openvibers/openvibe.media Commits
Restore drills: MEDIA_DRILL=1 makes Media a read-only instance on a restored copy of media.db, so `ovhost drill media` can boot one next to production. It refuses to start unless DB_PATH is set and outside the checkout and /opt/openvibe.media, HOST is loopback and PORT is not 4100, before the database is opened. It then starts only its HTTP server: no app seeding or JWKS refresh, no tiering sweep, health job, junk sweep, clip re-cuts, copy verification, jobs worker, disk guardian, thumbnail cleanup, object purge, backfill, orphan-recording finalize or Events relay, and webhooks are never sent (their URLs in the apps table are production's). It writes no file and creates no directory, and never opens a file path from the database: every byte route (/v and /c bytes, /t, /a, /f, /o, paste screenshots, live frames, legacy thumbnails) answers 503 while watch pages and /browse still render. Everything but GET/HEAD/OPTIONS answers 403, /auth/* 503, and net/fetch/child_process/dgram/listen are guarded so nothing leaves the process (only git runs: no ffmpeg or ffprobe). /api/ready drops the storage and remote-tier checks and says mode: drill. test/drill-mode.test.js boots the real server that way against a copy whose app has a webhook URL and checks each point
Changes
@@@ -11,6 +11,11 @@ MEDIA_PUBLIC_URL=https://openvibe.media# Database (better-sqlite3, WAL)DB_PATH=./data/media.db++# Restore-drill mode, set only by `ovhost drill media` (OpenVibe.Host docs/restore-drills.md): reads++# only from a restored copy, no stored bytes, no jobs, webhooks, events or ffmpeg; refuses to start++# unless DB_PATH is outside the checkout, HOST is loopback and PORT is not 4100 (server/drill.js).++# MEDIA_DRILL=1++# Storage paths (all created on boot)VOD_PATH=./data/vodsCLIPS_PATH=./data/clips@@@ -23,6 +23,7 @@ server/config.js env config (PORT, DB_PATH, *_PATH, MEDIA_B2_*/R2_*, RTP pool, OV_NETWORK_URL)auth.js tenancy + auth middleware, JWKS fetch, app seedingwebhooks.js HMAC-signed outbound webhooks++ drill.js MEDIA_DRILL: restore-drill mode (reads only, no bytes, no jobs, nothing leaves)db/schema.sql apps, vods, clips, pastes(+likes/comments), files, content_views, media_settings,media_objects/locations/relationships/variants/jobs/holds/invariant_violationsdb/database.js better-sqlite3 helpers, all app_id-scoped; legacy row importer@@@ -265,6 +266,29 @@ rolled-back change; the webhook follows the commit and carries `event_id` = that(absent when the outbox is off), so an app that reads both paths (Live during its webhook →Events transition) handles each outcome once. Webhooks stay until every consumer reads Events.++## Restore drills (`MEDIA_DRILL=1`)++++`ovhost drill media` (OpenVibe.Host `docs/restore-drills.md`) restores `media.db` from the latest++backup and starts a second Media from this checkout on 127.0.0.1:14100 with `MEDIA_DRILL=1` and++`DB_PATH` on the copy. In that mode (`server/drill.js`) Media:++++- refuses to start unless `DB_PATH` is set and outside the checkout and `/opt/openvibe.media`, `HOST`++ is loopback and `PORT` is not 4100 (before the database is opened);++- starts only its HTTP server: no app seeding, JWKS refresh, tiering sweep, health job, junk sweep,++ clip re-cuts, copy verification, jobs worker, disk guardian, thumbnail cleanup, object purge,++ backfill, orphan-recording finalize or Events relay; webhooks are never sent;++- writes no file and creates no directory (the storage checks leave `/api/ready`, as do the remote++ tiers; it reports `"mode": "drill"`);++- never opens a file path from the database: every byte route (`/v` and `/c` bytes, `/t`, `/a`, `/f`,++ `/o`, `/p/:slug/screenshot`, `/live/:sel/frame.jpg`, `/api/thumbnails/:name`) answers 503;++ watch pages and the `/browse` index still render from the copy;++- connects to nothing and runs no program but `git` (no ffmpeg or ffprobe), opens no UDP socket and++ listens on nothing but its port;++- answers 403 to every method but GET, HEAD and OPTIONS, and 503 to `/auth/*`.++++`test/drill-mode.test.js` boots the real server that way against a copy whose app has a webhook URL++and checks each point.++## Health, readiness and metrics| path | behavior |@@@ -21,7 +21,7 @@ const db = require('../db/database');const { tenantAuth } = require('../auth');const ASSET_DIR = config.assets.path;--if (!fs.existsSync(ASSET_DIR)) fs.mkdirSync(ASSET_DIR, { recursive: true });++if (!require('../drill').enabled && !fs.existsSync(ASSET_DIR)) fs.mkdirSync(ASSET_DIR, { recursive: true }); // a restore drill writes nothingconst upload = multer({ storage: multer.memoryStorage(), limits: { fileSize: 10 * 1024 * 1024 } });const router = express.Router({ mergeParams: true });nnew file mode 100644@@@ -0,0 +1,243 @@++'use strict';++/**++ * Restore-drill mode: MEDIA_DRILL=1 (`ovhost drill media`, see OpenVibe.Host docs/restore-drills.md).++ *++ * A drill starts a second Media from the production checkout, with the production env file, against++ * a restored copy of media.db on a spare loopback port, and compares its public reads with++ * production's. That instance must serve reads from the copy and do nothing else. With MEDIA_DRILL:++ *++ * - assertSafe(): Media refuses to start unless DB_PATH is set and lies outside the checkout and++ * outside /opt/openvibe.media (so it cannot be production's media.db), PORT is set and is not++ * production's 4100, and HOST is loopback. It runs before the database is opened.++ * - installGuards(): no outbound connection (net.Socket#connect, fetch), so no webhook, event, B2/R2++ * or Network call leaves the process; no program other than git (child_process: no ffmpeg or++ * ffprobe ever; git only reads the checkout for /release.json); no UDP socket (RTP ingest); nothing++ * listens but the drill's own HTTP port.++ * - readOnly: 403 for every request that is not GET, HEAD or OPTIONS, on every path.++ * - refuseBytes(): every route that would stream a stored file (or redirect to its B2/R2 copy)++ * answers 503. The database holds absolute paths to production's files; a drill never opens them.++ * Watch pages and the media index render from the database only and still answer.++ * - server/index.js opens the database and serves HTTP, and starts nothing else: no app seeding, no++ * JWKS refresh, no health job, junk sweep, tiering sweep, clip re-cuts, copy verification, jobs++ * worker (thumbnails, split/remux, invariant scans), disk guardian, thumbnail cleanup, object purge,++ * backfill, orphan-recording finalize or Events relay; webhooks are never sent. Timers that modules++ * start when they are loaded check `enabled` as well. Readiness does not probe storage (a drill++ * writes none) or the remote tiers.++ *++ * The guards are there so that anything this list missed fails closed instead of reaching production.++ */++const path = require('path');++const fs = require('fs');++++const TRUE = new Set(['1', 'true', 'on', 'yes']);++++/** MEDIA_DRILL as a switch (1/true/on/yes). */++function parse(value) {++ return TRUE.has(String(value == null ? '' : value).trim().toLowerCase());++}++++const enabled = parse(process.env.MEDIA_DRILL);++++const REPO_ROOT = path.resolve(__dirname, '..');++/** Where production runs (OpenVibe.Host inventory: repo /opt/openvibe.media, database data/media.db). */++const PRODUCTION_ROOT = '/opt/openvibe.media';++const PRODUCTION_PORT = 4100;++const LOOPBACK_HOSTS = new Set(['127.0.0.1', '::1', 'localhost', '::ffff:127.0.0.1']);++++function isLoopbackHost(host) {++ const h = String(host || '').trim().toLowerCase().replace(/^\[|\]$/g, '');++ return LOOPBACK_HOSTS.has(h) || /^127(\.\d{1,3}){3}$/.test(h);++}++++/** The real location of p, following symlinks on the part of it that exists. */++function realish(p) {++ let head = path.resolve(p);++ const tail = [];++ for (;;) {++ try { return path.join(fs.realpathSync(head), ...tail.reverse()); } catch { /* not there (yet) */ }++ const parent = path.dirname(head);++ if (parent === head) return path.resolve(p);++ tail.push(path.basename(head));++ head = parent;++ }++}++++function inside(p, root) {++ const rel = path.relative(root, p);++ return rel === '' || (!!rel && !rel.startsWith('..') && !path.isAbsolute(rel));++}++++/**++ * Why this environment is not safe for a drill ([] = safe). Pure, for tests.++ * `repoRoot` is the checkout the process runs from; `cwd` resolves relative paths.++ */++function problems(env = process.env, { repoRoot = REPO_ROOT, cwd = process.cwd() } = {}) {++ const out = [];++ const roots = [...new Set([repoRoot, realish(repoRoot), PRODUCTION_ROOT])];++ const productionDb = new Set([path.join(PRODUCTION_ROOT, 'data', 'media.db'), path.join(repoRoot, 'data', 'media.db'), path.join(cwd, 'data', 'media.db')]);++ if (!env.DB_PATH) {++ out.push('DB_PATH is not set; it must name the restored copy of media.db');++ } else {++ const abs = path.resolve(cwd, env.DB_PATH);++ const real = realish(abs);++ const root = roots.find((r) => inside(abs, r) || inside(real, r));++ if (productionDb.has(abs) || productionDb.has(real)) out.push(`DB_PATH (${abs}) is production's database; point it at the restored copy`);++ else if (root) out.push(`DB_PATH (${abs}${real !== abs ? ` → ${real}` : ''}) is inside ${root}; a drill writes nothing in the checkout or in production's data`);++ }++ if (!isLoopbackHost(env.HOST)) out.push(`HOST (${env.HOST || 'unset, i.e. 0.0.0.0'}) is not loopback; a drill binds 127.0.0.1 only`);++ const port = Number(env.PORT);++ if (!env.PORT || !Number.isInteger(port) || port < 1 || port > 65535) out.push('PORT is not set; it must be the drill\'s own port');++ else if (port === PRODUCTION_PORT) out.push(`PORT ${port} is production's port`);++ return out;++}++++class DrillRefused extends Error {++ constructor(list) {++ super(`MEDIA_DRILL: refusing to start a restore-drill instance: ${list.join('; ')}`);++ this.code = 'MEDIA_DRILL_UNSAFE';++ this.problems = list;++ }++}++++/** Throws DrillRefused unless the environment is safe for a drill. */++function assertSafe(env = process.env, opts) {++ const list = problems(env, opts);++ if (list.length) throw new DrillRefused(list);++}++++// ── Guards ───────────────────────────────────────────────────++++const blocked = []; // what the guards refused (newest last, bounded): { at, kind, target }++function note(kind, target) {++ blocked.push({ at: new Date().toISOString(), kind, target: String(target).slice(0, 200) });++ if (blocked.length > 50) blocked.shift();++ console.warn(`[Drill] refused ${kind}: ${target}`);++}++++function drillError(message, code) {++ const err = new Error(`${message} (MEDIA_DRILL: a restore-drill instance does not do this)`);++ err.code = code;++ return err;++}++++/** Where a net.Socket#connect call is going: { host, port } or { path }. */++function connectTarget(args) {++ let a = args[0];++ if (Array.isArray(a)) a = a[0]; // net.connect() hands Socket#connect its normalized [options, cb]++ if (a && typeof a === 'object') return a.path != null ? { path: String(a.path) } : { host: a.host, port: Number(a.port) };++ if (typeof a === 'string' && !/^\d+$/.test(a)) return { path: a };++ return { host: typeof args[1] === 'string' ? args[1] : undefined, port: Number(a) };++}++++const ALLOWED_PROGRAMS = new Set(['git']);++++function programOf(fn, args) {++ const first = args[0];++ if (fn === 'exec' || fn === 'execSync') return String(first || '').trim().split(/\s+/)[0] || '';++ if (fn === 'fork') return 'node';++ return String(first || '');++}++++let installed = false;++++/**++ * Replace the process's ways out with ones that refuse (drill only). `port` is the drill's HTTP++ * port: the one address the process may connect to (itself, on loopback) and the one it may listen on.++ */++function installGuards({ port = Number(process.env.PORT) } = {}) {++ if (installed) return;++ installed = true;++ const net = require('net');++ const util = require('util');++++ const isSelf = (t) => t && t.path == null && t.port === port && (t.host == null || isLoopbackHost(t.host));++++ const connect = net.Socket.prototype.connect;++ net.Socket.prototype.connect = function drillConnect(...args) {++ const t = connectTarget(args);++ if (isSelf(t)) return connect.apply(this, args);++ const where = t.path != null ? t.path : `${t.host || 'localhost'}:${t.port}`;++ note('connection', where);++ const err = drillError(`connect ECONNREFUSED ${where}`, 'ECONNREFUSED');++ err.syscall = 'connect';++ process.nextTick(() => this.destroy(err));++ return this;++ };++++ if (typeof globalThis.fetch === 'function') {++ const fetch = globalThis.fetch;++ globalThis.fetch = function drillFetch(input, init) {++ let url = null;++ try { url = new URL(typeof input === 'string' || input instanceof URL ? String(input) : input.url); } catch { /* */ }++ if (url && isSelf({ host: url.hostname, port: Number(url.port || (url.protocol === 'https:' ? 443 : 80)) })) return fetch(input, init);++ note('fetch', url ? url.origin + url.pathname : String(input));++ return Promise.reject(new TypeError('fetch failed', { cause: drillError(`connect ECONNREFUSED ${url ? url.host : ''}`, 'ECONNREFUSED') }));++ };++ }++++ const cp = require('child_process');++ for (const fn of ['spawn', 'spawnSync', 'exec', 'execSync', 'execFile', 'execFileSync', 'fork']) {++ const orig = cp[fn];++ if (typeof orig !== 'function') continue;++ const check = (args) => {++ const program = programOf(fn, args);++ if (ALLOWED_PROGRAMS.has(path.basename(program))) return;++ note('program', program);++ throw drillError(`${fn}(${program}) refused`, 'EDRILL');++ };++ const guarded = function (...args) { check(args); return orig.apply(this, args); };++ const custom = orig[util.promisify.custom];++ if (custom) guarded[util.promisify.custom] = (...args) => { check(args); return custom(...args); };++ cp[fn] = guarded;++ }++++ const dgram = require('dgram');++ dgram.createSocket = function drillUdp() {++ note('udp socket', 'dgram.createSocket');++ throw drillError('UDP socket refused', 'EDRILL');++ };++++ const listen = net.Server.prototype.listen;++ net.Server.prototype.listen = function drillListen(...args) {++ const a = args[0];++ const p = a && typeof a === 'object' ? Number(a.port) : Number(a);++ if (p === port && !(a && typeof a === 'object' && (a.fd != null || a.path != null))) return listen.apply(this, args);++ note('listener', a && typeof a === 'object' ? JSON.stringify(a) : String(a));++ throw drillError(`listen ${a && typeof a === 'object' ? JSON.stringify(a) : a} refused`, 'EDRILL');++ };++}++++// ── HTTP ─────────────────────────────────────────────────────++++/** Express middleware: reads only. */++function readOnly(req, res, next) {++ if (req.method === 'GET' || req.method === 'HEAD' || req.method === 'OPTIONS') return next();++ res.status(403).json({ error: 'This is a restore-drill instance (MEDIA_DRILL): it serves reads only', code: 'media.drill_read_only' });++}++++/**++ * Answer a request for stored bytes: a drill never opens a file path from the database (they are++ * production's files) and never hands out a B2/R2 URL for one. Returns true when it answered.++ */++function refuseBytes(res) {++ if (!enabled) return false;++ if (!res.headersSent) {++ res.status(503).set('Retry-After', '3600').json({ error: 'This is a restore-drill instance (MEDIA_DRILL): it serves no stored media', code: 'media.drill_no_bytes' });++ }++ return true;++}++++module.exports = {++ enabled,++ parse,++ problems,++ assertSafe,++ DrillRefused,++ installGuards,++ readOnly,++ refuseBytes,++ blocked,++ isLoopbackHost,++ PRODUCTION_ROOT,++ REPO_ROOT,++};@@@ -56,6 +56,8 @@ function init({fetchImpl, intervalMs = 2000,} = {}) {if (outbox) return outbox;++ // A restore drill (MEDIA_DRILL) relays nothing: its outbox rows describe a restored copy.++ if (require('./drill').enabled) return null;if (process.env.EVENTS_PUBLISH === 'off' || !eventsUrl || !clientSecret) return null;const tokens = createServiceTokenClient({ tokenUrl: `${networkUrl}/oauth/token`, clientId, clientSecret, fetch: fetchImpl });const client = createClient({ baseUrls: { events: eventsUrl }, tokenProvider: tokens, fetch: fetchImpl, retries: 0 });@@@ -32,8 +32,9 @@ const { tenantAuth, tenantCors } = require('../auth');const router = express.Router({ mergeParams: true });router.use(tenantCors);++// (multer creates `dest` when it is built; a restore drill takes no upload and creates no directory.)const upload = multer({-- dest: path.join(config.files.path, '.tmp'),++ ...(require('../drill').enabled ? { storage: multer.memoryStorage() } : { dest: path.join(config.files.path, '.tmp') }),limits: { fileSize: config.files.maxSizeMb * 1024 * 1024 },});@@@ -8,6 +8,23 @@*/'use strict';++// Restore-drill mode (MEDIA_DRILL=1, `ovhost drill media`): refuse an unsafe environment before the++// database is opened, then cut every way out of the process but its own HTTP port. The instance++// serves reads from a restored copy of media.db, never opens a stored file and starts nothing else++// (server/drill.js).++require('dotenv').config();++const drill = require('./drill');++if (drill.enabled) {++ try {++ drill.assertSafe();++ } catch (err) {++ console.error(`[Drill] ${err.message}`);++ process.exit(1);++ }++ drill.installGuards();++ console.log(`[Drill] MEDIA_DRILL: restore-drill instance on ${process.env.HOST}:${process.env.PORT}, database ${require('path').resolve(process.env.DB_PATH)}. Reads only; no stored bytes, background work or outbound connections.`);++}++const fs = require('fs');const express = require('express');const config = require('./config');@@@ -16,14 +33,19 @@ const auth = require('./auth');// ── Bootstrap ────────────────────────────────────────────────--for (const dir of [config.vod.path, config.vod.clipsPath, config.pastes.path, config.thumbnails.path, config.files.path, config.objects.path]) {-- if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true });++// A drill creates no storage directory (it writes no file) and seeds nothing: the copy is production's.++if (!drill.enabled) {++ for (const dir of [config.vod.path, config.vod.clipsPath, config.pastes.path, config.thumbnails.path, config.files.path, config.objects.path]) {++ if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true });++ }}db.getDb(); // init schema (WAL)--auth.seedApps(); // upsert MEDIA_APPS_SEED / MEDIA_APP_KEYS--auth.ensureTokenOnlyApps(); // tenants reached only with Network service tokens (community)--auth.startJwksRefresh();++if (!drill.enabled) {++ auth.seedApps(); // upsert MEDIA_APPS_SEED / MEDIA_APP_KEYS++ auth.ensureTokenOnlyApps(); // tenants reached only with Network service tokens (community)++ auth.startJwksRefresh(); // a drill asks Network for nothing (app keys still verify against the copy)++}const app = express();app.set('trust proxy', true);@@@ -31,6 +53,9 @@ app.set('trust proxy', true);const release = require('openvibe-shared/release').createRelease({ service: 'media', root: require('path').join(__dirname, '..') });const observability = require('./observability');const instrumented = observability.instrument(app, { release: release.release });++// A restore-drill instance answers reads only: 403 for every other method, on every path (before the++// raw-body upload routes below).++if (drill.enabled) app.use(drill.readOnly);// Object API v2 content uploads read the raw request body, so they sit ahead of the body parsers.app.put('/api/v2/:app/objects/:id/content', ...require('./objects/routes').contentHandlers);app.put('/api/v2/:app/objects/:id/multipart/:uploadId/parts/:n', ...require('./objects/routes').partHandlers);@@@ -69,6 +94,8 @@ const userAuthConfig = {cookies: { secure: /^https:/.test(config.publicUrl) },oauth: { clientId: process.env.OV_OAUTH_CLIENT_ID || 'media', clientSecret: process.env.OV_OAUTH_CLIENT_SECRET || '', redirectUri: process.env.OV_OAUTH_REDIRECT_URI || `${config.publicUrl}/auth/callback` },};++// A restore drill signs nobody in: that would redeem codes at Network.++if (drill.enabled) app.use('/auth', (req, res) => res.status(503).json({ error: 'This is a restore-drill instance (MEDIA_DRILL): no sign-in', code: 'media.drill_no_sign_in' }));app.use('/auth', userAuth.createAuthRoutes(userAuthConfig, userAuth.createAuthClient(userAuthConfig)));app.use(express.urlencoded({ extended: true, limit: '2mb' }));@@@ -108,6 +135,7 @@ app.get('/healthz', (req, res) => {recorder: require('./vod/recorder'),events: require('./events'),remote: { configured: (n) => vodStorageForReady.providerConfigured(n), probe: (n) => vodStorageForReady.probeProvider(n) },++ drill: drill.enabled,});}@@@ -134,7 +162,7 @@ app.get('/api/v1/:app/stats/series/:metric', auth.tenantAuth(), (req, res) => {});try { require('./views/service').ensureSchema(); } catch (e) { console.warn('[Views] schema:', e.message); }--setInterval(() => { try { const n = require('./views/service').prune(30); if (n) console.log(`[Views] pruned ${n} stale visit row(s)`); } catch { /* */ } }, 12 * 3600 * 1000);++if (!drill.enabled) setInterval(() => { try { const n = require('./views/service').prune(30); if (n) console.log(`[Views] pruned ${n} stale visit row(s)`); } catch { /* */ } }, 12 * 3600 * 1000);app.use('/api/v1/:app/views', require('./views/routes'));app.use('/api/v1/:app/vods', require('./vod/routes'));app.use('/api/v1/:app/clips', require('./vod/clips-routes'));@@@ -168,56 +196,61 @@ function every(ms, fn) {timers.push(t);}--vodStorage.checkProviders()-- .then(() => vodStorage.migrateLegacy().catch(() => {}))-- .catch(err => console.warn('[Boot] Provider check failed:', err.message));--vodStorage.start(); // tiering sweep--healthJob.start(); // health scan + quarantine cleanup--require('./vod/clip-jobs').start(); // failed clip re-cuts (auto-retry, hot-fetch)--require('./objects/verify-job').start(); // scheduled copy verification (bounded batches; never deletes)--require('./jobs/worker').start(); // media_jobs worker (light + heavy lanes; proposals wait for their owner)--// Descriptor watchdog: a leak here once pinned 80 GB of deleted recordings to the disk.--every(5 * 60 * 1000, () => {-- try {-- const n = fs.readdirSync('/proc/self/fd').length;-- if (n > 1500) console.warn(`[Boot] ${n} open file descriptors — investigate a stream/handle leak (lsof -p ${process.pid} +L1)`);-- } catch { /* not linux */ }--});--every(60 * 1000, () => recorder.checkDisk()); // disk guardian--every(60 * 60 * 1000, () => thumbService.cleanupOldThumbnails()); // stale live thumbs--// Object model: purge native objects whose soft-delete retention has passed (held ones are kept).--every(60 * 60 * 1000, () => {-- try { const n = require('./objects/model').purgeExpired(); if (n) console.log(`[Objects] Purged ${n} expired deleted object(s)`); } catch (err) { console.warn('[Objects] purge:', err.message); }-- // Incomplete multipart uploads past MEDIA_MULTIPART_TTL_HOURS: their parts are deleted (the objects stay uploading).-- try { const r = require('./objects/multipart').purgeExpired(); if (r.expired || r.orphan_dirs) console.log(`[Objects] Multipart: ${r.expired} expired session(s), ${r.orphan_dirs} orphan part dir(s) removed`); } catch (err) { console.warn('[Objects] multipart purge:', err.message); }--});--// Project rows that have no media_object yet (first boot after the upgrade: all of them).--setTimeout(() => {-- try {-- const bf = require('./objects/backfill');-- const r = bf.backfill({ onlyMissing: true });-- if (r.totals.created || r.totals.updated || r.errors.length) console.log(`[Objects] Backfill: ${bf.summarize(r)}`);-- } catch (err) { console.warn('[Objects] Backfill failed:', err.message); }--}, 15 * 1000).unref?.();----// Recover from an unclean shutdown: rows stuck in is_recording with no live--// ffmpeg are finalized from whatever hit the disk.--setTimeout(() => {-- try {-- const stuck = db.all('SELECT id FROM vods WHERE is_recording = 1');-- for (const row of stuck) {-- if (recorder.isRecording(row.id)) continue;-- console.log(`[Boot] Finalizing orphaned recording vod ${row.id}`);-- require('./vod/finalize').finalizeVod(row.id).catch(() => {});++// A restore drill starts none of this: every job below reads or writes stored files, runs ffmpeg,++// talks to B2/R2 or announces outcomes to Live.++if (!drill.enabled) {++ vodStorage.checkProviders()++ .then(() => vodStorage.migrateLegacy().catch(() => {}))++ .catch(err => console.warn('[Boot] Provider check failed:', err.message));++ vodStorage.start(); // tiering sweep++ healthJob.start(); // health scan + quarantine cleanup++ require('./vod/clip-jobs').start(); // failed clip re-cuts (auto-retry, hot-fetch)++ require('./objects/verify-job').start(); // scheduled copy verification (bounded batches; never deletes)++ require('./jobs/worker').start(); // media_jobs worker (light + heavy lanes; proposals wait for their owner)++ // Descriptor watchdog: a leak here once pinned 80 GB of deleted recordings to the disk.++ every(5 * 60 * 1000, () => {++ try {++ const n = fs.readdirSync('/proc/self/fd').length;++ if (n > 1500) console.warn(`[Boot] ${n} open file descriptors — investigate a stream/handle leak (lsof -p ${process.pid} +L1)`);++ } catch { /* not linux */ }++ });++ every(60 * 1000, () => recorder.checkDisk()); // disk guardian++ every(60 * 60 * 1000, () => thumbService.cleanupOldThumbnails()); // stale live thumbs++ // Object model: purge native objects whose soft-delete retention has passed (held ones are kept).++ every(60 * 60 * 1000, () => {++ try { const n = require('./objects/model').purgeExpired(); if (n) console.log(`[Objects] Purged ${n} expired deleted object(s)`); } catch (err) { console.warn('[Objects] purge:', err.message); }++ // Incomplete multipart uploads past MEDIA_MULTIPART_TTL_HOURS: their parts are deleted (the objects stay uploading).++ try { const r = require('./objects/multipart').purgeExpired(); if (r.expired || r.orphan_dirs) console.log(`[Objects] Multipart: ${r.expired} expired session(s), ${r.orphan_dirs} orphan part dir(s) removed`); } catch (err) { console.warn('[Objects] multipart purge:', err.message); }++ });++ // Project rows that have no media_object yet (first boot after the upgrade: all of them).++ setTimeout(() => {++ try {++ const bf = require('./objects/backfill');++ const r = bf.backfill({ onlyMissing: true });++ if (r.totals.created || r.totals.updated || r.errors.length) console.log(`[Objects] Backfill: ${bf.summarize(r)}`);++ } catch (err) { console.warn('[Objects] Backfill failed:', err.message); }++ }, 15 * 1000).unref?.();++++ // Recover from an unclean shutdown: rows stuck in is_recording with no live++ // ffmpeg are finalized from whatever hit the disk.++ setTimeout(() => {++ try {++ const stuck = db.all('SELECT id FROM vods WHERE is_recording = 1');++ for (const row of stuck) {++ if (recorder.isRecording(row.id)) continue;++ console.log(`[Boot] Finalizing orphaned recording vod ${row.id}`);++ require('./vod/finalize').finalizeVod(row.id).catch(() => {});++ }++ } catch (err) {++ console.warn('[Boot] Orphan finalize sweep failed:', err.message);}-- } catch (err) {-- console.warn('[Boot] Orphan finalize sweep failed:', err.message);-- }--}, 5000).unref?.();++ }, 5000).unref?.();++}// ── Listen + graceful shutdown ───────────────────────────────const server = app.listen(config.port, config.host, () => {++ if (drill.enabled) { console.log(`[Drill] Ready: http://${config.host}:${config.port} (reads only)`); return; }console.log(`[Media] OpenVibe.Media listening on ${config.host}:${config.port} (${config.nodeEnv})`);console.log(`[Media] Data: db=${config.db.path} vods=${config.vod.path} clips=${config.vod.clipsPath}`);console.log(`[Media] RTP ingest pool: udp ${config.rtp.portMin}-${config.rtp.portMax} (127.0.0.1)`);@@@ -225,6 +258,9 @@ const server = app.listen(config.port, config.host, () => {try { require('./events').init(); } catch (err) { console.warn('[Events] not started:', err.message); }});++// A drill whose port is taken stops instead of running unready.++if (drill.enabled) server.once('error', (err) => { console.error(`[Drill] HTTP server: ${err.message}`); process.exit(1); });++let shuttingDown = false;function shutdown(signal) {if (shuttingDown) return;@@@ -599,6 +599,8 @@ const INLINE = /^(image\/(?!svg)|video\/|audio\/|application\/pdf$|text\/plain$)const publicRouter = express.Router();publicRouter.get('/:id', async (req, res) => {++ // A restore drill (MEDIA_DRILL) serves no stored bytes, local or by a B2/R2 redirect.++ if (require('../drill').refuseBytes(res)) return;try {const obj = model.getObject(String(req.params.id || ''));if (!obj) return res.status(404).json({ error: 'Not found' });@@@ -94,16 +94,19 @@ function domainMetrics(registry, { db, recorder, events }) {});}--function createMediaReadiness({ release, db, config, auth, recorder, events, remote }) {++function createMediaReadiness({ release, db, config, auth, recorder, events, remote, drill = false }) {const checks = [{ name: 'db', required: true, check: () => { const r = db.get('SELECT COUNT(*) AS n FROM apps'); return { ok: Number.isInteger(r.n), detail: { apps: r.n } }; } },];-- for (const [name, dir] of Object.entries(storageDirs(config))) {-- checks.push({ name: `storage_${name}`, required: true, check: () => writable(dir) });++ // A restore drill (MEDIA_DRILL) writes no file and asks no remote tier anything: neither is checked.++ if (!drill) {++ for (const [name, dir] of Object.entries(storageDirs(config))) {++ checks.push({ name: `storage_${name}`, required: true, check: () => writable(dir) });++ }}checks.push({ name: 'network_jwks', required: false, description: 'user sign-in and service tokens (app keys work without it)', check: () => auth.jwksLoaded() || 'Network public key not loaded yet' });for (const name of ['b2', 'r2']) {-- if (!remote.configured(name)) continue; // not configured: no check, rather than a pretend pass++ if (drill || !remote.configured(name)) continue; // not configured: no check, rather than a pretend passchecks.push({ name: `remote_${name}`, required: false, cacheMs: 60 * 1000, timeoutMs: 5000, check: () => remote.probe(name) });}checks.push({@@@ -129,7 +132,7 @@ function createMediaReadiness({ release, db, config, auth, recorder, events, remservice: 'media',release,checks,-- details: () => ({ recordings_in_progress: recorder.activeCount() }),++ details: () => ({ ...(drill ? { mode: 'drill' } : {}), recordings_in_progress: recorder.activeCount() }),});}@@@ -56,7 +56,7 @@ router.use((req, res, next) => {// ── Screenshot upload storage ───────────────────────────────const SCREENSHOTS_DIR = path.join(config.pastes.path, 'screenshots');--if (!fs.existsSync(SCREENSHOTS_DIR)) fs.mkdirSync(SCREENSHOTS_DIR, { recursive: true });++if (!require('../drill').enabled && !fs.existsSync(SCREENSHOTS_DIR)) fs.mkdirSync(SCREENSHOTS_DIR, { recursive: true }); // a restore drill writes nothingconst MIME_TO_EXT = { 'image/png': '.png', 'image/jpeg': '.jpg', 'image/webp': '.webp', 'image/gif': '.gif' };const screenshotStorage = multer.diskStorage({@@@ -706,13 +706,14 @@ router.post('/:slug/copy', tenantAuth({ allowUser: true }), (req, res) => {// Rate-limit state: IP → timestamp of last commentconst commentCooldowns = new Map();--const _cooldownSweep = setInterval(() => {++// (A restore drill takes no comments and runs no timers.)++const _cooldownSweep = require('../drill').enabled ? null : setInterval(() => {const now = Date.now();for (const [ip, ts] of commentCooldowns) {if (now - ts > 120_000) commentCooldowns.delete(ip);}}, 600_000);--if (_cooldownSweep.unref) _cooldownSweep.unref();++if (_cooldownSweep && _cooldownSweep.unref) _cooldownSweep.unref();router.get('/:slug/comments', tenantAuth({ allowUser: true }), (req, res) => {try {@@@ -27,6 +27,9 @@ const db = require('../db/database');const tools = require('../vod/media-tools');const { optionalIdentity } = require('../auth');const pages = require('./pages');++// A restore drill (MEDIA_DRILL) serves no stored bytes: every byte route answers 503 before it looks++// at a file path (the database's paths are production's files). Watch pages still render.++const drill = require('../drill');const router = express.Router();@@@ -71,6 +74,7 @@ function canAccessPrivate(record, req) {* aborts a range request). Deleted-but-open VOD files once held 80 GB of disk that way.*/function sendFileStream(res, filePath, opts) {++ if (drill.enabled) { try { res.destroy(); } catch { /* */ } return null; } // unreachable: every caller refuses firstconst stream = fs.createReadStream(filePath, opts);const done = () => { try { stream.destroy(); } catch { /* */ } };res.on('close', done); res.on('error', done); res.on('finish', done);@@@ -80,6 +84,7 @@ function sendFileStream(res, filePath, opts) {}function streamFileWithRange(req, res, filePath, extraHeaders = {}) {++ if (drill.refuseBytes(res)) return;const stat = fs.statSync(filePath);const range = req.headers.range;const ext = path.extname(filePath).toLowerCase();@@@ -132,6 +137,7 @@ async function serveMediaRecord(kind, record, req, res) {res.set('Cache-Control', visibility === 'public' ? 'public, max-age=60' : 'private, no-store');return res.type('html').send(pages.renderWatchPage(kind, record));}++ if (drill.refuseBytes(res)) return;trackUniqueView(kind, record.id, req, record.user_id);@@@ -226,6 +232,7 @@ router.get('/og-image.png', (req, res) => {// ── Thumbnails ───────────────────────────────────────────────router.get('/t/:id', (req, res) => {++ if (drill.refuseBytes(res)) return;require('../thumbnails/thumbnail-service').serveThumbnail(req, res);});@@@ -238,6 +245,7 @@ router.get('/t/:id', (req, res) => {// consumers can key off the status code (or pass ?format=json).// Cached 5s per slot (that cache IS the rate limit); CORS-open.router.get('/live/:sel/frame.jpg', async (req, res) => {++ if (drill.refuseBytes(res)) return; // ffprobe + ffmpeg on the live recordingconst frames = require('../thumbnails/live-frame-service');const wantJson = String(req.query.format || '') === 'json';const sendCard = async (status, label, subtitle) => {@@@ -299,6 +307,7 @@ router.get('/live/:sel/frame.jpg', async (req, res) => {// ── App assets (emotes / sounds) by id ───────────────────────router.get('/a/:id', (req, res) => {++ if (drill.refuseBytes(res)) return;try {const a = db.getAssetById(parseInt(req.params.id, 10));if (!a || !a.file_path || !fs.existsSync(a.file_path)) return res.status(404).json({ error: 'Not found' });@@@ -359,6 +368,7 @@ router.get('/v/:id/transcript.json', _devDataRoute((req) => {// regenerated, so resolve the vod/clip id from the name and redirect to the// current canonical URL; serve the exact file when it still exists.router.get('/api/thumbnails/:name', (req, res) => {++ if (drill.refuseBytes(res)) return;const name = path.basename(String(req.params.name || ''));const m = /^(vod|clip)-(\d+)-\d+\.(?:jpg|jpeg|png)$/i.exec(name);if (m) {@@@ -382,6 +392,7 @@ router.get('/api/thumbnails/:name', (req, res) => {// basenames on disk and NO files-table rows, so this serves straight from// PASTES_PATH/screenshots for exactly that namespace.router.get('/f/screenshots/:name', (req, res) => {++ if (drill.refuseBytes(res)) return;try {const name = path.basename(String(req.params.name || ''));const dir = path.join(require('../config').pastes.path, 'screenshots');@@@ -398,6 +409,7 @@ router.get('/f/screenshots/:name', (req, res) => {// ── Files ────────────────────────────────────────────────────router.get('/f/:key', (req, res) => {++ if (drill.refuseBytes(res)) return;try {const row = db.getFileByKey(String(req.params.key));if (!row) return res.status(404).json({ error: 'Not found' });@@@ -524,6 +536,7 @@ router.get('/p/:slug/raw', (req, res) => {});router.get('/p/:slug/screenshot', (req, res) => {++ if (drill.refuseBytes(res)) return;try {const paste = db.getPasteBySlug(String(req.params.slug));// Pastes made since the move live in Community only: send an unknown slug there, like@@@ -29,7 +29,8 @@ const THUMB_WIDTH = 640;const THUMB_QUALITY = 6; // ffmpeg qscale:v (2=best, 31=worst, 6 is good balance)const CLIENT_THUMB_WRITE_MIN_INTERVAL_MS = 15000;--if (!fs.existsSync(THUMB_DIR)) {++// (A restore drill writes nothing, directories included.)++if (!require('../drill').enabled && !fs.existsSync(THUMB_DIR)) {fs.mkdirSync(THUMB_DIR, { recursive: true });}@@@ -267,6 +268,7 @@ function saveLiveThumbnail(appId, streamId, imageData) {// ── Serve Thumbnail File (public /t/:id) ─────────────────────function serveThumbnail(req, res) {++ if (require('../drill').refuseBytes(res)) return; // a restore drill serves no stored filetry {const filename = path.basename(req.params.id || req.params.filename || ''); // prevent traversalconst filePath = path.join(THUMB_DIR, filename);@@@ -89,8 +89,9 @@ function createAuthClient(config) {return client.verifyToken(token);}-- // Warm the key cache at boot (non-fatal if the Network is down)-- ensureKey().catch(() => {});++ // Warm the key cache at boot (non-fatal if the Network is down). A restore drill asks Network++ // for nothing (MEDIA_DRILL; its /auth routes answer 503).++ if (!require('./drill').enabled) ensureKey().catch(() => {});return { client, ensureKey, verify };}@@@ -150,7 +150,8 @@ class StreamRecorder {this._diskState = 'ok'; // 'ok' | 'warning' | 'critical'const vodDir = path.resolve(config.vod.path);-- if (!fs.existsSync(vodDir)) fs.mkdirSync(vodDir, { recursive: true });++ // A restore drill (MEDIA_DRILL) records nothing and writes nothing, directories included.++ if (!require('../drill').enabled && !fs.existsSync(vodDir)) fs.mkdirSync(vodDir, { recursive: true });}isRecording(vodId) {@@@ -322,7 +322,8 @@ router.get('/:id', tenantAuth({ allowUser: true }), async (req, res) => {}// Self-heal a missing duration on a finished local VOD.-- if (!vod.is_recording && (!vod.duration_seconds || vod.duration_seconds <= 0) && vod.file_path && fs.existsSync(vod.file_path)) {++ // (Never in a restore drill: it opens no stored file and runs no ffprobe.)++ if (!require('../drill').enabled && !vod.is_recording && (!vod.duration_seconds || vod.duration_seconds <= 0) && vod.file_path && fs.existsSync(vod.file_path)) {const duration = await tools.probeVodDuration(vod.file_path);if (duration > 0) {const fileSize = tools.getFileSizeSafe(vod.file_path);@@@ -49,6 +49,8 @@ async function _post(url, rawBody, signature) {* @param {string} [opts.eventId] the OpenVibe.Events event_id of the same outcome*/async function sendWebhook(appOrId, event, data, { eventId = null } = {}) {++ // A restore drill (MEDIA_DRILL) never calls an app: the URLs in its apps table are production's.++ if (require('./drill').enabled) return false;let app = appOrId;if (typeof appOrId === 'string') {try { app = db.getApp(appOrId); } catch { app = null; }nnew file mode 100644@@@ -0,0 +1,310 @@++'use strict';++/**++ * MEDIA_DRILL=1 (server/drill.js): a restore-drill instance (`ovhost drill media`, OpenVibe.Host++ * docs/restore-drills.md) serves reads from a restored copy of media.db and does nothing else.++ *++ * - It refuses to start when DB_PATH is production's media.db or inside the checkout, when HOST is++ * not loopback or PORT is production's 4100, before the database is opened (a real++ * `node server/index.js`, in a child process).++ * - Booted for real (server/index.js, in this process) on a copy whose `live` app has a webhook++ * URL, with production-like settings (EVENTS_URL + client secret, B2/R2, app seeding, VOD_PATH …):++ * one listener, its HTTP port on 127.0.0.1; no setInterval and no long timer from Media's code;++ * no program but git (no ffmpeg/ffprobe); no outbound connection; no webhook reaches the app;++ * nothing written or created outside the copy's directory.++ * - The endpoints the Host inventory compares (/release.json, the /browse index) and the watch pages++ * render from the copy; every route that would stream a stored file answers 503 without opening++ * the path the database gives it.++ * - Writes answer 403, sign-in 503, and the copy's rows stay as they were.++ *++ * Run: node test/drill-mode.test.js++ */++const assert = require('assert');++const crypto = require('crypto');++const fs = require('fs');++const http = require('http');++const net = require('net');++const os = require('os');++const path = require('path');++const { spawnSync, execFileSync } = require('child_process');++++const REPO = path.resolve(__dirname, '..');++const SERVER = path.join(REPO, 'server');++const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'media-drill-test-'));++// "Production": its database and its storage, with real files the database points at.++const PROD = path.join(tmp, 'prod');++const P = { vods: path.join(PROD, 'vods'), clips: path.join(PROD, 'clips'), pastes: path.join(PROD, 'pastes'), files: path.join(PROD, 'files'), objects: path.join(PROD, 'objects'), assets: path.join(PROD, 'assets'), thumbs: path.join(PROD, 'thumbnails') };++// The drill: the restored copy in its own directory; storage the way the Host inventory sets it++// (every directory but thumbnails moved under the drill directory, never created).++const DRILL = path.join(tmp, 'drill');++const DB_PATH = path.join(DRILL, 'db', 'media.db');++const STORAGE = path.join(DRILL, 'storage');++++let failures = 0;++async function check(name, fn) {++ try { await fn(); console.log(` ✓ ${name}`); } catch (err) { failures++; console.error(` ✗ ${name}\n ${err.stack || err.message}`); }++}++function freePort() {++ return new Promise((resolve) => { const s = net.createServer().listen(0, '127.0.0.1', () => { const p = s.address().port; s.close(() => resolve(p)); }); });++}++/** Every file and directory under dir with its size and mtime. */++function snapshot(dir) {++ const out = {};++ const walk = (d) => { for (const e of fs.readdirSync(d, { withFileTypes: true })) { const p = path.join(d, e.name); const s = fs.statSync(p); out[path.relative(dir, p)] = `${e.isDirectory() ? 'dir' : s.size}:${s.mtimeMs}`; if (e.isDirectory()) walk(p); } };++ walk(dir);++ return out;++}++++const drillMod = require('../server/drill'); // MEDIA_DRILL is unset here: `enabled` is false until the boot below++const safe = { DB_PATH, HOST: '127.0.0.1', PORT: '14100' };++++(async () => {++ console.log('drill-mode: environment guard');++ await check('MEDIA_DRILL parses like a switch and is off by default', () => {++ for (const v of ['1', 'true', 'on', 'yes']) assert.strictEqual(drillMod.parse(v), true, v);++ for (const v of ['', '0', 'false', 'off', undefined]) assert.strictEqual(drillMod.parse(v), false, String(v));++ assert.strictEqual(drillMod.enabled, false);++ });++ await check('a safe environment passes', () => assert.deepStrictEqual(drillMod.problems(safe), []));++ await check('refuses production\'s media.db (the checkout\'s, /opt/openvibe.media\'s, the relative default)', () => {++ for (const p of [path.join(REPO, 'data', 'media.db'), '/opt/openvibe.media/data/media.db']) {++ assert.ok(drillMod.problems({ ...safe, DB_PATH: p }).some((m) => /production's database/.test(m)), p);++ }++ assert.ok(drillMod.problems({ ...safe, DB_PATH: './data/media.db' }, { cwd: REPO }).some((m) => /production's database/.test(m)));++ assert.ok(drillMod.problems({ ...safe, DB_PATH: '' }).some((m) => /DB_PATH is not set/.test(m)));++ });++ await check('refuses a copy inside the checkout or /opt/openvibe.media, also through a symlink', () => {++ assert.ok(drillMod.problems({ ...safe, DB_PATH: path.join(REPO, 'drill.db') }).some((m) => /is inside/.test(m)));++ assert.ok(drillMod.problems({ ...safe, DB_PATH: '/opt/openvibe.media/drill/media.db' }).some((m) => /is inside \/opt\/openvibe\.media/.test(m)));++ const link = path.join(tmp, 'sneaky');++ fs.symlinkSync(REPO, link);++ assert.ok(drillMod.problems({ ...safe, DB_PATH: path.join(link, 'copy.db') }).some((m) => /is inside/.test(m)));++ fs.unlinkSync(link);++ });++ await check('refuses a non-loopback HOST and production\'s PORT', () => {++ for (const h of ['', '0.0.0.0', '::', '10.0.0.4']) assert.ok(drillMod.problems({ ...safe, HOST: h }).some((m) => /not loopback/.test(m)), h);++ for (const h of ['127.0.0.1', '::1', 'localhost']) assert.deepStrictEqual(drillMod.problems({ ...safe, HOST: h }), [], h);++ assert.ok(drillMod.problems({ ...safe, PORT: '4100' }).some((m) => /production's port/.test(m)));++ assert.ok(drillMod.problems({ ...safe, PORT: '' }).some((m) => /PORT is not set/.test(m)));++ });++++ // ── A real `node server/index.js` refuses before opening anything ──++ console.log('drill-mode: refusal at boot');++ const refusalPort = await freePort();++ const boot = (env) => spawnSync(process.execPath, [path.join(SERVER, 'index.js')], {++ cwd: REPO, encoding: 'utf8', timeout: 30000,++ env: { ...process.env, MEDIA_DRILL: '1', HOST: '127.0.0.1', PORT: String(refusalPort), ...env },++ });++ const stamp = (p) => { try { const s = fs.statSync(p); return `${s.size}:${s.mtimeMs}`; } catch { return 'absent'; } };++ await check('DB_PATH = production\'s media.db: exit 1, the file untouched', () => {++ const prod = path.join(REPO, 'data', 'media.db');++ const before = [prod, `${prod}-wal`, `${prod}-shm`].map(stamp);++ const r = boot({ DB_PATH: prod });++ assert.strictEqual(r.status, 1, r.stdout + r.stderr);++ assert.match(r.stderr, /refusing to start a restore-drill instance: DB_PATH .* is production's database/);++ assert.deepStrictEqual([prod, `${prod}-wal`, `${prod}-shm`].map(stamp), before);++ });++ await check('HOST 0.0.0.0: exit 1 before the database is created', () => {++ const other = path.join(tmp, 'never', 'media.db');++ const r = boot({ DB_PATH: other, HOST: '0.0.0.0' });++ assert.strictEqual(r.status, 1, r.stdout + r.stderr);++ assert.match(r.stderr, /HOST \(0\.0\.0\.0\) is not loopback/);++ assert.ok(!fs.existsSync(path.dirname(other)));++ });++++ // ── Production's database and files (a normal-mode process writes them) ──++ // The `live` app's webhook goes to a sink here: a drill must never call it.++ const sinkHits = [];++ const sink = http.createServer((req, res) => { sinkHits.push(`${req.method} ${req.url}`); res.end('ok'); });++ await new Promise((r) => sink.listen(0, '127.0.0.1', r));++ const SINK = `http://127.0.0.1:${sink.address().port}/media-webhook`;++ for (const d of Object.values(P)) fs.mkdirSync(d, { recursive: true });++ const bytes = { vod: path.join(P.vods, 'vod-1.mp4'), rec: path.join(P.vods, 'vod-2.mp4'), clip: path.join(P.clips, 'clip-1.mp4'), asset: path.join(P.assets, 'wave.png'), shot: path.join(P.pastes, 'shot-1.png'), file: path.join(P.files, 'live', 'f1.txt') };++ fs.mkdirSync(path.dirname(bytes.file), { recursive: true });++ for (const f of Object.values(bytes)) fs.writeFileSync(f, `bytes of ${path.basename(f)}`);++ fs.writeFileSync(path.join(P.thumbs, 'vod-1-1.jpg'), 'jpeg');++ const prodDb = path.join(PROD, 'media.db');++ const seed = spawnSync(process.execPath, ['-e', `++ const db = require('./server/db/database');++ db.getDb();++ require('./server/views/service').ensureSchema();++ const b = ${JSON.stringify(bytes)};++ db.run("INSERT INTO apps (app_id, name, api_key_hash, webhook_url, webhook_secret) VALUES ('live', 'OpenVibe.Live', ?, ?, 'whsec')", [require('crypto').createHash('sha256').update('k'.repeat(40)).digest('hex'), ${JSON.stringify(SINK)}]);++ db.run("INSERT INTO vods (app_id, title, file_path, thumbnail_url, duration_seconds, is_public, visibility) VALUES ('live', 'Restored VOD', ?, '/t/vod-1-1.jpg', 60, 1, 'public')", [b.vod]);++ db.run("INSERT INTO vods (app_id, title, file_path, is_public, visibility, is_recording) VALUES ('live', 'Still recording', ?, 1, 'public', 1)", [b.rec]);++ db.run("INSERT INTO clips (app_id, vod_id, title, file_path, duration_seconds, status) VALUES ('live', 1, 'Restored clip', ?, 10, 'ready')", [b.clip]);++ db.run("INSERT INTO assets (app_id, kind, name, file_path, mime) VALUES ('live', 'emote', 'drillWave', ?, 'image/png')", [b.asset]);++ db.run("INSERT INTO pastes (app_id, slug, type, title, screenshot_path, visibility) VALUES ('live', 'shot1', 'screenshot', 'Shot', ?, 'public')", [b.shot]);++ db.run("INSERT INTO files (key, app_id, original_name, size, mime) VALUES ('f1.txt', 'live', 'f1.txt', 13, 'text/plain')");++ require('./server/objects/backfill').backfill({ onlyMissing: true });++ db.close();++ `], { cwd: REPO, encoding: 'utf8', env: { ...process.env, DB_PATH: prodDb, VOD_PATH: P.vods, CLIPS_PATH: P.clips, PASTES_PATH: P.pastes, FILES_PATH: P.files, OBJECTS_PATH: P.objects, ASSETS_PATH: P.assets, THUMBNAILS_PATH: P.thumbs } });++ assert.strictEqual(seed.status, 0, `seeding failed:\n${seed.stdout}\n${seed.stderr}`);++ // The restore: a copy of the database file in the drill's own directory.++ fs.mkdirSync(path.dirname(DB_PATH), { recursive: true });++ fs.copyFileSync(prodDb, DB_PATH);++ const Database = require('better-sqlite3');++ const COUNTED = ['media_objects', 'vods', 'clips', 'apps', 'media_locations', 'media_jobs', 'event_outbox'];++ const counts = () => { const d = new Database(DB_PATH, { readonly: true }); try { return Object.fromEntries(COUNTED.map((t) => { try { return [t, d.prepare(`SELECT count(*) AS n FROM ${t}`).get().n]; } catch { return [t, null]; } })); } finally { d.close(); } };++ const countsBefore = counts();++ const objectId = (() => { const d = new Database(DB_PATH, { readonly: true }); try { return d.prepare("SELECT id FROM media_objects WHERE kind = 'vod' ORDER BY id LIMIT 1").get().id; } finally { d.close(); } })();++ const prodBefore = snapshot(PROD);++++ // ── Boot the real server in drill mode, in this process, with spies ──++ const port = await freePort();++ Object.assign(process.env, {++ MEDIA_DRILL: '1', DB_PATH, HOST: '127.0.0.1', PORT: String(port), NODE_ENV: 'production',++ // What the production env file sets; a normal boot would act on all of it.++ EVENTS_URL: 'http://127.0.0.1:9', OV_OAUTH_CLIENT_SECRET: 's'.repeat(40), INTERNAL_API_KEY: 'k'.repeat(40),++ MEDIA_APP_KEYS: `live:${'n'.repeat(40)},games:${'g'.repeat(40)}`,++ MEDIA_B2_ENDPOINT: 'https://s3.example.invalid', MEDIA_B2_BUCKET: 'b', MEDIA_B2_KEY_ID: 'id', MEDIA_B2_APP_KEY: 'key',++ MEDIA_R2_ENDPOINT: 'https://r2.example.invalid', MEDIA_R2_BUCKET: 'r', MEDIA_R2_ACCESS_KEY_ID: 'id', MEDIA_R2_SECRET_ACCESS_KEY: 'key',++ // The Host inventory's storage: nothing but the thumbnail listing points at production.++ VOD_PATH: path.join(STORAGE, 'vods'), CLIPS_PATH: path.join(STORAGE, 'clips'), PASTES_PATH: path.join(STORAGE, 'pastes'),++ FILES_PATH: path.join(STORAGE, 'files'), OBJECTS_PATH: path.join(STORAGE, 'objects'), ASSETS_PATH: path.join(STORAGE, 'assets'),++ THUMBNAILS_PATH: P.thumbs,++ });++ const inServer = (stack) => {++ const frame = stack.split('\n').slice(2).find((l) => l.includes(REPO) && !l.includes(`${path.sep}node_modules${path.sep}`) && !l.includes(__filename));++ return frame && frame.includes(`${SERVER}${path.sep}`) ? frame.trim() : null;++ };++ const spied = { intervals: [], longTimeouts: [], listens: [], programs: [], dbPathTouches: [] };++ const realSetInterval = global.setInterval;++ global.setInterval = function (fn, ms, ...a) { spied.intervals.push({ ms, at: (new Error().stack.split('\n')[2] || '').trim() }); return realSetInterval(fn, ms, ...a); };++ const realSetTimeout = global.setTimeout;++ global.setTimeout = function (fn, ms, ...a) { if (ms >= 1000) { const at = inServer(new Error().stack); if (at) spied.longTimeouts.push({ ms, at }); } return realSetTimeout(fn, ms, ...a); };++ const realListen = net.Server.prototype.listen;++ net.Server.prototype.listen = function (...a) { spied.listens.push(a[0]); return realListen.apply(this, a); };++ const cp = require('child_process');++ for (const fn of ['spawn', 'spawnSync', 'exec', 'execSync', 'execFile', 'execFileSync', 'fork']) {++ const orig = cp[fn];++ cp[fn] = function (...a) { spied.programs.push(fn === 'exec' || fn === 'execSync' ? String(a[0]).split(/\s+/)[0] : String(a[0])); return orig.apply(this, a); };++ }++ // Any look at a file path the database holds (production's media) is recorded.++ const dbPaths = new Set(Object.values(bytes));++ for (const fn of ['createReadStream', 'statSync', 'existsSync', 'openSync', 'readFileSync', 'stat', 'open', 'readFile']) {++ const orig = fs[fn];++ fs[fn] = function (p, ...a) { if (dbPaths.has(String(p))) spied.dbPathTouches.push(`${fn} ${p}`); return orig.call(this, p, ...a); };++ }++++ delete require.cache[require.resolve('../server/drill')]; // loaded above with MEDIA_DRILL unset++ require('../server/index.js');++ const drill = require('../server/drill');++ const base = `http://127.0.0.1:${port}`;++ const request = (method, p, { headers = {}, body = null } = {}) => new Promise((resolve, reject) => {++ const req = http.request(base + p, { method, headers: { ...(body ? { 'Content-Type': 'application/json' } : {}), ...headers } }, (res) => {++ let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, text: b, json: (() => { try { return JSON.parse(b); } catch { return null; } })() }));++ });++ req.on('error', reject);++ if (body) req.write(typeof body === 'string' ? body : JSON.stringify(body));++ req.end();++ });++ let ready = null;++ for (let i = 0; i < 150 && !(ready && ready.status === 200); i++) {++ try { ready = await request('GET', '/api/ready'); } catch { /* not listening yet */ }++ if (!ready || ready.status !== 200) await new Promise((r) => realSetTimeout(r, 100));++ }++ await new Promise((r) => realSetTimeout(r, 1500)); // let anything deferred show itself++++ console.log('drill-mode: a booted drill instance');++ await check('/api/ready is 200, mode: drill, and checks no storage and no remote tier', () => {++ assert.strictEqual(ready && ready.status, 200, ready && ready.text);++ assert.strictEqual(ready.json.ready, true);++ assert.strictEqual(ready.json.mode, 'drill');++ assert.deepStrictEqual(Object.keys(ready.json.checks).filter((k) => /^(storage|remote)_/.test(k)), []);++ assert.strictEqual(ready.json.checks.events_outbox.detail.enabled, false);++ });++ await check('one listener: its HTTP port on 127.0.0.1 (no RTP, no other server)', () => {++ assert.deepStrictEqual(spied.listens.map(String), [String(port)]);++ const servers = process._getActiveHandles().filter((h) => h instanceof net.Server && h.listening && h !== sink);++ assert.deepStrictEqual(servers.map((s) => s.address()), [{ address: '127.0.0.1', family: 'IPv4', port }]);++ });++ await check('no timers: no setInterval at all, no timer of a second or more from Media\'s code', () => {++ assert.deepStrictEqual(spied.intervals, [], JSON.stringify(spied.intervals, null, 1));++ assert.deepStrictEqual(spied.longTimeouts, [], JSON.stringify(spied.longTimeouts, null, 1));++ });++ await check('no program but git ran, nothing tried to leave, no webhook reached the app', () => {++ assert.ok(spied.programs.every((p) => path.basename(p) === 'git'), spied.programs.join(', '));++ assert.deepStrictEqual(drill.blocked, []);++ assert.deepStrictEqual(sinkHits, []);++ });++ await check('no app seeding: the copy\'s apps are production\'s, untouched by MEDIA_APP_KEYS', () => {++ assert.deepStrictEqual(counts(), countsBefore);++ });++++ console.log('drill-mode: reads');++ const head = execFileSync('git', ['rev-parse', '--short=12', 'HEAD'], { cwd: REPO, encoding: 'utf8' }).trim();++ await check('the endpoints the Host inventory compares render from the copy and ask nobody', async () => {++ const rel = await request('GET', '/release.json');++ assert.strictEqual(rel.status, 200);++ assert.strictEqual(rel.json.release, head);++ const videos = await request('GET', '/browse?tab=videos');++ assert.strictEqual(videos.status, 200);++ assert.ok(videos.text.includes('Restored VOD') && !videos.text.includes('Still recording'));++ assert.match(videos.text, /Thumbnails<span class="n">1</, 'the thumbnail listing is production\'s directory');++ const clips = await request('GET', '/browse?tab=clips');++ assert.strictEqual(clips.status, 200);++ assert.ok(clips.text.includes('Restored clip'));++ assert.strictEqual((await request('GET', '/')).status, 200);++ assert.strictEqual((await request('GET', '/healthz')).status, 200);++ const page = await request('GET', '/v/1', { headers: { Accept: 'text/html' } });++ assert.strictEqual(page.status, 200);++ assert.ok(page.text.includes('Restored VOD'), 'the watch page renders from the row');++ assert.deepStrictEqual(drill.blocked, []);++ assert.deepStrictEqual(spied.dbPathTouches, []);++ });++ await check('every route that would stream a stored file answers 503 without opening the database\'s path', async () => {++ const byteRoutes = ['/v/1', '/v/1?raw=1', '/v/vod-1.mp4', '/c/1', '/c/1?raw=1', `/o/${objectId}`, '/t/vod-1-1.jpg', '/api/thumbnails/vod-1-1.jpg',++ '/a/1', '/f/f1.txt', '/f/screenshots/shot-1.png', '/p/shot1/screenshot', '/live/1/frame.jpg'];++ for (const p of byteRoutes) {++ const r = await request('GET', p, { headers: { Range: 'bytes=0-' } });++ assert.strictEqual(r.status, 503, `${p}: ${r.status} ${r.text.slice(0, 120)}`);++ assert.strictEqual(r.json && r.json.code, 'media.drill_no_bytes', p);++ }++ assert.deepStrictEqual(spied.dbPathTouches, [], 'no stored file was looked at');++ assert.deepStrictEqual(spied.programs.filter((p) => path.basename(p) !== 'git'), [], 'no ffmpeg/ffprobe');++ });++++ console.log('drill-mode: writes, sign-in, side effects');++ await check('writes answer 403 on every path and change nothing; sign-in answers 503', async () => {++ const auth = { Authorization: `Bearer ${'k'.repeat(40)}` };++ const writes = [++ ['POST', '/api/v1/live/vods', { title: 'x' }], ['PUT', '/api/v1/live/vods/1', { title: 'y' }], ['DELETE', '/api/v1/live/vods/1', null],++ ['POST', '/api/v1/live/clips', { vod_id: 1 }], ['PUT', `/api/v2/live/objects/${objectId}/content`, 'raw bytes'], ['POST', '/api/v2/live/jobs', { type: 'thumbnail' }],++ ['POST', '/api/v1/live/admin/storage/tiers/sweep', {}], ['POST', '/internal/avatar-ingest', {}], ['PATCH', '/api/v1/live/vods/1', {}],++ ];++ for (const [m, p, body] of writes) {++ const r = await request(m, p, { headers: auth, body });++ assert.strictEqual(r.status, 403, `${m} ${p}: ${r.status}`);++ assert.strictEqual(r.json && r.json.code, 'media.drill_read_only', `${m} ${p}`);++ }++ const login = await request('GET', '/auth/login');++ assert.strictEqual(login.status, 503);++ assert.deepStrictEqual(counts(), countsBefore);++ });++ await check('webhooks and the Events relay stay off even when called', async () => {++ assert.strictEqual(await require('../server/webhooks').sendWebhook('live', 'vod.ready', { id: 1 }), false);++ assert.strictEqual(require('../server/events').init(), null);++ assert.deepStrictEqual(sinkHits, []);++ });++ await check('the guards refuse a connection, ffmpeg, a listener and a UDP socket', async () => {++ await assert.rejects(fetch('https://openvibe.live/internal/media-webhook'), /fetch failed/);++ const sock = net.connect({ host: '127.0.0.1', port: sink.address().port });++ const err = await new Promise((resolve) => sock.on('error', resolve));++ assert.strictEqual(err.code, 'ECONNREFUSED');++ assert.deepStrictEqual(sinkHits, [], 'the connection never reached the app');++ assert.throws(() => cp.spawn('ffmpeg', ['-version']), (e) => e.code === 'EDRILL');++ assert.throws(() => net.createServer().listen(12000), (e) => e.code === 'EDRILL');++ assert.throws(() => require('dgram').createSocket('udp4'), (e) => e.code === 'EDRILL');++ });++ await check('production\'s files are untouched and the drill created no storage directory', () => {++ assert.deepStrictEqual(snapshot(PROD), prodBefore);++ assert.ok(!fs.existsSync(STORAGE), `${STORAGE} was created`);++ assert.deepStrictEqual(fs.readdirSync(DRILL), ['db']);++ });++++ sink.close();++ try { fs.rmSync(tmp, { recursive: true, force: true }); } catch { /* */ }++ if (failures) { console.error(`\n${failures} drill-mode check(s) failed`); process.exit(1); }++ console.log('\ndrill-mode: all checks passed');++ process.exit(0);++})().catch((err) => { console.error(err); process.exit(1); });
@@ -11,6 +11,11 @@ MEDIA_PUBLIC_URL=https://openvibe.media
@@ -11,6 +11,11 @@ MEDIA_PUBLIC_URL=https://openvibe.media
# Database (better-sqlite3, WAL)
# Database (better-sqlite3, WAL)
DB_PATH=./data/media.db
DB_PATH=./data/media.db
+# Restore-drill mode, set only by `ovhost drill media` (OpenVibe.Host docs/restore-drills.md): reads
+# only from a restored copy, no stored bytes, no jobs, webhooks, events or ffmpeg; refuses to start
+# unless DB_PATH is outside the checkout, HOST is loopback and PORT is not 4100 (server/drill.js).
+# MEDIA_DRILL=1
+
# Storage paths (all created on boot)
# Storage paths (all created on boot)
VOD_PATH=./data/vods
VOD_PATH=./data/vods
CLIPS_PATH=./data/clips
CLIPS_PATH=./data/clips
@@ -23,6 +23,7 @@ server/
@@ -23,6 +23,7 @@ server/
config.js env config (PORT, DB_PATH, *_PATH, MEDIA_B2_*/R2_*, RTP pool, OV_NETWORK_URL)
config.js env config (PORT, DB_PATH, *_PATH, MEDIA_B2_*/R2_*, RTP pool, OV_NETWORK_URL)
auth.js tenancy + auth middleware, JWKS fetch, app seeding
auth.js tenancy + auth middleware, JWKS fetch, app seeding
webhooks.js HMAC-signed outbound webhooks
webhooks.js HMAC-signed outbound webhooks
+ drill.js MEDIA_DRILL: restore-drill mode (reads only, no bytes, no jobs, nothing leaves)
db/schema.sql apps, vods, clips, pastes(+likes/comments), files, content_views, media_settings,
db/schema.sql apps, vods, clips, pastes(+likes/comments), files, content_views, media_settings,
media_objects/locations/relationships/variants/jobs/holds/invariant_violations
media_objects/locations/relationships/variants/jobs/holds/invariant_violations
db/database.js better-sqlite3 helpers, all app_id-scoped; legacy row importer
db/database.js better-sqlite3 helpers, all app_id-scoped; legacy row importer
@@ -265,6 +266,29 @@ rolled-back change; the webhook follows the commit and carries `event_id` = that
@@ -265,6 +266,29 @@ rolled-back change; the webhook follows the commit and carries `event_id` = that
(absent when the outbox is off), so an app that reads both paths (Live during its webhook →
(absent when the outbox is off), so an app that reads both paths (Live during its webhook →
Events transition) handles each outcome once. Webhooks stay until every consumer reads Events.
Events transition) handles each outcome once. Webhooks stay until every consumer reads Events.
+## Restore drills (`MEDIA_DRILL=1`)
+
+`ovhost drill media` (OpenVibe.Host `docs/restore-drills.md`) restores `media.db` from the latest
+backup and starts a second Media from this checkout on 127.0.0.1:14100 with `MEDIA_DRILL=1` and
+`DB_PATH` on the copy. In that mode (`server/drill.js`) Media:
+
+- refuses to start unless `DB_PATH` is set and outside the checkout and `/opt/openvibe.media`, `HOST`
+ is loopback and `PORT` is not 4100 (before the database is opened);
+- starts only its HTTP server: no app seeding, JWKS refresh, tiering sweep, health job, junk sweep,
+ clip re-cuts, copy verification, jobs worker, disk guardian, thumbnail cleanup, object purge,
+ backfill, orphan-recording finalize or Events relay; webhooks are never sent;
+- writes no file and creates no directory (the storage checks leave `/api/ready`, as do the remote
+ tiers; it reports `"mode": "drill"`);
+- never opens a file path from the database: every byte route (`/v` and `/c` bytes, `/t`, `/a`, `/f`,
+ `/o`, `/p/:slug/screenshot`, `/live/:sel/frame.jpg`, `/api/thumbnails/:name`) answers 503;
+ watch pages and the `/browse` index still render from the copy;
+- connects to nothing and runs no program but `git` (no ffmpeg or ffprobe), opens no UDP socket and
+ listens on nothing but its port;
+- answers 403 to every method but GET, HEAD and OPTIONS, and 503 to `/auth/*`.
+
+`test/drill-mode.test.js` boots the real server that way against a copy whose app has a webhook URL
+and checks each point.
+
## Health, readiness and metrics
## Health, readiness and metrics
| path | behavior |
| path | behavior |
@@ -21,7 +21,7 @@ const db = require('../db/database');
@@ -21,7 +21,7 @@ const db = require('../db/database');
const { tenantAuth } = require('../auth');
const { tenantAuth } = require('../auth');
const ASSET_DIR = config.assets.path;
const ASSET_DIR = config.assets.path;
-if (!fs.existsSync(ASSET_DIR)) fs.mkdirSync(ASSET_DIR, { recursive: true });
+if (!require('../drill').enabled && !fs.existsSync(ASSET_DIR)) fs.mkdirSync(ASSET_DIR, { recursive: true }); // a restore drill writes nothing
const upload = multer({ storage: multer.memoryStorage(), limits: { fileSize: 10 * 1024 * 1024 } });
const upload = multer({ storage: multer.memoryStorage(), limits: { fileSize: 10 * 1024 * 1024 } });
const router = express.Router({ mergeParams: true });
const router = express.Router({ mergeParams: true });
new file mode 100644
new file mode 100644
@@ -0,0 +1,243 @@
@@ -0,0 +1,243 @@
+'use strict';
+/**
+ * Restore-drill mode: MEDIA_DRILL=1 (`ovhost drill media`, see OpenVibe.Host docs/restore-drills.md).
+ *
+ * A drill starts a second Media from the production checkout, with the production env file, against
+ * a restored copy of media.db on a spare loopback port, and compares its public reads with
+ * production's. That instance must serve reads from the copy and do nothing else. With MEDIA_DRILL:
+ *
+ * - assertSafe(): Media refuses to start unless DB_PATH is set and lies outside the checkout and
+ * outside /opt/openvibe.media (so it cannot be production's media.db), PORT is set and is not
+ * production's 4100, and HOST is loopback. It runs before the database is opened.
+ * - installGuards(): no outbound connection (net.Socket#connect, fetch), so no webhook, event, B2/R2
+ * or Network call leaves the process; no program other than git (child_process: no ffmpeg or
+ * ffprobe ever; git only reads the checkout for /release.json); no UDP socket (RTP ingest); nothing
+ * listens but the drill's own HTTP port.
+ * - readOnly: 403 for every request that is not GET, HEAD or OPTIONS, on every path.
+ * - refuseBytes(): every route that would stream a stored file (or redirect to its B2/R2 copy)
+ * answers 503. The database holds absolute paths to production's files; a drill never opens them.
+ * Watch pages and the media index render from the database only and still answer.
+ * - server/index.js opens the database and serves HTTP, and starts nothing else: no app seeding, no
+ * JWKS refresh, no health job, junk sweep, tiering sweep, clip re-cuts, copy verification, jobs
+ * worker (thumbnails, split/remux, invariant scans), disk guardian, thumbnail cleanup, object purge,
+ * backfill, orphan-recording finalize or Events relay; webhooks are never sent. Timers that modules
+ * start when they are loaded check `enabled` as well. Readiness does not probe storage (a drill
+ * writes none) or the remote tiers.
+ *
+ * The guards are there so that anything this list missed fails closed instead of reaching production.
+ */
+const path = require('path');
+const fs = require('fs');
+
+const TRUE = new Set(['1', 'true', 'on', 'yes']);
+
+/** MEDIA_DRILL as a switch (1/true/on/yes). */
+function parse(value) {
+ return TRUE.has(String(value == null ? '' : value).trim().toLowerCase());
+}
+
+const enabled = parse(process.env.MEDIA_DRILL);
+
+const REPO_ROOT = path.resolve(__dirname, '..');
+/** Where production runs (OpenVibe.Host inventory: repo /opt/openvibe.media, database data/media.db). */
+const PRODUCTION_ROOT = '/opt/openvibe.media';
+const PRODUCTION_PORT = 4100;
+const LOOPBACK_HOSTS = new Set(['127.0.0.1', '::1', 'localhost', '::ffff:127.0.0.1']);
+
+function isLoopbackHost(host) {
+ const h = String(host || '').trim().toLowerCase().replace(/^\[|\]$/g, '');
+ return LOOPBACK_HOSTS.has(h) || /^127(\.\d{1,3}){3}$/.test(h);
+}
+
+/** The real location of p, following symlinks on the part of it that exists. */
+function realish(p) {
+ let head = path.resolve(p);
+ const tail = [];
+ for (;;) {
+ try { return path.join(fs.realpathSync(head), ...tail.reverse()); } catch { /* not there (yet) */ }
+ const parent = path.dirname(head);
+ if (parent === head) return path.resolve(p);
+ tail.push(path.basename(head));
+ head = parent;
+ }
+}
+
+function inside(p, root) {
+ const rel = path.relative(root, p);
+ return rel === '' || (!!rel && !rel.startsWith('..') && !path.isAbsolute(rel));
+}
+
+/**
+ * Why this environment is not safe for a drill ([] = safe). Pure, for tests.
+ * `repoRoot` is the checkout the process runs from; `cwd` resolves relative paths.
+ */
+function problems(env = process.env, { repoRoot = REPO_ROOT, cwd = process.cwd() } = {}) {
+ const out = [];
+ const roots = [...new Set([repoRoot, realish(repoRoot), PRODUCTION_ROOT])];
+ const productionDb = new Set([path.join(PRODUCTION_ROOT, 'data', 'media.db'), path.join(repoRoot, 'data', 'media.db'), path.join(cwd, 'data', 'media.db')]);
+ if (!env.DB_PATH) {
+ out.push('DB_PATH is not set; it must name the restored copy of media.db');
+ } else {
+ const abs = path.resolve(cwd, env.DB_PATH);
+ const real = realish(abs);
+ const root = roots.find((r) => inside(abs, r) || inside(real, r));
+ if (productionDb.has(abs) || productionDb.has(real)) out.push(`DB_PATH (${abs}) is production's database; point it at the restored copy`);
+ else if (root) out.push(`DB_PATH (${abs}${real !== abs ? ` → ${real}` : ''}) is inside ${root}; a drill writes nothing in the checkout or in production's data`);
+ }
+ if (!isLoopbackHost(env.HOST)) out.push(`HOST (${env.HOST || 'unset, i.e. 0.0.0.0'}) is not loopback; a drill binds 127.0.0.1 only`);
+ const port = Number(env.PORT);
+ if (!env.PORT || !Number.isInteger(port) || port < 1 || port > 65535) out.push('PORT is not set; it must be the drill\'s own port');
+ else if (port === PRODUCTION_PORT) out.push(`PORT ${port} is production's port`);
+ return out;
+}
+
+class DrillRefused extends Error {
+ constructor(list) {
+ super(`MEDIA_DRILL: refusing to start a restore-drill instance: ${list.join('; ')}`);
+ this.code = 'MEDIA_DRILL_UNSAFE';
+ this.problems = list;
+ }
+}
+
+/** Throws DrillRefused unless the environment is safe for a drill. */
+function assertSafe(env = process.env, opts) {
+ const list = problems(env, opts);
+ if (list.length) throw new DrillRefused(list);
+}
+
+// ── Guards ───────────────────────────────────────────────────
+
+const blocked = []; // what the guards refused (newest last, bounded): { at, kind, target }
+function note(kind, target) {
+ blocked.push({ at: new Date().toISOString(), kind, target: String(target).slice(0, 200) });
+ if (blocked.length > 50) blocked.shift();
+ console.warn(`[Drill] refused ${kind}: ${target}`);
+}
+
+function drillError(message, code) {
+ const err = new Error(`${message} (MEDIA_DRILL: a restore-drill instance does not do this)`);
+ err.code = code;
+ return err;
+}
+
+/** Where a net.Socket#connect call is going: { host, port } or { path }. */
+function connectTarget(args) {
+ let a = args[0];
+ if (Array.isArray(a)) a = a[0]; // net.connect() hands Socket#connect its normalized [options, cb]
+ if (a && typeof a === 'object') return a.path != null ? { path: String(a.path) } : { host: a.host, port: Number(a.port) };
+ if (typeof a === 'string' && !/^\d+$/.test(a)) return { path: a };
+ return { host: typeof args[1] === 'string' ? args[1] : undefined, port: Number(a) };
+}
+
+const ALLOWED_PROGRAMS = new Set(['git']);
+
+function programOf(fn, args) {
+ const first = args[0];
+ if (fn === 'exec' || fn === 'execSync') return String(first || '').trim().split(/\s+/)[0] || '';
+ if (fn === 'fork') return 'node';
+ return String(first || '');
+}
+
+let installed = false;
+
+/**
+ * Replace the process's ways out with ones that refuse (drill only). `port` is the drill's HTTP
+ * port: the one address the process may connect to (itself, on loopback) and the one it may listen on.
+ */
+function installGuards({ port = Number(process.env.PORT) } = {}) {
+ if (installed) return;
+ installed = true;
+ const net = require('net');
+ const util = require('util');
+
+ const isSelf = (t) => t && t.path == null && t.port === port && (t.host == null || isLoopbackHost(t.host));
+
+ const connect = net.Socket.prototype.connect;
+ net.Socket.prototype.connect = function drillConnect(...args) {
+ const t = connectTarget(args);
+ if (isSelf(t)) return connect.apply(this, args);
+ const where = t.path != null ? t.path : `${t.host || 'localhost'}:${t.port}`;
+ note('connection', where);
+ const err = drillError(`connect ECONNREFUSED ${where}`, 'ECONNREFUSED');
+ err.syscall = 'connect';
+ process.nextTick(() => this.destroy(err));
+ return this;
+ };
+
+ if (typeof globalThis.fetch === 'function') {
+ const fetch = globalThis.fetch;
+ globalThis.fetch = function drillFetch(input, init) {
+ let url = null;
+ try { url = new URL(typeof input === 'string' || input instanceof URL ? String(input) : input.url); } catch { /* */ }
+ if (url && isSelf({ host: url.hostname, port: Number(url.port || (url.protocol === 'https:' ? 443 : 80)) })) return fetch(input, init);
+ note('fetch', url ? url.origin + url.pathname : String(input));
+ return Promise.reject(new TypeError('fetch failed', { cause: drillError(`connect ECONNREFUSED ${url ? url.host : ''}`, 'ECONNREFUSED') }));
+ };
+ }
+
+ const cp = require('child_process');
+ for (const fn of ['spawn', 'spawnSync', 'exec', 'execSync', 'execFile', 'execFileSync', 'fork']) {
+ const orig = cp[fn];
+ if (typeof orig !== 'function') continue;
+ const check = (args) => {
+ const program = programOf(fn, args);
+ if (ALLOWED_PROGRAMS.has(path.basename(program))) return;
+ note('program', program);
+ throw drillError(`${fn}(${program}) refused`, 'EDRILL');
+ };
+ const guarded = function (...args) { check(args); return orig.apply(this, args); };
+ const custom = orig[util.promisify.custom];
+ if (custom) guarded[util.promisify.custom] = (...args) => { check(args); return custom(...args); };
+ cp[fn] = guarded;
+ }
+
+ const dgram = require('dgram');
+ dgram.createSocket = function drillUdp() {
+ note('udp socket', 'dgram.createSocket');
+ throw drillError('UDP socket refused', 'EDRILL');
+ };
+
+ const listen = net.Server.prototype.listen;
+ net.Server.prototype.listen = function drillListen(...args) {
+ const a = args[0];
+ const p = a && typeof a === 'object' ? Number(a.port) : Number(a);
+ if (p === port && !(a && typeof a === 'object' && (a.fd != null || a.path != null))) return listen.apply(this, args);
+ note('listener', a && typeof a === 'object' ? JSON.stringify(a) : String(a));
+ throw drillError(`listen ${a && typeof a === 'object' ? JSON.stringify(a) : a} refused`, 'EDRILL');
+ };
+}
+
+// ── HTTP ─────────────────────────────────────────────────────
+
+/** Express middleware: reads only. */
+function readOnly(req, res, next) {
+ if (req.method === 'GET' || req.method === 'HEAD' || req.method === 'OPTIONS') return next();
+ res.status(403).json({ error: 'This is a restore-drill instance (MEDIA_DRILL): it serves reads only', code: 'media.drill_read_only' });
+}
+
+/**
+ * Answer a request for stored bytes: a drill never opens a file path from the database (they are
+ * production's files) and never hands out a B2/R2 URL for one. Returns true when it answered.
+ */
+function refuseBytes(res) {
+ if (!enabled) return false;
+ if (!res.headersSent) {
+ res.status(503).set('Retry-After', '3600').json({ error: 'This is a restore-drill instance (MEDIA_DRILL): it serves no stored media', code: 'media.drill_no_bytes' });
+ }
+ return true;
+}
+
+module.exports = {
+ enabled,
+ parse,
+ problems,
+ assertSafe,
+ DrillRefused,
+ installGuards,
+ readOnly,
+ refuseBytes,
+ blocked,
+ isLoopbackHost,
+ PRODUCTION_ROOT,
+ REPO_ROOT,
+};
@@ -56,6 +56,8 @@ function init({
@@ -56,6 +56,8 @@ function init({
fetchImpl, intervalMs = 2000,
fetchImpl, intervalMs = 2000,
} = {}) {
} = {}) {
if (outbox) return outbox;
if (outbox) return outbox;
+ // A restore drill (MEDIA_DRILL) relays nothing: its outbox rows describe a restored copy.
+ if (require('./drill').enabled) return null;
if (process.env.EVENTS_PUBLISH === 'off' || !eventsUrl || !clientSecret) return null;
if (process.env.EVENTS_PUBLISH === 'off' || !eventsUrl || !clientSecret) return null;
const tokens = createServiceTokenClient({ tokenUrl: `${networkUrl}/oauth/token`, clientId, clientSecret, fetch: fetchImpl });
const tokens = createServiceTokenClient({ tokenUrl: `${networkUrl}/oauth/token`, clientId, clientSecret, fetch: fetchImpl });
const client = createClient({ baseUrls: { events: eventsUrl }, tokenProvider: tokens, fetch: fetchImpl, retries: 0 });
const client = createClient({ baseUrls: { events: eventsUrl }, tokenProvider: tokens, fetch: fetchImpl, retries: 0 });
@@ -32,8 +32,9 @@ const { tenantAuth, tenantCors } = require('../auth');
@@ -32,8 +32,9 @@ const { tenantAuth, tenantCors } = require('../auth');
const router = express.Router({ mergeParams: true });
const router = express.Router({ mergeParams: true });
router.use(tenantCors);
router.use(tenantCors);
+// (multer creates `dest` when it is built; a restore drill takes no upload and creates no directory.)
const upload = multer({
const upload = multer({
- dest: path.join(config.files.path, '.tmp'),
+ ...(require('../drill').enabled ? { storage: multer.memoryStorage() } : { dest: path.join(config.files.path, '.tmp') }),
limits: { fileSize: config.files.maxSizeMb * 1024 * 1024 },
limits: { fileSize: config.files.maxSizeMb * 1024 * 1024 },
});
});
@@ -8,6 +8,23 @@
@@ -8,6 +8,23 @@
*/
*/
'use strict';
'use strict';
+// Restore-drill mode (MEDIA_DRILL=1, `ovhost drill media`): refuse an unsafe environment before the
+// database is opened, then cut every way out of the process but its own HTTP port. The instance
+// serves reads from a restored copy of media.db, never opens a stored file and starts nothing else
+// (server/drill.js).
+require('dotenv').config();
+const drill = require('./drill');
+if (drill.enabled) {
+ try {
+ drill.assertSafe();
+ } catch (err) {
+ console.error(`[Drill] ${err.message}`);
+ process.exit(1);
+ }
+ drill.installGuards();
+ console.log(`[Drill] MEDIA_DRILL: restore-drill instance on ${process.env.HOST}:${process.env.PORT}, database ${require('path').resolve(process.env.DB_PATH)}. Reads only; no stored bytes, background work or outbound connections.`);
+}
+
const fs = require('fs');
const fs = require('fs');
const express = require('express');
const express = require('express');
const config = require('./config');
const config = require('./config');
@@ -16,14 +33,19 @@ const auth = require('./auth');
@@ -16,14 +33,19 @@ const auth = require('./auth');
// ── Bootstrap ────────────────────────────────────────────────
// ── Bootstrap ────────────────────────────────────────────────
-for (const dir of [config.vod.path, config.vod.clipsPath, config.pastes.path, config.thumbnails.path, config.files.path, config.objects.path]) {
- if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true });
+// A drill creates no storage directory (it writes no file) and seeds nothing: the copy is production's.
+if (!drill.enabled) {
+ for (const dir of [config.vod.path, config.vod.clipsPath, config.pastes.path, config.thumbnails.path, config.files.path, config.objects.path]) {
+ if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true });
+ }
}
}
db.getDb(); // init schema (WAL)
db.getDb(); // init schema (WAL)
-auth.seedApps(); // upsert MEDIA_APPS_SEED / MEDIA_APP_KEYS
-auth.ensureTokenOnlyApps(); // tenants reached only with Network service tokens (community)
-auth.startJwksRefresh();
+if (!drill.enabled) {
+ auth.seedApps(); // upsert MEDIA_APPS_SEED / MEDIA_APP_KEYS
+ auth.ensureTokenOnlyApps(); // tenants reached only with Network service tokens (community)
+ auth.startJwksRefresh(); // a drill asks Network for nothing (app keys still verify against the copy)
+}
const app = express();
const app = express();
app.set('trust proxy', true);
app.set('trust proxy', true);
@@ -31,6 +53,9 @@ app.set('trust proxy', true);
@@ -31,6 +53,9 @@ app.set('trust proxy', true);
const release = require('openvibe-shared/release').createRelease({ service: 'media', root: require('path').join(__dirname, '..') });
const release = require('openvibe-shared/release').createRelease({ service: 'media', root: require('path').join(__dirname, '..') });
const observability = require('./observability');
const observability = require('./observability');
const instrumented = observability.instrument(app, { release: release.release });
const instrumented = observability.instrument(app, { release: release.release });
+// A restore-drill instance answers reads only: 403 for every other method, on every path (before the
+// raw-body upload routes below).
+if (drill.enabled) app.use(drill.readOnly);
// Object API v2 content uploads read the raw request body, so they sit ahead of the body parsers.
// Object API v2 content uploads read the raw request body, so they sit ahead of the body parsers.
app.put('/api/v2/:app/objects/:id/content', ...require('./objects/routes').contentHandlers);
app.put('/api/v2/:app/objects/:id/content', ...require('./objects/routes').contentHandlers);
app.put('/api/v2/:app/objects/:id/multipart/:uploadId/parts/:n', ...require('./objects/routes').partHandlers);
app.put('/api/v2/:app/objects/:id/multipart/:uploadId/parts/:n', ...require('./objects/routes').partHandlers);
@@ -69,6 +94,8 @@ const userAuthConfig = {
@@ -69,6 +94,8 @@ const userAuthConfig = {
cookies: { secure: /^https:/.test(config.publicUrl) },
cookies: { secure: /^https:/.test(config.publicUrl) },
oauth: { clientId: process.env.OV_OAUTH_CLIENT_ID || 'media', clientSecret: process.env.OV_OAUTH_CLIENT_SECRET || '', redirectUri: process.env.OV_OAUTH_REDIRECT_URI || `${config.publicUrl}/auth/callback` },
oauth: { clientId: process.env.OV_OAUTH_CLIENT_ID || 'media', clientSecret: process.env.OV_OAUTH_CLIENT_SECRET || '', redirectUri: process.env.OV_OAUTH_REDIRECT_URI || `${config.publicUrl}/auth/callback` },
};
};
+// A restore drill signs nobody in: that would redeem codes at Network.
+if (drill.enabled) app.use('/auth', (req, res) => res.status(503).json({ error: 'This is a restore-drill instance (MEDIA_DRILL): no sign-in', code: 'media.drill_no_sign_in' }));
app.use('/auth', userAuth.createAuthRoutes(userAuthConfig, userAuth.createAuthClient(userAuthConfig)));
app.use('/auth', userAuth.createAuthRoutes(userAuthConfig, userAuth.createAuthClient(userAuthConfig)));
app.use(express.urlencoded({ extended: true, limit: '2mb' }));
app.use(express.urlencoded({ extended: true, limit: '2mb' }));
@@ -108,6 +135,7 @@ app.get('/healthz', (req, res) => {
@@ -108,6 +135,7 @@ app.get('/healthz', (req, res) => {
recorder: require('./vod/recorder'),
recorder: require('./vod/recorder'),
events: require('./events'),
events: require('./events'),
remote: { configured: (n) => vodStorageForReady.providerConfigured(n), probe: (n) => vodStorageForReady.probeProvider(n) },
remote: { configured: (n) => vodStorageForReady.providerConfigured(n), probe: (n) => vodStorageForReady.probeProvider(n) },
+ drill: drill.enabled,
});
});
}
}
@@ -134,7 +162,7 @@ app.get('/api/v1/:app/stats/series/:metric', auth.tenantAuth(), (req, res) => {
@@ -134,7 +162,7 @@ app.get('/api/v1/:app/stats/series/:metric', auth.tenantAuth(), (req, res) => {
});
});
try { require('./views/service').ensureSchema(); } catch (e) { console.warn('[Views] schema:', e.message); }
try { require('./views/service').ensureSchema(); } catch (e) { console.warn('[Views] schema:', e.message); }
-setInterval(() => { try { const n = require('./views/service').prune(30); if (n) console.log(`[Views] pruned ${n} stale visit row(s)`); } catch { /* */ } }, 12 * 3600 * 1000);
+if (!drill.enabled) setInterval(() => { try { const n = require('./views/service').prune(30); if (n) console.log(`[Views] pruned ${n} stale visit row(s)`); } catch { /* */ } }, 12 * 3600 * 1000);
app.use('/api/v1/:app/views', require('./views/routes'));
app.use('/api/v1/:app/views', require('./views/routes'));
app.use('/api/v1/:app/vods', require('./vod/routes'));
app.use('/api/v1/:app/vods', require('./vod/routes'));
app.use('/api/v1/:app/clips', require('./vod/clips-routes'));
app.use('/api/v1/:app/clips', require('./vod/clips-routes'));
@@ -168,56 +196,61 @@ function every(ms, fn) {
@@ -168,56 +196,61 @@ function every(ms, fn) {
timers.push(t);
timers.push(t);
}
}
-vodStorage.checkProviders()
- .then(() => vodStorage.migrateLegacy().catch(() => {}))
- .catch(err => console.warn('[Boot] Provider check failed:', err.message));
-vodStorage.start(); // tiering sweep
-healthJob.start(); // health scan + quarantine cleanup
-require('./vod/clip-jobs').start(); // failed clip re-cuts (auto-retry, hot-fetch)
-require('./objects/verify-job').start(); // scheduled copy verification (bounded batches; never deletes)
-require('./jobs/worker').start(); // media_jobs worker (light + heavy lanes; proposals wait for their owner)
-// Descriptor watchdog: a leak here once pinned 80 GB of deleted recordings to the disk.
-every(5 * 60 * 1000, () => {
- try {
- const n = fs.readdirSync('/proc/self/fd').length;
- if (n > 1500) console.warn(`[Boot] ${n} open file descriptors — investigate a stream/handle leak (lsof -p ${process.pid} +L1)`);
- } catch { /* not linux */ }
-});
-every(60 * 1000, () => recorder.checkDisk()); // disk guardian
-every(60 * 60 * 1000, () => thumbService.cleanupOldThumbnails()); // stale live thumbs
-// Object model: purge native objects whose soft-delete retention has passed (held ones are kept).
-every(60 * 60 * 1000, () => {
- try { const n = require('./objects/model').purgeExpired(); if (n) console.log(`[Objects] Purged ${n} expired deleted object(s)`); } catch (err) { console.warn('[Objects] purge:', err.message); }
- // Incomplete multipart uploads past MEDIA_MULTIPART_TTL_HOURS: their parts are deleted (the objects stay uploading).
- try { const r = require('./objects/multipart').purgeExpired(); if (r.expired || r.orphan_dirs) console.log(`[Objects] Multipart: ${r.expired} expired session(s), ${r.orphan_dirs} orphan part dir(s) removed`); } catch (err) { console.warn('[Objects] multipart purge:', err.message); }
-});
-// Project rows that have no media_object yet (first boot after the upgrade: all of them).
-setTimeout(() => {
- try {
- const bf = require('./objects/backfill');
- const r = bf.backfill({ onlyMissing: true });
- if (r.totals.created || r.totals.updated || r.errors.length) console.log(`[Objects] Backfill: ${bf.summarize(r)}`);
- } catch (err) { console.warn('[Objects] Backfill failed:', err.message); }
-}, 15 * 1000).unref?.();
-
-// Recover from an unclean shutdown: rows stuck in is_recording with no live
-// ffmpeg are finalized from whatever hit the disk.
-setTimeout(() => {
- try {
- const stuck = db.all('SELECT id FROM vods WHERE is_recording = 1');
- for (const row of stuck) {
- if (recorder.isRecording(row.id)) continue;
- console.log(`[Boot] Finalizing orphaned recording vod ${row.id}`);
- require('./vod/finalize').finalizeVod(row.id).catch(() => {});
+// A restore drill starts none of this: every job below reads or writes stored files, runs ffmpeg,
+// talks to B2/R2 or announces outcomes to Live.
+if (!drill.enabled) {
+ vodStorage.checkProviders()
+ .then(() => vodStorage.migrateLegacy().catch(() => {}))
+ .catch(err => console.warn('[Boot] Provider check failed:', err.message));
+ vodStorage.start(); // tiering sweep
+ healthJob.start(); // health scan + quarantine cleanup
+ require('./vod/clip-jobs').start(); // failed clip re-cuts (auto-retry, hot-fetch)
+ require('./objects/verify-job').start(); // scheduled copy verification (bounded batches; never deletes)
+ require('./jobs/worker').start(); // media_jobs worker (light + heavy lanes; proposals wait for their owner)
+ // Descriptor watchdog: a leak here once pinned 80 GB of deleted recordings to the disk.
+ every(5 * 60 * 1000, () => {
+ try {
+ const n = fs.readdirSync('/proc/self/fd').length;
+ if (n > 1500) console.warn(`[Boot] ${n} open file descriptors — investigate a stream/handle leak (lsof -p ${process.pid} +L1)`);
+ } catch { /* not linux */ }
+ });
+ every(60 * 1000, () => recorder.checkDisk()); // disk guardian
+ every(60 * 60 * 1000, () => thumbService.cleanupOldThumbnails()); // stale live thumbs
+ // Object model: purge native objects whose soft-delete retention has passed (held ones are kept).
+ every(60 * 60 * 1000, () => {
+ try { const n = require('./objects/model').purgeExpired(); if (n) console.log(`[Objects] Purged ${n} expired deleted object(s)`); } catch (err) { console.warn('[Objects] purge:', err.message); }
+ // Incomplete multipart uploads past MEDIA_MULTIPART_TTL_HOURS: their parts are deleted (the objects stay uploading).
+ try { const r = require('./objects/multipart').purgeExpired(); if (r.expired || r.orphan_dirs) console.log(`[Objects] Multipart: ${r.expired} expired session(s), ${r.orphan_dirs} orphan part dir(s) removed`); } catch (err) { console.warn('[Objects] multipart purge:', err.message); }
+ });
+ // Project rows that have no media_object yet (first boot after the upgrade: all of them).
+ setTimeout(() => {
+ try {
+ const bf = require('./objects/backfill');
+ const r = bf.backfill({ onlyMissing: true });
+ if (r.totals.created || r.totals.updated || r.errors.length) console.log(`[Objects] Backfill: ${bf.summarize(r)}`);
+ } catch (err) { console.warn('[Objects] Backfill failed:', err.message); }
+ }, 15 * 1000).unref?.();
+
+ // Recover from an unclean shutdown: rows stuck in is_recording with no live
+ // ffmpeg are finalized from whatever hit the disk.
+ setTimeout(() => {
+ try {
+ const stuck = db.all('SELECT id FROM vods WHERE is_recording = 1');
+ for (const row of stuck) {
+ if (recorder.isRecording(row.id)) continue;
+ console.log(`[Boot] Finalizing orphaned recording vod ${row.id}`);
+ require('./vod/finalize').finalizeVod(row.id).catch(() => {});
+ }
+ } catch (err) {
+ console.warn('[Boot] Orphan finalize sweep failed:', err.message);
}
}
- } catch (err) {
- console.warn('[Boot] Orphan finalize sweep failed:', err.message);
- }
-}, 5000).unref?.();
+ }, 5000).unref?.();
+}
// ── Listen + graceful shutdown ───────────────────────────────
// ── Listen + graceful shutdown ───────────────────────────────
const server = app.listen(config.port, config.host, () => {
const server = app.listen(config.port, config.host, () => {
+ if (drill.enabled) { console.log(`[Drill] Ready: http://${config.host}:${config.port} (reads only)`); return; }
console.log(`[Media] OpenVibe.Media listening on ${config.host}:${config.port} (${config.nodeEnv})`);
console.log(`[Media] OpenVibe.Media listening on ${config.host}:${config.port} (${config.nodeEnv})`);
console.log(`[Media] Data: db=${config.db.path} vods=${config.vod.path} clips=${config.vod.clipsPath}`);
console.log(`[Media] Data: db=${config.db.path} vods=${config.vod.path} clips=${config.vod.clipsPath}`);
console.log(`[Media] RTP ingest pool: udp ${config.rtp.portMin}-${config.rtp.portMax} (127.0.0.1)`);
console.log(`[Media] RTP ingest pool: udp ${config.rtp.portMin}-${config.rtp.portMax} (127.0.0.1)`);
@@ -225,6 +258,9 @@ const server = app.listen(config.port, config.host, () => {
@@ -225,6 +258,9 @@ const server = app.listen(config.port, config.host, () => {
try { require('./events').init(); } catch (err) { console.warn('[Events] not started:', err.message); }
try { require('./events').init(); } catch (err) { console.warn('[Events] not started:', err.message); }
});
});
+// A drill whose port is taken stops instead of running unready.
+if (drill.enabled) server.once('error', (err) => { console.error(`[Drill] HTTP server: ${err.message}`); process.exit(1); });
+
let shuttingDown = false;
let shuttingDown = false;
function shutdown(signal) {
function shutdown(signal) {
if (shuttingDown) return;
if (shuttingDown) return;
@@ -599,6 +599,8 @@ const INLINE = /^(image\/(?!svg)|video\/|audio\/|application\/pdf$|text\/plain$)
@@ -599,6 +599,8 @@ const INLINE = /^(image\/(?!svg)|video\/|audio\/|application\/pdf$|text\/plain$)
const publicRouter = express.Router();
const publicRouter = express.Router();
publicRouter.get('/:id', async (req, res) => {
publicRouter.get('/:id', async (req, res) => {
+ // A restore drill (MEDIA_DRILL) serves no stored bytes, local or by a B2/R2 redirect.
+ if (require('../drill').refuseBytes(res)) return;
try {
try {
const obj = model.getObject(String(req.params.id || ''));
const obj = model.getObject(String(req.params.id || ''));
if (!obj) return res.status(404).json({ error: 'Not found' });
if (!obj) return res.status(404).json({ error: 'Not found' });
@@ -94,16 +94,19 @@ function domainMetrics(registry, { db, recorder, events }) {
@@ -94,16 +94,19 @@ function domainMetrics(registry, { db, recorder, events }) {
});
});
}
}
-function createMediaReadiness({ release, db, config, auth, recorder, events, remote }) {
+function createMediaReadiness({ release, db, config, auth, recorder, events, remote, drill = false }) {
const checks = [
const checks = [
{ name: 'db', required: true, check: () => { const r = db.get('SELECT COUNT(*) AS n FROM apps'); return { ok: Number.isInteger(r.n), detail: { apps: r.n } }; } },
{ name: 'db', required: true, check: () => { const r = db.get('SELECT COUNT(*) AS n FROM apps'); return { ok: Number.isInteger(r.n), detail: { apps: r.n } }; } },
];
];
- for (const [name, dir] of Object.entries(storageDirs(config))) {
- checks.push({ name: `storage_${name}`, required: true, check: () => writable(dir) });
+ // A restore drill (MEDIA_DRILL) writes no file and asks no remote tier anything: neither is checked.
+ if (!drill) {
+ for (const [name, dir] of Object.entries(storageDirs(config))) {
+ checks.push({ name: `storage_${name}`, required: true, check: () => writable(dir) });
+ }
}
}
checks.push({ name: 'network_jwks', required: false, description: 'user sign-in and service tokens (app keys work without it)', check: () => auth.jwksLoaded() || 'Network public key not loaded yet' });
checks.push({ name: 'network_jwks', required: false, description: 'user sign-in and service tokens (app keys work without it)', check: () => auth.jwksLoaded() || 'Network public key not loaded yet' });
for (const name of ['b2', 'r2']) {
for (const name of ['b2', 'r2']) {
- if (!remote.configured(name)) continue; // not configured: no check, rather than a pretend pass
+ if (drill || !remote.configured(name)) continue; // not configured: no check, rather than a pretend pass
checks.push({ name: `remote_${name}`, required: false, cacheMs: 60 * 1000, timeoutMs: 5000, check: () => remote.probe(name) });
checks.push({ name: `remote_${name}`, required: false, cacheMs: 60 * 1000, timeoutMs: 5000, check: () => remote.probe(name) });
}
}
checks.push({
checks.push({
@@ -129,7 +132,7 @@ function createMediaReadiness({ release, db, config, auth, recorder, events, rem
@@ -129,7 +132,7 @@ function createMediaReadiness({ release, db, config, auth, recorder, events, rem
service: 'media',
service: 'media',
release,
release,
checks,
checks,
- details: () => ({ recordings_in_progress: recorder.activeCount() }),
+ details: () => ({ ...(drill ? { mode: 'drill' } : {}), recordings_in_progress: recorder.activeCount() }),
});
});
}
}
@@ -56,7 +56,7 @@ router.use((req, res, next) => {
@@ -56,7 +56,7 @@ router.use((req, res, next) => {
// ── Screenshot upload storage ───────────────────────────────
// ── Screenshot upload storage ───────────────────────────────
const SCREENSHOTS_DIR = path.join(config.pastes.path, 'screenshots');
const SCREENSHOTS_DIR = path.join(config.pastes.path, 'screenshots');
-if (!fs.existsSync(SCREENSHOTS_DIR)) fs.mkdirSync(SCREENSHOTS_DIR, { recursive: true });
+if (!require('../drill').enabled && !fs.existsSync(SCREENSHOTS_DIR)) fs.mkdirSync(SCREENSHOTS_DIR, { recursive: true }); // a restore drill writes nothing
const MIME_TO_EXT = { 'image/png': '.png', 'image/jpeg': '.jpg', 'image/webp': '.webp', 'image/gif': '.gif' };
const MIME_TO_EXT = { 'image/png': '.png', 'image/jpeg': '.jpg', 'image/webp': '.webp', 'image/gif': '.gif' };
const screenshotStorage = multer.diskStorage({
const screenshotStorage = multer.diskStorage({
@@ -706,13 +706,14 @@ router.post('/:slug/copy', tenantAuth({ allowUser: true }), (req, res) => {
@@ -706,13 +706,14 @@ router.post('/:slug/copy', tenantAuth({ allowUser: true }), (req, res) => {
// Rate-limit state: IP → timestamp of last comment
// Rate-limit state: IP → timestamp of last comment
const commentCooldowns = new Map();
const commentCooldowns = new Map();
-const _cooldownSweep = setInterval(() => {
+// (A restore drill takes no comments and runs no timers.)
+const _cooldownSweep = require('../drill').enabled ? null : setInterval(() => {
const now = Date.now();
const now = Date.now();
for (const [ip, ts] of commentCooldowns) {
for (const [ip, ts] of commentCooldowns) {
if (now - ts > 120_000) commentCooldowns.delete(ip);
if (now - ts > 120_000) commentCooldowns.delete(ip);
}
}
}, 600_000);
}, 600_000);
-if (_cooldownSweep.unref) _cooldownSweep.unref();
+if (_cooldownSweep && _cooldownSweep.unref) _cooldownSweep.unref();
router.get('/:slug/comments', tenantAuth({ allowUser: true }), (req, res) => {
router.get('/:slug/comments', tenantAuth({ allowUser: true }), (req, res) => {
try {
try {
@@ -27,6 +27,9 @@ const db = require('../db/database');
@@ -27,6 +27,9 @@ const db = require('../db/database');
const tools = require('../vod/media-tools');
const tools = require('../vod/media-tools');
const { optionalIdentity } = require('../auth');
const { optionalIdentity } = require('../auth');
const pages = require('./pages');
const pages = require('./pages');
+// A restore drill (MEDIA_DRILL) serves no stored bytes: every byte route answers 503 before it looks
+// at a file path (the database's paths are production's files). Watch pages still render.
+const drill = require('../drill');
const router = express.Router();
const router = express.Router();
@@ -71,6 +74,7 @@ function canAccessPrivate(record, req) {
@@ -71,6 +74,7 @@ function canAccessPrivate(record, req) {
* aborts a range request). Deleted-but-open VOD files once held 80 GB of disk that way.
* aborts a range request). Deleted-but-open VOD files once held 80 GB of disk that way.
*/
*/
function sendFileStream(res, filePath, opts) {
function sendFileStream(res, filePath, opts) {
+ if (drill.enabled) { try { res.destroy(); } catch { /* */ } return null; } // unreachable: every caller refuses first
const stream = fs.createReadStream(filePath, opts);
const stream = fs.createReadStream(filePath, opts);
const done = () => { try { stream.destroy(); } catch { /* */ } };
const done = () => { try { stream.destroy(); } catch { /* */ } };
res.on('close', done); res.on('error', done); res.on('finish', done);
res.on('close', done); res.on('error', done); res.on('finish', done);
@@ -80,6 +84,7 @@ function sendFileStream(res, filePath, opts) {
@@ -80,6 +84,7 @@ function sendFileStream(res, filePath, opts) {
}
}
function streamFileWithRange(req, res, filePath, extraHeaders = {}) {
function streamFileWithRange(req, res, filePath, extraHeaders = {}) {
+ if (drill.refuseBytes(res)) return;
const stat = fs.statSync(filePath);
const stat = fs.statSync(filePath);
const range = req.headers.range;
const range = req.headers.range;
const ext = path.extname(filePath).toLowerCase();
const ext = path.extname(filePath).toLowerCase();
@@ -132,6 +137,7 @@ async function serveMediaRecord(kind, record, req, res) {
@@ -132,6 +137,7 @@ async function serveMediaRecord(kind, record, req, res) {
res.set('Cache-Control', visibility === 'public' ? 'public, max-age=60' : 'private, no-store');
res.set('Cache-Control', visibility === 'public' ? 'public, max-age=60' : 'private, no-store');
return res.type('html').send(pages.renderWatchPage(kind, record));
return res.type('html').send(pages.renderWatchPage(kind, record));
}
}
+ if (drill.refuseBytes(res)) return;
trackUniqueView(kind, record.id, req, record.user_id);
trackUniqueView(kind, record.id, req, record.user_id);
@@ -226,6 +232,7 @@ router.get('/og-image.png', (req, res) => {
@@ -226,6 +232,7 @@ router.get('/og-image.png', (req, res) => {
// ── Thumbnails ───────────────────────────────────────────────
// ── Thumbnails ───────────────────────────────────────────────
router.get('/t/:id', (req, res) => {
router.get('/t/:id', (req, res) => {
+ if (drill.refuseBytes(res)) return;
require('../thumbnails/thumbnail-service').serveThumbnail(req, res);
require('../thumbnails/thumbnail-service').serveThumbnail(req, res);
});
});
@@ -238,6 +245,7 @@ router.get('/t/:id', (req, res) => {
@@ -238,6 +245,7 @@ router.get('/t/:id', (req, res) => {
// consumers can key off the status code (or pass ?format=json).
// consumers can key off the status code (or pass ?format=json).
// Cached 5s per slot (that cache IS the rate limit); CORS-open.
// Cached 5s per slot (that cache IS the rate limit); CORS-open.
router.get('/live/:sel/frame.jpg', async (req, res) => {
router.get('/live/:sel/frame.jpg', async (req, res) => {
+ if (drill.refuseBytes(res)) return; // ffprobe + ffmpeg on the live recording
const frames = require('../thumbnails/live-frame-service');
const frames = require('../thumbnails/live-frame-service');
const wantJson = String(req.query.format || '') === 'json';
const wantJson = String(req.query.format || '') === 'json';
const sendCard = async (status, label, subtitle) => {
const sendCard = async (status, label, subtitle) => {
@@ -299,6 +307,7 @@ router.get('/live/:sel/frame.jpg', async (req, res) => {
@@ -299,6 +307,7 @@ router.get('/live/:sel/frame.jpg', async (req, res) => {
// ── App assets (emotes / sounds) by id ───────────────────────
// ── App assets (emotes / sounds) by id ───────────────────────
router.get('/a/:id', (req, res) => {
router.get('/a/:id', (req, res) => {
+ if (drill.refuseBytes(res)) return;
try {
try {
const a = db.getAssetById(parseInt(req.params.id, 10));
const a = db.getAssetById(parseInt(req.params.id, 10));
if (!a || !a.file_path || !fs.existsSync(a.file_path)) return res.status(404).json({ error: 'Not found' });
if (!a || !a.file_path || !fs.existsSync(a.file_path)) return res.status(404).json({ error: 'Not found' });
@@ -359,6 +368,7 @@ router.get('/v/:id/transcript.json', _devDataRoute((req) => {
@@ -359,6 +368,7 @@ router.get('/v/:id/transcript.json', _devDataRoute((req) => {
// regenerated, so resolve the vod/clip id from the name and redirect to the
// regenerated, so resolve the vod/clip id from the name and redirect to the
// current canonical URL; serve the exact file when it still exists.
// current canonical URL; serve the exact file when it still exists.
router.get('/api/thumbnails/:name', (req, res) => {
router.get('/api/thumbnails/:name', (req, res) => {
+ if (drill.refuseBytes(res)) return;
const name = path.basename(String(req.params.name || ''));
const name = path.basename(String(req.params.name || ''));
const m = /^(vod|clip)-(\d+)-\d+\.(?:jpg|jpeg|png)$/i.exec(name);
const m = /^(vod|clip)-(\d+)-\d+\.(?:jpg|jpeg|png)$/i.exec(name);
if (m) {
if (m) {
@@ -382,6 +392,7 @@ router.get('/api/thumbnails/:name', (req, res) => {
@@ -382,6 +392,7 @@ router.get('/api/thumbnails/:name', (req, res) => {
// basenames on disk and NO files-table rows, so this serves straight from
// basenames on disk and NO files-table rows, so this serves straight from
// PASTES_PATH/screenshots for exactly that namespace.
// PASTES_PATH/screenshots for exactly that namespace.
router.get('/f/screenshots/:name', (req, res) => {
router.get('/f/screenshots/:name', (req, res) => {
+ if (drill.refuseBytes(res)) return;
try {
try {
const name = path.basename(String(req.params.name || ''));
const name = path.basename(String(req.params.name || ''));
const dir = path.join(require('../config').pastes.path, 'screenshots');
const dir = path.join(require('../config').pastes.path, 'screenshots');
@@ -398,6 +409,7 @@ router.get('/f/screenshots/:name', (req, res) => {
@@ -398,6 +409,7 @@ router.get('/f/screenshots/:name', (req, res) => {
// ── Files ────────────────────────────────────────────────────
// ── Files ────────────────────────────────────────────────────
router.get('/f/:key', (req, res) => {
router.get('/f/:key', (req, res) => {
+ if (drill.refuseBytes(res)) return;
try {
try {
const row = db.getFileByKey(String(req.params.key));
const row = db.getFileByKey(String(req.params.key));
if (!row) return res.status(404).json({ error: 'Not found' });
if (!row) return res.status(404).json({ error: 'Not found' });
@@ -524,6 +536,7 @@ router.get('/p/:slug/raw', (req, res) => {
@@ -524,6 +536,7 @@ router.get('/p/:slug/raw', (req, res) => {
});
});
router.get('/p/:slug/screenshot', (req, res) => {
router.get('/p/:slug/screenshot', (req, res) => {
+ if (drill.refuseBytes(res)) return;
try {
try {
const paste = db.getPasteBySlug(String(req.params.slug));
const paste = db.getPasteBySlug(String(req.params.slug));
// Pastes made since the move live in Community only: send an unknown slug there, like
// Pastes made since the move live in Community only: send an unknown slug there, like
@@ -29,7 +29,8 @@ const THUMB_WIDTH = 640;
@@ -29,7 +29,8 @@ const THUMB_WIDTH = 640;
const THUMB_QUALITY = 6; // ffmpeg qscale:v (2=best, 31=worst, 6 is good balance)
const THUMB_QUALITY = 6; // ffmpeg qscale:v (2=best, 31=worst, 6 is good balance)
const CLIENT_THUMB_WRITE_MIN_INTERVAL_MS = 15000;
const CLIENT_THUMB_WRITE_MIN_INTERVAL_MS = 15000;
-if (!fs.existsSync(THUMB_DIR)) {
+// (A restore drill writes nothing, directories included.)
+if (!require('../drill').enabled && !fs.existsSync(THUMB_DIR)) {
fs.mkdirSync(THUMB_DIR, { recursive: true });
fs.mkdirSync(THUMB_DIR, { recursive: true });
}
}
@@ -267,6 +268,7 @@ function saveLiveThumbnail(appId, streamId, imageData) {
@@ -267,6 +268,7 @@ function saveLiveThumbnail(appId, streamId, imageData) {
// ── Serve Thumbnail File (public /t/:id) ─────────────────────
// ── Serve Thumbnail File (public /t/:id) ─────────────────────
function serveThumbnail(req, res) {
function serveThumbnail(req, res) {
+ if (require('../drill').refuseBytes(res)) return; // a restore drill serves no stored file
try {
try {
const filename = path.basename(req.params.id || req.params.filename || ''); // prevent traversal
const filename = path.basename(req.params.id || req.params.filename || ''); // prevent traversal
const filePath = path.join(THUMB_DIR, filename);
const filePath = path.join(THUMB_DIR, filename);
@@ -89,8 +89,9 @@ function createAuthClient(config) {
@@ -89,8 +89,9 @@ function createAuthClient(config) {
return client.verifyToken(token);
return client.verifyToken(token);
}
}
- // Warm the key cache at boot (non-fatal if the Network is down)
- ensureKey().catch(() => {});
+ // Warm the key cache at boot (non-fatal if the Network is down). A restore drill asks Network
+ // for nothing (MEDIA_DRILL; its /auth routes answer 503).
+ if (!require('./drill').enabled) ensureKey().catch(() => {});
return { client, ensureKey, verify };
return { client, ensureKey, verify };
}
}
@@ -150,7 +150,8 @@ class StreamRecorder {
@@ -150,7 +150,8 @@ class StreamRecorder {
this._diskState = 'ok'; // 'ok' | 'warning' | 'critical'
this._diskState = 'ok'; // 'ok' | 'warning' | 'critical'
const vodDir = path.resolve(config.vod.path);
const vodDir = path.resolve(config.vod.path);
- if (!fs.existsSync(vodDir)) fs.mkdirSync(vodDir, { recursive: true });
+ // A restore drill (MEDIA_DRILL) records nothing and writes nothing, directories included.
+ if (!require('../drill').enabled && !fs.existsSync(vodDir)) fs.mkdirSync(vodDir, { recursive: true });
}
}
isRecording(vodId) {
isRecording(vodId) {
@@ -322,7 +322,8 @@ router.get('/:id', tenantAuth({ allowUser: true }), async (req, res) => {
@@ -322,7 +322,8 @@ router.get('/:id', tenantAuth({ allowUser: true }), async (req, res) => {
}
}
// Self-heal a missing duration on a finished local VOD.
// Self-heal a missing duration on a finished local VOD.
- if (!vod.is_recording && (!vod.duration_seconds || vod.duration_seconds <= 0) && vod.file_path && fs.existsSync(vod.file_path)) {
+ // (Never in a restore drill: it opens no stored file and runs no ffprobe.)
+ if (!require('../drill').enabled && !vod.is_recording && (!vod.duration_seconds || vod.duration_seconds <= 0) && vod.file_path && fs.existsSync(vod.file_path)) {
const duration = await tools.probeVodDuration(vod.file_path);
const duration = await tools.probeVodDuration(vod.file_path);
if (duration > 0) {
if (duration > 0) {
const fileSize = tools.getFileSizeSafe(vod.file_path);
const fileSize = tools.getFileSizeSafe(vod.file_path);
@@ -49,6 +49,8 @@ async function _post(url, rawBody, signature) {
@@ -49,6 +49,8 @@ async function _post(url, rawBody, signature) {
* @param {string} [opts.eventId] the OpenVibe.Events event_id of the same outcome
* @param {string} [opts.eventId] the OpenVibe.Events event_id of the same outcome
*/
*/
async function sendWebhook(appOrId, event, data, { eventId = null } = {}) {
async function sendWebhook(appOrId, event, data, { eventId = null } = {}) {
+ // A restore drill (MEDIA_DRILL) never calls an app: the URLs in its apps table are production's.
+ if (require('./drill').enabled) return false;
let app = appOrId;
let app = appOrId;
if (typeof appOrId === 'string') {
if (typeof appOrId === 'string') {
try { app = db.getApp(appOrId); } catch { app = null; }
try { app = db.getApp(appOrId); } catch { app = null; }
new file mode 100644
new file mode 100644
@@ -0,0 +1,310 @@
@@ -0,0 +1,310 @@
+'use strict';
+/**
+ * MEDIA_DRILL=1 (server/drill.js): a restore-drill instance (`ovhost drill media`, OpenVibe.Host
+ * docs/restore-drills.md) serves reads from a restored copy of media.db and does nothing else.
+ *
+ * - It refuses to start when DB_PATH is production's media.db or inside the checkout, when HOST is
+ * not loopback or PORT is production's 4100, before the database is opened (a real
+ * `node server/index.js`, in a child process).
+ * - Booted for real (server/index.js, in this process) on a copy whose `live` app has a webhook
+ * URL, with production-like settings (EVENTS_URL + client secret, B2/R2, app seeding, VOD_PATH …):
+ * one listener, its HTTP port on 127.0.0.1; no setInterval and no long timer from Media's code;
+ * no program but git (no ffmpeg/ffprobe); no outbound connection; no webhook reaches the app;
+ * nothing written or created outside the copy's directory.
+ * - The endpoints the Host inventory compares (/release.json, the /browse index) and the watch pages
+ * render from the copy; every route that would stream a stored file answers 503 without opening
+ * the path the database gives it.
+ * - Writes answer 403, sign-in 503, and the copy's rows stay as they were.
+ *
+ * Run: node test/drill-mode.test.js
+ */
+const assert = require('assert');
+const crypto = require('crypto');
+const fs = require('fs');
+const http = require('http');
+const net = require('net');
+const os = require('os');
+const path = require('path');
+const { spawnSync, execFileSync } = require('child_process');
+
+const REPO = path.resolve(__dirname, '..');
+const SERVER = path.join(REPO, 'server');
+const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'media-drill-test-'));
+// "Production": its database and its storage, with real files the database points at.
+const PROD = path.join(tmp, 'prod');
+const P = { vods: path.join(PROD, 'vods'), clips: path.join(PROD, 'clips'), pastes: path.join(PROD, 'pastes'), files: path.join(PROD, 'files'), objects: path.join(PROD, 'objects'), assets: path.join(PROD, 'assets'), thumbs: path.join(PROD, 'thumbnails') };
+// The drill: the restored copy in its own directory; storage the way the Host inventory sets it
+// (every directory but thumbnails moved under the drill directory, never created).
+const DRILL = path.join(tmp, 'drill');
+const DB_PATH = path.join(DRILL, 'db', 'media.db');
+const STORAGE = path.join(DRILL, 'storage');
+
+let failures = 0;
+async function check(name, fn) {
+ try { await fn(); console.log(` ✓ ${name}`); } catch (err) { failures++; console.error(` ✗ ${name}\n ${err.stack || err.message}`); }
+}
+function freePort() {
+ return new Promise((resolve) => { const s = net.createServer().listen(0, '127.0.0.1', () => { const p = s.address().port; s.close(() => resolve(p)); }); });
+}
+/** Every file and directory under dir with its size and mtime. */
+function snapshot(dir) {
+ const out = {};
+ const walk = (d) => { for (const e of fs.readdirSync(d, { withFileTypes: true })) { const p = path.join(d, e.name); const s = fs.statSync(p); out[path.relative(dir, p)] = `${e.isDirectory() ? 'dir' : s.size}:${s.mtimeMs}`; if (e.isDirectory()) walk(p); } };
+ walk(dir);
+ return out;
+}
+
+const drillMod = require('../server/drill'); // MEDIA_DRILL is unset here: `enabled` is false until the boot below
+const safe = { DB_PATH, HOST: '127.0.0.1', PORT: '14100' };
+
+(async () => {
+ console.log('drill-mode: environment guard');
+ await check('MEDIA_DRILL parses like a switch and is off by default', () => {
+ for (const v of ['1', 'true', 'on', 'yes']) assert.strictEqual(drillMod.parse(v), true, v);
+ for (const v of ['', '0', 'false', 'off', undefined]) assert.strictEqual(drillMod.parse(v), false, String(v));
+ assert.strictEqual(drillMod.enabled, false);
+ });
+ await check('a safe environment passes', () => assert.deepStrictEqual(drillMod.problems(safe), []));
+ await check('refuses production\'s media.db (the checkout\'s, /opt/openvibe.media\'s, the relative default)', () => {
+ for (const p of [path.join(REPO, 'data', 'media.db'), '/opt/openvibe.media/data/media.db']) {
+ assert.ok(drillMod.problems({ ...safe, DB_PATH: p }).some((m) => /production's database/.test(m)), p);
+ }
+ assert.ok(drillMod.problems({ ...safe, DB_PATH: './data/media.db' }, { cwd: REPO }).some((m) => /production's database/.test(m)));
+ assert.ok(drillMod.problems({ ...safe, DB_PATH: '' }).some((m) => /DB_PATH is not set/.test(m)));
+ });
+ await check('refuses a copy inside the checkout or /opt/openvibe.media, also through a symlink', () => {
+ assert.ok(drillMod.problems({ ...safe, DB_PATH: path.join(REPO, 'drill.db') }).some((m) => /is inside/.test(m)));
+ assert.ok(drillMod.problems({ ...safe, DB_PATH: '/opt/openvibe.media/drill/media.db' }).some((m) => /is inside \/opt\/openvibe\.media/.test(m)));
+ const link = path.join(tmp, 'sneaky');
+ fs.symlinkSync(REPO, link);
+ assert.ok(drillMod.problems({ ...safe, DB_PATH: path.join(link, 'copy.db') }).some((m) => /is inside/.test(m)));
+ fs.unlinkSync(link);
+ });
+ await check('refuses a non-loopback HOST and production\'s PORT', () => {
+ for (const h of ['', '0.0.0.0', '::', '10.0.0.4']) assert.ok(drillMod.problems({ ...safe, HOST: h }).some((m) => /not loopback/.test(m)), h);
+ for (const h of ['127.0.0.1', '::1', 'localhost']) assert.deepStrictEqual(drillMod.problems({ ...safe, HOST: h }), [], h);
+ assert.ok(drillMod.problems({ ...safe, PORT: '4100' }).some((m) => /production's port/.test(m)));
+ assert.ok(drillMod.problems({ ...safe, PORT: '' }).some((m) => /PORT is not set/.test(m)));
+ });
+
+ // ── A real `node server/index.js` refuses before opening anything ──
+ console.log('drill-mode: refusal at boot');
+ const refusalPort = await freePort();
+ const boot = (env) => spawnSync(process.execPath, [path.join(SERVER, 'index.js')], {
+ cwd: REPO, encoding: 'utf8', timeout: 30000,
+ env: { ...process.env, MEDIA_DRILL: '1', HOST: '127.0.0.1', PORT: String(refusalPort), ...env },
+ });
+ const stamp = (p) => { try { const s = fs.statSync(p); return `${s.size}:${s.mtimeMs}`; } catch { return 'absent'; } };
+ await check('DB_PATH = production\'s media.db: exit 1, the file untouched', () => {
+ const prod = path.join(REPO, 'data', 'media.db');
+ const before = [prod, `${prod}-wal`, `${prod}-shm`].map(stamp);
+ const r = boot({ DB_PATH: prod });
+ assert.strictEqual(r.status, 1, r.stdout + r.stderr);
+ assert.match(r.stderr, /refusing to start a restore-drill instance: DB_PATH .* is production's database/);
+ assert.deepStrictEqual([prod, `${prod}-wal`, `${prod}-shm`].map(stamp), before);
+ });
+ await check('HOST 0.0.0.0: exit 1 before the database is created', () => {
+ const other = path.join(tmp, 'never', 'media.db');
+ const r = boot({ DB_PATH: other, HOST: '0.0.0.0' });
+ assert.strictEqual(r.status, 1, r.stdout + r.stderr);
+ assert.match(r.stderr, /HOST \(0\.0\.0\.0\) is not loopback/);
+ assert.ok(!fs.existsSync(path.dirname(other)));
+ });
+
+ // ── Production's database and files (a normal-mode process writes them) ──
+ // The `live` app's webhook goes to a sink here: a drill must never call it.
+ const sinkHits = [];
+ const sink = http.createServer((req, res) => { sinkHits.push(`${req.method} ${req.url}`); res.end('ok'); });
+ await new Promise((r) => sink.listen(0, '127.0.0.1', r));
+ const SINK = `http://127.0.0.1:${sink.address().port}/media-webhook`;
+ for (const d of Object.values(P)) fs.mkdirSync(d, { recursive: true });
+ const bytes = { vod: path.join(P.vods, 'vod-1.mp4'), rec: path.join(P.vods, 'vod-2.mp4'), clip: path.join(P.clips, 'clip-1.mp4'), asset: path.join(P.assets, 'wave.png'), shot: path.join(P.pastes, 'shot-1.png'), file: path.join(P.files, 'live', 'f1.txt') };
+ fs.mkdirSync(path.dirname(bytes.file), { recursive: true });
+ for (const f of Object.values(bytes)) fs.writeFileSync(f, `bytes of ${path.basename(f)}`);
+ fs.writeFileSync(path.join(P.thumbs, 'vod-1-1.jpg'), 'jpeg');
+ const prodDb = path.join(PROD, 'media.db');
+ const seed = spawnSync(process.execPath, ['-e', `
+ const db = require('./server/db/database');
+ db.getDb();
+ require('./server/views/service').ensureSchema();
+ const b = ${JSON.stringify(bytes)};
+ db.run("INSERT INTO apps (app_id, name, api_key_hash, webhook_url, webhook_secret) VALUES ('live', 'OpenVibe.Live', ?, ?, 'whsec')", [require('crypto').createHash('sha256').update('k'.repeat(40)).digest('hex'), ${JSON.stringify(SINK)}]);
+ db.run("INSERT INTO vods (app_id, title, file_path, thumbnail_url, duration_seconds, is_public, visibility) VALUES ('live', 'Restored VOD', ?, '/t/vod-1-1.jpg', 60, 1, 'public')", [b.vod]);
+ db.run("INSERT INTO vods (app_id, title, file_path, is_public, visibility, is_recording) VALUES ('live', 'Still recording', ?, 1, 'public', 1)", [b.rec]);
+ db.run("INSERT INTO clips (app_id, vod_id, title, file_path, duration_seconds, status) VALUES ('live', 1, 'Restored clip', ?, 10, 'ready')", [b.clip]);
+ db.run("INSERT INTO assets (app_id, kind, name, file_path, mime) VALUES ('live', 'emote', 'drillWave', ?, 'image/png')", [b.asset]);
+ db.run("INSERT INTO pastes (app_id, slug, type, title, screenshot_path, visibility) VALUES ('live', 'shot1', 'screenshot', 'Shot', ?, 'public')", [b.shot]);
+ db.run("INSERT INTO files (key, app_id, original_name, size, mime) VALUES ('f1.txt', 'live', 'f1.txt', 13, 'text/plain')");
+ require('./server/objects/backfill').backfill({ onlyMissing: true });
+ db.close();
+ `], { cwd: REPO, encoding: 'utf8', env: { ...process.env, DB_PATH: prodDb, VOD_PATH: P.vods, CLIPS_PATH: P.clips, PASTES_PATH: P.pastes, FILES_PATH: P.files, OBJECTS_PATH: P.objects, ASSETS_PATH: P.assets, THUMBNAILS_PATH: P.thumbs } });
+ assert.strictEqual(seed.status, 0, `seeding failed:\n${seed.stdout}\n${seed.stderr}`);
+ // The restore: a copy of the database file in the drill's own directory.
+ fs.mkdirSync(path.dirname(DB_PATH), { recursive: true });
+ fs.copyFileSync(prodDb, DB_PATH);
+ const Database = require('better-sqlite3');
+ const COUNTED = ['media_objects', 'vods', 'clips', 'apps', 'media_locations', 'media_jobs', 'event_outbox'];
+ const counts = () => { const d = new Database(DB_PATH, { readonly: true }); try { return Object.fromEntries(COUNTED.map((t) => { try { return [t, d.prepare(`SELECT count(*) AS n FROM ${t}`).get().n]; } catch { return [t, null]; } })); } finally { d.close(); } };
+ const countsBefore = counts();
+ const objectId = (() => { const d = new Database(DB_PATH, { readonly: true }); try { return d.prepare("SELECT id FROM media_objects WHERE kind = 'vod' ORDER BY id LIMIT 1").get().id; } finally { d.close(); } })();
+ const prodBefore = snapshot(PROD);
+
+ // ── Boot the real server in drill mode, in this process, with spies ──
+ const port = await freePort();
+ Object.assign(process.env, {
+ MEDIA_DRILL: '1', DB_PATH, HOST: '127.0.0.1', PORT: String(port), NODE_ENV: 'production',
+ // What the production env file sets; a normal boot would act on all of it.
+ EVENTS_URL: 'http://127.0.0.1:9', OV_OAUTH_CLIENT_SECRET: 's'.repeat(40), INTERNAL_API_KEY: 'k'.repeat(40),
+ MEDIA_APP_KEYS: `live:${'n'.repeat(40)},games:${'g'.repeat(40)}`,
+ MEDIA_B2_ENDPOINT: 'https://s3.example.invalid', MEDIA_B2_BUCKET: 'b', MEDIA_B2_KEY_ID: 'id', MEDIA_B2_APP_KEY: 'key',
+ MEDIA_R2_ENDPOINT: 'https://r2.example.invalid', MEDIA_R2_BUCKET: 'r', MEDIA_R2_ACCESS_KEY_ID: 'id', MEDIA_R2_SECRET_ACCESS_KEY: 'key',
+ // The Host inventory's storage: nothing but the thumbnail listing points at production.
+ VOD_PATH: path.join(STORAGE, 'vods'), CLIPS_PATH: path.join(STORAGE, 'clips'), PASTES_PATH: path.join(STORAGE, 'pastes'),
+ FILES_PATH: path.join(STORAGE, 'files'), OBJECTS_PATH: path.join(STORAGE, 'objects'), ASSETS_PATH: path.join(STORAGE, 'assets'),
+ THUMBNAILS_PATH: P.thumbs,
+ });
+ const inServer = (stack) => {
+ const frame = stack.split('\n').slice(2).find((l) => l.includes(REPO) && !l.includes(`${path.sep}node_modules${path.sep}`) && !l.includes(__filename));
+ return frame && frame.includes(`${SERVER}${path.sep}`) ? frame.trim() : null;
+ };
+ const spied = { intervals: [], longTimeouts: [], listens: [], programs: [], dbPathTouches: [] };
+ const realSetInterval = global.setInterval;
+ global.setInterval = function (fn, ms, ...a) { spied.intervals.push({ ms, at: (new Error().stack.split('\n')[2] || '').trim() }); return realSetInterval(fn, ms, ...a); };
+ const realSetTimeout = global.setTimeout;
+ global.setTimeout = function (fn, ms, ...a) { if (ms >= 1000) { const at = inServer(new Error().stack); if (at) spied.longTimeouts.push({ ms, at }); } return realSetTimeout(fn, ms, ...a); };
+ const realListen = net.Server.prototype.listen;
+ net.Server.prototype.listen = function (...a) { spied.listens.push(a[0]); return realListen.apply(this, a); };
+ const cp = require('child_process');
+ for (const fn of ['spawn', 'spawnSync', 'exec', 'execSync', 'execFile', 'execFileSync', 'fork']) {
+ const orig = cp[fn];
+ cp[fn] = function (...a) { spied.programs.push(fn === 'exec' || fn === 'execSync' ? String(a[0]).split(/\s+/)[0] : String(a[0])); return orig.apply(this, a); };
+ }
+ // Any look at a file path the database holds (production's media) is recorded.
+ const dbPaths = new Set(Object.values(bytes));
+ for (const fn of ['createReadStream', 'statSync', 'existsSync', 'openSync', 'readFileSync', 'stat', 'open', 'readFile']) {
+ const orig = fs[fn];
+ fs[fn] = function (p, ...a) { if (dbPaths.has(String(p))) spied.dbPathTouches.push(`${fn} ${p}`); return orig.call(this, p, ...a); };
+ }
+
+ delete require.cache[require.resolve('../server/drill')]; // loaded above with MEDIA_DRILL unset
+ require('../server/index.js');
+ const drill = require('../server/drill');
+ const base = `http://127.0.0.1:${port}`;
+ const request = (method, p, { headers = {}, body = null } = {}) => new Promise((resolve, reject) => {
+ const req = http.request(base + p, { method, headers: { ...(body ? { 'Content-Type': 'application/json' } : {}), ...headers } }, (res) => {
+ let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, text: b, json: (() => { try { return JSON.parse(b); } catch { return null; } })() }));
+ });
+ req.on('error', reject);
+ if (body) req.write(typeof body === 'string' ? body : JSON.stringify(body));
+ req.end();
+ });
+ let ready = null;
+ for (let i = 0; i < 150 && !(ready && ready.status === 200); i++) {
+ try { ready = await request('GET', '/api/ready'); } catch { /* not listening yet */ }
+ if (!ready || ready.status !== 200) await new Promise((r) => realSetTimeout(r, 100));
+ }
+ await new Promise((r) => realSetTimeout(r, 1500)); // let anything deferred show itself
+
+ console.log('drill-mode: a booted drill instance');
+ await check('/api/ready is 200, mode: drill, and checks no storage and no remote tier', () => {
+ assert.strictEqual(ready && ready.status, 200, ready && ready.text);
+ assert.strictEqual(ready.json.ready, true);
+ assert.strictEqual(ready.json.mode, 'drill');
+ assert.deepStrictEqual(Object.keys(ready.json.checks).filter((k) => /^(storage|remote)_/.test(k)), []);
+ assert.strictEqual(ready.json.checks.events_outbox.detail.enabled, false);
+ });
+ await check('one listener: its HTTP port on 127.0.0.1 (no RTP, no other server)', () => {
+ assert.deepStrictEqual(spied.listens.map(String), [String(port)]);
+ const servers = process._getActiveHandles().filter((h) => h instanceof net.Server && h.listening && h !== sink);
+ assert.deepStrictEqual(servers.map((s) => s.address()), [{ address: '127.0.0.1', family: 'IPv4', port }]);
+ });
+ await check('no timers: no setInterval at all, no timer of a second or more from Media\'s code', () => {
+ assert.deepStrictEqual(spied.intervals, [], JSON.stringify(spied.intervals, null, 1));
+ assert.deepStrictEqual(spied.longTimeouts, [], JSON.stringify(spied.longTimeouts, null, 1));
+ });
+ await check('no program but git ran, nothing tried to leave, no webhook reached the app', () => {
+ assert.ok(spied.programs.every((p) => path.basename(p) === 'git'), spied.programs.join(', '));
+ assert.deepStrictEqual(drill.blocked, []);
+ assert.deepStrictEqual(sinkHits, []);
+ });
+ await check('no app seeding: the copy\'s apps are production\'s, untouched by MEDIA_APP_KEYS', () => {
+ assert.deepStrictEqual(counts(), countsBefore);
+ });
+
+ console.log('drill-mode: reads');
+ const head = execFileSync('git', ['rev-parse', '--short=12', 'HEAD'], { cwd: REPO, encoding: 'utf8' }).trim();
+ await check('the endpoints the Host inventory compares render from the copy and ask nobody', async () => {
+ const rel = await request('GET', '/release.json');
+ assert.strictEqual(rel.status, 200);
+ assert.strictEqual(rel.json.release, head);
+ const videos = await request('GET', '/browse?tab=videos');
+ assert.strictEqual(videos.status, 200);
+ assert.ok(videos.text.includes('Restored VOD') && !videos.text.includes('Still recording'));
+ assert.match(videos.text, /Thumbnails<span class="n">1</, 'the thumbnail listing is production\'s directory');
+ const clips = await request('GET', '/browse?tab=clips');
+ assert.strictEqual(clips.status, 200);
+ assert.ok(clips.text.includes('Restored clip'));
+ assert.strictEqual((await request('GET', '/')).status, 200);
+ assert.strictEqual((await request('GET', '/healthz')).status, 200);
+ const page = await request('GET', '/v/1', { headers: { Accept: 'text/html' } });
+ assert.strictEqual(page.status, 200);
+ assert.ok(page.text.includes('Restored VOD'), 'the watch page renders from the row');
+ assert.deepStrictEqual(drill.blocked, []);
+ assert.deepStrictEqual(spied.dbPathTouches, []);
+ });
+ await check('every route that would stream a stored file answers 503 without opening the database\'s path', async () => {
+ const byteRoutes = ['/v/1', '/v/1?raw=1', '/v/vod-1.mp4', '/c/1', '/c/1?raw=1', `/o/${objectId}`, '/t/vod-1-1.jpg', '/api/thumbnails/vod-1-1.jpg',
+ '/a/1', '/f/f1.txt', '/f/screenshots/shot-1.png', '/p/shot1/screenshot', '/live/1/frame.jpg'];
+ for (const p of byteRoutes) {
+ const r = await request('GET', p, { headers: { Range: 'bytes=0-' } });
+ assert.strictEqual(r.status, 503, `${p}: ${r.status} ${r.text.slice(0, 120)}`);
+ assert.strictEqual(r.json && r.json.code, 'media.drill_no_bytes', p);
+ }
+ assert.deepStrictEqual(spied.dbPathTouches, [], 'no stored file was looked at');
+ assert.deepStrictEqual(spied.programs.filter((p) => path.basename(p) !== 'git'), [], 'no ffmpeg/ffprobe');
+ });
+
+ console.log('drill-mode: writes, sign-in, side effects');
+ await check('writes answer 403 on every path and change nothing; sign-in answers 503', async () => {
+ const auth = { Authorization: `Bearer ${'k'.repeat(40)}` };
+ const writes = [
+ ['POST', '/api/v1/live/vods', { title: 'x' }], ['PUT', '/api/v1/live/vods/1', { title: 'y' }], ['DELETE', '/api/v1/live/vods/1', null],
+ ['POST', '/api/v1/live/clips', { vod_id: 1 }], ['PUT', `/api/v2/live/objects/${objectId}/content`, 'raw bytes'], ['POST', '/api/v2/live/jobs', { type: 'thumbnail' }],
+ ['POST', '/api/v1/live/admin/storage/tiers/sweep', {}], ['POST', '/internal/avatar-ingest', {}], ['PATCH', '/api/v1/live/vods/1', {}],
+ ];
+ for (const [m, p, body] of writes) {
+ const r = await request(m, p, { headers: auth, body });
+ assert.strictEqual(r.status, 403, `${m} ${p}: ${r.status}`);
+ assert.strictEqual(r.json && r.json.code, 'media.drill_read_only', `${m} ${p}`);
+ }
+ const login = await request('GET', '/auth/login');
+ assert.strictEqual(login.status, 503);
+ assert.deepStrictEqual(counts(), countsBefore);
+ });
+ await check('webhooks and the Events relay stay off even when called', async () => {
+ assert.strictEqual(await require('../server/webhooks').sendWebhook('live', 'vod.ready', { id: 1 }), false);
+ assert.strictEqual(require('../server/events').init(), null);
+ assert.deepStrictEqual(sinkHits, []);
+ });
+ await check('the guards refuse a connection, ffmpeg, a listener and a UDP socket', async () => {
+ await assert.rejects(fetch('https://openvibe.live/internal/media-webhook'), /fetch failed/);
+ const sock = net.connect({ host: '127.0.0.1', port: sink.address().port });
+ const err = await new Promise((resolve) => sock.on('error', resolve));
+ assert.strictEqual(err.code, 'ECONNREFUSED');
+ assert.deepStrictEqual(sinkHits, [], 'the connection never reached the app');
+ assert.throws(() => cp.spawn('ffmpeg', ['-version']), (e) => e.code === 'EDRILL');
+ assert.throws(() => net.createServer().listen(12000), (e) => e.code === 'EDRILL');
+ assert.throws(() => require('dgram').createSocket('udp4'), (e) => e.code === 'EDRILL');
+ });
+ await check('production\'s files are untouched and the drill created no storage directory', () => {
+ assert.deepStrictEqual(snapshot(PROD), prodBefore);
+ assert.ok(!fs.existsSync(STORAGE), `${STORAGE} was created`);
+ assert.deepStrictEqual(fs.readdirSync(DRILL), ['db']);
+ });
+
+ sink.close();
+ try { fs.rmSync(tmp, { recursive: true, force: true }); } catch { /* */ }
+ if (failures) { console.error(`\n${failures} drill-mode check(s) failed`); process.exit(1); }
+ console.log('\ndrill-mode: all checks passed');
+ process.exit(0);
+})().catch((err) => { console.error(err); process.exit(1); });