openvibers/openvibe.media Commits
Storage orphan report (WS-G task 8), report only: server/vod/orphans-report.js buildStorageReport compares what storage holds with what the database names and lists files under the data directories and B2/R2 keys that no row names (with hints: a recorder name whose VOD row is gone, bytes of a deleted object, an R2 copy left after its VOD went back to B2, parked vods-orphans/ keys, parts of a session that no longer exists, a temp of an upload that never finished), copies the database records for live objects that are not there (no local file, a key absent from its bucket's listing), and multipart uploads left open (media_uploads sessions, expired ones flagged, and the buckets' own via ListMultipartUploads, new read-only vodStorage.listMultipartUploads). Live thumbnails and temps younger than an hour are counted as expected. It never deletes, moves or writes a file, an object or a row; it writes <data>/reports/storage-orphans-<time>.json. The job storage.orphans.scan runs it every MEDIA_ORPHAN_SCAN_DAYS (30; 0 = never) from the jobs worker (off in a restore drill) under the system app _media: no apps row, no media.job events, and POST /api/v2/:app/jobs refuses it (403 media.job.forbidden). On demand: scripts/vods-orphans-report.js --storage [--no-remote]. test/storage-orphans.test.js snapshots files, tables and buckets before and after, and its fake S3 refuses and counts any write
Changes
@@@ -52,7 +52,8 @@ server/cancellation), routes (/api/v2/:app/jobs), types: thumbnail.regenerate,invariant.scan (proposes split/remux), object.split, object.remux,vod.finalize (orphans and failed finalizes, with backoff),-- vod.duration.reconcile (stored vs measured durations, local and B2/R2)++ vod.duration.reconcile (stored vs measured durations, local and B2/R2),++ storage.orphans.scan (monthly storage orphan report, service-wide, report only)client-ip.js trust proxy = loopback; req.ip is the only client address(openvibe-shared v1.5.1, openvibe-contracts v0.33.0, openvibe-sdk v0.5.0: pinned release tarballs, installed by npm)scripts/smoke-test.sh end-to-end smoke test (boots a temp instance)@@@ -60,7 +61,9 @@ scripts/backfill-objects.js / reconcile-objects.js / object-invariant.js / no-goscripts/media-jobs.js list jobs, run the size-invariant scan (dry run by default), approve/cancel proposalsscripts/r2-eviction-drill.js evict one VOD's R2 copy, prove B2 serves it, re-warm R2; JSON artifact (dry run by default)scripts/vod-duration-reconcile.js stored VOD durations vs the real files (dry run by default; --apply --backup; --rollback)--scripts/vods-orphans-report.js read-only report on the B2 vods-orphans/ prefix (matches, sizes, recommendations)++scripts/vods-orphans-report.js read-only report on the B2 vods-orphans/ prefix (matches, sizes, recommendations);++ --storage: the whole storage against the database (files and keys no row names,++ copies that are not there, open multipart uploads), as the monthly job does```## Visitor sign-in@@@ -410,6 +413,10 @@ model current. Full reference: **[docs/object-model.md](docs/object-model.md)**.read-only by default; `--verify` HEADs B2/R2 and records location states.Reports missing canonical copies, lost local files, size/hash mismatches,orphan locations, deleted-but-still-served objects and unprojected rows.++- **Orphan report** — monthly job `storage.orphans.scan` (`MEDIA_ORPHAN_SCAN_DAYS`, 30) and++ `node scripts/vods-orphans-report.js --storage [--no-remote]`: local files and B2/R2 keys no row++ names, recorded copies that are not there, open multipart uploads (Media's and the buckets').++ Report only: it never deletes anything ([docs](docs/object-model.md#storage-orphan-report)).- **Public object-size invariant** — `MEDIA_PUBLIC_OBJECT_MAX_MB` (500) /`_TARGET_MB` (256) / `_WARN_MB` (384): v2 refuses oversized public playbackuploads; `node scripts/object-invariant.js` lists offenders and records them in@@@ -367,9 +367,45 @@ The exit code is 1 when issues were found.| `missing_projection` | an inherited row with no object yet (run the backfill) || `incomplete_multipart` | a multipart session still open past its expiry (the hourly purge should have removed it) |--**Not covered yet:** orphan *provider* objects (bucket keys with no row, which needs bucket--listing). `server/objects/reconcile.js` takes a `head` function,--so tests run it against a fake provider.++Bucket keys and local files that no row names are the [storage orphan report](#storage-orphan-report)'s.++`server/objects/reconcile.js` takes a `head` function, so tests run it against a fake provider.++++## Storage orphan report++++Code: `server/vod/orphans-report.js` (`buildStorageReport`), the job `storage.orphans.scan`++(`server/jobs/storage-orphans.js`), `scripts/vods-orphans-report.js --storage`. Test: `test/storage-orphans.test.js`++(a fake S3 through the real SDK that refuses and counts any write).++++It compares what storage holds with what the database names, and **only reports**: it walks the data++directories, lists the B2/R2 buckets (keys and open multipart uploads), reads the database and writes++`<data>/reports/storage-orphans-<time>.json`. It never deletes, moves or changes a file, a bucket object++or a row.++++| section | what it lists |++|---|---|++| `unreferenced_local` | files under `VOD_PATH`, `CLIPS_PATH`, `THUMBNAILS_PATH`, `FILES_PATH`, `PASTES_PATH`, `OBJECTS_PATH`, `ASSETS_PATH` that no row names (`media_locations`, `vods` with their sidecars, masters and chunk segments, `clips`, `files`, `pastes`, `assets`, open upload sessions, unfinished jobs). A file whose only row is a deleted object's location counts, unless that object is native and inside its retention period, or held. Largest first, with a hint: a recorder name whose VOD row is gone, bytes of a deleted object, parts of a session that no longer exists, a temp file of an upload that never finished |++| `unreferenced_remote` | B2/R2 keys that no `media_locations` row and no `vods`/`clips` row names; hints for `vods-orphans/` (see below), an R2 copy left behind after its VOD went back to B2, legacy paste screenshots, recorder names |++| `missing` | copies the database records for objects that are not deleted, whose bytes are not there: no local file, or a key absent from its bucket's listing |++| `multipart_local` | Media's own multipart sessions still open (`media_uploads` active or completing), with `expired` when past `expires_at` |++| `multipart_remote` | multipart uploads the buckets still hold open (a killed upload leaves its parts stored and billed) |++++Totals count everything; each list holds at most `limit` items (2000). Live thumbnails++(`stream-<app>-<id>.jpg`) and upload temps or partial downloads younger than an hour are counted as++`expected`, not listed. A provider that is not configured, or whose listing fails, is marked not listed++and nothing is concluded about its keys.++++**When it runs.** The job worker queues `storage.orphans.scan` every `MEDIA_ORPHAN_SCAN_DAYS` (30; `0`++= never). Storage is shared by every tenant, so the job belongs to none: it runs under the system app++`_media` (no apps row, so no API credential reaches it), announces no `media.job.*` event, and++`POST /api/v2/:app/jobs` refuses it with 403 `media.job.forbidden`. The worker does not start in a++restore drill. On demand:++++```++node scripts/vods-orphans-report.js --storage [--no-remote] [--limit 2000] [--out report.json] [--json] [--db ./data/media.db]++```++++`--no-remote` lists no bucket. The detailed per-object recommendations for the `vods-orphans/` prefix++stay with the prefix report below (`scripts/vods-orphans-report.js` without `--storage`).## Scheduled verification@@@ -509,6 +545,7 @@ requeued (or failed when out of attempts). Handlers checkpoint progress and resu| `object.remux` | heavy | Stream-copy remux of the whole source (seek index, duration, MP4 faststart) into one new private object; also the source's `remux` variant || `object.hash` | light | sha256 of ready objects' present local copies that have no hash yet, onto the object (`content_hash`, `metadata.hash_basis`) and the local location (`checksum`). Tenant-wide batch (`params { limit (1-500, 50), budget_mb }`) or one object. Scheduled per tenant every `MEDIA_HASH_INTERVAL_MIN`. Result `{ hashed, bytes, skipped, remaining }`. See [Scheduled verification](#scheduled-verification) || `vod.duration.reconcile` | heavy | One bounded batch of the tenant's finished VODs: stored duration vs a measurement of the real file, local or the B2/R2 copy (ffprobe over a presigned URL, ranged reads). `params { limit (1-200, 25), after_id?, apply (false), confirm_remote (false) }`; without `after_id` it walks the library from where the last run stopped. Repairs only confirmed, clearly wrong values when `apply`. Result `{ counts, range, next_after_id, report }`. See [Duration reconciliation](#duration-reconciliation) |++| `storage.orphans.scan` | light | The [storage orphan report](#storage-orphan-report), service-wide, under the system app `_media`; report only. `params { limit (1-100000, 2000) }`. Result `{ totals, providers, summary, report }` (the report file under `<data>/reports/`). Scheduled every `MEDIA_ORPHAN_SCAN_DAYS`; tenants cannot enqueue it (403) || `vod.finalize` | finalize | Finalizes a recording whose finalize failed or never ran. `params { vod_id }`. Queued by the orphan sweep (a row still `is_recording = 1` that no recorder, chunk upload or finalize holds and whose file has been idle for `MEDIA_FINALIZE_ORPHAN_GRACE_S`, 180 s; at boot and every `MEDIA_FINALIZE_SWEEP_S`, 120 s) and by finalize itself when it could not settle a recording (nothing measurable, a stat failure, a throw; first run after 5 min). Still unmeasurable: retried (5 min, 15 min, 45 min, 2 h 15, 6 h; 6 attempts), then the VOD stays `needs_review`, hidden. Never stops a live recording. Result `{ outcome: ready \| needs_review \| corrupt \| deleted \| skipped, duration_seconds, duration_source }` |Split and remux refuse a source that is not ready, a tenant quota the output would exceed, and too@@@ -634,6 +671,7 @@ Not run against production yet: it needs the owner's go-ahead.| `MEDIA_FINALIZE_ORPHAN_GRACE_S` / `MEDIA_FINALIZE_SWEEP_S` | 180 / 120 | an orphaned recording's file must be idle this long; the orphan sweep runs this often || `MEDIA_HASH_INTERVAL_MIN` | 15 | queue `object.hash` per tenant this often (`0` = only on demand) || `MEDIA_HASH_BUDGET_MB` / `_MAX_FILE_MB` / `_SETTLE_S` | 4096 / 20480 / 120 | bytes read per run; largest file hashed; a file changed this recently waits |++| `MEDIA_ORPHAN_SCAN_DAYS` | 30 | queue the storage orphan report this often (`0` = only on demand) || `MEDIA_DURATION_RECONCILE_HOURS` | 0 | schedule `vod.duration.reconcile` per tenant this often (`0` = only on demand) || `MEDIA_DURATION_RECONCILE_APPLY` / `_BATCH` | off / 25 | scheduled runs repair (`1`) or only report; VODs per scheduled run || `MEDIA_JOBS_HEAVY_WHILE_RECORDING` | off | `1` lets split/remux run while a recording is being written |@@@ -657,6 +695,5 @@ Not run against production yet: it needs the owner's go-ahead.- Copy/move aliases, lifecycle rules and an S3-compatible façade.- Clip cutting and the finalize remux still run inline, not as jobs (thumbnail regeneration was the firstoperation moved onto the job system).--- Orphan-bucket-object detection.- The segment-native timeline.- App assets (emotes and sounds, the `assets` table) are not projected yet. The `asset` kind exists for v2 uploads.@@@ -1,15 +1,23 @@#!/usr/bin/env node'use strict';/**-- * Report on the B2 `vods-orphans/` prefix (hazard H3). Read-only: it lists the prefix (ListObjectsV2),-- * HEADs each match's canonical copy, reads the database, and writes a report. It never deletes, moves,-- * copies or writes an object or a row (server/vod/orphans-report.js has the rules).++ * Orphan reports (server/vod/orphans-report.js has the rules). Read-only: they list, HEAD and read the++ * database, and write a report file. They never delete, move, copy or write an object, a file or a row.** node scripts/vods-orphans-report.js [--prefix vods-orphans/] [--provider b2] [--out <report.json>] [--json] [--db <media.db>]*-- * Prints one line per object (key, size, matched VOD/clip, recommendation) and the totals per-- * recommendation, and writes the full report (default <data>/reports/vods-orphans-<time>.json).-- * B2 credentials come from the environment (/etc/openvibe/media.env names). Exit 0 done, 1 error, 2 usage.++ * The B2 `vods-orphans/` prefix (hazard H3): lists the prefix (ListObjectsV2), HEADs each match's++ * canonical copy, prints one line per object (key, size, matched VOD/clip, recommendation) and the++ * totals per recommendation, and writes the full report (default <data>/reports/vods-orphans-<time>.json).++ *++ * node scripts/vods-orphans-report.js --storage [--no-remote] [--limit 2000] [--out <report.json>] [--json] [--db <media.db>]++ *++ * The whole storage against the database (the storage.orphans.scan job, on demand): files under the data++ * directories and B2/R2 keys that no row names, copies the database records that are not there, and++ * multipart uploads left open (Media's sessions and the buckets'). --no-remote lists no bucket. Prints++ * the totals and the largest items, and writes <data>/reports/storage-orphans-<time>.json.++ *++ * B2/R2 credentials come from the environment (/etc/openvibe/media.env names). Exit 0 done, 1 error, 2 usage.*/const fs = require('fs');const path = require('path');@@@ -18,7 +26,8 @@ const args = process.argv.slice(2);const has = (name) => args.includes(`--${name}`);const arg = (name) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] && !args[i + 1].startsWith('--') ? args[i + 1] : null; };const usage = (msg) => { console.error(`vods-orphans-report: ${msg}`); process.exit(2); };--for (const flag of ['prefix', 'provider', 'out', 'db']) if (has(flag) && !arg(flag)) usage(`--${flag} needs a value`);++for (const flag of ['prefix', 'provider', 'out', 'db', 'limit']) if (has(flag) && !arg(flag)) usage(`--${flag} needs a value`);++if (has('limit') && !(Number.isInteger(Number(arg('limit'))) && Number(arg('limit')) > 0)) usage('--limit must be a positive integer');if (arg('db')) process.env.DB_PATH = path.resolve(arg('db'));const provider = arg('provider') || 'b2';if (!['b2', 'r2'].includes(provider)) usage('--provider must be b2 or r2');@@@ -32,7 +41,32 @@ const JSON_OUT = has('json');const log = (...a) => { if (!JSON_OUT) console.log(...a); };const mb = (b) => `${(b / 1048576).toFixed(1)} MB`;++async function storageReport() {++ const providers = has('no-remote') ? [] : null;++ const report = await orphans.buildStorageReport({ providers, limit: has('limit') ? Number(arg('limit')) : 2000 });++ const file = orphans.writeStorageReport(report, arg('out') ? path.resolve(arg('out')) : null);++ const t = report.totals;++ log(`Storage orphan report (read-only), ${report.generated_at}`);++ for (const r of report.scope.roots) log(` local ${r.name.padEnd(11)} ${r.dir}${r.exists ? ` (${r.files} file(s))` : ' (missing)'}`);++ for (const [p, e] of Object.entries(report.scope.providers)) log(` ${p.padEnd(17)} ${e.listed ? `${e.bucket || ''}: ${e.keys} key(s), ${mb(e.bytes)}` : `not listed${e.error ? `: ${e.error}` : ''}`}`);++ if (has('no-remote')) log(' buckets not listed (--no-remote)');++ log('');++ log(`Unreferenced local files: ${t.unreferenced_local.files} (${mb(t.unreferenced_local.bytes)})`);++ for (const f of report.unreferenced_local.slice(0, 25)) log(` ${mb(f.size).padStart(11)} ${f.root}/${f.path}${f.hint ? ` — ${f.hint}` : ''}`);++ for (const [p, x] of Object.entries(t.unreferenced_remote)) log(`Unreferenced ${p} keys: ${x.keys} (${mb(x.bytes)})`);++ for (const k of report.unreferenced_remote.slice(0, 25)) log(` ${mb(k.size).padStart(11)} ${k.provider}:${k.key}${k.hint ? ` — ${k.hint}` : ''}`);++ log(`Copies the database records that are not there: ${t.missing.locations}`);++ for (const m of report.missing.slice(0, 25)) log(` ${m.object_id} ${m.legacy_ref || m.kind} ${m.provider} (recorded ${m.recorded_state})`);++ log(`Open multipart uploads: ${t.multipart_local.open} here (${t.multipart_local.expired} expired)${Object.entries(t.multipart_remote).map(([p, n]) => `, ${n} in ${p}`).join('')}`);++ log(`Expected, not listed: ${t.expected.live_thumbnails} live thumbnail(s), ${t.expected.in_flight} upload(s) in flight`);++ log('');++ log('Nothing was deleted or moved.');++ log(`Report: ${file}`);++ if (JSON_OUT) console.log(JSON.stringify({ ...report, report: file }, null, 2));++}++(async () => {++ if (has('storage')) { await storageReport(); db.close(); process.exit(0); }if (!vodStorage.providerConfigured(provider)) usage(`${provider} is not configured (MEDIA_${provider.toUpperCase()}_* not in the environment)`);const report = await orphans.buildReport({ provider, prefix: arg('prefix') || orphans.PREFIX });const file = arg('out') ? path.resolve(arg('out')) : path.join(path.dirname(config.db.path), 'reports', `vods-orphans-${report.generated_at.replace(/[:.]/g, '-')}.json`);@@@ -19,7 +19,7 @@* Watch pages and the media index render from the database only and still answer.* - server/index.js opens the database and serves HTTP, and starts nothing else: no app seeding, no* JWKS refresh, no health job, junk sweep, tiering sweep, clip re-cuts, copy verification, jobs-- * worker (thumbnails, split/remux, invariant scans), disk guardian, thumbnail cleanup, object purge,++ * worker (thumbnails, split/remux, invariant scans, storage orphan scans), disk guardian, thumbnail cleanup, object purge,* backfill, orphan-recording finalize or Events relay; webhooks are never sent. Timers that modules* start when they are loaded check `enabled` as well. Readiness does not probe storage (a drill* writes none) or the remote tiers.@@@ -126,6 +126,8 @@ function record(webhookEvent, appId, data) {function recordJob(transition, job) {const priority = JOB_TRANSITIONS[transition];if (!priority) throw new Error(`unknown job transition ${transition}`);++ // A service-wide maintenance job (queue.SYSTEM_APP) is no tenant's: nothing to announce.++ if (job.app_id === require('./jobs/queue').SYSTEM_APP) return null;return enqueue(`media.job.${transition}`, job.app_id, { type: 'job', id: String(job.id) }, job, priority);}@@@ -28,6 +28,9 @@ const STATUSES = ['proposed', 'queued', 'running', 'succeeded', 'failed', 'canceconst FINISHED = ['succeeded', 'failed', 'cancelled'];const ACTIVE = ['queued', 'running'];const MAX_ACTIVE_PER_TENANT = 50; // queued + running jobs one tenant may have (proposals excluded)++// Service-wide maintenance jobs (storage.orphans.scan) belong to no tenant: they run under this app id,++// which has no apps row, so no API credential reaches them, and they announce no events.++const SYSTEM_APP = '_media';const bus = new EventEmitter();bus.setMaxListeners(0);@@@ -394,7 +397,7 @@ function prune({ days = 30, types = ['thumbnail.regenerate', 'object.hash'] } =}module.exports = {-- STATUSES, FINISHED, ACTIVE, MAX_ACTIVE_PER_TENANT, JobError, bus,++ STATUSES, FINISHED, ACTIVE, MAX_ACTIVE_PER_TENANT, SYSTEM_APP, JobError, bus,register, typeSpec, typeNames, requestHash, parseJson,get, getForApp, jobPublic, jobEvent, list, counts,enqueue, approve, cancel,nnew file mode 100644@@@ -0,0 +1,73 @@++/**++ * OpenVibe.Media — job type storage.orphans.scan (light lane): the storage orphan report++ * (server/vod/orphans-report.js buildStorageReport) across the whole service. Report only: it walks the++ * data directories, lists the B2/R2 buckets (keys and open multipart uploads), reads the database and++ * writes <data>/reports/storage-orphans-<time>.json. It never deletes, moves or changes a file, a bucket++ * object or a row.++ *++ * params { limit (1-100000, default 2000): items listed per section; the totals count everything }++ * result { totals, providers, summary, report } report = the file name under <data>/reports/++ *++ * Storage is shared by every tenant, so the job belongs to none: it runs under queue.SYSTEM_APP, which no++ * API credential reaches (no apps row), announces no media.job.* event, and cannot be enqueued through++ * /api/v2/:app/jobs (validate refuses every tenant). Scheduled every MEDIA_ORPHAN_SCAN_DAYS (30; 0 = never)++ * by the worker, which does not start in a restore drill. On demand: scripts/vods-orphans-report.js --storage.++ */++'use strict';++++const path = require('path');++const queue = require('./queue');++const orphans = require('../vod/orphans-report');++++const { JobError } = queue;++const TYPE = 'storage.orphans.scan';++++function envInt(name, fallback) {++ const v = parseInt(process.env[name] || '', 10);++ return Number.isFinite(v) ? v : fallback;++}++++function checkParams(params) {++ const p = params || {};++ const limit = p.limit == null ? 2000 : Number(p.limit);++ if (!Number.isInteger(limit) || limit < 1 || limit > 100000) throw new JobError('media.job.invalid', 'params.limit must be 1-100000', { permanent: true });++ return { limit };++}++++function validate({ appId, obj, params }) {++ if (appId !== queue.SYSTEM_APP) {++ throw new JobError('media.job.forbidden', `${TYPE} covers the whole service's storage: it runs on its schedule or from scripts/vods-orphans-report.js --storage`, { status: 403, permanent: true });++ }++ if (obj) throw new JobError('media.job.invalid', `${TYPE} covers the whole storage: leave object_id out`, { permanent: true });++ return checkParams(params);++}++++async function run(job) {++ const p = checkParams(job.params);++ const report = await orphans.buildStorageReport({ limit: p.limit });++ report.job_id = job.id;++ const file = orphans.writeStorageReport(report);++ const summary = orphans.summarizeStorage(report);++ console.log(`[Orphans] ${summary} (report ${path.basename(file)}; nothing was deleted)`);++ return { totals: report.totals, providers: report.scope.providers, summary, report: path.basename(file) };++}++++/** Queue one scan every MEDIA_ORPHAN_SCAN_DAYS (0 = never). Returns how many were queued (0 or 1). */++function schedule(nowMs = Date.now()) {++ const days = envInt('MEDIA_ORPHAN_SCAN_DAYS', 30);++ if (!(days > 0)) return 0;++ const period = Math.floor(nowMs / (days * 86400 * 1000));++ try {++ const r = queue.enqueue({ appId: queue.SYSTEM_APP, type: TYPE, params: {}, idempotencyKey: `${TYPE}:${days}d:${period}`, createdBy: 'system:schedule' });++ return r.created ? 1 : 0;++ } catch (err) {++ console.warn(`[Jobs] could not schedule ${TYPE}: ${err.message}`);++ return 0;++ }++}++++module.exports = {++ TYPE,++ spec: { lane: 'light', maxAttempts: 2, timeoutMs: 2 * 60 * 60 * 1000, needsObject: false, validate, run },++ schedule,++};@@@ -8,6 +8,7 @@* vod.finalize finalize finalize a recording whose finalize failed or never ran (orphans), with backoff* vod.duration.reconcile heavy stored VOD durations vs a measurement of the real file (local or B2/R2), one batch* object.hash light sha256 of local copies with no content hash yet, one bounded batch (or one object)++ * storage.orphans.scan light the storage orphan report, service-wide (report only; never deletes)*/'use strict';@@@ -20,5 +21,6 @@ queue.register('object.remux', require('./derive').remux);queue.register('vod.finalize', require('./vod-finalize').spec);queue.register('vod.duration.reconcile', require('./duration-reconcile').spec);queue.register('object.hash', require('./content-hash').spec);++queue.register('storage.orphans.scan', require('./storage-orphans').spec);module.exports = { names: () => queue.typeNames() };@@@ -16,7 +16,8 @@** Scheduling: the size-invariant validator (invariant.scan) is enqueued per tenant every* MEDIA_INVARIANT_SCAN_HOURS (it proposes split/remux jobs; it never runs them). Orphaned recordings-- * get a vod.finalize every MEDIA_FINALIZE_SWEEP_S (server/jobs/vod-finalize.js).++ * get a vod.finalize every MEDIA_FINALIZE_SWEEP_S (server/jobs/vod-finalize.js). The storage orphan++ * report (storage.orphans.scan, report only) runs every MEDIA_ORPHAN_SCAN_DAYS under queue.SYSTEM_APP.*/'use strict';@@@ -149,6 +150,7 @@ async function tick() {scheduleInvariantScans(now);try { require('./duration-reconcile').schedule(now); } catch (err) { console.warn('[Jobs] duration reconcile schedule:', err.message); }try { require('./content-hash').schedule(now); } catch (err) { console.warn('[Jobs] content hash schedule:', err.message); }++ try { require('./storage-orphans').schedule(now); } catch (err) { console.warn('[Jobs] storage orphan scan schedule:', err.message); }}try { require('./vod-finalize').sweepOrphans({ now }); } catch (err) { console.warn('[Jobs] orphan sweep:', err.message); }if (now - lastPruneAt > 6 * 3600 * 1000) {@@@ -1,6 +1,14 @@/**-- * OpenVibe.Media — report on the B2 `vods-orphans/` prefix (hazard H3: 55 objects, 14.5 GB on-- * 2026-09-21). Report only: it lists, matches and recommends; it never deletes, moves or writes.++ * OpenVibe.Media — orphan reports. Report only: they list, match and recommend; they never delete,++ * move or write a file, a bucket object or a row.++ *++ * buildReport() the B2 `vods-orphans/` prefix (hazard H3: 55 objects, 14.5 GB on 2026-09-21)++ * buildStorageReport() the whole storage against the database (roadmap WS-G task 8): files and keys++ * no row names, copies the database records that are not there, and multipart++ * uploads left open (Media's own and the buckets'). The storage.orphans.scan job++ * runs it monthly; scripts/vods-orphans-report.js --storage on demand.++ *++ * ── The vods-orphans/ prefix ──** Each object under the prefix is matched to a VOD or clip by its file name (the basename of* vods.file_path / storage_key, clips.file_path; else the vod-<app>-<id>-<ms>.<ext> name the@@@ -22,6 +30,7 @@const fs = require('fs');const path = require('path');++const config = require('../config');const db = require('../db/database');const PREFIX = 'vods-orphans/';@@@ -136,4 +145,321 @@ async function buildReport({ provider = 'b2', prefix = PREFIX, list = null } = {};}--module.exports = { PREFIX, RECOMMENDATIONS, parseRecordingName, assess, buildReport };++// ═══════════════════════════════════════════════════════════════++// The whole storage against the database (WS-G task 8)++// ═══════════════════════════════════════════════════════════════++//++// unreferenced_local files under the data directories that no row names (media_locations, vods,++// clips, files, pastes, assets, open upload sessions, unfinished jobs). A file whose++// only row is a deleted object's location counts (its bytes outlived the object),++// unless the object is a native one still inside its retention period, or held.++// unreferenced_remote B2/R2 keys that no media_locations row or vods/clips row names++// missing copies the database records for objects that are not deleted, whose bytes are not++// there: no local file, or a key absent from its bucket's listing++// multipart_local Media's own upload sessions still open (media_uploads active/completing), expired++// ones flagged; part directories no open session owns are unreferenced_local++// multipart_remote multipart uploads the buckets still hold open (ListMultipartUploads)++//++// Expected files that no row names are counted, not listed: live thumbnails (stream-<app>-<id>.jpg,++// refreshed in place while a stream is live) and upload temps / partial downloads younger than an hour.++// A provider that is not configured, or whose listing fails, is reported as not listed; nothing is++// concluded about its keys.++++const STORAGE_SECTIONS = ['unreferenced_local', 'unreferenced_remote', 'missing', 'multipart_local', 'multipart_remote'];++const IN_FLIGHT_MS = 60 * 60 * 1000;++const MAX_WALK = 1000000;++++function localRoots() {++ const seen = new Set();++ return [++ ['vods', config.vod.path], ['clips', config.vod.clipsPath], ['thumbnails', config.thumbnails.path], ['files', config.files.path],++ ['pastes', config.pastes.path], ['objects', config.objects.path], ['assets', config.assets.path],++ ].map(([name, dir]) => ({ name, dir: path.resolve(dir) })).filter(r => (seen.has(r.dir) ? false : seen.add(r.dir)));++}++++/** Every regular file under one root. Symlinks are not followed; a root nested inside it is walked on its own. */++function walkRoot(root, rootDirs, out) {++ const stack = [root.dir];++ let seen = 0;++ while (stack.length) {++ const d = stack.pop();++ let entries;++ try { entries = fs.readdirSync(d, { withFileTypes: true }); } catch { continue; }++ for (const e of entries) {++ const p = path.join(d, e.name);++ if (e.isDirectory()) { if (!rootDirs.has(p)) stack.push(p); continue; }++ if (!e.isFile()) continue;++ if (out.files.length >= MAX_WALK) { out.truncated = true; return seen; }++ let st;++ try { st = fs.statSync(p); } catch { continue; }++ out.files.push({ root: root.name, abs: p, path: path.relative(root.dir, p), size: st.size, mtimeMs: st.mtimeMs });++ seen++;++ }++ }++ return seen;++}++++/** A location keeps its bytes wanted: its object exists and is not deleted, or is a native object inside its retention, or is held. */++function locationWanted(l) {++ if (!l.object_id) return false;++ return l.lifecycle_status !== 'deleted' || (!l.legacy_ref && !l.purged) || !!l.held;++}++++const LOCATION_ROWS = `SELECT l.provider, l.bucket, l.key, o.id AS object_id, o.lifecycle_status, o.legacy_ref,++ CASE WHEN json_valid(o.metadata) THEN json_extract(o.metadata, '$.purged_at') END AS purged,++ EXISTS (SELECT 1 FROM media_holds h WHERE h.object_id = o.id AND h.released_at IS NULL) AS held++ FROM media_locations l LEFT JOIN media_objects o ON o.id = l.object_id`;++++const sidecarsOf = (p) => [p.replace(/\.webm$/, '.seekable.webm'), p.replace(/\.mp4$/, '.seekable.mp4'), p.replace(/\.webm$/, '.master.mkv')].filter(x => x !== p);++++/** What the database names on local disk: { wanted: Set(abs path), deleted: Map(abs path → location), segmentBases, sessions, jobs }. */++function localReferences() {++ const vs = vodStorage();++ const model = require('../objects/model');++ const wanted = new Set();++ const add = (p) => { if (p) wanted.add(path.resolve(String(p))); };++ const deleted = new Map();++ for (const l of db.all(`${LOCATION_ROWS} WHERE l.provider = 'local'`)) {++ if (locationWanted(l)) add(l.key); else deleted.set(path.resolve(String(l.key)), l);++ }++ const thumb = (u) => { const name = model.thumbFileFromUrl(u); if (name) add(path.join(config.thumbnails.path, name)); };++ const segmentBases = new Set(); // browser-chunk segments <base>.seg-<n>-<ms>.webm belong to <base>.webm++ for (const v of db.all('SELECT id, file_path, master_file_path, thumbnail_url FROM vods')) {++ if (v.file_path) {++ for (const f of [v.file_path, vs.localPathForVod(v)]) {++ add(f);++ for (const x of sidecarsOf(path.resolve(f))) add(x);++ segmentBases.add(path.join(path.dirname(path.resolve(f)), path.basename(f).replace(/\.webm$/, '')));++ }++ }++ add(v.master_file_path);++ thumb(v.thumbnail_url);++ }++ for (const c of db.all('SELECT file_path, thumbnail_url FROM clips')) {++ if (c.file_path) { add(c.file_path); add(path.join(config.vod.clipsPath, path.basename(c.file_path))); }++ thumb(c.thumbnail_url);++ }++ for (const f of db.all('SELECT key, app_id FROM files')) add(path.join(config.files.path, f.app_id, f.key));++ for (const p of db.all('SELECT screenshot_path FROM pastes WHERE screenshot_path IS NOT NULL')) add(p.screenshot_path);++ for (const a of db.all('SELECT file_path FROM assets WHERE file_path IS NOT NULL')) add(a.file_path);++ const sessions = new Set(db.all("SELECT id FROM media_uploads WHERE status IN ('active', 'completing')").map(r => r.id));++ const jobs = new Set(db.all("SELECT id FROM media_jobs WHERE status IN ('queued', 'running')").map(r => r.id));++ return { wanted, deleted, segmentBases, sessions, jobs };++}++++/** Is this local file named by the database (or one of the expected kinds)? 'wanted' | 'live_thumbnail' | 'in_flight' | null. */++function classifyLocal(f, refs, now) {++ if (refs.wanted.has(f.abs)) return 'wanted';++ const base = path.basename(f.abs);++ const seg = /^(.+)\.seg-\d+-\d+\.webm$/.exec(base);++ if (seg && refs.segmentBases.has(path.join(path.dirname(f.abs), seg[1]))) return 'wanted';++ const [top, sub] = f.path.split(path.sep);++ if (f.root === 'objects' && top === '.parts' && refs.sessions.has(sub)) return 'wanted';++ if (f.root === 'objects' && top === '.jobs' && refs.jobs.has(sub)) return 'wanted';++ if (f.root === 'thumbnails' && /^stream-.+\.jpg$/.test(base)) return 'live_thumbnail';++ const transient = (f.root === 'objects' && top === '.tmp') || base.endsWith('.download') || (f.root === 'objects' && top === '.parts');++ if (transient && now - f.mtimeMs < IN_FLIGHT_MS) return 'in_flight';++ return null;++}++++/** A recorder-style name (vod-<app>-<id>-<ms>.<ext>): whose recording, and does its row still exist? */++function recordingNameHint(base) {++ const parsed = parseRecordingName(base);++ if (!parsed) return null;++ const row = db.get('SELECT id, file_path FROM vods WHERE id = ? AND app_id = ?', [parsed.id, parsed.app]);++ if (!row) return `a recording of vod ${parsed.id} (${parsed.app}), whose row is gone`;++ return `named for vod ${parsed.id} (${parsed.app}), whose row points at ${path.basename(row.file_path || '') || 'no file'}`;++}++++function localHint(f, refs) {++ const base = path.basename(f.abs);++ const del = refs.deleted.get(f.abs);++ if (del) return del.object_id ? `bytes of deleted object ${del.object_id}${del.legacy_ref ? ` (${del.legacy_ref})` : ''}` : 'named only by a location row whose object does not exist';++ if (f.root === 'objects') {++ const [top, sub] = f.path.split(path.sep);++ if (top === '.parts') {++ const s = db.get('SELECT status FROM media_uploads WHERE id = ?', [sub]);++ return s ? `parts of multipart upload ${sub} (${s.status})` : `parts of multipart upload ${sub}, which has no session`;++ }++ if (top === '.jobs') {++ const j = db.get('SELECT status FROM media_jobs WHERE id = ?', [sub]);++ return j ? `work files of job ${sub} (${j.status})` : `work files of job ${sub}, which does not exist`;++ }++ if (top === '.tmp') return 'an upload that never finished (temp file)';++ const o = db.get('SELECT id, lifecycle_status FROM media_objects WHERE id = ?', [base]);++ return o ? `named for object ${o.id} (${o.lifecycle_status}), which records no local copy here` : `named for object ${base}, which does not exist`;++ }++ if (base.endsWith('.download')) return 'a restore download that never finished';++ if (/\.seekable\.(webm|mp4)$/.test(base)) return 'a DVR sidecar no VOD row names';++ if (/\.master\.mkv$/.test(base)) return recordingNameHint(base.replace(/\.master\.mkv$/, '.webm')) || 'a master archive no VOD row names';++ return recordingNameHint(base);++}++++/** What the database names in one provider's bucket: { wanted: Set(key), deleted: Map(key → location), rows: Map(key → row) }. */++function remoteReferences(provider) {++ const vs = vodStorage();++ const bucket = vs.bucketFor(provider);++ const wanted = new Set();++ const deleted = new Map();++ for (const l of db.all(`${LOCATION_ROWS} WHERE l.provider = ?`, [provider])) {++ if (l.bucket && bucket && l.bucket !== bucket) continue; // a copy recorded in another bucket++ if (locationWanted(l)) wanted.add(l.key); else deleted.set(l.key, l);++ }++ const rows = new Map(); // every key a vods/clips row points at, with the row (stale R2 copies are recognised by it)++ for (const table of ['vods', 'clips']) {++ for (const r of db.all(`SELECT id, file_path, storage_provider, storage_key FROM ${table} WHERE COALESCE(file_path, '') != ''`)) {++ const p = vs.providerOf(r);++ if (p === 'local' && !r.storage_key) continue;++ const key = vs.keyForVod(r);++ rows.set(key, { kind: table === 'vods' ? 'vod' : 'clip', id: r.id, provider: p });++ if (provider === 'b2' || p === 'r2') wanted.add(key);++ }++ }++ return { wanted, deleted, rows };++}++++function remoteHint(provider, key, refs) {++ if (key.startsWith(PREFIX)) return `parked in ${PREFIX}: scripts/vods-orphans-report.js says what each one is`;++ const del = refs.deleted.get(key);++ if (del) return del.object_id ? `copy of deleted object ${del.object_id}${del.legacy_ref ? ` (${del.legacy_ref})` : ''}` : 'named only by a location row whose object does not exist';++ const row = refs.rows.get(key);++ if (row && provider === 'r2') return `an R2 copy of ${row.kind} ${row.id}, which is served from ${row.provider}`;++ const shot = /paste-screenshot-(\d+)\.[a-z0-9]+$/i.exec(key);++ if (shot) return db.get('SELECT 1 AS x FROM pastes WHERE id = ?', [Number(shot[1])]) ? `a legacy screenshot of paste ${shot[1]}` : `a legacy screenshot of paste ${shot[1]}, whose row is gone`;++ return recordingNameHint(path.basename(key));++}++++/**++ * The storage report. Options (all injectable for tests):++ * providers which buckets to list (default: the configured ones of b2, r2)++ * list (provider) → [{ key, size, last_modified }] default: every key (ListObjectsV2)++ * listUploads (provider) → [{ key, upload_id, initiated }] default: ListMultipartUploads++ * limit items listed per section (totals always count everything), default 2000++ */++async function buildStorageReport({ providers = null, list = null, listUploads = null, limit = 2000, now = Date.now() } = {}) {++ const vs = vodStorage();++ const provs = providers || ['b2', 'r2'].filter(p => vs.providerConfigured(p));++ const lister = list || ((p) => vs.listObjects(p, ''));++ const uploadLister = listUploads || ((p) => vs.listMultipartUploads(p));++ const cap = Math.max(1, Number(limit) || 2000);++ const report = {++ kind: 'media.storage_orphans.report', version: 1, generated_at: new Date(now).toISOString(), read_only: true,++ scope: { roots: [], providers: {}, walk_truncated: false },++ totals: {++ unreferenced_local: { files: 0, bytes: 0, by_root: {} }, unreferenced_remote: {},++ missing: { locations: 0, by_provider: {} }, multipart_local: { open: 0, expired: 0 }, multipart_remote: {},++ expected: { live_thumbnails: 0, in_flight: 0 },++ },++ truncated: {},++ note: 'Report only: nothing was deleted, moved or written. Each item is for an operator to look at.',++ };++ for (const k of STORAGE_SECTIONS) { report[k] = []; report.truncated[k] = false; }++ const push = (section, item) => { if (report[section].length < cap) report[section].push(item); else report.truncated[section] = true; };++++ // ── Local disk ──++ const roots = localRoots();++ const rootDirs = new Set(roots.map(r => r.dir));++ const walked = { files: [], truncated: false };++ for (const root of roots) {++ const exists = fs.existsSync(root.dir);++ const files = exists ? walkRoot(root, rootDirs, walked) : 0;++ report.scope.roots.push({ name: root.name, dir: root.dir, exists, files });++ }++ report.scope.walk_truncated = walked.truncated;++ const lrefs = localReferences();++ const orphanLocal = [];++ for (const f of walked.files) {++ const c = classifyLocal(f, lrefs, now);++ if (c === 'wanted') continue;++ if (c === 'live_thumbnail') { report.totals.expected.live_thumbnails++; continue; }++ if (c === 'in_flight') { report.totals.expected.in_flight++; continue; }++ orphanLocal.push(f);++ const t = report.totals.unreferenced_local;++ t.files++; t.bytes += f.size;++ t.by_root[f.root] = t.by_root[f.root] || { files: 0, bytes: 0 };++ t.by_root[f.root].files++; t.by_root[f.root].bytes += f.size;++ }++ orphanLocal.sort((a, b) => b.size - a.size);++ for (const f of orphanLocal.slice(0, cap)) push('unreferenced_local', { root: f.root, path: f.path, size: f.size, modified: new Date(f.mtimeMs).toISOString(), hint: localHint(f, lrefs) });++ if (orphanLocal.length > cap) report.truncated.unreferenced_local = true;++++ // ── Buckets ──++ const listed = new Map(); // provider → Map(key → size)++ for (const p of provs) {++ const entry = { configured: vs.providerConfigured(p), bucket: vs.bucketFor(p), listed: false };++ report.scope.providers[p] = entry;++ let keys;++ try { keys = await lister(p); } catch (err) { entry.error = err.message; continue; }++ entry.listed = true;++ entry.keys = keys.length;++ entry.bytes = keys.reduce((n, k) => n + (Number(k.size) || 0), 0);++ listed.set(p, new Map(keys.map(k => [k.key, k])));++ const rrefs = remoteReferences(p);++ const orphans = keys.filter(k => !rrefs.wanted.has(k.key)).sort((a, b) => (Number(b.size) || 0) - (Number(a.size) || 0));++ report.totals.unreferenced_remote[p] = { keys: orphans.length, bytes: orphans.reduce((n, k) => n + (Number(k.size) || 0), 0) };++ for (const k of orphans.slice(0, cap)) push('unreferenced_remote', { provider: p, key: k.key, size: Number(k.size) || 0, last_modified: k.last_modified || null, hint: remoteHint(p, k.key, rrefs) });++ if (orphans.length > cap) report.truncated.unreferenced_remote = true;++ let uploads = [];++ try { uploads = await uploadLister(p); } catch (err) { entry.multipart_error = err.message; }++ report.totals.multipart_remote[p] = uploads.length;++ for (const u of uploads) push('multipart_remote', { provider: p, key: u.key, upload_id: u.upload_id, initiated: u.initiated || null, key_named_by_a_row: rrefs.wanted.has(u.key) });++ }++++ // ── Copies the database records whose bytes are not there ──++ for (const l of db.all(`SELECT l.provider, l.bucket, l.key, l.state, l.verified_at, o.id AS object_id, o.app_id, o.kind, o.legacy_ref, o.lifecycle_status++ FROM media_locations l JOIN media_objects o ON o.id = l.object_id WHERE o.lifecycle_status != 'deleted' ORDER BY l.id`)) {++ let there;++ if (l.provider === 'local') {++ try { there = fs.statSync(l.key).isFile(); } catch { there = false; }++ } else {++ const keys = listed.get(l.provider);++ if (!keys) continue; // not listed: nothing is known++ if (l.bucket && vs.bucketFor(l.provider) && l.bucket !== vs.bucketFor(l.provider)) continue;++ there = keys.has(l.key);++ }++ if (there) continue;++ report.totals.missing.locations++;++ report.totals.missing.by_provider[l.provider] = (report.totals.missing.by_provider[l.provider] || 0) + 1;++ push('missing', { object_id: l.object_id, app_id: l.app_id, kind: l.kind, legacy_ref: l.legacy_ref || null, lifecycle_status: l.lifecycle_status,++ provider: l.provider, key: l.key, recorded_state: l.state, verified_at: l.verified_at || null });++ }++++ // ── Media's own multipart sessions still open ──++ for (const u of db.all(`SELECT u.*, (SELECT COUNT(*) FROM media_upload_parts p WHERE p.upload_id = u.id) AS parts_received,++ (SELECT COALESCE(SUM(p.size_bytes), 0) FROM media_upload_parts p WHERE p.upload_id = u.id) AS bytes_received,++ (u.expires_at < datetime('now')) AS expired++ FROM media_uploads u WHERE u.status IN ('active', 'completing') ORDER BY u.created_at`)) {++ report.totals.multipart_local.open++;++ if (u.expired) report.totals.multipart_local.expired++;++ push('multipart_local', { upload_id: u.id, object_id: u.object_id, app_id: u.app_id, status: u.status, parts_received: u.parts_received,++ parts_expected: u.parts_expected, bytes_received: u.bytes_received, total_size: u.total_size, created_at: u.created_at, expires_at: u.expires_at, expired: !!u.expired });++ }++ return report;++}++++function reportsDir() {++ return path.join(path.dirname(config.db.path), 'reports');++}++++/** Write a storage report as JSON (default <data>/reports/storage-orphans-<time>.json). Returns the path. */++function writeStorageReport(report, file = null) {++ const out = file || path.join(reportsDir(), `storage-orphans-${report.generated_at.replace(/[:.]/g, '-')}.json`);++ fs.mkdirSync(path.dirname(out), { recursive: true });++ fs.writeFileSync(out, JSON.stringify(report, null, 1));++ return out;++}++++/** One line per section for logs and the job result. */++function summarizeStorage(report) {++ const t = report.totals;++ const mb = (b) => `${(b / 1048576).toFixed(1)} MB`;++ const remote = Object.entries(t.unreferenced_remote).map(([p, x]) => `${p} ${x.keys} key(s) ${mb(x.bytes)}`).join(', ') || 'no bucket listed';++ return `unreferenced local ${t.unreferenced_local.files} file(s) ${mb(t.unreferenced_local.bytes)}; unreferenced remote: ${remote}; `++ + `missing copies ${t.missing.locations}; open multipart: ${t.multipart_local.open} here (${t.multipart_local.expired} expired), `++ + `${Object.entries(t.multipart_remote).map(([p, n]) => `${n} in ${p}`).join(', ') || 'buckets not listed'}`;++}++++module.exports = {++ PREFIX, RECOMMENDATIONS, parseRecordingName, assess, buildReport,++ STORAGE_SECTIONS, buildStorageReport, writeStorageReport, summarizeStorage, localRoots,++};@@@ -393,6 +393,27 @@ async function listObjects(provider, prefix) {return out;}++/**++ * Multipart uploads still open in one provider's bucket (started, never completed or aborted: a killed++ * process leaves them, and their parts are stored and billed): [{ key, upload_id, initiated }] (read-only).++ */++async function listMultipartUploads(provider, prefix = '') {++ const client = clientFor(provider);++ if (!client) throw new Error(`${provider} is not configured`);++ loadSdk();++ const out = [];++ let keyMarker, idMarker;++ for (;;) {++ const r = await client.send(new S3.ListMultipartUploadsCommand({ Bucket: PROVIDER_ENV[provider].bucket, Prefix: prefix || undefined,++ KeyMarker: keyMarker, UploadIdMarker: idMarker, MaxUploads: 1000 }));++ for (const u of (r.Uploads || [])) out.push({ key: u.Key, upload_id: u.UploadId, initiated: u.Initiated ? new Date(u.Initiated).toISOString() : null });++ if (!r.IsTruncated || !r.NextKeyMarker) break;++ keyMarker = r.NextKeyMarker;++ idMarker = r.NextUploadIdMarker;++ }++ return out;++}++async function presignGet(provider, key, expiresInSeconds = 900) {const client = clientFor(provider);if (!client) return null;@@@ -1378,6 +1399,7 @@ module.exports = {endpointFor,headObject,listObjects,++ listMultipartUploads,keyForVod,localPathForVod,resolvePlayback,nnew file mode 100644@@@ -0,0 +1,239 @@++'use strict';++// The storage orphan report (server/vod/orphans-report.js buildStorageReport; job storage.orphans.scan;++// scripts/vods-orphans-report.js --storage): files under the data directories and B2/R2 keys that no row++// names, copies the database records that are not there, and multipart uploads left open, here and in the++// buckets. Report only: the test snapshots every file, the tables and the buckets before and after and++// finds them unchanged, and a fake S3 (spoken to by the real AWS SDK) counts any write or delete. The job++// runs service-wide under queue.SYSTEM_APP, is scheduled every MEDIA_ORPHAN_SCAN_DAYS, and tenants cannot++// enqueue it.++const assert = require('assert');++const fs = require('fs');++const os = require('os');++const path = require('path');++const http = require('http');++const crypto = require('crypto');++const { execFileSync } = require('child_process');++++const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-orphans-'));++const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++++const buckets = { 'b2-bucket': new Map(), 'r2-bucket': new Map() };++const openUploads = { 'b2-bucket': [{ key: 'vods/half-uploaded.webm', id: 'up-b2-1' }], 'r2-bucket': [] };++let writes = 0;++const xml = (v) => String(v).replace(/&/g, '&').replace(/</g, '<');++const s3 = http.createServer((req, res) => {++ req.resume();++ req.on('end', () => {++ const u = new URL(req.url, 'http://x');++ const [, bucket, ...rest] = decodeURIComponent(u.pathname).split('/');++ const key = rest.join('/');++ const store = buckets[bucket];++ if (!store) { res.writeHead(404); return res.end(); }++ if (req.method !== 'GET' && req.method !== 'HEAD') { writes++; res.writeHead(403, { 'Content-Type': 'application/xml' }); return res.end('<Error><Code>AccessDenied</Code></Error>'); }++ if (!key && u.searchParams.has('uploads')) {++ res.writeHead(200, { 'Content-Type': 'application/xml' });++ return res.end(`<?xml version="1.0" encoding="UTF-8"?><ListMultipartUploadsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Bucket>${bucket}</Bucket><IsTruncated>false</IsTruncated>${++ openUploads[bucket].map(x => `<Upload><Key>${xml(x.key)}</Key><UploadId>${x.id}</UploadId><Initiated>2026-09-01T00:00:00.000Z</Initiated></Upload>`).join('')}</ListMultipartUploadsResult>`);++ }++ if (!key && u.searchParams.get('list-type') === '2') {++ const prefix = u.searchParams.get('prefix') || '';++ const keys = [...store.keys()].filter(k => k.startsWith(prefix)).sort();++ res.writeHead(200, { 'Content-Type': 'application/xml' });++ return res.end(`<?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>${bucket}</Name><Prefix>${xml(prefix)}</Prefix><KeyCount>${keys.length}</KeyCount><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated>${++ keys.map(k => `<Contents><Key>${xml(k)}</Key><LastModified>2026-09-21T00:00:00.000Z</LastModified><ETag>"e"</ETag><Size>${store.get(k).length}</Size><StorageClass>STANDARD</StorageClass></Contents>`).join('')}</ListBucketResult>`);++ }++ const obj = store.get(key);++ if (!obj) { res.writeHead(404); return res.end(); }++ res.writeHead(200, { 'Content-Length': obj.length, ETag: '"e"' });++ res.end(req.method === 'GET' ? obj : undefined);++ });++});++++(async () => {++ await new Promise((r) => s3.listen(0, '127.0.0.1', r));++ const endpoint = `http://127.0.0.1:${s3.address().port}`;++ const env = {++ DB_PATH: path.join(tmp, 'data', 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),++ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), ASSETS_PATH: dir('assets'),++ MEDIA_B2_ENDPOINT: endpoint, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_KEY_ID: 'k', MEDIA_B2_APP_KEY: 's', MEDIA_B2_REGION: 'us-west-004',++ MEDIA_R2_ENDPOINT: endpoint, MEDIA_R2_BUCKET: 'r2-bucket', MEDIA_R2_ACCESS_KEY_ID: 'k', MEDIA_R2_SECRET_ACCESS_KEY: 's',++ AWS_REQUEST_CHECKSUM_CALCULATION: 'WHEN_REQUIRED', AWS_RESPONSE_CHECKSUM_VALIDATION: 'WHEN_REQUIRED',++ MEDIA_PUBLIC_URL: 'https://media.test',++ };++ Object.assign(process.env, env);++ delete process.env.MEDIA_ORPHAN_SCAN_DAYS;++ const db = require('../server/db/database');++ const model = require('../server/objects/model');++ const queue = require('../server/jobs/queue');++ require('../server/jobs/types');++ const orphans = require('../server/vod/orphans-report');++ const storageJob = require('../server/jobs/storage-orphans');++ db.upsertApp({ app_id: 'live', api_key: 'live-key-orphans' });++ const raw = db.getDb();++ const write = (d, name, bytes = 16, ageMs = 0) => {++ const p = path.join(d, name);++ fs.mkdirSync(path.dirname(p), { recursive: true });++ fs.writeFileSync(p, Buffer.alloc(bytes, 7));++ if (ageMs) { const t = (Date.now() - ageMs) / 1000; fs.utimesSync(p, t, t); }++ return p;++ };++ const HOUR = 3600 * 1000;++++ // ── Fixtures: what the database names ──++ const ins = raw.prepare(`INSERT INTO vods (id, app_id, title, file_path, file_size, is_public, visibility, duration_seconds, storage_provider, storage_key, thumbnail_url)++ VALUES (?, 'live', ?, ?, ?, 1, 'public', 60, ?, ?, ?)`);++ ins.run(1, 'local', write(env.VOD_PATH, 'vod-live-1-1758000000001.webm', 40), 40, 'local', null, '/t/vod-1-111.jpg');++ write(env.THUMBNAILS_PATH, 'vod-1-111.jpg');++ ins.run(2, 'in B2', '/old/vod-live-2-1758000000002.webm', 20, 'b2', 'vods/vod-live-2-1758000000002.webm', null);++ buckets['b2-bucket'].set('vods/vod-live-2-1758000000002.webm', Buffer.alloc(20));++ ins.run(3, 'served from R2', '/old/vod-live-3-1758000000003.webm', 30, 'r2', 'vods/vod-live-3-1758000000003.webm', null);++ buckets['b2-bucket'].set('vods/vod-live-3-1758000000003.webm', Buffer.alloc(30));++ buckets['r2-bucket'].set('vods/vod-live-3-1758000000003.webm', Buffer.alloc(30));++ ins.run(4, 'back in B2, R2 copy left', '/old/vod-live-4-1758000000004.webm', 50, 'b2', 'vods/vod-live-4-1758000000004.webm', null);++ buckets['b2-bucket'].set('vods/vod-live-4-1758000000004.webm', Buffer.alloc(50));++ buckets['r2-bucket'].set('vods/vod-live-4-1758000000004.webm', Buffer.alloc(50));++ ins.run(5, 'local file lost', path.join(env.VOD_PATH, 'vod-live-5-1758000000005.webm'), 10, 'local', null, null);++ ins.run(6, 'B2 key lost', '/old/vod-live-6-1758000000006.webm', 60, 'b2', 'vods/vod-live-6-1758000000006.webm', null);++ for (const id of [1, 2, 3, 4, 5, 6]) model.sync('vod', id);++ raw.prepare(`INSERT INTO files (key, app_id, original_name, size, mime) VALUES ('abc123-notes.txt', 'live', 'notes.txt', 16, 'text/plain')`).run();++ write(path.join(env.FILES_PATH, 'live'), 'abc123-notes.txt');++ model.sync('file', 'abc123-notes.txt');++ // A native object soft-deleted inside its retention period keeps its bytes wanted.++ const kept = model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', visibility: 'private', size_bytes: 16 });++ const keptPath = write(path.join(env.OBJECTS_PATH, 'live'), kept, 16);++ model.upsertLocation(kept, { provider: 'local', key: keptPath, state: 'present', size_bytes: 16, verified: true });++ model.softDelete(model.getObject(kept));++ // A clip deleted by its row: the object is deleted, its file stayed behind.++ raw.prepare(`INSERT INTO clips (id, app_id, vod_id, title, file_path, status) VALUES (21, 'live', 1, 'gone clip', ?, 'ready')`).run(write(env.CLIPS_PATH, 'clip-live-21.webm', 12));++ model.sync('clip', 21);++ const clipObj = db.get('SELECT object_id FROM clips WHERE id = 21').object_id;++ raw.prepare('DELETE FROM clips WHERE id = 21').run();++ // Media's own multipart sessions: one open and expired (its parts wanted), one expired session's parts dir with no session.++ raw.prepare(`INSERT INTO media_uploads (id, object_id, app_id, part_size, total_size, parts_expected, status, expires_at)++ VALUES ('mup_OPEN', ?, 'live', 8, 16, 2, 'active', datetime('now', '-1 hour'))`).run(kept);++ raw.prepare("INSERT INTO media_upload_parts (upload_id, part_number, size_bytes, sha256) VALUES ('mup_OPEN', 1, 8, 'x')").run();++ write(path.join(env.OBJECTS_PATH, '.parts', 'mup_OPEN'), '1', 8, 3 * HOUR);++++ // ── What storage holds that no row names ──++ write(env.VOD_PATH, 'vod-live-99-1758000000099.webm', 1000); // the recorder's name; row 99 never existed++ write(env.VOD_PATH, 'mystery.bin', 500);++ write(env.VOD_PATH, 'vod-live-1-1758000000001.seekable.webm', 5); // a sidecar of vod 1: wanted++ write(env.THUMBNAILS_PATH, 'stream-live-1.jpg'); // a live thumbnail: expected++ write(path.join(env.OBJECTS_PATH, '.tmp'), 'med_OLD-abcd', 30, 3 * HOUR); // an upload that never finished++ write(path.join(env.OBJECTS_PATH, '.tmp'), 'med_NOW-abcd', 30); // one in flight right now++ write(path.join(env.OBJECTS_PATH, '.parts', 'mup_GONE'), '1', 8, 3 * HOUR); // parts with no session++ buckets['b2-bucket'].set('vods/stray.webm', Buffer.alloc(70));++ buckets['b2-bucket'].set('vods-orphans/vod-live-7-1758000000007.webm', Buffer.alloc(80));++++ // ── Everything before, to prove nothing changes ──++ const snapshotFiles = () => {++ const out = [];++ const walk = (d) => { for (const e of fs.readdirSync(d, { withFileTypes: true })) { const p = path.join(d, e.name); if (e.isDirectory()) { if (p !== path.join(tmp, 'data')) walk(p); } else out.push(`${path.relative(tmp, p)}:${fs.statSync(p).size}:${fs.statSync(p).mtimeMs}`); } };++ walk(tmp);++ return out.sort();++ };++ const TABLES = ['vods', 'clips', 'files', 'pastes', 'media_objects', 'media_locations', 'media_holds', 'media_uploads', 'media_upload_parts'];++ const snapshotDb = () => Object.fromEntries(TABLES.map(t => [t, crypto.createHash('sha256').update(JSON.stringify(raw.prepare(`SELECT * FROM ${t} ORDER BY rowid`).all())).digest('hex')]));++ const snapshotBuckets = () => JSON.stringify(Object.entries(buckets).map(([b, m]) => [b, [...m.keys()].sort()]));++ const before = { files: snapshotFiles(), db: snapshotDb(), buckets: snapshotBuckets() };++++ const report = await orphans.buildStorageReport({ limit: 50 });++++ // ── Unreferenced local files ──++ const local = Object.fromEntries(report.unreferenced_local.map(f => [`${f.root}/${f.path.split(path.sep).join('/')}`, f]));++ assert.deepStrictEqual(Object.keys(local).sort(), [++ 'clips/clip-live-21.webm', 'objects/.parts/mup_GONE/1', 'objects/.tmp/med_OLD-abcd', 'vods/mystery.bin', 'vods/vod-live-99-1758000000099.webm',++ ], 'exactly the files no row names');++ assert.ok(/whose row is gone/.test(local['vods/vod-live-99-1758000000099.webm'].hint), 'a recorder name says whose recording it was');++ assert.strictEqual(local['clips/clip-live-21.webm'].hint, `bytes of deleted object ${clipObj} (legacy:live:clip:21)`);++ assert.ok(/no session/.test(local['objects/.parts/mup_GONE/1'].hint));++ assert.ok(/never finished/.test(local['objects/.tmp/med_OLD-abcd'].hint));++ assert.strictEqual(local['vods/mystery.bin'].hint, null);++ assert.strictEqual(report.unreferenced_local[0].path, 'vod-live-99-1758000000099.webm', 'largest first');++ assert.deepStrictEqual([report.totals.unreferenced_local.files, report.totals.unreferenced_local.bytes], [5, 1000 + 500 + 30 + 8 + 12]);++ assert.deepStrictEqual(report.totals.expected, { live_thumbnails: 1, in_flight: 1 }, 'live thumbnails and uploads in flight are expected, not orphans');++ console.log('✅ unreferenced local files: recordings without a row, strays, bytes of deleted objects, stale temps and parts');++++ // ── Unreferenced bucket keys ──++ const remote = Object.fromEntries(report.unreferenced_remote.map(k => [`${k.provider}:${k.key}`, k]));++ assert.deepStrictEqual(Object.keys(remote).sort(), ['b2:vods-orphans/vod-live-7-1758000000007.webm', 'b2:vods/stray.webm', 'r2:vods/vod-live-4-1758000000004.webm']);++ assert.ok(/vods-orphans-report/.test(remote['b2:vods-orphans/vod-live-7-1758000000007.webm'].hint), 'parked recordings point at the prefix report');++ assert.strictEqual(remote['r2:vods/vod-live-4-1758000000004.webm'].hint, 'an R2 copy of vod 4, which is served from b2');++ assert.deepStrictEqual(report.totals.unreferenced_remote, { b2: { keys: 2, bytes: 150 }, r2: { keys: 1, bytes: 50 } });++ assert.deepStrictEqual([report.scope.providers.b2.listed, report.scope.providers.b2.keys, report.scope.providers.r2.keys], [true, 5, 2]);++ console.log('✅ unreferenced bucket keys: strays, parked recordings, an R2 copy left behind by a demotion');++++ // ── Copies the database records that are not there ──++ const missing = report.missing.map(m => `${m.legacy_ref}:${m.provider}`).sort();++ assert.deepStrictEqual(missing, ['legacy:live:vod:5:local', 'legacy:live:vod:6:b2'], 'a lost local file and a lost B2 key');++ assert.deepStrictEqual(report.totals.missing, { locations: 2, by_provider: { local: 1, b2: 1 } });++ console.log('✅ missing copies: rows whose location is not in storage');++++ // ── Multipart uploads left open ──++ assert.deepStrictEqual(report.multipart_local.map(u => [u.upload_id, u.expired, u.parts_received]), [['mup_OPEN', true, 1]]);++ assert.deepStrictEqual(report.totals.multipart_local, { open: 1, expired: 1 });++ assert.deepStrictEqual(report.multipart_remote.map(u => [u.provider, u.key, u.upload_id, u.key_named_by_a_row]), [['b2', 'vods/half-uploaded.webm', 'up-b2-1', false]]);++ assert.deepStrictEqual(report.totals.multipart_remote, { b2: 1, r2: 0 });++ console.log('✅ open multipart uploads: Media\'s expired sessions and the buckets\' own');++++ // ── Report only ──++ assert.deepStrictEqual(snapshotFiles(), before.files, 'no file was deleted, moved or touched');++ assert.deepStrictEqual(snapshotDb(), before.db, 'no row changed');++ assert.strictEqual(snapshotBuckets(), before.buckets, 'no bucket key changed');++ assert.strictEqual(writes, 0, 'no PUT, POST or DELETE reached a bucket');++ const src = fs.readFileSync(path.join(__dirname, '..', 'server', 'vod', 'orphans-report.js'), 'utf8');++ assert.ok(!/unlink|rmSync|rmdirSync|renameSync|DeleteObject|deleteObject|uploadFile|copyBetweenProviders|\b(DELETE|UPDATE|INSERT)\s/.test(src), 'the report module holds no delete, move or write');++ console.log('✅ report only: files, rows and buckets are unchanged; the module has no delete, move or write');++++ // ── The job: service-wide, scheduled, not for tenants ──++ assert.strictEqual(storageJob.schedule(Date.now()), 1, 'the monthly scan is queued');++ assert.strictEqual(storageJob.schedule(Date.now()), 0, 'once per period');++ process.env.MEDIA_ORPHAN_SCAN_DAYS = '0';++ assert.strictEqual(storageJob.schedule(Date.now() + 90 * 86400 * 1000), 0, 'MEDIA_ORPHAN_SCAN_DAYS=0 schedules nothing');++ delete process.env.MEDIA_ORPHAN_SCAN_DAYS;++ const job = db.get("SELECT * FROM media_jobs WHERE job_type = 'storage.orphans.scan'");++ assert.deepStrictEqual([job.app_id, job.status, job.created_by], [queue.SYSTEM_APP, 'queued', 'system:schedule']);++ const done = await require('../server/jobs/worker').runNow(job.id);++ assert.strictEqual(done.status, 'succeeded', done.error || '');++ const result = JSON.parse(done.result);++ assert.strictEqual(result.totals.unreferenced_local.files, 5);++ const reportFile = path.join(tmp, 'data', 'reports', result.report);++ assert.ok(fs.existsSync(reportFile), 'the job writes its report under <data>/reports');++ assert.strictEqual(JSON.parse(fs.readFileSync(reportFile, 'utf8')).job_id, job.id);++ assert.deepStrictEqual(snapshotFiles(), before.files, 'the job deleted nothing either');++ assert.strictEqual(writes, 0);++++ const express = require('express');++ const app = express();++ app.use(express.json());++ app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));++ const server = await new Promise((r) => { const s = app.listen(0, '127.0.0.1', () => r(s)); });++ const post = (p, body) => new Promise((resolve, reject) => {++ const data = JSON.stringify(body);++ const rq = http.request({ host: '127.0.0.1', port: server.address().port, path: p, method: 'POST',++ headers: { Authorization: 'Bearer live-key-orphans', 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(data) } },++ (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: JSON.parse(b || '{}') })); });++ rq.on('error', reject);++ rq.end(data);++ });++ const r = await post('/api/v2/live/jobs', { type: 'storage.orphans.scan' });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'media.job.forbidden'], 'a tenant cannot run the service-wide scan');++ server.close();++ console.log('✅ storage.orphans.scan: scheduled once per period under the system app, writes its report, refused to tenants');++++ // ── The script, on demand ──++ const out = path.join(tmp, 'script-report.json');++ const text = execFileSync(process.execPath, [path.join(__dirname, '..', 'scripts', 'vods-orphans-report.js'), '--storage', '--no-remote', '--out', out],++ { env: { ...process.env, ...env }, encoding: 'utf8' });++ assert.ok(/Unreferenced local files: 5/.test(text) && /Nothing was deleted or moved/.test(text), text);++ const scripted = JSON.parse(fs.readFileSync(out, 'utf8'));++ assert.deepStrictEqual([scripted.totals.unreferenced_local.files, Object.keys(scripted.scope.providers).length], [5, 0], '--no-remote lists no bucket');++ assert.deepStrictEqual(snapshotFiles().filter(f => !f.startsWith('script-report.json')), before.files);++ console.log('✅ scripts/vods-orphans-report.js --storage runs the same report on demand');++++ s3.close();++ db.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('storage orphans: all checks passed');++ process.exit(0);++})().catch((err) => { console.error(err); process.exit(1); });
@@ -52,7 +52,8 @@ server/
@@ -52,7 +52,8 @@ server/
cancellation), routes (/api/v2/:app/jobs), types: thumbnail.regenerate,
cancellation), routes (/api/v2/:app/jobs), types: thumbnail.regenerate,
invariant.scan (proposes split/remux), object.split, object.remux,
invariant.scan (proposes split/remux), object.split, object.remux,
vod.finalize (orphans and failed finalizes, with backoff),
vod.finalize (orphans and failed finalizes, with backoff),
- vod.duration.reconcile (stored vs measured durations, local and B2/R2)
+ vod.duration.reconcile (stored vs measured durations, local and B2/R2),
+ storage.orphans.scan (monthly storage orphan report, service-wide, report only)
client-ip.js trust proxy = loopback; req.ip is the only client address
client-ip.js trust proxy = loopback; req.ip is the only client address
(openvibe-shared v1.5.1, openvibe-contracts v0.33.0, openvibe-sdk v0.5.0: pinned release tarballs, installed by npm)
(openvibe-shared v1.5.1, openvibe-contracts v0.33.0, openvibe-sdk v0.5.0: pinned release tarballs, installed by npm)
scripts/smoke-test.sh end-to-end smoke test (boots a temp instance)
scripts/smoke-test.sh end-to-end smoke test (boots a temp instance)
@@ -60,7 +61,9 @@ scripts/backfill-objects.js / reconcile-objects.js / object-invariant.js / no-go
@@ -60,7 +61,9 @@ scripts/backfill-objects.js / reconcile-objects.js / object-invariant.js / no-go
scripts/media-jobs.js list jobs, run the size-invariant scan (dry run by default), approve/cancel proposals
scripts/media-jobs.js list jobs, run the size-invariant scan (dry run by default), approve/cancel proposals
scripts/r2-eviction-drill.js evict one VOD's R2 copy, prove B2 serves it, re-warm R2; JSON artifact (dry run by default)
scripts/r2-eviction-drill.js evict one VOD's R2 copy, prove B2 serves it, re-warm R2; JSON artifact (dry run by default)
scripts/vod-duration-reconcile.js stored VOD durations vs the real files (dry run by default; --apply --backup; --rollback)
scripts/vod-duration-reconcile.js stored VOD durations vs the real files (dry run by default; --apply --backup; --rollback)
-scripts/vods-orphans-report.js read-only report on the B2 vods-orphans/ prefix (matches, sizes, recommendations)
+scripts/vods-orphans-report.js read-only report on the B2 vods-orphans/ prefix (matches, sizes, recommendations);
+ --storage: the whole storage against the database (files and keys no row names,
+ copies that are not there, open multipart uploads), as the monthly job does
```
```
## Visitor sign-in
## Visitor sign-in
@@ -410,6 +413,10 @@ model current. Full reference: **[docs/object-model.md](docs/object-model.md)**.
@@ -410,6 +413,10 @@ model current. Full reference: **[docs/object-model.md](docs/object-model.md)**.
read-only by default; `--verify` HEADs B2/R2 and records location states.
read-only by default; `--verify` HEADs B2/R2 and records location states.
Reports missing canonical copies, lost local files, size/hash mismatches,
Reports missing canonical copies, lost local files, size/hash mismatches,
orphan locations, deleted-but-still-served objects and unprojected rows.
orphan locations, deleted-but-still-served objects and unprojected rows.
+- **Orphan report** — monthly job `storage.orphans.scan` (`MEDIA_ORPHAN_SCAN_DAYS`, 30) and
+ `node scripts/vods-orphans-report.js --storage [--no-remote]`: local files and B2/R2 keys no row
+ names, recorded copies that are not there, open multipart uploads (Media's and the buckets').
+ Report only: it never deletes anything ([docs](docs/object-model.md#storage-orphan-report)).
- **Public object-size invariant** — `MEDIA_PUBLIC_OBJECT_MAX_MB` (500) /
- **Public object-size invariant** — `MEDIA_PUBLIC_OBJECT_MAX_MB` (500) /
`_TARGET_MB` (256) / `_WARN_MB` (384): v2 refuses oversized public playback
`_TARGET_MB` (256) / `_WARN_MB` (384): v2 refuses oversized public playback
uploads; `node scripts/object-invariant.js` lists offenders and records them in
uploads; `node scripts/object-invariant.js` lists offenders and records them in
@@ -367,9 +367,45 @@ The exit code is 1 when issues were found.
@@ -367,9 +367,45 @@ The exit code is 1 when issues were found.
| `missing_projection` | an inherited row with no object yet (run the backfill) |
| `missing_projection` | an inherited row with no object yet (run the backfill) |
| `incomplete_multipart` | a multipart session still open past its expiry (the hourly purge should have removed it) |
| `incomplete_multipart` | a multipart session still open past its expiry (the hourly purge should have removed it) |
-**Not covered yet:** orphan *provider* objects (bucket keys with no row, which needs bucket
-listing). `server/objects/reconcile.js` takes a `head` function,
-so tests run it against a fake provider.
+Bucket keys and local files that no row names are the [storage orphan report](#storage-orphan-report)'s.
+`server/objects/reconcile.js` takes a `head` function, so tests run it against a fake provider.
+
+## Storage orphan report
+
+Code: `server/vod/orphans-report.js` (`buildStorageReport`), the job `storage.orphans.scan`
+(`server/jobs/storage-orphans.js`), `scripts/vods-orphans-report.js --storage`. Test: `test/storage-orphans.test.js`
+(a fake S3 through the real SDK that refuses and counts any write).
+
+It compares what storage holds with what the database names, and **only reports**: it walks the data
+directories, lists the B2/R2 buckets (keys and open multipart uploads), reads the database and writes
+`<data>/reports/storage-orphans-<time>.json`. It never deletes, moves or changes a file, a bucket object
+or a row.
+
+| section | what it lists |
+|---|---|
+| `unreferenced_local` | files under `VOD_PATH`, `CLIPS_PATH`, `THUMBNAILS_PATH`, `FILES_PATH`, `PASTES_PATH`, `OBJECTS_PATH`, `ASSETS_PATH` that no row names (`media_locations`, `vods` with their sidecars, masters and chunk segments, `clips`, `files`, `pastes`, `assets`, open upload sessions, unfinished jobs). A file whose only row is a deleted object's location counts, unless that object is native and inside its retention period, or held. Largest first, with a hint: a recorder name whose VOD row is gone, bytes of a deleted object, parts of a session that no longer exists, a temp file of an upload that never finished |
+| `unreferenced_remote` | B2/R2 keys that no `media_locations` row and no `vods`/`clips` row names; hints for `vods-orphans/` (see below), an R2 copy left behind after its VOD went back to B2, legacy paste screenshots, recorder names |
+| `missing` | copies the database records for objects that are not deleted, whose bytes are not there: no local file, or a key absent from its bucket's listing |
+| `multipart_local` | Media's own multipart sessions still open (`media_uploads` active or completing), with `expired` when past `expires_at` |
+| `multipart_remote` | multipart uploads the buckets still hold open (a killed upload leaves its parts stored and billed) |
+
+Totals count everything; each list holds at most `limit` items (2000). Live thumbnails
+(`stream-<app>-<id>.jpg`) and upload temps or partial downloads younger than an hour are counted as
+`expected`, not listed. A provider that is not configured, or whose listing fails, is marked not listed
+and nothing is concluded about its keys.
+
+**When it runs.** The job worker queues `storage.orphans.scan` every `MEDIA_ORPHAN_SCAN_DAYS` (30; `0`
+= never). Storage is shared by every tenant, so the job belongs to none: it runs under the system app
+`_media` (no apps row, so no API credential reaches it), announces no `media.job.*` event, and
+`POST /api/v2/:app/jobs` refuses it with 403 `media.job.forbidden`. The worker does not start in a
+restore drill. On demand:
+
+```
+node scripts/vods-orphans-report.js --storage [--no-remote] [--limit 2000] [--out report.json] [--json] [--db ./data/media.db]
+```
+
+`--no-remote` lists no bucket. The detailed per-object recommendations for the `vods-orphans/` prefix
+stay with the prefix report below (`scripts/vods-orphans-report.js` without `--storage`).
## Scheduled verification
## Scheduled verification
@@ -509,6 +545,7 @@ requeued (or failed when out of attempts). Handlers checkpoint progress and resu
@@ -509,6 +545,7 @@ requeued (or failed when out of attempts). Handlers checkpoint progress and resu
| `object.remux` | heavy | Stream-copy remux of the whole source (seek index, duration, MP4 faststart) into one new private object; also the source's `remux` variant |
| `object.remux` | heavy | Stream-copy remux of the whole source (seek index, duration, MP4 faststart) into one new private object; also the source's `remux` variant |
| `object.hash` | light | sha256 of ready objects' present local copies that have no hash yet, onto the object (`content_hash`, `metadata.hash_basis`) and the local location (`checksum`). Tenant-wide batch (`params { limit (1-500, 50), budget_mb }`) or one object. Scheduled per tenant every `MEDIA_HASH_INTERVAL_MIN`. Result `{ hashed, bytes, skipped, remaining }`. See [Scheduled verification](#scheduled-verification) |
| `object.hash` | light | sha256 of ready objects' present local copies that have no hash yet, onto the object (`content_hash`, `metadata.hash_basis`) and the local location (`checksum`). Tenant-wide batch (`params { limit (1-500, 50), budget_mb }`) or one object. Scheduled per tenant every `MEDIA_HASH_INTERVAL_MIN`. Result `{ hashed, bytes, skipped, remaining }`. See [Scheduled verification](#scheduled-verification) |
| `vod.duration.reconcile` | heavy | One bounded batch of the tenant's finished VODs: stored duration vs a measurement of the real file, local or the B2/R2 copy (ffprobe over a presigned URL, ranged reads). `params { limit (1-200, 25), after_id?, apply (false), confirm_remote (false) }`; without `after_id` it walks the library from where the last run stopped. Repairs only confirmed, clearly wrong values when `apply`. Result `{ counts, range, next_after_id, report }`. See [Duration reconciliation](#duration-reconciliation) |
| `vod.duration.reconcile` | heavy | One bounded batch of the tenant's finished VODs: stored duration vs a measurement of the real file, local or the B2/R2 copy (ffprobe over a presigned URL, ranged reads). `params { limit (1-200, 25), after_id?, apply (false), confirm_remote (false) }`; without `after_id` it walks the library from where the last run stopped. Repairs only confirmed, clearly wrong values when `apply`. Result `{ counts, range, next_after_id, report }`. See [Duration reconciliation](#duration-reconciliation) |
+| `storage.orphans.scan` | light | The [storage orphan report](#storage-orphan-report), service-wide, under the system app `_media`; report only. `params { limit (1-100000, 2000) }`. Result `{ totals, providers, summary, report }` (the report file under `<data>/reports/`). Scheduled every `MEDIA_ORPHAN_SCAN_DAYS`; tenants cannot enqueue it (403) |
| `vod.finalize` | finalize | Finalizes a recording whose finalize failed or never ran. `params { vod_id }`. Queued by the orphan sweep (a row still `is_recording = 1` that no recorder, chunk upload or finalize holds and whose file has been idle for `MEDIA_FINALIZE_ORPHAN_GRACE_S`, 180 s; at boot and every `MEDIA_FINALIZE_SWEEP_S`, 120 s) and by finalize itself when it could not settle a recording (nothing measurable, a stat failure, a throw; first run after 5 min). Still unmeasurable: retried (5 min, 15 min, 45 min, 2 h 15, 6 h; 6 attempts), then the VOD stays `needs_review`, hidden. Never stops a live recording. Result `{ outcome: ready \| needs_review \| corrupt \| deleted \| skipped, duration_seconds, duration_source }` |
| `vod.finalize` | finalize | Finalizes a recording whose finalize failed or never ran. `params { vod_id }`. Queued by the orphan sweep (a row still `is_recording = 1` that no recorder, chunk upload or finalize holds and whose file has been idle for `MEDIA_FINALIZE_ORPHAN_GRACE_S`, 180 s; at boot and every `MEDIA_FINALIZE_SWEEP_S`, 120 s) and by finalize itself when it could not settle a recording (nothing measurable, a stat failure, a throw; first run after 5 min). Still unmeasurable: retried (5 min, 15 min, 45 min, 2 h 15, 6 h; 6 attempts), then the VOD stays `needs_review`, hidden. Never stops a live recording. Result `{ outcome: ready \| needs_review \| corrupt \| deleted \| skipped, duration_seconds, duration_source }` |
Split and remux refuse a source that is not ready, a tenant quota the output would exceed, and too
Split and remux refuse a source that is not ready, a tenant quota the output would exceed, and too
@@ -634,6 +671,7 @@ Not run against production yet: it needs the owner's go-ahead.
@@ -634,6 +671,7 @@ Not run against production yet: it needs the owner's go-ahead.
| `MEDIA_FINALIZE_ORPHAN_GRACE_S` / `MEDIA_FINALIZE_SWEEP_S` | 180 / 120 | an orphaned recording's file must be idle this long; the orphan sweep runs this often |
| `MEDIA_FINALIZE_ORPHAN_GRACE_S` / `MEDIA_FINALIZE_SWEEP_S` | 180 / 120 | an orphaned recording's file must be idle this long; the orphan sweep runs this often |
| `MEDIA_HASH_INTERVAL_MIN` | 15 | queue `object.hash` per tenant this often (`0` = only on demand) |
| `MEDIA_HASH_INTERVAL_MIN` | 15 | queue `object.hash` per tenant this often (`0` = only on demand) |
| `MEDIA_HASH_BUDGET_MB` / `_MAX_FILE_MB` / `_SETTLE_S` | 4096 / 20480 / 120 | bytes read per run; largest file hashed; a file changed this recently waits |
| `MEDIA_HASH_BUDGET_MB` / `_MAX_FILE_MB` / `_SETTLE_S` | 4096 / 20480 / 120 | bytes read per run; largest file hashed; a file changed this recently waits |
+| `MEDIA_ORPHAN_SCAN_DAYS` | 30 | queue the storage orphan report this often (`0` = only on demand) |
| `MEDIA_DURATION_RECONCILE_HOURS` | 0 | schedule `vod.duration.reconcile` per tenant this often (`0` = only on demand) |
| `MEDIA_DURATION_RECONCILE_HOURS` | 0 | schedule `vod.duration.reconcile` per tenant this often (`0` = only on demand) |
| `MEDIA_DURATION_RECONCILE_APPLY` / `_BATCH` | off / 25 | scheduled runs repair (`1`) or only report; VODs per scheduled run |
| `MEDIA_DURATION_RECONCILE_APPLY` / `_BATCH` | off / 25 | scheduled runs repair (`1`) or only report; VODs per scheduled run |
| `MEDIA_JOBS_HEAVY_WHILE_RECORDING` | off | `1` lets split/remux run while a recording is being written |
| `MEDIA_JOBS_HEAVY_WHILE_RECORDING` | off | `1` lets split/remux run while a recording is being written |
@@ -657,6 +695,5 @@ Not run against production yet: it needs the owner's go-ahead.
@@ -657,6 +695,5 @@ Not run against production yet: it needs the owner's go-ahead.
- Copy/move aliases, lifecycle rules and an S3-compatible façade.
- Copy/move aliases, lifecycle rules and an S3-compatible façade.
- Clip cutting and the finalize remux still run inline, not as jobs (thumbnail regeneration was the first
- Clip cutting and the finalize remux still run inline, not as jobs (thumbnail regeneration was the first
operation moved onto the job system).
operation moved onto the job system).
-- Orphan-bucket-object detection.
- The segment-native timeline.
- The segment-native timeline.
- App assets (emotes and sounds, the `assets` table) are not projected yet. The `asset` kind exists for v2 uploads.
- App assets (emotes and sounds, the `assets` table) are not projected yet. The `asset` kind exists for v2 uploads.
@@ -1,15 +1,23 @@
@@ -1,15 +1,23 @@
#!/usr/bin/env node
#!/usr/bin/env node
'use strict';
'use strict';
/**
/**
- * Report on the B2 `vods-orphans/` prefix (hazard H3). Read-only: it lists the prefix (ListObjectsV2),
- * HEADs each match's canonical copy, reads the database, and writes a report. It never deletes, moves,
- * copies or writes an object or a row (server/vod/orphans-report.js has the rules).
+ * Orphan reports (server/vod/orphans-report.js has the rules). Read-only: they list, HEAD and read the
+ * database, and write a report file. They never delete, move, copy or write an object, a file or a row.
*
*
* node scripts/vods-orphans-report.js [--prefix vods-orphans/] [--provider b2] [--out <report.json>] [--json] [--db <media.db>]
* node scripts/vods-orphans-report.js [--prefix vods-orphans/] [--provider b2] [--out <report.json>] [--json] [--db <media.db>]
*
*
- * Prints one line per object (key, size, matched VOD/clip, recommendation) and the totals per
- * recommendation, and writes the full report (default <data>/reports/vods-orphans-<time>.json).
- * B2 credentials come from the environment (/etc/openvibe/media.env names). Exit 0 done, 1 error, 2 usage.
+ * The B2 `vods-orphans/` prefix (hazard H3): lists the prefix (ListObjectsV2), HEADs each match's
+ * canonical copy, prints one line per object (key, size, matched VOD/clip, recommendation) and the
+ * totals per recommendation, and writes the full report (default <data>/reports/vods-orphans-<time>.json).
+ *
+ * node scripts/vods-orphans-report.js --storage [--no-remote] [--limit 2000] [--out <report.json>] [--json] [--db <media.db>]
+ *
+ * The whole storage against the database (the storage.orphans.scan job, on demand): files under the data
+ * directories and B2/R2 keys that no row names, copies the database records that are not there, and
+ * multipart uploads left open (Media's sessions and the buckets'). --no-remote lists no bucket. Prints
+ * the totals and the largest items, and writes <data>/reports/storage-orphans-<time>.json.
+ *
+ * B2/R2 credentials come from the environment (/etc/openvibe/media.env names). Exit 0 done, 1 error, 2 usage.
*/
*/
const fs = require('fs');
const fs = require('fs');
const path = require('path');
const path = require('path');
@@ -18,7 +26,8 @@ const args = process.argv.slice(2);
@@ -18,7 +26,8 @@ const args = process.argv.slice(2);
const has = (name) => args.includes(`--${name}`);
const has = (name) => args.includes(`--${name}`);
const arg = (name) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] && !args[i + 1].startsWith('--') ? args[i + 1] : null; };
const arg = (name) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] && !args[i + 1].startsWith('--') ? args[i + 1] : null; };
const usage = (msg) => { console.error(`vods-orphans-report: ${msg}`); process.exit(2); };
const usage = (msg) => { console.error(`vods-orphans-report: ${msg}`); process.exit(2); };
-for (const flag of ['prefix', 'provider', 'out', 'db']) if (has(flag) && !arg(flag)) usage(`--${flag} needs a value`);
+for (const flag of ['prefix', 'provider', 'out', 'db', 'limit']) if (has(flag) && !arg(flag)) usage(`--${flag} needs a value`);
+if (has('limit') && !(Number.isInteger(Number(arg('limit'))) && Number(arg('limit')) > 0)) usage('--limit must be a positive integer');
if (arg('db')) process.env.DB_PATH = path.resolve(arg('db'));
if (arg('db')) process.env.DB_PATH = path.resolve(arg('db'));
const provider = arg('provider') || 'b2';
const provider = arg('provider') || 'b2';
if (!['b2', 'r2'].includes(provider)) usage('--provider must be b2 or r2');
if (!['b2', 'r2'].includes(provider)) usage('--provider must be b2 or r2');
@@ -32,7 +41,32 @@ const JSON_OUT = has('json');
@@ -32,7 +41,32 @@ const JSON_OUT = has('json');
const log = (...a) => { if (!JSON_OUT) console.log(...a); };
const log = (...a) => { if (!JSON_OUT) console.log(...a); };
const mb = (b) => `${(b / 1048576).toFixed(1)} MB`;
const mb = (b) => `${(b / 1048576).toFixed(1)} MB`;
+async function storageReport() {
+ const providers = has('no-remote') ? [] : null;
+ const report = await orphans.buildStorageReport({ providers, limit: has('limit') ? Number(arg('limit')) : 2000 });
+ const file = orphans.writeStorageReport(report, arg('out') ? path.resolve(arg('out')) : null);
+ const t = report.totals;
+ log(`Storage orphan report (read-only), ${report.generated_at}`);
+ for (const r of report.scope.roots) log(` local ${r.name.padEnd(11)} ${r.dir}${r.exists ? ` (${r.files} file(s))` : ' (missing)'}`);
+ for (const [p, e] of Object.entries(report.scope.providers)) log(` ${p.padEnd(17)} ${e.listed ? `${e.bucket || ''}: ${e.keys} key(s), ${mb(e.bytes)}` : `not listed${e.error ? `: ${e.error}` : ''}`}`);
+ if (has('no-remote')) log(' buckets not listed (--no-remote)');
+ log('');
+ log(`Unreferenced local files: ${t.unreferenced_local.files} (${mb(t.unreferenced_local.bytes)})`);
+ for (const f of report.unreferenced_local.slice(0, 25)) log(` ${mb(f.size).padStart(11)} ${f.root}/${f.path}${f.hint ? ` — ${f.hint}` : ''}`);
+ for (const [p, x] of Object.entries(t.unreferenced_remote)) log(`Unreferenced ${p} keys: ${x.keys} (${mb(x.bytes)})`);
+ for (const k of report.unreferenced_remote.slice(0, 25)) log(` ${mb(k.size).padStart(11)} ${k.provider}:${k.key}${k.hint ? ` — ${k.hint}` : ''}`);
+ log(`Copies the database records that are not there: ${t.missing.locations}`);
+ for (const m of report.missing.slice(0, 25)) log(` ${m.object_id} ${m.legacy_ref || m.kind} ${m.provider} (recorded ${m.recorded_state})`);
+ log(`Open multipart uploads: ${t.multipart_local.open} here (${t.multipart_local.expired} expired)${Object.entries(t.multipart_remote).map(([p, n]) => `, ${n} in ${p}`).join('')}`);
+ log(`Expected, not listed: ${t.expected.live_thumbnails} live thumbnail(s), ${t.expected.in_flight} upload(s) in flight`);
+ log('');
+ log('Nothing was deleted or moved.');
+ log(`Report: ${file}`);
+ if (JSON_OUT) console.log(JSON.stringify({ ...report, report: file }, null, 2));
+}
+
(async () => {
(async () => {
+ if (has('storage')) { await storageReport(); db.close(); process.exit(0); }
if (!vodStorage.providerConfigured(provider)) usage(`${provider} is not configured (MEDIA_${provider.toUpperCase()}_* not in the environment)`);
if (!vodStorage.providerConfigured(provider)) usage(`${provider} is not configured (MEDIA_${provider.toUpperCase()}_* not in the environment)`);
const report = await orphans.buildReport({ provider, prefix: arg('prefix') || orphans.PREFIX });
const report = await orphans.buildReport({ provider, prefix: arg('prefix') || orphans.PREFIX });
const file = arg('out') ? path.resolve(arg('out')) : path.join(path.dirname(config.db.path), 'reports', `vods-orphans-${report.generated_at.replace(/[:.]/g, '-')}.json`);
const file = arg('out') ? path.resolve(arg('out')) : path.join(path.dirname(config.db.path), 'reports', `vods-orphans-${report.generated_at.replace(/[:.]/g, '-')}.json`);
@@ -19,7 +19,7 @@
@@ -19,7 +19,7 @@
* Watch pages and the media index render from the database only and still answer.
* Watch pages and the media index render from the database only and still answer.
* - server/index.js opens the database and serves HTTP, and starts nothing else: no app seeding, no
* - server/index.js opens the database and serves HTTP, and starts nothing else: no app seeding, no
* JWKS refresh, no health job, junk sweep, tiering sweep, clip re-cuts, copy verification, jobs
* JWKS refresh, no health job, junk sweep, tiering sweep, clip re-cuts, copy verification, jobs
- * worker (thumbnails, split/remux, invariant scans), disk guardian, thumbnail cleanup, object purge,
+ * worker (thumbnails, split/remux, invariant scans, storage orphan scans), disk guardian, thumbnail cleanup, object purge,
* backfill, orphan-recording finalize or Events relay; webhooks are never sent. Timers that modules
* backfill, orphan-recording finalize or Events relay; webhooks are never sent. Timers that modules
* start when they are loaded check `enabled` as well. Readiness does not probe storage (a drill
* start when they are loaded check `enabled` as well. Readiness does not probe storage (a drill
* writes none) or the remote tiers.
* writes none) or the remote tiers.
@@ -126,6 +126,8 @@ function record(webhookEvent, appId, data) {
@@ -126,6 +126,8 @@ function record(webhookEvent, appId, data) {
function recordJob(transition, job) {
function recordJob(transition, job) {
const priority = JOB_TRANSITIONS[transition];
const priority = JOB_TRANSITIONS[transition];
if (!priority) throw new Error(`unknown job transition ${transition}`);
if (!priority) throw new Error(`unknown job transition ${transition}`);
+ // A service-wide maintenance job (queue.SYSTEM_APP) is no tenant's: nothing to announce.
+ if (job.app_id === require('./jobs/queue').SYSTEM_APP) return null;
return enqueue(`media.job.${transition}`, job.app_id, { type: 'job', id: String(job.id) }, job, priority);
return enqueue(`media.job.${transition}`, job.app_id, { type: 'job', id: String(job.id) }, job, priority);
}
}
@@ -28,6 +28,9 @@ const STATUSES = ['proposed', 'queued', 'running', 'succeeded', 'failed', 'cance
@@ -28,6 +28,9 @@ const STATUSES = ['proposed', 'queued', 'running', 'succeeded', 'failed', 'cance
const FINISHED = ['succeeded', 'failed', 'cancelled'];
const FINISHED = ['succeeded', 'failed', 'cancelled'];
const ACTIVE = ['queued', 'running'];
const ACTIVE = ['queued', 'running'];
const MAX_ACTIVE_PER_TENANT = 50; // queued + running jobs one tenant may have (proposals excluded)
const MAX_ACTIVE_PER_TENANT = 50; // queued + running jobs one tenant may have (proposals excluded)
+// Service-wide maintenance jobs (storage.orphans.scan) belong to no tenant: they run under this app id,
+// which has no apps row, so no API credential reaches them, and they announce no events.
+const SYSTEM_APP = '_media';
const bus = new EventEmitter();
const bus = new EventEmitter();
bus.setMaxListeners(0);
bus.setMaxListeners(0);
@@ -394,7 +397,7 @@ function prune({ days = 30, types = ['thumbnail.regenerate', 'object.hash'] } =
@@ -394,7 +397,7 @@ function prune({ days = 30, types = ['thumbnail.regenerate', 'object.hash'] } =
}
}
module.exports = {
module.exports = {
- STATUSES, FINISHED, ACTIVE, MAX_ACTIVE_PER_TENANT, JobError, bus,
+ STATUSES, FINISHED, ACTIVE, MAX_ACTIVE_PER_TENANT, SYSTEM_APP, JobError, bus,
register, typeSpec, typeNames, requestHash, parseJson,
register, typeSpec, typeNames, requestHash, parseJson,
get, getForApp, jobPublic, jobEvent, list, counts,
get, getForApp, jobPublic, jobEvent, list, counts,
enqueue, approve, cancel,
enqueue, approve, cancel,
new file mode 100644
new file mode 100644
@@ -0,0 +1,73 @@
@@ -0,0 +1,73 @@
+/**
+ * OpenVibe.Media — job type storage.orphans.scan (light lane): the storage orphan report
+ * (server/vod/orphans-report.js buildStorageReport) across the whole service. Report only: it walks the
+ * data directories, lists the B2/R2 buckets (keys and open multipart uploads), reads the database and
+ * writes <data>/reports/storage-orphans-<time>.json. It never deletes, moves or changes a file, a bucket
+ * object or a row.
+ *
+ * params { limit (1-100000, default 2000): items listed per section; the totals count everything }
+ * result { totals, providers, summary, report } report = the file name under <data>/reports/
+ *
+ * Storage is shared by every tenant, so the job belongs to none: it runs under queue.SYSTEM_APP, which no
+ * API credential reaches (no apps row), announces no media.job.* event, and cannot be enqueued through
+ * /api/v2/:app/jobs (validate refuses every tenant). Scheduled every MEDIA_ORPHAN_SCAN_DAYS (30; 0 = never)
+ * by the worker, which does not start in a restore drill. On demand: scripts/vods-orphans-report.js --storage.
+ */
+'use strict';
+
+const path = require('path');
+const queue = require('./queue');
+const orphans = require('../vod/orphans-report');
+
+const { JobError } = queue;
+const TYPE = 'storage.orphans.scan';
+
+function envInt(name, fallback) {
+ const v = parseInt(process.env[name] || '', 10);
+ return Number.isFinite(v) ? v : fallback;
+}
+
+function checkParams(params) {
+ const p = params || {};
+ const limit = p.limit == null ? 2000 : Number(p.limit);
+ if (!Number.isInteger(limit) || limit < 1 || limit > 100000) throw new JobError('media.job.invalid', 'params.limit must be 1-100000', { permanent: true });
+ return { limit };
+}
+
+function validate({ appId, obj, params }) {
+ if (appId !== queue.SYSTEM_APP) {
+ throw new JobError('media.job.forbidden', `${TYPE} covers the whole service's storage: it runs on its schedule or from scripts/vods-orphans-report.js --storage`, { status: 403, permanent: true });
+ }
+ if (obj) throw new JobError('media.job.invalid', `${TYPE} covers the whole storage: leave object_id out`, { permanent: true });
+ return checkParams(params);
+}
+
+async function run(job) {
+ const p = checkParams(job.params);
+ const report = await orphans.buildStorageReport({ limit: p.limit });
+ report.job_id = job.id;
+ const file = orphans.writeStorageReport(report);
+ const summary = orphans.summarizeStorage(report);
+ console.log(`[Orphans] ${summary} (report ${path.basename(file)}; nothing was deleted)`);
+ return { totals: report.totals, providers: report.scope.providers, summary, report: path.basename(file) };
+}
+
+/** Queue one scan every MEDIA_ORPHAN_SCAN_DAYS (0 = never). Returns how many were queued (0 or 1). */
+function schedule(nowMs = Date.now()) {
+ const days = envInt('MEDIA_ORPHAN_SCAN_DAYS', 30);
+ if (!(days > 0)) return 0;
+ const period = Math.floor(nowMs / (days * 86400 * 1000));
+ try {
+ const r = queue.enqueue({ appId: queue.SYSTEM_APP, type: TYPE, params: {}, idempotencyKey: `${TYPE}:${days}d:${period}`, createdBy: 'system:schedule' });
+ return r.created ? 1 : 0;
+ } catch (err) {
+ console.warn(`[Jobs] could not schedule ${TYPE}: ${err.message}`);
+ return 0;
+ }
+}
+
+module.exports = {
+ TYPE,
+ spec: { lane: 'light', maxAttempts: 2, timeoutMs: 2 * 60 * 60 * 1000, needsObject: false, validate, run },
+ schedule,
+};
@@ -8,6 +8,7 @@
@@ -8,6 +8,7 @@
* vod.finalize finalize finalize a recording whose finalize failed or never ran (orphans), with backoff
* vod.finalize finalize finalize a recording whose finalize failed or never ran (orphans), with backoff
* vod.duration.reconcile heavy stored VOD durations vs a measurement of the real file (local or B2/R2), one batch
* vod.duration.reconcile heavy stored VOD durations vs a measurement of the real file (local or B2/R2), one batch
* object.hash light sha256 of local copies with no content hash yet, one bounded batch (or one object)
* object.hash light sha256 of local copies with no content hash yet, one bounded batch (or one object)
+ * storage.orphans.scan light the storage orphan report, service-wide (report only; never deletes)
*/
*/
'use strict';
'use strict';
@@ -20,5 +21,6 @@ queue.register('object.remux', require('./derive').remux);
@@ -20,5 +21,6 @@ queue.register('object.remux', require('./derive').remux);
queue.register('vod.finalize', require('./vod-finalize').spec);
queue.register('vod.finalize', require('./vod-finalize').spec);
queue.register('vod.duration.reconcile', require('./duration-reconcile').spec);
queue.register('vod.duration.reconcile', require('./duration-reconcile').spec);
queue.register('object.hash', require('./content-hash').spec);
queue.register('object.hash', require('./content-hash').spec);
+queue.register('storage.orphans.scan', require('./storage-orphans').spec);
module.exports = { names: () => queue.typeNames() };
module.exports = { names: () => queue.typeNames() };
@@ -16,7 +16,8 @@
@@ -16,7 +16,8 @@
*
*
* Scheduling: the size-invariant validator (invariant.scan) is enqueued per tenant every
* Scheduling: the size-invariant validator (invariant.scan) is enqueued per tenant every
* MEDIA_INVARIANT_SCAN_HOURS (it proposes split/remux jobs; it never runs them). Orphaned recordings
* MEDIA_INVARIANT_SCAN_HOURS (it proposes split/remux jobs; it never runs them). Orphaned recordings
- * get a vod.finalize every MEDIA_FINALIZE_SWEEP_S (server/jobs/vod-finalize.js).
+ * get a vod.finalize every MEDIA_FINALIZE_SWEEP_S (server/jobs/vod-finalize.js). The storage orphan
+ * report (storage.orphans.scan, report only) runs every MEDIA_ORPHAN_SCAN_DAYS under queue.SYSTEM_APP.
*/
*/
'use strict';
'use strict';
@@ -149,6 +150,7 @@ async function tick() {
@@ -149,6 +150,7 @@ async function tick() {
scheduleInvariantScans(now);
scheduleInvariantScans(now);
try { require('./duration-reconcile').schedule(now); } catch (err) { console.warn('[Jobs] duration reconcile schedule:', err.message); }
try { require('./duration-reconcile').schedule(now); } catch (err) { console.warn('[Jobs] duration reconcile schedule:', err.message); }
try { require('./content-hash').schedule(now); } catch (err) { console.warn('[Jobs] content hash schedule:', err.message); }
try { require('./content-hash').schedule(now); } catch (err) { console.warn('[Jobs] content hash schedule:', err.message); }
+ try { require('./storage-orphans').schedule(now); } catch (err) { console.warn('[Jobs] storage orphan scan schedule:', err.message); }
}
}
try { require('./vod-finalize').sweepOrphans({ now }); } catch (err) { console.warn('[Jobs] orphan sweep:', err.message); }
try { require('./vod-finalize').sweepOrphans({ now }); } catch (err) { console.warn('[Jobs] orphan sweep:', err.message); }
if (now - lastPruneAt > 6 * 3600 * 1000) {
if (now - lastPruneAt > 6 * 3600 * 1000) {
@@ -1,6 +1,14 @@
@@ -1,6 +1,14 @@
/**
/**
- * OpenVibe.Media — report on the B2 `vods-orphans/` prefix (hazard H3: 55 objects, 14.5 GB on
- * 2026-09-21). Report only: it lists, matches and recommends; it never deletes, moves or writes.
+ * OpenVibe.Media — orphan reports. Report only: they list, match and recommend; they never delete,
+ * move or write a file, a bucket object or a row.
+ *
+ * buildReport() the B2 `vods-orphans/` prefix (hazard H3: 55 objects, 14.5 GB on 2026-09-21)
+ * buildStorageReport() the whole storage against the database (roadmap WS-G task 8): files and keys
+ * no row names, copies the database records that are not there, and multipart
+ * uploads left open (Media's own and the buckets'). The storage.orphans.scan job
+ * runs it monthly; scripts/vods-orphans-report.js --storage on demand.
+ *
+ * ── The vods-orphans/ prefix ──
*
*
* Each object under the prefix is matched to a VOD or clip by its file name (the basename of
* Each object under the prefix is matched to a VOD or clip by its file name (the basename of
* vods.file_path / storage_key, clips.file_path; else the vod-<app>-<id>-<ms>.<ext> name the
* vods.file_path / storage_key, clips.file_path; else the vod-<app>-<id>-<ms>.<ext> name the
@@ -22,6 +30,7 @@
@@ -22,6 +30,7 @@
const fs = require('fs');
const fs = require('fs');
const path = require('path');
const path = require('path');
+const config = require('../config');
const db = require('../db/database');
const db = require('../db/database');
const PREFIX = 'vods-orphans/';
const PREFIX = 'vods-orphans/';
@@ -136,4 +145,321 @@ async function buildReport({ provider = 'b2', prefix = PREFIX, list = null } = {
@@ -136,4 +145,321 @@ async function buildReport({ provider = 'b2', prefix = PREFIX, list = null } = {
};
};
}
}
-module.exports = { PREFIX, RECOMMENDATIONS, parseRecordingName, assess, buildReport };
+// ═══════════════════════════════════════════════════════════════
+// The whole storage against the database (WS-G task 8)
+// ═══════════════════════════════════════════════════════════════
+//
+// unreferenced_local files under the data directories that no row names (media_locations, vods,
+// clips, files, pastes, assets, open upload sessions, unfinished jobs). A file whose
+// only row is a deleted object's location counts (its bytes outlived the object),
+// unless the object is a native one still inside its retention period, or held.
+// unreferenced_remote B2/R2 keys that no media_locations row or vods/clips row names
+// missing copies the database records for objects that are not deleted, whose bytes are not
+// there: no local file, or a key absent from its bucket's listing
+// multipart_local Media's own upload sessions still open (media_uploads active/completing), expired
+// ones flagged; part directories no open session owns are unreferenced_local
+// multipart_remote multipart uploads the buckets still hold open (ListMultipartUploads)
+//
+// Expected files that no row names are counted, not listed: live thumbnails (stream-<app>-<id>.jpg,
+// refreshed in place while a stream is live) and upload temps / partial downloads younger than an hour.
+// A provider that is not configured, or whose listing fails, is reported as not listed; nothing is
+// concluded about its keys.
+
+const STORAGE_SECTIONS = ['unreferenced_local', 'unreferenced_remote', 'missing', 'multipart_local', 'multipart_remote'];
+const IN_FLIGHT_MS = 60 * 60 * 1000;
+const MAX_WALK = 1000000;
+
+function localRoots() {
+ const seen = new Set();
+ return [
+ ['vods', config.vod.path], ['clips', config.vod.clipsPath], ['thumbnails', config.thumbnails.path], ['files', config.files.path],
+ ['pastes', config.pastes.path], ['objects', config.objects.path], ['assets', config.assets.path],
+ ].map(([name, dir]) => ({ name, dir: path.resolve(dir) })).filter(r => (seen.has(r.dir) ? false : seen.add(r.dir)));
+}
+
+/** Every regular file under one root. Symlinks are not followed; a root nested inside it is walked on its own. */
+function walkRoot(root, rootDirs, out) {
+ const stack = [root.dir];
+ let seen = 0;
+ while (stack.length) {
+ const d = stack.pop();
+ let entries;
+ try { entries = fs.readdirSync(d, { withFileTypes: true }); } catch { continue; }
+ for (const e of entries) {
+ const p = path.join(d, e.name);
+ if (e.isDirectory()) { if (!rootDirs.has(p)) stack.push(p); continue; }
+ if (!e.isFile()) continue;
+ if (out.files.length >= MAX_WALK) { out.truncated = true; return seen; }
+ let st;
+ try { st = fs.statSync(p); } catch { continue; }
+ out.files.push({ root: root.name, abs: p, path: path.relative(root.dir, p), size: st.size, mtimeMs: st.mtimeMs });
+ seen++;
+ }
+ }
+ return seen;
+}
+
+/** A location keeps its bytes wanted: its object exists and is not deleted, or is a native object inside its retention, or is held. */
+function locationWanted(l) {
+ if (!l.object_id) return false;
+ return l.lifecycle_status !== 'deleted' || (!l.legacy_ref && !l.purged) || !!l.held;
+}
+
+const LOCATION_ROWS = `SELECT l.provider, l.bucket, l.key, o.id AS object_id, o.lifecycle_status, o.legacy_ref,
+ CASE WHEN json_valid(o.metadata) THEN json_extract(o.metadata, '$.purged_at') END AS purged,
+ EXISTS (SELECT 1 FROM media_holds h WHERE h.object_id = o.id AND h.released_at IS NULL) AS held
+ FROM media_locations l LEFT JOIN media_objects o ON o.id = l.object_id`;
+
+const sidecarsOf = (p) => [p.replace(/\.webm$/, '.seekable.webm'), p.replace(/\.mp4$/, '.seekable.mp4'), p.replace(/\.webm$/, '.master.mkv')].filter(x => x !== p);
+
+/** What the database names on local disk: { wanted: Set(abs path), deleted: Map(abs path → location), segmentBases, sessions, jobs }. */
+function localReferences() {
+ const vs = vodStorage();
+ const model = require('../objects/model');
+ const wanted = new Set();
+ const add = (p) => { if (p) wanted.add(path.resolve(String(p))); };
+ const deleted = new Map();
+ for (const l of db.all(`${LOCATION_ROWS} WHERE l.provider = 'local'`)) {
+ if (locationWanted(l)) add(l.key); else deleted.set(path.resolve(String(l.key)), l);
+ }
+ const thumb = (u) => { const name = model.thumbFileFromUrl(u); if (name) add(path.join(config.thumbnails.path, name)); };
+ const segmentBases = new Set(); // browser-chunk segments <base>.seg-<n>-<ms>.webm belong to <base>.webm
+ for (const v of db.all('SELECT id, file_path, master_file_path, thumbnail_url FROM vods')) {
+ if (v.file_path) {
+ for (const f of [v.file_path, vs.localPathForVod(v)]) {
+ add(f);
+ for (const x of sidecarsOf(path.resolve(f))) add(x);
+ segmentBases.add(path.join(path.dirname(path.resolve(f)), path.basename(f).replace(/\.webm$/, '')));
+ }
+ }
+ add(v.master_file_path);
+ thumb(v.thumbnail_url);
+ }
+ for (const c of db.all('SELECT file_path, thumbnail_url FROM clips')) {
+ if (c.file_path) { add(c.file_path); add(path.join(config.vod.clipsPath, path.basename(c.file_path))); }
+ thumb(c.thumbnail_url);
+ }
+ for (const f of db.all('SELECT key, app_id FROM files')) add(path.join(config.files.path, f.app_id, f.key));
+ for (const p of db.all('SELECT screenshot_path FROM pastes WHERE screenshot_path IS NOT NULL')) add(p.screenshot_path);
+ for (const a of db.all('SELECT file_path FROM assets WHERE file_path IS NOT NULL')) add(a.file_path);
+ const sessions = new Set(db.all("SELECT id FROM media_uploads WHERE status IN ('active', 'completing')").map(r => r.id));
+ const jobs = new Set(db.all("SELECT id FROM media_jobs WHERE status IN ('queued', 'running')").map(r => r.id));
+ return { wanted, deleted, segmentBases, sessions, jobs };
+}
+
+/** Is this local file named by the database (or one of the expected kinds)? 'wanted' | 'live_thumbnail' | 'in_flight' | null. */
+function classifyLocal(f, refs, now) {
+ if (refs.wanted.has(f.abs)) return 'wanted';
+ const base = path.basename(f.abs);
+ const seg = /^(.+)\.seg-\d+-\d+\.webm$/.exec(base);
+ if (seg && refs.segmentBases.has(path.join(path.dirname(f.abs), seg[1]))) return 'wanted';
+ const [top, sub] = f.path.split(path.sep);
+ if (f.root === 'objects' && top === '.parts' && refs.sessions.has(sub)) return 'wanted';
+ if (f.root === 'objects' && top === '.jobs' && refs.jobs.has(sub)) return 'wanted';
+ if (f.root === 'thumbnails' && /^stream-.+\.jpg$/.test(base)) return 'live_thumbnail';
+ const transient = (f.root === 'objects' && top === '.tmp') || base.endsWith('.download') || (f.root === 'objects' && top === '.parts');
+ if (transient && now - f.mtimeMs < IN_FLIGHT_MS) return 'in_flight';
+ return null;
+}
+
+/** A recorder-style name (vod-<app>-<id>-<ms>.<ext>): whose recording, and does its row still exist? */
+function recordingNameHint(base) {
+ const parsed = parseRecordingName(base);
+ if (!parsed) return null;
+ const row = db.get('SELECT id, file_path FROM vods WHERE id = ? AND app_id = ?', [parsed.id, parsed.app]);
+ if (!row) return `a recording of vod ${parsed.id} (${parsed.app}), whose row is gone`;
+ return `named for vod ${parsed.id} (${parsed.app}), whose row points at ${path.basename(row.file_path || '') || 'no file'}`;
+}
+
+function localHint(f, refs) {
+ const base = path.basename(f.abs);
+ const del = refs.deleted.get(f.abs);
+ if (del) return del.object_id ? `bytes of deleted object ${del.object_id}${del.legacy_ref ? ` (${del.legacy_ref})` : ''}` : 'named only by a location row whose object does not exist';
+ if (f.root === 'objects') {
+ const [top, sub] = f.path.split(path.sep);
+ if (top === '.parts') {
+ const s = db.get('SELECT status FROM media_uploads WHERE id = ?', [sub]);
+ return s ? `parts of multipart upload ${sub} (${s.status})` : `parts of multipart upload ${sub}, which has no session`;
+ }
+ if (top === '.jobs') {
+ const j = db.get('SELECT status FROM media_jobs WHERE id = ?', [sub]);
+ return j ? `work files of job ${sub} (${j.status})` : `work files of job ${sub}, which does not exist`;
+ }
+ if (top === '.tmp') return 'an upload that never finished (temp file)';
+ const o = db.get('SELECT id, lifecycle_status FROM media_objects WHERE id = ?', [base]);
+ return o ? `named for object ${o.id} (${o.lifecycle_status}), which records no local copy here` : `named for object ${base}, which does not exist`;
+ }
+ if (base.endsWith('.download')) return 'a restore download that never finished';
+ if (/\.seekable\.(webm|mp4)$/.test(base)) return 'a DVR sidecar no VOD row names';
+ if (/\.master\.mkv$/.test(base)) return recordingNameHint(base.replace(/\.master\.mkv$/, '.webm')) || 'a master archive no VOD row names';
+ return recordingNameHint(base);
+}
+
+/** What the database names in one provider's bucket: { wanted: Set(key), deleted: Map(key → location), rows: Map(key → row) }. */
+function remoteReferences(provider) {
+ const vs = vodStorage();
+ const bucket = vs.bucketFor(provider);
+ const wanted = new Set();
+ const deleted = new Map();
+ for (const l of db.all(`${LOCATION_ROWS} WHERE l.provider = ?`, [provider])) {
+ if (l.bucket && bucket && l.bucket !== bucket) continue; // a copy recorded in another bucket
+ if (locationWanted(l)) wanted.add(l.key); else deleted.set(l.key, l);
+ }
+ const rows = new Map(); // every key a vods/clips row points at, with the row (stale R2 copies are recognised by it)
+ for (const table of ['vods', 'clips']) {
+ for (const r of db.all(`SELECT id, file_path, storage_provider, storage_key FROM ${table} WHERE COALESCE(file_path, '') != ''`)) {
+ const p = vs.providerOf(r);
+ if (p === 'local' && !r.storage_key) continue;
+ const key = vs.keyForVod(r);
+ rows.set(key, { kind: table === 'vods' ? 'vod' : 'clip', id: r.id, provider: p });
+ if (provider === 'b2' || p === 'r2') wanted.add(key);
+ }
+ }
+ return { wanted, deleted, rows };
+}
+
+function remoteHint(provider, key, refs) {
+ if (key.startsWith(PREFIX)) return `parked in ${PREFIX}: scripts/vods-orphans-report.js says what each one is`;
+ const del = refs.deleted.get(key);
+ if (del) return del.object_id ? `copy of deleted object ${del.object_id}${del.legacy_ref ? ` (${del.legacy_ref})` : ''}` : 'named only by a location row whose object does not exist';
+ const row = refs.rows.get(key);
+ if (row && provider === 'r2') return `an R2 copy of ${row.kind} ${row.id}, which is served from ${row.provider}`;
+ const shot = /paste-screenshot-(\d+)\.[a-z0-9]+$/i.exec(key);
+ if (shot) return db.get('SELECT 1 AS x FROM pastes WHERE id = ?', [Number(shot[1])]) ? `a legacy screenshot of paste ${shot[1]}` : `a legacy screenshot of paste ${shot[1]}, whose row is gone`;
+ return recordingNameHint(path.basename(key));
+}
+
+/**
+ * The storage report. Options (all injectable for tests):
+ * providers which buckets to list (default: the configured ones of b2, r2)
+ * list (provider) → [{ key, size, last_modified }] default: every key (ListObjectsV2)
+ * listUploads (provider) → [{ key, upload_id, initiated }] default: ListMultipartUploads
+ * limit items listed per section (totals always count everything), default 2000
+ */
+async function buildStorageReport({ providers = null, list = null, listUploads = null, limit = 2000, now = Date.now() } = {}) {
+ const vs = vodStorage();
+ const provs = providers || ['b2', 'r2'].filter(p => vs.providerConfigured(p));
+ const lister = list || ((p) => vs.listObjects(p, ''));
+ const uploadLister = listUploads || ((p) => vs.listMultipartUploads(p));
+ const cap = Math.max(1, Number(limit) || 2000);
+ const report = {
+ kind: 'media.storage_orphans.report', version: 1, generated_at: new Date(now).toISOString(), read_only: true,
+ scope: { roots: [], providers: {}, walk_truncated: false },
+ totals: {
+ unreferenced_local: { files: 0, bytes: 0, by_root: {} }, unreferenced_remote: {},
+ missing: { locations: 0, by_provider: {} }, multipart_local: { open: 0, expired: 0 }, multipart_remote: {},
+ expected: { live_thumbnails: 0, in_flight: 0 },
+ },
+ truncated: {},
+ note: 'Report only: nothing was deleted, moved or written. Each item is for an operator to look at.',
+ };
+ for (const k of STORAGE_SECTIONS) { report[k] = []; report.truncated[k] = false; }
+ const push = (section, item) => { if (report[section].length < cap) report[section].push(item); else report.truncated[section] = true; };
+
+ // ── Local disk ──
+ const roots = localRoots();
+ const rootDirs = new Set(roots.map(r => r.dir));
+ const walked = { files: [], truncated: false };
+ for (const root of roots) {
+ const exists = fs.existsSync(root.dir);
+ const files = exists ? walkRoot(root, rootDirs, walked) : 0;
+ report.scope.roots.push({ name: root.name, dir: root.dir, exists, files });
+ }
+ report.scope.walk_truncated = walked.truncated;
+ const lrefs = localReferences();
+ const orphanLocal = [];
+ for (const f of walked.files) {
+ const c = classifyLocal(f, lrefs, now);
+ if (c === 'wanted') continue;
+ if (c === 'live_thumbnail') { report.totals.expected.live_thumbnails++; continue; }
+ if (c === 'in_flight') { report.totals.expected.in_flight++; continue; }
+ orphanLocal.push(f);
+ const t = report.totals.unreferenced_local;
+ t.files++; t.bytes += f.size;
+ t.by_root[f.root] = t.by_root[f.root] || { files: 0, bytes: 0 };
+ t.by_root[f.root].files++; t.by_root[f.root].bytes += f.size;
+ }
+ orphanLocal.sort((a, b) => b.size - a.size);
+ for (const f of orphanLocal.slice(0, cap)) push('unreferenced_local', { root: f.root, path: f.path, size: f.size, modified: new Date(f.mtimeMs).toISOString(), hint: localHint(f, lrefs) });
+ if (orphanLocal.length > cap) report.truncated.unreferenced_local = true;
+
+ // ── Buckets ──
+ const listed = new Map(); // provider → Map(key → size)
+ for (const p of provs) {
+ const entry = { configured: vs.providerConfigured(p), bucket: vs.bucketFor(p), listed: false };
+ report.scope.providers[p] = entry;
+ let keys;
+ try { keys = await lister(p); } catch (err) { entry.error = err.message; continue; }
+ entry.listed = true;
+ entry.keys = keys.length;
+ entry.bytes = keys.reduce((n, k) => n + (Number(k.size) || 0), 0);
+ listed.set(p, new Map(keys.map(k => [k.key, k])));
+ const rrefs = remoteReferences(p);
+ const orphans = keys.filter(k => !rrefs.wanted.has(k.key)).sort((a, b) => (Number(b.size) || 0) - (Number(a.size) || 0));
+ report.totals.unreferenced_remote[p] = { keys: orphans.length, bytes: orphans.reduce((n, k) => n + (Number(k.size) || 0), 0) };
+ for (const k of orphans.slice(0, cap)) push('unreferenced_remote', { provider: p, key: k.key, size: Number(k.size) || 0, last_modified: k.last_modified || null, hint: remoteHint(p, k.key, rrefs) });
+ if (orphans.length > cap) report.truncated.unreferenced_remote = true;
+ let uploads = [];
+ try { uploads = await uploadLister(p); } catch (err) { entry.multipart_error = err.message; }
+ report.totals.multipart_remote[p] = uploads.length;
+ for (const u of uploads) push('multipart_remote', { provider: p, key: u.key, upload_id: u.upload_id, initiated: u.initiated || null, key_named_by_a_row: rrefs.wanted.has(u.key) });
+ }
+
+ // ── Copies the database records whose bytes are not there ──
+ for (const l of db.all(`SELECT l.provider, l.bucket, l.key, l.state, l.verified_at, o.id AS object_id, o.app_id, o.kind, o.legacy_ref, o.lifecycle_status
+ FROM media_locations l JOIN media_objects o ON o.id = l.object_id WHERE o.lifecycle_status != 'deleted' ORDER BY l.id`)) {
+ let there;
+ if (l.provider === 'local') {
+ try { there = fs.statSync(l.key).isFile(); } catch { there = false; }
+ } else {
+ const keys = listed.get(l.provider);
+ if (!keys) continue; // not listed: nothing is known
+ if (l.bucket && vs.bucketFor(l.provider) && l.bucket !== vs.bucketFor(l.provider)) continue;
+ there = keys.has(l.key);
+ }
+ if (there) continue;
+ report.totals.missing.locations++;
+ report.totals.missing.by_provider[l.provider] = (report.totals.missing.by_provider[l.provider] || 0) + 1;
+ push('missing', { object_id: l.object_id, app_id: l.app_id, kind: l.kind, legacy_ref: l.legacy_ref || null, lifecycle_status: l.lifecycle_status,
+ provider: l.provider, key: l.key, recorded_state: l.state, verified_at: l.verified_at || null });
+ }
+
+ // ── Media's own multipart sessions still open ──
+ for (const u of db.all(`SELECT u.*, (SELECT COUNT(*) FROM media_upload_parts p WHERE p.upload_id = u.id) AS parts_received,
+ (SELECT COALESCE(SUM(p.size_bytes), 0) FROM media_upload_parts p WHERE p.upload_id = u.id) AS bytes_received,
+ (u.expires_at < datetime('now')) AS expired
+ FROM media_uploads u WHERE u.status IN ('active', 'completing') ORDER BY u.created_at`)) {
+ report.totals.multipart_local.open++;
+ if (u.expired) report.totals.multipart_local.expired++;
+ push('multipart_local', { upload_id: u.id, object_id: u.object_id, app_id: u.app_id, status: u.status, parts_received: u.parts_received,
+ parts_expected: u.parts_expected, bytes_received: u.bytes_received, total_size: u.total_size, created_at: u.created_at, expires_at: u.expires_at, expired: !!u.expired });
+ }
+ return report;
+}
+
+function reportsDir() {
+ return path.join(path.dirname(config.db.path), 'reports');
+}
+
+/** Write a storage report as JSON (default <data>/reports/storage-orphans-<time>.json). Returns the path. */
+function writeStorageReport(report, file = null) {
+ const out = file || path.join(reportsDir(), `storage-orphans-${report.generated_at.replace(/[:.]/g, '-')}.json`);
+ fs.mkdirSync(path.dirname(out), { recursive: true });
+ fs.writeFileSync(out, JSON.stringify(report, null, 1));
+ return out;
+}
+
+/** One line per section for logs and the job result. */
+function summarizeStorage(report) {
+ const t = report.totals;
+ const mb = (b) => `${(b / 1048576).toFixed(1)} MB`;
+ const remote = Object.entries(t.unreferenced_remote).map(([p, x]) => `${p} ${x.keys} key(s) ${mb(x.bytes)}`).join(', ') || 'no bucket listed';
+ return `unreferenced local ${t.unreferenced_local.files} file(s) ${mb(t.unreferenced_local.bytes)}; unreferenced remote: ${remote}; `
+ + `missing copies ${t.missing.locations}; open multipart: ${t.multipart_local.open} here (${t.multipart_local.expired} expired), `
+ + `${Object.entries(t.multipart_remote).map(([p, n]) => `${n} in ${p}`).join(', ') || 'buckets not listed'}`;
+}
+
+module.exports = {
+ PREFIX, RECOMMENDATIONS, parseRecordingName, assess, buildReport,
+ STORAGE_SECTIONS, buildStorageReport, writeStorageReport, summarizeStorage, localRoots,
+};
@@ -393,6 +393,27 @@ async function listObjects(provider, prefix) {
@@ -393,6 +393,27 @@ async function listObjects(provider, prefix) {
return out;
return out;
}
}
+/**
+ * Multipart uploads still open in one provider's bucket (started, never completed or aborted: a killed
+ * process leaves them, and their parts are stored and billed): [{ key, upload_id, initiated }] (read-only).
+ */
+async function listMultipartUploads(provider, prefix = '') {
+ const client = clientFor(provider);
+ if (!client) throw new Error(`${provider} is not configured`);
+ loadSdk();
+ const out = [];
+ let keyMarker, idMarker;
+ for (;;) {
+ const r = await client.send(new S3.ListMultipartUploadsCommand({ Bucket: PROVIDER_ENV[provider].bucket, Prefix: prefix || undefined,
+ KeyMarker: keyMarker, UploadIdMarker: idMarker, MaxUploads: 1000 }));
+ for (const u of (r.Uploads || [])) out.push({ key: u.Key, upload_id: u.UploadId, initiated: u.Initiated ? new Date(u.Initiated).toISOString() : null });
+ if (!r.IsTruncated || !r.NextKeyMarker) break;
+ keyMarker = r.NextKeyMarker;
+ idMarker = r.NextUploadIdMarker;
+ }
+ return out;
+}
+
async function presignGet(provider, key, expiresInSeconds = 900) {
async function presignGet(provider, key, expiresInSeconds = 900) {
const client = clientFor(provider);
const client = clientFor(provider);
if (!client) return null;
if (!client) return null;
@@ -1378,6 +1399,7 @@ module.exports = {
@@ -1378,6 +1399,7 @@ module.exports = {
endpointFor,
endpointFor,
headObject,
headObject,
listObjects,
listObjects,
+ listMultipartUploads,
keyForVod,
keyForVod,
localPathForVod,
localPathForVod,
resolvePlayback,
resolvePlayback,
new file mode 100644
new file mode 100644
@@ -0,0 +1,239 @@
@@ -0,0 +1,239 @@
+'use strict';
+// The storage orphan report (server/vod/orphans-report.js buildStorageReport; job storage.orphans.scan;
+// scripts/vods-orphans-report.js --storage): files under the data directories and B2/R2 keys that no row
+// names, copies the database records that are not there, and multipart uploads left open, here and in the
+// buckets. Report only: the test snapshots every file, the tables and the buckets before and after and
+// finds them unchanged, and a fake S3 (spoken to by the real AWS SDK) counts any write or delete. The job
+// runs service-wide under queue.SYSTEM_APP, is scheduled every MEDIA_ORPHAN_SCAN_DAYS, and tenants cannot
+// enqueue it.
+const assert = require('assert');
+const fs = require('fs');
+const os = require('os');
+const path = require('path');
+const http = require('http');
+const crypto = require('crypto');
+const { execFileSync } = require('child_process');
+
+const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-orphans-'));
+const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+
+const buckets = { 'b2-bucket': new Map(), 'r2-bucket': new Map() };
+const openUploads = { 'b2-bucket': [{ key: 'vods/half-uploaded.webm', id: 'up-b2-1' }], 'r2-bucket': [] };
+let writes = 0;
+const xml = (v) => String(v).replace(/&/g, '&').replace(/</g, '<');
+const s3 = http.createServer((req, res) => {
+ req.resume();
+ req.on('end', () => {
+ const u = new URL(req.url, 'http://x');
+ const [, bucket, ...rest] = decodeURIComponent(u.pathname).split('/');
+ const key = rest.join('/');
+ const store = buckets[bucket];
+ if (!store) { res.writeHead(404); return res.end(); }
+ if (req.method !== 'GET' && req.method !== 'HEAD') { writes++; res.writeHead(403, { 'Content-Type': 'application/xml' }); return res.end('<Error><Code>AccessDenied</Code></Error>'); }
+ if (!key && u.searchParams.has('uploads')) {
+ res.writeHead(200, { 'Content-Type': 'application/xml' });
+ return res.end(`<?xml version="1.0" encoding="UTF-8"?><ListMultipartUploadsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Bucket>${bucket}</Bucket><IsTruncated>false</IsTruncated>${
+ openUploads[bucket].map(x => `<Upload><Key>${xml(x.key)}</Key><UploadId>${x.id}</UploadId><Initiated>2026-09-01T00:00:00.000Z</Initiated></Upload>`).join('')}</ListMultipartUploadsResult>`);
+ }
+ if (!key && u.searchParams.get('list-type') === '2') {
+ const prefix = u.searchParams.get('prefix') || '';
+ const keys = [...store.keys()].filter(k => k.startsWith(prefix)).sort();
+ res.writeHead(200, { 'Content-Type': 'application/xml' });
+ return res.end(`<?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>${bucket}</Name><Prefix>${xml(prefix)}</Prefix><KeyCount>${keys.length}</KeyCount><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated>${
+ keys.map(k => `<Contents><Key>${xml(k)}</Key><LastModified>2026-09-21T00:00:00.000Z</LastModified><ETag>"e"</ETag><Size>${store.get(k).length}</Size><StorageClass>STANDARD</StorageClass></Contents>`).join('')}</ListBucketResult>`);
+ }
+ const obj = store.get(key);
+ if (!obj) { res.writeHead(404); return res.end(); }
+ res.writeHead(200, { 'Content-Length': obj.length, ETag: '"e"' });
+ res.end(req.method === 'GET' ? obj : undefined);
+ });
+});
+
+(async () => {
+ await new Promise((r) => s3.listen(0, '127.0.0.1', r));
+ const endpoint = `http://127.0.0.1:${s3.address().port}`;
+ const env = {
+ DB_PATH: path.join(tmp, 'data', 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
+ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), ASSETS_PATH: dir('assets'),
+ MEDIA_B2_ENDPOINT: endpoint, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_KEY_ID: 'k', MEDIA_B2_APP_KEY: 's', MEDIA_B2_REGION: 'us-west-004',
+ MEDIA_R2_ENDPOINT: endpoint, MEDIA_R2_BUCKET: 'r2-bucket', MEDIA_R2_ACCESS_KEY_ID: 'k', MEDIA_R2_SECRET_ACCESS_KEY: 's',
+ AWS_REQUEST_CHECKSUM_CALCULATION: 'WHEN_REQUIRED', AWS_RESPONSE_CHECKSUM_VALIDATION: 'WHEN_REQUIRED',
+ MEDIA_PUBLIC_URL: 'https://media.test',
+ };
+ Object.assign(process.env, env);
+ delete process.env.MEDIA_ORPHAN_SCAN_DAYS;
+ const db = require('../server/db/database');
+ const model = require('../server/objects/model');
+ const queue = require('../server/jobs/queue');
+ require('../server/jobs/types');
+ const orphans = require('../server/vod/orphans-report');
+ const storageJob = require('../server/jobs/storage-orphans');
+ db.upsertApp({ app_id: 'live', api_key: 'live-key-orphans' });
+ const raw = db.getDb();
+ const write = (d, name, bytes = 16, ageMs = 0) => {
+ const p = path.join(d, name);
+ fs.mkdirSync(path.dirname(p), { recursive: true });
+ fs.writeFileSync(p, Buffer.alloc(bytes, 7));
+ if (ageMs) { const t = (Date.now() - ageMs) / 1000; fs.utimesSync(p, t, t); }
+ return p;
+ };
+ const HOUR = 3600 * 1000;
+
+ // ── Fixtures: what the database names ──
+ const ins = raw.prepare(`INSERT INTO vods (id, app_id, title, file_path, file_size, is_public, visibility, duration_seconds, storage_provider, storage_key, thumbnail_url)
+ VALUES (?, 'live', ?, ?, ?, 1, 'public', 60, ?, ?, ?)`);
+ ins.run(1, 'local', write(env.VOD_PATH, 'vod-live-1-1758000000001.webm', 40), 40, 'local', null, '/t/vod-1-111.jpg');
+ write(env.THUMBNAILS_PATH, 'vod-1-111.jpg');
+ ins.run(2, 'in B2', '/old/vod-live-2-1758000000002.webm', 20, 'b2', 'vods/vod-live-2-1758000000002.webm', null);
+ buckets['b2-bucket'].set('vods/vod-live-2-1758000000002.webm', Buffer.alloc(20));
+ ins.run(3, 'served from R2', '/old/vod-live-3-1758000000003.webm', 30, 'r2', 'vods/vod-live-3-1758000000003.webm', null);
+ buckets['b2-bucket'].set('vods/vod-live-3-1758000000003.webm', Buffer.alloc(30));
+ buckets['r2-bucket'].set('vods/vod-live-3-1758000000003.webm', Buffer.alloc(30));
+ ins.run(4, 'back in B2, R2 copy left', '/old/vod-live-4-1758000000004.webm', 50, 'b2', 'vods/vod-live-4-1758000000004.webm', null);
+ buckets['b2-bucket'].set('vods/vod-live-4-1758000000004.webm', Buffer.alloc(50));
+ buckets['r2-bucket'].set('vods/vod-live-4-1758000000004.webm', Buffer.alloc(50));
+ ins.run(5, 'local file lost', path.join(env.VOD_PATH, 'vod-live-5-1758000000005.webm'), 10, 'local', null, null);
+ ins.run(6, 'B2 key lost', '/old/vod-live-6-1758000000006.webm', 60, 'b2', 'vods/vod-live-6-1758000000006.webm', null);
+ for (const id of [1, 2, 3, 4, 5, 6]) model.sync('vod', id);
+ raw.prepare(`INSERT INTO files (key, app_id, original_name, size, mime) VALUES ('abc123-notes.txt', 'live', 'notes.txt', 16, 'text/plain')`).run();
+ write(path.join(env.FILES_PATH, 'live'), 'abc123-notes.txt');
+ model.sync('file', 'abc123-notes.txt');
+ // A native object soft-deleted inside its retention period keeps its bytes wanted.
+ const kept = model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', visibility: 'private', size_bytes: 16 });
+ const keptPath = write(path.join(env.OBJECTS_PATH, 'live'), kept, 16);
+ model.upsertLocation(kept, { provider: 'local', key: keptPath, state: 'present', size_bytes: 16, verified: true });
+ model.softDelete(model.getObject(kept));
+ // A clip deleted by its row: the object is deleted, its file stayed behind.
+ raw.prepare(`INSERT INTO clips (id, app_id, vod_id, title, file_path, status) VALUES (21, 'live', 1, 'gone clip', ?, 'ready')`).run(write(env.CLIPS_PATH, 'clip-live-21.webm', 12));
+ model.sync('clip', 21);
+ const clipObj = db.get('SELECT object_id FROM clips WHERE id = 21').object_id;
+ raw.prepare('DELETE FROM clips WHERE id = 21').run();
+ // Media's own multipart sessions: one open and expired (its parts wanted), one expired session's parts dir with no session.
+ raw.prepare(`INSERT INTO media_uploads (id, object_id, app_id, part_size, total_size, parts_expected, status, expires_at)
+ VALUES ('mup_OPEN', ?, 'live', 8, 16, 2, 'active', datetime('now', '-1 hour'))`).run(kept);
+ raw.prepare("INSERT INTO media_upload_parts (upload_id, part_number, size_bytes, sha256) VALUES ('mup_OPEN', 1, 8, 'x')").run();
+ write(path.join(env.OBJECTS_PATH, '.parts', 'mup_OPEN'), '1', 8, 3 * HOUR);
+
+ // ── What storage holds that no row names ──
+ write(env.VOD_PATH, 'vod-live-99-1758000000099.webm', 1000); // the recorder's name; row 99 never existed
+ write(env.VOD_PATH, 'mystery.bin', 500);
+ write(env.VOD_PATH, 'vod-live-1-1758000000001.seekable.webm', 5); // a sidecar of vod 1: wanted
+ write(env.THUMBNAILS_PATH, 'stream-live-1.jpg'); // a live thumbnail: expected
+ write(path.join(env.OBJECTS_PATH, '.tmp'), 'med_OLD-abcd', 30, 3 * HOUR); // an upload that never finished
+ write(path.join(env.OBJECTS_PATH, '.tmp'), 'med_NOW-abcd', 30); // one in flight right now
+ write(path.join(env.OBJECTS_PATH, '.parts', 'mup_GONE'), '1', 8, 3 * HOUR); // parts with no session
+ buckets['b2-bucket'].set('vods/stray.webm', Buffer.alloc(70));
+ buckets['b2-bucket'].set('vods-orphans/vod-live-7-1758000000007.webm', Buffer.alloc(80));
+
+ // ── Everything before, to prove nothing changes ──
+ const snapshotFiles = () => {
+ const out = [];
+ const walk = (d) => { for (const e of fs.readdirSync(d, { withFileTypes: true })) { const p = path.join(d, e.name); if (e.isDirectory()) { if (p !== path.join(tmp, 'data')) walk(p); } else out.push(`${path.relative(tmp, p)}:${fs.statSync(p).size}:${fs.statSync(p).mtimeMs}`); } };
+ walk(tmp);
+ return out.sort();
+ };
+ const TABLES = ['vods', 'clips', 'files', 'pastes', 'media_objects', 'media_locations', 'media_holds', 'media_uploads', 'media_upload_parts'];
+ const snapshotDb = () => Object.fromEntries(TABLES.map(t => [t, crypto.createHash('sha256').update(JSON.stringify(raw.prepare(`SELECT * FROM ${t} ORDER BY rowid`).all())).digest('hex')]));
+ const snapshotBuckets = () => JSON.stringify(Object.entries(buckets).map(([b, m]) => [b, [...m.keys()].sort()]));
+ const before = { files: snapshotFiles(), db: snapshotDb(), buckets: snapshotBuckets() };
+
+ const report = await orphans.buildStorageReport({ limit: 50 });
+
+ // ── Unreferenced local files ──
+ const local = Object.fromEntries(report.unreferenced_local.map(f => [`${f.root}/${f.path.split(path.sep).join('/')}`, f]));
+ assert.deepStrictEqual(Object.keys(local).sort(), [
+ 'clips/clip-live-21.webm', 'objects/.parts/mup_GONE/1', 'objects/.tmp/med_OLD-abcd', 'vods/mystery.bin', 'vods/vod-live-99-1758000000099.webm',
+ ], 'exactly the files no row names');
+ assert.ok(/whose row is gone/.test(local['vods/vod-live-99-1758000000099.webm'].hint), 'a recorder name says whose recording it was');
+ assert.strictEqual(local['clips/clip-live-21.webm'].hint, `bytes of deleted object ${clipObj} (legacy:live:clip:21)`);
+ assert.ok(/no session/.test(local['objects/.parts/mup_GONE/1'].hint));
+ assert.ok(/never finished/.test(local['objects/.tmp/med_OLD-abcd'].hint));
+ assert.strictEqual(local['vods/mystery.bin'].hint, null);
+ assert.strictEqual(report.unreferenced_local[0].path, 'vod-live-99-1758000000099.webm', 'largest first');
+ assert.deepStrictEqual([report.totals.unreferenced_local.files, report.totals.unreferenced_local.bytes], [5, 1000 + 500 + 30 + 8 + 12]);
+ assert.deepStrictEqual(report.totals.expected, { live_thumbnails: 1, in_flight: 1 }, 'live thumbnails and uploads in flight are expected, not orphans');
+ console.log('✅ unreferenced local files: recordings without a row, strays, bytes of deleted objects, stale temps and parts');
+
+ // ── Unreferenced bucket keys ──
+ const remote = Object.fromEntries(report.unreferenced_remote.map(k => [`${k.provider}:${k.key}`, k]));
+ assert.deepStrictEqual(Object.keys(remote).sort(), ['b2:vods-orphans/vod-live-7-1758000000007.webm', 'b2:vods/stray.webm', 'r2:vods/vod-live-4-1758000000004.webm']);
+ assert.ok(/vods-orphans-report/.test(remote['b2:vods-orphans/vod-live-7-1758000000007.webm'].hint), 'parked recordings point at the prefix report');
+ assert.strictEqual(remote['r2:vods/vod-live-4-1758000000004.webm'].hint, 'an R2 copy of vod 4, which is served from b2');
+ assert.deepStrictEqual(report.totals.unreferenced_remote, { b2: { keys: 2, bytes: 150 }, r2: { keys: 1, bytes: 50 } });
+ assert.deepStrictEqual([report.scope.providers.b2.listed, report.scope.providers.b2.keys, report.scope.providers.r2.keys], [true, 5, 2]);
+ console.log('✅ unreferenced bucket keys: strays, parked recordings, an R2 copy left behind by a demotion');
+
+ // ── Copies the database records that are not there ──
+ const missing = report.missing.map(m => `${m.legacy_ref}:${m.provider}`).sort();
+ assert.deepStrictEqual(missing, ['legacy:live:vod:5:local', 'legacy:live:vod:6:b2'], 'a lost local file and a lost B2 key');
+ assert.deepStrictEqual(report.totals.missing, { locations: 2, by_provider: { local: 1, b2: 1 } });
+ console.log('✅ missing copies: rows whose location is not in storage');
+
+ // ── Multipart uploads left open ──
+ assert.deepStrictEqual(report.multipart_local.map(u => [u.upload_id, u.expired, u.parts_received]), [['mup_OPEN', true, 1]]);
+ assert.deepStrictEqual(report.totals.multipart_local, { open: 1, expired: 1 });
+ assert.deepStrictEqual(report.multipart_remote.map(u => [u.provider, u.key, u.upload_id, u.key_named_by_a_row]), [['b2', 'vods/half-uploaded.webm', 'up-b2-1', false]]);
+ assert.deepStrictEqual(report.totals.multipart_remote, { b2: 1, r2: 0 });
+ console.log('✅ open multipart uploads: Media\'s expired sessions and the buckets\' own');
+
+ // ── Report only ──
+ assert.deepStrictEqual(snapshotFiles(), before.files, 'no file was deleted, moved or touched');
+ assert.deepStrictEqual(snapshotDb(), before.db, 'no row changed');
+ assert.strictEqual(snapshotBuckets(), before.buckets, 'no bucket key changed');
+ assert.strictEqual(writes, 0, 'no PUT, POST or DELETE reached a bucket');
+ const src = fs.readFileSync(path.join(__dirname, '..', 'server', 'vod', 'orphans-report.js'), 'utf8');
+ assert.ok(!/unlink|rmSync|rmdirSync|renameSync|DeleteObject|deleteObject|uploadFile|copyBetweenProviders|\b(DELETE|UPDATE|INSERT)\s/.test(src), 'the report module holds no delete, move or write');
+ console.log('✅ report only: files, rows and buckets are unchanged; the module has no delete, move or write');
+
+ // ── The job: service-wide, scheduled, not for tenants ──
+ assert.strictEqual(storageJob.schedule(Date.now()), 1, 'the monthly scan is queued');
+ assert.strictEqual(storageJob.schedule(Date.now()), 0, 'once per period');
+ process.env.MEDIA_ORPHAN_SCAN_DAYS = '0';
+ assert.strictEqual(storageJob.schedule(Date.now() + 90 * 86400 * 1000), 0, 'MEDIA_ORPHAN_SCAN_DAYS=0 schedules nothing');
+ delete process.env.MEDIA_ORPHAN_SCAN_DAYS;
+ const job = db.get("SELECT * FROM media_jobs WHERE job_type = 'storage.orphans.scan'");
+ assert.deepStrictEqual([job.app_id, job.status, job.created_by], [queue.SYSTEM_APP, 'queued', 'system:schedule']);
+ const done = await require('../server/jobs/worker').runNow(job.id);
+ assert.strictEqual(done.status, 'succeeded', done.error || '');
+ const result = JSON.parse(done.result);
+ assert.strictEqual(result.totals.unreferenced_local.files, 5);
+ const reportFile = path.join(tmp, 'data', 'reports', result.report);
+ assert.ok(fs.existsSync(reportFile), 'the job writes its report under <data>/reports');
+ assert.strictEqual(JSON.parse(fs.readFileSync(reportFile, 'utf8')).job_id, job.id);
+ assert.deepStrictEqual(snapshotFiles(), before.files, 'the job deleted nothing either');
+ assert.strictEqual(writes, 0);
+
+ const express = require('express');
+ const app = express();
+ app.use(express.json());
+ app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));
+ const server = await new Promise((r) => { const s = app.listen(0, '127.0.0.1', () => r(s)); });
+ const post = (p, body) => new Promise((resolve, reject) => {
+ const data = JSON.stringify(body);
+ const rq = http.request({ host: '127.0.0.1', port: server.address().port, path: p, method: 'POST',
+ headers: { Authorization: 'Bearer live-key-orphans', 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(data) } },
+ (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: JSON.parse(b || '{}') })); });
+ rq.on('error', reject);
+ rq.end(data);
+ });
+ const r = await post('/api/v2/live/jobs', { type: 'storage.orphans.scan' });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'media.job.forbidden'], 'a tenant cannot run the service-wide scan');
+ server.close();
+ console.log('✅ storage.orphans.scan: scheduled once per period under the system app, writes its report, refused to tenants');
+
+ // ── The script, on demand ──
+ const out = path.join(tmp, 'script-report.json');
+ const text = execFileSync(process.execPath, [path.join(__dirname, '..', 'scripts', 'vods-orphans-report.js'), '--storage', '--no-remote', '--out', out],
+ { env: { ...process.env, ...env }, encoding: 'utf8' });
+ assert.ok(/Unreferenced local files: 5/.test(text) && /Nothing was deleted or moved/.test(text), text);
+ const scripted = JSON.parse(fs.readFileSync(out, 'utf8'));
+ assert.deepStrictEqual([scripted.totals.unreferenced_local.files, Object.keys(scripted.scope.providers).length], [5, 0], '--no-remote lists no bucket');
+ assert.deepStrictEqual(snapshotFiles().filter(f => !f.startsWith('script-report.json')), before.files);
+ console.log('✅ scripts/vods-orphans-report.js --storage runs the same report on demand');
+
+ s3.close();
+ db.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('storage orphans: all checks passed');
+ process.exit(0);
+})().catch((err) => { console.error(err); process.exit(1); });