openvibers/openvibe.media Commits

Storage orphan report (WS-G task 8), report only: server/vod/orphans-report.js buildStorageReport compares what storage holds with what the database names and lists files under the data directories and B2/R2 keys that no row names (with hints: a recorder name whose VOD row is gone, bytes of a deleted object, an R2 copy left after its VOD went back to B2, parked vods-orphans/ keys, parts of a session that no longer exists, a temp of an upload that never finished), copies the database records for live objects that are not there (no local file, a key absent from its bucket's listing), and multipart uploads left open (media_uploads sessions, expired ones flagged, and the buckets' own via ListMultipartUploads, new read-only vodStorage.listMultipartUploads). Live thumbnails and temps younger than an hour are counted as expected. It never deletes, moves or writes a file, an object or a row; it writes <data>/reports/storage-orphans-<time>.json. The job storage.orphans.scan runs it every MEDIA_ORPHAN_SCAN_DAYS (30; 0 = never) from the jobs worker (off in a restore drill) under the system app _media: no apps row, no media.job events, and POST /api/v2/:app/jobs refuses it (403 media.job.forbidden). On demand: scripts/vods-orphans-report.js --storage [--no-remote]. test/storage-orphans.test.js snapshots files, tables and buckets before and after, and its fake S3 refuses and counts any write

OP OpenVibers committed 25 Sept 2026 · 95a449d024 · 12 files, +766 −19


Changes

ddiff --git a/README.md b/README.md
iindex da32bb4..598fb5c 100644
---- a/README.md
++++ b/README.md
@@@ -52,7 +52,8 @@ server/
cancellation), routes (/api/v2/:app/jobs), types: thumbnail.regenerate,
invariant.scan (proposes split/remux), object.split, object.remux,
vod.finalize (orphans and failed finalizes, with backoff),
-- vod.duration.reconcile (stored vs measured durations, local and B2/R2)
++ vod.duration.reconcile (stored vs measured durations, local and B2/R2),
++ storage.orphans.scan (monthly storage orphan report, service-wide, report only)
client-ip.js trust proxy = loopback; req.ip is the only client address
(openvibe-shared v1.5.1, openvibe-contracts v0.33.0, openvibe-sdk v0.5.0: pinned release tarballs, installed by npm)
scripts/smoke-test.sh end-to-end smoke test (boots a temp instance)
@@@ -60,7 +61,9 @@ scripts/backfill-objects.js / reconcile-objects.js / object-invariant.js / no-go
scripts/media-jobs.js list jobs, run the size-invariant scan (dry run by default), approve/cancel proposals
scripts/r2-eviction-drill.js evict one VOD's R2 copy, prove B2 serves it, re-warm R2; JSON artifact (dry run by default)
scripts/vod-duration-reconcile.js stored VOD durations vs the real files (dry run by default; --apply --backup; --rollback)
--scripts/vods-orphans-report.js read-only report on the B2 vods-orphans/ prefix (matches, sizes, recommendations)
++scripts/vods-orphans-report.js read-only report on the B2 vods-orphans/ prefix (matches, sizes, recommendations);
++ --storage: the whole storage against the database (files and keys no row names,
++ copies that are not there, open multipart uploads), as the monthly job does
```
## Visitor sign-in
@@@ -410,6 +413,10 @@ model current. Full reference: **[docs/object-model.md](docs/object-model.md)**.
read-only by default; `--verify` HEADs B2/R2 and records location states.
Reports missing canonical copies, lost local files, size/hash mismatches,
orphan locations, deleted-but-still-served objects and unprojected rows.
++- **Orphan report** — monthly job `storage.orphans.scan` (`MEDIA_ORPHAN_SCAN_DAYS`, 30) and
++ `node scripts/vods-orphans-report.js --storage [--no-remote]`: local files and B2/R2 keys no row
++ names, recorded copies that are not there, open multipart uploads (Media's and the buckets').
++ Report only: it never deletes anything ([docs](docs/object-model.md#storage-orphan-report)).
- **Public object-size invariant** — `MEDIA_PUBLIC_OBJECT_MAX_MB` (500) /
`_TARGET_MB` (256) / `_WARN_MB` (384): v2 refuses oversized public playback
uploads; `node scripts/object-invariant.js` lists offenders and records them in
ddiff --git a/docs/object-model.md b/docs/object-model.md
iindex 42728b8..990ffbf 100644
---- a/docs/object-model.md
++++ b/docs/object-model.md
@@@ -367,9 +367,45 @@ The exit code is 1 when issues were found.
| `missing_projection` | an inherited row with no object yet (run the backfill) |
| `incomplete_multipart` | a multipart session still open past its expiry (the hourly purge should have removed it) |
--**Not covered yet:** orphan *provider* objects (bucket keys with no row, which needs bucket
--listing). `server/objects/reconcile.js` takes a `head` function,
--so tests run it against a fake provider.
++Bucket keys and local files that no row names are the [storage orphan report](#storage-orphan-report)'s.
++`server/objects/reconcile.js` takes a `head` function, so tests run it against a fake provider.
++
++## Storage orphan report
++
++Code: `server/vod/orphans-report.js` (`buildStorageReport`), the job `storage.orphans.scan`
++(`server/jobs/storage-orphans.js`), `scripts/vods-orphans-report.js --storage`. Test: `test/storage-orphans.test.js`
++(a fake S3 through the real SDK that refuses and counts any write).
++
++It compares what storage holds with what the database names, and **only reports**: it walks the data
++directories, lists the B2/R2 buckets (keys and open multipart uploads), reads the database and writes
++`<data>/reports/storage-orphans-<time>.json`. It never deletes, moves or changes a file, a bucket object
++or a row.
++
++| section | what it lists |
++|---|---|
++| `unreferenced_local` | files under `VOD_PATH`, `CLIPS_PATH`, `THUMBNAILS_PATH`, `FILES_PATH`, `PASTES_PATH`, `OBJECTS_PATH`, `ASSETS_PATH` that no row names (`media_locations`, `vods` with their sidecars, masters and chunk segments, `clips`, `files`, `pastes`, `assets`, open upload sessions, unfinished jobs). A file whose only row is a deleted object's location counts, unless that object is native and inside its retention period, or held. Largest first, with a hint: a recorder name whose VOD row is gone, bytes of a deleted object, parts of a session that no longer exists, a temp file of an upload that never finished |
++| `unreferenced_remote` | B2/R2 keys that no `media_locations` row and no `vods`/`clips` row names; hints for `vods-orphans/` (see below), an R2 copy left behind after its VOD went back to B2, legacy paste screenshots, recorder names |
++| `missing` | copies the database records for objects that are not deleted, whose bytes are not there: no local file, or a key absent from its bucket's listing |
++| `multipart_local` | Media's own multipart sessions still open (`media_uploads` active or completing), with `expired` when past `expires_at` |
++| `multipart_remote` | multipart uploads the buckets still hold open (a killed upload leaves its parts stored and billed) |
++
++Totals count everything; each list holds at most `limit` items (2000). Live thumbnails
++(`stream-<app>-<id>.jpg`) and upload temps or partial downloads younger than an hour are counted as
++`expected`, not listed. A provider that is not configured, or whose listing fails, is marked not listed
++and nothing is concluded about its keys.
++
++**When it runs.** The job worker queues `storage.orphans.scan` every `MEDIA_ORPHAN_SCAN_DAYS` (30; `0`
++= never). Storage is shared by every tenant, so the job belongs to none: it runs under the system app
++`_media` (no apps row, so no API credential reaches it), announces no `media.job.*` event, and
++`POST /api/v2/:app/jobs` refuses it with 403 `media.job.forbidden`. The worker does not start in a
++restore drill. On demand:
++
++```
++node scripts/vods-orphans-report.js --storage [--no-remote] [--limit 2000] [--out report.json] [--json] [--db ./data/media.db]
++```
++
++`--no-remote` lists no bucket. The detailed per-object recommendations for the `vods-orphans/` prefix
++stay with the prefix report below (`scripts/vods-orphans-report.js` without `--storage`).
## Scheduled verification
@@@ -509,6 +545,7 @@ requeued (or failed when out of attempts). Handlers checkpoint progress and resu
| `object.remux` | heavy | Stream-copy remux of the whole source (seek index, duration, MP4 faststart) into one new private object; also the source's `remux` variant |
| `object.hash` | light | sha256 of ready objects' present local copies that have no hash yet, onto the object (`content_hash`, `metadata.hash_basis`) and the local location (`checksum`). Tenant-wide batch (`params { limit (1-500, 50), budget_mb }`) or one object. Scheduled per tenant every `MEDIA_HASH_INTERVAL_MIN`. Result `{ hashed, bytes, skipped, remaining }`. See [Scheduled verification](#scheduled-verification) |
| `vod.duration.reconcile` | heavy | One bounded batch of the tenant's finished VODs: stored duration vs a measurement of the real file, local or the B2/R2 copy (ffprobe over a presigned URL, ranged reads). `params { limit (1-200, 25), after_id?, apply (false), confirm_remote (false) }`; without `after_id` it walks the library from where the last run stopped. Repairs only confirmed, clearly wrong values when `apply`. Result `{ counts, range, next_after_id, report }`. See [Duration reconciliation](#duration-reconciliation) |
++| `storage.orphans.scan` | light | The [storage orphan report](#storage-orphan-report), service-wide, under the system app `_media`; report only. `params { limit (1-100000, 2000) }`. Result `{ totals, providers, summary, report }` (the report file under `<data>/reports/`). Scheduled every `MEDIA_ORPHAN_SCAN_DAYS`; tenants cannot enqueue it (403) |
| `vod.finalize` | finalize | Finalizes a recording whose finalize failed or never ran. `params { vod_id }`. Queued by the orphan sweep (a row still `is_recording = 1` that no recorder, chunk upload or finalize holds and whose file has been idle for `MEDIA_FINALIZE_ORPHAN_GRACE_S`, 180 s; at boot and every `MEDIA_FINALIZE_SWEEP_S`, 120 s) and by finalize itself when it could not settle a recording (nothing measurable, a stat failure, a throw; first run after 5 min). Still unmeasurable: retried (5 min, 15 min, 45 min, 2 h 15, 6 h; 6 attempts), then the VOD stays `needs_review`, hidden. Never stops a live recording. Result `{ outcome: ready \| needs_review \| corrupt \| deleted \| skipped, duration_seconds, duration_source }` |
Split and remux refuse a source that is not ready, a tenant quota the output would exceed, and too
@@@ -634,6 +671,7 @@ Not run against production yet: it needs the owner's go-ahead.
| `MEDIA_FINALIZE_ORPHAN_GRACE_S` / `MEDIA_FINALIZE_SWEEP_S` | 180 / 120 | an orphaned recording's file must be idle this long; the orphan sweep runs this often |
| `MEDIA_HASH_INTERVAL_MIN` | 15 | queue `object.hash` per tenant this often (`0` = only on demand) |
| `MEDIA_HASH_BUDGET_MB` / `_MAX_FILE_MB` / `_SETTLE_S` | 4096 / 20480 / 120 | bytes read per run; largest file hashed; a file changed this recently waits |
++| `MEDIA_ORPHAN_SCAN_DAYS` | 30 | queue the storage orphan report this often (`0` = only on demand) |
| `MEDIA_DURATION_RECONCILE_HOURS` | 0 | schedule `vod.duration.reconcile` per tenant this often (`0` = only on demand) |
| `MEDIA_DURATION_RECONCILE_APPLY` / `_BATCH` | off / 25 | scheduled runs repair (`1`) or only report; VODs per scheduled run |
| `MEDIA_JOBS_HEAVY_WHILE_RECORDING` | off | `1` lets split/remux run while a recording is being written |
@@@ -657,6 +695,5 @@ Not run against production yet: it needs the owner's go-ahead.
- Copy/move aliases, lifecycle rules and an S3-compatible façade.
- Clip cutting and the finalize remux still run inline, not as jobs (thumbnail regeneration was the first
operation moved onto the job system).
--- Orphan-bucket-object detection.
- The segment-native timeline.
- App assets (emotes and sounds, the `assets` table) are not projected yet. The `asset` kind exists for v2 uploads.
ddiff --git a/scripts/vods-orphans-report.js b/scripts/vods-orphans-report.js
iindex baeb4fe..0fc431e 100755
---- a/scripts/vods-orphans-report.js
++++ b/scripts/vods-orphans-report.js
@@@ -1,15 +1,23 @@
#!/usr/bin/env node
'use strict';
/**
-- * Report on the B2 `vods-orphans/` prefix (hazard H3). Read-only: it lists the prefix (ListObjectsV2),
-- * HEADs each match's canonical copy, reads the database, and writes a report. It never deletes, moves,
-- * copies or writes an object or a row (server/vod/orphans-report.js has the rules).
++ * Orphan reports (server/vod/orphans-report.js has the rules). Read-only: they list, HEAD and read the
++ * database, and write a report file. They never delete, move, copy or write an object, a file or a row.
*
* node scripts/vods-orphans-report.js [--prefix vods-orphans/] [--provider b2] [--out <report.json>] [--json] [--db <media.db>]
*
-- * Prints one line per object (key, size, matched VOD/clip, recommendation) and the totals per
-- * recommendation, and writes the full report (default <data>/reports/vods-orphans-<time>.json).
-- * B2 credentials come from the environment (/etc/openvibe/media.env names). Exit 0 done, 1 error, 2 usage.
++ * The B2 `vods-orphans/` prefix (hazard H3): lists the prefix (ListObjectsV2), HEADs each match's
++ * canonical copy, prints one line per object (key, size, matched VOD/clip, recommendation) and the
++ * totals per recommendation, and writes the full report (default <data>/reports/vods-orphans-<time>.json).
++ *
++ * node scripts/vods-orphans-report.js --storage [--no-remote] [--limit 2000] [--out <report.json>] [--json] [--db <media.db>]
++ *
++ * The whole storage against the database (the storage.orphans.scan job, on demand): files under the data
++ * directories and B2/R2 keys that no row names, copies the database records that are not there, and
++ * multipart uploads left open (Media's sessions and the buckets'). --no-remote lists no bucket. Prints
++ * the totals and the largest items, and writes <data>/reports/storage-orphans-<time>.json.
++ *
++ * B2/R2 credentials come from the environment (/etc/openvibe/media.env names). Exit 0 done, 1 error, 2 usage.
*/
const fs = require('fs');
const path = require('path');
@@@ -18,7 +26,8 @@ const args = process.argv.slice(2);
const has = (name) => args.includes(`--${name}`);
const arg = (name) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] && !args[i + 1].startsWith('--') ? args[i + 1] : null; };
const usage = (msg) => { console.error(`vods-orphans-report: ${msg}`); process.exit(2); };
--for (const flag of ['prefix', 'provider', 'out', 'db']) if (has(flag) && !arg(flag)) usage(`--${flag} needs a value`);
++for (const flag of ['prefix', 'provider', 'out', 'db', 'limit']) if (has(flag) && !arg(flag)) usage(`--${flag} needs a value`);
++if (has('limit') && !(Number.isInteger(Number(arg('limit'))) && Number(arg('limit')) > 0)) usage('--limit must be a positive integer');
if (arg('db')) process.env.DB_PATH = path.resolve(arg('db'));
const provider = arg('provider') || 'b2';
if (!['b2', 'r2'].includes(provider)) usage('--provider must be b2 or r2');
@@@ -32,7 +41,32 @@ const JSON_OUT = has('json');
const log = (...a) => { if (!JSON_OUT) console.log(...a); };
const mb = (b) => `${(b / 1048576).toFixed(1)} MB`;
++async function storageReport() {
++ const providers = has('no-remote') ? [] : null;
++ const report = await orphans.buildStorageReport({ providers, limit: has('limit') ? Number(arg('limit')) : 2000 });
++ const file = orphans.writeStorageReport(report, arg('out') ? path.resolve(arg('out')) : null);
++ const t = report.totals;
++ log(`Storage orphan report (read-only), ${report.generated_at}`);
++ for (const r of report.scope.roots) log(` local ${r.name.padEnd(11)} ${r.dir}${r.exists ? ` (${r.files} file(s))` : ' (missing)'}`);
++ for (const [p, e] of Object.entries(report.scope.providers)) log(` ${p.padEnd(17)} ${e.listed ? `${e.bucket || ''}: ${e.keys} key(s), ${mb(e.bytes)}` : `not listed${e.error ? `: ${e.error}` : ''}`}`);
++ if (has('no-remote')) log(' buckets not listed (--no-remote)');
++ log('');
++ log(`Unreferenced local files: ${t.unreferenced_local.files} (${mb(t.unreferenced_local.bytes)})`);
++ for (const f of report.unreferenced_local.slice(0, 25)) log(` ${mb(f.size).padStart(11)} ${f.root}/${f.path}${f.hint ? ` — ${f.hint}` : ''}`);
++ for (const [p, x] of Object.entries(t.unreferenced_remote)) log(`Unreferenced ${p} keys: ${x.keys} (${mb(x.bytes)})`);
++ for (const k of report.unreferenced_remote.slice(0, 25)) log(` ${mb(k.size).padStart(11)} ${k.provider}:${k.key}${k.hint ? ` — ${k.hint}` : ''}`);
++ log(`Copies the database records that are not there: ${t.missing.locations}`);
++ for (const m of report.missing.slice(0, 25)) log(` ${m.object_id} ${m.legacy_ref || m.kind} ${m.provider} (recorded ${m.recorded_state})`);
++ log(`Open multipart uploads: ${t.multipart_local.open} here (${t.multipart_local.expired} expired)${Object.entries(t.multipart_remote).map(([p, n]) => `, ${n} in ${p}`).join('')}`);
++ log(`Expected, not listed: ${t.expected.live_thumbnails} live thumbnail(s), ${t.expected.in_flight} upload(s) in flight`);
++ log('');
++ log('Nothing was deleted or moved.');
++ log(`Report: ${file}`);
++ if (JSON_OUT) console.log(JSON.stringify({ ...report, report: file }, null, 2));
++}
++
(async () => {
++ if (has('storage')) { await storageReport(); db.close(); process.exit(0); }
if (!vodStorage.providerConfigured(provider)) usage(`${provider} is not configured (MEDIA_${provider.toUpperCase()}_* not in the environment)`);
const report = await orphans.buildReport({ provider, prefix: arg('prefix') || orphans.PREFIX });
const file = arg('out') ? path.resolve(arg('out')) : path.join(path.dirname(config.db.path), 'reports', `vods-orphans-${report.generated_at.replace(/[:.]/g, '-')}.json`);
ddiff --git a/server/drill.js b/server/drill.js
iindex cf55271..4a5de2b 100644
---- a/server/drill.js
++++ b/server/drill.js
@@@ -19,7 +19,7 @@
* Watch pages and the media index render from the database only and still answer.
* - server/index.js opens the database and serves HTTP, and starts nothing else: no app seeding, no
* JWKS refresh, no health job, junk sweep, tiering sweep, clip re-cuts, copy verification, jobs
-- * worker (thumbnails, split/remux, invariant scans), disk guardian, thumbnail cleanup, object purge,
++ * worker (thumbnails, split/remux, invariant scans, storage orphan scans), disk guardian, thumbnail cleanup, object purge,
* backfill, orphan-recording finalize or Events relay; webhooks are never sent. Timers that modules
* start when they are loaded check `enabled` as well. Readiness does not probe storage (a drill
* writes none) or the remote tiers.
ddiff --git a/server/events.js b/server/events.js
iindex 5a7cca1..f432b3f 100644
---- a/server/events.js
++++ b/server/events.js
@@@ -126,6 +126,8 @@ function record(webhookEvent, appId, data) {
function recordJob(transition, job) {
const priority = JOB_TRANSITIONS[transition];
if (!priority) throw new Error(`unknown job transition ${transition}`);
++ // A service-wide maintenance job (queue.SYSTEM_APP) is no tenant's: nothing to announce.
++ if (job.app_id === require('./jobs/queue').SYSTEM_APP) return null;
return enqueue(`media.job.${transition}`, job.app_id, { type: 'job', id: String(job.id) }, job, priority);
}
ddiff --git a/server/jobs/queue.js b/server/jobs/queue.js
iindex e84db4d..e889f48 100644
---- a/server/jobs/queue.js
++++ b/server/jobs/queue.js
@@@ -28,6 +28,9 @@ const STATUSES = ['proposed', 'queued', 'running', 'succeeded', 'failed', 'cance
const FINISHED = ['succeeded', 'failed', 'cancelled'];
const ACTIVE = ['queued', 'running'];
const MAX_ACTIVE_PER_TENANT = 50; // queued + running jobs one tenant may have (proposals excluded)
++// Service-wide maintenance jobs (storage.orphans.scan) belong to no tenant: they run under this app id,
++// which has no apps row, so no API credential reaches them, and they announce no events.
++const SYSTEM_APP = '_media';
const bus = new EventEmitter();
bus.setMaxListeners(0);
@@@ -394,7 +397,7 @@ function prune({ days = 30, types = ['thumbnail.regenerate', 'object.hash'] } =
}
module.exports = {
-- STATUSES, FINISHED, ACTIVE, MAX_ACTIVE_PER_TENANT, JobError, bus,
++ STATUSES, FINISHED, ACTIVE, MAX_ACTIVE_PER_TENANT, SYSTEM_APP, JobError, bus,
register, typeSpec, typeNames, requestHash, parseJson,
get, getForApp, jobPublic, jobEvent, list, counts,
enqueue, approve, cancel,
ddiff --git a/server/jobs/storage-orphans.js b/server/jobs/storage-orphans.js
nnew file mode 100644
iindex 0000000..ad55296
---- /dev/null
++++ b/server/jobs/storage-orphans.js
@@@ -0,0 +1,73 @@
++/**
++ * OpenVibe.Media — job type storage.orphans.scan (light lane): the storage orphan report
++ * (server/vod/orphans-report.js buildStorageReport) across the whole service. Report only: it walks the
++ * data directories, lists the B2/R2 buckets (keys and open multipart uploads), reads the database and
++ * writes <data>/reports/storage-orphans-<time>.json. It never deletes, moves or changes a file, a bucket
++ * object or a row.
++ *
++ * params { limit (1-100000, default 2000): items listed per section; the totals count everything }
++ * result { totals, providers, summary, report } report = the file name under <data>/reports/
++ *
++ * Storage is shared by every tenant, so the job belongs to none: it runs under queue.SYSTEM_APP, which no
++ * API credential reaches (no apps row), announces no media.job.* event, and cannot be enqueued through
++ * /api/v2/:app/jobs (validate refuses every tenant). Scheduled every MEDIA_ORPHAN_SCAN_DAYS (30; 0 = never)
++ * by the worker, which does not start in a restore drill. On demand: scripts/vods-orphans-report.js --storage.
++ */
++'use strict';
++
++const path = require('path');
++const queue = require('./queue');
++const orphans = require('../vod/orphans-report');
++
++const { JobError } = queue;
++const TYPE = 'storage.orphans.scan';
++
++function envInt(name, fallback) {
++ const v = parseInt(process.env[name] || '', 10);
++ return Number.isFinite(v) ? v : fallback;
++}
++
++function checkParams(params) {
++ const p = params || {};
++ const limit = p.limit == null ? 2000 : Number(p.limit);
++ if (!Number.isInteger(limit) || limit < 1 || limit > 100000) throw new JobError('media.job.invalid', 'params.limit must be 1-100000', { permanent: true });
++ return { limit };
++}
++
++function validate({ appId, obj, params }) {
++ if (appId !== queue.SYSTEM_APP) {
++ throw new JobError('media.job.forbidden', `${TYPE} covers the whole service's storage: it runs on its schedule or from scripts/vods-orphans-report.js --storage`, { status: 403, permanent: true });
++ }
++ if (obj) throw new JobError('media.job.invalid', `${TYPE} covers the whole storage: leave object_id out`, { permanent: true });
++ return checkParams(params);
++}
++
++async function run(job) {
++ const p = checkParams(job.params);
++ const report = await orphans.buildStorageReport({ limit: p.limit });
++ report.job_id = job.id;
++ const file = orphans.writeStorageReport(report);
++ const summary = orphans.summarizeStorage(report);
++ console.log(`[Orphans] ${summary} (report ${path.basename(file)}; nothing was deleted)`);
++ return { totals: report.totals, providers: report.scope.providers, summary, report: path.basename(file) };
++}
++
++/** Queue one scan every MEDIA_ORPHAN_SCAN_DAYS (0 = never). Returns how many were queued (0 or 1). */
++function schedule(nowMs = Date.now()) {
++ const days = envInt('MEDIA_ORPHAN_SCAN_DAYS', 30);
++ if (!(days > 0)) return 0;
++ const period = Math.floor(nowMs / (days * 86400 * 1000));
++ try {
++ const r = queue.enqueue({ appId: queue.SYSTEM_APP, type: TYPE, params: {}, idempotencyKey: `${TYPE}:${days}d:${period}`, createdBy: 'system:schedule' });
++ return r.created ? 1 : 0;
++ } catch (err) {
++ console.warn(`[Jobs] could not schedule ${TYPE}: ${err.message}`);
++ return 0;
++ }
++}
++
++module.exports = {
++ TYPE,
++ spec: { lane: 'light', maxAttempts: 2, timeoutMs: 2 * 60 * 60 * 1000, needsObject: false, validate, run },
++ schedule,
++};
ddiff --git a/server/jobs/types.js b/server/jobs/types.js
iindex 7fa8ad2..da9a123 100644
---- a/server/jobs/types.js
++++ b/server/jobs/types.js
@@@ -8,6 +8,7 @@
* vod.finalize finalize finalize a recording whose finalize failed or never ran (orphans), with backoff
* vod.duration.reconcile heavy stored VOD durations vs a measurement of the real file (local or B2/R2), one batch
* object.hash light sha256 of local copies with no content hash yet, one bounded batch (or one object)
++ * storage.orphans.scan light the storage orphan report, service-wide (report only; never deletes)
*/
'use strict';
@@@ -20,5 +21,6 @@ queue.register('object.remux', require('./derive').remux);
queue.register('vod.finalize', require('./vod-finalize').spec);
queue.register('vod.duration.reconcile', require('./duration-reconcile').spec);
queue.register('object.hash', require('./content-hash').spec);
++queue.register('storage.orphans.scan', require('./storage-orphans').spec);
module.exports = { names: () => queue.typeNames() };
ddiff --git a/server/jobs/worker.js b/server/jobs/worker.js
iindex 1594e27..6812646 100644
---- a/server/jobs/worker.js
++++ b/server/jobs/worker.js
@@@ -16,7 +16,8 @@
*
* Scheduling: the size-invariant validator (invariant.scan) is enqueued per tenant every
* MEDIA_INVARIANT_SCAN_HOURS (it proposes split/remux jobs; it never runs them). Orphaned recordings
-- * get a vod.finalize every MEDIA_FINALIZE_SWEEP_S (server/jobs/vod-finalize.js).
++ * get a vod.finalize every MEDIA_FINALIZE_SWEEP_S (server/jobs/vod-finalize.js). The storage orphan
++ * report (storage.orphans.scan, report only) runs every MEDIA_ORPHAN_SCAN_DAYS under queue.SYSTEM_APP.
*/
'use strict';
@@@ -149,6 +150,7 @@ async function tick() {
scheduleInvariantScans(now);
try { require('./duration-reconcile').schedule(now); } catch (err) { console.warn('[Jobs] duration reconcile schedule:', err.message); }
try { require('./content-hash').schedule(now); } catch (err) { console.warn('[Jobs] content hash schedule:', err.message); }
++ try { require('./storage-orphans').schedule(now); } catch (err) { console.warn('[Jobs] storage orphan scan schedule:', err.message); }
}
try { require('./vod-finalize').sweepOrphans({ now }); } catch (err) { console.warn('[Jobs] orphan sweep:', err.message); }
if (now - lastPruneAt > 6 * 3600 * 1000) {
ddiff --git a/server/vod/orphans-report.js b/server/vod/orphans-report.js
iindex 67cdad5..391382e 100644
---- a/server/vod/orphans-report.js
++++ b/server/vod/orphans-report.js
@@@ -1,6 +1,14 @@
/**
-- * OpenVibe.Media — report on the B2 `vods-orphans/` prefix (hazard H3: 55 objects, 14.5 GB on
-- * 2026-09-21). Report only: it lists, matches and recommends; it never deletes, moves or writes.
++ * OpenVibe.Media — orphan reports. Report only: they list, match and recommend; they never delete,
++ * move or write a file, a bucket object or a row.
++ *
++ * buildReport() the B2 `vods-orphans/` prefix (hazard H3: 55 objects, 14.5 GB on 2026-09-21)
++ * buildStorageReport() the whole storage against the database (roadmap WS-G task 8): files and keys
++ * no row names, copies the database records that are not there, and multipart
++ * uploads left open (Media's own and the buckets'). The storage.orphans.scan job
++ * runs it monthly; scripts/vods-orphans-report.js --storage on demand.
++ *
++ * ── The vods-orphans/ prefix ──
*
* Each object under the prefix is matched to a VOD or clip by its file name (the basename of
* vods.file_path / storage_key, clips.file_path; else the vod-<app>-<id>-<ms>.<ext> name the
@@@ -22,6 +30,7 @@
const fs = require('fs');
const path = require('path');
++const config = require('../config');
const db = require('../db/database');
const PREFIX = 'vods-orphans/';
@@@ -136,4 +145,321 @@ async function buildReport({ provider = 'b2', prefix = PREFIX, list = null } = {
};
}
--module.exports = { PREFIX, RECOMMENDATIONS, parseRecordingName, assess, buildReport };
++// ═══════════════════════════════════════════════════════════════
++// The whole storage against the database (WS-G task 8)
++// ═══════════════════════════════════════════════════════════════
++//
++// unreferenced_local files under the data directories that no row names (media_locations, vods,
++// clips, files, pastes, assets, open upload sessions, unfinished jobs). A file whose
++// only row is a deleted object's location counts (its bytes outlived the object),
++// unless the object is a native one still inside its retention period, or held.
++// unreferenced_remote B2/R2 keys that no media_locations row or vods/clips row names
++// missing copies the database records for objects that are not deleted, whose bytes are not
++// there: no local file, or a key absent from its bucket's listing
++// multipart_local Media's own upload sessions still open (media_uploads active/completing), expired
++// ones flagged; part directories no open session owns are unreferenced_local
++// multipart_remote multipart uploads the buckets still hold open (ListMultipartUploads)
++//
++// Expected files that no row names are counted, not listed: live thumbnails (stream-<app>-<id>.jpg,
++// refreshed in place while a stream is live) and upload temps / partial downloads younger than an hour.
++// A provider that is not configured, or whose listing fails, is reported as not listed; nothing is
++// concluded about its keys.
++
++const STORAGE_SECTIONS = ['unreferenced_local', 'unreferenced_remote', 'missing', 'multipart_local', 'multipart_remote'];
++const IN_FLIGHT_MS = 60 * 60 * 1000;
++const MAX_WALK = 1000000;
++
++function localRoots() {
++ const seen = new Set();
++ return [
++ ['vods', config.vod.path], ['clips', config.vod.clipsPath], ['thumbnails', config.thumbnails.path], ['files', config.files.path],
++ ['pastes', config.pastes.path], ['objects', config.objects.path], ['assets', config.assets.path],
++ ].map(([name, dir]) => ({ name, dir: path.resolve(dir) })).filter(r => (seen.has(r.dir) ? false : seen.add(r.dir)));
++}
++
++/** Every regular file under one root. Symlinks are not followed; a root nested inside it is walked on its own. */
++function walkRoot(root, rootDirs, out) {
++ const stack = [root.dir];
++ let seen = 0;
++ while (stack.length) {
++ const d = stack.pop();
++ let entries;
++ try { entries = fs.readdirSync(d, { withFileTypes: true }); } catch { continue; }
++ for (const e of entries) {
++ const p = path.join(d, e.name);
++ if (e.isDirectory()) { if (!rootDirs.has(p)) stack.push(p); continue; }
++ if (!e.isFile()) continue;
++ if (out.files.length >= MAX_WALK) { out.truncated = true; return seen; }
++ let st;
++ try { st = fs.statSync(p); } catch { continue; }
++ out.files.push({ root: root.name, abs: p, path: path.relative(root.dir, p), size: st.size, mtimeMs: st.mtimeMs });
++ seen++;
++ }
++ }
++ return seen;
++}
++
++/** A location keeps its bytes wanted: its object exists and is not deleted, or is a native object inside its retention, or is held. */
++function locationWanted(l) {
++ if (!l.object_id) return false;
++ return l.lifecycle_status !== 'deleted' || (!l.legacy_ref && !l.purged) || !!l.held;
++}
++
++const LOCATION_ROWS = `SELECT l.provider, l.bucket, l.key, o.id AS object_id, o.lifecycle_status, o.legacy_ref,
++ CASE WHEN json_valid(o.metadata) THEN json_extract(o.metadata, '$.purged_at') END AS purged,
++ EXISTS (SELECT 1 FROM media_holds h WHERE h.object_id = o.id AND h.released_at IS NULL) AS held
++ FROM media_locations l LEFT JOIN media_objects o ON o.id = l.object_id`;
++
++const sidecarsOf = (p) => [p.replace(/\.webm$/, '.seekable.webm'), p.replace(/\.mp4$/, '.seekable.mp4'), p.replace(/\.webm$/, '.master.mkv')].filter(x => x !== p);
++
++/** What the database names on local disk: { wanted: Set(abs path), deleted: Map(abs path → location), segmentBases, sessions, jobs }. */
++function localReferences() {
++ const vs = vodStorage();
++ const model = require('../objects/model');
++ const wanted = new Set();
++ const add = (p) => { if (p) wanted.add(path.resolve(String(p))); };
++ const deleted = new Map();
++ for (const l of db.all(`${LOCATION_ROWS} WHERE l.provider = 'local'`)) {
++ if (locationWanted(l)) add(l.key); else deleted.set(path.resolve(String(l.key)), l);
++ }
++ const thumb = (u) => { const name = model.thumbFileFromUrl(u); if (name) add(path.join(config.thumbnails.path, name)); };
++ const segmentBases = new Set(); // browser-chunk segments <base>.seg-<n>-<ms>.webm belong to <base>.webm
++ for (const v of db.all('SELECT id, file_path, master_file_path, thumbnail_url FROM vods')) {
++ if (v.file_path) {
++ for (const f of [v.file_path, vs.localPathForVod(v)]) {
++ add(f);
++ for (const x of sidecarsOf(path.resolve(f))) add(x);
++ segmentBases.add(path.join(path.dirname(path.resolve(f)), path.basename(f).replace(/\.webm$/, '')));
++ }
++ }
++ add(v.master_file_path);
++ thumb(v.thumbnail_url);
++ }
++ for (const c of db.all('SELECT file_path, thumbnail_url FROM clips')) {
++ if (c.file_path) { add(c.file_path); add(path.join(config.vod.clipsPath, path.basename(c.file_path))); }
++ thumb(c.thumbnail_url);
++ }
++ for (const f of db.all('SELECT key, app_id FROM files')) add(path.join(config.files.path, f.app_id, f.key));
++ for (const p of db.all('SELECT screenshot_path FROM pastes WHERE screenshot_path IS NOT NULL')) add(p.screenshot_path);
++ for (const a of db.all('SELECT file_path FROM assets WHERE file_path IS NOT NULL')) add(a.file_path);
++ const sessions = new Set(db.all("SELECT id FROM media_uploads WHERE status IN ('active', 'completing')").map(r => r.id));
++ const jobs = new Set(db.all("SELECT id FROM media_jobs WHERE status IN ('queued', 'running')").map(r => r.id));
++ return { wanted, deleted, segmentBases, sessions, jobs };
++}
++
++/** Is this local file named by the database (or one of the expected kinds)? 'wanted' | 'live_thumbnail' | 'in_flight' | null. */
++function classifyLocal(f, refs, now) {
++ if (refs.wanted.has(f.abs)) return 'wanted';
++ const base = path.basename(f.abs);
++ const seg = /^(.+)\.seg-\d+-\d+\.webm$/.exec(base);
++ if (seg && refs.segmentBases.has(path.join(path.dirname(f.abs), seg[1]))) return 'wanted';
++ const [top, sub] = f.path.split(path.sep);
++ if (f.root === 'objects' && top === '.parts' && refs.sessions.has(sub)) return 'wanted';
++ if (f.root === 'objects' && top === '.jobs' && refs.jobs.has(sub)) return 'wanted';
++ if (f.root === 'thumbnails' && /^stream-.+\.jpg$/.test(base)) return 'live_thumbnail';
++ const transient = (f.root === 'objects' && top === '.tmp') || base.endsWith('.download') || (f.root === 'objects' && top === '.parts');
++ if (transient && now - f.mtimeMs < IN_FLIGHT_MS) return 'in_flight';
++ return null;
++}
++
++/** A recorder-style name (vod-<app>-<id>-<ms>.<ext>): whose recording, and does its row still exist? */
++function recordingNameHint(base) {
++ const parsed = parseRecordingName(base);
++ if (!parsed) return null;
++ const row = db.get('SELECT id, file_path FROM vods WHERE id = ? AND app_id = ?', [parsed.id, parsed.app]);
++ if (!row) return `a recording of vod ${parsed.id} (${parsed.app}), whose row is gone`;
++ return `named for vod ${parsed.id} (${parsed.app}), whose row points at ${path.basename(row.file_path || '') || 'no file'}`;
++}
++
++function localHint(f, refs) {
++ const base = path.basename(f.abs);
++ const del = refs.deleted.get(f.abs);
++ if (del) return del.object_id ? `bytes of deleted object ${del.object_id}${del.legacy_ref ? ` (${del.legacy_ref})` : ''}` : 'named only by a location row whose object does not exist';
++ if (f.root === 'objects') {
++ const [top, sub] = f.path.split(path.sep);
++ if (top === '.parts') {
++ const s = db.get('SELECT status FROM media_uploads WHERE id = ?', [sub]);
++ return s ? `parts of multipart upload ${sub} (${s.status})` : `parts of multipart upload ${sub}, which has no session`;
++ }
++ if (top === '.jobs') {
++ const j = db.get('SELECT status FROM media_jobs WHERE id = ?', [sub]);
++ return j ? `work files of job ${sub} (${j.status})` : `work files of job ${sub}, which does not exist`;
++ }
++ if (top === '.tmp') return 'an upload that never finished (temp file)';
++ const o = db.get('SELECT id, lifecycle_status FROM media_objects WHERE id = ?', [base]);
++ return o ? `named for object ${o.id} (${o.lifecycle_status}), which records no local copy here` : `named for object ${base}, which does not exist`;
++ }
++ if (base.endsWith('.download')) return 'a restore download that never finished';
++ if (/\.seekable\.(webm|mp4)$/.test(base)) return 'a DVR sidecar no VOD row names';
++ if (/\.master\.mkv$/.test(base)) return recordingNameHint(base.replace(/\.master\.mkv$/, '.webm')) || 'a master archive no VOD row names';
++ return recordingNameHint(base);
++}
++
++/** What the database names in one provider's bucket: { wanted: Set(key), deleted: Map(key → location), rows: Map(key → row) }. */
++function remoteReferences(provider) {
++ const vs = vodStorage();
++ const bucket = vs.bucketFor(provider);
++ const wanted = new Set();
++ const deleted = new Map();
++ for (const l of db.all(`${LOCATION_ROWS} WHERE l.provider = ?`, [provider])) {
++ if (l.bucket && bucket && l.bucket !== bucket) continue; // a copy recorded in another bucket
++ if (locationWanted(l)) wanted.add(l.key); else deleted.set(l.key, l);
++ }
++ const rows = new Map(); // every key a vods/clips row points at, with the row (stale R2 copies are recognised by it)
++ for (const table of ['vods', 'clips']) {
++ for (const r of db.all(`SELECT id, file_path, storage_provider, storage_key FROM ${table} WHERE COALESCE(file_path, '') != ''`)) {
++ const p = vs.providerOf(r);
++ if (p === 'local' && !r.storage_key) continue;
++ const key = vs.keyForVod(r);
++ rows.set(key, { kind: table === 'vods' ? 'vod' : 'clip', id: r.id, provider: p });
++ if (provider === 'b2' || p === 'r2') wanted.add(key);
++ }
++ }
++ return { wanted, deleted, rows };
++}
++
++function remoteHint(provider, key, refs) {
++ if (key.startsWith(PREFIX)) return `parked in ${PREFIX}: scripts/vods-orphans-report.js says what each one is`;
++ const del = refs.deleted.get(key);
++ if (del) return del.object_id ? `copy of deleted object ${del.object_id}${del.legacy_ref ? ` (${del.legacy_ref})` : ''}` : 'named only by a location row whose object does not exist';
++ const row = refs.rows.get(key);
++ if (row && provider === 'r2') return `an R2 copy of ${row.kind} ${row.id}, which is served from ${row.provider}`;
++ const shot = /paste-screenshot-(\d+)\.[a-z0-9]+$/i.exec(key);
++ if (shot) return db.get('SELECT 1 AS x FROM pastes WHERE id = ?', [Number(shot[1])]) ? `a legacy screenshot of paste ${shot[1]}` : `a legacy screenshot of paste ${shot[1]}, whose row is gone`;
++ return recordingNameHint(path.basename(key));
++}
++
++/**
++ * The storage report. Options (all injectable for tests):
++ * providers which buckets to list (default: the configured ones of b2, r2)
++ * list (provider) → [{ key, size, last_modified }] default: every key (ListObjectsV2)
++ * listUploads (provider) → [{ key, upload_id, initiated }] default: ListMultipartUploads
++ * limit items listed per section (totals always count everything), default 2000
++ */
++async function buildStorageReport({ providers = null, list = null, listUploads = null, limit = 2000, now = Date.now() } = {}) {
++ const vs = vodStorage();
++ const provs = providers || ['b2', 'r2'].filter(p => vs.providerConfigured(p));
++ const lister = list || ((p) => vs.listObjects(p, ''));
++ const uploadLister = listUploads || ((p) => vs.listMultipartUploads(p));
++ const cap = Math.max(1, Number(limit) || 2000);
++ const report = {
++ kind: 'media.storage_orphans.report', version: 1, generated_at: new Date(now).toISOString(), read_only: true,
++ scope: { roots: [], providers: {}, walk_truncated: false },
++ totals: {
++ unreferenced_local: { files: 0, bytes: 0, by_root: {} }, unreferenced_remote: {},
++ missing: { locations: 0, by_provider: {} }, multipart_local: { open: 0, expired: 0 }, multipart_remote: {},
++ expected: { live_thumbnails: 0, in_flight: 0 },
++ },
++ truncated: {},
++ note: 'Report only: nothing was deleted, moved or written. Each item is for an operator to look at.',
++ };
++ for (const k of STORAGE_SECTIONS) { report[k] = []; report.truncated[k] = false; }
++ const push = (section, item) => { if (report[section].length < cap) report[section].push(item); else report.truncated[section] = true; };
++
++ // ── Local disk ──
++ const roots = localRoots();
++ const rootDirs = new Set(roots.map(r => r.dir));
++ const walked = { files: [], truncated: false };
++ for (const root of roots) {
++ const exists = fs.existsSync(root.dir);
++ const files = exists ? walkRoot(root, rootDirs, walked) : 0;
++ report.scope.roots.push({ name: root.name, dir: root.dir, exists, files });
++ }
++ report.scope.walk_truncated = walked.truncated;
++ const lrefs = localReferences();
++ const orphanLocal = [];
++ for (const f of walked.files) {
++ const c = classifyLocal(f, lrefs, now);
++ if (c === 'wanted') continue;
++ if (c === 'live_thumbnail') { report.totals.expected.live_thumbnails++; continue; }
++ if (c === 'in_flight') { report.totals.expected.in_flight++; continue; }
++ orphanLocal.push(f);
++ const t = report.totals.unreferenced_local;
++ t.files++; t.bytes += f.size;
++ t.by_root[f.root] = t.by_root[f.root] || { files: 0, bytes: 0 };
++ t.by_root[f.root].files++; t.by_root[f.root].bytes += f.size;
++ }
++ orphanLocal.sort((a, b) => b.size - a.size);
++ for (const f of orphanLocal.slice(0, cap)) push('unreferenced_local', { root: f.root, path: f.path, size: f.size, modified: new Date(f.mtimeMs).toISOString(), hint: localHint(f, lrefs) });
++ if (orphanLocal.length > cap) report.truncated.unreferenced_local = true;
++
++ // ── Buckets ──
++ const listed = new Map(); // provider → Map(key → size)
++ for (const p of provs) {
++ const entry = { configured: vs.providerConfigured(p), bucket: vs.bucketFor(p), listed: false };
++ report.scope.providers[p] = entry;
++ let keys;
++ try { keys = await lister(p); } catch (err) { entry.error = err.message; continue; }
++ entry.listed = true;
++ entry.keys = keys.length;
++ entry.bytes = keys.reduce((n, k) => n + (Number(k.size) || 0), 0);
++ listed.set(p, new Map(keys.map(k => [k.key, k])));
++ const rrefs = remoteReferences(p);
++ const orphans = keys.filter(k => !rrefs.wanted.has(k.key)).sort((a, b) => (Number(b.size) || 0) - (Number(a.size) || 0));
++ report.totals.unreferenced_remote[p] = { keys: orphans.length, bytes: orphans.reduce((n, k) => n + (Number(k.size) || 0), 0) };
++ for (const k of orphans.slice(0, cap)) push('unreferenced_remote', { provider: p, key: k.key, size: Number(k.size) || 0, last_modified: k.last_modified || null, hint: remoteHint(p, k.key, rrefs) });
++ if (orphans.length > cap) report.truncated.unreferenced_remote = true;
++ let uploads = [];
++ try { uploads = await uploadLister(p); } catch (err) { entry.multipart_error = err.message; }
++ report.totals.multipart_remote[p] = uploads.length;
++ for (const u of uploads) push('multipart_remote', { provider: p, key: u.key, upload_id: u.upload_id, initiated: u.initiated || null, key_named_by_a_row: rrefs.wanted.has(u.key) });
++ }
++
++ // ── Copies the database records whose bytes are not there ──
++ for (const l of db.all(`SELECT l.provider, l.bucket, l.key, l.state, l.verified_at, o.id AS object_id, o.app_id, o.kind, o.legacy_ref, o.lifecycle_status
++ FROM media_locations l JOIN media_objects o ON o.id = l.object_id WHERE o.lifecycle_status != 'deleted' ORDER BY l.id`)) {
++ let there;
++ if (l.provider === 'local') {
++ try { there = fs.statSync(l.key).isFile(); } catch { there = false; }
++ } else {
++ const keys = listed.get(l.provider);
++ if (!keys) continue; // not listed: nothing is known
++ if (l.bucket && vs.bucketFor(l.provider) && l.bucket !== vs.bucketFor(l.provider)) continue;
++ there = keys.has(l.key);
++ }
++ if (there) continue;
++ report.totals.missing.locations++;
++ report.totals.missing.by_provider[l.provider] = (report.totals.missing.by_provider[l.provider] || 0) + 1;
++ push('missing', { object_id: l.object_id, app_id: l.app_id, kind: l.kind, legacy_ref: l.legacy_ref || null, lifecycle_status: l.lifecycle_status,
++ provider: l.provider, key: l.key, recorded_state: l.state, verified_at: l.verified_at || null });
++ }
++
++ // ── Media's own multipart sessions still open ──
++ for (const u of db.all(`SELECT u.*, (SELECT COUNT(*) FROM media_upload_parts p WHERE p.upload_id = u.id) AS parts_received,
++ (SELECT COALESCE(SUM(p.size_bytes), 0) FROM media_upload_parts p WHERE p.upload_id = u.id) AS bytes_received,
++ (u.expires_at < datetime('now')) AS expired
++ FROM media_uploads u WHERE u.status IN ('active', 'completing') ORDER BY u.created_at`)) {
++ report.totals.multipart_local.open++;
++ if (u.expired) report.totals.multipart_local.expired++;
++ push('multipart_local', { upload_id: u.id, object_id: u.object_id, app_id: u.app_id, status: u.status, parts_received: u.parts_received,
++ parts_expected: u.parts_expected, bytes_received: u.bytes_received, total_size: u.total_size, created_at: u.created_at, expires_at: u.expires_at, expired: !!u.expired });
++ }
++ return report;
++}
++
++function reportsDir() {
++ return path.join(path.dirname(config.db.path), 'reports');
++}
++
++/** Write a storage report as JSON (default <data>/reports/storage-orphans-<time>.json). Returns the path. */
++function writeStorageReport(report, file = null) {
++ const out = file || path.join(reportsDir(), `storage-orphans-${report.generated_at.replace(/[:.]/g, '-')}.json`);
++ fs.mkdirSync(path.dirname(out), { recursive: true });
++ fs.writeFileSync(out, JSON.stringify(report, null, 1));
++ return out;
++}
++
++/** One line per section for logs and the job result. */
++function summarizeStorage(report) {
++ const t = report.totals;
++ const mb = (b) => `${(b / 1048576).toFixed(1)} MB`;
++ const remote = Object.entries(t.unreferenced_remote).map(([p, x]) => `${p} ${x.keys} key(s) ${mb(x.bytes)}`).join(', ') || 'no bucket listed';
++ return `unreferenced local ${t.unreferenced_local.files} file(s) ${mb(t.unreferenced_local.bytes)}; unreferenced remote: ${remote}; `
++ + `missing copies ${t.missing.locations}; open multipart: ${t.multipart_local.open} here (${t.multipart_local.expired} expired), `
++ + `${Object.entries(t.multipart_remote).map(([p, n]) => `${n} in ${p}`).join(', ') || 'buckets not listed'}`;
++}
++
++module.exports = {
++ PREFIX, RECOMMENDATIONS, parseRecordingName, assess, buildReport,
++ STORAGE_SECTIONS, buildStorageReport, writeStorageReport, summarizeStorage, localRoots,
++};
ddiff --git a/server/vod/vod-storage.js b/server/vod/vod-storage.js
iindex f72b9ea..01ecdb3 100644
---- a/server/vod/vod-storage.js
++++ b/server/vod/vod-storage.js
@@@ -393,6 +393,27 @@ async function listObjects(provider, prefix) {
return out;
}
++/**
++ * Multipart uploads still open in one provider's bucket (started, never completed or aborted: a killed
++ * process leaves them, and their parts are stored and billed): [{ key, upload_id, initiated }] (read-only).
++ */
++async function listMultipartUploads(provider, prefix = '') {
++ const client = clientFor(provider);
++ if (!client) throw new Error(`${provider} is not configured`);
++ loadSdk();
++ const out = [];
++ let keyMarker, idMarker;
++ for (;;) {
++ const r = await client.send(new S3.ListMultipartUploadsCommand({ Bucket: PROVIDER_ENV[provider].bucket, Prefix: prefix || undefined,
++ KeyMarker: keyMarker, UploadIdMarker: idMarker, MaxUploads: 1000 }));
++ for (const u of (r.Uploads || [])) out.push({ key: u.Key, upload_id: u.UploadId, initiated: u.Initiated ? new Date(u.Initiated).toISOString() : null });
++ if (!r.IsTruncated || !r.NextKeyMarker) break;
++ keyMarker = r.NextKeyMarker;
++ idMarker = r.NextUploadIdMarker;
++ }
++ return out;
++}
++
async function presignGet(provider, key, expiresInSeconds = 900) {
const client = clientFor(provider);
if (!client) return null;
@@@ -1378,6 +1399,7 @@ module.exports = {
endpointFor,
headObject,
listObjects,
++ listMultipartUploads,
keyForVod,
localPathForVod,
resolvePlayback,
ddiff --git a/test/storage-orphans.test.js b/test/storage-orphans.test.js
nnew file mode 100644
iindex 0000000..1adff2e
---- /dev/null
++++ b/test/storage-orphans.test.js
@@@ -0,0 +1,239 @@
++'use strict';
++// The storage orphan report (server/vod/orphans-report.js buildStorageReport; job storage.orphans.scan;
++// scripts/vods-orphans-report.js --storage): files under the data directories and B2/R2 keys that no row
++// names, copies the database records that are not there, and multipart uploads left open, here and in the
++// buckets. Report only: the test snapshots every file, the tables and the buckets before and after and
++// finds them unchanged, and a fake S3 (spoken to by the real AWS SDK) counts any write or delete. The job
++// runs service-wide under queue.SYSTEM_APP, is scheduled every MEDIA_ORPHAN_SCAN_DAYS, and tenants cannot
++// enqueue it.
++const assert = require('assert');
++const fs = require('fs');
++const os = require('os');
++const path = require('path');
++const http = require('http');
++const crypto = require('crypto');
++const { execFileSync } = require('child_process');
++
++const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-orphans-'));
++const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
++
++const buckets = { 'b2-bucket': new Map(), 'r2-bucket': new Map() };
++const openUploads = { 'b2-bucket': [{ key: 'vods/half-uploaded.webm', id: 'up-b2-1' }], 'r2-bucket': [] };
++let writes = 0;
++const xml = (v) => String(v).replace(/&/g, '&amp;').replace(/</g, '&lt;');
++const s3 = http.createServer((req, res) => {
++ req.resume();
++ req.on('end', () => {
++ const u = new URL(req.url, 'http://x');
++ const [, bucket, ...rest] = decodeURIComponent(u.pathname).split('/');
++ const key = rest.join('/');
++ const store = buckets[bucket];
++ if (!store) { res.writeHead(404); return res.end(); }
++ if (req.method !== 'GET' && req.method !== 'HEAD') { writes++; res.writeHead(403, { 'Content-Type': 'application/xml' }); return res.end('<Error><Code>AccessDenied</Code></Error>'); }
++ if (!key && u.searchParams.has('uploads')) {
++ res.writeHead(200, { 'Content-Type': 'application/xml' });
++ return res.end(`<?xml version="1.0" encoding="UTF-8"?><ListMultipartUploadsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Bucket>${bucket}</Bucket><IsTruncated>false</IsTruncated>${
++ openUploads[bucket].map(x => `<Upload><Key>${xml(x.key)}</Key><UploadId>${x.id}</UploadId><Initiated>2026-09-01T00:00:00.000Z</Initiated></Upload>`).join('')}</ListMultipartUploadsResult>`);
++ }
++ if (!key && u.searchParams.get('list-type') === '2') {
++ const prefix = u.searchParams.get('prefix') || '';
++ const keys = [...store.keys()].filter(k => k.startsWith(prefix)).sort();
++ res.writeHead(200, { 'Content-Type': 'application/xml' });
++ return res.end(`<?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>${bucket}</Name><Prefix>${xml(prefix)}</Prefix><KeyCount>${keys.length}</KeyCount><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated>${
++ keys.map(k => `<Contents><Key>${xml(k)}</Key><LastModified>2026-09-21T00:00:00.000Z</LastModified><ETag>"e"</ETag><Size>${store.get(k).length}</Size><StorageClass>STANDARD</StorageClass></Contents>`).join('')}</ListBucketResult>`);
++ }
++ const obj = store.get(key);
++ if (!obj) { res.writeHead(404); return res.end(); }
++ res.writeHead(200, { 'Content-Length': obj.length, ETag: '"e"' });
++ res.end(req.method === 'GET' ? obj : undefined);
++ });
++});
++
++(async () => {
++ await new Promise((r) => s3.listen(0, '127.0.0.1', r));
++ const endpoint = `http://127.0.0.1:${s3.address().port}`;
++ const env = {
++ DB_PATH: path.join(tmp, 'data', 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
++ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), ASSETS_PATH: dir('assets'),
++ MEDIA_B2_ENDPOINT: endpoint, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_KEY_ID: 'k', MEDIA_B2_APP_KEY: 's', MEDIA_B2_REGION: 'us-west-004',
++ MEDIA_R2_ENDPOINT: endpoint, MEDIA_R2_BUCKET: 'r2-bucket', MEDIA_R2_ACCESS_KEY_ID: 'k', MEDIA_R2_SECRET_ACCESS_KEY: 's',
++ AWS_REQUEST_CHECKSUM_CALCULATION: 'WHEN_REQUIRED', AWS_RESPONSE_CHECKSUM_VALIDATION: 'WHEN_REQUIRED',
++ MEDIA_PUBLIC_URL: 'https://media.test',
++ };
++ Object.assign(process.env, env);
++ delete process.env.MEDIA_ORPHAN_SCAN_DAYS;
++ const db = require('../server/db/database');
++ const model = require('../server/objects/model');
++ const queue = require('../server/jobs/queue');
++ require('../server/jobs/types');
++ const orphans = require('../server/vod/orphans-report');
++ const storageJob = require('../server/jobs/storage-orphans');
++ db.upsertApp({ app_id: 'live', api_key: 'live-key-orphans' });
++ const raw = db.getDb();
++ const write = (d, name, bytes = 16, ageMs = 0) => {
++ const p = path.join(d, name);
++ fs.mkdirSync(path.dirname(p), { recursive: true });
++ fs.writeFileSync(p, Buffer.alloc(bytes, 7));
++ if (ageMs) { const t = (Date.now() - ageMs) / 1000; fs.utimesSync(p, t, t); }
++ return p;
++ };
++ const HOUR = 3600 * 1000;
++
++ // ── Fixtures: what the database names ──
++ const ins = raw.prepare(`INSERT INTO vods (id, app_id, title, file_path, file_size, is_public, visibility, duration_seconds, storage_provider, storage_key, thumbnail_url)
++ VALUES (?, 'live', ?, ?, ?, 1, 'public', 60, ?, ?, ?)`);
++ ins.run(1, 'local', write(env.VOD_PATH, 'vod-live-1-1758000000001.webm', 40), 40, 'local', null, '/t/vod-1-111.jpg');
++ write(env.THUMBNAILS_PATH, 'vod-1-111.jpg');
++ ins.run(2, 'in B2', '/old/vod-live-2-1758000000002.webm', 20, 'b2', 'vods/vod-live-2-1758000000002.webm', null);
++ buckets['b2-bucket'].set('vods/vod-live-2-1758000000002.webm', Buffer.alloc(20));
++ ins.run(3, 'served from R2', '/old/vod-live-3-1758000000003.webm', 30, 'r2', 'vods/vod-live-3-1758000000003.webm', null);
++ buckets['b2-bucket'].set('vods/vod-live-3-1758000000003.webm', Buffer.alloc(30));
++ buckets['r2-bucket'].set('vods/vod-live-3-1758000000003.webm', Buffer.alloc(30));
++ ins.run(4, 'back in B2, R2 copy left', '/old/vod-live-4-1758000000004.webm', 50, 'b2', 'vods/vod-live-4-1758000000004.webm', null);
++ buckets['b2-bucket'].set('vods/vod-live-4-1758000000004.webm', Buffer.alloc(50));
++ buckets['r2-bucket'].set('vods/vod-live-4-1758000000004.webm', Buffer.alloc(50));
++ ins.run(5, 'local file lost', path.join(env.VOD_PATH, 'vod-live-5-1758000000005.webm'), 10, 'local', null, null);
++ ins.run(6, 'B2 key lost', '/old/vod-live-6-1758000000006.webm', 60, 'b2', 'vods/vod-live-6-1758000000006.webm', null);
++ for (const id of [1, 2, 3, 4, 5, 6]) model.sync('vod', id);
++ raw.prepare(`INSERT INTO files (key, app_id, original_name, size, mime) VALUES ('abc123-notes.txt', 'live', 'notes.txt', 16, 'text/plain')`).run();
++ write(path.join(env.FILES_PATH, 'live'), 'abc123-notes.txt');
++ model.sync('file', 'abc123-notes.txt');
++ // A native object soft-deleted inside its retention period keeps its bytes wanted.
++ const kept = model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', visibility: 'private', size_bytes: 16 });
++ const keptPath = write(path.join(env.OBJECTS_PATH, 'live'), kept, 16);
++ model.upsertLocation(kept, { provider: 'local', key: keptPath, state: 'present', size_bytes: 16, verified: true });
++ model.softDelete(model.getObject(kept));
++ // A clip deleted by its row: the object is deleted, its file stayed behind.
++ raw.prepare(`INSERT INTO clips (id, app_id, vod_id, title, file_path, status) VALUES (21, 'live', 1, 'gone clip', ?, 'ready')`).run(write(env.CLIPS_PATH, 'clip-live-21.webm', 12));
++ model.sync('clip', 21);
++ const clipObj = db.get('SELECT object_id FROM clips WHERE id = 21').object_id;
++ raw.prepare('DELETE FROM clips WHERE id = 21').run();
++ // Media's own multipart sessions: one open and expired (its parts wanted), one expired session's parts dir with no session.
++ raw.prepare(`INSERT INTO media_uploads (id, object_id, app_id, part_size, total_size, parts_expected, status, expires_at)
++ VALUES ('mup_OPEN', ?, 'live', 8, 16, 2, 'active', datetime('now', '-1 hour'))`).run(kept);
++ raw.prepare("INSERT INTO media_upload_parts (upload_id, part_number, size_bytes, sha256) VALUES ('mup_OPEN', 1, 8, 'x')").run();
++ write(path.join(env.OBJECTS_PATH, '.parts', 'mup_OPEN'), '1', 8, 3 * HOUR);
++
++ // ── What storage holds that no row names ──
++ write(env.VOD_PATH, 'vod-live-99-1758000000099.webm', 1000); // the recorder's name; row 99 never existed
++ write(env.VOD_PATH, 'mystery.bin', 500);
++ write(env.VOD_PATH, 'vod-live-1-1758000000001.seekable.webm', 5); // a sidecar of vod 1: wanted
++ write(env.THUMBNAILS_PATH, 'stream-live-1.jpg'); // a live thumbnail: expected
++ write(path.join(env.OBJECTS_PATH, '.tmp'), 'med_OLD-abcd', 30, 3 * HOUR); // an upload that never finished
++ write(path.join(env.OBJECTS_PATH, '.tmp'), 'med_NOW-abcd', 30); // one in flight right now
++ write(path.join(env.OBJECTS_PATH, '.parts', 'mup_GONE'), '1', 8, 3 * HOUR); // parts with no session
++ buckets['b2-bucket'].set('vods/stray.webm', Buffer.alloc(70));
++ buckets['b2-bucket'].set('vods-orphans/vod-live-7-1758000000007.webm', Buffer.alloc(80));
++
++ // ── Everything before, to prove nothing changes ──
++ const snapshotFiles = () => {
++ const out = [];
++ const walk = (d) => { for (const e of fs.readdirSync(d, { withFileTypes: true })) { const p = path.join(d, e.name); if (e.isDirectory()) { if (p !== path.join(tmp, 'data')) walk(p); } else out.push(`${path.relative(tmp, p)}:${fs.statSync(p).size}:${fs.statSync(p).mtimeMs}`); } };
++ walk(tmp);
++ return out.sort();
++ };
++ const TABLES = ['vods', 'clips', 'files', 'pastes', 'media_objects', 'media_locations', 'media_holds', 'media_uploads', 'media_upload_parts'];
++ const snapshotDb = () => Object.fromEntries(TABLES.map(t => [t, crypto.createHash('sha256').update(JSON.stringify(raw.prepare(`SELECT * FROM ${t} ORDER BY rowid`).all())).digest('hex')]));
++ const snapshotBuckets = () => JSON.stringify(Object.entries(buckets).map(([b, m]) => [b, [...m.keys()].sort()]));
++ const before = { files: snapshotFiles(), db: snapshotDb(), buckets: snapshotBuckets() };
++
++ const report = await orphans.buildStorageReport({ limit: 50 });
++
++ // ── Unreferenced local files ──
++ const local = Object.fromEntries(report.unreferenced_local.map(f => [`${f.root}/${f.path.split(path.sep).join('/')}`, f]));
++ assert.deepStrictEqual(Object.keys(local).sort(), [
++ 'clips/clip-live-21.webm', 'objects/.parts/mup_GONE/1', 'objects/.tmp/med_OLD-abcd', 'vods/mystery.bin', 'vods/vod-live-99-1758000000099.webm',
++ ], 'exactly the files no row names');
++ assert.ok(/whose row is gone/.test(local['vods/vod-live-99-1758000000099.webm'].hint), 'a recorder name says whose recording it was');
++ assert.strictEqual(local['clips/clip-live-21.webm'].hint, `bytes of deleted object ${clipObj} (legacy:live:clip:21)`);
++ assert.ok(/no session/.test(local['objects/.parts/mup_GONE/1'].hint));
++ assert.ok(/never finished/.test(local['objects/.tmp/med_OLD-abcd'].hint));
++ assert.strictEqual(local['vods/mystery.bin'].hint, null);
++ assert.strictEqual(report.unreferenced_local[0].path, 'vod-live-99-1758000000099.webm', 'largest first');
++ assert.deepStrictEqual([report.totals.unreferenced_local.files, report.totals.unreferenced_local.bytes], [5, 1000 + 500 + 30 + 8 + 12]);
++ assert.deepStrictEqual(report.totals.expected, { live_thumbnails: 1, in_flight: 1 }, 'live thumbnails and uploads in flight are expected, not orphans');
++ console.log('✅ unreferenced local files: recordings without a row, strays, bytes of deleted objects, stale temps and parts');
++
++ // ── Unreferenced bucket keys ──
++ const remote = Object.fromEntries(report.unreferenced_remote.map(k => [`${k.provider}:${k.key}`, k]));
++ assert.deepStrictEqual(Object.keys(remote).sort(), ['b2:vods-orphans/vod-live-7-1758000000007.webm', 'b2:vods/stray.webm', 'r2:vods/vod-live-4-1758000000004.webm']);
++ assert.ok(/vods-orphans-report/.test(remote['b2:vods-orphans/vod-live-7-1758000000007.webm'].hint), 'parked recordings point at the prefix report');
++ assert.strictEqual(remote['r2:vods/vod-live-4-1758000000004.webm'].hint, 'an R2 copy of vod 4, which is served from b2');
++ assert.deepStrictEqual(report.totals.unreferenced_remote, { b2: { keys: 2, bytes: 150 }, r2: { keys: 1, bytes: 50 } });
++ assert.deepStrictEqual([report.scope.providers.b2.listed, report.scope.providers.b2.keys, report.scope.providers.r2.keys], [true, 5, 2]);
++ console.log('✅ unreferenced bucket keys: strays, parked recordings, an R2 copy left behind by a demotion');
++
++ // ── Copies the database records that are not there ──
++ const missing = report.missing.map(m => `${m.legacy_ref}:${m.provider}`).sort();
++ assert.deepStrictEqual(missing, ['legacy:live:vod:5:local', 'legacy:live:vod:6:b2'], 'a lost local file and a lost B2 key');
++ assert.deepStrictEqual(report.totals.missing, { locations: 2, by_provider: { local: 1, b2: 1 } });
++ console.log('✅ missing copies: rows whose location is not in storage');
++
++ // ── Multipart uploads left open ──
++ assert.deepStrictEqual(report.multipart_local.map(u => [u.upload_id, u.expired, u.parts_received]), [['mup_OPEN', true, 1]]);
++ assert.deepStrictEqual(report.totals.multipart_local, { open: 1, expired: 1 });
++ assert.deepStrictEqual(report.multipart_remote.map(u => [u.provider, u.key, u.upload_id, u.key_named_by_a_row]), [['b2', 'vods/half-uploaded.webm', 'up-b2-1', false]]);
++ assert.deepStrictEqual(report.totals.multipart_remote, { b2: 1, r2: 0 });
++ console.log('✅ open multipart uploads: Media\'s expired sessions and the buckets\' own');
++
++ // ── Report only ──
++ assert.deepStrictEqual(snapshotFiles(), before.files, 'no file was deleted, moved or touched');
++ assert.deepStrictEqual(snapshotDb(), before.db, 'no row changed');
++ assert.strictEqual(snapshotBuckets(), before.buckets, 'no bucket key changed');
++ assert.strictEqual(writes, 0, 'no PUT, POST or DELETE reached a bucket');
++ const src = fs.readFileSync(path.join(__dirname, '..', 'server', 'vod', 'orphans-report.js'), 'utf8');
++ assert.ok(!/unlink|rmSync|rmdirSync|renameSync|DeleteObject|deleteObject|uploadFile|copyBetweenProviders|\b(DELETE|UPDATE|INSERT)\s/.test(src), 'the report module holds no delete, move or write');
++ console.log('✅ report only: files, rows and buckets are unchanged; the module has no delete, move or write');
++
++ // ── The job: service-wide, scheduled, not for tenants ──
++ assert.strictEqual(storageJob.schedule(Date.now()), 1, 'the monthly scan is queued');
++ assert.strictEqual(storageJob.schedule(Date.now()), 0, 'once per period');
++ process.env.MEDIA_ORPHAN_SCAN_DAYS = '0';
++ assert.strictEqual(storageJob.schedule(Date.now() + 90 * 86400 * 1000), 0, 'MEDIA_ORPHAN_SCAN_DAYS=0 schedules nothing');
++ delete process.env.MEDIA_ORPHAN_SCAN_DAYS;
++ const job = db.get("SELECT * FROM media_jobs WHERE job_type = 'storage.orphans.scan'");
++ assert.deepStrictEqual([job.app_id, job.status, job.created_by], [queue.SYSTEM_APP, 'queued', 'system:schedule']);
++ const done = await require('../server/jobs/worker').runNow(job.id);
++ assert.strictEqual(done.status, 'succeeded', done.error || '');
++ const result = JSON.parse(done.result);
++ assert.strictEqual(result.totals.unreferenced_local.files, 5);
++ const reportFile = path.join(tmp, 'data', 'reports', result.report);
++ assert.ok(fs.existsSync(reportFile), 'the job writes its report under <data>/reports');
++ assert.strictEqual(JSON.parse(fs.readFileSync(reportFile, 'utf8')).job_id, job.id);
++ assert.deepStrictEqual(snapshotFiles(), before.files, 'the job deleted nothing either');
++ assert.strictEqual(writes, 0);
++
++ const express = require('express');
++ const app = express();
++ app.use(express.json());
++ app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));
++ const server = await new Promise((r) => { const s = app.listen(0, '127.0.0.1', () => r(s)); });
++ const post = (p, body) => new Promise((resolve, reject) => {
++ const data = JSON.stringify(body);
++ const rq = http.request({ host: '127.0.0.1', port: server.address().port, path: p, method: 'POST',
++ headers: { Authorization: 'Bearer live-key-orphans', 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(data) } },
++ (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: JSON.parse(b || '{}') })); });
++ rq.on('error', reject);
++ rq.end(data);
++ });
++ const r = await post('/api/v2/live/jobs', { type: 'storage.orphans.scan' });
++ assert.deepStrictEqual([r.status, r.body.code], [403, 'media.job.forbidden'], 'a tenant cannot run the service-wide scan');
++ server.close();
++ console.log('✅ storage.orphans.scan: scheduled once per period under the system app, writes its report, refused to tenants');
++
++ // ── The script, on demand ──
++ const out = path.join(tmp, 'script-report.json');
++ const text = execFileSync(process.execPath, [path.join(__dirname, '..', 'scripts', 'vods-orphans-report.js'), '--storage', '--no-remote', '--out', out],
++ { env: { ...process.env, ...env }, encoding: 'utf8' });
++ assert.ok(/Unreferenced local files: 5/.test(text) && /Nothing was deleted or moved/.test(text), text);
++ const scripted = JSON.parse(fs.readFileSync(out, 'utf8'));
++ assert.deepStrictEqual([scripted.totals.unreferenced_local.files, Object.keys(scripted.scope.providers).length], [5, 0], '--no-remote lists no bucket');
++ assert.deepStrictEqual(snapshotFiles().filter(f => !f.startsWith('script-report.json')), before.files);
++ console.log('✅ scripts/vods-orphans-report.js --storage runs the same report on demand');
++
++ s3.close();
++ db.close();
++ fs.rmSync(tmp, { recursive: true, force: true });
++ console.log('storage orphans: all checks passed');
++ process.exit(0);
++})().catch((err) => { console.error(err); process.exit(1); });