openvibers/openvibe.media Commits
Record: decode-gate finalize and repair broken stream copies (#392)
Changes
@@@ -2,6 +2,21 @@## Unreleased++- **A recording that does not decode is never published (2026-11-10).** A recording is a stream copy of a WebRTC++ ingest, and WebRTC does not retransmit to a recording consumer: a lost RTP fragment arrives as a truncated H.264++ access unit, the muxer writes it, ffprobe calls the file healthy, and Chrome stops the whole playback on it++ (MEDIA_ERR_DECODE; VODs 6297/6298, the robot phone's streams, each had a truncated IDR — the keyframe at 1.175 s / 3.120 s — which ffmpeg concealed until a later P-frame at 1.7 s / 3.6 s, while Chrome rejected the IDR).++ `finalize` now decodes the opening 10 s with `ffmpeg -xerror` (`server/vod/decode-check.js`): a decoder error line++ matters even when a single-threaded ffmpeg exits 0, while the null muxer's DTS warnings from variable frame timing++ do not. A failing recording is measured, marked `needs_repair`, quarantined and hidden; `server/vod/vod-repair.js`++ re-encodes it in place (H.264/AAC, original timing and duration, verified output), settles the row, lifts the++ quarantine, restores visibility, announces `vod.ready` and re-cuts its timeline so the streaming playlist serves the++ repaired bytes too. The health job repairs `needs_repair` rows first on its next idle pass, and a deep scan that++ finds a decode fault repairs it instead of quarantining it as corrupt. Operators can use++ `node scripts/repair-vod-decode.js --ids … [--full] [--apply]`. Tests: `test/decode-check.test.js` (a corrupt++ P-frame fixture generated with ffmpeg) and `test/vod-decode-repair.test.js` (finalize marks, repair publishes).++ The Media recorder also logs incomplete-RTP-frame warnings, so packet loss is visible at record time.++- Avatar ingestion, image variants, and the upload-time `POST /api/v2/:app/objects/normalize-image` operation share`server/images/orientation.js`. The operation applies EXIF orientation to pixels before Sharp drops the sourcemetadata, for first-party uploaders such as Community. Test: `test/image-orientation.test.js` covers all eight values.@@@ -196,6 +196,21 @@ med_xyz/{captions/*.vtt, storyboard.webp+.vtt, waveform, metadata}With transcoding enabled, new jobs use a `:transcode` idempotency suffix. Lazy discovery retries an earliercopy-only `codec_unsupported` failure once under that key. A failed transcode job is not queued again; the hourlylazy backfill cap still applies.++- **F3.d, shipped: the decode gate (2026-11-10).** A recording is a stream copy of a WebRTC ingest, so a lost RTP++ fragment is a truncated H.264 access unit that the muxer writes without decoding. ffprobe sees a healthy file and++ ffmpeg conceals the frame, but Chrome stops the whole playback on it (`MEDIA_ERR_DECODE`; VODs 6297/6298, the robot++ phone's 2026-11-10 streams, each had a truncated IDR — the keyframe at 1.175 s / 3.120 s, the large burst a lossy uplink is most likely to lose a fragment of; ffmpeg concealed it and only complained at a later P-frame at 1.7 s / 3.6 s, while Chrome rejected the IDR). `server/vod/decode-check.js` decodes++ the opening 10 s with `ffmpeg -xerror` and treats a decoder error line as a fault even when a single-threaded ffmpeg++ exits 0 (the null muxer's DTS warnings from variable frame timing are not a fault). `finalize` runs it before the++ ready transition: a failing recording is measured, marked `needs_repair`, quarantined and hidden, and its bytes are++ kept. `server/vod/vod-repair.js` re-encodes that file in place to H.264 High/AAC on its original timing++ (`-fps_mode passthrough`, keyframe every 4 s), verifies the output, settles duration/size/health in the row and its++ object together, lifts the quarantine, restores the owner's visibility, announces `vod.ready` and queues an++ `object.cmaf` re-cut under `object.cmaf:<id>:decode_repair:<file version>` so the source playlist serves the++ repaired bytes too. The health job takes `needs_repair` rows first on its next idle pass, and a deep scan that++ finds `decode_failed` repairs instead of quarantining. The operator path is++ `node scripts/repair-vod-decode.js [--ids …] [--full] [--apply]` (dry run by default). Tests:++ `test/decode-check.test.js`, `test/vod-decode-repair.test.js`.- **F3.5, shipped: virtual clips over the timeline (§4).**- **Local timeline cache, shipped:** `OBJECTS_PATH/.timeline` is bounded by`MEDIA_TIMELINE_LOCAL_MAX_BYTES` (10 GiB by default; 0 keeps no new local copies once durable) andnnew file mode 100644@@@ -0,0 +1,79 @@++#!/usr/bin/env node++'use strict';++/**++ * Repair recordings that do not decode (server/vod/vod-repair.js), for the operator.++ *++ * A recording is a stream copy of a WebRTC ingest: a lost RTP fragment leaves a truncated H.264 access unit. The muxer++ * writes it, ffprobe calls the file healthy, and a browser stops the whole playback on it (VODs 6297/6298, the robot++ * phone's 2026-11-10 streams: one corrupt P-frame each). The finalize decode check marks such a recording++ * `needs_repair` and hides it; the health job repairs it on its next idle pass. This script is the explicit path for++ * a known id (and for recordings finalized before the check existed).++ *++ * node scripts/repair-vod-decode.js [--ids 6297,6298] [--days 7] [--limit 50] [--app live] [--full]++ * [--no-recut] [--apply] [--json]++ * Dry run (the default): decode the opening 10 s of each finished VOD (--full: the whole file) and report the++ * ones that fail, with the first decoder errors. Changes nothing.++ * --apply: re-encode each broken file in place (the corrupt access unit becomes the decoder's concealed picture;++ * duration and the original timing are kept), verify it, update the duration/size/health through Media's own row++ * writes, lift the quarantine, announce vod.ready and queue a fresh object.cmaf cut (--no-recut skips the cut)++ * so the streaming playlist serves the repaired bytes too. The running service's worker cuts it on its next poll.++ * (the database is the service's: DATABASE_URL, and MEDIA_HLS_ENABLED for the re-cut, from its env file)++ *++ * Exit codes: 0 done, 1 error, 2 bad usage.++ */++(async () => {++ const args = process.argv.slice(2);++ const has = (name) => args.includes(`--${name}`);++ const arg = (name, d = null) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] && !args[i + 1].startsWith('--') ? args[i + 1] : d; };++ const usage = (msg) => { console.error(`repair-vod-decode: ${msg}`); process.exit(2); };++ for (const flag of ['ids', 'days', 'limit', 'app']) if (has(flag) && !arg(flag)) usage(`--${flag} needs a value`);++ const APPLY = has('apply');++ const FULL = has('full');++ const RECUT = !has('no-recut');++ const days = Number(arg('days', 7));++ const limit = Number(arg('limit', 50));++ if (!(days > 0) || !(limit > 0)) usage('--days and --limit are positive numbers');++ const ids = arg('ids') ? arg('ids').split(',').map(Number).filter((n) => n > 0) : null;++ if (has('ids') && (!ids || !ids.length)) usage('--ids needs one or more VOD ids');++++ await require('../server/db/database').initDb(); // PostgreSQL (DATABASE_URL), as the service++ const db = require('../server/db/database');++ const repair = require('../server/vod/vod-repair');++ if (APPLY) {++ await require('../server/events').initWriter();++ if (RECUT && !require('../server/config').hls.enabled) console.warn('repair-vod-decode: MEDIA_HLS_ENABLED is off — files are repaired, no timeline is cut');++ }++++ const since = new Date(Date.now() - days * 86400000).toISOString().slice(0, 19).replace('T', ' ');++ const vods = ids++ ? await db.all('SELECT * FROM vods WHERE id = ANY(?) ORDER BY id DESC', [ids])++ : await db.all(`SELECT * FROM vods WHERE COALESCE(is_recording, 0) = 0 AND created_at >= ?++ ${has('app') ? 'AND app_id = ?' : ''} ORDER BY id DESC LIMIT ?`, has('app') ? [since, arg('app'), limit] : [since, limit]);++ if (!vods.length) usage('no VODs matched');++++ const report = [];++ for (const vod of vods) {++ const out = { vod: vod.id, title: vod.title, health: vod.health_status };++ report.push(out);++ try {++ const r = await repair.repairVod(vod, { full: FULL, apply: APPLY, recut: RECUT, announceReady: APPLY });++ Object.assign(out, r);++ } catch (err) {++ out.error = String(err.message || err).slice(0, 300);++ }++ }++ if (has('json')) console.log(JSON.stringify(report, null, 2));++ else {++ for (const r of report) {++ const what = r.error ? `error: ${r.error}` : r.skipped === 'clean' ? 'clean'++ : r.skipped === 'needs_repair' && !APPLY ? `needs repair (${(r.before.errors || [])[0] || `exit ${r.before.code}`})`++ : r.skipped ? `skip: ${r.skipped}`++ : r.ok ? `repaired → ${r.duration}s, ${(r.size / 1048576).toFixed(1)}MB${r.recut ? `, re-cut queued ${r.recut}` : ''}${r.recutError ? ` (re-cut error: ${r.recutError})` : ''}`++ : 'failed';++ console.log(`vod ${r.vod} ${what}`);++ }++ const broken = report.filter((r) => r.before && !r.before.ok).length;++ console.log(`${report.length} checked, ${broken} do not decode${APPLY ? `, ${report.filter((r) => r.ok && !r.skipped).length} repaired` : broken && !FULL ? ' (dry run: --apply repairs; --full decodes the whole file)' : ''}`);++ }++ process.exit(report.some((r) => r.error) ? 1 : 0);++})().catch((err) => { console.error(err); process.exit(1); });@@@ -425,9 +425,11 @@ async function getVodsNeedingHealthScan({ staleDays = 30, limit = 3 } = {}) {return await all(`SELECT * FROM vodsWHERE COALESCE(is_recording, 0) = 0AND (health_status IS NULL OR health_status NOT IN ('corrupt','zero_byte','missing_file'))-- AND (last_health_scan_at IS NULL OR last_health_scan_at <= datetime('now', ?)++ AND (health_status = 'needs_repair'++ OR last_health_scan_at IS NULL OR last_health_scan_at <= datetime('now', ?)OR (COALESCE(file_size, 0) BETWEEN 1 AND 1023 AND COALESCE(health_status, 'ok') = 'ok'))-- ORDER BY (last_health_scan_at IS NULL) DESC, (COALESCE(file_size, 0) BETWEEN 1 AND 1023) DESC, last_health_scan_at ASC++ ORDER BY (health_status = 'needs_repair') DESC, (last_health_scan_at IS NULL) DESC,++ (COALESCE(file_size, 0) BETWEEN 1 AND 1023) DESC, last_health_scan_at ASCLIMIT ?`, [`-${Math.max(1, staleDays)} days`, limit]);}@@@ -148,4 +148,25 @@ async function queueRendition(appId, objectId, { rendition = null } = {}) {return first;}--module.exports = { queueCmaf, queuePack, queueRendition, isVirtualClip };++/**++ * Cut a ready object's timeline again from its current bytes, even when a source timeline already exists. A repair++ * after a bad recording replaces the file (server/vod/vod-repair.js), so the old rows name the broken copy: this queues++ * object.cmaf under a key unique to that file version. The job replaces the source rows in one transaction and deletes++ * the previous cut's bytes, so the same hls_url starts serving the repaired video. → the job id, or null.++ */++async function queueRecut(appId, objectId, { reason = 'repair', version = null, segmentSeconds = null } = {}) {++ if (!config.hls.enabled || !appId || !objectId) return null;++ const obj = await require('./model').getObject(objectId);++ if (!obj || obj.lifecycle_status !== 'ready' || !require('../jobs/derive').isMediaObject(obj)) return null;++ const queue = require('../jobs/queue');++ const params = require('../jobs/cmaf').spec.validate({ appId, obj, params: segmentSeconds != null ? { segment_seconds: segmentSeconds } : {} });++ const idempotencyKey = `object.cmaf:${obj.id}:${reason}${version ? `:${version}` : ''}`;++ const r = await queue.enqueue({++ appId, type: 'object.cmaf', objectId: obj.id, params,++ dedupeActive: true, idempotencyKey, createdBy: `system:${reason}`,++ });++ if (r.created) require('../jobs/worker').kick();++ return r.job ? r.job.id : null;++}++++module.exports = { queueCmaf, queuePack, queueRendition, queueRecut, isVirtualClip };nnew file mode 100644@@@ -0,0 +1,151 @@++'use strict';++/**++ * OpenVibe.Media — does a finished recording decode, and can a broken one be made good again?++ *++ * A recording is a stream copy of what the broadcaster sent. WebRTC has no retransmission to a recording consumer, so++ * a lost RTP packet arrives as a truncated H.264 access unit: ffmpeg's muxer writes it (it does not decode), and++ * browsers are not forgiving — Chrome stops the whole pipeline with MEDIA_ERR_DECODE on the first bad frame. The robot++ * phone's 2026-11-10 VODs 6297/6298 each had a truncated IDR (the keyframe at 1.175 s / 3.120 s, the large burst a++ * lossy uplink is most likely to lose a fragment of); ffmpeg concealed it and only complained at a later P-frame that++ * referenced it (1.7 s / 3.6 s), while Chrome rejected the IDR itself. A plain ffprobe never noticed either.++ *++ * scan(file, { seconds = 10, timeoutMs }) decode the first seconds with `ffmpeg -xerror` and read the decoder's++ * error lines. `ok` only when the exit code is 0 AND no serious decode++ * error was logged: a tiny 1-thread stream can exit 0 on a bad frame, a++ * big multi-thread one exits 183, and the null muxer's "non monotonically++ * increasing dts" is a VFR artefact, never a fault. `seconds: 0` decodes++ * the whole file (the operator repair script does this).++ * repair(file, { durationSeconds }) re-encode the video to H.264/AAC in a temporary MP4 beside the file, verify++ * that output's first seconds and its duration, then rename it over the++ * file. A broken span cannot be cut out of a stream copy (every P-frame++ * after it references the bad picture), so a decode + clean encode is the++ * repair; the caller only does it on a file that failed `scan`.++ *++ * `repair` re-encodes and cannot recover the lost bytes: it replaces the corrupt access unit with the decoder's++ * concealed picture, so the result is a valid stream with a short glitch at the repair point. Video is H.264 High++ * (browser-safe) on its original timing (`-fps_mode passthrough`), with a keyframe every 4 s for the player and the++ * CMAF cut; audio is AAC. The file is replaced atomically on the same filesystem, so a reader never sees a half file.++ */++const fs = require('fs');++const path = require('path');++const { spawn } = require('child_process');++const tools = require('./media-tools');++++const FIRST_SECONDS = 10;++const SCAN_TIMEOUT_MS = 120000;++const REPAIR_MIN_FREE_BYTES = 1024 * 1024 * 1024;++const KEYFRAME_SECONDS = 4;++++// A decoder error means the bytes are not the stream they claim to be. The null muxer's DTS complaints are the++// variable frame timing of a wall-clock-timestamped RTP capture, not a fault: every healthy recording logs them.++const SERIOUS = /error while decoding|corrupted macroblock|corrupt decoded frame|Error processing packet|Invalid level prefix|cbp too large|negative number of zero coeffs|Invalid NAL unit size|missing picture in access unit|concealing \d+ corrupted|decode_slice_header error|no frame!/i;++++function _run(args, timeoutMs) {++ return new Promise((resolve) => {++ let proc;++ try { proc = spawn('ffmpeg', args, { stdio: ['ignore', 'ignore', 'pipe'] }); } catch (err) { return resolve({ code: -1, stderr: err.message, timedOut: false }); }++ try { require('os').setPriority(proc.pid, 15); } catch { /* best effort */ }++ let stderr = '';++ let timedOut = false;++ const timer = setTimeout(() => { timedOut = true; try { proc.kill('SIGKILL'); } catch { /* gone */ } }, timeoutMs);++ proc.stderr.on('data', (d) => { stderr = (stderr + d.toString()).slice(-20000); });++ proc.on('error', (err) => { clearTimeout(timer); resolve({ code: -1, stderr: err.message, timedOut }); });++ proc.on('close', (code) => { clearTimeout(timer); resolve({ code, stderr, timedOut }); });++ });++}++++function seriousLines(stderr) {++ return String(stderr || '').split('\n').map((l) => l.trim()).filter((l) => l && SERIOUS.test(l));++}++++/**++ * Decode `filePath`'s first `seconds` (0 = the whole file) and say whether it is clean.++ * → { ok, code, timedOut, errors, seconds }++ */++async function scan(filePath, { seconds = FIRST_SECONDS, timeoutMs = SCAN_TIMEOUT_MS } = {}) {++ const args = ['-nostdin', '-v', 'warning', '-xerror', '-i', filePath, '-map', '0:v:0?', '-map', '0:a:0?'];++ if (seconds > 0) args.push('-t', String(seconds));++ // The null muxer duplicates/drops frames to a constant rate by default, which invents duplicate DTS on a++ // variable-timestamp capture; passthrough decodes the frames as they are, like a player's decoder does.++ args.push('-fps_mode', 'passthrough', '-f', 'null', '-');++ const r = await _run(args, timeoutMs);++ const errors = seriousLines(r.stderr);++ // "The input would not open" (missing file, unreadable container) is not a decode fault: finalize's stat and probe++ // paths already settle that. ffmpeg returns the same AVERROR_INVALIDDATA exit (183) for both, so an unopenable++ // input is told apart by its open error lines.++ const openFailed = /Error opening input|moov atom not found|detected only with low score|could not find codec parameters/i.test(r.stderr);++ return {++ ok: r.code === 0 && !r.timedOut && errors.length === 0,++ fault: errors.length > 0 || (r.code === 183 && !openFailed),++ code: r.code,++ timedOut: r.timedOut,++ errors: errors.slice(0, 8),++ seconds,++ };++}++++function _freeBytes(dir) {++ try {++ const st = fs.statfsSync(dir);++ return Number(st.bavail) * Number(st.bsize);++ } catch { return Infinity; }++}++++function _timeoutMs(durationSeconds) {++ const estimated = (Number(durationSeconds) || 300) * 4000 + 60000;++ return Math.max(5 * 60000, Math.min(3 * 3600 * 1000, estimated));++}++++/**++ * Re-encode a broken MP4 in place (atomically) and verify the result. Refuses anything that is not a local MP4 with++ * a video stream, or when there is no room for the temporary file.++ * → { ok, error?, duration?, size? }++ */++async function repair(filePath, { durationSeconds = 0, timeoutMs = null } = {}) {++ if (!/\.mp4$/i.test(String(filePath))) return { ok: false, error: 'not_an_mp4' };++ let stat;++ try { stat = fs.statSync(filePath); } catch (err) { return { ok: false, error: `missing: ${err.message}` }; }++ const info = await tools.probeVodInfo(filePath);++ const video = (info.streams || []).find((s) => s.codec_type === 'video');++ if (!video) return { ok: false, error: 'no_video' };++ if (_freeBytes(path.dirname(filePath)) < stat.size + REPAIR_MIN_FREE_BYTES) return { ok: false, error: 'insufficient_disk' };++++ const tmp = `${filePath}.repair-${process.pid}.mp4`;++ await tools.removeFile(tmp);++ const fullRange = /^(pc|j)$/i.test(String(video.color_range || '')) || /^yuvj/i.test(String(video.pix_fmt || ''));++ const args = [++ '-y', '-nostdin', '-v', 'error',++ '-fflags', '+genpts', '-i', filePath,++ '-map', '0:v:0', '-map', '0:a:0?',++ ...(fullRange ? ['-vf', 'scale=in_range=pc:out_range=tv'] : []),++ '-c:v', 'libx264', '-preset', 'veryfast', '-crf', '21', '-pix_fmt', 'yuv420p', '-profile:v', 'high',++ '-fps_mode', 'passthrough',++ '-force_key_frames', `expr:gte(t,n_forced*${KEYFRAME_SECONDS})`, '-sc_threshold', '0',++ '-c:a', 'aac', '-b:a', '160k',++ '-map_metadata', '-1', '-movflags', '+faststart', '-max_muxing_queue_size', '1024',++ '-f', 'mp4', tmp,++ ];++ const r = await _run(args, timeoutMs || _timeoutMs(durationSeconds || info.duration));++ if (r.code !== 0 || r.timedOut || !fs.existsSync(tmp)) {++ await tools.removeFile(tmp);++ const tail = String(r.stderr || '').trim().split('\n').slice(-3).join(' ').slice(0, 300);++ return { ok: false, error: `encode_failed: ${r.timedOut ? 'timed out' : `exit ${r.code}`}${tail ? `: ${tail}` : ''}` };++ }++ const verified = await scan(tmp);++ const out = await tools.probeVodInfo(tmp);++ if (!verified.ok || !(out.duration > 0)) {++ await tools.removeFile(tmp);++ return { ok: false, error: `verify_failed: ${verified.errors[0] || 'no duration'}` };++ }++ try {++ fs.renameSync(tmp, filePath);++ } catch (err) {++ await tools.removeFile(tmp);++ return { ok: false, error: `replace_failed: ${err.message}` };++ }++ // The live DVR sidecars describe the bytes we just replaced.++ await tools.removeFile(filePath.replace(/\.mp4$/i, '.seekable.mp4'));++ return { ok: true, duration: out.duration, size: fs.statSync(filePath).size };++}++++module.exports = { scan, repair, seriousLines, FIRST_SECONDS };@@@ -192,7 +192,7 @@ async function _queueRetry(vodId, reason, opts = {}) {// Health issues finalize itself records when it cannot settle a recording. A later successful// finalize (the vod.finalize retry job) lifts the quarantine they caused.--const FINALIZE_ISSUES = ['probe_failed', 'inflated_duration', 'stat_failed', 'finalize_failed'];++const FINALIZE_ISSUES = ['probe_failed', 'inflated_duration', 'stat_failed', 'finalize_failed', 'decode_failed'];function _issuesOf(vod) {try { const v = JSON.parse(vod.health_issues_json || '[]'); return Array.isArray(v) ? v : []; } catch { return []; }@@@ -349,6 +349,22 @@ async function _doFinalize(vodId, opts) {return await db.getVodById(vodId);}++ // ── Does it decode? ──++ // A stream copy can carry a truncated access unit (a lost RTP fragment): ffmpeg's muxer writes it without decoding,++ // ffprobe sees a healthy file, and Chrome stops the whole playback on the first bad frame (VODs 6297/6298,++ // 2026-11-10). Decode the opening seconds before the ready transition. A recording that fails is marked++ // needs_repair and hidden; the health job re-encodes it (server/vod/vod-repair.js), lifts the quarantine and++ // re-cuts its timeline. Failure here is never fatal to the bytes: the file stays for that repair.++ let decode = { ok: true };++ try { decode = await require('./decode-check').scan(filePath); }++ catch (err) { console.warn(`[VOD] vod ${vodId}: decode check could not run (${err.message})`); }++ if (decode.fault) {++ console.warn(`[VOD] vod ${vodId}: first ${decode.seconds}s do not decode (${decode.errors[0] || `exit ${decode.code}${decode.timedOut ? ', timed out' : ''}`}) — needs_repair`);++ await _commitVodOutcome(vod, 'vod.failed', async () => await db.run(`UPDATE vods SET is_recording = 0, duration_seconds = ?, duration_source = ?, health_status = 'needs_repair', health_issues_json = ?, probe_duration_seconds = ?, probe_format_json = ?, last_health_scan_at = datetime('now'), quarantined_at = datetime('now'), is_public = 0 WHERE id = ?`,++ [stored, durationSource, JSON.stringify(['decode_failed', ...issues]), measured, probeFormatJson, vodId]));++ return await db.getVodById(vodId);++ }++let stat;try {stat = fs.statSync(filePath);@@@ -22,6 +22,7 @@ const path = require('path');const db = require('../db/database');const config = require('../config');const scanner = require('./health-scanner');++const repair = require('./vod-repair');const TICK_MS = 5 * 60 * 1000; // scan pass every 5 minutesconst SCAN_STALE_DAYS = 45; // re-scan a healthy VOD at most this often@@@ -43,11 +44,44 @@ function _anyRecording() {// (fast, non-destructive); the expensive full decode + re-encode only runs to// REPAIR a file that already looks broken, and only while idle (`deep`).async function _scanOne(vod, { deep }) {++ // A recording the finalize decode check marked needs_repair (or an earlier repair failed): re-encode it, verify it,++ // lift its quarantine and re-cut its timeline. Only while idle, like the other heavy recovery work.++ if (vod.health_status === 'needs_repair') {++ if (!deep) return { id: vod.id, status: 'deferred' };++ try {++ const r = await repair.repairVod(vod, { apply: true, recut: true, announceReady: true });++ if (r.ok) {++ console.log(`[VOD-Health] Repaired vod ${vod.id} (${r.skipped || `${r.duration}s`})${r.recut ? `, re-cut queued ${r.recut}` : ''}`);++ return { id: vod.id, status: 'repaired' };++ }++ console.warn(`[VOD-Health] Repair failed for vod ${vod.id}: ${r.error || r.skipped}`);++ } catch (e) {++ console.warn(`[VOD-Health] Repair error for vod ${vod.id}:`, e.message);++ }++ // Never probe a failed repair into 'ok': the row stays needs_repair and hidden, and the next idle pass retries.++ return { id: vod.id, status: 'needs_repair' };++ }++let scan = await scanner.scanVod(vod, { decode: false, repairDuration: true, quarantineBad: false });const broken = ['corrupt', 'zero_byte', 'missing_file', 'needs_review'].includes(scan.status)|| (scan.issues || []).some(i => /decode_failed|probe_failed|invalid_duration/.test(i));++ // A deep scan found a decode fault on a recording that predates the finalize check (or whose row was rated ok++ // before): repair it here instead of quarantining it as corrupt. The same re-encode the needs_repair path uses.++ if (deep && vod.file_path && (scan.issues || []).includes('decode_failed')) {++ try {++ const r = await repair.repairVod(vod, { apply: true, recut: true, announceReady: true });++ if (r.ok) {++ console.log(`[VOD-Health] Repaired vod ${vod.id} after a deep decode scan (${r.skipped || `${r.duration}s`})${r.recut ? `, re-cut queued ${r.recut}` : ''}`);++ return { id: vod.id, status: 'repaired' };++ }++ console.warn(`[VOD-Health] Repair after a decode scan failed for vod ${vod.id}: ${r.error || r.skipped}`);++ } catch (e) {++ console.warn(`[VOD-Health] Repair after a decode scan errored for vod ${vod.id}:`, e.message);++ }++ }++if (!broken) {// Healthy (or duration just repaired) — record the clean scan.try { await db.updateVodHealth(vod.id, { status: 'ok', issues: scan.issues || [] }); } catch { /* */ }@@@ -36,17 +36,12 @@ function probeMediaInfo(filePath) {}function decodeVodFile(filePath) {-- return new Promise((resolve) => {-- const proc = spawn('ffmpeg', ['-v', 'warning', '-xerror', '-i', filePath, '-map', '0:v:0', '-f', 'null', '-']);-- let stderr = '';-- proc.stderr.on('data', d => stderr += d.toString());-- proc.on('close', (code) => {-- const warnings = stderr.split('\n').filter(line => /error|warn|corrupt|invalid|timestamp|RTP|concealing|Non-monotonous DTS/i.test(line));-- resolve({ code, stderr, warnings, ok: code === 0 });-- });-- proc.on('error', () => resolve({ code: -1, stderr: '', warnings: [], ok: false }));-- setTimeout(() => { try { proc.kill(); } catch {} }, 60000);-- });++ // Same check as finalize's opening-seconds gate (server/vod/decode-check.js), over the whole file: a decoder error++ // line is a fault even when ffmpeg exits 0 (a single-threaded decode of a small stream conceals the frame and++ // carries on), while the null muxer's DTS complaints from a variable-timestamp recording are not.++ return require('./decode-check').scan(filePath, { seconds: 0, timeoutMs: 60000 }).then((r) => ({++ code: r.code, stderr: '', warnings: r.errors, ok: r.ok, timedOut: r.timedOut,++ }));}// The shared remux (media-tools): a kill budget that scales with the file (this copy's flat 30 s cut@@@ -273,6 +273,16 @@ class StreamRecorder {const line = data.toString();const rec = this.activeRecordings.get(vodId);_trackFfmpegDiagnostics(line, rec);++ // A lost RTP fragment is not an ffmpeg "Error": ffmpeg's H.264 depacketizer reassembles what arrived and++ // the muxer copies it, so the file gets a truncated access unit that browsers refuse to play (VODs++ // 6297/6298, 2026-11-10). Keep the loss visible in the log; the finalize decode check decides what the++ // recording needs. Counted, so a lossy stream cannot flood the journal.++ if (rec && /missed \d+ packets|dropping old packet|packet too late|invalid nal unit size|RTP: missed/i.test(line)) {++ rec.rtpLossWarnings = (rec.rtpLossWarnings || 0) + 1;++ if (rec.rtpLossWarnings === 1 || rec.rtpLossWarnings % 10 === 0) {++ console.warn(`[VOD] vod ${vodId}: ${rec.rtpLossWarnings} incomplete RTP frame warning(s) — last: ${line.trim().slice(0, 160)}`);++ }++ }if (line.includes('Error') || line.includes('error')) {if (_isControlledFfmpegError(line, rec?._expectedShutdown)) return;console.error(`[VOD] FFmpeg error (${label}, vod ${vodId}):`, line.trim());nnew file mode 100644@@@ -0,0 +1,114 @@++'use strict';++/**++ * OpenVibe.Media — repair a finished recording that does not decode.++ *++ * A stream copy can carry a truncated H.264 access unit (a lost RTP fragment in the WebRTC ingest). The muxer writes++ * it, ffprobe calls the file healthy, and a browser stops the whole playback on it (VODs 6297/6298, 2026-11-10). The++ * finalize decode check (server/vod/decode-check.js, server/vod/finalize.js) marks such a recording `needs_repair`++ * and hides it; this module is what turns it back into a playable one:++ *++ * scan (first 10 s, or the whole file for an operator run) → a clean file is left alone;++ * re-encode the file in place (decode-check.repair: the corrupt access unit becomes the decoder's concealed picture,++ * so the output is a valid stream) → probe again;++ * write duration/size/health in the row and its object together, lift the quarantine, and announce vod.ready;++ * cut the timeline again when HLS is on, so the streaming playlist serves the repaired bytes too.++ *++ * The bytes are replaced atomically by decode-check; this module never deletes a recording it could not repair. A VOD++ * on B2/R2 with no local copy is left for the storage tiering path to bring back (→ skipped: 'not_local').++ */++const fs = require('fs');++const db = require('../db/database');++const tools = require('./media-tools');++const decode = require('./decode-check');++const objects = () => require('../objects/model');++const { announce } = require('../announce');++++const FULL_SCAN_TIMEOUT_MS = 30 * 60 * 1000;++++/** Is this VOD one a repair can work on: a local file, not held, not still recording. */++function eligible(vod) {++ if (!vod || !vod.file_path || vod.is_recording) return { ok: false, reason: 'not_finished' };++ if (!fs.existsSync(vod.file_path)) return { ok: false, reason: 'not_local' };++ if (!/\.mp4$/i.test(vod.file_path)) return { ok: false, reason: 'not_mp4' };++ return { ok: true };++}++++/**++ * Write the settled, playable state: duration/size/probe, health ok, quarantine lifted, visibility restored — and++ * announce vod.ready when asked. Row and object commit together (objects.withObject).++ */++async function publishReady(vod, { duration, size, format = null, issues = ['decode_repaired'], announceReady = true }) {++ const stored = Math.round(duration);++ const change = async () => await objects().withObject('vod', vod.id, async () => await db.run(++ `UPDATE vods SET is_recording = 0, duration_seconds = ?, duration_source = 'probe', file_size = ?,++ probe_duration_seconds = ?, probe_format_json = ?, health_status = 'ok', health_issues_json = ?,++ last_health_scan_at = datetime('now'), quarantined_at = NULL,++ is_public = CASE WHEN COALESCE(visibility, 'public') = 'public' THEN 1 ELSE 0 END WHERE id = ?`,++ [stored, size, duration, JSON.stringify(format || {}), JSON.stringify(issues), vod.id]++ ));++ if (announceReady) {++ await announce(vod.app_id, 'vod.ready', {++ change,++ payload: async () => require('./finalize').vodPublic(await db.getVodById(vod.id), { hls: false }),++ });++ } else {++ await change();++ }++ return stored;++}++++/**++ * Scan (and with `apply`, repair) one VOD.++ * → { vod, ok, skipped?, error?, duration?, size?, recut?, before?, after? }++ */++async function repairVod(vod, { full = false, apply = false, recut = true, announceReady = true } = {}) {++ const out = { vod: vod && vod.id, ok: false };++ const can = eligible(vod);++ if (!can.ok) return { ...out, skipped: can.reason };++ try { if (await objects().isHeldRow(vod)) return { ...out, skipped: 'held' }; } catch { /* the scan is read-only */ }++ const scanOpts = { seconds: full ? 0 : decode.FIRST_SECONDS, timeoutMs: full ? FULL_SCAN_TIMEOUT_MS : undefined };++ const before = await decode.scan(vod.file_path, scanOpts);++ out.before = { ok: before.ok, code: before.code, timedOut: before.timedOut, errors: before.errors };++ if (before.ok) {++ // Already clean (a scan marked needs_repair on a transient error, or an earlier repair's row write failed):++ // settle the row without touching the bytes.++ if (apply && vod.health_status !== 'ok') {++ const info = await tools.probeVodInfo(vod.file_path);++ const stat = fs.statSync(vod.file_path);++ const duration = info.duration > 0 ? info.duration : Number(vod.duration_seconds) || 0;++ try {++ const stored = await publishReady(vod, { duration, size: stat.size, format: info.format, issues: ['decode_check_ok'], announceReady });++ return { ...out, ok: true, duration: stored, size: stat.size, issues: ['decode_check_ok'] };++ } catch (err) { return { ...out, error: `stored_failed: ${err.message}` }; }++ }++ return { ...out, ok: true, skipped: 'clean' };++ }++ if (!apply) return { ...out, ok: true, skipped: 'needs_repair' };++++ const fixed = await decode.repair(vod.file_path, { durationSeconds: vod.duration_seconds });++ if (!fixed.ok) return { ...out, error: fixed.error };++++ const info = await tools.probeVodInfo(vod.file_path);++ const duration = info.duration > 0 ? info.duration : Number(vod.duration_seconds) || 0;++ const stat = fs.statSync(vod.file_path);++ const after = await decode.scan(vod.file_path, scanOpts);++ out.after = { ok: after.ok, errors: after.errors };++ if (!after.ok) return { ...out, error: `still_broken: ${after.errors[0] || 'decode'}` };++++ let stored;++ try {++ stored = await publishReady(vod, { duration, size: stat.size, format: info.format, issues: ['decode_repaired'], announceReady });++ } catch (err) {++ return { ...out, error: `stored_failed: ${err.message}` };++ }++++ const version = `${stat.size}-${Math.round(stat.mtimeMs)}`;++ let recutJob = null;++ if (recut && vod.object_id) {++ try { recutJob = await require('../objects/timeline-queue').queueRecut(vod.app_id, vod.object_id, { reason: 'decode_repair', version }); }++ catch (err) { out.recutError = err.message; }++ }++ return { ...out, ok: true, duration: stored, size: stat.size, recut: recutJob };++}++++module.exports = { repairVod, publishReady, eligible, FULL_SCAN_TIMEOUT_MS };nnew file mode 100644@@@ -0,0 +1,63 @@++'use strict';++// Decode check + repair (server/vod/decode-check.js): a lost RTP access unit is a truncated H.264 slice; ffmpeg's++// muxer copies it, Chrome fails the whole playback on it (VODs 6297/6298, 2026-11-10). `scan` must catch it even when++// ffmpeg exits 0, and `repair` must turn the broken fixture into a stream that decodes cleanly. The fixture is made++// here with ffmpeg — no production media.++const assert = require('assert');++const fs = require('fs');++const os = require('os');++const path = require('path');++const { spawnSync } = require('child_process');++++const hasFfmpeg = spawnSync('ffmpeg', ['-version']).status === 0;++const hasX264 = hasFfmpeg && /libx264/.test(String(spawnSync('ffmpeg', ['-hide_banner', '-encoders']).stdout));++++if (!hasX264) {++ console.log('decode-check: skipped (ffmpeg with libx264 not found)');++ process.exit(0);++}++++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-decode-'));++ const clean = path.join(tmp, 'clean.mp4');++ const make = spawnSync('ffmpeg', ['-v', 'error', '-y',++ '-f', 'lavfi', '-i', 'testsrc=duration=6:size=320x180:rate=10',++ '-f', 'lavfi', '-i', 'sine=frequency=440:duration=6',++ '-c:v', 'libx264', '-preset', 'ultrafast', '-pix_fmt', 'yuv420p',++ '-g', '20', '-keyint_min', '20', '-sc_threshold', '0',++ '-c:a', 'aac', '-b:a', '64k', '-movflags', '+faststart', clean], { encoding: null });++ assert.strictEqual(make.status, 0, String(make.stderr));++++ const decode = require('../server/vod/decode-check');++++ // ── 1. A clean recording passes ──++ const good = await decode.scan(clean);++ assert.strictEqual(good.ok, true, JSON.stringify(good));++++ // ── 2. Corrupt one P-frame the way a lost RTP fragment corrupts it: flip bytes in the middle of the slice. ──++ const probe = spawnSync('ffprobe', ['-v', 'error', '-select_streams', 'v:0', '-show_packets', '-of', 'json', clean], { encoding: 'utf8' });++ const packets = JSON.parse(probe.stdout).packets || [];++ const victim = packets.find((p) => !String(p.flags || '').includes('K') && Number(p.size) > 300 && Number(p.pos) > 0);++ assert.ok(victim, 'the fixture has a P-frame to corrupt');++ const bad = path.join(tmp, 'bad.mp4');++ const bytes = fs.readFileSync(clean);++ const at = Number(victim.pos) + 4 + 120; // past the length prefix, into the slice data++ for (let i = 0; i < 12; i++) bytes[at + i] ^= 0xff;++ fs.writeFileSync(bad, bytes);++++ const broken = await decode.scan(bad);++ assert.strictEqual(broken.ok, false, 'the corrupt access unit is detected');++ assert.ok(broken.errors.some((l) => /error while decoding|corrupt decoded frame|Error processing packet/i.test(l)),++ `a decoder error is reported: ${JSON.stringify(broken.errors)}`);++++ // ── 3. Repair replaces the corrupt unit with the decoder's concealed picture and verifies the output ──++ const fixed = await decode.repair(bad);++ assert.strictEqual(fixed.ok, true, JSON.stringify(fixed));++ const after = await decode.scan(bad);++ assert.strictEqual(after.ok, true, `the repaired file decodes cleanly: ${JSON.stringify(after)}`);++ assert.ok(fixed.duration >= 5.5 && fixed.duration <= 6.6, `duration is kept (${fixed.duration}s)`);++ assert.ok(fixed.size > 1024);++++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('✅ decode-check: a corrupt H.264 access unit is detected even on ffmpeg exit 0, and repair re-encodes a clean file with the duration kept');++})().catch((err) => { console.error(err); process.exit(1); });@@@ -29,7 +29,7 @@ const express = require('express');const model = require('../server/objects/model');const timeline = require('../server/objects/timeline');const signing = require('../server/objects/signing');-- const { queueCmaf, queuePack } = require('../server/objects/timeline-queue');++ const { queueCmaf, queuePack, queueRecut } = require('../server/objects/timeline-queue');const tools = require('../server/vod/media-tools');const thumbService = require('../server/thumbnails/thumbnail-service');const vodStorage = require('../server/vod/vod-storage');@@@ -111,6 +111,27 @@ const express = require('express');assert.strictEqual((await jobs('object.cmaf', cut)).length, 0, 'a timeline already there: nothing queued');console.log('✅ nothing queued with the flag off, for a non-media or missing object, or once the timeline exists');++ // ── queueRecut: a repaired file gets a new cut even though a timeline exists, keyed by the file version ──++ const recutId = await queueRecut('live', cut, { reason: 'decode_repair', version: '111-222' });++ assert.ok(recutId && /^mjob_/.test(recutId), 'a repaired file queues a re-cut over the existing timeline');++ let recutJobs = await jobs('object.cmaf', cut);++ assert.strictEqual(recutJobs.length, 1);++ assert.deepStrictEqual([recutJobs[0].status, recutJobs[0].created_by, recutJobs[0].idempotency_key],++ ['queued', 'system:decode_repair', `object.cmaf:${cut}:decode_repair:111-222`]);++ assert.strictEqual(await queueRecut('live', cut, { reason: 'decode_repair', version: '111-222' }), recutId,++ 'the same repaired file joins its queued re-cut');++ // dedupeActive is per object and type: while the cut is queued, a newer version joins it rather than++ // running two cuts at once. The next repair after it settles gets its new key and cuts again.++ assert.strictEqual(await queueRecut('live', cut, { reason: 'decode_repair', version: '333-444' }), recutId,++ 'a newer version while the cut is active joins it');++ recutJobs = await jobs('object.cmaf', cut);++ assert.strictEqual(recutJobs.length, 1);++ config.hls.enabled = false;++ assert.strictEqual(await queueRecut('live', cut, { reason: 'decode_repair', version: '555-666' }), null);++ config.hls.enabled = true;++ assert.strictEqual((await jobs('object.cmaf', cut)).length, 1, 'flag off: nothing queued');++ console.log('✅ queueRecut: a repaired file re-cuts over the existing timeline, joining by file version');++// ── queuePack: a cut queues one object.pack, a re-run joins it; flag off or non-media queue nothing ──const packOff = await media();config.hls.enabled = false;nnew file mode 100644@@@ -0,0 +1,100 @@++'use strict';++// A recording that does not decode must never reach a player: finalize marks it needs_repair and hides it, and++// Media's own repair re-encodes the broken file, verifies it, and lifts the quarantine (server/vod/decode-check.js,++// server/vod/finalize.js, server/vod/vod-repair.js). Fixtures are made here with ffmpeg; no production media.++const assert = require('assert');++const fs = require('fs');++const os = require('os');++const path = require('path');++const { spawnSync } = require('child_process');++++const hasFfmpeg = spawnSync('ffmpeg', ['-version']).status === 0;++const hasX264 = hasFfmpeg && /libx264/.test(String(spawnSync('ffmpeg', ['-hide_banner', '-encoders']).stdout));++if (!hasX264) { console.log('vod-decode-repair: skipped (ffmpeg with libx264 not found)'); process.exit(0); }++++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-vod-repair-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ process.env.VOD_PATH = dir('vods');++ process.env.CLIPS_PATH = dir('clips');++ process.env.THUMBNAILS_PATH = dir('thumbnails');++ process.env.FILES_PATH = dir('files');++ process.env.PASTES_PATH = dir('pastes');++ process.env.OBJECTS_PATH = dir('objects');++ process.env.MEDIA_JOBS_ENABLED = '0';++ process.env.MEDIA_HLS_ENABLED = '0';++++ const db = require('../server/db/database');++ const finalize = require('../server/vod/finalize');++ const repair = require('../server/vod/vod-repair');++ const decode = require('../server/vod/decode-check');++ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });++++ const makeVideo = (file) => {++ const r = spawnSync('ffmpeg', ['-v', 'error', '-y',++ '-f', 'lavfi', '-i', 'testsrc=duration=6:size=320x180:rate=10',++ '-f', 'lavfi', '-i', 'sine=frequency=440:duration=6',++ '-c:v', 'libx264', '-preset', 'ultrafast', '-pix_fmt', 'yuv420p',++ '-g', '20', '-keyint_min', '20', '-sc_threshold', '0',++ '-c:a', 'aac', '-b:a', '64k', '-movflags', '+faststart', file], { encoding: null });++ assert.strictEqual(r.status, 0, String(r.stderr));++ };++ const corruptOneFrame = (file) => {++ const probe = spawnSync('ffprobe', ['-v', 'error', '-select_streams', 'v:0', '-show_packets', '-of', 'json', file], { encoding: 'utf8' });++ const victim = (JSON.parse(probe.stdout).packets || []).find((p) => !String(p.flags || '').includes('K') && Number(p.size) > 300 && Number(p.pos) > 0);++ assert.ok(victim, 'the fixture has a P-frame to corrupt');++ const bytes = fs.readFileSync(file);++ const at = Number(victim.pos) + 4 + 120;++ for (let i = 0; i < 12; i++) bytes[at + i] ^= 0xff;++ fs.writeFileSync(file, bytes);++ };++ async function makeVod(file) {++ const r = await db.run(`INSERT INTO vods (app_id, user_id, title, file_path, is_recording, duration_seconds, is_public, visibility)++ VALUES ('live', 7, 'robot', ?, 1, 0, 1, 'public') RETURNING id`, [file]);++ return Number(r.lastInsertRowid);++ }++ const row = async (id) => await db.get('SELECT * FROM vods WHERE id = ?', [id]);++++ // ── 1. Finalize a corrupt recording: it is measured, hidden and marked needs_repair, never published ──++ const bad = path.join(process.env.VOD_PATH, 'robot.mp4');++ makeVideo(bad);++ corruptOneFrame(bad);++ const badId = await makeVod(bad);++ await finalize.finalizeVod(badId);++ const broken = await row(badId);++ assert.strictEqual(broken.health_status, 'needs_repair', JSON.stringify(broken.health_issues_json));++ assert.strictEqual(broken.is_recording, 0);++ assert.strictEqual(broken.is_public, 0, 'a broken recording is hidden until it is repaired');++ assert.ok(broken.quarantined_at, 'and quarantined for repair');++ assert.ok(JSON.parse(broken.health_issues_json).includes('decode_failed'));++ assert.ok(broken.duration_seconds >= 5 && broken.duration_seconds <= 7, `its measured length is kept (${broken.duration_seconds}s)`);++ assert.ok(fs.existsSync(bad), 'the bytes stay for the repair');++ console.log('✅ finalize: a corrupt recording is needs_repair, hidden, and its bytes are kept');++++ // ── 2. Media's repair: re-encode, verify, lift the quarantine, restore the owner's visibility ──++ const fixed = await repair.repairVod(await row(badId), { apply: true, recut: false, announceReady: false });++ assert.strictEqual(fixed.ok, true, JSON.stringify(fixed));++ assert.ok(!fixed.skipped, 'it was not already clean');++ const good = await row(badId);++ assert.strictEqual(good.health_status, 'ok');++ assert.strictEqual(good.quarantined_at, null, 'the quarantine is lifted');++ assert.strictEqual(good.is_public, 1, 'back to the visibility its owner chose');++ assert.ok(good.duration_seconds >= 5 && good.duration_seconds <= 7, `duration kept (${good.duration_seconds}s)`);++ assert.ok(fs.statSync(bad).size > 1024);++ assert.strictEqual((await decode.scan(bad)).ok, true, 'the repaired file decodes cleanly');++ console.log('✅ repair: the file is re-encoded in place, verified, un-quarantined and public again');++++ // ── 3. A clean recording is not touched and the check does not cry wolf on variable timestamps ──++ const clean = path.join(process.env.VOD_PATH, 'clean.mp4');++ makeVideo(clean);++ const cleanId = await makeVod(clean);++ const cleanSize = fs.statSync(clean).size;++ await finalize.finalizeVod(cleanId);++ const ok = await row(cleanId);++ assert.strictEqual(ok.health_status, 'ok');++ assert.strictEqual(ok.is_public, 1);++ assert.strictEqual(fs.statSync(clean).size, cleanSize, 'a clean file is not re-encoded');++ console.log('✅ finalize: a clean recording is published without touching its bytes');++++ fs.rmSync(tmp, { recursive: true, force: true });++})().catch((err) => { console.error(err); process.exit(1); });
@@ -2,6 +2,21 @@
@@ -2,6 +2,21 @@
## Unreleased
## Unreleased
+- **A recording that does not decode is never published (2026-11-10).** A recording is a stream copy of a WebRTC
+ ingest, and WebRTC does not retransmit to a recording consumer: a lost RTP fragment arrives as a truncated H.264
+ access unit, the muxer writes it, ffprobe calls the file healthy, and Chrome stops the whole playback on it
+ (MEDIA_ERR_DECODE; VODs 6297/6298, the robot phone's streams, each had a truncated IDR — the keyframe at 1.175 s / 3.120 s — which ffmpeg concealed until a later P-frame at 1.7 s / 3.6 s, while Chrome rejected the IDR).
+ `finalize` now decodes the opening 10 s with `ffmpeg -xerror` (`server/vod/decode-check.js`): a decoder error line
+ matters even when a single-threaded ffmpeg exits 0, while the null muxer's DTS warnings from variable frame timing
+ do not. A failing recording is measured, marked `needs_repair`, quarantined and hidden; `server/vod/vod-repair.js`
+ re-encodes it in place (H.264/AAC, original timing and duration, verified output), settles the row, lifts the
+ quarantine, restores visibility, announces `vod.ready` and re-cuts its timeline so the streaming playlist serves the
+ repaired bytes too. The health job repairs `needs_repair` rows first on its next idle pass, and a deep scan that
+ finds a decode fault repairs it instead of quarantining it as corrupt. Operators can use
+ `node scripts/repair-vod-decode.js --ids … [--full] [--apply]`. Tests: `test/decode-check.test.js` (a corrupt
+ P-frame fixture generated with ffmpeg) and `test/vod-decode-repair.test.js` (finalize marks, repair publishes).
+ The Media recorder also logs incomplete-RTP-frame warnings, so packet loss is visible at record time.
+
- Avatar ingestion, image variants, and the upload-time `POST /api/v2/:app/objects/normalize-image` operation share
- Avatar ingestion, image variants, and the upload-time `POST /api/v2/:app/objects/normalize-image` operation share
`server/images/orientation.js`. The operation applies EXIF orientation to pixels before Sharp drops the source
`server/images/orientation.js`. The operation applies EXIF orientation to pixels before Sharp drops the source
metadata, for first-party uploaders such as Community. Test: `test/image-orientation.test.js` covers all eight values.
metadata, for first-party uploaders such as Community. Test: `test/image-orientation.test.js` covers all eight values.
@@ -196,6 +196,21 @@ med_xyz/{captions/*.vtt, storyboard.webp+.vtt, waveform, metadata}
@@ -196,6 +196,21 @@ med_xyz/{captions/*.vtt, storyboard.webp+.vtt, waveform, metadata}
With transcoding enabled, new jobs use a `:transcode` idempotency suffix. Lazy discovery retries an earlier
With transcoding enabled, new jobs use a `:transcode` idempotency suffix. Lazy discovery retries an earlier
copy-only `codec_unsupported` failure once under that key. A failed transcode job is not queued again; the hourly
copy-only `codec_unsupported` failure once under that key. A failed transcode job is not queued again; the hourly
lazy backfill cap still applies.
lazy backfill cap still applies.
+- **F3.d, shipped: the decode gate (2026-11-10).** A recording is a stream copy of a WebRTC ingest, so a lost RTP
+ fragment is a truncated H.264 access unit that the muxer writes without decoding. ffprobe sees a healthy file and
+ ffmpeg conceals the frame, but Chrome stops the whole playback on it (`MEDIA_ERR_DECODE`; VODs 6297/6298, the robot
+ phone's 2026-11-10 streams, each had a truncated IDR — the keyframe at 1.175 s / 3.120 s, the large burst a lossy uplink is most likely to lose a fragment of; ffmpeg concealed it and only complained at a later P-frame at 1.7 s / 3.6 s, while Chrome rejected the IDR). `server/vod/decode-check.js` decodes
+ the opening 10 s with `ffmpeg -xerror` and treats a decoder error line as a fault even when a single-threaded ffmpeg
+ exits 0 (the null muxer's DTS warnings from variable frame timing are not a fault). `finalize` runs it before the
+ ready transition: a failing recording is measured, marked `needs_repair`, quarantined and hidden, and its bytes are
+ kept. `server/vod/vod-repair.js` re-encodes that file in place to H.264 High/AAC on its original timing
+ (`-fps_mode passthrough`, keyframe every 4 s), verifies the output, settles duration/size/health in the row and its
+ object together, lifts the quarantine, restores the owner's visibility, announces `vod.ready` and queues an
+ `object.cmaf` re-cut under `object.cmaf:<id>:decode_repair:<file version>` so the source playlist serves the
+ repaired bytes too. The health job takes `needs_repair` rows first on its next idle pass, and a deep scan that
+ finds `decode_failed` repairs instead of quarantining. The operator path is
+ `node scripts/repair-vod-decode.js [--ids …] [--full] [--apply]` (dry run by default). Tests:
+ `test/decode-check.test.js`, `test/vod-decode-repair.test.js`.
- **F3.5, shipped: virtual clips over the timeline (§4).**
- **F3.5, shipped: virtual clips over the timeline (§4).**
- **Local timeline cache, shipped:** `OBJECTS_PATH/.timeline` is bounded by
- **Local timeline cache, shipped:** `OBJECTS_PATH/.timeline` is bounded by
`MEDIA_TIMELINE_LOCAL_MAX_BYTES` (10 GiB by default; 0 keeps no new local copies once durable) and
`MEDIA_TIMELINE_LOCAL_MAX_BYTES` (10 GiB by default; 0 keeps no new local copies once durable) and
new file mode 100644
new file mode 100644
@@ -0,0 +1,79 @@
@@ -0,0 +1,79 @@
+#!/usr/bin/env node
+'use strict';
+/**
+ * Repair recordings that do not decode (server/vod/vod-repair.js), for the operator.
+ *
+ * A recording is a stream copy of a WebRTC ingest: a lost RTP fragment leaves a truncated H.264 access unit. The muxer
+ * writes it, ffprobe calls the file healthy, and a browser stops the whole playback on it (VODs 6297/6298, the robot
+ * phone's 2026-11-10 streams: one corrupt P-frame each). The finalize decode check marks such a recording
+ * `needs_repair` and hides it; the health job repairs it on its next idle pass. This script is the explicit path for
+ * a known id (and for recordings finalized before the check existed).
+ *
+ * node scripts/repair-vod-decode.js [--ids 6297,6298] [--days 7] [--limit 50] [--app live] [--full]
+ * [--no-recut] [--apply] [--json]
+ * Dry run (the default): decode the opening 10 s of each finished VOD (--full: the whole file) and report the
+ * ones that fail, with the first decoder errors. Changes nothing.
+ * --apply: re-encode each broken file in place (the corrupt access unit becomes the decoder's concealed picture;
+ * duration and the original timing are kept), verify it, update the duration/size/health through Media's own row
+ * writes, lift the quarantine, announce vod.ready and queue a fresh object.cmaf cut (--no-recut skips the cut)
+ * so the streaming playlist serves the repaired bytes too. The running service's worker cuts it on its next poll.
+ * (the database is the service's: DATABASE_URL, and MEDIA_HLS_ENABLED for the re-cut, from its env file)
+ *
+ * Exit codes: 0 done, 1 error, 2 bad usage.
+ */
+(async () => {
+ const args = process.argv.slice(2);
+ const has = (name) => args.includes(`--${name}`);
+ const arg = (name, d = null) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] && !args[i + 1].startsWith('--') ? args[i + 1] : d; };
+ const usage = (msg) => { console.error(`repair-vod-decode: ${msg}`); process.exit(2); };
+ for (const flag of ['ids', 'days', 'limit', 'app']) if (has(flag) && !arg(flag)) usage(`--${flag} needs a value`);
+ const APPLY = has('apply');
+ const FULL = has('full');
+ const RECUT = !has('no-recut');
+ const days = Number(arg('days', 7));
+ const limit = Number(arg('limit', 50));
+ if (!(days > 0) || !(limit > 0)) usage('--days and --limit are positive numbers');
+ const ids = arg('ids') ? arg('ids').split(',').map(Number).filter((n) => n > 0) : null;
+ if (has('ids') && (!ids || !ids.length)) usage('--ids needs one or more VOD ids');
+
+ await require('../server/db/database').initDb(); // PostgreSQL (DATABASE_URL), as the service
+ const db = require('../server/db/database');
+ const repair = require('../server/vod/vod-repair');
+ if (APPLY) {
+ await require('../server/events').initWriter();
+ if (RECUT && !require('../server/config').hls.enabled) console.warn('repair-vod-decode: MEDIA_HLS_ENABLED is off — files are repaired, no timeline is cut');
+ }
+
+ const since = new Date(Date.now() - days * 86400000).toISOString().slice(0, 19).replace('T', ' ');
+ const vods = ids
+ ? await db.all('SELECT * FROM vods WHERE id = ANY(?) ORDER BY id DESC', [ids])
+ : await db.all(`SELECT * FROM vods WHERE COALESCE(is_recording, 0) = 0 AND created_at >= ?
+ ${has('app') ? 'AND app_id = ?' : ''} ORDER BY id DESC LIMIT ?`, has('app') ? [since, arg('app'), limit] : [since, limit]);
+ if (!vods.length) usage('no VODs matched');
+
+ const report = [];
+ for (const vod of vods) {
+ const out = { vod: vod.id, title: vod.title, health: vod.health_status };
+ report.push(out);
+ try {
+ const r = await repair.repairVod(vod, { full: FULL, apply: APPLY, recut: RECUT, announceReady: APPLY });
+ Object.assign(out, r);
+ } catch (err) {
+ out.error = String(err.message || err).slice(0, 300);
+ }
+ }
+ if (has('json')) console.log(JSON.stringify(report, null, 2));
+ else {
+ for (const r of report) {
+ const what = r.error ? `error: ${r.error}` : r.skipped === 'clean' ? 'clean'
+ : r.skipped === 'needs_repair' && !APPLY ? `needs repair (${(r.before.errors || [])[0] || `exit ${r.before.code}`})`
+ : r.skipped ? `skip: ${r.skipped}`
+ : r.ok ? `repaired → ${r.duration}s, ${(r.size / 1048576).toFixed(1)}MB${r.recut ? `, re-cut queued ${r.recut}` : ''}${r.recutError ? ` (re-cut error: ${r.recutError})` : ''}`
+ : 'failed';
+ console.log(`vod ${r.vod} ${what}`);
+ }
+ const broken = report.filter((r) => r.before && !r.before.ok).length;
+ console.log(`${report.length} checked, ${broken} do not decode${APPLY ? `, ${report.filter((r) => r.ok && !r.skipped).length} repaired` : broken && !FULL ? ' (dry run: --apply repairs; --full decodes the whole file)' : ''}`);
+ }
+ process.exit(report.some((r) => r.error) ? 1 : 0);
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -425,9 +425,11 @@ async function getVodsNeedingHealthScan({ staleDays = 30, limit = 3 } = {}) {
@@ -425,9 +425,11 @@ async function getVodsNeedingHealthScan({ staleDays = 30, limit = 3 } = {}) {
return await all(`SELECT * FROM vods
return await all(`SELECT * FROM vods
WHERE COALESCE(is_recording, 0) = 0
WHERE COALESCE(is_recording, 0) = 0
AND (health_status IS NULL OR health_status NOT IN ('corrupt','zero_byte','missing_file'))
AND (health_status IS NULL OR health_status NOT IN ('corrupt','zero_byte','missing_file'))
- AND (last_health_scan_at IS NULL OR last_health_scan_at <= datetime('now', ?)
+ AND (health_status = 'needs_repair'
+ OR last_health_scan_at IS NULL OR last_health_scan_at <= datetime('now', ?)
OR (COALESCE(file_size, 0) BETWEEN 1 AND 1023 AND COALESCE(health_status, 'ok') = 'ok'))
OR (COALESCE(file_size, 0) BETWEEN 1 AND 1023 AND COALESCE(health_status, 'ok') = 'ok'))
- ORDER BY (last_health_scan_at IS NULL) DESC, (COALESCE(file_size, 0) BETWEEN 1 AND 1023) DESC, last_health_scan_at ASC
+ ORDER BY (health_status = 'needs_repair') DESC, (last_health_scan_at IS NULL) DESC,
+ (COALESCE(file_size, 0) BETWEEN 1 AND 1023) DESC, last_health_scan_at ASC
LIMIT ?`, [`-${Math.max(1, staleDays)} days`, limit]);
LIMIT ?`, [`-${Math.max(1, staleDays)} days`, limit]);
}
}
@@ -148,4 +148,25 @@ async function queueRendition(appId, objectId, { rendition = null } = {}) {
@@ -148,4 +148,25 @@ async function queueRendition(appId, objectId, { rendition = null } = {}) {
return first;
return first;
}
}
-module.exports = { queueCmaf, queuePack, queueRendition, isVirtualClip };
+/**
+ * Cut a ready object's timeline again from its current bytes, even when a source timeline already exists. A repair
+ * after a bad recording replaces the file (server/vod/vod-repair.js), so the old rows name the broken copy: this queues
+ * object.cmaf under a key unique to that file version. The job replaces the source rows in one transaction and deletes
+ * the previous cut's bytes, so the same hls_url starts serving the repaired video. → the job id, or null.
+ */
+async function queueRecut(appId, objectId, { reason = 'repair', version = null, segmentSeconds = null } = {}) {
+ if (!config.hls.enabled || !appId || !objectId) return null;
+ const obj = await require('./model').getObject(objectId);
+ if (!obj || obj.lifecycle_status !== 'ready' || !require('../jobs/derive').isMediaObject(obj)) return null;
+ const queue = require('../jobs/queue');
+ const params = require('../jobs/cmaf').spec.validate({ appId, obj, params: segmentSeconds != null ? { segment_seconds: segmentSeconds } : {} });
+ const idempotencyKey = `object.cmaf:${obj.id}:${reason}${version ? `:${version}` : ''}`;
+ const r = await queue.enqueue({
+ appId, type: 'object.cmaf', objectId: obj.id, params,
+ dedupeActive: true, idempotencyKey, createdBy: `system:${reason}`,
+ });
+ if (r.created) require('../jobs/worker').kick();
+ return r.job ? r.job.id : null;
+}
+
+module.exports = { queueCmaf, queuePack, queueRendition, queueRecut, isVirtualClip };
new file mode 100644
new file mode 100644
@@ -0,0 +1,151 @@
@@ -0,0 +1,151 @@
+'use strict';
+/**
+ * OpenVibe.Media — does a finished recording decode, and can a broken one be made good again?
+ *
+ * A recording is a stream copy of what the broadcaster sent. WebRTC has no retransmission to a recording consumer, so
+ * a lost RTP packet arrives as a truncated H.264 access unit: ffmpeg's muxer writes it (it does not decode), and
+ * browsers are not forgiving — Chrome stops the whole pipeline with MEDIA_ERR_DECODE on the first bad frame. The robot
+ * phone's 2026-11-10 VODs 6297/6298 each had a truncated IDR (the keyframe at 1.175 s / 3.120 s, the large burst a
+ * lossy uplink is most likely to lose a fragment of); ffmpeg concealed it and only complained at a later P-frame that
+ * referenced it (1.7 s / 3.6 s), while Chrome rejected the IDR itself. A plain ffprobe never noticed either.
+ *
+ * scan(file, { seconds = 10, timeoutMs }) decode the first seconds with `ffmpeg -xerror` and read the decoder's
+ * error lines. `ok` only when the exit code is 0 AND no serious decode
+ * error was logged: a tiny 1-thread stream can exit 0 on a bad frame, a
+ * big multi-thread one exits 183, and the null muxer's "non monotonically
+ * increasing dts" is a VFR artefact, never a fault. `seconds: 0` decodes
+ * the whole file (the operator repair script does this).
+ * repair(file, { durationSeconds }) re-encode the video to H.264/AAC in a temporary MP4 beside the file, verify
+ * that output's first seconds and its duration, then rename it over the
+ * file. A broken span cannot be cut out of a stream copy (every P-frame
+ * after it references the bad picture), so a decode + clean encode is the
+ * repair; the caller only does it on a file that failed `scan`.
+ *
+ * `repair` re-encodes and cannot recover the lost bytes: it replaces the corrupt access unit with the decoder's
+ * concealed picture, so the result is a valid stream with a short glitch at the repair point. Video is H.264 High
+ * (browser-safe) on its original timing (`-fps_mode passthrough`), with a keyframe every 4 s for the player and the
+ * CMAF cut; audio is AAC. The file is replaced atomically on the same filesystem, so a reader never sees a half file.
+ */
+const fs = require('fs');
+const path = require('path');
+const { spawn } = require('child_process');
+const tools = require('./media-tools');
+
+const FIRST_SECONDS = 10;
+const SCAN_TIMEOUT_MS = 120000;
+const REPAIR_MIN_FREE_BYTES = 1024 * 1024 * 1024;
+const KEYFRAME_SECONDS = 4;
+
+// A decoder error means the bytes are not the stream they claim to be. The null muxer's DTS complaints are the
+// variable frame timing of a wall-clock-timestamped RTP capture, not a fault: every healthy recording logs them.
+const SERIOUS = /error while decoding|corrupted macroblock|corrupt decoded frame|Error processing packet|Invalid level prefix|cbp too large|negative number of zero coeffs|Invalid NAL unit size|missing picture in access unit|concealing \d+ corrupted|decode_slice_header error|no frame!/i;
+
+function _run(args, timeoutMs) {
+ return new Promise((resolve) => {
+ let proc;
+ try { proc = spawn('ffmpeg', args, { stdio: ['ignore', 'ignore', 'pipe'] }); } catch (err) { return resolve({ code: -1, stderr: err.message, timedOut: false }); }
+ try { require('os').setPriority(proc.pid, 15); } catch { /* best effort */ }
+ let stderr = '';
+ let timedOut = false;
+ const timer = setTimeout(() => { timedOut = true; try { proc.kill('SIGKILL'); } catch { /* gone */ } }, timeoutMs);
+ proc.stderr.on('data', (d) => { stderr = (stderr + d.toString()).slice(-20000); });
+ proc.on('error', (err) => { clearTimeout(timer); resolve({ code: -1, stderr: err.message, timedOut }); });
+ proc.on('close', (code) => { clearTimeout(timer); resolve({ code, stderr, timedOut }); });
+ });
+}
+
+function seriousLines(stderr) {
+ return String(stderr || '').split('\n').map((l) => l.trim()).filter((l) => l && SERIOUS.test(l));
+}
+
+/**
+ * Decode `filePath`'s first `seconds` (0 = the whole file) and say whether it is clean.
+ * → { ok, code, timedOut, errors, seconds }
+ */
+async function scan(filePath, { seconds = FIRST_SECONDS, timeoutMs = SCAN_TIMEOUT_MS } = {}) {
+ const args = ['-nostdin', '-v', 'warning', '-xerror', '-i', filePath, '-map', '0:v:0?', '-map', '0:a:0?'];
+ if (seconds > 0) args.push('-t', String(seconds));
+ // The null muxer duplicates/drops frames to a constant rate by default, which invents duplicate DTS on a
+ // variable-timestamp capture; passthrough decodes the frames as they are, like a player's decoder does.
+ args.push('-fps_mode', 'passthrough', '-f', 'null', '-');
+ const r = await _run(args, timeoutMs);
+ const errors = seriousLines(r.stderr);
+ // "The input would not open" (missing file, unreadable container) is not a decode fault: finalize's stat and probe
+ // paths already settle that. ffmpeg returns the same AVERROR_INVALIDDATA exit (183) for both, so an unopenable
+ // input is told apart by its open error lines.
+ const openFailed = /Error opening input|moov atom not found|detected only with low score|could not find codec parameters/i.test(r.stderr);
+ return {
+ ok: r.code === 0 && !r.timedOut && errors.length === 0,
+ fault: errors.length > 0 || (r.code === 183 && !openFailed),
+ code: r.code,
+ timedOut: r.timedOut,
+ errors: errors.slice(0, 8),
+ seconds,
+ };
+}
+
+function _freeBytes(dir) {
+ try {
+ const st = fs.statfsSync(dir);
+ return Number(st.bavail) * Number(st.bsize);
+ } catch { return Infinity; }
+}
+
+function _timeoutMs(durationSeconds) {
+ const estimated = (Number(durationSeconds) || 300) * 4000 + 60000;
+ return Math.max(5 * 60000, Math.min(3 * 3600 * 1000, estimated));
+}
+
+/**
+ * Re-encode a broken MP4 in place (atomically) and verify the result. Refuses anything that is not a local MP4 with
+ * a video stream, or when there is no room for the temporary file.
+ * → { ok, error?, duration?, size? }
+ */
+async function repair(filePath, { durationSeconds = 0, timeoutMs = null } = {}) {
+ if (!/\.mp4$/i.test(String(filePath))) return { ok: false, error: 'not_an_mp4' };
+ let stat;
+ try { stat = fs.statSync(filePath); } catch (err) { return { ok: false, error: `missing: ${err.message}` }; }
+ const info = await tools.probeVodInfo(filePath);
+ const video = (info.streams || []).find((s) => s.codec_type === 'video');
+ if (!video) return { ok: false, error: 'no_video' };
+ if (_freeBytes(path.dirname(filePath)) < stat.size + REPAIR_MIN_FREE_BYTES) return { ok: false, error: 'insufficient_disk' };
+
+ const tmp = `${filePath}.repair-${process.pid}.mp4`;
+ await tools.removeFile(tmp);
+ const fullRange = /^(pc|j)$/i.test(String(video.color_range || '')) || /^yuvj/i.test(String(video.pix_fmt || ''));
+ const args = [
+ '-y', '-nostdin', '-v', 'error',
+ '-fflags', '+genpts', '-i', filePath,
+ '-map', '0:v:0', '-map', '0:a:0?',
+ ...(fullRange ? ['-vf', 'scale=in_range=pc:out_range=tv'] : []),
+ '-c:v', 'libx264', '-preset', 'veryfast', '-crf', '21', '-pix_fmt', 'yuv420p', '-profile:v', 'high',
+ '-fps_mode', 'passthrough',
+ '-force_key_frames', `expr:gte(t,n_forced*${KEYFRAME_SECONDS})`, '-sc_threshold', '0',
+ '-c:a', 'aac', '-b:a', '160k',
+ '-map_metadata', '-1', '-movflags', '+faststart', '-max_muxing_queue_size', '1024',
+ '-f', 'mp4', tmp,
+ ];
+ const r = await _run(args, timeoutMs || _timeoutMs(durationSeconds || info.duration));
+ if (r.code !== 0 || r.timedOut || !fs.existsSync(tmp)) {
+ await tools.removeFile(tmp);
+ const tail = String(r.stderr || '').trim().split('\n').slice(-3).join(' ').slice(0, 300);
+ return { ok: false, error: `encode_failed: ${r.timedOut ? 'timed out' : `exit ${r.code}`}${tail ? `: ${tail}` : ''}` };
+ }
+ const verified = await scan(tmp);
+ const out = await tools.probeVodInfo(tmp);
+ if (!verified.ok || !(out.duration > 0)) {
+ await tools.removeFile(tmp);
+ return { ok: false, error: `verify_failed: ${verified.errors[0] || 'no duration'}` };
+ }
+ try {
+ fs.renameSync(tmp, filePath);
+ } catch (err) {
+ await tools.removeFile(tmp);
+ return { ok: false, error: `replace_failed: ${err.message}` };
+ }
+ // The live DVR sidecars describe the bytes we just replaced.
+ await tools.removeFile(filePath.replace(/\.mp4$/i, '.seekable.mp4'));
+ return { ok: true, duration: out.duration, size: fs.statSync(filePath).size };
+}
+
+module.exports = { scan, repair, seriousLines, FIRST_SECONDS };
@@ -192,7 +192,7 @@ async function _queueRetry(vodId, reason, opts = {}) {
@@ -192,7 +192,7 @@ async function _queueRetry(vodId, reason, opts = {}) {
// Health issues finalize itself records when it cannot settle a recording. A later successful
// Health issues finalize itself records when it cannot settle a recording. A later successful
// finalize (the vod.finalize retry job) lifts the quarantine they caused.
// finalize (the vod.finalize retry job) lifts the quarantine they caused.
-const FINALIZE_ISSUES = ['probe_failed', 'inflated_duration', 'stat_failed', 'finalize_failed'];
+const FINALIZE_ISSUES = ['probe_failed', 'inflated_duration', 'stat_failed', 'finalize_failed', 'decode_failed'];
function _issuesOf(vod) {
function _issuesOf(vod) {
try { const v = JSON.parse(vod.health_issues_json || '[]'); return Array.isArray(v) ? v : []; } catch { return []; }
try { const v = JSON.parse(vod.health_issues_json || '[]'); return Array.isArray(v) ? v : []; } catch { return []; }
@@ -349,6 +349,22 @@ async function _doFinalize(vodId, opts) {
@@ -349,6 +349,22 @@ async function _doFinalize(vodId, opts) {
return await db.getVodById(vodId);
return await db.getVodById(vodId);
}
}
+ // ── Does it decode? ──
+ // A stream copy can carry a truncated access unit (a lost RTP fragment): ffmpeg's muxer writes it without decoding,
+ // ffprobe sees a healthy file, and Chrome stops the whole playback on the first bad frame (VODs 6297/6298,
+ // 2026-11-10). Decode the opening seconds before the ready transition. A recording that fails is marked
+ // needs_repair and hidden; the health job re-encodes it (server/vod/vod-repair.js), lifts the quarantine and
+ // re-cuts its timeline. Failure here is never fatal to the bytes: the file stays for that repair.
+ let decode = { ok: true };
+ try { decode = await require('./decode-check').scan(filePath); }
+ catch (err) { console.warn(`[VOD] vod ${vodId}: decode check could not run (${err.message})`); }
+ if (decode.fault) {
+ console.warn(`[VOD] vod ${vodId}: first ${decode.seconds}s do not decode (${decode.errors[0] || `exit ${decode.code}${decode.timedOut ? ', timed out' : ''}`}) — needs_repair`);
+ await _commitVodOutcome(vod, 'vod.failed', async () => await db.run(`UPDATE vods SET is_recording = 0, duration_seconds = ?, duration_source = ?, health_status = 'needs_repair', health_issues_json = ?, probe_duration_seconds = ?, probe_format_json = ?, last_health_scan_at = datetime('now'), quarantined_at = datetime('now'), is_public = 0 WHERE id = ?`,
+ [stored, durationSource, JSON.stringify(['decode_failed', ...issues]), measured, probeFormatJson, vodId]));
+ return await db.getVodById(vodId);
+ }
+
let stat;
let stat;
try {
try {
stat = fs.statSync(filePath);
stat = fs.statSync(filePath);
@@ -22,6 +22,7 @@ const path = require('path');
@@ -22,6 +22,7 @@ const path = require('path');
const db = require('../db/database');
const db = require('../db/database');
const config = require('../config');
const config = require('../config');
const scanner = require('./health-scanner');
const scanner = require('./health-scanner');
+const repair = require('./vod-repair');
const TICK_MS = 5 * 60 * 1000; // scan pass every 5 minutes
const TICK_MS = 5 * 60 * 1000; // scan pass every 5 minutes
const SCAN_STALE_DAYS = 45; // re-scan a healthy VOD at most this often
const SCAN_STALE_DAYS = 45; // re-scan a healthy VOD at most this often
@@ -43,11 +44,44 @@ function _anyRecording() {
@@ -43,11 +44,44 @@ function _anyRecording() {
// (fast, non-destructive); the expensive full decode + re-encode only runs to
// (fast, non-destructive); the expensive full decode + re-encode only runs to
// REPAIR a file that already looks broken, and only while idle (`deep`).
// REPAIR a file that already looks broken, and only while idle (`deep`).
async function _scanOne(vod, { deep }) {
async function _scanOne(vod, { deep }) {
+ // A recording the finalize decode check marked needs_repair (or an earlier repair failed): re-encode it, verify it,
+ // lift its quarantine and re-cut its timeline. Only while idle, like the other heavy recovery work.
+ if (vod.health_status === 'needs_repair') {
+ if (!deep) return { id: vod.id, status: 'deferred' };
+ try {
+ const r = await repair.repairVod(vod, { apply: true, recut: true, announceReady: true });
+ if (r.ok) {
+ console.log(`[VOD-Health] Repaired vod ${vod.id} (${r.skipped || `${r.duration}s`})${r.recut ? `, re-cut queued ${r.recut}` : ''}`);
+ return { id: vod.id, status: 'repaired' };
+ }
+ console.warn(`[VOD-Health] Repair failed for vod ${vod.id}: ${r.error || r.skipped}`);
+ } catch (e) {
+ console.warn(`[VOD-Health] Repair error for vod ${vod.id}:`, e.message);
+ }
+ // Never probe a failed repair into 'ok': the row stays needs_repair and hidden, and the next idle pass retries.
+ return { id: vod.id, status: 'needs_repair' };
+ }
+
let scan = await scanner.scanVod(vod, { decode: false, repairDuration: true, quarantineBad: false });
let scan = await scanner.scanVod(vod, { decode: false, repairDuration: true, quarantineBad: false });
const broken = ['corrupt', 'zero_byte', 'missing_file', 'needs_review'].includes(scan.status)
const broken = ['corrupt', 'zero_byte', 'missing_file', 'needs_review'].includes(scan.status)
|| (scan.issues || []).some(i => /decode_failed|probe_failed|invalid_duration/.test(i));
|| (scan.issues || []).some(i => /decode_failed|probe_failed|invalid_duration/.test(i));
+ // A deep scan found a decode fault on a recording that predates the finalize check (or whose row was rated ok
+ // before): repair it here instead of quarantining it as corrupt. The same re-encode the needs_repair path uses.
+ if (deep && vod.file_path && (scan.issues || []).includes('decode_failed')) {
+ try {
+ const r = await repair.repairVod(vod, { apply: true, recut: true, announceReady: true });
+ if (r.ok) {
+ console.log(`[VOD-Health] Repaired vod ${vod.id} after a deep decode scan (${r.skipped || `${r.duration}s`})${r.recut ? `, re-cut queued ${r.recut}` : ''}`);
+ return { id: vod.id, status: 'repaired' };
+ }
+ console.warn(`[VOD-Health] Repair after a decode scan failed for vod ${vod.id}: ${r.error || r.skipped}`);
+ } catch (e) {
+ console.warn(`[VOD-Health] Repair after a decode scan errored for vod ${vod.id}:`, e.message);
+ }
+ }
+
if (!broken) {
if (!broken) {
// Healthy (or duration just repaired) — record the clean scan.
// Healthy (or duration just repaired) — record the clean scan.
try { await db.updateVodHealth(vod.id, { status: 'ok', issues: scan.issues || [] }); } catch { /* */ }
try { await db.updateVodHealth(vod.id, { status: 'ok', issues: scan.issues || [] }); } catch { /* */ }
@@ -36,17 +36,12 @@ function probeMediaInfo(filePath) {
@@ -36,17 +36,12 @@ function probeMediaInfo(filePath) {
}
}
function decodeVodFile(filePath) {
function decodeVodFile(filePath) {
- return new Promise((resolve) => {
- const proc = spawn('ffmpeg', ['-v', 'warning', '-xerror', '-i', filePath, '-map', '0:v:0', '-f', 'null', '-']);
- let stderr = '';
- proc.stderr.on('data', d => stderr += d.toString());
- proc.on('close', (code) => {
- const warnings = stderr.split('\n').filter(line => /error|warn|corrupt|invalid|timestamp|RTP|concealing|Non-monotonous DTS/i.test(line));
- resolve({ code, stderr, warnings, ok: code === 0 });
- });
- proc.on('error', () => resolve({ code: -1, stderr: '', warnings: [], ok: false }));
- setTimeout(() => { try { proc.kill(); } catch {} }, 60000);
- });
+ // Same check as finalize's opening-seconds gate (server/vod/decode-check.js), over the whole file: a decoder error
+ // line is a fault even when ffmpeg exits 0 (a single-threaded decode of a small stream conceals the frame and
+ // carries on), while the null muxer's DTS complaints from a variable-timestamp recording are not.
+ return require('./decode-check').scan(filePath, { seconds: 0, timeoutMs: 60000 }).then((r) => ({
+ code: r.code, stderr: '', warnings: r.errors, ok: r.ok, timedOut: r.timedOut,
+ }));
}
}
// The shared remux (media-tools): a kill budget that scales with the file (this copy's flat 30 s cut
// The shared remux (media-tools): a kill budget that scales with the file (this copy's flat 30 s cut
@@ -273,6 +273,16 @@ class StreamRecorder {
@@ -273,6 +273,16 @@ class StreamRecorder {
const line = data.toString();
const line = data.toString();
const rec = this.activeRecordings.get(vodId);
const rec = this.activeRecordings.get(vodId);
_trackFfmpegDiagnostics(line, rec);
_trackFfmpegDiagnostics(line, rec);
+ // A lost RTP fragment is not an ffmpeg "Error": ffmpeg's H.264 depacketizer reassembles what arrived and
+ // the muxer copies it, so the file gets a truncated access unit that browsers refuse to play (VODs
+ // 6297/6298, 2026-11-10). Keep the loss visible in the log; the finalize decode check decides what the
+ // recording needs. Counted, so a lossy stream cannot flood the journal.
+ if (rec && /missed \d+ packets|dropping old packet|packet too late|invalid nal unit size|RTP: missed/i.test(line)) {
+ rec.rtpLossWarnings = (rec.rtpLossWarnings || 0) + 1;
+ if (rec.rtpLossWarnings === 1 || rec.rtpLossWarnings % 10 === 0) {
+ console.warn(`[VOD] vod ${vodId}: ${rec.rtpLossWarnings} incomplete RTP frame warning(s) — last: ${line.trim().slice(0, 160)}`);
+ }
+ }
if (line.includes('Error') || line.includes('error')) {
if (line.includes('Error') || line.includes('error')) {
if (_isControlledFfmpegError(line, rec?._expectedShutdown)) return;
if (_isControlledFfmpegError(line, rec?._expectedShutdown)) return;
console.error(`[VOD] FFmpeg error (${label}, vod ${vodId}):`, line.trim());
console.error(`[VOD] FFmpeg error (${label}, vod ${vodId}):`, line.trim());
new file mode 100644
new file mode 100644
@@ -0,0 +1,114 @@
@@ -0,0 +1,114 @@
+'use strict';
+/**
+ * OpenVibe.Media — repair a finished recording that does not decode.
+ *
+ * A stream copy can carry a truncated H.264 access unit (a lost RTP fragment in the WebRTC ingest). The muxer writes
+ * it, ffprobe calls the file healthy, and a browser stops the whole playback on it (VODs 6297/6298, 2026-11-10). The
+ * finalize decode check (server/vod/decode-check.js, server/vod/finalize.js) marks such a recording `needs_repair`
+ * and hides it; this module is what turns it back into a playable one:
+ *
+ * scan (first 10 s, or the whole file for an operator run) → a clean file is left alone;
+ * re-encode the file in place (decode-check.repair: the corrupt access unit becomes the decoder's concealed picture,
+ * so the output is a valid stream) → probe again;
+ * write duration/size/health in the row and its object together, lift the quarantine, and announce vod.ready;
+ * cut the timeline again when HLS is on, so the streaming playlist serves the repaired bytes too.
+ *
+ * The bytes are replaced atomically by decode-check; this module never deletes a recording it could not repair. A VOD
+ * on B2/R2 with no local copy is left for the storage tiering path to bring back (→ skipped: 'not_local').
+ */
+const fs = require('fs');
+const db = require('../db/database');
+const tools = require('./media-tools');
+const decode = require('./decode-check');
+const objects = () => require('../objects/model');
+const { announce } = require('../announce');
+
+const FULL_SCAN_TIMEOUT_MS = 30 * 60 * 1000;
+
+/** Is this VOD one a repair can work on: a local file, not held, not still recording. */
+function eligible(vod) {
+ if (!vod || !vod.file_path || vod.is_recording) return { ok: false, reason: 'not_finished' };
+ if (!fs.existsSync(vod.file_path)) return { ok: false, reason: 'not_local' };
+ if (!/\.mp4$/i.test(vod.file_path)) return { ok: false, reason: 'not_mp4' };
+ return { ok: true };
+}
+
+/**
+ * Write the settled, playable state: duration/size/probe, health ok, quarantine lifted, visibility restored — and
+ * announce vod.ready when asked. Row and object commit together (objects.withObject).
+ */
+async function publishReady(vod, { duration, size, format = null, issues = ['decode_repaired'], announceReady = true }) {
+ const stored = Math.round(duration);
+ const change = async () => await objects().withObject('vod', vod.id, async () => await db.run(
+ `UPDATE vods SET is_recording = 0, duration_seconds = ?, duration_source = 'probe', file_size = ?,
+ probe_duration_seconds = ?, probe_format_json = ?, health_status = 'ok', health_issues_json = ?,
+ last_health_scan_at = datetime('now'), quarantined_at = NULL,
+ is_public = CASE WHEN COALESCE(visibility, 'public') = 'public' THEN 1 ELSE 0 END WHERE id = ?`,
+ [stored, size, duration, JSON.stringify(format || {}), JSON.stringify(issues), vod.id]
+ ));
+ if (announceReady) {
+ await announce(vod.app_id, 'vod.ready', {
+ change,
+ payload: async () => require('./finalize').vodPublic(await db.getVodById(vod.id), { hls: false }),
+ });
+ } else {
+ await change();
+ }
+ return stored;
+}
+
+/**
+ * Scan (and with `apply`, repair) one VOD.
+ * → { vod, ok, skipped?, error?, duration?, size?, recut?, before?, after? }
+ */
+async function repairVod(vod, { full = false, apply = false, recut = true, announceReady = true } = {}) {
+ const out = { vod: vod && vod.id, ok: false };
+ const can = eligible(vod);
+ if (!can.ok) return { ...out, skipped: can.reason };
+ try { if (await objects().isHeldRow(vod)) return { ...out, skipped: 'held' }; } catch { /* the scan is read-only */ }
+ const scanOpts = { seconds: full ? 0 : decode.FIRST_SECONDS, timeoutMs: full ? FULL_SCAN_TIMEOUT_MS : undefined };
+ const before = await decode.scan(vod.file_path, scanOpts);
+ out.before = { ok: before.ok, code: before.code, timedOut: before.timedOut, errors: before.errors };
+ if (before.ok) {
+ // Already clean (a scan marked needs_repair on a transient error, or an earlier repair's row write failed):
+ // settle the row without touching the bytes.
+ if (apply && vod.health_status !== 'ok') {
+ const info = await tools.probeVodInfo(vod.file_path);
+ const stat = fs.statSync(vod.file_path);
+ const duration = info.duration > 0 ? info.duration : Number(vod.duration_seconds) || 0;
+ try {
+ const stored = await publishReady(vod, { duration, size: stat.size, format: info.format, issues: ['decode_check_ok'], announceReady });
+ return { ...out, ok: true, duration: stored, size: stat.size, issues: ['decode_check_ok'] };
+ } catch (err) { return { ...out, error: `stored_failed: ${err.message}` }; }
+ }
+ return { ...out, ok: true, skipped: 'clean' };
+ }
+ if (!apply) return { ...out, ok: true, skipped: 'needs_repair' };
+
+ const fixed = await decode.repair(vod.file_path, { durationSeconds: vod.duration_seconds });
+ if (!fixed.ok) return { ...out, error: fixed.error };
+
+ const info = await tools.probeVodInfo(vod.file_path);
+ const duration = info.duration > 0 ? info.duration : Number(vod.duration_seconds) || 0;
+ const stat = fs.statSync(vod.file_path);
+ const after = await decode.scan(vod.file_path, scanOpts);
+ out.after = { ok: after.ok, errors: after.errors };
+ if (!after.ok) return { ...out, error: `still_broken: ${after.errors[0] || 'decode'}` };
+
+ let stored;
+ try {
+ stored = await publishReady(vod, { duration, size: stat.size, format: info.format, issues: ['decode_repaired'], announceReady });
+ } catch (err) {
+ return { ...out, error: `stored_failed: ${err.message}` };
+ }
+
+ const version = `${stat.size}-${Math.round(stat.mtimeMs)}`;
+ let recutJob = null;
+ if (recut && vod.object_id) {
+ try { recutJob = await require('../objects/timeline-queue').queueRecut(vod.app_id, vod.object_id, { reason: 'decode_repair', version }); }
+ catch (err) { out.recutError = err.message; }
+ }
+ return { ...out, ok: true, duration: stored, size: stat.size, recut: recutJob };
+}
+
+module.exports = { repairVod, publishReady, eligible, FULL_SCAN_TIMEOUT_MS };
new file mode 100644
new file mode 100644
@@ -0,0 +1,63 @@
@@ -0,0 +1,63 @@
+'use strict';
+// Decode check + repair (server/vod/decode-check.js): a lost RTP access unit is a truncated H.264 slice; ffmpeg's
+// muxer copies it, Chrome fails the whole playback on it (VODs 6297/6298, 2026-11-10). `scan` must catch it even when
+// ffmpeg exits 0, and `repair` must turn the broken fixture into a stream that decodes cleanly. The fixture is made
+// here with ffmpeg — no production media.
+const assert = require('assert');
+const fs = require('fs');
+const os = require('os');
+const path = require('path');
+const { spawnSync } = require('child_process');
+
+const hasFfmpeg = spawnSync('ffmpeg', ['-version']).status === 0;
+const hasX264 = hasFfmpeg && /libx264/.test(String(spawnSync('ffmpeg', ['-hide_banner', '-encoders']).stdout));
+
+if (!hasX264) {
+ console.log('decode-check: skipped (ffmpeg with libx264 not found)');
+ process.exit(0);
+}
+
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-decode-'));
+ const clean = path.join(tmp, 'clean.mp4');
+ const make = spawnSync('ffmpeg', ['-v', 'error', '-y',
+ '-f', 'lavfi', '-i', 'testsrc=duration=6:size=320x180:rate=10',
+ '-f', 'lavfi', '-i', 'sine=frequency=440:duration=6',
+ '-c:v', 'libx264', '-preset', 'ultrafast', '-pix_fmt', 'yuv420p',
+ '-g', '20', '-keyint_min', '20', '-sc_threshold', '0',
+ '-c:a', 'aac', '-b:a', '64k', '-movflags', '+faststart', clean], { encoding: null });
+ assert.strictEqual(make.status, 0, String(make.stderr));
+
+ const decode = require('../server/vod/decode-check');
+
+ // ── 1. A clean recording passes ──
+ const good = await decode.scan(clean);
+ assert.strictEqual(good.ok, true, JSON.stringify(good));
+
+ // ── 2. Corrupt one P-frame the way a lost RTP fragment corrupts it: flip bytes in the middle of the slice. ──
+ const probe = spawnSync('ffprobe', ['-v', 'error', '-select_streams', 'v:0', '-show_packets', '-of', 'json', clean], { encoding: 'utf8' });
+ const packets = JSON.parse(probe.stdout).packets || [];
+ const victim = packets.find((p) => !String(p.flags || '').includes('K') && Number(p.size) > 300 && Number(p.pos) > 0);
+ assert.ok(victim, 'the fixture has a P-frame to corrupt');
+ const bad = path.join(tmp, 'bad.mp4');
+ const bytes = fs.readFileSync(clean);
+ const at = Number(victim.pos) + 4 + 120; // past the length prefix, into the slice data
+ for (let i = 0; i < 12; i++) bytes[at + i] ^= 0xff;
+ fs.writeFileSync(bad, bytes);
+
+ const broken = await decode.scan(bad);
+ assert.strictEqual(broken.ok, false, 'the corrupt access unit is detected');
+ assert.ok(broken.errors.some((l) => /error while decoding|corrupt decoded frame|Error processing packet/i.test(l)),
+ `a decoder error is reported: ${JSON.stringify(broken.errors)}`);
+
+ // ── 3. Repair replaces the corrupt unit with the decoder's concealed picture and verifies the output ──
+ const fixed = await decode.repair(bad);
+ assert.strictEqual(fixed.ok, true, JSON.stringify(fixed));
+ const after = await decode.scan(bad);
+ assert.strictEqual(after.ok, true, `the repaired file decodes cleanly: ${JSON.stringify(after)}`);
+ assert.ok(fixed.duration >= 5.5 && fixed.duration <= 6.6, `duration is kept (${fixed.duration}s)`);
+ assert.ok(fixed.size > 1024);
+
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('✅ decode-check: a corrupt H.264 access unit is detected even on ffmpeg exit 0, and repair re-encodes a clean file with the duration kept');
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -29,7 +29,7 @@ const express = require('express');
@@ -29,7 +29,7 @@ const express = require('express');
const model = require('../server/objects/model');
const model = require('../server/objects/model');
const timeline = require('../server/objects/timeline');
const timeline = require('../server/objects/timeline');
const signing = require('../server/objects/signing');
const signing = require('../server/objects/signing');
- const { queueCmaf, queuePack } = require('../server/objects/timeline-queue');
+ const { queueCmaf, queuePack, queueRecut } = require('../server/objects/timeline-queue');
const tools = require('../server/vod/media-tools');
const tools = require('../server/vod/media-tools');
const thumbService = require('../server/thumbnails/thumbnail-service');
const thumbService = require('../server/thumbnails/thumbnail-service');
const vodStorage = require('../server/vod/vod-storage');
const vodStorage = require('../server/vod/vod-storage');
@@ -111,6 +111,27 @@ const express = require('express');
@@ -111,6 +111,27 @@ const express = require('express');
assert.strictEqual((await jobs('object.cmaf', cut)).length, 0, 'a timeline already there: nothing queued');
assert.strictEqual((await jobs('object.cmaf', cut)).length, 0, 'a timeline already there: nothing queued');
console.log('✅ nothing queued with the flag off, for a non-media or missing object, or once the timeline exists');
console.log('✅ nothing queued with the flag off, for a non-media or missing object, or once the timeline exists');
+ // ── queueRecut: a repaired file gets a new cut even though a timeline exists, keyed by the file version ──
+ const recutId = await queueRecut('live', cut, { reason: 'decode_repair', version: '111-222' });
+ assert.ok(recutId && /^mjob_/.test(recutId), 'a repaired file queues a re-cut over the existing timeline');
+ let recutJobs = await jobs('object.cmaf', cut);
+ assert.strictEqual(recutJobs.length, 1);
+ assert.deepStrictEqual([recutJobs[0].status, recutJobs[0].created_by, recutJobs[0].idempotency_key],
+ ['queued', 'system:decode_repair', `object.cmaf:${cut}:decode_repair:111-222`]);
+ assert.strictEqual(await queueRecut('live', cut, { reason: 'decode_repair', version: '111-222' }), recutId,
+ 'the same repaired file joins its queued re-cut');
+ // dedupeActive is per object and type: while the cut is queued, a newer version joins it rather than
+ // running two cuts at once. The next repair after it settles gets its new key and cuts again.
+ assert.strictEqual(await queueRecut('live', cut, { reason: 'decode_repair', version: '333-444' }), recutId,
+ 'a newer version while the cut is active joins it');
+ recutJobs = await jobs('object.cmaf', cut);
+ assert.strictEqual(recutJobs.length, 1);
+ config.hls.enabled = false;
+ assert.strictEqual(await queueRecut('live', cut, { reason: 'decode_repair', version: '555-666' }), null);
+ config.hls.enabled = true;
+ assert.strictEqual((await jobs('object.cmaf', cut)).length, 1, 'flag off: nothing queued');
+ console.log('✅ queueRecut: a repaired file re-cuts over the existing timeline, joining by file version');
+
// ── queuePack: a cut queues one object.pack, a re-run joins it; flag off or non-media queue nothing ──
// ── queuePack: a cut queues one object.pack, a re-run joins it; flag off or non-media queue nothing ──
const packOff = await media();
const packOff = await media();
config.hls.enabled = false;
config.hls.enabled = false;
new file mode 100644
new file mode 100644
@@ -0,0 +1,100 @@
@@ -0,0 +1,100 @@
+'use strict';
+// A recording that does not decode must never reach a player: finalize marks it needs_repair and hides it, and
+// Media's own repair re-encodes the broken file, verifies it, and lifts the quarantine (server/vod/decode-check.js,
+// server/vod/finalize.js, server/vod/vod-repair.js). Fixtures are made here with ffmpeg; no production media.
+const assert = require('assert');
+const fs = require('fs');
+const os = require('os');
+const path = require('path');
+const { spawnSync } = require('child_process');
+
+const hasFfmpeg = spawnSync('ffmpeg', ['-version']).status === 0;
+const hasX264 = hasFfmpeg && /libx264/.test(String(spawnSync('ffmpeg', ['-hide_banner', '-encoders']).stdout));
+if (!hasX264) { console.log('vod-decode-repair: skipped (ffmpeg with libx264 not found)'); process.exit(0); }
+
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-vod-repair-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ process.env.VOD_PATH = dir('vods');
+ process.env.CLIPS_PATH = dir('clips');
+ process.env.THUMBNAILS_PATH = dir('thumbnails');
+ process.env.FILES_PATH = dir('files');
+ process.env.PASTES_PATH = dir('pastes');
+ process.env.OBJECTS_PATH = dir('objects');
+ process.env.MEDIA_JOBS_ENABLED = '0';
+ process.env.MEDIA_HLS_ENABLED = '0';
+
+ const db = require('../server/db/database');
+ const finalize = require('../server/vod/finalize');
+ const repair = require('../server/vod/vod-repair');
+ const decode = require('../server/vod/decode-check');
+ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });
+
+ const makeVideo = (file) => {
+ const r = spawnSync('ffmpeg', ['-v', 'error', '-y',
+ '-f', 'lavfi', '-i', 'testsrc=duration=6:size=320x180:rate=10',
+ '-f', 'lavfi', '-i', 'sine=frequency=440:duration=6',
+ '-c:v', 'libx264', '-preset', 'ultrafast', '-pix_fmt', 'yuv420p',
+ '-g', '20', '-keyint_min', '20', '-sc_threshold', '0',
+ '-c:a', 'aac', '-b:a', '64k', '-movflags', '+faststart', file], { encoding: null });
+ assert.strictEqual(r.status, 0, String(r.stderr));
+ };
+ const corruptOneFrame = (file) => {
+ const probe = spawnSync('ffprobe', ['-v', 'error', '-select_streams', 'v:0', '-show_packets', '-of', 'json', file], { encoding: 'utf8' });
+ const victim = (JSON.parse(probe.stdout).packets || []).find((p) => !String(p.flags || '').includes('K') && Number(p.size) > 300 && Number(p.pos) > 0);
+ assert.ok(victim, 'the fixture has a P-frame to corrupt');
+ const bytes = fs.readFileSync(file);
+ const at = Number(victim.pos) + 4 + 120;
+ for (let i = 0; i < 12; i++) bytes[at + i] ^= 0xff;
+ fs.writeFileSync(file, bytes);
+ };
+ async function makeVod(file) {
+ const r = await db.run(`INSERT INTO vods (app_id, user_id, title, file_path, is_recording, duration_seconds, is_public, visibility)
+ VALUES ('live', 7, 'robot', ?, 1, 0, 1, 'public') RETURNING id`, [file]);
+ return Number(r.lastInsertRowid);
+ }
+ const row = async (id) => await db.get('SELECT * FROM vods WHERE id = ?', [id]);
+
+ // ── 1. Finalize a corrupt recording: it is measured, hidden and marked needs_repair, never published ──
+ const bad = path.join(process.env.VOD_PATH, 'robot.mp4');
+ makeVideo(bad);
+ corruptOneFrame(bad);
+ const badId = await makeVod(bad);
+ await finalize.finalizeVod(badId);
+ const broken = await row(badId);
+ assert.strictEqual(broken.health_status, 'needs_repair', JSON.stringify(broken.health_issues_json));
+ assert.strictEqual(broken.is_recording, 0);
+ assert.strictEqual(broken.is_public, 0, 'a broken recording is hidden until it is repaired');
+ assert.ok(broken.quarantined_at, 'and quarantined for repair');
+ assert.ok(JSON.parse(broken.health_issues_json).includes('decode_failed'));
+ assert.ok(broken.duration_seconds >= 5 && broken.duration_seconds <= 7, `its measured length is kept (${broken.duration_seconds}s)`);
+ assert.ok(fs.existsSync(bad), 'the bytes stay for the repair');
+ console.log('✅ finalize: a corrupt recording is needs_repair, hidden, and its bytes are kept');
+
+ // ── 2. Media's repair: re-encode, verify, lift the quarantine, restore the owner's visibility ──
+ const fixed = await repair.repairVod(await row(badId), { apply: true, recut: false, announceReady: false });
+ assert.strictEqual(fixed.ok, true, JSON.stringify(fixed));
+ assert.ok(!fixed.skipped, 'it was not already clean');
+ const good = await row(badId);
+ assert.strictEqual(good.health_status, 'ok');
+ assert.strictEqual(good.quarantined_at, null, 'the quarantine is lifted');
+ assert.strictEqual(good.is_public, 1, 'back to the visibility its owner chose');
+ assert.ok(good.duration_seconds >= 5 && good.duration_seconds <= 7, `duration kept (${good.duration_seconds}s)`);
+ assert.ok(fs.statSync(bad).size > 1024);
+ assert.strictEqual((await decode.scan(bad)).ok, true, 'the repaired file decodes cleanly');
+ console.log('✅ repair: the file is re-encoded in place, verified, un-quarantined and public again');
+
+ // ── 3. A clean recording is not touched and the check does not cry wolf on variable timestamps ──
+ const clean = path.join(process.env.VOD_PATH, 'clean.mp4');
+ makeVideo(clean);
+ const cleanId = await makeVod(clean);
+ const cleanSize = fs.statSync(clean).size;
+ await finalize.finalizeVod(cleanId);
+ const ok = await row(cleanId);
+ assert.strictEqual(ok.health_status, 'ok');
+ assert.strictEqual(ok.is_public, 1);
+ assert.strictEqual(fs.statSync(clean).size, cleanSize, 'a clean file is not re-encoded');
+ console.log('✅ finalize: a clean recording is published without touching its bytes');
+
+ fs.rmSync(tmp, { recursive: true, force: true });
+})().catch((err) => { console.error(err); process.exit(1); });