openvibers/openvibe.media Commits

Operator retirement for a no-good-copy object (#323)

OP OpenVibers committed 7 days ago · ff3e8c2435 · 6 files, +300 −4

scripts/no-good-copy-report.js --retire <med_id> [--apply] retires a reported object through the ordinary soft delete (retention, media.object.deleted, restorable), after refusing a public object, one with a good copy, one of 1 MiB or more, or one under a hold. A projected VOD/clip has its v1 row made private so the inherited route stops serving it. The dry run prints the plan and changes nothing.

Changes

ddiff --git a/CHANGELOG.md b/CHANGELOG.md
iindex 785efdd..9a9cf6e 100644
---- a/CHANGELOG.md
++++ b/CHANGELOG.md
@@@ -2,6 +2,12 @@
## Unreleased
++- Operator retirement for an object with no good copy (`scripts/no-good-copy-report.js --retire <med_id> [--apply]`):
++ the dry run changes nothing; only an object in the report, not public and under 1 MiB is retired (public, good-copy,
++ 1 MiB+ and held objects are refused with the reason). The delete is the ordinary soft delete (bytes kept,
++ retention, `media.object.deleted`, restorable), and a projected VOD/clip has its v1 row made private so the inherited
++ route stops serving it. Test: `test/retire-no-good-copy.test.js`.
++
- A legacy Live paste screenshot can be retired for OpenVibe.Community (2026-11-06). The internal
`POST /internal/community/legacy-paste-screenshots/:id/retire` route is loopback and service-token only
(`media.object.delete`, principal `svc:community`) and soft-deletes exactly a `legacy:live:paste:<slug>`
ddiff --git a/docs/object-model.md b/docs/object-model.md
iindex 15c541f..55e1e15 100644
---- a/docs/object-model.md
++++ b/docs/object-model.md
@@@ -610,6 +610,14 @@ that only moved to B2/R2 keeps it. Test: `test/content-hash.test.js`.
For each object it prints the owner (app, `owner_user_id`, `owner_subject`), `legacy_ref`, the `vods`/`clips`/`files` rows that point at it, its relationships, variants and holds, every location with its state and `verified_at`, and the last verification. The report opens the database read-only. It exits 1 when there are any.
++ Retire one of those objects instead of only listing it:
++
++ ```
++ node scripts/no-good-copy-report.js --retire <med_id> [--retire <med_id> ...] [--apply]
++ ```
++
++ The dry run (no `--apply`) prints what it would do and writes nothing. Only an object that is in the report (no present or pending copy), is not public and is under 1 MiB is retired; anything else — a public object, one with a good copy, one of 1 MiB or more, one under a retention hold — is refused with the reason and nothing is changed. The delete is the ordinary soft delete (`model.softDelete`): the bytes stay, `metadata.retention_until` is set, the same `media.object.deleted` event is queued as for any other delete, and `POST /:id/restore` brings the object back inside the retention period. A projected VOD or clip also has its v1 row made private (`visibility private`, `is_public 0`) so `/v/:id` and `/c/:id` stop serving it; the row is kept, not removed, so the retirement stays reversible. A file object is refused because `/f/:key` serves every file without auth. Test: `test/retire-no-good-copy.test.js`.
++
## Public object-size invariant
This is roadmap W4 deliverable 5, stated as **policy** rather than a copied constant:
ddiff --git a/scripts/no-good-copy-report.js b/scripts/no-good-copy-report.js
iindex 6c90473..3be4f8c 100644
---- a/scripts/no-good-copy-report.js
++++ b/scripts/no-good-copy-report.js
@@@ -8,19 +8,66 @@
*
* node scripts/no-good-copy-report.js [--app live] [--json] [--out report.json]
*
-- * Exit code 0 when there are none, 1 when some were found, 2 on error.
++ * Retire one or more of those objects instead (scripts/no-good-copy-report.js --retire):
++ *
++ * node scripts/no-good-copy-report.js --retire <med_id> [--retire <med_id> ...] [--apply]
++ *
++ * The dry run (no --apply) prints what it would do and changes nothing. Only an object that is in the
++ * report, not public and under 1 MiB is retired, through the ordinary soft delete (bytes kept for the
++ * retention period, media.object.deleted queued, restorable) with its v1 VOD/clip row hidden. Anything
++ * else is refused with the reason and nothing is changed.
++ *
++ * Exit code: 0 when the report found none (or every named object can be / was retired), 1 when the
++ * report found some (or one named object was refused), 2 on error.
*/
const fs = require('fs');
--const path = require('path');
(async () => {
await require('../server/db/database').initDb(); // PostgreSQL (DATABASE_URL), as the service
const args = process.argv.slice(2);
const has = (name) => args.includes(`--${name}`);
const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 ? args[i + 1] : dflt; };
++ const retireIds = [];
++ for (let i = 0; i < args.length; i++) {
++ if (args[i] !== '--retire') continue;
++ const value = args[i + 1];
++ if (!value || value.startsWith('--')) { retireIds.push(''); continue; }
++ retireIds.push(...String(value).split(','));
++ }
try {
-- // Read-only: the report runs in a READ ONLY transaction (PostgreSQL refuses any write inside it).
++ // ── Retire named objects (the one write path of this script) ──
++ if (retireIds.length) {
++ const { assess, describe, retire } = require('../server/objects/retire');
++ const ids = [...new Set(retireIds.map((s) => s.trim()).filter(Boolean))];
++ if (!ids.length) {
++ console.error('--retire needs one or more med_… object ids');
++ process.exit(2);
++ }
++ const plans = [];
++ for (const id of ids) plans.push(await assess(id));
++ const refused = plans.filter((p) => !p.ok);
++ if (refused.length) {
++ for (const p of refused) console.log(`${p.id} — refused: ${p.reason}`);
++ console.log('Nothing was changed.');
++ process.exit(1);
++ }
++ if (!has('apply')) {
++ for (const p of plans) console.log(`${describe(p)}\n dry run: nothing was changed; re-run with --apply to retire it\n`);
++ process.exit(0);
++ }
++ const done = await retire(plans);
++ for (const d of done) {
++ console.log(`${d.id} retired: ${d.lifecycle_status}; retention until ${d.retention_until || `the end of the ${require('../server/config').objects.retentionDays}-day period`}; restorable`);
++ }
++ process.exit(0);
++ }
++ if (has('apply')) {
++ console.error('--apply only goes with --retire <med_id>');
++ process.exit(2);
++ }
++
++ // ── Report (read-only: the report runs in a READ ONLY transaction, PostgreSQL refuses any write) ──
const { buildReport, formatReport } = require('../server/objects/copy-report');
const d = require('../server/db/database').getDb();
const q = {
ddiff --git a/server/objects/copy-report.js b/server/objects/copy-report.js
iindex 9035ec1..40f2ebb 100644
---- a/server/objects/copy-report.js
++++ b/server/objects/copy-report.js
@@@ -105,4 +105,4 @@ function formatReport(report) {
return out.join('\n');
}
--module.exports = { NO_GOOD_COPY_WHERE, countNoGoodCopy, listNoGoodCopy, buildReport, formatReport };
++module.exports = { NO_GOOD_COPY_WHERE, countNoGoodCopy, listNoGoodCopy, references, buildReport, formatReport };
ddiff --git a/server/objects/retire.js b/server/objects/retire.js
nnew file mode 100644
iindex 0000000..6fd2005
---- /dev/null
++++ b/server/objects/retire.js
@@@ -0,0 +1,121 @@
++'use strict';
++/**
++ * Operator retirement of a no-good-copy object (scripts/no-good-copy-report.js --retire).
++ *
++ * A ready object with no good copy cannot be repaired when every copy it has is missing or corrupt. The report can
++ * only list it; this is the one operator path that retires it, and only the safe shape (docs/object-model.md,
++ * "scheduled verification"):
++ *
++ * - it is in the no-good-copy report (listNoGoodCopy: ready, not virtual, no present/pending copy);
++ * - it is not public (a public object, or a public v1 row, is refused);
++ * - it is under 1 MiB: a stub or a broken small upload, never a real recording;
++ * - no retention hold applies.
++ *
++ * The delete is the ordinary soft delete (model.softDelete), the same one Media's own delete uses: lifecycle_status
++ * becomes deleted, the bytes stay for MEDIA_DELETE_RETENTION_DAYS (metadata.retention_until), the media_objects
++ * trigger queues the same media.object.deleted event as any other delete (so Live and Search forget the item), and
++ * POST /:id/restore brings it back inside the retention period.
++ *
++ * A projected VOD or clip is also hidden from its inherited route the way the owning app's delete hides it: its v1
++ * row is made private (visibility = private, is_public = 0) through db.setVodVisibility / setClipVisibility, so the
++ * row and the object agree. The row is kept, not removed, which is what keeps the retirement reversible; a VOD that
++ * is public is refused rather than hidden. A file row keeps being served by /f/:key whatever its object says, so a
++ * file object is refused too.
++ */
++const db = require('../db/database');
++const model = require('./model');
++const copyReport = require('./copy-report');
++
++const MAX_BYTES = 1024 * 1024; // strictly under 1 MiB
++const BY = 'operator:no-good-copy-report';
++const MED_ID = /^med_[0-9A-HJKMNP-TV-Z]{26}$/;
++
++function query() {
++ const d = db.getDb();
++ return {
++ all: async (sql, params = []) => await d.prepare(sql).all(...params),
++ get: async (sql, params = []) => await d.prepare(sql).get(...params),
++ };
++}
++
++/** The v1 route's own visibility rule (vods/clips row → public, unlisted or private). */
++function rowVisibility(row) {
++ return row.visibility || (row.is_public ? 'public' : 'private');
++}
++
++function humanBytes(n) {
++ n = Number(n) || 0;
++ if (n < 1024) return `${n} B`;
++ if (n < 1048576) return `${(n / 1024).toFixed(1)} KB`;
++ return `${(n / 1048576).toFixed(1)} MB`;
++}
++
++/**
++ * Decide whether an object may be retired. Reads only. Returns
++ * { ok: true, id, obj, size, rows, locations } or { ok: false, id, obj?, reason }.
++ */
++async function assess(id, { q = null } = {}) {
++ if (!MED_ID.test(String(id || ''))) return { ok: false, id, reason: 'not a med_… object id' };
++ const obj = await model.getObject(id);
++ if (!obj) return { ok: false, id, reason: 'no such object' };
++ if (obj.lifecycle_status === 'deleted') return { ok: false, id, obj, reason: 'it is already deleted' };
++ const reads = q || query();
++ const inReport = (await copyReport.listNoGoodCopy(reads, { appId: obj.app_id })).some((o) => o.id === obj.id);
++ if (!inReport) return { ok: false, id, obj, reason: 'not in the no-good-copy report (it has a good copy, is virtual, or is not ready)' };
++ if (obj.visibility === 'public') return { ok: false, id, obj, reason: 'it is public' };
++ const size = Number(obj.size_bytes) || 0;
++ if (size >= MAX_BYTES) return { ok: false, id, obj, reason: `${humanBytes(size)} is not under 1 MiB` };
++ if (await model.isHeld(obj.id)) return { ok: false, id, obj, reason: 'it is under a retention hold' };
++
++ const refs = await copyReport.references(reads, obj);
++ if (refs.files.length) return { ok: false, id, obj, reason: `its files/${refs.files[0].key} row is served publicly by /f/:key` };
++ const publicVod = refs.vods.find((r) => rowVisibility(r) === 'public');
++ if (publicVod) return { ok: false, id, obj, reason: `its vods#${publicVod.id} row is public` };
++ const publicClip = refs.clips.find((r) => rowVisibility(r) === 'public');
++ if (publicClip) return { ok: false, id, obj, reason: `its clips#${publicClip.id} row is public` };
++
++ return {
++ ok: true, id, obj, size,
++ rows: [
++ ...refs.vods.map((r) => ({ table: 'vods', id: r.id })),
++ ...refs.clips.map((r) => ({ table: 'clips', id: r.id })),
++ ],
++ locations: await model.listLocations(obj.id),
++ };
++}
++
++/** What a dry run (or the apply itself) does, in operator-readable lines. */
++function describe(plan) {
++ const o = plan.obj;
++ const out = [`${o.id} ${o.kind} ${humanBytes(plan.size)} ${o.visibility} ${o.legacy_ref || '(native object)'}`];
++ out.push(` action: soft delete (bytes kept, ${require('../config').objects.retentionDays}-day retention; restorable; this script deletes no bytes)`);
++ out.push(' event: media.object.deleted (Live and Search forget the item)');
++ if (plan.rows.length) out.push(` hide: ${plan.rows.map((r) => `${r.table}#${r.id} (visibility private, is_public 0)`).join(', ')}`);
++ if (!plan.locations.length) out.push(' report: no good copy (no locations recorded)');
++ else out.push(` report: no good copy (${plan.locations.map((l) => `${l.provider} ${l.state}`).join(', ')})`);
++ return out.join('\n');
++}
++
++/**
++ * Retire the assessed plans in one transaction: hide their v1 rows, then soft-delete each object through
++ * model.softDelete (with the event and retention metadata). Returns the deleted objects' key facts.
++ */
++async function retire(plans) {
++ const d = db.getDb();
++ await d.tx(async () => {
++ for (const plan of plans) {
++ for (const row of plan.rows) {
++ if (row.table === 'vods') await db.setVodVisibility(row.id, 'private');
++ else if (row.table === 'clips') await db.setClipVisibility(row.id, 'private');
++ }
++ await model.softDelete(plan.obj, { by: BY });
++ }
++ });
++ return await Promise.all(plans.map(async (plan) => {
++ const obj = await model.getObject(plan.obj.id);
++ const md = model.parseJson(obj.metadata, {});
++ return { id: obj.id, lifecycle_status: obj.lifecycle_status, deleted_at: obj.deleted_at || null, retention_until: md.retention_until || null };
++ }));
++}
++
++module.exports = { MAX_BYTES, BY, MED_ID, assess, describe, retire, rowVisibility, humanBytes };
ddiff --git a/test/retire-no-good-copy.test.js b/test/retire-no-good-copy.test.js
nnew file mode 100644
iindex 0000000..261d295
---- /dev/null
++++ b/test/retire-no-good-copy.test.js
@@@ -0,0 +1,114 @@
++'use strict';
++// Operator retirement of a no-good-copy object (scripts/no-good-copy-report.js --retire): the dry run prints and
++// changes nothing; a stub is soft-deleted (retention metadata, restorable, media.object.deleted staged) with its v1
++// VOD row hidden, and the readiness count (the same count /api/ready's object_copies check reads) goes to 0; public
++// objects, objects with a good copy, objects of 1 MiB or more and held objects are refused with the reason.
++const assert = require('assert');
++const fs = require('fs');
++const os = require('os');
++const path = require('path');
++
++(async () => {
++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-retire-'));
++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
++ const env = {
++ VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
++ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), ASSETS_PATH: dir('assets'),
++ MEDIA_PUBLIC_URL: 'https://media.test', OV_NETWORK_URL: 'http://127.0.0.1:9',
++ };
++ for (const k of ['MEDIA_B2_ENDPOINT', 'MEDIA_B2_BUCKET', 'MEDIA_R2_ENDPOINT', 'MEDIA_R2_BUCKET', 'EVENTS_URL', 'MEDIA_VERIFY_ENABLED']) process.env[k] = '';
++ Object.assign(process.env, env);
++
++ const db = require('../server/db/database');
++ const model = require('../server/objects/model');
++ const copyReport = require('../server/objects/copy-report');
++ const verifyJob = require('../server/objects/verify-job');
++ const vodStorage = require('../server/vod/vod-storage');
++ const { runScript } = require('./helpers/run-script');
++ await vodStorage.tierConfig.init(vodStorage.DEFAULTS);
++ await require('../server/objects/tier-policy').init();
++ const d = db.getDb();
++ const MB = 1024 * 1024;
++
++ let nextVod = 6027;
++ const insertVod = d.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, duration_seconds, visibility, is_public, storage_provider, clips_only, is_recording, created_at)
++ OVERRIDING SYSTEM VALUE VALUES (?, 'live', 5, ?, ?, ?, 1, ?, ?, 'local', 0, 0, '2026-11-01 00:00:00') RETURNING id`);
++ /** A VOD row projected onto its object; the file exists only when `bytes` is written. */
++ async function vod({ title = 'V', size = 8855, visibility = null, isPublic = 0, bytes = null }) {
++ const id = nextVod++;
++ const p = path.join(env.VOD_PATH, `vod-${id}.webm`);
++ if (bytes != null) { fs.mkdirSync(path.dirname(p), { recursive: true }); fs.writeFileSync(p, Buffer.alloc(bytes, 7)); }
++ await insertVod.run(id, title, p, size, visibility, isPublic);
++ await model.syncVod(id);
++ const objectId = (await d.prepare('SELECT object_id FROM vods WHERE id = ?').get(id)).object_id;
++ return { id, objectId, obj: await model.getObject(objectId) };
++ }
++
++ const script = path.join(__dirname, '..', 'scripts', 'no-good-copy-report.js');
++ const run = async (a) => await runScript(script, a, { env });
++ const state = async (v) => { const o = await model.getObject(v.objectId); return [o.lifecycle_status, o.visibility, o.deleted_at || null]; };
++
++ // ── The stub (the report's shape): a 1-second, 8.6 KB recording whose only copy is gone, never public ──
++ const stub = await vod({ title: 'recording stub' });
++ assert.strictEqual(stub.obj.visibility, 'private');
++ assert.strictEqual(await copyReport.countNoGoodCopy(db), 1);
++ assert.strictEqual((await verifyJob.status()).no_good_copy, 1);
++
++ // ── Dry run ──
++ const dry = await run(['--retire', stub.objectId]);
++ assert.strictEqual(dry.code, 0, dry.err);
++ assert.match(dry.out, /soft delete/);
++ assert.match(dry.out, /vods#6027 \(visibility private, is_public 0\)/);
++ assert.match(dry.out, /dry run: nothing was changed/);
++ assert.deepStrictEqual(await state(stub), ['ready', 'private', null]);
++ console.log('✅ the dry run prints what it would do and writes nothing');
++
++ // ── Apply: the stub is soft-deleted, its v1 row hidden, the readiness count is 0 ──
++ const applied = await run(['--retire', stub.objectId, '--apply']);
++ assert.strictEqual(applied.code, 0, applied.err);
++ const gone = await model.getObject(stub.objectId);
++ assert.strictEqual(gone.lifecycle_status, 'deleted');
++ assert.ok(gone.deleted_at, 'deleted_at is set');
++ const md = model.parseJson(gone.metadata, {});
++ assert.ok(Date.parse(md.retention_until) > Date.now(), `retention_until is in the future (${md.retention_until})`);
++ const row = await d.prepare('SELECT visibility, is_public FROM vods WHERE id = 6027').get();
++ assert.deepStrictEqual([row.visibility, Number(row.is_public)], ['private', 0], 'the v1 VOD row is hidden');
++ const change = await d.prepare("SELECT change FROM media_object_changes WHERE object_id = ? ORDER BY id DESC LIMIT 1").get(stub.objectId);
++ assert.strictEqual(change && change.change, 'deleted', 'media.object.deleted is staged for the object');
++ assert.strictEqual(await copyReport.countNoGoodCopy(db), 0, 'the readiness object_copies count is 0');
++ assert.strictEqual((await verifyJob.status()).no_good_copy, 0);
++ console.log('✅ retires the stub: soft delete, retention, hidden v1 row, media.object.deleted, readiness count 0');
++
++ // ── Refusals: nothing is changed, the reason says why ──
++ const good = await vod({ title: 'good', size: 100, bytes: 100 });
++ const publicVod = await vod({ title: 'public', visibility: 'public', isPublic: 1 });
++ const big = await vod({ title: 'big', size: MB });
++ const held = await vod({ title: 'held' });
++ await model.placeHold({ object_id: held.objectId, kind: 'admin', reason: 'test hold', created_by: 'test' });
++ assert.strictEqual(await copyReport.countNoGoodCopy(db), 3, 'public, big and held; not the one with a good copy');
++ for (const [v, reason] of [
++ [publicVod, /it is public/],
++ [good, /not in the no-good-copy report/],
++ [big, /not under 1 MiB/],
++ [held, /retention hold/],
++ ]) {
++ const before = await state(v);
++ const r = await run(['--retire', v.objectId]);
++ assert.strictEqual(r.code, 1, `${v.objectId} refused ${r.err || ''}`);
++ assert.match(r.out, reason);
++ assert.match(r.out, /Nothing was changed/);
++ assert.deepStrictEqual(await state(v), before, 'a refused retirement writes nothing');
++ }
++ console.log('✅ refuses a public object, one with a good copy, one of 1 MiB and one under a hold, with the reason');
++
++ // ── Restorable ──
++ const back = await model.restore(await model.getObject(stub.objectId));
++ assert.strictEqual(back.lifecycle_status, 'ready', 'a restore inside retention brings it back');
++ assert.strictEqual(await copyReport.countNoGoodCopy(db), 4, 'and it is a no-good-copy object again');
++ console.log('✅ restorable inside the retention period');
++
++ await db.close();
++ fs.rmSync(tmp, { recursive: true, force: true });
++ console.log('retire-no-good-copy: all checks passed');
++ process.exit(0);
++})().catch((err) => { console.error(err); process.exit(1); });