openvibers/openvibe.media Commits
Retire C-75: no boot backfill, no finalize follow-up, no drift tooling (#74)
Changes
@@@ -103,7 +103,7 @@ server/me/ the object explorer: /me and /api/v2/me (a signed-in person's own objects, uploads,derivatives and usage) and the operator views /me/ops — see docs/object-model.md#object-explorerobjects/ canonical object model: model (projections, holds), routes (/api/v2 + /o),-- backfill, reconcile, invariant, signing (presigned URLs), multipart,++ reconcile, invariant, signing (presigned URLs), multipart,content-type, verify-job (scheduled copy verification), copy-report,readiness (metadata / bytes_verified / playable), popularity (unique viewersper day, no identifiers), tier-policy + tiering (native objects in the R2 cache) —@@@ -127,7 +127,7 @@ server/client-ip.js trust proxy = loopback; req.ip is the only client address(openvibe-shared v3.0.0, openvibe-contracts v0.127.0, openvibe-sdk v0.38.0: pinned release tarballs, installed by npm)scripts/smoke-test.sh end-to-end smoke test (boots a temp instance)--scripts/reconcile-objects.js / object-invariant.js / object-drift-report.js / no-good-copy-report.js object-model operator tools++scripts/reconcile-objects.js / object-invariant.js / no-good-copy-report.js object-model operator toolsscripts/media-jobs.js list jobs, run the size-invariant scan (dry run by default), approve/cancel proposalsscripts/r2-eviction-drill.js evict one VOD's R2 copy, prove B2 serves it, re-warm R2; JSON artifact (dry run by default)scripts/vod-duration-reconcile.js stored VOD durations vs the real files (dry run by default; --apply --backup; --rollback)@@@ -148,7 +148,7 @@ other OpenVibe services call the API.subject, in every tenant: uploads in progress, copies, derivatives, visibility, lifecycle and their ownusage per tenant and namespace (`GET /api/v2/me/objects`, `/objects/:id`, `/usage`). Read-only: changesstay with the apps. Network staff with `staff.site.view` get the operator views at `/me/ops` (failed--jobs, missing media, backfill, tiering, outbound webhooks — who is configured and what this process++jobs, missing media, projection, tiering, outbound webhooks — who is configured and what this processhas sent — and usage recompute with `staff.site.configure`). Details in[docs/object-model.md](docs/object-model.md#object-explorer).@@@ -356,7 +356,7 @@ directories are shared across apps) and responses carry a `note` saying so.| POST | `/admin/storage/holds` | `{ object_id \| vod_id \| clip_id, reason, kind?, note?, placed_by? }` → 201: the object is kept (no delete, no tier move) and so are the clips cut from a held VOD. Not for calls acting for a user (403) || POST | `/admin/storage/holds/:holdId/release` | `{ released_by }` (also `DELETE /admin/storage/holds/:holdId`); 409 when already released. Holds stay on record || GET | `/admin/storage/buckets` | sanitized bucket status per provider: `{ configured, endpoint, bucket, region, healthy, reachable }` via a live HeadBucket probe — **credentials are never returned** |--| GET | `/admin/storage/ops` | this app's operator report (as `/me/ops` shows it to Network staff): failed jobs, missing media, backfill, tiering, the namespaces' usage snapshot. Not for calls acting for a user (403) |++| GET | `/admin/storage/ops` | this app's operator report (as `/me/ops` shows it to Network staff): failed jobs, missing media, projection, tiering, the namespaces' usage snapshot. Not for calls acting for a user (403) || POST | `/admin/storage/ops/recompute` | refresh this app's namespaces' usage snapshot from the rows |### Webhooks (outbound)@@@ -427,7 +427,7 @@ mode (`server/drill.js`) Media:4100 (before the database is opened), and opens the copy without migrating it;- starts only its HTTP server: no app seeding, JWKS refresh, tiering sweep, health job, junk sweep,clip re-cuts, copy verification, jobs worker, disk guardian, thumbnail cleanup, object purge,-- backfill, orphan-recording finalize or Events relay; webhooks are never sent;++ orphan-recording finalize or Events relay; webhooks are never sent;- writes no file and creates no directory (the storage checks leave `/api/ready`, as do the remotetiers; it reports `"mode": "drill"`);- never opens a file path from the database: every byte route (`/v` and `/c` bytes, `/t`, `/a`, `/f`,@@@ -578,16 +578,7 @@ Every stored blob is a **media object** (`med_<ULID>`, `media_objects`) with oneexisting route and response is unchanged, and the old write paths keep themodel current. Full reference: **[docs/object-model.md](docs/object-model.md)**.--- **Backfill** — the row→object projection (`server/objects/backfill.js`), not a-- CLI script: one object per vod, clip, file, screenshot, avatar and thumbnail;-- idempotent; never moves bytes or calls B2/R2 (remote copies stay `pending`). The-- service runs the only-missing form 15 s after boot, for rows with no `object_id`-- yet, and stores its report in `media_settings.objects.backfill.last_report`-- (also surfaced by `/me/ops`). To see whether any row still lacks an object — or-- disagrees with the object on record — run the read-only drift report:-- `node scripts/object-drift-report.js [--app live] [--limit 20] [--json]-- [--out report.json]`. Zero drift across a release is the signal to drop the boot-- backfill ([details](docs/object-model.md#backfill)).++- **Object projection** — C-75 was retired on 2026-10-10 after production drift was 0 of 3036 rows. Every write makes its media object in the same transaction (`server/objects/model.js` `withObject`). Operators retain `scripts/reconcile-objects.js` and `scripts/object-invariant.js` as checks ([details](docs/object-model.md#keeping-the-model-current-object-first-writes)).- **API** — `/api/v2/:app/objects`: init → `PUT /:id/content` (sha256, size,quota) → `/:id/complete`; `GET /:id`, cursor `GET /`, soft `DELETE /:id`(bytes kept `MEDIA_DELETE_RETENTION_DAYS`), `/:id/restore`, `/:id/download`@@@ -9,7 +9,7 @@ its v1 pastes API was retired in T10 step 2 (`/api/v1/:app/pastes` is unmounted;`/p/:slug` routes remain). Other existing URLs and v1 responses areunchanged. New code talks to objects through the [v2 API](#object-api-v2).--Code: `server/objects/` (`model.js`, `routes.js`, `backfill.js`, `reconcile.js`, `invariant.js`,++Code: `server/objects/` (`model.js`, `routes.js`, `reconcile.js`, `invariant.js`,`signing.js`, `verify-job.js`, `copy-report.js`, `multipart.js`, `content-type.js`, `readiness.js`, `namespaces.js`,`namespace-routes.js`, `popularity.js`, `tier-policy.js`, `tiering.js`) and the job system in`server/jobs/` (`queue.js`, `worker.js`, `routes.js`, `types.js`, `thumbnail.js`, `invariant-scan.js`,@@@ -44,7 +44,7 @@ state resets only when the key changes.### Ids and legacy refs--Ids come from `openvibe-contracts` `ids.newId('media')`. A backfilled object takes its ULID time++Ids come from `openvibe-contracts` `ids.newId('media')`. A projected legacy object takes its ULID timefrom the original row's `created_at`, so ids sort by creation time. Projected objects also carrythe transitional `media.media-ref@1` legacy form in `legacy_ref`, which is unique:@@@ -66,8 +66,7 @@ Every v2 route that takes `:id` accepts either form. Native v2 objects have `leg### How a row maps to its object`model.sync(kind, id)` derives the object and its locations from the row as it is at that moment--(the per-kind `<kind>Projection(row)` functions). The backfill, every write and the drift report use--the same functions.++(the per-kind `<kind>Projection(row)` functions). Every write uses the same functions.- **vod / clip.**- Local copy: the file under `VOD_PATH` (vods, resolved by basename as playback does) or the clip's own path.@@@ -81,29 +80,12 @@ the same functions.- **screenshot / avatar.** `screenshot_path`. Text pastes have no bytes and no object; paste text belongs to Community.- **thumbnail.** One object per vod/clip, reached through the parent's `thumbnail` variant and **updated in place** when the picture is regenerated (live recordings refresh it every couple of minutes). Visibility is `public` when the parent is public and `unlisted` otherwise, because `/t/<name>` serves anyone who has the name. An external `thumbnail_url` is skipped.--## Backfill----`backfill()` in `server/objects/backfill.js` is a module, not a CLI: it is called from code, and the service calls it--itself with `{ onlyMissing: true }` 15 s after boot. Read-only questions about it are answered by the--[drift report](#drift-report). It creates one object per vod, clip, file, screenshot/avatar paste--and thumbnail, with relationships and thumbnail variants.----- **Idempotent.** Objects are keyed by `legacy_ref`, so a second run updates them in place and creates nothing.--- **Never moves or deletes bytes, and never calls B2/R2.** A local copy is `present` when the file exists and `missing` when it does not. Remote copies stay `pending` until `reconcile-objects.js --verify` checks them.--- **One transaction,** with one savepoint per row: a failing row is reported and leaves nothing half-written. A dry run (`{ dryRun: true }`) rolls everything back and only reports.--- **Report:** counts per kind (seen, created, updated, skipped), location states, skipped rows with a reason, and errors. A real run stores the report in `media_settings` under `objects.backfill.last_report`; `lastReport()` reads it back and `/me/ops` shows it.--- **At boot:** the service runs the `onlyMissing` form itself 15 s after start, for rows with no `object_id`. The first boot after the upgrade therefore projects everything. With object-first writes it should find nothing; it stays until the [drift report](#drift-report) shows zero drift across a release.----Skip reasons today: `clips-only recording (ephemeral, never published)`, `screenshot paste without a file path`, `external thumbnail url`.----The backfill's own transaction holds row and table locks for its duration, typically seconds, so it runs only--where the service controls it: the boot backfill.++The historical projection of imported rows is preserved for tests in `test/helpers/project-rows.js`. It is not a production path.### Keeping the model current: object-first writesEvery write to a projected row writes its object in the same PostgreSQL transaction--(`model.withObject(kind, ids, write)`; roadmap WS-G task 1, which retires compatibility shim C-75,--"write the row, then sync the object"). The row change and the re-projection commit together or++(`model.withObject(kind, ids, write)`; roadmap WS-G task 1; compatibility shim C-75 was retired on 2026-10-10). The row change and the re-projection commit together ornot at all, so a crash or an error between them can no longer leave the object behind its row. Awrite whose object cannot be written fails as a whole and changes nothing. Inside`webhooks.announce()` the object joins the outcome's transaction, and its object events are queued@@@ -134,28 +116,13 @@ content hashes come from the content-hash job, and remote copies stay `pending`reconciliation verifies them. Two writes record something that already happened outside thedatabase, so they never lose the row to its object: the recorder's start (ffmpeg is running) and thetier moves (the bytes have moved). If their combined transaction fails, the row is written alone and--re-projected after it (`withObjectOrRow`), with a warning. After a move that removed the local file,--and after a finalize that threw part-way, a follow-up re-projection re-reads the disk. Columns no++re-projected after it (`withObjectOrRow`), with a warning. After a move that removed the local file, a follow-up re-projection re-reads the disk. Columns noobject carries (view counts, `master_file_path`, `cut_attempts`, paste likes and AI fields) are--written without one. `db.importLegacyRows` (the cutover import) is not converted: the boot backfill--projects what it inserts.----### Drift report----```--node scripts/object-drift-report.js [--app live] [--limit 20] [--json] [--out report.json]--```++written without one. `db.importLegacyRows` is the historical cutover import.--For every projected row, and each vod/clip thumbnail, the report derives what the object must say--with the same projection functions and compares it with the object on record: `kind`, `visibility`,--`lifecycle_status`, `size_bytes`, `legacy_ref`, `app_id`, `owner_app`, `owner_user_id`, and the link--(the row's `object_id`, or the parent's `thumbnail` variant). A row is `missing` (no object),--`unlinked` (the link names another object) or `mismatch` (a field differs). It prints counts per--kind and up to `--limit` examples of each. The database is opened read-only and nothing is written;--the exit code is always 0.++### C-75 retirement--The boot backfill (rows with no object) and the finalize follow-up stay until a release has run with--zero drift. They are removed in a later, dated step.++C-75 was retired on 2026-10-10 after production drift was 0 of 3036 rows (vod 736, clip 363, file 29, screenshot 819, avatar 3, thumbnail 1086). Every write makes its media object in the same transaction through `model.withObject`. Operators can still check objects with `scripts/reconcile-objects.js` and `scripts/object-invariant.js`.## Owner subjects@@@ -447,7 +414,7 @@ cheap and renders in a drill.- `jobs`: counts by status; failed jobs of the last 7 days by type and error code; the most recent failures with their error text.- `missing`: ready objects with no good copy; copies recorded missing or corrupt; copies by provider and state; quarantined VODs; the last verification run.--- `backfill`: the last object backfill that changed something; rows with no object yet; objects whose owner subject is unresolved; whether upload reservations were seeded.++- `projection`: rows with no object yet; objects whose owner subject is unresolved; whether upload reservations were seeded.- `tiering`: providers and policy; VODs and clips per provider; objects per canonical copy (native or projected); VODs eligible to offload; R2 decisions of the last 24 h and recent refusals and failures; the sweep's state in this process; and `native_objects`: the [object tiering](#tiering-of-native-objects)'s activation gate and policy, native objects in R2, how many are eligible to promote now, its decisions of the last 24 h (dry runs included) and the most recent ones, and its last sweep.- `namespaces`: every namespace with quota and usage snapshot.- `webhooks`: the apps with a configured outbound webhook (app id and host only — never the URL or secret) and, per app, what this process has sent since it loaded (`sent`, `failed`, the last event and attempt/ok times, the last error). In-memory: counts reset on restart and scope to the app when the report is scoped.@@@ -553,7 +520,7 @@ The exit code is 1 when issues were found.| `size_mismatch` / `hash_mismatch` | the bytes differ from the object's size or sha256 || `orphan_location` | a location row whose object does not exist || `deleted_publicly_reachable` | a deleted object whose inherited row still serves it, or a thumbnail of a deleted object that is still served by name |--| `missing_projection` | an inherited row with no object yet (run the backfill) |++| `missing_projection` | an inherited row with no object yet (investigate the write path) || `incomplete_multipart` | a multipart session still open past its expiry (the hourly purge should have removed it) |Bucket keys and local files that no row names are the [storage orphan report](#storage-orphan-report)'s.ddeleted file mode 100644@@@ -1,44 +0,0 @@--#!/usr/bin/env node--'use strict';--/**-- * Object drift report (WS-G task 1, retiring C-75; server/objects/drift.js has the rules): every-- * projected vods/clips/files/pastes row, and each vod/clip thumbnail, whose media_object is missing,-- * not linked, or disagrees with the row on kind, visibility, lifecycle status, size, legacy ref or-- * owner/app. Prints counts per kind and up to --limit examples of each. READ-ONLY: the database is-- * opened read-only and only local files are stat'ed (paths from the service's environment:-- * VOD_PATH, CLIPS_PATH, FILES_PATH, THUMBNAILS_PATH, as the projection uses them).-- *-- * node scripts/object-drift-report.js [--app live] [--limit 20] [--json] [--out report.json]-- *-- * Exit code 0 always: it is a report (an error is printed to stderr). Zero drift across a release-- * is the signal to remove the boot backfill (server/index.js) and the finalize follow-up.-- */--const fs = require('fs');--const path = require('path');----(async () => {-- await require('../server/db/database').initDb(); // PostgreSQL (DATABASE_URL), as the service-- const args = process.argv.slice(2);-- const has = (name) => args.includes(`--${name}`);-- const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] && !args[i + 1].startsWith('--') ? args[i + 1] : dflt; };---- try {-- // Read-only: the report runs in a READ ONLY transaction (PostgreSQL refuses any write inside it).-- const { buildReport, formatReport } = require('../server/objects/drift');-- const limit = Math.max(0, parseInt(arg('limit', '20'), 10) || 0);-- const d = require('../server/db/database').getDb();-- const q = {-- all: async (sql, params = []) => await d.prepare(sql).all(...params),-- get: async (sql, params = []) => await d.prepare(sql).get(...params),-- };-- const report = await d.tx(async (t) => {-- await t.query('SET TRANSACTION READ ONLY');-- return await buildReport(q, { appId: arg('app', null), limit });-- });-- if (arg('out')) fs.writeFileSync(arg('out'), JSON.stringify(report, null, 2));-- console.log(has('json') ? JSON.stringify(report, null, 2) : formatReport(report));-- } catch (err) {-- console.error(`object-drift-report: ${err.message}`);-- }-- process.exit(0);--})().catch((err) => { console.error(err); process.exit(1); });@@@ -35,7 +35,7 @@* and each change is logged as an [Admin] line* POST /holds/:holdId/release { released_by } (also DELETE /holds/:holdId); 409 when already released* GET /ops this app's operator report (server/me/ops.js, as openvibe.media/me/ops shows-- * it to Network staff): failed jobs, missing media, backfill, tiering, namespaces++ * it to Network staff): failed jobs, missing media, projection, tiering, namespaces* POST /ops/recompute refresh this app's namespaces' usage snapshot from the rows*/'use strict';@@@ -665,7 +665,7 @@ function _opsStaffOnly(req, res) {return false;}--// GET /ops?limit — failed jobs, missing media, backfill status, tiering diagnostics and the namespaces'++// GET /ops?limit — failed jobs, missing media, projection status, tiering diagnostics and the namespaces'// usage snapshot of this app (server/me/ops.js; the sweep and the provider switches are service-wide).router.get('/ops', async (req, res) => {try {@@@ -75,7 +75,7 @@ function heldSql(ref) {}/**-- * Object-first write (WS-G task 1, retiring C-75): run write() — a change to a projected++ * Object-first write (WS-G task 1, C-75 retired 2026-10-10): run write() — a change to a projected* vods/clips/files/pastes row — and re-project the row's media_object in the same transaction* (objects/model.js withObject). Throws, with nothing written, when either part fails.*/@@@ -21,7 +21,7 @@* - server/index.js opens the database and serves HTTP, and starts nothing else: no app seeding, no* JWKS refresh, no health job, junk sweep, tiering sweep, clip re-cuts, copy verification, jobs* worker (thumbnails, split/remux, invariant scans, storage orphan scans), disk guardian, thumbnail cleanup, object purge,-- * backfill, orphan-recording finalize or Events relay; webhooks are never sent. Timers that modules++ * orphan-recording finalize or Events relay; webhooks are never sent. Timers that modules* start when they are loaded check `enabled` as well. Readiness does not probe storage (a drill* writes none) or the remote tiers.*@@@ -303,18 +303,8 @@ const ready = (async () => {await ns.reconcileAll();} catch (err) { console.warn('[Namespaces] sweep:', err.message); }});-- // Project rows that have no media_object yet (first boot after the upgrade: all of them).-- // C-75 catch-up: every write now makes its object in the same transaction (objects/model.js-- // withObject, WS-G task 1), so this should find nothing. It stays until a release has run with-- // `node scripts/object-drift-report.js` showing zero drift; then it goes, with the finalize-- // follow-up re-projection (vod/finalize.js), in a dated later step.++ // Take the first namespaces usage snapshot after boot (then hourly and after each write).setTimeout(async () => {-- try {-- const bf = require('./objects/backfill');-- const r = await bf.backfill({ onlyMissing: true });-- if (r.totals.created || r.totals.updated || r.errors.length) console.log(`[Objects] Backfill: ${bf.summarize(r)}`);-- } catch (err) { console.warn('[Objects] Backfill failed:', err.message); }-- // The namespaces' usage snapshot, first taken here (then hourly, and after each write).try { await require('./objects/namespaces').reconcileAll(); } catch (err) { console.warn('[Namespaces] reconcile:', err.message); }}, 15 * 1000).unref?.();@@@ -2,7 +2,7 @@* OpenVibe.Media — operator views over the object platform (roadmap WS-G task 12;* docs/object-model.md#operator-views).*-- * report({ appId, limit }) failed jobs, missing media, backfill status, tiering diagnostics, the++ * report({ appId, limit }) failed jobs, missing media, projection status, tiering diagnostics, the* namespaces' usage snapshot and the outbound-webhooks section; appId* narrows everything that has a tenant (the tiering sweep and the provider* switches are service-wide)@@@ -98,15 +98,14 @@ async function missing(appId, limit) {locations: { by_provider_state: byState, missing_or_corrupt: bad },vods_quarantined: quarantined,verification,-- note: 'From what the database records (scheduled verification, tier moves, reconciliation). A full walk of storage is the storage.orphans.scan report; drift between rows and objects is scripts/object-drift-report.js.',++ note: 'From what the database records (scheduled verification, tier moves, reconciliation). A full walk of storage is the storage.orphans.scan report.',};}--// ── Backfill ─────────────────────────────────────────────────++// ── Projection ───────────────────────────────────────────────--async function backfill(appId) {++async function projection(appId) {const s = scoped(appId);-- const last = await require('../objects/backfill').lastReport();const unprojected = {vods: (await db.get(`SELECT COUNT(*) AS n FROM vods WHERE object_id IS NULL AND COALESCE(clips_only, 0) = 0${s.sql}`, s.params)).n,clips: (await db.get(`SELECT COUNT(*) AS n FROM clips WHERE object_id IS NULL${s.sql}`, s.params)).n,@@@ -116,16 +115,10 @@ async function backfill(appId) {const owner = await db.all(`SELECT app_id, COUNT(*) AS n FROM media_objects WHERE owner_subject IS NULL AND owner_user_id IS NOT NULLAND lifecycle_status != 'deleted'${s.sql} GROUP BY app_id ORDER BY n DESC`, s.params);return {-- objects: last ? {-- finished_at: iso(last.finished_at) || null, dry_run: !!last.dry_run, only_missing: !!last.only_missing,-- totals: last.totals || null, locations: last.locations || null,-- errors: Array.isArray(last.errors) ? last.errors.length : 0,-- skipped: Array.isArray(last.skipped) ? last.skipped.length : 0,-- } : null,unprojected,owner_subject: { missing: owner.reduce((a, r) => a + r.n, 0), by_app: owner.map(r => ({ app_id: r.app_id, count: r.n })) },namespace_reservations_seeded: !!await db.get("SELECT 1 AS x FROM media_settings WHERE key = 'namespaces.reservations_seeded'"),-- note: 'objects: the last boot or scripted backfill that changed something (media_settings objects.backfill.last_report). unprojected: rows with no object yet. owner_subject: objects the owner-subject job has not resolved.',++ note: 'unprojected: rows with no object yet. owner_subject: objects the owner-subject job has not resolved.',};}@@@ -238,7 +231,7 @@ async function report({ appId = null, limit = 50 } = {}) {tenants: (await db.all('SELECT app_id FROM apps ORDER BY app_id')).map(r => r.app_id),jobs: await jobs(appId, n),missing: await missing(appId, n),-- backfill: await backfill(appId),++ projection: await projection(appId),tiering: await tiering(appId, n),namespaces: await namespaceSnapshot(appId),webhooks: await webhooks(appId),@@@ -332,7 +332,7 @@ function renderOps({ person, report, canRecompute, recomputed = null }) {const kv = (obj) => Object.entries(obj || {}).map(([k, v]) => `${esc(k.replace(/_/g, ' '))} ${num(v)}`).join(' · ') || 'none';const t = r.tiering;const no = t.native_objects || null;-- const bf = r.backfill;++ const projection = r.projection;const m = r.missing;const body = `<h1>Media operations</h1>@@@ -346,7 +346,7 @@ function renderOps({ person, report, canRecompute, recomputed = null }) {${card(num(r.jobs.failed.total), 'failed jobs')}${card(num(m.no_good_copy.total), 'ready objects with no good copy')}${card(num(m.locations.missing_or_corrupt.length), 'missing or corrupt copies listed')}-- ${card(num(Object.values(bf.unprojected).reduce((a, b) => a + b, 0)), 'rows without an object')}++ ${card(num(Object.values(projection.unprojected).reduce((a, b) => a + b, 0)), 'rows without an object')}${card(num(t.pending_offload), 'VODs eligible to offload')}</div>@@@ -392,14 +392,13 @@ function renderOps({ person, report, canRecompute, recomputed = null }) {${table('Copies by provider and state', ['Provider', 'State', 'Copies'], m.locations.by_provider_state.map(x => [esc(x.provider), esc(x.state), num(x.count)]), { numeric: [2] }) || '<p class="empty">None.</p>'}</section>-- <section aria-labelledby="backfill-h">-- <h2 id="backfill-h">Backfill</h2>-- <p class="hint">${esc(bf.note)}</p>++ <section aria-labelledby="projection-h">++ <h2 id="projection-h">Projection</h2>++ <p class="hint">${esc(projection.note)}</p><dl class="facts">-- <dt>Last object backfill</dt><dd>${bf.objects ? `${esc(fmtWhen(bf.objects.finished_at))}${bf.objects.dry_run ? ' (dry run)' : ''}: ${bf.objects.totals ? kv(bf.objects.totals) : 'no totals'}; ${num(bf.objects.errors)} errors, ${num(bf.objects.skipped)} skipped` : 'none recorded'}</dd>-- <dt>Rows without an object</dt><dd>${kv(bf.unprojected)}</dd>-- <dt>Owner subject unresolved</dt><dd>${num(bf.owner_subject.missing)}${bf.owner_subject.by_app.length ? ` (${bf.owner_subject.by_app.map(a => `${esc(a.app_id)} ${num(a.count)}`).join(', ')})` : ''}</dd>-- <dt>Upload reservations seeded</dt><dd>${bf.namespace_reservations_seeded ? 'yes' : 'no'}</dd>++ <dt>Rows without an object</dt><dd>${kv(projection.unprojected)}</dd>++ <dt>Owner subject unresolved</dt><dd>${num(projection.owner_subject.missing)}${projection.owner_subject.by_app.length ? ` (${projection.owner_subject.by_app.map(a => `${esc(a.app_id)} ${num(a.count)}`).join(', ')})` : ''}</dd>++ <dt>Upload reservations seeded</dt><dd>${projection.namespace_reservations_seeded ? 'yes' : 'no'}</dd></dl></section>@@@ -441,7 +440,7 @@ function renderOps({ person, report, canRecompute, recomputed = null }) {}), { numeric: [2, 3] }) || '<p class="empty">No app is configured to receive webhooks.</p>'}</section><p class="hint">JSON: <a href="/api/v2/me/ops${scopeQs}">/api/v2/me/ops</a>.</p>`;-- return doc({ title: 'Media operations', description: 'Operator views of OpenVibe.Media: failed jobs, usage, missing media, backfill, tiering and webhooks.', body });++ return doc({ title: 'Media operations', description: 'Operator views of OpenVibe.Media: failed jobs, usage, missing media, projection, tiering and webhooks.', body });}module.exports = {@@@ -15,7 +15,7 @@* Operator views, for Network staff (contracts staff map, ADR-022): staff.site.view reads,* staff.site.configure recomputes (POST, same-origin only when the credential is the cookie):*-- * GET /api/v2/me/ops[?app] failed jobs, missing media, backfill, tiering, namespaces (me/ops.js)++ * GET /api/v2/me/ops[?app] failed jobs, missing media, projection, tiering, namespaces (me/ops.js)* POST /api/v2/me/ops/recompute[?app] refresh the namespaces' usage snapshot* GET /me/ops, POST /me/ops/recompute the page and its form*ddeleted file mode 100644@@@ -1,111 +0,0 @@--/**-- * OpenVibe.Media — object backfill-- *-- * One media_object (+ locations, relationships, thumbnail variant) per existing-- * vod, clip, file, screenshot/avatar paste and vod/clip thumbnail, derived by the-- * same sync functions every write uses (model.withObject). Idempotent: objects are keyed by their-- * legacy ref, so a re-run updates in place and creates nothing new.-- *-- * Never touches bytes and never calls B2/R2: local copies are 'present' when the-- * file exists on disk and 'missing' when it does not; remote copies stay 'pending'-- * until `scripts/reconcile-objects.js --verify` HEADs them.-- *-- * Runs in one transaction; dryRun rolls it back and only reports.-- */--'use strict';----const db = require('../db/database');--const model = require('./model');----const SETTING_KEY = 'objects.backfill.last_report';----function emptyCounts() {-- return { seen: 0, created: 0, updated: 0, skipped: 0 };--}----async function backfill({ dryRun = false, onlyMissing = false } = {}) {-- const report = {-- started_at: new Date().toISOString(), dry_run: !!dryRun, only_missing: !!onlyMissing,-- counts: { vod: emptyCounts(), clip: emptyCounts(), file: emptyCounts(), screenshot: emptyCounts(), avatar: emptyCounts(), thumbnail: emptyCounts() },-- locations: { present: 0, missing: 0, pending: 0 },-- skipped: [],-- errors: [],-- };-- const where = onlyMissing ? ' WHERE object_id IS NULL' : '';---- const tally = (kind, table, id, r) => {-- const c = report.counts[kind];-- c.seen++;-- if (!r) return;-- if (r.skipped) { c.skipped++; report.skipped.push({ table, id, reason: r.skipped }); return; }-- c[r.created ? 'created' : 'updated']++;-- for (const s of r.locations || []) if (s in report.locations) report.locations[s]++;-- };-- const thumb = (table, id, r) => {-- if (!r || !r.thumbnail) return;-- tally('thumbnail', `${table}.thumbnail_url`, id, r.thumbnail);-- };-- const d = db.getDb();-- // One savepoint per row (a nested transaction): a row that fails leaves nothing half-written behind.-- const each = async (table, sql, fn) => {-- for (const row of await db.all(sql)) {-- try { await d.tx(async () => { await fn(row); }); } catch (err) { report.errors.push({ table, id: row.id ?? row.key, error: err.message }); }-- }-- };---- // One transaction for the run; a dry run rolls it back at the end.-- const DRY = Symbol('dry run');-- try {-- await d.tx(async () => {-- // VODs before clips, so clip_of can point at the vod's object.-- await each('vods', `SELECT * FROM vods${where} ORDER BY id`, async (row) => {-- const r = await model.syncVod(row);-- tally('vod', 'vods', row.id, r);-- thumb('vods', row.id, r);-- });-- await each('clips', `SELECT * FROM clips${where} ORDER BY id`, async (row) => {-- const r = await model.syncClip(row);-- tally('clip', 'clips', row.id, r);-- thumb('clips', row.id, r);-- });-- await each('files', `SELECT * FROM files${where} ORDER BY created_at, key`, async (row) => {-- tally('file', 'files', row.key, await model.syncFile(row));-- });-- await each('pastes', `SELECT * FROM pastes${where ? where + ' AND' : ' WHERE'} type = 'screenshot' ORDER BY id`, async (row) => {-- const r = await model.syncPaste(row);-- const kind = (r && r.kind) || (String(row.metadata || '').includes('"kind":"avatar"') ? 'avatar' : 'screenshot');-- tally(kind, 'pastes', row.id, r);-- });-- if (dryRun) throw DRY;-- });-- } catch (err) {-- if (err !== DRY) throw err;-- }---- report.finished_at = new Date().toISOString();-- report.totals = Object.values(report.counts).reduce((t, c) => {-- for (const k of Object.keys(t)) t[k] += c[k];-- return t;-- }, emptyCounts());-- if (!dryRun && (!onlyMissing || report.totals.created || report.totals.updated)) {-- // Keep the latest report (skipped rows capped) for operators: media_settings.objects.backfill.last_report.-- const stored = { ...report, skipped: report.skipped.slice(0, 500), skipped_truncated: report.skipped.length > 500 };-- await db.run(`INSERT INTO media_settings (key, value, description, type) VALUES (?, ?, 'Last object backfill report', 'json')-- ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = ov_now()`, [SETTING_KEY, JSON.stringify(stored)]);-- }-- return report;--}----async function lastReport() {-- const row = await db.get('SELECT value FROM media_settings WHERE key = ?', [SETTING_KEY]);-- return row ? model.parseJson(row.value, null) : null;--}----function summarize(report) {-- const parts = Object.entries(report.counts).filter(([, c]) => c.seen)-- .map(([k, c]) => `${k} ${c.created}+/${c.updated}~/${c.skipped} skipped`);-- return `${report.dry_run ? '[dry run] ' : ''}${parts.join(', ') || 'nothing to do'}; locations present ${report.locations.present}, `-- + `missing ${report.locations.missing}, pending ${report.locations.pending}${report.errors.length ? `; ${report.errors.length} error(s)` : ''}`;--}----module.exports = { backfill, lastReport, summarize, SETTING_KEY };ddeleted file mode 100644@@@ -1,152 +0,0 @@--/**-- * OpenVibe.Media — object drift report (WS-G task 1: retiring compatibility shim C-75)-- *-- * Every write to an inherited vods/clips/files/pastes row now makes its media_object in the same-- * transaction (model.withObject). This report checks that nothing has drifted anyway: for each-- * projected row it derives what the object must say, with the same <kind>Projection() functions-- * the writes use, and compares that with the object on record, on the fields a sync writes:-- *-- * kind, visibility, lifecycle_status, size_bytes, legacy_ref, app_id, owner_app, owner_user_id-- *-- * and the link to it (the row's object_id; for a vod/clip thumbnail, its parent's thumbnail-- * variant). Each checked row is ok, or has one or more of:-- * missing no object under its legacy ref, nor at the row's object_id-- * unlinked the row's object_id (the thumbnail variant) is not the object its legacy ref names-- * mismatch the object disagrees with the row on one of the fields above-- * Rows that have no object by design (clips-only recordings, screenshot pastes without a file,-- * external thumbnail URLs) are counted as skipped; text pastes are not projected and not listed.-- *-- * READ-ONLY. Pure SQL over a small `q` interface ({ all(sql, params), get(sql, params) }), so the-- * script (scripts/object-drift-report.js) runs it on a database opened read-only; beyond that it-- * only stats local files, as the projection does. Nothing is written, re-projected or moved.-- */--'use strict';----const model = require('./model');----const FIELDS = ['kind', 'visibility', 'lifecycle_status', 'size_bytes', 'legacy_ref', 'app_id', 'owner_app', 'owner_user_id'];--const KINDS = ['vod', 'clip', 'file', 'screenshot', 'avatar', 'thumbnail'];--const PAGE = 500;--const OBJECT_COLS = `id, ${FIELDS.join(', ')}`;----/** The compared fields as project() writes them from a projection input. */--function expectedFields(p) {-- return {-- kind: p.kind, visibility: p.visibility, lifecycle_status: p.lifecycle_status, size_bytes: Number(p.size_bytes) || 0,-- legacy_ref: p.legacy_ref, app_id: p.app_id, owner_app: p.owner_app || p.app_id, owner_user_id: p.owner_user_id ?? null,-- };--}----// The store hands bigint back as a number or a string and TEXT as a string; an app-local user id may be either.--const same = (a, b) => (a == null ? '' : String(a)) === (b == null ? '' : String(b));----/** Every row of `table` (optionally one tenant's) in key order, a page at a time. */--async function* eachRow(q, table, keyCol, { appId = null, where = null } = {}) {-- const conds = [];-- const params = [];-- if (appId) { conds.push('app_id = ?'); params.push(appId); }-- if (where) conds.push(where);-- let last = null;-- for (;;) {-- const c = last == null ? conds : [...conds, `${keyCol} > ?`];-- const page = await q.all(`SELECT * FROM ${table}${c.length ? ` WHERE ${c.join(' AND ')}` : ''} ORDER BY ${keyCol} LIMIT ${PAGE}`,-- last == null ? params : [...params, last]);-- for (const r of page) yield r;-- if (page.length < PAGE) return;-- last = page[page.length - 1][keyCol];-- }--}----async function buildReport(q, { appId = null, limit = 20 } = {}) {-- const report = {-- generated_at: new Date().toISOString(),-- app_id: appId,-- limit,-- fields: FIELDS,-- kinds: Object.fromEntries(KINDS.map(k => [k, { rows: 0, ok: 0, skipped: 0, drifted: 0, missing: 0, unlinked: 0, mismatch: 0, examples: [] }])),-- field_counts: Object.fromEntries(FIELDS.map(f => [f, 0])),-- total_rows: 0,-- total_drift: 0,-- };-- // Objects by legacy ref, loaded once (the compared columns only); a row's object_id is looked up when its ref finds none.-- const byRef = new Map((await q.all(`SELECT ${OBJECT_COLS} FROM media_objects WHERE legacy_ref IS NOT NULL`)).map(o => [o.legacy_ref, o]));-- const byId = async (id) => (id ? await q.get(`SELECT ${OBJECT_COLS} FROM media_objects WHERE id = ?`, [id]) || null : null);---- /** Compare one row's projection with its object. Returns the object found (or null). */-- const check = async (kind, where, p, linkedId) => {-- const k = report.kinds[kind];-- k.rows++;-- report.total_rows++;-- const found = byRef.get(p.legacy_ref) || await byId(linkedId);-- const problems = [];-- const diff = {};-- if (!found) {-- problems.push('missing');-- } else {-- if (!same(linkedId, found.id)) { problems.push('unlinked'); diff.object_id = { row: linkedId || null, object: found.id }; }-- const want = expectedFields(p);-- for (const f of FIELDS) {-- if (same(want[f], found[f])) continue;-- diff[f] = { row: want[f], object: found[f] ?? null };-- report.field_counts[f]++;-- }-- if (FIELDS.some(f => f in diff)) problems.push('mismatch');-- }-- if (!problems.length) { k.ok++; return found; }-- k.drifted++;-- report.total_drift++;-- for (const pr of problems) k[pr]++;-- if (k.examples.length < limit) k.examples.push({ ...where, legacy_ref: p.legacy_ref, object_id: found ? found.id : null, problems, diff });-- return found;-- };-- const skip = (kind) => { report.kinds[kind].skipped++; };-- const thumbnailOf = async (row, table, parentKind, parent) => {-- const tp = model.thumbnailProjection(row, parentKind);-- if (!tp) return;-- if (tp.skipped) return skip('thumbnail');-- const variant = parent ? await q.get("SELECT derived_object_id FROM media_variants WHERE object_id = ? AND variant_name = 'thumbnail'", [parent.id]) : null;-- await check('thumbnail', { table, id: row.id, of: parentKind }, tp, variant ? variant.derived_object_id : null);-- };---- for await (const row of await eachRow(q, 'vods', 'id', { appId })) {-- const p = model.vodProjection(row);-- if (p.skipped) { skip('vod'); continue; }-- await thumbnailOf(row, 'vods', 'vod', await check('vod', { table: 'vods', id: row.id }, p, row.object_id));-- }-- for await (const row of await eachRow(q, 'clips', 'id', { appId })) {-- await thumbnailOf(row, 'clips', 'clip', await check('clip', { table: 'clips', id: row.id }, model.clipProjection(row), row.object_id));-- }-- for await (const row of await eachRow(q, 'files', 'key', { appId })) {-- await check('file', { table: 'files', id: row.key }, model.fileProjection(row), row.object_id);-- }-- for await (const row of await eachRow(q, 'pastes', 'id', { appId, where: "type = 'screenshot'" })) {-- const p = model.pasteProjection(row);-- if (p.skipped) { skip('screenshot'); continue; }-- await check(p.kind, { table: 'pastes', id: row.id, slug: row.slug }, p, row.object_id);-- }-- return report;--}----const fmt = (v) => (v == null ? 'null' : typeof v === 'string' ? v : JSON.stringify(v));----function formatReport(r) {-- const lines = [`Object drift report — ${r.generated_at}${r.app_id ? ` (app ${r.app_id})` : ' (all apps)'}`];-- lines.push(` ${'kind'.padEnd(11)} ${'rows'.padStart(7)} ${'ok'.padStart(7)} ${'drifted'.padStart(8)} ${'missing'.padStart(8)} ${'unlinked'.padStart(9)} ${'mismatch'.padStart(9)} ${'skipped'.padStart(8)}`);-- for (const [kind, c] of Object.entries(r.kinds)) {-- lines.push(` ${kind.padEnd(11)} ${String(c.rows).padStart(7)} ${String(c.ok).padStart(7)} ${String(c.drifted).padStart(8)} ${String(c.missing).padStart(8)} ${String(c.unlinked).padStart(9)} ${String(c.mismatch).padStart(9)} ${String(c.skipped).padStart(8)}`);-- }-- const fields = Object.entries(r.field_counts).filter(([, n]) => n).map(([f, n]) => `${f} ${n}`);-- lines.push(` fields that disagree: ${fields.join(', ') || 'none'}`);-- lines.push(` total drift: ${r.total_drift} of ${r.total_rows} row(s)${r.total_drift ? '' : ' — none'}`);-- for (const [kind, c] of Object.entries(r.kinds)) {-- if (!c.examples.length) continue;-- lines.push(` ${kind} examples (${c.examples.length} of ${c.drifted}):`);-- for (const e of c.examples) {-- const what = Object.entries(e.diff).map(([f, d]) => `${f} row=${fmt(d.row)} object=${fmt(d.object)}`).join('; ');-- lines.push(` ${e.table} ${e.id}${e.of ? ` (${e.of} thumbnail)` : ''} ${e.object_id || '-'} ${e.problems.join('+')}${what ? `: ${what}` : ` (${e.legacy_ref})`}`);-- }-- }-- return lines.join('\n');--}----module.exports = { buildReport, formatReport, expectedFields, FIELDS, KINDS };@@@ -5,10 +5,8 @@* row per provider copy. The inherited vods/clips/files/pastes rows are typed* projections over objects: each carries object_id, and the sync*() functions* below derive the object + locations from the row as it is right now. The same-- * functions back the one-off backfill and every write on the old APIs, which-- * writes the row and its object in one transaction (withObject, WS-G task 1), so-- * the model cannot drift between "imported" and "written since". The drift-- * report (./drift.js) checks that it has not.++ * functions back every write on the old APIs, which writes the row and its object++ * in one transaction (withObject, WS-G task 1).** Location states: present (verified: the local file exists / a HEAD answered),* missing (verified absent), pending (believed there, not yet verified — every@@@ -49,7 +47,7 @@ function parseJson(s, fallback) {try { return JSON.parse(s); } catch { return fallback; }}--/** Epoch ms from a `YYYY-MM-DD HH:MM:SS` (UTC) text timestamp, as the migrations store them, so backfilled ids sort by original creation time. */++/** Epoch ms from a `YYYY-MM-DD HH:MM:SS` (UTC) text timestamp, as the migrations store them, so projected legacy ids sort by original creation time. */function toMs(createdAt) {const ms = Date.parse(String(createdAt || '').replace(' ', 'T') + (/[zZ]|[+-]\d\d:?\d\d$/.test(String(createdAt || '')) ? '' : 'Z'));return Number.isFinite(ms) ? ms : Date.now();@@@ -302,9 +300,8 @@ async function recordInvariant(objectId) {// ── Per-kind projections ─────────────────────────────────────// <kind>Projection(row) derives what the object behind one inherited row must say (project()'s input).--// Read-only (it stats local files, never writes), and shared by the sync*() writers below and the--// drift report (./drift.js), so the report checks exactly what a write would have written. A row--// that has no object of its own answers { skipped: reason }.++// Read-only (it stats local files, never writes), and shared by the sync*() writers below.++// A row that has no object of its own answers { skipped: reason }.function vodLifecycle(row) {const s = db.vodStatus(row);@@@ -519,7 +516,7 @@ async function safeSync(kind, id) {}/**-- * Object-first write (WS-G task 1; retires compatibility shim C-75, "write the row, then sync its++ * Object-first write (WS-G task 1; C-75 retired 2026-10-10, "write the row, then sync its* object"): write() changes inherited rows and the objects behind them are re-projected in the SAME* transaction, so both commit or neither does: no crash or error between the two can leave* an object behind its row. kind vod|clip|file|paste; ids the row id (a file's key), a list of them,@@@ -546,7 +543,7 @@ async function withObject(kind, ids, write) {* withObject() for a row write that records something already done outside the database (a* recorder's ffmpeg is running): the row must say so whatever happens to its object. When the* combined transaction fails, the row is written alone and the object re-projected after it (the-- * pre-WS-G two steps; logged, and the drift report lists the row until a later write catches up).++ * pre-WS-G two steps; logged, and the row's next write brings its object level).*/async function withObjectOrRow(kind, ids, write) {try { return await withObject(kind, ids, write); } catch (err) {@@@ -4,13 +4,12 @@** Every MEDIA_OWNER_SUBJECT_INTERVAL_MIN minutes (first run a minute after boot), objects that name* an app-local owner (owner_app + owner_user_id) and no owner_subject get their owner's canonical-- * subject from Network (server/objects/owner-subject.js). This is how objects created since the-- * backfill get one: the upload and projection paths never wait on Network, and an object whose owner-- * Network does not know yet (a Live account not linked to Network) is filled on a later run once it is.++ * subject from Network (server/objects/owner-subject.js). This is how new objects get one: the upload++ * and projection paths never wait on Network, and an object whose owner Network does not know yet (a++ * Live account not linked to Network) is filled on a later run once it is.* When there is nothing to fill, a run asks Network nothing.*-- * MEDIA_OWNER_SUBJECT_SYNC=0 turns it off (do that before rolling a backfill back, or the next run-- * fills the rows again).++ * MEDIA_OWNER_SUBJECT_SYNC=0 turns it off.*/const db = require('../db/database');const config = require('../config');@@@ -16,7 +16,7 @@* size_mismatch / hash_mismatch bytes differ from the object's size / sha256* orphan_location location rows whose object does not exist* deleted_publicly_reachable deleted object still served (legacy row, or a thumbnail of it)-- * missing_projection inherited rows with no object yet (run the backfill)++ * missing_projection inherited rows with no object yet (investigate the write path)* incomplete_multipart multipart sessions still open past their expiry (the hourly purge* removes their parts; one listed here means the purge is not running)*/@@@ -166,11 +166,6 @@ async function finalizeVod(vodId, opts = {}) {throw err;} finally {_finalizing.delete(vodId);-- // Every outcome (ready, quarantined, failed) now commits with its object (withObject); a deleted-- // row's object is marked by its trigger. This follow-up only re-reads the disk after a finalize-- // that threw part-way (a merge or remux rewrote the file, no row write followed). C-75 leftover:-- // it goes with the boot backfill (server/index.js) once the drift report shows zero drift.-- await objects().safeSync('vod', vodId);}}@@@ -141,7 +141,7 @@ const { spawnSync, execFileSync } = require('child_process');await db.run("INSERT INTO assets (id, app_id, kind, name, file_path, mime) OVERRIDING SYSTEM VALUE VALUES (1, 'live', 'emote', 'drillWave', ?, 'image/png')", [bytes.asset]);await db.run("INSERT INTO pastes (id, app_id, slug, type, title, screenshot_path, visibility) OVERRIDING SYSTEM VALUE VALUES (1, 'live', 'shot1', 'screenshot', 'Shot', ?, 'public')", [bytes.shot]);await db.run("INSERT INTO files (key, app_id, original_name, size, mime) VALUES ('f1.txt', 'live', 'f1.txt', 13, 'text/plain')");-- await require('../server/objects/backfill').backfill({ onlyMissing: true });++ await require('./helpers/project-rows').projectRows({ onlyMissing: true });const COUNTED = ['media_objects', 'vods', 'clips', 'apps', 'media_locations', 'media_jobs', 'event_outbox'];const counts = async () => Object.fromEntries(await Promise.all(COUNTED.map(async (t) => [t, Number((await db.get(`SELECT COUNT(*) AS n FROM ${t}`)).n)])));const countsBefore = await counts();nnew file mode 100644@@@ -0,0 +1,98 @@++/**++ * OpenVibe.Media — test-only legacy row projection++ *++ * One media_object (+ locations, relationships, thumbnail variant) per existing++ * vod, clip, file, screenshot/avatar paste and vod/clip thumbnail, derived by the++ * same sync functions every write uses (model.withObject). Idempotent: objects are keyed by their++ * legacy ref, so a re-run updates in place and creates nothing new.++ *++ * Never touches bytes and never calls B2/R2: local copies are 'present' when the++ * file exists on disk and 'missing' when it does not; remote copies stay 'pending'++ * until `scripts/reconcile-objects.js --verify` HEADs them.++ *++ * Runs in one transaction; dryRun rolls it back and only reports.++ */++'use strict';++++const db = require('../../server/db/database');++const model = require('../../server/objects/model');++++function emptyCounts() {++ return { seen: 0, created: 0, updated: 0, skipped: 0 };++}++++async function projectRows({ dryRun = false, onlyMissing = false } = {}) {++ const report = {++ started_at: new Date().toISOString(), dry_run: !!dryRun, only_missing: !!onlyMissing,++ counts: { vod: emptyCounts(), clip: emptyCounts(), file: emptyCounts(), screenshot: emptyCounts(), avatar: emptyCounts(), thumbnail: emptyCounts() },++ locations: { present: 0, missing: 0, pending: 0 },++ skipped: [],++ errors: [],++ };++ const where = onlyMissing ? ' WHERE object_id IS NULL' : '';++++ const tally = (kind, table, id, r) => {++ const c = report.counts[kind];++ c.seen++;++ if (!r) return;++ if (r.skipped) { c.skipped++; report.skipped.push({ table, id, reason: r.skipped }); return; }++ c[r.created ? 'created' : 'updated']++;++ for (const s of r.locations || []) if (s in report.locations) report.locations[s]++;++ };++ const thumb = (table, id, r) => {++ if (!r || !r.thumbnail) return;++ tally('thumbnail', `${table}.thumbnail_url`, id, r.thumbnail);++ };++ const d = db.getDb();++ // One savepoint per row (a nested transaction): a row that fails leaves nothing half-written behind.++ const each = async (table, sql, fn) => {++ for (const row of await db.all(sql)) {++ try { await d.tx(async () => { await fn(row); }); } catch (err) { report.errors.push({ table, id: row.id ?? row.key, error: err.message }); }++ }++ };++++ // One transaction for the run; a dry run rolls it back at the end.++ const DRY = Symbol('dry run');++ try {++ await d.tx(async () => {++ // VODs before clips, so clip_of can point at the vod's object.++ await each('vods', `SELECT * FROM vods${where} ORDER BY id`, async (row) => {++ const r = await model.syncVod(row);++ tally('vod', 'vods', row.id, r);++ thumb('vods', row.id, r);++ });++ await each('clips', `SELECT * FROM clips${where} ORDER BY id`, async (row) => {++ const r = await model.syncClip(row);++ tally('clip', 'clips', row.id, r);++ thumb('clips', row.id, r);++ });++ await each('files', `SELECT * FROM files${where} ORDER BY created_at, key`, async (row) => {++ tally('file', 'files', row.key, await model.syncFile(row));++ });++ await each('pastes', `SELECT * FROM pastes${where ? where + ' AND' : ' WHERE'} type = 'screenshot' ORDER BY id`, async (row) => {++ const r = await model.syncPaste(row);++ const kind = (r && r.kind) || (String(row.metadata || '').includes('"kind":"avatar"') ? 'avatar' : 'screenshot');++ tally(kind, 'pastes', row.id, r);++ });++ if (dryRun) throw DRY;++ });++ } catch (err) {++ if (err !== DRY) throw err;++ }++++ report.finished_at = new Date().toISOString();++ report.totals = Object.values(report.counts).reduce((t, c) => {++ for (const k of Object.keys(t)) t[k] += c[k];++ return t;++ }, emptyCounts());++ return report;++}++++function summarize(report) {++ const parts = Object.entries(report.counts).filter(([, c]) => c.seen)++ .map(([k, c]) => `${k} ${c.created}+/${c.updated}~/${c.skipped} skipped`);++ return `${report.dry_run ? '[dry run] ' : ''}${parts.join(', ') || 'nothing to do'}; locations present ${report.locations.present}, `++ + `missing ${report.locations.missing}, pending ${report.locations.pending}${report.errors.length ? `; ${report.errors.length} error(s)` : ''}`;++}++++module.exports = { projectRows, summarize };@@@ -336,7 +336,7 @@ const http = require('http');assert.deepStrictEqual(rep.jobs.failed.last_7_days, [{ type: 'object.sprite', error_code: 'no_video', count: 1 }]);assert.ok(rep.missing.no_good_copy.objects.some(o => o.id === lost), 'a ready object whose only copy is missing');assert.ok(rep.missing.locations.missing_or_corrupt.some(l => l.object_id === lost && l.provider === 'b2'));-- assert.ok(rep.backfill.unprojected && rep.backfill.owner_subject.missing >= 1);++ assert.ok(rep.projection.unprojected && rep.projection.owner_subject.missing >= 1);assert.ok(rep.tiering.objects.some(o => o.origin === 'native') && rep.tiering.sweep && 'pending_offload' in rep.tiering);assert.ok(rep.namespaces.some(n => n.namespace === 'live.avatars'));// Webhooks section: host only, no URL path and no secret anywhere in the payload; counts are@@@ -352,7 +352,7 @@ const http = require('http');assert.ok(r.status === 200 && r.text.includes('mjob_TESTFAILED') && r.text.includes('action="/me/ops/recompute"') && r.text.includes('noindex'));assert.ok(r.text.includes('webhooks-h') && r.text.includes('hooks.example.test'), 'the page shows the webhooks section and the configured host');assert.ok(!r.text.includes(HOOK_URL) && !r.text.includes('ov-private-hook-path') && !r.text.includes(HOOK_SECRET), 'the page never shows the webhook URL path or secret');-- console.log('✅ operator views: staff.site.view only (not users, not global mods); failed jobs, missing media, backfill, tiering, namespaces');++ console.log('✅ operator views: staff.site.view only (not users, not global mods); failed jobs, missing media, projection, tiering, namespaces');// ── Recompute: staff.site.configure, same-origin ──await db.run("UPDATE media_namespaces SET used_bytes = 0, used_objects = 0, reconciled_at = NULL WHERE namespace = 'live'");@@@ -50,7 +50,9 @@ const SEED_PG = `await db.run("INSERT INTO pastes (app_id, slug, type, title, content, visibility) VALUES ('live', 'n1paste', 'paste', 'N-1 paste', 'hello from N-1', 'public')");await db.run("INSERT INTO files (key, app_id, original_name, size, mime) VALUES ('n1file.txt', 'live', 'n1file.txt', 17, 'text/plain')");put('FILES_PATH', 'live/n1file.txt'); put('THUMBNAILS_PATH', 'vod-1-1.jpg');-- await require('./server/objects/backfill').backfill({ onlyMissing: true });++ // N-1 releases before C-75's retirement project rows with server/objects/backfill.js; later ones with the test helper.++ const projectRows = fs.existsSync('./test/helpers/project-rows.js') ? require('./test/helpers/project-rows').projectRows : require('./server/objects/backfill').backfill;++ await projectRows({ onlyMissing: true });const obj = await db.get("SELECT id FROM media_objects WHERE kind = 'vod' ORDER BY created_at, id LIMIT 1");fs.writeFileSync(path.join(process.env.N1_DATA, 'n1-ids.json'), JSON.stringify({ object: obj ? obj.id : null }));await db.close();@@@ -1,6 +1,6 @@'use strict';// Namespaces and grants (roadmap WS-G task 2; server/objects/namespaces.js, server/auth.js VERBS):--// - the backfill on open: developer-project objects move to app.<project_id>[.sandbox], every tenant++// - namespace setup on open: developer-project objects move to app.<project_id>[.sandbox], every tenant// and every namespace an object names gets a row, uploads in progress get a reservation (once);// - five verbs per namespace (read, list, write, delete, transform), the older ids still granting the// newer verbs, strict_verbs refusing them; child namespaces and grants that name only a child;@@@ -1,10 +1,9 @@'use strict';--// Object-first writes (WS-G task 1, retiring compatibility shim C-75): every write to an inherited++// Object-first writes (WS-G task 1, C-75 retired 2026-10-10): every write to an inherited// vods/clips/files/pastes row makes or updates its media_object in the SAME PostgreSQL transaction// (objects/model.js withObject). For each main write path the object is there, linked and agreeing--// with the row, as soon as the write returns; when the object cannot be written, the row write--// rolls back with it (neither row); and the drift report (server/objects/drift.js,--// scripts/object-drift-report.js) writes nothing and finds drift planted behind the model's back.++// with the row as soon as the write returns; when the object cannot be written, the row write++// rolls back with it (neither row).const assert = require('assert');const fs = require('fs');const os = require('os');@@@ -24,7 +23,6 @@ const sharp = require('sharp');const db = require('../server/db/database');const model = require('../server/objects/model');-- const drift = require('../server/objects/drift');const tools = require('../server/vod/media-tools');const thumbService = require('../server/thumbnails/thumbnail-service');const cutter = require('../server/vod/clip-cutter');@@@ -37,7 +35,6 @@ const sharp = require('sharp');await db.upsertApp({ app_id: 'live', api_key: 'live-key-object-first' });const raw = db.getDb();-- const q = { all: async (sql, p = []) => await raw.prepare(sql).all(...p), get: async (sql, p = []) => await raw.prepare(sql).get(...p) };const count = async (table) => (await raw.prepare(`SELECT COUNT(*) AS n FROM ${table}`).get()).n;// No ffmpeg in these paths: the thumbnail generators and the probes answer as a real file would.@@@ -56,15 +53,11 @@ const sharp = require('sharp');const obj = await model.getObject(row.object_id);assert.ok(obj, `${label}: object exists right after the write`);const p = { vods: model.vodProjection, clips: model.clipProjection, files: model.fileProjection, pastes: model.pasteProjection }[table](row);-- const want = drift.expectedFields(p);-- for (const f of drift.FIELDS) assert.strictEqual(String(obj[f] ?? ''), String(want[f] ?? ''), `${label}: object ${f}`);++ for (const [field, value] of Object.entries({ kind: p.kind, visibility: p.visibility, lifecycle_status: p.lifecycle_status, size_bytes: Number(p.size_bytes) || 0, legacy_ref: p.legacy_ref, app_id: p.app_id, owner_app: p.owner_app || p.app_id, owner_user_id: p.owner_user_id })) {++ assert.strictEqual(String(obj[field] ?? ''), String(value ?? ''), `${label}: object ${field}`);++ }return { row, obj };}-- const noDrift = async (label) => {-- const r = await drift.buildReport(q);-- assert.strictEqual(r.total_drift, 0, `${label}:\n${drift.formatReport(r)}`);-- return r;-- };const app = express();app.use(express.json({ limit: '5mb' }));@@@ -224,9 +217,6 @@ const sharp = require('sharp');assert.deepStrictEqual([(await linked('pastes', 'id', avatarId, 'avatar')).obj.kind, (await linked('pastes', 'id', avatarId, 'avatar')).obj.legacy_ref], ['avatar', 'legacy:network:avatar:av1']);console.log('✅ file, screenshot-paste and avatar writes commit with their object; deletes mark it deleted');-- const clean = await noDrift('after every write path');-- assert.ok(clean.total_rows >= 8, 'the report checked the rows written above');--// ── 4. Atomicity: an object write that fails takes the row write with it ──// Every object write fails (as a trigger refusing it would): the model's INSERT/UPDATE of media_objects throws.const realRun = db.run;@@@ -264,56 +254,8 @@ const sharp = require('sharp');await unchanged('thrown after the insert: neither row');db.run = realRun;await finalizeVod(vodId, { fromJob: true });-- await noDrift('after the refused writes');console.log('✅ a write whose object cannot be written leaves neither row (inserts, updates, announce path, a throw after the insert)');-- // ── 5. Drift report: read-only, finds planted drift ──-- await raw.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path) OVERRIDING SYSTEM VALUE VALUES (900, 'live', 3, 'Planted: no object', NULL) RETURNING id").run();-- const upObj = (await db.get('SELECT object_id FROM clips WHERE id = ?', [upId])).object_id;-- await raw.prepare("UPDATE media_objects SET visibility = 'private' WHERE id = ?").run(upObj);-- const fileObj = (await db.get('SELECT object_id FROM files WHERE key = ?', [fileKey])).object_id;-- await raw.prepare('UPDATE media_objects SET size_bytes = 1, owner_user_id = 99 WHERE id = ?').run(fileObj);-- await raw.prepare('UPDATE pastes SET object_id = NULL WHERE slug = ?').run(slug);-- const cutObj = (await db.get('SELECT object_id FROM clips WHERE id = ?', [cutId])).object_id;-- await raw.prepare("UPDATE media_objects SET lifecycle_status = 'deleted' WHERE id = ?").run(cutObj);---- const snapshot = async () => (await Promise.all(['vods', 'clips', 'files', 'pastes', 'media_objects', 'media_locations', 'media_variants', 'media_relationships']-- .map(async t => JSON.stringify(await raw.prepare(`SELECT * FROM ${t} ORDER BY ${t}::text`).all())))).join('|');-- const snap = await snapshot();-- const rep = await drift.buildReport(q, { limit: 5 });-- assert.strictEqual(await snapshot(), snap, 'the report changed nothing');---- assert.strictEqual(rep.total_drift, 5, drift.formatReport(rep));-- assert.deepStrictEqual([rep.kinds.vod.missing, rep.kinds.clip.mismatch, rep.kinds.file.mismatch, rep.kinds.screenshot.unlinked], [1, 2, 1, 1]);-- assert.deepStrictEqual([rep.field_counts.visibility, rep.field_counts.size_bytes, rep.field_counts.owner_user_id, rep.field_counts.lifecycle_status], [1, 1, 1, 1]);-- const ex = (kind, id) => rep.kinds[kind].examples.find(e => String(e.id) === String(id));-- assert.deepStrictEqual(ex('vod', 900).problems, ['missing']);-- assert.deepStrictEqual(ex('clip', upId).diff.visibility, { row: 'public', object: 'private' });-- assert.deepStrictEqual(ex('clip', cutId).diff.lifecycle_status, { row: 'failed', object: 'deleted' });-- assert.deepStrictEqual(Object.keys(ex('file', fileKey).diff).sort(), ['owner_user_id', 'size_bytes']);-- assert.deepStrictEqual(ex('screenshot', (await db.get('SELECT id FROM pastes WHERE slug = ?', [slug])).id).problems, ['unlinked']);-- assert.strictEqual(rep.kinds.vod.skipped, 1, 'the clips-only recording is skipped, not drift');-- assert.strictEqual((await drift.buildReport(q, { appId: 'network' })).total_drift, 0, '--app narrows the report');-- assert.match(drift.formatReport(rep), /total drift: 5 of \d+ row\(s\)/);---- // The script: read-only, exit 0 whatever it finds.-- const { runScript } = require('./helpers/run-script');-- const script = path.join(__dirname, '..', 'scripts', 'object-drift-report.js');-- const r1 = await runScript(script, ['--json', '--limit', '1']);-- assert.strictEqual(r1.code, 0, r1.err);-- const out = JSON.parse(r1.out);-- assert.strictEqual(out.total_drift, 5);-- assert.ok(Object.values(out.kinds).every(k => k.examples.length <= 1), '--limit caps the examples per kind');-- assert.match((await runScript(script, [])).out, /clip examples \(2 of 2\)/);-- assert.strictEqual(await snapshot(), snap, 'the script changed nothing either');---- // Re-projecting the planted rows clears the drift the report listed.-- for (const id of [900]) await model.sync('vod', id);-- await model.sync('clip', upId); await model.sync('clip', cutId); await model.sync('file', fileKey);-- await model.sync('paste', (await db.get('SELECT id FROM pastes WHERE slug = ?', [slug])).id);-- await noDrift('after re-projecting the planted drift');-- console.log('✅ drift report: read-only (report and script), counts per kind with examples, finds missing / unlinked / mismatched objects');--server.close();fs.rmSync(tmp, { recursive: true, force: true });console.log('✅ All object-first write tests passed');@@@ -1,5 +1,5 @@'use strict';--// Canonical object model (roadmap Wave 4): the backfill projects every vod, clip, file, screenshot,++// Canonical object model (roadmap Wave 4): the test helper projects every vod, clip, file, screenshot,// avatar and thumbnail row onto media_objects + media_locations without touching bytes or providers;// it is idempotent; the write hooks keep the model current; holds block deletion and tier moves on// every path; the public object-size invariant is recorded as policy.@@@ -27,7 +27,7 @@ const path = require('path');const contracts = require('openvibe-contracts');const db = require('../server/db/database');const model = require('../server/objects/model');-- const { backfill } = require('../server/objects/backfill');++ const { projectRows } = require('./helpers/project-rows');const invariant = require('../server/objects/invariant');const vodStorage = require('../server/vod/vod-storage');// The revisioned tier policies load once the database is open (server/index.js does this at boot).@@@ -68,16 +68,15 @@ const path = require('path');const mtimes = [vodFile, clipFile, shot, avatar].map(p => fs.statSync(p).mtimeMs);// ── Dry run: a full report, nothing written ──-- const dry = await backfill({ dryRun: true });++ const dry = await projectRows({ dryRun: true });assert.strictEqual(dry.dry_run, true);assert.deepStrictEqual(dry.counts.vod, { seen: 7, created: 6, updated: 0, skipped: 1 });assert.strictEqual((await d.prepare('SELECT COUNT(*) c FROM media_objects').get()).c, 0, 'dry run writes no objects');assert.strictEqual((await d.prepare('SELECT COUNT(*) c FROM vods WHERE object_id IS NOT NULL').get()).c, 0, 'dry run links no rows');-- assert.strictEqual(model.parseJson((await d.prepare("SELECT value FROM media_settings WHERE key = 'objects.backfill.last_report'").get())?.value, null), null, 'dry run stores no report');console.log('✅ dry run reports without writing');// ── Real run ──-- const r = await backfill();++ const r = await projectRows();assert.deepStrictEqual(r.counts.vod, { seen: 7, created: 6, updated: 0, skipped: 1 });assert.deepStrictEqual(r.counts.clip, { seen: 2, created: 2, updated: 0, skipped: 0 });assert.deepStrictEqual(r.counts.file, { seen: 1, created: 1, updated: 0, skipped: 0 });@@@ -86,7 +85,6 @@ const path = require('path');assert.deepStrictEqual(r.counts.thumbnail, { seen: 2, created: 1, updated: 0, skipped: 1 });assert.deepStrictEqual(r.skipped.map(s => s.reason).sort(), ['clips-only recording (ephemeral, never published)', 'external thumbnail url', 'screenshot paste without a file path']);assert.deepStrictEqual(r.errors, []);-- assert.ok(await d.prepare("SELECT value FROM media_settings WHERE key = 'objects.backfill.last_report'").get(), 'report recorded');assert.deepStrictEqual([vodFile, clipFile, shot, avatar].map(p => fs.statSync(p).mtimeMs), mtimes, 'no bytes touched');const v1 = await objOf('vods', 1), v2 = await objOf('vods', 2), v3 = await objOf('vods', 3), v4 = await objOf('vods', 4), v6 = await objOf('vods', 6), v7 = await objOf('vods', 7);@@@ -130,18 +128,18 @@ const path = require('path');const th = await model.getObject(tv.derived_object_id);assert.deepStrictEqual([th.kind, th.legacy_ref, th.visibility, (await locs(th)).local.state], ['thumbnail', 'legacy:live:thumbnail:vod-1-111.jpg', 'public', 'present']);assert.ok(await d.prepare("SELECT 1 FROM media_relationships WHERE from_object_id = ? AND relation = 'thumbnail_of' AND to_object_id = ?").get(th.id, v1.id));-- console.log('✅ backfill: one object per row, locations mirror where the bytes are, relationships + thumbnail variants');++ console.log('✅ projection: one object per row, locations mirror where the bytes are, relationships + thumbnail variants');// ── Idempotent ──const countAll = async () => (await Promise.all(['media_objects', 'media_locations', 'media_relationships', 'media_variants'].map(async t => (await d.prepare(`SELECT COUNT(*) c FROM ${t}`).get()).c)));const before = await countAll();-- const again = await backfill();++ const again = await projectRows();assert.strictEqual(again.totals.created, 0, 'a re-run creates nothing');assert.strictEqual(again.counts.vod.updated, 6);assert.deepStrictEqual(await countAll(), before);assert.strictEqual((await objOf('vods', 1)).id, v1.id, 'ids are stable');-- assert.strictEqual((await backfill({ onlyMissing: true })).totals.created, 0, 'only-missing finds nothing new');-- console.log('✅ backfill is idempotent');++ assert.strictEqual((await projectRows({ onlyMissing: true })).totals.created, 0, 'only-missing finds nothing new');++ console.log('✅ projection is idempotent');// ── Remote knowledge survives a re-projection ──await model.setLocationState((await locs(v2)).b2.id, { state: 'present', size_bytes: 5 * MB });@@@ -243,6 +241,6 @@ const path = require('path');console.log('✅ native soft delete keeps bytes for the retention period; purge skips held objects');fs.rmSync(tmp, { recursive: true, force: true });-- console.log('objects model + backfill: all checks passed');++ console.log('objects model + projection: all checks passed');})().catch((err) => { console.error(err); process.exit(1); });})().catch((err) => { console.error(err); process.exit(1); });@@@ -2,7 +2,7 @@// Reconciliation of the object model against the bytes, with a fake B2/R2 provider: read-only by// default, --verify records what the HEADs found; it detects missing canonical copies, lost local// files, size/hash mismatches, orphan locations, deleted-but-still-served objects and unprojected--// rows. Plus the three scripts end to end (backfill --dry-run, reconcile, invariant).++// rows. Plus test-only projection and the reconcile and invariant scripts end to end.const assert = require('assert');const fs = require('fs');const os = require('os');@@@ -40,19 +40,19 @@ const crypto = require('crypto');write(path.join(env.FILES_PATH, 'live', 'k1-f.bin'), fileBytes);await d.prepare("INSERT INTO files (key, app_id, size, mime, sha256) VALUES ('k1-f.bin', 'live', ?, 'application/octet-stream', ?)").run(fileBytes.length, crypto.createHash('sha256').update(fileBytes).digest('hex'));-- // The backfill (server/objects/backfill.js; the service runs it at boot): a dry run reports and writes nothing.++ // Test-only legacy row projection: a dry run reports and writes nothing.const { runScript } = require('./helpers/run-script');const run = async (script, args = []) => await runScript(path.join(__dirname, '..', 'scripts', script), args, { env });-- const bf = require('../server/objects/backfill');-- const dry = await bf.backfill({ dryRun: true });++ const bf = require('./helpers/project-rows');++ const dry = await bf.projectRows({ dryRun: true });assert.deepStrictEqual([dry.dry_run, dry.counts.vod.created, dry.counts.file.created], [true, 5, 1]);assert.strictEqual((await d.prepare('SELECT COUNT(*) c FROM media_objects').get()).c, 0, 'a dry run wrote nothing');-- const real = await bf.backfill();++ const real = await bf.projectRows();assert.match(bf.summarize(real), /vod 5\+\/0~\/0 skipped/);assert.strictEqual((await d.prepare('SELECT COUNT(*) c FROM media_objects').get()).c, 6);-- console.log('✅ backfill: a dry run reports only; a real run projects');++ console.log('✅ projection: a dry run reports only; a real run projects');-- fs.unlinkSync(path.join(env.VOD_PATH, 'e.webm')); // vod 5 loses its file after backfill++ fs.unlinkSync(path.join(env.VOD_PATH, 'e.webm')); // vod 5 loses its file after projectionconst objId = async (vodId) => (await d.prepare('SELECT object_id FROM vods WHERE id = ?').get(vodId)).object_id;// Anomalies the reconciler must notice:await d.prepare("INSERT INTO media_locations (object_id, provider, key, state) VALUES ('med_01JAB2C3D4E5F6G7H8J9K0MNPQ', 'local', '/nowhere', 'present') RETURNING id").run(); // orphan location@@@ -132,6 +132,8 @@ const crypto = require('crypto');const { finalizeVod } = require('../server/vod/finalize');await raw.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, is_recording) OVERRIDING SYSTEM VALUEVALUES (20, 'live', 5, 'Ghost', ?, 1, 1) RETURNING id`).run(path.join(process.env.VOD_PATH, 'never-written.webm'));++ // A recording row has its object from the start (createVod writes both); this fixture row is projected the same way.++ await require('../server/objects/model').sync('vod', 20);failOutboxInserts(true);await finalizeVod(20);failOutboxInserts(false);
@@ -103,7 +103,7 @@ server/
@@ -103,7 +103,7 @@ server/
me/ the object explorer: /me and /api/v2/me (a signed-in person's own objects, uploads,
me/ the object explorer: /me and /api/v2/me (a signed-in person's own objects, uploads,
derivatives and usage) and the operator views /me/ops — see docs/object-model.md#object-explorer
derivatives and usage) and the operator views /me/ops — see docs/object-model.md#object-explorer
objects/ canonical object model: model (projections, holds), routes (/api/v2 + /o),
objects/ canonical object model: model (projections, holds), routes (/api/v2 + /o),
- backfill, reconcile, invariant, signing (presigned URLs), multipart,
+ reconcile, invariant, signing (presigned URLs), multipart,
content-type, verify-job (scheduled copy verification), copy-report,
content-type, verify-job (scheduled copy verification), copy-report,
readiness (metadata / bytes_verified / playable), popularity (unique viewers
readiness (metadata / bytes_verified / playable), popularity (unique viewers
per day, no identifiers), tier-policy + tiering (native objects in the R2 cache) —
per day, no identifiers), tier-policy + tiering (native objects in the R2 cache) —
@@ -127,7 +127,7 @@ server/
@@ -127,7 +127,7 @@ server/
client-ip.js trust proxy = loopback; req.ip is the only client address
client-ip.js trust proxy = loopback; req.ip is the only client address
(openvibe-shared v3.0.0, openvibe-contracts v0.127.0, openvibe-sdk v0.38.0: pinned release tarballs, installed by npm)
(openvibe-shared v3.0.0, openvibe-contracts v0.127.0, openvibe-sdk v0.38.0: pinned release tarballs, installed by npm)
scripts/smoke-test.sh end-to-end smoke test (boots a temp instance)
scripts/smoke-test.sh end-to-end smoke test (boots a temp instance)
-scripts/reconcile-objects.js / object-invariant.js / object-drift-report.js / no-good-copy-report.js object-model operator tools
+scripts/reconcile-objects.js / object-invariant.js / no-good-copy-report.js object-model operator tools
scripts/media-jobs.js list jobs, run the size-invariant scan (dry run by default), approve/cancel proposals
scripts/media-jobs.js list jobs, run the size-invariant scan (dry run by default), approve/cancel proposals
scripts/r2-eviction-drill.js evict one VOD's R2 copy, prove B2 serves it, re-warm R2; JSON artifact (dry run by default)
scripts/r2-eviction-drill.js evict one VOD's R2 copy, prove B2 serves it, re-warm R2; JSON artifact (dry run by default)
scripts/vod-duration-reconcile.js stored VOD durations vs the real files (dry run by default; --apply --backup; --rollback)
scripts/vod-duration-reconcile.js stored VOD durations vs the real files (dry run by default; --apply --backup; --rollback)
@@ -148,7 +148,7 @@ other OpenVibe services call the API.
@@ -148,7 +148,7 @@ other OpenVibe services call the API.
subject, in every tenant: uploads in progress, copies, derivatives, visibility, lifecycle and their own
subject, in every tenant: uploads in progress, copies, derivatives, visibility, lifecycle and their own
usage per tenant and namespace (`GET /api/v2/me/objects`, `/objects/:id`, `/usage`). Read-only: changes
usage per tenant and namespace (`GET /api/v2/me/objects`, `/objects/:id`, `/usage`). Read-only: changes
stay with the apps. Network staff with `staff.site.view` get the operator views at `/me/ops` (failed
stay with the apps. Network staff with `staff.site.view` get the operator views at `/me/ops` (failed
-jobs, missing media, backfill, tiering, outbound webhooks — who is configured and what this process
+jobs, missing media, projection, tiering, outbound webhooks — who is configured and what this process
has sent — and usage recompute with `staff.site.configure`). Details in
has sent — and usage recompute with `staff.site.configure`). Details in
[docs/object-model.md](docs/object-model.md#object-explorer).
[docs/object-model.md](docs/object-model.md#object-explorer).
@@ -356,7 +356,7 @@ directories are shared across apps) and responses carry a `note` saying so.
@@ -356,7 +356,7 @@ directories are shared across apps) and responses carry a `note` saying so.
| POST | `/admin/storage/holds` | `{ object_id \| vod_id \| clip_id, reason, kind?, note?, placed_by? }` → 201: the object is kept (no delete, no tier move) and so are the clips cut from a held VOD. Not for calls acting for a user (403) |
| POST | `/admin/storage/holds` | `{ object_id \| vod_id \| clip_id, reason, kind?, note?, placed_by? }` → 201: the object is kept (no delete, no tier move) and so are the clips cut from a held VOD. Not for calls acting for a user (403) |
| POST | `/admin/storage/holds/:holdId/release` | `{ released_by }` (also `DELETE /admin/storage/holds/:holdId`); 409 when already released. Holds stay on record |
| POST | `/admin/storage/holds/:holdId/release` | `{ released_by }` (also `DELETE /admin/storage/holds/:holdId`); 409 when already released. Holds stay on record |
| GET | `/admin/storage/buckets` | sanitized bucket status per provider: `{ configured, endpoint, bucket, region, healthy, reachable }` via a live HeadBucket probe — **credentials are never returned** |
| GET | `/admin/storage/buckets` | sanitized bucket status per provider: `{ configured, endpoint, bucket, region, healthy, reachable }` via a live HeadBucket probe — **credentials are never returned** |
-| GET | `/admin/storage/ops` | this app's operator report (as `/me/ops` shows it to Network staff): failed jobs, missing media, backfill, tiering, the namespaces' usage snapshot. Not for calls acting for a user (403) |
+| GET | `/admin/storage/ops` | this app's operator report (as `/me/ops` shows it to Network staff): failed jobs, missing media, projection, tiering, the namespaces' usage snapshot. Not for calls acting for a user (403) |
| POST | `/admin/storage/ops/recompute` | refresh this app's namespaces' usage snapshot from the rows |
| POST | `/admin/storage/ops/recompute` | refresh this app's namespaces' usage snapshot from the rows |
### Webhooks (outbound)
### Webhooks (outbound)
@@ -427,7 +427,7 @@ mode (`server/drill.js`) Media:
@@ -427,7 +427,7 @@ mode (`server/drill.js`) Media:
4100 (before the database is opened), and opens the copy without migrating it;
4100 (before the database is opened), and opens the copy without migrating it;
- starts only its HTTP server: no app seeding, JWKS refresh, tiering sweep, health job, junk sweep,
- starts only its HTTP server: no app seeding, JWKS refresh, tiering sweep, health job, junk sweep,
clip re-cuts, copy verification, jobs worker, disk guardian, thumbnail cleanup, object purge,
clip re-cuts, copy verification, jobs worker, disk guardian, thumbnail cleanup, object purge,
- backfill, orphan-recording finalize or Events relay; webhooks are never sent;
+ orphan-recording finalize or Events relay; webhooks are never sent;
- writes no file and creates no directory (the storage checks leave `/api/ready`, as do the remote
- writes no file and creates no directory (the storage checks leave `/api/ready`, as do the remote
tiers; it reports `"mode": "drill"`);
tiers; it reports `"mode": "drill"`);
- never opens a file path from the database: every byte route (`/v` and `/c` bytes, `/t`, `/a`, `/f`,
- never opens a file path from the database: every byte route (`/v` and `/c` bytes, `/t`, `/a`, `/f`,
@@ -578,16 +578,7 @@ Every stored blob is a **media object** (`med_<ULID>`, `media_objects`) with one
@@ -578,16 +578,7 @@ Every stored blob is a **media object** (`med_<ULID>`, `media_objects`) with one
existing route and response is unchanged, and the old write paths keep the
existing route and response is unchanged, and the old write paths keep the
model current. Full reference: **[docs/object-model.md](docs/object-model.md)**.
model current. Full reference: **[docs/object-model.md](docs/object-model.md)**.
-- **Backfill** — the row→object projection (`server/objects/backfill.js`), not a
- CLI script: one object per vod, clip, file, screenshot, avatar and thumbnail;
- idempotent; never moves bytes or calls B2/R2 (remote copies stay `pending`). The
- service runs the only-missing form 15 s after boot, for rows with no `object_id`
- yet, and stores its report in `media_settings.objects.backfill.last_report`
- (also surfaced by `/me/ops`). To see whether any row still lacks an object — or
- disagrees with the object on record — run the read-only drift report:
- `node scripts/object-drift-report.js [--app live] [--limit 20] [--json]
- [--out report.json]`. Zero drift across a release is the signal to drop the boot
- backfill ([details](docs/object-model.md#backfill)).
+- **Object projection** — C-75 was retired on 2026-10-10 after production drift was 0 of 3036 rows. Every write makes its media object in the same transaction (`server/objects/model.js` `withObject`). Operators retain `scripts/reconcile-objects.js` and `scripts/object-invariant.js` as checks ([details](docs/object-model.md#keeping-the-model-current-object-first-writes)).
- **API** — `/api/v2/:app/objects`: init → `PUT /:id/content` (sha256, size,
- **API** — `/api/v2/:app/objects`: init → `PUT /:id/content` (sha256, size,
quota) → `/:id/complete`; `GET /:id`, cursor `GET /`, soft `DELETE /:id`
quota) → `/:id/complete`; `GET /:id`, cursor `GET /`, soft `DELETE /:id`
(bytes kept `MEDIA_DELETE_RETENTION_DAYS`), `/:id/restore`, `/:id/download`
(bytes kept `MEDIA_DELETE_RETENTION_DAYS`), `/:id/restore`, `/:id/download`
@@ -9,7 +9,7 @@ its v1 pastes API was retired in T10 step 2 (`/api/v1/:app/pastes` is unmounted;
@@ -9,7 +9,7 @@ its v1 pastes API was retired in T10 step 2 (`/api/v1/:app/pastes` is unmounted;
`/p/:slug` routes remain). Other existing URLs and v1 responses are
`/p/:slug` routes remain). Other existing URLs and v1 responses are
unchanged. New code talks to objects through the [v2 API](#object-api-v2).
unchanged. New code talks to objects through the [v2 API](#object-api-v2).
-Code: `server/objects/` (`model.js`, `routes.js`, `backfill.js`, `reconcile.js`, `invariant.js`,
+Code: `server/objects/` (`model.js`, `routes.js`, `reconcile.js`, `invariant.js`,
`signing.js`, `verify-job.js`, `copy-report.js`, `multipart.js`, `content-type.js`, `readiness.js`, `namespaces.js`,
`signing.js`, `verify-job.js`, `copy-report.js`, `multipart.js`, `content-type.js`, `readiness.js`, `namespaces.js`,
`namespace-routes.js`, `popularity.js`, `tier-policy.js`, `tiering.js`) and the job system in
`namespace-routes.js`, `popularity.js`, `tier-policy.js`, `tiering.js`) and the job system in
`server/jobs/` (`queue.js`, `worker.js`, `routes.js`, `types.js`, `thumbnail.js`, `invariant-scan.js`,
`server/jobs/` (`queue.js`, `worker.js`, `routes.js`, `types.js`, `thumbnail.js`, `invariant-scan.js`,
@@ -44,7 +44,7 @@ state resets only when the key changes.
@@ -44,7 +44,7 @@ state resets only when the key changes.
### Ids and legacy refs
### Ids and legacy refs
-Ids come from `openvibe-contracts` `ids.newId('media')`. A backfilled object takes its ULID time
+Ids come from `openvibe-contracts` `ids.newId('media')`. A projected legacy object takes its ULID time
from the original row's `created_at`, so ids sort by creation time. Projected objects also carry
from the original row's `created_at`, so ids sort by creation time. Projected objects also carry
the transitional `media.media-ref@1` legacy form in `legacy_ref`, which is unique:
the transitional `media.media-ref@1` legacy form in `legacy_ref`, which is unique:
@@ -66,8 +66,7 @@ Every v2 route that takes `:id` accepts either form. Native v2 objects have `leg
@@ -66,8 +66,7 @@ Every v2 route that takes `:id` accepts either form. Native v2 objects have `leg
### How a row maps to its object
### How a row maps to its object
`model.sync(kind, id)` derives the object and its locations from the row as it is at that moment
`model.sync(kind, id)` derives the object and its locations from the row as it is at that moment
-(the per-kind `<kind>Projection(row)` functions). The backfill, every write and the drift report use
-the same functions.
+(the per-kind `<kind>Projection(row)` functions). Every write uses the same functions.
- **vod / clip.**
- **vod / clip.**
- Local copy: the file under `VOD_PATH` (vods, resolved by basename as playback does) or the clip's own path.
- Local copy: the file under `VOD_PATH` (vods, resolved by basename as playback does) or the clip's own path.
@@ -81,29 +80,12 @@ the same functions.
@@ -81,29 +80,12 @@ the same functions.
- **screenshot / avatar.** `screenshot_path`. Text pastes have no bytes and no object; paste text belongs to Community.
- **screenshot / avatar.** `screenshot_path`. Text pastes have no bytes and no object; paste text belongs to Community.
- **thumbnail.** One object per vod/clip, reached through the parent's `thumbnail` variant and **updated in place** when the picture is regenerated (live recordings refresh it every couple of minutes). Visibility is `public` when the parent is public and `unlisted` otherwise, because `/t/<name>` serves anyone who has the name. An external `thumbnail_url` is skipped.
- **thumbnail.** One object per vod/clip, reached through the parent's `thumbnail` variant and **updated in place** when the picture is regenerated (live recordings refresh it every couple of minutes). Visibility is `public` when the parent is public and `unlisted` otherwise, because `/t/<name>` serves anyone who has the name. An external `thumbnail_url` is skipped.
-## Backfill
-
-`backfill()` in `server/objects/backfill.js` is a module, not a CLI: it is called from code, and the service calls it
-itself with `{ onlyMissing: true }` 15 s after boot. Read-only questions about it are answered by the
-[drift report](#drift-report). It creates one object per vod, clip, file, screenshot/avatar paste
-and thumbnail, with relationships and thumbnail variants.
-
-- **Idempotent.** Objects are keyed by `legacy_ref`, so a second run updates them in place and creates nothing.
-- **Never moves or deletes bytes, and never calls B2/R2.** A local copy is `present` when the file exists and `missing` when it does not. Remote copies stay `pending` until `reconcile-objects.js --verify` checks them.
-- **One transaction,** with one savepoint per row: a failing row is reported and leaves nothing half-written. A dry run (`{ dryRun: true }`) rolls everything back and only reports.
-- **Report:** counts per kind (seen, created, updated, skipped), location states, skipped rows with a reason, and errors. A real run stores the report in `media_settings` under `objects.backfill.last_report`; `lastReport()` reads it back and `/me/ops` shows it.
-- **At boot:** the service runs the `onlyMissing` form itself 15 s after start, for rows with no `object_id`. The first boot after the upgrade therefore projects everything. With object-first writes it should find nothing; it stays until the [drift report](#drift-report) shows zero drift across a release.
-
-Skip reasons today: `clips-only recording (ephemeral, never published)`, `screenshot paste without a file path`, `external thumbnail url`.
-
-The backfill's own transaction holds row and table locks for its duration, typically seconds, so it runs only
-where the service controls it: the boot backfill.
+The historical projection of imported rows is preserved for tests in `test/helpers/project-rows.js`. It is not a production path.
### Keeping the model current: object-first writes
### Keeping the model current: object-first writes
Every write to a projected row writes its object in the same PostgreSQL transaction
Every write to a projected row writes its object in the same PostgreSQL transaction
-(`model.withObject(kind, ids, write)`; roadmap WS-G task 1, which retires compatibility shim C-75,
-"write the row, then sync the object"). The row change and the re-projection commit together or
+(`model.withObject(kind, ids, write)`; roadmap WS-G task 1; compatibility shim C-75 was retired on 2026-10-10). The row change and the re-projection commit together or
not at all, so a crash or an error between them can no longer leave the object behind its row. A
not at all, so a crash or an error between them can no longer leave the object behind its row. A
write whose object cannot be written fails as a whole and changes nothing. Inside
write whose object cannot be written fails as a whole and changes nothing. Inside
`webhooks.announce()` the object joins the outcome's transaction, and its object events are queued
`webhooks.announce()` the object joins the outcome's transaction, and its object events are queued
@@ -134,28 +116,13 @@ content hashes come from the content-hash job, and remote copies stay `pending`
@@ -134,28 +116,13 @@ content hashes come from the content-hash job, and remote copies stay `pending`
reconciliation verifies them. Two writes record something that already happened outside the
reconciliation verifies them. Two writes record something that already happened outside the
database, so they never lose the row to its object: the recorder's start (ffmpeg is running) and the
database, so they never lose the row to its object: the recorder's start (ffmpeg is running) and the
tier moves (the bytes have moved). If their combined transaction fails, the row is written alone and
tier moves (the bytes have moved). If their combined transaction fails, the row is written alone and
-re-projected after it (`withObjectOrRow`), with a warning. After a move that removed the local file,
-and after a finalize that threw part-way, a follow-up re-projection re-reads the disk. Columns no
+re-projected after it (`withObjectOrRow`), with a warning. After a move that removed the local file, a follow-up re-projection re-reads the disk. Columns no
object carries (view counts, `master_file_path`, `cut_attempts`, paste likes and AI fields) are
object carries (view counts, `master_file_path`, `cut_attempts`, paste likes and AI fields) are
-written without one. `db.importLegacyRows` (the cutover import) is not converted: the boot backfill
-projects what it inserts.
-
-### Drift report
-
-```
-node scripts/object-drift-report.js [--app live] [--limit 20] [--json] [--out report.json]
-```
+written without one. `db.importLegacyRows` is the historical cutover import.
-For every projected row, and each vod/clip thumbnail, the report derives what the object must say
-with the same projection functions and compares it with the object on record: `kind`, `visibility`,
-`lifecycle_status`, `size_bytes`, `legacy_ref`, `app_id`, `owner_app`, `owner_user_id`, and the link
-(the row's `object_id`, or the parent's `thumbnail` variant). A row is `missing` (no object),
-`unlinked` (the link names another object) or `mismatch` (a field differs). It prints counts per
-kind and up to `--limit` examples of each. The database is opened read-only and nothing is written;
-the exit code is always 0.
+### C-75 retirement
-The boot backfill (rows with no object) and the finalize follow-up stay until a release has run with
-zero drift. They are removed in a later, dated step.
+C-75 was retired on 2026-10-10 after production drift was 0 of 3036 rows (vod 736, clip 363, file 29, screenshot 819, avatar 3, thumbnail 1086). Every write makes its media object in the same transaction through `model.withObject`. Operators can still check objects with `scripts/reconcile-objects.js` and `scripts/object-invariant.js`.
## Owner subjects
## Owner subjects
@@ -447,7 +414,7 @@ cheap and renders in a drill.
@@ -447,7 +414,7 @@ cheap and renders in a drill.
- `jobs`: counts by status; failed jobs of the last 7 days by type and error code; the most recent failures with their error text.
- `jobs`: counts by status; failed jobs of the last 7 days by type and error code; the most recent failures with their error text.
- `missing`: ready objects with no good copy; copies recorded missing or corrupt; copies by provider and state; quarantined VODs; the last verification run.
- `missing`: ready objects with no good copy; copies recorded missing or corrupt; copies by provider and state; quarantined VODs; the last verification run.
-- `backfill`: the last object backfill that changed something; rows with no object yet; objects whose owner subject is unresolved; whether upload reservations were seeded.
+- `projection`: rows with no object yet; objects whose owner subject is unresolved; whether upload reservations were seeded.
- `tiering`: providers and policy; VODs and clips per provider; objects per canonical copy (native or projected); VODs eligible to offload; R2 decisions of the last 24 h and recent refusals and failures; the sweep's state in this process; and `native_objects`: the [object tiering](#tiering-of-native-objects)'s activation gate and policy, native objects in R2, how many are eligible to promote now, its decisions of the last 24 h (dry runs included) and the most recent ones, and its last sweep.
- `tiering`: providers and policy; VODs and clips per provider; objects per canonical copy (native or projected); VODs eligible to offload; R2 decisions of the last 24 h and recent refusals and failures; the sweep's state in this process; and `native_objects`: the [object tiering](#tiering-of-native-objects)'s activation gate and policy, native objects in R2, how many are eligible to promote now, its decisions of the last 24 h (dry runs included) and the most recent ones, and its last sweep.
- `namespaces`: every namespace with quota and usage snapshot.
- `namespaces`: every namespace with quota and usage snapshot.
- `webhooks`: the apps with a configured outbound webhook (app id and host only — never the URL or secret) and, per app, what this process has sent since it loaded (`sent`, `failed`, the last event and attempt/ok times, the last error). In-memory: counts reset on restart and scope to the app when the report is scoped.
- `webhooks`: the apps with a configured outbound webhook (app id and host only — never the URL or secret) and, per app, what this process has sent since it loaded (`sent`, `failed`, the last event and attempt/ok times, the last error). In-memory: counts reset on restart and scope to the app when the report is scoped.
@@ -553,7 +520,7 @@ The exit code is 1 when issues were found.
@@ -553,7 +520,7 @@ The exit code is 1 when issues were found.
| `size_mismatch` / `hash_mismatch` | the bytes differ from the object's size or sha256 |
| `size_mismatch` / `hash_mismatch` | the bytes differ from the object's size or sha256 |
| `orphan_location` | a location row whose object does not exist |
| `orphan_location` | a location row whose object does not exist |
| `deleted_publicly_reachable` | a deleted object whose inherited row still serves it, or a thumbnail of a deleted object that is still served by name |
| `deleted_publicly_reachable` | a deleted object whose inherited row still serves it, or a thumbnail of a deleted object that is still served by name |
-| `missing_projection` | an inherited row with no object yet (run the backfill) |
+| `missing_projection` | an inherited row with no object yet (investigate the write path) |
| `incomplete_multipart` | a multipart session still open past its expiry (the hourly purge should have removed it) |
| `incomplete_multipart` | a multipart session still open past its expiry (the hourly purge should have removed it) |
Bucket keys and local files that no row names are the [storage orphan report](#storage-orphan-report)'s.
Bucket keys and local files that no row names are the [storage orphan report](#storage-orphan-report)'s.
deleted file mode 100644
deleted file mode 100644
@@ -1,44 +0,0 @@
@@ -1,44 +0,0 @@
-#!/usr/bin/env node
-'use strict';
-/**
- * Object drift report (WS-G task 1, retiring C-75; server/objects/drift.js has the rules): every
- * projected vods/clips/files/pastes row, and each vod/clip thumbnail, whose media_object is missing,
- * not linked, or disagrees with the row on kind, visibility, lifecycle status, size, legacy ref or
- * owner/app. Prints counts per kind and up to --limit examples of each. READ-ONLY: the database is
- * opened read-only and only local files are stat'ed (paths from the service's environment:
- * VOD_PATH, CLIPS_PATH, FILES_PATH, THUMBNAILS_PATH, as the projection uses them).
- *
- * node scripts/object-drift-report.js [--app live] [--limit 20] [--json] [--out report.json]
- *
- * Exit code 0 always: it is a report (an error is printed to stderr). Zero drift across a release
- * is the signal to remove the boot backfill (server/index.js) and the finalize follow-up.
- */
-const fs = require('fs');
-const path = require('path');
-
-(async () => {
- await require('../server/db/database').initDb(); // PostgreSQL (DATABASE_URL), as the service
- const args = process.argv.slice(2);
- const has = (name) => args.includes(`--${name}`);
- const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] && !args[i + 1].startsWith('--') ? args[i + 1] : dflt; };
-
- try {
- // Read-only: the report runs in a READ ONLY transaction (PostgreSQL refuses any write inside it).
- const { buildReport, formatReport } = require('../server/objects/drift');
- const limit = Math.max(0, parseInt(arg('limit', '20'), 10) || 0);
- const d = require('../server/db/database').getDb();
- const q = {
- all: async (sql, params = []) => await d.prepare(sql).all(...params),
- get: async (sql, params = []) => await d.prepare(sql).get(...params),
- };
- const report = await d.tx(async (t) => {
- await t.query('SET TRANSACTION READ ONLY');
- return await buildReport(q, { appId: arg('app', null), limit });
- });
- if (arg('out')) fs.writeFileSync(arg('out'), JSON.stringify(report, null, 2));
- console.log(has('json') ? JSON.stringify(report, null, 2) : formatReport(report));
- } catch (err) {
- console.error(`object-drift-report: ${err.message}`);
- }
- process.exit(0);
-})().catch((err) => { console.error(err); process.exit(1); });
@@ -35,7 +35,7 @@
@@ -35,7 +35,7 @@
* and each change is logged as an [Admin] line
* and each change is logged as an [Admin] line
* POST /holds/:holdId/release { released_by } (also DELETE /holds/:holdId); 409 when already released
* POST /holds/:holdId/release { released_by } (also DELETE /holds/:holdId); 409 when already released
* GET /ops this app's operator report (server/me/ops.js, as openvibe.media/me/ops shows
* GET /ops this app's operator report (server/me/ops.js, as openvibe.media/me/ops shows
- * it to Network staff): failed jobs, missing media, backfill, tiering, namespaces
+ * it to Network staff): failed jobs, missing media, projection, tiering, namespaces
* POST /ops/recompute refresh this app's namespaces' usage snapshot from the rows
* POST /ops/recompute refresh this app's namespaces' usage snapshot from the rows
*/
*/
'use strict';
'use strict';
@@ -665,7 +665,7 @@ function _opsStaffOnly(req, res) {
@@ -665,7 +665,7 @@ function _opsStaffOnly(req, res) {
return false;
return false;
}
}
-// GET /ops?limit — failed jobs, missing media, backfill status, tiering diagnostics and the namespaces'
+// GET /ops?limit — failed jobs, missing media, projection status, tiering diagnostics and the namespaces'
// usage snapshot of this app (server/me/ops.js; the sweep and the provider switches are service-wide).
// usage snapshot of this app (server/me/ops.js; the sweep and the provider switches are service-wide).
router.get('/ops', async (req, res) => {
router.get('/ops', async (req, res) => {
try {
try {
@@ -75,7 +75,7 @@ function heldSql(ref) {
@@ -75,7 +75,7 @@ function heldSql(ref) {
}
}
/**
/**
- * Object-first write (WS-G task 1, retiring C-75): run write() — a change to a projected
+ * Object-first write (WS-G task 1, C-75 retired 2026-10-10): run write() — a change to a projected
* vods/clips/files/pastes row — and re-project the row's media_object in the same transaction
* vods/clips/files/pastes row — and re-project the row's media_object in the same transaction
* (objects/model.js withObject). Throws, with nothing written, when either part fails.
* (objects/model.js withObject). Throws, with nothing written, when either part fails.
*/
*/
@@ -21,7 +21,7 @@
@@ -21,7 +21,7 @@
* - server/index.js opens the database and serves HTTP, and starts nothing else: no app seeding, no
* - server/index.js opens the database and serves HTTP, and starts nothing else: no app seeding, no
* JWKS refresh, no health job, junk sweep, tiering sweep, clip re-cuts, copy verification, jobs
* JWKS refresh, no health job, junk sweep, tiering sweep, clip re-cuts, copy verification, jobs
* worker (thumbnails, split/remux, invariant scans, storage orphan scans), disk guardian, thumbnail cleanup, object purge,
* worker (thumbnails, split/remux, invariant scans, storage orphan scans), disk guardian, thumbnail cleanup, object purge,
- * backfill, orphan-recording finalize or Events relay; webhooks are never sent. Timers that modules
+ * orphan-recording finalize or Events relay; webhooks are never sent. Timers that modules
* start when they are loaded check `enabled` as well. Readiness does not probe storage (a drill
* start when they are loaded check `enabled` as well. Readiness does not probe storage (a drill
* writes none) or the remote tiers.
* writes none) or the remote tiers.
*
*
@@ -303,18 +303,8 @@ const ready = (async () => {
@@ -303,18 +303,8 @@ const ready = (async () => {
await ns.reconcileAll();
await ns.reconcileAll();
} catch (err) { console.warn('[Namespaces] sweep:', err.message); }
} catch (err) { console.warn('[Namespaces] sweep:', err.message); }
});
});
- // Project rows that have no media_object yet (first boot after the upgrade: all of them).
- // C-75 catch-up: every write now makes its object in the same transaction (objects/model.js
- // withObject, WS-G task 1), so this should find nothing. It stays until a release has run with
- // `node scripts/object-drift-report.js` showing zero drift; then it goes, with the finalize
- // follow-up re-projection (vod/finalize.js), in a dated later step.
+ // Take the first namespaces usage snapshot after boot (then hourly and after each write).
setTimeout(async () => {
setTimeout(async () => {
- try {
- const bf = require('./objects/backfill');
- const r = await bf.backfill({ onlyMissing: true });
- if (r.totals.created || r.totals.updated || r.errors.length) console.log(`[Objects] Backfill: ${bf.summarize(r)}`);
- } catch (err) { console.warn('[Objects] Backfill failed:', err.message); }
- // The namespaces' usage snapshot, first taken here (then hourly, and after each write).
try { await require('./objects/namespaces').reconcileAll(); } catch (err) { console.warn('[Namespaces] reconcile:', err.message); }
try { await require('./objects/namespaces').reconcileAll(); } catch (err) { console.warn('[Namespaces] reconcile:', err.message); }
}, 15 * 1000).unref?.();
}, 15 * 1000).unref?.();
@@ -2,7 +2,7 @@
@@ -2,7 +2,7 @@
* OpenVibe.Media — operator views over the object platform (roadmap WS-G task 12;
* OpenVibe.Media — operator views over the object platform (roadmap WS-G task 12;
* docs/object-model.md#operator-views).
* docs/object-model.md#operator-views).
*
*
- * report({ appId, limit }) failed jobs, missing media, backfill status, tiering diagnostics, the
+ * report({ appId, limit }) failed jobs, missing media, projection status, tiering diagnostics, the
* namespaces' usage snapshot and the outbound-webhooks section; appId
* namespaces' usage snapshot and the outbound-webhooks section; appId
* narrows everything that has a tenant (the tiering sweep and the provider
* narrows everything that has a tenant (the tiering sweep and the provider
* switches are service-wide)
* switches are service-wide)
@@ -98,15 +98,14 @@ async function missing(appId, limit) {
@@ -98,15 +98,14 @@ async function missing(appId, limit) {
locations: { by_provider_state: byState, missing_or_corrupt: bad },
locations: { by_provider_state: byState, missing_or_corrupt: bad },
vods_quarantined: quarantined,
vods_quarantined: quarantined,
verification,
verification,
- note: 'From what the database records (scheduled verification, tier moves, reconciliation). A full walk of storage is the storage.orphans.scan report; drift between rows and objects is scripts/object-drift-report.js.',
+ note: 'From what the database records (scheduled verification, tier moves, reconciliation). A full walk of storage is the storage.orphans.scan report.',
};
};
}
}
-// ── Backfill ─────────────────────────────────────────────────
+// ── Projection ───────────────────────────────────────────────
-async function backfill(appId) {
+async function projection(appId) {
const s = scoped(appId);
const s = scoped(appId);
- const last = await require('../objects/backfill').lastReport();
const unprojected = {
const unprojected = {
vods: (await db.get(`SELECT COUNT(*) AS n FROM vods WHERE object_id IS NULL AND COALESCE(clips_only, 0) = 0${s.sql}`, s.params)).n,
vods: (await db.get(`SELECT COUNT(*) AS n FROM vods WHERE object_id IS NULL AND COALESCE(clips_only, 0) = 0${s.sql}`, s.params)).n,
clips: (await db.get(`SELECT COUNT(*) AS n FROM clips WHERE object_id IS NULL${s.sql}`, s.params)).n,
clips: (await db.get(`SELECT COUNT(*) AS n FROM clips WHERE object_id IS NULL${s.sql}`, s.params)).n,
@@ -116,16 +115,10 @@ async function backfill(appId) {
@@ -116,16 +115,10 @@ async function backfill(appId) {
const owner = await db.all(`SELECT app_id, COUNT(*) AS n FROM media_objects WHERE owner_subject IS NULL AND owner_user_id IS NOT NULL
const owner = await db.all(`SELECT app_id, COUNT(*) AS n FROM media_objects WHERE owner_subject IS NULL AND owner_user_id IS NOT NULL
AND lifecycle_status != 'deleted'${s.sql} GROUP BY app_id ORDER BY n DESC`, s.params);
AND lifecycle_status != 'deleted'${s.sql} GROUP BY app_id ORDER BY n DESC`, s.params);
return {
return {
- objects: last ? {
- finished_at: iso(last.finished_at) || null, dry_run: !!last.dry_run, only_missing: !!last.only_missing,
- totals: last.totals || null, locations: last.locations || null,
- errors: Array.isArray(last.errors) ? last.errors.length : 0,
- skipped: Array.isArray(last.skipped) ? last.skipped.length : 0,
- } : null,
unprojected,
unprojected,
owner_subject: { missing: owner.reduce((a, r) => a + r.n, 0), by_app: owner.map(r => ({ app_id: r.app_id, count: r.n })) },
owner_subject: { missing: owner.reduce((a, r) => a + r.n, 0), by_app: owner.map(r => ({ app_id: r.app_id, count: r.n })) },
namespace_reservations_seeded: !!await db.get("SELECT 1 AS x FROM media_settings WHERE key = 'namespaces.reservations_seeded'"),
namespace_reservations_seeded: !!await db.get("SELECT 1 AS x FROM media_settings WHERE key = 'namespaces.reservations_seeded'"),
- note: 'objects: the last boot or scripted backfill that changed something (media_settings objects.backfill.last_report). unprojected: rows with no object yet. owner_subject: objects the owner-subject job has not resolved.',
+ note: 'unprojected: rows with no object yet. owner_subject: objects the owner-subject job has not resolved.',
};
};
}
}
@@ -238,7 +231,7 @@ async function report({ appId = null, limit = 50 } = {}) {
@@ -238,7 +231,7 @@ async function report({ appId = null, limit = 50 } = {}) {
tenants: (await db.all('SELECT app_id FROM apps ORDER BY app_id')).map(r => r.app_id),
tenants: (await db.all('SELECT app_id FROM apps ORDER BY app_id')).map(r => r.app_id),
jobs: await jobs(appId, n),
jobs: await jobs(appId, n),
missing: await missing(appId, n),
missing: await missing(appId, n),
- backfill: await backfill(appId),
+ projection: await projection(appId),
tiering: await tiering(appId, n),
tiering: await tiering(appId, n),
namespaces: await namespaceSnapshot(appId),
namespaces: await namespaceSnapshot(appId),
webhooks: await webhooks(appId),
webhooks: await webhooks(appId),
@@ -332,7 +332,7 @@ function renderOps({ person, report, canRecompute, recomputed = null }) {
@@ -332,7 +332,7 @@ function renderOps({ person, report, canRecompute, recomputed = null }) {
const kv = (obj) => Object.entries(obj || {}).map(([k, v]) => `${esc(k.replace(/_/g, ' '))} ${num(v)}`).join(' · ') || 'none';
const kv = (obj) => Object.entries(obj || {}).map(([k, v]) => `${esc(k.replace(/_/g, ' '))} ${num(v)}`).join(' · ') || 'none';
const t = r.tiering;
const t = r.tiering;
const no = t.native_objects || null;
const no = t.native_objects || null;
- const bf = r.backfill;
+ const projection = r.projection;
const m = r.missing;
const m = r.missing;
const body = `
const body = `
<h1>Media operations</h1>
<h1>Media operations</h1>
@@ -346,7 +346,7 @@ function renderOps({ person, report, canRecompute, recomputed = null }) {
@@ -346,7 +346,7 @@ function renderOps({ person, report, canRecompute, recomputed = null }) {
${card(num(r.jobs.failed.total), 'failed jobs')}
${card(num(r.jobs.failed.total), 'failed jobs')}
${card(num(m.no_good_copy.total), 'ready objects with no good copy')}
${card(num(m.no_good_copy.total), 'ready objects with no good copy')}
${card(num(m.locations.missing_or_corrupt.length), 'missing or corrupt copies listed')}
${card(num(m.locations.missing_or_corrupt.length), 'missing or corrupt copies listed')}
- ${card(num(Object.values(bf.unprojected).reduce((a, b) => a + b, 0)), 'rows without an object')}
+ ${card(num(Object.values(projection.unprojected).reduce((a, b) => a + b, 0)), 'rows without an object')}
${card(num(t.pending_offload), 'VODs eligible to offload')}
${card(num(t.pending_offload), 'VODs eligible to offload')}
</div>
</div>
@@ -392,14 +392,13 @@ function renderOps({ person, report, canRecompute, recomputed = null }) {
@@ -392,14 +392,13 @@ function renderOps({ person, report, canRecompute, recomputed = null }) {
${table('Copies by provider and state', ['Provider', 'State', 'Copies'], m.locations.by_provider_state.map(x => [esc(x.provider), esc(x.state), num(x.count)]), { numeric: [2] }) || '<p class="empty">None.</p>'}
${table('Copies by provider and state', ['Provider', 'State', 'Copies'], m.locations.by_provider_state.map(x => [esc(x.provider), esc(x.state), num(x.count)]), { numeric: [2] }) || '<p class="empty">None.</p>'}
</section>
</section>
- <section aria-labelledby="backfill-h">
- <h2 id="backfill-h">Backfill</h2>
- <p class="hint">${esc(bf.note)}</p>
+ <section aria-labelledby="projection-h">
+ <h2 id="projection-h">Projection</h2>
+ <p class="hint">${esc(projection.note)}</p>
<dl class="facts">
<dl class="facts">
- <dt>Last object backfill</dt><dd>${bf.objects ? `${esc(fmtWhen(bf.objects.finished_at))}${bf.objects.dry_run ? ' (dry run)' : ''}: ${bf.objects.totals ? kv(bf.objects.totals) : 'no totals'}; ${num(bf.objects.errors)} errors, ${num(bf.objects.skipped)} skipped` : 'none recorded'}</dd>
- <dt>Rows without an object</dt><dd>${kv(bf.unprojected)}</dd>
- <dt>Owner subject unresolved</dt><dd>${num(bf.owner_subject.missing)}${bf.owner_subject.by_app.length ? ` (${bf.owner_subject.by_app.map(a => `${esc(a.app_id)} ${num(a.count)}`).join(', ')})` : ''}</dd>
- <dt>Upload reservations seeded</dt><dd>${bf.namespace_reservations_seeded ? 'yes' : 'no'}</dd>
+ <dt>Rows without an object</dt><dd>${kv(projection.unprojected)}</dd>
+ <dt>Owner subject unresolved</dt><dd>${num(projection.owner_subject.missing)}${projection.owner_subject.by_app.length ? ` (${projection.owner_subject.by_app.map(a => `${esc(a.app_id)} ${num(a.count)}`).join(', ')})` : ''}</dd>
+ <dt>Upload reservations seeded</dt><dd>${projection.namespace_reservations_seeded ? 'yes' : 'no'}</dd>
</dl>
</dl>
</section>
</section>
@@ -441,7 +440,7 @@ function renderOps({ person, report, canRecompute, recomputed = null }) {
@@ -441,7 +440,7 @@ function renderOps({ person, report, canRecompute, recomputed = null }) {
}), { numeric: [2, 3] }) || '<p class="empty">No app is configured to receive webhooks.</p>'}
}), { numeric: [2, 3] }) || '<p class="empty">No app is configured to receive webhooks.</p>'}
</section>
</section>
<p class="hint">JSON: <a href="/api/v2/me/ops${scopeQs}">/api/v2/me/ops</a>.</p>`;
<p class="hint">JSON: <a href="/api/v2/me/ops${scopeQs}">/api/v2/me/ops</a>.</p>`;
- return doc({ title: 'Media operations', description: 'Operator views of OpenVibe.Media: failed jobs, usage, missing media, backfill, tiering and webhooks.', body });
+ return doc({ title: 'Media operations', description: 'Operator views of OpenVibe.Media: failed jobs, usage, missing media, projection, tiering and webhooks.', body });
}
}
module.exports = {
module.exports = {
@@ -15,7 +15,7 @@
@@ -15,7 +15,7 @@
* Operator views, for Network staff (contracts staff map, ADR-022): staff.site.view reads,
* Operator views, for Network staff (contracts staff map, ADR-022): staff.site.view reads,
* staff.site.configure recomputes (POST, same-origin only when the credential is the cookie):
* staff.site.configure recomputes (POST, same-origin only when the credential is the cookie):
*
*
- * GET /api/v2/me/ops[?app] failed jobs, missing media, backfill, tiering, namespaces (me/ops.js)
+ * GET /api/v2/me/ops[?app] failed jobs, missing media, projection, tiering, namespaces (me/ops.js)
* POST /api/v2/me/ops/recompute[?app] refresh the namespaces' usage snapshot
* POST /api/v2/me/ops/recompute[?app] refresh the namespaces' usage snapshot
* GET /me/ops, POST /me/ops/recompute the page and its form
* GET /me/ops, POST /me/ops/recompute the page and its form
*
*
deleted file mode 100644
deleted file mode 100644
@@ -1,111 +0,0 @@
@@ -1,111 +0,0 @@
-/**
- * OpenVibe.Media — object backfill
- *
- * One media_object (+ locations, relationships, thumbnail variant) per existing
- * vod, clip, file, screenshot/avatar paste and vod/clip thumbnail, derived by the
- * same sync functions every write uses (model.withObject). Idempotent: objects are keyed by their
- * legacy ref, so a re-run updates in place and creates nothing new.
- *
- * Never touches bytes and never calls B2/R2: local copies are 'present' when the
- * file exists on disk and 'missing' when it does not; remote copies stay 'pending'
- * until `scripts/reconcile-objects.js --verify` HEADs them.
- *
- * Runs in one transaction; dryRun rolls it back and only reports.
- */
-'use strict';
-
-const db = require('../db/database');
-const model = require('./model');
-
-const SETTING_KEY = 'objects.backfill.last_report';
-
-function emptyCounts() {
- return { seen: 0, created: 0, updated: 0, skipped: 0 };
-}
-
-async function backfill({ dryRun = false, onlyMissing = false } = {}) {
- const report = {
- started_at: new Date().toISOString(), dry_run: !!dryRun, only_missing: !!onlyMissing,
- counts: { vod: emptyCounts(), clip: emptyCounts(), file: emptyCounts(), screenshot: emptyCounts(), avatar: emptyCounts(), thumbnail: emptyCounts() },
- locations: { present: 0, missing: 0, pending: 0 },
- skipped: [],
- errors: [],
- };
- const where = onlyMissing ? ' WHERE object_id IS NULL' : '';
-
- const tally = (kind, table, id, r) => {
- const c = report.counts[kind];
- c.seen++;
- if (!r) return;
- if (r.skipped) { c.skipped++; report.skipped.push({ table, id, reason: r.skipped }); return; }
- c[r.created ? 'created' : 'updated']++;
- for (const s of r.locations || []) if (s in report.locations) report.locations[s]++;
- };
- const thumb = (table, id, r) => {
- if (!r || !r.thumbnail) return;
- tally('thumbnail', `${table}.thumbnail_url`, id, r.thumbnail);
- };
- const d = db.getDb();
- // One savepoint per row (a nested transaction): a row that fails leaves nothing half-written behind.
- const each = async (table, sql, fn) => {
- for (const row of await db.all(sql)) {
- try { await d.tx(async () => { await fn(row); }); } catch (err) { report.errors.push({ table, id: row.id ?? row.key, error: err.message }); }
- }
- };
-
- // One transaction for the run; a dry run rolls it back at the end.
- const DRY = Symbol('dry run');
- try {
- await d.tx(async () => {
- // VODs before clips, so clip_of can point at the vod's object.
- await each('vods', `SELECT * FROM vods${where} ORDER BY id`, async (row) => {
- const r = await model.syncVod(row);
- tally('vod', 'vods', row.id, r);
- thumb('vods', row.id, r);
- });
- await each('clips', `SELECT * FROM clips${where} ORDER BY id`, async (row) => {
- const r = await model.syncClip(row);
- tally('clip', 'clips', row.id, r);
- thumb('clips', row.id, r);
- });
- await each('files', `SELECT * FROM files${where} ORDER BY created_at, key`, async (row) => {
- tally('file', 'files', row.key, await model.syncFile(row));
- });
- await each('pastes', `SELECT * FROM pastes${where ? where + ' AND' : ' WHERE'} type = 'screenshot' ORDER BY id`, async (row) => {
- const r = await model.syncPaste(row);
- const kind = (r && r.kind) || (String(row.metadata || '').includes('"kind":"avatar"') ? 'avatar' : 'screenshot');
- tally(kind, 'pastes', row.id, r);
- });
- if (dryRun) throw DRY;
- });
- } catch (err) {
- if (err !== DRY) throw err;
- }
-
- report.finished_at = new Date().toISOString();
- report.totals = Object.values(report.counts).reduce((t, c) => {
- for (const k of Object.keys(t)) t[k] += c[k];
- return t;
- }, emptyCounts());
- if (!dryRun && (!onlyMissing || report.totals.created || report.totals.updated)) {
- // Keep the latest report (skipped rows capped) for operators: media_settings.objects.backfill.last_report.
- const stored = { ...report, skipped: report.skipped.slice(0, 500), skipped_truncated: report.skipped.length > 500 };
- await db.run(`INSERT INTO media_settings (key, value, description, type) VALUES (?, ?, 'Last object backfill report', 'json')
- ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = ov_now()`, [SETTING_KEY, JSON.stringify(stored)]);
- }
- return report;
-}
-
-async function lastReport() {
- const row = await db.get('SELECT value FROM media_settings WHERE key = ?', [SETTING_KEY]);
- return row ? model.parseJson(row.value, null) : null;
-}
-
-function summarize(report) {
- const parts = Object.entries(report.counts).filter(([, c]) => c.seen)
- .map(([k, c]) => `${k} ${c.created}+/${c.updated}~/${c.skipped} skipped`);
- return `${report.dry_run ? '[dry run] ' : ''}${parts.join(', ') || 'nothing to do'}; locations present ${report.locations.present}, `
- + `missing ${report.locations.missing}, pending ${report.locations.pending}${report.errors.length ? `; ${report.errors.length} error(s)` : ''}`;
-}
-
-module.exports = { backfill, lastReport, summarize, SETTING_KEY };
deleted file mode 100644
deleted file mode 100644
@@ -1,152 +0,0 @@
@@ -1,152 +0,0 @@
-/**
- * OpenVibe.Media — object drift report (WS-G task 1: retiring compatibility shim C-75)
- *
- * Every write to an inherited vods/clips/files/pastes row now makes its media_object in the same
- * transaction (model.withObject). This report checks that nothing has drifted anyway: for each
- * projected row it derives what the object must say, with the same <kind>Projection() functions
- * the writes use, and compares that with the object on record, on the fields a sync writes:
- *
- * kind, visibility, lifecycle_status, size_bytes, legacy_ref, app_id, owner_app, owner_user_id
- *
- * and the link to it (the row's object_id; for a vod/clip thumbnail, its parent's thumbnail
- * variant). Each checked row is ok, or has one or more of:
- * missing no object under its legacy ref, nor at the row's object_id
- * unlinked the row's object_id (the thumbnail variant) is not the object its legacy ref names
- * mismatch the object disagrees with the row on one of the fields above
- * Rows that have no object by design (clips-only recordings, screenshot pastes without a file,
- * external thumbnail URLs) are counted as skipped; text pastes are not projected and not listed.
- *
- * READ-ONLY. Pure SQL over a small `q` interface ({ all(sql, params), get(sql, params) }), so the
- * script (scripts/object-drift-report.js) runs it on a database opened read-only; beyond that it
- * only stats local files, as the projection does. Nothing is written, re-projected or moved.
- */
-'use strict';
-
-const model = require('./model');
-
-const FIELDS = ['kind', 'visibility', 'lifecycle_status', 'size_bytes', 'legacy_ref', 'app_id', 'owner_app', 'owner_user_id'];
-const KINDS = ['vod', 'clip', 'file', 'screenshot', 'avatar', 'thumbnail'];
-const PAGE = 500;
-const OBJECT_COLS = `id, ${FIELDS.join(', ')}`;
-
-/** The compared fields as project() writes them from a projection input. */
-function expectedFields(p) {
- return {
- kind: p.kind, visibility: p.visibility, lifecycle_status: p.lifecycle_status, size_bytes: Number(p.size_bytes) || 0,
- legacy_ref: p.legacy_ref, app_id: p.app_id, owner_app: p.owner_app || p.app_id, owner_user_id: p.owner_user_id ?? null,
- };
-}
-
-// The store hands bigint back as a number or a string and TEXT as a string; an app-local user id may be either.
-const same = (a, b) => (a == null ? '' : String(a)) === (b == null ? '' : String(b));
-
-/** Every row of `table` (optionally one tenant's) in key order, a page at a time. */
-async function* eachRow(q, table, keyCol, { appId = null, where = null } = {}) {
- const conds = [];
- const params = [];
- if (appId) { conds.push('app_id = ?'); params.push(appId); }
- if (where) conds.push(where);
- let last = null;
- for (;;) {
- const c = last == null ? conds : [...conds, `${keyCol} > ?`];
- const page = await q.all(`SELECT * FROM ${table}${c.length ? ` WHERE ${c.join(' AND ')}` : ''} ORDER BY ${keyCol} LIMIT ${PAGE}`,
- last == null ? params : [...params, last]);
- for (const r of page) yield r;
- if (page.length < PAGE) return;
- last = page[page.length - 1][keyCol];
- }
-}
-
-async function buildReport(q, { appId = null, limit = 20 } = {}) {
- const report = {
- generated_at: new Date().toISOString(),
- app_id: appId,
- limit,
- fields: FIELDS,
- kinds: Object.fromEntries(KINDS.map(k => [k, { rows: 0, ok: 0, skipped: 0, drifted: 0, missing: 0, unlinked: 0, mismatch: 0, examples: [] }])),
- field_counts: Object.fromEntries(FIELDS.map(f => [f, 0])),
- total_rows: 0,
- total_drift: 0,
- };
- // Objects by legacy ref, loaded once (the compared columns only); a row's object_id is looked up when its ref finds none.
- const byRef = new Map((await q.all(`SELECT ${OBJECT_COLS} FROM media_objects WHERE legacy_ref IS NOT NULL`)).map(o => [o.legacy_ref, o]));
- const byId = async (id) => (id ? await q.get(`SELECT ${OBJECT_COLS} FROM media_objects WHERE id = ?`, [id]) || null : null);
-
- /** Compare one row's projection with its object. Returns the object found (or null). */
- const check = async (kind, where, p, linkedId) => {
- const k = report.kinds[kind];
- k.rows++;
- report.total_rows++;
- const found = byRef.get(p.legacy_ref) || await byId(linkedId);
- const problems = [];
- const diff = {};
- if (!found) {
- problems.push('missing');
- } else {
- if (!same(linkedId, found.id)) { problems.push('unlinked'); diff.object_id = { row: linkedId || null, object: found.id }; }
- const want = expectedFields(p);
- for (const f of FIELDS) {
- if (same(want[f], found[f])) continue;
- diff[f] = { row: want[f], object: found[f] ?? null };
- report.field_counts[f]++;
- }
- if (FIELDS.some(f => f in diff)) problems.push('mismatch');
- }
- if (!problems.length) { k.ok++; return found; }
- k.drifted++;
- report.total_drift++;
- for (const pr of problems) k[pr]++;
- if (k.examples.length < limit) k.examples.push({ ...where, legacy_ref: p.legacy_ref, object_id: found ? found.id : null, problems, diff });
- return found;
- };
- const skip = (kind) => { report.kinds[kind].skipped++; };
- const thumbnailOf = async (row, table, parentKind, parent) => {
- const tp = model.thumbnailProjection(row, parentKind);
- if (!tp) return;
- if (tp.skipped) return skip('thumbnail');
- const variant = parent ? await q.get("SELECT derived_object_id FROM media_variants WHERE object_id = ? AND variant_name = 'thumbnail'", [parent.id]) : null;
- await check('thumbnail', { table, id: row.id, of: parentKind }, tp, variant ? variant.derived_object_id : null);
- };
-
- for await (const row of await eachRow(q, 'vods', 'id', { appId })) {
- const p = model.vodProjection(row);
- if (p.skipped) { skip('vod'); continue; }
- await thumbnailOf(row, 'vods', 'vod', await check('vod', { table: 'vods', id: row.id }, p, row.object_id));
- }
- for await (const row of await eachRow(q, 'clips', 'id', { appId })) {
- await thumbnailOf(row, 'clips', 'clip', await check('clip', { table: 'clips', id: row.id }, model.clipProjection(row), row.object_id));
- }
- for await (const row of await eachRow(q, 'files', 'key', { appId })) {
- await check('file', { table: 'files', id: row.key }, model.fileProjection(row), row.object_id);
- }
- for await (const row of await eachRow(q, 'pastes', 'id', { appId, where: "type = 'screenshot'" })) {
- const p = model.pasteProjection(row);
- if (p.skipped) { skip('screenshot'); continue; }
- await check(p.kind, { table: 'pastes', id: row.id, slug: row.slug }, p, row.object_id);
- }
- return report;
-}
-
-const fmt = (v) => (v == null ? 'null' : typeof v === 'string' ? v : JSON.stringify(v));
-
-function formatReport(r) {
- const lines = [`Object drift report — ${r.generated_at}${r.app_id ? ` (app ${r.app_id})` : ' (all apps)'}`];
- lines.push(` ${'kind'.padEnd(11)} ${'rows'.padStart(7)} ${'ok'.padStart(7)} ${'drifted'.padStart(8)} ${'missing'.padStart(8)} ${'unlinked'.padStart(9)} ${'mismatch'.padStart(9)} ${'skipped'.padStart(8)}`);
- for (const [kind, c] of Object.entries(r.kinds)) {
- lines.push(` ${kind.padEnd(11)} ${String(c.rows).padStart(7)} ${String(c.ok).padStart(7)} ${String(c.drifted).padStart(8)} ${String(c.missing).padStart(8)} ${String(c.unlinked).padStart(9)} ${String(c.mismatch).padStart(9)} ${String(c.skipped).padStart(8)}`);
- }
- const fields = Object.entries(r.field_counts).filter(([, n]) => n).map(([f, n]) => `${f} ${n}`);
- lines.push(` fields that disagree: ${fields.join(', ') || 'none'}`);
- lines.push(` total drift: ${r.total_drift} of ${r.total_rows} row(s)${r.total_drift ? '' : ' — none'}`);
- for (const [kind, c] of Object.entries(r.kinds)) {
- if (!c.examples.length) continue;
- lines.push(` ${kind} examples (${c.examples.length} of ${c.drifted}):`);
- for (const e of c.examples) {
- const what = Object.entries(e.diff).map(([f, d]) => `${f} row=${fmt(d.row)} object=${fmt(d.object)}`).join('; ');
- lines.push(` ${e.table} ${e.id}${e.of ? ` (${e.of} thumbnail)` : ''} ${e.object_id || '-'} ${e.problems.join('+')}${what ? `: ${what}` : ` (${e.legacy_ref})`}`);
- }
- }
- return lines.join('\n');
-}
-
-module.exports = { buildReport, formatReport, expectedFields, FIELDS, KINDS };
@@ -5,10 +5,8 @@
@@ -5,10 +5,8 @@
* row per provider copy. The inherited vods/clips/files/pastes rows are typed
* row per provider copy. The inherited vods/clips/files/pastes rows are typed
* projections over objects: each carries object_id, and the sync*() functions
* projections over objects: each carries object_id, and the sync*() functions
* below derive the object + locations from the row as it is right now. The same
* below derive the object + locations from the row as it is right now. The same
- * functions back the one-off backfill and every write on the old APIs, which
- * writes the row and its object in one transaction (withObject, WS-G task 1), so
- * the model cannot drift between "imported" and "written since". The drift
- * report (./drift.js) checks that it has not.
+ * functions back every write on the old APIs, which writes the row and its object
+ * in one transaction (withObject, WS-G task 1).
*
*
* Location states: present (verified: the local file exists / a HEAD answered),
* Location states: present (verified: the local file exists / a HEAD answered),
* missing (verified absent), pending (believed there, not yet verified — every
* missing (verified absent), pending (believed there, not yet verified — every
@@ -49,7 +47,7 @@ function parseJson(s, fallback) {
@@ -49,7 +47,7 @@ function parseJson(s, fallback) {
try { return JSON.parse(s); } catch { return fallback; }
try { return JSON.parse(s); } catch { return fallback; }
}
}
-/** Epoch ms from a `YYYY-MM-DD HH:MM:SS` (UTC) text timestamp, as the migrations store them, so backfilled ids sort by original creation time. */
+/** Epoch ms from a `YYYY-MM-DD HH:MM:SS` (UTC) text timestamp, as the migrations store them, so projected legacy ids sort by original creation time. */
function toMs(createdAt) {
function toMs(createdAt) {
const ms = Date.parse(String(createdAt || '').replace(' ', 'T') + (/[zZ]|[+-]\d\d:?\d\d$/.test(String(createdAt || '')) ? '' : 'Z'));
const ms = Date.parse(String(createdAt || '').replace(' ', 'T') + (/[zZ]|[+-]\d\d:?\d\d$/.test(String(createdAt || '')) ? '' : 'Z'));
return Number.isFinite(ms) ? ms : Date.now();
return Number.isFinite(ms) ? ms : Date.now();
@@ -302,9 +300,8 @@ async function recordInvariant(objectId) {
@@ -302,9 +300,8 @@ async function recordInvariant(objectId) {
// ── Per-kind projections ─────────────────────────────────────
// ── Per-kind projections ─────────────────────────────────────
// <kind>Projection(row) derives what the object behind one inherited row must say (project()'s input).
// <kind>Projection(row) derives what the object behind one inherited row must say (project()'s input).
-// Read-only (it stats local files, never writes), and shared by the sync*() writers below and the
-// drift report (./drift.js), so the report checks exactly what a write would have written. A row
-// that has no object of its own answers { skipped: reason }.
+// Read-only (it stats local files, never writes), and shared by the sync*() writers below.
+// A row that has no object of its own answers { skipped: reason }.
function vodLifecycle(row) {
function vodLifecycle(row) {
const s = db.vodStatus(row);
const s = db.vodStatus(row);
@@ -519,7 +516,7 @@ async function safeSync(kind, id) {
@@ -519,7 +516,7 @@ async function safeSync(kind, id) {
}
}
/**
/**
- * Object-first write (WS-G task 1; retires compatibility shim C-75, "write the row, then sync its
+ * Object-first write (WS-G task 1; C-75 retired 2026-10-10, "write the row, then sync its
* object"): write() changes inherited rows and the objects behind them are re-projected in the SAME
* object"): write() changes inherited rows and the objects behind them are re-projected in the SAME
* transaction, so both commit or neither does: no crash or error between the two can leave
* transaction, so both commit or neither does: no crash or error between the two can leave
* an object behind its row. kind vod|clip|file|paste; ids the row id (a file's key), a list of them,
* an object behind its row. kind vod|clip|file|paste; ids the row id (a file's key), a list of them,
@@ -546,7 +543,7 @@ async function withObject(kind, ids, write) {
@@ -546,7 +543,7 @@ async function withObject(kind, ids, write) {
* withObject() for a row write that records something already done outside the database (a
* withObject() for a row write that records something already done outside the database (a
* recorder's ffmpeg is running): the row must say so whatever happens to its object. When the
* recorder's ffmpeg is running): the row must say so whatever happens to its object. When the
* combined transaction fails, the row is written alone and the object re-projected after it (the
* combined transaction fails, the row is written alone and the object re-projected after it (the
- * pre-WS-G two steps; logged, and the drift report lists the row until a later write catches up).
+ * pre-WS-G two steps; logged, and the row's next write brings its object level).
*/
*/
async function withObjectOrRow(kind, ids, write) {
async function withObjectOrRow(kind, ids, write) {
try { return await withObject(kind, ids, write); } catch (err) {
try { return await withObject(kind, ids, write); } catch (err) {
@@ -4,13 +4,12 @@
@@ -4,13 +4,12 @@
*
*
* Every MEDIA_OWNER_SUBJECT_INTERVAL_MIN minutes (first run a minute after boot), objects that name
* Every MEDIA_OWNER_SUBJECT_INTERVAL_MIN minutes (first run a minute after boot), objects that name
* an app-local owner (owner_app + owner_user_id) and no owner_subject get their owner's canonical
* an app-local owner (owner_app + owner_user_id) and no owner_subject get their owner's canonical
- * subject from Network (server/objects/owner-subject.js). This is how objects created since the
- * backfill get one: the upload and projection paths never wait on Network, and an object whose owner
- * Network does not know yet (a Live account not linked to Network) is filled on a later run once it is.
+ * subject from Network (server/objects/owner-subject.js). This is how new objects get one: the upload
+ * and projection paths never wait on Network, and an object whose owner Network does not know yet (a
+ * Live account not linked to Network) is filled on a later run once it is.
* When there is nothing to fill, a run asks Network nothing.
* When there is nothing to fill, a run asks Network nothing.
*
*
- * MEDIA_OWNER_SUBJECT_SYNC=0 turns it off (do that before rolling a backfill back, or the next run
- * fills the rows again).
+ * MEDIA_OWNER_SUBJECT_SYNC=0 turns it off.
*/
*/
const db = require('../db/database');
const db = require('../db/database');
const config = require('../config');
const config = require('../config');
@@ -16,7 +16,7 @@
@@ -16,7 +16,7 @@
* size_mismatch / hash_mismatch bytes differ from the object's size / sha256
* size_mismatch / hash_mismatch bytes differ from the object's size / sha256
* orphan_location location rows whose object does not exist
* orphan_location location rows whose object does not exist
* deleted_publicly_reachable deleted object still served (legacy row, or a thumbnail of it)
* deleted_publicly_reachable deleted object still served (legacy row, or a thumbnail of it)
- * missing_projection inherited rows with no object yet (run the backfill)
+ * missing_projection inherited rows with no object yet (investigate the write path)
* incomplete_multipart multipart sessions still open past their expiry (the hourly purge
* incomplete_multipart multipart sessions still open past their expiry (the hourly purge
* removes their parts; one listed here means the purge is not running)
* removes their parts; one listed here means the purge is not running)
*/
*/
@@ -166,11 +166,6 @@ async function finalizeVod(vodId, opts = {}) {
@@ -166,11 +166,6 @@ async function finalizeVod(vodId, opts = {}) {
throw err;
throw err;
} finally {
} finally {
_finalizing.delete(vodId);
_finalizing.delete(vodId);
- // Every outcome (ready, quarantined, failed) now commits with its object (withObject); a deleted
- // row's object is marked by its trigger. This follow-up only re-reads the disk after a finalize
- // that threw part-way (a merge or remux rewrote the file, no row write followed). C-75 leftover:
- // it goes with the boot backfill (server/index.js) once the drift report shows zero drift.
- await objects().safeSync('vod', vodId);
}
}
}
}
@@ -141,7 +141,7 @@ const { spawnSync, execFileSync } = require('child_process');
@@ -141,7 +141,7 @@ const { spawnSync, execFileSync } = require('child_process');
await db.run("INSERT INTO assets (id, app_id, kind, name, file_path, mime) OVERRIDING SYSTEM VALUE VALUES (1, 'live', 'emote', 'drillWave', ?, 'image/png')", [bytes.asset]);
await db.run("INSERT INTO assets (id, app_id, kind, name, file_path, mime) OVERRIDING SYSTEM VALUE VALUES (1, 'live', 'emote', 'drillWave', ?, 'image/png')", [bytes.asset]);
await db.run("INSERT INTO pastes (id, app_id, slug, type, title, screenshot_path, visibility) OVERRIDING SYSTEM VALUE VALUES (1, 'live', 'shot1', 'screenshot', 'Shot', ?, 'public')", [bytes.shot]);
await db.run("INSERT INTO pastes (id, app_id, slug, type, title, screenshot_path, visibility) OVERRIDING SYSTEM VALUE VALUES (1, 'live', 'shot1', 'screenshot', 'Shot', ?, 'public')", [bytes.shot]);
await db.run("INSERT INTO files (key, app_id, original_name, size, mime) VALUES ('f1.txt', 'live', 'f1.txt', 13, 'text/plain')");
await db.run("INSERT INTO files (key, app_id, original_name, size, mime) VALUES ('f1.txt', 'live', 'f1.txt', 13, 'text/plain')");
- await require('../server/objects/backfill').backfill({ onlyMissing: true });
+ await require('./helpers/project-rows').projectRows({ onlyMissing: true });
const COUNTED = ['media_objects', 'vods', 'clips', 'apps', 'media_locations', 'media_jobs', 'event_outbox'];
const COUNTED = ['media_objects', 'vods', 'clips', 'apps', 'media_locations', 'media_jobs', 'event_outbox'];
const counts = async () => Object.fromEntries(await Promise.all(COUNTED.map(async (t) => [t, Number((await db.get(`SELECT COUNT(*) AS n FROM ${t}`)).n)])));
const counts = async () => Object.fromEntries(await Promise.all(COUNTED.map(async (t) => [t, Number((await db.get(`SELECT COUNT(*) AS n FROM ${t}`)).n)])));
const countsBefore = await counts();
const countsBefore = await counts();
new file mode 100644
new file mode 100644
@@ -0,0 +1,98 @@
@@ -0,0 +1,98 @@
+/**
+ * OpenVibe.Media — test-only legacy row projection
+ *
+ * One media_object (+ locations, relationships, thumbnail variant) per existing
+ * vod, clip, file, screenshot/avatar paste and vod/clip thumbnail, derived by the
+ * same sync functions every write uses (model.withObject). Idempotent: objects are keyed by their
+ * legacy ref, so a re-run updates in place and creates nothing new.
+ *
+ * Never touches bytes and never calls B2/R2: local copies are 'present' when the
+ * file exists on disk and 'missing' when it does not; remote copies stay 'pending'
+ * until `scripts/reconcile-objects.js --verify` HEADs them.
+ *
+ * Runs in one transaction; dryRun rolls it back and only reports.
+ */
+'use strict';
+
+const db = require('../../server/db/database');
+const model = require('../../server/objects/model');
+
+function emptyCounts() {
+ return { seen: 0, created: 0, updated: 0, skipped: 0 };
+}
+
+async function projectRows({ dryRun = false, onlyMissing = false } = {}) {
+ const report = {
+ started_at: new Date().toISOString(), dry_run: !!dryRun, only_missing: !!onlyMissing,
+ counts: { vod: emptyCounts(), clip: emptyCounts(), file: emptyCounts(), screenshot: emptyCounts(), avatar: emptyCounts(), thumbnail: emptyCounts() },
+ locations: { present: 0, missing: 0, pending: 0 },
+ skipped: [],
+ errors: [],
+ };
+ const where = onlyMissing ? ' WHERE object_id IS NULL' : '';
+
+ const tally = (kind, table, id, r) => {
+ const c = report.counts[kind];
+ c.seen++;
+ if (!r) return;
+ if (r.skipped) { c.skipped++; report.skipped.push({ table, id, reason: r.skipped }); return; }
+ c[r.created ? 'created' : 'updated']++;
+ for (const s of r.locations || []) if (s in report.locations) report.locations[s]++;
+ };
+ const thumb = (table, id, r) => {
+ if (!r || !r.thumbnail) return;
+ tally('thumbnail', `${table}.thumbnail_url`, id, r.thumbnail);
+ };
+ const d = db.getDb();
+ // One savepoint per row (a nested transaction): a row that fails leaves nothing half-written behind.
+ const each = async (table, sql, fn) => {
+ for (const row of await db.all(sql)) {
+ try { await d.tx(async () => { await fn(row); }); } catch (err) { report.errors.push({ table, id: row.id ?? row.key, error: err.message }); }
+ }
+ };
+
+ // One transaction for the run; a dry run rolls it back at the end.
+ const DRY = Symbol('dry run');
+ try {
+ await d.tx(async () => {
+ // VODs before clips, so clip_of can point at the vod's object.
+ await each('vods', `SELECT * FROM vods${where} ORDER BY id`, async (row) => {
+ const r = await model.syncVod(row);
+ tally('vod', 'vods', row.id, r);
+ thumb('vods', row.id, r);
+ });
+ await each('clips', `SELECT * FROM clips${where} ORDER BY id`, async (row) => {
+ const r = await model.syncClip(row);
+ tally('clip', 'clips', row.id, r);
+ thumb('clips', row.id, r);
+ });
+ await each('files', `SELECT * FROM files${where} ORDER BY created_at, key`, async (row) => {
+ tally('file', 'files', row.key, await model.syncFile(row));
+ });
+ await each('pastes', `SELECT * FROM pastes${where ? where + ' AND' : ' WHERE'} type = 'screenshot' ORDER BY id`, async (row) => {
+ const r = await model.syncPaste(row);
+ const kind = (r && r.kind) || (String(row.metadata || '').includes('"kind":"avatar"') ? 'avatar' : 'screenshot');
+ tally(kind, 'pastes', row.id, r);
+ });
+ if (dryRun) throw DRY;
+ });
+ } catch (err) {
+ if (err !== DRY) throw err;
+ }
+
+ report.finished_at = new Date().toISOString();
+ report.totals = Object.values(report.counts).reduce((t, c) => {
+ for (const k of Object.keys(t)) t[k] += c[k];
+ return t;
+ }, emptyCounts());
+ return report;
+}
+
+function summarize(report) {
+ const parts = Object.entries(report.counts).filter(([, c]) => c.seen)
+ .map(([k, c]) => `${k} ${c.created}+/${c.updated}~/${c.skipped} skipped`);
+ return `${report.dry_run ? '[dry run] ' : ''}${parts.join(', ') || 'nothing to do'}; locations present ${report.locations.present}, `
+ + `missing ${report.locations.missing}, pending ${report.locations.pending}${report.errors.length ? `; ${report.errors.length} error(s)` : ''}`;
+}
+
+module.exports = { projectRows, summarize };
@@ -336,7 +336,7 @@ const http = require('http');
@@ -336,7 +336,7 @@ const http = require('http');
assert.deepStrictEqual(rep.jobs.failed.last_7_days, [{ type: 'object.sprite', error_code: 'no_video', count: 1 }]);
assert.deepStrictEqual(rep.jobs.failed.last_7_days, [{ type: 'object.sprite', error_code: 'no_video', count: 1 }]);
assert.ok(rep.missing.no_good_copy.objects.some(o => o.id === lost), 'a ready object whose only copy is missing');
assert.ok(rep.missing.no_good_copy.objects.some(o => o.id === lost), 'a ready object whose only copy is missing');
assert.ok(rep.missing.locations.missing_or_corrupt.some(l => l.object_id === lost && l.provider === 'b2'));
assert.ok(rep.missing.locations.missing_or_corrupt.some(l => l.object_id === lost && l.provider === 'b2'));
- assert.ok(rep.backfill.unprojected && rep.backfill.owner_subject.missing >= 1);
+ assert.ok(rep.projection.unprojected && rep.projection.owner_subject.missing >= 1);
assert.ok(rep.tiering.objects.some(o => o.origin === 'native') && rep.tiering.sweep && 'pending_offload' in rep.tiering);
assert.ok(rep.tiering.objects.some(o => o.origin === 'native') && rep.tiering.sweep && 'pending_offload' in rep.tiering);
assert.ok(rep.namespaces.some(n => n.namespace === 'live.avatars'));
assert.ok(rep.namespaces.some(n => n.namespace === 'live.avatars'));
// Webhooks section: host only, no URL path and no secret anywhere in the payload; counts are
// Webhooks section: host only, no URL path and no secret anywhere in the payload; counts are
@@ -352,7 +352,7 @@ const http = require('http');
@@ -352,7 +352,7 @@ const http = require('http');
assert.ok(r.status === 200 && r.text.includes('mjob_TESTFAILED') && r.text.includes('action="/me/ops/recompute"') && r.text.includes('noindex'));
assert.ok(r.status === 200 && r.text.includes('mjob_TESTFAILED') && r.text.includes('action="/me/ops/recompute"') && r.text.includes('noindex'));
assert.ok(r.text.includes('webhooks-h') && r.text.includes('hooks.example.test'), 'the page shows the webhooks section and the configured host');
assert.ok(r.text.includes('webhooks-h') && r.text.includes('hooks.example.test'), 'the page shows the webhooks section and the configured host');
assert.ok(!r.text.includes(HOOK_URL) && !r.text.includes('ov-private-hook-path') && !r.text.includes(HOOK_SECRET), 'the page never shows the webhook URL path or secret');
assert.ok(!r.text.includes(HOOK_URL) && !r.text.includes('ov-private-hook-path') && !r.text.includes(HOOK_SECRET), 'the page never shows the webhook URL path or secret');
- console.log('✅ operator views: staff.site.view only (not users, not global mods); failed jobs, missing media, backfill, tiering, namespaces');
+ console.log('✅ operator views: staff.site.view only (not users, not global mods); failed jobs, missing media, projection, tiering, namespaces');
// ── Recompute: staff.site.configure, same-origin ──
// ── Recompute: staff.site.configure, same-origin ──
await db.run("UPDATE media_namespaces SET used_bytes = 0, used_objects = 0, reconciled_at = NULL WHERE namespace = 'live'");
await db.run("UPDATE media_namespaces SET used_bytes = 0, used_objects = 0, reconciled_at = NULL WHERE namespace = 'live'");
@@ -50,7 +50,9 @@ const SEED_PG = `
@@ -50,7 +50,9 @@ const SEED_PG = `
await db.run("INSERT INTO pastes (app_id, slug, type, title, content, visibility) VALUES ('live', 'n1paste', 'paste', 'N-1 paste', 'hello from N-1', 'public')");
await db.run("INSERT INTO pastes (app_id, slug, type, title, content, visibility) VALUES ('live', 'n1paste', 'paste', 'N-1 paste', 'hello from N-1', 'public')");
await db.run("INSERT INTO files (key, app_id, original_name, size, mime) VALUES ('n1file.txt', 'live', 'n1file.txt', 17, 'text/plain')");
await db.run("INSERT INTO files (key, app_id, original_name, size, mime) VALUES ('n1file.txt', 'live', 'n1file.txt', 17, 'text/plain')");
put('FILES_PATH', 'live/n1file.txt'); put('THUMBNAILS_PATH', 'vod-1-1.jpg');
put('FILES_PATH', 'live/n1file.txt'); put('THUMBNAILS_PATH', 'vod-1-1.jpg');
- await require('./server/objects/backfill').backfill({ onlyMissing: true });
+ // N-1 releases before C-75's retirement project rows with server/objects/backfill.js; later ones with the test helper.
+ const projectRows = fs.existsSync('./test/helpers/project-rows.js') ? require('./test/helpers/project-rows').projectRows : require('./server/objects/backfill').backfill;
+ await projectRows({ onlyMissing: true });
const obj = await db.get("SELECT id FROM media_objects WHERE kind = 'vod' ORDER BY created_at, id LIMIT 1");
const obj = await db.get("SELECT id FROM media_objects WHERE kind = 'vod' ORDER BY created_at, id LIMIT 1");
fs.writeFileSync(path.join(process.env.N1_DATA, 'n1-ids.json'), JSON.stringify({ object: obj ? obj.id : null }));
fs.writeFileSync(path.join(process.env.N1_DATA, 'n1-ids.json'), JSON.stringify({ object: obj ? obj.id : null }));
await db.close();
await db.close();
@@ -1,6 +1,6 @@
@@ -1,6 +1,6 @@
'use strict';
'use strict';
// Namespaces and grants (roadmap WS-G task 2; server/objects/namespaces.js, server/auth.js VERBS):
// Namespaces and grants (roadmap WS-G task 2; server/objects/namespaces.js, server/auth.js VERBS):
-// - the backfill on open: developer-project objects move to app.<project_id>[.sandbox], every tenant
+// - namespace setup on open: developer-project objects move to app.<project_id>[.sandbox], every tenant
// and every namespace an object names gets a row, uploads in progress get a reservation (once);
// and every namespace an object names gets a row, uploads in progress get a reservation (once);
// - five verbs per namespace (read, list, write, delete, transform), the older ids still granting the
// - five verbs per namespace (read, list, write, delete, transform), the older ids still granting the
// newer verbs, strict_verbs refusing them; child namespaces and grants that name only a child;
// newer verbs, strict_verbs refusing them; child namespaces and grants that name only a child;
@@ -1,10 +1,9 @@
@@ -1,10 +1,9 @@
'use strict';
'use strict';
-// Object-first writes (WS-G task 1, retiring compatibility shim C-75): every write to an inherited
+// Object-first writes (WS-G task 1, C-75 retired 2026-10-10): every write to an inherited
// vods/clips/files/pastes row makes or updates its media_object in the SAME PostgreSQL transaction
// vods/clips/files/pastes row makes or updates its media_object in the SAME PostgreSQL transaction
// (objects/model.js withObject). For each main write path the object is there, linked and agreeing
// (objects/model.js withObject). For each main write path the object is there, linked and agreeing
-// with the row, as soon as the write returns; when the object cannot be written, the row write
-// rolls back with it (neither row); and the drift report (server/objects/drift.js,
-// scripts/object-drift-report.js) writes nothing and finds drift planted behind the model's back.
+// with the row as soon as the write returns; when the object cannot be written, the row write
+// rolls back with it (neither row).
const assert = require('assert');
const assert = require('assert');
const fs = require('fs');
const fs = require('fs');
const os = require('os');
const os = require('os');
@@ -24,7 +23,6 @@ const sharp = require('sharp');
@@ -24,7 +23,6 @@ const sharp = require('sharp');
const db = require('../server/db/database');
const db = require('../server/db/database');
const model = require('../server/objects/model');
const model = require('../server/objects/model');
- const drift = require('../server/objects/drift');
const tools = require('../server/vod/media-tools');
const tools = require('../server/vod/media-tools');
const thumbService = require('../server/thumbnails/thumbnail-service');
const thumbService = require('../server/thumbnails/thumbnail-service');
const cutter = require('../server/vod/clip-cutter');
const cutter = require('../server/vod/clip-cutter');
@@ -37,7 +35,6 @@ const sharp = require('sharp');
@@ -37,7 +35,6 @@ const sharp = require('sharp');
await db.upsertApp({ app_id: 'live', api_key: 'live-key-object-first' });
await db.upsertApp({ app_id: 'live', api_key: 'live-key-object-first' });
const raw = db.getDb();
const raw = db.getDb();
- const q = { all: async (sql, p = []) => await raw.prepare(sql).all(...p), get: async (sql, p = []) => await raw.prepare(sql).get(...p) };
const count = async (table) => (await raw.prepare(`SELECT COUNT(*) AS n FROM ${table}`).get()).n;
const count = async (table) => (await raw.prepare(`SELECT COUNT(*) AS n FROM ${table}`).get()).n;
// No ffmpeg in these paths: the thumbnail generators and the probes answer as a real file would.
// No ffmpeg in these paths: the thumbnail generators and the probes answer as a real file would.
@@ -56,15 +53,11 @@ const sharp = require('sharp');
@@ -56,15 +53,11 @@ const sharp = require('sharp');
const obj = await model.getObject(row.object_id);
const obj = await model.getObject(row.object_id);
assert.ok(obj, `${label}: object exists right after the write`);
assert.ok(obj, `${label}: object exists right after the write`);
const p = { vods: model.vodProjection, clips: model.clipProjection, files: model.fileProjection, pastes: model.pasteProjection }[table](row);
const p = { vods: model.vodProjection, clips: model.clipProjection, files: model.fileProjection, pastes: model.pasteProjection }[table](row);
- const want = drift.expectedFields(p);
- for (const f of drift.FIELDS) assert.strictEqual(String(obj[f] ?? ''), String(want[f] ?? ''), `${label}: object ${f}`);
+ for (const [field, value] of Object.entries({ kind: p.kind, visibility: p.visibility, lifecycle_status: p.lifecycle_status, size_bytes: Number(p.size_bytes) || 0, legacy_ref: p.legacy_ref, app_id: p.app_id, owner_app: p.owner_app || p.app_id, owner_user_id: p.owner_user_id })) {
+ assert.strictEqual(String(obj[field] ?? ''), String(value ?? ''), `${label}: object ${field}`);
+ }
return { row, obj };
return { row, obj };
}
}
- const noDrift = async (label) => {
- const r = await drift.buildReport(q);
- assert.strictEqual(r.total_drift, 0, `${label}:\n${drift.formatReport(r)}`);
- return r;
- };
const app = express();
const app = express();
app.use(express.json({ limit: '5mb' }));
app.use(express.json({ limit: '5mb' }));
@@ -224,9 +217,6 @@ const sharp = require('sharp');
@@ -224,9 +217,6 @@ const sharp = require('sharp');
assert.deepStrictEqual([(await linked('pastes', 'id', avatarId, 'avatar')).obj.kind, (await linked('pastes', 'id', avatarId, 'avatar')).obj.legacy_ref], ['avatar', 'legacy:network:avatar:av1']);
assert.deepStrictEqual([(await linked('pastes', 'id', avatarId, 'avatar')).obj.kind, (await linked('pastes', 'id', avatarId, 'avatar')).obj.legacy_ref], ['avatar', 'legacy:network:avatar:av1']);
console.log('✅ file, screenshot-paste and avatar writes commit with their object; deletes mark it deleted');
console.log('✅ file, screenshot-paste and avatar writes commit with their object; deletes mark it deleted');
- const clean = await noDrift('after every write path');
- assert.ok(clean.total_rows >= 8, 'the report checked the rows written above');
-
// ── 4. Atomicity: an object write that fails takes the row write with it ──
// ── 4. Atomicity: an object write that fails takes the row write with it ──
// Every object write fails (as a trigger refusing it would): the model's INSERT/UPDATE of media_objects throws.
// Every object write fails (as a trigger refusing it would): the model's INSERT/UPDATE of media_objects throws.
const realRun = db.run;
const realRun = db.run;
@@ -264,56 +254,8 @@ const sharp = require('sharp');
@@ -264,56 +254,8 @@ const sharp = require('sharp');
await unchanged('thrown after the insert: neither row');
await unchanged('thrown after the insert: neither row');
db.run = realRun;
db.run = realRun;
await finalizeVod(vodId, { fromJob: true });
await finalizeVod(vodId, { fromJob: true });
- await noDrift('after the refused writes');
console.log('✅ a write whose object cannot be written leaves neither row (inserts, updates, announce path, a throw after the insert)');
console.log('✅ a write whose object cannot be written leaves neither row (inserts, updates, announce path, a throw after the insert)');
- // ── 5. Drift report: read-only, finds planted drift ──
- await raw.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path) OVERRIDING SYSTEM VALUE VALUES (900, 'live', 3, 'Planted: no object', NULL) RETURNING id").run();
- const upObj = (await db.get('SELECT object_id FROM clips WHERE id = ?', [upId])).object_id;
- await raw.prepare("UPDATE media_objects SET visibility = 'private' WHERE id = ?").run(upObj);
- const fileObj = (await db.get('SELECT object_id FROM files WHERE key = ?', [fileKey])).object_id;
- await raw.prepare('UPDATE media_objects SET size_bytes = 1, owner_user_id = 99 WHERE id = ?').run(fileObj);
- await raw.prepare('UPDATE pastes SET object_id = NULL WHERE slug = ?').run(slug);
- const cutObj = (await db.get('SELECT object_id FROM clips WHERE id = ?', [cutId])).object_id;
- await raw.prepare("UPDATE media_objects SET lifecycle_status = 'deleted' WHERE id = ?").run(cutObj);
-
- const snapshot = async () => (await Promise.all(['vods', 'clips', 'files', 'pastes', 'media_objects', 'media_locations', 'media_variants', 'media_relationships']
- .map(async t => JSON.stringify(await raw.prepare(`SELECT * FROM ${t} ORDER BY ${t}::text`).all())))).join('|');
- const snap = await snapshot();
- const rep = await drift.buildReport(q, { limit: 5 });
- assert.strictEqual(await snapshot(), snap, 'the report changed nothing');
-
- assert.strictEqual(rep.total_drift, 5, drift.formatReport(rep));
- assert.deepStrictEqual([rep.kinds.vod.missing, rep.kinds.clip.mismatch, rep.kinds.file.mismatch, rep.kinds.screenshot.unlinked], [1, 2, 1, 1]);
- assert.deepStrictEqual([rep.field_counts.visibility, rep.field_counts.size_bytes, rep.field_counts.owner_user_id, rep.field_counts.lifecycle_status], [1, 1, 1, 1]);
- const ex = (kind, id) => rep.kinds[kind].examples.find(e => String(e.id) === String(id));
- assert.deepStrictEqual(ex('vod', 900).problems, ['missing']);
- assert.deepStrictEqual(ex('clip', upId).diff.visibility, { row: 'public', object: 'private' });
- assert.deepStrictEqual(ex('clip', cutId).diff.lifecycle_status, { row: 'failed', object: 'deleted' });
- assert.deepStrictEqual(Object.keys(ex('file', fileKey).diff).sort(), ['owner_user_id', 'size_bytes']);
- assert.deepStrictEqual(ex('screenshot', (await db.get('SELECT id FROM pastes WHERE slug = ?', [slug])).id).problems, ['unlinked']);
- assert.strictEqual(rep.kinds.vod.skipped, 1, 'the clips-only recording is skipped, not drift');
- assert.strictEqual((await drift.buildReport(q, { appId: 'network' })).total_drift, 0, '--app narrows the report');
- assert.match(drift.formatReport(rep), /total drift: 5 of \d+ row\(s\)/);
-
- // The script: read-only, exit 0 whatever it finds.
- const { runScript } = require('./helpers/run-script');
- const script = path.join(__dirname, '..', 'scripts', 'object-drift-report.js');
- const r1 = await runScript(script, ['--json', '--limit', '1']);
- assert.strictEqual(r1.code, 0, r1.err);
- const out = JSON.parse(r1.out);
- assert.strictEqual(out.total_drift, 5);
- assert.ok(Object.values(out.kinds).every(k => k.examples.length <= 1), '--limit caps the examples per kind');
- assert.match((await runScript(script, [])).out, /clip examples \(2 of 2\)/);
- assert.strictEqual(await snapshot(), snap, 'the script changed nothing either');
-
- // Re-projecting the planted rows clears the drift the report listed.
- for (const id of [900]) await model.sync('vod', id);
- await model.sync('clip', upId); await model.sync('clip', cutId); await model.sync('file', fileKey);
- await model.sync('paste', (await db.get('SELECT id FROM pastes WHERE slug = ?', [slug])).id);
- await noDrift('after re-projecting the planted drift');
- console.log('✅ drift report: read-only (report and script), counts per kind with examples, finds missing / unlinked / mismatched objects');
-
server.close();
server.close();
fs.rmSync(tmp, { recursive: true, force: true });
fs.rmSync(tmp, { recursive: true, force: true });
console.log('✅ All object-first write tests passed');
console.log('✅ All object-first write tests passed');
@@ -1,5 +1,5 @@
@@ -1,5 +1,5 @@
'use strict';
'use strict';
-// Canonical object model (roadmap Wave 4): the backfill projects every vod, clip, file, screenshot,
+// Canonical object model (roadmap Wave 4): the test helper projects every vod, clip, file, screenshot,
// avatar and thumbnail row onto media_objects + media_locations without touching bytes or providers;
// avatar and thumbnail row onto media_objects + media_locations without touching bytes or providers;
// it is idempotent; the write hooks keep the model current; holds block deletion and tier moves on
// it is idempotent; the write hooks keep the model current; holds block deletion and tier moves on
// every path; the public object-size invariant is recorded as policy.
// every path; the public object-size invariant is recorded as policy.
@@ -27,7 +27,7 @@ const path = require('path');
@@ -27,7 +27,7 @@ const path = require('path');
const contracts = require('openvibe-contracts');
const contracts = require('openvibe-contracts');
const db = require('../server/db/database');
const db = require('../server/db/database');
const model = require('../server/objects/model');
const model = require('../server/objects/model');
- const { backfill } = require('../server/objects/backfill');
+ const { projectRows } = require('./helpers/project-rows');
const invariant = require('../server/objects/invariant');
const invariant = require('../server/objects/invariant');
const vodStorage = require('../server/vod/vod-storage');
const vodStorage = require('../server/vod/vod-storage');
// The revisioned tier policies load once the database is open (server/index.js does this at boot).
// The revisioned tier policies load once the database is open (server/index.js does this at boot).
@@ -68,16 +68,15 @@ const path = require('path');
@@ -68,16 +68,15 @@ const path = require('path');
const mtimes = [vodFile, clipFile, shot, avatar].map(p => fs.statSync(p).mtimeMs);
const mtimes = [vodFile, clipFile, shot, avatar].map(p => fs.statSync(p).mtimeMs);
// ── Dry run: a full report, nothing written ──
// ── Dry run: a full report, nothing written ──
- const dry = await backfill({ dryRun: true });
+ const dry = await projectRows({ dryRun: true });
assert.strictEqual(dry.dry_run, true);
assert.strictEqual(dry.dry_run, true);
assert.deepStrictEqual(dry.counts.vod, { seen: 7, created: 6, updated: 0, skipped: 1 });
assert.deepStrictEqual(dry.counts.vod, { seen: 7, created: 6, updated: 0, skipped: 1 });
assert.strictEqual((await d.prepare('SELECT COUNT(*) c FROM media_objects').get()).c, 0, 'dry run writes no objects');
assert.strictEqual((await d.prepare('SELECT COUNT(*) c FROM media_objects').get()).c, 0, 'dry run writes no objects');
assert.strictEqual((await d.prepare('SELECT COUNT(*) c FROM vods WHERE object_id IS NOT NULL').get()).c, 0, 'dry run links no rows');
assert.strictEqual((await d.prepare('SELECT COUNT(*) c FROM vods WHERE object_id IS NOT NULL').get()).c, 0, 'dry run links no rows');
- assert.strictEqual(model.parseJson((await d.prepare("SELECT value FROM media_settings WHERE key = 'objects.backfill.last_report'").get())?.value, null), null, 'dry run stores no report');
console.log('✅ dry run reports without writing');
console.log('✅ dry run reports without writing');
// ── Real run ──
// ── Real run ──
- const r = await backfill();
+ const r = await projectRows();
assert.deepStrictEqual(r.counts.vod, { seen: 7, created: 6, updated: 0, skipped: 1 });
assert.deepStrictEqual(r.counts.vod, { seen: 7, created: 6, updated: 0, skipped: 1 });
assert.deepStrictEqual(r.counts.clip, { seen: 2, created: 2, updated: 0, skipped: 0 });
assert.deepStrictEqual(r.counts.clip, { seen: 2, created: 2, updated: 0, skipped: 0 });
assert.deepStrictEqual(r.counts.file, { seen: 1, created: 1, updated: 0, skipped: 0 });
assert.deepStrictEqual(r.counts.file, { seen: 1, created: 1, updated: 0, skipped: 0 });
@@ -86,7 +85,6 @@ const path = require('path');
@@ -86,7 +85,6 @@ const path = require('path');
assert.deepStrictEqual(r.counts.thumbnail, { seen: 2, created: 1, updated: 0, skipped: 1 });
assert.deepStrictEqual(r.counts.thumbnail, { seen: 2, created: 1, updated: 0, skipped: 1 });
assert.deepStrictEqual(r.skipped.map(s => s.reason).sort(), ['clips-only recording (ephemeral, never published)', 'external thumbnail url', 'screenshot paste without a file path']);
assert.deepStrictEqual(r.skipped.map(s => s.reason).sort(), ['clips-only recording (ephemeral, never published)', 'external thumbnail url', 'screenshot paste without a file path']);
assert.deepStrictEqual(r.errors, []);
assert.deepStrictEqual(r.errors, []);
- assert.ok(await d.prepare("SELECT value FROM media_settings WHERE key = 'objects.backfill.last_report'").get(), 'report recorded');
assert.deepStrictEqual([vodFile, clipFile, shot, avatar].map(p => fs.statSync(p).mtimeMs), mtimes, 'no bytes touched');
assert.deepStrictEqual([vodFile, clipFile, shot, avatar].map(p => fs.statSync(p).mtimeMs), mtimes, 'no bytes touched');
const v1 = await objOf('vods', 1), v2 = await objOf('vods', 2), v3 = await objOf('vods', 3), v4 = await objOf('vods', 4), v6 = await objOf('vods', 6), v7 = await objOf('vods', 7);
const v1 = await objOf('vods', 1), v2 = await objOf('vods', 2), v3 = await objOf('vods', 3), v4 = await objOf('vods', 4), v6 = await objOf('vods', 6), v7 = await objOf('vods', 7);
@@ -130,18 +128,18 @@ const path = require('path');
@@ -130,18 +128,18 @@ const path = require('path');
const th = await model.getObject(tv.derived_object_id);
const th = await model.getObject(tv.derived_object_id);
assert.deepStrictEqual([th.kind, th.legacy_ref, th.visibility, (await locs(th)).local.state], ['thumbnail', 'legacy:live:thumbnail:vod-1-111.jpg', 'public', 'present']);
assert.deepStrictEqual([th.kind, th.legacy_ref, th.visibility, (await locs(th)).local.state], ['thumbnail', 'legacy:live:thumbnail:vod-1-111.jpg', 'public', 'present']);
assert.ok(await d.prepare("SELECT 1 FROM media_relationships WHERE from_object_id = ? AND relation = 'thumbnail_of' AND to_object_id = ?").get(th.id, v1.id));
assert.ok(await d.prepare("SELECT 1 FROM media_relationships WHERE from_object_id = ? AND relation = 'thumbnail_of' AND to_object_id = ?").get(th.id, v1.id));
- console.log('✅ backfill: one object per row, locations mirror where the bytes are, relationships + thumbnail variants');
+ console.log('✅ projection: one object per row, locations mirror where the bytes are, relationships + thumbnail variants');
// ── Idempotent ──
// ── Idempotent ──
const countAll = async () => (await Promise.all(['media_objects', 'media_locations', 'media_relationships', 'media_variants'].map(async t => (await d.prepare(`SELECT COUNT(*) c FROM ${t}`).get()).c)));
const countAll = async () => (await Promise.all(['media_objects', 'media_locations', 'media_relationships', 'media_variants'].map(async t => (await d.prepare(`SELECT COUNT(*) c FROM ${t}`).get()).c)));
const before = await countAll();
const before = await countAll();
- const again = await backfill();
+ const again = await projectRows();
assert.strictEqual(again.totals.created, 0, 'a re-run creates nothing');
assert.strictEqual(again.totals.created, 0, 'a re-run creates nothing');
assert.strictEqual(again.counts.vod.updated, 6);
assert.strictEqual(again.counts.vod.updated, 6);
assert.deepStrictEqual(await countAll(), before);
assert.deepStrictEqual(await countAll(), before);
assert.strictEqual((await objOf('vods', 1)).id, v1.id, 'ids are stable');
assert.strictEqual((await objOf('vods', 1)).id, v1.id, 'ids are stable');
- assert.strictEqual((await backfill({ onlyMissing: true })).totals.created, 0, 'only-missing finds nothing new');
- console.log('✅ backfill is idempotent');
+ assert.strictEqual((await projectRows({ onlyMissing: true })).totals.created, 0, 'only-missing finds nothing new');
+ console.log('✅ projection is idempotent');
// ── Remote knowledge survives a re-projection ──
// ── Remote knowledge survives a re-projection ──
await model.setLocationState((await locs(v2)).b2.id, { state: 'present', size_bytes: 5 * MB });
await model.setLocationState((await locs(v2)).b2.id, { state: 'present', size_bytes: 5 * MB });
@@ -243,6 +241,6 @@ const path = require('path');
@@ -243,6 +241,6 @@ const path = require('path');
console.log('✅ native soft delete keeps bytes for the retention period; purge skips held objects');
console.log('✅ native soft delete keeps bytes for the retention period; purge skips held objects');
fs.rmSync(tmp, { recursive: true, force: true });
fs.rmSync(tmp, { recursive: true, force: true });
- console.log('objects model + backfill: all checks passed');
+ console.log('objects model + projection: all checks passed');
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
@@ -2,7 +2,7 @@
@@ -2,7 +2,7 @@
// Reconciliation of the object model against the bytes, with a fake B2/R2 provider: read-only by
// Reconciliation of the object model against the bytes, with a fake B2/R2 provider: read-only by
// default, --verify records what the HEADs found; it detects missing canonical copies, lost local
// default, --verify records what the HEADs found; it detects missing canonical copies, lost local
// files, size/hash mismatches, orphan locations, deleted-but-still-served objects and unprojected
// files, size/hash mismatches, orphan locations, deleted-but-still-served objects and unprojected
-// rows. Plus the three scripts end to end (backfill --dry-run, reconcile, invariant).
+// rows. Plus test-only projection and the reconcile and invariant scripts end to end.
const assert = require('assert');
const assert = require('assert');
const fs = require('fs');
const fs = require('fs');
const os = require('os');
const os = require('os');
@@ -40,19 +40,19 @@ const crypto = require('crypto');
@@ -40,19 +40,19 @@ const crypto = require('crypto');
write(path.join(env.FILES_PATH, 'live', 'k1-f.bin'), fileBytes);
write(path.join(env.FILES_PATH, 'live', 'k1-f.bin'), fileBytes);
await d.prepare("INSERT INTO files (key, app_id, size, mime, sha256) VALUES ('k1-f.bin', 'live', ?, 'application/octet-stream', ?)").run(fileBytes.length, crypto.createHash('sha256').update(fileBytes).digest('hex'));
await d.prepare("INSERT INTO files (key, app_id, size, mime, sha256) VALUES ('k1-f.bin', 'live', ?, 'application/octet-stream', ?)").run(fileBytes.length, crypto.createHash('sha256').update(fileBytes).digest('hex'));
- // The backfill (server/objects/backfill.js; the service runs it at boot): a dry run reports and writes nothing.
+ // Test-only legacy row projection: a dry run reports and writes nothing.
const { runScript } = require('./helpers/run-script');
const { runScript } = require('./helpers/run-script');
const run = async (script, args = []) => await runScript(path.join(__dirname, '..', 'scripts', script), args, { env });
const run = async (script, args = []) => await runScript(path.join(__dirname, '..', 'scripts', script), args, { env });
- const bf = require('../server/objects/backfill');
- const dry = await bf.backfill({ dryRun: true });
+ const bf = require('./helpers/project-rows');
+ const dry = await bf.projectRows({ dryRun: true });
assert.deepStrictEqual([dry.dry_run, dry.counts.vod.created, dry.counts.file.created], [true, 5, 1]);
assert.deepStrictEqual([dry.dry_run, dry.counts.vod.created, dry.counts.file.created], [true, 5, 1]);
assert.strictEqual((await d.prepare('SELECT COUNT(*) c FROM media_objects').get()).c, 0, 'a dry run wrote nothing');
assert.strictEqual((await d.prepare('SELECT COUNT(*) c FROM media_objects').get()).c, 0, 'a dry run wrote nothing');
- const real = await bf.backfill();
+ const real = await bf.projectRows();
assert.match(bf.summarize(real), /vod 5\+\/0~\/0 skipped/);
assert.match(bf.summarize(real), /vod 5\+\/0~\/0 skipped/);
assert.strictEqual((await d.prepare('SELECT COUNT(*) c FROM media_objects').get()).c, 6);
assert.strictEqual((await d.prepare('SELECT COUNT(*) c FROM media_objects').get()).c, 6);
- console.log('✅ backfill: a dry run reports only; a real run projects');
+ console.log('✅ projection: a dry run reports only; a real run projects');
- fs.unlinkSync(path.join(env.VOD_PATH, 'e.webm')); // vod 5 loses its file after backfill
+ fs.unlinkSync(path.join(env.VOD_PATH, 'e.webm')); // vod 5 loses its file after projection
const objId = async (vodId) => (await d.prepare('SELECT object_id FROM vods WHERE id = ?').get(vodId)).object_id;
const objId = async (vodId) => (await d.prepare('SELECT object_id FROM vods WHERE id = ?').get(vodId)).object_id;
// Anomalies the reconciler must notice:
// Anomalies the reconciler must notice:
await d.prepare("INSERT INTO media_locations (object_id, provider, key, state) VALUES ('med_01JAB2C3D4E5F6G7H8J9K0MNPQ', 'local', '/nowhere', 'present') RETURNING id").run(); // orphan location
await d.prepare("INSERT INTO media_locations (object_id, provider, key, state) VALUES ('med_01JAB2C3D4E5F6G7H8J9K0MNPQ', 'local', '/nowhere', 'present') RETURNING id").run(); // orphan location
@@ -132,6 +132,8 @@ const crypto = require('crypto');
@@ -132,6 +132,8 @@ const crypto = require('crypto');
const { finalizeVod } = require('../server/vod/finalize');
const { finalizeVod } = require('../server/vod/finalize');
await raw.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, is_recording) OVERRIDING SYSTEM VALUE
await raw.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, is_recording) OVERRIDING SYSTEM VALUE
VALUES (20, 'live', 5, 'Ghost', ?, 1, 1) RETURNING id`).run(path.join(process.env.VOD_PATH, 'never-written.webm'));
VALUES (20, 'live', 5, 'Ghost', ?, 1, 1) RETURNING id`).run(path.join(process.env.VOD_PATH, 'never-written.webm'));
+ // A recording row has its object from the start (createVod writes both); this fixture row is projected the same way.
+ await require('../server/objects/model').sync('vod', 20);
failOutboxInserts(true);
failOutboxInserts(true);
await finalizeVod(20);
await finalizeVod(20);
failOutboxInserts(false);
failOutboxInserts(false);