openvibers/openvibe.media Commits
Document the object backfill as a boot-time module and the drift report
Changes
@@@ -497,10 +497,16 @@ Every stored blob is a **media object** (`med_<ULID>`, `media_objects`) with oneexisting route and response is unchanged, and the old write paths keep themodel current. Full reference: **[docs/object-model.md](docs/object-model.md)**.--- **Backfill** — `backfill()` in `server/objects/backfill.js`: one object per-- vod, clip, file, screenshot, avatar and thumbnail; idempotent; never moves-- bytes or calls B2/R2 (remote copies stay `pending`). It is a module, not a-- CLI: the service calls it itself with `onlyMissing: true` 15 s after boot.++- **Backfill** — the row→object projection (`server/objects/backfill.js`), not a++ CLI script: one object per vod, clip, file, screenshot, avatar and thumbnail;++ idempotent; never moves bytes or calls B2/R2 (remote copies stay `pending`). The++ service runs the only-missing form 15 s after boot, for rows with no `object_id`++ yet, and stores its report in `media_settings.objects.backfill.last_report`++ (also surfaced by `/me/ops`). To see whether any row still lacks an object — or++ disagrees with the object on record — run the read-only drift report:++ `node scripts/object-drift-report.js [--app live] [--limit 20] [--json]++ [--out report.json]`. Zero drift across a release is the signal to drop the boot++ backfill ([details](docs/object-model.md#backfill)).- **API** — `/api/v2/:app/objects`: init → `PUT /:id/content` (sha256, size,quota) → `/:id/complete`; `GET /:id`, cursor `GET /`, soft `DELETE /:id`(bytes kept `MEDIA_DELETE_RETENTION_DAYS`), `/:id/restore`, `/:id/download`@@@ -11,7 +11,7 @@ Code: `server/objects/` (`model.js`, `routes.js`, `backfill.js`, `reconcile.js`,`signing.js`, `verify-job.js`, `copy-report.js`, `multipart.js`, `content-type.js`, `readiness.js`, `namespaces.js`,`namespace-routes.js`, `popularity.js`, `tier-policy.js`, `tiering.js`) and the job system in`server/jobs/` (`queue.js`, `worker.js`, `routes.js`, `types.js`, `thumbnail.js`, `invariant-scan.js`,--`derive.js`). Schema: the bottom of `server/db/schema.sql`. Tests: `test/objects-*.test.js`, `test/jobs*.test.js`,++`derive.js`). Schema: `migrations/0001_initial.sql`. Tests: `test/objects-*.test.js`, `test/jobs*.test.js`,`test/r2-eviction-drill.test.js`, `test/object-tiering.test.js`.## Tables@@@ -82,19 +82,20 @@ the same functions.## Backfill`backfill()` in `server/objects/backfill.js` is a module, not a CLI: it is called from code, and the service calls it--itself with `{ onlyMissing: true }` 15 s after boot. It creates one object per vod, clip, file, screenshot/avatar paste++itself with `{ onlyMissing: true }` 15 s after boot. Read-only questions about it are answered by the++[drift report](#drift-report). It creates one object per vod, clip, file, screenshot/avatar pasteand thumbnail, with relationships and thumbnail variants.- **Idempotent.** Objects are keyed by `legacy_ref`, so a second run updates them in place and creates nothing.- **Never moves or deletes bytes, and never calls B2/R2.** A local copy is `present` when the file exists and `missing` when it does not. Remote copies stay `pending` until `reconcile-objects.js --verify` checks them.- **One transaction,** with one savepoint per row: a failing row is reported and leaves nothing half-written. A dry run (`{ dryRun: true }`) rolls everything back and only reports.--- **Report:** counts per kind (seen, created, updated, skipped), location states, skipped rows with a reason, and errors. A real run stores the report in `media_settings` under `objects.backfill.last_report`.++- **Report:** counts per kind (seen, created, updated, skipped), location states, skipped rows with a reason, and errors. A real run stores the report in `media_settings` under `objects.backfill.last_report`; `lastReport()` reads it back and `/me/ops` shows it.- **At boot:** the service runs the `onlyMissing` form itself 15 s after start, for rows with no `object_id`. The first boot after the upgrade therefore projects everything. With object-first writes it should find nothing; it stays until the [drift report](#drift-report) shows zero drift across a release.Skip reasons today: `clips-only recording (ephemeral, never published)`, `screenshot paste without a file path`, `external thumbnail url`.--The backfill's own transaction holds row and table locks for its duration, typically seconds. Prefer the boot backfill; a--manual call should wait until traffic is quiet.++The backfill's own transaction holds row and table locks for its duration, typically seconds, so it runs only++where the service controls it: the boot backfill.### Keeping the model current: object-first writes@@@ -170,10 +171,10 @@ user ids. An object gets it in one of three ways:fills their objects. Upload and projection paths never wait on Network. When nothing is missing, arun asks Network nothing. An owner Network does not know (a Live account not linked to Network) stays`NULL` until a later run finds it.--- **From the one-off backfill** `scripts/backfill-owner-subject.js`, for everything that existed before-- the job.++- **From the one-off backfill** that existed before the job. That script is retired; the job above fills++ what is left of it, so nothing needs running by hand.--Rules for both job and backfill:++Rules for both job and historical backfill:- A non-null `owner_subject` is never overwritten. Every write is guarded by `owner_subject IS NULL`and by the owner it was resolved for.@@@ -185,26 +186,9 @@ Rules for both job and backfill:`identity.subject.resolve`, audience `openvibe.network`). There is no fallback credential: when Networkhas not granted it, the pass fails and the job retries.--```--node scripts/backfill-owner-subject.js [--batch 500] [--json] # dry run: counts per tenant--node scripts/backfill-owner-subject.js --apply --backup <file.json> [--batch 500] # fill--node scripts/backfill-owner-subject.js --rollback <file.json> [--apply] # undo (dry without --apply)--```----- **The dry run** (the default) opens the database read-only and prints per tenant: objects, already-- set, to fill, unresolvable, unsupported tenant and no owner.--- **`--apply --backup <file.json>`** writes the rollback file `<file.json>` (0600) with the rows it is-- about to change, and fills them in `--batch` transactions. The database's own safety net is-- PostgreSQL's point-in-time recovery (pgBackRest), not an online copy. Finally it rewrites the file to-- list exactly the rows it changed. It refuses an existing backup name, and a re-run fills only what is-- still missing.--- **`--rollback <file.json> --apply`** sets `owner_subject` back to `NULL` on each listed row that-- still carries the subject the backfill wrote for the same owner. Rows changed since are counted and-- left alone. Without `--apply` it only reports, which also verifies an applied backfill. Stop the job-- first (`MEDIA_OWNER_SUBJECT_SYNC=0`, restart), or its next run fills the rows again.----The script opens the database directly, not through `server/db/database.js`, so none of the service's--boot work runs (for example, marking in-progress clip cuts failed).++To stop or pace the job: `MEDIA_OWNER_SUBJECT_SYNC=0` turns it off, `MEDIA_OWNER_SUBJECT_INTERVAL_MIN`++(10) sets the interval. There is no rollback command; PostgreSQL's point-in-time recovery (pgBackRest) is++the safety net.## Object API v2
@@ -497,10 +497,16 @@ Every stored blob is a **media object** (`med_<ULID>`, `media_objects`) with one
@@ -497,10 +497,16 @@ Every stored blob is a **media object** (`med_<ULID>`, `media_objects`) with one
existing route and response is unchanged, and the old write paths keep the
existing route and response is unchanged, and the old write paths keep the
model current. Full reference: **[docs/object-model.md](docs/object-model.md)**.
model current. Full reference: **[docs/object-model.md](docs/object-model.md)**.
-- **Backfill** — `backfill()` in `server/objects/backfill.js`: one object per
- vod, clip, file, screenshot, avatar and thumbnail; idempotent; never moves
- bytes or calls B2/R2 (remote copies stay `pending`). It is a module, not a
- CLI: the service calls it itself with `onlyMissing: true` 15 s after boot.
+- **Backfill** — the row→object projection (`server/objects/backfill.js`), not a
+ CLI script: one object per vod, clip, file, screenshot, avatar and thumbnail;
+ idempotent; never moves bytes or calls B2/R2 (remote copies stay `pending`). The
+ service runs the only-missing form 15 s after boot, for rows with no `object_id`
+ yet, and stores its report in `media_settings.objects.backfill.last_report`
+ (also surfaced by `/me/ops`). To see whether any row still lacks an object — or
+ disagrees with the object on record — run the read-only drift report:
+ `node scripts/object-drift-report.js [--app live] [--limit 20] [--json]
+ [--out report.json]`. Zero drift across a release is the signal to drop the boot
+ backfill ([details](docs/object-model.md#backfill)).
- **API** — `/api/v2/:app/objects`: init → `PUT /:id/content` (sha256, size,
- **API** — `/api/v2/:app/objects`: init → `PUT /:id/content` (sha256, size,
quota) → `/:id/complete`; `GET /:id`, cursor `GET /`, soft `DELETE /:id`
quota) → `/:id/complete`; `GET /:id`, cursor `GET /`, soft `DELETE /:id`
(bytes kept `MEDIA_DELETE_RETENTION_DAYS`), `/:id/restore`, `/:id/download`
(bytes kept `MEDIA_DELETE_RETENTION_DAYS`), `/:id/restore`, `/:id/download`
@@ -11,7 +11,7 @@ Code: `server/objects/` (`model.js`, `routes.js`, `backfill.js`, `reconcile.js`,
@@ -11,7 +11,7 @@ Code: `server/objects/` (`model.js`, `routes.js`, `backfill.js`, `reconcile.js`,
`signing.js`, `verify-job.js`, `copy-report.js`, `multipart.js`, `content-type.js`, `readiness.js`, `namespaces.js`,
`signing.js`, `verify-job.js`, `copy-report.js`, `multipart.js`, `content-type.js`, `readiness.js`, `namespaces.js`,
`namespace-routes.js`, `popularity.js`, `tier-policy.js`, `tiering.js`) and the job system in
`namespace-routes.js`, `popularity.js`, `tier-policy.js`, `tiering.js`) and the job system in
`server/jobs/` (`queue.js`, `worker.js`, `routes.js`, `types.js`, `thumbnail.js`, `invariant-scan.js`,
`server/jobs/` (`queue.js`, `worker.js`, `routes.js`, `types.js`, `thumbnail.js`, `invariant-scan.js`,
-`derive.js`). Schema: the bottom of `server/db/schema.sql`. Tests: `test/objects-*.test.js`, `test/jobs*.test.js`,
+`derive.js`). Schema: `migrations/0001_initial.sql`. Tests: `test/objects-*.test.js`, `test/jobs*.test.js`,
`test/r2-eviction-drill.test.js`, `test/object-tiering.test.js`.
`test/r2-eviction-drill.test.js`, `test/object-tiering.test.js`.
## Tables
## Tables
@@ -82,19 +82,20 @@ the same functions.
@@ -82,19 +82,20 @@ the same functions.
## Backfill
## Backfill
`backfill()` in `server/objects/backfill.js` is a module, not a CLI: it is called from code, and the service calls it
`backfill()` in `server/objects/backfill.js` is a module, not a CLI: it is called from code, and the service calls it
-itself with `{ onlyMissing: true }` 15 s after boot. It creates one object per vod, clip, file, screenshot/avatar paste
+itself with `{ onlyMissing: true }` 15 s after boot. Read-only questions about it are answered by the
+[drift report](#drift-report). It creates one object per vod, clip, file, screenshot/avatar paste
and thumbnail, with relationships and thumbnail variants.
and thumbnail, with relationships and thumbnail variants.
- **Idempotent.** Objects are keyed by `legacy_ref`, so a second run updates them in place and creates nothing.
- **Idempotent.** Objects are keyed by `legacy_ref`, so a second run updates them in place and creates nothing.
- **Never moves or deletes bytes, and never calls B2/R2.** A local copy is `present` when the file exists and `missing` when it does not. Remote copies stay `pending` until `reconcile-objects.js --verify` checks them.
- **Never moves or deletes bytes, and never calls B2/R2.** A local copy is `present` when the file exists and `missing` when it does not. Remote copies stay `pending` until `reconcile-objects.js --verify` checks them.
- **One transaction,** with one savepoint per row: a failing row is reported and leaves nothing half-written. A dry run (`{ dryRun: true }`) rolls everything back and only reports.
- **One transaction,** with one savepoint per row: a failing row is reported and leaves nothing half-written. A dry run (`{ dryRun: true }`) rolls everything back and only reports.
-- **Report:** counts per kind (seen, created, updated, skipped), location states, skipped rows with a reason, and errors. A real run stores the report in `media_settings` under `objects.backfill.last_report`.
+- **Report:** counts per kind (seen, created, updated, skipped), location states, skipped rows with a reason, and errors. A real run stores the report in `media_settings` under `objects.backfill.last_report`; `lastReport()` reads it back and `/me/ops` shows it.
- **At boot:** the service runs the `onlyMissing` form itself 15 s after start, for rows with no `object_id`. The first boot after the upgrade therefore projects everything. With object-first writes it should find nothing; it stays until the [drift report](#drift-report) shows zero drift across a release.
- **At boot:** the service runs the `onlyMissing` form itself 15 s after start, for rows with no `object_id`. The first boot after the upgrade therefore projects everything. With object-first writes it should find nothing; it stays until the [drift report](#drift-report) shows zero drift across a release.
Skip reasons today: `clips-only recording (ephemeral, never published)`, `screenshot paste without a file path`, `external thumbnail url`.
Skip reasons today: `clips-only recording (ephemeral, never published)`, `screenshot paste without a file path`, `external thumbnail url`.
-The backfill's own transaction holds row and table locks for its duration, typically seconds. Prefer the boot backfill; a
-manual call should wait until traffic is quiet.
+The backfill's own transaction holds row and table locks for its duration, typically seconds, so it runs only
+where the service controls it: the boot backfill.
### Keeping the model current: object-first writes
### Keeping the model current: object-first writes
@@ -170,10 +171,10 @@ user ids. An object gets it in one of three ways:
@@ -170,10 +171,10 @@ user ids. An object gets it in one of three ways:
fills their objects. Upload and projection paths never wait on Network. When nothing is missing, a
fills their objects. Upload and projection paths never wait on Network. When nothing is missing, a
run asks Network nothing. An owner Network does not know (a Live account not linked to Network) stays
run asks Network nothing. An owner Network does not know (a Live account not linked to Network) stays
`NULL` until a later run finds it.
`NULL` until a later run finds it.
-- **From the one-off backfill** `scripts/backfill-owner-subject.js`, for everything that existed before
- the job.
+- **From the one-off backfill** that existed before the job. That script is retired; the job above fills
+ what is left of it, so nothing needs running by hand.
-Rules for both job and backfill:
+Rules for both job and historical backfill:
- A non-null `owner_subject` is never overwritten. Every write is guarded by `owner_subject IS NULL`
- A non-null `owner_subject` is never overwritten. Every write is guarded by `owner_subject IS NULL`
and by the owner it was resolved for.
and by the owner it was resolved for.
@@ -185,26 +186,9 @@ Rules for both job and backfill:
@@ -185,26 +186,9 @@ Rules for both job and backfill:
`identity.subject.resolve`, audience `openvibe.network`). There is no fallback credential: when Network
`identity.subject.resolve`, audience `openvibe.network`). There is no fallback credential: when Network
has not granted it, the pass fails and the job retries.
has not granted it, the pass fails and the job retries.
-```
-node scripts/backfill-owner-subject.js [--batch 500] [--json] # dry run: counts per tenant
-node scripts/backfill-owner-subject.js --apply --backup <file.json> [--batch 500] # fill
-node scripts/backfill-owner-subject.js --rollback <file.json> [--apply] # undo (dry without --apply)
-```
-
-- **The dry run** (the default) opens the database read-only and prints per tenant: objects, already
- set, to fill, unresolvable, unsupported tenant and no owner.
-- **`--apply --backup <file.json>`** writes the rollback file `<file.json>` (0600) with the rows it is
- about to change, and fills them in `--batch` transactions. The database's own safety net is
- PostgreSQL's point-in-time recovery (pgBackRest), not an online copy. Finally it rewrites the file to
- list exactly the rows it changed. It refuses an existing backup name, and a re-run fills only what is
- still missing.
-- **`--rollback <file.json> --apply`** sets `owner_subject` back to `NULL` on each listed row that
- still carries the subject the backfill wrote for the same owner. Rows changed since are counted and
- left alone. Without `--apply` it only reports, which also verifies an applied backfill. Stop the job
- first (`MEDIA_OWNER_SUBJECT_SYNC=0`, restart), or its next run fills the rows again.
-
-The script opens the database directly, not through `server/db/database.js`, so none of the service's
-boot work runs (for example, marking in-progress clip cuts failed).
+To stop or pace the job: `MEDIA_OWNER_SUBJECT_SYNC=0` turns it off, `MEDIA_OWNER_SUBJECT_INTERVAL_MIN`
+(10) sets the interval. There is no rollback command; PostgreSQL's point-in-time recovery (pgBackRest) is
+the safety net.
## Object API v2
## Object API v2