openvibers/openvibe.media Commits
Namespaces and grants (roadmap WS-G task 2): every namespace is a media_namespaces row (tenant, owner, policy, quota_bytes/quota_objects, used/reserved snapshot). A tenant's root is its app id, or app.<project_id> / app.<project_id>.sandbox for a developer project; children (<root>.<segment>, up to 3, MEDIA_NAMESPACE_MAX_CHILDREN per tenant) are made by the first init that names one. Network tokens are checked per namespace for five verbs (server/auth.js VERBS): read, list (media.object.list), write, delete (media.object.delete), transform (media.derivative.create); the older ids keep granting the newer verbs (read lists, upload deletes and transforms) unless the namespace's policy says strict_verbs, and an app token's legacy ns [prj_…] means app.<project_id>.*. Quotas cover a namespace's subtree up to the tenant (bytes: the root inherits apps.quota_bytes): init reserves the declared size in the same transaction as the object (413 media.quota.exceeded / media.quota.objects_exceeded), the stored bytes re-reserve, complete re-checks the real size and settles, a multipart abort or expiry releases, and the hourly sweep fails uploads whose reservation passed MEDIA_UPLOAD_RESERVATION_HOURS (72) and frees their bytes. Policy: kinds, visibilities, max_object_bytes. Owner read GET /api/v2/:app/namespaces[/:namespace]; operator scripts/namespaces.js (list, --set quotas and policy). Backfill on open (idempotent): developer-project objects move to app.<project_id>[.sandbox], every tenant and named namespace gets a row, and once, uploads in progress get a reservation for the full window. Test: namespaces.test.js (new); docs/object-model.md, README
Changes
@@@ -100,15 +100,18 @@ Credential types on `/api/v1/:app/...`:`allowed_origins` (CORS is reflected for allow-listed origins only).3. **Network principal token** (`sub svc:…`, `app:…`, `mod:…`; RS256, audience `openvibe.media`) —-- accepted only on routes that name a capability: files upload/delete and every objects v2 write-- (`media.object.upload`), files list/meta and objects v2 reads (`media.object.read`), and only for-- the `:app` namespaces in the token's `ns`.++ accepted only on routes that name a verb, each checked for the namespace it works in against the++ token's `ns`: read (`media.object.read`), list (`media.object.list`, or read), write++ (`media.object.upload`), delete (`media.object.delete`, or upload) and transform++ (`media.derivative.create`, or upload). The tenant's root namespace is `:app`; children, quotas++ and policy are in [docs/object-model.md](docs/object-model.md#namespaces-grants-and-quotas).### Developer-project tenants (ADR-014)A developer app from OpenVibe.Network (token `sub app:app_<ULID>`, `project_id prj_<ULID>`,--`env sandbox|production`, `ns [project_id]`) holding `media.object.upload` / `media.object.read`--gets a tenant keyed by its project id, **created on first use**. The URL always names the project++`env sandbox|production`, `ns [project_id, app.<project_id>.*]`; an older `ns [project_id]` means the++same) holding media verbs gets a tenant keyed by its project id, **created on first use**. Its++namespaces are `app.<project_id>` (production) and `app.<project_id>.sandbox`, with children below. The URL always names the project(`/api/v1/prj_<ULID>/files`, `/api/v2/prj_<ULID>/objects`, so openvibe-sdk's`createMediaClient({ app: projectId })` works); the token's `env` picks one of two separate tenants:@@@ -7,7 +7,8 @@ projection over an object through its `object_id` column. Every existing URL andunchanged. New code talks to objects through the [v2 API](#object-api-v2).Code: `server/objects/` (`model.js`, `routes.js`, `backfill.js`, `reconcile.js`, `invariant.js`,--`signing.js`, `verify-job.js`, `copy-report.js`, `multipart.js`, `content-type.js`, `readiness.js`) and the job system in++`signing.js`, `verify-job.js`, `copy-report.js`, `multipart.js`, `content-type.js`, `readiness.js`, `namespaces.js`,++`namespace-routes.js`) and the job system in`server/jobs/` (`queue.js`, `worker.js`, `routes.js`, `types.js`, `thumbnail.js`, `invariant-scan.js`,`derive.js`). Schema: the bottom of `server/db/schema.sql`. Tests: `test/objects-*.test.js`, `test/jobs*.test.js`,`test/r2-eviction-drill.test.js`.@@@ -16,13 +17,15 @@ Code: `server/objects/` (`model.js`, `routes.js`, `backfill.js`, `reconcile.js`,| table | what it holds ||---|---|--| `media_objects` | `id` (med_…), `app_id` (tenant), `namespace` (capability namespace; = app today), `kind` (`vod` `clip` `file` `thumbnail` `screenshot` `avatar` `asset`), `owner_subject` (usr_… when known), `owner_app` + `owner_user_id` (legacy owner in the app's own user-id space), `visibility` (`public` `unlisted` `private`), `lifecycle_status` (`uploading` `ready` `failed` `archived` `deleted`), `mime_type`, `size_bytes`, `content_hash` (sha256), `canonical_provider` + `canonical_key`, `legacy_ref`, `metadata` (JSON), `created_at` / `updated_at` / `deleted_at` |++| `media_objects` | `id` (med_…), `app_id` (tenant), `namespace` (a [`media_namespaces`](#namespaces-grants-and-quotas) row: the tenant's root or a child of it), `kind` (`vod` `clip` `file` `thumbnail` `screenshot` `avatar` `asset`), `owner_subject` (usr_… when known), `owner_app` + `owner_user_id` (legacy owner in the app's own user-id space), `visibility` (`public` `unlisted` `private`), `lifecycle_status` (`uploading` `ready` `failed` `archived` `deleted`), `mime_type`, `size_bytes`, `content_hash` (sha256), `canonical_provider` + `canonical_key`, `legacy_ref`, `metadata` (JSON), `created_at` / `updated_at` / `deleted_at` || `media_locations` | one row per provider copy (`UNIQUE(object_id, provider)`): `provider` (`local` `b2` `r2`), `bucket`, `key` (absolute path for local, object key for B2/R2), `storage_class` (`hot` local, `cold` B2, `cache` R2), `state`, `checksum`, `size_bytes`, `verified_at` || `media_relationships` | `from_object_id`, `relation`, `to_object_id`, `metadata`. In use: `clip_of` (clip to vod, with start/end), `thumbnail_of` (thumbnail to vod/clip). Reserved: `derived_from`, `screenshot_of` || `media_variants` | `object_id`, `variant_name`, `derived_object_id`, `recipe`. In use: `thumbnail` || `media_jobs` | derivative/maintenance jobs: `id` (mjob_…), `app_id`, `object_id`, `job_type`, `status`, `idempotency_key`, `params`, `result`, `attempts`/`max_attempts`, `run_after`, `lease_until`, `lease_token`, `checkpoint`, `error`/`error_code`, `cancel_requested`, `created_by`, `decided_by`. See [Jobs](#jobs) || `media_uploads`, `media_upload_parts` | multipart upload sessions and the parts received (size, sha256). See [Multipart uploads](#multipart-uploads) || `media_holds` | retention holds (with a staff `note`), see [Holds](#retention-holds) |++| `media_namespaces` | every namespace: `namespace`, `app_id` (tenant), `parent`, `owner` (`service:<app>` or `project:<prj_…>`), `policy` (JSON), `quota_bytes`, `quota_objects`, and the usage snapshot `used_bytes`, `used_objects`, `reserved_bytes`, `reserved_objects`, `reconciled_at`. See [Namespaces](#namespaces-grants-and-quotas) |++| `media_quota_reservations` | quota held by an upload in progress: `object_id`, `app_id`, `namespace`, `bytes`, `expires_at` || `media_invariant_violations` | public playback objects over the size policy, see [Invariant](#public-object-size-invariant) |Location `state`:@@@ -206,7 +209,7 @@ All routes live under `/api/v2/:app/objects`.**Auth.** One of:- the app's API key;--- a Network service token that grants `media.object.upload` (writes) or `media.object.read` (reads) for the namespace `:app`. `tenantAuth({ capability })` enforces this, as it does on v1.++- a Network service token that holds the route's [verb](#namespaces-grants-and-quotas) for the object's namespace: write `media.object.upload`, read `media.object.read`, list `media.object.list`, delete (and restore) `media.object.delete`. The older ids still grant the newer verbs (read lists, upload deletes), unless the namespace's policy is `strict_verbs`. The tenant's root namespace is `:app`.- for `:app = prj_<ULID>` (a developer-project tenant, ADR-014): only that project's app tokens, production or sandbox. The token's `env` selects the tenant (`prj_<ULID>` or `prj_<ULID>-sandbox`), and upload URLs always name the project. Sandbox objects never get a `public_url`; `/download` signs them and `/o/:id` needs the signature, whatever their visibility. Retention holds are not available to app tokens. See the README's "Developer-project tenants".**Owner.**@@@ -218,7 +221,7 @@ All routes live under `/api/v2/:app/objects`.| method | path | notes ||---|---|---|--| POST | `/` | init. `{ kind, visibility (default private), mime_type, size_bytes, filename, content_hash (expected sha256), metadata (≤ 16 KB), user_id, upload_ttl (60-86400 s), multipart, part_size }` returns 201 `{ id, object, upload: { method: 'PUT', url, token, expires_at, max_bytes, content_type, complete_url, multipart_url } }`. `url` is a [presigned PUT URL](#presigned-uploads). With `multipart: true` (required above `MEDIA_OBJECT_MAX_MB`): `method: 'multipart'`, `url: null`, and `upload.multipart` is the new [session](#multipart-uploads). Checks at init: `size_bytes` against `MEDIA_OBJECT_MAX_MB` (single part) or `MEDIA_MULTIPART_MAX_MB` (413), the quota including reservations (413 `media.quota.exceeded`), the public-size invariant (422) and the [content type](#content-types) for the kind (415 `media.object.unsupported_type`) |++| POST | `/` | init. `{ namespace (default the tenant's root; a name below it, full or relative), kind, visibility (default private), mime_type, size_bytes, filename, content_hash (expected sha256), metadata (≤ 16 KB), user_id, upload_ttl (60-86400 s), multipart, part_size }` returns 201 `{ id, object, upload: { method: 'PUT', url, token, expires_at, max_bytes, content_type, complete_url, multipart_url } }`. `url` is a [presigned PUT URL](#presigned-uploads). With `multipart: true` (required above `MEDIA_OBJECT_MAX_MB`): `method: 'multipart'`, `url: null`, and `upload.multipart` is the new [session](#multipart-uploads). Checks at init: `size_bytes` against `MEDIA_OBJECT_MAX_MB` (single part) or `MEDIA_MULTIPART_MAX_MB` (413), the namespace's policy (422 `media.namespace.policy_denied`, 413 `media.object.too_large`), every quota from the namespace up to the tenant including reservations (413 `media.quota.exceeded` / `media.quota.objects_exceeded`; the declared size is then reserved), the public-size invariant (422) and the [content type](#content-types) for the kind (415 `media.object.unsupported_type`) || POST | `/:id/upload-url` | a fresh presigned single-PUT URL for an uploading object (`{ ttl }` 60-86400 s) || PUT | `/:id/content` | the bytes, single part. Auth is the upload token (`?token=` from init, or `X-Upload-Token`) or the usual credential. Streams to disk computing sha256. Refuses more than the declared size or the limit (413; an object declared above `MEDIA_OBJECT_MAX_MB` goes multipart), a size mismatch (400), quota overrun (413), a type that does not suit the kind (415) and a PUT while a multipart session is open (409). Stored at `OBJECTS_PATH/<app>/<id>` with a verified local location. Mounted ahead of the JSON body parser, so any Content-Type is taken as raw bytes; the Content-Type becomes `mime_type` when init set none. Can be repeated while the object is `uploading` || POST | `/:id/complete` | checks the expected hash (422 `media.object.hash_mismatch`), the invariant, the quota and the bytes against the type (415 `media.object.content_mismatch`). Sets `ready` and sends the `media.object.uploaded` webhook. Also accepts the upload token |@@@ -226,9 +229,9 @@ All routes live under `/api/v2/:app/objects`.| GET | `/:id/multipart/:uploadId` | the session: `parts` received (number, size, sha256), `missing`, `received_bytes`, `expires_at` || PUT | `/:id/multipart/:uploadId/parts/:n` | one part, raw bytes of exactly its size; optional `X-Content-SHA256` || POST | `/:id/multipart/:uploadId/complete` | `{ content_hash, parts: [{ part_number, sha256 }] }` (both optional): assembles, then the checks of `/complete` |--| DELETE | `/:id/multipart/:uploadId` | abort: parts deleted, the object stays `uploading` |++| DELETE | `/:id/multipart/:uploadId` | abort: parts deleted and the quota reservation released, the object stays `uploading` (a new session or upload URL reserves again) || GET | `/:id` | metadata: providers and states of each copy, never paths or keys, and its [readiness](#readiness) |--| GET | `/` | cursor list, newest first. `?limit (≤200)&cursor&kind&visibility&status&owner (usr_…)&user_id&include_deleted` returns `{ objects, next_cursor }` |++| GET | `/` | cursor list, newest first. `?limit (≤200)&cursor&namespace (and below)&kind&visibility&status&owner (usr_…)&user_id&include_deleted` returns `{ objects, next_cursor }`. A Network token sees only the namespaces it may list || DELETE | `/:id` | soft delete: `lifecycle_status = deleted`, and the bytes are kept for `MEDIA_DELETE_RETENTION_DAYS` (default 30). 409 `media.object.held` under a hold. 409 `media.object.legacy_managed` for projected objects, which are deleted through their v1 route || POST | `/:id/restore` | undo a soft delete within the retention period (410 once purged) || GET | `/:id/download` | public or unlisted: 302 to the public location (the inherited URL for projected objects such as `/v/:id`, `/c/:id`, `/f/:key`, `/t/:name`, `/p/:slug/screenshot`; otherwise `/o/:id`). Private: `{ url, expires_at }` signed for `?ttl` seconds (30 to 3600, default `MEDIA_SIGNED_URL_TTL_S` = 300), or a 302 with `?redirect=1`. `?format=json` always answers JSON. 410 when deleted |@@@ -311,14 +314,90 @@ Code: `server/objects/content-type.js`.- Everything else is served as an attachment with `nosniff`, so its declared type is taken as given.The declared type wins over the PUT's `Content-Type` (declared at init: the PUT header is ignored).--**Quota.** `apps.quota_bytes` is compared against v1 file bytes plus native objects that are--`uploading` (their declared size counts as a reservation) or `ready`. Soft-deleted objects do not--count, except in developer-project tenants, where they count until their bytes are purged. The v1 files route still counts only v1 files, so its response is unchanged.++**Quota.** Per namespace, up to the tenant: see [Namespaces, grants and quotas](#namespaces-grants-and-quotas).**Purge.** Every hour, the service removes the bytes of native objects whose retention period haspassed and that have no active hold. The object row stays `deleted` with `metadata.purged_at`.Projected objects are never purged by this job.++## Namespaces, grants and quotas++++Roadmap WS-G task 2. Code: `server/objects/namespaces.js` (rows, policy, quotas, reservations),++`server/auth.js` (`VERBS`, `tenantAuth({ verb, namespaced })`, `namespaceGrant`),++`server/objects/namespace-routes.js`, `scripts/namespaces.js`. Test: `test/namespaces.test.js`.++++**Namespaces are rows.** Every object's `namespace` is a `media_namespaces` row. A tenant has one root:++++| tenant | root namespace |++|---|---|++| first-party (`live`, `community`, `tools`, …) | its app id |++| developer project, production (`prj_<ULID>`) | `app.<project_id>` |++| developer project, sandbox (`prj_<ULID>-sandbox`) | `app.<project_id>.sandbox` |++++Children sit below a root (`<root>.<segment>`, up to three segments of `a-z 0-9 _ -`). Init creates one++the first time it is named (`{ namespace: "avatars" }` or the full name), at most++`MEDIA_NAMESPACE_MAX_CHILDREN` per tenant (413 `media.quota.namespaces_exceeded`). `owner` is++`service:<app_id>` or `project:<prj_…>`. v1 files, projected rows and job outputs live in the root++(outputs in their source's namespace).++++**Grants: five verbs, one capability each, per namespace.** Network issues them (service principals'++`principal_grants`, developer apps' grants) and the token's `ns` names the namespaces.++++| verb | capability | also granted by (tokens issued before the split) | routes |++|---|---|---|---|++| read | `media.object.read` | | object metadata, download, holds, a multipart session, a job, one namespace, v1 file meta |++| list | `media.object.list` | `media.object.read` | object list, job list, namespace list, v1 file list |++| write | `media.object.upload` | | init, upload URL, content, complete, multipart, v1 file upload |++| delete | `media.object.delete` | `media.object.upload` | object delete and restore, v1 file delete |++| transform | `media.derivative.create` | `media.object.upload` | job create, approve, cancel (the job's object's namespace; the root for a tenant-wide job) |++++A namespace whose policy sets `strict_verbs: true` accepts only the verb's own capability. A grant++of `ns: ["community"]` is the root alone; `community.*` covers its children; a token for one child++(`community.avatars`) reaches the tenant but only that namespace. A developer app token's `ns` entry++equal to its `project_id` (what Network issued before) means `app.<project_id>.*`; Network now issues++both. The app key and upload/session tokens are unrestricted within their tenant or object.++++**Policy** (JSON, set by the operator; children inherit it key by key): `kinds`, `visibilities`++(422 `media.namespace.policy_denied` at init), `max_object_bytes` (413 `media.object.too_large`),++`strict_verbs`.++++**Quotas.** `quota_bytes` and `quota_objects` limit a namespace's subtree. `quota_bytes` NULL++inherits: the root takes its tenant's `apps.quota_bytes`, a child has no limit of its own; 0 is no++limit. Usage counts ready native objects (in a developer project also soft-deleted ones until their++bytes are purged), the tenant's v1 files (at the root) and reservations:++++1. **Init** checks every limit from the namespace up to the root with the declared size and one more++ object, and reserves the declared size (`media_quota_reservations`), in one transaction with the++ object row.++2. **The bytes** (PUT, or a new multipart session or upload URL) are checked again and replace the++ reservation's size.++3. **Complete** checks the real size against everything else and settles the reservation: the ready++ object counts from then on. The snapshot is reconciled.++4. **Abort** of a multipart session (or its expiry) releases the reservation down to bytes still stored.++ Deleting an upload settles it.++5. **Expiry.** A reservation lasts `MEDIA_UPLOAD_RESERVATION_HOURS` (72) after the upload's last step.++ The hourly sweep then marks the upload `failed` (`metadata.failure: upload_expired`), deletes its++ stored bytes and drops the reservation, unless a multipart session is still open.++++Refusals are 413 problems: `media.quota.exceeded` (`namespace`, `quota_bytes`, `used_bytes`,++`reserved_bytes`) or `media.quota.objects_exceeded` (`namespace`, `quota_objects`, `used_objects`,++`reserved_objects`). The v1 file upload keeps its JSON shape and adds `code`. Split and remux check++the source namespace's quotas. The `used_*`/`reserved_*` columns are a snapshot: checks always count++from the rows. It is refreshed at complete, delete and restore, when read, and hourly.++++**Reads.** `GET /api/v2/:app/namespaces` lists the tenant's namespaces the caller may list, and++`GET /api/v2/:app/namespaces/:namespace` reads one (full or relative name): owner, own and++effective policy, quota (`bytes`, `objects`, `bytes_from: tenant|namespace`) and usage, counted when++read. The operator lists every tenant's namespaces and sets quotas and policy with++`node scripts/namespaces.js [--app <tenant>] [--json]` and++`--set <namespace> [--quota-bytes <n>|inherit] [--quota-objects <n>|none] [--policy '<json>'] [--dry-run]`.++++**Backfill** (every open, idempotent; `server/db/database.js` `migrateNamespaces`): a developer++project's objects move from the tenant id (`prj_…`, `prj_…-sandbox`) to `app.<project_id>[.sandbox]`;++every tenant gets its root row and every namespace an object names gets a row. Once++(`media_settings` `namespaces.reservations_seeded`), uploads already in progress get a reservation of++their declared size for the full window from that open.++## ReadinessCode: `server/objects/readiness.js`. Test: `test/readiness.test.js`.@@@ -580,8 +659,8 @@ may be queued or running (429 `media.job.too_many`).**Cancellation, by the owner.** A proposed or queued job is cancelled at once. A running job gets`cancel_requested`; the worker aborts it (its ffmpeg is killed) and it ends `cancelled`. Finished--jobs answer 409. The owner is the tenant (app key, service token for the namespace, or the project's--app token); a caller acting for one of the app's users (`X-OV-User-Id`) sees and decides only the jobs++jobs answer 409. The owner is the tenant (app key, a service token or the project's app token holding++the `transform` [verb](#namespaces-grants-and-quotas) for the job's object's namespace); a caller acting for one of the app's users (`X-OV-User-Id`) sees and decides only the jobsit created and the jobs on objects it owns.**Worker.** In-process, polling every `MEDIA_JOBS_POLL_MS`, in three lanes: `light`@@@ -731,6 +810,8 @@ Not run against production yet: it needs the owner's go-ahead.| `MEDIA_MULTIPART_MIN_PART_MB` / `_MAX_PART_MB` / `_PART_MB` | 5 / 256 / 64 | part size bounds and default || `MEDIA_MULTIPART_TTL_HOURS` | 24 | an unfinished multipart session is purged after this || `MEDIA_UPLOAD_MIN_FREE_MB` | 10240 | free disk kept in reserve by multipart uploads and split/remux jobs |++| `MEDIA_UPLOAD_RESERVATION_HOURS` | 72 | an upload's quota reservation lasts this long after its last step; then the hourly sweep fails it and frees its bytes |++| `MEDIA_NAMESPACE_MAX_CHILDREN` | 100 | namespaces a tenant may have below its root || `MEDIA_JOBS_ENABLED` | on | `0` stops the job worker || `MEDIA_JOBS_POLL_MS` | 5000 | worker poll interval || `MEDIA_JOBS_LIGHT_CONCURRENCY` / `_HEAVY_CONCURRENCY` / `_FINALIZE_CONCURRENCY` | 2 / 1 / 1 | jobs per lane |nnew file mode 100644@@@ -0,0 +1,77 @@++#!/usr/bin/env node++'use strict';++/**++ * Namespaces for the operator (roadmap WS-G task 2; server/objects/namespaces.js). Lists every++ * namespace with its owner, quota and usage, and sets a namespace's quotas and policy.++ *++ * node scripts/namespaces.js [--app <tenant>] [--json] list (usage counted from the rows now)++ * node scripts/namespaces.js --set <namespace> [--quota-bytes <n>|inherit] [--quota-objects <n>|none]++ * [--policy '<json>'] [--dry-run]++ *++ * --quota-bytes: 0 is no limit; `inherit` clears it (a root then takes its tenant's apps.quota_bytes,++ * a child has none of its own). --quota-objects: 0 or `none` is no limit. --policy replaces the++ * namespace's own policy (kinds, visibilities, max_object_bytes, strict_verbs; children inherit it,++ * key by key). Listing refreshes the usage snapshot; nothing else is written without --set.++ */++const args = process.argv.slice(2);++const has = (name) => args.includes(`--${name}`);++const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] != null && !args[i + 1].startsWith('--') ? args[i + 1] : dflt; };++++function fail(msg) { console.error(`namespaces: ${msg}`); process.exit(1); }++const human = (b) => (b == null ? '-' : b >= 1024 ** 3 ? `${(b / 1024 ** 3).toFixed(2)} GB` : b >= 1024 ** 2 ? `${(b / 1024 ** 2).toFixed(1)} MB` : `${b} B`);++++try {++ const db = require('../server/db/database');++ const namespaces = require('../server/objects/namespaces');++++ if (arg('set')) {++ const row = namespaces.get(arg('set'));++ if (!row) fail(`no namespace ${arg('set')}`);++ const sets = {};++ if (arg('quota-bytes') != null) {++ const v = arg('quota-bytes');++ if (v === 'inherit') sets.quota_bytes = null;++ else if (/^\d+$/.test(v)) sets.quota_bytes = Number(v);++ else fail('--quota-bytes takes a whole number of bytes or `inherit`');++ }++ if (arg('quota-objects') != null) {++ const v = arg('quota-objects');++ if (v === 'none') sets.quota_objects = null;++ else if (/^\d+$/.test(v)) sets.quota_objects = Number(v) || null;++ else fail('--quota-objects takes a whole number or `none`');++ }++ if (arg('policy') != null) {++ let p;++ try { p = JSON.parse(arg('policy')); } catch { fail('--policy must be JSON'); }++ const v = namespaces.validatePolicy(p);++ if (v.error) fail(v.error);++ sets.policy = JSON.stringify(v.policy);++ }++ if (!Object.keys(sets).length) fail('--set needs --quota-bytes, --quota-objects or --policy');++ console.log(`${row.namespace}: ${Object.entries(sets).map(([k, v]) => `${k} ${row[k] ?? 'NULL'} -> ${v ?? 'NULL'}`).join(', ')}${has('dry-run') ? ' (dry run)' : ''}`);++ if (!has('dry-run')) {++ db.run(`UPDATE media_namespaces SET ${Object.keys(sets).map(k => `${k} = ?`).join(', ')}, updated_at = CURRENT_TIMESTAMP WHERE namespace = ?`, [...Object.values(sets), row.namespace]);++ }++ process.exit(0);++ }++++ const rows = arg('app') ? namespaces.listForTenant(arg('app')) : db.all('SELECT * FROM media_namespaces ORDER BY app_id, namespace');++ const apps = new Map();++ const out = rows.map((row) => {++ if (!apps.has(row.app_id)) apps.set(row.app_id, db.getApp(row.app_id) || { app_id: row.app_id });++ const app = apps.get(row.app_id);++ return { app_id: row.app_id, ...namespaces.publicShape(app, row, namespaces.reconcile(app, row.namespace)) };++ });++ if (has('json')) { console.log(JSON.stringify({ namespaces: out }, null, 2)); process.exit(0); }++ for (const n of out) {++ const u = n.usage;++ console.log(`${n.namespace} [${n.app_id}${n.root ? ', root' : ''}] ${n.owner}`++ + ` used ${human(u.used_bytes)} / ${n.quota.bytes ? human(n.quota.bytes) : 'no limit'}${n.quota.bytes_from === 'tenant' ? ' (tenant)' : ''}`++ + `, ${u.used_objects} object(s)${n.quota.objects ? ` / ${n.quota.objects}` : ''}`++ + (u.reserved_objects ? `, ${u.reserved_objects} upload(s) reserving ${human(u.reserved_bytes)}` : '')++ + (Object.keys(n.policy).length ? ` policy ${JSON.stringify(n.policy)}` : ''));++ }++ if (!out.length) console.log('no namespaces');++} catch (err) {++ fail(err.message);++}@@@ -12,6 +12,9 @@** App seeding: MEDIA_APPS_SEED (JSON array) and/or MEDIA_APP_KEYS ("id:key,…")* are upserted on boot; keys are stored as sha256 hashes.++ *++ * Network principal tokens (service and developer app tokens) are checked per namespace for one of++ * five verbs: read, list, write, delete, transform (VERBS below; objects/namespaces.js has the rows).*/'use strict';@@@ -116,23 +119,96 @@ function problem(res, status, code, detail) {return http.sendProblem(res, status, code, { detail });}++// ── Namespace grants: five verbs (roadmap WS-G task 2; docs/object-model.md#namespaces-grants-and-quotas) ──++++/**++ * Each verb is one capability, checked for one namespace against the token's `ns`. The first id is++ * the verb's own; the broader ids after it grant it too, so the tokens Network issued before the++ * verbs were split keep working (upload also deletes and transforms, read also lists). A namespace++ * whose policy sets strict_verbs accepts only the verb's own id.++ */++const VERBS = Object.freeze({++ read: ['media.object.read'],++ list: ['media.object.list', 'media.object.read'],++ write: ['media.object.upload'],++ delete: ['media.object.delete', 'media.object.upload'],++ transform: ['media.derivative.create', 'media.object.upload'],++});++++/**++ * The namespaces a token was granted. A developer app token's own project id (the `ns` Network++ * issued before namespaces were rows) means the whole project, app.<project_id>.* (both its tenants).++ */++function grantedNamespaces(claims) {++ const ns = claims && Array.isArray(claims.ns) ? claims.ns.map(String) : [];++ if (!claims || !isAppPrincipal(claims) || !claims.project_id) return ns;++ return [...new Set(ns.map(n => (n === claims.project_id ? `app.${claims.project_id}.*` : n)))];++}++++/** Does the grant list cover `root` or anything below it (a token for one child reaches its tenant)? */++function touchesTenant(nsList, root) {++ if (capabilities.namespaceAllowed(nsList, root)) return true;++ return nsList.some(n => (n.endsWith('.*') ? n.slice(0, -2) : n).startsWith(`${root}.`));++}++++/**++ * May a token with `claims` do `verb` in `namespace`? { allowed, code, reason }; namespace undefined++ * checks the capability only. The ids are Media's own, so this does not need the installed contracts++ * catalog to know the newer ones (media.object.list, media.object.delete).++ */++function verbAllowed(claims, verb, namespace, { strict = false } = {}) {++ const ids = VERBS[verb];++ if (!ids) return { allowed: false, code: 'capability.unknown', reason: `no verb ${verb}` };++ const held = (strict ? ids.slice(0, 1) : ids).some(id => capabilities.grants(claims && claims.cap, id));++ if (!held) return { allowed: false, code: 'capability.denied', reason: `${ids[0]} not granted${strict ? ` (${namespace} accepts only it)` : ''}` };++ if (namespace !== undefined && !capabilities.namespaceAllowed(grantedNamespaces(claims), namespace)) {++ return { allowed: false, code: 'capability.namespace_denied', reason: `namespace ${namespace} not granted` };++ }++ return { allowed: true, code: null, reason: null };++}++++/**++ * The tenant-level decision in tenantAuth. A `verb` route that is not `namespaced` works in the++ * tenant's root and is decided here in full. A `namespaced` route (objects v2, the job actions, the++ * namespace reads) checks each namespace it touches with namespaceGrant(); here the token must hold++ * the verb and some namespace of this tenant. A `capability` route checks it for the root, as before.++ */++function tenantGrant(claims, { capability, verb, namespaced }, appId, root) {++ if (!verb) return capabilities.check({ ...claims, ns: grantedNamespaces(claims) }, capability, { namespace: root });++ if (!namespaced) return verbAllowed(claims, verb, root, { strict: require('./objects/namespaces').isStrict(appId, root) });++ const c = verbAllowed(claims, verb, undefined);++ if (!c.allowed) return c;++ return touchesTenant(grantedNamespaces(claims), root) ? c : { allowed: false, code: 'capability.namespace_denied', reason: `namespace ${root} not granted` };++}++++/**++ * May this request do `verb` in `namespace` of its tenant? The app key (and an upload or session++ * token, scoped to its object already) may do anything; a Network token as its grant says.++ */++function namespaceGrant(req, verb, namespace) {++ if (!req.grant) return { allowed: true, code: null, reason: null };++ return verbAllowed(req.grant, verb, namespace, { strict: require('./objects/namespaces').isStrict(req.appId, namespace) });++}++// ── Developer-project tenants (roadmap Wave 20, ADR-014) ─────const PROJECT_ID_RE = /^prj_[0-9A-HJKMNP-TV-Z]{26}$/;const isAppPrincipal = (claims) => !!claims && (claims.actor_type === 'app' || /^app:/.test(String(claims.sub || '')));/**-- * An app token on /api/v1/<project_id>/files or /api/v2/<project_id>/objects. The path names the-- * project; the token's env picks the tenant row (production prj_<ULID>, sandbox prj_<ULID>-sandbox),-- * which is created on first use with the default quota. Nothing is created unless the token's-- * project_id is the path's project and the token holds this route's capability for it.++ * An app token on /api/v1/<project_id>/files or /api/v2/<project_id>/(objects|jobs|namespaces). The++ * path names the project; the token's env picks the tenant row (production prj_<ULID>, sandbox++ * prj_<ULID>-sandbox), which is created on first use with the default quota. Its namespaces are++ * app.<project_id> and app.<project_id>.sandbox with their children. Nothing is created unless the++ * token's project_id is the path's project and the token holds this route's verb there.*/--function projectTenant(req, res, next, projectId, claims, capability) {++function projectTenant(req, res, next, projectId, claims, grant) {if (claims.project_id !== projectId) {return problem(res, 403, 'capability.namespace_denied', `this app token belongs to ${claims.project_id || 'no project'}, not ${projectId}`);}const env = claims.env === 'sandbox' ? 'sandbox' : 'production';-- const c = capabilities.check(claims, capability, { namespace: projectId });++ const tenantId = db.projectTenantId(projectId, env);++ const c = tenantGrant(claims, grant, tenantId, db.rootNamespace({ app_id: tenantId, project_id: projectId, env }));if (!c.allowed) return problem(res, 403, c.code, c.reason);let tenant;try {@@@ -147,6 +223,7 @@ function projectTenant(req, res, next, projectId, claims, capability) {req.authType = 'app';req.tenantEnv = env;req.principal = { sub: claims.sub, cap: claims.cap, jti: claims.jti, project_id: projectId, env };++ req.grant = claims;return next();}@@@ -169,6 +246,7 @@ function ensureTokenOnlyApps() {['blog', 'OpenVibe.Blog', 10], // Wave 16 post attachments]) {if (!db.getApp(appId)) db.run("INSERT INTO apps (app_id, name, api_key_hash, quota_bytes) VALUES (?, ?, '', ?)", [appId, name, quotaGb * 1024 ** 3]);++ db.ensureRootNamespace(db.getApp(appId));}}@@@ -217,16 +295,24 @@ function actingUserId(req) {* @param {object} opts* @param {boolean} [opts.allowUser=false] also accept a Network user JWT* (browser endpoints); Origin, when present, must be allow-listed.++ * @param {string} [opts.verb] read | list | write | delete | transform: the route accepts Network++ * tokens that hold the verb (VERBS) for the tenant's root namespace++ * @param {boolean} [opts.namespaced=false] with `verb`: the route checks each namespace it touches++ * (namespaceGrant); a token holding the verb for any namespace of the tenant gets through++ * @param {string} [opts.capability] instead of a verb: one capability, for the root namespace*/--function tenantAuth({ allowUser = false, capability = null } = {}) {++function tenantAuth({ allowUser = false, capability = null, verb = null, namespaced = false } = {}) {++ if (verb && !VERBS[verb]) throw new Error(`tenantAuth: unknown verb ${verb}`);++ const named = verb ? VERBS[verb][0] : capability;++ const grant = { capability, verb, namespaced };return (req, res, next) => {const appId = String(req.params.app || '').trim();const token = bearerToken(req);// 0) Principal tokens first, so a sandbox token is refused as such (never a 404 that hints// at which tenants exist). Only developer-project tenant routes (a route that names its-- // capability, under /<project_id>/) opt in to sandbox tokens.-- const appRoute = !!capability && PROJECT_ID_RE.test(appId);++ // verb or capability, under /<project_id>/) opt in to sandbox tokens.++ const appRoute = !!named && PROJECT_ID_RE.test(appId);let svc = null;if (token && token.split('.').length === 3) {const r = verifyServiceTokenResult(token, { acceptSandbox: appRoute });@@@ -238,10 +324,10 @@ function tenantAuth({ allowUser = false, capability = null } = {}) {if (svc && isAppPrincipal(svc)) {// Developer apps reach only their own project's tenant, and only on routes that name a capability.if (!appRoute) {-- return problem(res, 403, capability ? 'capability.namespace_denied' : 'capability.denied',-- capability ? 'app tokens reach only /<project_id>/ tenants' : 'app tokens are not accepted on this route');++ return problem(res, 403, named ? 'capability.namespace_denied' : 'capability.denied',++ named ? 'app tokens reach only /<project_id>/ tenants' : 'app tokens are not accepted on this route');}-- return projectTenant(req, res, next, appId, svc, capability);++ return projectTenant(req, res, next, appId, svc, grant);}if (svc && svc.env === 'sandbox') return problem(res, 401, 'token.sandbox_refused', 'only developer-app sandbox tokens are accepted, on their own project tenant');@@@ -270,14 +356,16 @@ function tenantAuth({ allowUser = false, capability = null } = {}) {}// 1b) A service-principal token from OpenVibe.Network (roadmap Wave 1, ADR-003). Accepted only on-- // routes that name the capability they perform, and only for the :app namespaces the token was-- // granted. It carries the app's authority for that one action — no acting user.++ // routes that name the verb (or capability) they perform, and only for the namespaces of this++ // tenant the token was granted (the root is the :app id). It carries the app's authority for++ // that one action — no acting user.if (svc) {-- if (!capability) return problem(res, 403, 'capability.denied', 'service tokens are not accepted on this route');-- const c = capabilities.check(svc, capability, { namespace: appId });++ if (!named) return problem(res, 403, 'capability.denied', 'service tokens are not accepted on this route');++ const c = tenantGrant(svc, grant, appId, db.rootNamespace(app));if (!c.allowed) return problem(res, 403, c.code, c.reason);req.authType = 'app';req.principal = { sub: svc.sub, cap: svc.cap, jti: svc.jti };++ req.grant = svc;return next();}@@@ -398,6 +486,10 @@ function seedApps() {module.exports = {tenantAuth,tenantPath,++ VERBS,++ verbAllowed,++ namespaceGrant,++ grantedNamespaces,ensureTokenOnlyApps,verifyServiceToken,verifyServiceTokenResult,@@@ -62,6 +62,10 @@ const config = {multipartDefaultPartMb: intEnv('MEDIA_MULTIPART_PART_MB', 64),multipartTtlHours: intEnv('MEDIA_MULTIPART_TTL_HOURS', 24), // an unfinished session's parts are purged after thisuploadMinFreeMb: intEnv('MEDIA_UPLOAD_MIN_FREE_MB', 10240), // multipart needs 2x its size free plus this++ // Namespaces and quotas (server/objects/namespaces.js): an upload holds its quota reservation this long++ // after its last step (init, a fresh URL, the bytes); then the hourly sweep fails it and frees its bytes.++ reservationHours: Math.max(1, intEnv('MEDIA_UPLOAD_RESERVATION_HOURS', 72)),++ maxChildNamespaces: Math.max(0, intEnv('MEDIA_NAMESPACE_MAX_CHILDREN', 100)), // children per tenant},// Job system (server/jobs/; docs/object-model.md#jobs).@@@ -29,6 +29,7 @@ function getDb() {database.exec(schema);migrateColumns();migrateJobsTable(schema);++ migrateNamespaces();ensureObjectTriggers();normalizeThumbnailUrls();seedSettings();@@@ -265,7 +266,9 @@ function upsertApp({ app_id, name, api_key, webhook_url, webhook_secret, allowedallowed_origins = excluded.allowed_origins,quota_bytes = excluded.quota_bytes`,[app_id, name || app_id, hashApiKey(api_key), webhook_url || null, webhook_secret || null, origins, quota_bytes || 0]);-- return getApp(app_id);++ const row = getApp(app_id);++ ensureRootNamespace(row);++ return row;}/**@@@ -289,6 +292,7 @@ function ensureProjectTenant(projectId, env, quotaBytes) {err.code = 'media.tenant.conflict';throw err;}++ ensureRootNamespace(row);return row;}@@@ -303,6 +307,78 @@ function appAllowedOrigins(app) {try { return JSON.parse(app.allowed_origins || '[]'); } catch { return []; }}++// ── Namespaces (roadmap WS-G task 2; objects/namespaces.js) ──++++/**++ * A tenant's root namespace: its app id for a first-party tenant; app.<project_id> for a developer++ * project's production tenant and app.<project_id>.sandbox for its sandbox tenant (so a grant of++ * app.<project_id>.* covers both, and the token's env still picks the tenant).++ */++function rootNamespace(app) {++ if (!app || !app.app_id) return null;++ if (app.project_id) return app.env === 'sandbox' ? `app.${app.project_id}.sandbox` : `app.${app.project_id}`;++ return app.app_id;++}++++/** Who owns a tenant's namespaces: service:<app_id> (a first-party tenant) or project:<prj_…>. */++function namespaceOwner(app) {++ return app.project_id ? `project:${app.project_id}` : `service:${app.app_id}`;++}++++/** The tenant's root namespace row, created when missing. Returns its name. */++function ensureRootNamespace(app) {++ const ns = rootNamespace(app);++ if (!ns) return null;++ run('INSERT OR IGNORE INTO media_namespaces (namespace, app_id, parent, owner) VALUES (?, ?, NULL, ?)', [ns, app.app_id, namespaceOwner(app)]);++ return ns;++}++++/**++ * Namespaces as rows (WS-G task 2). Idempotent, on every open:++ * 1. a developer project's objects move from the tenant id (prj_<ULID>, prj_<ULID>-sandbox), which++ * was their namespace before namespaces were rows, to app.<project_id>[.sandbox];++ * 2. every tenant has its root row, and every namespace an object names has a row (a child with the++ * chain up to its root);++ * 3. once (setting namespaces.reservations_seeded): every native upload still in progress holds a++ * quota reservation of its declared size, expiring MEDIA_UPLOAD_RESERVATION_HOURS after that++ * first open, so an upload that began before reservations existed is not cut short.++ */++function migrateNamespaces() {++ const insert = database.prepare('INSERT OR IGNORE INTO media_namespaces (namespace, app_id, parent, owner) VALUES (?, ?, ?, ?)');++ database.transaction(() => {++ const apps = new Map(database.prepare('SELECT * FROM apps').all().map(a => [a.app_id, a]));++ const move = database.prepare('UPDATE media_objects SET namespace = ? WHERE app_id = ? AND namespace = ?');++ let moved = 0;++ for (const app of apps.values()) {++ if (app.project_id) moved += move.run(rootNamespace(app), app.app_id, app.app_id).changes;++ insert.run(rootNamespace(app), app.app_id, null, namespaceOwner(app));++ }++ if (moved) console.warn(`[DB] Moved ${moved} developer-project object(s) to their app.<project_id> namespace`);++ const missing = database.prepare(`SELECT DISTINCT o.app_id, o.namespace FROM media_objects o++ LEFT JOIN media_namespaces n ON n.namespace = o.namespace WHERE n.namespace IS NULL`).all();++ for (const m of missing) {++ const app = apps.get(m.app_id) || { app_id: m.app_id };++ const root = rootNamespace(app);++ insert.run(root, app.app_id, null, namespaceOwner(app));++ if (m.namespace === root) continue;++ if (!m.namespace.startsWith(`${root}.`)) { insert.run(m.namespace, app.app_id, null, namespaceOwner(app)); continue; }++ let parent = root;++ for (const seg of m.namespace.slice(root.length + 1).split('.')) {++ insert.run(`${parent}.${seg}`, app.app_id, parent, namespaceOwner(app));++ parent = `${parent}.${seg}`;++ }++ }++ const seeded = database.prepare("SELECT value FROM media_settings WHERE key = 'namespaces.reservations_seeded'").get();++ if (!seeded) {++ const n = database.prepare(`INSERT OR IGNORE INTO media_quota_reservations (object_id, app_id, namespace, bytes, expires_at)++ SELECT id, app_id, namespace, size_bytes, datetime('now', ?) FROM media_objects WHERE lifecycle_status = 'uploading' AND legacy_ref IS NULL`)++ .run(`+${Math.max(1, config.objects.reservationHours)} hours`).changes;++ database.prepare("INSERT OR REPLACE INTO media_settings (key, value, type) VALUES ('namespaces.reservations_seeded', ?, 'string')").run(new Date().toISOString());++ if (n) console.warn(`[DB] ${n} upload(s) in progress now hold a quota reservation`);++ }++ })();++}++// ── VOD helpers ──────────────────────────────────────────────// The row and its media_object in one transaction (a clips-only recording has no object).@@@ -820,6 +896,8 @@ module.exports = {getSetting, setSetting,// appshashApiKey, getApp, listApps, upsertApp, appAllowedOrigins, projectTenantId, ensureProjectTenant, isSandboxTenant,++ // namespaces++ rootNamespace, namespaceOwner, ensureRootNamespace,// vodscreateVod, getVodById, getVodByFileBasename, listVods, countVods, setVodVisibility, vodStatus,latestVodThumbsByManagedStreams, getAppStats, getAppStatSeries,@@@ -215,7 +215,7 @@ CREATE UNIQUE INDEX IF NOT EXISTS idx_assets_identity ON assets(app_id, kind, naCREATE TABLE IF NOT EXISTS media_objects (id TEXT PRIMARY KEY, -- med_<ULID> (time-sortable)app_id TEXT NOT NULL, -- tenant-- namespace TEXT NOT NULL, -- capability namespace (= app_id today; projects later)++ namespace TEXT NOT NULL, -- a media_namespaces row: the tenant's root (app id, or app.<project_id>[.sandbox]) or a child of itkind TEXT NOT NULL CHECK(kind IN ('vod', 'clip', 'file', 'thumbnail', 'screenshot', 'avatar', 'asset')),owner_subject TEXT, -- canonical subject (usr_<ULID>) when knownowner_app TEXT, -- legacy owner: the app whose user-id space owner_user_id is in@@@ -237,6 +237,47 @@ CREATE INDEX IF NOT EXISTS idx_media_objects_app ON media_objects(app_id, id);CREATE INDEX IF NOT EXISTS idx_media_objects_kind ON media_objects(app_id, kind);CREATE INDEX IF NOT EXISTS idx_media_objects_owner ON media_objects(owner_subject);CREATE INDEX IF NOT EXISTS idx_media_objects_lifecycle ON media_objects(lifecycle_status);++CREATE INDEX IF NOT EXISTS idx_media_objects_namespace ON media_objects(app_id, namespace, lifecycle_status);++++-- Namespaces (roadmap WS-G task 2; server/objects/namespaces.js, docs/object-model.md#namespaces-grants-and-quotas).++-- Every namespace an object names is a row. A tenant (apps row) has one root: its app id, or++-- app.<project_id> (app.<project_id>.sandbox) for a developer project, and children below it++-- (<root>.<segment>…) made by the first upload that names one. Quotas: NULL inherits (a root takes its++-- tenant's apps.quota_bytes; a child has no limit of its own), 0 is no limit. The used_* and reserved_*++-- columns are a snapshot of the subtree (the namespace and its children) taken by reconcile(); quota++-- checks count from the object and reservation rows themselves.++CREATE TABLE IF NOT EXISTS media_namespaces (++ namespace TEXT PRIMARY KEY,++ app_id TEXT NOT NULL, -- the tenant it belongs to (and whose quota it counts against)++ parent TEXT, -- NULL for a root++ owner TEXT NOT NULL, -- service:<app_id> (a first-party tenant) | project:<prj_…> (a developer project)++ policy TEXT NOT NULL DEFAULT '{}', -- JSON: kinds, visibilities, max_object_bytes, strict_verbs (children inherit)++ quota_bytes INTEGER,++ quota_objects INTEGER,++ used_bytes INTEGER NOT NULL DEFAULT 0,++ used_objects INTEGER NOT NULL DEFAULT 0,++ reserved_bytes INTEGER NOT NULL DEFAULT 0,++ reserved_objects INTEGER NOT NULL DEFAULT 0,++ reconciled_at DATETIME,++ created_at DATETIME DEFAULT CURRENT_TIMESTAMP,++ updated_at DATETIME DEFAULT CURRENT_TIMESTAMP++);++CREATE INDEX IF NOT EXISTS idx_media_namespaces_app ON media_namespaces(app_id, namespace);++++-- Quota held by an upload in progress (a native object still `uploading`): its declared size from++-- init, the bytes received once they are stored. Settled at complete (the object's own size counts++-- from then on), released at abort (down to the bytes still stored, if any) and swept at expiry.++CREATE TABLE IF NOT EXISTS media_quota_reservations (++ object_id TEXT PRIMARY KEY,++ app_id TEXT NOT NULL,++ namespace TEXT NOT NULL,++ bytes INTEGER NOT NULL DEFAULT 0,++ expires_at DATETIME NOT NULL, -- the expiry sweep fails the upload after this (unless a multipart session is open)++ created_at DATETIME DEFAULT CURRENT_TIMESTAMP,++ updated_at DATETIME DEFAULT CURRENT_TIMESTAMP++);++CREATE INDEX IF NOT EXISTS idx_media_quota_reservations_ns ON media_quota_reservations(app_id, namespace);++CREATE INDEX IF NOT EXISTS idx_media_quota_reservations_expiry ON media_quota_reservations(expires_at);-- Where the bytes are. One row per provider copy; the canonical one is named on the object.CREATE TABLE IF NOT EXISTS media_locations (@@@ -10,11 +10,14 @@** Files live at FILES_PATH/<app>/<key> with key = <sha256-prefix>-<name>.* Public serving (Content-Type + Range) is at GET /f/:key.-- * Per-app quota: apps.quota_bytes (0 = unlimited) checked against the tenant's-- * stored bytes (v1 files + native v2 objects, objects/model.usedBytes).++ * Files live in the tenant's root namespace. Its quotas (bytes: apps.quota_bytes unless the root++ * sets its own, 0 = unlimited; objects: the root's quota_objects) count v1 files, native v2 objects++ * and uploads in progress (objects/namespaces.js).*-- * Network principal tokens: upload + delete need media.object.upload, list + meta-- * need media.object.read, for namespace = :app. Developer-project tenants++ * Network principal tokens, for the tenant's root namespace (server/auth.js VERBS): upload needs++ * write (media.object.upload), delete needs delete (media.object.delete, or media.object.upload),++ * list needs list (media.object.list, or media.object.read), meta needs read (media.object.read).++ * Developer-project tenants* (:app = prj_<ULID>, app tokens only, ADR-014): sandbox files are never served* publicly — their `url` is a short-lived signed /f/:key URL.*/@@@ -27,6 +30,7 @@ const fs = require('fs');const multer = require('multer');const db = require('../db/database');const config = require('../config');++const namespaces = require('../objects/namespaces');const { tenantAuth, tenantCors } = require('../auth');const router = express.Router({ mergeParams: true });@@@ -81,7 +85,7 @@ function sha256File(filePath) {}// ── Upload ───────────────────────────────────────────────────--router.post('/', tenantAuth({ allowUser: true, capability: 'media.object.upload' }), upload.single('file'), async (req, res) => {++router.post('/', tenantAuth({ allowUser: true, verb: 'write' }), upload.single('file'), async (req, res) => {try {if (!req.file) return res.status(400).json({ error: 'No file uploaded (multipart field: file)' });@@@ -104,20 +108,14 @@ router.post('/', tenantAuth({ allowUser: true, capability: 'media.object.upload'return res.status(200).json({ ...filePublic(existing), deduplicated: true });}-- // Per-app quota check against quota_bytes (0 = unlimited): files + native objects. Checked-- // after the last await, so nothing else is stored between this check and the row below++ // The root namespace's quotas (bytes and objects: files + native objects + uploads in progress).++ // Checked after the last await, so nothing else is stored between this check and the row below// (concurrent uploads each saw the same usage when it ran before hashing).-- const quota = Number(req.appRow.quota_bytes) || 0;-- if (quota > 0) {-- const used = require('../objects/model').usedBytes(req.appId);-- if (used + req.file.size > quota) {-- try { fs.unlinkSync(req.file.path); } catch { /* */ }-- return res.status(413).json({-- error: 'App file quota exceeded',-- quota_bytes: quota,-- used_bytes: used,-- });-- }++ const root = db.rootNamespace(req.appRow);++ const q = namespaces.checkQuota(req.appRow, root, { bytes: req.file.size, objects: 1 });++ if (q) {++ try { fs.unlinkSync(req.file.path); } catch { /* */ }++ return res.status(413).json({ error: q.code === 'media.quota.objects_exceeded' ? 'App object quota exceeded' : 'App file quota exceeded', code: q.code, ...q.extra });}const dest = path.join(appDir(req.appId), key);@@@ -140,6 +138,7 @@ router.post('/', tenantAuth({ allowUser: true, capability: 'media.object.upload'});const row = db.getFileByKey(key, req.appId);++ namespaces.reconcileChain(req.appRow, root);console.log(`[Files] Stored ${key} for app ${req.appId} (${(req.file.size / 1024).toFixed(1)} KB)`);res.status(201).json(filePublic(row));} catch (err) {@@@ -150,7 +149,7 @@ router.post('/', tenantAuth({ allowUser: true, capability: 'media.object.upload'});// ── List ─────────────────────────────────────────────────────--router.get('/', tenantAuth({ allowUser: true, capability: 'media.object.read' }), (req, res) => {++router.get('/', tenantAuth({ allowUser: true, verb: 'list' }), (req, res) => {try {const limit = Math.min(Math.max(parseInt(req.query.limit || '100', 10), 1), 500);const offset = Math.max(parseInt(req.query.offset || '0', 10), 0);@@@ -167,7 +166,7 @@ router.get('/', tenantAuth({ allowUser: true, capability: 'media.object.read' })});// ── Meta ─────────────────────────────────────────────────────--router.get('/:key', tenantAuth({ allowUser: true, capability: 'media.object.read' }), (req, res) => {++router.get('/:key', tenantAuth({ allowUser: true, verb: 'read' }), (req, res) => {try {const row = db.getFileByKey(String(req.params.key), req.appId);if (!row) return res.status(404).json({ error: 'File not found' });@@@ -178,7 +177,7 @@ router.get('/:key', tenantAuth({ allowUser: true, capability: 'media.object.read});// ── Delete ───────────────────────────────────────────────────--router.delete('/:key', tenantAuth({ allowUser: true, capability: 'media.object.upload' }), (req, res) => {++router.delete('/:key', tenantAuth({ allowUser: true, verb: 'delete' }), (req, res) => {try {const row = db.getFileByKey(String(req.params.key), req.appId);if (!row) return res.status(404).json({ error: 'File not found' });@@@ -190,6 +189,7 @@ router.delete('/:key', tenantAuth({ allowUser: true, capability: 'media.object.uconst filePath = filePathForKey(row);try { if (fs.existsSync(filePath)) fs.unlinkSync(filePath); } catch { /* */ }db.deleteFileRow(row.key);++ namespaces.reconcileChain(req.appRow, db.rootNamespace(req.appRow));res.json({ message: 'File deleted' });} catch (err) {res.status(500).json({ error: 'Failed to delete file' });@@@ -183,6 +183,7 @@ app.use('/api/v1/:app/assets', require('./assets/routes'));app.use('/api/v1/:app/admin/storage', require('./admin/routes'));app.use('/api/v2/:app/objects', require('./objects/routes')); // canonical object API (docs/object-model.md)app.use('/api/v2/:app/jobs', require('./jobs/routes')); // media jobs: thumbnails, split/remux, invariant scans++app.use('/api/v2/:app/namespaces', require('./objects/namespace-routes')); // a tenant's namespaces: policy, quotas, usageapp.use('/o', require('./objects/routes').publicRouter); // object bytes (public, or signed)app.use('/', require('./public/routes')); // /v /c /p /t /fapp.use(require('./not-found').notFound); // nothing matched: 404 with Media's CSP (Cloudflare's beacon allowed)@@@ -233,6 +234,14 @@ if (!drill.enabled) {try { const n = require('./objects/model').purgeExpired(); if (n) console.log(`[Objects] Purged ${n} expired deleted object(s)`); } catch (err) { console.warn('[Objects] purge:', err.message); }// Incomplete multipart uploads past MEDIA_MULTIPART_TTL_HOURS: their parts are deleted (the objects stay uploading).try { const r = require('./objects/multipart').purgeExpired(); if (r.expired || r.orphan_dirs) console.log(`[Objects] Multipart: ${r.expired} expired session(s), ${r.orphan_dirs} orphan part dir(s) removed`); } catch (err) { console.warn('[Objects] multipart purge:', err.message); }++ // Namespaces: uploads whose quota reservation expired are failed and their bytes freed; then every++ // namespace's usage snapshot is refreshed (v1 files, deletes and purges included).++ try {++ const ns = require('./objects/namespaces');++ const r = ns.expireReservations();++ if (r.expired || r.dropped) console.log(`[Namespaces] ${r.expired} abandoned upload(s) expired, ${r.dropped} stale reservation(s) dropped`);++ ns.reconcileAll();++ } catch (err) { console.warn('[Namespaces] sweep:', err.message); }});// Project rows that have no media_object yet (first boot after the upgrade: all of them).// C-75 catch-up: every write now makes its object in the same transaction (objects/model.js@@@ -245,6 +254,8 @@ if (!drill.enabled) {const r = bf.backfill({ onlyMissing: true });if (r.totals.created || r.totals.updated || r.errors.length) console.log(`[Objects] Backfill: ${bf.summarize(r)}`);} catch (err) { console.warn('[Objects] Backfill failed:', err.message); }++ // The namespaces' usage snapshot, first taken here (then hourly, and after each write).++ try { require('./objects/namespaces').reconcileAll(); } catch (err) { console.warn('[Namespaces] reconcile:', err.message); }}, 15 * 1000).unref?.();// Recover from an unclean shutdown: rows stuck in is_recording with no live ffmpeg are finalized@@@ -146,11 +146,10 @@ function checkRoom(obj, needBytes) {if (free < needBytes + reserve) {throw new JobError('insufficient_disk', `Not enough free disk: need ${Math.ceil((needBytes + reserve) / MB)} MB, ${Math.floor(free / MB)} MB free`, { retryAfterS: 1800 });}++ // Outputs go to the source's namespace: every quota from there up to the tenant must have the room.const app = db.getApp(obj.app_id);-- const quota = Number(app && app.quota_bytes) || 0;-- if (quota && model.usedBytes(obj.app_id) + needBytes > quota) {-- throw new JobError('quota_exceeded', 'The tenant storage quota would be exceeded', { permanent: true });-- }++ const q = app ? require('../objects/namespaces').checkQuota(app, obj.namespace || db.rootNamespace(app), { bytes: needBytes, objects: 1 }) : null;++ if (q) throw new JobError('quota_exceeded', `The storage quota would be exceeded (${q.detail})`, { permanent: true });}function workDir(jobId) {@@@ -182,7 +181,7 @@ async function adopt({ src, file, ext, job, relation, metadata, variant, saveChedb.run(`INSERT INTO media_objects (id, app_id, namespace, kind, owner_subject, owner_app, owner_user_id, visibility, lifecycle_status,mime_type, size_bytes, content_hash, canonical_provider, canonical_key, legacy_ref, metadata)VALUES (?, ?, ?, ?, ?, ?, ?, 'private', 'ready', ?, ?, ?, 'local', ?, NULL, ?)`,-- [id, src.app_id, src.namespace || src.app_id, kind || src.kind, src.owner_subject || null, src.owner_app || src.app_id, src.owner_user_id ?? null,++ [id, src.app_id, src.namespace || db.rootNamespace(db.getApp(src.app_id) || { app_id: src.app_id }), kind || src.kind, src.owner_subject || null, src.owner_app || src.app_id, src.owner_user_id ?? null,mimeFor(ext, src), size, hash, dest,JSON.stringify({ title: md.title || null, derived_from: src.id, job_id: job.id, filename: `${src.id}${suffix ? `-${suffix}` : metadata.part ? `-part${metadata.part}` : '-remux'}${ext}`, ...metadata })]);model.upsertLocation(id, { provider: 'local', key: dest, state: 'present', size_bytes: size, checksum: hash, verified: true });@@@ -9,11 +9,13 @@* POST /:jobId/cancel cancel (also DELETE /:jobId): proposed/queued at once -> 200;* running -> 202 { job } with cancel_requested until it stops; finished -> 409*-- * Auth as the objects API: the app key or a Network service token granting media.object.read (reads)-- * or media.object.upload (create, approve, cancel) for namespace :app; a developer project's app-- * tokens on its own tenant. With X-OV-User-Id (app key) the caller acts for one of the app's users:-- * it sees and decides only jobs it created or jobs on objects it owns, and creates jobs on its own-- * objects only. Errors are RFC 9457 problems.++ * Auth as the objects API: the app key, or a Network token holding the verb (server/auth.js VERBS):++ * list (GET /) and read (GET /:jobId) for the tenant's root namespace; transform (create, approve,++ * cancel: media.derivative.create, or media.object.upload as before) for the namespace of the job's++ * object (the root for a job with none); a developer project's app tokens on its own tenant. With++ * X-OV-User-Id (app key) the caller acts for one of the app's users: it sees and decides only jobs++ * it created or jobs on objects it owns, and creates jobs on its own objects only. Errors are RFC++ * 9457 problems.*/'use strict';@@@ -21,10 +23,12 @@ const express = require('express');const { http } = require('openvibe-contracts');const model = require('../objects/model');const queue = require('./queue');--const { tenantAuth, tenantCors } = require('../auth');++const db = require('../db/database');++const { tenantAuth, tenantCors, namespaceGrant } = require('../auth');--const write = tenantAuth({ capability: 'media.object.upload' });--const read = tenantAuth({ capability: 'media.object.read' });++const transform = tenantAuth({ verb: 'transform', namespaced: true });++const read = tenantAuth({ verb: 'read' });++const list = tenantAuth({ verb: 'list' });function problem(res, status, code, detail, extra) {return http.sendProblem(res, status, code, { detail, extra });@@@ -48,6 +52,13 @@ function load(req, res) {return job;}++/** Transform is granted per namespace: the job's object's, or the tenant's root for a job with none (else answered). */++function mayTransform(req, res, obj) {++ const g = namespaceGrant(req, 'transform', obj ? obj.namespace : db.rootNamespace(req.appRow));++ if (!g.allowed) { problem(res, 403, g.code, g.reason); return false; }++ return true;++}++function sendJobError(res, err) {if (err instanceof queue.JobError) {if (err.retryAfterS) res.set('Retry-After', String(err.retryAfterS));@@@ -60,7 +71,7 @@ function sendJobError(res, err) {const router = express.Router({ mergeParams: true });router.use(tenantCors);--router.get('/', read, (req, res) => {++router.get('/', list, (req, res) => {try {const q = req.query;if (q.cursor && !/^mjob_[0-9A-Za-z_]{1,40}$/.test(String(q.cursor))) return problem(res, 400, 'media.job.invalid', 'Bad cursor');@@@ -73,7 +84,7 @@ router.get('/', read, (req, res) => {} catch (err) { sendJobError(res, err); }});--router.post('/', write, (req, res) => {++router.post('/', transform, (req, res) => {try {const b = req.body || {};const type = String(b.type || '');@@@ -89,6 +100,7 @@ router.post('/', write, (req, res) => {} else if (spec.needsObject) {return problem(res, 400, 'media.job.invalid', `${type} needs object_id`);}++ if (!mayTransform(req, res, obj)) return;if (b.params != null && (typeof b.params !== 'object' || Array.isArray(b.params) || JSON.stringify(b.params).length > 16384)) {return problem(res, 400, 'media.job.invalid', 'params must be an object of at most 16 KB');}@@@ -110,10 +122,10 @@ router.get('/:jobId', read, (req, res) => {if (job) res.json({ job: queue.jobPublic(job) });});--router.post('/:jobId/approve', write, (req, res) => {++router.post('/:jobId/approve', transform, (req, res) => {try {const job = load(req, res);-- if (!job) return;++ if (!job || !mayTransform(req, res, job.object_id ? model.getObject(job.object_id) : null)) return;if (job.status !== 'proposed') return problem(res, 409, 'media.job.not_proposed', `The job is ${job.status}; only proposals are approved`);const out = queue.approve(job.id, { by: who(req) });if (!out) return problem(res, 409, 'media.job.not_proposed', 'The job was decided meanwhile');@@@ -125,13 +137,13 @@ router.post('/:jobId/approve', write, (req, res) => {function cancelHandler(req, res) {try {const job = load(req, res);-- if (!job) return;++ if (!job || !mayTransform(req, res, job.object_id ? model.getObject(job.object_id) : null)) return;const out = queue.cancel(job.id, { by: who(req) });if (out.finished) return problem(res, 409, 'media.job.finished', `The job already ${out.job.status}`);res.status(out.pending ? 202 : 200).json({ job: queue.jobPublic(out.job) });} catch (err) { sendJobError(res, err); }}--router.post('/:jobId/cancel', write, cancelHandler);--router.delete('/:jobId', write, cancelHandler);++router.post('/:jobId/cancel', transform, cancelHandler);++router.delete('/:jobId', transform, cancelHandler);module.exports = router;@@@ -136,13 +136,18 @@ function isHeldRow(row) {const OBJECT_COLUMNS = ['app_id', 'namespace', 'kind', 'owner_subject', 'owner_app', 'owner_user_id', 'visibility','lifecycle_status', 'mime_type', 'size_bytes', 'content_hash', 'canonical_provider', 'canonical_key', 'legacy_ref', 'metadata', 'deleted_at'];++/** The root namespace of tenant `appId`: its app id, or app.<project_id>[.sandbox] for a developer project. */++function tenantRoot(appId) {++ return db.rootNamespace(db.getApp(appId) || { app_id: appId });++}++function createObject(o) {if (!KINDS.includes(o.kind)) throw new Error(`unknown kind ${o.kind}`);const id = ids.newId('media', o.createdMs);db.run(`INSERT INTO media_objects (id, app_id, namespace, kind, owner_subject, owner_app, owner_user_id, visibility,lifecycle_status, mime_type, size_bytes, content_hash, canonical_provider, canonical_key, legacy_ref, metadata, created_at)VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, COALESCE(?, CURRENT_TIMESTAMP))`,-- [id, o.app_id, o.namespace || o.app_id, o.kind, o.owner_subject || null, o.owner_app || null, o.owner_user_id ?? null,++ [id, o.app_id, o.namespace || tenantRoot(o.app_id), o.kind, o.owner_subject || null, o.owner_app || null, o.owner_user_id ?? null,VISIBILITIES.includes(o.visibility) ? o.visibility : 'private', LIFECYCLES.includes(o.lifecycle_status) ? o.lifecycle_status : 'uploading',o.mime_type || null, Number(o.size_bytes) || 0, o.content_hash || null, o.canonical_provider || null, o.canonical_key || null,o.legacy_ref || null, JSON.stringify(o.metadata || {}), o.created_at || null]);@@@ -219,7 +224,7 @@ function project(p) {const found = getObjectByLegacyRef(p.legacy_ref) || (p.existingId ? getObject(p.existingId) : null);const canon = p.locations.find(l => l.provider === p.canonical) || p.locations[0] || null;const fields = {-- app_id: p.app_id, namespace: p.app_id, kind: p.kind,++ app_id: p.app_id, namespace: tenantRoot(p.app_id), kind: p.kind,owner_app: p.owner_app || p.app_id, owner_user_id: p.owner_user_id ?? null,visibility: p.visibility, lifecycle_status: p.lifecycle_status,mime_type: p.mime_type || null, size_bytes: Number(p.size_bytes) || 0, content_hash: p.content_hash || null,@@@ -633,6 +638,7 @@ function softDelete(obj, { by = null } = {}) {db.getDb().transaction(() => {db.run(`UPDATE media_objects SET lifecycle_status = 'deleted', deleted_at = CURRENT_TIMESTAMP, metadata = ?, updated_at = CURRENT_TIMESTAMPWHERE id = ?`, [JSON.stringify(md), obj.id]);++ require('./namespaces').settle(obj.id); // an upload deleted before it completed holds no quotaevents.recordObjectChanges(); // media.object.deleted commits with the delete})();events.kick();@@@ -673,17 +679,15 @@ function purgeExpired({ retentionDays = config.objects.retentionDays } = {}) {return purged;}--/** Bytes counted against apps.quota_bytes: v1 files + live native objects (reservations included). */++/**++ * Bytes counted against the tenant's quota: v1 files, ready native objects (in a developer project++ * also soft-deleted ones until purged) and the reservations of uploads in progress++ * (objects/namespaces.usage of the tenant's root, which is the whole tenant).++ */function usedBytes(appId, excludeId = null) {-- // Developer-project tenants (third-party uploaders) also pay for soft-deleted bytes until they are-- // purged: those stay on disk for the retention period, and upload -> delete -> upload would-- // otherwise grow the disk without bound.-- const app = db.getApp(appId);-- const retained = app && app.project_id ? " OR (lifecycle_status = 'deleted' AND json_extract(metadata, '$.purged_at') IS NULL)" : '';-- const native = db.get(`SELECT COALESCE(SUM(size_bytes), 0) AS b FROM media_objects-- WHERE app_id = ? AND legacy_ref IS NULL AND (lifecycle_status IN ('uploading', 'ready')${retained}) AND id != ?`,-- [appId, excludeId || '']).b;-- return db.appFilesBytes(appId) + native;++ const app = db.getApp(appId) || { app_id: appId };++ const u = require('./namespaces').usage(app, db.rootNamespace(app), { excludeId });++ return u.used_bytes + u.reserved_bytes;}// ── Public shape ─────────────────────────────────────────────@@@ -11,7 +11,8 @@* Complete checks that every part is there (and matches the client's per-part sha256 when given),* concatenates them into OBJECTS_PATH/<app>/<id> while hashing the whole object, and then the route* runs the same checks as a single-part complete. Abort (or expiry after MEDIA_MULTIPART_TTL_HOURS)-- * deletes the parts; the object stays `uploading` so it can be sent again.++ * deletes the parts and releases the session's quota reservation; the object stays `uploading` so it++ * can be sent again (a new session or upload URL reserves again).*/'use strict';@@@ -83,7 +84,7 @@ function initiate(obj, { partSize: requested } = {}) {const free = freeBytes();if (free < need) return { status: 507, code: 'media.storage.insufficient', error: `Not enough free space for this upload (${Math.floor(free / MB)} MB free)` };const prior = activeFor(obj.id);-- if (prior) abort(prior.id);++ if (prior) abort(prior.id, 'aborted', { release: false }); // replaced: the object's reservation carries overconst id = `mup_${ids.ulid()}`;const parts = Math.max(1, Math.ceil(total / chosen.size));db.run(`INSERT INTO media_uploads (id, object_id, app_id, part_size, total_size, parts_expected, expires_at)@@@ -227,10 +228,15 @@ function removeParts(uploadId) {db.run('DELETE FROM media_upload_parts WHERE upload_id = ?', [uploadId]);}--/** Delete a session's parts; the object stays uploading. */--function abort(uploadId, status = 'aborted') {++/**++ * Delete a session's parts; the object stays uploading. Its quota reservation is released down to++ * what is still stored for it (objects/namespaces.release), unless a new session replaces this one.++ */++function abort(uploadId, status = 'aborted', { release = true } = {}) {++ const session = getSession(uploadId);db.run("UPDATE media_uploads SET status = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ? AND status IN ('active', 'completing')", [status, uploadId]);removeParts(uploadId);++ if (release && session) require('./namespaces').release(session.object_id);}/**nnew file mode 100644@@@ -0,0 +1,57 @@++/**++ * OpenVibe.Media — a tenant's namespaces (mounted at /api/v2/:app/namespaces;++ * docs/object-model.md#namespaces-grants-and-quotas)++ *++ * GET / the namespaces the caller may list: owner, policy, quota and usage++ * GET /:namespace one (its full name, or a name relative to the tenant's root)++ *++ * The owner's read. The app key sees every namespace of its tenant; a Network token (a first-party++ * service's, or a developer project's app token on /<project_id>/) sees those it holds list++ * (GET /: media.object.list, or media.object.read) or read (GET /:namespace: media.object.read) for.++ * Usage is counted from the rows when read, and the snapshot columns are refreshed with it. Quotas and++ * policies are set by the operator (scripts/namespaces.js).++ */++'use strict';++++const express = require('express');++const { http } = require('openvibe-contracts');++const namespaces = require('./namespaces');++const { tenantAuth, tenantCors, namespaceGrant } = require('../auth');++++const router = express.Router({ mergeParams: true });++router.use(tenantCors);++++function problem(res, status, code, detail) {++ return http.sendProblem(res, status, code, { detail });++}++++function shape(req, row) {++ return namespaces.publicShape(req.appRow, row, namespaces.reconcile(req.appRow, row.namespace));++}++++router.get('/', tenantAuth({ verb: 'list', namespaced: true }), (req, res) => {++ try {++ const rows = namespaces.listForTenant(req.appId).filter(r => namespaceGrant(req, 'list', r.namespace).allowed);++ res.json({ namespaces: rows.map(r => shape(req, r)) });++ } catch (err) {++ console.error('[Namespaces] list error:', err.message);++ problem(res, 500, 'media.namespace.list_failed', 'Failed to list namespaces');++ }++});++++router.get('/:namespace', tenantAuth({ verb: 'read', namespaced: true }), (req, res) => {++ try {++ const named = namespaces.resolveName(req.appRow, req.params.namespace);++ if (named.error) return problem(res, 400, 'media.namespace.invalid', named.error);++ const g = namespaceGrant(req, 'read', named.namespace);++ if (!g.allowed) return problem(res, 403, g.code, g.reason);++ const row = namespaces.get(named.namespace);++ if (!row || row.app_id !== req.appId) return problem(res, 404, 'media.namespace.not_found', 'No such namespace in this tenant');++ res.json(shape(req, row));++ } catch (err) {++ console.error('[Namespaces] read error:', err.message);++ problem(res, 500, 'media.namespace.read_failed', 'Failed to read the namespace');++ }++});++++module.exports = router;nnew file mode 100644@@@ -0,0 +1,334 @@++/**++ * OpenVibe.Media — namespaces, quotas and upload reservations (roadmap WS-G task 2;++ * docs/object-model.md#namespaces-grants-and-quotas)++ *++ * Every object lives in a namespace, and every namespace is a media_namespaces row: the tenant it++ * belongs to, its owner, a policy, quotas and a snapshot of its usage. A tenant has one root:++ *++ * first-party tenant its app id live, community, tools, …++ * developer project, production app.<project_id> (tenant prj_<ULID>)++ * developer project, sandbox app.<project_id>.sandbox (tenant prj_<ULID>-sandbox)++ *++ * and children below it (<root>.<segment>, up to three segments), made by the first upload that++ * names one. Which verbs a token may use in which namespace is decided in server/auth.js; this module++ * keeps the rows, the policy and the quotas.++ *++ * Quotas. A namespace's limit covers its subtree (itself and its children). quota_bytes NULL inherits++ * (the root takes its tenant's apps.quota_bytes, a child has no limit of its own), 0 is no limit.++ * Usage counts native objects that are ready (in a developer project, also soft-deleted ones until++ * their bytes are purged), the tenant's v1 files at the root, and the reservations of uploads in++ * progress. An upload reserves its declared size at init (checked against every limit from its++ * namespace up to the root), re-reserves the bytes it actually stored, and is checked again with its++ * real size at complete, when the reservation is settled; abort releases it down to the bytes still++ * stored; the hourly sweep fails an upload whose reservation expired and frees its bytes. Checks count++ * from the rows, never from the snapshot; reconcile() refreshes the snapshot.++ */++'use strict';++++const fs = require('fs');++const path = require('path');++const db = require('../db/database');++const config = require('../config');++++const SEGMENT_RE = /^[a-z0-9][a-z0-9_-]{0,39}$/;++const MAX_DEPTH = 3;++const POLICY_KEYS = ['kinds', 'visibilities', 'max_object_bytes', 'strict_verbs'];++++function parsePolicy(s) {++ try {++ const p = typeof s === 'string' ? JSON.parse(s || '{}') : s;++ return p && typeof p === 'object' && !Array.isArray(p) ? p : {};++ } catch { return {}; }++}++++/** A policy an operator sets: only the known keys, each of the right shape. Returns { policy } or { error }. */++function validatePolicy(p) {++ if (!p || typeof p !== 'object' || Array.isArray(p)) return { error: 'policy must be an object' };++ const unknown = Object.keys(p).filter(k => !POLICY_KEYS.includes(k));++ if (unknown.length) return { error: `unknown policy key(s): ${unknown.join(', ')} (known: ${POLICY_KEYS.join(', ')})` };++ const model = require('./model');++ if (p.kinds != null && !(Array.isArray(p.kinds) && p.kinds.every(k => model.KINDS.includes(k)))) return { error: `kinds must be a list of ${model.KINDS.join(', ')}` };++ if (p.visibilities != null && !(Array.isArray(p.visibilities) && p.visibilities.every(v => model.VISIBILITIES.includes(v)))) return { error: `visibilities must be a list of ${model.VISIBILITIES.join(', ')}` };++ if (p.max_object_bytes != null && !(Number.isInteger(p.max_object_bytes) && p.max_object_bytes >= 0)) return { error: 'max_object_bytes must be a whole number of bytes (0 = no limit)' };++ if (p.strict_verbs != null && typeof p.strict_verbs !== 'boolean') return { error: 'strict_verbs must be true or false' };++ return { policy: p };++}++++const sqlTime = (ms) => new Date(ms).toISOString().replace('T', ' ').slice(0, 19);++++// ── Rows ─────────────────────────────────────────────────────++++function get(namespace) {++ return namespace ? db.get('SELECT * FROM media_namespaces WHERE namespace = ?', [String(namespace)]) : null;++}++++function listForTenant(appId) {++ return db.all('SELECT * FROM media_namespaces WHERE app_id = ? ORDER BY namespace', [appId]);++}++++/** Is `ns` the namespace `base` or below it? */++function within(base, ns) {++ return ns === base || String(ns || '').startsWith(`${base}.`);++}++++/** The tenant's rows from its root down to `namespace` (those that exist), root first. */++function chain(appId, namespace) {++ const parts = String(namespace || '').split('.');++ const names = parts.map((_, i) => parts.slice(0, i + 1).join('.'));++ if (!names.length) return [];++ const rows = db.all(`SELECT * FROM media_namespaces WHERE app_id = ? AND namespace IN (${names.map(() => '?').join(', ')})`, [appId, ...names]);++ return rows.sort((a, b) => a.namespace.length - b.namespace.length);++}++++/** Policy in force in `namespace`: the root's, overridden key by key by each child on the way down. */++function effectivePolicy(appId, namespace) {++ return Object.assign({}, ...chain(appId, namespace).map(r => parsePolicy(r.policy)));++}++++function isStrict(appId, namespace) {++ return effectivePolicy(appId, namespace).strict_verbs === true;++}++++/**++ * The namespace a request names in tenant `app`: nothing = the root; the root's full name or a name++ * below it; or a name relative to the root ('avatars' = '<root>.avatars'). Returns { namespace } or { error }.++ */++function resolveName(app, requested) {++ const root = db.rootNamespace(app);++ if (requested == null || requested === '') return { namespace: root };++ const s = String(requested).trim();++ if (s === root) return { namespace: root };++ const rel = s.startsWith(`${root}.`) ? s.slice(root.length + 1) : s;++ const segs = rel.split('.');++ if (segs.length > MAX_DEPTH || !segs.every(x => SEGMENT_RE.test(x)) || `${root}.${rel}`.length > 200) {++ return { error: `namespace must be ${root} or a name below it: up to ${MAX_DEPTH} segments of a-z, 0-9, _ and - (e.g. ${root}.avatars)` };++ }++ // The production root's `sandbox` child would read as the project's sandbox tenant.++ if (app.project_id && app.env !== 'sandbox' && segs[0] === 'sandbox') return { error: `${root}.sandbox is the project's sandbox tenant` };++ return { namespace: `${root}.${rel}` };++}++++/**++ * The row for `namespace` in tenant `app`, with the chain above it created when missing. Returns the++ * row, or { error, status, code } when the tenant already has MEDIA_NAMESPACE_MAX_CHILDREN children++ * or the name belongs to another tenant.++ */++function ensure(app, namespace) {++ const root = db.ensureRootNamespace(app);++ const found = get(namespace);++ if (found) {++ return found.app_id === app.app_id ? found++ : { error: `namespace ${namespace} belongs to another tenant`, status: 409, code: 'media.namespace.conflict' };++ }++ if (!within(root, namespace)) return { error: `namespace ${namespace} is not below ${root}`, status: 400, code: 'media.namespace.invalid' };++ const segs = namespace.slice(root.length + 1).split('.');++ const names = segs.map((_, i) => `${root}.${segs.slice(0, i + 1).join('.')}`);++ const missing = names.filter(n => !get(n));++ const children = db.get('SELECT COUNT(*) AS n FROM media_namespaces WHERE app_id = ? AND parent IS NOT NULL', [app.app_id]).n;++ if (children + missing.length > config.objects.maxChildNamespaces) {++ return { error: `a tenant has at most ${config.objects.maxChildNamespaces} namespaces below its root`, status: 413, code: 'media.quota.namespaces_exceeded' };++ }++ let parent = root;++ for (const name of names) {++ db.run('INSERT OR IGNORE INTO media_namespaces (namespace, app_id, parent, owner) VALUES (?, ?, ?, ?)', [name, app.app_id, parent, db.namespaceOwner(app)]);++ parent = name;++ }++ return get(namespace);++}++++// ── Usage and quotas ─────────────────────────────────────────++++/**++ * What the subtree of `namespace` uses now, from the rows: { used_bytes, used_objects, reserved_bytes,++ * reserved_objects }. The root's subtree is the whole tenant, v1 files included. `excludeId` leaves++ * one object (and its reservation) out, for re-checking that object.++ */++function usage(app, namespace, { excludeId = null } = {}) {++ const root = db.rootNamespace(app);++ const whole = namespace === root;++ const where = whole ? '' : ' AND (o.namespace = @ns OR substr(o.namespace, 1, @plen) = @prefix)';++ const rwhere = whole ? '' : ' AND (r.namespace = @ns OR substr(r.namespace, 1, @plen) = @prefix)';++ const p = { app: app.app_id, ns: namespace, plen: String(namespace).length + 1, prefix: `${namespace}.`, ex: excludeId || '' };++ // Developer projects (third-party uploaders) pay for soft-deleted bytes until they are purged: they stay++ // on disk for the retention period, and upload -> delete -> upload would otherwise grow the disk.++ const retained = app.project_id ? " OR (o.lifecycle_status = 'deleted' AND json_extract(o.metadata, '$.purged_at') IS NULL)" : '';++ const used = db.get(`SELECT COUNT(*) AS n, COALESCE(SUM(o.size_bytes), 0) AS b FROM media_objects o++ WHERE o.app_id = @app AND o.legacy_ref IS NULL AND (o.lifecycle_status = 'ready'${retained}) AND o.id != @ex${where}`, p);++ // A reservation counts while its object is still uploading (a settled or failed one never double-counts).++ const reserved = db.get(`SELECT COUNT(*) AS n, COALESCE(SUM(r.bytes), 0) AS b FROM media_quota_reservations r++ JOIN media_objects o ON o.id = r.object_id AND o.lifecycle_status = 'uploading'++ WHERE r.app_id = @app AND r.object_id != @ex${rwhere}`, p);++ const files = whole ? db.get('SELECT COUNT(*) AS n, COALESCE(SUM(size), 0) AS b FROM files WHERE app_id = ?', [app.app_id]) : { n: 0, b: 0 };++ return { used_bytes: used.b + files.b, used_objects: used.n + files.n, reserved_bytes: reserved.b, reserved_objects: reserved.n };++}++++/** The limits of one row: { bytes, objects, bytes_from } (0 = none). A root with no quota_bytes takes its tenant's. */++function limitsOf(app, row) {++ const inherits = row.quota_bytes == null;++ const bytes = inherits ? (row.parent == null ? Number(app.quota_bytes) || 0 : 0) : Number(row.quota_bytes) || 0;++ return { bytes, objects: Number(row.quota_objects) || 0, bytes_from: bytes ? (inherits ? 'tenant' : 'namespace') : null };++}++++/**++ * Would adding `bytes` and `objects` to `namespace` break a limit? Checked for the namespace and every++ * ancestor with a limit (the root always carries the tenant's quota), and against the effective++ * policy's max_object_bytes. Returns null, or { status, code, detail, extra } for a problem answer.++ */++function checkQuota(app, namespace, { bytes = 0, objects = 0, excludeId = null } = {}) {++ const root = db.rootNamespace(app);++ const rows = chain(app.app_id, namespace);++ if (!rows.length || rows[0].namespace !== root) rows.unshift({ namespace: root, parent: null, quota_bytes: null, quota_objects: null });++ const policy = Object.assign({}, ...rows.map(r => parsePolicy(r.policy)));++ const maxObject = Number(policy.max_object_bytes) || 0;++ if (maxObject && bytes > maxObject) {++ return { status: 413, code: 'media.object.too_large', detail: `Objects in ${namespace} are limited to ${maxObject} bytes`, extra: { namespace, max_object_bytes: maxObject } };++ }++ for (const row of rows.slice().reverse()) {++ const lim = limitsOf(app, row);++ if (!lim.bytes && !lim.objects) continue;++ const u = usage(app, row.namespace, { excludeId });++ if (lim.bytes && u.used_bytes + u.reserved_bytes + bytes > lim.bytes) {++ return {++ status: 413, code: 'media.quota.exceeded', detail: `The storage quota of ${row.namespace} would be exceeded`,++ extra: { namespace: row.namespace, quota_bytes: lim.bytes, used_bytes: u.used_bytes, reserved_bytes: u.reserved_bytes },++ };++ }++ if (lim.objects && objects > 0 && u.used_objects + u.reserved_objects + objects > lim.objects) {++ return {++ status: 413, code: 'media.quota.objects_exceeded', detail: `The object quota of ${row.namespace} would be exceeded`,++ extra: { namespace: row.namespace, quota_objects: lim.objects, used_objects: u.used_objects, reserved_objects: u.reserved_objects },++ };++ }++ }++ return null;++}++++// ── Reservations ─────────────────────────────────────────────++++function reservation(objectId) {++ return objectId ? db.get('SELECT * FROM media_quota_reservations WHERE object_id = ?', [objectId]) : null;++}++++/** Hold `bytes` of quota for the upload of `obj` (replacing what it held); expiry moves forward, never back. */++function reserve(obj, bytes, { hours = config.objects.reservationHours } = {}) {++ db.run(`INSERT INTO media_quota_reservations (object_id, app_id, namespace, bytes, expires_at) VALUES (?, ?, ?, ?, ?)++ ON CONFLICT(object_id) DO UPDATE SET bytes = excluded.bytes, namespace = excluded.namespace,++ expires_at = MAX(expires_at, excluded.expires_at), updated_at = CURRENT_TIMESTAMP`,++ [obj.id, obj.app_id, obj.namespace, Math.max(0, Number(bytes) || 0), sqlTime(Date.now() + hours * 3600 * 1000)]);++}++++/** An upload step happened: its reservation's expiry moves forward. */++function touch(objectId, { hours = config.objects.reservationHours } = {}) {++ db.run('UPDATE media_quota_reservations SET expires_at = MAX(expires_at, ?), updated_at = CURRENT_TIMESTAMP WHERE object_id = ?',++ [sqlTime(Date.now() + hours * 3600 * 1000), objectId]);++}++++/** The upload ended (complete, or the object was deleted): the object's own row counts from now on. */++function settle(objectId) {++ db.run('DELETE FROM media_quota_reservations WHERE object_id = ?', [objectId]);++}++++/** An aborted or expired session: keep holding only what is still stored for the object (a PUT's bytes). */++function release(objectId) {++ const obj = db.get('SELECT lifecycle_status FROM media_objects WHERE id = ?', [objectId]);++ const stored = obj && obj.lifecycle_status === 'uploading'++ ? db.get("SELECT size_bytes FROM media_locations WHERE object_id = ? AND provider = 'local' AND state = 'present'", [objectId]) : null;++ if (stored && Number(stored.size_bytes) > 0) db.run('UPDATE media_quota_reservations SET bytes = ?, updated_at = CURRENT_TIMESTAMP WHERE object_id = ?', [Number(stored.size_bytes), objectId]);++ else settle(objectId);++}++++/**++ * The hourly sweep: an upload whose reservation expired, with no multipart session open, is failed++ * (metadata.failure upload_expired), its stored bytes are deleted and its reservation goes; a++ * reservation whose object is no longer uploading is dropped. Returns { expired, dropped }.++ */++function expireReservations() {++ const rows = db.all(`SELECT r.object_id, r.app_id, r.namespace, o.lifecycle_status, o.legacy_ref, o.metadata FROM media_quota_reservations r++ LEFT JOIN media_objects o ON o.id = r.object_id WHERE r.expires_at < datetime('now') ORDER BY r.expires_at LIMIT 1000`);++ const multipart = require('./multipart');++ const root = path.resolve(config.objects.path) + path.sep;++ const touched = new Map();++ let expired = 0, dropped = 0;++ for (const r of rows) {++ if (r.lifecycle_status !== 'uploading' || r.legacy_ref) { settle(r.object_id); dropped++; continue; }++ if (multipart.activeFor(r.object_id)) continue; // its session expires on its own, then this does++ try {++ const md = parsePolicy(r.metadata);++ md.failure = 'upload_expired';++ md.expired_at = new Date().toISOString();++ const locs = db.all("SELECT * FROM media_locations WHERE object_id = ? AND provider = 'local'", [r.object_id]);++ db.getDb().transaction(() => {++ db.run("UPDATE media_objects SET lifecycle_status = 'failed', metadata = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ? AND lifecycle_status = 'uploading'",++ [JSON.stringify(md), r.object_id]);++ for (const l of locs) db.run('DELETE FROM media_locations WHERE id = ?', [l.id]);++ settle(r.object_id);++ })();++ for (const l of locs) if (path.resolve(l.key).startsWith(root)) { try { fs.unlinkSync(l.key); } catch { /* already gone */ } }++ expired++;++ touched.set(`${r.app_id}\n${r.namespace}`, r);++ } catch (err) {++ // A retention hold refuses the lifecycle change: the reservation stays until the hold is released.++ console.warn(`[Namespaces] Could not expire the upload of ${r.object_id}: ${err.message}`);++ }++ }++ for (const r of touched.values()) { const app = db.getApp(r.app_id); if (app) reconcileChain(app, r.namespace); }++ return { expired, dropped };++}++++// ── Snapshot (the used_* / reserved_* columns) ───────────────++++function reconcile(app, namespace) {++ const u = usage(app, namespace);++ db.run(`UPDATE media_namespaces SET used_bytes = ?, used_objects = ?, reserved_bytes = ?, reserved_objects = ?,++ reconciled_at = CURRENT_TIMESTAMP, updated_at = CURRENT_TIMESTAMP WHERE namespace = ? AND app_id = ?`,++ [u.used_bytes, u.used_objects, u.reserved_bytes, u.reserved_objects, namespace, app.app_id]);++ return u;++}++++/** Refresh `namespace` and every namespace above it (after a write changed its usage). */++function reconcileChain(app, namespace) {++ try {++ for (const row of chain(app.app_id, namespace)) reconcile(app, row.namespace);++ } catch (err) { console.warn(`[Namespaces] reconcile ${namespace}: ${err.message}`); }++}++++/** Refresh every row (the hourly job). Returns how many. */++function reconcileAll() {++ const apps = new Map();++ let n = 0;++ for (const row of db.all('SELECT namespace, app_id FROM media_namespaces ORDER BY app_id, namespace')) {++ if (!apps.has(row.app_id)) apps.set(row.app_id, db.getApp(row.app_id) || { app_id: row.app_id });++ reconcile(apps.get(row.app_id), row.namespace);++ n++;++ }++ return n;++}++++// ── Public shape ─────────────────────────────────────────────++++/** A namespace as its owner reads it; `fresh` is usage() just computed (else the snapshot is shown). */++function publicShape(app, row, fresh = null) {++ const lim = limitsOf(app, row);++ const u = fresh || { used_bytes: row.used_bytes, used_objects: row.used_objects, reserved_bytes: row.reserved_bytes, reserved_objects: row.reserved_objects };++ const out = {++ namespace: row.namespace,++ parent: row.parent || null,++ root: row.parent == null,++ owner: row.owner,++ policy: parsePolicy(row.policy),++ effective_policy: effectivePolicy(row.app_id, row.namespace),++ quota: { bytes: lim.bytes || null, objects: lim.objects || null, bytes_from: lim.bytes_from },++ usage: { ...u, reconciled_at: fresh ? new Date().toISOString() : row.reconciled_at || null },++ created_at: row.created_at,++ };++ if (db.isSandboxTenant(row.app_id)) out.sandbox = true;++ return out;++}++++module.exports = {++ SEGMENT_RE, POLICY_KEYS, parsePolicy, validatePolicy,++ get, listForTenant, within, chain, effectivePolicy, isStrict, resolveName, ensure,++ usage, limitsOf, checkQuota,++ reservation, reserve, touch, settle, release, expireReservations,++ reconcile, reconcileChain, reconcileAll, publicShape,++};@@@ -12,7 +12,7 @@* POST /:id/multipart/:uploadId/complete assemble + the same checks as complete → ready* DELETE /:id/multipart/:uploadId abort (parts deleted; the object stays uploading)* GET /:id metadata (id = med_… or legacy:<app>:<kind>:<id>)-- * GET / cursor list (?kind&visibility&status&owner&user_id&include_deleted&limit&cursor)++ * GET / cursor list (?namespace&kind&visibility&status&owner&user_id&include_deleted&limit&cursor)* DELETE /:id soft delete (bytes kept MEDIA_DELETE_RETENTION_DAYS); 409 when held* POST /:id/restore undo a soft delete inside the retention period* GET /:id/download 302 to the public location, or a signed short-lived URL for private objects@@@ -21,8 +21,13 @@* DELETE /:id/holds/:holdId release it (app key only)* Staff place and release holds by vod/clip id at /api/v1/:app/admin/storage/holds.*-- * Auth: the app's API key, or a Network service token granting media.object.upload-- * (writes) / media.object.read (reads) for namespace = :app. Developer-project tenants++ * Auth: the app's API key, or a Network token holding the route's verb for the object's namespace++ * (server/auth.js VERBS): write (init, content, complete, multipart) media.object.upload; read++ * (metadata, download, holds) media.object.read; list media.object.list; delete and restore++ * media.object.delete. The older ids keep granting the newer verbs (read lists, upload deletes).++ * Init takes `namespace` (the tenant's root by default, or a name below it; objects/namespaces.js),++ * reserves its declared size against every quota from there up to the tenant, and complete settles++ * the reservation against the bytes received. Developer-project tenants* (:app = prj_<ULID>) take their project's app tokens only (server/auth.js); their sandbox* objects are served only through signed URLs, whatever their visibility. X-OV-Subject names the* owner (usr_…); X-OV-User-Id (app key only) acts as one of the app's users. The@@@ -46,12 +51,16 @@ const invariant = require('./invariant');const signing = require('./signing');const ctypes = require('./content-type');const multipart = require('./multipart');--const { tenantAuth, tenantCors, tenantPath } = require('../auth');++const namespaces = require('./namespaces');++const { tenantAuth, tenantCors, tenantPath, namespaceGrant } = require('../auth');const { announce } = require('../webhooks');const MB = 1024 * 1024;--const upload = tenantAuth({ capability: 'media.object.upload' });--const read = tenantAuth({ capability: 'media.object.read' });++// Each route checks its verb for the namespace it works in (the object's, or the one init names).++const upload = tenantAuth({ verb: 'write', namespaced: true });++const read = tenantAuth({ verb: 'read', namespaced: true });++const list = tenantAuth({ verb: 'list', namespaced: true });++const remove = tenantAuth({ verb: 'delete', namespaced: true });const appOnly = tenantAuth();function problem(res, status, code, detail, extra) {@@@ -77,9 +86,12 @@ function canWrite(req, obj) {return req.authType !== 'user' || (obj.owner_user_id != null && obj.owner_user_id === req.userId);}--function load(req, res) {++/** The object named in the URL, in this tenant and visible to the caller, if the caller may `verb` in its namespace (else answered). */++function load(req, res, verb = 'read') {const obj = model.resolveObject(String(req.params.id || ''), req.appId);if (!obj || !canSee(req, obj)) { problem(res, 404, 'media.object.not_found', 'No such object in this namespace'); return null; }++ const g = namespaceGrant(req, verb, obj.namespace);++ if (!g.allowed) { problem(res, 403, g.code, g.reason); return null; }return obj;}@@@ -107,9 +119,9 @@ function contentAuth(req, res, next) {if (tokenTenant(req, res, obj)) next();}--/** Multipart session token (from POST /:id/multipart) or the usual tenant credential with `capability`. */--function multipartAuth(capability) {-- const fallback = tenantAuth({ capability });++/** Multipart session token (from POST /:id/multipart) or the usual tenant credential with `verb`. */++function multipartAuth(verb) {++ const fallback = tenantAuth({ verb, namespaced: true });return (req, res, next) => {const token = req.query.token || req.headers['x-upload-token'];if (!token) return fallback(req, res, next);@@@ -161,15 +173,25 @@ function sanitizeFilename(name) {return path.basename(String(name)).replace(/[^\w.\- ]/g, '_').slice(0, 200) || null;}--function quotaCheck(req, res, addBytes, excludeId = null) {-- const quota = Number(req.appRow.quota_bytes) || 0;-- if (!quota) return true;-- const used = model.usedBytes(req.appId, excludeId);-- if (used + addBytes <= quota) return true;-- problem(res, 413, 'media.quota.exceeded', 'App storage quota exceeded', { quota_bytes: quota, used_bytes: used });++/**++ * Room for `bytes` (and `objects` new objects) in `namespace`: every quota from there up to the++ * tenant, and the policy's max_object_bytes (objects/namespaces.checkQuota). Answers 413 when not.++ */++function quotaCheck(req, res, namespace, { bytes = 0, objects = 0, excludeId = null } = {}) {++ const q = namespaces.checkQuota(req.appRow, namespace, { bytes, objects, excludeId });++ if (!q) return true;++ problem(res, q.status, q.code, q.detail, q.extra);return false;}++/** An upload step that has no reservation (it was released at abort) takes one again, checked like init. */++function reserveAgain(req, res, obj, bytes) {++ const held = namespaces.reservation(obj.id);++ if (!quotaCheck(req, res, obj.namespace, { bytes, objects: held ? 0 : 1, excludeId: obj.id })) return false;++ namespaces.reserve(obj, bytes);++ return true;++}++// ── Content upload (mounted ahead of the JSON body parser — see index.js) ──/** Stream the request body to `tmp`, hashing as it goes; stop past `cap` bytes. */@@@ -196,7 +218,7 @@ function receive(req, tmp, cap) {}async function putContent(req, res) {-- const obj = load(req, res);++ const obj = load(req, res, 'write');if (!obj) { req.resume(); return; }if (!canWrite(req, obj)) { req.resume(); return problem(res, 403, 'media.object.forbidden', 'Not your object'); }if (obj.legacy_ref) { req.resume(); return problem(res, 409, 'media.object.legacy_managed', 'This object is written through the v1 API'); }@@@ -224,6 +246,7 @@ async function putContent(req, res) {const tmpDir = path.join(config.objects.path, '.tmp');fs.mkdirSync(tmpDir, { recursive: true });const tmp = path.join(tmpDir, `${obj.id}-${crypto.randomBytes(4).toString('hex')}`);++ namespaces.touch(obj.id); // the upload is active: its reservation does not expire under itconst got = await receive(req, tmp, cap);const drop = () => { try { fs.unlinkSync(tmp); } catch { /* not written */ } };if (got.error) {@@@ -233,7 +256,12 @@ async function putContent(req, res) {}if (declared && got.bytes !== declared) { drop(); return problem(res, 400, 'media.object.size_mismatch', `Declared ${declared} bytes, received ${got.bytes}`); }if (!got.bytes) { drop(); return problem(res, 400, 'media.object.empty', 'No bytes received'); }-- if (!quotaCheck(req, res, got.bytes, obj.id)) { drop(); return; }++ // The object may have been completed, deleted or expired while the bytes streamed in.++ const current = model.getObject(obj.id);++ if (!current || current.lifecycle_status !== 'uploading') { drop(); return problem(res, 409, 'media.object.not_uploading', `Object is ${current ? current.lifecycle_status : 'gone'}`); }++ // The real size replaces the declared one in this upload's reservation.++ const held = namespaces.reservation(obj.id);++ if (!quotaCheck(req, res, obj.namespace, { bytes: got.bytes, objects: held ? 0 : 1, excludeId: obj.id })) { drop(); return; }try {const dest = model.objectFilePath(obj);@@@ -245,6 +273,7 @@ async function putContent(req, res) {canonical_provider: 'local', canonical_key: dest,});model.upsertLocation(obj.id, { provider: 'local', key: dest, state: 'present', size_bytes: got.bytes, checksum: got.sha256, verified: true });++ namespaces.reserve(current, got.bytes);})();res.json({ id: obj.id, size_bytes: got.bytes, content_hash: got.sha256 });} catch (err) {@@@ -266,6 +295,15 @@ router.post('/', upload, (req, res) => {if (!model.KINDS.includes(kind)) return problem(res, 400, 'media.object.invalid', `kind must be one of ${model.KINDS.join(', ')}`);const visibility = String(b.visibility || 'private');if (!model.VISIBILITIES.includes(visibility)) return problem(res, 400, 'media.object.invalid', 'visibility must be public, unlisted or private');++ // The namespace: the tenant's root unless the body names one below it; the caller must hold write there.++ const named = namespaces.resolveName(req.appRow, b.namespace);++ if (named.error) return problem(res, 400, 'media.namespace.invalid', named.error);++ const namespace = named.namespace;++ const g = namespaceGrant(req, 'write', namespace);++ if (!g.allowed) return problem(res, 403, g.code, g.reason);++ const policy = namespaces.effectivePolicy(req.appId, namespace);++ if (Array.isArray(policy.kinds) && !policy.kinds.includes(kind)) return problem(res, 422, 'media.namespace.policy_denied', `${namespace} takes kinds ${policy.kinds.join(', ') || '(none)'}`);++ if (Array.isArray(policy.visibilities) && !policy.visibilities.includes(visibility)) return problem(res, 422, 'media.namespace.policy_denied', `${namespace} takes visibilities ${policy.visibilities.join(', ') || '(none)'}`);let size = 0;if (b.size_bytes != null) {size = Number(b.size_bytes);@@@ -285,7 +323,6 @@ router.post('/', upload, (req, res) => {}const subject = subjectFrom(req);if (subject === undefined) return problem(res, 400, 'media.object.invalid', 'X-OV-Subject must be a user subject (usr_…)');-- if (!quotaCheck(req, res, size)) return;const mime = b.mime_type && /^[\w.+-]+\/[\w.+-]+$/.test(String(b.mime_type)) ? String(b.mime_type).toLowerCase() : null;if (mime) {@@@ -300,15 +337,28 @@ router.post('/', upload, (req, res) => {if (b.content_hash) metadata.expected_sha256 = String(b.content_hash).toLowerCase();if (req.principal) metadata.created_by = req.principal.sub;-- const id = model.createObject({-- app_id: req.appId, namespace: req.appId, kind, owner_subject: subject, owner_app: req.appId, owner_user_id: userId,-- visibility, lifecycle_status: 'uploading', mime_type: mime, size_bytes: size, metadata,-- });++ // The namespace row, the quota check, the object and its reservation commit together, so two++ // inits racing for the last of a quota cannot both get it.++ const made = db.getDb().transaction(() => {++ const row = namespaces.ensure(req.appRow, namespace);++ if (row.error) return { problem: { status: row.status, code: row.code, detail: row.error } };++ const q = namespaces.checkQuota(req.appRow, namespace, { bytes: size, objects: 1 });++ if (q) return { problem: q };++ const newId = model.createObject({++ app_id: req.appId, namespace, kind, owner_subject: subject, owner_app: req.appId, owner_user_id: userId,++ visibility, lifecycle_status: 'uploading', mime_type: mime, size_bytes: size, metadata,++ });++ namespaces.reserve(model.getObject(newId), size);++ return { id: newId };++ })();++ if (made.problem) return problem(res, made.problem.status, made.problem.code, made.problem.detail, made.problem.extra);++ const id = made.id;const obj = model.getObject(id);let session = null;if (wantParts) {session = multipart.initiate(obj, { partSize: b.part_size });if (session.error) {++ namespaces.settle(id);db.run('DELETE FROM media_objects WHERE id = ?', [id]); // nothing was stored for itreturn problem(res, session.status, session.code, session.error);}@@@ -346,22 +396,25 @@ function completeUpload(req, res, obj, { expectedHash = null } = {}) {if (invariant.wouldViolate(obj)) {return problem(res, 422, 'media.invariant.public_object_too_large', `Public playback objects are limited to ${config.objects.publicMaxMb} MB`);}-- if (!quotaCheck(req, res, Number(obj.size_bytes) || 0, obj.id)) return;++ // Reconciled against the bytes actually stored: the object's real size, everything else as it is now.++ if (!quotaCheck(req, res, obj.namespace, { bytes: Number(obj.size_bytes) || 0, objects: namespaces.reservation(obj.id) ? 0 : 1, excludeId: obj.id })) return;const typeWhy = ctypes.kindProblem(obj.kind, obj.mime_type) || ctypes.contentProblem(obj.kind, obj.mime_type, ctypes.readHead(loc.key));if (typeWhy) return problem(res, 415, 'media.object.content_mismatch', typeWhy);const { data: body } = announce(req.appId, 'media.object.uploaded', {change: () => {model.updateObject(obj.id, { lifecycle_status: 'ready' });++ namespaces.settle(obj.id); // the ready object counts from now oninvariant.record(model.getObject(obj.id));},payload: () => model.objectPublic(model.getObject(obj.id)),});++ namespaces.reconcileChain(req.appRow, obj.namespace);return res.json(body);}/** The object named in the URL, writable by this caller, native and still uploading (else answered). */function loadUploading(req, res) {-- const obj = load(req, res);++ const obj = load(req, res, 'write');if (!obj) return null;if (!canWrite(req, obj)) { problem(res, 403, 'media.object.forbidden', 'Not your object'); return null; }if (obj.legacy_ref) { problem(res, 409, 'media.object.legacy_managed', 'This object is written through the v1 API'); return null; }@@@ -371,7 +424,7 @@ function loadUploading(req, res) {router.post('/:id/complete', contentAuth, tenantCors, (req, res) => {try {-- const obj = load(req, res);++ const obj = load(req, res, 'write');if (!obj) return;if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');if (obj.lifecycle_status === 'ready' && !obj.legacy_ref) return res.json(model.objectPublic(obj));@@@ -391,19 +444,22 @@ router.post('/:id/upload-url', upload, (req, res) => {const ttl = req.body && req.body.ttl != null ? Number(req.body.ttl) : (req.query.ttl != null ? Number(req.query.ttl) : undefined);if (ttl !== undefined && !(ttl >= 60 && ttl <= 86400)) return problem(res, 400, 'media.object.invalid', 'ttl must be 60-86400 seconds');if (Number(obj.size_bytes) > config.objects.maxUploadMb * MB) return problem(res, 413, 'media.object.too_large', 'This object is uploaded in parts: POST /multipart');++ // A fresh URL is a fresh attempt: the upload holds its declared size again (after an abort released it).++ const stored = Number((db.get("SELECT size_bytes FROM media_locations WHERE object_id = ? AND provider = 'local' AND state = 'present'", [obj.id]) || {}).size_bytes) || 0;++ if (!reserveAgain(req, res, obj, Math.max(Number(obj.size_bytes) || 0, stored))) return;res.json(presignedPut(req, obj, ttl));});// ── Multipart uploads ────────────────────────────────────────--const mpRead = multipartAuth('media.object.read');--const mpWrite = multipartAuth('media.object.upload');++const mpRead = multipartAuth('read');++const mpWrite = multipartAuth('write');/** The session named in the URL, belonging to the object, still open (else answered). */function loadSession(req, res, obj, { open = true } = {}) {const session = multipart.getSession(String(req.params.uploadId || ''));if (!session || session.object_id !== obj.id) { problem(res, 404, 'media.upload.not_found', 'No such upload for this object'); return null; }-- if (open && session.status === 'active' && multipart.isExpired(session)) multipart.abort(session.id, 'expired');++ if (open && session.status === 'active' && multipart.isExpired(session)) multipart.abort(session.id, 'expired'); // releases its reservationconst now = multipart.getSession(session.id);if (open && now.status !== 'active') { problem(res, 409, 'media.upload.not_active', `The upload is ${now.status}`); return null; }return now;@@@ -423,11 +479,13 @@ router.post('/:id/multipart', upload, (req, res) => {if (invariant.wouldViolate({ kind: obj.kind, visibility: obj.visibility, size_bytes: size })) {return problem(res, 422, 'media.invariant.public_object_too_large', `Public playback objects are limited to ${config.objects.publicMaxMb} MB`);}-- if (!quotaCheck(req, res, size, obj.id)) return;++ if (!reserveAgain(req, res, obj, size)) return;model.updateObject(obj.id, { size_bytes: size });current = model.getObject(obj.id);} else if (b.size_bytes != null && Number(b.size_bytes) !== Number(obj.size_bytes)) {return problem(res, 400, 'media.object.size_mismatch', `The object declares ${obj.size_bytes} bytes`);++ } else if (!reserveAgain(req, res, obj, Number(obj.size_bytes))) {++ return; // a session after an abort holds the declared size again}const session = multipart.initiate(current, { partSize: b.part_size });if (session.error) return problem(res, session.status, session.code, session.error);@@@ -439,7 +497,7 @@ router.post('/:id/multipart', upload, (req, res) => {});router.get('/:id/multipart/:uploadId', mpRead, (req, res) => {-- const obj = load(req, res);++ const obj = load(req, res, 'read');if (!obj) return;const session = loadSession(req, res, obj, { open: false });if (!session) return;@@@ -485,21 +543,44 @@ router.post('/:id/multipart/:uploadId/complete', mpWrite, tenantCors, async (req});router.delete('/:id/multipart/:uploadId', mpWrite, (req, res) => {-- const obj = load(req, res);++ const obj = load(req, res, 'write');if (!obj) return;if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');const session = loadSession(req, res, obj, { open: false });if (!session) return;if (session.status === 'completed') return problem(res, 409, 'media.upload.not_active', 'The upload is completed');-- multipart.abort(session.id);++ multipart.abort(session.id); // parts deleted; the quota it held is releasedres.json(multipart.sessionPublic(multipart.getSession(session.id), { parts: false }));});--router.get('/', read, (req, res) => {++router.get('/', list, (req, res) => {try {const q = req.query;const limit = Math.min(Math.max(parseInt(q.limit || '50', 10) || 50, 1), 200);const conds = ['app_id = ?'], params = [req.appId];++ // ?namespace= narrows the list to one namespace and those below it.++ if (q.namespace != null && q.namespace !== '') {++ const named = namespaces.resolveName(req.appRow, q.namespace);++ if (named.error) return problem(res, 400, 'media.namespace.invalid', named.error);++ const g = namespaceGrant(req, 'list', named.namespace);++ if (!g.allowed) return problem(res, 403, g.code, g.reason);++ conds.push('(namespace = ? OR substr(namespace, 1, ?) = ?)');++ params.push(named.namespace, named.namespace.length + 1, `${named.namespace}.`);++ }++ // A Network token lists only the namespaces it may list here.++ if (req.grant) {++ const rows = namespaces.listForTenant(req.appId);++ const listable = rows.filter(r => namespaceGrant(req, 'list', r.namespace).allowed).map(r => r.namespace);++ if (!listable.length) {++ // Nothing listable: refused when the verb itself is (a strict namespace), else an empty page.++ const g = namespaceGrant(req, 'list', db.rootNamespace(req.appRow));++ if (!g.allowed && g.code === 'capability.denied') return problem(res, 403, g.code, g.reason);++ conds.push('0');++ } else if (listable.length < rows.length) {++ conds.push(`namespace IN (${listable.map(() => '?').join(', ')})`);++ params.push(...listable);++ }++ }if (q.cursor) {if (!/^med_[0-9A-HJKMNP-TV-Z]{26}$/.test(String(q.cursor))) return problem(res, 400, 'media.object.invalid', 'Bad cursor');conds.push('id < ?'); params.push(String(q.cursor));@@@ -521,19 +602,21 @@ router.get('/', read, (req, res) => {});router.get('/:id', read, (req, res) => {-- const obj = load(req, res);++ const obj = load(req, res, 'read');if (obj) res.json(model.objectPublic(obj));});--router.delete('/:id', upload, (req, res) => {++router.delete('/:id', remove, (req, res) => {try {-- const obj = load(req, res);++ const obj = load(req, res, 'delete');if (!obj) return;if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');if (obj.legacy_ref) return problem(res, 409, 'media.object.legacy_managed', 'Delete this object through its v1 route (vods, clips, files, pastes)');if (model.isHeld(obj.id)) return problem(res, 409, 'media.object.held', 'Object is under a retention hold');if (obj.lifecycle_status === 'deleted') return res.json(model.objectPublic(obj));-- res.json(model.objectPublic(model.softDelete(obj, { by: req.principal ? req.principal.sub : `app:${req.appId}` })));++ const gone = model.softDelete(obj, { by: req.principal ? req.principal.sub : `app:${req.appId}` });++ namespaces.reconcileChain(req.appRow, obj.namespace);++ res.json(model.objectPublic(gone));} catch (err) {if (err.code === 'media.object.held' || /retention hold/.test(err.message)) return problem(res, 409, 'media.object.held', 'Object is under a retention hold');console.error('[Objects] delete error:', err.message);@@@ -541,18 +624,22 @@ router.delete('/:id', upload, (req, res) => {}});--router.post('/:id/restore', upload, (req, res) => {-- const obj = load(req, res);++router.post('/:id/restore', remove, (req, res) => {++ const obj = load(req, res, 'delete');if (!obj) return;if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');if (obj.lifecycle_status !== 'deleted' || obj.legacy_ref) return problem(res, 409, 'media.object.not_deleted', 'Only soft-deleted native objects can be restored');++ // A restored object counts again: it needs the room (in a developer project it never stopped counting).++ const md = model.parseJson(obj.metadata, {});++ if (md.pre_delete_status !== 'uploading' && !md.purged_at && !quotaCheck(req, res, obj.namespace, { bytes: Number(obj.size_bytes) || 0, objects: 1, excludeId: obj.id })) return;const back = model.restore(obj);if (!back) return problem(res, 410, 'media.object.purged', 'The retention period has passed and the bytes are gone');++ namespaces.reconcileChain(req.appRow, obj.namespace);res.json(model.objectPublic(back));});router.get('/:id/download', read, (req, res) => {-- const obj = load(req, res);++ const obj = load(req, res, 'read');if (!obj) return;if (obj.lifecycle_status === 'deleted') return problem(res, 410, 'media.object.deleted', 'Object was deleted');if (obj.lifecycle_status !== 'ready') return problem(res, 409, 'media.object.not_ready', `Object is ${obj.lifecycle_status}`);@@@ -571,7 +658,7 @@ router.get('/:id/download', read, (req, res) => {// ── Retention holds ──────────────────────────────────────────router.get('/:id/holds', read, (req, res) => {-- const obj = load(req, res);++ const obj = load(req, res, 'read');if (!obj) return;res.json({object_id: obj.id,@@@ -654,7 +741,7 @@ publicRouter.get('/:id', async (req, res) => {/** PUT /:id/multipart/:uploadId/parts/:n (raw body; mounted ahead of the JSON body parser). */async function putPart(req, res) {-- const obj = load(req, res);++ const obj = load(req, res, 'write');if (!obj) { req.resume(); return; }if (!canWrite(req, obj)) { req.resume(); return problem(res, 403, 'media.object.forbidden', 'Not your object'); }if (obj.lifecycle_status !== 'uploading') { req.resume(); return problem(res, 409, 'media.object.not_uploading', `Object is ${obj.lifecycle_status}`); }nnew file mode 100644@@@ -0,0 +1,373 @@++'use strict';++// Namespaces and grants (roadmap WS-G task 2; server/objects/namespaces.js, server/auth.js VERBS):++// - the backfill on open: developer-project objects move to app.<project_id>[.sandbox], every tenant++// and every namespace an object names gets a row, uploads in progress get a reservation (once);++// - five verbs per namespace (read, list, write, delete, transform), the older ids still granting the++// newer verbs, strict_verbs refusing them; child namespaces and grants that name only a child;++// - developer app tokens: their legacy ns [prj_…] means app.<project_id>.*, the new ns works the same;++// - quotas per namespace (bytes and objects, up to the tenant), reserved at init, re-reserved with the++// bytes stored, settled at complete, released at abort, swept at expiry; the policy's kinds,++// visibilities and max_object_bytes; the owner's read of namespaces and usage.++const assert = require('assert');++const fs = require('fs');++const os = require('os');++const path = require('path');++const http = require('http');++const crypto = require('crypto');++const Database = require('better-sqlite3');++++const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-namespaces-'));++const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++process.env.DB_PATH = path.join(tmp, 'media.db');++process.env.VOD_PATH = dir('vods');++process.env.FILES_PATH = dir('files');++process.env.OBJECTS_PATH = dir('objects');++process.env.THUMBNAILS_PATH = dir('thumbnails');++process.env.OV_NETWORK_URL = 'https://openvibe.network';++process.env.MEDIA_PUBLIC_URL = 'https://media.test';++process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';++process.env.MEDIA_NAMESPACE_MAX_CHILDREN = '6';++process.env.MEDIA_MULTIPART_MIN_PART_MB = '1';++process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';++++const { serviceAuth, ids } = require('openvibe-contracts');++const PA = `prj_${ids.ulid()}`;++const PB = `prj_${ids.ulid()}`;++const APP_A = `app_${ids.ulid()}`;++++// ── A database from before namespaces were rows ──++{++ const h = new Database(process.env.DB_PATH);++ h.exec(fs.readFileSync(path.join(__dirname, '..', 'server', 'db', 'schema.sql'), 'utf8'));++ h.exec('ALTER TABLE apps ADD COLUMN project_id TEXT; ALTER TABLE apps ADD COLUMN env TEXT;');++ h.prepare("INSERT INTO apps (app_id, name, api_key_hash, quota_bytes) VALUES ('live', 'Live', ?, 0)").run(crypto.createHash('sha256').update('live-key').digest('hex'));++ h.prepare("INSERT INTO apps (app_id, name, api_key_hash, quota_bytes, project_id, env) VALUES (?, 'p', '', 1000000, ?, 'production')").run(PA, PA);++ h.prepare("INSERT INTO apps (app_id, name, api_key_hash, quota_bytes, project_id, env) VALUES (?, 'p', '', 1000000, ?, 'sandbox')").run(`${PA}-sandbox`, PA);++ const put = h.prepare("INSERT INTO media_objects (id, app_id, namespace, kind, lifecycle_status, size_bytes) VALUES (?, ?, ?, 'file', ?, ?)");++ put.run(`med_${ids.ulid()}`, PA, PA, 'ready', 10);++ put.run(`med_${ids.ulid()}`, `${PA}-sandbox`, `${PA}-sandbox`, 'ready', 20);++ put.run(`med_${ids.ulid()}`, 'live', 'live.legacy.old', 'ready', 30);++ put.run('med_01M0PRE0UPLOAD0000000000AA', 'live', 'live', 'uploading', 100);++ h.close();++}++++const db = require('../server/db/database');++const auth = require('../server/auth');++const namespaces = require('../server/objects/namespaces');++const model = require('../server/objects/model');++const objectRoutes = require('../server/objects/routes');++const express = require('express');++++const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });++auth._setNetworkPublicKeyForTests(keys.publicKey);++const now = Math.floor(Date.now() / 1000);++const jti = () => `tok_${crypto.randomBytes(6).toString('hex')}`;++const svc = (cap, ns = ['community']) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: 'svc:community', actor_type: 'service', aud: ['openvibe.media'], cap, ns, iat: now, exp: now + 300, jti: jti() }, keys.privateKey);++const appTok = ({ cap = ['media.object.upload', 'media.object.read'], ns = [PA], env = 'production', project = PA } = {}) => serviceAuth.signServiceToken({++ iss: 'https://openvibe.network', sub: `app:${APP_A}`, actor_type: 'app', aud: ['openvibe.media'], cap, ns, project_id: project, env, iat: now, exp: now + 300, jti: jti(),++}, keys.privateKey);++++// ── 1. The backfill on open ──++{++ db.getDb();++ const nsOf = (app) => db.all('SELECT namespace FROM media_objects WHERE app_id = ? ORDER BY namespace', [app]).map(r => r.namespace);++ assert.deepStrictEqual(nsOf(PA), [`app.${PA}`], 'a project object moves to app.<project_id>');++ assert.deepStrictEqual(nsOf(`${PA}-sandbox`), [`app.${PA}.sandbox`], 'its sandbox object to app.<project_id>.sandbox');++ const rows = Object.fromEntries(db.all('SELECT * FROM media_namespaces').map(r => [r.namespace, r]));++ assert.deepStrictEqual([rows.live.owner, rows.live.parent, rows[`app.${PA}`].owner, rows[`app.${PA}.sandbox`].app_id], ['service:live', null, `project:${PA}`, `${PA}-sandbox`]);++ assert.deepStrictEqual([rows['live.legacy'].parent, rows['live.legacy.old'].parent], ['live', 'live.legacy'], 'a child an object names gets a row, with the chain up to its root');++ const pre = namespaces.reservation('med_01M0PRE0UPLOAD0000000000AA');++ assert.ok(pre && pre.bytes === 100 && pre.namespace === 'live', 'an upload already in progress holds its declared size');++ assert.ok(Date.parse(pre.expires_at.replace(' ', 'T') + 'Z') > Date.now() + 70 * 3600e3, 'for the full reservation window from now');++ // Idempotent, and the reservation seed runs once: a released reservation does not come back.++ namespaces.settle('med_01M0PRE0UPLOAD0000000000AA');++ db.close();++ db.getDb();++ assert.strictEqual(namespaces.reservation('med_01M0PRE0UPLOAD0000000000AA'), undefined, 'the seed ran once');++ assert.deepStrictEqual(nsOf(PA), [`app.${PA}`]);++ assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_namespaces').n, Object.keys(rows).length, 'no duplicate rows on a second open');++ db.run("UPDATE media_objects SET lifecycle_status = 'failed' WHERE id = 'med_01M0PRE0UPLOAD0000000000AA'");++ console.log('✅ backfill: project objects under app.<project_id>, a row per namespace, reservations seeded once');++}++++db.upsertApp({ app_id: 'tiny', api_key: 'tiny-key', quota_bytes: 100 });++auth.ensureTokenOnlyApps();++assert.ok(namespaces.get('tiny') && namespaces.get('community'), 'new tenants get their root row');++++const app = express();++app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);++app.put('/api/v2/:app/objects/:id/multipart/:uploadId/parts/:n', ...objectRoutes.partHandlers);++app.use(express.json());++app.use('/api/v1/:app/files', require('../server/files/routes'));++app.use('/api/v2/:app/objects', objectRoutes);++app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));++app.use('/api/v2/:app/namespaces', require('../server/objects/namespace-routes'));++const server = http.createServer(app);++++(async () => {++ await new Promise((r) => server.listen(0, '127.0.0.1', r));++ const base = `http://127.0.0.1:${server.address().port}`;++ const local = (u) => base + String(u).replace('https://media.test', '');++ const call = async (method, p, { bearer = 'live-key', body, raw, headers = {} } = {}) => {++ const h = { ...headers };++ if (bearer) h.authorization = `Bearer ${bearer}`;++ let payload;++ if (raw !== undefined) payload = raw;++ else if (body !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }++ const res = await fetch(p.startsWith('http') ? p : base + p, { method, headers: h, body: payload, redirect: 'manual' });++ const text = await res.text();++ let json = null; try { json = JSON.parse(text); } catch { /* not json */ }++ return { status: res.status, body: json, text };++ };++ const C = '/api/v2/community/objects';++ // init → PUT → complete; returns the ready object (or the failing answer).++ const store = async (url, bearer, bytes, extra = {}) => {++ let r = await call('POST', url, { bearer, body: { kind: 'file', mime_type: 'application/octet-stream', size_bytes: bytes.length, ...extra } });++ if (r.status !== 201) return r;++ const id = r.body.id;++ r = await call('PUT', local(r.body.upload.url), { bearer: null, raw: bytes, headers: { 'content-type': 'application/octet-stream' } });++ if (r.status !== 200) return r;++ return call('POST', `${url}/${id}/complete`, { bearer, body: {} });++ };++++ // ── 2. Five verbs; the older ids keep granting the newer ones ──++ const UP = ['media.object.upload'];++ let r = await store(C, svc([...UP, 'media.object.read']), Buffer.from('root object'));++ assert.strictEqual(r.status, 200, JSON.stringify(r.body));++ const rootObj = r.body;++ assert.strictEqual(rootObj.namespace, 'community', 'the root namespace by default');++ r = await call('GET', C, { bearer: svc(['media.object.read']) });++ assert.strictEqual(r.status, 200, 'read still lists (tokens from before the split)');++ r = await call('GET', C, { bearer: svc(['media.object.list']) });++ assert.deepStrictEqual([r.status, r.body.objects.map(o => o.id)], [200, [rootObj.id]], 'media.object.list lists');++ r = await call('GET', `${C}/${rootObj.id}`, { bearer: svc(['media.object.list']) });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'list is not read');++ r = await call('POST', C, { bearer: svc(['media.object.delete']), body: { kind: 'file' } });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'delete is not write');++ r = await call('DELETE', `${C}/${rootObj.id}`, { bearer: svc(['media.object.read']) });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'read does not delete');++ r = await call('DELETE', `${C}/${rootObj.id}`, { bearer: svc(['media.object.delete']) });++ assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'deleted'], 'media.object.delete deletes');++ r = await call('POST', `${C}/${rootObj.id}/restore`, { bearer: svc(UP) });++ assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'ready'], 'upload still restores (and deletes)');++ r = await call('POST', '/api/v2/community/jobs', { bearer: svc(['media.object.read']), body: { type: 'invariant.scan' } });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'transform needs its grant');++ r = await call('POST', '/api/v2/community/jobs', { bearer: svc(['media.derivative.create']), body: { type: 'invariant.scan' } });++ assert.strictEqual(r.status, 202, `media.derivative.create transforms: ${JSON.stringify(r.body)}`);++ const job = r.body.job;++ r = await call('POST', `/api/v2/community/jobs/${job.id}/cancel`, { bearer: svc(UP) });++ assert.strictEqual(r.status, 200, 'upload still transforms');++ r = await call('GET', '/api/v2/community/jobs', { bearer: svc(['media.object.list']) });++ assert.strictEqual(r.status, 200, 'jobs are listed with list');++ r = await call('GET', `/api/v1/community/files`, { bearer: svc(['media.object.list']) });++ assert.strictEqual(r.status, 200, 'v1 files list with list');++ r = await call('POST', C, { bearer: svc(UP, ['live']), body: {} });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'another tenant\'s namespace');++++ // strict_verbs: only the verb's own id counts in that namespace.++ db.run(`UPDATE media_namespaces SET policy = '{"strict_verbs":true}' WHERE namespace = 'community'`);++ r = await call('GET', C, { bearer: svc(['media.object.read']) });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'strict: read no longer lists');++ r = await call('DELETE', `${C}/${rootObj.id}`, { bearer: svc(UP) });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'strict: upload no longer deletes');++ r = await call('GET', C, { bearer: svc(['media.object.list']) });++ assert.strictEqual(r.status, 200);++ db.run("UPDATE media_namespaces SET policy = '{}' WHERE namespace = 'community'");++ console.log('✅ verbs: read, list, write, delete, transform; older ids still grant; strict_verbs refuses them');++++ // ── 3. Child namespaces; grants that name only a child ──++ r = await store(C, svc(UP, ['community.*']), Buffer.from('an avatar'), { namespace: 'avatars' });++ assert.deepStrictEqual([r.status, r.body.namespace], [200, 'community.avatars'], 'a relative name lands below the root');++ const avatar = r.body;++ assert.strictEqual(namespaces.get('community.avatars').parent, 'community');++ r = await call('POST', C, { bearer: svc(UP), body: { namespace: 'avatars' } });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'a grant of the root alone is not its children');++ for (const bad of ['Avatars', 'a..b', 'a.b.c.d', '../x', 'live', '-x']) {++ r = await call('POST', C, { bearer: svc(UP, ['community.*']), body: { namespace: bad } });++ if (bad === 'live') { assert.deepStrictEqual([r.status, r.body.namespace === undefined], [201, true]); continue; } // relative: community.live++ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.namespace.invalid'], bad);++ }++ assert.ok(namespaces.get('community.live'), '"live" is relative to the tenant');++ const onlyAvatars = (cap) => svc(cap, ['community.avatars']);++ r = await call('POST', C, { bearer: onlyAvatars(UP), body: {} });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'a child grant does not write the root');++ r = await store(C, onlyAvatars(UP), Buffer.from('another avatar'), { namespace: 'community.avatars' });++ assert.deepStrictEqual([r.status, r.body.namespace], [200, 'community.avatars'], 'the full name works too');++ r = await call('GET', C, { bearer: onlyAvatars(['media.object.list']) });++ assert.ok(r.body.objects.length === 2 && r.body.objects.every(o => o.namespace === 'community.avatars'), 'a child grant lists only its namespace');++ r = await call('GET', `${C}/${rootObj.id}`, { bearer: onlyAvatars(['media.object.read']) });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'nor reads the root');++ r = await call('GET', `${C}?namespace=avatars`, { bearer: svc(['media.object.list'], ['community.*']) });++ assert.ok(r.body.objects.length === 2 && r.body.objects.every(o => o.namespace === 'community.avatars'), '?namespace= narrows the list');++ r = await call('GET', `${C}?namespace=community`, { bearer: onlyAvatars(['media.object.list']) });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied']);++ r = await call('POST', '/api/v2/community/jobs', { bearer: onlyAvatars(['media.derivative.create']), body: { type: 'object.hash', object_id: avatar.id } });++ assert.ok(r.status === 202 || r.status === 400, `transform on an object in the granted child: ${JSON.stringify(r.body)}`);++ assert.notStrictEqual(r.body.code, 'capability.namespace_denied');++ r = await call('POST', '/api/v2/community/jobs', { bearer: onlyAvatars(['media.derivative.create']), body: { type: 'invariant.scan' } });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'a tenant-wide job is the root\'s');++ // At most MEDIA_NAMESPACE_MAX_CHILDREN (6) below a root: avatars, live, then four more.++ for (const n of ['c1', 'c2', 'c3.d']) {++ r = await call('POST', C, { bearer: svc(UP, ['community.*']), body: { namespace: n } });++ assert.strictEqual(r.status, 201, `${n}: ${JSON.stringify(r.body)}`);++ }++ r = await call('POST', C, { bearer: svc(UP, ['community.*']), body: { namespace: 'c4' } });++ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.namespaces_exceeded']);++ console.log('✅ child namespaces: created by the first upload, granted and listed per namespace, bounded per tenant');++++ // ── 4. Developer app tokens: app.<project_id>.* ──++ const A = `/api/v2/${PA}/objects`;++ r = await store(A, appTok(), Buffer.from('legacy ns'), { namespace: 'uploads' });++ assert.deepStrictEqual([r.status, r.body.namespace], [200, `app.${PA}.uploads`], 'a legacy ns [project_id] means the whole project');++ r = await store(A, appTok({ ns: [PA, `app.${PA}.*`] }), Buffer.from('new ns'));++ assert.deepStrictEqual([r.status, r.body.namespace], [200, `app.${PA}`], 'the ns Network issues now');++ r = await call('POST', A, { bearer: appTok({ ns: [`app.${PA}.uploads`] }), body: {} });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'a token for one child of the project');++ r = await call('POST', A, { bearer: appTok({ ns: [`app.${PA}.uploads`] }), body: { namespace: 'uploads' } });++ assert.strictEqual(r.status, 201);++ r = await call('POST', A, { bearer: appTok({ ns: [`app.${PB}.*`] }), body: {} });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'another project\'s namespace');++ r = await call('POST', A, { bearer: appTok(), body: { namespace: 'sandbox' } });++ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.namespace.invalid'], 'the sandbox tenant is not a child of production');++ r = await store(A, appTok({ env: 'sandbox' }), Buffer.from('sandbox'), { namespace: 'uploads' });++ assert.deepStrictEqual([r.status, r.body.namespace], [200, `app.${PA}.sandbox.uploads`], 'sandbox tokens work in the sandbox tenant');++ r = await call('GET', `/api/v2/${PA}/namespaces`, { bearer: appTok() });++ assert.deepStrictEqual(r.body.namespaces.map(n => n.namespace), [`app.${PA}`, `app.${PA}.uploads`], 'the owner reads its production namespaces');++ const rootNs = r.body.namespaces[0];++ assert.deepStrictEqual([rootNs.root, rootNs.owner, rootNs.quota.bytes, rootNs.quota.bytes_from, rootNs.usage.used_objects, rootNs.usage.used_bytes],++ [true, `project:${PA}`, 1000000, 'tenant', 3, 10 + 9 + 6], 'root usage covers the tenant; the quota is the tenant\'s');++ assert.strictEqual(r.body.namespaces[1].usage.reserved_objects, 1, 'the open upload in uploads is a reservation');++ r = await call('GET', `/api/v2/${PA}/namespaces/uploads`, { bearer: appTok({ cap: ['media.object.read'] }) });++ assert.deepStrictEqual([r.status, r.body.namespace, r.body.usage.used_bytes], [200, `app.${PA}.uploads`, 9]);++ r = await call('GET', `/api/v2/${PA}/namespaces`, { bearer: appTok({ env: 'sandbox' }) });++ assert.deepStrictEqual(r.body.namespaces.map(n => [n.namespace, n.sandbox]), [[`app.${PA}.sandbox`, true], [`app.${PA}.sandbox.uploads`, true]]);++ r = await call('GET', `/api/v2/${PA}/namespaces/nope`, { bearer: appTok() });++ assert.deepStrictEqual([r.status, r.body.code], [404, 'media.namespace.not_found']);++ r = await call('GET', '/api/v2/community/namespaces', { bearer: onlyAvatars(['media.object.list']) });++ assert.deepStrictEqual(r.body.namespaces.map(n => n.namespace), ['community.avatars'], 'a child grant sees its namespace only');++ r = await call('GET', '/api/v2/community/namespaces/community', { bearer: onlyAvatars(['media.object.read']) });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied']);++ r = await call('GET', '/api/v2/live/namespaces');++ assert.deepStrictEqual(r.body.namespaces.map(n => n.namespace), ['live', 'live.legacy', 'live.legacy.old'], 'the app key reads every namespace of its tenant');++ console.log('✅ developer tenants: app.<project_id>.* (legacy and new ns), sandbox apart; owner reads namespaces and usage');++++ // ── 5. Quotas: per namespace, reserved at init, reconciled at complete, released, swept ──++ const T = '/api/v2/tiny/objects';++ db.run("INSERT INTO media_namespaces (namespace, app_id, parent, owner, quota_bytes, quota_objects) VALUES ('tiny.box', 'tiny', 'tiny', 'service:tiny', 30, 2)");++ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 31 } });++ assert.deepStrictEqual([r.status, r.body.code, r.body.namespace, r.body.quota_bytes], [413, 'media.quota.exceeded', 'tiny.box', 30], 'the child\'s own byte quota');++ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 20 } });++ assert.strictEqual(r.status, 201);++ const boxed = r.body;++ assert.strictEqual(namespaces.reservation(boxed.id).bytes, 20, 'init reserves the declared size');++ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 11 } });++ assert.strictEqual(r.status, 413, 'the reservation counts');++ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 5 } });++ assert.strictEqual(r.status, 201);++ const second = r.body;++ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 1 } });++ assert.deepStrictEqual([r.status, r.body.code, r.body.quota_objects], [413, 'media.quota.objects_exceeded', 2], 'the child\'s object quota');++ r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 76 } });++ assert.deepStrictEqual([r.status, r.body.namespace], [413, 'tiny'], 'the tenant quota (100) covers its children\'s reservations (25)');++ // The bytes replace the declared size; complete settles the reservation, and the ready object counts.++ r = await call('PUT', local(boxed.upload.url), { bearer: null, raw: Buffer.alloc(20, 1) });++ assert.strictEqual(r.status, 200);++ r = await call('POST', `${T}/${boxed.id}/complete`, { bearer: 'tiny-key', body: {} });++ assert.strictEqual(r.status, 200);++ assert.strictEqual(namespaces.reservation(boxed.id), undefined, 'settled at complete');++ let box = namespaces.get('tiny.box');++ assert.deepStrictEqual([box.used_bytes, box.used_objects, box.reserved_bytes, box.reserved_objects], [20, 1, 5, 1], 'the snapshot is reconciled at complete');++ // Complete checks the real size: an undeclared upload that grew past the room is refused there.++ r = await call('POST', T, { bearer: 'tiny-key', body: {} });++ const grow = r.body;++ r = await call('PUT', `${T}/${grow.id}/content`, { bearer: 'tiny-key', raw: Buffer.alloc(76, 2) });++ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'the stored bytes are checked (20 + 5 + 76 > 100)');++ r = await call('PUT', `${T}/${grow.id}/content`, { bearer: 'tiny-key', raw: Buffer.alloc(40, 2) });++ assert.strictEqual(r.status, 200);++ assert.strictEqual(namespaces.reservation(grow.id).bytes, 40, 're-reserved with the bytes stored');++ db.run("UPDATE apps SET quota_bytes = 60 WHERE app_id = 'tiny'");++ r = await call('POST', `${T}/${grow.id}/complete`, { bearer: 'tiny-key', body: {} });++ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'and reconciled again at complete (20 + 5 + 40 > 60)');++ db.run("UPDATE apps SET quota_bytes = 100 WHERE app_id = 'tiny'");++ r = await call('POST', `${T}/${grow.id}/complete`, { bearer: 'tiny-key', body: {} });++ assert.strictEqual(r.status, 200);++ // Deleting an upload in progress frees its reservation.++ r = await call('DELETE', `${T}/${second.id}`, { bearer: 'tiny-key' });++ assert.strictEqual(r.status, 200);++ assert.strictEqual(namespaces.reservation(second.id), undefined, 'a deleted upload holds nothing');++ box = namespaces.get('tiny.box');++ assert.deepStrictEqual([box.reserved_bytes, box.reserved_objects], [0, 0]);++++ // Multipart: the session holds the declared size; abort releases it; a new session reserves again.++ db.run("UPDATE apps SET quota_bytes = ? WHERE app_id = 'tiny'", [3 * 1024 * 1024]);++ db.run("UPDATE media_namespaces SET quota_bytes = NULL, quota_objects = NULL WHERE namespace = 'tiny.box'");++ const MB = 1024 * 1024;++ r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 2 * MB, multipart: true, part_size: MB } });++ assert.strictEqual(r.status, 201, JSON.stringify(r.body));++ const mp = r.body;++ r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 2 * MB } });++ assert.strictEqual(r.status, 413, 'the open session holds its 2 MB');++ r = await call('DELETE', local(mp.upload.multipart.abort_url), { bearer: null });++ assert.strictEqual(r.status, 200);++ assert.strictEqual(namespaces.reservation(mp.id), undefined, 'abort released it');++ r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 2 * MB, multipart: true, part_size: MB } });++ assert.strictEqual(r.status, 201, 'the room is back');++ const other = r.body;++ r = await call('POST', `${T}/${mp.id}/multipart`, { bearer: 'tiny-key', body: { part_size: MB } });++ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'a new session for the aborted upload is checked again');++ r = await call('DELETE', local(other.upload.multipart.abort_url), { bearer: null });++ r = await call('POST', `${T}/${mp.id}/multipart`, { bearer: 'tiny-key', body: { part_size: MB } });++ assert.strictEqual(r.status, 201, 'and reserves again when there is room');++ assert.strictEqual(namespaces.reservation(mp.id).bytes, 2 * MB);++++ // Expiry: an abandoned upload is failed and its bytes freed; an open session keeps its upload.++ r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 3 } });++ const abandoned = r.body;++ await call('PUT', local(abandoned.upload.url), { bearer: null, raw: Buffer.from('abc') });++ const file = model.objectFilePath(model.getObject(abandoned.id));++ assert.ok(fs.existsSync(file));++ db.run("UPDATE media_quota_reservations SET expires_at = datetime('now', '-1 hour') WHERE object_id IN (?, ?)", [abandoned.id, mp.id]);++ const swept = namespaces.expireReservations();++ assert.deepStrictEqual(swept, { expired: 1, dropped: 0 }, 'only the upload with no open session');++ const gone = model.getObject(abandoned.id);++ assert.deepStrictEqual([gone.lifecycle_status, JSON.parse(gone.metadata).failure, fs.existsSync(file), namespaces.reservation(abandoned.id)],++ ['failed', 'upload_expired', false, undefined]);++ assert.ok(namespaces.reservation(mp.id), 'the multipart upload keeps its reservation while its session is open');++ r = await call('PUT', local(abandoned.upload.url), { bearer: null, raw: Buffer.from('abc') });++ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.not_uploading'], 'an expired upload takes no more bytes');++++ // v1 files count against the root's quotas, objects included.++ db.run("UPDATE media_namespaces SET quota_objects = 1 WHERE namespace = 'tiny'");++ const fd = new FormData();++ fd.append('file', new Blob([Buffer.from('v1 bytes')], { type: 'text/plain' }), 'x.txt');++ const f = await fetch(`${base}/api/v1/tiny/files`, { method: 'POST', headers: { authorization: 'Bearer tiny-key' }, body: fd });++ const fb = await f.json();++ assert.deepStrictEqual([f.status, fb.code, fb.namespace], [413, 'media.quota.objects_exceeded', 'tiny'], 'the v1 answer keeps its shape and says why');++ db.run("UPDATE media_namespaces SET quota_objects = NULL WHERE namespace = 'tiny'");++++ // Policy: kinds, visibilities and max_object_bytes, children inheriting and overriding.++ db.run(`UPDATE media_namespaces SET policy = '{"kinds":["file","asset"],"max_object_bytes":10}' WHERE namespace = 'tiny'`);++ db.run(`UPDATE media_namespaces SET policy = '{"visibilities":["private"],"max_object_bytes":0}' WHERE namespace = 'tiny.box'`);++ r = await call('POST', T, { bearer: 'tiny-key', body: { kind: 'vod' } });++ assert.deepStrictEqual([r.status, r.body.code], [422, 'media.namespace.policy_denied'], 'kinds');++ r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 11 } });++ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'max_object_bytes');++ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 11, visibility: 'private' } });++ assert.strictEqual(r.status, 201, 'a child overrides max_object_bytes');++ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', kind: 'vod', visibility: 'private' } });++ assert.strictEqual(r.status, 422, 'and inherits kinds');++ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', visibility: 'public' } });++ assert.deepStrictEqual([r.status, r.body.code], [422, 'media.namespace.policy_denied'], 'visibilities');++ assert.ok(namespaces.validatePolicy({ kinds: ['file'], strict_verbs: true }).policy);++ assert.ok(namespaces.validatePolicy({ nope: 1 }).error && namespaces.validatePolicy({ kinds: ['x'] }).error && namespaces.validatePolicy({ max_object_bytes: -1 }).error);++++ // The snapshot follows the rows when reconciled.++ db.run("UPDATE media_namespaces SET used_bytes = 999 WHERE namespace = 'tiny'");++ assert.ok(namespaces.reconcileAll() >= 3);++ assert.strictEqual(namespaces.get('tiny').used_bytes, model.usedBytes('tiny') - namespaces.get('tiny').reserved_bytes, 'the root\'s snapshot is the tenant\'s usage');++ console.log('✅ quotas: per namespace and tenant, bytes and objects; reserved, re-reserved, settled, released, swept; policy enforced');++++ server.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('namespaces and grants: all checks passed');++})().catch((err) => { console.error(err); process.exit(1); });
@@ -100,15 +100,18 @@ Credential types on `/api/v1/:app/...`:
@@ -100,15 +100,18 @@ Credential types on `/api/v1/:app/...`:
`allowed_origins` (CORS is reflected for allow-listed origins only).
`allowed_origins` (CORS is reflected for allow-listed origins only).
3. **Network principal token** (`sub svc:…`, `app:…`, `mod:…`; RS256, audience `openvibe.media`) —
3. **Network principal token** (`sub svc:…`, `app:…`, `mod:…`; RS256, audience `openvibe.media`) —
- accepted only on routes that name a capability: files upload/delete and every objects v2 write
- (`media.object.upload`), files list/meta and objects v2 reads (`media.object.read`), and only for
- the `:app` namespaces in the token's `ns`.
+ accepted only on routes that name a verb, each checked for the namespace it works in against the
+ token's `ns`: read (`media.object.read`), list (`media.object.list`, or read), write
+ (`media.object.upload`), delete (`media.object.delete`, or upload) and transform
+ (`media.derivative.create`, or upload). The tenant's root namespace is `:app`; children, quotas
+ and policy are in [docs/object-model.md](docs/object-model.md#namespaces-grants-and-quotas).
### Developer-project tenants (ADR-014)
### Developer-project tenants (ADR-014)
A developer app from OpenVibe.Network (token `sub app:app_<ULID>`, `project_id prj_<ULID>`,
A developer app from OpenVibe.Network (token `sub app:app_<ULID>`, `project_id prj_<ULID>`,
-`env sandbox|production`, `ns [project_id]`) holding `media.object.upload` / `media.object.read`
-gets a tenant keyed by its project id, **created on first use**. The URL always names the project
+`env sandbox|production`, `ns [project_id, app.<project_id>.*]`; an older `ns [project_id]` means the
+same) holding media verbs gets a tenant keyed by its project id, **created on first use**. Its
+namespaces are `app.<project_id>` (production) and `app.<project_id>.sandbox`, with children below. The URL always names the project
(`/api/v1/prj_<ULID>/files`, `/api/v2/prj_<ULID>/objects`, so openvibe-sdk's
(`/api/v1/prj_<ULID>/files`, `/api/v2/prj_<ULID>/objects`, so openvibe-sdk's
`createMediaClient({ app: projectId })` works); the token's `env` picks one of two separate tenants:
`createMediaClient({ app: projectId })` works); the token's `env` picks one of two separate tenants:
@@ -7,7 +7,8 @@ projection over an object through its `object_id` column. Every existing URL and
@@ -7,7 +7,8 @@ projection over an object through its `object_id` column. Every existing URL and
unchanged. New code talks to objects through the [v2 API](#object-api-v2).
unchanged. New code talks to objects through the [v2 API](#object-api-v2).
Code: `server/objects/` (`model.js`, `routes.js`, `backfill.js`, `reconcile.js`, `invariant.js`,
Code: `server/objects/` (`model.js`, `routes.js`, `backfill.js`, `reconcile.js`, `invariant.js`,
-`signing.js`, `verify-job.js`, `copy-report.js`, `multipart.js`, `content-type.js`, `readiness.js`) and the job system in
+`signing.js`, `verify-job.js`, `copy-report.js`, `multipart.js`, `content-type.js`, `readiness.js`, `namespaces.js`,
+`namespace-routes.js`) and the job system in
`server/jobs/` (`queue.js`, `worker.js`, `routes.js`, `types.js`, `thumbnail.js`, `invariant-scan.js`,
`server/jobs/` (`queue.js`, `worker.js`, `routes.js`, `types.js`, `thumbnail.js`, `invariant-scan.js`,
`derive.js`). Schema: the bottom of `server/db/schema.sql`. Tests: `test/objects-*.test.js`, `test/jobs*.test.js`,
`derive.js`). Schema: the bottom of `server/db/schema.sql`. Tests: `test/objects-*.test.js`, `test/jobs*.test.js`,
`test/r2-eviction-drill.test.js`.
`test/r2-eviction-drill.test.js`.
@@ -16,13 +17,15 @@ Code: `server/objects/` (`model.js`, `routes.js`, `backfill.js`, `reconcile.js`,
@@ -16,13 +17,15 @@ Code: `server/objects/` (`model.js`, `routes.js`, `backfill.js`, `reconcile.js`,
| table | what it holds |
| table | what it holds |
|---|---|
|---|---|
-| `media_objects` | `id` (med_…), `app_id` (tenant), `namespace` (capability namespace; = app today), `kind` (`vod` `clip` `file` `thumbnail` `screenshot` `avatar` `asset`), `owner_subject` (usr_… when known), `owner_app` + `owner_user_id` (legacy owner in the app's own user-id space), `visibility` (`public` `unlisted` `private`), `lifecycle_status` (`uploading` `ready` `failed` `archived` `deleted`), `mime_type`, `size_bytes`, `content_hash` (sha256), `canonical_provider` + `canonical_key`, `legacy_ref`, `metadata` (JSON), `created_at` / `updated_at` / `deleted_at` |
+| `media_objects` | `id` (med_…), `app_id` (tenant), `namespace` (a [`media_namespaces`](#namespaces-grants-and-quotas) row: the tenant's root or a child of it), `kind` (`vod` `clip` `file` `thumbnail` `screenshot` `avatar` `asset`), `owner_subject` (usr_… when known), `owner_app` + `owner_user_id` (legacy owner in the app's own user-id space), `visibility` (`public` `unlisted` `private`), `lifecycle_status` (`uploading` `ready` `failed` `archived` `deleted`), `mime_type`, `size_bytes`, `content_hash` (sha256), `canonical_provider` + `canonical_key`, `legacy_ref`, `metadata` (JSON), `created_at` / `updated_at` / `deleted_at` |
| `media_locations` | one row per provider copy (`UNIQUE(object_id, provider)`): `provider` (`local` `b2` `r2`), `bucket`, `key` (absolute path for local, object key for B2/R2), `storage_class` (`hot` local, `cold` B2, `cache` R2), `state`, `checksum`, `size_bytes`, `verified_at` |
| `media_locations` | one row per provider copy (`UNIQUE(object_id, provider)`): `provider` (`local` `b2` `r2`), `bucket`, `key` (absolute path for local, object key for B2/R2), `storage_class` (`hot` local, `cold` B2, `cache` R2), `state`, `checksum`, `size_bytes`, `verified_at` |
| `media_relationships` | `from_object_id`, `relation`, `to_object_id`, `metadata`. In use: `clip_of` (clip to vod, with start/end), `thumbnail_of` (thumbnail to vod/clip). Reserved: `derived_from`, `screenshot_of` |
| `media_relationships` | `from_object_id`, `relation`, `to_object_id`, `metadata`. In use: `clip_of` (clip to vod, with start/end), `thumbnail_of` (thumbnail to vod/clip). Reserved: `derived_from`, `screenshot_of` |
| `media_variants` | `object_id`, `variant_name`, `derived_object_id`, `recipe`. In use: `thumbnail` |
| `media_variants` | `object_id`, `variant_name`, `derived_object_id`, `recipe`. In use: `thumbnail` |
| `media_jobs` | derivative/maintenance jobs: `id` (mjob_…), `app_id`, `object_id`, `job_type`, `status`, `idempotency_key`, `params`, `result`, `attempts`/`max_attempts`, `run_after`, `lease_until`, `lease_token`, `checkpoint`, `error`/`error_code`, `cancel_requested`, `created_by`, `decided_by`. See [Jobs](#jobs) |
| `media_jobs` | derivative/maintenance jobs: `id` (mjob_…), `app_id`, `object_id`, `job_type`, `status`, `idempotency_key`, `params`, `result`, `attempts`/`max_attempts`, `run_after`, `lease_until`, `lease_token`, `checkpoint`, `error`/`error_code`, `cancel_requested`, `created_by`, `decided_by`. See [Jobs](#jobs) |
| `media_uploads`, `media_upload_parts` | multipart upload sessions and the parts received (size, sha256). See [Multipart uploads](#multipart-uploads) |
| `media_uploads`, `media_upload_parts` | multipart upload sessions and the parts received (size, sha256). See [Multipart uploads](#multipart-uploads) |
| `media_holds` | retention holds (with a staff `note`), see [Holds](#retention-holds) |
| `media_holds` | retention holds (with a staff `note`), see [Holds](#retention-holds) |
+| `media_namespaces` | every namespace: `namespace`, `app_id` (tenant), `parent`, `owner` (`service:<app>` or `project:<prj_…>`), `policy` (JSON), `quota_bytes`, `quota_objects`, and the usage snapshot `used_bytes`, `used_objects`, `reserved_bytes`, `reserved_objects`, `reconciled_at`. See [Namespaces](#namespaces-grants-and-quotas) |
+| `media_quota_reservations` | quota held by an upload in progress: `object_id`, `app_id`, `namespace`, `bytes`, `expires_at` |
| `media_invariant_violations` | public playback objects over the size policy, see [Invariant](#public-object-size-invariant) |
| `media_invariant_violations` | public playback objects over the size policy, see [Invariant](#public-object-size-invariant) |
Location `state`:
Location `state`:
@@ -206,7 +209,7 @@ All routes live under `/api/v2/:app/objects`.
@@ -206,7 +209,7 @@ All routes live under `/api/v2/:app/objects`.
**Auth.** One of:
**Auth.** One of:
- the app's API key;
- the app's API key;
-- a Network service token that grants `media.object.upload` (writes) or `media.object.read` (reads) for the namespace `:app`. `tenantAuth({ capability })` enforces this, as it does on v1.
+- a Network service token that holds the route's [verb](#namespaces-grants-and-quotas) for the object's namespace: write `media.object.upload`, read `media.object.read`, list `media.object.list`, delete (and restore) `media.object.delete`. The older ids still grant the newer verbs (read lists, upload deletes), unless the namespace's policy is `strict_verbs`. The tenant's root namespace is `:app`.
- for `:app = prj_<ULID>` (a developer-project tenant, ADR-014): only that project's app tokens, production or sandbox. The token's `env` selects the tenant (`prj_<ULID>` or `prj_<ULID>-sandbox`), and upload URLs always name the project. Sandbox objects never get a `public_url`; `/download` signs them and `/o/:id` needs the signature, whatever their visibility. Retention holds are not available to app tokens. See the README's "Developer-project tenants".
- for `:app = prj_<ULID>` (a developer-project tenant, ADR-014): only that project's app tokens, production or sandbox. The token's `env` selects the tenant (`prj_<ULID>` or `prj_<ULID>-sandbox`), and upload URLs always name the project. Sandbox objects never get a `public_url`; `/download` signs them and `/o/:id` needs the signature, whatever their visibility. Retention holds are not available to app tokens. See the README's "Developer-project tenants".
**Owner.**
**Owner.**
@@ -218,7 +221,7 @@ All routes live under `/api/v2/:app/objects`.
@@ -218,7 +221,7 @@ All routes live under `/api/v2/:app/objects`.
| method | path | notes |
| method | path | notes |
|---|---|---|
|---|---|---|
-| POST | `/` | init. `{ kind, visibility (default private), mime_type, size_bytes, filename, content_hash (expected sha256), metadata (≤ 16 KB), user_id, upload_ttl (60-86400 s), multipart, part_size }` returns 201 `{ id, object, upload: { method: 'PUT', url, token, expires_at, max_bytes, content_type, complete_url, multipart_url } }`. `url` is a [presigned PUT URL](#presigned-uploads). With `multipart: true` (required above `MEDIA_OBJECT_MAX_MB`): `method: 'multipart'`, `url: null`, and `upload.multipart` is the new [session](#multipart-uploads). Checks at init: `size_bytes` against `MEDIA_OBJECT_MAX_MB` (single part) or `MEDIA_MULTIPART_MAX_MB` (413), the quota including reservations (413 `media.quota.exceeded`), the public-size invariant (422) and the [content type](#content-types) for the kind (415 `media.object.unsupported_type`) |
+| POST | `/` | init. `{ namespace (default the tenant's root; a name below it, full or relative), kind, visibility (default private), mime_type, size_bytes, filename, content_hash (expected sha256), metadata (≤ 16 KB), user_id, upload_ttl (60-86400 s), multipart, part_size }` returns 201 `{ id, object, upload: { method: 'PUT', url, token, expires_at, max_bytes, content_type, complete_url, multipart_url } }`. `url` is a [presigned PUT URL](#presigned-uploads). With `multipart: true` (required above `MEDIA_OBJECT_MAX_MB`): `method: 'multipart'`, `url: null`, and `upload.multipart` is the new [session](#multipart-uploads). Checks at init: `size_bytes` against `MEDIA_OBJECT_MAX_MB` (single part) or `MEDIA_MULTIPART_MAX_MB` (413), the namespace's policy (422 `media.namespace.policy_denied`, 413 `media.object.too_large`), every quota from the namespace up to the tenant including reservations (413 `media.quota.exceeded` / `media.quota.objects_exceeded`; the declared size is then reserved), the public-size invariant (422) and the [content type](#content-types) for the kind (415 `media.object.unsupported_type`) |
| POST | `/:id/upload-url` | a fresh presigned single-PUT URL for an uploading object (`{ ttl }` 60-86400 s) |
| POST | `/:id/upload-url` | a fresh presigned single-PUT URL for an uploading object (`{ ttl }` 60-86400 s) |
| PUT | `/:id/content` | the bytes, single part. Auth is the upload token (`?token=` from init, or `X-Upload-Token`) or the usual credential. Streams to disk computing sha256. Refuses more than the declared size or the limit (413; an object declared above `MEDIA_OBJECT_MAX_MB` goes multipart), a size mismatch (400), quota overrun (413), a type that does not suit the kind (415) and a PUT while a multipart session is open (409). Stored at `OBJECTS_PATH/<app>/<id>` with a verified local location. Mounted ahead of the JSON body parser, so any Content-Type is taken as raw bytes; the Content-Type becomes `mime_type` when init set none. Can be repeated while the object is `uploading` |
| PUT | `/:id/content` | the bytes, single part. Auth is the upload token (`?token=` from init, or `X-Upload-Token`) or the usual credential. Streams to disk computing sha256. Refuses more than the declared size or the limit (413; an object declared above `MEDIA_OBJECT_MAX_MB` goes multipart), a size mismatch (400), quota overrun (413), a type that does not suit the kind (415) and a PUT while a multipart session is open (409). Stored at `OBJECTS_PATH/<app>/<id>` with a verified local location. Mounted ahead of the JSON body parser, so any Content-Type is taken as raw bytes; the Content-Type becomes `mime_type` when init set none. Can be repeated while the object is `uploading` |
| POST | `/:id/complete` | checks the expected hash (422 `media.object.hash_mismatch`), the invariant, the quota and the bytes against the type (415 `media.object.content_mismatch`). Sets `ready` and sends the `media.object.uploaded` webhook. Also accepts the upload token |
| POST | `/:id/complete` | checks the expected hash (422 `media.object.hash_mismatch`), the invariant, the quota and the bytes against the type (415 `media.object.content_mismatch`). Sets `ready` and sends the `media.object.uploaded` webhook. Also accepts the upload token |
@@ -226,9 +229,9 @@ All routes live under `/api/v2/:app/objects`.
@@ -226,9 +229,9 @@ All routes live under `/api/v2/:app/objects`.
| GET | `/:id/multipart/:uploadId` | the session: `parts` received (number, size, sha256), `missing`, `received_bytes`, `expires_at` |
| GET | `/:id/multipart/:uploadId` | the session: `parts` received (number, size, sha256), `missing`, `received_bytes`, `expires_at` |
| PUT | `/:id/multipart/:uploadId/parts/:n` | one part, raw bytes of exactly its size; optional `X-Content-SHA256` |
| PUT | `/:id/multipart/:uploadId/parts/:n` | one part, raw bytes of exactly its size; optional `X-Content-SHA256` |
| POST | `/:id/multipart/:uploadId/complete` | `{ content_hash, parts: [{ part_number, sha256 }] }` (both optional): assembles, then the checks of `/complete` |
| POST | `/:id/multipart/:uploadId/complete` | `{ content_hash, parts: [{ part_number, sha256 }] }` (both optional): assembles, then the checks of `/complete` |
-| DELETE | `/:id/multipart/:uploadId` | abort: parts deleted, the object stays `uploading` |
+| DELETE | `/:id/multipart/:uploadId` | abort: parts deleted and the quota reservation released, the object stays `uploading` (a new session or upload URL reserves again) |
| GET | `/:id` | metadata: providers and states of each copy, never paths or keys, and its [readiness](#readiness) |
| GET | `/:id` | metadata: providers and states of each copy, never paths or keys, and its [readiness](#readiness) |
-| GET | `/` | cursor list, newest first. `?limit (≤200)&cursor&kind&visibility&status&owner (usr_…)&user_id&include_deleted` returns `{ objects, next_cursor }` |
+| GET | `/` | cursor list, newest first. `?limit (≤200)&cursor&namespace (and below)&kind&visibility&status&owner (usr_…)&user_id&include_deleted` returns `{ objects, next_cursor }`. A Network token sees only the namespaces it may list |
| DELETE | `/:id` | soft delete: `lifecycle_status = deleted`, and the bytes are kept for `MEDIA_DELETE_RETENTION_DAYS` (default 30). 409 `media.object.held` under a hold. 409 `media.object.legacy_managed` for projected objects, which are deleted through their v1 route |
| DELETE | `/:id` | soft delete: `lifecycle_status = deleted`, and the bytes are kept for `MEDIA_DELETE_RETENTION_DAYS` (default 30). 409 `media.object.held` under a hold. 409 `media.object.legacy_managed` for projected objects, which are deleted through their v1 route |
| POST | `/:id/restore` | undo a soft delete within the retention period (410 once purged) |
| POST | `/:id/restore` | undo a soft delete within the retention period (410 once purged) |
| GET | `/:id/download` | public or unlisted: 302 to the public location (the inherited URL for projected objects such as `/v/:id`, `/c/:id`, `/f/:key`, `/t/:name`, `/p/:slug/screenshot`; otherwise `/o/:id`). Private: `{ url, expires_at }` signed for `?ttl` seconds (30 to 3600, default `MEDIA_SIGNED_URL_TTL_S` = 300), or a 302 with `?redirect=1`. `?format=json` always answers JSON. 410 when deleted |
| GET | `/:id/download` | public or unlisted: 302 to the public location (the inherited URL for projected objects such as `/v/:id`, `/c/:id`, `/f/:key`, `/t/:name`, `/p/:slug/screenshot`; otherwise `/o/:id`). Private: `{ url, expires_at }` signed for `?ttl` seconds (30 to 3600, default `MEDIA_SIGNED_URL_TTL_S` = 300), or a 302 with `?redirect=1`. `?format=json` always answers JSON. 410 when deleted |
@@ -311,14 +314,90 @@ Code: `server/objects/content-type.js`.
@@ -311,14 +314,90 @@ Code: `server/objects/content-type.js`.
- Everything else is served as an attachment with `nosniff`, so its declared type is taken as given.
- Everything else is served as an attachment with `nosniff`, so its declared type is taken as given.
The declared type wins over the PUT's `Content-Type` (declared at init: the PUT header is ignored).
The declared type wins over the PUT's `Content-Type` (declared at init: the PUT header is ignored).
-**Quota.** `apps.quota_bytes` is compared against v1 file bytes plus native objects that are
-`uploading` (their declared size counts as a reservation) or `ready`. Soft-deleted objects do not
-count, except in developer-project tenants, where they count until their bytes are purged. The v1 files route still counts only v1 files, so its response is unchanged.
+**Quota.** Per namespace, up to the tenant: see [Namespaces, grants and quotas](#namespaces-grants-and-quotas).
**Purge.** Every hour, the service removes the bytes of native objects whose retention period has
**Purge.** Every hour, the service removes the bytes of native objects whose retention period has
passed and that have no active hold. The object row stays `deleted` with `metadata.purged_at`.
passed and that have no active hold. The object row stays `deleted` with `metadata.purged_at`.
Projected objects are never purged by this job.
Projected objects are never purged by this job.
+## Namespaces, grants and quotas
+
+Roadmap WS-G task 2. Code: `server/objects/namespaces.js` (rows, policy, quotas, reservations),
+`server/auth.js` (`VERBS`, `tenantAuth({ verb, namespaced })`, `namespaceGrant`),
+`server/objects/namespace-routes.js`, `scripts/namespaces.js`. Test: `test/namespaces.test.js`.
+
+**Namespaces are rows.** Every object's `namespace` is a `media_namespaces` row. A tenant has one root:
+
+| tenant | root namespace |
+|---|---|
+| first-party (`live`, `community`, `tools`, …) | its app id |
+| developer project, production (`prj_<ULID>`) | `app.<project_id>` |
+| developer project, sandbox (`prj_<ULID>-sandbox`) | `app.<project_id>.sandbox` |
+
+Children sit below a root (`<root>.<segment>`, up to three segments of `a-z 0-9 _ -`). Init creates one
+the first time it is named (`{ namespace: "avatars" }` or the full name), at most
+`MEDIA_NAMESPACE_MAX_CHILDREN` per tenant (413 `media.quota.namespaces_exceeded`). `owner` is
+`service:<app_id>` or `project:<prj_…>`. v1 files, projected rows and job outputs live in the root
+(outputs in their source's namespace).
+
+**Grants: five verbs, one capability each, per namespace.** Network issues them (service principals'
+`principal_grants`, developer apps' grants) and the token's `ns` names the namespaces.
+
+| verb | capability | also granted by (tokens issued before the split) | routes |
+|---|---|---|---|
+| read | `media.object.read` | | object metadata, download, holds, a multipart session, a job, one namespace, v1 file meta |
+| list | `media.object.list` | `media.object.read` | object list, job list, namespace list, v1 file list |
+| write | `media.object.upload` | | init, upload URL, content, complete, multipart, v1 file upload |
+| delete | `media.object.delete` | `media.object.upload` | object delete and restore, v1 file delete |
+| transform | `media.derivative.create` | `media.object.upload` | job create, approve, cancel (the job's object's namespace; the root for a tenant-wide job) |
+
+A namespace whose policy sets `strict_verbs: true` accepts only the verb's own capability. A grant
+of `ns: ["community"]` is the root alone; `community.*` covers its children; a token for one child
+(`community.avatars`) reaches the tenant but only that namespace. A developer app token's `ns` entry
+equal to its `project_id` (what Network issued before) means `app.<project_id>.*`; Network now issues
+both. The app key and upload/session tokens are unrestricted within their tenant or object.
+
+**Policy** (JSON, set by the operator; children inherit it key by key): `kinds`, `visibilities`
+(422 `media.namespace.policy_denied` at init), `max_object_bytes` (413 `media.object.too_large`),
+`strict_verbs`.
+
+**Quotas.** `quota_bytes` and `quota_objects` limit a namespace's subtree. `quota_bytes` NULL
+inherits: the root takes its tenant's `apps.quota_bytes`, a child has no limit of its own; 0 is no
+limit. Usage counts ready native objects (in a developer project also soft-deleted ones until their
+bytes are purged), the tenant's v1 files (at the root) and reservations:
+
+1. **Init** checks every limit from the namespace up to the root with the declared size and one more
+ object, and reserves the declared size (`media_quota_reservations`), in one transaction with the
+ object row.
+2. **The bytes** (PUT, or a new multipart session or upload URL) are checked again and replace the
+ reservation's size.
+3. **Complete** checks the real size against everything else and settles the reservation: the ready
+ object counts from then on. The snapshot is reconciled.
+4. **Abort** of a multipart session (or its expiry) releases the reservation down to bytes still stored.
+ Deleting an upload settles it.
+5. **Expiry.** A reservation lasts `MEDIA_UPLOAD_RESERVATION_HOURS` (72) after the upload's last step.
+ The hourly sweep then marks the upload `failed` (`metadata.failure: upload_expired`), deletes its
+ stored bytes and drops the reservation, unless a multipart session is still open.
+
+Refusals are 413 problems: `media.quota.exceeded` (`namespace`, `quota_bytes`, `used_bytes`,
+`reserved_bytes`) or `media.quota.objects_exceeded` (`namespace`, `quota_objects`, `used_objects`,
+`reserved_objects`). The v1 file upload keeps its JSON shape and adds `code`. Split and remux check
+the source namespace's quotas. The `used_*`/`reserved_*` columns are a snapshot: checks always count
+from the rows. It is refreshed at complete, delete and restore, when read, and hourly.
+
+**Reads.** `GET /api/v2/:app/namespaces` lists the tenant's namespaces the caller may list, and
+`GET /api/v2/:app/namespaces/:namespace` reads one (full or relative name): owner, own and
+effective policy, quota (`bytes`, `objects`, `bytes_from: tenant|namespace`) and usage, counted when
+read. The operator lists every tenant's namespaces and sets quotas and policy with
+`node scripts/namespaces.js [--app <tenant>] [--json]` and
+`--set <namespace> [--quota-bytes <n>|inherit] [--quota-objects <n>|none] [--policy '<json>'] [--dry-run]`.
+
+**Backfill** (every open, idempotent; `server/db/database.js` `migrateNamespaces`): a developer
+project's objects move from the tenant id (`prj_…`, `prj_…-sandbox`) to `app.<project_id>[.sandbox]`;
+every tenant gets its root row and every namespace an object names gets a row. Once
+(`media_settings` `namespaces.reservations_seeded`), uploads already in progress get a reservation of
+their declared size for the full window from that open.
+
## Readiness
## Readiness
Code: `server/objects/readiness.js`. Test: `test/readiness.test.js`.
Code: `server/objects/readiness.js`. Test: `test/readiness.test.js`.
@@ -580,8 +659,8 @@ may be queued or running (429 `media.job.too_many`).
@@ -580,8 +659,8 @@ may be queued or running (429 `media.job.too_many`).
**Cancellation, by the owner.** A proposed or queued job is cancelled at once. A running job gets
**Cancellation, by the owner.** A proposed or queued job is cancelled at once. A running job gets
`cancel_requested`; the worker aborts it (its ffmpeg is killed) and it ends `cancelled`. Finished
`cancel_requested`; the worker aborts it (its ffmpeg is killed) and it ends `cancelled`. Finished
-jobs answer 409. The owner is the tenant (app key, service token for the namespace, or the project's
-app token); a caller acting for one of the app's users (`X-OV-User-Id`) sees and decides only the jobs
+jobs answer 409. The owner is the tenant (app key, a service token or the project's app token holding
+the `transform` [verb](#namespaces-grants-and-quotas) for the job's object's namespace); a caller acting for one of the app's users (`X-OV-User-Id`) sees and decides only the jobs
it created and the jobs on objects it owns.
it created and the jobs on objects it owns.
**Worker.** In-process, polling every `MEDIA_JOBS_POLL_MS`, in three lanes: `light`
**Worker.** In-process, polling every `MEDIA_JOBS_POLL_MS`, in three lanes: `light`
@@ -731,6 +810,8 @@ Not run against production yet: it needs the owner's go-ahead.
@@ -731,6 +810,8 @@ Not run against production yet: it needs the owner's go-ahead.
| `MEDIA_MULTIPART_MIN_PART_MB` / `_MAX_PART_MB` / `_PART_MB` | 5 / 256 / 64 | part size bounds and default |
| `MEDIA_MULTIPART_MIN_PART_MB` / `_MAX_PART_MB` / `_PART_MB` | 5 / 256 / 64 | part size bounds and default |
| `MEDIA_MULTIPART_TTL_HOURS` | 24 | an unfinished multipart session is purged after this |
| `MEDIA_MULTIPART_TTL_HOURS` | 24 | an unfinished multipart session is purged after this |
| `MEDIA_UPLOAD_MIN_FREE_MB` | 10240 | free disk kept in reserve by multipart uploads and split/remux jobs |
| `MEDIA_UPLOAD_MIN_FREE_MB` | 10240 | free disk kept in reserve by multipart uploads and split/remux jobs |
+| `MEDIA_UPLOAD_RESERVATION_HOURS` | 72 | an upload's quota reservation lasts this long after its last step; then the hourly sweep fails it and frees its bytes |
+| `MEDIA_NAMESPACE_MAX_CHILDREN` | 100 | namespaces a tenant may have below its root |
| `MEDIA_JOBS_ENABLED` | on | `0` stops the job worker |
| `MEDIA_JOBS_ENABLED` | on | `0` stops the job worker |
| `MEDIA_JOBS_POLL_MS` | 5000 | worker poll interval |
| `MEDIA_JOBS_POLL_MS` | 5000 | worker poll interval |
| `MEDIA_JOBS_LIGHT_CONCURRENCY` / `_HEAVY_CONCURRENCY` / `_FINALIZE_CONCURRENCY` | 2 / 1 / 1 | jobs per lane |
| `MEDIA_JOBS_LIGHT_CONCURRENCY` / `_HEAVY_CONCURRENCY` / `_FINALIZE_CONCURRENCY` | 2 / 1 / 1 | jobs per lane |
new file mode 100644
new file mode 100644
@@ -0,0 +1,77 @@
@@ -0,0 +1,77 @@
+#!/usr/bin/env node
+'use strict';
+/**
+ * Namespaces for the operator (roadmap WS-G task 2; server/objects/namespaces.js). Lists every
+ * namespace with its owner, quota and usage, and sets a namespace's quotas and policy.
+ *
+ * node scripts/namespaces.js [--app <tenant>] [--json] list (usage counted from the rows now)
+ * node scripts/namespaces.js --set <namespace> [--quota-bytes <n>|inherit] [--quota-objects <n>|none]
+ * [--policy '<json>'] [--dry-run]
+ *
+ * --quota-bytes: 0 is no limit; `inherit` clears it (a root then takes its tenant's apps.quota_bytes,
+ * a child has none of its own). --quota-objects: 0 or `none` is no limit. --policy replaces the
+ * namespace's own policy (kinds, visibilities, max_object_bytes, strict_verbs; children inherit it,
+ * key by key). Listing refreshes the usage snapshot; nothing else is written without --set.
+ */
+const args = process.argv.slice(2);
+const has = (name) => args.includes(`--${name}`);
+const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 && args[i + 1] != null && !args[i + 1].startsWith('--') ? args[i + 1] : dflt; };
+
+function fail(msg) { console.error(`namespaces: ${msg}`); process.exit(1); }
+const human = (b) => (b == null ? '-' : b >= 1024 ** 3 ? `${(b / 1024 ** 3).toFixed(2)} GB` : b >= 1024 ** 2 ? `${(b / 1024 ** 2).toFixed(1)} MB` : `${b} B`);
+
+try {
+ const db = require('../server/db/database');
+ const namespaces = require('../server/objects/namespaces');
+
+ if (arg('set')) {
+ const row = namespaces.get(arg('set'));
+ if (!row) fail(`no namespace ${arg('set')}`);
+ const sets = {};
+ if (arg('quota-bytes') != null) {
+ const v = arg('quota-bytes');
+ if (v === 'inherit') sets.quota_bytes = null;
+ else if (/^\d+$/.test(v)) sets.quota_bytes = Number(v);
+ else fail('--quota-bytes takes a whole number of bytes or `inherit`');
+ }
+ if (arg('quota-objects') != null) {
+ const v = arg('quota-objects');
+ if (v === 'none') sets.quota_objects = null;
+ else if (/^\d+$/.test(v)) sets.quota_objects = Number(v) || null;
+ else fail('--quota-objects takes a whole number or `none`');
+ }
+ if (arg('policy') != null) {
+ let p;
+ try { p = JSON.parse(arg('policy')); } catch { fail('--policy must be JSON'); }
+ const v = namespaces.validatePolicy(p);
+ if (v.error) fail(v.error);
+ sets.policy = JSON.stringify(v.policy);
+ }
+ if (!Object.keys(sets).length) fail('--set needs --quota-bytes, --quota-objects or --policy');
+ console.log(`${row.namespace}: ${Object.entries(sets).map(([k, v]) => `${k} ${row[k] ?? 'NULL'} -> ${v ?? 'NULL'}`).join(', ')}${has('dry-run') ? ' (dry run)' : ''}`);
+ if (!has('dry-run')) {
+ db.run(`UPDATE media_namespaces SET ${Object.keys(sets).map(k => `${k} = ?`).join(', ')}, updated_at = CURRENT_TIMESTAMP WHERE namespace = ?`, [...Object.values(sets), row.namespace]);
+ }
+ process.exit(0);
+ }
+
+ const rows = arg('app') ? namespaces.listForTenant(arg('app')) : db.all('SELECT * FROM media_namespaces ORDER BY app_id, namespace');
+ const apps = new Map();
+ const out = rows.map((row) => {
+ if (!apps.has(row.app_id)) apps.set(row.app_id, db.getApp(row.app_id) || { app_id: row.app_id });
+ const app = apps.get(row.app_id);
+ return { app_id: row.app_id, ...namespaces.publicShape(app, row, namespaces.reconcile(app, row.namespace)) };
+ });
+ if (has('json')) { console.log(JSON.stringify({ namespaces: out }, null, 2)); process.exit(0); }
+ for (const n of out) {
+ const u = n.usage;
+ console.log(`${n.namespace} [${n.app_id}${n.root ? ', root' : ''}] ${n.owner}`
+ + ` used ${human(u.used_bytes)} / ${n.quota.bytes ? human(n.quota.bytes) : 'no limit'}${n.quota.bytes_from === 'tenant' ? ' (tenant)' : ''}`
+ + `, ${u.used_objects} object(s)${n.quota.objects ? ` / ${n.quota.objects}` : ''}`
+ + (u.reserved_objects ? `, ${u.reserved_objects} upload(s) reserving ${human(u.reserved_bytes)}` : '')
+ + (Object.keys(n.policy).length ? ` policy ${JSON.stringify(n.policy)}` : ''));
+ }
+ if (!out.length) console.log('no namespaces');
+} catch (err) {
+ fail(err.message);
+}
@@ -12,6 +12,9 @@
@@ -12,6 +12,9 @@
*
*
* App seeding: MEDIA_APPS_SEED (JSON array) and/or MEDIA_APP_KEYS ("id:key,…")
* App seeding: MEDIA_APPS_SEED (JSON array) and/or MEDIA_APP_KEYS ("id:key,…")
* are upserted on boot; keys are stored as sha256 hashes.
* are upserted on boot; keys are stored as sha256 hashes.
+ *
+ * Network principal tokens (service and developer app tokens) are checked per namespace for one of
+ * five verbs: read, list, write, delete, transform (VERBS below; objects/namespaces.js has the rows).
*/
*/
'use strict';
'use strict';
@@ -116,23 +119,96 @@ function problem(res, status, code, detail) {
@@ -116,23 +119,96 @@ function problem(res, status, code, detail) {
return http.sendProblem(res, status, code, { detail });
return http.sendProblem(res, status, code, { detail });
}
}
+// ── Namespace grants: five verbs (roadmap WS-G task 2; docs/object-model.md#namespaces-grants-and-quotas) ──
+
+/**
+ * Each verb is one capability, checked for one namespace against the token's `ns`. The first id is
+ * the verb's own; the broader ids after it grant it too, so the tokens Network issued before the
+ * verbs were split keep working (upload also deletes and transforms, read also lists). A namespace
+ * whose policy sets strict_verbs accepts only the verb's own id.
+ */
+const VERBS = Object.freeze({
+ read: ['media.object.read'],
+ list: ['media.object.list', 'media.object.read'],
+ write: ['media.object.upload'],
+ delete: ['media.object.delete', 'media.object.upload'],
+ transform: ['media.derivative.create', 'media.object.upload'],
+});
+
+/**
+ * The namespaces a token was granted. A developer app token's own project id (the `ns` Network
+ * issued before namespaces were rows) means the whole project, app.<project_id>.* (both its tenants).
+ */
+function grantedNamespaces(claims) {
+ const ns = claims && Array.isArray(claims.ns) ? claims.ns.map(String) : [];
+ if (!claims || !isAppPrincipal(claims) || !claims.project_id) return ns;
+ return [...new Set(ns.map(n => (n === claims.project_id ? `app.${claims.project_id}.*` : n)))];
+}
+
+/** Does the grant list cover `root` or anything below it (a token for one child reaches its tenant)? */
+function touchesTenant(nsList, root) {
+ if (capabilities.namespaceAllowed(nsList, root)) return true;
+ return nsList.some(n => (n.endsWith('.*') ? n.slice(0, -2) : n).startsWith(`${root}.`));
+}
+
+/**
+ * May a token with `claims` do `verb` in `namespace`? { allowed, code, reason }; namespace undefined
+ * checks the capability only. The ids are Media's own, so this does not need the installed contracts
+ * catalog to know the newer ones (media.object.list, media.object.delete).
+ */
+function verbAllowed(claims, verb, namespace, { strict = false } = {}) {
+ const ids = VERBS[verb];
+ if (!ids) return { allowed: false, code: 'capability.unknown', reason: `no verb ${verb}` };
+ const held = (strict ? ids.slice(0, 1) : ids).some(id => capabilities.grants(claims && claims.cap, id));
+ if (!held) return { allowed: false, code: 'capability.denied', reason: `${ids[0]} not granted${strict ? ` (${namespace} accepts only it)` : ''}` };
+ if (namespace !== undefined && !capabilities.namespaceAllowed(grantedNamespaces(claims), namespace)) {
+ return { allowed: false, code: 'capability.namespace_denied', reason: `namespace ${namespace} not granted` };
+ }
+ return { allowed: true, code: null, reason: null };
+}
+
+/**
+ * The tenant-level decision in tenantAuth. A `verb` route that is not `namespaced` works in the
+ * tenant's root and is decided here in full. A `namespaced` route (objects v2, the job actions, the
+ * namespace reads) checks each namespace it touches with namespaceGrant(); here the token must hold
+ * the verb and some namespace of this tenant. A `capability` route checks it for the root, as before.
+ */
+function tenantGrant(claims, { capability, verb, namespaced }, appId, root) {
+ if (!verb) return capabilities.check({ ...claims, ns: grantedNamespaces(claims) }, capability, { namespace: root });
+ if (!namespaced) return verbAllowed(claims, verb, root, { strict: require('./objects/namespaces').isStrict(appId, root) });
+ const c = verbAllowed(claims, verb, undefined);
+ if (!c.allowed) return c;
+ return touchesTenant(grantedNamespaces(claims), root) ? c : { allowed: false, code: 'capability.namespace_denied', reason: `namespace ${root} not granted` };
+}
+
+/**
+ * May this request do `verb` in `namespace` of its tenant? The app key (and an upload or session
+ * token, scoped to its object already) may do anything; a Network token as its grant says.
+ */
+function namespaceGrant(req, verb, namespace) {
+ if (!req.grant) return { allowed: true, code: null, reason: null };
+ return verbAllowed(req.grant, verb, namespace, { strict: require('./objects/namespaces').isStrict(req.appId, namespace) });
+}
+
// ── Developer-project tenants (roadmap Wave 20, ADR-014) ─────
// ── Developer-project tenants (roadmap Wave 20, ADR-014) ─────
const PROJECT_ID_RE = /^prj_[0-9A-HJKMNP-TV-Z]{26}$/;
const PROJECT_ID_RE = /^prj_[0-9A-HJKMNP-TV-Z]{26}$/;
const isAppPrincipal = (claims) => !!claims && (claims.actor_type === 'app' || /^app:/.test(String(claims.sub || '')));
const isAppPrincipal = (claims) => !!claims && (claims.actor_type === 'app' || /^app:/.test(String(claims.sub || '')));
/**
/**
- * An app token on /api/v1/<project_id>/files or /api/v2/<project_id>/objects. The path names the
- * project; the token's env picks the tenant row (production prj_<ULID>, sandbox prj_<ULID>-sandbox),
- * which is created on first use with the default quota. Nothing is created unless the token's
- * project_id is the path's project and the token holds this route's capability for it.
+ * An app token on /api/v1/<project_id>/files or /api/v2/<project_id>/(objects|jobs|namespaces). The
+ * path names the project; the token's env picks the tenant row (production prj_<ULID>, sandbox
+ * prj_<ULID>-sandbox), which is created on first use with the default quota. Its namespaces are
+ * app.<project_id> and app.<project_id>.sandbox with their children. Nothing is created unless the
+ * token's project_id is the path's project and the token holds this route's verb there.
*/
*/
-function projectTenant(req, res, next, projectId, claims, capability) {
+function projectTenant(req, res, next, projectId, claims, grant) {
if (claims.project_id !== projectId) {
if (claims.project_id !== projectId) {
return problem(res, 403, 'capability.namespace_denied', `this app token belongs to ${claims.project_id || 'no project'}, not ${projectId}`);
return problem(res, 403, 'capability.namespace_denied', `this app token belongs to ${claims.project_id || 'no project'}, not ${projectId}`);
}
}
const env = claims.env === 'sandbox' ? 'sandbox' : 'production';
const env = claims.env === 'sandbox' ? 'sandbox' : 'production';
- const c = capabilities.check(claims, capability, { namespace: projectId });
+ const tenantId = db.projectTenantId(projectId, env);
+ const c = tenantGrant(claims, grant, tenantId, db.rootNamespace({ app_id: tenantId, project_id: projectId, env }));
if (!c.allowed) return problem(res, 403, c.code, c.reason);
if (!c.allowed) return problem(res, 403, c.code, c.reason);
let tenant;
let tenant;
try {
try {
@@ -147,6 +223,7 @@ function projectTenant(req, res, next, projectId, claims, capability) {
@@ -147,6 +223,7 @@ function projectTenant(req, res, next, projectId, claims, capability) {
req.authType = 'app';
req.authType = 'app';
req.tenantEnv = env;
req.tenantEnv = env;
req.principal = { sub: claims.sub, cap: claims.cap, jti: claims.jti, project_id: projectId, env };
req.principal = { sub: claims.sub, cap: claims.cap, jti: claims.jti, project_id: projectId, env };
+ req.grant = claims;
return next();
return next();
}
}
@@ -169,6 +246,7 @@ function ensureTokenOnlyApps() {
@@ -169,6 +246,7 @@ function ensureTokenOnlyApps() {
['blog', 'OpenVibe.Blog', 10], // Wave 16 post attachments
['blog', 'OpenVibe.Blog', 10], // Wave 16 post attachments
]) {
]) {
if (!db.getApp(appId)) db.run("INSERT INTO apps (app_id, name, api_key_hash, quota_bytes) VALUES (?, ?, '', ?)", [appId, name, quotaGb * 1024 ** 3]);
if (!db.getApp(appId)) db.run("INSERT INTO apps (app_id, name, api_key_hash, quota_bytes) VALUES (?, ?, '', ?)", [appId, name, quotaGb * 1024 ** 3]);
+ db.ensureRootNamespace(db.getApp(appId));
}
}
}
}
@@ -217,16 +295,24 @@ function actingUserId(req) {
@@ -217,16 +295,24 @@ function actingUserId(req) {
* @param {object} opts
* @param {object} opts
* @param {boolean} [opts.allowUser=false] also accept a Network user JWT
* @param {boolean} [opts.allowUser=false] also accept a Network user JWT
* (browser endpoints); Origin, when present, must be allow-listed.
* (browser endpoints); Origin, when present, must be allow-listed.
+ * @param {string} [opts.verb] read | list | write | delete | transform: the route accepts Network
+ * tokens that hold the verb (VERBS) for the tenant's root namespace
+ * @param {boolean} [opts.namespaced=false] with `verb`: the route checks each namespace it touches
+ * (namespaceGrant); a token holding the verb for any namespace of the tenant gets through
+ * @param {string} [opts.capability] instead of a verb: one capability, for the root namespace
*/
*/
-function tenantAuth({ allowUser = false, capability = null } = {}) {
+function tenantAuth({ allowUser = false, capability = null, verb = null, namespaced = false } = {}) {
+ if (verb && !VERBS[verb]) throw new Error(`tenantAuth: unknown verb ${verb}`);
+ const named = verb ? VERBS[verb][0] : capability;
+ const grant = { capability, verb, namespaced };
return (req, res, next) => {
return (req, res, next) => {
const appId = String(req.params.app || '').trim();
const appId = String(req.params.app || '').trim();
const token = bearerToken(req);
const token = bearerToken(req);
// 0) Principal tokens first, so a sandbox token is refused as such (never a 404 that hints
// 0) Principal tokens first, so a sandbox token is refused as such (never a 404 that hints
// at which tenants exist). Only developer-project tenant routes (a route that names its
// at which tenants exist). Only developer-project tenant routes (a route that names its
- // capability, under /<project_id>/) opt in to sandbox tokens.
- const appRoute = !!capability && PROJECT_ID_RE.test(appId);
+ // verb or capability, under /<project_id>/) opt in to sandbox tokens.
+ const appRoute = !!named && PROJECT_ID_RE.test(appId);
let svc = null;
let svc = null;
if (token && token.split('.').length === 3) {
if (token && token.split('.').length === 3) {
const r = verifyServiceTokenResult(token, { acceptSandbox: appRoute });
const r = verifyServiceTokenResult(token, { acceptSandbox: appRoute });
@@ -238,10 +324,10 @@ function tenantAuth({ allowUser = false, capability = null } = {}) {
@@ -238,10 +324,10 @@ function tenantAuth({ allowUser = false, capability = null } = {}) {
if (svc && isAppPrincipal(svc)) {
if (svc && isAppPrincipal(svc)) {
// Developer apps reach only their own project's tenant, and only on routes that name a capability.
// Developer apps reach only their own project's tenant, and only on routes that name a capability.
if (!appRoute) {
if (!appRoute) {
- return problem(res, 403, capability ? 'capability.namespace_denied' : 'capability.denied',
- capability ? 'app tokens reach only /<project_id>/ tenants' : 'app tokens are not accepted on this route');
+ return problem(res, 403, named ? 'capability.namespace_denied' : 'capability.denied',
+ named ? 'app tokens reach only /<project_id>/ tenants' : 'app tokens are not accepted on this route');
}
}
- return projectTenant(req, res, next, appId, svc, capability);
+ return projectTenant(req, res, next, appId, svc, grant);
}
}
if (svc && svc.env === 'sandbox') return problem(res, 401, 'token.sandbox_refused', 'only developer-app sandbox tokens are accepted, on their own project tenant');
if (svc && svc.env === 'sandbox') return problem(res, 401, 'token.sandbox_refused', 'only developer-app sandbox tokens are accepted, on their own project tenant');
@@ -270,14 +356,16 @@ function tenantAuth({ allowUser = false, capability = null } = {}) {
@@ -270,14 +356,16 @@ function tenantAuth({ allowUser = false, capability = null } = {}) {
}
}
// 1b) A service-principal token from OpenVibe.Network (roadmap Wave 1, ADR-003). Accepted only on
// 1b) A service-principal token from OpenVibe.Network (roadmap Wave 1, ADR-003). Accepted only on
- // routes that name the capability they perform, and only for the :app namespaces the token was
- // granted. It carries the app's authority for that one action — no acting user.
+ // routes that name the verb (or capability) they perform, and only for the namespaces of this
+ // tenant the token was granted (the root is the :app id). It carries the app's authority for
+ // that one action — no acting user.
if (svc) {
if (svc) {
- if (!capability) return problem(res, 403, 'capability.denied', 'service tokens are not accepted on this route');
- const c = capabilities.check(svc, capability, { namespace: appId });
+ if (!named) return problem(res, 403, 'capability.denied', 'service tokens are not accepted on this route');
+ const c = tenantGrant(svc, grant, appId, db.rootNamespace(app));
if (!c.allowed) return problem(res, 403, c.code, c.reason);
if (!c.allowed) return problem(res, 403, c.code, c.reason);
req.authType = 'app';
req.authType = 'app';
req.principal = { sub: svc.sub, cap: svc.cap, jti: svc.jti };
req.principal = { sub: svc.sub, cap: svc.cap, jti: svc.jti };
+ req.grant = svc;
return next();
return next();
}
}
@@ -398,6 +486,10 @@ function seedApps() {
@@ -398,6 +486,10 @@ function seedApps() {
module.exports = {
module.exports = {
tenantAuth,
tenantAuth,
tenantPath,
tenantPath,
+ VERBS,
+ verbAllowed,
+ namespaceGrant,
+ grantedNamespaces,
ensureTokenOnlyApps,
ensureTokenOnlyApps,
verifyServiceToken,
verifyServiceToken,
verifyServiceTokenResult,
verifyServiceTokenResult,
@@ -62,6 +62,10 @@ const config = {
@@ -62,6 +62,10 @@ const config = {
multipartDefaultPartMb: intEnv('MEDIA_MULTIPART_PART_MB', 64),
multipartDefaultPartMb: intEnv('MEDIA_MULTIPART_PART_MB', 64),
multipartTtlHours: intEnv('MEDIA_MULTIPART_TTL_HOURS', 24), // an unfinished session's parts are purged after this
multipartTtlHours: intEnv('MEDIA_MULTIPART_TTL_HOURS', 24), // an unfinished session's parts are purged after this
uploadMinFreeMb: intEnv('MEDIA_UPLOAD_MIN_FREE_MB', 10240), // multipart needs 2x its size free plus this
uploadMinFreeMb: intEnv('MEDIA_UPLOAD_MIN_FREE_MB', 10240), // multipart needs 2x its size free plus this
+ // Namespaces and quotas (server/objects/namespaces.js): an upload holds its quota reservation this long
+ // after its last step (init, a fresh URL, the bytes); then the hourly sweep fails it and frees its bytes.
+ reservationHours: Math.max(1, intEnv('MEDIA_UPLOAD_RESERVATION_HOURS', 72)),
+ maxChildNamespaces: Math.max(0, intEnv('MEDIA_NAMESPACE_MAX_CHILDREN', 100)), // children per tenant
},
},
// Job system (server/jobs/; docs/object-model.md#jobs).
// Job system (server/jobs/; docs/object-model.md#jobs).
@@ -29,6 +29,7 @@ function getDb() {
@@ -29,6 +29,7 @@ function getDb() {
database.exec(schema);
database.exec(schema);
migrateColumns();
migrateColumns();
migrateJobsTable(schema);
migrateJobsTable(schema);
+ migrateNamespaces();
ensureObjectTriggers();
ensureObjectTriggers();
normalizeThumbnailUrls();
normalizeThumbnailUrls();
seedSettings();
seedSettings();
@@ -265,7 +266,9 @@ function upsertApp({ app_id, name, api_key, webhook_url, webhook_secret, allowed
@@ -265,7 +266,9 @@ function upsertApp({ app_id, name, api_key, webhook_url, webhook_secret, allowed
allowed_origins = excluded.allowed_origins,
allowed_origins = excluded.allowed_origins,
quota_bytes = excluded.quota_bytes`,
quota_bytes = excluded.quota_bytes`,
[app_id, name || app_id, hashApiKey(api_key), webhook_url || null, webhook_secret || null, origins, quota_bytes || 0]);
[app_id, name || app_id, hashApiKey(api_key), webhook_url || null, webhook_secret || null, origins, quota_bytes || 0]);
- return getApp(app_id);
+ const row = getApp(app_id);
+ ensureRootNamespace(row);
+ return row;
}
}
/**
/**
@@ -289,6 +292,7 @@ function ensureProjectTenant(projectId, env, quotaBytes) {
@@ -289,6 +292,7 @@ function ensureProjectTenant(projectId, env, quotaBytes) {
err.code = 'media.tenant.conflict';
err.code = 'media.tenant.conflict';
throw err;
throw err;
}
}
+ ensureRootNamespace(row);
return row;
return row;
}
}
@@ -303,6 +307,78 @@ function appAllowedOrigins(app) {
@@ -303,6 +307,78 @@ function appAllowedOrigins(app) {
try { return JSON.parse(app.allowed_origins || '[]'); } catch { return []; }
try { return JSON.parse(app.allowed_origins || '[]'); } catch { return []; }
}
}
+// ── Namespaces (roadmap WS-G task 2; objects/namespaces.js) ──
+
+/**
+ * A tenant's root namespace: its app id for a first-party tenant; app.<project_id> for a developer
+ * project's production tenant and app.<project_id>.sandbox for its sandbox tenant (so a grant of
+ * app.<project_id>.* covers both, and the token's env still picks the tenant).
+ */
+function rootNamespace(app) {
+ if (!app || !app.app_id) return null;
+ if (app.project_id) return app.env === 'sandbox' ? `app.${app.project_id}.sandbox` : `app.${app.project_id}`;
+ return app.app_id;
+}
+
+/** Who owns a tenant's namespaces: service:<app_id> (a first-party tenant) or project:<prj_…>. */
+function namespaceOwner(app) {
+ return app.project_id ? `project:${app.project_id}` : `service:${app.app_id}`;
+}
+
+/** The tenant's root namespace row, created when missing. Returns its name. */
+function ensureRootNamespace(app) {
+ const ns = rootNamespace(app);
+ if (!ns) return null;
+ run('INSERT OR IGNORE INTO media_namespaces (namespace, app_id, parent, owner) VALUES (?, ?, NULL, ?)', [ns, app.app_id, namespaceOwner(app)]);
+ return ns;
+}
+
+/**
+ * Namespaces as rows (WS-G task 2). Idempotent, on every open:
+ * 1. a developer project's objects move from the tenant id (prj_<ULID>, prj_<ULID>-sandbox), which
+ * was their namespace before namespaces were rows, to app.<project_id>[.sandbox];
+ * 2. every tenant has its root row, and every namespace an object names has a row (a child with the
+ * chain up to its root);
+ * 3. once (setting namespaces.reservations_seeded): every native upload still in progress holds a
+ * quota reservation of its declared size, expiring MEDIA_UPLOAD_RESERVATION_HOURS after that
+ * first open, so an upload that began before reservations existed is not cut short.
+ */
+function migrateNamespaces() {
+ const insert = database.prepare('INSERT OR IGNORE INTO media_namespaces (namespace, app_id, parent, owner) VALUES (?, ?, ?, ?)');
+ database.transaction(() => {
+ const apps = new Map(database.prepare('SELECT * FROM apps').all().map(a => [a.app_id, a]));
+ const move = database.prepare('UPDATE media_objects SET namespace = ? WHERE app_id = ? AND namespace = ?');
+ let moved = 0;
+ for (const app of apps.values()) {
+ if (app.project_id) moved += move.run(rootNamespace(app), app.app_id, app.app_id).changes;
+ insert.run(rootNamespace(app), app.app_id, null, namespaceOwner(app));
+ }
+ if (moved) console.warn(`[DB] Moved ${moved} developer-project object(s) to their app.<project_id> namespace`);
+ const missing = database.prepare(`SELECT DISTINCT o.app_id, o.namespace FROM media_objects o
+ LEFT JOIN media_namespaces n ON n.namespace = o.namespace WHERE n.namespace IS NULL`).all();
+ for (const m of missing) {
+ const app = apps.get(m.app_id) || { app_id: m.app_id };
+ const root = rootNamespace(app);
+ insert.run(root, app.app_id, null, namespaceOwner(app));
+ if (m.namespace === root) continue;
+ if (!m.namespace.startsWith(`${root}.`)) { insert.run(m.namespace, app.app_id, null, namespaceOwner(app)); continue; }
+ let parent = root;
+ for (const seg of m.namespace.slice(root.length + 1).split('.')) {
+ insert.run(`${parent}.${seg}`, app.app_id, parent, namespaceOwner(app));
+ parent = `${parent}.${seg}`;
+ }
+ }
+ const seeded = database.prepare("SELECT value FROM media_settings WHERE key = 'namespaces.reservations_seeded'").get();
+ if (!seeded) {
+ const n = database.prepare(`INSERT OR IGNORE INTO media_quota_reservations (object_id, app_id, namespace, bytes, expires_at)
+ SELECT id, app_id, namespace, size_bytes, datetime('now', ?) FROM media_objects WHERE lifecycle_status = 'uploading' AND legacy_ref IS NULL`)
+ .run(`+${Math.max(1, config.objects.reservationHours)} hours`).changes;
+ database.prepare("INSERT OR REPLACE INTO media_settings (key, value, type) VALUES ('namespaces.reservations_seeded', ?, 'string')").run(new Date().toISOString());
+ if (n) console.warn(`[DB] ${n} upload(s) in progress now hold a quota reservation`);
+ }
+ })();
+}
+
// ── VOD helpers ──────────────────────────────────────────────
// ── VOD helpers ──────────────────────────────────────────────
// The row and its media_object in one transaction (a clips-only recording has no object).
// The row and its media_object in one transaction (a clips-only recording has no object).
@@ -820,6 +896,8 @@ module.exports = {
@@ -820,6 +896,8 @@ module.exports = {
getSetting, setSetting,
getSetting, setSetting,
// apps
// apps
hashApiKey, getApp, listApps, upsertApp, appAllowedOrigins, projectTenantId, ensureProjectTenant, isSandboxTenant,
hashApiKey, getApp, listApps, upsertApp, appAllowedOrigins, projectTenantId, ensureProjectTenant, isSandboxTenant,
+ // namespaces
+ rootNamespace, namespaceOwner, ensureRootNamespace,
// vods
// vods
createVod, getVodById, getVodByFileBasename, listVods, countVods, setVodVisibility, vodStatus,
createVod, getVodById, getVodByFileBasename, listVods, countVods, setVodVisibility, vodStatus,
latestVodThumbsByManagedStreams, getAppStats, getAppStatSeries,
latestVodThumbsByManagedStreams, getAppStats, getAppStatSeries,
@@ -215,7 +215,7 @@ CREATE UNIQUE INDEX IF NOT EXISTS idx_assets_identity ON assets(app_id, kind, na
@@ -215,7 +215,7 @@ CREATE UNIQUE INDEX IF NOT EXISTS idx_assets_identity ON assets(app_id, kind, na
CREATE TABLE IF NOT EXISTS media_objects (
CREATE TABLE IF NOT EXISTS media_objects (
id TEXT PRIMARY KEY, -- med_<ULID> (time-sortable)
id TEXT PRIMARY KEY, -- med_<ULID> (time-sortable)
app_id TEXT NOT NULL, -- tenant
app_id TEXT NOT NULL, -- tenant
- namespace TEXT NOT NULL, -- capability namespace (= app_id today; projects later)
+ namespace TEXT NOT NULL, -- a media_namespaces row: the tenant's root (app id, or app.<project_id>[.sandbox]) or a child of it
kind TEXT NOT NULL CHECK(kind IN ('vod', 'clip', 'file', 'thumbnail', 'screenshot', 'avatar', 'asset')),
kind TEXT NOT NULL CHECK(kind IN ('vod', 'clip', 'file', 'thumbnail', 'screenshot', 'avatar', 'asset')),
owner_subject TEXT, -- canonical subject (usr_<ULID>) when known
owner_subject TEXT, -- canonical subject (usr_<ULID>) when known
owner_app TEXT, -- legacy owner: the app whose user-id space owner_user_id is in
owner_app TEXT, -- legacy owner: the app whose user-id space owner_user_id is in
@@ -237,6 +237,47 @@ CREATE INDEX IF NOT EXISTS idx_media_objects_app ON media_objects(app_id, id);
@@ -237,6 +237,47 @@ CREATE INDEX IF NOT EXISTS idx_media_objects_app ON media_objects(app_id, id);
CREATE INDEX IF NOT EXISTS idx_media_objects_kind ON media_objects(app_id, kind);
CREATE INDEX IF NOT EXISTS idx_media_objects_kind ON media_objects(app_id, kind);
CREATE INDEX IF NOT EXISTS idx_media_objects_owner ON media_objects(owner_subject);
CREATE INDEX IF NOT EXISTS idx_media_objects_owner ON media_objects(owner_subject);
CREATE INDEX IF NOT EXISTS idx_media_objects_lifecycle ON media_objects(lifecycle_status);
CREATE INDEX IF NOT EXISTS idx_media_objects_lifecycle ON media_objects(lifecycle_status);
+CREATE INDEX IF NOT EXISTS idx_media_objects_namespace ON media_objects(app_id, namespace, lifecycle_status);
+
+-- Namespaces (roadmap WS-G task 2; server/objects/namespaces.js, docs/object-model.md#namespaces-grants-and-quotas).
+-- Every namespace an object names is a row. A tenant (apps row) has one root: its app id, or
+-- app.<project_id> (app.<project_id>.sandbox) for a developer project, and children below it
+-- (<root>.<segment>…) made by the first upload that names one. Quotas: NULL inherits (a root takes its
+-- tenant's apps.quota_bytes; a child has no limit of its own), 0 is no limit. The used_* and reserved_*
+-- columns are a snapshot of the subtree (the namespace and its children) taken by reconcile(); quota
+-- checks count from the object and reservation rows themselves.
+CREATE TABLE IF NOT EXISTS media_namespaces (
+ namespace TEXT PRIMARY KEY,
+ app_id TEXT NOT NULL, -- the tenant it belongs to (and whose quota it counts against)
+ parent TEXT, -- NULL for a root
+ owner TEXT NOT NULL, -- service:<app_id> (a first-party tenant) | project:<prj_…> (a developer project)
+ policy TEXT NOT NULL DEFAULT '{}', -- JSON: kinds, visibilities, max_object_bytes, strict_verbs (children inherit)
+ quota_bytes INTEGER,
+ quota_objects INTEGER,
+ used_bytes INTEGER NOT NULL DEFAULT 0,
+ used_objects INTEGER NOT NULL DEFAULT 0,
+ reserved_bytes INTEGER NOT NULL DEFAULT 0,
+ reserved_objects INTEGER NOT NULL DEFAULT 0,
+ reconciled_at DATETIME,
+ created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
+ updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
+);
+CREATE INDEX IF NOT EXISTS idx_media_namespaces_app ON media_namespaces(app_id, namespace);
+
+-- Quota held by an upload in progress (a native object still `uploading`): its declared size from
+-- init, the bytes received once they are stored. Settled at complete (the object's own size counts
+-- from then on), released at abort (down to the bytes still stored, if any) and swept at expiry.
+CREATE TABLE IF NOT EXISTS media_quota_reservations (
+ object_id TEXT PRIMARY KEY,
+ app_id TEXT NOT NULL,
+ namespace TEXT NOT NULL,
+ bytes INTEGER NOT NULL DEFAULT 0,
+ expires_at DATETIME NOT NULL, -- the expiry sweep fails the upload after this (unless a multipart session is open)
+ created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
+ updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
+);
+CREATE INDEX IF NOT EXISTS idx_media_quota_reservations_ns ON media_quota_reservations(app_id, namespace);
+CREATE INDEX IF NOT EXISTS idx_media_quota_reservations_expiry ON media_quota_reservations(expires_at);
-- Where the bytes are. One row per provider copy; the canonical one is named on the object.
-- Where the bytes are. One row per provider copy; the canonical one is named on the object.
CREATE TABLE IF NOT EXISTS media_locations (
CREATE TABLE IF NOT EXISTS media_locations (
@@ -10,11 +10,14 @@
@@ -10,11 +10,14 @@
*
*
* Files live at FILES_PATH/<app>/<key> with key = <sha256-prefix>-<name>.
* Files live at FILES_PATH/<app>/<key> with key = <sha256-prefix>-<name>.
* Public serving (Content-Type + Range) is at GET /f/:key.
* Public serving (Content-Type + Range) is at GET /f/:key.
- * Per-app quota: apps.quota_bytes (0 = unlimited) checked against the tenant's
- * stored bytes (v1 files + native v2 objects, objects/model.usedBytes).
+ * Files live in the tenant's root namespace. Its quotas (bytes: apps.quota_bytes unless the root
+ * sets its own, 0 = unlimited; objects: the root's quota_objects) count v1 files, native v2 objects
+ * and uploads in progress (objects/namespaces.js).
*
*
- * Network principal tokens: upload + delete need media.object.upload, list + meta
- * need media.object.read, for namespace = :app. Developer-project tenants
+ * Network principal tokens, for the tenant's root namespace (server/auth.js VERBS): upload needs
+ * write (media.object.upload), delete needs delete (media.object.delete, or media.object.upload),
+ * list needs list (media.object.list, or media.object.read), meta needs read (media.object.read).
+ * Developer-project tenants
* (:app = prj_<ULID>, app tokens only, ADR-014): sandbox files are never served
* (:app = prj_<ULID>, app tokens only, ADR-014): sandbox files are never served
* publicly — their `url` is a short-lived signed /f/:key URL.
* publicly — their `url` is a short-lived signed /f/:key URL.
*/
*/
@@ -27,6 +30,7 @@ const fs = require('fs');
@@ -27,6 +30,7 @@ const fs = require('fs');
const multer = require('multer');
const multer = require('multer');
const db = require('../db/database');
const db = require('../db/database');
const config = require('../config');
const config = require('../config');
+const namespaces = require('../objects/namespaces');
const { tenantAuth, tenantCors } = require('../auth');
const { tenantAuth, tenantCors } = require('../auth');
const router = express.Router({ mergeParams: true });
const router = express.Router({ mergeParams: true });
@@ -81,7 +85,7 @@ function sha256File(filePath) {
@@ -81,7 +85,7 @@ function sha256File(filePath) {
}
}
// ── Upload ───────────────────────────────────────────────────
// ── Upload ───────────────────────────────────────────────────
-router.post('/', tenantAuth({ allowUser: true, capability: 'media.object.upload' }), upload.single('file'), async (req, res) => {
+router.post('/', tenantAuth({ allowUser: true, verb: 'write' }), upload.single('file'), async (req, res) => {
try {
try {
if (!req.file) return res.status(400).json({ error: 'No file uploaded (multipart field: file)' });
if (!req.file) return res.status(400).json({ error: 'No file uploaded (multipart field: file)' });
@@ -104,20 +108,14 @@ router.post('/', tenantAuth({ allowUser: true, capability: 'media.object.upload'
@@ -104,20 +108,14 @@ router.post('/', tenantAuth({ allowUser: true, capability: 'media.object.upload'
return res.status(200).json({ ...filePublic(existing), deduplicated: true });
return res.status(200).json({ ...filePublic(existing), deduplicated: true });
}
}
- // Per-app quota check against quota_bytes (0 = unlimited): files + native objects. Checked
- // after the last await, so nothing else is stored between this check and the row below
+ // The root namespace's quotas (bytes and objects: files + native objects + uploads in progress).
+ // Checked after the last await, so nothing else is stored between this check and the row below
// (concurrent uploads each saw the same usage when it ran before hashing).
// (concurrent uploads each saw the same usage when it ran before hashing).
- const quota = Number(req.appRow.quota_bytes) || 0;
- if (quota > 0) {
- const used = require('../objects/model').usedBytes(req.appId);
- if (used + req.file.size > quota) {
- try { fs.unlinkSync(req.file.path); } catch { /* */ }
- return res.status(413).json({
- error: 'App file quota exceeded',
- quota_bytes: quota,
- used_bytes: used,
- });
- }
+ const root = db.rootNamespace(req.appRow);
+ const q = namespaces.checkQuota(req.appRow, root, { bytes: req.file.size, objects: 1 });
+ if (q) {
+ try { fs.unlinkSync(req.file.path); } catch { /* */ }
+ return res.status(413).json({ error: q.code === 'media.quota.objects_exceeded' ? 'App object quota exceeded' : 'App file quota exceeded', code: q.code, ...q.extra });
}
}
const dest = path.join(appDir(req.appId), key);
const dest = path.join(appDir(req.appId), key);
@@ -140,6 +138,7 @@ router.post('/', tenantAuth({ allowUser: true, capability: 'media.object.upload'
@@ -140,6 +138,7 @@ router.post('/', tenantAuth({ allowUser: true, capability: 'media.object.upload'
});
});
const row = db.getFileByKey(key, req.appId);
const row = db.getFileByKey(key, req.appId);
+ namespaces.reconcileChain(req.appRow, root);
console.log(`[Files] Stored ${key} for app ${req.appId} (${(req.file.size / 1024).toFixed(1)} KB)`);
console.log(`[Files] Stored ${key} for app ${req.appId} (${(req.file.size / 1024).toFixed(1)} KB)`);
res.status(201).json(filePublic(row));
res.status(201).json(filePublic(row));
} catch (err) {
} catch (err) {
@@ -150,7 +149,7 @@ router.post('/', tenantAuth({ allowUser: true, capability: 'media.object.upload'
@@ -150,7 +149,7 @@ router.post('/', tenantAuth({ allowUser: true, capability: 'media.object.upload'
});
});
// ── List ─────────────────────────────────────────────────────
// ── List ─────────────────────────────────────────────────────
-router.get('/', tenantAuth({ allowUser: true, capability: 'media.object.read' }), (req, res) => {
+router.get('/', tenantAuth({ allowUser: true, verb: 'list' }), (req, res) => {
try {
try {
const limit = Math.min(Math.max(parseInt(req.query.limit || '100', 10), 1), 500);
const limit = Math.min(Math.max(parseInt(req.query.limit || '100', 10), 1), 500);
const offset = Math.max(parseInt(req.query.offset || '0', 10), 0);
const offset = Math.max(parseInt(req.query.offset || '0', 10), 0);
@@ -167,7 +166,7 @@ router.get('/', tenantAuth({ allowUser: true, capability: 'media.object.read' })
@@ -167,7 +166,7 @@ router.get('/', tenantAuth({ allowUser: true, capability: 'media.object.read' })
});
});
// ── Meta ─────────────────────────────────────────────────────
// ── Meta ─────────────────────────────────────────────────────
-router.get('/:key', tenantAuth({ allowUser: true, capability: 'media.object.read' }), (req, res) => {
+router.get('/:key', tenantAuth({ allowUser: true, verb: 'read' }), (req, res) => {
try {
try {
const row = db.getFileByKey(String(req.params.key), req.appId);
const row = db.getFileByKey(String(req.params.key), req.appId);
if (!row) return res.status(404).json({ error: 'File not found' });
if (!row) return res.status(404).json({ error: 'File not found' });
@@ -178,7 +177,7 @@ router.get('/:key', tenantAuth({ allowUser: true, capability: 'media.object.read
@@ -178,7 +177,7 @@ router.get('/:key', tenantAuth({ allowUser: true, capability: 'media.object.read
});
});
// ── Delete ───────────────────────────────────────────────────
// ── Delete ───────────────────────────────────────────────────
-router.delete('/:key', tenantAuth({ allowUser: true, capability: 'media.object.upload' }), (req, res) => {
+router.delete('/:key', tenantAuth({ allowUser: true, verb: 'delete' }), (req, res) => {
try {
try {
const row = db.getFileByKey(String(req.params.key), req.appId);
const row = db.getFileByKey(String(req.params.key), req.appId);
if (!row) return res.status(404).json({ error: 'File not found' });
if (!row) return res.status(404).json({ error: 'File not found' });
@@ -190,6 +189,7 @@ router.delete('/:key', tenantAuth({ allowUser: true, capability: 'media.object.u
@@ -190,6 +189,7 @@ router.delete('/:key', tenantAuth({ allowUser: true, capability: 'media.object.u
const filePath = filePathForKey(row);
const filePath = filePathForKey(row);
try { if (fs.existsSync(filePath)) fs.unlinkSync(filePath); } catch { /* */ }
try { if (fs.existsSync(filePath)) fs.unlinkSync(filePath); } catch { /* */ }
db.deleteFileRow(row.key);
db.deleteFileRow(row.key);
+ namespaces.reconcileChain(req.appRow, db.rootNamespace(req.appRow));
res.json({ message: 'File deleted' });
res.json({ message: 'File deleted' });
} catch (err) {
} catch (err) {
res.status(500).json({ error: 'Failed to delete file' });
res.status(500).json({ error: 'Failed to delete file' });
@@ -183,6 +183,7 @@ app.use('/api/v1/:app/assets', require('./assets/routes'));
@@ -183,6 +183,7 @@ app.use('/api/v1/:app/assets', require('./assets/routes'));
app.use('/api/v1/:app/admin/storage', require('./admin/routes'));
app.use('/api/v1/:app/admin/storage', require('./admin/routes'));
app.use('/api/v2/:app/objects', require('./objects/routes')); // canonical object API (docs/object-model.md)
app.use('/api/v2/:app/objects', require('./objects/routes')); // canonical object API (docs/object-model.md)
app.use('/api/v2/:app/jobs', require('./jobs/routes')); // media jobs: thumbnails, split/remux, invariant scans
app.use('/api/v2/:app/jobs', require('./jobs/routes')); // media jobs: thumbnails, split/remux, invariant scans
+app.use('/api/v2/:app/namespaces', require('./objects/namespace-routes')); // a tenant's namespaces: policy, quotas, usage
app.use('/o', require('./objects/routes').publicRouter); // object bytes (public, or signed)
app.use('/o', require('./objects/routes').publicRouter); // object bytes (public, or signed)
app.use('/', require('./public/routes')); // /v /c /p /t /f
app.use('/', require('./public/routes')); // /v /c /p /t /f
app.use(require('./not-found').notFound); // nothing matched: 404 with Media's CSP (Cloudflare's beacon allowed)
app.use(require('./not-found').notFound); // nothing matched: 404 with Media's CSP (Cloudflare's beacon allowed)
@@ -233,6 +234,14 @@ if (!drill.enabled) {
@@ -233,6 +234,14 @@ if (!drill.enabled) {
try { const n = require('./objects/model').purgeExpired(); if (n) console.log(`[Objects] Purged ${n} expired deleted object(s)`); } catch (err) { console.warn('[Objects] purge:', err.message); }
try { const n = require('./objects/model').purgeExpired(); if (n) console.log(`[Objects] Purged ${n} expired deleted object(s)`); } catch (err) { console.warn('[Objects] purge:', err.message); }
// Incomplete multipart uploads past MEDIA_MULTIPART_TTL_HOURS: their parts are deleted (the objects stay uploading).
// Incomplete multipart uploads past MEDIA_MULTIPART_TTL_HOURS: their parts are deleted (the objects stay uploading).
try { const r = require('./objects/multipart').purgeExpired(); if (r.expired || r.orphan_dirs) console.log(`[Objects] Multipart: ${r.expired} expired session(s), ${r.orphan_dirs} orphan part dir(s) removed`); } catch (err) { console.warn('[Objects] multipart purge:', err.message); }
try { const r = require('./objects/multipart').purgeExpired(); if (r.expired || r.orphan_dirs) console.log(`[Objects] Multipart: ${r.expired} expired session(s), ${r.orphan_dirs} orphan part dir(s) removed`); } catch (err) { console.warn('[Objects] multipart purge:', err.message); }
+ // Namespaces: uploads whose quota reservation expired are failed and their bytes freed; then every
+ // namespace's usage snapshot is refreshed (v1 files, deletes and purges included).
+ try {
+ const ns = require('./objects/namespaces');
+ const r = ns.expireReservations();
+ if (r.expired || r.dropped) console.log(`[Namespaces] ${r.expired} abandoned upload(s) expired, ${r.dropped} stale reservation(s) dropped`);
+ ns.reconcileAll();
+ } catch (err) { console.warn('[Namespaces] sweep:', err.message); }
});
});
// Project rows that have no media_object yet (first boot after the upgrade: all of them).
// Project rows that have no media_object yet (first boot after the upgrade: all of them).
// C-75 catch-up: every write now makes its object in the same transaction (objects/model.js
// C-75 catch-up: every write now makes its object in the same transaction (objects/model.js
@@ -245,6 +254,8 @@ if (!drill.enabled) {
@@ -245,6 +254,8 @@ if (!drill.enabled) {
const r = bf.backfill({ onlyMissing: true });
const r = bf.backfill({ onlyMissing: true });
if (r.totals.created || r.totals.updated || r.errors.length) console.log(`[Objects] Backfill: ${bf.summarize(r)}`);
if (r.totals.created || r.totals.updated || r.errors.length) console.log(`[Objects] Backfill: ${bf.summarize(r)}`);
} catch (err) { console.warn('[Objects] Backfill failed:', err.message); }
} catch (err) { console.warn('[Objects] Backfill failed:', err.message); }
+ // The namespaces' usage snapshot, first taken here (then hourly, and after each write).
+ try { require('./objects/namespaces').reconcileAll(); } catch (err) { console.warn('[Namespaces] reconcile:', err.message); }
}, 15 * 1000).unref?.();
}, 15 * 1000).unref?.();
// Recover from an unclean shutdown: rows stuck in is_recording with no live ffmpeg are finalized
// Recover from an unclean shutdown: rows stuck in is_recording with no live ffmpeg are finalized
@@ -146,11 +146,10 @@ function checkRoom(obj, needBytes) {
@@ -146,11 +146,10 @@ function checkRoom(obj, needBytes) {
if (free < needBytes + reserve) {
if (free < needBytes + reserve) {
throw new JobError('insufficient_disk', `Not enough free disk: need ${Math.ceil((needBytes + reserve) / MB)} MB, ${Math.floor(free / MB)} MB free`, { retryAfterS: 1800 });
throw new JobError('insufficient_disk', `Not enough free disk: need ${Math.ceil((needBytes + reserve) / MB)} MB, ${Math.floor(free / MB)} MB free`, { retryAfterS: 1800 });
}
}
+ // Outputs go to the source's namespace: every quota from there up to the tenant must have the room.
const app = db.getApp(obj.app_id);
const app = db.getApp(obj.app_id);
- const quota = Number(app && app.quota_bytes) || 0;
- if (quota && model.usedBytes(obj.app_id) + needBytes > quota) {
- throw new JobError('quota_exceeded', 'The tenant storage quota would be exceeded', { permanent: true });
- }
+ const q = app ? require('../objects/namespaces').checkQuota(app, obj.namespace || db.rootNamespace(app), { bytes: needBytes, objects: 1 }) : null;
+ if (q) throw new JobError('quota_exceeded', `The storage quota would be exceeded (${q.detail})`, { permanent: true });
}
}
function workDir(jobId) {
function workDir(jobId) {
@@ -182,7 +181,7 @@ async function adopt({ src, file, ext, job, relation, metadata, variant, saveChe
@@ -182,7 +181,7 @@ async function adopt({ src, file, ext, job, relation, metadata, variant, saveChe
db.run(`INSERT INTO media_objects (id, app_id, namespace, kind, owner_subject, owner_app, owner_user_id, visibility, lifecycle_status,
db.run(`INSERT INTO media_objects (id, app_id, namespace, kind, owner_subject, owner_app, owner_user_id, visibility, lifecycle_status,
mime_type, size_bytes, content_hash, canonical_provider, canonical_key, legacy_ref, metadata)
mime_type, size_bytes, content_hash, canonical_provider, canonical_key, legacy_ref, metadata)
VALUES (?, ?, ?, ?, ?, ?, ?, 'private', 'ready', ?, ?, ?, 'local', ?, NULL, ?)`,
VALUES (?, ?, ?, ?, ?, ?, ?, 'private', 'ready', ?, ?, ?, 'local', ?, NULL, ?)`,
- [id, src.app_id, src.namespace || src.app_id, kind || src.kind, src.owner_subject || null, src.owner_app || src.app_id, src.owner_user_id ?? null,
+ [id, src.app_id, src.namespace || db.rootNamespace(db.getApp(src.app_id) || { app_id: src.app_id }), kind || src.kind, src.owner_subject || null, src.owner_app || src.app_id, src.owner_user_id ?? null,
mimeFor(ext, src), size, hash, dest,
mimeFor(ext, src), size, hash, dest,
JSON.stringify({ title: md.title || null, derived_from: src.id, job_id: job.id, filename: `${src.id}${suffix ? `-${suffix}` : metadata.part ? `-part${metadata.part}` : '-remux'}${ext}`, ...metadata })]);
JSON.stringify({ title: md.title || null, derived_from: src.id, job_id: job.id, filename: `${src.id}${suffix ? `-${suffix}` : metadata.part ? `-part${metadata.part}` : '-remux'}${ext}`, ...metadata })]);
model.upsertLocation(id, { provider: 'local', key: dest, state: 'present', size_bytes: size, checksum: hash, verified: true });
model.upsertLocation(id, { provider: 'local', key: dest, state: 'present', size_bytes: size, checksum: hash, verified: true });
@@ -9,11 +9,13 @@
@@ -9,11 +9,13 @@
* POST /:jobId/cancel cancel (also DELETE /:jobId): proposed/queued at once -> 200;
* POST /:jobId/cancel cancel (also DELETE /:jobId): proposed/queued at once -> 200;
* running -> 202 { job } with cancel_requested until it stops; finished -> 409
* running -> 202 { job } with cancel_requested until it stops; finished -> 409
*
*
- * Auth as the objects API: the app key or a Network service token granting media.object.read (reads)
- * or media.object.upload (create, approve, cancel) for namespace :app; a developer project's app
- * tokens on its own tenant. With X-OV-User-Id (app key) the caller acts for one of the app's users:
- * it sees and decides only jobs it created or jobs on objects it owns, and creates jobs on its own
- * objects only. Errors are RFC 9457 problems.
+ * Auth as the objects API: the app key, or a Network token holding the verb (server/auth.js VERBS):
+ * list (GET /) and read (GET /:jobId) for the tenant's root namespace; transform (create, approve,
+ * cancel: media.derivative.create, or media.object.upload as before) for the namespace of the job's
+ * object (the root for a job with none); a developer project's app tokens on its own tenant. With
+ * X-OV-User-Id (app key) the caller acts for one of the app's users: it sees and decides only jobs
+ * it created or jobs on objects it owns, and creates jobs on its own objects only. Errors are RFC
+ * 9457 problems.
*/
*/
'use strict';
'use strict';
@@ -21,10 +23,12 @@ const express = require('express');
@@ -21,10 +23,12 @@ const express = require('express');
const { http } = require('openvibe-contracts');
const { http } = require('openvibe-contracts');
const model = require('../objects/model');
const model = require('../objects/model');
const queue = require('./queue');
const queue = require('./queue');
-const { tenantAuth, tenantCors } = require('../auth');
+const db = require('../db/database');
+const { tenantAuth, tenantCors, namespaceGrant } = require('../auth');
-const write = tenantAuth({ capability: 'media.object.upload' });
-const read = tenantAuth({ capability: 'media.object.read' });
+const transform = tenantAuth({ verb: 'transform', namespaced: true });
+const read = tenantAuth({ verb: 'read' });
+const list = tenantAuth({ verb: 'list' });
function problem(res, status, code, detail, extra) {
function problem(res, status, code, detail, extra) {
return http.sendProblem(res, status, code, { detail, extra });
return http.sendProblem(res, status, code, { detail, extra });
@@ -48,6 +52,13 @@ function load(req, res) {
@@ -48,6 +52,13 @@ function load(req, res) {
return job;
return job;
}
}
+/** Transform is granted per namespace: the job's object's, or the tenant's root for a job with none (else answered). */
+function mayTransform(req, res, obj) {
+ const g = namespaceGrant(req, 'transform', obj ? obj.namespace : db.rootNamespace(req.appRow));
+ if (!g.allowed) { problem(res, 403, g.code, g.reason); return false; }
+ return true;
+}
+
function sendJobError(res, err) {
function sendJobError(res, err) {
if (err instanceof queue.JobError) {
if (err instanceof queue.JobError) {
if (err.retryAfterS) res.set('Retry-After', String(err.retryAfterS));
if (err.retryAfterS) res.set('Retry-After', String(err.retryAfterS));
@@ -60,7 +71,7 @@ function sendJobError(res, err) {
@@ -60,7 +71,7 @@ function sendJobError(res, err) {
const router = express.Router({ mergeParams: true });
const router = express.Router({ mergeParams: true });
router.use(tenantCors);
router.use(tenantCors);
-router.get('/', read, (req, res) => {
+router.get('/', list, (req, res) => {
try {
try {
const q = req.query;
const q = req.query;
if (q.cursor && !/^mjob_[0-9A-Za-z_]{1,40}$/.test(String(q.cursor))) return problem(res, 400, 'media.job.invalid', 'Bad cursor');
if (q.cursor && !/^mjob_[0-9A-Za-z_]{1,40}$/.test(String(q.cursor))) return problem(res, 400, 'media.job.invalid', 'Bad cursor');
@@ -73,7 +84,7 @@ router.get('/', read, (req, res) => {
@@ -73,7 +84,7 @@ router.get('/', read, (req, res) => {
} catch (err) { sendJobError(res, err); }
} catch (err) { sendJobError(res, err); }
});
});
-router.post('/', write, (req, res) => {
+router.post('/', transform, (req, res) => {
try {
try {
const b = req.body || {};
const b = req.body || {};
const type = String(b.type || '');
const type = String(b.type || '');
@@ -89,6 +100,7 @@ router.post('/', write, (req, res) => {
@@ -89,6 +100,7 @@ router.post('/', write, (req, res) => {
} else if (spec.needsObject) {
} else if (spec.needsObject) {
return problem(res, 400, 'media.job.invalid', `${type} needs object_id`);
return problem(res, 400, 'media.job.invalid', `${type} needs object_id`);
}
}
+ if (!mayTransform(req, res, obj)) return;
if (b.params != null && (typeof b.params !== 'object' || Array.isArray(b.params) || JSON.stringify(b.params).length > 16384)) {
if (b.params != null && (typeof b.params !== 'object' || Array.isArray(b.params) || JSON.stringify(b.params).length > 16384)) {
return problem(res, 400, 'media.job.invalid', 'params must be an object of at most 16 KB');
return problem(res, 400, 'media.job.invalid', 'params must be an object of at most 16 KB');
}
}
@@ -110,10 +122,10 @@ router.get('/:jobId', read, (req, res) => {
@@ -110,10 +122,10 @@ router.get('/:jobId', read, (req, res) => {
if (job) res.json({ job: queue.jobPublic(job) });
if (job) res.json({ job: queue.jobPublic(job) });
});
});
-router.post('/:jobId/approve', write, (req, res) => {
+router.post('/:jobId/approve', transform, (req, res) => {
try {
try {
const job = load(req, res);
const job = load(req, res);
- if (!job) return;
+ if (!job || !mayTransform(req, res, job.object_id ? model.getObject(job.object_id) : null)) return;
if (job.status !== 'proposed') return problem(res, 409, 'media.job.not_proposed', `The job is ${job.status}; only proposals are approved`);
if (job.status !== 'proposed') return problem(res, 409, 'media.job.not_proposed', `The job is ${job.status}; only proposals are approved`);
const out = queue.approve(job.id, { by: who(req) });
const out = queue.approve(job.id, { by: who(req) });
if (!out) return problem(res, 409, 'media.job.not_proposed', 'The job was decided meanwhile');
if (!out) return problem(res, 409, 'media.job.not_proposed', 'The job was decided meanwhile');
@@ -125,13 +137,13 @@ router.post('/:jobId/approve', write, (req, res) => {
@@ -125,13 +137,13 @@ router.post('/:jobId/approve', write, (req, res) => {
function cancelHandler(req, res) {
function cancelHandler(req, res) {
try {
try {
const job = load(req, res);
const job = load(req, res);
- if (!job) return;
+ if (!job || !mayTransform(req, res, job.object_id ? model.getObject(job.object_id) : null)) return;
const out = queue.cancel(job.id, { by: who(req) });
const out = queue.cancel(job.id, { by: who(req) });
if (out.finished) return problem(res, 409, 'media.job.finished', `The job already ${out.job.status}`);
if (out.finished) return problem(res, 409, 'media.job.finished', `The job already ${out.job.status}`);
res.status(out.pending ? 202 : 200).json({ job: queue.jobPublic(out.job) });
res.status(out.pending ? 202 : 200).json({ job: queue.jobPublic(out.job) });
} catch (err) { sendJobError(res, err); }
} catch (err) { sendJobError(res, err); }
}
}
-router.post('/:jobId/cancel', write, cancelHandler);
-router.delete('/:jobId', write, cancelHandler);
+router.post('/:jobId/cancel', transform, cancelHandler);
+router.delete('/:jobId', transform, cancelHandler);
module.exports = router;
module.exports = router;
@@ -136,13 +136,18 @@ function isHeldRow(row) {
@@ -136,13 +136,18 @@ function isHeldRow(row) {
const OBJECT_COLUMNS = ['app_id', 'namespace', 'kind', 'owner_subject', 'owner_app', 'owner_user_id', 'visibility',
const OBJECT_COLUMNS = ['app_id', 'namespace', 'kind', 'owner_subject', 'owner_app', 'owner_user_id', 'visibility',
'lifecycle_status', 'mime_type', 'size_bytes', 'content_hash', 'canonical_provider', 'canonical_key', 'legacy_ref', 'metadata', 'deleted_at'];
'lifecycle_status', 'mime_type', 'size_bytes', 'content_hash', 'canonical_provider', 'canonical_key', 'legacy_ref', 'metadata', 'deleted_at'];
+/** The root namespace of tenant `appId`: its app id, or app.<project_id>[.sandbox] for a developer project. */
+function tenantRoot(appId) {
+ return db.rootNamespace(db.getApp(appId) || { app_id: appId });
+}
+
function createObject(o) {
function createObject(o) {
if (!KINDS.includes(o.kind)) throw new Error(`unknown kind ${o.kind}`);
if (!KINDS.includes(o.kind)) throw new Error(`unknown kind ${o.kind}`);
const id = ids.newId('media', o.createdMs);
const id = ids.newId('media', o.createdMs);
db.run(`INSERT INTO media_objects (id, app_id, namespace, kind, owner_subject, owner_app, owner_user_id, visibility,
db.run(`INSERT INTO media_objects (id, app_id, namespace, kind, owner_subject, owner_app, owner_user_id, visibility,
lifecycle_status, mime_type, size_bytes, content_hash, canonical_provider, canonical_key, legacy_ref, metadata, created_at)
lifecycle_status, mime_type, size_bytes, content_hash, canonical_provider, canonical_key, legacy_ref, metadata, created_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, COALESCE(?, CURRENT_TIMESTAMP))`,
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, COALESCE(?, CURRENT_TIMESTAMP))`,
- [id, o.app_id, o.namespace || o.app_id, o.kind, o.owner_subject || null, o.owner_app || null, o.owner_user_id ?? null,
+ [id, o.app_id, o.namespace || tenantRoot(o.app_id), o.kind, o.owner_subject || null, o.owner_app || null, o.owner_user_id ?? null,
VISIBILITIES.includes(o.visibility) ? o.visibility : 'private', LIFECYCLES.includes(o.lifecycle_status) ? o.lifecycle_status : 'uploading',
VISIBILITIES.includes(o.visibility) ? o.visibility : 'private', LIFECYCLES.includes(o.lifecycle_status) ? o.lifecycle_status : 'uploading',
o.mime_type || null, Number(o.size_bytes) || 0, o.content_hash || null, o.canonical_provider || null, o.canonical_key || null,
o.mime_type || null, Number(o.size_bytes) || 0, o.content_hash || null, o.canonical_provider || null, o.canonical_key || null,
o.legacy_ref || null, JSON.stringify(o.metadata || {}), o.created_at || null]);
o.legacy_ref || null, JSON.stringify(o.metadata || {}), o.created_at || null]);
@@ -219,7 +224,7 @@ function project(p) {
@@ -219,7 +224,7 @@ function project(p) {
const found = getObjectByLegacyRef(p.legacy_ref) || (p.existingId ? getObject(p.existingId) : null);
const found = getObjectByLegacyRef(p.legacy_ref) || (p.existingId ? getObject(p.existingId) : null);
const canon = p.locations.find(l => l.provider === p.canonical) || p.locations[0] || null;
const canon = p.locations.find(l => l.provider === p.canonical) || p.locations[0] || null;
const fields = {
const fields = {
- app_id: p.app_id, namespace: p.app_id, kind: p.kind,
+ app_id: p.app_id, namespace: tenantRoot(p.app_id), kind: p.kind,
owner_app: p.owner_app || p.app_id, owner_user_id: p.owner_user_id ?? null,
owner_app: p.owner_app || p.app_id, owner_user_id: p.owner_user_id ?? null,
visibility: p.visibility, lifecycle_status: p.lifecycle_status,
visibility: p.visibility, lifecycle_status: p.lifecycle_status,
mime_type: p.mime_type || null, size_bytes: Number(p.size_bytes) || 0, content_hash: p.content_hash || null,
mime_type: p.mime_type || null, size_bytes: Number(p.size_bytes) || 0, content_hash: p.content_hash || null,
@@ -633,6 +638,7 @@ function softDelete(obj, { by = null } = {}) {
@@ -633,6 +638,7 @@ function softDelete(obj, { by = null } = {}) {
db.getDb().transaction(() => {
db.getDb().transaction(() => {
db.run(`UPDATE media_objects SET lifecycle_status = 'deleted', deleted_at = CURRENT_TIMESTAMP, metadata = ?, updated_at = CURRENT_TIMESTAMP
db.run(`UPDATE media_objects SET lifecycle_status = 'deleted', deleted_at = CURRENT_TIMESTAMP, metadata = ?, updated_at = CURRENT_TIMESTAMP
WHERE id = ?`, [JSON.stringify(md), obj.id]);
WHERE id = ?`, [JSON.stringify(md), obj.id]);
+ require('./namespaces').settle(obj.id); // an upload deleted before it completed holds no quota
events.recordObjectChanges(); // media.object.deleted commits with the delete
events.recordObjectChanges(); // media.object.deleted commits with the delete
})();
})();
events.kick();
events.kick();
@@ -673,17 +679,15 @@ function purgeExpired({ retentionDays = config.objects.retentionDays } = {}) {
@@ -673,17 +679,15 @@ function purgeExpired({ retentionDays = config.objects.retentionDays } = {}) {
return purged;
return purged;
}
}
-/** Bytes counted against apps.quota_bytes: v1 files + live native objects (reservations included). */
+/**
+ * Bytes counted against the tenant's quota: v1 files, ready native objects (in a developer project
+ * also soft-deleted ones until purged) and the reservations of uploads in progress
+ * (objects/namespaces.usage of the tenant's root, which is the whole tenant).
+ */
function usedBytes(appId, excludeId = null) {
function usedBytes(appId, excludeId = null) {
- // Developer-project tenants (third-party uploaders) also pay for soft-deleted bytes until they are
- // purged: those stay on disk for the retention period, and upload -> delete -> upload would
- // otherwise grow the disk without bound.
- const app = db.getApp(appId);
- const retained = app && app.project_id ? " OR (lifecycle_status = 'deleted' AND json_extract(metadata, '$.purged_at') IS NULL)" : '';
- const native = db.get(`SELECT COALESCE(SUM(size_bytes), 0) AS b FROM media_objects
- WHERE app_id = ? AND legacy_ref IS NULL AND (lifecycle_status IN ('uploading', 'ready')${retained}) AND id != ?`,
- [appId, excludeId || '']).b;
- return db.appFilesBytes(appId) + native;
+ const app = db.getApp(appId) || { app_id: appId };
+ const u = require('./namespaces').usage(app, db.rootNamespace(app), { excludeId });
+ return u.used_bytes + u.reserved_bytes;
}
}
// ── Public shape ─────────────────────────────────────────────
// ── Public shape ─────────────────────────────────────────────
@@ -11,7 +11,8 @@
@@ -11,7 +11,8 @@
* Complete checks that every part is there (and matches the client's per-part sha256 when given),
* Complete checks that every part is there (and matches the client's per-part sha256 when given),
* concatenates them into OBJECTS_PATH/<app>/<id> while hashing the whole object, and then the route
* concatenates them into OBJECTS_PATH/<app>/<id> while hashing the whole object, and then the route
* runs the same checks as a single-part complete. Abort (or expiry after MEDIA_MULTIPART_TTL_HOURS)
* runs the same checks as a single-part complete. Abort (or expiry after MEDIA_MULTIPART_TTL_HOURS)
- * deletes the parts; the object stays `uploading` so it can be sent again.
+ * deletes the parts and releases the session's quota reservation; the object stays `uploading` so it
+ * can be sent again (a new session or upload URL reserves again).
*/
*/
'use strict';
'use strict';
@@ -83,7 +84,7 @@ function initiate(obj, { partSize: requested } = {}) {
@@ -83,7 +84,7 @@ function initiate(obj, { partSize: requested } = {}) {
const free = freeBytes();
const free = freeBytes();
if (free < need) return { status: 507, code: 'media.storage.insufficient', error: `Not enough free space for this upload (${Math.floor(free / MB)} MB free)` };
if (free < need) return { status: 507, code: 'media.storage.insufficient', error: `Not enough free space for this upload (${Math.floor(free / MB)} MB free)` };
const prior = activeFor(obj.id);
const prior = activeFor(obj.id);
- if (prior) abort(prior.id);
+ if (prior) abort(prior.id, 'aborted', { release: false }); // replaced: the object's reservation carries over
const id = `mup_${ids.ulid()}`;
const id = `mup_${ids.ulid()}`;
const parts = Math.max(1, Math.ceil(total / chosen.size));
const parts = Math.max(1, Math.ceil(total / chosen.size));
db.run(`INSERT INTO media_uploads (id, object_id, app_id, part_size, total_size, parts_expected, expires_at)
db.run(`INSERT INTO media_uploads (id, object_id, app_id, part_size, total_size, parts_expected, expires_at)
@@ -227,10 +228,15 @@ function removeParts(uploadId) {
@@ -227,10 +228,15 @@ function removeParts(uploadId) {
db.run('DELETE FROM media_upload_parts WHERE upload_id = ?', [uploadId]);
db.run('DELETE FROM media_upload_parts WHERE upload_id = ?', [uploadId]);
}
}
-/** Delete a session's parts; the object stays uploading. */
-function abort(uploadId, status = 'aborted') {
+/**
+ * Delete a session's parts; the object stays uploading. Its quota reservation is released down to
+ * what is still stored for it (objects/namespaces.release), unless a new session replaces this one.
+ */
+function abort(uploadId, status = 'aborted', { release = true } = {}) {
+ const session = getSession(uploadId);
db.run("UPDATE media_uploads SET status = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ? AND status IN ('active', 'completing')", [status, uploadId]);
db.run("UPDATE media_uploads SET status = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ? AND status IN ('active', 'completing')", [status, uploadId]);
removeParts(uploadId);
removeParts(uploadId);
+ if (release && session) require('./namespaces').release(session.object_id);
}
}
/**
/**
new file mode 100644
new file mode 100644
@@ -0,0 +1,57 @@
@@ -0,0 +1,57 @@
+/**
+ * OpenVibe.Media — a tenant's namespaces (mounted at /api/v2/:app/namespaces;
+ * docs/object-model.md#namespaces-grants-and-quotas)
+ *
+ * GET / the namespaces the caller may list: owner, policy, quota and usage
+ * GET /:namespace one (its full name, or a name relative to the tenant's root)
+ *
+ * The owner's read. The app key sees every namespace of its tenant; a Network token (a first-party
+ * service's, or a developer project's app token on /<project_id>/) sees those it holds list
+ * (GET /: media.object.list, or media.object.read) or read (GET /:namespace: media.object.read) for.
+ * Usage is counted from the rows when read, and the snapshot columns are refreshed with it. Quotas and
+ * policies are set by the operator (scripts/namespaces.js).
+ */
+'use strict';
+
+const express = require('express');
+const { http } = require('openvibe-contracts');
+const namespaces = require('./namespaces');
+const { tenantAuth, tenantCors, namespaceGrant } = require('../auth');
+
+const router = express.Router({ mergeParams: true });
+router.use(tenantCors);
+
+function problem(res, status, code, detail) {
+ return http.sendProblem(res, status, code, { detail });
+}
+
+function shape(req, row) {
+ return namespaces.publicShape(req.appRow, row, namespaces.reconcile(req.appRow, row.namespace));
+}
+
+router.get('/', tenantAuth({ verb: 'list', namespaced: true }), (req, res) => {
+ try {
+ const rows = namespaces.listForTenant(req.appId).filter(r => namespaceGrant(req, 'list', r.namespace).allowed);
+ res.json({ namespaces: rows.map(r => shape(req, r)) });
+ } catch (err) {
+ console.error('[Namespaces] list error:', err.message);
+ problem(res, 500, 'media.namespace.list_failed', 'Failed to list namespaces');
+ }
+});
+
+router.get('/:namespace', tenantAuth({ verb: 'read', namespaced: true }), (req, res) => {
+ try {
+ const named = namespaces.resolveName(req.appRow, req.params.namespace);
+ if (named.error) return problem(res, 400, 'media.namespace.invalid', named.error);
+ const g = namespaceGrant(req, 'read', named.namespace);
+ if (!g.allowed) return problem(res, 403, g.code, g.reason);
+ const row = namespaces.get(named.namespace);
+ if (!row || row.app_id !== req.appId) return problem(res, 404, 'media.namespace.not_found', 'No such namespace in this tenant');
+ res.json(shape(req, row));
+ } catch (err) {
+ console.error('[Namespaces] read error:', err.message);
+ problem(res, 500, 'media.namespace.read_failed', 'Failed to read the namespace');
+ }
+});
+
+module.exports = router;
new file mode 100644
new file mode 100644
@@ -0,0 +1,334 @@
@@ -0,0 +1,334 @@
+/**
+ * OpenVibe.Media — namespaces, quotas and upload reservations (roadmap WS-G task 2;
+ * docs/object-model.md#namespaces-grants-and-quotas)
+ *
+ * Every object lives in a namespace, and every namespace is a media_namespaces row: the tenant it
+ * belongs to, its owner, a policy, quotas and a snapshot of its usage. A tenant has one root:
+ *
+ * first-party tenant its app id live, community, tools, …
+ * developer project, production app.<project_id> (tenant prj_<ULID>)
+ * developer project, sandbox app.<project_id>.sandbox (tenant prj_<ULID>-sandbox)
+ *
+ * and children below it (<root>.<segment>, up to three segments), made by the first upload that
+ * names one. Which verbs a token may use in which namespace is decided in server/auth.js; this module
+ * keeps the rows, the policy and the quotas.
+ *
+ * Quotas. A namespace's limit covers its subtree (itself and its children). quota_bytes NULL inherits
+ * (the root takes its tenant's apps.quota_bytes, a child has no limit of its own), 0 is no limit.
+ * Usage counts native objects that are ready (in a developer project, also soft-deleted ones until
+ * their bytes are purged), the tenant's v1 files at the root, and the reservations of uploads in
+ * progress. An upload reserves its declared size at init (checked against every limit from its
+ * namespace up to the root), re-reserves the bytes it actually stored, and is checked again with its
+ * real size at complete, when the reservation is settled; abort releases it down to the bytes still
+ * stored; the hourly sweep fails an upload whose reservation expired and frees its bytes. Checks count
+ * from the rows, never from the snapshot; reconcile() refreshes the snapshot.
+ */
+'use strict';
+
+const fs = require('fs');
+const path = require('path');
+const db = require('../db/database');
+const config = require('../config');
+
+const SEGMENT_RE = /^[a-z0-9][a-z0-9_-]{0,39}$/;
+const MAX_DEPTH = 3;
+const POLICY_KEYS = ['kinds', 'visibilities', 'max_object_bytes', 'strict_verbs'];
+
+function parsePolicy(s) {
+ try {
+ const p = typeof s === 'string' ? JSON.parse(s || '{}') : s;
+ return p && typeof p === 'object' && !Array.isArray(p) ? p : {};
+ } catch { return {}; }
+}
+
+/** A policy an operator sets: only the known keys, each of the right shape. Returns { policy } or { error }. */
+function validatePolicy(p) {
+ if (!p || typeof p !== 'object' || Array.isArray(p)) return { error: 'policy must be an object' };
+ const unknown = Object.keys(p).filter(k => !POLICY_KEYS.includes(k));
+ if (unknown.length) return { error: `unknown policy key(s): ${unknown.join(', ')} (known: ${POLICY_KEYS.join(', ')})` };
+ const model = require('./model');
+ if (p.kinds != null && !(Array.isArray(p.kinds) && p.kinds.every(k => model.KINDS.includes(k)))) return { error: `kinds must be a list of ${model.KINDS.join(', ')}` };
+ if (p.visibilities != null && !(Array.isArray(p.visibilities) && p.visibilities.every(v => model.VISIBILITIES.includes(v)))) return { error: `visibilities must be a list of ${model.VISIBILITIES.join(', ')}` };
+ if (p.max_object_bytes != null && !(Number.isInteger(p.max_object_bytes) && p.max_object_bytes >= 0)) return { error: 'max_object_bytes must be a whole number of bytes (0 = no limit)' };
+ if (p.strict_verbs != null && typeof p.strict_verbs !== 'boolean') return { error: 'strict_verbs must be true or false' };
+ return { policy: p };
+}
+
+const sqlTime = (ms) => new Date(ms).toISOString().replace('T', ' ').slice(0, 19);
+
+// ── Rows ─────────────────────────────────────────────────────
+
+function get(namespace) {
+ return namespace ? db.get('SELECT * FROM media_namespaces WHERE namespace = ?', [String(namespace)]) : null;
+}
+
+function listForTenant(appId) {
+ return db.all('SELECT * FROM media_namespaces WHERE app_id = ? ORDER BY namespace', [appId]);
+}
+
+/** Is `ns` the namespace `base` or below it? */
+function within(base, ns) {
+ return ns === base || String(ns || '').startsWith(`${base}.`);
+}
+
+/** The tenant's rows from its root down to `namespace` (those that exist), root first. */
+function chain(appId, namespace) {
+ const parts = String(namespace || '').split('.');
+ const names = parts.map((_, i) => parts.slice(0, i + 1).join('.'));
+ if (!names.length) return [];
+ const rows = db.all(`SELECT * FROM media_namespaces WHERE app_id = ? AND namespace IN (${names.map(() => '?').join(', ')})`, [appId, ...names]);
+ return rows.sort((a, b) => a.namespace.length - b.namespace.length);
+}
+
+/** Policy in force in `namespace`: the root's, overridden key by key by each child on the way down. */
+function effectivePolicy(appId, namespace) {
+ return Object.assign({}, ...chain(appId, namespace).map(r => parsePolicy(r.policy)));
+}
+
+function isStrict(appId, namespace) {
+ return effectivePolicy(appId, namespace).strict_verbs === true;
+}
+
+/**
+ * The namespace a request names in tenant `app`: nothing = the root; the root's full name or a name
+ * below it; or a name relative to the root ('avatars' = '<root>.avatars'). Returns { namespace } or { error }.
+ */
+function resolveName(app, requested) {
+ const root = db.rootNamespace(app);
+ if (requested == null || requested === '') return { namespace: root };
+ const s = String(requested).trim();
+ if (s === root) return { namespace: root };
+ const rel = s.startsWith(`${root}.`) ? s.slice(root.length + 1) : s;
+ const segs = rel.split('.');
+ if (segs.length > MAX_DEPTH || !segs.every(x => SEGMENT_RE.test(x)) || `${root}.${rel}`.length > 200) {
+ return { error: `namespace must be ${root} or a name below it: up to ${MAX_DEPTH} segments of a-z, 0-9, _ and - (e.g. ${root}.avatars)` };
+ }
+ // The production root's `sandbox` child would read as the project's sandbox tenant.
+ if (app.project_id && app.env !== 'sandbox' && segs[0] === 'sandbox') return { error: `${root}.sandbox is the project's sandbox tenant` };
+ return { namespace: `${root}.${rel}` };
+}
+
+/**
+ * The row for `namespace` in tenant `app`, with the chain above it created when missing. Returns the
+ * row, or { error, status, code } when the tenant already has MEDIA_NAMESPACE_MAX_CHILDREN children
+ * or the name belongs to another tenant.
+ */
+function ensure(app, namespace) {
+ const root = db.ensureRootNamespace(app);
+ const found = get(namespace);
+ if (found) {
+ return found.app_id === app.app_id ? found
+ : { error: `namespace ${namespace} belongs to another tenant`, status: 409, code: 'media.namespace.conflict' };
+ }
+ if (!within(root, namespace)) return { error: `namespace ${namespace} is not below ${root}`, status: 400, code: 'media.namespace.invalid' };
+ const segs = namespace.slice(root.length + 1).split('.');
+ const names = segs.map((_, i) => `${root}.${segs.slice(0, i + 1).join('.')}`);
+ const missing = names.filter(n => !get(n));
+ const children = db.get('SELECT COUNT(*) AS n FROM media_namespaces WHERE app_id = ? AND parent IS NOT NULL', [app.app_id]).n;
+ if (children + missing.length > config.objects.maxChildNamespaces) {
+ return { error: `a tenant has at most ${config.objects.maxChildNamespaces} namespaces below its root`, status: 413, code: 'media.quota.namespaces_exceeded' };
+ }
+ let parent = root;
+ for (const name of names) {
+ db.run('INSERT OR IGNORE INTO media_namespaces (namespace, app_id, parent, owner) VALUES (?, ?, ?, ?)', [name, app.app_id, parent, db.namespaceOwner(app)]);
+ parent = name;
+ }
+ return get(namespace);
+}
+
+// ── Usage and quotas ─────────────────────────────────────────
+
+/**
+ * What the subtree of `namespace` uses now, from the rows: { used_bytes, used_objects, reserved_bytes,
+ * reserved_objects }. The root's subtree is the whole tenant, v1 files included. `excludeId` leaves
+ * one object (and its reservation) out, for re-checking that object.
+ */
+function usage(app, namespace, { excludeId = null } = {}) {
+ const root = db.rootNamespace(app);
+ const whole = namespace === root;
+ const where = whole ? '' : ' AND (o.namespace = @ns OR substr(o.namespace, 1, @plen) = @prefix)';
+ const rwhere = whole ? '' : ' AND (r.namespace = @ns OR substr(r.namespace, 1, @plen) = @prefix)';
+ const p = { app: app.app_id, ns: namespace, plen: String(namespace).length + 1, prefix: `${namespace}.`, ex: excludeId || '' };
+ // Developer projects (third-party uploaders) pay for soft-deleted bytes until they are purged: they stay
+ // on disk for the retention period, and upload -> delete -> upload would otherwise grow the disk.
+ const retained = app.project_id ? " OR (o.lifecycle_status = 'deleted' AND json_extract(o.metadata, '$.purged_at') IS NULL)" : '';
+ const used = db.get(`SELECT COUNT(*) AS n, COALESCE(SUM(o.size_bytes), 0) AS b FROM media_objects o
+ WHERE o.app_id = @app AND o.legacy_ref IS NULL AND (o.lifecycle_status = 'ready'${retained}) AND o.id != @ex${where}`, p);
+ // A reservation counts while its object is still uploading (a settled or failed one never double-counts).
+ const reserved = db.get(`SELECT COUNT(*) AS n, COALESCE(SUM(r.bytes), 0) AS b FROM media_quota_reservations r
+ JOIN media_objects o ON o.id = r.object_id AND o.lifecycle_status = 'uploading'
+ WHERE r.app_id = @app AND r.object_id != @ex${rwhere}`, p);
+ const files = whole ? db.get('SELECT COUNT(*) AS n, COALESCE(SUM(size), 0) AS b FROM files WHERE app_id = ?', [app.app_id]) : { n: 0, b: 0 };
+ return { used_bytes: used.b + files.b, used_objects: used.n + files.n, reserved_bytes: reserved.b, reserved_objects: reserved.n };
+}
+
+/** The limits of one row: { bytes, objects, bytes_from } (0 = none). A root with no quota_bytes takes its tenant's. */
+function limitsOf(app, row) {
+ const inherits = row.quota_bytes == null;
+ const bytes = inherits ? (row.parent == null ? Number(app.quota_bytes) || 0 : 0) : Number(row.quota_bytes) || 0;
+ return { bytes, objects: Number(row.quota_objects) || 0, bytes_from: bytes ? (inherits ? 'tenant' : 'namespace') : null };
+}
+
+/**
+ * Would adding `bytes` and `objects` to `namespace` break a limit? Checked for the namespace and every
+ * ancestor with a limit (the root always carries the tenant's quota), and against the effective
+ * policy's max_object_bytes. Returns null, or { status, code, detail, extra } for a problem answer.
+ */
+function checkQuota(app, namespace, { bytes = 0, objects = 0, excludeId = null } = {}) {
+ const root = db.rootNamespace(app);
+ const rows = chain(app.app_id, namespace);
+ if (!rows.length || rows[0].namespace !== root) rows.unshift({ namespace: root, parent: null, quota_bytes: null, quota_objects: null });
+ const policy = Object.assign({}, ...rows.map(r => parsePolicy(r.policy)));
+ const maxObject = Number(policy.max_object_bytes) || 0;
+ if (maxObject && bytes > maxObject) {
+ return { status: 413, code: 'media.object.too_large', detail: `Objects in ${namespace} are limited to ${maxObject} bytes`, extra: { namespace, max_object_bytes: maxObject } };
+ }
+ for (const row of rows.slice().reverse()) {
+ const lim = limitsOf(app, row);
+ if (!lim.bytes && !lim.objects) continue;
+ const u = usage(app, row.namespace, { excludeId });
+ if (lim.bytes && u.used_bytes + u.reserved_bytes + bytes > lim.bytes) {
+ return {
+ status: 413, code: 'media.quota.exceeded', detail: `The storage quota of ${row.namespace} would be exceeded`,
+ extra: { namespace: row.namespace, quota_bytes: lim.bytes, used_bytes: u.used_bytes, reserved_bytes: u.reserved_bytes },
+ };
+ }
+ if (lim.objects && objects > 0 && u.used_objects + u.reserved_objects + objects > lim.objects) {
+ return {
+ status: 413, code: 'media.quota.objects_exceeded', detail: `The object quota of ${row.namespace} would be exceeded`,
+ extra: { namespace: row.namespace, quota_objects: lim.objects, used_objects: u.used_objects, reserved_objects: u.reserved_objects },
+ };
+ }
+ }
+ return null;
+}
+
+// ── Reservations ─────────────────────────────────────────────
+
+function reservation(objectId) {
+ return objectId ? db.get('SELECT * FROM media_quota_reservations WHERE object_id = ?', [objectId]) : null;
+}
+
+/** Hold `bytes` of quota for the upload of `obj` (replacing what it held); expiry moves forward, never back. */
+function reserve(obj, bytes, { hours = config.objects.reservationHours } = {}) {
+ db.run(`INSERT INTO media_quota_reservations (object_id, app_id, namespace, bytes, expires_at) VALUES (?, ?, ?, ?, ?)
+ ON CONFLICT(object_id) DO UPDATE SET bytes = excluded.bytes, namespace = excluded.namespace,
+ expires_at = MAX(expires_at, excluded.expires_at), updated_at = CURRENT_TIMESTAMP`,
+ [obj.id, obj.app_id, obj.namespace, Math.max(0, Number(bytes) || 0), sqlTime(Date.now() + hours * 3600 * 1000)]);
+}
+
+/** An upload step happened: its reservation's expiry moves forward. */
+function touch(objectId, { hours = config.objects.reservationHours } = {}) {
+ db.run('UPDATE media_quota_reservations SET expires_at = MAX(expires_at, ?), updated_at = CURRENT_TIMESTAMP WHERE object_id = ?',
+ [sqlTime(Date.now() + hours * 3600 * 1000), objectId]);
+}
+
+/** The upload ended (complete, or the object was deleted): the object's own row counts from now on. */
+function settle(objectId) {
+ db.run('DELETE FROM media_quota_reservations WHERE object_id = ?', [objectId]);
+}
+
+/** An aborted or expired session: keep holding only what is still stored for the object (a PUT's bytes). */
+function release(objectId) {
+ const obj = db.get('SELECT lifecycle_status FROM media_objects WHERE id = ?', [objectId]);
+ const stored = obj && obj.lifecycle_status === 'uploading'
+ ? db.get("SELECT size_bytes FROM media_locations WHERE object_id = ? AND provider = 'local' AND state = 'present'", [objectId]) : null;
+ if (stored && Number(stored.size_bytes) > 0) db.run('UPDATE media_quota_reservations SET bytes = ?, updated_at = CURRENT_TIMESTAMP WHERE object_id = ?', [Number(stored.size_bytes), objectId]);
+ else settle(objectId);
+}
+
+/**
+ * The hourly sweep: an upload whose reservation expired, with no multipart session open, is failed
+ * (metadata.failure upload_expired), its stored bytes are deleted and its reservation goes; a
+ * reservation whose object is no longer uploading is dropped. Returns { expired, dropped }.
+ */
+function expireReservations() {
+ const rows = db.all(`SELECT r.object_id, r.app_id, r.namespace, o.lifecycle_status, o.legacy_ref, o.metadata FROM media_quota_reservations r
+ LEFT JOIN media_objects o ON o.id = r.object_id WHERE r.expires_at < datetime('now') ORDER BY r.expires_at LIMIT 1000`);
+ const multipart = require('./multipart');
+ const root = path.resolve(config.objects.path) + path.sep;
+ const touched = new Map();
+ let expired = 0, dropped = 0;
+ for (const r of rows) {
+ if (r.lifecycle_status !== 'uploading' || r.legacy_ref) { settle(r.object_id); dropped++; continue; }
+ if (multipart.activeFor(r.object_id)) continue; // its session expires on its own, then this does
+ try {
+ const md = parsePolicy(r.metadata);
+ md.failure = 'upload_expired';
+ md.expired_at = new Date().toISOString();
+ const locs = db.all("SELECT * FROM media_locations WHERE object_id = ? AND provider = 'local'", [r.object_id]);
+ db.getDb().transaction(() => {
+ db.run("UPDATE media_objects SET lifecycle_status = 'failed', metadata = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ? AND lifecycle_status = 'uploading'",
+ [JSON.stringify(md), r.object_id]);
+ for (const l of locs) db.run('DELETE FROM media_locations WHERE id = ?', [l.id]);
+ settle(r.object_id);
+ })();
+ for (const l of locs) if (path.resolve(l.key).startsWith(root)) { try { fs.unlinkSync(l.key); } catch { /* already gone */ } }
+ expired++;
+ touched.set(`${r.app_id}\n${r.namespace}`, r);
+ } catch (err) {
+ // A retention hold refuses the lifecycle change: the reservation stays until the hold is released.
+ console.warn(`[Namespaces] Could not expire the upload of ${r.object_id}: ${err.message}`);
+ }
+ }
+ for (const r of touched.values()) { const app = db.getApp(r.app_id); if (app) reconcileChain(app, r.namespace); }
+ return { expired, dropped };
+}
+
+// ── Snapshot (the used_* / reserved_* columns) ───────────────
+
+function reconcile(app, namespace) {
+ const u = usage(app, namespace);
+ db.run(`UPDATE media_namespaces SET used_bytes = ?, used_objects = ?, reserved_bytes = ?, reserved_objects = ?,
+ reconciled_at = CURRENT_TIMESTAMP, updated_at = CURRENT_TIMESTAMP WHERE namespace = ? AND app_id = ?`,
+ [u.used_bytes, u.used_objects, u.reserved_bytes, u.reserved_objects, namespace, app.app_id]);
+ return u;
+}
+
+/** Refresh `namespace` and every namespace above it (after a write changed its usage). */
+function reconcileChain(app, namespace) {
+ try {
+ for (const row of chain(app.app_id, namespace)) reconcile(app, row.namespace);
+ } catch (err) { console.warn(`[Namespaces] reconcile ${namespace}: ${err.message}`); }
+}
+
+/** Refresh every row (the hourly job). Returns how many. */
+function reconcileAll() {
+ const apps = new Map();
+ let n = 0;
+ for (const row of db.all('SELECT namespace, app_id FROM media_namespaces ORDER BY app_id, namespace')) {
+ if (!apps.has(row.app_id)) apps.set(row.app_id, db.getApp(row.app_id) || { app_id: row.app_id });
+ reconcile(apps.get(row.app_id), row.namespace);
+ n++;
+ }
+ return n;
+}
+
+// ── Public shape ─────────────────────────────────────────────
+
+/** A namespace as its owner reads it; `fresh` is usage() just computed (else the snapshot is shown). */
+function publicShape(app, row, fresh = null) {
+ const lim = limitsOf(app, row);
+ const u = fresh || { used_bytes: row.used_bytes, used_objects: row.used_objects, reserved_bytes: row.reserved_bytes, reserved_objects: row.reserved_objects };
+ const out = {
+ namespace: row.namespace,
+ parent: row.parent || null,
+ root: row.parent == null,
+ owner: row.owner,
+ policy: parsePolicy(row.policy),
+ effective_policy: effectivePolicy(row.app_id, row.namespace),
+ quota: { bytes: lim.bytes || null, objects: lim.objects || null, bytes_from: lim.bytes_from },
+ usage: { ...u, reconciled_at: fresh ? new Date().toISOString() : row.reconciled_at || null },
+ created_at: row.created_at,
+ };
+ if (db.isSandboxTenant(row.app_id)) out.sandbox = true;
+ return out;
+}
+
+module.exports = {
+ SEGMENT_RE, POLICY_KEYS, parsePolicy, validatePolicy,
+ get, listForTenant, within, chain, effectivePolicy, isStrict, resolveName, ensure,
+ usage, limitsOf, checkQuota,
+ reservation, reserve, touch, settle, release, expireReservations,
+ reconcile, reconcileChain, reconcileAll, publicShape,
+};
@@ -12,7 +12,7 @@
@@ -12,7 +12,7 @@
* POST /:id/multipart/:uploadId/complete assemble + the same checks as complete → ready
* POST /:id/multipart/:uploadId/complete assemble + the same checks as complete → ready
* DELETE /:id/multipart/:uploadId abort (parts deleted; the object stays uploading)
* DELETE /:id/multipart/:uploadId abort (parts deleted; the object stays uploading)
* GET /:id metadata (id = med_… or legacy:<app>:<kind>:<id>)
* GET /:id metadata (id = med_… or legacy:<app>:<kind>:<id>)
- * GET / cursor list (?kind&visibility&status&owner&user_id&include_deleted&limit&cursor)
+ * GET / cursor list (?namespace&kind&visibility&status&owner&user_id&include_deleted&limit&cursor)
* DELETE /:id soft delete (bytes kept MEDIA_DELETE_RETENTION_DAYS); 409 when held
* DELETE /:id soft delete (bytes kept MEDIA_DELETE_RETENTION_DAYS); 409 when held
* POST /:id/restore undo a soft delete inside the retention period
* POST /:id/restore undo a soft delete inside the retention period
* GET /:id/download 302 to the public location, or a signed short-lived URL for private objects
* GET /:id/download 302 to the public location, or a signed short-lived URL for private objects
@@ -21,8 +21,13 @@
@@ -21,8 +21,13 @@
* DELETE /:id/holds/:holdId release it (app key only)
* DELETE /:id/holds/:holdId release it (app key only)
* Staff place and release holds by vod/clip id at /api/v1/:app/admin/storage/holds.
* Staff place and release holds by vod/clip id at /api/v1/:app/admin/storage/holds.
*
*
- * Auth: the app's API key, or a Network service token granting media.object.upload
- * (writes) / media.object.read (reads) for namespace = :app. Developer-project tenants
+ * Auth: the app's API key, or a Network token holding the route's verb for the object's namespace
+ * (server/auth.js VERBS): write (init, content, complete, multipart) media.object.upload; read
+ * (metadata, download, holds) media.object.read; list media.object.list; delete and restore
+ * media.object.delete. The older ids keep granting the newer verbs (read lists, upload deletes).
+ * Init takes `namespace` (the tenant's root by default, or a name below it; objects/namespaces.js),
+ * reserves its declared size against every quota from there up to the tenant, and complete settles
+ * the reservation against the bytes received. Developer-project tenants
* (:app = prj_<ULID>) take their project's app tokens only (server/auth.js); their sandbox
* (:app = prj_<ULID>) take their project's app tokens only (server/auth.js); their sandbox
* objects are served only through signed URLs, whatever their visibility. X-OV-Subject names the
* objects are served only through signed URLs, whatever their visibility. X-OV-Subject names the
* owner (usr_…); X-OV-User-Id (app key only) acts as one of the app's users. The
* owner (usr_…); X-OV-User-Id (app key only) acts as one of the app's users. The
@@ -46,12 +51,16 @@ const invariant = require('./invariant');
@@ -46,12 +51,16 @@ const invariant = require('./invariant');
const signing = require('./signing');
const signing = require('./signing');
const ctypes = require('./content-type');
const ctypes = require('./content-type');
const multipart = require('./multipart');
const multipart = require('./multipart');
-const { tenantAuth, tenantCors, tenantPath } = require('../auth');
+const namespaces = require('./namespaces');
+const { tenantAuth, tenantCors, tenantPath, namespaceGrant } = require('../auth');
const { announce } = require('../webhooks');
const { announce } = require('../webhooks');
const MB = 1024 * 1024;
const MB = 1024 * 1024;
-const upload = tenantAuth({ capability: 'media.object.upload' });
-const read = tenantAuth({ capability: 'media.object.read' });
+// Each route checks its verb for the namespace it works in (the object's, or the one init names).
+const upload = tenantAuth({ verb: 'write', namespaced: true });
+const read = tenantAuth({ verb: 'read', namespaced: true });
+const list = tenantAuth({ verb: 'list', namespaced: true });
+const remove = tenantAuth({ verb: 'delete', namespaced: true });
const appOnly = tenantAuth();
const appOnly = tenantAuth();
function problem(res, status, code, detail, extra) {
function problem(res, status, code, detail, extra) {
@@ -77,9 +86,12 @@ function canWrite(req, obj) {
@@ -77,9 +86,12 @@ function canWrite(req, obj) {
return req.authType !== 'user' || (obj.owner_user_id != null && obj.owner_user_id === req.userId);
return req.authType !== 'user' || (obj.owner_user_id != null && obj.owner_user_id === req.userId);
}
}
-function load(req, res) {
+/** The object named in the URL, in this tenant and visible to the caller, if the caller may `verb` in its namespace (else answered). */
+function load(req, res, verb = 'read') {
const obj = model.resolveObject(String(req.params.id || ''), req.appId);
const obj = model.resolveObject(String(req.params.id || ''), req.appId);
if (!obj || !canSee(req, obj)) { problem(res, 404, 'media.object.not_found', 'No such object in this namespace'); return null; }
if (!obj || !canSee(req, obj)) { problem(res, 404, 'media.object.not_found', 'No such object in this namespace'); return null; }
+ const g = namespaceGrant(req, verb, obj.namespace);
+ if (!g.allowed) { problem(res, 403, g.code, g.reason); return null; }
return obj;
return obj;
}
}
@@ -107,9 +119,9 @@ function contentAuth(req, res, next) {
@@ -107,9 +119,9 @@ function contentAuth(req, res, next) {
if (tokenTenant(req, res, obj)) next();
if (tokenTenant(req, res, obj)) next();
}
}
-/** Multipart session token (from POST /:id/multipart) or the usual tenant credential with `capability`. */
-function multipartAuth(capability) {
- const fallback = tenantAuth({ capability });
+/** Multipart session token (from POST /:id/multipart) or the usual tenant credential with `verb`. */
+function multipartAuth(verb) {
+ const fallback = tenantAuth({ verb, namespaced: true });
return (req, res, next) => {
return (req, res, next) => {
const token = req.query.token || req.headers['x-upload-token'];
const token = req.query.token || req.headers['x-upload-token'];
if (!token) return fallback(req, res, next);
if (!token) return fallback(req, res, next);
@@ -161,15 +173,25 @@ function sanitizeFilename(name) {
@@ -161,15 +173,25 @@ function sanitizeFilename(name) {
return path.basename(String(name)).replace(/[^\w.\- ]/g, '_').slice(0, 200) || null;
return path.basename(String(name)).replace(/[^\w.\- ]/g, '_').slice(0, 200) || null;
}
}
-function quotaCheck(req, res, addBytes, excludeId = null) {
- const quota = Number(req.appRow.quota_bytes) || 0;
- if (!quota) return true;
- const used = model.usedBytes(req.appId, excludeId);
- if (used + addBytes <= quota) return true;
- problem(res, 413, 'media.quota.exceeded', 'App storage quota exceeded', { quota_bytes: quota, used_bytes: used });
+/**
+ * Room for `bytes` (and `objects` new objects) in `namespace`: every quota from there up to the
+ * tenant, and the policy's max_object_bytes (objects/namespaces.checkQuota). Answers 413 when not.
+ */
+function quotaCheck(req, res, namespace, { bytes = 0, objects = 0, excludeId = null } = {}) {
+ const q = namespaces.checkQuota(req.appRow, namespace, { bytes, objects, excludeId });
+ if (!q) return true;
+ problem(res, q.status, q.code, q.detail, q.extra);
return false;
return false;
}
}
+/** An upload step that has no reservation (it was released at abort) takes one again, checked like init. */
+function reserveAgain(req, res, obj, bytes) {
+ const held = namespaces.reservation(obj.id);
+ if (!quotaCheck(req, res, obj.namespace, { bytes, objects: held ? 0 : 1, excludeId: obj.id })) return false;
+ namespaces.reserve(obj, bytes);
+ return true;
+}
+
// ── Content upload (mounted ahead of the JSON body parser — see index.js) ──
// ── Content upload (mounted ahead of the JSON body parser — see index.js) ──
/** Stream the request body to `tmp`, hashing as it goes; stop past `cap` bytes. */
/** Stream the request body to `tmp`, hashing as it goes; stop past `cap` bytes. */
@@ -196,7 +218,7 @@ function receive(req, tmp, cap) {
@@ -196,7 +218,7 @@ function receive(req, tmp, cap) {
}
}
async function putContent(req, res) {
async function putContent(req, res) {
- const obj = load(req, res);
+ const obj = load(req, res, 'write');
if (!obj) { req.resume(); return; }
if (!obj) { req.resume(); return; }
if (!canWrite(req, obj)) { req.resume(); return problem(res, 403, 'media.object.forbidden', 'Not your object'); }
if (!canWrite(req, obj)) { req.resume(); return problem(res, 403, 'media.object.forbidden', 'Not your object'); }
if (obj.legacy_ref) { req.resume(); return problem(res, 409, 'media.object.legacy_managed', 'This object is written through the v1 API'); }
if (obj.legacy_ref) { req.resume(); return problem(res, 409, 'media.object.legacy_managed', 'This object is written through the v1 API'); }
@@ -224,6 +246,7 @@ async function putContent(req, res) {
@@ -224,6 +246,7 @@ async function putContent(req, res) {
const tmpDir = path.join(config.objects.path, '.tmp');
const tmpDir = path.join(config.objects.path, '.tmp');
fs.mkdirSync(tmpDir, { recursive: true });
fs.mkdirSync(tmpDir, { recursive: true });
const tmp = path.join(tmpDir, `${obj.id}-${crypto.randomBytes(4).toString('hex')}`);
const tmp = path.join(tmpDir, `${obj.id}-${crypto.randomBytes(4).toString('hex')}`);
+ namespaces.touch(obj.id); // the upload is active: its reservation does not expire under it
const got = await receive(req, tmp, cap);
const got = await receive(req, tmp, cap);
const drop = () => { try { fs.unlinkSync(tmp); } catch { /* not written */ } };
const drop = () => { try { fs.unlinkSync(tmp); } catch { /* not written */ } };
if (got.error) {
if (got.error) {
@@ -233,7 +256,12 @@ async function putContent(req, res) {
@@ -233,7 +256,12 @@ async function putContent(req, res) {
}
}
if (declared && got.bytes !== declared) { drop(); return problem(res, 400, 'media.object.size_mismatch', `Declared ${declared} bytes, received ${got.bytes}`); }
if (declared && got.bytes !== declared) { drop(); return problem(res, 400, 'media.object.size_mismatch', `Declared ${declared} bytes, received ${got.bytes}`); }
if (!got.bytes) { drop(); return problem(res, 400, 'media.object.empty', 'No bytes received'); }
if (!got.bytes) { drop(); return problem(res, 400, 'media.object.empty', 'No bytes received'); }
- if (!quotaCheck(req, res, got.bytes, obj.id)) { drop(); return; }
+ // The object may have been completed, deleted or expired while the bytes streamed in.
+ const current = model.getObject(obj.id);
+ if (!current || current.lifecycle_status !== 'uploading') { drop(); return problem(res, 409, 'media.object.not_uploading', `Object is ${current ? current.lifecycle_status : 'gone'}`); }
+ // The real size replaces the declared one in this upload's reservation.
+ const held = namespaces.reservation(obj.id);
+ if (!quotaCheck(req, res, obj.namespace, { bytes: got.bytes, objects: held ? 0 : 1, excludeId: obj.id })) { drop(); return; }
try {
try {
const dest = model.objectFilePath(obj);
const dest = model.objectFilePath(obj);
@@ -245,6 +273,7 @@ async function putContent(req, res) {
@@ -245,6 +273,7 @@ async function putContent(req, res) {
canonical_provider: 'local', canonical_key: dest,
canonical_provider: 'local', canonical_key: dest,
});
});
model.upsertLocation(obj.id, { provider: 'local', key: dest, state: 'present', size_bytes: got.bytes, checksum: got.sha256, verified: true });
model.upsertLocation(obj.id, { provider: 'local', key: dest, state: 'present', size_bytes: got.bytes, checksum: got.sha256, verified: true });
+ namespaces.reserve(current, got.bytes);
})();
})();
res.json({ id: obj.id, size_bytes: got.bytes, content_hash: got.sha256 });
res.json({ id: obj.id, size_bytes: got.bytes, content_hash: got.sha256 });
} catch (err) {
} catch (err) {
@@ -266,6 +295,15 @@ router.post('/', upload, (req, res) => {
@@ -266,6 +295,15 @@ router.post('/', upload, (req, res) => {
if (!model.KINDS.includes(kind)) return problem(res, 400, 'media.object.invalid', `kind must be one of ${model.KINDS.join(', ')}`);
if (!model.KINDS.includes(kind)) return problem(res, 400, 'media.object.invalid', `kind must be one of ${model.KINDS.join(', ')}`);
const visibility = String(b.visibility || 'private');
const visibility = String(b.visibility || 'private');
if (!model.VISIBILITIES.includes(visibility)) return problem(res, 400, 'media.object.invalid', 'visibility must be public, unlisted or private');
if (!model.VISIBILITIES.includes(visibility)) return problem(res, 400, 'media.object.invalid', 'visibility must be public, unlisted or private');
+ // The namespace: the tenant's root unless the body names one below it; the caller must hold write there.
+ const named = namespaces.resolveName(req.appRow, b.namespace);
+ if (named.error) return problem(res, 400, 'media.namespace.invalid', named.error);
+ const namespace = named.namespace;
+ const g = namespaceGrant(req, 'write', namespace);
+ if (!g.allowed) return problem(res, 403, g.code, g.reason);
+ const policy = namespaces.effectivePolicy(req.appId, namespace);
+ if (Array.isArray(policy.kinds) && !policy.kinds.includes(kind)) return problem(res, 422, 'media.namespace.policy_denied', `${namespace} takes kinds ${policy.kinds.join(', ') || '(none)'}`);
+ if (Array.isArray(policy.visibilities) && !policy.visibilities.includes(visibility)) return problem(res, 422, 'media.namespace.policy_denied', `${namespace} takes visibilities ${policy.visibilities.join(', ') || '(none)'}`);
let size = 0;
let size = 0;
if (b.size_bytes != null) {
if (b.size_bytes != null) {
size = Number(b.size_bytes);
size = Number(b.size_bytes);
@@ -285,7 +323,6 @@ router.post('/', upload, (req, res) => {
@@ -285,7 +323,6 @@ router.post('/', upload, (req, res) => {
}
}
const subject = subjectFrom(req);
const subject = subjectFrom(req);
if (subject === undefined) return problem(res, 400, 'media.object.invalid', 'X-OV-Subject must be a user subject (usr_…)');
if (subject === undefined) return problem(res, 400, 'media.object.invalid', 'X-OV-Subject must be a user subject (usr_…)');
- if (!quotaCheck(req, res, size)) return;
const mime = b.mime_type && /^[\w.+-]+\/[\w.+-]+$/.test(String(b.mime_type)) ? String(b.mime_type).toLowerCase() : null;
const mime = b.mime_type && /^[\w.+-]+\/[\w.+-]+$/.test(String(b.mime_type)) ? String(b.mime_type).toLowerCase() : null;
if (mime) {
if (mime) {
@@ -300,15 +337,28 @@ router.post('/', upload, (req, res) => {
@@ -300,15 +337,28 @@ router.post('/', upload, (req, res) => {
if (b.content_hash) metadata.expected_sha256 = String(b.content_hash).toLowerCase();
if (b.content_hash) metadata.expected_sha256 = String(b.content_hash).toLowerCase();
if (req.principal) metadata.created_by = req.principal.sub;
if (req.principal) metadata.created_by = req.principal.sub;
- const id = model.createObject({
- app_id: req.appId, namespace: req.appId, kind, owner_subject: subject, owner_app: req.appId, owner_user_id: userId,
- visibility, lifecycle_status: 'uploading', mime_type: mime, size_bytes: size, metadata,
- });
+ // The namespace row, the quota check, the object and its reservation commit together, so two
+ // inits racing for the last of a quota cannot both get it.
+ const made = db.getDb().transaction(() => {
+ const row = namespaces.ensure(req.appRow, namespace);
+ if (row.error) return { problem: { status: row.status, code: row.code, detail: row.error } };
+ const q = namespaces.checkQuota(req.appRow, namespace, { bytes: size, objects: 1 });
+ if (q) return { problem: q };
+ const newId = model.createObject({
+ app_id: req.appId, namespace, kind, owner_subject: subject, owner_app: req.appId, owner_user_id: userId,
+ visibility, lifecycle_status: 'uploading', mime_type: mime, size_bytes: size, metadata,
+ });
+ namespaces.reserve(model.getObject(newId), size);
+ return { id: newId };
+ })();
+ if (made.problem) return problem(res, made.problem.status, made.problem.code, made.problem.detail, made.problem.extra);
+ const id = made.id;
const obj = model.getObject(id);
const obj = model.getObject(id);
let session = null;
let session = null;
if (wantParts) {
if (wantParts) {
session = multipart.initiate(obj, { partSize: b.part_size });
session = multipart.initiate(obj, { partSize: b.part_size });
if (session.error) {
if (session.error) {
+ namespaces.settle(id);
db.run('DELETE FROM media_objects WHERE id = ?', [id]); // nothing was stored for it
db.run('DELETE FROM media_objects WHERE id = ?', [id]); // nothing was stored for it
return problem(res, session.status, session.code, session.error);
return problem(res, session.status, session.code, session.error);
}
}
@@ -346,22 +396,25 @@ function completeUpload(req, res, obj, { expectedHash = null } = {}) {
@@ -346,22 +396,25 @@ function completeUpload(req, res, obj, { expectedHash = null } = {}) {
if (invariant.wouldViolate(obj)) {
if (invariant.wouldViolate(obj)) {
return problem(res, 422, 'media.invariant.public_object_too_large', `Public playback objects are limited to ${config.objects.publicMaxMb} MB`);
return problem(res, 422, 'media.invariant.public_object_too_large', `Public playback objects are limited to ${config.objects.publicMaxMb} MB`);
}
}
- if (!quotaCheck(req, res, Number(obj.size_bytes) || 0, obj.id)) return;
+ // Reconciled against the bytes actually stored: the object's real size, everything else as it is now.
+ if (!quotaCheck(req, res, obj.namespace, { bytes: Number(obj.size_bytes) || 0, objects: namespaces.reservation(obj.id) ? 0 : 1, excludeId: obj.id })) return;
const typeWhy = ctypes.kindProblem(obj.kind, obj.mime_type) || ctypes.contentProblem(obj.kind, obj.mime_type, ctypes.readHead(loc.key));
const typeWhy = ctypes.kindProblem(obj.kind, obj.mime_type) || ctypes.contentProblem(obj.kind, obj.mime_type, ctypes.readHead(loc.key));
if (typeWhy) return problem(res, 415, 'media.object.content_mismatch', typeWhy);
if (typeWhy) return problem(res, 415, 'media.object.content_mismatch', typeWhy);
const { data: body } = announce(req.appId, 'media.object.uploaded', {
const { data: body } = announce(req.appId, 'media.object.uploaded', {
change: () => {
change: () => {
model.updateObject(obj.id, { lifecycle_status: 'ready' });
model.updateObject(obj.id, { lifecycle_status: 'ready' });
+ namespaces.settle(obj.id); // the ready object counts from now on
invariant.record(model.getObject(obj.id));
invariant.record(model.getObject(obj.id));
},
},
payload: () => model.objectPublic(model.getObject(obj.id)),
payload: () => model.objectPublic(model.getObject(obj.id)),
});
});
+ namespaces.reconcileChain(req.appRow, obj.namespace);
return res.json(body);
return res.json(body);
}
}
/** The object named in the URL, writable by this caller, native and still uploading (else answered). */
/** The object named in the URL, writable by this caller, native and still uploading (else answered). */
function loadUploading(req, res) {
function loadUploading(req, res) {
- const obj = load(req, res);
+ const obj = load(req, res, 'write');
if (!obj) return null;
if (!obj) return null;
if (!canWrite(req, obj)) { problem(res, 403, 'media.object.forbidden', 'Not your object'); return null; }
if (!canWrite(req, obj)) { problem(res, 403, 'media.object.forbidden', 'Not your object'); return null; }
if (obj.legacy_ref) { problem(res, 409, 'media.object.legacy_managed', 'This object is written through the v1 API'); return null; }
if (obj.legacy_ref) { problem(res, 409, 'media.object.legacy_managed', 'This object is written through the v1 API'); return null; }
@@ -371,7 +424,7 @@ function loadUploading(req, res) {
@@ -371,7 +424,7 @@ function loadUploading(req, res) {
router.post('/:id/complete', contentAuth, tenantCors, (req, res) => {
router.post('/:id/complete', contentAuth, tenantCors, (req, res) => {
try {
try {
- const obj = load(req, res);
+ const obj = load(req, res, 'write');
if (!obj) return;
if (!obj) return;
if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');
if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');
if (obj.lifecycle_status === 'ready' && !obj.legacy_ref) return res.json(model.objectPublic(obj));
if (obj.lifecycle_status === 'ready' && !obj.legacy_ref) return res.json(model.objectPublic(obj));
@@ -391,19 +444,22 @@ router.post('/:id/upload-url', upload, (req, res) => {
@@ -391,19 +444,22 @@ router.post('/:id/upload-url', upload, (req, res) => {
const ttl = req.body && req.body.ttl != null ? Number(req.body.ttl) : (req.query.ttl != null ? Number(req.query.ttl) : undefined);
const ttl = req.body && req.body.ttl != null ? Number(req.body.ttl) : (req.query.ttl != null ? Number(req.query.ttl) : undefined);
if (ttl !== undefined && !(ttl >= 60 && ttl <= 86400)) return problem(res, 400, 'media.object.invalid', 'ttl must be 60-86400 seconds');
if (ttl !== undefined && !(ttl >= 60 && ttl <= 86400)) return problem(res, 400, 'media.object.invalid', 'ttl must be 60-86400 seconds');
if (Number(obj.size_bytes) > config.objects.maxUploadMb * MB) return problem(res, 413, 'media.object.too_large', 'This object is uploaded in parts: POST /multipart');
if (Number(obj.size_bytes) > config.objects.maxUploadMb * MB) return problem(res, 413, 'media.object.too_large', 'This object is uploaded in parts: POST /multipart');
+ // A fresh URL is a fresh attempt: the upload holds its declared size again (after an abort released it).
+ const stored = Number((db.get("SELECT size_bytes FROM media_locations WHERE object_id = ? AND provider = 'local' AND state = 'present'", [obj.id]) || {}).size_bytes) || 0;
+ if (!reserveAgain(req, res, obj, Math.max(Number(obj.size_bytes) || 0, stored))) return;
res.json(presignedPut(req, obj, ttl));
res.json(presignedPut(req, obj, ttl));
});
});
// ── Multipart uploads ────────────────────────────────────────
// ── Multipart uploads ────────────────────────────────────────
-const mpRead = multipartAuth('media.object.read');
-const mpWrite = multipartAuth('media.object.upload');
+const mpRead = multipartAuth('read');
+const mpWrite = multipartAuth('write');
/** The session named in the URL, belonging to the object, still open (else answered). */
/** The session named in the URL, belonging to the object, still open (else answered). */
function loadSession(req, res, obj, { open = true } = {}) {
function loadSession(req, res, obj, { open = true } = {}) {
const session = multipart.getSession(String(req.params.uploadId || ''));
const session = multipart.getSession(String(req.params.uploadId || ''));
if (!session || session.object_id !== obj.id) { problem(res, 404, 'media.upload.not_found', 'No such upload for this object'); return null; }
if (!session || session.object_id !== obj.id) { problem(res, 404, 'media.upload.not_found', 'No such upload for this object'); return null; }
- if (open && session.status === 'active' && multipart.isExpired(session)) multipart.abort(session.id, 'expired');
+ if (open && session.status === 'active' && multipart.isExpired(session)) multipart.abort(session.id, 'expired'); // releases its reservation
const now = multipart.getSession(session.id);
const now = multipart.getSession(session.id);
if (open && now.status !== 'active') { problem(res, 409, 'media.upload.not_active', `The upload is ${now.status}`); return null; }
if (open && now.status !== 'active') { problem(res, 409, 'media.upload.not_active', `The upload is ${now.status}`); return null; }
return now;
return now;
@@ -423,11 +479,13 @@ router.post('/:id/multipart', upload, (req, res) => {
@@ -423,11 +479,13 @@ router.post('/:id/multipart', upload, (req, res) => {
if (invariant.wouldViolate({ kind: obj.kind, visibility: obj.visibility, size_bytes: size })) {
if (invariant.wouldViolate({ kind: obj.kind, visibility: obj.visibility, size_bytes: size })) {
return problem(res, 422, 'media.invariant.public_object_too_large', `Public playback objects are limited to ${config.objects.publicMaxMb} MB`);
return problem(res, 422, 'media.invariant.public_object_too_large', `Public playback objects are limited to ${config.objects.publicMaxMb} MB`);
}
}
- if (!quotaCheck(req, res, size, obj.id)) return;
+ if (!reserveAgain(req, res, obj, size)) return;
model.updateObject(obj.id, { size_bytes: size });
model.updateObject(obj.id, { size_bytes: size });
current = model.getObject(obj.id);
current = model.getObject(obj.id);
} else if (b.size_bytes != null && Number(b.size_bytes) !== Number(obj.size_bytes)) {
} else if (b.size_bytes != null && Number(b.size_bytes) !== Number(obj.size_bytes)) {
return problem(res, 400, 'media.object.size_mismatch', `The object declares ${obj.size_bytes} bytes`);
return problem(res, 400, 'media.object.size_mismatch', `The object declares ${obj.size_bytes} bytes`);
+ } else if (!reserveAgain(req, res, obj, Number(obj.size_bytes))) {
+ return; // a session after an abort holds the declared size again
}
}
const session = multipart.initiate(current, { partSize: b.part_size });
const session = multipart.initiate(current, { partSize: b.part_size });
if (session.error) return problem(res, session.status, session.code, session.error);
if (session.error) return problem(res, session.status, session.code, session.error);
@@ -439,7 +497,7 @@ router.post('/:id/multipart', upload, (req, res) => {
@@ -439,7 +497,7 @@ router.post('/:id/multipart', upload, (req, res) => {
});
});
router.get('/:id/multipart/:uploadId', mpRead, (req, res) => {
router.get('/:id/multipart/:uploadId', mpRead, (req, res) => {
- const obj = load(req, res);
+ const obj = load(req, res, 'read');
if (!obj) return;
if (!obj) return;
const session = loadSession(req, res, obj, { open: false });
const session = loadSession(req, res, obj, { open: false });
if (!session) return;
if (!session) return;
@@ -485,21 +543,44 @@ router.post('/:id/multipart/:uploadId/complete', mpWrite, tenantCors, async (req
@@ -485,21 +543,44 @@ router.post('/:id/multipart/:uploadId/complete', mpWrite, tenantCors, async (req
});
});
router.delete('/:id/multipart/:uploadId', mpWrite, (req, res) => {
router.delete('/:id/multipart/:uploadId', mpWrite, (req, res) => {
- const obj = load(req, res);
+ const obj = load(req, res, 'write');
if (!obj) return;
if (!obj) return;
if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');
if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');
const session = loadSession(req, res, obj, { open: false });
const session = loadSession(req, res, obj, { open: false });
if (!session) return;
if (!session) return;
if (session.status === 'completed') return problem(res, 409, 'media.upload.not_active', 'The upload is completed');
if (session.status === 'completed') return problem(res, 409, 'media.upload.not_active', 'The upload is completed');
- multipart.abort(session.id);
+ multipart.abort(session.id); // parts deleted; the quota it held is released
res.json(multipart.sessionPublic(multipart.getSession(session.id), { parts: false }));
res.json(multipart.sessionPublic(multipart.getSession(session.id), { parts: false }));
});
});
-router.get('/', read, (req, res) => {
+router.get('/', list, (req, res) => {
try {
try {
const q = req.query;
const q = req.query;
const limit = Math.min(Math.max(parseInt(q.limit || '50', 10) || 50, 1), 200);
const limit = Math.min(Math.max(parseInt(q.limit || '50', 10) || 50, 1), 200);
const conds = ['app_id = ?'], params = [req.appId];
const conds = ['app_id = ?'], params = [req.appId];
+ // ?namespace= narrows the list to one namespace and those below it.
+ if (q.namespace != null && q.namespace !== '') {
+ const named = namespaces.resolveName(req.appRow, q.namespace);
+ if (named.error) return problem(res, 400, 'media.namespace.invalid', named.error);
+ const g = namespaceGrant(req, 'list', named.namespace);
+ if (!g.allowed) return problem(res, 403, g.code, g.reason);
+ conds.push('(namespace = ? OR substr(namespace, 1, ?) = ?)');
+ params.push(named.namespace, named.namespace.length + 1, `${named.namespace}.`);
+ }
+ // A Network token lists only the namespaces it may list here.
+ if (req.grant) {
+ const rows = namespaces.listForTenant(req.appId);
+ const listable = rows.filter(r => namespaceGrant(req, 'list', r.namespace).allowed).map(r => r.namespace);
+ if (!listable.length) {
+ // Nothing listable: refused when the verb itself is (a strict namespace), else an empty page.
+ const g = namespaceGrant(req, 'list', db.rootNamespace(req.appRow));
+ if (!g.allowed && g.code === 'capability.denied') return problem(res, 403, g.code, g.reason);
+ conds.push('0');
+ } else if (listable.length < rows.length) {
+ conds.push(`namespace IN (${listable.map(() => '?').join(', ')})`);
+ params.push(...listable);
+ }
+ }
if (q.cursor) {
if (q.cursor) {
if (!/^med_[0-9A-HJKMNP-TV-Z]{26}$/.test(String(q.cursor))) return problem(res, 400, 'media.object.invalid', 'Bad cursor');
if (!/^med_[0-9A-HJKMNP-TV-Z]{26}$/.test(String(q.cursor))) return problem(res, 400, 'media.object.invalid', 'Bad cursor');
conds.push('id < ?'); params.push(String(q.cursor));
conds.push('id < ?'); params.push(String(q.cursor));
@@ -521,19 +602,21 @@ router.get('/', read, (req, res) => {
@@ -521,19 +602,21 @@ router.get('/', read, (req, res) => {
});
});
router.get('/:id', read, (req, res) => {
router.get('/:id', read, (req, res) => {
- const obj = load(req, res);
+ const obj = load(req, res, 'read');
if (obj) res.json(model.objectPublic(obj));
if (obj) res.json(model.objectPublic(obj));
});
});
-router.delete('/:id', upload, (req, res) => {
+router.delete('/:id', remove, (req, res) => {
try {
try {
- const obj = load(req, res);
+ const obj = load(req, res, 'delete');
if (!obj) return;
if (!obj) return;
if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');
if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');
if (obj.legacy_ref) return problem(res, 409, 'media.object.legacy_managed', 'Delete this object through its v1 route (vods, clips, files, pastes)');
if (obj.legacy_ref) return problem(res, 409, 'media.object.legacy_managed', 'Delete this object through its v1 route (vods, clips, files, pastes)');
if (model.isHeld(obj.id)) return problem(res, 409, 'media.object.held', 'Object is under a retention hold');
if (model.isHeld(obj.id)) return problem(res, 409, 'media.object.held', 'Object is under a retention hold');
if (obj.lifecycle_status === 'deleted') return res.json(model.objectPublic(obj));
if (obj.lifecycle_status === 'deleted') return res.json(model.objectPublic(obj));
- res.json(model.objectPublic(model.softDelete(obj, { by: req.principal ? req.principal.sub : `app:${req.appId}` })));
+ const gone = model.softDelete(obj, { by: req.principal ? req.principal.sub : `app:${req.appId}` });
+ namespaces.reconcileChain(req.appRow, obj.namespace);
+ res.json(model.objectPublic(gone));
} catch (err) {
} catch (err) {
if (err.code === 'media.object.held' || /retention hold/.test(err.message)) return problem(res, 409, 'media.object.held', 'Object is under a retention hold');
if (err.code === 'media.object.held' || /retention hold/.test(err.message)) return problem(res, 409, 'media.object.held', 'Object is under a retention hold');
console.error('[Objects] delete error:', err.message);
console.error('[Objects] delete error:', err.message);
@@ -541,18 +624,22 @@ router.delete('/:id', upload, (req, res) => {
@@ -541,18 +624,22 @@ router.delete('/:id', upload, (req, res) => {
}
}
});
});
-router.post('/:id/restore', upload, (req, res) => {
- const obj = load(req, res);
+router.post('/:id/restore', remove, (req, res) => {
+ const obj = load(req, res, 'delete');
if (!obj) return;
if (!obj) return;
if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');
if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');
if (obj.lifecycle_status !== 'deleted' || obj.legacy_ref) return problem(res, 409, 'media.object.not_deleted', 'Only soft-deleted native objects can be restored');
if (obj.lifecycle_status !== 'deleted' || obj.legacy_ref) return problem(res, 409, 'media.object.not_deleted', 'Only soft-deleted native objects can be restored');
+ // A restored object counts again: it needs the room (in a developer project it never stopped counting).
+ const md = model.parseJson(obj.metadata, {});
+ if (md.pre_delete_status !== 'uploading' && !md.purged_at && !quotaCheck(req, res, obj.namespace, { bytes: Number(obj.size_bytes) || 0, objects: 1, excludeId: obj.id })) return;
const back = model.restore(obj);
const back = model.restore(obj);
if (!back) return problem(res, 410, 'media.object.purged', 'The retention period has passed and the bytes are gone');
if (!back) return problem(res, 410, 'media.object.purged', 'The retention period has passed and the bytes are gone');
+ namespaces.reconcileChain(req.appRow, obj.namespace);
res.json(model.objectPublic(back));
res.json(model.objectPublic(back));
});
});
router.get('/:id/download', read, (req, res) => {
router.get('/:id/download', read, (req, res) => {
- const obj = load(req, res);
+ const obj = load(req, res, 'read');
if (!obj) return;
if (!obj) return;
if (obj.lifecycle_status === 'deleted') return problem(res, 410, 'media.object.deleted', 'Object was deleted');
if (obj.lifecycle_status === 'deleted') return problem(res, 410, 'media.object.deleted', 'Object was deleted');
if (obj.lifecycle_status !== 'ready') return problem(res, 409, 'media.object.not_ready', `Object is ${obj.lifecycle_status}`);
if (obj.lifecycle_status !== 'ready') return problem(res, 409, 'media.object.not_ready', `Object is ${obj.lifecycle_status}`);
@@ -571,7 +658,7 @@ router.get('/:id/download', read, (req, res) => {
@@ -571,7 +658,7 @@ router.get('/:id/download', read, (req, res) => {
// ── Retention holds ──────────────────────────────────────────
// ── Retention holds ──────────────────────────────────────────
router.get('/:id/holds', read, (req, res) => {
router.get('/:id/holds', read, (req, res) => {
- const obj = load(req, res);
+ const obj = load(req, res, 'read');
if (!obj) return;
if (!obj) return;
res.json({
res.json({
object_id: obj.id,
object_id: obj.id,
@@ -654,7 +741,7 @@ publicRouter.get('/:id', async (req, res) => {
@@ -654,7 +741,7 @@ publicRouter.get('/:id', async (req, res) => {
/** PUT /:id/multipart/:uploadId/parts/:n (raw body; mounted ahead of the JSON body parser). */
/** PUT /:id/multipart/:uploadId/parts/:n (raw body; mounted ahead of the JSON body parser). */
async function putPart(req, res) {
async function putPart(req, res) {
- const obj = load(req, res);
+ const obj = load(req, res, 'write');
if (!obj) { req.resume(); return; }
if (!obj) { req.resume(); return; }
if (!canWrite(req, obj)) { req.resume(); return problem(res, 403, 'media.object.forbidden', 'Not your object'); }
if (!canWrite(req, obj)) { req.resume(); return problem(res, 403, 'media.object.forbidden', 'Not your object'); }
if (obj.lifecycle_status !== 'uploading') { req.resume(); return problem(res, 409, 'media.object.not_uploading', `Object is ${obj.lifecycle_status}`); }
if (obj.lifecycle_status !== 'uploading') { req.resume(); return problem(res, 409, 'media.object.not_uploading', `Object is ${obj.lifecycle_status}`); }
new file mode 100644
new file mode 100644
@@ -0,0 +1,373 @@
@@ -0,0 +1,373 @@
+'use strict';
+// Namespaces and grants (roadmap WS-G task 2; server/objects/namespaces.js, server/auth.js VERBS):
+// - the backfill on open: developer-project objects move to app.<project_id>[.sandbox], every tenant
+// and every namespace an object names gets a row, uploads in progress get a reservation (once);
+// - five verbs per namespace (read, list, write, delete, transform), the older ids still granting the
+// newer verbs, strict_verbs refusing them; child namespaces and grants that name only a child;
+// - developer app tokens: their legacy ns [prj_…] means app.<project_id>.*, the new ns works the same;
+// - quotas per namespace (bytes and objects, up to the tenant), reserved at init, re-reserved with the
+// bytes stored, settled at complete, released at abort, swept at expiry; the policy's kinds,
+// visibilities and max_object_bytes; the owner's read of namespaces and usage.
+const assert = require('assert');
+const fs = require('fs');
+const os = require('os');
+const path = require('path');
+const http = require('http');
+const crypto = require('crypto');
+const Database = require('better-sqlite3');
+
+const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-namespaces-'));
+const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+process.env.DB_PATH = path.join(tmp, 'media.db');
+process.env.VOD_PATH = dir('vods');
+process.env.FILES_PATH = dir('files');
+process.env.OBJECTS_PATH = dir('objects');
+process.env.THUMBNAILS_PATH = dir('thumbnails');
+process.env.OV_NETWORK_URL = 'https://openvibe.network';
+process.env.MEDIA_PUBLIC_URL = 'https://media.test';
+process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';
+process.env.MEDIA_NAMESPACE_MAX_CHILDREN = '6';
+process.env.MEDIA_MULTIPART_MIN_PART_MB = '1';
+process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';
+
+const { serviceAuth, ids } = require('openvibe-contracts');
+const PA = `prj_${ids.ulid()}`;
+const PB = `prj_${ids.ulid()}`;
+const APP_A = `app_${ids.ulid()}`;
+
+// ── A database from before namespaces were rows ──
+{
+ const h = new Database(process.env.DB_PATH);
+ h.exec(fs.readFileSync(path.join(__dirname, '..', 'server', 'db', 'schema.sql'), 'utf8'));
+ h.exec('ALTER TABLE apps ADD COLUMN project_id TEXT; ALTER TABLE apps ADD COLUMN env TEXT;');
+ h.prepare("INSERT INTO apps (app_id, name, api_key_hash, quota_bytes) VALUES ('live', 'Live', ?, 0)").run(crypto.createHash('sha256').update('live-key').digest('hex'));
+ h.prepare("INSERT INTO apps (app_id, name, api_key_hash, quota_bytes, project_id, env) VALUES (?, 'p', '', 1000000, ?, 'production')").run(PA, PA);
+ h.prepare("INSERT INTO apps (app_id, name, api_key_hash, quota_bytes, project_id, env) VALUES (?, 'p', '', 1000000, ?, 'sandbox')").run(`${PA}-sandbox`, PA);
+ const put = h.prepare("INSERT INTO media_objects (id, app_id, namespace, kind, lifecycle_status, size_bytes) VALUES (?, ?, ?, 'file', ?, ?)");
+ put.run(`med_${ids.ulid()}`, PA, PA, 'ready', 10);
+ put.run(`med_${ids.ulid()}`, `${PA}-sandbox`, `${PA}-sandbox`, 'ready', 20);
+ put.run(`med_${ids.ulid()}`, 'live', 'live.legacy.old', 'ready', 30);
+ put.run('med_01M0PRE0UPLOAD0000000000AA', 'live', 'live', 'uploading', 100);
+ h.close();
+}
+
+const db = require('../server/db/database');
+const auth = require('../server/auth');
+const namespaces = require('../server/objects/namespaces');
+const model = require('../server/objects/model');
+const objectRoutes = require('../server/objects/routes');
+const express = require('express');
+
+const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });
+auth._setNetworkPublicKeyForTests(keys.publicKey);
+const now = Math.floor(Date.now() / 1000);
+const jti = () => `tok_${crypto.randomBytes(6).toString('hex')}`;
+const svc = (cap, ns = ['community']) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: 'svc:community', actor_type: 'service', aud: ['openvibe.media'], cap, ns, iat: now, exp: now + 300, jti: jti() }, keys.privateKey);
+const appTok = ({ cap = ['media.object.upload', 'media.object.read'], ns = [PA], env = 'production', project = PA } = {}) => serviceAuth.signServiceToken({
+ iss: 'https://openvibe.network', sub: `app:${APP_A}`, actor_type: 'app', aud: ['openvibe.media'], cap, ns, project_id: project, env, iat: now, exp: now + 300, jti: jti(),
+}, keys.privateKey);
+
+// ── 1. The backfill on open ──
+{
+ db.getDb();
+ const nsOf = (app) => db.all('SELECT namespace FROM media_objects WHERE app_id = ? ORDER BY namespace', [app]).map(r => r.namespace);
+ assert.deepStrictEqual(nsOf(PA), [`app.${PA}`], 'a project object moves to app.<project_id>');
+ assert.deepStrictEqual(nsOf(`${PA}-sandbox`), [`app.${PA}.sandbox`], 'its sandbox object to app.<project_id>.sandbox');
+ const rows = Object.fromEntries(db.all('SELECT * FROM media_namespaces').map(r => [r.namespace, r]));
+ assert.deepStrictEqual([rows.live.owner, rows.live.parent, rows[`app.${PA}`].owner, rows[`app.${PA}.sandbox`].app_id], ['service:live', null, `project:${PA}`, `${PA}-sandbox`]);
+ assert.deepStrictEqual([rows['live.legacy'].parent, rows['live.legacy.old'].parent], ['live', 'live.legacy'], 'a child an object names gets a row, with the chain up to its root');
+ const pre = namespaces.reservation('med_01M0PRE0UPLOAD0000000000AA');
+ assert.ok(pre && pre.bytes === 100 && pre.namespace === 'live', 'an upload already in progress holds its declared size');
+ assert.ok(Date.parse(pre.expires_at.replace(' ', 'T') + 'Z') > Date.now() + 70 * 3600e3, 'for the full reservation window from now');
+ // Idempotent, and the reservation seed runs once: a released reservation does not come back.
+ namespaces.settle('med_01M0PRE0UPLOAD0000000000AA');
+ db.close();
+ db.getDb();
+ assert.strictEqual(namespaces.reservation('med_01M0PRE0UPLOAD0000000000AA'), undefined, 'the seed ran once');
+ assert.deepStrictEqual(nsOf(PA), [`app.${PA}`]);
+ assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_namespaces').n, Object.keys(rows).length, 'no duplicate rows on a second open');
+ db.run("UPDATE media_objects SET lifecycle_status = 'failed' WHERE id = 'med_01M0PRE0UPLOAD0000000000AA'");
+ console.log('✅ backfill: project objects under app.<project_id>, a row per namespace, reservations seeded once');
+}
+
+db.upsertApp({ app_id: 'tiny', api_key: 'tiny-key', quota_bytes: 100 });
+auth.ensureTokenOnlyApps();
+assert.ok(namespaces.get('tiny') && namespaces.get('community'), 'new tenants get their root row');
+
+const app = express();
+app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);
+app.put('/api/v2/:app/objects/:id/multipart/:uploadId/parts/:n', ...objectRoutes.partHandlers);
+app.use(express.json());
+app.use('/api/v1/:app/files', require('../server/files/routes'));
+app.use('/api/v2/:app/objects', objectRoutes);
+app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));
+app.use('/api/v2/:app/namespaces', require('../server/objects/namespace-routes'));
+const server = http.createServer(app);
+
+(async () => {
+ await new Promise((r) => server.listen(0, '127.0.0.1', r));
+ const base = `http://127.0.0.1:${server.address().port}`;
+ const local = (u) => base + String(u).replace('https://media.test', '');
+ const call = async (method, p, { bearer = 'live-key', body, raw, headers = {} } = {}) => {
+ const h = { ...headers };
+ if (bearer) h.authorization = `Bearer ${bearer}`;
+ let payload;
+ if (raw !== undefined) payload = raw;
+ else if (body !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }
+ const res = await fetch(p.startsWith('http') ? p : base + p, { method, headers: h, body: payload, redirect: 'manual' });
+ const text = await res.text();
+ let json = null; try { json = JSON.parse(text); } catch { /* not json */ }
+ return { status: res.status, body: json, text };
+ };
+ const C = '/api/v2/community/objects';
+ // init → PUT → complete; returns the ready object (or the failing answer).
+ const store = async (url, bearer, bytes, extra = {}) => {
+ let r = await call('POST', url, { bearer, body: { kind: 'file', mime_type: 'application/octet-stream', size_bytes: bytes.length, ...extra } });
+ if (r.status !== 201) return r;
+ const id = r.body.id;
+ r = await call('PUT', local(r.body.upload.url), { bearer: null, raw: bytes, headers: { 'content-type': 'application/octet-stream' } });
+ if (r.status !== 200) return r;
+ return call('POST', `${url}/${id}/complete`, { bearer, body: {} });
+ };
+
+ // ── 2. Five verbs; the older ids keep granting the newer ones ──
+ const UP = ['media.object.upload'];
+ let r = await store(C, svc([...UP, 'media.object.read']), Buffer.from('root object'));
+ assert.strictEqual(r.status, 200, JSON.stringify(r.body));
+ const rootObj = r.body;
+ assert.strictEqual(rootObj.namespace, 'community', 'the root namespace by default');
+ r = await call('GET', C, { bearer: svc(['media.object.read']) });
+ assert.strictEqual(r.status, 200, 'read still lists (tokens from before the split)');
+ r = await call('GET', C, { bearer: svc(['media.object.list']) });
+ assert.deepStrictEqual([r.status, r.body.objects.map(o => o.id)], [200, [rootObj.id]], 'media.object.list lists');
+ r = await call('GET', `${C}/${rootObj.id}`, { bearer: svc(['media.object.list']) });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'list is not read');
+ r = await call('POST', C, { bearer: svc(['media.object.delete']), body: { kind: 'file' } });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'delete is not write');
+ r = await call('DELETE', `${C}/${rootObj.id}`, { bearer: svc(['media.object.read']) });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'read does not delete');
+ r = await call('DELETE', `${C}/${rootObj.id}`, { bearer: svc(['media.object.delete']) });
+ assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'deleted'], 'media.object.delete deletes');
+ r = await call('POST', `${C}/${rootObj.id}/restore`, { bearer: svc(UP) });
+ assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'ready'], 'upload still restores (and deletes)');
+ r = await call('POST', '/api/v2/community/jobs', { bearer: svc(['media.object.read']), body: { type: 'invariant.scan' } });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'transform needs its grant');
+ r = await call('POST', '/api/v2/community/jobs', { bearer: svc(['media.derivative.create']), body: { type: 'invariant.scan' } });
+ assert.strictEqual(r.status, 202, `media.derivative.create transforms: ${JSON.stringify(r.body)}`);
+ const job = r.body.job;
+ r = await call('POST', `/api/v2/community/jobs/${job.id}/cancel`, { bearer: svc(UP) });
+ assert.strictEqual(r.status, 200, 'upload still transforms');
+ r = await call('GET', '/api/v2/community/jobs', { bearer: svc(['media.object.list']) });
+ assert.strictEqual(r.status, 200, 'jobs are listed with list');
+ r = await call('GET', `/api/v1/community/files`, { bearer: svc(['media.object.list']) });
+ assert.strictEqual(r.status, 200, 'v1 files list with list');
+ r = await call('POST', C, { bearer: svc(UP, ['live']), body: {} });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'another tenant\'s namespace');
+
+ // strict_verbs: only the verb's own id counts in that namespace.
+ db.run(`UPDATE media_namespaces SET policy = '{"strict_verbs":true}' WHERE namespace = 'community'`);
+ r = await call('GET', C, { bearer: svc(['media.object.read']) });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'strict: read no longer lists');
+ r = await call('DELETE', `${C}/${rootObj.id}`, { bearer: svc(UP) });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'strict: upload no longer deletes');
+ r = await call('GET', C, { bearer: svc(['media.object.list']) });
+ assert.strictEqual(r.status, 200);
+ db.run("UPDATE media_namespaces SET policy = '{}' WHERE namespace = 'community'");
+ console.log('✅ verbs: read, list, write, delete, transform; older ids still grant; strict_verbs refuses them');
+
+ // ── 3. Child namespaces; grants that name only a child ──
+ r = await store(C, svc(UP, ['community.*']), Buffer.from('an avatar'), { namespace: 'avatars' });
+ assert.deepStrictEqual([r.status, r.body.namespace], [200, 'community.avatars'], 'a relative name lands below the root');
+ const avatar = r.body;
+ assert.strictEqual(namespaces.get('community.avatars').parent, 'community');
+ r = await call('POST', C, { bearer: svc(UP), body: { namespace: 'avatars' } });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'a grant of the root alone is not its children');
+ for (const bad of ['Avatars', 'a..b', 'a.b.c.d', '../x', 'live', '-x']) {
+ r = await call('POST', C, { bearer: svc(UP, ['community.*']), body: { namespace: bad } });
+ if (bad === 'live') { assert.deepStrictEqual([r.status, r.body.namespace === undefined], [201, true]); continue; } // relative: community.live
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.namespace.invalid'], bad);
+ }
+ assert.ok(namespaces.get('community.live'), '"live" is relative to the tenant');
+ const onlyAvatars = (cap) => svc(cap, ['community.avatars']);
+ r = await call('POST', C, { bearer: onlyAvatars(UP), body: {} });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'a child grant does not write the root');
+ r = await store(C, onlyAvatars(UP), Buffer.from('another avatar'), { namespace: 'community.avatars' });
+ assert.deepStrictEqual([r.status, r.body.namespace], [200, 'community.avatars'], 'the full name works too');
+ r = await call('GET', C, { bearer: onlyAvatars(['media.object.list']) });
+ assert.ok(r.body.objects.length === 2 && r.body.objects.every(o => o.namespace === 'community.avatars'), 'a child grant lists only its namespace');
+ r = await call('GET', `${C}/${rootObj.id}`, { bearer: onlyAvatars(['media.object.read']) });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'nor reads the root');
+ r = await call('GET', `${C}?namespace=avatars`, { bearer: svc(['media.object.list'], ['community.*']) });
+ assert.ok(r.body.objects.length === 2 && r.body.objects.every(o => o.namespace === 'community.avatars'), '?namespace= narrows the list');
+ r = await call('GET', `${C}?namespace=community`, { bearer: onlyAvatars(['media.object.list']) });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied']);
+ r = await call('POST', '/api/v2/community/jobs', { bearer: onlyAvatars(['media.derivative.create']), body: { type: 'object.hash', object_id: avatar.id } });
+ assert.ok(r.status === 202 || r.status === 400, `transform on an object in the granted child: ${JSON.stringify(r.body)}`);
+ assert.notStrictEqual(r.body.code, 'capability.namespace_denied');
+ r = await call('POST', '/api/v2/community/jobs', { bearer: onlyAvatars(['media.derivative.create']), body: { type: 'invariant.scan' } });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'a tenant-wide job is the root\'s');
+ // At most MEDIA_NAMESPACE_MAX_CHILDREN (6) below a root: avatars, live, then four more.
+ for (const n of ['c1', 'c2', 'c3.d']) {
+ r = await call('POST', C, { bearer: svc(UP, ['community.*']), body: { namespace: n } });
+ assert.strictEqual(r.status, 201, `${n}: ${JSON.stringify(r.body)}`);
+ }
+ r = await call('POST', C, { bearer: svc(UP, ['community.*']), body: { namespace: 'c4' } });
+ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.namespaces_exceeded']);
+ console.log('✅ child namespaces: created by the first upload, granted and listed per namespace, bounded per tenant');
+
+ // ── 4. Developer app tokens: app.<project_id>.* ──
+ const A = `/api/v2/${PA}/objects`;
+ r = await store(A, appTok(), Buffer.from('legacy ns'), { namespace: 'uploads' });
+ assert.deepStrictEqual([r.status, r.body.namespace], [200, `app.${PA}.uploads`], 'a legacy ns [project_id] means the whole project');
+ r = await store(A, appTok({ ns: [PA, `app.${PA}.*`] }), Buffer.from('new ns'));
+ assert.deepStrictEqual([r.status, r.body.namespace], [200, `app.${PA}`], 'the ns Network issues now');
+ r = await call('POST', A, { bearer: appTok({ ns: [`app.${PA}.uploads`] }), body: {} });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'a token for one child of the project');
+ r = await call('POST', A, { bearer: appTok({ ns: [`app.${PA}.uploads`] }), body: { namespace: 'uploads' } });
+ assert.strictEqual(r.status, 201);
+ r = await call('POST', A, { bearer: appTok({ ns: [`app.${PB}.*`] }), body: {} });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied'], 'another project\'s namespace');
+ r = await call('POST', A, { bearer: appTok(), body: { namespace: 'sandbox' } });
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.namespace.invalid'], 'the sandbox tenant is not a child of production');
+ r = await store(A, appTok({ env: 'sandbox' }), Buffer.from('sandbox'), { namespace: 'uploads' });
+ assert.deepStrictEqual([r.status, r.body.namespace], [200, `app.${PA}.sandbox.uploads`], 'sandbox tokens work in the sandbox tenant');
+ r = await call('GET', `/api/v2/${PA}/namespaces`, { bearer: appTok() });
+ assert.deepStrictEqual(r.body.namespaces.map(n => n.namespace), [`app.${PA}`, `app.${PA}.uploads`], 'the owner reads its production namespaces');
+ const rootNs = r.body.namespaces[0];
+ assert.deepStrictEqual([rootNs.root, rootNs.owner, rootNs.quota.bytes, rootNs.quota.bytes_from, rootNs.usage.used_objects, rootNs.usage.used_bytes],
+ [true, `project:${PA}`, 1000000, 'tenant', 3, 10 + 9 + 6], 'root usage covers the tenant; the quota is the tenant\'s');
+ assert.strictEqual(r.body.namespaces[1].usage.reserved_objects, 1, 'the open upload in uploads is a reservation');
+ r = await call('GET', `/api/v2/${PA}/namespaces/uploads`, { bearer: appTok({ cap: ['media.object.read'] }) });
+ assert.deepStrictEqual([r.status, r.body.namespace, r.body.usage.used_bytes], [200, `app.${PA}.uploads`, 9]);
+ r = await call('GET', `/api/v2/${PA}/namespaces`, { bearer: appTok({ env: 'sandbox' }) });
+ assert.deepStrictEqual(r.body.namespaces.map(n => [n.namespace, n.sandbox]), [[`app.${PA}.sandbox`, true], [`app.${PA}.sandbox.uploads`, true]]);
+ r = await call('GET', `/api/v2/${PA}/namespaces/nope`, { bearer: appTok() });
+ assert.deepStrictEqual([r.status, r.body.code], [404, 'media.namespace.not_found']);
+ r = await call('GET', '/api/v2/community/namespaces', { bearer: onlyAvatars(['media.object.list']) });
+ assert.deepStrictEqual(r.body.namespaces.map(n => n.namespace), ['community.avatars'], 'a child grant sees its namespace only');
+ r = await call('GET', '/api/v2/community/namespaces/community', { bearer: onlyAvatars(['media.object.read']) });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied']);
+ r = await call('GET', '/api/v2/live/namespaces');
+ assert.deepStrictEqual(r.body.namespaces.map(n => n.namespace), ['live', 'live.legacy', 'live.legacy.old'], 'the app key reads every namespace of its tenant');
+ console.log('✅ developer tenants: app.<project_id>.* (legacy and new ns), sandbox apart; owner reads namespaces and usage');
+
+ // ── 5. Quotas: per namespace, reserved at init, reconciled at complete, released, swept ──
+ const T = '/api/v2/tiny/objects';
+ db.run("INSERT INTO media_namespaces (namespace, app_id, parent, owner, quota_bytes, quota_objects) VALUES ('tiny.box', 'tiny', 'tiny', 'service:tiny', 30, 2)");
+ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 31 } });
+ assert.deepStrictEqual([r.status, r.body.code, r.body.namespace, r.body.quota_bytes], [413, 'media.quota.exceeded', 'tiny.box', 30], 'the child\'s own byte quota');
+ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 20 } });
+ assert.strictEqual(r.status, 201);
+ const boxed = r.body;
+ assert.strictEqual(namespaces.reservation(boxed.id).bytes, 20, 'init reserves the declared size');
+ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 11 } });
+ assert.strictEqual(r.status, 413, 'the reservation counts');
+ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 5 } });
+ assert.strictEqual(r.status, 201);
+ const second = r.body;
+ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 1 } });
+ assert.deepStrictEqual([r.status, r.body.code, r.body.quota_objects], [413, 'media.quota.objects_exceeded', 2], 'the child\'s object quota');
+ r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 76 } });
+ assert.deepStrictEqual([r.status, r.body.namespace], [413, 'tiny'], 'the tenant quota (100) covers its children\'s reservations (25)');
+ // The bytes replace the declared size; complete settles the reservation, and the ready object counts.
+ r = await call('PUT', local(boxed.upload.url), { bearer: null, raw: Buffer.alloc(20, 1) });
+ assert.strictEqual(r.status, 200);
+ r = await call('POST', `${T}/${boxed.id}/complete`, { bearer: 'tiny-key', body: {} });
+ assert.strictEqual(r.status, 200);
+ assert.strictEqual(namespaces.reservation(boxed.id), undefined, 'settled at complete');
+ let box = namespaces.get('tiny.box');
+ assert.deepStrictEqual([box.used_bytes, box.used_objects, box.reserved_bytes, box.reserved_objects], [20, 1, 5, 1], 'the snapshot is reconciled at complete');
+ // Complete checks the real size: an undeclared upload that grew past the room is refused there.
+ r = await call('POST', T, { bearer: 'tiny-key', body: {} });
+ const grow = r.body;
+ r = await call('PUT', `${T}/${grow.id}/content`, { bearer: 'tiny-key', raw: Buffer.alloc(76, 2) });
+ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'the stored bytes are checked (20 + 5 + 76 > 100)');
+ r = await call('PUT', `${T}/${grow.id}/content`, { bearer: 'tiny-key', raw: Buffer.alloc(40, 2) });
+ assert.strictEqual(r.status, 200);
+ assert.strictEqual(namespaces.reservation(grow.id).bytes, 40, 're-reserved with the bytes stored');
+ db.run("UPDATE apps SET quota_bytes = 60 WHERE app_id = 'tiny'");
+ r = await call('POST', `${T}/${grow.id}/complete`, { bearer: 'tiny-key', body: {} });
+ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'and reconciled again at complete (20 + 5 + 40 > 60)');
+ db.run("UPDATE apps SET quota_bytes = 100 WHERE app_id = 'tiny'");
+ r = await call('POST', `${T}/${grow.id}/complete`, { bearer: 'tiny-key', body: {} });
+ assert.strictEqual(r.status, 200);
+ // Deleting an upload in progress frees its reservation.
+ r = await call('DELETE', `${T}/${second.id}`, { bearer: 'tiny-key' });
+ assert.strictEqual(r.status, 200);
+ assert.strictEqual(namespaces.reservation(second.id), undefined, 'a deleted upload holds nothing');
+ box = namespaces.get('tiny.box');
+ assert.deepStrictEqual([box.reserved_bytes, box.reserved_objects], [0, 0]);
+
+ // Multipart: the session holds the declared size; abort releases it; a new session reserves again.
+ db.run("UPDATE apps SET quota_bytes = ? WHERE app_id = 'tiny'", [3 * 1024 * 1024]);
+ db.run("UPDATE media_namespaces SET quota_bytes = NULL, quota_objects = NULL WHERE namespace = 'tiny.box'");
+ const MB = 1024 * 1024;
+ r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 2 * MB, multipart: true, part_size: MB } });
+ assert.strictEqual(r.status, 201, JSON.stringify(r.body));
+ const mp = r.body;
+ r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 2 * MB } });
+ assert.strictEqual(r.status, 413, 'the open session holds its 2 MB');
+ r = await call('DELETE', local(mp.upload.multipart.abort_url), { bearer: null });
+ assert.strictEqual(r.status, 200);
+ assert.strictEqual(namespaces.reservation(mp.id), undefined, 'abort released it');
+ r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 2 * MB, multipart: true, part_size: MB } });
+ assert.strictEqual(r.status, 201, 'the room is back');
+ const other = r.body;
+ r = await call('POST', `${T}/${mp.id}/multipart`, { bearer: 'tiny-key', body: { part_size: MB } });
+ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'a new session for the aborted upload is checked again');
+ r = await call('DELETE', local(other.upload.multipart.abort_url), { bearer: null });
+ r = await call('POST', `${T}/${mp.id}/multipart`, { bearer: 'tiny-key', body: { part_size: MB } });
+ assert.strictEqual(r.status, 201, 'and reserves again when there is room');
+ assert.strictEqual(namespaces.reservation(mp.id).bytes, 2 * MB);
+
+ // Expiry: an abandoned upload is failed and its bytes freed; an open session keeps its upload.
+ r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 3 } });
+ const abandoned = r.body;
+ await call('PUT', local(abandoned.upload.url), { bearer: null, raw: Buffer.from('abc') });
+ const file = model.objectFilePath(model.getObject(abandoned.id));
+ assert.ok(fs.existsSync(file));
+ db.run("UPDATE media_quota_reservations SET expires_at = datetime('now', '-1 hour') WHERE object_id IN (?, ?)", [abandoned.id, mp.id]);
+ const swept = namespaces.expireReservations();
+ assert.deepStrictEqual(swept, { expired: 1, dropped: 0 }, 'only the upload with no open session');
+ const gone = model.getObject(abandoned.id);
+ assert.deepStrictEqual([gone.lifecycle_status, JSON.parse(gone.metadata).failure, fs.existsSync(file), namespaces.reservation(abandoned.id)],
+ ['failed', 'upload_expired', false, undefined]);
+ assert.ok(namespaces.reservation(mp.id), 'the multipart upload keeps its reservation while its session is open');
+ r = await call('PUT', local(abandoned.upload.url), { bearer: null, raw: Buffer.from('abc') });
+ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.not_uploading'], 'an expired upload takes no more bytes');
+
+ // v1 files count against the root's quotas, objects included.
+ db.run("UPDATE media_namespaces SET quota_objects = 1 WHERE namespace = 'tiny'");
+ const fd = new FormData();
+ fd.append('file', new Blob([Buffer.from('v1 bytes')], { type: 'text/plain' }), 'x.txt');
+ const f = await fetch(`${base}/api/v1/tiny/files`, { method: 'POST', headers: { authorization: 'Bearer tiny-key' }, body: fd });
+ const fb = await f.json();
+ assert.deepStrictEqual([f.status, fb.code, fb.namespace], [413, 'media.quota.objects_exceeded', 'tiny'], 'the v1 answer keeps its shape and says why');
+ db.run("UPDATE media_namespaces SET quota_objects = NULL WHERE namespace = 'tiny'");
+
+ // Policy: kinds, visibilities and max_object_bytes, children inheriting and overriding.
+ db.run(`UPDATE media_namespaces SET policy = '{"kinds":["file","asset"],"max_object_bytes":10}' WHERE namespace = 'tiny'`);
+ db.run(`UPDATE media_namespaces SET policy = '{"visibilities":["private"],"max_object_bytes":0}' WHERE namespace = 'tiny.box'`);
+ r = await call('POST', T, { bearer: 'tiny-key', body: { kind: 'vod' } });
+ assert.deepStrictEqual([r.status, r.body.code], [422, 'media.namespace.policy_denied'], 'kinds');
+ r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 11 } });
+ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'max_object_bytes');
+ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', size_bytes: 11, visibility: 'private' } });
+ assert.strictEqual(r.status, 201, 'a child overrides max_object_bytes');
+ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', kind: 'vod', visibility: 'private' } });
+ assert.strictEqual(r.status, 422, 'and inherits kinds');
+ r = await call('POST', T, { bearer: 'tiny-key', body: { namespace: 'box', visibility: 'public' } });
+ assert.deepStrictEqual([r.status, r.body.code], [422, 'media.namespace.policy_denied'], 'visibilities');
+ assert.ok(namespaces.validatePolicy({ kinds: ['file'], strict_verbs: true }).policy);
+ assert.ok(namespaces.validatePolicy({ nope: 1 }).error && namespaces.validatePolicy({ kinds: ['x'] }).error && namespaces.validatePolicy({ max_object_bytes: -1 }).error);
+
+ // The snapshot follows the rows when reconciled.
+ db.run("UPDATE media_namespaces SET used_bytes = 999 WHERE namespace = 'tiny'");
+ assert.ok(namespaces.reconcileAll() >= 3);
+ assert.strictEqual(namespaces.get('tiny').used_bytes, model.usedBytes('tiny') - namespaces.get('tiny').reserved_bytes, 'the root\'s snapshot is the tenant\'s usage');
+ console.log('✅ quotas: per namespace and tenant, bytes and objects; reserved, re-reserved, settled, released, swept; policy enforced');
+
+ server.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('namespaces and grants: all checks passed');
+})().catch((err) => { console.error(err); process.exit(1); });