openvibers/openvibe.media Commits
Media: the S3 surface and zones (ADR-031) — SigV4, access keys, zone projection and deletion, one object for two doors (#108)
Changes
@@@ -126,6 +126,12 @@ MEDIA_CAPABILITY_GATE=report#MEDIA_UPLOAD_TOKEN_TTL_S=3600 # default presigned upload URL lifetime#MEDIA_MULTIPART_MAX_MB=20480#MEDIA_MULTIPART_PART_MB=64 # bounds: MEDIA_MULTIPART_MIN_PART_MB=5, MEDIA_MULTIPART_MAX_PART_MB=256++++# ── S3 surface and zones (ADR-031; server/s3/, docs/s3.md) ──++# MEDIA_S3_KEY_SECRET (32+ characters) encrypts the S3 access keys' secrets; unset = the S3 surface answers 503 and no++# key can be issued. Changing it makes every issued key unusable. MEDIA_S3_HOSTS: the Host names the surface answers.++#MEDIA_S3_KEY_SECRET=++#MEDIA_S3_HOSTS=s3.openvibe.media,s3.openvibe.network#MEDIA_MULTIPART_TTL_HOURS=24#MEDIA_UPLOAD_MIN_FREE_MB=10240 # free disk kept in reserve by multipart uploads and split/remux jobs@@@ -60,6 +60,9 @@ key ([Tenancy & auth](#tenancy--auth)).|---|---|---|---|| `media.resource.read` | first-party | `GET /api/v1/resources`, `GET /api/v1/resources/:ovrn` | the authority resource index ([Resource index](#resource-index-apiv1resources-first-party)). Active in openvibe-contracts since 0.108.0; the route enforces it (a Network service token holding it, loopback, audience `openvibe.media`) || `media.resource.control` | first-party | `POST /api/v1/resources/control` | `delete` a person's (or their project's) object: the soft delete, restorable until its retention ends ([Resource index](#resource-index-apiv1resources-first-party)). Active in openvibe-contracts since 0.169.0 |++| `media.object.s3` | public (an access key) | the S3 surface at `s3.openvibe.media` ([docs/s3.md](docs/s3.md)) | PutObject, GetObject, HeadObject, DeleteObject(s), ListObjects(V2), ListBuckets, HeadBucket, presigned GET/PUT; a bucket is a zone |++| `media.access_key.manage` | first-party | `POST/GET/DELETE /api/v1/projects/:project/access-keys` | S3 access keys for a project admin, the secret shown once (Zone's console, Services) |++| `media.zone.provision` | first-party (Zone) | `PUT/GET /internal/zones/:id` | OpenVibe.Zone's zone records, applied by revision; usage and deletion progress |Called elsewhere, as the service principal `media` (the OAuth client `media`):@@@ -126,7 +129,7 @@ server/storage.orphans.scan (monthly storage orphan report, service-wide, report only),storage.move.cleanup (the alert for placement moves whose delete keeps failing, service-wide)client-ip.js trust proxy = loopback; req.ip is the only client address--(openvibe-shared v4.1.3, openvibe-contracts v0.169.0, openvibe-sdk v0.44.0: pinned release tarballs, installed by npm)++(openvibe-shared v4.1.3, openvibe-contracts v0.173.0, openvibe-sdk v0.44.0: pinned release tarballs, installed by npm)scripts/smoke-test.sh end-to-end smoke test (boots a temp instance)scripts/reconcile-objects.js / object-invariant.js / no-good-copy-report.js object-model operator toolsscripts/media-jobs.js list jobs, run the size-invariant scan (dry run by default), approve/cancel proposals@@@ -586,6 +589,12 @@ without a signature is a 404 like any other private one). A native object in theR2 popularity cache redirects to its verified R2 copy; a GET of a native objectcounts its viewer for the day, identifying no one (see Storage tiering).++## S3 surface and zones++++Media answers a subset of S3 at `https://s3.openvibe.media` (ADR-031). Each bucket is a zone of a project environment, and zones are managed by OpenVibe.Zone. Media issues and verifies the `ovk_` access keys, and every object stays an ordinary Media object.++++See [docs/s3.md](docs/s3.md) for the operations, the rules and the tests.++## Object model and API v2Every stored blob is a **media object** (`med_<ULID>`, `media_objects`) with onennew file mode 100644@@@ -0,0 +1,82 @@++# OpenVibe.Media's S3 surface (ADR-031): path style, https://s3.openvibe.media/<zone name>/<key>. Node answers every++# request whose Host is an S3 host (MEDIA_S3_HOSTS, server/s3/routes.js). proxy_pass has no URI part on purpose: nginx++# then passes the request line exactly as the client sent it, and SigV4 signs that path (a URI part would make nginx++# decode and re-encode it, and every key with an escaped character would fail its signature).++# Certificate: the openvibe.media wildcard. DNS: s3.openvibe.media needs its record (the owner's Cloudflare action);++# until it has one, s3.openvibe.network (the openvibe.network wildcard name and certificate) is the same surface, and++# OpenVibe.Zone names it (ZONE_S3_ENDPOINT). MEDIA_S3_HOSTS lists both. Cloudflare's proxy carries at most 100 MB a++# request on this plan, so one object over S3 is at most 100 MB until the name is DNS-only.++limit_req_zone $binary_remote_addr zone=media_s3:10m rate=50r/s;++++server {++ listen 80;++ listen [::]:80;++ server_name s3.openvibe.media s3.openvibe.network;++ location /.well-known/acme-challenge/ { root /var/www/html; }++ location / { return 301 https://$host$request_uri; }++}++server {++ listen 443 ssl http2;++ listen [::]:443 ssl http2;++ server_name s3.openvibe.media;++ ssl_certificate /etc/letsencrypt/live/openvibe.media/fullchain.pem;++ ssl_certificate_key /etc/letsencrypt/live/openvibe.media/privkey.pem;++++ # One PutObject carries at most MEDIA_OBJECT_MAX_MB (256 MB) of data; aws-chunked framing adds a little.++ client_max_body_size 300m;++ proxy_read_timeout 600s;++ proxy_send_timeout 600s;++ # Signed URLs carry long query strings.++ large_client_header_buffers 8 16k;++++ # No path is held back: every path is a zone and a key. (Node's own /metrics answers proxied callers 404, so Zone++ # refuses the zone name `metrics`.)++ location / {++ limit_req zone=media_s3 burst=200 nodelay;++ proxy_pass http://127.0.0.1:4100;++ proxy_http_version 1.1;++ proxy_set_header Host $host;++ proxy_set_header X-Media-Shield-Host "";++ proxy_set_header X-Media-Shield-Failed "";++ proxy_set_header X-Forwarded-Host "";++ proxy_set_header X-Real-IP $remote_addr;++ proxy_set_header X-Forwarded-For $remote_addr;++ proxy_set_header CF-Connecting-IP $remote_addr;++ proxy_set_header X-Forwarded-Proto $scheme;++ proxy_request_buffering off;++ proxy_buffering off;++ }++}++++server {++ listen 443 ssl http2;++ listen [::]:443 ssl http2;++ server_name s3.openvibe.network;++ ssl_certificate /etc/letsencrypt/live/openvibe.network/fullchain.pem;++ ssl_certificate_key /etc/letsencrypt/live/openvibe.network/privkey.pem;++++ # One PutObject carries at most MEDIA_OBJECT_MAX_MB (256 MB) of data; aws-chunked framing adds a little.++ client_max_body_size 300m;++ proxy_read_timeout 600s;++ proxy_send_timeout 600s;++ # Signed URLs carry long query strings.++ large_client_header_buffers 8 16k;++++ # No path is held back: every path is a zone and a key. (Node's own /metrics answers proxied callers 404, so Zone++ # refuses the zone name `metrics`.)++ location / {++ limit_req zone=media_s3 burst=200 nodelay;++ proxy_pass http://127.0.0.1:4100;++ proxy_http_version 1.1;++ proxy_set_header Host $host;++ proxy_set_header X-Media-Shield-Host "";++ proxy_set_header X-Media-Shield-Failed "";++ proxy_set_header X-Forwarded-Host "";++ proxy_set_header X-Real-IP $remote_addr;++ proxy_set_header X-Forwarded-For $remote_addr;++ proxy_set_header CF-Connecting-IP $remote_addr;++ proxy_set_header X-Forwarded-Proto $scheme;++ proxy_request_buffering off;++ proxy_buffering off;++ }++}nnew file mode 100644@@@ -0,0 +1,54 @@++# The S3 surface and zones++++OpenVibe.Media answers a subset of the Amazon S3 API at `https://s3.openvibe.media`, path style++(`/<bucket>/<key>`). The design is ADR-031 in OpenVibe.Contracts, with its two amendments. A **bucket is a zone**:++a named object zone of one project environment. Zones are made and deleted only through OpenVibe.Zone++(https://openvibe.zone).++++## Pieces++++| Piece | File | What it does |++|---|---|---|++| Signatures | `server/s3/sigv4.js` | Verifies SigV4 (Authorization header and presigned query) and decodes aws-chunked bodies, checking each chunk's signature and the trailers. |++| Access keys | `server/s3/keys.js` | `ovk_` keys for one project environment: read, write or both, and optionally limited to some zones. Secrets are encrypted with `MEDIA_S3_KEY_SECRET` and shown once. |++| Zones | `server/s3/zones.js` | Media's copy of Zone's records. It resolves bucket names, gives each zone's namespace its quota, and runs zone deletion. |++| S3 routes | `server/s3/routes.js` | The operations below, dispatched by `Host` (`MEDIA_S3_HOSTS`). |++| Control doors | `server/s3/admin-routes.js` | `PUT/GET /internal/zones/:id` (Zone only, `media.zone.provision`) and `/api/v1/projects/:project/access-keys` (`media.access_key.manage`, on behalf of a project admin). |++++## Operations++++**Supported:**++- `PutObject`, in one request, including aws-chunked bodies. `Content-MD5`, `x-amz-content-sha256` and the `x-amz-checksum-*` headers and trailers are checked.++- `GetObject` (with ranges, conditional headers and presigned response overrides) and `HeadObject`.++- `DeleteObject` and `DeleteObjects`.++- `ListObjectsV2` and `ListObjects` (prefix, delimiter and pages).++- `ListBuckets`, `HeadBucket` and `GetBucketLocation`.++- Presigned GET and PUT, valid for 7 days at most.++++**Not implemented:** multipart uploads, `CopyObject`, ACLs, policies, versioning, tagging and bucket deletion.++`CreateBucket` answers 200 for a zone that already exists, so tools that probe this way keep working.++++## Rules++++- **Two doors, one object.** An object written over S3 is an ordinary Media object: a `med_` id in the zone's namespace (`app.<project>[.sandbox].<zone id>`) with `zone_id` and `object_key` set. The native API reads, lists and deletes it, and the reverse works too.++- **The default zone.** It is the environment root. Over S3 it lists the environment's native objects under a derived key: the child namespace path, then the `med_` id. A key of that form cannot be written over S3.++- **Overwrite.** The new bytes are stored first. Then, in one transaction under the tenant's quota lock, the previous object goes through Media's deletion path and the new one becomes ready. A held object is neither overwritten nor deleted (`AccessDenied`).++- **Quotas.** A write is checked against the zone's `quota_bytes` and the environment's quota, before the bytes arrive and again after.++- **Zone deletion.** When Zone pushes state `deleting`, the zone answers `NoSuchBucket` at once. The minute sweep (`zones.sweepDeleting`) tombstones every object with no restore window and purges every copy. The status reports `objects_remaining` and `held_objects` until nothing is left. Then Zone marks the zone `deleted`: its name is retired, and keys scoped only to it are revoked.++- **No browser execution.** Every response carries `Content-Security-Policy: sandbox; default-src 'none'` and `nosniff`, whatever type the object declares.++++## Tests++++- `test/s3-zones.test.js` runs end to end with the AWS SDK for JavaScript v3: every operation, presigned URLs, isolation, quotas, holds, two doors and zone deletion.++- `test/s3-sigv4.test.js` checks the worked examples of the S3 API reference, including the chunk signature chain.++++## Using it++++```sh++aws configure set aws_access_key_id ovk_… # issued on openvibe.zone++aws configure set aws_secret_access_key … # shown once++aws configure set default.s3.multipart_threshold 256MB # no multipart yet: one request per object++aws --endpoint-url https://s3.openvibe.media s3 ls s3://assets/++rclone config create oz s3 provider Other endpoint https://s3.openvibe.media access_key_id ovk_… secret_access_key … \++ upload_cutoff 256M no_check_bucket true++```nnew file mode 100644@@@ -0,0 +1,50 @@++-- phase: expand++-- Named object zones and the S3 surface (ADR-031 and its 2026-10-02 amendment, OpenVibe.Zone).++--++-- media_zones is Media's projection of the zone records OpenVibe.Zone owns: Zone pushes each change with a higher++-- revision (PUT /internal/zones/:id) and Media resolves S3 bucket names, quotas and states from it. A name stays with++-- its zone for ever within one project environment, deleted or not, so a signed request names exactly one zone id.++CREATE TABLE media_zones (++ id text COLLATE "C" PRIMARY KEY, -- zon_<ULID>++ project_id text COLLATE "C" NOT NULL,++ env text COLLATE "C" NOT NULL CHECK (env IN ('sandbox', 'production')),++ name text COLLATE "C" NOT NULL,++ is_default boolean NOT NULL DEFAULT false,++ namespace text COLLATE "C" NOT NULL UNIQUE, -- the environment root, or <root>.<id>++ quota_bytes bigint,++ state text COLLATE "C" NOT NULL CHECK (state IN ('provisioning', 'active', 'deleting', 'deleted')),++ revision bigint NOT NULL,++ record jsonb NOT NULL, -- the zone.object-zone@1 Zone sent++ deletion_requested_at bigint,++ deletion_completed_at bigint,++ updated_at bigint NOT NULL,++ UNIQUE (project_id, env, name)++);++++-- S3 access keys (ADR-031: bound to one project environment, read and/or write, optionally to some zones). Media++-- verifies the SigV4 signature, so Media keeps the secret: encrypted with MEDIA_S3_KEY_SECRET, shown once at issue.++CREATE TABLE media_access_keys (++ id text COLLATE "C" PRIMARY KEY, -- ovk_<ULID>, the S3 access key id++ project_id text COLLATE "C" NOT NULL,++ env text COLLATE "C" NOT NULL CHECK (env IN ('sandbox', 'production')),++ name text NOT NULL,++ permissions text COLLATE "C" NOT NULL CHECK (permissions IN ('read', 'write', 'read-write')),++ zones jsonb NOT NULL DEFAULT '[]', -- [] = every zone of the environment++ secret_enc text COLLATE "C" NOT NULL, -- iv.tag.ciphertext (AES-256-GCM), base64url++ created_by text COLLATE "C" NOT NULL, -- usr_… who issued it++ created_at bigint NOT NULL,++ last_used_at bigint,++ revoked_at bigint,++ revoked_by text COLLATE "C"++);++CREATE INDEX media_access_keys_project ON media_access_keys (project_id, env) WHERE revoked_at IS NULL;++++-- An object's zone and its S3 key. Objects written over S3 carry both; a native object's key is derived (its med_ id,++-- below the child namespace path in the default zone) until the contract migration backfills every row.++ALTER TABLE media_objects ADD COLUMN zone_id text COLLATE "C";++ALTER TABLE media_objects ADD COLUMN object_key text COLLATE "C";++-- One live object per key in a zone. An S3 write stores its bytes first and then, in one transaction under the++-- tenant's quota lock, tombstones the previous object of the key and inserts the new one already ready.++CREATE UNIQUE INDEX media_objects_zone_key_live ON media_objects (zone_id, object_key)++ WHERE zone_id IS NOT NULL AND object_key IS NOT NULL AND lifecycle_status = 'ready';++CREATE INDEX media_objects_namespace_key ON media_objects (namespace, object_key) WHERE lifecycle_status = 'ready';@@@ -16,7 +16,7 @@"express": "^4.19.2","iovalkey": "^0.4.0","multer": "^1.4.5-lts.1",-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.169.0",++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.173.0","openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.44.0","openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.1.3","pg": "^8.23.0",@@@ -1847,9 +1847,9 @@}},"node_modules/openvibe-contracts": {-- "version": "0.169.0",-- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.169.0",-- "integrity": "sha512-Py2z8UaKGvLzG3atuIAnJKnRe74Te9cJRTaxqFq6m4fov6gaMXeS6rn7lH0tSIo0qBGD5trrT618ARXnklkp+A==",++ "version": "0.173.0",++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.173.0",++ "integrity": "sha512-6zHbU9zghIYDd3PPGtgMTsIgxrWwsoBHO/rvuLW1rghr6kRK6VwCmSMZRcjQbcPODPH1XrUAFuLg/T1K4nOCMQ==","license": "AGPL-3.0-only","dependencies": {"ajv": "^8.17.1",@@@ -1,7 +1,7 @@{"name": "openvibe-media","version": "1.0.0",-- "description": "OpenVibe.Media \u2014 multi-tenant media service (VOD ingest/recording, storage tiering, clips, pastes, files, thumbnails)",++ "description": "OpenVibe.Media — multi-tenant media service (VOD ingest/recording, storage tiering, clips, pastes, files, thumbnails)","main": "server/index.js","scripts": {"start": "node server/index.js",@@@ -20,7 +20,7 @@"express": "^4.19.2","iovalkey": "^0.4.0","multer": "^1.4.5-lts.1",-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.169.0",++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.173.0","openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.44.0","openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.1.3","pg": "^8.23.0",@@@ -88,6 +88,14 @@ const ready = (async () => {// Network's sign-out-everywhere cutoffs (server/revocations.js): signed, loopback-only. Before the JSON// parser, which would consume the raw body the signature is over.app.post('/internal/events', ...require('./revocations').handler());++ // The S3 surface (ADR-031, server/s3/routes.js) answers every request whose Host is an S3 host (MEDIA_S3_HOSTS,++ // s3.openvibe.media): nginx's vhost passes the request line untouched, because the client signed that exact path.++ // It reads raw bodies (PutObject, aws-chunked), so it sits ahead of the JSON parser. The zone and access-key doors++ // carry their own parsers and service-token guards.++ const s3 = require('./s3/routes');++ app.use((req, res, next) => (s3.isS3Host(req) ? s3(req, res, next) : next()));++ app.use('/internal/zones', require('./s3/admin-routes').internalRouter());++ app.use('/api/v1/projects/:project/access-keys', require('./s3/admin-routes').keysRouter());app.use(express.json({ limit: '2mb' }));// IndexNow (openvibe-shared/indexnow): created once at boot from INDEXNOW_KEY. Unset → off, the@@@ -284,6 +292,10 @@ const ready = (async () => {} catch { /* not linux */ }});every(60 * 1000, () => recorder.checkDisk()); // disk guardian++ // Zones being deleted (ADR-031 amendment): their objects are tombstoned and purged in batches, every minute.++ every(60 * 1000, async () => {++ try { const r = await require('./s3/zones').sweepDeleting(); if (r.tombstoned || r.purged) console.log(`[Zones] ${r.zones} deleting zone(s): ${r.tombstoned} tombstoned, ${r.purged} purged`); } catch (err) { console.warn('[Zones] deletion sweep:', err.message); }++ });every(60 * 60 * 1000, () => thumbService.cleanupOldThumbnails()); // stale live thumbs// Object model: purge native objects whose soft-delete retention has passed (held ones are kept).every(60 * 60 * 1000, async () => {@@@ -624,7 +624,7 @@ async function softDelete(obj, { by = null } = {}) {async function restore(obj) {const md = parseJson(obj.metadata, {});-- if (md.purged_at) return null;++ if (md.purged_at || md.no_restore) return null; // a zone deletion's tombstones are never restoredconst back = md.pre_delete_status && md.pre_delete_status !== 'deleted' ? md.pre_delete_status : 'ready';delete md.pre_delete_status; delete md.retention_until; delete md.deleted_by;await db.run(`UPDATE media_objects SET lifecycle_status = ?, deleted_at = NULL, metadata = ?, updated_at = ov_now() WHERE id = ?`,@@@ -642,27 +642,34 @@ async function restore(obj) {async function purgeExpired({ retentionDays = config.objects.retentionDays } = {}) {const rows = await db.all(`SELECT * FROM media_objects WHERE lifecycle_status = 'deleted' AND legacy_ref IS NULLAND deleted_at <= datetime('now', ?)`, [`-${Math.max(0, retentionDays)} days`]);-- const root = path.resolve(config.objects.path) + path.sep;let purged = 0;-- for (const obj of rows) {-- const md = parseJson(obj.metadata, {});-- if (md.purged_at || await isHeld(obj.id)) continue;-- if ((await listLocations(obj.id)).some(l => l.provider === 'r2')) continue;-- for (const l of await listLocations(obj.id)) {-- if (l.provider === 'local' && path.resolve(l.key).startsWith(root)) { try { fs.unlinkSync(l.key); } catch { /* already gone */ } }-- if (l.provider === 'local') await db.run('DELETE FROM media_locations WHERE id = ?', [l.id]);-- }-- // Its CMAF segments (F3.1) go with the bytes. A durable copy whose delete failed keeps its row, so this purge-- // is not marked done and the next pass retries it rather than losing the key.-- const tl = await require('./timeline').removeObject(obj.id);-- if (tl.pending) continue;-- md.purged_at = new Date().toISOString();-- await db.run('UPDATE media_objects SET metadata = ?, updated_at = ov_now() WHERE id = ?', [JSON.stringify(md), obj.id]);-- purged++;-- }++ for (const obj of rows) if (await purgeOne(obj)) purged++;return purged;}++/**++ * Erase one tombstoned native object's bytes now (purgeExpired's step, and a zone deletion's, which has no retention++ * period): its local copies, then its CMAF segments; marked purged only when every copy is gone. A held object, one++ * still in the R2 cache (tiering demotes it first) and a purged one are left alone. true when it is purged now.++ */++async function purgeOne(obj) {++ const md = parseJson(obj.metadata, {});++ if (obj.lifecycle_status !== 'deleted' || obj.legacy_ref || md.purged_at || await isHeld(obj.id)) return false;++ if ((await listLocations(obj.id)).some(l => l.provider === 'r2')) return false;++ const root = path.resolve(config.objects.path) + path.sep;++ for (const l of await listLocations(obj.id)) {++ if (l.provider === 'local' && path.resolve(l.key).startsWith(root)) { try { fs.unlinkSync(l.key); } catch { /* already gone */ } }++ if (l.provider === 'local') await db.run('DELETE FROM media_locations WHERE id = ?', [l.id]);++ }++ // Its CMAF segments (F3.1) go with the bytes. A durable copy whose delete failed keeps its row, so this purge++ // is not marked done and the next pass retries it rather than losing the key.++ const tl = await require('./timeline').removeObject(obj.id);++ if (tl.pending) return false;++ md.purged_at = new Date().toISOString();++ await db.run('UPDATE media_objects SET metadata = ?, updated_at = ov_now() WHERE id = ?', [JSON.stringify(md), obj.id]);++ return true;++}++/*** Bytes counted against the tenant's quota: v1 files, ready native objects (in a developer project* also soft-deleted ones until purged) and the reservations of uploads in progress@@@ -731,6 +738,6 @@ module.exports = {vodProjection, clipProjection, fileProjection, thumbnailProjection,syncVod, syncClip, syncFile, sync, safeSync, withObject, withObjectOrRow, afterTierMove,placeHold, releaseHold, holdPublic,-- objectFilePath, softDelete, restore, purgeExpired, usedBytes,++ objectFilePath, softDelete, restore, purgeExpired, purgeOne, usedBytes,legacyPublicUrl, objectPublic,};@@@ -31,6 +31,7 @@ const db = require('../db/database');const config = require('../config');const SEGMENT_RE = /^[a-z0-9][a-z0-9_-]{0,39}$/;++const ZONE_SEGMENT_RE = /^zon_[0-9A-HJKMNP-TV-Z]{26}$/;const MAX_DEPTH = 3;const POLICY_KEYS = ['kinds', 'visibilities', 'max_object_bytes', 'strict_verbs'];@@@ -100,6 +101,9 @@ function resolveName(app, requested) {if (s === root) return { namespace: root };const rel = s.startsWith(`${root}.`) ? s.slice(root.length + 1) : s;const segs = rel.split('.');++ // A named zone (OpenVibe.Zone) is the one child whose name is upper-case: <root>.zon_<ULID>, made only when Zone++ // provisions it (server/s3/zones.js); objects routes refuse one that does not exist.++ if (segs.length === 1 && ZONE_SEGMENT_RE.test(segs[0])) return { namespace: `${root}.${rel}`, zone: segs[0] };if (segs.length > MAX_DEPTH || !segs.every(x => SEGMENT_RE.test(x)) || `${root}.${rel}`.length > 200) {return { error: `namespace must be ${root} or a name below it: up to ${MAX_DEPTH} segments of a-z, 0-9, _ and - (e.g. ${root}.avatars)` };}@@@ -319,6 +319,10 @@ router.post('/', upload, limits('media.object.upload', UPLOAD), async (req, res)const namespace = named.namespace;const g = await namespaceGrant(req, 'write', namespace);if (!g.allowed) return problem(res, 403, g.code, g.reason);++ // A named zone (ADR-031 amendment): it must exist and be active, and it holds private objects only.++ const zone = named.zone ? await require('../s3/zones').byId(named.zone) : null;++ if (named.zone && (!zone || zone.state !== 'active' || zone.namespace !== namespace)) return problem(res, 404, 'media.zone.unknown', `no active zone ${named.zone} in this environment`);++ if (zone && visibility !== 'private') return problem(res, 422, 'media.zone.private', 'a named zone holds private objects only');const policy = await namespaces.effectivePolicy(req.appId, namespace);if (Array.isArray(policy.kinds) && !policy.kinds.includes(kind)) return problem(res, 422, 'media.namespace.policy_denied', `${namespace} takes kinds ${policy.kinds.join(', ') || '(none)'}`);if (Array.isArray(policy.visibilities) && !policy.visibilities.includes(visibility)) return problem(res, 422, 'media.namespace.policy_denied', `${namespace} takes visibilities ${policy.visibilities.join(', ') || '(none)'}`);@@@ -366,6 +370,8 @@ router.post('/', upload, limits('media.object.upload', UPLOAD), async (req, res)app_id: req.appId, namespace, kind, owner_subject: subject, owner_app: req.appId, owner_user_id: userId,visibility, lifecycle_status: 'uploading', mime_type: mime, size_bytes: size, metadata,});++ // In a named zone the object lists over S3 under its id until a key is given (only S3 writes give one).++ if (zone) await db.run('UPDATE media_objects SET zone_id = ?, object_key = ? WHERE id = ?', [zone.id, newId, newId]);await namespaces.reserve(await model.getObject(newId), size);return { id: newId };});nnew file mode 100644@@@ -0,0 +1,96 @@++'use strict';++/**++ * The two control doors of the S3 surface, both loopback-only and Network-service-token-only (server/service-guard.js):++ *++ * OpenVibe.Zone → Media (capability media.zone.provision), mounted at /internal/zones:++ * PUT /:id body zone.object-zone@1 → { applied, ...report } a zone change, applied when its revision is higher++ * GET /:id → report: { zone_id, revision, state, usage, deletion } what Zone shows and polls while deleting++ *++ * OpenVibe.Services → Media (capability media.access_key.manage), mounted at /api/v1/projects/:project/access-keys,++ * always on behalf of a person who is an admin or owner of the project (asked of Network, server/registry/projects.js):++ * POST / { env, name, permissions, zones, on_behalf_of } → 201 { key: media.access-key@1, secret }++ * GET /?on_behalf_of= → { keys: [media.access-key@1] }++ * DELETE /:id?on_behalf_of= → 200 media.access-key@1 (revoked; S3 refuses it at once)++ * The secret is in the 201 only; Media never shows it again.++ */++const express = require('express');++const { http } = require('openvibe-contracts');++const { guard } = require('../service-guard');++const zones = require('./zones');++const keys = require('./keys');++++const PROJECT_RE = /^prj_[0-9A-HJKMNP-TV-Z]{26}$/;++const SUBJECT_RE = /^usr_[0-9A-HJKMNP-TV-Z]{26}$/;++const problem = (res, status, code, detail) => http.sendProblem(res, status, code, { detail });++++function internalRouter() {++ const r = express.Router();++ r.use(guard('media.zone.provision'), express.json({ limit: '64kb' }));++ r.put('/:id', async (req, res, next) => {++ try {++ res.set('Cache-Control', 'no-store');++ if (!req.body || req.body.id !== req.params.id) return problem(res, 400, 'media.zone.invalid', 'the body is the zone named in the path');++ const out = await zones.provision(req.body);++ res.json({ applied: out.applied, ...await zones.report(out.zone) });++ } catch (err) {++ if (err instanceof zones.ZoneError) return problem(res, err.status, err.code, err.message);++ next(err);++ }++ });++ r.get('/:id', async (req, res, next) => {++ try {++ res.set('Cache-Control', 'no-store');++ const zone = await zones.byId(req.params.id);++ if (!zone) return problem(res, 404, 'media.zone.unknown', 'Media has no such zone');++ res.json(await zones.report(zone));++ } catch (err) { next(err); }++ });++ return r;++}++++function keysRouter({ projects = require('../registry/projects').createProjects() } = {}) {++ const r = express.Router({ mergeParams: true });++ r.use(guard('media.access_key.manage'), express.json({ limit: '16kb' }));++++ // A first-party service acts, and only for a person who administers the project.++ r.use(async (req, res, next) => {++ try {++ res.set('Cache-Control', 'no-store');++ if (!String(req.principal.sub || '').startsWith('svc:')) return problem(res, 403, 'capability.denied', 'media.access_key.manage requires a service token');++ if (!PROJECT_RE.test(req.params.project)) return problem(res, 404, 'media.project.unknown', 'no such project');++ const person = String((req.body && req.body.on_behalf_of) || req.query.on_behalf_of || '');++ if (!SUBJECT_RE.test(person)) return problem(res, 400, 'media.access_key.no_person', 'on_behalf_of names the person acting (usr_…)');++ let admin;++ try { admin = await projects.isAdmin(req.params.project, person); } catch (err) {++ return problem(res, 503, 'media.network.unavailable', `project roles cannot be read: ${err.message}`);++ }++ if (!admin) return problem(res, 403, 'media.access_key.not_admin', 'only an admin or owner of the project manages its access keys');++ req.person = person;++ next();++ } catch (err) { next(err); }++ });++++ r.post('/', async (req, res, next) => {++ try {++ const b = req.body || {};++ const out = await keys.issue({ projectId: req.params.project, env: b.env, name: b.name, permissions: b.permissions || 'read-write', zones: b.zones || [], createdBy: req.person });++ res.status(201).json(out);++ } catch (err) {++ if (err instanceof keys.KeyError) return problem(res, err.status, err.code, err.message);++ next(err);++ }++ });++ r.get('/', async (req, res, next) => {++ try { res.json({ keys: await keys.list(req.params.project) }); } catch (err) { next(err); }++ });++ r.delete('/:id', async (req, res, next) => {++ try {++ const out = await keys.revoke(req.params.project, req.params.id, req.person);++ if (!out) return problem(res, 404, 'media.access_key.unknown', 'no such key in this project');++ res.json(out);++ } catch (err) { next(err); }++ });++ return r;++}++++module.exports = { internalRouter, keysRouter };nnew file mode 100644@@@ -0,0 +1,130 @@++'use strict';++/**++ * S3 access keys (ADR-031): an `ovk_<ULID>` id and a secret, bound to one project environment, read and/or write,++ * and optionally to some of its zones. Media verifies SigV4 itself (server/s3/sigv4.js), so Media keeps the secret,++ * encrypted with MEDIA_S3_KEY_SECRET (AES-256-GCM); it is returned once, when the key is issued, and never again.++ *++ * issue({ projectId, env, name, permissions, zones, createdBy }) → { key, secret }++ * find(id) → the row with `secret` (decrypted), or null; revoked keys are null too++ * list(projectId) → public shapes, newest first (revoked included)++ * revoke(projectId, id, by) → the public shape, or null when there is no such key of that project++ * publicShape(row) → media.access-key@1++ * enabled() → whether keys can be issued and verified (the secret is configured)++ */++const crypto = require('crypto');++const { ids } = require('openvibe-contracts');++const db = require('../db/database');++++const PERMISSIONS = ['read', 'write', 'read-write'];++const ZONE_ID_RE = /^zon_[0-9A-HJKMNP-TV-Z]{26}$/;++const KEY_ID_RE = /^ovk_[0-9A-HJKMNP-TV-Z]{26}$/;++const MAX_KEYS_PER_ENV = 50;++const TOUCH_MS = 60_000;++++function masterKey() {++ const s = String(process.env.MEDIA_S3_KEY_SECRET || '');++ return s.length >= 32 ? crypto.createHash('sha256').update(s).digest() : null;++}++const enabled = () => !!masterKey();++++function encrypt(secret) {++ const iv = crypto.randomBytes(12);++ const c = crypto.createCipheriv('aes-256-gcm', masterKey(), iv);++ const enc = Buffer.concat([c.update(secret, 'utf8'), c.final()]);++ return [iv, c.getAuthTag(), enc].map((b) => b.toString('base64url')).join('.');++}++++function decrypt(blob) {++ const [iv, tag, enc] = String(blob).split('.').map((p) => Buffer.from(p, 'base64url'));++ const d = crypto.createDecipheriv('aes-256-gcm', masterKey(), iv);++ d.setAuthTag(tag);++ return Buffer.concat([d.update(enc), d.final()]).toString('utf8');++}++++const iso = (ms) => (ms == null ? null : new Date(Number(ms)).toISOString());++const parseZones = (z) => (Array.isArray(z) ? z : (() => { try { return JSON.parse(z || '[]'); } catch { return []; } })());++++function publicShape(r) {++ return {++ id: r.id,++ project_id: r.project_id,++ env: r.env,++ name: r.name,++ permissions: r.permissions,++ zones: parseZones(r.zones),++ created_by: r.created_by,++ created_at: iso(r.created_at),++ last_used_at: iso(r.last_used_at),++ revoked_at: iso(r.revoked_at),++ };++}++++class KeyError extends Error {++ constructor(status, code, message) { super(message); this.status = status; this.code = code; }++}++++async function issue({ projectId, env, name, permissions = 'read-write', zones = [], createdBy, now = Date.now() }) {++ if (!enabled()) throw new KeyError(503, 'media.s3.disabled', 'S3 access keys are not configured on this Media (MEDIA_S3_KEY_SECRET)');++ if (!['sandbox', 'production'].includes(env)) throw new KeyError(400, 'media.access_key.invalid', 'env is sandbox or production');++ if (!PERMISSIONS.includes(permissions)) throw new KeyError(400, 'media.access_key.invalid', `permissions is one of ${PERMISSIONS.join(', ')}`);++ const label = String(name || '').trim().slice(0, 80);++ if (!label) throw new KeyError(400, 'media.access_key.invalid', 'a key needs a name (what uses it)');++ if (!Array.isArray(zones) || zones.length > 20 || !zones.every((z) => ZONE_ID_RE.test(String(z)))) {++ throw new KeyError(400, 'media.access_key.invalid', 'zones is a list of at most 20 zone ids (zon_…); empty means every zone of the environment');++ }++ const uniq = [...new Set(zones)];++ for (const z of uniq) {++ const zone = await db.get('SELECT project_id, env, state FROM media_zones WHERE id = ?', [z]);++ if (!zone || zone.project_id !== projectId || zone.env !== env || !['provisioning', 'active'].includes(zone.state)) {++ throw new KeyError(404, 'media.access_key.unknown_zone', `${z} is not a zone of this project's ${env} environment`);++ }++ }++ const live = (await db.get('SELECT COUNT(*) AS n FROM media_access_keys WHERE project_id = ? AND env = ? AND revoked_at IS NULL', [projectId, env])).n;++ if (Number(live) >= MAX_KEYS_PER_ENV) throw new KeyError(409, 'media.access_key.limit', `an environment holds at most ${MAX_KEYS_PER_ENV} live keys; revoke one first`);++ const id = `ovk_${ids.ulid(now)}`;++ const secret = crypto.randomBytes(30).toString('base64url'); // 40 characters, as S3 secrets are++ await db.run(`INSERT INTO media_access_keys (id, project_id, env, name, permissions, zones, secret_enc, created_by, created_at)++ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`, [id, projectId, env, label, permissions, JSON.stringify(uniq), encrypt(secret), createdBy, now]);++ return { key: publicShape(await db.get('SELECT * FROM media_access_keys WHERE id = ?', [id])), secret };++}++++const touched = new Map();++async function find(id, { now = Date.now() } = {}) {++ if (!KEY_ID_RE.test(String(id || '')) || !enabled()) return null;++ const row = await db.get('SELECT * FROM media_access_keys WHERE id = ? AND revoked_at IS NULL', [id]);++ if (!row) return null;++ let secret;++ try { secret = decrypt(row.secret_enc); } catch { return null; } // a key encrypted under another secret++ if (!touched.has(id) || now - touched.get(id) > TOUCH_MS) {++ touched.set(id, now);++ await db.run('UPDATE media_access_keys SET last_used_at = ? WHERE id = ?', [now, id]);++ }++ return { ...row, zones: parseZones(row.zones), secret };++}++++async function list(projectId) {++ const rows = await db.all('SELECT * FROM media_access_keys WHERE project_id = ? ORDER BY created_at DESC, id DESC', [projectId]);++ return rows.map(publicShape);++}++++async function revoke(projectId, id, by, { now = Date.now() } = {}) {++ const row = await db.get('SELECT * FROM media_access_keys WHERE id = ? AND project_id = ?', [id, projectId]);++ if (!row) return null;++ if (row.revoked_at == null) await db.run('UPDATE media_access_keys SET revoked_at = ?, revoked_by = ? WHERE id = ?', [now, by, id]);++ touched.delete(id);++ return publicShape(await db.get('SELECT * FROM media_access_keys WHERE id = ?', [id]));++}++++/** Revoke every live key scoped only to `zoneId` (a zone that was deleted); keys naming other zones lose it. */++async function dropZone(zoneId, { now = Date.now() } = {}) {++ const rows = await db.all("SELECT id, zones FROM media_access_keys WHERE revoked_at IS NULL AND zones::text LIKE ?", [`%${zoneId}%`]);++ for (const r of rows) {++ const rest = parseZones(r.zones).filter((z) => z !== zoneId);++ if (!rest.length) await db.run("UPDATE media_access_keys SET revoked_at = ?, revoked_by = 'system:zone-deleted' WHERE id = ?", [now, r.id]);++ else await db.run('UPDATE media_access_keys SET zones = ? WHERE id = ?', [JSON.stringify(rest), r.id]);++ touched.delete(r.id);++ }++ return rows.length;++}++++module.exports = { issue, find, list, revoke, dropZone, publicShape, enabled, KeyError, PERMISSIONS, KEY_ID_RE, ZONE_ID_RE };nnew file mode 100644@@@ -0,0 +1,597 @@++'use strict';++/**++ * Media's S3 surface (ADR-031 and its zone amendment), path style: https://s3.openvibe.media/<zone name>/<key>.++ *++ * Served by Host (MEDIA_S3_HOSTS), not by path: nginx passes the request line untouched, since SigV4 signs the path.++ *++ * GET / ListBuckets: the active zones of the key's environment (those it may reach)++ * HEAD /<zone> HeadBucket++ * GET /<zone>?location GetBucketLocation++ * GET /<zone>?list-type=2 ListObjectsV2 (prefix, delimiter, max-keys, continuation-token, start-after)++ * GET /<zone> ListObjects (v1: prefix, delimiter, max-keys, marker)++ * PUT /<zone> CreateBucket: 200 for a zone that exists (clients probe this way), else NotImplemented++ * POST /<zone>?delete DeleteObjects++ * PUT /<zone>/<key> PutObject (one request; aws-chunked bodies too)++ * GET /<zone>/<key> GetObject (Range)++ * HEAD /<zone>/<key> HeadObject++ * DELETE /<zone>/<key> DeleteObject (Media's soft delete; refused under a retention hold)++ * everything else NotImplemented (multipart, copy, ACLs, policies, versioning, bucket deletion)++ *++ * Every request is signed (SigV4, header or presigned) with an access key (server/s3/keys.js). The key fixes the++ * project and environment, so a bucket name resolves to one zone id of that environment (server/s3/zones.js); a key++ * scoped to zones reaches only those. A request is authorised as the project's app would be: the same tenant, the++ * same quotas (the zone's ceiling and the environment's), the same objects as the native API (/api/v2/:project).++ *++ * An object written here is an ordinary Media object: a med_ id in the zone's namespace, with `zone_id` and its key.++ * A write stores the bytes first, then in one transaction under the tenant's quota lock tombstones the key's previous++ * object (Media's deletion path; refused while it is held) and makes the new one ready. The default zone also lists++ * the environment's native objects, under a derived key: the child namespace path and the med_ id.++ *++ * Responses never run in a browser: every one carries a sandbox CSP and nosniff, whatever type the object declares.++ */++const crypto = require('crypto');++const fs = require('fs');++const path = require('path');++const { pipeline } = require('stream/promises');++const { PassThrough } = require('stream');++const express = require('express');++const db = require('../db/database');++const config = require('../config');++const model = require('../objects/model');++const namespaces = require('../objects/namespaces');++const sigv4 = require('./sigv4');++const keys = require('./keys');++const zones = require('./zones');++++const MAX_KEY_BYTES = 1024;++const MAX_LIST = 1000;++const MAX_META_BYTES = 2048;++const DERIVED_KEY = /^(?:[a-z0-9][a-z0-9_-]{0,39}\/){0,3}med_[0-9A-HJKMNP-TV-Z]{26}$/;++const XMLNS = 'http://s3.amazonaws.com/doc/2006-03-01/';++++const STATUS = {++ AccessDenied: 403, InvalidAccessKeyId: 403, SignatureDoesNotMatch: 403, RequestTimeTooSkewed: 403,++ AuthorizationHeaderMalformed: 400, AuthorizationQueryParametersError: 400, InvalidRequest: 400, InvalidArgument: 400,++ KeyTooLongError: 400, MissingContentLength: 411, EntityTooLarge: 400, BadDigest: 400, XAmzContentSHA256Mismatch: 400,++ IncompleteBody: 400, MalformedXML: 400, InvalidRange: 416, NoSuchBucket: 404, NoSuchKey: 404, QuotaExceeded: 403,++ NotImplemented: 501, MethodNotAllowed: 405, InternalError: 500, ServiceUnavailable: 503, InvalidBucketName: 400,++};++++const esc = (s) => String(s).replace(/&/g, '&').replace(/</g, '<').replace(/>/g, '>').replace(/"/g, '"').replace(/'/g, ''');++const requestId = () => crypto.randomBytes(8).toString('hex').toUpperCase();++++function baseHeaders(res) {++ res.set('Content-Security-Policy', "sandbox; default-src 'none'");++ res.set('X-Content-Type-Options', 'nosniff');++ res.set('x-amz-request-id', res.locals.rid || (res.locals.rid = requestId()));++ res.set('Cache-Control', 'private, no-store');++}++++/** Headers as given (Express's res.set would add a charset to a text Content-Type; S3 answers the stored type). */++function setAll(res, headers) {++ for (const [k, v] of Object.entries(headers)) res.setHeader(k, v);++}++++function xml(res, status, body) {++ baseHeaders(res);++ res.status(status).type('application/xml').send(`<?xml version="1.0" encoding="UTF-8"?>\n${body}`);++}++++function s3error(req, res, code, message, extra = '') {++ const status = STATUS[code] || 400;++ baseHeaders(res);++ if (req.method === 'HEAD') return res.status(status).end();++ xml(res, status, `<Error><Code>${esc(code)}</Code><Message>${esc(message)}</Message><Resource>${esc(req.originalUrl.split('?')[0])}</Resource>`++ + `${extra}<RequestId>${res.locals.rid}</RequestId></Error>`);++}++++const etagOf = (obj) => {++ const md = model.parseJson(obj.metadata, {});++ return md.md5 ? `"${md.md5}"` : `"${String(obj.content_hash || obj.id).slice(0, 32)}-1"`;++};++const lastModified = (obj) => new Date(String(obj.updated_at || obj.created_at).replace(' ', 'T') + (String(obj.updated_at || obj.created_at).endsWith('Z') ? '' : 'Z'));++const isoOf = (d) => (Number.isFinite(d.getTime()) ? d.toISOString().replace(/\.\d{3}Z$/, '.000Z') : new Date(0).toISOString());++++// ── Authentication: the signature, the key, the tenant ───────++++async function authenticate(req, res, next) {++ res.locals.rid = requestId();++ if (!keys.enabled()) return s3error(req, res, 'ServiceUnavailable', 'The S3 surface is not configured on this Media');++ const parsed = sigv4.parse(req);++ if (parsed.error) return s3error(req, res, parsed.error.code, parsed.error.message);++ if (parsed.anonymous) return s3error(req, res, 'AccessDenied', 'Every request must be signed with an access key');++ const key = await keys.find(parsed.accessKeyId);++ if (!key) return s3error(req, res, 'InvalidAccessKeyId', 'The access key ID you provided does not exist in our records');++ const bad = sigv4.verify(req, parsed, key.secret);++ if (bad) return s3error(req, res, bad.code, bad.message);++ try {++ const quota = (key.env === 'sandbox' ? config.apps.sandboxQuotaMb : config.apps.projectQuotaMb) * 1024 * 1024;++ req.appRow = await db.ensureProjectTenant(key.project_id, key.env, quota);++ } catch (err) {++ return s3error(req, res, 'AccessDenied', err.message);++ }++ req.appId = req.appRow.app_id;++ req.s3 = { key, parsed };++ next();++}++++const mayRead = (key) => key.permissions === 'read' || key.permissions === 'read-write';++const mayWrite = (key) => key.permissions === 'write' || key.permissions === 'read-write';++++/** The zone a bucket name names for this key, or an S3 error answered (and null returned). */++async function zoneFor(req, res, bucket) {++ const { key } = req.s3;++ const zone = await zones.byName(key.project_id, key.env, bucket);++ if (!zone || zone.state !== 'active') { s3error(req, res, 'NoSuchBucket', 'The specified bucket does not exist'); return null; }++ if (key.zones.length && !key.zones.includes(zone.id)) { s3error(req, res, 'AccessDenied', 'This access key is not scoped to this bucket'); return null; }++ return zone;++}++++// ── Keys and objects ─────────────────────────────────────────++++/** The object key from the path (decoded once), or null when it is not a valid S3 key. */++function keyOf(rest) {++ let k;++ try { k = decodeURIComponent(rest); } catch { return null; }++ const bytes = Buffer.byteLength(k, 'utf8');++ return bytes >= 1 && bytes <= MAX_KEY_BYTES ? k : null;++}++++/** The ready object a key names in a zone (S3-written, or a native object's derived key in the default zone). */++async function findObject(zone, tenantId, key) {++ const row = await db.get(`SELECT * FROM media_objects WHERE zone_id = ? AND object_key = ? AND lifecycle_status = 'ready'`, [zone.id, key]);++ if (row) return row;++ if (!zone.is_default || !DERIVED_KEY.test(key)) return null;++ const parts = key.split('/');++ const id = parts.pop();++ const ns = parts.length ? `${zone.namespace}.${parts.join('.')}` : zone.namespace;++ return await db.get(`SELECT * FROM media_objects WHERE id = ? AND app_id = ? AND namespace = ? AND object_key IS NULL++ AND lifecycle_status = 'ready'`, [id, tenantId, ns]) || null;++}++++/** SQL for the key every object of a zone lists under (its own, or the derived one in the default zone). */++function effectiveKeySql(zone) {++ if (!zone.is_default) return 'o.object_key';++ const n = zone.namespace.length + 2;++ return `COALESCE(o.object_key, CASE WHEN o.namespace = '${zone.namespace.replace(/'/g, "''")}' THEN o.id++ ELSE replace(substr(o.namespace, ${n}), '.', '/') || '/' || o.id END)`;++}++++// ── Reading bytes ────────────────────────────────────────────++++/** bytes=a-b | a- | -n against `size` → { start, end } | 'invalid' | null (no Range). */++function parseRange(header, size) {++ if (!header) return null;++ const m = /^bytes=(\d*)-(\d*)$/.exec(String(header).trim());++ if (!m || (m[1] === '' && m[2] === '')) return null; // S3 ignores a range it cannot read++ let start, end;++ if (m[1] === '') { const n = Number(m[2]); if (n === 0) return 'invalid'; start = Math.max(0, size - n); end = size - 1; }++ else { start = Number(m[1]); end = m[2] === '' ? size - 1 : Math.min(Number(m[2]), size - 1); }++ if (start >= size || end < start) return 'invalid';++ return { start, end };++}++++function objectHeaders(obj) {++ const md = model.parseJson(obj.metadata, {});++ const h = {++ 'Content-Type': obj.mime_type || 'application/octet-stream',++ ETag: etagOf(obj),++ 'Last-Modified': lastModified(obj).toUTCString(),++ 'Accept-Ranges': 'bytes',++ };++ for (const [k, v] of Object.entries(md.s3_meta || {})) h[`x-amz-meta-${k}`] = v;++ if (md.content_disposition) h['Content-Disposition'] = md.content_disposition;++ if (md.content_encoding) h['Content-Encoding'] = md.content_encoding;++ if (md.cache_control) h['Cache-Control'] = md.cache_control;++ return h;++}++++/** Presigned GETs may override response headers, as S3 allows. */++function responseOverrides(req) {++ const o = {};++ const map = { 'response-content-type': 'Content-Type', 'response-content-disposition': 'Content-Disposition',++ 'response-cache-control': 'Cache-Control', 'response-content-encoding': 'Content-Encoding', 'response-content-language': 'Content-Language',++ 'response-expires': 'Expires' };++ for (const [q, h] of Object.entries(map)) if (typeof req.query[q] === 'string' && req.query[q].length <= 512) o[h] = req.query[q];++ return o;++}++++async function sendObject(req, res, obj, { head = false } = {}) {++ const size = Number(obj.size_bytes) || 0;++ const headers = { ...objectHeaders(obj), ...responseOverrides(req) };++ const inm = req.headers['if-none-match'];++ if (inm && inm.split(',').map((s) => s.trim()).includes(headers.ETag)) { baseHeaders(res); setAll(res, headers); return res.status(304).end(); }++ const ims = req.headers['if-modified-since'] ? new Date(req.headers['if-modified-since']) : null;++ if (ims && Number.isFinite(ims.getTime()) && lastModified(obj).getTime() <= ims.getTime() + 999) { baseHeaders(res); setAll(res, headers); return res.status(304).end(); }++ const im = req.headers['if-match'];++ if (im && !im.split(',').map((s) => s.trim()).includes(headers.ETag) && im.trim() !== '*') return s3error(req, res, 'PreconditionFailed', 'At least one of the preconditions you specified did not hold');++ const range = parseRange(req.headers.range, size);++ if (range === 'invalid') { res.set('Content-Range', `bytes */${size}`); return s3error(req, res, 'InvalidRange', 'The requested range is not satisfiable'); }++ baseHeaders(res);++ setAll(res, headers);++ if (range) {++ res.status(206).set('Content-Range', `bytes ${range.start}-${range.end}/${size}`).set('Content-Length', String(range.end - range.start + 1));++ } else res.status(200).set('Content-Length', String(size));++ if (head || req.method === 'HEAD') return res.end();++ if (require('../drill').refuseBytes(res)) return;++ const from = range || (size ? { start: 0, end: size - 1 } : null);++ if (!from) return res.end();++ // The placement router ranks the copies; a local one streams from disk, a remote one is fetched and passed on++ // (S3 clients do not follow redirects).++ const router = require('../placement/router');++ const decision = await router.route({ object: obj, purpose: 'download', session: router.sessionFor(req, obj.id), contentType: headers['Content-Type'], expiresIn: 300 });++ for (const loc of decision.candidates || []) {++ if (loc.provider === 'local') {++ if (!fs.existsSync(loc.key)) continue;++ return await pipeline(fs.createReadStream(loc.key, { start: from.start, end: from.end }), res).catch(() => res.destroy());++ }++ const url = (loc.provider === decision.provider && decision.url)++ || await require('../vod/vod-storage').presignGet(loc.provider, loc.key, 300, {}).catch(() => null);++ if (!url) continue;++ const up = await fetch(url, { headers: { Range: `bytes=${from.start}-${from.end}` }, signal: AbortSignal.timeout(30_000) }).catch(() => null);++ if (!up || !(up.status === 206 || up.status === 200) || !up.body) continue;++ const { Readable } = require('stream');++ return await pipeline(Readable.fromWeb(up.body), res).catch(() => res.destroy());++ }++ if (!res.headersSent) return s3error(req, res, 'InternalError', 'The object bytes are unavailable right now');++ res.destroy();++}++++// ── Writing ──────────────────────────────────────────────────++++const CRC_TABLE = (() => {++ const t = new Uint32Array(256);++ for (let n = 0; n < 256; n++) { let c = n; for (let k = 0; k < 8; k++) c = c & 1 ? 0xedb88320 ^ (c >>> 1) : c >>> 1; t[n] = c >>> 0; }++ return t;++})();++function crc32Update(crc, buf) {++ let c = crc ^ 0xffffffff;++ for (const b of buf) c = CRC_TABLE[(c ^ b) & 0xff] ^ (c >>> 8);++ return (c ^ 0xffffffff) >>> 0;++}++++/** x-amz-meta-* headers (at most 2 KB, as S3 allows), names lowercased. */++function userMeta(req) {++ const out = {};++ let bytes = 0;++ for (const [k, v] of Object.entries(req.headers)) {++ if (!k.startsWith('x-amz-meta-')) continue;++ const name = k.slice('x-amz-meta-'.length);++ const value = Array.isArray(v) ? v.join(',') : String(v);++ bytes += name.length + value.length;++ out[name] = value;++ }++ return bytes > MAX_META_BYTES ? null : out;++}++++/**++ * Stream the body (decoded from aws-chunked when it is) to a temporary file, hashing it: sha256, md5, crc32.++ * → { bytes, sha256, md5, crc32, trailers } | { error: { code, message } }++ */++async function receive(req, tmp, cap, parsed) {++ const sha = crypto.createHash('sha256'), md5 = crypto.createHash('md5'), sha1 = crypto.createHash('sha1');++ let crc = 0, n = 0, trailers = {};++ const counter = new PassThrough();++ counter.on('data', (chunk) => {++ n += chunk.length;++ if (n > cap) { counter.destroy(Object.assign(new Error('too large'), { code: 'EntityTooLarge' })); return; }++ sha.update(chunk); md5.update(chunk); sha1.update(chunk); crc = crc32Update(crc, chunk);++ });++ const stages = [req];++ if (sigv4.isStreaming(parsed)) stages.push(sigv4.chunkDecoder(parsed, { onTrailers: (t) => { trailers = t; } }));++ stages.push(counter, fs.createWriteStream(tmp));++ try {++ await pipeline(...stages);++ } catch (err) {++ return { error: { code: err.code && STATUS[err.code] ? err.code : 'IncompleteBody', message: err.code === 'EntityTooLarge' ? `Objects sent in one request are at most ${cap} bytes` : (err.message || 'The request body ended early') } };++ }++ const crcBuf = Buffer.alloc(4); crcBuf.writeUInt32BE(crc);++ return { bytes: n, sha256: sha.digest('hex'), md5: md5.digest(), sha1: sha1.digest('base64'), crc32: crcBuf.toString('base64'), trailers };++}++++/** Every integrity claim the client made about the body, checked: Content-MD5, x-amz-content-sha256, x-amz-checksum-*. */++function integrityProblem(req, parsed, got) {++ const md5 = req.headers['content-md5'];++ if (md5 && md5 !== got.md5.toString('base64')) return ['BadDigest', 'The Content-MD5 you specified did not match what we received'];++ if (/^[0-9a-f]{64}$/.test(parsed.payloadHash) && parsed.payloadHash !== got.sha256) {++ return ['XAmzContentSHA256Mismatch', 'The provided x-amz-content-sha256 header does not match what was computed'];++ }++ const claims = { ...got.trailers };++ for (const k of ['x-amz-checksum-crc32', 'x-amz-checksum-sha256', 'x-amz-checksum-sha1']) if (req.headers[k]) claims[k] = String(req.headers[k]);++ const ours = { 'x-amz-checksum-crc32': got.crc32, 'x-amz-checksum-sha256': Buffer.from(got.sha256, 'hex').toString('base64'), 'x-amz-checksum-sha1': got.sha1 };++ for (const [k, v] of Object.entries(claims)) if (ours[k] && ours[k] !== v) return ['BadDigest', `The ${k.slice('x-amz-checksum-'.length)} checksum you specified did not match what we received`];++ return null;++}++++async function putObject(req, res, zone, key) {++ const { key: akey, parsed } = req.s3;++ if (!mayWrite(akey)) { req.resume(); return s3error(req, res, 'AccessDenied', 'This access key is read-only'); }++ if (req.headers['x-amz-copy-source']) { req.resume(); return s3error(req, res, 'NotImplemented', 'CopyObject is not supported yet: read the object and write it again'); }++ const acl = String(req.headers['x-amz-acl'] || 'private');++ if (acl !== 'private' && acl !== 'bucket-owner-full-control') { req.resume(); return s3error(req, res, 'AccessDenied', 'Zones are private: objects cannot be made public over S3'); }++ if (DERIVED_KEY.test(key)) {++ req.resume();++ return s3error(req, res, 'InvalidArgument', 'Keys of the form [path/]med_<id> are the ids of objects written through the native API; choose another key');++ }++ const meta = userMeta(req);++ if (!meta) { req.resume(); return s3error(req, res, 'InvalidArgument', `User metadata is limited to ${MAX_META_BYTES} bytes`); }++ const declared = Number(sigv4.isStreaming(parsed) ? req.headers['x-amz-decoded-content-length'] : req.headers['content-length']);++ const cap = config.objects.maxUploadMb * 1024 * 1024;++ if (!Number.isFinite(declared)) { req.resume(); return s3error(req, res, 'MissingContentLength', 'You must provide the Content-Length HTTP header'); }++ if (declared > cap) { res.set('Connection', 'close'); req.resume(); return s3error(req, res, 'EntityTooLarge', `Objects sent in one request are at most ${cap} bytes`); }++ // Room for the declared size before a byte is stored; checked again with the real size below.++ const pre = await namespaces.checkQuota(req.appRow, zone.namespace, { bytes: declared, objects: 0 });++ if (pre) { res.set('Connection', 'close'); req.resume(); return s3error(req, res, 'QuotaExceeded', pre.detail); }++++ const tmpDir = path.join(config.objects.path, '.tmp');++ fs.mkdirSync(tmpDir, { recursive: true });++ const tmp = path.join(tmpDir, `s3-${crypto.randomBytes(8).toString('hex')}`);++ const drop = () => { try { fs.unlinkSync(tmp); } catch { /* not written */ } };++ const got = await receive(req, tmp, Math.max(declared, 0), parsed);++ if (got.error) { drop(); return s3error(req, res, got.error.code, got.error.message); }++ if (got.bytes !== declared) { drop(); return s3error(req, res, 'IncompleteBody', `You did not provide the number of bytes specified by the Content-Length HTTP header (${declared}; ${got.bytes} received)`); }++ const bad = integrityProblem(req, parsed, got);++ if (bad) { drop(); return s3error(req, res, bad[0], bad[1]); }++++ const mime = String(req.headers['content-type'] || 'application/octet-stream').split(';')[0].trim().toLowerCase().slice(0, 127) || 'application/octet-stream';++ const md = { filename: key.split('/').pop().slice(0, 200), md5: got.md5.toString('hex'), s3: true, created_by: akey.id };++ if (Object.keys(meta).length) md.s3_meta = meta;++ for (const [h, f] of [['content-disposition', 'content_disposition'], ['content-encoding', 'content_encoding'], ['cache-control', 'cache_control']]) {++ if (req.headers[h] && String(req.headers[h]).length <= 512) md[f] = String(req.headers[h]);++ }++ let outcome;++ try {++ const events = require('../events');++ outcome = await db.getDb().tx(async () => {++ const previous = await db.get(`SELECT * FROM media_objects WHERE zone_id = ? AND object_key = ? AND lifecycle_status = 'ready'`, [zone.id, key]);++ const q = await namespaces.checkQuota(req.appRow, zone.namespace, { bytes: got.bytes, objects: 1, excludeId: previous ? previous.id : null });++ if (q) return { error: ['QuotaExceeded', q.detail] };++ const fresh = await zones.byId(zone.id);++ if (!fresh || fresh.state !== 'active') return { error: ['NoSuchBucket', 'The specified bucket does not exist'] };++ if (previous) {++ if (await model.isHeld(previous.id)) return { error: ['AccessDenied', 'The object under this key is under a retention hold and cannot be replaced'] };++ const pmd = model.parseJson(previous.metadata, {});++ pmd.pre_delete_status = previous.lifecycle_status;++ pmd.retention_until = new Date(Date.now() + config.objects.retentionDays * 864e5).toISOString();++ pmd.deleted_by = `${akey.id} (overwrite)`;++ await db.run(`UPDATE media_objects SET lifecycle_status = 'deleted', deleted_at = ov_now(), metadata = ?, updated_at = ov_now() WHERE id = ?`,++ [JSON.stringify(pmd), previous.id]);++ await namespaces.settle(previous.id);++ }++ await namespaces.ensure(req.appRow, db.rootNamespace(req.appRow));++ const id = await model.createObject({++ app_id: req.appId, namespace: zone.namespace, kind: 'file', owner_subject: null, owner_app: req.appId, owner_user_id: null,++ visibility: 'private', lifecycle_status: 'uploading', mime_type: mime, size_bytes: got.bytes, metadata: md,++ });++ const obj = await model.getObject(id);++ const dest = model.objectFilePath(obj);++ fs.mkdirSync(path.dirname(dest), { recursive: true });++ fs.renameSync(tmp, dest);++ await db.run(`UPDATE media_objects SET lifecycle_status = 'ready', content_hash = ?, canonical_provider = 'local', canonical_key = ?,++ zone_id = ?, object_key = ?, updated_at = ov_now() WHERE id = ?`, [got.sha256, dest, zone.id, key, id]);++ await model.upsertLocation(id, { provider: 'local', key: dest, state: 'present', size_bytes: got.bytes, checksum: got.sha256, verified: true });++ const ready = await model.getObject(id);++ await require('../objects/invariant').record(ready);++ await events.record('media.object.uploaded', req.appId, await model.objectPublic(ready));++ await events.recordObjectChanges();++ return { obj: ready };++ });++ require('../events').kick();++ } catch (err) {++ drop();++ console.error('[S3] put error:', err.message);++ return s3error(req, res, 'InternalError', 'We encountered an internal error. Please try again.');++ }++ if (outcome.error) { drop(); return s3error(req, res, outcome.error[0], outcome.error[1]); }++ await namespaces.reconcileChain(req.appRow, zone.namespace).catch(() => {});++ baseHeaders(res);++ res.set('ETag', etagOf(outcome.obj));++ for (const k of ['x-amz-checksum-crc32', 'x-amz-checksum-sha256']) if (req.headers[k] || got.trailers[k]) res.set(k, req.headers[k] || got.trailers[k]);++ res.status(200).end();++}++++async function deleteOne(req, zone, key) {++ const obj = await findObject(zone, req.appId, key);++ if (!obj) return { ok: true }; // S3 answers 204 for a key that is not there++ if (await model.isHeld(obj.id)) return { error: ['AccessDenied', 'The object is under a retention hold'] };++ try {++ await model.softDelete(obj, { by: `${req.s3.key.id}` });++ } catch (err) {++ if (err instanceof model.HeldError) return { error: ['AccessDenied', 'The object is under a retention hold'] };++ throw err;++ }++ await namespaces.reconcileChain(req.appRow, obj.namespace).catch(() => {});++ return { ok: true };++}++++// ── Listing ──────────────────────────────────────────────────++++async function listObjects(req, res, zone, { v2 }) {++ const q = req.query;++ const prefix = typeof q.prefix === 'string' ? q.prefix : '';++ const delimiter = typeof q.delimiter === 'string' && q.delimiter ? q.delimiter : '';++ const maxKeys = Math.min(MAX_LIST, Math.max(0, Number.isFinite(Number(q['max-keys'])) ? Number(q['max-keys']) : MAX_LIST));++ let after = '';++ if (v2) {++ if (typeof q['continuation-token'] === 'string' && q['continuation-token']) {++ try { after = Buffer.from(q['continuation-token'], 'base64url').toString('utf8'); } catch { return s3error(req, res, 'InvalidArgument', 'The continuation token provided is incorrect'); }++ } else if (typeof q['start-after'] === 'string') after = q['start-after'];++ } else if (typeof q.marker === 'string') after = q.marker;++ const encodeUrl = q['encoding-type'] === 'url';++ const enc = (s) => (encodeUrl ? sigv4.awsEncode(s, true) : s);++++ const scope = zones.objectScope(zone, req.appId);++ const ek = effectiveKeySql(zone);++ const named = zone.is_default ? '' : ' AND o.zone_id = ?';++ const like = `${prefix.replace(/[\\%_]/g, (c) => `\\${c}`)}%`;++ const page = (cursor, limit) => db.all(`SELECT * FROM (SELECT o.*, ${ek} AS s3_key FROM media_objects o WHERE ${scope.where}${named}++ AND o.lifecycle_status = 'ready') x WHERE x.s3_key > ? AND x.s3_key LIKE ? ESCAPE '\\' ORDER BY x.s3_key LIMIT ?`,++ [...scope.args, ...(zone.is_default ? [] : [zone.id]), cursor, like, limit]);++ // Keys come in byte order. A delimiter folds every key below a common prefix into that prefix, so the cursor jumps++ // past the whole prefix (its last possible key) once the prefix is listed. A marker that is a common prefix itself++ // (a v1 NextMarker) is skipped the same way.++ const SKIP = '\u{10FFFF}';++ let cursor = delimiter && after && after.endsWith(delimiter) ? `${after}${SKIP}` : after;++ const contents = [], prefixes = [];++ let truncated = false;++ const full = () => contents.length + prefixes.length >= maxKeys;++ for (let pass = 0; pass < 100 && !full(); pass++) {++ const rows = await page(cursor, MAX_LIST);++ for (const r of rows) {++ if (full()) break;++ if (prefixes.length && r.s3_key.startsWith(prefixes[prefixes.length - 1])) continue;++ const i = delimiter ? r.s3_key.indexOf(delimiter, prefix.length) : -1;++ if (i >= 0) {++ const cp = r.s3_key.slice(0, i + delimiter.length);++ prefixes.push(cp);++ cursor = `${cp}${SKIP}`;++ } else {++ contents.push(r);++ cursor = r.s3_key;++ }++ }++ if (rows.length < MAX_LIST) break;++ }++ if (full()) truncated = (await page(cursor, 1)).length > 0;++ const last = cursor;++ const items = contents.map((o) => `<Contents><Key>${esc(enc(o.s3_key))}</Key><LastModified>${isoOf(lastModified(o))}</LastModified>`++ + `<ETag>${esc(etagOf(o))}</ETag><Size>${Number(o.size_bytes) || 0}</Size><StorageClass>STANDARD</StorageClass></Contents>`).join('');++ const cps = prefixes.map((p) => `<CommonPrefixes><Prefix>${esc(enc(p))}</Prefix></CommonPrefixes>`).join('');++ const common = `<Name>${esc(zone.name)}</Name><Prefix>${esc(enc(prefix))}</Prefix>${delimiter ? `<Delimiter>${esc(enc(delimiter))}</Delimiter>` : ''}`++ + `<MaxKeys>${maxKeys}</MaxKeys>${encodeUrl ? '<EncodingType>url</EncodingType>' : ''}<IsTruncated>${truncated}</IsTruncated>`;++ if (v2) {++ const token = typeof q['continuation-token'] === 'string' ? `<ContinuationToken>${esc(q['continuation-token'])}</ContinuationToken>` : '';++ const next = truncated ? `<NextContinuationToken>${Buffer.from(last).toString('base64url')}</NextContinuationToken>` : '';++ const sa = typeof q['start-after'] === 'string' ? `<StartAfter>${esc(enc(q['start-after']))}</StartAfter>` : '';++ return xml(res, 200, `<ListBucketResult xmlns="${XMLNS}">${common}<KeyCount>${contents.length + prefixes.length}</KeyCount>${token}${next}${sa}${items}${cps}</ListBucketResult>`);++ }++ const marker = `<Marker>${esc(enc(after))}</Marker>`;++ const lastListed = prefixes.length && (!contents.length || prefixes[prefixes.length - 1] > contents[contents.length - 1].s3_key)++ ? prefixes[prefixes.length - 1] : (contents.length ? contents[contents.length - 1].s3_key : '');++ const nextMarker = truncated && delimiter ? `<NextMarker>${esc(enc(lastListed))}</NextMarker>` : '';++ return xml(res, 200, `<ListBucketResult xmlns="${XMLNS}">${common}${marker}${nextMarker}${items}${cps}</ListBucketResult>`);++}++++async function listBuckets(req, res) {++ const { key } = req.s3;++ if (!mayRead(key)) return s3error(req, res, 'AccessDenied', 'This access key cannot read');++ const rows = (await zones.listActive(key.project_id, key.env)).filter((z) => !key.zones.length || key.zones.includes(z.id));++ const buckets = rows.map((z) => {++ const rec = model.parseJson(z.record, {});++ return `<Bucket><Name>${esc(z.name)}</Name><CreationDate>${esc(rec.created_at || new Date(Number(z.updated_at)).toISOString())}</CreationDate></Bucket>`;++ }).join('');++ xml(res, 200, `<ListAllMyBucketsResult xmlns="${XMLNS}"><Owner><ID>${esc(key.project_id)}</ID><DisplayName>${esc(key.project_id)}</DisplayName></Owner><Buckets>${buckets}</Buckets></ListAllMyBucketsResult>`);++}++++/** POST /<zone>?delete: the keys of a <Delete> document, each through DeleteObject's path. */++async function deleteObjects(req, res, zone) {++ if (!mayWrite(req.s3.key)) return s3error(req, res, 'AccessDenied', 'This access key is read-only');++ const chunks = [];++ let n = 0;++ try {++ const body = sigv4.isStreaming(req.s3.parsed) ? req.pipe(sigv4.chunkDecoder(req.s3.parsed)) : req;++ for await (const c of body) { n += c.length; if (n > 2 * 1024 * 1024) return s3error(req, res, 'MalformedXML', 'The delete request is too large'); chunks.push(c); }++ } catch (err) { return s3error(req, res, err.code && STATUS[err.code] ? err.code : 'IncompleteBody', err.message); }++ const raw = Buffer.concat(chunks);++ if (/^[0-9a-f]{64}$/.test(req.s3.parsed.payloadHash) && crypto.createHash('sha256').update(raw).digest('hex') !== req.s3.parsed.payloadHash) {++ return s3error(req, res, 'XAmzContentSHA256Mismatch', 'The provided x-amz-content-sha256 header does not match what was computed');++ }++ const text = raw.toString('utf8');++ const objectsXml = text.match(/<Object>[\s\S]*?<\/Object>/g) || [];++ if (!/<Delete[\s>]/.test(text) || objectsXml.length > MAX_LIST) return s3error(req, res, 'MalformedXML', 'The XML you provided was not well-formed or did not validate');++ const quiet = /<Quiet>\s*true\s*<\/Quiet>/i.test(text);++ const unxml = (s) => s.replace(/</g, '<').replace(/>/g, '>').replace(/"/g, '"').replace(/'/g, "'").replace(/&#(\d+);/g, (_, d) => String.fromCodePoint(Number(d))).replace(/&/g, '&');++ let out = '';++ for (const o of objectsXml) {++ const m = /<Key>([\s\S]*?)<\/Key>/.exec(o);++ const key = m ? unxml(m[1]) : '';++ if (!key || Buffer.byteLength(key) > MAX_KEY_BYTES) { out += `<Error><Key>${esc(key)}</Key><Code>InvalidArgument</Code><Message>Invalid key</Message></Error>`; continue; }++ const r = await deleteOne(req, zone, key);++ if (r.error) out += `<Error><Key>${esc(key)}</Key><Code>${r.error[0]}</Code><Message>${esc(r.error[1])}</Message></Error>`;++ else if (!quiet) out += `<Deleted><Key>${esc(key)}</Key></Deleted>`;++ }++ xml(res, 200, `<DeleteResult xmlns="${XMLNS}">${out}</DeleteResult>`);++}++++// ── Dispatch ─────────────────────────────────────────────────++++const router = express.Router();++router.use(authenticate);++++router.use(async (req, res) => {++ try {++ const p = sigv4.signedPath(req);++ const slash = p.indexOf('/', 1);++ const bucket = decodeURIComponent(slash < 0 ? p.slice(1) : p.slice(1, slash));++ const rest = slash < 0 ? '' : p.slice(slash + 1);++ const q = req.query;++ if (!bucket) {++ if (req.method === 'GET') return await listBuckets(req, res);++ return s3error(req, res, 'MethodNotAllowed', 'The specified method is not allowed against this resource');++ }++ if (!rest) {++ if (req.method === 'PUT' && !Object.keys(q).length) {++ // CreateBucket: zones are made through OpenVibe.Zone; a client that probes with it gets its zone back.++ const z = await zones.byName(req.s3.key.project_id, req.s3.key.env, bucket);++ if (z && z.state === 'active' && (!req.s3.key.zones.length || req.s3.key.zones.includes(z.id))) { req.resume(); baseHeaders(res); return res.status(200).end(); }++ req.resume();++ return s3error(req, res, 'NotImplemented', 'Create zones in OpenVibe.Zone (https://openvibe.zone): a zone is a bucket');++ }++ const zone = await zoneFor(req, res, bucket);++ if (!zone) return;++ if (req.method === 'HEAD') { if (!mayRead(req.s3.key)) return s3error(req, res, 'AccessDenied', ''); baseHeaders(res); res.set('x-amz-bucket-region', 'us-east-1'); return res.status(200).end(); }++ if (req.method === 'GET' && q.location !== undefined) return xml(res, 200, `<LocationConstraint xmlns="${XMLNS}"/>`);++ if (req.method === 'POST' && q.delete !== undefined) return await deleteObjects(req, res, zone);++ if (req.method === 'GET' && !['acl', 'policy', 'versioning', 'cors', 'lifecycle', 'tagging', 'uploads', 'versions', 'website', 'encryption', 'logging', 'notification', 'replication', 'object-lock', 'ownershipControls', 'publicAccessBlock', 'accelerate', 'requestPayment'].some((k) => q[k] !== undefined)) {++ if (!mayRead(req.s3.key)) return s3error(req, res, 'AccessDenied', 'This access key cannot read');++ return await listObjects(req, res, zone, { v2: q['list-type'] === '2' });++ }++ req.resume();++ return s3error(req, res, 'NotImplemented', 'A header or query you provided implies functionality that is not implemented');++ }++ const key = keyOf(rest);++ if (key == null) { req.resume(); return s3error(req, res, 'KeyTooLongError', `Keys are 1 to ${MAX_KEY_BYTES} bytes`); }++ const zone = await zoneFor(req, res, bucket);++ if (!zone) { req.resume(); return; }++ if (q.uploads !== undefined || q.uploadId !== undefined || q.partNumber !== undefined) {++ req.resume();++ return s3error(req, res, 'NotImplemented', 'Multipart uploads are not supported yet: send objects up to the single-request limit in one PUT');++ }++ if (['acl', 'tagging', 'retention', 'legal-hold', 'torrent', 'restore', 'attributes', 'versionId'].some((k) => q[k] !== undefined)) {++ req.resume();++ return s3error(req, res, 'NotImplemented', 'A header or query you provided implies functionality that is not implemented');++ }++ if (req.method === 'PUT') return await putObject(req, res, zone, key);++ if (req.method === 'GET' || req.method === 'HEAD') {++ if (!mayRead(req.s3.key)) return s3error(req, res, 'AccessDenied', 'This access key cannot read');++ const obj = await findObject(zone, req.appId, key);++ if (!obj) return s3error(req, res, 'NoSuchKey', 'The specified key does not exist.', `<Key>${esc(key)}</Key>`);++ return await sendObject(req, res, obj);++ }++ if (req.method === 'DELETE') {++ if (!mayWrite(req.s3.key)) return s3error(req, res, 'AccessDenied', 'This access key is read-only');++ const r = await deleteOne(req, zone, key);++ if (r.error) return s3error(req, res, r.error[0], r.error[1]);++ baseHeaders(res);++ return res.status(204).end();++ }++ req.resume();++ return s3error(req, res, 'MethodNotAllowed', 'The specified method is not allowed against this resource');++ } catch (err) {++ console.error('[S3] error:', err.message);++ if (!res.headersSent) return s3error(req, res, 'InternalError', 'We encountered an internal error. Please try again.');++ res.destroy();++ }++});++++/** Is this request for the S3 surface? Its Host is one of MEDIA_S3_HOSTS (default s3.openvibe.media). */++function isS3Host(req) {++ const hosts = String(process.env.MEDIA_S3_HOSTS || 's3.openvibe.media').split(',').map((h) => h.trim().toLowerCase()).filter(Boolean);++ const host = String(req.headers.host || '').toLowerCase().replace(/:\d+$/, '');++ return hosts.includes(host);++}++++module.exports = router;++module.exports.isS3Host = isS3Host;++module.exports.parseRange = parseRange;++module.exports.DERIVED_KEY = DERIVED_KEY;nnew file mode 100644@@@ -0,0 +1,290 @@++'use strict';++/**++ * AWS Signature Version 4 for Media's S3 surface (ADR-031): the server half. Media holds each access key's secret++ * (server/s3/keys.js) and recomputes the signature a client made, the way S3 does.++ *++ * parse(req) → { kind: 'header'|'presigned', accessKeyId, date, amzDate, region, scope,++ * signedHeaders, signature, payloadHash, expires } | { error }++ * verify(req, parsed, secret, { now }) → null | { code, message } (the S3 error to answer)++ * chunkDecoder(parsed, secret) → a Transform that turns an aws-chunked body into its bytes, checking every chunk++ * signature (STREAMING-AWS4-HMAC-SHA256-PAYLOAD[-TRAILER]) and reading the trailers++ *++ * Canonical forms follow S3, not the generic SigV4 rules: the path is decoded and encoded again exactly once, with++ * '/' kept, and is never normalised (S3 keys may hold '..', '//' and spaces). Header-signed requests must be within++ * 15 minutes of now; a presigned URL may live 7 days at most and may not be signed more than 15 minutes ahead.++ */++const crypto = require('crypto');++const { Transform } = require('stream');++++const ALGORITHM = 'AWS4-HMAC-SHA256';++const MAX_PRESIGN_S = 7 * 24 * 3600;++const SKEW_MS = 15 * 60 * 1000;++const EMPTY_SHA256 = 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855';++const STREAMING = new Set([++ 'STREAMING-AWS4-HMAC-SHA256-PAYLOAD',++ 'STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER',++ 'STREAMING-UNSIGNED-PAYLOAD-TRAILER',++]);++++const hmac = (key, data) => crypto.createHmac('sha256', key).update(data, 'utf8').digest();++const sha256hex = (data) => crypto.createHash('sha256').update(data).digest('hex');++++/** RFC 3986 encoding as AWS uses it: everything but A-Z a-z 0-9 - . _ ~ is %XX (upper case hex). */++function awsEncode(s, keepSlash = false) {++ let out = '';++ for (const b of Buffer.from(String(s), 'utf8')) {++ const c = String.fromCharCode(b);++ if (/[A-Za-z0-9\-._~]/.test(c) || (keepSlash && c === '/')) out += c;++ else out += `%${b.toString(16).toUpperCase().padStart(2, '0')}`;++ }++ return out;++}++++/** decodeURIComponent that leaves a malformed escape alone instead of throwing. */++function safeDecode(s) {++ try { return decodeURIComponent(s); } catch { return s; }++}++++/** The path as S3 canonicalises it: each segment decoded, then encoded once; '/' kept; nothing normalised. */++function canonicalPath(rawPath) {++ return String(rawPath || '/').split('/').map((seg) => awsEncode(safeDecode(seg))).join('/') || '/';++}++++/** [name, value] pairs of a raw query string, decoded ('+' is a space only in form encoding, never here). */++function queryPairs(rawQuery) {++ if (!rawQuery) return [];++ return rawQuery.split('&').filter(Boolean).map((p) => {++ const i = p.indexOf('=');++ return i < 0 ? [safeDecode(p), ''] : [safeDecode(p.slice(0, i)), safeDecode(p.slice(i + 1))];++ });++}++++function canonicalQuery(pairs) {++ return pairs.map(([k, v]) => [awsEncode(k), awsEncode(v)])++ .sort((a, b) => (a[0] < b[0] ? -1 : a[0] > b[0] ? 1 : a[1] < b[1] ? -1 : a[1] > b[1] ? 1 : 0))++ .map(([k, v]) => `${k}=${v}`).join('&');++}++++function headerValue(req, name) {++ const v = req.headers[name];++ if (v === undefined) return undefined;++ return (Array.isArray(v) ? v.join(',') : String(v)).trim().replace(/\s+/g, ' ');++}++++function splitUrl(req) {++ const raw = req.originalUrl || req.url || '/';++ const q = raw.indexOf('?');++ return { path: q < 0 ? raw : raw.slice(0, q), query: q < 0 ? '' : raw.slice(q + 1) };++}++++/** The path the client signed: the request path below the router's mount point (req.baseUrl), as sent. */++function signedPath(req) {++ const { path } = splitUrl(req);++ const base = req.baseUrl || '';++ return base && path.startsWith(base) ? path.slice(base.length) || '/' : path;++}++++function signingKey(secret, date, region, service) {++ return hmac(hmac(hmac(hmac(`AWS4${secret}`, date), region), service), 'aws4_request');++}++++/** Read the credentials a request carries (header or presigned query). Nothing is verified here. */++function parse(req) {++ const auth = headerValue(req, 'authorization');++ const { query } = splitUrl(req);++ const pairs = queryPairs(query);++ const q = Object.fromEntries(pairs);++ if (auth && auth.startsWith(`${ALGORITHM} `)) {++ const fields = Object.fromEntries(auth.slice(ALGORITHM.length + 1).split(',').map((f) => {++ const i = f.indexOf('=');++ return i < 0 ? [f.trim(), ''] : [f.slice(0, i).trim(), f.slice(i + 1).trim()];++ }));++ const cred = String(fields.Credential || '').split('/');++ if (cred.length !== 5 || cred[4] !== 'aws4_request' || !fields.SignedHeaders || !/^[0-9a-f]{64}$/.test(fields.Signature || '')) {++ return { error: { code: 'AuthorizationHeaderMalformed', message: 'The authorization header is malformed' } };++ }++ const amzDate = headerValue(req, 'x-amz-date');++ const payloadHash = headerValue(req, 'x-amz-content-sha256');++ if (!amzDate || !/^\d{8}T\d{6}Z$/.test(amzDate)) return { error: { code: 'AccessDenied', message: 'X-Amz-Date is required' } };++ if (!payloadHash) return { error: { code: 'InvalidRequest', message: 'Missing required header for this request: x-amz-content-sha256' } };++ return {++ kind: 'header', accessKeyId: cred[0], date: cred[1], region: cred[2], service: cred[3], amzDate,++ scope: cred.slice(1).join('/'), signedHeaders: fields.SignedHeaders.split(';'), signature: fields.Signature,++ payloadHash, pairs,++ };++ }++ if (q['X-Amz-Algorithm'] !== undefined) {++ if (q['X-Amz-Algorithm'] !== ALGORITHM) return { error: { code: 'AuthorizationQueryParametersError', message: 'X-Amz-Algorithm must be AWS4-HMAC-SHA256' } };++ const cred = String(q['X-Amz-Credential'] || '').split('/');++ const expires = Number(q['X-Amz-Expires']);++ if (cred.length !== 5 || cred[4] !== 'aws4_request' || !/^\d{8}T\d{6}Z$/.test(q['X-Amz-Date'] || '')++ || !q['X-Amz-SignedHeaders'] || !/^[0-9a-f]{64}$/.test(q['X-Amz-Signature'] || '')) {++ return { error: { code: 'AuthorizationQueryParametersError', message: 'The presigned URL is malformed' } };++ }++ if (!Number.isInteger(expires) || expires < 1 || expires > MAX_PRESIGN_S) {++ return { error: { code: 'AuthorizationQueryParametersError', message: 'X-Amz-Expires must be between 1 and 604800 seconds' } };++ }++ return {++ kind: 'presigned', accessKeyId: cred[0], date: cred[1], region: cred[2], service: cred[3], amzDate: q['X-Amz-Date'],++ scope: cred.slice(1).join('/'), signedHeaders: String(q['X-Amz-SignedHeaders']).split(';'), signature: q['X-Amz-Signature'],++ payloadHash: 'UNSIGNED-PAYLOAD', expires, pairs: pairs.filter(([k]) => k !== 'X-Amz-Signature'),++ };++ }++ return { error: null, anonymous: true };++}++++function amzDateMs(amzDate) {++ const m = /^(\d{4})(\d{2})(\d{2})T(\d{2})(\d{2})(\d{2})Z$/.exec(amzDate);++ return m ? Date.UTC(+m[1], +m[2] - 1, +m[3], +m[4], +m[5], +m[6]) : NaN;++}++++/** The signature the client should have sent, given the secret. */++function expectedSignature(req, parsed, secret) {++ const headers = parsed.signedHeaders.map((h) => {++ const v = h === 'host' ? (headerValue(req, 'host') || '') : headerValue(req, h);++ return `${h}:${v === undefined ? '' : v}\n`;++ }).join('');++ const canonical = [++ req.method,++ canonicalPath(signedPath(req)),++ canonicalQuery(parsed.pairs),++ headers,++ parsed.signedHeaders.join(';'),++ parsed.payloadHash,++ ].join('\n');++ const toSign = [ALGORITHM, parsed.amzDate, parsed.scope, sha256hex(canonical)].join('\n');++ const key = signingKey(secret, parsed.date, parsed.region, parsed.service);++ return { signature: crypto.createHmac('sha256', key).update(toSign, 'utf8').digest('hex'), key };++}++++function safeEqual(a, b) {++ const x = Buffer.from(String(a)), y = Buffer.from(String(b));++ return x.length === y.length && crypto.timingSafeEqual(x, y);++}++++/** null when the request is signed by `secret` and in time; else the S3 error ({ code, message }). */++function verify(req, parsed, secret, { now = Date.now() } = {}) {++ if (parsed.service !== 's3') return { code: 'AuthorizationHeaderMalformed', message: 'The credential scope must name the s3 service' };++ if (parsed.amzDate.slice(0, 8) !== parsed.date) return { code: 'AuthorizationHeaderMalformed', message: 'The credential date does not match X-Amz-Date' };++ if (!parsed.signedHeaders.includes('host')) return { code: 'AccessDenied', message: 'The host header must be signed' };++ const at = amzDateMs(parsed.amzDate);++ if (!Number.isFinite(at)) return { code: 'AccessDenied', message: 'X-Amz-Date is not a date' };++ if (parsed.kind === 'header' && Math.abs(now - at) > SKEW_MS) {++ return { code: 'RequestTimeTooSkewed', message: 'The difference between the request time and the current time is too large' };++ }++ if (parsed.kind === 'presigned') {++ if (at - now > SKEW_MS) return { code: 'AccessDenied', message: 'Request is not valid yet' };++ if (now > at + parsed.expires * 1000) return { code: 'AccessDenied', message: 'Request has expired' };++ }++ const { signature, key } = expectedSignature(req, parsed, secret);++ if (!safeEqual(signature, parsed.signature)) {++ return { code: 'SignatureDoesNotMatch', message: 'The request signature we calculated does not match the signature you provided' };++ }++ parsed.signingKey = key;++ return null;++}++++/** Is the payload an aws-chunked stream (x-amz-content-sha256 says so)? */++function isStreaming(parsed) {++ return STREAMING.has(parsed.payloadHash);++}++++/**++ * aws-chunked → bytes. Each chunk is `<hex size>[;chunk-signature=<sig>]\r\n<data>\r\n`, ending with a 0-size chunk,++ * then (for the -TRAILER forms) `name:value\r\n` trailers and, when signed, `x-amz-trailer-signature:<sig>\r\n`, and++ * a final `\r\n`. Signed chunks chain from the request's seed signature. The trailers are handed to `onTrailers` as++ * an object (x-amz-checksum-* values are checked by the caller). Errors are emitted with an S3 `code`.++ */++function chunkDecoder(parsed, { onTrailers = () => {} } = {}) {++ const signed = parsed.payloadHash.startsWith('STREAMING-AWS4-HMAC-SHA256-PAYLOAD');++ const scope = parsed.scope;++ let prev = parsed.signature;++ let buf = Buffer.alloc(0);++ let state = 'header'; // header → data → crlf → … → trailers → done++ let want = 0, chunkSig = null, chunkHash = null, trailerSigned = false;++ const trailers = {};++ const fail = (self, code, message) => { const e = new Error(message); e.code = code; self.destroy(e); };++ const sigOk = (stringToSign, sig) => safeEqual(crypto.createHmac('sha256', parsed.signingKey).update(stringToSign, 'utf8').digest('hex'), sig);++++ return new Transform({++ transform(chunk, _enc, cb) {++ buf = buf.length ? Buffer.concat([buf, chunk]) : chunk;++ try {++ for (;;) {++ if (state === 'header') {++ const i = buf.indexOf('\r\n');++ if (i < 0) { if (buf.length > 4096) return fail(this, 'IncompleteBody', 'Malformed chunk header'); break; }++ const line = buf.slice(0, i).toString('latin1');++ buf = buf.slice(i + 2);++ const [sizeHex, ...ext] = line.split(';');++ if (!/^[0-9a-fA-F]{1,16}$/.test(sizeHex)) return fail(this, 'IncompleteBody', 'Malformed chunk size');++ want = parseInt(sizeHex, 16);++ const sigExt = ext.find((e) => e.startsWith('chunk-signature='));++ chunkSig = sigExt ? sigExt.slice('chunk-signature='.length) : null;++ if (signed && !/^[0-9a-f]{64}$/.test(chunkSig || '')) return fail(this, 'SignatureDoesNotMatch', 'A chunk carries no signature');++ chunkHash = crypto.createHash('sha256');++ state = want === 0 ? 'last' : 'data';++ } else if (state === 'data') {++ if (!buf.length) break;++ const take = Math.min(want, buf.length);++ const part = buf.slice(0, take);++ buf = buf.slice(take);++ want -= take;++ chunkHash.update(part);++ this.push(part);++ if (want === 0) state = 'crlf';++ } else if (state === 'crlf') {++ if (buf.length < 2) break;++ if (buf[0] !== 13 || buf[1] !== 10) return fail(this, 'IncompleteBody', 'Malformed chunk terminator');++ buf = buf.slice(2);++ if (signed) {++ const s = ['AWS4-HMAC-SHA256-PAYLOAD', parsed.amzDate, scope, prev, EMPTY_SHA256, chunkHash.digest('hex')].join('\n');++ if (!sigOk(s, chunkSig)) return fail(this, 'SignatureDoesNotMatch', 'A chunk signature does not match');++ prev = chunkSig;++ }++ state = 'header';++ } else if (state === 'last') {++ if (signed) {++ const s = ['AWS4-HMAC-SHA256-PAYLOAD', parsed.amzDate, scope, prev, EMPTY_SHA256, EMPTY_SHA256].join('\n');++ if (!sigOk(s, chunkSig)) return fail(this, 'SignatureDoesNotMatch', 'The final chunk signature does not match');++ prev = chunkSig;++ }++ state = 'trailers';++ } else if (state === 'trailers') {++ const i = buf.indexOf('\r\n');++ if (i < 0) { if (buf.length > 8192) return fail(this, 'IncompleteBody', 'Malformed trailers'); break; }++ const line = buf.slice(0, i).toString('latin1');++ buf = buf.slice(i + 2);++ if (line === '') { state = 'done'; continue; }++ const c = line.indexOf(':');++ if (c < 0) return fail(this, 'IncompleteBody', 'Malformed trailer');++ const name = line.slice(0, c).trim().toLowerCase(), value = line.slice(c + 1).trim();++ if (name === 'x-amz-trailer-signature') {++ const body = Object.entries(trailers).map(([k, v]) => `${k}:${v}\n`).join('');++ const s = ['AWS4-HMAC-SHA256-TRAILER', parsed.amzDate, scope, prev, sha256hex(body)].join('\n');++ if (!sigOk(s, value)) return fail(this, 'SignatureDoesNotMatch', 'The trailer signature does not match');++ trailerSigned = true;++ } else trailers[name] = value;++ } else break;++ }++ cb();++ } catch (e) { cb(e); }++ },++ flush(cb) {++ // A body that ends right after the 0-size chunk (no trailers, no final CRLF) is complete too.++ if (state !== 'done' && !(state === 'trailers' && buf.length === 0)) {++ const e = new Error('The request body ended early'); e.code = 'IncompleteBody'; return cb(e);++ }++ if (signed && parsed.payloadHash.endsWith('-TRAILER') && Object.keys(trailers).length && !trailerSigned) {++ const e = new Error('Trailers are not signed'); e.code = 'SignatureDoesNotMatch'; return cb(e);++ }++ onTrailers(trailers);++ cb();++ },++ });++}++++module.exports = {++ parse, verify, isStreaming, chunkDecoder, awsEncode, canonicalPath, canonicalQuery, queryPairs, signingKey,++ expectedSignature, signedPath, EMPTY_SHA256, MAX_PRESIGN_S, SKEW_MS,++};nnew file mode 100644@@@ -0,0 +1,173 @@++'use strict';++/**++ * Media's projection of OpenVibe.Zone's zones (ADR-031 amendment 2026-10-02). Zone owns the record; it pushes every++ * change here with a higher revision (PUT /internal/zones/:id, server/s3/admin-routes.js), and Media uses the++ * projection to resolve S3 bucket names, enforce a zone's byte ceiling (its namespace row's quota_bytes, checked with++ * the environment's by objects/namespaces.checkQuota) and refuse requests to a zone that is going away.++ *++ * provision(record) → { applied, zone } (checkObjectZone first; a lower or equal revision changes nothing)++ * byName(projectId, env, name) / byId(id) / listActive(projectId, env)++ * status(zone) → { stored_bytes, objects, objects_remaining, held_objects, as_of }++ * sweepDeleting() → tombstones and purges what deleting zones still hold (the job runs it every minute)++ * objectScope(zone, app) → { where, args }: the SQL condition for the objects of a zone++ *++ * A zone's namespace is `<environment root>.<zone id>`; the default zone is the environment root itself and holds++ * every object of the environment that is not in a named zone (children included, legacy layouts unchanged).++ */++const { zones: zoneContracts } = require('openvibe-contracts');++const db = require('../db/database');++const config = require('../config');++const model = require('../objects/model');++++const BATCH = 200;++++class ZoneError extends Error {++ constructor(status, code, message) { super(message); this.status = status; this.code = code; }++}++++const rootOf = (projectId, env) => `app.${projectId}${env === 'sandbox' ? '.sandbox' : ''}`;++const tenantQuota = (env) => (env === 'sandbox' ? config.apps.sandboxQuotaMb : config.apps.projectQuotaMb) * 1024 * 1024;++++async function byId(id) {++ return await db.get('SELECT * FROM media_zones WHERE id = ?', [id]) || null;++}++++async function byName(projectId, env, name) {++ return await db.get('SELECT * FROM media_zones WHERE project_id = ? AND env = ? AND name = ?', [projectId, env, name]) || null;++}++++async function listActive(projectId, env) {++ return await db.all("SELECT * FROM media_zones WHERE project_id = ? AND env = ? AND state = 'active' ORDER BY name", [projectId, env]);++}++++/** The zone a namespace belongs to: the named zone it is, or the environment's default zone. */++async function forNamespace(app, namespace) {++ if (!app || !app.project_id) return null;++ const root = rootOf(app.project_id, app.env);++ const m = /^(.+)\.(zon_[0-9A-HJKMNP-TV-Z]{26})$/.exec(namespace);++ if (m && m[1] === root) return await byId(m[2]);++ if (namespace === root || namespace.startsWith(`${root}.`)) return await byName(app.project_id, app.env, 'default');++ return null;++}++++/** Apply one zone record from Zone. Idempotent by revision; throws ZoneError for a record Media must refuse. */++async function provision(record, { now = Date.now() } = {}) {++ const check = zoneContracts.checkObjectZone(record);++ if (!check.valid) throw new ZoneError(400, 'media.zone.invalid', JSON.stringify(check.errors).slice(0, 400));++ return await db.getDb().tx(async () => {++ const cur = await byId(record.id);++ if (cur && Number(cur.revision) >= record.revision) return { applied: false, zone: cur };++ if (cur && (cur.project_id !== record.project_id || cur.env !== record.env || cur.name !== record.name || cur.namespace !== record.media_namespace)) {++ throw new ZoneError(409, 'media.zone.immutable', 'a zone keeps its project, environment, name and namespace');++ }++ if (cur && cur.state === 'deleted' && record.state !== 'deleted') throw new ZoneError(409, 'media.zone.deleted', 'a deleted zone stays deleted');++ const named = await byName(record.project_id, record.env, record.name);++ if (named && named.id !== record.id) throw new ZoneError(409, 'media.zone.name_retired', `${record.name} belongs to ${named.id} in this environment`);++ const tenant = await db.ensureProjectTenant(record.project_id, record.env, tenantQuota(record.env));++ const root = db.rootNamespace(tenant);++ if (!record.default) {++ // The zone's own namespace row: its quota (a ceiling below the environment's) is enforced on every write.++ await db.run(`INSERT INTO media_namespaces (namespace, app_id, parent, owner, quota_bytes) VALUES (?, ?, ?, ?, ?)++ ON CONFLICT (namespace) DO UPDATE SET quota_bytes = excluded.quota_bytes, updated_at = ov_now()`,++ [record.media_namespace, tenant.app_id, root, db.namespaceOwner(tenant), record.quota_bytes ?? null]);++ }++ const deleting = ['deleting', 'deleted'].includes(record.state);++ await db.run(`INSERT INTO media_zones (id, project_id, env, name, is_default, namespace, quota_bytes, state, revision, record,++ deletion_requested_at, deletion_completed_at, updated_at)++ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, NULL, ?)++ ON CONFLICT (id) DO UPDATE SET quota_bytes = excluded.quota_bytes, state = excluded.state, revision = excluded.revision,++ record = excluded.record, updated_at = excluded.updated_at,++ deletion_requested_at = COALESCE(media_zones.deletion_requested_at, excluded.deletion_requested_at)`,++ [record.id, record.project_id, record.env, record.name, !!record.default, record.media_namespace, record.quota_bytes ?? null,++ record.state, record.revision, JSON.stringify(record), deleting ? now : null, now]);++ if (record.state === 'deleted') await require('./keys').dropZone(record.id, { now });++ return { applied: true, zone: await byId(record.id) };++ });++}++++/** SQL for the objects of a zone: its namespace, or (default zone) the environment's objects outside named zones. */++function objectScope(zone, tenantId) {++ if (!zone.is_default) return { where: 'o.namespace = ?', args: [zone.namespace] };++ return {++ where: "o.app_id = ? AND (o.namespace = ? OR (o.namespace LIKE ? AND o.namespace NOT LIKE ?))",++ args: [tenantId, zone.namespace, `${zone.namespace}.%`, `${zone.namespace}.zon\\_%`],++ };++}++++const tenantIdOf = (zone) => db.projectTenantId(zone.project_id, zone.env);++++async function status(zone, { now = Date.now() } = {}) {++ const s = objectScope(zone, tenantIdOf(zone));++ const live = await db.get(`SELECT COUNT(*) AS n, COALESCE(SUM(o.size_bytes), 0) AS b FROM media_objects o++ WHERE ${s.where} AND o.lifecycle_status = 'ready'`, s.args);++ const remaining = await db.get(`SELECT COUNT(*) AS n FROM media_objects o WHERE ${s.where}++ AND (o.lifecycle_status <> 'deleted' OR o.metadata NOT LIKE '%"purged_at"%')`, s.args);++ const held = await db.get(`SELECT COUNT(DISTINCT o.id) AS n FROM media_objects o JOIN media_holds h ON h.object_id = o.id++ WHERE ${s.where} AND h.released_at IS NULL AND (o.lifecycle_status <> 'deleted' OR o.metadata NOT LIKE '%"purged_at"%')`, s.args);++ return {++ stored_bytes: Number(live.b), objects: Number(live.n),++ objects_remaining: Number(remaining.n), held_objects: Number(held.n), as_of: new Date(now).toISOString(),++ };++}++++/** Is any object of the zone under a retention hold? (Zone asks before it starts a deletion: 409 zone.held.) */++async function hasHeld(zone) {++ return (await status(zone)).held_objects > 0;++}++++/**++ * One pass over the zones being deleted: every live object is tombstoned with no restore window (a held one too: its++ * hold delays only its purge), then every tombstoned object without a hold is purged, its copies erased. When nothing++ * is left the projection records the completion; Zone reads it from the status and marks the zone deleted.++ */++async function sweepDeleting({ now = Date.now() } = {}) {++ const zones = await db.all("SELECT * FROM media_zones WHERE state = 'deleting' AND deletion_completed_at IS NULL");++ let tombstoned = 0, purged = 0;++ for (const zone of zones) {++ const s = objectScope(zone, tenantIdOf(zone));++ const live = await db.all(`SELECT o.* FROM media_objects o WHERE ${s.where} AND o.lifecycle_status <> 'deleted' LIMIT ${BATCH}`, s.args);++ for (const obj of live) {++ const md = model.parseJson(obj.metadata, {});++ md.pre_delete_status = obj.lifecycle_status;++ md.retention_until = new Date(now).toISOString();++ md.deleted_by = `zone:${zone.id}`;++ md.no_restore = true;++ const events = require('../events');++ await db.getDb().tx(async () => {++ await db.run(`UPDATE media_objects SET lifecycle_status = 'deleted', deleted_at = ov_now(), metadata = ?, updated_at = ov_now()++ WHERE id = ? AND lifecycle_status <> 'deleted'`, [JSON.stringify(md), obj.id]);++ await require('../objects/namespaces').settle(obj.id);++ await events.recordObjectChanges();++ });++ events.kick();++ tombstoned++;++ }++ const dead = await db.all(`SELECT o.* FROM media_objects o WHERE ${s.where} AND o.lifecycle_status = 'deleted'++ AND o.metadata NOT LIKE '%"purged_at"%' LIMIT ${BATCH}`, s.args);++ for (const obj of dead) if (await model.purgeOne(obj)) purged++;++ const st = await status(zone, { now });++ if (st.objects_remaining === 0) {++ await db.run('UPDATE media_zones SET deletion_completed_at = ?, updated_at = ? WHERE id = ?', [now, now, zone.id]);++ }++ }++ return { zones: zones.length, tombstoned, purged };++}++++/** The zone.object-zone-usage@1-like snapshot Zone shows: the live objects and, while deleting, the progress. */++async function report(zone, opts) {++ const st = await status(zone, opts);++ return {++ zone_id: zone.id, revision: Number(zone.revision), state: zone.state,++ usage: { stored_bytes: st.stored_bytes, objects: st.objects, as_of: st.as_of },++ held_objects: st.held_objects, // Zone refuses to start a deletion while any object is held++ deletion: zone.deletion_requested_at == null ? null : {++ requested_at: new Date(Number(zone.deletion_requested_at)).toISOString(),++ objects_remaining: st.objects_remaining, held_objects: st.held_objects,++ completed_at: zone.deletion_completed_at == null ? null : new Date(Number(zone.deletion_completed_at)).toISOString(),++ },++ };++}++++module.exports = {++ provision, byId, byName, listActive, forNamespace, status, report, hasHeld, sweepDeleting, objectScope, rootOf, ZoneError,++};nnew file mode 100644@@@ -0,0 +1,71 @@++'use strict';++// server/s3/sigv4.js against the worked examples of the Amazon S3 API reference ("Signature Calculations for the++// Authorization Header", "Authenticating Requests: Using Query Parameters" and "Signature Calculations for the++// Authorization Header: Transferring Payload in Multiple Chunks"), with their published signatures. They pin the++// canonical forms independently of any SDK: the path, the query, the signed headers and the chunk signature chain.++const assert = require('assert');++const { Readable } = require('stream');++const sigv4 = require('../server/s3/sigv4');++++const SECRET = 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY';++const AT = Date.UTC(2013, 4, 24, 0, 0, 30);++const req = (method, url, headers) => ({ method, url, originalUrl: url, baseUrl: '', headers });++++(async () => {++ // GET Object with a Range, signed in the Authorization header.++ let r = req('GET', '/test.txt', {++ host: 'examplebucket.s3.amazonaws.com', range: 'bytes=0-9',++ 'x-amz-content-sha256': sigv4.EMPTY_SHA256, 'x-amz-date': '20130524T000000Z',++ authorization: 'AWS4-HMAC-SHA256 Credential=AKIAIOSFODNN7EXAMPLE/20130524/us-east-1/s3/aws4_request,SignedHeaders=host;range;x-amz-content-sha256;x-amz-date,Signature=f0e8bdb87c964420e857bd35b5d6ed310bd44f0170aba48dd91039c6036bdb41',++ });++ let p = sigv4.parse(r);++ assert.strictEqual(p.kind, 'header');++ assert.strictEqual(sigv4.verify(r, p, SECRET, { now: AT }), null, 'the published GET Object signature verifies');++ assert.strictEqual(sigv4.verify(r, p, 'x'.repeat(40), { now: AT }).code, 'SignatureDoesNotMatch');++ assert.strictEqual(sigv4.verify(r, p, SECRET, { now: AT + 20 * 60 * 1000 }).code, 'RequestTimeTooSkewed', 'header-signed requests are good for 15 minutes');++++ // A presigned GET (query parameters), valid 86400 seconds.++ const q = 'X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIOSFODNN7EXAMPLE%2F20130524%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20130524T000000Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host';++ r = req('GET', `/test.txt?${q}&X-Amz-Signature=aeeed9bbccd4d02ee5c0109b86d86835f995330da4c265957d157751f604d404`, { host: 'examplebucket.s3.amazonaws.com' });++ p = sigv4.parse(r);++ assert.deepStrictEqual([p.kind, p.expires, p.payloadHash], ['presigned', 86400, 'UNSIGNED-PAYLOAD']);++ assert.strictEqual(sigv4.verify(r, p, SECRET, { now: AT }), null, 'the published presigned URL verifies');++ assert.strictEqual(sigv4.verify(r, p, SECRET, { now: AT + 86401 * 1000 }).message, 'Request has expired');++ r = req('GET', `/test.txt?${q.replace('86400', '604801')}&X-Amz-Signature=${'0'.repeat(64)}`, { host: 'examplebucket.s3.amazonaws.com' });++ assert.strictEqual(sigv4.parse(r).error.code, 'AuthorizationQueryParametersError', 'a presigned URL lives 7 days at most');++++ // A chunked upload: 66560 bytes of 'a' in chunks of 65536, 1024 and 0, each signature chained to the last.++ const headers = {++ host: 's3.amazonaws.com', 'x-amz-date': '20130524T000000Z', 'x-amz-storage-class': 'REDUCED_REDUNDANCY',++ 'x-amz-content-sha256': 'STREAMING-AWS4-HMAC-SHA256-PAYLOAD', 'content-encoding': 'aws-chunked',++ 'x-amz-decoded-content-length': '66560', 'content-length': '66824',++ authorization: 'AWS4-HMAC-SHA256 Credential=AKIAIOSFODNN7EXAMPLE/20130524/us-east-1/s3/aws4_request,SignedHeaders=content-encoding;content-length;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-storage-class,Signature=4f232c4386841ef735655705268965c44a0e4690baa4adea153f7db9fa80a0a9',++ };++ r = req('PUT', '/examplebucket/chunkObject.txt', headers);++ p = sigv4.parse(r);++ assert.strictEqual(sigv4.verify(r, p, SECRET, { now: AT }), null, 'the published seed signature verifies');++ assert.ok(sigv4.isStreaming(p));++ const chunk = (n, sig) => Buffer.concat([Buffer.from(`${n.toString(16)};chunk-signature=${sig}\r\n`), Buffer.alloc(n, 'a'), Buffer.from('\r\n')]);++ const body = Buffer.concat([++ chunk(65536, 'ad80c730a21e5b8d04586a2213dd63b9a0e99e0e2307b0ade35a65485a288648'),++ chunk(1024, '0055627c9e194cb4542bae2aa5492e3c1575bbb81b612b7d234b86a503ef5497'),++ chunk(0, 'b6c6ea8a5354eaf15b3cb7646744f4275b71ea724fed81ceb9323e279d449df9'),++ ]);++ assert.strictEqual(body.length, 66824, 'the published body length');++ const decode = async (bytes, parsed) => {++ const out = [];++ for await (const c of Readable.from([bytes.subarray(0, 1000), bytes.subarray(1000, 40000), bytes.subarray(40000)]).pipe(sigv4.chunkDecoder(parsed))) out.push(c);++ return Buffer.concat(out);++ };++ const data = await decode(body, p);++ assert.ok(data.equals(Buffer.alloc(66560, 'a')), 'the chunks decode to the 66560 bytes, every signature checked');++ const tampered = Buffer.from(body);++ tampered[200] = 'b'.charCodeAt(0);++ await assert.rejects(decode(tampered, p), (e) => e.code === 'SignatureDoesNotMatch', 'a changed byte breaks its chunk signature');++ await assert.rejects(decode(body.subarray(0, 66000), p), (e) => e.code === 'IncompleteBody', 'a truncated body is refused');++++ // Canonical paths: decoded once, encoded once, '/' kept, nothing normalised.++ assert.strictEqual(sigv4.canonicalPath('/bucket/a%20b%2Bc/..//x~y'), '/bucket/a%20b%2Bc/..//x~y');++ assert.strictEqual(sigv4.canonicalPath('/bucket/caf%C3%A9'), '/bucket/caf%C3%A9');++ console.log('s3 sigv4: all checks passed');++})().catch((err) => { console.error(err); process.exit(1); });nnew file mode 100644@@@ -0,0 +1,279 @@++'use strict';++// ADR-031 and its zone amendment, end to end with the AWS SDK for JavaScript v3 against a local Media:++// zones provisioned by Zone's revisioned PUT, access keys issued on behalf of a project admin, every supported S3++// operation (PutObject plain and aws-chunked, GetObject with ranges, HeadObject, ListObjectsV2/v1 with prefixes,++// delimiters and pages, DeleteObject, DeleteObjects, ListBuckets, HeadBucket), presigned GET and PUT, two doors++// (S3 and the native API) on one object, isolation (zone-scoped and read-only keys, other environments), quotas,++// overwrite under a hold, the NotImplemented answers, and zone deletion with no name reuse.++const assert = require('assert');++const fs = require('fs');++const os = require('os');++const path = require('path');++const http = require('http');++const crypto = require('crypto');++const { Readable } = require('stream');++++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-s3-'));++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++ process.env.VOD_PATH = dir('vods');++ process.env.FILES_PATH = dir('files');++ process.env.OBJECTS_PATH = dir('objects');++ process.env.THUMBNAILS_PATH = dir('thumbnails');++ process.env.OV_NETWORK_URL = 'https://openvibe.network';++ process.env.MEDIA_PUBLIC_URL = 'https://media.test';++ process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';++ process.env.MEDIA_S3_KEY_SECRET = 'k'.repeat(40); // a test value: 32+ characters++ process.env.MEDIA_S3_HOSTS = '127.0.0.1';++++ const db = require('../server/db/database');++ const auth = require('../server/auth');++ const model = require('../server/objects/model');++ const zones = require('../server/s3/zones');++ const { serviceAuth, zones: zoneContracts, ids } = require('openvibe-contracts');++ const express = require('express');++ const S3 = require('@aws-sdk/client-s3');++ const { getSignedUrl } = require('@aws-sdk/s3-request-presigner');++++ const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });++ auth._setNetworkPublicKeyForTests(keys.publicKey);++ const now = Math.floor(Date.now() / 1000);++ const svc = (sub, cap, over = {}) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub, actor_type: 'service', aud: ['openvibe.media'], cap, iat: now, exp: now + 600, jti: `tok_${crypto.randomBytes(6).toString('hex')}`, ...over }, keys.privateKey);++ const ZONE_TOKEN = svc('svc:zone', ['media.zone.provision']);++ const SERVICES_TOKEN = svc('svc:services', ['media.access_key.manage']);++ const ADMIN = `usr_${ids.ulid()}`, STRANGER = `usr_${ids.ulid()}`;++ const PRJ = `prj_${ids.ulid()}`, OTHER = `prj_${ids.ulid()}`;++++ // Same mount order as server/index.js.++ const s3router = require('../server/s3/routes');++ const admin = require('../server/s3/admin-routes');++ const app = express();++ app.use((req, res, next) => (s3router.isS3Host(req) ? s3router(req, res, next) : next()));++ app.use('/internal/zones', admin.internalRouter());++ app.use('/api/v1/projects/:project/access-keys', admin.keysRouter({ projects: { isAdmin: async (p, s) => s === ADMIN && [PRJ, OTHER].includes(p) } }));++ app.put('/api/v2/:app/objects/:id/content', ...require('../server/objects/routes').contentHandlers);++ app.use(express.json());++ app.use('/api/v2/:app/objects', require('../server/objects/routes'));++ const server = http.createServer(app);++ await new Promise((r) => server.listen(0, '127.0.0.1', r));++ const port = server.address().port;++ // The internal doors are reached as localhost (a different Host than the S3 one) from loopback.++ const internal = async (method, p, { token, body } = {}) => {++ const res = await fetch(`http://localhost:${port}${p}`, { method, headers: { authorization: `Bearer ${token}`, 'content-type': 'application/json' }, body: body ? JSON.stringify(body) : undefined });++ return { status: res.status, body: await res.json().catch(() => null) };++ };++ const zoneRecord = (projectId, env, name, over = {}) => {++ const id = over.id || `zon_${ids.ulid()}`;++ const isDefault = name === 'default';++ const z = { id, project_id: projectId, env, name, default: isDefault, access: isDefault ? 'per-object' : 'private', state: 'active',++ quota_bytes: null, revision: 1, created_at: new Date().toISOString(), ...over };++ Object.assign(z, zoneContracts.identity(z));++ return z;++ };++ const provision = async (z) => internal('PUT', `/internal/zones/${z.id}`, { token: ZONE_TOKEN, body: z });++++ try {++ // ── Zones from Zone ──++ const def = zoneRecord(PRJ, 'production', 'default');++ const assets = zoneRecord(PRJ, 'production', 'assets');++ const logs = zoneRecord(PRJ, 'production', 'logs', { quota_bytes: 64 });++ const sandbox = zoneRecord(PRJ, 'sandbox', 'assets');++ const otherAssets = zoneRecord(OTHER, 'production', 'assets');++ for (const z of [def, assets, logs, sandbox, otherAssets]) {++ const r = await provision(z);++ assert.deepStrictEqual([r.status, r.body.applied, r.body.state], [200, true, 'active'], JSON.stringify(r.body));++ }++ let r = await provision(assets);++ assert.deepStrictEqual([r.status, r.body.applied], [200, false], 'a repeat (same revision) changes nothing');++ r = await provision({ ...zoneRecord(PRJ, 'production', 'assets'), revision: 1 });++ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.zone.name_retired'], 'a name belongs to one zone of its environment');++ r = await provision({ ...assets, media_namespace: `app.${OTHER}.${assets.id}`, revision: 2 });++ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.zone.invalid'], 'a record naming another project\'s namespace is refused (checkObjectZone)');++ r = await internal('PUT', `/internal/zones/${assets.id}`, { token: SERVICES_TOKEN, body: assets });++ assert.strictEqual(r.status, 403, 'only media.zone.provision provisions');++ console.log('✅ zones: provisioned by revision, names never reused, identity checked, capability required');++++ // ── Keys, on behalf of a project admin ──++ const issue = async (body, project = PRJ) => internal('POST', `/api/v1/projects/${project}/access-keys`, { token: SERVICES_TOKEN, body: { on_behalf_of: ADMIN, ...body } });++ r = await internal('POST', `/api/v1/projects/${PRJ}/access-keys`, { token: SERVICES_TOKEN, body: { on_behalf_of: STRANGER, env: 'production', name: 'x' } });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'media.access_key.not_admin'], 'only a project admin gets keys');++ const issued = await issue({ env: 'production', name: 'ci deploys' });++ assert.strictEqual(issued.status, 201, JSON.stringify(issued.body));++ const full = issued.body;++ assert.ok(/^ovk_/.test(full.key.id) && full.secret.length === 40 && !('secret' in (await internal('GET', `/api/v1/projects/${PRJ}/access-keys?on_behalf_of=${ADMIN}`, { token: SERVICES_TOKEN })).body.keys[0]), 'the secret is shown once');++ const scoped = (await issue({ env: 'production', name: 'logs only', zones: [logs.id] })).body;++ const reader = (await issue({ env: 'production', name: 'reader', permissions: 'read' })).body;++ const sbox = (await issue({ env: 'sandbox', name: 'sandbox' })).body;++ const other = (await issue({ env: 'production', name: 'other' }, OTHER)).body;++ r = await issue({ env: 'production', name: 'bad', zones: [sandbox.id] });++ assert.deepStrictEqual([r.status, r.body.code], [404, 'media.access_key.unknown_zone'], 'a key cannot name a zone of another environment');++ console.log('✅ access keys: issued for admins only, secret once, scopes checked');++++ const client = (k) => new S3.S3Client({ endpoint: `http://127.0.0.1:${port}`, region: 'us-east-1', forcePathStyle: true, credentials: { accessKeyId: k.key.id, secretAccessKey: k.secret } });++ const s3 = client(full);++ const body = Buffer.from('hello zones\n'.repeat(100));++ const md5 = crypto.createHash('md5').update(body).digest('hex');++++ // ── PutObject, GetObject, HeadObject ──++ let out = await s3.send(new S3.PutObjectCommand({ Bucket: 'assets', Key: 'site/index.html', Body: body, ContentType: 'text/html', Metadata: { mtime: '1700000000' } }));++ assert.strictEqual(out.ETag, `"${md5}"`, 'the ETag is the MD5 of the bytes, as S3 makes it');++ out = await s3.send(new S3.GetObjectCommand({ Bucket: 'assets', Key: 'site/index.html' }));++ assert.deepStrictEqual([Buffer.from(await out.Body.transformToByteArray()).equals(body), out.ContentType, out.Metadata.mtime], [true, 'text/html', '1700000000']);++ out = await s3.send(new S3.GetObjectCommand({ Bucket: 'assets', Key: 'site/index.html', Range: 'bytes=6-10' }));++ assert.deepStrictEqual([await out.Body.transformToString(), out.ContentRange], ['zones', `bytes 6-10/${body.length}`]);++ out = await s3.send(new S3.HeadObjectCommand({ Bucket: 'assets', Key: 'site/index.html' }));++ assert.deepStrictEqual([out.ContentLength, out.ETag], [body.length, `"${md5}"`]);++ // A streamed body of known length goes aws-chunked with a trailing checksum (the SDK's default).++ await s3.send(new S3.PutObjectCommand({ Bucket: 'assets', Key: 'stream/a b+c.bin', Body: Readable.from([body.subarray(0, 500), body.subarray(500)]), ContentLength: body.length }));++ out = await s3.send(new S3.GetObjectCommand({ Bucket: 'assets', Key: 'stream/a b+c.bin' }));++ assert.ok(Buffer.from(await out.Body.transformToByteArray()).equals(body), 'a streamed upload with a space and + in its key round-trips');++ const raw = await fetch(`http://127.0.0.1:${port}/assets/site/index.html`);++ assert.deepStrictEqual([raw.status, raw.headers.get('content-security-policy')], [403, "sandbox; default-src 'none'"], 'unsigned requests are refused, and nothing here runs in a browser');++ console.log('✅ PutObject (plain and aws-chunked), GetObject with a range, HeadObject, metadata, unsigned refused');++++ // ── Two doors: the object is an ordinary Media object ──++ const row = await db.get("SELECT * FROM media_objects WHERE zone_id = ? AND object_key = 'site/index.html' AND lifecycle_status = 'ready'", [assets.id]);++ assert.deepStrictEqual([row.namespace, row.app_id, row.content_hash, row.visibility], [assets.media_namespace, PRJ, crypto.createHash('sha256').update(body).digest('hex'), 'private']);++ const appTok = serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: `app:app_${ids.ulid()}`, actor_type: 'app', project_id: PRJ, env: 'production', aud: ['openvibe.media'], cap: ['media.object.upload', 'media.object.read', 'media.object.list', 'media.object.delete'], ns: [PRJ], iat: now, exp: now + 600, jti: 'tok_app1' }, keys.privateKey);++ const native = async (method, p, b) => { const res = await fetch(`http://localhost:${port}/api/v2/${PRJ}/objects${p}`, { method, headers: { authorization: `Bearer ${appTok}`, 'content-type': 'application/json' }, body: b ? JSON.stringify(b) : undefined }); return { status: res.status, body: await res.json().catch(() => null) }; };++ r = await native('GET', `/${row.id}`);++ assert.deepStrictEqual([r.status, r.body.content_hash], [200, row.content_hash], 'the native API reads an object written over S3');++ r = await native('POST', '', { namespace: assets.id, kind: 'file', size_bytes: 5, mime_type: 'text/plain', visibility: 'public' });++ assert.deepStrictEqual([r.status, r.body.code], [422, 'media.zone.private'], 'a named zone takes private objects only');++ r = await native('POST', '', { namespace: assets.id, kind: 'file', size_bytes: 5, mime_type: 'text/plain' });++ assert.strictEqual(r.status, 201, JSON.stringify(r.body));++ const nativeId = r.body.id;++ await fetch(`http://localhost:${port}/api/v2/${PRJ}/objects/${nativeId}/content`, { method: 'PUT', headers: { authorization: `Bearer ${appTok}`, 'content-type': 'text/plain' }, body: 'hello' });++ r = await native('POST', `/${nativeId}/complete`, {});++ assert.strictEqual(r.status, 200, JSON.stringify(r.body));++ out = await s3.send(new S3.GetObjectCommand({ Bucket: 'assets', Key: nativeId }));++ assert.strictEqual(await out.Body.transformToString(), 'hello', 'S3 reads a native object of the zone under its id');++ r = await native('POST', '', { namespace: `zon_${ids.ulid()}`, kind: 'file', size_bytes: 5, mime_type: 'text/plain' });++ assert.deepStrictEqual([r.status, r.body.code], [404, 'media.zone.unknown'], 'a zone that does not exist is not created by a write');++ // The default zone lists the environment's native objects under derived keys.++ r = await native('POST', '', { kind: 'file', size_bytes: 3, mime_type: 'text/plain', namespace: 'avatars' });++ const legacyId = r.body.id;++ await fetch(`http://localhost:${port}/api/v2/${PRJ}/objects/${legacyId}/content`, { method: 'PUT', headers: { authorization: `Bearer ${appTok}`, 'content-type': 'text/plain' }, body: 'old' });++ await native('POST', `/${legacyId}/complete`, {});++ out = await s3.send(new S3.ListObjectsV2Command({ Bucket: 'default' }));++ assert.deepStrictEqual((out.Contents || []).map((c) => c.Key), [`avatars/${legacyId}`], 'the default zone lists a native object under its namespace path and id, and nothing of a named zone');++ out = await s3.send(new S3.GetObjectCommand({ Bucket: 'default', Key: `avatars/${legacyId}` }));++ assert.strictEqual(await out.Body.transformToString(), 'old');++ await assert.rejects(s3.send(new S3.PutObjectCommand({ Bucket: 'default', Key: `avatars/${legacyId}`, Body: 'x' })), /InvalidArgument|med_/, 'a derived key cannot be written over S3');++ console.log('✅ two doors: one object for S3 and the native API; named zones private; the default zone lists native objects');++++ // ── Listing ──++ for (const k of ['photos/2024/a.jpg', 'photos/2024/b.jpg', 'photos/2025/c.jpg', 'photos/top.jpg', 'readme.txt']) {++ await s3.send(new S3.PutObjectCommand({ Bucket: 'assets', Key: k, Body: k }));++ }++ out = await s3.send(new S3.ListObjectsV2Command({ Bucket: 'assets', Prefix: 'photos/', Delimiter: '/' }));++ assert.deepStrictEqual([(out.Contents || []).map((c) => c.Key), (out.CommonPrefixes || []).map((c) => c.Prefix), out.KeyCount],++ [['photos/top.jpg'], ['photos/2024/', 'photos/2025/'], 3]);++ const seen = [];++ let token;++ do {++ out = await s3.send(new S3.ListObjectsV2Command({ Bucket: 'assets', MaxKeys: 2, ContinuationToken: token }));++ seen.push(...(out.Contents || []).map((c) => c.Key));++ token = out.NextContinuationToken;++ assert.strictEqual(!!token, !!out.IsTruncated);++ } while (token);++ assert.deepStrictEqual(seen, [nativeId, 'photos/2024/a.jpg', 'photos/2024/b.jpg', 'photos/2025/c.jpg', 'photos/top.jpg', 'readme.txt', 'site/index.html', 'stream/a b+c.bin'].sort((a, b) => (Buffer.compare(Buffer.from(a), Buffer.from(b)))), 'pages of two cover every key once, in byte order');++ const v1 = await s3.send(new S3.ListObjectsCommand({ Bucket: 'assets', Delimiter: '/', MaxKeys: 2 }));++ assert.deepStrictEqual([(v1.Contents || []).map((c) => c.Key), (v1.CommonPrefixes || []).map((c) => c.Prefix), v1.IsTruncated], [[nativeId], ['photos/'], true]);++ const v1b = await s3.send(new S3.ListObjectsCommand({ Bucket: 'assets', Delimiter: '/', Marker: v1.NextMarker }));++ assert.deepStrictEqual([(v1b.Contents || []).map((c) => c.Key), (v1b.CommonPrefixes || []).map((c) => c.Prefix)], [['readme.txt'], ['site/', 'stream/']], 'ListObjects v1 resumes after a common prefix marker');++ out = await s3.send(new S3.ListBucketsCommand({}));++ assert.deepStrictEqual(out.Buckets.map((b) => b.Name), ['assets', 'default', 'logs'], 'ListBuckets: the environment\'s zones, not the sandbox\'s or another project\'s');++ await s3.send(new S3.HeadBucketCommand({ Bucket: 'assets' }));++ await assert.rejects(s3.send(new S3.HeadBucketCommand({ Bucket: 'nope' })), (e) => e.$metadata.httpStatusCode === 404);++ console.log('✅ ListObjectsV2 (prefix, delimiter, pages), ListObjects v1, ListBuckets, HeadBucket');++++ // ── Presigned URLs ──++ const getUrl = await getSignedUrl(s3, new S3.GetObjectCommand({ Bucket: 'assets', Key: 'readme.txt' }), { expiresIn: 60 });++ let res = await fetch(getUrl);++ assert.deepStrictEqual([res.status, await res.text()], [200, 'readme.txt'], 'a presigned GET reads');++ const putUrl = await getSignedUrl(s3, new S3.PutObjectCommand({ Bucket: 'assets', Key: 'up/presigned.txt' }), { expiresIn: 60 });++ res = await fetch(putUrl, { method: 'PUT', body: 'via presigned put' });++ assert.strictEqual(res.status, 200, await res.text());++ out = await s3.send(new S3.GetObjectCommand({ Bucket: 'assets', Key: 'up/presigned.txt' }));++ assert.strictEqual(await out.Body.transformToString(), 'via presigned put');++ res = await fetch(getUrl.replace('readme.txt', 'site/index.html'));++ assert.strictEqual(res.status, 403, 'a presigned URL is for its own key only');++ const expired = await getSignedUrl(s3, new S3.GetObjectCommand({ Bucket: 'assets', Key: 'readme.txt' }), { expiresIn: 1, signingDate: new Date(Date.now() - 10_000) });++ res = await fetch(expired);++ assert.deepStrictEqual([res.status, /Request has expired/.test(await res.text())], [403, true], 'an expired URL is refused');++ console.log('✅ presigned GET and PUT; a URL reaches its own key only and expires');++++ // ── Isolation and permissions ──++ await assert.rejects(client(scoped).send(new S3.GetObjectCommand({ Bucket: 'assets', Key: 'readme.txt' })), /AccessDenied/, 'a zone-scoped key cannot reach another zone');++ await client(scoped).send(new S3.PutObjectCommand({ Bucket: 'logs', Key: 'a.log', Body: 'x' }));++ assert.deepStrictEqual((await client(scoped).send(new S3.ListBucketsCommand({}))).Buckets.map((b) => b.Name), ['logs']);++ await assert.rejects(client(reader).send(new S3.PutObjectCommand({ Bucket: 'assets', Key: 'nope', Body: 'x' })), /AccessDenied/, 'a read key cannot write');++ assert.strictEqual(await (await client(reader).send(new S3.GetObjectCommand({ Bucket: 'assets', Key: 'readme.txt' }))).Body.transformToString(), 'readme.txt');++ await client(sbox).send(new S3.PutObjectCommand({ Bucket: 'assets', Key: 'readme.txt', Body: 'sandbox copy' }));++ assert.strictEqual(await (await s3.send(new S3.GetObjectCommand({ Bucket: 'assets', Key: 'readme.txt' }))).Body.transformToString(), 'readme.txt', 'the sandbox\'s assets is another bucket');++ await assert.rejects(client(other).send(new S3.GetObjectCommand({ Bucket: 'assets', Key: 'readme.txt' })), /NoSuchKey/, 'another project\'s assets is another bucket');++ const forged = new S3.S3Client({ endpoint: `http://127.0.0.1:${port}`, region: 'us-east-1', forcePathStyle: true, credentials: { accessKeyId: full.key.id, secretAccessKey: 'x'.repeat(40) } });++ await assert.rejects(forged.send(new S3.ListBucketsCommand({})), /SignatureDoesNotMatch/);++ console.log('✅ isolation: zone-scoped, read-only, sandbox and other-project keys each stay in their place; forged signatures refused');++++ // ── Quotas, overwrite, holds, NotImplemented ──++ await assert.rejects(client(scoped).send(new S3.PutObjectCommand({ Bucket: 'logs', Key: 'big.log', Body: Buffer.alloc(100) })), /QuotaExceeded/, 'the zone\'s quota_bytes is enforced');++ await s3.send(new S3.PutObjectCommand({ Bucket: 'assets', Key: 'readme.txt', Body: 'v2' }));++ assert.strictEqual(await (await s3.send(new S3.GetObjectCommand({ Bucket: 'assets', Key: 'readme.txt' }))).Body.transformToString(), 'v2', 'an overwrite serves the new bytes');++ const versions = await db.all("SELECT lifecycle_status FROM media_objects WHERE zone_id = ? AND object_key = 'readme.txt' ORDER BY created_at", [assets.id]);++ assert.deepStrictEqual(versions.map((v) => v.lifecycle_status).sort(), ['deleted', 'ready'], 'the previous object went through the deletion path');++ const current = await db.get("SELECT * FROM media_objects WHERE zone_id = ? AND object_key = 'readme.txt' AND lifecycle_status = 'ready'", [assets.id]);++ await model.placeHold({ object_id: current.id, kind: 'evidence', reason: 'test' });++ await assert.rejects(s3.send(new S3.PutObjectCommand({ Bucket: 'assets', Key: 'readme.txt', Body: 'v3' })), /AccessDenied/, 'an overwrite of a held object is refused');++ await assert.rejects(s3.send(new S3.DeleteObjectCommand({ Bucket: 'assets', Key: 'readme.txt' })), /AccessDenied/, 'so is its deletion');++ await assert.rejects(s3.send(new S3.CreateMultipartUploadCommand({ Bucket: 'assets', Key: 'mp' })), /NotImplemented/);++ await assert.rejects(s3.send(new S3.CopyObjectCommand({ Bucket: 'assets', Key: 'copy', CopySource: 'assets/readme.txt' })), /NotImplemented/);++ await assert.rejects(s3.send(new S3.DeleteBucketCommand({ Bucket: 'assets' })), /NotImplemented|MethodNotAllowed/);++ console.log('✅ quotas, overwrite through the deletion path, holds refuse overwrite and delete, NotImplemented answers');++++ // ── Deleting objects ──++ await s3.send(new S3.DeleteObjectCommand({ Bucket: 'assets', Key: 'site/index.html' }));++ await assert.rejects(s3.send(new S3.GetObjectCommand({ Bucket: 'assets', Key: 'site/index.html' })), /NoSuchKey/);++ r = await native('GET', `/${row.id}`);++ assert.strictEqual(r.body.lifecycle_status, 'deleted', 'deleted over S3, deleted for the native API too');++ out = await s3.send(new S3.DeleteObjectsCommand({ Bucket: 'assets', Delete: { Objects: [{ Key: 'photos/2024/a.jpg' }, { Key: 'photos/2024/b.jpg' }, { Key: 'missing' }] } }));++ assert.deepStrictEqual((out.Deleted || []).map((d) => d.Key).sort(), ['missing', 'photos/2024/a.jpg', 'photos/2024/b.jpg']);++ out = await s3.send(new S3.ListObjectsV2Command({ Bucket: 'assets', Prefix: 'photos/2024/' }));++ assert.strictEqual(out.KeyCount, 0);++ console.log('✅ DeleteObject and DeleteObjects: both doors answer not found');++++ // ── Zone deletion: at once NoSuchBucket, objects tombstoned and purged, name retired, keys dropped ──++ await s3.send(new S3.PutObjectCommand({ Bucket: 'logs', Key: 'b.log', Body: 'y' }));++ const logsRow = await db.get("SELECT * FROM media_objects WHERE zone_id = ? AND object_key = 'b.log'", [logs.id]);++ const presignedLog = await getSignedUrl(s3, new S3.GetObjectCommand({ Bucket: 'logs', Key: 'b.log' }), { expiresIn: 600 });++ r = await provision({ ...logs, state: 'deleting', revision: 2, deletion: { requested_at: new Date().toISOString(), objects_remaining: 2, held_objects: 0, completed_at: null } });++ assert.deepStrictEqual([r.status, r.body.state], [200, 'deleting']);++ await assert.rejects(s3.send(new S3.GetObjectCommand({ Bucket: 'logs', Key: 'b.log' })), /NoSuchBucket/, 'a deleting zone answers NoSuchBucket at once');++ assert.strictEqual((await fetch(presignedLog)).status, 404, 'and so do its presigned URLs');++ const swept = await zones.sweepDeleting();++ assert.ok(swept.tombstoned >= 2 && swept.purged >= 2, JSON.stringify(swept));++ r = await internal('GET', `/internal/zones/${logs.id}`, { token: ZONE_TOKEN });++ assert.deepStrictEqual([r.body.deletion.objects_remaining, r.body.deletion.held_objects, !!r.body.deletion.completed_at], [0, 0, true], JSON.stringify(r.body));++ assert.ok(!fs.existsSync(model.objectFilePath(logsRow)), 'its bytes are gone');++ assert.strictEqual(await model.restore(await model.getObject(logsRow.id)), null, 'a zone deletion is not restorable');++ r = await provision({ ...logs, state: 'deleted', revision: 3, deletion: { requested_at: new Date().toISOString(), objects_remaining: 0, held_objects: 0, completed_at: new Date().toISOString() } });++ assert.strictEqual(r.body.state, 'deleted');++ r = await provision(zoneRecord(PRJ, 'production', 'logs'));++ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.zone.name_retired'], 'the name is retired, never given to a new zone');++ assert.strictEqual((await db.get('SELECT revoked_at FROM media_access_keys WHERE id = ?', [scoped.key.id])).revoked_at != null, true, 'a key scoped only to the deleted zone is revoked');++ assert.strictEqual((await fetch(presignedLog)).status, 404, 'a URL signed for the old zone never reaches anything again');++ console.log('✅ zone deletion: NoSuchBucket at once, tombstoned and purged, no restore, name retired, scoped keys revoked');++++ // ── Revocation ──++ r = await internal('DELETE', `/api/v1/projects/${PRJ}/access-keys/${reader.key.id}?on_behalf_of=${ADMIN}`, { token: SERVICES_TOKEN });++ assert.ok(r.body.revoked_at);++ await assert.rejects(client(reader).send(new S3.ListBucketsCommand({})), /InvalidAccessKeyId/, 'a revoked key is refused at once');++ console.log('✅ revocation is immediate');++ } finally {++ server.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ }++ console.log('s3 surface and zones: all checks passed');++ process.exit(0);++})().catch((err) => { console.error(err); process.exit(1); });@@@ -37,6 +37,7 @@ const crypto = require('crypto');MEDIA_B2_APP_KEY: S('b2-app-key'),MEDIA_R2_ACCESS_KEY_ID: S('r2-access-key-id'),MEDIA_R2_SECRET_ACCESS_KEY: S('r2-secret-access-key'),++ MEDIA_S3_KEY_SECRET: S('s3-key-secret'), // encrypts the S3 access keys' secrets (server/s3/keys.js)};const sentinels = new Map([...Object.entries(SECRET_ENV),@@@ -35,6 +35,7 @@ const https = require('https');'server/public/dev-data.js': 'the app\'s internal URL (APP_INTERNAL_URLS), ids and encoded names in the path','server/jobs/pack.js': 'a presigned GET on B2/R2 (operator-configured buckets) for a segment\'s durable copy','server/objects/routes.js': 'a ranged GET on a presigned B2/R2 URL (operator-configured buckets) for a packed segment\'s chunk',++ 'server/s3/routes.js': 'a ranged GET on a presigned B2/R2 URL (operator-configured buckets) for an S3 GetObject whose bytes are not on local disk','server/jobs/previews.js': 'a (ranged) GET on a presigned B2/R2 URL (operator-configured buckets) for a timeline segment, to cut a sprite frame','server/drill.js': 'restore drill: only this server itself','server/me/client.js': 'browser script, same-origin',
@@ -126,6 +126,12 @@ MEDIA_CAPABILITY_GATE=report
@@ -126,6 +126,12 @@ MEDIA_CAPABILITY_GATE=report
#MEDIA_UPLOAD_TOKEN_TTL_S=3600 # default presigned upload URL lifetime
#MEDIA_UPLOAD_TOKEN_TTL_S=3600 # default presigned upload URL lifetime
#MEDIA_MULTIPART_MAX_MB=20480
#MEDIA_MULTIPART_MAX_MB=20480
#MEDIA_MULTIPART_PART_MB=64 # bounds: MEDIA_MULTIPART_MIN_PART_MB=5, MEDIA_MULTIPART_MAX_PART_MB=256
#MEDIA_MULTIPART_PART_MB=64 # bounds: MEDIA_MULTIPART_MIN_PART_MB=5, MEDIA_MULTIPART_MAX_PART_MB=256
+
+# ── S3 surface and zones (ADR-031; server/s3/, docs/s3.md) ──
+# MEDIA_S3_KEY_SECRET (32+ characters) encrypts the S3 access keys' secrets; unset = the S3 surface answers 503 and no
+# key can be issued. Changing it makes every issued key unusable. MEDIA_S3_HOSTS: the Host names the surface answers.
+#MEDIA_S3_KEY_SECRET=
+#MEDIA_S3_HOSTS=s3.openvibe.media,s3.openvibe.network
#MEDIA_MULTIPART_TTL_HOURS=24
#MEDIA_MULTIPART_TTL_HOURS=24
#MEDIA_UPLOAD_MIN_FREE_MB=10240 # free disk kept in reserve by multipart uploads and split/remux jobs
#MEDIA_UPLOAD_MIN_FREE_MB=10240 # free disk kept in reserve by multipart uploads and split/remux jobs
@@ -60,6 +60,9 @@ key ([Tenancy & auth](#tenancy--auth)).
@@ -60,6 +60,9 @@ key ([Tenancy & auth](#tenancy--auth)).
|---|---|---|---|
|---|---|---|---|
| `media.resource.read` | first-party | `GET /api/v1/resources`, `GET /api/v1/resources/:ovrn` | the authority resource index ([Resource index](#resource-index-apiv1resources-first-party)). Active in openvibe-contracts since 0.108.0; the route enforces it (a Network service token holding it, loopback, audience `openvibe.media`) |
| `media.resource.read` | first-party | `GET /api/v1/resources`, `GET /api/v1/resources/:ovrn` | the authority resource index ([Resource index](#resource-index-apiv1resources-first-party)). Active in openvibe-contracts since 0.108.0; the route enforces it (a Network service token holding it, loopback, audience `openvibe.media`) |
| `media.resource.control` | first-party | `POST /api/v1/resources/control` | `delete` a person's (or their project's) object: the soft delete, restorable until its retention ends ([Resource index](#resource-index-apiv1resources-first-party)). Active in openvibe-contracts since 0.169.0 |
| `media.resource.control` | first-party | `POST /api/v1/resources/control` | `delete` a person's (or their project's) object: the soft delete, restorable until its retention ends ([Resource index](#resource-index-apiv1resources-first-party)). Active in openvibe-contracts since 0.169.0 |
+| `media.object.s3` | public (an access key) | the S3 surface at `s3.openvibe.media` ([docs/s3.md](docs/s3.md)) | PutObject, GetObject, HeadObject, DeleteObject(s), ListObjects(V2), ListBuckets, HeadBucket, presigned GET/PUT; a bucket is a zone |
+| `media.access_key.manage` | first-party | `POST/GET/DELETE /api/v1/projects/:project/access-keys` | S3 access keys for a project admin, the secret shown once (Zone's console, Services) |
+| `media.zone.provision` | first-party (Zone) | `PUT/GET /internal/zones/:id` | OpenVibe.Zone's zone records, applied by revision; usage and deletion progress |
Called elsewhere, as the service principal `media` (the OAuth client `media`):
Called elsewhere, as the service principal `media` (the OAuth client `media`):
@@ -126,7 +129,7 @@ server/
@@ -126,7 +129,7 @@ server/
storage.orphans.scan (monthly storage orphan report, service-wide, report only),
storage.orphans.scan (monthly storage orphan report, service-wide, report only),
storage.move.cleanup (the alert for placement moves whose delete keeps failing, service-wide)
storage.move.cleanup (the alert for placement moves whose delete keeps failing, service-wide)
client-ip.js trust proxy = loopback; req.ip is the only client address
client-ip.js trust proxy = loopback; req.ip is the only client address
-(openvibe-shared v4.1.3, openvibe-contracts v0.169.0, openvibe-sdk v0.44.0: pinned release tarballs, installed by npm)
+(openvibe-shared v4.1.3, openvibe-contracts v0.173.0, openvibe-sdk v0.44.0: pinned release tarballs, installed by npm)
scripts/smoke-test.sh end-to-end smoke test (boots a temp instance)
scripts/smoke-test.sh end-to-end smoke test (boots a temp instance)
scripts/reconcile-objects.js / object-invariant.js / no-good-copy-report.js object-model operator tools
scripts/reconcile-objects.js / object-invariant.js / no-good-copy-report.js object-model operator tools
scripts/media-jobs.js list jobs, run the size-invariant scan (dry run by default), approve/cancel proposals
scripts/media-jobs.js list jobs, run the size-invariant scan (dry run by default), approve/cancel proposals
@@ -586,6 +589,12 @@ without a signature is a 404 like any other private one). A native object in the
@@ -586,6 +589,12 @@ without a signature is a 404 like any other private one). A native object in the
R2 popularity cache redirects to its verified R2 copy; a GET of a native object
R2 popularity cache redirects to its verified R2 copy; a GET of a native object
counts its viewer for the day, identifying no one (see Storage tiering).
counts its viewer for the day, identifying no one (see Storage tiering).
+## S3 surface and zones
+
+Media answers a subset of S3 at `https://s3.openvibe.media` (ADR-031). Each bucket is a zone of a project environment, and zones are managed by OpenVibe.Zone. Media issues and verifies the `ovk_` access keys, and every object stays an ordinary Media object.
+
+See [docs/s3.md](docs/s3.md) for the operations, the rules and the tests.
+
## Object model and API v2
## Object model and API v2
Every stored blob is a **media object** (`med_<ULID>`, `media_objects`) with one
Every stored blob is a **media object** (`med_<ULID>`, `media_objects`) with one
new file mode 100644
new file mode 100644
@@ -0,0 +1,82 @@
@@ -0,0 +1,82 @@
+# OpenVibe.Media's S3 surface (ADR-031): path style, https://s3.openvibe.media/<zone name>/<key>. Node answers every
+# request whose Host is an S3 host (MEDIA_S3_HOSTS, server/s3/routes.js). proxy_pass has no URI part on purpose: nginx
+# then passes the request line exactly as the client sent it, and SigV4 signs that path (a URI part would make nginx
+# decode and re-encode it, and every key with an escaped character would fail its signature).
+# Certificate: the openvibe.media wildcard. DNS: s3.openvibe.media needs its record (the owner's Cloudflare action);
+# until it has one, s3.openvibe.network (the openvibe.network wildcard name and certificate) is the same surface, and
+# OpenVibe.Zone names it (ZONE_S3_ENDPOINT). MEDIA_S3_HOSTS lists both. Cloudflare's proxy carries at most 100 MB a
+# request on this plan, so one object over S3 is at most 100 MB until the name is DNS-only.
+limit_req_zone $binary_remote_addr zone=media_s3:10m rate=50r/s;
+
+server {
+ listen 80;
+ listen [::]:80;
+ server_name s3.openvibe.media s3.openvibe.network;
+ location /.well-known/acme-challenge/ { root /var/www/html; }
+ location / { return 301 https://$host$request_uri; }
+}
+server {
+ listen 443 ssl http2;
+ listen [::]:443 ssl http2;
+ server_name s3.openvibe.media;
+ ssl_certificate /etc/letsencrypt/live/openvibe.media/fullchain.pem;
+ ssl_certificate_key /etc/letsencrypt/live/openvibe.media/privkey.pem;
+
+ # One PutObject carries at most MEDIA_OBJECT_MAX_MB (256 MB) of data; aws-chunked framing adds a little.
+ client_max_body_size 300m;
+ proxy_read_timeout 600s;
+ proxy_send_timeout 600s;
+ # Signed URLs carry long query strings.
+ large_client_header_buffers 8 16k;
+
+ # No path is held back: every path is a zone and a key. (Node's own /metrics answers proxied callers 404, so Zone
+ # refuses the zone name `metrics`.)
+ location / {
+ limit_req zone=media_s3 burst=200 nodelay;
+ proxy_pass http://127.0.0.1:4100;
+ proxy_http_version 1.1;
+ proxy_set_header Host $host;
+ proxy_set_header X-Media-Shield-Host "";
+ proxy_set_header X-Media-Shield-Failed "";
+ proxy_set_header X-Forwarded-Host "";
+ proxy_set_header X-Real-IP $remote_addr;
+ proxy_set_header X-Forwarded-For $remote_addr;
+ proxy_set_header CF-Connecting-IP $remote_addr;
+ proxy_set_header X-Forwarded-Proto $scheme;
+ proxy_request_buffering off;
+ proxy_buffering off;
+ }
+}
+
+server {
+ listen 443 ssl http2;
+ listen [::]:443 ssl http2;
+ server_name s3.openvibe.network;
+ ssl_certificate /etc/letsencrypt/live/openvibe.network/fullchain.pem;
+ ssl_certificate_key /etc/letsencrypt/live/openvibe.network/privkey.pem;
+
+ # One PutObject carries at most MEDIA_OBJECT_MAX_MB (256 MB) of data; aws-chunked framing adds a little.
+ client_max_body_size 300m;
+ proxy_read_timeout 600s;
+ proxy_send_timeout 600s;
+ # Signed URLs carry long query strings.
+ large_client_header_buffers 8 16k;
+
+ # No path is held back: every path is a zone and a key. (Node's own /metrics answers proxied callers 404, so Zone
+ # refuses the zone name `metrics`.)
+ location / {
+ limit_req zone=media_s3 burst=200 nodelay;
+ proxy_pass http://127.0.0.1:4100;
+ proxy_http_version 1.1;
+ proxy_set_header Host $host;
+ proxy_set_header X-Media-Shield-Host "";
+ proxy_set_header X-Media-Shield-Failed "";
+ proxy_set_header X-Forwarded-Host "";
+ proxy_set_header X-Real-IP $remote_addr;
+ proxy_set_header X-Forwarded-For $remote_addr;
+ proxy_set_header CF-Connecting-IP $remote_addr;
+ proxy_set_header X-Forwarded-Proto $scheme;
+ proxy_request_buffering off;
+ proxy_buffering off;
+ }
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,54 @@
@@ -0,0 +1,54 @@
+# The S3 surface and zones
+
+OpenVibe.Media answers a subset of the Amazon S3 API at `https://s3.openvibe.media`, path style
+(`/<bucket>/<key>`). The design is ADR-031 in OpenVibe.Contracts, with its two amendments. A **bucket is a zone**:
+a named object zone of one project environment. Zones are made and deleted only through OpenVibe.Zone
+(https://openvibe.zone).
+
+## Pieces
+
+| Piece | File | What it does |
+|---|---|---|
+| Signatures | `server/s3/sigv4.js` | Verifies SigV4 (Authorization header and presigned query) and decodes aws-chunked bodies, checking each chunk's signature and the trailers. |
+| Access keys | `server/s3/keys.js` | `ovk_` keys for one project environment: read, write or both, and optionally limited to some zones. Secrets are encrypted with `MEDIA_S3_KEY_SECRET` and shown once. |
+| Zones | `server/s3/zones.js` | Media's copy of Zone's records. It resolves bucket names, gives each zone's namespace its quota, and runs zone deletion. |
+| S3 routes | `server/s3/routes.js` | The operations below, dispatched by `Host` (`MEDIA_S3_HOSTS`). |
+| Control doors | `server/s3/admin-routes.js` | `PUT/GET /internal/zones/:id` (Zone only, `media.zone.provision`) and `/api/v1/projects/:project/access-keys` (`media.access_key.manage`, on behalf of a project admin). |
+
+## Operations
+
+**Supported:**
+- `PutObject`, in one request, including aws-chunked bodies. `Content-MD5`, `x-amz-content-sha256` and the `x-amz-checksum-*` headers and trailers are checked.
+- `GetObject` (with ranges, conditional headers and presigned response overrides) and `HeadObject`.
+- `DeleteObject` and `DeleteObjects`.
+- `ListObjectsV2` and `ListObjects` (prefix, delimiter and pages).
+- `ListBuckets`, `HeadBucket` and `GetBucketLocation`.
+- Presigned GET and PUT, valid for 7 days at most.
+
+**Not implemented:** multipart uploads, `CopyObject`, ACLs, policies, versioning, tagging and bucket deletion.
+`CreateBucket` answers 200 for a zone that already exists, so tools that probe this way keep working.
+
+## Rules
+
+- **Two doors, one object.** An object written over S3 is an ordinary Media object: a `med_` id in the zone's namespace (`app.<project>[.sandbox].<zone id>`) with `zone_id` and `object_key` set. The native API reads, lists and deletes it, and the reverse works too.
+- **The default zone.** It is the environment root. Over S3 it lists the environment's native objects under a derived key: the child namespace path, then the `med_` id. A key of that form cannot be written over S3.
+- **Overwrite.** The new bytes are stored first. Then, in one transaction under the tenant's quota lock, the previous object goes through Media's deletion path and the new one becomes ready. A held object is neither overwritten nor deleted (`AccessDenied`).
+- **Quotas.** A write is checked against the zone's `quota_bytes` and the environment's quota, before the bytes arrive and again after.
+- **Zone deletion.** When Zone pushes state `deleting`, the zone answers `NoSuchBucket` at once. The minute sweep (`zones.sweepDeleting`) tombstones every object with no restore window and purges every copy. The status reports `objects_remaining` and `held_objects` until nothing is left. Then Zone marks the zone `deleted`: its name is retired, and keys scoped only to it are revoked.
+- **No browser execution.** Every response carries `Content-Security-Policy: sandbox; default-src 'none'` and `nosniff`, whatever type the object declares.
+
+## Tests
+
+- `test/s3-zones.test.js` runs end to end with the AWS SDK for JavaScript v3: every operation, presigned URLs, isolation, quotas, holds, two doors and zone deletion.
+- `test/s3-sigv4.test.js` checks the worked examples of the S3 API reference, including the chunk signature chain.
+
+## Using it
+
+```sh
+aws configure set aws_access_key_id ovk_… # issued on openvibe.zone
+aws configure set aws_secret_access_key … # shown once
+aws configure set default.s3.multipart_threshold 256MB # no multipart yet: one request per object
+aws --endpoint-url https://s3.openvibe.media s3 ls s3://assets/
+rclone config create oz s3 provider Other endpoint https://s3.openvibe.media access_key_id ovk_… secret_access_key … \
+ upload_cutoff 256M no_check_bucket true
+```
new file mode 100644
new file mode 100644
@@ -0,0 +1,50 @@
@@ -0,0 +1,50 @@
+-- phase: expand
+-- Named object zones and the S3 surface (ADR-031 and its 2026-10-02 amendment, OpenVibe.Zone).
+--
+-- media_zones is Media's projection of the zone records OpenVibe.Zone owns: Zone pushes each change with a higher
+-- revision (PUT /internal/zones/:id) and Media resolves S3 bucket names, quotas and states from it. A name stays with
+-- its zone for ever within one project environment, deleted or not, so a signed request names exactly one zone id.
+CREATE TABLE media_zones (
+ id text COLLATE "C" PRIMARY KEY, -- zon_<ULID>
+ project_id text COLLATE "C" NOT NULL,
+ env text COLLATE "C" NOT NULL CHECK (env IN ('sandbox', 'production')),
+ name text COLLATE "C" NOT NULL,
+ is_default boolean NOT NULL DEFAULT false,
+ namespace text COLLATE "C" NOT NULL UNIQUE, -- the environment root, or <root>.<id>
+ quota_bytes bigint,
+ state text COLLATE "C" NOT NULL CHECK (state IN ('provisioning', 'active', 'deleting', 'deleted')),
+ revision bigint NOT NULL,
+ record jsonb NOT NULL, -- the zone.object-zone@1 Zone sent
+ deletion_requested_at bigint,
+ deletion_completed_at bigint,
+ updated_at bigint NOT NULL,
+ UNIQUE (project_id, env, name)
+);
+
+-- S3 access keys (ADR-031: bound to one project environment, read and/or write, optionally to some zones). Media
+-- verifies the SigV4 signature, so Media keeps the secret: encrypted with MEDIA_S3_KEY_SECRET, shown once at issue.
+CREATE TABLE media_access_keys (
+ id text COLLATE "C" PRIMARY KEY, -- ovk_<ULID>, the S3 access key id
+ project_id text COLLATE "C" NOT NULL,
+ env text COLLATE "C" NOT NULL CHECK (env IN ('sandbox', 'production')),
+ name text NOT NULL,
+ permissions text COLLATE "C" NOT NULL CHECK (permissions IN ('read', 'write', 'read-write')),
+ zones jsonb NOT NULL DEFAULT '[]', -- [] = every zone of the environment
+ secret_enc text COLLATE "C" NOT NULL, -- iv.tag.ciphertext (AES-256-GCM), base64url
+ created_by text COLLATE "C" NOT NULL, -- usr_… who issued it
+ created_at bigint NOT NULL,
+ last_used_at bigint,
+ revoked_at bigint,
+ revoked_by text COLLATE "C"
+);
+CREATE INDEX media_access_keys_project ON media_access_keys (project_id, env) WHERE revoked_at IS NULL;
+
+-- An object's zone and its S3 key. Objects written over S3 carry both; a native object's key is derived (its med_ id,
+-- below the child namespace path in the default zone) until the contract migration backfills every row.
+ALTER TABLE media_objects ADD COLUMN zone_id text COLLATE "C";
+ALTER TABLE media_objects ADD COLUMN object_key text COLLATE "C";
+-- One live object per key in a zone. An S3 write stores its bytes first and then, in one transaction under the
+-- tenant's quota lock, tombstones the previous object of the key and inserts the new one already ready.
+CREATE UNIQUE INDEX media_objects_zone_key_live ON media_objects (zone_id, object_key)
+ WHERE zone_id IS NOT NULL AND object_key IS NOT NULL AND lifecycle_status = 'ready';
+CREATE INDEX media_objects_namespace_key ON media_objects (namespace, object_key) WHERE lifecycle_status = 'ready';
@@ -16,7 +16,7 @@
@@ -16,7 +16,7 @@
"express": "^4.19.2",
"express": "^4.19.2",
"iovalkey": "^0.4.0",
"iovalkey": "^0.4.0",
"multer": "^1.4.5-lts.1",
"multer": "^1.4.5-lts.1",
- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.169.0",
+ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.173.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.44.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.44.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.1.3",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.1.3",
"pg": "^8.23.0",
"pg": "^8.23.0",
@@ -1847,9 +1847,9 @@
@@ -1847,9 +1847,9 @@
}
}
},
},
"node_modules/openvibe-contracts": {
"node_modules/openvibe-contracts": {
- "version": "0.169.0",
- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.169.0",
- "integrity": "sha512-Py2z8UaKGvLzG3atuIAnJKnRe74Te9cJRTaxqFq6m4fov6gaMXeS6rn7lH0tSIo0qBGD5trrT618ARXnklkp+A==",
+ "version": "0.173.0",
+ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.173.0",
+ "integrity": "sha512-6zHbU9zghIYDd3PPGtgMTsIgxrWwsoBHO/rvuLW1rghr6kRK6VwCmSMZRcjQbcPODPH1XrUAFuLg/T1K4nOCMQ==",
"license": "AGPL-3.0-only",
"license": "AGPL-3.0-only",
"dependencies": {
"dependencies": {
"ajv": "^8.17.1",
"ajv": "^8.17.1",
@@ -1,7 +1,7 @@
@@ -1,7 +1,7 @@
{
{
"name": "openvibe-media",
"name": "openvibe-media",
"version": "1.0.0",
"version": "1.0.0",
- "description": "OpenVibe.Media \u2014 multi-tenant media service (VOD ingest/recording, storage tiering, clips, pastes, files, thumbnails)",
+ "description": "OpenVibe.Media — multi-tenant media service (VOD ingest/recording, storage tiering, clips, pastes, files, thumbnails)",
"main": "server/index.js",
"main": "server/index.js",
"scripts": {
"scripts": {
"start": "node server/index.js",
"start": "node server/index.js",
@@ -20,7 +20,7 @@
@@ -20,7 +20,7 @@
"express": "^4.19.2",
"express": "^4.19.2",
"iovalkey": "^0.4.0",
"iovalkey": "^0.4.0",
"multer": "^1.4.5-lts.1",
"multer": "^1.4.5-lts.1",
- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.169.0",
+ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.173.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.44.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.44.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.1.3",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.1.3",
"pg": "^8.23.0",
"pg": "^8.23.0",
@@ -88,6 +88,14 @@ const ready = (async () => {
@@ -88,6 +88,14 @@ const ready = (async () => {
// Network's sign-out-everywhere cutoffs (server/revocations.js): signed, loopback-only. Before the JSON
// Network's sign-out-everywhere cutoffs (server/revocations.js): signed, loopback-only. Before the JSON
// parser, which would consume the raw body the signature is over.
// parser, which would consume the raw body the signature is over.
app.post('/internal/events', ...require('./revocations').handler());
app.post('/internal/events', ...require('./revocations').handler());
+ // The S3 surface (ADR-031, server/s3/routes.js) answers every request whose Host is an S3 host (MEDIA_S3_HOSTS,
+ // s3.openvibe.media): nginx's vhost passes the request line untouched, because the client signed that exact path.
+ // It reads raw bodies (PutObject, aws-chunked), so it sits ahead of the JSON parser. The zone and access-key doors
+ // carry their own parsers and service-token guards.
+ const s3 = require('./s3/routes');
+ app.use((req, res, next) => (s3.isS3Host(req) ? s3(req, res, next) : next()));
+ app.use('/internal/zones', require('./s3/admin-routes').internalRouter());
+ app.use('/api/v1/projects/:project/access-keys', require('./s3/admin-routes').keysRouter());
app.use(express.json({ limit: '2mb' }));
app.use(express.json({ limit: '2mb' }));
// IndexNow (openvibe-shared/indexnow): created once at boot from INDEXNOW_KEY. Unset → off, the
// IndexNow (openvibe-shared/indexnow): created once at boot from INDEXNOW_KEY. Unset → off, the
@@ -284,6 +292,10 @@ const ready = (async () => {
@@ -284,6 +292,10 @@ const ready = (async () => {
} catch { /* not linux */ }
} catch { /* not linux */ }
});
});
every(60 * 1000, () => recorder.checkDisk()); // disk guardian
every(60 * 1000, () => recorder.checkDisk()); // disk guardian
+ // Zones being deleted (ADR-031 amendment): their objects are tombstoned and purged in batches, every minute.
+ every(60 * 1000, async () => {
+ try { const r = await require('./s3/zones').sweepDeleting(); if (r.tombstoned || r.purged) console.log(`[Zones] ${r.zones} deleting zone(s): ${r.tombstoned} tombstoned, ${r.purged} purged`); } catch (err) { console.warn('[Zones] deletion sweep:', err.message); }
+ });
every(60 * 60 * 1000, () => thumbService.cleanupOldThumbnails()); // stale live thumbs
every(60 * 60 * 1000, () => thumbService.cleanupOldThumbnails()); // stale live thumbs
// Object model: purge native objects whose soft-delete retention has passed (held ones are kept).
// Object model: purge native objects whose soft-delete retention has passed (held ones are kept).
every(60 * 60 * 1000, async () => {
every(60 * 60 * 1000, async () => {
@@ -624,7 +624,7 @@ async function softDelete(obj, { by = null } = {}) {
@@ -624,7 +624,7 @@ async function softDelete(obj, { by = null } = {}) {
async function restore(obj) {
async function restore(obj) {
const md = parseJson(obj.metadata, {});
const md = parseJson(obj.metadata, {});
- if (md.purged_at) return null;
+ if (md.purged_at || md.no_restore) return null; // a zone deletion's tombstones are never restored
const back = md.pre_delete_status && md.pre_delete_status !== 'deleted' ? md.pre_delete_status : 'ready';
const back = md.pre_delete_status && md.pre_delete_status !== 'deleted' ? md.pre_delete_status : 'ready';
delete md.pre_delete_status; delete md.retention_until; delete md.deleted_by;
delete md.pre_delete_status; delete md.retention_until; delete md.deleted_by;
await db.run(`UPDATE media_objects SET lifecycle_status = ?, deleted_at = NULL, metadata = ?, updated_at = ov_now() WHERE id = ?`,
await db.run(`UPDATE media_objects SET lifecycle_status = ?, deleted_at = NULL, metadata = ?, updated_at = ov_now() WHERE id = ?`,
@@ -642,27 +642,34 @@ async function restore(obj) {
@@ -642,27 +642,34 @@ async function restore(obj) {
async function purgeExpired({ retentionDays = config.objects.retentionDays } = {}) {
async function purgeExpired({ retentionDays = config.objects.retentionDays } = {}) {
const rows = await db.all(`SELECT * FROM media_objects WHERE lifecycle_status = 'deleted' AND legacy_ref IS NULL
const rows = await db.all(`SELECT * FROM media_objects WHERE lifecycle_status = 'deleted' AND legacy_ref IS NULL
AND deleted_at <= datetime('now', ?)`, [`-${Math.max(0, retentionDays)} days`]);
AND deleted_at <= datetime('now', ?)`, [`-${Math.max(0, retentionDays)} days`]);
- const root = path.resolve(config.objects.path) + path.sep;
let purged = 0;
let purged = 0;
- for (const obj of rows) {
- const md = parseJson(obj.metadata, {});
- if (md.purged_at || await isHeld(obj.id)) continue;
- if ((await listLocations(obj.id)).some(l => l.provider === 'r2')) continue;
- for (const l of await listLocations(obj.id)) {
- if (l.provider === 'local' && path.resolve(l.key).startsWith(root)) { try { fs.unlinkSync(l.key); } catch { /* already gone */ } }
- if (l.provider === 'local') await db.run('DELETE FROM media_locations WHERE id = ?', [l.id]);
- }
- // Its CMAF segments (F3.1) go with the bytes. A durable copy whose delete failed keeps its row, so this purge
- // is not marked done and the next pass retries it rather than losing the key.
- const tl = await require('./timeline').removeObject(obj.id);
- if (tl.pending) continue;
- md.purged_at = new Date().toISOString();
- await db.run('UPDATE media_objects SET metadata = ?, updated_at = ov_now() WHERE id = ?', [JSON.stringify(md), obj.id]);
- purged++;
- }
+ for (const obj of rows) if (await purgeOne(obj)) purged++;
return purged;
return purged;
}
}
+/**
+ * Erase one tombstoned native object's bytes now (purgeExpired's step, and a zone deletion's, which has no retention
+ * period): its local copies, then its CMAF segments; marked purged only when every copy is gone. A held object, one
+ * still in the R2 cache (tiering demotes it first) and a purged one are left alone. true when it is purged now.
+ */
+async function purgeOne(obj) {
+ const md = parseJson(obj.metadata, {});
+ if (obj.lifecycle_status !== 'deleted' || obj.legacy_ref || md.purged_at || await isHeld(obj.id)) return false;
+ if ((await listLocations(obj.id)).some(l => l.provider === 'r2')) return false;
+ const root = path.resolve(config.objects.path) + path.sep;
+ for (const l of await listLocations(obj.id)) {
+ if (l.provider === 'local' && path.resolve(l.key).startsWith(root)) { try { fs.unlinkSync(l.key); } catch { /* already gone */ } }
+ if (l.provider === 'local') await db.run('DELETE FROM media_locations WHERE id = ?', [l.id]);
+ }
+ // Its CMAF segments (F3.1) go with the bytes. A durable copy whose delete failed keeps its row, so this purge
+ // is not marked done and the next pass retries it rather than losing the key.
+ const tl = await require('./timeline').removeObject(obj.id);
+ if (tl.pending) return false;
+ md.purged_at = new Date().toISOString();
+ await db.run('UPDATE media_objects SET metadata = ?, updated_at = ov_now() WHERE id = ?', [JSON.stringify(md), obj.id]);
+ return true;
+}
+
/**
/**
* Bytes counted against the tenant's quota: v1 files, ready native objects (in a developer project
* Bytes counted against the tenant's quota: v1 files, ready native objects (in a developer project
* also soft-deleted ones until purged) and the reservations of uploads in progress
* also soft-deleted ones until purged) and the reservations of uploads in progress
@@ -731,6 +738,6 @@ module.exports = {
@@ -731,6 +738,6 @@ module.exports = {
vodProjection, clipProjection, fileProjection, thumbnailProjection,
vodProjection, clipProjection, fileProjection, thumbnailProjection,
syncVod, syncClip, syncFile, sync, safeSync, withObject, withObjectOrRow, afterTierMove,
syncVod, syncClip, syncFile, sync, safeSync, withObject, withObjectOrRow, afterTierMove,
placeHold, releaseHold, holdPublic,
placeHold, releaseHold, holdPublic,
- objectFilePath, softDelete, restore, purgeExpired, usedBytes,
+ objectFilePath, softDelete, restore, purgeExpired, purgeOne, usedBytes,
legacyPublicUrl, objectPublic,
legacyPublicUrl, objectPublic,
};
};
@@ -31,6 +31,7 @@ const db = require('../db/database');
@@ -31,6 +31,7 @@ const db = require('../db/database');
const config = require('../config');
const config = require('../config');
const SEGMENT_RE = /^[a-z0-9][a-z0-9_-]{0,39}$/;
const SEGMENT_RE = /^[a-z0-9][a-z0-9_-]{0,39}$/;
+const ZONE_SEGMENT_RE = /^zon_[0-9A-HJKMNP-TV-Z]{26}$/;
const MAX_DEPTH = 3;
const MAX_DEPTH = 3;
const POLICY_KEYS = ['kinds', 'visibilities', 'max_object_bytes', 'strict_verbs'];
const POLICY_KEYS = ['kinds', 'visibilities', 'max_object_bytes', 'strict_verbs'];
@@ -100,6 +101,9 @@ function resolveName(app, requested) {
@@ -100,6 +101,9 @@ function resolveName(app, requested) {
if (s === root) return { namespace: root };
if (s === root) return { namespace: root };
const rel = s.startsWith(`${root}.`) ? s.slice(root.length + 1) : s;
const rel = s.startsWith(`${root}.`) ? s.slice(root.length + 1) : s;
const segs = rel.split('.');
const segs = rel.split('.');
+ // A named zone (OpenVibe.Zone) is the one child whose name is upper-case: <root>.zon_<ULID>, made only when Zone
+ // provisions it (server/s3/zones.js); objects routes refuse one that does not exist.
+ if (segs.length === 1 && ZONE_SEGMENT_RE.test(segs[0])) return { namespace: `${root}.${rel}`, zone: segs[0] };
if (segs.length > MAX_DEPTH || !segs.every(x => SEGMENT_RE.test(x)) || `${root}.${rel}`.length > 200) {
if (segs.length > MAX_DEPTH || !segs.every(x => SEGMENT_RE.test(x)) || `${root}.${rel}`.length > 200) {
return { error: `namespace must be ${root} or a name below it: up to ${MAX_DEPTH} segments of a-z, 0-9, _ and - (e.g. ${root}.avatars)` };
return { error: `namespace must be ${root} or a name below it: up to ${MAX_DEPTH} segments of a-z, 0-9, _ and - (e.g. ${root}.avatars)` };
}
}
@@ -319,6 +319,10 @@ router.post('/', upload, limits('media.object.upload', UPLOAD), async (req, res)
@@ -319,6 +319,10 @@ router.post('/', upload, limits('media.object.upload', UPLOAD), async (req, res)
const namespace = named.namespace;
const namespace = named.namespace;
const g = await namespaceGrant(req, 'write', namespace);
const g = await namespaceGrant(req, 'write', namespace);
if (!g.allowed) return problem(res, 403, g.code, g.reason);
if (!g.allowed) return problem(res, 403, g.code, g.reason);
+ // A named zone (ADR-031 amendment): it must exist and be active, and it holds private objects only.
+ const zone = named.zone ? await require('../s3/zones').byId(named.zone) : null;
+ if (named.zone && (!zone || zone.state !== 'active' || zone.namespace !== namespace)) return problem(res, 404, 'media.zone.unknown', `no active zone ${named.zone} in this environment`);
+ if (zone && visibility !== 'private') return problem(res, 422, 'media.zone.private', 'a named zone holds private objects only');
const policy = await namespaces.effectivePolicy(req.appId, namespace);
const policy = await namespaces.effectivePolicy(req.appId, namespace);
if (Array.isArray(policy.kinds) && !policy.kinds.includes(kind)) return problem(res, 422, 'media.namespace.policy_denied', `${namespace} takes kinds ${policy.kinds.join(', ') || '(none)'}`);
if (Array.isArray(policy.kinds) && !policy.kinds.includes(kind)) return problem(res, 422, 'media.namespace.policy_denied', `${namespace} takes kinds ${policy.kinds.join(', ') || '(none)'}`);
if (Array.isArray(policy.visibilities) && !policy.visibilities.includes(visibility)) return problem(res, 422, 'media.namespace.policy_denied', `${namespace} takes visibilities ${policy.visibilities.join(', ') || '(none)'}`);
if (Array.isArray(policy.visibilities) && !policy.visibilities.includes(visibility)) return problem(res, 422, 'media.namespace.policy_denied', `${namespace} takes visibilities ${policy.visibilities.join(', ') || '(none)'}`);
@@ -366,6 +370,8 @@ router.post('/', upload, limits('media.object.upload', UPLOAD), async (req, res)
@@ -366,6 +370,8 @@ router.post('/', upload, limits('media.object.upload', UPLOAD), async (req, res)
app_id: req.appId, namespace, kind, owner_subject: subject, owner_app: req.appId, owner_user_id: userId,
app_id: req.appId, namespace, kind, owner_subject: subject, owner_app: req.appId, owner_user_id: userId,
visibility, lifecycle_status: 'uploading', mime_type: mime, size_bytes: size, metadata,
visibility, lifecycle_status: 'uploading', mime_type: mime, size_bytes: size, metadata,
});
});
+ // In a named zone the object lists over S3 under its id until a key is given (only S3 writes give one).
+ if (zone) await db.run('UPDATE media_objects SET zone_id = ?, object_key = ? WHERE id = ?', [zone.id, newId, newId]);
await namespaces.reserve(await model.getObject(newId), size);
await namespaces.reserve(await model.getObject(newId), size);
return { id: newId };
return { id: newId };
});
});
new file mode 100644
new file mode 100644
@@ -0,0 +1,96 @@
@@ -0,0 +1,96 @@
+'use strict';
+/**
+ * The two control doors of the S3 surface, both loopback-only and Network-service-token-only (server/service-guard.js):
+ *
+ * OpenVibe.Zone → Media (capability media.zone.provision), mounted at /internal/zones:
+ * PUT /:id body zone.object-zone@1 → { applied, ...report } a zone change, applied when its revision is higher
+ * GET /:id → report: { zone_id, revision, state, usage, deletion } what Zone shows and polls while deleting
+ *
+ * OpenVibe.Services → Media (capability media.access_key.manage), mounted at /api/v1/projects/:project/access-keys,
+ * always on behalf of a person who is an admin or owner of the project (asked of Network, server/registry/projects.js):
+ * POST / { env, name, permissions, zones, on_behalf_of } → 201 { key: media.access-key@1, secret }
+ * GET /?on_behalf_of= → { keys: [media.access-key@1] }
+ * DELETE /:id?on_behalf_of= → 200 media.access-key@1 (revoked; S3 refuses it at once)
+ * The secret is in the 201 only; Media never shows it again.
+ */
+const express = require('express');
+const { http } = require('openvibe-contracts');
+const { guard } = require('../service-guard');
+const zones = require('./zones');
+const keys = require('./keys');
+
+const PROJECT_RE = /^prj_[0-9A-HJKMNP-TV-Z]{26}$/;
+const SUBJECT_RE = /^usr_[0-9A-HJKMNP-TV-Z]{26}$/;
+const problem = (res, status, code, detail) => http.sendProblem(res, status, code, { detail });
+
+function internalRouter() {
+ const r = express.Router();
+ r.use(guard('media.zone.provision'), express.json({ limit: '64kb' }));
+ r.put('/:id', async (req, res, next) => {
+ try {
+ res.set('Cache-Control', 'no-store');
+ if (!req.body || req.body.id !== req.params.id) return problem(res, 400, 'media.zone.invalid', 'the body is the zone named in the path');
+ const out = await zones.provision(req.body);
+ res.json({ applied: out.applied, ...await zones.report(out.zone) });
+ } catch (err) {
+ if (err instanceof zones.ZoneError) return problem(res, err.status, err.code, err.message);
+ next(err);
+ }
+ });
+ r.get('/:id', async (req, res, next) => {
+ try {
+ res.set('Cache-Control', 'no-store');
+ const zone = await zones.byId(req.params.id);
+ if (!zone) return problem(res, 404, 'media.zone.unknown', 'Media has no such zone');
+ res.json(await zones.report(zone));
+ } catch (err) { next(err); }
+ });
+ return r;
+}
+
+function keysRouter({ projects = require('../registry/projects').createProjects() } = {}) {
+ const r = express.Router({ mergeParams: true });
+ r.use(guard('media.access_key.manage'), express.json({ limit: '16kb' }));
+
+ // A first-party service acts, and only for a person who administers the project.
+ r.use(async (req, res, next) => {
+ try {
+ res.set('Cache-Control', 'no-store');
+ if (!String(req.principal.sub || '').startsWith('svc:')) return problem(res, 403, 'capability.denied', 'media.access_key.manage requires a service token');
+ if (!PROJECT_RE.test(req.params.project)) return problem(res, 404, 'media.project.unknown', 'no such project');
+ const person = String((req.body && req.body.on_behalf_of) || req.query.on_behalf_of || '');
+ if (!SUBJECT_RE.test(person)) return problem(res, 400, 'media.access_key.no_person', 'on_behalf_of names the person acting (usr_…)');
+ let admin;
+ try { admin = await projects.isAdmin(req.params.project, person); } catch (err) {
+ return problem(res, 503, 'media.network.unavailable', `project roles cannot be read: ${err.message}`);
+ }
+ if (!admin) return problem(res, 403, 'media.access_key.not_admin', 'only an admin or owner of the project manages its access keys');
+ req.person = person;
+ next();
+ } catch (err) { next(err); }
+ });
+
+ r.post('/', async (req, res, next) => {
+ try {
+ const b = req.body || {};
+ const out = await keys.issue({ projectId: req.params.project, env: b.env, name: b.name, permissions: b.permissions || 'read-write', zones: b.zones || [], createdBy: req.person });
+ res.status(201).json(out);
+ } catch (err) {
+ if (err instanceof keys.KeyError) return problem(res, err.status, err.code, err.message);
+ next(err);
+ }
+ });
+ r.get('/', async (req, res, next) => {
+ try { res.json({ keys: await keys.list(req.params.project) }); } catch (err) { next(err); }
+ });
+ r.delete('/:id', async (req, res, next) => {
+ try {
+ const out = await keys.revoke(req.params.project, req.params.id, req.person);
+ if (!out) return problem(res, 404, 'media.access_key.unknown', 'no such key in this project');
+ res.json(out);
+ } catch (err) { next(err); }
+ });
+ return r;
+}
+
+module.exports = { internalRouter, keysRouter };
new file mode 100644
new file mode 100644
@@ -0,0 +1,130 @@
@@ -0,0 +1,130 @@
+'use strict';
+/**
+ * S3 access keys (ADR-031): an `ovk_<ULID>` id and a secret, bound to one project environment, read and/or write,
+ * and optionally to some of its zones. Media verifies SigV4 itself (server/s3/sigv4.js), so Media keeps the secret,
+ * encrypted with MEDIA_S3_KEY_SECRET (AES-256-GCM); it is returned once, when the key is issued, and never again.
+ *
+ * issue({ projectId, env, name, permissions, zones, createdBy }) → { key, secret }
+ * find(id) → the row with `secret` (decrypted), or null; revoked keys are null too
+ * list(projectId) → public shapes, newest first (revoked included)
+ * revoke(projectId, id, by) → the public shape, or null when there is no such key of that project
+ * publicShape(row) → media.access-key@1
+ * enabled() → whether keys can be issued and verified (the secret is configured)
+ */
+const crypto = require('crypto');
+const { ids } = require('openvibe-contracts');
+const db = require('../db/database');
+
+const PERMISSIONS = ['read', 'write', 'read-write'];
+const ZONE_ID_RE = /^zon_[0-9A-HJKMNP-TV-Z]{26}$/;
+const KEY_ID_RE = /^ovk_[0-9A-HJKMNP-TV-Z]{26}$/;
+const MAX_KEYS_PER_ENV = 50;
+const TOUCH_MS = 60_000;
+
+function masterKey() {
+ const s = String(process.env.MEDIA_S3_KEY_SECRET || '');
+ return s.length >= 32 ? crypto.createHash('sha256').update(s).digest() : null;
+}
+const enabled = () => !!masterKey();
+
+function encrypt(secret) {
+ const iv = crypto.randomBytes(12);
+ const c = crypto.createCipheriv('aes-256-gcm', masterKey(), iv);
+ const enc = Buffer.concat([c.update(secret, 'utf8'), c.final()]);
+ return [iv, c.getAuthTag(), enc].map((b) => b.toString('base64url')).join('.');
+}
+
+function decrypt(blob) {
+ const [iv, tag, enc] = String(blob).split('.').map((p) => Buffer.from(p, 'base64url'));
+ const d = crypto.createDecipheriv('aes-256-gcm', masterKey(), iv);
+ d.setAuthTag(tag);
+ return Buffer.concat([d.update(enc), d.final()]).toString('utf8');
+}
+
+const iso = (ms) => (ms == null ? null : new Date(Number(ms)).toISOString());
+const parseZones = (z) => (Array.isArray(z) ? z : (() => { try { return JSON.parse(z || '[]'); } catch { return []; } })());
+
+function publicShape(r) {
+ return {
+ id: r.id,
+ project_id: r.project_id,
+ env: r.env,
+ name: r.name,
+ permissions: r.permissions,
+ zones: parseZones(r.zones),
+ created_by: r.created_by,
+ created_at: iso(r.created_at),
+ last_used_at: iso(r.last_used_at),
+ revoked_at: iso(r.revoked_at),
+ };
+}
+
+class KeyError extends Error {
+ constructor(status, code, message) { super(message); this.status = status; this.code = code; }
+}
+
+async function issue({ projectId, env, name, permissions = 'read-write', zones = [], createdBy, now = Date.now() }) {
+ if (!enabled()) throw new KeyError(503, 'media.s3.disabled', 'S3 access keys are not configured on this Media (MEDIA_S3_KEY_SECRET)');
+ if (!['sandbox', 'production'].includes(env)) throw new KeyError(400, 'media.access_key.invalid', 'env is sandbox or production');
+ if (!PERMISSIONS.includes(permissions)) throw new KeyError(400, 'media.access_key.invalid', `permissions is one of ${PERMISSIONS.join(', ')}`);
+ const label = String(name || '').trim().slice(0, 80);
+ if (!label) throw new KeyError(400, 'media.access_key.invalid', 'a key needs a name (what uses it)');
+ if (!Array.isArray(zones) || zones.length > 20 || !zones.every((z) => ZONE_ID_RE.test(String(z)))) {
+ throw new KeyError(400, 'media.access_key.invalid', 'zones is a list of at most 20 zone ids (zon_…); empty means every zone of the environment');
+ }
+ const uniq = [...new Set(zones)];
+ for (const z of uniq) {
+ const zone = await db.get('SELECT project_id, env, state FROM media_zones WHERE id = ?', [z]);
+ if (!zone || zone.project_id !== projectId || zone.env !== env || !['provisioning', 'active'].includes(zone.state)) {
+ throw new KeyError(404, 'media.access_key.unknown_zone', `${z} is not a zone of this project's ${env} environment`);
+ }
+ }
+ const live = (await db.get('SELECT COUNT(*) AS n FROM media_access_keys WHERE project_id = ? AND env = ? AND revoked_at IS NULL', [projectId, env])).n;
+ if (Number(live) >= MAX_KEYS_PER_ENV) throw new KeyError(409, 'media.access_key.limit', `an environment holds at most ${MAX_KEYS_PER_ENV} live keys; revoke one first`);
+ const id = `ovk_${ids.ulid(now)}`;
+ const secret = crypto.randomBytes(30).toString('base64url'); // 40 characters, as S3 secrets are
+ await db.run(`INSERT INTO media_access_keys (id, project_id, env, name, permissions, zones, secret_enc, created_by, created_at)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`, [id, projectId, env, label, permissions, JSON.stringify(uniq), encrypt(secret), createdBy, now]);
+ return { key: publicShape(await db.get('SELECT * FROM media_access_keys WHERE id = ?', [id])), secret };
+}
+
+const touched = new Map();
+async function find(id, { now = Date.now() } = {}) {
+ if (!KEY_ID_RE.test(String(id || '')) || !enabled()) return null;
+ const row = await db.get('SELECT * FROM media_access_keys WHERE id = ? AND revoked_at IS NULL', [id]);
+ if (!row) return null;
+ let secret;
+ try { secret = decrypt(row.secret_enc); } catch { return null; } // a key encrypted under another secret
+ if (!touched.has(id) || now - touched.get(id) > TOUCH_MS) {
+ touched.set(id, now);
+ await db.run('UPDATE media_access_keys SET last_used_at = ? WHERE id = ?', [now, id]);
+ }
+ return { ...row, zones: parseZones(row.zones), secret };
+}
+
+async function list(projectId) {
+ const rows = await db.all('SELECT * FROM media_access_keys WHERE project_id = ? ORDER BY created_at DESC, id DESC', [projectId]);
+ return rows.map(publicShape);
+}
+
+async function revoke(projectId, id, by, { now = Date.now() } = {}) {
+ const row = await db.get('SELECT * FROM media_access_keys WHERE id = ? AND project_id = ?', [id, projectId]);
+ if (!row) return null;
+ if (row.revoked_at == null) await db.run('UPDATE media_access_keys SET revoked_at = ?, revoked_by = ? WHERE id = ?', [now, by, id]);
+ touched.delete(id);
+ return publicShape(await db.get('SELECT * FROM media_access_keys WHERE id = ?', [id]));
+}
+
+/** Revoke every live key scoped only to `zoneId` (a zone that was deleted); keys naming other zones lose it. */
+async function dropZone(zoneId, { now = Date.now() } = {}) {
+ const rows = await db.all("SELECT id, zones FROM media_access_keys WHERE revoked_at IS NULL AND zones::text LIKE ?", [`%${zoneId}%`]);
+ for (const r of rows) {
+ const rest = parseZones(r.zones).filter((z) => z !== zoneId);
+ if (!rest.length) await db.run("UPDATE media_access_keys SET revoked_at = ?, revoked_by = 'system:zone-deleted' WHERE id = ?", [now, r.id]);
+ else await db.run('UPDATE media_access_keys SET zones = ? WHERE id = ?', [JSON.stringify(rest), r.id]);
+ touched.delete(r.id);
+ }
+ return rows.length;
+}
+
+module.exports = { issue, find, list, revoke, dropZone, publicShape, enabled, KeyError, PERMISSIONS, KEY_ID_RE, ZONE_ID_RE };
new file mode 100644
new file mode 100644
@@ -0,0 +1,597 @@
@@ -0,0 +1,597 @@
+'use strict';
+/**
+ * Media's S3 surface (ADR-031 and its zone amendment), path style: https://s3.openvibe.media/<zone name>/<key>.
+ *
+ * Served by Host (MEDIA_S3_HOSTS), not by path: nginx passes the request line untouched, since SigV4 signs the path.
+ *
+ * GET / ListBuckets: the active zones of the key's environment (those it may reach)
+ * HEAD /<zone> HeadBucket
+ * GET /<zone>?location GetBucketLocation
+ * GET /<zone>?list-type=2 ListObjectsV2 (prefix, delimiter, max-keys, continuation-token, start-after)
+ * GET /<zone> ListObjects (v1: prefix, delimiter, max-keys, marker)
+ * PUT /<zone> CreateBucket: 200 for a zone that exists (clients probe this way), else NotImplemented
+ * POST /<zone>?delete DeleteObjects
+ * PUT /<zone>/<key> PutObject (one request; aws-chunked bodies too)
+ * GET /<zone>/<key> GetObject (Range)
+ * HEAD /<zone>/<key> HeadObject
+ * DELETE /<zone>/<key> DeleteObject (Media's soft delete; refused under a retention hold)
+ * everything else NotImplemented (multipart, copy, ACLs, policies, versioning, bucket deletion)
+ *
+ * Every request is signed (SigV4, header or presigned) with an access key (server/s3/keys.js). The key fixes the
+ * project and environment, so a bucket name resolves to one zone id of that environment (server/s3/zones.js); a key
+ * scoped to zones reaches only those. A request is authorised as the project's app would be: the same tenant, the
+ * same quotas (the zone's ceiling and the environment's), the same objects as the native API (/api/v2/:project).
+ *
+ * An object written here is an ordinary Media object: a med_ id in the zone's namespace, with `zone_id` and its key.
+ * A write stores the bytes first, then in one transaction under the tenant's quota lock tombstones the key's previous
+ * object (Media's deletion path; refused while it is held) and makes the new one ready. The default zone also lists
+ * the environment's native objects, under a derived key: the child namespace path and the med_ id.
+ *
+ * Responses never run in a browser: every one carries a sandbox CSP and nosniff, whatever type the object declares.
+ */
+const crypto = require('crypto');
+const fs = require('fs');
+const path = require('path');
+const { pipeline } = require('stream/promises');
+const { PassThrough } = require('stream');
+const express = require('express');
+const db = require('../db/database');
+const config = require('../config');
+const model = require('../objects/model');
+const namespaces = require('../objects/namespaces');
+const sigv4 = require('./sigv4');
+const keys = require('./keys');
+const zones = require('./zones');
+
+const MAX_KEY_BYTES = 1024;
+const MAX_LIST = 1000;
+const MAX_META_BYTES = 2048;
+const DERIVED_KEY = /^(?:[a-z0-9][a-z0-9_-]{0,39}\/){0,3}med_[0-9A-HJKMNP-TV-Z]{26}$/;
+const XMLNS = 'http://s3.amazonaws.com/doc/2006-03-01/';
+
+const STATUS = {
+ AccessDenied: 403, InvalidAccessKeyId: 403, SignatureDoesNotMatch: 403, RequestTimeTooSkewed: 403,
+ AuthorizationHeaderMalformed: 400, AuthorizationQueryParametersError: 400, InvalidRequest: 400, InvalidArgument: 400,
+ KeyTooLongError: 400, MissingContentLength: 411, EntityTooLarge: 400, BadDigest: 400, XAmzContentSHA256Mismatch: 400,
+ IncompleteBody: 400, MalformedXML: 400, InvalidRange: 416, NoSuchBucket: 404, NoSuchKey: 404, QuotaExceeded: 403,
+ NotImplemented: 501, MethodNotAllowed: 405, InternalError: 500, ServiceUnavailable: 503, InvalidBucketName: 400,
+};
+
+const esc = (s) => String(s).replace(/&/g, '&').replace(/</g, '<').replace(/>/g, '>').replace(/"/g, '"').replace(/'/g, ''');
+const requestId = () => crypto.randomBytes(8).toString('hex').toUpperCase();
+
+function baseHeaders(res) {
+ res.set('Content-Security-Policy', "sandbox; default-src 'none'");
+ res.set('X-Content-Type-Options', 'nosniff');
+ res.set('x-amz-request-id', res.locals.rid || (res.locals.rid = requestId()));
+ res.set('Cache-Control', 'private, no-store');
+}
+
+/** Headers as given (Express's res.set would add a charset to a text Content-Type; S3 answers the stored type). */
+function setAll(res, headers) {
+ for (const [k, v] of Object.entries(headers)) res.setHeader(k, v);
+}
+
+function xml(res, status, body) {
+ baseHeaders(res);
+ res.status(status).type('application/xml').send(`<?xml version="1.0" encoding="UTF-8"?>\n${body}`);
+}
+
+function s3error(req, res, code, message, extra = '') {
+ const status = STATUS[code] || 400;
+ baseHeaders(res);
+ if (req.method === 'HEAD') return res.status(status).end();
+ xml(res, status, `<Error><Code>${esc(code)}</Code><Message>${esc(message)}</Message><Resource>${esc(req.originalUrl.split('?')[0])}</Resource>`
+ + `${extra}<RequestId>${res.locals.rid}</RequestId></Error>`);
+}
+
+const etagOf = (obj) => {
+ const md = model.parseJson(obj.metadata, {});
+ return md.md5 ? `"${md.md5}"` : `"${String(obj.content_hash || obj.id).slice(0, 32)}-1"`;
+};
+const lastModified = (obj) => new Date(String(obj.updated_at || obj.created_at).replace(' ', 'T') + (String(obj.updated_at || obj.created_at).endsWith('Z') ? '' : 'Z'));
+const isoOf = (d) => (Number.isFinite(d.getTime()) ? d.toISOString().replace(/\.\d{3}Z$/, '.000Z') : new Date(0).toISOString());
+
+// ── Authentication: the signature, the key, the tenant ───────
+
+async function authenticate(req, res, next) {
+ res.locals.rid = requestId();
+ if (!keys.enabled()) return s3error(req, res, 'ServiceUnavailable', 'The S3 surface is not configured on this Media');
+ const parsed = sigv4.parse(req);
+ if (parsed.error) return s3error(req, res, parsed.error.code, parsed.error.message);
+ if (parsed.anonymous) return s3error(req, res, 'AccessDenied', 'Every request must be signed with an access key');
+ const key = await keys.find(parsed.accessKeyId);
+ if (!key) return s3error(req, res, 'InvalidAccessKeyId', 'The access key ID you provided does not exist in our records');
+ const bad = sigv4.verify(req, parsed, key.secret);
+ if (bad) return s3error(req, res, bad.code, bad.message);
+ try {
+ const quota = (key.env === 'sandbox' ? config.apps.sandboxQuotaMb : config.apps.projectQuotaMb) * 1024 * 1024;
+ req.appRow = await db.ensureProjectTenant(key.project_id, key.env, quota);
+ } catch (err) {
+ return s3error(req, res, 'AccessDenied', err.message);
+ }
+ req.appId = req.appRow.app_id;
+ req.s3 = { key, parsed };
+ next();
+}
+
+const mayRead = (key) => key.permissions === 'read' || key.permissions === 'read-write';
+const mayWrite = (key) => key.permissions === 'write' || key.permissions === 'read-write';
+
+/** The zone a bucket name names for this key, or an S3 error answered (and null returned). */
+async function zoneFor(req, res, bucket) {
+ const { key } = req.s3;
+ const zone = await zones.byName(key.project_id, key.env, bucket);
+ if (!zone || zone.state !== 'active') { s3error(req, res, 'NoSuchBucket', 'The specified bucket does not exist'); return null; }
+ if (key.zones.length && !key.zones.includes(zone.id)) { s3error(req, res, 'AccessDenied', 'This access key is not scoped to this bucket'); return null; }
+ return zone;
+}
+
+// ── Keys and objects ─────────────────────────────────────────
+
+/** The object key from the path (decoded once), or null when it is not a valid S3 key. */
+function keyOf(rest) {
+ let k;
+ try { k = decodeURIComponent(rest); } catch { return null; }
+ const bytes = Buffer.byteLength(k, 'utf8');
+ return bytes >= 1 && bytes <= MAX_KEY_BYTES ? k : null;
+}
+
+/** The ready object a key names in a zone (S3-written, or a native object's derived key in the default zone). */
+async function findObject(zone, tenantId, key) {
+ const row = await db.get(`SELECT * FROM media_objects WHERE zone_id = ? AND object_key = ? AND lifecycle_status = 'ready'`, [zone.id, key]);
+ if (row) return row;
+ if (!zone.is_default || !DERIVED_KEY.test(key)) return null;
+ const parts = key.split('/');
+ const id = parts.pop();
+ const ns = parts.length ? `${zone.namespace}.${parts.join('.')}` : zone.namespace;
+ return await db.get(`SELECT * FROM media_objects WHERE id = ? AND app_id = ? AND namespace = ? AND object_key IS NULL
+ AND lifecycle_status = 'ready'`, [id, tenantId, ns]) || null;
+}
+
+/** SQL for the key every object of a zone lists under (its own, or the derived one in the default zone). */
+function effectiveKeySql(zone) {
+ if (!zone.is_default) return 'o.object_key';
+ const n = zone.namespace.length + 2;
+ return `COALESCE(o.object_key, CASE WHEN o.namespace = '${zone.namespace.replace(/'/g, "''")}' THEN o.id
+ ELSE replace(substr(o.namespace, ${n}), '.', '/') || '/' || o.id END)`;
+}
+
+// ── Reading bytes ────────────────────────────────────────────
+
+/** bytes=a-b | a- | -n against `size` → { start, end } | 'invalid' | null (no Range). */
+function parseRange(header, size) {
+ if (!header) return null;
+ const m = /^bytes=(\d*)-(\d*)$/.exec(String(header).trim());
+ if (!m || (m[1] === '' && m[2] === '')) return null; // S3 ignores a range it cannot read
+ let start, end;
+ if (m[1] === '') { const n = Number(m[2]); if (n === 0) return 'invalid'; start = Math.max(0, size - n); end = size - 1; }
+ else { start = Number(m[1]); end = m[2] === '' ? size - 1 : Math.min(Number(m[2]), size - 1); }
+ if (start >= size || end < start) return 'invalid';
+ return { start, end };
+}
+
+function objectHeaders(obj) {
+ const md = model.parseJson(obj.metadata, {});
+ const h = {
+ 'Content-Type': obj.mime_type || 'application/octet-stream',
+ ETag: etagOf(obj),
+ 'Last-Modified': lastModified(obj).toUTCString(),
+ 'Accept-Ranges': 'bytes',
+ };
+ for (const [k, v] of Object.entries(md.s3_meta || {})) h[`x-amz-meta-${k}`] = v;
+ if (md.content_disposition) h['Content-Disposition'] = md.content_disposition;
+ if (md.content_encoding) h['Content-Encoding'] = md.content_encoding;
+ if (md.cache_control) h['Cache-Control'] = md.cache_control;
+ return h;
+}
+
+/** Presigned GETs may override response headers, as S3 allows. */
+function responseOverrides(req) {
+ const o = {};
+ const map = { 'response-content-type': 'Content-Type', 'response-content-disposition': 'Content-Disposition',
+ 'response-cache-control': 'Cache-Control', 'response-content-encoding': 'Content-Encoding', 'response-content-language': 'Content-Language',
+ 'response-expires': 'Expires' };
+ for (const [q, h] of Object.entries(map)) if (typeof req.query[q] === 'string' && req.query[q].length <= 512) o[h] = req.query[q];
+ return o;
+}
+
+async function sendObject(req, res, obj, { head = false } = {}) {
+ const size = Number(obj.size_bytes) || 0;
+ const headers = { ...objectHeaders(obj), ...responseOverrides(req) };
+ const inm = req.headers['if-none-match'];
+ if (inm && inm.split(',').map((s) => s.trim()).includes(headers.ETag)) { baseHeaders(res); setAll(res, headers); return res.status(304).end(); }
+ const ims = req.headers['if-modified-since'] ? new Date(req.headers['if-modified-since']) : null;
+ if (ims && Number.isFinite(ims.getTime()) && lastModified(obj).getTime() <= ims.getTime() + 999) { baseHeaders(res); setAll(res, headers); return res.status(304).end(); }
+ const im = req.headers['if-match'];
+ if (im && !im.split(',').map((s) => s.trim()).includes(headers.ETag) && im.trim() !== '*') return s3error(req, res, 'PreconditionFailed', 'At least one of the preconditions you specified did not hold');
+ const range = parseRange(req.headers.range, size);
+ if (range === 'invalid') { res.set('Content-Range', `bytes */${size}`); return s3error(req, res, 'InvalidRange', 'The requested range is not satisfiable'); }
+ baseHeaders(res);
+ setAll(res, headers);
+ if (range) {
+ res.status(206).set('Content-Range', `bytes ${range.start}-${range.end}/${size}`).set('Content-Length', String(range.end - range.start + 1));
+ } else res.status(200).set('Content-Length', String(size));
+ if (head || req.method === 'HEAD') return res.end();
+ if (require('../drill').refuseBytes(res)) return;
+ const from = range || (size ? { start: 0, end: size - 1 } : null);
+ if (!from) return res.end();
+ // The placement router ranks the copies; a local one streams from disk, a remote one is fetched and passed on
+ // (S3 clients do not follow redirects).
+ const router = require('../placement/router');
+ const decision = await router.route({ object: obj, purpose: 'download', session: router.sessionFor(req, obj.id), contentType: headers['Content-Type'], expiresIn: 300 });
+ for (const loc of decision.candidates || []) {
+ if (loc.provider === 'local') {
+ if (!fs.existsSync(loc.key)) continue;
+ return await pipeline(fs.createReadStream(loc.key, { start: from.start, end: from.end }), res).catch(() => res.destroy());
+ }
+ const url = (loc.provider === decision.provider && decision.url)
+ || await require('../vod/vod-storage').presignGet(loc.provider, loc.key, 300, {}).catch(() => null);
+ if (!url) continue;
+ const up = await fetch(url, { headers: { Range: `bytes=${from.start}-${from.end}` }, signal: AbortSignal.timeout(30_000) }).catch(() => null);
+ if (!up || !(up.status === 206 || up.status === 200) || !up.body) continue;
+ const { Readable } = require('stream');
+ return await pipeline(Readable.fromWeb(up.body), res).catch(() => res.destroy());
+ }
+ if (!res.headersSent) return s3error(req, res, 'InternalError', 'The object bytes are unavailable right now');
+ res.destroy();
+}
+
+// ── Writing ──────────────────────────────────────────────────
+
+const CRC_TABLE = (() => {
+ const t = new Uint32Array(256);
+ for (let n = 0; n < 256; n++) { let c = n; for (let k = 0; k < 8; k++) c = c & 1 ? 0xedb88320 ^ (c >>> 1) : c >>> 1; t[n] = c >>> 0; }
+ return t;
+})();
+function crc32Update(crc, buf) {
+ let c = crc ^ 0xffffffff;
+ for (const b of buf) c = CRC_TABLE[(c ^ b) & 0xff] ^ (c >>> 8);
+ return (c ^ 0xffffffff) >>> 0;
+}
+
+/** x-amz-meta-* headers (at most 2 KB, as S3 allows), names lowercased. */
+function userMeta(req) {
+ const out = {};
+ let bytes = 0;
+ for (const [k, v] of Object.entries(req.headers)) {
+ if (!k.startsWith('x-amz-meta-')) continue;
+ const name = k.slice('x-amz-meta-'.length);
+ const value = Array.isArray(v) ? v.join(',') : String(v);
+ bytes += name.length + value.length;
+ out[name] = value;
+ }
+ return bytes > MAX_META_BYTES ? null : out;
+}
+
+/**
+ * Stream the body (decoded from aws-chunked when it is) to a temporary file, hashing it: sha256, md5, crc32.
+ * → { bytes, sha256, md5, crc32, trailers } | { error: { code, message } }
+ */
+async function receive(req, tmp, cap, parsed) {
+ const sha = crypto.createHash('sha256'), md5 = crypto.createHash('md5'), sha1 = crypto.createHash('sha1');
+ let crc = 0, n = 0, trailers = {};
+ const counter = new PassThrough();
+ counter.on('data', (chunk) => {
+ n += chunk.length;
+ if (n > cap) { counter.destroy(Object.assign(new Error('too large'), { code: 'EntityTooLarge' })); return; }
+ sha.update(chunk); md5.update(chunk); sha1.update(chunk); crc = crc32Update(crc, chunk);
+ });
+ const stages = [req];
+ if (sigv4.isStreaming(parsed)) stages.push(sigv4.chunkDecoder(parsed, { onTrailers: (t) => { trailers = t; } }));
+ stages.push(counter, fs.createWriteStream(tmp));
+ try {
+ await pipeline(...stages);
+ } catch (err) {
+ return { error: { code: err.code && STATUS[err.code] ? err.code : 'IncompleteBody', message: err.code === 'EntityTooLarge' ? `Objects sent in one request are at most ${cap} bytes` : (err.message || 'The request body ended early') } };
+ }
+ const crcBuf = Buffer.alloc(4); crcBuf.writeUInt32BE(crc);
+ return { bytes: n, sha256: sha.digest('hex'), md5: md5.digest(), sha1: sha1.digest('base64'), crc32: crcBuf.toString('base64'), trailers };
+}
+
+/** Every integrity claim the client made about the body, checked: Content-MD5, x-amz-content-sha256, x-amz-checksum-*. */
+function integrityProblem(req, parsed, got) {
+ const md5 = req.headers['content-md5'];
+ if (md5 && md5 !== got.md5.toString('base64')) return ['BadDigest', 'The Content-MD5 you specified did not match what we received'];
+ if (/^[0-9a-f]{64}$/.test(parsed.payloadHash) && parsed.payloadHash !== got.sha256) {
+ return ['XAmzContentSHA256Mismatch', 'The provided x-amz-content-sha256 header does not match what was computed'];
+ }
+ const claims = { ...got.trailers };
+ for (const k of ['x-amz-checksum-crc32', 'x-amz-checksum-sha256', 'x-amz-checksum-sha1']) if (req.headers[k]) claims[k] = String(req.headers[k]);
+ const ours = { 'x-amz-checksum-crc32': got.crc32, 'x-amz-checksum-sha256': Buffer.from(got.sha256, 'hex').toString('base64'), 'x-amz-checksum-sha1': got.sha1 };
+ for (const [k, v] of Object.entries(claims)) if (ours[k] && ours[k] !== v) return ['BadDigest', `The ${k.slice('x-amz-checksum-'.length)} checksum you specified did not match what we received`];
+ return null;
+}
+
+async function putObject(req, res, zone, key) {
+ const { key: akey, parsed } = req.s3;
+ if (!mayWrite(akey)) { req.resume(); return s3error(req, res, 'AccessDenied', 'This access key is read-only'); }
+ if (req.headers['x-amz-copy-source']) { req.resume(); return s3error(req, res, 'NotImplemented', 'CopyObject is not supported yet: read the object and write it again'); }
+ const acl = String(req.headers['x-amz-acl'] || 'private');
+ if (acl !== 'private' && acl !== 'bucket-owner-full-control') { req.resume(); return s3error(req, res, 'AccessDenied', 'Zones are private: objects cannot be made public over S3'); }
+ if (DERIVED_KEY.test(key)) {
+ req.resume();
+ return s3error(req, res, 'InvalidArgument', 'Keys of the form [path/]med_<id> are the ids of objects written through the native API; choose another key');
+ }
+ const meta = userMeta(req);
+ if (!meta) { req.resume(); return s3error(req, res, 'InvalidArgument', `User metadata is limited to ${MAX_META_BYTES} bytes`); }
+ const declared = Number(sigv4.isStreaming(parsed) ? req.headers['x-amz-decoded-content-length'] : req.headers['content-length']);
+ const cap = config.objects.maxUploadMb * 1024 * 1024;
+ if (!Number.isFinite(declared)) { req.resume(); return s3error(req, res, 'MissingContentLength', 'You must provide the Content-Length HTTP header'); }
+ if (declared > cap) { res.set('Connection', 'close'); req.resume(); return s3error(req, res, 'EntityTooLarge', `Objects sent in one request are at most ${cap} bytes`); }
+ // Room for the declared size before a byte is stored; checked again with the real size below.
+ const pre = await namespaces.checkQuota(req.appRow, zone.namespace, { bytes: declared, objects: 0 });
+ if (pre) { res.set('Connection', 'close'); req.resume(); return s3error(req, res, 'QuotaExceeded', pre.detail); }
+
+ const tmpDir = path.join(config.objects.path, '.tmp');
+ fs.mkdirSync(tmpDir, { recursive: true });
+ const tmp = path.join(tmpDir, `s3-${crypto.randomBytes(8).toString('hex')}`);
+ const drop = () => { try { fs.unlinkSync(tmp); } catch { /* not written */ } };
+ const got = await receive(req, tmp, Math.max(declared, 0), parsed);
+ if (got.error) { drop(); return s3error(req, res, got.error.code, got.error.message); }
+ if (got.bytes !== declared) { drop(); return s3error(req, res, 'IncompleteBody', `You did not provide the number of bytes specified by the Content-Length HTTP header (${declared}; ${got.bytes} received)`); }
+ const bad = integrityProblem(req, parsed, got);
+ if (bad) { drop(); return s3error(req, res, bad[0], bad[1]); }
+
+ const mime = String(req.headers['content-type'] || 'application/octet-stream').split(';')[0].trim().toLowerCase().slice(0, 127) || 'application/octet-stream';
+ const md = { filename: key.split('/').pop().slice(0, 200), md5: got.md5.toString('hex'), s3: true, created_by: akey.id };
+ if (Object.keys(meta).length) md.s3_meta = meta;
+ for (const [h, f] of [['content-disposition', 'content_disposition'], ['content-encoding', 'content_encoding'], ['cache-control', 'cache_control']]) {
+ if (req.headers[h] && String(req.headers[h]).length <= 512) md[f] = String(req.headers[h]);
+ }
+ let outcome;
+ try {
+ const events = require('../events');
+ outcome = await db.getDb().tx(async () => {
+ const previous = await db.get(`SELECT * FROM media_objects WHERE zone_id = ? AND object_key = ? AND lifecycle_status = 'ready'`, [zone.id, key]);
+ const q = await namespaces.checkQuota(req.appRow, zone.namespace, { bytes: got.bytes, objects: 1, excludeId: previous ? previous.id : null });
+ if (q) return { error: ['QuotaExceeded', q.detail] };
+ const fresh = await zones.byId(zone.id);
+ if (!fresh || fresh.state !== 'active') return { error: ['NoSuchBucket', 'The specified bucket does not exist'] };
+ if (previous) {
+ if (await model.isHeld(previous.id)) return { error: ['AccessDenied', 'The object under this key is under a retention hold and cannot be replaced'] };
+ const pmd = model.parseJson(previous.metadata, {});
+ pmd.pre_delete_status = previous.lifecycle_status;
+ pmd.retention_until = new Date(Date.now() + config.objects.retentionDays * 864e5).toISOString();
+ pmd.deleted_by = `${akey.id} (overwrite)`;
+ await db.run(`UPDATE media_objects SET lifecycle_status = 'deleted', deleted_at = ov_now(), metadata = ?, updated_at = ov_now() WHERE id = ?`,
+ [JSON.stringify(pmd), previous.id]);
+ await namespaces.settle(previous.id);
+ }
+ await namespaces.ensure(req.appRow, db.rootNamespace(req.appRow));
+ const id = await model.createObject({
+ app_id: req.appId, namespace: zone.namespace, kind: 'file', owner_subject: null, owner_app: req.appId, owner_user_id: null,
+ visibility: 'private', lifecycle_status: 'uploading', mime_type: mime, size_bytes: got.bytes, metadata: md,
+ });
+ const obj = await model.getObject(id);
+ const dest = model.objectFilePath(obj);
+ fs.mkdirSync(path.dirname(dest), { recursive: true });
+ fs.renameSync(tmp, dest);
+ await db.run(`UPDATE media_objects SET lifecycle_status = 'ready', content_hash = ?, canonical_provider = 'local', canonical_key = ?,
+ zone_id = ?, object_key = ?, updated_at = ov_now() WHERE id = ?`, [got.sha256, dest, zone.id, key, id]);
+ await model.upsertLocation(id, { provider: 'local', key: dest, state: 'present', size_bytes: got.bytes, checksum: got.sha256, verified: true });
+ const ready = await model.getObject(id);
+ await require('../objects/invariant').record(ready);
+ await events.record('media.object.uploaded', req.appId, await model.objectPublic(ready));
+ await events.recordObjectChanges();
+ return { obj: ready };
+ });
+ require('../events').kick();
+ } catch (err) {
+ drop();
+ console.error('[S3] put error:', err.message);
+ return s3error(req, res, 'InternalError', 'We encountered an internal error. Please try again.');
+ }
+ if (outcome.error) { drop(); return s3error(req, res, outcome.error[0], outcome.error[1]); }
+ await namespaces.reconcileChain(req.appRow, zone.namespace).catch(() => {});
+ baseHeaders(res);
+ res.set('ETag', etagOf(outcome.obj));
+ for (const k of ['x-amz-checksum-crc32', 'x-amz-checksum-sha256']) if (req.headers[k] || got.trailers[k]) res.set(k, req.headers[k] || got.trailers[k]);
+ res.status(200).end();
+}
+
+async function deleteOne(req, zone, key) {
+ const obj = await findObject(zone, req.appId, key);
+ if (!obj) return { ok: true }; // S3 answers 204 for a key that is not there
+ if (await model.isHeld(obj.id)) return { error: ['AccessDenied', 'The object is under a retention hold'] };
+ try {
+ await model.softDelete(obj, { by: `${req.s3.key.id}` });
+ } catch (err) {
+ if (err instanceof model.HeldError) return { error: ['AccessDenied', 'The object is under a retention hold'] };
+ throw err;
+ }
+ await namespaces.reconcileChain(req.appRow, obj.namespace).catch(() => {});
+ return { ok: true };
+}
+
+// ── Listing ──────────────────────────────────────────────────
+
+async function listObjects(req, res, zone, { v2 }) {
+ const q = req.query;
+ const prefix = typeof q.prefix === 'string' ? q.prefix : '';
+ const delimiter = typeof q.delimiter === 'string' && q.delimiter ? q.delimiter : '';
+ const maxKeys = Math.min(MAX_LIST, Math.max(0, Number.isFinite(Number(q['max-keys'])) ? Number(q['max-keys']) : MAX_LIST));
+ let after = '';
+ if (v2) {
+ if (typeof q['continuation-token'] === 'string' && q['continuation-token']) {
+ try { after = Buffer.from(q['continuation-token'], 'base64url').toString('utf8'); } catch { return s3error(req, res, 'InvalidArgument', 'The continuation token provided is incorrect'); }
+ } else if (typeof q['start-after'] === 'string') after = q['start-after'];
+ } else if (typeof q.marker === 'string') after = q.marker;
+ const encodeUrl = q['encoding-type'] === 'url';
+ const enc = (s) => (encodeUrl ? sigv4.awsEncode(s, true) : s);
+
+ const scope = zones.objectScope(zone, req.appId);
+ const ek = effectiveKeySql(zone);
+ const named = zone.is_default ? '' : ' AND o.zone_id = ?';
+ const like = `${prefix.replace(/[\\%_]/g, (c) => `\\${c}`)}%`;
+ const page = (cursor, limit) => db.all(`SELECT * FROM (SELECT o.*, ${ek} AS s3_key FROM media_objects o WHERE ${scope.where}${named}
+ AND o.lifecycle_status = 'ready') x WHERE x.s3_key > ? AND x.s3_key LIKE ? ESCAPE '\\' ORDER BY x.s3_key LIMIT ?`,
+ [...scope.args, ...(zone.is_default ? [] : [zone.id]), cursor, like, limit]);
+ // Keys come in byte order. A delimiter folds every key below a common prefix into that prefix, so the cursor jumps
+ // past the whole prefix (its last possible key) once the prefix is listed. A marker that is a common prefix itself
+ // (a v1 NextMarker) is skipped the same way.
+ const SKIP = '\u{10FFFF}';
+ let cursor = delimiter && after && after.endsWith(delimiter) ? `${after}${SKIP}` : after;
+ const contents = [], prefixes = [];
+ let truncated = false;
+ const full = () => contents.length + prefixes.length >= maxKeys;
+ for (let pass = 0; pass < 100 && !full(); pass++) {
+ const rows = await page(cursor, MAX_LIST);
+ for (const r of rows) {
+ if (full()) break;
+ if (prefixes.length && r.s3_key.startsWith(prefixes[prefixes.length - 1])) continue;
+ const i = delimiter ? r.s3_key.indexOf(delimiter, prefix.length) : -1;
+ if (i >= 0) {
+ const cp = r.s3_key.slice(0, i + delimiter.length);
+ prefixes.push(cp);
+ cursor = `${cp}${SKIP}`;
+ } else {
+ contents.push(r);
+ cursor = r.s3_key;
+ }
+ }
+ if (rows.length < MAX_LIST) break;
+ }
+ if (full()) truncated = (await page(cursor, 1)).length > 0;
+ const last = cursor;
+ const items = contents.map((o) => `<Contents><Key>${esc(enc(o.s3_key))}</Key><LastModified>${isoOf(lastModified(o))}</LastModified>`
+ + `<ETag>${esc(etagOf(o))}</ETag><Size>${Number(o.size_bytes) || 0}</Size><StorageClass>STANDARD</StorageClass></Contents>`).join('');
+ const cps = prefixes.map((p) => `<CommonPrefixes><Prefix>${esc(enc(p))}</Prefix></CommonPrefixes>`).join('');
+ const common = `<Name>${esc(zone.name)}</Name><Prefix>${esc(enc(prefix))}</Prefix>${delimiter ? `<Delimiter>${esc(enc(delimiter))}</Delimiter>` : ''}`
+ + `<MaxKeys>${maxKeys}</MaxKeys>${encodeUrl ? '<EncodingType>url</EncodingType>' : ''}<IsTruncated>${truncated}</IsTruncated>`;
+ if (v2) {
+ const token = typeof q['continuation-token'] === 'string' ? `<ContinuationToken>${esc(q['continuation-token'])}</ContinuationToken>` : '';
+ const next = truncated ? `<NextContinuationToken>${Buffer.from(last).toString('base64url')}</NextContinuationToken>` : '';
+ const sa = typeof q['start-after'] === 'string' ? `<StartAfter>${esc(enc(q['start-after']))}</StartAfter>` : '';
+ return xml(res, 200, `<ListBucketResult xmlns="${XMLNS}">${common}<KeyCount>${contents.length + prefixes.length}</KeyCount>${token}${next}${sa}${items}${cps}</ListBucketResult>`);
+ }
+ const marker = `<Marker>${esc(enc(after))}</Marker>`;
+ const lastListed = prefixes.length && (!contents.length || prefixes[prefixes.length - 1] > contents[contents.length - 1].s3_key)
+ ? prefixes[prefixes.length - 1] : (contents.length ? contents[contents.length - 1].s3_key : '');
+ const nextMarker = truncated && delimiter ? `<NextMarker>${esc(enc(lastListed))}</NextMarker>` : '';
+ return xml(res, 200, `<ListBucketResult xmlns="${XMLNS}">${common}${marker}${nextMarker}${items}${cps}</ListBucketResult>`);
+}
+
+async function listBuckets(req, res) {
+ const { key } = req.s3;
+ if (!mayRead(key)) return s3error(req, res, 'AccessDenied', 'This access key cannot read');
+ const rows = (await zones.listActive(key.project_id, key.env)).filter((z) => !key.zones.length || key.zones.includes(z.id));
+ const buckets = rows.map((z) => {
+ const rec = model.parseJson(z.record, {});
+ return `<Bucket><Name>${esc(z.name)}</Name><CreationDate>${esc(rec.created_at || new Date(Number(z.updated_at)).toISOString())}</CreationDate></Bucket>`;
+ }).join('');
+ xml(res, 200, `<ListAllMyBucketsResult xmlns="${XMLNS}"><Owner><ID>${esc(key.project_id)}</ID><DisplayName>${esc(key.project_id)}</DisplayName></Owner><Buckets>${buckets}</Buckets></ListAllMyBucketsResult>`);
+}
+
+/** POST /<zone>?delete: the keys of a <Delete> document, each through DeleteObject's path. */
+async function deleteObjects(req, res, zone) {
+ if (!mayWrite(req.s3.key)) return s3error(req, res, 'AccessDenied', 'This access key is read-only');
+ const chunks = [];
+ let n = 0;
+ try {
+ const body = sigv4.isStreaming(req.s3.parsed) ? req.pipe(sigv4.chunkDecoder(req.s3.parsed)) : req;
+ for await (const c of body) { n += c.length; if (n > 2 * 1024 * 1024) return s3error(req, res, 'MalformedXML', 'The delete request is too large'); chunks.push(c); }
+ } catch (err) { return s3error(req, res, err.code && STATUS[err.code] ? err.code : 'IncompleteBody', err.message); }
+ const raw = Buffer.concat(chunks);
+ if (/^[0-9a-f]{64}$/.test(req.s3.parsed.payloadHash) && crypto.createHash('sha256').update(raw).digest('hex') !== req.s3.parsed.payloadHash) {
+ return s3error(req, res, 'XAmzContentSHA256Mismatch', 'The provided x-amz-content-sha256 header does not match what was computed');
+ }
+ const text = raw.toString('utf8');
+ const objectsXml = text.match(/<Object>[\s\S]*?<\/Object>/g) || [];
+ if (!/<Delete[\s>]/.test(text) || objectsXml.length > MAX_LIST) return s3error(req, res, 'MalformedXML', 'The XML you provided was not well-formed or did not validate');
+ const quiet = /<Quiet>\s*true\s*<\/Quiet>/i.test(text);
+ const unxml = (s) => s.replace(/</g, '<').replace(/>/g, '>').replace(/"/g, '"').replace(/'/g, "'").replace(/&#(\d+);/g, (_, d) => String.fromCodePoint(Number(d))).replace(/&/g, '&');
+ let out = '';
+ for (const o of objectsXml) {
+ const m = /<Key>([\s\S]*?)<\/Key>/.exec(o);
+ const key = m ? unxml(m[1]) : '';
+ if (!key || Buffer.byteLength(key) > MAX_KEY_BYTES) { out += `<Error><Key>${esc(key)}</Key><Code>InvalidArgument</Code><Message>Invalid key</Message></Error>`; continue; }
+ const r = await deleteOne(req, zone, key);
+ if (r.error) out += `<Error><Key>${esc(key)}</Key><Code>${r.error[0]}</Code><Message>${esc(r.error[1])}</Message></Error>`;
+ else if (!quiet) out += `<Deleted><Key>${esc(key)}</Key></Deleted>`;
+ }
+ xml(res, 200, `<DeleteResult xmlns="${XMLNS}">${out}</DeleteResult>`);
+}
+
+// ── Dispatch ─────────────────────────────────────────────────
+
+const router = express.Router();
+router.use(authenticate);
+
+router.use(async (req, res) => {
+ try {
+ const p = sigv4.signedPath(req);
+ const slash = p.indexOf('/', 1);
+ const bucket = decodeURIComponent(slash < 0 ? p.slice(1) : p.slice(1, slash));
+ const rest = slash < 0 ? '' : p.slice(slash + 1);
+ const q = req.query;
+ if (!bucket) {
+ if (req.method === 'GET') return await listBuckets(req, res);
+ return s3error(req, res, 'MethodNotAllowed', 'The specified method is not allowed against this resource');
+ }
+ if (!rest) {
+ if (req.method === 'PUT' && !Object.keys(q).length) {
+ // CreateBucket: zones are made through OpenVibe.Zone; a client that probes with it gets its zone back.
+ const z = await zones.byName(req.s3.key.project_id, req.s3.key.env, bucket);
+ if (z && z.state === 'active' && (!req.s3.key.zones.length || req.s3.key.zones.includes(z.id))) { req.resume(); baseHeaders(res); return res.status(200).end(); }
+ req.resume();
+ return s3error(req, res, 'NotImplemented', 'Create zones in OpenVibe.Zone (https://openvibe.zone): a zone is a bucket');
+ }
+ const zone = await zoneFor(req, res, bucket);
+ if (!zone) return;
+ if (req.method === 'HEAD') { if (!mayRead(req.s3.key)) return s3error(req, res, 'AccessDenied', ''); baseHeaders(res); res.set('x-amz-bucket-region', 'us-east-1'); return res.status(200).end(); }
+ if (req.method === 'GET' && q.location !== undefined) return xml(res, 200, `<LocationConstraint xmlns="${XMLNS}"/>`);
+ if (req.method === 'POST' && q.delete !== undefined) return await deleteObjects(req, res, zone);
+ if (req.method === 'GET' && !['acl', 'policy', 'versioning', 'cors', 'lifecycle', 'tagging', 'uploads', 'versions', 'website', 'encryption', 'logging', 'notification', 'replication', 'object-lock', 'ownershipControls', 'publicAccessBlock', 'accelerate', 'requestPayment'].some((k) => q[k] !== undefined)) {
+ if (!mayRead(req.s3.key)) return s3error(req, res, 'AccessDenied', 'This access key cannot read');
+ return await listObjects(req, res, zone, { v2: q['list-type'] === '2' });
+ }
+ req.resume();
+ return s3error(req, res, 'NotImplemented', 'A header or query you provided implies functionality that is not implemented');
+ }
+ const key = keyOf(rest);
+ if (key == null) { req.resume(); return s3error(req, res, 'KeyTooLongError', `Keys are 1 to ${MAX_KEY_BYTES} bytes`); }
+ const zone = await zoneFor(req, res, bucket);
+ if (!zone) { req.resume(); return; }
+ if (q.uploads !== undefined || q.uploadId !== undefined || q.partNumber !== undefined) {
+ req.resume();
+ return s3error(req, res, 'NotImplemented', 'Multipart uploads are not supported yet: send objects up to the single-request limit in one PUT');
+ }
+ if (['acl', 'tagging', 'retention', 'legal-hold', 'torrent', 'restore', 'attributes', 'versionId'].some((k) => q[k] !== undefined)) {
+ req.resume();
+ return s3error(req, res, 'NotImplemented', 'A header or query you provided implies functionality that is not implemented');
+ }
+ if (req.method === 'PUT') return await putObject(req, res, zone, key);
+ if (req.method === 'GET' || req.method === 'HEAD') {
+ if (!mayRead(req.s3.key)) return s3error(req, res, 'AccessDenied', 'This access key cannot read');
+ const obj = await findObject(zone, req.appId, key);
+ if (!obj) return s3error(req, res, 'NoSuchKey', 'The specified key does not exist.', `<Key>${esc(key)}</Key>`);
+ return await sendObject(req, res, obj);
+ }
+ if (req.method === 'DELETE') {
+ if (!mayWrite(req.s3.key)) return s3error(req, res, 'AccessDenied', 'This access key is read-only');
+ const r = await deleteOne(req, zone, key);
+ if (r.error) return s3error(req, res, r.error[0], r.error[1]);
+ baseHeaders(res);
+ return res.status(204).end();
+ }
+ req.resume();
+ return s3error(req, res, 'MethodNotAllowed', 'The specified method is not allowed against this resource');
+ } catch (err) {
+ console.error('[S3] error:', err.message);
+ if (!res.headersSent) return s3error(req, res, 'InternalError', 'We encountered an internal error. Please try again.');
+ res.destroy();
+ }
+});
+
+/** Is this request for the S3 surface? Its Host is one of MEDIA_S3_HOSTS (default s3.openvibe.media). */
+function isS3Host(req) {
+ const hosts = String(process.env.MEDIA_S3_HOSTS || 's3.openvibe.media').split(',').map((h) => h.trim().toLowerCase()).filter(Boolean);
+ const host = String(req.headers.host || '').toLowerCase().replace(/:\d+$/, '');
+ return hosts.includes(host);
+}
+
+module.exports = router;
+module.exports.isS3Host = isS3Host;
+module.exports.parseRange = parseRange;
+module.exports.DERIVED_KEY = DERIVED_KEY;
new file mode 100644
new file mode 100644
@@ -0,0 +1,290 @@
@@ -0,0 +1,290 @@
+'use strict';
+/**
+ * AWS Signature Version 4 for Media's S3 surface (ADR-031): the server half. Media holds each access key's secret
+ * (server/s3/keys.js) and recomputes the signature a client made, the way S3 does.
+ *
+ * parse(req) → { kind: 'header'|'presigned', accessKeyId, date, amzDate, region, scope,
+ * signedHeaders, signature, payloadHash, expires } | { error }
+ * verify(req, parsed, secret, { now }) → null | { code, message } (the S3 error to answer)
+ * chunkDecoder(parsed, secret) → a Transform that turns an aws-chunked body into its bytes, checking every chunk
+ * signature (STREAMING-AWS4-HMAC-SHA256-PAYLOAD[-TRAILER]) and reading the trailers
+ *
+ * Canonical forms follow S3, not the generic SigV4 rules: the path is decoded and encoded again exactly once, with
+ * '/' kept, and is never normalised (S3 keys may hold '..', '//' and spaces). Header-signed requests must be within
+ * 15 minutes of now; a presigned URL may live 7 days at most and may not be signed more than 15 minutes ahead.
+ */
+const crypto = require('crypto');
+const { Transform } = require('stream');
+
+const ALGORITHM = 'AWS4-HMAC-SHA256';
+const MAX_PRESIGN_S = 7 * 24 * 3600;
+const SKEW_MS = 15 * 60 * 1000;
+const EMPTY_SHA256 = 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855';
+const STREAMING = new Set([
+ 'STREAMING-AWS4-HMAC-SHA256-PAYLOAD',
+ 'STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER',
+ 'STREAMING-UNSIGNED-PAYLOAD-TRAILER',
+]);
+
+const hmac = (key, data) => crypto.createHmac('sha256', key).update(data, 'utf8').digest();
+const sha256hex = (data) => crypto.createHash('sha256').update(data).digest('hex');
+
+/** RFC 3986 encoding as AWS uses it: everything but A-Z a-z 0-9 - . _ ~ is %XX (upper case hex). */
+function awsEncode(s, keepSlash = false) {
+ let out = '';
+ for (const b of Buffer.from(String(s), 'utf8')) {
+ const c = String.fromCharCode(b);
+ if (/[A-Za-z0-9\-._~]/.test(c) || (keepSlash && c === '/')) out += c;
+ else out += `%${b.toString(16).toUpperCase().padStart(2, '0')}`;
+ }
+ return out;
+}
+
+/** decodeURIComponent that leaves a malformed escape alone instead of throwing. */
+function safeDecode(s) {
+ try { return decodeURIComponent(s); } catch { return s; }
+}
+
+/** The path as S3 canonicalises it: each segment decoded, then encoded once; '/' kept; nothing normalised. */
+function canonicalPath(rawPath) {
+ return String(rawPath || '/').split('/').map((seg) => awsEncode(safeDecode(seg))).join('/') || '/';
+}
+
+/** [name, value] pairs of a raw query string, decoded ('+' is a space only in form encoding, never here). */
+function queryPairs(rawQuery) {
+ if (!rawQuery) return [];
+ return rawQuery.split('&').filter(Boolean).map((p) => {
+ const i = p.indexOf('=');
+ return i < 0 ? [safeDecode(p), ''] : [safeDecode(p.slice(0, i)), safeDecode(p.slice(i + 1))];
+ });
+}
+
+function canonicalQuery(pairs) {
+ return pairs.map(([k, v]) => [awsEncode(k), awsEncode(v)])
+ .sort((a, b) => (a[0] < b[0] ? -1 : a[0] > b[0] ? 1 : a[1] < b[1] ? -1 : a[1] > b[1] ? 1 : 0))
+ .map(([k, v]) => `${k}=${v}`).join('&');
+}
+
+function headerValue(req, name) {
+ const v = req.headers[name];
+ if (v === undefined) return undefined;
+ return (Array.isArray(v) ? v.join(',') : String(v)).trim().replace(/\s+/g, ' ');
+}
+
+function splitUrl(req) {
+ const raw = req.originalUrl || req.url || '/';
+ const q = raw.indexOf('?');
+ return { path: q < 0 ? raw : raw.slice(0, q), query: q < 0 ? '' : raw.slice(q + 1) };
+}
+
+/** The path the client signed: the request path below the router's mount point (req.baseUrl), as sent. */
+function signedPath(req) {
+ const { path } = splitUrl(req);
+ const base = req.baseUrl || '';
+ return base && path.startsWith(base) ? path.slice(base.length) || '/' : path;
+}
+
+function signingKey(secret, date, region, service) {
+ return hmac(hmac(hmac(hmac(`AWS4${secret}`, date), region), service), 'aws4_request');
+}
+
+/** Read the credentials a request carries (header or presigned query). Nothing is verified here. */
+function parse(req) {
+ const auth = headerValue(req, 'authorization');
+ const { query } = splitUrl(req);
+ const pairs = queryPairs(query);
+ const q = Object.fromEntries(pairs);
+ if (auth && auth.startsWith(`${ALGORITHM} `)) {
+ const fields = Object.fromEntries(auth.slice(ALGORITHM.length + 1).split(',').map((f) => {
+ const i = f.indexOf('=');
+ return i < 0 ? [f.trim(), ''] : [f.slice(0, i).trim(), f.slice(i + 1).trim()];
+ }));
+ const cred = String(fields.Credential || '').split('/');
+ if (cred.length !== 5 || cred[4] !== 'aws4_request' || !fields.SignedHeaders || !/^[0-9a-f]{64}$/.test(fields.Signature || '')) {
+ return { error: { code: 'AuthorizationHeaderMalformed', message: 'The authorization header is malformed' } };
+ }
+ const amzDate = headerValue(req, 'x-amz-date');
+ const payloadHash = headerValue(req, 'x-amz-content-sha256');
+ if (!amzDate || !/^\d{8}T\d{6}Z$/.test(amzDate)) return { error: { code: 'AccessDenied', message: 'X-Amz-Date is required' } };
+ if (!payloadHash) return { error: { code: 'InvalidRequest', message: 'Missing required header for this request: x-amz-content-sha256' } };
+ return {
+ kind: 'header', accessKeyId: cred[0], date: cred[1], region: cred[2], service: cred[3], amzDate,
+ scope: cred.slice(1).join('/'), signedHeaders: fields.SignedHeaders.split(';'), signature: fields.Signature,
+ payloadHash, pairs,
+ };
+ }
+ if (q['X-Amz-Algorithm'] !== undefined) {
+ if (q['X-Amz-Algorithm'] !== ALGORITHM) return { error: { code: 'AuthorizationQueryParametersError', message: 'X-Amz-Algorithm must be AWS4-HMAC-SHA256' } };
+ const cred = String(q['X-Amz-Credential'] || '').split('/');
+ const expires = Number(q['X-Amz-Expires']);
+ if (cred.length !== 5 || cred[4] !== 'aws4_request' || !/^\d{8}T\d{6}Z$/.test(q['X-Amz-Date'] || '')
+ || !q['X-Amz-SignedHeaders'] || !/^[0-9a-f]{64}$/.test(q['X-Amz-Signature'] || '')) {
+ return { error: { code: 'AuthorizationQueryParametersError', message: 'The presigned URL is malformed' } };
+ }
+ if (!Number.isInteger(expires) || expires < 1 || expires > MAX_PRESIGN_S) {
+ return { error: { code: 'AuthorizationQueryParametersError', message: 'X-Amz-Expires must be between 1 and 604800 seconds' } };
+ }
+ return {
+ kind: 'presigned', accessKeyId: cred[0], date: cred[1], region: cred[2], service: cred[3], amzDate: q['X-Amz-Date'],
+ scope: cred.slice(1).join('/'), signedHeaders: String(q['X-Amz-SignedHeaders']).split(';'), signature: q['X-Amz-Signature'],
+ payloadHash: 'UNSIGNED-PAYLOAD', expires, pairs: pairs.filter(([k]) => k !== 'X-Amz-Signature'),
+ };
+ }
+ return { error: null, anonymous: true };
+}
+
+function amzDateMs(amzDate) {
+ const m = /^(\d{4})(\d{2})(\d{2})T(\d{2})(\d{2})(\d{2})Z$/.exec(amzDate);
+ return m ? Date.UTC(+m[1], +m[2] - 1, +m[3], +m[4], +m[5], +m[6]) : NaN;
+}
+
+/** The signature the client should have sent, given the secret. */
+function expectedSignature(req, parsed, secret) {
+ const headers = parsed.signedHeaders.map((h) => {
+ const v = h === 'host' ? (headerValue(req, 'host') || '') : headerValue(req, h);
+ return `${h}:${v === undefined ? '' : v}\n`;
+ }).join('');
+ const canonical = [
+ req.method,
+ canonicalPath(signedPath(req)),
+ canonicalQuery(parsed.pairs),
+ headers,
+ parsed.signedHeaders.join(';'),
+ parsed.payloadHash,
+ ].join('\n');
+ const toSign = [ALGORITHM, parsed.amzDate, parsed.scope, sha256hex(canonical)].join('\n');
+ const key = signingKey(secret, parsed.date, parsed.region, parsed.service);
+ return { signature: crypto.createHmac('sha256', key).update(toSign, 'utf8').digest('hex'), key };
+}
+
+function safeEqual(a, b) {
+ const x = Buffer.from(String(a)), y = Buffer.from(String(b));
+ return x.length === y.length && crypto.timingSafeEqual(x, y);
+}
+
+/** null when the request is signed by `secret` and in time; else the S3 error ({ code, message }). */
+function verify(req, parsed, secret, { now = Date.now() } = {}) {
+ if (parsed.service !== 's3') return { code: 'AuthorizationHeaderMalformed', message: 'The credential scope must name the s3 service' };
+ if (parsed.amzDate.slice(0, 8) !== parsed.date) return { code: 'AuthorizationHeaderMalformed', message: 'The credential date does not match X-Amz-Date' };
+ if (!parsed.signedHeaders.includes('host')) return { code: 'AccessDenied', message: 'The host header must be signed' };
+ const at = amzDateMs(parsed.amzDate);
+ if (!Number.isFinite(at)) return { code: 'AccessDenied', message: 'X-Amz-Date is not a date' };
+ if (parsed.kind === 'header' && Math.abs(now - at) > SKEW_MS) {
+ return { code: 'RequestTimeTooSkewed', message: 'The difference between the request time and the current time is too large' };
+ }
+ if (parsed.kind === 'presigned') {
+ if (at - now > SKEW_MS) return { code: 'AccessDenied', message: 'Request is not valid yet' };
+ if (now > at + parsed.expires * 1000) return { code: 'AccessDenied', message: 'Request has expired' };
+ }
+ const { signature, key } = expectedSignature(req, parsed, secret);
+ if (!safeEqual(signature, parsed.signature)) {
+ return { code: 'SignatureDoesNotMatch', message: 'The request signature we calculated does not match the signature you provided' };
+ }
+ parsed.signingKey = key;
+ return null;
+}
+
+/** Is the payload an aws-chunked stream (x-amz-content-sha256 says so)? */
+function isStreaming(parsed) {
+ return STREAMING.has(parsed.payloadHash);
+}
+
+/**
+ * aws-chunked → bytes. Each chunk is `<hex size>[;chunk-signature=<sig>]\r\n<data>\r\n`, ending with a 0-size chunk,
+ * then (for the -TRAILER forms) `name:value\r\n` trailers and, when signed, `x-amz-trailer-signature:<sig>\r\n`, and
+ * a final `\r\n`. Signed chunks chain from the request's seed signature. The trailers are handed to `onTrailers` as
+ * an object (x-amz-checksum-* values are checked by the caller). Errors are emitted with an S3 `code`.
+ */
+function chunkDecoder(parsed, { onTrailers = () => {} } = {}) {
+ const signed = parsed.payloadHash.startsWith('STREAMING-AWS4-HMAC-SHA256-PAYLOAD');
+ const scope = parsed.scope;
+ let prev = parsed.signature;
+ let buf = Buffer.alloc(0);
+ let state = 'header'; // header → data → crlf → … → trailers → done
+ let want = 0, chunkSig = null, chunkHash = null, trailerSigned = false;
+ const trailers = {};
+ const fail = (self, code, message) => { const e = new Error(message); e.code = code; self.destroy(e); };
+ const sigOk = (stringToSign, sig) => safeEqual(crypto.createHmac('sha256', parsed.signingKey).update(stringToSign, 'utf8').digest('hex'), sig);
+
+ return new Transform({
+ transform(chunk, _enc, cb) {
+ buf = buf.length ? Buffer.concat([buf, chunk]) : chunk;
+ try {
+ for (;;) {
+ if (state === 'header') {
+ const i = buf.indexOf('\r\n');
+ if (i < 0) { if (buf.length > 4096) return fail(this, 'IncompleteBody', 'Malformed chunk header'); break; }
+ const line = buf.slice(0, i).toString('latin1');
+ buf = buf.slice(i + 2);
+ const [sizeHex, ...ext] = line.split(';');
+ if (!/^[0-9a-fA-F]{1,16}$/.test(sizeHex)) return fail(this, 'IncompleteBody', 'Malformed chunk size');
+ want = parseInt(sizeHex, 16);
+ const sigExt = ext.find((e) => e.startsWith('chunk-signature='));
+ chunkSig = sigExt ? sigExt.slice('chunk-signature='.length) : null;
+ if (signed && !/^[0-9a-f]{64}$/.test(chunkSig || '')) return fail(this, 'SignatureDoesNotMatch', 'A chunk carries no signature');
+ chunkHash = crypto.createHash('sha256');
+ state = want === 0 ? 'last' : 'data';
+ } else if (state === 'data') {
+ if (!buf.length) break;
+ const take = Math.min(want, buf.length);
+ const part = buf.slice(0, take);
+ buf = buf.slice(take);
+ want -= take;
+ chunkHash.update(part);
+ this.push(part);
+ if (want === 0) state = 'crlf';
+ } else if (state === 'crlf') {
+ if (buf.length < 2) break;
+ if (buf[0] !== 13 || buf[1] !== 10) return fail(this, 'IncompleteBody', 'Malformed chunk terminator');
+ buf = buf.slice(2);
+ if (signed) {
+ const s = ['AWS4-HMAC-SHA256-PAYLOAD', parsed.amzDate, scope, prev, EMPTY_SHA256, chunkHash.digest('hex')].join('\n');
+ if (!sigOk(s, chunkSig)) return fail(this, 'SignatureDoesNotMatch', 'A chunk signature does not match');
+ prev = chunkSig;
+ }
+ state = 'header';
+ } else if (state === 'last') {
+ if (signed) {
+ const s = ['AWS4-HMAC-SHA256-PAYLOAD', parsed.amzDate, scope, prev, EMPTY_SHA256, EMPTY_SHA256].join('\n');
+ if (!sigOk(s, chunkSig)) return fail(this, 'SignatureDoesNotMatch', 'The final chunk signature does not match');
+ prev = chunkSig;
+ }
+ state = 'trailers';
+ } else if (state === 'trailers') {
+ const i = buf.indexOf('\r\n');
+ if (i < 0) { if (buf.length > 8192) return fail(this, 'IncompleteBody', 'Malformed trailers'); break; }
+ const line = buf.slice(0, i).toString('latin1');
+ buf = buf.slice(i + 2);
+ if (line === '') { state = 'done'; continue; }
+ const c = line.indexOf(':');
+ if (c < 0) return fail(this, 'IncompleteBody', 'Malformed trailer');
+ const name = line.slice(0, c).trim().toLowerCase(), value = line.slice(c + 1).trim();
+ if (name === 'x-amz-trailer-signature') {
+ const body = Object.entries(trailers).map(([k, v]) => `${k}:${v}\n`).join('');
+ const s = ['AWS4-HMAC-SHA256-TRAILER', parsed.amzDate, scope, prev, sha256hex(body)].join('\n');
+ if (!sigOk(s, value)) return fail(this, 'SignatureDoesNotMatch', 'The trailer signature does not match');
+ trailerSigned = true;
+ } else trailers[name] = value;
+ } else break;
+ }
+ cb();
+ } catch (e) { cb(e); }
+ },
+ flush(cb) {
+ // A body that ends right after the 0-size chunk (no trailers, no final CRLF) is complete too.
+ if (state !== 'done' && !(state === 'trailers' && buf.length === 0)) {
+ const e = new Error('The request body ended early'); e.code = 'IncompleteBody'; return cb(e);
+ }
+ if (signed && parsed.payloadHash.endsWith('-TRAILER') && Object.keys(trailers).length && !trailerSigned) {
+ const e = new Error('Trailers are not signed'); e.code = 'SignatureDoesNotMatch'; return cb(e);
+ }
+ onTrailers(trailers);
+ cb();
+ },
+ });
+}
+
+module.exports = {
+ parse, verify, isStreaming, chunkDecoder, awsEncode, canonicalPath, canonicalQuery, queryPairs, signingKey,
+ expectedSignature, signedPath, EMPTY_SHA256, MAX_PRESIGN_S, SKEW_MS,
+};
new file mode 100644
new file mode 100644
@@ -0,0 +1,173 @@
@@ -0,0 +1,173 @@
+'use strict';
+/**
+ * Media's projection of OpenVibe.Zone's zones (ADR-031 amendment 2026-10-02). Zone owns the record; it pushes every
+ * change here with a higher revision (PUT /internal/zones/:id, server/s3/admin-routes.js), and Media uses the
+ * projection to resolve S3 bucket names, enforce a zone's byte ceiling (its namespace row's quota_bytes, checked with
+ * the environment's by objects/namespaces.checkQuota) and refuse requests to a zone that is going away.
+ *
+ * provision(record) → { applied, zone } (checkObjectZone first; a lower or equal revision changes nothing)
+ * byName(projectId, env, name) / byId(id) / listActive(projectId, env)
+ * status(zone) → { stored_bytes, objects, objects_remaining, held_objects, as_of }
+ * sweepDeleting() → tombstones and purges what deleting zones still hold (the job runs it every minute)
+ * objectScope(zone, app) → { where, args }: the SQL condition for the objects of a zone
+ *
+ * A zone's namespace is `<environment root>.<zone id>`; the default zone is the environment root itself and holds
+ * every object of the environment that is not in a named zone (children included, legacy layouts unchanged).
+ */
+const { zones: zoneContracts } = require('openvibe-contracts');
+const db = require('../db/database');
+const config = require('../config');
+const model = require('../objects/model');
+
+const BATCH = 200;
+
+class ZoneError extends Error {
+ constructor(status, code, message) { super(message); this.status = status; this.code = code; }
+}
+
+const rootOf = (projectId, env) => `app.${projectId}${env === 'sandbox' ? '.sandbox' : ''}`;
+const tenantQuota = (env) => (env === 'sandbox' ? config.apps.sandboxQuotaMb : config.apps.projectQuotaMb) * 1024 * 1024;
+
+async function byId(id) {
+ return await db.get('SELECT * FROM media_zones WHERE id = ?', [id]) || null;
+}
+
+async function byName(projectId, env, name) {
+ return await db.get('SELECT * FROM media_zones WHERE project_id = ? AND env = ? AND name = ?', [projectId, env, name]) || null;
+}
+
+async function listActive(projectId, env) {
+ return await db.all("SELECT * FROM media_zones WHERE project_id = ? AND env = ? AND state = 'active' ORDER BY name", [projectId, env]);
+}
+
+/** The zone a namespace belongs to: the named zone it is, or the environment's default zone. */
+async function forNamespace(app, namespace) {
+ if (!app || !app.project_id) return null;
+ const root = rootOf(app.project_id, app.env);
+ const m = /^(.+)\.(zon_[0-9A-HJKMNP-TV-Z]{26})$/.exec(namespace);
+ if (m && m[1] === root) return await byId(m[2]);
+ if (namespace === root || namespace.startsWith(`${root}.`)) return await byName(app.project_id, app.env, 'default');
+ return null;
+}
+
+/** Apply one zone record from Zone. Idempotent by revision; throws ZoneError for a record Media must refuse. */
+async function provision(record, { now = Date.now() } = {}) {
+ const check = zoneContracts.checkObjectZone(record);
+ if (!check.valid) throw new ZoneError(400, 'media.zone.invalid', JSON.stringify(check.errors).slice(0, 400));
+ return await db.getDb().tx(async () => {
+ const cur = await byId(record.id);
+ if (cur && Number(cur.revision) >= record.revision) return { applied: false, zone: cur };
+ if (cur && (cur.project_id !== record.project_id || cur.env !== record.env || cur.name !== record.name || cur.namespace !== record.media_namespace)) {
+ throw new ZoneError(409, 'media.zone.immutable', 'a zone keeps its project, environment, name and namespace');
+ }
+ if (cur && cur.state === 'deleted' && record.state !== 'deleted') throw new ZoneError(409, 'media.zone.deleted', 'a deleted zone stays deleted');
+ const named = await byName(record.project_id, record.env, record.name);
+ if (named && named.id !== record.id) throw new ZoneError(409, 'media.zone.name_retired', `${record.name} belongs to ${named.id} in this environment`);
+ const tenant = await db.ensureProjectTenant(record.project_id, record.env, tenantQuota(record.env));
+ const root = db.rootNamespace(tenant);
+ if (!record.default) {
+ // The zone's own namespace row: its quota (a ceiling below the environment's) is enforced on every write.
+ await db.run(`INSERT INTO media_namespaces (namespace, app_id, parent, owner, quota_bytes) VALUES (?, ?, ?, ?, ?)
+ ON CONFLICT (namespace) DO UPDATE SET quota_bytes = excluded.quota_bytes, updated_at = ov_now()`,
+ [record.media_namespace, tenant.app_id, root, db.namespaceOwner(tenant), record.quota_bytes ?? null]);
+ }
+ const deleting = ['deleting', 'deleted'].includes(record.state);
+ await db.run(`INSERT INTO media_zones (id, project_id, env, name, is_default, namespace, quota_bytes, state, revision, record,
+ deletion_requested_at, deletion_completed_at, updated_at)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, NULL, ?)
+ ON CONFLICT (id) DO UPDATE SET quota_bytes = excluded.quota_bytes, state = excluded.state, revision = excluded.revision,
+ record = excluded.record, updated_at = excluded.updated_at,
+ deletion_requested_at = COALESCE(media_zones.deletion_requested_at, excluded.deletion_requested_at)`,
+ [record.id, record.project_id, record.env, record.name, !!record.default, record.media_namespace, record.quota_bytes ?? null,
+ record.state, record.revision, JSON.stringify(record), deleting ? now : null, now]);
+ if (record.state === 'deleted') await require('./keys').dropZone(record.id, { now });
+ return { applied: true, zone: await byId(record.id) };
+ });
+}
+
+/** SQL for the objects of a zone: its namespace, or (default zone) the environment's objects outside named zones. */
+function objectScope(zone, tenantId) {
+ if (!zone.is_default) return { where: 'o.namespace = ?', args: [zone.namespace] };
+ return {
+ where: "o.app_id = ? AND (o.namespace = ? OR (o.namespace LIKE ? AND o.namespace NOT LIKE ?))",
+ args: [tenantId, zone.namespace, `${zone.namespace}.%`, `${zone.namespace}.zon\\_%`],
+ };
+}
+
+const tenantIdOf = (zone) => db.projectTenantId(zone.project_id, zone.env);
+
+async function status(zone, { now = Date.now() } = {}) {
+ const s = objectScope(zone, tenantIdOf(zone));
+ const live = await db.get(`SELECT COUNT(*) AS n, COALESCE(SUM(o.size_bytes), 0) AS b FROM media_objects o
+ WHERE ${s.where} AND o.lifecycle_status = 'ready'`, s.args);
+ const remaining = await db.get(`SELECT COUNT(*) AS n FROM media_objects o WHERE ${s.where}
+ AND (o.lifecycle_status <> 'deleted' OR o.metadata NOT LIKE '%"purged_at"%')`, s.args);
+ const held = await db.get(`SELECT COUNT(DISTINCT o.id) AS n FROM media_objects o JOIN media_holds h ON h.object_id = o.id
+ WHERE ${s.where} AND h.released_at IS NULL AND (o.lifecycle_status <> 'deleted' OR o.metadata NOT LIKE '%"purged_at"%')`, s.args);
+ return {
+ stored_bytes: Number(live.b), objects: Number(live.n),
+ objects_remaining: Number(remaining.n), held_objects: Number(held.n), as_of: new Date(now).toISOString(),
+ };
+}
+
+/** Is any object of the zone under a retention hold? (Zone asks before it starts a deletion: 409 zone.held.) */
+async function hasHeld(zone) {
+ return (await status(zone)).held_objects > 0;
+}
+
+/**
+ * One pass over the zones being deleted: every live object is tombstoned with no restore window (a held one too: its
+ * hold delays only its purge), then every tombstoned object without a hold is purged, its copies erased. When nothing
+ * is left the projection records the completion; Zone reads it from the status and marks the zone deleted.
+ */
+async function sweepDeleting({ now = Date.now() } = {}) {
+ const zones = await db.all("SELECT * FROM media_zones WHERE state = 'deleting' AND deletion_completed_at IS NULL");
+ let tombstoned = 0, purged = 0;
+ for (const zone of zones) {
+ const s = objectScope(zone, tenantIdOf(zone));
+ const live = await db.all(`SELECT o.* FROM media_objects o WHERE ${s.where} AND o.lifecycle_status <> 'deleted' LIMIT ${BATCH}`, s.args);
+ for (const obj of live) {
+ const md = model.parseJson(obj.metadata, {});
+ md.pre_delete_status = obj.lifecycle_status;
+ md.retention_until = new Date(now).toISOString();
+ md.deleted_by = `zone:${zone.id}`;
+ md.no_restore = true;
+ const events = require('../events');
+ await db.getDb().tx(async () => {
+ await db.run(`UPDATE media_objects SET lifecycle_status = 'deleted', deleted_at = ov_now(), metadata = ?, updated_at = ov_now()
+ WHERE id = ? AND lifecycle_status <> 'deleted'`, [JSON.stringify(md), obj.id]);
+ await require('../objects/namespaces').settle(obj.id);
+ await events.recordObjectChanges();
+ });
+ events.kick();
+ tombstoned++;
+ }
+ const dead = await db.all(`SELECT o.* FROM media_objects o WHERE ${s.where} AND o.lifecycle_status = 'deleted'
+ AND o.metadata NOT LIKE '%"purged_at"%' LIMIT ${BATCH}`, s.args);
+ for (const obj of dead) if (await model.purgeOne(obj)) purged++;
+ const st = await status(zone, { now });
+ if (st.objects_remaining === 0) {
+ await db.run('UPDATE media_zones SET deletion_completed_at = ?, updated_at = ? WHERE id = ?', [now, now, zone.id]);
+ }
+ }
+ return { zones: zones.length, tombstoned, purged };
+}
+
+/** The zone.object-zone-usage@1-like snapshot Zone shows: the live objects and, while deleting, the progress. */
+async function report(zone, opts) {
+ const st = await status(zone, opts);
+ return {
+ zone_id: zone.id, revision: Number(zone.revision), state: zone.state,
+ usage: { stored_bytes: st.stored_bytes, objects: st.objects, as_of: st.as_of },
+ held_objects: st.held_objects, // Zone refuses to start a deletion while any object is held
+ deletion: zone.deletion_requested_at == null ? null : {
+ requested_at: new Date(Number(zone.deletion_requested_at)).toISOString(),
+ objects_remaining: st.objects_remaining, held_objects: st.held_objects,
+ completed_at: zone.deletion_completed_at == null ? null : new Date(Number(zone.deletion_completed_at)).toISOString(),
+ },
+ };
+}
+
+module.exports = {
+ provision, byId, byName, listActive, forNamespace, status, report, hasHeld, sweepDeleting, objectScope, rootOf, ZoneError,
+};
new file mode 100644
new file mode 100644
@@ -0,0 +1,71 @@
@@ -0,0 +1,71 @@
+'use strict';
+// server/s3/sigv4.js against the worked examples of the Amazon S3 API reference ("Signature Calculations for the
+// Authorization Header", "Authenticating Requests: Using Query Parameters" and "Signature Calculations for the
+// Authorization Header: Transferring Payload in Multiple Chunks"), with their published signatures. They pin the
+// canonical forms independently of any SDK: the path, the query, the signed headers and the chunk signature chain.
+const assert = require('assert');
+const { Readable } = require('stream');
+const sigv4 = require('../server/s3/sigv4');
+
+const SECRET = 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY';
+const AT = Date.UTC(2013, 4, 24, 0, 0, 30);
+const req = (method, url, headers) => ({ method, url, originalUrl: url, baseUrl: '', headers });
+
+(async () => {
+ // GET Object with a Range, signed in the Authorization header.
+ let r = req('GET', '/test.txt', {
+ host: 'examplebucket.s3.amazonaws.com', range: 'bytes=0-9',
+ 'x-amz-content-sha256': sigv4.EMPTY_SHA256, 'x-amz-date': '20130524T000000Z',
+ authorization: 'AWS4-HMAC-SHA256 Credential=AKIAIOSFODNN7EXAMPLE/20130524/us-east-1/s3/aws4_request,SignedHeaders=host;range;x-amz-content-sha256;x-amz-date,Signature=f0e8bdb87c964420e857bd35b5d6ed310bd44f0170aba48dd91039c6036bdb41',
+ });
+ let p = sigv4.parse(r);
+ assert.strictEqual(p.kind, 'header');
+ assert.strictEqual(sigv4.verify(r, p, SECRET, { now: AT }), null, 'the published GET Object signature verifies');
+ assert.strictEqual(sigv4.verify(r, p, 'x'.repeat(40), { now: AT }).code, 'SignatureDoesNotMatch');
+ assert.strictEqual(sigv4.verify(r, p, SECRET, { now: AT + 20 * 60 * 1000 }).code, 'RequestTimeTooSkewed', 'header-signed requests are good for 15 minutes');
+
+ // A presigned GET (query parameters), valid 86400 seconds.
+ const q = 'X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIOSFODNN7EXAMPLE%2F20130524%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20130524T000000Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host';
+ r = req('GET', `/test.txt?${q}&X-Amz-Signature=aeeed9bbccd4d02ee5c0109b86d86835f995330da4c265957d157751f604d404`, { host: 'examplebucket.s3.amazonaws.com' });
+ p = sigv4.parse(r);
+ assert.deepStrictEqual([p.kind, p.expires, p.payloadHash], ['presigned', 86400, 'UNSIGNED-PAYLOAD']);
+ assert.strictEqual(sigv4.verify(r, p, SECRET, { now: AT }), null, 'the published presigned URL verifies');
+ assert.strictEqual(sigv4.verify(r, p, SECRET, { now: AT + 86401 * 1000 }).message, 'Request has expired');
+ r = req('GET', `/test.txt?${q.replace('86400', '604801')}&X-Amz-Signature=${'0'.repeat(64)}`, { host: 'examplebucket.s3.amazonaws.com' });
+ assert.strictEqual(sigv4.parse(r).error.code, 'AuthorizationQueryParametersError', 'a presigned URL lives 7 days at most');
+
+ // A chunked upload: 66560 bytes of 'a' in chunks of 65536, 1024 and 0, each signature chained to the last.
+ const headers = {
+ host: 's3.amazonaws.com', 'x-amz-date': '20130524T000000Z', 'x-amz-storage-class': 'REDUCED_REDUNDANCY',
+ 'x-amz-content-sha256': 'STREAMING-AWS4-HMAC-SHA256-PAYLOAD', 'content-encoding': 'aws-chunked',
+ 'x-amz-decoded-content-length': '66560', 'content-length': '66824',
+ authorization: 'AWS4-HMAC-SHA256 Credential=AKIAIOSFODNN7EXAMPLE/20130524/us-east-1/s3/aws4_request,SignedHeaders=content-encoding;content-length;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-storage-class,Signature=4f232c4386841ef735655705268965c44a0e4690baa4adea153f7db9fa80a0a9',
+ };
+ r = req('PUT', '/examplebucket/chunkObject.txt', headers);
+ p = sigv4.parse(r);
+ assert.strictEqual(sigv4.verify(r, p, SECRET, { now: AT }), null, 'the published seed signature verifies');
+ assert.ok(sigv4.isStreaming(p));
+ const chunk = (n, sig) => Buffer.concat([Buffer.from(`${n.toString(16)};chunk-signature=${sig}\r\n`), Buffer.alloc(n, 'a'), Buffer.from('\r\n')]);
+ const body = Buffer.concat([
+ chunk(65536, 'ad80c730a21e5b8d04586a2213dd63b9a0e99e0e2307b0ade35a65485a288648'),
+ chunk(1024, '0055627c9e194cb4542bae2aa5492e3c1575bbb81b612b7d234b86a503ef5497'),
+ chunk(0, 'b6c6ea8a5354eaf15b3cb7646744f4275b71ea724fed81ceb9323e279d449df9'),
+ ]);
+ assert.strictEqual(body.length, 66824, 'the published body length');
+ const decode = async (bytes, parsed) => {
+ const out = [];
+ for await (const c of Readable.from([bytes.subarray(0, 1000), bytes.subarray(1000, 40000), bytes.subarray(40000)]).pipe(sigv4.chunkDecoder(parsed))) out.push(c);
+ return Buffer.concat(out);
+ };
+ const data = await decode(body, p);
+ assert.ok(data.equals(Buffer.alloc(66560, 'a')), 'the chunks decode to the 66560 bytes, every signature checked');
+ const tampered = Buffer.from(body);
+ tampered[200] = 'b'.charCodeAt(0);
+ await assert.rejects(decode(tampered, p), (e) => e.code === 'SignatureDoesNotMatch', 'a changed byte breaks its chunk signature');
+ await assert.rejects(decode(body.subarray(0, 66000), p), (e) => e.code === 'IncompleteBody', 'a truncated body is refused');
+
+ // Canonical paths: decoded once, encoded once, '/' kept, nothing normalised.
+ assert.strictEqual(sigv4.canonicalPath('/bucket/a%20b%2Bc/..//x~y'), '/bucket/a%20b%2Bc/..//x~y');
+ assert.strictEqual(sigv4.canonicalPath('/bucket/caf%C3%A9'), '/bucket/caf%C3%A9');
+ console.log('s3 sigv4: all checks passed');
+})().catch((err) => { console.error(err); process.exit(1); });
new file mode 100644
new file mode 100644
@@ -0,0 +1,279 @@
@@ -0,0 +1,279 @@
+'use strict';
+// ADR-031 and its zone amendment, end to end with the AWS SDK for JavaScript v3 against a local Media:
+// zones provisioned by Zone's revisioned PUT, access keys issued on behalf of a project admin, every supported S3
+// operation (PutObject plain and aws-chunked, GetObject with ranges, HeadObject, ListObjectsV2/v1 with prefixes,
+// delimiters and pages, DeleteObject, DeleteObjects, ListBuckets, HeadBucket), presigned GET and PUT, two doors
+// (S3 and the native API) on one object, isolation (zone-scoped and read-only keys, other environments), quotas,
+// overwrite under a hold, the NotImplemented answers, and zone deletion with no name reuse.
+const assert = require('assert');
+const fs = require('fs');
+const os = require('os');
+const path = require('path');
+const http = require('http');
+const crypto = require('crypto');
+const { Readable } = require('stream');
+
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-s3-'));
+ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+ process.env.VOD_PATH = dir('vods');
+ process.env.FILES_PATH = dir('files');
+ process.env.OBJECTS_PATH = dir('objects');
+ process.env.THUMBNAILS_PATH = dir('thumbnails');
+ process.env.OV_NETWORK_URL = 'https://openvibe.network';
+ process.env.MEDIA_PUBLIC_URL = 'https://media.test';
+ process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';
+ process.env.MEDIA_S3_KEY_SECRET = 'k'.repeat(40); // a test value: 32+ characters
+ process.env.MEDIA_S3_HOSTS = '127.0.0.1';
+
+ const db = require('../server/db/database');
+ const auth = require('../server/auth');
+ const model = require('../server/objects/model');
+ const zones = require('../server/s3/zones');
+ const { serviceAuth, zones: zoneContracts, ids } = require('openvibe-contracts');
+ const express = require('express');
+ const S3 = require('@aws-sdk/client-s3');
+ const { getSignedUrl } = require('@aws-sdk/s3-request-presigner');
+
+ const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });
+ auth._setNetworkPublicKeyForTests(keys.publicKey);
+ const now = Math.floor(Date.now() / 1000);
+ const svc = (sub, cap, over = {}) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub, actor_type: 'service', aud: ['openvibe.media'], cap, iat: now, exp: now + 600, jti: `tok_${crypto.randomBytes(6).toString('hex')}`, ...over }, keys.privateKey);
+ const ZONE_TOKEN = svc('svc:zone', ['media.zone.provision']);
+ const SERVICES_TOKEN = svc('svc:services', ['media.access_key.manage']);
+ const ADMIN = `usr_${ids.ulid()}`, STRANGER = `usr_${ids.ulid()}`;
+ const PRJ = `prj_${ids.ulid()}`, OTHER = `prj_${ids.ulid()}`;
+
+ // Same mount order as server/index.js.
+ const s3router = require('../server/s3/routes');
+ const admin = require('../server/s3/admin-routes');
+ const app = express();
+ app.use((req, res, next) => (s3router.isS3Host(req) ? s3router(req, res, next) : next()));
+ app.use('/internal/zones', admin.internalRouter());
+ app.use('/api/v1/projects/:project/access-keys', admin.keysRouter({ projects: { isAdmin: async (p, s) => s === ADMIN && [PRJ, OTHER].includes(p) } }));
+ app.put('/api/v2/:app/objects/:id/content', ...require('../server/objects/routes').contentHandlers);
+ app.use(express.json());
+ app.use('/api/v2/:app/objects', require('../server/objects/routes'));
+ const server = http.createServer(app);
+ await new Promise((r) => server.listen(0, '127.0.0.1', r));
+ const port = server.address().port;
+ // The internal doors are reached as localhost (a different Host than the S3 one) from loopback.
+ const internal = async (method, p, { token, body } = {}) => {
+ const res = await fetch(`http://localhost:${port}${p}`, { method, headers: { authorization: `Bearer ${token}`, 'content-type': 'application/json' }, body: body ? JSON.stringify(body) : undefined });
+ return { status: res.status, body: await res.json().catch(() => null) };
+ };
+ const zoneRecord = (projectId, env, name, over = {}) => {
+ const id = over.id || `zon_${ids.ulid()}`;
+ const isDefault = name === 'default';
+ const z = { id, project_id: projectId, env, name, default: isDefault, access: isDefault ? 'per-object' : 'private', state: 'active',
+ quota_bytes: null, revision: 1, created_at: new Date().toISOString(), ...over };
+ Object.assign(z, zoneContracts.identity(z));
+ return z;
+ };
+ const provision = async (z) => internal('PUT', `/internal/zones/${z.id}`, { token: ZONE_TOKEN, body: z });
+
+ try {
+ // ── Zones from Zone ──
+ const def = zoneRecord(PRJ, 'production', 'default');
+ const assets = zoneRecord(PRJ, 'production', 'assets');
+ const logs = zoneRecord(PRJ, 'production', 'logs', { quota_bytes: 64 });
+ const sandbox = zoneRecord(PRJ, 'sandbox', 'assets');
+ const otherAssets = zoneRecord(OTHER, 'production', 'assets');
+ for (const z of [def, assets, logs, sandbox, otherAssets]) {
+ const r = await provision(z);
+ assert.deepStrictEqual([r.status, r.body.applied, r.body.state], [200, true, 'active'], JSON.stringify(r.body));
+ }
+ let r = await provision(assets);
+ assert.deepStrictEqual([r.status, r.body.applied], [200, false], 'a repeat (same revision) changes nothing');
+ r = await provision({ ...zoneRecord(PRJ, 'production', 'assets'), revision: 1 });
+ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.zone.name_retired'], 'a name belongs to one zone of its environment');
+ r = await provision({ ...assets, media_namespace: `app.${OTHER}.${assets.id}`, revision: 2 });
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.zone.invalid'], 'a record naming another project\'s namespace is refused (checkObjectZone)');
+ r = await internal('PUT', `/internal/zones/${assets.id}`, { token: SERVICES_TOKEN, body: assets });
+ assert.strictEqual(r.status, 403, 'only media.zone.provision provisions');
+ console.log('✅ zones: provisioned by revision, names never reused, identity checked, capability required');
+
+ // ── Keys, on behalf of a project admin ──
+ const issue = async (body, project = PRJ) => internal('POST', `/api/v1/projects/${project}/access-keys`, { token: SERVICES_TOKEN, body: { on_behalf_of: ADMIN, ...body } });
+ r = await internal('POST', `/api/v1/projects/${PRJ}/access-keys`, { token: SERVICES_TOKEN, body: { on_behalf_of: STRANGER, env: 'production', name: 'x' } });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'media.access_key.not_admin'], 'only a project admin gets keys');
+ const issued = await issue({ env: 'production', name: 'ci deploys' });
+ assert.strictEqual(issued.status, 201, JSON.stringify(issued.body));
+ const full = issued.body;
+ assert.ok(/^ovk_/.test(full.key.id) && full.secret.length === 40 && !('secret' in (await internal('GET', `/api/v1/projects/${PRJ}/access-keys?on_behalf_of=${ADMIN}`, { token: SERVICES_TOKEN })).body.keys[0]), 'the secret is shown once');
+ const scoped = (await issue({ env: 'production', name: 'logs only', zones: [logs.id] })).body;
+ const reader = (await issue({ env: 'production', name: 'reader', permissions: 'read' })).body;
+ const sbox = (await issue({ env: 'sandbox', name: 'sandbox' })).body;
+ const other = (await issue({ env: 'production', name: 'other' }, OTHER)).body;
+ r = await issue({ env: 'production', name: 'bad', zones: [sandbox.id] });
+ assert.deepStrictEqual([r.status, r.body.code], [404, 'media.access_key.unknown_zone'], 'a key cannot name a zone of another environment');
+ console.log('✅ access keys: issued for admins only, secret once, scopes checked');
+
+ const client = (k) => new S3.S3Client({ endpoint: `http://127.0.0.1:${port}`, region: 'us-east-1', forcePathStyle: true, credentials: { accessKeyId: k.key.id, secretAccessKey: k.secret } });
+ const s3 = client(full);
+ const body = Buffer.from('hello zones\n'.repeat(100));
+ const md5 = crypto.createHash('md5').update(body).digest('hex');
+
+ // ── PutObject, GetObject, HeadObject ──
+ let out = await s3.send(new S3.PutObjectCommand({ Bucket: 'assets', Key: 'site/index.html', Body: body, ContentType: 'text/html', Metadata: { mtime: '1700000000' } }));
+ assert.strictEqual(out.ETag, `"${md5}"`, 'the ETag is the MD5 of the bytes, as S3 makes it');
+ out = await s3.send(new S3.GetObjectCommand({ Bucket: 'assets', Key: 'site/index.html' }));
+ assert.deepStrictEqual([Buffer.from(await out.Body.transformToByteArray()).equals(body), out.ContentType, out.Metadata.mtime], [true, 'text/html', '1700000000']);
+ out = await s3.send(new S3.GetObjectCommand({ Bucket: 'assets', Key: 'site/index.html', Range: 'bytes=6-10' }));
+ assert.deepStrictEqual([await out.Body.transformToString(), out.ContentRange], ['zones', `bytes 6-10/${body.length}`]);
+ out = await s3.send(new S3.HeadObjectCommand({ Bucket: 'assets', Key: 'site/index.html' }));
+ assert.deepStrictEqual([out.ContentLength, out.ETag], [body.length, `"${md5}"`]);
+ // A streamed body of known length goes aws-chunked with a trailing checksum (the SDK's default).
+ await s3.send(new S3.PutObjectCommand({ Bucket: 'assets', Key: 'stream/a b+c.bin', Body: Readable.from([body.subarray(0, 500), body.subarray(500)]), ContentLength: body.length }));
+ out = await s3.send(new S3.GetObjectCommand({ Bucket: 'assets', Key: 'stream/a b+c.bin' }));
+ assert.ok(Buffer.from(await out.Body.transformToByteArray()).equals(body), 'a streamed upload with a space and + in its key round-trips');
+ const raw = await fetch(`http://127.0.0.1:${port}/assets/site/index.html`);
+ assert.deepStrictEqual([raw.status, raw.headers.get('content-security-policy')], [403, "sandbox; default-src 'none'"], 'unsigned requests are refused, and nothing here runs in a browser');
+ console.log('✅ PutObject (plain and aws-chunked), GetObject with a range, HeadObject, metadata, unsigned refused');
+
+ // ── Two doors: the object is an ordinary Media object ──
+ const row = await db.get("SELECT * FROM media_objects WHERE zone_id = ? AND object_key = 'site/index.html' AND lifecycle_status = 'ready'", [assets.id]);
+ assert.deepStrictEqual([row.namespace, row.app_id, row.content_hash, row.visibility], [assets.media_namespace, PRJ, crypto.createHash('sha256').update(body).digest('hex'), 'private']);
+ const appTok = serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: `app:app_${ids.ulid()}`, actor_type: 'app', project_id: PRJ, env: 'production', aud: ['openvibe.media'], cap: ['media.object.upload', 'media.object.read', 'media.object.list', 'media.object.delete'], ns: [PRJ], iat: now, exp: now + 600, jti: 'tok_app1' }, keys.privateKey);
+ const native = async (method, p, b) => { const res = await fetch(`http://localhost:${port}/api/v2/${PRJ}/objects${p}`, { method, headers: { authorization: `Bearer ${appTok}`, 'content-type': 'application/json' }, body: b ? JSON.stringify(b) : undefined }); return { status: res.status, body: await res.json().catch(() => null) }; };
+ r = await native('GET', `/${row.id}`);
+ assert.deepStrictEqual([r.status, r.body.content_hash], [200, row.content_hash], 'the native API reads an object written over S3');
+ r = await native('POST', '', { namespace: assets.id, kind: 'file', size_bytes: 5, mime_type: 'text/plain', visibility: 'public' });
+ assert.deepStrictEqual([r.status, r.body.code], [422, 'media.zone.private'], 'a named zone takes private objects only');
+ r = await native('POST', '', { namespace: assets.id, kind: 'file', size_bytes: 5, mime_type: 'text/plain' });
+ assert.strictEqual(r.status, 201, JSON.stringify(r.body));
+ const nativeId = r.body.id;
+ await fetch(`http://localhost:${port}/api/v2/${PRJ}/objects/${nativeId}/content`, { method: 'PUT', headers: { authorization: `Bearer ${appTok}`, 'content-type': 'text/plain' }, body: 'hello' });
+ r = await native('POST', `/${nativeId}/complete`, {});
+ assert.strictEqual(r.status, 200, JSON.stringify(r.body));
+ out = await s3.send(new S3.GetObjectCommand({ Bucket: 'assets', Key: nativeId }));
+ assert.strictEqual(await out.Body.transformToString(), 'hello', 'S3 reads a native object of the zone under its id');
+ r = await native('POST', '', { namespace: `zon_${ids.ulid()}`, kind: 'file', size_bytes: 5, mime_type: 'text/plain' });
+ assert.deepStrictEqual([r.status, r.body.code], [404, 'media.zone.unknown'], 'a zone that does not exist is not created by a write');
+ // The default zone lists the environment's native objects under derived keys.
+ r = await native('POST', '', { kind: 'file', size_bytes: 3, mime_type: 'text/plain', namespace: 'avatars' });
+ const legacyId = r.body.id;
+ await fetch(`http://localhost:${port}/api/v2/${PRJ}/objects/${legacyId}/content`, { method: 'PUT', headers: { authorization: `Bearer ${appTok}`, 'content-type': 'text/plain' }, body: 'old' });
+ await native('POST', `/${legacyId}/complete`, {});
+ out = await s3.send(new S3.ListObjectsV2Command({ Bucket: 'default' }));
+ assert.deepStrictEqual((out.Contents || []).map((c) => c.Key), [`avatars/${legacyId}`], 'the default zone lists a native object under its namespace path and id, and nothing of a named zone');
+ out = await s3.send(new S3.GetObjectCommand({ Bucket: 'default', Key: `avatars/${legacyId}` }));
+ assert.strictEqual(await out.Body.transformToString(), 'old');
+ await assert.rejects(s3.send(new S3.PutObjectCommand({ Bucket: 'default', Key: `avatars/${legacyId}`, Body: 'x' })), /InvalidArgument|med_/, 'a derived key cannot be written over S3');
+ console.log('✅ two doors: one object for S3 and the native API; named zones private; the default zone lists native objects');
+
+ // ── Listing ──
+ for (const k of ['photos/2024/a.jpg', 'photos/2024/b.jpg', 'photos/2025/c.jpg', 'photos/top.jpg', 'readme.txt']) {
+ await s3.send(new S3.PutObjectCommand({ Bucket: 'assets', Key: k, Body: k }));
+ }
+ out = await s3.send(new S3.ListObjectsV2Command({ Bucket: 'assets', Prefix: 'photos/', Delimiter: '/' }));
+ assert.deepStrictEqual([(out.Contents || []).map((c) => c.Key), (out.CommonPrefixes || []).map((c) => c.Prefix), out.KeyCount],
+ [['photos/top.jpg'], ['photos/2024/', 'photos/2025/'], 3]);
+ const seen = [];
+ let token;
+ do {
+ out = await s3.send(new S3.ListObjectsV2Command({ Bucket: 'assets', MaxKeys: 2, ContinuationToken: token }));
+ seen.push(...(out.Contents || []).map((c) => c.Key));
+ token = out.NextContinuationToken;
+ assert.strictEqual(!!token, !!out.IsTruncated);
+ } while (token);
+ assert.deepStrictEqual(seen, [nativeId, 'photos/2024/a.jpg', 'photos/2024/b.jpg', 'photos/2025/c.jpg', 'photos/top.jpg', 'readme.txt', 'site/index.html', 'stream/a b+c.bin'].sort((a, b) => (Buffer.compare(Buffer.from(a), Buffer.from(b)))), 'pages of two cover every key once, in byte order');
+ const v1 = await s3.send(new S3.ListObjectsCommand({ Bucket: 'assets', Delimiter: '/', MaxKeys: 2 }));
+ assert.deepStrictEqual([(v1.Contents || []).map((c) => c.Key), (v1.CommonPrefixes || []).map((c) => c.Prefix), v1.IsTruncated], [[nativeId], ['photos/'], true]);
+ const v1b = await s3.send(new S3.ListObjectsCommand({ Bucket: 'assets', Delimiter: '/', Marker: v1.NextMarker }));
+ assert.deepStrictEqual([(v1b.Contents || []).map((c) => c.Key), (v1b.CommonPrefixes || []).map((c) => c.Prefix)], [['readme.txt'], ['site/', 'stream/']], 'ListObjects v1 resumes after a common prefix marker');
+ out = await s3.send(new S3.ListBucketsCommand({}));
+ assert.deepStrictEqual(out.Buckets.map((b) => b.Name), ['assets', 'default', 'logs'], 'ListBuckets: the environment\'s zones, not the sandbox\'s or another project\'s');
+ await s3.send(new S3.HeadBucketCommand({ Bucket: 'assets' }));
+ await assert.rejects(s3.send(new S3.HeadBucketCommand({ Bucket: 'nope' })), (e) => e.$metadata.httpStatusCode === 404);
+ console.log('✅ ListObjectsV2 (prefix, delimiter, pages), ListObjects v1, ListBuckets, HeadBucket');
+
+ // ── Presigned URLs ──
+ const getUrl = await getSignedUrl(s3, new S3.GetObjectCommand({ Bucket: 'assets', Key: 'readme.txt' }), { expiresIn: 60 });
+ let res = await fetch(getUrl);
+ assert.deepStrictEqual([res.status, await res.text()], [200, 'readme.txt'], 'a presigned GET reads');
+ const putUrl = await getSignedUrl(s3, new S3.PutObjectCommand({ Bucket: 'assets', Key: 'up/presigned.txt' }), { expiresIn: 60 });
+ res = await fetch(putUrl, { method: 'PUT', body: 'via presigned put' });
+ assert.strictEqual(res.status, 200, await res.text());
+ out = await s3.send(new S3.GetObjectCommand({ Bucket: 'assets', Key: 'up/presigned.txt' }));
+ assert.strictEqual(await out.Body.transformToString(), 'via presigned put');
+ res = await fetch(getUrl.replace('readme.txt', 'site/index.html'));
+ assert.strictEqual(res.status, 403, 'a presigned URL is for its own key only');
+ const expired = await getSignedUrl(s3, new S3.GetObjectCommand({ Bucket: 'assets', Key: 'readme.txt' }), { expiresIn: 1, signingDate: new Date(Date.now() - 10_000) });
+ res = await fetch(expired);
+ assert.deepStrictEqual([res.status, /Request has expired/.test(await res.text())], [403, true], 'an expired URL is refused');
+ console.log('✅ presigned GET and PUT; a URL reaches its own key only and expires');
+
+ // ── Isolation and permissions ──
+ await assert.rejects(client(scoped).send(new S3.GetObjectCommand({ Bucket: 'assets', Key: 'readme.txt' })), /AccessDenied/, 'a zone-scoped key cannot reach another zone');
+ await client(scoped).send(new S3.PutObjectCommand({ Bucket: 'logs', Key: 'a.log', Body: 'x' }));
+ assert.deepStrictEqual((await client(scoped).send(new S3.ListBucketsCommand({}))).Buckets.map((b) => b.Name), ['logs']);
+ await assert.rejects(client(reader).send(new S3.PutObjectCommand({ Bucket: 'assets', Key: 'nope', Body: 'x' })), /AccessDenied/, 'a read key cannot write');
+ assert.strictEqual(await (await client(reader).send(new S3.GetObjectCommand({ Bucket: 'assets', Key: 'readme.txt' }))).Body.transformToString(), 'readme.txt');
+ await client(sbox).send(new S3.PutObjectCommand({ Bucket: 'assets', Key: 'readme.txt', Body: 'sandbox copy' }));
+ assert.strictEqual(await (await s3.send(new S3.GetObjectCommand({ Bucket: 'assets', Key: 'readme.txt' }))).Body.transformToString(), 'readme.txt', 'the sandbox\'s assets is another bucket');
+ await assert.rejects(client(other).send(new S3.GetObjectCommand({ Bucket: 'assets', Key: 'readme.txt' })), /NoSuchKey/, 'another project\'s assets is another bucket');
+ const forged = new S3.S3Client({ endpoint: `http://127.0.0.1:${port}`, region: 'us-east-1', forcePathStyle: true, credentials: { accessKeyId: full.key.id, secretAccessKey: 'x'.repeat(40) } });
+ await assert.rejects(forged.send(new S3.ListBucketsCommand({})), /SignatureDoesNotMatch/);
+ console.log('✅ isolation: zone-scoped, read-only, sandbox and other-project keys each stay in their place; forged signatures refused');
+
+ // ── Quotas, overwrite, holds, NotImplemented ──
+ await assert.rejects(client(scoped).send(new S3.PutObjectCommand({ Bucket: 'logs', Key: 'big.log', Body: Buffer.alloc(100) })), /QuotaExceeded/, 'the zone\'s quota_bytes is enforced');
+ await s3.send(new S3.PutObjectCommand({ Bucket: 'assets', Key: 'readme.txt', Body: 'v2' }));
+ assert.strictEqual(await (await s3.send(new S3.GetObjectCommand({ Bucket: 'assets', Key: 'readme.txt' }))).Body.transformToString(), 'v2', 'an overwrite serves the new bytes');
+ const versions = await db.all("SELECT lifecycle_status FROM media_objects WHERE zone_id = ? AND object_key = 'readme.txt' ORDER BY created_at", [assets.id]);
+ assert.deepStrictEqual(versions.map((v) => v.lifecycle_status).sort(), ['deleted', 'ready'], 'the previous object went through the deletion path');
+ const current = await db.get("SELECT * FROM media_objects WHERE zone_id = ? AND object_key = 'readme.txt' AND lifecycle_status = 'ready'", [assets.id]);
+ await model.placeHold({ object_id: current.id, kind: 'evidence', reason: 'test' });
+ await assert.rejects(s3.send(new S3.PutObjectCommand({ Bucket: 'assets', Key: 'readme.txt', Body: 'v3' })), /AccessDenied/, 'an overwrite of a held object is refused');
+ await assert.rejects(s3.send(new S3.DeleteObjectCommand({ Bucket: 'assets', Key: 'readme.txt' })), /AccessDenied/, 'so is its deletion');
+ await assert.rejects(s3.send(new S3.CreateMultipartUploadCommand({ Bucket: 'assets', Key: 'mp' })), /NotImplemented/);
+ await assert.rejects(s3.send(new S3.CopyObjectCommand({ Bucket: 'assets', Key: 'copy', CopySource: 'assets/readme.txt' })), /NotImplemented/);
+ await assert.rejects(s3.send(new S3.DeleteBucketCommand({ Bucket: 'assets' })), /NotImplemented|MethodNotAllowed/);
+ console.log('✅ quotas, overwrite through the deletion path, holds refuse overwrite and delete, NotImplemented answers');
+
+ // ── Deleting objects ──
+ await s3.send(new S3.DeleteObjectCommand({ Bucket: 'assets', Key: 'site/index.html' }));
+ await assert.rejects(s3.send(new S3.GetObjectCommand({ Bucket: 'assets', Key: 'site/index.html' })), /NoSuchKey/);
+ r = await native('GET', `/${row.id}`);
+ assert.strictEqual(r.body.lifecycle_status, 'deleted', 'deleted over S3, deleted for the native API too');
+ out = await s3.send(new S3.DeleteObjectsCommand({ Bucket: 'assets', Delete: { Objects: [{ Key: 'photos/2024/a.jpg' }, { Key: 'photos/2024/b.jpg' }, { Key: 'missing' }] } }));
+ assert.deepStrictEqual((out.Deleted || []).map((d) => d.Key).sort(), ['missing', 'photos/2024/a.jpg', 'photos/2024/b.jpg']);
+ out = await s3.send(new S3.ListObjectsV2Command({ Bucket: 'assets', Prefix: 'photos/2024/' }));
+ assert.strictEqual(out.KeyCount, 0);
+ console.log('✅ DeleteObject and DeleteObjects: both doors answer not found');
+
+ // ── Zone deletion: at once NoSuchBucket, objects tombstoned and purged, name retired, keys dropped ──
+ await s3.send(new S3.PutObjectCommand({ Bucket: 'logs', Key: 'b.log', Body: 'y' }));
+ const logsRow = await db.get("SELECT * FROM media_objects WHERE zone_id = ? AND object_key = 'b.log'", [logs.id]);
+ const presignedLog = await getSignedUrl(s3, new S3.GetObjectCommand({ Bucket: 'logs', Key: 'b.log' }), { expiresIn: 600 });
+ r = await provision({ ...logs, state: 'deleting', revision: 2, deletion: { requested_at: new Date().toISOString(), objects_remaining: 2, held_objects: 0, completed_at: null } });
+ assert.deepStrictEqual([r.status, r.body.state], [200, 'deleting']);
+ await assert.rejects(s3.send(new S3.GetObjectCommand({ Bucket: 'logs', Key: 'b.log' })), /NoSuchBucket/, 'a deleting zone answers NoSuchBucket at once');
+ assert.strictEqual((await fetch(presignedLog)).status, 404, 'and so do its presigned URLs');
+ const swept = await zones.sweepDeleting();
+ assert.ok(swept.tombstoned >= 2 && swept.purged >= 2, JSON.stringify(swept));
+ r = await internal('GET', `/internal/zones/${logs.id}`, { token: ZONE_TOKEN });
+ assert.deepStrictEqual([r.body.deletion.objects_remaining, r.body.deletion.held_objects, !!r.body.deletion.completed_at], [0, 0, true], JSON.stringify(r.body));
+ assert.ok(!fs.existsSync(model.objectFilePath(logsRow)), 'its bytes are gone');
+ assert.strictEqual(await model.restore(await model.getObject(logsRow.id)), null, 'a zone deletion is not restorable');
+ r = await provision({ ...logs, state: 'deleted', revision: 3, deletion: { requested_at: new Date().toISOString(), objects_remaining: 0, held_objects: 0, completed_at: new Date().toISOString() } });
+ assert.strictEqual(r.body.state, 'deleted');
+ r = await provision(zoneRecord(PRJ, 'production', 'logs'));
+ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.zone.name_retired'], 'the name is retired, never given to a new zone');
+ assert.strictEqual((await db.get('SELECT revoked_at FROM media_access_keys WHERE id = ?', [scoped.key.id])).revoked_at != null, true, 'a key scoped only to the deleted zone is revoked');
+ assert.strictEqual((await fetch(presignedLog)).status, 404, 'a URL signed for the old zone never reaches anything again');
+ console.log('✅ zone deletion: NoSuchBucket at once, tombstoned and purged, no restore, name retired, scoped keys revoked');
+
+ // ── Revocation ──
+ r = await internal('DELETE', `/api/v1/projects/${PRJ}/access-keys/${reader.key.id}?on_behalf_of=${ADMIN}`, { token: SERVICES_TOKEN });
+ assert.ok(r.body.revoked_at);
+ await assert.rejects(client(reader).send(new S3.ListBucketsCommand({})), /InvalidAccessKeyId/, 'a revoked key is refused at once');
+ console.log('✅ revocation is immediate');
+ } finally {
+ server.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ }
+ console.log('s3 surface and zones: all checks passed');
+ process.exit(0);
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -37,6 +37,7 @@ const crypto = require('crypto');
@@ -37,6 +37,7 @@ const crypto = require('crypto');
MEDIA_B2_APP_KEY: S('b2-app-key'),
MEDIA_B2_APP_KEY: S('b2-app-key'),
MEDIA_R2_ACCESS_KEY_ID: S('r2-access-key-id'),
MEDIA_R2_ACCESS_KEY_ID: S('r2-access-key-id'),
MEDIA_R2_SECRET_ACCESS_KEY: S('r2-secret-access-key'),
MEDIA_R2_SECRET_ACCESS_KEY: S('r2-secret-access-key'),
+ MEDIA_S3_KEY_SECRET: S('s3-key-secret'), // encrypts the S3 access keys' secrets (server/s3/keys.js)
};
};
const sentinels = new Map([
const sentinels = new Map([
...Object.entries(SECRET_ENV),
...Object.entries(SECRET_ENV),
@@ -35,6 +35,7 @@ const https = require('https');
@@ -35,6 +35,7 @@ const https = require('https');
'server/public/dev-data.js': 'the app\'s internal URL (APP_INTERNAL_URLS), ids and encoded names in the path',
'server/public/dev-data.js': 'the app\'s internal URL (APP_INTERNAL_URLS), ids and encoded names in the path',
'server/jobs/pack.js': 'a presigned GET on B2/R2 (operator-configured buckets) for a segment\'s durable copy',
'server/jobs/pack.js': 'a presigned GET on B2/R2 (operator-configured buckets) for a segment\'s durable copy',
'server/objects/routes.js': 'a ranged GET on a presigned B2/R2 URL (operator-configured buckets) for a packed segment\'s chunk',
'server/objects/routes.js': 'a ranged GET on a presigned B2/R2 URL (operator-configured buckets) for a packed segment\'s chunk',
+ 'server/s3/routes.js': 'a ranged GET on a presigned B2/R2 URL (operator-configured buckets) for an S3 GetObject whose bytes are not on local disk',
'server/jobs/previews.js': 'a (ranged) GET on a presigned B2/R2 URL (operator-configured buckets) for a timeline segment, to cut a sprite frame',
'server/jobs/previews.js': 'a (ranged) GET on a presigned B2/R2 URL (operator-configured buckets) for a timeline segment, to cut a sprite frame',
'server/drill.js': 'restore drill: only this server itself',
'server/drill.js': 'restore drill: only this server itself',
'server/me/client.js': 'browser script, same-origin',
'server/me/client.js': 'browser script, same-origin',