openvibers/openvibe.host Commits

Store tenant objects in OpenVibe.Media with local disk as the read cache (HOST_OBJECT_STORE=media; off by default) (#24)

OP OpenVibers committed 6 Oct 2026 · 5c5cba046d · 14 files, +878 −39

* Store tenant objects in OpenVibe.Media with local disk as the read cache (HOST_OBJECT_STORE=media; off by default)

## Changes (file:line)

- `server/storage.js:38` — `MEDIA_SCOPE` / `MEDIA_AUDIENCE`; `:110` `createLocalStore` keeps today's store byte-for-byte and adds the sync `ensure` read path; `:152` `createMediaStore` (local disk = cache, Media = source of truth); `:192` `put` caches locally then writes through with `createObjectsClient` (private, `content_hash`=sha256, `metadata.project_id`); `:211` `ensure` serves cache, else fetches from Media, verifies sha256, then caches; `:231`/`:243` `remove`/`removeProject` delete only what the caller determined unreferenced; `:268` `createBlobStore(root, { media })` — `media` absent ⇒ exactly the old local store.
- `server/app.js:82` — builds `mediaStore` from config only when `HOST_OBJECT_STORE=media` (nothing built at module load); `:93` passes it to `createBlobStore`.
- `server/config.js:62` — `objectStore.mode` (`media`|`local`, default `local`); `:65` `media.url` (`OV_MEDIA_URL`, default `http://127.0.0.1:4100`), `namespace` (`OV_MEDIA_NAMESPACE`, default `host`), `scope`.
- `server/domain/deploys.js:188` — object writes are awaited; a Media refusal fails the deploy `502 storage.object_store` (recorded, with log), never a local-only success; `:329`/`:348` `unlinkBlobs` async so GC deletes the Media object too.
- `server/domain/projects.js:162` — project removal awaits `blobs.removeProject` (deletes the project's Media objects).
- `server/worker.js:40` — orphan sweep removes through the store, so the unreferenced Media object goes with the cache; grace/“no `host_blobs` row” semantics unchanged.
- `server/http/tenant.js:182`/`:213` — `send` is async and gets bytes via `blobs.ensure` (cache hit, or Media fetch + sha256 verify; a mismatch is a 500, never served).
- Docs: `README.md:266` new “Storage” section, plus Model/Serving/Depends-on/Grants/Not-done edits; `.env.example:34`; `docs/threat-review.md` (asset table, serving path, alpha note, residual risk); `STATUS.json:44` `artifactStorage`.
- Tests: `test/stageb/media.js` (fake Media Object API v2), `test/host-storage-media.test.js` (put-through; deploy fails when Media refuses; local-miss re-fetch + tamper refused; per-project dedup; GC only when unreferenced; project removal; unset ⇒ zero Media calls).

## What Host needs from Network

- Audience **`openvibe.media`**, from Host's service principal `svc:host` (its existing `OV_OAUTH_CLIENT_ID`/`OV_OAUTH_CLIENT_SECRET`). Grants (requested as scope):
  - `[host, media.object.upload, openvibe.media]`
  - `[host, media.object.read, openvibe.media]`
  - `[host, media.object.list, openvibe.media]`
  - `[host, media.object.delete, openvibe.media]`
- Namespace **`host`** (Host's own tenant in Media must exist). Objects are stored in the root `host` namespace, keyed per project by `metadata.project_id` with sha256 as `content_hash`; a `ns` grant of `host.*` covers the subtree if child namespaces are ever used. (Confirms Tools' `tools`/`tools.*` convention.)

Note: pinned `openvibe-sdk` v0.26.0 does **not** export `openvibe-sdk/objects`; `createObjectsClient` is exported from `openvibe-sdk/media` (`require('openvibe-sdk/media')`), which is what the store uses.

## Test results

- `nice -n 15 ov test --no-cache test/host-storage-media.test.js test/host-isolation.test.js` → **2 passed**.
- Changed-module/serving tests → **13 passed**: `deploy`, `host-uploads`, `host-preview`, `host-lifecycle`, `host-git-source`, `host-contracts-events`, `host-rollback`, `host-quota-auth`, `host-abuse`, `host-sitemap`, `asset-c

* Media object store: re-check references before any delete, empty files stay local, direct uploads, per-blob serialisation, coalesced streamed re-fetch with a timeout, logged leftovers and sweep failures (review)

All eight findings are fixed, each with a test, and the store changes are exercised by the allowed suites.

**Fixes**

1. **Early delete of a re-used blob** — `remove` now scans for every match, then calls the injected `isReferenced(projectId, sha)` (a prepared `host_blobs` lookup) right before deleting; if a row exists it skips both the Media and cache delete and returns false. Wired in `server/app.js:93-97`, callback in `server/storage.js:325-336`. Missing `await` added at `server/domain/sites.js:116`.
2. **Zero-byte files** — `put` writes length-0 data with `local.put` only and never calls Media; the serving path already short-circuits `size === 0` (`server/storage.js:264`).
3. **Quadratic scan** — a fresh blob (local.put returned true) is uploaded directly; only an already-cached blob is looked up (`server/storage.js:266-272`). The SDK v2 `list` has no `content_hash`/metadata filter, noted in the comment.
4. **Duplicate Media objects** — puts are serialised per `(project, sha)` by an in-process promise map (`server/storage.js:201-209,289`), and `remove` deletes every matching object, not just the first (`server/storage.js:327-332`).
5. **Read path** — concurrent `ensure` misses share one in-flight promise (`server/storage.js:312`), the download streams into `tmp` while hashing then renames (`server/storage.js:228-257`), and the raw fetch carries `AbortSignal.timeout` (`server/storage.js:234`).
6. **Failed deploy** — the shas of objects left behind are logged at error and recorded in the deploy log (`server/domain/deploys.js:196-201`).
7. **Worker sweep** — a `storage.media` delete failure is logged at warn with project and sha (`server/worker.js:41-47`).
8. **Docs** — removed the "never deleted" overclaim; README now states the `host_blobs` re-check, the cache-only empty files, and that the sweep reaps only locally cached blobs (`README.md:285-292`), same precision in the `server/storage.js` header (`:22-27`) and `docs/threat-review.md`.

**Tests** — added 9 checks to `test/host-storage-media.test.js` (zero-byte, direct-upload/cached-lookup, concurrent-put coalescing, remove-every-duplicate, re-reference guard, ensure coalescing/tmp-rename, fetch timeout, failed-deploy sha log, worker warn), plus fake-Media controls in `test/stageb/media.js`.

**Results**
```
nice -n 15 ov test --no-cache test/host-storage-media.test.js test/host-isolation.test.js \
  test/deploy.test.js test/host-uploads.test.js test/host-lifecycle.test.js
→ 5 passed in 12s
```
(`host-storage-media.test.js` runs 17 checks, all green.)

Changes

ddiff --git a/.env.example b/.env.example
iindex 9015bea..e8457ae 100644
---- a/.env.example
++++ b/.env.example
@@@ -31,6 +31,15 @@ VALKEY_PREFIX=
HOST_DB_PATH=./data/host.db
HOST_STORAGE_DIR=./data/objects
++# Object storage. Unset/empty: local disk only (data/objects), exactly as before. `media`: every
++# deploy object is also written through to OpenVibe.Media (private, Host's namespace `host`, keyed
++# per project) and local disk becomes the read cache. Needs OV_MEDIA_URL and OV_OAUTH_CLIENT_SECRET
++# (Host's Network service principal with media.object.upload/read/list/delete for namespace `host.*`,
++# and a `host` tenant in Media).
++HOST_OBJECT_STORE=
++OV_MEDIA_URL=http://127.0.0.1:4100
++#OV_MEDIA_NAMESPACE=host
++
# OpenVibe.Network — SSO (OAuth2), JWKS, client-credentials tokens
OV_NETWORK_URL=https://openvibe.network
OV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000
ddiff --git a/README.md b/README.md
iindex 13fa2de..ed484c2 100644
---- a/README.md
++++ b/README.md
@@@ -53,7 +53,7 @@ Stage A adds a few safety rules of its own:
- OpenVibe.Network (Stage B): SSO for the dashboard, the JWKS that verifies user and service tokens, client-credentials tokens for the outbox relay.
- OpenVibe.Events (Stage B): `host.*` events through the `openvibe-sdk` v0.26.0 transactional outbox (openvibe-sdk/limits for the per-actor limits).
- `openvibe-shared` v2.6.0 (Stage B): shared chrome (Host's own pages are composed with `openvibe-shared/shell`), legal pages, `/release.json`, `/metrics`, `/api/ready`.
--- OpenVibe.Media: not yet. The roadmap stores artifacts "through Media where practical"; Stage B keeps them on local disk for now (see [Not done yet](#not-done-yet-stage-b)).
++- OpenVibe.Media (Stage B, opt-in): with `HOST_OBJECT_STORE=media`, deploy objects are written through to Media's Object API v2 (`openvibe-sdk/media`) and local disk becomes the read cache; unset keeps them on local disk (see [Storage](#storage)).
## Capabilities
@@@ -258,11 +258,48 @@ A Node service (`server/`, Express 4, better-sqlite3, port **4910**, service id
- **Projects** (`prj_<ULID>`) have an owner (a `usr_` subject), an environment (`production` or `sandbox`), members with a role (`owner` > `maintainer` > `deployer`), and optionally the `network_project_id` of the OpenVibe.Network project (ADR-014: Network owns projects; until it has them, Host creates one for its owner). Every Host row and every stored object is keyed by the project id.
- **Sites** (`site_<ULID>`) have a name (one DNS label, 3–40 characters, reserved names refused) that is their default host, `<name>.openvibe.host`, and a pointer to the active deploy. A deleted site's name is held for 30 days so nobody else can serve content on links that still point there.
--- **Deploys** (`dpl_<ULID>`) are immutable artifacts: a manifest (`host.deploy-manifest@1`: path, sha256, size and content type of every file, file count, total bytes) with its own sha256, file rows, and the upload/validation log. Database triggers refuse any update of a deploy's artifact columns or file rows. The bytes are content-addressed per project: `<HOST_STORAGE_DIR>/projects/<prj_…>/<aa>/<sha256>`.
++- **Deploys** (`dpl_<ULID>`) are immutable artifacts: a manifest (`host.deploy-manifest@1`: path, sha256, size and content type of every file, file count, total bytes) with its own sha256, file rows, and the upload/validation log. Database triggers refuse any update of a deploy's artifact columns or file rows. The bytes are content-addressed per project: `<HOST_STORAGE_DIR>/projects/<prj_…>/<aa>/<sha256>`, with `HOST_OBJECT_STORE=media` writing them through to OpenVibe.Media as well ([Storage](#storage)).
- **Domains**: every site has its default domain. A custom domain is added as `pending` and becomes `verified` (and served) once `_openvibe-host.<hostname>` has the TXT record `openvibe-host-verification=<token>`. A verified name belongs to one site. OpenVibe domains (every domain in the released service manifests, `openvibe.<tld>`, the sites domain itself) can never be claimed. The worker re-checks pending domains (they fail after `HOST_DOMAIN_PENDING_DAYS`) and re-checks verified ones daily: a TXT record gone for `HOST_DOMAIN_LAPSE_DAYS` lapses the domain and it stops being served.
- **Quotas** per project, enforced by Host: storage bytes (objects stored once per project, so re-uploading unchanged files costs nothing), deploys per rolling 24 hours (checked before the body is read; failed uploads count), files per deploy, bytes per file, sites, custom domains. A deploy's own size before deduplication may not exceed the storage quota. Defaults come from `HOST_QUOTA_*` / `HOST_SANDBOX_QUOTA_*`; staff override them per project. Sandbox projects get smaller quotas, no custom domains, and `X-Robots-Tag: noindex`.
- **Build logs**: Stage B has no build step. Each deploy's log records what was received, every validation problem, what was stored, the manifest digest and the activation, and says that nothing was executed. Refused uploads are recorded as `failed` deploys, so their logs are visible too.
++### Storage
++
++Deploy objects are content-addressed and stored **per project** (ADR-014):
++`<HOST_STORAGE_DIR>/projects/<prj_…>/<aa>/<sha256>`. The same bytes uploaded by two projects are
++stored twice, and storage is accounted and deleted per project. Every read goes through
++`server/storage.js` and is keyed by a validated project id and the manifest's sha256, so a request
++can only ever name its own project's objects.
++
++With **`HOST_OBJECT_STORE=media`** (opt-in; unset is the local store above, unchanged) local disk
++becomes the read cache and **OpenVibe.Media** is the source of truth:
++
++- every object is written through to Media's Object API v2 (`openvibe-sdk/media`
++ `createObjectsClient`) as a **private** object in Host's own namespace (`OV_MEDIA_NAMESPACE`,
++ default `host`), keyed per project by `metadata.project_id` with the file's sha256 as
++ `content_hash`. A deploy is not reported as stored until Media acknowledged every object; a Media
++ failure fails the deploy (`502 storage.object_store`) with its log, never a silent local-only
++ success;
++- serving reads the local cache; a miss re-fetches the object from Media, verifies its sha256 and
++ only then caches and serves it (a mismatch is a `500`, never served);
++- an object is deleted from Media exactly where the local store would delete the blob (deploy GC,
++ project removal, the hourly orphan sweep), and the delete re-checks `host_blobs` just before it
++ runs, so an object a deploy re-referenced while the delete was in flight keeps its Media copy;
++- an empty file (0 bytes) is stored in the local cache only: Media's upload refuses empty
++ objects, and the serving path never asks Media for a size-0 file;
++- the hourly orphan sweep walks the **local cache files** only, so a Media object with no cache
++ file (for example a duplicate left by two uploads that raced before puts were serialised) is
++ not reaped by the sweep; project removal deletes it, and puts are serialised per
++ `(project, sha256)` so two deploys sharing a new sha do not create a duplicate in the first
++ place.
++
++Host authenticates to Media with its Network service principal (client credentials, audience
++`openvibe.media`), which must hold `media.object.upload`, `media.object.read`, `media.object.list`
++and `media.object.delete` for the `host` namespace and everything below it (`host.*`), and a `host`
++tenant must exist in Media. `OV_MEDIA_URL` is the internal Media base URL (default
++`http://127.0.0.1:4100`). `ovhost backup host` still archives the local cache; in the Media mode the
++Media objects are the durable copy.
++
### Uploads
`POST /api/v1/sites/:id/deploys[?activate=1][&root=dist]` accepts a tar or tar.gz archive (`Content-Type: application/gzip`, `application/x-tar`, `application/octet-stream`) or `multipart/form-data` (`archive=<one archive>` or `files=<file>…`, each part's filename being the path; `strip=folder` removes a browser folder upload's shared top folder). The archive is parsed in memory and never extracted. The whole upload is refused, and recorded as a failed deploy, when any of these is true:
@@@ -277,7 +314,7 @@ A Node service (`server/`, Express 4, better-sqlite3, port **4910**, service id
### Serving
--Requests resolve their site once from the `Host` header, read the site's active deploy pointer once, and then read only that deploy's immutable rows. The file's bytes are opened at a path built from the site's project id and the manifest's sha256, never from the URL, so no path, encoding or `Host` trick can reach another project's objects. Absolute-form request targets must name the same host as the `Host` header.
++Requests resolve their site once from the `Host` header, read the site's active deploy pointer once, and then read only that deploy's immutable rows. The file's bytes come from the blob store keyed by the site's project id and the manifest's sha256 — the local disk cache, or, with `HOST_OBJECT_STORE=media`, the object fetched back from Media and sha256-verified ([Storage](#storage)) — never from the URL, so no path, encoding or `Host` trick can reach another project's objects. Absolute-form request targets must name the same host as the `Host` header.
- `GET`/`HEAD` only (405 otherwise). `/` and `/dir/` serve `index.html`; `/page` also tries `page.html`; `/dir` redirects to `/dir/` when `dir/index.html` exists.
- `Content-Type` from the manifest; strong `ETag` (the sha256) with `304` on `If-None-Match`; single byte ranges (`206`/`416`).
@@@ -384,6 +421,7 @@ Server-rendered pages with the shared chrome (`openvibe-shared` v2.6.0 `shell.pa
- OpenVibe.Network OAuth client **`host`**, redirect `https://openvibe.host/auth/callback`, scope `profile theme`. The same client is the service principal `svc:host`.
- Grant `[host, events.event.publish, openvibe.events]`.
++- With `HOST_OBJECT_STORE=media`: grants `[host, media.object.upload, openvibe.media]`, `[host, media.object.read, openvibe.media]`, `[host, media.object.list, openvibe.media]` and `[host, media.object.delete, openvibe.media]`, covering the `host` namespace and its subtree (`host.*`), plus a `host` tenant in Media. Network-side; Host asks for the verbs as its scope.
- Callers of Host get `[<client>, host.site.manage | host.deploy.create | host.domain.manage, openvibe.host]` as needed. None exist yet (Codes, the expected first caller, is not built).
- Released in `openvibe-contracts` v0.83.0 (CI contract check blocking); v0.32.0 adds the takedown routes to `host.site.manage.implementedBy`.
@@@ -404,11 +442,11 @@ The service runs on the host (loopback, since 2026-09-23) and the pending tenant
### Not done yet (Stage B)
- Uploads are held in memory while they are validated (bounded by `HOST_MAX_UPLOAD_BYTES` and `HOST_MAX_UNPACKED_BYTES`), so the service needs that much headroom per concurrent upload.
--- Objects live on the service's local disk, not in OpenVibe.Media; there is no replication. `ovhost backup host` archives `/var/lib/openvibe-host-api/objects` (the inventory's `objects` entry) alongside `ov_host`, uploads the encrypted archive off-host, and a restore drill extracts it and checks every blob's sha256. The database (`ov_host` on the host's data role, ADR-035; schema in [migrations/](migrations/)) is backed up with the others by pgBackRest.
++- Objects are on the service's local disk by default; there is no replication. `HOST_OBJECT_STORE=media` (opt-in) additionally writes every object through to OpenVibe.Media and keeps local disk as the read cache. `ovhost backup host` archives `/var/lib/openvibe-host-api/objects` (the inventory's `objects` entry) alongside `ov_host`, uploads the encrypted archive off-host, and a restore drill extracts it and checks every blob's sha256. The database (`ov_host` on the host's data role, ADR-035; schema in [migrations/](migrations/)) is backed up with the others by pgBackRest.
- Certificates for custom domains are issued and installed by `ovhost certs renew --install` (step 8) on the `openvibe-certs.timer`; certbot stays outside the Host service and no key file is read.
- Projects are created in Host. When Network has projects (ADR-014), Host should accept only Network project ids and read membership from Network.
- `openvibe.host` is not on the Public Suffix List, so a tenant page can still set `Domain=openvibe.host` cookies (a cookie bomb breaks the dashboard and other tenant sites for that visitor). The protections above do not depend on it. [docs/threat-review.md §5](docs/threat-review.md#5-the-public-suffix-list-question) records the decision (launch without it) and the recommended follow-up (dashboard off the tenant zone, then list the zone).
--- Tenant objects are in `ovhost backup`: the inventory declares them as `objects` (`/var/lib/openvibe-host-api/objects`), the backup archives the tree, the off-host copy encrypts it, and a drill extracts it and checks every blob's sha256.
++- Tenant objects are in `ovhost backup`: the inventory declares them as `objects` (`/var/lib/openvibe-host-api/objects`), the backup archives the tree, the off-host copy encrypts it, and a drill extracts it and checks every blob's sha256. With `HOST_OBJECT_STORE=media`, this archives the read cache and Media holds the durable objects.
- Git deploys are Phase 1 only (see [Git deploys](#git-deploys-phase-1)): the project's own CI builds and posts the output. Host never clones, fetches, holds a repository credential or builds; in-Host builds belong to Run (plan T14). There is no webhook: a push reaches Host only through the CI's call.
- Tenant sites publish their own sitemap/robots/feed; when a deploy ships no `sitemap.xml` or `robots.txt`, Host generates one (the manifest's HTML pages on the host the request came in on, so a verified custom domain gets its own `sitemap.xml`; sandbox projects get `Disallow: /` and are `noindex`). A tenant's own file at either path always wins. The dashboard host serves `/robots.txt` and `/sitemap.xml` (front page and legal pages).
- The Codes portal (Wave 20) does not exist, so there is no public developer onboarding for Host yet.
ddiff --git a/STATUS.json b/STATUS.json
iindex 146fc81..aa5e10f 100644
---- a/STATUS.json
++++ b/STATUS.json
@@@ -41,7 +41,7 @@
"B": "alpha: running on the host on loopback :4910 (2026-09-23 audit), not launched; launch-rule items met in code and tests, launch steps in docs/launch.md, threat review in docs/threat-review.md",
"C": "not started (deliberately: blocked until isolation and metering are proven)"
},
-- "artifactStorage": "local disk, content-addressed per project (not OpenVibe.Media yet)",
++ "artifactStorage": "local disk, content-addressed per project; HOST_OBJECT_STORE=media writes objects through to OpenVibe.Media with local disk as the read cache (opt-in, off by default)",
"customDomainCertificates": "operator-issued (certbot), not automated",
"installedOnHost": true,
"deployed": "loopback only, not launched",
ddiff --git a/docs/threat-review.md b/docs/threat-review.md
iindex e9d23e0..e2fcf5b 100644
---- a/docs/threat-review.md
++++ b/docs/threat-review.md
@@@ -18,7 +18,7 @@ Status column says which risks are **fixed**, **mitigated**, **accepted** (with
| Asset | Where it lives |
|---|---|
--| Other tenants' files, deploy history, logs and domains | `host_*` tables in `/var/lib/openvibe-host-api/host.db`; objects in `<HOST_STORAGE_DIR>/projects/<prj_…>/<aa>/<sha256>` |
++| Other tenants' files, deploy history, logs and domains | `host_*` tables in the `host` database; objects in `<HOST_STORAGE_DIR>/projects/<prj_…>/<aa>/<sha256>`, or, with `HOST_OBJECT_STORE=media`, in Host's own namespace in OpenVibe.Media (local disk is then the read cache) |
| OpenVibe sessions (Network SSO tokens, other products' cookies) | browsers; Network |
| The dashboard session and form tokens | `__Host-ov_host_session` cookie (`server/auth/sso.js:59-115`); `HOST_FORM_SECRET` |
| Host's own credentials (`OV_OAUTH_CLIENT_SECRET`, `HOST_FORM_SECRET`) | `/etc/openvibe/host.env` (0600) |
@@@ -50,9 +50,10 @@ It reads the site's active deploy pointer once, then only that deploy's immutabl
percent-decoded segment by segment, and encoded separators (`%2f %5c %00`) are refused
(`candidates`, `:108`). The decoded path must pass the same `checkPath` as uploads
(`server/artifacts/paths.js:26`), and is then looked up in the manifest table, never on the
--filesystem. The bytes are opened at `blobs.pathFor(row.project_id, row.sha256)` (`:154`,
--`server/storage.js:35`): a validated `prj_` id plus a validated lowercase sha256. That path is never
--built from the URL. Absolute-form request targets must name the same host as `Host` (`:186-196`).
++filesystem. The bytes come from `blobs.ensure(row.project_id, row.sha256)`: a validated `prj_` id plus a
++validated lowercase sha256, which is the local cache file, or (with `HOST_OBJECT_STORE=media`) the
++object fetched back from Media, sha256-verified before it is cached or served (`server/storage.js`).
++That key is never built from the URL. Absolute-form request targets must name the same host as `Host` (`:186-196`).
Identical bytes uploaded by two projects are stored twice. Deleting one project's copy can never
break another's, and no tenant can confirm that another tenant has some content by uploading it.
@@@ -229,9 +230,13 @@ Cloudflare. **Added:** every template ovhost renders now sends `X-Forwarded-For`
request reaching the origin directly could choose the address that rate limits and logs use. The
installed vhosts were fixed by hand on 2026-09-23 (`30592c0`). `TRUST_PROXY` now defaults to 1.
--Accepted for alpha: objects live only on the local disk and are not in `ovhost backup` (only
--`host.db` is). A lost disk loses tenant sites, and tenants keep their source. README "Not done yet"
--says so.
++Objects live on the local disk (archived by `ovhost backup`, whose drill checks every blob's
++sha256), or, with `HOST_OBJECT_STORE=media`, are written through to OpenVibe.Media with local disk
++as the read cache. Tenants keep their source either way. A Media delete re-checks `host_blobs`
++just before it runs, so a deploy that re-referenced a blob mid-sweep is not left without its
++durable copy; the hourly orphan sweep still walks only the local cache, so a Media-only orphan
++is left to project removal. Empty files are cache-only (Media's upload refuses 0 bytes). README
++[Storage](../README.md#storage).
## 7. Abuse and takedown
@@@ -285,6 +290,6 @@ the history of any site.
| Unverified or lapsed custom domains reach nginx's default server | medium (confusion, mis-served content on a stranger's domain) | operator: catch-all `default_server` (docs/launch.md step 0) |
| Cookie bomb across tenants and the dashboard (no PSL) | low (per-visitor nuisance; takedown exists) | owner decision (§5) |
| No abuse mailbox/owner | medium (process, not code) | owner |
--| Objects not backed up | medium for tenants (alpha, documented) | fixed: the inventory's `objects` entry archives the store, the off-host copy encrypts it, and a drill restores it and checks every blob's sha256 |
++| Objects not backed up | medium for tenants (alpha, documented) | fixed: the inventory's `objects` entry archives the store, the off-host copy encrypts it, and a drill restores it and checks every blob's sha256. With `HOST_OBJECT_STORE=media` the archive is the read cache and Media holds the objects |
| Decompressing an upload blocked the event loop (`gunzipSync`) | done: async `zlib.gunzip` on the libuv threadpool (`server/artifacts/archive.js`), same `maxOutputLength` cap | done |
| Takedown routes not in released `openvibe-contracts` `host.site.manage.implementedBy` | none at runtime (the contract check passes) | Contracts: next release (docs/launch.md) |
ddiff --git a/server/app.js b/server/app.js
iindex 8478059..a91f6a8 100644
---- a/server/app.js
++++ b/server/app.js
@@@ -76,7 +76,25 @@ async function createApp(opts = {}) {
const log = opts.log || console;
// PostgreSQL (ADR-035): opened and migrated here unless the caller (a test, a script) hands in a store.
const store = opts.store || await openStore(config, { now: opts.now, log });
-- const blobs = createBlobStore(opts.storageDir || config.storageDir);
++ // The object store: local disk, written through to OpenVibe.Media when HOST_OBJECT_STORE=media.
++ // The Media client is built here (not at module load) and fetches a token lazily; unset config
++ // leaves media null and the store is exactly the local one.
++ const mediaStore = config.objectStore.mode === 'media' ? {
++ url: config.media.url,
++ namespace: config.media.namespace,
++ scope: config.media.scope,
++ clientId: config.oauth.clientId,
++ clientSecret: config.oauth.clientSecret,
++ tokenUrl: `${config.networkInternalUrl}/oauth/token`,
++ ...(opts.fetchImpl ? { fetchImpl: opts.fetchImpl } : {}),
++ } : null;
++ // The Media delete re-checks this just before it runs: a deploy may have re-used the sha since
++ // the caller decided the blob was unreferenced, and Media holds the durable copy.
++ const blobReferenced = store.db.prepare('SELECT 1 FROM host_blobs WHERE project_id = ? AND sha256 = ?');
++ const blobs = createBlobStore(opts.storageDir || config.storageDir, {
++ media: mediaStore,
++ isReferenced: async (projectId, sha256) => Boolean(await blobReferenced.get(projectId, sha256)),
++ });
require('./http/errors').setLogger(log);
// IndexNow (openvibe-shared/indexnow): created once at boot from INDEXNOW_KEY. Unset → off, nothing
ddiff --git a/server/config.js b/server/config.js
iindex 4debfc9..1d285b1 100644
---- a/server/config.js
++++ b/server/config.js
@@@ -54,6 +54,20 @@ function load(env = process.env) {
// Content-addressed deploy objects, one directory per project (tenancy keyed by project id).
storageDir: env.HOST_STORAGE_DIR || './data/objects',
++ // Tenant objects: local disk is the store and the read cache. HOST_OBJECT_STORE=media writes
++ // every object through to OpenVibe.Media as well (Host's own namespace `host`, keyed per
++ // project, sha256 as content_hash) and serves a local miss from there; unset keeps the local
++ // store exactly as it is. OV_MEDIA_URL is the internal Media base URL; the credential is
++ // Host's usual Network service principal (OV_OAUTH_CLIENT_ID/SECRET), audience openvibe.media.
++ objectStore: {
++ mode: String(env.HOST_OBJECT_STORE || '').toLowerCase() === 'media' ? 'media' : 'local',
++ },
++ media: {
++ url: trim(env.OV_MEDIA_URL || 'http://127.0.0.1:4100'),
++ namespace: String(env.OV_MEDIA_NAMESPACE || 'host').trim() || 'host',
++ scope: 'media.object.upload media.object.read media.object.list media.object.delete',
++ },
++
// OpenVibe.Network: SSO (OAuth2 authorization server), JWKS, client-credentials tokens.
networkUrl: trim(env.OV_NETWORK_URL || 'https://openvibe.network'),
networkInternalUrl: trim(env.OV_NETWORK_INTERNAL_URL || 'http://127.0.0.1:4000'),
ddiff --git a/server/domain/deploys.js b/server/domain/deploys.js
iindex 6f146fd..d8f528e 100644
---- a/server/domain/deploys.js
++++ b/server/domain/deploys.js
@@@ -163,8 +163,8 @@ function createDeploys({ store, config = null, access, projects, sites, blobs, o
const isPreview = preview === true;
const storedSource = isPreview && !isGit ? 'preview' : source;
if (isPreview) activate = false;
-- const fail = async (status, code, problems) => {
-- const r = await recordFailure(viewer, ctx, { source: storedSource, code, problems, notes, traceparent });
++ const fail = async (status, code, problems, extraNotes = []) => {
++ const r = await recordFailure(viewer, ctx, { source: storedSource, code, problems, notes: [...notes, ...extraNotes], traceparent });
throw new ApiError(status, code, problems[0] ? problems[0].message : code, { deploy_id: r.deploy.id, log: r.log.map((l) => `${l.level}: ${l.message}`) });
};
@@@ -180,8 +180,26 @@ function createDeploys({ store, config = null, access, projects, sites, blobs, o
await fail(413, 'quota.storage', [{ code: 'quota.storage', message: `this deploy adds ${fmtBytes(newBytes)}; the project stores ${fmtBytes(used)} of ${fmtBytes(limits.quota.storageBytes)}` }]);
}
-- // Objects first (idempotent, content-addressed); rows only after they are durable.
-- for (const f of fresh.values()) blobs.put(project.id, f.sha256, f.data);
++ // Objects first (idempotent, content-addressed); rows only after they are durable. With the
++ // Media store this is where a deploy waits for Media to acknowledge every object: a refusal
++ // fails the deploy, recorded with its log, instead of publishing a local-only success.
++ const stored = [];
++ let current = null;
++ try {
++ for (const f of fresh.values()) {
++ current = f.sha256;
++ await blobs.put(project.id, f.sha256, f.data);
++ stored.push(f.sha256);
++ }
++ } catch (err) {
++ // Name every object this failed deploy left behind so an orphan is traceable: the ones
++ // Media already acknowledged, plus the one being written when it refused.
++ const left = [...stored, ...(current && !stored.includes(current) ? [current] : [])];
++ log.error('[Host] object store refused a deploy:', project.id, err.code || err.message,
++ `objects left by the failed deploy: ${left.length ? left.join(' ') : 'none'}`);
++ await fail(502, 'storage.object_store', [{ code: 'storage.object_store', message: 'the object store could not store this deploy; nothing was published' }],
++ [`object store: objects left by the failed deploy, which no deploy row references (a sweep or the next deploy reclaims them): ${left.join(' ') || 'none'}`]);
++ }
const id = newId('deploy', store.now());
const now = store.now();
@@@ -319,9 +337,9 @@ function createDeploys({ store, config = null, access, projects, sites, blobs, o
return freed;
}
-- function unlinkBlobs(projectId, shas) {
++ async function unlinkBlobs(projectId, shas) {
for (const sha of shas) {
-- try { blobs.remove(projectId, sha); } catch (err) { log.warn('[Host] could not remove object:', err.message); }
++ try { await blobs.remove(projectId, sha); } catch (err) { log.warn('[Host] could not remove object:', err.message); }
}
}
@@@ -338,7 +356,7 @@ function createDeploys({ store, config = null, access, projects, sites, blobs, o
await q.clearPreviewIf.run(site.id, deploy.id);
freed = await collectGarbage(project.id);
});
-- unlinkBlobs(project.id, freed);
++ await unlinkBlobs(project.id, freed);
return { deleted: true, objects_removed: freed.length };
}
ddiff --git a/server/domain/projects.js b/server/domain/projects.js
iindex c4ee73a..3be56c2 100644
---- a/server/domain/projects.js
++++ b/server/domain/projects.js
@@@ -159,7 +159,7 @@ function createProjects({ store, config, access, blobs, takedowns, log = console
await db.prepare('DELETE FROM host_domains WHERE project_id = ?').run(project.id);
await q.markDeleted.run(now, now, project.id);
});
-- try { blobs.removeProject(project.id); } catch (err) { log.warn('[Host] could not remove project objects:', err.message); }
++ try { await blobs.removeProject(project.id); } catch (err) { log.warn('[Host] could not remove project objects:', err.message); }
// IndexNow: every site of the project left the index (never a sandbox project, which is noindex).
if (project.environment !== 'sandbox') for (const s of doomed) announce(`${s.name}.${config.sitesDomain}`, ['/', '/sitemap.xml']);
return { deleted: true };
ddiff --git a/server/domain/sites.js b/server/domain/sites.js
iindex 79ce57d..893116e 100644
---- a/server/domain/sites.js
++++ b/server/domain/sites.js
@@@ -111,7 +111,9 @@ function createSites({ store, config, access, projects, takedowns, indexnow = nu
await markDeleted(site, now);
freed = await deploys.collectGarbage(project.id);
});
-- deploys.unlinkBlobs(project.id, freed);
++ // Await it: with the Media store the deletes are slow and detached otherwise, and the caller
++ // (project/site removal) should finish only once the durable objects are gone.
++ await deploys.unlinkBlobs(project.id, freed);
announceSite(site, project);
return { deleted: true };
}
ddiff --git a/server/http/tenant.js b/server/http/tenant.js
iindex 0d06a74..4a8bc24 100644
---- a/server/http/tenant.js
++++ b/server/http/tenant.js
@@@ -8,8 +8,9 @@
*
* A request resolves its site ONCE from the Host header, reads the site's active deploy pointer
* ONCE, and from then on reads only immutable rows of that deploy (host_deploy_files). The file's
-- * bytes come from <storage>/projects/<that site's project>/<sha256>: the path is built from the
-- * project id and the manifest's hash, never from the URL, so no URL, encoding or Host trick can
++ * bytes come from the blob store keyed by the project id and the manifest's hash — the local disk
++ * cache, or, with HOST_OBJECT_STORE=media, the object fetched back from OpenVibe.Media and verified
++ * (server/storage.js). The key is never built from the URL, so no URL, encoding or Host trick can
* reach another project's objects.
*
* Responses: GET/HEAD only; Content-Type from the manifest; strong ETag (the sha256); 304 on
@@@ -178,7 +179,7 @@ function createTenantServer({ store, config, blobs, takedowns = null, siteConfig
return { list: [p, `${p}.html`], dirIndex: `${p}/index.html`, dir: false, path: p };
}
-- function send(req, res, site, row, { status = 200, cache, deployId = site.active_deploy_id }) {
++ async function send(req, res, site, row, { status = 200, cache, deployId = site.active_deploy_id }) {
const etag = `"${row.sha256}"`;
res.setHeader('ETag', etag);
res.setHeader('Content-Type', row.content_type);
@@@ -203,9 +204,19 @@ function createTenantServer({ store, config, blobs, takedowns = null, siteConfig
} else {
res.statusCode = status;
}
-- res.setHeader('Content-Length', row.size === 0 ? 0 : end - start + 1);
-- if (req.method === 'HEAD' || row.size === 0) return res.end();
-- const file = blobs.pathFor(row.project_id, row.sha256);
++ const length = row.size === 0 ? 0 : end - start + 1;
++ if (req.method === 'HEAD' || row.size === 0) { res.setHeader('Content-Length', length); return res.end(); }
++ // The local cache, or (Media store) the object fetched from Media, its sha256 verified, and
++ // cached. A miss that cannot be satisfied is a 500: bytes of the wrong content are never served.
++ let file;
++ try {
++ file = await blobs.ensure(row.project_id, row.sha256);
++ } catch (err) {
++ log.error('[Host] object missing or unreadable:', row.project_id, row.sha256, err.code || err.message);
++ if (!res.headersSent) { res.removeHeader('Content-Range'); return plain(res, 500, 'This file could not be read. Try again later.'); }
++ return res.destroy();
++ }
++ res.setHeader('Content-Length', length);
const stream = fs.createReadStream(file, { start, end });
stream.on('error', (err) => {
log.error('[Host] object missing or unreadable:', row.project_id, row.sha256, err.code || err.message);
ddiff --git a/server/storage.js b/server/storage.js
iindex 4246bf3..ca162f2 100644
---- a/server/storage.js
++++ b/server/storage.js
@@@ -10,17 +10,56 @@
* directory. The same bytes uploaded by two projects are stored twice: storage is accounted and
* deleted per project (ADR-014: tenancy keyed by project id).
*
-- * Stage B keeps objects on local disk. The roadmap prefers OpenVibe.Media "where practical"; that
-- * adapter does not exist yet (see README).
++ * Two modes, selected by config (HOST_OBJECT_STORE):
++ *
++ * local (default, and what an unset HOST_OBJECT_STORE gets) — the layout above, exactly as before.
++ * media — the same local layout is the READ cache, and every put is written through to
++ * OpenVibe.Media's Object API v2 (openvibe-sdk/media createObjectsClient) as a private
++ * object in Host's own namespace, keyed per project with the file's sha256 as content_hash.
++ * A deploy is not reported as stored until Media acknowledged the write; a Media failure
++ * rejects the put and the deploy fails, never a silent local-only success. ensure() serves
++ * the local cache and, on a miss, streams the object back from Media while hashing it,
++ * verifies the sha256 and only then renames it into the cache and serves it. Media objects
++ * are deleted only where the local store would delete the blob today (deploy GC, project
++ * removal, the worker's orphan sweep), and the delete re-checks host_blobs just before it
++ * runs, so a Media object a deploy has since re-referenced is not deleted. The worker's
++ * sweep walks only locally cached blobs, so a Media-only orphan is left to project
++ * removal. A zero-byte file is cache-only (Media's upload refuses empty objects) and is
++ * never fetched. A put is serialised per (project, sha256) and remove deletes every
++ * matching object, so two deploys sharing a new sha cannot leave a duplicate behind.
++ *
++ * Nothing in this module reaches the network on load: the Media client is built inside
++ * createMediaStore(), only when the Media store is selected, and it fetches a token lazily.
*/
const crypto = require('crypto');
const fs = require('fs');
const path = require('path');
++const { Readable, Transform } = require('stream');
++const { pipeline } = require('stream/promises');
const { isId } = require('./ids');
const SHA_RE = /^[0-9a-f]{64}$/;
++// What Host's Network service principal must be granted for audience openvibe.media, in Host's own
++// namespace (`host` / `host.*`). Media's v2 object API checks a verb per namespace: upload, read,
++// list, delete. Media's older ids keep granting the newer verbs, but asking for each verb is exact.
++const MEDIA_SCOPE = 'media.object.upload media.object.read media.object.list media.object.delete';
++const MEDIA_AUDIENCE = 'openvibe.media';
++// A Media download that has not answered within this long is abandoned and the read fails with a
++// clean 500 instead of hanging. The SDK client has its own timeouts; this raw fetch does not.
++const MEDIA_FETCH_TIMEOUT_MS = 30 * 1000;
++
++const sha256Of = (data) => crypto.createHash('sha256').update(data).digest('hex');
++
++/** A put/ensure/delete failure against Media, with a stable code. Never carries a secret. */
++function mediaFailure(action, err) {
++ const e = new Error(err && err.message ? `${action}: ${err.message}` : action);
++ e.code = 'storage.media';
++ if (err) e.cause = err;
++ return e;
++}
--function createBlobStore(root) {
++/** The local, content-addressed store. `ensure` is the synchronous read path (a cache hit is the file). */
++function createLocalStore(root) {
const base = path.resolve(root);
const projectsDir = path.join(base, 'projects');
const tmpDir = path.join(base, 'tmp');
@@@ -40,7 +79,7 @@ function createBlobStore(root) {
/** Store bytes whose sha256 the caller computed; verified again here. Idempotent. */
function put(projectId, sha256, data) {
const dest = pathFor(projectId, sha256);
-- const actual = crypto.createHash('sha256').update(data).digest('hex');
++ const actual = sha256Of(data);
if (actual !== sha256) throw new Error('blob digest mismatch');
try {
const st = fs.statSync(dest);
@@@ -107,7 +146,226 @@ function createBlobStore(root) {
return true;
}
-- return { root: base, pathFor, put, exists, remove, removeProject, list, sweepTmp, writable, freeBytes };
++ /** The path to serve: a local cache hit is already the file. */
++ function ensure(projectId, sha256) {
++ return pathFor(projectId, sha256);
++ }
++
++ return { root: base, tmpDir, pathFor, put, exists, remove, removeProject, list, sweepTmp, writable, freeBytes, ensure };
++}
++
++/**
++ * The Media-backed store: local disk is the read cache, Media is the source of truth. It exposes the
++ * same interface as the local store; put/ensure/remove/removeProject become async (they talk to
++ * Media), and the rest delegate to the cache.
++ *
++ * opts.isReferenced(projectId, sha256) -> whether host_blobs still has a row, i.e. whether a deploy
++ * still references the blob. remove() re-checks it just before deleting, so a blob a concurrent
++ * deploy re-used after the caller found it unreferenced keeps its Media copy.
++ */
++function createMediaStore(local, media, { isReferenced } = {}) {
++ const { createObjectsClient } = require('openvibe-sdk/media'); // v2 object API
++ const { createServiceTokenClient } = require('openvibe-sdk/auth');
++ if (!media || !media.url) throw new Error('HOST_OBJECT_STORE=media needs OV_MEDIA_URL (the internal OpenVibe.Media base URL)');
++ if (!media.clientSecret) throw new Error('HOST_OBJECT_STORE=media needs OV_OAUTH_CLIENT_SECRET: Host stores objects as its own Media tenant through a Network service token');
++ const tokenClient = createServiceTokenClient({
++ tokenUrl: media.tokenUrl,
++ clientId: media.clientId,
++ clientSecret: media.clientSecret,
++ audience: MEDIA_AUDIENCE,
++ scope: media.scope || MEDIA_SCOPE,
++ ...(media.fetchImpl ? { fetch: media.fetchImpl } : {}),
++ });
++ const objects = createObjectsClient({
++ app: media.namespace || 'host',
++ baseUrl: String(media.url).replace(/\/+$/, ''),
++ tokenClient,
++ ...(media.fetchImpl ? { fetch: media.fetchImpl } : {}),
++ });
++ const fetchBytes = media.fetchImpl || globalThis.fetch;
++ const fetchTimeoutMs = Number(media.fetchTimeoutMs) > 0 ? Number(media.fetchTimeoutMs) : MEDIA_FETCH_TIMEOUT_MS;
++ // (project, sha256) -> Media object id, so a deploy does not list Host's namespace once per file.
++ // A restart simply repopulates it; deleting an object drops its entry. Only an optimisation: an
++ // empty map still finds every object by its content_hash.
++ const known = new Map();
++ const keyOf = (projectId, sha256) => `${projectId}\n${sha256}`;
++ // In-flight tasks by key: a concurrent put of the same new blob waits for the first upload
++ // instead of racing it into a duplicate object, and a burst of cache misses for one blob shares
++ // a single download. Keys are prefixed so a put and an ensure never block each other.
++ const inflight = new Map();
++
++ const isOurs = (o, projectId, sha256) => Boolean(o)
++ && o.content_hash === sha256
++ && o.metadata && o.metadata.project_id === projectId;
++
++ /** Run fn after every earlier task with the same key, whichever way it settled. */
++ function serialise(map, key, fn) {
++ const prev = map.get(key) || Promise.resolve();
++ const run = prev.then(fn, fn);
++ const tail = run.then(() => {}, () => {}); // the queue's tail never rejects
++ map.set(key, tail);
++ tail.then(() => { if (map.get(key) === tail) map.delete(key); });
++ return run;
++ }
++
++ /** Every Media object id for this (project, sha256): there can be duplicates from before puts were serialised. */
++ async function findAllIds(projectId, sha256) {
++ const ids = [];
++ for await (const o of objects.iterate({})) if (isOurs(o, projectId, sha256)) ids.push(o.id);
++ return ids;
++ }
++
++ async function findId(projectId, sha256) {
++ const key = keyOf(projectId, sha256);
++ const hit = known.get(key);
++ if (hit) return hit;
++ const ids = await findAllIds(projectId, sha256);
++ if (!ids.length) return null;
++ known.set(key, ids[0]);
++ return ids[0];
++ }
++
++ /** Stream a Media object into a temp file while hashing it; rename into the cache only on a match. */
++ async function downloadToCache(projectId, sha256, url) {
++ const dest = local.pathFor(projectId, sha256);
++ const tmp = path.join(local.tmpDir, `${crypto.randomBytes(12).toString('hex')}.part`);
++ const hash = crypto.createHash('sha256');
++ try {
++ const res = await fetchBytes(url, { signal: AbortSignal.timeout(fetchTimeoutMs) });
++ if (!res.ok) throw new Error(`download answered ${res.status}`);
++ fs.mkdirSync(path.dirname(dest), { recursive: true });
++ if (res.body) {
++ const meter = new Transform({ transform(chunk, _enc, cb) { hash.update(chunk); cb(null, chunk); } });
++ await pipeline(Readable.fromWeb(res.body), meter, fs.createWriteStream(tmp));
++ } else {
++ const buf = Buffer.from(await res.arrayBuffer());
++ hash.update(buf);
++ fs.writeFileSync(tmp, buf);
++ }
++ const actual = hash.digest('hex');
++ if (actual !== sha256) {
++ const e = new Error(`object ${sha256}: Media returned bytes whose sha256 is ${actual}; refused`);
++ e.code = 'storage.corrupt';
++ throw e;
++ }
++ fs.renameSync(tmp, dest);
++ } catch (err) {
++ fs.rmSync(tmp, { force: true });
++ if (err && err.code === 'storage.corrupt') throw err;
++ throw mediaFailure(`could not fetch object ${sha256} from Media`, err);
++ }
++ }
++
++ async function doPut(projectId, sha256, data) {
++ if (!SHA_RE.test(String(sha256))) throw new TypeError('invalid sha256');
++ // Media's upload refuses zero bytes ("nothing to upload"); an empty file is a valid manifest
++ // entry, so it lives in the local cache only. ensure() serves it from there and, since the
++ // serving path short-circuits size 0, never asks Media (which has no object for it).
++ if (data.length === 0) return local.put(projectId, sha256, data);
++ const changed = local.put(projectId, sha256, data); // the cache first; verifies the digest
++ // A blob this process just wrote is new here: upload it directly. Listing Host's whole
++ // namespace once per file would be quadratic on a 300-file deploy, and Media's v2 list API has
++ // no content_hash or metadata filter. Only a blob that was already cached (a previous deploy
++ // in this process, or an earlier run) is looked up, so a restart does not upload a second copy.
++ if (!changed) {
++ if (await findId(projectId, sha256)) return changed;
++ }
++ let obj;
++ try {
++ obj = await objects.upload(data, {
++ kind: 'file', visibility: 'private', mimeType: 'application/octet-stream',
++ filename: sha256, contentHash: sha256,
++ metadata: { project_id: projectId, sha256 },
++ });
++ } catch (err) {
++ throw mediaFailure(`Media refused to store object ${sha256}`, err);
++ }
++ known.set(keyOf(projectId, sha256), obj.id);
++ return changed;
++ }
++
++ /** Cache the bytes locally (as today), then write through to Media. Rejects if Media refuses. */
++ function put(projectId, sha256, data) {
++ return serialise(inflight, `put:${keyOf(projectId, sha256)}`, () => doPut(projectId, sha256, data));
++ }
++
++ async function doEnsure(projectId, sha256) {
++ if (local.exists(projectId, sha256)) return local.pathFor(projectId, sha256); // a peer cached it while we waited
++ const id = await findId(projectId, sha256);
++ if (!id) throw mediaFailure(`object ${sha256} is not in Media`, null);
++ let url;
++ try {
++ ({ url } = await objects.signedUrl(id, { ttl: 300 }));
++ } catch (err) {
++ throw mediaFailure(`could not fetch object ${sha256} from Media`, err);
++ }
++ await downloadToCache(projectId, sha256, url);
++ return local.pathFor(projectId, sha256);
++ }
++
++ /**
++ * Serve the cache; on a miss fetch from Media, verify the sha256, cache it, then serve. Concurrent
++ * misses for one blob share a single download.
++ */
++ function ensure(projectId, sha256) {
++ if (local.exists(projectId, sha256)) return Promise.resolve(local.pathFor(projectId, sha256));
++ return serialise(inflight, `ensure:${keyOf(projectId, sha256)}`, () => doEnsure(projectId, sha256));
++ }
++
++ /** Delete the Media object(s) and the cache file. Callers pass only blobs no deploy references. */
++ async function remove(projectId, sha256) {
++ if (!SHA_RE.test(String(sha256))) throw new TypeError('invalid sha256');
++ let ids;
++ try {
++ ids = await findAllIds(projectId, sha256);
++ } catch (err) {
++ throw mediaFailure(`Media could not delete object ${sha256}`, err);
++ }
++ // Between the caller's "unreferenced" decision and this delete a concurrent deploy may have
++ // re-used the sha. host_blobs is the authority: if a row now exists, keep Media's durable copy
++ // and the cache file. The check is after the (slow) namespace scan, right before the delete.
++ if (isReferenced && await isReferenced(projectId, sha256)) return false;
++ if (ids.length) {
++ try {
++ for (const id of ids) await objects.delete(id);
++ } catch (err) {
++ throw mediaFailure(`Media could not delete object ${sha256}`, err);
++ }
++ known.delete(keyOf(projectId, sha256));
++ }
++ return local.remove(projectId, sha256);
++ }
++
++ /** Delete every Media object of the project and the project's cache directory. */
++ async function removeProject(projectId) {
++ if (!isId('project', projectId)) throw new TypeError('invalid project id');
++ let failure = null;
++ try {
++ for await (const o of objects.iterate({})) {
++ if (o && o.metadata && o.metadata.project_id === projectId) {
++ try {
++ await objects.delete(o.id);
++ if (o.content_hash) known.delete(keyOf(projectId, o.content_hash));
++ } catch (err) { failure = failure || err; }
++ }
++ }
++ } catch (err) { failure = failure || err; }
++ local.removeProject(projectId);
++ if (failure) throw mediaFailure(`Media could not delete the objects of ${projectId}`, failure);
++ }
++
++ return { ...local, put, ensure, remove, removeProject };
++}
++
++/**
++ * root: the local object directory. opts.media, when set, selects the Media-backed store (its
++ * presence, built from config, is the `HOST_OBJECT_STORE=media` switch); without it the local store
++ * behaves exactly as before. opts.isReferenced is the Media store's host_blobs re-check on delete.
++ */
++function createBlobStore(root, opts = {}) {
++ const local = createLocalStore(root);
++ if (!opts.media) return local;
++ return createMediaStore(local, opts.media, { isReferenced: opts.isReferenced });
}
--module.exports = { createBlobStore, SHA_RE };
++module.exports = { createBlobStore, SHA_RE, MEDIA_SCOPE, MEDIA_AUDIENCE };
ddiff --git a/server/worker.js b/server/worker.js
iindex f9e4b71..acd6595 100644
---- a/server/worker.js
++++ b/server/worker.js
@@@ -36,9 +36,16 @@ function createWorker({ config, store, domains, blobs, outbox, log = console })
if (await known.get(o.projectId, o.sha256)) continue;
try {
if (Date.now() - fs.statSync(o.file).mtimeMs < graceMs) continue;
-- fs.unlinkSync(o.file);
-- removed++;
-- } catch { /* raced with a write or already gone */ }
++ // Through the store, so with the Media store the unreferenced Media object goes too.
++ if (await blobs.remove(o.projectId, o.sha256)) removed++;
++ } catch (err) {
++ // A storage.media failure is a delete Media refused (grants, an outage): the sweep would
++ // otherwise look like it worked while the object leaks. Anything else raced a write or
++ // the file is already gone.
++ if (err && err.code === 'storage.media') {
++ log.warn('[Host] object sweep could not delete a Media object:', o.projectId, o.sha256, err.message);
++ }
++ }
}
removed += blobs.sweepTmp(graceMs);
return removed;
ddiff --git a/test/host-storage-media.test.js b/test/host-storage-media.test.js
nnew file mode 100644
iindex 0000000..b1b3496
---- /dev/null
++++ b/test/host-storage-media.test.js
@@@ -0,0 +1,309 @@
++'use strict';
++/**
++ * Host's Media-backed object store (HOST_OBJECT_STORE=media, plan T12): local disk is the read cache,
++ * OpenVibe.Media is the source of truth. Against a fake Media Object API v2 (test/stageb/media.js)
++ * and the same boot() as the other Stage B suites.
++ *
++ * - a deploy writes every object through to Media and serves from the local cache;
++ * - a deploy fails (502 storage.object_store) when Media refuses the write;
++ * - a local cache miss re-fetches from Media and verifies the sha256; a tampered object is refused;
++ * - identical bytes in one project are stored in Media once;
++ * - GC deletes a Media object only when no deploy references it;
++ * - project removal deletes the project's Media objects;
++ * - with HOST_OBJECT_STORE unset nothing calls Media at all.
++ */
++const assert = require('assert');
++const crypto = require('crypto');
++const fs = require('fs');
++const path = require('path');
++const { boot, check, done } = require('./stageb/boot');
++const { startMedia } = require('./stageb/media');
++const { createBlobStore, MEDIA_SCOPE } = require('../server/storage');
++const { createWorker } = require('../server/worker');
++
++const cachePath = (t, projectId, sha) => path.join(t.dir, 'objects', 'projects', projectId, sha.slice(0, 2), sha);
++const shaOf = (s) => crypto.createHash('sha256').update(s).digest('hex');
++/** A store built directly (not through the app) with the test's fake Media and mock Network. */
++const directStore = (t, media, { dir, isReferenced, fetchTimeoutMs } = {}) => createBlobStore(
++ dir || fs.mkdtempSync(path.join(t.dir, 'direct-')),
++ {
++ media: {
++ url: media.url, namespace: 'host', scope: MEDIA_SCOPE, clientId: 'host', clientSecret: t.SECRET,
++ tokenUrl: `${t.network.url}/oauth/token`,
++ ...(fetchTimeoutMs ? { fetchTimeoutMs } : {}),
++ },
++ isReferenced,
++ },
++);
++const filesOf = async (t, as, deployId) => (await t.api('GET', `/api/v1/deploys/${deployId}`, { as })).json().deploy.files;
++
++(async () => {
++ const media = await startMedia();
++ const t = await boot({ env: { HOST_OBJECT_STORE: 'media', OV_MEDIA_URL: media.url } });
++ const alice = t.user('alice');
++
++ await check('booting with HOST_OBJECT_STORE=media selects the Media store', async () => {
++ assert.strictEqual(t.config.objectStore.mode, 'media');
++ assert.strictEqual(t.config.media.url, media.url);
++ });
++
++ const pa = await t.project(alice, 'A');
++ const sa = await t.site(alice, pa.id, 'mediasite');
++ let firstFiles = [];
++ await check('a deploy writes every object through to Media, private and keyed per project', async () => {
++ const d = await t.deploy(alice, sa.id, { 'index.html': 'HELLO MEDIA', 'app.js': 'console.log(1)' });
++ firstFiles = await filesOf(t, alice, d.id);
++ assert.strictEqual(firstFiles.length, 2);
++ for (const f of firstFiles) {
++ const o = media.byHash(pa.id, f.sha256);
++ assert.ok(o, `Media holds ${f.path}`);
++ assert.strictEqual(o.visibility, 'private');
++ assert.strictEqual(o.lifecycle_status, 'ready');
++ assert.strictEqual(o.metadata.project_id, pa.id);
++ assert.ok(fs.existsSync(cachePath(t, pa.id, f.sha256)), `${f.path} is cached locally`);
++ }
++ assert.strictEqual((await t.get('mediasite.openvibe.host', '/')).text, 'HELLO MEDIA');
++ });
++
++ await check('a deploy fails when Media refuses the write (never a local-only success)', async () => {
++ const pb = await t.project(alice, 'B');
++ const sb = await t.site(alice, pb.id, 'refuses');
++ media.failUpload = true;
++ try {
++ const r = await t.upload(alice, sb.id, { 'index.html': 'NOPE' });
++ assert.strictEqual(r.status, 502, r.text);
++ assert.strictEqual(r.json().code, 'storage.object_store');
++ assert.ok(r.json().deploy_id, 'the failed deploy is recorded');
++ assert.strictEqual(media.readyFor(pb.id).length, 0, 'nothing reached Media as ready');
++ assert.strictEqual((await t.get('refuses.openvibe.host', '/')).status, 404, 'nothing is served');
++ } finally { media.failUpload = false; }
++ // The next deploy of the same project goes through once Media recovers.
++ await t.deploy(alice, sb.id, { 'index.html': 'RECOVERED' });
++ assert.strictEqual((await t.get('refuses.openvibe.host', '/')).text, 'RECOVERED');
++ });
++
++ await check('a local miss re-fetches from Media and verifies the sha256; tampered bytes are refused', async () => {
++ const sha = firstFiles.find((f) => f.path === 'index.html').sha256;
++ const local = cachePath(t, pa.id, sha);
++ fs.rmSync(local);
++ assert.ok(!fs.existsSync(local), 'evicted the local cache');
++ const ok = await t.get('mediasite.openvibe.host', '/');
++ assert.strictEqual(ok.status, 200, ok.text);
++ assert.strictEqual(ok.text, 'HELLO MEDIA');
++ assert.ok(fs.existsSync(local), 'a verified fetch is cached again');
++
++ const obj = media.byHash(pa.id, sha);
++ fs.rmSync(local);
++ media.tamper.add(obj.id);
++ const bad = await t.get('mediasite.openvibe.host', '/');
++ assert.strictEqual(bad.status, 500);
++ assert.ok(!bad.text.includes('HELLO MEDIA'), 'tampered bytes are never served');
++ assert.ok(!fs.existsSync(local), 'tampered bytes are never cached');
++ media.tamper.delete(obj.id);
++ });
++
++ await check('identical bytes in one project are stored in Media once', async () => {
++ const before = media.countInits();
++ const d = await t.deploy(alice, sa.id, { 'index.html': 'HELLO MEDIA', 'extra.txt': 'EXTRA' });
++ const files = await filesOf(t, alice, d.id);
++ assert.strictEqual(media.countInits(), before + 1, 'only the new file is uploaded; index.html is reused');
++ // One Media object per unique sha of the project (index.html, app.js, extra.txt).
++ const uniq = new Set([...firstFiles, ...files].map((f) => f.sha256));
++ assert.strictEqual(uniq.size, 3);
++ assert.strictEqual(media.readyFor(pa.id).length, uniq.size);
++ });
++
++ await check('GC deletes a Media object only when no remaining deploy references it', async () => {
++ const pc = await t.project(alice, 'C');
++ const sc = await t.site(alice, pc.id, 'gamma');
++ const g1 = await t.deploy(alice, sc.id, { 'a.html': 'AAA', 'shared.txt': 'SHARED' });
++ const g2 = await t.deploy(alice, sc.id, { 'b.html': 'BBB', 'shared.txt': 'SHARED' }); // now active
++ assert.ok(g2.active);
++ const f1 = await filesOf(t, alice, g1.id);
++ const shaA = f1.find((f) => f.path === 'a.html').sha256;
++ const shaShared = f1.find((f) => f.path === 'shared.txt').sha256;
++ const del = await t.api('DELETE', `/api/v1/deploys/${g1.id}`, { as: alice });
++ assert.strictEqual(del.status, 200, del.text);
++ assert.strictEqual(media.byHash(pc.id, shaA), null, 'the unreferenced object is deleted from Media');
++ assert.ok(media.byHash(pc.id, shaShared), 'an object a remaining deploy references is kept');
++ assert.strictEqual((await t.get('gamma.openvibe.host', '/shared.txt')).text, 'SHARED');
++ });
++
++ await check('project removal deletes the project\'s Media objects, and nobody else\'s', async () => {
++ const pd = await t.project(alice, 'D');
++ const sd = await t.site(alice, pd.id, 'removed');
++ const d1 = await t.deploy(alice, sd.id, { 'x.html': 'XXXX' });
++ const shaX = (await filesOf(t, alice, d1.id)).find((f) => f.path === 'x.html').sha256;
++ const pe = await t.project(alice, 'E');
++ const se = await t.site(alice, pe.id, 'kept');
++ const d2 = await t.deploy(alice, se.id, { 'y.html': 'YYYY' });
++ const shaY = (await filesOf(t, alice, d2.id)).find((f) => f.path === 'y.html').sha256;
++ assert.ok(media.byHash(pd.id, shaX) && media.byHash(pe.id, shaY));
++ const rm = await t.api('DELETE', `/api/v1/projects/${pd.id}`, { as: alice });
++ assert.strictEqual(rm.status, 200, rm.text);
++ assert.strictEqual(media.byHash(pd.id, shaX), null, 'the removed project\'s object is gone from Media');
++ assert.ok(media.byHash(pe.id, shaY), 'another project\'s object is untouched');
++ assert.strictEqual((await t.get('kept.openvibe.host', '/y.html')).text, 'YYYY');
++ });
++
++ await check('with HOST_OBJECT_STORE unset nothing calls Media', async () => {
++ const media2 = await startMedia();
++ const t2 = await boot({ env: { OV_MEDIA_URL: media2.url } });
++ try {
++ const bob = t2.user('bob');
++ const p = await t2.project(bob, 'L');
++ const s = await t2.site(bob, p.id, 'localonly');
++ await t2.deploy(bob, s.id, { 'index.html': 'LOCAL ONLY' });
++ assert.strictEqual(media2.calls.length, 0, 'the local store never talks to Media');
++ assert.strictEqual((await t2.get('localonly.openvibe.host', '/')).text, 'LOCAL ONLY');
++ } finally { await t2.close(); await media2.close(); }
++ });
++
++ await check('a zero-byte file is stored locally only and never sent to Media', async () => {
++ const pz = await t.project(alice, 'Z');
++ const sz = await t.site(alice, pz.id, 'zero');
++ const before = media.countInits();
++ const d = await t.deploy(alice, sz.id, { 'index.html': 'ZERO SITE', 'empty.txt': '' });
++ const empty = (await filesOf(t, alice, d.id)).find((f) => f.path === 'empty.txt');
++ assert.strictEqual(empty.size, 0);
++ assert.strictEqual(media.countInits(), before + 1, 'only index.html is uploaded; the empty file is not');
++ assert.strictEqual(media.byHash(pz.id, empty.sha256), null, 'Media has no object for the empty file');
++ assert.ok(fs.existsSync(cachePath(t, pz.id, empty.sha256)), 'the empty file is in the local cache');
++ const r = await t.get('zero.openvibe.host', '/empty.txt');
++ assert.strictEqual(r.status, 200);
++ assert.strictEqual(r.text, '');
++ });
++
++ await check('put uploads a fresh blob directly and looks up only an already-cached one', async () => {
++ const dir = fs.mkdtempSync(path.join(t.dir, 'direct-put-'));
++ const s1 = directStore(t, media, { dir });
++ const blob = Buffer.from('DIRECT UPLOAD');
++ const h = shaOf('DIRECT UPLOAD');
++ const lists0 = media.countLists();
++ await s1.put(pa.id, h, blob);
++ assert.strictEqual(media.countLists(), lists0, 'a fresh blob is uploaded without listing the namespace');
++ assert.ok(media.byHash(pa.id, h));
++ // A restart: a new store over the same warm cache, with no in-memory id map.
++ const s2 = directStore(t, media, { dir });
++ const inits = media.countInits();
++ const lists1 = media.countLists();
++ await s2.put(pa.id, h, blob);
++ assert.strictEqual(media.countInits(), inits, 'the cached blob is not uploaded a second time');
++ assert.ok(media.countLists() > lists1, 'the already-cached blob is looked up once');
++ assert.strictEqual(media.ready().filter((o) => o.metadata.project_id === pa.id && o.content_hash === h).length, 1);
++ });
++
++ await check('concurrent puts of one new sha upload it once', async () => {
++ const store = directStore(t, media, { dir: fs.mkdtempSync(path.join(t.dir, 'direct-race-')) });
++ const blob = Buffer.from('RACE');
++ const h = shaOf('RACE');
++ const inits = media.countInits();
++ const changed = await Promise.all([store.put(pa.id, h, blob), store.put(pa.id, h, blob), store.put(pa.id, h, blob)]);
++ assert.deepStrictEqual(changed, [true, false, false]);
++ assert.strictEqual(media.countInits(), inits + 1, 'three concurrent puts, one upload');
++ assert.strictEqual(media.ready().filter((o) => o.metadata.project_id === pa.id && o.content_hash === h).length, 1);
++ });
++
++ await check('remove deletes every Media object for a blob, not just the first', async () => {
++ const store = directStore(t, media, { dir: fs.mkdtempSync(path.join(t.dir, 'direct-dupes-')) });
++ const blob = Buffer.from('DUPLICATE');
++ const h = shaOf('DUPLICATE');
++ media.seed(pa.id, blob);
++ media.seed(pa.id, blob); // two pre-existing duplicates (from before puts were serialised)
++ await store.put(pa.id, h, blob); // plus the cache and one more object
++ const countOf = () => media.ready().filter((o) => o.metadata.project_id === pa.id && o.content_hash === h).length;
++ assert.strictEqual(countOf(), 3);
++ assert.strictEqual(await store.remove(pa.id, h), true);
++ assert.strictEqual(countOf(), 0, 'every duplicate is gone');
++ assert.ok(!fs.existsSync(store.pathFor(pa.id, h)), 'the cache file is gone');
++ });
++
++ await check('remove keeps a Media object and cache that host_blobs re-references mid-sweep', async () => {
++ let referenced = false;
++ const store = directStore(t, media, {
++ dir: fs.mkdtempSync(path.join(t.dir, 'direct-ref-')),
++ isReferenced: async () => referenced,
++ });
++ const blob = Buffer.from('RE-REFERENCED');
++ const h = shaOf('RE-REFERENCED');
++ await store.put(pa.id, h, blob);
++ referenced = true; // a concurrent deploy reused the sha after the caller decided it was unreferenced
++ assert.strictEqual(await store.remove(pa.id, h), false, 'remove reports nothing removed');
++ assert.ok(media.byHash(pa.id, h), 'the durable copy stays in Media');
++ assert.ok(fs.existsSync(store.pathFor(pa.id, h)), 'the cache stays');
++ referenced = false;
++ assert.strictEqual(await store.remove(pa.id, h), true);
++ assert.strictEqual(media.byHash(pa.id, h), null, 'once unreferenced, remove deletes it');
++ assert.ok(!fs.existsSync(store.pathFor(pa.id, h)));
++ });
++
++ await check('concurrent cache misses for one blob stream it in once and rename it in', async () => {
++ const store = directStore(t, media, { dir: fs.mkdtempSync(path.join(t.dir, 'direct-coalesce-')) });
++ const blob = Buffer.from('COALESCE ME');
++ const h = shaOf('COALESCE ME');
++ await store.put(pa.id, h, blob);
++ fs.rmSync(store.pathFor(pa.id, h));
++ const downloads = media.countDownloads();
++ const files = await Promise.all([store.ensure(pa.id, h), store.ensure(pa.id, h), store.ensure(pa.id, h)]);
++ assert.deepStrictEqual(files, [store.pathFor(pa.id, h), store.pathFor(pa.id, h), store.pathFor(pa.id, h)]);
++ assert.strictEqual(media.countDownloads(), downloads + 1, 'one download for three misses');
++ assert.strictEqual(fs.readFileSync(files[0], 'utf8'), 'COALESCE ME');
++ assert.deepStrictEqual(fs.readdirSync(store.tmpDir).filter((f) => f.endsWith('.part')), [], 'no temp file is left behind');
++ });
++
++ await check('a Media download past the fetch timeout fails cleanly and caches nothing', async () => {
++ const store = directStore(t, media, { dir: fs.mkdtempSync(path.join(t.dir, 'direct-slow-')), fetchTimeoutMs: 60 });
++ const blob = Buffer.from('SLOW OBJECT');
++ const h = shaOf('SLOW OBJECT');
++ media.seed(pa.id, blob);
++ media.delayContentMs = 400;
++ try {
++ await assert.rejects(() => store.ensure(pa.id, h), (e) => e.code === 'storage.media');
++ } finally { media.delayContentMs = 0; }
++ assert.ok(!fs.existsSync(store.pathFor(pa.id, h)), 'nothing is cached from a failed download');
++ });
++
++ await check('a failed deploy names the objects it left in the deploy log', async () => {
++ const ph = await t.project(alice, 'H');
++ const sh = await t.site(alice, ph.id, 'partial');
++ const shaA = shaOf('AAAA PARTIAL');
++ media.failUploadAfter = media.acceptedInits() + 1; // the first object lands, the second is refused
++ let r;
++ try { r = await t.upload(alice, sh.id, { 'a.txt': 'AAAA PARTIAL', 'b.txt': 'BBBB PARTIAL' }); }
++ finally { media.failUploadAfter = null; }
++ assert.strictEqual(r.status, 502, r.text);
++ assert.strictEqual(r.json().code, 'storage.object_store');
++ assert.ok(media.byHash(ph.id, shaA), 'the first object was left in Media');
++ const log = r.json().log.join('\n');
++ assert.ok(log.includes(shaA), `the deploy log names the leftover object: ${log}`);
++ });
++
++ await check('the object sweep logs a Media delete failure at warn instead of swallowing it', async () => {
++ const dir = fs.mkdtempSync(path.join(t.dir, 'sweep-'));
++ const file = path.join(dir, 'blob');
++ fs.writeFileSync(file, 'x');
++ fs.utimesSync(file, new Date(0), new Date(0));
++ const sha = 'a'.repeat(64);
++ const warnings = [];
++ const worker = createWorker({
++ config: { worker: { enabled: false } },
++ store: { db: { prepare: () => ({ get: async () => null }) } }, // no host_blobs row
++ domains: {},
++ blobs: {
++ list: function* () { yield { projectId: pa.id, sha256: sha, file }; },
++ sweepTmp: () => 0,
++ remove: async () => { const e = new Error('media down'); e.code = 'storage.media'; throw e; },
++ },
++ outbox: {},
++ log: { log() {}, error() {}, warn: (...a) => warnings.push(a.join(' ')) },
++ });
++ const removed = await worker.sweepObjects({ graceMs: 0 });
++ assert.strictEqual(removed, 0, 'a failed delete is not counted as removed');
++ assert.strictEqual(warnings.length, 1, `expected exactly one warn, got ${JSON.stringify(warnings)}`);
++ assert.ok(warnings[0].includes('Media') && warnings[0].includes(sha) && warnings[0].includes('media down'), warnings[0]);
++ });
++
++ await t.close();
++ await media.close();
++ done();
++})();
ddiff --git a/test/stageb/media.js b/test/stageb/media.js
nnew file mode 100644
iindex 0000000..f625270
---- /dev/null
++++ b/test/stageb/media.js
@@@ -0,0 +1,150 @@
++'use strict';
++/**
++ * An in-process stand-in for OpenVibe.Media's Object API v2, wide enough for Host's Media object
++ * store: init (POST /objects), the content PUT, complete, metadata GET, list, the signed download
++ * link, and delete — the routes OpenVibe.Media serves from server/objects/routes.js, served here
++ * from memory. It follows test/stageb/mocks.js (an HTTP server on a random port) and test/s3-mock.js
++ * (a recorded command log plus injectable failures).
++ *
++ * const media = await startMedia();
++ * media.url the base URL to give Host (OV_MEDIA_URL)
++ * media.objects Map id -> object (bytes, content_hash, metadata, lifecycle_status)
++ * media.calls every request { method, url, bytes, auth }
++ * media.failUpload = true every init answers 403 (a deploy must then fail)
++ * media.failUploadAfter = n refuse inits once n have succeeded (fail mid-deploy)
++ * media.delayContentMs = ms delay the signed content GET (a stalled download)
++ * media.failDelete = true every delete answers 500
++ * media.tamper.add(id) the signed content GET answers other bytes
++ * media.countInits() init requests so far
++ * media.acceptedInits() inits that were allowed (the rest were refused)
++ * media.countLists() namespace list requests so far
++ * media.countDownloads() signed content GETs so far
++ * media.seed(projectId, bytes) put a ready object straight into the fake, returns its id
++ * media.readyFor(projectId) ready objects whose metadata.project_id is that project
++ * media.byHash(projectId, sha) the ready object for that project and content hash
++ * await media.close()
++ */
++const http = require('http');
++const crypto = require('crypto');
++const { ids } = require('openvibe-contracts');
++
++const sha256 = (buf) => crypto.createHash('sha256').update(buf).digest('hex');
++
++function startMedia({ app = 'host' } = {}) {
++ const objects = new Map();
++ const calls = [];
++ const media = { objects, calls, app, failUpload: false, failUploadAfter: null, delayContentMs: 0, failDelete: false, tamper: new Set() };
++ let acceptedInits = 0;
++ const basePath = `/api/v2/${encodeURIComponent(app)}/objects`;
++ const base = () => `http://127.0.0.1:${server.address().port}`;
++ const json = (res, status, obj) => { res.writeHead(status, { 'Content-Type': 'application/json' }); res.end(JSON.stringify(obj)); };
++ const publicShape = (o) => ({
++ id: o.id, app_id: app, namespace: o.namespace, kind: o.kind, visibility: o.visibility,
++ lifecycle_status: o.lifecycle_status, mime_type: o.mime_type, size_bytes: o.size_bytes,
++ content_hash: o.content_hash || null, metadata: o.metadata, public_url: null,
++ });
++
++ const server = http.createServer((req, res) => {
++ const chunks = [];
++ req.on('data', (c) => chunks.push(c));
++ req.on('end', () => {
++ const raw = Buffer.concat(chunks);
++ calls.push({ method: req.method, url: req.url, bytes: raw.length, auth: req.headers.authorization || null });
++ const p = new URL(req.url, 'http://localhost').pathname;
++ const rel = p === basePath ? '' : p.startsWith(`${basePath}/`) ? p.slice(basePath.length + 1) : null;
++ if (rel === null) return json(res, 404, { code: 'media.not_found' });
++
++ if (rel === '' && req.method === 'POST') {
++ if (media.failUpload || (media.failUploadAfter != null && acceptedInits >= media.failUploadAfter)) {
++ return json(res, 403, { code: 'media.object.denied', detail: 'the fake refuses every upload' });
++ }
++ acceptedInits++;
++ let body = {};
++ try { body = JSON.parse(raw.toString('utf8')); } catch { /* malformed: treat as empty */ }
++ const id = `med_${ids.ulid()}`;
++ const o = {
++ id, namespace: body.namespace || app, kind: body.kind || 'file', visibility: body.visibility || 'private',
++ lifecycle_status: 'uploading', mime_type: body.mime_type || null, size_bytes: Number(body.size_bytes) || 0,
++ content_hash: null, metadata: { ...(body.metadata || {}) }, bytes: null,
++ };
++ objects.set(id, o);
++ return json(res, 201, {
++ id, object: publicShape(o),
++ upload: { method: 'PUT', url: `${base()}${basePath}/${id}/content?token=tok`, token: 'tok', expires_at: new Date(Date.now() + 3600e3).toISOString(), max_bytes: o.size_bytes, content_type: o.mime_type },
++ });
++ }
++ if (rel === '' && req.method === 'GET') {
++ const all = [...objects.values()].filter((o) => o.lifecycle_status !== 'deleted');
++ return json(res, 200, { objects: all.map(publicShape), next_cursor: null, limit: 50 });
++ }
++
++ const m = /^([^/]+)(?:\/(.*))?$/.exec(rel);
++ const obj = m && objects.get(m[1]);
++ if (!obj) return json(res, 404, { code: 'media.object.not_found' });
++ const sub = m[2] || '';
++ if (sub === 'content' && req.method === 'PUT') {
++ obj.bytes = Buffer.from(raw);
++ obj.content_hash = sha256(obj.bytes);
++ obj.size_bytes = obj.bytes.length;
++ return json(res, 200, {});
++ }
++ if (sub === 'content' && req.method === 'GET') {
++ const sendContent = () => {
++ if (res.destroyed || res.writableEnded) return; // the client gave up (timeout)
++ if (media.tamper.has(obj.id)) { res.writeHead(200, { 'Content-Type': 'application/octet-stream' }); return res.end(Buffer.from('TAMPERED BYTES')); }
++ res.writeHead(200, { 'Content-Type': 'application/octet-stream' });
++ return res.end(obj.bytes || Buffer.alloc(0));
++ };
++ if (media.delayContentMs) { setTimeout(sendContent, media.delayContentMs); return; }
++ return sendContent();
++ }
++ if (sub === 'complete' && req.method === 'POST') {
++ let body = {};
++ try { body = JSON.parse(raw.toString('utf8')); } catch { /* */ }
++ const expected = String(body.content_hash || '').toLowerCase();
++ if (expected && expected !== obj.content_hash) return json(res, 422, { code: 'media.object.hash_mismatch', detail: 'content hash mismatch' });
++ obj.lifecycle_status = 'ready';
++ return json(res, 200, publicShape(obj));
++ }
++ if (sub === 'download' && req.method === 'GET') {
++ if (obj.lifecycle_status !== 'ready') return json(res, 409, { code: 'media.object.not_ready' });
++ return json(res, 200, { url: `${base()}${basePath}/${obj.id}/content?token=signed`, expires_at: new Date(Date.now() + 300e3).toISOString(), public: false });
++ }
++ if (sub === '' && req.method === 'GET') return json(res, 200, publicShape(obj));
++ if (sub === '' && req.method === 'DELETE') {
++ if (media.failDelete) return json(res, 500, { code: 'media.object.delete_failed' });
++ obj.lifecycle_status = 'deleted';
++ return json(res, 200, publicShape(obj));
++ }
++ return json(res, 404, { code: 'media.not_found' });
++ });
++ });
++
++ return new Promise((resolve) => {
++ server.listen(0, '127.0.0.1', () => {
++ media.url = base();
++ media.close = () => new Promise((r) => server.close(r));
++ media.countInits = () => calls.filter((c) => c.method === 'POST' && new URL(c.url, 'http://x').pathname === basePath).length;
++ media.acceptedInits = () => acceptedInits;
++ media.countLists = () => calls.filter((c) => c.method === 'GET' && new URL(c.url, 'http://x').pathname === basePath).length;
++ media.countDownloads = () => calls.filter((c) => c.method === 'GET' && new URL(c.url, 'http://x').pathname.endsWith('/content')).length;
++ media.seed = (projectId, bytes) => {
++ const b = Buffer.isBuffer(bytes) ? bytes : Buffer.from(bytes);
++ const h = sha256(b);
++ const id = `med_${ids.ulid()}`;
++ objects.set(id, {
++ id, namespace: app, kind: 'file', visibility: 'private', lifecycle_status: 'ready',
++ mime_type: 'application/octet-stream', size_bytes: b.length, content_hash: h,
++ metadata: { project_id: projectId, sha256: h }, bytes: b,
++ });
++ return id;
++ };
++ media.ready = () => [...objects.values()].filter((o) => o.lifecycle_status === 'ready');
++ media.readyFor = (projectId) => media.ready().filter((o) => o.metadata && o.metadata.project_id === projectId);
++ media.byHash = (projectId, sha) => media.ready().find((o) => o.metadata && o.metadata.project_id === projectId && o.content_hash === sha) || null;
++ resolve(media);
++ });
++ });
++}
++
++module.exports = { startMedia };