openvibers/openvibe.media Commits
Tiering for native v2 objects (roadmap WS-G task 11): popular objects get a verified R2 copy that /o/:id plays from, behind a site-level activation gate that is off by default. Popularity (server/objects/popularity.js, ADR-021): GET /o/:id counts a native object's viewer once per UTC day as HMAC-SHA256(the day's random salt, client network; IPv6 by /64), never an IP address or subject id; the day's hashes and salt are deleted once it is over (first view of the next day, hourly, every sweep), only daily counts stay (30 days); the 7-day figure sums them. Seeks, HEAD, Sec-GPC/DNT, crawlers and sandbox tenants are not counted. Policy: the revisioned namespace media.object_tier (server/objects/tier-policy.js; active=false, promoteMinUniqueViewers7d 500, viewed within 3 days, 16 MB to 4 GB, 3 promotions and 10 demotions per sweep, demote after 14 idle days), validated (types, ranges, min size below max, demoteIdleDays above promoteRecentAccessDays) and served by the admin config routes. Sweep and moves (server/objects/tiering.js), step 4 of the storage sweep, skipped under disk pressure and in a restore drill: with the gate off nothing moves (sweep or direct call) and would-be moves are logged as dry_run; holds refuse every move; only an object whose canonical copy is verified on record is promoted, the canonical copy is re-hashed, copied to R2 and the R2 copy's size and sha256 read back before it is recorded (else deleted, failed, 6 h back-off); a demotion (idle, or no longer ready) re-checks the canonical copy first and never removes the last good copy, and the purge waits for a deleted object's R2 copy. Every decision goes to media_object_tier_decisions with its inputs and the policy in force; GET /admin/storage/tiers/objects/policy and /decisions, /me/ops (native_objects) and media_tier_decisions_24h{target=vod|object} show them. The storage engine now exports uploadFile, which the scheduled verification's default re-upload called but could not reach. test/object-tiering.test.js runs it against a fake S3 through the real SDK.
Changes
@@@ -46,7 +46,8 @@ server/objects/ canonical object model: model (projections, holds), routes (/api/v2 + /o),backfill, reconcile, invariant, signing (presigned URLs), multipart,content-type, verify-job (scheduled copy verification), copy-report,-- readiness (metadata / bytes_verified / playable) —++ readiness (metadata / bytes_verified / playable), popularity (unique viewers++ per day, no identifiers), tier-policy + tiering (native objects in the R2 cache) —see docs/object-model.mdjobs/ media_jobs: queue (states, idempotency, media.job.* events in the statechange's transaction; payload queue.jobEvent: ids, state, ISO times,@@@ -290,6 +291,9 @@ directories are shared across apps) and responses carry a `note` saying so.| POST | `/admin/storage/tiers/bulk-move` | `{ ids: [...], target }` (max 50) → `{ moved, bytes, errors? }` || GET | `/admin/storage/tiers/policy` | **read-only R2 policy**: each threshold (`r2Enabled`, `r2MinViews`, `r2RecentAccessDays`, `r2MaxIdleDays`, `r2MaxPerSweep`) as `{ value, default, source: default\|setting }`, the promote and demote rules in words, the R2/B2 provider status, and this app's decisions: `last_24h` counts per action and outcome and the 20 most recent || GET | `/admin/storage/tiers/decisions?vod_id&action&outcome&limit&before_id` | this app's R2 promote/demote decisions, newest first: `{ id, decided_at, vod_id, object_id, action, from_provider, to_provider, outcome: done\|already\|refused\|failed, trigger: sweep\|admin\|clip\|drill\|manual, reason, inputs, thresholds, error }` → `{ decisions, next_before_id }` |++| GET | `/admin/storage/tiers/objects/policy` | native v2 objects' R2 tiering (below): the activation gate, each `media.object_tier` threshold as `{ value, default, source }`, the rules, this app's promotion candidates (with what would refuse each) and its decisions (24-hour counts and the 20 most recent) |++| GET | `/admin/storage/tiers/objects/decisions?object_id&action&outcome&limit&before_id` | this app's native object decisions, newest first (`outcome`: done\|already\|refused\|failed\|dry_run) → `{ decisions, next_before_id }` |++| GET / POST | `/admin/storage/config[/media.storage_tier\|/media.object_tier[/history\|/rollback]]` | the revisioned tier policies (openvibe-shared/config): read, apply `{ values, merge, reason }`, history, roll back || GET | `/admin/storage/holds?all&object_id&vod_id&clip_id&limit&before_id` | this app's retention holds, newest first (active only unless `all=1`) || POST | `/admin/storage/holds` | `{ object_id \| vod_id \| clip_id, reason, kind?, note?, placed_by? }` → 201: the object is kept (no delete, no tier move) and so are the clips cut from a held VOD. Not for calls acting for a user (403) || POST | `/admin/storage/holds/:holdId/release` | `{ released_by }` (also `DELETE /admin/storage/holds/:holdId`); 409 when already released. Holds stay on record |@@@ -412,7 +416,9 @@ apply it to an app acting for someone other than the owner.`GET /o/:id` serves object bytes by canonical id (`med_…`): public/unlistedobjects openly, private ones only with a valid signature from`/api/v2/:app/objects/:id/download`, deleted ones 410 (a deleted private object--without a signature is a 404 like any other private one).++without a signature is a 404 like any other private one). A native object in the++R2 popularity cache redirects to its verified R2 copy; a GET of a native object++counts its viewer for the day, identifying no one (see Storage tiering).## Object model and API v2@@@ -463,7 +469,22 @@ promote/demote); presigned-302 playback. Knobs live in `media_settings` underclip cut fetching an R2 VOD home, the eviction drill) is logged in `media_tier_decisions` with theinputs it saw (views, last access, provider, recording, hold, size), the thresholds in force and theirsource, the reason and the outcome; `/admin/storage/tiers/policy` shows the policy and the log, and--`media_tier_decisions_24h{action,outcome}` counts them on `/metrics` (`test/tier-decisions.test.js`). CLI:++`media_tier_decisions_24h{target="vod",action,outcome}` counts them on `/metrics` (`test/tier-decisions.test.js`).++++**Native v2 objects** (roadmap WS-G task 11; [docs](docs/object-model.md#tiering-of-native-objects)) keep++their canonical copy and, when popular, get a verified copy in R2 that `/o/:id` plays from. Popularity is++unique viewers per object per UTC day, counted on `/o/:id` without storing an IP address or subject id++(a daily-salted hash, deleted with its salt when the day ends; only the counts stay, 30 days), summed over++7 days. The policy is the revisioned namespace `media.object_tier`: promote at `promoteMinUniqueViewers7d`++(500) unique viewers, viewed within 3 days, 16 MB to 4 GB, at most 3 per sweep; demote after 14 idle days++or when the object is no longer ready. Its **activation gate `active` is off by default**: the sweep (step 4++of this one) then moves nothing and records what it would do as `dry_run` decisions. Holds block every++move, only a verified canonical copy is promoted (re-hashed; the R2 copy's size and sha256 are checked++before it is used), and a demotion never removes the last good copy. Decisions:++`media_object_tier_decisions`, `/admin/storage/tiers/objects/*`, `/me/ops`, and++`media_tier_decisions_24h{target="object"}` (`test/object-tiering.test.js`).++++The storage engine's CLI:```node server/vod/vod-storage.js check|migrate-legacy|drain [targetPct]@@@ -8,10 +8,10 @@ unchanged. New code talks to objects through the [v2 API](#object-api-v2).Code: `server/objects/` (`model.js`, `routes.js`, `backfill.js`, `reconcile.js`, `invariant.js`,`signing.js`, `verify-job.js`, `copy-report.js`, `multipart.js`, `content-type.js`, `readiness.js`, `namespaces.js`,--`namespace-routes.js`) and the job system in++`namespace-routes.js`, `popularity.js`, `tier-policy.js`, `tiering.js`) and the job system in`server/jobs/` (`queue.js`, `worker.js`, `routes.js`, `types.js`, `thumbnail.js`, `invariant-scan.js`,`derive.js`). Schema: the bottom of `server/db/schema.sql`. Tests: `test/objects-*.test.js`, `test/jobs*.test.js`,--`test/r2-eviction-drill.test.js`.++`test/r2-eviction-drill.test.js`, `test/object-tiering.test.js`.## Tables@@@ -244,7 +244,9 @@ All routes live under `/api/v2/:app/objects`.- Public and unlisted `ready` objects are served openly.- Private objects need a valid `?exp=&sig=` from `/download`. Without one the answer is 404, so a private object cannot be told apart from a missing one.- A deleted object answers 410 — after the check above, so a deleted private object is a 404 without a signature.--- A local copy streams with Range support. Otherwise the route redirects 302 to a presigned R2 URL, then a presigned B2 URL.++- A native object [promoted to the R2 cache](#tiering-of-native-objects) redirects 302 to its verified R2 copy (a presigned URL that answers with the type and `Content-Disposition` this route would send) while R2 is available.++- Otherwise a local copy streams with Range support, or the route redirects 302 to a presigned R2 URL, then a presigned B2 URL.++- A GET of a native object counts its viewer for the day, without identifying anyone ([popularity](#popularity)).- Headers: `X-Content-Type-Options: nosniff` and `X-Robots-Tag: noindex`. Content is `inline` only for images (not SVG), video, audio, PDF and plain text, and `attachment` for everything else.**Signing.** HMAC-SHA256 keyed with `MEDIA_SIGNING_SECRET` over a purpose and what the link may do:@@@ -318,7 +320,9 @@ Code: `server/objects/content-type.js`.**Purge.** Every hour, the service removes the bytes of native objects whose retention period haspassed and that have no active hold. The object row stays `deleted` with `metadata.purged_at`.--Projected objects are never purged by this job.++Projected objects are never purged by this job. An object that still has a copy in the R2 cache waits++until the [object tiering](#tiering-of-native-objects) has removed it (it demotes deleted objects), so a++purge never leaves the R2 copy behind.## Namespaces, grants and quotas@@@ -454,7 +458,7 @@ cheap and renders in a drill.- `jobs`: counts by status; failed jobs of the last 7 days by type and error code; the most recent failures with their error text.- `missing`: ready objects with no good copy; copies recorded missing or corrupt; copies by provider and state; quarantined VODs; the last verification run.- `backfill`: the last object backfill that changed something; rows with no object yet; objects whose owner subject is unresolved; whether upload reservations were seeded.--- `tiering`: providers and policy; VODs and clips per provider; objects per canonical copy (native or projected); VODs eligible to offload; R2 decisions of the last 24 h and recent refusals and failures; the sweep's state in this process.++- `tiering`: providers and policy; VODs and clips per provider; objects per canonical copy (native or projected); VODs eligible to offload; R2 decisions of the last 24 h and recent refusals and failures; the sweep's state in this process; and `native_objects`: the [object tiering](#tiering-of-native-objects)'s activation gate and policy, native objects in R2, how many are eligible to promote now, its decisions of the last 24 h (dry runs included) and the most recent ones, and its last sweep.- `namespaces`: every namespace with quota and usage snapshot.## Readiness@@@ -518,7 +522,7 @@ While any hold applies:- `deleteVodObjects` keeps the bytes.- Paste screenshot removal keeps the file, and censor answers 409.- A `BEFORE DELETE` trigger on each projected table, plus a trigger on `media_objects.lifecycle_status`, refuses the change, so paths nobody has hooked are still covered. The triggers are the `_v2` ones (clip-aware); an older database has its first-version guards replaced at start.--- **No tier move.** `moveToCold`, `moveToHot`, `promoteToR2` and `demoteFromR2` return `{ ok: false, held: true }` (the admin moves answer it, R2 moves are logged as `refused`), and the sweep counts these as `skippedHeld` instead of errors. A hold freezes an object's placement: `moveToHot` is refused too, because restoring flips the row to local and drops an R2 copy. A clip cut from a held VOD in B2/R2 is cut from the cloud copy instead of fetching the VOD home. Row updates that only record where the bytes already are (the sweep finding a local file gone while B2 has the copy, the legacy-tier migration) still happen: they move nothing.++- **No tier move.** `moveToCold`, `moveToHot`, `promoteToR2` and `demoteFromR2` return `{ ok: false, held: true }` (the admin moves answer it, R2 moves are logged as `refused`), and the sweep counts these as `skippedHeld` instead of errors. A held native object is never promoted to or demoted from the R2 cache either (logged as `refused`; a hold placed while its bytes were being copied wins, and the unrecorded copy is removed). A hold freezes an object's placement: `moveToHot` is refused too, because restoring flips the row to local and drops an R2 copy. A clip cut from a held VOD in B2/R2 is cut from the cloud copy instead of fetching the VOD home. Row updates that only record where the bytes already are (the sweep finding a local file gone while B2 has the copy, the legacy-tier migration) still happen: they move nothing.**Staff routes** (`/api/v1/:app/admin/storage/holds`, the app key like every admin route; a call actingfor one of the app's users, `X-OV-User-Id`, gets 403):@@@ -855,6 +859,105 @@ node scripts/r2-eviction-drill.js (--vod <id> | --pick) [--execute] [--base-urlNot run against production yet: it needs the owner's go-ahead.++## Tiering of native objects++++Roadmap WS-G task 11. Code: `server/objects/popularity.js`, `server/objects/tier-policy.js`,++`server/objects/tiering.js`. Test: `test/object-tiering.test.js` (a fake S3 through the real SDK).++++A native object (uploaded through `/api/v2`, `legacy_ref` NULL) keeps its canonical copy where the upload++put it: local disk (`OBJECTS_PATH`), or B2 for a B2 canonical copy. When it is popular it also gets a copy++in the Cloudflare R2 popularity cache (`storage_class` `cache`, key `objects/<app>/<id>`, or the B2 key),++and that copy becomes its playback source: `GET /o/:id` redirects to it while R2 is available, and the++canonical copy is the fallback. When the object goes idle, or is no longer `ready` (deleted), the R2 copy++is removed again. Projected objects (VODs, clips) keep tiering through `vod-storage.js` as before.++++### Popularity++++Unique viewers per object per UTC day, with no IP address or subject id stored (ADR-021):++++- **Who is a viewer.** The client's network: `req.ip` (see [Client addresses](../README.md#client-addresses)),++ an IPv6 address reduced to its /64. One network is one viewer, whatever its user agent: people behind++ one NAT count once, which errs on the conservative side, and a client cannot inflate a count.++- **What is stored.** `media_object_viewer_days` holds `HMAC-SHA256(salt of the day, network)`, 16 hex++ characters, per object and day; the salt is 32 random bytes per UTC day (`media_object_view_salts`).++ `media_object_views_daily` holds the count (`unique_viewers`, `last_viewed_at`) per object and day.++- **Rotation.** Once a UTC day is over, its hashes and its salt are deleted: by the first view of the next++ day, by the hourly maintenance and at the start of every tiering sweep. After that nothing links a count++ to an address, even with the database and every secret. The daily counts are kept 30 days.++- **The 7-day figure** is the sum of the last seven daily counts (today included): viewer-days. The same++ network on three days counts three times, as in the shared analytics' multi-day uniques.++- **Counted** on `GET /o/:id` of a ready, native object of a non-sandbox tenant, public or signed. **Not++ counted:** `HEAD`, range requests that do not start at byte 0 (a player's seeks), `Sec-GPC: 1` or++ `DNT: 1` (no hash is made at all), crawlers and clients without a user agent++ (`openvibe-shared/analytics` `privacy.optedOut` / `classifyRequest`). A response that Cloudflare serves++ from its cache never reaches Media and is not counted.++++### Policy++++The namespace `media.object_tier` of `openvibe-shared/config`, beside `media.storage_tier`: revisions with++who and why, validated as a whole, history and rollback through `…/admin/storage/config/media.object_tier`.++Revision 1 sets nothing, so every key is its built-in default until a revision sets it.++++| key | default | meaning |++|---|---|---|++| `active` | `false` | **the site-level activation gate.** Off: nothing moves (not through the sweep, not through a direct `promote()`/`demote()`); the sweep records what it would do as `dry_run` decisions |++| `promoteMinUniqueViewers7d` | 500 | unique viewers over the last 7 UTC days to promote (1 to 10⁹) |++| `promoteRecentAccessDays` | 3 | …and viewed within this many days, today included (1-30) |++| `promoteMinSizeMb` / `promoteMaxSizeMb` | 16 / 4096 | size bounds: a small object is cheap to serve locally; the upper bound caps the bytes one promotion hashes and reads back |++| `maxPromotionsPerSweep` | 3 | promotions (and dry runs) per sweep (0-100) |++| `demoteIdleDays` | 14 | no viewer for this many days, and in R2 at least that long: demote (1-30) |++| `maxDemotionsPerSweep` | 10 | demotions per sweep (0-500) |++++Refused (422 `config.invalid`, nothing changes): a wrong type, an unknown key, a value out of range,++`promoteMinSizeMb` not below `promoteMaxSizeMb`, and `demoteIdleDays` not above `promoteRecentAccessDays`++(an object viewed recently enough to promote would also be idle enough to demote).++++### Sweep and moves++++The sweep is step 4 of the storage sweep (`vod-storage.js runSweep`, every `sweepIntervalMs`, and++`POST /admin/storage/tiers/sweep`); its result is the storage sweep's `objects`. It is skipped while the++local disk still needs a drain, never starts in a restore drill (and refuses under `MEDIA_DRILL` anyway).++Each pass rotates the popularity counts, then:++++1. **Demotes** first (at most `maxDemotionsPerSweep`): R2 copies of native objects that are no longer++ `ready`, or idle for `demoteIdleDays` and in R2 at least that long.++2. **Promotes** (at most `maxPromotionsPerSweep`), most viewed first: ready native objects of non-sandbox++ tenants within the thresholds and without a present R2 copy.++++**Promotion** (`tiering.promote`): refused when the object is held, not ready, a sandbox object, or its++canonical copy is not verified on record (present, checked, not contradicted, and the object has a++sha256: the `object.hash` job supplies it for older objects); then the gate (`dry_run`); then R2 must be++available. The canonical copy is checked again (a local file is hashed and must match the sha256; a B2++copy must answer a HEAD with the right size), uploaded (or copied from B2), and the R2 copy is HEADed and++read back: its size and sha256 must match. Only then is the R2 location recorded `present` (with the++checksum), which makes it the playback source. A copy that does not verify is deleted and never recorded++(`failed`); a hold or a delete that arrived during the copy wins (`refused`, the copy removed).++++**Demotion** (`tiering.demote`): refused under a hold; then the gate (`dry_run`); then the canonical copy++must check out now (hashed again, or HEAD for B2). A copy that does not (missing, changed) keeps the R2++copy, which may be the last good one (`refused`). The R2 object is deleted, a HEAD must confirm it is gone++(`failed` otherwise, and the location stays), and the location row is removed.++++**Decisions.** Every outcome goes to `media_object_tier_decisions` (`done`, `already`, `refused`,++`failed`, `dry_run`) with the inputs it saw (`unique_viewers_7d`, `last_viewed_day`, size, kind,++visibility, lifecycle, the canonical copy's state, `held`, the R2 copy, `gate_active`, `r2_available`),++the policy in force (each value with its source) and the reason. A refusal or dry run the sweep repeats is++logged once a day per object, and a repeated dry run does not use up the sweep's budget, so over a day the++log shows what successive sweeps would move. An object whose move failed waits 6 hours. They are counted++in `media_tier_decisions_24h{target="object",action,outcome}` on `/metrics` (VOD decisions are++`target="vod"`) and shown in the [operator views](#operator-views).++++**Admin API** (`/api/v1/:app/admin/storage`, the app key, this app's objects only):++++| method | path | notes |++|---|---|---|++| GET | `/tiers/objects/policy` | `gate`, each threshold as `{ value, default, source }`, the rules in words, R2's status, `candidates` (what would be promoted now: `object_id`, `unique_viewers_7d`, `last_viewed_day`, `blocked_by` when something on record would refuse it; `?limit`), `decisions.last_24h` per action and outcome and the 20 most recent |++| GET | `/tiers/objects/decisions` | `?object_id&action=promote\|demote&outcome=done\|already\|refused\|failed\|dry_run&limit (≤200)&before_id` → `{ decisions, next_before_id }`, newest first |++| GET / POST | `/config/media.object_tier`, `…/history`, `…/rollback` | the policy's revisions (`POST …/media.object_tier` `{ values: { active: true }, merge: true, reason }` turns the gate on) |++++**With the gate on and then off again,** placement freezes: promoted objects keep playing from R2, and a++deleted object's R2 copy stays (so its purge waits) until the gate is on again.++## Configuration| env | default | meaning |@@@ -897,7 +1000,8 @@ Not run against production yet: it needs the owner's go-ahead.## Not in this pass- Presigned direct-to-bucket uploads. Presigned URLs are Media-signed and native bytes are stored locally.--- Tiering of native objects to B2/R2.++- Offloading native objects' canonical copy to B2. Popular native objects are cached in R2++ ([Tiering of native objects](#tiering-of-native-objects)); their canonical copy stays local.- Copy/move aliases, lifecycle rules and an S3-compatible façade.- Clip cutting and the finalize remux still run inline, not as jobs (thumbnail regeneration was the firstoperation moved onto the job system).@@@ -912,3 +1016,5 @@ The tiering thresholds (`DEFAULTS` in `server/vod/vod-storage.js`) are the names- **Revision 1** is what `media_settings` held.- **Mirror:** every activation writes the active revision's explicit keys through to `media_settings` (`storage_tier.*`) and removes the rest, so a release that reads those rows sees the same policy. The rows go in the ADR-028 contract step.- **Sweep:** a running sweep restarts with the new policy.++- **Native objects** have their own namespace, `media.object_tier` (`server/objects/tier-policy.js`): the++ same revisions and routes, nothing mirrored to `media_settings`. See [Tiering of native objects](#tiering-of-native-objects).@@@ -21,6 +21,12 @@* decisions (media_tier_decisions) with 24-hour counts* GET /tiers/decisions this app's R2 promote/demote decisions, newest first* (?vod_id&action=promote|demote&outcome&limit≤200&before_id)++ * GET /tiers/objects/policy native v2 objects (server/objects/tiering.js): the activation gate, each++ * threshold with its source, the rules, this app's promotion candidates (and what++ * would refuse each), and its decisions (media_object_tier_decisions) with 24-hour counts++ * GET /tiers/objects/decisions this app's native object decisions, newest first++ * (?object_id&action&outcome=done|already|refused|failed|dry_run&limit≤200&before_id)++ * GET /config… the revisioned policies media.storage_tier and media.object_tier (openvibe-shared/config)* GET /holds this app's retention holds, newest first (?all=1 includes released;* ?object_id|vod_id|clip_id; ?limit≤200&before_id)* POST /holds place one: { object_id (med_… or legacy ref) | vod_id | clip_id, reason, kind@@@ -347,7 +353,8 @@ router.put('/tiers/settings', async (req, res) => {let configRoutes = null;router.use('/config', (req, res, next) => {if (!configRoutes) {-- configRoutes = require('openvibe-shared/config').adminRoutes([vodStorage.tierConfig.get(vodStorage.DEFAULTS)], {++ // media.storage_tier (VODs) and media.object_tier (native v2 objects: thresholds and the activation gate).++ configRoutes = require('openvibe-shared/config').adminRoutes([vodStorage.tierConfig.get(vodStorage.DEFAULTS), require('../objects/tier-policy').get()], {basePath: '/', requireAdmin: (_rq, _rs, nx) => nx(), actor: (rq) => ({ type: 'service', id: rq.appId }),});}@@@ -415,6 +422,43 @@ router.get('/tiers/decisions', (req, res) => {}});++// ── Native v2 objects: policy, candidates and decision log (read-only) ──++// The policy changes through the config routes (…/config/media.object_tier); the sweep moves nothing while++// its activation gate (`active`) is off, and logs what it would do as dry_run decisions.++const objectTiers = () => require('../objects/tiering');++++router.get('/tiers/objects/policy', (req, res) => {++ try {++ const policy = require('../objects/tier-policy');++ const settings = policy.settings();++ res.json({++ gate: { active: !!settings.active, ...policy.thresholds(settings).active },++ thresholds: policy.thresholds(settings),++ rules: policy.describe(settings),++ provider: { r2: { configured: vodStorage.providerConfigured('r2'), available: vodStorage.providerAvailable('r2') } },++ candidates: objectTiers().candidates({ appId: req.appId, limit: req.query.limit, settings }),++ decisions: { last_24h: objectTiers().counts24h(req.appId), recent: objectTiers().listDecisions({ appId: req.appId, limit: 20 }).decisions },++ note: 'Read-only. The policy (media.object_tier) changes through /config/media.object_tier; candidates and decisions are this app\'s only.',++ });++ } catch (err) {++ console.error('[Admin] Object tier policy error:', err.message);++ res.status(500).json({ error: 'Failed to read the object tier policy' });++ }++});++++router.get('/tiers/objects/decisions', (req, res) => {++ try {++ const q = req.query;++ if (q.action && !['promote', 'demote'].includes(String(q.action))) return res.status(400).json({ error: 'action must be promote or demote' });++ if (q.outcome && !objectTiers().OUTCOMES.includes(String(q.outcome))) return res.status(400).json({ error: `outcome must be one of ${objectTiers().OUTCOMES.join(', ')}` });++ res.json(objectTiers().listDecisions({ appId: req.appId, objectId: q.object_id ? String(q.object_id) : null, action: q.action ? String(q.action) : null,++ outcome: q.outcome ? String(q.outcome) : null, beforeId: q.before_id != null ? parseInt(q.before_id, 10) || 0 : null, limit: q.limit }));++ } catch (err) {++ console.error('[Admin] Object tier decisions error:', err.message);++ res.status(500).json({ error: 'Failed to list object tier decisions' });++ }++});++// ── POST /tiers/sweep — trigger a sweep now ──────────────────router.post('/tiers/sweep', async (req, res) => {try {@@@ -429,6 +429,50 @@ CREATE TABLE IF NOT EXISTS media_tier_decisions (CREATE INDEX IF NOT EXISTS idx_media_tier_decisions_vod ON media_tier_decisions(vod_id, id);CREATE INDEX IF NOT EXISTS idx_media_tier_decisions_app ON media_tier_decisions(app_id, id);++-- Popularity of native v2 objects (server/objects/popularity.js; ADR-021): unique viewers per object per UTC++-- day, never an IP address or subject id. A viewer is HMAC-SHA256(the day's random salt, client network), and++-- the hashes and the salt of a day are deleted once it is over; only the daily counts stay (30 days).++CREATE TABLE IF NOT EXISTS media_object_view_salts (++ day TEXT PRIMARY KEY, -- YYYY-MM-DD (UTC)++ salt TEXT NOT NULL, -- 32 random bytes, hex; deleted with the day's hashes++ created_at DATETIME DEFAULT CURRENT_TIMESTAMP++);++CREATE TABLE IF NOT EXISTS media_object_viewer_days (++ day TEXT NOT NULL,++ object_id TEXT NOT NULL,++ viewer TEXT NOT NULL, -- 16 hex chars of HMAC-SHA256(salt of the day, client network)++ PRIMARY KEY (day, object_id, viewer)++) WITHOUT ROWID;++CREATE TABLE IF NOT EXISTS media_object_views_daily (++ object_id TEXT NOT NULL,++ day TEXT NOT NULL, -- YYYY-MM-DD (UTC)++ unique_viewers INTEGER NOT NULL DEFAULT 0,++ last_viewed_at TEXT, -- ISO 8601: the day's last new viewer++ PRIMARY KEY (object_id, day)++);++CREATE INDEX IF NOT EXISTS idx_media_object_views_daily_day ON media_object_views_daily(day, object_id);++++-- Tier decisions for native v2 objects (server/objects/tiering.js): every promotion to and demotion from the++-- R2 popularity cache the object sweep decided, including what it would have done while the activation gate++-- (media.object_tier `active`) is off (dry_run), with the inputs it saw and the policy in force.++CREATE TABLE IF NOT EXISTS media_object_tier_decisions (++ id INTEGER PRIMARY KEY AUTOINCREMENT,++ decided_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),++ object_id TEXT NOT NULL,++ app_id TEXT,++ action TEXT NOT NULL CHECK(action IN ('promote', 'demote')),++ from_provider TEXT,++ to_provider TEXT,++ outcome TEXT NOT NULL CHECK(outcome IN ('done', 'already', 'refused', 'failed', 'dry_run')),++ trigger TEXT NOT NULL, -- sweep | manual++ reason TEXT NOT NULL,++ inputs TEXT NOT NULL DEFAULT '{}', -- JSON: unique_viewers_7d, last_viewed_day, size, kind, lifecycle, canonical copy, held, R2 copy, gate++ thresholds TEXT NOT NULL DEFAULT '{}',-- JSON: the media.object_tier policy in force, each value with its source (default | setting)++ error TEXT++);++CREATE INDEX IF NOT EXISTS idx_media_object_tier_decisions_object ON media_object_tier_decisions(object_id, id);++CREATE INDEX IF NOT EXISTS idx_media_object_tier_decisions_app ON media_object_tier_decisions(app_id, id);++-- Object changes waiting to become events (server/events.js recordObjectChanges): media_objects-- triggers (database.js ensureObjectTriggers) write one row per visibility change and per deletion,-- in the transaction that makes the change, whatever path made it; the rows become@@@ -242,6 +242,8 @@ if (!drill.enabled) {try { const n = require('./objects/model').purgeExpired(); if (n) console.log(`[Objects] Purged ${n} expired deleted object(s)`); } catch (err) { console.warn('[Objects] purge:', err.message); }// Incomplete multipart uploads past MEDIA_MULTIPART_TTL_HOURS: their parts are deleted (the objects stay uploading).try { const r = require('./objects/multipart').purgeExpired(); if (r.expired || r.orphan_dirs) console.log(`[Objects] Multipart: ${r.expired} expired session(s), ${r.orphan_dirs} orphan part dir(s) removed`); } catch (err) { console.warn('[Objects] multipart purge:', err.message); }++ // Native objects' popularity (ADR-021): a finished UTC day's viewer hashes and salt are deleted, only its counts stay (30 days).++ try { const r = require('./objects/popularity').rotate(); if (r.hashes || r.salts || r.counts) console.log(`[Popularity] ${r.hashes} viewer hash(es) and ${r.salts} salt(s) of finished days deleted, ${r.counts} old daily count(s) pruned`); } catch (err) { console.warn('[Popularity] rotate:', err.message); }// Namespaces: uploads whose quota reservation expired are failed and their bytes freed; then every// namespace's usage snapshot is refreshed (v1 files, deletes and purges included).try {@@@ -156,6 +156,9 @@ function tiering(appId, limit) {FROM media_tier_decisions WHERE outcome IN ('refused', 'failed')${s.sql} ORDER BY id DESC LIMIT ?`, [...s.params, Math.min(limit, 20)]).map(r => ({ ...r, reason: clip(r.reason, 300), error: clip(r.error, 300) }));const sweep = vodStorage.sweepInfo();++ // Native v2 objects (objects/tiering.js): the activation gate, the policy, R2 copies, what would be promoted++ // now, and the decisions (dry runs included), from the database like everything here.++ const nativeObjects = require('../objects/tiering').report({ appId, limit: Math.min(limit, 20) });return {providers: { b2: vodStorage.providerConfigured('b2'), r2: vodStorage.providerConfigured('r2') },policy: { enabled: !!settings.enabled, r2_enabled: !!settings.r2Enabled, min_age_days: settings.minAgeDays, max_views_for_cold: settings.maxViewsForCold, min_last_access_days: settings.minLastAccessDays },@@@ -175,7 +178,8 @@ function tiering(appId, limit) {timestamp: sweep.lastResult.timestamp || null,} : null,},-- note: 'Native v2 objects stay on local storage until roadmap WS-G task 11 (tiering for native objects). The sweep and the provider switches are service-wide; counts follow the scope.',++ native_objects: nativeObjects,++ note: 'VODs tier by the media.storage_tier policy; native v2 objects keep their canonical copy and are cached in R2 by the media.object_tier policy, only while its activation gate is on (dry runs otherwise). The sweep, the gate and the provider switches are service-wide; counts follow the scope.',};}@@@ -331,6 +331,7 @@ function renderOps({ person, report, canRecompute, recomputed = null }) {const card = (value, label) => `<div class="card"><b>${value}</b><span>${esc(label)}</span></div>`;const kv = (obj) => Object.entries(obj || {}).map(([k, v]) => `${esc(k.replace(/_/g, ' '))} ${num(v)}`).join(' · ') || 'none';const t = r.tiering;++ const no = t.native_objects || null;const bf = r.backfill;const m = r.missing;const body = `@@@ -418,6 +419,16 @@ function renderOps({ person, report, canRecompute, recomputed = null }) {${table('Recent refused or failed tier moves', ['When', 'Tenant', 'VOD', 'Move', 'Outcome', 'Why'], t.recent_problems.map(d => [esc(fmtWhen(d.decided_at)), esc(d.app_id || ''), esc(d.vod_id), `${esc(d.action)} ${esc(d.from_provider || '')} → ${esc(d.to_provider || '')}`, esc(d.outcome), esc(d.error || d.reason || ''),])) || '<p class="empty">None.</p>'}++ ${no ? `<h3>Native objects</h3>++ <dl class="facts">++ <dt>Activation gate</dt><dd>${no.gate.active ? '<strong>on</strong>: the sweep moves objects' : 'off: dry runs only'} (${esc(no.gate.source)})</dd>++ <dt>Policy</dt><dd>promote at ${num(no.policy.promoteMinUniqueViewers7d)} unique viewers in 7 days, viewed within ${esc(no.policy.promoteRecentAccessDays)} d, ${esc(no.policy.promoteMinSizeMb)} to ${esc(no.policy.promoteMaxSizeMb)} MB · demote after ${esc(no.policy.demoteIdleDays)} idle days</dd>++ <dt>In R2</dt><dd>${num(no.r2_copies.count)} objects, ${esc(fmtBytes(no.r2_copies.bytes))} · eligible to promote now: ${num(no.eligible_to_promote)}</dd>++ <dt>Decisions, 24 h</dt><dd>promote: ${kv(no.decisions_24h.promote)} · demote: ${kv(no.decisions_24h.demote)}</dd>++ </dl>++ ${table('Recent native object tier decisions', ['When', 'Tenant', 'Object', 'Action', 'Outcome', 'Why'], no.recent.map(d => [++ esc(fmtWhen(d.decided_at)), esc(d.app_id || ''), `<code class="id">${esc(d.object_id)}</code>`, esc(d.action), esc(d.outcome.replace('_', ' ')), esc(d.error || d.reason || ''),++ ])) || '<p class="empty">None.</p>'}` : ''}</section><p class="hint">JSON: <a href="/api/v2/me/ops${scopeQs}">/api/v2/me/ops</a>.</p>`;return doc({ title: 'Media operations', description: 'Operator views of OpenVibe.Media: failed jobs, usage, missing media, backfill and tiering.', body });@@@ -658,7 +658,9 @@ function restore(obj) {/*** Remove the bytes of native (v2) objects deleted longer ago than the retention* period. Projected objects are never purged here — their bytes belong to the-- * inherited pipelines. Held objects are skipped.++ * inherited pipelines. Held objects are skipped, and so is an object that still++ * has a copy in the R2 popularity cache: the object tiering removes that first++ * (./tiering.js demotes deleted objects), so a purge never leaves it behind.*/function purgeExpired({ retentionDays = config.objects.retentionDays } = {}) {const rows = db.all(`SELECT * FROM media_objects WHERE lifecycle_status = 'deleted' AND legacy_ref IS NULL@@@ -668,6 +670,7 @@ function purgeExpired({ retentionDays = config.objects.retentionDays } = {}) {for (const obj of rows) {const md = parseJson(obj.metadata, {});if (md.purged_at || isHeld(obj.id)) continue;++ if (listLocations(obj.id).some(l => l.provider === 'r2')) continue;for (const l of listLocations(obj.id)) {if (l.provider === 'local' && path.resolve(l.key).startsWith(root)) { try { fs.unlinkSync(l.key); } catch { /* already gone */ } }if (l.provider === 'local') db.run('DELETE FROM media_locations WHERE id = ?', [l.id]);nnew file mode 100644@@@ -0,0 +1,180 @@++/**++ * OpenVibe.Media — popularity of native v2 objects, counted without identifying anyone (roadmap WS-G task 11;++ * ADR-021; docs/object-model.md#tiering-of-native-objects).++ *++ * record(obj, req) GET /o/:id: the request's viewer counts once per object per UTC day++ * stats({ ids, days }) unique viewers per object over the last `days` UTC days (today included)++ * rotate() the hashes and the salt of every finished day are deleted; counts past 30 days too++ *++ * A viewer is the client's network: req.ip (server/client-ip.js: the address nginx saw, never a header a++ * direct caller sends), an IPv6 address reduced to its /64. It is never stored. A row holds HMAC-SHA256(salt++ * of the day, network), 16 hex characters (media_object_viewer_days), and the salt is 32 random bytes per UTC++ * day (media_object_view_salts). Once the day is over, rotate() deletes that day's hashes and its salt: from++ * then on nothing links a count to an address, not even with the database and every secret. What stays is one++ * count per object and day (media_object_views_daily: unique_viewers, last_viewed_at), for 30 days. The same++ * network on two days is two viewer-days, so the 7-day figure is the sum of seven daily counts. One address++ * is one viewer: people behind one NAT count once (the conservative side), and changing the user agent does++ * not make a client count again.++ *++ * Not counted: HEAD and range requests past the first byte (a player's seeks), requests with Sec-GPC: 1 or++ * DNT: 1 (openvibe-shared/analytics privacy.optedOut: no hash is made at all), crawlers and clients without a++ * user agent (privacy.classifyRequest). Projected objects (vods, clips, files) keep their own view counts++ * (server/views/service.js). Nothing runs at load; a restore drill never reaches record() (/o answers 503++ * first) and never rotates.++ */++'use strict';++++const crypto = require('crypto');++const net = require('net');++const db = require('../db/database');++const { privacy } = require('openvibe-shared/analytics');++const { clientIp } = require('../client-ip');++++const WINDOW_DAYS = 7; // the rolling window the tiering policy reads++const KEEP_DAYS = 30; // daily counts are kept this long (ADR-021's bound for raw analytics, applied to the counts too)++const SEEN_MAX = 200000; // in-memory "already counted today" keys before the set starts over++++/** YYYY-MM-DD (UTC) of an epoch-ms time. */++function dayOf(ms = Date.now()) {++ return new Date(ms).toISOString().slice(0, 10);++}++++/** The day `n` days after `day` (negative: before). */++function addDays(day, n) {++ return dayOf(Date.parse(`${day}T00:00:00Z`) + n * 864e5);++}++++/** The first day of the `days`-day window that ends on (and includes) `today`. */++function windowStart(today, days = WINDOW_DAYS) {++ return addDays(today, -(Math.max(1, Number(days) || 1) - 1));++}++++/** The network a viewer is counted as: an IPv4 address (also when IPv4-mapped), or the /64 of an IPv6 address. */++function networkOf(ip) {++ let s = String(ip || '').trim().toLowerCase();++ if (s.startsWith('::ffff:') && net.isIPv4(s.slice(7))) s = s.slice(7);++ if (net.isIPv4(s)) return s;++ s = s.split('%')[0];++ if (!net.isIPv6(s)) return s || 'unknown';++ const [head, tail] = s.includes('::') ? s.split('::') : [s, null];++ const h = head ? head.split(':') : [];++ const t = tail ? tail.split(':') : [];++ // An embedded IPv4 tail stands for two groups; only the first four groups are kept, which it never reaches.++ const groups = [...h, ...(tail === null ? [] : Array(Math.max(0, 8 - h.length - t.length)).fill('0')), ...t];++ return `${groups.slice(0, 4).map((g) => parseInt(g || '0', 16).toString(16)).join(':')}::/64`;++}++++// ── The salt of the day ──────────────────────────────────────++++let _saltDay = null, _salt = null;++++/** The day's random salt: read, or made by the first view of the day (a racing writer keeps the first). */++function daySalt(day) {++ if (_saltDay === day && _salt) return _salt;++ let row = db.get('SELECT salt FROM media_object_view_salts WHERE day = ?', [day]);++ if (!row) {++ db.run('INSERT OR IGNORE INTO media_object_view_salts (day, salt) VALUES (?, ?)', [day, crypto.randomBytes(32).toString('hex')]);++ row = db.get('SELECT salt FROM media_object_view_salts WHERE day = ?', [day]);++ }++ _saltDay = day;++ _salt = row.salt;++ return _salt;++}++++/** The stored form of a viewer on one day: 16 hex chars of HMAC-SHA256(salt of the day, network). */++function viewerHash(network, day) {++ return crypto.createHmac('sha256', daySalt(day)).update(String(network)).digest('hex').slice(0, 16);++}++++// ── Recording ────────────────────────────────────────────────++++let _day = null;++const _seen = new Set(); // `${object}|${viewer}` counted today in this process: no write for a repeat++++/** Why this request is not counted, or null when it is a view. */++function notCounted(req) {++ if (!req || req.method !== 'GET') return 'method';++ const range = req.headers && req.headers.range;++ if (range && !/^bytes=0-/.test(String(range))) return 'range';++ if (privacy.optedOut(req.headers)) return 'opted_out';++ if (privacy.classifyRequest(req).isBot) return 'bot';++ return null;++}++++/**++ * Count the request's viewer for a native object (legacy_ref NULL), at most once per UTC day. Never throws.++ * → { counted, reason } (reason: not_native, method, range, opted_out, bot, seen, error; null when counted)++ */++function record(obj, req, { now = Date.now() } = {}) {++ try {++ if (!obj || !obj.id || obj.legacy_ref) return { counted: false, reason: 'not_native' };++ const why = notCounted(req);++ if (why) return { counted: false, reason: why };++ const day = dayOf(now);++ if (day !== _day) {++ // The first view of a new day: yesterday's hashes and salt go now (the hourly rotation would too).++ _day = day;++ _seen.clear();++ rotate({ now });++ }++ const viewer = viewerHash(networkOf(clientIp(req)), day);++ const seenKey = `${obj.id}|${viewer}`;++ if (_seen.has(seenKey)) return { counted: false, reason: 'seen' };++ const at = new Date(now).toISOString();++ const added = db.getDb().transaction(() => {++ const r = db.run('INSERT OR IGNORE INTO media_object_viewer_days (day, object_id, viewer) VALUES (?, ?, ?)', [day, obj.id, viewer]);++ if (!r.changes) return false;++ db.run(`INSERT INTO media_object_views_daily (object_id, day, unique_viewers, last_viewed_at) VALUES (?, ?, 1, ?)++ ON CONFLICT(object_id, day) DO UPDATE SET unique_viewers = unique_viewers + 1, last_viewed_at = excluded.last_viewed_at`, [obj.id, day, at]);++ return true;++ })();++ if (_seen.size >= SEEN_MAX) _seen.clear();++ _seen.add(seenKey);++ return { counted: added, reason: added ? null : 'seen' };++ } catch (err) {++ console.warn('[Popularity] view not recorded:', err.message);++ return { counted: false, reason: 'error' };++ }++}++++/**++ * Delete what must not outlive its day: every viewer hash and salt of a day before today, and daily counts++ * older than KEEP_DAYS. → { hashes, salts, counts } (rows deleted)++ */++function rotate({ now = Date.now() } = {}) {++ const today = dayOf(now);++ const out = db.getDb().transaction(() => ({++ hashes: db.run('DELETE FROM media_object_viewer_days WHERE day < ?', [today]).changes,++ salts: db.run('DELETE FROM media_object_view_salts WHERE day < ?', [today]).changes,++ counts: db.run('DELETE FROM media_object_views_daily WHERE day < ?', [addDays(today, -KEEP_DAYS)]).changes,++ }))();++ if (_saltDay && _saltDay < today) { _saltDay = null; _salt = null; }++ return out;++}++++/**++ * Unique viewers over the last `days` UTC days (today included), per object: Map(object_id → { unique_viewers,++ * last_viewed_day, last_viewed_at, days_viewed }). `ids` narrows it (else every object with a view in the window).++ */++function stats({ ids = null, days = WINDOW_DAYS, now = Date.now() } = {}) {++ const since = windowStart(dayOf(now), days);++ const list = ids == null ? null : [].concat(ids).filter(Boolean);++ if (list && !list.length) return new Map();++ const rows = db.all(`SELECT object_id, SUM(unique_viewers) AS viewers, MAX(day) AS last_day, MAX(last_viewed_at) AS last_at, COUNT(*) AS days++ FROM media_object_views_daily WHERE day >= ?${list ? ` AND object_id IN (${list.map(() => '?').join(',')})` : ''}++ GROUP BY object_id`, [since, ...(list || [])]);++ return new Map(rows.map((r) => [r.object_id, { unique_viewers: r.viewers, last_viewed_day: r.last_day, last_viewed_at: r.last_at, days_viewed: r.days }]));++}++++/** The last day anyone viewed the object (in the kept counts), or null. */++function lastViewedDay(objectId) {++ const r = db.get('SELECT MAX(day) AS d FROM media_object_views_daily WHERE object_id = ? AND unique_viewers > 0', [objectId]);++ return (r && r.d) || null;++}++++/** Test hook: forget the in-memory day, salt and repeats (as a restart would). */++function _reset() {++ _day = null; _seen.clear(); _saltDay = null; _salt = null;++}++++module.exports = { WINDOW_DAYS, KEEP_DAYS, dayOf, addDays, windowStart, networkOf, viewerHash, record, rotate, stats, lastViewedDay, _reset };@@@ -722,10 +722,20 @@ publicRouter.get('/:id', async (req, res) => {'Cache-Control': obj.visibility === 'private' || sandbox ? 'private, no-store' : 'public, max-age=3600','Content-Disposition': `${INLINE.test(mime) ? 'inline' : 'attachment'}; filename="${name}"`,};++ // A viewer of a native object, counted once a day without being identified (objects/popularity.js); the++ // object tiering reads these counts. Sandbox objects are never tiered, so they are not counted.++ if (!obj.legacy_ref && !sandbox) require('./popularity').record(obj, req);const locs = model.listLocations(obj.id);++ const vodStorage = require('../vod/vod-storage');++ // A native object promoted to the R2 popularity cache (objects/tiering.js) plays from its verified R2 copy++ // while R2 is available, with the headers this route would send; the canonical copy is the fallback.++ const cached = !obj.legacy_ref && locs.find(l => l.provider === 'r2' && l.state === 'present' && l.verified_at);++ if (cached && vodStorage.providerAvailable('r2')) {++ const url = await vodStorage.presignGet('r2', cached.key, 300, { contentType: mime, contentDisposition: headers['Content-Disposition'] }).catch(() => null);++ if (url) { res.set('Cache-Control', 'private, max-age=0'); res.set('X-Robots-Tag', 'noindex'); return res.redirect(302, url); }++ }const local = locs.find(l => l.provider === 'local' && l.state !== 'missing' && fs.existsSync(l.key));if (local) return require('../public/routes').streamFileWithRange(req, res, local.key, headers);-- const vodStorage = require('../vod/vod-storage');for (const p of ['r2', 'b2']) {const l = locs.find(x => x.provider === p && x.state !== 'missing' && x.state !== 'corrupt');if (!l || !vodStorage.providerConfigured(p)) continue;nnew file mode 100644@@@ -0,0 +1,120 @@++/**++ * OpenVibe.Media — the tiering policy for native v2 objects as revisioned configuration (roadmap WS-G task 11;++ * docs/object-model.md#tiering-of-native-objects).++ *++ * The namespace `media.object_tier` of openvibe-shared/config, beside the VOD policy (`media.storage_tier`,++ * server/vod/tier-config.js): every change is a revision with who and why, validated as a whole before it++ * applies (types, ranges, and the order the thresholds need), with history and rollback through the admin++ * config routes (GET/POST /api/v1/:app/admin/storage/config/media.object_tier…).++ *++ * active the site-level activation gate. Off (the default): the sweep moves nothing and++ * only records what it would do (outcome dry_run)++ * promoteMinUniqueViewers7d unique viewers over the last 7 UTC days (server/objects/popularity.js) to promote++ * promoteRecentAccessDays …and viewed within this many days (today included)++ * promoteMinSizeMb / MaxSizeMb size bounds: small objects are cheap to serve locally; the upper bound caps++ * the bytes one promotion reads back from R2 to verify its sha256++ * maxPromotionsPerSweep moves (and would-be moves) per sweep++ * demoteIdleDays no viewer for this many days (and in R2 at least that long): demote++ * maxDemotionsPerSweep++ *++ * Revision 1 sets nothing (every key is its built-in default, so a release that changes a default is not++ * pinned to the old one); nothing is mirrored anywhere: no earlier release reads these keys.++ */++'use strict';++const config = require('openvibe-shared/config');++const db = require('../db/database');++++const DEFAULTS = {++ active: false,++ promoteMinUniqueViewers7d: 500,++ promoteRecentAccessDays: 3,++ promoteMinSizeMb: 16,++ promoteMaxSizeMb: 4096,++ maxPromotionsPerSweep: 3,++ demoteIdleDays: 14,++ maxDemotionsPerSweep: 10,++};++++// Daily counts are kept 30 days (popularity.KEEP_DAYS): a longer look-back would see nothing.++const MAX_LOOKBACK_DAYS = 30;++++const SCHEMA = {++ type: 'object',++ additionalProperties: false,++ properties: {++ active: { type: 'boolean' },++ promoteMinUniqueViewers7d: { type: 'integer', minimum: 1, maximum: 1e9 },++ promoteRecentAccessDays: { type: 'integer', minimum: 1, maximum: MAX_LOOKBACK_DAYS },++ promoteMinSizeMb: { type: 'number', minimum: 0, maximum: 1048576 },++ promoteMaxSizeMb: { type: 'number', minimum: 1, maximum: 1048576 },++ maxPromotionsPerSweep: { type: 'integer', minimum: 0, maximum: 100 },++ demoteIdleDays: { type: 'integer', minimum: 1, maximum: MAX_LOOKBACK_DAYS },++ maxDemotionsPerSweep: { type: 'integer', minimum: 0, maximum: 500 },++ },++};++++/** Rules across keys: a size window that is not empty, and no object both promoted and demoted (flapping). */++function validate(v) {++ const errors = [];++ if (v.promoteMinSizeMb >= v.promoteMaxSizeMb) errors.push('promoteMinSizeMb must be below promoteMaxSizeMb');++ if (v.demoteIdleDays <= v.promoteRecentAccessDays) errors.push('demoteIdleDays must be above promoteRecentAccessDays (an object viewed recently enough to promote would also be idle enough to demote)');++ return errors.length ? errors : true;++}++++let store = null;++++/** The store (created on first use, once the database is open). */++function get() {++ if (store) return store;++ store = config.createConfigStore({++ db: db.getDb(), service: 'media', namespace: 'media.object_tier',++ defaults: DEFAULTS, schema: SCHEMA, validate,++ // Revision 1 is empty: the defaults apply, and a key set later shows as a setting.++ legacy: () => ({}),++ log: { info: (m) => console.log(`[ObjectTiers] ${m}`), warn: (m) => console.warn(`[ObjectTiers] ${m}`), error: (m) => console.error(`[ObjectTiers] ${m}`) },++ });++ return store;++}++++/** Every key's effective value (the defaults under the active revision). */++function settings() {++ try { return { ...DEFAULTS, ...get().get() }; } catch (err) { console.warn('[ObjectTiers] config unavailable:', err.message); return { ...DEFAULTS }; }++}++++/** Change some keys: one validated revision merged over the active one → the new snapshot (throws ConfigError 422/409). */++function set(updates, { actor = null, reason = null } = {}) {++ return get().apply(updates, { merge: true, actor, reason });++}++++/** Keys the active revision sets explicitly (anything else is its default). */++function explicitKeys() {++ const s = get();++ if (s.revision() == null) return new Set();++ const snap = s.snapshot(s.revision());++ return new Set(Object.keys((snap && snap.values) || {}));++}++++/** Each key as { value, default, source: default | setting }. */++function thresholds(values = settings()) {++ let explicit = new Set();++ try { explicit = explicitKeys(); } catch { /* defaults only */ }++ const out = {};++ for (const k of Object.keys(DEFAULTS)) out[k] = { value: values[k], default: DEFAULTS[k], source: explicit.has(k) ? 'setting' : 'default' };++ return out;++}++++/** The rules the sweep applies, in words. */++function describe(v = settings()) {++ return {++ gate: v.active ? 'active: the sweep moves objects' : 'off (active = false): the sweep moves nothing and records what it would do as dry_run decisions',++ promote: `a ready native (v2) object of a non-sandbox tenant with at least promoteMinUniqueViewers7d (${v.promoteMinUniqueViewers7d}) unique viewers over the last 7 UTC days, `++ + `viewed within promoteRecentAccessDays (${v.promoteRecentAccessDays} days), between promoteMinSizeMb (${v.promoteMinSizeMb}) and promoteMaxSizeMb (${v.promoteMaxSizeMb}) MB, `++ + 'whose canonical copy is verified (present, checked, sha256 on record) and not under a hold; the canonical copy is hashed again, copied to R2, and the R2 copy '++ + `checked (size and sha256) before it becomes the playback source; at most maxPromotionsPerSweep (${v.maxPromotionsPerSweep}) per sweep`,++ demote: `an R2 copy of a native object that is no longer ready (deleted, failed, archived), or whose object has had no viewer for demoteIdleDays (${v.demoteIdleDays} days) `++ + `and has been in R2 at least that long; removed only after the canonical copy is confirmed good (hashed again), so the last good copy is never the one removed, `++ + `and never under a hold; at most maxDemotionsPerSweep (${v.maxDemotionsPerSweep}) per sweep`,++ };++}++++module.exports = { DEFAULTS, SCHEMA, validate, get, settings, set, explicitKeys, thresholds, describe, _reset: () => { store = null; } };nnew file mode 100644@@@ -0,0 +1,469 @@++/**++ * OpenVibe.Media — tiering of native v2 objects (roadmap WS-G task 11; docs/object-model.md#tiering-of-native-objects).++ *++ * A native object (uploaded through /api/v2: legacy_ref NULL) keeps its canonical copy where the upload put it++ * (local disk, OBJECTS_PATH; a B2 canonical copy is handled the same way). A popular one also gets a copy in++ * the Cloudflare R2 popularity cache, which becomes its playback source (GET /o/:id redirects there while R2++ * is available; the canonical copy is the fallback), and loses it when it goes idle or stops being ready.++ * Popularity is ./popularity.js (unique viewers per UTC day, no identifiers); the thresholds and the++ * site-level activation gate are the revisioned policy media.object_tier (./tier-policy.js).++ *++ * runSweep({ trigger }) one pass: rotate the popularity counts, then promote and demote by the policy.++ * Step 4 of the storage sweep (server/vod/vod-storage.js runSweep), which a restore++ * drill never starts (and this refuses under MEDIA_DRILL anyway)++ * promote(id, ctx) add a verified R2 copy: the canonical copy is hashed again (sha256 = content_hash),++ * uploaded (or copied from B2), and the R2 copy HEADed and read back (size and sha256)++ * before it is recorded present, which makes it the playback source++ * demote(id, ctx) remove the R2 copy, after the canonical copy is confirmed good (hashed again)++ * candidates({ appId }) what the policy would promote now, and what would refuse it (database only)++ * report({ appId }) the operator views' part (database only)++ *++ * The gate: while `active` is false nothing moves, here or through promote()/demote(); the sweep records what++ * it would do as dry_run decisions. A hold freezes placement: a held object is never promoted or demoted++ * (refused). Only an object whose canonical copy is verified on record (present, checked, sha256 on record)++ * is promoted, and a demotion never removes the last good copy: the canonical copy must check out first.++ * Every decision (done, already, refused, failed, dry_run) goes to media_object_tier_decisions with its++ * inputs and the policy in force, and counts in media_tier_decisions_24h{target="object"} on /metrics. A++ * refusal or dry run the sweep would repeat is logged once a day per object (a repeated dry run does not use++ * up the sweep's budget, so over a day the log shows what successive sweeps would move); an object whose move++ * failed waits FAILED_BACKOFF_H hours before the sweep tries it again.++ */++'use strict';++++const fs = require('fs');++const crypto = require('crypto');++const db = require('../db/database');++const drill = require('../drill');++const policy = require('./tier-policy');++const popularity = require('./popularity');++++const MB = 1024 * 1024;++const FAILED_BACKOFF_H = 6;++const OUTCOMES = ['done', 'already', 'refused', 'failed', 'dry_run'];++++const vodStorage = () => require('../vod/vod-storage');++const model = () => require('./model');++++const sweepState = { running: false, lastRunAt: null, lastResult: null };++++function sha256File(p) {++ return new Promise((resolve, reject) => {++ const h = crypto.createHash('sha256');++ const s = fs.createReadStream(p);++ s.on('data', (d) => h.update(d));++ s.on('end', () => resolve(h.digest('hex')));++ s.on('error', reject);++ });++}++++/** SQLite 'YYYY-MM-DD HH:MM:SS' (UTC) or ISO → epoch ms (NaN when unreadable). */++function msOf(t) {++ const s = String(t || '');++ return Date.parse(/[zZ]|[+-]\d\d:?\d\d$/.test(s) ? s : `${s.replace(' ', 'T')}Z`);++}++++// ── The canonical copy ───────────────────────────────────────++++function canonicalLocation(obj, locs) {++ return locs.find((l) => l.provider === (obj.canonical_provider || 'local')) || null;++}++++/** Is the canonical copy verified on record (no I/O)? → null, or why not. */++function canonicalUnverified(obj, locs) {++ const l = canonicalLocation(obj, locs);++ if (!l) return 'no canonical copy on record';++ if (l.provider === 'r2') return 'the canonical copy is the R2 cache';++ if (!obj.content_hash) return 'no sha256 on record to check a copy against (the object.hash job has not reached it)';++ if (l.state !== 'present' || !l.verified_at) return `the canonical ${l.provider} copy is ${l.state}${l.verified_at ? '' : ' and was never checked'}`;++ if (l.checksum && l.checksum !== obj.content_hash) return `the canonical ${l.provider} copy's checksum contradicts the object's sha256`;++ return null;++}++++/**++ * Check the canonical copy now: a local file must exist with the object's size and sha256; a B2 copy must++ * answer a HEAD with the object's size. → { ok, why, source: { file } | { b2key } }++ */++async function confirmCanonical(obj, loc) {++ if (!loc) return { ok: false, why: 'no canonical copy on record' };++ if (!obj.content_hash) return { ok: false, why: 'no sha256 on record to check the canonical copy against' };++ const size = Number(obj.size_bytes);++ if (loc.provider === 'local') {++ let st = null;++ try { st = fs.statSync(loc.key); } catch { /* missing */ }++ if (!st || !st.isFile()) return { ok: false, why: 'the canonical local file is missing' };++ if (st.size !== size) return { ok: false, why: `the canonical local file is ${st.size} bytes, not ${size}` };++ const sum = await sha256File(loc.key).catch(() => null);++ if (sum !== obj.content_hash) return { ok: false, why: sum ? 'the canonical local file does not match the sha256 on record' : 'the canonical local file could not be read' };++ return { ok: true, source: { file: loc.key } };++ }++ if (loc.provider === 'b2') {++ const head = await vodStorage().headObject('b2', loc.key).catch((err) => ({ error: err.message }));++ if (!head) return { ok: false, why: 'the canonical B2 copy is missing' };++ if (head.error) return { ok: false, why: `the canonical B2 copy could not be checked: ${head.error}` };++ if (head.size !== size) return { ok: false, why: `the canonical B2 copy is ${head.size} bytes, not ${size}` };++ return { ok: true, source: { b2key: loc.key } };++ }++ return { ok: false, why: `a canonical copy in ${loc.provider} cannot be checked here` };++}++++/** Where the R2 copy goes: the canonical key for a B2 canonical copy (copied key to key), else objects/<app>/<id>. */++function r2KeyFor(obj, canon) {++ return canon && canon.provider === 'b2' ? canon.key : `objects/${obj.app_id}/${obj.id}`;++}++++// ── Decisions ────────────────────────────────────────────────++++function thresholdSnapshot(settings) {++ const out = {};++ for (const [k, t] of Object.entries(policy.thresholds(settings))) out[k] = { value: t.value, source: t.source };++ return out;++}++++/** What a decision saw: the object's popularity, size, lifecycle, copies, hold and the gate. */++function inputsOf(obj, { locs, pop, held, settings }) {++ const canon = canonicalLocation(obj, locs);++ const r2 = locs.find((l) => l.provider === 'r2');++ return {++ unique_viewers_7d: pop ? pop.unique_viewers : 0,++ last_viewed_day: pop ? pop.last_viewed_day : popularity.lastViewedDay(obj.id),++ size_bytes: Number(obj.size_bytes) || 0,++ kind: obj.kind, visibility: obj.visibility, lifecycle_status: obj.lifecycle_status,++ content_hash: !!obj.content_hash,++ canonical: canon ? { provider: canon.provider, state: canon.state, verified_at: canon.verified_at || null,++ checksum_matches: !!(canon.checksum && obj.content_hash && canon.checksum === obj.content_hash) } : null,++ held,++ r2: r2 ? { state: r2.state, since: r2.created_at || null } : null,++ gate_active: !!settings.active,++ r2_available: vodStorage().providerAvailable('r2'),++ };++}++++/** Log one decision. Never throws: the move already happened (or did not). → the row id, or null */++function recordDecision({ obj, action, from, to, outcome, trigger, reason, inputs, settings, error = null }) {++ try {++ return db.run(`INSERT INTO media_object_tier_decisions (object_id, app_id, action, from_provider, to_provider, outcome, trigger, reason, inputs, thresholds, error)++ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,++ [obj.id, obj.app_id || null, action, from || null, to || null, outcome, String(trigger || 'manual').slice(0, 40), String(reason || '').slice(0, 1000),++ JSON.stringify(inputs || {}), JSON.stringify(thresholdSnapshot(settings)), error == null ? null : String(error).slice(0, 500)]).lastInsertRowid;++ } catch (err) {++ console.warn(`[ObjectTiers] decision for ${obj && obj.id} not logged: ${err.message}`);++ return null;++ }++}++++/** The sweep logged the same outcome for this object and action within the last day. */++function loggedToday(objectId, action, outcome) {++ return !!db.get(`SELECT 1 AS x FROM media_object_tier_decisions WHERE object_id = ? AND action = ? AND outcome = ? AND trigger = 'sweep'++ AND decided_at >= strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-1 day') LIMIT 1`, [objectId, action, outcome]);++}++++/** A move of this object failed within the back-off window. */++function inBackoff(objectId, action) {++ return !!db.get(`SELECT 1 AS x FROM media_object_tier_decisions WHERE object_id = ? AND action = ? AND outcome = 'failed'++ AND decided_at >= strftime('%Y-%m-%dT%H:%M:%fZ', 'now', ?) LIMIT 1`, [objectId, action, `-${FAILED_BACKOFF_H} hours`]);++}++++/**++ * Shared frame of promote()/demote(): loads the object and its facts, and returns log(outcome, reason, extra)++ * which records the decision (the sweep's repeated refusals and dry runs once a day) and shapes the answer.++ */++function frame(objectId, action, ctx) {++ const settings = ctx.settings || policy.settings();++ const obj = model().getObject(String(objectId || ''));++ if (!obj) return { missing: true };++ const locs = model().listLocations(obj.id);++ const pop = ctx.pop !== undefined ? ctx.pop : (popularity.stats({ ids: [obj.id], now: ctx.now }).get(obj.id) || null);++ const held = model().isHeld(obj.id);++ const inputs = inputsOf(obj, { locs, pop, held, settings });++ const canon = canonicalLocation(obj, locs);++ const trigger = ctx.trigger || 'manual';++ const why = ctx.reason || 'requested';++ const from = action === 'promote' ? (canon ? canon.provider : null) : 'r2';++ const to = action === 'promote' ? 'r2' : (canon ? canon.provider : null);++ const log = (outcome, reason, extra = {}) => {++ const repeat = trigger === 'sweep' && (outcome === 'refused' || outcome === 'dry_run') && loggedToday(obj.id, action, outcome);++ if (!repeat) recordDecision({ obj, action, from, to, outcome, trigger, reason, inputs, settings, error: extra.error });++ return { ok: outcome === 'done' || outcome === 'already', outcome, object_id: obj.id, repeat, ...extra };++ };++ return { obj, locs, canon, held, settings, why, log };++}++++/** Why an object may not be promoted at all, from the record (no I/O), or null. */++function promoteRefusal(obj, { locs, held }) {++ if (held) return 'under a retention hold (a hold freezes placement)';++ if (obj.legacy_ref) return 'not a native object (projected objects tier with their VOD)';++ if (obj.lifecycle_status !== 'ready') return `the object is ${obj.lifecycle_status}`;++ if (db.isSandboxTenant(obj.app_id)) return 'developer sandbox objects are never tiered';++ return canonicalUnverified(obj, locs);++}++++// ── Moves ────────────────────────────────────────────────────++++/**++ * Promote one native object to the R2 popularity cache and log the decision. ctx { trigger, reason, settings, pop }.++ * → { ok, outcome: done | already | refused | failed | dry_run, error?, bytes? }++ */++async function promote(objectId, ctx = {}) {++ const f = frame(objectId, 'promote', ctx);++ if (f.missing) return { ok: false, outcome: 'refused', error: 'object not found' };++ const { obj, locs, canon, held, settings, why, log } = f;++ if (held) return log('refused', `${why}; refused: under a retention hold (a hold freezes placement)`, { error: 'held', held: true });++ const r2Now = locs.find((l) => l.provider === 'r2');++ if (r2Now && r2Now.state === 'present' && r2Now.verified_at) return log('already', `${why}; the R2 copy is already present`);++ const refusal = promoteRefusal(obj, { locs, held });++ if (refusal) return log('refused', `${why}; refused: ${refusal}`, { error: refusal });++ const vs = vodStorage();++ if (!settings.active) return log('dry_run', `gate off (active = false): would promote: ${why}${vs.providerAvailable('r2') ? '' : ' (R2 is not available now)'}`);++ if (!vs.providerAvailable('r2')) return log('refused', `${why}; refused: R2 is not available`, { error: 'R2 not available' });++++ const check = await confirmCanonical(obj, canon);++ if (!check.ok) return log('refused', `${why}; refused: ${check.why}`, { error: check.why });++ const key = r2KeyFor(obj, canon);++ const size = Number(obj.size_bytes);++ try {++ if (check.source.file) await vs.uploadFile('r2', key, check.source.file, obj.mime_type || 'application/octet-stream');++ else await vs.copyBetweenProviders('b2', 'r2', key);++ const head = await vs.headObject('r2', key);++ if (!head || head.size !== size) throw new Error(`the R2 copy is ${head ? `${head.size} bytes` : 'missing'}, not ${size} bytes`);++ const read = await vs.sha256Object('r2', key);++ if (!read || read.sha256 !== obj.content_hash) throw new Error('the R2 copy read back does not match the sha256 on record');++ } catch (err) {++ await vs.deleteObject('r2', key).catch(() => {}); // never leave an unrecorded or unverified copy behind++ return log('failed', `${why}; the copy to R2 did not verify`, { error: err.message });++ }++++ // A hold placed (or a delete made) while the bytes were copied wins: the new copy was never recorded, so++ // removing it moves nothing anyone could see.++ const now = model().getObject(obj.id);++ if (!now || model().isHeld(obj.id) || now.lifecycle_status !== 'ready') {++ await vs.deleteObject('r2', key).catch(() => {});++ return log('refused', `${why}; refused: the object was ${!now ? 'removed' : now.lifecycle_status !== 'ready' ? now.lifecycle_status : 'put under a hold'} during the copy (the copy was removed)`,++ { error: 'changed during the copy' });++ }++ db.getDb().transaction(() => {++ // A fresh row: its created_at is when the object entered R2 (the demotion's "in R2 at least that long").++ db.run("DELETE FROM media_locations WHERE object_id = ? AND provider = 'r2'", [obj.id]);++ model().upsertLocation(obj.id, { provider: 'r2', bucket: vs.bucketFor('r2'), key, storage_class: 'cache', state: 'present',++ checksum: obj.content_hash, size_bytes: size, verified: true });++ model().setLocationState(canon.id, { state: 'present', size_bytes: size });++ })();++ console.log(`[ObjectTiers] ${obj.id} (${obj.app_id}) promoted to R2: ${key} (${(size / MB).toFixed(1)} MB)`);++ return log('done', why, { bytes: size });++}++++/**++ * Demote one native object from the R2 popularity cache (its R2 copy is deleted) and log the decision.++ * ctx as for promote(). → { ok, outcome, error? }++ */++async function demote(objectId, ctx = {}) {++ const f = frame(objectId, 'demote', ctx);++ if (f.missing) return { ok: false, outcome: 'refused', error: 'object not found' };++ const { obj, locs, canon, held, settings, why, log } = f;++ if (held) return log('refused', `${why}; refused: under a retention hold (a hold freezes placement)`, { error: 'held', held: true });++ const r2 = locs.find((l) => l.provider === 'r2');++ if (!r2) return log('already', `${why}; there is no R2 copy`);++ const vs = vodStorage();++ if (!settings.active) return log('dry_run', `gate off (active = false): would demote: ${why}`);++ if (!vs.providerConfigured('r2')) return log('refused', `${why}; refused: R2 is not configured`, { error: 'R2 not configured' });++ if (!canon || canon.provider === 'r2') return log('refused', `${why}; refused: no canonical copy besides R2 (the R2 copy is kept)`, { error: 'no canonical copy' });++++ // The last good copy is never the one removed: the canonical copy must check out now.++ const check = await confirmCanonical(obj, canon);++ if (!check.ok) return log('refused', `${why}; refused: ${check.why}; the R2 copy is kept (it may be the last good copy)`, { error: check.why });++ try {++ await vs.deleteObject('r2', r2.key);++ const still = await vs.headObject('r2', r2.key);++ if (still) throw new Error('the R2 copy is still there after the delete');++ } catch (err) {++ return log('failed', `${why}; the R2 copy could not be removed`, { error: err.message });++ }++ db.getDb().transaction(() => {++ db.run('DELETE FROM media_locations WHERE id = ?', [r2.id]);++ model().setLocationState(canon.id, { state: 'present', size_bytes: Number(obj.size_bytes) });++ })();++ console.log(`[ObjectTiers] ${obj.id} (${obj.app_id}) demoted from R2 (${why})`);++ return log('done', why);++}++++// ── Candidates ───────────────────────────────────────────────++++/** Ready native objects that pass the popularity, recency and size thresholds and have no R2 copy yet, most viewed first. */++function promotionCandidates(settings, { now = Date.now(), appId = null, limit = 200 } = {}) {++ const today = popularity.dayOf(now);++ return db.all(`SELECT o.*, p.viewers AS unique_viewers_7d, p.last_day AS last_viewed_day++ FROM (SELECT object_id, SUM(unique_viewers) AS viewers, MAX(day) AS last_day FROM media_object_views_daily++ WHERE day >= ? AND unique_viewers > 0 GROUP BY object_id) p++ JOIN media_objects o ON o.id = p.object_id++ WHERE o.legacy_ref IS NULL AND o.lifecycle_status = 'ready'++ AND p.viewers >= ? AND p.last_day >= ? AND o.size_bytes >= ? AND o.size_bytes <= ?++ AND NOT EXISTS (SELECT 1 FROM media_locations l WHERE l.object_id = o.id AND l.provider = 'r2' AND l.state = 'present')${appId ? ' AND o.app_id = ?' : ''}++ ORDER BY p.viewers DESC, o.id LIMIT ?`,++ [popularity.windowStart(today, popularity.WINDOW_DAYS), settings.promoteMinUniqueViewers7d, popularity.windowStart(today, settings.promoteRecentAccessDays),++ Math.ceil(settings.promoteMinSizeMb * MB), Math.floor(settings.promoteMaxSizeMb * MB), ...(appId ? [appId] : []), limit]);++}++++/** Native objects with an R2 copy that should lose it: not ready any more, or idle. → [{ obj row, reason }] */++function demotionCandidates(settings, { now = Date.now(), appId = null } = {}) {++ const idleFrom = popularity.windowStart(popularity.dayOf(now), settings.demoteIdleDays);++ const rows = db.all(`SELECT o.*, l.created_at AS r2_since,++ (SELECT MAX(d.day) FROM media_object_views_daily d WHERE d.object_id = o.id AND d.unique_viewers > 0) AS last_viewed_day++ FROM media_locations l JOIN media_objects o ON o.id = l.object_id++ WHERE l.provider = 'r2' AND o.legacy_ref IS NULL${appId ? ' AND o.app_id = ?' : ''}++ ORDER BY (o.lifecycle_status = 'ready'), o.id`, appId ? [appId] : []);++ const out = [];++ for (const r of rows) {++ if (r.lifecycle_status !== 'ready') { out.push({ row: r, reason: `the object is ${r.lifecycle_status}` }); continue; }++ const idle = !r.last_viewed_day || r.last_viewed_day < idleFrom;++ const since = msOf(r.r2_since);++ const settled = !Number.isFinite(since) || now - since >= settings.demoteIdleDays * 864e5;++ if (idle && settled) {++ out.push({ row: r, reason: `no viewer since ${r.last_viewed_day || 'the kept counts began'} (idle for demoteIdleDays ${settings.demoteIdleDays}); in R2 since ${r.r2_since || 'unknown'}` });++ }++ }++ return out;++}++++function promoteReason(c, s) {++ return `${c.unique_viewers_7d} unique viewers in 7 days >= promoteMinUniqueViewers7d ${s.promoteMinUniqueViewers7d}; last viewed ${c.last_viewed_day} `++ + `(within promoteRecentAccessDays ${s.promoteRecentAccessDays}); ${(Number(c.size_bytes) / MB).toFixed(1)} MB (promoteMinSizeMb ${s.promoteMinSizeMb} to promoteMaxSizeMb ${s.promoteMaxSizeMb})`;++}++++/** What the policy would promote now, each with what (on record) would refuse it; database only. */++function candidates({ appId = null, limit = 20, now = Date.now(), settings = policy.settings() } = {}) {++ return promotionCandidates(settings, { now, appId, limit: Math.min(Math.max(parseInt(limit, 10) || 20, 1), 200) }).map((c) => ({++ object_id: c.id, app_id: c.app_id, kind: c.kind, size_bytes: c.size_bytes, unique_viewers_7d: c.unique_viewers_7d, last_viewed_day: c.last_viewed_day,++ blocked_by: promoteRefusal(c, { locs: model().listLocations(c.id), held: model().isHeld(c.id) }),++ }));++}++++// ── The sweep ────────────────────────────────────────────────++++function tally(out, r) {++ if (r.repeat) out.repeats++;++ else if (r.outcome === 'done') out[r.action === 'demote' ? 'demoted' : 'promoted']++;++ else if (r.outcome === 'dry_run') out[r.action === 'demote' ? 'would_demote' : 'would_promote']++;++ else out[r.outcome] = (out[r.outcome] || 0) + 1;++ if (r.outcome === 'failed') out.errors.push({ object_id: r.object_id, action: r.action, error: r.error });++}++++/**++ * One pass over native objects. → { gate, promoted, demoted, would_promote, would_demote, already, refused, failed,++ * repeats, skipped_backoff, candidates: { promote, demote }, rotated, errors }++ */++async function runSweep({ trigger = 'sweep', now = Date.now() } = {}) {++ if (drill.enabled) return { skipped: true, reason: 'restore drill (MEDIA_DRILL)' };++ if (sweepState.running) return { skipped: true, reason: 'already running' };++ sweepState.running = true;++ try {++ const settings = policy.settings();++ const out = { gate: !!settings.active, promoted: 0, demoted: 0, would_promote: 0, would_demote: 0, already: 0, refused: 0, failed: 0,++ repeats: 0, skipped_backoff: 0, candidates: { promote: 0, demote: 0 }, rotated: popularity.rotate({ now }), errors: [] };++++ // Demotions first: they free the R2 cache, and a deleted object's copy should not wait behind promotions.++ const demote_ = demotionCandidates(settings, { now });++ out.candidates.demote = demote_.length;++ let budget = settings.maxDemotionsPerSweep;++ for (const { row, reason } of demote_) {++ if (budget <= 0) break;++ if (inBackoff(row.id, 'demote')) { out.skipped_backoff++; continue; }++ const r = await demote(row.id, { trigger, reason, settings, now });++ tally(out, { ...r, action: 'demote' });++ if (!r.repeat && ['done', 'failed', 'dry_run'].includes(r.outcome)) budget--;++ }++++ const promote_ = promotionCandidates(settings, { now, limit: Math.max(50, settings.maxPromotionsPerSweep * 10) })++ .filter((c) => !db.isSandboxTenant(c.app_id));++ out.candidates.promote = promote_.length;++ budget = settings.maxPromotionsPerSweep;++ for (const c of promote_) {++ if (budget <= 0) break;++ if (inBackoff(c.id, 'promote')) { out.skipped_backoff++; continue; }++ const r = await promote(c.id, { trigger, reason: promoteReason(c, settings), settings, now,++ pop: { unique_viewers: c.unique_viewers_7d, last_viewed_day: c.last_viewed_day } });++ tally(out, { ...r, action: 'promote' });++ if (!r.repeat && ['done', 'failed', 'dry_run'].includes(r.outcome)) budget--;++ }++++ if (out.promoted || out.demoted || out.failed) {++ console.log(`[ObjectTiers] Sweep: ${out.promoted} promoted to R2, ${out.demoted} demoted, ${out.failed} failed, ${out.refused} refused`);++ }++ if (!out.errors.length) delete out.errors;++ out.timestamp = new Date(now).toISOString();++ sweepState.lastRunAt = Date.now();++ sweepState.lastResult = out;++ return out;++ } catch (err) {++ console.error('[ObjectTiers] Sweep error:', err.message);++ return { error: err.message };++ } finally {++ sweepState.running = false;++ }++}++++// ── Operator views ───────────────────────────────────────────++++function decisionPublic(r) {++ const parse = (v) => { try { return JSON.parse(v || '{}'); } catch { return {}; } };++ return {++ id: r.id, decided_at: r.decided_at, object_id: r.object_id, app_id: r.app_id || null, action: r.action,++ from_provider: r.from_provider, to_provider: r.to_provider, outcome: r.outcome, trigger: r.trigger, reason: r.reason,++ inputs: parse(r.inputs), thresholds: parse(r.thresholds), error: r.error || null,++ };++}++++/** Decisions of the last 24 hours by action and outcome (appId narrows them). */++function counts24h(appId = null) {++ const empty = () => Object.fromEntries(OUTCOMES.map((o) => [o, 0]));++ const out = { promote: empty(), demote: empty() };++ for (const r of db.all(`SELECT action, outcome, COUNT(*) AS n FROM media_object_tier_decisions++ WHERE decided_at >= strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-1 day')${appId ? ' AND app_id = ?' : ''} GROUP BY action, outcome`, appId ? [appId] : [])) {++ if (out[r.action]) out[r.action][r.outcome] = r.n;++ }++ return out;++}++++/** A page of decisions, newest first: { decisions, next_before_id }. */++function listDecisions({ appId = null, objectId = null, action = null, outcome = null, beforeId = null, limit = 50 } = {}) {++ const conds = ['1 = 1'], params = [];++ if (appId) { conds.push('app_id = ?'); params.push(appId); }++ if (objectId) { conds.push('object_id = ?'); params.push(objectId); }++ if (action) { conds.push('action = ?'); params.push(action); }++ if (outcome) { conds.push('outcome = ?'); params.push(outcome); }++ if (beforeId != null) { conds.push('id < ?'); params.push(Number(beforeId) || 0); }++ const n = Math.min(Math.max(parseInt(limit, 10) || 50, 1), 200);++ const rows = db.all(`SELECT * FROM media_object_tier_decisions WHERE ${conds.join(' AND ')} ORDER BY id DESC LIMIT ?`, [...params, n + 1]);++ const page = rows.slice(0, n);++ return { decisions: page.map(decisionPublic), next_before_id: rows.length > n ? page[page.length - 1].id : null, limit: n };++}++++/** The operator views' part (server/me/ops.js): database only, no provider or file. */++function report({ appId = null, limit = 20 } = {}) {++ const settings = policy.settings();++ const s = appId ? { sql: ' AND o.app_id = ?', params: [appId] } : { sql: '', params: [] };++ const r2 = db.get(`SELECT COUNT(*) AS n, COALESCE(SUM(o.size_bytes), 0) AS b FROM media_locations l JOIN media_objects o ON o.id = l.object_id++ WHERE l.provider = 'r2' AND o.legacy_ref IS NULL AND l.state = 'present'${s.sql}`, s.params);++ const eligible = promotionCandidates(settings, { appId, limit: 100000 }).filter((c) => !db.isSandboxTenant(c.app_id)).length;++ const n = Math.min(Math.max(parseInt(limit, 10) || 20, 1), 50);++ const last = sweepState.lastResult;++ return {++ gate: { active: !!settings.active, source: policy.thresholds(settings).active.source },++ policy: Object.fromEntries(Object.keys(policy.DEFAULTS).filter((k) => k !== 'active').map((k) => [k, settings[k]])),++ r2_copies: { count: r2.n, bytes: r2.b },++ eligible_to_promote: eligible,++ decisions_24h: counts24h(appId),++ recent: listDecisions({ appId, limit: n }).decisions.map((d) => ({ id: d.id, decided_at: d.decided_at, app_id: d.app_id, object_id: d.object_id,++ action: d.action, outcome: d.outcome, trigger: d.trigger, reason: String(d.reason || '').slice(0, 300), error: d.error ? String(d.error).slice(0, 300) : null })),++ sweep: {++ last_run_at: sweepState.lastRunAt ? new Date(sweepState.lastRunAt).toISOString() : null,++ last_result: last ? { gate: last.gate, promoted: last.promoted, demoted: last.demoted, would_promote: last.would_promote, would_demote: last.would_demote,++ refused: last.refused, failed: last.failed, error: last.error || null } : null,++ },++ };++}++++module.exports = {++ FAILED_BACKOFF_H, OUTCOMES,++ runSweep, promote, demote, candidates, promotionCandidates, demotionCandidates,++ canonicalUnverified, confirmCanonical, r2KeyFor,++ counts24h, listDecisions, decisionPublic, report,++};@@@ -56,10 +56,13 @@ function storageDirs(config) {function domainMetrics(registry, { db, recorder, events }) {registry.gauge({ name: 'media_recordings_in_progress', help: 'Recordings ffmpeg is writing right now', collect: () => recorder.activeCount() });++ // target vod: media_tier_decisions (VODs); object: media_object_tier_decisions (native v2 objects), whose++ // outcome dry_run is what the object sweep would have done while its activation gate is off.registry.gauge({-- name: 'media_tier_decisions_24h', help: 'R2 promotions and demotions decided in the last 24 hours, by action and outcome (media_tier_decisions)', labelNames: ['action', 'outcome'],-- collect: () => db.all(`SELECT action, outcome, COUNT(*) AS n FROM media_tier_decisions WHERE decided_at >= strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-1 day')-- GROUP BY action, outcome`).map((r) => ({ labels: { action: r.action, outcome: r.outcome }, value: r.n })),++ name: 'media_tier_decisions_24h', help: 'R2 promotions and demotions decided in the last 24 hours, by target (vod, object), action and outcome', labelNames: ['target', 'action', 'outcome'],++ collect: () => [['vod', 'media_tier_decisions'], ['object', 'media_object_tier_decisions']].flatMap(([target, table]) => db.all(`SELECT action, outcome, COUNT(*) AS n++ FROM ${table} WHERE decided_at >= strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-1 day') GROUP BY action, outcome`)++ .map((r) => ({ labels: { target, action: r.action, outcome: r.outcome }, value: r.n }))),});registry.gauge({name: 'media_object_locations', help: 'Stored copies of media objects by provider and verification state', labelNames: ['provider', 'state'],@@@ -25,6 +25,7 @@* low-water mark regardless of the popularity thresholds* 2. promotes popular VODs to R2* 3. demotes stale R2 VODs back to B2-only++ * 4. tiers native v2 objects (server/objects/tiering.js: its own policy and activation gate)*/'use strict';@@@ -219,6 +220,11 @@ function clientFor(name) {return clients[name];}++/** Configured, and not known to be down (the last HeadBucket or readiness probe did not fail). */++function providerAvailable(name) {++ return providerConfigured(name) && providerHealthy[name] !== false;++}++function bucketFor(name) {return (PROVIDER_ENV[name] && PROVIDER_ENV[name].bucket) || null;}@@@ -345,6 +351,27 @@ async function copyBetweenProviders(srcProvider, dstProvider, key) {return { size: head.size };}++/**++ * Read one stored object through and hash it: { sha256, size }, or null when it is not there. Used to check a++ * copy byte for byte after it was written (R2 egress is free; the object tiering bounds the size it reads).++ */++async function sha256Object(provider, key) {++ const client = clientFor(provider);++ if (!client) throw new Error(`Provider ${provider} not configured`);++ loadSdk();++ let obj;++ try {++ obj = await client.send(new S3.GetObjectCommand({ Bucket: PROVIDER_ENV[provider].bucket, Key: key }));++ } catch (err) {++ if (err?.$metadata?.httpStatusCode === 404 || err?.name === 'NoSuchKey' || err?.name === 'NotFound') return null;++ throw err;++ }++ const hash = require('crypto').createHash('sha256');++ let size = 0;++ for await (const chunk of obj.Body) { hash.update(chunk); size += chunk.length; }++ return { sha256: hash.digest('hex'), size };++}++async function deleteObject(provider, key) {const client = clientFor(provider);if (!client) return;@@@ -417,18 +444,23 @@ async function listMultipartUploads(provider, prefix = '') {return out;}--async function presignGet(provider, key, expiresInSeconds = 900) {++/**++ * A presigned GET URL. `overrides` { contentType, contentDisposition } become the response headers the store++ * answers with (the object tiering passes what /o/:id would have sent, so an attachment stays one).++ */++async function presignGet(provider, key, expiresInSeconds = 900, overrides = {}) {const client = clientFor(provider);if (!client) return null;loadSdk();// Force the response MIME so the browser plays the media even when the stored// object metadata is generic (legacy clips were uploaded as octet-stream).const ext = path.extname(key || '').toLowerCase();-- const mime = { '.webm': 'video/webm', '.mp4': 'video/mp4', '.mkv': 'video/x-matroska', '.jpg': 'image/jpeg', '.jpeg': 'image/jpeg', '.png': 'image/png', '.webp': 'image/webp' }[ext];++ const mime = overrides.contentType || { '.webm': 'video/webm', '.mp4': 'video/mp4', '.mkv': 'video/x-matroska', '.jpg': 'image/jpeg', '.jpeg': 'image/jpeg', '.png': 'image/png', '.webp': 'image/webp' }[ext];return Presigner.getSignedUrl(client, new S3.GetObjectCommand({Bucket: PROVIDER_ENV[provider].bucket,Key: key,...(mime ? { ResponseContentType: mime } : {}),++ ...(overrides.contentDisposition ? { ResponseContentDisposition: overrides.contentDisposition } : {}),}), { expiresIn: expiresInSeconds });}@@@ -1067,6 +1099,11 @@ async function runSweep() {}}++ // 4) Native v2 objects (server/objects/tiering.js): their own policy (media.object_tier) and activation++ // gate, after the VODs; not while the disk still needs a drain (a promotion frees no local space).++ const objectsResult = stillNeedsDrain ? { skipped: true, reason: 'disk pressure: the drain goes first' }++ : await require('../objects/tiering').runSweep({ trigger: 'sweep' }).catch((err) => ({ error: err.message }));++const summary = {checked: candidates.length,migrated,@@@ -1082,6 +1119,7 @@ async function runSweep() {diskPct: diskAfter.usePct,freeGb: Number((diskAfter.available / GB).toFixed(1)),errors: errors.length ? errors : undefined,++ objects: objectsResult,timestamp: new Date().toISOString(),};sweepState.lastRunAt = Date.now();@@@ -1422,9 +1460,13 @@ module.exports = {providerOf,isRemote,providerConfigured,++ providerAvailable,bucketFor,endpointFor,headObject,++ uploadFile,++ copyBetweenProviders,++ sha256Object,listObjects,listMultipartUploads,keyForVod,nnew file mode 100644@@@ -0,0 +1,473 @@++'use strict';++// Tiering of native v2 objects (roadmap WS-G task 11): popularity counted per object and UTC day without an++// IP address or subject id (server/objects/popularity.js), rolled into a 7-day figure; the revisioned policy++// media.object_tier with its activation gate off by default (server/objects/tier-policy.js); and the sweep++// and moves (server/objects/tiering.js): dry runs while the gate is off, thresholds, holds, the verified++// canonical copy, promotion (copy, then size and sha256 of the R2 copy) and demotion (never the last good++// copy), every decision logged and counted, the admin API, the operator views and /o/:id playing from R2.++// A fake S3 (B2 and R2 buckets) is spoken to by the real AWS SDK through the storage engine.++const assert = require('assert');++const fs = require('fs');++const os = require('os');++const path = require('path');++const http = require('http');++const crypto = require('crypto');++++const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-objtier-'));++const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++const sha256 = (b) => crypto.createHash('sha256').update(b).digest('hex');++++const buckets = { 'b2-bucket': new Map(), 'r2-bucket': new Map() };++const fault = { refusePut: null, corruptPut: null, keepOnDelete: null };++function decodeAwsChunked(buf) {++ const out = [];++ let i = 0;++ for (;;) {++ const nl = buf.indexOf('\r\n', i);++ const size = parseInt(buf.subarray(i, nl).toString().split(';')[0], 16);++ if (!size) break;++ out.push(buf.subarray(nl + 2, nl + 2 + size));++ i = nl + 2 + size + 2;++ }++ return Buffer.concat(out);++}++const s3 = http.createServer((req, res) => {++ const chunks = [];++ req.on('data', (c) => chunks.push(c));++ req.on('end', () => {++ const u = new URL(req.url, 'http://x');++ const [, bucket, ...rest] = decodeURIComponent(u.pathname).split('/');++ const key = rest.join('/');++ const store = buckets[bucket];++ if (!store) { res.writeHead(404); return res.end(); }++ if (!key && req.method === 'HEAD') { res.writeHead(200); return res.end(); }++ const obj = store.get(key);++ if (req.method === 'HEAD' || req.method === 'GET') {++ if (!obj) { res.writeHead(404, { 'Content-Type': 'application/xml' }); return res.end(req.method === 'GET' ? '<Error><Code>NoSuchKey</Code></Error>' : undefined); }++ res.writeHead(200, { 'Content-Length': obj.length, ETag: '"e"', 'Content-Type': 'application/octet-stream' });++ return res.end(req.method === 'GET' ? obj : undefined);++ }++ if (req.method === 'PUT') {++ if (fault.refusePut === bucket) { res.writeHead(500, { 'Content-Type': 'application/xml' }); return res.end('<Error><Code>InternalError</Code></Error>'); }++ let body = Buffer.concat(chunks);++ if (req.headers['x-amz-decoded-content-length'] || /aws-chunked/.test(req.headers['content-encoding'] || '')) body = decodeAwsChunked(body);++ if (fault.corruptPut === bucket) { body = Buffer.from(body); body[0] ^= 0xff; } // same size, other bytes++ store.set(key, body);++ res.writeHead(200, { ETag: '"e"' });++ return res.end();++ }++ if (req.method === 'DELETE') { if (fault.keepOnDelete !== bucket) store.delete(key); res.writeHead(204); return res.end(); }++ res.writeHead(405); res.end();++ });++});++++const BROWSER = 'Mozilla/5.0 (X11; Linux x86_64; rv:120.0) Gecko/20100101 Firefox/120.0';++const SAFARI = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 14_0) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15';++++(async () => {++ await new Promise((r) => s3.listen(0, '127.0.0.1', r));++ const endpoint = `http://127.0.0.1:${s3.address().port}`;++ Object.assign(process.env, {++ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),++ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),++ MEDIA_B2_ENDPOINT: endpoint, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_KEY_ID: 'k', MEDIA_B2_APP_KEY: 's', MEDIA_B2_REGION: 'us-west-004',++ MEDIA_R2_ENDPOINT: endpoint, MEDIA_R2_BUCKET: 'r2-bucket', MEDIA_R2_ACCESS_KEY_ID: 'k', MEDIA_R2_SECRET_ACCESS_KEY: 's',++ AWS_REQUEST_CHECKSUM_CALCULATION: 'WHEN_REQUIRED', AWS_RESPONSE_CHECKSUM_VALIDATION: 'WHEN_REQUIRED',++ });++ const db = require('../server/db/database');++ const model = require('../server/objects/model');++ const storage = require('../server/vod/vod-storage');++ const popularity = require('../server/objects/popularity');++ const policy = require('../server/objects/tier-policy');++ const tiering = require('../server/objects/tiering');++ db.upsertApp({ app_id: 'live', api_key: 'live-key-objtier' });++ db.upsertApp({ app_id: 'games', api_key: 'games-key-objtier' });++ // Keep the VOD sweep's disk-pressure drain out of this test (whatever this machine's disk looks like).++ await storage.setSettings({ hotDiskPressurePct: 100, criticalDiskPct: 100, minFreeGb: 0 }, { reason: 'test: no pressure drain' });++++ const today = popularity.dayOf(Date.now());++ const addObject = ({ app = 'live', size = 4096, mime = 'application/octet-stream', verified = true, hash = true, canonical = 'local' } = {}) => {++ const bytes = crypto.randomBytes(size);++ const sum = sha256(bytes);++ const id = model.createObject({ app_id: app, kind: 'file', visibility: 'public', lifecycle_status: 'ready', mime_type: mime, size_bytes: size,++ content_hash: hash ? sum : null, canonical_provider: canonical, metadata: { filename: 'x.bin' } });++ let file = null;++ if (canonical === 'local') {++ file = path.join(process.env.OBJECTS_PATH, app, id);++ fs.mkdirSync(path.dirname(file), { recursive: true });++ fs.writeFileSync(file, bytes);++ model.updateObject(id, { canonical_key: file });++ model.upsertLocation(id, { provider: 'local', key: file, state: verified ? 'present' : 'pending', size_bytes: size, checksum: hash ? sum : null, verified });++ } else {++ const key = `objects/${app}/${id}`;++ buckets['b2-bucket'].set(key, bytes);++ model.updateObject(id, { canonical_key: key });++ model.upsertLocation(id, { provider: 'b2', bucket: 'b2-bucket', key, state: 'present', size_bytes: size, verified: true });++ }++ return { id, bytes, sum, file, app };++ };++ const setViews = (id, entries) => {++ db.run('DELETE FROM media_object_views_daily WHERE object_id = ?', [id]);++ for (const [offset, n] of entries) {++ db.run('INSERT INTO media_object_views_daily (object_id, day, unique_viewers, last_viewed_at) VALUES (?, ?, ?, ?)',++ [id, popularity.addDays(today, offset), n, `${popularity.addDays(today, offset)}T12:00:00.000Z`]);++ }++ };++ const r2Row = (id) => db.get("SELECT * FROM media_locations WHERE object_id = ? AND provider = 'r2'", [id]);++ const ageR2 = (id, days) => db.run("UPDATE media_locations SET created_at = datetime('now', ?) WHERE object_id = ? AND provider = 'r2'", [`-${days} days`, id]);++ const decisions = (id) => db.all('SELECT * FROM media_object_tier_decisions WHERE object_id = ? ORDER BY id', [id]);++ const last = (id) => { const d = decisions(id); return d[d.length - 1]; };++ const fake = (ip, headers = {}, method = 'GET') => ({ method, ip, path: '/o/x', url: '/o/x', headers: { 'user-agent': BROWSER, ...headers } });++++ // ── 1. Unique viewers per object and day, with no IP address or subject id stored ──++ const seen = addObject();++ const rec = (ip, headers, method) => popularity.record(model.getObject(seen.id), fake(ip, headers, method));++ assert.deepStrictEqual(rec('203.0.113.5'), { counted: true, reason: null });++ assert.strictEqual(rec('203.0.113.5', { 'user-agent': SAFARI }).reason, 'seen', 'one network is one viewer, whatever its user agent');++ popularity._reset(); // a restart: the same day's salt comes back from the database++ assert.strictEqual(rec('203.0.113.5').reason, 'seen', 'and still once after a restart');++ assert.strictEqual(rec('198.51.100.7').counted, true);++ assert.strictEqual(rec('::ffff:198.51.100.7').reason, 'seen', 'an IPv4-mapped address is the IPv4 address');++ assert.strictEqual(rec('2001:db8:1:2::10').counted, true);++ assert.strictEqual(rec('2001:db8:1:2:aaaa:bbbb:cccc:1').reason, 'seen', 'an IPv6 /64 is one viewer');++ assert.strictEqual(rec('2001:db8:1:3::10').counted, true);++ assert.strictEqual(rec('192.0.2.10', { range: 'bytes=5000-' }).reason, 'range', 'a seek is not a view');++ assert.strictEqual(rec('192.0.2.10', { range: 'bytes=0-' }).counted, true);++ assert.strictEqual(rec('192.0.2.11', {}, 'HEAD').reason, 'method');++ assert.strictEqual(rec('192.0.2.12', { 'sec-gpc': '1' }).reason, 'opted_out');++ assert.strictEqual(rec('192.0.2.13', { dnt: '1' }).reason, 'opted_out');++ assert.strictEqual(rec('192.0.2.14', { 'user-agent': 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' }).reason, 'bot');++ assert.strictEqual(rec('192.0.2.15', { 'user-agent': '' }).reason, 'bot');++ const projected = model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', legacy_ref: 'legacy:live:file:abc' });++ assert.strictEqual(popularity.record(model.getObject(projected), fake('192.0.2.16')).reason, 'not_native');++ const todayRow = db.get('SELECT * FROM media_object_views_daily WHERE object_id = ? AND day = ?', [seen.id, today]);++ assert.strictEqual(todayRow.unique_viewers, 5);++ assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_object_viewer_days WHERE object_id = ?', [seen.id]).n, 5, 'an opted-out, bot, HEAD or seek request makes no hash at all');++ const stored = JSON.stringify([db.all('SELECT * FROM media_object_viewer_days'), db.all('SELECT * FROM media_object_views_daily'), db.all('SELECT day FROM media_object_view_salts')]);++ for (const ip of ['203.0.113.5', '198.51.100.7', '2001:db8', '192.0.2.10', 'usr_']) assert.ok(!stored.includes(ip), `nothing stored names ${ip}`);++ for (const v of db.all('SELECT viewer FROM media_object_viewer_days').map((r) => r.viewer)) {++ assert.ok(/^[0-9a-f]{16}$/.test(v));++ assert.notStrictEqual(v, sha256('203.0.113.5').slice(0, 16), 'keyed by the salt of the day, not a plain hash of the address');++ }++ assert.deepStrictEqual(['10.1.2.3', '::ffff:10.1.2.3', '2001:DB8::1', '2001:db8:0:0:ffff::9', 'fe80::1%eth0'].map(popularity.networkOf),++ ['10.1.2.3', '10.1.2.3', '2001:db8:0:0::/64', '2001:db8:0:0::/64', 'fe80:0:0:0::/64']);++ console.log('✅ a viewer counts once per object and UTC day (one network, IPv6 by /64), never a seek, HEAD, bot or opted-out request, and no address is stored');++++ // ── 2. Hashes and salt die with their day; counts roll into a 7-day window, kept 30 days ──++ const roll = addObject();++ const yesterday = popularity.addDays(today, -1);++ assert.strictEqual(popularity.record(model.getObject(roll.id), fake('203.0.113.9'), { now: Date.parse(`${yesterday}T12:00:00Z`) }).counted, true);++ const oldHash = db.get('SELECT viewer FROM media_object_viewer_days WHERE object_id = ? AND day = ?', [roll.id, yesterday]).viewer;++ assert.ok(db.get('SELECT 1 AS x FROM media_object_view_salts WHERE day = ?', [yesterday]));++ assert.strictEqual(popularity.record(model.getObject(roll.id), fake('203.0.113.9')).counted, true, 'the same network on the next day is a new viewer-day');++ assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_object_viewer_days WHERE day < ?', [today]).n, 0, 'the first view of a day deletes the finished days\' hashes');++ assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_object_view_salts WHERE day < ?', [today]).n, 0, '...and their salts');++ assert.notStrictEqual(db.get('SELECT viewer FROM media_object_viewer_days WHERE object_id = ? AND day = ?', [roll.id, today]).viewer, oldHash, 'a new salt every day: days cannot be linked');++ assert.strictEqual(db.get('SELECT unique_viewers FROM media_object_views_daily WHERE object_id = ? AND day = ?', [roll.id, yesterday]).unique_viewers, 1, 'the count stays');++ const rs = popularity.stats({ ids: [roll.id] }).get(roll.id);++ assert.deepStrictEqual([rs.unique_viewers, rs.days_viewed, rs.last_viewed_day], [2, 2, today], 'two viewer-days');++ const win = addObject();++ setViews(win.id, [[-8, 100], [-6, 200], [0, 150]]);++ assert.strictEqual(popularity.stats({ ids: [win.id] }).get(win.id).unique_viewers, 350, 'the last 7 UTC days, today included');++ assert.strictEqual(popularity.stats({ ids: [win.id], days: 9 }).get(win.id).unique_viewers, 450);++ db.run('INSERT INTO media_object_views_daily (object_id, day, unique_viewers) VALUES (?, ?, 1), (?, ?, 1)', [win.id, popularity.addDays(today, -30), win.id, popularity.addDays(today, -31)]);++ const rot = popularity.rotate();++ assert.strictEqual(rot.counts, 1);++ assert.deepStrictEqual(db.all('SELECT day FROM media_object_views_daily WHERE object_id = ? ORDER BY day', [win.id]).map((r) => r.day),++ [popularity.addDays(today, -30), popularity.addDays(today, -8), popularity.addDays(today, -6), today], 'daily counts are kept 30 days');++ console.log('✅ a finished day\'s hashes and salt are deleted (counts stay); the 7-day window sums the daily counts; counts go after 30 days');++++ // ── 3. The policy: revisioned, validated, the gate off by default ──++ assert.deepStrictEqual(policy.DEFAULTS, { active: false, promoteMinUniqueViewers7d: 500, promoteRecentAccessDays: 3, promoteMinSizeMb: 16,++ promoteMaxSizeMb: 4096, maxPromotionsPerSweep: 3, demoteIdleDays: 14, maxDemotionsPerSweep: 10 });++ assert.strictEqual(policy.settings().active, false);++ const rev0 = policy.get().revision();++ for (const [updates, why] of [++ [{ promoteMinSizeMb: 10, promoteMaxSizeMb: 5 }, /promoteMinSizeMb must be below promoteMaxSizeMb/],++ [{ demoteIdleDays: 3 }, /demoteIdleDays must be above promoteRecentAccessDays/],++ [{ promoteMinUniqueViewers7d: 0 }, /must be >= 1/],++ [{ active: 'yes' }, /must be boolean/],++ [{ maxPromotionsPerSweep: 1.5 }, /must be integer/],++ [{ demoteIdleDays: 31 }, /must be <= 30/],++ [{ surprise: 1 }, /not a known key/],++ ]) {++ await assert.rejects(policy.set(updates, { reason: 'test' }), (e) => e.status === 422 && e.code === 'config.invalid' && why.test(e.message), JSON.stringify(updates));++ }++ assert.strictEqual(policy.get().revision(), rev0, 'a refused change changes nothing');++ await policy.set({ promoteMinSizeMb: 0.001, promoteMaxSizeMb: 1 }, { actor: { type: 'service', id: 'live' }, reason: 'test: small objects' });++ const th = policy.thresholds();++ assert.deepStrictEqual([th.promoteMinSizeMb.source, th.promoteMinUniqueViewers7d.source, th.active.source], ['setting', 'default', 'default']);++ assert.deepStrictEqual(th.promoteMinUniqueViewers7d, { value: 500, default: 500, source: 'default' });++ console.log('✅ media.object_tier: gate off and 500 viewers by default; types, ranges, key order and unknown keys refused (422), nothing changed');++++ // ── 4. Gate off: the sweep moves nothing and records what it would do ──++ db.run('DELETE FROM media_object_views_daily');++ const pop = addObject();++ setViews(pop.id, [[0, 600]]);++ const cached = addObject(); // a copy in R2 from earlier, idle for a month++ buckets['r2-bucket'].set(`objects/live/${cached.id}`, cached.bytes);++ model.upsertLocation(cached.id, { provider: 'r2', bucket: 'r2-bucket', key: `objects/live/${cached.id}`, storage_class: 'cache', state: 'present', size_bytes: 4096, checksum: cached.sum, verified: true });++ ageR2(cached.id, 30);++ const r2Keys = buckets['r2-bucket'].size;++ let s = await tiering.runSweep();++ assert.deepStrictEqual([s.gate, s.promoted, s.demoted, s.would_promote, s.would_demote], [false, 0, 0, 1, 1], JSON.stringify(s));++ assert.strictEqual(buckets['r2-bucket'].size, r2Keys, 'nothing was copied or deleted');++ assert.ok(!r2Row(pop.id) && r2Row(cached.id));++ let d = last(pop.id);++ assert.deepStrictEqual([d.action, d.outcome, d.trigger, d.from_provider, d.to_provider], ['promote', 'dry_run', 'sweep', 'local', 'r2']);++ assert.ok(/^gate off \(active = false\): would promote: 600 unique viewers in 7 days >= promoteMinUniqueViewers7d 500/.test(d.reason), d.reason);++ const din = JSON.parse(d.inputs);++ assert.deepStrictEqual([din.unique_viewers_7d, din.last_viewed_day, din.gate_active, din.held, din.canonical.provider, din.canonical.checksum_matches], [600, today, false, false, 'local', true]);++ assert.deepStrictEqual(JSON.parse(d.thresholds).active, { value: false, source: 'default' });++ assert.deepStrictEqual([last(cached.id).action, last(cached.id).outcome], ['demote', 'dry_run']);++ s = await tiering.runSweep();++ assert.deepStrictEqual([s.would_promote, s.would_demote, s.repeats], [0, 0, 2], 'a repeated dry run is logged once a day');++ assert.strictEqual(decisions(pop.id).length, 1);++ const manual = await tiering.promote(pop.id, { reason: 'asked by hand' });++ assert.deepStrictEqual([manual.ok, manual.outcome], [false, 'dry_run'], 'the gate holds for a direct call too');++ assert.ok(!r2Row(pop.id));++ console.log('✅ gate off: dry_run decisions (with inputs and thresholds), once a day, and nothing moves, from the sweep or a direct call');++++ // ── 5. Thresholds: viewers, recency, window and size ──++ const few = addObject(); setViews(few.id, [[0, 499]]);++ const stale = addObject(); setViews(stale.id, [[-5, 900]]); // popular, but not viewed within 3 days++ const outside = addObject(); setViews(outside.id, [[-7, 450], [0, 100]]); // day -7 is outside the 7-day window++ const spread = addObject(); setViews(spread.id, [[-6, 250], [-3, 150], [0, 100]]);++ const tiny = addObject({ size: 512 }); setViews(tiny.id, [[0, 900]]); // under promoteMinSizeMb (0.001 MB)++ const huge = addObject({ size: 1100000 }); setViews(huge.id, [[0, 900]]); // over promoteMaxSizeMb (1 MB)++ const projectId = 'prj_01J8Z3Q4X5Y6Z7A8B9C0D1E2F3';++ db.ensureProjectTenant(projectId, 'sandbox', 0);++ const sandboxed = addObject({ app: `${projectId}-sandbox` }); setViews(sandboxed.id, [[0, 900]]);++ const ids = tiering.promotionCandidates(policy.settings()).map((c) => c.id);++ assert.ok(ids.includes(pop.id) && ids.includes(spread.id), 'at 500 or more within the window');++ for (const o of [few, stale, outside, tiny, huge]) assert.ok(!ids.includes(o.id), `not a candidate: ${o.id}`);++ s = await tiering.runSweep();++ assert.ok(!decisions(sandboxed.id).length, 'a sandbox object is never considered');++ assert.strictEqual(tiering.candidates().find((c) => c.object_id === sandboxed.id).blocked_by, 'developer sandbox objects are never tiered');++ assert.strictEqual(last(spread.id).outcome, 'dry_run');++ for (const o of [few, stale, outside, tiny, huge]) assert.strictEqual(decisions(o.id).length, 0);++ console.log('✅ thresholds: 7-day viewers (sum of the window), recent view, size bounds; sandbox tenants never tier');++++ // ── 6. Gate on: holds and an unverified canonical copy refuse; a released hold promotes ──++ await policy.set({ active: true }, { actor: { type: 'service', id: 'live' }, reason: 'test: gate on' });++ db.run('DELETE FROM media_object_views_daily');++ const hot = addObject(); setViews(hot.id, [[0, 700]]);++ const hold = model.placeHold({ object_id: hot.id, kind: 'admin', reason: 'test' });++ const unchecked = addObject({ verified: false }); setViews(unchecked.id, [[0, 700]]);++ const nohash = addObject({ hash: false }); setViews(nohash.id, [[0, 700]]);++ const tampered = addObject(); setViews(tampered.id, [[0, 700]]);++ const other = Buffer.from(tampered.bytes); other[10] ^= 0xff; fs.writeFileSync(tampered.file, other); // on record verified; on disk not++ s = await tiering.runSweep();++ assert.deepStrictEqual([s.gate, s.promoted, s.refused], [true, 0, 4], JSON.stringify(s));++ assert.ok(/retention hold/.test(last(hot.id).reason) && JSON.parse(last(hot.id).inputs).held === true);++ assert.ok(/never checked/.test(last(unchecked.id).reason), last(unchecked.id).reason);++ assert.ok(/no sha256 on record/.test(last(nohash.id).reason), last(nohash.id).reason);++ assert.ok(/does not match the sha256 on record/.test(last(tampered.id).reason), last(tampered.id).reason);++ for (const o of [hot, unchecked, nohash, tampered]) { assert.ok(!r2Row(o.id)); assert.ok(!buckets['r2-bucket'].has(`objects/live/${o.id}`)); }++ s = await tiering.runSweep();++ assert.strictEqual(decisions(hot.id).length, 1, 'a repeated refusal is logged once a day');++ model.releaseHold(hold.id, 'test');++ s = await tiering.runSweep();++ assert.strictEqual(s.promoted, 1);++ console.log('✅ gate on: a hold, a canonical copy never checked, no sha256 or bytes that no longer match refuse the promotion; released, it promotes');++++ // ── 7. Promotion: copy, verify size and sha256, record; /o/:id plays from R2 ──++ let r = r2Row(hot.id);++ assert.deepStrictEqual([r.state, r.storage_class, r.key, r.checksum, r.size_bytes, !!r.verified_at], ['present', 'cache', `objects/live/${hot.id}`, hot.sum, 4096, true]);++ assert.ok(buckets['r2-bucket'].get(`objects/live/${hot.id}`).equals(hot.bytes), 'the R2 copy is the same bytes');++ assert.ok(fs.existsSync(hot.file) && db.get("SELECT state FROM media_locations WHERE object_id = ? AND provider = 'local'", [hot.id]).state === 'present', 'the canonical copy stays');++ d = last(hot.id);++ assert.deepStrictEqual([d.action, d.outcome, d.trigger, d.from_provider, d.to_provider, d.error], ['promote', 'done', 'sweep', 'local', 'r2', null]);++++ const express = require('express');++ const app = express();++ app.set('trust proxy', 'loopback');++ app.use(express.json());++ app.use('/api/v1/:app/admin/storage', require('../server/admin/routes'));++ app.use('/o', require('../server/objects/routes').publicRouter);++ const server = await new Promise((ok) => { const x = app.listen(0, '127.0.0.1', () => ok(x)); });++ const base = `http://127.0.0.1:${server.address().port}`;++ const get = (url, headers = {}) => new Promise((resolve, reject) => {++ http.get(url, { headers: { 'User-Agent': BROWSER, ...headers } }, (res) => {++ const b = []; res.on('data', (c) => b.push(c)); res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body: Buffer.concat(b) }));++ }).on('error', reject);++ });++ const call = async (p, key = 'live-key-objtier') => { const x = await get(base + p, { Authorization: `Bearer ${key}` }); return { status: x.status, body: JSON.parse(x.body.toString() || '{}') }; };++++ const web = addObject();++ let g = await get(`${base}/o/${web.id}`, { 'X-Forwarded-For': '192.0.2.44' });++ assert.strictEqual(g.status, 200);++ assert.ok(g.body.equals(web.bytes), 'served from the local canonical copy before');++ await get(`${base}/o/${web.id}`, { 'X-Forwarded-For': '192.0.2.45', 'Sec-GPC': '1' });++ assert.strictEqual(popularity.stats({ ids: [web.id] }).get(web.id).unique_viewers, 1, '/o/:id counts its viewers (not the opted-out one)');++ assert.strictEqual((await tiering.promote(web.id, { trigger: 'manual', reason: 'test' })).outcome, 'done');++ g = await get(`${base}/o/${web.id}`, { 'X-Forwarded-For': '192.0.2.46' });++ assert.strictEqual(g.status, 302);++ const loc = new URL(g.headers.location);++ assert.ok(loc.pathname.endsWith(`/r2-bucket/objects/live/${web.id}`), g.headers.location);++ assert.ok(/^attachment; filename="x\.bin"$/.test(loc.searchParams.get('response-content-disposition')), 'the presigned answer keeps the headers /o would send');++ assert.strictEqual(loc.searchParams.get('response-content-type'), 'application/octet-stream');++ assert.ok((await get(g.headers.location)).body.equals(web.bytes), 'and R2 serves the same bytes');++ assert.strictEqual(popularity.stats({ ids: [web.id] }).get(web.id).unique_viewers, 2);++++ fault.corruptPut = 'r2-bucket';++ const bad = addObject();++ let m = await tiering.promote(bad.id, { reason: 'test' });++ fault.corruptPut = null;++ assert.deepStrictEqual([m.ok, m.outcome], [false, 'failed']);++ assert.ok(/does not match the sha256/.test(m.error), m.error);++ assert.ok(!buckets['r2-bucket'].has(`objects/live/${bad.id}`) && !r2Row(bad.id), 'an R2 copy that does not verify is removed and never recorded');++ assert.ok(last(bad.id).error);++ fault.refusePut = 'r2-bucket';++ const refusedPut = addObject();++ m = await tiering.promote(refusedPut.id, { reason: 'test' });++ fault.refusePut = null;++ assert.deepStrictEqual([m.outcome, !!r2Row(refusedPut.id)], ['failed', false]);++ db.run('DELETE FROM media_object_views_daily WHERE object_id != ?', [bad.id]);++ setViews(bad.id, [[0, 700]]);++ s = await tiering.runSweep();++ assert.deepStrictEqual([s.skipped_backoff, s.promoted, decisions(bad.id).length], [1, 0, 1], 'a failed object waits before the next try');++++ const fromB2 = addObject({ canonical: 'b2' });++ m = await tiering.promote(fromB2.id, { reason: 'test' });++ assert.strictEqual(m.outcome, 'done', JSON.stringify(m));++ assert.ok(buckets['r2-bucket'].get(`objects/live/${fromB2.id}`).equals(fromB2.bytes), 'a B2 canonical copy is copied to R2 under its key');++ assert.strictEqual(last(fromB2.id).from_provider, 'b2');++ assert.strictEqual((await tiering.promote(fromB2.id, { reason: 'again' })).outcome, 'already');++ console.log('✅ promotion copies the canonical copy, checks the R2 copy\'s size and sha256 before recording it; /o/:id then redirects to R2 with its own headers; failures leave nothing and back off');++++ // ── 8. Demotion: idle or not ready; never the last good copy; holds; purge waits for it ──++ db.run('DELETE FROM media_object_views_daily');++ setViews(hot.id, [[-20, 900]]); ageR2(hot.id, 20); // idle 20 days, in R2 20 days++ const fresh = addObject(); await tiering.promote(fresh.id, { reason: 'test' }); // no viewer, but in R2 since now++ setViews(web.id, [[0, 3]]); ageR2(web.id, 20); // viewed today++ const lost = addObject(); await tiering.promote(lost.id, { reason: 'test' }); ageR2(lost.id, 20);++ fs.unlinkSync(lost.file); // the canonical copy is gone: R2 is the last good copy++ const drifted = addObject(); await tiering.promote(drifted.id, { reason: 'test' }); ageR2(drifted.id, 20);++ const flipped = Buffer.from(drifted.bytes); flipped[0] ^= 0xff; fs.writeFileSync(drifted.file, flipped);++ const pinned = addObject(); await tiering.promote(pinned.id, { reason: 'test' }); ageR2(pinned.id, 20);++ const pin = model.placeHold({ object_id: pinned.id, kind: 'creator_pin', reason: 'test' });++ const gone = addObject(); await tiering.promote(gone.id, { reason: 'test' }); setViews(gone.id, [[0, 900]]);++ model.softDelete(model.getObject(gone.id), { by: 'test' });++ const sticky = addObject(); await tiering.promote(sticky.id, { reason: 'test' }); ageR2(sticky.id, 20);++ fault.keepOnDelete = 'r2-bucket';++ // Only the sticky object sees the store keep its bytes: demote it on its own first.++ m = await tiering.demote(sticky.id, { reason: 'test' });++ fault.keepOnDelete = null;++ assert.deepStrictEqual([m.outcome, !!r2Row(sticky.id)], ['failed', true]);++ assert.ok(/still there/.test(m.error));++ // Seven hours ago: past the back-off, so the sweep below tries again (and still within the 24-hour counts).++ db.run("UPDATE media_object_tier_decisions SET decided_at = strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-7 hours') WHERE object_id = ? AND outcome = 'failed'", [sticky.id]);++ await policy.set({ maxDemotionsPerSweep: 20 }, { reason: 'test: room for every demotion' });++ s = await tiering.runSweep();++ assert.ok(!r2Row(hot.id) && !buckets['r2-bucket'].has(`objects/live/${hot.id}`), 'an idle object leaves R2');++ assert.ok(fs.existsSync(hot.file) && db.get("SELECT state FROM media_locations WHERE object_id = ? AND provider = 'local'", [hot.id]).state === 'present');++ d = last(hot.id);++ assert.deepStrictEqual([d.action, d.outcome, d.from_provider, d.to_provider], ['demote', 'done', 'r2', 'local']);++ assert.ok(/idle for demoteIdleDays 14/.test(d.reason), d.reason);++ assert.ok(r2Row(fresh.id) && r2Row(web.id), 'an object in R2 for less than demoteIdleDays, or viewed, stays');++ assert.ok(r2Row(lost.id) && buckets['r2-bucket'].has(`objects/live/${lost.id}`), 'the last good copy is never removed');++ assert.deepStrictEqual([last(lost.id).outcome, /the canonical local file is missing; the R2 copy is kept/.test(last(lost.id).reason)], ['refused', true]);++ assert.ok(r2Row(drifted.id) && /does not match the sha256/.test(last(drifted.id).reason), 'nor one whose canonical copy no longer hashes right');++ assert.ok(r2Row(pinned.id) && /retention hold/.test(last(pinned.id).reason), 'a hold blocks the demotion');++ assert.ok(!r2Row(gone.id) && /the object is deleted/.test(last(gone.id).reason), 'a deleted object leaves R2 however popular');++ assert.ok(!r2Row(cached.id), 'the month-idle copy from before goes too');++ assert.ok(!r2Row(sticky.id), 'and the one the store kept once, now that it lets go');++ g = await get(`${base}/o/${lost.id}`);++ assert.strictEqual(g.status, 302, 'the object whose canonical copy is lost still plays from R2');++ g = await get(`${base}/o/${hot.id}`);++ assert.strictEqual(g.status, 200, 'a demoted object plays from its canonical copy again');++ model.releaseHold(pin.id, 'test');++++ const purged = addObject(); await tiering.promote(purged.id, { reason: 'test' });++ model.softDelete(model.getObject(purged.id), { by: 'test' });++ db.run("UPDATE media_objects SET deleted_at = datetime('now', '-40 days') WHERE id = ?", [purged.id]);++ model.purgeExpired();++ assert.ok(fs.existsSync(purged.file) && r2Row(purged.id), 'the purge waits while the object still has an R2 copy');++ await tiering.demote(purged.id, { trigger: 'sweep', reason: 'the object is deleted' });++ model.purgeExpired();++ assert.ok(!fs.existsSync(purged.file) && !r2Row(purged.id), 'and purges once the R2 copy is gone');++ console.log('✅ demotion: idle (and settled) or not ready; refused under a hold or when the canonical copy does not check out; a failed delete keeps the row; the purge waits for it');++++ // ── 9. The decision log: admin API, metric, operator views ──++ const gamesObj = addObject({ app: 'games' });++ await tiering.promote(gamesObj.id, { reason: 'games' });++ const cand = addObject(); setViews(cand.id, [[0, 800]]);++ let a = await call('/api/v1/live/admin/storage/tiers/objects/policy');++ assert.strictEqual(a.status, 200, JSON.stringify(a.body));++ assert.deepStrictEqual([a.body.gate.active, a.body.gate.source], [true, 'setting']);++ assert.deepStrictEqual(a.body.thresholds.promoteMinUniqueViewers7d, { value: 500, default: 500, source: 'default' });++ assert.ok(/promoteMinUniqueViewers7d \(500\)/.test(a.body.rules.promote) && /last good copy/.test(a.body.rules.demote));++ assert.deepStrictEqual(a.body.candidates.map((c) => [c.object_id, c.unique_viewers_7d, c.blocked_by]), [[cand.id, 800, null]]);++ assert.strictEqual(a.body.provider.r2.configured, true);++ const c24 = a.body.decisions.last_24h;++ assert.ok(c24.promote.dry_run >= 3 && c24.promote.done >= 10 && c24.promote.refused >= 4 && c24.promote.failed >= 2 && c24.promote.already === 1, JSON.stringify(c24));++ assert.ok(c24.demote.done >= 4 && c24.demote.refused >= 3 && c24.demote.failed === 1 && c24.demote.dry_run === 1, JSON.stringify(c24));++ assert.ok(a.body.decisions.recent.every((x) => x.app_id === 'live') && a.body.decisions.recent[0].inputs && a.body.decisions.recent[0].thresholds);++ a = await call('/api/v1/live/admin/storage/tiers/objects/decisions?outcome=dry_run&limit=2');++ assert.deepStrictEqual([a.body.decisions.length, a.body.decisions.every((x) => x.outcome === 'dry_run'), a.body.next_before_id != null], [2, true, true]);++ a = await call(`/api/v1/live/admin/storage/tiers/objects/decisions?object_id=${bad.id}`);++ assert.deepStrictEqual(a.body.decisions.map((x) => x.outcome), ['failed']);++ assert.strictEqual((await call('/api/v1/live/admin/storage/tiers/objects/decisions?action=move')).status, 400);++ assert.strictEqual((await call('/api/v1/live/admin/storage/tiers/objects/decisions?outcome=nope')).status, 400);++ a = await call('/api/v1/games/admin/storage/tiers/objects/decisions', 'games-key-objtier');++ assert.deepStrictEqual(a.body.decisions.map((x) => x.object_id), [gamesObj.id], 'each app sees its own');++ assert.ok((await call('/api/v1/live/admin/storage/tiers/objects/policy', 'wrong-key')).status >= 401);++ a = await call('/api/v1/live/admin/storage/config');++ const ns = a.body.namespaces.find((n) => n.namespace === 'media.object_tier');++ assert.ok(ns && ns.values.active === true && ns.values.promoteMinUniqueViewers7d === 500, 'the policy is in the config routes');++ a = await call('/api/v1/live/admin/storage/config/media.object_tier/history');++ assert.deepStrictEqual(a.body.snapshots.map((x) => x.state), ['active', 'superseded', 'superseded', 'superseded']);++ const post = (p, body) => new Promise((resolve, reject) => {++ const data = JSON.stringify(body);++ const rq = http.request(base + p, { method: 'POST', headers: { Authorization: 'Bearer live-key-objtier', 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(data) } },++ (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: JSON.parse(b || '{}') })); });++ rq.on('error', reject);++ rq.end(data);++ });++ a = await post('/api/v1/live/admin/storage/config/media.object_tier', { values: { active: false }, merge: true, reason: 'gate off through the route' });++ assert.strictEqual(a.status, 200, JSON.stringify(a.body));++ assert.strictEqual(policy.settings().active, false);++ a = await post('/api/v1/live/admin/storage/config/media.object_tier', { values: { active: true }, merge: true, reason: 'and on again' });++ assert.deepStrictEqual([a.status, policy.settings().active, policy.thresholds().active.source], [200, true, 'setting']);++ a = await post('/api/v1/live/admin/storage/config/media.object_tier', { values: { demoteIdleDays: 2 }, merge: true, reason: 'flapping' });++ assert.deepStrictEqual([a.status, policy.settings().demoteIdleDays], [422, 14], 'the route refuses what the rules refuse');++++ db.run("INSERT INTO media_tier_decisions (vod_id, app_id, action, outcome, trigger, reason) VALUES (1, 'live', 'promote', 'done', 'admin', 'test')");++ const registry = require('openvibe-shared/metrics').createRegistry();++ require('../server/observability').domainMetrics(registry, { db, recorder: { activeCount: () => 0 }, events: { status: () => ({ enabled: false }) } });++ const text = registry.metrics();++ const all24 = tiering.counts24h();++ assert.ok(text.includes(`media_tier_decisions_24h{target="object",action="promote",outcome="done"} ${all24.promote.done}`), text);++ assert.ok(text.includes(`media_tier_decisions_24h{target="object",action="promote",outcome="dry_run"} ${all24.promote.dry_run}`));++ assert.ok(text.includes('media_tier_decisions_24h{target="vod",action="promote",outcome="done"} 1'));++++ const report = require('../server/me/ops').report({ appId: 'live' });++ const no = report.tiering.native_objects;++ const liveR2 = db.get("SELECT COUNT(*) AS n FROM media_locations l JOIN media_objects o ON o.id = l.object_id WHERE l.provider = 'r2' AND l.state = 'present' AND o.app_id = 'live'").n;++ assert.deepStrictEqual([no.gate.active, no.r2_copies.count, no.eligible_to_promote, no.decisions_24h.promote.done], [true, liveR2, 1, c24.promote.done]);++ assert.ok(no.recent.length && no.sweep.last_result && no.sweep.last_run_at);++ assert.ok(!/until roadmap WS-G task 11/.test(report.tiering.note));++ const html = require('../server/me/pages').renderOps({ person: { subject: 'usr_01J8Z3Q4X5Y6Z7A8B9C0D1E2F3' }, report, canRecompute: false });++ assert.ok(/Native objects/.test(html) && /Activation gate/.test(html) && html.includes(bad.id), 'the /me/ops page shows the gate and the recent decisions');++ console.log('✅ decisions: /tiers/objects/policy and /decisions (filters, pages, app-scoped), the config routes, media_tier_decisions_24h{target}, and the operator views');++++ // ── 10. The storage sweep runs it (step 4); a restore drill never does ──++ const vodSweep = await storage.runSweep();++ assert.ok(vodSweep.objects && vodSweep.objects.gate === true && typeof vodSweep.objects.promoted === 'number', JSON.stringify(vodSweep));++ assert.strictEqual(r2Row(cand.id) && r2Row(cand.id).state, 'present', 'the storage sweep promoted the candidate');++ // The storage engine's upload is exported (the scheduled verification's default re-upload calls it too).++ assert.deepStrictEqual(['uploadFile', 'copyBetweenProviders', 'sha256Object', 'providerAvailable'].map((f) => typeof storage[f]), ['function', 'function', 'function', 'function']);++ const drill = require('../server/drill');++ drill.enabled = true;++ assert.deepStrictEqual(await tiering.runSweep(), { skipped: true, reason: 'restore drill (MEDIA_DRILL)' });++ drill.enabled = false;++ console.log('✅ the storage sweep runs the object tiering as its step 4; under MEDIA_DRILL it does nothing');++++ server.close();++ s3.close();++ db.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('\n✅ All object tiering tests passed');++ process.exit(0);++})().catch((err) => { console.error(err); process.exit(1); });
@@ -46,7 +46,8 @@ server/
@@ -46,7 +46,8 @@ server/
objects/ canonical object model: model (projections, holds), routes (/api/v2 + /o),
objects/ canonical object model: model (projections, holds), routes (/api/v2 + /o),
backfill, reconcile, invariant, signing (presigned URLs), multipart,
backfill, reconcile, invariant, signing (presigned URLs), multipart,
content-type, verify-job (scheduled copy verification), copy-report,
content-type, verify-job (scheduled copy verification), copy-report,
- readiness (metadata / bytes_verified / playable) —
+ readiness (metadata / bytes_verified / playable), popularity (unique viewers
+ per day, no identifiers), tier-policy + tiering (native objects in the R2 cache) —
see docs/object-model.md
see docs/object-model.md
jobs/ media_jobs: queue (states, idempotency, media.job.* events in the state
jobs/ media_jobs: queue (states, idempotency, media.job.* events in the state
change's transaction; payload queue.jobEvent: ids, state, ISO times,
change's transaction; payload queue.jobEvent: ids, state, ISO times,
@@ -290,6 +291,9 @@ directories are shared across apps) and responses carry a `note` saying so.
@@ -290,6 +291,9 @@ directories are shared across apps) and responses carry a `note` saying so.
| POST | `/admin/storage/tiers/bulk-move` | `{ ids: [...], target }` (max 50) → `{ moved, bytes, errors? }` |
| POST | `/admin/storage/tiers/bulk-move` | `{ ids: [...], target }` (max 50) → `{ moved, bytes, errors? }` |
| GET | `/admin/storage/tiers/policy` | **read-only R2 policy**: each threshold (`r2Enabled`, `r2MinViews`, `r2RecentAccessDays`, `r2MaxIdleDays`, `r2MaxPerSweep`) as `{ value, default, source: default\|setting }`, the promote and demote rules in words, the R2/B2 provider status, and this app's decisions: `last_24h` counts per action and outcome and the 20 most recent |
| GET | `/admin/storage/tiers/policy` | **read-only R2 policy**: each threshold (`r2Enabled`, `r2MinViews`, `r2RecentAccessDays`, `r2MaxIdleDays`, `r2MaxPerSweep`) as `{ value, default, source: default\|setting }`, the promote and demote rules in words, the R2/B2 provider status, and this app's decisions: `last_24h` counts per action and outcome and the 20 most recent |
| GET | `/admin/storage/tiers/decisions?vod_id&action&outcome&limit&before_id` | this app's R2 promote/demote decisions, newest first: `{ id, decided_at, vod_id, object_id, action, from_provider, to_provider, outcome: done\|already\|refused\|failed, trigger: sweep\|admin\|clip\|drill\|manual, reason, inputs, thresholds, error }` → `{ decisions, next_before_id }` |
| GET | `/admin/storage/tiers/decisions?vod_id&action&outcome&limit&before_id` | this app's R2 promote/demote decisions, newest first: `{ id, decided_at, vod_id, object_id, action, from_provider, to_provider, outcome: done\|already\|refused\|failed, trigger: sweep\|admin\|clip\|drill\|manual, reason, inputs, thresholds, error }` → `{ decisions, next_before_id }` |
+| GET | `/admin/storage/tiers/objects/policy` | native v2 objects' R2 tiering (below): the activation gate, each `media.object_tier` threshold as `{ value, default, source }`, the rules, this app's promotion candidates (with what would refuse each) and its decisions (24-hour counts and the 20 most recent) |
+| GET | `/admin/storage/tiers/objects/decisions?object_id&action&outcome&limit&before_id` | this app's native object decisions, newest first (`outcome`: done\|already\|refused\|failed\|dry_run) → `{ decisions, next_before_id }` |
+| GET / POST | `/admin/storage/config[/media.storage_tier\|/media.object_tier[/history\|/rollback]]` | the revisioned tier policies (openvibe-shared/config): read, apply `{ values, merge, reason }`, history, roll back |
| GET | `/admin/storage/holds?all&object_id&vod_id&clip_id&limit&before_id` | this app's retention holds, newest first (active only unless `all=1`) |
| GET | `/admin/storage/holds?all&object_id&vod_id&clip_id&limit&before_id` | this app's retention holds, newest first (active only unless `all=1`) |
| POST | `/admin/storage/holds` | `{ object_id \| vod_id \| clip_id, reason, kind?, note?, placed_by? }` → 201: the object is kept (no delete, no tier move) and so are the clips cut from a held VOD. Not for calls acting for a user (403) |
| POST | `/admin/storage/holds` | `{ object_id \| vod_id \| clip_id, reason, kind?, note?, placed_by? }` → 201: the object is kept (no delete, no tier move) and so are the clips cut from a held VOD. Not for calls acting for a user (403) |
| POST | `/admin/storage/holds/:holdId/release` | `{ released_by }` (also `DELETE /admin/storage/holds/:holdId`); 409 when already released. Holds stay on record |
| POST | `/admin/storage/holds/:holdId/release` | `{ released_by }` (also `DELETE /admin/storage/holds/:holdId`); 409 when already released. Holds stay on record |
@@ -412,7 +416,9 @@ apply it to an app acting for someone other than the owner.
@@ -412,7 +416,9 @@ apply it to an app acting for someone other than the owner.
`GET /o/:id` serves object bytes by canonical id (`med_…`): public/unlisted
`GET /o/:id` serves object bytes by canonical id (`med_…`): public/unlisted
objects openly, private ones only with a valid signature from
objects openly, private ones only with a valid signature from
`/api/v2/:app/objects/:id/download`, deleted ones 410 (a deleted private object
`/api/v2/:app/objects/:id/download`, deleted ones 410 (a deleted private object
-without a signature is a 404 like any other private one).
+without a signature is a 404 like any other private one). A native object in the
+R2 popularity cache redirects to its verified R2 copy; a GET of a native object
+counts its viewer for the day, identifying no one (see Storage tiering).
## Object model and API v2
## Object model and API v2
@@ -463,7 +469,22 @@ promote/demote); presigned-302 playback. Knobs live in `media_settings` under
@@ -463,7 +469,22 @@ promote/demote); presigned-302 playback. Knobs live in `media_settings` under
clip cut fetching an R2 VOD home, the eviction drill) is logged in `media_tier_decisions` with the
clip cut fetching an R2 VOD home, the eviction drill) is logged in `media_tier_decisions` with the
inputs it saw (views, last access, provider, recording, hold, size), the thresholds in force and their
inputs it saw (views, last access, provider, recording, hold, size), the thresholds in force and their
source, the reason and the outcome; `/admin/storage/tiers/policy` shows the policy and the log, and
source, the reason and the outcome; `/admin/storage/tiers/policy` shows the policy and the log, and
-`media_tier_decisions_24h{action,outcome}` counts them on `/metrics` (`test/tier-decisions.test.js`). CLI:
+`media_tier_decisions_24h{target="vod",action,outcome}` counts them on `/metrics` (`test/tier-decisions.test.js`).
+
+**Native v2 objects** (roadmap WS-G task 11; [docs](docs/object-model.md#tiering-of-native-objects)) keep
+their canonical copy and, when popular, get a verified copy in R2 that `/o/:id` plays from. Popularity is
+unique viewers per object per UTC day, counted on `/o/:id` without storing an IP address or subject id
+(a daily-salted hash, deleted with its salt when the day ends; only the counts stay, 30 days), summed over
+7 days. The policy is the revisioned namespace `media.object_tier`: promote at `promoteMinUniqueViewers7d`
+(500) unique viewers, viewed within 3 days, 16 MB to 4 GB, at most 3 per sweep; demote after 14 idle days
+or when the object is no longer ready. Its **activation gate `active` is off by default**: the sweep (step 4
+of this one) then moves nothing and records what it would do as `dry_run` decisions. Holds block every
+move, only a verified canonical copy is promoted (re-hashed; the R2 copy's size and sha256 are checked
+before it is used), and a demotion never removes the last good copy. Decisions:
+`media_object_tier_decisions`, `/admin/storage/tiers/objects/*`, `/me/ops`, and
+`media_tier_decisions_24h{target="object"}` (`test/object-tiering.test.js`).
+
+The storage engine's CLI:
```
```
node server/vod/vod-storage.js check|migrate-legacy|drain [targetPct]
node server/vod/vod-storage.js check|migrate-legacy|drain [targetPct]
@@ -8,10 +8,10 @@ unchanged. New code talks to objects through the [v2 API](#object-api-v2).
@@ -8,10 +8,10 @@ unchanged. New code talks to objects through the [v2 API](#object-api-v2).
Code: `server/objects/` (`model.js`, `routes.js`, `backfill.js`, `reconcile.js`, `invariant.js`,
Code: `server/objects/` (`model.js`, `routes.js`, `backfill.js`, `reconcile.js`, `invariant.js`,
`signing.js`, `verify-job.js`, `copy-report.js`, `multipart.js`, `content-type.js`, `readiness.js`, `namespaces.js`,
`signing.js`, `verify-job.js`, `copy-report.js`, `multipart.js`, `content-type.js`, `readiness.js`, `namespaces.js`,
-`namespace-routes.js`) and the job system in
+`namespace-routes.js`, `popularity.js`, `tier-policy.js`, `tiering.js`) and the job system in
`server/jobs/` (`queue.js`, `worker.js`, `routes.js`, `types.js`, `thumbnail.js`, `invariant-scan.js`,
`server/jobs/` (`queue.js`, `worker.js`, `routes.js`, `types.js`, `thumbnail.js`, `invariant-scan.js`,
`derive.js`). Schema: the bottom of `server/db/schema.sql`. Tests: `test/objects-*.test.js`, `test/jobs*.test.js`,
`derive.js`). Schema: the bottom of `server/db/schema.sql`. Tests: `test/objects-*.test.js`, `test/jobs*.test.js`,
-`test/r2-eviction-drill.test.js`.
+`test/r2-eviction-drill.test.js`, `test/object-tiering.test.js`.
## Tables
## Tables
@@ -244,7 +244,9 @@ All routes live under `/api/v2/:app/objects`.
@@ -244,7 +244,9 @@ All routes live under `/api/v2/:app/objects`.
- Public and unlisted `ready` objects are served openly.
- Public and unlisted `ready` objects are served openly.
- Private objects need a valid `?exp=&sig=` from `/download`. Without one the answer is 404, so a private object cannot be told apart from a missing one.
- Private objects need a valid `?exp=&sig=` from `/download`. Without one the answer is 404, so a private object cannot be told apart from a missing one.
- A deleted object answers 410 — after the check above, so a deleted private object is a 404 without a signature.
- A deleted object answers 410 — after the check above, so a deleted private object is a 404 without a signature.
-- A local copy streams with Range support. Otherwise the route redirects 302 to a presigned R2 URL, then a presigned B2 URL.
+- A native object [promoted to the R2 cache](#tiering-of-native-objects) redirects 302 to its verified R2 copy (a presigned URL that answers with the type and `Content-Disposition` this route would send) while R2 is available.
+- Otherwise a local copy streams with Range support, or the route redirects 302 to a presigned R2 URL, then a presigned B2 URL.
+- A GET of a native object counts its viewer for the day, without identifying anyone ([popularity](#popularity)).
- Headers: `X-Content-Type-Options: nosniff` and `X-Robots-Tag: noindex`. Content is `inline` only for images (not SVG), video, audio, PDF and plain text, and `attachment` for everything else.
- Headers: `X-Content-Type-Options: nosniff` and `X-Robots-Tag: noindex`. Content is `inline` only for images (not SVG), video, audio, PDF and plain text, and `attachment` for everything else.
**Signing.** HMAC-SHA256 keyed with `MEDIA_SIGNING_SECRET` over a purpose and what the link may do:
**Signing.** HMAC-SHA256 keyed with `MEDIA_SIGNING_SECRET` over a purpose and what the link may do:
@@ -318,7 +320,9 @@ Code: `server/objects/content-type.js`.
@@ -318,7 +320,9 @@ Code: `server/objects/content-type.js`.
**Purge.** Every hour, the service removes the bytes of native objects whose retention period has
**Purge.** Every hour, the service removes the bytes of native objects whose retention period has
passed and that have no active hold. The object row stays `deleted` with `metadata.purged_at`.
passed and that have no active hold. The object row stays `deleted` with `metadata.purged_at`.
-Projected objects are never purged by this job.
+Projected objects are never purged by this job. An object that still has a copy in the R2 cache waits
+until the [object tiering](#tiering-of-native-objects) has removed it (it demotes deleted objects), so a
+purge never leaves the R2 copy behind.
## Namespaces, grants and quotas
## Namespaces, grants and quotas
@@ -454,7 +458,7 @@ cheap and renders in a drill.
@@ -454,7 +458,7 @@ cheap and renders in a drill.
- `jobs`: counts by status; failed jobs of the last 7 days by type and error code; the most recent failures with their error text.
- `jobs`: counts by status; failed jobs of the last 7 days by type and error code; the most recent failures with their error text.
- `missing`: ready objects with no good copy; copies recorded missing or corrupt; copies by provider and state; quarantined VODs; the last verification run.
- `missing`: ready objects with no good copy; copies recorded missing or corrupt; copies by provider and state; quarantined VODs; the last verification run.
- `backfill`: the last object backfill that changed something; rows with no object yet; objects whose owner subject is unresolved; whether upload reservations were seeded.
- `backfill`: the last object backfill that changed something; rows with no object yet; objects whose owner subject is unresolved; whether upload reservations were seeded.
-- `tiering`: providers and policy; VODs and clips per provider; objects per canonical copy (native or projected); VODs eligible to offload; R2 decisions of the last 24 h and recent refusals and failures; the sweep's state in this process.
+- `tiering`: providers and policy; VODs and clips per provider; objects per canonical copy (native or projected); VODs eligible to offload; R2 decisions of the last 24 h and recent refusals and failures; the sweep's state in this process; and `native_objects`: the [object tiering](#tiering-of-native-objects)'s activation gate and policy, native objects in R2, how many are eligible to promote now, its decisions of the last 24 h (dry runs included) and the most recent ones, and its last sweep.
- `namespaces`: every namespace with quota and usage snapshot.
- `namespaces`: every namespace with quota and usage snapshot.
## Readiness
## Readiness
@@ -518,7 +522,7 @@ While any hold applies:
@@ -518,7 +522,7 @@ While any hold applies:
- `deleteVodObjects` keeps the bytes.
- `deleteVodObjects` keeps the bytes.
- Paste screenshot removal keeps the file, and censor answers 409.
- Paste screenshot removal keeps the file, and censor answers 409.
- A `BEFORE DELETE` trigger on each projected table, plus a trigger on `media_objects.lifecycle_status`, refuses the change, so paths nobody has hooked are still covered. The triggers are the `_v2` ones (clip-aware); an older database has its first-version guards replaced at start.
- A `BEFORE DELETE` trigger on each projected table, plus a trigger on `media_objects.lifecycle_status`, refuses the change, so paths nobody has hooked are still covered. The triggers are the `_v2` ones (clip-aware); an older database has its first-version guards replaced at start.
-- **No tier move.** `moveToCold`, `moveToHot`, `promoteToR2` and `demoteFromR2` return `{ ok: false, held: true }` (the admin moves answer it, R2 moves are logged as `refused`), and the sweep counts these as `skippedHeld` instead of errors. A hold freezes an object's placement: `moveToHot` is refused too, because restoring flips the row to local and drops an R2 copy. A clip cut from a held VOD in B2/R2 is cut from the cloud copy instead of fetching the VOD home. Row updates that only record where the bytes already are (the sweep finding a local file gone while B2 has the copy, the legacy-tier migration) still happen: they move nothing.
+- **No tier move.** `moveToCold`, `moveToHot`, `promoteToR2` and `demoteFromR2` return `{ ok: false, held: true }` (the admin moves answer it, R2 moves are logged as `refused`), and the sweep counts these as `skippedHeld` instead of errors. A held native object is never promoted to or demoted from the R2 cache either (logged as `refused`; a hold placed while its bytes were being copied wins, and the unrecorded copy is removed). A hold freezes an object's placement: `moveToHot` is refused too, because restoring flips the row to local and drops an R2 copy. A clip cut from a held VOD in B2/R2 is cut from the cloud copy instead of fetching the VOD home. Row updates that only record where the bytes already are (the sweep finding a local file gone while B2 has the copy, the legacy-tier migration) still happen: they move nothing.
**Staff routes** (`/api/v1/:app/admin/storage/holds`, the app key like every admin route; a call acting
**Staff routes** (`/api/v1/:app/admin/storage/holds`, the app key like every admin route; a call acting
for one of the app's users, `X-OV-User-Id`, gets 403):
for one of the app's users, `X-OV-User-Id`, gets 403):
@@ -855,6 +859,105 @@ node scripts/r2-eviction-drill.js (--vod <id> | --pick) [--execute] [--base-url
@@ -855,6 +859,105 @@ node scripts/r2-eviction-drill.js (--vod <id> | --pick) [--execute] [--base-url
Not run against production yet: it needs the owner's go-ahead.
Not run against production yet: it needs the owner's go-ahead.
+## Tiering of native objects
+
+Roadmap WS-G task 11. Code: `server/objects/popularity.js`, `server/objects/tier-policy.js`,
+`server/objects/tiering.js`. Test: `test/object-tiering.test.js` (a fake S3 through the real SDK).
+
+A native object (uploaded through `/api/v2`, `legacy_ref` NULL) keeps its canonical copy where the upload
+put it: local disk (`OBJECTS_PATH`), or B2 for a B2 canonical copy. When it is popular it also gets a copy
+in the Cloudflare R2 popularity cache (`storage_class` `cache`, key `objects/<app>/<id>`, or the B2 key),
+and that copy becomes its playback source: `GET /o/:id` redirects to it while R2 is available, and the
+canonical copy is the fallback. When the object goes idle, or is no longer `ready` (deleted), the R2 copy
+is removed again. Projected objects (VODs, clips) keep tiering through `vod-storage.js` as before.
+
+### Popularity
+
+Unique viewers per object per UTC day, with no IP address or subject id stored (ADR-021):
+
+- **Who is a viewer.** The client's network: `req.ip` (see [Client addresses](../README.md#client-addresses)),
+ an IPv6 address reduced to its /64. One network is one viewer, whatever its user agent: people behind
+ one NAT count once, which errs on the conservative side, and a client cannot inflate a count.
+- **What is stored.** `media_object_viewer_days` holds `HMAC-SHA256(salt of the day, network)`, 16 hex
+ characters, per object and day; the salt is 32 random bytes per UTC day (`media_object_view_salts`).
+ `media_object_views_daily` holds the count (`unique_viewers`, `last_viewed_at`) per object and day.
+- **Rotation.** Once a UTC day is over, its hashes and its salt are deleted: by the first view of the next
+ day, by the hourly maintenance and at the start of every tiering sweep. After that nothing links a count
+ to an address, even with the database and every secret. The daily counts are kept 30 days.
+- **The 7-day figure** is the sum of the last seven daily counts (today included): viewer-days. The same
+ network on three days counts three times, as in the shared analytics' multi-day uniques.
+- **Counted** on `GET /o/:id` of a ready, native object of a non-sandbox tenant, public or signed. **Not
+ counted:** `HEAD`, range requests that do not start at byte 0 (a player's seeks), `Sec-GPC: 1` or
+ `DNT: 1` (no hash is made at all), crawlers and clients without a user agent
+ (`openvibe-shared/analytics` `privacy.optedOut` / `classifyRequest`). A response that Cloudflare serves
+ from its cache never reaches Media and is not counted.
+
+### Policy
+
+The namespace `media.object_tier` of `openvibe-shared/config`, beside `media.storage_tier`: revisions with
+who and why, validated as a whole, history and rollback through `…/admin/storage/config/media.object_tier`.
+Revision 1 sets nothing, so every key is its built-in default until a revision sets it.
+
+| key | default | meaning |
+|---|---|---|
+| `active` | `false` | **the site-level activation gate.** Off: nothing moves (not through the sweep, not through a direct `promote()`/`demote()`); the sweep records what it would do as `dry_run` decisions |
+| `promoteMinUniqueViewers7d` | 500 | unique viewers over the last 7 UTC days to promote (1 to 10⁹) |
+| `promoteRecentAccessDays` | 3 | …and viewed within this many days, today included (1-30) |
+| `promoteMinSizeMb` / `promoteMaxSizeMb` | 16 / 4096 | size bounds: a small object is cheap to serve locally; the upper bound caps the bytes one promotion hashes and reads back |
+| `maxPromotionsPerSweep` | 3 | promotions (and dry runs) per sweep (0-100) |
+| `demoteIdleDays` | 14 | no viewer for this many days, and in R2 at least that long: demote (1-30) |
+| `maxDemotionsPerSweep` | 10 | demotions per sweep (0-500) |
+
+Refused (422 `config.invalid`, nothing changes): a wrong type, an unknown key, a value out of range,
+`promoteMinSizeMb` not below `promoteMaxSizeMb`, and `demoteIdleDays` not above `promoteRecentAccessDays`
+(an object viewed recently enough to promote would also be idle enough to demote).
+
+### Sweep and moves
+
+The sweep is step 4 of the storage sweep (`vod-storage.js runSweep`, every `sweepIntervalMs`, and
+`POST /admin/storage/tiers/sweep`); its result is the storage sweep's `objects`. It is skipped while the
+local disk still needs a drain, never starts in a restore drill (and refuses under `MEDIA_DRILL` anyway).
+Each pass rotates the popularity counts, then:
+
+1. **Demotes** first (at most `maxDemotionsPerSweep`): R2 copies of native objects that are no longer
+ `ready`, or idle for `demoteIdleDays` and in R2 at least that long.
+2. **Promotes** (at most `maxPromotionsPerSweep`), most viewed first: ready native objects of non-sandbox
+ tenants within the thresholds and without a present R2 copy.
+
+**Promotion** (`tiering.promote`): refused when the object is held, not ready, a sandbox object, or its
+canonical copy is not verified on record (present, checked, not contradicted, and the object has a
+sha256: the `object.hash` job supplies it for older objects); then the gate (`dry_run`); then R2 must be
+available. The canonical copy is checked again (a local file is hashed and must match the sha256; a B2
+copy must answer a HEAD with the right size), uploaded (or copied from B2), and the R2 copy is HEADed and
+read back: its size and sha256 must match. Only then is the R2 location recorded `present` (with the
+checksum), which makes it the playback source. A copy that does not verify is deleted and never recorded
+(`failed`); a hold or a delete that arrived during the copy wins (`refused`, the copy removed).
+
+**Demotion** (`tiering.demote`): refused under a hold; then the gate (`dry_run`); then the canonical copy
+must check out now (hashed again, or HEAD for B2). A copy that does not (missing, changed) keeps the R2
+copy, which may be the last good one (`refused`). The R2 object is deleted, a HEAD must confirm it is gone
+(`failed` otherwise, and the location stays), and the location row is removed.
+
+**Decisions.** Every outcome goes to `media_object_tier_decisions` (`done`, `already`, `refused`,
+`failed`, `dry_run`) with the inputs it saw (`unique_viewers_7d`, `last_viewed_day`, size, kind,
+visibility, lifecycle, the canonical copy's state, `held`, the R2 copy, `gate_active`, `r2_available`),
+the policy in force (each value with its source) and the reason. A refusal or dry run the sweep repeats is
+logged once a day per object, and a repeated dry run does not use up the sweep's budget, so over a day the
+log shows what successive sweeps would move. An object whose move failed waits 6 hours. They are counted
+in `media_tier_decisions_24h{target="object",action,outcome}` on `/metrics` (VOD decisions are
+`target="vod"`) and shown in the [operator views](#operator-views).
+
+**Admin API** (`/api/v1/:app/admin/storage`, the app key, this app's objects only):
+
+| method | path | notes |
+|---|---|---|
+| GET | `/tiers/objects/policy` | `gate`, each threshold as `{ value, default, source }`, the rules in words, R2's status, `candidates` (what would be promoted now: `object_id`, `unique_viewers_7d`, `last_viewed_day`, `blocked_by` when something on record would refuse it; `?limit`), `decisions.last_24h` per action and outcome and the 20 most recent |
+| GET | `/tiers/objects/decisions` | `?object_id&action=promote\|demote&outcome=done\|already\|refused\|failed\|dry_run&limit (≤200)&before_id` → `{ decisions, next_before_id }`, newest first |
+| GET / POST | `/config/media.object_tier`, `…/history`, `…/rollback` | the policy's revisions (`POST …/media.object_tier` `{ values: { active: true }, merge: true, reason }` turns the gate on) |
+
+**With the gate on and then off again,** placement freezes: promoted objects keep playing from R2, and a
+deleted object's R2 copy stays (so its purge waits) until the gate is on again.
+
## Configuration
## Configuration
| env | default | meaning |
| env | default | meaning |
@@ -897,7 +1000,8 @@ Not run against production yet: it needs the owner's go-ahead.
@@ -897,7 +1000,8 @@ Not run against production yet: it needs the owner's go-ahead.
## Not in this pass
## Not in this pass
- Presigned direct-to-bucket uploads. Presigned URLs are Media-signed and native bytes are stored locally.
- Presigned direct-to-bucket uploads. Presigned URLs are Media-signed and native bytes are stored locally.
-- Tiering of native objects to B2/R2.
+- Offloading native objects' canonical copy to B2. Popular native objects are cached in R2
+ ([Tiering of native objects](#tiering-of-native-objects)); their canonical copy stays local.
- Copy/move aliases, lifecycle rules and an S3-compatible façade.
- Copy/move aliases, lifecycle rules and an S3-compatible façade.
- Clip cutting and the finalize remux still run inline, not as jobs (thumbnail regeneration was the first
- Clip cutting and the finalize remux still run inline, not as jobs (thumbnail regeneration was the first
operation moved onto the job system).
operation moved onto the job system).
@@ -912,3 +1016,5 @@ The tiering thresholds (`DEFAULTS` in `server/vod/vod-storage.js`) are the names
@@ -912,3 +1016,5 @@ The tiering thresholds (`DEFAULTS` in `server/vod/vod-storage.js`) are the names
- **Revision 1** is what `media_settings` held.
- **Revision 1** is what `media_settings` held.
- **Mirror:** every activation writes the active revision's explicit keys through to `media_settings` (`storage_tier.*`) and removes the rest, so a release that reads those rows sees the same policy. The rows go in the ADR-028 contract step.
- **Mirror:** every activation writes the active revision's explicit keys through to `media_settings` (`storage_tier.*`) and removes the rest, so a release that reads those rows sees the same policy. The rows go in the ADR-028 contract step.
- **Sweep:** a running sweep restarts with the new policy.
- **Sweep:** a running sweep restarts with the new policy.
+- **Native objects** have their own namespace, `media.object_tier` (`server/objects/tier-policy.js`): the
+ same revisions and routes, nothing mirrored to `media_settings`. See [Tiering of native objects](#tiering-of-native-objects).
@@ -21,6 +21,12 @@
@@ -21,6 +21,12 @@
* decisions (media_tier_decisions) with 24-hour counts
* decisions (media_tier_decisions) with 24-hour counts
* GET /tiers/decisions this app's R2 promote/demote decisions, newest first
* GET /tiers/decisions this app's R2 promote/demote decisions, newest first
* (?vod_id&action=promote|demote&outcome&limit≤200&before_id)
* (?vod_id&action=promote|demote&outcome&limit≤200&before_id)
+ * GET /tiers/objects/policy native v2 objects (server/objects/tiering.js): the activation gate, each
+ * threshold with its source, the rules, this app's promotion candidates (and what
+ * would refuse each), and its decisions (media_object_tier_decisions) with 24-hour counts
+ * GET /tiers/objects/decisions this app's native object decisions, newest first
+ * (?object_id&action&outcome=done|already|refused|failed|dry_run&limit≤200&before_id)
+ * GET /config… the revisioned policies media.storage_tier and media.object_tier (openvibe-shared/config)
* GET /holds this app's retention holds, newest first (?all=1 includes released;
* GET /holds this app's retention holds, newest first (?all=1 includes released;
* ?object_id|vod_id|clip_id; ?limit≤200&before_id)
* ?object_id|vod_id|clip_id; ?limit≤200&before_id)
* POST /holds place one: { object_id (med_… or legacy ref) | vod_id | clip_id, reason, kind
* POST /holds place one: { object_id (med_… or legacy ref) | vod_id | clip_id, reason, kind
@@ -347,7 +353,8 @@ router.put('/tiers/settings', async (req, res) => {
@@ -347,7 +353,8 @@ router.put('/tiers/settings', async (req, res) => {
let configRoutes = null;
let configRoutes = null;
router.use('/config', (req, res, next) => {
router.use('/config', (req, res, next) => {
if (!configRoutes) {
if (!configRoutes) {
- configRoutes = require('openvibe-shared/config').adminRoutes([vodStorage.tierConfig.get(vodStorage.DEFAULTS)], {
+ // media.storage_tier (VODs) and media.object_tier (native v2 objects: thresholds and the activation gate).
+ configRoutes = require('openvibe-shared/config').adminRoutes([vodStorage.tierConfig.get(vodStorage.DEFAULTS), require('../objects/tier-policy').get()], {
basePath: '/', requireAdmin: (_rq, _rs, nx) => nx(), actor: (rq) => ({ type: 'service', id: rq.appId }),
basePath: '/', requireAdmin: (_rq, _rs, nx) => nx(), actor: (rq) => ({ type: 'service', id: rq.appId }),
});
});
}
}
@@ -415,6 +422,43 @@ router.get('/tiers/decisions', (req, res) => {
@@ -415,6 +422,43 @@ router.get('/tiers/decisions', (req, res) => {
}
}
});
});
+// ── Native v2 objects: policy, candidates and decision log (read-only) ──
+// The policy changes through the config routes (…/config/media.object_tier); the sweep moves nothing while
+// its activation gate (`active`) is off, and logs what it would do as dry_run decisions.
+const objectTiers = () => require('../objects/tiering');
+
+router.get('/tiers/objects/policy', (req, res) => {
+ try {
+ const policy = require('../objects/tier-policy');
+ const settings = policy.settings();
+ res.json({
+ gate: { active: !!settings.active, ...policy.thresholds(settings).active },
+ thresholds: policy.thresholds(settings),
+ rules: policy.describe(settings),
+ provider: { r2: { configured: vodStorage.providerConfigured('r2'), available: vodStorage.providerAvailable('r2') } },
+ candidates: objectTiers().candidates({ appId: req.appId, limit: req.query.limit, settings }),
+ decisions: { last_24h: objectTiers().counts24h(req.appId), recent: objectTiers().listDecisions({ appId: req.appId, limit: 20 }).decisions },
+ note: 'Read-only. The policy (media.object_tier) changes through /config/media.object_tier; candidates and decisions are this app\'s only.',
+ });
+ } catch (err) {
+ console.error('[Admin] Object tier policy error:', err.message);
+ res.status(500).json({ error: 'Failed to read the object tier policy' });
+ }
+});
+
+router.get('/tiers/objects/decisions', (req, res) => {
+ try {
+ const q = req.query;
+ if (q.action && !['promote', 'demote'].includes(String(q.action))) return res.status(400).json({ error: 'action must be promote or demote' });
+ if (q.outcome && !objectTiers().OUTCOMES.includes(String(q.outcome))) return res.status(400).json({ error: `outcome must be one of ${objectTiers().OUTCOMES.join(', ')}` });
+ res.json(objectTiers().listDecisions({ appId: req.appId, objectId: q.object_id ? String(q.object_id) : null, action: q.action ? String(q.action) : null,
+ outcome: q.outcome ? String(q.outcome) : null, beforeId: q.before_id != null ? parseInt(q.before_id, 10) || 0 : null, limit: q.limit }));
+ } catch (err) {
+ console.error('[Admin] Object tier decisions error:', err.message);
+ res.status(500).json({ error: 'Failed to list object tier decisions' });
+ }
+});
+
// ── POST /tiers/sweep — trigger a sweep now ──────────────────
// ── POST /tiers/sweep — trigger a sweep now ──────────────────
router.post('/tiers/sweep', async (req, res) => {
router.post('/tiers/sweep', async (req, res) => {
try {
try {
@@ -429,6 +429,50 @@ CREATE TABLE IF NOT EXISTS media_tier_decisions (
@@ -429,6 +429,50 @@ CREATE TABLE IF NOT EXISTS media_tier_decisions (
CREATE INDEX IF NOT EXISTS idx_media_tier_decisions_vod ON media_tier_decisions(vod_id, id);
CREATE INDEX IF NOT EXISTS idx_media_tier_decisions_vod ON media_tier_decisions(vod_id, id);
CREATE INDEX IF NOT EXISTS idx_media_tier_decisions_app ON media_tier_decisions(app_id, id);
CREATE INDEX IF NOT EXISTS idx_media_tier_decisions_app ON media_tier_decisions(app_id, id);
+-- Popularity of native v2 objects (server/objects/popularity.js; ADR-021): unique viewers per object per UTC
+-- day, never an IP address or subject id. A viewer is HMAC-SHA256(the day's random salt, client network), and
+-- the hashes and the salt of a day are deleted once it is over; only the daily counts stay (30 days).
+CREATE TABLE IF NOT EXISTS media_object_view_salts (
+ day TEXT PRIMARY KEY, -- YYYY-MM-DD (UTC)
+ salt TEXT NOT NULL, -- 32 random bytes, hex; deleted with the day's hashes
+ created_at DATETIME DEFAULT CURRENT_TIMESTAMP
+);
+CREATE TABLE IF NOT EXISTS media_object_viewer_days (
+ day TEXT NOT NULL,
+ object_id TEXT NOT NULL,
+ viewer TEXT NOT NULL, -- 16 hex chars of HMAC-SHA256(salt of the day, client network)
+ PRIMARY KEY (day, object_id, viewer)
+) WITHOUT ROWID;
+CREATE TABLE IF NOT EXISTS media_object_views_daily (
+ object_id TEXT NOT NULL,
+ day TEXT NOT NULL, -- YYYY-MM-DD (UTC)
+ unique_viewers INTEGER NOT NULL DEFAULT 0,
+ last_viewed_at TEXT, -- ISO 8601: the day's last new viewer
+ PRIMARY KEY (object_id, day)
+);
+CREATE INDEX IF NOT EXISTS idx_media_object_views_daily_day ON media_object_views_daily(day, object_id);
+
+-- Tier decisions for native v2 objects (server/objects/tiering.js): every promotion to and demotion from the
+-- R2 popularity cache the object sweep decided, including what it would have done while the activation gate
+-- (media.object_tier `active`) is off (dry_run), with the inputs it saw and the policy in force.
+CREATE TABLE IF NOT EXISTS media_object_tier_decisions (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ decided_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
+ object_id TEXT NOT NULL,
+ app_id TEXT,
+ action TEXT NOT NULL CHECK(action IN ('promote', 'demote')),
+ from_provider TEXT,
+ to_provider TEXT,
+ outcome TEXT NOT NULL CHECK(outcome IN ('done', 'already', 'refused', 'failed', 'dry_run')),
+ trigger TEXT NOT NULL, -- sweep | manual
+ reason TEXT NOT NULL,
+ inputs TEXT NOT NULL DEFAULT '{}', -- JSON: unique_viewers_7d, last_viewed_day, size, kind, lifecycle, canonical copy, held, R2 copy, gate
+ thresholds TEXT NOT NULL DEFAULT '{}',-- JSON: the media.object_tier policy in force, each value with its source (default | setting)
+ error TEXT
+);
+CREATE INDEX IF NOT EXISTS idx_media_object_tier_decisions_object ON media_object_tier_decisions(object_id, id);
+CREATE INDEX IF NOT EXISTS idx_media_object_tier_decisions_app ON media_object_tier_decisions(app_id, id);
+
-- Object changes waiting to become events (server/events.js recordObjectChanges): media_objects
-- Object changes waiting to become events (server/events.js recordObjectChanges): media_objects
-- triggers (database.js ensureObjectTriggers) write one row per visibility change and per deletion,
-- triggers (database.js ensureObjectTriggers) write one row per visibility change and per deletion,
-- in the transaction that makes the change, whatever path made it; the rows become
-- in the transaction that makes the change, whatever path made it; the rows become
@@ -242,6 +242,8 @@ if (!drill.enabled) {
@@ -242,6 +242,8 @@ if (!drill.enabled) {
try { const n = require('./objects/model').purgeExpired(); if (n) console.log(`[Objects] Purged ${n} expired deleted object(s)`); } catch (err) { console.warn('[Objects] purge:', err.message); }
try { const n = require('./objects/model').purgeExpired(); if (n) console.log(`[Objects] Purged ${n} expired deleted object(s)`); } catch (err) { console.warn('[Objects] purge:', err.message); }
// Incomplete multipart uploads past MEDIA_MULTIPART_TTL_HOURS: their parts are deleted (the objects stay uploading).
// Incomplete multipart uploads past MEDIA_MULTIPART_TTL_HOURS: their parts are deleted (the objects stay uploading).
try { const r = require('./objects/multipart').purgeExpired(); if (r.expired || r.orphan_dirs) console.log(`[Objects] Multipart: ${r.expired} expired session(s), ${r.orphan_dirs} orphan part dir(s) removed`); } catch (err) { console.warn('[Objects] multipart purge:', err.message); }
try { const r = require('./objects/multipart').purgeExpired(); if (r.expired || r.orphan_dirs) console.log(`[Objects] Multipart: ${r.expired} expired session(s), ${r.orphan_dirs} orphan part dir(s) removed`); } catch (err) { console.warn('[Objects] multipart purge:', err.message); }
+ // Native objects' popularity (ADR-021): a finished UTC day's viewer hashes and salt are deleted, only its counts stay (30 days).
+ try { const r = require('./objects/popularity').rotate(); if (r.hashes || r.salts || r.counts) console.log(`[Popularity] ${r.hashes} viewer hash(es) and ${r.salts} salt(s) of finished days deleted, ${r.counts} old daily count(s) pruned`); } catch (err) { console.warn('[Popularity] rotate:', err.message); }
// Namespaces: uploads whose quota reservation expired are failed and their bytes freed; then every
// Namespaces: uploads whose quota reservation expired are failed and their bytes freed; then every
// namespace's usage snapshot is refreshed (v1 files, deletes and purges included).
// namespace's usage snapshot is refreshed (v1 files, deletes and purges included).
try {
try {
@@ -156,6 +156,9 @@ function tiering(appId, limit) {
@@ -156,6 +156,9 @@ function tiering(appId, limit) {
FROM media_tier_decisions WHERE outcome IN ('refused', 'failed')${s.sql} ORDER BY id DESC LIMIT ?`, [...s.params, Math.min(limit, 20)])
FROM media_tier_decisions WHERE outcome IN ('refused', 'failed')${s.sql} ORDER BY id DESC LIMIT ?`, [...s.params, Math.min(limit, 20)])
.map(r => ({ ...r, reason: clip(r.reason, 300), error: clip(r.error, 300) }));
.map(r => ({ ...r, reason: clip(r.reason, 300), error: clip(r.error, 300) }));
const sweep = vodStorage.sweepInfo();
const sweep = vodStorage.sweepInfo();
+ // Native v2 objects (objects/tiering.js): the activation gate, the policy, R2 copies, what would be promoted
+ // now, and the decisions (dry runs included), from the database like everything here.
+ const nativeObjects = require('../objects/tiering').report({ appId, limit: Math.min(limit, 20) });
return {
return {
providers: { b2: vodStorage.providerConfigured('b2'), r2: vodStorage.providerConfigured('r2') },
providers: { b2: vodStorage.providerConfigured('b2'), r2: vodStorage.providerConfigured('r2') },
policy: { enabled: !!settings.enabled, r2_enabled: !!settings.r2Enabled, min_age_days: settings.minAgeDays, max_views_for_cold: settings.maxViewsForCold, min_last_access_days: settings.minLastAccessDays },
policy: { enabled: !!settings.enabled, r2_enabled: !!settings.r2Enabled, min_age_days: settings.minAgeDays, max_views_for_cold: settings.maxViewsForCold, min_last_access_days: settings.minLastAccessDays },
@@ -175,7 +178,8 @@ function tiering(appId, limit) {
@@ -175,7 +178,8 @@ function tiering(appId, limit) {
timestamp: sweep.lastResult.timestamp || null,
timestamp: sweep.lastResult.timestamp || null,
} : null,
} : null,
},
},
- note: 'Native v2 objects stay on local storage until roadmap WS-G task 11 (tiering for native objects). The sweep and the provider switches are service-wide; counts follow the scope.',
+ native_objects: nativeObjects,
+ note: 'VODs tier by the media.storage_tier policy; native v2 objects keep their canonical copy and are cached in R2 by the media.object_tier policy, only while its activation gate is on (dry runs otherwise). The sweep, the gate and the provider switches are service-wide; counts follow the scope.',
};
};
}
}
@@ -331,6 +331,7 @@ function renderOps({ person, report, canRecompute, recomputed = null }) {
@@ -331,6 +331,7 @@ function renderOps({ person, report, canRecompute, recomputed = null }) {
const card = (value, label) => `<div class="card"><b>${value}</b><span>${esc(label)}</span></div>`;
const card = (value, label) => `<div class="card"><b>${value}</b><span>${esc(label)}</span></div>`;
const kv = (obj) => Object.entries(obj || {}).map(([k, v]) => `${esc(k.replace(/_/g, ' '))} ${num(v)}`).join(' · ') || 'none';
const kv = (obj) => Object.entries(obj || {}).map(([k, v]) => `${esc(k.replace(/_/g, ' '))} ${num(v)}`).join(' · ') || 'none';
const t = r.tiering;
const t = r.tiering;
+ const no = t.native_objects || null;
const bf = r.backfill;
const bf = r.backfill;
const m = r.missing;
const m = r.missing;
const body = `
const body = `
@@ -418,6 +419,16 @@ function renderOps({ person, report, canRecompute, recomputed = null }) {
@@ -418,6 +419,16 @@ function renderOps({ person, report, canRecompute, recomputed = null }) {
${table('Recent refused or failed tier moves', ['When', 'Tenant', 'VOD', 'Move', 'Outcome', 'Why'], t.recent_problems.map(d => [
${table('Recent refused or failed tier moves', ['When', 'Tenant', 'VOD', 'Move', 'Outcome', 'Why'], t.recent_problems.map(d => [
esc(fmtWhen(d.decided_at)), esc(d.app_id || ''), esc(d.vod_id), `${esc(d.action)} ${esc(d.from_provider || '')} → ${esc(d.to_provider || '')}`, esc(d.outcome), esc(d.error || d.reason || ''),
esc(fmtWhen(d.decided_at)), esc(d.app_id || ''), esc(d.vod_id), `${esc(d.action)} ${esc(d.from_provider || '')} → ${esc(d.to_provider || '')}`, esc(d.outcome), esc(d.error || d.reason || ''),
])) || '<p class="empty">None.</p>'}
])) || '<p class="empty">None.</p>'}
+ ${no ? `<h3>Native objects</h3>
+ <dl class="facts">
+ <dt>Activation gate</dt><dd>${no.gate.active ? '<strong>on</strong>: the sweep moves objects' : 'off: dry runs only'} (${esc(no.gate.source)})</dd>
+ <dt>Policy</dt><dd>promote at ${num(no.policy.promoteMinUniqueViewers7d)} unique viewers in 7 days, viewed within ${esc(no.policy.promoteRecentAccessDays)} d, ${esc(no.policy.promoteMinSizeMb)} to ${esc(no.policy.promoteMaxSizeMb)} MB · demote after ${esc(no.policy.demoteIdleDays)} idle days</dd>
+ <dt>In R2</dt><dd>${num(no.r2_copies.count)} objects, ${esc(fmtBytes(no.r2_copies.bytes))} · eligible to promote now: ${num(no.eligible_to_promote)}</dd>
+ <dt>Decisions, 24 h</dt><dd>promote: ${kv(no.decisions_24h.promote)} · demote: ${kv(no.decisions_24h.demote)}</dd>
+ </dl>
+ ${table('Recent native object tier decisions', ['When', 'Tenant', 'Object', 'Action', 'Outcome', 'Why'], no.recent.map(d => [
+ esc(fmtWhen(d.decided_at)), esc(d.app_id || ''), `<code class="id">${esc(d.object_id)}</code>`, esc(d.action), esc(d.outcome.replace('_', ' ')), esc(d.error || d.reason || ''),
+ ])) || '<p class="empty">None.</p>'}` : ''}
</section>
</section>
<p class="hint">JSON: <a href="/api/v2/me/ops${scopeQs}">/api/v2/me/ops</a>.</p>`;
<p class="hint">JSON: <a href="/api/v2/me/ops${scopeQs}">/api/v2/me/ops</a>.</p>`;
return doc({ title: 'Media operations', description: 'Operator views of OpenVibe.Media: failed jobs, usage, missing media, backfill and tiering.', body });
return doc({ title: 'Media operations', description: 'Operator views of OpenVibe.Media: failed jobs, usage, missing media, backfill and tiering.', body });
@@ -658,7 +658,9 @@ function restore(obj) {
@@ -658,7 +658,9 @@ function restore(obj) {
/**
/**
* Remove the bytes of native (v2) objects deleted longer ago than the retention
* Remove the bytes of native (v2) objects deleted longer ago than the retention
* period. Projected objects are never purged here — their bytes belong to the
* period. Projected objects are never purged here — their bytes belong to the
- * inherited pipelines. Held objects are skipped.
+ * inherited pipelines. Held objects are skipped, and so is an object that still
+ * has a copy in the R2 popularity cache: the object tiering removes that first
+ * (./tiering.js demotes deleted objects), so a purge never leaves it behind.
*/
*/
function purgeExpired({ retentionDays = config.objects.retentionDays } = {}) {
function purgeExpired({ retentionDays = config.objects.retentionDays } = {}) {
const rows = db.all(`SELECT * FROM media_objects WHERE lifecycle_status = 'deleted' AND legacy_ref IS NULL
const rows = db.all(`SELECT * FROM media_objects WHERE lifecycle_status = 'deleted' AND legacy_ref IS NULL
@@ -668,6 +670,7 @@ function purgeExpired({ retentionDays = config.objects.retentionDays } = {}) {
@@ -668,6 +670,7 @@ function purgeExpired({ retentionDays = config.objects.retentionDays } = {}) {
for (const obj of rows) {
for (const obj of rows) {
const md = parseJson(obj.metadata, {});
const md = parseJson(obj.metadata, {});
if (md.purged_at || isHeld(obj.id)) continue;
if (md.purged_at || isHeld(obj.id)) continue;
+ if (listLocations(obj.id).some(l => l.provider === 'r2')) continue;
for (const l of listLocations(obj.id)) {
for (const l of listLocations(obj.id)) {
if (l.provider === 'local' && path.resolve(l.key).startsWith(root)) { try { fs.unlinkSync(l.key); } catch { /* already gone */ } }
if (l.provider === 'local' && path.resolve(l.key).startsWith(root)) { try { fs.unlinkSync(l.key); } catch { /* already gone */ } }
if (l.provider === 'local') db.run('DELETE FROM media_locations WHERE id = ?', [l.id]);
if (l.provider === 'local') db.run('DELETE FROM media_locations WHERE id = ?', [l.id]);
new file mode 100644
new file mode 100644
@@ -0,0 +1,180 @@
@@ -0,0 +1,180 @@
+/**
+ * OpenVibe.Media — popularity of native v2 objects, counted without identifying anyone (roadmap WS-G task 11;
+ * ADR-021; docs/object-model.md#tiering-of-native-objects).
+ *
+ * record(obj, req) GET /o/:id: the request's viewer counts once per object per UTC day
+ * stats({ ids, days }) unique viewers per object over the last `days` UTC days (today included)
+ * rotate() the hashes and the salt of every finished day are deleted; counts past 30 days too
+ *
+ * A viewer is the client's network: req.ip (server/client-ip.js: the address nginx saw, never a header a
+ * direct caller sends), an IPv6 address reduced to its /64. It is never stored. A row holds HMAC-SHA256(salt
+ * of the day, network), 16 hex characters (media_object_viewer_days), and the salt is 32 random bytes per UTC
+ * day (media_object_view_salts). Once the day is over, rotate() deletes that day's hashes and its salt: from
+ * then on nothing links a count to an address, not even with the database and every secret. What stays is one
+ * count per object and day (media_object_views_daily: unique_viewers, last_viewed_at), for 30 days. The same
+ * network on two days is two viewer-days, so the 7-day figure is the sum of seven daily counts. One address
+ * is one viewer: people behind one NAT count once (the conservative side), and changing the user agent does
+ * not make a client count again.
+ *
+ * Not counted: HEAD and range requests past the first byte (a player's seeks), requests with Sec-GPC: 1 or
+ * DNT: 1 (openvibe-shared/analytics privacy.optedOut: no hash is made at all), crawlers and clients without a
+ * user agent (privacy.classifyRequest). Projected objects (vods, clips, files) keep their own view counts
+ * (server/views/service.js). Nothing runs at load; a restore drill never reaches record() (/o answers 503
+ * first) and never rotates.
+ */
+'use strict';
+
+const crypto = require('crypto');
+const net = require('net');
+const db = require('../db/database');
+const { privacy } = require('openvibe-shared/analytics');
+const { clientIp } = require('../client-ip');
+
+const WINDOW_DAYS = 7; // the rolling window the tiering policy reads
+const KEEP_DAYS = 30; // daily counts are kept this long (ADR-021's bound for raw analytics, applied to the counts too)
+const SEEN_MAX = 200000; // in-memory "already counted today" keys before the set starts over
+
+/** YYYY-MM-DD (UTC) of an epoch-ms time. */
+function dayOf(ms = Date.now()) {
+ return new Date(ms).toISOString().slice(0, 10);
+}
+
+/** The day `n` days after `day` (negative: before). */
+function addDays(day, n) {
+ return dayOf(Date.parse(`${day}T00:00:00Z`) + n * 864e5);
+}
+
+/** The first day of the `days`-day window that ends on (and includes) `today`. */
+function windowStart(today, days = WINDOW_DAYS) {
+ return addDays(today, -(Math.max(1, Number(days) || 1) - 1));
+}
+
+/** The network a viewer is counted as: an IPv4 address (also when IPv4-mapped), or the /64 of an IPv6 address. */
+function networkOf(ip) {
+ let s = String(ip || '').trim().toLowerCase();
+ if (s.startsWith('::ffff:') && net.isIPv4(s.slice(7))) s = s.slice(7);
+ if (net.isIPv4(s)) return s;
+ s = s.split('%')[0];
+ if (!net.isIPv6(s)) return s || 'unknown';
+ const [head, tail] = s.includes('::') ? s.split('::') : [s, null];
+ const h = head ? head.split(':') : [];
+ const t = tail ? tail.split(':') : [];
+ // An embedded IPv4 tail stands for two groups; only the first four groups are kept, which it never reaches.
+ const groups = [...h, ...(tail === null ? [] : Array(Math.max(0, 8 - h.length - t.length)).fill('0')), ...t];
+ return `${groups.slice(0, 4).map((g) => parseInt(g || '0', 16).toString(16)).join(':')}::/64`;
+}
+
+// ── The salt of the day ──────────────────────────────────────
+
+let _saltDay = null, _salt = null;
+
+/** The day's random salt: read, or made by the first view of the day (a racing writer keeps the first). */
+function daySalt(day) {
+ if (_saltDay === day && _salt) return _salt;
+ let row = db.get('SELECT salt FROM media_object_view_salts WHERE day = ?', [day]);
+ if (!row) {
+ db.run('INSERT OR IGNORE INTO media_object_view_salts (day, salt) VALUES (?, ?)', [day, crypto.randomBytes(32).toString('hex')]);
+ row = db.get('SELECT salt FROM media_object_view_salts WHERE day = ?', [day]);
+ }
+ _saltDay = day;
+ _salt = row.salt;
+ return _salt;
+}
+
+/** The stored form of a viewer on one day: 16 hex chars of HMAC-SHA256(salt of the day, network). */
+function viewerHash(network, day) {
+ return crypto.createHmac('sha256', daySalt(day)).update(String(network)).digest('hex').slice(0, 16);
+}
+
+// ── Recording ────────────────────────────────────────────────
+
+let _day = null;
+const _seen = new Set(); // `${object}|${viewer}` counted today in this process: no write for a repeat
+
+/** Why this request is not counted, or null when it is a view. */
+function notCounted(req) {
+ if (!req || req.method !== 'GET') return 'method';
+ const range = req.headers && req.headers.range;
+ if (range && !/^bytes=0-/.test(String(range))) return 'range';
+ if (privacy.optedOut(req.headers)) return 'opted_out';
+ if (privacy.classifyRequest(req).isBot) return 'bot';
+ return null;
+}
+
+/**
+ * Count the request's viewer for a native object (legacy_ref NULL), at most once per UTC day. Never throws.
+ * → { counted, reason } (reason: not_native, method, range, opted_out, bot, seen, error; null when counted)
+ */
+function record(obj, req, { now = Date.now() } = {}) {
+ try {
+ if (!obj || !obj.id || obj.legacy_ref) return { counted: false, reason: 'not_native' };
+ const why = notCounted(req);
+ if (why) return { counted: false, reason: why };
+ const day = dayOf(now);
+ if (day !== _day) {
+ // The first view of a new day: yesterday's hashes and salt go now (the hourly rotation would too).
+ _day = day;
+ _seen.clear();
+ rotate({ now });
+ }
+ const viewer = viewerHash(networkOf(clientIp(req)), day);
+ const seenKey = `${obj.id}|${viewer}`;
+ if (_seen.has(seenKey)) return { counted: false, reason: 'seen' };
+ const at = new Date(now).toISOString();
+ const added = db.getDb().transaction(() => {
+ const r = db.run('INSERT OR IGNORE INTO media_object_viewer_days (day, object_id, viewer) VALUES (?, ?, ?)', [day, obj.id, viewer]);
+ if (!r.changes) return false;
+ db.run(`INSERT INTO media_object_views_daily (object_id, day, unique_viewers, last_viewed_at) VALUES (?, ?, 1, ?)
+ ON CONFLICT(object_id, day) DO UPDATE SET unique_viewers = unique_viewers + 1, last_viewed_at = excluded.last_viewed_at`, [obj.id, day, at]);
+ return true;
+ })();
+ if (_seen.size >= SEEN_MAX) _seen.clear();
+ _seen.add(seenKey);
+ return { counted: added, reason: added ? null : 'seen' };
+ } catch (err) {
+ console.warn('[Popularity] view not recorded:', err.message);
+ return { counted: false, reason: 'error' };
+ }
+}
+
+/**
+ * Delete what must not outlive its day: every viewer hash and salt of a day before today, and daily counts
+ * older than KEEP_DAYS. → { hashes, salts, counts } (rows deleted)
+ */
+function rotate({ now = Date.now() } = {}) {
+ const today = dayOf(now);
+ const out = db.getDb().transaction(() => ({
+ hashes: db.run('DELETE FROM media_object_viewer_days WHERE day < ?', [today]).changes,
+ salts: db.run('DELETE FROM media_object_view_salts WHERE day < ?', [today]).changes,
+ counts: db.run('DELETE FROM media_object_views_daily WHERE day < ?', [addDays(today, -KEEP_DAYS)]).changes,
+ }))();
+ if (_saltDay && _saltDay < today) { _saltDay = null; _salt = null; }
+ return out;
+}
+
+/**
+ * Unique viewers over the last `days` UTC days (today included), per object: Map(object_id → { unique_viewers,
+ * last_viewed_day, last_viewed_at, days_viewed }). `ids` narrows it (else every object with a view in the window).
+ */
+function stats({ ids = null, days = WINDOW_DAYS, now = Date.now() } = {}) {
+ const since = windowStart(dayOf(now), days);
+ const list = ids == null ? null : [].concat(ids).filter(Boolean);
+ if (list && !list.length) return new Map();
+ const rows = db.all(`SELECT object_id, SUM(unique_viewers) AS viewers, MAX(day) AS last_day, MAX(last_viewed_at) AS last_at, COUNT(*) AS days
+ FROM media_object_views_daily WHERE day >= ?${list ? ` AND object_id IN (${list.map(() => '?').join(',')})` : ''}
+ GROUP BY object_id`, [since, ...(list || [])]);
+ return new Map(rows.map((r) => [r.object_id, { unique_viewers: r.viewers, last_viewed_day: r.last_day, last_viewed_at: r.last_at, days_viewed: r.days }]));
+}
+
+/** The last day anyone viewed the object (in the kept counts), or null. */
+function lastViewedDay(objectId) {
+ const r = db.get('SELECT MAX(day) AS d FROM media_object_views_daily WHERE object_id = ? AND unique_viewers > 0', [objectId]);
+ return (r && r.d) || null;
+}
+
+/** Test hook: forget the in-memory day, salt and repeats (as a restart would). */
+function _reset() {
+ _day = null; _seen.clear(); _saltDay = null; _salt = null;
+}
+
+module.exports = { WINDOW_DAYS, KEEP_DAYS, dayOf, addDays, windowStart, networkOf, viewerHash, record, rotate, stats, lastViewedDay, _reset };
@@ -722,10 +722,20 @@ publicRouter.get('/:id', async (req, res) => {
@@ -722,10 +722,20 @@ publicRouter.get('/:id', async (req, res) => {
'Cache-Control': obj.visibility === 'private' || sandbox ? 'private, no-store' : 'public, max-age=3600',
'Cache-Control': obj.visibility === 'private' || sandbox ? 'private, no-store' : 'public, max-age=3600',
'Content-Disposition': `${INLINE.test(mime) ? 'inline' : 'attachment'}; filename="${name}"`,
'Content-Disposition': `${INLINE.test(mime) ? 'inline' : 'attachment'}; filename="${name}"`,
};
};
+ // A viewer of a native object, counted once a day without being identified (objects/popularity.js); the
+ // object tiering reads these counts. Sandbox objects are never tiered, so they are not counted.
+ if (!obj.legacy_ref && !sandbox) require('./popularity').record(obj, req);
const locs = model.listLocations(obj.id);
const locs = model.listLocations(obj.id);
+ const vodStorage = require('../vod/vod-storage');
+ // A native object promoted to the R2 popularity cache (objects/tiering.js) plays from its verified R2 copy
+ // while R2 is available, with the headers this route would send; the canonical copy is the fallback.
+ const cached = !obj.legacy_ref && locs.find(l => l.provider === 'r2' && l.state === 'present' && l.verified_at);
+ if (cached && vodStorage.providerAvailable('r2')) {
+ const url = await vodStorage.presignGet('r2', cached.key, 300, { contentType: mime, contentDisposition: headers['Content-Disposition'] }).catch(() => null);
+ if (url) { res.set('Cache-Control', 'private, max-age=0'); res.set('X-Robots-Tag', 'noindex'); return res.redirect(302, url); }
+ }
const local = locs.find(l => l.provider === 'local' && l.state !== 'missing' && fs.existsSync(l.key));
const local = locs.find(l => l.provider === 'local' && l.state !== 'missing' && fs.existsSync(l.key));
if (local) return require('../public/routes').streamFileWithRange(req, res, local.key, headers);
if (local) return require('../public/routes').streamFileWithRange(req, res, local.key, headers);
- const vodStorage = require('../vod/vod-storage');
for (const p of ['r2', 'b2']) {
for (const p of ['r2', 'b2']) {
const l = locs.find(x => x.provider === p && x.state !== 'missing' && x.state !== 'corrupt');
const l = locs.find(x => x.provider === p && x.state !== 'missing' && x.state !== 'corrupt');
if (!l || !vodStorage.providerConfigured(p)) continue;
if (!l || !vodStorage.providerConfigured(p)) continue;
new file mode 100644
new file mode 100644
@@ -0,0 +1,120 @@
@@ -0,0 +1,120 @@
+/**
+ * OpenVibe.Media — the tiering policy for native v2 objects as revisioned configuration (roadmap WS-G task 11;
+ * docs/object-model.md#tiering-of-native-objects).
+ *
+ * The namespace `media.object_tier` of openvibe-shared/config, beside the VOD policy (`media.storage_tier`,
+ * server/vod/tier-config.js): every change is a revision with who and why, validated as a whole before it
+ * applies (types, ranges, and the order the thresholds need), with history and rollback through the admin
+ * config routes (GET/POST /api/v1/:app/admin/storage/config/media.object_tier…).
+ *
+ * active the site-level activation gate. Off (the default): the sweep moves nothing and
+ * only records what it would do (outcome dry_run)
+ * promoteMinUniqueViewers7d unique viewers over the last 7 UTC days (server/objects/popularity.js) to promote
+ * promoteRecentAccessDays …and viewed within this many days (today included)
+ * promoteMinSizeMb / MaxSizeMb size bounds: small objects are cheap to serve locally; the upper bound caps
+ * the bytes one promotion reads back from R2 to verify its sha256
+ * maxPromotionsPerSweep moves (and would-be moves) per sweep
+ * demoteIdleDays no viewer for this many days (and in R2 at least that long): demote
+ * maxDemotionsPerSweep
+ *
+ * Revision 1 sets nothing (every key is its built-in default, so a release that changes a default is not
+ * pinned to the old one); nothing is mirrored anywhere: no earlier release reads these keys.
+ */
+'use strict';
+const config = require('openvibe-shared/config');
+const db = require('../db/database');
+
+const DEFAULTS = {
+ active: false,
+ promoteMinUniqueViewers7d: 500,
+ promoteRecentAccessDays: 3,
+ promoteMinSizeMb: 16,
+ promoteMaxSizeMb: 4096,
+ maxPromotionsPerSweep: 3,
+ demoteIdleDays: 14,
+ maxDemotionsPerSweep: 10,
+};
+
+// Daily counts are kept 30 days (popularity.KEEP_DAYS): a longer look-back would see nothing.
+const MAX_LOOKBACK_DAYS = 30;
+
+const SCHEMA = {
+ type: 'object',
+ additionalProperties: false,
+ properties: {
+ active: { type: 'boolean' },
+ promoteMinUniqueViewers7d: { type: 'integer', minimum: 1, maximum: 1e9 },
+ promoteRecentAccessDays: { type: 'integer', minimum: 1, maximum: MAX_LOOKBACK_DAYS },
+ promoteMinSizeMb: { type: 'number', minimum: 0, maximum: 1048576 },
+ promoteMaxSizeMb: { type: 'number', minimum: 1, maximum: 1048576 },
+ maxPromotionsPerSweep: { type: 'integer', minimum: 0, maximum: 100 },
+ demoteIdleDays: { type: 'integer', minimum: 1, maximum: MAX_LOOKBACK_DAYS },
+ maxDemotionsPerSweep: { type: 'integer', minimum: 0, maximum: 500 },
+ },
+};
+
+/** Rules across keys: a size window that is not empty, and no object both promoted and demoted (flapping). */
+function validate(v) {
+ const errors = [];
+ if (v.promoteMinSizeMb >= v.promoteMaxSizeMb) errors.push('promoteMinSizeMb must be below promoteMaxSizeMb');
+ if (v.demoteIdleDays <= v.promoteRecentAccessDays) errors.push('demoteIdleDays must be above promoteRecentAccessDays (an object viewed recently enough to promote would also be idle enough to demote)');
+ return errors.length ? errors : true;
+}
+
+let store = null;
+
+/** The store (created on first use, once the database is open). */
+function get() {
+ if (store) return store;
+ store = config.createConfigStore({
+ db: db.getDb(), service: 'media', namespace: 'media.object_tier',
+ defaults: DEFAULTS, schema: SCHEMA, validate,
+ // Revision 1 is empty: the defaults apply, and a key set later shows as a setting.
+ legacy: () => ({}),
+ log: { info: (m) => console.log(`[ObjectTiers] ${m}`), warn: (m) => console.warn(`[ObjectTiers] ${m}`), error: (m) => console.error(`[ObjectTiers] ${m}`) },
+ });
+ return store;
+}
+
+/** Every key's effective value (the defaults under the active revision). */
+function settings() {
+ try { return { ...DEFAULTS, ...get().get() }; } catch (err) { console.warn('[ObjectTiers] config unavailable:', err.message); return { ...DEFAULTS }; }
+}
+
+/** Change some keys: one validated revision merged over the active one → the new snapshot (throws ConfigError 422/409). */
+function set(updates, { actor = null, reason = null } = {}) {
+ return get().apply(updates, { merge: true, actor, reason });
+}
+
+/** Keys the active revision sets explicitly (anything else is its default). */
+function explicitKeys() {
+ const s = get();
+ if (s.revision() == null) return new Set();
+ const snap = s.snapshot(s.revision());
+ return new Set(Object.keys((snap && snap.values) || {}));
+}
+
+/** Each key as { value, default, source: default | setting }. */
+function thresholds(values = settings()) {
+ let explicit = new Set();
+ try { explicit = explicitKeys(); } catch { /* defaults only */ }
+ const out = {};
+ for (const k of Object.keys(DEFAULTS)) out[k] = { value: values[k], default: DEFAULTS[k], source: explicit.has(k) ? 'setting' : 'default' };
+ return out;
+}
+
+/** The rules the sweep applies, in words. */
+function describe(v = settings()) {
+ return {
+ gate: v.active ? 'active: the sweep moves objects' : 'off (active = false): the sweep moves nothing and records what it would do as dry_run decisions',
+ promote: `a ready native (v2) object of a non-sandbox tenant with at least promoteMinUniqueViewers7d (${v.promoteMinUniqueViewers7d}) unique viewers over the last 7 UTC days, `
+ + `viewed within promoteRecentAccessDays (${v.promoteRecentAccessDays} days), between promoteMinSizeMb (${v.promoteMinSizeMb}) and promoteMaxSizeMb (${v.promoteMaxSizeMb}) MB, `
+ + 'whose canonical copy is verified (present, checked, sha256 on record) and not under a hold; the canonical copy is hashed again, copied to R2, and the R2 copy '
+ + `checked (size and sha256) before it becomes the playback source; at most maxPromotionsPerSweep (${v.maxPromotionsPerSweep}) per sweep`,
+ demote: `an R2 copy of a native object that is no longer ready (deleted, failed, archived), or whose object has had no viewer for demoteIdleDays (${v.demoteIdleDays} days) `
+ + `and has been in R2 at least that long; removed only after the canonical copy is confirmed good (hashed again), so the last good copy is never the one removed, `
+ + `and never under a hold; at most maxDemotionsPerSweep (${v.maxDemotionsPerSweep}) per sweep`,
+ };
+}
+
+module.exports = { DEFAULTS, SCHEMA, validate, get, settings, set, explicitKeys, thresholds, describe, _reset: () => { store = null; } };
new file mode 100644
new file mode 100644
@@ -0,0 +1,469 @@
@@ -0,0 +1,469 @@
+/**
+ * OpenVibe.Media — tiering of native v2 objects (roadmap WS-G task 11; docs/object-model.md#tiering-of-native-objects).
+ *
+ * A native object (uploaded through /api/v2: legacy_ref NULL) keeps its canonical copy where the upload put it
+ * (local disk, OBJECTS_PATH; a B2 canonical copy is handled the same way). A popular one also gets a copy in
+ * the Cloudflare R2 popularity cache, which becomes its playback source (GET /o/:id redirects there while R2
+ * is available; the canonical copy is the fallback), and loses it when it goes idle or stops being ready.
+ * Popularity is ./popularity.js (unique viewers per UTC day, no identifiers); the thresholds and the
+ * site-level activation gate are the revisioned policy media.object_tier (./tier-policy.js).
+ *
+ * runSweep({ trigger }) one pass: rotate the popularity counts, then promote and demote by the policy.
+ * Step 4 of the storage sweep (server/vod/vod-storage.js runSweep), which a restore
+ * drill never starts (and this refuses under MEDIA_DRILL anyway)
+ * promote(id, ctx) add a verified R2 copy: the canonical copy is hashed again (sha256 = content_hash),
+ * uploaded (or copied from B2), and the R2 copy HEADed and read back (size and sha256)
+ * before it is recorded present, which makes it the playback source
+ * demote(id, ctx) remove the R2 copy, after the canonical copy is confirmed good (hashed again)
+ * candidates({ appId }) what the policy would promote now, and what would refuse it (database only)
+ * report({ appId }) the operator views' part (database only)
+ *
+ * The gate: while `active` is false nothing moves, here or through promote()/demote(); the sweep records what
+ * it would do as dry_run decisions. A hold freezes placement: a held object is never promoted or demoted
+ * (refused). Only an object whose canonical copy is verified on record (present, checked, sha256 on record)
+ * is promoted, and a demotion never removes the last good copy: the canonical copy must check out first.
+ * Every decision (done, already, refused, failed, dry_run) goes to media_object_tier_decisions with its
+ * inputs and the policy in force, and counts in media_tier_decisions_24h{target="object"} on /metrics. A
+ * refusal or dry run the sweep would repeat is logged once a day per object (a repeated dry run does not use
+ * up the sweep's budget, so over a day the log shows what successive sweeps would move); an object whose move
+ * failed waits FAILED_BACKOFF_H hours before the sweep tries it again.
+ */
+'use strict';
+
+const fs = require('fs');
+const crypto = require('crypto');
+const db = require('../db/database');
+const drill = require('../drill');
+const policy = require('./tier-policy');
+const popularity = require('./popularity');
+
+const MB = 1024 * 1024;
+const FAILED_BACKOFF_H = 6;
+const OUTCOMES = ['done', 'already', 'refused', 'failed', 'dry_run'];
+
+const vodStorage = () => require('../vod/vod-storage');
+const model = () => require('./model');
+
+const sweepState = { running: false, lastRunAt: null, lastResult: null };
+
+function sha256File(p) {
+ return new Promise((resolve, reject) => {
+ const h = crypto.createHash('sha256');
+ const s = fs.createReadStream(p);
+ s.on('data', (d) => h.update(d));
+ s.on('end', () => resolve(h.digest('hex')));
+ s.on('error', reject);
+ });
+}
+
+/** SQLite 'YYYY-MM-DD HH:MM:SS' (UTC) or ISO → epoch ms (NaN when unreadable). */
+function msOf(t) {
+ const s = String(t || '');
+ return Date.parse(/[zZ]|[+-]\d\d:?\d\d$/.test(s) ? s : `${s.replace(' ', 'T')}Z`);
+}
+
+// ── The canonical copy ───────────────────────────────────────
+
+function canonicalLocation(obj, locs) {
+ return locs.find((l) => l.provider === (obj.canonical_provider || 'local')) || null;
+}
+
+/** Is the canonical copy verified on record (no I/O)? → null, or why not. */
+function canonicalUnverified(obj, locs) {
+ const l = canonicalLocation(obj, locs);
+ if (!l) return 'no canonical copy on record';
+ if (l.provider === 'r2') return 'the canonical copy is the R2 cache';
+ if (!obj.content_hash) return 'no sha256 on record to check a copy against (the object.hash job has not reached it)';
+ if (l.state !== 'present' || !l.verified_at) return `the canonical ${l.provider} copy is ${l.state}${l.verified_at ? '' : ' and was never checked'}`;
+ if (l.checksum && l.checksum !== obj.content_hash) return `the canonical ${l.provider} copy's checksum contradicts the object's sha256`;
+ return null;
+}
+
+/**
+ * Check the canonical copy now: a local file must exist with the object's size and sha256; a B2 copy must
+ * answer a HEAD with the object's size. → { ok, why, source: { file } | { b2key } }
+ */
+async function confirmCanonical(obj, loc) {
+ if (!loc) return { ok: false, why: 'no canonical copy on record' };
+ if (!obj.content_hash) return { ok: false, why: 'no sha256 on record to check the canonical copy against' };
+ const size = Number(obj.size_bytes);
+ if (loc.provider === 'local') {
+ let st = null;
+ try { st = fs.statSync(loc.key); } catch { /* missing */ }
+ if (!st || !st.isFile()) return { ok: false, why: 'the canonical local file is missing' };
+ if (st.size !== size) return { ok: false, why: `the canonical local file is ${st.size} bytes, not ${size}` };
+ const sum = await sha256File(loc.key).catch(() => null);
+ if (sum !== obj.content_hash) return { ok: false, why: sum ? 'the canonical local file does not match the sha256 on record' : 'the canonical local file could not be read' };
+ return { ok: true, source: { file: loc.key } };
+ }
+ if (loc.provider === 'b2') {
+ const head = await vodStorage().headObject('b2', loc.key).catch((err) => ({ error: err.message }));
+ if (!head) return { ok: false, why: 'the canonical B2 copy is missing' };
+ if (head.error) return { ok: false, why: `the canonical B2 copy could not be checked: ${head.error}` };
+ if (head.size !== size) return { ok: false, why: `the canonical B2 copy is ${head.size} bytes, not ${size}` };
+ return { ok: true, source: { b2key: loc.key } };
+ }
+ return { ok: false, why: `a canonical copy in ${loc.provider} cannot be checked here` };
+}
+
+/** Where the R2 copy goes: the canonical key for a B2 canonical copy (copied key to key), else objects/<app>/<id>. */
+function r2KeyFor(obj, canon) {
+ return canon && canon.provider === 'b2' ? canon.key : `objects/${obj.app_id}/${obj.id}`;
+}
+
+// ── Decisions ────────────────────────────────────────────────
+
+function thresholdSnapshot(settings) {
+ const out = {};
+ for (const [k, t] of Object.entries(policy.thresholds(settings))) out[k] = { value: t.value, source: t.source };
+ return out;
+}
+
+/** What a decision saw: the object's popularity, size, lifecycle, copies, hold and the gate. */
+function inputsOf(obj, { locs, pop, held, settings }) {
+ const canon = canonicalLocation(obj, locs);
+ const r2 = locs.find((l) => l.provider === 'r2');
+ return {
+ unique_viewers_7d: pop ? pop.unique_viewers : 0,
+ last_viewed_day: pop ? pop.last_viewed_day : popularity.lastViewedDay(obj.id),
+ size_bytes: Number(obj.size_bytes) || 0,
+ kind: obj.kind, visibility: obj.visibility, lifecycle_status: obj.lifecycle_status,
+ content_hash: !!obj.content_hash,
+ canonical: canon ? { provider: canon.provider, state: canon.state, verified_at: canon.verified_at || null,
+ checksum_matches: !!(canon.checksum && obj.content_hash && canon.checksum === obj.content_hash) } : null,
+ held,
+ r2: r2 ? { state: r2.state, since: r2.created_at || null } : null,
+ gate_active: !!settings.active,
+ r2_available: vodStorage().providerAvailable('r2'),
+ };
+}
+
+/** Log one decision. Never throws: the move already happened (or did not). → the row id, or null */
+function recordDecision({ obj, action, from, to, outcome, trigger, reason, inputs, settings, error = null }) {
+ try {
+ return db.run(`INSERT INTO media_object_tier_decisions (object_id, app_id, action, from_provider, to_provider, outcome, trigger, reason, inputs, thresholds, error)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
+ [obj.id, obj.app_id || null, action, from || null, to || null, outcome, String(trigger || 'manual').slice(0, 40), String(reason || '').slice(0, 1000),
+ JSON.stringify(inputs || {}), JSON.stringify(thresholdSnapshot(settings)), error == null ? null : String(error).slice(0, 500)]).lastInsertRowid;
+ } catch (err) {
+ console.warn(`[ObjectTiers] decision for ${obj && obj.id} not logged: ${err.message}`);
+ return null;
+ }
+}
+
+/** The sweep logged the same outcome for this object and action within the last day. */
+function loggedToday(objectId, action, outcome) {
+ return !!db.get(`SELECT 1 AS x FROM media_object_tier_decisions WHERE object_id = ? AND action = ? AND outcome = ? AND trigger = 'sweep'
+ AND decided_at >= strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-1 day') LIMIT 1`, [objectId, action, outcome]);
+}
+
+/** A move of this object failed within the back-off window. */
+function inBackoff(objectId, action) {
+ return !!db.get(`SELECT 1 AS x FROM media_object_tier_decisions WHERE object_id = ? AND action = ? AND outcome = 'failed'
+ AND decided_at >= strftime('%Y-%m-%dT%H:%M:%fZ', 'now', ?) LIMIT 1`, [objectId, action, `-${FAILED_BACKOFF_H} hours`]);
+}
+
+/**
+ * Shared frame of promote()/demote(): loads the object and its facts, and returns log(outcome, reason, extra)
+ * which records the decision (the sweep's repeated refusals and dry runs once a day) and shapes the answer.
+ */
+function frame(objectId, action, ctx) {
+ const settings = ctx.settings || policy.settings();
+ const obj = model().getObject(String(objectId || ''));
+ if (!obj) return { missing: true };
+ const locs = model().listLocations(obj.id);
+ const pop = ctx.pop !== undefined ? ctx.pop : (popularity.stats({ ids: [obj.id], now: ctx.now }).get(obj.id) || null);
+ const held = model().isHeld(obj.id);
+ const inputs = inputsOf(obj, { locs, pop, held, settings });
+ const canon = canonicalLocation(obj, locs);
+ const trigger = ctx.trigger || 'manual';
+ const why = ctx.reason || 'requested';
+ const from = action === 'promote' ? (canon ? canon.provider : null) : 'r2';
+ const to = action === 'promote' ? 'r2' : (canon ? canon.provider : null);
+ const log = (outcome, reason, extra = {}) => {
+ const repeat = trigger === 'sweep' && (outcome === 'refused' || outcome === 'dry_run') && loggedToday(obj.id, action, outcome);
+ if (!repeat) recordDecision({ obj, action, from, to, outcome, trigger, reason, inputs, settings, error: extra.error });
+ return { ok: outcome === 'done' || outcome === 'already', outcome, object_id: obj.id, repeat, ...extra };
+ };
+ return { obj, locs, canon, held, settings, why, log };
+}
+
+/** Why an object may not be promoted at all, from the record (no I/O), or null. */
+function promoteRefusal(obj, { locs, held }) {
+ if (held) return 'under a retention hold (a hold freezes placement)';
+ if (obj.legacy_ref) return 'not a native object (projected objects tier with their VOD)';
+ if (obj.lifecycle_status !== 'ready') return `the object is ${obj.lifecycle_status}`;
+ if (db.isSandboxTenant(obj.app_id)) return 'developer sandbox objects are never tiered';
+ return canonicalUnverified(obj, locs);
+}
+
+// ── Moves ────────────────────────────────────────────────────
+
+/**
+ * Promote one native object to the R2 popularity cache and log the decision. ctx { trigger, reason, settings, pop }.
+ * → { ok, outcome: done | already | refused | failed | dry_run, error?, bytes? }
+ */
+async function promote(objectId, ctx = {}) {
+ const f = frame(objectId, 'promote', ctx);
+ if (f.missing) return { ok: false, outcome: 'refused', error: 'object not found' };
+ const { obj, locs, canon, held, settings, why, log } = f;
+ if (held) return log('refused', `${why}; refused: under a retention hold (a hold freezes placement)`, { error: 'held', held: true });
+ const r2Now = locs.find((l) => l.provider === 'r2');
+ if (r2Now && r2Now.state === 'present' && r2Now.verified_at) return log('already', `${why}; the R2 copy is already present`);
+ const refusal = promoteRefusal(obj, { locs, held });
+ if (refusal) return log('refused', `${why}; refused: ${refusal}`, { error: refusal });
+ const vs = vodStorage();
+ if (!settings.active) return log('dry_run', `gate off (active = false): would promote: ${why}${vs.providerAvailable('r2') ? '' : ' (R2 is not available now)'}`);
+ if (!vs.providerAvailable('r2')) return log('refused', `${why}; refused: R2 is not available`, { error: 'R2 not available' });
+
+ const check = await confirmCanonical(obj, canon);
+ if (!check.ok) return log('refused', `${why}; refused: ${check.why}`, { error: check.why });
+ const key = r2KeyFor(obj, canon);
+ const size = Number(obj.size_bytes);
+ try {
+ if (check.source.file) await vs.uploadFile('r2', key, check.source.file, obj.mime_type || 'application/octet-stream');
+ else await vs.copyBetweenProviders('b2', 'r2', key);
+ const head = await vs.headObject('r2', key);
+ if (!head || head.size !== size) throw new Error(`the R2 copy is ${head ? `${head.size} bytes` : 'missing'}, not ${size} bytes`);
+ const read = await vs.sha256Object('r2', key);
+ if (!read || read.sha256 !== obj.content_hash) throw new Error('the R2 copy read back does not match the sha256 on record');
+ } catch (err) {
+ await vs.deleteObject('r2', key).catch(() => {}); // never leave an unrecorded or unverified copy behind
+ return log('failed', `${why}; the copy to R2 did not verify`, { error: err.message });
+ }
+
+ // A hold placed (or a delete made) while the bytes were copied wins: the new copy was never recorded, so
+ // removing it moves nothing anyone could see.
+ const now = model().getObject(obj.id);
+ if (!now || model().isHeld(obj.id) || now.lifecycle_status !== 'ready') {
+ await vs.deleteObject('r2', key).catch(() => {});
+ return log('refused', `${why}; refused: the object was ${!now ? 'removed' : now.lifecycle_status !== 'ready' ? now.lifecycle_status : 'put under a hold'} during the copy (the copy was removed)`,
+ { error: 'changed during the copy' });
+ }
+ db.getDb().transaction(() => {
+ // A fresh row: its created_at is when the object entered R2 (the demotion's "in R2 at least that long").
+ db.run("DELETE FROM media_locations WHERE object_id = ? AND provider = 'r2'", [obj.id]);
+ model().upsertLocation(obj.id, { provider: 'r2', bucket: vs.bucketFor('r2'), key, storage_class: 'cache', state: 'present',
+ checksum: obj.content_hash, size_bytes: size, verified: true });
+ model().setLocationState(canon.id, { state: 'present', size_bytes: size });
+ })();
+ console.log(`[ObjectTiers] ${obj.id} (${obj.app_id}) promoted to R2: ${key} (${(size / MB).toFixed(1)} MB)`);
+ return log('done', why, { bytes: size });
+}
+
+/**
+ * Demote one native object from the R2 popularity cache (its R2 copy is deleted) and log the decision.
+ * ctx as for promote(). → { ok, outcome, error? }
+ */
+async function demote(objectId, ctx = {}) {
+ const f = frame(objectId, 'demote', ctx);
+ if (f.missing) return { ok: false, outcome: 'refused', error: 'object not found' };
+ const { obj, locs, canon, held, settings, why, log } = f;
+ if (held) return log('refused', `${why}; refused: under a retention hold (a hold freezes placement)`, { error: 'held', held: true });
+ const r2 = locs.find((l) => l.provider === 'r2');
+ if (!r2) return log('already', `${why}; there is no R2 copy`);
+ const vs = vodStorage();
+ if (!settings.active) return log('dry_run', `gate off (active = false): would demote: ${why}`);
+ if (!vs.providerConfigured('r2')) return log('refused', `${why}; refused: R2 is not configured`, { error: 'R2 not configured' });
+ if (!canon || canon.provider === 'r2') return log('refused', `${why}; refused: no canonical copy besides R2 (the R2 copy is kept)`, { error: 'no canonical copy' });
+
+ // The last good copy is never the one removed: the canonical copy must check out now.
+ const check = await confirmCanonical(obj, canon);
+ if (!check.ok) return log('refused', `${why}; refused: ${check.why}; the R2 copy is kept (it may be the last good copy)`, { error: check.why });
+ try {
+ await vs.deleteObject('r2', r2.key);
+ const still = await vs.headObject('r2', r2.key);
+ if (still) throw new Error('the R2 copy is still there after the delete');
+ } catch (err) {
+ return log('failed', `${why}; the R2 copy could not be removed`, { error: err.message });
+ }
+ db.getDb().transaction(() => {
+ db.run('DELETE FROM media_locations WHERE id = ?', [r2.id]);
+ model().setLocationState(canon.id, { state: 'present', size_bytes: Number(obj.size_bytes) });
+ })();
+ console.log(`[ObjectTiers] ${obj.id} (${obj.app_id}) demoted from R2 (${why})`);
+ return log('done', why);
+}
+
+// ── Candidates ───────────────────────────────────────────────
+
+/** Ready native objects that pass the popularity, recency and size thresholds and have no R2 copy yet, most viewed first. */
+function promotionCandidates(settings, { now = Date.now(), appId = null, limit = 200 } = {}) {
+ const today = popularity.dayOf(now);
+ return db.all(`SELECT o.*, p.viewers AS unique_viewers_7d, p.last_day AS last_viewed_day
+ FROM (SELECT object_id, SUM(unique_viewers) AS viewers, MAX(day) AS last_day FROM media_object_views_daily
+ WHERE day >= ? AND unique_viewers > 0 GROUP BY object_id) p
+ JOIN media_objects o ON o.id = p.object_id
+ WHERE o.legacy_ref IS NULL AND o.lifecycle_status = 'ready'
+ AND p.viewers >= ? AND p.last_day >= ? AND o.size_bytes >= ? AND o.size_bytes <= ?
+ AND NOT EXISTS (SELECT 1 FROM media_locations l WHERE l.object_id = o.id AND l.provider = 'r2' AND l.state = 'present')${appId ? ' AND o.app_id = ?' : ''}
+ ORDER BY p.viewers DESC, o.id LIMIT ?`,
+ [popularity.windowStart(today, popularity.WINDOW_DAYS), settings.promoteMinUniqueViewers7d, popularity.windowStart(today, settings.promoteRecentAccessDays),
+ Math.ceil(settings.promoteMinSizeMb * MB), Math.floor(settings.promoteMaxSizeMb * MB), ...(appId ? [appId] : []), limit]);
+}
+
+/** Native objects with an R2 copy that should lose it: not ready any more, or idle. → [{ obj row, reason }] */
+function demotionCandidates(settings, { now = Date.now(), appId = null } = {}) {
+ const idleFrom = popularity.windowStart(popularity.dayOf(now), settings.demoteIdleDays);
+ const rows = db.all(`SELECT o.*, l.created_at AS r2_since,
+ (SELECT MAX(d.day) FROM media_object_views_daily d WHERE d.object_id = o.id AND d.unique_viewers > 0) AS last_viewed_day
+ FROM media_locations l JOIN media_objects o ON o.id = l.object_id
+ WHERE l.provider = 'r2' AND o.legacy_ref IS NULL${appId ? ' AND o.app_id = ?' : ''}
+ ORDER BY (o.lifecycle_status = 'ready'), o.id`, appId ? [appId] : []);
+ const out = [];
+ for (const r of rows) {
+ if (r.lifecycle_status !== 'ready') { out.push({ row: r, reason: `the object is ${r.lifecycle_status}` }); continue; }
+ const idle = !r.last_viewed_day || r.last_viewed_day < idleFrom;
+ const since = msOf(r.r2_since);
+ const settled = !Number.isFinite(since) || now - since >= settings.demoteIdleDays * 864e5;
+ if (idle && settled) {
+ out.push({ row: r, reason: `no viewer since ${r.last_viewed_day || 'the kept counts began'} (idle for demoteIdleDays ${settings.demoteIdleDays}); in R2 since ${r.r2_since || 'unknown'}` });
+ }
+ }
+ return out;
+}
+
+function promoteReason(c, s) {
+ return `${c.unique_viewers_7d} unique viewers in 7 days >= promoteMinUniqueViewers7d ${s.promoteMinUniqueViewers7d}; last viewed ${c.last_viewed_day} `
+ + `(within promoteRecentAccessDays ${s.promoteRecentAccessDays}); ${(Number(c.size_bytes) / MB).toFixed(1)} MB (promoteMinSizeMb ${s.promoteMinSizeMb} to promoteMaxSizeMb ${s.promoteMaxSizeMb})`;
+}
+
+/** What the policy would promote now, each with what (on record) would refuse it; database only. */
+function candidates({ appId = null, limit = 20, now = Date.now(), settings = policy.settings() } = {}) {
+ return promotionCandidates(settings, { now, appId, limit: Math.min(Math.max(parseInt(limit, 10) || 20, 1), 200) }).map((c) => ({
+ object_id: c.id, app_id: c.app_id, kind: c.kind, size_bytes: c.size_bytes, unique_viewers_7d: c.unique_viewers_7d, last_viewed_day: c.last_viewed_day,
+ blocked_by: promoteRefusal(c, { locs: model().listLocations(c.id), held: model().isHeld(c.id) }),
+ }));
+}
+
+// ── The sweep ────────────────────────────────────────────────
+
+function tally(out, r) {
+ if (r.repeat) out.repeats++;
+ else if (r.outcome === 'done') out[r.action === 'demote' ? 'demoted' : 'promoted']++;
+ else if (r.outcome === 'dry_run') out[r.action === 'demote' ? 'would_demote' : 'would_promote']++;
+ else out[r.outcome] = (out[r.outcome] || 0) + 1;
+ if (r.outcome === 'failed') out.errors.push({ object_id: r.object_id, action: r.action, error: r.error });
+}
+
+/**
+ * One pass over native objects. → { gate, promoted, demoted, would_promote, would_demote, already, refused, failed,
+ * repeats, skipped_backoff, candidates: { promote, demote }, rotated, errors }
+ */
+async function runSweep({ trigger = 'sweep', now = Date.now() } = {}) {
+ if (drill.enabled) return { skipped: true, reason: 'restore drill (MEDIA_DRILL)' };
+ if (sweepState.running) return { skipped: true, reason: 'already running' };
+ sweepState.running = true;
+ try {
+ const settings = policy.settings();
+ const out = { gate: !!settings.active, promoted: 0, demoted: 0, would_promote: 0, would_demote: 0, already: 0, refused: 0, failed: 0,
+ repeats: 0, skipped_backoff: 0, candidates: { promote: 0, demote: 0 }, rotated: popularity.rotate({ now }), errors: [] };
+
+ // Demotions first: they free the R2 cache, and a deleted object's copy should not wait behind promotions.
+ const demote_ = demotionCandidates(settings, { now });
+ out.candidates.demote = demote_.length;
+ let budget = settings.maxDemotionsPerSweep;
+ for (const { row, reason } of demote_) {
+ if (budget <= 0) break;
+ if (inBackoff(row.id, 'demote')) { out.skipped_backoff++; continue; }
+ const r = await demote(row.id, { trigger, reason, settings, now });
+ tally(out, { ...r, action: 'demote' });
+ if (!r.repeat && ['done', 'failed', 'dry_run'].includes(r.outcome)) budget--;
+ }
+
+ const promote_ = promotionCandidates(settings, { now, limit: Math.max(50, settings.maxPromotionsPerSweep * 10) })
+ .filter((c) => !db.isSandboxTenant(c.app_id));
+ out.candidates.promote = promote_.length;
+ budget = settings.maxPromotionsPerSweep;
+ for (const c of promote_) {
+ if (budget <= 0) break;
+ if (inBackoff(c.id, 'promote')) { out.skipped_backoff++; continue; }
+ const r = await promote(c.id, { trigger, reason: promoteReason(c, settings), settings, now,
+ pop: { unique_viewers: c.unique_viewers_7d, last_viewed_day: c.last_viewed_day } });
+ tally(out, { ...r, action: 'promote' });
+ if (!r.repeat && ['done', 'failed', 'dry_run'].includes(r.outcome)) budget--;
+ }
+
+ if (out.promoted || out.demoted || out.failed) {
+ console.log(`[ObjectTiers] Sweep: ${out.promoted} promoted to R2, ${out.demoted} demoted, ${out.failed} failed, ${out.refused} refused`);
+ }
+ if (!out.errors.length) delete out.errors;
+ out.timestamp = new Date(now).toISOString();
+ sweepState.lastRunAt = Date.now();
+ sweepState.lastResult = out;
+ return out;
+ } catch (err) {
+ console.error('[ObjectTiers] Sweep error:', err.message);
+ return { error: err.message };
+ } finally {
+ sweepState.running = false;
+ }
+}
+
+// ── Operator views ───────────────────────────────────────────
+
+function decisionPublic(r) {
+ const parse = (v) => { try { return JSON.parse(v || '{}'); } catch { return {}; } };
+ return {
+ id: r.id, decided_at: r.decided_at, object_id: r.object_id, app_id: r.app_id || null, action: r.action,
+ from_provider: r.from_provider, to_provider: r.to_provider, outcome: r.outcome, trigger: r.trigger, reason: r.reason,
+ inputs: parse(r.inputs), thresholds: parse(r.thresholds), error: r.error || null,
+ };
+}
+
+/** Decisions of the last 24 hours by action and outcome (appId narrows them). */
+function counts24h(appId = null) {
+ const empty = () => Object.fromEntries(OUTCOMES.map((o) => [o, 0]));
+ const out = { promote: empty(), demote: empty() };
+ for (const r of db.all(`SELECT action, outcome, COUNT(*) AS n FROM media_object_tier_decisions
+ WHERE decided_at >= strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-1 day')${appId ? ' AND app_id = ?' : ''} GROUP BY action, outcome`, appId ? [appId] : [])) {
+ if (out[r.action]) out[r.action][r.outcome] = r.n;
+ }
+ return out;
+}
+
+/** A page of decisions, newest first: { decisions, next_before_id }. */
+function listDecisions({ appId = null, objectId = null, action = null, outcome = null, beforeId = null, limit = 50 } = {}) {
+ const conds = ['1 = 1'], params = [];
+ if (appId) { conds.push('app_id = ?'); params.push(appId); }
+ if (objectId) { conds.push('object_id = ?'); params.push(objectId); }
+ if (action) { conds.push('action = ?'); params.push(action); }
+ if (outcome) { conds.push('outcome = ?'); params.push(outcome); }
+ if (beforeId != null) { conds.push('id < ?'); params.push(Number(beforeId) || 0); }
+ const n = Math.min(Math.max(parseInt(limit, 10) || 50, 1), 200);
+ const rows = db.all(`SELECT * FROM media_object_tier_decisions WHERE ${conds.join(' AND ')} ORDER BY id DESC LIMIT ?`, [...params, n + 1]);
+ const page = rows.slice(0, n);
+ return { decisions: page.map(decisionPublic), next_before_id: rows.length > n ? page[page.length - 1].id : null, limit: n };
+}
+
+/** The operator views' part (server/me/ops.js): database only, no provider or file. */
+function report({ appId = null, limit = 20 } = {}) {
+ const settings = policy.settings();
+ const s = appId ? { sql: ' AND o.app_id = ?', params: [appId] } : { sql: '', params: [] };
+ const r2 = db.get(`SELECT COUNT(*) AS n, COALESCE(SUM(o.size_bytes), 0) AS b FROM media_locations l JOIN media_objects o ON o.id = l.object_id
+ WHERE l.provider = 'r2' AND o.legacy_ref IS NULL AND l.state = 'present'${s.sql}`, s.params);
+ const eligible = promotionCandidates(settings, { appId, limit: 100000 }).filter((c) => !db.isSandboxTenant(c.app_id)).length;
+ const n = Math.min(Math.max(parseInt(limit, 10) || 20, 1), 50);
+ const last = sweepState.lastResult;
+ return {
+ gate: { active: !!settings.active, source: policy.thresholds(settings).active.source },
+ policy: Object.fromEntries(Object.keys(policy.DEFAULTS).filter((k) => k !== 'active').map((k) => [k, settings[k]])),
+ r2_copies: { count: r2.n, bytes: r2.b },
+ eligible_to_promote: eligible,
+ decisions_24h: counts24h(appId),
+ recent: listDecisions({ appId, limit: n }).decisions.map((d) => ({ id: d.id, decided_at: d.decided_at, app_id: d.app_id, object_id: d.object_id,
+ action: d.action, outcome: d.outcome, trigger: d.trigger, reason: String(d.reason || '').slice(0, 300), error: d.error ? String(d.error).slice(0, 300) : null })),
+ sweep: {
+ last_run_at: sweepState.lastRunAt ? new Date(sweepState.lastRunAt).toISOString() : null,
+ last_result: last ? { gate: last.gate, promoted: last.promoted, demoted: last.demoted, would_promote: last.would_promote, would_demote: last.would_demote,
+ refused: last.refused, failed: last.failed, error: last.error || null } : null,
+ },
+ };
+}
+
+module.exports = {
+ FAILED_BACKOFF_H, OUTCOMES,
+ runSweep, promote, demote, candidates, promotionCandidates, demotionCandidates,
+ canonicalUnverified, confirmCanonical, r2KeyFor,
+ counts24h, listDecisions, decisionPublic, report,
+};
@@ -56,10 +56,13 @@ function storageDirs(config) {
@@ -56,10 +56,13 @@ function storageDirs(config) {
function domainMetrics(registry, { db, recorder, events }) {
function domainMetrics(registry, { db, recorder, events }) {
registry.gauge({ name: 'media_recordings_in_progress', help: 'Recordings ffmpeg is writing right now', collect: () => recorder.activeCount() });
registry.gauge({ name: 'media_recordings_in_progress', help: 'Recordings ffmpeg is writing right now', collect: () => recorder.activeCount() });
+ // target vod: media_tier_decisions (VODs); object: media_object_tier_decisions (native v2 objects), whose
+ // outcome dry_run is what the object sweep would have done while its activation gate is off.
registry.gauge({
registry.gauge({
- name: 'media_tier_decisions_24h', help: 'R2 promotions and demotions decided in the last 24 hours, by action and outcome (media_tier_decisions)', labelNames: ['action', 'outcome'],
- collect: () => db.all(`SELECT action, outcome, COUNT(*) AS n FROM media_tier_decisions WHERE decided_at >= strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-1 day')
- GROUP BY action, outcome`).map((r) => ({ labels: { action: r.action, outcome: r.outcome }, value: r.n })),
+ name: 'media_tier_decisions_24h', help: 'R2 promotions and demotions decided in the last 24 hours, by target (vod, object), action and outcome', labelNames: ['target', 'action', 'outcome'],
+ collect: () => [['vod', 'media_tier_decisions'], ['object', 'media_object_tier_decisions']].flatMap(([target, table]) => db.all(`SELECT action, outcome, COUNT(*) AS n
+ FROM ${table} WHERE decided_at >= strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-1 day') GROUP BY action, outcome`)
+ .map((r) => ({ labels: { target, action: r.action, outcome: r.outcome }, value: r.n }))),
});
});
registry.gauge({
registry.gauge({
name: 'media_object_locations', help: 'Stored copies of media objects by provider and verification state', labelNames: ['provider', 'state'],
name: 'media_object_locations', help: 'Stored copies of media objects by provider and verification state', labelNames: ['provider', 'state'],
@@ -25,6 +25,7 @@
@@ -25,6 +25,7 @@
* low-water mark regardless of the popularity thresholds
* low-water mark regardless of the popularity thresholds
* 2. promotes popular VODs to R2
* 2. promotes popular VODs to R2
* 3. demotes stale R2 VODs back to B2-only
* 3. demotes stale R2 VODs back to B2-only
+ * 4. tiers native v2 objects (server/objects/tiering.js: its own policy and activation gate)
*/
*/
'use strict';
'use strict';
@@ -219,6 +220,11 @@ function clientFor(name) {
@@ -219,6 +220,11 @@ function clientFor(name) {
return clients[name];
return clients[name];
}
}
+/** Configured, and not known to be down (the last HeadBucket or readiness probe did not fail). */
+function providerAvailable(name) {
+ return providerConfigured(name) && providerHealthy[name] !== false;
+}
+
function bucketFor(name) {
function bucketFor(name) {
return (PROVIDER_ENV[name] && PROVIDER_ENV[name].bucket) || null;
return (PROVIDER_ENV[name] && PROVIDER_ENV[name].bucket) || null;
}
}
@@ -345,6 +351,27 @@ async function copyBetweenProviders(srcProvider, dstProvider, key) {
@@ -345,6 +351,27 @@ async function copyBetweenProviders(srcProvider, dstProvider, key) {
return { size: head.size };
return { size: head.size };
}
}
+/**
+ * Read one stored object through and hash it: { sha256, size }, or null when it is not there. Used to check a
+ * copy byte for byte after it was written (R2 egress is free; the object tiering bounds the size it reads).
+ */
+async function sha256Object(provider, key) {
+ const client = clientFor(provider);
+ if (!client) throw new Error(`Provider ${provider} not configured`);
+ loadSdk();
+ let obj;
+ try {
+ obj = await client.send(new S3.GetObjectCommand({ Bucket: PROVIDER_ENV[provider].bucket, Key: key }));
+ } catch (err) {
+ if (err?.$metadata?.httpStatusCode === 404 || err?.name === 'NoSuchKey' || err?.name === 'NotFound') return null;
+ throw err;
+ }
+ const hash = require('crypto').createHash('sha256');
+ let size = 0;
+ for await (const chunk of obj.Body) { hash.update(chunk); size += chunk.length; }
+ return { sha256: hash.digest('hex'), size };
+}
+
async function deleteObject(provider, key) {
async function deleteObject(provider, key) {
const client = clientFor(provider);
const client = clientFor(provider);
if (!client) return;
if (!client) return;
@@ -417,18 +444,23 @@ async function listMultipartUploads(provider, prefix = '') {
@@ -417,18 +444,23 @@ async function listMultipartUploads(provider, prefix = '') {
return out;
return out;
}
}
-async function presignGet(provider, key, expiresInSeconds = 900) {
+/**
+ * A presigned GET URL. `overrides` { contentType, contentDisposition } become the response headers the store
+ * answers with (the object tiering passes what /o/:id would have sent, so an attachment stays one).
+ */
+async function presignGet(provider, key, expiresInSeconds = 900, overrides = {}) {
const client = clientFor(provider);
const client = clientFor(provider);
if (!client) return null;
if (!client) return null;
loadSdk();
loadSdk();
// Force the response MIME so the browser plays the media even when the stored
// Force the response MIME so the browser plays the media even when the stored
// object metadata is generic (legacy clips were uploaded as octet-stream).
// object metadata is generic (legacy clips were uploaded as octet-stream).
const ext = path.extname(key || '').toLowerCase();
const ext = path.extname(key || '').toLowerCase();
- const mime = { '.webm': 'video/webm', '.mp4': 'video/mp4', '.mkv': 'video/x-matroska', '.jpg': 'image/jpeg', '.jpeg': 'image/jpeg', '.png': 'image/png', '.webp': 'image/webp' }[ext];
+ const mime = overrides.contentType || { '.webm': 'video/webm', '.mp4': 'video/mp4', '.mkv': 'video/x-matroska', '.jpg': 'image/jpeg', '.jpeg': 'image/jpeg', '.png': 'image/png', '.webp': 'image/webp' }[ext];
return Presigner.getSignedUrl(client, new S3.GetObjectCommand({
return Presigner.getSignedUrl(client, new S3.GetObjectCommand({
Bucket: PROVIDER_ENV[provider].bucket,
Bucket: PROVIDER_ENV[provider].bucket,
Key: key,
Key: key,
...(mime ? { ResponseContentType: mime } : {}),
...(mime ? { ResponseContentType: mime } : {}),
+ ...(overrides.contentDisposition ? { ResponseContentDisposition: overrides.contentDisposition } : {}),
}), { expiresIn: expiresInSeconds });
}), { expiresIn: expiresInSeconds });
}
}
@@ -1067,6 +1099,11 @@ async function runSweep() {
@@ -1067,6 +1099,11 @@ async function runSweep() {
}
}
}
}
+ // 4) Native v2 objects (server/objects/tiering.js): their own policy (media.object_tier) and activation
+ // gate, after the VODs; not while the disk still needs a drain (a promotion frees no local space).
+ const objectsResult = stillNeedsDrain ? { skipped: true, reason: 'disk pressure: the drain goes first' }
+ : await require('../objects/tiering').runSweep({ trigger: 'sweep' }).catch((err) => ({ error: err.message }));
+
const summary = {
const summary = {
checked: candidates.length,
checked: candidates.length,
migrated,
migrated,
@@ -1082,6 +1119,7 @@ async function runSweep() {
@@ -1082,6 +1119,7 @@ async function runSweep() {
diskPct: diskAfter.usePct,
diskPct: diskAfter.usePct,
freeGb: Number((diskAfter.available / GB).toFixed(1)),
freeGb: Number((diskAfter.available / GB).toFixed(1)),
errors: errors.length ? errors : undefined,
errors: errors.length ? errors : undefined,
+ objects: objectsResult,
timestamp: new Date().toISOString(),
timestamp: new Date().toISOString(),
};
};
sweepState.lastRunAt = Date.now();
sweepState.lastRunAt = Date.now();
@@ -1422,9 +1460,13 @@ module.exports = {
@@ -1422,9 +1460,13 @@ module.exports = {
providerOf,
providerOf,
isRemote,
isRemote,
providerConfigured,
providerConfigured,
+ providerAvailable,
bucketFor,
bucketFor,
endpointFor,
endpointFor,
headObject,
headObject,
+ uploadFile,
+ copyBetweenProviders,
+ sha256Object,
listObjects,
listObjects,
listMultipartUploads,
listMultipartUploads,
keyForVod,
keyForVod,
new file mode 100644
new file mode 100644
@@ -0,0 +1,473 @@
@@ -0,0 +1,473 @@
+'use strict';
+// Tiering of native v2 objects (roadmap WS-G task 11): popularity counted per object and UTC day without an
+// IP address or subject id (server/objects/popularity.js), rolled into a 7-day figure; the revisioned policy
+// media.object_tier with its activation gate off by default (server/objects/tier-policy.js); and the sweep
+// and moves (server/objects/tiering.js): dry runs while the gate is off, thresholds, holds, the verified
+// canonical copy, promotion (copy, then size and sha256 of the R2 copy) and demotion (never the last good
+// copy), every decision logged and counted, the admin API, the operator views and /o/:id playing from R2.
+// A fake S3 (B2 and R2 buckets) is spoken to by the real AWS SDK through the storage engine.
+const assert = require('assert');
+const fs = require('fs');
+const os = require('os');
+const path = require('path');
+const http = require('http');
+const crypto = require('crypto');
+
+const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-objtier-'));
+const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+const sha256 = (b) => crypto.createHash('sha256').update(b).digest('hex');
+
+const buckets = { 'b2-bucket': new Map(), 'r2-bucket': new Map() };
+const fault = { refusePut: null, corruptPut: null, keepOnDelete: null };
+function decodeAwsChunked(buf) {
+ const out = [];
+ let i = 0;
+ for (;;) {
+ const nl = buf.indexOf('\r\n', i);
+ const size = parseInt(buf.subarray(i, nl).toString().split(';')[0], 16);
+ if (!size) break;
+ out.push(buf.subarray(nl + 2, nl + 2 + size));
+ i = nl + 2 + size + 2;
+ }
+ return Buffer.concat(out);
+}
+const s3 = http.createServer((req, res) => {
+ const chunks = [];
+ req.on('data', (c) => chunks.push(c));
+ req.on('end', () => {
+ const u = new URL(req.url, 'http://x');
+ const [, bucket, ...rest] = decodeURIComponent(u.pathname).split('/');
+ const key = rest.join('/');
+ const store = buckets[bucket];
+ if (!store) { res.writeHead(404); return res.end(); }
+ if (!key && req.method === 'HEAD') { res.writeHead(200); return res.end(); }
+ const obj = store.get(key);
+ if (req.method === 'HEAD' || req.method === 'GET') {
+ if (!obj) { res.writeHead(404, { 'Content-Type': 'application/xml' }); return res.end(req.method === 'GET' ? '<Error><Code>NoSuchKey</Code></Error>' : undefined); }
+ res.writeHead(200, { 'Content-Length': obj.length, ETag: '"e"', 'Content-Type': 'application/octet-stream' });
+ return res.end(req.method === 'GET' ? obj : undefined);
+ }
+ if (req.method === 'PUT') {
+ if (fault.refusePut === bucket) { res.writeHead(500, { 'Content-Type': 'application/xml' }); return res.end('<Error><Code>InternalError</Code></Error>'); }
+ let body = Buffer.concat(chunks);
+ if (req.headers['x-amz-decoded-content-length'] || /aws-chunked/.test(req.headers['content-encoding'] || '')) body = decodeAwsChunked(body);
+ if (fault.corruptPut === bucket) { body = Buffer.from(body); body[0] ^= 0xff; } // same size, other bytes
+ store.set(key, body);
+ res.writeHead(200, { ETag: '"e"' });
+ return res.end();
+ }
+ if (req.method === 'DELETE') { if (fault.keepOnDelete !== bucket) store.delete(key); res.writeHead(204); return res.end(); }
+ res.writeHead(405); res.end();
+ });
+});
+
+const BROWSER = 'Mozilla/5.0 (X11; Linux x86_64; rv:120.0) Gecko/20100101 Firefox/120.0';
+const SAFARI = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 14_0) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15';
+
+(async () => {
+ await new Promise((r) => s3.listen(0, '127.0.0.1', r));
+ const endpoint = `http://127.0.0.1:${s3.address().port}`;
+ Object.assign(process.env, {
+ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
+ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),
+ MEDIA_B2_ENDPOINT: endpoint, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_KEY_ID: 'k', MEDIA_B2_APP_KEY: 's', MEDIA_B2_REGION: 'us-west-004',
+ MEDIA_R2_ENDPOINT: endpoint, MEDIA_R2_BUCKET: 'r2-bucket', MEDIA_R2_ACCESS_KEY_ID: 'k', MEDIA_R2_SECRET_ACCESS_KEY: 's',
+ AWS_REQUEST_CHECKSUM_CALCULATION: 'WHEN_REQUIRED', AWS_RESPONSE_CHECKSUM_VALIDATION: 'WHEN_REQUIRED',
+ });
+ const db = require('../server/db/database');
+ const model = require('../server/objects/model');
+ const storage = require('../server/vod/vod-storage');
+ const popularity = require('../server/objects/popularity');
+ const policy = require('../server/objects/tier-policy');
+ const tiering = require('../server/objects/tiering');
+ db.upsertApp({ app_id: 'live', api_key: 'live-key-objtier' });
+ db.upsertApp({ app_id: 'games', api_key: 'games-key-objtier' });
+ // Keep the VOD sweep's disk-pressure drain out of this test (whatever this machine's disk looks like).
+ await storage.setSettings({ hotDiskPressurePct: 100, criticalDiskPct: 100, minFreeGb: 0 }, { reason: 'test: no pressure drain' });
+
+ const today = popularity.dayOf(Date.now());
+ const addObject = ({ app = 'live', size = 4096, mime = 'application/octet-stream', verified = true, hash = true, canonical = 'local' } = {}) => {
+ const bytes = crypto.randomBytes(size);
+ const sum = sha256(bytes);
+ const id = model.createObject({ app_id: app, kind: 'file', visibility: 'public', lifecycle_status: 'ready', mime_type: mime, size_bytes: size,
+ content_hash: hash ? sum : null, canonical_provider: canonical, metadata: { filename: 'x.bin' } });
+ let file = null;
+ if (canonical === 'local') {
+ file = path.join(process.env.OBJECTS_PATH, app, id);
+ fs.mkdirSync(path.dirname(file), { recursive: true });
+ fs.writeFileSync(file, bytes);
+ model.updateObject(id, { canonical_key: file });
+ model.upsertLocation(id, { provider: 'local', key: file, state: verified ? 'present' : 'pending', size_bytes: size, checksum: hash ? sum : null, verified });
+ } else {
+ const key = `objects/${app}/${id}`;
+ buckets['b2-bucket'].set(key, bytes);
+ model.updateObject(id, { canonical_key: key });
+ model.upsertLocation(id, { provider: 'b2', bucket: 'b2-bucket', key, state: 'present', size_bytes: size, verified: true });
+ }
+ return { id, bytes, sum, file, app };
+ };
+ const setViews = (id, entries) => {
+ db.run('DELETE FROM media_object_views_daily WHERE object_id = ?', [id]);
+ for (const [offset, n] of entries) {
+ db.run('INSERT INTO media_object_views_daily (object_id, day, unique_viewers, last_viewed_at) VALUES (?, ?, ?, ?)',
+ [id, popularity.addDays(today, offset), n, `${popularity.addDays(today, offset)}T12:00:00.000Z`]);
+ }
+ };
+ const r2Row = (id) => db.get("SELECT * FROM media_locations WHERE object_id = ? AND provider = 'r2'", [id]);
+ const ageR2 = (id, days) => db.run("UPDATE media_locations SET created_at = datetime('now', ?) WHERE object_id = ? AND provider = 'r2'", [`-${days} days`, id]);
+ const decisions = (id) => db.all('SELECT * FROM media_object_tier_decisions WHERE object_id = ? ORDER BY id', [id]);
+ const last = (id) => { const d = decisions(id); return d[d.length - 1]; };
+ const fake = (ip, headers = {}, method = 'GET') => ({ method, ip, path: '/o/x', url: '/o/x', headers: { 'user-agent': BROWSER, ...headers } });
+
+ // ── 1. Unique viewers per object and day, with no IP address or subject id stored ──
+ const seen = addObject();
+ const rec = (ip, headers, method) => popularity.record(model.getObject(seen.id), fake(ip, headers, method));
+ assert.deepStrictEqual(rec('203.0.113.5'), { counted: true, reason: null });
+ assert.strictEqual(rec('203.0.113.5', { 'user-agent': SAFARI }).reason, 'seen', 'one network is one viewer, whatever its user agent');
+ popularity._reset(); // a restart: the same day's salt comes back from the database
+ assert.strictEqual(rec('203.0.113.5').reason, 'seen', 'and still once after a restart');
+ assert.strictEqual(rec('198.51.100.7').counted, true);
+ assert.strictEqual(rec('::ffff:198.51.100.7').reason, 'seen', 'an IPv4-mapped address is the IPv4 address');
+ assert.strictEqual(rec('2001:db8:1:2::10').counted, true);
+ assert.strictEqual(rec('2001:db8:1:2:aaaa:bbbb:cccc:1').reason, 'seen', 'an IPv6 /64 is one viewer');
+ assert.strictEqual(rec('2001:db8:1:3::10').counted, true);
+ assert.strictEqual(rec('192.0.2.10', { range: 'bytes=5000-' }).reason, 'range', 'a seek is not a view');
+ assert.strictEqual(rec('192.0.2.10', { range: 'bytes=0-' }).counted, true);
+ assert.strictEqual(rec('192.0.2.11', {}, 'HEAD').reason, 'method');
+ assert.strictEqual(rec('192.0.2.12', { 'sec-gpc': '1' }).reason, 'opted_out');
+ assert.strictEqual(rec('192.0.2.13', { dnt: '1' }).reason, 'opted_out');
+ assert.strictEqual(rec('192.0.2.14', { 'user-agent': 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' }).reason, 'bot');
+ assert.strictEqual(rec('192.0.2.15', { 'user-agent': '' }).reason, 'bot');
+ const projected = model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', legacy_ref: 'legacy:live:file:abc' });
+ assert.strictEqual(popularity.record(model.getObject(projected), fake('192.0.2.16')).reason, 'not_native');
+ const todayRow = db.get('SELECT * FROM media_object_views_daily WHERE object_id = ? AND day = ?', [seen.id, today]);
+ assert.strictEqual(todayRow.unique_viewers, 5);
+ assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_object_viewer_days WHERE object_id = ?', [seen.id]).n, 5, 'an opted-out, bot, HEAD or seek request makes no hash at all');
+ const stored = JSON.stringify([db.all('SELECT * FROM media_object_viewer_days'), db.all('SELECT * FROM media_object_views_daily'), db.all('SELECT day FROM media_object_view_salts')]);
+ for (const ip of ['203.0.113.5', '198.51.100.7', '2001:db8', '192.0.2.10', 'usr_']) assert.ok(!stored.includes(ip), `nothing stored names ${ip}`);
+ for (const v of db.all('SELECT viewer FROM media_object_viewer_days').map((r) => r.viewer)) {
+ assert.ok(/^[0-9a-f]{16}$/.test(v));
+ assert.notStrictEqual(v, sha256('203.0.113.5').slice(0, 16), 'keyed by the salt of the day, not a plain hash of the address');
+ }
+ assert.deepStrictEqual(['10.1.2.3', '::ffff:10.1.2.3', '2001:DB8::1', '2001:db8:0:0:ffff::9', 'fe80::1%eth0'].map(popularity.networkOf),
+ ['10.1.2.3', '10.1.2.3', '2001:db8:0:0::/64', '2001:db8:0:0::/64', 'fe80:0:0:0::/64']);
+ console.log('✅ a viewer counts once per object and UTC day (one network, IPv6 by /64), never a seek, HEAD, bot or opted-out request, and no address is stored');
+
+ // ── 2. Hashes and salt die with their day; counts roll into a 7-day window, kept 30 days ──
+ const roll = addObject();
+ const yesterday = popularity.addDays(today, -1);
+ assert.strictEqual(popularity.record(model.getObject(roll.id), fake('203.0.113.9'), { now: Date.parse(`${yesterday}T12:00:00Z`) }).counted, true);
+ const oldHash = db.get('SELECT viewer FROM media_object_viewer_days WHERE object_id = ? AND day = ?', [roll.id, yesterday]).viewer;
+ assert.ok(db.get('SELECT 1 AS x FROM media_object_view_salts WHERE day = ?', [yesterday]));
+ assert.strictEqual(popularity.record(model.getObject(roll.id), fake('203.0.113.9')).counted, true, 'the same network on the next day is a new viewer-day');
+ assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_object_viewer_days WHERE day < ?', [today]).n, 0, 'the first view of a day deletes the finished days\' hashes');
+ assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_object_view_salts WHERE day < ?', [today]).n, 0, '...and their salts');
+ assert.notStrictEqual(db.get('SELECT viewer FROM media_object_viewer_days WHERE object_id = ? AND day = ?', [roll.id, today]).viewer, oldHash, 'a new salt every day: days cannot be linked');
+ assert.strictEqual(db.get('SELECT unique_viewers FROM media_object_views_daily WHERE object_id = ? AND day = ?', [roll.id, yesterday]).unique_viewers, 1, 'the count stays');
+ const rs = popularity.stats({ ids: [roll.id] }).get(roll.id);
+ assert.deepStrictEqual([rs.unique_viewers, rs.days_viewed, rs.last_viewed_day], [2, 2, today], 'two viewer-days');
+ const win = addObject();
+ setViews(win.id, [[-8, 100], [-6, 200], [0, 150]]);
+ assert.strictEqual(popularity.stats({ ids: [win.id] }).get(win.id).unique_viewers, 350, 'the last 7 UTC days, today included');
+ assert.strictEqual(popularity.stats({ ids: [win.id], days: 9 }).get(win.id).unique_viewers, 450);
+ db.run('INSERT INTO media_object_views_daily (object_id, day, unique_viewers) VALUES (?, ?, 1), (?, ?, 1)', [win.id, popularity.addDays(today, -30), win.id, popularity.addDays(today, -31)]);
+ const rot = popularity.rotate();
+ assert.strictEqual(rot.counts, 1);
+ assert.deepStrictEqual(db.all('SELECT day FROM media_object_views_daily WHERE object_id = ? ORDER BY day', [win.id]).map((r) => r.day),
+ [popularity.addDays(today, -30), popularity.addDays(today, -8), popularity.addDays(today, -6), today], 'daily counts are kept 30 days');
+ console.log('✅ a finished day\'s hashes and salt are deleted (counts stay); the 7-day window sums the daily counts; counts go after 30 days');
+
+ // ── 3. The policy: revisioned, validated, the gate off by default ──
+ assert.deepStrictEqual(policy.DEFAULTS, { active: false, promoteMinUniqueViewers7d: 500, promoteRecentAccessDays: 3, promoteMinSizeMb: 16,
+ promoteMaxSizeMb: 4096, maxPromotionsPerSweep: 3, demoteIdleDays: 14, maxDemotionsPerSweep: 10 });
+ assert.strictEqual(policy.settings().active, false);
+ const rev0 = policy.get().revision();
+ for (const [updates, why] of [
+ [{ promoteMinSizeMb: 10, promoteMaxSizeMb: 5 }, /promoteMinSizeMb must be below promoteMaxSizeMb/],
+ [{ demoteIdleDays: 3 }, /demoteIdleDays must be above promoteRecentAccessDays/],
+ [{ promoteMinUniqueViewers7d: 0 }, /must be >= 1/],
+ [{ active: 'yes' }, /must be boolean/],
+ [{ maxPromotionsPerSweep: 1.5 }, /must be integer/],
+ [{ demoteIdleDays: 31 }, /must be <= 30/],
+ [{ surprise: 1 }, /not a known key/],
+ ]) {
+ await assert.rejects(policy.set(updates, { reason: 'test' }), (e) => e.status === 422 && e.code === 'config.invalid' && why.test(e.message), JSON.stringify(updates));
+ }
+ assert.strictEqual(policy.get().revision(), rev0, 'a refused change changes nothing');
+ await policy.set({ promoteMinSizeMb: 0.001, promoteMaxSizeMb: 1 }, { actor: { type: 'service', id: 'live' }, reason: 'test: small objects' });
+ const th = policy.thresholds();
+ assert.deepStrictEqual([th.promoteMinSizeMb.source, th.promoteMinUniqueViewers7d.source, th.active.source], ['setting', 'default', 'default']);
+ assert.deepStrictEqual(th.promoteMinUniqueViewers7d, { value: 500, default: 500, source: 'default' });
+ console.log('✅ media.object_tier: gate off and 500 viewers by default; types, ranges, key order and unknown keys refused (422), nothing changed');
+
+ // ── 4. Gate off: the sweep moves nothing and records what it would do ──
+ db.run('DELETE FROM media_object_views_daily');
+ const pop = addObject();
+ setViews(pop.id, [[0, 600]]);
+ const cached = addObject(); // a copy in R2 from earlier, idle for a month
+ buckets['r2-bucket'].set(`objects/live/${cached.id}`, cached.bytes);
+ model.upsertLocation(cached.id, { provider: 'r2', bucket: 'r2-bucket', key: `objects/live/${cached.id}`, storage_class: 'cache', state: 'present', size_bytes: 4096, checksum: cached.sum, verified: true });
+ ageR2(cached.id, 30);
+ const r2Keys = buckets['r2-bucket'].size;
+ let s = await tiering.runSweep();
+ assert.deepStrictEqual([s.gate, s.promoted, s.demoted, s.would_promote, s.would_demote], [false, 0, 0, 1, 1], JSON.stringify(s));
+ assert.strictEqual(buckets['r2-bucket'].size, r2Keys, 'nothing was copied or deleted');
+ assert.ok(!r2Row(pop.id) && r2Row(cached.id));
+ let d = last(pop.id);
+ assert.deepStrictEqual([d.action, d.outcome, d.trigger, d.from_provider, d.to_provider], ['promote', 'dry_run', 'sweep', 'local', 'r2']);
+ assert.ok(/^gate off \(active = false\): would promote: 600 unique viewers in 7 days >= promoteMinUniqueViewers7d 500/.test(d.reason), d.reason);
+ const din = JSON.parse(d.inputs);
+ assert.deepStrictEqual([din.unique_viewers_7d, din.last_viewed_day, din.gate_active, din.held, din.canonical.provider, din.canonical.checksum_matches], [600, today, false, false, 'local', true]);
+ assert.deepStrictEqual(JSON.parse(d.thresholds).active, { value: false, source: 'default' });
+ assert.deepStrictEqual([last(cached.id).action, last(cached.id).outcome], ['demote', 'dry_run']);
+ s = await tiering.runSweep();
+ assert.deepStrictEqual([s.would_promote, s.would_demote, s.repeats], [0, 0, 2], 'a repeated dry run is logged once a day');
+ assert.strictEqual(decisions(pop.id).length, 1);
+ const manual = await tiering.promote(pop.id, { reason: 'asked by hand' });
+ assert.deepStrictEqual([manual.ok, manual.outcome], [false, 'dry_run'], 'the gate holds for a direct call too');
+ assert.ok(!r2Row(pop.id));
+ console.log('✅ gate off: dry_run decisions (with inputs and thresholds), once a day, and nothing moves, from the sweep or a direct call');
+
+ // ── 5. Thresholds: viewers, recency, window and size ──
+ const few = addObject(); setViews(few.id, [[0, 499]]);
+ const stale = addObject(); setViews(stale.id, [[-5, 900]]); // popular, but not viewed within 3 days
+ const outside = addObject(); setViews(outside.id, [[-7, 450], [0, 100]]); // day -7 is outside the 7-day window
+ const spread = addObject(); setViews(spread.id, [[-6, 250], [-3, 150], [0, 100]]);
+ const tiny = addObject({ size: 512 }); setViews(tiny.id, [[0, 900]]); // under promoteMinSizeMb (0.001 MB)
+ const huge = addObject({ size: 1100000 }); setViews(huge.id, [[0, 900]]); // over promoteMaxSizeMb (1 MB)
+ const projectId = 'prj_01J8Z3Q4X5Y6Z7A8B9C0D1E2F3';
+ db.ensureProjectTenant(projectId, 'sandbox', 0);
+ const sandboxed = addObject({ app: `${projectId}-sandbox` }); setViews(sandboxed.id, [[0, 900]]);
+ const ids = tiering.promotionCandidates(policy.settings()).map((c) => c.id);
+ assert.ok(ids.includes(pop.id) && ids.includes(spread.id), 'at 500 or more within the window');
+ for (const o of [few, stale, outside, tiny, huge]) assert.ok(!ids.includes(o.id), `not a candidate: ${o.id}`);
+ s = await tiering.runSweep();
+ assert.ok(!decisions(sandboxed.id).length, 'a sandbox object is never considered');
+ assert.strictEqual(tiering.candidates().find((c) => c.object_id === sandboxed.id).blocked_by, 'developer sandbox objects are never tiered');
+ assert.strictEqual(last(spread.id).outcome, 'dry_run');
+ for (const o of [few, stale, outside, tiny, huge]) assert.strictEqual(decisions(o.id).length, 0);
+ console.log('✅ thresholds: 7-day viewers (sum of the window), recent view, size bounds; sandbox tenants never tier');
+
+ // ── 6. Gate on: holds and an unverified canonical copy refuse; a released hold promotes ──
+ await policy.set({ active: true }, { actor: { type: 'service', id: 'live' }, reason: 'test: gate on' });
+ db.run('DELETE FROM media_object_views_daily');
+ const hot = addObject(); setViews(hot.id, [[0, 700]]);
+ const hold = model.placeHold({ object_id: hot.id, kind: 'admin', reason: 'test' });
+ const unchecked = addObject({ verified: false }); setViews(unchecked.id, [[0, 700]]);
+ const nohash = addObject({ hash: false }); setViews(nohash.id, [[0, 700]]);
+ const tampered = addObject(); setViews(tampered.id, [[0, 700]]);
+ const other = Buffer.from(tampered.bytes); other[10] ^= 0xff; fs.writeFileSync(tampered.file, other); // on record verified; on disk not
+ s = await tiering.runSweep();
+ assert.deepStrictEqual([s.gate, s.promoted, s.refused], [true, 0, 4], JSON.stringify(s));
+ assert.ok(/retention hold/.test(last(hot.id).reason) && JSON.parse(last(hot.id).inputs).held === true);
+ assert.ok(/never checked/.test(last(unchecked.id).reason), last(unchecked.id).reason);
+ assert.ok(/no sha256 on record/.test(last(nohash.id).reason), last(nohash.id).reason);
+ assert.ok(/does not match the sha256 on record/.test(last(tampered.id).reason), last(tampered.id).reason);
+ for (const o of [hot, unchecked, nohash, tampered]) { assert.ok(!r2Row(o.id)); assert.ok(!buckets['r2-bucket'].has(`objects/live/${o.id}`)); }
+ s = await tiering.runSweep();
+ assert.strictEqual(decisions(hot.id).length, 1, 'a repeated refusal is logged once a day');
+ model.releaseHold(hold.id, 'test');
+ s = await tiering.runSweep();
+ assert.strictEqual(s.promoted, 1);
+ console.log('✅ gate on: a hold, a canonical copy never checked, no sha256 or bytes that no longer match refuse the promotion; released, it promotes');
+
+ // ── 7. Promotion: copy, verify size and sha256, record; /o/:id plays from R2 ──
+ let r = r2Row(hot.id);
+ assert.deepStrictEqual([r.state, r.storage_class, r.key, r.checksum, r.size_bytes, !!r.verified_at], ['present', 'cache', `objects/live/${hot.id}`, hot.sum, 4096, true]);
+ assert.ok(buckets['r2-bucket'].get(`objects/live/${hot.id}`).equals(hot.bytes), 'the R2 copy is the same bytes');
+ assert.ok(fs.existsSync(hot.file) && db.get("SELECT state FROM media_locations WHERE object_id = ? AND provider = 'local'", [hot.id]).state === 'present', 'the canonical copy stays');
+ d = last(hot.id);
+ assert.deepStrictEqual([d.action, d.outcome, d.trigger, d.from_provider, d.to_provider, d.error], ['promote', 'done', 'sweep', 'local', 'r2', null]);
+
+ const express = require('express');
+ const app = express();
+ app.set('trust proxy', 'loopback');
+ app.use(express.json());
+ app.use('/api/v1/:app/admin/storage', require('../server/admin/routes'));
+ app.use('/o', require('../server/objects/routes').publicRouter);
+ const server = await new Promise((ok) => { const x = app.listen(0, '127.0.0.1', () => ok(x)); });
+ const base = `http://127.0.0.1:${server.address().port}`;
+ const get = (url, headers = {}) => new Promise((resolve, reject) => {
+ http.get(url, { headers: { 'User-Agent': BROWSER, ...headers } }, (res) => {
+ const b = []; res.on('data', (c) => b.push(c)); res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body: Buffer.concat(b) }));
+ }).on('error', reject);
+ });
+ const call = async (p, key = 'live-key-objtier') => { const x = await get(base + p, { Authorization: `Bearer ${key}` }); return { status: x.status, body: JSON.parse(x.body.toString() || '{}') }; };
+
+ const web = addObject();
+ let g = await get(`${base}/o/${web.id}`, { 'X-Forwarded-For': '192.0.2.44' });
+ assert.strictEqual(g.status, 200);
+ assert.ok(g.body.equals(web.bytes), 'served from the local canonical copy before');
+ await get(`${base}/o/${web.id}`, { 'X-Forwarded-For': '192.0.2.45', 'Sec-GPC': '1' });
+ assert.strictEqual(popularity.stats({ ids: [web.id] }).get(web.id).unique_viewers, 1, '/o/:id counts its viewers (not the opted-out one)');
+ assert.strictEqual((await tiering.promote(web.id, { trigger: 'manual', reason: 'test' })).outcome, 'done');
+ g = await get(`${base}/o/${web.id}`, { 'X-Forwarded-For': '192.0.2.46' });
+ assert.strictEqual(g.status, 302);
+ const loc = new URL(g.headers.location);
+ assert.ok(loc.pathname.endsWith(`/r2-bucket/objects/live/${web.id}`), g.headers.location);
+ assert.ok(/^attachment; filename="x\.bin"$/.test(loc.searchParams.get('response-content-disposition')), 'the presigned answer keeps the headers /o would send');
+ assert.strictEqual(loc.searchParams.get('response-content-type'), 'application/octet-stream');
+ assert.ok((await get(g.headers.location)).body.equals(web.bytes), 'and R2 serves the same bytes');
+ assert.strictEqual(popularity.stats({ ids: [web.id] }).get(web.id).unique_viewers, 2);
+
+ fault.corruptPut = 'r2-bucket';
+ const bad = addObject();
+ let m = await tiering.promote(bad.id, { reason: 'test' });
+ fault.corruptPut = null;
+ assert.deepStrictEqual([m.ok, m.outcome], [false, 'failed']);
+ assert.ok(/does not match the sha256/.test(m.error), m.error);
+ assert.ok(!buckets['r2-bucket'].has(`objects/live/${bad.id}`) && !r2Row(bad.id), 'an R2 copy that does not verify is removed and never recorded');
+ assert.ok(last(bad.id).error);
+ fault.refusePut = 'r2-bucket';
+ const refusedPut = addObject();
+ m = await tiering.promote(refusedPut.id, { reason: 'test' });
+ fault.refusePut = null;
+ assert.deepStrictEqual([m.outcome, !!r2Row(refusedPut.id)], ['failed', false]);
+ db.run('DELETE FROM media_object_views_daily WHERE object_id != ?', [bad.id]);
+ setViews(bad.id, [[0, 700]]);
+ s = await tiering.runSweep();
+ assert.deepStrictEqual([s.skipped_backoff, s.promoted, decisions(bad.id).length], [1, 0, 1], 'a failed object waits before the next try');
+
+ const fromB2 = addObject({ canonical: 'b2' });
+ m = await tiering.promote(fromB2.id, { reason: 'test' });
+ assert.strictEqual(m.outcome, 'done', JSON.stringify(m));
+ assert.ok(buckets['r2-bucket'].get(`objects/live/${fromB2.id}`).equals(fromB2.bytes), 'a B2 canonical copy is copied to R2 under its key');
+ assert.strictEqual(last(fromB2.id).from_provider, 'b2');
+ assert.strictEqual((await tiering.promote(fromB2.id, { reason: 'again' })).outcome, 'already');
+ console.log('✅ promotion copies the canonical copy, checks the R2 copy\'s size and sha256 before recording it; /o/:id then redirects to R2 with its own headers; failures leave nothing and back off');
+
+ // ── 8. Demotion: idle or not ready; never the last good copy; holds; purge waits for it ──
+ db.run('DELETE FROM media_object_views_daily');
+ setViews(hot.id, [[-20, 900]]); ageR2(hot.id, 20); // idle 20 days, in R2 20 days
+ const fresh = addObject(); await tiering.promote(fresh.id, { reason: 'test' }); // no viewer, but in R2 since now
+ setViews(web.id, [[0, 3]]); ageR2(web.id, 20); // viewed today
+ const lost = addObject(); await tiering.promote(lost.id, { reason: 'test' }); ageR2(lost.id, 20);
+ fs.unlinkSync(lost.file); // the canonical copy is gone: R2 is the last good copy
+ const drifted = addObject(); await tiering.promote(drifted.id, { reason: 'test' }); ageR2(drifted.id, 20);
+ const flipped = Buffer.from(drifted.bytes); flipped[0] ^= 0xff; fs.writeFileSync(drifted.file, flipped);
+ const pinned = addObject(); await tiering.promote(pinned.id, { reason: 'test' }); ageR2(pinned.id, 20);
+ const pin = model.placeHold({ object_id: pinned.id, kind: 'creator_pin', reason: 'test' });
+ const gone = addObject(); await tiering.promote(gone.id, { reason: 'test' }); setViews(gone.id, [[0, 900]]);
+ model.softDelete(model.getObject(gone.id), { by: 'test' });
+ const sticky = addObject(); await tiering.promote(sticky.id, { reason: 'test' }); ageR2(sticky.id, 20);
+ fault.keepOnDelete = 'r2-bucket';
+ // Only the sticky object sees the store keep its bytes: demote it on its own first.
+ m = await tiering.demote(sticky.id, { reason: 'test' });
+ fault.keepOnDelete = null;
+ assert.deepStrictEqual([m.outcome, !!r2Row(sticky.id)], ['failed', true]);
+ assert.ok(/still there/.test(m.error));
+ // Seven hours ago: past the back-off, so the sweep below tries again (and still within the 24-hour counts).
+ db.run("UPDATE media_object_tier_decisions SET decided_at = strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-7 hours') WHERE object_id = ? AND outcome = 'failed'", [sticky.id]);
+ await policy.set({ maxDemotionsPerSweep: 20 }, { reason: 'test: room for every demotion' });
+ s = await tiering.runSweep();
+ assert.ok(!r2Row(hot.id) && !buckets['r2-bucket'].has(`objects/live/${hot.id}`), 'an idle object leaves R2');
+ assert.ok(fs.existsSync(hot.file) && db.get("SELECT state FROM media_locations WHERE object_id = ? AND provider = 'local'", [hot.id]).state === 'present');
+ d = last(hot.id);
+ assert.deepStrictEqual([d.action, d.outcome, d.from_provider, d.to_provider], ['demote', 'done', 'r2', 'local']);
+ assert.ok(/idle for demoteIdleDays 14/.test(d.reason), d.reason);
+ assert.ok(r2Row(fresh.id) && r2Row(web.id), 'an object in R2 for less than demoteIdleDays, or viewed, stays');
+ assert.ok(r2Row(lost.id) && buckets['r2-bucket'].has(`objects/live/${lost.id}`), 'the last good copy is never removed');
+ assert.deepStrictEqual([last(lost.id).outcome, /the canonical local file is missing; the R2 copy is kept/.test(last(lost.id).reason)], ['refused', true]);
+ assert.ok(r2Row(drifted.id) && /does not match the sha256/.test(last(drifted.id).reason), 'nor one whose canonical copy no longer hashes right');
+ assert.ok(r2Row(pinned.id) && /retention hold/.test(last(pinned.id).reason), 'a hold blocks the demotion');
+ assert.ok(!r2Row(gone.id) && /the object is deleted/.test(last(gone.id).reason), 'a deleted object leaves R2 however popular');
+ assert.ok(!r2Row(cached.id), 'the month-idle copy from before goes too');
+ assert.ok(!r2Row(sticky.id), 'and the one the store kept once, now that it lets go');
+ g = await get(`${base}/o/${lost.id}`);
+ assert.strictEqual(g.status, 302, 'the object whose canonical copy is lost still plays from R2');
+ g = await get(`${base}/o/${hot.id}`);
+ assert.strictEqual(g.status, 200, 'a demoted object plays from its canonical copy again');
+ model.releaseHold(pin.id, 'test');
+
+ const purged = addObject(); await tiering.promote(purged.id, { reason: 'test' });
+ model.softDelete(model.getObject(purged.id), { by: 'test' });
+ db.run("UPDATE media_objects SET deleted_at = datetime('now', '-40 days') WHERE id = ?", [purged.id]);
+ model.purgeExpired();
+ assert.ok(fs.existsSync(purged.file) && r2Row(purged.id), 'the purge waits while the object still has an R2 copy');
+ await tiering.demote(purged.id, { trigger: 'sweep', reason: 'the object is deleted' });
+ model.purgeExpired();
+ assert.ok(!fs.existsSync(purged.file) && !r2Row(purged.id), 'and purges once the R2 copy is gone');
+ console.log('✅ demotion: idle (and settled) or not ready; refused under a hold or when the canonical copy does not check out; a failed delete keeps the row; the purge waits for it');
+
+ // ── 9. The decision log: admin API, metric, operator views ──
+ const gamesObj = addObject({ app: 'games' });
+ await tiering.promote(gamesObj.id, { reason: 'games' });
+ const cand = addObject(); setViews(cand.id, [[0, 800]]);
+ let a = await call('/api/v1/live/admin/storage/tiers/objects/policy');
+ assert.strictEqual(a.status, 200, JSON.stringify(a.body));
+ assert.deepStrictEqual([a.body.gate.active, a.body.gate.source], [true, 'setting']);
+ assert.deepStrictEqual(a.body.thresholds.promoteMinUniqueViewers7d, { value: 500, default: 500, source: 'default' });
+ assert.ok(/promoteMinUniqueViewers7d \(500\)/.test(a.body.rules.promote) && /last good copy/.test(a.body.rules.demote));
+ assert.deepStrictEqual(a.body.candidates.map((c) => [c.object_id, c.unique_viewers_7d, c.blocked_by]), [[cand.id, 800, null]]);
+ assert.strictEqual(a.body.provider.r2.configured, true);
+ const c24 = a.body.decisions.last_24h;
+ assert.ok(c24.promote.dry_run >= 3 && c24.promote.done >= 10 && c24.promote.refused >= 4 && c24.promote.failed >= 2 && c24.promote.already === 1, JSON.stringify(c24));
+ assert.ok(c24.demote.done >= 4 && c24.demote.refused >= 3 && c24.demote.failed === 1 && c24.demote.dry_run === 1, JSON.stringify(c24));
+ assert.ok(a.body.decisions.recent.every((x) => x.app_id === 'live') && a.body.decisions.recent[0].inputs && a.body.decisions.recent[0].thresholds);
+ a = await call('/api/v1/live/admin/storage/tiers/objects/decisions?outcome=dry_run&limit=2');
+ assert.deepStrictEqual([a.body.decisions.length, a.body.decisions.every((x) => x.outcome === 'dry_run'), a.body.next_before_id != null], [2, true, true]);
+ a = await call(`/api/v1/live/admin/storage/tiers/objects/decisions?object_id=${bad.id}`);
+ assert.deepStrictEqual(a.body.decisions.map((x) => x.outcome), ['failed']);
+ assert.strictEqual((await call('/api/v1/live/admin/storage/tiers/objects/decisions?action=move')).status, 400);
+ assert.strictEqual((await call('/api/v1/live/admin/storage/tiers/objects/decisions?outcome=nope')).status, 400);
+ a = await call('/api/v1/games/admin/storage/tiers/objects/decisions', 'games-key-objtier');
+ assert.deepStrictEqual(a.body.decisions.map((x) => x.object_id), [gamesObj.id], 'each app sees its own');
+ assert.ok((await call('/api/v1/live/admin/storage/tiers/objects/policy', 'wrong-key')).status >= 401);
+ a = await call('/api/v1/live/admin/storage/config');
+ const ns = a.body.namespaces.find((n) => n.namespace === 'media.object_tier');
+ assert.ok(ns && ns.values.active === true && ns.values.promoteMinUniqueViewers7d === 500, 'the policy is in the config routes');
+ a = await call('/api/v1/live/admin/storage/config/media.object_tier/history');
+ assert.deepStrictEqual(a.body.snapshots.map((x) => x.state), ['active', 'superseded', 'superseded', 'superseded']);
+ const post = (p, body) => new Promise((resolve, reject) => {
+ const data = JSON.stringify(body);
+ const rq = http.request(base + p, { method: 'POST', headers: { Authorization: 'Bearer live-key-objtier', 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(data) } },
+ (res) => { let b = ''; res.on('data', (c) => { b += c; }); res.on('end', () => resolve({ status: res.statusCode, body: JSON.parse(b || '{}') })); });
+ rq.on('error', reject);
+ rq.end(data);
+ });
+ a = await post('/api/v1/live/admin/storage/config/media.object_tier', { values: { active: false }, merge: true, reason: 'gate off through the route' });
+ assert.strictEqual(a.status, 200, JSON.stringify(a.body));
+ assert.strictEqual(policy.settings().active, false);
+ a = await post('/api/v1/live/admin/storage/config/media.object_tier', { values: { active: true }, merge: true, reason: 'and on again' });
+ assert.deepStrictEqual([a.status, policy.settings().active, policy.thresholds().active.source], [200, true, 'setting']);
+ a = await post('/api/v1/live/admin/storage/config/media.object_tier', { values: { demoteIdleDays: 2 }, merge: true, reason: 'flapping' });
+ assert.deepStrictEqual([a.status, policy.settings().demoteIdleDays], [422, 14], 'the route refuses what the rules refuse');
+
+ db.run("INSERT INTO media_tier_decisions (vod_id, app_id, action, outcome, trigger, reason) VALUES (1, 'live', 'promote', 'done', 'admin', 'test')");
+ const registry = require('openvibe-shared/metrics').createRegistry();
+ require('../server/observability').domainMetrics(registry, { db, recorder: { activeCount: () => 0 }, events: { status: () => ({ enabled: false }) } });
+ const text = registry.metrics();
+ const all24 = tiering.counts24h();
+ assert.ok(text.includes(`media_tier_decisions_24h{target="object",action="promote",outcome="done"} ${all24.promote.done}`), text);
+ assert.ok(text.includes(`media_tier_decisions_24h{target="object",action="promote",outcome="dry_run"} ${all24.promote.dry_run}`));
+ assert.ok(text.includes('media_tier_decisions_24h{target="vod",action="promote",outcome="done"} 1'));
+
+ const report = require('../server/me/ops').report({ appId: 'live' });
+ const no = report.tiering.native_objects;
+ const liveR2 = db.get("SELECT COUNT(*) AS n FROM media_locations l JOIN media_objects o ON o.id = l.object_id WHERE l.provider = 'r2' AND l.state = 'present' AND o.app_id = 'live'").n;
+ assert.deepStrictEqual([no.gate.active, no.r2_copies.count, no.eligible_to_promote, no.decisions_24h.promote.done], [true, liveR2, 1, c24.promote.done]);
+ assert.ok(no.recent.length && no.sweep.last_result && no.sweep.last_run_at);
+ assert.ok(!/until roadmap WS-G task 11/.test(report.tiering.note));
+ const html = require('../server/me/pages').renderOps({ person: { subject: 'usr_01J8Z3Q4X5Y6Z7A8B9C0D1E2F3' }, report, canRecompute: false });
+ assert.ok(/Native objects/.test(html) && /Activation gate/.test(html) && html.includes(bad.id), 'the /me/ops page shows the gate and the recent decisions');
+ console.log('✅ decisions: /tiers/objects/policy and /decisions (filters, pages, app-scoped), the config routes, media_tier_decisions_24h{target}, and the operator views');
+
+ // ── 10. The storage sweep runs it (step 4); a restore drill never does ──
+ const vodSweep = await storage.runSweep();
+ assert.ok(vodSweep.objects && vodSweep.objects.gate === true && typeof vodSweep.objects.promoted === 'number', JSON.stringify(vodSweep));
+ assert.strictEqual(r2Row(cand.id) && r2Row(cand.id).state, 'present', 'the storage sweep promoted the candidate');
+ // The storage engine's upload is exported (the scheduled verification's default re-upload calls it too).
+ assert.deepStrictEqual(['uploadFile', 'copyBetweenProviders', 'sha256Object', 'providerAvailable'].map((f) => typeof storage[f]), ['function', 'function', 'function', 'function']);
+ const drill = require('../server/drill');
+ drill.enabled = true;
+ assert.deepStrictEqual(await tiering.runSweep(), { skipped: true, reason: 'restore drill (MEDIA_DRILL)' });
+ drill.enabled = false;
+ console.log('✅ the storage sweep runs the object tiering as its step 4; under MEDIA_DRILL it does nothing');
+
+ server.close();
+ s3.close();
+ db.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('\n✅ All object tiering tests passed');
+ process.exit(0);
+})().catch((err) => { console.error(err); process.exit(1); });