openvibers/openvibe.media Commits
Route every storage class through the placement sweep (#32)
Changes
@@@ -556,7 +556,11 @@ unique viewers per object per UTC day, counted on `/o/:id` without storing an IPor when the object is no longer ready. Its **activation gate `active` is off by default**: the sweep (step 4of this one) then moves nothing and records what it would do as `dry_run` decisions. Holds block everymove, only a verified canonical copy is promoted (re-hashed; the R2 copy's size and sha256 are checked--before it is used), and a demotion never removes the last good copy. Decisions:++before it is used), and a demotion never removes the last good copy. Every object classifies into one of the six++storage classes (`video`, `image`, `download`, `game-asset`, `attachment`, `backup`): kind and mime type decide++video and image, and an app may declare any class in the object's metadata (`metadata.class`, or `media_class` /++`placement_class`), so each class's budgets, hysteresis band and R2 storage ceiling act on that class alone.++Decisions:`media_object_tier_decisions`, `/admin/storage/tiers/objects/*`, `/me/ops`, and`media_tier_decisions_24h{target="object"}` (`test/object-tiering.test.js`).@@@ -37,6 +37,7 @@"placement demand rollups on Valkey (F2.4): reads per object × region × 5-minute bucket, hotness over the hour, media.object.hot once per object per hour","tiering on demand (F2.5): the object sweep promotes and demotes on the hour's Valkey reads against per-class hysteresis bands, one MGET per page, falling back to the daily PostgreSQL counts for a whole sweep when Valkey fails","placement constraints (F2.6): sweep moves refused while their provider fails its class (breaker or probe), and a per-class monthly R2 storage ceiling (maxHotUsdPerMonth) applied in value order, each explained in the decision's inputs",++ "every storage class in the native-object sweep: kind and mime type classify video/image/download, an app may declare game-asset/attachment/backup (or any of the six) in the object's metadata (metadata.class, or media_class/placement_class), and each class spends its own per-sweep budget, hysteresis band and R2 storage ceiling","segment-native video (F3.1, behind MEDIA_HLS_ENABLED): the media_timeline index, the object.cmaf job (CMAF/fMP4 segments of a finished video on its own keyframes, sha256 and durability per row, idempotent) and HLS playlists written from the timeline at /o/:id/master.m3u8 under the object's access checks","packing after finalize (F3.3, behind MEDIA_HLS_ENABLED): the object.pack job concatenates durable segments into ~60 s chunk objects verified by sha256, each media_timeline row naming its byte range; the segment route serves a packed segment as a ranged read of its chunk; the per-segment keys are deleted once the chunk is durable; segment providers come from the placement router; a finished object.cmaf queues object.pack by itself (dedupeActive: a rerun joins an active pack, a later cut queues a new one)"],@@@ -261,8 +261,17 @@ formats (AVIF/WebP), preview clips and AI thumbnails are `rebuildable = true`:only the moves made; `projected` also counts the dry runs, and the ceiling (and `inputs.budget.usd_per_month`) is heldagainst it, so dry runs (gate off) are refused as a live sweep would be, without changing `spend` (a dry runrepeated the same day is logged once but still counts, and a dry-run demotion checks its canonical copy like a live-- one before it frees any projected cost). Native objects classify only as `video`, `image` or `download` today, so a ceiling (or budget)-- set on `game-asset`, `attachment` or `backup` decides nothing yet.++ one before it frees any projected cost). Every native object classifies into one of the six classes, so a budget,++ hysteresis band or ceiling set on any of them acts on that class's objects: `kind` and `mime_type` decide `video`++ and `image` (everything else is `download`), and an app may declare **`game-asset`, `attachment` or `backup`** for++ one of its objects in the object's metadata — `metadata.class`, or the explicit `media_class` / `placement_class`.++ A declaration is honoured only when kind and mime type would otherwise say `download`, so content alone decides++ `video` and `image`: a name outside those three (a `video` or `image` declaration included, or one on a video or++ image object) is ignored and kind and mime type decide. The declaration is limited this way because budgets,++ hysteresis bands and R2 ceilings are global per class — without the limit any app could declare a class on a large++ video and spend another class's budget, starving or filling it, and be billed for it in the sweep's spend++ (`hotSpend()`). An `attachment` or `backup` object therefore promotes and demotes under its own budget, and a++ `game-asset` ceiling refuses that class alone.## 7. Delivery: sticky, measured, canaried@@@ -326,7 +335,9 @@ metric. Budgets per class and provider with a forecast; `media.provider.cost_thrdecision log with class and reason, the events above, Valkey rollups of demand (F2.4, shipped) and the sweep'seligibility on them with per-class hysteresis and a PostgreSQL fallback (F2.5, shipped), the provider-class gateand per-class monthly R2 storage ceilings on its moves (F2.6, shipped), the move cleanup job and its alert after-- three failures (§6, shipped). Still open in F2: every class (not only native objects ↔ R2) through the one sweep.++ three failures (§6, shipped). Every storage class (all six) now acts in the one sweep: native objects classify by++ kind and mime type, or by a class the app declares in the object's metadata (§6, shipped). Still open in F2: the++ projected objects (VODs and clips) keep their own tiering path (`media_tier_decisions`) instead of the sweep.- **F3 segment-native video:** the timeline index and the CMAF/HLS source representation of finished video (F3.1,shipped; §3), packing into ~60 s chunks (F3.3, shipped; §3), the timeline queued at finalize (and lazily ondownload), signed playlists outliving a download URL and the faststart MP4 fallback (F3.c, shipped; §3), virtual and@@@ -209,7 +209,7 @@ All routes live under `/api/v2/:app/objects`.| method | path | notes ||---|---|---|--| POST | `/` | init. `{ namespace (default the tenant's root; a name below it, full or relative), kind, visibility (default private), mime_type, size_bytes, filename, content_hash (expected sha256), metadata (≤ 16 KB), user_id, upload_ttl (60-86400 s), multipart, part_size }` returns 201 `{ id, object, upload: { method: 'PUT', url, token, expires_at, max_bytes, content_type, complete_url, multipart_url } }`. `url` is a [presigned PUT URL](#presigned-uploads). With `multipart: true` (required above `MEDIA_OBJECT_MAX_MB`): `method: 'multipart'`, `url: null`, and `upload.multipart` is the new [session](#multipart-uploads). Checks at init: `size_bytes` against `MEDIA_OBJECT_MAX_MB` (single part) or `MEDIA_MULTIPART_MAX_MB` (413), the namespace's policy (422 `media.namespace.policy_denied`, 413 `media.object.too_large`), every quota from the namespace up to the tenant including reservations (413 `media.quota.exceeded` / `media.quota.objects_exceeded`; the declared size is then reserved), the public-size invariant (422) and the [content type](#content-types) for the kind (415 `media.object.unsupported_type`) |++| POST | `/` | init. `{ namespace (default the tenant's root; a name below it, full or relative), kind, visibility (default private), mime_type, size_bytes, filename, content_hash (expected sha256), metadata (≤ 16 KB; `metadata.class` declares the object's storage class, see [Tiering of native objects](#tiering-of-native-objects)), user_id, upload_ttl (60-86400 s), multipart, part_size }` returns 201 `{ id, object, upload: { method: 'PUT', url, token, expires_at, max_bytes, content_type, complete_url, multipart_url } }`. `url` is a [presigned PUT URL](#presigned-uploads). With `multipart: true` (required above `MEDIA_OBJECT_MAX_MB`): `method: 'multipart'`, `url: null`, and `upload.multipart` is the new [session](#multipart-uploads). Checks at init: `size_bytes` against `MEDIA_OBJECT_MAX_MB` (single part) or `MEDIA_MULTIPART_MAX_MB` (413), the namespace's policy (422 `media.namespace.policy_denied`, 413 `media.object.too_large`), every quota from the namespace up to the tenant including reservations (413 `media.quota.exceeded` / `media.quota.objects_exceeded`; the declared size is then reserved), the public-size invariant (422) and the [content type](#content-types) for the kind (415 `media.object.unsupported_type`) || POST | `/:id/upload-url` | a fresh presigned single-PUT URL for an uploading object (`{ ttl }` 60-86400 s) || PUT | `/:id/content` | the bytes, single part. Auth is the upload token (`?token=` from init, or `X-Upload-Token`) or the usual credential. Streams to disk computing sha256. Refuses more than the declared size or the limit (413; an object declared above `MEDIA_OBJECT_MAX_MB` goes multipart), a size mismatch (400), quota overrun (413), a type that does not suit the kind (415) and a PUT while a multipart session is open (409). Stored at `OBJECTS_PATH/<app>/<id>` with a verified local location. Mounted ahead of the JSON body parser, so any Content-Type is taken as raw bytes; the Content-Type becomes `mime_type` when init set none. Can be repeated while the object is `uploading` || POST | `/:id/complete` | checks the expected hash (422 `media.object.hash_mismatch`), the invariant, the quota and the bytes against the type (415 `media.object.content_mismatch`). Sets `ready` and sends the `media.object.uploaded` webhook. Also accepts the upload token |@@@ -921,6 +921,17 @@ Each pass rotates the popularity counts, then:500, so no class is crowded out of its budget by a most-viewed prefix.Budgets and minimum residency are per class (`media.storage_policy`); a refusal or back-off spends no slot.++**Classes.** Every native object is classified into one of the six storage classes (`video`, `image`,++`download`, `game-asset`, `attachment`, `backup`), and each class has its own per-sweep budgets, minimum++residency, hysteresis band and optional monthly R2 storage ceiling in `media.storage_policy`. `kind` and++`mime_type` decide `video` and `image` (video/audio and their kinds, image and its kinds); everything else is++`download` unless the app declared `game-asset`, `attachment` or `backup` for the object in its metadata:++`metadata.class`, or the explicit `media_class` / `placement_class`. A declaration is honoured only when kind++and mime type would say `download`, so content alone decides `video` and `image` (budgets and ceilings are++global per class, so a declaration could otherwise bill a video to another class's); any other name is ignored.++The declared class is what the sweep's eligibility, ranking and budgets read, and the class's++R2 copies are what its ceiling counts, so a ceiling on `game-asset` refuses that class alone.++**Promotion** (`tiering.promote`): refused when the object is held, not ready, a sandbox object, or itscanonical copy is not verified on record (present, checked, not contradicted, and the object has asha256: the `object.hash` job supplies it for older objects); then the gate (`dry_run`); then R2 must be@@@ -151,10 +151,26 @@ const fmtUsd = (v) => `$${Number(Number(v).toPrecision(4))}`;/** What each class's present R2 copies cost a month at list price, gross of R2's account-wide free tier: { video: 0.12, image: 0, ... }. */async function hotSpend() {const out = Object.fromEntries(storagePolicy.CLASSES.map((c) => [c, 0]));-- const rows = await db.all(`SELECT o.kind, o.mime_type, SUM(COALESCE(l.size_bytes, o.size_bytes, 0))::bigint AS bytes-- FROM media_locations l JOIN media_objects o ON o.id = l.object_id-- WHERE l.provider = 'r2' AND l.state = 'present' GROUP BY o.kind, o.mime_type`);-- for (const r of rows) out[storagePolicy.classOf(r)] += hotUsdPerMonth(r.bytes);++ // Aggregate in SQL: one row per (declared class, kind, mime_type) group, not per copy — a copy carries up to++ // 16 KB of metadata, and R2 copies (the hot cache) run to many thousands; the groups are a few dozen. The three++ // metadata keys classOf() reads are extracted here with the SQLite-compatible json_extract() the migrations++ // define over the text metadata column (guard with json_valid, so unreadable JSON declares nothing, as++ // declaredClass() does), lower-cased and trimmed like normalizeClass(), and summed per group. classOf() then++ // maps each group's representative fields to its class — the one place the rules live.++ const rows = await db.all(`SELECT grp.kind, grp.mime_type, grp.declared_class, grp.declared_media_class, grp.declared_placement_class,++ SUM(grp.bytes)::bigint AS bytes++ FROM (SELECT o.kind, o.mime_type,++ CASE WHEN json_valid(o.metadata) THEN lower(trim(json_extract(o.metadata, '$.class'))) END AS declared_class,++ CASE WHEN json_valid(o.metadata) THEN lower(trim(json_extract(o.metadata, '$.media_class'))) END AS declared_media_class,++ CASE WHEN json_valid(o.metadata) THEN lower(trim(json_extract(o.metadata, '$.placement_class'))) END AS declared_placement_class,++ COALESCE(l.size_bytes, o.size_bytes, 0) AS bytes++ FROM media_locations l JOIN media_objects o ON o.id = l.object_id++ WHERE l.provider = 'r2' AND l.state = 'present') grp++ GROUP BY grp.kind, grp.mime_type, grp.declared_class, grp.declared_media_class, grp.declared_placement_class`);++ for (const r of rows) {++ out[storagePolicy.classOf({ kind: r.kind, mime_type: r.mime_type,++ metadata: { class: r.declared_class, media_class: r.declared_media_class, placement_class: r.declared_placement_class } })] += hotUsdPerMonth(r.bytes);++ }return out;}@@@ -104,8 +104,40 @@ function classPolicy(name) {return settings().classes[name];}--/** The placement class of a native object: video/audio → video, image/thumbnail → image, everything else → download. */--function classOf(obj) {++// An app may declare its object's class in the object's metadata: `class`, or the explicit `media_class` /++// `placement_class`. Only one of CLASSES counts; anything else is ignored and the kind and mime type decide.++// A declaration is honoured only when kind and mime type would otherwise classify the object as `download`,++// and only to one of DECLARABLE_CLASSES: the roles content cannot express. Video and image are decided by++// content, never declared. Budgets, hysteresis bands and R2 ceilings are global per class, so letting a++// declaration win unconditionally would let one app put a large video on another class's budget, starve that++// class, and be billed for it in the sweep's spend (server/objects/tiering.js hotSpend()).++const DECLARED_KEYS = ['class', 'media_class', 'placement_class'];++const DECLARABLE_CLASSES = ['game-asset', 'attachment', 'backup'];++++/** A known class name (case-folded), or null for anything else. */++function normalizeClass(value) {++ if (value == null || typeof value === 'object') return null;++ const name = String(value).trim().toLowerCase();++ return CLASSES.includes(name) ? name : null;++}++++/** The class an app declared for the object — its metadata as an object or its JSON text — or null. */++function declaredClass(obj) {++ if (!obj) return null;++ const direct = normalizeClass(obj.class);++ if (direct) return direct;++ let md = obj.metadata;++ if (typeof md === 'string') { try { md = JSON.parse(md); } catch { return null; } }++ if (!md || typeof md !== 'object') return null;++ for (const key of DECLARED_KEYS) {++ const name = normalizeClass(md[key]);++ if (name) return name;++ }++ return null;++}++++/** What kind and mime type alone decide: video/audio → video, image/thumbnail → image, everything else → download. */++function classFromContent(obj) {const kind = obj && obj.kind;if (kind === 'vod' || kind === 'clip') return 'video';if (kind === 'thumbnail' || kind === 'screenshot' || kind === 'avatar') return 'image';@@@ -115,6 +147,20 @@ function classOf(obj) {return 'download';}++/**++ * The placement class of a native object: video/audio → video, image/thumbnail → image, everything else →++ * download, unless the app declared one of DECLARABLE_CLASSES (game-asset, attachment, backup) for it and the++ * content alone would say download. All six classes (CLASSES) act in the one sweep: this decides each object's++ * hysteresis band, per-sweep budget and R2 storage ceiling. A declared class never overrides content, so one++ * app cannot bill a video or image to another class's global budget, band or ceiling.++ */++function classOf(obj) {++ const base = classFromContent(obj);++ if (base !== 'download') return base;++ const declared = declaredClass(obj);++ return declared && DECLARABLE_CLASSES.includes(declared) ? declared : base;++}++function budgetFor(name) {const { maxPromotionsPerSweep, maxDemotionsPerSweep } = classPolicy(name);return { maxPromotionsPerSweep, maxDemotionsPerSweep };@@@ -91,11 +91,11 @@ const crypto = require('crypto');await storage.setSettings({ hotDiskPressurePct: 100, criticalDiskPct: 100, minFreeGb: 0 }, { reason: 'test: no pressure drain' });const today = popularity.dayOf(Date.now());-- const addObject = async ({ app = 'live', size = 4096, mime = 'application/octet-stream', verified = true, hash = true, canonical = 'local' } = {}) => {++ const addObject = async ({ app = 'live', size = 4096, mime = 'application/octet-stream', verified = true, hash = true, canonical = 'local', meta = {} } = {}) => {const bytes = crypto.randomBytes(size);const sum = sha256(bytes);const id = await model.createObject({ app_id: app, kind: 'file', visibility: 'public', lifecycle_status: 'ready', mime_type: mime, size_bytes: size,-- content_hash: hash ? sum : null, canonical_provider: canonical, metadata: { filename: 'x.bin' } });++ content_hash: hash ? sum : null, canonical_provider: canonical, metadata: { filename: 'x.bin', ...meta } });let file = null;if (canonical === 'local') {file = path.join(process.env.OBJECTS_PATH, app, id);@@@ -773,6 +773,75 @@ const crypto = require('crypto');for (const v of candViews) await db.run('INSERT INTO media_object_views_daily (object_id, day, unique_viewers, last_viewed_at) VALUES (?, ?, ?, ?)', [v.object_id, v.day, v.unique_viewers, v.last_viewed_at]);console.log('✅ constraints: a provider failing its class (breaker or probe) refuses the move and keeps every copy; the monthly R2 storage ceiling closes a class in value order');++ // ── 9g. Every storage class through the one sweep: an app declares its object's class in the metadata ──++ await db.run('DELETE FROM media_object_views_daily');++ const att = await addObject({ meta: { class: 'attachment' } });++ const bak = await addObject({ meta: { class: 'backup' } });++ const dl0 = await addObject();++ for (const o of [att, bak, dl0]) await setViews(o.id, [[0, 900]]);++ assert.deepStrictEqual([sp.classOf(await model.getObject(att.id)), sp.classOf(await model.getObject(bak.id)), sp.classOf(await model.getObject(dl0.id))],++ ['attachment', 'backup', 'download'], 'an app-declared class rides on the metadata the sweep reads');++ await sp.set({ classes: { download: { maxPromotionsPerSweep: 0 }, attachment: { maxPromotionsPerSweep: 1 }, backup: { maxPromotionsPerSweep: 1 } } },++ { reason: 'test: per-class budgets for attachment and backup' });++ s = await tiering.runSweep();++ assert.ok(await r2Row(att.id) && await r2Row(bak.id), 'attachment and backup promote under their own budgets');++ assert.ok(!await r2Row(dl0.id) && (await decisions(dl0.id)).length === 0, 'the download budget of 0 spends nothing on them');++ assert.strictEqual(JSON.parse((await last(att.id)).inputs).class, 'attachment');++ assert.ok(/class attachment, value per dollar/.test((await last(att.id)).reason), (await last(att.id)).reason);++ // A class's R2 spend counts its own copies: a declared class is never billed to its mime type's class.++ const spendBefore = await tiering.hotSpend();++ const gasPrice = tiering.hotUsdPerMonth(4096);++ const gas = await addObject({ meta: { class: 'game-asset' } });++ await inR2(gas, 30);++ const spendAfter = await tiering.hotSpend();++ assert.ok(Math.abs(spendAfter['game-asset'] - (spendBefore['game-asset'] + gasPrice)) < 1e-12, `the game-asset copy spends on game-asset: ${JSON.stringify(spendAfter)}`);++ assert.ok(Math.abs(spendAfter.download - spendBefore.download) < 1e-12, 'and not on download, which its mime type alone would say');++ // hotSpend aggregates in SQL: several copies in one (declared class, kind, mime type) group come back as++ // one row with their bytes summed, rather than one row each carrying the copy's whole metadata.++ for (let i = 0; i < 3; i++) await inR2(await addObject({ meta: { class: 'backup' } }), 30);++ const realAll = db.all;++ let groupedRows = null;++ db.all = async (sql, params) => { const rows = await realAll.call(db, sql, params); if (/SUM\(grp\.bytes\)/.test(sql)) groupedRows = rows; return rows; };++ const groupedSpend = await tiering.hotSpend();++ db.all = realAll;++ const group = groupedRows.filter((r) => r.declared_class === 'backup' && r.kind === 'file' && r.mime_type === 'application/octet-stream');++ const groupCopies = await db.get(`SELECT COUNT(*) AS n, COALESCE(SUM(COALESCE(l.size_bytes, o.size_bytes, 0)), 0)::bigint AS bytes++ FROM media_locations l JOIN media_objects o ON o.id = l.object_id++ WHERE l.provider = 'r2' AND l.state = 'present' AND o.kind = 'file' AND o.mime_type = 'application/octet-stream'++ AND lower(trim(json_extract(o.metadata, '$.class'))) = 'backup'`);++ assert.ok(Number(groupCopies.n) >= 3, `several copies in the group (${groupCopies.n})`);++ assert.strictEqual(group.length, 1, 'the copies are grouped into one row');++ assert.strictEqual(Number(group[0].bytes), Number(groupCopies.bytes), 'their bytes are summed in that row');++ assert.ok(Math.abs(groupedSpend.backup - tiering.hotUsdPerMonth(groupCopies.bytes)) < 1e-12, 'the class spend is the summed group');++ // A demotion spends the class's own budget too.++ const attR2 = await addObject({ meta: { class: 'attachment' } });++ const dlR2 = await addObject();++ await inR2(attR2, 30); await inR2(dlR2, 30);++ await sp.set({ classes: { download: { maxDemotionsPerSweep: 0 }, attachment: { maxDemotionsPerSweep: 1 } } }, { reason: 'test: per-class demotion budgets' });++ s = await tiering.runSweep();++ assert.ok(!await r2Row(attR2.id) && await r2Row(dlR2.id), 'the attachment demotes under its own budget; the download budget of 0 keeps its copy');++ assert.strictEqual(JSON.parse((await last(attR2.id)).inputs).class, 'attachment');++ // A ceiling on one class refuses only that class.++ const gAs = [await addObject({ meta: { class: 'game-asset' } }), await addObject({ meta: { class: 'game-asset' } })];++ const dl3 = await addObject();++ for (const o of [...gAs, dl3]) await setViews(o.id, [[0, 900]]);++ await sp.set({ classes: { 'game-asset': { maxHotUsdPerMonth: 0 }, download: { maxPromotionsPerSweep: 3, maxDemotionsPerSweep: 20 } } },++ { reason: 'test: a game-asset ceiling' });++ s = await tiering.runSweep();++ assert.ok(!await r2Row(gAs[0].id) && !await r2Row(gAs[1].id), 'the game-asset ceiling refuses the class');++ assert.ok(await r2Row(dl3.id), 'the download class is unaffected');++ d = await last(gAs[0].id);++ assert.deepStrictEqual([d.action, d.outcome, JSON.parse(d.inputs).class], ['promote', 'refused', 'game-asset']);++ assert.ok(/class game-asset has maxHotUsdPerMonth \$0/.test(d.reason), d.reason);++ assert.strictEqual((await decisions(gAs[1].id)).length, 0, 'the refusal closed game-asset only');++ assert.deepStrictEqual([s.constraints.hot_usd_per_month['game-asset'].ceiling, s.constraints.refused.budget], [0, 1], JSON.stringify(s.constraints));++ console.log('✅ every class through one sweep: declared attachment/backup promote and demote under their own budgets, and a game-asset ceiling refuses only that class');++++ await sp.set({ classes: { download: { maxPromotionsPerSweep: 3, maxDemotionsPerSweep: 20 }, attachment: { maxPromotionsPerSweep: 3, maxDemotionsPerSweep: 10 },++ backup: { maxPromotionsPerSweep: 3, maxDemotionsPerSweep: 10 }, 'game-asset': { maxHotUsdPerMonth: 1e9 } } }, { reason: 'test: class policy back' });++ await db.run('DELETE FROM media_object_views_daily');++ for (const v of candViews) await db.run('INSERT INTO media_object_views_daily (object_id, day, unique_viewers, last_viewed_at) VALUES (?, ?, ?, ?)', [v.object_id, v.day, v.unique_viewers, v.last_viewed_at]);++// ── 10. The storage sweep runs it (step 4); a restore drill never does ──const vodSweep = await storage.runSweep();assert.ok(vodSweep.objects && vodSweep.objects.gate === true && typeof vodSweep.objects.promoted === 'number', JSON.stringify(vodSweep));@@@ -28,11 +28,38 @@ const path = require('path');}assert.strictEqual(policy.mayMove({ class: 'video', lastMovedAt: 1000, now: 1000 }), true, 'zero residency is off');assert.throws(() => policy.budgetFor('unknown'), RangeError);-- const cls = (kind, mime_type) => policy.classOf({ kind, mime_type });++ const cls = (kind, mime_type, metadata) => policy.classOf({ kind, mime_type, metadata });assert.deepStrictEqual([cls('vod'), cls('clip'), cls('file', 'audio/mpeg'), cls('file', 'Video/MP4'), cls('asset', 'video/webm')], Array(5).fill('video'));assert.deepStrictEqual([cls('thumbnail'), cls('screenshot'), cls('avatar'), cls('file', 'image/png')], Array(4).fill('image'));assert.deepStrictEqual([cls('file', 'application/zip'), cls('file'), cls('asset', null), policy.classOf(null)], Array(4).fill('download'));++ // Every class acts: an app declares its object's role in the object's metadata (class, or the explicit++ // media_class / placement_class). A declaration is honoured only when kind and mime type would say++ // download, and only for game-asset / attachment / backup: content alone decides video and image.++ assert.deepStrictEqual([++ cls('file', 'application/zip', { class: 'attachment' }),++ cls('file', null, { class: 'backup' }),++ cls('asset', null, { media_class: 'game-asset' }),++ cls('file', 'video/mp4', { placement_class: 'game-asset' }),++ cls('file', 'application/octet-stream', { class: 'Video' }),++ ], ['attachment', 'backup', 'game-asset', 'video', 'download']);++ assert.strictEqual(policy.classOf({ kind: 'file', metadata: JSON.stringify({ class: 'backup' }) }), 'backup', 'metadata read as its JSON text too');++ assert.deepStrictEqual([cls('file', 'application/zip', { class: 'nope' }), cls('file', null, { class: 7 }),++ cls('file', null, 'not json metadata'), cls('file', null, { class: ['attachment'] })], Array(4).fill('download'),++ 'an unknown, non-string or unreadable declaration is ignored and kind/mime decide');++ // Class escape: a declaration can never override content, so one app cannot bill a large video (or an++ // image) to another class's global budget, hysteresis band or R2 ceiling.++ assert.strictEqual(cls('file', 'video/mp4', { class: 'backup' }), 'video', 'a declared backup does not steal video');++ assert.strictEqual(cls('vod', null, { class: 'attachment' }), 'video', 'a VOD kind stays video');++ assert.strictEqual(cls('file', 'image/png', { placement_class: 'attachment' }), 'image', 'an image stays image');++ assert.strictEqual(cls('thumbnail', null, { media_class: 'game-asset' }), 'image', 'a thumbnail stays image');++ assert.strictEqual(cls('file', 'application/zip', { class: 'backup' }), 'backup', 'a download may be a backup');++ assert.strictEqual(cls('file', 'application/zip', { class: 'video' }), 'download', 'but never declared video');++ assert.strictEqual(cls('file', 'application/zip', { class: 'image' }), 'download', 'nor declared image');++ assert.deepStrictEqual([policy.budgetFor('attachment'), policy.hotCeilingFor('backup'), policy.bandFor('game-asset')],++ [{ maxPromotionsPerSweep: 3, maxDemotionsPerSweep: 10 }, null, { promoteReadsPerHour: 60, demoteReadsPerHour: 6 }],++ 'the classes an app declares have their own policy like any other');++await policy.set({ classes: { image: { maxPromotionsPerSweep: 2, maxDemotionsPerSweep: 4, minResidencyMs: 60_000 } } },{ actor: { type: 'service', id: 'media-test' }, reason: 'test image placement policy' });assert.strictEqual(policy.get().revision(), 2, 'set creates a new revision');
@@ -556,7 +556,11 @@ unique viewers per object per UTC day, counted on `/o/:id` without storing an IP
@@ -556,7 +556,11 @@ unique viewers per object per UTC day, counted on `/o/:id` without storing an IP
or when the object is no longer ready. Its **activation gate `active` is off by default**: the sweep (step 4
or when the object is no longer ready. Its **activation gate `active` is off by default**: the sweep (step 4
of this one) then moves nothing and records what it would do as `dry_run` decisions. Holds block every
of this one) then moves nothing and records what it would do as `dry_run` decisions. Holds block every
move, only a verified canonical copy is promoted (re-hashed; the R2 copy's size and sha256 are checked
move, only a verified canonical copy is promoted (re-hashed; the R2 copy's size and sha256 are checked
-before it is used), and a demotion never removes the last good copy. Decisions:
+before it is used), and a demotion never removes the last good copy. Every object classifies into one of the six
+storage classes (`video`, `image`, `download`, `game-asset`, `attachment`, `backup`): kind and mime type decide
+video and image, and an app may declare any class in the object's metadata (`metadata.class`, or `media_class` /
+`placement_class`), so each class's budgets, hysteresis band and R2 storage ceiling act on that class alone.
+Decisions:
`media_object_tier_decisions`, `/admin/storage/tiers/objects/*`, `/me/ops`, and
`media_object_tier_decisions`, `/admin/storage/tiers/objects/*`, `/me/ops`, and
`media_tier_decisions_24h{target="object"}` (`test/object-tiering.test.js`).
`media_tier_decisions_24h{target="object"}` (`test/object-tiering.test.js`).
@@ -37,6 +37,7 @@
@@ -37,6 +37,7 @@
"placement demand rollups on Valkey (F2.4): reads per object × region × 5-minute bucket, hotness over the hour, media.object.hot once per object per hour",
"placement demand rollups on Valkey (F2.4): reads per object × region × 5-minute bucket, hotness over the hour, media.object.hot once per object per hour",
"tiering on demand (F2.5): the object sweep promotes and demotes on the hour's Valkey reads against per-class hysteresis bands, one MGET per page, falling back to the daily PostgreSQL counts for a whole sweep when Valkey fails",
"tiering on demand (F2.5): the object sweep promotes and demotes on the hour's Valkey reads against per-class hysteresis bands, one MGET per page, falling back to the daily PostgreSQL counts for a whole sweep when Valkey fails",
"placement constraints (F2.6): sweep moves refused while their provider fails its class (breaker or probe), and a per-class monthly R2 storage ceiling (maxHotUsdPerMonth) applied in value order, each explained in the decision's inputs",
"placement constraints (F2.6): sweep moves refused while their provider fails its class (breaker or probe), and a per-class monthly R2 storage ceiling (maxHotUsdPerMonth) applied in value order, each explained in the decision's inputs",
+ "every storage class in the native-object sweep: kind and mime type classify video/image/download, an app may declare game-asset/attachment/backup (or any of the six) in the object's metadata (metadata.class, or media_class/placement_class), and each class spends its own per-sweep budget, hysteresis band and R2 storage ceiling",
"segment-native video (F3.1, behind MEDIA_HLS_ENABLED): the media_timeline index, the object.cmaf job (CMAF/fMP4 segments of a finished video on its own keyframes, sha256 and durability per row, idempotent) and HLS playlists written from the timeline at /o/:id/master.m3u8 under the object's access checks",
"segment-native video (F3.1, behind MEDIA_HLS_ENABLED): the media_timeline index, the object.cmaf job (CMAF/fMP4 segments of a finished video on its own keyframes, sha256 and durability per row, idempotent) and HLS playlists written from the timeline at /o/:id/master.m3u8 under the object's access checks",
"packing after finalize (F3.3, behind MEDIA_HLS_ENABLED): the object.pack job concatenates durable segments into ~60 s chunk objects verified by sha256, each media_timeline row naming its byte range; the segment route serves a packed segment as a ranged read of its chunk; the per-segment keys are deleted once the chunk is durable; segment providers come from the placement router; a finished object.cmaf queues object.pack by itself (dedupeActive: a rerun joins an active pack, a later cut queues a new one)"
"packing after finalize (F3.3, behind MEDIA_HLS_ENABLED): the object.pack job concatenates durable segments into ~60 s chunk objects verified by sha256, each media_timeline row naming its byte range; the segment route serves a packed segment as a ranged read of its chunk; the per-segment keys are deleted once the chunk is durable; segment providers come from the placement router; a finished object.cmaf queues object.pack by itself (dedupeActive: a rerun joins an active pack, a later cut queues a new one)"
],
],
@@ -261,8 +261,17 @@ formats (AVIF/WebP), preview clips and AI thumbnails are `rebuildable = true`:
@@ -261,8 +261,17 @@ formats (AVIF/WebP), preview clips and AI thumbnails are `rebuildable = true`:
only the moves made; `projected` also counts the dry runs, and the ceiling (and `inputs.budget.usd_per_month`) is held
only the moves made; `projected` also counts the dry runs, and the ceiling (and `inputs.budget.usd_per_month`) is held
against it, so dry runs (gate off) are refused as a live sweep would be, without changing `spend` (a dry run
against it, so dry runs (gate off) are refused as a live sweep would be, without changing `spend` (a dry run
repeated the same day is logged once but still counts, and a dry-run demotion checks its canonical copy like a live
repeated the same day is logged once but still counts, and a dry-run demotion checks its canonical copy like a live
- one before it frees any projected cost). Native objects classify only as `video`, `image` or `download` today, so a ceiling (or budget)
- set on `game-asset`, `attachment` or `backup` decides nothing yet.
+ one before it frees any projected cost). Every native object classifies into one of the six classes, so a budget,
+ hysteresis band or ceiling set on any of them acts on that class's objects: `kind` and `mime_type` decide `video`
+ and `image` (everything else is `download`), and an app may declare **`game-asset`, `attachment` or `backup`** for
+ one of its objects in the object's metadata — `metadata.class`, or the explicit `media_class` / `placement_class`.
+ A declaration is honoured only when kind and mime type would otherwise say `download`, so content alone decides
+ `video` and `image`: a name outside those three (a `video` or `image` declaration included, or one on a video or
+ image object) is ignored and kind and mime type decide. The declaration is limited this way because budgets,
+ hysteresis bands and R2 ceilings are global per class — without the limit any app could declare a class on a large
+ video and spend another class's budget, starving or filling it, and be billed for it in the sweep's spend
+ (`hotSpend()`). An `attachment` or `backup` object therefore promotes and demotes under its own budget, and a
+ `game-asset` ceiling refuses that class alone.
## 7. Delivery: sticky, measured, canaried
## 7. Delivery: sticky, measured, canaried
@@ -326,7 +335,9 @@ metric. Budgets per class and provider with a forecast; `media.provider.cost_thr
@@ -326,7 +335,9 @@ metric. Budgets per class and provider with a forecast; `media.provider.cost_thr
decision log with class and reason, the events above, Valkey rollups of demand (F2.4, shipped) and the sweep's
decision log with class and reason, the events above, Valkey rollups of demand (F2.4, shipped) and the sweep's
eligibility on them with per-class hysteresis and a PostgreSQL fallback (F2.5, shipped), the provider-class gate
eligibility on them with per-class hysteresis and a PostgreSQL fallback (F2.5, shipped), the provider-class gate
and per-class monthly R2 storage ceilings on its moves (F2.6, shipped), the move cleanup job and its alert after
and per-class monthly R2 storage ceilings on its moves (F2.6, shipped), the move cleanup job and its alert after
- three failures (§6, shipped). Still open in F2: every class (not only native objects ↔ R2) through the one sweep.
+ three failures (§6, shipped). Every storage class (all six) now acts in the one sweep: native objects classify by
+ kind and mime type, or by a class the app declares in the object's metadata (§6, shipped). Still open in F2: the
+ projected objects (VODs and clips) keep their own tiering path (`media_tier_decisions`) instead of the sweep.
- **F3 segment-native video:** the timeline index and the CMAF/HLS source representation of finished video (F3.1,
- **F3 segment-native video:** the timeline index and the CMAF/HLS source representation of finished video (F3.1,
shipped; §3), packing into ~60 s chunks (F3.3, shipped; §3), the timeline queued at finalize (and lazily on
shipped; §3), packing into ~60 s chunks (F3.3, shipped; §3), the timeline queued at finalize (and lazily on
download), signed playlists outliving a download URL and the faststart MP4 fallback (F3.c, shipped; §3), virtual and
download), signed playlists outliving a download URL and the faststart MP4 fallback (F3.c, shipped; §3), virtual and
@@ -209,7 +209,7 @@ All routes live under `/api/v2/:app/objects`.
@@ -209,7 +209,7 @@ All routes live under `/api/v2/:app/objects`.
| method | path | notes |
| method | path | notes |
|---|---|---|
|---|---|---|
-| POST | `/` | init. `{ namespace (default the tenant's root; a name below it, full or relative), kind, visibility (default private), mime_type, size_bytes, filename, content_hash (expected sha256), metadata (≤ 16 KB), user_id, upload_ttl (60-86400 s), multipart, part_size }` returns 201 `{ id, object, upload: { method: 'PUT', url, token, expires_at, max_bytes, content_type, complete_url, multipart_url } }`. `url` is a [presigned PUT URL](#presigned-uploads). With `multipart: true` (required above `MEDIA_OBJECT_MAX_MB`): `method: 'multipart'`, `url: null`, and `upload.multipart` is the new [session](#multipart-uploads). Checks at init: `size_bytes` against `MEDIA_OBJECT_MAX_MB` (single part) or `MEDIA_MULTIPART_MAX_MB` (413), the namespace's policy (422 `media.namespace.policy_denied`, 413 `media.object.too_large`), every quota from the namespace up to the tenant including reservations (413 `media.quota.exceeded` / `media.quota.objects_exceeded`; the declared size is then reserved), the public-size invariant (422) and the [content type](#content-types) for the kind (415 `media.object.unsupported_type`) |
+| POST | `/` | init. `{ namespace (default the tenant's root; a name below it, full or relative), kind, visibility (default private), mime_type, size_bytes, filename, content_hash (expected sha256), metadata (≤ 16 KB; `metadata.class` declares the object's storage class, see [Tiering of native objects](#tiering-of-native-objects)), user_id, upload_ttl (60-86400 s), multipart, part_size }` returns 201 `{ id, object, upload: { method: 'PUT', url, token, expires_at, max_bytes, content_type, complete_url, multipart_url } }`. `url` is a [presigned PUT URL](#presigned-uploads). With `multipart: true` (required above `MEDIA_OBJECT_MAX_MB`): `method: 'multipart'`, `url: null`, and `upload.multipart` is the new [session](#multipart-uploads). Checks at init: `size_bytes` against `MEDIA_OBJECT_MAX_MB` (single part) or `MEDIA_MULTIPART_MAX_MB` (413), the namespace's policy (422 `media.namespace.policy_denied`, 413 `media.object.too_large`), every quota from the namespace up to the tenant including reservations (413 `media.quota.exceeded` / `media.quota.objects_exceeded`; the declared size is then reserved), the public-size invariant (422) and the [content type](#content-types) for the kind (415 `media.object.unsupported_type`) |
| POST | `/:id/upload-url` | a fresh presigned single-PUT URL for an uploading object (`{ ttl }` 60-86400 s) |
| POST | `/:id/upload-url` | a fresh presigned single-PUT URL for an uploading object (`{ ttl }` 60-86400 s) |
| PUT | `/:id/content` | the bytes, single part. Auth is the upload token (`?token=` from init, or `X-Upload-Token`) or the usual credential. Streams to disk computing sha256. Refuses more than the declared size or the limit (413; an object declared above `MEDIA_OBJECT_MAX_MB` goes multipart), a size mismatch (400), quota overrun (413), a type that does not suit the kind (415) and a PUT while a multipart session is open (409). Stored at `OBJECTS_PATH/<app>/<id>` with a verified local location. Mounted ahead of the JSON body parser, so any Content-Type is taken as raw bytes; the Content-Type becomes `mime_type` when init set none. Can be repeated while the object is `uploading` |
| PUT | `/:id/content` | the bytes, single part. Auth is the upload token (`?token=` from init, or `X-Upload-Token`) or the usual credential. Streams to disk computing sha256. Refuses more than the declared size or the limit (413; an object declared above `MEDIA_OBJECT_MAX_MB` goes multipart), a size mismatch (400), quota overrun (413), a type that does not suit the kind (415) and a PUT while a multipart session is open (409). Stored at `OBJECTS_PATH/<app>/<id>` with a verified local location. Mounted ahead of the JSON body parser, so any Content-Type is taken as raw bytes; the Content-Type becomes `mime_type` when init set none. Can be repeated while the object is `uploading` |
| POST | `/:id/complete` | checks the expected hash (422 `media.object.hash_mismatch`), the invariant, the quota and the bytes against the type (415 `media.object.content_mismatch`). Sets `ready` and sends the `media.object.uploaded` webhook. Also accepts the upload token |
| POST | `/:id/complete` | checks the expected hash (422 `media.object.hash_mismatch`), the invariant, the quota and the bytes against the type (415 `media.object.content_mismatch`). Sets `ready` and sends the `media.object.uploaded` webhook. Also accepts the upload token |
@@ -921,6 +921,17 @@ Each pass rotates the popularity counts, then:
@@ -921,6 +921,17 @@ Each pass rotates the popularity counts, then:
500, so no class is crowded out of its budget by a most-viewed prefix.
500, so no class is crowded out of its budget by a most-viewed prefix.
Budgets and minimum residency are per class (`media.storage_policy`); a refusal or back-off spends no slot.
Budgets and minimum residency are per class (`media.storage_policy`); a refusal or back-off spends no slot.
+**Classes.** Every native object is classified into one of the six storage classes (`video`, `image`,
+`download`, `game-asset`, `attachment`, `backup`), and each class has its own per-sweep budgets, minimum
+residency, hysteresis band and optional monthly R2 storage ceiling in `media.storage_policy`. `kind` and
+`mime_type` decide `video` and `image` (video/audio and their kinds, image and its kinds); everything else is
+`download` unless the app declared `game-asset`, `attachment` or `backup` for the object in its metadata:
+`metadata.class`, or the explicit `media_class` / `placement_class`. A declaration is honoured only when kind
+and mime type would say `download`, so content alone decides `video` and `image` (budgets and ceilings are
+global per class, so a declaration could otherwise bill a video to another class's); any other name is ignored.
+The declared class is what the sweep's eligibility, ranking and budgets read, and the class's
+R2 copies are what its ceiling counts, so a ceiling on `game-asset` refuses that class alone.
+
**Promotion** (`tiering.promote`): refused when the object is held, not ready, a sandbox object, or its
**Promotion** (`tiering.promote`): refused when the object is held, not ready, a sandbox object, or its
canonical copy is not verified on record (present, checked, not contradicted, and the object has a
canonical copy is not verified on record (present, checked, not contradicted, and the object has a
sha256: the `object.hash` job supplies it for older objects); then the gate (`dry_run`); then R2 must be
sha256: the `object.hash` job supplies it for older objects); then the gate (`dry_run`); then R2 must be
@@ -151,10 +151,26 @@ const fmtUsd = (v) => `$${Number(Number(v).toPrecision(4))}`;
@@ -151,10 +151,26 @@ const fmtUsd = (v) => `$${Number(Number(v).toPrecision(4))}`;
/** What each class's present R2 copies cost a month at list price, gross of R2's account-wide free tier: { video: 0.12, image: 0, ... }. */
/** What each class's present R2 copies cost a month at list price, gross of R2's account-wide free tier: { video: 0.12, image: 0, ... }. */
async function hotSpend() {
async function hotSpend() {
const out = Object.fromEntries(storagePolicy.CLASSES.map((c) => [c, 0]));
const out = Object.fromEntries(storagePolicy.CLASSES.map((c) => [c, 0]));
- const rows = await db.all(`SELECT o.kind, o.mime_type, SUM(COALESCE(l.size_bytes, o.size_bytes, 0))::bigint AS bytes
- FROM media_locations l JOIN media_objects o ON o.id = l.object_id
- WHERE l.provider = 'r2' AND l.state = 'present' GROUP BY o.kind, o.mime_type`);
- for (const r of rows) out[storagePolicy.classOf(r)] += hotUsdPerMonth(r.bytes);
+ // Aggregate in SQL: one row per (declared class, kind, mime_type) group, not per copy — a copy carries up to
+ // 16 KB of metadata, and R2 copies (the hot cache) run to many thousands; the groups are a few dozen. The three
+ // metadata keys classOf() reads are extracted here with the SQLite-compatible json_extract() the migrations
+ // define over the text metadata column (guard with json_valid, so unreadable JSON declares nothing, as
+ // declaredClass() does), lower-cased and trimmed like normalizeClass(), and summed per group. classOf() then
+ // maps each group's representative fields to its class — the one place the rules live.
+ const rows = await db.all(`SELECT grp.kind, grp.mime_type, grp.declared_class, grp.declared_media_class, grp.declared_placement_class,
+ SUM(grp.bytes)::bigint AS bytes
+ FROM (SELECT o.kind, o.mime_type,
+ CASE WHEN json_valid(o.metadata) THEN lower(trim(json_extract(o.metadata, '$.class'))) END AS declared_class,
+ CASE WHEN json_valid(o.metadata) THEN lower(trim(json_extract(o.metadata, '$.media_class'))) END AS declared_media_class,
+ CASE WHEN json_valid(o.metadata) THEN lower(trim(json_extract(o.metadata, '$.placement_class'))) END AS declared_placement_class,
+ COALESCE(l.size_bytes, o.size_bytes, 0) AS bytes
+ FROM media_locations l JOIN media_objects o ON o.id = l.object_id
+ WHERE l.provider = 'r2' AND l.state = 'present') grp
+ GROUP BY grp.kind, grp.mime_type, grp.declared_class, grp.declared_media_class, grp.declared_placement_class`);
+ for (const r of rows) {
+ out[storagePolicy.classOf({ kind: r.kind, mime_type: r.mime_type,
+ metadata: { class: r.declared_class, media_class: r.declared_media_class, placement_class: r.declared_placement_class } })] += hotUsdPerMonth(r.bytes);
+ }
return out;
return out;
}
}
@@ -104,8 +104,40 @@ function classPolicy(name) {
@@ -104,8 +104,40 @@ function classPolicy(name) {
return settings().classes[name];
return settings().classes[name];
}
}
-/** The placement class of a native object: video/audio → video, image/thumbnail → image, everything else → download. */
-function classOf(obj) {
+// An app may declare its object's class in the object's metadata: `class`, or the explicit `media_class` /
+// `placement_class`. Only one of CLASSES counts; anything else is ignored and the kind and mime type decide.
+// A declaration is honoured only when kind and mime type would otherwise classify the object as `download`,
+// and only to one of DECLARABLE_CLASSES: the roles content cannot express. Video and image are decided by
+// content, never declared. Budgets, hysteresis bands and R2 ceilings are global per class, so letting a
+// declaration win unconditionally would let one app put a large video on another class's budget, starve that
+// class, and be billed for it in the sweep's spend (server/objects/tiering.js hotSpend()).
+const DECLARED_KEYS = ['class', 'media_class', 'placement_class'];
+const DECLARABLE_CLASSES = ['game-asset', 'attachment', 'backup'];
+
+/** A known class name (case-folded), or null for anything else. */
+function normalizeClass(value) {
+ if (value == null || typeof value === 'object') return null;
+ const name = String(value).trim().toLowerCase();
+ return CLASSES.includes(name) ? name : null;
+}
+
+/** The class an app declared for the object — its metadata as an object or its JSON text — or null. */
+function declaredClass(obj) {
+ if (!obj) return null;
+ const direct = normalizeClass(obj.class);
+ if (direct) return direct;
+ let md = obj.metadata;
+ if (typeof md === 'string') { try { md = JSON.parse(md); } catch { return null; } }
+ if (!md || typeof md !== 'object') return null;
+ for (const key of DECLARED_KEYS) {
+ const name = normalizeClass(md[key]);
+ if (name) return name;
+ }
+ return null;
+}
+
+/** What kind and mime type alone decide: video/audio → video, image/thumbnail → image, everything else → download. */
+function classFromContent(obj) {
const kind = obj && obj.kind;
const kind = obj && obj.kind;
if (kind === 'vod' || kind === 'clip') return 'video';
if (kind === 'vod' || kind === 'clip') return 'video';
if (kind === 'thumbnail' || kind === 'screenshot' || kind === 'avatar') return 'image';
if (kind === 'thumbnail' || kind === 'screenshot' || kind === 'avatar') return 'image';
@@ -115,6 +147,20 @@ function classOf(obj) {
@@ -115,6 +147,20 @@ function classOf(obj) {
return 'download';
return 'download';
}
}
+/**
+ * The placement class of a native object: video/audio → video, image/thumbnail → image, everything else →
+ * download, unless the app declared one of DECLARABLE_CLASSES (game-asset, attachment, backup) for it and the
+ * content alone would say download. All six classes (CLASSES) act in the one sweep: this decides each object's
+ * hysteresis band, per-sweep budget and R2 storage ceiling. A declared class never overrides content, so one
+ * app cannot bill a video or image to another class's global budget, band or ceiling.
+ */
+function classOf(obj) {
+ const base = classFromContent(obj);
+ if (base !== 'download') return base;
+ const declared = declaredClass(obj);
+ return declared && DECLARABLE_CLASSES.includes(declared) ? declared : base;
+}
+
function budgetFor(name) {
function budgetFor(name) {
const { maxPromotionsPerSweep, maxDemotionsPerSweep } = classPolicy(name);
const { maxPromotionsPerSweep, maxDemotionsPerSweep } = classPolicy(name);
return { maxPromotionsPerSweep, maxDemotionsPerSweep };
return { maxPromotionsPerSweep, maxDemotionsPerSweep };
@@ -91,11 +91,11 @@ const crypto = require('crypto');
@@ -91,11 +91,11 @@ const crypto = require('crypto');
await storage.setSettings({ hotDiskPressurePct: 100, criticalDiskPct: 100, minFreeGb: 0 }, { reason: 'test: no pressure drain' });
await storage.setSettings({ hotDiskPressurePct: 100, criticalDiskPct: 100, minFreeGb: 0 }, { reason: 'test: no pressure drain' });
const today = popularity.dayOf(Date.now());
const today = popularity.dayOf(Date.now());
- const addObject = async ({ app = 'live', size = 4096, mime = 'application/octet-stream', verified = true, hash = true, canonical = 'local' } = {}) => {
+ const addObject = async ({ app = 'live', size = 4096, mime = 'application/octet-stream', verified = true, hash = true, canonical = 'local', meta = {} } = {}) => {
const bytes = crypto.randomBytes(size);
const bytes = crypto.randomBytes(size);
const sum = sha256(bytes);
const sum = sha256(bytes);
const id = await model.createObject({ app_id: app, kind: 'file', visibility: 'public', lifecycle_status: 'ready', mime_type: mime, size_bytes: size,
const id = await model.createObject({ app_id: app, kind: 'file', visibility: 'public', lifecycle_status: 'ready', mime_type: mime, size_bytes: size,
- content_hash: hash ? sum : null, canonical_provider: canonical, metadata: { filename: 'x.bin' } });
+ content_hash: hash ? sum : null, canonical_provider: canonical, metadata: { filename: 'x.bin', ...meta } });
let file = null;
let file = null;
if (canonical === 'local') {
if (canonical === 'local') {
file = path.join(process.env.OBJECTS_PATH, app, id);
file = path.join(process.env.OBJECTS_PATH, app, id);
@@ -773,6 +773,75 @@ const crypto = require('crypto');
@@ -773,6 +773,75 @@ const crypto = require('crypto');
for (const v of candViews) await db.run('INSERT INTO media_object_views_daily (object_id, day, unique_viewers, last_viewed_at) VALUES (?, ?, ?, ?)', [v.object_id, v.day, v.unique_viewers, v.last_viewed_at]);
for (const v of candViews) await db.run('INSERT INTO media_object_views_daily (object_id, day, unique_viewers, last_viewed_at) VALUES (?, ?, ?, ?)', [v.object_id, v.day, v.unique_viewers, v.last_viewed_at]);
console.log('✅ constraints: a provider failing its class (breaker or probe) refuses the move and keeps every copy; the monthly R2 storage ceiling closes a class in value order');
console.log('✅ constraints: a provider failing its class (breaker or probe) refuses the move and keeps every copy; the monthly R2 storage ceiling closes a class in value order');
+ // ── 9g. Every storage class through the one sweep: an app declares its object's class in the metadata ──
+ await db.run('DELETE FROM media_object_views_daily');
+ const att = await addObject({ meta: { class: 'attachment' } });
+ const bak = await addObject({ meta: { class: 'backup' } });
+ const dl0 = await addObject();
+ for (const o of [att, bak, dl0]) await setViews(o.id, [[0, 900]]);
+ assert.deepStrictEqual([sp.classOf(await model.getObject(att.id)), sp.classOf(await model.getObject(bak.id)), sp.classOf(await model.getObject(dl0.id))],
+ ['attachment', 'backup', 'download'], 'an app-declared class rides on the metadata the sweep reads');
+ await sp.set({ classes: { download: { maxPromotionsPerSweep: 0 }, attachment: { maxPromotionsPerSweep: 1 }, backup: { maxPromotionsPerSweep: 1 } } },
+ { reason: 'test: per-class budgets for attachment and backup' });
+ s = await tiering.runSweep();
+ assert.ok(await r2Row(att.id) && await r2Row(bak.id), 'attachment and backup promote under their own budgets');
+ assert.ok(!await r2Row(dl0.id) && (await decisions(dl0.id)).length === 0, 'the download budget of 0 spends nothing on them');
+ assert.strictEqual(JSON.parse((await last(att.id)).inputs).class, 'attachment');
+ assert.ok(/class attachment, value per dollar/.test((await last(att.id)).reason), (await last(att.id)).reason);
+ // A class's R2 spend counts its own copies: a declared class is never billed to its mime type's class.
+ const spendBefore = await tiering.hotSpend();
+ const gasPrice = tiering.hotUsdPerMonth(4096);
+ const gas = await addObject({ meta: { class: 'game-asset' } });
+ await inR2(gas, 30);
+ const spendAfter = await tiering.hotSpend();
+ assert.ok(Math.abs(spendAfter['game-asset'] - (spendBefore['game-asset'] + gasPrice)) < 1e-12, `the game-asset copy spends on game-asset: ${JSON.stringify(spendAfter)}`);
+ assert.ok(Math.abs(spendAfter.download - spendBefore.download) < 1e-12, 'and not on download, which its mime type alone would say');
+ // hotSpend aggregates in SQL: several copies in one (declared class, kind, mime type) group come back as
+ // one row with their bytes summed, rather than one row each carrying the copy's whole metadata.
+ for (let i = 0; i < 3; i++) await inR2(await addObject({ meta: { class: 'backup' } }), 30);
+ const realAll = db.all;
+ let groupedRows = null;
+ db.all = async (sql, params) => { const rows = await realAll.call(db, sql, params); if (/SUM\(grp\.bytes\)/.test(sql)) groupedRows = rows; return rows; };
+ const groupedSpend = await tiering.hotSpend();
+ db.all = realAll;
+ const group = groupedRows.filter((r) => r.declared_class === 'backup' && r.kind === 'file' && r.mime_type === 'application/octet-stream');
+ const groupCopies = await db.get(`SELECT COUNT(*) AS n, COALESCE(SUM(COALESCE(l.size_bytes, o.size_bytes, 0)), 0)::bigint AS bytes
+ FROM media_locations l JOIN media_objects o ON o.id = l.object_id
+ WHERE l.provider = 'r2' AND l.state = 'present' AND o.kind = 'file' AND o.mime_type = 'application/octet-stream'
+ AND lower(trim(json_extract(o.metadata, '$.class'))) = 'backup'`);
+ assert.ok(Number(groupCopies.n) >= 3, `several copies in the group (${groupCopies.n})`);
+ assert.strictEqual(group.length, 1, 'the copies are grouped into one row');
+ assert.strictEqual(Number(group[0].bytes), Number(groupCopies.bytes), 'their bytes are summed in that row');
+ assert.ok(Math.abs(groupedSpend.backup - tiering.hotUsdPerMonth(groupCopies.bytes)) < 1e-12, 'the class spend is the summed group');
+ // A demotion spends the class's own budget too.
+ const attR2 = await addObject({ meta: { class: 'attachment' } });
+ const dlR2 = await addObject();
+ await inR2(attR2, 30); await inR2(dlR2, 30);
+ await sp.set({ classes: { download: { maxDemotionsPerSweep: 0 }, attachment: { maxDemotionsPerSweep: 1 } } }, { reason: 'test: per-class demotion budgets' });
+ s = await tiering.runSweep();
+ assert.ok(!await r2Row(attR2.id) && await r2Row(dlR2.id), 'the attachment demotes under its own budget; the download budget of 0 keeps its copy');
+ assert.strictEqual(JSON.parse((await last(attR2.id)).inputs).class, 'attachment');
+ // A ceiling on one class refuses only that class.
+ const gAs = [await addObject({ meta: { class: 'game-asset' } }), await addObject({ meta: { class: 'game-asset' } })];
+ const dl3 = await addObject();
+ for (const o of [...gAs, dl3]) await setViews(o.id, [[0, 900]]);
+ await sp.set({ classes: { 'game-asset': { maxHotUsdPerMonth: 0 }, download: { maxPromotionsPerSweep: 3, maxDemotionsPerSweep: 20 } } },
+ { reason: 'test: a game-asset ceiling' });
+ s = await tiering.runSweep();
+ assert.ok(!await r2Row(gAs[0].id) && !await r2Row(gAs[1].id), 'the game-asset ceiling refuses the class');
+ assert.ok(await r2Row(dl3.id), 'the download class is unaffected');
+ d = await last(gAs[0].id);
+ assert.deepStrictEqual([d.action, d.outcome, JSON.parse(d.inputs).class], ['promote', 'refused', 'game-asset']);
+ assert.ok(/class game-asset has maxHotUsdPerMonth \$0/.test(d.reason), d.reason);
+ assert.strictEqual((await decisions(gAs[1].id)).length, 0, 'the refusal closed game-asset only');
+ assert.deepStrictEqual([s.constraints.hot_usd_per_month['game-asset'].ceiling, s.constraints.refused.budget], [0, 1], JSON.stringify(s.constraints));
+ console.log('✅ every class through one sweep: declared attachment/backup promote and demote under their own budgets, and a game-asset ceiling refuses only that class');
+
+ await sp.set({ classes: { download: { maxPromotionsPerSweep: 3, maxDemotionsPerSweep: 20 }, attachment: { maxPromotionsPerSweep: 3, maxDemotionsPerSweep: 10 },
+ backup: { maxPromotionsPerSweep: 3, maxDemotionsPerSweep: 10 }, 'game-asset': { maxHotUsdPerMonth: 1e9 } } }, { reason: 'test: class policy back' });
+ await db.run('DELETE FROM media_object_views_daily');
+ for (const v of candViews) await db.run('INSERT INTO media_object_views_daily (object_id, day, unique_viewers, last_viewed_at) VALUES (?, ?, ?, ?)', [v.object_id, v.day, v.unique_viewers, v.last_viewed_at]);
+
// ── 10. The storage sweep runs it (step 4); a restore drill never does ──
// ── 10. The storage sweep runs it (step 4); a restore drill never does ──
const vodSweep = await storage.runSweep();
const vodSweep = await storage.runSweep();
assert.ok(vodSweep.objects && vodSweep.objects.gate === true && typeof vodSweep.objects.promoted === 'number', JSON.stringify(vodSweep));
assert.ok(vodSweep.objects && vodSweep.objects.gate === true && typeof vodSweep.objects.promoted === 'number', JSON.stringify(vodSweep));
@@ -28,11 +28,38 @@ const path = require('path');
@@ -28,11 +28,38 @@ const path = require('path');
}
}
assert.strictEqual(policy.mayMove({ class: 'video', lastMovedAt: 1000, now: 1000 }), true, 'zero residency is off');
assert.strictEqual(policy.mayMove({ class: 'video', lastMovedAt: 1000, now: 1000 }), true, 'zero residency is off');
assert.throws(() => policy.budgetFor('unknown'), RangeError);
assert.throws(() => policy.budgetFor('unknown'), RangeError);
- const cls = (kind, mime_type) => policy.classOf({ kind, mime_type });
+ const cls = (kind, mime_type, metadata) => policy.classOf({ kind, mime_type, metadata });
assert.deepStrictEqual([cls('vod'), cls('clip'), cls('file', 'audio/mpeg'), cls('file', 'Video/MP4'), cls('asset', 'video/webm')], Array(5).fill('video'));
assert.deepStrictEqual([cls('vod'), cls('clip'), cls('file', 'audio/mpeg'), cls('file', 'Video/MP4'), cls('asset', 'video/webm')], Array(5).fill('video'));
assert.deepStrictEqual([cls('thumbnail'), cls('screenshot'), cls('avatar'), cls('file', 'image/png')], Array(4).fill('image'));
assert.deepStrictEqual([cls('thumbnail'), cls('screenshot'), cls('avatar'), cls('file', 'image/png')], Array(4).fill('image'));
assert.deepStrictEqual([cls('file', 'application/zip'), cls('file'), cls('asset', null), policy.classOf(null)], Array(4).fill('download'));
assert.deepStrictEqual([cls('file', 'application/zip'), cls('file'), cls('asset', null), policy.classOf(null)], Array(4).fill('download'));
+ // Every class acts: an app declares its object's role in the object's metadata (class, or the explicit
+ // media_class / placement_class). A declaration is honoured only when kind and mime type would say
+ // download, and only for game-asset / attachment / backup: content alone decides video and image.
+ assert.deepStrictEqual([
+ cls('file', 'application/zip', { class: 'attachment' }),
+ cls('file', null, { class: 'backup' }),
+ cls('asset', null, { media_class: 'game-asset' }),
+ cls('file', 'video/mp4', { placement_class: 'game-asset' }),
+ cls('file', 'application/octet-stream', { class: 'Video' }),
+ ], ['attachment', 'backup', 'game-asset', 'video', 'download']);
+ assert.strictEqual(policy.classOf({ kind: 'file', metadata: JSON.stringify({ class: 'backup' }) }), 'backup', 'metadata read as its JSON text too');
+ assert.deepStrictEqual([cls('file', 'application/zip', { class: 'nope' }), cls('file', null, { class: 7 }),
+ cls('file', null, 'not json metadata'), cls('file', null, { class: ['attachment'] })], Array(4).fill('download'),
+ 'an unknown, non-string or unreadable declaration is ignored and kind/mime decide');
+ // Class escape: a declaration can never override content, so one app cannot bill a large video (or an
+ // image) to another class's global budget, hysteresis band or R2 ceiling.
+ assert.strictEqual(cls('file', 'video/mp4', { class: 'backup' }), 'video', 'a declared backup does not steal video');
+ assert.strictEqual(cls('vod', null, { class: 'attachment' }), 'video', 'a VOD kind stays video');
+ assert.strictEqual(cls('file', 'image/png', { placement_class: 'attachment' }), 'image', 'an image stays image');
+ assert.strictEqual(cls('thumbnail', null, { media_class: 'game-asset' }), 'image', 'a thumbnail stays image');
+ assert.strictEqual(cls('file', 'application/zip', { class: 'backup' }), 'backup', 'a download may be a backup');
+ assert.strictEqual(cls('file', 'application/zip', { class: 'video' }), 'download', 'but never declared video');
+ assert.strictEqual(cls('file', 'application/zip', { class: 'image' }), 'download', 'nor declared image');
+ assert.deepStrictEqual([policy.budgetFor('attachment'), policy.hotCeilingFor('backup'), policy.bandFor('game-asset')],
+ [{ maxPromotionsPerSweep: 3, maxDemotionsPerSweep: 10 }, null, { promoteReadsPerHour: 60, demoteReadsPerHour: 6 }],
+ 'the classes an app declares have their own policy like any other');
+
await policy.set({ classes: { image: { maxPromotionsPerSweep: 2, maxDemotionsPerSweep: 4, minResidencyMs: 60_000 } } },
await policy.set({ classes: { image: { maxPromotionsPerSweep: 2, maxDemotionsPerSweep: 4, minResidencyMs: 60_000 } } },
{ actor: { type: 'service', id: 'media-test' }, reason: 'test image placement policy' });
{ actor: { type: 'service', id: 'media-test' }, reason: 'test image placement policy' });
assert.strictEqual(policy.get().revision(), 2, 'set creates a new revision');
assert.strictEqual(policy.get().revision(), 2, 'set creates a new revision');