openvibers/openvibe.events Commits
Move pruned events into a replay tier and read both with one cursor
Changes
@@@ -40,8 +40,21 @@ VALKEY_PREFIX=#NATS_STREAM=OV_EVENTS_STREAM#EVENTS_CARRIERS=pg-v1,valkey-v1,nats-v1,nats-js-v1#EVENTS_CARRIERS_DISABLED=--# Events (and their deliveries) older than this are pruned; resuming from before it yields a gap.++# Retention in tiers (ADR-042 decision 8; README "Retention: three tiers"). Hot events (and their deliveries) older++# than this move to the replay tier (events_archive) at the hourly prune.EVENTS_RETENTION_DAYS=30++# Days (from receipt) an event stays readable from the replay tier: pulls, SSE resume and by-id reads; resuming from++# before it yields a gap. 0 = no replay tier: old events are deleted, as before, and events_archive is emptied.++EVENTS_REPLAY_RETENTION_DAYS=365++# The archive tier, operator only (node scripts/events-archive.js export|restore --month YYYY-MM): a local directory,++# or an S3-compatible bucket when EVENTS_ARCHIVE_S3_BUCKET is set (path-style, https; SigV4, no dependency).++#EVENTS_ARCHIVE_DIR=/var/lib/openvibe/events-archive++#EVENTS_ARCHIVE_S3_ENDPOINT=https://s3.example.com++#EVENTS_ARCHIVE_S3_BUCKET=++#EVENTS_ARCHIVE_S3_REGION=us-east-1++#EVENTS_ARCHIVE_S3_ACCESS_KEY_ID=++#EVENTS_ARCHIVE_S3_SECRET_ACCESS_KEY=++#EVENTS_ARCHIVE_S3_PREFIX=events/# Accepted event ids are remembered this long, so late re-publishes stay duplicates.EVENTS_RECEIPT_RETENTION_DAYS=90#EVENTS_PRUNE_INTERVAL_MS=3600000@@@ -112,7 +112,31 @@ Each delivery is `POST <endpoint>` with body `{ "event": <envelope>, "seq": n }`Operators: `GET /api/v1/deliveries?status=dead` is the dead-letter queue; `POST /api/v1/deliveries/replay { subscription_id, event_ids: [...] }` or `{ subscription_id, from_seq }` requeues retained events (that is also how a new subscription catches up on history).--Pull consumers: start with `GET /api/v1/events?topic=media.vod.*&limit=100`, then request `GET /api/v1/events?topic=media.vod.*&after=<next_cursor>&limit=100`, using the previous response's opaque `next_cursor` ([ADR-042](https://github.com/OpenVibers/OpenVibe.Contracts/blob/main/docs/adr/ADR-042-events-fabric.md) decision 7). The response carries `{ events: [{ seq, cursor, event }], next_cursor, next_after_seq, latest_seq }`, plus `gap: { from_seq, to_seq }` when the position is older than retention or from another retention epoch. Use `next_cursor` even when no events match, since it advances past scanned events. Numeric `after_seq=<position>` remains accepted for one release; when both are supplied, `after` takes precedence. The numeric `seq`, `next_after_seq` and `latest_seq` fields also remain for one release. `PUT /api/v1/checkpoints { topic, cursor }` stores a consumer's position here if it has nowhere better (an opaque cursor string is accepted in place of the integer; the answer carries its `epoch`, `carrier` and `next_cursor`).++Pull consumers: start with `GET /api/v1/events?topic=media.vod.*&limit=100`, then request `GET /api/v1/events?topic=media.vod.*&after=<next_cursor>&limit=100`, using the previous response's opaque `next_cursor` ([ADR-042](https://github.com/OpenVibers/OpenVibe.Contracts/blob/main/docs/adr/ADR-042-events-fabric.md) decision 7). The response carries `{ events: [{ seq, cursor, event }], next_cursor, next_after_seq, latest_seq }`, plus `gap: { from_seq, to_seq }` when the position is older than both the hot store and the replay tier (below) or from another retention epoch. Use `next_cursor` even when no events match, since it advances past scanned events. Numeric `after_seq=<position>` remains accepted for one release; when both are supplied, `after` takes precedence. The numeric `seq`, `next_after_seq` and `latest_seq` fields also remain for one release. `PUT /api/v1/checkpoints { topic, cursor }` stores a consumer's position here if it has nowhere better (an opaque cursor string is accepted in place of the integer; the answer carries its `epoch`, `carrier` and `next_cursor`).++++### Retention: three tiers++++[ADR-042](https://github.com/OpenVibers/OpenVibe.Contracts/blob/main/docs/adr/ADR-042-events-fabric.md) decision 8. An event lives in one tier at a time:++++| Tier | Where | Kept | Read by |++|---|---|---|---|++| hot | `events` (with its deliveries) | `EVENTS_RETENTION_DAYS` (30) | everything: delivery, pulls, SSE, by id, replay to subscriptions |++| replay | `events_archive`, same database: the filter columns plus a gzip'd body, no delivery rows | `EVENTS_REPLAY_RETENTION_DAYS` (365) from receipt; `0` = no replay tier | pulls, SSE resume and `GET /api/v1/events/:id`, with the same cursor |++| archive | one NDJSON object per month (gzip, with a manifest), outside the database | until an operator deletes it | nothing online; an operator restores a month into replay |++++The hourly prune (`EVENTS_PRUNE_INTERVAL_MS`) moves each hot event past `EVENTS_RETENTION_DAYS` into `events_archive` in the transaction that deletes it: 500 rows at a time, locked `FOR UPDATE SKIP LOCKED`, inserted idempotently on `event_id`, so two Events processes pruning at once each move different rows and no event is moved twice. Its deliveries go with the hot row (operator replay to a subscription, `POST /api/v1/deliveries/replay`, reaches hot events only). Replay rows past `EVENTS_REPLAY_RETENTION_DAYS` are then deleted. A pull or SSE resume runs from a cursor in the replay tier into the hot store without a `gap`; `gap` (the same shape) still means the position is older than both tiers or from another epoch. Redaction tombstones replay rows exactly as hot ones, the sandbox and environment rules read them the same way, and a re-publish of an archived `event_id` is still `duplicate: true` with its original `seq` and `cursor`. Sandbox events never enter replay: they are deleted at `EVENTS_APP_SANDBOX_RETENTION_DAYS`, as before; publish receipts keep `EVENTS_RECEIPT_RETENTION_DAYS`.++++`EVENTS_REPLAY_RETENTION_DAYS=0` keeps the old behaviour exactly: events past `EVENTS_RETENTION_DAYS` are deleted, and the next prune also empties `events_archive`, because no rule would prune the rows left there otherwise. `GET /limits.json` publishes the window as `replay_retention_days`.++++The archive tier is an operator job, never run automatically (`DATABASE_URL` as for the service):++++```sh++node scripts/events-archive.js status # replay rows per month, oldest hot event++node scripts/events-archive.js export --month 2026-01 [--older-than-days 90]++node scripts/events-archive.js restore --month 2026-01 [--hold-days 30]++```++++`export` writes the closed month's replay rows older than the cutoff to `<YYYY>/events-<YYYY-MM>.ndjson.gz` (one row per line: every column of the hot row, payload and actor as their stored JSON text, plus `epoch` and `archived_at`) and `<YYYY>/events-<YYYY-MM>.json` (`count`, `sha256`). It deletes those rows from `events_archive` only after reading the object back and matching both, and only rows unchanged since it read them (a row redacted meanwhile stays; rerun). Re-running, or exporting the rest of a month later, merges into the month's object. `restore` checks the object against its manifest, then inserts it back into `events_archive`, idempotent on `event_id`, held for `--hold-days` so the replay prune does not delete it at once. A redaction published after a month was exported does not reach its object, so restore only what you need. Export and restore of one month are serialised by a PostgreSQL advisory lock. Storage (`scripts/archive-storage.js`): a local directory, `EVENTS_ARCHIVE_DIR` (default `data/archive`), or an S3-compatible bucket when `EVENTS_ARCHIVE_S3_BUCKET` is set (`EVENTS_ARCHIVE_S3_ENDPOINT`, `_REGION`, `_ACCESS_KEY_ID`, `_SECRET_ACCESS_KEY`, `_PREFIX`; path-style, SigV4 signed with node:crypto and sent with node's `fetch`, no dependency). Each object is built in memory.## The fabric: carriers, the planner and explain@@@ -252,14 +276,14 @@ void connect();- Conversely, a user JWT that carries `typ` or `purpose` (a ticket, a FedCM assertion) is not a session here.- A person's topic: `network.notification.created` (Network's outbox; visibility `subject`, subject the recipient). `topics=network.notification.*` streams a person's own notifications and nobody else's. There is no `user:<id>` topic: it is not a valid pattern (400 `realtime.bad_topic`), and subject visibility already does its job.- Visibility: `public` events go to anyone subscribed to the topic; `subject` events only to the user whose subject id (`usr_…`) is the event's `actor.id` or its user `subject.id`; `internal` events never reach a browser. A guessed topic yields nothing.--- Resume: the SSE `id` is the cursor (ADR-042 decision 7; a bare seq is accepted for one release), so the browser's automatic `Last-Event-ID` (or `?last_event_id=`) replays what was missed. A cursor older than retention — or from another retention epoch (`reason: "epoch"`) — first gets `event: gap` (`{ reason, from_seq, to_seq }`), as does a replay that hits `REALTIME_REPLAY_MAX`.++- Resume: the SSE `id` is the cursor (ADR-042 decision 7; a bare seq is accepted for one release), so the browser's automatic `Last-Event-ID` (or `?last_event_id=`) replays what was missed. A cursor older than both the hot store and the replay tier — or from another retention epoch (`reason: "epoch"`) — first gets `event: gap` (`{ reason, from_seq, to_seq }`), as does a replay that hits `REALTIME_REPLAY_MAX`.- Public replay window: browsers (signed out or signed in) are replayed `public` events received in the last `REALTIME_PUBLIC_REPLAY_SECONDS` (300; 0 = none), enough to ride out a reconnect. Older public events are not replayed: the stream opens with `event: gap` (`reason: "public_window"`, `window_seconds`), and the client refetches state from the owning service. `subject` events addressed to the viewer, and service viewers, keep the whole retention. This keeps the stream from paging through a month of public history, chat lines included; nothing in the network needs more (no browser surface replays public events today).- Redacted events are replayed as their tombstones ([Redaction](#redaction)).- Heartbeat comment every 25 s; at most 20 topics per connection and `REALTIME_MAX_CONNECTIONS` (2000) overall; CORS with credentials for `https://*.openvibe.*` only.## Owns--- `events`, `subscriptions`, `deliveries`, `consumer_checkpoints`, `idempotency_receipts`, `app_revocations`, `app_usage` (PostgreSQL `ov_events`, ADR-035; the realtime fan-out is in-process, one Events process)++- `events`, `events_archive` (the replay tier), `subscriptions`, `deliveries`, `consumer_checkpoints`, `idempotency_receipts`, `app_revocations`, `app_usage` (PostgreSQL `ov_events`, ADR-035; the realtime fan-out is in-process, one Events process)- developer-app event scope, sandbox separation and per-project Events quotas (ADR-014)- canonical event envelope (event_id, trace_id, type, version, source, actor, subject + revision, payload)- priority classes `critical|important|low`, loop guards, backpressure, DLQ and replay@@@ -280,6 +304,7 @@ void connect();- event persists before consumer delivery (`test/delivery.test.js`)- kill a consumer mid-processing; replay creates exactly one effect (`test/outbox-inbox.test.js`)- browser reconnect resumes from a cursor or reports a gap (`test/realtime.test.js`)++- a hot-pruned event stays readable through replay, a cursor across the tier boundary reports no false gap, two concurrent prunes move each event once, replay pruning answers `gap`, export → restore round-trips byte for byte, and `EVENTS_REPLAY_RETENTION_DAYS=0` behaves as before the tiers (`test/archive.test.js`)- a guessed private topic yields no data (`test/realtime.test.js`)- a developer app cannot publish, read or subscribe outside its project, sandbox never meets production, app webhooks reach public addresses only, quotas hold (`test/apps.test.js`)- a project's publishing and deliveries are counted per hour, refusals and failed attempts as errors, and each closed hour is sent once as `events.usage.recorded`, never visible to apps (`test/usage.test.js`)@@@ -20,12 +20,13 @@"capabilitiesRegistered": true,"deployed": true,"deployment": "Deployed with `sudo ovhost deploy events`: /opt/openvibe.events, unit openvibe-events, 127.0.0.1:4300; public at https://events.openvibe.network (realtime SSE, health, token-guarded /api/v1/events, /api/v1/subscriptions, /api/v1/checkpoints; /api/v1/deliveries host-local); real traffic since 2026-09-23 (Live, Media, Chat, Games, Network and OpenRe publish; on 2026-09-23 8 subscriptions, held by Search, Tips, VIP, News and Reviews); restore drill passed 2026-09-23",-- "updated": "2026-09-29",++ "updated": "2026-10-04","features": ["durable event store with the canonical envelope","subscriptions with signed webhook delivery (v1 and v2), retry, dead letters and replay","priority classes and backpressure","pull consumers with cursors and checkpoints",++ "tiered retention (ADR-042 decision 8): hot events past EVENTS_RETENTION_DAYS move to the events_archive replay tier (EVENTS_REPLAY_RETENTION_DAYS, 365; 0 = off), read with the same cursor; monthly NDJSON archive export and restore by operator job (scripts/events-archive.js; local directory or S3-compatible bucket)","realtime SSE with cursor resume, gap reports and Network tickets","developer-app publish, read and subscribe with sandbox separation, SSRF-guarded webhooks and project quotas","hourly project usage rollups (events.usage.recorded)",nnew file mode 100644@@@ -0,0 +1,30 @@++-- PostgreSQL seed for the 0006_events_archive cutover rehearsal (docs/cutover-12.md). Loaded after migrations++-- 0001-0005, before 0006: the shape production has today. Times are relative to now() (ms), so the rehearsal's prune++-- with the defaults (EVENTS_RETENTION_DAYS=30, EVENTS_REPLAY_RETENTION_DAYS=365) sees:++-- seq 1 production, 400 days old -> moved to replay, then dropped by the replay prune (past 365 days)++-- seq 2 production, 60 days old -> moved to replay (kept), its two deliveries go with the hot row++-- seq 3 production, 45 days old, tombstoned (redacted) -> moved to replay, still a tombstone++-- seq 4 sandbox, 45 days old -> deleted (sandbox never enters replay)++-- seq 5 production, 1 day old -> stays hot++-- The epoch stays 0 and the sequence counter at 5, as a live store's would.++INSERT INTO events (id, seq, event_type, version, source, actor, subject_type, subject_id, subject_revision, trace_id, priority,++ visibility, occurred_at, received_at, payload, hops, publisher, request_id, project_id, env, size_bytes, redacted_at, redacted_by, redacts)++SELECT v.id, v.seq, v.event_type, 1, v.source, '{"type":"service","id":"svc:live"}', 'stream', v.subject_id, NULL,++ 'trc_rehearsal_' || v.seq, 'important', 'public', '2026-01-01T00:00:00.000Z',++ (extract(epoch FROM now()) * 1000)::bigint - v.days::bigint * 86400000, v.payload, 1, v.publisher, NULL, v.project_id, v.env,++ length(v.payload), v.redacted_at, v.redacted_by, 0++FROM (VALUES++ ('evt_rehearsal_0001', 1, 'live.stream.started', 'live', 'str_1', '{"title":"oldest"}', 'svc:live', NULL, 'production', 400, NULL::bigint, NULL),++ ('evt_rehearsal_0002', 2, 'live.stream.started', 'live', 'str_2', '{"title":"replay ✓"}', 'svc:live', NULL, 'production', 60, NULL, NULL),++ ('evt_rehearsal_0003', 3, 'live.stream.started', 'live', 'str_3', '{"redacted":true,"redacted_at":1,"redacted_by":"evt_rehearsal_0005"}', 'svc:live', NULL, 'production', 45, 1, 'evt_rehearsal_0005'),++ ('evt_rehearsal_0004', 4, 'app.prj_rehearsal.tick', 'app', 'str_4', '{"n":4}', 'app:prj_rehearsal', 'prj_rehearsal', 'sandbox', 45, NULL, NULL),++ ('evt_rehearsal_0005', 5, 'live.stream.started', 'live', 'str_5', '{"title":"hot"}', 'svc:live', NULL, 'production', 1, NULL, NULL)++) AS v(id, seq, event_type, source, subject_id, payload, publisher, project_id, env, days, redacted_at, redacted_by);++++UPDATE sequences SET value = 5 WHERE name = 'events';++++INSERT INTO subscriptions (id, consumer, topic_pattern, endpoint, secret, enabled, created_at, updated_at)++VALUES ('sub_rehearsal', 'search', 'live.stream.*', 'http://127.0.0.1:9/hook', 'rehearsal-secret-rehearsal-secret', 1, 0, 0);++INSERT INTO deliveries (event_id, subscription_id, seq, priority, status, delivered_at, created_at, updated_at)++VALUES ('evt_rehearsal_0002', 'sub_rehearsal', 2, 1, 'delivered', 0, 0, 0),++ ('evt_rehearsal_0005', 'sub_rehearsal', 5, 1, 'pending', NULL, 0, 0);nnew file mode 100644@@@ -0,0 +1,56 @@++# Cutover: the replay tier (`0006_events_archive`, ADR-042 decision 8)++++One additive migration (`-- phase: expand`): a new table `events_archive` and its four indexes. Nothing in `events`,++`deliveries` or any other table changes; no backfill runs at migration time. The release that ships it starts moving++hot events past `EVENTS_RETENTION_DAYS` (30) into `events_archive` at its hourly prune instead of deleting them, and++reads span both tables. `EVENTS_REPLAY_RETENTION_DAYS` defaults to 365; `0` is the old behaviour exactly.++++## Before++++1. Health: `ov access run openvibe-ovh health events` answers ` ready `.++2. Backup: the host's nightly `ov_events` dump is from today, or take one (`pg_dump -Fc ov_events`, OpenVibe.Host++ docs/backups.md). The migration only adds a table, so the backup is for the release, not for the migration.++3. Size the first prune: `SELECT COUNT(*) FROM events WHERE received_at < (extract(epoch FROM now()) * 1000)::bigint - 30 * 86400000;`++ is 0 or close to it on a store that prunes hourly today; whatever it is, the prune moves 500 rows per transaction and++ at most 100 000 per run, the next run continues.++++## Cutover++++1. Deploy the release (the pipeline's `ovhost deploy events`). On start the service applies `0006_events_archive.sql`++ as the owner role (`DATABASE_DIRECT_URL`): `CREATE TABLE events_archive …` plus its indexes. On an empty table this++ takes milliseconds and holds no lock on `events`.++2. `/etc/openvibe/events.env`: nothing to add for the default (365 days of replay). To keep today's behaviour, set++ `EVENTS_REPLAY_RETENTION_DAYS=0` and restart.++3. Check: `curl -s https://events.openvibe.network/limits.json` lists `replay_retention_days`; after the first prune++ (start + up to `EVENTS_PRUNE_INTERVAL_MS`) the journal (`ov access run openvibe-ovh journal openvibe-events.service 200`)++ shows `[retention] pruned N hot events (N moved to replay), …` when anything was due, and++ `node scripts/events-archive.js status` (on the host, with the service env) lists the replay rows per month.++++The archive tier (`scripts/events-archive.js export|restore --month YYYY-MM`) is an operator job, never part of this++cutover; see README "Retention: three tiers".++++## Rollback++++- Release only: roll back to the previous release. It never reads `events_archive`; the table and the rows moved into++ it stay, unread, and the old release deletes hot events past retention as before. Rolling forward again finds them++ in place (cursors into them keep working, the epoch is unchanged).++- The table too (only if the release is abandoned): after rolling back, `DROP TABLE events_archive;` as the owner and++ delete its row from the migrations ledger, `DELETE FROM ov_migrations WHERE id = '0006';`++ (the ledger table of openvibe-sdk/db). The events in it are lost; restore them from the backup if they matter.++++## Rehearsal++++The seed is PostgreSQL ([cutover-12-seed.sql](cutover-12-seed.sql), loaded after 0001–0005, before 0006): five hot++events at 400, 60, 45 (a tombstone), 45 (sandbox) and 1 day(s) old, a subscription and two deliveries. The commands++apply 0006 through the service's own `openDb` (a second run applies nothing), run the tiered prune with the++defaults, and check the result: the 400-day event is gone from both tiers, the 60- and 45-day production events are in++the replay tier (the tombstone still a tombstone, the delivery gone with the hot row), the sandbox event is deleted, the++1-day event stays hot with its pending delivery, one scan from the start spans both tiers in seq order, a by-id read++finds the archived event, and a second prune moves nothing.++++```rehearse++seed: docs/cutover-12-seed.sql++node scripts/events-archive.js status++node -e "const a=require('assert');const {load}=require('./server/config');const {openDb,createStore}=require('./server/store');(async()=>{const db=await openDb(load(),{log:{log(){},warn(){}}});const s=createStore(db);const r=await s.prune({retentionDays:30,replayRetentionDays:365});console.log(JSON.stringify(r));a.strictEqual(r.archived,3);a.strictEqual(r.replay,1);const n=async(t)=>(await db.prepare('SELECT COUNT(*)::int AS n FROM '+t).get()).n;a.strictEqual(await n('events'),1);a.strictEqual(await n('events_archive'),2);a.strictEqual(await n('deliveries'),1);const {rows}=await s.scan(0);a.deepStrictEqual(rows.map(x=>x.seq),[2,3,5]);a.strictEqual(rows[1].redacted_by,'evt_rehearsal_0005');const e=await s.getEvent('evt_rehearsal_0002');a.strictEqual(JSON.parse(e.payload).title,'replay ✓');a.strictEqual(await s.getEvent('evt_rehearsal_0001'),null);a.strictEqual((await s.prune({retentionDays:30,replayRetentionDays:365})).archived,0);await db.close();console.log('rehearsal ok')})().catch(e=>{console.error(e);process.exit(1)})"++node scripts/events-archive.js status++```nnew file mode 100644@@@ -0,0 +1,38 @@++-- phase: expand++-- ADR-042 decision 8: three tiers. hot = `events` (pruned at retention.hot, EVENTS_RETENTION_DAYS); replay =++-- `events_archive` in the same database, pruned at retention.replay (EVENTS_REPLAY_RETENTION_DAYS); archive = monthly++-- NDJSON objects outside the database (scripts/events-archive.js, operator only). The prune MOVES a hot event here in++-- the transaction that deletes it, so a position is always in exactly one tier. No delivery rows follow it.++--++-- A row keeps, as columns, what the read path filters and orders on: identity, topic, the cursor position (seq in the++-- epoch it was moved in: the same pair server/cursor.js encodes), owner and visibility, the redaction markers and++-- received_at. Everything else of the event row (actor, payload, trace, …) is one gzip'd JSON object in `body`++-- (server/archive.js), restored to the hot row's exact column values on read. Additive only: nothing in `events` changes.++CREATE TABLE events_archive (++ id text COLLATE "C" PRIMARY KEY,++ seq bigint NOT NULL,++ epoch bigint NOT NULL,++ event_type text COLLATE "C" NOT NULL,++ source text COLLATE "C" NOT NULL,++ subject_type text COLLATE "C" NOT NULL,++ subject_id text COLLATE "C" NOT NULL,++ visibility text COLLATE "C" NOT NULL,++ project_id text COLLATE "C",++ env text COLLATE "C" NOT NULL DEFAULT 'production',++ received_at bigint NOT NULL,++ redacted_at bigint,++ redacts bigint NOT NULL DEFAULT 0,++ body bytea NOT NULL,++ archived_at bigint NOT NULL,++ -- An operator restore (scripts/events-archive.js restore) keeps the rows it brings back past retention.replay++ -- until this time (ms); NULL for rows the prune moved.++ hold_until bigint++);++-- Pull and scan: seq order within the current epoch (also the one position per epoch: a re-imported store whose seq++-- collides with an archived row of the same epoch makes the move fail instead of silently losing the hot row).++CREATE UNIQUE INDEX idx_events_archive_position ON events_archive(epoch, seq);++CREATE INDEX idx_events_archive_type_seq ON events_archive(event_type, seq);++-- Replay pruning and the monthly export.++CREATE INDEX idx_events_archive_received ON events_archive(received_at);++-- Redaction by subject reaches archived rows too.++CREATE INDEX idx_events_archive_subject ON events_archive(source, subject_type, subject_id);nnew file mode 100644@@@ -0,0 +1,112 @@++'use strict';++/**++ * Where the archive tier's monthly objects live (ADR-042 decision 8; scripts/events-archive.js only — never an online++ * path). Two adapters with one shape, { name, put(key, buf), get(key) -> Buffer | null }:++ *++ * local a directory, EVENTS_ARCHIVE_DIR (default data/archive). A put writes a temporary file, fsyncs it and++ * renames it into place, so an object is either whole or absent.++ * s3 an S3-compatible bucket, only when EVENTS_ARCHIVE_S3_BUCKET is set: path-style requests to++ * EVENTS_ARCHIVE_S3_ENDPOINT (https://…), signed with AWS Signature V4 by node:crypto and sent with node's++ * fetch — no dependency. EVENTS_ARCHIVE_S3_REGION (default us-east-1), EVENTS_ARCHIVE_S3_ACCESS_KEY_ID,++ * EVENTS_ARCHIVE_S3_SECRET_ACCESS_KEY, EVENTS_ARCHIVE_S3_PREFIX (default events/). The body is signed too++ * (x-amz-content-sha256), so the store refuses a body that changed on the way.++ */++const fs = require('fs');++const path = require('path');++const crypto = require('crypto');++++const KEY_RE = /^[a-z0-9][a-z0-9._/-]{0,200}$/;++const checkKey = (key) => {++ if (!KEY_RE.test(key) || key.includes('..')) throw new Error(`archive storage: bad object key ${JSON.stringify(key)}`);++ return key;++};++++function localStorage({ dir }) {++ const root = path.resolve(dir);++ return {++ name: `local:${root}`,++ async put(key, buf) {++ const file = path.join(root, checkKey(key));++ fs.mkdirSync(path.dirname(file), { recursive: true });++ const tmp = `${file}.tmp-${process.pid}-${crypto.randomBytes(4).toString('hex')}`;++ const fd = fs.openSync(tmp, 'w', 0o640);++ try { fs.writeSync(fd, buf); fs.fsyncSync(fd); } finally { fs.closeSync(fd); }++ fs.renameSync(tmp, file);++ },++ async get(key) {++ try { return fs.readFileSync(path.join(root, checkKey(key))); } catch (err) { if (err.code === 'ENOENT') return null; throw err; }++ },++ };++}++++const sha256 = (data) => crypto.createHash('sha256').update(data).digest('hex');++const hmac = (key, data) => crypto.createHmac('sha256', key).update(data).digest();++/** RFC 3986 encoding of one path segment, as SigV4 wants it. */++const encodeSegment = (s) => encodeURIComponent(s).replace(/[!'()*]/g, c => `%${c.charCodeAt(0).toString(16).toUpperCase()}`);++++/**++ * AWS Signature V4 for one request: the Authorization header value. `headers` are the headers to sign (lower-case++ * names; host included), `path` the already-encoded absolute path, `query` the canonical query string ('' for none).++ */++function signV4({ method, path: p, query = '', headers, payloadHash, region, service, accessKeyId, secretAccessKey, amzDate }) {++ const names = Object.keys(headers).map(h => h.toLowerCase()).sort();++ const lower = Object.fromEntries(Object.entries(headers).map(([k, v]) => [k.toLowerCase(), String(v).trim().replace(/\s+/g, ' ')]));++ const signedHeaders = names.join(';');++ const canonical = [method, p, query, names.map(n => `${n}:${lower[n]}\n`).join(''), signedHeaders, payloadHash].join('\n');++ const day = amzDate.slice(0, 8);++ const scope = `${day}/${region}/${service}/aws4_request`;++ const toSign = ['AWS4-HMAC-SHA256', amzDate, scope, sha256(canonical)].join('\n');++ const key = hmac(hmac(hmac(hmac(`AWS4${secretAccessKey}`, day), region), service), 'aws4_request');++ const signature = crypto.createHmac('sha256', key).update(toSign).digest('hex');++ return `AWS4-HMAC-SHA256 Credential=${accessKeyId}/${scope}, SignedHeaders=${signedHeaders}, Signature=${signature}`;++}++++function s3Storage({ endpoint, bucket, region = 'us-east-1', accessKeyId, secretAccessKey, prefix = 'events/', fetchImpl = fetch, now = () => new Date() }) {++ if (!endpoint || !bucket || !accessKeyId || !secretAccessKey) {++ throw new Error('archive storage: s3 needs EVENTS_ARCHIVE_S3_ENDPOINT, _BUCKET, _ACCESS_KEY_ID and _SECRET_ACCESS_KEY');++ }++ const base = new URL(endpoint);++ if (base.protocol !== 'https:' && !['127.0.0.1', 'localhost', '[::1]'].includes(base.hostname)) {++ throw new Error('archive storage: EVENTS_ARCHIVE_S3_ENDPOINT must be https (http only on loopback)');++ }++ const basePath = base.pathname.replace(/\/+$/, '');++ async function request(method, key, body) {++ const objectPath = `${basePath}/${encodeSegment(bucket)}/${checkKey(`${prefix}${key}`).split('/').map(encodeSegment).join('/')}`;++ const amzDate = now().toISOString().replace(/[-:]/g, '').replace(/\.\d{3}/, '');++ const payloadHash = sha256(body || '');++ const headers = { host: base.host, 'x-amz-content-sha256': payloadHash, 'x-amz-date': amzDate };++ const authorization = signV4({ method, path: objectPath, headers, payloadHash, region, service: 's3', accessKeyId, secretAccessKey, amzDate });++ const res = await fetchImpl(`${base.origin}${objectPath}`, {++ method, body, redirect: 'manual',++ headers: { 'x-amz-content-sha256': payloadHash, 'x-amz-date': amzDate, authorization },++ });++ return res;++ }++ return {++ name: `s3:${base.host}/${bucket}/${prefix}`,++ async put(key, buf) {++ const res = await request('PUT', key, buf);++ if (!res.ok) throw new Error(`archive storage: PUT ${key} answered ${res.status}`);++ },++ async get(key) {++ const res = await request('GET', key);++ if (res.status === 404) return null;++ if (!res.ok) throw new Error(`archive storage: GET ${key} answered ${res.status}`);++ return Buffer.from(await res.arrayBuffer());++ },++ };++}++++/** The adapter the environment names: s3 when EVENTS_ARCHIVE_S3_BUCKET is set, else the local directory. */++function fromEnv(env = process.env, { fetchImpl } = {}) {++ if (env.EVENTS_ARCHIVE_S3_BUCKET) {++ return s3Storage({++ endpoint: env.EVENTS_ARCHIVE_S3_ENDPOINT, bucket: env.EVENTS_ARCHIVE_S3_BUCKET, region: env.EVENTS_ARCHIVE_S3_REGION || undefined,++ accessKeyId: env.EVENTS_ARCHIVE_S3_ACCESS_KEY_ID, secretAccessKey: env.EVENTS_ARCHIVE_S3_SECRET_ACCESS_KEY,++ prefix: env.EVENTS_ARCHIVE_S3_PREFIX ?? undefined, ...(fetchImpl ? { fetchImpl } : {}),++ });++ }++ return localStorage({ dir: env.EVENTS_ARCHIVE_DIR || path.join(__dirname, '..', 'data', 'archive') });++}++++module.exports = { localStorage, s3Storage, fromEnv, signV4 };nnew file mode 100644@@@ -0,0 +1,189 @@++#!/usr/bin/env node++'use strict';++/**++ * The archive tier (ADR-042 decision 8): monthly NDJSON objects outside the database, written and read back only by++ * this operator job. Nothing runs it automatically and no online path reads an object.++ *++ * node scripts/events-archive.js export --month YYYY-MM [--older-than-days 90]++ * node scripts/events-archive.js restore --month YYYY-MM [--hold-days 30]++ * node scripts/events-archive.js status++ *++ * export the replay-tier rows (events_archive) received in that closed month and older than --older-than-days go++ * into one object, <YYYY>/events-<YYYY-MM>.ndjson.gz (one event row per line: the hot row's exact column++ * values — payload and actor as their stored JSON text — plus its epoch and archived_at), and a manifest++ * beside it, <YYYY>/events-<YYYY-MM>.json ({ format, month, count, sha256, bytes, exported_at }). The rows++ * are deleted from events_archive only after the object was read back and matched the manifest's count and++ * sha256, and only those still as exported (a row redacted meanwhile stays; rerun the export). A rerun, or an++ * export of what is left of a month, merges into the month's object: rows already in it are kept, a row++ * still in the database replaces its line.++ * restore reads the month's object back (its sha256 and count checked against the manifest first) into++ * events_archive, idempotent on event_id, held for --hold-days from now: the replay prune leaves a restored++ * row alone until then, whatever its age. Pulls see restored rows of the current epoch.++ * status rows per month in the replay tier, and the hot store's oldest event.++ *++ * Storage: scripts/archive-storage.js — EVENTS_ARCHIVE_DIR (local directory, default data/archive), or an++ * S3-compatible bucket when EVENTS_ARCHIVE_S3_BUCKET is set. The database is DATABASE_URL (PostgreSQL, ADR-035).++ * Export and restore of one month are serialised by a PostgreSQL advisory lock.++ */++const crypto = require('crypto');++const zlib = require('zlib');++const archive = require('../server/archive');++++const DAY_MS = 24 * 60 * 60 * 1000;++const FORMAT = 'openvibe.events.archive@1';++const LINE = [...archive.COLUMNS, ...archive.BODY, 'epoch', 'archived_at'];++const CHUNK = 1000;++++const sha256 = (buf) => crypto.createHash('sha256').update(buf).digest('hex');++++/** [start, end) of a UTC month, in ms. Throws on anything but YYYY-MM. */++function monthRange(month) {++ const m = /^(\d{4})-(0[1-9]|1[0-2])$/.exec(String(month || ''));++ if (!m) throw new Error('--month must be YYYY-MM');++ return [Date.UTC(Number(m[1]), Number(m[2]) - 1, 1), Date.UTC(Number(m[1]), Number(m[2]), 1)];++}++const objectKey = (month) => `${month.slice(0, 4)}/events-${month}.ndjson.gz`;++const manifestKey = (month) => `${month.slice(0, 4)}/events-${month}.json`;++++/** An events_archive row → its NDJSON line object, keys in a fixed order. */++function toLine(a) {++ const row = archive.unpack(a);++ const out = {};++ for (const k of LINE) out[k] = k === 'archived_at' ? a.archived_at : row[k];++ return out;++}++++/** The month's object as line objects, after checking it against its manifest; null when there is none. */++async function readObject(storage, month) {++ const [manifestBuf, gz] = [await storage.get(manifestKey(month)), await storage.get(objectKey(month))];++ if (!manifestBuf && !gz) return null;++ if (!manifestBuf || !gz) throw new Error(`archive ${month}: object and manifest must both exist (found only the ${gz ? 'object' : 'manifest'})`);++ const manifest = JSON.parse(manifestBuf.toString('utf8'));++ if (manifest.format !== FORMAT || manifest.month !== month) throw new Error(`archive ${month}: the manifest is not a ${FORMAT} manifest of this month`);++ if (sha256(gz) !== manifest.sha256) throw new Error(`archive ${month}: sha256 of the object does not match its manifest`);++ const text = zlib.gunzipSync(gz).toString('utf8');++ const lines = text ? text.split('\n').filter(Boolean).map((l) => JSON.parse(l)) : [];++ if (lines.length !== manifest.count) throw new Error(`archive ${month}: the object holds ${lines.length} rows, its manifest says ${manifest.count}`);++ return { manifest, lines };++}++++/** Serialise export and restore of one month across processes (inside the caller's transaction). */++const lockMonth = (db, month) => db.prepare("SELECT pg_advisory_xact_lock(hashtext('events-archive:' || ?)) AS ok").get(month);++++/**++ * Export `month` (see the header). Returns { month, exported, count, deleted, changed, object } — exported: rows read++ * from the database this run; count: rows in the object; deleted: rows removed from events_archive; changed: rows++ * left in place because they changed (were redacted) after they were read.++ */++async function exportMonth({ db, storage, month, now = Date.now(), olderThanDays = 90 }) {++ const [start, end] = monthRange(month);++ if (end > now) throw new Error(`archive ${month}: the month has not ended yet`);++ if (!(Number.isInteger(olderThanDays) && olderThanDays >= 0)) throw new Error('--older-than-days must be an integer >= 0');++ const cutoff = Math.min(end, now - olderThanDays * DAY_MS);++ return await db.tx(async () => {++ await lockMonth(db, month);++ const fresh = [];++ if (cutoff > start) {++ const page = db.prepare(`SELECT * FROM events_archive WHERE received_at >= ? AND received_at < ? AND (epoch, seq) > (?, ?)++ ORDER BY epoch, seq LIMIT ?`);++ let at = [-1, -1];++ for (;;) {++ const rows = await page.all(start, cutoff, at[0], at[1], CHUNK);++ for (const r of rows) fresh.push(toLine(r));++ if (rows.length < CHUNK) break;++ at = [rows[rows.length - 1].epoch, rows[rows.length - 1].seq];++ }++ }++ const existing = await readObject(storage, month);++ if (!fresh.length && !existing) return { month, exported: 0, count: 0, deleted: 0, changed: 0, object: null };++ const merged = new Map((existing ? existing.lines : []).map((l) => [l.id, l]));++ for (const l of fresh) merged.set(l.id, l);++ const lines = [...merged.values()].sort((x, y) => x.epoch - y.epoch || x.seq - y.seq);++ const gz = zlib.gzipSync(Buffer.from(lines.map((l) => `${JSON.stringify(l)}\n`).join(''), 'utf8'));++ const manifest = { format: FORMAT, month, count: lines.length, sha256: sha256(gz), bytes: gz.length, exported_at: new Date(now).toISOString() };++ await storage.put(objectKey(month), gz);++ await storage.put(manifestKey(month), Buffer.from(`${JSON.stringify(manifest, null, 2)}\n`));++++ // Verified before anything is deleted: the object as stored has the manifest's sha256 and count, and every++ // row read from the database this run is in it, line for line.++ const back = await readObject(storage, month);++ if (!back || back.manifest.sha256 !== manifest.sha256) throw new Error(`archive ${month}: the stored object is not the one written; nothing deleted`);++ const stored = new Map(back.lines.map((l) => [l.id, JSON.stringify(l)]));++ for (const l of fresh) {++ if (stored.get(l.id) !== JSON.stringify(l)) throw new Error(`archive ${month}: row ${l.id} is missing from the stored object; nothing deleted`);++ }++++ // Delete only what is still as exported: a row redacted since it was read keeps its (tombstoned) replay copy.++ let deleted = 0;++ let changed = 0;++ const lock = db.prepare('SELECT id, redacted_at FROM events_archive WHERE id = ANY(?) FOR UPDATE');++ const drop = db.prepare('DELETE FROM events_archive WHERE id = ANY(?)');++ for (let i = 0; i < fresh.length; i += CHUNK) {++ const part = fresh.slice(i, i + CHUNK);++ const exported = new Map(part.map((l) => [l.id, l.redacted_at ?? null]));++ const current = await lock.all(part.map((l) => l.id));++ const same = current.filter((r) => (r.redacted_at ?? null) === exported.get(r.id)).map((r) => r.id);++ changed += current.length - same.length;++ if (same.length) deleted += (await drop.run(same)).changes;++ }++ return { month, exported: fresh.length, count: lines.length, deleted, changed, object: objectKey(month) };++ });++}++++/** Restore `month` into events_archive (see the header). Returns { month, count, inserted, skipped }. */++async function restoreMonth({ db, storage, month, now = Date.now(), holdDays = 30 }) {++ monthRange(month);++ if (!(Number.isInteger(holdDays) && holdDays >= 1)) throw new Error('--hold-days must be an integer >= 1');++ return await db.tx(async () => {++ await lockMonth(db, month);++ const obj = await readObject(storage, month);++ if (!obj) throw new Error(`archive ${month}: no object in ${storage.name}`);++ const put = db.prepare(`INSERT INTO events_archive (${archive.COLUMNS.join(', ')}, epoch, body, archived_at, hold_until)++ VALUES (${archive.COLUMNS.map(() => '?').join(', ')}, ?, ?, ?, ?) ON CONFLICT (id) DO NOTHING`);++ let inserted = 0;++ for (const l of obj.lines) {++ inserted += (await put.run(...archive.COLUMNS.map((c) => l[c]), l.epoch, archive.pack(l), l.archived_at, now + holdDays * DAY_MS)).changes;++ }++ return { month, count: obj.lines.length, inserted, skipped: obj.lines.length - inserted };++ });++}++++/** Rows per UTC month in the replay tier, and the hot store's oldest event. */++async function status({ db }) {++ const months = await db.prepare(`SELECT to_char(to_timestamp(received_at / 1000.0) AT TIME ZONE 'UTC', 'YYYY-MM') AS month,++ COUNT(*)::bigint AS rows, MIN(seq) AS first_seq, MAX(seq) AS last_seq, COUNT(hold_until)::bigint AS held++ FROM events_archive GROUP BY 1 ORDER BY 1`).all();++ const hot = await db.prepare('SELECT MIN(seq) AS oldest_seq, MIN(received_at) AS oldest_received_at, COUNT(*)::bigint AS rows FROM events').get();++ return { replay: months, hot };++}++++async function main(argv = process.argv.slice(2)) {++ const [command, ...rest] = argv;++ const opt = (k, d) => { const i = rest.indexOf(k); return i >= 0 ? rest[i + 1] : d; };++ if (!['export', 'restore', 'status'].includes(command)) {++ console.error('usage: events-archive.js export --month YYYY-MM [--older-than-days 90] | restore --month YYYY-MM [--hold-days 30] | status');++ return 2;++ }++ require('dotenv').config();++ const { load } = require('../server/config');++ const { openDb } = require('../server/store');++ const { fromEnv } = require('./archive-storage');++ const db = await openDb(load(), { log: { log() {}, warn: console.warn } });++ try {++ const storage = fromEnv(process.env);++ let r;++ if (command === 'export') r = await exportMonth({ db, storage, month: opt('--month'), olderThanDays: Number(opt('--older-than-days', '90')) });++ else if (command === 'restore') r = await restoreMonth({ db, storage, month: opt('--month'), holdDays: Number(opt('--hold-days', '30')) });++ else r = await status({ db });++ console.log(JSON.stringify({ storage: command === 'status' ? undefined : storage.name, ...r }, null, 2));++ return 0;++ } finally {++ await db.close();++ }++}++++if (require.main === module) {++ main().then((code) => process.exit(code), (err) => { console.error(`events-archive: ${err.message}`); process.exit(1); });++}++++module.exports = { exportMonth, restoreMonth, status, monthRange, objectKey, manifestKey, FORMAT };@@@ -6,9 +6,10 @@* -> { events: [{ seq, cursor, event }], next_after_seq, next_cursor, latest_seq, gap? }* `after=` takes an opaque cursor beside after_seq (ADR-042 decision 7); a cursor from another* retention epoch answers `gap` — never a silent restart. `gap` ({ from_seq, to_seq }) means-- * events after the position were already pruned by retention (or the epoch changed).++ * events after the position were already pruned by retention (or the epoch changed). The pull++ * spans the hot store and the replay tier (decision 8) with the one cursor: `gap` only outside both.* Keep next_cursor as the cursor; it moves past events that did not match.-- * GET /api/v1/events/:event_id -> { seq, cursor, event }++ * GET /api/v1/events/:event_id -> { seq, cursor, event } (hot or replay)* GET /api/v1/checkpoints?topic=… / PUT /api/v1/checkpoints { topic, cursor }* a consumer's own stored cursor per topic pattern (consumer = calling principal). `cursor` is* the numeric position; PUT also accepts an opaque cursor string, and both answers carry the@@@ -111,7 +112,8 @@ function readRouter({ store, auth, worker, limits }) {const row = await store.getEvent(String(req.params.id));const visible = row && (req.principal.kind === 'app' ? apps.visibleToApp(row, req.principal) : (row.env || 'production') === 'production');if (!visible) return http.sendProblem(res, 404, 'events.not_found', { detail: 'no such event (or pruned by retention)', ctx: req.ov });-- return res.json({ seq: row.seq, cursor: cursor.encode(row.seq, await store.epoch()), event: rowToEnvelope(row) });++ // A replay-tier row names the epoch its position belongs to; a hot row is in the current one.++ return res.json({ seq: row.seq, cursor: cursor.encode(row.seq, row.epoch ?? await store.epoch()), event: rowToEnvelope(row) });});const consumerOf = (req) => req.principal.service || req.principal.sub;nnew file mode 100644@@@ -0,0 +1,33 @@++'use strict';++/**++ * The replay tier's row shape (ADR-042 decision 8; migrations/0006_events_archive.sql). An archived row keeps as columns++ * what the read path filters and orders on (COLUMNS); every other column of the hot row (BODY) is one gzip'd JSON++ * object in `body`. unpack() gives back the hot row's exact column values — actor and payload are the stored JSON++ * text, not re-serialised — so rowToEnvelope, the visibility and redaction rules and the export see one shape for++ * both tiers. Compression is node:zlib gzip: no dependency, and the export (scripts/events-archive.js) uses it too.++ */++const zlib = require('zlib');++++/** Columns events_archive keeps as they are in `events`. */++const COLUMNS = ['id', 'seq', 'event_type', 'source', 'subject_type', 'subject_id', 'visibility', 'project_id', 'env', 'received_at', 'redacted_at', 'redacts'];++/** Columns of `events` that live in the compressed body. */++const BODY = ['version', 'actor', 'subject_revision', 'trace_id', 'priority', 'occurred_at', 'payload', 'hops', 'publisher', 'request_id', 'size_bytes', 'redacted_by'];++++/** The compressed body of a hot row (or of an unpacked one). */++function pack(row) {++ const body = {};++ for (const c of BODY) body[c] = row[c] === undefined ? null : row[c];++ return zlib.gzipSync(Buffer.from(JSON.stringify(body), 'utf8'));++}++++/** An events_archive row → the hot row it was, plus the epoch its position belongs to. */++function unpack(a) {++ const body = JSON.parse(zlib.gunzipSync(a.body).toString('utf8'));++ const row = {};++ for (const c of COLUMNS) row[c] = a[c];++ for (const c of BODY) row[c] = body[c] === undefined ? null : body[c];++ row.epoch = a.epoch;++ return row;++}++++module.exports = { pack, unpack, COLUMNS, BODY };@@@ -111,6 +111,9 @@ function load(env = process.env) {stream: env.NATS_STREAM || 'OV_EVENTS_STREAM',},retentionDays: int(env.EVENTS_RETENTION_DAYS, 30),++ // retention.replay (ADR-042 decision 8): days an event stays readable in events_archive after it leaves the++ // hot store, counted from receipt like retentionDays. 0: no replay tier (pruned events are deleted, as before).++ replayRetentionDays: Math.max(0, int(env.EVENTS_REPLAY_RETENTION_DAYS, 365)),receiptRetentionDays: int(env.EVENTS_RECEIPT_RETENTION_DAYS, 90),pruneIntervalMs: int(env.EVENTS_PRUNE_INTERVAL_MS, 60 * 60 * 1000),@@@ -67,8 +67,13 @@ async function start({const prune = async () => {try {-- const r = await store.prune({ retentionDays: config.retentionDays, receiptRetentionDays: config.receiptRetentionDays, sandboxRetentionDays: config.apps.sandboxRetentionDays });-- if (r.events || r.receipts) log.log(`[retention] pruned ${r.events} events, ${r.receipts} receipts`);++ // Tiers (ADR-042 decision 8): hot events past EVENTS_RETENTION_DAYS move to the replay tier, which keeps them++ // for EVENTS_REPLAY_RETENTION_DAYS (0: no replay tier, deleted as before).++ const r = await store.prune({++ retentionDays: config.retentionDays, replayRetentionDays: config.replayRetentionDays,++ receiptRetentionDays: config.receiptRetentionDays, sandboxRetentionDays: config.apps.sandboxRetentionDays,++ });++ if (r.events || r.replay || r.receipts) log.log(`[retention] pruned ${r.events} hot events (${r.archived} moved to replay), ${r.replay} replay events, ${r.receipts} receipts`);} catch (err) {log.error(`[retention] prune failed: ${err.message}`);}@@@ -24,6 +24,10 @@ function limitsOf(config) {production: off(q.production.retainedBytes), sandbox: off(q.sandbox.retainedBytes), exceeded: '429 events.quota_exceeded' },{ id: 'retention_days', label: 'Days an event is kept', capability: 'events.app.read', unit: 'days',production: config.retentionDays, sandbox: config.apps.sandboxRetentionDays, exceeded: 'pruned' },++ // The replay window (ADR-042 decision 8): an event past retention_days is still pulled and replayed from the++ // replay tier until replay_retention_days after receipt (0 = no replay tier). Sandbox events never enter it.++ { id: 'replay_retention_days', label: 'Days an event can still be replayed', capability: 'events.app.read', unit: 'days',++ production: config.replayRetentionDays, sandbox: 0, exceeded: 'gap' },{ id: 'subscriptions', label: 'Webhook subscriptions', capability: 'events.app.subscribe', unit: 'count',production: off(q.production.maxSubscriptions), sandbox: off(q.sandbox.maxSubscriptions), exceeded: '429 events.quota_exceeded' },{ id: 'payload_bytes', label: 'Payload size of one event', capability: 'events.app.publish', unit: 'bytes',@@@ -19,8 +19,9 @@* Developer-app events (app.<project_key>.*, events.app.publish) are never streamed here, to anyone.** Resume: `Last-Event-ID` (or ?last_event_id=) is a cursor, or a bare seq for one release. Missed-- * events are replayed first; when the cursor is older than retention — or belongs to another-- * retention epoch — an `event: gap` message comes first, so the client knows to refetch state.++ * events are replayed first, from the replay tier and then the hot store (ADR-042 decision 8); when the++ * cursor is older than both — or belongs to another retention epoch — an `event: gap` message comes++ * first, so the client knows to refetch state.* Heartbeat comment every 25 s.** Public replay window: a browser (signed out or signed in) is replayed `public` events received in@@@ -5,6 +5,9 @@* consumer delivery". Global order is `seq`, handed out from a counter that never goes backwards,* even after retention has pruned the newest rows. Beside it a cursor (./cursor.js) names a seq plus* the retention epoch — ADR-042 decision 7; seq stays on the wire for one release.++ *++ * Retention is tiered (ADR-042 decision 8): prune() moves a hot event older than retention.hot into events_archive++ * (./archive.js) in the transaction that deletes it, and pulls, scans and by-id reads span both tiers with one cursor.*/const fs = require('fs');const path = require('path');@@@ -13,6 +16,7 @@ const topics = require('./topics');const apps = require('./apps');const redaction = require('./redaction');const cursor = require('./cursor');++const archive = require('./archive');const { carrierClass, orderingKey } = require('./fabric/policy');const { createUsage, deliveryCode } = require('./usage');@@@ -25,6 +29,26 @@ const DEV_PGLITE = path.join(__dirname, '..', 'data', 'pglite');const PUBLISH_RECEIPT = 'events:publish'; // receipts' consumer column for accepted publishes++/**++ * Rows with seq > ? (and `filter`) from the hot store and the replay tier's current epoch, in seq order, LIMIT ?.++ * Both branches return the hot row's columns plus `body` (NULL for a hot row; a replay row's compressed rest).++ */++function spanTiers(filter) {++ return `SELECT * FROM (++ SELECT ${[...archive.COLUMNS, ...archive.BODY].join(', ')}, NULL::bytea AS body FROM events WHERE seq > ? ${filter}++ UNION ALL++ SELECT ${archive.COLUMNS.join(', ')}, ${archive.BODY.map(c => `NULL AS ${c}`).join(', ')}, body FROM events_archive++ WHERE seq > ? ${filter} AND epoch = (SELECT value FROM store_epoch WHERE name = 'events')++ ) t ORDER BY seq LIMIT ?`;++}++++/** A spanTiers row → the hot row shape. */++function fromTier(r) {++ if (r.body) return archive.unpack(r);++ delete r.body;++ return r;++}++/*** The serving handle (ADR-035): DATABASE_URL through PgBouncer; in development without it, an embedded PGlite database* in data/pglite. Migrations run first, as the owner (DATABASE_DIRECT_URL), or on the embedded handle. The columns added@@@ -101,7 +125,12 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usageaddReceipt: db.prepare('INSERT INTO idempotency_receipts (consumer, event_id, processed_at) VALUES (?, ?, ?) ON CONFLICT DO NOTHING'),nextSeq: db.prepare("UPDATE sequences SET value = value + 1 WHERE name = 'events' RETURNING value"),lastSeq: db.prepare("SELECT value FROM sequences WHERE name = 'events'"),-- minSeq: db.prepare('SELECT MIN(seq) AS s FROM events'),++ // The oldest position either tier still holds; replay rows count in the current epoch only.++ minSeq: db.prepare(`SELECT LEAST((SELECT MIN(seq) FROM events),++ (SELECT MIN(seq) FROM events_archive WHERE epoch = (SELECT value FROM store_epoch WHERE name = 'events'))) AS s`),++ getArchived: db.prepare('SELECT * FROM events_archive WHERE id = ?'),++ archivedBySubject: db.prepare(`SELECT * FROM events_archive WHERE source = ? AND subject_type = ? AND subject_id = ?++ AND redacted_at IS NULL AND redacts = 0`),epoch: db.prepare("SELECT value FROM store_epoch WHERE name = 'events'"),bumpEpoch: db.prepare("UPDATE store_epoch SET value = value + 1 WHERE name = 'events' RETURNING value"),// Hop depth counts only rows the publisher's own tenancy can have caused: a first-party@@@ -122,7 +151,8 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usageAND redacted_at IS NULL AND redacts = 0`),applyTombstone: db.prepare(`UPDATE events SET payload = @payload, actor = @actor, size_bytes = @size_bytes,redacted_at = @redacted_at, redacted_by = @redacted_by WHERE id = @id AND redacted_at IS NULL`),-- firstSeqSince: db.prepare('SELECT MIN(seq) AS s FROM events WHERE received_at >= ?'),++ firstSeqSince: db.prepare(`SELECT LEAST((SELECT MIN(seq) FROM events WHERE received_at >= ?),++ (SELECT MIN(seq) FROM events_archive WHERE received_at >= ? AND epoch = (SELECT value FROM store_epoch WHERE name = 'events'))) AS s`),enabledSubs: db.prepare('SELECT id, topic_pattern, project_id, env FROM subscriptions WHERE enabled = 1'),projectRecent: db.prepare('SELECT COUNT(*) AS n FROM events WHERE project_id = ? AND env = ? AND received_at > ?'),projectBytes: db.prepare('SELECT COALESCE(SUM(size_bytes), 0)::bigint AS b FROM events WHERE project_id = ? AND env = ?'),@@@ -137,8 +167,9 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage// The carrier a (subscription, ordering key) currently has a backlog on: what pins new deliveries of the key.backlogCarrier: db.prepare(`SELECT carrier FROM deliveries WHERE subscription_id = ? AND ordering_key = ? AND status IN ('pending', 'failed')AND carrier IS NOT NULL ORDER BY seq LIMIT 1`),-- afterSeq: db.prepare('SELECT * FROM events WHERE seq > ? ORDER BY seq LIMIT ?'),-- afterSeqLike: db.prepare("SELECT * FROM events WHERE seq > ? AND event_type ILIKE ? ESCAPE '\\' ORDER BY seq LIMIT ?"),++ // One statement over both tiers (one snapshot): an event the prune moves between two reads can be in neither.++ afterSeq: db.prepare(spanTiers('')),++ afterSeqLike: db.prepare(spanTiers("AND event_type ILIKE ? ESCAPE '\\'")),};// ── Events ───────────────────────────────────────────────@@@ -167,12 +198,13 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usagebytes = (await q.projectBytes.get(project.projectId, project.env)).b;}for (const env of items) {-- const existing = await q.getEvent.get(env.event_id);++ // An id is stored once across the tiers: one moved to replay is still a duplicate (with its position).++ const existing = await q.getEvent.get(env.event_id) || await q.getArchived.get(env.event_id);if (existing) {if (existing.source !== env.source || existing.event_type !== env.event_type) {throw new StoreError(409, 'events.id_conflict', `event ${env.event_id} already exists with different content`);}-- results.push({ event_id: env.event_id, seq: existing.seq, duplicate: true, cursor: cursor.encode(existing.seq, epoch) });++ results.push({ event_id: env.event_id, seq: existing.seq, duplicate: true, cursor: cursor.encode(existing.seq, existing.epoch ?? epoch) });continue;}if (await q.getReceipt.get(PUBLISH_RECEIPT, env.event_id)) {@@@ -286,13 +318,15 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage/*** Turn the targets of `directive` (server/redaction.js) into tombstones on behalf of `by` (the-- * stored row of the redacting event). Throws StoreError 403 when an event id names an event of-- * another owner. Returns [{ id, seq, columns }] for the rows it rewrote.++ * stored row of the redacting event), in either tier. Throws StoreError 403 when an event id names++ * an event of another owner. Returns [{ id, seq, columns }] for the rows it rewrote.*/async function applyRedaction(by, directive, now) {-- const targets = new Map();++ const targets = new Map(); // id -> the row (hot, or unpacked from replay)for (const id of directive.eventIds) {-- const r = await q.getEvent.get(id);++ const hot = await q.getEvent.get(id);++ const archived = hot ? null : await q.getArchived.get(id);++ const r = hot || (archived && archive.unpack(archived));if (!r) continue; // never stored, or pruned alreadyif (!redaction.sameOwner(by, r)) {throw new StoreError(403, 'events.redaction_not_allowed', `event ${id} belongs to ${r.source}; ${by.source} may redact only its own events`,@@@ -305,17 +339,33 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usagefor (const r of await q.redactBySubject.all(by.source, directive.subjectType, subjectId)) {if (redaction.sameOwner(by, r)) targets.set(r.id, r);}++ for (const a of await q.archivedBySubject.all(by.source, directive.subjectType, subjectId)) {++ const r = archive.unpack(a);++ if (redaction.sameOwner(by, r)) targets.set(r.id, r);++ }}}const out = [];for (const r of targets.values()) {if (r.id === by.id) continue;const columns = redaction.tombstone(r, { by: by.id, at: now });-- if ((await q.applyTombstone.run({ id: r.id, ...columns })).changes) out.push({ id: r.id, seq: r.seq, columns });++ // A hot row the prune moved since it was read is tombstoned where it is now: the move holds the row lock++ // until it commits, so the hot UPDATE finds nothing and the replay row is visible to the next statement.++ if ((await q.applyTombstone.run({ id: r.id, ...columns })).changes || await tombstoneArchived(r.id, columns)) {++ out.push({ id: r.id, seq: r.seq, columns });++ }}return out;}++ /** Rewrite a replay row as a tombstone (its body re-packed). False when it is not there or already redacted. */++ async function tombstoneArchived(id, columns) {++ const a = await db.prepare('SELECT * FROM events_archive WHERE id = ? AND redacted_at IS NULL FOR UPDATE').get(id);++ if (!a) return false;++ const row = { ...archive.unpack(a), ...columns };++ return (await db.prepare('UPDATE events_archive SET body = ?, redacted_at = ? WHERE id = ?').run(archive.pack(row), columns.redacted_at, id)).changes === 1;++ }++/*** Operator redaction (scripts/redact-backfill.js): what an event of first-party `source` carrying* `directive` (the payload.redacts object) would have redacted. The tombstones name `by` as@@@ -354,8 +404,12 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usagereturn r ? r.revoked_at : null;}++ /** The stored row of an event, hot or replay (an unpacked replay row also carries its `epoch`); null when neither tier has it. */async function getEvent(id) {-- return await q.getEvent.get(id) || null;++ const hot = await q.getEvent.get(id);++ if (hot) return hot;++ const archived = await q.getArchived.get(id);++ return archived ? archive.unpack(archived) : null;}/** The resolved delivery policy (../fabric/policy.js) for an event type, or null when no library is attached. */@@@ -369,7 +423,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage/** First seq received at or after `ms`; the next seq to be handed out when there is none. */async function firstSeqSince(ms) {-- const m = (await q.firstSeqSince.get(ms)).s;++ const m = (await q.firstSeqSince.get(ms, ms)).s;return m == null ? await lastSeq() + 1 : m;}@@@ -393,7 +447,8 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage/*** Events with seq > afterSeq that match any of `patterns`, in order, filtered by `accept(row)`.* Scans at most `scanMax` rows; returns { rows, cursor } where cursor is the last seq examined-- * (so a pull consumer's cursor moves past rows that did not match).++ * (so a pull consumer's cursor moves past rows that did not match). Spans the replay tier and the++ * hot store as one sequence (ADR-042 decision 8): replay rows come back unpacked, in the hot shape.*/async function scan(afterSeq, { patterns = ['*'], limit = 100, scanMax = 5000, accept = () => true } = {}) {const rows = [];@@@ -401,7 +456,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usagelet scanned = 0;const single = patterns.length === 1 && patterns[0] !== '*' ? topics.toLike(patterns[0]) : null;for (;;) {-- const page = single ? await q.afterSeqLike.all(cursor, single, 500) : await q.afterSeq.all(cursor, 500);++ const page = (single ? await q.afterSeqLike.all(cursor, single, cursor, single, 500) : await q.afterSeq.all(cursor, cursor, 500)).map(fromTier);for (const row of page) {scanned++;cursor = row.seq;@@@ -677,13 +732,55 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage// ── Retention ────────────────────────────────────────────/**-- * Drop events (and their deliveries) older than retentionDays (sandbox events older than-- * sandboxRetentionDays, when shorter), and old publish receipts.++ * Move up to `limit` hot events received before `cutoff` into the replay tier, in one transaction: the rows are++ * locked FOR UPDATE SKIP LOCKED, copied into events_archive (idempotent on the event id) and deleted (their++ * deliveries go with them). A row another prune — or a redaction, or a claim cascading on it — holds is skipped,++ * never waited on: two Events processes pruning at once take disjoint rows, and a row one of them moved is gone++ * from `events` when the other's next batch looks. Returns how many hot rows it removed.*/-- async function prune({ retentionDays = 30, receiptRetentionDays = 90, sandboxRetentionDays = retentionDays, now = clock.now() } = {}) {-- let events = (await db.prepare('DELETE FROM events WHERE received_at < ?').run(now - retentionDays * DAY_MS)).changes;-- if (sandboxRetentionDays < retentionDays) {-- events += (await db.prepare("DELETE FROM events WHERE env = 'sandbox' AND received_at < ?").run(now - sandboxRetentionDays * DAY_MS)).changes;++ const moveBatch = async (cutoff, limit, now) => await db.tx(async () => {++ const rows = await db.prepare(`SELECT * FROM events WHERE received_at < ? AND env != 'sandbox' ORDER BY seq LIMIT ?++ FOR UPDATE SKIP LOCKED`).all(cutoff, limit);++ if (!rows.length) return 0;++ const at = (await q.epoch.get()).value;++ const put = db.prepare(`INSERT INTO events_archive (${archive.COLUMNS.join(', ')}, epoch, body, archived_at)++ VALUES (${archive.COLUMNS.map(() => '?').join(', ')}, ?, ?, ?) ON CONFLICT (id) DO NOTHING`);++ for (const r of rows) await put.run(...archive.COLUMNS.map(c => r[c]), at, archive.pack(r), now);++ return (await db.prepare('DELETE FROM events WHERE id = ANY(?)').run(rows.map(r => r.id))).changes;++ });++++ /**++ * Retention in tiers (ADR-042 decision 8). With replayRetentionDays > 0, events older than retentionDays move++ * from the hot store into the replay tier (moveBatch, `batchSize` at a time, at most `maxBatches` per call; the++ * next call continues), and replay rows older than replayRetentionDays are dropped. With 0 (the default here;++ * the service passes EVENTS_REPLAY_RETENTION_DAYS) there is no replay tier: old events are deleted as before and++ * the replay tier is emptied, so nothing is kept without a rule that prunes it. Rows an operator restored stay++ * until their hold_until either way. Sandbox events never enter replay: they are deleted older than++ * sandboxRetentionDays (or retentionDays, if shorter). Old publish receipts go as before.++ */++ async function prune({ retentionDays = 30, replayRetentionDays = 0, receiptRetentionDays = 90, sandboxRetentionDays = retentionDays,++ now = clock.now(), batchSize = 500, maxBatches = 200 } = {}) {++ const hotCutoff = now - retentionDays * DAY_MS;++ let events = 0;++ let archived = 0;++ let replay = 0;++ if (replayRetentionDays > 0) {++ events += (await db.prepare("DELETE FROM events WHERE env = 'sandbox' AND received_at < ?")++ .run(now - Math.min(retentionDays, sandboxRetentionDays) * DAY_MS)).changes;++ for (let i = 0; i < maxBatches; i++) {++ const n = await moveBatch(hotCutoff, batchSize, now);++ archived += n;++ if (n < batchSize) break; // done, or the rest is locked by a concurrent prune that is moving it++ }++ events += archived;++ replay = (await db.prepare('DELETE FROM events_archive WHERE received_at < ? AND (hold_until IS NULL OR hold_until < ?)')++ .run(now - replayRetentionDays * DAY_MS, now)).changes;++ } else {++ events += (await db.prepare('DELETE FROM events WHERE received_at < ?').run(hotCutoff)).changes;++ if (sandboxRetentionDays < retentionDays) {++ events += (await db.prepare("DELETE FROM events WHERE env = 'sandbox' AND received_at < ?").run(now - sandboxRetentionDays * DAY_MS)).changes;++ }++ replay = (await db.prepare('DELETE FROM events_archive WHERE hold_until IS NULL OR hold_until < ?').run(now)).changes;}const receipts = (await db.prepare('DELETE FROM idempotency_receipts WHERE processed_at < ?').run(now - Math.max(receiptRetentionDays, retentionDays) * DAY_MS)).changes;@@@ -693,7 +790,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usageSELECT 1 FROM delivery_placements q WHERE q.carrier_class = p.carrier_classAND q.ordering_key IS NOT DISTINCT FROM p.ordering_key AND q.decided_at > p.decided_at)`).run(now - retentionDays * DAY_MS)).changes;-- return { events, receipts, placements };++ return { events, archived, replay, receipts, placements };}async function ping() {nnew file mode 100644@@@ -0,0 +1,277 @@++'use strict';++// Tiered retention (ADR-042 decision 8): hot (`events`) → replay (`events_archive`, same database) → archive (monthly++// NDJSON objects, scripts/events-archive.js). Reads span hot and replay with one cursor; `gap` only outside both.++const assert = require('assert');++const http = require('http');++const zlib = require('zlib');++const { ids } = require('openvibe-contracts');++const apps = require('../server/apps');++const cursor = require('../server/cursor');++const { load } = require('../server/config');++const { limitsOf } = require('../server/limits');++const { createStore } = require('../server/store');++const { exportMonth, restoreMonth, objectKey, manifestKey } = require('../scripts/events-archive');++const { localStorage, s3Storage, fromEnv, signV4 } = require('../scripts/archive-storage');++const { boot, request, serviceToken, appToken, envelope, sse, suite, tmpDir } = require('./helpers');++++const t = suite('archive');++const DAY = 86400000;++let h;++const live = serviceToken('live', ['events.event.publish']);++const media = serviceToken('media', ['events.event.publish']);++const reader = serviceToken('games', ['events.event.read']);++const T0 = Date.now();++const later = (days) => T0 + days * DAY;++const tiered = (days, over = {}) => ({ retentionDays: 30, replayRetentionDays: 365, now: later(days), ...over });++const open = [];++++async function publish(token, body) {++ const r = await request(h.base, 'POST', '/api/v1/events', { token, body });++ assert.strictEqual(r.status, 201, r.text);++ return r.body;++}++const pull = async (q, token = reader) => await request(h.base, 'GET', `/api/v1/events?${q}`, { token });++const count = async (table, id) => (await h.db.prepare(`SELECT COUNT(*)::int AS n FROM ${table} WHERE id = ?`).get(id)).n;++++t('boot', async () => { h = await boot(); });++++t('a hot-pruned event is readable through replay: by id and by pull, unchanged', async () => {++ const e = await publish(live, envelope('live', { event_type: 'live.tier.one', visibility: 'public', payload: { n: 1, s: 'é ✓' } }));++ const before = await request(h.base, 'GET', `/api/v1/events/${e.event_id}`, { token: reader });++ const r = await h.store.prune(tiered(31));++ assert.ok(r.archived >= 1 && r.events >= r.archived, JSON.stringify(r));++ assert.deepStrictEqual([await count('events', e.event_id), await count('events_archive', e.event_id)], [0, 1], 'moved, not copied');++ const after = await request(h.base, 'GET', `/api/v1/events/${e.event_id}`, { token: reader });++ assert.strictEqual(after.status, 200, after.text);++ assert.deepStrictEqual(after.body, before.body, 'the same seq, cursor and envelope from the replay tier');++ const p = await pull('topic=live.tier.one&after_seq=0');++ assert.deepStrictEqual(p.body.events.map(x => x.event), [before.body.event]);++ assert.strictEqual(p.body.gap, undefined);++ // A publish of the same id is a duplicate with its position, never a second row.++ const again = await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope('live', { event_id: e.event_id, event_type: 'live.tier.one' }) });++ assert.strictEqual(again.status, 200, again.text);++ assert.deepStrictEqual([again.body.duplicate, again.body.seq, again.body.cursor], [true, e.seq, e.cursor]);++});++++t('a cursor across the hot/replay boundary reports no false gap (pull and SSE)', async () => {++ const a = await publish(live, envelope('live', { event_type: 'live.tier.cross' }));++ const b = await publish(live, envelope('live', { event_type: 'live.tier.cross' }));++ await h.store.prune(tiered(31));++ assert.strictEqual(await count('events_archive', b.event_id), 1);++ const c = await publish(live, envelope('live', { event_type: 'live.tier.cross' }));++++ let r = await pull(`topic=live.tier.cross&after=${a.cursor}&limit=1`);++ assert.deepStrictEqual([r.body.events.map(x => x.seq), r.body.gap], [[b.seq], undefined], 'replay');++ assert.strictEqual(r.body.events[0].cursor, b.cursor);++ r = await pull(`topic=live.tier.cross&after=${r.body.next_cursor}&limit=1`);++ assert.deepStrictEqual([r.body.events.map(x => x.seq), r.body.gap], [[c.seq], undefined], 'then hot, no gap between');++ r = await pull(`topic=*&after_seq=${a.seq - 1}`);++ assert.deepStrictEqual(r.body.events.map(x => x.seq), [a.seq, b.seq, c.seq]);++ assert.strictEqual(r.body.gap, undefined);++++ const s = await sse(h.base, '/realtime/stream?topics=live.tier.cross', { headers: { Authorization: `Bearer ${serviceToken('live', ['events.event.read'])}`, 'Last-Event-ID': a.cursor } });++ open.push(s);++ await s.waitFor(x => x.events().length === 2);++ assert.deepStrictEqual(s.events().map(x => x.seq), [b.seq, c.seq]);++ assert.deepStrictEqual(s.gaps(), []);++ assert.ok(s.messages.some(m => m.id === b.cursor), 'the SSE id of a replayed event is its cursor');++ s.close();++});++++t('redaction reaches replay rows (by id and by subject); the owner rule holds there too', async () => {++ const x = await publish(live, envelope('live', { event_type: 'live.tier.secret', subject: { type: 'stream', id: 'tier-red-1' }, payload: { secret: 'a' } }));++ const y = await publish(live, envelope('live', { event_type: 'live.tier.secret', subject: { type: 'stream', id: 'tier-red-2' }, payload: { secret: 'b' } }));++ await h.store.prune(tiered(31));++ assert.strictEqual(await count('events_archive', x.event_id) + await count('events_archive', y.event_id), 2);++ const refused = await request(h.base, 'POST', '/api/v1/events', { token: media, body: envelope('media', { event_type: 'media.tier.redact', payload: { redacts: { event_ids: [x.event_id] } } }) });++ assert.deepStrictEqual([refused.status, refused.body.code], [403, 'events.redaction_not_allowed']);++ await publish(live, envelope('live', { event_type: 'live.tier.redact', payload: { redacts: { event_ids: [y.event_id], subject_type: 'stream', subject_ids: ['tier-red-1'] } } }));++ for (const e of [x, y]) {++ const r = await request(h.base, 'GET', `/api/v1/events/${e.event_id}`, { token: reader });++ assert.strictEqual(r.body.event.payload.redacted, true, JSON.stringify(r.body));++ assert.deepStrictEqual(r.body.event.actor, { type: 'service', id: 'live' });++ assert.strictEqual(r.body.seq, e.seq);++ }++ const p = await pull('topic=live.tier.secret&after_seq=0');++ assert.ok(p.body.events.every(e => e.event.payload.redacted === true && !('secret' in e.event.payload)));++});++++t('sandbox and environment rules apply to replay rows exactly as to hot ones', async () => {++ const prj = `prj_${ids.ulid()}`;++ const [prodApp, sandApp] = [ids.newId('app'), ids.newId('app')];++ const cap = ['events.app.publish', 'events.app.read'];++ const prod = appToken({ appId: prodApp, projectId: prj, env: 'production', cap });++ const sand = appToken({ appId: sandApp, projectId: prj, env: 'sandbox', cap });++ const appEvent = (appId) => envelope('x', { source: apps.appSource(appId), event_type: `app.${apps.projectKey(prj)}.order.created`, actor: { type: 'app', id: appId }, subject: { type: 'order', id: '1' }, visibility: 'internal' });++ const pe = await publish(prod, appEvent(prodApp));++ const se = await publish(sand, appEvent(sandApp));++ await h.store.prune(tiered(31, { sandboxRetentionDays: 7 }));++ assert.deepStrictEqual([await count('events_archive', pe.event_id), await count('events_archive', se.event_id), await count('events', se.event_id)], [1, 0, 0],++ 'a production event moves; a sandbox event is deleted, never kept in replay');++ const topic = `topic=app.${apps.projectKey(prj)}.*&after_seq=0`;++ assert.deepStrictEqual((await pull(topic, prod)).body.events.map(e => e.seq), [pe.seq], 'the app reads its own replay rows');++ assert.deepStrictEqual((await pull(topic, sand)).body.events, [], 'never across environments');++ assert.deepStrictEqual((await pull(topic)).body.events.map(e => e.seq), [pe.seq], 'first-party: production app events through app.*');++ assert.deepStrictEqual((await pull('topic=*&after_seq=0')).body.events.filter(e => e.event.event_id === pe.event_id), [], '…and only through app.*');++ assert.strictEqual((await request(h.base, 'GET', `/api/v1/events/${pe.event_id}`, { token: sand })).status, 404);++ assert.strictEqual((await request(h.base, 'GET', `/api/v1/events/${pe.event_id}`, { token: prod })).status, 200);++});++++t('two concurrent prunes move each event exactly once', async () => {++ await h.store.prune(tiered(31)); // the other tests' hot rows first: the race below is over these 60 only++ const mine = [];++ for (let i = 0; i < 60; i++) mine.push((await publish(live, envelope('live', { event_type: 'live.tier.race', payload: { i } }))).event_id);++ // Two stores on one database: separate pooled connections, as two Events processes would have.++ const s1 = createStore(h.db, { clock: h.clock });++ const s2 = createStore(h.db, { clock: h.clock });++ const [r1, r2] = await Promise.all([s1.prune(tiered(31, { batchSize: 7 })), s2.prune(tiered(31, { batchSize: 7 }))]);++ assert.strictEqual(r1.archived + r2.archived, 60, JSON.stringify([r1, r2]));++ const rows = await h.db.prepare('SELECT id, seq FROM events_archive WHERE id = ANY(?) ORDER BY seq').all(mine);++ assert.deepStrictEqual(rows.map(r => r.id), mine, 'every one in replay, once, in order');++ assert.strictEqual((await h.db.prepare('SELECT COUNT(*)::int AS n FROM events WHERE id = ANY(?)').get(mine)).n, 0);++ const again = await h.store.prune(tiered(31));++ assert.strictEqual(again.archived, 0, 'nothing left to move');++});++++t('export → restore round-trips byte-identical rows; the rows leave replay only after the object verifies', async () => {++ const storage = localStorage({ dir: tmpDir() });++ const month = new Date(h.clock.now()).toISOString().slice(0, 7);++ const exportAt = later(400); // the month is closed and older than the cutoff++ const inMonth = await h.db.prepare('SELECT * FROM events_archive ORDER BY seq').all();++ assert.ok(inMonth.length >= 60);++ const before = await Promise.all(inMonth.map(r => h.store.getEvent(r.id)));++++ const r = await exportMonth({ db: h.db, storage, month, now: exportAt, olderThanDays: 90 });++ assert.deepStrictEqual([r.exported, r.count, r.deleted, r.changed], [inMonth.length, inMonth.length, inMonth.length, 0]);++ assert.strictEqual((await h.db.prepare('SELECT COUNT(*)::int AS n FROM events_archive').get()).n, 0);++ const lines = zlib.gunzipSync(await storage.get(objectKey(month))).toString('utf8').trim().split('\n').map(l => JSON.parse(l));++ assert.strictEqual(lines.length, inMonth.length);++ assert.strictEqual(JSON.parse(await storage.get(manifestKey(month))).count, inMonth.length);++ assert.strictEqual((await request(h.base, 'GET', `/api/v1/events/${inMonth[0].id}`, { token: reader })).status, 404, 'the archive tier is offline');++ const gapped = await pull(`topic=*&after_seq=0`);++ assert.ok(gapped.body.gap, 'a cursor into what went to the archive tier is a gap');++++ // A rerun is idempotent: the object keeps its rows, nothing is deleted twice.++ const rerun = await exportMonth({ db: h.db, storage, month, now: exportAt, olderThanDays: 90 });++ assert.deepStrictEqual([rerun.exported, rerun.count, rerun.deleted], [0, inMonth.length, 0]);++++ const back = await restoreMonth({ db: h.db, storage, month, now: exportAt, holdDays: 30 });++ assert.deepStrictEqual([back.count, back.inserted, back.skipped], [inMonth.length, inMonth.length, 0]);++ assert.deepStrictEqual((await restoreMonth({ db: h.db, storage, month, now: exportAt })).inserted, 0, 'idempotent on event_id');++ const after = await Promise.all(inMonth.map(r => h.store.getEvent(r.id)));++ assert.deepStrictEqual(after, before, 'every column, payload and actor text byte for byte');++ const restored = await h.db.prepare('SELECT * FROM events_archive ORDER BY seq').all();++ for (let i = 0; i < restored.length; i++) {++ assert.strictEqual(zlib.gunzipSync(restored[i].body).toString('utf8'), zlib.gunzipSync(inMonth[i].body).toString('utf8'));++ assert.deepStrictEqual({ ...restored[i], body: null, hold_until: null }, { ...inMonth[i], body: null, hold_until: null });++ }++ // Held: a replay prune leaves restored rows alone until hold_until, whatever their age.++ const held = async () => (await h.db.prepare('SELECT COUNT(*)::int AS n FROM events_archive WHERE id = ANY(?)').get(inMonth.map(x => x.id))).n;++ await h.store.prune({ ...tiered(0), now: exportAt });++ assert.strictEqual(await held(), inMonth.length);++ await h.store.prune({ ...tiered(0), now: exportAt + 31 * DAY });++ assert.strictEqual(await held(), 0);++++ // A tampered object is refused before anything is restored.++ const gz = await storage.get(objectKey(month));++ gz[gz.length - 5] ^= 0xff;++ await storage.put(objectKey(month), gz);++ await assert.rejects(restoreMonth({ db: h.db, storage, month, now: exportAt }), /sha256/);++ await assert.rejects(exportMonth({ db: h.db, storage, month: '2999-01', now: exportAt }), /not ended/);++});++++t('replay pruning answers `gap` (the existing shape), pull and SSE', async () => {++ const a = await publish(live, envelope('live', { event_type: 'live.tier.gone', visibility: 'public' }));++ await publish(live, envelope('live', { event_type: 'live.tier.gone', visibility: 'public' }));++ await h.store.prune(tiered(31));++ const r0 = await pull(`topic=live.tier.gone&after=${a.cursor}`);++ assert.strictEqual(r0.body.gap, undefined, 'still in replay: no gap');++ await h.store.prune(tiered(366));++ const kept = await publish(live, envelope('live', { event_type: 'live.tier.gone' }));++ const r = await pull(`topic=live.tier.gone&after=${a.cursor}`);++ assert.deepStrictEqual(r.body.gap, { from_seq: a.seq + 1, to_seq: kept.seq - 1 });++ assert.deepStrictEqual(r.body.events.map(e => e.seq), [kept.seq]);++ const s = await sse(h.base, '/realtime/stream?topics=live.tier.gone', { headers: { Authorization: `Bearer ${serviceToken('live', ['events.event.read'])}`, 'Last-Event-ID': a.cursor } });++ open.push(s);++ await s.waitFor(x => x.events().length === 1);++ assert.deepStrictEqual(s.gaps().map(g => [g.reason, g.from_seq, g.to_seq]), [['retention', a.seq + 1, kept.seq - 1]]);++ s.close();++});++++t('EVENTS_REPLAY_RETENTION_DAYS=0 behaves exactly as before the tiers', async () => {++ assert.strictEqual(load({}).replayRetentionDays, 365);++ assert.strictEqual(load({ EVENTS_REPLAY_RETENTION_DAYS: '0' }).replayRetentionDays, 0);++ const x = await boot({ env: { EVENTS_REPLAY_RETENTION_DAYS: '0' } });++ try {++ const pub = async () => (await request(x.base, 'POST', '/api/v1/events', { token: live, body: envelope('live', { event_type: 'live.tier.zero' }) })).body;++ const old = await pub();++ // A leftover replay row (the tier was on before) is emptied by the next prune: no rule would prune it otherwise.++ await x.store.prune({ ...tiered(31), replayRetentionDays: 365 });++ assert.strictEqual((await x.db.prepare('SELECT COUNT(*)::int AS n FROM events_archive').get()).n, 1);++ const r = await x.store.prune({ retentionDays: x.config.retentionDays, replayRetentionDays: x.config.replayRetentionDays, now: later(31) });++ assert.deepStrictEqual([r.archived, r.replay], [0, 1]);++ const old2 = await pub();++ const r2 = await x.store.prune({ retentionDays: 30, replayRetentionDays: 0, now: later(31) });++ assert.deepStrictEqual([r2.events, r2.archived], [1, 0], 'deleted, not moved');++ assert.strictEqual((await x.db.prepare('SELECT COUNT(*)::int AS n FROM events_archive').get()).n, 0);++ const kept = await pub();++ const p = await request(x.base, 'GET', `/api/v1/events?topic=live.tier.zero&after=${old.cursor}`, { token: reader });++ assert.deepStrictEqual(p.body.gap, { from_seq: old.seq + 1, to_seq: kept.seq - 1 });++ assert.strictEqual((await request(x.base, 'GET', `/api/v1/events/${old2.event_id}`, { token: reader })).status, 404);++ const lim = limitsOf(x.config).limits.find(l => l.id === 'replay_retention_days');++ assert.deepStrictEqual([lim.production, lim.sandbox], [0, 0]);++ } finally {++ await x.stop();++ }++ const lim = limitsOf(h.config).limits.find(l => l.id === 'replay_retention_days');++ assert.deepStrictEqual([lim.production, lim.sandbox, lim.unit], [365, 0, 'days']);++});++++t('archive storage: SigV4 matches AWS\'s published vectors; the s3 adapter round-trips through a signing stub', async () => {++ const empty = 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855';++ assert.match(signV4({ method: 'GET', path: '/', headers: { host: 'example.amazonaws.com', 'x-amz-date': '20150830T123600Z' }, payloadHash: empty,++ region: 'us-east-1', service: 'service', accessKeyId: 'AKIDEXAMPLE', secretAccessKey: 'wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY', amzDate: '20150830T123600Z' }),++ /Signature=5fa00fa31553b73ebf1942676e86291e8372ff2a2260956d9b8aae1d763fbf31$/);++ assert.match(signV4({ method: 'GET', path: '/test.txt', headers: { host: 'examplebucket.s3.amazonaws.com', range: 'bytes=0-9', 'x-amz-content-sha256': empty, 'x-amz-date': '20130524T000000Z' },++ payloadHash: empty, region: 'us-east-1', service: 's3', accessKeyId: 'AKIAIOSFODNN7EXAMPLE', secretAccessKey: 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY', amzDate: '20130524T000000Z' }),++ /SignedHeaders=host;range;x-amz-content-sha256;x-amz-date, Signature=f0e8bdb87c964420e857bd35b5d6ed310bd44f0170aba48dd91039c6036bdb41$/);++++ // A stub S3: checks each request's signature with the same secret and keeps objects in memory.++ const objects = new Map();++ const creds = { accessKeyId: 'AKIDTEST', secretAccessKey: 'secret-test' };++ const server = http.createServer((req, res) => {++ const chunks = [];++ req.on('data', c => chunks.push(c));++ req.on('end', () => {++ const body = Buffer.concat(chunks);++ const headers = { host: req.headers.host, 'x-amz-content-sha256': req.headers['x-amz-content-sha256'], 'x-amz-date': req.headers['x-amz-date'] };++ const want = signV4({ method: req.method, path: req.url, headers, payloadHash: require('crypto').createHash('sha256').update(body).digest('hex'),++ region: 'eu-west-3', service: 's3', ...creds, amzDate: req.headers['x-amz-date'] });++ if (req.headers.authorization !== want) { res.writeHead(403); return res.end(); }++ if (req.method === 'PUT') { objects.set(req.url, body); res.writeHead(200); return res.end(); }++ if (!objects.has(req.url)) { res.writeHead(404); return res.end(); }++ res.writeHead(200); return res.end(objects.get(req.url));++ });++ });++ await new Promise(r => server.listen(0, '127.0.0.1', r));++ try {++ const env = { EVENTS_ARCHIVE_S3_BUCKET: 'ov-events', EVENTS_ARCHIVE_S3_ENDPOINT: `http://127.0.0.1:${server.address().port}`, EVENTS_ARCHIVE_S3_REGION: 'eu-west-3',++ EVENTS_ARCHIVE_S3_ACCESS_KEY_ID: creds.accessKeyId, EVENTS_ARCHIVE_S3_SECRET_ACCESS_KEY: creds.secretAccessKey };++ const s3 = fromEnv(env);++ assert.match(s3.name, /^s3:/);++ await s3.put('2026/events-2026-01.json', Buffer.from('{"a":1}'));++ assert.deepStrictEqual([...objects.keys()], ['/ov-events/events/2026/events-2026-01.json']);++ assert.strictEqual((await s3.get('2026/events-2026-01.json')).toString(), '{"a":1}');++ assert.strictEqual(await s3.get('2026/missing.json'), null);++ await assert.rejects(fromEnv({ ...env, EVENTS_ARCHIVE_S3_SECRET_ACCESS_KEY: 'wrong' }).put('x.json', Buffer.from('1')), /answered 403/);++ assert.throws(() => s3Storage({ ...creds, endpoint: 'http://s3.example.com', bucket: 'b' }), /https/);++ assert.match(fromEnv({ EVENTS_ARCHIVE_DIR: tmpDir() }).name, /^local:/);++ await assert.rejects(s3.put('../escape', Buffer.from('1')), /bad object key/);++ } finally {++ server.close();++ }++});++++t('stop', async () => { for (const c of open) c.close(); await h.stop(); });++++t.run();
@@ -40,8 +40,21 @@ VALKEY_PREFIX=
@@ -40,8 +40,21 @@ VALKEY_PREFIX=
#NATS_STREAM=OV_EVENTS_STREAM
#NATS_STREAM=OV_EVENTS_STREAM
#EVENTS_CARRIERS=pg-v1,valkey-v1,nats-v1,nats-js-v1
#EVENTS_CARRIERS=pg-v1,valkey-v1,nats-v1,nats-js-v1
#EVENTS_CARRIERS_DISABLED=
#EVENTS_CARRIERS_DISABLED=
-# Events (and their deliveries) older than this are pruned; resuming from before it yields a gap.
+# Retention in tiers (ADR-042 decision 8; README "Retention: three tiers"). Hot events (and their deliveries) older
+# than this move to the replay tier (events_archive) at the hourly prune.
EVENTS_RETENTION_DAYS=30
EVENTS_RETENTION_DAYS=30
+# Days (from receipt) an event stays readable from the replay tier: pulls, SSE resume and by-id reads; resuming from
+# before it yields a gap. 0 = no replay tier: old events are deleted, as before, and events_archive is emptied.
+EVENTS_REPLAY_RETENTION_DAYS=365
+# The archive tier, operator only (node scripts/events-archive.js export|restore --month YYYY-MM): a local directory,
+# or an S3-compatible bucket when EVENTS_ARCHIVE_S3_BUCKET is set (path-style, https; SigV4, no dependency).
+#EVENTS_ARCHIVE_DIR=/var/lib/openvibe/events-archive
+#EVENTS_ARCHIVE_S3_ENDPOINT=https://s3.example.com
+#EVENTS_ARCHIVE_S3_BUCKET=
+#EVENTS_ARCHIVE_S3_REGION=us-east-1
+#EVENTS_ARCHIVE_S3_ACCESS_KEY_ID=
+#EVENTS_ARCHIVE_S3_SECRET_ACCESS_KEY=
+#EVENTS_ARCHIVE_S3_PREFIX=events/
# Accepted event ids are remembered this long, so late re-publishes stay duplicates.
# Accepted event ids are remembered this long, so late re-publishes stay duplicates.
EVENTS_RECEIPT_RETENTION_DAYS=90
EVENTS_RECEIPT_RETENTION_DAYS=90
#EVENTS_PRUNE_INTERVAL_MS=3600000
#EVENTS_PRUNE_INTERVAL_MS=3600000
@@ -112,7 +112,31 @@ Each delivery is `POST <endpoint>` with body `{ "event": <envelope>, "seq": n }`
@@ -112,7 +112,31 @@ Each delivery is `POST <endpoint>` with body `{ "event": <envelope>, "seq": n }`
Operators: `GET /api/v1/deliveries?status=dead` is the dead-letter queue; `POST /api/v1/deliveries/replay { subscription_id, event_ids: [...] }` or `{ subscription_id, from_seq }` requeues retained events (that is also how a new subscription catches up on history).
Operators: `GET /api/v1/deliveries?status=dead` is the dead-letter queue; `POST /api/v1/deliveries/replay { subscription_id, event_ids: [...] }` or `{ subscription_id, from_seq }` requeues retained events (that is also how a new subscription catches up on history).
-Pull consumers: start with `GET /api/v1/events?topic=media.vod.*&limit=100`, then request `GET /api/v1/events?topic=media.vod.*&after=<next_cursor>&limit=100`, using the previous response's opaque `next_cursor` ([ADR-042](https://github.com/OpenVibers/OpenVibe.Contracts/blob/main/docs/adr/ADR-042-events-fabric.md) decision 7). The response carries `{ events: [{ seq, cursor, event }], next_cursor, next_after_seq, latest_seq }`, plus `gap: { from_seq, to_seq }` when the position is older than retention or from another retention epoch. Use `next_cursor` even when no events match, since it advances past scanned events. Numeric `after_seq=<position>` remains accepted for one release; when both are supplied, `after` takes precedence. The numeric `seq`, `next_after_seq` and `latest_seq` fields also remain for one release. `PUT /api/v1/checkpoints { topic, cursor }` stores a consumer's position here if it has nowhere better (an opaque cursor string is accepted in place of the integer; the answer carries its `epoch`, `carrier` and `next_cursor`).
+Pull consumers: start with `GET /api/v1/events?topic=media.vod.*&limit=100`, then request `GET /api/v1/events?topic=media.vod.*&after=<next_cursor>&limit=100`, using the previous response's opaque `next_cursor` ([ADR-042](https://github.com/OpenVibers/OpenVibe.Contracts/blob/main/docs/adr/ADR-042-events-fabric.md) decision 7). The response carries `{ events: [{ seq, cursor, event }], next_cursor, next_after_seq, latest_seq }`, plus `gap: { from_seq, to_seq }` when the position is older than both the hot store and the replay tier (below) or from another retention epoch. Use `next_cursor` even when no events match, since it advances past scanned events. Numeric `after_seq=<position>` remains accepted for one release; when both are supplied, `after` takes precedence. The numeric `seq`, `next_after_seq` and `latest_seq` fields also remain for one release. `PUT /api/v1/checkpoints { topic, cursor }` stores a consumer's position here if it has nowhere better (an opaque cursor string is accepted in place of the integer; the answer carries its `epoch`, `carrier` and `next_cursor`).
+
+### Retention: three tiers
+
+[ADR-042](https://github.com/OpenVibers/OpenVibe.Contracts/blob/main/docs/adr/ADR-042-events-fabric.md) decision 8. An event lives in one tier at a time:
+
+| Tier | Where | Kept | Read by |
+|---|---|---|---|
+| hot | `events` (with its deliveries) | `EVENTS_RETENTION_DAYS` (30) | everything: delivery, pulls, SSE, by id, replay to subscriptions |
+| replay | `events_archive`, same database: the filter columns plus a gzip'd body, no delivery rows | `EVENTS_REPLAY_RETENTION_DAYS` (365) from receipt; `0` = no replay tier | pulls, SSE resume and `GET /api/v1/events/:id`, with the same cursor |
+| archive | one NDJSON object per month (gzip, with a manifest), outside the database | until an operator deletes it | nothing online; an operator restores a month into replay |
+
+The hourly prune (`EVENTS_PRUNE_INTERVAL_MS`) moves each hot event past `EVENTS_RETENTION_DAYS` into `events_archive` in the transaction that deletes it: 500 rows at a time, locked `FOR UPDATE SKIP LOCKED`, inserted idempotently on `event_id`, so two Events processes pruning at once each move different rows and no event is moved twice. Its deliveries go with the hot row (operator replay to a subscription, `POST /api/v1/deliveries/replay`, reaches hot events only). Replay rows past `EVENTS_REPLAY_RETENTION_DAYS` are then deleted. A pull or SSE resume runs from a cursor in the replay tier into the hot store without a `gap`; `gap` (the same shape) still means the position is older than both tiers or from another epoch. Redaction tombstones replay rows exactly as hot ones, the sandbox and environment rules read them the same way, and a re-publish of an archived `event_id` is still `duplicate: true` with its original `seq` and `cursor`. Sandbox events never enter replay: they are deleted at `EVENTS_APP_SANDBOX_RETENTION_DAYS`, as before; publish receipts keep `EVENTS_RECEIPT_RETENTION_DAYS`.
+
+`EVENTS_REPLAY_RETENTION_DAYS=0` keeps the old behaviour exactly: events past `EVENTS_RETENTION_DAYS` are deleted, and the next prune also empties `events_archive`, because no rule would prune the rows left there otherwise. `GET /limits.json` publishes the window as `replay_retention_days`.
+
+The archive tier is an operator job, never run automatically (`DATABASE_URL` as for the service):
+
+```sh
+node scripts/events-archive.js status # replay rows per month, oldest hot event
+node scripts/events-archive.js export --month 2026-01 [--older-than-days 90]
+node scripts/events-archive.js restore --month 2026-01 [--hold-days 30]
+```
+
+`export` writes the closed month's replay rows older than the cutoff to `<YYYY>/events-<YYYY-MM>.ndjson.gz` (one row per line: every column of the hot row, payload and actor as their stored JSON text, plus `epoch` and `archived_at`) and `<YYYY>/events-<YYYY-MM>.json` (`count`, `sha256`). It deletes those rows from `events_archive` only after reading the object back and matching both, and only rows unchanged since it read them (a row redacted meanwhile stays; rerun). Re-running, or exporting the rest of a month later, merges into the month's object. `restore` checks the object against its manifest, then inserts it back into `events_archive`, idempotent on `event_id`, held for `--hold-days` so the replay prune does not delete it at once. A redaction published after a month was exported does not reach its object, so restore only what you need. Export and restore of one month are serialised by a PostgreSQL advisory lock. Storage (`scripts/archive-storage.js`): a local directory, `EVENTS_ARCHIVE_DIR` (default `data/archive`), or an S3-compatible bucket when `EVENTS_ARCHIVE_S3_BUCKET` is set (`EVENTS_ARCHIVE_S3_ENDPOINT`, `_REGION`, `_ACCESS_KEY_ID`, `_SECRET_ACCESS_KEY`, `_PREFIX`; path-style, SigV4 signed with node:crypto and sent with node's `fetch`, no dependency). Each object is built in memory.
## The fabric: carriers, the planner and explain
## The fabric: carriers, the planner and explain
@@ -252,14 +276,14 @@ void connect();
@@ -252,14 +276,14 @@ void connect();
- Conversely, a user JWT that carries `typ` or `purpose` (a ticket, a FedCM assertion) is not a session here.
- Conversely, a user JWT that carries `typ` or `purpose` (a ticket, a FedCM assertion) is not a session here.
- A person's topic: `network.notification.created` (Network's outbox; visibility `subject`, subject the recipient). `topics=network.notification.*` streams a person's own notifications and nobody else's. There is no `user:<id>` topic: it is not a valid pattern (400 `realtime.bad_topic`), and subject visibility already does its job.
- A person's topic: `network.notification.created` (Network's outbox; visibility `subject`, subject the recipient). `topics=network.notification.*` streams a person's own notifications and nobody else's. There is no `user:<id>` topic: it is not a valid pattern (400 `realtime.bad_topic`), and subject visibility already does its job.
- Visibility: `public` events go to anyone subscribed to the topic; `subject` events only to the user whose subject id (`usr_…`) is the event's `actor.id` or its user `subject.id`; `internal` events never reach a browser. A guessed topic yields nothing.
- Visibility: `public` events go to anyone subscribed to the topic; `subject` events only to the user whose subject id (`usr_…`) is the event's `actor.id` or its user `subject.id`; `internal` events never reach a browser. A guessed topic yields nothing.
-- Resume: the SSE `id` is the cursor (ADR-042 decision 7; a bare seq is accepted for one release), so the browser's automatic `Last-Event-ID` (or `?last_event_id=`) replays what was missed. A cursor older than retention — or from another retention epoch (`reason: "epoch"`) — first gets `event: gap` (`{ reason, from_seq, to_seq }`), as does a replay that hits `REALTIME_REPLAY_MAX`.
+- Resume: the SSE `id` is the cursor (ADR-042 decision 7; a bare seq is accepted for one release), so the browser's automatic `Last-Event-ID` (or `?last_event_id=`) replays what was missed. A cursor older than both the hot store and the replay tier — or from another retention epoch (`reason: "epoch"`) — first gets `event: gap` (`{ reason, from_seq, to_seq }`), as does a replay that hits `REALTIME_REPLAY_MAX`.
- Public replay window: browsers (signed out or signed in) are replayed `public` events received in the last `REALTIME_PUBLIC_REPLAY_SECONDS` (300; 0 = none), enough to ride out a reconnect. Older public events are not replayed: the stream opens with `event: gap` (`reason: "public_window"`, `window_seconds`), and the client refetches state from the owning service. `subject` events addressed to the viewer, and service viewers, keep the whole retention. This keeps the stream from paging through a month of public history, chat lines included; nothing in the network needs more (no browser surface replays public events today).
- Public replay window: browsers (signed out or signed in) are replayed `public` events received in the last `REALTIME_PUBLIC_REPLAY_SECONDS` (300; 0 = none), enough to ride out a reconnect. Older public events are not replayed: the stream opens with `event: gap` (`reason: "public_window"`, `window_seconds`), and the client refetches state from the owning service. `subject` events addressed to the viewer, and service viewers, keep the whole retention. This keeps the stream from paging through a month of public history, chat lines included; nothing in the network needs more (no browser surface replays public events today).
- Redacted events are replayed as their tombstones ([Redaction](#redaction)).
- Redacted events are replayed as their tombstones ([Redaction](#redaction)).
- Heartbeat comment every 25 s; at most 20 topics per connection and `REALTIME_MAX_CONNECTIONS` (2000) overall; CORS with credentials for `https://*.openvibe.*` only.
- Heartbeat comment every 25 s; at most 20 topics per connection and `REALTIME_MAX_CONNECTIONS` (2000) overall; CORS with credentials for `https://*.openvibe.*` only.
## Owns
## Owns
-- `events`, `subscriptions`, `deliveries`, `consumer_checkpoints`, `idempotency_receipts`, `app_revocations`, `app_usage` (PostgreSQL `ov_events`, ADR-035; the realtime fan-out is in-process, one Events process)
+- `events`, `events_archive` (the replay tier), `subscriptions`, `deliveries`, `consumer_checkpoints`, `idempotency_receipts`, `app_revocations`, `app_usage` (PostgreSQL `ov_events`, ADR-035; the realtime fan-out is in-process, one Events process)
- developer-app event scope, sandbox separation and per-project Events quotas (ADR-014)
- developer-app event scope, sandbox separation and per-project Events quotas (ADR-014)
- canonical event envelope (event_id, trace_id, type, version, source, actor, subject + revision, payload)
- canonical event envelope (event_id, trace_id, type, version, source, actor, subject + revision, payload)
- priority classes `critical|important|low`, loop guards, backpressure, DLQ and replay
- priority classes `critical|important|low`, loop guards, backpressure, DLQ and replay
@@ -280,6 +304,7 @@ void connect();
@@ -280,6 +304,7 @@ void connect();
- event persists before consumer delivery (`test/delivery.test.js`)
- event persists before consumer delivery (`test/delivery.test.js`)
- kill a consumer mid-processing; replay creates exactly one effect (`test/outbox-inbox.test.js`)
- kill a consumer mid-processing; replay creates exactly one effect (`test/outbox-inbox.test.js`)
- browser reconnect resumes from a cursor or reports a gap (`test/realtime.test.js`)
- browser reconnect resumes from a cursor or reports a gap (`test/realtime.test.js`)
+- a hot-pruned event stays readable through replay, a cursor across the tier boundary reports no false gap, two concurrent prunes move each event once, replay pruning answers `gap`, export → restore round-trips byte for byte, and `EVENTS_REPLAY_RETENTION_DAYS=0` behaves as before the tiers (`test/archive.test.js`)
- a guessed private topic yields no data (`test/realtime.test.js`)
- a guessed private topic yields no data (`test/realtime.test.js`)
- a developer app cannot publish, read or subscribe outside its project, sandbox never meets production, app webhooks reach public addresses only, quotas hold (`test/apps.test.js`)
- a developer app cannot publish, read or subscribe outside its project, sandbox never meets production, app webhooks reach public addresses only, quotas hold (`test/apps.test.js`)
- a project's publishing and deliveries are counted per hour, refusals and failed attempts as errors, and each closed hour is sent once as `events.usage.recorded`, never visible to apps (`test/usage.test.js`)
- a project's publishing and deliveries are counted per hour, refusals and failed attempts as errors, and each closed hour is sent once as `events.usage.recorded`, never visible to apps (`test/usage.test.js`)
@@ -20,12 +20,13 @@
@@ -20,12 +20,13 @@
"capabilitiesRegistered": true,
"capabilitiesRegistered": true,
"deployed": true,
"deployed": true,
"deployment": "Deployed with `sudo ovhost deploy events`: /opt/openvibe.events, unit openvibe-events, 127.0.0.1:4300; public at https://events.openvibe.network (realtime SSE, health, token-guarded /api/v1/events, /api/v1/subscriptions, /api/v1/checkpoints; /api/v1/deliveries host-local); real traffic since 2026-09-23 (Live, Media, Chat, Games, Network and OpenRe publish; on 2026-09-23 8 subscriptions, held by Search, Tips, VIP, News and Reviews); restore drill passed 2026-09-23",
"deployment": "Deployed with `sudo ovhost deploy events`: /opt/openvibe.events, unit openvibe-events, 127.0.0.1:4300; public at https://events.openvibe.network (realtime SSE, health, token-guarded /api/v1/events, /api/v1/subscriptions, /api/v1/checkpoints; /api/v1/deliveries host-local); real traffic since 2026-09-23 (Live, Media, Chat, Games, Network and OpenRe publish; on 2026-09-23 8 subscriptions, held by Search, Tips, VIP, News and Reviews); restore drill passed 2026-09-23",
- "updated": "2026-09-29",
+ "updated": "2026-10-04",
"features": [
"features": [
"durable event store with the canonical envelope",
"durable event store with the canonical envelope",
"subscriptions with signed webhook delivery (v1 and v2), retry, dead letters and replay",
"subscriptions with signed webhook delivery (v1 and v2), retry, dead letters and replay",
"priority classes and backpressure",
"priority classes and backpressure",
"pull consumers with cursors and checkpoints",
"pull consumers with cursors and checkpoints",
+ "tiered retention (ADR-042 decision 8): hot events past EVENTS_RETENTION_DAYS move to the events_archive replay tier (EVENTS_REPLAY_RETENTION_DAYS, 365; 0 = off), read with the same cursor; monthly NDJSON archive export and restore by operator job (scripts/events-archive.js; local directory or S3-compatible bucket)",
"realtime SSE with cursor resume, gap reports and Network tickets",
"realtime SSE with cursor resume, gap reports and Network tickets",
"developer-app publish, read and subscribe with sandbox separation, SSRF-guarded webhooks and project quotas",
"developer-app publish, read and subscribe with sandbox separation, SSRF-guarded webhooks and project quotas",
"hourly project usage rollups (events.usage.recorded)",
"hourly project usage rollups (events.usage.recorded)",
new file mode 100644
new file mode 100644
@@ -0,0 +1,30 @@
@@ -0,0 +1,30 @@
+-- PostgreSQL seed for the 0006_events_archive cutover rehearsal (docs/cutover-12.md). Loaded after migrations
+-- 0001-0005, before 0006: the shape production has today. Times are relative to now() (ms), so the rehearsal's prune
+-- with the defaults (EVENTS_RETENTION_DAYS=30, EVENTS_REPLAY_RETENTION_DAYS=365) sees:
+-- seq 1 production, 400 days old -> moved to replay, then dropped by the replay prune (past 365 days)
+-- seq 2 production, 60 days old -> moved to replay (kept), its two deliveries go with the hot row
+-- seq 3 production, 45 days old, tombstoned (redacted) -> moved to replay, still a tombstone
+-- seq 4 sandbox, 45 days old -> deleted (sandbox never enters replay)
+-- seq 5 production, 1 day old -> stays hot
+-- The epoch stays 0 and the sequence counter at 5, as a live store's would.
+INSERT INTO events (id, seq, event_type, version, source, actor, subject_type, subject_id, subject_revision, trace_id, priority,
+ visibility, occurred_at, received_at, payload, hops, publisher, request_id, project_id, env, size_bytes, redacted_at, redacted_by, redacts)
+SELECT v.id, v.seq, v.event_type, 1, v.source, '{"type":"service","id":"svc:live"}', 'stream', v.subject_id, NULL,
+ 'trc_rehearsal_' || v.seq, 'important', 'public', '2026-01-01T00:00:00.000Z',
+ (extract(epoch FROM now()) * 1000)::bigint - v.days::bigint * 86400000, v.payload, 1, v.publisher, NULL, v.project_id, v.env,
+ length(v.payload), v.redacted_at, v.redacted_by, 0
+FROM (VALUES
+ ('evt_rehearsal_0001', 1, 'live.stream.started', 'live', 'str_1', '{"title":"oldest"}', 'svc:live', NULL, 'production', 400, NULL::bigint, NULL),
+ ('evt_rehearsal_0002', 2, 'live.stream.started', 'live', 'str_2', '{"title":"replay ✓"}', 'svc:live', NULL, 'production', 60, NULL, NULL),
+ ('evt_rehearsal_0003', 3, 'live.stream.started', 'live', 'str_3', '{"redacted":true,"redacted_at":1,"redacted_by":"evt_rehearsal_0005"}', 'svc:live', NULL, 'production', 45, 1, 'evt_rehearsal_0005'),
+ ('evt_rehearsal_0004', 4, 'app.prj_rehearsal.tick', 'app', 'str_4', '{"n":4}', 'app:prj_rehearsal', 'prj_rehearsal', 'sandbox', 45, NULL, NULL),
+ ('evt_rehearsal_0005', 5, 'live.stream.started', 'live', 'str_5', '{"title":"hot"}', 'svc:live', NULL, 'production', 1, NULL, NULL)
+) AS v(id, seq, event_type, source, subject_id, payload, publisher, project_id, env, days, redacted_at, redacted_by);
+
+UPDATE sequences SET value = 5 WHERE name = 'events';
+
+INSERT INTO subscriptions (id, consumer, topic_pattern, endpoint, secret, enabled, created_at, updated_at)
+VALUES ('sub_rehearsal', 'search', 'live.stream.*', 'http://127.0.0.1:9/hook', 'rehearsal-secret-rehearsal-secret', 1, 0, 0);
+INSERT INTO deliveries (event_id, subscription_id, seq, priority, status, delivered_at, created_at, updated_at)
+VALUES ('evt_rehearsal_0002', 'sub_rehearsal', 2, 1, 'delivered', 0, 0, 0),
+ ('evt_rehearsal_0005', 'sub_rehearsal', 5, 1, 'pending', NULL, 0, 0);
new file mode 100644
new file mode 100644
@@ -0,0 +1,56 @@
@@ -0,0 +1,56 @@
+# Cutover: the replay tier (`0006_events_archive`, ADR-042 decision 8)
+
+One additive migration (`-- phase: expand`): a new table `events_archive` and its four indexes. Nothing in `events`,
+`deliveries` or any other table changes; no backfill runs at migration time. The release that ships it starts moving
+hot events past `EVENTS_RETENTION_DAYS` (30) into `events_archive` at its hourly prune instead of deleting them, and
+reads span both tables. `EVENTS_REPLAY_RETENTION_DAYS` defaults to 365; `0` is the old behaviour exactly.
+
+## Before
+
+1. Health: `ov access run openvibe-ovh health events` answers ` ready `.
+2. Backup: the host's nightly `ov_events` dump is from today, or take one (`pg_dump -Fc ov_events`, OpenVibe.Host
+ docs/backups.md). The migration only adds a table, so the backup is for the release, not for the migration.
+3. Size the first prune: `SELECT COUNT(*) FROM events WHERE received_at < (extract(epoch FROM now()) * 1000)::bigint - 30 * 86400000;`
+ is 0 or close to it on a store that prunes hourly today; whatever it is, the prune moves 500 rows per transaction and
+ at most 100 000 per run, the next run continues.
+
+## Cutover
+
+1. Deploy the release (the pipeline's `ovhost deploy events`). On start the service applies `0006_events_archive.sql`
+ as the owner role (`DATABASE_DIRECT_URL`): `CREATE TABLE events_archive …` plus its indexes. On an empty table this
+ takes milliseconds and holds no lock on `events`.
+2. `/etc/openvibe/events.env`: nothing to add for the default (365 days of replay). To keep today's behaviour, set
+ `EVENTS_REPLAY_RETENTION_DAYS=0` and restart.
+3. Check: `curl -s https://events.openvibe.network/limits.json` lists `replay_retention_days`; after the first prune
+ (start + up to `EVENTS_PRUNE_INTERVAL_MS`) the journal (`ov access run openvibe-ovh journal openvibe-events.service 200`)
+ shows `[retention] pruned N hot events (N moved to replay), …` when anything was due, and
+ `node scripts/events-archive.js status` (on the host, with the service env) lists the replay rows per month.
+
+The archive tier (`scripts/events-archive.js export|restore --month YYYY-MM`) is an operator job, never part of this
+cutover; see README "Retention: three tiers".
+
+## Rollback
+
+- Release only: roll back to the previous release. It never reads `events_archive`; the table and the rows moved into
+ it stay, unread, and the old release deletes hot events past retention as before. Rolling forward again finds them
+ in place (cursors into them keep working, the epoch is unchanged).
+- The table too (only if the release is abandoned): after rolling back, `DROP TABLE events_archive;` as the owner and
+ delete its row from the migrations ledger, `DELETE FROM ov_migrations WHERE id = '0006';`
+ (the ledger table of openvibe-sdk/db). The events in it are lost; restore them from the backup if they matter.
+
+## Rehearsal
+
+The seed is PostgreSQL ([cutover-12-seed.sql](cutover-12-seed.sql), loaded after 0001–0005, before 0006): five hot
+events at 400, 60, 45 (a tombstone), 45 (sandbox) and 1 day(s) old, a subscription and two deliveries. The commands
+apply 0006 through the service's own `openDb` (a second run applies nothing), run the tiered prune with the
+defaults, and check the result: the 400-day event is gone from both tiers, the 60- and 45-day production events are in
+the replay tier (the tombstone still a tombstone, the delivery gone with the hot row), the sandbox event is deleted, the
+1-day event stays hot with its pending delivery, one scan from the start spans both tiers in seq order, a by-id read
+finds the archived event, and a second prune moves nothing.
+
+```rehearse
+seed: docs/cutover-12-seed.sql
+node scripts/events-archive.js status
+node -e "const a=require('assert');const {load}=require('./server/config');const {openDb,createStore}=require('./server/store');(async()=>{const db=await openDb(load(),{log:{log(){},warn(){}}});const s=createStore(db);const r=await s.prune({retentionDays:30,replayRetentionDays:365});console.log(JSON.stringify(r));a.strictEqual(r.archived,3);a.strictEqual(r.replay,1);const n=async(t)=>(await db.prepare('SELECT COUNT(*)::int AS n FROM '+t).get()).n;a.strictEqual(await n('events'),1);a.strictEqual(await n('events_archive'),2);a.strictEqual(await n('deliveries'),1);const {rows}=await s.scan(0);a.deepStrictEqual(rows.map(x=>x.seq),[2,3,5]);a.strictEqual(rows[1].redacted_by,'evt_rehearsal_0005');const e=await s.getEvent('evt_rehearsal_0002');a.strictEqual(JSON.parse(e.payload).title,'replay ✓');a.strictEqual(await s.getEvent('evt_rehearsal_0001'),null);a.strictEqual((await s.prune({retentionDays:30,replayRetentionDays:365})).archived,0);await db.close();console.log('rehearsal ok')})().catch(e=>{console.error(e);process.exit(1)})"
+node scripts/events-archive.js status
+```
new file mode 100644
new file mode 100644
@@ -0,0 +1,38 @@
@@ -0,0 +1,38 @@
+-- phase: expand
+-- ADR-042 decision 8: three tiers. hot = `events` (pruned at retention.hot, EVENTS_RETENTION_DAYS); replay =
+-- `events_archive` in the same database, pruned at retention.replay (EVENTS_REPLAY_RETENTION_DAYS); archive = monthly
+-- NDJSON objects outside the database (scripts/events-archive.js, operator only). The prune MOVES a hot event here in
+-- the transaction that deletes it, so a position is always in exactly one tier. No delivery rows follow it.
+--
+-- A row keeps, as columns, what the read path filters and orders on: identity, topic, the cursor position (seq in the
+-- epoch it was moved in: the same pair server/cursor.js encodes), owner and visibility, the redaction markers and
+-- received_at. Everything else of the event row (actor, payload, trace, …) is one gzip'd JSON object in `body`
+-- (server/archive.js), restored to the hot row's exact column values on read. Additive only: nothing in `events` changes.
+CREATE TABLE events_archive (
+ id text COLLATE "C" PRIMARY KEY,
+ seq bigint NOT NULL,
+ epoch bigint NOT NULL,
+ event_type text COLLATE "C" NOT NULL,
+ source text COLLATE "C" NOT NULL,
+ subject_type text COLLATE "C" NOT NULL,
+ subject_id text COLLATE "C" NOT NULL,
+ visibility text COLLATE "C" NOT NULL,
+ project_id text COLLATE "C",
+ env text COLLATE "C" NOT NULL DEFAULT 'production',
+ received_at bigint NOT NULL,
+ redacted_at bigint,
+ redacts bigint NOT NULL DEFAULT 0,
+ body bytea NOT NULL,
+ archived_at bigint NOT NULL,
+ -- An operator restore (scripts/events-archive.js restore) keeps the rows it brings back past retention.replay
+ -- until this time (ms); NULL for rows the prune moved.
+ hold_until bigint
+);
+-- Pull and scan: seq order within the current epoch (also the one position per epoch: a re-imported store whose seq
+-- collides with an archived row of the same epoch makes the move fail instead of silently losing the hot row).
+CREATE UNIQUE INDEX idx_events_archive_position ON events_archive(epoch, seq);
+CREATE INDEX idx_events_archive_type_seq ON events_archive(event_type, seq);
+-- Replay pruning and the monthly export.
+CREATE INDEX idx_events_archive_received ON events_archive(received_at);
+-- Redaction by subject reaches archived rows too.
+CREATE INDEX idx_events_archive_subject ON events_archive(source, subject_type, subject_id);
new file mode 100644
new file mode 100644
@@ -0,0 +1,112 @@
@@ -0,0 +1,112 @@
+'use strict';
+/**
+ * Where the archive tier's monthly objects live (ADR-042 decision 8; scripts/events-archive.js only — never an online
+ * path). Two adapters with one shape, { name, put(key, buf), get(key) -> Buffer | null }:
+ *
+ * local a directory, EVENTS_ARCHIVE_DIR (default data/archive). A put writes a temporary file, fsyncs it and
+ * renames it into place, so an object is either whole or absent.
+ * s3 an S3-compatible bucket, only when EVENTS_ARCHIVE_S3_BUCKET is set: path-style requests to
+ * EVENTS_ARCHIVE_S3_ENDPOINT (https://…), signed with AWS Signature V4 by node:crypto and sent with node's
+ * fetch — no dependency. EVENTS_ARCHIVE_S3_REGION (default us-east-1), EVENTS_ARCHIVE_S3_ACCESS_KEY_ID,
+ * EVENTS_ARCHIVE_S3_SECRET_ACCESS_KEY, EVENTS_ARCHIVE_S3_PREFIX (default events/). The body is signed too
+ * (x-amz-content-sha256), so the store refuses a body that changed on the way.
+ */
+const fs = require('fs');
+const path = require('path');
+const crypto = require('crypto');
+
+const KEY_RE = /^[a-z0-9][a-z0-9._/-]{0,200}$/;
+const checkKey = (key) => {
+ if (!KEY_RE.test(key) || key.includes('..')) throw new Error(`archive storage: bad object key ${JSON.stringify(key)}`);
+ return key;
+};
+
+function localStorage({ dir }) {
+ const root = path.resolve(dir);
+ return {
+ name: `local:${root}`,
+ async put(key, buf) {
+ const file = path.join(root, checkKey(key));
+ fs.mkdirSync(path.dirname(file), { recursive: true });
+ const tmp = `${file}.tmp-${process.pid}-${crypto.randomBytes(4).toString('hex')}`;
+ const fd = fs.openSync(tmp, 'w', 0o640);
+ try { fs.writeSync(fd, buf); fs.fsyncSync(fd); } finally { fs.closeSync(fd); }
+ fs.renameSync(tmp, file);
+ },
+ async get(key) {
+ try { return fs.readFileSync(path.join(root, checkKey(key))); } catch (err) { if (err.code === 'ENOENT') return null; throw err; }
+ },
+ };
+}
+
+const sha256 = (data) => crypto.createHash('sha256').update(data).digest('hex');
+const hmac = (key, data) => crypto.createHmac('sha256', key).update(data).digest();
+/** RFC 3986 encoding of one path segment, as SigV4 wants it. */
+const encodeSegment = (s) => encodeURIComponent(s).replace(/[!'()*]/g, c => `%${c.charCodeAt(0).toString(16).toUpperCase()}`);
+
+/**
+ * AWS Signature V4 for one request: the Authorization header value. `headers` are the headers to sign (lower-case
+ * names; host included), `path` the already-encoded absolute path, `query` the canonical query string ('' for none).
+ */
+function signV4({ method, path: p, query = '', headers, payloadHash, region, service, accessKeyId, secretAccessKey, amzDate }) {
+ const names = Object.keys(headers).map(h => h.toLowerCase()).sort();
+ const lower = Object.fromEntries(Object.entries(headers).map(([k, v]) => [k.toLowerCase(), String(v).trim().replace(/\s+/g, ' ')]));
+ const signedHeaders = names.join(';');
+ const canonical = [method, p, query, names.map(n => `${n}:${lower[n]}\n`).join(''), signedHeaders, payloadHash].join('\n');
+ const day = amzDate.slice(0, 8);
+ const scope = `${day}/${region}/${service}/aws4_request`;
+ const toSign = ['AWS4-HMAC-SHA256', amzDate, scope, sha256(canonical)].join('\n');
+ const key = hmac(hmac(hmac(hmac(`AWS4${secretAccessKey}`, day), region), service), 'aws4_request');
+ const signature = crypto.createHmac('sha256', key).update(toSign).digest('hex');
+ return `AWS4-HMAC-SHA256 Credential=${accessKeyId}/${scope}, SignedHeaders=${signedHeaders}, Signature=${signature}`;
+}
+
+function s3Storage({ endpoint, bucket, region = 'us-east-1', accessKeyId, secretAccessKey, prefix = 'events/', fetchImpl = fetch, now = () => new Date() }) {
+ if (!endpoint || !bucket || !accessKeyId || !secretAccessKey) {
+ throw new Error('archive storage: s3 needs EVENTS_ARCHIVE_S3_ENDPOINT, _BUCKET, _ACCESS_KEY_ID and _SECRET_ACCESS_KEY');
+ }
+ const base = new URL(endpoint);
+ if (base.protocol !== 'https:' && !['127.0.0.1', 'localhost', '[::1]'].includes(base.hostname)) {
+ throw new Error('archive storage: EVENTS_ARCHIVE_S3_ENDPOINT must be https (http only on loopback)');
+ }
+ const basePath = base.pathname.replace(/\/+$/, '');
+ async function request(method, key, body) {
+ const objectPath = `${basePath}/${encodeSegment(bucket)}/${checkKey(`${prefix}${key}`).split('/').map(encodeSegment).join('/')}`;
+ const amzDate = now().toISOString().replace(/[-:]/g, '').replace(/\.\d{3}/, '');
+ const payloadHash = sha256(body || '');
+ const headers = { host: base.host, 'x-amz-content-sha256': payloadHash, 'x-amz-date': amzDate };
+ const authorization = signV4({ method, path: objectPath, headers, payloadHash, region, service: 's3', accessKeyId, secretAccessKey, amzDate });
+ const res = await fetchImpl(`${base.origin}${objectPath}`, {
+ method, body, redirect: 'manual',
+ headers: { 'x-amz-content-sha256': payloadHash, 'x-amz-date': amzDate, authorization },
+ });
+ return res;
+ }
+ return {
+ name: `s3:${base.host}/${bucket}/${prefix}`,
+ async put(key, buf) {
+ const res = await request('PUT', key, buf);
+ if (!res.ok) throw new Error(`archive storage: PUT ${key} answered ${res.status}`);
+ },
+ async get(key) {
+ const res = await request('GET', key);
+ if (res.status === 404) return null;
+ if (!res.ok) throw new Error(`archive storage: GET ${key} answered ${res.status}`);
+ return Buffer.from(await res.arrayBuffer());
+ },
+ };
+}
+
+/** The adapter the environment names: s3 when EVENTS_ARCHIVE_S3_BUCKET is set, else the local directory. */
+function fromEnv(env = process.env, { fetchImpl } = {}) {
+ if (env.EVENTS_ARCHIVE_S3_BUCKET) {
+ return s3Storage({
+ endpoint: env.EVENTS_ARCHIVE_S3_ENDPOINT, bucket: env.EVENTS_ARCHIVE_S3_BUCKET, region: env.EVENTS_ARCHIVE_S3_REGION || undefined,
+ accessKeyId: env.EVENTS_ARCHIVE_S3_ACCESS_KEY_ID, secretAccessKey: env.EVENTS_ARCHIVE_S3_SECRET_ACCESS_KEY,
+ prefix: env.EVENTS_ARCHIVE_S3_PREFIX ?? undefined, ...(fetchImpl ? { fetchImpl } : {}),
+ });
+ }
+ return localStorage({ dir: env.EVENTS_ARCHIVE_DIR || path.join(__dirname, '..', 'data', 'archive') });
+}
+
+module.exports = { localStorage, s3Storage, fromEnv, signV4 };
new file mode 100644
new file mode 100644
@@ -0,0 +1,189 @@
@@ -0,0 +1,189 @@
+#!/usr/bin/env node
+'use strict';
+/**
+ * The archive tier (ADR-042 decision 8): monthly NDJSON objects outside the database, written and read back only by
+ * this operator job. Nothing runs it automatically and no online path reads an object.
+ *
+ * node scripts/events-archive.js export --month YYYY-MM [--older-than-days 90]
+ * node scripts/events-archive.js restore --month YYYY-MM [--hold-days 30]
+ * node scripts/events-archive.js status
+ *
+ * export the replay-tier rows (events_archive) received in that closed month and older than --older-than-days go
+ * into one object, <YYYY>/events-<YYYY-MM>.ndjson.gz (one event row per line: the hot row's exact column
+ * values — payload and actor as their stored JSON text — plus its epoch and archived_at), and a manifest
+ * beside it, <YYYY>/events-<YYYY-MM>.json ({ format, month, count, sha256, bytes, exported_at }). The rows
+ * are deleted from events_archive only after the object was read back and matched the manifest's count and
+ * sha256, and only those still as exported (a row redacted meanwhile stays; rerun the export). A rerun, or an
+ * export of what is left of a month, merges into the month's object: rows already in it are kept, a row
+ * still in the database replaces its line.
+ * restore reads the month's object back (its sha256 and count checked against the manifest first) into
+ * events_archive, idempotent on event_id, held for --hold-days from now: the replay prune leaves a restored
+ * row alone until then, whatever its age. Pulls see restored rows of the current epoch.
+ * status rows per month in the replay tier, and the hot store's oldest event.
+ *
+ * Storage: scripts/archive-storage.js — EVENTS_ARCHIVE_DIR (local directory, default data/archive), or an
+ * S3-compatible bucket when EVENTS_ARCHIVE_S3_BUCKET is set. The database is DATABASE_URL (PostgreSQL, ADR-035).
+ * Export and restore of one month are serialised by a PostgreSQL advisory lock.
+ */
+const crypto = require('crypto');
+const zlib = require('zlib');
+const archive = require('../server/archive');
+
+const DAY_MS = 24 * 60 * 60 * 1000;
+const FORMAT = 'openvibe.events.archive@1';
+const LINE = [...archive.COLUMNS, ...archive.BODY, 'epoch', 'archived_at'];
+const CHUNK = 1000;
+
+const sha256 = (buf) => crypto.createHash('sha256').update(buf).digest('hex');
+
+/** [start, end) of a UTC month, in ms. Throws on anything but YYYY-MM. */
+function monthRange(month) {
+ const m = /^(\d{4})-(0[1-9]|1[0-2])$/.exec(String(month || ''));
+ if (!m) throw new Error('--month must be YYYY-MM');
+ return [Date.UTC(Number(m[1]), Number(m[2]) - 1, 1), Date.UTC(Number(m[1]), Number(m[2]), 1)];
+}
+const objectKey = (month) => `${month.slice(0, 4)}/events-${month}.ndjson.gz`;
+const manifestKey = (month) => `${month.slice(0, 4)}/events-${month}.json`;
+
+/** An events_archive row → its NDJSON line object, keys in a fixed order. */
+function toLine(a) {
+ const row = archive.unpack(a);
+ const out = {};
+ for (const k of LINE) out[k] = k === 'archived_at' ? a.archived_at : row[k];
+ return out;
+}
+
+/** The month's object as line objects, after checking it against its manifest; null when there is none. */
+async function readObject(storage, month) {
+ const [manifestBuf, gz] = [await storage.get(manifestKey(month)), await storage.get(objectKey(month))];
+ if (!manifestBuf && !gz) return null;
+ if (!manifestBuf || !gz) throw new Error(`archive ${month}: object and manifest must both exist (found only the ${gz ? 'object' : 'manifest'})`);
+ const manifest = JSON.parse(manifestBuf.toString('utf8'));
+ if (manifest.format !== FORMAT || manifest.month !== month) throw new Error(`archive ${month}: the manifest is not a ${FORMAT} manifest of this month`);
+ if (sha256(gz) !== manifest.sha256) throw new Error(`archive ${month}: sha256 of the object does not match its manifest`);
+ const text = zlib.gunzipSync(gz).toString('utf8');
+ const lines = text ? text.split('\n').filter(Boolean).map((l) => JSON.parse(l)) : [];
+ if (lines.length !== manifest.count) throw new Error(`archive ${month}: the object holds ${lines.length} rows, its manifest says ${manifest.count}`);
+ return { manifest, lines };
+}
+
+/** Serialise export and restore of one month across processes (inside the caller's transaction). */
+const lockMonth = (db, month) => db.prepare("SELECT pg_advisory_xact_lock(hashtext('events-archive:' || ?)) AS ok").get(month);
+
+/**
+ * Export `month` (see the header). Returns { month, exported, count, deleted, changed, object } — exported: rows read
+ * from the database this run; count: rows in the object; deleted: rows removed from events_archive; changed: rows
+ * left in place because they changed (were redacted) after they were read.
+ */
+async function exportMonth({ db, storage, month, now = Date.now(), olderThanDays = 90 }) {
+ const [start, end] = monthRange(month);
+ if (end > now) throw new Error(`archive ${month}: the month has not ended yet`);
+ if (!(Number.isInteger(olderThanDays) && olderThanDays >= 0)) throw new Error('--older-than-days must be an integer >= 0');
+ const cutoff = Math.min(end, now - olderThanDays * DAY_MS);
+ return await db.tx(async () => {
+ await lockMonth(db, month);
+ const fresh = [];
+ if (cutoff > start) {
+ const page = db.prepare(`SELECT * FROM events_archive WHERE received_at >= ? AND received_at < ? AND (epoch, seq) > (?, ?)
+ ORDER BY epoch, seq LIMIT ?`);
+ let at = [-1, -1];
+ for (;;) {
+ const rows = await page.all(start, cutoff, at[0], at[1], CHUNK);
+ for (const r of rows) fresh.push(toLine(r));
+ if (rows.length < CHUNK) break;
+ at = [rows[rows.length - 1].epoch, rows[rows.length - 1].seq];
+ }
+ }
+ const existing = await readObject(storage, month);
+ if (!fresh.length && !existing) return { month, exported: 0, count: 0, deleted: 0, changed: 0, object: null };
+ const merged = new Map((existing ? existing.lines : []).map((l) => [l.id, l]));
+ for (const l of fresh) merged.set(l.id, l);
+ const lines = [...merged.values()].sort((x, y) => x.epoch - y.epoch || x.seq - y.seq);
+ const gz = zlib.gzipSync(Buffer.from(lines.map((l) => `${JSON.stringify(l)}\n`).join(''), 'utf8'));
+ const manifest = { format: FORMAT, month, count: lines.length, sha256: sha256(gz), bytes: gz.length, exported_at: new Date(now).toISOString() };
+ await storage.put(objectKey(month), gz);
+ await storage.put(manifestKey(month), Buffer.from(`${JSON.stringify(manifest, null, 2)}\n`));
+
+ // Verified before anything is deleted: the object as stored has the manifest's sha256 and count, and every
+ // row read from the database this run is in it, line for line.
+ const back = await readObject(storage, month);
+ if (!back || back.manifest.sha256 !== manifest.sha256) throw new Error(`archive ${month}: the stored object is not the one written; nothing deleted`);
+ const stored = new Map(back.lines.map((l) => [l.id, JSON.stringify(l)]));
+ for (const l of fresh) {
+ if (stored.get(l.id) !== JSON.stringify(l)) throw new Error(`archive ${month}: row ${l.id} is missing from the stored object; nothing deleted`);
+ }
+
+ // Delete only what is still as exported: a row redacted since it was read keeps its (tombstoned) replay copy.
+ let deleted = 0;
+ let changed = 0;
+ const lock = db.prepare('SELECT id, redacted_at FROM events_archive WHERE id = ANY(?) FOR UPDATE');
+ const drop = db.prepare('DELETE FROM events_archive WHERE id = ANY(?)');
+ for (let i = 0; i < fresh.length; i += CHUNK) {
+ const part = fresh.slice(i, i + CHUNK);
+ const exported = new Map(part.map((l) => [l.id, l.redacted_at ?? null]));
+ const current = await lock.all(part.map((l) => l.id));
+ const same = current.filter((r) => (r.redacted_at ?? null) === exported.get(r.id)).map((r) => r.id);
+ changed += current.length - same.length;
+ if (same.length) deleted += (await drop.run(same)).changes;
+ }
+ return { month, exported: fresh.length, count: lines.length, deleted, changed, object: objectKey(month) };
+ });
+}
+
+/** Restore `month` into events_archive (see the header). Returns { month, count, inserted, skipped }. */
+async function restoreMonth({ db, storage, month, now = Date.now(), holdDays = 30 }) {
+ monthRange(month);
+ if (!(Number.isInteger(holdDays) && holdDays >= 1)) throw new Error('--hold-days must be an integer >= 1');
+ return await db.tx(async () => {
+ await lockMonth(db, month);
+ const obj = await readObject(storage, month);
+ if (!obj) throw new Error(`archive ${month}: no object in ${storage.name}`);
+ const put = db.prepare(`INSERT INTO events_archive (${archive.COLUMNS.join(', ')}, epoch, body, archived_at, hold_until)
+ VALUES (${archive.COLUMNS.map(() => '?').join(', ')}, ?, ?, ?, ?) ON CONFLICT (id) DO NOTHING`);
+ let inserted = 0;
+ for (const l of obj.lines) {
+ inserted += (await put.run(...archive.COLUMNS.map((c) => l[c]), l.epoch, archive.pack(l), l.archived_at, now + holdDays * DAY_MS)).changes;
+ }
+ return { month, count: obj.lines.length, inserted, skipped: obj.lines.length - inserted };
+ });
+}
+
+/** Rows per UTC month in the replay tier, and the hot store's oldest event. */
+async function status({ db }) {
+ const months = await db.prepare(`SELECT to_char(to_timestamp(received_at / 1000.0) AT TIME ZONE 'UTC', 'YYYY-MM') AS month,
+ COUNT(*)::bigint AS rows, MIN(seq) AS first_seq, MAX(seq) AS last_seq, COUNT(hold_until)::bigint AS held
+ FROM events_archive GROUP BY 1 ORDER BY 1`).all();
+ const hot = await db.prepare('SELECT MIN(seq) AS oldest_seq, MIN(received_at) AS oldest_received_at, COUNT(*)::bigint AS rows FROM events').get();
+ return { replay: months, hot };
+}
+
+async function main(argv = process.argv.slice(2)) {
+ const [command, ...rest] = argv;
+ const opt = (k, d) => { const i = rest.indexOf(k); return i >= 0 ? rest[i + 1] : d; };
+ if (!['export', 'restore', 'status'].includes(command)) {
+ console.error('usage: events-archive.js export --month YYYY-MM [--older-than-days 90] | restore --month YYYY-MM [--hold-days 30] | status');
+ return 2;
+ }
+ require('dotenv').config();
+ const { load } = require('../server/config');
+ const { openDb } = require('../server/store');
+ const { fromEnv } = require('./archive-storage');
+ const db = await openDb(load(), { log: { log() {}, warn: console.warn } });
+ try {
+ const storage = fromEnv(process.env);
+ let r;
+ if (command === 'export') r = await exportMonth({ db, storage, month: opt('--month'), olderThanDays: Number(opt('--older-than-days', '90')) });
+ else if (command === 'restore') r = await restoreMonth({ db, storage, month: opt('--month'), holdDays: Number(opt('--hold-days', '30')) });
+ else r = await status({ db });
+ console.log(JSON.stringify({ storage: command === 'status' ? undefined : storage.name, ...r }, null, 2));
+ return 0;
+ } finally {
+ await db.close();
+ }
+}
+
+if (require.main === module) {
+ main().then((code) => process.exit(code), (err) => { console.error(`events-archive: ${err.message}`); process.exit(1); });
+}
+
+module.exports = { exportMonth, restoreMonth, status, monthRange, objectKey, manifestKey, FORMAT };
@@ -6,9 +6,10 @@
@@ -6,9 +6,10 @@
* -> { events: [{ seq, cursor, event }], next_after_seq, next_cursor, latest_seq, gap? }
* -> { events: [{ seq, cursor, event }], next_after_seq, next_cursor, latest_seq, gap? }
* `after=` takes an opaque cursor beside after_seq (ADR-042 decision 7); a cursor from another
* `after=` takes an opaque cursor beside after_seq (ADR-042 decision 7); a cursor from another
* retention epoch answers `gap` — never a silent restart. `gap` ({ from_seq, to_seq }) means
* retention epoch answers `gap` — never a silent restart. `gap` ({ from_seq, to_seq }) means
- * events after the position were already pruned by retention (or the epoch changed).
+ * events after the position were already pruned by retention (or the epoch changed). The pull
+ * spans the hot store and the replay tier (decision 8) with the one cursor: `gap` only outside both.
* Keep next_cursor as the cursor; it moves past events that did not match.
* Keep next_cursor as the cursor; it moves past events that did not match.
- * GET /api/v1/events/:event_id -> { seq, cursor, event }
+ * GET /api/v1/events/:event_id -> { seq, cursor, event } (hot or replay)
* GET /api/v1/checkpoints?topic=… / PUT /api/v1/checkpoints { topic, cursor }
* GET /api/v1/checkpoints?topic=… / PUT /api/v1/checkpoints { topic, cursor }
* a consumer's own stored cursor per topic pattern (consumer = calling principal). `cursor` is
* a consumer's own stored cursor per topic pattern (consumer = calling principal). `cursor` is
* the numeric position; PUT also accepts an opaque cursor string, and both answers carry the
* the numeric position; PUT also accepts an opaque cursor string, and both answers carry the
@@ -111,7 +112,8 @@ function readRouter({ store, auth, worker, limits }) {
@@ -111,7 +112,8 @@ function readRouter({ store, auth, worker, limits }) {
const row = await store.getEvent(String(req.params.id));
const row = await store.getEvent(String(req.params.id));
const visible = row && (req.principal.kind === 'app' ? apps.visibleToApp(row, req.principal) : (row.env || 'production') === 'production');
const visible = row && (req.principal.kind === 'app' ? apps.visibleToApp(row, req.principal) : (row.env || 'production') === 'production');
if (!visible) return http.sendProblem(res, 404, 'events.not_found', { detail: 'no such event (or pruned by retention)', ctx: req.ov });
if (!visible) return http.sendProblem(res, 404, 'events.not_found', { detail: 'no such event (or pruned by retention)', ctx: req.ov });
- return res.json({ seq: row.seq, cursor: cursor.encode(row.seq, await store.epoch()), event: rowToEnvelope(row) });
+ // A replay-tier row names the epoch its position belongs to; a hot row is in the current one.
+ return res.json({ seq: row.seq, cursor: cursor.encode(row.seq, row.epoch ?? await store.epoch()), event: rowToEnvelope(row) });
});
});
const consumerOf = (req) => req.principal.service || req.principal.sub;
const consumerOf = (req) => req.principal.service || req.principal.sub;
new file mode 100644
new file mode 100644
@@ -0,0 +1,33 @@
@@ -0,0 +1,33 @@
+'use strict';
+/**
+ * The replay tier's row shape (ADR-042 decision 8; migrations/0006_events_archive.sql). An archived row keeps as columns
+ * what the read path filters and orders on (COLUMNS); every other column of the hot row (BODY) is one gzip'd JSON
+ * object in `body`. unpack() gives back the hot row's exact column values — actor and payload are the stored JSON
+ * text, not re-serialised — so rowToEnvelope, the visibility and redaction rules and the export see one shape for
+ * both tiers. Compression is node:zlib gzip: no dependency, and the export (scripts/events-archive.js) uses it too.
+ */
+const zlib = require('zlib');
+
+/** Columns events_archive keeps as they are in `events`. */
+const COLUMNS = ['id', 'seq', 'event_type', 'source', 'subject_type', 'subject_id', 'visibility', 'project_id', 'env', 'received_at', 'redacted_at', 'redacts'];
+/** Columns of `events` that live in the compressed body. */
+const BODY = ['version', 'actor', 'subject_revision', 'trace_id', 'priority', 'occurred_at', 'payload', 'hops', 'publisher', 'request_id', 'size_bytes', 'redacted_by'];
+
+/** The compressed body of a hot row (or of an unpacked one). */
+function pack(row) {
+ const body = {};
+ for (const c of BODY) body[c] = row[c] === undefined ? null : row[c];
+ return zlib.gzipSync(Buffer.from(JSON.stringify(body), 'utf8'));
+}
+
+/** An events_archive row → the hot row it was, plus the epoch its position belongs to. */
+function unpack(a) {
+ const body = JSON.parse(zlib.gunzipSync(a.body).toString('utf8'));
+ const row = {};
+ for (const c of COLUMNS) row[c] = a[c];
+ for (const c of BODY) row[c] = body[c] === undefined ? null : body[c];
+ row.epoch = a.epoch;
+ return row;
+}
+
+module.exports = { pack, unpack, COLUMNS, BODY };
@@ -111,6 +111,9 @@ function load(env = process.env) {
@@ -111,6 +111,9 @@ function load(env = process.env) {
stream: env.NATS_STREAM || 'OV_EVENTS_STREAM',
stream: env.NATS_STREAM || 'OV_EVENTS_STREAM',
},
},
retentionDays: int(env.EVENTS_RETENTION_DAYS, 30),
retentionDays: int(env.EVENTS_RETENTION_DAYS, 30),
+ // retention.replay (ADR-042 decision 8): days an event stays readable in events_archive after it leaves the
+ // hot store, counted from receipt like retentionDays. 0: no replay tier (pruned events are deleted, as before).
+ replayRetentionDays: Math.max(0, int(env.EVENTS_REPLAY_RETENTION_DAYS, 365)),
receiptRetentionDays: int(env.EVENTS_RECEIPT_RETENTION_DAYS, 90),
receiptRetentionDays: int(env.EVENTS_RECEIPT_RETENTION_DAYS, 90),
pruneIntervalMs: int(env.EVENTS_PRUNE_INTERVAL_MS, 60 * 60 * 1000),
pruneIntervalMs: int(env.EVENTS_PRUNE_INTERVAL_MS, 60 * 60 * 1000),
@@ -67,8 +67,13 @@ async function start({
@@ -67,8 +67,13 @@ async function start({
const prune = async () => {
const prune = async () => {
try {
try {
- const r = await store.prune({ retentionDays: config.retentionDays, receiptRetentionDays: config.receiptRetentionDays, sandboxRetentionDays: config.apps.sandboxRetentionDays });
- if (r.events || r.receipts) log.log(`[retention] pruned ${r.events} events, ${r.receipts} receipts`);
+ // Tiers (ADR-042 decision 8): hot events past EVENTS_RETENTION_DAYS move to the replay tier, which keeps them
+ // for EVENTS_REPLAY_RETENTION_DAYS (0: no replay tier, deleted as before).
+ const r = await store.prune({
+ retentionDays: config.retentionDays, replayRetentionDays: config.replayRetentionDays,
+ receiptRetentionDays: config.receiptRetentionDays, sandboxRetentionDays: config.apps.sandboxRetentionDays,
+ });
+ if (r.events || r.replay || r.receipts) log.log(`[retention] pruned ${r.events} hot events (${r.archived} moved to replay), ${r.replay} replay events, ${r.receipts} receipts`);
} catch (err) {
} catch (err) {
log.error(`[retention] prune failed: ${err.message}`);
log.error(`[retention] prune failed: ${err.message}`);
}
}
@@ -24,6 +24,10 @@ function limitsOf(config) {
@@ -24,6 +24,10 @@ function limitsOf(config) {
production: off(q.production.retainedBytes), sandbox: off(q.sandbox.retainedBytes), exceeded: '429 events.quota_exceeded' },
production: off(q.production.retainedBytes), sandbox: off(q.sandbox.retainedBytes), exceeded: '429 events.quota_exceeded' },
{ id: 'retention_days', label: 'Days an event is kept', capability: 'events.app.read', unit: 'days',
{ id: 'retention_days', label: 'Days an event is kept', capability: 'events.app.read', unit: 'days',
production: config.retentionDays, sandbox: config.apps.sandboxRetentionDays, exceeded: 'pruned' },
production: config.retentionDays, sandbox: config.apps.sandboxRetentionDays, exceeded: 'pruned' },
+ // The replay window (ADR-042 decision 8): an event past retention_days is still pulled and replayed from the
+ // replay tier until replay_retention_days after receipt (0 = no replay tier). Sandbox events never enter it.
+ { id: 'replay_retention_days', label: 'Days an event can still be replayed', capability: 'events.app.read', unit: 'days',
+ production: config.replayRetentionDays, sandbox: 0, exceeded: 'gap' },
{ id: 'subscriptions', label: 'Webhook subscriptions', capability: 'events.app.subscribe', unit: 'count',
{ id: 'subscriptions', label: 'Webhook subscriptions', capability: 'events.app.subscribe', unit: 'count',
production: off(q.production.maxSubscriptions), sandbox: off(q.sandbox.maxSubscriptions), exceeded: '429 events.quota_exceeded' },
production: off(q.production.maxSubscriptions), sandbox: off(q.sandbox.maxSubscriptions), exceeded: '429 events.quota_exceeded' },
{ id: 'payload_bytes', label: 'Payload size of one event', capability: 'events.app.publish', unit: 'bytes',
{ id: 'payload_bytes', label: 'Payload size of one event', capability: 'events.app.publish', unit: 'bytes',
@@ -19,8 +19,9 @@
@@ -19,8 +19,9 @@
* Developer-app events (app.<project_key>.*, events.app.publish) are never streamed here, to anyone.
* Developer-app events (app.<project_key>.*, events.app.publish) are never streamed here, to anyone.
*
*
* Resume: `Last-Event-ID` (or ?last_event_id=) is a cursor, or a bare seq for one release. Missed
* Resume: `Last-Event-ID` (or ?last_event_id=) is a cursor, or a bare seq for one release. Missed
- * events are replayed first; when the cursor is older than retention — or belongs to another
- * retention epoch — an `event: gap` message comes first, so the client knows to refetch state.
+ * events are replayed first, from the replay tier and then the hot store (ADR-042 decision 8); when the
+ * cursor is older than both — or belongs to another retention epoch — an `event: gap` message comes
+ * first, so the client knows to refetch state.
* Heartbeat comment every 25 s.
* Heartbeat comment every 25 s.
*
*
* Public replay window: a browser (signed out or signed in) is replayed `public` events received in
* Public replay window: a browser (signed out or signed in) is replayed `public` events received in
@@ -5,6 +5,9 @@
@@ -5,6 +5,9 @@
* consumer delivery". Global order is `seq`, handed out from a counter that never goes backwards,
* consumer delivery". Global order is `seq`, handed out from a counter that never goes backwards,
* even after retention has pruned the newest rows. Beside it a cursor (./cursor.js) names a seq plus
* even after retention has pruned the newest rows. Beside it a cursor (./cursor.js) names a seq plus
* the retention epoch — ADR-042 decision 7; seq stays on the wire for one release.
* the retention epoch — ADR-042 decision 7; seq stays on the wire for one release.
+ *
+ * Retention is tiered (ADR-042 decision 8): prune() moves a hot event older than retention.hot into events_archive
+ * (./archive.js) in the transaction that deletes it, and pulls, scans and by-id reads span both tiers with one cursor.
*/
*/
const fs = require('fs');
const fs = require('fs');
const path = require('path');
const path = require('path');
@@ -13,6 +16,7 @@ const topics = require('./topics');
@@ -13,6 +16,7 @@ const topics = require('./topics');
const apps = require('./apps');
const apps = require('./apps');
const redaction = require('./redaction');
const redaction = require('./redaction');
const cursor = require('./cursor');
const cursor = require('./cursor');
+const archive = require('./archive');
const { carrierClass, orderingKey } = require('./fabric/policy');
const { carrierClass, orderingKey } = require('./fabric/policy');
const { createUsage, deliveryCode } = require('./usage');
const { createUsage, deliveryCode } = require('./usage');
@@ -25,6 +29,26 @@ const DEV_PGLITE = path.join(__dirname, '..', 'data', 'pglite');
@@ -25,6 +29,26 @@ const DEV_PGLITE = path.join(__dirname, '..', 'data', 'pglite');
const PUBLISH_RECEIPT = 'events:publish'; // receipts' consumer column for accepted publishes
const PUBLISH_RECEIPT = 'events:publish'; // receipts' consumer column for accepted publishes
+/**
+ * Rows with seq > ? (and `filter`) from the hot store and the replay tier's current epoch, in seq order, LIMIT ?.
+ * Both branches return the hot row's columns plus `body` (NULL for a hot row; a replay row's compressed rest).
+ */
+function spanTiers(filter) {
+ return `SELECT * FROM (
+ SELECT ${[...archive.COLUMNS, ...archive.BODY].join(', ')}, NULL::bytea AS body FROM events WHERE seq > ? ${filter}
+ UNION ALL
+ SELECT ${archive.COLUMNS.join(', ')}, ${archive.BODY.map(c => `NULL AS ${c}`).join(', ')}, body FROM events_archive
+ WHERE seq > ? ${filter} AND epoch = (SELECT value FROM store_epoch WHERE name = 'events')
+ ) t ORDER BY seq LIMIT ?`;
+}
+
+/** A spanTiers row → the hot row shape. */
+function fromTier(r) {
+ if (r.body) return archive.unpack(r);
+ delete r.body;
+ return r;
+}
+
/**
/**
* The serving handle (ADR-035): DATABASE_URL through PgBouncer; in development without it, an embedded PGlite database
* The serving handle (ADR-035): DATABASE_URL through PgBouncer; in development without it, an embedded PGlite database
* in data/pglite. Migrations run first, as the owner (DATABASE_DIRECT_URL), or on the embedded handle. The columns added
* in data/pglite. Migrations run first, as the owner (DATABASE_DIRECT_URL), or on the embedded handle. The columns added
@@ -101,7 +125,12 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -101,7 +125,12 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
addReceipt: db.prepare('INSERT INTO idempotency_receipts (consumer, event_id, processed_at) VALUES (?, ?, ?) ON CONFLICT DO NOTHING'),
addReceipt: db.prepare('INSERT INTO idempotency_receipts (consumer, event_id, processed_at) VALUES (?, ?, ?) ON CONFLICT DO NOTHING'),
nextSeq: db.prepare("UPDATE sequences SET value = value + 1 WHERE name = 'events' RETURNING value"),
nextSeq: db.prepare("UPDATE sequences SET value = value + 1 WHERE name = 'events' RETURNING value"),
lastSeq: db.prepare("SELECT value FROM sequences WHERE name = 'events'"),
lastSeq: db.prepare("SELECT value FROM sequences WHERE name = 'events'"),
- minSeq: db.prepare('SELECT MIN(seq) AS s FROM events'),
+ // The oldest position either tier still holds; replay rows count in the current epoch only.
+ minSeq: db.prepare(`SELECT LEAST((SELECT MIN(seq) FROM events),
+ (SELECT MIN(seq) FROM events_archive WHERE epoch = (SELECT value FROM store_epoch WHERE name = 'events'))) AS s`),
+ getArchived: db.prepare('SELECT * FROM events_archive WHERE id = ?'),
+ archivedBySubject: db.prepare(`SELECT * FROM events_archive WHERE source = ? AND subject_type = ? AND subject_id = ?
+ AND redacted_at IS NULL AND redacts = 0`),
epoch: db.prepare("SELECT value FROM store_epoch WHERE name = 'events'"),
epoch: db.prepare("SELECT value FROM store_epoch WHERE name = 'events'"),
bumpEpoch: db.prepare("UPDATE store_epoch SET value = value + 1 WHERE name = 'events' RETURNING value"),
bumpEpoch: db.prepare("UPDATE store_epoch SET value = value + 1 WHERE name = 'events' RETURNING value"),
// Hop depth counts only rows the publisher's own tenancy can have caused: a first-party
// Hop depth counts only rows the publisher's own tenancy can have caused: a first-party
@@ -122,7 +151,8 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -122,7 +151,8 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
AND redacted_at IS NULL AND redacts = 0`),
AND redacted_at IS NULL AND redacts = 0`),
applyTombstone: db.prepare(`UPDATE events SET payload = @payload, actor = @actor, size_bytes = @size_bytes,
applyTombstone: db.prepare(`UPDATE events SET payload = @payload, actor = @actor, size_bytes = @size_bytes,
redacted_at = @redacted_at, redacted_by = @redacted_by WHERE id = @id AND redacted_at IS NULL`),
redacted_at = @redacted_at, redacted_by = @redacted_by WHERE id = @id AND redacted_at IS NULL`),
- firstSeqSince: db.prepare('SELECT MIN(seq) AS s FROM events WHERE received_at >= ?'),
+ firstSeqSince: db.prepare(`SELECT LEAST((SELECT MIN(seq) FROM events WHERE received_at >= ?),
+ (SELECT MIN(seq) FROM events_archive WHERE received_at >= ? AND epoch = (SELECT value FROM store_epoch WHERE name = 'events'))) AS s`),
enabledSubs: db.prepare('SELECT id, topic_pattern, project_id, env FROM subscriptions WHERE enabled = 1'),
enabledSubs: db.prepare('SELECT id, topic_pattern, project_id, env FROM subscriptions WHERE enabled = 1'),
projectRecent: db.prepare('SELECT COUNT(*) AS n FROM events WHERE project_id = ? AND env = ? AND received_at > ?'),
projectRecent: db.prepare('SELECT COUNT(*) AS n FROM events WHERE project_id = ? AND env = ? AND received_at > ?'),
projectBytes: db.prepare('SELECT COALESCE(SUM(size_bytes), 0)::bigint AS b FROM events WHERE project_id = ? AND env = ?'),
projectBytes: db.prepare('SELECT COALESCE(SUM(size_bytes), 0)::bigint AS b FROM events WHERE project_id = ? AND env = ?'),
@@ -137,8 +167,9 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -137,8 +167,9 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
// The carrier a (subscription, ordering key) currently has a backlog on: what pins new deliveries of the key.
// The carrier a (subscription, ordering key) currently has a backlog on: what pins new deliveries of the key.
backlogCarrier: db.prepare(`SELECT carrier FROM deliveries WHERE subscription_id = ? AND ordering_key = ? AND status IN ('pending', 'failed')
backlogCarrier: db.prepare(`SELECT carrier FROM deliveries WHERE subscription_id = ? AND ordering_key = ? AND status IN ('pending', 'failed')
AND carrier IS NOT NULL ORDER BY seq LIMIT 1`),
AND carrier IS NOT NULL ORDER BY seq LIMIT 1`),
- afterSeq: db.prepare('SELECT * FROM events WHERE seq > ? ORDER BY seq LIMIT ?'),
- afterSeqLike: db.prepare("SELECT * FROM events WHERE seq > ? AND event_type ILIKE ? ESCAPE '\\' ORDER BY seq LIMIT ?"),
+ // One statement over both tiers (one snapshot): an event the prune moves between two reads can be in neither.
+ afterSeq: db.prepare(spanTiers('')),
+ afterSeqLike: db.prepare(spanTiers("AND event_type ILIKE ? ESCAPE '\\'")),
};
};
// ── Events ───────────────────────────────────────────────
// ── Events ───────────────────────────────────────────────
@@ -167,12 +198,13 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -167,12 +198,13 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
bytes = (await q.projectBytes.get(project.projectId, project.env)).b;
bytes = (await q.projectBytes.get(project.projectId, project.env)).b;
}
}
for (const env of items) {
for (const env of items) {
- const existing = await q.getEvent.get(env.event_id);
+ // An id is stored once across the tiers: one moved to replay is still a duplicate (with its position).
+ const existing = await q.getEvent.get(env.event_id) || await q.getArchived.get(env.event_id);
if (existing) {
if (existing) {
if (existing.source !== env.source || existing.event_type !== env.event_type) {
if (existing.source !== env.source || existing.event_type !== env.event_type) {
throw new StoreError(409, 'events.id_conflict', `event ${env.event_id} already exists with different content`);
throw new StoreError(409, 'events.id_conflict', `event ${env.event_id} already exists with different content`);
}
}
- results.push({ event_id: env.event_id, seq: existing.seq, duplicate: true, cursor: cursor.encode(existing.seq, epoch) });
+ results.push({ event_id: env.event_id, seq: existing.seq, duplicate: true, cursor: cursor.encode(existing.seq, existing.epoch ?? epoch) });
continue;
continue;
}
}
if (await q.getReceipt.get(PUBLISH_RECEIPT, env.event_id)) {
if (await q.getReceipt.get(PUBLISH_RECEIPT, env.event_id)) {
@@ -286,13 +318,15 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -286,13 +318,15 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
/**
/**
* Turn the targets of `directive` (server/redaction.js) into tombstones on behalf of `by` (the
* Turn the targets of `directive` (server/redaction.js) into tombstones on behalf of `by` (the
- * stored row of the redacting event). Throws StoreError 403 when an event id names an event of
- * another owner. Returns [{ id, seq, columns }] for the rows it rewrote.
+ * stored row of the redacting event), in either tier. Throws StoreError 403 when an event id names
+ * an event of another owner. Returns [{ id, seq, columns }] for the rows it rewrote.
*/
*/
async function applyRedaction(by, directive, now) {
async function applyRedaction(by, directive, now) {
- const targets = new Map();
+ const targets = new Map(); // id -> the row (hot, or unpacked from replay)
for (const id of directive.eventIds) {
for (const id of directive.eventIds) {
- const r = await q.getEvent.get(id);
+ const hot = await q.getEvent.get(id);
+ const archived = hot ? null : await q.getArchived.get(id);
+ const r = hot || (archived && archive.unpack(archived));
if (!r) continue; // never stored, or pruned already
if (!r) continue; // never stored, or pruned already
if (!redaction.sameOwner(by, r)) {
if (!redaction.sameOwner(by, r)) {
throw new StoreError(403, 'events.redaction_not_allowed', `event ${id} belongs to ${r.source}; ${by.source} may redact only its own events`,
throw new StoreError(403, 'events.redaction_not_allowed', `event ${id} belongs to ${r.source}; ${by.source} may redact only its own events`,
@@ -305,17 +339,33 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -305,17 +339,33 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
for (const r of await q.redactBySubject.all(by.source, directive.subjectType, subjectId)) {
for (const r of await q.redactBySubject.all(by.source, directive.subjectType, subjectId)) {
if (redaction.sameOwner(by, r)) targets.set(r.id, r);
if (redaction.sameOwner(by, r)) targets.set(r.id, r);
}
}
+ for (const a of await q.archivedBySubject.all(by.source, directive.subjectType, subjectId)) {
+ const r = archive.unpack(a);
+ if (redaction.sameOwner(by, r)) targets.set(r.id, r);
+ }
}
}
}
}
const out = [];
const out = [];
for (const r of targets.values()) {
for (const r of targets.values()) {
if (r.id === by.id) continue;
if (r.id === by.id) continue;
const columns = redaction.tombstone(r, { by: by.id, at: now });
const columns = redaction.tombstone(r, { by: by.id, at: now });
- if ((await q.applyTombstone.run({ id: r.id, ...columns })).changes) out.push({ id: r.id, seq: r.seq, columns });
+ // A hot row the prune moved since it was read is tombstoned where it is now: the move holds the row lock
+ // until it commits, so the hot UPDATE finds nothing and the replay row is visible to the next statement.
+ if ((await q.applyTombstone.run({ id: r.id, ...columns })).changes || await tombstoneArchived(r.id, columns)) {
+ out.push({ id: r.id, seq: r.seq, columns });
+ }
}
}
return out;
return out;
}
}
+ /** Rewrite a replay row as a tombstone (its body re-packed). False when it is not there or already redacted. */
+ async function tombstoneArchived(id, columns) {
+ const a = await db.prepare('SELECT * FROM events_archive WHERE id = ? AND redacted_at IS NULL FOR UPDATE').get(id);
+ if (!a) return false;
+ const row = { ...archive.unpack(a), ...columns };
+ return (await db.prepare('UPDATE events_archive SET body = ?, redacted_at = ? WHERE id = ?').run(archive.pack(row), columns.redacted_at, id)).changes === 1;
+ }
+
/**
/**
* Operator redaction (scripts/redact-backfill.js): what an event of first-party `source` carrying
* Operator redaction (scripts/redact-backfill.js): what an event of first-party `source` carrying
* `directive` (the payload.redacts object) would have redacted. The tombstones name `by` as
* `directive` (the payload.redacts object) would have redacted. The tombstones name `by` as
@@ -354,8 +404,12 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -354,8 +404,12 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
return r ? r.revoked_at : null;
return r ? r.revoked_at : null;
}
}
+ /** The stored row of an event, hot or replay (an unpacked replay row also carries its `epoch`); null when neither tier has it. */
async function getEvent(id) {
async function getEvent(id) {
- return await q.getEvent.get(id) || null;
+ const hot = await q.getEvent.get(id);
+ if (hot) return hot;
+ const archived = await q.getArchived.get(id);
+ return archived ? archive.unpack(archived) : null;
}
}
/** The resolved delivery policy (../fabric/policy.js) for an event type, or null when no library is attached. */
/** The resolved delivery policy (../fabric/policy.js) for an event type, or null when no library is attached. */
@@ -369,7 +423,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -369,7 +423,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
/** First seq received at or after `ms`; the next seq to be handed out when there is none. */
/** First seq received at or after `ms`; the next seq to be handed out when there is none. */
async function firstSeqSince(ms) {
async function firstSeqSince(ms) {
- const m = (await q.firstSeqSince.get(ms)).s;
+ const m = (await q.firstSeqSince.get(ms, ms)).s;
return m == null ? await lastSeq() + 1 : m;
return m == null ? await lastSeq() + 1 : m;
}
}
@@ -393,7 +447,8 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -393,7 +447,8 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
/**
/**
* Events with seq > afterSeq that match any of `patterns`, in order, filtered by `accept(row)`.
* Events with seq > afterSeq that match any of `patterns`, in order, filtered by `accept(row)`.
* Scans at most `scanMax` rows; returns { rows, cursor } where cursor is the last seq examined
* Scans at most `scanMax` rows; returns { rows, cursor } where cursor is the last seq examined
- * (so a pull consumer's cursor moves past rows that did not match).
+ * (so a pull consumer's cursor moves past rows that did not match). Spans the replay tier and the
+ * hot store as one sequence (ADR-042 decision 8): replay rows come back unpacked, in the hot shape.
*/
*/
async function scan(afterSeq, { patterns = ['*'], limit = 100, scanMax = 5000, accept = () => true } = {}) {
async function scan(afterSeq, { patterns = ['*'], limit = 100, scanMax = 5000, accept = () => true } = {}) {
const rows = [];
const rows = [];
@@ -401,7 +456,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -401,7 +456,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
let scanned = 0;
let scanned = 0;
const single = patterns.length === 1 && patterns[0] !== '*' ? topics.toLike(patterns[0]) : null;
const single = patterns.length === 1 && patterns[0] !== '*' ? topics.toLike(patterns[0]) : null;
for (;;) {
for (;;) {
- const page = single ? await q.afterSeqLike.all(cursor, single, 500) : await q.afterSeq.all(cursor, 500);
+ const page = (single ? await q.afterSeqLike.all(cursor, single, cursor, single, 500) : await q.afterSeq.all(cursor, cursor, 500)).map(fromTier);
for (const row of page) {
for (const row of page) {
scanned++;
scanned++;
cursor = row.seq;
cursor = row.seq;
@@ -677,13 +732,55 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -677,13 +732,55 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
// ── Retention ────────────────────────────────────────────
// ── Retention ────────────────────────────────────────────
/**
/**
- * Drop events (and their deliveries) older than retentionDays (sandbox events older than
- * sandboxRetentionDays, when shorter), and old publish receipts.
+ * Move up to `limit` hot events received before `cutoff` into the replay tier, in one transaction: the rows are
+ * locked FOR UPDATE SKIP LOCKED, copied into events_archive (idempotent on the event id) and deleted (their
+ * deliveries go with them). A row another prune — or a redaction, or a claim cascading on it — holds is skipped,
+ * never waited on: two Events processes pruning at once take disjoint rows, and a row one of them moved is gone
+ * from `events` when the other's next batch looks. Returns how many hot rows it removed.
*/
*/
- async function prune({ retentionDays = 30, receiptRetentionDays = 90, sandboxRetentionDays = retentionDays, now = clock.now() } = {}) {
- let events = (await db.prepare('DELETE FROM events WHERE received_at < ?').run(now - retentionDays * DAY_MS)).changes;
- if (sandboxRetentionDays < retentionDays) {
- events += (await db.prepare("DELETE FROM events WHERE env = 'sandbox' AND received_at < ?").run(now - sandboxRetentionDays * DAY_MS)).changes;
+ const moveBatch = async (cutoff, limit, now) => await db.tx(async () => {
+ const rows = await db.prepare(`SELECT * FROM events WHERE received_at < ? AND env != 'sandbox' ORDER BY seq LIMIT ?
+ FOR UPDATE SKIP LOCKED`).all(cutoff, limit);
+ if (!rows.length) return 0;
+ const at = (await q.epoch.get()).value;
+ const put = db.prepare(`INSERT INTO events_archive (${archive.COLUMNS.join(', ')}, epoch, body, archived_at)
+ VALUES (${archive.COLUMNS.map(() => '?').join(', ')}, ?, ?, ?) ON CONFLICT (id) DO NOTHING`);
+ for (const r of rows) await put.run(...archive.COLUMNS.map(c => r[c]), at, archive.pack(r), now);
+ return (await db.prepare('DELETE FROM events WHERE id = ANY(?)').run(rows.map(r => r.id))).changes;
+ });
+
+ /**
+ * Retention in tiers (ADR-042 decision 8). With replayRetentionDays > 0, events older than retentionDays move
+ * from the hot store into the replay tier (moveBatch, `batchSize` at a time, at most `maxBatches` per call; the
+ * next call continues), and replay rows older than replayRetentionDays are dropped. With 0 (the default here;
+ * the service passes EVENTS_REPLAY_RETENTION_DAYS) there is no replay tier: old events are deleted as before and
+ * the replay tier is emptied, so nothing is kept without a rule that prunes it. Rows an operator restored stay
+ * until their hold_until either way. Sandbox events never enter replay: they are deleted older than
+ * sandboxRetentionDays (or retentionDays, if shorter). Old publish receipts go as before.
+ */
+ async function prune({ retentionDays = 30, replayRetentionDays = 0, receiptRetentionDays = 90, sandboxRetentionDays = retentionDays,
+ now = clock.now(), batchSize = 500, maxBatches = 200 } = {}) {
+ const hotCutoff = now - retentionDays * DAY_MS;
+ let events = 0;
+ let archived = 0;
+ let replay = 0;
+ if (replayRetentionDays > 0) {
+ events += (await db.prepare("DELETE FROM events WHERE env = 'sandbox' AND received_at < ?")
+ .run(now - Math.min(retentionDays, sandboxRetentionDays) * DAY_MS)).changes;
+ for (let i = 0; i < maxBatches; i++) {
+ const n = await moveBatch(hotCutoff, batchSize, now);
+ archived += n;
+ if (n < batchSize) break; // done, or the rest is locked by a concurrent prune that is moving it
+ }
+ events += archived;
+ replay = (await db.prepare('DELETE FROM events_archive WHERE received_at < ? AND (hold_until IS NULL OR hold_until < ?)')
+ .run(now - replayRetentionDays * DAY_MS, now)).changes;
+ } else {
+ events += (await db.prepare('DELETE FROM events WHERE received_at < ?').run(hotCutoff)).changes;
+ if (sandboxRetentionDays < retentionDays) {
+ events += (await db.prepare("DELETE FROM events WHERE env = 'sandbox' AND received_at < ?").run(now - sandboxRetentionDays * DAY_MS)).changes;
+ }
+ replay = (await db.prepare('DELETE FROM events_archive WHERE hold_until IS NULL OR hold_until < ?').run(now)).changes;
}
}
const receipts = (await db.prepare('DELETE FROM idempotency_receipts WHERE processed_at < ?')
const receipts = (await db.prepare('DELETE FROM idempotency_receipts WHERE processed_at < ?')
.run(now - Math.max(receiptRetentionDays, retentionDays) * DAY_MS)).changes;
.run(now - Math.max(receiptRetentionDays, retentionDays) * DAY_MS)).changes;
@@ -693,7 +790,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -693,7 +790,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
SELECT 1 FROM delivery_placements q WHERE q.carrier_class = p.carrier_class
SELECT 1 FROM delivery_placements q WHERE q.carrier_class = p.carrier_class
AND q.ordering_key IS NOT DISTINCT FROM p.ordering_key AND q.decided_at > p.decided_at)`)
AND q.ordering_key IS NOT DISTINCT FROM p.ordering_key AND q.decided_at > p.decided_at)`)
.run(now - retentionDays * DAY_MS)).changes;
.run(now - retentionDays * DAY_MS)).changes;
- return { events, receipts, placements };
+ return { events, archived, replay, receipts, placements };
}
}
async function ping() {
async function ping() {
new file mode 100644
new file mode 100644
@@ -0,0 +1,277 @@
@@ -0,0 +1,277 @@
+'use strict';
+// Tiered retention (ADR-042 decision 8): hot (`events`) → replay (`events_archive`, same database) → archive (monthly
+// NDJSON objects, scripts/events-archive.js). Reads span hot and replay with one cursor; `gap` only outside both.
+const assert = require('assert');
+const http = require('http');
+const zlib = require('zlib');
+const { ids } = require('openvibe-contracts');
+const apps = require('../server/apps');
+const cursor = require('../server/cursor');
+const { load } = require('../server/config');
+const { limitsOf } = require('../server/limits');
+const { createStore } = require('../server/store');
+const { exportMonth, restoreMonth, objectKey, manifestKey } = require('../scripts/events-archive');
+const { localStorage, s3Storage, fromEnv, signV4 } = require('../scripts/archive-storage');
+const { boot, request, serviceToken, appToken, envelope, sse, suite, tmpDir } = require('./helpers');
+
+const t = suite('archive');
+const DAY = 86400000;
+let h;
+const live = serviceToken('live', ['events.event.publish']);
+const media = serviceToken('media', ['events.event.publish']);
+const reader = serviceToken('games', ['events.event.read']);
+const T0 = Date.now();
+const later = (days) => T0 + days * DAY;
+const tiered = (days, over = {}) => ({ retentionDays: 30, replayRetentionDays: 365, now: later(days), ...over });
+const open = [];
+
+async function publish(token, body) {
+ const r = await request(h.base, 'POST', '/api/v1/events', { token, body });
+ assert.strictEqual(r.status, 201, r.text);
+ return r.body;
+}
+const pull = async (q, token = reader) => await request(h.base, 'GET', `/api/v1/events?${q}`, { token });
+const count = async (table, id) => (await h.db.prepare(`SELECT COUNT(*)::int AS n FROM ${table} WHERE id = ?`).get(id)).n;
+
+t('boot', async () => { h = await boot(); });
+
+t('a hot-pruned event is readable through replay: by id and by pull, unchanged', async () => {
+ const e = await publish(live, envelope('live', { event_type: 'live.tier.one', visibility: 'public', payload: { n: 1, s: 'é ✓' } }));
+ const before = await request(h.base, 'GET', `/api/v1/events/${e.event_id}`, { token: reader });
+ const r = await h.store.prune(tiered(31));
+ assert.ok(r.archived >= 1 && r.events >= r.archived, JSON.stringify(r));
+ assert.deepStrictEqual([await count('events', e.event_id), await count('events_archive', e.event_id)], [0, 1], 'moved, not copied');
+ const after = await request(h.base, 'GET', `/api/v1/events/${e.event_id}`, { token: reader });
+ assert.strictEqual(after.status, 200, after.text);
+ assert.deepStrictEqual(after.body, before.body, 'the same seq, cursor and envelope from the replay tier');
+ const p = await pull('topic=live.tier.one&after_seq=0');
+ assert.deepStrictEqual(p.body.events.map(x => x.event), [before.body.event]);
+ assert.strictEqual(p.body.gap, undefined);
+ // A publish of the same id is a duplicate with its position, never a second row.
+ const again = await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope('live', { event_id: e.event_id, event_type: 'live.tier.one' }) });
+ assert.strictEqual(again.status, 200, again.text);
+ assert.deepStrictEqual([again.body.duplicate, again.body.seq, again.body.cursor], [true, e.seq, e.cursor]);
+});
+
+t('a cursor across the hot/replay boundary reports no false gap (pull and SSE)', async () => {
+ const a = await publish(live, envelope('live', { event_type: 'live.tier.cross' }));
+ const b = await publish(live, envelope('live', { event_type: 'live.tier.cross' }));
+ await h.store.prune(tiered(31));
+ assert.strictEqual(await count('events_archive', b.event_id), 1);
+ const c = await publish(live, envelope('live', { event_type: 'live.tier.cross' }));
+
+ let r = await pull(`topic=live.tier.cross&after=${a.cursor}&limit=1`);
+ assert.deepStrictEqual([r.body.events.map(x => x.seq), r.body.gap], [[b.seq], undefined], 'replay');
+ assert.strictEqual(r.body.events[0].cursor, b.cursor);
+ r = await pull(`topic=live.tier.cross&after=${r.body.next_cursor}&limit=1`);
+ assert.deepStrictEqual([r.body.events.map(x => x.seq), r.body.gap], [[c.seq], undefined], 'then hot, no gap between');
+ r = await pull(`topic=*&after_seq=${a.seq - 1}`);
+ assert.deepStrictEqual(r.body.events.map(x => x.seq), [a.seq, b.seq, c.seq]);
+ assert.strictEqual(r.body.gap, undefined);
+
+ const s = await sse(h.base, '/realtime/stream?topics=live.tier.cross', { headers: { Authorization: `Bearer ${serviceToken('live', ['events.event.read'])}`, 'Last-Event-ID': a.cursor } });
+ open.push(s);
+ await s.waitFor(x => x.events().length === 2);
+ assert.deepStrictEqual(s.events().map(x => x.seq), [b.seq, c.seq]);
+ assert.deepStrictEqual(s.gaps(), []);
+ assert.ok(s.messages.some(m => m.id === b.cursor), 'the SSE id of a replayed event is its cursor');
+ s.close();
+});
+
+t('redaction reaches replay rows (by id and by subject); the owner rule holds there too', async () => {
+ const x = await publish(live, envelope('live', { event_type: 'live.tier.secret', subject: { type: 'stream', id: 'tier-red-1' }, payload: { secret: 'a' } }));
+ const y = await publish(live, envelope('live', { event_type: 'live.tier.secret', subject: { type: 'stream', id: 'tier-red-2' }, payload: { secret: 'b' } }));
+ await h.store.prune(tiered(31));
+ assert.strictEqual(await count('events_archive', x.event_id) + await count('events_archive', y.event_id), 2);
+ const refused = await request(h.base, 'POST', '/api/v1/events', { token: media, body: envelope('media', { event_type: 'media.tier.redact', payload: { redacts: { event_ids: [x.event_id] } } }) });
+ assert.deepStrictEqual([refused.status, refused.body.code], [403, 'events.redaction_not_allowed']);
+ await publish(live, envelope('live', { event_type: 'live.tier.redact', payload: { redacts: { event_ids: [y.event_id], subject_type: 'stream', subject_ids: ['tier-red-1'] } } }));
+ for (const e of [x, y]) {
+ const r = await request(h.base, 'GET', `/api/v1/events/${e.event_id}`, { token: reader });
+ assert.strictEqual(r.body.event.payload.redacted, true, JSON.stringify(r.body));
+ assert.deepStrictEqual(r.body.event.actor, { type: 'service', id: 'live' });
+ assert.strictEqual(r.body.seq, e.seq);
+ }
+ const p = await pull('topic=live.tier.secret&after_seq=0');
+ assert.ok(p.body.events.every(e => e.event.payload.redacted === true && !('secret' in e.event.payload)));
+});
+
+t('sandbox and environment rules apply to replay rows exactly as to hot ones', async () => {
+ const prj = `prj_${ids.ulid()}`;
+ const [prodApp, sandApp] = [ids.newId('app'), ids.newId('app')];
+ const cap = ['events.app.publish', 'events.app.read'];
+ const prod = appToken({ appId: prodApp, projectId: prj, env: 'production', cap });
+ const sand = appToken({ appId: sandApp, projectId: prj, env: 'sandbox', cap });
+ const appEvent = (appId) => envelope('x', { source: apps.appSource(appId), event_type: `app.${apps.projectKey(prj)}.order.created`, actor: { type: 'app', id: appId }, subject: { type: 'order', id: '1' }, visibility: 'internal' });
+ const pe = await publish(prod, appEvent(prodApp));
+ const se = await publish(sand, appEvent(sandApp));
+ await h.store.prune(tiered(31, { sandboxRetentionDays: 7 }));
+ assert.deepStrictEqual([await count('events_archive', pe.event_id), await count('events_archive', se.event_id), await count('events', se.event_id)], [1, 0, 0],
+ 'a production event moves; a sandbox event is deleted, never kept in replay');
+ const topic = `topic=app.${apps.projectKey(prj)}.*&after_seq=0`;
+ assert.deepStrictEqual((await pull(topic, prod)).body.events.map(e => e.seq), [pe.seq], 'the app reads its own replay rows');
+ assert.deepStrictEqual((await pull(topic, sand)).body.events, [], 'never across environments');
+ assert.deepStrictEqual((await pull(topic)).body.events.map(e => e.seq), [pe.seq], 'first-party: production app events through app.*');
+ assert.deepStrictEqual((await pull('topic=*&after_seq=0')).body.events.filter(e => e.event.event_id === pe.event_id), [], '…and only through app.*');
+ assert.strictEqual((await request(h.base, 'GET', `/api/v1/events/${pe.event_id}`, { token: sand })).status, 404);
+ assert.strictEqual((await request(h.base, 'GET', `/api/v1/events/${pe.event_id}`, { token: prod })).status, 200);
+});
+
+t('two concurrent prunes move each event exactly once', async () => {
+ await h.store.prune(tiered(31)); // the other tests' hot rows first: the race below is over these 60 only
+ const mine = [];
+ for (let i = 0; i < 60; i++) mine.push((await publish(live, envelope('live', { event_type: 'live.tier.race', payload: { i } }))).event_id);
+ // Two stores on one database: separate pooled connections, as two Events processes would have.
+ const s1 = createStore(h.db, { clock: h.clock });
+ const s2 = createStore(h.db, { clock: h.clock });
+ const [r1, r2] = await Promise.all([s1.prune(tiered(31, { batchSize: 7 })), s2.prune(tiered(31, { batchSize: 7 }))]);
+ assert.strictEqual(r1.archived + r2.archived, 60, JSON.stringify([r1, r2]));
+ const rows = await h.db.prepare('SELECT id, seq FROM events_archive WHERE id = ANY(?) ORDER BY seq').all(mine);
+ assert.deepStrictEqual(rows.map(r => r.id), mine, 'every one in replay, once, in order');
+ assert.strictEqual((await h.db.prepare('SELECT COUNT(*)::int AS n FROM events WHERE id = ANY(?)').get(mine)).n, 0);
+ const again = await h.store.prune(tiered(31));
+ assert.strictEqual(again.archived, 0, 'nothing left to move');
+});
+
+t('export → restore round-trips byte-identical rows; the rows leave replay only after the object verifies', async () => {
+ const storage = localStorage({ dir: tmpDir() });
+ const month = new Date(h.clock.now()).toISOString().slice(0, 7);
+ const exportAt = later(400); // the month is closed and older than the cutoff
+ const inMonth = await h.db.prepare('SELECT * FROM events_archive ORDER BY seq').all();
+ assert.ok(inMonth.length >= 60);
+ const before = await Promise.all(inMonth.map(r => h.store.getEvent(r.id)));
+
+ const r = await exportMonth({ db: h.db, storage, month, now: exportAt, olderThanDays: 90 });
+ assert.deepStrictEqual([r.exported, r.count, r.deleted, r.changed], [inMonth.length, inMonth.length, inMonth.length, 0]);
+ assert.strictEqual((await h.db.prepare('SELECT COUNT(*)::int AS n FROM events_archive').get()).n, 0);
+ const lines = zlib.gunzipSync(await storage.get(objectKey(month))).toString('utf8').trim().split('\n').map(l => JSON.parse(l));
+ assert.strictEqual(lines.length, inMonth.length);
+ assert.strictEqual(JSON.parse(await storage.get(manifestKey(month))).count, inMonth.length);
+ assert.strictEqual((await request(h.base, 'GET', `/api/v1/events/${inMonth[0].id}`, { token: reader })).status, 404, 'the archive tier is offline');
+ const gapped = await pull(`topic=*&after_seq=0`);
+ assert.ok(gapped.body.gap, 'a cursor into what went to the archive tier is a gap');
+
+ // A rerun is idempotent: the object keeps its rows, nothing is deleted twice.
+ const rerun = await exportMonth({ db: h.db, storage, month, now: exportAt, olderThanDays: 90 });
+ assert.deepStrictEqual([rerun.exported, rerun.count, rerun.deleted], [0, inMonth.length, 0]);
+
+ const back = await restoreMonth({ db: h.db, storage, month, now: exportAt, holdDays: 30 });
+ assert.deepStrictEqual([back.count, back.inserted, back.skipped], [inMonth.length, inMonth.length, 0]);
+ assert.deepStrictEqual((await restoreMonth({ db: h.db, storage, month, now: exportAt })).inserted, 0, 'idempotent on event_id');
+ const after = await Promise.all(inMonth.map(r => h.store.getEvent(r.id)));
+ assert.deepStrictEqual(after, before, 'every column, payload and actor text byte for byte');
+ const restored = await h.db.prepare('SELECT * FROM events_archive ORDER BY seq').all();
+ for (let i = 0; i < restored.length; i++) {
+ assert.strictEqual(zlib.gunzipSync(restored[i].body).toString('utf8'), zlib.gunzipSync(inMonth[i].body).toString('utf8'));
+ assert.deepStrictEqual({ ...restored[i], body: null, hold_until: null }, { ...inMonth[i], body: null, hold_until: null });
+ }
+ // Held: a replay prune leaves restored rows alone until hold_until, whatever their age.
+ const held = async () => (await h.db.prepare('SELECT COUNT(*)::int AS n FROM events_archive WHERE id = ANY(?)').get(inMonth.map(x => x.id))).n;
+ await h.store.prune({ ...tiered(0), now: exportAt });
+ assert.strictEqual(await held(), inMonth.length);
+ await h.store.prune({ ...tiered(0), now: exportAt + 31 * DAY });
+ assert.strictEqual(await held(), 0);
+
+ // A tampered object is refused before anything is restored.
+ const gz = await storage.get(objectKey(month));
+ gz[gz.length - 5] ^= 0xff;
+ await storage.put(objectKey(month), gz);
+ await assert.rejects(restoreMonth({ db: h.db, storage, month, now: exportAt }), /sha256/);
+ await assert.rejects(exportMonth({ db: h.db, storage, month: '2999-01', now: exportAt }), /not ended/);
+});
+
+t('replay pruning answers `gap` (the existing shape), pull and SSE', async () => {
+ const a = await publish(live, envelope('live', { event_type: 'live.tier.gone', visibility: 'public' }));
+ await publish(live, envelope('live', { event_type: 'live.tier.gone', visibility: 'public' }));
+ await h.store.prune(tiered(31));
+ const r0 = await pull(`topic=live.tier.gone&after=${a.cursor}`);
+ assert.strictEqual(r0.body.gap, undefined, 'still in replay: no gap');
+ await h.store.prune(tiered(366));
+ const kept = await publish(live, envelope('live', { event_type: 'live.tier.gone' }));
+ const r = await pull(`topic=live.tier.gone&after=${a.cursor}`);
+ assert.deepStrictEqual(r.body.gap, { from_seq: a.seq + 1, to_seq: kept.seq - 1 });
+ assert.deepStrictEqual(r.body.events.map(e => e.seq), [kept.seq]);
+ const s = await sse(h.base, '/realtime/stream?topics=live.tier.gone', { headers: { Authorization: `Bearer ${serviceToken('live', ['events.event.read'])}`, 'Last-Event-ID': a.cursor } });
+ open.push(s);
+ await s.waitFor(x => x.events().length === 1);
+ assert.deepStrictEqual(s.gaps().map(g => [g.reason, g.from_seq, g.to_seq]), [['retention', a.seq + 1, kept.seq - 1]]);
+ s.close();
+});
+
+t('EVENTS_REPLAY_RETENTION_DAYS=0 behaves exactly as before the tiers', async () => {
+ assert.strictEqual(load({}).replayRetentionDays, 365);
+ assert.strictEqual(load({ EVENTS_REPLAY_RETENTION_DAYS: '0' }).replayRetentionDays, 0);
+ const x = await boot({ env: { EVENTS_REPLAY_RETENTION_DAYS: '0' } });
+ try {
+ const pub = async () => (await request(x.base, 'POST', '/api/v1/events', { token: live, body: envelope('live', { event_type: 'live.tier.zero' }) })).body;
+ const old = await pub();
+ // A leftover replay row (the tier was on before) is emptied by the next prune: no rule would prune it otherwise.
+ await x.store.prune({ ...tiered(31), replayRetentionDays: 365 });
+ assert.strictEqual((await x.db.prepare('SELECT COUNT(*)::int AS n FROM events_archive').get()).n, 1);
+ const r = await x.store.prune({ retentionDays: x.config.retentionDays, replayRetentionDays: x.config.replayRetentionDays, now: later(31) });
+ assert.deepStrictEqual([r.archived, r.replay], [0, 1]);
+ const old2 = await pub();
+ const r2 = await x.store.prune({ retentionDays: 30, replayRetentionDays: 0, now: later(31) });
+ assert.deepStrictEqual([r2.events, r2.archived], [1, 0], 'deleted, not moved');
+ assert.strictEqual((await x.db.prepare('SELECT COUNT(*)::int AS n FROM events_archive').get()).n, 0);
+ const kept = await pub();
+ const p = await request(x.base, 'GET', `/api/v1/events?topic=live.tier.zero&after=${old.cursor}`, { token: reader });
+ assert.deepStrictEqual(p.body.gap, { from_seq: old.seq + 1, to_seq: kept.seq - 1 });
+ assert.strictEqual((await request(x.base, 'GET', `/api/v1/events/${old2.event_id}`, { token: reader })).status, 404);
+ const lim = limitsOf(x.config).limits.find(l => l.id === 'replay_retention_days');
+ assert.deepStrictEqual([lim.production, lim.sandbox], [0, 0]);
+ } finally {
+ await x.stop();
+ }
+ const lim = limitsOf(h.config).limits.find(l => l.id === 'replay_retention_days');
+ assert.deepStrictEqual([lim.production, lim.sandbox, lim.unit], [365, 0, 'days']);
+});
+
+t('archive storage: SigV4 matches AWS\'s published vectors; the s3 adapter round-trips through a signing stub', async () => {
+ const empty = 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855';
+ assert.match(signV4({ method: 'GET', path: '/', headers: { host: 'example.amazonaws.com', 'x-amz-date': '20150830T123600Z' }, payloadHash: empty,
+ region: 'us-east-1', service: 'service', accessKeyId: 'AKIDEXAMPLE', secretAccessKey: 'wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY', amzDate: '20150830T123600Z' }),
+ /Signature=5fa00fa31553b73ebf1942676e86291e8372ff2a2260956d9b8aae1d763fbf31$/);
+ assert.match(signV4({ method: 'GET', path: '/test.txt', headers: { host: 'examplebucket.s3.amazonaws.com', range: 'bytes=0-9', 'x-amz-content-sha256': empty, 'x-amz-date': '20130524T000000Z' },
+ payloadHash: empty, region: 'us-east-1', service: 's3', accessKeyId: 'AKIAIOSFODNN7EXAMPLE', secretAccessKey: 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY', amzDate: '20130524T000000Z' }),
+ /SignedHeaders=host;range;x-amz-content-sha256;x-amz-date, Signature=f0e8bdb87c964420e857bd35b5d6ed310bd44f0170aba48dd91039c6036bdb41$/);
+
+ // A stub S3: checks each request's signature with the same secret and keeps objects in memory.
+ const objects = new Map();
+ const creds = { accessKeyId: 'AKIDTEST', secretAccessKey: 'secret-test' };
+ const server = http.createServer((req, res) => {
+ const chunks = [];
+ req.on('data', c => chunks.push(c));
+ req.on('end', () => {
+ const body = Buffer.concat(chunks);
+ const headers = { host: req.headers.host, 'x-amz-content-sha256': req.headers['x-amz-content-sha256'], 'x-amz-date': req.headers['x-amz-date'] };
+ const want = signV4({ method: req.method, path: req.url, headers, payloadHash: require('crypto').createHash('sha256').update(body).digest('hex'),
+ region: 'eu-west-3', service: 's3', ...creds, amzDate: req.headers['x-amz-date'] });
+ if (req.headers.authorization !== want) { res.writeHead(403); return res.end(); }
+ if (req.method === 'PUT') { objects.set(req.url, body); res.writeHead(200); return res.end(); }
+ if (!objects.has(req.url)) { res.writeHead(404); return res.end(); }
+ res.writeHead(200); return res.end(objects.get(req.url));
+ });
+ });
+ await new Promise(r => server.listen(0, '127.0.0.1', r));
+ try {
+ const env = { EVENTS_ARCHIVE_S3_BUCKET: 'ov-events', EVENTS_ARCHIVE_S3_ENDPOINT: `http://127.0.0.1:${server.address().port}`, EVENTS_ARCHIVE_S3_REGION: 'eu-west-3',
+ EVENTS_ARCHIVE_S3_ACCESS_KEY_ID: creds.accessKeyId, EVENTS_ARCHIVE_S3_SECRET_ACCESS_KEY: creds.secretAccessKey };
+ const s3 = fromEnv(env);
+ assert.match(s3.name, /^s3:/);
+ await s3.put('2026/events-2026-01.json', Buffer.from('{"a":1}'));
+ assert.deepStrictEqual([...objects.keys()], ['/ov-events/events/2026/events-2026-01.json']);
+ assert.strictEqual((await s3.get('2026/events-2026-01.json')).toString(), '{"a":1}');
+ assert.strictEqual(await s3.get('2026/missing.json'), null);
+ await assert.rejects(fromEnv({ ...env, EVENTS_ARCHIVE_S3_SECRET_ACCESS_KEY: 'wrong' }).put('x.json', Buffer.from('1')), /answered 403/);
+ assert.throws(() => s3Storage({ ...creds, endpoint: 'http://s3.example.com', bucket: 'b' }), /https/);
+ assert.match(fromEnv({ EVENTS_ARCHIVE_DIR: tmpDir() }).name, /^local:/);
+ await assert.rejects(s3.put('../escape', Buffer.from('1')), /bad object key/);
+ } finally {
+ server.close();
+ }
+});
+
+t('stop', async () => { for (const c of open) c.close(); await h.stop(); });
+
+t.run();