openvibers/openvibe.events Commits
Events: no per-event sequence number on the consumer wire (#60)
Changes
@@@ -76,10 +76,10 @@ project's hourly publishing and delivery rollup, for Network; never visible to a`POST /api/v1/events` takes one `events.event-envelope@1` envelope, or `{ "events": [ … ] }` (up to 100, stored atomically).- `source` must be the calling service (`svc:live` publishes `source: "live"`), and `event_type` must start with a prefix that source owns (`live.*`, `media.*`, `network.*`, `community.*`, `chat.*`, `openre.*`, `billing.*`, `tips.*`, `vip.*`, `ai.*`, `games.*`, `tools.*`, `codes.*`, `host.*`, `bot.*`, `watch.*`, `run.*`, `space.*`, and the publication products (`wiki.*`, `blog.*`, `news.*`, `reviews.*`, `deals.*`, `coupons.*`, `trade.*`, `sources.*`, `search.*`); `EVENTS_SOURCE_PREFIXES` overrides). A network-wide prefix may be published by any first-party service: `provider.*` carries storage provider health/capacity telemetry (`provider.health.degraded`, `provider.capacity.warning`), so whichever service observes a provider can report it; `EVENTS_SHARED_PREFIXES` overrides.--- Idempotent on `event_id`: a repeat answers `200 { event_id, seq, duplicate: true }` and is never stored twice (even after retention, for `EVENTS_RECEIPT_RETENTION_DAYS`).++- Idempotent on `event_id`: a repeat answers `200 { event_id, cursor, duplicate: true }` (`pruned: true` and no cursor when retention already removed the stored copy) and is never stored twice (even after retention, for `EVENTS_RECEIPT_RETENTION_DAYS`).- Missing `trace_id` is taken from the request's `traceparent`; `priority` defaults to `important`, `visibility` to `internal`.- Loop guard: an event whose trace already carries 8 hops (the cross-service chain depth, or the same source/type/subject repeating in the trace) is refused with `409 events.loop_detected`. The chain depth of a first-party publish counts first-party events only (developer-app events in the same trace never count toward it), so an app cannot poison a trace it has seen.--- Each accepted event gets a global `seq` and is committed with one delivery row per matching subscription before anything is sent.++- Each accepted event gets a position in the hot store (handed out only as an opaque cursor; there is no global order on the wire) and is committed with one delivery row per matching subscription before anything is sent.### Redaction@@@ -92,8 +92,8 @@ A producer takes back what it published (a deleted chat message) with a directiv```- `redacts.event_ids` (up to 1000 `evt_…`) names events directly; `subject_type` + `subject_ids` (up to 1000) names every stored event of the same source about those subjects.--- In the transaction that stores the directive, each target becomes a **tombstone**: `payload` is replaced by `{ "redacted": true, "redacted_at": "<ISO>", "redacted_by": "<the redacting event_id>" }` and `actor` by the producer itself (`{ "type": "service", "id": "<source>" }`, or the app). `event_id`, `seq`, `event_type`, `subject`, `timestamp` and `visibility` stay, so sequences have no holes. On PostgreSQL the replaced row version is a dead tuple until autovacuum reclaims it (the `events` table vacuums after 1% of its rows change, so within minutes); until then, and in backups taken before the redaction, the old bytes still exist on the host. No read path serves them.--- Every read path serves the tombstone from then on: pull, `GET /api/v1/events/:id`, SSE replay (anonymous, signed in or service), queued deliveries and DLQ replays. First-party consumers get the tombstone at the original seq and then the deletion event itself; they should check `payload.redacted`.++- In the transaction that stores the directive, each target becomes a **tombstone**: `payload` is replaced by `{ "redacted": true, "redacted_at": "<ISO>", "redacted_by": "<the redacting event_id>" }` and `actor` by the producer itself (`{ "type": "service", "id": "<source>" }`, or the app). `event_id`, its position, `event_type`, `subject`, `timestamp` and `visibility` stay, so a reader's walk has no holes. On PostgreSQL the replaced row version is a dead tuple until autovacuum reclaims it (the `events` table vacuums after 1% of its rows change, so within minutes); until then, and in backups taken before the redaction, the old bytes still exist on the host. No read path serves them.++- Every read path serves the tombstone from then on: pull, `GET /api/v1/events/:id`, SSE replay (anonymous, signed in or service), queued deliveries and DLQ replays. First-party consumers get the tombstone at its original position and then the deletion event itself; they should check `payload.redacted`.- Authority is the publish rule: only the owning source (for an app, its own project and environment) can redact. Naming another source's event by id refuses the whole publish with `403 events.redaction_not_allowed`; a subject match never reaches another source. A malformed directive is `422 events.invalid_redaction`. Events carrying a directive are never redacted themselves. No capability beyond `events.event.publish` is needed.- Redaction applies to what is stored when the directive arrives; it does not block later events about the same subject.- Events stored before a producer published deletions: the one-time `scripts/redact-backfill.js` (chat only) ran on 2026-09-24 and was retired with the move to PostgreSQL (it is in git history).@@@ -108,13 +108,13 @@ POST /api/v1/subscriptionsTopic patterns are dot-separated segments where `*` stands for one or more whole segments (`media.vod.*`, `*.created`, `media.*.ready`, `*`). Endpoints must be `http(s)` on `127.0.0.1` or `openvibe.<tld>`/its subdomains (`EVENTS_ENDPOINT_HOSTS`); redirects are never followed. A caller may supply its own `secret` (a string of 32..256 characters; the same bound on `POST /api/v1/subscriptions/:id/rotate-secret`), otherwise Events generates a `whsec_…` one; either way it is returned once, on creation, and the consumer needs it to verify the v2 signature. The consumer is the calling service, and re-creating a subscription it already holds for the same `topic_pattern` and `endpoint` answers `409 events.subscription_exists` with the existing `subscription_id`, so an ensure-at-boot script re-runs safely.--Each delivery is `POST <endpoint>` with body `{ "event": <envelope>, "seq": n }` and headers `X-OpenVibe-Event-Id`, `X-OpenVibe-Event-Type`, `X-OpenVibe-Seq`, `X-OpenVibe-Subscription-Id`, `X-OpenVibe-Delivery-Attempt`, `X-OpenVibe-Timestamp: <unix seconds>`, `X-OpenVibe-Signature-V2: t=<that timestamp>,v2=<HMAC-SHA256 of "<t>.<raw body>">` and `traceparent` (the event's trace). During secret rotation, the v2 header also carries a second `v2=` signature made with the previous secret. Any 2xx is delivered. Anything else is retried after 1 s, 5 s, 30 s, 2 min, 10 min, 1 h (then hourly) up to 8 attempts, after which the delivery is `dead`. Priority classes go first (`critical`, `important`, `low`, then seq); one delivery per subscription is in flight at a time and at most 20 overall. Delivery is at least once and not strictly ordered; consumers dedupe with an inbox and order with `subject.revision` (`createPgOrderedInbox` below does both).++Each delivery is `POST <endpoint>` with body `{ "event": <envelope> }` (no sequence number: dedupe on the event id) and headers `X-OpenVibe-Event-Id`, `X-OpenVibe-Event-Type`, `X-OpenVibe-Subscription-Id`, `X-OpenVibe-Delivery-Attempt`, `X-OpenVibe-Timestamp: <unix seconds>`, `X-OpenVibe-Signature-V2: t=<that timestamp>,v2=<HMAC-SHA256 of "<t>.<raw body>">` and `traceparent` (the event's trace). During secret rotation, the v2 header also carries a second `v2=` signature made with the previous secret. Any 2xx is delivered. Anything else is retried after 1 s, 5 s, 30 s, 2 min, 10 min, 1 h (then hourly) up to 8 attempts, after which the delivery is `dead`. Priority classes go first (`critical`, `important`, `low`, then seq); one delivery per subscription is in flight at a time and at most 20 overall. Delivery is at least once and not strictly ordered; consumers dedupe with an inbox and order with `subject.revision` (`createPgOrderedInbox` below does both).**Replay window (signature v2).** Events sends only the v2 signature. It signs the timestamp together with the raw body, and every attempt, retries included, is signed afresh with the time it is sent. Consumers check it with `verifyDeliveryV2(raw, headers, secret, { toleranceSec = 300, now })` (here) or openvibe-sdk ≥ 0.4.0 `parseDelivery(raw, headers, secret, { requireV2: true })` and reject anything more than 300 s from their clock either way. Reject a missing, wrong or stale v2 signature; do not fall back to v1.Operators: `GET /api/v1/deliveries?status=dead` is the dead-letter queue; `POST /api/v1/deliveries/replay { subscription_id, event_ids: [...] }` or `{ subscription_id, from_seq }` requeues retained events (that is also how a new subscription catches up on history).--Pull consumers: start with `GET /api/v1/events?topic=media.vod.*&limit=100`, then request `GET /api/v1/events?topic=media.vod.*&after=<next_cursor>&limit=100`, using the previous response's opaque `next_cursor` ([ADR-042](https://github.com/OpenVibers/OpenVibe.Contracts/blob/main/docs/adr/ADR-042-events-fabric.md) decision 7). The response carries `{ events: [{ seq, cursor, event }], next_cursor, latest_cursor }` (`latest_cursor` is the head as an opaque cursor, for a consumer that starts at "now" without history), plus `gap: { from_seq, to_seq }` when the position is older than both the hot store and the replay tier (below) or from another retention epoch. Use `next_cursor` even when no events match, since it advances past scanned events. A position is only ever an opaque cursor: `after_seq` is refused with 400 (never read as "from the start"), and the per-event `seq` is informational. `PUT /api/v1/checkpoints { topic, cursor, carrier? }` stores a consumer's cursor here if it has nowhere better (a cursor string; a number is refused); `GET /api/v1/checkpoints?topic=` and the PUT answer `{ consumer, topic, cursor, carrier, updated_at }`, with `cursor` as stored (null when none is).++Pull consumers: start with `GET /api/v1/events?topic=media.vod.*&limit=100`, then request `GET /api/v1/events?topic=media.vod.*&after=<next_cursor>&limit=100`, using the previous response's opaque `next_cursor` ([ADR-042](https://github.com/OpenVibers/OpenVibe.Contracts/blob/main/docs/adr/ADR-042-events-fabric.md) decision 7). The response carries `{ events: [{ cursor, event }], next_cursor, latest_cursor }` (`latest_cursor` is the head as an opaque cursor, for a consumer that starts at "now" without history), plus `gap: { from_seq, to_seq }` when the position is older than both the hot store and the replay tier (below) or from another retention epoch. Use `next_cursor` even when no events match, since it advances past scanned events. A position is only ever an opaque cursor: `after_seq` is refused with 400 (never read as "from the start"), and there is no per-event sequence number (dedupe on `event.event_id`). `PUT /api/v1/checkpoints { topic, cursor, carrier? }` stores a consumer's cursor here if it has nowhere better (a cursor string; a number is refused); `GET /api/v1/checkpoints?topic=` and the PUT answer `{ consumer, topic, cursor, carrier, updated_at }`, with `cursor` as stored (null when none is).### Retention: three tiers@@@ -126,7 +126,7 @@ Pull consumers: start with `GET /api/v1/events?topic=media.vod.*&limit=100`, the| replay | `events_archive`, same database: the filter columns plus a gzip'd body, no delivery rows | `EVENTS_REPLAY_RETENTION_DAYS` (365) from receipt; `0` = no replay tier | pulls, SSE resume and `GET /api/v1/events/:id`, with the same cursor || archive | one NDJSON object per month (gzip, with a manifest), outside the database | until an operator deletes it | nothing online; an operator restores a month into replay |--The hourly prune (`EVENTS_PRUNE_INTERVAL_MS`) moves each hot event past `EVENTS_RETENTION_DAYS` into `events_archive` in the transaction that deletes it: 500 rows at a time, locked `FOR UPDATE SKIP LOCKED`, inserted idempotently on `event_id`, so two Events processes pruning at once each move different rows and no event is moved twice. Its deliveries go with the hot row (operator replay to a subscription, `POST /api/v1/deliveries/replay`, reaches hot events only). Replay rows past `EVENTS_REPLAY_RETENTION_DAYS` are then deleted. A pull or SSE resume runs from a cursor in the replay tier into the hot store without a `gap`; `gap` (the same shape) still means the position is older than both tiers or from another epoch. Redaction tombstones replay rows exactly as hot ones, the sandbox and environment rules read them the same way, and a re-publish of an archived `event_id` is still `duplicate: true` with its original `seq` and `cursor`. Sandbox events never enter replay: they are deleted at `EVENTS_APP_SANDBOX_RETENTION_DAYS`, as before; publish receipts keep `EVENTS_RECEIPT_RETENTION_DAYS`.++The hourly prune (`EVENTS_PRUNE_INTERVAL_MS`) moves each hot event past `EVENTS_RETENTION_DAYS` into `events_archive` in the transaction that deletes it: 500 rows at a time, locked `FOR UPDATE SKIP LOCKED`, inserted idempotently on `event_id`, so two Events processes pruning at once each move different rows and no event is moved twice. Its deliveries go with the hot row (operator replay to a subscription, `POST /api/v1/deliveries/replay`, reaches hot events only). Replay rows past `EVENTS_REPLAY_RETENTION_DAYS` are then deleted. A pull or SSE resume runs from a cursor in the replay tier into the hot store without a `gap`; `gap` (the same shape) still means the position is older than both tiers or from another epoch. Redaction tombstones replay rows exactly as hot ones, the sandbox and environment rules read them the same way, and a re-publish of an archived `event_id` is still `duplicate: true` with its original `cursor`. Sandbox events never enter replay: they are deleted at `EVENTS_APP_SANDBOX_RETENTION_DAYS`, as before; publish receipts keep `EVENTS_RECEIPT_RETENTION_DAYS`.`EVENTS_REPLAY_RETENTION_DAYS=0` keeps the old behaviour exactly: events past `EVENTS_RETENTION_DAYS` are deleted, and the next prune also empties `events_archive`, because no rule would prune the rows left there otherwise. `GET /limits.json` publishes the window as `replay_retention_days`.@@@ -273,7 +273,7 @@ async function connect() {url.searchParams.set('ticket', ticket);if (lastEventId != null) url.searchParams.set('last_event_id', lastEventId);const stream = new EventSource(url);-- stream.onmessage = (m) => { lastEventId = m.lastEventId; const { seq, event } = JSON.parse(m.data); };++ stream.onmessage = (m) => { lastEventId = m.lastEventId; const { event } = JSON.parse(m.data); };stream.addEventListener('gap', (m) => { /* events were missed: refetch state */ });stream.onerror = () => { stream.close(); setTimeout(connect, 1000); };} catch (err) {@@@ -387,7 +387,7 @@ implemented service.Part of the [OpenVibe network](https://openvibe.network). Built in the open by [OpenVibers](https://github.com/OpenVibers).<!-- versions:start -->--- openvibe-contracts: v0.132.0++- openvibe-contracts: v0.136.1- openvibe-sdk: v0.42.0- openvibe-shared: v3.0.1<!-- versions:end -->@@@ -7,7 +7,7 @@"runtime": "node 22, express, PostgreSQL 18 (openvibe-sdk/db, async) + Valkey 9","port": 4300,"domain": "openvibe.events",-- "contracts": "openvibe-contracts v0.132.0",++ "contracts": "openvibe-contracts v0.136.1","sdk": "openvibe-sdk v0.42.0","shared": "openvibe-shared v3.0.1","capabilities": [@@@ -37,5 +37,5 @@"carrier fabric (ADR-042): the pg-v1 poll beside a valkey-v1 push carrier for QUEUE (streams) and TOPIC (SSE fan-out) an optional nats-v1 NATS Core TOPIC carrier (NATS_URL) and an optional nats-js-v1 JetStream STREAM carrier (NATS_URL, NATS_JETSTREAM; deploy/nats) and an optional cloud-v1 SQS-compatible QUEUE carrier (CLOUD_QUEUE_URL; SigV4, no dependency), with a placement planner, per-key pinning, delivery_placements history and a GET /api/v1/placement explain API; EVENTS_CARRIERS[_DISABLED] switches it","at_most_once delivery semantics (a failed attempt is final)"],-- "updated": "2026-10-10"++ "updated": "2026-10-11"}@@@ -154,8 +154,8 @@ class PublishError extends Error {/*** createPublisher({ eventsUrl, tokenClient, fetchImpl?, timeoutMs? })-- * publish(envelope | [envelopes], { traceparent }) -> { event_id, seq, cursor, duplicate } | { results: [...] }-- * (`cursor` is the opaque position beside `seq`; ADR-042 decision 7. Hand it back as `after=` or `Last-Event-ID`.)++ * publish(envelope | [envelopes], { traceparent }) -> { event_id, cursor, duplicate } | { results: [...] }++ * (`cursor` is the event's opaque position; ADR-042 decision 7. Hand it back as `after=` or `Last-Event-ID`.)* A 401 invalidates the cached token and retries once.*/function createPublisher({ eventsUrl, tokenClient, fetchImpl = globalThis.fetch, timeoutMs = 10000 }) {@@@ -12,7 +12,7 @@"dotenv": "^16.4.7","express": "^4.21.2","iovalkey": "^0.4.0",-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.132.0",++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.136.1","openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.42.0","openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v3.0.1","pg": "^8.23.0"@@@ -828,9 +828,9 @@}},"node_modules/openvibe-contracts": {-- "version": "0.132.0",-- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.132.0",-- "integrity": "sha512-08GuHutggWYtGcGrnBLilWco/K715Bl7ZAdfYX7kXeeHguWTIhbWZsSF3JYcH9iq6KKtRB7XCAbmPFKScleD9Q==",++ "version": "0.136.1",++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.136.1",++ "integrity": "sha512-G+xlUJEle2MlrEnVrKha5PfKLTh27cYomf2ioTZFwMSK5+Vy2BZlt5kkAOOmZ9zyJKDAL2CkEBlJy9rsUOY8UA==","license": "AGPL-3.0-only","dependencies": {"ajv": "^8.17.1",@@@ -24,7 +24,7 @@"dotenv": "^16.4.7","express": "^4.21.2","iovalkey": "^0.4.0",-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.132.0",++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.136.1","openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.42.0","openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v3.0.1","pg": "^8.23.0"@@@ -3,15 +3,16 @@* POST /api/v1/events (service token, audience openvibe.events, capability events.event.publish;* or a developer app token with events.app.publish)*-- * body: <envelope> -> 201 { event_id, seq, cursor, duplicate: false } | 200 on a repeat-- * body: { events: [...] } -> 201 { results: [{ event_id, seq, cursor, duplicate }] } (<= 100, atomic)++ * body: <envelope> -> 201 { event_id, cursor, duplicate: false } | 200 on a repeat++ * body: { events: [...] } -> 201 { results: [{ event_id, cursor, duplicate }] } (<= 100, atomic)*-- * `cursor` (ADR-042 decision 7) is the opaque position beside seq; `seq` and the global-order promise-- * stay for one release, then go.++ * `cursor` (ADR-042 decision 7) is the event's opaque position; there is no sequence number on the wire and no++ * global order. A repeat whose stored copy retention already removed answers { event_id, duplicate: true,++ * pruned: true } with no cursor.** Every envelope must validate against events.event-envelope@1; `source` must be the calling* service (svc:live -> 'live'); `event_type` must start with a prefix that source owns. A-- * re-published event_id is answered with the stored seq and never stored twice.++ * re-published event_id is answered with its stored position and never stored twice.** Developer apps (ADR-014, server/apps.js): `source` must be the app's `app-<lowercased ULID>`,* `event_type` must start with `app.<project_key>.`, and `actor` must be the app itself or the user@@@ -3,16 +3,17 @@* Pull consumers (service token with events.event.read):** GET /api/v1/events?topic=media.vod.*[,…][&after=<cursor>]&limit=100-- * -> { events: [{ seq, cursor, event }], next_cursor, latest_cursor, gap? }++ * -> { events: [{ cursor, event }], next_cursor, latest_cursor, gap? }* A position is only ever an opaque cursor (ADR-042 decision 7): `after=` takes one, and without it the* page starts at the oldest retained event. `after_seq` is refused (400), never read as "from the start".-- * `latest_cursor` is the head: a consumer that starts at "now" (skipping history) stores it. The per-event-- * `seq` is informational. A cursor from another retention epoch answers `gap` — never a silent restart.++ * `latest_cursor` is the head: a consumer that starts at "now" (skipping history) stores it. There is no++ * per-event sequence number (dedupe on event.event_id). A cursor from another retention epoch answers `gap`++ * — never a silent restart.* `gap` ({ from_seq, to_seq }) means* events after the position were already pruned by retention (or the epoch changed). The pull* spans the hot store and the replay tier (decision 8) with the one cursor: `gap` only outside both.* Keep next_cursor as the cursor; it moves past events that did not match.-- * GET /api/v1/events/:event_id -> { seq, cursor, event } (hot or replay)++ * GET /api/v1/events/:event_id -> { cursor, event } (hot or replay)* GET /api/v1/checkpoints?topic=… / PUT /api/v1/checkpoints { topic, cursor, carrier? }* a consumer's own stored cursor per topic pattern (consumer = calling principal): `cursor` is an* opaque cursor string (a number is refused), and both answers return it as stored, with its carrier@@@ -105,7 +106,7 @@ function readRouter({ store, auth, worker, limits }) {from = oldest - 1;}const { rows, cursor: scanned } = await store.scan(from, { patterns, limit, accept: acceptFor(req.principal, patterns) });-- out.events = rows.map(r => ({ seq: r.seq, cursor: cursor.encode(r.seq, epoch), event: rowToEnvelope(r) }));++ out.events = rows.map(r => ({ cursor: cursor.encode(r.seq, epoch), event: rowToEnvelope(r) }));out.next_cursor = cursor.encode(scanned, epoch);out.latest_cursor = cursor.encode(await store.lastSeq(), epoch);res.json(out);@@@ -116,7 +117,7 @@ function readRouter({ store, auth, worker, limits }) {const visible = row && (req.principal.kind === 'app' ? apps.visibleToApp(row, req.principal) : (row.env || 'production') === 'production');if (!visible) return http.sendProblem(res, 404, 'events.not_found', { detail: 'no such event (or pruned by retention)', ctx: req.ov });// A replay-tier row names the epoch its position belongs to; a hot row is in the current one.-- return res.json({ seq: row.seq, cursor: cursor.encode(row.seq, row.epoch ?? await store.epoch()), event: rowToEnvelope(row) });++ return res.json({ cursor: cursor.encode(row.seq, row.epoch ?? await store.epoch()), event: rowToEnvelope(row) });});const consumerOf = (req) => req.principal.service || req.principal.sub;@@@ -8,8 +8,8 @@* minted by Network for the signed-in person: how pages on every OpenVibe site open a stream without a* third-party cookie), a Bearer user or service token, the ov_token cookie, or nobody.*-- * Every message is `id: <cursor>` (ADR-042 decision 7; a bare seq is still accepted for one release) +-- * `data: {"seq":n,"event":<envelope>}`. Visibility decides who sees an event:++ * Every message is `id: <cursor>` (ADR-042 decision 7) + `data: {"event":<envelope>}`; there is no sequence++ * number (a client dedupes on event.event_id). Visibility decides who sees an event:* public anyone subscribed to a matching topic (signed-out visitors too, unless disabled)* subject only the user whose subject id is the event's actor.id or its subject.id (subject* type user); a guessed topic yields nothing for anyone else. A person's topic is an@@@ -91,7 +91,7 @@ function createRealtime({ store, auth, config, clock = { now: () => Date.now() }function sendEvent(conn, row) {conn.lastSeq = row.seq;-- write(conn, `id: ${cursor.encode(row.seq, conn.epoch)}\ndata: ${JSON.stringify({ seq: row.seq, event: rowToEnvelope(row) })}\n\n`);++ write(conn, `id: ${cursor.encode(row.seq, conn.epoch)}\ndata: ${JSON.stringify({ event: rowToEnvelope(row) })}\n\n`);}function sendGap(conn, gap) {@@@ -204,11 +204,11 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usageif (existing.source !== env.source || existing.event_type !== env.event_type) {throw new StoreError(409, 'events.id_conflict', `event ${env.event_id} already exists with different content`);}-- results.push({ event_id: env.event_id, seq: existing.seq, duplicate: true, cursor: cursor.encode(existing.seq, existing.epoch ?? epoch) });++ results.push({ event_id: env.event_id, duplicate: true, cursor: cursor.encode(existing.seq, existing.epoch ?? epoch) });continue;}if (await q.getReceipt.get(PUBLISH_RECEIPT, env.event_id)) {-- results.push({ event_id: env.event_id, seq: null, duplicate: true, pruned: true });++ results.push({ event_id: env.event_id, duplicate: true, pruned: true });continue;}// Loop guard: depth of the cross-service chain in this trace, and how often this exact@@@ -293,7 +293,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usageif (carrier) deliveries.push({ event_id: env.event_id, subscription_id: s.id, carrier, carrier_class: klass, ordering_key: key });}await revocationHook(row, now);-- results.push({ event_id: env.event_id, seq, duplicate: false, cursor: cursor.encode(seq, epoch) });++ results.push({ event_id: env.event_id, duplicate: false, cursor: cursor.encode(seq, epoch) });inserted.push(row);if (directive) {const redacted = await applyRedaction(row, directive, now);@@@ -2,9 +2,9 @@/*** Delivery worker. Pending deliveries are POSTed to their subscription endpoints as*-- * { "event": <envelope>, "seq": <n> }++ * { "event": <envelope> }*-- * with X-OpenVibe-Event-Id, X-OpenVibe-Signature (sha256=<HMAC of the raw body with the++ * (no sequence number: a receiver dedupes on event.event_id) with X-OpenVibe-Event-Id, X-OpenVibe-Signature (sha256=<HMAC of the raw body with the* subscription secret>), X-OpenVibe-Timestamp (unix seconds, new on every attempt) and* X-OpenVibe-Signature-V2 (t=<that timestamp>,v2=<HMAC of "<t>.<raw body>">; consumers reject it* outside ±300 s, so a captured delivery cannot be replayed later), traceparent (the event's trace). A 2xx is delivered; anything else@@@ -73,14 +73,13 @@ function createWorker({ store, config, clock = { now: () => Date.now() }, fetchIconst { maxAttempts, backoffMs } = policy(sub);let outcome;try {-- const body = JSON.stringify({ event: rowToEnvelope(row), seq: row.seq });++ const body = JSON.stringify({ event: rowToEnvelope(row) });const timestamp = Math.floor(clock.now() / 1000); // per attempt: a retry is signed afreshconst headers = {'Content-Type': 'application/json','User-Agent': 'OpenVibe.Events/0.1','X-OpenVibe-Event-Id': row.id,'X-OpenVibe-Event-Type': row.event_type,-- 'X-OpenVibe-Seq': String(row.seq),'X-OpenVibe-Subscription-Id': sub.id,'X-OpenVibe-Delivery-Attempt': String(attempt),'X-OpenVibe-Hops': String(row.hops),@@@ -43,14 +43,14 @@ t('a hot-pruned event is readable through replay: by id and by pull, unchanged',assert.deepStrictEqual([await count('events', e.event_id), await count('events_archive', e.event_id)], [0, 1], 'moved, not copied');const after = await request(h.base, 'GET', `/api/v1/events/${e.event_id}`, { token: reader });assert.strictEqual(after.status, 200, after.text);-- assert.deepStrictEqual(after.body, before.body, 'the same seq, cursor and envelope from the replay tier');++ assert.deepStrictEqual(after.body, before.body, 'the same cursor and envelope from the replay tier');const p = await pull('topic=live.tier.one');assert.deepStrictEqual(p.body.events.map(x => x.event), [before.body.event]);assert.strictEqual(p.body.gap, undefined);// A publish of the same id is a duplicate with its position, never a second row.const again = await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope('live', { event_id: e.event_id, event_type: 'live.tier.one' }) });assert.strictEqual(again.status, 200, again.text);-- assert.deepStrictEqual([again.body.duplicate, again.body.seq, again.body.cursor], [true, e.seq, e.cursor]);++ assert.deepStrictEqual([again.body.duplicate, again.body.cursor], [true, e.cursor]);});t('a cursor across the hot/replay boundary reports no false gap (pull and SSE)', async () => {@@@ -61,18 +61,18 @@ t('a cursor across the hot/replay boundary reports no false gap (pull and SSE)',const c = await publish(live, envelope('live', { event_type: 'live.tier.cross' }));let r = await pull(`topic=live.tier.cross&after=${a.cursor}&limit=1`);-- assert.deepStrictEqual([r.body.events.map(x => x.seq), r.body.gap], [[b.seq], undefined], 'replay');++ assert.deepStrictEqual([r.body.events.map(x => x.cursor), r.body.gap], [[b.cursor], undefined], 'replay');assert.strictEqual(r.body.events[0].cursor, b.cursor);r = await pull(`topic=live.tier.cross&after=${r.body.next_cursor}&limit=1`);-- assert.deepStrictEqual([r.body.events.map(x => x.seq), r.body.gap], [[c.seq], undefined], 'then hot, no gap between');-- r = await pull(`topic=*&after=${await cursorAt(h, a.seq - 1)}`);-- assert.deepStrictEqual(r.body.events.map(x => x.seq), [a.seq, b.seq, c.seq]);++ assert.deepStrictEqual([r.body.events.map(x => x.cursor), r.body.gap], [[c.cursor], undefined], 'then hot, no gap between');++ r = await pull(`topic=*&after=${await cursorAt(h, cursor.decode(a.cursor).seq - 1)}`);++ assert.deepStrictEqual(r.body.events.map(x => x.cursor), [a.cursor, b.cursor, c.cursor]);assert.strictEqual(r.body.gap, undefined);const s = await sse(h.base, '/realtime/stream?topics=live.tier.cross', { headers: { Authorization: `Bearer ${serviceToken('live', ['events.event.read'])}`, 'Last-Event-ID': a.cursor } });open.push(s);await s.waitFor(x => x.events().length === 2);-- assert.deepStrictEqual(s.events().map(x => x.seq), [b.seq, c.seq]);++ assert.deepStrictEqual(s.events().map(x => x.event.event_id), [b.event_id, c.event_id]);assert.deepStrictEqual(s.gaps(), []);assert.ok(s.messages.some(m => m.id === b.cursor), 'the SSE id of a replayed event is its cursor');s.close();@@@ -90,7 +90,7 @@ t('redaction reaches replay rows (by id and by subject); the owner rule holds thconst r = await request(h.base, 'GET', `/api/v1/events/${e.event_id}`, { token: reader });assert.strictEqual(r.body.event.payload.redacted, true, JSON.stringify(r.body));assert.deepStrictEqual(r.body.event.actor, { type: 'service', id: 'live' });-- assert.strictEqual(r.body.seq, e.seq);++ assert.strictEqual(r.body.cursor, e.cursor);}const p = await pull('topic=live.tier.secret');assert.ok(p.body.events.every(e => e.event.payload.redacted === true && !('secret' in e.event.payload)));@@@ -109,9 +109,9 @@ t('sandbox and environment rules apply to replay rows exactly as to hot ones', aassert.deepStrictEqual([await count('events_archive', pe.event_id), await count('events_archive', se.event_id), await count('events', se.event_id)], [1, 0, 0],'a production event moves; a sandbox event is deleted, never kept in replay');const topic = `topic=app.${apps.projectKey(prj)}.*`;-- assert.deepStrictEqual((await pull(topic, prod)).body.events.map(e => e.seq), [pe.seq], 'the app reads its own replay rows');++ assert.deepStrictEqual((await pull(topic, prod)).body.events.map(e => e.cursor), [pe.cursor], 'the app reads its own replay rows');assert.deepStrictEqual((await pull(topic, sand)).body.events, [], 'never across environments');-- assert.deepStrictEqual((await pull(topic)).body.events.map(e => e.seq), [pe.seq], 'first-party: production app events through app.*');++ assert.deepStrictEqual((await pull(topic)).body.events.map(e => e.cursor), [pe.cursor], 'first-party: production app events through app.*');assert.deepStrictEqual((await pull('topic=*')).body.events.filter(e => e.event.event_id === pe.event_id), [], '…and only through app.*');assert.strictEqual((await request(h.base, 'GET', `/api/v1/events/${pe.event_id}`, { token: sand })).status, 404);assert.strictEqual((await request(h.base, 'GET', `/api/v1/events/${pe.event_id}`, { token: prod })).status, 200);@@@ -189,12 +189,12 @@ t('replay pruning answers `gap` (the existing shape), pull and SSE', async () =>await h.store.prune(tiered(366));const kept = await publish(live, envelope('live', { event_type: 'live.tier.gone' }));const r = await pull(`topic=live.tier.gone&after=${a.cursor}`);-- assert.deepStrictEqual(r.body.gap, { from_seq: a.seq + 1, to_seq: kept.seq - 1 });-- assert.deepStrictEqual(r.body.events.map(e => e.seq), [kept.seq]);++ assert.deepStrictEqual(r.body.gap, { from_seq: cursor.decode(a.cursor).seq + 1, to_seq: cursor.decode(kept.cursor).seq - 1 });++ assert.deepStrictEqual(r.body.events.map(e => e.cursor), [kept.cursor]);const s = await sse(h.base, '/realtime/stream?topics=live.tier.gone', { headers: { Authorization: `Bearer ${serviceToken('live', ['events.event.read'])}`, 'Last-Event-ID': a.cursor } });open.push(s);await s.waitFor(x => x.events().length === 1);-- assert.deepStrictEqual(s.gaps().map(g => [g.reason, g.from_seq, g.to_seq]), [['retention', a.seq + 1, kept.seq - 1]]);++ assert.deepStrictEqual(s.gaps().map(g => [g.reason, g.from_seq, g.to_seq]), [['retention', cursor.decode(a.cursor).seq + 1, cursor.decode(kept.cursor).seq - 1]]);s.close();});@@@ -216,7 +216,7 @@ t('EVENTS_REPLAY_RETENTION_DAYS=0 behaves exactly as before the tiers', async ()assert.strictEqual((await x.db.prepare('SELECT COUNT(*)::int AS n FROM events_archive').get()).n, 0);const kept = await pub();const p = await request(x.base, 'GET', `/api/v1/events?topic=live.tier.zero&after=${old.cursor}`, { token: reader });-- assert.deepStrictEqual(p.body.gap, { from_seq: old.seq + 1, to_seq: kept.seq - 1 });++ assert.deepStrictEqual(p.body.gap, { from_seq: cursor.decode(old.cursor).seq + 1, to_seq: cursor.decode(kept.cursor).seq - 1 });assert.strictEqual((await request(x.base, 'GET', `/api/v1/events/${old2.event_id}`, { token: reader })).status, 404);const lim = limitsOf(x.config).limits.find(l => l.id === 'replay_retention_days');assert.deepStrictEqual([lim.production, lim.sandbox], [0, 0]);@@@ -1,9 +1,8 @@'use strict';/*** ADR-042 decision 7: consumer positions are cursors only. A cursor is `c1.<epoch>.<base64url(seq)>`;-- * event answers retain seq and carry a cursor, pull accepts `after=<cursor>`, SSE resumes from-- * cursor IDs, and checkpoints store and return opaque cursor strings. Unchanged publish and single-- * event answers are validated against the pinned openvibe-contracts.++ * event answers carry a cursor without seq, pull accepts `after=<cursor>`, SSE resumes from++ * cursor IDs, and checkpoints store and return opaque cursor strings.*/const assert = require('assert');const { validate } = require('openvibe-contracts');@@@ -43,22 +42,23 @@ t('publish and read answers carry cursors', async () => {const one = await publish();const pv = validate('events.publish-result@1', one);assert.ok(pv.valid, JSON.stringify(pv.errors));-- assert.deepStrictEqual(cursor.decode(one.cursor), { seq: one.seq, epoch });++ assert.deepStrictEqual(cursor.decode(one.cursor), { seq: (await h.store.getEvent(one.event_id)).seq, epoch });const batch = await request(h.base, 'POST', '/api/v1/events', { token: live, body: { events: [envelope('live', { event_type: 'live.vod.ready' }), envelope('live', { event_type: 'live.vod.ready' })] } });assert.ok(validate('events.publish-result@1', batch.body).valid);-- for (const r of batch.body.results) assert.deepStrictEqual(cursor.decode(r.cursor), { seq: r.seq, epoch });++ for (const r of batch.body.results) assert.deepStrictEqual(cursor.decode(r.cursor), { seq: (await h.store.getEvent(r.event_id)).seq, epoch });const page = await request(h.base, 'GET', '/api/v1/events?topic=live.vod.*', { token: reader });assert.ok(typeof page.body.next_cursor === 'string' && typeof page.body.latest_cursor === 'string');assert.ok(page.body.events.length >= 3);-- for (const e of page.body.events) assert.deepStrictEqual(cursor.decode(e.cursor), { seq: e.seq, epoch });-- assert.deepStrictEqual(cursor.decode(page.body.next_cursor), { seq: page.body.events.at(-1).seq, epoch });++ for (const e of page.body.events) assert.deepStrictEqual(cursor.decode(e.cursor), { seq: (await h.store.getEvent(e.event.event_id)).seq, epoch });++ assert.strictEqual(page.body.next_cursor, page.body.events.at(-1).cursor);assert.ok(!('next_after_seq' in page.body) && !('latest_seq' in page.body));const single = await request(h.base, 'GET', `/api/v1/events/${one.event_id}`, { token: reader });assert.ok(validate('events.read-result@1', single.body).valid);-- assert.deepStrictEqual(cursor.decode(single.body.cursor), { seq: single.body.seq, epoch });++ assert.strictEqual(single.body.cursor, one.cursor);++ assert.ok(!Object.hasOwn(single.body, 'seq'));});t('pull accepts after=<cursor>; next_cursor round-trips a page and after_seq is refused', async () => {@@@ -68,8 +68,8 @@ t('pull accepts after=<cursor>; next_cursor round-trips a page and after_seq isassert.strictEqual(r.body.next_cursor, first.cursor);r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after=${encodeURIComponent(r.body.next_cursor)}`, { token: reader });assert.strictEqual(r.status, 200, r.text);-- assert.ok(r.body.events.length >= 1 && r.body.events.every(e => e.seq > first.seq));-- for (const value of ['0', String(first.seq)]) {++ assert.ok(r.body.events.length >= 1 && r.body.events.every(e => cursor.decode(e.cursor).seq > cursor.decode(first.cursor).seq));++ for (const value of ['0', String(cursor.decode(first.cursor).seq)]) {r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after_seq=${value}`, { token: reader });assert.strictEqual(r.status, 400);assert.strictEqual(r.body.code, 'events.bad_request');@@@ -104,22 +104,24 @@ t('SSE id is the cursor; Last-Event-ID resumes from a cursor, and a bare seq staconst s1 = await publish({ visibility: 'public', event_type: 'live.stream.a' });await c1.waitFor(x => x.events().length === 1);const msg = c1.messages.find(m => !m.event);-- assert.deepStrictEqual(cursor.decode(msg.id), { seq: s1.seq, epoch });++ assert.strictEqual(msg.id, s1.cursor);++ assert.strictEqual(cursor.decode(msg.id).epoch, epoch);++ assert.ok(!Object.hasOwn(JSON.parse(msg.data), 'seq'));c1.close();const s2 = await publish({ visibility: 'public', event_type: 'live.stream.b' });const byCursor = await sse(h.base, `/realtime/stream?topics=live.stream.*&last_event_id=${encodeURIComponent(msg.id)}`, { headers: { Authorization: `Bearer ${readerSvc}` } });await byCursor.waitFor(x => x.events().length === 1);-- assert.deepStrictEqual(byCursor.events().map(e => e.seq), [s2.seq]);++ assert.deepStrictEqual(byCursor.events().map(e => e.seq), [cursor.decode(s2.cursor).seq]);assert.deepStrictEqual(byCursor.gaps(), []);byCursor.close();-- const bySeq = await sse(h.base, `/realtime/stream?topics=live.stream.*&last_event_id=${s1.seq}`, { headers: { Authorization: `Bearer ${readerSvc}` } });++ const bySeq = await sse(h.base, `/realtime/stream?topics=live.stream.*&last_event_id=${cursor.decode(s1.cursor).seq}`, { headers: { Authorization: `Bearer ${readerSvc}` } });assert.deepStrictEqual(bySeq.events(), [], 'a bare number does not replay missed events');assert.deepStrictEqual(bySeq.gaps(), []);const s3 = await publish({ visibility: 'public', event_type: 'live.stream.c' });await bySeq.waitFor(x => x.events().length === 1);-- assert.deepStrictEqual(bySeq.events().map(e => e.seq), [s3.seq]);++ assert.deepStrictEqual(bySeq.events().map(e => e.seq), [cursor.decode(s3.cursor).seq]);bySeq.close();});@@@ -134,7 +136,7 @@ t('SSE reports a gap when the cursor belongs to another epoch', async () => {await again.waitFor(x => x.gaps().length >= 1);assert.strictEqual(again.gaps()[0].reason, 'epoch');// The retained matching event is replayed after the gap.-- await again.waitFor(x => x.events().some(e => e.seq === s.seq));++ await again.waitFor(x => x.events().some(e => e.seq === cursor.decode(s.cursor).seq));again.close();});@@@ -2,6 +2,7 @@const assert = require('assert');const crypto = require('crypto');const { verifyDelivery, verifyDeliveryV2, signV2, sign } = require('../lib/client');++const cursor = require('../server/cursor');const { boot, request, serviceToken, envelope, subscriber, suite, sleep } = require('./helpers');const t = suite('delivery');@@@ -21,7 +22,7 @@ async function publish(env) {t('signV2 / verifyDeliveryV2: HMAC of "<t>.<raw body>", ±300 s, constant-time', async () => {const secret = `whsec_${'cd'.repeat(32)}`;-- const raw = Buffer.from(JSON.stringify({ event: { event_id: 'evt_1' }, seq: 3 }));++ const raw = Buffer.from(JSON.stringify({ event: { event_id: 'evt_1' } }));const now = 1790000000000;const t = Math.floor(now / 1000);const v2 = signV2(raw, secret, t);@@@ -36,7 +37,7 @@ t('signV2 / verifyDeliveryV2: HMAC of "<t>.<raw body>", ±300 s, constant-time',assert.ok(!verifyDeliveryV2(raw, headers, secret, { now: now - 301000 }), 'too far in the future');assert.ok(verifyDeliveryV2(raw, headers, secret, { now: now + 3600000, toleranceSec: 3601 }));assert.ok(!verifyDeliveryV2(raw, headers, 'whsec_other', { now }));-- assert.ok(!verifyDeliveryV2(Buffer.from(raw.toString().replace('3', '4')), headers, secret, { now }), 'body changed');++ assert.ok(!verifyDeliveryV2(Buffer.from(raw.toString().replace('evt_1', 'evt_2')), headers, secret, { now }), 'body changed');assert.ok(!verifyDeliveryV2(raw, { ...headers, 'x-openvibe-signature-v2': v2.replace(`t=${t}`, `t=${t + 1}`) }, secret, { now }), 'timestamp changed');assert.ok(!verifyDeliveryV2(raw, { ...headers, 'x-openvibe-timestamp': String(t + 1) }, secret, { now }), 'X-OpenVibe-Timestamp must match t');assert.ok(verifyDeliveryV2(raw, { 'x-openvibe-signature-v2': `${v2},v2=${'0'.repeat(64)}` }, secret, { now }), 'one of several v2 values');@@@ -136,7 +137,7 @@ t('success: signed POST (v2 only), verified by verifyDeliveryV2; trace propagateconst stub = await subscriber();const sub = (await subscribe(media, { topic_pattern: 'live.stream.*', endpoint: stub.url })).body;const env = envelope('live', { trace_id: '4bf92f3577b34da6a3ce929d0e0e4736' });-- const { seq } = await publish(env);++ await publish(env);assert.strictEqual((await h.store.getDelivery(env.event_id, sub.id)).status, 'pending', 'persisted before delivery');await h.worker.drain();assert.strictEqual(stub.calls.length, 1);@@@ -154,7 +155,8 @@ t('success: signed POST (v2 only), verified by verifyDeliveryV2; trace propagateassert.strictEqual(call.headers['x-openvibe-event-id'], env.event_id);assert.match(call.headers.traceparent, /^00-4bf92f3577b34da6a3ce929d0e0e4736-[0-9a-f]{16}-01$/);assert.deepStrictEqual(call.body.event.payload, env.payload);-- assert.strictEqual(call.body.seq, seq);++ assert.strictEqual(call.body.seq, undefined);++ assert.strictEqual(call.headers['x-openvibe-seq'], undefined);const d = await h.store.getDelivery(env.event_id, sub.id);assert.strictEqual(d.status, 'delivered');assert.strictEqual(d.attempt, 1);@@@ -216,14 +218,14 @@ t('replay from_seq re-sends retained matching events (new subscription catches uconst e1 = envelope('live', { event_type: 'live.history.a' });const e2 = envelope('live', { event_type: 'live.history.b' });const e3 = envelope('live', { event_type: 'live.other.c' });-- const { seq } = await publish(e1);++ const { cursor: firstCursor } = await publish(e1);await publish(e2);await publish(e3);const stub = await subscriber();const sub = (await subscribe(media, { topic_pattern: 'live.history.*', endpoint: stub.url })).body;await h.worker.drain();assert.strictEqual(stub.calls.length, 0, 'subscriptions only get events published after they exist');-- const rp = await request(h.base, 'POST', '/api/v1/deliveries/replay', { token: admin, body: { subscription_id: sub.id, from_seq: seq } });++ const rp = await request(h.base, 'POST', '/api/v1/deliveries/replay', { token: admin, body: { subscription_id: sub.id, from_seq: cursor.decode(firstCursor).seq } });assert.strictEqual(rp.body.queued, 2);await h.worker.drain();assert.deepStrictEqual(stub.calls.map(c => c.body.event.event_id), [e1.event_id, e2.event_id]);@@@ -8,6 +8,7 @@ const assert = require('assert');const crypto = require('crypto');const { ids, serviceAuth } = require('openvibe-contracts');const apps = require('../server/apps');++const cursor = require('../server/cursor');const { ISSUER, privateKey, boot, request, appToken, envelope, suite } = require('./helpers');const t = suite('export-tokens');@@@ -38,7 +39,7 @@ t('boot and publish as the project\'s app, in both environments (and another probody: envelope('x', { source: apps.appSource(appId), event_type: `app.${apps.projectKey(project)}.order.n${n}`, actor: { type: 'app', id: appId }, visibility: 'internal' }),});assert.strictEqual(r.status, 201, r.text);-- return r.body.seq;++ return cursor.decode(r.body.cursor).seq;};for (let i = 0; i < 5; i++) seqs.production.push(await publish(P, 'production', i));for (let i = 0; i < 2; i++) seqs.sandbox.push(await publish(P, 'sandbox', i));@@@ -51,14 +52,14 @@ t('an export token pulls the project\'s events of its environment, page by pagefor (let page = 0; page < 10; page++) {const r = await pull(exportToken('production'), after, 2);assert.strictEqual(r.status, 200, r.text);-- got.push(...r.body.events.map(e => e.seq));++ got.push(...r.body.events.map(e => cursor.decode(e.cursor).seq));if (r.body.next_cursor === r.body.latest_cursor || r.body.next_cursor === after) break;after = r.body.next_cursor;}assert.deepStrictEqual(got, seqs.production);const s = await pull(exportToken('sandbox'));assert.strictEqual(s.status, 200, s.text);-- assert.deepStrictEqual(s.body.events.map(e => e.seq), seqs.sandbox, 'the sandbox token sees the sandbox only');++ assert.deepStrictEqual(s.body.events.map(e => cursor.decode(e.cursor).seq), seqs.sandbox, 'the sandbox token sees the sandbox only');});t('it reads nothing else and writes nothing', async () => {@@@ -220,7 +220,8 @@ t('valkey-v1 QUEUE: two instances, exactly once, per-key order, fast; a down carawait putPolicy(a.base, 'live.q.*', keyed('subject'));const KEYS = ['K0', 'K1', 'K2', 'K3', 'K4'];const byId = new Map();-- const firstSeq = new Map(KEYS.map((k) => [k, []]));++ const firstIds = new Map(KEYS.map((k) => [k, []]));++ const expectedIds = new Map(KEYS.map((k) => [k, []]));stub.calls.length = 0;const seen = new Map();// Publish 50 events over 5 keys, recording the publish time per event id.@@@ -229,6 +230,7 @@ t('valkey-v1 QUEUE: two instances, exactly once, per-key order, fast; a down carfor (const k of KEYS) {const env2 = envelope('live', { event_type: 'live.q.x', subject: { type: 'stream', id: k } });publishedAt.set(env2.event_id, Date.now());++ expectedIds.get(k).push(env2.event_id);await publish(a.base, env2);}}@@@ -236,13 +238,13 @@ t('valkey-v1 QUEUE: two instances, exactly once, per-key order, fast; a down carawait waitFor(() => {for (const c of stub.calls) {const id = c.body.event.event_id;-- if (!seen.has(id)) { seen.set(id, { at: Date.now(), seq: c.body.seq }); byId.set(id, (byId.get(id) || 0) + 1); const L = firstSeq.get(c.body.event.subject.id); if (L && !L.includes(c.body.seq)) L.push(c.body.seq); }++ if (!seen.has(id)) { seen.set(id, { at: Date.now() }); byId.set(id, (byId.get(id) || 0) + 1); const L = firstIds.get(c.body.event.subject.id); if (L && !L.includes(id)) L.push(id); }}return seen.size >= 50;}, 9000);assert.strictEqual(byId.size, 50, 'every event of every key was delivered');assert.ok([...byId.values()].every((n) => n === 1), 'each delivery was sent exactly once');-- for (const k of KEYS) { const L = firstSeq.get(k); assert.deepStrictEqual(L, [...L].sort((x, y) => x - y), `${k}: first delivery per seq is in order`); }++ for (const k of KEYS) assert.deepStrictEqual(firstIds.get(k), expectedIds.get(k), `${k}: first deliveries follow publish order`);const lats = [...publishedAt.entries()].map(([id, at]) => seen.get(id).at - at).sort((x, y) => x - y);const median = lats[Math.floor(lats.length / 2)];assert.ok(median < 5000, `median publish->send latency ${median}ms is below the 5 s poll interval`);@@@ -252,7 +254,7 @@ t('valkey-v1 QUEUE: two instances, exactly once, per-key order, fast; a down carconst before = seen.size;for (let i = 0; i < 5; i++) await publish(a.base, envelope('live', { event_type: 'live.q.x', subject: { type: 'stream', id: 'K9' } }));await waitFor(() => {-- for (const c of stub.calls) { const id = c.body.event.event_id; if (!seen.has(id)) { seen.set(id, { at: Date.now(), seq: c.body.seq }); byId.set(id, (byId.get(id) || 0) + 1); } }++ for (const c of stub.calls) { const id = c.body.event.event_id; if (!seen.has(id)) { seen.set(id, { at: Date.now() }); byId.set(id, (byId.get(id) || 0) + 1); } }return seen.size >= before + 5;}, 12000);assert.ok([...byId.values()].every((n) => n === 1), 'nothing was lost or doubled when the carrier went down');@@@ -148,7 +148,7 @@ async function subscriber(respond = () => 204) {};}--/** Open an SSE stream and parse it. */++/** Open an SSE stream and parse it; events() derives a test-only seq from each message id cursor. */function sse(base, pathAndQuery, { headers = {} } = {}) {return new Promise((resolve, reject) => {const messages = [];@@@ -161,7 +161,7 @@ function sse(base, pathAndQuery, { headers = {} } = {}) {headers: res.headers,messages,body: '',-- events: () => messages.filter(m => !m.event || m.event === 'message').map(m => JSON.parse(m.data)),++ events: () => messages.filter(m => !m.event || m.event === 'message').map(m => ({ ...JSON.parse(m.data), seq: require('../server/cursor').decode(m.id).seq })),gaps: () => messages.filter(m => m.event === 'gap').map(m => JSON.parse(m.data)),comments: [],waitFor(pred, ms = 2000) {@@@ -147,34 +147,39 @@ t('per-key order under retries: a successor waits for its head, another key goesawait h.store.setSubscriptionEnabled(sub.id, false);});--t('per-key order across two workers with a lease expiry, and never two distinct seqs of one key in flight', async () => {++t('per-key order across two workers with a lease expiry, and never two distinct events of one key in flight', async () => {const stub = await subscriber(() => 204);const sub = await subscribe('ord3', 'live.ord3.*', stub.url);await putPolicy('live.ord3.*', keyed('subject'));const keys = ['K0', 'K1', 'K2'];const byId = new Map(); // event_id -> deliveries-- const firstSeqs = new Map(keys.map((k) => [k, []])); // key -> seqs in first-delivery order-- const inflight = new Map(); // key -> Set(seq)-- let distinctPeak = 0; // the most distinct seqs of one key ever in flight at once++ const firstIds = new Map(keys.map((k) => [k, []])); // key -> event ids in first-delivery order++ const inflight = new Map(); // key -> Set(event_id)++ let distinctPeak = 0; // the most distinct events of one key ever in flight at onceconst fetchImpl = async (url, init) => {-- const { event, seq } = JSON.parse(init.body);++ const { event } = JSON.parse(init.body);const k = event.subject.id;byId.set(event.event_id, (byId.get(event.event_id) || 0) + 1);const set = inflight.get(k) || new Set();inflight.set(k, set);-- set.add(seq);++ set.add(event.event_id);distinctPeak = Math.max(distinctPeak, set.size);await sleep(2);-- set.delete(seq);-- const list = firstSeqs.get(k);-- if (!list.includes(seq)) list.push(seq);++ set.delete(event.event_id);++ const list = firstIds.get(k);++ if (!list.includes(event.event_id)) list.push(event.event_id);return new Response(null, { status: 204 });};-- // 30 events over 3 keys, interleaved so the global seq order is not the per-key order.++ // 30 events over 3 keys, interleaved across keys.++ const expectedIds = new Map(keys.map((k) => [k, []]));for (let i = 0; i < 10; i++) {-- for (const k of keys) await publish(envelope('live', { event_type: 'live.ord3.x', subject: { type: 'stream', id: k } }));++ for (const k of keys) {++ const env = envelope('live', { event_type: 'live.ord3.x', subject: { type: 'stream', id: k } });++ expectedIds.get(k).push(env.event_id);++ await publish(env);++ }}const a = createWorker({ store: h.store, config: h.config, clock: h.clock, fetchImpl, log: silent });@@@ -189,10 +194,10 @@ t('per-key order across two workers with a lease expiry, and never two distinctassert.strictEqual(byId.size, 30, 'every event of every key was delivered');assert.ok([...byId.values()].every((n) => n >= 1), 'every event delivered at least once');for (const k of keys) {-- const list = firstSeqs.get(k);-- assert.deepStrictEqual(list, [...list].sort((x, y) => x - y), `${k}: the first delivery of each seq is in seq order (${list})`);++ const list = firstIds.get(k);++ assert.deepStrictEqual(list, expectedIds.get(k), `${k}: first deliveries follow publish order`);}-- assert.strictEqual(distinctPeak, 1, 'never two distinct seqs of one key in flight at once');++ assert.strictEqual(distinctPeak, 1, 'never two distinct events of one key in flight at once');await deletePolicy('live.ord3.*');await stub.close();@@@ -205,17 +210,17 @@ t('max_inflight = 2: two keys run together, never two of one key, never three atawait putPolicy('live.ord4.*', keyed('subject'));await h.store.db.prepare('UPDATE subscriptions SET max_inflight = 2 WHERE id = ?').run(sub.id);-- const inflight = []; // { key, seq } currently being sent++ const inflight = []; // { key, event id } currently being sentlet totalPeak = 0;let keyPeak = 0;const fetchImpl = async (url, init) => {-- const { event, seq } = JSON.parse(init.body);++ const { event } = JSON.parse(init.body);const k = event.subject.id;-- inflight.push({ k, seq });++ inflight.push({ k, id: event.event_id });totalPeak = Math.max(totalPeak, inflight.length);keyPeak = Math.max(keyPeak, inflight.filter((x) => x.k === k).length);await sleep(4);-- inflight.splice(inflight.findIndex((x) => x.k === k && x.seq === seq), 1);++ inflight.splice(inflight.findIndex((x) => x.k === k && x.id === event.event_id), 1);return new Response(null, { status: 204 });};@@@ -97,7 +97,7 @@ t('outbox: enqueue inside a rolled-back transaction publishes nothing; committedr = await outbox.flush();assert.strictEqual(r.sent, 1);assert.strictEqual(await countIn(committed.event_id), 1, 'still exactly one event');-- assert.ok(Number((await db.one('SELECT seq FROM event_outbox WHERE event_id = $1', [committed.event_id])).seq) > 0);++ assert.strictEqual((await db.one('SELECT seq FROM event_outbox WHERE event_id = $1', [committed.event_id])).seq, null, 'Events hands out no sequence number (ADR-042): nothing is recorded');} finally { await db.close(); }});@@@ -28,17 +28,18 @@ t('a family grant (events.*) covers events.event.publish', async () => {assert.strictEqual(r.status, 201);});--t('valid envelope -> 201 with a seq; defaults filled', async () => {++t('valid envelope -> 201 with a cursor; defaults filled', async () => {const env = envelope('live');delete env.priority; delete env.visibility;const r = await request(h.base, 'POST', '/api/v1/events', { token: live, body: env, headers: { traceparent: '00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01' } });assert.strictEqual(r.status, 201, r.text);assert.strictEqual(r.body.event_id, env.event_id);-- assert.ok(Number.isInteger(r.body.seq) && r.body.seq > 0);++ assert.ok(typeof r.body.cursor === 'string');++ assert.ok(!Object.hasOwn(r.body, 'seq'));assert.strictEqual(r.body.duplicate, false);assert.ok(validate('events.publish-result@1', r.body).valid, JSON.stringify(validate('events.publish-result@1', r.body).errors));-- assert.deepStrictEqual(cursor.decode(r.body.cursor), { seq: r.body.seq, epoch: await h.store.epoch() });const row = await h.store.getEvent(env.event_id);++ assert.deepStrictEqual(cursor.decode(r.body.cursor), { seq: row.seq, epoch: await h.store.epoch() });assert.strictEqual(row.priority, 'important');assert.strictEqual(row.visibility, 'internal');assert.strictEqual(row.trace_id, '0af7651916cd43dd8448eb211c80319c', 'trace taken from traceparent');@@@ -153,14 +154,14 @@ t('app principals cannot publish as a service', async () => {assert.strictEqual(r.body.code, 'token.invalid_claims');});--t('idempotent repeat: same event_id -> 200, same seq, stored once', async () => {++t('idempotent repeat: same event_id -> 200, same cursor, stored once', async () => {const env = envelope();const a = await request(h.base, 'POST', '/api/v1/events', { token: live, body: env });const b = await request(h.base, 'POST', '/api/v1/events', { token: live, body: env });assert.strictEqual(a.status, 201);assert.strictEqual(b.status, 200);assert.strictEqual(b.body.duplicate, true);-- assert.strictEqual(b.body.seq, a.body.seq);++ assert.strictEqual(b.body.cursor, a.body.cursor);assert.strictEqual((await h.db.prepare('SELECT COUNT(*) AS n FROM events WHERE id = ?').get(env.event_id)).n, 1);const c = await request(h.base, 'POST', '/api/v1/events', { token: live, body: { ...env, event_type: 'live.stream.ended' } });assert.strictEqual(c.status, 409);@@@ -172,7 +173,10 @@ t('batch: atomic, <= 100, results per event', async () => {let r = await request(h.base, 'POST', '/api/v1/events', { token: live, body: { events } });assert.strictEqual(r.status, 201);assert.strictEqual(r.body.results.length, 3);-- assert.ok(r.body.results[0].seq < r.body.results[1].seq && r.body.results[1].seq < r.body.results[2].seq);++ assert.ok(r.body.results.every(item => typeof item.cursor === 'string' && !Object.hasOwn(item, 'seq')));++ assert.deepStrictEqual(r.body.results.map(item => item.event_id), events.map(event => event.event_id));++ const positions = r.body.results.map(item => cursor.decode(item.cursor).seq);++ assert.ok(positions[0] < positions[1] && positions[1] < positions[2]);const before = await h.store.lastSeq();r = await request(h.base, 'POST', '/api/v1/events', { token: live, body: { events: [envelope(), envelope('media')] } });@@@ -231,6 +235,7 @@ t('publish receipts outlive retention: a pruned id is still a duplicate', asyncassert.strictEqual(r.status, 200);assert.strictEqual(r.body.duplicate, true);assert.strictEqual(r.body.pruned, true);++ assert.ok(!Object.hasOwn(r.body, 'cursor') && !Object.hasOwn(r.body, 'seq'));});t('stop', async () => { await h.stop(); });@@@ -54,24 +54,27 @@ t('config: network-wide prefixes are shared, source namespaces stay owned', () =t('boot', async () => { h = await boot(); });t('pull: cursor, topic filter, internal events included for services', async () => {-- const s1 = (await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope('live', { event_type: 'live.vod.ready' }) })).body.seq;++ const c1 = (await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope('live', { event_type: 'live.vod.ready' }) })).body.cursor;await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope('live', { event_type: 'live.chat.sent' }) });-- const s3 = (await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope('live', { event_type: 'live.vod.deleted' }) })).body.seq;++ const c3 = (await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope('live', { event_type: 'live.vod.deleted' }) })).body.cursor;let r = await request(h.base, 'GET', '/api/v1/events?topic=live.vod.*&limit=1', { token: reader });assert.strictEqual(r.status, 200, r.text);assert.ok(validate('events.read-result@1', r.body).valid, JSON.stringify(validate('events.read-result@1', r.body).errors));-- for (const e of r.body.events) assert.deepStrictEqual(cursor.decode(e.cursor), { seq: e.seq, epoch: await h.store.epoch() });-- assert.strictEqual(r.body.next_cursor, await cursorAt(h, s1));++ for (const e of r.body.events) {++ assert.ok(!Object.hasOwn(e, 'seq'));++ assert.deepStrictEqual(cursor.decode(e.cursor).epoch, await h.store.epoch());++ }++ assert.strictEqual(r.body.next_cursor, c1);assert.ok(!('next_after_seq' in r.body) && !('latest_seq' in r.body));-- assert.deepStrictEqual(r.body.events.map(e => e.seq), [s1]);++ assert.deepStrictEqual(r.body.events.map(e => e.cursor), [c1]);assert.strictEqual(r.body.events[0].event.visibility, 'internal');r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after=${r.body.next_cursor}`, { token: reader });-- assert.deepStrictEqual(r.body.events.map(e => e.seq), [s3]);-- assert.strictEqual(r.body.next_cursor, await cursorAt(h, s3));++ assert.deepStrictEqual(r.body.events.map(e => e.cursor), [c3]);++ assert.strictEqual(r.body.next_cursor, c3);r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after=${r.body.next_cursor}`, { token: reader });assert.deepStrictEqual(r.body.events, []);-- assert.strictEqual(r.body.latest_cursor, await cursorAt(h, s3));++ assert.strictEqual(r.body.latest_cursor, c3);assert.match(r.body.latest_cursor, /^c1\./, 'the head as an opaque cursor');{// Starting at the head with latest_cursor reads nothing old, and the next event after it arrives.@@@ -82,9 +85,9 @@ t('pull: cursor, topic filter, internal events included for services', async ()assert.strictEqual(fromHead.body.next_cursor, head);}-- const id3 = (await h.db.prepare('SELECT id FROM events WHERE seq = ?').get(s3)).id;++ const id3 = (await h.db.prepare('SELECT id FROM events WHERE seq = ?').get(cursor.decode(c3).seq)).id;r = await request(h.base, 'GET', `/api/v1/events/${id3}`, { token: reader });-- assert.strictEqual(r.body.seq, s3);++ assert.strictEqual(r.body.cursor, c3);r = await request(h.base, 'GET', '/api/v1/events?topic=live.*', { token: live });assert.strictEqual(r.status, 403, 'events.event.read is required');r = await request(h.base, 'GET', '/api/v1/events?limit=5000', { token: reader });@@@ -94,10 +97,10 @@ t('pull: cursor, topic filter, internal events included for services', async ()t('pull: a cursor older than retention reports the gap', async () => {const before = await h.store.lastSeq();await h.store.prune({ retentionDays: 30, now: Date.now() + 31 * 86400000 });-- const s = (await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope() })).body.seq;++ const c = (await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope() })).body.cursor;const r = await request(h.base, 'GET', `/api/v1/events?after=${await cursorAt(h, 1)}`, { token: reader });assert.deepStrictEqual(r.body.gap, { from_seq: 2, to_seq: before });-- assert.deepStrictEqual(r.body.events.map(e => e.seq), [s]);++ assert.deepStrictEqual(r.body.events.map(e => e.cursor), [c]);});t('pull: after_seq is refused even at zero', async () => {@@@ -41,7 +41,7 @@ t('realtime example gets a new ticket and resumes from the last cursor', async (assert.strictEqual(streams[0].url.searchParams.get('topics'), 'network.notification.*');assert.strictEqual(streams[0].url.searchParams.has('last_event_id'), false);-- streams[0].onmessage({ lastEventId: 'opaque-cursor', data: '{"seq":1,"event":{}}' });++ streams[0].onmessage({ lastEventId: 'opaque-cursor', data: '{"event":{}}' });streams[0].onerror();assert.strictEqual(streams[0].closed, true);retries.shift()();@@@ -8,6 +8,7 @@const assert = require('assert');const crypto = require('crypto');const { ids } = require('openvibe-contracts');++const cursorCodec = require('../server/cursor');const { boot, request, serviceToken, userToken, realtimeTicket, envelope, sse, sleep, suite, ISSUER, cursorAt } = require('./helpers');const t = suite('realtime-tickets');@@@ -27,7 +28,7 @@ async function stream(q, headers) {async function publish(token, env) {const r = await request(h.base, 'POST', '/api/v1/events', { token, body: env });assert.strictEqual(r.status, 201, r.text);-- return r.body.seq;++ return cursorCodec.decode(r.body.cursor).seq;}/** network.notification.created for `who`, as Network's outbox writes it. */const notified = (who, unread = 1) => envelope('network', {@@@ -1,6 +1,7 @@'use strict';const assert = require('assert');const { ids } = require('openvibe-contracts');++const cursor = require('../server/cursor');const { boot, request, serviceToken, userToken, envelope, sse, sleep, suite, cursorAt } = require('./helpers');const t = suite('realtime');@@@ -19,7 +20,7 @@ async function stream(q, headers) {async function publish(token, env) {const r = await request(h.base, 'POST', '/api/v1/events', { token, body: env });assert.strictEqual(r.status, 201, r.text);-- return r.body.seq;++ return cursor.decode(r.body.cursor).seq;}const quiet = () => sleep(150);@@@ -50,6 +51,7 @@ t('visibility: public to everyone, subject only to that user, internal only to sassert.deepStrictEqual(seqs(a), [pub, toAliceAsSubject, byAliceAsActor]);assert.deepStrictEqual(seqs(b), [pub], 'bob sees nothing addressed to alice');assert.deepStrictEqual(seqs(svc), [pub, internal, toAliceAsSubject, byAliceAsActor]);++ assert.ok(!Object.hasOwn(JSON.parse(anon.messages.find(m => !m.event).data), 'seq'));const { encode } = require('../server/cursor');assert.strictEqual(a.messages.find(m => m.data.includes('hi alice')).id, encode(toAliceAsSubject, await h.store.epoch()), 'SSE id is the cursor');for (const c of [anon, a, b, svc]) c.close();@@@ -5,7 +5,7 @@* The leak this closes: a chat message deleted in OpenVibe.Chat stayed replayable here, text and* anon id included, for the whole retention, to anyone (public visibility, anonymous SSE replay).* Now Chat's chat.message.deleted carries payload.redacts, the stored chat.message.created becomes-- * a tombstone at the same seq on every read path, and browsers are replayed public events of the++ * a tombstone at the same stored position on every read path, and browsers are replayed public events of the* last few minutes only.*/const assert = require('assert');@@@ -13,6 +13,7 @@ const fs = require('fs');const path = require('path');const { ids } = require('openvibe-contracts');const apps = require('../server/apps');++const cursor = require('../server/cursor');const { parseDirective } = require('../server/redaction');const { boot, request, serviceToken, userToken, appToken, envelope, subscriber, sse, sleep, suite, cursorAt } = require('./helpers');@@@ -96,7 +97,7 @@ t('a first-party subscriber is queued the original before the deletion', async (t('before deletion the message is replayable (the leak as it was)', async () => {const env = created(101);msgEventId = env.event_id;-- msgSeq = (await publish(chat, env)).seq;++ msgSeq = cursor.decode((await publish(chat, env)).cursor).seq;const anon = await stream('topics=chat.message.*', { 'Last-Event-ID': await cursorAt(h, msgSeq - 1) });await anon.waitFor(c => c.events().length === 1);assert.strictEqual(anon.events()[0].event.payload.text, SECRET_TEXT);@@@ -107,7 +108,7 @@ t('chat.message.deleted turns the stored event into a tombstone at the same seq'const env = deleted([101]);delEventId = env.event_id;const r = await publish(chat, env);-- delSeq = r.seq;++ delSeq = cursor.decode(r.cursor).seq;assert.strictEqual(delSeq, msgSeq + 1);const row = await h.store.getEvent(msgEventId);assert.strictEqual(row.seq, msgSeq, 'seq unchanged');@@@ -145,15 +146,15 @@ t('signed-in SSE replay (cookie and Bearer) and service replay: the tombstone tot('pull (/api/v1/events), by id, and developer-app reads: the tombstone', async () => {let r = await request(h.base, 'GET', `/api/v1/events?topic=chat.message.*&after=${await cursorAt(h, msgSeq - 1)}`, { token: reader });assert.strictEqual(r.status, 200);-- assert.deepStrictEqual(r.body.events.map(e => e.seq), [msgSeq, delSeq], 'no hole in the sequence');++ assert.deepStrictEqual(r.body.events.map(e => cursor.decode(e.cursor).seq), [msgSeq, delSeq], 'no hole in the sequence');assertTombstone(r.body.events[0].event, delEventId);r = await request(h.base, 'GET', `/api/v1/events/${msgEventId}`, { token: reader });assertTombstone(r.body.event, delEventId);-- assert.strictEqual(r.body.seq, msgSeq);++ assert.strictEqual(cursor.decode(r.body.cursor).seq, msgSeq);const app = appToken({ env: 'production', cap: ['events.app.read'] });r = await request(h.base, 'GET', `/api/v1/events?topic=chat.*&after=${await cursorAt(h, msgSeq - 1)}`, { token: app });assert.strictEqual(r.status, 200, r.text);-- assert.deepStrictEqual(r.body.events.map(e => e.seq), [msgSeq, delSeq]);++ assert.deepStrictEqual(r.body.events.map(e => cursor.decode(e.cursor).seq), [msgSeq, delSeq]);assertTombstone(r.body.events[0].event, delEventId);assert.ok(!leaks(r.body));});@@@ -161,7 +162,7 @@ t('pull (/api/v1/events), by id, and developer-app reads: the tombstone', asynct('durable delivery: the consumer gets the tombstone and then the deletion, in order', async () => {await h.worker.drain();const got = stub.calls.map(c => c.body);-- assert.deepStrictEqual(got.map(b => b.seq), [msgSeq, delSeq]);++ assert.deepStrictEqual(got.map(b => b.event.event_id), [msgEventId, delEventId]);assertTombstone(got[0].event, delEventId);assert.strictEqual(got[1].event.event_type, 'chat.message.deleted');assert.ok(!leaks(stub.calls.map(c => c.rawBody.toString())));@@@ -200,7 +201,7 @@ t('re-publishing a deletion is a duplicate; a second deletion leaves tombstonest('only the owning service can redact', async () => {const env = created(201);-- const seq = (await publish(chat, env)).seq;++ const seq = cursor.decode((await publish(chat, env)).cursor).seq;// Live names Chat's event by id: refused, and nothing of the batch is stored.const liveEvent = envelope('live', { payload: { redacts: { event_ids: [env.event_id] } } });let r = await request(h.base, 'POST', '/api/v1/events', { token: live, body: { events: [envelope('live'), liveEvent] } });@@@ -264,7 +265,7 @@ t('operator redaction (backfill): same owner rule, idempotent', async () => {t('public replay window: browsers get a gap for public events older than 5 minutes', async () => {const w = await boot({ env: { REALTIME_PUBLIC_REPLAY_SECONDS: '300' } });try {-- const post = async (token, body) => (await request(w.base, 'POST', '/api/v1/events', { token, body })).body.seq;++ const post = async (token, body) => cursor.decode((await request(w.base, 'POST', '/api/v1/events', { token, body })).body.cursor).seq;const net = serviceToken('network', ['events.event.publish']);const oldPublic = await post(chat, created(1));const oldMine = await post(net, envelope('network', { event_type: 'network.notification.created', visibility: 'subject', subject: { type: 'user', id: alice } }));@@@ -296,7 +297,7 @@ t('public replay window: browsers get a gap for public events older than 5 minutconst none = await boot({ env: { REALTIME_PUBLIC_REPLAY_SECONDS: '0' } });try {-- const s = await (async () => (await request(none.base, 'POST', '/api/v1/events', { token: chat, body: created(3) })).body.seq)();++ const s = await (async () => cursor.decode((await request(none.base, 'POST', '/api/v1/events', { token: chat, body: created(3) })).body.cursor).seq)();const c = await sse(none.base, '/realtime/stream?topics=chat.message.*', { headers: { 'Last-Event-ID': await cursorAt(none, s - 1) } });await c.waitFor(x => x.gaps().length === 1);await sleep(50);
@@ -76,10 +76,10 @@ project's hourly publishing and delivery rollup, for Network; never visible to a
@@ -76,10 +76,10 @@ project's hourly publishing and delivery rollup, for Network; never visible to a
`POST /api/v1/events` takes one `events.event-envelope@1` envelope, or `{ "events": [ … ] }` (up to 100, stored atomically).
`POST /api/v1/events` takes one `events.event-envelope@1` envelope, or `{ "events": [ … ] }` (up to 100, stored atomically).
- `source` must be the calling service (`svc:live` publishes `source: "live"`), and `event_type` must start with a prefix that source owns (`live.*`, `media.*`, `network.*`, `community.*`, `chat.*`, `openre.*`, `billing.*`, `tips.*`, `vip.*`, `ai.*`, `games.*`, `tools.*`, `codes.*`, `host.*`, `bot.*`, `watch.*`, `run.*`, `space.*`, and the publication products (`wiki.*`, `blog.*`, `news.*`, `reviews.*`, `deals.*`, `coupons.*`, `trade.*`, `sources.*`, `search.*`); `EVENTS_SOURCE_PREFIXES` overrides). A network-wide prefix may be published by any first-party service: `provider.*` carries storage provider health/capacity telemetry (`provider.health.degraded`, `provider.capacity.warning`), so whichever service observes a provider can report it; `EVENTS_SHARED_PREFIXES` overrides.
- `source` must be the calling service (`svc:live` publishes `source: "live"`), and `event_type` must start with a prefix that source owns (`live.*`, `media.*`, `network.*`, `community.*`, `chat.*`, `openre.*`, `billing.*`, `tips.*`, `vip.*`, `ai.*`, `games.*`, `tools.*`, `codes.*`, `host.*`, `bot.*`, `watch.*`, `run.*`, `space.*`, and the publication products (`wiki.*`, `blog.*`, `news.*`, `reviews.*`, `deals.*`, `coupons.*`, `trade.*`, `sources.*`, `search.*`); `EVENTS_SOURCE_PREFIXES` overrides). A network-wide prefix may be published by any first-party service: `provider.*` carries storage provider health/capacity telemetry (`provider.health.degraded`, `provider.capacity.warning`), so whichever service observes a provider can report it; `EVENTS_SHARED_PREFIXES` overrides.
-- Idempotent on `event_id`: a repeat answers `200 { event_id, seq, duplicate: true }` and is never stored twice (even after retention, for `EVENTS_RECEIPT_RETENTION_DAYS`).
+- Idempotent on `event_id`: a repeat answers `200 { event_id, cursor, duplicate: true }` (`pruned: true` and no cursor when retention already removed the stored copy) and is never stored twice (even after retention, for `EVENTS_RECEIPT_RETENTION_DAYS`).
- Missing `trace_id` is taken from the request's `traceparent`; `priority` defaults to `important`, `visibility` to `internal`.
- Missing `trace_id` is taken from the request's `traceparent`; `priority` defaults to `important`, `visibility` to `internal`.
- Loop guard: an event whose trace already carries 8 hops (the cross-service chain depth, or the same source/type/subject repeating in the trace) is refused with `409 events.loop_detected`. The chain depth of a first-party publish counts first-party events only (developer-app events in the same trace never count toward it), so an app cannot poison a trace it has seen.
- Loop guard: an event whose trace already carries 8 hops (the cross-service chain depth, or the same source/type/subject repeating in the trace) is refused with `409 events.loop_detected`. The chain depth of a first-party publish counts first-party events only (developer-app events in the same trace never count toward it), so an app cannot poison a trace it has seen.
-- Each accepted event gets a global `seq` and is committed with one delivery row per matching subscription before anything is sent.
+- Each accepted event gets a position in the hot store (handed out only as an opaque cursor; there is no global order on the wire) and is committed with one delivery row per matching subscription before anything is sent.
### Redaction
### Redaction
@@ -92,8 +92,8 @@ A producer takes back what it published (a deleted chat message) with a directiv
@@ -92,8 +92,8 @@ A producer takes back what it published (a deleted chat message) with a directiv
```
```
- `redacts.event_ids` (up to 1000 `evt_…`) names events directly; `subject_type` + `subject_ids` (up to 1000) names every stored event of the same source about those subjects.
- `redacts.event_ids` (up to 1000 `evt_…`) names events directly; `subject_type` + `subject_ids` (up to 1000) names every stored event of the same source about those subjects.
-- In the transaction that stores the directive, each target becomes a **tombstone**: `payload` is replaced by `{ "redacted": true, "redacted_at": "<ISO>", "redacted_by": "<the redacting event_id>" }` and `actor` by the producer itself (`{ "type": "service", "id": "<source>" }`, or the app). `event_id`, `seq`, `event_type`, `subject`, `timestamp` and `visibility` stay, so sequences have no holes. On PostgreSQL the replaced row version is a dead tuple until autovacuum reclaims it (the `events` table vacuums after 1% of its rows change, so within minutes); until then, and in backups taken before the redaction, the old bytes still exist on the host. No read path serves them.
-- Every read path serves the tombstone from then on: pull, `GET /api/v1/events/:id`, SSE replay (anonymous, signed in or service), queued deliveries and DLQ replays. First-party consumers get the tombstone at the original seq and then the deletion event itself; they should check `payload.redacted`.
+- In the transaction that stores the directive, each target becomes a **tombstone**: `payload` is replaced by `{ "redacted": true, "redacted_at": "<ISO>", "redacted_by": "<the redacting event_id>" }` and `actor` by the producer itself (`{ "type": "service", "id": "<source>" }`, or the app). `event_id`, its position, `event_type`, `subject`, `timestamp` and `visibility` stay, so a reader's walk has no holes. On PostgreSQL the replaced row version is a dead tuple until autovacuum reclaims it (the `events` table vacuums after 1% of its rows change, so within minutes); until then, and in backups taken before the redaction, the old bytes still exist on the host. No read path serves them.
+- Every read path serves the tombstone from then on: pull, `GET /api/v1/events/:id`, SSE replay (anonymous, signed in or service), queued deliveries and DLQ replays. First-party consumers get the tombstone at its original position and then the deletion event itself; they should check `payload.redacted`.
- Authority is the publish rule: only the owning source (for an app, its own project and environment) can redact. Naming another source's event by id refuses the whole publish with `403 events.redaction_not_allowed`; a subject match never reaches another source. A malformed directive is `422 events.invalid_redaction`. Events carrying a directive are never redacted themselves. No capability beyond `events.event.publish` is needed.
- Authority is the publish rule: only the owning source (for an app, its own project and environment) can redact. Naming another source's event by id refuses the whole publish with `403 events.redaction_not_allowed`; a subject match never reaches another source. A malformed directive is `422 events.invalid_redaction`. Events carrying a directive are never redacted themselves. No capability beyond `events.event.publish` is needed.
- Redaction applies to what is stored when the directive arrives; it does not block later events about the same subject.
- Redaction applies to what is stored when the directive arrives; it does not block later events about the same subject.
- Events stored before a producer published deletions: the one-time `scripts/redact-backfill.js` (chat only) ran on 2026-09-24 and was retired with the move to PostgreSQL (it is in git history).
- Events stored before a producer published deletions: the one-time `scripts/redact-backfill.js` (chat only) ran on 2026-09-24 and was retired with the move to PostgreSQL (it is in git history).
@@ -108,13 +108,13 @@ POST /api/v1/subscriptions
@@ -108,13 +108,13 @@ POST /api/v1/subscriptions
Topic patterns are dot-separated segments where `*` stands for one or more whole segments (`media.vod.*`, `*.created`, `media.*.ready`, `*`). Endpoints must be `http(s)` on `127.0.0.1` or `openvibe.<tld>`/its subdomains (`EVENTS_ENDPOINT_HOSTS`); redirects are never followed. A caller may supply its own `secret` (a string of 32..256 characters; the same bound on `POST /api/v1/subscriptions/:id/rotate-secret`), otherwise Events generates a `whsec_…` one; either way it is returned once, on creation, and the consumer needs it to verify the v2 signature. The consumer is the calling service, and re-creating a subscription it already holds for the same `topic_pattern` and `endpoint` answers `409 events.subscription_exists` with the existing `subscription_id`, so an ensure-at-boot script re-runs safely.
Topic patterns are dot-separated segments where `*` stands for one or more whole segments (`media.vod.*`, `*.created`, `media.*.ready`, `*`). Endpoints must be `http(s)` on `127.0.0.1` or `openvibe.<tld>`/its subdomains (`EVENTS_ENDPOINT_HOSTS`); redirects are never followed. A caller may supply its own `secret` (a string of 32..256 characters; the same bound on `POST /api/v1/subscriptions/:id/rotate-secret`), otherwise Events generates a `whsec_…` one; either way it is returned once, on creation, and the consumer needs it to verify the v2 signature. The consumer is the calling service, and re-creating a subscription it already holds for the same `topic_pattern` and `endpoint` answers `409 events.subscription_exists` with the existing `subscription_id`, so an ensure-at-boot script re-runs safely.
-Each delivery is `POST <endpoint>` with body `{ "event": <envelope>, "seq": n }` and headers `X-OpenVibe-Event-Id`, `X-OpenVibe-Event-Type`, `X-OpenVibe-Seq`, `X-OpenVibe-Subscription-Id`, `X-OpenVibe-Delivery-Attempt`, `X-OpenVibe-Timestamp: <unix seconds>`, `X-OpenVibe-Signature-V2: t=<that timestamp>,v2=<HMAC-SHA256 of "<t>.<raw body>">` and `traceparent` (the event's trace). During secret rotation, the v2 header also carries a second `v2=` signature made with the previous secret. Any 2xx is delivered. Anything else is retried after 1 s, 5 s, 30 s, 2 min, 10 min, 1 h (then hourly) up to 8 attempts, after which the delivery is `dead`. Priority classes go first (`critical`, `important`, `low`, then seq); one delivery per subscription is in flight at a time and at most 20 overall. Delivery is at least once and not strictly ordered; consumers dedupe with an inbox and order with `subject.revision` (`createPgOrderedInbox` below does both).
+Each delivery is `POST <endpoint>` with body `{ "event": <envelope> }` (no sequence number: dedupe on the event id) and headers `X-OpenVibe-Event-Id`, `X-OpenVibe-Event-Type`, `X-OpenVibe-Subscription-Id`, `X-OpenVibe-Delivery-Attempt`, `X-OpenVibe-Timestamp: <unix seconds>`, `X-OpenVibe-Signature-V2: t=<that timestamp>,v2=<HMAC-SHA256 of "<t>.<raw body>">` and `traceparent` (the event's trace). During secret rotation, the v2 header also carries a second `v2=` signature made with the previous secret. Any 2xx is delivered. Anything else is retried after 1 s, 5 s, 30 s, 2 min, 10 min, 1 h (then hourly) up to 8 attempts, after which the delivery is `dead`. Priority classes go first (`critical`, `important`, `low`, then seq); one delivery per subscription is in flight at a time and at most 20 overall. Delivery is at least once and not strictly ordered; consumers dedupe with an inbox and order with `subject.revision` (`createPgOrderedInbox` below does both).
**Replay window (signature v2).** Events sends only the v2 signature. It signs the timestamp together with the raw body, and every attempt, retries included, is signed afresh with the time it is sent. Consumers check it with `verifyDeliveryV2(raw, headers, secret, { toleranceSec = 300, now })` (here) or openvibe-sdk ≥ 0.4.0 `parseDelivery(raw, headers, secret, { requireV2: true })` and reject anything more than 300 s from their clock either way. Reject a missing, wrong or stale v2 signature; do not fall back to v1.
**Replay window (signature v2).** Events sends only the v2 signature. It signs the timestamp together with the raw body, and every attempt, retries included, is signed afresh with the time it is sent. Consumers check it with `verifyDeliveryV2(raw, headers, secret, { toleranceSec = 300, now })` (here) or openvibe-sdk ≥ 0.4.0 `parseDelivery(raw, headers, secret, { requireV2: true })` and reject anything more than 300 s from their clock either way. Reject a missing, wrong or stale v2 signature; do not fall back to v1.
Operators: `GET /api/v1/deliveries?status=dead` is the dead-letter queue; `POST /api/v1/deliveries/replay { subscription_id, event_ids: [...] }` or `{ subscription_id, from_seq }` requeues retained events (that is also how a new subscription catches up on history).
Operators: `GET /api/v1/deliveries?status=dead` is the dead-letter queue; `POST /api/v1/deliveries/replay { subscription_id, event_ids: [...] }` or `{ subscription_id, from_seq }` requeues retained events (that is also how a new subscription catches up on history).
-Pull consumers: start with `GET /api/v1/events?topic=media.vod.*&limit=100`, then request `GET /api/v1/events?topic=media.vod.*&after=<next_cursor>&limit=100`, using the previous response's opaque `next_cursor` ([ADR-042](https://github.com/OpenVibers/OpenVibe.Contracts/blob/main/docs/adr/ADR-042-events-fabric.md) decision 7). The response carries `{ events: [{ seq, cursor, event }], next_cursor, latest_cursor }` (`latest_cursor` is the head as an opaque cursor, for a consumer that starts at "now" without history), plus `gap: { from_seq, to_seq }` when the position is older than both the hot store and the replay tier (below) or from another retention epoch. Use `next_cursor` even when no events match, since it advances past scanned events. A position is only ever an opaque cursor: `after_seq` is refused with 400 (never read as "from the start"), and the per-event `seq` is informational. `PUT /api/v1/checkpoints { topic, cursor, carrier? }` stores a consumer's cursor here if it has nowhere better (a cursor string; a number is refused); `GET /api/v1/checkpoints?topic=` and the PUT answer `{ consumer, topic, cursor, carrier, updated_at }`, with `cursor` as stored (null when none is).
+Pull consumers: start with `GET /api/v1/events?topic=media.vod.*&limit=100`, then request `GET /api/v1/events?topic=media.vod.*&after=<next_cursor>&limit=100`, using the previous response's opaque `next_cursor` ([ADR-042](https://github.com/OpenVibers/OpenVibe.Contracts/blob/main/docs/adr/ADR-042-events-fabric.md) decision 7). The response carries `{ events: [{ cursor, event }], next_cursor, latest_cursor }` (`latest_cursor` is the head as an opaque cursor, for a consumer that starts at "now" without history), plus `gap: { from_seq, to_seq }` when the position is older than both the hot store and the replay tier (below) or from another retention epoch. Use `next_cursor` even when no events match, since it advances past scanned events. A position is only ever an opaque cursor: `after_seq` is refused with 400 (never read as "from the start"), and there is no per-event sequence number (dedupe on `event.event_id`). `PUT /api/v1/checkpoints { topic, cursor, carrier? }` stores a consumer's cursor here if it has nowhere better (a cursor string; a number is refused); `GET /api/v1/checkpoints?topic=` and the PUT answer `{ consumer, topic, cursor, carrier, updated_at }`, with `cursor` as stored (null when none is).
### Retention: three tiers
### Retention: three tiers
@@ -126,7 +126,7 @@ Pull consumers: start with `GET /api/v1/events?topic=media.vod.*&limit=100`, the
@@ -126,7 +126,7 @@ Pull consumers: start with `GET /api/v1/events?topic=media.vod.*&limit=100`, the
| replay | `events_archive`, same database: the filter columns plus a gzip'd body, no delivery rows | `EVENTS_REPLAY_RETENTION_DAYS` (365) from receipt; `0` = no replay tier | pulls, SSE resume and `GET /api/v1/events/:id`, with the same cursor |
| replay | `events_archive`, same database: the filter columns plus a gzip'd body, no delivery rows | `EVENTS_REPLAY_RETENTION_DAYS` (365) from receipt; `0` = no replay tier | pulls, SSE resume and `GET /api/v1/events/:id`, with the same cursor |
| archive | one NDJSON object per month (gzip, with a manifest), outside the database | until an operator deletes it | nothing online; an operator restores a month into replay |
| archive | one NDJSON object per month (gzip, with a manifest), outside the database | until an operator deletes it | nothing online; an operator restores a month into replay |
-The hourly prune (`EVENTS_PRUNE_INTERVAL_MS`) moves each hot event past `EVENTS_RETENTION_DAYS` into `events_archive` in the transaction that deletes it: 500 rows at a time, locked `FOR UPDATE SKIP LOCKED`, inserted idempotently on `event_id`, so two Events processes pruning at once each move different rows and no event is moved twice. Its deliveries go with the hot row (operator replay to a subscription, `POST /api/v1/deliveries/replay`, reaches hot events only). Replay rows past `EVENTS_REPLAY_RETENTION_DAYS` are then deleted. A pull or SSE resume runs from a cursor in the replay tier into the hot store without a `gap`; `gap` (the same shape) still means the position is older than both tiers or from another epoch. Redaction tombstones replay rows exactly as hot ones, the sandbox and environment rules read them the same way, and a re-publish of an archived `event_id` is still `duplicate: true` with its original `seq` and `cursor`. Sandbox events never enter replay: they are deleted at `EVENTS_APP_SANDBOX_RETENTION_DAYS`, as before; publish receipts keep `EVENTS_RECEIPT_RETENTION_DAYS`.
+The hourly prune (`EVENTS_PRUNE_INTERVAL_MS`) moves each hot event past `EVENTS_RETENTION_DAYS` into `events_archive` in the transaction that deletes it: 500 rows at a time, locked `FOR UPDATE SKIP LOCKED`, inserted idempotently on `event_id`, so two Events processes pruning at once each move different rows and no event is moved twice. Its deliveries go with the hot row (operator replay to a subscription, `POST /api/v1/deliveries/replay`, reaches hot events only). Replay rows past `EVENTS_REPLAY_RETENTION_DAYS` are then deleted. A pull or SSE resume runs from a cursor in the replay tier into the hot store without a `gap`; `gap` (the same shape) still means the position is older than both tiers or from another epoch. Redaction tombstones replay rows exactly as hot ones, the sandbox and environment rules read them the same way, and a re-publish of an archived `event_id` is still `duplicate: true` with its original `cursor`. Sandbox events never enter replay: they are deleted at `EVENTS_APP_SANDBOX_RETENTION_DAYS`, as before; publish receipts keep `EVENTS_RECEIPT_RETENTION_DAYS`.
`EVENTS_REPLAY_RETENTION_DAYS=0` keeps the old behaviour exactly: events past `EVENTS_RETENTION_DAYS` are deleted, and the next prune also empties `events_archive`, because no rule would prune the rows left there otherwise. `GET /limits.json` publishes the window as `replay_retention_days`.
`EVENTS_REPLAY_RETENTION_DAYS=0` keeps the old behaviour exactly: events past `EVENTS_RETENTION_DAYS` are deleted, and the next prune also empties `events_archive`, because no rule would prune the rows left there otherwise. `GET /limits.json` publishes the window as `replay_retention_days`.
@@ -273,7 +273,7 @@ async function connect() {
@@ -273,7 +273,7 @@ async function connect() {
url.searchParams.set('ticket', ticket);
url.searchParams.set('ticket', ticket);
if (lastEventId != null) url.searchParams.set('last_event_id', lastEventId);
if (lastEventId != null) url.searchParams.set('last_event_id', lastEventId);
const stream = new EventSource(url);
const stream = new EventSource(url);
- stream.onmessage = (m) => { lastEventId = m.lastEventId; const { seq, event } = JSON.parse(m.data); };
+ stream.onmessage = (m) => { lastEventId = m.lastEventId; const { event } = JSON.parse(m.data); };
stream.addEventListener('gap', (m) => { /* events were missed: refetch state */ });
stream.addEventListener('gap', (m) => { /* events were missed: refetch state */ });
stream.onerror = () => { stream.close(); setTimeout(connect, 1000); };
stream.onerror = () => { stream.close(); setTimeout(connect, 1000); };
} catch (err) {
} catch (err) {
@@ -387,7 +387,7 @@ implemented service.
@@ -387,7 +387,7 @@ implemented service.
Part of the [OpenVibe network](https://openvibe.network). Built in the open by [OpenVibers](https://github.com/OpenVibers).
Part of the [OpenVibe network](https://openvibe.network). Built in the open by [OpenVibers](https://github.com/OpenVibers).
<!-- versions:start -->
<!-- versions:start -->
-- openvibe-contracts: v0.132.0
+- openvibe-contracts: v0.136.1
- openvibe-sdk: v0.42.0
- openvibe-sdk: v0.42.0
- openvibe-shared: v3.0.1
- openvibe-shared: v3.0.1
<!-- versions:end -->
<!-- versions:end -->
@@ -7,7 +7,7 @@
@@ -7,7 +7,7 @@
"runtime": "node 22, express, PostgreSQL 18 (openvibe-sdk/db, async) + Valkey 9",
"runtime": "node 22, express, PostgreSQL 18 (openvibe-sdk/db, async) + Valkey 9",
"port": 4300,
"port": 4300,
"domain": "openvibe.events",
"domain": "openvibe.events",
- "contracts": "openvibe-contracts v0.132.0",
+ "contracts": "openvibe-contracts v0.136.1",
"sdk": "openvibe-sdk v0.42.0",
"sdk": "openvibe-sdk v0.42.0",
"shared": "openvibe-shared v3.0.1",
"shared": "openvibe-shared v3.0.1",
"capabilities": [
"capabilities": [
@@ -37,5 +37,5 @@
@@ -37,5 +37,5 @@
"carrier fabric (ADR-042): the pg-v1 poll beside a valkey-v1 push carrier for QUEUE (streams) and TOPIC (SSE fan-out) an optional nats-v1 NATS Core TOPIC carrier (NATS_URL) and an optional nats-js-v1 JetStream STREAM carrier (NATS_URL, NATS_JETSTREAM; deploy/nats) and an optional cloud-v1 SQS-compatible QUEUE carrier (CLOUD_QUEUE_URL; SigV4, no dependency), with a placement planner, per-key pinning, delivery_placements history and a GET /api/v1/placement explain API; EVENTS_CARRIERS[_DISABLED] switches it",
"carrier fabric (ADR-042): the pg-v1 poll beside a valkey-v1 push carrier for QUEUE (streams) and TOPIC (SSE fan-out) an optional nats-v1 NATS Core TOPIC carrier (NATS_URL) and an optional nats-js-v1 JetStream STREAM carrier (NATS_URL, NATS_JETSTREAM; deploy/nats) and an optional cloud-v1 SQS-compatible QUEUE carrier (CLOUD_QUEUE_URL; SigV4, no dependency), with a placement planner, per-key pinning, delivery_placements history and a GET /api/v1/placement explain API; EVENTS_CARRIERS[_DISABLED] switches it",
"at_most_once delivery semantics (a failed attempt is final)"
"at_most_once delivery semantics (a failed attempt is final)"
],
],
- "updated": "2026-10-10"
+ "updated": "2026-10-11"
}
}
@@ -154,8 +154,8 @@ class PublishError extends Error {
@@ -154,8 +154,8 @@ class PublishError extends Error {
/**
/**
* createPublisher({ eventsUrl, tokenClient, fetchImpl?, timeoutMs? })
* createPublisher({ eventsUrl, tokenClient, fetchImpl?, timeoutMs? })
- * publish(envelope | [envelopes], { traceparent }) -> { event_id, seq, cursor, duplicate } | { results: [...] }
- * (`cursor` is the opaque position beside `seq`; ADR-042 decision 7. Hand it back as `after=` or `Last-Event-ID`.)
+ * publish(envelope | [envelopes], { traceparent }) -> { event_id, cursor, duplicate } | { results: [...] }
+ * (`cursor` is the event's opaque position; ADR-042 decision 7. Hand it back as `after=` or `Last-Event-ID`.)
* A 401 invalidates the cached token and retries once.
* A 401 invalidates the cached token and retries once.
*/
*/
function createPublisher({ eventsUrl, tokenClient, fetchImpl = globalThis.fetch, timeoutMs = 10000 }) {
function createPublisher({ eventsUrl, tokenClient, fetchImpl = globalThis.fetch, timeoutMs = 10000 }) {
@@ -12,7 +12,7 @@
@@ -12,7 +12,7 @@
"dotenv": "^16.4.7",
"dotenv": "^16.4.7",
"express": "^4.21.2",
"express": "^4.21.2",
"iovalkey": "^0.4.0",
"iovalkey": "^0.4.0",
- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.132.0",
+ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.136.1",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.42.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.42.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v3.0.1",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v3.0.1",
"pg": "^8.23.0"
"pg": "^8.23.0"
@@ -828,9 +828,9 @@
@@ -828,9 +828,9 @@
}
}
},
},
"node_modules/openvibe-contracts": {
"node_modules/openvibe-contracts": {
- "version": "0.132.0",
- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.132.0",
- "integrity": "sha512-08GuHutggWYtGcGrnBLilWco/K715Bl7ZAdfYX7kXeeHguWTIhbWZsSF3JYcH9iq6KKtRB7XCAbmPFKScleD9Q==",
+ "version": "0.136.1",
+ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.136.1",
+ "integrity": "sha512-G+xlUJEle2MlrEnVrKha5PfKLTh27cYomf2ioTZFwMSK5+Vy2BZlt5kkAOOmZ9zyJKDAL2CkEBlJy9rsUOY8UA==",
"license": "AGPL-3.0-only",
"license": "AGPL-3.0-only",
"dependencies": {
"dependencies": {
"ajv": "^8.17.1",
"ajv": "^8.17.1",
@@ -24,7 +24,7 @@
@@ -24,7 +24,7 @@
"dotenv": "^16.4.7",
"dotenv": "^16.4.7",
"express": "^4.21.2",
"express": "^4.21.2",
"iovalkey": "^0.4.0",
"iovalkey": "^0.4.0",
- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.132.0",
+ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.136.1",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.42.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.42.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v3.0.1",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v3.0.1",
"pg": "^8.23.0"
"pg": "^8.23.0"
@@ -3,15 +3,16 @@
@@ -3,15 +3,16 @@
* POST /api/v1/events (service token, audience openvibe.events, capability events.event.publish;
* POST /api/v1/events (service token, audience openvibe.events, capability events.event.publish;
* or a developer app token with events.app.publish)
* or a developer app token with events.app.publish)
*
*
- * body: <envelope> -> 201 { event_id, seq, cursor, duplicate: false } | 200 on a repeat
- * body: { events: [...] } -> 201 { results: [{ event_id, seq, cursor, duplicate }] } (<= 100, atomic)
+ * body: <envelope> -> 201 { event_id, cursor, duplicate: false } | 200 on a repeat
+ * body: { events: [...] } -> 201 { results: [{ event_id, cursor, duplicate }] } (<= 100, atomic)
*
*
- * `cursor` (ADR-042 decision 7) is the opaque position beside seq; `seq` and the global-order promise
- * stay for one release, then go.
+ * `cursor` (ADR-042 decision 7) is the event's opaque position; there is no sequence number on the wire and no
+ * global order. A repeat whose stored copy retention already removed answers { event_id, duplicate: true,
+ * pruned: true } with no cursor.
*
*
* Every envelope must validate against events.event-envelope@1; `source` must be the calling
* Every envelope must validate against events.event-envelope@1; `source` must be the calling
* service (svc:live -> 'live'); `event_type` must start with a prefix that source owns. A
* service (svc:live -> 'live'); `event_type` must start with a prefix that source owns. A
- * re-published event_id is answered with the stored seq and never stored twice.
+ * re-published event_id is answered with its stored position and never stored twice.
*
*
* Developer apps (ADR-014, server/apps.js): `source` must be the app's `app-<lowercased ULID>`,
* Developer apps (ADR-014, server/apps.js): `source` must be the app's `app-<lowercased ULID>`,
* `event_type` must start with `app.<project_key>.`, and `actor` must be the app itself or the user
* `event_type` must start with `app.<project_key>.`, and `actor` must be the app itself or the user
@@ -3,16 +3,17 @@
@@ -3,16 +3,17 @@
* Pull consumers (service token with events.event.read):
* Pull consumers (service token with events.event.read):
*
*
* GET /api/v1/events?topic=media.vod.*[,…][&after=<cursor>]&limit=100
* GET /api/v1/events?topic=media.vod.*[,…][&after=<cursor>]&limit=100
- * -> { events: [{ seq, cursor, event }], next_cursor, latest_cursor, gap? }
+ * -> { events: [{ cursor, event }], next_cursor, latest_cursor, gap? }
* A position is only ever an opaque cursor (ADR-042 decision 7): `after=` takes one, and without it the
* A position is only ever an opaque cursor (ADR-042 decision 7): `after=` takes one, and without it the
* page starts at the oldest retained event. `after_seq` is refused (400), never read as "from the start".
* page starts at the oldest retained event. `after_seq` is refused (400), never read as "from the start".
- * `latest_cursor` is the head: a consumer that starts at "now" (skipping history) stores it. The per-event
- * `seq` is informational. A cursor from another retention epoch answers `gap` — never a silent restart.
+ * `latest_cursor` is the head: a consumer that starts at "now" (skipping history) stores it. There is no
+ * per-event sequence number (dedupe on event.event_id). A cursor from another retention epoch answers `gap`
+ * — never a silent restart.
* `gap` ({ from_seq, to_seq }) means
* `gap` ({ from_seq, to_seq }) means
* events after the position were already pruned by retention (or the epoch changed). The pull
* events after the position were already pruned by retention (or the epoch changed). The pull
* spans the hot store and the replay tier (decision 8) with the one cursor: `gap` only outside both.
* spans the hot store and the replay tier (decision 8) with the one cursor: `gap` only outside both.
* Keep next_cursor as the cursor; it moves past events that did not match.
* Keep next_cursor as the cursor; it moves past events that did not match.
- * GET /api/v1/events/:event_id -> { seq, cursor, event } (hot or replay)
+ * GET /api/v1/events/:event_id -> { cursor, event } (hot or replay)
* GET /api/v1/checkpoints?topic=… / PUT /api/v1/checkpoints { topic, cursor, carrier? }
* GET /api/v1/checkpoints?topic=… / PUT /api/v1/checkpoints { topic, cursor, carrier? }
* a consumer's own stored cursor per topic pattern (consumer = calling principal): `cursor` is an
* a consumer's own stored cursor per topic pattern (consumer = calling principal): `cursor` is an
* opaque cursor string (a number is refused), and both answers return it as stored, with its carrier
* opaque cursor string (a number is refused), and both answers return it as stored, with its carrier
@@ -105,7 +106,7 @@ function readRouter({ store, auth, worker, limits }) {
@@ -105,7 +106,7 @@ function readRouter({ store, auth, worker, limits }) {
from = oldest - 1;
from = oldest - 1;
}
}
const { rows, cursor: scanned } = await store.scan(from, { patterns, limit, accept: acceptFor(req.principal, patterns) });
const { rows, cursor: scanned } = await store.scan(from, { patterns, limit, accept: acceptFor(req.principal, patterns) });
- out.events = rows.map(r => ({ seq: r.seq, cursor: cursor.encode(r.seq, epoch), event: rowToEnvelope(r) }));
+ out.events = rows.map(r => ({ cursor: cursor.encode(r.seq, epoch), event: rowToEnvelope(r) }));
out.next_cursor = cursor.encode(scanned, epoch);
out.next_cursor = cursor.encode(scanned, epoch);
out.latest_cursor = cursor.encode(await store.lastSeq(), epoch);
out.latest_cursor = cursor.encode(await store.lastSeq(), epoch);
res.json(out);
res.json(out);
@@ -116,7 +117,7 @@ function readRouter({ store, auth, worker, limits }) {
@@ -116,7 +117,7 @@ function readRouter({ store, auth, worker, limits }) {
const visible = row && (req.principal.kind === 'app' ? apps.visibleToApp(row, req.principal) : (row.env || 'production') === 'production');
const visible = row && (req.principal.kind === 'app' ? apps.visibleToApp(row, req.principal) : (row.env || 'production') === 'production');
if (!visible) return http.sendProblem(res, 404, 'events.not_found', { detail: 'no such event (or pruned by retention)', ctx: req.ov });
if (!visible) return http.sendProblem(res, 404, 'events.not_found', { detail: 'no such event (or pruned by retention)', ctx: req.ov });
// A replay-tier row names the epoch its position belongs to; a hot row is in the current one.
// A replay-tier row names the epoch its position belongs to; a hot row is in the current one.
- return res.json({ seq: row.seq, cursor: cursor.encode(row.seq, row.epoch ?? await store.epoch()), event: rowToEnvelope(row) });
+ return res.json({ cursor: cursor.encode(row.seq, row.epoch ?? await store.epoch()), event: rowToEnvelope(row) });
});
});
const consumerOf = (req) => req.principal.service || req.principal.sub;
const consumerOf = (req) => req.principal.service || req.principal.sub;
@@ -8,8 +8,8 @@
@@ -8,8 +8,8 @@
* minted by Network for the signed-in person: how pages on every OpenVibe site open a stream without a
* minted by Network for the signed-in person: how pages on every OpenVibe site open a stream without a
* third-party cookie), a Bearer user or service token, the ov_token cookie, or nobody.
* third-party cookie), a Bearer user or service token, the ov_token cookie, or nobody.
*
*
- * Every message is `id: <cursor>` (ADR-042 decision 7; a bare seq is still accepted for one release) +
- * `data: {"seq":n,"event":<envelope>}`. Visibility decides who sees an event:
+ * Every message is `id: <cursor>` (ADR-042 decision 7) + `data: {"event":<envelope>}`; there is no sequence
+ * number (a client dedupes on event.event_id). Visibility decides who sees an event:
* public anyone subscribed to a matching topic (signed-out visitors too, unless disabled)
* public anyone subscribed to a matching topic (signed-out visitors too, unless disabled)
* subject only the user whose subject id is the event's actor.id or its subject.id (subject
* subject only the user whose subject id is the event's actor.id or its subject.id (subject
* type user); a guessed topic yields nothing for anyone else. A person's topic is an
* type user); a guessed topic yields nothing for anyone else. A person's topic is an
@@ -91,7 +91,7 @@ function createRealtime({ store, auth, config, clock = { now: () => Date.now() }
@@ -91,7 +91,7 @@ function createRealtime({ store, auth, config, clock = { now: () => Date.now() }
function sendEvent(conn, row) {
function sendEvent(conn, row) {
conn.lastSeq = row.seq;
conn.lastSeq = row.seq;
- write(conn, `id: ${cursor.encode(row.seq, conn.epoch)}\ndata: ${JSON.stringify({ seq: row.seq, event: rowToEnvelope(row) })}\n\n`);
+ write(conn, `id: ${cursor.encode(row.seq, conn.epoch)}\ndata: ${JSON.stringify({ event: rowToEnvelope(row) })}\n\n`);
}
}
function sendGap(conn, gap) {
function sendGap(conn, gap) {
@@ -204,11 +204,11 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -204,11 +204,11 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
if (existing.source !== env.source || existing.event_type !== env.event_type) {
if (existing.source !== env.source || existing.event_type !== env.event_type) {
throw new StoreError(409, 'events.id_conflict', `event ${env.event_id} already exists with different content`);
throw new StoreError(409, 'events.id_conflict', `event ${env.event_id} already exists with different content`);
}
}
- results.push({ event_id: env.event_id, seq: existing.seq, duplicate: true, cursor: cursor.encode(existing.seq, existing.epoch ?? epoch) });
+ results.push({ event_id: env.event_id, duplicate: true, cursor: cursor.encode(existing.seq, existing.epoch ?? epoch) });
continue;
continue;
}
}
if (await q.getReceipt.get(PUBLISH_RECEIPT, env.event_id)) {
if (await q.getReceipt.get(PUBLISH_RECEIPT, env.event_id)) {
- results.push({ event_id: env.event_id, seq: null, duplicate: true, pruned: true });
+ results.push({ event_id: env.event_id, duplicate: true, pruned: true });
continue;
continue;
}
}
// Loop guard: depth of the cross-service chain in this trace, and how often this exact
// Loop guard: depth of the cross-service chain in this trace, and how often this exact
@@ -293,7 +293,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -293,7 +293,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
if (carrier) deliveries.push({ event_id: env.event_id, subscription_id: s.id, carrier, carrier_class: klass, ordering_key: key });
if (carrier) deliveries.push({ event_id: env.event_id, subscription_id: s.id, carrier, carrier_class: klass, ordering_key: key });
}
}
await revocationHook(row, now);
await revocationHook(row, now);
- results.push({ event_id: env.event_id, seq, duplicate: false, cursor: cursor.encode(seq, epoch) });
+ results.push({ event_id: env.event_id, duplicate: false, cursor: cursor.encode(seq, epoch) });
inserted.push(row);
inserted.push(row);
if (directive) {
if (directive) {
const redacted = await applyRedaction(row, directive, now);
const redacted = await applyRedaction(row, directive, now);
@@ -2,9 +2,9 @@
@@ -2,9 +2,9 @@
/**
/**
* Delivery worker. Pending deliveries are POSTed to their subscription endpoints as
* Delivery worker. Pending deliveries are POSTed to their subscription endpoints as
*
*
- * { "event": <envelope>, "seq": <n> }
+ * { "event": <envelope> }
*
*
- * with X-OpenVibe-Event-Id, X-OpenVibe-Signature (sha256=<HMAC of the raw body with the
+ * (no sequence number: a receiver dedupes on event.event_id) with X-OpenVibe-Event-Id, X-OpenVibe-Signature (sha256=<HMAC of the raw body with the
* subscription secret>), X-OpenVibe-Timestamp (unix seconds, new on every attempt) and
* subscription secret>), X-OpenVibe-Timestamp (unix seconds, new on every attempt) and
* X-OpenVibe-Signature-V2 (t=<that timestamp>,v2=<HMAC of "<t>.<raw body>">; consumers reject it
* X-OpenVibe-Signature-V2 (t=<that timestamp>,v2=<HMAC of "<t>.<raw body>">; consumers reject it
* outside ±300 s, so a captured delivery cannot be replayed later), traceparent (the event's trace). A 2xx is delivered; anything else
* outside ±300 s, so a captured delivery cannot be replayed later), traceparent (the event's trace). A 2xx is delivered; anything else
@@ -73,14 +73,13 @@ function createWorker({ store, config, clock = { now: () => Date.now() }, fetchI
@@ -73,14 +73,13 @@ function createWorker({ store, config, clock = { now: () => Date.now() }, fetchI
const { maxAttempts, backoffMs } = policy(sub);
const { maxAttempts, backoffMs } = policy(sub);
let outcome;
let outcome;
try {
try {
- const body = JSON.stringify({ event: rowToEnvelope(row), seq: row.seq });
+ const body = JSON.stringify({ event: rowToEnvelope(row) });
const timestamp = Math.floor(clock.now() / 1000); // per attempt: a retry is signed afresh
const timestamp = Math.floor(clock.now() / 1000); // per attempt: a retry is signed afresh
const headers = {
const headers = {
'Content-Type': 'application/json',
'Content-Type': 'application/json',
'User-Agent': 'OpenVibe.Events/0.1',
'User-Agent': 'OpenVibe.Events/0.1',
'X-OpenVibe-Event-Id': row.id,
'X-OpenVibe-Event-Id': row.id,
'X-OpenVibe-Event-Type': row.event_type,
'X-OpenVibe-Event-Type': row.event_type,
- 'X-OpenVibe-Seq': String(row.seq),
'X-OpenVibe-Subscription-Id': sub.id,
'X-OpenVibe-Subscription-Id': sub.id,
'X-OpenVibe-Delivery-Attempt': String(attempt),
'X-OpenVibe-Delivery-Attempt': String(attempt),
'X-OpenVibe-Hops': String(row.hops),
'X-OpenVibe-Hops': String(row.hops),
@@ -43,14 +43,14 @@ t('a hot-pruned event is readable through replay: by id and by pull, unchanged',
@@ -43,14 +43,14 @@ t('a hot-pruned event is readable through replay: by id and by pull, unchanged',
assert.deepStrictEqual([await count('events', e.event_id), await count('events_archive', e.event_id)], [0, 1], 'moved, not copied');
assert.deepStrictEqual([await count('events', e.event_id), await count('events_archive', e.event_id)], [0, 1], 'moved, not copied');
const after = await request(h.base, 'GET', `/api/v1/events/${e.event_id}`, { token: reader });
const after = await request(h.base, 'GET', `/api/v1/events/${e.event_id}`, { token: reader });
assert.strictEqual(after.status, 200, after.text);
assert.strictEqual(after.status, 200, after.text);
- assert.deepStrictEqual(after.body, before.body, 'the same seq, cursor and envelope from the replay tier');
+ assert.deepStrictEqual(after.body, before.body, 'the same cursor and envelope from the replay tier');
const p = await pull('topic=live.tier.one');
const p = await pull('topic=live.tier.one');
assert.deepStrictEqual(p.body.events.map(x => x.event), [before.body.event]);
assert.deepStrictEqual(p.body.events.map(x => x.event), [before.body.event]);
assert.strictEqual(p.body.gap, undefined);
assert.strictEqual(p.body.gap, undefined);
// A publish of the same id is a duplicate with its position, never a second row.
// A publish of the same id is a duplicate with its position, never a second row.
const again = await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope('live', { event_id: e.event_id, event_type: 'live.tier.one' }) });
const again = await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope('live', { event_id: e.event_id, event_type: 'live.tier.one' }) });
assert.strictEqual(again.status, 200, again.text);
assert.strictEqual(again.status, 200, again.text);
- assert.deepStrictEqual([again.body.duplicate, again.body.seq, again.body.cursor], [true, e.seq, e.cursor]);
+ assert.deepStrictEqual([again.body.duplicate, again.body.cursor], [true, e.cursor]);
});
});
t('a cursor across the hot/replay boundary reports no false gap (pull and SSE)', async () => {
t('a cursor across the hot/replay boundary reports no false gap (pull and SSE)', async () => {
@@ -61,18 +61,18 @@ t('a cursor across the hot/replay boundary reports no false gap (pull and SSE)',
@@ -61,18 +61,18 @@ t('a cursor across the hot/replay boundary reports no false gap (pull and SSE)',
const c = await publish(live, envelope('live', { event_type: 'live.tier.cross' }));
const c = await publish(live, envelope('live', { event_type: 'live.tier.cross' }));
let r = await pull(`topic=live.tier.cross&after=${a.cursor}&limit=1`);
let r = await pull(`topic=live.tier.cross&after=${a.cursor}&limit=1`);
- assert.deepStrictEqual([r.body.events.map(x => x.seq), r.body.gap], [[b.seq], undefined], 'replay');
+ assert.deepStrictEqual([r.body.events.map(x => x.cursor), r.body.gap], [[b.cursor], undefined], 'replay');
assert.strictEqual(r.body.events[0].cursor, b.cursor);
assert.strictEqual(r.body.events[0].cursor, b.cursor);
r = await pull(`topic=live.tier.cross&after=${r.body.next_cursor}&limit=1`);
r = await pull(`topic=live.tier.cross&after=${r.body.next_cursor}&limit=1`);
- assert.deepStrictEqual([r.body.events.map(x => x.seq), r.body.gap], [[c.seq], undefined], 'then hot, no gap between');
- r = await pull(`topic=*&after=${await cursorAt(h, a.seq - 1)}`);
- assert.deepStrictEqual(r.body.events.map(x => x.seq), [a.seq, b.seq, c.seq]);
+ assert.deepStrictEqual([r.body.events.map(x => x.cursor), r.body.gap], [[c.cursor], undefined], 'then hot, no gap between');
+ r = await pull(`topic=*&after=${await cursorAt(h, cursor.decode(a.cursor).seq - 1)}`);
+ assert.deepStrictEqual(r.body.events.map(x => x.cursor), [a.cursor, b.cursor, c.cursor]);
assert.strictEqual(r.body.gap, undefined);
assert.strictEqual(r.body.gap, undefined);
const s = await sse(h.base, '/realtime/stream?topics=live.tier.cross', { headers: { Authorization: `Bearer ${serviceToken('live', ['events.event.read'])}`, 'Last-Event-ID': a.cursor } });
const s = await sse(h.base, '/realtime/stream?topics=live.tier.cross', { headers: { Authorization: `Bearer ${serviceToken('live', ['events.event.read'])}`, 'Last-Event-ID': a.cursor } });
open.push(s);
open.push(s);
await s.waitFor(x => x.events().length === 2);
await s.waitFor(x => x.events().length === 2);
- assert.deepStrictEqual(s.events().map(x => x.seq), [b.seq, c.seq]);
+ assert.deepStrictEqual(s.events().map(x => x.event.event_id), [b.event_id, c.event_id]);
assert.deepStrictEqual(s.gaps(), []);
assert.deepStrictEqual(s.gaps(), []);
assert.ok(s.messages.some(m => m.id === b.cursor), 'the SSE id of a replayed event is its cursor');
assert.ok(s.messages.some(m => m.id === b.cursor), 'the SSE id of a replayed event is its cursor');
s.close();
s.close();
@@ -90,7 +90,7 @@ t('redaction reaches replay rows (by id and by subject); the owner rule holds th
@@ -90,7 +90,7 @@ t('redaction reaches replay rows (by id and by subject); the owner rule holds th
const r = await request(h.base, 'GET', `/api/v1/events/${e.event_id}`, { token: reader });
const r = await request(h.base, 'GET', `/api/v1/events/${e.event_id}`, { token: reader });
assert.strictEqual(r.body.event.payload.redacted, true, JSON.stringify(r.body));
assert.strictEqual(r.body.event.payload.redacted, true, JSON.stringify(r.body));
assert.deepStrictEqual(r.body.event.actor, { type: 'service', id: 'live' });
assert.deepStrictEqual(r.body.event.actor, { type: 'service', id: 'live' });
- assert.strictEqual(r.body.seq, e.seq);
+ assert.strictEqual(r.body.cursor, e.cursor);
}
}
const p = await pull('topic=live.tier.secret');
const p = await pull('topic=live.tier.secret');
assert.ok(p.body.events.every(e => e.event.payload.redacted === true && !('secret' in e.event.payload)));
assert.ok(p.body.events.every(e => e.event.payload.redacted === true && !('secret' in e.event.payload)));
@@ -109,9 +109,9 @@ t('sandbox and environment rules apply to replay rows exactly as to hot ones', a
@@ -109,9 +109,9 @@ t('sandbox and environment rules apply to replay rows exactly as to hot ones', a
assert.deepStrictEqual([await count('events_archive', pe.event_id), await count('events_archive', se.event_id), await count('events', se.event_id)], [1, 0, 0],
assert.deepStrictEqual([await count('events_archive', pe.event_id), await count('events_archive', se.event_id), await count('events', se.event_id)], [1, 0, 0],
'a production event moves; a sandbox event is deleted, never kept in replay');
'a production event moves; a sandbox event is deleted, never kept in replay');
const topic = `topic=app.${apps.projectKey(prj)}.*`;
const topic = `topic=app.${apps.projectKey(prj)}.*`;
- assert.deepStrictEqual((await pull(topic, prod)).body.events.map(e => e.seq), [pe.seq], 'the app reads its own replay rows');
+ assert.deepStrictEqual((await pull(topic, prod)).body.events.map(e => e.cursor), [pe.cursor], 'the app reads its own replay rows');
assert.deepStrictEqual((await pull(topic, sand)).body.events, [], 'never across environments');
assert.deepStrictEqual((await pull(topic, sand)).body.events, [], 'never across environments');
- assert.deepStrictEqual((await pull(topic)).body.events.map(e => e.seq), [pe.seq], 'first-party: production app events through app.*');
+ assert.deepStrictEqual((await pull(topic)).body.events.map(e => e.cursor), [pe.cursor], 'first-party: production app events through app.*');
assert.deepStrictEqual((await pull('topic=*')).body.events.filter(e => e.event.event_id === pe.event_id), [], '…and only through app.*');
assert.deepStrictEqual((await pull('topic=*')).body.events.filter(e => e.event.event_id === pe.event_id), [], '…and only through app.*');
assert.strictEqual((await request(h.base, 'GET', `/api/v1/events/${pe.event_id}`, { token: sand })).status, 404);
assert.strictEqual((await request(h.base, 'GET', `/api/v1/events/${pe.event_id}`, { token: sand })).status, 404);
assert.strictEqual((await request(h.base, 'GET', `/api/v1/events/${pe.event_id}`, { token: prod })).status, 200);
assert.strictEqual((await request(h.base, 'GET', `/api/v1/events/${pe.event_id}`, { token: prod })).status, 200);
@@ -189,12 +189,12 @@ t('replay pruning answers `gap` (the existing shape), pull and SSE', async () =>
@@ -189,12 +189,12 @@ t('replay pruning answers `gap` (the existing shape), pull and SSE', async () =>
await h.store.prune(tiered(366));
await h.store.prune(tiered(366));
const kept = await publish(live, envelope('live', { event_type: 'live.tier.gone' }));
const kept = await publish(live, envelope('live', { event_type: 'live.tier.gone' }));
const r = await pull(`topic=live.tier.gone&after=${a.cursor}`);
const r = await pull(`topic=live.tier.gone&after=${a.cursor}`);
- assert.deepStrictEqual(r.body.gap, { from_seq: a.seq + 1, to_seq: kept.seq - 1 });
- assert.deepStrictEqual(r.body.events.map(e => e.seq), [kept.seq]);
+ assert.deepStrictEqual(r.body.gap, { from_seq: cursor.decode(a.cursor).seq + 1, to_seq: cursor.decode(kept.cursor).seq - 1 });
+ assert.deepStrictEqual(r.body.events.map(e => e.cursor), [kept.cursor]);
const s = await sse(h.base, '/realtime/stream?topics=live.tier.gone', { headers: { Authorization: `Bearer ${serviceToken('live', ['events.event.read'])}`, 'Last-Event-ID': a.cursor } });
const s = await sse(h.base, '/realtime/stream?topics=live.tier.gone', { headers: { Authorization: `Bearer ${serviceToken('live', ['events.event.read'])}`, 'Last-Event-ID': a.cursor } });
open.push(s);
open.push(s);
await s.waitFor(x => x.events().length === 1);
await s.waitFor(x => x.events().length === 1);
- assert.deepStrictEqual(s.gaps().map(g => [g.reason, g.from_seq, g.to_seq]), [['retention', a.seq + 1, kept.seq - 1]]);
+ assert.deepStrictEqual(s.gaps().map(g => [g.reason, g.from_seq, g.to_seq]), [['retention', cursor.decode(a.cursor).seq + 1, cursor.decode(kept.cursor).seq - 1]]);
s.close();
s.close();
});
});
@@ -216,7 +216,7 @@ t('EVENTS_REPLAY_RETENTION_DAYS=0 behaves exactly as before the tiers', async ()
@@ -216,7 +216,7 @@ t('EVENTS_REPLAY_RETENTION_DAYS=0 behaves exactly as before the tiers', async ()
assert.strictEqual((await x.db.prepare('SELECT COUNT(*)::int AS n FROM events_archive').get()).n, 0);
assert.strictEqual((await x.db.prepare('SELECT COUNT(*)::int AS n FROM events_archive').get()).n, 0);
const kept = await pub();
const kept = await pub();
const p = await request(x.base, 'GET', `/api/v1/events?topic=live.tier.zero&after=${old.cursor}`, { token: reader });
const p = await request(x.base, 'GET', `/api/v1/events?topic=live.tier.zero&after=${old.cursor}`, { token: reader });
- assert.deepStrictEqual(p.body.gap, { from_seq: old.seq + 1, to_seq: kept.seq - 1 });
+ assert.deepStrictEqual(p.body.gap, { from_seq: cursor.decode(old.cursor).seq + 1, to_seq: cursor.decode(kept.cursor).seq - 1 });
assert.strictEqual((await request(x.base, 'GET', `/api/v1/events/${old2.event_id}`, { token: reader })).status, 404);
assert.strictEqual((await request(x.base, 'GET', `/api/v1/events/${old2.event_id}`, { token: reader })).status, 404);
const lim = limitsOf(x.config).limits.find(l => l.id === 'replay_retention_days');
const lim = limitsOf(x.config).limits.find(l => l.id === 'replay_retention_days');
assert.deepStrictEqual([lim.production, lim.sandbox], [0, 0]);
assert.deepStrictEqual([lim.production, lim.sandbox], [0, 0]);
@@ -1,9 +1,8 @@
@@ -1,9 +1,8 @@
'use strict';
'use strict';
/**
/**
* ADR-042 decision 7: consumer positions are cursors only. A cursor is `c1.<epoch>.<base64url(seq)>`;
* ADR-042 decision 7: consumer positions are cursors only. A cursor is `c1.<epoch>.<base64url(seq)>`;
- * event answers retain seq and carry a cursor, pull accepts `after=<cursor>`, SSE resumes from
- * cursor IDs, and checkpoints store and return opaque cursor strings. Unchanged publish and single
- * event answers are validated against the pinned openvibe-contracts.
+ * event answers carry a cursor without seq, pull accepts `after=<cursor>`, SSE resumes from
+ * cursor IDs, and checkpoints store and return opaque cursor strings.
*/
*/
const assert = require('assert');
const assert = require('assert');
const { validate } = require('openvibe-contracts');
const { validate } = require('openvibe-contracts');
@@ -43,22 +42,23 @@ t('publish and read answers carry cursors', async () => {
@@ -43,22 +42,23 @@ t('publish and read answers carry cursors', async () => {
const one = await publish();
const one = await publish();
const pv = validate('events.publish-result@1', one);
const pv = validate('events.publish-result@1', one);
assert.ok(pv.valid, JSON.stringify(pv.errors));
assert.ok(pv.valid, JSON.stringify(pv.errors));
- assert.deepStrictEqual(cursor.decode(one.cursor), { seq: one.seq, epoch });
+ assert.deepStrictEqual(cursor.decode(one.cursor), { seq: (await h.store.getEvent(one.event_id)).seq, epoch });
const batch = await request(h.base, 'POST', '/api/v1/events', { token: live, body: { events: [envelope('live', { event_type: 'live.vod.ready' }), envelope('live', { event_type: 'live.vod.ready' })] } });
const batch = await request(h.base, 'POST', '/api/v1/events', { token: live, body: { events: [envelope('live', { event_type: 'live.vod.ready' }), envelope('live', { event_type: 'live.vod.ready' })] } });
assert.ok(validate('events.publish-result@1', batch.body).valid);
assert.ok(validate('events.publish-result@1', batch.body).valid);
- for (const r of batch.body.results) assert.deepStrictEqual(cursor.decode(r.cursor), { seq: r.seq, epoch });
+ for (const r of batch.body.results) assert.deepStrictEqual(cursor.decode(r.cursor), { seq: (await h.store.getEvent(r.event_id)).seq, epoch });
const page = await request(h.base, 'GET', '/api/v1/events?topic=live.vod.*', { token: reader });
const page = await request(h.base, 'GET', '/api/v1/events?topic=live.vod.*', { token: reader });
assert.ok(typeof page.body.next_cursor === 'string' && typeof page.body.latest_cursor === 'string');
assert.ok(typeof page.body.next_cursor === 'string' && typeof page.body.latest_cursor === 'string');
assert.ok(page.body.events.length >= 3);
assert.ok(page.body.events.length >= 3);
- for (const e of page.body.events) assert.deepStrictEqual(cursor.decode(e.cursor), { seq: e.seq, epoch });
- assert.deepStrictEqual(cursor.decode(page.body.next_cursor), { seq: page.body.events.at(-1).seq, epoch });
+ for (const e of page.body.events) assert.deepStrictEqual(cursor.decode(e.cursor), { seq: (await h.store.getEvent(e.event.event_id)).seq, epoch });
+ assert.strictEqual(page.body.next_cursor, page.body.events.at(-1).cursor);
assert.ok(!('next_after_seq' in page.body) && !('latest_seq' in page.body));
assert.ok(!('next_after_seq' in page.body) && !('latest_seq' in page.body));
const single = await request(h.base, 'GET', `/api/v1/events/${one.event_id}`, { token: reader });
const single = await request(h.base, 'GET', `/api/v1/events/${one.event_id}`, { token: reader });
assert.ok(validate('events.read-result@1', single.body).valid);
assert.ok(validate('events.read-result@1', single.body).valid);
- assert.deepStrictEqual(cursor.decode(single.body.cursor), { seq: single.body.seq, epoch });
+ assert.strictEqual(single.body.cursor, one.cursor);
+ assert.ok(!Object.hasOwn(single.body, 'seq'));
});
});
t('pull accepts after=<cursor>; next_cursor round-trips a page and after_seq is refused', async () => {
t('pull accepts after=<cursor>; next_cursor round-trips a page and after_seq is refused', async () => {
@@ -68,8 +68,8 @@ t('pull accepts after=<cursor>; next_cursor round-trips a page and after_seq is
@@ -68,8 +68,8 @@ t('pull accepts after=<cursor>; next_cursor round-trips a page and after_seq is
assert.strictEqual(r.body.next_cursor, first.cursor);
assert.strictEqual(r.body.next_cursor, first.cursor);
r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after=${encodeURIComponent(r.body.next_cursor)}`, { token: reader });
r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after=${encodeURIComponent(r.body.next_cursor)}`, { token: reader });
assert.strictEqual(r.status, 200, r.text);
assert.strictEqual(r.status, 200, r.text);
- assert.ok(r.body.events.length >= 1 && r.body.events.every(e => e.seq > first.seq));
- for (const value of ['0', String(first.seq)]) {
+ assert.ok(r.body.events.length >= 1 && r.body.events.every(e => cursor.decode(e.cursor).seq > cursor.decode(first.cursor).seq));
+ for (const value of ['0', String(cursor.decode(first.cursor).seq)]) {
r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after_seq=${value}`, { token: reader });
r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after_seq=${value}`, { token: reader });
assert.strictEqual(r.status, 400);
assert.strictEqual(r.status, 400);
assert.strictEqual(r.body.code, 'events.bad_request');
assert.strictEqual(r.body.code, 'events.bad_request');
@@ -104,22 +104,24 @@ t('SSE id is the cursor; Last-Event-ID resumes from a cursor, and a bare seq sta
@@ -104,22 +104,24 @@ t('SSE id is the cursor; Last-Event-ID resumes from a cursor, and a bare seq sta
const s1 = await publish({ visibility: 'public', event_type: 'live.stream.a' });
const s1 = await publish({ visibility: 'public', event_type: 'live.stream.a' });
await c1.waitFor(x => x.events().length === 1);
await c1.waitFor(x => x.events().length === 1);
const msg = c1.messages.find(m => !m.event);
const msg = c1.messages.find(m => !m.event);
- assert.deepStrictEqual(cursor.decode(msg.id), { seq: s1.seq, epoch });
+ assert.strictEqual(msg.id, s1.cursor);
+ assert.strictEqual(cursor.decode(msg.id).epoch, epoch);
+ assert.ok(!Object.hasOwn(JSON.parse(msg.data), 'seq'));
c1.close();
c1.close();
const s2 = await publish({ visibility: 'public', event_type: 'live.stream.b' });
const s2 = await publish({ visibility: 'public', event_type: 'live.stream.b' });
const byCursor = await sse(h.base, `/realtime/stream?topics=live.stream.*&last_event_id=${encodeURIComponent(msg.id)}`, { headers: { Authorization: `Bearer ${readerSvc}` } });
const byCursor = await sse(h.base, `/realtime/stream?topics=live.stream.*&last_event_id=${encodeURIComponent(msg.id)}`, { headers: { Authorization: `Bearer ${readerSvc}` } });
await byCursor.waitFor(x => x.events().length === 1);
await byCursor.waitFor(x => x.events().length === 1);
- assert.deepStrictEqual(byCursor.events().map(e => e.seq), [s2.seq]);
+ assert.deepStrictEqual(byCursor.events().map(e => e.seq), [cursor.decode(s2.cursor).seq]);
assert.deepStrictEqual(byCursor.gaps(), []);
assert.deepStrictEqual(byCursor.gaps(), []);
byCursor.close();
byCursor.close();
- const bySeq = await sse(h.base, `/realtime/stream?topics=live.stream.*&last_event_id=${s1.seq}`, { headers: { Authorization: `Bearer ${readerSvc}` } });
+ const bySeq = await sse(h.base, `/realtime/stream?topics=live.stream.*&last_event_id=${cursor.decode(s1.cursor).seq}`, { headers: { Authorization: `Bearer ${readerSvc}` } });
assert.deepStrictEqual(bySeq.events(), [], 'a bare number does not replay missed events');
assert.deepStrictEqual(bySeq.events(), [], 'a bare number does not replay missed events');
assert.deepStrictEqual(bySeq.gaps(), []);
assert.deepStrictEqual(bySeq.gaps(), []);
const s3 = await publish({ visibility: 'public', event_type: 'live.stream.c' });
const s3 = await publish({ visibility: 'public', event_type: 'live.stream.c' });
await bySeq.waitFor(x => x.events().length === 1);
await bySeq.waitFor(x => x.events().length === 1);
- assert.deepStrictEqual(bySeq.events().map(e => e.seq), [s3.seq]);
+ assert.deepStrictEqual(bySeq.events().map(e => e.seq), [cursor.decode(s3.cursor).seq]);
bySeq.close();
bySeq.close();
});
});
@@ -134,7 +136,7 @@ t('SSE reports a gap when the cursor belongs to another epoch', async () => {
@@ -134,7 +136,7 @@ t('SSE reports a gap when the cursor belongs to another epoch', async () => {
await again.waitFor(x => x.gaps().length >= 1);
await again.waitFor(x => x.gaps().length >= 1);
assert.strictEqual(again.gaps()[0].reason, 'epoch');
assert.strictEqual(again.gaps()[0].reason, 'epoch');
// The retained matching event is replayed after the gap.
// The retained matching event is replayed after the gap.
- await again.waitFor(x => x.events().some(e => e.seq === s.seq));
+ await again.waitFor(x => x.events().some(e => e.seq === cursor.decode(s.cursor).seq));
again.close();
again.close();
});
});
@@ -2,6 +2,7 @@
@@ -2,6 +2,7 @@
const assert = require('assert');
const assert = require('assert');
const crypto = require('crypto');
const crypto = require('crypto');
const { verifyDelivery, verifyDeliveryV2, signV2, sign } = require('../lib/client');
const { verifyDelivery, verifyDeliveryV2, signV2, sign } = require('../lib/client');
+const cursor = require('../server/cursor');
const { boot, request, serviceToken, envelope, subscriber, suite, sleep } = require('./helpers');
const { boot, request, serviceToken, envelope, subscriber, suite, sleep } = require('./helpers');
const t = suite('delivery');
const t = suite('delivery');
@@ -21,7 +22,7 @@ async function publish(env) {
@@ -21,7 +22,7 @@ async function publish(env) {
t('signV2 / verifyDeliveryV2: HMAC of "<t>.<raw body>", ±300 s, constant-time', async () => {
t('signV2 / verifyDeliveryV2: HMAC of "<t>.<raw body>", ±300 s, constant-time', async () => {
const secret = `whsec_${'cd'.repeat(32)}`;
const secret = `whsec_${'cd'.repeat(32)}`;
- const raw = Buffer.from(JSON.stringify({ event: { event_id: 'evt_1' }, seq: 3 }));
+ const raw = Buffer.from(JSON.stringify({ event: { event_id: 'evt_1' } }));
const now = 1790000000000;
const now = 1790000000000;
const t = Math.floor(now / 1000);
const t = Math.floor(now / 1000);
const v2 = signV2(raw, secret, t);
const v2 = signV2(raw, secret, t);
@@ -36,7 +37,7 @@ t('signV2 / verifyDeliveryV2: HMAC of "<t>.<raw body>", ±300 s, constant-time',
@@ -36,7 +37,7 @@ t('signV2 / verifyDeliveryV2: HMAC of "<t>.<raw body>", ±300 s, constant-time',
assert.ok(!verifyDeliveryV2(raw, headers, secret, { now: now - 301000 }), 'too far in the future');
assert.ok(!verifyDeliveryV2(raw, headers, secret, { now: now - 301000 }), 'too far in the future');
assert.ok(verifyDeliveryV2(raw, headers, secret, { now: now + 3600000, toleranceSec: 3601 }));
assert.ok(verifyDeliveryV2(raw, headers, secret, { now: now + 3600000, toleranceSec: 3601 }));
assert.ok(!verifyDeliveryV2(raw, headers, 'whsec_other', { now }));
assert.ok(!verifyDeliveryV2(raw, headers, 'whsec_other', { now }));
- assert.ok(!verifyDeliveryV2(Buffer.from(raw.toString().replace('3', '4')), headers, secret, { now }), 'body changed');
+ assert.ok(!verifyDeliveryV2(Buffer.from(raw.toString().replace('evt_1', 'evt_2')), headers, secret, { now }), 'body changed');
assert.ok(!verifyDeliveryV2(raw, { ...headers, 'x-openvibe-signature-v2': v2.replace(`t=${t}`, `t=${t + 1}`) }, secret, { now }), 'timestamp changed');
assert.ok(!verifyDeliveryV2(raw, { ...headers, 'x-openvibe-signature-v2': v2.replace(`t=${t}`, `t=${t + 1}`) }, secret, { now }), 'timestamp changed');
assert.ok(!verifyDeliveryV2(raw, { ...headers, 'x-openvibe-timestamp': String(t + 1) }, secret, { now }), 'X-OpenVibe-Timestamp must match t');
assert.ok(!verifyDeliveryV2(raw, { ...headers, 'x-openvibe-timestamp': String(t + 1) }, secret, { now }), 'X-OpenVibe-Timestamp must match t');
assert.ok(verifyDeliveryV2(raw, { 'x-openvibe-signature-v2': `${v2},v2=${'0'.repeat(64)}` }, secret, { now }), 'one of several v2 values');
assert.ok(verifyDeliveryV2(raw, { 'x-openvibe-signature-v2': `${v2},v2=${'0'.repeat(64)}` }, secret, { now }), 'one of several v2 values');
@@ -136,7 +137,7 @@ t('success: signed POST (v2 only), verified by verifyDeliveryV2; trace propagate
@@ -136,7 +137,7 @@ t('success: signed POST (v2 only), verified by verifyDeliveryV2; trace propagate
const stub = await subscriber();
const stub = await subscriber();
const sub = (await subscribe(media, { topic_pattern: 'live.stream.*', endpoint: stub.url })).body;
const sub = (await subscribe(media, { topic_pattern: 'live.stream.*', endpoint: stub.url })).body;
const env = envelope('live', { trace_id: '4bf92f3577b34da6a3ce929d0e0e4736' });
const env = envelope('live', { trace_id: '4bf92f3577b34da6a3ce929d0e0e4736' });
- const { seq } = await publish(env);
+ await publish(env);
assert.strictEqual((await h.store.getDelivery(env.event_id, sub.id)).status, 'pending', 'persisted before delivery');
assert.strictEqual((await h.store.getDelivery(env.event_id, sub.id)).status, 'pending', 'persisted before delivery');
await h.worker.drain();
await h.worker.drain();
assert.strictEqual(stub.calls.length, 1);
assert.strictEqual(stub.calls.length, 1);
@@ -154,7 +155,8 @@ t('success: signed POST (v2 only), verified by verifyDeliveryV2; trace propagate
@@ -154,7 +155,8 @@ t('success: signed POST (v2 only), verified by verifyDeliveryV2; trace propagate
assert.strictEqual(call.headers['x-openvibe-event-id'], env.event_id);
assert.strictEqual(call.headers['x-openvibe-event-id'], env.event_id);
assert.match(call.headers.traceparent, /^00-4bf92f3577b34da6a3ce929d0e0e4736-[0-9a-f]{16}-01$/);
assert.match(call.headers.traceparent, /^00-4bf92f3577b34da6a3ce929d0e0e4736-[0-9a-f]{16}-01$/);
assert.deepStrictEqual(call.body.event.payload, env.payload);
assert.deepStrictEqual(call.body.event.payload, env.payload);
- assert.strictEqual(call.body.seq, seq);
+ assert.strictEqual(call.body.seq, undefined);
+ assert.strictEqual(call.headers['x-openvibe-seq'], undefined);
const d = await h.store.getDelivery(env.event_id, sub.id);
const d = await h.store.getDelivery(env.event_id, sub.id);
assert.strictEqual(d.status, 'delivered');
assert.strictEqual(d.status, 'delivered');
assert.strictEqual(d.attempt, 1);
assert.strictEqual(d.attempt, 1);
@@ -216,14 +218,14 @@ t('replay from_seq re-sends retained matching events (new subscription catches u
@@ -216,14 +218,14 @@ t('replay from_seq re-sends retained matching events (new subscription catches u
const e1 = envelope('live', { event_type: 'live.history.a' });
const e1 = envelope('live', { event_type: 'live.history.a' });
const e2 = envelope('live', { event_type: 'live.history.b' });
const e2 = envelope('live', { event_type: 'live.history.b' });
const e3 = envelope('live', { event_type: 'live.other.c' });
const e3 = envelope('live', { event_type: 'live.other.c' });
- const { seq } = await publish(e1);
+ const { cursor: firstCursor } = await publish(e1);
await publish(e2);
await publish(e2);
await publish(e3);
await publish(e3);
const stub = await subscriber();
const stub = await subscriber();
const sub = (await subscribe(media, { topic_pattern: 'live.history.*', endpoint: stub.url })).body;
const sub = (await subscribe(media, { topic_pattern: 'live.history.*', endpoint: stub.url })).body;
await h.worker.drain();
await h.worker.drain();
assert.strictEqual(stub.calls.length, 0, 'subscriptions only get events published after they exist');
assert.strictEqual(stub.calls.length, 0, 'subscriptions only get events published after they exist');
- const rp = await request(h.base, 'POST', '/api/v1/deliveries/replay', { token: admin, body: { subscription_id: sub.id, from_seq: seq } });
+ const rp = await request(h.base, 'POST', '/api/v1/deliveries/replay', { token: admin, body: { subscription_id: sub.id, from_seq: cursor.decode(firstCursor).seq } });
assert.strictEqual(rp.body.queued, 2);
assert.strictEqual(rp.body.queued, 2);
await h.worker.drain();
await h.worker.drain();
assert.deepStrictEqual(stub.calls.map(c => c.body.event.event_id), [e1.event_id, e2.event_id]);
assert.deepStrictEqual(stub.calls.map(c => c.body.event.event_id), [e1.event_id, e2.event_id]);
@@ -8,6 +8,7 @@ const assert = require('assert');
@@ -8,6 +8,7 @@ const assert = require('assert');
const crypto = require('crypto');
const crypto = require('crypto');
const { ids, serviceAuth } = require('openvibe-contracts');
const { ids, serviceAuth } = require('openvibe-contracts');
const apps = require('../server/apps');
const apps = require('../server/apps');
+const cursor = require('../server/cursor');
const { ISSUER, privateKey, boot, request, appToken, envelope, suite } = require('./helpers');
const { ISSUER, privateKey, boot, request, appToken, envelope, suite } = require('./helpers');
const t = suite('export-tokens');
const t = suite('export-tokens');
@@ -38,7 +39,7 @@ t('boot and publish as the project\'s app, in both environments (and another pro
@@ -38,7 +39,7 @@ t('boot and publish as the project\'s app, in both environments (and another pro
body: envelope('x', { source: apps.appSource(appId), event_type: `app.${apps.projectKey(project)}.order.n${n}`, actor: { type: 'app', id: appId }, visibility: 'internal' }),
body: envelope('x', { source: apps.appSource(appId), event_type: `app.${apps.projectKey(project)}.order.n${n}`, actor: { type: 'app', id: appId }, visibility: 'internal' }),
});
});
assert.strictEqual(r.status, 201, r.text);
assert.strictEqual(r.status, 201, r.text);
- return r.body.seq;
+ return cursor.decode(r.body.cursor).seq;
};
};
for (let i = 0; i < 5; i++) seqs.production.push(await publish(P, 'production', i));
for (let i = 0; i < 5; i++) seqs.production.push(await publish(P, 'production', i));
for (let i = 0; i < 2; i++) seqs.sandbox.push(await publish(P, 'sandbox', i));
for (let i = 0; i < 2; i++) seqs.sandbox.push(await publish(P, 'sandbox', i));
@@ -51,14 +52,14 @@ t('an export token pulls the project\'s events of its environment, page by page
@@ -51,14 +52,14 @@ t('an export token pulls the project\'s events of its environment, page by page
for (let page = 0; page < 10; page++) {
for (let page = 0; page < 10; page++) {
const r = await pull(exportToken('production'), after, 2);
const r = await pull(exportToken('production'), after, 2);
assert.strictEqual(r.status, 200, r.text);
assert.strictEqual(r.status, 200, r.text);
- got.push(...r.body.events.map(e => e.seq));
+ got.push(...r.body.events.map(e => cursor.decode(e.cursor).seq));
if (r.body.next_cursor === r.body.latest_cursor || r.body.next_cursor === after) break;
if (r.body.next_cursor === r.body.latest_cursor || r.body.next_cursor === after) break;
after = r.body.next_cursor;
after = r.body.next_cursor;
}
}
assert.deepStrictEqual(got, seqs.production);
assert.deepStrictEqual(got, seqs.production);
const s = await pull(exportToken('sandbox'));
const s = await pull(exportToken('sandbox'));
assert.strictEqual(s.status, 200, s.text);
assert.strictEqual(s.status, 200, s.text);
- assert.deepStrictEqual(s.body.events.map(e => e.seq), seqs.sandbox, 'the sandbox token sees the sandbox only');
+ assert.deepStrictEqual(s.body.events.map(e => cursor.decode(e.cursor).seq), seqs.sandbox, 'the sandbox token sees the sandbox only');
});
});
t('it reads nothing else and writes nothing', async () => {
t('it reads nothing else and writes nothing', async () => {
@@ -220,7 +220,8 @@ t('valkey-v1 QUEUE: two instances, exactly once, per-key order, fast; a down car
@@ -220,7 +220,8 @@ t('valkey-v1 QUEUE: two instances, exactly once, per-key order, fast; a down car
await putPolicy(a.base, 'live.q.*', keyed('subject'));
await putPolicy(a.base, 'live.q.*', keyed('subject'));
const KEYS = ['K0', 'K1', 'K2', 'K3', 'K4'];
const KEYS = ['K0', 'K1', 'K2', 'K3', 'K4'];
const byId = new Map();
const byId = new Map();
- const firstSeq = new Map(KEYS.map((k) => [k, []]));
+ const firstIds = new Map(KEYS.map((k) => [k, []]));
+ const expectedIds = new Map(KEYS.map((k) => [k, []]));
stub.calls.length = 0;
stub.calls.length = 0;
const seen = new Map();
const seen = new Map();
// Publish 50 events over 5 keys, recording the publish time per event id.
// Publish 50 events over 5 keys, recording the publish time per event id.
@@ -229,6 +230,7 @@ t('valkey-v1 QUEUE: two instances, exactly once, per-key order, fast; a down car
@@ -229,6 +230,7 @@ t('valkey-v1 QUEUE: two instances, exactly once, per-key order, fast; a down car
for (const k of KEYS) {
for (const k of KEYS) {
const env2 = envelope('live', { event_type: 'live.q.x', subject: { type: 'stream', id: k } });
const env2 = envelope('live', { event_type: 'live.q.x', subject: { type: 'stream', id: k } });
publishedAt.set(env2.event_id, Date.now());
publishedAt.set(env2.event_id, Date.now());
+ expectedIds.get(k).push(env2.event_id);
await publish(a.base, env2);
await publish(a.base, env2);
}
}
}
}
@@ -236,13 +238,13 @@ t('valkey-v1 QUEUE: two instances, exactly once, per-key order, fast; a down car
@@ -236,13 +238,13 @@ t('valkey-v1 QUEUE: two instances, exactly once, per-key order, fast; a down car
await waitFor(() => {
await waitFor(() => {
for (const c of stub.calls) {
for (const c of stub.calls) {
const id = c.body.event.event_id;
const id = c.body.event.event_id;
- if (!seen.has(id)) { seen.set(id, { at: Date.now(), seq: c.body.seq }); byId.set(id, (byId.get(id) || 0) + 1); const L = firstSeq.get(c.body.event.subject.id); if (L && !L.includes(c.body.seq)) L.push(c.body.seq); }
+ if (!seen.has(id)) { seen.set(id, { at: Date.now() }); byId.set(id, (byId.get(id) || 0) + 1); const L = firstIds.get(c.body.event.subject.id); if (L && !L.includes(id)) L.push(id); }
}
}
return seen.size >= 50;
return seen.size >= 50;
}, 9000);
}, 9000);
assert.strictEqual(byId.size, 50, 'every event of every key was delivered');
assert.strictEqual(byId.size, 50, 'every event of every key was delivered');
assert.ok([...byId.values()].every((n) => n === 1), 'each delivery was sent exactly once');
assert.ok([...byId.values()].every((n) => n === 1), 'each delivery was sent exactly once');
- for (const k of KEYS) { const L = firstSeq.get(k); assert.deepStrictEqual(L, [...L].sort((x, y) => x - y), `${k}: first delivery per seq is in order`); }
+ for (const k of KEYS) assert.deepStrictEqual(firstIds.get(k), expectedIds.get(k), `${k}: first deliveries follow publish order`);
const lats = [...publishedAt.entries()].map(([id, at]) => seen.get(id).at - at).sort((x, y) => x - y);
const lats = [...publishedAt.entries()].map(([id, at]) => seen.get(id).at - at).sort((x, y) => x - y);
const median = lats[Math.floor(lats.length / 2)];
const median = lats[Math.floor(lats.length / 2)];
assert.ok(median < 5000, `median publish->send latency ${median}ms is below the 5 s poll interval`);
assert.ok(median < 5000, `median publish->send latency ${median}ms is below the 5 s poll interval`);
@@ -252,7 +254,7 @@ t('valkey-v1 QUEUE: two instances, exactly once, per-key order, fast; a down car
@@ -252,7 +254,7 @@ t('valkey-v1 QUEUE: two instances, exactly once, per-key order, fast; a down car
const before = seen.size;
const before = seen.size;
for (let i = 0; i < 5; i++) await publish(a.base, envelope('live', { event_type: 'live.q.x', subject: { type: 'stream', id: 'K9' } }));
for (let i = 0; i < 5; i++) await publish(a.base, envelope('live', { event_type: 'live.q.x', subject: { type: 'stream', id: 'K9' } }));
await waitFor(() => {
await waitFor(() => {
- for (const c of stub.calls) { const id = c.body.event.event_id; if (!seen.has(id)) { seen.set(id, { at: Date.now(), seq: c.body.seq }); byId.set(id, (byId.get(id) || 0) + 1); } }
+ for (const c of stub.calls) { const id = c.body.event.event_id; if (!seen.has(id)) { seen.set(id, { at: Date.now() }); byId.set(id, (byId.get(id) || 0) + 1); } }
return seen.size >= before + 5;
return seen.size >= before + 5;
}, 12000);
}, 12000);
assert.ok([...byId.values()].every((n) => n === 1), 'nothing was lost or doubled when the carrier went down');
assert.ok([...byId.values()].every((n) => n === 1), 'nothing was lost or doubled when the carrier went down');
@@ -148,7 +148,7 @@ async function subscriber(respond = () => 204) {
@@ -148,7 +148,7 @@ async function subscriber(respond = () => 204) {
};
};
}
}
-/** Open an SSE stream and parse it. */
+/** Open an SSE stream and parse it; events() derives a test-only seq from each message id cursor. */
function sse(base, pathAndQuery, { headers = {} } = {}) {
function sse(base, pathAndQuery, { headers = {} } = {}) {
return new Promise((resolve, reject) => {
return new Promise((resolve, reject) => {
const messages = [];
const messages = [];
@@ -161,7 +161,7 @@ function sse(base, pathAndQuery, { headers = {} } = {}) {
@@ -161,7 +161,7 @@ function sse(base, pathAndQuery, { headers = {} } = {}) {
headers: res.headers,
headers: res.headers,
messages,
messages,
body: '',
body: '',
- events: () => messages.filter(m => !m.event || m.event === 'message').map(m => JSON.parse(m.data)),
+ events: () => messages.filter(m => !m.event || m.event === 'message').map(m => ({ ...JSON.parse(m.data), seq: require('../server/cursor').decode(m.id).seq })),
gaps: () => messages.filter(m => m.event === 'gap').map(m => JSON.parse(m.data)),
gaps: () => messages.filter(m => m.event === 'gap').map(m => JSON.parse(m.data)),
comments: [],
comments: [],
waitFor(pred, ms = 2000) {
waitFor(pred, ms = 2000) {
@@ -147,34 +147,39 @@ t('per-key order under retries: a successor waits for its head, another key goes
@@ -147,34 +147,39 @@ t('per-key order under retries: a successor waits for its head, another key goes
await h.store.setSubscriptionEnabled(sub.id, false);
await h.store.setSubscriptionEnabled(sub.id, false);
});
});
-t('per-key order across two workers with a lease expiry, and never two distinct seqs of one key in flight', async () => {
+t('per-key order across two workers with a lease expiry, and never two distinct events of one key in flight', async () => {
const stub = await subscriber(() => 204);
const stub = await subscriber(() => 204);
const sub = await subscribe('ord3', 'live.ord3.*', stub.url);
const sub = await subscribe('ord3', 'live.ord3.*', stub.url);
await putPolicy('live.ord3.*', keyed('subject'));
await putPolicy('live.ord3.*', keyed('subject'));
const keys = ['K0', 'K1', 'K2'];
const keys = ['K0', 'K1', 'K2'];
const byId = new Map(); // event_id -> deliveries
const byId = new Map(); // event_id -> deliveries
- const firstSeqs = new Map(keys.map((k) => [k, []])); // key -> seqs in first-delivery order
- const inflight = new Map(); // key -> Set(seq)
- let distinctPeak = 0; // the most distinct seqs of one key ever in flight at once
+ const firstIds = new Map(keys.map((k) => [k, []])); // key -> event ids in first-delivery order
+ const inflight = new Map(); // key -> Set(event_id)
+ let distinctPeak = 0; // the most distinct events of one key ever in flight at once
const fetchImpl = async (url, init) => {
const fetchImpl = async (url, init) => {
- const { event, seq } = JSON.parse(init.body);
+ const { event } = JSON.parse(init.body);
const k = event.subject.id;
const k = event.subject.id;
byId.set(event.event_id, (byId.get(event.event_id) || 0) + 1);
byId.set(event.event_id, (byId.get(event.event_id) || 0) + 1);
const set = inflight.get(k) || new Set();
const set = inflight.get(k) || new Set();
inflight.set(k, set);
inflight.set(k, set);
- set.add(seq);
+ set.add(event.event_id);
distinctPeak = Math.max(distinctPeak, set.size);
distinctPeak = Math.max(distinctPeak, set.size);
await sleep(2);
await sleep(2);
- set.delete(seq);
- const list = firstSeqs.get(k);
- if (!list.includes(seq)) list.push(seq);
+ set.delete(event.event_id);
+ const list = firstIds.get(k);
+ if (!list.includes(event.event_id)) list.push(event.event_id);
return new Response(null, { status: 204 });
return new Response(null, { status: 204 });
};
};
- // 30 events over 3 keys, interleaved so the global seq order is not the per-key order.
+ // 30 events over 3 keys, interleaved across keys.
+ const expectedIds = new Map(keys.map((k) => [k, []]));
for (let i = 0; i < 10; i++) {
for (let i = 0; i < 10; i++) {
- for (const k of keys) await publish(envelope('live', { event_type: 'live.ord3.x', subject: { type: 'stream', id: k } }));
+ for (const k of keys) {
+ const env = envelope('live', { event_type: 'live.ord3.x', subject: { type: 'stream', id: k } });
+ expectedIds.get(k).push(env.event_id);
+ await publish(env);
+ }
}
}
const a = createWorker({ store: h.store, config: h.config, clock: h.clock, fetchImpl, log: silent });
const a = createWorker({ store: h.store, config: h.config, clock: h.clock, fetchImpl, log: silent });
@@ -189,10 +194,10 @@ t('per-key order across two workers with a lease expiry, and never two distinct
@@ -189,10 +194,10 @@ t('per-key order across two workers with a lease expiry, and never two distinct
assert.strictEqual(byId.size, 30, 'every event of every key was delivered');
assert.strictEqual(byId.size, 30, 'every event of every key was delivered');
assert.ok([...byId.values()].every((n) => n >= 1), 'every event delivered at least once');
assert.ok([...byId.values()].every((n) => n >= 1), 'every event delivered at least once');
for (const k of keys) {
for (const k of keys) {
- const list = firstSeqs.get(k);
- assert.deepStrictEqual(list, [...list].sort((x, y) => x - y), `${k}: the first delivery of each seq is in seq order (${list})`);
+ const list = firstIds.get(k);
+ assert.deepStrictEqual(list, expectedIds.get(k), `${k}: first deliveries follow publish order`);
}
}
- assert.strictEqual(distinctPeak, 1, 'never two distinct seqs of one key in flight at once');
+ assert.strictEqual(distinctPeak, 1, 'never two distinct events of one key in flight at once');
await deletePolicy('live.ord3.*');
await deletePolicy('live.ord3.*');
await stub.close();
await stub.close();
@@ -205,17 +210,17 @@ t('max_inflight = 2: two keys run together, never two of one key, never three at
@@ -205,17 +210,17 @@ t('max_inflight = 2: two keys run together, never two of one key, never three at
await putPolicy('live.ord4.*', keyed('subject'));
await putPolicy('live.ord4.*', keyed('subject'));
await h.store.db.prepare('UPDATE subscriptions SET max_inflight = 2 WHERE id = ?').run(sub.id);
await h.store.db.prepare('UPDATE subscriptions SET max_inflight = 2 WHERE id = ?').run(sub.id);
- const inflight = []; // { key, seq } currently being sent
+ const inflight = []; // { key, event id } currently being sent
let totalPeak = 0;
let totalPeak = 0;
let keyPeak = 0;
let keyPeak = 0;
const fetchImpl = async (url, init) => {
const fetchImpl = async (url, init) => {
- const { event, seq } = JSON.parse(init.body);
+ const { event } = JSON.parse(init.body);
const k = event.subject.id;
const k = event.subject.id;
- inflight.push({ k, seq });
+ inflight.push({ k, id: event.event_id });
totalPeak = Math.max(totalPeak, inflight.length);
totalPeak = Math.max(totalPeak, inflight.length);
keyPeak = Math.max(keyPeak, inflight.filter((x) => x.k === k).length);
keyPeak = Math.max(keyPeak, inflight.filter((x) => x.k === k).length);
await sleep(4);
await sleep(4);
- inflight.splice(inflight.findIndex((x) => x.k === k && x.seq === seq), 1);
+ inflight.splice(inflight.findIndex((x) => x.k === k && x.id === event.event_id), 1);
return new Response(null, { status: 204 });
return new Response(null, { status: 204 });
};
};
@@ -97,7 +97,7 @@ t('outbox: enqueue inside a rolled-back transaction publishes nothing; committed
@@ -97,7 +97,7 @@ t('outbox: enqueue inside a rolled-back transaction publishes nothing; committed
r = await outbox.flush();
r = await outbox.flush();
assert.strictEqual(r.sent, 1);
assert.strictEqual(r.sent, 1);
assert.strictEqual(await countIn(committed.event_id), 1, 'still exactly one event');
assert.strictEqual(await countIn(committed.event_id), 1, 'still exactly one event');
- assert.ok(Number((await db.one('SELECT seq FROM event_outbox WHERE event_id = $1', [committed.event_id])).seq) > 0);
+ assert.strictEqual((await db.one('SELECT seq FROM event_outbox WHERE event_id = $1', [committed.event_id])).seq, null, 'Events hands out no sequence number (ADR-042): nothing is recorded');
} finally { await db.close(); }
} finally { await db.close(); }
});
});
@@ -28,17 +28,18 @@ t('a family grant (events.*) covers events.event.publish', async () => {
@@ -28,17 +28,18 @@ t('a family grant (events.*) covers events.event.publish', async () => {
assert.strictEqual(r.status, 201);
assert.strictEqual(r.status, 201);
});
});
-t('valid envelope -> 201 with a seq; defaults filled', async () => {
+t('valid envelope -> 201 with a cursor; defaults filled', async () => {
const env = envelope('live');
const env = envelope('live');
delete env.priority; delete env.visibility;
delete env.priority; delete env.visibility;
const r = await request(h.base, 'POST', '/api/v1/events', { token: live, body: env, headers: { traceparent: '00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01' } });
const r = await request(h.base, 'POST', '/api/v1/events', { token: live, body: env, headers: { traceparent: '00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01' } });
assert.strictEqual(r.status, 201, r.text);
assert.strictEqual(r.status, 201, r.text);
assert.strictEqual(r.body.event_id, env.event_id);
assert.strictEqual(r.body.event_id, env.event_id);
- assert.ok(Number.isInteger(r.body.seq) && r.body.seq > 0);
+ assert.ok(typeof r.body.cursor === 'string');
+ assert.ok(!Object.hasOwn(r.body, 'seq'));
assert.strictEqual(r.body.duplicate, false);
assert.strictEqual(r.body.duplicate, false);
assert.ok(validate('events.publish-result@1', r.body).valid, JSON.stringify(validate('events.publish-result@1', r.body).errors));
assert.ok(validate('events.publish-result@1', r.body).valid, JSON.stringify(validate('events.publish-result@1', r.body).errors));
- assert.deepStrictEqual(cursor.decode(r.body.cursor), { seq: r.body.seq, epoch: await h.store.epoch() });
const row = await h.store.getEvent(env.event_id);
const row = await h.store.getEvent(env.event_id);
+ assert.deepStrictEqual(cursor.decode(r.body.cursor), { seq: row.seq, epoch: await h.store.epoch() });
assert.strictEqual(row.priority, 'important');
assert.strictEqual(row.priority, 'important');
assert.strictEqual(row.visibility, 'internal');
assert.strictEqual(row.visibility, 'internal');
assert.strictEqual(row.trace_id, '0af7651916cd43dd8448eb211c80319c', 'trace taken from traceparent');
assert.strictEqual(row.trace_id, '0af7651916cd43dd8448eb211c80319c', 'trace taken from traceparent');
@@ -153,14 +154,14 @@ t('app principals cannot publish as a service', async () => {
@@ -153,14 +154,14 @@ t('app principals cannot publish as a service', async () => {
assert.strictEqual(r.body.code, 'token.invalid_claims');
assert.strictEqual(r.body.code, 'token.invalid_claims');
});
});
-t('idempotent repeat: same event_id -> 200, same seq, stored once', async () => {
+t('idempotent repeat: same event_id -> 200, same cursor, stored once', async () => {
const env = envelope();
const env = envelope();
const a = await request(h.base, 'POST', '/api/v1/events', { token: live, body: env });
const a = await request(h.base, 'POST', '/api/v1/events', { token: live, body: env });
const b = await request(h.base, 'POST', '/api/v1/events', { token: live, body: env });
const b = await request(h.base, 'POST', '/api/v1/events', { token: live, body: env });
assert.strictEqual(a.status, 201);
assert.strictEqual(a.status, 201);
assert.strictEqual(b.status, 200);
assert.strictEqual(b.status, 200);
assert.strictEqual(b.body.duplicate, true);
assert.strictEqual(b.body.duplicate, true);
- assert.strictEqual(b.body.seq, a.body.seq);
+ assert.strictEqual(b.body.cursor, a.body.cursor);
assert.strictEqual((await h.db.prepare('SELECT COUNT(*) AS n FROM events WHERE id = ?').get(env.event_id)).n, 1);
assert.strictEqual((await h.db.prepare('SELECT COUNT(*) AS n FROM events WHERE id = ?').get(env.event_id)).n, 1);
const c = await request(h.base, 'POST', '/api/v1/events', { token: live, body: { ...env, event_type: 'live.stream.ended' } });
const c = await request(h.base, 'POST', '/api/v1/events', { token: live, body: { ...env, event_type: 'live.stream.ended' } });
assert.strictEqual(c.status, 409);
assert.strictEqual(c.status, 409);
@@ -172,7 +173,10 @@ t('batch: atomic, <= 100, results per event', async () => {
@@ -172,7 +173,10 @@ t('batch: atomic, <= 100, results per event', async () => {
let r = await request(h.base, 'POST', '/api/v1/events', { token: live, body: { events } });
let r = await request(h.base, 'POST', '/api/v1/events', { token: live, body: { events } });
assert.strictEqual(r.status, 201);
assert.strictEqual(r.status, 201);
assert.strictEqual(r.body.results.length, 3);
assert.strictEqual(r.body.results.length, 3);
- assert.ok(r.body.results[0].seq < r.body.results[1].seq && r.body.results[1].seq < r.body.results[2].seq);
+ assert.ok(r.body.results.every(item => typeof item.cursor === 'string' && !Object.hasOwn(item, 'seq')));
+ assert.deepStrictEqual(r.body.results.map(item => item.event_id), events.map(event => event.event_id));
+ const positions = r.body.results.map(item => cursor.decode(item.cursor).seq);
+ assert.ok(positions[0] < positions[1] && positions[1] < positions[2]);
const before = await h.store.lastSeq();
const before = await h.store.lastSeq();
r = await request(h.base, 'POST', '/api/v1/events', { token: live, body: { events: [envelope(), envelope('media')] } });
r = await request(h.base, 'POST', '/api/v1/events', { token: live, body: { events: [envelope(), envelope('media')] } });
@@ -231,6 +235,7 @@ t('publish receipts outlive retention: a pruned id is still a duplicate', async
@@ -231,6 +235,7 @@ t('publish receipts outlive retention: a pruned id is still a duplicate', async
assert.strictEqual(r.status, 200);
assert.strictEqual(r.status, 200);
assert.strictEqual(r.body.duplicate, true);
assert.strictEqual(r.body.duplicate, true);
assert.strictEqual(r.body.pruned, true);
assert.strictEqual(r.body.pruned, true);
+ assert.ok(!Object.hasOwn(r.body, 'cursor') && !Object.hasOwn(r.body, 'seq'));
});
});
t('stop', async () => { await h.stop(); });
t('stop', async () => { await h.stop(); });
@@ -54,24 +54,27 @@ t('config: network-wide prefixes are shared, source namespaces stay owned', () =
@@ -54,24 +54,27 @@ t('config: network-wide prefixes are shared, source namespaces stay owned', () =
t('boot', async () => { h = await boot(); });
t('boot', async () => { h = await boot(); });
t('pull: cursor, topic filter, internal events included for services', async () => {
t('pull: cursor, topic filter, internal events included for services', async () => {
- const s1 = (await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope('live', { event_type: 'live.vod.ready' }) })).body.seq;
+ const c1 = (await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope('live', { event_type: 'live.vod.ready' }) })).body.cursor;
await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope('live', { event_type: 'live.chat.sent' }) });
await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope('live', { event_type: 'live.chat.sent' }) });
- const s3 = (await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope('live', { event_type: 'live.vod.deleted' }) })).body.seq;
+ const c3 = (await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope('live', { event_type: 'live.vod.deleted' }) })).body.cursor;
let r = await request(h.base, 'GET', '/api/v1/events?topic=live.vod.*&limit=1', { token: reader });
let r = await request(h.base, 'GET', '/api/v1/events?topic=live.vod.*&limit=1', { token: reader });
assert.strictEqual(r.status, 200, r.text);
assert.strictEqual(r.status, 200, r.text);
assert.ok(validate('events.read-result@1', r.body).valid, JSON.stringify(validate('events.read-result@1', r.body).errors));
assert.ok(validate('events.read-result@1', r.body).valid, JSON.stringify(validate('events.read-result@1', r.body).errors));
- for (const e of r.body.events) assert.deepStrictEqual(cursor.decode(e.cursor), { seq: e.seq, epoch: await h.store.epoch() });
- assert.strictEqual(r.body.next_cursor, await cursorAt(h, s1));
+ for (const e of r.body.events) {
+ assert.ok(!Object.hasOwn(e, 'seq'));
+ assert.deepStrictEqual(cursor.decode(e.cursor).epoch, await h.store.epoch());
+ }
+ assert.strictEqual(r.body.next_cursor, c1);
assert.ok(!('next_after_seq' in r.body) && !('latest_seq' in r.body));
assert.ok(!('next_after_seq' in r.body) && !('latest_seq' in r.body));
- assert.deepStrictEqual(r.body.events.map(e => e.seq), [s1]);
+ assert.deepStrictEqual(r.body.events.map(e => e.cursor), [c1]);
assert.strictEqual(r.body.events[0].event.visibility, 'internal');
assert.strictEqual(r.body.events[0].event.visibility, 'internal');
r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after=${r.body.next_cursor}`, { token: reader });
r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after=${r.body.next_cursor}`, { token: reader });
- assert.deepStrictEqual(r.body.events.map(e => e.seq), [s3]);
- assert.strictEqual(r.body.next_cursor, await cursorAt(h, s3));
+ assert.deepStrictEqual(r.body.events.map(e => e.cursor), [c3]);
+ assert.strictEqual(r.body.next_cursor, c3);
r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after=${r.body.next_cursor}`, { token: reader });
r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after=${r.body.next_cursor}`, { token: reader });
assert.deepStrictEqual(r.body.events, []);
assert.deepStrictEqual(r.body.events, []);
- assert.strictEqual(r.body.latest_cursor, await cursorAt(h, s3));
+ assert.strictEqual(r.body.latest_cursor, c3);
assert.match(r.body.latest_cursor, /^c1\./, 'the head as an opaque cursor');
assert.match(r.body.latest_cursor, /^c1\./, 'the head as an opaque cursor');
{
{
// Starting at the head with latest_cursor reads nothing old, and the next event after it arrives.
// Starting at the head with latest_cursor reads nothing old, and the next event after it arrives.
@@ -82,9 +85,9 @@ t('pull: cursor, topic filter, internal events included for services', async ()
@@ -82,9 +85,9 @@ t('pull: cursor, topic filter, internal events included for services', async ()
assert.strictEqual(fromHead.body.next_cursor, head);
assert.strictEqual(fromHead.body.next_cursor, head);
}
}
- const id3 = (await h.db.prepare('SELECT id FROM events WHERE seq = ?').get(s3)).id;
+ const id3 = (await h.db.prepare('SELECT id FROM events WHERE seq = ?').get(cursor.decode(c3).seq)).id;
r = await request(h.base, 'GET', `/api/v1/events/${id3}`, { token: reader });
r = await request(h.base, 'GET', `/api/v1/events/${id3}`, { token: reader });
- assert.strictEqual(r.body.seq, s3);
+ assert.strictEqual(r.body.cursor, c3);
r = await request(h.base, 'GET', '/api/v1/events?topic=live.*', { token: live });
r = await request(h.base, 'GET', '/api/v1/events?topic=live.*', { token: live });
assert.strictEqual(r.status, 403, 'events.event.read is required');
assert.strictEqual(r.status, 403, 'events.event.read is required');
r = await request(h.base, 'GET', '/api/v1/events?limit=5000', { token: reader });
r = await request(h.base, 'GET', '/api/v1/events?limit=5000', { token: reader });
@@ -94,10 +97,10 @@ t('pull: cursor, topic filter, internal events included for services', async ()
@@ -94,10 +97,10 @@ t('pull: cursor, topic filter, internal events included for services', async ()
t('pull: a cursor older than retention reports the gap', async () => {
t('pull: a cursor older than retention reports the gap', async () => {
const before = await h.store.lastSeq();
const before = await h.store.lastSeq();
await h.store.prune({ retentionDays: 30, now: Date.now() + 31 * 86400000 });
await h.store.prune({ retentionDays: 30, now: Date.now() + 31 * 86400000 });
- const s = (await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope() })).body.seq;
+ const c = (await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope() })).body.cursor;
const r = await request(h.base, 'GET', `/api/v1/events?after=${await cursorAt(h, 1)}`, { token: reader });
const r = await request(h.base, 'GET', `/api/v1/events?after=${await cursorAt(h, 1)}`, { token: reader });
assert.deepStrictEqual(r.body.gap, { from_seq: 2, to_seq: before });
assert.deepStrictEqual(r.body.gap, { from_seq: 2, to_seq: before });
- assert.deepStrictEqual(r.body.events.map(e => e.seq), [s]);
+ assert.deepStrictEqual(r.body.events.map(e => e.cursor), [c]);
});
});
t('pull: after_seq is refused even at zero', async () => {
t('pull: after_seq is refused even at zero', async () => {
@@ -41,7 +41,7 @@ t('realtime example gets a new ticket and resumes from the last cursor', async (
@@ -41,7 +41,7 @@ t('realtime example gets a new ticket and resumes from the last cursor', async (
assert.strictEqual(streams[0].url.searchParams.get('topics'), 'network.notification.*');
assert.strictEqual(streams[0].url.searchParams.get('topics'), 'network.notification.*');
assert.strictEqual(streams[0].url.searchParams.has('last_event_id'), false);
assert.strictEqual(streams[0].url.searchParams.has('last_event_id'), false);
- streams[0].onmessage({ lastEventId: 'opaque-cursor', data: '{"seq":1,"event":{}}' });
+ streams[0].onmessage({ lastEventId: 'opaque-cursor', data: '{"event":{}}' });
streams[0].onerror();
streams[0].onerror();
assert.strictEqual(streams[0].closed, true);
assert.strictEqual(streams[0].closed, true);
retries.shift()();
retries.shift()();
@@ -8,6 +8,7 @@
@@ -8,6 +8,7 @@
const assert = require('assert');
const assert = require('assert');
const crypto = require('crypto');
const crypto = require('crypto');
const { ids } = require('openvibe-contracts');
const { ids } = require('openvibe-contracts');
+const cursorCodec = require('../server/cursor');
const { boot, request, serviceToken, userToken, realtimeTicket, envelope, sse, sleep, suite, ISSUER, cursorAt } = require('./helpers');
const { boot, request, serviceToken, userToken, realtimeTicket, envelope, sse, sleep, suite, ISSUER, cursorAt } = require('./helpers');
const t = suite('realtime-tickets');
const t = suite('realtime-tickets');
@@ -27,7 +28,7 @@ async function stream(q, headers) {
@@ -27,7 +28,7 @@ async function stream(q, headers) {
async function publish(token, env) {
async function publish(token, env) {
const r = await request(h.base, 'POST', '/api/v1/events', { token, body: env });
const r = await request(h.base, 'POST', '/api/v1/events', { token, body: env });
assert.strictEqual(r.status, 201, r.text);
assert.strictEqual(r.status, 201, r.text);
- return r.body.seq;
+ return cursorCodec.decode(r.body.cursor).seq;
}
}
/** network.notification.created for `who`, as Network's outbox writes it. */
/** network.notification.created for `who`, as Network's outbox writes it. */
const notified = (who, unread = 1) => envelope('network', {
const notified = (who, unread = 1) => envelope('network', {
@@ -1,6 +1,7 @@
@@ -1,6 +1,7 @@
'use strict';
'use strict';
const assert = require('assert');
const assert = require('assert');
const { ids } = require('openvibe-contracts');
const { ids } = require('openvibe-contracts');
+const cursor = require('../server/cursor');
const { boot, request, serviceToken, userToken, envelope, sse, sleep, suite, cursorAt } = require('./helpers');
const { boot, request, serviceToken, userToken, envelope, sse, sleep, suite, cursorAt } = require('./helpers');
const t = suite('realtime');
const t = suite('realtime');
@@ -19,7 +20,7 @@ async function stream(q, headers) {
@@ -19,7 +20,7 @@ async function stream(q, headers) {
async function publish(token, env) {
async function publish(token, env) {
const r = await request(h.base, 'POST', '/api/v1/events', { token, body: env });
const r = await request(h.base, 'POST', '/api/v1/events', { token, body: env });
assert.strictEqual(r.status, 201, r.text);
assert.strictEqual(r.status, 201, r.text);
- return r.body.seq;
+ return cursor.decode(r.body.cursor).seq;
}
}
const quiet = () => sleep(150);
const quiet = () => sleep(150);
@@ -50,6 +51,7 @@ t('visibility: public to everyone, subject only to that user, internal only to s
@@ -50,6 +51,7 @@ t('visibility: public to everyone, subject only to that user, internal only to s
assert.deepStrictEqual(seqs(a), [pub, toAliceAsSubject, byAliceAsActor]);
assert.deepStrictEqual(seqs(a), [pub, toAliceAsSubject, byAliceAsActor]);
assert.deepStrictEqual(seqs(b), [pub], 'bob sees nothing addressed to alice');
assert.deepStrictEqual(seqs(b), [pub], 'bob sees nothing addressed to alice');
assert.deepStrictEqual(seqs(svc), [pub, internal, toAliceAsSubject, byAliceAsActor]);
assert.deepStrictEqual(seqs(svc), [pub, internal, toAliceAsSubject, byAliceAsActor]);
+ assert.ok(!Object.hasOwn(JSON.parse(anon.messages.find(m => !m.event).data), 'seq'));
const { encode } = require('../server/cursor');
const { encode } = require('../server/cursor');
assert.strictEqual(a.messages.find(m => m.data.includes('hi alice')).id, encode(toAliceAsSubject, await h.store.epoch()), 'SSE id is the cursor');
assert.strictEqual(a.messages.find(m => m.data.includes('hi alice')).id, encode(toAliceAsSubject, await h.store.epoch()), 'SSE id is the cursor');
for (const c of [anon, a, b, svc]) c.close();
for (const c of [anon, a, b, svc]) c.close();
@@ -5,7 +5,7 @@
@@ -5,7 +5,7 @@
* The leak this closes: a chat message deleted in OpenVibe.Chat stayed replayable here, text and
* The leak this closes: a chat message deleted in OpenVibe.Chat stayed replayable here, text and
* anon id included, for the whole retention, to anyone (public visibility, anonymous SSE replay).
* anon id included, for the whole retention, to anyone (public visibility, anonymous SSE replay).
* Now Chat's chat.message.deleted carries payload.redacts, the stored chat.message.created becomes
* Now Chat's chat.message.deleted carries payload.redacts, the stored chat.message.created becomes
- * a tombstone at the same seq on every read path, and browsers are replayed public events of the
+ * a tombstone at the same stored position on every read path, and browsers are replayed public events of the
* last few minutes only.
* last few minutes only.
*/
*/
const assert = require('assert');
const assert = require('assert');
@@ -13,6 +13,7 @@ const fs = require('fs');
@@ -13,6 +13,7 @@ const fs = require('fs');
const path = require('path');
const path = require('path');
const { ids } = require('openvibe-contracts');
const { ids } = require('openvibe-contracts');
const apps = require('../server/apps');
const apps = require('../server/apps');
+const cursor = require('../server/cursor');
const { parseDirective } = require('../server/redaction');
const { parseDirective } = require('../server/redaction');
const { boot, request, serviceToken, userToken, appToken, envelope, subscriber, sse, sleep, suite, cursorAt } = require('./helpers');
const { boot, request, serviceToken, userToken, appToken, envelope, subscriber, sse, sleep, suite, cursorAt } = require('./helpers');
@@ -96,7 +97,7 @@ t('a first-party subscriber is queued the original before the deletion', async (
@@ -96,7 +97,7 @@ t('a first-party subscriber is queued the original before the deletion', async (
t('before deletion the message is replayable (the leak as it was)', async () => {
t('before deletion the message is replayable (the leak as it was)', async () => {
const env = created(101);
const env = created(101);
msgEventId = env.event_id;
msgEventId = env.event_id;
- msgSeq = (await publish(chat, env)).seq;
+ msgSeq = cursor.decode((await publish(chat, env)).cursor).seq;
const anon = await stream('topics=chat.message.*', { 'Last-Event-ID': await cursorAt(h, msgSeq - 1) });
const anon = await stream('topics=chat.message.*', { 'Last-Event-ID': await cursorAt(h, msgSeq - 1) });
await anon.waitFor(c => c.events().length === 1);
await anon.waitFor(c => c.events().length === 1);
assert.strictEqual(anon.events()[0].event.payload.text, SECRET_TEXT);
assert.strictEqual(anon.events()[0].event.payload.text, SECRET_TEXT);
@@ -107,7 +108,7 @@ t('chat.message.deleted turns the stored event into a tombstone at the same seq'
@@ -107,7 +108,7 @@ t('chat.message.deleted turns the stored event into a tombstone at the same seq'
const env = deleted([101]);
const env = deleted([101]);
delEventId = env.event_id;
delEventId = env.event_id;
const r = await publish(chat, env);
const r = await publish(chat, env);
- delSeq = r.seq;
+ delSeq = cursor.decode(r.cursor).seq;
assert.strictEqual(delSeq, msgSeq + 1);
assert.strictEqual(delSeq, msgSeq + 1);
const row = await h.store.getEvent(msgEventId);
const row = await h.store.getEvent(msgEventId);
assert.strictEqual(row.seq, msgSeq, 'seq unchanged');
assert.strictEqual(row.seq, msgSeq, 'seq unchanged');
@@ -145,15 +146,15 @@ t('signed-in SSE replay (cookie and Bearer) and service replay: the tombstone to
@@ -145,15 +146,15 @@ t('signed-in SSE replay (cookie and Bearer) and service replay: the tombstone to
t('pull (/api/v1/events), by id, and developer-app reads: the tombstone', async () => {
t('pull (/api/v1/events), by id, and developer-app reads: the tombstone', async () => {
let r = await request(h.base, 'GET', `/api/v1/events?topic=chat.message.*&after=${await cursorAt(h, msgSeq - 1)}`, { token: reader });
let r = await request(h.base, 'GET', `/api/v1/events?topic=chat.message.*&after=${await cursorAt(h, msgSeq - 1)}`, { token: reader });
assert.strictEqual(r.status, 200);
assert.strictEqual(r.status, 200);
- assert.deepStrictEqual(r.body.events.map(e => e.seq), [msgSeq, delSeq], 'no hole in the sequence');
+ assert.deepStrictEqual(r.body.events.map(e => cursor.decode(e.cursor).seq), [msgSeq, delSeq], 'no hole in the sequence');
assertTombstone(r.body.events[0].event, delEventId);
assertTombstone(r.body.events[0].event, delEventId);
r = await request(h.base, 'GET', `/api/v1/events/${msgEventId}`, { token: reader });
r = await request(h.base, 'GET', `/api/v1/events/${msgEventId}`, { token: reader });
assertTombstone(r.body.event, delEventId);
assertTombstone(r.body.event, delEventId);
- assert.strictEqual(r.body.seq, msgSeq);
+ assert.strictEqual(cursor.decode(r.body.cursor).seq, msgSeq);
const app = appToken({ env: 'production', cap: ['events.app.read'] });
const app = appToken({ env: 'production', cap: ['events.app.read'] });
r = await request(h.base, 'GET', `/api/v1/events?topic=chat.*&after=${await cursorAt(h, msgSeq - 1)}`, { token: app });
r = await request(h.base, 'GET', `/api/v1/events?topic=chat.*&after=${await cursorAt(h, msgSeq - 1)}`, { token: app });
assert.strictEqual(r.status, 200, r.text);
assert.strictEqual(r.status, 200, r.text);
- assert.deepStrictEqual(r.body.events.map(e => e.seq), [msgSeq, delSeq]);
+ assert.deepStrictEqual(r.body.events.map(e => cursor.decode(e.cursor).seq), [msgSeq, delSeq]);
assertTombstone(r.body.events[0].event, delEventId);
assertTombstone(r.body.events[0].event, delEventId);
assert.ok(!leaks(r.body));
assert.ok(!leaks(r.body));
});
});
@@ -161,7 +162,7 @@ t('pull (/api/v1/events), by id, and developer-app reads: the tombstone', async
@@ -161,7 +162,7 @@ t('pull (/api/v1/events), by id, and developer-app reads: the tombstone', async
t('durable delivery: the consumer gets the tombstone and then the deletion, in order', async () => {
t('durable delivery: the consumer gets the tombstone and then the deletion, in order', async () => {
await h.worker.drain();
await h.worker.drain();
const got = stub.calls.map(c => c.body);
const got = stub.calls.map(c => c.body);
- assert.deepStrictEqual(got.map(b => b.seq), [msgSeq, delSeq]);
+ assert.deepStrictEqual(got.map(b => b.event.event_id), [msgEventId, delEventId]);
assertTombstone(got[0].event, delEventId);
assertTombstone(got[0].event, delEventId);
assert.strictEqual(got[1].event.event_type, 'chat.message.deleted');
assert.strictEqual(got[1].event.event_type, 'chat.message.deleted');
assert.ok(!leaks(stub.calls.map(c => c.rawBody.toString())));
assert.ok(!leaks(stub.calls.map(c => c.rawBody.toString())));
@@ -200,7 +201,7 @@ t('re-publishing a deletion is a duplicate; a second deletion leaves tombstones
@@ -200,7 +201,7 @@ t('re-publishing a deletion is a duplicate; a second deletion leaves tombstones
t('only the owning service can redact', async () => {
t('only the owning service can redact', async () => {
const env = created(201);
const env = created(201);
- const seq = (await publish(chat, env)).seq;
+ const seq = cursor.decode((await publish(chat, env)).cursor).seq;
// Live names Chat's event by id: refused, and nothing of the batch is stored.
// Live names Chat's event by id: refused, and nothing of the batch is stored.
const liveEvent = envelope('live', { payload: { redacts: { event_ids: [env.event_id] } } });
const liveEvent = envelope('live', { payload: { redacts: { event_ids: [env.event_id] } } });
let r = await request(h.base, 'POST', '/api/v1/events', { token: live, body: { events: [envelope('live'), liveEvent] } });
let r = await request(h.base, 'POST', '/api/v1/events', { token: live, body: { events: [envelope('live'), liveEvent] } });
@@ -264,7 +265,7 @@ t('operator redaction (backfill): same owner rule, idempotent', async () => {
@@ -264,7 +265,7 @@ t('operator redaction (backfill): same owner rule, idempotent', async () => {
t('public replay window: browsers get a gap for public events older than 5 minutes', async () => {
t('public replay window: browsers get a gap for public events older than 5 minutes', async () => {
const w = await boot({ env: { REALTIME_PUBLIC_REPLAY_SECONDS: '300' } });
const w = await boot({ env: { REALTIME_PUBLIC_REPLAY_SECONDS: '300' } });
try {
try {
- const post = async (token, body) => (await request(w.base, 'POST', '/api/v1/events', { token, body })).body.seq;
+ const post = async (token, body) => cursor.decode((await request(w.base, 'POST', '/api/v1/events', { token, body })).body.cursor).seq;
const net = serviceToken('network', ['events.event.publish']);
const net = serviceToken('network', ['events.event.publish']);
const oldPublic = await post(chat, created(1));
const oldPublic = await post(chat, created(1));
const oldMine = await post(net, envelope('network', { event_type: 'network.notification.created', visibility: 'subject', subject: { type: 'user', id: alice } }));
const oldMine = await post(net, envelope('network', { event_type: 'network.notification.created', visibility: 'subject', subject: { type: 'user', id: alice } }));
@@ -296,7 +297,7 @@ t('public replay window: browsers get a gap for public events older than 5 minut
@@ -296,7 +297,7 @@ t('public replay window: browsers get a gap for public events older than 5 minut
const none = await boot({ env: { REALTIME_PUBLIC_REPLAY_SECONDS: '0' } });
const none = await boot({ env: { REALTIME_PUBLIC_REPLAY_SECONDS: '0' } });
try {
try {
- const s = await (async () => (await request(none.base, 'POST', '/api/v1/events', { token: chat, body: created(3) })).body.seq)();
+ const s = await (async () => cursor.decode((await request(none.base, 'POST', '/api/v1/events', { token: chat, body: created(3) })).body.cursor).seq)();
const c = await sse(none.base, '/realtime/stream?topics=chat.message.*', { headers: { 'Last-Event-ID': await cursorAt(none, s - 1) } });
const c = await sse(none.base, '/realtime/stream?topics=chat.message.*', { headers: { 'Last-Event-ID': await cursorAt(none, s - 1) } });
await c.waitFor(x => x.gaps().length === 1);
await c.waitFor(x => x.gaps().length === 1);
await sleep(50);
await sleep(50);