openvibers/openvibe.events Commits
Events: a position is an opaque cursor only (after_seq, numeric checkpoints and bare-seq resume retired) (#56)
Changes
@@@ -114,7 +114,7 @@ Each delivery is `POST <endpoint>` with body `{ "event": <envelope>, "seq": n }`Operators: `GET /api/v1/deliveries?status=dead` is the dead-letter queue; `POST /api/v1/deliveries/replay { subscription_id, event_ids: [...] }` or `{ subscription_id, from_seq }` requeues retained events (that is also how a new subscription catches up on history).--Pull consumers: start with `GET /api/v1/events?topic=media.vod.*&limit=100`, then request `GET /api/v1/events?topic=media.vod.*&after=<next_cursor>&limit=100`, using the previous response's opaque `next_cursor` ([ADR-042](https://github.com/OpenVibers/OpenVibe.Contracts/blob/main/docs/adr/ADR-042-events-fabric.md) decision 7). The response carries `{ events: [{ seq, cursor, event }], next_cursor, next_after_seq, latest_seq, latest_cursor }` (`latest_cursor` is the head as an opaque cursor, for a consumer that starts at "now" without history), plus `gap: { from_seq, to_seq }` when the position is older than both the hot store and the replay tier (below) or from another retention epoch. Use `next_cursor` even when no events match, since it advances past scanned events. Numeric `after_seq=<position>` remains accepted for one release; when both are supplied, `after` takes precedence. The numeric `seq`, `next_after_seq` and `latest_seq` fields also remain for one release. `PUT /api/v1/checkpoints { topic, cursor }` stores a consumer's position here if it has nowhere better (an opaque cursor string is accepted in place of the integer; the answer carries its `epoch`, `carrier` and `next_cursor`).++Pull consumers: start with `GET /api/v1/events?topic=media.vod.*&limit=100`, then request `GET /api/v1/events?topic=media.vod.*&after=<next_cursor>&limit=100`, using the previous response's opaque `next_cursor` ([ADR-042](https://github.com/OpenVibers/OpenVibe.Contracts/blob/main/docs/adr/ADR-042-events-fabric.md) decision 7). The response carries `{ events: [{ seq, cursor, event }], next_cursor, latest_cursor }` (`latest_cursor` is the head as an opaque cursor, for a consumer that starts at "now" without history), plus `gap: { from_seq, to_seq }` when the position is older than both the hot store and the replay tier (below) or from another retention epoch. Use `next_cursor` even when no events match, since it advances past scanned events. A position is only ever an opaque cursor: `after_seq` is refused with 400 (never read as "from the start"), and the per-event `seq` is informational. `PUT /api/v1/checkpoints { topic, cursor, carrier? }` stores a consumer's cursor here if it has nowhere better (a cursor string; a number is refused); `GET /api/v1/checkpoints?topic=` and the PUT answer `{ consumer, topic, cursor, carrier, updated_at }`, with `cursor` as stored (null when none is).### Retention: three tiers@@@ -290,7 +290,7 @@ void connect();- Conversely, a user JWT that carries `typ` or `purpose` (a ticket, a FedCM assertion) is not a session here.- A person's topic: `network.notification.created` (Network's outbox; visibility `subject`, subject the recipient). `topics=network.notification.*` streams a person's own notifications and nobody else's. There is no `user:<id>` topic: it is not a valid pattern (400 `realtime.bad_topic`), and subject visibility already does its job.- Visibility: `public` events go to anyone subscribed to the topic; `subject` events only to the user whose subject id (`usr_…`) is the event's `actor.id` or its user `subject.id`; `internal` events never reach a browser. A guessed topic yields nothing.--- Resume: the SSE `id` is the cursor (ADR-042 decision 7; a bare seq is accepted for one release), so the browser's automatic `Last-Event-ID` (or `?last_event_id=`) replays what was missed. A cursor older than both the hot store and the replay tier — or from another retention epoch (`reason: "epoch"`) — first gets `event: gap` (`{ reason, from_seq, to_seq }`), as does a replay that hits `REALTIME_REPLAY_MAX`.++- Resume: the SSE `id` is the cursor (ADR-042 decision 7), so the browser's automatic `Last-Event-ID` (or `?last_event_id=`) replays what was missed. Only a cursor is a position: anything else, a bare number included, is ignored and the stream starts at the head. A cursor older than both the hot store and the replay tier — or from another retention epoch (`reason: "epoch"`) — first gets `event: gap` (`{ reason, from_seq, to_seq }`), as does a replay that hits `REALTIME_REPLAY_MAX`.- Public replay window: browsers (signed out or signed in) are replayed `public` events received in the last `REALTIME_PUBLIC_REPLAY_SECONDS` (300; 0 = none), enough to ride out a reconnect. Older public events are not replayed: the stream opens with `event: gap` (`reason: "public_window"`, `window_seconds`), and the client refetches state from the owning service. `subject` events addressed to the viewer, and service viewers, keep the whole retention. This keeps the stream from paging through a month of public history, chat lines included; nothing in the network needs more (no browser surface replays public events today).- Redacted events are replayed as their tombstones ([Redaction](#redaction)).- Heartbeat comment every 25 s; at most 20 topics per connection and `REALTIME_MAX_CONNECTIONS` (2000) overall; CORS with credentials for `https://*.openvibe.*` only.@@@ -387,7 +387,7 @@ implemented service.Part of the [OpenVibe network](https://openvibe.network). Built in the open by [OpenVibers](https://github.com/OpenVibers).<!-- versions:start -->--- openvibe-contracts: v0.129.0++- openvibe-contracts: v0.132.0- openvibe-sdk: v0.40.0- openvibe-shared: v3.0.0<!-- versions:end -->@@@ -7,7 +7,7 @@"runtime": "node 22, express, PostgreSQL 18 (openvibe-sdk/db, async) + Valkey 9","port": 4300,"domain": "openvibe.events",-- "contracts": "openvibe-contracts v0.129.0",++ "contracts": "openvibe-contracts v0.132.0","sdk": "openvibe-sdk v0.40.0","shared": "openvibe-shared v3.0.0","capabilities": [@@@ -12,7 +12,7 @@"dotenv": "^16.4.7","express": "^4.21.2","iovalkey": "^0.4.0",-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.129.0",++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.132.0","openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.40.0","openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v3.0.0","pg": "^8.23.0"@@@ -828,9 +828,9 @@}},"node_modules/openvibe-contracts": {-- "version": "0.129.0",-- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.129.0",-- "integrity": "sha512-Ly0F/ap6wmAf8hYpNrOhMLsU3cZYBjsFDW2P1F/xVmHVsRDzAPAZDXtFcohP0UNeZgyxgrR0BI0pPG5guD/Efw==",++ "version": "0.132.0",++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.132.0",++ "integrity": "sha512-08GuHutggWYtGcGrnBLilWco/K715Bl7ZAdfYX7kXeeHguWTIhbWZsSF3JYcH9iq6KKtRB7XCAbmPFKScleD9Q==","license": "AGPL-3.0-only","dependencies": {"ajv": "^8.17.1",@@@ -24,7 +24,7 @@"dotenv": "^16.4.7","express": "^4.21.2","iovalkey": "^0.4.0",-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.129.0",++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.132.0","openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.40.0","openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v3.0.0","pg": "^8.23.0"@@@ -2,20 +2,21 @@/*** Pull consumers (service token with events.event.read):*-- * GET /api/v1/events?topic=media.vod.*[,…]&after_seq=0&limit=100-- * -> { events: [{ seq, cursor, event }], next_after_seq, next_cursor, latest_seq, latest_cursor, gap? }-- * `latest_cursor` is the head as an opaque cursor: a consumer that starts at "now" (skipping history)-- * stores it, so it never needs the numeric latest_seq.-- * `after=` takes an opaque cursor beside after_seq (ADR-042 decision 7); a cursor from another-- * retention epoch answers `gap` — never a silent restart. `gap` ({ from_seq, to_seq }) means++ * GET /api/v1/events?topic=media.vod.*[,…][&after=<cursor>]&limit=100++ * -> { events: [{ seq, cursor, event }], next_cursor, latest_cursor, gap? }++ * A position is only ever an opaque cursor (ADR-042 decision 7): `after=` takes one, and without it the++ * page starts at the oldest retained event. `after_seq` is refused (400), never read as "from the start".++ * `latest_cursor` is the head: a consumer that starts at "now" (skipping history) stores it. The per-event++ * `seq` is informational. A cursor from another retention epoch answers `gap` — never a silent restart.++ * `gap` ({ from_seq, to_seq }) means* events after the position were already pruned by retention (or the epoch changed). The pull* spans the hot store and the replay tier (decision 8) with the one cursor: `gap` only outside both.* Keep next_cursor as the cursor; it moves past events that did not match.* GET /api/v1/events/:event_id -> { seq, cursor, event } (hot or replay)-- * GET /api/v1/checkpoints?topic=… / PUT /api/v1/checkpoints { topic, cursor }-- * a consumer's own stored cursor per topic pattern (consumer = calling principal). `cursor` is-- * the numeric position; PUT also accepts an opaque cursor string, and both answers carry the-- * position's epoch and carrier so the cursor can be rebuilt.++ * GET /api/v1/checkpoints?topic=… / PUT /api/v1/checkpoints { topic, cursor, carrier? }++ * a consumer's own stored cursor per topic pattern (consumer = calling principal): `cursor` is an++ * opaque cursor string (a number is refused), and both answers return it as stored, with its carrier++ * ({ consumer, topic, cursor, carrier, updated_at }; cursor null when none is stored).** Developer apps (events.app.read) use the same routes with an app token: only their project's* events in the token's environment plus public first-party events (server/apps.js); every topic@@@ -79,18 +80,19 @@ function readRouter({ store, auth, worker, limits }) {if (Number.isNaN(limit)) {return http.sendProblem(res, 400, 'events.bad_request', { detail: 'limit must be 1..1000', ctx });}++ // A position is an opaque cursor and nothing else: a number would silently restart a consumer that still sends one.++ if (req.query.after_seq !== undefined) {++ return http.sendProblem(res, 400, 'events.bad_request', { detail: "after_seq is retired: pass after=<cursor> (a page's next_cursor, or latest_cursor for the head)", ctx });++ }const epoch = await store.epoch();-- // `after=` is an opaque cursor; `after_seq` stays for one release. A cursor from another epoch cannot be-- // mapped onto this store, so it is answered with a gap (never treated as a position here).-- let after = intParam(req.query.after_seq, 0, 0, Number.MAX_SAFE_INTEGER);++ // A cursor from another epoch cannot be mapped onto this store, so it is answered with a gap.++ let after = 0;let epochMismatch = false;if (req.query.after !== undefined && String(req.query.after) !== '') {const decoded = cursor.decode(String(req.query.after));if (!decoded) return http.sendProblem(res, 400, 'events.bad_request', { detail: 'after must be an opaque cursor', ctx });after = decoded.seq;epochMismatch = decoded.epoch !== epoch;-- } else if (Number.isNaN(after)) {-- return http.sendProblem(res, 400, 'events.bad_request', { detail: 'after_seq must be >= 0', ctx });}const oldest = await store.oldestSeq();const out = {};@@@ -104,10 +106,8 @@ function readRouter({ store, auth, worker, limits }) {}const { rows, cursor: scanned } = await store.scan(from, { patterns, limit, accept: acceptFor(req.principal, patterns) });out.events = rows.map(r => ({ seq: r.seq, cursor: cursor.encode(r.seq, epoch), event: rowToEnvelope(r) }));-- out.next_after_seq = scanned;out.next_cursor = cursor.encode(scanned, epoch);-- out.latest_seq = await store.lastSeq();-- out.latest_cursor = cursor.encode(out.latest_seq, epoch);++ out.latest_cursor = cursor.encode(await store.lastSeq(), epoch);res.json(out);});@@@ -120,6 +120,10 @@ function readRouter({ store, auth, worker, limits }) {});const consumerOf = (req) => req.principal.service || req.principal.sub;++ // The stored position and epoch, handed back as the opaque cursor they make (null when nothing is stored).++ const checkpointView = (consumer, topic, cp) => ({++ consumer, topic, cursor: cp ? cursor.encode(Number(cp.cursor), Number(cp.epoch)) : null, carrier: cp ? cp.carrier || null : null, updated_at: cp ? cp.updated_at : null,++ });router.get('/api/v1/checkpoints', canRead, limits('events.checkpoint.read'), async (req, res) => {const topic = String(req.query.topic || '');@@@ -127,36 +131,21 @@ function readRouter({ store, auth, worker, limits }) {const scopeErr = scopeError(req.principal, [topic]);if (scopeErr) return http.sendProblem(res, 403, 'events.topic_not_allowed', { detail: scopeErr, ctx: req.ov });const cp = await store.getCheckpoint(consumerOf(req), topic);-- const epoch = cp ? cp.epoch : await store.epoch();-- const position = cp ? cp.cursor : 0;-- res.json({-- consumer: consumerOf(req), topic, cursor: position, epoch, carrier: cp ? cp.carrier : null,-- next_cursor: cursor.encode(position, epoch), updated_at: cp ? cp.updated_at : null,-- });++ res.json(checkpointView(consumerOf(req), topic, cp));});router.put('/api/v1/checkpoints', canRead, limits('events.checkpoint.write', { minute: 600, hour: 20000 }), async (req, res) => {const b = req.body || {};-- // `cursor` stays the numeric position; an opaque cursor string is accepted in its place and decoded to the-- // position and epoch it names (ADR-042 decision 7). `carrier` records the carrier the position is on.-- let position = null;-- let atEpoch = null;-- if (typeof b.cursor === 'string') {-- const decoded = cursor.decode(b.cursor);-- if (!decoded) return http.sendProblem(res, 400, 'events.bad_request', { detail: 'cursor must be an opaque cursor or an integer >= 0', ctx: req.ov });-- position = decoded.seq;-- atEpoch = decoded.epoch;-- } else if (Number.isInteger(b.cursor) && b.cursor >= 0) {-- position = b.cursor;-- if (Number.isInteger(b.epoch) && b.epoch >= 0) atEpoch = b.epoch;-- }-- if (!topics.isValidPattern(b.topic) || position === null) {-- return http.sendProblem(res, 400, 'events.bad_request', { detail: 'topic (pattern) and cursor (integer >= 0 or an opaque cursor) are required', ctx: req.ov });++ // `cursor` is an opaque cursor, decoded to the position and epoch it names (ADR-042 decision 7); a number is++ // refused. `carrier` records the carrier the position is on.++ const decoded = typeof b.cursor === 'string' ? cursor.decode(b.cursor) : null;++ if (!topics.isValidPattern(b.topic) || !decoded) {++ return http.sendProblem(res, 400, 'events.bad_request', { detail: "topic (pattern) and cursor (an opaque cursor: a page's next_cursor) are required", ctx: req.ov });}const scopeErr = scopeError(req.principal, [b.topic]);if (scopeErr) return http.sendProblem(res, 403, 'events.topic_not_allowed', { detail: scopeErr, ctx: req.ov });-- const cp = await store.setCheckpoint(consumerOf(req), b.topic, position, { epoch: atEpoch, carrier: typeof b.carrier === 'string' && b.carrier ? b.carrier : null });-- res.json({ consumer: consumerOf(req), topic: b.topic, ...cp, next_cursor: cursor.encode(cp.cursor, cp.epoch) });++ const cp = await store.setCheckpoint(consumerOf(req), b.topic, decoded.seq, { epoch: decoded.epoch, carrier: typeof b.carrier === 'string' && b.carrier ? b.carrier : null });++ res.json(checkpointView(consumerOf(req), b.topic, cp));});router.get('/api/v1/deliveries', isAdmin, limits('events.delivery.list'), async (req, res) => {@@@ -18,7 +18,8 @@* internal service principals (token with events.event.read) only, never a browser* Developer-app events (app.<project_key>.*, events.app.publish) are never streamed here, to anyone.*-- * Resume: `Last-Event-ID` (or ?last_event_id=) is a cursor, or a bare seq for one release. Missed++ * Resume: `Last-Event-ID` (or ?last_event_id=) is an opaque cursor; anything else (a bare number included) is not a++ * position and is ignored, so the stream starts at the head. Missed* events are replayed first, from the replay tier and then the hot store (ADR-042 decision 8); when the* cursor is older than both — or belongs to another retention epoch — an `event: gap` message comes* first, so the client knows to refetch state.@@@ -131,16 +132,12 @@ function createRealtime({ store, auth, config, clock = { now: () => Date.now() }const epoch = await store.epoch();const rawLast = req.headers['last-event-id'] ?? req.query.last_event_id;-- // A cursor, or — for one release — a bare seq. Anything else is ignored, as before.++ // Only a cursor is a position (ADR-042 decision 7); anything else is ignored and the stream starts at the head.let lastId = null;let epochMismatch = false;if (rawLast != null) {-- const raw = String(rawLast).trim();-- if (/^\d{1,15}$/.test(raw)) lastId = Number(raw);-- else {-- const decoded = cursor.decode(raw);-- if (decoded) { lastId = decoded.seq; epochMismatch = decoded.epoch !== epoch; }-- }++ const decoded = cursor.decode(String(rawLast).trim());++ if (decoded) { lastId = decoded.seq; epochMismatch = decoded.epoch !== epoch; }}res.status(200);@@@ -724,13 +724,14 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage/*** A checkpoint is a cursor in parts: `cursor` is the numeric position, `epoch` the retention epoch it belongs to-- * and `carrier` the carrier that position is on (ADR-042 decision 7). `cursor` stays numeric on the wire.++ * and `carrier` the carrier that position is on (ADR-042 decision 7). The wire only ever carries the opaque cursor++ * these parts make (server/api/read.js).*/async function getCheckpoint(consumer, topicPattern) {const r = await db.prepare('SELECT cursor, epoch, carrier, updated_at FROM consumer_checkpoints WHERE consumer = ? AND topic_pattern = ?').get(consumer, topicPattern);return r ? { cursor: r.cursor, epoch: r.epoch, carrier: r.carrier || null, updated_at: new Date(r.updated_at).toISOString() } : null;}-- /** Store a position. `epoch` defaults to the current retention epoch (a legacy numeric cursor). */++ /** Store a position (decoded from a cursor). `epoch` defaults to the current retention epoch. */async function setCheckpoint(consumer, topicPattern, position, { epoch: atEpoch = null, carrier = null } = {}) {const at = atEpoch == null ? await epoch() : atEpoch;await db.prepare(`INSERT INTO consumer_checkpoints (consumer, topic_pattern, cursor, epoch, carrier, updated_at) VALUES (?, ?, ?, ?, ?, ?)@@@ -8,7 +8,7 @@ const { verifyDeliveryV2 } = require('../lib/client');const apps = require('../server/apps');const { createGuardedPost, isPublicAddress } = require('../server/egress');const { load } = require('../server/config');--const { boot, request, serviceToken, appToken, envelope, subscriber, suite, sleep } = require('./helpers');++const { boot, request, serviceToken, appToken, envelope, subscriber, suite, sleep, cursorAt } = require('./helpers');const t = suite('apps');const ALL = ['events.app.publish', 'events.app.read', 'events.app.subscribe'];@@@ -72,7 +72,7 @@ function appEvent(appId, projectId, name = 'order.created', over = {}) {});}const publish = async (token, body) => await request(h.base, 'POST', '/api/v1/events', { token, body });--const pull = async (token, topic, after = 0) => await request(h.base, 'GET', `/api/v1/events?topic=${encodeURIComponent(topic)}&after_seq=${after}`, { token });++const pull = async (token, topic, after = null) => await request(h.base, 'GET', `/api/v1/events?topic=${encodeURIComponent(topic)}${after === null || after === 0 ? '' : `&after=${after}`}`, { token });t('boot', async () => {await new Promise(r => receiver.listen(0, '127.0.0.1', r));@@@ -177,9 +177,9 @@ t('read: own project in the same environment, plus public first-party events', aassert.strictEqual(r.status, 404, 'sandbox events are invisible to first-party readers');// Checkpoints are per app and follow the same scope.-- r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: tokA(), body: { topic: `app.${keyA}.*`, cursor: 3 } });++ r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: tokA(), body: { topic: `app.${keyA}.*`, cursor: await cursorAt(h, 3) } });assert.strictEqual(r.status, 200); assert.strictEqual(r.body.consumer, `app:${appA}`);-- r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: tokA(), body: { topic: `app.${keyB}.*`, cursor: 3 } });++ r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: tokA(), body: { topic: `app.${keyB}.*`, cursor: await cursorAt(h, 3) } });assert.strictEqual(r.status, 403);r = await request(h.base, 'GET', `/api/v1/checkpoints?topic=app.${keyA}.*`, { token: tokB() });assert.strictEqual(r.status, 403);@@@ -12,7 +12,7 @@ const { limitsOf } = require('../server/limits');const { createStore } = require('../server/store');const { exportMonth, restoreMonth, objectKey, manifestKey } = require('../scripts/events-archive');const { localStorage, s3Storage, fromEnv, signV4 } = require('../scripts/archive-storage');--const { boot, request, serviceToken, appToken, envelope, sse, suite, tmpDir } = require('./helpers');++const { boot, request, serviceToken, appToken, envelope, sse, suite, tmpDir, cursorAt } = require('./helpers');const t = suite('archive');const DAY = 86400000;@@@ -44,7 +44,7 @@ t('a hot-pruned event is readable through replay: by id and by pull, unchanged',const after = await request(h.base, 'GET', `/api/v1/events/${e.event_id}`, { token: reader });assert.strictEqual(after.status, 200, after.text);assert.deepStrictEqual(after.body, before.body, 'the same seq, cursor and envelope from the replay tier');-- const p = await pull('topic=live.tier.one&after_seq=0');++ const p = await pull('topic=live.tier.one');assert.deepStrictEqual(p.body.events.map(x => x.event), [before.body.event]);assert.strictEqual(p.body.gap, undefined);// A publish of the same id is a duplicate with its position, never a second row.@@@ -65,7 +65,7 @@ t('a cursor across the hot/replay boundary reports no false gap (pull and SSE)',assert.strictEqual(r.body.events[0].cursor, b.cursor);r = await pull(`topic=live.tier.cross&after=${r.body.next_cursor}&limit=1`);assert.deepStrictEqual([r.body.events.map(x => x.seq), r.body.gap], [[c.seq], undefined], 'then hot, no gap between');-- r = await pull(`topic=*&after_seq=${a.seq - 1}`);++ r = await pull(`topic=*&after=${await cursorAt(h, a.seq - 1)}`);assert.deepStrictEqual(r.body.events.map(x => x.seq), [a.seq, b.seq, c.seq]);assert.strictEqual(r.body.gap, undefined);@@@ -92,7 +92,7 @@ t('redaction reaches replay rows (by id and by subject); the owner rule holds thassert.deepStrictEqual(r.body.event.actor, { type: 'service', id: 'live' });assert.strictEqual(r.body.seq, e.seq);}-- const p = await pull('topic=live.tier.secret&after_seq=0');++ const p = await pull('topic=live.tier.secret');assert.ok(p.body.events.every(e => e.event.payload.redacted === true && !('secret' in e.event.payload)));});@@@ -108,11 +108,11 @@ t('sandbox and environment rules apply to replay rows exactly as to hot ones', aawait h.store.prune(tiered(31, { sandboxRetentionDays: 7 }));assert.deepStrictEqual([await count('events_archive', pe.event_id), await count('events_archive', se.event_id), await count('events', se.event_id)], [1, 0, 0],'a production event moves; a sandbox event is deleted, never kept in replay');-- const topic = `topic=app.${apps.projectKey(prj)}.*&after_seq=0`;++ const topic = `topic=app.${apps.projectKey(prj)}.*`;assert.deepStrictEqual((await pull(topic, prod)).body.events.map(e => e.seq), [pe.seq], 'the app reads its own replay rows');assert.deepStrictEqual((await pull(topic, sand)).body.events, [], 'never across environments');assert.deepStrictEqual((await pull(topic)).body.events.map(e => e.seq), [pe.seq], 'first-party: production app events through app.*');-- assert.deepStrictEqual((await pull('topic=*&after_seq=0')).body.events.filter(e => e.event.event_id === pe.event_id), [], '…and only through app.*');++ assert.deepStrictEqual((await pull('topic=*')).body.events.filter(e => e.event.event_id === pe.event_id), [], '…and only through app.*');assert.strictEqual((await request(h.base, 'GET', `/api/v1/events/${pe.event_id}`, { token: sand })).status, 404);assert.strictEqual((await request(h.base, 'GET', `/api/v1/events/${pe.event_id}`, { token: prod })).status, 200);});@@@ -148,7 +148,7 @@ t('export → restore round-trips byte-identical rows; the rows leave replay onlassert.strictEqual(lines.length, inMonth.length);assert.strictEqual(JSON.parse(await storage.get(manifestKey(month))).count, inMonth.length);assert.strictEqual((await request(h.base, 'GET', `/api/v1/events/${inMonth[0].id}`, { token: reader })).status, 404, 'the archive tier is offline');-- const gapped = await pull(`topic=*&after_seq=0`);++ const gapped = await pull(`topic=*&after=${await cursorAt(h, 0)}`);assert.ok(gapped.body.gap, 'a cursor into what went to the archive tier is a gap');// A rerun is idempotent: the object keeps its rows, nothing is deleted twice.@@@ -1,10 +1,9 @@'use strict';/**-- * ADR-042 decision 7: opaque cursors beside the global seq. A cursor is `c1.<epoch>.<base64url(seq)>`;-- * every answer that carries a seq also carries its cursor, pull accepts `after=<cursor>` beside-- * `after_seq`, the SSE `id` is the cursor (a bare seq is accepted for one release), and checkpoints-- * store and return the position's epoch. The contract answers are validated against the pinned-- * openvibe-contracts (0.81.0, the release that added the cursor fields).++ * ADR-042 decision 7: consumer positions are cursors only. A cursor is `c1.<epoch>.<base64url(seq)>`;++ * event answers retain seq and carry a cursor, pull accepts `after=<cursor>`, SSE resumes from++ * cursor IDs, and checkpoints store and return opaque cursor strings. Unchanged publish and single++ * event answers are validated against the pinned openvibe-contracts.*/const assert = require('assert');const { validate } = require('openvibe-contracts');@@@ -39,7 +38,7 @@ t('encode/decode round trip (never throws on malformed input)', () => {}});--t('publish and read answers carry cursors, validated against events.publish-result@1 / read-result@1', async () => {++t('publish and read answers carry cursors', async () => {const epoch = await h.store.epoch();const one = await publish();const pv = validate('events.publish-result@1', one);@@@ -51,28 +50,30 @@ t('publish and read answers carry cursors, validated against events.publish-resufor (const r of batch.body.results) assert.deepStrictEqual(cursor.decode(r.cursor), { seq: r.seq, epoch });const page = await request(h.base, 'GET', '/api/v1/events?topic=live.vod.*', { token: reader });-- const rv = validate('events.read-result@1', page.body);-- assert.ok(rv.valid, JSON.stringify(rv.errors));++ assert.ok(typeof page.body.next_cursor === 'string' && typeof page.body.latest_cursor === 'string');assert.ok(page.body.events.length >= 3);for (const e of page.body.events) assert.deepStrictEqual(cursor.decode(e.cursor), { seq: e.seq, epoch });-- assert.deepStrictEqual(cursor.decode(page.body.next_cursor), { seq: page.body.next_after_seq, epoch });++ assert.deepStrictEqual(cursor.decode(page.body.next_cursor), { seq: page.body.events.at(-1).seq, epoch });++ assert.ok(!('next_after_seq' in page.body) && !('latest_seq' in page.body));const single = await request(h.base, 'GET', `/api/v1/events/${one.event_id}`, { token: reader });assert.ok(validate('events.read-result@1', single.body).valid);assert.deepStrictEqual(cursor.decode(single.body.cursor), { seq: single.body.seq, epoch });});--t('pull accepts after=<cursor> beside after_seq; next_cursor round-trips a page', async () => {++t('pull accepts after=<cursor>; next_cursor round-trips a page and after_seq is refused', async () => {let r = await request(h.base, 'GET', '/api/v1/events?topic=live.vod.*&limit=1', { token: reader });assert.strictEqual(r.body.events.length, 1);const first = r.body.events[0];-- assert.strictEqual(r.body.next_after_seq, first.seq);++ assert.strictEqual(r.body.next_cursor, first.cursor);r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after=${encodeURIComponent(r.body.next_cursor)}`, { token: reader });assert.strictEqual(r.status, 200, r.text);assert.ok(r.body.events.length >= 1 && r.body.events.every(e => e.seq > first.seq));-- // after_seq still works, unchanged.-- r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after_seq=${first.seq}`, { token: reader });-- assert.ok(r.body.events.every(e => e.seq > first.seq));++ for (const value of ['0', String(first.seq)]) {++ r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after_seq=${value}`, { token: reader });++ assert.strictEqual(r.status, 400);++ assert.strictEqual(r.body.code, 'events.bad_request');++ }});t('a malformed after= is refused, not treated as a position', async () => {@@@ -97,7 +98,7 @@ t('a cursor from another epoch answers the gap shape, never a silent restart', aassert.strictEqual(cursor.decode(r.body.next_cursor).epoch, olderEpoch + 1, 'position moves in the new epoch');});--t('SSE id is the cursor; Last-Event-ID resumes from a cursor, and a bare seq still works', async () => {++t('SSE id is the cursor; Last-Event-ID resumes from a cursor, and a bare seq starts at the head', async () => {const epoch = await h.store.epoch();const c1 = await sse(h.base, '/realtime/stream?topics=live.stream.*', { headers: { Authorization: `Bearer ${readerSvc}` } });const s1 = await publish({ visibility: 'public', event_type: 'live.stream.a' });@@@ -113,8 +114,9 @@ t('SSE id is the cursor; Last-Event-ID resumes from a cursor, and a bare seq stiassert.deepStrictEqual(byCursor.gaps(), []);byCursor.close();-- // A bare seq is still accepted for this release.-- const bySeq = await sse(h.base, `/realtime/stream?topics=live.stream.*&last_event_id=${s2.seq}`, { headers: { Authorization: `Bearer ${readerSvc}` } });++ const bySeq = await sse(h.base, `/realtime/stream?topics=live.stream.*&last_event_id=${s1.seq}`, { headers: { Authorization: `Bearer ${readerSvc}` } });++ assert.deepStrictEqual(bySeq.events(), [], 'a bare number does not replay missed events');++ assert.deepStrictEqual(bySeq.gaps(), []);const s3 = await publish({ visibility: 'public', event_type: 'live.stream.c' });await bySeq.waitFor(x => x.events().length === 1);assert.deepStrictEqual(bySeq.events().map(e => e.seq), [s3.seq]);@@@ -136,27 +138,22 @@ t('SSE reports a gap when the cursor belongs to another epoch', async () => {again.close();});--t('checkpoint round trip: an opaque cursor in, the position and epoch out', async () => {++t('checkpoint round trip: an opaque cursor in and out', async () => {const pub = await publish();-- const epoch = cursor.decode(pub.cursor).epoch;let r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: reader, body: { topic: 'live.vod.*', cursor: pub.cursor } });assert.strictEqual(r.status, 200, r.text);-- assert.ok(validate('events.read-result@1', r.body).valid, JSON.stringify(r.body));-- assert.strictEqual(r.body.cursor, pub.seq, 'cursor stays the numeric position');-- assert.strictEqual(r.body.epoch, epoch);-- assert.strictEqual(r.body.next_cursor, pub.cursor, 'the opaque cursor comes back, ready for after=');++ assert.strictEqual(r.body.topic, 'live.vod.*');++ assert.strictEqual(r.body.cursor, pub.cursor);++ assert.ok(!('epoch' in r.body) && !('next_cursor' in r.body));r = await request(h.base, 'GET', '/api/v1/checkpoints?topic=live.vod.*', { token: reader });-- assert.strictEqual(r.body.cursor, pub.seq);-- assert.strictEqual(r.body.epoch, epoch);-- assert.strictEqual(r.body.next_cursor, pub.cursor);-- assert.strictEqual(cursor.encode(r.body.cursor, r.body.epoch), pub.cursor);++ assert.strictEqual(r.body.cursor, pub.cursor);++ assert.ok(!('epoch' in r.body) && !('next_cursor' in r.body));-- // A plain integer still works and takes the current epoch.++ // A plain integer is no longer a consumer position.r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: reader, body: { topic: 'live.vod.*', cursor: 5 } });-- assert.strictEqual(r.body.cursor, 5);-- assert.strictEqual(r.body.epoch, await h.store.epoch());-- assert.deepStrictEqual(cursor.decode(r.body.next_cursor), { seq: 5, epoch: await h.store.epoch() });++ assert.strictEqual(r.status, 400);++ assert.strictEqual(r.body.code, 'events.bad_request');// A malformed cursor string is refused.r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: reader, body: { topic: 'live.vod.*', cursor: 'nope' } });@@@ -28,7 +28,7 @@ function exportToken(env, { project = P, cap = ['events.app.read'] } = {}) {let h;const seqs = { production: [], sandbox: [] };--const pull = async (token, after = 0, limit = 100, topic = `app.${key}.*`) => await request(h.base, 'GET', `/api/v1/events?topic=${encodeURIComponent(topic)}&after_seq=${after}&limit=${limit}`, { token });++const pull = async (token, after = null, limit = 100, topic = `app.${key}.*`) => await request(h.base, 'GET', `/api/v1/events?topic=${encodeURIComponent(topic)}${after === null || after === 0 ? '' : `&after=${after}`}&limit=${limit}`, { token });t('boot and publish as the project\'s app, in both environments (and another project)', async () => {h = await boot();@@@ -47,15 +47,13 @@ t('boot and publish as the project\'s app, in both environments (and another prot('an export token pulls the project\'s events of its environment, page by page to the end', async () => {const got = [];-- let after = 0;-- let latest = null;++ let after = null;for (let page = 0; page < 10; page++) {const r = await pull(exportToken('production'), after, 2);assert.strictEqual(r.status, 200, r.text);-- latest = latest === null ? r.body.latest_seq : latest;got.push(...r.body.events.map(e => e.seq));-- if (r.body.next_after_seq <= after || r.body.next_after_seq >= latest) break;-- after = r.body.next_after_seq;++ if (r.body.next_cursor === r.body.latest_cursor || r.body.next_cursor === after) break;++ after = r.body.next_cursor;}assert.deepStrictEqual(got, seqs.production);const s = await pull(exportToken('sandbox'));@@@ -94,6 +94,9 @@ async function boot({ env = {}, clock = manualClock(), worker = 'manual', fetchI};}++/** Events' opaque cursor for a position in this boot's store (tests build positions the way Events hands them out). */++async function cursorAt(h, seq) { return require('../server/cursor').encode(seq, await h.store.epoch()); }++async function request(base, method, p, { token, body, headers = {}, raw } = {}) {const h = { ...headers };if (token) h.Authorization = `Bearer ${token}`;@@@ -216,5 +219,5 @@ function suite(name) {module.exports = {ISSUER, privateKey, publicKey, silent, serviceToken, appToken, userToken, realtimeTicket, manualClock, tmpDir, boot, request,-- envelope, subscriber, sse, sleep, suite,++ envelope, subscriber, sse, sleep, suite, cursorAt,};@@@ -57,7 +57,7 @@ t('deliveries feed the latency histogram and attempt counter; a dead one degradet('/metrics: loopback only, templates not ids, Events gauges', async () => {const id = (await h.db.prepare('SELECT id FROM events ORDER BY seq LIMIT 1').get()).id;await request(h.base, 'GET', `/api/v1/events/${id}`, { token: reader });-- await request(h.base, 'GET', '/api/v1/events?topic=live.*&after_seq=0', { token: reader });++ await request(h.base, 'GET', '/api/v1/events?topic=live.*', { token: reader });await request(h.base, 'GET', '/no/such/12345');let m = await get(h.base, '/metrics', { 'X-Forwarded-For': '203.0.113.7' });assert.strictEqual(m.status, 404, 'through a proxy: not found');@@@ -2,12 +2,12 @@const assert = require('assert');const crypto = require('crypto');const nodeHttp = require('http');--const { validate } = require('openvibe-contracts');const cursor = require('../server/cursor');const topics = require('../server/topics');const { hasCap, allows } = require('../server/auth');const { load } = require('../server/config');--const { boot, request, serviceToken, envelope, publicKey, suite, sleep } = require('./helpers');++const { validate } = require('openvibe-contracts');++const { boot, request, serviceToken, envelope, publicKey, suite, sleep, cursorAt } = require('./helpers');const t = suite('pull-health');let h;@@@ -58,19 +58,20 @@ t('pull: cursor, topic filter, internal events included for services', async ()await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope('live', { event_type: 'live.chat.sent' }) });const s3 = (await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope('live', { event_type: 'live.vod.deleted' }) })).body.seq;-- let r = await request(h.base, 'GET', '/api/v1/events?topic=live.vod.*&after_seq=0&limit=1', { token: reader });++ let r = await request(h.base, 'GET', '/api/v1/events?topic=live.vod.*&limit=1', { token: reader });assert.strictEqual(r.status, 200, r.text);assert.ok(validate('events.read-result@1', r.body).valid, JSON.stringify(validate('events.read-result@1', r.body).errors));for (const e of r.body.events) assert.deepStrictEqual(cursor.decode(e.cursor), { seq: e.seq, epoch: await h.store.epoch() });-- assert.strictEqual(cursor.decode(r.body.next_cursor).seq, r.body.next_after_seq);++ assert.strictEqual(r.body.next_cursor, await cursorAt(h, s1));++ assert.ok(!('next_after_seq' in r.body) && !('latest_seq' in r.body));assert.deepStrictEqual(r.body.events.map(e => e.seq), [s1]);assert.strictEqual(r.body.events[0].event.visibility, 'internal');-- r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after_seq=${r.body.next_after_seq}`, { token: reader });++ r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after=${r.body.next_cursor}`, { token: reader });assert.deepStrictEqual(r.body.events.map(e => e.seq), [s3]);-- assert.strictEqual(r.body.next_after_seq, s3);-- r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after_seq=${r.body.next_after_seq}`, { token: reader });++ assert.strictEqual(r.body.next_cursor, await cursorAt(h, s3));++ r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after=${r.body.next_cursor}`, { token: reader });assert.deepStrictEqual(r.body.events, []);-- assert.strictEqual(r.body.latest_seq, s3);++ assert.strictEqual(r.body.latest_cursor, await cursorAt(h, s3));assert.match(r.body.latest_cursor, /^c1\./, 'the head as an opaque cursor');{// Starting at the head with latest_cursor reads nothing old, and the next event after it arrives.@@@ -94,21 +95,35 @@ t('pull: a cursor older than retention reports the gap', async () => {const before = await h.store.lastSeq();await h.store.prune({ retentionDays: 30, now: Date.now() + 31 * 86400000 });const s = (await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope() })).body.seq;-- const r = await request(h.base, 'GET', '/api/v1/events?after_seq=1', { token: reader });++ const r = await request(h.base, 'GET', `/api/v1/events?after=${await cursorAt(h, 1)}`, { token: reader });assert.deepStrictEqual(r.body.gap, { from_seq: 2, to_seq: before });assert.deepStrictEqual(r.body.events.map(e => e.seq), [s]);});++t('pull: after_seq is refused even at zero', async () => {++ const r = await request(h.base, 'GET', '/api/v1/events?topic=live.vod.*&after_seq=0', { token: reader });++ assert.strictEqual(r.status, 400);++ assert.strictEqual(r.body.code, 'events.bad_request');++});++t('checkpoints: per consumer and topic', async () => {let r = await request(h.base, 'GET', '/api/v1/checkpoints?topic=live.vod.*', { token: reader });-- assert.strictEqual(r.body.cursor, 0);++ assert.strictEqual(r.body.cursor, null);++ assert.strictEqual(r.body.carrier, null);++ assert.strictEqual(r.body.updated_at, null);r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: reader, body: { topic: 'live.vod.*', cursor: 42 } });-- assert.strictEqual(r.body.cursor, 42);++ assert.strictEqual(r.status, 400);++ assert.strictEqual(r.body.code, 'events.bad_request');++ const position = await cursorAt(h, 42);++ r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: reader, body: { topic: 'live.vod.*', cursor: position } });++ assert.strictEqual(r.status, 200);++ assert.strictEqual(r.body.cursor, position);++ assert.ok(validate('events.read-result@1', r.body).valid, 'the checkpoint answer is a contract checkpoint');r = await request(h.base, 'GET', '/api/v1/checkpoints?topic=live.vod.*', { token: reader });-- assert.strictEqual(r.body.cursor, 42);++ assert.strictEqual(r.body.cursor, position);assert.strictEqual(r.body.consumer, 'games');r = await request(h.base, 'GET', '/api/v1/checkpoints?topic=live.vod.*', { token: serviceToken('tools', ['events.event.read']) });-- assert.strictEqual(r.body.cursor, 0, 'another consumer has its own');++ assert.strictEqual(r.body.cursor, null, 'another consumer has its own');});t('health and ready', async () => {@@@ -8,7 +8,7 @@const assert = require('assert');const crypto = require('crypto');const { ids } = require('openvibe-contracts');--const { boot, request, serviceToken, userToken, realtimeTicket, envelope, sse, sleep, suite, ISSUER } = require('./helpers');++const { boot, request, serviceToken, userToken, realtimeTicket, envelope, sse, sleep, suite, ISSUER, cursorAt } = require('./helpers');const t = suite('realtime-tickets');let h;@@@ -77,7 +77,7 @@ t('a guessed user:<other> topic yields nothing: refused as a pattern, and no patassert.deepStrictEqual(guess.events(), [], 'Alice\'s ticket sees none of Bob\'s events and no internal one');assert.deepStrictEqual(anon.events(), []);// Replay too: from before those events, Alice's ticket still gets none of them.-- const replay = await stream(`topics=*&ticket=${realtimeTicket({ subjectId: alice })}&last_event_id=0`);++ const replay = await stream(`topics=*&ticket=${realtimeTicket({ subjectId: alice })}&last_event_id=${await cursorAt(h, 0)}`);await quiet();assert.ok(replay.events().every(e => e.event.subject.id !== bob && e.event.visibility !== 'internal'), 'replay never includes Bob\'s or internal events');guess.close(); anon.close(); replay.close();@@@ -139,7 +139,7 @@ t('disconnect and resume from the cursor with a fresh ticket: the missed events,const missed1 = await publish(network, notified(alice, 2));await publish(network, notified(bob, 9));const missed2 = await publish(network, notified(alice, 3));-- const a2 = await stream(`topics=${TOPIC}&ticket=${realtimeTicket({ subjectId: alice })}&last_event_id=${cursor}`);++ const a2 = await stream(`topics=${TOPIC}&ticket=${realtimeTicket({ subjectId: alice })}&last_event_id=${await cursorAt(h, cursor)}`);await a2.waitFor(c => c.events().length === 2);assert.deepStrictEqual(a2.events().map(e => e.seq), [missed1, missed2]);assert.deepStrictEqual(a2.events().map(e => e.event.payload.unread_count), [2, 3]);@@@ -154,7 +154,7 @@ t('a cursor older than retention is reported as a gap before the events kept', aconst old = await publish(network, notified(alice, 5));await h.store.prune({ retentionDays: 30, now: Date.now() + 31 * 86400000 });const kept = await publish(network, notified(alice, 6));-- const c = await stream(`topics=${TOPIC}&ticket=${realtimeTicket({ subjectId: alice })}&last_event_id=${old - 1}`);++ const c = await stream(`topics=${TOPIC}&ticket=${realtimeTicket({ subjectId: alice })}&last_event_id=${await cursorAt(h, old - 1)}`);await c.waitFor(x => x.events().length === 1);assert.strictEqual(c.gaps().length, 1);assert.strictEqual(c.gaps()[0].reason, 'retention');@@@ -1,7 +1,7 @@'use strict';const assert = require('assert');const { ids } = require('openvibe-contracts');--const { boot, request, serviceToken, userToken, envelope, sse, sleep, suite } = require('./helpers');++const { boot, request, serviceToken, userToken, envelope, sse, sleep, suite, cursorAt } = require('./helpers');const t = suite('realtime');let h;@@@ -72,7 +72,7 @@ t('Last-Event-ID resume replays what was missed, in order, then goes live', asynawait publish(live, envelope('live', { visibility: 'public', event_type: 'live.chat.not_subscribed' }));await publish(live, envelope('live', { visibility: 'internal', event_type: 'live.stream.c' }));const missed2 = await publish(live, envelope('live', { visibility: 'public', event_type: 'live.stream.d' }));-- const c = await stream('topics=live.stream.*', { 'Last-Event-ID': String(first) });++ const c = await stream('topics=live.stream.*', { 'Last-Event-ID': await cursorAt(h, first) });await c.waitFor(x => x.events().length === 2);assert.deepStrictEqual(c.events().map(e => e.seq), [missed1, missed2]);assert.deepStrictEqual(c.gaps(), []);@@@ -81,17 +81,32 @@ t('Last-Event-ID resume replays what was missed, in order, then goes live', asynassert.strictEqual(c.events()[2].seq, liveSeq);c.close();// Query-string form for clients that cannot set headers.-- const q = await stream(`topics=live.stream.*&last_event_id=${missed2}`);++ const q = await stream(`topics=live.stream.*&last_event_id=${await cursorAt(h, missed2)}`);await q.waitFor(x => x.events().length === 1);assert.strictEqual(q.events()[0].seq, liveSeq);q.close();});++t('a bare-number Last-Event-ID starts at the head without replay or gap', async () => {++ const first = await publish(live, envelope('live', { visibility: 'public' }));++ const missed = await publish(live, envelope('live', { visibility: 'public' }));++ const bare = await stream('topics=live.stream.*', { 'Last-Event-ID': String(first) });++ await quiet();++ assert.deepStrictEqual(bare.events(), []);++ assert.deepStrictEqual(bare.gaps(), []);++ bare.close();++ const resumed = await stream('topics=live.stream.*', { 'Last-Event-ID': await cursorAt(h, first) });++ await resumed.waitFor(x => x.events().some(e => e.seq === missed));++ assert.deepStrictEqual(resumed.events().map(e => e.seq), [missed]);++ assert.deepStrictEqual(resumed.gaps(), []);++ resumed.close();++});++t('a cursor older than retention gets `event: gap` first', async () => {const old = await publish(live, envelope('live', { visibility: 'public' }));await h.store.prune({ retentionDays: 30, now: Date.now() + 31 * 86400000 });const kept = await publish(live, envelope('live', { visibility: 'public' }));-- const c = await stream('topics=live.stream.*', { 'Last-Event-ID': String(old - 1) });++ const c = await stream('topics=live.stream.*', { 'Last-Event-ID': await cursorAt(h, old - 1) });await c.waitFor(x => x.events().length === 1);const gaps = c.gaps();assert.strictEqual(gaps.length, 1);@@@ -101,7 +116,7 @@ t('a cursor older than retention gets `event: gap` first', async () => {assert.ok(c.messages.findIndex(m => m.event === 'gap') < c.messages.findIndex(m => !m.event), 'gap comes before events');assert.strictEqual(c.events()[0].seq, kept);c.close();-- const ahead = await stream('topics=live.stream.*', { 'Last-Event-ID': String(kept + 1000) });++ const ahead = await stream('topics=live.stream.*', { 'Last-Event-ID': await cursorAt(h, kept + 1000) });await ahead.waitFor(x => x.gaps().length === 1);assert.strictEqual(ahead.gaps()[0].reason, 'cursor_ahead');ahead.close();@@@ -14,7 +14,7 @@ const path = require('path');const { ids } = require('openvibe-contracts');const apps = require('../server/apps');const { parseDirective } = require('../server/redaction');--const { boot, request, serviceToken, userToken, appToken, envelope, subscriber, sse, sleep, suite } = require('./helpers');++const { boot, request, serviceToken, userToken, appToken, envelope, subscriber, sse, sleep, suite, cursorAt } = require('./helpers');const t = suite('redaction');let h;@@@ -97,7 +97,7 @@ t('before deletion the message is replayable (the leak as it was)', async () =>const env = created(101);msgEventId = env.event_id;msgSeq = (await publish(chat, env)).seq;-- const anon = await stream('topics=chat.message.*', { 'Last-Event-ID': String(msgSeq - 1) });++ const anon = await stream('topics=chat.message.*', { 'Last-Event-ID': await cursorAt(h, msgSeq - 1) });await anon.waitFor(c => c.events().length === 1);assert.strictEqual(anon.events()[0].event.payload.text, SECRET_TEXT);anon.close();@@@ -119,7 +119,7 @@ t('chat.message.deleted turns the stored event into a tombstone at the same seq'});t('anonymous SSE replay: the tombstone, never the text', async () => {-- const anon = await stream('topics=chat.message.*', { 'Last-Event-ID': String(msgSeq - 1) });++ const anon = await stream('topics=chat.message.*', { 'Last-Event-ID': await cursorAt(h, msgSeq - 1) });await anon.waitFor(c => c.events().length === 2);const [a, b] = anon.events();assert.deepStrictEqual([a.seq, b.seq], [msgSeq, delSeq], 'sequence intact');@@@ -130,9 +130,9 @@ t('anonymous SSE replay: the tombstone, never the text', async () => {});t('signed-in SSE replay (cookie and Bearer) and service replay: the tombstone too', async () => {-- const byCookie = await stream('topics=chat.message.*', { Cookie: `ov_token=${userToken({ subjectId: alice })}`, 'Last-Event-ID': String(msgSeq - 1) });-- const byBearer = await stream('topics=chat.*', { Authorization: `Bearer ${userToken()}`, 'Last-Event-ID': String(msgSeq - 1) });-- const svc = await stream('topics=chat.message.*', { Authorization: `Bearer ${serviceToken('search', ['events.event.read'])}`, 'Last-Event-ID': String(msgSeq - 1) });++ const byCookie = await stream('topics=chat.message.*', { Cookie: `ov_token=${userToken({ subjectId: alice })}`, 'Last-Event-ID': await cursorAt(h, msgSeq - 1) });++ const byBearer = await stream('topics=chat.*', { Authorization: `Bearer ${userToken()}`, 'Last-Event-ID': await cursorAt(h, msgSeq - 1) });++ const svc = await stream('topics=chat.message.*', { Authorization: `Bearer ${serviceToken('search', ['events.event.read'])}`, 'Last-Event-ID': await cursorAt(h, msgSeq - 1) });for (const c of [byCookie, byBearer, svc]) {await c.waitFor(x => x.events().length === 2);assert.deepStrictEqual(c.events().map(e => e.seq), [msgSeq, delSeq]);@@@ -143,7 +143,7 @@ t('signed-in SSE replay (cookie and Bearer) and service replay: the tombstone to});t('pull (/api/v1/events), by id, and developer-app reads: the tombstone', async () => {-- let r = await request(h.base, 'GET', `/api/v1/events?topic=chat.message.*&after_seq=${msgSeq - 1}`, { token: reader });++ let r = await request(h.base, 'GET', `/api/v1/events?topic=chat.message.*&after=${await cursorAt(h, msgSeq - 1)}`, { token: reader });assert.strictEqual(r.status, 200);assert.deepStrictEqual(r.body.events.map(e => e.seq), [msgSeq, delSeq], 'no hole in the sequence');assertTombstone(r.body.events[0].event, delEventId);@@@ -151,7 +151,7 @@ t('pull (/api/v1/events), by id, and developer-app reads: the tombstone', asyncassertTombstone(r.body.event, delEventId);assert.strictEqual(r.body.seq, msgSeq);const app = appToken({ env: 'production', cap: ['events.app.read'] });-- r = await request(h.base, 'GET', `/api/v1/events?topic=chat.*&after_seq=${msgSeq - 1}`, { token: app });++ r = await request(h.base, 'GET', `/api/v1/events?topic=chat.*&after=${await cursorAt(h, msgSeq - 1)}`, { token: app });assert.strictEqual(r.status, 200, r.text);assert.deepStrictEqual(r.body.events.map(e => e.seq), [msgSeq, delSeq]);assertTombstone(r.body.events[0].event, delEventId);@@@ -272,7 +272,7 @@ t('public replay window: browsers get a gap for public events older than 5 minutconst recent = await post(chat, created(2));const topics = 'topics=chat.message.*,network.notification.*';-- const anon = await sse(w.base, `/realtime/stream?${topics}`, { headers: { 'Last-Event-ID': String(oldPublic - 1) } });++ const anon = await sse(w.base, `/realtime/stream?${topics}`, { headers: { 'Last-Event-ID': await cursorAt(w, oldPublic - 1) } });await anon.waitFor(c => c.events().length === 1);assert.deepStrictEqual(anon.events().map(e => e.seq), [recent]);const gap = anon.gaps()[0];@@@ -280,16 +280,16 @@ t('public replay window: browsers get a gap for public events older than 5 minutassert.deepStrictEqual([gap.from_seq, gap.to_seq, gap.window_seconds], [oldPublic, recent - 1, 300]);assert.ok(anon.messages.findIndex(m => m.event === 'gap') < anon.messages.findIndex(m => !m.event), 'gap first');-- const user = await sse(w.base, `/realtime/stream?${topics}`, { headers: { Cookie: `ov_token=${userToken({ subjectId: alice })}`, 'Last-Event-ID': String(oldPublic - 1) } });++ const user = await sse(w.base, `/realtime/stream?${topics}`, { headers: { Cookie: `ov_token=${userToken({ subjectId: alice })}`, 'Last-Event-ID': await cursorAt(w, oldPublic - 1) } });await user.waitFor(c => c.events().length === 2);assert.deepStrictEqual(user.events().map(e => e.seq), [oldMine, recent], 'own subject events keep the whole retention; old public ones do not');assert.strictEqual(user.gaps()[0].reason, 'public_window');-- const svc = await sse(w.base, `/realtime/stream?${topics}`, { headers: { Authorization: `Bearer ${serviceToken('search', ['events.event.read'])}`, 'Last-Event-ID': String(oldPublic - 1) } });++ const svc = await sse(w.base, `/realtime/stream?${topics}`, { headers: { Authorization: `Bearer ${serviceToken('search', ['events.event.read'])}`, 'Last-Event-ID': await cursorAt(w, oldPublic - 1) } });await svc.waitFor(c => c.events().length === 3);assert.deepStrictEqual(svc.gaps(), [], 'services are not windowed');-- const fresh = await sse(w.base, `/realtime/stream?${topics}`, { headers: { 'Last-Event-ID': String(recent - 1) } });++ const fresh = await sse(w.base, `/realtime/stream?${topics}`, { headers: { 'Last-Event-ID': await cursorAt(w, recent - 1) } });await fresh.waitFor(c => c.events().length === 1);assert.deepStrictEqual(fresh.gaps(), [], 'a reconnect inside the window is seamless');for (const c of [anon, user, svc, fresh]) c.close();@@@ -297,7 +297,7 @@ t('public replay window: browsers get a gap for public events older than 5 minutconst none = await boot({ env: { REALTIME_PUBLIC_REPLAY_SECONDS: '0' } });try {const s = await (async () => (await request(none.base, 'POST', '/api/v1/events', { token: chat, body: created(3) })).body.seq)();-- const c = await sse(none.base, '/realtime/stream?topics=chat.message.*', { headers: { 'Last-Event-ID': String(s - 1) } });++ const c = await sse(none.base, '/realtime/stream?topics=chat.message.*', { headers: { 'Last-Event-ID': await cursorAt(none, s - 1) } });await c.waitFor(x => x.gaps().length === 1);await sleep(50);assert.deepStrictEqual(c.events(), [], 'window 0: no public replay at all');@@@ -13,7 +13,7 @@// positive controls show each route works for its owner.// node test/security-idor.test.jsconst assert = require('assert');--const { suite, appToken, envelope, request } = require('./helpers');++const { suite, appToken, envelope, request, cursorAt } = require('./helpers');const { buildWorld, ALL } = require('./security-world');const apps = require('../server/apps');@@@ -116,7 +116,7 @@ t('deliveries, replay and checkpoints across consumers', async () => {}// Checkpoints of another project's pattern.for (const [token, topic] of [[tok.appA, `app.${ids.keyB}.*`], [tok.appB, `app.${ids.keyA}.*`], [tok.appA, '*']]) {-- const r = await request(w.base, 'PUT', '/api/v1/checkpoints', { token, body: { topic, cursor: 99 } });++ const r = await request(w.base, 'PUT', '/api/v1/checkpoints', { token, body: { topic, cursor: await cursorAt(w.h, 99) } });if (r.status !== 403) bad.push(`checkpoint ${topic} → ${r.status}`);}assert.deepStrictEqual(bad, [], bad.join('\n'));@@@ -15,7 +15,7 @@// - subscription listings, single reads and checkpoints are the caller's own// node test/security-private.test.jsconst assert = require('assert');--const { suite, appToken, request } = require('./helpers');++const { suite, appToken, request, cursorAt } = require('./helpers');const { buildWorld } = require('./security-world');const crawler = require('./security-crawl');@@@ -38,7 +38,7 @@ t('pull reads: each caller gets exactly its scope, by every topic pattern and evconst token = who === 'anonymous' ? undefined : tok[who] || tok.userX;let got = [];for (const topic of topicsList) {-- const r = await request(w.base, 'GET', `/api/v1/events?topic=${encodeURIComponent(topic)}&after_seq=0&limit=1000`, { token });++ const r = await request(w.base, 'GET', `/api/v1/events?topic=${encodeURIComponent(topic)}&limit=1000`, { token });got = got.concat(seenIn(r.text));}for (const [name, id] of Object.entries(ev)) {@@@ -57,7 +57,7 @@ t('pull reads: each caller gets exactly its scope, by every topic pattern and evt('realtime: visibility by viewer, replayed from the start, app events never streamed', async () => {const { tok, ids } = w;const topics = encodeURIComponent('live.*,network.notification.*,app.*');-- const open = async (headers = {}, extra = '') => crawler.hit(w.base, 'GET', `/realtime/stream?topics=${topics}&last_event_id=0${extra}`, { headers, streamMs: 400 });++ const open = async (headers = {}, extra = '') => crawler.hit(w.base, 'GET', `/realtime/stream?topics=${topics}&last_event_id=${await cursorAt(w.h, 0)}${extra}`, { headers, streamMs: 400 });const cases = [['anonymous', {}, '', ['public']],['x by ticket', {}, `&ticket=${w.realtimeTicket({ subjectId: ids.x })}`, ['public', 'subjectX']],@@@ -131,18 +131,18 @@ t('subscriptions and checkpoints are the caller\'s own', async () => {if (r.status < 400) bad.push(`${who} reads live's subscription (${r.status})`);}// Checkpoints are keyed by the calling principal.-- let r = await request(w.base, 'PUT', '/api/v1/checkpoints', { token: tok.live, body: { topic: 'live.*', cursor: 5 } });++ let r = await request(w.base, 'PUT', '/api/v1/checkpoints', { token: tok.live, body: { topic: 'live.*', cursor: await cursorAt(w.h, 5) } });assert.strictEqual(r.status, 200);-- r = await request(w.base, 'PUT', '/api/v1/checkpoints', { token: tok.appA, body: { topic: `app.${ids.keyA}.*`, cursor: 3 } });++ r = await request(w.base, 'PUT', '/api/v1/checkpoints', { token: tok.appA, body: { topic: `app.${ids.keyA}.*`, cursor: await cursorAt(w.h, 3) } });assert.strictEqual(r.status, 200);for (const [who, topic] of [['media', 'live.*'], ['reader', 'live.*'], ['appA2', `app.${ids.keyA}.*`]]) {r = await request(w.base, 'GET', `/api/v1/checkpoints?topic=${encodeURIComponent(topic)}`, { token: tok[who] });-- if (r.status === 200 && r.body.cursor !== 0) bad.push(`${who} reads another consumer's checkpoint (${r.body.cursor})`);++ if (r.status === 200 && r.body.cursor !== null) bad.push(`${who} reads another consumer's checkpoint (${r.body.cursor})`);}r = await request(w.base, 'GET', `/api/v1/checkpoints?topic=${encodeURIComponent(`app.${ids.keyA}.*`)}`, { token: tok.appB });assert.strictEqual(r.status, 403, 'another project\'s pattern is refused');r = await request(w.base, 'GET', '/api/v1/checkpoints?topic=live.*', { token: tok.live });-- assert.strictEqual(r.body.cursor, 5, 'positive control');++ assert.strictEqual(r.body.cursor, await cursorAt(w.h, 5), 'positive control');assert.deepStrictEqual(bad, [], bad.join('\n'));});@@@ -18,7 +18,7 @@ const assert = require('assert');const crypto = require('crypto');const fs = require('fs');const path = require('path');--const { suite, serviceToken, appToken, userToken, ISSUER } = require('./helpers');++const { suite, serviceToken, appToken, userToken, ISSUER, cursorAt } = require('./helpers');const { buildWorld } = require('./security-world');const crawler = require('./security-crawl');@@@ -89,8 +89,8 @@ t('crawl: no GET, refusal or error carries a secret, a token or the key', asyncreturn n;};const values = (name) => ({ id: [subs.live.id, subs.media.id, subs.appA.id, subs.appB.id, ev.public, ev.internal, ev.appA, 'sub_00000000000000000000000000', 'x', '..%2f'] }[name] || ['x']);-- const queries = ['topic=*', 'topic=live.*', `topic=app.${ids.keyA}.*`, `topic=app.${ids.keyB}.*,live.*`, 'status=dead', `subscription_id=${subs.appB.id}`, 'after_seq=0&limit=1000',-- 'topics=live.*,network.notification.*&last_event_id=0', 'debug=1&include=secret'];++ const queries = ['topic=*', 'topic=live.*', `topic=app.${ids.keyA}.*`, `topic=app.${ids.keyB}.*,live.*`, 'status=dead', `subscription_id=${subs.appB.id}`, 'limit=1000',++ `topics=live.*,network.notification.*&last_event_id=${await cursorAt(w.h, 0)}`, 'debug=1&include=secret'];const paths = crawler.pathsFor(w.routes, values, { method: 'get', queries, extra: ['/nope', '/api/nope', '/.env', '/data/pglite/PG_VERSION', '/package.json', '/api/v1/subscriptions/../../package.json'] });const gets = await crawler.crawl(w.base, paths, people, needlesFor);const found = [...gets.found];@@@ -102,7 +102,7 @@ t('after the hour: one events.usage.recorded per rollup, valid, internal, delive});t('apps never see the rollups', async () => {-- const r = await request(h.base, 'GET', '/api/v1/events?topic=events.*&after_seq=0', { token });++ const r = await request(h.base, 'GET', '/api/v1/events?topic=events.*', { token });assert.strictEqual(r.status, 200);assert.strictEqual(r.body.events.length, 0);});
@@ -114,7 +114,7 @@ Each delivery is `POST <endpoint>` with body `{ "event": <envelope>, "seq": n }`
@@ -114,7 +114,7 @@ Each delivery is `POST <endpoint>` with body `{ "event": <envelope>, "seq": n }`
Operators: `GET /api/v1/deliveries?status=dead` is the dead-letter queue; `POST /api/v1/deliveries/replay { subscription_id, event_ids: [...] }` or `{ subscription_id, from_seq }` requeues retained events (that is also how a new subscription catches up on history).
Operators: `GET /api/v1/deliveries?status=dead` is the dead-letter queue; `POST /api/v1/deliveries/replay { subscription_id, event_ids: [...] }` or `{ subscription_id, from_seq }` requeues retained events (that is also how a new subscription catches up on history).
-Pull consumers: start with `GET /api/v1/events?topic=media.vod.*&limit=100`, then request `GET /api/v1/events?topic=media.vod.*&after=<next_cursor>&limit=100`, using the previous response's opaque `next_cursor` ([ADR-042](https://github.com/OpenVibers/OpenVibe.Contracts/blob/main/docs/adr/ADR-042-events-fabric.md) decision 7). The response carries `{ events: [{ seq, cursor, event }], next_cursor, next_after_seq, latest_seq, latest_cursor }` (`latest_cursor` is the head as an opaque cursor, for a consumer that starts at "now" without history), plus `gap: { from_seq, to_seq }` when the position is older than both the hot store and the replay tier (below) or from another retention epoch. Use `next_cursor` even when no events match, since it advances past scanned events. Numeric `after_seq=<position>` remains accepted for one release; when both are supplied, `after` takes precedence. The numeric `seq`, `next_after_seq` and `latest_seq` fields also remain for one release. `PUT /api/v1/checkpoints { topic, cursor }` stores a consumer's position here if it has nowhere better (an opaque cursor string is accepted in place of the integer; the answer carries its `epoch`, `carrier` and `next_cursor`).
+Pull consumers: start with `GET /api/v1/events?topic=media.vod.*&limit=100`, then request `GET /api/v1/events?topic=media.vod.*&after=<next_cursor>&limit=100`, using the previous response's opaque `next_cursor` ([ADR-042](https://github.com/OpenVibers/OpenVibe.Contracts/blob/main/docs/adr/ADR-042-events-fabric.md) decision 7). The response carries `{ events: [{ seq, cursor, event }], next_cursor, latest_cursor }` (`latest_cursor` is the head as an opaque cursor, for a consumer that starts at "now" without history), plus `gap: { from_seq, to_seq }` when the position is older than both the hot store and the replay tier (below) or from another retention epoch. Use `next_cursor` even when no events match, since it advances past scanned events. A position is only ever an opaque cursor: `after_seq` is refused with 400 (never read as "from the start"), and the per-event `seq` is informational. `PUT /api/v1/checkpoints { topic, cursor, carrier? }` stores a consumer's cursor here if it has nowhere better (a cursor string; a number is refused); `GET /api/v1/checkpoints?topic=` and the PUT answer `{ consumer, topic, cursor, carrier, updated_at }`, with `cursor` as stored (null when none is).
### Retention: three tiers
### Retention: three tiers
@@ -290,7 +290,7 @@ void connect();
@@ -290,7 +290,7 @@ void connect();
- Conversely, a user JWT that carries `typ` or `purpose` (a ticket, a FedCM assertion) is not a session here.
- Conversely, a user JWT that carries `typ` or `purpose` (a ticket, a FedCM assertion) is not a session here.
- A person's topic: `network.notification.created` (Network's outbox; visibility `subject`, subject the recipient). `topics=network.notification.*` streams a person's own notifications and nobody else's. There is no `user:<id>` topic: it is not a valid pattern (400 `realtime.bad_topic`), and subject visibility already does its job.
- A person's topic: `network.notification.created` (Network's outbox; visibility `subject`, subject the recipient). `topics=network.notification.*` streams a person's own notifications and nobody else's. There is no `user:<id>` topic: it is not a valid pattern (400 `realtime.bad_topic`), and subject visibility already does its job.
- Visibility: `public` events go to anyone subscribed to the topic; `subject` events only to the user whose subject id (`usr_…`) is the event's `actor.id` or its user `subject.id`; `internal` events never reach a browser. A guessed topic yields nothing.
- Visibility: `public` events go to anyone subscribed to the topic; `subject` events only to the user whose subject id (`usr_…`) is the event's `actor.id` or its user `subject.id`; `internal` events never reach a browser. A guessed topic yields nothing.
-- Resume: the SSE `id` is the cursor (ADR-042 decision 7; a bare seq is accepted for one release), so the browser's automatic `Last-Event-ID` (or `?last_event_id=`) replays what was missed. A cursor older than both the hot store and the replay tier — or from another retention epoch (`reason: "epoch"`) — first gets `event: gap` (`{ reason, from_seq, to_seq }`), as does a replay that hits `REALTIME_REPLAY_MAX`.
+- Resume: the SSE `id` is the cursor (ADR-042 decision 7), so the browser's automatic `Last-Event-ID` (or `?last_event_id=`) replays what was missed. Only a cursor is a position: anything else, a bare number included, is ignored and the stream starts at the head. A cursor older than both the hot store and the replay tier — or from another retention epoch (`reason: "epoch"`) — first gets `event: gap` (`{ reason, from_seq, to_seq }`), as does a replay that hits `REALTIME_REPLAY_MAX`.
- Public replay window: browsers (signed out or signed in) are replayed `public` events received in the last `REALTIME_PUBLIC_REPLAY_SECONDS` (300; 0 = none), enough to ride out a reconnect. Older public events are not replayed: the stream opens with `event: gap` (`reason: "public_window"`, `window_seconds`), and the client refetches state from the owning service. `subject` events addressed to the viewer, and service viewers, keep the whole retention. This keeps the stream from paging through a month of public history, chat lines included; nothing in the network needs more (no browser surface replays public events today).
- Public replay window: browsers (signed out or signed in) are replayed `public` events received in the last `REALTIME_PUBLIC_REPLAY_SECONDS` (300; 0 = none), enough to ride out a reconnect. Older public events are not replayed: the stream opens with `event: gap` (`reason: "public_window"`, `window_seconds`), and the client refetches state from the owning service. `subject` events addressed to the viewer, and service viewers, keep the whole retention. This keeps the stream from paging through a month of public history, chat lines included; nothing in the network needs more (no browser surface replays public events today).
- Redacted events are replayed as their tombstones ([Redaction](#redaction)).
- Redacted events are replayed as their tombstones ([Redaction](#redaction)).
- Heartbeat comment every 25 s; at most 20 topics per connection and `REALTIME_MAX_CONNECTIONS` (2000) overall; CORS with credentials for `https://*.openvibe.*` only.
- Heartbeat comment every 25 s; at most 20 topics per connection and `REALTIME_MAX_CONNECTIONS` (2000) overall; CORS with credentials for `https://*.openvibe.*` only.
@@ -387,7 +387,7 @@ implemented service.
@@ -387,7 +387,7 @@ implemented service.
Part of the [OpenVibe network](https://openvibe.network). Built in the open by [OpenVibers](https://github.com/OpenVibers).
Part of the [OpenVibe network](https://openvibe.network). Built in the open by [OpenVibers](https://github.com/OpenVibers).
<!-- versions:start -->
<!-- versions:start -->
-- openvibe-contracts: v0.129.0
+- openvibe-contracts: v0.132.0
- openvibe-sdk: v0.40.0
- openvibe-sdk: v0.40.0
- openvibe-shared: v3.0.0
- openvibe-shared: v3.0.0
<!-- versions:end -->
<!-- versions:end -->
@@ -7,7 +7,7 @@
@@ -7,7 +7,7 @@
"runtime": "node 22, express, PostgreSQL 18 (openvibe-sdk/db, async) + Valkey 9",
"runtime": "node 22, express, PostgreSQL 18 (openvibe-sdk/db, async) + Valkey 9",
"port": 4300,
"port": 4300,
"domain": "openvibe.events",
"domain": "openvibe.events",
- "contracts": "openvibe-contracts v0.129.0",
+ "contracts": "openvibe-contracts v0.132.0",
"sdk": "openvibe-sdk v0.40.0",
"sdk": "openvibe-sdk v0.40.0",
"shared": "openvibe-shared v3.0.0",
"shared": "openvibe-shared v3.0.0",
"capabilities": [
"capabilities": [
@@ -12,7 +12,7 @@
@@ -12,7 +12,7 @@
"dotenv": "^16.4.7",
"dotenv": "^16.4.7",
"express": "^4.21.2",
"express": "^4.21.2",
"iovalkey": "^0.4.0",
"iovalkey": "^0.4.0",
- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.129.0",
+ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.132.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.40.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.40.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v3.0.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v3.0.0",
"pg": "^8.23.0"
"pg": "^8.23.0"
@@ -828,9 +828,9 @@
@@ -828,9 +828,9 @@
}
}
},
},
"node_modules/openvibe-contracts": {
"node_modules/openvibe-contracts": {
- "version": "0.129.0",
- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.129.0",
- "integrity": "sha512-Ly0F/ap6wmAf8hYpNrOhMLsU3cZYBjsFDW2P1F/xVmHVsRDzAPAZDXtFcohP0UNeZgyxgrR0BI0pPG5guD/Efw==",
+ "version": "0.132.0",
+ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.132.0",
+ "integrity": "sha512-08GuHutggWYtGcGrnBLilWco/K715Bl7ZAdfYX7kXeeHguWTIhbWZsSF3JYcH9iq6KKtRB7XCAbmPFKScleD9Q==",
"license": "AGPL-3.0-only",
"license": "AGPL-3.0-only",
"dependencies": {
"dependencies": {
"ajv": "^8.17.1",
"ajv": "^8.17.1",
@@ -24,7 +24,7 @@
@@ -24,7 +24,7 @@
"dotenv": "^16.4.7",
"dotenv": "^16.4.7",
"express": "^4.21.2",
"express": "^4.21.2",
"iovalkey": "^0.4.0",
"iovalkey": "^0.4.0",
- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.129.0",
+ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.132.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.40.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.40.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v3.0.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v3.0.0",
"pg": "^8.23.0"
"pg": "^8.23.0"
@@ -2,20 +2,21 @@
@@ -2,20 +2,21 @@
/**
/**
* Pull consumers (service token with events.event.read):
* Pull consumers (service token with events.event.read):
*
*
- * GET /api/v1/events?topic=media.vod.*[,…]&after_seq=0&limit=100
- * -> { events: [{ seq, cursor, event }], next_after_seq, next_cursor, latest_seq, latest_cursor, gap? }
- * `latest_cursor` is the head as an opaque cursor: a consumer that starts at "now" (skipping history)
- * stores it, so it never needs the numeric latest_seq.
- * `after=` takes an opaque cursor beside after_seq (ADR-042 decision 7); a cursor from another
- * retention epoch answers `gap` — never a silent restart. `gap` ({ from_seq, to_seq }) means
+ * GET /api/v1/events?topic=media.vod.*[,…][&after=<cursor>]&limit=100
+ * -> { events: [{ seq, cursor, event }], next_cursor, latest_cursor, gap? }
+ * A position is only ever an opaque cursor (ADR-042 decision 7): `after=` takes one, and without it the
+ * page starts at the oldest retained event. `after_seq` is refused (400), never read as "from the start".
+ * `latest_cursor` is the head: a consumer that starts at "now" (skipping history) stores it. The per-event
+ * `seq` is informational. A cursor from another retention epoch answers `gap` — never a silent restart.
+ * `gap` ({ from_seq, to_seq }) means
* events after the position were already pruned by retention (or the epoch changed). The pull
* events after the position were already pruned by retention (or the epoch changed). The pull
* spans the hot store and the replay tier (decision 8) with the one cursor: `gap` only outside both.
* spans the hot store and the replay tier (decision 8) with the one cursor: `gap` only outside both.
* Keep next_cursor as the cursor; it moves past events that did not match.
* Keep next_cursor as the cursor; it moves past events that did not match.
* GET /api/v1/events/:event_id -> { seq, cursor, event } (hot or replay)
* GET /api/v1/events/:event_id -> { seq, cursor, event } (hot or replay)
- * GET /api/v1/checkpoints?topic=… / PUT /api/v1/checkpoints { topic, cursor }
- * a consumer's own stored cursor per topic pattern (consumer = calling principal). `cursor` is
- * the numeric position; PUT also accepts an opaque cursor string, and both answers carry the
- * position's epoch and carrier so the cursor can be rebuilt.
+ * GET /api/v1/checkpoints?topic=… / PUT /api/v1/checkpoints { topic, cursor, carrier? }
+ * a consumer's own stored cursor per topic pattern (consumer = calling principal): `cursor` is an
+ * opaque cursor string (a number is refused), and both answers return it as stored, with its carrier
+ * ({ consumer, topic, cursor, carrier, updated_at }; cursor null when none is stored).
*
*
* Developer apps (events.app.read) use the same routes with an app token: only their project's
* Developer apps (events.app.read) use the same routes with an app token: only their project's
* events in the token's environment plus public first-party events (server/apps.js); every topic
* events in the token's environment plus public first-party events (server/apps.js); every topic
@@ -79,18 +80,19 @@ function readRouter({ store, auth, worker, limits }) {
@@ -79,18 +80,19 @@ function readRouter({ store, auth, worker, limits }) {
if (Number.isNaN(limit)) {
if (Number.isNaN(limit)) {
return http.sendProblem(res, 400, 'events.bad_request', { detail: 'limit must be 1..1000', ctx });
return http.sendProblem(res, 400, 'events.bad_request', { detail: 'limit must be 1..1000', ctx });
}
}
+ // A position is an opaque cursor and nothing else: a number would silently restart a consumer that still sends one.
+ if (req.query.after_seq !== undefined) {
+ return http.sendProblem(res, 400, 'events.bad_request', { detail: "after_seq is retired: pass after=<cursor> (a page's next_cursor, or latest_cursor for the head)", ctx });
+ }
const epoch = await store.epoch();
const epoch = await store.epoch();
- // `after=` is an opaque cursor; `after_seq` stays for one release. A cursor from another epoch cannot be
- // mapped onto this store, so it is answered with a gap (never treated as a position here).
- let after = intParam(req.query.after_seq, 0, 0, Number.MAX_SAFE_INTEGER);
+ // A cursor from another epoch cannot be mapped onto this store, so it is answered with a gap.
+ let after = 0;
let epochMismatch = false;
let epochMismatch = false;
if (req.query.after !== undefined && String(req.query.after) !== '') {
if (req.query.after !== undefined && String(req.query.after) !== '') {
const decoded = cursor.decode(String(req.query.after));
const decoded = cursor.decode(String(req.query.after));
if (!decoded) return http.sendProblem(res, 400, 'events.bad_request', { detail: 'after must be an opaque cursor', ctx });
if (!decoded) return http.sendProblem(res, 400, 'events.bad_request', { detail: 'after must be an opaque cursor', ctx });
after = decoded.seq;
after = decoded.seq;
epochMismatch = decoded.epoch !== epoch;
epochMismatch = decoded.epoch !== epoch;
- } else if (Number.isNaN(after)) {
- return http.sendProblem(res, 400, 'events.bad_request', { detail: 'after_seq must be >= 0', ctx });
}
}
const oldest = await store.oldestSeq();
const oldest = await store.oldestSeq();
const out = {};
const out = {};
@@ -104,10 +106,8 @@ function readRouter({ store, auth, worker, limits }) {
@@ -104,10 +106,8 @@ function readRouter({ store, auth, worker, limits }) {
}
}
const { rows, cursor: scanned } = await store.scan(from, { patterns, limit, accept: acceptFor(req.principal, patterns) });
const { rows, cursor: scanned } = await store.scan(from, { patterns, limit, accept: acceptFor(req.principal, patterns) });
out.events = rows.map(r => ({ seq: r.seq, cursor: cursor.encode(r.seq, epoch), event: rowToEnvelope(r) }));
out.events = rows.map(r => ({ seq: r.seq, cursor: cursor.encode(r.seq, epoch), event: rowToEnvelope(r) }));
- out.next_after_seq = scanned;
out.next_cursor = cursor.encode(scanned, epoch);
out.next_cursor = cursor.encode(scanned, epoch);
- out.latest_seq = await store.lastSeq();
- out.latest_cursor = cursor.encode(out.latest_seq, epoch);
+ out.latest_cursor = cursor.encode(await store.lastSeq(), epoch);
res.json(out);
res.json(out);
});
});
@@ -120,6 +120,10 @@ function readRouter({ store, auth, worker, limits }) {
@@ -120,6 +120,10 @@ function readRouter({ store, auth, worker, limits }) {
});
});
const consumerOf = (req) => req.principal.service || req.principal.sub;
const consumerOf = (req) => req.principal.service || req.principal.sub;
+ // The stored position and epoch, handed back as the opaque cursor they make (null when nothing is stored).
+ const checkpointView = (consumer, topic, cp) => ({
+ consumer, topic, cursor: cp ? cursor.encode(Number(cp.cursor), Number(cp.epoch)) : null, carrier: cp ? cp.carrier || null : null, updated_at: cp ? cp.updated_at : null,
+ });
router.get('/api/v1/checkpoints', canRead, limits('events.checkpoint.read'), async (req, res) => {
router.get('/api/v1/checkpoints', canRead, limits('events.checkpoint.read'), async (req, res) => {
const topic = String(req.query.topic || '');
const topic = String(req.query.topic || '');
@@ -127,36 +131,21 @@ function readRouter({ store, auth, worker, limits }) {
@@ -127,36 +131,21 @@ function readRouter({ store, auth, worker, limits }) {
const scopeErr = scopeError(req.principal, [topic]);
const scopeErr = scopeError(req.principal, [topic]);
if (scopeErr) return http.sendProblem(res, 403, 'events.topic_not_allowed', { detail: scopeErr, ctx: req.ov });
if (scopeErr) return http.sendProblem(res, 403, 'events.topic_not_allowed', { detail: scopeErr, ctx: req.ov });
const cp = await store.getCheckpoint(consumerOf(req), topic);
const cp = await store.getCheckpoint(consumerOf(req), topic);
- const epoch = cp ? cp.epoch : await store.epoch();
- const position = cp ? cp.cursor : 0;
- res.json({
- consumer: consumerOf(req), topic, cursor: position, epoch, carrier: cp ? cp.carrier : null,
- next_cursor: cursor.encode(position, epoch), updated_at: cp ? cp.updated_at : null,
- });
+ res.json(checkpointView(consumerOf(req), topic, cp));
});
});
router.put('/api/v1/checkpoints', canRead, limits('events.checkpoint.write', { minute: 600, hour: 20000 }), async (req, res) => {
router.put('/api/v1/checkpoints', canRead, limits('events.checkpoint.write', { minute: 600, hour: 20000 }), async (req, res) => {
const b = req.body || {};
const b = req.body || {};
- // `cursor` stays the numeric position; an opaque cursor string is accepted in its place and decoded to the
- // position and epoch it names (ADR-042 decision 7). `carrier` records the carrier the position is on.
- let position = null;
- let atEpoch = null;
- if (typeof b.cursor === 'string') {
- const decoded = cursor.decode(b.cursor);
- if (!decoded) return http.sendProblem(res, 400, 'events.bad_request', { detail: 'cursor must be an opaque cursor or an integer >= 0', ctx: req.ov });
- position = decoded.seq;
- atEpoch = decoded.epoch;
- } else if (Number.isInteger(b.cursor) && b.cursor >= 0) {
- position = b.cursor;
- if (Number.isInteger(b.epoch) && b.epoch >= 0) atEpoch = b.epoch;
- }
- if (!topics.isValidPattern(b.topic) || position === null) {
- return http.sendProblem(res, 400, 'events.bad_request', { detail: 'topic (pattern) and cursor (integer >= 0 or an opaque cursor) are required', ctx: req.ov });
+ // `cursor` is an opaque cursor, decoded to the position and epoch it names (ADR-042 decision 7); a number is
+ // refused. `carrier` records the carrier the position is on.
+ const decoded = typeof b.cursor === 'string' ? cursor.decode(b.cursor) : null;
+ if (!topics.isValidPattern(b.topic) || !decoded) {
+ return http.sendProblem(res, 400, 'events.bad_request', { detail: "topic (pattern) and cursor (an opaque cursor: a page's next_cursor) are required", ctx: req.ov });
}
}
const scopeErr = scopeError(req.principal, [b.topic]);
const scopeErr = scopeError(req.principal, [b.topic]);
if (scopeErr) return http.sendProblem(res, 403, 'events.topic_not_allowed', { detail: scopeErr, ctx: req.ov });
if (scopeErr) return http.sendProblem(res, 403, 'events.topic_not_allowed', { detail: scopeErr, ctx: req.ov });
- const cp = await store.setCheckpoint(consumerOf(req), b.topic, position, { epoch: atEpoch, carrier: typeof b.carrier === 'string' && b.carrier ? b.carrier : null });
- res.json({ consumer: consumerOf(req), topic: b.topic, ...cp, next_cursor: cursor.encode(cp.cursor, cp.epoch) });
+ const cp = await store.setCheckpoint(consumerOf(req), b.topic, decoded.seq, { epoch: decoded.epoch, carrier: typeof b.carrier === 'string' && b.carrier ? b.carrier : null });
+ res.json(checkpointView(consumerOf(req), b.topic, cp));
});
});
router.get('/api/v1/deliveries', isAdmin, limits('events.delivery.list'), async (req, res) => {
router.get('/api/v1/deliveries', isAdmin, limits('events.delivery.list'), async (req, res) => {
@@ -18,7 +18,8 @@
@@ -18,7 +18,8 @@
* internal service principals (token with events.event.read) only, never a browser
* internal service principals (token with events.event.read) only, never a browser
* Developer-app events (app.<project_key>.*, events.app.publish) are never streamed here, to anyone.
* Developer-app events (app.<project_key>.*, events.app.publish) are never streamed here, to anyone.
*
*
- * Resume: `Last-Event-ID` (or ?last_event_id=) is a cursor, or a bare seq for one release. Missed
+ * Resume: `Last-Event-ID` (or ?last_event_id=) is an opaque cursor; anything else (a bare number included) is not a
+ * position and is ignored, so the stream starts at the head. Missed
* events are replayed first, from the replay tier and then the hot store (ADR-042 decision 8); when the
* events are replayed first, from the replay tier and then the hot store (ADR-042 decision 8); when the
* cursor is older than both — or belongs to another retention epoch — an `event: gap` message comes
* cursor is older than both — or belongs to another retention epoch — an `event: gap` message comes
* first, so the client knows to refetch state.
* first, so the client knows to refetch state.
@@ -131,16 +132,12 @@ function createRealtime({ store, auth, config, clock = { now: () => Date.now() }
@@ -131,16 +132,12 @@ function createRealtime({ store, auth, config, clock = { now: () => Date.now() }
const epoch = await store.epoch();
const epoch = await store.epoch();
const rawLast = req.headers['last-event-id'] ?? req.query.last_event_id;
const rawLast = req.headers['last-event-id'] ?? req.query.last_event_id;
- // A cursor, or — for one release — a bare seq. Anything else is ignored, as before.
+ // Only a cursor is a position (ADR-042 decision 7); anything else is ignored and the stream starts at the head.
let lastId = null;
let lastId = null;
let epochMismatch = false;
let epochMismatch = false;
if (rawLast != null) {
if (rawLast != null) {
- const raw = String(rawLast).trim();
- if (/^\d{1,15}$/.test(raw)) lastId = Number(raw);
- else {
- const decoded = cursor.decode(raw);
- if (decoded) { lastId = decoded.seq; epochMismatch = decoded.epoch !== epoch; }
- }
+ const decoded = cursor.decode(String(rawLast).trim());
+ if (decoded) { lastId = decoded.seq; epochMismatch = decoded.epoch !== epoch; }
}
}
res.status(200);
res.status(200);
@@ -724,13 +724,14 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -724,13 +724,14 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
/**
/**
* A checkpoint is a cursor in parts: `cursor` is the numeric position, `epoch` the retention epoch it belongs to
* A checkpoint is a cursor in parts: `cursor` is the numeric position, `epoch` the retention epoch it belongs to
- * and `carrier` the carrier that position is on (ADR-042 decision 7). `cursor` stays numeric on the wire.
+ * and `carrier` the carrier that position is on (ADR-042 decision 7). The wire only ever carries the opaque cursor
+ * these parts make (server/api/read.js).
*/
*/
async function getCheckpoint(consumer, topicPattern) {
async function getCheckpoint(consumer, topicPattern) {
const r = await db.prepare('SELECT cursor, epoch, carrier, updated_at FROM consumer_checkpoints WHERE consumer = ? AND topic_pattern = ?').get(consumer, topicPattern);
const r = await db.prepare('SELECT cursor, epoch, carrier, updated_at FROM consumer_checkpoints WHERE consumer = ? AND topic_pattern = ?').get(consumer, topicPattern);
return r ? { cursor: r.cursor, epoch: r.epoch, carrier: r.carrier || null, updated_at: new Date(r.updated_at).toISOString() } : null;
return r ? { cursor: r.cursor, epoch: r.epoch, carrier: r.carrier || null, updated_at: new Date(r.updated_at).toISOString() } : null;
}
}
- /** Store a position. `epoch` defaults to the current retention epoch (a legacy numeric cursor). */
+ /** Store a position (decoded from a cursor). `epoch` defaults to the current retention epoch. */
async function setCheckpoint(consumer, topicPattern, position, { epoch: atEpoch = null, carrier = null } = {}) {
async function setCheckpoint(consumer, topicPattern, position, { epoch: atEpoch = null, carrier = null } = {}) {
const at = atEpoch == null ? await epoch() : atEpoch;
const at = atEpoch == null ? await epoch() : atEpoch;
await db.prepare(`INSERT INTO consumer_checkpoints (consumer, topic_pattern, cursor, epoch, carrier, updated_at) VALUES (?, ?, ?, ?, ?, ?)
await db.prepare(`INSERT INTO consumer_checkpoints (consumer, topic_pattern, cursor, epoch, carrier, updated_at) VALUES (?, ?, ?, ?, ?, ?)
@@ -8,7 +8,7 @@ const { verifyDeliveryV2 } = require('../lib/client');
@@ -8,7 +8,7 @@ const { verifyDeliveryV2 } = require('../lib/client');
const apps = require('../server/apps');
const apps = require('../server/apps');
const { createGuardedPost, isPublicAddress } = require('../server/egress');
const { createGuardedPost, isPublicAddress } = require('../server/egress');
const { load } = require('../server/config');
const { load } = require('../server/config');
-const { boot, request, serviceToken, appToken, envelope, subscriber, suite, sleep } = require('./helpers');
+const { boot, request, serviceToken, appToken, envelope, subscriber, suite, sleep, cursorAt } = require('./helpers');
const t = suite('apps');
const t = suite('apps');
const ALL = ['events.app.publish', 'events.app.read', 'events.app.subscribe'];
const ALL = ['events.app.publish', 'events.app.read', 'events.app.subscribe'];
@@ -72,7 +72,7 @@ function appEvent(appId, projectId, name = 'order.created', over = {}) {
@@ -72,7 +72,7 @@ function appEvent(appId, projectId, name = 'order.created', over = {}) {
});
});
}
}
const publish = async (token, body) => await request(h.base, 'POST', '/api/v1/events', { token, body });
const publish = async (token, body) => await request(h.base, 'POST', '/api/v1/events', { token, body });
-const pull = async (token, topic, after = 0) => await request(h.base, 'GET', `/api/v1/events?topic=${encodeURIComponent(topic)}&after_seq=${after}`, { token });
+const pull = async (token, topic, after = null) => await request(h.base, 'GET', `/api/v1/events?topic=${encodeURIComponent(topic)}${after === null || after === 0 ? '' : `&after=${after}`}`, { token });
t('boot', async () => {
t('boot', async () => {
await new Promise(r => receiver.listen(0, '127.0.0.1', r));
await new Promise(r => receiver.listen(0, '127.0.0.1', r));
@@ -177,9 +177,9 @@ t('read: own project in the same environment, plus public first-party events', a
@@ -177,9 +177,9 @@ t('read: own project in the same environment, plus public first-party events', a
assert.strictEqual(r.status, 404, 'sandbox events are invisible to first-party readers');
assert.strictEqual(r.status, 404, 'sandbox events are invisible to first-party readers');
// Checkpoints are per app and follow the same scope.
// Checkpoints are per app and follow the same scope.
- r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: tokA(), body: { topic: `app.${keyA}.*`, cursor: 3 } });
+ r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: tokA(), body: { topic: `app.${keyA}.*`, cursor: await cursorAt(h, 3) } });
assert.strictEqual(r.status, 200); assert.strictEqual(r.body.consumer, `app:${appA}`);
assert.strictEqual(r.status, 200); assert.strictEqual(r.body.consumer, `app:${appA}`);
- r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: tokA(), body: { topic: `app.${keyB}.*`, cursor: 3 } });
+ r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: tokA(), body: { topic: `app.${keyB}.*`, cursor: await cursorAt(h, 3) } });
assert.strictEqual(r.status, 403);
assert.strictEqual(r.status, 403);
r = await request(h.base, 'GET', `/api/v1/checkpoints?topic=app.${keyA}.*`, { token: tokB() });
r = await request(h.base, 'GET', `/api/v1/checkpoints?topic=app.${keyA}.*`, { token: tokB() });
assert.strictEqual(r.status, 403);
assert.strictEqual(r.status, 403);
@@ -12,7 +12,7 @@ const { limitsOf } = require('../server/limits');
@@ -12,7 +12,7 @@ const { limitsOf } = require('../server/limits');
const { createStore } = require('../server/store');
const { createStore } = require('../server/store');
const { exportMonth, restoreMonth, objectKey, manifestKey } = require('../scripts/events-archive');
const { exportMonth, restoreMonth, objectKey, manifestKey } = require('../scripts/events-archive');
const { localStorage, s3Storage, fromEnv, signV4 } = require('../scripts/archive-storage');
const { localStorage, s3Storage, fromEnv, signV4 } = require('../scripts/archive-storage');
-const { boot, request, serviceToken, appToken, envelope, sse, suite, tmpDir } = require('./helpers');
+const { boot, request, serviceToken, appToken, envelope, sse, suite, tmpDir, cursorAt } = require('./helpers');
const t = suite('archive');
const t = suite('archive');
const DAY = 86400000;
const DAY = 86400000;
@@ -44,7 +44,7 @@ t('a hot-pruned event is readable through replay: by id and by pull, unchanged',
@@ -44,7 +44,7 @@ t('a hot-pruned event is readable through replay: by id and by pull, unchanged',
const after = await request(h.base, 'GET', `/api/v1/events/${e.event_id}`, { token: reader });
const after = await request(h.base, 'GET', `/api/v1/events/${e.event_id}`, { token: reader });
assert.strictEqual(after.status, 200, after.text);
assert.strictEqual(after.status, 200, after.text);
assert.deepStrictEqual(after.body, before.body, 'the same seq, cursor and envelope from the replay tier');
assert.deepStrictEqual(after.body, before.body, 'the same seq, cursor and envelope from the replay tier');
- const p = await pull('topic=live.tier.one&after_seq=0');
+ const p = await pull('topic=live.tier.one');
assert.deepStrictEqual(p.body.events.map(x => x.event), [before.body.event]);
assert.deepStrictEqual(p.body.events.map(x => x.event), [before.body.event]);
assert.strictEqual(p.body.gap, undefined);
assert.strictEqual(p.body.gap, undefined);
// A publish of the same id is a duplicate with its position, never a second row.
// A publish of the same id is a duplicate with its position, never a second row.
@@ -65,7 +65,7 @@ t('a cursor across the hot/replay boundary reports no false gap (pull and SSE)',
@@ -65,7 +65,7 @@ t('a cursor across the hot/replay boundary reports no false gap (pull and SSE)',
assert.strictEqual(r.body.events[0].cursor, b.cursor);
assert.strictEqual(r.body.events[0].cursor, b.cursor);
r = await pull(`topic=live.tier.cross&after=${r.body.next_cursor}&limit=1`);
r = await pull(`topic=live.tier.cross&after=${r.body.next_cursor}&limit=1`);
assert.deepStrictEqual([r.body.events.map(x => x.seq), r.body.gap], [[c.seq], undefined], 'then hot, no gap between');
assert.deepStrictEqual([r.body.events.map(x => x.seq), r.body.gap], [[c.seq], undefined], 'then hot, no gap between');
- r = await pull(`topic=*&after_seq=${a.seq - 1}`);
+ r = await pull(`topic=*&after=${await cursorAt(h, a.seq - 1)}`);
assert.deepStrictEqual(r.body.events.map(x => x.seq), [a.seq, b.seq, c.seq]);
assert.deepStrictEqual(r.body.events.map(x => x.seq), [a.seq, b.seq, c.seq]);
assert.strictEqual(r.body.gap, undefined);
assert.strictEqual(r.body.gap, undefined);
@@ -92,7 +92,7 @@ t('redaction reaches replay rows (by id and by subject); the owner rule holds th
@@ -92,7 +92,7 @@ t('redaction reaches replay rows (by id and by subject); the owner rule holds th
assert.deepStrictEqual(r.body.event.actor, { type: 'service', id: 'live' });
assert.deepStrictEqual(r.body.event.actor, { type: 'service', id: 'live' });
assert.strictEqual(r.body.seq, e.seq);
assert.strictEqual(r.body.seq, e.seq);
}
}
- const p = await pull('topic=live.tier.secret&after_seq=0');
+ const p = await pull('topic=live.tier.secret');
assert.ok(p.body.events.every(e => e.event.payload.redacted === true && !('secret' in e.event.payload)));
assert.ok(p.body.events.every(e => e.event.payload.redacted === true && !('secret' in e.event.payload)));
});
});
@@ -108,11 +108,11 @@ t('sandbox and environment rules apply to replay rows exactly as to hot ones', a
@@ -108,11 +108,11 @@ t('sandbox and environment rules apply to replay rows exactly as to hot ones', a
await h.store.prune(tiered(31, { sandboxRetentionDays: 7 }));
await h.store.prune(tiered(31, { sandboxRetentionDays: 7 }));
assert.deepStrictEqual([await count('events_archive', pe.event_id), await count('events_archive', se.event_id), await count('events', se.event_id)], [1, 0, 0],
assert.deepStrictEqual([await count('events_archive', pe.event_id), await count('events_archive', se.event_id), await count('events', se.event_id)], [1, 0, 0],
'a production event moves; a sandbox event is deleted, never kept in replay');
'a production event moves; a sandbox event is deleted, never kept in replay');
- const topic = `topic=app.${apps.projectKey(prj)}.*&after_seq=0`;
+ const topic = `topic=app.${apps.projectKey(prj)}.*`;
assert.deepStrictEqual((await pull(topic, prod)).body.events.map(e => e.seq), [pe.seq], 'the app reads its own replay rows');
assert.deepStrictEqual((await pull(topic, prod)).body.events.map(e => e.seq), [pe.seq], 'the app reads its own replay rows');
assert.deepStrictEqual((await pull(topic, sand)).body.events, [], 'never across environments');
assert.deepStrictEqual((await pull(topic, sand)).body.events, [], 'never across environments');
assert.deepStrictEqual((await pull(topic)).body.events.map(e => e.seq), [pe.seq], 'first-party: production app events through app.*');
assert.deepStrictEqual((await pull(topic)).body.events.map(e => e.seq), [pe.seq], 'first-party: production app events through app.*');
- assert.deepStrictEqual((await pull('topic=*&after_seq=0')).body.events.filter(e => e.event.event_id === pe.event_id), [], '…and only through app.*');
+ assert.deepStrictEqual((await pull('topic=*')).body.events.filter(e => e.event.event_id === pe.event_id), [], '…and only through app.*');
assert.strictEqual((await request(h.base, 'GET', `/api/v1/events/${pe.event_id}`, { token: sand })).status, 404);
assert.strictEqual((await request(h.base, 'GET', `/api/v1/events/${pe.event_id}`, { token: sand })).status, 404);
assert.strictEqual((await request(h.base, 'GET', `/api/v1/events/${pe.event_id}`, { token: prod })).status, 200);
assert.strictEqual((await request(h.base, 'GET', `/api/v1/events/${pe.event_id}`, { token: prod })).status, 200);
});
});
@@ -148,7 +148,7 @@ t('export → restore round-trips byte-identical rows; the rows leave replay onl
@@ -148,7 +148,7 @@ t('export → restore round-trips byte-identical rows; the rows leave replay onl
assert.strictEqual(lines.length, inMonth.length);
assert.strictEqual(lines.length, inMonth.length);
assert.strictEqual(JSON.parse(await storage.get(manifestKey(month))).count, inMonth.length);
assert.strictEqual(JSON.parse(await storage.get(manifestKey(month))).count, inMonth.length);
assert.strictEqual((await request(h.base, 'GET', `/api/v1/events/${inMonth[0].id}`, { token: reader })).status, 404, 'the archive tier is offline');
assert.strictEqual((await request(h.base, 'GET', `/api/v1/events/${inMonth[0].id}`, { token: reader })).status, 404, 'the archive tier is offline');
- const gapped = await pull(`topic=*&after_seq=0`);
+ const gapped = await pull(`topic=*&after=${await cursorAt(h, 0)}`);
assert.ok(gapped.body.gap, 'a cursor into what went to the archive tier is a gap');
assert.ok(gapped.body.gap, 'a cursor into what went to the archive tier is a gap');
// A rerun is idempotent: the object keeps its rows, nothing is deleted twice.
// A rerun is idempotent: the object keeps its rows, nothing is deleted twice.
@@ -1,10 +1,9 @@
@@ -1,10 +1,9 @@
'use strict';
'use strict';
/**
/**
- * ADR-042 decision 7: opaque cursors beside the global seq. A cursor is `c1.<epoch>.<base64url(seq)>`;
- * every answer that carries a seq also carries its cursor, pull accepts `after=<cursor>` beside
- * `after_seq`, the SSE `id` is the cursor (a bare seq is accepted for one release), and checkpoints
- * store and return the position's epoch. The contract answers are validated against the pinned
- * openvibe-contracts (0.81.0, the release that added the cursor fields).
+ * ADR-042 decision 7: consumer positions are cursors only. A cursor is `c1.<epoch>.<base64url(seq)>`;
+ * event answers retain seq and carry a cursor, pull accepts `after=<cursor>`, SSE resumes from
+ * cursor IDs, and checkpoints store and return opaque cursor strings. Unchanged publish and single
+ * event answers are validated against the pinned openvibe-contracts.
*/
*/
const assert = require('assert');
const assert = require('assert');
const { validate } = require('openvibe-contracts');
const { validate } = require('openvibe-contracts');
@@ -39,7 +38,7 @@ t('encode/decode round trip (never throws on malformed input)', () => {
@@ -39,7 +38,7 @@ t('encode/decode round trip (never throws on malformed input)', () => {
}
}
});
});
-t('publish and read answers carry cursors, validated against events.publish-result@1 / read-result@1', async () => {
+t('publish and read answers carry cursors', async () => {
const epoch = await h.store.epoch();
const epoch = await h.store.epoch();
const one = await publish();
const one = await publish();
const pv = validate('events.publish-result@1', one);
const pv = validate('events.publish-result@1', one);
@@ -51,28 +50,30 @@ t('publish and read answers carry cursors, validated against events.publish-resu
@@ -51,28 +50,30 @@ t('publish and read answers carry cursors, validated against events.publish-resu
for (const r of batch.body.results) assert.deepStrictEqual(cursor.decode(r.cursor), { seq: r.seq, epoch });
for (const r of batch.body.results) assert.deepStrictEqual(cursor.decode(r.cursor), { seq: r.seq, epoch });
const page = await request(h.base, 'GET', '/api/v1/events?topic=live.vod.*', { token: reader });
const page = await request(h.base, 'GET', '/api/v1/events?topic=live.vod.*', { token: reader });
- const rv = validate('events.read-result@1', page.body);
- assert.ok(rv.valid, JSON.stringify(rv.errors));
+ assert.ok(typeof page.body.next_cursor === 'string' && typeof page.body.latest_cursor === 'string');
assert.ok(page.body.events.length >= 3);
assert.ok(page.body.events.length >= 3);
for (const e of page.body.events) assert.deepStrictEqual(cursor.decode(e.cursor), { seq: e.seq, epoch });
for (const e of page.body.events) assert.deepStrictEqual(cursor.decode(e.cursor), { seq: e.seq, epoch });
- assert.deepStrictEqual(cursor.decode(page.body.next_cursor), { seq: page.body.next_after_seq, epoch });
+ assert.deepStrictEqual(cursor.decode(page.body.next_cursor), { seq: page.body.events.at(-1).seq, epoch });
+ assert.ok(!('next_after_seq' in page.body) && !('latest_seq' in page.body));
const single = await request(h.base, 'GET', `/api/v1/events/${one.event_id}`, { token: reader });
const single = await request(h.base, 'GET', `/api/v1/events/${one.event_id}`, { token: reader });
assert.ok(validate('events.read-result@1', single.body).valid);
assert.ok(validate('events.read-result@1', single.body).valid);
assert.deepStrictEqual(cursor.decode(single.body.cursor), { seq: single.body.seq, epoch });
assert.deepStrictEqual(cursor.decode(single.body.cursor), { seq: single.body.seq, epoch });
});
});
-t('pull accepts after=<cursor> beside after_seq; next_cursor round-trips a page', async () => {
+t('pull accepts after=<cursor>; next_cursor round-trips a page and after_seq is refused', async () => {
let r = await request(h.base, 'GET', '/api/v1/events?topic=live.vod.*&limit=1', { token: reader });
let r = await request(h.base, 'GET', '/api/v1/events?topic=live.vod.*&limit=1', { token: reader });
assert.strictEqual(r.body.events.length, 1);
assert.strictEqual(r.body.events.length, 1);
const first = r.body.events[0];
const first = r.body.events[0];
- assert.strictEqual(r.body.next_after_seq, first.seq);
+ assert.strictEqual(r.body.next_cursor, first.cursor);
r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after=${encodeURIComponent(r.body.next_cursor)}`, { token: reader });
r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after=${encodeURIComponent(r.body.next_cursor)}`, { token: reader });
assert.strictEqual(r.status, 200, r.text);
assert.strictEqual(r.status, 200, r.text);
assert.ok(r.body.events.length >= 1 && r.body.events.every(e => e.seq > first.seq));
assert.ok(r.body.events.length >= 1 && r.body.events.every(e => e.seq > first.seq));
- // after_seq still works, unchanged.
- r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after_seq=${first.seq}`, { token: reader });
- assert.ok(r.body.events.every(e => e.seq > first.seq));
+ for (const value of ['0', String(first.seq)]) {
+ r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after_seq=${value}`, { token: reader });
+ assert.strictEqual(r.status, 400);
+ assert.strictEqual(r.body.code, 'events.bad_request');
+ }
});
});
t('a malformed after= is refused, not treated as a position', async () => {
t('a malformed after= is refused, not treated as a position', async () => {
@@ -97,7 +98,7 @@ t('a cursor from another epoch answers the gap shape, never a silent restart', a
@@ -97,7 +98,7 @@ t('a cursor from another epoch answers the gap shape, never a silent restart', a
assert.strictEqual(cursor.decode(r.body.next_cursor).epoch, olderEpoch + 1, 'position moves in the new epoch');
assert.strictEqual(cursor.decode(r.body.next_cursor).epoch, olderEpoch + 1, 'position moves in the new epoch');
});
});
-t('SSE id is the cursor; Last-Event-ID resumes from a cursor, and a bare seq still works', async () => {
+t('SSE id is the cursor; Last-Event-ID resumes from a cursor, and a bare seq starts at the head', async () => {
const epoch = await h.store.epoch();
const epoch = await h.store.epoch();
const c1 = await sse(h.base, '/realtime/stream?topics=live.stream.*', { headers: { Authorization: `Bearer ${readerSvc}` } });
const c1 = await sse(h.base, '/realtime/stream?topics=live.stream.*', { headers: { Authorization: `Bearer ${readerSvc}` } });
const s1 = await publish({ visibility: 'public', event_type: 'live.stream.a' });
const s1 = await publish({ visibility: 'public', event_type: 'live.stream.a' });
@@ -113,8 +114,9 @@ t('SSE id is the cursor; Last-Event-ID resumes from a cursor, and a bare seq sti
@@ -113,8 +114,9 @@ t('SSE id is the cursor; Last-Event-ID resumes from a cursor, and a bare seq sti
assert.deepStrictEqual(byCursor.gaps(), []);
assert.deepStrictEqual(byCursor.gaps(), []);
byCursor.close();
byCursor.close();
- // A bare seq is still accepted for this release.
- const bySeq = await sse(h.base, `/realtime/stream?topics=live.stream.*&last_event_id=${s2.seq}`, { headers: { Authorization: `Bearer ${readerSvc}` } });
+ const bySeq = await sse(h.base, `/realtime/stream?topics=live.stream.*&last_event_id=${s1.seq}`, { headers: { Authorization: `Bearer ${readerSvc}` } });
+ assert.deepStrictEqual(bySeq.events(), [], 'a bare number does not replay missed events');
+ assert.deepStrictEqual(bySeq.gaps(), []);
const s3 = await publish({ visibility: 'public', event_type: 'live.stream.c' });
const s3 = await publish({ visibility: 'public', event_type: 'live.stream.c' });
await bySeq.waitFor(x => x.events().length === 1);
await bySeq.waitFor(x => x.events().length === 1);
assert.deepStrictEqual(bySeq.events().map(e => e.seq), [s3.seq]);
assert.deepStrictEqual(bySeq.events().map(e => e.seq), [s3.seq]);
@@ -136,27 +138,22 @@ t('SSE reports a gap when the cursor belongs to another epoch', async () => {
@@ -136,27 +138,22 @@ t('SSE reports a gap when the cursor belongs to another epoch', async () => {
again.close();
again.close();
});
});
-t('checkpoint round trip: an opaque cursor in, the position and epoch out', async () => {
+t('checkpoint round trip: an opaque cursor in and out', async () => {
const pub = await publish();
const pub = await publish();
- const epoch = cursor.decode(pub.cursor).epoch;
let r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: reader, body: { topic: 'live.vod.*', cursor: pub.cursor } });
let r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: reader, body: { topic: 'live.vod.*', cursor: pub.cursor } });
assert.strictEqual(r.status, 200, r.text);
assert.strictEqual(r.status, 200, r.text);
- assert.ok(validate('events.read-result@1', r.body).valid, JSON.stringify(r.body));
- assert.strictEqual(r.body.cursor, pub.seq, 'cursor stays the numeric position');
- assert.strictEqual(r.body.epoch, epoch);
- assert.strictEqual(r.body.next_cursor, pub.cursor, 'the opaque cursor comes back, ready for after=');
+ assert.strictEqual(r.body.topic, 'live.vod.*');
+ assert.strictEqual(r.body.cursor, pub.cursor);
+ assert.ok(!('epoch' in r.body) && !('next_cursor' in r.body));
r = await request(h.base, 'GET', '/api/v1/checkpoints?topic=live.vod.*', { token: reader });
r = await request(h.base, 'GET', '/api/v1/checkpoints?topic=live.vod.*', { token: reader });
- assert.strictEqual(r.body.cursor, pub.seq);
- assert.strictEqual(r.body.epoch, epoch);
- assert.strictEqual(r.body.next_cursor, pub.cursor);
- assert.strictEqual(cursor.encode(r.body.cursor, r.body.epoch), pub.cursor);
+ assert.strictEqual(r.body.cursor, pub.cursor);
+ assert.ok(!('epoch' in r.body) && !('next_cursor' in r.body));
- // A plain integer still works and takes the current epoch.
+ // A plain integer is no longer a consumer position.
r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: reader, body: { topic: 'live.vod.*', cursor: 5 } });
r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: reader, body: { topic: 'live.vod.*', cursor: 5 } });
- assert.strictEqual(r.body.cursor, 5);
- assert.strictEqual(r.body.epoch, await h.store.epoch());
- assert.deepStrictEqual(cursor.decode(r.body.next_cursor), { seq: 5, epoch: await h.store.epoch() });
+ assert.strictEqual(r.status, 400);
+ assert.strictEqual(r.body.code, 'events.bad_request');
// A malformed cursor string is refused.
// A malformed cursor string is refused.
r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: reader, body: { topic: 'live.vod.*', cursor: 'nope' } });
r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: reader, body: { topic: 'live.vod.*', cursor: 'nope' } });
@@ -28,7 +28,7 @@ function exportToken(env, { project = P, cap = ['events.app.read'] } = {}) {
@@ -28,7 +28,7 @@ function exportToken(env, { project = P, cap = ['events.app.read'] } = {}) {
let h;
let h;
const seqs = { production: [], sandbox: [] };
const seqs = { production: [], sandbox: [] };
-const pull = async (token, after = 0, limit = 100, topic = `app.${key}.*`) => await request(h.base, 'GET', `/api/v1/events?topic=${encodeURIComponent(topic)}&after_seq=${after}&limit=${limit}`, { token });
+const pull = async (token, after = null, limit = 100, topic = `app.${key}.*`) => await request(h.base, 'GET', `/api/v1/events?topic=${encodeURIComponent(topic)}${after === null || after === 0 ? '' : `&after=${after}`}&limit=${limit}`, { token });
t('boot and publish as the project\'s app, in both environments (and another project)', async () => {
t('boot and publish as the project\'s app, in both environments (and another project)', async () => {
h = await boot();
h = await boot();
@@ -47,15 +47,13 @@ t('boot and publish as the project\'s app, in both environments (and another pro
@@ -47,15 +47,13 @@ t('boot and publish as the project\'s app, in both environments (and another pro
t('an export token pulls the project\'s events of its environment, page by page to the end', async () => {
t('an export token pulls the project\'s events of its environment, page by page to the end', async () => {
const got = [];
const got = [];
- let after = 0;
- let latest = null;
+ let after = null;
for (let page = 0; page < 10; page++) {
for (let page = 0; page < 10; page++) {
const r = await pull(exportToken('production'), after, 2);
const r = await pull(exportToken('production'), after, 2);
assert.strictEqual(r.status, 200, r.text);
assert.strictEqual(r.status, 200, r.text);
- latest = latest === null ? r.body.latest_seq : latest;
got.push(...r.body.events.map(e => e.seq));
got.push(...r.body.events.map(e => e.seq));
- if (r.body.next_after_seq <= after || r.body.next_after_seq >= latest) break;
- after = r.body.next_after_seq;
+ if (r.body.next_cursor === r.body.latest_cursor || r.body.next_cursor === after) break;
+ after = r.body.next_cursor;
}
}
assert.deepStrictEqual(got, seqs.production);
assert.deepStrictEqual(got, seqs.production);
const s = await pull(exportToken('sandbox'));
const s = await pull(exportToken('sandbox'));
@@ -94,6 +94,9 @@ async function boot({ env = {}, clock = manualClock(), worker = 'manual', fetchI
@@ -94,6 +94,9 @@ async function boot({ env = {}, clock = manualClock(), worker = 'manual', fetchI
};
};
}
}
+/** Events' opaque cursor for a position in this boot's store (tests build positions the way Events hands them out). */
+async function cursorAt(h, seq) { return require('../server/cursor').encode(seq, await h.store.epoch()); }
+
async function request(base, method, p, { token, body, headers = {}, raw } = {}) {
async function request(base, method, p, { token, body, headers = {}, raw } = {}) {
const h = { ...headers };
const h = { ...headers };
if (token) h.Authorization = `Bearer ${token}`;
if (token) h.Authorization = `Bearer ${token}`;
@@ -216,5 +219,5 @@ function suite(name) {
@@ -216,5 +219,5 @@ function suite(name) {
module.exports = {
module.exports = {
ISSUER, privateKey, publicKey, silent, serviceToken, appToken, userToken, realtimeTicket, manualClock, tmpDir, boot, request,
ISSUER, privateKey, publicKey, silent, serviceToken, appToken, userToken, realtimeTicket, manualClock, tmpDir, boot, request,
- envelope, subscriber, sse, sleep, suite,
+ envelope, subscriber, sse, sleep, suite, cursorAt,
};
};
@@ -57,7 +57,7 @@ t('deliveries feed the latency histogram and attempt counter; a dead one degrade
@@ -57,7 +57,7 @@ t('deliveries feed the latency histogram and attempt counter; a dead one degrade
t('/metrics: loopback only, templates not ids, Events gauges', async () => {
t('/metrics: loopback only, templates not ids, Events gauges', async () => {
const id = (await h.db.prepare('SELECT id FROM events ORDER BY seq LIMIT 1').get()).id;
const id = (await h.db.prepare('SELECT id FROM events ORDER BY seq LIMIT 1').get()).id;
await request(h.base, 'GET', `/api/v1/events/${id}`, { token: reader });
await request(h.base, 'GET', `/api/v1/events/${id}`, { token: reader });
- await request(h.base, 'GET', '/api/v1/events?topic=live.*&after_seq=0', { token: reader });
+ await request(h.base, 'GET', '/api/v1/events?topic=live.*', { token: reader });
await request(h.base, 'GET', '/no/such/12345');
await request(h.base, 'GET', '/no/such/12345');
let m = await get(h.base, '/metrics', { 'X-Forwarded-For': '203.0.113.7' });
let m = await get(h.base, '/metrics', { 'X-Forwarded-For': '203.0.113.7' });
assert.strictEqual(m.status, 404, 'through a proxy: not found');
assert.strictEqual(m.status, 404, 'through a proxy: not found');
@@ -2,12 +2,12 @@
@@ -2,12 +2,12 @@
const assert = require('assert');
const assert = require('assert');
const crypto = require('crypto');
const crypto = require('crypto');
const nodeHttp = require('http');
const nodeHttp = require('http');
-const { validate } = require('openvibe-contracts');
const cursor = require('../server/cursor');
const cursor = require('../server/cursor');
const topics = require('../server/topics');
const topics = require('../server/topics');
const { hasCap, allows } = require('../server/auth');
const { hasCap, allows } = require('../server/auth');
const { load } = require('../server/config');
const { load } = require('../server/config');
-const { boot, request, serviceToken, envelope, publicKey, suite, sleep } = require('./helpers');
+const { validate } = require('openvibe-contracts');
+const { boot, request, serviceToken, envelope, publicKey, suite, sleep, cursorAt } = require('./helpers');
const t = suite('pull-health');
const t = suite('pull-health');
let h;
let h;
@@ -58,19 +58,20 @@ t('pull: cursor, topic filter, internal events included for services', async ()
@@ -58,19 +58,20 @@ t('pull: cursor, topic filter, internal events included for services', async ()
await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope('live', { event_type: 'live.chat.sent' }) });
await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope('live', { event_type: 'live.chat.sent' }) });
const s3 = (await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope('live', { event_type: 'live.vod.deleted' }) })).body.seq;
const s3 = (await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope('live', { event_type: 'live.vod.deleted' }) })).body.seq;
- let r = await request(h.base, 'GET', '/api/v1/events?topic=live.vod.*&after_seq=0&limit=1', { token: reader });
+ let r = await request(h.base, 'GET', '/api/v1/events?topic=live.vod.*&limit=1', { token: reader });
assert.strictEqual(r.status, 200, r.text);
assert.strictEqual(r.status, 200, r.text);
assert.ok(validate('events.read-result@1', r.body).valid, JSON.stringify(validate('events.read-result@1', r.body).errors));
assert.ok(validate('events.read-result@1', r.body).valid, JSON.stringify(validate('events.read-result@1', r.body).errors));
for (const e of r.body.events) assert.deepStrictEqual(cursor.decode(e.cursor), { seq: e.seq, epoch: await h.store.epoch() });
for (const e of r.body.events) assert.deepStrictEqual(cursor.decode(e.cursor), { seq: e.seq, epoch: await h.store.epoch() });
- assert.strictEqual(cursor.decode(r.body.next_cursor).seq, r.body.next_after_seq);
+ assert.strictEqual(r.body.next_cursor, await cursorAt(h, s1));
+ assert.ok(!('next_after_seq' in r.body) && !('latest_seq' in r.body));
assert.deepStrictEqual(r.body.events.map(e => e.seq), [s1]);
assert.deepStrictEqual(r.body.events.map(e => e.seq), [s1]);
assert.strictEqual(r.body.events[0].event.visibility, 'internal');
assert.strictEqual(r.body.events[0].event.visibility, 'internal');
- r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after_seq=${r.body.next_after_seq}`, { token: reader });
+ r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after=${r.body.next_cursor}`, { token: reader });
assert.deepStrictEqual(r.body.events.map(e => e.seq), [s3]);
assert.deepStrictEqual(r.body.events.map(e => e.seq), [s3]);
- assert.strictEqual(r.body.next_after_seq, s3);
- r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after_seq=${r.body.next_after_seq}`, { token: reader });
+ assert.strictEqual(r.body.next_cursor, await cursorAt(h, s3));
+ r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after=${r.body.next_cursor}`, { token: reader });
assert.deepStrictEqual(r.body.events, []);
assert.deepStrictEqual(r.body.events, []);
- assert.strictEqual(r.body.latest_seq, s3);
+ assert.strictEqual(r.body.latest_cursor, await cursorAt(h, s3));
assert.match(r.body.latest_cursor, /^c1\./, 'the head as an opaque cursor');
assert.match(r.body.latest_cursor, /^c1\./, 'the head as an opaque cursor');
{
{
// Starting at the head with latest_cursor reads nothing old, and the next event after it arrives.
// Starting at the head with latest_cursor reads nothing old, and the next event after it arrives.
@@ -94,21 +95,35 @@ t('pull: a cursor older than retention reports the gap', async () => {
@@ -94,21 +95,35 @@ t('pull: a cursor older than retention reports the gap', async () => {
const before = await h.store.lastSeq();
const before = await h.store.lastSeq();
await h.store.prune({ retentionDays: 30, now: Date.now() + 31 * 86400000 });
await h.store.prune({ retentionDays: 30, now: Date.now() + 31 * 86400000 });
const s = (await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope() })).body.seq;
const s = (await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope() })).body.seq;
- const r = await request(h.base, 'GET', '/api/v1/events?after_seq=1', { token: reader });
+ const r = await request(h.base, 'GET', `/api/v1/events?after=${await cursorAt(h, 1)}`, { token: reader });
assert.deepStrictEqual(r.body.gap, { from_seq: 2, to_seq: before });
assert.deepStrictEqual(r.body.gap, { from_seq: 2, to_seq: before });
assert.deepStrictEqual(r.body.events.map(e => e.seq), [s]);
assert.deepStrictEqual(r.body.events.map(e => e.seq), [s]);
});
});
+t('pull: after_seq is refused even at zero', async () => {
+ const r = await request(h.base, 'GET', '/api/v1/events?topic=live.vod.*&after_seq=0', { token: reader });
+ assert.strictEqual(r.status, 400);
+ assert.strictEqual(r.body.code, 'events.bad_request');
+});
+
t('checkpoints: per consumer and topic', async () => {
t('checkpoints: per consumer and topic', async () => {
let r = await request(h.base, 'GET', '/api/v1/checkpoints?topic=live.vod.*', { token: reader });
let r = await request(h.base, 'GET', '/api/v1/checkpoints?topic=live.vod.*', { token: reader });
- assert.strictEqual(r.body.cursor, 0);
+ assert.strictEqual(r.body.cursor, null);
+ assert.strictEqual(r.body.carrier, null);
+ assert.strictEqual(r.body.updated_at, null);
r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: reader, body: { topic: 'live.vod.*', cursor: 42 } });
r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: reader, body: { topic: 'live.vod.*', cursor: 42 } });
- assert.strictEqual(r.body.cursor, 42);
+ assert.strictEqual(r.status, 400);
+ assert.strictEqual(r.body.code, 'events.bad_request');
+ const position = await cursorAt(h, 42);
+ r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: reader, body: { topic: 'live.vod.*', cursor: position } });
+ assert.strictEqual(r.status, 200);
+ assert.strictEqual(r.body.cursor, position);
+ assert.ok(validate('events.read-result@1', r.body).valid, 'the checkpoint answer is a contract checkpoint');
r = await request(h.base, 'GET', '/api/v1/checkpoints?topic=live.vod.*', { token: reader });
r = await request(h.base, 'GET', '/api/v1/checkpoints?topic=live.vod.*', { token: reader });
- assert.strictEqual(r.body.cursor, 42);
+ assert.strictEqual(r.body.cursor, position);
assert.strictEqual(r.body.consumer, 'games');
assert.strictEqual(r.body.consumer, 'games');
r = await request(h.base, 'GET', '/api/v1/checkpoints?topic=live.vod.*', { token: serviceToken('tools', ['events.event.read']) });
r = await request(h.base, 'GET', '/api/v1/checkpoints?topic=live.vod.*', { token: serviceToken('tools', ['events.event.read']) });
- assert.strictEqual(r.body.cursor, 0, 'another consumer has its own');
+ assert.strictEqual(r.body.cursor, null, 'another consumer has its own');
});
});
t('health and ready', async () => {
t('health and ready', async () => {
@@ -8,7 +8,7 @@
@@ -8,7 +8,7 @@
const assert = require('assert');
const assert = require('assert');
const crypto = require('crypto');
const crypto = require('crypto');
const { ids } = require('openvibe-contracts');
const { ids } = require('openvibe-contracts');
-const { boot, request, serviceToken, userToken, realtimeTicket, envelope, sse, sleep, suite, ISSUER } = require('./helpers');
+const { boot, request, serviceToken, userToken, realtimeTicket, envelope, sse, sleep, suite, ISSUER, cursorAt } = require('./helpers');
const t = suite('realtime-tickets');
const t = suite('realtime-tickets');
let h;
let h;
@@ -77,7 +77,7 @@ t('a guessed user:<other> topic yields nothing: refused as a pattern, and no pat
@@ -77,7 +77,7 @@ t('a guessed user:<other> topic yields nothing: refused as a pattern, and no pat
assert.deepStrictEqual(guess.events(), [], 'Alice\'s ticket sees none of Bob\'s events and no internal one');
assert.deepStrictEqual(guess.events(), [], 'Alice\'s ticket sees none of Bob\'s events and no internal one');
assert.deepStrictEqual(anon.events(), []);
assert.deepStrictEqual(anon.events(), []);
// Replay too: from before those events, Alice's ticket still gets none of them.
// Replay too: from before those events, Alice's ticket still gets none of them.
- const replay = await stream(`topics=*&ticket=${realtimeTicket({ subjectId: alice })}&last_event_id=0`);
+ const replay = await stream(`topics=*&ticket=${realtimeTicket({ subjectId: alice })}&last_event_id=${await cursorAt(h, 0)}`);
await quiet();
await quiet();
assert.ok(replay.events().every(e => e.event.subject.id !== bob && e.event.visibility !== 'internal'), 'replay never includes Bob\'s or internal events');
assert.ok(replay.events().every(e => e.event.subject.id !== bob && e.event.visibility !== 'internal'), 'replay never includes Bob\'s or internal events');
guess.close(); anon.close(); replay.close();
guess.close(); anon.close(); replay.close();
@@ -139,7 +139,7 @@ t('disconnect and resume from the cursor with a fresh ticket: the missed events,
@@ -139,7 +139,7 @@ t('disconnect and resume from the cursor with a fresh ticket: the missed events,
const missed1 = await publish(network, notified(alice, 2));
const missed1 = await publish(network, notified(alice, 2));
await publish(network, notified(bob, 9));
await publish(network, notified(bob, 9));
const missed2 = await publish(network, notified(alice, 3));
const missed2 = await publish(network, notified(alice, 3));
- const a2 = await stream(`topics=${TOPIC}&ticket=${realtimeTicket({ subjectId: alice })}&last_event_id=${cursor}`);
+ const a2 = await stream(`topics=${TOPIC}&ticket=${realtimeTicket({ subjectId: alice })}&last_event_id=${await cursorAt(h, cursor)}`);
await a2.waitFor(c => c.events().length === 2);
await a2.waitFor(c => c.events().length === 2);
assert.deepStrictEqual(a2.events().map(e => e.seq), [missed1, missed2]);
assert.deepStrictEqual(a2.events().map(e => e.seq), [missed1, missed2]);
assert.deepStrictEqual(a2.events().map(e => e.event.payload.unread_count), [2, 3]);
assert.deepStrictEqual(a2.events().map(e => e.event.payload.unread_count), [2, 3]);
@@ -154,7 +154,7 @@ t('a cursor older than retention is reported as a gap before the events kept', a
@@ -154,7 +154,7 @@ t('a cursor older than retention is reported as a gap before the events kept', a
const old = await publish(network, notified(alice, 5));
const old = await publish(network, notified(alice, 5));
await h.store.prune({ retentionDays: 30, now: Date.now() + 31 * 86400000 });
await h.store.prune({ retentionDays: 30, now: Date.now() + 31 * 86400000 });
const kept = await publish(network, notified(alice, 6));
const kept = await publish(network, notified(alice, 6));
- const c = await stream(`topics=${TOPIC}&ticket=${realtimeTicket({ subjectId: alice })}&last_event_id=${old - 1}`);
+ const c = await stream(`topics=${TOPIC}&ticket=${realtimeTicket({ subjectId: alice })}&last_event_id=${await cursorAt(h, old - 1)}`);
await c.waitFor(x => x.events().length === 1);
await c.waitFor(x => x.events().length === 1);
assert.strictEqual(c.gaps().length, 1);
assert.strictEqual(c.gaps().length, 1);
assert.strictEqual(c.gaps()[0].reason, 'retention');
assert.strictEqual(c.gaps()[0].reason, 'retention');
@@ -1,7 +1,7 @@
@@ -1,7 +1,7 @@
'use strict';
'use strict';
const assert = require('assert');
const assert = require('assert');
const { ids } = require('openvibe-contracts');
const { ids } = require('openvibe-contracts');
-const { boot, request, serviceToken, userToken, envelope, sse, sleep, suite } = require('./helpers');
+const { boot, request, serviceToken, userToken, envelope, sse, sleep, suite, cursorAt } = require('./helpers');
const t = suite('realtime');
const t = suite('realtime');
let h;
let h;
@@ -72,7 +72,7 @@ t('Last-Event-ID resume replays what was missed, in order, then goes live', asyn
@@ -72,7 +72,7 @@ t('Last-Event-ID resume replays what was missed, in order, then goes live', asyn
await publish(live, envelope('live', { visibility: 'public', event_type: 'live.chat.not_subscribed' }));
await publish(live, envelope('live', { visibility: 'public', event_type: 'live.chat.not_subscribed' }));
await publish(live, envelope('live', { visibility: 'internal', event_type: 'live.stream.c' }));
await publish(live, envelope('live', { visibility: 'internal', event_type: 'live.stream.c' }));
const missed2 = await publish(live, envelope('live', { visibility: 'public', event_type: 'live.stream.d' }));
const missed2 = await publish(live, envelope('live', { visibility: 'public', event_type: 'live.stream.d' }));
- const c = await stream('topics=live.stream.*', { 'Last-Event-ID': String(first) });
+ const c = await stream('topics=live.stream.*', { 'Last-Event-ID': await cursorAt(h, first) });
await c.waitFor(x => x.events().length === 2);
await c.waitFor(x => x.events().length === 2);
assert.deepStrictEqual(c.events().map(e => e.seq), [missed1, missed2]);
assert.deepStrictEqual(c.events().map(e => e.seq), [missed1, missed2]);
assert.deepStrictEqual(c.gaps(), []);
assert.deepStrictEqual(c.gaps(), []);
@@ -81,17 +81,32 @@ t('Last-Event-ID resume replays what was missed, in order, then goes live', asyn
@@ -81,17 +81,32 @@ t('Last-Event-ID resume replays what was missed, in order, then goes live', asyn
assert.strictEqual(c.events()[2].seq, liveSeq);
assert.strictEqual(c.events()[2].seq, liveSeq);
c.close();
c.close();
// Query-string form for clients that cannot set headers.
// Query-string form for clients that cannot set headers.
- const q = await stream(`topics=live.stream.*&last_event_id=${missed2}`);
+ const q = await stream(`topics=live.stream.*&last_event_id=${await cursorAt(h, missed2)}`);
await q.waitFor(x => x.events().length === 1);
await q.waitFor(x => x.events().length === 1);
assert.strictEqual(q.events()[0].seq, liveSeq);
assert.strictEqual(q.events()[0].seq, liveSeq);
q.close();
q.close();
});
});
+t('a bare-number Last-Event-ID starts at the head without replay or gap', async () => {
+ const first = await publish(live, envelope('live', { visibility: 'public' }));
+ const missed = await publish(live, envelope('live', { visibility: 'public' }));
+ const bare = await stream('topics=live.stream.*', { 'Last-Event-ID': String(first) });
+ await quiet();
+ assert.deepStrictEqual(bare.events(), []);
+ assert.deepStrictEqual(bare.gaps(), []);
+ bare.close();
+ const resumed = await stream('topics=live.stream.*', { 'Last-Event-ID': await cursorAt(h, first) });
+ await resumed.waitFor(x => x.events().some(e => e.seq === missed));
+ assert.deepStrictEqual(resumed.events().map(e => e.seq), [missed]);
+ assert.deepStrictEqual(resumed.gaps(), []);
+ resumed.close();
+});
+
t('a cursor older than retention gets `event: gap` first', async () => {
t('a cursor older than retention gets `event: gap` first', async () => {
const old = await publish(live, envelope('live', { visibility: 'public' }));
const old = await publish(live, envelope('live', { visibility: 'public' }));
await h.store.prune({ retentionDays: 30, now: Date.now() + 31 * 86400000 });
await h.store.prune({ retentionDays: 30, now: Date.now() + 31 * 86400000 });
const kept = await publish(live, envelope('live', { visibility: 'public' }));
const kept = await publish(live, envelope('live', { visibility: 'public' }));
- const c = await stream('topics=live.stream.*', { 'Last-Event-ID': String(old - 1) });
+ const c = await stream('topics=live.stream.*', { 'Last-Event-ID': await cursorAt(h, old - 1) });
await c.waitFor(x => x.events().length === 1);
await c.waitFor(x => x.events().length === 1);
const gaps = c.gaps();
const gaps = c.gaps();
assert.strictEqual(gaps.length, 1);
assert.strictEqual(gaps.length, 1);
@@ -101,7 +116,7 @@ t('a cursor older than retention gets `event: gap` first', async () => {
@@ -101,7 +116,7 @@ t('a cursor older than retention gets `event: gap` first', async () => {
assert.ok(c.messages.findIndex(m => m.event === 'gap') < c.messages.findIndex(m => !m.event), 'gap comes before events');
assert.ok(c.messages.findIndex(m => m.event === 'gap') < c.messages.findIndex(m => !m.event), 'gap comes before events');
assert.strictEqual(c.events()[0].seq, kept);
assert.strictEqual(c.events()[0].seq, kept);
c.close();
c.close();
- const ahead = await stream('topics=live.stream.*', { 'Last-Event-ID': String(kept + 1000) });
+ const ahead = await stream('topics=live.stream.*', { 'Last-Event-ID': await cursorAt(h, kept + 1000) });
await ahead.waitFor(x => x.gaps().length === 1);
await ahead.waitFor(x => x.gaps().length === 1);
assert.strictEqual(ahead.gaps()[0].reason, 'cursor_ahead');
assert.strictEqual(ahead.gaps()[0].reason, 'cursor_ahead');
ahead.close();
ahead.close();
@@ -14,7 +14,7 @@ const path = require('path');
@@ -14,7 +14,7 @@ const path = require('path');
const { ids } = require('openvibe-contracts');
const { ids } = require('openvibe-contracts');
const apps = require('../server/apps');
const apps = require('../server/apps');
const { parseDirective } = require('../server/redaction');
const { parseDirective } = require('../server/redaction');
-const { boot, request, serviceToken, userToken, appToken, envelope, subscriber, sse, sleep, suite } = require('./helpers');
+const { boot, request, serviceToken, userToken, appToken, envelope, subscriber, sse, sleep, suite, cursorAt } = require('./helpers');
const t = suite('redaction');
const t = suite('redaction');
let h;
let h;
@@ -97,7 +97,7 @@ t('before deletion the message is replayable (the leak as it was)', async () =>
@@ -97,7 +97,7 @@ t('before deletion the message is replayable (the leak as it was)', async () =>
const env = created(101);
const env = created(101);
msgEventId = env.event_id;
msgEventId = env.event_id;
msgSeq = (await publish(chat, env)).seq;
msgSeq = (await publish(chat, env)).seq;
- const anon = await stream('topics=chat.message.*', { 'Last-Event-ID': String(msgSeq - 1) });
+ const anon = await stream('topics=chat.message.*', { 'Last-Event-ID': await cursorAt(h, msgSeq - 1) });
await anon.waitFor(c => c.events().length === 1);
await anon.waitFor(c => c.events().length === 1);
assert.strictEqual(anon.events()[0].event.payload.text, SECRET_TEXT);
assert.strictEqual(anon.events()[0].event.payload.text, SECRET_TEXT);
anon.close();
anon.close();
@@ -119,7 +119,7 @@ t('chat.message.deleted turns the stored event into a tombstone at the same seq'
@@ -119,7 +119,7 @@ t('chat.message.deleted turns the stored event into a tombstone at the same seq'
});
});
t('anonymous SSE replay: the tombstone, never the text', async () => {
t('anonymous SSE replay: the tombstone, never the text', async () => {
- const anon = await stream('topics=chat.message.*', { 'Last-Event-ID': String(msgSeq - 1) });
+ const anon = await stream('topics=chat.message.*', { 'Last-Event-ID': await cursorAt(h, msgSeq - 1) });
await anon.waitFor(c => c.events().length === 2);
await anon.waitFor(c => c.events().length === 2);
const [a, b] = anon.events();
const [a, b] = anon.events();
assert.deepStrictEqual([a.seq, b.seq], [msgSeq, delSeq], 'sequence intact');
assert.deepStrictEqual([a.seq, b.seq], [msgSeq, delSeq], 'sequence intact');
@@ -130,9 +130,9 @@ t('anonymous SSE replay: the tombstone, never the text', async () => {
@@ -130,9 +130,9 @@ t('anonymous SSE replay: the tombstone, never the text', async () => {
});
});
t('signed-in SSE replay (cookie and Bearer) and service replay: the tombstone too', async () => {
t('signed-in SSE replay (cookie and Bearer) and service replay: the tombstone too', async () => {
- const byCookie = await stream('topics=chat.message.*', { Cookie: `ov_token=${userToken({ subjectId: alice })}`, 'Last-Event-ID': String(msgSeq - 1) });
- const byBearer = await stream('topics=chat.*', { Authorization: `Bearer ${userToken()}`, 'Last-Event-ID': String(msgSeq - 1) });
- const svc = await stream('topics=chat.message.*', { Authorization: `Bearer ${serviceToken('search', ['events.event.read'])}`, 'Last-Event-ID': String(msgSeq - 1) });
+ const byCookie = await stream('topics=chat.message.*', { Cookie: `ov_token=${userToken({ subjectId: alice })}`, 'Last-Event-ID': await cursorAt(h, msgSeq - 1) });
+ const byBearer = await stream('topics=chat.*', { Authorization: `Bearer ${userToken()}`, 'Last-Event-ID': await cursorAt(h, msgSeq - 1) });
+ const svc = await stream('topics=chat.message.*', { Authorization: `Bearer ${serviceToken('search', ['events.event.read'])}`, 'Last-Event-ID': await cursorAt(h, msgSeq - 1) });
for (const c of [byCookie, byBearer, svc]) {
for (const c of [byCookie, byBearer, svc]) {
await c.waitFor(x => x.events().length === 2);
await c.waitFor(x => x.events().length === 2);
assert.deepStrictEqual(c.events().map(e => e.seq), [msgSeq, delSeq]);
assert.deepStrictEqual(c.events().map(e => e.seq), [msgSeq, delSeq]);
@@ -143,7 +143,7 @@ t('signed-in SSE replay (cookie and Bearer) and service replay: the tombstone to
@@ -143,7 +143,7 @@ t('signed-in SSE replay (cookie and Bearer) and service replay: the tombstone to
});
});
t('pull (/api/v1/events), by id, and developer-app reads: the tombstone', async () => {
t('pull (/api/v1/events), by id, and developer-app reads: the tombstone', async () => {
- let r = await request(h.base, 'GET', `/api/v1/events?topic=chat.message.*&after_seq=${msgSeq - 1}`, { token: reader });
+ let r = await request(h.base, 'GET', `/api/v1/events?topic=chat.message.*&after=${await cursorAt(h, msgSeq - 1)}`, { token: reader });
assert.strictEqual(r.status, 200);
assert.strictEqual(r.status, 200);
assert.deepStrictEqual(r.body.events.map(e => e.seq), [msgSeq, delSeq], 'no hole in the sequence');
assert.deepStrictEqual(r.body.events.map(e => e.seq), [msgSeq, delSeq], 'no hole in the sequence');
assertTombstone(r.body.events[0].event, delEventId);
assertTombstone(r.body.events[0].event, delEventId);
@@ -151,7 +151,7 @@ t('pull (/api/v1/events), by id, and developer-app reads: the tombstone', async
@@ -151,7 +151,7 @@ t('pull (/api/v1/events), by id, and developer-app reads: the tombstone', async
assertTombstone(r.body.event, delEventId);
assertTombstone(r.body.event, delEventId);
assert.strictEqual(r.body.seq, msgSeq);
assert.strictEqual(r.body.seq, msgSeq);
const app = appToken({ env: 'production', cap: ['events.app.read'] });
const app = appToken({ env: 'production', cap: ['events.app.read'] });
- r = await request(h.base, 'GET', `/api/v1/events?topic=chat.*&after_seq=${msgSeq - 1}`, { token: app });
+ r = await request(h.base, 'GET', `/api/v1/events?topic=chat.*&after=${await cursorAt(h, msgSeq - 1)}`, { token: app });
assert.strictEqual(r.status, 200, r.text);
assert.strictEqual(r.status, 200, r.text);
assert.deepStrictEqual(r.body.events.map(e => e.seq), [msgSeq, delSeq]);
assert.deepStrictEqual(r.body.events.map(e => e.seq), [msgSeq, delSeq]);
assertTombstone(r.body.events[0].event, delEventId);
assertTombstone(r.body.events[0].event, delEventId);
@@ -272,7 +272,7 @@ t('public replay window: browsers get a gap for public events older than 5 minut
@@ -272,7 +272,7 @@ t('public replay window: browsers get a gap for public events older than 5 minut
const recent = await post(chat, created(2));
const recent = await post(chat, created(2));
const topics = 'topics=chat.message.*,network.notification.*';
const topics = 'topics=chat.message.*,network.notification.*';
- const anon = await sse(w.base, `/realtime/stream?${topics}`, { headers: { 'Last-Event-ID': String(oldPublic - 1) } });
+ const anon = await sse(w.base, `/realtime/stream?${topics}`, { headers: { 'Last-Event-ID': await cursorAt(w, oldPublic - 1) } });
await anon.waitFor(c => c.events().length === 1);
await anon.waitFor(c => c.events().length === 1);
assert.deepStrictEqual(anon.events().map(e => e.seq), [recent]);
assert.deepStrictEqual(anon.events().map(e => e.seq), [recent]);
const gap = anon.gaps()[0];
const gap = anon.gaps()[0];
@@ -280,16 +280,16 @@ t('public replay window: browsers get a gap for public events older than 5 minut
@@ -280,16 +280,16 @@ t('public replay window: browsers get a gap for public events older than 5 minut
assert.deepStrictEqual([gap.from_seq, gap.to_seq, gap.window_seconds], [oldPublic, recent - 1, 300]);
assert.deepStrictEqual([gap.from_seq, gap.to_seq, gap.window_seconds], [oldPublic, recent - 1, 300]);
assert.ok(anon.messages.findIndex(m => m.event === 'gap') < anon.messages.findIndex(m => !m.event), 'gap first');
assert.ok(anon.messages.findIndex(m => m.event === 'gap') < anon.messages.findIndex(m => !m.event), 'gap first');
- const user = await sse(w.base, `/realtime/stream?${topics}`, { headers: { Cookie: `ov_token=${userToken({ subjectId: alice })}`, 'Last-Event-ID': String(oldPublic - 1) } });
+ const user = await sse(w.base, `/realtime/stream?${topics}`, { headers: { Cookie: `ov_token=${userToken({ subjectId: alice })}`, 'Last-Event-ID': await cursorAt(w, oldPublic - 1) } });
await user.waitFor(c => c.events().length === 2);
await user.waitFor(c => c.events().length === 2);
assert.deepStrictEqual(user.events().map(e => e.seq), [oldMine, recent], 'own subject events keep the whole retention; old public ones do not');
assert.deepStrictEqual(user.events().map(e => e.seq), [oldMine, recent], 'own subject events keep the whole retention; old public ones do not');
assert.strictEqual(user.gaps()[0].reason, 'public_window');
assert.strictEqual(user.gaps()[0].reason, 'public_window');
- const svc = await sse(w.base, `/realtime/stream?${topics}`, { headers: { Authorization: `Bearer ${serviceToken('search', ['events.event.read'])}`, 'Last-Event-ID': String(oldPublic - 1) } });
+ const svc = await sse(w.base, `/realtime/stream?${topics}`, { headers: { Authorization: `Bearer ${serviceToken('search', ['events.event.read'])}`, 'Last-Event-ID': await cursorAt(w, oldPublic - 1) } });
await svc.waitFor(c => c.events().length === 3);
await svc.waitFor(c => c.events().length === 3);
assert.deepStrictEqual(svc.gaps(), [], 'services are not windowed');
assert.deepStrictEqual(svc.gaps(), [], 'services are not windowed');
- const fresh = await sse(w.base, `/realtime/stream?${topics}`, { headers: { 'Last-Event-ID': String(recent - 1) } });
+ const fresh = await sse(w.base, `/realtime/stream?${topics}`, { headers: { 'Last-Event-ID': await cursorAt(w, recent - 1) } });
await fresh.waitFor(c => c.events().length === 1);
await fresh.waitFor(c => c.events().length === 1);
assert.deepStrictEqual(fresh.gaps(), [], 'a reconnect inside the window is seamless');
assert.deepStrictEqual(fresh.gaps(), [], 'a reconnect inside the window is seamless');
for (const c of [anon, user, svc, fresh]) c.close();
for (const c of [anon, user, svc, fresh]) c.close();
@@ -297,7 +297,7 @@ t('public replay window: browsers get a gap for public events older than 5 minut
@@ -297,7 +297,7 @@ t('public replay window: browsers get a gap for public events older than 5 minut
const none = await boot({ env: { REALTIME_PUBLIC_REPLAY_SECONDS: '0' } });
const none = await boot({ env: { REALTIME_PUBLIC_REPLAY_SECONDS: '0' } });
try {
try {
const s = await (async () => (await request(none.base, 'POST', '/api/v1/events', { token: chat, body: created(3) })).body.seq)();
const s = await (async () => (await request(none.base, 'POST', '/api/v1/events', { token: chat, body: created(3) })).body.seq)();
- const c = await sse(none.base, '/realtime/stream?topics=chat.message.*', { headers: { 'Last-Event-ID': String(s - 1) } });
+ const c = await sse(none.base, '/realtime/stream?topics=chat.message.*', { headers: { 'Last-Event-ID': await cursorAt(none, s - 1) } });
await c.waitFor(x => x.gaps().length === 1);
await c.waitFor(x => x.gaps().length === 1);
await sleep(50);
await sleep(50);
assert.deepStrictEqual(c.events(), [], 'window 0: no public replay at all');
assert.deepStrictEqual(c.events(), [], 'window 0: no public replay at all');
@@ -13,7 +13,7 @@
@@ -13,7 +13,7 @@
// positive controls show each route works for its owner.
// positive controls show each route works for its owner.
// node test/security-idor.test.js
// node test/security-idor.test.js
const assert = require('assert');
const assert = require('assert');
-const { suite, appToken, envelope, request } = require('./helpers');
+const { suite, appToken, envelope, request, cursorAt } = require('./helpers');
const { buildWorld, ALL } = require('./security-world');
const { buildWorld, ALL } = require('./security-world');
const apps = require('../server/apps');
const apps = require('../server/apps');
@@ -116,7 +116,7 @@ t('deliveries, replay and checkpoints across consumers', async () => {
@@ -116,7 +116,7 @@ t('deliveries, replay and checkpoints across consumers', async () => {
}
}
// Checkpoints of another project's pattern.
// Checkpoints of another project's pattern.
for (const [token, topic] of [[tok.appA, `app.${ids.keyB}.*`], [tok.appB, `app.${ids.keyA}.*`], [tok.appA, '*']]) {
for (const [token, topic] of [[tok.appA, `app.${ids.keyB}.*`], [tok.appB, `app.${ids.keyA}.*`], [tok.appA, '*']]) {
- const r = await request(w.base, 'PUT', '/api/v1/checkpoints', { token, body: { topic, cursor: 99 } });
+ const r = await request(w.base, 'PUT', '/api/v1/checkpoints', { token, body: { topic, cursor: await cursorAt(w.h, 99) } });
if (r.status !== 403) bad.push(`checkpoint ${topic} → ${r.status}`);
if (r.status !== 403) bad.push(`checkpoint ${topic} → ${r.status}`);
}
}
assert.deepStrictEqual(bad, [], bad.join('\n'));
assert.deepStrictEqual(bad, [], bad.join('\n'));
@@ -15,7 +15,7 @@
@@ -15,7 +15,7 @@
// - subscription listings, single reads and checkpoints are the caller's own
// - subscription listings, single reads and checkpoints are the caller's own
// node test/security-private.test.js
// node test/security-private.test.js
const assert = require('assert');
const assert = require('assert');
-const { suite, appToken, request } = require('./helpers');
+const { suite, appToken, request, cursorAt } = require('./helpers');
const { buildWorld } = require('./security-world');
const { buildWorld } = require('./security-world');
const crawler = require('./security-crawl');
const crawler = require('./security-crawl');
@@ -38,7 +38,7 @@ t('pull reads: each caller gets exactly its scope, by every topic pattern and ev
@@ -38,7 +38,7 @@ t('pull reads: each caller gets exactly its scope, by every topic pattern and ev
const token = who === 'anonymous' ? undefined : tok[who] || tok.userX;
const token = who === 'anonymous' ? undefined : tok[who] || tok.userX;
let got = [];
let got = [];
for (const topic of topicsList) {
for (const topic of topicsList) {
- const r = await request(w.base, 'GET', `/api/v1/events?topic=${encodeURIComponent(topic)}&after_seq=0&limit=1000`, { token });
+ const r = await request(w.base, 'GET', `/api/v1/events?topic=${encodeURIComponent(topic)}&limit=1000`, { token });
got = got.concat(seenIn(r.text));
got = got.concat(seenIn(r.text));
}
}
for (const [name, id] of Object.entries(ev)) {
for (const [name, id] of Object.entries(ev)) {
@@ -57,7 +57,7 @@ t('pull reads: each caller gets exactly its scope, by every topic pattern and ev
@@ -57,7 +57,7 @@ t('pull reads: each caller gets exactly its scope, by every topic pattern and ev
t('realtime: visibility by viewer, replayed from the start, app events never streamed', async () => {
t('realtime: visibility by viewer, replayed from the start, app events never streamed', async () => {
const { tok, ids } = w;
const { tok, ids } = w;
const topics = encodeURIComponent('live.*,network.notification.*,app.*');
const topics = encodeURIComponent('live.*,network.notification.*,app.*');
- const open = async (headers = {}, extra = '') => crawler.hit(w.base, 'GET', `/realtime/stream?topics=${topics}&last_event_id=0${extra}`, { headers, streamMs: 400 });
+ const open = async (headers = {}, extra = '') => crawler.hit(w.base, 'GET', `/realtime/stream?topics=${topics}&last_event_id=${await cursorAt(w.h, 0)}${extra}`, { headers, streamMs: 400 });
const cases = [
const cases = [
['anonymous', {}, '', ['public']],
['anonymous', {}, '', ['public']],
['x by ticket', {}, `&ticket=${w.realtimeTicket({ subjectId: ids.x })}`, ['public', 'subjectX']],
['x by ticket', {}, `&ticket=${w.realtimeTicket({ subjectId: ids.x })}`, ['public', 'subjectX']],
@@ -131,18 +131,18 @@ t('subscriptions and checkpoints are the caller\'s own', async () => {
@@ -131,18 +131,18 @@ t('subscriptions and checkpoints are the caller\'s own', async () => {
if (r.status < 400) bad.push(`${who} reads live's subscription (${r.status})`);
if (r.status < 400) bad.push(`${who} reads live's subscription (${r.status})`);
}
}
// Checkpoints are keyed by the calling principal.
// Checkpoints are keyed by the calling principal.
- let r = await request(w.base, 'PUT', '/api/v1/checkpoints', { token: tok.live, body: { topic: 'live.*', cursor: 5 } });
+ let r = await request(w.base, 'PUT', '/api/v1/checkpoints', { token: tok.live, body: { topic: 'live.*', cursor: await cursorAt(w.h, 5) } });
assert.strictEqual(r.status, 200);
assert.strictEqual(r.status, 200);
- r = await request(w.base, 'PUT', '/api/v1/checkpoints', { token: tok.appA, body: { topic: `app.${ids.keyA}.*`, cursor: 3 } });
+ r = await request(w.base, 'PUT', '/api/v1/checkpoints', { token: tok.appA, body: { topic: `app.${ids.keyA}.*`, cursor: await cursorAt(w.h, 3) } });
assert.strictEqual(r.status, 200);
assert.strictEqual(r.status, 200);
for (const [who, topic] of [['media', 'live.*'], ['reader', 'live.*'], ['appA2', `app.${ids.keyA}.*`]]) {
for (const [who, topic] of [['media', 'live.*'], ['reader', 'live.*'], ['appA2', `app.${ids.keyA}.*`]]) {
r = await request(w.base, 'GET', `/api/v1/checkpoints?topic=${encodeURIComponent(topic)}`, { token: tok[who] });
r = await request(w.base, 'GET', `/api/v1/checkpoints?topic=${encodeURIComponent(topic)}`, { token: tok[who] });
- if (r.status === 200 && r.body.cursor !== 0) bad.push(`${who} reads another consumer's checkpoint (${r.body.cursor})`);
+ if (r.status === 200 && r.body.cursor !== null) bad.push(`${who} reads another consumer's checkpoint (${r.body.cursor})`);
}
}
r = await request(w.base, 'GET', `/api/v1/checkpoints?topic=${encodeURIComponent(`app.${ids.keyA}.*`)}`, { token: tok.appB });
r = await request(w.base, 'GET', `/api/v1/checkpoints?topic=${encodeURIComponent(`app.${ids.keyA}.*`)}`, { token: tok.appB });
assert.strictEqual(r.status, 403, 'another project\'s pattern is refused');
assert.strictEqual(r.status, 403, 'another project\'s pattern is refused');
r = await request(w.base, 'GET', '/api/v1/checkpoints?topic=live.*', { token: tok.live });
r = await request(w.base, 'GET', '/api/v1/checkpoints?topic=live.*', { token: tok.live });
- assert.strictEqual(r.body.cursor, 5, 'positive control');
+ assert.strictEqual(r.body.cursor, await cursorAt(w.h, 5), 'positive control');
assert.deepStrictEqual(bad, [], bad.join('\n'));
assert.deepStrictEqual(bad, [], bad.join('\n'));
});
});
@@ -18,7 +18,7 @@ const assert = require('assert');
@@ -18,7 +18,7 @@ const assert = require('assert');
const crypto = require('crypto');
const crypto = require('crypto');
const fs = require('fs');
const fs = require('fs');
const path = require('path');
const path = require('path');
-const { suite, serviceToken, appToken, userToken, ISSUER } = require('./helpers');
+const { suite, serviceToken, appToken, userToken, ISSUER, cursorAt } = require('./helpers');
const { buildWorld } = require('./security-world');
const { buildWorld } = require('./security-world');
const crawler = require('./security-crawl');
const crawler = require('./security-crawl');
@@ -89,8 +89,8 @@ t('crawl: no GET, refusal or error carries a secret, a token or the key', async
@@ -89,8 +89,8 @@ t('crawl: no GET, refusal or error carries a secret, a token or the key', async
return n;
return n;
};
};
const values = (name) => ({ id: [subs.live.id, subs.media.id, subs.appA.id, subs.appB.id, ev.public, ev.internal, ev.appA, 'sub_00000000000000000000000000', 'x', '..%2f'] }[name] || ['x']);
const values = (name) => ({ id: [subs.live.id, subs.media.id, subs.appA.id, subs.appB.id, ev.public, ev.internal, ev.appA, 'sub_00000000000000000000000000', 'x', '..%2f'] }[name] || ['x']);
- const queries = ['topic=*', 'topic=live.*', `topic=app.${ids.keyA}.*`, `topic=app.${ids.keyB}.*,live.*`, 'status=dead', `subscription_id=${subs.appB.id}`, 'after_seq=0&limit=1000',
- 'topics=live.*,network.notification.*&last_event_id=0', 'debug=1&include=secret'];
+ const queries = ['topic=*', 'topic=live.*', `topic=app.${ids.keyA}.*`, `topic=app.${ids.keyB}.*,live.*`, 'status=dead', `subscription_id=${subs.appB.id}`, 'limit=1000',
+ `topics=live.*,network.notification.*&last_event_id=${await cursorAt(w.h, 0)}`, 'debug=1&include=secret'];
const paths = crawler.pathsFor(w.routes, values, { method: 'get', queries, extra: ['/nope', '/api/nope', '/.env', '/data/pglite/PG_VERSION', '/package.json', '/api/v1/subscriptions/../../package.json'] });
const paths = crawler.pathsFor(w.routes, values, { method: 'get', queries, extra: ['/nope', '/api/nope', '/.env', '/data/pglite/PG_VERSION', '/package.json', '/api/v1/subscriptions/../../package.json'] });
const gets = await crawler.crawl(w.base, paths, people, needlesFor);
const gets = await crawler.crawl(w.base, paths, people, needlesFor);
const found = [...gets.found];
const found = [...gets.found];
@@ -102,7 +102,7 @@ t('after the hour: one events.usage.recorded per rollup, valid, internal, delive
@@ -102,7 +102,7 @@ t('after the hour: one events.usage.recorded per rollup, valid, internal, delive
});
});
t('apps never see the rollups', async () => {
t('apps never see the rollups', async () => {
- const r = await request(h.base, 'GET', '/api/v1/events?topic=events.*&after_seq=0', { token });
+ const r = await request(h.base, 'GET', '/api/v1/events?topic=events.*', { token });
assert.strictEqual(r.status, 200);
assert.strictEqual(r.status, 200);
assert.strictEqual(r.body.events.length, 0);
assert.strictEqual(r.body.events.length, 0);
});
});