openvibers/openvibe.events Commits

Events: a position is an opaque cursor only (after_seq, numeric checkpoints and bare-seq resume retired) (#56)

OP OpenVibers committed 10 Oct 2026 · 3a2d41c64b · 21 files, +173 −158

* openvibe-contracts v0.132.0 (was v0.129.0)

Opened by scripts/pin-bump.js in OpenVibers/OpenVibe.Shared after the v0.132.0 release.

* Events: a position is an opaque cursor only (after_seq, numeric checkpoints and bare-seq resume retired)

Plan T7, ADR-042 decision 7: a position on OpenVibe.Events is an opaque cursor and nothing else (Contracts 0.132.0). This PR carries the openvibe-contracts v0.132.0 pin bump (#55, folded in).

Every consumer reads by cursor now:
- Community's relay (Community #56, deployed);
- openvibe-sdk 0.42.0 (pull, iterate, checkpoints, realtime);
- Shared 3.0.1 (the bell and the release watcher);
- Examples #14;
- Chat's e2e test (#64);
- Services' project archive, which already did.

**Changes:**
- **Pull:** `after_seq` (any value, 0 included) is a 400 `events.bad_request` naming `after=`. Before, a consumer that still sent a number would have been silently re-read or restarted. Without `after=` a page starts at the oldest retained event. Pages carry `next_cursor` and `latest_cursor` and no longer `next_after_seq`/`latest_seq`.
- **Checkpoints:** `PUT { topic, cursor, carrier? }` takes an opaque cursor; a number is refused. Both answers are `{ consumer, topic, cursor, carrier, updated_at }`: the stored position and epoch handed back as the cursor they make, or `cursor: null` when none is stored. They no longer return `epoch`/`next_cursor` beside a numeric cursor. Storage is unchanged (position, epoch and carrier columns).
- **Realtime:** `Last-Event-ID` / `last_event_id=` resumes only from a cursor. A bare number is no longer a position: it is ignored like any other non-cursor, so the stream starts at the head (the SSE id has been the cursor since the cursor release).
- **Unchanged:**
  - the per-event `seq` (now documented as informational);
  - `gap` ranges;
  - the operator routes (`/api/v1/deliveries?after_seq=`, `replay { from_seq }`);
  - the `latest_seq` operator numbers in `/api/ready` and `/metrics`.
- README and the route comments describe the cursor-only API.

**Tests:** every test now builds positions as cursors through a `cursorAt(h, seq)` helper. New assertions:
- `after_seq=0` is a 400;
- pages have no numeric fields and validate against `events.read-result@1` 0.132.0;
- checkpoints: none stored → `cursor: null`, a number → 400, a cursor round-trips and validates;
- a bare-number `Last-Event-ID` replays nothing and reports no gap, while the cursor of the same position replays the missed event.

`ov test`: 31 reached files green (the NATS fabric files skip without a NATS server).

**Deploy order:** Community (#56) is already live on SDK 0.42. Services on older SDK pins only publish (no pulls), so nothing that reads is left on a numeric position. A site still on a Shared pin older than 3.0.1 sends its bell's numeric `last_event_id`; after this, that reconnect starts at the head instead of replaying, and the bell's count poll fills in. The pin bot moves those sites to 3.0.1.

Changes

ddiff --git a/README.md b/README.md
iindex cf62ef6..5343ddf 100644
---- a/README.md
++++ b/README.md
@@@ -114,7 +114,7 @@ Each delivery is `POST <endpoint>` with body `{ "event": <envelope>, "seq": n }`
Operators: `GET /api/v1/deliveries?status=dead` is the dead-letter queue; `POST /api/v1/deliveries/replay { subscription_id, event_ids: [...] }` or `{ subscription_id, from_seq }` requeues retained events (that is also how a new subscription catches up on history).
--Pull consumers: start with `GET /api/v1/events?topic=media.vod.*&limit=100`, then request `GET /api/v1/events?topic=media.vod.*&after=<next_cursor>&limit=100`, using the previous response's opaque `next_cursor` ([ADR-042](https://github.com/OpenVibers/OpenVibe.Contracts/blob/main/docs/adr/ADR-042-events-fabric.md) decision 7). The response carries `{ events: [{ seq, cursor, event }], next_cursor, next_after_seq, latest_seq, latest_cursor }` (`latest_cursor` is the head as an opaque cursor, for a consumer that starts at "now" without history), plus `gap: { from_seq, to_seq }` when the position is older than both the hot store and the replay tier (below) or from another retention epoch. Use `next_cursor` even when no events match, since it advances past scanned events. Numeric `after_seq=<position>` remains accepted for one release; when both are supplied, `after` takes precedence. The numeric `seq`, `next_after_seq` and `latest_seq` fields also remain for one release. `PUT /api/v1/checkpoints { topic, cursor }` stores a consumer's position here if it has nowhere better (an opaque cursor string is accepted in place of the integer; the answer carries its `epoch`, `carrier` and `next_cursor`).
++Pull consumers: start with `GET /api/v1/events?topic=media.vod.*&limit=100`, then request `GET /api/v1/events?topic=media.vod.*&after=<next_cursor>&limit=100`, using the previous response's opaque `next_cursor` ([ADR-042](https://github.com/OpenVibers/OpenVibe.Contracts/blob/main/docs/adr/ADR-042-events-fabric.md) decision 7). The response carries `{ events: [{ seq, cursor, event }], next_cursor, latest_cursor }` (`latest_cursor` is the head as an opaque cursor, for a consumer that starts at "now" without history), plus `gap: { from_seq, to_seq }` when the position is older than both the hot store and the replay tier (below) or from another retention epoch. Use `next_cursor` even when no events match, since it advances past scanned events. A position is only ever an opaque cursor: `after_seq` is refused with 400 (never read as "from the start"), and the per-event `seq` is informational. `PUT /api/v1/checkpoints { topic, cursor, carrier? }` stores a consumer's cursor here if it has nowhere better (a cursor string; a number is refused); `GET /api/v1/checkpoints?topic=` and the PUT answer `{ consumer, topic, cursor, carrier, updated_at }`, with `cursor` as stored (null when none is).
### Retention: three tiers
@@@ -290,7 +290,7 @@ void connect();
- Conversely, a user JWT that carries `typ` or `purpose` (a ticket, a FedCM assertion) is not a session here.
- A person's topic: `network.notification.created` (Network's outbox; visibility `subject`, subject the recipient). `topics=network.notification.*` streams a person's own notifications and nobody else's. There is no `user:<id>` topic: it is not a valid pattern (400 `realtime.bad_topic`), and subject visibility already does its job.
- Visibility: `public` events go to anyone subscribed to the topic; `subject` events only to the user whose subject id (`usr_…`) is the event's `actor.id` or its user `subject.id`; `internal` events never reach a browser. A guessed topic yields nothing.
--- Resume: the SSE `id` is the cursor (ADR-042 decision 7; a bare seq is accepted for one release), so the browser's automatic `Last-Event-ID` (or `?last_event_id=`) replays what was missed. A cursor older than both the hot store and the replay tier — or from another retention epoch (`reason: "epoch"`) — first gets `event: gap` (`{ reason, from_seq, to_seq }`), as does a replay that hits `REALTIME_REPLAY_MAX`.
++- Resume: the SSE `id` is the cursor (ADR-042 decision 7), so the browser's automatic `Last-Event-ID` (or `?last_event_id=`) replays what was missed. Only a cursor is a position: anything else, a bare number included, is ignored and the stream starts at the head. A cursor older than both the hot store and the replay tier — or from another retention epoch (`reason: "epoch"`) — first gets `event: gap` (`{ reason, from_seq, to_seq }`), as does a replay that hits `REALTIME_REPLAY_MAX`.
- Public replay window: browsers (signed out or signed in) are replayed `public` events received in the last `REALTIME_PUBLIC_REPLAY_SECONDS` (300; 0 = none), enough to ride out a reconnect. Older public events are not replayed: the stream opens with `event: gap` (`reason: "public_window"`, `window_seconds`), and the client refetches state from the owning service. `subject` events addressed to the viewer, and service viewers, keep the whole retention. This keeps the stream from paging through a month of public history, chat lines included; nothing in the network needs more (no browser surface replays public events today).
- Redacted events are replayed as their tombstones ([Redaction](#redaction)).
- Heartbeat comment every 25 s; at most 20 topics per connection and `REALTIME_MAX_CONNECTIONS` (2000) overall; CORS with credentials for `https://*.openvibe.*` only.
@@@ -387,7 +387,7 @@ implemented service.
Part of the [OpenVibe network](https://openvibe.network). Built in the open by [OpenVibers](https://github.com/OpenVibers).
<!-- versions:start -->
--- openvibe-contracts: v0.129.0
++- openvibe-contracts: v0.132.0
- openvibe-sdk: v0.40.0
- openvibe-shared: v3.0.0
<!-- versions:end -->
ddiff --git a/STATUS.json b/STATUS.json
iindex f5dba30..865aa20 100644
---- a/STATUS.json
++++ b/STATUS.json
@@@ -7,7 +7,7 @@
"runtime": "node 22, express, PostgreSQL 18 (openvibe-sdk/db, async) + Valkey 9",
"port": 4300,
"domain": "openvibe.events",
-- "contracts": "openvibe-contracts v0.129.0",
++ "contracts": "openvibe-contracts v0.132.0",
"sdk": "openvibe-sdk v0.40.0",
"shared": "openvibe-shared v3.0.0",
"capabilities": [
ddiff --git a/package-lock.json b/package-lock.json
iindex 9c40a92..abf4b97 100644
---- a/package-lock.json
++++ b/package-lock.json
@@@ -12,7 +12,7 @@
"dotenv": "^16.4.7",
"express": "^4.21.2",
"iovalkey": "^0.4.0",
-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.129.0",
++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.132.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.40.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v3.0.0",
"pg": "^8.23.0"
@@@ -828,9 +828,9 @@
}
},
"node_modules/openvibe-contracts": {
-- "version": "0.129.0",
-- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.129.0",
-- "integrity": "sha512-Ly0F/ap6wmAf8hYpNrOhMLsU3cZYBjsFDW2P1F/xVmHVsRDzAPAZDXtFcohP0UNeZgyxgrR0BI0pPG5guD/Efw==",
++ "version": "0.132.0",
++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.132.0",
++ "integrity": "sha512-08GuHutggWYtGcGrnBLilWco/K715Bl7ZAdfYX7kXeeHguWTIhbWZsSF3JYcH9iq6KKtRB7XCAbmPFKScleD9Q==",
"license": "AGPL-3.0-only",
"dependencies": {
"ajv": "^8.17.1",
ddiff --git a/package.json b/package.json
iindex 8d3b9dc..c0f15a5 100644
---- a/package.json
++++ b/package.json
@@@ -24,7 +24,7 @@
"dotenv": "^16.4.7",
"express": "^4.21.2",
"iovalkey": "^0.4.0",
-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.129.0",
++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.132.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.40.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v3.0.0",
"pg": "^8.23.0"
ddiff --git a/server/api/read.js b/server/api/read.js
iindex c9cacd4..fed472f 100644
---- a/server/api/read.js
++++ b/server/api/read.js
@@@ -2,20 +2,21 @@
/**
* Pull consumers (service token with events.event.read):
*
-- * GET /api/v1/events?topic=media.vod.*[,…]&after_seq=0&limit=100
-- * -> { events: [{ seq, cursor, event }], next_after_seq, next_cursor, latest_seq, latest_cursor, gap? }
-- * `latest_cursor` is the head as an opaque cursor: a consumer that starts at "now" (skipping history)
-- * stores it, so it never needs the numeric latest_seq.
-- * `after=` takes an opaque cursor beside after_seq (ADR-042 decision 7); a cursor from another
-- * retention epoch answers `gap` — never a silent restart. `gap` ({ from_seq, to_seq }) means
++ * GET /api/v1/events?topic=media.vod.*[,…][&after=<cursor>]&limit=100
++ * -> { events: [{ seq, cursor, event }], next_cursor, latest_cursor, gap? }
++ * A position is only ever an opaque cursor (ADR-042 decision 7): `after=` takes one, and without it the
++ * page starts at the oldest retained event. `after_seq` is refused (400), never read as "from the start".
++ * `latest_cursor` is the head: a consumer that starts at "now" (skipping history) stores it. The per-event
++ * `seq` is informational. A cursor from another retention epoch answers `gap` — never a silent restart.
++ * `gap` ({ from_seq, to_seq }) means
* events after the position were already pruned by retention (or the epoch changed). The pull
* spans the hot store and the replay tier (decision 8) with the one cursor: `gap` only outside both.
* Keep next_cursor as the cursor; it moves past events that did not match.
* GET /api/v1/events/:event_id -> { seq, cursor, event } (hot or replay)
-- * GET /api/v1/checkpoints?topic=… / PUT /api/v1/checkpoints { topic, cursor }
-- * a consumer's own stored cursor per topic pattern (consumer = calling principal). `cursor` is
-- * the numeric position; PUT also accepts an opaque cursor string, and both answers carry the
-- * position's epoch and carrier so the cursor can be rebuilt.
++ * GET /api/v1/checkpoints?topic=… / PUT /api/v1/checkpoints { topic, cursor, carrier? }
++ * a consumer's own stored cursor per topic pattern (consumer = calling principal): `cursor` is an
++ * opaque cursor string (a number is refused), and both answers return it as stored, with its carrier
++ * ({ consumer, topic, cursor, carrier, updated_at }; cursor null when none is stored).
*
* Developer apps (events.app.read) use the same routes with an app token: only their project's
* events in the token's environment plus public first-party events (server/apps.js); every topic
@@@ -79,18 +80,19 @@ function readRouter({ store, auth, worker, limits }) {
if (Number.isNaN(limit)) {
return http.sendProblem(res, 400, 'events.bad_request', { detail: 'limit must be 1..1000', ctx });
}
++ // A position is an opaque cursor and nothing else: a number would silently restart a consumer that still sends one.
++ if (req.query.after_seq !== undefined) {
++ return http.sendProblem(res, 400, 'events.bad_request', { detail: "after_seq is retired: pass after=<cursor> (a page's next_cursor, or latest_cursor for the head)", ctx });
++ }
const epoch = await store.epoch();
-- // `after=` is an opaque cursor; `after_seq` stays for one release. A cursor from another epoch cannot be
-- // mapped onto this store, so it is answered with a gap (never treated as a position here).
-- let after = intParam(req.query.after_seq, 0, 0, Number.MAX_SAFE_INTEGER);
++ // A cursor from another epoch cannot be mapped onto this store, so it is answered with a gap.
++ let after = 0;
let epochMismatch = false;
if (req.query.after !== undefined && String(req.query.after) !== '') {
const decoded = cursor.decode(String(req.query.after));
if (!decoded) return http.sendProblem(res, 400, 'events.bad_request', { detail: 'after must be an opaque cursor', ctx });
after = decoded.seq;
epochMismatch = decoded.epoch !== epoch;
-- } else if (Number.isNaN(after)) {
-- return http.sendProblem(res, 400, 'events.bad_request', { detail: 'after_seq must be >= 0', ctx });
}
const oldest = await store.oldestSeq();
const out = {};
@@@ -104,10 +106,8 @@ function readRouter({ store, auth, worker, limits }) {
}
const { rows, cursor: scanned } = await store.scan(from, { patterns, limit, accept: acceptFor(req.principal, patterns) });
out.events = rows.map(r => ({ seq: r.seq, cursor: cursor.encode(r.seq, epoch), event: rowToEnvelope(r) }));
-- out.next_after_seq = scanned;
out.next_cursor = cursor.encode(scanned, epoch);
-- out.latest_seq = await store.lastSeq();
-- out.latest_cursor = cursor.encode(out.latest_seq, epoch);
++ out.latest_cursor = cursor.encode(await store.lastSeq(), epoch);
res.json(out);
});
@@@ -120,6 +120,10 @@ function readRouter({ store, auth, worker, limits }) {
});
const consumerOf = (req) => req.principal.service || req.principal.sub;
++ // The stored position and epoch, handed back as the opaque cursor they make (null when nothing is stored).
++ const checkpointView = (consumer, topic, cp) => ({
++ consumer, topic, cursor: cp ? cursor.encode(Number(cp.cursor), Number(cp.epoch)) : null, carrier: cp ? cp.carrier || null : null, updated_at: cp ? cp.updated_at : null,
++ });
router.get('/api/v1/checkpoints', canRead, limits('events.checkpoint.read'), async (req, res) => {
const topic = String(req.query.topic || '');
@@@ -127,36 +131,21 @@ function readRouter({ store, auth, worker, limits }) {
const scopeErr = scopeError(req.principal, [topic]);
if (scopeErr) return http.sendProblem(res, 403, 'events.topic_not_allowed', { detail: scopeErr, ctx: req.ov });
const cp = await store.getCheckpoint(consumerOf(req), topic);
-- const epoch = cp ? cp.epoch : await store.epoch();
-- const position = cp ? cp.cursor : 0;
-- res.json({
-- consumer: consumerOf(req), topic, cursor: position, epoch, carrier: cp ? cp.carrier : null,
-- next_cursor: cursor.encode(position, epoch), updated_at: cp ? cp.updated_at : null,
-- });
++ res.json(checkpointView(consumerOf(req), topic, cp));
});
router.put('/api/v1/checkpoints', canRead, limits('events.checkpoint.write', { minute: 600, hour: 20000 }), async (req, res) => {
const b = req.body || {};
-- // `cursor` stays the numeric position; an opaque cursor string is accepted in its place and decoded to the
-- // position and epoch it names (ADR-042 decision 7). `carrier` records the carrier the position is on.
-- let position = null;
-- let atEpoch = null;
-- if (typeof b.cursor === 'string') {
-- const decoded = cursor.decode(b.cursor);
-- if (!decoded) return http.sendProblem(res, 400, 'events.bad_request', { detail: 'cursor must be an opaque cursor or an integer >= 0', ctx: req.ov });
-- position = decoded.seq;
-- atEpoch = decoded.epoch;
-- } else if (Number.isInteger(b.cursor) && b.cursor >= 0) {
-- position = b.cursor;
-- if (Number.isInteger(b.epoch) && b.epoch >= 0) atEpoch = b.epoch;
-- }
-- if (!topics.isValidPattern(b.topic) || position === null) {
-- return http.sendProblem(res, 400, 'events.bad_request', { detail: 'topic (pattern) and cursor (integer >= 0 or an opaque cursor) are required', ctx: req.ov });
++ // `cursor` is an opaque cursor, decoded to the position and epoch it names (ADR-042 decision 7); a number is
++ // refused. `carrier` records the carrier the position is on.
++ const decoded = typeof b.cursor === 'string' ? cursor.decode(b.cursor) : null;
++ if (!topics.isValidPattern(b.topic) || !decoded) {
++ return http.sendProblem(res, 400, 'events.bad_request', { detail: "topic (pattern) and cursor (an opaque cursor: a page's next_cursor) are required", ctx: req.ov });
}
const scopeErr = scopeError(req.principal, [b.topic]);
if (scopeErr) return http.sendProblem(res, 403, 'events.topic_not_allowed', { detail: scopeErr, ctx: req.ov });
-- const cp = await store.setCheckpoint(consumerOf(req), b.topic, position, { epoch: atEpoch, carrier: typeof b.carrier === 'string' && b.carrier ? b.carrier : null });
-- res.json({ consumer: consumerOf(req), topic: b.topic, ...cp, next_cursor: cursor.encode(cp.cursor, cp.epoch) });
++ const cp = await store.setCheckpoint(consumerOf(req), b.topic, decoded.seq, { epoch: decoded.epoch, carrier: typeof b.carrier === 'string' && b.carrier ? b.carrier : null });
++ res.json(checkpointView(consumerOf(req), b.topic, cp));
});
router.get('/api/v1/deliveries', isAdmin, limits('events.delivery.list'), async (req, res) => {
ddiff --git a/server/realtime.js b/server/realtime.js
iindex c550c96..c643bb1 100644
---- a/server/realtime.js
++++ b/server/realtime.js
@@@ -18,7 +18,8 @@
* internal service principals (token with events.event.read) only, never a browser
* Developer-app events (app.<project_key>.*, events.app.publish) are never streamed here, to anyone.
*
-- * Resume: `Last-Event-ID` (or ?last_event_id=) is a cursor, or a bare seq for one release. Missed
++ * Resume: `Last-Event-ID` (or ?last_event_id=) is an opaque cursor; anything else (a bare number included) is not a
++ * position and is ignored, so the stream starts at the head. Missed
* events are replayed first, from the replay tier and then the hot store (ADR-042 decision 8); when the
* cursor is older than both — or belongs to another retention epoch — an `event: gap` message comes
* first, so the client knows to refetch state.
@@@ -131,16 +132,12 @@ function createRealtime({ store, auth, config, clock = { now: () => Date.now() }
const epoch = await store.epoch();
const rawLast = req.headers['last-event-id'] ?? req.query.last_event_id;
-- // A cursor, or — for one release — a bare seq. Anything else is ignored, as before.
++ // Only a cursor is a position (ADR-042 decision 7); anything else is ignored and the stream starts at the head.
let lastId = null;
let epochMismatch = false;
if (rawLast != null) {
-- const raw = String(rawLast).trim();
-- if (/^\d{1,15}$/.test(raw)) lastId = Number(raw);
-- else {
-- const decoded = cursor.decode(raw);
-- if (decoded) { lastId = decoded.seq; epochMismatch = decoded.epoch !== epoch; }
-- }
++ const decoded = cursor.decode(String(rawLast).trim());
++ if (decoded) { lastId = decoded.seq; epochMismatch = decoded.epoch !== epoch; }
}
res.status(200);
ddiff --git a/server/store.js b/server/store.js
iindex 30a8bb8..3ef6bae 100644
---- a/server/store.js
++++ b/server/store.js
@@@ -724,13 +724,14 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
/**
* A checkpoint is a cursor in parts: `cursor` is the numeric position, `epoch` the retention epoch it belongs to
-- * and `carrier` the carrier that position is on (ADR-042 decision 7). `cursor` stays numeric on the wire.
++ * and `carrier` the carrier that position is on (ADR-042 decision 7). The wire only ever carries the opaque cursor
++ * these parts make (server/api/read.js).
*/
async function getCheckpoint(consumer, topicPattern) {
const r = await db.prepare('SELECT cursor, epoch, carrier, updated_at FROM consumer_checkpoints WHERE consumer = ? AND topic_pattern = ?').get(consumer, topicPattern);
return r ? { cursor: r.cursor, epoch: r.epoch, carrier: r.carrier || null, updated_at: new Date(r.updated_at).toISOString() } : null;
}
-- /** Store a position. `epoch` defaults to the current retention epoch (a legacy numeric cursor). */
++ /** Store a position (decoded from a cursor). `epoch` defaults to the current retention epoch. */
async function setCheckpoint(consumer, topicPattern, position, { epoch: atEpoch = null, carrier = null } = {}) {
const at = atEpoch == null ? await epoch() : atEpoch;
await db.prepare(`INSERT INTO consumer_checkpoints (consumer, topic_pattern, cursor, epoch, carrier, updated_at) VALUES (?, ?, ?, ?, ?, ?)
ddiff --git a/test/apps.test.js b/test/apps.test.js
iindex d1a8bcc..6b9f1bf 100644
---- a/test/apps.test.js
++++ b/test/apps.test.js
@@@ -8,7 +8,7 @@ const { verifyDeliveryV2 } = require('../lib/client');
const apps = require('../server/apps');
const { createGuardedPost, isPublicAddress } = require('../server/egress');
const { load } = require('../server/config');
--const { boot, request, serviceToken, appToken, envelope, subscriber, suite, sleep } = require('./helpers');
++const { boot, request, serviceToken, appToken, envelope, subscriber, suite, sleep, cursorAt } = require('./helpers');
const t = suite('apps');
const ALL = ['events.app.publish', 'events.app.read', 'events.app.subscribe'];
@@@ -72,7 +72,7 @@ function appEvent(appId, projectId, name = 'order.created', over = {}) {
});
}
const publish = async (token, body) => await request(h.base, 'POST', '/api/v1/events', { token, body });
--const pull = async (token, topic, after = 0) => await request(h.base, 'GET', `/api/v1/events?topic=${encodeURIComponent(topic)}&after_seq=${after}`, { token });
++const pull = async (token, topic, after = null) => await request(h.base, 'GET', `/api/v1/events?topic=${encodeURIComponent(topic)}${after === null || after === 0 ? '' : `&after=${after}`}`, { token });
t('boot', async () => {
await new Promise(r => receiver.listen(0, '127.0.0.1', r));
@@@ -177,9 +177,9 @@ t('read: own project in the same environment, plus public first-party events', a
assert.strictEqual(r.status, 404, 'sandbox events are invisible to first-party readers');
// Checkpoints are per app and follow the same scope.
-- r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: tokA(), body: { topic: `app.${keyA}.*`, cursor: 3 } });
++ r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: tokA(), body: { topic: `app.${keyA}.*`, cursor: await cursorAt(h, 3) } });
assert.strictEqual(r.status, 200); assert.strictEqual(r.body.consumer, `app:${appA}`);
-- r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: tokA(), body: { topic: `app.${keyB}.*`, cursor: 3 } });
++ r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: tokA(), body: { topic: `app.${keyB}.*`, cursor: await cursorAt(h, 3) } });
assert.strictEqual(r.status, 403);
r = await request(h.base, 'GET', `/api/v1/checkpoints?topic=app.${keyA}.*`, { token: tokB() });
assert.strictEqual(r.status, 403);
ddiff --git a/test/archive.test.js b/test/archive.test.js
iindex d3d2d11..0de4633 100644
---- a/test/archive.test.js
++++ b/test/archive.test.js
@@@ -12,7 +12,7 @@ const { limitsOf } = require('../server/limits');
const { createStore } = require('../server/store');
const { exportMonth, restoreMonth, objectKey, manifestKey } = require('../scripts/events-archive');
const { localStorage, s3Storage, fromEnv, signV4 } = require('../scripts/archive-storage');
--const { boot, request, serviceToken, appToken, envelope, sse, suite, tmpDir } = require('./helpers');
++const { boot, request, serviceToken, appToken, envelope, sse, suite, tmpDir, cursorAt } = require('./helpers');
const t = suite('archive');
const DAY = 86400000;
@@@ -44,7 +44,7 @@ t('a hot-pruned event is readable through replay: by id and by pull, unchanged',
const after = await request(h.base, 'GET', `/api/v1/events/${e.event_id}`, { token: reader });
assert.strictEqual(after.status, 200, after.text);
assert.deepStrictEqual(after.body, before.body, 'the same seq, cursor and envelope from the replay tier');
-- const p = await pull('topic=live.tier.one&after_seq=0');
++ const p = await pull('topic=live.tier.one');
assert.deepStrictEqual(p.body.events.map(x => x.event), [before.body.event]);
assert.strictEqual(p.body.gap, undefined);
// A publish of the same id is a duplicate with its position, never a second row.
@@@ -65,7 +65,7 @@ t('a cursor across the hot/replay boundary reports no false gap (pull and SSE)',
assert.strictEqual(r.body.events[0].cursor, b.cursor);
r = await pull(`topic=live.tier.cross&after=${r.body.next_cursor}&limit=1`);
assert.deepStrictEqual([r.body.events.map(x => x.seq), r.body.gap], [[c.seq], undefined], 'then hot, no gap between');
-- r = await pull(`topic=*&after_seq=${a.seq - 1}`);
++ r = await pull(`topic=*&after=${await cursorAt(h, a.seq - 1)}`);
assert.deepStrictEqual(r.body.events.map(x => x.seq), [a.seq, b.seq, c.seq]);
assert.strictEqual(r.body.gap, undefined);
@@@ -92,7 +92,7 @@ t('redaction reaches replay rows (by id and by subject); the owner rule holds th
assert.deepStrictEqual(r.body.event.actor, { type: 'service', id: 'live' });
assert.strictEqual(r.body.seq, e.seq);
}
-- const p = await pull('topic=live.tier.secret&after_seq=0');
++ const p = await pull('topic=live.tier.secret');
assert.ok(p.body.events.every(e => e.event.payload.redacted === true && !('secret' in e.event.payload)));
});
@@@ -108,11 +108,11 @@ t('sandbox and environment rules apply to replay rows exactly as to hot ones', a
await h.store.prune(tiered(31, { sandboxRetentionDays: 7 }));
assert.deepStrictEqual([await count('events_archive', pe.event_id), await count('events_archive', se.event_id), await count('events', se.event_id)], [1, 0, 0],
'a production event moves; a sandbox event is deleted, never kept in replay');
-- const topic = `topic=app.${apps.projectKey(prj)}.*&after_seq=0`;
++ const topic = `topic=app.${apps.projectKey(prj)}.*`;
assert.deepStrictEqual((await pull(topic, prod)).body.events.map(e => e.seq), [pe.seq], 'the app reads its own replay rows');
assert.deepStrictEqual((await pull(topic, sand)).body.events, [], 'never across environments');
assert.deepStrictEqual((await pull(topic)).body.events.map(e => e.seq), [pe.seq], 'first-party: production app events through app.*');
-- assert.deepStrictEqual((await pull('topic=*&after_seq=0')).body.events.filter(e => e.event.event_id === pe.event_id), [], '…and only through app.*');
++ assert.deepStrictEqual((await pull('topic=*')).body.events.filter(e => e.event.event_id === pe.event_id), [], '…and only through app.*');
assert.strictEqual((await request(h.base, 'GET', `/api/v1/events/${pe.event_id}`, { token: sand })).status, 404);
assert.strictEqual((await request(h.base, 'GET', `/api/v1/events/${pe.event_id}`, { token: prod })).status, 200);
});
@@@ -148,7 +148,7 @@ t('export → restore round-trips byte-identical rows; the rows leave replay onl
assert.strictEqual(lines.length, inMonth.length);
assert.strictEqual(JSON.parse(await storage.get(manifestKey(month))).count, inMonth.length);
assert.strictEqual((await request(h.base, 'GET', `/api/v1/events/${inMonth[0].id}`, { token: reader })).status, 404, 'the archive tier is offline');
-- const gapped = await pull(`topic=*&after_seq=0`);
++ const gapped = await pull(`topic=*&after=${await cursorAt(h, 0)}`);
assert.ok(gapped.body.gap, 'a cursor into what went to the archive tier is a gap');
// A rerun is idempotent: the object keeps its rows, nothing is deleted twice.
ddiff --git a/test/cursor.test.js b/test/cursor.test.js
iindex c587f2f..a969ef9 100644
---- a/test/cursor.test.js
++++ b/test/cursor.test.js
@@@ -1,10 +1,9 @@
'use strict';
/**
-- * ADR-042 decision 7: opaque cursors beside the global seq. A cursor is `c1.<epoch>.<base64url(seq)>`;
-- * every answer that carries a seq also carries its cursor, pull accepts `after=<cursor>` beside
-- * `after_seq`, the SSE `id` is the cursor (a bare seq is accepted for one release), and checkpoints
-- * store and return the position's epoch. The contract answers are validated against the pinned
-- * openvibe-contracts (0.81.0, the release that added the cursor fields).
++ * ADR-042 decision 7: consumer positions are cursors only. A cursor is `c1.<epoch>.<base64url(seq)>`;
++ * event answers retain seq and carry a cursor, pull accepts `after=<cursor>`, SSE resumes from
++ * cursor IDs, and checkpoints store and return opaque cursor strings. Unchanged publish and single
++ * event answers are validated against the pinned openvibe-contracts.
*/
const assert = require('assert');
const { validate } = require('openvibe-contracts');
@@@ -39,7 +38,7 @@ t('encode/decode round trip (never throws on malformed input)', () => {
}
});
--t('publish and read answers carry cursors, validated against events.publish-result@1 / read-result@1', async () => {
++t('publish and read answers carry cursors', async () => {
const epoch = await h.store.epoch();
const one = await publish();
const pv = validate('events.publish-result@1', one);
@@@ -51,28 +50,30 @@ t('publish and read answers carry cursors, validated against events.publish-resu
for (const r of batch.body.results) assert.deepStrictEqual(cursor.decode(r.cursor), { seq: r.seq, epoch });
const page = await request(h.base, 'GET', '/api/v1/events?topic=live.vod.*', { token: reader });
-- const rv = validate('events.read-result@1', page.body);
-- assert.ok(rv.valid, JSON.stringify(rv.errors));
++ assert.ok(typeof page.body.next_cursor === 'string' && typeof page.body.latest_cursor === 'string');
assert.ok(page.body.events.length >= 3);
for (const e of page.body.events) assert.deepStrictEqual(cursor.decode(e.cursor), { seq: e.seq, epoch });
-- assert.deepStrictEqual(cursor.decode(page.body.next_cursor), { seq: page.body.next_after_seq, epoch });
++ assert.deepStrictEqual(cursor.decode(page.body.next_cursor), { seq: page.body.events.at(-1).seq, epoch });
++ assert.ok(!('next_after_seq' in page.body) && !('latest_seq' in page.body));
const single = await request(h.base, 'GET', `/api/v1/events/${one.event_id}`, { token: reader });
assert.ok(validate('events.read-result@1', single.body).valid);
assert.deepStrictEqual(cursor.decode(single.body.cursor), { seq: single.body.seq, epoch });
});
--t('pull accepts after=<cursor> beside after_seq; next_cursor round-trips a page', async () => {
++t('pull accepts after=<cursor>; next_cursor round-trips a page and after_seq is refused', async () => {
let r = await request(h.base, 'GET', '/api/v1/events?topic=live.vod.*&limit=1', { token: reader });
assert.strictEqual(r.body.events.length, 1);
const first = r.body.events[0];
-- assert.strictEqual(r.body.next_after_seq, first.seq);
++ assert.strictEqual(r.body.next_cursor, first.cursor);
r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after=${encodeURIComponent(r.body.next_cursor)}`, { token: reader });
assert.strictEqual(r.status, 200, r.text);
assert.ok(r.body.events.length >= 1 && r.body.events.every(e => e.seq > first.seq));
-- // after_seq still works, unchanged.
-- r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after_seq=${first.seq}`, { token: reader });
-- assert.ok(r.body.events.every(e => e.seq > first.seq));
++ for (const value of ['0', String(first.seq)]) {
++ r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after_seq=${value}`, { token: reader });
++ assert.strictEqual(r.status, 400);
++ assert.strictEqual(r.body.code, 'events.bad_request');
++ }
});
t('a malformed after= is refused, not treated as a position', async () => {
@@@ -97,7 +98,7 @@ t('a cursor from another epoch answers the gap shape, never a silent restart', a
assert.strictEqual(cursor.decode(r.body.next_cursor).epoch, olderEpoch + 1, 'position moves in the new epoch');
});
--t('SSE id is the cursor; Last-Event-ID resumes from a cursor, and a bare seq still works', async () => {
++t('SSE id is the cursor; Last-Event-ID resumes from a cursor, and a bare seq starts at the head', async () => {
const epoch = await h.store.epoch();
const c1 = await sse(h.base, '/realtime/stream?topics=live.stream.*', { headers: { Authorization: `Bearer ${readerSvc}` } });
const s1 = await publish({ visibility: 'public', event_type: 'live.stream.a' });
@@@ -113,8 +114,9 @@ t('SSE id is the cursor; Last-Event-ID resumes from a cursor, and a bare seq sti
assert.deepStrictEqual(byCursor.gaps(), []);
byCursor.close();
-- // A bare seq is still accepted for this release.
-- const bySeq = await sse(h.base, `/realtime/stream?topics=live.stream.*&last_event_id=${s2.seq}`, { headers: { Authorization: `Bearer ${readerSvc}` } });
++ const bySeq = await sse(h.base, `/realtime/stream?topics=live.stream.*&last_event_id=${s1.seq}`, { headers: { Authorization: `Bearer ${readerSvc}` } });
++ assert.deepStrictEqual(bySeq.events(), [], 'a bare number does not replay missed events');
++ assert.deepStrictEqual(bySeq.gaps(), []);
const s3 = await publish({ visibility: 'public', event_type: 'live.stream.c' });
await bySeq.waitFor(x => x.events().length === 1);
assert.deepStrictEqual(bySeq.events().map(e => e.seq), [s3.seq]);
@@@ -136,27 +138,22 @@ t('SSE reports a gap when the cursor belongs to another epoch', async () => {
again.close();
});
--t('checkpoint round trip: an opaque cursor in, the position and epoch out', async () => {
++t('checkpoint round trip: an opaque cursor in and out', async () => {
const pub = await publish();
-- const epoch = cursor.decode(pub.cursor).epoch;
let r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: reader, body: { topic: 'live.vod.*', cursor: pub.cursor } });
assert.strictEqual(r.status, 200, r.text);
-- assert.ok(validate('events.read-result@1', r.body).valid, JSON.stringify(r.body));
-- assert.strictEqual(r.body.cursor, pub.seq, 'cursor stays the numeric position');
-- assert.strictEqual(r.body.epoch, epoch);
-- assert.strictEqual(r.body.next_cursor, pub.cursor, 'the opaque cursor comes back, ready for after=');
++ assert.strictEqual(r.body.topic, 'live.vod.*');
++ assert.strictEqual(r.body.cursor, pub.cursor);
++ assert.ok(!('epoch' in r.body) && !('next_cursor' in r.body));
r = await request(h.base, 'GET', '/api/v1/checkpoints?topic=live.vod.*', { token: reader });
-- assert.strictEqual(r.body.cursor, pub.seq);
-- assert.strictEqual(r.body.epoch, epoch);
-- assert.strictEqual(r.body.next_cursor, pub.cursor);
-- assert.strictEqual(cursor.encode(r.body.cursor, r.body.epoch), pub.cursor);
++ assert.strictEqual(r.body.cursor, pub.cursor);
++ assert.ok(!('epoch' in r.body) && !('next_cursor' in r.body));
-- // A plain integer still works and takes the current epoch.
++ // A plain integer is no longer a consumer position.
r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: reader, body: { topic: 'live.vod.*', cursor: 5 } });
-- assert.strictEqual(r.body.cursor, 5);
-- assert.strictEqual(r.body.epoch, await h.store.epoch());
-- assert.deepStrictEqual(cursor.decode(r.body.next_cursor), { seq: 5, epoch: await h.store.epoch() });
++ assert.strictEqual(r.status, 400);
++ assert.strictEqual(r.body.code, 'events.bad_request');
// A malformed cursor string is refused.
r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: reader, body: { topic: 'live.vod.*', cursor: 'nope' } });
ddiff --git a/test/export-tokens.test.js b/test/export-tokens.test.js
iindex 7cb71b3..c0438ea 100644
---- a/test/export-tokens.test.js
++++ b/test/export-tokens.test.js
@@@ -28,7 +28,7 @@ function exportToken(env, { project = P, cap = ['events.app.read'] } = {}) {
let h;
const seqs = { production: [], sandbox: [] };
--const pull = async (token, after = 0, limit = 100, topic = `app.${key}.*`) => await request(h.base, 'GET', `/api/v1/events?topic=${encodeURIComponent(topic)}&after_seq=${after}&limit=${limit}`, { token });
++const pull = async (token, after = null, limit = 100, topic = `app.${key}.*`) => await request(h.base, 'GET', `/api/v1/events?topic=${encodeURIComponent(topic)}${after === null || after === 0 ? '' : `&after=${after}`}&limit=${limit}`, { token });
t('boot and publish as the project\'s app, in both environments (and another project)', async () => {
h = await boot();
@@@ -47,15 +47,13 @@ t('boot and publish as the project\'s app, in both environments (and another pro
t('an export token pulls the project\'s events of its environment, page by page to the end', async () => {
const got = [];
-- let after = 0;
-- let latest = null;
++ let after = null;
for (let page = 0; page < 10; page++) {
const r = await pull(exportToken('production'), after, 2);
assert.strictEqual(r.status, 200, r.text);
-- latest = latest === null ? r.body.latest_seq : latest;
got.push(...r.body.events.map(e => e.seq));
-- if (r.body.next_after_seq <= after || r.body.next_after_seq >= latest) break;
-- after = r.body.next_after_seq;
++ if (r.body.next_cursor === r.body.latest_cursor || r.body.next_cursor === after) break;
++ after = r.body.next_cursor;
}
assert.deepStrictEqual(got, seqs.production);
const s = await pull(exportToken('sandbox'));
ddiff --git a/test/helpers.js b/test/helpers.js
iindex c93719e..3fbb183 100644
---- a/test/helpers.js
++++ b/test/helpers.js
@@@ -94,6 +94,9 @@ async function boot({ env = {}, clock = manualClock(), worker = 'manual', fetchI
};
}
++/** Events' opaque cursor for a position in this boot's store (tests build positions the way Events hands them out). */
++async function cursorAt(h, seq) { return require('../server/cursor').encode(seq, await h.store.epoch()); }
++
async function request(base, method, p, { token, body, headers = {}, raw } = {}) {
const h = { ...headers };
if (token) h.Authorization = `Bearer ${token}`;
@@@ -216,5 +219,5 @@ function suite(name) {
module.exports = {
ISSUER, privateKey, publicKey, silent, serviceToken, appToken, userToken, realtimeTicket, manualClock, tmpDir, boot, request,
-- envelope, subscriber, sse, sleep, suite,
++ envelope, subscriber, sse, sleep, suite, cursorAt,
};
ddiff --git a/test/observability.test.js b/test/observability.test.js
iindex e68b856..5ee1054 100644
---- a/test/observability.test.js
++++ b/test/observability.test.js
@@@ -57,7 +57,7 @@ t('deliveries feed the latency histogram and attempt counter; a dead one degrade
t('/metrics: loopback only, templates not ids, Events gauges', async () => {
const id = (await h.db.prepare('SELECT id FROM events ORDER BY seq LIMIT 1').get()).id;
await request(h.base, 'GET', `/api/v1/events/${id}`, { token: reader });
-- await request(h.base, 'GET', '/api/v1/events?topic=live.*&after_seq=0', { token: reader });
++ await request(h.base, 'GET', '/api/v1/events?topic=live.*', { token: reader });
await request(h.base, 'GET', '/no/such/12345');
let m = await get(h.base, '/metrics', { 'X-Forwarded-For': '203.0.113.7' });
assert.strictEqual(m.status, 404, 'through a proxy: not found');
ddiff --git a/test/pull-health.test.js b/test/pull-health.test.js
iindex b6ac63f..bbb8164 100644
---- a/test/pull-health.test.js
++++ b/test/pull-health.test.js
@@@ -2,12 +2,12 @@
const assert = require('assert');
const crypto = require('crypto');
const nodeHttp = require('http');
--const { validate } = require('openvibe-contracts');
const cursor = require('../server/cursor');
const topics = require('../server/topics');
const { hasCap, allows } = require('../server/auth');
const { load } = require('../server/config');
--const { boot, request, serviceToken, envelope, publicKey, suite, sleep } = require('./helpers');
++const { validate } = require('openvibe-contracts');
++const { boot, request, serviceToken, envelope, publicKey, suite, sleep, cursorAt } = require('./helpers');
const t = suite('pull-health');
let h;
@@@ -58,19 +58,20 @@ t('pull: cursor, topic filter, internal events included for services', async ()
await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope('live', { event_type: 'live.chat.sent' }) });
const s3 = (await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope('live', { event_type: 'live.vod.deleted' }) })).body.seq;
-- let r = await request(h.base, 'GET', '/api/v1/events?topic=live.vod.*&after_seq=0&limit=1', { token: reader });
++ let r = await request(h.base, 'GET', '/api/v1/events?topic=live.vod.*&limit=1', { token: reader });
assert.strictEqual(r.status, 200, r.text);
assert.ok(validate('events.read-result@1', r.body).valid, JSON.stringify(validate('events.read-result@1', r.body).errors));
for (const e of r.body.events) assert.deepStrictEqual(cursor.decode(e.cursor), { seq: e.seq, epoch: await h.store.epoch() });
-- assert.strictEqual(cursor.decode(r.body.next_cursor).seq, r.body.next_after_seq);
++ assert.strictEqual(r.body.next_cursor, await cursorAt(h, s1));
++ assert.ok(!('next_after_seq' in r.body) && !('latest_seq' in r.body));
assert.deepStrictEqual(r.body.events.map(e => e.seq), [s1]);
assert.strictEqual(r.body.events[0].event.visibility, 'internal');
-- r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after_seq=${r.body.next_after_seq}`, { token: reader });
++ r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after=${r.body.next_cursor}`, { token: reader });
assert.deepStrictEqual(r.body.events.map(e => e.seq), [s3]);
-- assert.strictEqual(r.body.next_after_seq, s3);
-- r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after_seq=${r.body.next_after_seq}`, { token: reader });
++ assert.strictEqual(r.body.next_cursor, await cursorAt(h, s3));
++ r = await request(h.base, 'GET', `/api/v1/events?topic=live.vod.*&after=${r.body.next_cursor}`, { token: reader });
assert.deepStrictEqual(r.body.events, []);
-- assert.strictEqual(r.body.latest_seq, s3);
++ assert.strictEqual(r.body.latest_cursor, await cursorAt(h, s3));
assert.match(r.body.latest_cursor, /^c1\./, 'the head as an opaque cursor');
{
// Starting at the head with latest_cursor reads nothing old, and the next event after it arrives.
@@@ -94,21 +95,35 @@ t('pull: a cursor older than retention reports the gap', async () => {
const before = await h.store.lastSeq();
await h.store.prune({ retentionDays: 30, now: Date.now() + 31 * 86400000 });
const s = (await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope() })).body.seq;
-- const r = await request(h.base, 'GET', '/api/v1/events?after_seq=1', { token: reader });
++ const r = await request(h.base, 'GET', `/api/v1/events?after=${await cursorAt(h, 1)}`, { token: reader });
assert.deepStrictEqual(r.body.gap, { from_seq: 2, to_seq: before });
assert.deepStrictEqual(r.body.events.map(e => e.seq), [s]);
});
++t('pull: after_seq is refused even at zero', async () => {
++ const r = await request(h.base, 'GET', '/api/v1/events?topic=live.vod.*&after_seq=0', { token: reader });
++ assert.strictEqual(r.status, 400);
++ assert.strictEqual(r.body.code, 'events.bad_request');
++});
++
t('checkpoints: per consumer and topic', async () => {
let r = await request(h.base, 'GET', '/api/v1/checkpoints?topic=live.vod.*', { token: reader });
-- assert.strictEqual(r.body.cursor, 0);
++ assert.strictEqual(r.body.cursor, null);
++ assert.strictEqual(r.body.carrier, null);
++ assert.strictEqual(r.body.updated_at, null);
r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: reader, body: { topic: 'live.vod.*', cursor: 42 } });
-- assert.strictEqual(r.body.cursor, 42);
++ assert.strictEqual(r.status, 400);
++ assert.strictEqual(r.body.code, 'events.bad_request');
++ const position = await cursorAt(h, 42);
++ r = await request(h.base, 'PUT', '/api/v1/checkpoints', { token: reader, body: { topic: 'live.vod.*', cursor: position } });
++ assert.strictEqual(r.status, 200);
++ assert.strictEqual(r.body.cursor, position);
++ assert.ok(validate('events.read-result@1', r.body).valid, 'the checkpoint answer is a contract checkpoint');
r = await request(h.base, 'GET', '/api/v1/checkpoints?topic=live.vod.*', { token: reader });
-- assert.strictEqual(r.body.cursor, 42);
++ assert.strictEqual(r.body.cursor, position);
assert.strictEqual(r.body.consumer, 'games');
r = await request(h.base, 'GET', '/api/v1/checkpoints?topic=live.vod.*', { token: serviceToken('tools', ['events.event.read']) });
-- assert.strictEqual(r.body.cursor, 0, 'another consumer has its own');
++ assert.strictEqual(r.body.cursor, null, 'another consumer has its own');
});
t('health and ready', async () => {
ddiff --git a/test/realtime-tickets.test.js b/test/realtime-tickets.test.js
iindex f393a7d..68abc0f 100644
---- a/test/realtime-tickets.test.js
++++ b/test/realtime-tickets.test.js
@@@ -8,7 +8,7 @@
const assert = require('assert');
const crypto = require('crypto');
const { ids } = require('openvibe-contracts');
--const { boot, request, serviceToken, userToken, realtimeTicket, envelope, sse, sleep, suite, ISSUER } = require('./helpers');
++const { boot, request, serviceToken, userToken, realtimeTicket, envelope, sse, sleep, suite, ISSUER, cursorAt } = require('./helpers');
const t = suite('realtime-tickets');
let h;
@@@ -77,7 +77,7 @@ t('a guessed user:<other> topic yields nothing: refused as a pattern, and no pat
assert.deepStrictEqual(guess.events(), [], 'Alice\'s ticket sees none of Bob\'s events and no internal one');
assert.deepStrictEqual(anon.events(), []);
// Replay too: from before those events, Alice's ticket still gets none of them.
-- const replay = await stream(`topics=*&ticket=${realtimeTicket({ subjectId: alice })}&last_event_id=0`);
++ const replay = await stream(`topics=*&ticket=${realtimeTicket({ subjectId: alice })}&last_event_id=${await cursorAt(h, 0)}`);
await quiet();
assert.ok(replay.events().every(e => e.event.subject.id !== bob && e.event.visibility !== 'internal'), 'replay never includes Bob\'s or internal events');
guess.close(); anon.close(); replay.close();
@@@ -139,7 +139,7 @@ t('disconnect and resume from the cursor with a fresh ticket: the missed events,
const missed1 = await publish(network, notified(alice, 2));
await publish(network, notified(bob, 9));
const missed2 = await publish(network, notified(alice, 3));
-- const a2 = await stream(`topics=${TOPIC}&ticket=${realtimeTicket({ subjectId: alice })}&last_event_id=${cursor}`);
++ const a2 = await stream(`topics=${TOPIC}&ticket=${realtimeTicket({ subjectId: alice })}&last_event_id=${await cursorAt(h, cursor)}`);
await a2.waitFor(c => c.events().length === 2);
assert.deepStrictEqual(a2.events().map(e => e.seq), [missed1, missed2]);
assert.deepStrictEqual(a2.events().map(e => e.event.payload.unread_count), [2, 3]);
@@@ -154,7 +154,7 @@ t('a cursor older than retention is reported as a gap before the events kept', a
const old = await publish(network, notified(alice, 5));
await h.store.prune({ retentionDays: 30, now: Date.now() + 31 * 86400000 });
const kept = await publish(network, notified(alice, 6));
-- const c = await stream(`topics=${TOPIC}&ticket=${realtimeTicket({ subjectId: alice })}&last_event_id=${old - 1}`);
++ const c = await stream(`topics=${TOPIC}&ticket=${realtimeTicket({ subjectId: alice })}&last_event_id=${await cursorAt(h, old - 1)}`);
await c.waitFor(x => x.events().length === 1);
assert.strictEqual(c.gaps().length, 1);
assert.strictEqual(c.gaps()[0].reason, 'retention');
ddiff --git a/test/realtime.test.js b/test/realtime.test.js
iindex 272f99e..520c4a8 100644
---- a/test/realtime.test.js
++++ b/test/realtime.test.js
@@@ -1,7 +1,7 @@
'use strict';
const assert = require('assert');
const { ids } = require('openvibe-contracts');
--const { boot, request, serviceToken, userToken, envelope, sse, sleep, suite } = require('./helpers');
++const { boot, request, serviceToken, userToken, envelope, sse, sleep, suite, cursorAt } = require('./helpers');
const t = suite('realtime');
let h;
@@@ -72,7 +72,7 @@ t('Last-Event-ID resume replays what was missed, in order, then goes live', asyn
await publish(live, envelope('live', { visibility: 'public', event_type: 'live.chat.not_subscribed' }));
await publish(live, envelope('live', { visibility: 'internal', event_type: 'live.stream.c' }));
const missed2 = await publish(live, envelope('live', { visibility: 'public', event_type: 'live.stream.d' }));
-- const c = await stream('topics=live.stream.*', { 'Last-Event-ID': String(first) });
++ const c = await stream('topics=live.stream.*', { 'Last-Event-ID': await cursorAt(h, first) });
await c.waitFor(x => x.events().length === 2);
assert.deepStrictEqual(c.events().map(e => e.seq), [missed1, missed2]);
assert.deepStrictEqual(c.gaps(), []);
@@@ -81,17 +81,32 @@ t('Last-Event-ID resume replays what was missed, in order, then goes live', asyn
assert.strictEqual(c.events()[2].seq, liveSeq);
c.close();
// Query-string form for clients that cannot set headers.
-- const q = await stream(`topics=live.stream.*&last_event_id=${missed2}`);
++ const q = await stream(`topics=live.stream.*&last_event_id=${await cursorAt(h, missed2)}`);
await q.waitFor(x => x.events().length === 1);
assert.strictEqual(q.events()[0].seq, liveSeq);
q.close();
});
++t('a bare-number Last-Event-ID starts at the head without replay or gap', async () => {
++ const first = await publish(live, envelope('live', { visibility: 'public' }));
++ const missed = await publish(live, envelope('live', { visibility: 'public' }));
++ const bare = await stream('topics=live.stream.*', { 'Last-Event-ID': String(first) });
++ await quiet();
++ assert.deepStrictEqual(bare.events(), []);
++ assert.deepStrictEqual(bare.gaps(), []);
++ bare.close();
++ const resumed = await stream('topics=live.stream.*', { 'Last-Event-ID': await cursorAt(h, first) });
++ await resumed.waitFor(x => x.events().some(e => e.seq === missed));
++ assert.deepStrictEqual(resumed.events().map(e => e.seq), [missed]);
++ assert.deepStrictEqual(resumed.gaps(), []);
++ resumed.close();
++});
++
t('a cursor older than retention gets `event: gap` first', async () => {
const old = await publish(live, envelope('live', { visibility: 'public' }));
await h.store.prune({ retentionDays: 30, now: Date.now() + 31 * 86400000 });
const kept = await publish(live, envelope('live', { visibility: 'public' }));
-- const c = await stream('topics=live.stream.*', { 'Last-Event-ID': String(old - 1) });
++ const c = await stream('topics=live.stream.*', { 'Last-Event-ID': await cursorAt(h, old - 1) });
await c.waitFor(x => x.events().length === 1);
const gaps = c.gaps();
assert.strictEqual(gaps.length, 1);
@@@ -101,7 +116,7 @@ t('a cursor older than retention gets `event: gap` first', async () => {
assert.ok(c.messages.findIndex(m => m.event === 'gap') < c.messages.findIndex(m => !m.event), 'gap comes before events');
assert.strictEqual(c.events()[0].seq, kept);
c.close();
-- const ahead = await stream('topics=live.stream.*', { 'Last-Event-ID': String(kept + 1000) });
++ const ahead = await stream('topics=live.stream.*', { 'Last-Event-ID': await cursorAt(h, kept + 1000) });
await ahead.waitFor(x => x.gaps().length === 1);
assert.strictEqual(ahead.gaps()[0].reason, 'cursor_ahead');
ahead.close();
ddiff --git a/test/redaction.test.js b/test/redaction.test.js
iindex 1409f4d..366aa26 100644
---- a/test/redaction.test.js
++++ b/test/redaction.test.js
@@@ -14,7 +14,7 @@ const path = require('path');
const { ids } = require('openvibe-contracts');
const apps = require('../server/apps');
const { parseDirective } = require('../server/redaction');
--const { boot, request, serviceToken, userToken, appToken, envelope, subscriber, sse, sleep, suite } = require('./helpers');
++const { boot, request, serviceToken, userToken, appToken, envelope, subscriber, sse, sleep, suite, cursorAt } = require('./helpers');
const t = suite('redaction');
let h;
@@@ -97,7 +97,7 @@ t('before deletion the message is replayable (the leak as it was)', async () =>
const env = created(101);
msgEventId = env.event_id;
msgSeq = (await publish(chat, env)).seq;
-- const anon = await stream('topics=chat.message.*', { 'Last-Event-ID': String(msgSeq - 1) });
++ const anon = await stream('topics=chat.message.*', { 'Last-Event-ID': await cursorAt(h, msgSeq - 1) });
await anon.waitFor(c => c.events().length === 1);
assert.strictEqual(anon.events()[0].event.payload.text, SECRET_TEXT);
anon.close();
@@@ -119,7 +119,7 @@ t('chat.message.deleted turns the stored event into a tombstone at the same seq'
});
t('anonymous SSE replay: the tombstone, never the text', async () => {
-- const anon = await stream('topics=chat.message.*', { 'Last-Event-ID': String(msgSeq - 1) });
++ const anon = await stream('topics=chat.message.*', { 'Last-Event-ID': await cursorAt(h, msgSeq - 1) });
await anon.waitFor(c => c.events().length === 2);
const [a, b] = anon.events();
assert.deepStrictEqual([a.seq, b.seq], [msgSeq, delSeq], 'sequence intact');
@@@ -130,9 +130,9 @@ t('anonymous SSE replay: the tombstone, never the text', async () => {
});
t('signed-in SSE replay (cookie and Bearer) and service replay: the tombstone too', async () => {
-- const byCookie = await stream('topics=chat.message.*', { Cookie: `ov_token=${userToken({ subjectId: alice })}`, 'Last-Event-ID': String(msgSeq - 1) });
-- const byBearer = await stream('topics=chat.*', { Authorization: `Bearer ${userToken()}`, 'Last-Event-ID': String(msgSeq - 1) });
-- const svc = await stream('topics=chat.message.*', { Authorization: `Bearer ${serviceToken('search', ['events.event.read'])}`, 'Last-Event-ID': String(msgSeq - 1) });
++ const byCookie = await stream('topics=chat.message.*', { Cookie: `ov_token=${userToken({ subjectId: alice })}`, 'Last-Event-ID': await cursorAt(h, msgSeq - 1) });
++ const byBearer = await stream('topics=chat.*', { Authorization: `Bearer ${userToken()}`, 'Last-Event-ID': await cursorAt(h, msgSeq - 1) });
++ const svc = await stream('topics=chat.message.*', { Authorization: `Bearer ${serviceToken('search', ['events.event.read'])}`, 'Last-Event-ID': await cursorAt(h, msgSeq - 1) });
for (const c of [byCookie, byBearer, svc]) {
await c.waitFor(x => x.events().length === 2);
assert.deepStrictEqual(c.events().map(e => e.seq), [msgSeq, delSeq]);
@@@ -143,7 +143,7 @@ t('signed-in SSE replay (cookie and Bearer) and service replay: the tombstone to
});
t('pull (/api/v1/events), by id, and developer-app reads: the tombstone', async () => {
-- let r = await request(h.base, 'GET', `/api/v1/events?topic=chat.message.*&after_seq=${msgSeq - 1}`, { token: reader });
++ let r = await request(h.base, 'GET', `/api/v1/events?topic=chat.message.*&after=${await cursorAt(h, msgSeq - 1)}`, { token: reader });
assert.strictEqual(r.status, 200);
assert.deepStrictEqual(r.body.events.map(e => e.seq), [msgSeq, delSeq], 'no hole in the sequence');
assertTombstone(r.body.events[0].event, delEventId);
@@@ -151,7 +151,7 @@ t('pull (/api/v1/events), by id, and developer-app reads: the tombstone', async
assertTombstone(r.body.event, delEventId);
assert.strictEqual(r.body.seq, msgSeq);
const app = appToken({ env: 'production', cap: ['events.app.read'] });
-- r = await request(h.base, 'GET', `/api/v1/events?topic=chat.*&after_seq=${msgSeq - 1}`, { token: app });
++ r = await request(h.base, 'GET', `/api/v1/events?topic=chat.*&after=${await cursorAt(h, msgSeq - 1)}`, { token: app });
assert.strictEqual(r.status, 200, r.text);
assert.deepStrictEqual(r.body.events.map(e => e.seq), [msgSeq, delSeq]);
assertTombstone(r.body.events[0].event, delEventId);
@@@ -272,7 +272,7 @@ t('public replay window: browsers get a gap for public events older than 5 minut
const recent = await post(chat, created(2));
const topics = 'topics=chat.message.*,network.notification.*';
-- const anon = await sse(w.base, `/realtime/stream?${topics}`, { headers: { 'Last-Event-ID': String(oldPublic - 1) } });
++ const anon = await sse(w.base, `/realtime/stream?${topics}`, { headers: { 'Last-Event-ID': await cursorAt(w, oldPublic - 1) } });
await anon.waitFor(c => c.events().length === 1);
assert.deepStrictEqual(anon.events().map(e => e.seq), [recent]);
const gap = anon.gaps()[0];
@@@ -280,16 +280,16 @@ t('public replay window: browsers get a gap for public events older than 5 minut
assert.deepStrictEqual([gap.from_seq, gap.to_seq, gap.window_seconds], [oldPublic, recent - 1, 300]);
assert.ok(anon.messages.findIndex(m => m.event === 'gap') < anon.messages.findIndex(m => !m.event), 'gap first');
-- const user = await sse(w.base, `/realtime/stream?${topics}`, { headers: { Cookie: `ov_token=${userToken({ subjectId: alice })}`, 'Last-Event-ID': String(oldPublic - 1) } });
++ const user = await sse(w.base, `/realtime/stream?${topics}`, { headers: { Cookie: `ov_token=${userToken({ subjectId: alice })}`, 'Last-Event-ID': await cursorAt(w, oldPublic - 1) } });
await user.waitFor(c => c.events().length === 2);
assert.deepStrictEqual(user.events().map(e => e.seq), [oldMine, recent], 'own subject events keep the whole retention; old public ones do not');
assert.strictEqual(user.gaps()[0].reason, 'public_window');
-- const svc = await sse(w.base, `/realtime/stream?${topics}`, { headers: { Authorization: `Bearer ${serviceToken('search', ['events.event.read'])}`, 'Last-Event-ID': String(oldPublic - 1) } });
++ const svc = await sse(w.base, `/realtime/stream?${topics}`, { headers: { Authorization: `Bearer ${serviceToken('search', ['events.event.read'])}`, 'Last-Event-ID': await cursorAt(w, oldPublic - 1) } });
await svc.waitFor(c => c.events().length === 3);
assert.deepStrictEqual(svc.gaps(), [], 'services are not windowed');
-- const fresh = await sse(w.base, `/realtime/stream?${topics}`, { headers: { 'Last-Event-ID': String(recent - 1) } });
++ const fresh = await sse(w.base, `/realtime/stream?${topics}`, { headers: { 'Last-Event-ID': await cursorAt(w, recent - 1) } });
await fresh.waitFor(c => c.events().length === 1);
assert.deepStrictEqual(fresh.gaps(), [], 'a reconnect inside the window is seamless');
for (const c of [anon, user, svc, fresh]) c.close();
@@@ -297,7 +297,7 @@ t('public replay window: browsers get a gap for public events older than 5 minut
const none = await boot({ env: { REALTIME_PUBLIC_REPLAY_SECONDS: '0' } });
try {
const s = await (async () => (await request(none.base, 'POST', '/api/v1/events', { token: chat, body: created(3) })).body.seq)();
-- const c = await sse(none.base, '/realtime/stream?topics=chat.message.*', { headers: { 'Last-Event-ID': String(s - 1) } });
++ const c = await sse(none.base, '/realtime/stream?topics=chat.message.*', { headers: { 'Last-Event-ID': await cursorAt(none, s - 1) } });
await c.waitFor(x => x.gaps().length === 1);
await sleep(50);
assert.deepStrictEqual(c.events(), [], 'window 0: no public replay at all');
ddiff --git a/test/security-idor.test.js b/test/security-idor.test.js
iindex 0bee3f7..2026fca 100644
---- a/test/security-idor.test.js
++++ b/test/security-idor.test.js
@@@ -13,7 +13,7 @@
// positive controls show each route works for its owner.
// node test/security-idor.test.js
const assert = require('assert');
--const { suite, appToken, envelope, request } = require('./helpers');
++const { suite, appToken, envelope, request, cursorAt } = require('./helpers');
const { buildWorld, ALL } = require('./security-world');
const apps = require('../server/apps');
@@@ -116,7 +116,7 @@ t('deliveries, replay and checkpoints across consumers', async () => {
}
// Checkpoints of another project's pattern.
for (const [token, topic] of [[tok.appA, `app.${ids.keyB}.*`], [tok.appB, `app.${ids.keyA}.*`], [tok.appA, '*']]) {
-- const r = await request(w.base, 'PUT', '/api/v1/checkpoints', { token, body: { topic, cursor: 99 } });
++ const r = await request(w.base, 'PUT', '/api/v1/checkpoints', { token, body: { topic, cursor: await cursorAt(w.h, 99) } });
if (r.status !== 403) bad.push(`checkpoint ${topic} → ${r.status}`);
}
assert.deepStrictEqual(bad, [], bad.join('\n'));
ddiff --git a/test/security-private.test.js b/test/security-private.test.js
iindex 78a51f7..3db01cf 100644
---- a/test/security-private.test.js
++++ b/test/security-private.test.js
@@@ -15,7 +15,7 @@
// - subscription listings, single reads and checkpoints are the caller's own
// node test/security-private.test.js
const assert = require('assert');
--const { suite, appToken, request } = require('./helpers');
++const { suite, appToken, request, cursorAt } = require('./helpers');
const { buildWorld } = require('./security-world');
const crawler = require('./security-crawl');
@@@ -38,7 +38,7 @@ t('pull reads: each caller gets exactly its scope, by every topic pattern and ev
const token = who === 'anonymous' ? undefined : tok[who] || tok.userX;
let got = [];
for (const topic of topicsList) {
-- const r = await request(w.base, 'GET', `/api/v1/events?topic=${encodeURIComponent(topic)}&after_seq=0&limit=1000`, { token });
++ const r = await request(w.base, 'GET', `/api/v1/events?topic=${encodeURIComponent(topic)}&limit=1000`, { token });
got = got.concat(seenIn(r.text));
}
for (const [name, id] of Object.entries(ev)) {
@@@ -57,7 +57,7 @@ t('pull reads: each caller gets exactly its scope, by every topic pattern and ev
t('realtime: visibility by viewer, replayed from the start, app events never streamed', async () => {
const { tok, ids } = w;
const topics = encodeURIComponent('live.*,network.notification.*,app.*');
-- const open = async (headers = {}, extra = '') => crawler.hit(w.base, 'GET', `/realtime/stream?topics=${topics}&last_event_id=0${extra}`, { headers, streamMs: 400 });
++ const open = async (headers = {}, extra = '') => crawler.hit(w.base, 'GET', `/realtime/stream?topics=${topics}&last_event_id=${await cursorAt(w.h, 0)}${extra}`, { headers, streamMs: 400 });
const cases = [
['anonymous', {}, '', ['public']],
['x by ticket', {}, `&ticket=${w.realtimeTicket({ subjectId: ids.x })}`, ['public', 'subjectX']],
@@@ -131,18 +131,18 @@ t('subscriptions and checkpoints are the caller\'s own', async () => {
if (r.status < 400) bad.push(`${who} reads live's subscription (${r.status})`);
}
// Checkpoints are keyed by the calling principal.
-- let r = await request(w.base, 'PUT', '/api/v1/checkpoints', { token: tok.live, body: { topic: 'live.*', cursor: 5 } });
++ let r = await request(w.base, 'PUT', '/api/v1/checkpoints', { token: tok.live, body: { topic: 'live.*', cursor: await cursorAt(w.h, 5) } });
assert.strictEqual(r.status, 200);
-- r = await request(w.base, 'PUT', '/api/v1/checkpoints', { token: tok.appA, body: { topic: `app.${ids.keyA}.*`, cursor: 3 } });
++ r = await request(w.base, 'PUT', '/api/v1/checkpoints', { token: tok.appA, body: { topic: `app.${ids.keyA}.*`, cursor: await cursorAt(w.h, 3) } });
assert.strictEqual(r.status, 200);
for (const [who, topic] of [['media', 'live.*'], ['reader', 'live.*'], ['appA2', `app.${ids.keyA}.*`]]) {
r = await request(w.base, 'GET', `/api/v1/checkpoints?topic=${encodeURIComponent(topic)}`, { token: tok[who] });
-- if (r.status === 200 && r.body.cursor !== 0) bad.push(`${who} reads another consumer's checkpoint (${r.body.cursor})`);
++ if (r.status === 200 && r.body.cursor !== null) bad.push(`${who} reads another consumer's checkpoint (${r.body.cursor})`);
}
r = await request(w.base, 'GET', `/api/v1/checkpoints?topic=${encodeURIComponent(`app.${ids.keyA}.*`)}`, { token: tok.appB });
assert.strictEqual(r.status, 403, 'another project\'s pattern is refused');
r = await request(w.base, 'GET', '/api/v1/checkpoints?topic=live.*', { token: tok.live });
-- assert.strictEqual(r.body.cursor, 5, 'positive control');
++ assert.strictEqual(r.body.cursor, await cursorAt(w.h, 5), 'positive control');
assert.deepStrictEqual(bad, [], bad.join('\n'));
});
ddiff --git a/test/security-secrets.test.js b/test/security-secrets.test.js
iindex 114010b..b0ae8ee 100644
---- a/test/security-secrets.test.js
++++ b/test/security-secrets.test.js
@@@ -18,7 +18,7 @@ const assert = require('assert');
const crypto = require('crypto');
const fs = require('fs');
const path = require('path');
--const { suite, serviceToken, appToken, userToken, ISSUER } = require('./helpers');
++const { suite, serviceToken, appToken, userToken, ISSUER, cursorAt } = require('./helpers');
const { buildWorld } = require('./security-world');
const crawler = require('./security-crawl');
@@@ -89,8 +89,8 @@ t('crawl: no GET, refusal or error carries a secret, a token or the key', async
return n;
};
const values = (name) => ({ id: [subs.live.id, subs.media.id, subs.appA.id, subs.appB.id, ev.public, ev.internal, ev.appA, 'sub_00000000000000000000000000', 'x', '..%2f'] }[name] || ['x']);
-- const queries = ['topic=*', 'topic=live.*', `topic=app.${ids.keyA}.*`, `topic=app.${ids.keyB}.*,live.*`, 'status=dead', `subscription_id=${subs.appB.id}`, 'after_seq=0&limit=1000',
-- 'topics=live.*,network.notification.*&last_event_id=0', 'debug=1&include=secret'];
++ const queries = ['topic=*', 'topic=live.*', `topic=app.${ids.keyA}.*`, `topic=app.${ids.keyB}.*,live.*`, 'status=dead', `subscription_id=${subs.appB.id}`, 'limit=1000',
++ `topics=live.*,network.notification.*&last_event_id=${await cursorAt(w.h, 0)}`, 'debug=1&include=secret'];
const paths = crawler.pathsFor(w.routes, values, { method: 'get', queries, extra: ['/nope', '/api/nope', '/.env', '/data/pglite/PG_VERSION', '/package.json', '/api/v1/subscriptions/../../package.json'] });
const gets = await crawler.crawl(w.base, paths, people, needlesFor);
const found = [...gets.found];
ddiff --git a/test/usage.test.js b/test/usage.test.js
iindex 26d2943..be6af17 100644
---- a/test/usage.test.js
++++ b/test/usage.test.js
@@@ -102,7 +102,7 @@ t('after the hour: one events.usage.recorded per rollup, valid, internal, delive
});
t('apps never see the rollups', async () => {
-- const r = await request(h.base, 'GET', '/api/v1/events?topic=events.*&after_seq=0', { token });
++ const r = await request(h.base, 'GET', '/api/v1/events?topic=events.*', { token });
assert.strictEqual(r.status, 200);
assert.strictEqual(r.body.events.length, 0);
});