openvibers/openvibe.events Commits
Events on PostgreSQL (ADR-035, roadmap WS-X2, repository pass G3).
Changes
@@@ -2,7 +2,7 @@# OpenVibe.Events — Environment Configuration# Copy to .env and fill in values.# In production this lives at /etc/openvibe/events.env (0600); the systemd unit sets--# NODE_ENV, PORT and EVENTS_DB_PATH itself.++# NODE_ENV and PORT itself.# ═══════════════════════════════════════════════════════════════# Server@@@ -23,7 +23,14 @@ OV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000# A browser's user JWT is accepted when its aud contains one of these.#EVENTS_USER_AUDIENCES=openvibe.events,openvibe.network--# Store (SQLite). Production: /var/lib/openvibe-events/events.db (systemd StateDirectory).++# PostgreSQL (ADR-035), written by OpenVibe.Host roles/data/add-service.sh events: the runtime role through PgBouncer,++# the owner on a direct connection (migrations and the one-time import), and Valkey (shared limit counters).++# Without DATABASE_URL, development uses an embedded PGlite database in data/pglite.++DATABASE_URL=++DATABASE_DIRECT_URL=++VALKEY_URL=++VALKEY_PREFIX=++# The SQLite file of releases before PostgreSQL: read once by scripts/migrate-to-postgres.js.EVENTS_DB_PATH=./data/events.db# Events (and their deliveries) older than this are pruned; resuming from before it yields a gap.EVENTS_RETENTION_DAYS=30@@@ -9,6 +9,9 @@ jobs:uses: OpenVibers/OpenVibe.Shared/.github/workflows/test.yml@423fe519b5fe9594cc81b670e5ba5cb9f21f6176with:contracts-service: events++ # ADR-035: PostgreSQL 18 + PgBouncer (transaction mode) + Valkey 9 containers (openvibe-sdk++ # scripts/test-services.sh). npm test runs every file on PGlite; test:pg runs them all through PgBouncer.++ test: 'eval "$(node_modules/openvibe-sdk/scripts/test-services.sh up)" && npm test && npm run test:pg'# Secret scan (gitleaks) and dependency audit, shared by every OpenVibers repo (Track Q).security:uses: OpenVibers/OpenVibe.Shared/.github/workflows/security.yml@423fe519b5fe9594cc81b670e5ba5cb9f21f6176@@@ -90,11 +90,11 @@ A producer takes back what it published (a deleted chat message) with a directiv```- `redacts.event_ids` (up to 1000 `evt_…`) names events directly; `subject_type` + `subject_ids` (up to 1000) names every stored event of the same source about those subjects.--- In the transaction that stores the directive, each target becomes a **tombstone**: `payload` is replaced by `{ "redacted": true, "redacted_at": "<ISO>", "redacted_by": "<the redacting event_id>" }` and `actor` by the producer itself (`{ "type": "service", "id": "<source>" }`, or the app). `event_id`, `seq`, `event_type`, `subject`, `timestamp` and `visibility` stay, so sequences have no holes. The old payload is overwritten on disk (`secure_delete`), not just hidden.++- In the transaction that stores the directive, each target becomes a **tombstone**: `payload` is replaced by `{ "redacted": true, "redacted_at": "<ISO>", "redacted_by": "<the redacting event_id>" }` and `actor` by the producer itself (`{ "type": "service", "id": "<source>" }`, or the app). `event_id`, `seq`, `event_type`, `subject`, `timestamp` and `visibility` stay, so sequences have no holes. On PostgreSQL the replaced row version is a dead tuple until autovacuum reclaims it (the `events` table vacuums after 1% of its rows change, so within minutes); until then, and in backups taken before the redaction, the old bytes still exist on the host. No read path serves them.- Every read path serves the tombstone from then on: pull, `GET /api/v1/events/:id`, SSE replay (anonymous, signed in or service), queued deliveries and DLQ replays. First-party consumers get the tombstone at the original seq and then the deletion event itself; they should check `payload.redacted`.- Authority is the publish rule: only the owning source (for an app, its own project and environment) can redact. Naming another source's event by id refuses the whole publish with `403 events.redaction_not_allowed`; a subject match never reaches another source. A malformed directive is `422 events.invalid_redaction`. Events carrying a directive are never redacted themselves. No capability beyond `events.event.publish` is needed.- Redaction applies to what is stored when the directive arrives; it does not block later events about the same subject.--- Events stored before a producer published deletions: [scripts/redact-backfill.js](scripts/redact-backfill.js) (chat only; dry run by default, `--apply` needs `--backup <new file>`).++- Events stored before a producer published deletions: the one-time `scripts/redact-backfill.js` (chat only) ran on 2026-09-24 and was retired with the move to PostgreSQL (it is in git history).## Subscriptions and delivery@@@ -169,7 +169,8 @@ Quotas recorded in Network (`dev_quotas`) are not read yet: that needs `network.const events = require('openvibe-events');const tokenClient = contracts.serviceAuth.createTokenClient({ tokenUrl: `${NETWORK}/oauth/token`, clientId, clientSecret, audience: 'openvibe.events' });--// Producer: transactional outbox in the service's own better-sqlite3 database++// Producer: transactional outbox in the service's own better-sqlite3 database (a service on PostgreSQL uses++// openvibe-sdk/events createPgOutbox / createPgInbox instead: same envelope, receipts and relay, async)const outbox = events.createOutbox(db, { publisher: events.createPublisher({ eventsUrl: 'http://127.0.0.1:4300', tokenClient }) });outbox.ensureSchema();db.transaction(() => {@@@ -215,7 +216,7 @@ es.addEventListener('gap', (m) => { /* events were missed: refetch state */ });## Owns--- `events`, `subscriptions`, `deliveries`, `consumer_checkpoints`, `idempotency_receipts`, `app_revocations`, `app_usage` (SQLite today; the plan's PostgreSQL + Redis fanout is a later step)++- `events`, `subscriptions`, `deliveries`, `consumer_checkpoints`, `idempotency_receipts`, `app_revocations`, `app_usage` (PostgreSQL `ov_events`, ADR-035; the realtime fan-out is in-process, one Events process)- developer-app event scope, sandbox separation and per-project Events quotas (ADR-014)- canonical event envelope (event_id, trace_id, type, version, source, actor, subject + revision, payload)- priority classes `critical|important|low`, loop guards, backpressure, DLQ and replay@@@ -270,13 +271,15 @@ Reporting a vulnerability: [SECURITY.md](SECURITY.md). The rules the code keeps:Production deploys with `sudo ovhost deploy events` on the host (strategy `git-checkout`: fetch,fast-forward `/opt/openvibe.events`, install on a lockfile change, restart, wait for `/api/ready`).--The unit is `openvibe-events.service` on `127.0.0.1:4300`, the env file `/etc/openvibe/events.env`. The store is--`/var/lib/openvibe-events/events.db`; nginx serves `events.openvibe.network` from++The unit is `openvibe-events.service` on `127.0.0.1:4300`, the env file `/etc/openvibe/events.env`. The database is++`ov_events` on the host's data role (`sudo /opt/openvibe.host/roles/data/add-service.sh events` writes its settings); the++release migrates it at boot. The one-time move from SQLite is `scripts/migrate-to-postgres.js` (openvibe-sdk++`runSqliteMigration`, with a `--pglite` rehearsal mode), run while the service is stopped; the old++`/var/lib/openvibe-events/events.db` stays read-only for 7 days as the rollback. nginx serves `events.openvibe.network` from[deploy/nginx/events.openvibe.network.conf](deploy/nginx/events.openvibe.network.conf). ovhost treats openrealtime connections as a report-only drain, so `--wait-idle` waits for them.Rollback: ovhost puts the previous sha back by itself when `/api/ready` does not answer 2xx after the--restart; afterwards `sudo ovhost rollback events --to <sha>`. Nothing blocks a rollback: the schema--code only adds tables and columns.++restart; afterwards `sudo ovhost rollback events --to <sha>`. Migrations only add tables and columns.`EVENTS_LIMITS=off` turns the per-actor limits off without a deploy.## Launch rule@@@ -4,11 +4,19 @@"code": true,"wave": "W3","plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026)",-- "runtime": "node 22, express, better-sqlite3",++ "runtime": "node 22, express, PostgreSQL 18 (openvibe-sdk/db, async) + Valkey 9","port": 4300,"domain": "events.openvibe.network",-- "contracts": "openvibe-contracts v0.63.0",-- "capabilities": ["events.event.publish", "events.subscription.manage", "events.event.read", "events.delivery.admin", "events.app.publish", "events.app.read", "events.app.subscribe"],++ "contracts": "openvibe-contracts v0.76.0",++ "capabilities": [++ "events.event.publish",++ "events.subscription.manage",++ "events.event.read",++ "events.delivery.admin",++ "events.app.publish",++ "events.app.read",++ "events.app.subscribe"++ ],"capabilitiesRegistered": true,"deployed": true,"deployment": "Deployed with `sudo ovhost deploy events`: /opt/openvibe.events, unit openvibe-events, 127.0.0.1:4300; public at https://events.openvibe.network (realtime SSE, health, token-guarded /api/v1/events, /api/v1/subscriptions, /api/v1/checkpoints; /api/v1/deliveries host-local); real traffic since 2026-09-23 (Live, Media, Chat, Games, Network and OpenRe publish; on 2026-09-23 8 subscriptions, held by Search, Tips, VIP, News and Reviews); restore drill passed 2026-09-23",@@@ -22,6 +30,6 @@"developer-app publish, read and subscribe with sandbox separation, SSRF-guarded webhooks and project quotas","hourly project usage rollups (events.usage.recorded)","redaction tombstones",-- "per-actor limits (openvibe-sdk/limits)"++ "per-actor limits (openvibe-sdk/limits; shared on Valkey)"]}@@@ -2,8 +2,8 @@Description=OpenVibe.Events — durable events, signed delivery, dead letters, replay and the SSE realtime gateway# Network (openvibe-network) signs the tokens this service verifies. Events starts without it# and keeps retrying the JWKS fetch; /api/ready stays 503 until the key has loaded.--After=network.target--Wants=openvibe-network.service++After=network.target postgresql.service pgbouncer.service valkey-server.service++Wants=openvibe-network.service postgresql.service pgbouncer.service valkey-server.service[Service]Type=simple@@@ -26,7 +26,8 @@ NoNewPrivileges=truePrivateTmp=trueProtectSystem=strictProtectHome=true--# The code tree stays read-only; the event store lives in /var/lib/openvibe-events.++# The code tree stays read-only. The event store is PostgreSQL (ov_events); /var/lib/openvibe-events keeps the++# pre-PostgreSQL SQLite file (read-only) for the rollback window.StateDirectory=openvibe-eventsEnvironment=EVENTS_DB_PATH=/var/lib/openvibe-events/events.dbnnew file mode 100644@@@ -0,0 +1,129 @@++-- phase: expand++-- OpenVibe.Events on PostgreSQL (ADR-035, roadmap WS-X2): the tables as they were on SQLite (converted by openvibe-sdk++-- tools/asyncify/sqlite-schema-to-pg: text COLLATE "C" compares like SQLite, integers are bigint, identities keep their ids),++-- then the openvibe-publishing stores and the openvibe-sdk inbox and outbox. Generated once on 2026-09-28; never edited after it runs.++++CREATE TABLE sequences (++ name text COLLATE "C" PRIMARY KEY,++ value bigint NOT NULL++);++CREATE TABLE events (++ id text COLLATE "C" PRIMARY KEY,++ seq bigint NOT NULL UNIQUE,++ event_type text COLLATE "C" NOT NULL,++ version bigint NOT NULL,++ source text COLLATE "C" NOT NULL,++ actor text COLLATE "C" NOT NULL,++ subject_type text COLLATE "C" NOT NULL,++ subject_id text COLLATE "C" NOT NULL,++ subject_revision bigint,++ trace_id text COLLATE "C" NOT NULL,++ priority text COLLATE "C" NOT NULL,++ visibility text COLLATE "C" NOT NULL,++ occurred_at text COLLATE "C" NOT NULL,++ received_at bigint NOT NULL,++ payload text COLLATE "C" NOT NULL,++ hops bigint NOT NULL DEFAULT 1,++ publisher text COLLATE "C" NOT NULL,++ request_id text COLLATE "C"++);++CREATE INDEX idx_events_trace ON events(trace_id);++CREATE INDEX idx_events_received ON events(received_at);++CREATE INDEX idx_events_type_seq ON events(event_type, seq);++++CREATE TABLE subscriptions (++ id text COLLATE "C" PRIMARY KEY,++ consumer text COLLATE "C" NOT NULL,++ topic_pattern text COLLATE "C" NOT NULL,++ endpoint text COLLATE "C" NOT NULL,++ secret text COLLATE "C" NOT NULL,++ enabled bigint NOT NULL DEFAULT 1,++ retry_policy text COLLATE "C",++ created_at bigint NOT NULL,++ updated_at bigint NOT NULL++);++CREATE INDEX idx_subscriptions_consumer ON subscriptions(consumer);++++CREATE TABLE deliveries (++ event_id text COLLATE "C" NOT NULL REFERENCES events(id) ON DELETE CASCADE,++ subscription_id text COLLATE "C" NOT NULL REFERENCES subscriptions(id) ON DELETE CASCADE,++ seq bigint NOT NULL,++ priority bigint NOT NULL,++ attempt bigint NOT NULL DEFAULT 0,++ status text COLLATE "C" NOT NULL CHECK (status IN ('pending', 'delivered', 'failed', 'dead')),++ next_attempt_at bigint,++ last_error text COLLATE "C",++ last_status bigint,++ delivered_at bigint,++ created_at bigint NOT NULL,++ updated_at bigint NOT NULL,++ PRIMARY KEY (event_id, subscription_id)++);++CREATE INDEX idx_deliveries_due ON deliveries(status, next_attempt_at);++CREATE INDEX idx_deliveries_sub ON deliveries(subscription_id, status, seq);++++CREATE TABLE consumer_checkpoints (++ consumer text COLLATE "C" NOT NULL,++ topic_pattern text COLLATE "C" NOT NULL,++ cursor bigint NOT NULL,++ updated_at bigint NOT NULL,++ PRIMARY KEY (consumer, topic_pattern)++);++++-- Developer apps whose access Network revoked (network.app.revoked -> scope 'app') or whose++-- events.app.subscribe grant it withdrew (scope 'events.app.subscribe'). An app token issued at or++-- before revoked_at (ms, the revocation's own timestamp) is refused for that scope.++CREATE TABLE app_revocations (++ consumer text COLLATE "C" NOT NULL,++ scope text COLLATE "C" NOT NULL,++ revoked_at bigint NOT NULL,++ PRIMARY KEY (consumer, scope)++);++++-- Publish receipts: remember accepted event ids for longer than the events themselves, so a++-- producer that re-publishes an old id after retention is still answered "duplicate".++CREATE TABLE idempotency_receipts (++ consumer text COLLATE "C" NOT NULL,++ event_id text COLLATE "C" NOT NULL,++ processed_at bigint NOT NULL,++ PRIMARY KEY (consumer, event_id)++);++++-- Columns and indexes added after the first release (server/store.js ADDED_COLUMNS: developer apps, ADR-014, and redaction).++ALTER TABLE events ADD COLUMN project_id text COLLATE "C";++ALTER TABLE events ADD COLUMN env text COLLATE "C" NOT NULL DEFAULT 'production';++ALTER TABLE events ADD COLUMN size_bytes bigint NOT NULL DEFAULT 0;++ALTER TABLE events ADD COLUMN redacted_at bigint;++ALTER TABLE events ADD COLUMN redacted_by text COLLATE "C";++ALTER TABLE events ADD COLUMN redacts bigint NOT NULL DEFAULT 0;++ALTER TABLE subscriptions ADD COLUMN project_id text COLLATE "C";++ALTER TABLE subscriptions ADD COLUMN env text COLLATE "C" NOT NULL DEFAULT 'production';++ALTER TABLE subscriptions ADD COLUMN previous_secret text COLLATE "C";++ALTER TABLE subscriptions ADD COLUMN previous_secret_until bigint;++CREATE INDEX idx_events_project ON events(project_id, env, received_at) WHERE project_id IS NOT NULL;++CREATE INDEX idx_subscriptions_project ON subscriptions(project_id, env) WHERE project_id IS NOT NULL;++CREATE INDEX idx_events_subject ON events(source, subject_type, subject_id);++++-- The global event sequence (an import replaces the row with SQLite's).++INSERT INTO sequences (name, value) VALUES ('events', 0) ON CONFLICT DO NOTHING;++++-- server/usage.js: per-app usage windows (was created by the module).++CREATE TABLE app_usage (++ project_id text COLLATE "C" NOT NULL,++ env text COLLATE "C" NOT NULL,++ capability text COLLATE "C" NOT NULL,++ unit text COLLATE "C" NOT NULL,++ window_start bigint NOT NULL,++ quantity bigint NOT NULL DEFAULT 0,++ errors bigint NOT NULL DEFAULT 0,++ error_codes text COLLATE "C" NOT NULL DEFAULT '{}',++ samples text COLLATE "C" NOT NULL DEFAULT '[]',++ revision bigint NOT NULL DEFAULT 1,++ event_id text COLLATE "C",++ emitted_at bigint,++ PRIMARY KEY (project_id, env, capability, unit, window_start)++);++CREATE INDEX idx_app_usage_open ON app_usage(emitted_at, window_start);++++-- Redaction rewrites payloads in place and PostgreSQL has no secure_delete: reclaim the replaced row versions soon.++ALTER TABLE events SET (autovacuum_vacuum_scale_factor = 0.01, autovacuum_vacuum_threshold = 50);@@@ -12,14 +12,25 @@"better-sqlite3": "^11.10.0","dotenv": "^16.4.7","express": "^4.21.2",-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.63.0",-- "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0",-- "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0"++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",++ "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.20.4",++ "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.28.0",++ "pg": "^8.23.0"++ },++ "devDependencies": {++ "@electric-sql/pglite": "^0.5.8"},"engines": {"node": ">=20.0.0"}},++ "node_modules/@electric-sql/pglite": {++ "version": "0.5.8",++ "resolved": "https://registry.npmjs.org/@electric-sql/pglite/-/pglite-0.5.8.tgz",++ "integrity": "sha512-n9tsbUOhwx2epK1V0ZG9Ar4SHWUju04dhmzZXiSBXwBoleOvIfals33NAaWgagQVAL4Rbvx/Ptsu3P+pA09f6Q==",++ "devOptional": true,++ "license": "Apache-2.0"++ },"node_modules/accepts": {"version": "1.3.8","resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.8.tgz",@@@ -968,9 +979,9 @@}},"node_modules/openvibe-contracts": {-- "version": "0.63.0",-- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.63.0",-- "integrity": "sha512-vmc1b1vWg9M5fm57X50MlZEZHkP2IshHwBXt68+kHDV85nyg6FxaYyUWa4ibi1eeFqzMcECPlILSRTOgg2Xbgw==",++ "version": "0.76.0",++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",++ "integrity": "sha512-zTdx776KWrcJKm1JTnf+0+BnsvswdvXy1ldPyse1hSqhnW/SSr98sX696eRetFUhTl8qVM1xSRCykqIUbqItPw==","license": "AGPL-3.0-only","dependencies": {"ajv": "^8.17.1",@@@ -984,18 +995,38 @@}},"node_modules/openvibe-sdk": {-- "version": "0.12.0",-- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0",-- "integrity": "sha512-iG1h905O1f7y/PnP8xLA6GCVzXEBDfSaBj2tc1ucNreWg2sT/AJWosb8NW0WRMK5WS4sIQ6tMsprmDLj/fjjeQ==",++ "version": "0.20.4",++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.20.4",++ "integrity": "sha512-IMvvBrFhCVO4sm2Ii694BslcnKHSV7YABXviLqrDI/2lTr9NKIH80z3HXxwMUgj0fwZDWBWcNgqPs6OaWIB0GQ==","license": "MIT","engines": {"node": ">=20"++ },++ "peerDependencies": {++ "@electric-sql/pglite": "^0.5.8",++ "better-sqlite3": ">=11",++ "iovalkey": "^0.4.0",++ "pg": "^8.23.0"++ },++ "peerDependenciesMeta": {++ "@electric-sql/pglite": {++ "optional": true++ },++ "better-sqlite3": {++ "optional": true++ },++ "iovalkey": {++ "optional": true++ },++ "pg": {++ "optional": true++ }}},"node_modules/openvibe-shared": {-- "version": "1.25.0",-- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0",-- "integrity": "sha512-oqZ+PvljrWi8IEbtg6bDYai17ZV2sALFx/ezMptniELSr10TgFUiGUQt+b1IWwiHv0T2Y1U2iSrkirJxMNgjSw==",++ "version": "1.28.0",++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.28.0",++ "integrity": "sha512-WLd11lDG75veVi7mFwngquK2nqIQV86Z14Ad/xE2dYvyMT5mhKJxxxlKUoskpdyLeap0ucMvzzS8Tg7vPGwYoQ==","license": "MIT","dependencies": {"jsonwebtoken": "^9.0.0"@@@ -1016,6 +1047,134 @@"integrity": "sha512-A/AGNMFN3c8bOlvV9RreMdrv7jsmF9XIfDeCd87+I8RNg6s78BhJxMu69NEMHBSJFxKidViTEdruRwEk/WIKqA==","license": "MIT"},++ "node_modules/pg": {++ "version": "8.23.0",++ "resolved": "https://registry.npmjs.org/pg/-/pg-8.23.0.tgz",++ "integrity": "sha512-Ip2EQCngowJLGOfCwkFhPXU7/ljlhn6Rxlmy4XYfL2Y+vyRM59+8uR2xqRWKdYmbXmxCFOAmKxBuSUCdF34qLg==",++ "license": "MIT",++ "dependencies": {++ "pg-connection-string": "^2.14.0",++ "pg-pool": "^3.14.0",++ "pg-protocol": "^1.16.0",++ "pg-types": "2.2.0",++ "pgpass": "1.0.5"++ },++ "engines": {++ "node": ">= 16.0.0"++ },++ "optionalDependencies": {++ "pg-cloudflare": "^1.4.0"++ },++ "peerDependencies": {++ "pg-native": ">=3.0.1"++ },++ "peerDependenciesMeta": {++ "pg-native": {++ "optional": true++ }++ }++ },++ "node_modules/pg-cloudflare": {++ "version": "1.4.0",++ "resolved": "https://registry.npmjs.org/pg-cloudflare/-/pg-cloudflare-1.4.0.tgz",++ "integrity": "sha512-Vo7z/6rrQYxpNRylp4Tlob2elzbh+N/MOQbxFVWCxS7oEx6jF53GTJFxK2WWpKuBRkmiin4Mt+xofFDjx09R0A==",++ "license": "MIT",++ "optional": true++ },++ "node_modules/pg-connection-string": {++ "version": "2.14.0",++ "resolved": "https://registry.npmjs.org/pg-connection-string/-/pg-connection-string-2.14.0.tgz",++ "integrity": "sha512-XwWDGcLRGCXAR8F/AM5bG7Q+A3Wm2s6QeEjlOKZLlH3UYcguiqCWKyWXVag5TLTIjR7oOJUY8kcADaZgWPyLeg==",++ "license": "MIT"++ },++ "node_modules/pg-int8": {++ "version": "1.0.1",++ "resolved": "https://registry.npmjs.org/pg-int8/-/pg-int8-1.0.1.tgz",++ "integrity": "sha512-WCtabS6t3c8SkpDBUlb1kjOs7l66xsGdKpIPZsg4wR+B3+u9UAum2odSsF9tnvxg80h4ZxLWMy4pRjOsFIqQpw==",++ "license": "ISC",++ "engines": {++ "node": ">=4.0.0"++ }++ },++ "node_modules/pg-pool": {++ "version": "3.14.0",++ "resolved": "https://registry.npmjs.org/pg-pool/-/pg-pool-3.14.0.tgz",++ "integrity": "sha512-gKtPkFdQPU3DksooVLi9LsjZxrsBUZIpa+7aVx+LV5pNh0KzP4Zleud2po+ConrxbuXGBJ6Hfer6hdgpIBpBaw==",++ "license": "MIT",++ "peerDependencies": {++ "pg": ">=8.0"++ }++ },++ "node_modules/pg-protocol": {++ "version": "1.16.0",++ "resolved": "https://registry.npmjs.org/pg-protocol/-/pg-protocol-1.16.0.tgz",++ "integrity": "sha512-sILXutLVjCLjcDuOmvhX5e2Z4cS5qG/6Bu3VkpFwdf/633ElGLpEh9bgmuI5I4sqKqkifQiGyiCcx1HdtrK7tg==",++ "license": "MIT"++ },++ "node_modules/pg-types": {++ "version": "2.2.0",++ "resolved": "https://registry.npmjs.org/pg-types/-/pg-types-2.2.0.tgz",++ "integrity": "sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA==",++ "license": "MIT",++ "dependencies": {++ "pg-int8": "1.0.1",++ "postgres-array": "~2.0.0",++ "postgres-bytea": "~1.0.0",++ "postgres-date": "~1.0.4",++ "postgres-interval": "^1.1.0"++ },++ "engines": {++ "node": ">=4"++ }++ },++ "node_modules/pgpass": {++ "version": "1.0.5",++ "resolved": "https://registry.npmjs.org/pgpass/-/pgpass-1.0.5.tgz",++ "integrity": "sha512-FdW9r/jQZhSeohs1Z3sI1yxFQNFvMcnmfuj4WBMUTxOrAyLMaTcE1aAMBiTlbMNaXvBCQuVi0R7hd8udDSP7ug==",++ "license": "MIT",++ "dependencies": {++ "split2": "^4.1.0"++ }++ },++ "node_modules/postgres-array": {++ "version": "2.0.0",++ "resolved": "https://registry.npmjs.org/postgres-array/-/postgres-array-2.0.0.tgz",++ "integrity": "sha512-VpZrUqU5A69eQyW2c5CA1jtLecCsN2U/bD6VilrFDWq5+5UIEVO7nazS3TEcHf1zuPYO/sqGvUvW62g86RXZuA==",++ "license": "MIT",++ "engines": {++ "node": ">=4"++ }++ },++ "node_modules/postgres-bytea": {++ "version": "1.0.1",++ "resolved": "https://registry.npmjs.org/postgres-bytea/-/postgres-bytea-1.0.1.tgz",++ "integrity": "sha512-5+5HqXnsZPE65IJZSMkZtURARZelel2oXUEO8rH83VS/hxH5vv1uHquPg5wZs8yMAfdv971IU+kcPUczi7NVBQ==",++ "license": "MIT",++ "engines": {++ "node": ">=0.10.0"++ }++ },++ "node_modules/postgres-date": {++ "version": "1.0.7",++ "resolved": "https://registry.npmjs.org/postgres-date/-/postgres-date-1.0.7.tgz",++ "integrity": "sha512-suDmjLVQg78nMK2UZ454hAG+OAW+HQPZ6n++TNDUX+L0+uUlLywnoxJKDou51Zm+zTCjrCl0Nq6J9C5hP9vK/Q==",++ "license": "MIT",++ "engines": {++ "node": ">=0.10.0"++ }++ },++ "node_modules/postgres-interval": {++ "version": "1.2.0",++ "resolved": "https://registry.npmjs.org/postgres-interval/-/postgres-interval-1.2.0.tgz",++ "integrity": "sha512-9ZhXKM/rw350N1ovuWHbGxnGh/SNJ4cnxHiM0rxE4VN41wsg8P8zWn9hv/buK00RP4WvlOyr/RBDiptyxVbkZQ==",++ "license": "MIT",++ "dependencies": {++ "xtend": "^4.0.0"++ },++ "engines": {++ "node": ">=0.10.0"++ }++ },"node_modules/prebuild-install": {"version": "7.1.3","resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz",@@@ -1354,6 +1513,15 @@"simple-concat": "^1.0.0"}},++ "node_modules/split2": {++ "version": "4.2.0",++ "resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz",++ "integrity": "sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==",++ "license": "ISC",++ "engines": {++ "node": ">= 10.x"++ }++ },"node_modules/statuses": {"version": "2.0.2","resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz",@@@ -1481,6 +1649,15 @@"resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz","integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==","license": "ISC"++ },++ "node_modules/xtend": {++ "version": "4.0.2",++ "resolved": "https://registry.npmjs.org/xtend/-/xtend-4.0.2.tgz",++ "integrity": "sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ==",++ "license": "MIT",++ "engines": {++ "node": ">=0.4"++ }}}}@@@ -7,9 +7,16 @@"scripts": {"start": "node server/index.js","dev": "NODE_ENV=development node server/index.js",-- "test": "node test/run.js"++ "test": "node test/run.js",++ "test:pg": "EVENTS_TEST_STORE=pg node test/run.js"},-- "keywords": ["openvibe", "events", "outbox", "webhooks", "sse"],++ "keywords": [++ "openvibe",++ "events",++ "outbox",++ "webhooks",++ "sse"++ ],"author": "OpenVibers","license": "AGPL-3.0-only","repository": "github:OpenVibers/OpenVibe.Events",@@@ -17,11 +24,15 @@"better-sqlite3": "^11.10.0","dotenv": "^16.4.7","express": "^4.21.2",-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.63.0",-- "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0",-- "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0"++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",++ "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.20.4",++ "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.28.0",++ "pg": "^8.23.0"},"engines": {"node": ">=20.0.0"++ },++ "devDependencies": {++ "@electric-sql/pglite": "^0.5.8"}}nnew file mode 100755@@@ -0,0 +1,26 @@++#!/usr/bin/env node++'use strict';++/**++ * The one-time move of Events' SQLite database (EVENTS_DB_PATH) into its PostgreSQL schema (ADR-035; the procedure is++ * openvibe-sdk docs/migrating-to-postgresql.md, section 6, carried out by openvibe-sdk/db runSqliteMigration).++ *++ * node scripts/migrate-to-postgres.js [--sqlite <file>] [--pglite] [--json]++ *++ * Applies migrations/ as the owner (DATABASE_DIRECT_URL), copies every table into emptied tables, verifies row counts++ * and checksums, and exits 1 unless everything verified. The SQLite file is opened read-only. Every table keeps its++ * name and columns.++ */++require('dotenv').config();++const { runSqliteMigration } = require('openvibe-sdk/db');++const { load } = require('../server/config');++const { MIGRATIONS } = require('../server/store');++++const TABLES = {};++++if (require.main === module) {++ const config = load();++ runSqliteMigration({ service: 'events', sqlite: config.dbPath, directUrl: config.db.directUrl, migrations: MIGRATIONS, tables: TABLES })++ .then((code) => process.exit(code), (err) => { console.error(`migrate-to-postgres failed: ${err.message}`); process.exit(1); });++}++++module.exports = { TABLES };ddeleted file mode 100644@@@ -1,175 +0,0 @@--#!/usr/bin/env node--/**-- * One-off backfill: redact the stored chat events of messages OpenVibe.Chat deleted before Chat-- * published chat.message.deleted (server/redaction.js does it for every deletion since).-- *-- * node scripts/redact-backfill.js --chat-db <chat.db> # dry run: counts only-- * node scripts/redact-backfill.js --chat-db <chat.db> --apply --backup <new file> # backup, then redact-- *-- * Options:-- * --chat-db <file> Chat's database, opened read-only (production: /var/lib/openvibe-chat/chat.db)-- * --db <file> the Events database (default: $EVENTS_DB_PATH, else data/events.db; relative-- * paths from the repo root; production: /var/lib/openvibe-events/events.db)-- * --batch <n> messages per write transaction (default 500, at most 1000)-- * --include-missing also redact events about messages Chat no longer has at all (hard-deleted)-- *-- * A chat event is redacted when it is a first-party `chat` event about a `chat_message` subject-- * that Chat marks deleted (is_deleted = 1, or an auto-delete time that has passed). It becomes the-- * same tombstone a live chat.message.deleted produces (payload { redacted, redacted_at,-- * redacted_by: "backfill" }, actor service:chat), at the same seq. Messages Chat does not have at-- * all are counted and left alone, unless --include-missing (Chat hard-deleted them). Idempotent: redacted events are never selected again, so a-- * second run changes nothing.-- *-- * --apply refuses to run without --backup <file>: a new file, mode 0600, an sqlite online backup of-- * the whole Events database, verified with quick_check and a row count before anything is changed.-- * That backup still holds the text being removed: delete it once the run is verified. Safe while-- * Events is up (short transactions; secure_delete zeroes the old payloads). Run as the service-- * user, with the Events release that knows redaction deployed (it adds the columns).-- */--'use strict';--const fs = require('fs');--const path = require('path');--const Database = require('better-sqlite3');--const { createStore } = require('../server/store');----const ROOT = path.join(__dirname, '..');--const HELP_LINES = 26; // header lines 3..26--const BY = 'backfill';----function parseArgs(argv) {-- const a = { apply: false, backup: null, chatDb: null, db: null, batch: 500 };-- for (let i = 0; i < argv.length; i++) {-- const k = argv[i];-- const val = () => { if (i + 1 >= argv.length) throw new Error(`${k} needs a value`); return argv[++i]; };-- if (k === '--apply') a.apply = true;-- else if (k === '--backup') a.backup = val();-- else if (k === '--chat-db') a.chatDb = val();-- else if (k === '--db') a.db = val();-- else if (k === '--batch') a.batch = Number(val());-- else if (k === '--include-missing') a.includeMissing = true;-- else if (k === '-h' || k === '--help') a.help = true;-- else throw new Error(`unknown option ${k}`);-- }-- if (!Number.isInteger(a.batch) || a.batch < 1 || a.batch > 1000) throw new Error('--batch must be 1..1000');-- return a;--}----/** The database the server uses: --db, else $EVENTS_DB_PATH, else data/events.db (relative to the repo root). */--function defaultDbPath(args, env = process.env) {-- return path.resolve(ROOT, args.db || env.EVENTS_DB_PATH || path.join('data', 'events.db'));--}----const mb = (n) => `${(n / 1048576).toFixed(1)} MB`;--function fileBytes(f) { try { return fs.statSync(f).size; } catch { return 0; } }--function freeBytes(dir) { try { const s = fs.statfsSync(dir); return s.bavail * s.bsize; } catch { return null; } }--const chunks = (list, n) => Array.from({ length: Math.ceil(list.length / n) }, (_, i) => list.slice(i * n, (i + 1) * n));----/** Unredacted first-party chat events about chat messages: [{ id, seq, subject_id }]. */--function candidates(db) {-- return db.prepare(`SELECT id, seq, subject_id FROM events-- WHERE source = 'chat' AND subject_type = 'chat_message' AND project_id IS NULL AND redacted_at IS NULL AND redacts = 0-- ORDER BY seq`).all();--}----/** Split message ids into those Chat marks deleted and those it does not have. */--function classify(chat, messageIds) {-- const deleted = new Set();-- const present = new Set();-- for (const part of chunks(messageIds, 500)) {-- const marks = part.map(() => '?').join(',');-- for (const r of chat.prepare(`SELECT id, is_deleted,-- (auto_delete_at IS NOT NULL AND auto_delete_at <= CURRENT_TIMESTAMP) AS expired-- FROM chat_messages WHERE id IN (${marks})`).all(...part.map(Number))) {-- present.add(String(r.id));-- if (r.is_deleted || r.expired) deleted.add(String(r.id));-- }-- }-- return { deleted: [...deleted], missing: messageIds.filter(id => !present.has(id)) };--}----async function main(argv, log = console.log) {-- let args;-- try { args = parseArgs(argv); } catch (e) { log(`error: ${e.message}`); return 2; }-- if (args.help) { log(fs.readFileSync(__filename, 'utf8').split('\n').slice(2, HELP_LINES).join('\n')); return 0; }-- if (!args.chatDb) { log('error: --chat-db <chat.db> is required'); return 2; }-- const dbPath = defaultDbPath(args);-- const chatPath = path.resolve(args.chatDb);-- for (const f of [dbPath, chatPath]) if (!fs.existsSync(f)) { log(`error: ${f} does not exist`); return 2; }-- if (args.apply && !args.backup) {-- log('refusing --apply without --backup <new file> (an sqlite online backup of the Events database)');-- return 2;-- }---- const db = new Database(dbPath, args.apply ? { fileMustExist: true } : { readonly: true, fileMustExist: true });-- const chat = new Database(chatPath, { readonly: true, fileMustExist: true });-- try {-- db.pragma('busy_timeout = 5000');-- chat.pragma('busy_timeout = 5000');-- const cols = new Set(db.prepare('PRAGMA table_info(events)').all().map(c => c.name));-- if (!cols.has('redacted_at') || !cols.has('redacts')) {-- log('error: this Events database has no redaction columns yet; deploy and start the Events release with redaction first');-- return 2;-- }-- const rows = candidates(db);-- const byMessage = new Map();-- for (const r of rows) {-- if (!byMessage.has(r.subject_id)) byMessage.set(r.subject_id, []);-- byMessage.get(r.subject_id).push(r);-- }-- const { deleted, missing } = classify(chat, [...byMessage.keys()]);-- // --include-missing: a message Chat no longer has at all was hard-deleted (by hand, or by its-- // expiry sweep), so its text must not outlive it here either.-- const gone = args.includeMissing ? deleted.concat(missing) : deleted;-- const toRedact = gone.flatMap(id => byMessage.get(id)).sort((x, y) => x.seq - y.seq);-- const walBytes = fileBytes(dbPath + '-wal');-- log(`events db ${dbPath} (${mb(fileBytes(dbPath))} + ${mb(walBytes)} WAL)`);-- log(`chat db ${chatPath} (read-only)`);-- log(`chat events about messages, not redacted: ${rows.length} events, ${byMessage.size} messages`);-- log(`deleted in Chat: ${deleted.length} messages, ${toRedact.length} events to redact${toRedact.length ? ` (seq ${toRedact[0].seq}..${toRedact.at(-1).seq})` : ''}`);-- log(`not in Chat's database: ${missing.length} messages${args.includeMissing ? ' (redacted: --include-missing)' : ' (left alone; --include-missing redacts them)'}`);---- if (!args.apply) {-- log('\ndry run: nothing changed. Re-run with --apply --backup <new file>.');-- return 0;-- }---- const target = path.resolve(args.backup);-- if (fs.existsSync(target)) { log(`error: backup target ${target} already exists; choose a new file`); return 2; }-- const need = fileBytes(dbPath) + walBytes;-- const free = freeBytes(path.dirname(target));-- if (free != null && free < need * 1.1) { log(`error: ${mb(free)} free at ${path.dirname(target)}, the backup needs about ${mb(need)}`); return 2; }-- const total = db.prepare('SELECT COUNT(*) FROM events').pluck().get();-- // A full copy of the event store, deleted text included: owner-only from the first byte.-- const umask = process.umask(0o077);-- try { await db.backup(target); } finally { process.umask(umask); }-- fs.chmodSync(target, 0o600);-- const b = new Database(target, { readonly: true });-- try {-- const ok = b.pragma('quick_check', { simple: true });-- const n = b.prepare('SELECT COUNT(*) FROM events').pluck().get();-- if (ok !== 'ok' || n < total) { log(`error: backup check failed (quick_check=${ok}, events=${n} of ${total}); nothing changed`); return 1; }-- } finally { b.close(); }-- log(`backup ${target} (${mb(fileBytes(target))}, mode 0600, quick_check ok). It still holds the deleted text: remove it once this run is verified.`);---- db.pragma('secure_delete = ON');-- const store = createStore(db);-- let redacted = 0;-- for (const part of chunks(gone, args.batch)) {-- redacted += store.redact('chat', { subject_type: 'chat_message', subject_ids: part }, { by: BY }).length;-- }-- log(`redacted ${redacted} events`);-- try { db.pragma('wal_checkpoint(TRUNCATE)'); } catch { /* the server's next checkpoint does it */ }-- const left = classify(chat, [...new Set(candidates(db).map(r => r.subject_id))]).deleted.length;-- log(`now ${left} deleted messages with unredacted events`);-- return left ? 1 : 0;-- } finally {-- chat.close();-- db.close();-- }--}----if (require.main === module) {-- main(process.argv.slice(2)).then((code) => process.exit(code), (err) => { console.error(err); process.exit(1); });--}----module.exports = { main, parseArgs, defaultDbPath };@@@ -3,7 +3,7 @@/*** Operator rotation of a first-party subscription's secret, without the value ever being printed.*-- * node scripts/rotate-subscription-secret.js --db <events.db> (--subscription <sub_…> | --consumer <service>)++ * node scripts/rotate-subscription-secret.js (--subscription <sub_…> | --consumer <service>) (DATABASE_URL: the Events database)* --env-file </etc/openvibe/<consumer>.env> --env-var <NAME> [--overlap-s 86400] [--apply]** With --apply it (1) sets a new random secret on the subscription, keeping the old one signing next to it@@@ -15,12 +15,13 @@*/const fs = require('fs');const crypto = require('crypto');--const Database = require('better-sqlite3');++require('dotenv').config();++const { load } = require('../server/config');++const { openDb } = require('../server/store');const args = process.argv.slice(2);const opt = (k, d) => { const i = args.indexOf(k); return i >= 0 ? args[i + 1] : d; };const apply = args.includes('--apply');--const dbPath = opt('--db');const subId = opt('--subscription');const consumer = opt('--consumer'); // every first-party subscription of that service, to ONE new secret (they share one env variable)const envFile = opt('--env-file');@@@ -28,35 +29,36 @@ const envVar = opt('--env-var');const overlapS = Number(opt('--overlap-s', '86400'));function fail(msg) { console.error(`rotate-subscription-secret: ${msg}`); process.exit(1); }--if (!dbPath || !(subId || consumer) || !envFile || !envVar) fail('--db, --subscription or --consumer, --env-file and --env-var are required');++if (!(subId || consumer) || !envFile || !envVar) fail('--subscription or --consumer, --env-file and --env-var are required (the database is DATABASE_URL: PostgreSQL, ADR-035)');if (!/^[A-Z][A-Z0-9_]{1,63}$/.test(envVar)) fail('--env-var must be an env variable name');if (!Number.isFinite(overlapS) || overlapS < 0 || overlapS > 7 * 86400) fail('--overlap-s is 0..604800');if (!fs.existsSync(envFile)) fail(`${envFile} does not exist`);--const db = new Database(dbPath, { fileMustExist: true });++(async () => {++const db = await openDb(load(), { log: { log() {}, warn: console.warn } });const subs = consumer-- ? db.prepare('SELECT id, consumer, topic_pattern, project_id FROM subscriptions WHERE consumer = ? AND project_id IS NULL').all(consumer)-- : db.prepare('SELECT id, consumer, topic_pattern, project_id FROM subscriptions WHERE id = ?').all(subId);++ ? await db.prepare('SELECT id, consumer, topic_pattern, project_id FROM subscriptions WHERE consumer = ? AND project_id IS NULL').all(consumer)++ : await db.prepare('SELECT id, consumer, topic_pattern, project_id FROM subscriptions WHERE id = ?').all(subId);if (!subs.length) fail(`no subscription for ${consumer || subId}`);if (subs.some((x) => x.project_id)) fail('developer-app subscriptions rotate through the API (POST /api/v1/subscriptions/:id/rotate-secret)');const sub = subs[0];--const cols = new Set(db.prepare('PRAGMA table_info(subscriptions)').all().map((c) => c.name));--if (!cols.has('previous_secret')) fail('this Events database predates rotation; deploy Events first (it adds the columns at boot)');const lines = fs.readFileSync(envFile, 'utf8').split('\n');const has = lines.some((l) => l.startsWith(`${envVar}=`));console.log(`${subs.length} subscription(s) of ${sub.consumer} (${subs.map((x) => x.topic_pattern).join(', ')}); ${envFile}: ${envVar} ${has ? 'will be replaced' : 'will be appended'}; overlap ${overlapS}s`);--if (!apply) { console.log('dry run: nothing changed (add --apply)'); process.exit(0); }++if (!apply) { console.log('dry run: nothing changed (add --apply)'); await db.close(); return; }const secret = `whsec_${crypto.randomBytes(32).toString('hex')}`;const now = Date.now();fs.copyFileSync(envFile, `${envFile}.bak-${new Date(now).toISOString().replace(/[:.]/g, '-')}`);--db.transaction(() => {++await db.tx(async () => {const up = db.prepare('UPDATE subscriptions SET previous_secret = secret, previous_secret_until = ?, secret = ?, updated_at = ? WHERE id = ?');-- for (const x of subs) up.run(now + overlapS * 1000, secret, now, x.id);--})();++ for (const x of subs) await up.run(now + overlapS * 1000, secret, now, x.id);++});let out;if (has) out = lines.map((l) => (l.startsWith(`${envVar}=`) ? `${envVar}=${secret}` : l));else { out = lines.slice(); while (out.length && out[out.length - 1] === '') out.pop(); out.push(`${envVar}=${secret}`, ''); }fs.writeFileSync(envFile, out.join('\n'), { mode: fs.statSync(envFile).mode });console.log(`rotated: both secrets sign until ${new Date(now + overlapS * 1000).toISOString()}; ${envVar} updated. Restart ${sub.consumer} now.`);++await db.close();++})().catch((err) => fail(err.message));@@@ -14,10 +14,10 @@* Never limited: /api/health, /api/ready, /release.json, /limits.json, /metrics, and the realtime* stream (a connection, capped by REALTIME_MAX_CONNECTIONS and REALTIME_MAX_TOPICS).*/--const { createActorLimiter } = require('openvibe-sdk/limits');++const { createActorLimiter, createValkeyLimitStore } = require('openvibe-sdk/limits');/** limits(name, own) middleware for one app; `clock` is the service's (manual in tests). */--function createLimits({ config, clock = { now: () => Date.now() }, metrics = null, log = console }) {++function createLimits({ config, clock = { now: () => Date.now() }, metrics = null, log = console, valkey = null }) {// Every named limit with its numbers, as the routes declare them (published in /limits.json as rate_limits).const registered = new Map();const record = (name, own = {}) => registered.set(name, { minute: own.minute != null ? own.minute : config.limits.minute, hour: own.hour != null ? own.hour : config.limits.hour });@@@ -33,6 +33,8 @@ function createLimits({ config, clock = { now: () => Date.now() }, metrics = nulconst limiter = createActorLimiter({limits: { minute: config.limits.minute, hour: config.limits.hour },now: () => clock.now(),++ // Shared across processes on Valkey (ADR-035) when VALKEY_URL is set; in-process otherwise.++ ...(valkey ? { store: createValkeyLimitStore(valkey) } : {}),onLimited(e) {// The actor is a principal id (svc:…, app:app_…), never a token.log.warn(`[limits] ${e.name}: ${e.actor} refused, over ${e.limit} per ${e.window}`);@@@ -96,24 +96,24 @@ function publishRouter({ config, store, auth, worker, realtime, limits }) {return { env };}-- router.post('/api/v1/events', auth.appOrService(CAPS.publish, CAPS.appPublish, { requireService: true }), publishLimit, (req, res) => {++ router.post('/api/v1/events', auth.appOrService(CAPS.publish, CAPS.appPublish, { requireService: true }), publishLimit, async (req, res) => {const ctx = req.ov;const body = req.body;const p = req.principal;// Every refusal of an app's publish is one error in its project's usage.-- const refuse = (status, code, o = {}, ref = null) => {++ const refuse = async (status, code, o = {}, ref = null) => {if (p.kind === 'app') {try {-- store.usage.record({ projectId: p.projectId, env: p.env, capability: 'events.app.publish', unit: 'events', error: { code, status, traceId: ctx.traceId, ref } });++ await store.usage.record({ projectId: p.projectId, env: p.env, capability: 'events.app.publish', unit: 'events', error: { code, status, traceId: ctx.traceId, ref } });} catch (err) { console.error(`[usage] refusal not counted: ${err.message}`); }}return http.sendProblem(res, status, code, { ...o, ctx });};const isBatch = body && typeof body === 'object' && !Array.isArray(body) && Array.isArray(body.events) && body.event_id === undefined;const items = isBatch ? body.events : [body];-- if (isBatch && !items.length) return refuse(400, 'events.bad_request', { detail: 'events must not be empty' });++ if (isBatch && !items.length) return await refuse(400, 'events.bad_request', { detail: 'events must not be empty' });if (items.length > config.maxBatch) {-- return refuse(413, 'events.batch_too_large', { detail: `at most ${config.maxBatch} events per request` });++ return await refuse(413, 'events.batch_too_large', { detail: `at most ${config.maxBatch} events per request` });}const envelopes = [];@@@ -122,10 +122,10 @@ function publishRouter({ config, store, auth, worker, realtime, limits }) {const r = checkOne(items[i], req.principal, ctx);const ref = items[i] && typeof items[i] === 'object' ? items[i].event_id : null;if (r.env && seen.has(r.env.event_id)) {-- return refuse(422, 'events.invalid_envelope', { detail: `events[${i}]: event_id repeated within the batch`, extra: { index: i } }, ref);++ return await refuse(422, 'events.invalid_envelope', { detail: `events[${i}]: event_id repeated within the batch`, extra: { index: i } }, ref);}if (!r.env) {-- return refuse(r.status, r.code, {++ return await refuse(r.status, r.code, {detail: isBatch ? `events[${i}]: ${r.detail}` : r.detail, errors: r.errors, extra: isBatch ? { index: i } : undefined,}, ref);}@@@ -136,11 +136,11 @@ function publishRouter({ config, store, auth, worker, realtime, limits }) {let out;try {const project = p.kind === 'app' ? { projectId: p.projectId, env: p.env, ...config.apps.quotas[p.env] } : null;-- out = store.insertBatch(envelopes, { publisher: p.sub, requestId: ctx.requestId, project });++ out = await store.insertBatch(envelopes, { publisher: p.sub, requestId: ctx.requestId, project });} catch (err) {if (err instanceof StoreError) {if (err.status === 429 && err.extra && err.extra.retry_after) res.setHeader('Retry-After', String(err.extra.retry_after));-- return refuse(err.status, err.code, { detail: err.message, extra: err.extra }, err.extra && err.extra.event_id);++ return await refuse(err.status, err.code, { detail: err.message, extra: err.extra }, err.extra && err.extra.event_id);}throw err;}@@@ -59,7 +59,7 @@ function readRouter({ store, auth, worker, limits }) {// may store its checkpoint after each page: 600 a minute each, so a catch-up is never throttled.const pull = limits('events.event.pull', { minute: 600, hour: 20000 });-- router.get('/api/v1/events', canRead, pull, (req, res) => {++ router.get('/api/v1/events', canRead, pull, async (req, res) => {const ctx = req.ov;const patterns = String(req.query.topic || '*').split(',').map(s => s.trim()).filter(Boolean);if (!patterns.length || patterns.length > 20 || !patterns.every(topics.isValidPattern)) {@@@ -72,22 +72,22 @@ function readRouter({ store, auth, worker, limits }) {if (Number.isNaN(after) || Number.isNaN(limit)) {return http.sendProblem(res, 400, 'events.bad_request', { detail: 'after_seq must be >= 0 and limit 1..1000', ctx });}-- const oldest = store.oldestSeq();++ const oldest = await store.oldestSeq();const out = {};let from = after;if (after < oldest - 1) {out.gap = { from_seq: after + 1, to_seq: oldest - 1 };from = oldest - 1;}-- const { rows, cursor } = store.scan(from, { patterns, limit, accept: acceptFor(req.principal, patterns) });++ const { rows, cursor } = await store.scan(from, { patterns, limit, accept: acceptFor(req.principal, patterns) });out.events = rows.map(r => ({ seq: r.seq, event: rowToEnvelope(r) }));out.next_after_seq = cursor;-- out.latest_seq = store.lastSeq();++ out.latest_seq = await store.lastSeq();res.json(out);});-- router.get('/api/v1/events/:id', canRead, limits('events.event.read'), (req, res) => {-- const row = store.getEvent(String(req.params.id));++ router.get('/api/v1/events/:id', canRead, limits('events.event.read'), async (req, res) => {++ const row = await store.getEvent(String(req.params.id));const visible = row && (req.principal.kind === 'app' ? apps.visibleToApp(row, req.principal) : (row.env || 'production') === 'production');if (!visible) return http.sendProblem(res, 404, 'events.not_found', { detail: 'no such event (or pruned by retention)', ctx: req.ov });return res.json({ seq: row.seq, event: rowToEnvelope(row) });@@@ -95,27 +95,27 @@ function readRouter({ store, auth, worker, limits }) {const consumerOf = (req) => req.principal.service || req.principal.sub;-- router.get('/api/v1/checkpoints', canRead, limits('events.checkpoint.read'), (req, res) => {++ router.get('/api/v1/checkpoints', canRead, limits('events.checkpoint.read'), async (req, res) => {const topic = String(req.query.topic || '');if (!topics.isValidPattern(topic)) return http.sendProblem(res, 400, 'events.bad_topic', { detail: 'topic is required', ctx: req.ov });const scopeErr = scopeError(req.principal, [topic]);if (scopeErr) return http.sendProblem(res, 403, 'events.topic_not_allowed', { detail: scopeErr, ctx: req.ov });-- const cp = store.getCheckpoint(consumerOf(req), topic);++ const cp = await store.getCheckpoint(consumerOf(req), topic);res.json({ consumer: consumerOf(req), topic, cursor: cp ? cp.cursor : 0, updated_at: cp ? cp.updated_at : null });});-- router.put('/api/v1/checkpoints', canRead, limits('events.checkpoint.write', { minute: 600, hour: 20000 }), (req, res) => {++ router.put('/api/v1/checkpoints', canRead, limits('events.checkpoint.write', { minute: 600, hour: 20000 }), async (req, res) => {const b = req.body || {};if (!topics.isValidPattern(b.topic) || !Number.isInteger(b.cursor) || b.cursor < 0) {return http.sendProblem(res, 400, 'events.bad_request', { detail: 'topic (pattern) and cursor (integer >= 0) are required', ctx: req.ov });}const scopeErr = scopeError(req.principal, [b.topic]);if (scopeErr) return http.sendProblem(res, 403, 'events.topic_not_allowed', { detail: scopeErr, ctx: req.ov });-- const cp = store.setCheckpoint(consumerOf(req), b.topic, b.cursor);++ const cp = await store.setCheckpoint(consumerOf(req), b.topic, b.cursor);res.json({ consumer: consumerOf(req), topic: b.topic, ...cp });});-- router.get('/api/v1/deliveries', isAdmin, limits('events.delivery.list'), (req, res) => {++ router.get('/api/v1/deliveries', isAdmin, limits('events.delivery.list'), async (req, res) => {const status = req.query.status ? String(req.query.status) : undefined;if (status && !['pending', 'delivered', 'failed', 'dead'].includes(status)) {return http.sendProblem(res, 400, 'events.bad_request', { detail: 'status is pending|delivered|failed|dead', ctx: req.ov });@@@ -123,7 +123,7 @@ function readRouter({ store, auth, worker, limits }) {const limit = intParam(req.query.limit, 100, 1, 1000);const afterSeq = intParam(req.query.after_seq, 0, 0, Number.MAX_SAFE_INTEGER);if (Number.isNaN(limit) || Number.isNaN(afterSeq)) return http.sendProblem(res, 400, 'events.bad_request', { detail: 'bad limit or after_seq', ctx: req.ov });-- const rows = store.listDeliveries({ status, subscriptionId: req.query.subscription_id ? String(req.query.subscription_id) : undefined, limit, afterSeq });++ const rows = await store.listDeliveries({ status, subscriptionId: req.query.subscription_id ? String(req.query.subscription_id) : undefined, limit, afterSeq });return res.json({deliveries: rows.map(d => ({event_id: d.event_id,@@@ -136,16 +136,16 @@ function readRouter({ store, auth, worker, limits }) {last_error: d.last_error,delivered_at: d.delivered_at ? new Date(d.delivered_at).toISOString() : null,})),-- counts: store.deliveryCounts(),++ counts: await store.deliveryCounts(),});});// A replay requeues up to a whole subscription's retained history: an operator's action, a few// times at most.-- router.post('/api/v1/deliveries/replay', isAdmin, limits('events.delivery.replay', { minute: 6, hour: 60 }), (req, res) => {++ router.post('/api/v1/deliveries/replay', isAdmin, limits('events.delivery.replay', { minute: 6, hour: 60 }), async (req, res) => {const ctx = req.ov;const b = req.body || {};-- const sub = typeof b.subscription_id === 'string' ? store.getSubscription(b.subscription_id) : null;++ const sub = typeof b.subscription_id === 'string' ? await store.getSubscription(b.subscription_id) : null;if (!sub) return http.sendProblem(res, 404, 'events.not_found', { detail: 'no such subscription', ctx });const hasIds = Array.isArray(b.event_ids);const hasFrom = Number.isInteger(b.from_seq) && b.from_seq >= 0;@@@ -153,7 +153,7 @@ function readRouter({ store, auth, worker, limits }) {if (hasIds && (b.event_ids.length > 1000 || !b.event_ids.every(id => typeof id === 'string'))) {return http.sendProblem(res, 400, 'events.bad_request', { detail: 'event_ids: at most 1000 ids', ctx });}-- const queued = store.requeue(sub, hasIds ? { eventIds: b.event_ids } : { fromSeq: b.from_seq });++ const queued = await store.requeue(sub, hasIds ? { eventIds: b.event_ids } : { fromSeq: b.from_seq });worker.kick();return res.json({ subscription_id: sub.id, queued });});@@@ -57,10 +57,10 @@ function subscriptionsRouter({ config, store, auth, dnsLookup, limits }) {const change = { minute: 30, hour: 300 };/** The app's events.app.subscribe grant was withdrawn after this token was issued. */-- function subscribeRevoked(req, res) {++ async function subscribeRevoked(req, res) {const p = req.principal;if (p.kind !== 'app') return false;-- const at = store.revokedAt(p.sub, CAPS.appSubscribe);++ const at = await store.revokedAt(p.sub, CAPS.appSubscribe);if (at && p.iat * 1000 <= at) {http.sendProblem(res, 403, 'capability.denied', { detail: `${CAPS.appSubscribe} was revoked after this token was issued`, ctx: req.ov });return true;@@@ -71,8 +71,8 @@ function subscriptionsRouter({ config, store, auth, dnsLookup, limits }) {/** An app's subscriptions are those of its token's project and environment too (never across sandbox/production). */const inScope = (req, sub) => req.principal.kind !== 'app' || (sub.project_id === req.principal.projectId && (sub.env || 'production') === req.principal.env);-- function own(req, res) {-- const sub = store.getSubscription(String(req.params.id));++ async function own(req, res) {++ const sub = await store.getSubscription(String(req.params.id));if (!sub || sub.consumer !== consumerOf(req) || !inScope(req, sub)) {http.sendProblem(res, 404, 'events.not_found', { detail: 'no such subscription', ctx: req.ov });return null;@@@ -96,7 +96,7 @@ function subscriptionsRouter({ config, store, auth, dnsLookup, limits }) {return http.sendProblem(res, 422, 'events.bad_topic', { detail: 'topic_pattern must be dot-separated segments of [a-z0-9_] or *', ctx });}const app = req.principal.kind === 'app' ? req.principal : null;-- if (subscribeRevoked(req, res)) return undefined;++ if (await subscribeRevoked(req, res)) return undefined;if (app) {const err = apps.patternScopeError(pattern, app);if (err) return http.sendProblem(res, 403, 'events.topic_not_allowed', { detail: err, ctx });@@@ -111,22 +111,22 @@ function subscriptionsRouter({ config, store, auth, dnsLookup, limits }) {const consumer = consumerOf(req);const endpoint = ep.url.toString();-- const dup = store.listSubscriptions(consumer).find(s => s.topic_pattern === pattern && s.endpoint === endpoint);++ const dup = (await store.listSubscriptions(consumer)).find(s => s.topic_pattern === pattern && s.endpoint === endpoint);if (dup) {return http.sendProblem(res, 409, 'events.subscription_exists', { detail: 'this consumer already subscribes that endpoint to that topic', ctx, extra: { subscription_id: dup.id } });}-- if (store.countSubscriptions(consumer) >= config.maxSubscriptionsPerConsumer) {++ if (await store.countSubscriptions(consumer) >= config.maxSubscriptionsPerConsumer) {return http.sendProblem(res, 409, 'events.subscription_limit', { detail: `at most ${config.maxSubscriptionsPerConsumer} subscriptions per consumer`, ctx });}if (app) {const max = config.apps.quotas[app.env].maxSubscriptions;-- if (max && store.countProjectSubscriptions(app.projectId, app.env) >= max) {++ if (max && await store.countProjectSubscriptions(app.projectId, app.env) >= max) {return http.sendProblem(res, 429, 'events.quota_exceeded', {detail: `at most ${max} subscriptions per project in ${app.env}`, ctx, extra: { quota: 'subscriptions', limit: max },});}}-- const row = store.createSubscription({++ const row = await store.createSubscription({id: `sub_${ids.ulid()}`,consumer,topicPattern: pattern,@@@ -139,20 +139,20 @@ function subscriptionsRouter({ config, store, auth, dnsLookup, limits }) {return res.status(201).json(subscriptionView(row, { withSecret: true }));}-- router.get('/api/v1/subscriptions', guard, limits('events.subscription.list'), (req, res) => {-- res.json({ subscriptions: store.listSubscriptions(consumerOf(req)).filter(s => inScope(req, s)).map(s => subscriptionView(s)) });++ router.get('/api/v1/subscriptions', guard, limits('events.subscription.list'), async (req, res) => {++ res.json({ subscriptions: (await store.listSubscriptions(consumerOf(req))).filter(s => inScope(req, s)).map(s => subscriptionView(s)) });});-- router.get('/api/v1/subscriptions/:id', guard, limits('events.subscription.read'), (req, res) => {-- const sub = own(req, res);++ router.get('/api/v1/subscriptions/:id', guard, limits('events.subscription.read'), async (req, res) => {++ const sub = await own(req, res);if (sub) res.json(subscriptionView(sub));});// POST /api/v1/subscriptions/:id/rotate-secret { secret?, overlap_s? } -> the new secret (shown once). The old// secret keeps signing next to it for overlap_s (default 1 day, at most 7), so the consumer can switch without// missing a delivery.-- router.post('/api/v1/subscriptions/:id/rotate-secret', guard, limits('events.subscription.rotate', { minute: 10, hour: 100 }), express.json({ limit: '4kb' }), (req, res) => {-- const sub = own(req, res);++ router.post('/api/v1/subscriptions/:id/rotate-secret', guard, limits('events.subscription.rotate', { minute: 10, hour: 100 }), express.json({ limit: '4kb' }), async (req, res) => {++ const sub = await own(req, res);if (!sub) return;const b = req.body || {};if (b.secret !== undefined && (typeof b.secret !== 'string' || b.secret.length < 32 || b.secret.length > 256)) {@@@ -163,16 +163,16 @@ function subscriptionsRouter({ config, store, auth, dnsLookup, limits }) {return http.sendProblem(res, 422, 'events.bad_request', { detail: 'overlap_s is 0..604800', ctx: req.ov });}const secret = b.secret || `whsec_${crypto.randomBytes(32).toString('hex')}`;-- const out = store.rotateSubscriptionSecret(sub.id, secret, overlapS * 1000);++ const out = await store.rotateSubscriptionSecret(sub.id, secret, overlapS * 1000);res.json({ ...subscriptionView(out), secret, previous_secret_valid_until: new Date(out.previous_secret_until).toISOString() });});const toggle = limits('events.subscription.toggle', change);for (const [action, enabled] of [['disable', false], ['enable', true]]) {-- router.post(`/api/v1/subscriptions/:id/${action}`, guard, toggle, (req, res) => {-- if (enabled && subscribeRevoked(req, res)) return;-- const sub = own(req, res);-- if (sub) res.json(subscriptionView(store.setSubscriptionEnabled(sub.id, enabled)));++ router.post(`/api/v1/subscriptions/:id/${action}`, guard, toggle, async (req, res) => {++ if (enabled && await subscribeRevoked(req, res)) return;++ const sub = await own(req, res);++ if (sub) res.json(subscriptionView(await store.setSubscriptionEnabled(sub.id, enabled)));});}@@@ -13,7 +13,7 @@ const { mountLimits } = require('./limits');const { createLimits } = require('./actor-limits');const pkg = require('../package.json');--function createApp({ config, store, auth, keys, worker, realtime, metrics, dnsLookup, clock, log = console }) {++function createApp({ config, store, auth, keys, worker, realtime, metrics, dnsLookup, clock, log = console, valkey = null }) {const app = express();app.disable('x-powered-by');app.set('trust proxy', 'loopback');@@@ -44,13 +44,15 @@ function createApp({ config, store, auth, keys, worker, realtime, metrics, dnsLo// Readiness (openvibe-shared/ready): 503 only when a required check fails. A DLQ past// EVENTS_DLQ_DEGRADED_AT degrades the service (still ready: new events are accepted and delivered).const checks = [-- { name: 'db', required: true, check: () => store.ping() },++ // A real round trip that names the store (postgresql / pglite).++ { name: 'db', required: true, check: async () => { const r = await store.db.ready(); return r.ok ? { ok: true, detail: r.detail } : r.error; } },++ { name: 'valkey', required: false, check: async () => (valkey ? valkey.ready() : { skipped: 'VALKEY_URL not set: per-actor limits count in this process only' }) },{ name: 'network_jwks', required: true, check: () => keys.loaded() || 'Network signing key not loaded yet' },];if (config.worker.enabled) checks.push({ name: 'delivery_worker', required: true, check: () => worker.running() || 'delivery worker is not running' });checks.push({-- name: 'dlq', required: false, check: () => {-- const depth = store.deliveryCounts().dead;++ name: 'dlq', required: false, check: async () => {++ const depth = (await store.deliveryCounts()).dead;return depth > config.dlqDegradedAt? { ok: false, error: `${depth} dead deliveries (threshold ${config.dlqDegradedAt})`, detail: { depth, threshold: config.dlqDegradedAt } }: { ok: true, detail: { depth, threshold: config.dlqDegradedAt } };@@@ -58,11 +60,11 @@ function createApp({ config, store, auth, keys, worker, realtime, metrics, dnsLo});const readiness = createReadiness({service: 'events', release: release.release, checks,-- details: (body) => {++ details: async (body) => {const dbOk = body.checks.db.status === 'ok';return {-- latest_seq: dbOk ? store.lastSeq() : null,-- deliveries: dbOk ? store.deliveryCounts() : null,++ latest_seq: dbOk ? await store.lastSeq() : null,++ deliveries: dbOk ? await store.deliveryCounts() : null,worker: { enabled: config.worker.enabled, ...worker.stats() },realtime_connections: realtime.count(),};@@@ -72,7 +74,7 @@ function createApp({ config, store, auth, keys, worker, realtime, metrics, dnsLo// One per-actor limiter for the capability routes below (server/actor-limits.js); health, ready,// release.json, limits.json, metrics and the realtime stream above are never limited.-- const limits = createLimits({ config, clock, metrics, log });++ const limits = createLimits({ config, clock, metrics, log, valkey });app.use(publishRouter({ config, store, auth, worker, realtime, limits }));app.use(subscriptionsRouter({ config, store, auth, dnsLookup, limits }));app.use(readRouter({ store, auth, worker, limits }));@@@ -72,7 +72,7 @@ function createKeyStore({ urls = [], pem = null, fetchImpl = globalThis.fetch, lreturn null;}-- function start() {++ async function start() {if (pem) return Promise.resolve(key);const attempt = async () => {const k = await fetchOnce();@@@ -84,7 +84,7 @@ function createKeyStore({ urls = [], pem = null, fetchImpl = globalThis.fetch, l};refreshTimer = setInterval(() => { fetchOnce().catch(() => {}); }, 6 * 60 * 60 * 1000);refreshTimer.unref?.();-- return attempt();++ return await attempt();}function stop() {@@@ -259,7 +259,7 @@ function createAuth({ config, keys, store = null }) {*/function appOrService(serviceCap, appCap, { requireService = false } = {}) {const serviceGuard = requireCap(serviceCap, { requireService });-- return function appOrServiceGuard(req, res, next) {++ return async function appOrServiceGuard(req, res, next) {const ctx = req.ov;const token = bearer(req);if (!token || !config.apps.enabled) return serviceGuard(req, res, next);@@@ -268,7 +268,7 @@ function createAuth({ config, keys, store = null }) {const principal = apps.appPrincipal(r.claims);if (principal.error) return http.sendProblem(res, 401, 'token.invalid_claims', { detail: principal.error, ctx });// Network revoked this app after the token was issued (network.app.revoked reached us).-- const revoked = store && store.revokedAt(principal.sub, 'app');++ const revoked = store && await store.revokedAt(principal.sub, 'app');if (revoked && r.claims.iat * 1000 <= revoked) return http.sendProblem(res, 401, 'token.revoked', { detail: 'this app was revoked', ctx });const c = allows(r.claims, appCap);if (!c.allowed) return http.sendProblem(res, 403, c.code, { detail: c.reason, ctx });@@@ -64,6 +64,11 @@ function load(env = process.env) {// openvibe.events user tokens yet; the network-wide audience is the stable one).userAudiences: list(env.EVENTS_USER_AUDIENCES, ['openvibe.events', 'openvibe.network']),++ // PostgreSQL (ADR-035): DATABASE_URL serves (PgBouncer), DATABASE_DIRECT_URL migrates (owner role).++ db: { url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '' },++ // Valkey (ADR-035): per-actor limit counters shared across processes; without it they count in this process.++ valkey: { url: env.VALKEY_URL || '', prefix: env.VALKEY_PREFIX || 'ov:events:' },++ // The SQLite file of releases before the switch: read once by scripts/migrate-to-postgres.js.dbPath: env.EVENTS_DB_PATH || './data/events.db',retentionDays: int(env.EVENTS_RETENTION_DAYS, 30),receiptRetentionDays: int(env.EVENTS_RECEIPT_RETENTION_DAYS, 90),@@@ -17,11 +17,12 @@ const { createMetrics } = require('./metrics');const { createGuardedPost } = require('./egress');async function start({-- config, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, deliveryFetch = fetchImpl,++ config, db: givenDb = null, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, deliveryFetch = fetchImpl,appPost = undefined, dnsLookup = undefined, log = console, listen = true,} = {}) {config = config || load();-- const db = openDb(config.dbPath);++ // PostgreSQL (ADR-035): opened and migrated here unless the caller (a test) hands in a migrated handle.++ const db = givenDb || await openDb(config, { log });const store = createStore(db, { clock, maxHops: config.maxHops, usage: config.usage });const keys = createKeyStore({ urls: [config.networkInternalUrl, config.networkUrl], pem: config.networkPublicKey, fetchImpl, log });const auth = createAuth({ config, keys, store });@@@ -31,16 +32,17 @@ async function start({const worker = createWorker({ store, config, clock, fetchImpl: deliveryFetch, appPost: appPost || createGuardedPost({ lookup: dnsLookup }), log, observe: metrics.observe });const realtime = createRealtime({ store, auth, config, clock, log });metrics.bind({ realtime });-- const app = createApp({ config, store, auth, keys, worker, realtime, metrics, dnsLookup, clock, log });++ const valkey = config.valkey.url ? require('openvibe-sdk/valkey').createValkey({ url: config.valkey.url, prefix: config.valkey.prefix, log }) : null;++ const app = createApp({ config, store, auth, keys, worker, realtime, metrics, dnsLookup, clock, log, valkey });// The key loads in the background (retrying while Network boots); /api/ready says when it has.const keyLoaded = keys.start().catch(() => null);if (config.worker.enabled) worker.start();realtime.start();-- const prune = () => {++ const prune = async () => {try {-- const r = store.prune({ retentionDays: config.retentionDays, receiptRetentionDays: config.receiptRetentionDays, sandboxRetentionDays: config.apps.sandboxRetentionDays });++ const r = await store.prune({ retentionDays: config.retentionDays, receiptRetentionDays: config.receiptRetentionDays, sandboxRetentionDays: config.apps.sandboxRetentionDays });if (r.events || r.receipts) log.log(`[retention] pruned ${r.events} events, ${r.receipts} receipts`);} catch (err) {log.error(`[retention] prune failed: ${err.message}`);@@@ -48,13 +50,13 @@ async function start({};const pruneTimer = setInterval(prune, config.pruneIntervalMs);pruneTimer.unref?.();-- if (config.worker.enabled) prune();++ if (config.worker.enabled) await prune();// Project usage rollups (server/usage.js): each closed hour is stored as events.usage.recorded and// fanned out like a published event.-- const flushUsage = (opts) => {++ const flushUsage = async (opts) => {try {-- const r = store.flushUsage(opts);++ const r = await store.flushUsage(opts);if (r.stored.length) {realtime.publish(r.stored);worker.kick();@@@ -91,7 +93,8 @@ async function start({server.closeAllConnections?.();await new Promise(resolve => server.close(() => resolve()));}-- db.close();++ if (valkey) await valkey.close();++ if (!givenDb) await db.close();}return { config, db, store, keys, keyLoaded, auth, worker, realtime, metrics, app, server, close, prune, flushUsage };@@@ -22,10 +22,10 @@ function createMetrics({ store }) {registry.gauge({name: 'events_deliveries', help: 'Deliveries by status', labelNames: ['status'],-- collect: () => Object.entries(store.deliveryCounts()).map(([status, value]) => ({ labels: { status }, value })),++ collect: async () => Object.entries(await store.deliveryCounts()).map(([status, value]) => ({ labels: { status }, value })),});-- registry.gauge({ name: 'events_dlq_depth', help: 'Dead deliveries (the DLQ) waiting for replay', collect: () => store.deliveryCounts().dead });-- registry.gauge({ name: 'events_latest_seq', help: 'Last sequence number assigned to an event', collect: () => store.lastSeq() });++ registry.gauge({ name: 'events_dlq_depth', help: 'Dead deliveries (the DLQ) waiting for replay', collect: async () => (await store.deliveryCounts()).dead });++ registry.gauge({ name: 'events_latest_seq', help: 'Last sequence number assigned to an event', collect: async () => await store.lastSeq() });registry.gauge({ name: 'events_realtime_connections', help: 'Open realtime (SSE) connections', collect: () => (realtime ? realtime.count() : undefined) });const latency = registry.histogram({ name: 'events_delivery_latency_seconds', help: 'Time from acceptance to successful delivery, retries included', labelNames: ['priority'], buckets: LATENCY_BUCKETS });const attempts = registry.counter({ name: 'events_delivery_attempts_total', help: 'Delivery attempts by outcome', labelNames: ['outcome'] });@@@ -106,7 +106,7 @@ function createRealtime({ store, auth, config, clock = { now: () => Date.now() }return [...new Set(list)];}-- function handler(req, res) {++ async function handler(req, res) {const ctx = req.ov;const wanted = parseTopics(req.query.topics);if (!wanted.length) return http.sendProblem(res, 400, 'realtime.bad_request', { detail: 'topics=<pattern>[,<pattern>] is required', ctx });@@@ -138,46 +138,55 @@ function createRealtime({ store, auth, config, clock = { now: () => Date.now() }req.socket.setTimeout(0);req.socket.setNoDelay(true);-- const conn = { res, viewer, patterns: wanted, lastSeq: lastId ?? store.lastSeq(), closed: false };++ // The connection takes its slot and joins the fan-out before the first await, so the cap holds and nothing++ // published during the catch-up is lost: those rows wait in `pending` and follow the replay, skipping what it sent.++ const conn = { res, viewer, patterns: wanted, lastSeq: lastId, pending: [], closed: false };++ conns.add(conn);++ res.on('close', () => close(conn));write(conn, `retry: 3000\n: connected ${viewer.kind}\n\n`);---- // Catch up synchronously (better-sqlite3 is synchronous, so no event can be published in-- // between), then join the live fan-out.-- if (lastId != null) {-- const latest = store.lastSeq();-- const oldest = store.oldestSeq();-- if (lastId > latest) {-- sendGap(conn, { reason: 'cursor_ahead', from_seq: latest + 1, to_seq: lastId, latest_seq: latest });-- conn.lastSeq = latest;-- } else {-- let cursor = lastId;-- if (lastId < oldest - 1) {-- sendGap(conn, { reason: 'retention', from_seq: lastId + 1, to_seq: oldest - 1, latest_seq: latest });-- cursor = oldest - 1;-- }-- // Browsers: public events older than the window are not replayed (a gap says so).-- const publicFrom = viewer.kind === 'service' ? 0-- : opts.publicReplaySeconds > 0 ? store.firstSeqSince(clock.now() - opts.publicReplaySeconds * 1000) : latest + 1;-- if (cursor < publicFrom - 1) {-- sendGap(conn, { reason: 'public_window', from_seq: cursor + 1, to_seq: publicFrom - 1, latest_seq: latest, window_seconds: opts.publicReplaySeconds });-- // Signed out, only public events are visible: nothing before the window to scan.-- if (viewer.kind === 'anonymous') cursor = publicFrom - 1;-- }-- const accept = (row) => visibleTo(viewer, row) && !(row.visibility === 'public' && row.seq < publicFrom);-- const { rows, cursor: scanned } = store.scan(cursor, {-- patterns: wanted, limit: opts.replayMax, scanMax: opts.replayMax * 50, accept,-- });-- for (const row of rows) sendEvent(conn, row);-- if (scanned < latest) {-- // Replay stopped at its limit: everything after `scanned` is reported as a gap.-- sendGap(conn, { reason: 'replay_limit', from_seq: scanned + 1, to_seq: latest, latest_seq: latest });++ try {++ if (lastId == null) conn.lastSeq = await store.lastSeq();++ if (lastId != null) {++ const latest = await store.lastSeq();++ const oldest = await store.oldestSeq();++ if (lastId > latest) {++ sendGap(conn, { reason: 'cursor_ahead', from_seq: latest + 1, to_seq: lastId, latest_seq: latest });++ conn.lastSeq = latest;++ } else {++ let cursor = lastId;++ if (lastId < oldest - 1) {++ sendGap(conn, { reason: 'retention', from_seq: lastId + 1, to_seq: oldest - 1, latest_seq: latest });++ cursor = oldest - 1;++ }++ // Browsers: public events older than the window are not replayed (a gap says so).++ const publicFrom = viewer.kind === 'service' ? 0++ : opts.publicReplaySeconds > 0 ? await store.firstSeqSince(clock.now() - opts.publicReplaySeconds * 1000) : latest + 1;++ if (cursor < publicFrom - 1) {++ sendGap(conn, { reason: 'public_window', from_seq: cursor + 1, to_seq: publicFrom - 1, latest_seq: latest, window_seconds: opts.publicReplaySeconds });++ // Signed out, only public events are visible: nothing before the window to scan.++ if (viewer.kind === 'anonymous') cursor = publicFrom - 1;++ }++ const accept = (row) => visibleTo(viewer, row) && !(row.visibility === 'public' && row.seq < publicFrom);++ const { rows, cursor: scanned } = await store.scan(cursor, {++ patterns: wanted, limit: opts.replayMax, scanMax: opts.replayMax * 50, accept,++ });++ for (const row of rows) sendEvent(conn, row);++ if (scanned < latest) {++ // Replay stopped at its limit: everything after `scanned` is reported as a gap.++ sendGap(conn, { reason: 'replay_limit', from_seq: scanned + 1, to_seq: latest, latest_seq: latest });++ }++ conn.lastSeq = latest;}-- conn.lastSeq = latest;}++ } catch (err) {++ // The headers are sent: a failed catch-up can only end the stream (the client reconnects with its Last-Event-ID).++ log.error(`[realtime] catch-up failed: ${err.message}`);++ close(conn);++ return undefined;}---- conns.add(conn);-- res.on('close', () => close(conn));++ const pending = conn.pending;++ conn.pending = null;++ for (const row of pending) if (row.seq > conn.lastSeq) sendEvent(conn, row);return undefined;}@@@ -186,10 +195,11 @@ function createRealtime({ store, auth, config, clock = { now: () => Date.now() }if (!conns.size) return;for (const row of rows) {for (const conn of conns) {-- if (conn.closed || row.seq <= conn.lastSeq) continue;++ if (conn.closed || (!conn.pending && row.seq <= conn.lastSeq)) continue;if (!conn.patterns.some(p => topics.matches(p, row.event_type))) continue;if (!visibleTo(conn.viewer, row)) continue;-- sendEvent(conn, row);++ if (conn.pending) conn.pending.push(row);++ else sendEvent(conn, row);}}}@@@ -7,7 +7,7 @@*/const fs = require('fs');const path = require('path');--const Database = require('better-sqlite3');++const { createDb } = require('openvibe-sdk/db');const topics = require('./topics');const apps = require('./apps');const redaction = require('./redaction');@@@ -17,134 +17,29 @@ const PRIORITY_RANK = { critical: 0, important: 1, low: 2 };const RANK_PRIORITY = ['critical', 'important', 'low'];const DAY_MS = 24 * 60 * 60 * 1000;--const SCHEMA = `--CREATE TABLE IF NOT EXISTS sequences (-- name TEXT PRIMARY KEY,-- value INTEGER NOT NULL--);--CREATE TABLE IF NOT EXISTS events (-- id TEXT PRIMARY KEY,-- seq INTEGER NOT NULL UNIQUE,-- event_type TEXT NOT NULL,-- version INTEGER NOT NULL,-- source TEXT NOT NULL,-- actor TEXT NOT NULL,-- subject_type TEXT NOT NULL,-- subject_id TEXT NOT NULL,-- subject_revision INTEGER,-- trace_id TEXT NOT NULL,-- priority TEXT NOT NULL,-- visibility TEXT NOT NULL,-- occurred_at TEXT NOT NULL,-- received_at INTEGER NOT NULL,-- payload TEXT NOT NULL,-- hops INTEGER NOT NULL DEFAULT 1,-- publisher TEXT NOT NULL,-- request_id TEXT--);--CREATE INDEX IF NOT EXISTS idx_events_trace ON events(trace_id);--CREATE INDEX IF NOT EXISTS idx_events_received ON events(received_at);--CREATE INDEX IF NOT EXISTS idx_events_type_seq ON events(event_type, seq);----CREATE TABLE IF NOT EXISTS subscriptions (-- id TEXT PRIMARY KEY,-- consumer TEXT NOT NULL,-- topic_pattern TEXT NOT NULL,-- endpoint TEXT NOT NULL,-- secret TEXT NOT NULL,-- enabled INTEGER NOT NULL DEFAULT 1,-- retry_policy TEXT,-- created_at INTEGER NOT NULL,-- updated_at INTEGER NOT NULL--);--CREATE INDEX IF NOT EXISTS idx_subscriptions_consumer ON subscriptions(consumer);----CREATE TABLE IF NOT EXISTS deliveries (-- event_id TEXT NOT NULL REFERENCES events(id) ON DELETE CASCADE,-- subscription_id TEXT NOT NULL REFERENCES subscriptions(id) ON DELETE CASCADE,-- seq INTEGER NOT NULL,-- priority INTEGER NOT NULL,-- attempt INTEGER NOT NULL DEFAULT 0,-- status TEXT NOT NULL CHECK (status IN ('pending', 'delivered', 'failed', 'dead')),-- next_attempt_at INTEGER,-- last_error TEXT,-- last_status INTEGER,-- delivered_at INTEGER,-- created_at INTEGER NOT NULL,-- updated_at INTEGER NOT NULL,-- PRIMARY KEY (event_id, subscription_id)--);--CREATE INDEX IF NOT EXISTS idx_deliveries_due ON deliveries(status, next_attempt_at);--CREATE INDEX IF NOT EXISTS idx_deliveries_sub ON deliveries(subscription_id, status, seq);----CREATE TABLE IF NOT EXISTS consumer_checkpoints (-- consumer TEXT NOT NULL,-- topic_pattern TEXT NOT NULL,-- cursor INTEGER NOT NULL,-- updated_at INTEGER NOT NULL,-- PRIMARY KEY (consumer, topic_pattern)--);----CREATE TABLE IF NOT EXISTS app_revocations (-- consumer TEXT NOT NULL,-- scope TEXT NOT NULL,-- revoked_at INTEGER NOT NULL,-- PRIMARY KEY (consumer, scope)--);----CREATE TABLE IF NOT EXISTS idempotency_receipts (-- consumer TEXT NOT NULL,-- event_id TEXT NOT NULL,-- processed_at INTEGER NOT NULL,-- PRIMARY KEY (consumer, event_id)--);--`;++const MIGRATIONS = path.join(__dirname, '..', 'migrations');++const DEV_PGLITE = path.join(__dirname, '..', 'data', 'pglite');const PUBLISH_RECEIPT = 'events:publish'; // receipts' consumer column for accepted publishes/**-- * Columns added after the first release (developer apps, ADR-014). Existing databases get them-- * with ALTER TABLE; every row stored before is a first-party production row.-- * events.project_id / env / size_bytes who owns an app event, its environment, stored bytes-- * subscriptions.project_id / env an app subscription's scope (NULL project = first-party)-- * and redaction (server/redaction.js):-- * events.redacted_at / redacted_by set when the row became a tombstone (ms, the redacting event_id)-- * events.redacts 1 when the row carries a payload.redacts directive++ * The serving handle (ADR-035): DATABASE_URL through PgBouncer; in development without it, an embedded PGlite database++ * in data/pglite. Migrations run first, as the owner (DATABASE_DIRECT_URL), or on the embedded handle. The columns added++ * after the first release (developer apps, ADR-014; redaction) and the sequence row are in migrations/0001_initial.sql.*/--const ADDED_COLUMNS = {-- events: [['project_id', 'TEXT'], ['env', "TEXT NOT NULL DEFAULT 'production'"], ['size_bytes', 'INTEGER NOT NULL DEFAULT 0'],-- ['redacted_at', 'INTEGER'], ['redacted_by', 'TEXT'], ['redacts', 'INTEGER NOT NULL DEFAULT 0']],-- subscriptions: [['project_id', 'TEXT'], ['env', "TEXT NOT NULL DEFAULT 'production'"], ['previous_secret', 'TEXT'], ['previous_secret_until', 'INTEGER']],--};----function migrate(db) {-- for (const [table, cols] of Object.entries(ADDED_COLUMNS)) {-- const have = new Set(db.prepare(`PRAGMA table_info(${table})`).all().map(c => c.name));-- for (const [name, type] of cols) if (!have.has(name)) db.exec(`ALTER TABLE ${table} ADD COLUMN ${name} ${type}`);++async function openDb(config, { log = console, registry } = {}) {++ if (!config.db.url) {++ if (config.isProduction) throw new Error('DATABASE_URL is not set: production serves from PostgreSQL (OpenVibe.Host roles/data add-service.sh events)');++ log.warn(`[events] DATABASE_URL unset: embedded PGlite database in ${DEV_PGLITE} (development only, one process)`);++ fs.mkdirSync(DEV_PGLITE, { recursive: true });++ const db = createDb({ pglite: DEV_PGLITE, service: 'events', registry, log });++ await db.migrate({ dir: MIGRATIONS, log });++ return db;}-- db.exec(`CREATE INDEX IF NOT EXISTS idx_events_project ON events(project_id, env, received_at) WHERE project_id IS NOT NULL;-- CREATE INDEX IF NOT EXISTS idx_subscriptions_project ON subscriptions(project_id, env) WHERE project_id IS NOT NULL;-- CREATE INDEX IF NOT EXISTS idx_events_subject ON events(source, subject_type, subject_id);`);--}----function openDb(dbPath) {-- if (dbPath !== ':memory:') fs.mkdirSync(path.dirname(path.resolve(dbPath)), { recursive: true });-- const db = new Database(dbPath);-- db.pragma('journal_mode = WAL');-- db.pragma('synchronous = NORMAL');-- db.pragma('foreign_keys = ON');-- db.pragma('busy_timeout = 5000');-- // A redacted payload (and a pruned row) is overwritten with zeros, not left in free pages.-- db.pragma('secure_delete = ON');-- db.exec(SCHEMA);-- migrate(db);-- db.prepare("INSERT OR IGNORE INTO sequences (name, value) VALUES ('events', 0)").run();-- return db;++ if (!config.db.directUrl) throw new Error('DATABASE_DIRECT_URL is not set: migrations run with the owner role on a direct connection');++ const owner = createDb({ url: config.db.directUrl, service: 'events-migrate', max: 1, log });++ try { await owner.migrate({ dir: MIGRATIONS, log }); } finally { await owner.close(); }++ return createDb({ url: config.db.url, service: 'events', registry, log });}class StoreError extends Error {@@@ -200,7 +95,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usageconst q = {getEvent: db.prepare('SELECT * FROM events WHERE id = ?'),getReceipt: db.prepare('SELECT processed_at FROM idempotency_receipts WHERE consumer = ? AND event_id = ?'),-- addReceipt: db.prepare('INSERT OR IGNORE INTO idempotency_receipts (consumer, event_id, processed_at) VALUES (?, ?, ?)'),++ addReceipt: db.prepare('INSERT INTO idempotency_receipts (consumer, event_id, processed_at) VALUES (?, ?, ?) ON CONFLICT DO NOTHING'),nextSeq: db.prepare("UPDATE sequences SET value = value + 1 WHERE name = 'events' RETURNING value"),lastSeq: db.prepare("SELECT value FROM sequences WHERE name = 'events'"),minSeq: db.prepare('SELECT MIN(seq) AS s FROM events'),@@@ -225,15 +120,15 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usagefirstSeqSince: db.prepare('SELECT MIN(seq) AS s FROM events WHERE received_at >= ?'),enabledSubs: db.prepare('SELECT id, topic_pattern, project_id, env FROM subscriptions WHERE enabled = 1'),projectRecent: db.prepare('SELECT COUNT(*) AS n FROM events WHERE project_id = ? AND env = ? AND received_at > ?'),-- projectBytes: db.prepare('SELECT COALESCE(SUM(size_bytes), 0) AS b FROM events WHERE project_id = ? AND env = ?'),++ projectBytes: db.prepare('SELECT COALESCE(SUM(size_bytes), 0)::bigint AS b FROM events WHERE project_id = ? AND env = ?'),disableAppSubs: db.prepare('UPDATE subscriptions SET enabled = 0, updated_at = ? WHERE consumer = ? AND enabled = 1'),revoke: db.prepare(`INSERT INTO app_revocations (consumer, scope, revoked_at) VALUES (?, ?, ?)-- ON CONFLICT(consumer, scope) DO UPDATE SET revoked_at = MAX(revoked_at, excluded.revoked_at)`),++ ON CONFLICT(consumer, scope) DO UPDATE SET revoked_at = GREATEST(app_revocations.revoked_at, excluded.revoked_at)`),revokedAt: db.prepare('SELECT revoked_at FROM app_revocations WHERE consumer = ? AND scope = ?'),-- insertDelivery: db.prepare(`INSERT OR IGNORE INTO deliveries (event_id, subscription_id, seq, priority, attempt, status,-- next_attempt_at, created_at, updated_at) VALUES (?, ?, ?, ?, 0, 'pending', ?, ?, ?)`),++ insertDelivery: db.prepare(`INSERT INTO deliveries (event_id, subscription_id, seq, priority, attempt, status,++ next_attempt_at, created_at, updated_at) VALUES (?, ?, ?, ?, 0, 'pending', ?, ?, ?) ON CONFLICT DO NOTHING`),afterSeq: db.prepare('SELECT * FROM events WHERE seq > ? ORDER BY seq LIMIT ?'),-- afterSeqGlob: db.prepare('SELECT * FROM events WHERE seq > ? AND event_type GLOB ? ORDER BY seq LIMIT ?'),++ afterSeqLike: db.prepare("SELECT * FROM events WHERE seq > ? AND event_type ILIKE ? ESCAPE '\\' ORDER BY seq LIMIT ?"),};// ── Events ───────────────────────────────────────────────@@@ -247,19 +142,19 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage* carry project_id and env, and the project's rate and retained-bytes quotas are checked here,* inside the transaction, against what is already stored.*/-- const insertBatch = db.transaction((items, { publisher, requestId, project = null }) => {++ const insertBatch = async (items, { publisher, requestId, project = null }) => await db.tx(async () => {const now = clock.now();const results = [];const inserted = [];-- const subs = q.enabledSubs.all();++ const subs = await q.enabledSubs.all();let recent = null;let bytes = null;if (project) {-- recent = q.projectRecent.get(project.projectId, project.env, now - 60 * 1000).n;-- bytes = q.projectBytes.get(project.projectId, project.env).b;++ recent = (await q.projectRecent.get(project.projectId, project.env, now - 60 * 1000)).n;++ bytes = (await q.projectBytes.get(project.projectId, project.env)).b;}for (const env of items) {-- const existing = q.getEvent.get(env.event_id);++ const existing = await q.getEvent.get(env.event_id);if (existing) {if (existing.source !== env.source || existing.event_type !== env.event_type) {throw new StoreError(409, 'events.id_conflict', `event ${env.event_id} already exists with different content`);@@@ -267,17 +162,17 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usageresults.push({ event_id: env.event_id, seq: existing.seq, duplicate: true });continue;}-- if (q.getReceipt.get(PUBLISH_RECEIPT, env.event_id)) {++ if (await q.getReceipt.get(PUBLISH_RECEIPT, env.event_id)) {results.push({ event_id: env.event_id, seq: null, duplicate: true, pruned: true });continue;}// Loop guard: depth of the cross-service chain in this trace, and how often this exact// (source, type, subject) already happened in the trace from other publish calls.const chainRow = project-- ? q.chainHopsApp.get(env.trace_id, env.source, project.projectId, project.env)-- : q.chainHops.get(env.trace_id, env.source);++ ? await q.chainHopsApp.get(env.trace_id, env.source, project.projectId, project.env)++ : await q.chainHops.get(env.trace_id, env.source);const chain = (chainRow.h || 0) + 1;-- const repeats = q.selfRepeats.get(env.trace_id, env.source, env.event_type, env.subject.type, env.subject.id, requestId || '').n + 1;++ const repeats = (await q.selfRepeats.get(env.trace_id, env.source, env.event_type, env.subject.type, env.subject.id, requestId || '')).n + 1;const hops = Math.max(chain, repeats);if (hops > maxHops) {throw new StoreError(409, 'events.loop_detected',@@@ -299,7 +194,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage{ quota: 'retained_bytes', limit: project.retainedBytes, used: bytes - sizeBytes });}}-- const seq = q.nextSeq.get().value;++ const seq = (await q.nextSeq.get()).value;const row = {id: env.event_id,seq,@@@ -327,18 +222,18 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usageconst directive = redaction.parseDirective(env.payload);if (directive && directive.error) throw new StoreError(422, 'events.invalid_redaction', directive.error, { event_id: env.event_id });if (directive) row.redacts = 1;-- q.insertEvent.run(row);-- q.addReceipt.run(PUBLISH_RECEIPT, env.event_id, now);++ await q.insertEvent.run(row);++ await q.addReceipt.run(PUBLISH_RECEIPT, env.event_id, now);for (const s of subs) {if (apps.deliverable(s, row)) {-- q.insertDelivery.run(env.event_id, s.id, seq, PRIORITY_RANK[env.priority], now, now, now);++ await q.insertDelivery.run(env.event_id, s.id, seq, PRIORITY_RANK[env.priority], now, now, now);}}-- revocationHook(row, now);++ await revocationHook(row, now);results.push({ event_id: env.event_id, seq, duplicate: false });inserted.push(row);if (directive) {-- const redacted = applyRedaction(row, directive, now);++ const redacted = await applyRedaction(row, directive, now);// Rows stored earlier in this batch are fanned out after the commit from memory: they// must go out as the tombstones they now are.for (const t of redacted) {@@@ -349,7 +244,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage}// An app publish counts toward its project's usage in the same transaction as the rows.if (project && inserted.length) {-- usage.record({ projectId: project.projectId, env: project.env, capability: 'events.app.publish', unit: 'events', quantity: inserted.length, at: now });++ await usage.record({ projectId: project.projectId, env: project.env, capability: 'events.app.publish', unit: 'events', quantity: inserted.length, at: now });}return { results, inserted };});@@@ -359,10 +254,10 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage* stored row of the redacting event). Throws StoreError 403 when an event id names an event of* another owner. Returns [{ id, seq, columns }] for the rows it rewrote.*/-- function applyRedaction(by, directive, now) {++ async function applyRedaction(by, directive, now) {const targets = new Map();for (const id of directive.eventIds) {-- const r = q.getEvent.get(id);++ const r = await q.getEvent.get(id);if (!r) continue; // never stored, or pruned alreadyif (!redaction.sameOwner(by, r)) {throw new StoreError(403, 'events.redaction_not_allowed', `event ${id} belongs to ${r.source}; ${by.source} may redact only its own events`,@@@ -372,7 +267,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage}if (directive.subjectType) {for (const subjectId of directive.subjectIds) {-- for (const r of q.redactBySubject.all(by.source, directive.subjectType, subjectId)) {++ for (const r of await q.redactBySubject.all(by.source, directive.subjectType, subjectId)) {if (redaction.sameOwner(by, r)) targets.set(r.id, r);}}@@@ -381,7 +276,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usagefor (const r of targets.values()) {if (r.id === by.id) continue;const columns = redaction.tombstone(r, { by: by.id, at: now });-- if (q.applyTombstone.run({ id: r.id, ...columns }).changes) out.push({ id: r.id, seq: r.seq, columns });++ if ((await q.applyTombstone.run({ id: r.id, ...columns })).changes) out.push({ id: r.id, seq: r.seq, columns });}return out;}@@@ -391,10 +286,10 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage* `directive` (the payload.redacts object) would have redacted. The tombstones name `by` as* redacted_by. Same owner rule as a publish. Returns the rewritten [{ id, seq }].*/-- const redact = db.transaction((source, directive, { by = 'operator', now = clock.now() } = {}) => {++ const redact = async (source, directive, { by = 'operator', now = clock.now() } = {}) => await db.tx(async () => {const parsed = redaction.parseDirective({ redacts: directive });if (!parsed || parsed.error) throw new StoreError(422, 'events.invalid_redaction', parsed ? parsed.error : 'nothing to redact');-- return applyRedaction({ id: by, source, project_id: null, env: 'production' }, parsed, now).map(({ id, seq }) => ({ id, seq }));++ return (await applyRedaction({ id: by, source, project_id: null, env: 'production' }, parsed, now)).map(({ id, seq }) => ({ id, seq }));});/**@@@ -403,7 +298,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage* app's subscriptions stop at once, in the same transaction that stores the event, so a* revocation reaches deliveries without waiting for anything else.*/-- function revocationHook(row, now) {++ async function revocationHook(row, now) {if (row.source !== 'network' || row.project_id || row.subject_type !== 'app') return;let payload = {};try { payload = JSON.parse(row.payload) || {}; } catch { payload = {}; }@@@ -414,34 +309,34 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usageif (!scope || !/^app_[0-9A-HJKMNP-TV-Z]{26}$/.test(row.subject_id)) return;const consumer = `app:${row.subject_id}`;const at = Date.parse(row.occurred_at);-- q.revoke.run(consumer, scope, Number.isFinite(at) ? at : now);-- q.disableAppSubs.run(now, consumer);++ await q.revoke.run(consumer, scope, Number.isFinite(at) ? at : now);++ await q.disableAppSubs.run(now, consumer);}/** When Network revoked `consumer` (app:app_…) for `scope` ('app' | 'events.app.subscribe'), in ms; or null. */-- function revokedAt(consumer, scope) {-- const r = q.revokedAt.get(consumer, scope);++ async function revokedAt(consumer, scope) {++ const r = await q.revokedAt.get(consumer, scope);return r ? r.revoked_at : null;}-- function getEvent(id) {-- return q.getEvent.get(id) || null;++ async function getEvent(id) {++ return await q.getEvent.get(id) || null;}-- function lastSeq() {-- return q.lastSeq.get().value;++ async function lastSeq() {++ return (await q.lastSeq.get()).value;}/** First seq received at or after `ms`; the next seq to be handed out when there is none. */-- function firstSeqSince(ms) {-- const m = q.firstSeqSince.get(ms).s;-- return m == null ? lastSeq() + 1 : m;++ async function firstSeqSince(ms) {++ const m = (await q.firstSeqSince.get(ms)).s;++ return m == null ? await lastSeq() + 1 : m;}/** Oldest seq still stored; when the table is empty, the next seq to be handed out. */-- function oldestSeq() {-- const m = q.minSeq.get().s;-- return m == null ? lastSeq() + 1 : m;++ async function oldestSeq() {++ const m = (await q.minSeq.get()).s;++ return m == null ? await lastSeq() + 1 : m;}/**@@@ -449,13 +344,13 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage* Scans at most `scanMax` rows; returns { rows, cursor } where cursor is the last seq examined* (so a pull consumer's cursor moves past rows that did not match).*/-- function scan(afterSeq, { patterns = ['*'], limit = 100, scanMax = 5000, accept = () => true } = {}) {++ async function scan(afterSeq, { patterns = ['*'], limit = 100, scanMax = 5000, accept = () => true } = {}) {const rows = [];let cursor = afterSeq;let scanned = 0;-- const single = patterns.length === 1 && patterns[0] !== '*' ? topics.toGlob(patterns[0]) : null;++ const single = patterns.length === 1 && patterns[0] !== '*' ? topics.toLike(patterns[0]) : null;for (;;) {-- const page = single ? q.afterSeqGlob.all(cursor, single, 500) : q.afterSeq.all(cursor, 500);++ const page = single ? await q.afterSeqLike.all(cursor, single, 500) : await q.afterSeq.all(cursor, 500);for (const row of page) {scanned++;cursor = row.seq;@@@ -464,9 +359,9 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage}if (rows.length >= limit || scanned >= scanMax) break;if (page.length < 500) {-- // Reached the end. With the GLOB narrowing, rows that were never fetched cannot match,++ // Reached the end. With the ILIKE narrowing, rows that were never fetched cannot match,// so the cursor may move up to the latest seq handed out.-- if (single) cursor = Math.max(cursor, lastSeq());++ if (single) cursor = Math.max(cursor, await lastSeq());break;}}@@@ -475,37 +370,37 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage// ── Subscriptions ────────────────────────────────────────-- function createSubscription({ id, consumer, topicPattern, endpoint, secret, retryPolicy, projectId = null, env = 'production' }) {++ async function createSubscription({ id, consumer, topicPattern, endpoint, secret, retryPolicy, projectId = null, env = 'production' }) {const now = clock.now();-- db.prepare(`INSERT INTO subscriptions (id, consumer, topic_pattern, endpoint, secret, enabled, retry_policy, created_at, updated_at, project_id, env)++ await db.prepare(`INSERT INTO subscriptions (id, consumer, topic_pattern, endpoint, secret, enabled, retry_policy, created_at, updated_at, project_id, env)VALUES (?, ?, ?, ?, ?, 1, ?, ?, ?, ?, ?)`).run(id, consumer, topicPattern, endpoint, secret, retryPolicy ? JSON.stringify(retryPolicy) : null, now, now, projectId, env);-- return getSubscription(id);++ return await getSubscription(id);}-- function countProjectSubscriptions(projectId, env) {-- return db.prepare('SELECT COUNT(*) AS n FROM subscriptions WHERE project_id = ? AND env = ?').get(projectId, env).n;++ async function countProjectSubscriptions(projectId, env) {++ return (await db.prepare('SELECT COUNT(*) AS n FROM subscriptions WHERE project_id = ? AND env = ?').get(projectId, env)).n;}-- function getSubscription(id) {-- return db.prepare('SELECT * FROM subscriptions WHERE id = ?').get(id) || null;++ async function getSubscription(id) {++ return await db.prepare('SELECT * FROM subscriptions WHERE id = ?').get(id) || null;}-- function listSubscriptions(consumer) {++ async function listSubscriptions(consumer) {return consumer-- ? db.prepare('SELECT * FROM subscriptions WHERE consumer = ? ORDER BY created_at, id').all(consumer)-- : db.prepare('SELECT * FROM subscriptions ORDER BY created_at, id').all();++ ? await db.prepare('SELECT * FROM subscriptions WHERE consumer = ? ORDER BY created_at, id').all(consumer)++ : await db.prepare('SELECT * FROM subscriptions ORDER BY created_at, id').all();}-- function countSubscriptions(consumer) {-- return db.prepare('SELECT COUNT(*) AS n FROM subscriptions WHERE consumer = ?').get(consumer).n;++ async function countSubscriptions(consumer) {++ return (await db.prepare('SELECT COUNT(*) AS n FROM subscriptions WHERE consumer = ?').get(consumer)).n;}/** Rotate a subscription's secret: the old one keeps signing (next to the new) until overlapMs from now. */-- function rotateSubscriptionSecret(id, secret, overlapMs) {++ async function rotateSubscriptionSecret(id, secret, overlapMs) {const now = clock.now();-- db.prepare('UPDATE subscriptions SET previous_secret = secret, previous_secret_until = ?, secret = ?, updated_at = ? WHERE id = ?')++ await db.prepare('UPDATE subscriptions SET previous_secret = secret, previous_secret_until = ?, secret = ?, updated_at = ? WHERE id = ?').run(now + Math.max(0, overlapMs), secret, now, id);-- return getSubscription(id);++ return await getSubscription(id);}-- function setSubscriptionEnabled(id, enabled) {-- db.prepare('UPDATE subscriptions SET enabled = ?, updated_at = ? WHERE id = ?').run(enabled ? 1 : 0, clock.now(), id);-- return getSubscription(id);++ async function setSubscriptionEnabled(id, enabled) {++ await db.prepare('UPDATE subscriptions SET enabled = ?, updated_at = ? WHERE id = ?').run(enabled ? 1 : 0, clock.now(), id);++ return await getSubscription(id);}// ── Deliveries ───────────────────────────────────────────@@@ -514,9 +409,9 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage* Next due delivery per subscription (at most one each, so per-subscription concurrency stays 1),* ordered by priority class, then seq. `busy` are subscription ids with a delivery in flight.*/-- function dueDeliveries(now, limit, busy = [], { firstPartyOnly = false } = {}) {++ async function dueDeliveries(now, limit, busy = [], { firstPartyOnly = false } = {}) {// app_project_id: set for developer-app subscriptions (the worker caps their share of slots).-- const rows = db.prepare(`++ const rows = await db.prepare(`SELECT * FROM (SELECT d.*, s.project_id AS app_project_id, ROW_NUMBER() OVER (PARTITION BY d.subscription_id ORDER BY d.priority, d.seq) AS rnFROM deliveries d JOIN subscriptions s ON s.id = d.subscription_id@@@ -532,20 +427,20 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage* subscription: the attempt counts toward the project's events.app.subscribe usage in the same* transaction.*/-- const recordAttempt = db.transaction((eventId, subscriptionId, outcome, app = null) => {++ const recordAttempt = async (eventId, subscriptionId, outcome, app = null) => await db.tx(async () => {const now = clock.now();if (app) {-- usage.record({++ await usage.record({projectId: app.projectId, env: app.env, capability: 'events.app.subscribe', unit: 'deliveries', quantity: 1, at: now,error: outcome.ok ? null : { code: deliveryCode(outcome), status: outcome.status || undefined, traceId: app.traceId, ref: eventId },});}if (outcome.ok) {-- db.prepare(`UPDATE deliveries SET status = 'delivered', attempt = ?, delivered_at = ?, last_status = ?, last_error = NULL,++ await db.prepare(`UPDATE deliveries SET status = 'delivered', attempt = ?, delivered_at = ?, last_status = ?, last_error = NULL,next_attempt_at = NULL, updated_at = ? WHERE event_id = ? AND subscription_id = ?`).run(outcome.attempt, now, outcome.status ?? null, now, eventId, subscriptionId);} else {-- db.prepare(`UPDATE deliveries SET status = ?, attempt = ?, last_status = ?, last_error = ?, next_attempt_at = ?, updated_at = ?++ await db.prepare(`UPDATE deliveries SET status = ?, attempt = ?, last_status = ?, last_error = ?, next_attempt_at = ?, updated_at = ?WHERE event_id = ? AND subscription_id = ?`).run(outcome.dead ? 'dead' : 'failed', outcome.attempt, outcome.status ?? null, String(outcome.error || '').slice(0, 500),outcome.dead ? null : outcome.nextAttemptAt, now, eventId, subscriptionId);@@@ -553,37 +448,37 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage});/** Send the closed hours' usage rollups (./usage.js); returns the stored rows. */-- function flushUsage(opts) {-- return usage.flush(insertBatch, opts);++ async function flushUsage(opts) {++ return await usage.flush(insertBatch, opts);}-- function getDelivery(eventId, subscriptionId) {-- return db.prepare('SELECT * FROM deliveries WHERE event_id = ? AND subscription_id = ?').get(eventId, subscriptionId) || null;++ async function getDelivery(eventId, subscriptionId) {++ return await db.prepare('SELECT * FROM deliveries WHERE event_id = ? AND subscription_id = ?').get(eventId, subscriptionId) || null;}-- function listDeliveries({ status, subscriptionId, limit = 100, afterSeq = 0 } = {}) {++ async function listDeliveries({ status, subscriptionId, limit = 100, afterSeq = 0 } = {}) {const where = ['seq > ?'];const args = [afterSeq];if (status) { where.push('status = ?'); args.push(status); }if (subscriptionId) { where.push('subscription_id = ?'); args.push(subscriptionId); }args.push(limit);-- return db.prepare(`SELECT * FROM deliveries WHERE ${where.join(' AND ')} ORDER BY seq, subscription_id LIMIT ?`).all(...args);++ return await db.prepare(`SELECT * FROM deliveries WHERE ${where.join(' AND ')} ORDER BY seq, subscription_id LIMIT ?`).all(...args);}/*** Queue (again) the given events for one subscription: reset to pending, attempt 0, due now.* Only retained events that match the subscription's topic are queued. Returns the count.*/-- const requeue = db.transaction((sub, { fromSeq, eventIds, max = 10000 }) => {++ const requeue = async (sub, { fromSeq, eventIds, max = 10000 }) => await db.tx(async () => {const now = clock.now();let rows;const cols = 'id, seq, priority, event_type, visibility, project_id, env';if (Array.isArray(eventIds)) {const get = db.prepare(`SELECT ${cols} FROM events WHERE id = ?`);-- rows = eventIds.map(id => get.get(id)).filter(Boolean);++ rows = (await Promise.all(eventIds.map(async id => await get.get(id)))).filter(Boolean);} else {-- rows = db.prepare(`SELECT ${cols} FROM events WHERE seq >= ? AND event_type GLOB ? ORDER BY seq LIMIT ?`)-- .all(fromSeq, topics.toGlob(sub.topic_pattern), max * 2);++ rows = await db.prepare(`SELECT ${cols} FROM events WHERE seq >= ? AND event_type ILIKE ? ESCAPE '\\' ORDER BY seq LIMIT ?`)++ .all(fromSeq, topics.toLike(sub.topic_pattern), max * 2);}// Replay never widens a subscription's scope (app scope, sandbox separation).rows = rows.filter(r => apps.deliverable(sub, r)).slice(0, max);@@@ -591,27 +486,27 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usageVALUES (?, ?, ?, ?, 0, 'pending', ?, ?, ?)ON CONFLICT(event_id, subscription_id) DO UPDATE SET status = 'pending', attempt = 0, next_attempt_at = excluded.next_attempt_at,last_error = NULL, last_status = NULL, delivered_at = NULL, updated_at = excluded.updated_at`);-- for (const r of rows) up.run(r.id, sub.id, r.seq, PRIORITY_RANK[r.priority], now, now, now);++ for (const r of rows) await up.run(r.id, sub.id, r.seq, PRIORITY_RANK[r.priority], now, now, now);return rows.length;});-- function deliveryCounts() {++ async function deliveryCounts() {const out = { pending: 0, delivered: 0, failed: 0, dead: 0 };-- for (const r of db.prepare('SELECT status, COUNT(*) AS n FROM deliveries GROUP BY status').all()) out[r.status] = r.n;++ for (const r of await db.prepare('SELECT status, COUNT(*) AS n FROM deliveries GROUP BY status').all()) out[r.status] = r.n;return out;}// ── Checkpoints ──────────────────────────────────────────-- function getCheckpoint(consumer, topicPattern) {-- const r = db.prepare('SELECT cursor, updated_at FROM consumer_checkpoints WHERE consumer = ? AND topic_pattern = ?').get(consumer, topicPattern);++ async function getCheckpoint(consumer, topicPattern) {++ const r = await db.prepare('SELECT cursor, updated_at FROM consumer_checkpoints WHERE consumer = ? AND topic_pattern = ?').get(consumer, topicPattern);return r ? { cursor: r.cursor, updated_at: new Date(r.updated_at).toISOString() } : null;}-- function setCheckpoint(consumer, topicPattern, cursor) {-- db.prepare(`INSERT INTO consumer_checkpoints (consumer, topic_pattern, cursor, updated_at) VALUES (?, ?, ?, ?)++ async function setCheckpoint(consumer, topicPattern, cursor) {++ await db.prepare(`INSERT INTO consumer_checkpoints (consumer, topic_pattern, cursor, updated_at) VALUES (?, ?, ?, ?)ON CONFLICT(consumer, topic_pattern) DO UPDATE SET cursor = excluded.cursor, updated_at = excluded.updated_at`).run(consumer, topicPattern, cursor, clock.now());-- return getCheckpoint(consumer, topicPattern);++ return await getCheckpoint(consumer, topicPattern);}// ── Retention ────────────────────────────────────────────@@@ -620,18 +515,18 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage* Drop events (and their deliveries) older than retentionDays (sandbox events older than* sandboxRetentionDays, when shorter), and old publish receipts.*/-- function prune({ retentionDays = 30, receiptRetentionDays = 90, sandboxRetentionDays = retentionDays, now = clock.now() } = {}) {-- let events = db.prepare('DELETE FROM events WHERE received_at < ?').run(now - retentionDays * DAY_MS).changes;++ async function prune({ retentionDays = 30, receiptRetentionDays = 90, sandboxRetentionDays = retentionDays, now = clock.now() } = {}) {++ let events = (await db.prepare('DELETE FROM events WHERE received_at < ?').run(now - retentionDays * DAY_MS)).changes;if (sandboxRetentionDays < retentionDays) {-- events += db.prepare("DELETE FROM events WHERE env = 'sandbox' AND received_at < ?").run(now - sandboxRetentionDays * DAY_MS).changes;++ events += (await db.prepare("DELETE FROM events WHERE env = 'sandbox' AND received_at < ?").run(now - sandboxRetentionDays * DAY_MS)).changes;}-- const receipts = db.prepare('DELETE FROM idempotency_receipts WHERE processed_at < ?')-- .run(now - Math.max(receiptRetentionDays, retentionDays) * DAY_MS).changes;++ const receipts = (await db.prepare('DELETE FROM idempotency_receipts WHERE processed_at < ?')++ .run(now - Math.max(receiptRetentionDays, retentionDays) * DAY_MS)).changes;return { events, receipts };}-- function ping() {-- return db.prepare('SELECT 1 AS ok').get().ok === 1;++ async function ping() {++ return (await db.prepare('SELECT 1 AS ok').get()).ok === 1;}return {@@@ -642,4 +537,4 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage};}--module.exports = { openDb, createStore, rowToEnvelope, subscriptionView, StoreError, PRIORITY_RANK, RANK_PRIORITY };++module.exports = { openDb, MIGRATIONS, createStore, rowToEnvelope, subscriptionView, StoreError, PRIORITY_RANK, RANK_PRIORITY };@@@ -34,13 +34,13 @@ function matches(pattern, eventType) {}/**-- * A SQLite GLOB that selects a superset of what the pattern matches (GLOB `*` also spans dots),-- * so the SQL narrows the scan and matches() makes the exact decision. Valid patterns only contain-- * [a-z0-9_.*], none of which GLOB treats specially except `*`, so the pattern is its own GLOB.++ * A ILIKE pattern (PostgreSQL, ESCAPE '\\') that selects a superset of what the pattern matches (`%` also spans dots),++ * so the SQL narrows the scan and matches() makes the exact decision. Valid patterns only contain [a-z0-9_.*]: `_` is++ * escaped (it is a ILIKE wildcard) and `*` becomes `%`. ILIKE compares exactly, as GLOB did.*/--function toGlob(pattern) {++function toLike(pattern) {if (!isValidPattern(pattern)) throw new TypeError(`invalid topic pattern "${pattern}"`);-- return pattern;++ return pattern.replace(/_/g, '\\_').replace(/\*/g, '%');}--module.exports = { isValidPattern, compile, matches, toGlob };++module.exports = { isValidPattern, compile, matches, toLike };@@@ -36,24 +36,7 @@ const TRACE_RE = /^[0-9a-f]{32}$/;const REF_RE = /^evt_[0-9A-HJKMNP-TV-Z]{26}$/;const PROJECT_RE = /^prj_[0-9A-HJKMNP-TV-Z]{26}$/;--const SCHEMA = `--CREATE TABLE IF NOT EXISTS app_usage (-- project_id TEXT NOT NULL,-- env TEXT NOT NULL,-- capability TEXT NOT NULL,-- unit TEXT NOT NULL,-- window_start INTEGER NOT NULL,-- quantity INTEGER NOT NULL DEFAULT 0,-- errors INTEGER NOT NULL DEFAULT 0,-- error_codes TEXT NOT NULL DEFAULT '{}',-- samples TEXT NOT NULL DEFAULT '[]',-- revision INTEGER NOT NULL DEFAULT 1,-- event_id TEXT,-- emitted_at INTEGER,-- PRIMARY KEY (project_id, env, capability, unit, window_start)--);--CREATE INDEX IF NOT EXISTS idx_app_usage_open ON app_usage(emitted_at, window_start);--`;++// The app_usage table is in migrations/0001_initial.sql.const hourOf = (ms) => Math.floor(ms / HOUR_MS) * HOUR_MS;const parse = (s, d) => { try { return JSON.parse(s); } catch { return d; } };@@@ -71,13 +54,12 @@ function deliveryCode(outcome) {* @param {boolean} [o.enabled=true] false (EVENTS_USAGE=off): nothing is counted*/function createUsage(db, { clock = { now: () => Date.now() }, enabled = true } = {}) {-- db.exec(SCHEMA);const q = {add: db.prepare(`INSERT INTO app_usage (project_id, env, capability, unit, window_start, quantity, errors)VALUES (@project_id, @env, @capability, @unit, @window_start, @quantity, @errors)ON CONFLICT(project_id, env, capability, unit, window_start) DO UPDATE SET-- quantity = quantity + excluded.quantity, errors = errors + excluded.errors,-- revision = revision + (CASE WHEN emitted_at IS NULL THEN 0 ELSE 1 END), emitted_at = NULL`),++ quantity = app_usage.quantity + excluded.quantity, errors = app_usage.errors + excluded.errors,++ revision = app_usage.revision + (CASE WHEN app_usage.emitted_at IS NULL THEN 0 ELSE 1 END), emitted_at = NULL`),get: db.prepare('SELECT error_codes, samples FROM app_usage WHERE project_id = ? AND env = ? AND capability = ? AND unit = ? AND window_start = ?'),setErrors: db.prepare(`UPDATE app_usage SET error_codes = ?, samples = ?WHERE project_id = ? AND env = ? AND capability = ? AND unit = ? AND window_start = ?`),@@@ -91,13 +73,13 @@ function createUsage(db, { clock = { now: () => Date.now() }, enabled = true } =* Count usage (and at most one failure) for a project, inside the caller's transaction when there* is one. `error` = { code, status?, traceId?, ref? } counts one error with a sample.*/-- const record = db.transaction(({ projectId, env, capability, unit, quantity = 0, error = null, at = clock.now() }) => {++ const record = async ({ projectId, env, capability, unit, quantity = 0, error = null, at = clock.now() }) => await db.tx(async () => {if (!enabled || !PROJECT_RE.test(String(projectId || '')) || (env !== 'sandbox' && env !== 'production')) return;const key = { project_id: projectId, env, capability, unit, window_start: hourOf(at) };-- q.add.run({ ...key, quantity: Math.max(0, Math.trunc(Number(quantity) || 0)), errors: error ? 1 : 0 });++ await q.add.run({ ...key, quantity: Math.max(0, Math.trunc(Number(quantity) || 0)), errors: error ? 1 : 0 });if (!error) return;const code = CODE_RE.test(String(error.code || '')) ? String(error.code) : 'events.error';-- const row = q.get.get(projectId, env, capability, unit, key.window_start);++ const row = await q.get.get(projectId, env, capability, unit, key.window_start);const codes = parse(row.error_codes, {});if (codes[code] || Object.keys(codes).length < MAX_CODES) codes[code] = (codes[code] || 0) + 1;const sample = { at: new Date(at).toISOString(), code };@@@ -105,7 +87,7 @@ function createUsage(db, { clock = { now: () => Date.now() }, enabled = true } =if (TRACE_RE.test(String(error.traceId || ''))) sample.trace_id = error.traceId;if (REF_RE.test(String(error.ref || ''))) sample.ref = error.ref;const samples = [sample, ...parse(row.samples, [])].slice(0, MAX_SAMPLES);-- q.setErrors.run(JSON.stringify(codes), JSON.stringify(samples), projectId, env, capability, unit, key.window_start);++ await q.setErrors.run(JSON.stringify(codes), JSON.stringify(samples), projectId, env, capability, unit, key.window_start);});function payloadOf(row) {@@@ -126,10 +108,10 @@ function createUsage(db, { clock = { now: () => Date.now() }, enabled = true } =* Store every closed hour's rollups as events.usage.recorded (at most `limit`). `insertBatch` is* the store's; the returned rows are what it stored, for the realtime fan-out and the worker kick.*/-- function flush(insertBatch, { now = clock.now(), limit = 500 } = {}) {++ async function flush(insertBatch, { now = clock.now(), limit = 500 } = {}) {const stored = [];let invalid = 0;-- const rows = q.due.all(hourOf(now - GRACE_MS) - HOUR_MS, limit);++ const rows = await q.due.all(hourOf(now - GRACE_MS) - HOUR_MS, limit);for (const row of rows) {const payload = payloadOf(row);const envelope = {@@@ -145,19 +127,19 @@ function createUsage(db, { clock = { now: () => Date.now() }, enabled = true } =invalid++;continue;}-- db.transaction(() => {-- const out = insertBatch([envelope], { publisher: 'svc:events', requestId: null });-- q.sent.run(now, envelope.event_id, row.project_id, row.env, row.capability, row.unit, row.window_start);++ await db.tx(async () => {++ const out = await insertBatch([envelope], { publisher: 'svc:events', requestId: null });++ await q.sent.run(now, envelope.event_id, row.project_id, row.env, row.capability, row.unit, row.window_start);stored.push(...out.inserted);-- })();++ });}-- q.prune.run(now - KEEP_SENT_MS);++ await q.prune.run(now - KEEP_SENT_MS);return { stored, invalid };}return {record, flush, payloadOf, deliveryCode,-- pending: () => q.pending.get().n,++ pending: async () => (await q.pending.get()).n,};}@@@ -61,8 +61,8 @@ function createWorker({ store, config, clock = { now: () => Date.now() }, fetchI}async function send(delivery) {-- const sub = store.getSubscription(delivery.subscription_id);-- const row = store.getEvent(delivery.event_id);++ const sub = await store.getSubscription(delivery.subscription_id);++ const row = await store.getEvent(delivery.event_id);const attempt = delivery.attempt + 1;if (!sub || !row) return; // pruned or removed while queued; cascade deletes the deliveryconst { maxAttempts, backoffMs } = policy(sub);@@@ -112,7 +112,7 @@ function createWorker({ store, config, clock = { now: () => Date.now() }, fetchI}// A developer app's subscription: the attempt counts toward its project's usage (./usage.js).const app = sub.project_id ? { projectId: sub.project_id, env: sub.env || 'production', traceId: row.trace_id } : null;-- store.recordAttempt(delivery.event_id, delivery.subscription_id, outcome, app);++ await store.recordAttempt(delivery.event_id, delivery.subscription_id, outcome, app);if (observe) {try {observe.attempt(outcome.ok ? 'delivered' : outcome.dead ? 'dead' : 'retry');@@@ -122,8 +122,21 @@ function createWorker({ store, config, clock = { now: () => Date.now() }, fetchI}}-- /** Start every due delivery there is room for. Returns how many were started. */++ // One pass at a time: the due query and start() must not interleave, or a second pass would read the rows the++ // first is starting and send them twice. A call during a pass gets one more pass right after it.++ let passing = null;++ let queued = null;function dispatch() {++ if (!passing) {++ passing = dispatchPass().finally(() => { passing = null; });++ return passing;++ }++ if (!queued) queued = passing.catch(() => {}).then(() => { queued = null; return dispatch(); });++ return queued;++ }++++ /** Start every due delivery there is room for. Returns how many were started. */++ async function dispatchPass() {lastTickAt = Date.now();const room = opts.maxInflight - inflight.size;if (room <= 0) return 0;@@@ -138,10 +151,10 @@ function createWorker({ store, config, clock = { now: () => Date.now() }, fetchIstarted++;}};-- startAll(store.dueDeliveries(clock.now(), room, [...busy], { firstPartyOnly: appInflight >= opts.maxAppInflight }));++ startAll(await store.dueDeliveries(clock.now(), room, [...busy], { firstPartyOnly: appInflight >= opts.maxAppInflight }));// App deliveries (which can outrank first-party ones by priority) filled the app share: the// rest of the room goes to first-party subscriptions.-- if (skippedApp && started < room) startAll(store.dueDeliveries(clock.now(), room - started, [...busy], { firstPartyOnly: true }));++ if (skippedApp && started < room) startAll(await store.dueDeliveries(clock.now(), room - started, [...busy], { firstPartyOnly: true }));return started;}@@@ -163,9 +176,9 @@ function createWorker({ store, config, clock = { now: () => Date.now() }, fetchIif (!running) return;if (timer && ms > 0) return; // a tick is already comingclearTimeout(timer);-- timer = setTimeout(() => {++ timer = setTimeout(async () => {timer = null;-- try { dispatch(); } catch (err) { log.error(`[worker] tick failed: ${err.stack || err}`); }++ try { await dispatch(); } catch (err) { log.error(`[worker] tick failed: ${err.stack || err}`); }schedule(opts.intervalMs);}, ms);timer.unref?.();@@@ -174,7 +187,7 @@ function createWorker({ store, config, clock = { now: () => Date.now() }, fetchI/** Deliver everything due now (and whatever becomes due as a result), then resolve. Tests. */async function drain() {for (;;) {-- const started = dispatch();++ const started = await dispatch();if (!started && !inflight.size) return;if (inflight.size) await Promise.race([...inflight]);}@@@ -27,8 +27,8 @@ const appEvent = () => envelope('x', {source: apps.appSource(appId), event_type: `app.${apps.projectKey(projectId)}.order.created`,actor: { type: 'app', id: appId }, subject: { type: 'order', id: '42' }, visibility: 'internal',});--const get = (token, p) => request(h.base, 'GET', p, { token });--const publish = (token, body) => request(h.base, 'POST', '/api/v1/events', { token, body });++const get = async (token, p) => await request(h.base, 'GET', p, { token });++const publish = async (token, body) => await request(h.base, 'POST', '/api/v1/events', { token, body });t('boot (EVENTS_LIMITS_MINUTE=3; the app event quota off, so only the per-actor limit answers)', async () => {h = await boot({ clock, env: { EVENTS_LIMITS_MINUTE: '3', EVENTS_LIMITS_HOUR: '100', EVENTS_APP_PUBLISH_PER_MINUTE: '0' } });@@@ -65,10 +65,10 @@ t('publish: an app gets 60 requests a minute; the 61st is refused before anythinconst r = await publish(app, appEvent());assert.strictEqual(r.status, 201, r.text);}-- const before = h.store.lastSeq();++ const before = await h.store.lastSeq();const r = await publish(app, appEvent());assert.deepStrictEqual([r.status, r.body.code, r.headers.get('retry-after')], [429, 'rate_limited', '60']);-- assert.strictEqual(h.store.lastSeq(), before, 'nothing stored');++ assert.strictEqual(await h.store.lastSeq(), before, 'nothing stored');assert.strictEqual((await publish(live, envelope('live'))).status, 201, 'a service still publishes');});@@@ -122,7 +122,7 @@ t('worker: app deliveries never take every in-flight slot', async () => {}const r = await request(h.base, 'POST', '/api/v1/events', { token: serviceToken('live', ['events.event.publish']), body: envelope('live') });assert.strictEqual(r.status, 201);-- h.worker.dispatch();++ await h.worker.dispatch();for (let i = 0; i < 40 && !firstParty.calls.length; i++) await sleep(25);assert.strictEqual(firstParty.calls.length, 1, 'the first-party delivery got a slot');assert.ok(pending.length >= 1, 'app deliveries still run');@@@ -71,8 +71,8 @@ function appEvent(appId, projectId, name = 'order.created', over = {}) {actor: { type: 'app', id: appId }, subject: { type: 'order', id: '42' }, visibility: 'internal', ...over,});}--const publish = (token, body) => request(h.base, 'POST', '/api/v1/events', { token, body });--const pull = (token, topic, after = 0) => request(h.base, 'GET', `/api/v1/events?topic=${encodeURIComponent(topic)}&after_seq=${after}`, { token });++const publish = async (token, body) => await request(h.base, 'POST', '/api/v1/events', { token, body });++const pull = async (token, topic, after = 0) => await request(h.base, 'GET', `/api/v1/events?topic=${encodeURIComponent(topic)}&after_seq=${after}`, { token });t('boot', async () => {await new Promise(r => receiver.listen(0, '127.0.0.1', r));@@@ -95,7 +95,7 @@ t('project_key and app source are defined exactly (and fit the envelope contractt('publish: an app publishes app.<project_key>.* only, as its own source and actor', async () => {let r = await publish(tokA(), appEvent(appA, prjA));assert.strictEqual(r.status, 201, r.text);-- const row = h.store.getEvent(r.body.event_id);++ const row = await h.store.getEvent(r.body.event_id);assert.strictEqual(row.project_id, prjA);assert.strictEqual(row.env, 'sandbox');assert.strictEqual(row.publisher, `app:${appA}`);@@@ -187,7 +187,7 @@ t('read: own project in the same environment, plus public first-party events', alet subA;t('subscribe: scoped patterns, public https endpoints only', async () => {-- const sub = (token, body) => request(h.base, 'POST', '/api/v1/subscriptions', { token, body });++ const sub = async (token, body) => await request(h.base, 'POST', '/api/v1/subscriptions', { token, body });for (const endpoint of ['http://hooks.example.com/x', 'https://127.0.0.1/x', 'https://[::1]/x', 'https://10.1.2.3/x', 'https://169.254.169.254/latest','https://evil.example.com/x', 'https://localhost/x', 'https://u:[email protected]/x', 'https://nowhere.example.com/x', 'https://intranet/x']) {const r = await sub(tokA(), { topic_pattern: `app.${keyA}.*`, endpoint });@@@ -282,10 +282,10 @@ t('delivery-time SSRF guard: re-resolved on every attempt, redirects never folloconst ev = await publish(tokB(), appEvent(appB, prjB, 'rebind.test'));const ev2 = await publish(tokB(), appEvent(appB, prjB, 'redirect.test'));await h.worker.drain();-- const d = h.store.getDelivery(ev.body.event_id, r.body.id);++ const d = await h.store.getDelivery(ev.body.event_id, r.body.id);assert.strictEqual(d.status, 'dead', 'a private address at delivery time is refused for good');assert.match(d.last_error, /non-public/);-- const d2 = h.store.getDelivery(ev2.body.event_id, redir.body.id);++ const d2 = await h.store.getDelivery(ev2.body.event_id, redir.body.id);assert.strictEqual(d2.status, 'failed'); assert.strictEqual(d2.last_status, 302);assert.deepStrictEqual(seen.map(s => s.url).filter(u => u !== '/b-project'), ['/redirect'], 'one request, the redirect was not followed, nothing reached the rebound host');});@@@ -293,7 +293,7 @@ t('delivery-time SSRF guard: re-resolved on every attempt, redirects never follot('quotas: per-project publish rate and retained bytes', async () => {const h2 = await boot({ appPost: testPost, dnsLookup: fakeLookup, env: { EVENTS_APP_PUBLISH_PER_MINUTE: '3', EVENTS_APP_SANDBOX_RETAINED_BYTES: '3000' } });try {-- const pub = (token, body) => request(h2.base, 'POST', '/api/v1/events', { token, body });++ const pub = async (token, body) => await request(h2.base, 'POST', '/api/v1/events', { token, body });for (let i = 0; i < 3; i++) assert.strictEqual((await pub(tokA2(), appEvent(appA2, prjA))).status, 201);let r = await pub(tokA2(), appEvent(appA2, prjA));assert.strictEqual(r.status, 429); assert.strictEqual(r.body.code, 'events.quota_exceeded');@@@ -339,7 +339,7 @@ t('revocation from Network stops an app\'s subscriptions and old tokens', asyncsubject: { type: 'app', id: appA }, payload: { project_id: prjA, environment: 'sandbox', reason: 'test' },}));assert.strictEqual(r.status, 201, r.text);-- assert.ok(h.store.listSubscriptions(`app:${appA}`).every(s => !s.enabled));++ assert.ok((await h.store.listSubscriptions(`app:${appA}`)).every(s => !s.enabled));r = await pull(tokA(), `app.${keyA}.*`);assert.strictEqual(r.status, 401); assert.strictEqual(r.body.code, 'token.revoked');r = await publish(tokA(), appEvent(appA, prjA));@@@ -358,7 +358,7 @@ t('realtime never streams app events, to anyone', async () => {await anon.waitFor(c => c.events().some(e => e.event.event_id === marker.body.event_id));assert.ok(!svc.events().some(e => e.event.event_type.startsWith('app.')));assert.ok(!anon.events().some(e => e.event.event_type.startsWith('app.')));-- svc.close(); anon.close();++ await svc.close(); anon.close();});t('stop', async () => {nnew file mode 100644@@@ -0,0 +1,11 @@++'use strict';++/**++ * A migrated database for one test boot (ADR-035), from openvibe-sdk/testing: PGlite by default; with++ * EVENTS_TEST_STORE=pg (npm run test:pg) the PostgreSQL + PgBouncer containers, with roles and a schema of its own.++ */++const { createTestDb, pgAvailable } = require('openvibe-sdk/testing');++const { MIGRATIONS } = require('../server/store');++++const testDb = ({ store = process.env.EVENTS_TEST_STORE || 'pglite', max = 4 } = {}) => createTestDb({ migrations: MIGRATIONS, store, service: 'events', max });++++module.exports = { testDb, pgAvailable };@@@ -11,7 +11,7 @@ const media = serviceToken('media', ['events.subscription.manage']);const admin = serviceToken('ops', ['events.delivery.admin']);async function subscribe(token, body) {-- return request(h.base, 'POST', '/api/v1/subscriptions', { token, body });++ return await request(h.base, 'POST', '/api/v1/subscriptions', { token, body });}async function publish(env) {const r = await request(h.base, 'POST', '/api/v1/events', { token: live, body: env });@@@ -100,7 +100,7 @@ t('success: signed POST (v2 only), verified by verifyDeliveryV2; trace propagateconst sub = (await subscribe(media, { topic_pattern: 'live.stream.*', endpoint: stub.url })).body;const env = envelope('live', { trace_id: '4bf92f3577b34da6a3ce929d0e0e4736' });const { seq } = await publish(env);-- assert.strictEqual(h.store.getDelivery(env.event_id, sub.id).status, 'pending', 'persisted before delivery');++ assert.strictEqual((await h.store.getDelivery(env.event_id, sub.id)).status, 'pending', 'persisted before delivery');await h.worker.drain();assert.strictEqual(stub.calls.length, 1);const call = stub.calls[0];@@@ -118,7 +118,7 @@ t('success: signed POST (v2 only), verified by verifyDeliveryV2; trace propagateassert.match(call.headers.traceparent, /^00-4bf92f3577b34da6a3ce929d0e0e4736-[0-9a-f]{16}-01$/);assert.deepStrictEqual(call.body.event.payload, env.payload);assert.strictEqual(call.body.seq, seq);-- const d = h.store.getDelivery(env.event_id, sub.id);++ const d = await h.store.getDelivery(env.event_id, sub.id);assert.strictEqual(d.status, 'delivered');assert.strictEqual(d.attempt, 1);await request(h.base, 'POST', `/api/v1/subscriptions/${sub.id}/disable`, { token: media });@@@ -139,7 +139,7 @@ t('retries with backoff (injectable clock), then DLQ after 8 attempts, then replassert.strictEqual(call.headers['x-openvibe-timestamp'], String(Math.floor(h.clock.now() / 1000)), 'every retry is signed with a fresh timestamp');assert.ok(verifyDeliveryV2(call.rawBody, call.headers, sub.secret, { now: h.clock.now() }));if (attempt > 1) assert.notStrictEqual(call.headers['x-openvibe-signature-v2'], stub.calls[attempt - 2].headers['x-openvibe-signature-v2']);-- const d = h.store.getDelivery(env.event_id, sub.id);++ const d = await h.store.getDelivery(env.event_id, sub.id);assert.strictEqual(d.attempt, attempt);if (attempt < 8) {assert.strictEqual(d.status, 'failed');@@@ -170,7 +170,7 @@ t('retries with backoff (injectable clock), then DLQ after 8 attempts, then replassert.strictEqual(rp.body.queued, 1);await h.worker.drain();assert.strictEqual(stub.calls.length, 9);-- assert.strictEqual(h.store.getDelivery(env.event_id, sub.id).status, 'delivered');++ assert.strictEqual((await h.store.getDelivery(env.event_id, sub.id)).status, 'delivered');await request(h.base, 'POST', `/api/v1/subscriptions/${sub.id}/disable`, { token: media });await stub.close();});@@@ -203,13 +203,13 @@ t('priority classes: critical before important before low', async () => {await publish(envelope('live', { event_type: 'live.prio.x', priority: 'low' }));await publish(envelope('live', { event_type: 'live.prio.x', priority: 'important' }));await publish(envelope('live', { event_type: 'live.prio.x', priority: 'critical' }));-- const due = h.store.dueDeliveries(h.clock.now(), 100);++ const due = await h.store.dueDeliveries(h.clock.now(), 100);assert.strictEqual(due.length, 3, 'one per subscription');assert.ok(due.every(d => d.priority === 0), 'each subscription\'s head is its critical event');await h.worker.drain();for (const o of orders) assert.deepStrictEqual(o, ['critical', 'important', 'low']);for (const s of stubs) await s.close();-- for (const sub of h.store.listSubscriptions()) h.store.setSubscriptionEnabled(sub.id, false);++ for (const sub of await h.store.listSubscriptions()) await h.store.setSubscriptionEnabled(sub.id, false);});t('backpressure: one in flight per subscription, <= maxInflight (20) overall', async () => {@@@ -234,7 +234,7 @@ t('backpressure: one in flight per subscription, <= maxInflight (20) overall', aawait bp.worker.drain();assert.strictEqual(peak, 20, `peak in flight ${peak}`);assert.strictEqual(perSubPeak, 1);-- assert.strictEqual(bp.store.deliveryCounts().delivered, 75);++ assert.strictEqual((await bp.store.deliveryCounts()).delivered, 75);await bp.stop();});@@@ -28,7 +28,7 @@ function exportToken(env, { project = P, cap = ['events.app.read'] } = {}) {let h;const seqs = { production: [], sandbox: [] };--const pull = (token, after = 0, limit = 100, topic = `app.${key}.*`) => request(h.base, 'GET', `/api/v1/events?topic=${encodeURIComponent(topic)}&after_seq=${after}&limit=${limit}`, { token });++const pull = async (token, after = 0, limit = 100, topic = `app.${key}.*`) => await request(h.base, 'GET', `/api/v1/events?topic=${encodeURIComponent(topic)}&after_seq=${after}&limit=${limit}`, { token });t('boot and publish as the project\'s app, in both environments (and another project)', async () => {h = await boot();@@@ -82,11 +82,13 @@ async function boot({ env = {}, clock = manualClock(), worker = 'manual', fetchIEVENTS_WORKER: worker === 'manual' ? 'off' : 'on',...env,});-- const h = await start({ config, clock, log: silent, fetchImpl, deliveryFetch, appPost, dnsLookup });++ // One database per boot (PGlite, or EVENTS_TEST_STORE=pg: the containers), dropped when the boot stops.++ const testdb = await require('./db').testDb();++ const h = await start({ config, db: testdb.db, clock, log: silent, fetchImpl, deliveryFetch, appPost, dnsLookup });const base = `http://127.0.0.1:${h.server.address().port}`;return {...h, base, clock, dir,-- async stop() { await h.close(); fs.rmSync(dir, { recursive: true, force: true }); },++ async stop() { await h.close(); await testdb.close(); fs.rmSync(dir, { recursive: true, force: true }); },};}@@@ -24,9 +24,9 @@ t('/api/ready: every check reports status, latency and checked_at; worker is reqassert.strictEqual(r.status, 200, r.text);assert.strictEqual(r.body.status, 'ready');assert.strictEqual(r.body.service, 'events');-- assert.deepStrictEqual(Object.keys(r.body.checks), ['db', 'network_jwks', 'delivery_worker', 'dlq']);++ assert.deepStrictEqual(Object.keys(r.body.checks), ['db', 'valkey', 'network_jwks', 'delivery_worker', 'dlq']);for (const [name, c] of Object.entries(r.body.checks)) {-- assert.strictEqual(c.status, 'ok', name);++ assert.strictEqual(c.status, name === 'valkey' ? 'skipped' : 'ok', name); // test boots have no VALKEY_URLassert.strictEqual(typeof c.latency_ms, 'number');assert.ok(Date.parse(c.checked_at));}@@@ -55,7 +55,7 @@ t('deliveries feed the latency histogram and attempt counter; a dead one degrade});t('/metrics: loopback only, templates not ids, Events gauges', async () => {-- const id = h.db.prepare('SELECT id FROM events ORDER BY seq LIMIT 1').get().id;++ const id = (await h.db.prepare('SELECT id FROM events ORDER BY seq LIMIT 1').get()).id;await request(h.base, 'GET', `/api/v1/events/${id}`, { token: reader });await request(h.base, 'GET', '/api/v1/events?topic=live.*&after_seq=0', { token: reader });await request(h.base, 'GET', '/no/such/12345');@@@ -13,8 +13,8 @@ let network;let tokenClient;const admin = serviceToken('ops', ['events.delivery.admin']);--function countIn(id) {-- return h.db.prepare('SELECT COUNT(*) AS n FROM events WHERE id = ?').get(id).n;++async function countIn(id) {++ return (await h.db.prepare('SELECT COUNT(*) AS n FROM events WHERE id = ?').get(id)).n; // Events' own database (PostgreSQL)}t('boot Events + a stub Network token endpoint', async () => {@@@ -46,7 +46,7 @@ t('publisher fills event_id, timestamp and trace (from traceparent)', async () =subject: { type: 'stream', id: '7' }, payload: {},}, { traceparent: '00-11111111111111111111111111111111-2222222222222222-01' });assert.match(out.event_id, /^evt_/);-- assert.strictEqual(h.store.getEvent(out.event_id).trace_id, '11111111111111111111111111111111');++ assert.strictEqual((await h.store.getEvent(out.event_id)).trace_id, '11111111111111111111111111111111');assert.strictEqual(network.issued(), 1);const batch = await publisher.publish([envelope(), envelope()]);assert.strictEqual(batch.results.length, 2);@@@ -71,7 +71,7 @@ t('outbox: enqueue inside a rolled-back transaction publishes nothing; committedassert.strictEqual(outbox.pending(), 0);let r = await outbox.flush();assert.deepStrictEqual(r, { sent: 0, failed: 0, rejected: 0 });-- assert.strictEqual(countIn(rolledBack.event_id), 0, 'rolled back: never published');++ assert.strictEqual(await countIn(rolledBack.event_id), 0, 'rolled back: never published');let committed;db.transaction(() => {@@@ -83,13 +83,13 @@ t('outbox: enqueue inside a rolled-back transaction publishes nothing; committedassert.strictEqual(r.sent, 1);r = await outbox.flush();assert.strictEqual(r.sent, 0, 'sent rows are not sent again');-- assert.strictEqual(countIn(committed.event_id), 1);++ assert.strictEqual(await countIn(committed.event_id), 1);// Relay crashed after Events accepted but before the row was marked: at-least-once resend is a duplicate.db.prepare('UPDATE event_outbox SET sent_at = NULL WHERE event_id = ?').run(committed.event_id);r = await outbox.flush();assert.strictEqual(r.sent, 1);-- assert.strictEqual(countIn(committed.event_id), 1, 'still exactly one event');++ assert.strictEqual(await countIn(committed.event_id), 1, 'still exactly one event');assert.ok(db.prepare('SELECT seq FROM event_outbox WHERE event_id = ?').get(committed.event_id).seq > 0);db.close();});@@@ -106,7 +106,7 @@ t('outbox: a poisoned row is isolated and rejected; a down Events is retried lat})();const r = await outbox.flush();assert.deepStrictEqual(r, { sent: 1, failed: 0, rejected: 1 });-- assert.strictEqual(countIn(good.event_id), 1);++ assert.strictEqual(await countIn(good.event_id), 1);assert.ok(db.prepare('SELECT rejected_at FROM event_outbox WHERE event_id = ?').get(bad.event_id).rejected_at);const down = events.createOutbox(db, { publisher: events.createPublisher({ eventsUrl: 'http://127.0.0.1:9', tokenClient }), now: () => clock });@@@ -120,7 +120,7 @@ t('outbox: a poisoned row is isolated and rejected; a down Events is retried lat// Events is back: the regular relay picks it up once due.clock += 1000;assert.strictEqual((await outbox.flush()).sent, 1);-- assert.strictEqual(countIn(later.event_id), 1);++ assert.strictEqual(await countIn(later.event_id), 1);db.close();});@@@ -161,12 +161,12 @@ t('consumer crash mid-processing + redelivery + replay -> exactly one effect (crconst env = envelope('live', { event_type: 'live.tip.sent', payload: { amount: 5 } });await request(h.base, 'POST', '/api/v1/events', { token: live, body: env });await h.worker.drain();-- const d = h.store.getDelivery(env.event_id, sub.id);++ const d = await h.store.getDelivery(env.event_id, sub.id);assert.strictEqual(d.status, 'failed', `${first}: the crashed attempt is retried`);assert.strictEqual(effects(env.event_id), first === 'crash-after-commit' ? 1 : 0);h.clock.advance(1000);await h.worker.drain();-- assert.strictEqual(h.store.getDelivery(env.event_id, sub.id).status, 'delivered');++ assert.strictEqual((await h.store.getDelivery(env.event_id, sub.id)).status, 'delivered');assert.strictEqual(effects(env.event_id), 1, `${first}: exactly one effect after redelivery`);const rp = await request(h.base, 'POST', '/api/v1/deliveries/replay', { token: admin, body: { subscription_id: sub.id, event_ids: [env.event_id] } });@@@ -35,7 +35,7 @@ t('valid envelope -> 201 with a seq; defaults filled', async () => {assert.strictEqual(r.body.event_id, env.event_id);assert.ok(Number.isInteger(r.body.seq) && r.body.seq > 0);assert.strictEqual(r.body.duplicate, false);-- const row = h.store.getEvent(env.event_id);++ const row = await h.store.getEvent(env.event_id);assert.strictEqual(row.priority, 'important');assert.strictEqual(row.visibility, 'internal');assert.strictEqual(row.trace_id, '0af7651916cd43dd8448eb211c80319c', 'trace taken from traceparent');@@@ -100,7 +100,7 @@ t('idempotent repeat: same event_id -> 200, same seq, stored once', async () =>assert.strictEqual(b.status, 200);assert.strictEqual(b.body.duplicate, true);assert.strictEqual(b.body.seq, a.body.seq);-- assert.strictEqual(h.db.prepare('SELECT COUNT(*) AS n FROM events WHERE id = ?').get(env.event_id).n, 1);++ assert.strictEqual((await h.db.prepare('SELECT COUNT(*) AS n FROM events WHERE id = ?').get(env.event_id)).n, 1);const c = await request(h.base, 'POST', '/api/v1/events', { token: live, body: { ...env, event_type: 'live.stream.ended' } });assert.strictEqual(c.status, 409);assert.strictEqual(c.body.code, 'events.id_conflict');@@@ -113,11 +113,11 @@ t('batch: atomic, <= 100, results per event', async () => {assert.strictEqual(r.body.results.length, 3);assert.ok(r.body.results[0].seq < r.body.results[1].seq && r.body.results[1].seq < r.body.results[2].seq);-- const before = h.store.lastSeq();++ const before = await h.store.lastSeq();r = await request(h.base, 'POST', '/api/v1/events', { token: live, body: { events: [envelope(), envelope('media')] } });assert.strictEqual(r.status, 403);assert.strictEqual(r.body.index, 1);-- assert.strictEqual(h.store.lastSeq(), before, 'nothing of a rejected batch is stored');++ assert.strictEqual(await h.store.lastSeq(), before, 'nothing of a rejected batch is stored');r = await request(h.base, 'POST', '/api/v1/events', { token: live, body: { events: Array.from({ length: 101 }, () => envelope()) } });assert.strictEqual(r.status, 413);@@@ -164,8 +164,8 @@ t('loop guard: one service re-triggering itself on the same subject', async () =t('publish receipts outlive retention: a pruned id is still a duplicate', async () => {const env = envelope();await request(h.base, 'POST', '/api/v1/events', { token: live, body: env });-- h.store.prune({ retentionDays: 30, receiptRetentionDays: 90, now: Date.now() + 31 * 86400000 });-- assert.strictEqual(h.store.getEvent(env.event_id), null);++ await h.store.prune({ retentionDays: 30, receiptRetentionDays: 90, now: Date.now() + 31 * 86400000 });++ assert.strictEqual(await h.store.getEvent(env.event_id), null);const r = await request(h.base, 'POST', '/api/v1/events', { token: live, body: env });assert.strictEqual(r.status, 200);assert.strictEqual(r.body.duplicate, true);@@@ -58,7 +58,7 @@ t('pull: cursor, topic filter, internal events included for services', async ()assert.deepStrictEqual(r.body.events, []);assert.strictEqual(r.body.latest_seq, s3);-- const id3 = h.db.prepare('SELECT id FROM events WHERE seq = ?').get(s3).id;++ const id3 = (await h.db.prepare('SELECT id FROM events WHERE seq = ?').get(s3)).id;r = await request(h.base, 'GET', `/api/v1/events/${id3}`, { token: reader });assert.strictEqual(r.body.seq, s3);r = await request(h.base, 'GET', '/api/v1/events?topic=live.*', { token: live });@@@ -68,8 +68,8 @@ t('pull: cursor, topic filter, internal events included for services', async ()});t('pull: a cursor older than retention reports the gap', async () => {-- const before = h.store.lastSeq();-- h.store.prune({ retentionDays: 30, now: Date.now() + 31 * 86400000 });++ const before = await h.store.lastSeq();++ await h.store.prune({ retentionDays: 30, now: Date.now() + 31 * 86400000 });const s = (await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope() })).body.seq;const r = await request(h.base, 'GET', '/api/v1/events?after_seq=1', { token: reader });assert.deepStrictEqual(r.body.gap, { from_seq: 2, to_seq: before });@@@ -94,7 +94,7 @@ t('health and ready', async () => {r = await request(h.base, 'GET', '/api/ready');assert.strictEqual(r.status, 200);assert.strictEqual(r.body.ready, true);-- assert.deepStrictEqual(Object.keys(r.body.checks), ['db', 'network_jwks', 'dlq'], 'worker off in this boot: no delivery_worker check');++ assert.deepStrictEqual(Object.keys(r.body.checks), ['db', 'valkey', 'network_jwks', 'dlq'], 'worker off in this boot: no delivery_worker check');assert.strictEqual(r.body.checks.db.status, 'ok');assert.strictEqual(r.body.checks.network_jwks.status, 'ok');r = await request(h.base, 'GET', '/nope');@@@ -152,7 +152,7 @@ t('disconnect and resume from the cursor with a fresh ticket: the missed events,t('a cursor older than retention is reported as a gap before the events kept', async () => {const old = await publish(network, notified(alice, 5));-- h.store.prune({ retentionDays: 30, now: Date.now() + 31 * 86400000 });++ await h.store.prune({ retentionDays: 30, now: Date.now() + 31 * 86400000 });const kept = await publish(network, notified(alice, 6));const c = await stream(`topics=${TOPIC}&ticket=${realtimeTicket({ subjectId: alice })}&last_event_id=${old - 1}`);await c.waitFor(x => x.events().length === 1);@@@ -88,7 +88,7 @@ t('Last-Event-ID resume replays what was missed, in order, then goes live', asynt('a cursor older than retention gets `event: gap` first', async () => {const old = await publish(live, envelope('live', { visibility: 'public' }));-- h.store.prune({ retentionDays: 30, now: Date.now() + 31 * 86400000 });++ await h.store.prune({ retentionDays: 30, now: Date.now() + 31 * 86400000 });const kept = await publish(live, envelope('live', { visibility: 'public' }));const c = await stream('topics=live.stream.*', { 'Last-Event-ID': String(old - 1) });await c.waitFor(x => x.events().length === 1);ddeleted file mode 100644@@@ -1,128 +0,0 @@--'use strict';--/**-- * scripts/redact-backfill.js: redacts the stored chat events of messages Chat deleted before it-- * published chat.message.deleted. Dry run by default, --apply only with a verified --backup,-- * idempotent, first-party chat events only.-- */--const assert = require('assert');--const fs = require('fs');--const path = require('path');--const Database = require('better-sqlite3');--const { ids } = require('openvibe-contracts');--const { openDb, createStore } = require('../server/store');--const { main } = require('../scripts/redact-backfill');--const { tmpDir, envelope, suite } = require('./helpers');----const t = suite('redact-backfill');--const dir = tmpDir();--const eventsPath = path.join(dir, 'events.db');--const chatPath = path.join(dir, 'chat.db');--const user = ids.newId('user');--let store, db;--const ev = {};----function created(messageId, text) {-- return envelope('chat', {-- event_type: 'chat.message.created', visibility: 'public', actor: { type: 'user', id: user },-- subject: { type: 'chat_message', id: String(messageId) }, payload: { message_id: messageId, anon_id: 'anon77', text },-- });--}--async function run(...argv) {-- const lines = [];-- const code = await main(['--db', eventsPath, '--chat-db', chatPath, ...argv], (l) => lines.push(l));-- return { code, out: lines.join('\n') };--}----t('setup: an Events store with chat events, a Chat database with deleted messages', () => {-- db = openDb(eventsPath);-- store = createStore(db);-- const batch = [created(1, 'deleted one'), created(2, 'still here'), created(3, 'expired one'), created(4, 'never in chat'),-- envelope('live', { subject: { type: 'chat_message', id: '1' }, payload: { note: 'live about 1' } })];-- [ev.m1, ev.m2, ev.m3, ev.m4, ev.live] = batch.map(e => e.event_id);-- store.insertBatch(batch.map(e => ({ ...e, trace_id: 'a'.repeat(32), priority: 'important', visibility: e.visibility || 'internal' })), { publisher: 'svc:test' });-- const chat = new Database(chatPath);-- chat.exec(`CREATE TABLE chat_messages (id INTEGER PRIMARY KEY, message TEXT, is_deleted INTEGER DEFAULT 0, auto_delete_at DATETIME);-- INSERT INTO chat_messages (id, message, is_deleted, auto_delete_at) VALUES-- (1, 'deleted one', 1, NULL), (2, 'still here', 0, '2999-01-01 00:00:00'), (3, 'expired one', 0, '2000-01-01 00:00:00');`);-- chat.close();--});----t('dry run: counts, changes nothing', async () => {-- const r = await run();-- assert.strictEqual(r.code, 0, r.out);-- assert.match(r.out, /not redacted: 4 events, 4 messages/);-- assert.match(r.out, /deleted in Chat: 2 messages, 2 events to redact/);-- assert.match(r.out, /not in Chat's database: 1 messages \(left alone; --include-missing redacts them\)/);-- assert.match(r.out, /dry run: nothing changed/);-- for (const id of [ev.m1, ev.m3]) assert.ok(!store.getEvent(id).redacted_at);--});----t('--apply refuses without --backup, and with an existing backup file', async () => {-- let r = await run('--apply');-- assert.strictEqual(r.code, 2);-- assert.match(r.out, /refusing --apply without --backup/);-- const existing = path.join(dir, 'exists.db');-- fs.writeFileSync(existing, 'x');-- r = await run('--apply', '--backup', existing);-- assert.strictEqual(r.code, 2);-- assert.match(r.out, /already exists/);-- assert.ok(!store.getEvent(ev.m1).redacted_at, 'nothing changed');-- r = await main(['--db', eventsPath], () => {});-- assert.strictEqual(r, 2, '--chat-db is required');--});----t('--apply --backup: verified 0600 backup first, then exactly the deleted messages are redacted', async () => {-- const backup = path.join(dir, 'backup-1.db');-- const r = await run('--apply', '--backup', backup);-- assert.strictEqual(r.code, 0, r.out);-- assert.match(r.out, /redacted {3}2 events/);-- assert.strictEqual(fs.statSync(backup).mode & 0o777, 0o600);-- const b = new Database(backup, { readonly: true });-- assert.match(b.prepare('SELECT payload FROM events WHERE id = ?').get(ev.m1).payload, /deleted one/, 'the backup is the state before');-- b.close();-- for (const id of [ev.m1, ev.m3]) {-- const row = store.getEvent(id);-- assert.strictEqual(row.redacted_by, 'backfill');-- assert.deepStrictEqual(Object.keys(JSON.parse(row.payload)).sort(), ['redacted', 'redacted_at', 'redacted_by']);-- assert.deepStrictEqual(JSON.parse(row.actor), { type: 'service', id: 'chat' });-- }-- assert.match(store.getEvent(ev.m2).payload, /still here/, 'a live message is untouched');-- assert.match(store.getEvent(ev.m4).payload, /never in chat/, 'unknown messages are left alone');-- assert.ok(!store.getEvent(ev.live).redacted_at, "another source's event about the same id is untouched");-- const seqs = db.prepare('SELECT seq FROM events ORDER BY seq').pluck().all();-- assert.deepStrictEqual(seqs, [1, 2, 3, 4, 5], 'no row removed, sequence intact');--});----t('--include-missing: a message Chat no longer has (hard-deleted) is redacted too', async () => {-- const r = await run('--apply', '--include-missing', '--backup', path.join(dir, 'backup-missing.db'));-- assert.strictEqual(r.code, 0, r.out);-- assert.match(r.out, /not in Chat's database: 1 messages \(redacted: --include-missing\)/);-- const row = store.getEvent(ev.m4);-- assert.strictEqual(row.redacted_by, 'backfill');-- assert.ok(!/never in chat/.test(row.payload));-- assert.match(store.getEvent(ev.m2).payload, /still here/, 'a live message is still untouched');--});----t('idempotent: a second run finds nothing and changes nothing', async () => {-- const before = db.prepare('SELECT id, payload, redacted_at FROM events ORDER BY seq').all();-- const r = await run('--apply', '--backup', path.join(dir, 'backup-2.db'));-- assert.strictEqual(r.code, 0, r.out);-- assert.match(r.out, /deleted in Chat: 0 messages, 0 events to redact/);-- assert.match(r.out, /redacted {3}0 events/);-- assert.deepStrictEqual(db.prepare('SELECT id, payload, redacted_at FROM events ORDER BY seq').all(), before);--});----t('refuses a database from before redaction', async () => {-- const old = path.join(dir, 'old.db');-- const o = new Database(old);-- o.exec('CREATE TABLE events (id TEXT PRIMARY KEY, seq INTEGER, payload TEXT)');-- o.close();-- const lines = [];-- const code = await main(['--db', old, '--chat-db', chatPath], (l) => lines.push(l));-- assert.strictEqual(code, 2);-- assert.match(lines.join('\n'), /no redaction columns/);--});----t('close', () => { db.close(); });----t.run();@@@ -109,13 +109,13 @@ t('chat.message.deleted turns the stored event into a tombstone at the same seq'const r = await publish(chat, env);delSeq = r.seq;assert.strictEqual(delSeq, msgSeq + 1);-- const row = h.store.getEvent(msgEventId);++ const row = await h.store.getEvent(msgEventId);assert.strictEqual(row.seq, msgSeq, 'seq unchanged');assert.strictEqual(row.redacted_by, delEventId);assert.ok(row.redacted_at > 0);assert.ok(!row.payload.includes('555-0199') && !row.actor.includes(alice));-- assert.strictEqual(h.store.getEvent(delEventId).redacts, 1);-- assert.ok(!h.store.getEvent(delEventId).redacted_at, 'the deletion event itself is kept as is');++ assert.strictEqual((await h.store.getEvent(delEventId)).redacts, 1);++ assert.ok(!(await h.store.getEvent(delEventId)).redacted_at, 'the deletion event itself is kept as is');});t('anonymous SSE replay: the tombstone, never the text', async () => {@@@ -194,8 +194,8 @@ t('re-publishing a deletion is a duplicate; a second deletion leaves tombstonesassert.strictEqual(again.body.duplicate, true);const second = deleted([101]);await publish(chat, second);-- assert.strictEqual(h.store.getEvent(msgEventId).redacted_by, delEventId, 'first redaction stands');-- assert.ok(!h.store.getEvent(delEventId).redacted_at, 'a deletion event is never redacted by a later one');++ assert.strictEqual((await h.store.getEvent(msgEventId)).redacted_by, delEventId, 'first redaction stands');++ assert.ok(!(await h.store.getEvent(delEventId)).redacted_at, 'a deletion event is never redacted by a later one');});t('only the owning service can redact', async () => {@@@ -206,7 +206,7 @@ t('only the owning service can redact', async () => {let r = await request(h.base, 'POST', '/api/v1/events', { token: live, body: { events: [envelope('live'), liveEvent] } });assert.strictEqual(r.status, 403, r.text);assert.strictEqual(r.body.code, 'events.redaction_not_allowed');-- assert.strictEqual(h.store.getEvent(liveEvent.event_id), null);++ assert.strictEqual(await h.store.getEvent(liveEvent.event_id), null);// Live by subject: its own namespace only, so Chat's event is untouched.await publish(live, envelope('live', { payload: { redacts: { subject_type: 'chat_message', subject_ids: ['201'] } } }));// Nor can Live publish Chat's deletion (source must be the caller).@@@ -232,20 +232,20 @@ t('only the owning service can redact', async () => {r = await request(h.base, 'POST', '/api/v1/events', { token: chat, body: deleted([201], { payload: { redacts: { subject_type: 'chat_message' } } }) });assert.strictEqual(r.status, 422);assert.strictEqual(r.body.code, 'events.invalid_redaction');-- const row = h.store.getEvent(env.event_id);++ const row = await h.store.getEvent(env.event_id);assert.ok(!row.redacted_at, 'still intact after every refused or foreign attempt');assert.ok(row.payload.includes('555-0199'));// The owner can.await publish(chat, deleted([201]));-- assert.ok(h.store.getEvent(env.event_id).redacted_at);-- assert.strictEqual(h.store.getEvent(env.event_id).seq, seq);++ assert.ok((await h.store.getEvent(env.event_id)).redacted_at);++ assert.strictEqual((await h.store.getEvent(env.event_id)).seq, seq);});--t('the text is gone from the database file, not just hidden', async () => {-- h.db.pragma('wal_checkpoint(TRUNCATE)');-- const bytes = fs.readFileSync(path.join(h.dir, 'events.db'));-- assert.ok(!bytes.includes(Buffer.from('555-0199')), 'secure_delete zeroed the old payloads');-- assert.ok(!bytes.includes(Buffer.from('anon4242')));++t('the text is gone from the stored rows, not just hidden', async () => {++ // PostgreSQL (ADR-035): the redacted row no longer holds the text; the old tuple versions are reclaimed by autovacuum.++ const rows = await h.db.prepare('SELECT payload::text AS p FROM events').all();++ assert.ok(!rows.some((r) => String(r.p).includes('555-0199')), 'no stored payload keeps the redacted text');++ assert.ok(!rows.some((r) => String(r.p).includes('anon4242')));});t('operator redaction (backfill): same owner rule, idempotent', async () => {@@@ -253,12 +253,12 @@ t('operator redaction (backfill): same owner rule, idempotent', async () => {await publish(chat, env);const other = envelope('live', { subject: { type: 'chat_message', id: '301' } });await publish(live, other);-- const done = h.store.redact('chat', { subject_type: 'chat_message', subject_ids: ['301'] }, { by: 'backfill' });++ const done = await h.store.redact('chat', { subject_type: 'chat_message', subject_ids: ['301'] }, { by: 'backfill' });assert.deepStrictEqual(done.map(d => d.id), [env.event_id]);-- assert.strictEqual(h.store.getEvent(env.event_id).redacted_by, 'backfill');-- assert.ok(!h.store.getEvent(other.event_id).redacted_at, "another source's event with the same subject is untouched");-- assert.deepStrictEqual(h.store.redact('chat', { subject_type: 'chat_message', subject_ids: ['301'] }, { by: 'backfill' }), [], 'second run: nothing');-- assert.throws(() => h.store.redact('chat', { event_ids: [other.event_id] }), /may redact only its own/);++ assert.strictEqual((await h.store.getEvent(env.event_id)).redacted_by, 'backfill');++ assert.ok(!(await h.store.getEvent(other.event_id)).redacted_at, "another source's event with the same subject is untouched");++ assert.deepStrictEqual(await h.store.redact('chat', { subject_type: 'chat_message', subject_ids: ['301'] }, { by: 'backfill' }), [], 'second run: nothing');++ await assert.rejects(async () => await h.store.redact('chat', { event_ids: [other.event_id] }), /may redact only its own/);});t('public replay window: browsers get a gap for public events older than 5 minutes', async () => {@@@ -19,7 +19,7 @@ const apps = require('../server/apps');const t = suite('security-idor');let w;--const snap = () => JSON.stringify(['subscriptions', 'events', 'deliveries', 'consumer_checkpoints'].map((tb) => w.h.db.prepare(`SELECT * FROM "${tb}" ORDER BY rowid`).all()));++const snap = async () => JSON.stringify((await Promise.all(['subscriptions', 'events', 'deliveries', 'consumer_checkpoints'].map(async (tb) => await w.h.db.prepare(`SELECT * FROM "${tb}" ORDER BY 1, 2`).all()))));t('world', async () => { w = await buildWorld(); });@@@ -36,7 +36,7 @@ t('another consumer\'s subscription: 404 on every route, nothing changes', async'A2 as sandbox': appToken({ appId: ids.A2, projectId: ids.PA, env: 'sandbox', cap: ALL }),};const bad = [];-- const before = snap();++ const before = await snap();const tryAll = async (who, token, id) => {for (const [m, p, body] of [['GET', `/api/v1/subscriptions/${id}`], ['POST', `/api/v1/subscriptions/${id}/rotate-secret`, { secret: 'x'.repeat(40) }],['POST', `/api/v1/subscriptions/${id}/disable`], ['POST', `/api/v1/subscriptions/${id}/enable`]]) {@@@ -50,7 +50,7 @@ t('another consumer\'s subscription: 404 on every route, nothing changes', asyncawait tryAll('A naming project B', crafted['A naming project B'], subs.appA.id);await tryAll('A2 as sandbox', crafted['A2 as sandbox'], subs.appA2.id);assert.deepStrictEqual(bad, [], bad.join('\n'));-- assert.strictEqual(snap(), before, 'the refusals changed nothing');++ assert.strictEqual(await snap(), before, 'the refusals changed nothing');// Lists: the crafted tokens list nothing of A's.for (const [who, token] of Object.entries(crafted)) {const r = await request(w.base, 'GET', '/api/v1/subscriptions', { token });@@@ -68,7 +68,7 @@ t('another consumer\'s subscription: 404 on every route, nothing changes', asynct('publishing into another namespace, source or project is refused and stores nothing', async () => {const { tok, ids } = w;const ev = (appId, projectId, over = {}) => envelope('x', { source: apps.appSource(appId), event_type: `app.${apps.projectKey(projectId)}.order.created`, actor: { type: 'app', id: appId }, subject: { type: 'order', id: '1' }, ...over });-- const before = snap();++ const before = await snap();const tries = [['app A into B\'s namespace', tok.appA, ev(ids.A, ids.PB)],['app A as B\'s source', tok.appA, ev(ids.A, ids.PA, { source: apps.appSource(ids.B) })],@@@ -91,7 +91,7 @@ t('publishing into another namespace, source or project is refused and stores noif (r2.status < 400 && !(r2.body && Array.isArray(r2.body.results) && r2.body.results.every((x) => x.status >= 400 || x.error))) bad.push(`${what} (batch) → ${r2.status} ${r2.text.slice(0, 120)}`);}assert.deepStrictEqual(bad, [], bad.join('\n'));-- assert.strictEqual(snap(), before, 'nothing was stored or queued');++ assert.strictEqual(await snap(), before, 'nothing was stored or queued');// Positive controls: each publishes into its own.assert.strictEqual((await request(w.base, 'POST', '/api/v1/events', { token: tok.appB, body: ev(ids.B, ids.PB) })).status, 201);assert.strictEqual((await request(w.base, 'POST', '/api/v1/events', { token: tok.live, body: envelope('live') })).status, 201);@@@ -99,7 +99,7 @@ t('publishing into another namespace, source or project is refused and stores not('deliveries, replay and checkpoints across consumers', async () => {const { tok, subs, ids, ev } = w;-- const before = snap();++ const before = await snap();const bad = [];for (const who of ['appA', 'appB', 'appA2', 'live', 'media', 'reader']) {for (const [m, p, body] of [['GET', `/api/v1/deliveries?subscription_id=${subs.appB.id}`], ['GET', '/api/v1/deliveries?status=dead'],@@@ -120,7 +120,7 @@ t('deliveries, replay and checkpoints across consumers', async () => {if (r.status !== 403) bad.push(`checkpoint ${topic} → ${r.status}`);}assert.deepStrictEqual(bad, [], bad.join('\n'));-- assert.strictEqual(snap(), before, 'the refusals changed nothing');++ assert.strictEqual(await snap(), before, 'the refusals changed nothing');const r = await request(w.base, 'POST', '/api/v1/deliveries/replay', { token: tok.ops, body: { subscription_id: subs.appB.id, from_seq: 0 } });assert.strictEqual(r.status, 200, 'the operator replays (positive control)');});@@@ -129,10 +129,10 @@ t('deliveries, the log and the tables carry no secret', async () => {const inLog = crawler.leaks({ text: w.logged.join('\n'), headers: {} }, all).map((l) => l.label);assert.deepStrictEqual(inLog, [], 'nothing secret was logged');const db = w.h.db;-- const tables = db.prepare("SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%'").all().map((r) => r.name).filter((n) => n !== 'subscriptions');++ const tables = (await db.prepare("SELECT table_name AS name FROM information_schema.tables WHERE table_schema = current_schema()").all()).map((r) => r.name).filter((n) => n !== 'subscriptions');assert.ok(tables.includes('events') && tables.includes('deliveries'));const hits = [];-- for (const tb of tables) for (const l of crawler.leaks({ text: JSON.stringify(db.prepare(`SELECT * FROM "${tb}"`).all()), headers: {} }, all)) hits.push(`${tb}: ${l.label}`);++ for (const tb of tables) for (const l of crawler.leaks({ text: JSON.stringify(await db.prepare(`SELECT * FROM "${tb}"`).all()), headers: {} }, all)) hits.push(`${tb}: ${l.label}`);assert.deepStrictEqual(hits, [], 'no secret outside the subscriptions table (events, deliveries, usage outbox)');});@@@ -107,7 +107,7 @@ const prj = `prj_${ids.ulid()}`;const app = ids.newId('app');const key = apps.projectKey(prj);const tokApp = () => appToken({ appId: app, projectId: prj, env: 'sandbox', cap: ALL });--const sub = (token, endpoint, topic = `app.${key}.*`) => request(h.base, 'POST', '/api/v1/subscriptions', { token, body: { topic_pattern: topic, endpoint } });++const sub = async (token, endpoint, topic = `app.${key}.*`) => await request(h.base, 'POST', '/api/v1/subscriptions', { token, body: { topic_pattern: topic, endpoint } });t('boot', async () => {await new Promise((r) => receiver.listen(0, '127.0.0.1', r));@@@ -158,7 +158,7 @@ t('DNS rebinding: public at subscribe, internal at delivery: dead at once, nothiconst e = await request(h.base, 'POST', '/api/v1/events', { token: tokApp(), body: envelope('x', { source: apps.appSource(app), event_type: `app.${key}.rebind.hit`, actor: { type: 'app', id: app } }) });assert.strictEqual(e.status, 201, e.text);await h.worker.drain();-- const d = h.store.getDelivery(e.body.event_id, r.body.id);++ const d = await h.store.getDelivery(e.body.event_id, r.body.id);assert.strictEqual(d.status, 'dead', 'refused for good');assert.match(d.last_error, /non-public/);assert.ok(!received.some((x) => x.url === '/rebind'), 'nothing reached the rebound address');@@@ -172,7 +172,7 @@ t('redirects are never followed (app endpoints and first-party ones)', async ()received.length = 0; internalHits = 0;const e = await request(h.base, 'POST', '/api/v1/events', { token: tokApp(), body: envelope('x', { source: apps.appSource(app), event_type: `app.${key}.redir.hit`, actor: { type: 'app', id: app } }) });await h.worker.drain();-- const d = h.store.getDelivery(e.body.event_id, r.body.id);++ const d = await h.store.getDelivery(e.body.event_id, r.body.id);assert.strictEqual(d.last_status, 302, 'a 3xx is a failed attempt');assert.notStrictEqual(d.status, 'delivered');assert.strictEqual(received.filter((x) => x.url === '/redirect').length, 1, 'one request to the endpoint, none to where it pointed');@@@ -185,7 +185,7 @@ t('redirects are never followed (app endpoints and first-party ones)', async ()const le = await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope('live') });await h.worker.drain();assert.strictEqual(fp.calls.length, 1);-- assert.strictEqual(h.store.getDelivery(le.body.event_id, s.body.id).last_status, 302);++ assert.strictEqual((await h.store.getDelivery(le.body.event_id, s.body.id)).last_status, 302);assert.strictEqual(internalHits, 0);await fp.close();});@@@ -32,7 +32,7 @@ const ALL = ['events.app.publish', 'events.app.read', 'events.app.subscribe'];async function buildWorld({ env = {} } = {}) {const dir = tmpDir();const logged = [];-- const keep = (...a) => logged.push(a.map((x) => (x && x.stack) || String(x)).join(' '));++ const keep = (...a) => { logged.push(a.map((x) => (x && x.stack) || String(x)).join(' ')); if (process.env.DEBUG) console.error(...a); };const log = { log: keep, info: keep, warn: keep, error: keep };// App endpoints: a fake DNS, and a local receiver standing in for every public address.@@@ -63,7 +63,8 @@ async function buildWorld({ env = {} } = {}) {const clock = manualClock();const config = load({ NODE_ENV: 'test', PORT: '0', EVENTS_DB_PATH: path.join(dir, 'events.db'), OV_NETWORK_PUBLIC_KEY: publicKey, EVENTS_WORKER: 'off', ...env });-- const h = await start({ config, clock, log, appPost, dnsLookup });++ const testdb = await require('./db').testDb(); // a database of its own (PGlite, or the containers under test:pg)++ const h = await start({ config, db: testdb.db, clock, log, appPost, dnsLookup });const base = `http://127.0.0.1:${h.server.address().port}`;const routes = crawler.listRoutes(h.server);@@@ -84,7 +85,7 @@ async function buildWorld({ env = {} } = {}) {userX: userToken({ subjectId: x }),userY: userToken({ subjectId: y }),};-- const call = (token, method, p, body, headers = {}) => request(base, method, p, { token, body, headers });++ const call = async (token, method, p, body, headers = {}) => await request(base, method, p, { token, body, headers });const must = (r, status, what) => { if (r.status !== status) throw new Error(`${what}: ${r.status} ${r.text.slice(0, 300)}`); return r; };const shownOnce = [];const once = (what, r) => { shownOnce.push({ what, status: r.status, cache: r.headers.get('cache-control') || '' }); return r; };@@@ -133,7 +134,7 @@ async function buildWorld({ env = {} } = {}) {return {h, base, dir, config, clock, routes, logged, secrets, subs, ev, markers, tok, shownOnce, call, fp, appSeen, dnsTable, dnsLookup,ids: { PA, PB, A, A2, B, x, y, keyA, keyB }, privateKey, realtimeTicket,-- stop: async () => { await fp.close(); await new Promise((r) => receiver.close(r)); await h.close(); fs.rmSync(dir, { recursive: true, force: true }); },++ stop: async () => { await fp.close(); await new Promise((r) => receiver.close(r)); await h.close(); await testdb.close(); fs.rmSync(dir, { recursive: true, force: true }); },};}@@@ -23,7 +23,7 @@ const appEvent = (over = {}) => ({const TRACE = '4bf92f3577b34da6a3ce929d0e0e4736';let h, network, statuses = [];--const rows = () => h.db.prepare('SELECT * FROM app_usage ORDER BY capability').all();++const rows = async () => await h.db.prepare('SELECT * FROM app_usage ORDER BY capability').all();t('boot: an app subscription and a first-party usage subscriber', async () => {// The hour starts at a round time so the test controls when it closes.@@@ -34,8 +34,8 @@ t('boot: an app subscription and a first-party usage subscriber', async () => {});h.clock.t = clockStart;network = await subscriber();-- h.store.createSubscription({ id: `sub_${ids.ulid()}`, consumer: source, topicPattern: `app.${key}.*`, endpoint: 'https://hooks.example.com/h', secret: 'a'.repeat(40), projectId: prj, env: 'sandbox' });-- h.store.createSubscription({ id: `sub_${ids.ulid()}`, consumer: 'network', topicPattern: 'events.usage.recorded', endpoint: network.url, secret: 'b'.repeat(40) });++ await h.store.createSubscription({ id: `sub_${ids.ulid()}`, consumer: source, topicPattern: `app.${key}.*`, endpoint: 'https://hooks.example.com/h', secret: 'a'.repeat(40), projectId: prj, env: 'sandbox' });++ await h.store.createSubscription({ id: `sub_${ids.ulid()}`, consumer: 'network', topicPattern: 'events.usage.recorded', endpoint: network.url, secret: 'b'.repeat(40) });});t('stored app events count; a repeated event_id does not', async () => {@@@ -44,7 +44,7 @@ t('stored app events count; a repeated event_id does not', async () => {assert.strictEqual(r.status, 201);r = await request(h.base, 'POST', '/api/v1/events', { token, body: batch[0] });assert.strictEqual(r.status, 200, 'a duplicate');-- const [pub] = rows();++ const [pub] = await rows();assert.deepStrictEqual([pub.project_id, pub.env, pub.capability, pub.unit, pub.quantity, pub.errors], [prj, 'sandbox', 'events.app.publish', 'events', 3, 0]);assert.strictEqual(pub.window_start % HOUR, 0, 'an hour window');});@@@ -55,7 +55,7 @@ t('refusals count as errors with code, status, trace id and event id; nothing ofassert.strictEqual(r.status, 403);r = await request(h.base, 'POST', '/api/v1/events', { token, body: { events: [appEvent(), appEvent(), appEvent()] } });assert.strictEqual(r.status, 429, 'past the per-minute quota: the whole batch is refused');-- const [pub] = rows();++ const [pub] = await rows();assert.strictEqual(pub.quantity, 3);assert.strictEqual(pub.errors, 2);assert.deepStrictEqual(JSON.parse(pub.error_codes), { 'events.type_not_allowed': 1, 'events.quota_exceeded': 1 });@@@ -68,7 +68,7 @@ t('refusals count as errors with code, status, trace id and event id; nothing oft('webhook delivery attempts count, a non-2xx as an error with the event and its trace', async () => {statuses = [500];await h.worker.drain();-- const sub = rows().find(r => r.capability === 'events.app.subscribe');++ const sub = (await rows()).find(r => r.capability === 'events.app.subscribe');assert.deepStrictEqual([sub.unit, sub.quantity, sub.errors], ['deliveries', 3, 1]);assert.deepStrictEqual(JSON.parse(sub.error_codes), { 'events.delivery.http_500': 1 });const [s] = JSON.parse(sub.samples);@@@ -76,14 +76,14 @@ t('webhook delivery attempts count, a non-2xx as an error with the event and its});t('nothing is sent before the hour closes', async () => {-- const out = h.flushUsage();++ const out = await h.flushUsage();assert.strictEqual(out.stored.length, 0);-- assert.strictEqual(h.store.usage.pending(), 2);++ assert.strictEqual(await h.store.usage.pending(), 2);});t('after the hour: one events.usage.recorded per rollup, valid, internal, delivered to first-party subscribers', async () => {h.clock.advance(HOUR);-- const out = h.flushUsage();++ const out = await h.flushUsage();assert.strictEqual(out.stored.length, 2);for (const row of out.stored) {assert.deepStrictEqual([row.event_type, row.source, row.visibility, row.priority, row.subject_type, row.subject_id, row.project_id], ['events.usage.recorded', 'events', 'internal', 'low', 'project', prj, null]);@@@ -92,7 +92,7 @@ t('after the hour: one events.usage.recorded per rollup, valid, internal, deliveconst text = JSON.stringify(payload);assert.ok(!text.includes(appId) && !text.includes(source) && !text.includes('order.created'), 'no app, source or event content');}-- assert.strictEqual(h.flushUsage().stored.length, 0, 'sent once');++ assert.strictEqual((await h.flushUsage()).stored.length, 0, 'sent once');await h.worker.drain();const got = network.calls.map(c => c.body.event);assert.strictEqual(got.length, 2);@@@ -108,9 +108,9 @@ t('apps never see the rollups', async () => {});t('a count in an hour already sent re-sends it as revision 2', async () => {-- const sent = rows().find(r => r.capability === 'events.app.publish');-- h.store.usage.record({ projectId: prj, env: 'sandbox', capability: 'events.app.publish', unit: 'events', quantity: 1, at: sent.window_start + 1000 });-- const out = h.flushUsage();++ const sent = (await rows()).find(r => r.capability === 'events.app.publish');++ await h.store.usage.record({ projectId: prj, env: 'sandbox', capability: 'events.app.publish', unit: 'events', quantity: 1, at: sent.window_start + 1000 });++ const out = await h.flushUsage();assert.strictEqual(out.stored.length, 1);const p = JSON.parse(out.stored[0].payload);assert.deepStrictEqual([p.quantity, p.revision], [4, 2], 'the corrected totals');@@@ -121,7 +121,7 @@ t('EVENTS_USAGE=off counts nothing', async () => {try {const r = await request(h2.base, 'POST', '/api/v1/events', { token, body: appEvent() });assert.strictEqual(r.status, 201);-- assert.strictEqual(h2.db.prepare('SELECT COUNT(*) AS n FROM app_usage').get().n, 0);++ assert.strictEqual((await h2.db.prepare('SELECT COUNT(*) AS n FROM app_usage').get()).n, 0);} finally { await h2.stop(); }});
@@ -2,7 +2,7 @@
@@ -2,7 +2,7 @@
# OpenVibe.Events — Environment Configuration
# OpenVibe.Events — Environment Configuration
# Copy to .env and fill in values.
# Copy to .env and fill in values.
# In production this lives at /etc/openvibe/events.env (0600); the systemd unit sets
# In production this lives at /etc/openvibe/events.env (0600); the systemd unit sets
-# NODE_ENV, PORT and EVENTS_DB_PATH itself.
+# NODE_ENV and PORT itself.
# ═══════════════════════════════════════════════════════════════
# ═══════════════════════════════════════════════════════════════
# Server
# Server
@@ -23,7 +23,14 @@ OV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000
@@ -23,7 +23,14 @@ OV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000
# A browser's user JWT is accepted when its aud contains one of these.
# A browser's user JWT is accepted when its aud contains one of these.
#EVENTS_USER_AUDIENCES=openvibe.events,openvibe.network
#EVENTS_USER_AUDIENCES=openvibe.events,openvibe.network
-# Store (SQLite). Production: /var/lib/openvibe-events/events.db (systemd StateDirectory).
+# PostgreSQL (ADR-035), written by OpenVibe.Host roles/data/add-service.sh events: the runtime role through PgBouncer,
+# the owner on a direct connection (migrations and the one-time import), and Valkey (shared limit counters).
+# Without DATABASE_URL, development uses an embedded PGlite database in data/pglite.
+DATABASE_URL=
+DATABASE_DIRECT_URL=
+VALKEY_URL=
+VALKEY_PREFIX=
+# The SQLite file of releases before PostgreSQL: read once by scripts/migrate-to-postgres.js.
EVENTS_DB_PATH=./data/events.db
EVENTS_DB_PATH=./data/events.db
# Events (and their deliveries) older than this are pruned; resuming from before it yields a gap.
# Events (and their deliveries) older than this are pruned; resuming from before it yields a gap.
EVENTS_RETENTION_DAYS=30
EVENTS_RETENTION_DAYS=30
@@ -9,6 +9,9 @@ jobs:
@@ -9,6 +9,9 @@ jobs:
uses: OpenVibers/OpenVibe.Shared/.github/workflows/test.yml@423fe519b5fe9594cc81b670e5ba5cb9f21f6176
uses: OpenVibers/OpenVibe.Shared/.github/workflows/test.yml@423fe519b5fe9594cc81b670e5ba5cb9f21f6176
with:
with:
contracts-service: events
contracts-service: events
+ # ADR-035: PostgreSQL 18 + PgBouncer (transaction mode) + Valkey 9 containers (openvibe-sdk
+ # scripts/test-services.sh). npm test runs every file on PGlite; test:pg runs them all through PgBouncer.
+ test: 'eval "$(node_modules/openvibe-sdk/scripts/test-services.sh up)" && npm test && npm run test:pg'
# Secret scan (gitleaks) and dependency audit, shared by every OpenVibers repo (Track Q).
# Secret scan (gitleaks) and dependency audit, shared by every OpenVibers repo (Track Q).
security:
security:
uses: OpenVibers/OpenVibe.Shared/.github/workflows/security.yml@423fe519b5fe9594cc81b670e5ba5cb9f21f6176
uses: OpenVibers/OpenVibe.Shared/.github/workflows/security.yml@423fe519b5fe9594cc81b670e5ba5cb9f21f6176
@@ -90,11 +90,11 @@ A producer takes back what it published (a deleted chat message) with a directiv
@@ -90,11 +90,11 @@ A producer takes back what it published (a deleted chat message) with a directiv
```
```
- `redacts.event_ids` (up to 1000 `evt_…`) names events directly; `subject_type` + `subject_ids` (up to 1000) names every stored event of the same source about those subjects.
- `redacts.event_ids` (up to 1000 `evt_…`) names events directly; `subject_type` + `subject_ids` (up to 1000) names every stored event of the same source about those subjects.
-- In the transaction that stores the directive, each target becomes a **tombstone**: `payload` is replaced by `{ "redacted": true, "redacted_at": "<ISO>", "redacted_by": "<the redacting event_id>" }` and `actor` by the producer itself (`{ "type": "service", "id": "<source>" }`, or the app). `event_id`, `seq`, `event_type`, `subject`, `timestamp` and `visibility` stay, so sequences have no holes. The old payload is overwritten on disk (`secure_delete`), not just hidden.
+- In the transaction that stores the directive, each target becomes a **tombstone**: `payload` is replaced by `{ "redacted": true, "redacted_at": "<ISO>", "redacted_by": "<the redacting event_id>" }` and `actor` by the producer itself (`{ "type": "service", "id": "<source>" }`, or the app). `event_id`, `seq`, `event_type`, `subject`, `timestamp` and `visibility` stay, so sequences have no holes. On PostgreSQL the replaced row version is a dead tuple until autovacuum reclaims it (the `events` table vacuums after 1% of its rows change, so within minutes); until then, and in backups taken before the redaction, the old bytes still exist on the host. No read path serves them.
- Every read path serves the tombstone from then on: pull, `GET /api/v1/events/:id`, SSE replay (anonymous, signed in or service), queued deliveries and DLQ replays. First-party consumers get the tombstone at the original seq and then the deletion event itself; they should check `payload.redacted`.
- Every read path serves the tombstone from then on: pull, `GET /api/v1/events/:id`, SSE replay (anonymous, signed in or service), queued deliveries and DLQ replays. First-party consumers get the tombstone at the original seq and then the deletion event itself; they should check `payload.redacted`.
- Authority is the publish rule: only the owning source (for an app, its own project and environment) can redact. Naming another source's event by id refuses the whole publish with `403 events.redaction_not_allowed`; a subject match never reaches another source. A malformed directive is `422 events.invalid_redaction`. Events carrying a directive are never redacted themselves. No capability beyond `events.event.publish` is needed.
- Authority is the publish rule: only the owning source (for an app, its own project and environment) can redact. Naming another source's event by id refuses the whole publish with `403 events.redaction_not_allowed`; a subject match never reaches another source. A malformed directive is `422 events.invalid_redaction`. Events carrying a directive are never redacted themselves. No capability beyond `events.event.publish` is needed.
- Redaction applies to what is stored when the directive arrives; it does not block later events about the same subject.
- Redaction applies to what is stored when the directive arrives; it does not block later events about the same subject.
-- Events stored before a producer published deletions: [scripts/redact-backfill.js](scripts/redact-backfill.js) (chat only; dry run by default, `--apply` needs `--backup <new file>`).
+- Events stored before a producer published deletions: the one-time `scripts/redact-backfill.js` (chat only) ran on 2026-09-24 and was retired with the move to PostgreSQL (it is in git history).
## Subscriptions and delivery
## Subscriptions and delivery
@@ -169,7 +169,8 @@ Quotas recorded in Network (`dev_quotas`) are not read yet: that needs `network.
@@ -169,7 +169,8 @@ Quotas recorded in Network (`dev_quotas`) are not read yet: that needs `network.
const events = require('openvibe-events');
const events = require('openvibe-events');
const tokenClient = contracts.serviceAuth.createTokenClient({ tokenUrl: `${NETWORK}/oauth/token`, clientId, clientSecret, audience: 'openvibe.events' });
const tokenClient = contracts.serviceAuth.createTokenClient({ tokenUrl: `${NETWORK}/oauth/token`, clientId, clientSecret, audience: 'openvibe.events' });
-// Producer: transactional outbox in the service's own better-sqlite3 database
+// Producer: transactional outbox in the service's own better-sqlite3 database (a service on PostgreSQL uses
+// openvibe-sdk/events createPgOutbox / createPgInbox instead: same envelope, receipts and relay, async)
const outbox = events.createOutbox(db, { publisher: events.createPublisher({ eventsUrl: 'http://127.0.0.1:4300', tokenClient }) });
const outbox = events.createOutbox(db, { publisher: events.createPublisher({ eventsUrl: 'http://127.0.0.1:4300', tokenClient }) });
outbox.ensureSchema();
outbox.ensureSchema();
db.transaction(() => {
db.transaction(() => {
@@ -215,7 +216,7 @@ es.addEventListener('gap', (m) => { /* events were missed: refetch state */ });
@@ -215,7 +216,7 @@ es.addEventListener('gap', (m) => { /* events were missed: refetch state */ });
## Owns
## Owns
-- `events`, `subscriptions`, `deliveries`, `consumer_checkpoints`, `idempotency_receipts`, `app_revocations`, `app_usage` (SQLite today; the plan's PostgreSQL + Redis fanout is a later step)
+- `events`, `subscriptions`, `deliveries`, `consumer_checkpoints`, `idempotency_receipts`, `app_revocations`, `app_usage` (PostgreSQL `ov_events`, ADR-035; the realtime fan-out is in-process, one Events process)
- developer-app event scope, sandbox separation and per-project Events quotas (ADR-014)
- developer-app event scope, sandbox separation and per-project Events quotas (ADR-014)
- canonical event envelope (event_id, trace_id, type, version, source, actor, subject + revision, payload)
- canonical event envelope (event_id, trace_id, type, version, source, actor, subject + revision, payload)
- priority classes `critical|important|low`, loop guards, backpressure, DLQ and replay
- priority classes `critical|important|low`, loop guards, backpressure, DLQ and replay
@@ -270,13 +271,15 @@ Reporting a vulnerability: [SECURITY.md](SECURITY.md). The rules the code keeps:
@@ -270,13 +271,15 @@ Reporting a vulnerability: [SECURITY.md](SECURITY.md). The rules the code keeps:
Production deploys with `sudo ovhost deploy events` on the host (strategy `git-checkout`: fetch,
Production deploys with `sudo ovhost deploy events` on the host (strategy `git-checkout`: fetch,
fast-forward `/opt/openvibe.events`, install on a lockfile change, restart, wait for `/api/ready`).
fast-forward `/opt/openvibe.events`, install on a lockfile change, restart, wait for `/api/ready`).
-The unit is `openvibe-events.service` on `127.0.0.1:4300`, the env file `/etc/openvibe/events.env`. The store is
-`/var/lib/openvibe-events/events.db`; nginx serves `events.openvibe.network` from
+The unit is `openvibe-events.service` on `127.0.0.1:4300`, the env file `/etc/openvibe/events.env`. The database is
+`ov_events` on the host's data role (`sudo /opt/openvibe.host/roles/data/add-service.sh events` writes its settings); the
+release migrates it at boot. The one-time move from SQLite is `scripts/migrate-to-postgres.js` (openvibe-sdk
+`runSqliteMigration`, with a `--pglite` rehearsal mode), run while the service is stopped; the old
+`/var/lib/openvibe-events/events.db` stays read-only for 7 days as the rollback. nginx serves `events.openvibe.network` from
[deploy/nginx/events.openvibe.network.conf](deploy/nginx/events.openvibe.network.conf). ovhost treats open
[deploy/nginx/events.openvibe.network.conf](deploy/nginx/events.openvibe.network.conf). ovhost treats open
realtime connections as a report-only drain, so `--wait-idle` waits for them.
realtime connections as a report-only drain, so `--wait-idle` waits for them.
Rollback: ovhost puts the previous sha back by itself when `/api/ready` does not answer 2xx after the
Rollback: ovhost puts the previous sha back by itself when `/api/ready` does not answer 2xx after the
-restart; afterwards `sudo ovhost rollback events --to <sha>`. Nothing blocks a rollback: the schema
-code only adds tables and columns.
+restart; afterwards `sudo ovhost rollback events --to <sha>`. Migrations only add tables and columns.
`EVENTS_LIMITS=off` turns the per-actor limits off without a deploy.
`EVENTS_LIMITS=off` turns the per-actor limits off without a deploy.
## Launch rule
## Launch rule
@@ -4,11 +4,19 @@
@@ -4,11 +4,19 @@
"code": true,
"code": true,
"wave": "W3",
"wave": "W3",
"plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026)",
"plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026)",
- "runtime": "node 22, express, better-sqlite3",
+ "runtime": "node 22, express, PostgreSQL 18 (openvibe-sdk/db, async) + Valkey 9",
"port": 4300,
"port": 4300,
"domain": "events.openvibe.network",
"domain": "events.openvibe.network",
- "contracts": "openvibe-contracts v0.63.0",
- "capabilities": ["events.event.publish", "events.subscription.manage", "events.event.read", "events.delivery.admin", "events.app.publish", "events.app.read", "events.app.subscribe"],
+ "contracts": "openvibe-contracts v0.76.0",
+ "capabilities": [
+ "events.event.publish",
+ "events.subscription.manage",
+ "events.event.read",
+ "events.delivery.admin",
+ "events.app.publish",
+ "events.app.read",
+ "events.app.subscribe"
+ ],
"capabilitiesRegistered": true,
"capabilitiesRegistered": true,
"deployed": true,
"deployed": true,
"deployment": "Deployed with `sudo ovhost deploy events`: /opt/openvibe.events, unit openvibe-events, 127.0.0.1:4300; public at https://events.openvibe.network (realtime SSE, health, token-guarded /api/v1/events, /api/v1/subscriptions, /api/v1/checkpoints; /api/v1/deliveries host-local); real traffic since 2026-09-23 (Live, Media, Chat, Games, Network and OpenRe publish; on 2026-09-23 8 subscriptions, held by Search, Tips, VIP, News and Reviews); restore drill passed 2026-09-23",
"deployment": "Deployed with `sudo ovhost deploy events`: /opt/openvibe.events, unit openvibe-events, 127.0.0.1:4300; public at https://events.openvibe.network (realtime SSE, health, token-guarded /api/v1/events, /api/v1/subscriptions, /api/v1/checkpoints; /api/v1/deliveries host-local); real traffic since 2026-09-23 (Live, Media, Chat, Games, Network and OpenRe publish; on 2026-09-23 8 subscriptions, held by Search, Tips, VIP, News and Reviews); restore drill passed 2026-09-23",
@@ -22,6 +30,6 @@
@@ -22,6 +30,6 @@
"developer-app publish, read and subscribe with sandbox separation, SSRF-guarded webhooks and project quotas",
"developer-app publish, read and subscribe with sandbox separation, SSRF-guarded webhooks and project quotas",
"hourly project usage rollups (events.usage.recorded)",
"hourly project usage rollups (events.usage.recorded)",
"redaction tombstones",
"redaction tombstones",
- "per-actor limits (openvibe-sdk/limits)"
+ "per-actor limits (openvibe-sdk/limits; shared on Valkey)"
]
]
}
}
@@ -2,8 +2,8 @@
@@ -2,8 +2,8 @@
Description=OpenVibe.Events — durable events, signed delivery, dead letters, replay and the SSE realtime gateway
Description=OpenVibe.Events — durable events, signed delivery, dead letters, replay and the SSE realtime gateway
# Network (openvibe-network) signs the tokens this service verifies. Events starts without it
# Network (openvibe-network) signs the tokens this service verifies. Events starts without it
# and keeps retrying the JWKS fetch; /api/ready stays 503 until the key has loaded.
# and keeps retrying the JWKS fetch; /api/ready stays 503 until the key has loaded.
-After=network.target
-Wants=openvibe-network.service
+After=network.target postgresql.service pgbouncer.service valkey-server.service
+Wants=openvibe-network.service postgresql.service pgbouncer.service valkey-server.service
[Service]
[Service]
Type=simple
Type=simple
@@ -26,7 +26,8 @@ NoNewPrivileges=true
@@ -26,7 +26,8 @@ NoNewPrivileges=true
PrivateTmp=true
PrivateTmp=true
ProtectSystem=strict
ProtectSystem=strict
ProtectHome=true
ProtectHome=true
-# The code tree stays read-only; the event store lives in /var/lib/openvibe-events.
+# The code tree stays read-only. The event store is PostgreSQL (ov_events); /var/lib/openvibe-events keeps the
+# pre-PostgreSQL SQLite file (read-only) for the rollback window.
StateDirectory=openvibe-events
StateDirectory=openvibe-events
Environment=EVENTS_DB_PATH=/var/lib/openvibe-events/events.db
Environment=EVENTS_DB_PATH=/var/lib/openvibe-events/events.db
new file mode 100644
new file mode 100644
@@ -0,0 +1,129 @@
@@ -0,0 +1,129 @@
+-- phase: expand
+-- OpenVibe.Events on PostgreSQL (ADR-035, roadmap WS-X2): the tables as they were on SQLite (converted by openvibe-sdk
+-- tools/asyncify/sqlite-schema-to-pg: text COLLATE "C" compares like SQLite, integers are bigint, identities keep their ids),
+-- then the openvibe-publishing stores and the openvibe-sdk inbox and outbox. Generated once on 2026-09-28; never edited after it runs.
+
+CREATE TABLE sequences (
+ name text COLLATE "C" PRIMARY KEY,
+ value bigint NOT NULL
+);
+CREATE TABLE events (
+ id text COLLATE "C" PRIMARY KEY,
+ seq bigint NOT NULL UNIQUE,
+ event_type text COLLATE "C" NOT NULL,
+ version bigint NOT NULL,
+ source text COLLATE "C" NOT NULL,
+ actor text COLLATE "C" NOT NULL,
+ subject_type text COLLATE "C" NOT NULL,
+ subject_id text COLLATE "C" NOT NULL,
+ subject_revision bigint,
+ trace_id text COLLATE "C" NOT NULL,
+ priority text COLLATE "C" NOT NULL,
+ visibility text COLLATE "C" NOT NULL,
+ occurred_at text COLLATE "C" NOT NULL,
+ received_at bigint NOT NULL,
+ payload text COLLATE "C" NOT NULL,
+ hops bigint NOT NULL DEFAULT 1,
+ publisher text COLLATE "C" NOT NULL,
+ request_id text COLLATE "C"
+);
+CREATE INDEX idx_events_trace ON events(trace_id);
+CREATE INDEX idx_events_received ON events(received_at);
+CREATE INDEX idx_events_type_seq ON events(event_type, seq);
+
+CREATE TABLE subscriptions (
+ id text COLLATE "C" PRIMARY KEY,
+ consumer text COLLATE "C" NOT NULL,
+ topic_pattern text COLLATE "C" NOT NULL,
+ endpoint text COLLATE "C" NOT NULL,
+ secret text COLLATE "C" NOT NULL,
+ enabled bigint NOT NULL DEFAULT 1,
+ retry_policy text COLLATE "C",
+ created_at bigint NOT NULL,
+ updated_at bigint NOT NULL
+);
+CREATE INDEX idx_subscriptions_consumer ON subscriptions(consumer);
+
+CREATE TABLE deliveries (
+ event_id text COLLATE "C" NOT NULL REFERENCES events(id) ON DELETE CASCADE,
+ subscription_id text COLLATE "C" NOT NULL REFERENCES subscriptions(id) ON DELETE CASCADE,
+ seq bigint NOT NULL,
+ priority bigint NOT NULL,
+ attempt bigint NOT NULL DEFAULT 0,
+ status text COLLATE "C" NOT NULL CHECK (status IN ('pending', 'delivered', 'failed', 'dead')),
+ next_attempt_at bigint,
+ last_error text COLLATE "C",
+ last_status bigint,
+ delivered_at bigint,
+ created_at bigint NOT NULL,
+ updated_at bigint NOT NULL,
+ PRIMARY KEY (event_id, subscription_id)
+);
+CREATE INDEX idx_deliveries_due ON deliveries(status, next_attempt_at);
+CREATE INDEX idx_deliveries_sub ON deliveries(subscription_id, status, seq);
+
+CREATE TABLE consumer_checkpoints (
+ consumer text COLLATE "C" NOT NULL,
+ topic_pattern text COLLATE "C" NOT NULL,
+ cursor bigint NOT NULL,
+ updated_at bigint NOT NULL,
+ PRIMARY KEY (consumer, topic_pattern)
+);
+
+-- Developer apps whose access Network revoked (network.app.revoked -> scope 'app') or whose
+-- events.app.subscribe grant it withdrew (scope 'events.app.subscribe'). An app token issued at or
+-- before revoked_at (ms, the revocation's own timestamp) is refused for that scope.
+CREATE TABLE app_revocations (
+ consumer text COLLATE "C" NOT NULL,
+ scope text COLLATE "C" NOT NULL,
+ revoked_at bigint NOT NULL,
+ PRIMARY KEY (consumer, scope)
+);
+
+-- Publish receipts: remember accepted event ids for longer than the events themselves, so a
+-- producer that re-publishes an old id after retention is still answered "duplicate".
+CREATE TABLE idempotency_receipts (
+ consumer text COLLATE "C" NOT NULL,
+ event_id text COLLATE "C" NOT NULL,
+ processed_at bigint NOT NULL,
+ PRIMARY KEY (consumer, event_id)
+);
+
+-- Columns and indexes added after the first release (server/store.js ADDED_COLUMNS: developer apps, ADR-014, and redaction).
+ALTER TABLE events ADD COLUMN project_id text COLLATE "C";
+ALTER TABLE events ADD COLUMN env text COLLATE "C" NOT NULL DEFAULT 'production';
+ALTER TABLE events ADD COLUMN size_bytes bigint NOT NULL DEFAULT 0;
+ALTER TABLE events ADD COLUMN redacted_at bigint;
+ALTER TABLE events ADD COLUMN redacted_by text COLLATE "C";
+ALTER TABLE events ADD COLUMN redacts bigint NOT NULL DEFAULT 0;
+ALTER TABLE subscriptions ADD COLUMN project_id text COLLATE "C";
+ALTER TABLE subscriptions ADD COLUMN env text COLLATE "C" NOT NULL DEFAULT 'production';
+ALTER TABLE subscriptions ADD COLUMN previous_secret text COLLATE "C";
+ALTER TABLE subscriptions ADD COLUMN previous_secret_until bigint;
+CREATE INDEX idx_events_project ON events(project_id, env, received_at) WHERE project_id IS NOT NULL;
+CREATE INDEX idx_subscriptions_project ON subscriptions(project_id, env) WHERE project_id IS NOT NULL;
+CREATE INDEX idx_events_subject ON events(source, subject_type, subject_id);
+
+-- The global event sequence (an import replaces the row with SQLite's).
+INSERT INTO sequences (name, value) VALUES ('events', 0) ON CONFLICT DO NOTHING;
+
+-- server/usage.js: per-app usage windows (was created by the module).
+CREATE TABLE app_usage (
+ project_id text COLLATE "C" NOT NULL,
+ env text COLLATE "C" NOT NULL,
+ capability text COLLATE "C" NOT NULL,
+ unit text COLLATE "C" NOT NULL,
+ window_start bigint NOT NULL,
+ quantity bigint NOT NULL DEFAULT 0,
+ errors bigint NOT NULL DEFAULT 0,
+ error_codes text COLLATE "C" NOT NULL DEFAULT '{}',
+ samples text COLLATE "C" NOT NULL DEFAULT '[]',
+ revision bigint NOT NULL DEFAULT 1,
+ event_id text COLLATE "C",
+ emitted_at bigint,
+ PRIMARY KEY (project_id, env, capability, unit, window_start)
+);
+CREATE INDEX idx_app_usage_open ON app_usage(emitted_at, window_start);
+
+-- Redaction rewrites payloads in place and PostgreSQL has no secure_delete: reclaim the replaced row versions soon.
+ALTER TABLE events SET (autovacuum_vacuum_scale_factor = 0.01, autovacuum_vacuum_threshold = 50);
@@ -12,14 +12,25 @@
@@ -12,14 +12,25 @@
"better-sqlite3": "^11.10.0",
"better-sqlite3": "^11.10.0",
"dotenv": "^16.4.7",
"dotenv": "^16.4.7",
"express": "^4.21.2",
"express": "^4.21.2",
- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.63.0",
- "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0",
- "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0"
+ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",
+ "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.20.4",
+ "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.28.0",
+ "pg": "^8.23.0"
+ },
+ "devDependencies": {
+ "@electric-sql/pglite": "^0.5.8"
},
},
"engines": {
"engines": {
"node": ">=20.0.0"
"node": ">=20.0.0"
}
}
},
},
+ "node_modules/@electric-sql/pglite": {
+ "version": "0.5.8",
+ "resolved": "https://registry.npmjs.org/@electric-sql/pglite/-/pglite-0.5.8.tgz",
+ "integrity": "sha512-n9tsbUOhwx2epK1V0ZG9Ar4SHWUju04dhmzZXiSBXwBoleOvIfals33NAaWgagQVAL4Rbvx/Ptsu3P+pA09f6Q==",
+ "devOptional": true,
+ "license": "Apache-2.0"
+ },
"node_modules/accepts": {
"node_modules/accepts": {
"version": "1.3.8",
"version": "1.3.8",
"resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.8.tgz",
"resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.8.tgz",
@@ -968,9 +979,9 @@
@@ -968,9 +979,9 @@
}
}
},
},
"node_modules/openvibe-contracts": {
"node_modules/openvibe-contracts": {
- "version": "0.63.0",
- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.63.0",
- "integrity": "sha512-vmc1b1vWg9M5fm57X50MlZEZHkP2IshHwBXt68+kHDV85nyg6FxaYyUWa4ibi1eeFqzMcECPlILSRTOgg2Xbgw==",
+ "version": "0.76.0",
+ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",
+ "integrity": "sha512-zTdx776KWrcJKm1JTnf+0+BnsvswdvXy1ldPyse1hSqhnW/SSr98sX696eRetFUhTl8qVM1xSRCykqIUbqItPw==",
"license": "AGPL-3.0-only",
"license": "AGPL-3.0-only",
"dependencies": {
"dependencies": {
"ajv": "^8.17.1",
"ajv": "^8.17.1",
@@ -984,18 +995,38 @@
@@ -984,18 +995,38 @@
}
}
},
},
"node_modules/openvibe-sdk": {
"node_modules/openvibe-sdk": {
- "version": "0.12.0",
- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0",
- "integrity": "sha512-iG1h905O1f7y/PnP8xLA6GCVzXEBDfSaBj2tc1ucNreWg2sT/AJWosb8NW0WRMK5WS4sIQ6tMsprmDLj/fjjeQ==",
+ "version": "0.20.4",
+ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.20.4",
+ "integrity": "sha512-IMvvBrFhCVO4sm2Ii694BslcnKHSV7YABXviLqrDI/2lTr9NKIH80z3HXxwMUgj0fwZDWBWcNgqPs6OaWIB0GQ==",
"license": "MIT",
"license": "MIT",
"engines": {
"engines": {
"node": ">=20"
"node": ">=20"
+ },
+ "peerDependencies": {
+ "@electric-sql/pglite": "^0.5.8",
+ "better-sqlite3": ">=11",
+ "iovalkey": "^0.4.0",
+ "pg": "^8.23.0"
+ },
+ "peerDependenciesMeta": {
+ "@electric-sql/pglite": {
+ "optional": true
+ },
+ "better-sqlite3": {
+ "optional": true
+ },
+ "iovalkey": {
+ "optional": true
+ },
+ "pg": {
+ "optional": true
+ }
}
}
},
},
"node_modules/openvibe-shared": {
"node_modules/openvibe-shared": {
- "version": "1.25.0",
- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0",
- "integrity": "sha512-oqZ+PvljrWi8IEbtg6bDYai17ZV2sALFx/ezMptniELSr10TgFUiGUQt+b1IWwiHv0T2Y1U2iSrkirJxMNgjSw==",
+ "version": "1.28.0",
+ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.28.0",
+ "integrity": "sha512-WLd11lDG75veVi7mFwngquK2nqIQV86Z14Ad/xE2dYvyMT5mhKJxxxlKUoskpdyLeap0ucMvzzS8Tg7vPGwYoQ==",
"license": "MIT",
"license": "MIT",
"dependencies": {
"dependencies": {
"jsonwebtoken": "^9.0.0"
"jsonwebtoken": "^9.0.0"
@@ -1016,6 +1047,134 @@
@@ -1016,6 +1047,134 @@
"integrity": "sha512-A/AGNMFN3c8bOlvV9RreMdrv7jsmF9XIfDeCd87+I8RNg6s78BhJxMu69NEMHBSJFxKidViTEdruRwEk/WIKqA==",
"integrity": "sha512-A/AGNMFN3c8bOlvV9RreMdrv7jsmF9XIfDeCd87+I8RNg6s78BhJxMu69NEMHBSJFxKidViTEdruRwEk/WIKqA==",
"license": "MIT"
"license": "MIT"
},
},
+ "node_modules/pg": {
+ "version": "8.23.0",
+ "resolved": "https://registry.npmjs.org/pg/-/pg-8.23.0.tgz",
+ "integrity": "sha512-Ip2EQCngowJLGOfCwkFhPXU7/ljlhn6Rxlmy4XYfL2Y+vyRM59+8uR2xqRWKdYmbXmxCFOAmKxBuSUCdF34qLg==",
+ "license": "MIT",
+ "dependencies": {
+ "pg-connection-string": "^2.14.0",
+ "pg-pool": "^3.14.0",
+ "pg-protocol": "^1.16.0",
+ "pg-types": "2.2.0",
+ "pgpass": "1.0.5"
+ },
+ "engines": {
+ "node": ">= 16.0.0"
+ },
+ "optionalDependencies": {
+ "pg-cloudflare": "^1.4.0"
+ },
+ "peerDependencies": {
+ "pg-native": ">=3.0.1"
+ },
+ "peerDependenciesMeta": {
+ "pg-native": {
+ "optional": true
+ }
+ }
+ },
+ "node_modules/pg-cloudflare": {
+ "version": "1.4.0",
+ "resolved": "https://registry.npmjs.org/pg-cloudflare/-/pg-cloudflare-1.4.0.tgz",
+ "integrity": "sha512-Vo7z/6rrQYxpNRylp4Tlob2elzbh+N/MOQbxFVWCxS7oEx6jF53GTJFxK2WWpKuBRkmiin4Mt+xofFDjx09R0A==",
+ "license": "MIT",
+ "optional": true
+ },
+ "node_modules/pg-connection-string": {
+ "version": "2.14.0",
+ "resolved": "https://registry.npmjs.org/pg-connection-string/-/pg-connection-string-2.14.0.tgz",
+ "integrity": "sha512-XwWDGcLRGCXAR8F/AM5bG7Q+A3Wm2s6QeEjlOKZLlH3UYcguiqCWKyWXVag5TLTIjR7oOJUY8kcADaZgWPyLeg==",
+ "license": "MIT"
+ },
+ "node_modules/pg-int8": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/pg-int8/-/pg-int8-1.0.1.tgz",
+ "integrity": "sha512-WCtabS6t3c8SkpDBUlb1kjOs7l66xsGdKpIPZsg4wR+B3+u9UAum2odSsF9tnvxg80h4ZxLWMy4pRjOsFIqQpw==",
+ "license": "ISC",
+ "engines": {
+ "node": ">=4.0.0"
+ }
+ },
+ "node_modules/pg-pool": {
+ "version": "3.14.0",
+ "resolved": "https://registry.npmjs.org/pg-pool/-/pg-pool-3.14.0.tgz",
+ "integrity": "sha512-gKtPkFdQPU3DksooVLi9LsjZxrsBUZIpa+7aVx+LV5pNh0KzP4Zleud2po+ConrxbuXGBJ6Hfer6hdgpIBpBaw==",
+ "license": "MIT",
+ "peerDependencies": {
+ "pg": ">=8.0"
+ }
+ },
+ "node_modules/pg-protocol": {
+ "version": "1.16.0",
+ "resolved": "https://registry.npmjs.org/pg-protocol/-/pg-protocol-1.16.0.tgz",
+ "integrity": "sha512-sILXutLVjCLjcDuOmvhX5e2Z4cS5qG/6Bu3VkpFwdf/633ElGLpEh9bgmuI5I4sqKqkifQiGyiCcx1HdtrK7tg==",
+ "license": "MIT"
+ },
+ "node_modules/pg-types": {
+ "version": "2.2.0",
+ "resolved": "https://registry.npmjs.org/pg-types/-/pg-types-2.2.0.tgz",
+ "integrity": "sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA==",
+ "license": "MIT",
+ "dependencies": {
+ "pg-int8": "1.0.1",
+ "postgres-array": "~2.0.0",
+ "postgres-bytea": "~1.0.0",
+ "postgres-date": "~1.0.4",
+ "postgres-interval": "^1.1.0"
+ },
+ "engines": {
+ "node": ">=4"
+ }
+ },
+ "node_modules/pgpass": {
+ "version": "1.0.5",
+ "resolved": "https://registry.npmjs.org/pgpass/-/pgpass-1.0.5.tgz",
+ "integrity": "sha512-FdW9r/jQZhSeohs1Z3sI1yxFQNFvMcnmfuj4WBMUTxOrAyLMaTcE1aAMBiTlbMNaXvBCQuVi0R7hd8udDSP7ug==",
+ "license": "MIT",
+ "dependencies": {
+ "split2": "^4.1.0"
+ }
+ },
+ "node_modules/postgres-array": {
+ "version": "2.0.0",
+ "resolved": "https://registry.npmjs.org/postgres-array/-/postgres-array-2.0.0.tgz",
+ "integrity": "sha512-VpZrUqU5A69eQyW2c5CA1jtLecCsN2U/bD6VilrFDWq5+5UIEVO7nazS3TEcHf1zuPYO/sqGvUvW62g86RXZuA==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=4"
+ }
+ },
+ "node_modules/postgres-bytea": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/postgres-bytea/-/postgres-bytea-1.0.1.tgz",
+ "integrity": "sha512-5+5HqXnsZPE65IJZSMkZtURARZelel2oXUEO8rH83VS/hxH5vv1uHquPg5wZs8yMAfdv971IU+kcPUczi7NVBQ==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/postgres-date": {
+ "version": "1.0.7",
+ "resolved": "https://registry.npmjs.org/postgres-date/-/postgres-date-1.0.7.tgz",
+ "integrity": "sha512-suDmjLVQg78nMK2UZ454hAG+OAW+HQPZ6n++TNDUX+L0+uUlLywnoxJKDou51Zm+zTCjrCl0Nq6J9C5hP9vK/Q==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/postgres-interval": {
+ "version": "1.2.0",
+ "resolved": "https://registry.npmjs.org/postgres-interval/-/postgres-interval-1.2.0.tgz",
+ "integrity": "sha512-9ZhXKM/rw350N1ovuWHbGxnGh/SNJ4cnxHiM0rxE4VN41wsg8P8zWn9hv/buK00RP4WvlOyr/RBDiptyxVbkZQ==",
+ "license": "MIT",
+ "dependencies": {
+ "xtend": "^4.0.0"
+ },
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
"node_modules/prebuild-install": {
"node_modules/prebuild-install": {
"version": "7.1.3",
"version": "7.1.3",
"resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz",
"resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz",
@@ -1354,6 +1513,15 @@
@@ -1354,6 +1513,15 @@
"simple-concat": "^1.0.0"
"simple-concat": "^1.0.0"
}
}
},
},
+ "node_modules/split2": {
+ "version": "4.2.0",
+ "resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz",
+ "integrity": "sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==",
+ "license": "ISC",
+ "engines": {
+ "node": ">= 10.x"
+ }
+ },
"node_modules/statuses": {
"node_modules/statuses": {
"version": "2.0.2",
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz",
"resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz",
@@ -1481,6 +1649,15 @@
@@ -1481,6 +1649,15 @@
"resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",
"resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",
"integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==",
"integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==",
"license": "ISC"
"license": "ISC"
+ },
+ "node_modules/xtend": {
+ "version": "4.0.2",
+ "resolved": "https://registry.npmjs.org/xtend/-/xtend-4.0.2.tgz",
+ "integrity": "sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=0.4"
+ }
}
}
}
}
}
}
@@ -7,9 +7,16 @@
@@ -7,9 +7,16 @@
"scripts": {
"scripts": {
"start": "node server/index.js",
"start": "node server/index.js",
"dev": "NODE_ENV=development node server/index.js",
"dev": "NODE_ENV=development node server/index.js",
- "test": "node test/run.js"
+ "test": "node test/run.js",
+ "test:pg": "EVENTS_TEST_STORE=pg node test/run.js"
},
},
- "keywords": ["openvibe", "events", "outbox", "webhooks", "sse"],
+ "keywords": [
+ "openvibe",
+ "events",
+ "outbox",
+ "webhooks",
+ "sse"
+ ],
"author": "OpenVibers",
"author": "OpenVibers",
"license": "AGPL-3.0-only",
"license": "AGPL-3.0-only",
"repository": "github:OpenVibers/OpenVibe.Events",
"repository": "github:OpenVibers/OpenVibe.Events",
@@ -17,11 +24,15 @@
@@ -17,11 +24,15 @@
"better-sqlite3": "^11.10.0",
"better-sqlite3": "^11.10.0",
"dotenv": "^16.4.7",
"dotenv": "^16.4.7",
"express": "^4.21.2",
"express": "^4.21.2",
- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.63.0",
- "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0",
- "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0"
+ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",
+ "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.20.4",
+ "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.28.0",
+ "pg": "^8.23.0"
},
},
"engines": {
"engines": {
"node": ">=20.0.0"
"node": ">=20.0.0"
+ },
+ "devDependencies": {
+ "@electric-sql/pglite": "^0.5.8"
}
}
}
}
new file mode 100755
new file mode 100755
@@ -0,0 +1,26 @@
@@ -0,0 +1,26 @@
+#!/usr/bin/env node
+'use strict';
+/**
+ * The one-time move of Events' SQLite database (EVENTS_DB_PATH) into its PostgreSQL schema (ADR-035; the procedure is
+ * openvibe-sdk docs/migrating-to-postgresql.md, section 6, carried out by openvibe-sdk/db runSqliteMigration).
+ *
+ * node scripts/migrate-to-postgres.js [--sqlite <file>] [--pglite] [--json]
+ *
+ * Applies migrations/ as the owner (DATABASE_DIRECT_URL), copies every table into emptied tables, verifies row counts
+ * and checksums, and exits 1 unless everything verified. The SQLite file is opened read-only. Every table keeps its
+ * name and columns.
+ */
+require('dotenv').config();
+const { runSqliteMigration } = require('openvibe-sdk/db');
+const { load } = require('../server/config');
+const { MIGRATIONS } = require('../server/store');
+
+const TABLES = {};
+
+if (require.main === module) {
+ const config = load();
+ runSqliteMigration({ service: 'events', sqlite: config.dbPath, directUrl: config.db.directUrl, migrations: MIGRATIONS, tables: TABLES })
+ .then((code) => process.exit(code), (err) => { console.error(`migrate-to-postgres failed: ${err.message}`); process.exit(1); });
+}
+
+module.exports = { TABLES };
deleted file mode 100644
deleted file mode 100644
@@ -1,175 +0,0 @@
@@ -1,175 +0,0 @@
-#!/usr/bin/env node
-/**
- * One-off backfill: redact the stored chat events of messages OpenVibe.Chat deleted before Chat
- * published chat.message.deleted (server/redaction.js does it for every deletion since).
- *
- * node scripts/redact-backfill.js --chat-db <chat.db> # dry run: counts only
- * node scripts/redact-backfill.js --chat-db <chat.db> --apply --backup <new file> # backup, then redact
- *
- * Options:
- * --chat-db <file> Chat's database, opened read-only (production: /var/lib/openvibe-chat/chat.db)
- * --db <file> the Events database (default: $EVENTS_DB_PATH, else data/events.db; relative
- * paths from the repo root; production: /var/lib/openvibe-events/events.db)
- * --batch <n> messages per write transaction (default 500, at most 1000)
- * --include-missing also redact events about messages Chat no longer has at all (hard-deleted)
- *
- * A chat event is redacted when it is a first-party `chat` event about a `chat_message` subject
- * that Chat marks deleted (is_deleted = 1, or an auto-delete time that has passed). It becomes the
- * same tombstone a live chat.message.deleted produces (payload { redacted, redacted_at,
- * redacted_by: "backfill" }, actor service:chat), at the same seq. Messages Chat does not have at
- * all are counted and left alone, unless --include-missing (Chat hard-deleted them). Idempotent: redacted events are never selected again, so a
- * second run changes nothing.
- *
- * --apply refuses to run without --backup <file>: a new file, mode 0600, an sqlite online backup of
- * the whole Events database, verified with quick_check and a row count before anything is changed.
- * That backup still holds the text being removed: delete it once the run is verified. Safe while
- * Events is up (short transactions; secure_delete zeroes the old payloads). Run as the service
- * user, with the Events release that knows redaction deployed (it adds the columns).
- */
-'use strict';
-const fs = require('fs');
-const path = require('path');
-const Database = require('better-sqlite3');
-const { createStore } = require('../server/store');
-
-const ROOT = path.join(__dirname, '..');
-const HELP_LINES = 26; // header lines 3..26
-const BY = 'backfill';
-
-function parseArgs(argv) {
- const a = { apply: false, backup: null, chatDb: null, db: null, batch: 500 };
- for (let i = 0; i < argv.length; i++) {
- const k = argv[i];
- const val = () => { if (i + 1 >= argv.length) throw new Error(`${k} needs a value`); return argv[++i]; };
- if (k === '--apply') a.apply = true;
- else if (k === '--backup') a.backup = val();
- else if (k === '--chat-db') a.chatDb = val();
- else if (k === '--db') a.db = val();
- else if (k === '--batch') a.batch = Number(val());
- else if (k === '--include-missing') a.includeMissing = true;
- else if (k === '-h' || k === '--help') a.help = true;
- else throw new Error(`unknown option ${k}`);
- }
- if (!Number.isInteger(a.batch) || a.batch < 1 || a.batch > 1000) throw new Error('--batch must be 1..1000');
- return a;
-}
-
-/** The database the server uses: --db, else $EVENTS_DB_PATH, else data/events.db (relative to the repo root). */
-function defaultDbPath(args, env = process.env) {
- return path.resolve(ROOT, args.db || env.EVENTS_DB_PATH || path.join('data', 'events.db'));
-}
-
-const mb = (n) => `${(n / 1048576).toFixed(1)} MB`;
-function fileBytes(f) { try { return fs.statSync(f).size; } catch { return 0; } }
-function freeBytes(dir) { try { const s = fs.statfsSync(dir); return s.bavail * s.bsize; } catch { return null; } }
-const chunks = (list, n) => Array.from({ length: Math.ceil(list.length / n) }, (_, i) => list.slice(i * n, (i + 1) * n));
-
-/** Unredacted first-party chat events about chat messages: [{ id, seq, subject_id }]. */
-function candidates(db) {
- return db.prepare(`SELECT id, seq, subject_id FROM events
- WHERE source = 'chat' AND subject_type = 'chat_message' AND project_id IS NULL AND redacted_at IS NULL AND redacts = 0
- ORDER BY seq`).all();
-}
-
-/** Split message ids into those Chat marks deleted and those it does not have. */
-function classify(chat, messageIds) {
- const deleted = new Set();
- const present = new Set();
- for (const part of chunks(messageIds, 500)) {
- const marks = part.map(() => '?').join(',');
- for (const r of chat.prepare(`SELECT id, is_deleted,
- (auto_delete_at IS NOT NULL AND auto_delete_at <= CURRENT_TIMESTAMP) AS expired
- FROM chat_messages WHERE id IN (${marks})`).all(...part.map(Number))) {
- present.add(String(r.id));
- if (r.is_deleted || r.expired) deleted.add(String(r.id));
- }
- }
- return { deleted: [...deleted], missing: messageIds.filter(id => !present.has(id)) };
-}
-
-async function main(argv, log = console.log) {
- let args;
- try { args = parseArgs(argv); } catch (e) { log(`error: ${e.message}`); return 2; }
- if (args.help) { log(fs.readFileSync(__filename, 'utf8').split('\n').slice(2, HELP_LINES).join('\n')); return 0; }
- if (!args.chatDb) { log('error: --chat-db <chat.db> is required'); return 2; }
- const dbPath = defaultDbPath(args);
- const chatPath = path.resolve(args.chatDb);
- for (const f of [dbPath, chatPath]) if (!fs.existsSync(f)) { log(`error: ${f} does not exist`); return 2; }
- if (args.apply && !args.backup) {
- log('refusing --apply without --backup <new file> (an sqlite online backup of the Events database)');
- return 2;
- }
-
- const db = new Database(dbPath, args.apply ? { fileMustExist: true } : { readonly: true, fileMustExist: true });
- const chat = new Database(chatPath, { readonly: true, fileMustExist: true });
- try {
- db.pragma('busy_timeout = 5000');
- chat.pragma('busy_timeout = 5000');
- const cols = new Set(db.prepare('PRAGMA table_info(events)').all().map(c => c.name));
- if (!cols.has('redacted_at') || !cols.has('redacts')) {
- log('error: this Events database has no redaction columns yet; deploy and start the Events release with redaction first');
- return 2;
- }
- const rows = candidates(db);
- const byMessage = new Map();
- for (const r of rows) {
- if (!byMessage.has(r.subject_id)) byMessage.set(r.subject_id, []);
- byMessage.get(r.subject_id).push(r);
- }
- const { deleted, missing } = classify(chat, [...byMessage.keys()]);
- // --include-missing: a message Chat no longer has at all was hard-deleted (by hand, or by its
- // expiry sweep), so its text must not outlive it here either.
- const gone = args.includeMissing ? deleted.concat(missing) : deleted;
- const toRedact = gone.flatMap(id => byMessage.get(id)).sort((x, y) => x.seq - y.seq);
- const walBytes = fileBytes(dbPath + '-wal');
- log(`events db ${dbPath} (${mb(fileBytes(dbPath))} + ${mb(walBytes)} WAL)`);
- log(`chat db ${chatPath} (read-only)`);
- log(`chat events about messages, not redacted: ${rows.length} events, ${byMessage.size} messages`);
- log(`deleted in Chat: ${deleted.length} messages, ${toRedact.length} events to redact${toRedact.length ? ` (seq ${toRedact[0].seq}..${toRedact.at(-1).seq})` : ''}`);
- log(`not in Chat's database: ${missing.length} messages${args.includeMissing ? ' (redacted: --include-missing)' : ' (left alone; --include-missing redacts them)'}`);
-
- if (!args.apply) {
- log('\ndry run: nothing changed. Re-run with --apply --backup <new file>.');
- return 0;
- }
-
- const target = path.resolve(args.backup);
- if (fs.existsSync(target)) { log(`error: backup target ${target} already exists; choose a new file`); return 2; }
- const need = fileBytes(dbPath) + walBytes;
- const free = freeBytes(path.dirname(target));
- if (free != null && free < need * 1.1) { log(`error: ${mb(free)} free at ${path.dirname(target)}, the backup needs about ${mb(need)}`); return 2; }
- const total = db.prepare('SELECT COUNT(*) FROM events').pluck().get();
- // A full copy of the event store, deleted text included: owner-only from the first byte.
- const umask = process.umask(0o077);
- try { await db.backup(target); } finally { process.umask(umask); }
- fs.chmodSync(target, 0o600);
- const b = new Database(target, { readonly: true });
- try {
- const ok = b.pragma('quick_check', { simple: true });
- const n = b.prepare('SELECT COUNT(*) FROM events').pluck().get();
- if (ok !== 'ok' || n < total) { log(`error: backup check failed (quick_check=${ok}, events=${n} of ${total}); nothing changed`); return 1; }
- } finally { b.close(); }
- log(`backup ${target} (${mb(fileBytes(target))}, mode 0600, quick_check ok). It still holds the deleted text: remove it once this run is verified.`);
-
- db.pragma('secure_delete = ON');
- const store = createStore(db);
- let redacted = 0;
- for (const part of chunks(gone, args.batch)) {
- redacted += store.redact('chat', { subject_type: 'chat_message', subject_ids: part }, { by: BY }).length;
- }
- log(`redacted ${redacted} events`);
- try { db.pragma('wal_checkpoint(TRUNCATE)'); } catch { /* the server's next checkpoint does it */ }
- const left = classify(chat, [...new Set(candidates(db).map(r => r.subject_id))]).deleted.length;
- log(`now ${left} deleted messages with unredacted events`);
- return left ? 1 : 0;
- } finally {
- chat.close();
- db.close();
- }
-}
-
-if (require.main === module) {
- main(process.argv.slice(2)).then((code) => process.exit(code), (err) => { console.error(err); process.exit(1); });
-}
-
-module.exports = { main, parseArgs, defaultDbPath };
@@ -3,7 +3,7 @@
@@ -3,7 +3,7 @@
/**
/**
* Operator rotation of a first-party subscription's secret, without the value ever being printed.
* Operator rotation of a first-party subscription's secret, without the value ever being printed.
*
*
- * node scripts/rotate-subscription-secret.js --db <events.db> (--subscription <sub_…> | --consumer <service>)
+ * node scripts/rotate-subscription-secret.js (--subscription <sub_…> | --consumer <service>) (DATABASE_URL: the Events database)
* --env-file </etc/openvibe/<consumer>.env> --env-var <NAME> [--overlap-s 86400] [--apply]
* --env-file </etc/openvibe/<consumer>.env> --env-var <NAME> [--overlap-s 86400] [--apply]
*
*
* With --apply it (1) sets a new random secret on the subscription, keeping the old one signing next to it
* With --apply it (1) sets a new random secret on the subscription, keeping the old one signing next to it
@@ -15,12 +15,13 @@
@@ -15,12 +15,13 @@
*/
*/
const fs = require('fs');
const fs = require('fs');
const crypto = require('crypto');
const crypto = require('crypto');
-const Database = require('better-sqlite3');
+require('dotenv').config();
+const { load } = require('../server/config');
+const { openDb } = require('../server/store');
const args = process.argv.slice(2);
const args = process.argv.slice(2);
const opt = (k, d) => { const i = args.indexOf(k); return i >= 0 ? args[i + 1] : d; };
const opt = (k, d) => { const i = args.indexOf(k); return i >= 0 ? args[i + 1] : d; };
const apply = args.includes('--apply');
const apply = args.includes('--apply');
-const dbPath = opt('--db');
const subId = opt('--subscription');
const subId = opt('--subscription');
const consumer = opt('--consumer'); // every first-party subscription of that service, to ONE new secret (they share one env variable)
const consumer = opt('--consumer'); // every first-party subscription of that service, to ONE new secret (they share one env variable)
const envFile = opt('--env-file');
const envFile = opt('--env-file');
@@ -28,35 +29,36 @@ const envVar = opt('--env-var');
@@ -28,35 +29,36 @@ const envVar = opt('--env-var');
const overlapS = Number(opt('--overlap-s', '86400'));
const overlapS = Number(opt('--overlap-s', '86400'));
function fail(msg) { console.error(`rotate-subscription-secret: ${msg}`); process.exit(1); }
function fail(msg) { console.error(`rotate-subscription-secret: ${msg}`); process.exit(1); }
-if (!dbPath || !(subId || consumer) || !envFile || !envVar) fail('--db, --subscription or --consumer, --env-file and --env-var are required');
+if (!(subId || consumer) || !envFile || !envVar) fail('--subscription or --consumer, --env-file and --env-var are required (the database is DATABASE_URL: PostgreSQL, ADR-035)');
if (!/^[A-Z][A-Z0-9_]{1,63}$/.test(envVar)) fail('--env-var must be an env variable name');
if (!/^[A-Z][A-Z0-9_]{1,63}$/.test(envVar)) fail('--env-var must be an env variable name');
if (!Number.isFinite(overlapS) || overlapS < 0 || overlapS > 7 * 86400) fail('--overlap-s is 0..604800');
if (!Number.isFinite(overlapS) || overlapS < 0 || overlapS > 7 * 86400) fail('--overlap-s is 0..604800');
if (!fs.existsSync(envFile)) fail(`${envFile} does not exist`);
if (!fs.existsSync(envFile)) fail(`${envFile} does not exist`);
-const db = new Database(dbPath, { fileMustExist: true });
+(async () => {
+const db = await openDb(load(), { log: { log() {}, warn: console.warn } });
const subs = consumer
const subs = consumer
- ? db.prepare('SELECT id, consumer, topic_pattern, project_id FROM subscriptions WHERE consumer = ? AND project_id IS NULL').all(consumer)
- : db.prepare('SELECT id, consumer, topic_pattern, project_id FROM subscriptions WHERE id = ?').all(subId);
+ ? await db.prepare('SELECT id, consumer, topic_pattern, project_id FROM subscriptions WHERE consumer = ? AND project_id IS NULL').all(consumer)
+ : await db.prepare('SELECT id, consumer, topic_pattern, project_id FROM subscriptions WHERE id = ?').all(subId);
if (!subs.length) fail(`no subscription for ${consumer || subId}`);
if (!subs.length) fail(`no subscription for ${consumer || subId}`);
if (subs.some((x) => x.project_id)) fail('developer-app subscriptions rotate through the API (POST /api/v1/subscriptions/:id/rotate-secret)');
if (subs.some((x) => x.project_id)) fail('developer-app subscriptions rotate through the API (POST /api/v1/subscriptions/:id/rotate-secret)');
const sub = subs[0];
const sub = subs[0];
-const cols = new Set(db.prepare('PRAGMA table_info(subscriptions)').all().map((c) => c.name));
-if (!cols.has('previous_secret')) fail('this Events database predates rotation; deploy Events first (it adds the columns at boot)');
const lines = fs.readFileSync(envFile, 'utf8').split('\n');
const lines = fs.readFileSync(envFile, 'utf8').split('\n');
const has = lines.some((l) => l.startsWith(`${envVar}=`));
const has = lines.some((l) => l.startsWith(`${envVar}=`));
console.log(`${subs.length} subscription(s) of ${sub.consumer} (${subs.map((x) => x.topic_pattern).join(', ')}); ${envFile}: ${envVar} ${has ? 'will be replaced' : 'will be appended'}; overlap ${overlapS}s`);
console.log(`${subs.length} subscription(s) of ${sub.consumer} (${subs.map((x) => x.topic_pattern).join(', ')}); ${envFile}: ${envVar} ${has ? 'will be replaced' : 'will be appended'}; overlap ${overlapS}s`);
-if (!apply) { console.log('dry run: nothing changed (add --apply)'); process.exit(0); }
+if (!apply) { console.log('dry run: nothing changed (add --apply)'); await db.close(); return; }
const secret = `whsec_${crypto.randomBytes(32).toString('hex')}`;
const secret = `whsec_${crypto.randomBytes(32).toString('hex')}`;
const now = Date.now();
const now = Date.now();
fs.copyFileSync(envFile, `${envFile}.bak-${new Date(now).toISOString().replace(/[:.]/g, '-')}`);
fs.copyFileSync(envFile, `${envFile}.bak-${new Date(now).toISOString().replace(/[:.]/g, '-')}`);
-db.transaction(() => {
+await db.tx(async () => {
const up = db.prepare('UPDATE subscriptions SET previous_secret = secret, previous_secret_until = ?, secret = ?, updated_at = ? WHERE id = ?');
const up = db.prepare('UPDATE subscriptions SET previous_secret = secret, previous_secret_until = ?, secret = ?, updated_at = ? WHERE id = ?');
- for (const x of subs) up.run(now + overlapS * 1000, secret, now, x.id);
-})();
+ for (const x of subs) await up.run(now + overlapS * 1000, secret, now, x.id);
+});
let out;
let out;
if (has) out = lines.map((l) => (l.startsWith(`${envVar}=`) ? `${envVar}=${secret}` : l));
if (has) out = lines.map((l) => (l.startsWith(`${envVar}=`) ? `${envVar}=${secret}` : l));
else { out = lines.slice(); while (out.length && out[out.length - 1] === '') out.pop(); out.push(`${envVar}=${secret}`, ''); }
else { out = lines.slice(); while (out.length && out[out.length - 1] === '') out.pop(); out.push(`${envVar}=${secret}`, ''); }
fs.writeFileSync(envFile, out.join('\n'), { mode: fs.statSync(envFile).mode });
fs.writeFileSync(envFile, out.join('\n'), { mode: fs.statSync(envFile).mode });
console.log(`rotated: both secrets sign until ${new Date(now + overlapS * 1000).toISOString()}; ${envVar} updated. Restart ${sub.consumer} now.`);
console.log(`rotated: both secrets sign until ${new Date(now + overlapS * 1000).toISOString()}; ${envVar} updated. Restart ${sub.consumer} now.`);
+await db.close();
+})().catch((err) => fail(err.message));
@@ -14,10 +14,10 @@
@@ -14,10 +14,10 @@
* Never limited: /api/health, /api/ready, /release.json, /limits.json, /metrics, and the realtime
* Never limited: /api/health, /api/ready, /release.json, /limits.json, /metrics, and the realtime
* stream (a connection, capped by REALTIME_MAX_CONNECTIONS and REALTIME_MAX_TOPICS).
* stream (a connection, capped by REALTIME_MAX_CONNECTIONS and REALTIME_MAX_TOPICS).
*/
*/
-const { createActorLimiter } = require('openvibe-sdk/limits');
+const { createActorLimiter, createValkeyLimitStore } = require('openvibe-sdk/limits');
/** limits(name, own) middleware for one app; `clock` is the service's (manual in tests). */
/** limits(name, own) middleware for one app; `clock` is the service's (manual in tests). */
-function createLimits({ config, clock = { now: () => Date.now() }, metrics = null, log = console }) {
+function createLimits({ config, clock = { now: () => Date.now() }, metrics = null, log = console, valkey = null }) {
// Every named limit with its numbers, as the routes declare them (published in /limits.json as rate_limits).
// Every named limit with its numbers, as the routes declare them (published in /limits.json as rate_limits).
const registered = new Map();
const registered = new Map();
const record = (name, own = {}) => registered.set(name, { minute: own.minute != null ? own.minute : config.limits.minute, hour: own.hour != null ? own.hour : config.limits.hour });
const record = (name, own = {}) => registered.set(name, { minute: own.minute != null ? own.minute : config.limits.minute, hour: own.hour != null ? own.hour : config.limits.hour });
@@ -33,6 +33,8 @@ function createLimits({ config, clock = { now: () => Date.now() }, metrics = nul
@@ -33,6 +33,8 @@ function createLimits({ config, clock = { now: () => Date.now() }, metrics = nul
const limiter = createActorLimiter({
const limiter = createActorLimiter({
limits: { minute: config.limits.minute, hour: config.limits.hour },
limits: { minute: config.limits.minute, hour: config.limits.hour },
now: () => clock.now(),
now: () => clock.now(),
+ // Shared across processes on Valkey (ADR-035) when VALKEY_URL is set; in-process otherwise.
+ ...(valkey ? { store: createValkeyLimitStore(valkey) } : {}),
onLimited(e) {
onLimited(e) {
// The actor is a principal id (svc:…, app:app_…), never a token.
// The actor is a principal id (svc:…, app:app_…), never a token.
log.warn(`[limits] ${e.name}: ${e.actor} refused, over ${e.limit} per ${e.window}`);
log.warn(`[limits] ${e.name}: ${e.actor} refused, over ${e.limit} per ${e.window}`);
@@ -96,24 +96,24 @@ function publishRouter({ config, store, auth, worker, realtime, limits }) {
@@ -96,24 +96,24 @@ function publishRouter({ config, store, auth, worker, realtime, limits }) {
return { env };
return { env };
}
}
- router.post('/api/v1/events', auth.appOrService(CAPS.publish, CAPS.appPublish, { requireService: true }), publishLimit, (req, res) => {
+ router.post('/api/v1/events', auth.appOrService(CAPS.publish, CAPS.appPublish, { requireService: true }), publishLimit, async (req, res) => {
const ctx = req.ov;
const ctx = req.ov;
const body = req.body;
const body = req.body;
const p = req.principal;
const p = req.principal;
// Every refusal of an app's publish is one error in its project's usage.
// Every refusal of an app's publish is one error in its project's usage.
- const refuse = (status, code, o = {}, ref = null) => {
+ const refuse = async (status, code, o = {}, ref = null) => {
if (p.kind === 'app') {
if (p.kind === 'app') {
try {
try {
- store.usage.record({ projectId: p.projectId, env: p.env, capability: 'events.app.publish', unit: 'events', error: { code, status, traceId: ctx.traceId, ref } });
+ await store.usage.record({ projectId: p.projectId, env: p.env, capability: 'events.app.publish', unit: 'events', error: { code, status, traceId: ctx.traceId, ref } });
} catch (err) { console.error(`[usage] refusal not counted: ${err.message}`); }
} catch (err) { console.error(`[usage] refusal not counted: ${err.message}`); }
}
}
return http.sendProblem(res, status, code, { ...o, ctx });
return http.sendProblem(res, status, code, { ...o, ctx });
};
};
const isBatch = body && typeof body === 'object' && !Array.isArray(body) && Array.isArray(body.events) && body.event_id === undefined;
const isBatch = body && typeof body === 'object' && !Array.isArray(body) && Array.isArray(body.events) && body.event_id === undefined;
const items = isBatch ? body.events : [body];
const items = isBatch ? body.events : [body];
- if (isBatch && !items.length) return refuse(400, 'events.bad_request', { detail: 'events must not be empty' });
+ if (isBatch && !items.length) return await refuse(400, 'events.bad_request', { detail: 'events must not be empty' });
if (items.length > config.maxBatch) {
if (items.length > config.maxBatch) {
- return refuse(413, 'events.batch_too_large', { detail: `at most ${config.maxBatch} events per request` });
+ return await refuse(413, 'events.batch_too_large', { detail: `at most ${config.maxBatch} events per request` });
}
}
const envelopes = [];
const envelopes = [];
@@ -122,10 +122,10 @@ function publishRouter({ config, store, auth, worker, realtime, limits }) {
@@ -122,10 +122,10 @@ function publishRouter({ config, store, auth, worker, realtime, limits }) {
const r = checkOne(items[i], req.principal, ctx);
const r = checkOne(items[i], req.principal, ctx);
const ref = items[i] && typeof items[i] === 'object' ? items[i].event_id : null;
const ref = items[i] && typeof items[i] === 'object' ? items[i].event_id : null;
if (r.env && seen.has(r.env.event_id)) {
if (r.env && seen.has(r.env.event_id)) {
- return refuse(422, 'events.invalid_envelope', { detail: `events[${i}]: event_id repeated within the batch`, extra: { index: i } }, ref);
+ return await refuse(422, 'events.invalid_envelope', { detail: `events[${i}]: event_id repeated within the batch`, extra: { index: i } }, ref);
}
}
if (!r.env) {
if (!r.env) {
- return refuse(r.status, r.code, {
+ return await refuse(r.status, r.code, {
detail: isBatch ? `events[${i}]: ${r.detail}` : r.detail, errors: r.errors, extra: isBatch ? { index: i } : undefined,
detail: isBatch ? `events[${i}]: ${r.detail}` : r.detail, errors: r.errors, extra: isBatch ? { index: i } : undefined,
}, ref);
}, ref);
}
}
@@ -136,11 +136,11 @@ function publishRouter({ config, store, auth, worker, realtime, limits }) {
@@ -136,11 +136,11 @@ function publishRouter({ config, store, auth, worker, realtime, limits }) {
let out;
let out;
try {
try {
const project = p.kind === 'app' ? { projectId: p.projectId, env: p.env, ...config.apps.quotas[p.env] } : null;
const project = p.kind === 'app' ? { projectId: p.projectId, env: p.env, ...config.apps.quotas[p.env] } : null;
- out = store.insertBatch(envelopes, { publisher: p.sub, requestId: ctx.requestId, project });
+ out = await store.insertBatch(envelopes, { publisher: p.sub, requestId: ctx.requestId, project });
} catch (err) {
} catch (err) {
if (err instanceof StoreError) {
if (err instanceof StoreError) {
if (err.status === 429 && err.extra && err.extra.retry_after) res.setHeader('Retry-After', String(err.extra.retry_after));
if (err.status === 429 && err.extra && err.extra.retry_after) res.setHeader('Retry-After', String(err.extra.retry_after));
- return refuse(err.status, err.code, { detail: err.message, extra: err.extra }, err.extra && err.extra.event_id);
+ return await refuse(err.status, err.code, { detail: err.message, extra: err.extra }, err.extra && err.extra.event_id);
}
}
throw err;
throw err;
}
}
@@ -59,7 +59,7 @@ function readRouter({ store, auth, worker, limits }) {
@@ -59,7 +59,7 @@ function readRouter({ store, auth, worker, limits }) {
// may store its checkpoint after each page: 600 a minute each, so a catch-up is never throttled.
// may store its checkpoint after each page: 600 a minute each, so a catch-up is never throttled.
const pull = limits('events.event.pull', { minute: 600, hour: 20000 });
const pull = limits('events.event.pull', { minute: 600, hour: 20000 });
- router.get('/api/v1/events', canRead, pull, (req, res) => {
+ router.get('/api/v1/events', canRead, pull, async (req, res) => {
const ctx = req.ov;
const ctx = req.ov;
const patterns = String(req.query.topic || '*').split(',').map(s => s.trim()).filter(Boolean);
const patterns = String(req.query.topic || '*').split(',').map(s => s.trim()).filter(Boolean);
if (!patterns.length || patterns.length > 20 || !patterns.every(topics.isValidPattern)) {
if (!patterns.length || patterns.length > 20 || !patterns.every(topics.isValidPattern)) {
@@ -72,22 +72,22 @@ function readRouter({ store, auth, worker, limits }) {
@@ -72,22 +72,22 @@ function readRouter({ store, auth, worker, limits }) {
if (Number.isNaN(after) || Number.isNaN(limit)) {
if (Number.isNaN(after) || Number.isNaN(limit)) {
return http.sendProblem(res, 400, 'events.bad_request', { detail: 'after_seq must be >= 0 and limit 1..1000', ctx });
return http.sendProblem(res, 400, 'events.bad_request', { detail: 'after_seq must be >= 0 and limit 1..1000', ctx });
}
}
- const oldest = store.oldestSeq();
+ const oldest = await store.oldestSeq();
const out = {};
const out = {};
let from = after;
let from = after;
if (after < oldest - 1) {
if (after < oldest - 1) {
out.gap = { from_seq: after + 1, to_seq: oldest - 1 };
out.gap = { from_seq: after + 1, to_seq: oldest - 1 };
from = oldest - 1;
from = oldest - 1;
}
}
- const { rows, cursor } = store.scan(from, { patterns, limit, accept: acceptFor(req.principal, patterns) });
+ const { rows, cursor } = await store.scan(from, { patterns, limit, accept: acceptFor(req.principal, patterns) });
out.events = rows.map(r => ({ seq: r.seq, event: rowToEnvelope(r) }));
out.events = rows.map(r => ({ seq: r.seq, event: rowToEnvelope(r) }));
out.next_after_seq = cursor;
out.next_after_seq = cursor;
- out.latest_seq = store.lastSeq();
+ out.latest_seq = await store.lastSeq();
res.json(out);
res.json(out);
});
});
- router.get('/api/v1/events/:id', canRead, limits('events.event.read'), (req, res) => {
- const row = store.getEvent(String(req.params.id));
+ router.get('/api/v1/events/:id', canRead, limits('events.event.read'), async (req, res) => {
+ const row = await store.getEvent(String(req.params.id));
const visible = row && (req.principal.kind === 'app' ? apps.visibleToApp(row, req.principal) : (row.env || 'production') === 'production');
const visible = row && (req.principal.kind === 'app' ? apps.visibleToApp(row, req.principal) : (row.env || 'production') === 'production');
if (!visible) return http.sendProblem(res, 404, 'events.not_found', { detail: 'no such event (or pruned by retention)', ctx: req.ov });
if (!visible) return http.sendProblem(res, 404, 'events.not_found', { detail: 'no such event (or pruned by retention)', ctx: req.ov });
return res.json({ seq: row.seq, event: rowToEnvelope(row) });
return res.json({ seq: row.seq, event: rowToEnvelope(row) });
@@ -95,27 +95,27 @@ function readRouter({ store, auth, worker, limits }) {
@@ -95,27 +95,27 @@ function readRouter({ store, auth, worker, limits }) {
const consumerOf = (req) => req.principal.service || req.principal.sub;
const consumerOf = (req) => req.principal.service || req.principal.sub;
- router.get('/api/v1/checkpoints', canRead, limits('events.checkpoint.read'), (req, res) => {
+ router.get('/api/v1/checkpoints', canRead, limits('events.checkpoint.read'), async (req, res) => {
const topic = String(req.query.topic || '');
const topic = String(req.query.topic || '');
if (!topics.isValidPattern(topic)) return http.sendProblem(res, 400, 'events.bad_topic', { detail: 'topic is required', ctx: req.ov });
if (!topics.isValidPattern(topic)) return http.sendProblem(res, 400, 'events.bad_topic', { detail: 'topic is required', ctx: req.ov });
const scopeErr = scopeError(req.principal, [topic]);
const scopeErr = scopeError(req.principal, [topic]);
if (scopeErr) return http.sendProblem(res, 403, 'events.topic_not_allowed', { detail: scopeErr, ctx: req.ov });
if (scopeErr) return http.sendProblem(res, 403, 'events.topic_not_allowed', { detail: scopeErr, ctx: req.ov });
- const cp = store.getCheckpoint(consumerOf(req), topic);
+ const cp = await store.getCheckpoint(consumerOf(req), topic);
res.json({ consumer: consumerOf(req), topic, cursor: cp ? cp.cursor : 0, updated_at: cp ? cp.updated_at : null });
res.json({ consumer: consumerOf(req), topic, cursor: cp ? cp.cursor : 0, updated_at: cp ? cp.updated_at : null });
});
});
- router.put('/api/v1/checkpoints', canRead, limits('events.checkpoint.write', { minute: 600, hour: 20000 }), (req, res) => {
+ router.put('/api/v1/checkpoints', canRead, limits('events.checkpoint.write', { minute: 600, hour: 20000 }), async (req, res) => {
const b = req.body || {};
const b = req.body || {};
if (!topics.isValidPattern(b.topic) || !Number.isInteger(b.cursor) || b.cursor < 0) {
if (!topics.isValidPattern(b.topic) || !Number.isInteger(b.cursor) || b.cursor < 0) {
return http.sendProblem(res, 400, 'events.bad_request', { detail: 'topic (pattern) and cursor (integer >= 0) are required', ctx: req.ov });
return http.sendProblem(res, 400, 'events.bad_request', { detail: 'topic (pattern) and cursor (integer >= 0) are required', ctx: req.ov });
}
}
const scopeErr = scopeError(req.principal, [b.topic]);
const scopeErr = scopeError(req.principal, [b.topic]);
if (scopeErr) return http.sendProblem(res, 403, 'events.topic_not_allowed', { detail: scopeErr, ctx: req.ov });
if (scopeErr) return http.sendProblem(res, 403, 'events.topic_not_allowed', { detail: scopeErr, ctx: req.ov });
- const cp = store.setCheckpoint(consumerOf(req), b.topic, b.cursor);
+ const cp = await store.setCheckpoint(consumerOf(req), b.topic, b.cursor);
res.json({ consumer: consumerOf(req), topic: b.topic, ...cp });
res.json({ consumer: consumerOf(req), topic: b.topic, ...cp });
});
});
- router.get('/api/v1/deliveries', isAdmin, limits('events.delivery.list'), (req, res) => {
+ router.get('/api/v1/deliveries', isAdmin, limits('events.delivery.list'), async (req, res) => {
const status = req.query.status ? String(req.query.status) : undefined;
const status = req.query.status ? String(req.query.status) : undefined;
if (status && !['pending', 'delivered', 'failed', 'dead'].includes(status)) {
if (status && !['pending', 'delivered', 'failed', 'dead'].includes(status)) {
return http.sendProblem(res, 400, 'events.bad_request', { detail: 'status is pending|delivered|failed|dead', ctx: req.ov });
return http.sendProblem(res, 400, 'events.bad_request', { detail: 'status is pending|delivered|failed|dead', ctx: req.ov });
@@ -123,7 +123,7 @@ function readRouter({ store, auth, worker, limits }) {
@@ -123,7 +123,7 @@ function readRouter({ store, auth, worker, limits }) {
const limit = intParam(req.query.limit, 100, 1, 1000);
const limit = intParam(req.query.limit, 100, 1, 1000);
const afterSeq = intParam(req.query.after_seq, 0, 0, Number.MAX_SAFE_INTEGER);
const afterSeq = intParam(req.query.after_seq, 0, 0, Number.MAX_SAFE_INTEGER);
if (Number.isNaN(limit) || Number.isNaN(afterSeq)) return http.sendProblem(res, 400, 'events.bad_request', { detail: 'bad limit or after_seq', ctx: req.ov });
if (Number.isNaN(limit) || Number.isNaN(afterSeq)) return http.sendProblem(res, 400, 'events.bad_request', { detail: 'bad limit or after_seq', ctx: req.ov });
- const rows = store.listDeliveries({ status, subscriptionId: req.query.subscription_id ? String(req.query.subscription_id) : undefined, limit, afterSeq });
+ const rows = await store.listDeliveries({ status, subscriptionId: req.query.subscription_id ? String(req.query.subscription_id) : undefined, limit, afterSeq });
return res.json({
return res.json({
deliveries: rows.map(d => ({
deliveries: rows.map(d => ({
event_id: d.event_id,
event_id: d.event_id,
@@ -136,16 +136,16 @@ function readRouter({ store, auth, worker, limits }) {
@@ -136,16 +136,16 @@ function readRouter({ store, auth, worker, limits }) {
last_error: d.last_error,
last_error: d.last_error,
delivered_at: d.delivered_at ? new Date(d.delivered_at).toISOString() : null,
delivered_at: d.delivered_at ? new Date(d.delivered_at).toISOString() : null,
})),
})),
- counts: store.deliveryCounts(),
+ counts: await store.deliveryCounts(),
});
});
});
});
// A replay requeues up to a whole subscription's retained history: an operator's action, a few
// A replay requeues up to a whole subscription's retained history: an operator's action, a few
// times at most.
// times at most.
- router.post('/api/v1/deliveries/replay', isAdmin, limits('events.delivery.replay', { minute: 6, hour: 60 }), (req, res) => {
+ router.post('/api/v1/deliveries/replay', isAdmin, limits('events.delivery.replay', { minute: 6, hour: 60 }), async (req, res) => {
const ctx = req.ov;
const ctx = req.ov;
const b = req.body || {};
const b = req.body || {};
- const sub = typeof b.subscription_id === 'string' ? store.getSubscription(b.subscription_id) : null;
+ const sub = typeof b.subscription_id === 'string' ? await store.getSubscription(b.subscription_id) : null;
if (!sub) return http.sendProblem(res, 404, 'events.not_found', { detail: 'no such subscription', ctx });
if (!sub) return http.sendProblem(res, 404, 'events.not_found', { detail: 'no such subscription', ctx });
const hasIds = Array.isArray(b.event_ids);
const hasIds = Array.isArray(b.event_ids);
const hasFrom = Number.isInteger(b.from_seq) && b.from_seq >= 0;
const hasFrom = Number.isInteger(b.from_seq) && b.from_seq >= 0;
@@ -153,7 +153,7 @@ function readRouter({ store, auth, worker, limits }) {
@@ -153,7 +153,7 @@ function readRouter({ store, auth, worker, limits }) {
if (hasIds && (b.event_ids.length > 1000 || !b.event_ids.every(id => typeof id === 'string'))) {
if (hasIds && (b.event_ids.length > 1000 || !b.event_ids.every(id => typeof id === 'string'))) {
return http.sendProblem(res, 400, 'events.bad_request', { detail: 'event_ids: at most 1000 ids', ctx });
return http.sendProblem(res, 400, 'events.bad_request', { detail: 'event_ids: at most 1000 ids', ctx });
}
}
- const queued = store.requeue(sub, hasIds ? { eventIds: b.event_ids } : { fromSeq: b.from_seq });
+ const queued = await store.requeue(sub, hasIds ? { eventIds: b.event_ids } : { fromSeq: b.from_seq });
worker.kick();
worker.kick();
return res.json({ subscription_id: sub.id, queued });
return res.json({ subscription_id: sub.id, queued });
});
});
@@ -57,10 +57,10 @@ function subscriptionsRouter({ config, store, auth, dnsLookup, limits }) {
@@ -57,10 +57,10 @@ function subscriptionsRouter({ config, store, auth, dnsLookup, limits }) {
const change = { minute: 30, hour: 300 };
const change = { minute: 30, hour: 300 };
/** The app's events.app.subscribe grant was withdrawn after this token was issued. */
/** The app's events.app.subscribe grant was withdrawn after this token was issued. */
- function subscribeRevoked(req, res) {
+ async function subscribeRevoked(req, res) {
const p = req.principal;
const p = req.principal;
if (p.kind !== 'app') return false;
if (p.kind !== 'app') return false;
- const at = store.revokedAt(p.sub, CAPS.appSubscribe);
+ const at = await store.revokedAt(p.sub, CAPS.appSubscribe);
if (at && p.iat * 1000 <= at) {
if (at && p.iat * 1000 <= at) {
http.sendProblem(res, 403, 'capability.denied', { detail: `${CAPS.appSubscribe} was revoked after this token was issued`, ctx: req.ov });
http.sendProblem(res, 403, 'capability.denied', { detail: `${CAPS.appSubscribe} was revoked after this token was issued`, ctx: req.ov });
return true;
return true;
@@ -71,8 +71,8 @@ function subscriptionsRouter({ config, store, auth, dnsLookup, limits }) {
@@ -71,8 +71,8 @@ function subscriptionsRouter({ config, store, auth, dnsLookup, limits }) {
/** An app's subscriptions are those of its token's project and environment too (never across sandbox/production). */
/** An app's subscriptions are those of its token's project and environment too (never across sandbox/production). */
const inScope = (req, sub) => req.principal.kind !== 'app' || (sub.project_id === req.principal.projectId && (sub.env || 'production') === req.principal.env);
const inScope = (req, sub) => req.principal.kind !== 'app' || (sub.project_id === req.principal.projectId && (sub.env || 'production') === req.principal.env);
- function own(req, res) {
- const sub = store.getSubscription(String(req.params.id));
+ async function own(req, res) {
+ const sub = await store.getSubscription(String(req.params.id));
if (!sub || sub.consumer !== consumerOf(req) || !inScope(req, sub)) {
if (!sub || sub.consumer !== consumerOf(req) || !inScope(req, sub)) {
http.sendProblem(res, 404, 'events.not_found', { detail: 'no such subscription', ctx: req.ov });
http.sendProblem(res, 404, 'events.not_found', { detail: 'no such subscription', ctx: req.ov });
return null;
return null;
@@ -96,7 +96,7 @@ function subscriptionsRouter({ config, store, auth, dnsLookup, limits }) {
@@ -96,7 +96,7 @@ function subscriptionsRouter({ config, store, auth, dnsLookup, limits }) {
return http.sendProblem(res, 422, 'events.bad_topic', { detail: 'topic_pattern must be dot-separated segments of [a-z0-9_] or *', ctx });
return http.sendProblem(res, 422, 'events.bad_topic', { detail: 'topic_pattern must be dot-separated segments of [a-z0-9_] or *', ctx });
}
}
const app = req.principal.kind === 'app' ? req.principal : null;
const app = req.principal.kind === 'app' ? req.principal : null;
- if (subscribeRevoked(req, res)) return undefined;
+ if (await subscribeRevoked(req, res)) return undefined;
if (app) {
if (app) {
const err = apps.patternScopeError(pattern, app);
const err = apps.patternScopeError(pattern, app);
if (err) return http.sendProblem(res, 403, 'events.topic_not_allowed', { detail: err, ctx });
if (err) return http.sendProblem(res, 403, 'events.topic_not_allowed', { detail: err, ctx });
@@ -111,22 +111,22 @@ function subscriptionsRouter({ config, store, auth, dnsLookup, limits }) {
@@ -111,22 +111,22 @@ function subscriptionsRouter({ config, store, auth, dnsLookup, limits }) {
const consumer = consumerOf(req);
const consumer = consumerOf(req);
const endpoint = ep.url.toString();
const endpoint = ep.url.toString();
- const dup = store.listSubscriptions(consumer).find(s => s.topic_pattern === pattern && s.endpoint === endpoint);
+ const dup = (await store.listSubscriptions(consumer)).find(s => s.topic_pattern === pattern && s.endpoint === endpoint);
if (dup) {
if (dup) {
return http.sendProblem(res, 409, 'events.subscription_exists', { detail: 'this consumer already subscribes that endpoint to that topic', ctx, extra: { subscription_id: dup.id } });
return http.sendProblem(res, 409, 'events.subscription_exists', { detail: 'this consumer already subscribes that endpoint to that topic', ctx, extra: { subscription_id: dup.id } });
}
}
- if (store.countSubscriptions(consumer) >= config.maxSubscriptionsPerConsumer) {
+ if (await store.countSubscriptions(consumer) >= config.maxSubscriptionsPerConsumer) {
return http.sendProblem(res, 409, 'events.subscription_limit', { detail: `at most ${config.maxSubscriptionsPerConsumer} subscriptions per consumer`, ctx });
return http.sendProblem(res, 409, 'events.subscription_limit', { detail: `at most ${config.maxSubscriptionsPerConsumer} subscriptions per consumer`, ctx });
}
}
if (app) {
if (app) {
const max = config.apps.quotas[app.env].maxSubscriptions;
const max = config.apps.quotas[app.env].maxSubscriptions;
- if (max && store.countProjectSubscriptions(app.projectId, app.env) >= max) {
+ if (max && await store.countProjectSubscriptions(app.projectId, app.env) >= max) {
return http.sendProblem(res, 429, 'events.quota_exceeded', {
return http.sendProblem(res, 429, 'events.quota_exceeded', {
detail: `at most ${max} subscriptions per project in ${app.env}`, ctx, extra: { quota: 'subscriptions', limit: max },
detail: `at most ${max} subscriptions per project in ${app.env}`, ctx, extra: { quota: 'subscriptions', limit: max },
});
});
}
}
}
}
- const row = store.createSubscription({
+ const row = await store.createSubscription({
id: `sub_${ids.ulid()}`,
id: `sub_${ids.ulid()}`,
consumer,
consumer,
topicPattern: pattern,
topicPattern: pattern,
@@ -139,20 +139,20 @@ function subscriptionsRouter({ config, store, auth, dnsLookup, limits }) {
@@ -139,20 +139,20 @@ function subscriptionsRouter({ config, store, auth, dnsLookup, limits }) {
return res.status(201).json(subscriptionView(row, { withSecret: true }));
return res.status(201).json(subscriptionView(row, { withSecret: true }));
}
}
- router.get('/api/v1/subscriptions', guard, limits('events.subscription.list'), (req, res) => {
- res.json({ subscriptions: store.listSubscriptions(consumerOf(req)).filter(s => inScope(req, s)).map(s => subscriptionView(s)) });
+ router.get('/api/v1/subscriptions', guard, limits('events.subscription.list'), async (req, res) => {
+ res.json({ subscriptions: (await store.listSubscriptions(consumerOf(req))).filter(s => inScope(req, s)).map(s => subscriptionView(s)) });
});
});
- router.get('/api/v1/subscriptions/:id', guard, limits('events.subscription.read'), (req, res) => {
- const sub = own(req, res);
+ router.get('/api/v1/subscriptions/:id', guard, limits('events.subscription.read'), async (req, res) => {
+ const sub = await own(req, res);
if (sub) res.json(subscriptionView(sub));
if (sub) res.json(subscriptionView(sub));
});
});
// POST /api/v1/subscriptions/:id/rotate-secret { secret?, overlap_s? } -> the new secret (shown once). The old
// POST /api/v1/subscriptions/:id/rotate-secret { secret?, overlap_s? } -> the new secret (shown once). The old
// secret keeps signing next to it for overlap_s (default 1 day, at most 7), so the consumer can switch without
// secret keeps signing next to it for overlap_s (default 1 day, at most 7), so the consumer can switch without
// missing a delivery.
// missing a delivery.
- router.post('/api/v1/subscriptions/:id/rotate-secret', guard, limits('events.subscription.rotate', { minute: 10, hour: 100 }), express.json({ limit: '4kb' }), (req, res) => {
- const sub = own(req, res);
+ router.post('/api/v1/subscriptions/:id/rotate-secret', guard, limits('events.subscription.rotate', { minute: 10, hour: 100 }), express.json({ limit: '4kb' }), async (req, res) => {
+ const sub = await own(req, res);
if (!sub) return;
if (!sub) return;
const b = req.body || {};
const b = req.body || {};
if (b.secret !== undefined && (typeof b.secret !== 'string' || b.secret.length < 32 || b.secret.length > 256)) {
if (b.secret !== undefined && (typeof b.secret !== 'string' || b.secret.length < 32 || b.secret.length > 256)) {
@@ -163,16 +163,16 @@ function subscriptionsRouter({ config, store, auth, dnsLookup, limits }) {
@@ -163,16 +163,16 @@ function subscriptionsRouter({ config, store, auth, dnsLookup, limits }) {
return http.sendProblem(res, 422, 'events.bad_request', { detail: 'overlap_s is 0..604800', ctx: req.ov });
return http.sendProblem(res, 422, 'events.bad_request', { detail: 'overlap_s is 0..604800', ctx: req.ov });
}
}
const secret = b.secret || `whsec_${crypto.randomBytes(32).toString('hex')}`;
const secret = b.secret || `whsec_${crypto.randomBytes(32).toString('hex')}`;
- const out = store.rotateSubscriptionSecret(sub.id, secret, overlapS * 1000);
+ const out = await store.rotateSubscriptionSecret(sub.id, secret, overlapS * 1000);
res.json({ ...subscriptionView(out), secret, previous_secret_valid_until: new Date(out.previous_secret_until).toISOString() });
res.json({ ...subscriptionView(out), secret, previous_secret_valid_until: new Date(out.previous_secret_until).toISOString() });
});
});
const toggle = limits('events.subscription.toggle', change);
const toggle = limits('events.subscription.toggle', change);
for (const [action, enabled] of [['disable', false], ['enable', true]]) {
for (const [action, enabled] of [['disable', false], ['enable', true]]) {
- router.post(`/api/v1/subscriptions/:id/${action}`, guard, toggle, (req, res) => {
- if (enabled && subscribeRevoked(req, res)) return;
- const sub = own(req, res);
- if (sub) res.json(subscriptionView(store.setSubscriptionEnabled(sub.id, enabled)));
+ router.post(`/api/v1/subscriptions/:id/${action}`, guard, toggle, async (req, res) => {
+ if (enabled && await subscribeRevoked(req, res)) return;
+ const sub = await own(req, res);
+ if (sub) res.json(subscriptionView(await store.setSubscriptionEnabled(sub.id, enabled)));
});
});
}
}
@@ -13,7 +13,7 @@ const { mountLimits } = require('./limits');
@@ -13,7 +13,7 @@ const { mountLimits } = require('./limits');
const { createLimits } = require('./actor-limits');
const { createLimits } = require('./actor-limits');
const pkg = require('../package.json');
const pkg = require('../package.json');
-function createApp({ config, store, auth, keys, worker, realtime, metrics, dnsLookup, clock, log = console }) {
+function createApp({ config, store, auth, keys, worker, realtime, metrics, dnsLookup, clock, log = console, valkey = null }) {
const app = express();
const app = express();
app.disable('x-powered-by');
app.disable('x-powered-by');
app.set('trust proxy', 'loopback');
app.set('trust proxy', 'loopback');
@@ -44,13 +44,15 @@ function createApp({ config, store, auth, keys, worker, realtime, metrics, dnsLo
@@ -44,13 +44,15 @@ function createApp({ config, store, auth, keys, worker, realtime, metrics, dnsLo
// Readiness (openvibe-shared/ready): 503 only when a required check fails. A DLQ past
// Readiness (openvibe-shared/ready): 503 only when a required check fails. A DLQ past
// EVENTS_DLQ_DEGRADED_AT degrades the service (still ready: new events are accepted and delivered).
// EVENTS_DLQ_DEGRADED_AT degrades the service (still ready: new events are accepted and delivered).
const checks = [
const checks = [
- { name: 'db', required: true, check: () => store.ping() },
+ // A real round trip that names the store (postgresql / pglite).
+ { name: 'db', required: true, check: async () => { const r = await store.db.ready(); return r.ok ? { ok: true, detail: r.detail } : r.error; } },
+ { name: 'valkey', required: false, check: async () => (valkey ? valkey.ready() : { skipped: 'VALKEY_URL not set: per-actor limits count in this process only' }) },
{ name: 'network_jwks', required: true, check: () => keys.loaded() || 'Network signing key not loaded yet' },
{ name: 'network_jwks', required: true, check: () => keys.loaded() || 'Network signing key not loaded yet' },
];
];
if (config.worker.enabled) checks.push({ name: 'delivery_worker', required: true, check: () => worker.running() || 'delivery worker is not running' });
if (config.worker.enabled) checks.push({ name: 'delivery_worker', required: true, check: () => worker.running() || 'delivery worker is not running' });
checks.push({
checks.push({
- name: 'dlq', required: false, check: () => {
- const depth = store.deliveryCounts().dead;
+ name: 'dlq', required: false, check: async () => {
+ const depth = (await store.deliveryCounts()).dead;
return depth > config.dlqDegradedAt
return depth > config.dlqDegradedAt
? { ok: false, error: `${depth} dead deliveries (threshold ${config.dlqDegradedAt})`, detail: { depth, threshold: config.dlqDegradedAt } }
? { ok: false, error: `${depth} dead deliveries (threshold ${config.dlqDegradedAt})`, detail: { depth, threshold: config.dlqDegradedAt } }
: { ok: true, detail: { depth, threshold: config.dlqDegradedAt } };
: { ok: true, detail: { depth, threshold: config.dlqDegradedAt } };
@@ -58,11 +60,11 @@ function createApp({ config, store, auth, keys, worker, realtime, metrics, dnsLo
@@ -58,11 +60,11 @@ function createApp({ config, store, auth, keys, worker, realtime, metrics, dnsLo
});
});
const readiness = createReadiness({
const readiness = createReadiness({
service: 'events', release: release.release, checks,
service: 'events', release: release.release, checks,
- details: (body) => {
+ details: async (body) => {
const dbOk = body.checks.db.status === 'ok';
const dbOk = body.checks.db.status === 'ok';
return {
return {
- latest_seq: dbOk ? store.lastSeq() : null,
- deliveries: dbOk ? store.deliveryCounts() : null,
+ latest_seq: dbOk ? await store.lastSeq() : null,
+ deliveries: dbOk ? await store.deliveryCounts() : null,
worker: { enabled: config.worker.enabled, ...worker.stats() },
worker: { enabled: config.worker.enabled, ...worker.stats() },
realtime_connections: realtime.count(),
realtime_connections: realtime.count(),
};
};
@@ -72,7 +74,7 @@ function createApp({ config, store, auth, keys, worker, realtime, metrics, dnsLo
@@ -72,7 +74,7 @@ function createApp({ config, store, auth, keys, worker, realtime, metrics, dnsLo
// One per-actor limiter for the capability routes below (server/actor-limits.js); health, ready,
// One per-actor limiter for the capability routes below (server/actor-limits.js); health, ready,
// release.json, limits.json, metrics and the realtime stream above are never limited.
// release.json, limits.json, metrics and the realtime stream above are never limited.
- const limits = createLimits({ config, clock, metrics, log });
+ const limits = createLimits({ config, clock, metrics, log, valkey });
app.use(publishRouter({ config, store, auth, worker, realtime, limits }));
app.use(publishRouter({ config, store, auth, worker, realtime, limits }));
app.use(subscriptionsRouter({ config, store, auth, dnsLookup, limits }));
app.use(subscriptionsRouter({ config, store, auth, dnsLookup, limits }));
app.use(readRouter({ store, auth, worker, limits }));
app.use(readRouter({ store, auth, worker, limits }));
@@ -72,7 +72,7 @@ function createKeyStore({ urls = [], pem = null, fetchImpl = globalThis.fetch, l
@@ -72,7 +72,7 @@ function createKeyStore({ urls = [], pem = null, fetchImpl = globalThis.fetch, l
return null;
return null;
}
}
- function start() {
+ async function start() {
if (pem) return Promise.resolve(key);
if (pem) return Promise.resolve(key);
const attempt = async () => {
const attempt = async () => {
const k = await fetchOnce();
const k = await fetchOnce();
@@ -84,7 +84,7 @@ function createKeyStore({ urls = [], pem = null, fetchImpl = globalThis.fetch, l
@@ -84,7 +84,7 @@ function createKeyStore({ urls = [], pem = null, fetchImpl = globalThis.fetch, l
};
};
refreshTimer = setInterval(() => { fetchOnce().catch(() => {}); }, 6 * 60 * 60 * 1000);
refreshTimer = setInterval(() => { fetchOnce().catch(() => {}); }, 6 * 60 * 60 * 1000);
refreshTimer.unref?.();
refreshTimer.unref?.();
- return attempt();
+ return await attempt();
}
}
function stop() {
function stop() {
@@ -259,7 +259,7 @@ function createAuth({ config, keys, store = null }) {
@@ -259,7 +259,7 @@ function createAuth({ config, keys, store = null }) {
*/
*/
function appOrService(serviceCap, appCap, { requireService = false } = {}) {
function appOrService(serviceCap, appCap, { requireService = false } = {}) {
const serviceGuard = requireCap(serviceCap, { requireService });
const serviceGuard = requireCap(serviceCap, { requireService });
- return function appOrServiceGuard(req, res, next) {
+ return async function appOrServiceGuard(req, res, next) {
const ctx = req.ov;
const ctx = req.ov;
const token = bearer(req);
const token = bearer(req);
if (!token || !config.apps.enabled) return serviceGuard(req, res, next);
if (!token || !config.apps.enabled) return serviceGuard(req, res, next);
@@ -268,7 +268,7 @@ function createAuth({ config, keys, store = null }) {
@@ -268,7 +268,7 @@ function createAuth({ config, keys, store = null }) {
const principal = apps.appPrincipal(r.claims);
const principal = apps.appPrincipal(r.claims);
if (principal.error) return http.sendProblem(res, 401, 'token.invalid_claims', { detail: principal.error, ctx });
if (principal.error) return http.sendProblem(res, 401, 'token.invalid_claims', { detail: principal.error, ctx });
// Network revoked this app after the token was issued (network.app.revoked reached us).
// Network revoked this app after the token was issued (network.app.revoked reached us).
- const revoked = store && store.revokedAt(principal.sub, 'app');
+ const revoked = store && await store.revokedAt(principal.sub, 'app');
if (revoked && r.claims.iat * 1000 <= revoked) return http.sendProblem(res, 401, 'token.revoked', { detail: 'this app was revoked', ctx });
if (revoked && r.claims.iat * 1000 <= revoked) return http.sendProblem(res, 401, 'token.revoked', { detail: 'this app was revoked', ctx });
const c = allows(r.claims, appCap);
const c = allows(r.claims, appCap);
if (!c.allowed) return http.sendProblem(res, 403, c.code, { detail: c.reason, ctx });
if (!c.allowed) return http.sendProblem(res, 403, c.code, { detail: c.reason, ctx });
@@ -64,6 +64,11 @@ function load(env = process.env) {
@@ -64,6 +64,11 @@ function load(env = process.env) {
// openvibe.events user tokens yet; the network-wide audience is the stable one).
// openvibe.events user tokens yet; the network-wide audience is the stable one).
userAudiences: list(env.EVENTS_USER_AUDIENCES, ['openvibe.events', 'openvibe.network']),
userAudiences: list(env.EVENTS_USER_AUDIENCES, ['openvibe.events', 'openvibe.network']),
+ // PostgreSQL (ADR-035): DATABASE_URL serves (PgBouncer), DATABASE_DIRECT_URL migrates (owner role).
+ db: { url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '' },
+ // Valkey (ADR-035): per-actor limit counters shared across processes; without it they count in this process.
+ valkey: { url: env.VALKEY_URL || '', prefix: env.VALKEY_PREFIX || 'ov:events:' },
+ // The SQLite file of releases before the switch: read once by scripts/migrate-to-postgres.js.
dbPath: env.EVENTS_DB_PATH || './data/events.db',
dbPath: env.EVENTS_DB_PATH || './data/events.db',
retentionDays: int(env.EVENTS_RETENTION_DAYS, 30),
retentionDays: int(env.EVENTS_RETENTION_DAYS, 30),
receiptRetentionDays: int(env.EVENTS_RECEIPT_RETENTION_DAYS, 90),
receiptRetentionDays: int(env.EVENTS_RECEIPT_RETENTION_DAYS, 90),
@@ -17,11 +17,12 @@ const { createMetrics } = require('./metrics');
@@ -17,11 +17,12 @@ const { createMetrics } = require('./metrics');
const { createGuardedPost } = require('./egress');
const { createGuardedPost } = require('./egress');
async function start({
async function start({
- config, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, deliveryFetch = fetchImpl,
+ config, db: givenDb = null, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, deliveryFetch = fetchImpl,
appPost = undefined, dnsLookup = undefined, log = console, listen = true,
appPost = undefined, dnsLookup = undefined, log = console, listen = true,
} = {}) {
} = {}) {
config = config || load();
config = config || load();
- const db = openDb(config.dbPath);
+ // PostgreSQL (ADR-035): opened and migrated here unless the caller (a test) hands in a migrated handle.
+ const db = givenDb || await openDb(config, { log });
const store = createStore(db, { clock, maxHops: config.maxHops, usage: config.usage });
const store = createStore(db, { clock, maxHops: config.maxHops, usage: config.usage });
const keys = createKeyStore({ urls: [config.networkInternalUrl, config.networkUrl], pem: config.networkPublicKey, fetchImpl, log });
const keys = createKeyStore({ urls: [config.networkInternalUrl, config.networkUrl], pem: config.networkPublicKey, fetchImpl, log });
const auth = createAuth({ config, keys, store });
const auth = createAuth({ config, keys, store });
@@ -31,16 +32,17 @@ async function start({
@@ -31,16 +32,17 @@ async function start({
const worker = createWorker({ store, config, clock, fetchImpl: deliveryFetch, appPost: appPost || createGuardedPost({ lookup: dnsLookup }), log, observe: metrics.observe });
const worker = createWorker({ store, config, clock, fetchImpl: deliveryFetch, appPost: appPost || createGuardedPost({ lookup: dnsLookup }), log, observe: metrics.observe });
const realtime = createRealtime({ store, auth, config, clock, log });
const realtime = createRealtime({ store, auth, config, clock, log });
metrics.bind({ realtime });
metrics.bind({ realtime });
- const app = createApp({ config, store, auth, keys, worker, realtime, metrics, dnsLookup, clock, log });
+ const valkey = config.valkey.url ? require('openvibe-sdk/valkey').createValkey({ url: config.valkey.url, prefix: config.valkey.prefix, log }) : null;
+ const app = createApp({ config, store, auth, keys, worker, realtime, metrics, dnsLookup, clock, log, valkey });
// The key loads in the background (retrying while Network boots); /api/ready says when it has.
// The key loads in the background (retrying while Network boots); /api/ready says when it has.
const keyLoaded = keys.start().catch(() => null);
const keyLoaded = keys.start().catch(() => null);
if (config.worker.enabled) worker.start();
if (config.worker.enabled) worker.start();
realtime.start();
realtime.start();
- const prune = () => {
+ const prune = async () => {
try {
try {
- const r = store.prune({ retentionDays: config.retentionDays, receiptRetentionDays: config.receiptRetentionDays, sandboxRetentionDays: config.apps.sandboxRetentionDays });
+ const r = await store.prune({ retentionDays: config.retentionDays, receiptRetentionDays: config.receiptRetentionDays, sandboxRetentionDays: config.apps.sandboxRetentionDays });
if (r.events || r.receipts) log.log(`[retention] pruned ${r.events} events, ${r.receipts} receipts`);
if (r.events || r.receipts) log.log(`[retention] pruned ${r.events} events, ${r.receipts} receipts`);
} catch (err) {
} catch (err) {
log.error(`[retention] prune failed: ${err.message}`);
log.error(`[retention] prune failed: ${err.message}`);
@@ -48,13 +50,13 @@ async function start({
@@ -48,13 +50,13 @@ async function start({
};
};
const pruneTimer = setInterval(prune, config.pruneIntervalMs);
const pruneTimer = setInterval(prune, config.pruneIntervalMs);
pruneTimer.unref?.();
pruneTimer.unref?.();
- if (config.worker.enabled) prune();
+ if (config.worker.enabled) await prune();
// Project usage rollups (server/usage.js): each closed hour is stored as events.usage.recorded and
// Project usage rollups (server/usage.js): each closed hour is stored as events.usage.recorded and
// fanned out like a published event.
// fanned out like a published event.
- const flushUsage = (opts) => {
+ const flushUsage = async (opts) => {
try {
try {
- const r = store.flushUsage(opts);
+ const r = await store.flushUsage(opts);
if (r.stored.length) {
if (r.stored.length) {
realtime.publish(r.stored);
realtime.publish(r.stored);
worker.kick();
worker.kick();
@@ -91,7 +93,8 @@ async function start({
@@ -91,7 +93,8 @@ async function start({
server.closeAllConnections?.();
server.closeAllConnections?.();
await new Promise(resolve => server.close(() => resolve()));
await new Promise(resolve => server.close(() => resolve()));
}
}
- db.close();
+ if (valkey) await valkey.close();
+ if (!givenDb) await db.close();
}
}
return { config, db, store, keys, keyLoaded, auth, worker, realtime, metrics, app, server, close, prune, flushUsage };
return { config, db, store, keys, keyLoaded, auth, worker, realtime, metrics, app, server, close, prune, flushUsage };
@@ -22,10 +22,10 @@ function createMetrics({ store }) {
@@ -22,10 +22,10 @@ function createMetrics({ store }) {
registry.gauge({
registry.gauge({
name: 'events_deliveries', help: 'Deliveries by status', labelNames: ['status'],
name: 'events_deliveries', help: 'Deliveries by status', labelNames: ['status'],
- collect: () => Object.entries(store.deliveryCounts()).map(([status, value]) => ({ labels: { status }, value })),
+ collect: async () => Object.entries(await store.deliveryCounts()).map(([status, value]) => ({ labels: { status }, value })),
});
});
- registry.gauge({ name: 'events_dlq_depth', help: 'Dead deliveries (the DLQ) waiting for replay', collect: () => store.deliveryCounts().dead });
- registry.gauge({ name: 'events_latest_seq', help: 'Last sequence number assigned to an event', collect: () => store.lastSeq() });
+ registry.gauge({ name: 'events_dlq_depth', help: 'Dead deliveries (the DLQ) waiting for replay', collect: async () => (await store.deliveryCounts()).dead });
+ registry.gauge({ name: 'events_latest_seq', help: 'Last sequence number assigned to an event', collect: async () => await store.lastSeq() });
registry.gauge({ name: 'events_realtime_connections', help: 'Open realtime (SSE) connections', collect: () => (realtime ? realtime.count() : undefined) });
registry.gauge({ name: 'events_realtime_connections', help: 'Open realtime (SSE) connections', collect: () => (realtime ? realtime.count() : undefined) });
const latency = registry.histogram({ name: 'events_delivery_latency_seconds', help: 'Time from acceptance to successful delivery, retries included', labelNames: ['priority'], buckets: LATENCY_BUCKETS });
const latency = registry.histogram({ name: 'events_delivery_latency_seconds', help: 'Time from acceptance to successful delivery, retries included', labelNames: ['priority'], buckets: LATENCY_BUCKETS });
const attempts = registry.counter({ name: 'events_delivery_attempts_total', help: 'Delivery attempts by outcome', labelNames: ['outcome'] });
const attempts = registry.counter({ name: 'events_delivery_attempts_total', help: 'Delivery attempts by outcome', labelNames: ['outcome'] });
@@ -106,7 +106,7 @@ function createRealtime({ store, auth, config, clock = { now: () => Date.now() }
@@ -106,7 +106,7 @@ function createRealtime({ store, auth, config, clock = { now: () => Date.now() }
return [...new Set(list)];
return [...new Set(list)];
}
}
- function handler(req, res) {
+ async function handler(req, res) {
const ctx = req.ov;
const ctx = req.ov;
const wanted = parseTopics(req.query.topics);
const wanted = parseTopics(req.query.topics);
if (!wanted.length) return http.sendProblem(res, 400, 'realtime.bad_request', { detail: 'topics=<pattern>[,<pattern>] is required', ctx });
if (!wanted.length) return http.sendProblem(res, 400, 'realtime.bad_request', { detail: 'topics=<pattern>[,<pattern>] is required', ctx });
@@ -138,46 +138,55 @@ function createRealtime({ store, auth, config, clock = { now: () => Date.now() }
@@ -138,46 +138,55 @@ function createRealtime({ store, auth, config, clock = { now: () => Date.now() }
req.socket.setTimeout(0);
req.socket.setTimeout(0);
req.socket.setNoDelay(true);
req.socket.setNoDelay(true);
- const conn = { res, viewer, patterns: wanted, lastSeq: lastId ?? store.lastSeq(), closed: false };
+ // The connection takes its slot and joins the fan-out before the first await, so the cap holds and nothing
+ // published during the catch-up is lost: those rows wait in `pending` and follow the replay, skipping what it sent.
+ const conn = { res, viewer, patterns: wanted, lastSeq: lastId, pending: [], closed: false };
+ conns.add(conn);
+ res.on('close', () => close(conn));
write(conn, `retry: 3000\n: connected ${viewer.kind}\n\n`);
write(conn, `retry: 3000\n: connected ${viewer.kind}\n\n`);
-
- // Catch up synchronously (better-sqlite3 is synchronous, so no event can be published in
- // between), then join the live fan-out.
- if (lastId != null) {
- const latest = store.lastSeq();
- const oldest = store.oldestSeq();
- if (lastId > latest) {
- sendGap(conn, { reason: 'cursor_ahead', from_seq: latest + 1, to_seq: lastId, latest_seq: latest });
- conn.lastSeq = latest;
- } else {
- let cursor = lastId;
- if (lastId < oldest - 1) {
- sendGap(conn, { reason: 'retention', from_seq: lastId + 1, to_seq: oldest - 1, latest_seq: latest });
- cursor = oldest - 1;
- }
- // Browsers: public events older than the window are not replayed (a gap says so).
- const publicFrom = viewer.kind === 'service' ? 0
- : opts.publicReplaySeconds > 0 ? store.firstSeqSince(clock.now() - opts.publicReplaySeconds * 1000) : latest + 1;
- if (cursor < publicFrom - 1) {
- sendGap(conn, { reason: 'public_window', from_seq: cursor + 1, to_seq: publicFrom - 1, latest_seq: latest, window_seconds: opts.publicReplaySeconds });
- // Signed out, only public events are visible: nothing before the window to scan.
- if (viewer.kind === 'anonymous') cursor = publicFrom - 1;
- }
- const accept = (row) => visibleTo(viewer, row) && !(row.visibility === 'public' && row.seq < publicFrom);
- const { rows, cursor: scanned } = store.scan(cursor, {
- patterns: wanted, limit: opts.replayMax, scanMax: opts.replayMax * 50, accept,
- });
- for (const row of rows) sendEvent(conn, row);
- if (scanned < latest) {
- // Replay stopped at its limit: everything after `scanned` is reported as a gap.
- sendGap(conn, { reason: 'replay_limit', from_seq: scanned + 1, to_seq: latest, latest_seq: latest });
+ try {
+ if (lastId == null) conn.lastSeq = await store.lastSeq();
+ if (lastId != null) {
+ const latest = await store.lastSeq();
+ const oldest = await store.oldestSeq();
+ if (lastId > latest) {
+ sendGap(conn, { reason: 'cursor_ahead', from_seq: latest + 1, to_seq: lastId, latest_seq: latest });
+ conn.lastSeq = latest;
+ } else {
+ let cursor = lastId;
+ if (lastId < oldest - 1) {
+ sendGap(conn, { reason: 'retention', from_seq: lastId + 1, to_seq: oldest - 1, latest_seq: latest });
+ cursor = oldest - 1;
+ }
+ // Browsers: public events older than the window are not replayed (a gap says so).
+ const publicFrom = viewer.kind === 'service' ? 0
+ : opts.publicReplaySeconds > 0 ? await store.firstSeqSince(clock.now() - opts.publicReplaySeconds * 1000) : latest + 1;
+ if (cursor < publicFrom - 1) {
+ sendGap(conn, { reason: 'public_window', from_seq: cursor + 1, to_seq: publicFrom - 1, latest_seq: latest, window_seconds: opts.publicReplaySeconds });
+ // Signed out, only public events are visible: nothing before the window to scan.
+ if (viewer.kind === 'anonymous') cursor = publicFrom - 1;
+ }
+ const accept = (row) => visibleTo(viewer, row) && !(row.visibility === 'public' && row.seq < publicFrom);
+ const { rows, cursor: scanned } = await store.scan(cursor, {
+ patterns: wanted, limit: opts.replayMax, scanMax: opts.replayMax * 50, accept,
+ });
+ for (const row of rows) sendEvent(conn, row);
+ if (scanned < latest) {
+ // Replay stopped at its limit: everything after `scanned` is reported as a gap.
+ sendGap(conn, { reason: 'replay_limit', from_seq: scanned + 1, to_seq: latest, latest_seq: latest });
+ }
+ conn.lastSeq = latest;
}
}
- conn.lastSeq = latest;
}
}
+ } catch (err) {
+ // The headers are sent: a failed catch-up can only end the stream (the client reconnects with its Last-Event-ID).
+ log.error(`[realtime] catch-up failed: ${err.message}`);
+ close(conn);
+ return undefined;
}
}
-
- conns.add(conn);
- res.on('close', () => close(conn));
+ const pending = conn.pending;
+ conn.pending = null;
+ for (const row of pending) if (row.seq > conn.lastSeq) sendEvent(conn, row);
return undefined;
return undefined;
}
}
@@ -186,10 +195,11 @@ function createRealtime({ store, auth, config, clock = { now: () => Date.now() }
@@ -186,10 +195,11 @@ function createRealtime({ store, auth, config, clock = { now: () => Date.now() }
if (!conns.size) return;
if (!conns.size) return;
for (const row of rows) {
for (const row of rows) {
for (const conn of conns) {
for (const conn of conns) {
- if (conn.closed || row.seq <= conn.lastSeq) continue;
+ if (conn.closed || (!conn.pending && row.seq <= conn.lastSeq)) continue;
if (!conn.patterns.some(p => topics.matches(p, row.event_type))) continue;
if (!conn.patterns.some(p => topics.matches(p, row.event_type))) continue;
if (!visibleTo(conn.viewer, row)) continue;
if (!visibleTo(conn.viewer, row)) continue;
- sendEvent(conn, row);
+ if (conn.pending) conn.pending.push(row);
+ else sendEvent(conn, row);
}
}
}
}
}
}
@@ -7,7 +7,7 @@
@@ -7,7 +7,7 @@
*/
*/
const fs = require('fs');
const fs = require('fs');
const path = require('path');
const path = require('path');
-const Database = require('better-sqlite3');
+const { createDb } = require('openvibe-sdk/db');
const topics = require('./topics');
const topics = require('./topics');
const apps = require('./apps');
const apps = require('./apps');
const redaction = require('./redaction');
const redaction = require('./redaction');
@@ -17,134 +17,29 @@ const PRIORITY_RANK = { critical: 0, important: 1, low: 2 };
@@ -17,134 +17,29 @@ const PRIORITY_RANK = { critical: 0, important: 1, low: 2 };
const RANK_PRIORITY = ['critical', 'important', 'low'];
const RANK_PRIORITY = ['critical', 'important', 'low'];
const DAY_MS = 24 * 60 * 60 * 1000;
const DAY_MS = 24 * 60 * 60 * 1000;
-const SCHEMA = `
-CREATE TABLE IF NOT EXISTS sequences (
- name TEXT PRIMARY KEY,
- value INTEGER NOT NULL
-);
-CREATE TABLE IF NOT EXISTS events (
- id TEXT PRIMARY KEY,
- seq INTEGER NOT NULL UNIQUE,
- event_type TEXT NOT NULL,
- version INTEGER NOT NULL,
- source TEXT NOT NULL,
- actor TEXT NOT NULL,
- subject_type TEXT NOT NULL,
- subject_id TEXT NOT NULL,
- subject_revision INTEGER,
- trace_id TEXT NOT NULL,
- priority TEXT NOT NULL,
- visibility TEXT NOT NULL,
- occurred_at TEXT NOT NULL,
- received_at INTEGER NOT NULL,
- payload TEXT NOT NULL,
- hops INTEGER NOT NULL DEFAULT 1,
- publisher TEXT NOT NULL,
- request_id TEXT
-);
-CREATE INDEX IF NOT EXISTS idx_events_trace ON events(trace_id);
-CREATE INDEX IF NOT EXISTS idx_events_received ON events(received_at);
-CREATE INDEX IF NOT EXISTS idx_events_type_seq ON events(event_type, seq);
-
-CREATE TABLE IF NOT EXISTS subscriptions (
- id TEXT PRIMARY KEY,
- consumer TEXT NOT NULL,
- topic_pattern TEXT NOT NULL,
- endpoint TEXT NOT NULL,
- secret TEXT NOT NULL,
- enabled INTEGER NOT NULL DEFAULT 1,
- retry_policy TEXT,
- created_at INTEGER NOT NULL,
- updated_at INTEGER NOT NULL
-);
-CREATE INDEX IF NOT EXISTS idx_subscriptions_consumer ON subscriptions(consumer);
-
-CREATE TABLE IF NOT EXISTS deliveries (
- event_id TEXT NOT NULL REFERENCES events(id) ON DELETE CASCADE,
- subscription_id TEXT NOT NULL REFERENCES subscriptions(id) ON DELETE CASCADE,
- seq INTEGER NOT NULL,
- priority INTEGER NOT NULL,
- attempt INTEGER NOT NULL DEFAULT 0,
- status TEXT NOT NULL CHECK (status IN ('pending', 'delivered', 'failed', 'dead')),
- next_attempt_at INTEGER,
- last_error TEXT,
- last_status INTEGER,
- delivered_at INTEGER,
- created_at INTEGER NOT NULL,
- updated_at INTEGER NOT NULL,
- PRIMARY KEY (event_id, subscription_id)
-);
-CREATE INDEX IF NOT EXISTS idx_deliveries_due ON deliveries(status, next_attempt_at);
-CREATE INDEX IF NOT EXISTS idx_deliveries_sub ON deliveries(subscription_id, status, seq);
-
-CREATE TABLE IF NOT EXISTS consumer_checkpoints (
- consumer TEXT NOT NULL,
- topic_pattern TEXT NOT NULL,
- cursor INTEGER NOT NULL,
- updated_at INTEGER NOT NULL,
- PRIMARY KEY (consumer, topic_pattern)
-);
-
-CREATE TABLE IF NOT EXISTS app_revocations (
- consumer TEXT NOT NULL,
- scope TEXT NOT NULL,
- revoked_at INTEGER NOT NULL,
- PRIMARY KEY (consumer, scope)
-);
-
-CREATE TABLE IF NOT EXISTS idempotency_receipts (
- consumer TEXT NOT NULL,
- event_id TEXT NOT NULL,
- processed_at INTEGER NOT NULL,
- PRIMARY KEY (consumer, event_id)
-);
-`;
+const MIGRATIONS = path.join(__dirname, '..', 'migrations');
+const DEV_PGLITE = path.join(__dirname, '..', 'data', 'pglite');
const PUBLISH_RECEIPT = 'events:publish'; // receipts' consumer column for accepted publishes
const PUBLISH_RECEIPT = 'events:publish'; // receipts' consumer column for accepted publishes
/**
/**
- * Columns added after the first release (developer apps, ADR-014). Existing databases get them
- * with ALTER TABLE; every row stored before is a first-party production row.
- * events.project_id / env / size_bytes who owns an app event, its environment, stored bytes
- * subscriptions.project_id / env an app subscription's scope (NULL project = first-party)
- * and redaction (server/redaction.js):
- * events.redacted_at / redacted_by set when the row became a tombstone (ms, the redacting event_id)
- * events.redacts 1 when the row carries a payload.redacts directive
+ * The serving handle (ADR-035): DATABASE_URL through PgBouncer; in development without it, an embedded PGlite database
+ * in data/pglite. Migrations run first, as the owner (DATABASE_DIRECT_URL), or on the embedded handle. The columns added
+ * after the first release (developer apps, ADR-014; redaction) and the sequence row are in migrations/0001_initial.sql.
*/
*/
-const ADDED_COLUMNS = {
- events: [['project_id', 'TEXT'], ['env', "TEXT NOT NULL DEFAULT 'production'"], ['size_bytes', 'INTEGER NOT NULL DEFAULT 0'],
- ['redacted_at', 'INTEGER'], ['redacted_by', 'TEXT'], ['redacts', 'INTEGER NOT NULL DEFAULT 0']],
- subscriptions: [['project_id', 'TEXT'], ['env', "TEXT NOT NULL DEFAULT 'production'"], ['previous_secret', 'TEXT'], ['previous_secret_until', 'INTEGER']],
-};
-
-function migrate(db) {
- for (const [table, cols] of Object.entries(ADDED_COLUMNS)) {
- const have = new Set(db.prepare(`PRAGMA table_info(${table})`).all().map(c => c.name));
- for (const [name, type] of cols) if (!have.has(name)) db.exec(`ALTER TABLE ${table} ADD COLUMN ${name} ${type}`);
+async function openDb(config, { log = console, registry } = {}) {
+ if (!config.db.url) {
+ if (config.isProduction) throw new Error('DATABASE_URL is not set: production serves from PostgreSQL (OpenVibe.Host roles/data add-service.sh events)');
+ log.warn(`[events] DATABASE_URL unset: embedded PGlite database in ${DEV_PGLITE} (development only, one process)`);
+ fs.mkdirSync(DEV_PGLITE, { recursive: true });
+ const db = createDb({ pglite: DEV_PGLITE, service: 'events', registry, log });
+ await db.migrate({ dir: MIGRATIONS, log });
+ return db;
}
}
- db.exec(`CREATE INDEX IF NOT EXISTS idx_events_project ON events(project_id, env, received_at) WHERE project_id IS NOT NULL;
- CREATE INDEX IF NOT EXISTS idx_subscriptions_project ON subscriptions(project_id, env) WHERE project_id IS NOT NULL;
- CREATE INDEX IF NOT EXISTS idx_events_subject ON events(source, subject_type, subject_id);`);
-}
-
-function openDb(dbPath) {
- if (dbPath !== ':memory:') fs.mkdirSync(path.dirname(path.resolve(dbPath)), { recursive: true });
- const db = new Database(dbPath);
- db.pragma('journal_mode = WAL');
- db.pragma('synchronous = NORMAL');
- db.pragma('foreign_keys = ON');
- db.pragma('busy_timeout = 5000');
- // A redacted payload (and a pruned row) is overwritten with zeros, not left in free pages.
- db.pragma('secure_delete = ON');
- db.exec(SCHEMA);
- migrate(db);
- db.prepare("INSERT OR IGNORE INTO sequences (name, value) VALUES ('events', 0)").run();
- return db;
+ if (!config.db.directUrl) throw new Error('DATABASE_DIRECT_URL is not set: migrations run with the owner role on a direct connection');
+ const owner = createDb({ url: config.db.directUrl, service: 'events-migrate', max: 1, log });
+ try { await owner.migrate({ dir: MIGRATIONS, log }); } finally { await owner.close(); }
+ return createDb({ url: config.db.url, service: 'events', registry, log });
}
}
class StoreError extends Error {
class StoreError extends Error {
@@ -200,7 +95,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -200,7 +95,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
const q = {
const q = {
getEvent: db.prepare('SELECT * FROM events WHERE id = ?'),
getEvent: db.prepare('SELECT * FROM events WHERE id = ?'),
getReceipt: db.prepare('SELECT processed_at FROM idempotency_receipts WHERE consumer = ? AND event_id = ?'),
getReceipt: db.prepare('SELECT processed_at FROM idempotency_receipts WHERE consumer = ? AND event_id = ?'),
- addReceipt: db.prepare('INSERT OR IGNORE INTO idempotency_receipts (consumer, event_id, processed_at) VALUES (?, ?, ?)'),
+ addReceipt: db.prepare('INSERT INTO idempotency_receipts (consumer, event_id, processed_at) VALUES (?, ?, ?) ON CONFLICT DO NOTHING'),
nextSeq: db.prepare("UPDATE sequences SET value = value + 1 WHERE name = 'events' RETURNING value"),
nextSeq: db.prepare("UPDATE sequences SET value = value + 1 WHERE name = 'events' RETURNING value"),
lastSeq: db.prepare("SELECT value FROM sequences WHERE name = 'events'"),
lastSeq: db.prepare("SELECT value FROM sequences WHERE name = 'events'"),
minSeq: db.prepare('SELECT MIN(seq) AS s FROM events'),
minSeq: db.prepare('SELECT MIN(seq) AS s FROM events'),
@@ -225,15 +120,15 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -225,15 +120,15 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
firstSeqSince: db.prepare('SELECT MIN(seq) AS s FROM events WHERE received_at >= ?'),
firstSeqSince: db.prepare('SELECT MIN(seq) AS s FROM events WHERE received_at >= ?'),
enabledSubs: db.prepare('SELECT id, topic_pattern, project_id, env FROM subscriptions WHERE enabled = 1'),
enabledSubs: db.prepare('SELECT id, topic_pattern, project_id, env FROM subscriptions WHERE enabled = 1'),
projectRecent: db.prepare('SELECT COUNT(*) AS n FROM events WHERE project_id = ? AND env = ? AND received_at > ?'),
projectRecent: db.prepare('SELECT COUNT(*) AS n FROM events WHERE project_id = ? AND env = ? AND received_at > ?'),
- projectBytes: db.prepare('SELECT COALESCE(SUM(size_bytes), 0) AS b FROM events WHERE project_id = ? AND env = ?'),
+ projectBytes: db.prepare('SELECT COALESCE(SUM(size_bytes), 0)::bigint AS b FROM events WHERE project_id = ? AND env = ?'),
disableAppSubs: db.prepare('UPDATE subscriptions SET enabled = 0, updated_at = ? WHERE consumer = ? AND enabled = 1'),
disableAppSubs: db.prepare('UPDATE subscriptions SET enabled = 0, updated_at = ? WHERE consumer = ? AND enabled = 1'),
revoke: db.prepare(`INSERT INTO app_revocations (consumer, scope, revoked_at) VALUES (?, ?, ?)
revoke: db.prepare(`INSERT INTO app_revocations (consumer, scope, revoked_at) VALUES (?, ?, ?)
- ON CONFLICT(consumer, scope) DO UPDATE SET revoked_at = MAX(revoked_at, excluded.revoked_at)`),
+ ON CONFLICT(consumer, scope) DO UPDATE SET revoked_at = GREATEST(app_revocations.revoked_at, excluded.revoked_at)`),
revokedAt: db.prepare('SELECT revoked_at FROM app_revocations WHERE consumer = ? AND scope = ?'),
revokedAt: db.prepare('SELECT revoked_at FROM app_revocations WHERE consumer = ? AND scope = ?'),
- insertDelivery: db.prepare(`INSERT OR IGNORE INTO deliveries (event_id, subscription_id, seq, priority, attempt, status,
- next_attempt_at, created_at, updated_at) VALUES (?, ?, ?, ?, 0, 'pending', ?, ?, ?)`),
+ insertDelivery: db.prepare(`INSERT INTO deliveries (event_id, subscription_id, seq, priority, attempt, status,
+ next_attempt_at, created_at, updated_at) VALUES (?, ?, ?, ?, 0, 'pending', ?, ?, ?) ON CONFLICT DO NOTHING`),
afterSeq: db.prepare('SELECT * FROM events WHERE seq > ? ORDER BY seq LIMIT ?'),
afterSeq: db.prepare('SELECT * FROM events WHERE seq > ? ORDER BY seq LIMIT ?'),
- afterSeqGlob: db.prepare('SELECT * FROM events WHERE seq > ? AND event_type GLOB ? ORDER BY seq LIMIT ?'),
+ afterSeqLike: db.prepare("SELECT * FROM events WHERE seq > ? AND event_type ILIKE ? ESCAPE '\\' ORDER BY seq LIMIT ?"),
};
};
// ── Events ───────────────────────────────────────────────
// ── Events ───────────────────────────────────────────────
@@ -247,19 +142,19 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -247,19 +142,19 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
* carry project_id and env, and the project's rate and retained-bytes quotas are checked here,
* carry project_id and env, and the project's rate and retained-bytes quotas are checked here,
* inside the transaction, against what is already stored.
* inside the transaction, against what is already stored.
*/
*/
- const insertBatch = db.transaction((items, { publisher, requestId, project = null }) => {
+ const insertBatch = async (items, { publisher, requestId, project = null }) => await db.tx(async () => {
const now = clock.now();
const now = clock.now();
const results = [];
const results = [];
const inserted = [];
const inserted = [];
- const subs = q.enabledSubs.all();
+ const subs = await q.enabledSubs.all();
let recent = null;
let recent = null;
let bytes = null;
let bytes = null;
if (project) {
if (project) {
- recent = q.projectRecent.get(project.projectId, project.env, now - 60 * 1000).n;
- bytes = q.projectBytes.get(project.projectId, project.env).b;
+ recent = (await q.projectRecent.get(project.projectId, project.env, now - 60 * 1000)).n;
+ bytes = (await q.projectBytes.get(project.projectId, project.env)).b;
}
}
for (const env of items) {
for (const env of items) {
- const existing = q.getEvent.get(env.event_id);
+ const existing = await q.getEvent.get(env.event_id);
if (existing) {
if (existing) {
if (existing.source !== env.source || existing.event_type !== env.event_type) {
if (existing.source !== env.source || existing.event_type !== env.event_type) {
throw new StoreError(409, 'events.id_conflict', `event ${env.event_id} already exists with different content`);
throw new StoreError(409, 'events.id_conflict', `event ${env.event_id} already exists with different content`);
@@ -267,17 +162,17 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -267,17 +162,17 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
results.push({ event_id: env.event_id, seq: existing.seq, duplicate: true });
results.push({ event_id: env.event_id, seq: existing.seq, duplicate: true });
continue;
continue;
}
}
- if (q.getReceipt.get(PUBLISH_RECEIPT, env.event_id)) {
+ if (await q.getReceipt.get(PUBLISH_RECEIPT, env.event_id)) {
results.push({ event_id: env.event_id, seq: null, duplicate: true, pruned: true });
results.push({ event_id: env.event_id, seq: null, duplicate: true, pruned: true });
continue;
continue;
}
}
// Loop guard: depth of the cross-service chain in this trace, and how often this exact
// Loop guard: depth of the cross-service chain in this trace, and how often this exact
// (source, type, subject) already happened in the trace from other publish calls.
// (source, type, subject) already happened in the trace from other publish calls.
const chainRow = project
const chainRow = project
- ? q.chainHopsApp.get(env.trace_id, env.source, project.projectId, project.env)
- : q.chainHops.get(env.trace_id, env.source);
+ ? await q.chainHopsApp.get(env.trace_id, env.source, project.projectId, project.env)
+ : await q.chainHops.get(env.trace_id, env.source);
const chain = (chainRow.h || 0) + 1;
const chain = (chainRow.h || 0) + 1;
- const repeats = q.selfRepeats.get(env.trace_id, env.source, env.event_type, env.subject.type, env.subject.id, requestId || '').n + 1;
+ const repeats = (await q.selfRepeats.get(env.trace_id, env.source, env.event_type, env.subject.type, env.subject.id, requestId || '')).n + 1;
const hops = Math.max(chain, repeats);
const hops = Math.max(chain, repeats);
if (hops > maxHops) {
if (hops > maxHops) {
throw new StoreError(409, 'events.loop_detected',
throw new StoreError(409, 'events.loop_detected',
@@ -299,7 +194,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -299,7 +194,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
{ quota: 'retained_bytes', limit: project.retainedBytes, used: bytes - sizeBytes });
{ quota: 'retained_bytes', limit: project.retainedBytes, used: bytes - sizeBytes });
}
}
}
}
- const seq = q.nextSeq.get().value;
+ const seq = (await q.nextSeq.get()).value;
const row = {
const row = {
id: env.event_id,
id: env.event_id,
seq,
seq,
@@ -327,18 +222,18 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -327,18 +222,18 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
const directive = redaction.parseDirective(env.payload);
const directive = redaction.parseDirective(env.payload);
if (directive && directive.error) throw new StoreError(422, 'events.invalid_redaction', directive.error, { event_id: env.event_id });
if (directive && directive.error) throw new StoreError(422, 'events.invalid_redaction', directive.error, { event_id: env.event_id });
if (directive) row.redacts = 1;
if (directive) row.redacts = 1;
- q.insertEvent.run(row);
- q.addReceipt.run(PUBLISH_RECEIPT, env.event_id, now);
+ await q.insertEvent.run(row);
+ await q.addReceipt.run(PUBLISH_RECEIPT, env.event_id, now);
for (const s of subs) {
for (const s of subs) {
if (apps.deliverable(s, row)) {
if (apps.deliverable(s, row)) {
- q.insertDelivery.run(env.event_id, s.id, seq, PRIORITY_RANK[env.priority], now, now, now);
+ await q.insertDelivery.run(env.event_id, s.id, seq, PRIORITY_RANK[env.priority], now, now, now);
}
}
}
}
- revocationHook(row, now);
+ await revocationHook(row, now);
results.push({ event_id: env.event_id, seq, duplicate: false });
results.push({ event_id: env.event_id, seq, duplicate: false });
inserted.push(row);
inserted.push(row);
if (directive) {
if (directive) {
- const redacted = applyRedaction(row, directive, now);
+ const redacted = await applyRedaction(row, directive, now);
// Rows stored earlier in this batch are fanned out after the commit from memory: they
// Rows stored earlier in this batch are fanned out after the commit from memory: they
// must go out as the tombstones they now are.
// must go out as the tombstones they now are.
for (const t of redacted) {
for (const t of redacted) {
@@ -349,7 +244,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -349,7 +244,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
}
}
// An app publish counts toward its project's usage in the same transaction as the rows.
// An app publish counts toward its project's usage in the same transaction as the rows.
if (project && inserted.length) {
if (project && inserted.length) {
- usage.record({ projectId: project.projectId, env: project.env, capability: 'events.app.publish', unit: 'events', quantity: inserted.length, at: now });
+ await usage.record({ projectId: project.projectId, env: project.env, capability: 'events.app.publish', unit: 'events', quantity: inserted.length, at: now });
}
}
return { results, inserted };
return { results, inserted };
});
});
@@ -359,10 +254,10 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -359,10 +254,10 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
* stored row of the redacting event). Throws StoreError 403 when an event id names an event of
* stored row of the redacting event). Throws StoreError 403 when an event id names an event of
* another owner. Returns [{ id, seq, columns }] for the rows it rewrote.
* another owner. Returns [{ id, seq, columns }] for the rows it rewrote.
*/
*/
- function applyRedaction(by, directive, now) {
+ async function applyRedaction(by, directive, now) {
const targets = new Map();
const targets = new Map();
for (const id of directive.eventIds) {
for (const id of directive.eventIds) {
- const r = q.getEvent.get(id);
+ const r = await q.getEvent.get(id);
if (!r) continue; // never stored, or pruned already
if (!r) continue; // never stored, or pruned already
if (!redaction.sameOwner(by, r)) {
if (!redaction.sameOwner(by, r)) {
throw new StoreError(403, 'events.redaction_not_allowed', `event ${id} belongs to ${r.source}; ${by.source} may redact only its own events`,
throw new StoreError(403, 'events.redaction_not_allowed', `event ${id} belongs to ${r.source}; ${by.source} may redact only its own events`,
@@ -372,7 +267,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -372,7 +267,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
}
}
if (directive.subjectType) {
if (directive.subjectType) {
for (const subjectId of directive.subjectIds) {
for (const subjectId of directive.subjectIds) {
- for (const r of q.redactBySubject.all(by.source, directive.subjectType, subjectId)) {
+ for (const r of await q.redactBySubject.all(by.source, directive.subjectType, subjectId)) {
if (redaction.sameOwner(by, r)) targets.set(r.id, r);
if (redaction.sameOwner(by, r)) targets.set(r.id, r);
}
}
}
}
@@ -381,7 +276,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -381,7 +276,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
for (const r of targets.values()) {
for (const r of targets.values()) {
if (r.id === by.id) continue;
if (r.id === by.id) continue;
const columns = redaction.tombstone(r, { by: by.id, at: now });
const columns = redaction.tombstone(r, { by: by.id, at: now });
- if (q.applyTombstone.run({ id: r.id, ...columns }).changes) out.push({ id: r.id, seq: r.seq, columns });
+ if ((await q.applyTombstone.run({ id: r.id, ...columns })).changes) out.push({ id: r.id, seq: r.seq, columns });
}
}
return out;
return out;
}
}
@@ -391,10 +286,10 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -391,10 +286,10 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
* `directive` (the payload.redacts object) would have redacted. The tombstones name `by` as
* `directive` (the payload.redacts object) would have redacted. The tombstones name `by` as
* redacted_by. Same owner rule as a publish. Returns the rewritten [{ id, seq }].
* redacted_by. Same owner rule as a publish. Returns the rewritten [{ id, seq }].
*/
*/
- const redact = db.transaction((source, directive, { by = 'operator', now = clock.now() } = {}) => {
+ const redact = async (source, directive, { by = 'operator', now = clock.now() } = {}) => await db.tx(async () => {
const parsed = redaction.parseDirective({ redacts: directive });
const parsed = redaction.parseDirective({ redacts: directive });
if (!parsed || parsed.error) throw new StoreError(422, 'events.invalid_redaction', parsed ? parsed.error : 'nothing to redact');
if (!parsed || parsed.error) throw new StoreError(422, 'events.invalid_redaction', parsed ? parsed.error : 'nothing to redact');
- return applyRedaction({ id: by, source, project_id: null, env: 'production' }, parsed, now).map(({ id, seq }) => ({ id, seq }));
+ return (await applyRedaction({ id: by, source, project_id: null, env: 'production' }, parsed, now)).map(({ id, seq }) => ({ id, seq }));
});
});
/**
/**
@@ -403,7 +298,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -403,7 +298,7 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
* app's subscriptions stop at once, in the same transaction that stores the event, so a
* app's subscriptions stop at once, in the same transaction that stores the event, so a
* revocation reaches deliveries without waiting for anything else.
* revocation reaches deliveries without waiting for anything else.
*/
*/
- function revocationHook(row, now) {
+ async function revocationHook(row, now) {
if (row.source !== 'network' || row.project_id || row.subject_type !== 'app') return;
if (row.source !== 'network' || row.project_id || row.subject_type !== 'app') return;
let payload = {};
let payload = {};
try { payload = JSON.parse(row.payload) || {}; } catch { payload = {}; }
try { payload = JSON.parse(row.payload) || {}; } catch { payload = {}; }
@@ -414,34 +309,34 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -414,34 +309,34 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
if (!scope || !/^app_[0-9A-HJKMNP-TV-Z]{26}$/.test(row.subject_id)) return;
if (!scope || !/^app_[0-9A-HJKMNP-TV-Z]{26}$/.test(row.subject_id)) return;
const consumer = `app:${row.subject_id}`;
const consumer = `app:${row.subject_id}`;
const at = Date.parse(row.occurred_at);
const at = Date.parse(row.occurred_at);
- q.revoke.run(consumer, scope, Number.isFinite(at) ? at : now);
- q.disableAppSubs.run(now, consumer);
+ await q.revoke.run(consumer, scope, Number.isFinite(at) ? at : now);
+ await q.disableAppSubs.run(now, consumer);
}
}
/** When Network revoked `consumer` (app:app_…) for `scope` ('app' | 'events.app.subscribe'), in ms; or null. */
/** When Network revoked `consumer` (app:app_…) for `scope` ('app' | 'events.app.subscribe'), in ms; or null. */
- function revokedAt(consumer, scope) {
- const r = q.revokedAt.get(consumer, scope);
+ async function revokedAt(consumer, scope) {
+ const r = await q.revokedAt.get(consumer, scope);
return r ? r.revoked_at : null;
return r ? r.revoked_at : null;
}
}
- function getEvent(id) {
- return q.getEvent.get(id) || null;
+ async function getEvent(id) {
+ return await q.getEvent.get(id) || null;
}
}
- function lastSeq() {
- return q.lastSeq.get().value;
+ async function lastSeq() {
+ return (await q.lastSeq.get()).value;
}
}
/** First seq received at or after `ms`; the next seq to be handed out when there is none. */
/** First seq received at or after `ms`; the next seq to be handed out when there is none. */
- function firstSeqSince(ms) {
- const m = q.firstSeqSince.get(ms).s;
- return m == null ? lastSeq() + 1 : m;
+ async function firstSeqSince(ms) {
+ const m = (await q.firstSeqSince.get(ms)).s;
+ return m == null ? await lastSeq() + 1 : m;
}
}
/** Oldest seq still stored; when the table is empty, the next seq to be handed out. */
/** Oldest seq still stored; when the table is empty, the next seq to be handed out. */
- function oldestSeq() {
- const m = q.minSeq.get().s;
- return m == null ? lastSeq() + 1 : m;
+ async function oldestSeq() {
+ const m = (await q.minSeq.get()).s;
+ return m == null ? await lastSeq() + 1 : m;
}
}
/**
/**
@@ -449,13 +344,13 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -449,13 +344,13 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
* Scans at most `scanMax` rows; returns { rows, cursor } where cursor is the last seq examined
* Scans at most `scanMax` rows; returns { rows, cursor } where cursor is the last seq examined
* (so a pull consumer's cursor moves past rows that did not match).
* (so a pull consumer's cursor moves past rows that did not match).
*/
*/
- function scan(afterSeq, { patterns = ['*'], limit = 100, scanMax = 5000, accept = () => true } = {}) {
+ async function scan(afterSeq, { patterns = ['*'], limit = 100, scanMax = 5000, accept = () => true } = {}) {
const rows = [];
const rows = [];
let cursor = afterSeq;
let cursor = afterSeq;
let scanned = 0;
let scanned = 0;
- const single = patterns.length === 1 && patterns[0] !== '*' ? topics.toGlob(patterns[0]) : null;
+ const single = patterns.length === 1 && patterns[0] !== '*' ? topics.toLike(patterns[0]) : null;
for (;;) {
for (;;) {
- const page = single ? q.afterSeqGlob.all(cursor, single, 500) : q.afterSeq.all(cursor, 500);
+ const page = single ? await q.afterSeqLike.all(cursor, single, 500) : await q.afterSeq.all(cursor, 500);
for (const row of page) {
for (const row of page) {
scanned++;
scanned++;
cursor = row.seq;
cursor = row.seq;
@@ -464,9 +359,9 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -464,9 +359,9 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
}
}
if (rows.length >= limit || scanned >= scanMax) break;
if (rows.length >= limit || scanned >= scanMax) break;
if (page.length < 500) {
if (page.length < 500) {
- // Reached the end. With the GLOB narrowing, rows that were never fetched cannot match,
+ // Reached the end. With the ILIKE narrowing, rows that were never fetched cannot match,
// so the cursor may move up to the latest seq handed out.
// so the cursor may move up to the latest seq handed out.
- if (single) cursor = Math.max(cursor, lastSeq());
+ if (single) cursor = Math.max(cursor, await lastSeq());
break;
break;
}
}
}
}
@@ -475,37 +370,37 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -475,37 +370,37 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
// ── Subscriptions ────────────────────────────────────────
// ── Subscriptions ────────────────────────────────────────
- function createSubscription({ id, consumer, topicPattern, endpoint, secret, retryPolicy, projectId = null, env = 'production' }) {
+ async function createSubscription({ id, consumer, topicPattern, endpoint, secret, retryPolicy, projectId = null, env = 'production' }) {
const now = clock.now();
const now = clock.now();
- db.prepare(`INSERT INTO subscriptions (id, consumer, topic_pattern, endpoint, secret, enabled, retry_policy, created_at, updated_at, project_id, env)
+ await db.prepare(`INSERT INTO subscriptions (id, consumer, topic_pattern, endpoint, secret, enabled, retry_policy, created_at, updated_at, project_id, env)
VALUES (?, ?, ?, ?, ?, 1, ?, ?, ?, ?, ?)`).run(id, consumer, topicPattern, endpoint, secret, retryPolicy ? JSON.stringify(retryPolicy) : null, now, now, projectId, env);
VALUES (?, ?, ?, ?, ?, 1, ?, ?, ?, ?, ?)`).run(id, consumer, topicPattern, endpoint, secret, retryPolicy ? JSON.stringify(retryPolicy) : null, now, now, projectId, env);
- return getSubscription(id);
+ return await getSubscription(id);
}
}
- function countProjectSubscriptions(projectId, env) {
- return db.prepare('SELECT COUNT(*) AS n FROM subscriptions WHERE project_id = ? AND env = ?').get(projectId, env).n;
+ async function countProjectSubscriptions(projectId, env) {
+ return (await db.prepare('SELECT COUNT(*) AS n FROM subscriptions WHERE project_id = ? AND env = ?').get(projectId, env)).n;
}
}
- function getSubscription(id) {
- return db.prepare('SELECT * FROM subscriptions WHERE id = ?').get(id) || null;
+ async function getSubscription(id) {
+ return await db.prepare('SELECT * FROM subscriptions WHERE id = ?').get(id) || null;
}
}
- function listSubscriptions(consumer) {
+ async function listSubscriptions(consumer) {
return consumer
return consumer
- ? db.prepare('SELECT * FROM subscriptions WHERE consumer = ? ORDER BY created_at, id').all(consumer)
- : db.prepare('SELECT * FROM subscriptions ORDER BY created_at, id').all();
+ ? await db.prepare('SELECT * FROM subscriptions WHERE consumer = ? ORDER BY created_at, id').all(consumer)
+ : await db.prepare('SELECT * FROM subscriptions ORDER BY created_at, id').all();
}
}
- function countSubscriptions(consumer) {
- return db.prepare('SELECT COUNT(*) AS n FROM subscriptions WHERE consumer = ?').get(consumer).n;
+ async function countSubscriptions(consumer) {
+ return (await db.prepare('SELECT COUNT(*) AS n FROM subscriptions WHERE consumer = ?').get(consumer)).n;
}
}
/** Rotate a subscription's secret: the old one keeps signing (next to the new) until overlapMs from now. */
/** Rotate a subscription's secret: the old one keeps signing (next to the new) until overlapMs from now. */
- function rotateSubscriptionSecret(id, secret, overlapMs) {
+ async function rotateSubscriptionSecret(id, secret, overlapMs) {
const now = clock.now();
const now = clock.now();
- db.prepare('UPDATE subscriptions SET previous_secret = secret, previous_secret_until = ?, secret = ?, updated_at = ? WHERE id = ?')
+ await db.prepare('UPDATE subscriptions SET previous_secret = secret, previous_secret_until = ?, secret = ?, updated_at = ? WHERE id = ?')
.run(now + Math.max(0, overlapMs), secret, now, id);
.run(now + Math.max(0, overlapMs), secret, now, id);
- return getSubscription(id);
+ return await getSubscription(id);
}
}
- function setSubscriptionEnabled(id, enabled) {
- db.prepare('UPDATE subscriptions SET enabled = ?, updated_at = ? WHERE id = ?').run(enabled ? 1 : 0, clock.now(), id);
- return getSubscription(id);
+ async function setSubscriptionEnabled(id, enabled) {
+ await db.prepare('UPDATE subscriptions SET enabled = ?, updated_at = ? WHERE id = ?').run(enabled ? 1 : 0, clock.now(), id);
+ return await getSubscription(id);
}
}
// ── Deliveries ───────────────────────────────────────────
// ── Deliveries ───────────────────────────────────────────
@@ -514,9 +409,9 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -514,9 +409,9 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
* Next due delivery per subscription (at most one each, so per-subscription concurrency stays 1),
* Next due delivery per subscription (at most one each, so per-subscription concurrency stays 1),
* ordered by priority class, then seq. `busy` are subscription ids with a delivery in flight.
* ordered by priority class, then seq. `busy` are subscription ids with a delivery in flight.
*/
*/
- function dueDeliveries(now, limit, busy = [], { firstPartyOnly = false } = {}) {
+ async function dueDeliveries(now, limit, busy = [], { firstPartyOnly = false } = {}) {
// app_project_id: set for developer-app subscriptions (the worker caps their share of slots).
// app_project_id: set for developer-app subscriptions (the worker caps their share of slots).
- const rows = db.prepare(`
+ const rows = await db.prepare(`
SELECT * FROM (
SELECT * FROM (
SELECT d.*, s.project_id AS app_project_id, ROW_NUMBER() OVER (PARTITION BY d.subscription_id ORDER BY d.priority, d.seq) AS rn
SELECT d.*, s.project_id AS app_project_id, ROW_NUMBER() OVER (PARTITION BY d.subscription_id ORDER BY d.priority, d.seq) AS rn
FROM deliveries d JOIN subscriptions s ON s.id = d.subscription_id
FROM deliveries d JOIN subscriptions s ON s.id = d.subscription_id
@@ -532,20 +427,20 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -532,20 +427,20 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
* subscription: the attempt counts toward the project's events.app.subscribe usage in the same
* subscription: the attempt counts toward the project's events.app.subscribe usage in the same
* transaction.
* transaction.
*/
*/
- const recordAttempt = db.transaction((eventId, subscriptionId, outcome, app = null) => {
+ const recordAttempt = async (eventId, subscriptionId, outcome, app = null) => await db.tx(async () => {
const now = clock.now();
const now = clock.now();
if (app) {
if (app) {
- usage.record({
+ await usage.record({
projectId: app.projectId, env: app.env, capability: 'events.app.subscribe', unit: 'deliveries', quantity: 1, at: now,
projectId: app.projectId, env: app.env, capability: 'events.app.subscribe', unit: 'deliveries', quantity: 1, at: now,
error: outcome.ok ? null : { code: deliveryCode(outcome), status: outcome.status || undefined, traceId: app.traceId, ref: eventId },
error: outcome.ok ? null : { code: deliveryCode(outcome), status: outcome.status || undefined, traceId: app.traceId, ref: eventId },
});
});
}
}
if (outcome.ok) {
if (outcome.ok) {
- db.prepare(`UPDATE deliveries SET status = 'delivered', attempt = ?, delivered_at = ?, last_status = ?, last_error = NULL,
+ await db.prepare(`UPDATE deliveries SET status = 'delivered', attempt = ?, delivered_at = ?, last_status = ?, last_error = NULL,
next_attempt_at = NULL, updated_at = ? WHERE event_id = ? AND subscription_id = ?`)
next_attempt_at = NULL, updated_at = ? WHERE event_id = ? AND subscription_id = ?`)
.run(outcome.attempt, now, outcome.status ?? null, now, eventId, subscriptionId);
.run(outcome.attempt, now, outcome.status ?? null, now, eventId, subscriptionId);
} else {
} else {
- db.prepare(`UPDATE deliveries SET status = ?, attempt = ?, last_status = ?, last_error = ?, next_attempt_at = ?, updated_at = ?
+ await db.prepare(`UPDATE deliveries SET status = ?, attempt = ?, last_status = ?, last_error = ?, next_attempt_at = ?, updated_at = ?
WHERE event_id = ? AND subscription_id = ?`)
WHERE event_id = ? AND subscription_id = ?`)
.run(outcome.dead ? 'dead' : 'failed', outcome.attempt, outcome.status ?? null, String(outcome.error || '').slice(0, 500),
.run(outcome.dead ? 'dead' : 'failed', outcome.attempt, outcome.status ?? null, String(outcome.error || '').slice(0, 500),
outcome.dead ? null : outcome.nextAttemptAt, now, eventId, subscriptionId);
outcome.dead ? null : outcome.nextAttemptAt, now, eventId, subscriptionId);
@@ -553,37 +448,37 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -553,37 +448,37 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
});
});
/** Send the closed hours' usage rollups (./usage.js); returns the stored rows. */
/** Send the closed hours' usage rollups (./usage.js); returns the stored rows. */
- function flushUsage(opts) {
- return usage.flush(insertBatch, opts);
+ async function flushUsage(opts) {
+ return await usage.flush(insertBatch, opts);
}
}
- function getDelivery(eventId, subscriptionId) {
- return db.prepare('SELECT * FROM deliveries WHERE event_id = ? AND subscription_id = ?').get(eventId, subscriptionId) || null;
+ async function getDelivery(eventId, subscriptionId) {
+ return await db.prepare('SELECT * FROM deliveries WHERE event_id = ? AND subscription_id = ?').get(eventId, subscriptionId) || null;
}
}
- function listDeliveries({ status, subscriptionId, limit = 100, afterSeq = 0 } = {}) {
+ async function listDeliveries({ status, subscriptionId, limit = 100, afterSeq = 0 } = {}) {
const where = ['seq > ?'];
const where = ['seq > ?'];
const args = [afterSeq];
const args = [afterSeq];
if (status) { where.push('status = ?'); args.push(status); }
if (status) { where.push('status = ?'); args.push(status); }
if (subscriptionId) { where.push('subscription_id = ?'); args.push(subscriptionId); }
if (subscriptionId) { where.push('subscription_id = ?'); args.push(subscriptionId); }
args.push(limit);
args.push(limit);
- return db.prepare(`SELECT * FROM deliveries WHERE ${where.join(' AND ')} ORDER BY seq, subscription_id LIMIT ?`).all(...args);
+ return await db.prepare(`SELECT * FROM deliveries WHERE ${where.join(' AND ')} ORDER BY seq, subscription_id LIMIT ?`).all(...args);
}
}
/**
/**
* Queue (again) the given events for one subscription: reset to pending, attempt 0, due now.
* Queue (again) the given events for one subscription: reset to pending, attempt 0, due now.
* Only retained events that match the subscription's topic are queued. Returns the count.
* Only retained events that match the subscription's topic are queued. Returns the count.
*/
*/
- const requeue = db.transaction((sub, { fromSeq, eventIds, max = 10000 }) => {
+ const requeue = async (sub, { fromSeq, eventIds, max = 10000 }) => await db.tx(async () => {
const now = clock.now();
const now = clock.now();
let rows;
let rows;
const cols = 'id, seq, priority, event_type, visibility, project_id, env';
const cols = 'id, seq, priority, event_type, visibility, project_id, env';
if (Array.isArray(eventIds)) {
if (Array.isArray(eventIds)) {
const get = db.prepare(`SELECT ${cols} FROM events WHERE id = ?`);
const get = db.prepare(`SELECT ${cols} FROM events WHERE id = ?`);
- rows = eventIds.map(id => get.get(id)).filter(Boolean);
+ rows = (await Promise.all(eventIds.map(async id => await get.get(id)))).filter(Boolean);
} else {
} else {
- rows = db.prepare(`SELECT ${cols} FROM events WHERE seq >= ? AND event_type GLOB ? ORDER BY seq LIMIT ?`)
- .all(fromSeq, topics.toGlob(sub.topic_pattern), max * 2);
+ rows = await db.prepare(`SELECT ${cols} FROM events WHERE seq >= ? AND event_type ILIKE ? ESCAPE '\\' ORDER BY seq LIMIT ?`)
+ .all(fromSeq, topics.toLike(sub.topic_pattern), max * 2);
}
}
// Replay never widens a subscription's scope (app scope, sandbox separation).
// Replay never widens a subscription's scope (app scope, sandbox separation).
rows = rows.filter(r => apps.deliverable(sub, r)).slice(0, max);
rows = rows.filter(r => apps.deliverable(sub, r)).slice(0, max);
@@ -591,27 +486,27 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -591,27 +486,27 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
VALUES (?, ?, ?, ?, 0, 'pending', ?, ?, ?)
VALUES (?, ?, ?, ?, 0, 'pending', ?, ?, ?)
ON CONFLICT(event_id, subscription_id) DO UPDATE SET status = 'pending', attempt = 0, next_attempt_at = excluded.next_attempt_at,
ON CONFLICT(event_id, subscription_id) DO UPDATE SET status = 'pending', attempt = 0, next_attempt_at = excluded.next_attempt_at,
last_error = NULL, last_status = NULL, delivered_at = NULL, updated_at = excluded.updated_at`);
last_error = NULL, last_status = NULL, delivered_at = NULL, updated_at = excluded.updated_at`);
- for (const r of rows) up.run(r.id, sub.id, r.seq, PRIORITY_RANK[r.priority], now, now, now);
+ for (const r of rows) await up.run(r.id, sub.id, r.seq, PRIORITY_RANK[r.priority], now, now, now);
return rows.length;
return rows.length;
});
});
- function deliveryCounts() {
+ async function deliveryCounts() {
const out = { pending: 0, delivered: 0, failed: 0, dead: 0 };
const out = { pending: 0, delivered: 0, failed: 0, dead: 0 };
- for (const r of db.prepare('SELECT status, COUNT(*) AS n FROM deliveries GROUP BY status').all()) out[r.status] = r.n;
+ for (const r of await db.prepare('SELECT status, COUNT(*) AS n FROM deliveries GROUP BY status').all()) out[r.status] = r.n;
return out;
return out;
}
}
// ── Checkpoints ──────────────────────────────────────────
// ── Checkpoints ──────────────────────────────────────────
- function getCheckpoint(consumer, topicPattern) {
- const r = db.prepare('SELECT cursor, updated_at FROM consumer_checkpoints WHERE consumer = ? AND topic_pattern = ?').get(consumer, topicPattern);
+ async function getCheckpoint(consumer, topicPattern) {
+ const r = await db.prepare('SELECT cursor, updated_at FROM consumer_checkpoints WHERE consumer = ? AND topic_pattern = ?').get(consumer, topicPattern);
return r ? { cursor: r.cursor, updated_at: new Date(r.updated_at).toISOString() } : null;
return r ? { cursor: r.cursor, updated_at: new Date(r.updated_at).toISOString() } : null;
}
}
- function setCheckpoint(consumer, topicPattern, cursor) {
- db.prepare(`INSERT INTO consumer_checkpoints (consumer, topic_pattern, cursor, updated_at) VALUES (?, ?, ?, ?)
+ async function setCheckpoint(consumer, topicPattern, cursor) {
+ await db.prepare(`INSERT INTO consumer_checkpoints (consumer, topic_pattern, cursor, updated_at) VALUES (?, ?, ?, ?)
ON CONFLICT(consumer, topic_pattern) DO UPDATE SET cursor = excluded.cursor, updated_at = excluded.updated_at`)
ON CONFLICT(consumer, topic_pattern) DO UPDATE SET cursor = excluded.cursor, updated_at = excluded.updated_at`)
.run(consumer, topicPattern, cursor, clock.now());
.run(consumer, topicPattern, cursor, clock.now());
- return getCheckpoint(consumer, topicPattern);
+ return await getCheckpoint(consumer, topicPattern);
}
}
// ── Retention ────────────────────────────────────────────
// ── Retention ────────────────────────────────────────────
@@ -620,18 +515,18 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -620,18 +515,18 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
* Drop events (and their deliveries) older than retentionDays (sandbox events older than
* Drop events (and their deliveries) older than retentionDays (sandbox events older than
* sandboxRetentionDays, when shorter), and old publish receipts.
* sandboxRetentionDays, when shorter), and old publish receipts.
*/
*/
- function prune({ retentionDays = 30, receiptRetentionDays = 90, sandboxRetentionDays = retentionDays, now = clock.now() } = {}) {
- let events = db.prepare('DELETE FROM events WHERE received_at < ?').run(now - retentionDays * DAY_MS).changes;
+ async function prune({ retentionDays = 30, receiptRetentionDays = 90, sandboxRetentionDays = retentionDays, now = clock.now() } = {}) {
+ let events = (await db.prepare('DELETE FROM events WHERE received_at < ?').run(now - retentionDays * DAY_MS)).changes;
if (sandboxRetentionDays < retentionDays) {
if (sandboxRetentionDays < retentionDays) {
- events += db.prepare("DELETE FROM events WHERE env = 'sandbox' AND received_at < ?").run(now - sandboxRetentionDays * DAY_MS).changes;
+ events += (await db.prepare("DELETE FROM events WHERE env = 'sandbox' AND received_at < ?").run(now - sandboxRetentionDays * DAY_MS)).changes;
}
}
- const receipts = db.prepare('DELETE FROM idempotency_receipts WHERE processed_at < ?')
- .run(now - Math.max(receiptRetentionDays, retentionDays) * DAY_MS).changes;
+ const receipts = (await db.prepare('DELETE FROM idempotency_receipts WHERE processed_at < ?')
+ .run(now - Math.max(receiptRetentionDays, retentionDays) * DAY_MS)).changes;
return { events, receipts };
return { events, receipts };
}
}
- function ping() {
- return db.prepare('SELECT 1 AS ok').get().ok === 1;
+ async function ping() {
+ return (await db.prepare('SELECT 1 AS ok').get()).ok === 1;
}
}
return {
return {
@@ -642,4 +537,4 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
@@ -642,4 +537,4 @@ function createStore(db, { clock = { now: () => Date.now() }, maxHops = 8, usage
};
};
}
}
-module.exports = { openDb, createStore, rowToEnvelope, subscriptionView, StoreError, PRIORITY_RANK, RANK_PRIORITY };
+module.exports = { openDb, MIGRATIONS, createStore, rowToEnvelope, subscriptionView, StoreError, PRIORITY_RANK, RANK_PRIORITY };
@@ -34,13 +34,13 @@ function matches(pattern, eventType) {
@@ -34,13 +34,13 @@ function matches(pattern, eventType) {
}
}
/**
/**
- * A SQLite GLOB that selects a superset of what the pattern matches (GLOB `*` also spans dots),
- * so the SQL narrows the scan and matches() makes the exact decision. Valid patterns only contain
- * [a-z0-9_.*], none of which GLOB treats specially except `*`, so the pattern is its own GLOB.
+ * A ILIKE pattern (PostgreSQL, ESCAPE '\\') that selects a superset of what the pattern matches (`%` also spans dots),
+ * so the SQL narrows the scan and matches() makes the exact decision. Valid patterns only contain [a-z0-9_.*]: `_` is
+ * escaped (it is a ILIKE wildcard) and `*` becomes `%`. ILIKE compares exactly, as GLOB did.
*/
*/
-function toGlob(pattern) {
+function toLike(pattern) {
if (!isValidPattern(pattern)) throw new TypeError(`invalid topic pattern "${pattern}"`);
if (!isValidPattern(pattern)) throw new TypeError(`invalid topic pattern "${pattern}"`);
- return pattern;
+ return pattern.replace(/_/g, '\\_').replace(/\*/g, '%');
}
}
-module.exports = { isValidPattern, compile, matches, toGlob };
+module.exports = { isValidPattern, compile, matches, toLike };
@@ -36,24 +36,7 @@ const TRACE_RE = /^[0-9a-f]{32}$/;
@@ -36,24 +36,7 @@ const TRACE_RE = /^[0-9a-f]{32}$/;
const REF_RE = /^evt_[0-9A-HJKMNP-TV-Z]{26}$/;
const REF_RE = /^evt_[0-9A-HJKMNP-TV-Z]{26}$/;
const PROJECT_RE = /^prj_[0-9A-HJKMNP-TV-Z]{26}$/;
const PROJECT_RE = /^prj_[0-9A-HJKMNP-TV-Z]{26}$/;
-const SCHEMA = `
-CREATE TABLE IF NOT EXISTS app_usage (
- project_id TEXT NOT NULL,
- env TEXT NOT NULL,
- capability TEXT NOT NULL,
- unit TEXT NOT NULL,
- window_start INTEGER NOT NULL,
- quantity INTEGER NOT NULL DEFAULT 0,
- errors INTEGER NOT NULL DEFAULT 0,
- error_codes TEXT NOT NULL DEFAULT '{}',
- samples TEXT NOT NULL DEFAULT '[]',
- revision INTEGER NOT NULL DEFAULT 1,
- event_id TEXT,
- emitted_at INTEGER,
- PRIMARY KEY (project_id, env, capability, unit, window_start)
-);
-CREATE INDEX IF NOT EXISTS idx_app_usage_open ON app_usage(emitted_at, window_start);
-`;
+// The app_usage table is in migrations/0001_initial.sql.
const hourOf = (ms) => Math.floor(ms / HOUR_MS) * HOUR_MS;
const hourOf = (ms) => Math.floor(ms / HOUR_MS) * HOUR_MS;
const parse = (s, d) => { try { return JSON.parse(s); } catch { return d; } };
const parse = (s, d) => { try { return JSON.parse(s); } catch { return d; } };
@@ -71,13 +54,12 @@ function deliveryCode(outcome) {
@@ -71,13 +54,12 @@ function deliveryCode(outcome) {
* @param {boolean} [o.enabled=true] false (EVENTS_USAGE=off): nothing is counted
* @param {boolean} [o.enabled=true] false (EVENTS_USAGE=off): nothing is counted
*/
*/
function createUsage(db, { clock = { now: () => Date.now() }, enabled = true } = {}) {
function createUsage(db, { clock = { now: () => Date.now() }, enabled = true } = {}) {
- db.exec(SCHEMA);
const q = {
const q = {
add: db.prepare(`INSERT INTO app_usage (project_id, env, capability, unit, window_start, quantity, errors)
add: db.prepare(`INSERT INTO app_usage (project_id, env, capability, unit, window_start, quantity, errors)
VALUES (@project_id, @env, @capability, @unit, @window_start, @quantity, @errors)
VALUES (@project_id, @env, @capability, @unit, @window_start, @quantity, @errors)
ON CONFLICT(project_id, env, capability, unit, window_start) DO UPDATE SET
ON CONFLICT(project_id, env, capability, unit, window_start) DO UPDATE SET
- quantity = quantity + excluded.quantity, errors = errors + excluded.errors,
- revision = revision + (CASE WHEN emitted_at IS NULL THEN 0 ELSE 1 END), emitted_at = NULL`),
+ quantity = app_usage.quantity + excluded.quantity, errors = app_usage.errors + excluded.errors,
+ revision = app_usage.revision + (CASE WHEN app_usage.emitted_at IS NULL THEN 0 ELSE 1 END), emitted_at = NULL`),
get: db.prepare('SELECT error_codes, samples FROM app_usage WHERE project_id = ? AND env = ? AND capability = ? AND unit = ? AND window_start = ?'),
get: db.prepare('SELECT error_codes, samples FROM app_usage WHERE project_id = ? AND env = ? AND capability = ? AND unit = ? AND window_start = ?'),
setErrors: db.prepare(`UPDATE app_usage SET error_codes = ?, samples = ?
setErrors: db.prepare(`UPDATE app_usage SET error_codes = ?, samples = ?
WHERE project_id = ? AND env = ? AND capability = ? AND unit = ? AND window_start = ?`),
WHERE project_id = ? AND env = ? AND capability = ? AND unit = ? AND window_start = ?`),
@@ -91,13 +73,13 @@ function createUsage(db, { clock = { now: () => Date.now() }, enabled = true } =
@@ -91,13 +73,13 @@ function createUsage(db, { clock = { now: () => Date.now() }, enabled = true } =
* Count usage (and at most one failure) for a project, inside the caller's transaction when there
* Count usage (and at most one failure) for a project, inside the caller's transaction when there
* is one. `error` = { code, status?, traceId?, ref? } counts one error with a sample.
* is one. `error` = { code, status?, traceId?, ref? } counts one error with a sample.
*/
*/
- const record = db.transaction(({ projectId, env, capability, unit, quantity = 0, error = null, at = clock.now() }) => {
+ const record = async ({ projectId, env, capability, unit, quantity = 0, error = null, at = clock.now() }) => await db.tx(async () => {
if (!enabled || !PROJECT_RE.test(String(projectId || '')) || (env !== 'sandbox' && env !== 'production')) return;
if (!enabled || !PROJECT_RE.test(String(projectId || '')) || (env !== 'sandbox' && env !== 'production')) return;
const key = { project_id: projectId, env, capability, unit, window_start: hourOf(at) };
const key = { project_id: projectId, env, capability, unit, window_start: hourOf(at) };
- q.add.run({ ...key, quantity: Math.max(0, Math.trunc(Number(quantity) || 0)), errors: error ? 1 : 0 });
+ await q.add.run({ ...key, quantity: Math.max(0, Math.trunc(Number(quantity) || 0)), errors: error ? 1 : 0 });
if (!error) return;
if (!error) return;
const code = CODE_RE.test(String(error.code || '')) ? String(error.code) : 'events.error';
const code = CODE_RE.test(String(error.code || '')) ? String(error.code) : 'events.error';
- const row = q.get.get(projectId, env, capability, unit, key.window_start);
+ const row = await q.get.get(projectId, env, capability, unit, key.window_start);
const codes = parse(row.error_codes, {});
const codes = parse(row.error_codes, {});
if (codes[code] || Object.keys(codes).length < MAX_CODES) codes[code] = (codes[code] || 0) + 1;
if (codes[code] || Object.keys(codes).length < MAX_CODES) codes[code] = (codes[code] || 0) + 1;
const sample = { at: new Date(at).toISOString(), code };
const sample = { at: new Date(at).toISOString(), code };
@@ -105,7 +87,7 @@ function createUsage(db, { clock = { now: () => Date.now() }, enabled = true } =
@@ -105,7 +87,7 @@ function createUsage(db, { clock = { now: () => Date.now() }, enabled = true } =
if (TRACE_RE.test(String(error.traceId || ''))) sample.trace_id = error.traceId;
if (TRACE_RE.test(String(error.traceId || ''))) sample.trace_id = error.traceId;
if (REF_RE.test(String(error.ref || ''))) sample.ref = error.ref;
if (REF_RE.test(String(error.ref || ''))) sample.ref = error.ref;
const samples = [sample, ...parse(row.samples, [])].slice(0, MAX_SAMPLES);
const samples = [sample, ...parse(row.samples, [])].slice(0, MAX_SAMPLES);
- q.setErrors.run(JSON.stringify(codes), JSON.stringify(samples), projectId, env, capability, unit, key.window_start);
+ await q.setErrors.run(JSON.stringify(codes), JSON.stringify(samples), projectId, env, capability, unit, key.window_start);
});
});
function payloadOf(row) {
function payloadOf(row) {
@@ -126,10 +108,10 @@ function createUsage(db, { clock = { now: () => Date.now() }, enabled = true } =
@@ -126,10 +108,10 @@ function createUsage(db, { clock = { now: () => Date.now() }, enabled = true } =
* Store every closed hour's rollups as events.usage.recorded (at most `limit`). `insertBatch` is
* Store every closed hour's rollups as events.usage.recorded (at most `limit`). `insertBatch` is
* the store's; the returned rows are what it stored, for the realtime fan-out and the worker kick.
* the store's; the returned rows are what it stored, for the realtime fan-out and the worker kick.
*/
*/
- function flush(insertBatch, { now = clock.now(), limit = 500 } = {}) {
+ async function flush(insertBatch, { now = clock.now(), limit = 500 } = {}) {
const stored = [];
const stored = [];
let invalid = 0;
let invalid = 0;
- const rows = q.due.all(hourOf(now - GRACE_MS) - HOUR_MS, limit);
+ const rows = await q.due.all(hourOf(now - GRACE_MS) - HOUR_MS, limit);
for (const row of rows) {
for (const row of rows) {
const payload = payloadOf(row);
const payload = payloadOf(row);
const envelope = {
const envelope = {
@@ -145,19 +127,19 @@ function createUsage(db, { clock = { now: () => Date.now() }, enabled = true } =
@@ -145,19 +127,19 @@ function createUsage(db, { clock = { now: () => Date.now() }, enabled = true } =
invalid++;
invalid++;
continue;
continue;
}
}
- db.transaction(() => {
- const out = insertBatch([envelope], { publisher: 'svc:events', requestId: null });
- q.sent.run(now, envelope.event_id, row.project_id, row.env, row.capability, row.unit, row.window_start);
+ await db.tx(async () => {
+ const out = await insertBatch([envelope], { publisher: 'svc:events', requestId: null });
+ await q.sent.run(now, envelope.event_id, row.project_id, row.env, row.capability, row.unit, row.window_start);
stored.push(...out.inserted);
stored.push(...out.inserted);
- })();
+ });
}
}
- q.prune.run(now - KEEP_SENT_MS);
+ await q.prune.run(now - KEEP_SENT_MS);
return { stored, invalid };
return { stored, invalid };
}
}
return {
return {
record, flush, payloadOf, deliveryCode,
record, flush, payloadOf, deliveryCode,
- pending: () => q.pending.get().n,
+ pending: async () => (await q.pending.get()).n,
};
};
}
}
@@ -61,8 +61,8 @@ function createWorker({ store, config, clock = { now: () => Date.now() }, fetchI
@@ -61,8 +61,8 @@ function createWorker({ store, config, clock = { now: () => Date.now() }, fetchI
}
}
async function send(delivery) {
async function send(delivery) {
- const sub = store.getSubscription(delivery.subscription_id);
- const row = store.getEvent(delivery.event_id);
+ const sub = await store.getSubscription(delivery.subscription_id);
+ const row = await store.getEvent(delivery.event_id);
const attempt = delivery.attempt + 1;
const attempt = delivery.attempt + 1;
if (!sub || !row) return; // pruned or removed while queued; cascade deletes the delivery
if (!sub || !row) return; // pruned or removed while queued; cascade deletes the delivery
const { maxAttempts, backoffMs } = policy(sub);
const { maxAttempts, backoffMs } = policy(sub);
@@ -112,7 +112,7 @@ function createWorker({ store, config, clock = { now: () => Date.now() }, fetchI
@@ -112,7 +112,7 @@ function createWorker({ store, config, clock = { now: () => Date.now() }, fetchI
}
}
// A developer app's subscription: the attempt counts toward its project's usage (./usage.js).
// A developer app's subscription: the attempt counts toward its project's usage (./usage.js).
const app = sub.project_id ? { projectId: sub.project_id, env: sub.env || 'production', traceId: row.trace_id } : null;
const app = sub.project_id ? { projectId: sub.project_id, env: sub.env || 'production', traceId: row.trace_id } : null;
- store.recordAttempt(delivery.event_id, delivery.subscription_id, outcome, app);
+ await store.recordAttempt(delivery.event_id, delivery.subscription_id, outcome, app);
if (observe) {
if (observe) {
try {
try {
observe.attempt(outcome.ok ? 'delivered' : outcome.dead ? 'dead' : 'retry');
observe.attempt(outcome.ok ? 'delivered' : outcome.dead ? 'dead' : 'retry');
@@ -122,8 +122,21 @@ function createWorker({ store, config, clock = { now: () => Date.now() }, fetchI
@@ -122,8 +122,21 @@ function createWorker({ store, config, clock = { now: () => Date.now() }, fetchI
}
}
}
}
- /** Start every due delivery there is room for. Returns how many were started. */
+ // One pass at a time: the due query and start() must not interleave, or a second pass would read the rows the
+ // first is starting and send them twice. A call during a pass gets one more pass right after it.
+ let passing = null;
+ let queued = null;
function dispatch() {
function dispatch() {
+ if (!passing) {
+ passing = dispatchPass().finally(() => { passing = null; });
+ return passing;
+ }
+ if (!queued) queued = passing.catch(() => {}).then(() => { queued = null; return dispatch(); });
+ return queued;
+ }
+
+ /** Start every due delivery there is room for. Returns how many were started. */
+ async function dispatchPass() {
lastTickAt = Date.now();
lastTickAt = Date.now();
const room = opts.maxInflight - inflight.size;
const room = opts.maxInflight - inflight.size;
if (room <= 0) return 0;
if (room <= 0) return 0;
@@ -138,10 +151,10 @@ function createWorker({ store, config, clock = { now: () => Date.now() }, fetchI
@@ -138,10 +151,10 @@ function createWorker({ store, config, clock = { now: () => Date.now() }, fetchI
started++;
started++;
}
}
};
};
- startAll(store.dueDeliveries(clock.now(), room, [...busy], { firstPartyOnly: appInflight >= opts.maxAppInflight }));
+ startAll(await store.dueDeliveries(clock.now(), room, [...busy], { firstPartyOnly: appInflight >= opts.maxAppInflight }));
// App deliveries (which can outrank first-party ones by priority) filled the app share: the
// App deliveries (which can outrank first-party ones by priority) filled the app share: the
// rest of the room goes to first-party subscriptions.
// rest of the room goes to first-party subscriptions.
- if (skippedApp && started < room) startAll(store.dueDeliveries(clock.now(), room - started, [...busy], { firstPartyOnly: true }));
+ if (skippedApp && started < room) startAll(await store.dueDeliveries(clock.now(), room - started, [...busy], { firstPartyOnly: true }));
return started;
return started;
}
}
@@ -163,9 +176,9 @@ function createWorker({ store, config, clock = { now: () => Date.now() }, fetchI
@@ -163,9 +176,9 @@ function createWorker({ store, config, clock = { now: () => Date.now() }, fetchI
if (!running) return;
if (!running) return;
if (timer && ms > 0) return; // a tick is already coming
if (timer && ms > 0) return; // a tick is already coming
clearTimeout(timer);
clearTimeout(timer);
- timer = setTimeout(() => {
+ timer = setTimeout(async () => {
timer = null;
timer = null;
- try { dispatch(); } catch (err) { log.error(`[worker] tick failed: ${err.stack || err}`); }
+ try { await dispatch(); } catch (err) { log.error(`[worker] tick failed: ${err.stack || err}`); }
schedule(opts.intervalMs);
schedule(opts.intervalMs);
}, ms);
}, ms);
timer.unref?.();
timer.unref?.();
@@ -174,7 +187,7 @@ function createWorker({ store, config, clock = { now: () => Date.now() }, fetchI
@@ -174,7 +187,7 @@ function createWorker({ store, config, clock = { now: () => Date.now() }, fetchI
/** Deliver everything due now (and whatever becomes due as a result), then resolve. Tests. */
/** Deliver everything due now (and whatever becomes due as a result), then resolve. Tests. */
async function drain() {
async function drain() {
for (;;) {
for (;;) {
- const started = dispatch();
+ const started = await dispatch();
if (!started && !inflight.size) return;
if (!started && !inflight.size) return;
if (inflight.size) await Promise.race([...inflight]);
if (inflight.size) await Promise.race([...inflight]);
}
}
@@ -27,8 +27,8 @@ const appEvent = () => envelope('x', {
@@ -27,8 +27,8 @@ const appEvent = () => envelope('x', {
source: apps.appSource(appId), event_type: `app.${apps.projectKey(projectId)}.order.created`,
source: apps.appSource(appId), event_type: `app.${apps.projectKey(projectId)}.order.created`,
actor: { type: 'app', id: appId }, subject: { type: 'order', id: '42' }, visibility: 'internal',
actor: { type: 'app', id: appId }, subject: { type: 'order', id: '42' }, visibility: 'internal',
});
});
-const get = (token, p) => request(h.base, 'GET', p, { token });
-const publish = (token, body) => request(h.base, 'POST', '/api/v1/events', { token, body });
+const get = async (token, p) => await request(h.base, 'GET', p, { token });
+const publish = async (token, body) => await request(h.base, 'POST', '/api/v1/events', { token, body });
t('boot (EVENTS_LIMITS_MINUTE=3; the app event quota off, so only the per-actor limit answers)', async () => {
t('boot (EVENTS_LIMITS_MINUTE=3; the app event quota off, so only the per-actor limit answers)', async () => {
h = await boot({ clock, env: { EVENTS_LIMITS_MINUTE: '3', EVENTS_LIMITS_HOUR: '100', EVENTS_APP_PUBLISH_PER_MINUTE: '0' } });
h = await boot({ clock, env: { EVENTS_LIMITS_MINUTE: '3', EVENTS_LIMITS_HOUR: '100', EVENTS_APP_PUBLISH_PER_MINUTE: '0' } });
@@ -65,10 +65,10 @@ t('publish: an app gets 60 requests a minute; the 61st is refused before anythin
@@ -65,10 +65,10 @@ t('publish: an app gets 60 requests a minute; the 61st is refused before anythin
const r = await publish(app, appEvent());
const r = await publish(app, appEvent());
assert.strictEqual(r.status, 201, r.text);
assert.strictEqual(r.status, 201, r.text);
}
}
- const before = h.store.lastSeq();
+ const before = await h.store.lastSeq();
const r = await publish(app, appEvent());
const r = await publish(app, appEvent());
assert.deepStrictEqual([r.status, r.body.code, r.headers.get('retry-after')], [429, 'rate_limited', '60']);
assert.deepStrictEqual([r.status, r.body.code, r.headers.get('retry-after')], [429, 'rate_limited', '60']);
- assert.strictEqual(h.store.lastSeq(), before, 'nothing stored');
+ assert.strictEqual(await h.store.lastSeq(), before, 'nothing stored');
assert.strictEqual((await publish(live, envelope('live'))).status, 201, 'a service still publishes');
assert.strictEqual((await publish(live, envelope('live'))).status, 201, 'a service still publishes');
});
});
@@ -122,7 +122,7 @@ t('worker: app deliveries never take every in-flight slot', async () => {
@@ -122,7 +122,7 @@ t('worker: app deliveries never take every in-flight slot', async () => {
}
}
const r = await request(h.base, 'POST', '/api/v1/events', { token: serviceToken('live', ['events.event.publish']), body: envelope('live') });
const r = await request(h.base, 'POST', '/api/v1/events', { token: serviceToken('live', ['events.event.publish']), body: envelope('live') });
assert.strictEqual(r.status, 201);
assert.strictEqual(r.status, 201);
- h.worker.dispatch();
+ await h.worker.dispatch();
for (let i = 0; i < 40 && !firstParty.calls.length; i++) await sleep(25);
for (let i = 0; i < 40 && !firstParty.calls.length; i++) await sleep(25);
assert.strictEqual(firstParty.calls.length, 1, 'the first-party delivery got a slot');
assert.strictEqual(firstParty.calls.length, 1, 'the first-party delivery got a slot');
assert.ok(pending.length >= 1, 'app deliveries still run');
assert.ok(pending.length >= 1, 'app deliveries still run');
@@ -71,8 +71,8 @@ function appEvent(appId, projectId, name = 'order.created', over = {}) {
@@ -71,8 +71,8 @@ function appEvent(appId, projectId, name = 'order.created', over = {}) {
actor: { type: 'app', id: appId }, subject: { type: 'order', id: '42' }, visibility: 'internal', ...over,
actor: { type: 'app', id: appId }, subject: { type: 'order', id: '42' }, visibility: 'internal', ...over,
});
});
}
}
-const publish = (token, body) => request(h.base, 'POST', '/api/v1/events', { token, body });
-const pull = (token, topic, after = 0) => request(h.base, 'GET', `/api/v1/events?topic=${encodeURIComponent(topic)}&after_seq=${after}`, { token });
+const publish = async (token, body) => await request(h.base, 'POST', '/api/v1/events', { token, body });
+const pull = async (token, topic, after = 0) => await request(h.base, 'GET', `/api/v1/events?topic=${encodeURIComponent(topic)}&after_seq=${after}`, { token });
t('boot', async () => {
t('boot', async () => {
await new Promise(r => receiver.listen(0, '127.0.0.1', r));
await new Promise(r => receiver.listen(0, '127.0.0.1', r));
@@ -95,7 +95,7 @@ t('project_key and app source are defined exactly (and fit the envelope contract
@@ -95,7 +95,7 @@ t('project_key and app source are defined exactly (and fit the envelope contract
t('publish: an app publishes app.<project_key>.* only, as its own source and actor', async () => {
t('publish: an app publishes app.<project_key>.* only, as its own source and actor', async () => {
let r = await publish(tokA(), appEvent(appA, prjA));
let r = await publish(tokA(), appEvent(appA, prjA));
assert.strictEqual(r.status, 201, r.text);
assert.strictEqual(r.status, 201, r.text);
- const row = h.store.getEvent(r.body.event_id);
+ const row = await h.store.getEvent(r.body.event_id);
assert.strictEqual(row.project_id, prjA);
assert.strictEqual(row.project_id, prjA);
assert.strictEqual(row.env, 'sandbox');
assert.strictEqual(row.env, 'sandbox');
assert.strictEqual(row.publisher, `app:${appA}`);
assert.strictEqual(row.publisher, `app:${appA}`);
@@ -187,7 +187,7 @@ t('read: own project in the same environment, plus public first-party events', a
@@ -187,7 +187,7 @@ t('read: own project in the same environment, plus public first-party events', a
let subA;
let subA;
t('subscribe: scoped patterns, public https endpoints only', async () => {
t('subscribe: scoped patterns, public https endpoints only', async () => {
- const sub = (token, body) => request(h.base, 'POST', '/api/v1/subscriptions', { token, body });
+ const sub = async (token, body) => await request(h.base, 'POST', '/api/v1/subscriptions', { token, body });
for (const endpoint of ['http://hooks.example.com/x', 'https://127.0.0.1/x', 'https://[::1]/x', 'https://10.1.2.3/x', 'https://169.254.169.254/latest',
for (const endpoint of ['http://hooks.example.com/x', 'https://127.0.0.1/x', 'https://[::1]/x', 'https://10.1.2.3/x', 'https://169.254.169.254/latest',
'https://evil.example.com/x', 'https://localhost/x', 'https://u:[email protected]/x', 'https://nowhere.example.com/x', 'https://intranet/x']) {
'https://evil.example.com/x', 'https://localhost/x', 'https://u:[email protected]/x', 'https://nowhere.example.com/x', 'https://intranet/x']) {
const r = await sub(tokA(), { topic_pattern: `app.${keyA}.*`, endpoint });
const r = await sub(tokA(), { topic_pattern: `app.${keyA}.*`, endpoint });
@@ -282,10 +282,10 @@ t('delivery-time SSRF guard: re-resolved on every attempt, redirects never follo
@@ -282,10 +282,10 @@ t('delivery-time SSRF guard: re-resolved on every attempt, redirects never follo
const ev = await publish(tokB(), appEvent(appB, prjB, 'rebind.test'));
const ev = await publish(tokB(), appEvent(appB, prjB, 'rebind.test'));
const ev2 = await publish(tokB(), appEvent(appB, prjB, 'redirect.test'));
const ev2 = await publish(tokB(), appEvent(appB, prjB, 'redirect.test'));
await h.worker.drain();
await h.worker.drain();
- const d = h.store.getDelivery(ev.body.event_id, r.body.id);
+ const d = await h.store.getDelivery(ev.body.event_id, r.body.id);
assert.strictEqual(d.status, 'dead', 'a private address at delivery time is refused for good');
assert.strictEqual(d.status, 'dead', 'a private address at delivery time is refused for good');
assert.match(d.last_error, /non-public/);
assert.match(d.last_error, /non-public/);
- const d2 = h.store.getDelivery(ev2.body.event_id, redir.body.id);
+ const d2 = await h.store.getDelivery(ev2.body.event_id, redir.body.id);
assert.strictEqual(d2.status, 'failed'); assert.strictEqual(d2.last_status, 302);
assert.strictEqual(d2.status, 'failed'); assert.strictEqual(d2.last_status, 302);
assert.deepStrictEqual(seen.map(s => s.url).filter(u => u !== '/b-project'), ['/redirect'], 'one request, the redirect was not followed, nothing reached the rebound host');
assert.deepStrictEqual(seen.map(s => s.url).filter(u => u !== '/b-project'), ['/redirect'], 'one request, the redirect was not followed, nothing reached the rebound host');
});
});
@@ -293,7 +293,7 @@ t('delivery-time SSRF guard: re-resolved on every attempt, redirects never follo
@@ -293,7 +293,7 @@ t('delivery-time SSRF guard: re-resolved on every attempt, redirects never follo
t('quotas: per-project publish rate and retained bytes', async () => {
t('quotas: per-project publish rate and retained bytes', async () => {
const h2 = await boot({ appPost: testPost, dnsLookup: fakeLookup, env: { EVENTS_APP_PUBLISH_PER_MINUTE: '3', EVENTS_APP_SANDBOX_RETAINED_BYTES: '3000' } });
const h2 = await boot({ appPost: testPost, dnsLookup: fakeLookup, env: { EVENTS_APP_PUBLISH_PER_MINUTE: '3', EVENTS_APP_SANDBOX_RETAINED_BYTES: '3000' } });
try {
try {
- const pub = (token, body) => request(h2.base, 'POST', '/api/v1/events', { token, body });
+ const pub = async (token, body) => await request(h2.base, 'POST', '/api/v1/events', { token, body });
for (let i = 0; i < 3; i++) assert.strictEqual((await pub(tokA2(), appEvent(appA2, prjA))).status, 201);
for (let i = 0; i < 3; i++) assert.strictEqual((await pub(tokA2(), appEvent(appA2, prjA))).status, 201);
let r = await pub(tokA2(), appEvent(appA2, prjA));
let r = await pub(tokA2(), appEvent(appA2, prjA));
assert.strictEqual(r.status, 429); assert.strictEqual(r.body.code, 'events.quota_exceeded');
assert.strictEqual(r.status, 429); assert.strictEqual(r.body.code, 'events.quota_exceeded');
@@ -339,7 +339,7 @@ t('revocation from Network stops an app\'s subscriptions and old tokens', async
@@ -339,7 +339,7 @@ t('revocation from Network stops an app\'s subscriptions and old tokens', async
subject: { type: 'app', id: appA }, payload: { project_id: prjA, environment: 'sandbox', reason: 'test' },
subject: { type: 'app', id: appA }, payload: { project_id: prjA, environment: 'sandbox', reason: 'test' },
}));
}));
assert.strictEqual(r.status, 201, r.text);
assert.strictEqual(r.status, 201, r.text);
- assert.ok(h.store.listSubscriptions(`app:${appA}`).every(s => !s.enabled));
+ assert.ok((await h.store.listSubscriptions(`app:${appA}`)).every(s => !s.enabled));
r = await pull(tokA(), `app.${keyA}.*`);
r = await pull(tokA(), `app.${keyA}.*`);
assert.strictEqual(r.status, 401); assert.strictEqual(r.body.code, 'token.revoked');
assert.strictEqual(r.status, 401); assert.strictEqual(r.body.code, 'token.revoked');
r = await publish(tokA(), appEvent(appA, prjA));
r = await publish(tokA(), appEvent(appA, prjA));
@@ -358,7 +358,7 @@ t('realtime never streams app events, to anyone', async () => {
@@ -358,7 +358,7 @@ t('realtime never streams app events, to anyone', async () => {
await anon.waitFor(c => c.events().some(e => e.event.event_id === marker.body.event_id));
await anon.waitFor(c => c.events().some(e => e.event.event_id === marker.body.event_id));
assert.ok(!svc.events().some(e => e.event.event_type.startsWith('app.')));
assert.ok(!svc.events().some(e => e.event.event_type.startsWith('app.')));
assert.ok(!anon.events().some(e => e.event.event_type.startsWith('app.')));
assert.ok(!anon.events().some(e => e.event.event_type.startsWith('app.')));
- svc.close(); anon.close();
+ await svc.close(); anon.close();
});
});
t('stop', async () => {
t('stop', async () => {
new file mode 100644
new file mode 100644
@@ -0,0 +1,11 @@
@@ -0,0 +1,11 @@
+'use strict';
+/**
+ * A migrated database for one test boot (ADR-035), from openvibe-sdk/testing: PGlite by default; with
+ * EVENTS_TEST_STORE=pg (npm run test:pg) the PostgreSQL + PgBouncer containers, with roles and a schema of its own.
+ */
+const { createTestDb, pgAvailable } = require('openvibe-sdk/testing');
+const { MIGRATIONS } = require('../server/store');
+
+const testDb = ({ store = process.env.EVENTS_TEST_STORE || 'pglite', max = 4 } = {}) => createTestDb({ migrations: MIGRATIONS, store, service: 'events', max });
+
+module.exports = { testDb, pgAvailable };
@@ -11,7 +11,7 @@ const media = serviceToken('media', ['events.subscription.manage']);
@@ -11,7 +11,7 @@ const media = serviceToken('media', ['events.subscription.manage']);
const admin = serviceToken('ops', ['events.delivery.admin']);
const admin = serviceToken('ops', ['events.delivery.admin']);
async function subscribe(token, body) {
async function subscribe(token, body) {
- return request(h.base, 'POST', '/api/v1/subscriptions', { token, body });
+ return await request(h.base, 'POST', '/api/v1/subscriptions', { token, body });
}
}
async function publish(env) {
async function publish(env) {
const r = await request(h.base, 'POST', '/api/v1/events', { token: live, body: env });
const r = await request(h.base, 'POST', '/api/v1/events', { token: live, body: env });
@@ -100,7 +100,7 @@ t('success: signed POST (v2 only), verified by verifyDeliveryV2; trace propagate
@@ -100,7 +100,7 @@ t('success: signed POST (v2 only), verified by verifyDeliveryV2; trace propagate
const sub = (await subscribe(media, { topic_pattern: 'live.stream.*', endpoint: stub.url })).body;
const sub = (await subscribe(media, { topic_pattern: 'live.stream.*', endpoint: stub.url })).body;
const env = envelope('live', { trace_id: '4bf92f3577b34da6a3ce929d0e0e4736' });
const env = envelope('live', { trace_id: '4bf92f3577b34da6a3ce929d0e0e4736' });
const { seq } = await publish(env);
const { seq } = await publish(env);
- assert.strictEqual(h.store.getDelivery(env.event_id, sub.id).status, 'pending', 'persisted before delivery');
+ assert.strictEqual((await h.store.getDelivery(env.event_id, sub.id)).status, 'pending', 'persisted before delivery');
await h.worker.drain();
await h.worker.drain();
assert.strictEqual(stub.calls.length, 1);
assert.strictEqual(stub.calls.length, 1);
const call = stub.calls[0];
const call = stub.calls[0];
@@ -118,7 +118,7 @@ t('success: signed POST (v2 only), verified by verifyDeliveryV2; trace propagate
@@ -118,7 +118,7 @@ t('success: signed POST (v2 only), verified by verifyDeliveryV2; trace propagate
assert.match(call.headers.traceparent, /^00-4bf92f3577b34da6a3ce929d0e0e4736-[0-9a-f]{16}-01$/);
assert.match(call.headers.traceparent, /^00-4bf92f3577b34da6a3ce929d0e0e4736-[0-9a-f]{16}-01$/);
assert.deepStrictEqual(call.body.event.payload, env.payload);
assert.deepStrictEqual(call.body.event.payload, env.payload);
assert.strictEqual(call.body.seq, seq);
assert.strictEqual(call.body.seq, seq);
- const d = h.store.getDelivery(env.event_id, sub.id);
+ const d = await h.store.getDelivery(env.event_id, sub.id);
assert.strictEqual(d.status, 'delivered');
assert.strictEqual(d.status, 'delivered');
assert.strictEqual(d.attempt, 1);
assert.strictEqual(d.attempt, 1);
await request(h.base, 'POST', `/api/v1/subscriptions/${sub.id}/disable`, { token: media });
await request(h.base, 'POST', `/api/v1/subscriptions/${sub.id}/disable`, { token: media });
@@ -139,7 +139,7 @@ t('retries with backoff (injectable clock), then DLQ after 8 attempts, then repl
@@ -139,7 +139,7 @@ t('retries with backoff (injectable clock), then DLQ after 8 attempts, then repl
assert.strictEqual(call.headers['x-openvibe-timestamp'], String(Math.floor(h.clock.now() / 1000)), 'every retry is signed with a fresh timestamp');
assert.strictEqual(call.headers['x-openvibe-timestamp'], String(Math.floor(h.clock.now() / 1000)), 'every retry is signed with a fresh timestamp');
assert.ok(verifyDeliveryV2(call.rawBody, call.headers, sub.secret, { now: h.clock.now() }));
assert.ok(verifyDeliveryV2(call.rawBody, call.headers, sub.secret, { now: h.clock.now() }));
if (attempt > 1) assert.notStrictEqual(call.headers['x-openvibe-signature-v2'], stub.calls[attempt - 2].headers['x-openvibe-signature-v2']);
if (attempt > 1) assert.notStrictEqual(call.headers['x-openvibe-signature-v2'], stub.calls[attempt - 2].headers['x-openvibe-signature-v2']);
- const d = h.store.getDelivery(env.event_id, sub.id);
+ const d = await h.store.getDelivery(env.event_id, sub.id);
assert.strictEqual(d.attempt, attempt);
assert.strictEqual(d.attempt, attempt);
if (attempt < 8) {
if (attempt < 8) {
assert.strictEqual(d.status, 'failed');
assert.strictEqual(d.status, 'failed');
@@ -170,7 +170,7 @@ t('retries with backoff (injectable clock), then DLQ after 8 attempts, then repl
@@ -170,7 +170,7 @@ t('retries with backoff (injectable clock), then DLQ after 8 attempts, then repl
assert.strictEqual(rp.body.queued, 1);
assert.strictEqual(rp.body.queued, 1);
await h.worker.drain();
await h.worker.drain();
assert.strictEqual(stub.calls.length, 9);
assert.strictEqual(stub.calls.length, 9);
- assert.strictEqual(h.store.getDelivery(env.event_id, sub.id).status, 'delivered');
+ assert.strictEqual((await h.store.getDelivery(env.event_id, sub.id)).status, 'delivered');
await request(h.base, 'POST', `/api/v1/subscriptions/${sub.id}/disable`, { token: media });
await request(h.base, 'POST', `/api/v1/subscriptions/${sub.id}/disable`, { token: media });
await stub.close();
await stub.close();
});
});
@@ -203,13 +203,13 @@ t('priority classes: critical before important before low', async () => {
@@ -203,13 +203,13 @@ t('priority classes: critical before important before low', async () => {
await publish(envelope('live', { event_type: 'live.prio.x', priority: 'low' }));
await publish(envelope('live', { event_type: 'live.prio.x', priority: 'low' }));
await publish(envelope('live', { event_type: 'live.prio.x', priority: 'important' }));
await publish(envelope('live', { event_type: 'live.prio.x', priority: 'important' }));
await publish(envelope('live', { event_type: 'live.prio.x', priority: 'critical' }));
await publish(envelope('live', { event_type: 'live.prio.x', priority: 'critical' }));
- const due = h.store.dueDeliveries(h.clock.now(), 100);
+ const due = await h.store.dueDeliveries(h.clock.now(), 100);
assert.strictEqual(due.length, 3, 'one per subscription');
assert.strictEqual(due.length, 3, 'one per subscription');
assert.ok(due.every(d => d.priority === 0), 'each subscription\'s head is its critical event');
assert.ok(due.every(d => d.priority === 0), 'each subscription\'s head is its critical event');
await h.worker.drain();
await h.worker.drain();
for (const o of orders) assert.deepStrictEqual(o, ['critical', 'important', 'low']);
for (const o of orders) assert.deepStrictEqual(o, ['critical', 'important', 'low']);
for (const s of stubs) await s.close();
for (const s of stubs) await s.close();
- for (const sub of h.store.listSubscriptions()) h.store.setSubscriptionEnabled(sub.id, false);
+ for (const sub of await h.store.listSubscriptions()) await h.store.setSubscriptionEnabled(sub.id, false);
});
});
t('backpressure: one in flight per subscription, <= maxInflight (20) overall', async () => {
t('backpressure: one in flight per subscription, <= maxInflight (20) overall', async () => {
@@ -234,7 +234,7 @@ t('backpressure: one in flight per subscription, <= maxInflight (20) overall', a
@@ -234,7 +234,7 @@ t('backpressure: one in flight per subscription, <= maxInflight (20) overall', a
await bp.worker.drain();
await bp.worker.drain();
assert.strictEqual(peak, 20, `peak in flight ${peak}`);
assert.strictEqual(peak, 20, `peak in flight ${peak}`);
assert.strictEqual(perSubPeak, 1);
assert.strictEqual(perSubPeak, 1);
- assert.strictEqual(bp.store.deliveryCounts().delivered, 75);
+ assert.strictEqual((await bp.store.deliveryCounts()).delivered, 75);
await bp.stop();
await bp.stop();
});
});
@@ -28,7 +28,7 @@ function exportToken(env, { project = P, cap = ['events.app.read'] } = {}) {
@@ -28,7 +28,7 @@ function exportToken(env, { project = P, cap = ['events.app.read'] } = {}) {
let h;
let h;
const seqs = { production: [], sandbox: [] };
const seqs = { production: [], sandbox: [] };
-const pull = (token, after = 0, limit = 100, topic = `app.${key}.*`) => request(h.base, 'GET', `/api/v1/events?topic=${encodeURIComponent(topic)}&after_seq=${after}&limit=${limit}`, { token });
+const pull = async (token, after = 0, limit = 100, topic = `app.${key}.*`) => await request(h.base, 'GET', `/api/v1/events?topic=${encodeURIComponent(topic)}&after_seq=${after}&limit=${limit}`, { token });
t('boot and publish as the project\'s app, in both environments (and another project)', async () => {
t('boot and publish as the project\'s app, in both environments (and another project)', async () => {
h = await boot();
h = await boot();
@@ -82,11 +82,13 @@ async function boot({ env = {}, clock = manualClock(), worker = 'manual', fetchI
@@ -82,11 +82,13 @@ async function boot({ env = {}, clock = manualClock(), worker = 'manual', fetchI
EVENTS_WORKER: worker === 'manual' ? 'off' : 'on',
EVENTS_WORKER: worker === 'manual' ? 'off' : 'on',
...env,
...env,
});
});
- const h = await start({ config, clock, log: silent, fetchImpl, deliveryFetch, appPost, dnsLookup });
+ // One database per boot (PGlite, or EVENTS_TEST_STORE=pg: the containers), dropped when the boot stops.
+ const testdb = await require('./db').testDb();
+ const h = await start({ config, db: testdb.db, clock, log: silent, fetchImpl, deliveryFetch, appPost, dnsLookup });
const base = `http://127.0.0.1:${h.server.address().port}`;
const base = `http://127.0.0.1:${h.server.address().port}`;
return {
return {
...h, base, clock, dir,
...h, base, clock, dir,
- async stop() { await h.close(); fs.rmSync(dir, { recursive: true, force: true }); },
+ async stop() { await h.close(); await testdb.close(); fs.rmSync(dir, { recursive: true, force: true }); },
};
};
}
}
@@ -24,9 +24,9 @@ t('/api/ready: every check reports status, latency and checked_at; worker is req
@@ -24,9 +24,9 @@ t('/api/ready: every check reports status, latency and checked_at; worker is req
assert.strictEqual(r.status, 200, r.text);
assert.strictEqual(r.status, 200, r.text);
assert.strictEqual(r.body.status, 'ready');
assert.strictEqual(r.body.status, 'ready');
assert.strictEqual(r.body.service, 'events');
assert.strictEqual(r.body.service, 'events');
- assert.deepStrictEqual(Object.keys(r.body.checks), ['db', 'network_jwks', 'delivery_worker', 'dlq']);
+ assert.deepStrictEqual(Object.keys(r.body.checks), ['db', 'valkey', 'network_jwks', 'delivery_worker', 'dlq']);
for (const [name, c] of Object.entries(r.body.checks)) {
for (const [name, c] of Object.entries(r.body.checks)) {
- assert.strictEqual(c.status, 'ok', name);
+ assert.strictEqual(c.status, name === 'valkey' ? 'skipped' : 'ok', name); // test boots have no VALKEY_URL
assert.strictEqual(typeof c.latency_ms, 'number');
assert.strictEqual(typeof c.latency_ms, 'number');
assert.ok(Date.parse(c.checked_at));
assert.ok(Date.parse(c.checked_at));
}
}
@@ -55,7 +55,7 @@ t('deliveries feed the latency histogram and attempt counter; a dead one degrade
@@ -55,7 +55,7 @@ t('deliveries feed the latency histogram and attempt counter; a dead one degrade
});
});
t('/metrics: loopback only, templates not ids, Events gauges', async () => {
t('/metrics: loopback only, templates not ids, Events gauges', async () => {
- const id = h.db.prepare('SELECT id FROM events ORDER BY seq LIMIT 1').get().id;
+ const id = (await h.db.prepare('SELECT id FROM events ORDER BY seq LIMIT 1').get()).id;
await request(h.base, 'GET', `/api/v1/events/${id}`, { token: reader });
await request(h.base, 'GET', `/api/v1/events/${id}`, { token: reader });
await request(h.base, 'GET', '/api/v1/events?topic=live.*&after_seq=0', { token: reader });
await request(h.base, 'GET', '/api/v1/events?topic=live.*&after_seq=0', { token: reader });
await request(h.base, 'GET', '/no/such/12345');
await request(h.base, 'GET', '/no/such/12345');
@@ -13,8 +13,8 @@ let network;
@@ -13,8 +13,8 @@ let network;
let tokenClient;
let tokenClient;
const admin = serviceToken('ops', ['events.delivery.admin']);
const admin = serviceToken('ops', ['events.delivery.admin']);
-function countIn(id) {
- return h.db.prepare('SELECT COUNT(*) AS n FROM events WHERE id = ?').get(id).n;
+async function countIn(id) {
+ return (await h.db.prepare('SELECT COUNT(*) AS n FROM events WHERE id = ?').get(id)).n; // Events' own database (PostgreSQL)
}
}
t('boot Events + a stub Network token endpoint', async () => {
t('boot Events + a stub Network token endpoint', async () => {
@@ -46,7 +46,7 @@ t('publisher fills event_id, timestamp and trace (from traceparent)', async () =
@@ -46,7 +46,7 @@ t('publisher fills event_id, timestamp and trace (from traceparent)', async () =
subject: { type: 'stream', id: '7' }, payload: {},
subject: { type: 'stream', id: '7' }, payload: {},
}, { traceparent: '00-11111111111111111111111111111111-2222222222222222-01' });
}, { traceparent: '00-11111111111111111111111111111111-2222222222222222-01' });
assert.match(out.event_id, /^evt_/);
assert.match(out.event_id, /^evt_/);
- assert.strictEqual(h.store.getEvent(out.event_id).trace_id, '11111111111111111111111111111111');
+ assert.strictEqual((await h.store.getEvent(out.event_id)).trace_id, '11111111111111111111111111111111');
assert.strictEqual(network.issued(), 1);
assert.strictEqual(network.issued(), 1);
const batch = await publisher.publish([envelope(), envelope()]);
const batch = await publisher.publish([envelope(), envelope()]);
assert.strictEqual(batch.results.length, 2);
assert.strictEqual(batch.results.length, 2);
@@ -71,7 +71,7 @@ t('outbox: enqueue inside a rolled-back transaction publishes nothing; committed
@@ -71,7 +71,7 @@ t('outbox: enqueue inside a rolled-back transaction publishes nothing; committed
assert.strictEqual(outbox.pending(), 0);
assert.strictEqual(outbox.pending(), 0);
let r = await outbox.flush();
let r = await outbox.flush();
assert.deepStrictEqual(r, { sent: 0, failed: 0, rejected: 0 });
assert.deepStrictEqual(r, { sent: 0, failed: 0, rejected: 0 });
- assert.strictEqual(countIn(rolledBack.event_id), 0, 'rolled back: never published');
+ assert.strictEqual(await countIn(rolledBack.event_id), 0, 'rolled back: never published');
let committed;
let committed;
db.transaction(() => {
db.transaction(() => {
@@ -83,13 +83,13 @@ t('outbox: enqueue inside a rolled-back transaction publishes nothing; committed
@@ -83,13 +83,13 @@ t('outbox: enqueue inside a rolled-back transaction publishes nothing; committed
assert.strictEqual(r.sent, 1);
assert.strictEqual(r.sent, 1);
r = await outbox.flush();
r = await outbox.flush();
assert.strictEqual(r.sent, 0, 'sent rows are not sent again');
assert.strictEqual(r.sent, 0, 'sent rows are not sent again');
- assert.strictEqual(countIn(committed.event_id), 1);
+ assert.strictEqual(await countIn(committed.event_id), 1);
// Relay crashed after Events accepted but before the row was marked: at-least-once resend is a duplicate.
// Relay crashed after Events accepted but before the row was marked: at-least-once resend is a duplicate.
db.prepare('UPDATE event_outbox SET sent_at = NULL WHERE event_id = ?').run(committed.event_id);
db.prepare('UPDATE event_outbox SET sent_at = NULL WHERE event_id = ?').run(committed.event_id);
r = await outbox.flush();
r = await outbox.flush();
assert.strictEqual(r.sent, 1);
assert.strictEqual(r.sent, 1);
- assert.strictEqual(countIn(committed.event_id), 1, 'still exactly one event');
+ assert.strictEqual(await countIn(committed.event_id), 1, 'still exactly one event');
assert.ok(db.prepare('SELECT seq FROM event_outbox WHERE event_id = ?').get(committed.event_id).seq > 0);
assert.ok(db.prepare('SELECT seq FROM event_outbox WHERE event_id = ?').get(committed.event_id).seq > 0);
db.close();
db.close();
});
});
@@ -106,7 +106,7 @@ t('outbox: a poisoned row is isolated and rejected; a down Events is retried lat
@@ -106,7 +106,7 @@ t('outbox: a poisoned row is isolated and rejected; a down Events is retried lat
})();
})();
const r = await outbox.flush();
const r = await outbox.flush();
assert.deepStrictEqual(r, { sent: 1, failed: 0, rejected: 1 });
assert.deepStrictEqual(r, { sent: 1, failed: 0, rejected: 1 });
- assert.strictEqual(countIn(good.event_id), 1);
+ assert.strictEqual(await countIn(good.event_id), 1);
assert.ok(db.prepare('SELECT rejected_at FROM event_outbox WHERE event_id = ?').get(bad.event_id).rejected_at);
assert.ok(db.prepare('SELECT rejected_at FROM event_outbox WHERE event_id = ?').get(bad.event_id).rejected_at);
const down = events.createOutbox(db, { publisher: events.createPublisher({ eventsUrl: 'http://127.0.0.1:9', tokenClient }), now: () => clock });
const down = events.createOutbox(db, { publisher: events.createPublisher({ eventsUrl: 'http://127.0.0.1:9', tokenClient }), now: () => clock });
@@ -120,7 +120,7 @@ t('outbox: a poisoned row is isolated and rejected; a down Events is retried lat
@@ -120,7 +120,7 @@ t('outbox: a poisoned row is isolated and rejected; a down Events is retried lat
// Events is back: the regular relay picks it up once due.
// Events is back: the regular relay picks it up once due.
clock += 1000;
clock += 1000;
assert.strictEqual((await outbox.flush()).sent, 1);
assert.strictEqual((await outbox.flush()).sent, 1);
- assert.strictEqual(countIn(later.event_id), 1);
+ assert.strictEqual(await countIn(later.event_id), 1);
db.close();
db.close();
});
});
@@ -161,12 +161,12 @@ t('consumer crash mid-processing + redelivery + replay -> exactly one effect (cr
@@ -161,12 +161,12 @@ t('consumer crash mid-processing + redelivery + replay -> exactly one effect (cr
const env = envelope('live', { event_type: 'live.tip.sent', payload: { amount: 5 } });
const env = envelope('live', { event_type: 'live.tip.sent', payload: { amount: 5 } });
await request(h.base, 'POST', '/api/v1/events', { token: live, body: env });
await request(h.base, 'POST', '/api/v1/events', { token: live, body: env });
await h.worker.drain();
await h.worker.drain();
- const d = h.store.getDelivery(env.event_id, sub.id);
+ const d = await h.store.getDelivery(env.event_id, sub.id);
assert.strictEqual(d.status, 'failed', `${first}: the crashed attempt is retried`);
assert.strictEqual(d.status, 'failed', `${first}: the crashed attempt is retried`);
assert.strictEqual(effects(env.event_id), first === 'crash-after-commit' ? 1 : 0);
assert.strictEqual(effects(env.event_id), first === 'crash-after-commit' ? 1 : 0);
h.clock.advance(1000);
h.clock.advance(1000);
await h.worker.drain();
await h.worker.drain();
- assert.strictEqual(h.store.getDelivery(env.event_id, sub.id).status, 'delivered');
+ assert.strictEqual((await h.store.getDelivery(env.event_id, sub.id)).status, 'delivered');
assert.strictEqual(effects(env.event_id), 1, `${first}: exactly one effect after redelivery`);
assert.strictEqual(effects(env.event_id), 1, `${first}: exactly one effect after redelivery`);
const rp = await request(h.base, 'POST', '/api/v1/deliveries/replay', { token: admin, body: { subscription_id: sub.id, event_ids: [env.event_id] } });
const rp = await request(h.base, 'POST', '/api/v1/deliveries/replay', { token: admin, body: { subscription_id: sub.id, event_ids: [env.event_id] } });
@@ -35,7 +35,7 @@ t('valid envelope -> 201 with a seq; defaults filled', async () => {
@@ -35,7 +35,7 @@ t('valid envelope -> 201 with a seq; defaults filled', async () => {
assert.strictEqual(r.body.event_id, env.event_id);
assert.strictEqual(r.body.event_id, env.event_id);
assert.ok(Number.isInteger(r.body.seq) && r.body.seq > 0);
assert.ok(Number.isInteger(r.body.seq) && r.body.seq > 0);
assert.strictEqual(r.body.duplicate, false);
assert.strictEqual(r.body.duplicate, false);
- const row = h.store.getEvent(env.event_id);
+ const row = await h.store.getEvent(env.event_id);
assert.strictEqual(row.priority, 'important');
assert.strictEqual(row.priority, 'important');
assert.strictEqual(row.visibility, 'internal');
assert.strictEqual(row.visibility, 'internal');
assert.strictEqual(row.trace_id, '0af7651916cd43dd8448eb211c80319c', 'trace taken from traceparent');
assert.strictEqual(row.trace_id, '0af7651916cd43dd8448eb211c80319c', 'trace taken from traceparent');
@@ -100,7 +100,7 @@ t('idempotent repeat: same event_id -> 200, same seq, stored once', async () =>
@@ -100,7 +100,7 @@ t('idempotent repeat: same event_id -> 200, same seq, stored once', async () =>
assert.strictEqual(b.status, 200);
assert.strictEqual(b.status, 200);
assert.strictEqual(b.body.duplicate, true);
assert.strictEqual(b.body.duplicate, true);
assert.strictEqual(b.body.seq, a.body.seq);
assert.strictEqual(b.body.seq, a.body.seq);
- assert.strictEqual(h.db.prepare('SELECT COUNT(*) AS n FROM events WHERE id = ?').get(env.event_id).n, 1);
+ assert.strictEqual((await h.db.prepare('SELECT COUNT(*) AS n FROM events WHERE id = ?').get(env.event_id)).n, 1);
const c = await request(h.base, 'POST', '/api/v1/events', { token: live, body: { ...env, event_type: 'live.stream.ended' } });
const c = await request(h.base, 'POST', '/api/v1/events', { token: live, body: { ...env, event_type: 'live.stream.ended' } });
assert.strictEqual(c.status, 409);
assert.strictEqual(c.status, 409);
assert.strictEqual(c.body.code, 'events.id_conflict');
assert.strictEqual(c.body.code, 'events.id_conflict');
@@ -113,11 +113,11 @@ t('batch: atomic, <= 100, results per event', async () => {
@@ -113,11 +113,11 @@ t('batch: atomic, <= 100, results per event', async () => {
assert.strictEqual(r.body.results.length, 3);
assert.strictEqual(r.body.results.length, 3);
assert.ok(r.body.results[0].seq < r.body.results[1].seq && r.body.results[1].seq < r.body.results[2].seq);
assert.ok(r.body.results[0].seq < r.body.results[1].seq && r.body.results[1].seq < r.body.results[2].seq);
- const before = h.store.lastSeq();
+ const before = await h.store.lastSeq();
r = await request(h.base, 'POST', '/api/v1/events', { token: live, body: { events: [envelope(), envelope('media')] } });
r = await request(h.base, 'POST', '/api/v1/events', { token: live, body: { events: [envelope(), envelope('media')] } });
assert.strictEqual(r.status, 403);
assert.strictEqual(r.status, 403);
assert.strictEqual(r.body.index, 1);
assert.strictEqual(r.body.index, 1);
- assert.strictEqual(h.store.lastSeq(), before, 'nothing of a rejected batch is stored');
+ assert.strictEqual(await h.store.lastSeq(), before, 'nothing of a rejected batch is stored');
r = await request(h.base, 'POST', '/api/v1/events', { token: live, body: { events: Array.from({ length: 101 }, () => envelope()) } });
r = await request(h.base, 'POST', '/api/v1/events', { token: live, body: { events: Array.from({ length: 101 }, () => envelope()) } });
assert.strictEqual(r.status, 413);
assert.strictEqual(r.status, 413);
@@ -164,8 +164,8 @@ t('loop guard: one service re-triggering itself on the same subject', async () =
@@ -164,8 +164,8 @@ t('loop guard: one service re-triggering itself on the same subject', async () =
t('publish receipts outlive retention: a pruned id is still a duplicate', async () => {
t('publish receipts outlive retention: a pruned id is still a duplicate', async () => {
const env = envelope();
const env = envelope();
await request(h.base, 'POST', '/api/v1/events', { token: live, body: env });
await request(h.base, 'POST', '/api/v1/events', { token: live, body: env });
- h.store.prune({ retentionDays: 30, receiptRetentionDays: 90, now: Date.now() + 31 * 86400000 });
- assert.strictEqual(h.store.getEvent(env.event_id), null);
+ await h.store.prune({ retentionDays: 30, receiptRetentionDays: 90, now: Date.now() + 31 * 86400000 });
+ assert.strictEqual(await h.store.getEvent(env.event_id), null);
const r = await request(h.base, 'POST', '/api/v1/events', { token: live, body: env });
const r = await request(h.base, 'POST', '/api/v1/events', { token: live, body: env });
assert.strictEqual(r.status, 200);
assert.strictEqual(r.status, 200);
assert.strictEqual(r.body.duplicate, true);
assert.strictEqual(r.body.duplicate, true);
@@ -58,7 +58,7 @@ t('pull: cursor, topic filter, internal events included for services', async ()
@@ -58,7 +58,7 @@ t('pull: cursor, topic filter, internal events included for services', async ()
assert.deepStrictEqual(r.body.events, []);
assert.deepStrictEqual(r.body.events, []);
assert.strictEqual(r.body.latest_seq, s3);
assert.strictEqual(r.body.latest_seq, s3);
- const id3 = h.db.prepare('SELECT id FROM events WHERE seq = ?').get(s3).id;
+ const id3 = (await h.db.prepare('SELECT id FROM events WHERE seq = ?').get(s3)).id;
r = await request(h.base, 'GET', `/api/v1/events/${id3}`, { token: reader });
r = await request(h.base, 'GET', `/api/v1/events/${id3}`, { token: reader });
assert.strictEqual(r.body.seq, s3);
assert.strictEqual(r.body.seq, s3);
r = await request(h.base, 'GET', '/api/v1/events?topic=live.*', { token: live });
r = await request(h.base, 'GET', '/api/v1/events?topic=live.*', { token: live });
@@ -68,8 +68,8 @@ t('pull: cursor, topic filter, internal events included for services', async ()
@@ -68,8 +68,8 @@ t('pull: cursor, topic filter, internal events included for services', async ()
});
});
t('pull: a cursor older than retention reports the gap', async () => {
t('pull: a cursor older than retention reports the gap', async () => {
- const before = h.store.lastSeq();
- h.store.prune({ retentionDays: 30, now: Date.now() + 31 * 86400000 });
+ const before = await h.store.lastSeq();
+ await h.store.prune({ retentionDays: 30, now: Date.now() + 31 * 86400000 });
const s = (await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope() })).body.seq;
const s = (await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope() })).body.seq;
const r = await request(h.base, 'GET', '/api/v1/events?after_seq=1', { token: reader });
const r = await request(h.base, 'GET', '/api/v1/events?after_seq=1', { token: reader });
assert.deepStrictEqual(r.body.gap, { from_seq: 2, to_seq: before });
assert.deepStrictEqual(r.body.gap, { from_seq: 2, to_seq: before });
@@ -94,7 +94,7 @@ t('health and ready', async () => {
@@ -94,7 +94,7 @@ t('health and ready', async () => {
r = await request(h.base, 'GET', '/api/ready');
r = await request(h.base, 'GET', '/api/ready');
assert.strictEqual(r.status, 200);
assert.strictEqual(r.status, 200);
assert.strictEqual(r.body.ready, true);
assert.strictEqual(r.body.ready, true);
- assert.deepStrictEqual(Object.keys(r.body.checks), ['db', 'network_jwks', 'dlq'], 'worker off in this boot: no delivery_worker check');
+ assert.deepStrictEqual(Object.keys(r.body.checks), ['db', 'valkey', 'network_jwks', 'dlq'], 'worker off in this boot: no delivery_worker check');
assert.strictEqual(r.body.checks.db.status, 'ok');
assert.strictEqual(r.body.checks.db.status, 'ok');
assert.strictEqual(r.body.checks.network_jwks.status, 'ok');
assert.strictEqual(r.body.checks.network_jwks.status, 'ok');
r = await request(h.base, 'GET', '/nope');
r = await request(h.base, 'GET', '/nope');
@@ -152,7 +152,7 @@ t('disconnect and resume from the cursor with a fresh ticket: the missed events,
@@ -152,7 +152,7 @@ t('disconnect and resume from the cursor with a fresh ticket: the missed events,
t('a cursor older than retention is reported as a gap before the events kept', async () => {
t('a cursor older than retention is reported as a gap before the events kept', async () => {
const old = await publish(network, notified(alice, 5));
const old = await publish(network, notified(alice, 5));
- h.store.prune({ retentionDays: 30, now: Date.now() + 31 * 86400000 });
+ await h.store.prune({ retentionDays: 30, now: Date.now() + 31 * 86400000 });
const kept = await publish(network, notified(alice, 6));
const kept = await publish(network, notified(alice, 6));
const c = await stream(`topics=${TOPIC}&ticket=${realtimeTicket({ subjectId: alice })}&last_event_id=${old - 1}`);
const c = await stream(`topics=${TOPIC}&ticket=${realtimeTicket({ subjectId: alice })}&last_event_id=${old - 1}`);
await c.waitFor(x => x.events().length === 1);
await c.waitFor(x => x.events().length === 1);
@@ -88,7 +88,7 @@ t('Last-Event-ID resume replays what was missed, in order, then goes live', asyn
@@ -88,7 +88,7 @@ t('Last-Event-ID resume replays what was missed, in order, then goes live', asyn
t('a cursor older than retention gets `event: gap` first', async () => {
t('a cursor older than retention gets `event: gap` first', async () => {
const old = await publish(live, envelope('live', { visibility: 'public' }));
const old = await publish(live, envelope('live', { visibility: 'public' }));
- h.store.prune({ retentionDays: 30, now: Date.now() + 31 * 86400000 });
+ await h.store.prune({ retentionDays: 30, now: Date.now() + 31 * 86400000 });
const kept = await publish(live, envelope('live', { visibility: 'public' }));
const kept = await publish(live, envelope('live', { visibility: 'public' }));
const c = await stream('topics=live.stream.*', { 'Last-Event-ID': String(old - 1) });
const c = await stream('topics=live.stream.*', { 'Last-Event-ID': String(old - 1) });
await c.waitFor(x => x.events().length === 1);
await c.waitFor(x => x.events().length === 1);
deleted file mode 100644
deleted file mode 100644
@@ -1,128 +0,0 @@
@@ -1,128 +0,0 @@
-'use strict';
-/**
- * scripts/redact-backfill.js: redacts the stored chat events of messages Chat deleted before it
- * published chat.message.deleted. Dry run by default, --apply only with a verified --backup,
- * idempotent, first-party chat events only.
- */
-const assert = require('assert');
-const fs = require('fs');
-const path = require('path');
-const Database = require('better-sqlite3');
-const { ids } = require('openvibe-contracts');
-const { openDb, createStore } = require('../server/store');
-const { main } = require('../scripts/redact-backfill');
-const { tmpDir, envelope, suite } = require('./helpers');
-
-const t = suite('redact-backfill');
-const dir = tmpDir();
-const eventsPath = path.join(dir, 'events.db');
-const chatPath = path.join(dir, 'chat.db');
-const user = ids.newId('user');
-let store, db;
-const ev = {};
-
-function created(messageId, text) {
- return envelope('chat', {
- event_type: 'chat.message.created', visibility: 'public', actor: { type: 'user', id: user },
- subject: { type: 'chat_message', id: String(messageId) }, payload: { message_id: messageId, anon_id: 'anon77', text },
- });
-}
-async function run(...argv) {
- const lines = [];
- const code = await main(['--db', eventsPath, '--chat-db', chatPath, ...argv], (l) => lines.push(l));
- return { code, out: lines.join('\n') };
-}
-
-t('setup: an Events store with chat events, a Chat database with deleted messages', () => {
- db = openDb(eventsPath);
- store = createStore(db);
- const batch = [created(1, 'deleted one'), created(2, 'still here'), created(3, 'expired one'), created(4, 'never in chat'),
- envelope('live', { subject: { type: 'chat_message', id: '1' }, payload: { note: 'live about 1' } })];
- [ev.m1, ev.m2, ev.m3, ev.m4, ev.live] = batch.map(e => e.event_id);
- store.insertBatch(batch.map(e => ({ ...e, trace_id: 'a'.repeat(32), priority: 'important', visibility: e.visibility || 'internal' })), { publisher: 'svc:test' });
- const chat = new Database(chatPath);
- chat.exec(`CREATE TABLE chat_messages (id INTEGER PRIMARY KEY, message TEXT, is_deleted INTEGER DEFAULT 0, auto_delete_at DATETIME);
- INSERT INTO chat_messages (id, message, is_deleted, auto_delete_at) VALUES
- (1, 'deleted one', 1, NULL), (2, 'still here', 0, '2999-01-01 00:00:00'), (3, 'expired one', 0, '2000-01-01 00:00:00');`);
- chat.close();
-});
-
-t('dry run: counts, changes nothing', async () => {
- const r = await run();
- assert.strictEqual(r.code, 0, r.out);
- assert.match(r.out, /not redacted: 4 events, 4 messages/);
- assert.match(r.out, /deleted in Chat: 2 messages, 2 events to redact/);
- assert.match(r.out, /not in Chat's database: 1 messages \(left alone; --include-missing redacts them\)/);
- assert.match(r.out, /dry run: nothing changed/);
- for (const id of [ev.m1, ev.m3]) assert.ok(!store.getEvent(id).redacted_at);
-});
-
-t('--apply refuses without --backup, and with an existing backup file', async () => {
- let r = await run('--apply');
- assert.strictEqual(r.code, 2);
- assert.match(r.out, /refusing --apply without --backup/);
- const existing = path.join(dir, 'exists.db');
- fs.writeFileSync(existing, 'x');
- r = await run('--apply', '--backup', existing);
- assert.strictEqual(r.code, 2);
- assert.match(r.out, /already exists/);
- assert.ok(!store.getEvent(ev.m1).redacted_at, 'nothing changed');
- r = await main(['--db', eventsPath], () => {});
- assert.strictEqual(r, 2, '--chat-db is required');
-});
-
-t('--apply --backup: verified 0600 backup first, then exactly the deleted messages are redacted', async () => {
- const backup = path.join(dir, 'backup-1.db');
- const r = await run('--apply', '--backup', backup);
- assert.strictEqual(r.code, 0, r.out);
- assert.match(r.out, /redacted {3}2 events/);
- assert.strictEqual(fs.statSync(backup).mode & 0o777, 0o600);
- const b = new Database(backup, { readonly: true });
- assert.match(b.prepare('SELECT payload FROM events WHERE id = ?').get(ev.m1).payload, /deleted one/, 'the backup is the state before');
- b.close();
- for (const id of [ev.m1, ev.m3]) {
- const row = store.getEvent(id);
- assert.strictEqual(row.redacted_by, 'backfill');
- assert.deepStrictEqual(Object.keys(JSON.parse(row.payload)).sort(), ['redacted', 'redacted_at', 'redacted_by']);
- assert.deepStrictEqual(JSON.parse(row.actor), { type: 'service', id: 'chat' });
- }
- assert.match(store.getEvent(ev.m2).payload, /still here/, 'a live message is untouched');
- assert.match(store.getEvent(ev.m4).payload, /never in chat/, 'unknown messages are left alone');
- assert.ok(!store.getEvent(ev.live).redacted_at, "another source's event about the same id is untouched");
- const seqs = db.prepare('SELECT seq FROM events ORDER BY seq').pluck().all();
- assert.deepStrictEqual(seqs, [1, 2, 3, 4, 5], 'no row removed, sequence intact');
-});
-
-t('--include-missing: a message Chat no longer has (hard-deleted) is redacted too', async () => {
- const r = await run('--apply', '--include-missing', '--backup', path.join(dir, 'backup-missing.db'));
- assert.strictEqual(r.code, 0, r.out);
- assert.match(r.out, /not in Chat's database: 1 messages \(redacted: --include-missing\)/);
- const row = store.getEvent(ev.m4);
- assert.strictEqual(row.redacted_by, 'backfill');
- assert.ok(!/never in chat/.test(row.payload));
- assert.match(store.getEvent(ev.m2).payload, /still here/, 'a live message is still untouched');
-});
-
-t('idempotent: a second run finds nothing and changes nothing', async () => {
- const before = db.prepare('SELECT id, payload, redacted_at FROM events ORDER BY seq').all();
- const r = await run('--apply', '--backup', path.join(dir, 'backup-2.db'));
- assert.strictEqual(r.code, 0, r.out);
- assert.match(r.out, /deleted in Chat: 0 messages, 0 events to redact/);
- assert.match(r.out, /redacted {3}0 events/);
- assert.deepStrictEqual(db.prepare('SELECT id, payload, redacted_at FROM events ORDER BY seq').all(), before);
-});
-
-t('refuses a database from before redaction', async () => {
- const old = path.join(dir, 'old.db');
- const o = new Database(old);
- o.exec('CREATE TABLE events (id TEXT PRIMARY KEY, seq INTEGER, payload TEXT)');
- o.close();
- const lines = [];
- const code = await main(['--db', old, '--chat-db', chatPath], (l) => lines.push(l));
- assert.strictEqual(code, 2);
- assert.match(lines.join('\n'), /no redaction columns/);
-});
-
-t('close', () => { db.close(); });
-
-t.run();
@@ -109,13 +109,13 @@ t('chat.message.deleted turns the stored event into a tombstone at the same seq'
@@ -109,13 +109,13 @@ t('chat.message.deleted turns the stored event into a tombstone at the same seq'
const r = await publish(chat, env);
const r = await publish(chat, env);
delSeq = r.seq;
delSeq = r.seq;
assert.strictEqual(delSeq, msgSeq + 1);
assert.strictEqual(delSeq, msgSeq + 1);
- const row = h.store.getEvent(msgEventId);
+ const row = await h.store.getEvent(msgEventId);
assert.strictEqual(row.seq, msgSeq, 'seq unchanged');
assert.strictEqual(row.seq, msgSeq, 'seq unchanged');
assert.strictEqual(row.redacted_by, delEventId);
assert.strictEqual(row.redacted_by, delEventId);
assert.ok(row.redacted_at > 0);
assert.ok(row.redacted_at > 0);
assert.ok(!row.payload.includes('555-0199') && !row.actor.includes(alice));
assert.ok(!row.payload.includes('555-0199') && !row.actor.includes(alice));
- assert.strictEqual(h.store.getEvent(delEventId).redacts, 1);
- assert.ok(!h.store.getEvent(delEventId).redacted_at, 'the deletion event itself is kept as is');
+ assert.strictEqual((await h.store.getEvent(delEventId)).redacts, 1);
+ assert.ok(!(await h.store.getEvent(delEventId)).redacted_at, 'the deletion event itself is kept as is');
});
});
t('anonymous SSE replay: the tombstone, never the text', async () => {
t('anonymous SSE replay: the tombstone, never the text', async () => {
@@ -194,8 +194,8 @@ t('re-publishing a deletion is a duplicate; a second deletion leaves tombstones
@@ -194,8 +194,8 @@ t('re-publishing a deletion is a duplicate; a second deletion leaves tombstones
assert.strictEqual(again.body.duplicate, true);
assert.strictEqual(again.body.duplicate, true);
const second = deleted([101]);
const second = deleted([101]);
await publish(chat, second);
await publish(chat, second);
- assert.strictEqual(h.store.getEvent(msgEventId).redacted_by, delEventId, 'first redaction stands');
- assert.ok(!h.store.getEvent(delEventId).redacted_at, 'a deletion event is never redacted by a later one');
+ assert.strictEqual((await h.store.getEvent(msgEventId)).redacted_by, delEventId, 'first redaction stands');
+ assert.ok(!(await h.store.getEvent(delEventId)).redacted_at, 'a deletion event is never redacted by a later one');
});
});
t('only the owning service can redact', async () => {
t('only the owning service can redact', async () => {
@@ -206,7 +206,7 @@ t('only the owning service can redact', async () => {
@@ -206,7 +206,7 @@ t('only the owning service can redact', async () => {
let r = await request(h.base, 'POST', '/api/v1/events', { token: live, body: { events: [envelope('live'), liveEvent] } });
let r = await request(h.base, 'POST', '/api/v1/events', { token: live, body: { events: [envelope('live'), liveEvent] } });
assert.strictEqual(r.status, 403, r.text);
assert.strictEqual(r.status, 403, r.text);
assert.strictEqual(r.body.code, 'events.redaction_not_allowed');
assert.strictEqual(r.body.code, 'events.redaction_not_allowed');
- assert.strictEqual(h.store.getEvent(liveEvent.event_id), null);
+ assert.strictEqual(await h.store.getEvent(liveEvent.event_id), null);
// Live by subject: its own namespace only, so Chat's event is untouched.
// Live by subject: its own namespace only, so Chat's event is untouched.
await publish(live, envelope('live', { payload: { redacts: { subject_type: 'chat_message', subject_ids: ['201'] } } }));
await publish(live, envelope('live', { payload: { redacts: { subject_type: 'chat_message', subject_ids: ['201'] } } }));
// Nor can Live publish Chat's deletion (source must be the caller).
// Nor can Live publish Chat's deletion (source must be the caller).
@@ -232,20 +232,20 @@ t('only the owning service can redact', async () => {
@@ -232,20 +232,20 @@ t('only the owning service can redact', async () => {
r = await request(h.base, 'POST', '/api/v1/events', { token: chat, body: deleted([201], { payload: { redacts: { subject_type: 'chat_message' } } }) });
r = await request(h.base, 'POST', '/api/v1/events', { token: chat, body: deleted([201], { payload: { redacts: { subject_type: 'chat_message' } } }) });
assert.strictEqual(r.status, 422);
assert.strictEqual(r.status, 422);
assert.strictEqual(r.body.code, 'events.invalid_redaction');
assert.strictEqual(r.body.code, 'events.invalid_redaction');
- const row = h.store.getEvent(env.event_id);
+ const row = await h.store.getEvent(env.event_id);
assert.ok(!row.redacted_at, 'still intact after every refused or foreign attempt');
assert.ok(!row.redacted_at, 'still intact after every refused or foreign attempt');
assert.ok(row.payload.includes('555-0199'));
assert.ok(row.payload.includes('555-0199'));
// The owner can.
// The owner can.
await publish(chat, deleted([201]));
await publish(chat, deleted([201]));
- assert.ok(h.store.getEvent(env.event_id).redacted_at);
- assert.strictEqual(h.store.getEvent(env.event_id).seq, seq);
+ assert.ok((await h.store.getEvent(env.event_id)).redacted_at);
+ assert.strictEqual((await h.store.getEvent(env.event_id)).seq, seq);
});
});
-t('the text is gone from the database file, not just hidden', async () => {
- h.db.pragma('wal_checkpoint(TRUNCATE)');
- const bytes = fs.readFileSync(path.join(h.dir, 'events.db'));
- assert.ok(!bytes.includes(Buffer.from('555-0199')), 'secure_delete zeroed the old payloads');
- assert.ok(!bytes.includes(Buffer.from('anon4242')));
+t('the text is gone from the stored rows, not just hidden', async () => {
+ // PostgreSQL (ADR-035): the redacted row no longer holds the text; the old tuple versions are reclaimed by autovacuum.
+ const rows = await h.db.prepare('SELECT payload::text AS p FROM events').all();
+ assert.ok(!rows.some((r) => String(r.p).includes('555-0199')), 'no stored payload keeps the redacted text');
+ assert.ok(!rows.some((r) => String(r.p).includes('anon4242')));
});
});
t('operator redaction (backfill): same owner rule, idempotent', async () => {
t('operator redaction (backfill): same owner rule, idempotent', async () => {
@@ -253,12 +253,12 @@ t('operator redaction (backfill): same owner rule, idempotent', async () => {
@@ -253,12 +253,12 @@ t('operator redaction (backfill): same owner rule, idempotent', async () => {
await publish(chat, env);
await publish(chat, env);
const other = envelope('live', { subject: { type: 'chat_message', id: '301' } });
const other = envelope('live', { subject: { type: 'chat_message', id: '301' } });
await publish(live, other);
await publish(live, other);
- const done = h.store.redact('chat', { subject_type: 'chat_message', subject_ids: ['301'] }, { by: 'backfill' });
+ const done = await h.store.redact('chat', { subject_type: 'chat_message', subject_ids: ['301'] }, { by: 'backfill' });
assert.deepStrictEqual(done.map(d => d.id), [env.event_id]);
assert.deepStrictEqual(done.map(d => d.id), [env.event_id]);
- assert.strictEqual(h.store.getEvent(env.event_id).redacted_by, 'backfill');
- assert.ok(!h.store.getEvent(other.event_id).redacted_at, "another source's event with the same subject is untouched");
- assert.deepStrictEqual(h.store.redact('chat', { subject_type: 'chat_message', subject_ids: ['301'] }, { by: 'backfill' }), [], 'second run: nothing');
- assert.throws(() => h.store.redact('chat', { event_ids: [other.event_id] }), /may redact only its own/);
+ assert.strictEqual((await h.store.getEvent(env.event_id)).redacted_by, 'backfill');
+ assert.ok(!(await h.store.getEvent(other.event_id)).redacted_at, "another source's event with the same subject is untouched");
+ assert.deepStrictEqual(await h.store.redact('chat', { subject_type: 'chat_message', subject_ids: ['301'] }, { by: 'backfill' }), [], 'second run: nothing');
+ await assert.rejects(async () => await h.store.redact('chat', { event_ids: [other.event_id] }), /may redact only its own/);
});
});
t('public replay window: browsers get a gap for public events older than 5 minutes', async () => {
t('public replay window: browsers get a gap for public events older than 5 minutes', async () => {
@@ -19,7 +19,7 @@ const apps = require('../server/apps');
@@ -19,7 +19,7 @@ const apps = require('../server/apps');
const t = suite('security-idor');
const t = suite('security-idor');
let w;
let w;
-const snap = () => JSON.stringify(['subscriptions', 'events', 'deliveries', 'consumer_checkpoints'].map((tb) => w.h.db.prepare(`SELECT * FROM "${tb}" ORDER BY rowid`).all()));
+const snap = async () => JSON.stringify((await Promise.all(['subscriptions', 'events', 'deliveries', 'consumer_checkpoints'].map(async (tb) => await w.h.db.prepare(`SELECT * FROM "${tb}" ORDER BY 1, 2`).all()))));
t('world', async () => { w = await buildWorld(); });
t('world', async () => { w = await buildWorld(); });
@@ -36,7 +36,7 @@ t('another consumer\'s subscription: 404 on every route, nothing changes', async
@@ -36,7 +36,7 @@ t('another consumer\'s subscription: 404 on every route, nothing changes', async
'A2 as sandbox': appToken({ appId: ids.A2, projectId: ids.PA, env: 'sandbox', cap: ALL }),
'A2 as sandbox': appToken({ appId: ids.A2, projectId: ids.PA, env: 'sandbox', cap: ALL }),
};
};
const bad = [];
const bad = [];
- const before = snap();
+ const before = await snap();
const tryAll = async (who, token, id) => {
const tryAll = async (who, token, id) => {
for (const [m, p, body] of [['GET', `/api/v1/subscriptions/${id}`], ['POST', `/api/v1/subscriptions/${id}/rotate-secret`, { secret: 'x'.repeat(40) }],
for (const [m, p, body] of [['GET', `/api/v1/subscriptions/${id}`], ['POST', `/api/v1/subscriptions/${id}/rotate-secret`, { secret: 'x'.repeat(40) }],
['POST', `/api/v1/subscriptions/${id}/disable`], ['POST', `/api/v1/subscriptions/${id}/enable`]]) {
['POST', `/api/v1/subscriptions/${id}/disable`], ['POST', `/api/v1/subscriptions/${id}/enable`]]) {
@@ -50,7 +50,7 @@ t('another consumer\'s subscription: 404 on every route, nothing changes', async
@@ -50,7 +50,7 @@ t('another consumer\'s subscription: 404 on every route, nothing changes', async
await tryAll('A naming project B', crafted['A naming project B'], subs.appA.id);
await tryAll('A naming project B', crafted['A naming project B'], subs.appA.id);
await tryAll('A2 as sandbox', crafted['A2 as sandbox'], subs.appA2.id);
await tryAll('A2 as sandbox', crafted['A2 as sandbox'], subs.appA2.id);
assert.deepStrictEqual(bad, [], bad.join('\n'));
assert.deepStrictEqual(bad, [], bad.join('\n'));
- assert.strictEqual(snap(), before, 'the refusals changed nothing');
+ assert.strictEqual(await snap(), before, 'the refusals changed nothing');
// Lists: the crafted tokens list nothing of A's.
// Lists: the crafted tokens list nothing of A's.
for (const [who, token] of Object.entries(crafted)) {
for (const [who, token] of Object.entries(crafted)) {
const r = await request(w.base, 'GET', '/api/v1/subscriptions', { token });
const r = await request(w.base, 'GET', '/api/v1/subscriptions', { token });
@@ -68,7 +68,7 @@ t('another consumer\'s subscription: 404 on every route, nothing changes', async
@@ -68,7 +68,7 @@ t('another consumer\'s subscription: 404 on every route, nothing changes', async
t('publishing into another namespace, source or project is refused and stores nothing', async () => {
t('publishing into another namespace, source or project is refused and stores nothing', async () => {
const { tok, ids } = w;
const { tok, ids } = w;
const ev = (appId, projectId, over = {}) => envelope('x', { source: apps.appSource(appId), event_type: `app.${apps.projectKey(projectId)}.order.created`, actor: { type: 'app', id: appId }, subject: { type: 'order', id: '1' }, ...over });
const ev = (appId, projectId, over = {}) => envelope('x', { source: apps.appSource(appId), event_type: `app.${apps.projectKey(projectId)}.order.created`, actor: { type: 'app', id: appId }, subject: { type: 'order', id: '1' }, ...over });
- const before = snap();
+ const before = await snap();
const tries = [
const tries = [
['app A into B\'s namespace', tok.appA, ev(ids.A, ids.PB)],
['app A into B\'s namespace', tok.appA, ev(ids.A, ids.PB)],
['app A as B\'s source', tok.appA, ev(ids.A, ids.PA, { source: apps.appSource(ids.B) })],
['app A as B\'s source', tok.appA, ev(ids.A, ids.PA, { source: apps.appSource(ids.B) })],
@@ -91,7 +91,7 @@ t('publishing into another namespace, source or project is refused and stores no
@@ -91,7 +91,7 @@ t('publishing into another namespace, source or project is refused and stores no
if (r2.status < 400 && !(r2.body && Array.isArray(r2.body.results) && r2.body.results.every((x) => x.status >= 400 || x.error))) bad.push(`${what} (batch) → ${r2.status} ${r2.text.slice(0, 120)}`);
if (r2.status < 400 && !(r2.body && Array.isArray(r2.body.results) && r2.body.results.every((x) => x.status >= 400 || x.error))) bad.push(`${what} (batch) → ${r2.status} ${r2.text.slice(0, 120)}`);
}
}
assert.deepStrictEqual(bad, [], bad.join('\n'));
assert.deepStrictEqual(bad, [], bad.join('\n'));
- assert.strictEqual(snap(), before, 'nothing was stored or queued');
+ assert.strictEqual(await snap(), before, 'nothing was stored or queued');
// Positive controls: each publishes into its own.
// Positive controls: each publishes into its own.
assert.strictEqual((await request(w.base, 'POST', '/api/v1/events', { token: tok.appB, body: ev(ids.B, ids.PB) })).status, 201);
assert.strictEqual((await request(w.base, 'POST', '/api/v1/events', { token: tok.appB, body: ev(ids.B, ids.PB) })).status, 201);
assert.strictEqual((await request(w.base, 'POST', '/api/v1/events', { token: tok.live, body: envelope('live') })).status, 201);
assert.strictEqual((await request(w.base, 'POST', '/api/v1/events', { token: tok.live, body: envelope('live') })).status, 201);
@@ -99,7 +99,7 @@ t('publishing into another namespace, source or project is refused and stores no
@@ -99,7 +99,7 @@ t('publishing into another namespace, source or project is refused and stores no
t('deliveries, replay and checkpoints across consumers', async () => {
t('deliveries, replay and checkpoints across consumers', async () => {
const { tok, subs, ids, ev } = w;
const { tok, subs, ids, ev } = w;
- const before = snap();
+ const before = await snap();
const bad = [];
const bad = [];
for (const who of ['appA', 'appB', 'appA2', 'live', 'media', 'reader']) {
for (const who of ['appA', 'appB', 'appA2', 'live', 'media', 'reader']) {
for (const [m, p, body] of [['GET', `/api/v1/deliveries?subscription_id=${subs.appB.id}`], ['GET', '/api/v1/deliveries?status=dead'],
for (const [m, p, body] of [['GET', `/api/v1/deliveries?subscription_id=${subs.appB.id}`], ['GET', '/api/v1/deliveries?status=dead'],
@@ -120,7 +120,7 @@ t('deliveries, replay and checkpoints across consumers', async () => {
@@ -120,7 +120,7 @@ t('deliveries, replay and checkpoints across consumers', async () => {
if (r.status !== 403) bad.push(`checkpoint ${topic} → ${r.status}`);
if (r.status !== 403) bad.push(`checkpoint ${topic} → ${r.status}`);
}
}
assert.deepStrictEqual(bad, [], bad.join('\n'));
assert.deepStrictEqual(bad, [], bad.join('\n'));
- assert.strictEqual(snap(), before, 'the refusals changed nothing');
+ assert.strictEqual(await snap(), before, 'the refusals changed nothing');
const r = await request(w.base, 'POST', '/api/v1/deliveries/replay', { token: tok.ops, body: { subscription_id: subs.appB.id, from_seq: 0 } });
const r = await request(w.base, 'POST', '/api/v1/deliveries/replay', { token: tok.ops, body: { subscription_id: subs.appB.id, from_seq: 0 } });
assert.strictEqual(r.status, 200, 'the operator replays (positive control)');
assert.strictEqual(r.status, 200, 'the operator replays (positive control)');
});
});
@@ -129,10 +129,10 @@ t('deliveries, the log and the tables carry no secret', async () => {
@@ -129,10 +129,10 @@ t('deliveries, the log and the tables carry no secret', async () => {
const inLog = crawler.leaks({ text: w.logged.join('\n'), headers: {} }, all).map((l) => l.label);
const inLog = crawler.leaks({ text: w.logged.join('\n'), headers: {} }, all).map((l) => l.label);
assert.deepStrictEqual(inLog, [], 'nothing secret was logged');
assert.deepStrictEqual(inLog, [], 'nothing secret was logged');
const db = w.h.db;
const db = w.h.db;
- const tables = db.prepare("SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%'").all().map((r) => r.name).filter((n) => n !== 'subscriptions');
+ const tables = (await db.prepare("SELECT table_name AS name FROM information_schema.tables WHERE table_schema = current_schema()").all()).map((r) => r.name).filter((n) => n !== 'subscriptions');
assert.ok(tables.includes('events') && tables.includes('deliveries'));
assert.ok(tables.includes('events') && tables.includes('deliveries'));
const hits = [];
const hits = [];
- for (const tb of tables) for (const l of crawler.leaks({ text: JSON.stringify(db.prepare(`SELECT * FROM "${tb}"`).all()), headers: {} }, all)) hits.push(`${tb}: ${l.label}`);
+ for (const tb of tables) for (const l of crawler.leaks({ text: JSON.stringify(await db.prepare(`SELECT * FROM "${tb}"`).all()), headers: {} }, all)) hits.push(`${tb}: ${l.label}`);
assert.deepStrictEqual(hits, [], 'no secret outside the subscriptions table (events, deliveries, usage outbox)');
assert.deepStrictEqual(hits, [], 'no secret outside the subscriptions table (events, deliveries, usage outbox)');
});
});
@@ -107,7 +107,7 @@ const prj = `prj_${ids.ulid()}`;
@@ -107,7 +107,7 @@ const prj = `prj_${ids.ulid()}`;
const app = ids.newId('app');
const app = ids.newId('app');
const key = apps.projectKey(prj);
const key = apps.projectKey(prj);
const tokApp = () => appToken({ appId: app, projectId: prj, env: 'sandbox', cap: ALL });
const tokApp = () => appToken({ appId: app, projectId: prj, env: 'sandbox', cap: ALL });
-const sub = (token, endpoint, topic = `app.${key}.*`) => request(h.base, 'POST', '/api/v1/subscriptions', { token, body: { topic_pattern: topic, endpoint } });
+const sub = async (token, endpoint, topic = `app.${key}.*`) => await request(h.base, 'POST', '/api/v1/subscriptions', { token, body: { topic_pattern: topic, endpoint } });
t('boot', async () => {
t('boot', async () => {
await new Promise((r) => receiver.listen(0, '127.0.0.1', r));
await new Promise((r) => receiver.listen(0, '127.0.0.1', r));
@@ -158,7 +158,7 @@ t('DNS rebinding: public at subscribe, internal at delivery: dead at once, nothi
@@ -158,7 +158,7 @@ t('DNS rebinding: public at subscribe, internal at delivery: dead at once, nothi
const e = await request(h.base, 'POST', '/api/v1/events', { token: tokApp(), body: envelope('x', { source: apps.appSource(app), event_type: `app.${key}.rebind.hit`, actor: { type: 'app', id: app } }) });
const e = await request(h.base, 'POST', '/api/v1/events', { token: tokApp(), body: envelope('x', { source: apps.appSource(app), event_type: `app.${key}.rebind.hit`, actor: { type: 'app', id: app } }) });
assert.strictEqual(e.status, 201, e.text);
assert.strictEqual(e.status, 201, e.text);
await h.worker.drain();
await h.worker.drain();
- const d = h.store.getDelivery(e.body.event_id, r.body.id);
+ const d = await h.store.getDelivery(e.body.event_id, r.body.id);
assert.strictEqual(d.status, 'dead', 'refused for good');
assert.strictEqual(d.status, 'dead', 'refused for good');
assert.match(d.last_error, /non-public/);
assert.match(d.last_error, /non-public/);
assert.ok(!received.some((x) => x.url === '/rebind'), 'nothing reached the rebound address');
assert.ok(!received.some((x) => x.url === '/rebind'), 'nothing reached the rebound address');
@@ -172,7 +172,7 @@ t('redirects are never followed (app endpoints and first-party ones)', async ()
@@ -172,7 +172,7 @@ t('redirects are never followed (app endpoints and first-party ones)', async ()
received.length = 0; internalHits = 0;
received.length = 0; internalHits = 0;
const e = await request(h.base, 'POST', '/api/v1/events', { token: tokApp(), body: envelope('x', { source: apps.appSource(app), event_type: `app.${key}.redir.hit`, actor: { type: 'app', id: app } }) });
const e = await request(h.base, 'POST', '/api/v1/events', { token: tokApp(), body: envelope('x', { source: apps.appSource(app), event_type: `app.${key}.redir.hit`, actor: { type: 'app', id: app } }) });
await h.worker.drain();
await h.worker.drain();
- const d = h.store.getDelivery(e.body.event_id, r.body.id);
+ const d = await h.store.getDelivery(e.body.event_id, r.body.id);
assert.strictEqual(d.last_status, 302, 'a 3xx is a failed attempt');
assert.strictEqual(d.last_status, 302, 'a 3xx is a failed attempt');
assert.notStrictEqual(d.status, 'delivered');
assert.notStrictEqual(d.status, 'delivered');
assert.strictEqual(received.filter((x) => x.url === '/redirect').length, 1, 'one request to the endpoint, none to where it pointed');
assert.strictEqual(received.filter((x) => x.url === '/redirect').length, 1, 'one request to the endpoint, none to where it pointed');
@@ -185,7 +185,7 @@ t('redirects are never followed (app endpoints and first-party ones)', async ()
@@ -185,7 +185,7 @@ t('redirects are never followed (app endpoints and first-party ones)', async ()
const le = await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope('live') });
const le = await request(h.base, 'POST', '/api/v1/events', { token: live, body: envelope('live') });
await h.worker.drain();
await h.worker.drain();
assert.strictEqual(fp.calls.length, 1);
assert.strictEqual(fp.calls.length, 1);
- assert.strictEqual(h.store.getDelivery(le.body.event_id, s.body.id).last_status, 302);
+ assert.strictEqual((await h.store.getDelivery(le.body.event_id, s.body.id)).last_status, 302);
assert.strictEqual(internalHits, 0);
assert.strictEqual(internalHits, 0);
await fp.close();
await fp.close();
});
});
@@ -32,7 +32,7 @@ const ALL = ['events.app.publish', 'events.app.read', 'events.app.subscribe'];
@@ -32,7 +32,7 @@ const ALL = ['events.app.publish', 'events.app.read', 'events.app.subscribe'];
async function buildWorld({ env = {} } = {}) {
async function buildWorld({ env = {} } = {}) {
const dir = tmpDir();
const dir = tmpDir();
const logged = [];
const logged = [];
- const keep = (...a) => logged.push(a.map((x) => (x && x.stack) || String(x)).join(' '));
+ const keep = (...a) => { logged.push(a.map((x) => (x && x.stack) || String(x)).join(' ')); if (process.env.DEBUG) console.error(...a); };
const log = { log: keep, info: keep, warn: keep, error: keep };
const log = { log: keep, info: keep, warn: keep, error: keep };
// App endpoints: a fake DNS, and a local receiver standing in for every public address.
// App endpoints: a fake DNS, and a local receiver standing in for every public address.
@@ -63,7 +63,8 @@ async function buildWorld({ env = {} } = {}) {
@@ -63,7 +63,8 @@ async function buildWorld({ env = {} } = {}) {
const clock = manualClock();
const clock = manualClock();
const config = load({ NODE_ENV: 'test', PORT: '0', EVENTS_DB_PATH: path.join(dir, 'events.db'), OV_NETWORK_PUBLIC_KEY: publicKey, EVENTS_WORKER: 'off', ...env });
const config = load({ NODE_ENV: 'test', PORT: '0', EVENTS_DB_PATH: path.join(dir, 'events.db'), OV_NETWORK_PUBLIC_KEY: publicKey, EVENTS_WORKER: 'off', ...env });
- const h = await start({ config, clock, log, appPost, dnsLookup });
+ const testdb = await require('./db').testDb(); // a database of its own (PGlite, or the containers under test:pg)
+ const h = await start({ config, db: testdb.db, clock, log, appPost, dnsLookup });
const base = `http://127.0.0.1:${h.server.address().port}`;
const base = `http://127.0.0.1:${h.server.address().port}`;
const routes = crawler.listRoutes(h.server);
const routes = crawler.listRoutes(h.server);
@@ -84,7 +85,7 @@ async function buildWorld({ env = {} } = {}) {
@@ -84,7 +85,7 @@ async function buildWorld({ env = {} } = {}) {
userX: userToken({ subjectId: x }),
userX: userToken({ subjectId: x }),
userY: userToken({ subjectId: y }),
userY: userToken({ subjectId: y }),
};
};
- const call = (token, method, p, body, headers = {}) => request(base, method, p, { token, body, headers });
+ const call = async (token, method, p, body, headers = {}) => await request(base, method, p, { token, body, headers });
const must = (r, status, what) => { if (r.status !== status) throw new Error(`${what}: ${r.status} ${r.text.slice(0, 300)}`); return r; };
const must = (r, status, what) => { if (r.status !== status) throw new Error(`${what}: ${r.status} ${r.text.slice(0, 300)}`); return r; };
const shownOnce = [];
const shownOnce = [];
const once = (what, r) => { shownOnce.push({ what, status: r.status, cache: r.headers.get('cache-control') || '' }); return r; };
const once = (what, r) => { shownOnce.push({ what, status: r.status, cache: r.headers.get('cache-control') || '' }); return r; };
@@ -133,7 +134,7 @@ async function buildWorld({ env = {} } = {}) {
@@ -133,7 +134,7 @@ async function buildWorld({ env = {} } = {}) {
return {
return {
h, base, dir, config, clock, routes, logged, secrets, subs, ev, markers, tok, shownOnce, call, fp, appSeen, dnsTable, dnsLookup,
h, base, dir, config, clock, routes, logged, secrets, subs, ev, markers, tok, shownOnce, call, fp, appSeen, dnsTable, dnsLookup,
ids: { PA, PB, A, A2, B, x, y, keyA, keyB }, privateKey, realtimeTicket,
ids: { PA, PB, A, A2, B, x, y, keyA, keyB }, privateKey, realtimeTicket,
- stop: async () => { await fp.close(); await new Promise((r) => receiver.close(r)); await h.close(); fs.rmSync(dir, { recursive: true, force: true }); },
+ stop: async () => { await fp.close(); await new Promise((r) => receiver.close(r)); await h.close(); await testdb.close(); fs.rmSync(dir, { recursive: true, force: true }); },
};
};
}
}
@@ -23,7 +23,7 @@ const appEvent = (over = {}) => ({
@@ -23,7 +23,7 @@ const appEvent = (over = {}) => ({
const TRACE = '4bf92f3577b34da6a3ce929d0e0e4736';
const TRACE = '4bf92f3577b34da6a3ce929d0e0e4736';
let h, network, statuses = [];
let h, network, statuses = [];
-const rows = () => h.db.prepare('SELECT * FROM app_usage ORDER BY capability').all();
+const rows = async () => await h.db.prepare('SELECT * FROM app_usage ORDER BY capability').all();
t('boot: an app subscription and a first-party usage subscriber', async () => {
t('boot: an app subscription and a first-party usage subscriber', async () => {
// The hour starts at a round time so the test controls when it closes.
// The hour starts at a round time so the test controls when it closes.
@@ -34,8 +34,8 @@ t('boot: an app subscription and a first-party usage subscriber', async () => {
@@ -34,8 +34,8 @@ t('boot: an app subscription and a first-party usage subscriber', async () => {
});
});
h.clock.t = clockStart;
h.clock.t = clockStart;
network = await subscriber();
network = await subscriber();
- h.store.createSubscription({ id: `sub_${ids.ulid()}`, consumer: source, topicPattern: `app.${key}.*`, endpoint: 'https://hooks.example.com/h', secret: 'a'.repeat(40), projectId: prj, env: 'sandbox' });
- h.store.createSubscription({ id: `sub_${ids.ulid()}`, consumer: 'network', topicPattern: 'events.usage.recorded', endpoint: network.url, secret: 'b'.repeat(40) });
+ await h.store.createSubscription({ id: `sub_${ids.ulid()}`, consumer: source, topicPattern: `app.${key}.*`, endpoint: 'https://hooks.example.com/h', secret: 'a'.repeat(40), projectId: prj, env: 'sandbox' });
+ await h.store.createSubscription({ id: `sub_${ids.ulid()}`, consumer: 'network', topicPattern: 'events.usage.recorded', endpoint: network.url, secret: 'b'.repeat(40) });
});
});
t('stored app events count; a repeated event_id does not', async () => {
t('stored app events count; a repeated event_id does not', async () => {
@@ -44,7 +44,7 @@ t('stored app events count; a repeated event_id does not', async () => {
@@ -44,7 +44,7 @@ t('stored app events count; a repeated event_id does not', async () => {
assert.strictEqual(r.status, 201);
assert.strictEqual(r.status, 201);
r = await request(h.base, 'POST', '/api/v1/events', { token, body: batch[0] });
r = await request(h.base, 'POST', '/api/v1/events', { token, body: batch[0] });
assert.strictEqual(r.status, 200, 'a duplicate');
assert.strictEqual(r.status, 200, 'a duplicate');
- const [pub] = rows();
+ const [pub] = await rows();
assert.deepStrictEqual([pub.project_id, pub.env, pub.capability, pub.unit, pub.quantity, pub.errors], [prj, 'sandbox', 'events.app.publish', 'events', 3, 0]);
assert.deepStrictEqual([pub.project_id, pub.env, pub.capability, pub.unit, pub.quantity, pub.errors], [prj, 'sandbox', 'events.app.publish', 'events', 3, 0]);
assert.strictEqual(pub.window_start % HOUR, 0, 'an hour window');
assert.strictEqual(pub.window_start % HOUR, 0, 'an hour window');
});
});
@@ -55,7 +55,7 @@ t('refusals count as errors with code, status, trace id and event id; nothing of
@@ -55,7 +55,7 @@ t('refusals count as errors with code, status, trace id and event id; nothing of
assert.strictEqual(r.status, 403);
assert.strictEqual(r.status, 403);
r = await request(h.base, 'POST', '/api/v1/events', { token, body: { events: [appEvent(), appEvent(), appEvent()] } });
r = await request(h.base, 'POST', '/api/v1/events', { token, body: { events: [appEvent(), appEvent(), appEvent()] } });
assert.strictEqual(r.status, 429, 'past the per-minute quota: the whole batch is refused');
assert.strictEqual(r.status, 429, 'past the per-minute quota: the whole batch is refused');
- const [pub] = rows();
+ const [pub] = await rows();
assert.strictEqual(pub.quantity, 3);
assert.strictEqual(pub.quantity, 3);
assert.strictEqual(pub.errors, 2);
assert.strictEqual(pub.errors, 2);
assert.deepStrictEqual(JSON.parse(pub.error_codes), { 'events.type_not_allowed': 1, 'events.quota_exceeded': 1 });
assert.deepStrictEqual(JSON.parse(pub.error_codes), { 'events.type_not_allowed': 1, 'events.quota_exceeded': 1 });
@@ -68,7 +68,7 @@ t('refusals count as errors with code, status, trace id and event id; nothing of
@@ -68,7 +68,7 @@ t('refusals count as errors with code, status, trace id and event id; nothing of
t('webhook delivery attempts count, a non-2xx as an error with the event and its trace', async () => {
t('webhook delivery attempts count, a non-2xx as an error with the event and its trace', async () => {
statuses = [500];
statuses = [500];
await h.worker.drain();
await h.worker.drain();
- const sub = rows().find(r => r.capability === 'events.app.subscribe');
+ const sub = (await rows()).find(r => r.capability === 'events.app.subscribe');
assert.deepStrictEqual([sub.unit, sub.quantity, sub.errors], ['deliveries', 3, 1]);
assert.deepStrictEqual([sub.unit, sub.quantity, sub.errors], ['deliveries', 3, 1]);
assert.deepStrictEqual(JSON.parse(sub.error_codes), { 'events.delivery.http_500': 1 });
assert.deepStrictEqual(JSON.parse(sub.error_codes), { 'events.delivery.http_500': 1 });
const [s] = JSON.parse(sub.samples);
const [s] = JSON.parse(sub.samples);
@@ -76,14 +76,14 @@ t('webhook delivery attempts count, a non-2xx as an error with the event and its
@@ -76,14 +76,14 @@ t('webhook delivery attempts count, a non-2xx as an error with the event and its
});
});
t('nothing is sent before the hour closes', async () => {
t('nothing is sent before the hour closes', async () => {
- const out = h.flushUsage();
+ const out = await h.flushUsage();
assert.strictEqual(out.stored.length, 0);
assert.strictEqual(out.stored.length, 0);
- assert.strictEqual(h.store.usage.pending(), 2);
+ assert.strictEqual(await h.store.usage.pending(), 2);
});
});
t('after the hour: one events.usage.recorded per rollup, valid, internal, delivered to first-party subscribers', async () => {
t('after the hour: one events.usage.recorded per rollup, valid, internal, delivered to first-party subscribers', async () => {
h.clock.advance(HOUR);
h.clock.advance(HOUR);
- const out = h.flushUsage();
+ const out = await h.flushUsage();
assert.strictEqual(out.stored.length, 2);
assert.strictEqual(out.stored.length, 2);
for (const row of out.stored) {
for (const row of out.stored) {
assert.deepStrictEqual([row.event_type, row.source, row.visibility, row.priority, row.subject_type, row.subject_id, row.project_id], ['events.usage.recorded', 'events', 'internal', 'low', 'project', prj, null]);
assert.deepStrictEqual([row.event_type, row.source, row.visibility, row.priority, row.subject_type, row.subject_id, row.project_id], ['events.usage.recorded', 'events', 'internal', 'low', 'project', prj, null]);
@@ -92,7 +92,7 @@ t('after the hour: one events.usage.recorded per rollup, valid, internal, delive
@@ -92,7 +92,7 @@ t('after the hour: one events.usage.recorded per rollup, valid, internal, delive
const text = JSON.stringify(payload);
const text = JSON.stringify(payload);
assert.ok(!text.includes(appId) && !text.includes(source) && !text.includes('order.created'), 'no app, source or event content');
assert.ok(!text.includes(appId) && !text.includes(source) && !text.includes('order.created'), 'no app, source or event content');
}
}
- assert.strictEqual(h.flushUsage().stored.length, 0, 'sent once');
+ assert.strictEqual((await h.flushUsage()).stored.length, 0, 'sent once');
await h.worker.drain();
await h.worker.drain();
const got = network.calls.map(c => c.body.event);
const got = network.calls.map(c => c.body.event);
assert.strictEqual(got.length, 2);
assert.strictEqual(got.length, 2);
@@ -108,9 +108,9 @@ t('apps never see the rollups', async () => {
@@ -108,9 +108,9 @@ t('apps never see the rollups', async () => {
});
});
t('a count in an hour already sent re-sends it as revision 2', async () => {
t('a count in an hour already sent re-sends it as revision 2', async () => {
- const sent = rows().find(r => r.capability === 'events.app.publish');
- h.store.usage.record({ projectId: prj, env: 'sandbox', capability: 'events.app.publish', unit: 'events', quantity: 1, at: sent.window_start + 1000 });
- const out = h.flushUsage();
+ const sent = (await rows()).find(r => r.capability === 'events.app.publish');
+ await h.store.usage.record({ projectId: prj, env: 'sandbox', capability: 'events.app.publish', unit: 'events', quantity: 1, at: sent.window_start + 1000 });
+ const out = await h.flushUsage();
assert.strictEqual(out.stored.length, 1);
assert.strictEqual(out.stored.length, 1);
const p = JSON.parse(out.stored[0].payload);
const p = JSON.parse(out.stored[0].payload);
assert.deepStrictEqual([p.quantity, p.revision], [4, 2], 'the corrected totals');
assert.deepStrictEqual([p.quantity, p.revision], [4, 2], 'the corrected totals');
@@ -121,7 +121,7 @@ t('EVENTS_USAGE=off counts nothing', async () => {
@@ -121,7 +121,7 @@ t('EVENTS_USAGE=off counts nothing', async () => {
try {
try {
const r = await request(h2.base, 'POST', '/api/v1/events', { token, body: appEvent() });
const r = await request(h2.base, 'POST', '/api/v1/events', { token, body: appEvent() });
assert.strictEqual(r.status, 201);
assert.strictEqual(r.status, 201);
- assert.strictEqual(h2.db.prepare('SELECT COUNT(*) AS n FROM app_usage').get().n, 0);
+ assert.strictEqual((await h2.db.prepare('SELECT COUNT(*) AS n FROM app_usage').get()).n, 0);
} finally { await h2.stop(); }
} finally { await h2.stop(); }
});
});