openvibers/openvibe.search Commits
Search moves to PostgreSQL (ADR-035, roadmap WS-X2, repository-pass group G2), with a PostgreSQL full-text engine.
Changes
@@@ -28,6 +28,14 @@ OV_OAUTH_CLIENT_ID=searchOV_OAUTH_CLIENT_SECRET=# Store (SQLite, FTS5). Production: /var/lib/openvibe-search/search.db (systemd StateDirectory).++# PostgreSQL (ADR-035), written by OpenVibe.Host roles/data/add-service.sh search: the runtime role through PgBouncer,++# the owner on a direct connection (migrations and the one-time import), and Valkey (shared limit counters).++# Without DATABASE_URL, development uses an embedded PGlite database in data/pglite.++DATABASE_URL=++DATABASE_DIRECT_URL=++VALKEY_URL=++VALKEY_PREFIX=++# The SQLite file of releases before PostgreSQL: read once by scripts/migrate-to-postgres.js.SEARCH_DB_PATH=./data/search.db# OpenVibe.Events. The outbox relay runs only when EVENTS_URL is set.@@@ -9,6 +9,9 @@ jobs:uses: OpenVibers/OpenVibe.Shared/.github/workflows/test.yml@423fe519b5fe9594cc81b670e5ba5cb9f21f6176with:contracts-service: search++ # ADR-035: PostgreSQL 18 + PgBouncer (transaction mode) + Valkey 9 containers (openvibe-sdk++ # scripts/test-services.sh). npm test runs every file on PGlite; test:pg runs them all through PgBouncer.++ test: 'eval "$(node_modules/openvibe-sdk/scripts/test-services.sh up)" && npm test && npm run test:pg'# Secret scan (gitleaks) and dependency audit, shared by every OpenVibers repo (Track Q).security:uses: OpenVibers/OpenVibe.Shared/.github/workflows/security.yml@423fe519b5fe9594cc81b670e5ba5cb9f21f6176@@@ -30,7 +30,7 @@ npm test # every test/*.test.js, temp databases, no network neededNode 22 in production (`fnm exec --using=22.22.1 npm test`). Production: `/opt/openvibe.search`,env `/etc/openvibe/search.env`, unit [deploy/systemd/openvibe-search.service](deploy/systemd/openvibe-search.service),--index `/var/lib/openvibe-search/search.db`, nginx [deploy/nginx/search.openvibe.network.conf](deploy/nginx/search.openvibe.network.conf)++database `ov_search` on the host's data role (ADR-035; `sudo /opt/openvibe.host/roles/data/add-service.sh search`; the one-time move from `/var/lib/openvibe-search/search.db` is `scripts/migrate-to-postgres.js`, and the full-text index rebuilds itself from the documents at boot), nginx [deploy/nginx/search.openvibe.network.conf](deploy/nginx/search.openvibe.network.conf)(the public vhost exposes the search page, the GET query API, saved searches and health; the ownerAPI and the Events webhook are loopback-only, and owners and Events call `127.0.0.1:4710`).@@@ -173,9 +173,9 @@ the `search.document.removed` event it records:## Owns- the index document contract (`search.index-document@1`, released in openvibe-contracts v0.12.0; proposal in [docs/contracts-proposal/](docs/contracts-proposal/))--- `documents`, `doc_acl`, `doc_facets`, `fts_public`, `fts_restricted`, `idempotency_receipts`,-- `ingest_rejections`, `event_outbox`, `removals`, `cdn_purges`, `saved_searches` (SQLite; engine-- decision in [docs/adr-engine.md](docs/adr-engine.md))++- `documents`, `doc_acl`, `doc_facets`, `search_fts` (the full-text index), `idempotency_receipts`,++ `ingest_rejections`, `event_outbox`, `removals`, `cdn_purges`, `saved_searches` (PostgreSQL, schema in++ [migrations/](migrations/); engine decision in [docs/adr-engine.md](docs/adr-engine.md))- the query API, its ranking and ACL filtering, the search page and saved searches; deletion andvisibility-change propagation, including the CDN purge of removed public URLs@@@ -189,6 +189,7 @@ the `search.document.removed` event it records:## Depends on- OpenVibe.Contracts (service tokens, problem details, ids, the event envelope)++- PostgreSQL 18 (OpenVibe.Host `roles/data/`, ADR-035): documents and the full-text index, async through `openvibe-sdk/db`- OpenVibe.Network (signing key; service principal `search`)- OpenVibe.Events (index-document deliveries in, `search.document.*` out)@@@ -209,7 +210,7 @@ Called elsewhere, as the service principal `search` (client credentials from Ope| OpenVibe.Events | `events.event.publish` (audience `openvibe.events`) | the outbox relays `search.document.indexed` / `search.document.removed` || OpenVibe.Events | `events.subscription.manage` (`npm run subscribe`, once) | the HMAC-signed subscription to `*.index_document.*` that delivers index events |--Released in `openvibe-contracts` v0.12.0 with the service manifest (this repo pins v0.49.0);++Released in `openvibe-contracts` v0.76.0 with the service manifest (this repo pins v0.49.0);[server/auth.js](server/auth.js) decides them with the contracts grant rule, and CI runs`openvibe-contracts-check --service search`.@@@ -294,7 +295,7 @@ fast-forward `/opt/openvibe.search`, install on a lockfile change, restart, wait| Unit | `openvibe-search.service` ([deploy/systemd/](deploy/systemd/openvibe-search.service)), runs as `ubuntu` || Port | `127.0.0.1:4710` || Env file | `/etc/openvibe/search.env` |--| Data | `/var/lib/openvibe-search/search.db` (`StateDirectory`) |++| Data | PostgreSQL `ov_search` (`DATABASE_URL` through PgBouncer; migrations on `DATABASE_DIRECT_URL`) || Public host | nginx [deploy/nginx/search.openvibe.network.conf](deploy/nginx/search.openvibe.network.conf) (below) |Rollback: ovhost puts the previous sha back by itself when `/api/ready` does not answer 2xx after the@@@ -7,14 +7,22 @@"runtime": "node 22, express, better-sqlite3 (FTS5)","port": 4710,"domain": "search.openvibe.network (vhost in deploy/nginx; the service manifest records the search page and query API as public since 2026-09-23)",-- "contracts": "openvibe-contracts v0.49.0",-- "capabilities": ["search.document.write", "search.query.delegate", "search.query.run"],++ "contracts": "openvibe-contracts v0.76.0",++ "capabilities": [++ "search.document.write",++ "search.query.delegate",++ "search.query.run"++ ],"capabilitiesRegistered": true,-- "contractsProposed": ["search.index-document@1 (released in openvibe-contracts v0.12.0)"],++ "contractsProposed": [++ "search.index-document@1 (released in openvibe-contracts v0.12.0)"++ ],"engine": "sqlite-fts5 (docs/adr-engine.md)","deployed": true,"deployment": "Deployed with `sudo ovhost deploy search`: /opt/openvibe.search, unit openvibe-search, 127.0.0.1:4710, principal search, Events subscription *.index_document.*, accepted event owners sources, wiki, blog, news, reviews, trade, deals, coupons. On 2026-09-23 the production index held 10 documents, all tombstones (wiki), 0 public. Restore drill passed 2026-09-23.",-- "indexedProducers": ["wiki (tombstones only so far)"],++ "indexedProducers": [++ "wiki (tombstones only so far)"++ ],"features": ["ACL-filtered FTS5 query API","facets",@@@ -2,8 +2,8 @@Description=OpenVibe.Search — permission-aware index and query API (SQLite FTS5)# Network (openvibe-network) signs the tokens this service verifies. Search starts without it# and keeps retrying the JWKS fetch; /api/ready stays 503 until the key has loaded.--After=network.target--Wants=openvibe-network.service++After=network.target postgresql.service pgbouncer.service valkey-server.service++Wants=openvibe-network.service postgresql.service pgbouncer.service valkey-server.service[Service]Type=simple@@@ -51,3 +51,16 @@ is no corpus to measure: no product indexes anything today.(Japanese/Chinese get no word segmentation; a later engine or an ICU tokenizer fixes that).- Moving to PostgreSQL is an engine swap plus a re-index from owners (the reconciliation APIexists for exactly that), not a contract change.++++## Update 2026-09-28: PostgreSQL full-text search (ADR-035)++++Search moved to PostgreSQL with every other service. The engine interface held: `server/engine/pg.js` replaced++`server/engine/fts5.js` and nothing else learned the engine changed. One table, `search_fts`, split by audience++(exposure 3 public listed, 1 restricted), with a `tsvector` weighted title A, summary B, body C (FTS5's 10 / 4 / 1 as++`ts_rank` weights) and a GIN index. Text is folded (diacritics removed) and split into runs of letters and digits before++the `simple` configuration sees it, so tokens are exactly FTS5 unicode61's: on a copy of production, the match counts++for sampled terms were identical. The rank is minus `ts_rank` times the same freshness boost, so ascending order and++the keyset cursors work unchanged. Snippets are `ts_headline` over the summary and body (the title when only it++matches), computed for the returned page only. The index is derived: at boot, when its counts disagree with the++documents table (after the one-time import, whose FTS5 tables are not copied), it is rebuilt from `documents.doc`.++nnew file mode 100644@@@ -0,0 +1,153 @@++-- phase: expand++-- OpenVibe.Search on PostgreSQL (ADR-035, roadmap WS-X2): the tables as they were on SQLite (converted by openvibe-sdk++-- tools/asyncify/sqlite-schema-to-pg: text COLLATE "C" compares like SQLite, integers are bigint, identities keep their ids),++-- then the openvibe-publishing stores and the openvibe-sdk inbox and outbox. Generated once on 2026-09-28; never edited after it runs.++++CREATE TABLE documents (++ rid bigint GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,++ owner text COLLATE "C" NOT NULL,++ type text COLLATE "C" NOT NULL,++ id text COLLATE "C" NOT NULL,++ revision bigint NOT NULL,++ deleted bigint NOT NULL DEFAULT 0,++ visibility text COLLATE "C",++ publication_state text COLLATE "C",++ exposure bigint NOT NULL DEFAULT 0, -- document.js EXPOSURE (0 none .. 3 public listed)++ index_decision text COLLATE "C" NOT NULL,++ noindex_reasons text COLLATE "C" NOT NULL DEFAULT '[]',++ canonical_url text COLLATE "C",++ title text COLLATE "C",++ language text COLLATE "C",++ published_at text COLLATE "C",++ updated_at text COLLATE "C",++ sort_at text COLLATE "C" NOT NULL DEFAULT '', -- updated_at, else published_at: browse order++ doc text COLLATE "C" NOT NULL, -- normalized document JSON (tombstone: identity only)++ hash text COLLATE "C" NOT NULL,++ via text COLLATE "C" NOT NULL, -- event | api++ event_id text COLLATE "C",++ indexed_at bigint NOT NULL,++ UNIQUE (owner, type, id)++);++CREATE INDEX idx_documents_exposure ON documents (exposure, sort_at, rid);++CREATE INDEX idx_documents_owner ON documents (owner, type, id);++++-- ACL keys of servable non-public documents: kind s = subject, g = group, e = entitlement.++CREATE TABLE doc_acl (++ rid bigint NOT NULL,++ kind text COLLATE "C" NOT NULL CHECK (kind IN ('s', 'g', 'e')),++ value text COLLATE "C" NOT NULL,++ PRIMARY KEY (rid, kind, value)++);++CREATE INDEX idx_doc_acl_value ON doc_acl (kind, value, rid);++++CREATE TABLE doc_facets (++ rid bigint NOT NULL,++ key text COLLATE "C" NOT NULL,++ value text COLLATE "C" NOT NULL,++ PRIMARY KEY (rid, key, value)++);++CREATE INDEX idx_doc_facets_value ON doc_facets (key, value, rid);++++-- Inbox: one receipt per consumed event (exactly-once effects), with what happened.++CREATE TABLE idempotency_receipts (++ consumer text COLLATE "C" NOT NULL,++ event_id text COLLATE "C" NOT NULL,++ outcome text COLLATE "C" NOT NULL,++ processed_at bigint NOT NULL,++ PRIMARY KEY (consumer, event_id)++);++++-- Index-document events Search refused (bad document, wrong owner, subject mismatch). Owners++-- read their own through GET /api/v1/owners/:owner/rejections.++CREATE TABLE ingest_rejections (++ seq bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,++ event_id text COLLATE "C" NOT NULL,++ event_type text COLLATE "C",++ owner text COLLATE "C",++ type text COLLATE "C",++ id text COLLATE "C",++ revision bigint,++ code text COLLATE "C" NOT NULL,++ detail text COLLATE "C",++ at bigint NOT NULL++);++CREATE INDEX idx_ingest_rejections_owner ON ingest_rejections (owner, seq);++++-- server/events/outbox.js ensureSchema()++CREATE TABLE event_outbox (++ seq bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,++ event_id text COLLATE "C" NOT NULL UNIQUE,++ event_type text COLLATE "C" NOT NULL,++ envelope text COLLATE "C" NOT NULL,++ created_at bigint NOT NULL,++ attempts bigint NOT NULL DEFAULT 0,++ next_attempt_at bigint NOT NULL DEFAULT 0,++ sent_at bigint,++ rejected_at bigint,++ last_error text COLLATE "C"++);++CREATE INDEX idx_event_outbox_due ON event_outbox(sent_at, rejected_at, next_attempt_at);++++-- server/purge.js ensureSchema()++CREATE TABLE removals (++ seq bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,++ event_id text COLLATE "C" NOT NULL, -- the search.document.removed event++ owner text COLLATE "C" NOT NULL,++ type text COLLATE "C" NOT NULL,++ id text COLLATE "C" NOT NULL,++ revision bigint NOT NULL,++ reason text COLLATE "C" NOT NULL,++ previous_exposure text COLLATE "C" NOT NULL,++ exposure text COLLATE "C" NOT NULL,++ canonical_url text COLLATE "C", -- only a URL that had been public++ at bigint NOT NULL++);++CREATE INDEX idx_removals_owner ON removals (owner, seq);++++CREATE TABLE cdn_purges (++ seq bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,++ removal_seq bigint NOT NULL,++ provider text COLLATE "C" NOT NULL DEFAULT 'cloudflare',++ zone_id text COLLATE "C",++ url text COLLATE "C" NOT NULL,++ state text COLLATE "C" NOT NULL CHECK (state IN ('pending', 'purged', 'failed', 'skipped')),++ detail text COLLATE "C",++ attempts bigint NOT NULL DEFAULT 0,++ next_attempt_at bigint NOT NULL DEFAULT 0,++ created_at bigint NOT NULL,++ done_at bigint++);++-- One pending purge per URL: a sitemap shared by many removals is purged once per batch.++CREATE UNIQUE INDEX idx_cdn_purges_pending ON cdn_purges (zone_id, url) WHERE state = 'pending';++CREATE INDEX idx_cdn_purges_due ON cdn_purges (state, next_attempt_at, seq);++++-- server/saved.js ensureSchema()++CREATE TABLE saved_searches (++ id text COLLATE "C" PRIMARY KEY,++ subject text COLLATE "C" NOT NULL, -- usr_… (the person who saved it)++ name text COLLATE "C" NOT NULL,++ q text COLLATE "C" NOT NULL DEFAULT '',++ filters text COLLATE "C" NOT NULL, -- canonical JSON {owner,type,language,facets}++ query_hash text COLLATE "C" NOT NULL, -- same query saved twice = one row++ created_at bigint NOT NULL,++ updated_at bigint NOT NULL,++ last_run_at bigint,++ UNIQUE (subject, query_hash)++);++CREATE INDEX idx_saved_searches_subject ON saved_searches (subject, created_at);++++-- The full-text index (server/engine/pg.js): one row per servable document, split by audience (exposure 3 public++-- listed: all an anonymous query ever reads; 1 restricted: ACL-checked in the same SQL). Drafts, unpublished, deleted++-- and noindex documents have no row. Title, summary and body carry weights A, B and C. Derived from documents.doc and++-- rebuilt at boot when its counts disagree with the documents table (after the one-time import, for instance).++CREATE TABLE search_fts (++ rid bigint PRIMARY KEY REFERENCES documents(rid) ON DELETE CASCADE,++ audience bigint NOT NULL,++ fresh_at bigint, -- epoch ms of published_at, else updated_at (the freshness boost)++ title text NOT NULL,++ summary text NOT NULL,++ body text NOT NULL,++ tsv tsvector NOT NULL++);++CREATE INDEX search_fts_tsv ON search_fts USING gin (tsv);++CREATE INDEX search_fts_audience ON search_fts (audience, rid);@@@ -14,14 +14,25 @@"better-sqlite3": "^11.10.0","dotenv": "^16.4.7","express": "^4.21.2",-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.49.0",-- "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0",-- "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0"++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",++ "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.20.0",++ "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.27.0",++ "pg": "^8.23.0"++ },++ "devDependencies": {++ "@electric-sql/pglite": "^0.5.8"},"engines": {"node": ">=22.0.0"}},++ "node_modules/@electric-sql/pglite": {++ "version": "0.5.8",++ "resolved": "https://registry.npmjs.org/@electric-sql/pglite/-/pglite-0.5.8.tgz",++ "integrity": "sha512-n9tsbUOhwx2epK1V0ZG9Ar4SHWUju04dhmzZXiSBXwBoleOvIfals33NAaWgagQVAL4Rbvx/Ptsu3P+pA09f6Q==",++ "devOptional": true,++ "license": "Apache-2.0"++ },"node_modules/accepts": {"version": "1.3.8","resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.8.tgz",@@@ -970,9 +981,9 @@}},"node_modules/openvibe-contracts": {-- "version": "0.49.0",-- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.49.0",-- "integrity": "sha512-lapsLoBm3ZpndgiOMELdCBh2PJtuuyqj6SUhQsfwJkxnLdQfyhXfRFgiuLh+iQrg4kqtG9611191Mp8zAfk7fQ==",++ "version": "0.76.0",++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",++ "integrity": "sha512-zTdx776KWrcJKm1JTnf+0+BnsvswdvXy1ldPyse1hSqhnW/SSr98sX696eRetFUhTl8qVM1xSRCykqIUbqItPw==","license": "AGPL-3.0-only","dependencies": {"ajv": "^8.17.1",@@@ -986,18 +997,38 @@}},"node_modules/openvibe-sdk": {-- "version": "0.12.0",-- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0",-- "integrity": "sha512-iG1h905O1f7y/PnP8xLA6GCVzXEBDfSaBj2tc1ucNreWg2sT/AJWosb8NW0WRMK5WS4sIQ6tMsprmDLj/fjjeQ==",++ "version": "0.20.0",++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.20.0",++ "integrity": "sha512-CqZFfVuSv3ZPoyG1LuWCAdvo8TG2ZuUCeGaqTsxDKS3gsymSFV2QXPq/5QUotTd82WShTGUKKhTgcyp9ggUMfQ==","license": "MIT","engines": {"node": ">=20"++ },++ "peerDependencies": {++ "@electric-sql/pglite": "^0.5.8",++ "better-sqlite3": ">=11",++ "iovalkey": "^0.4.0",++ "pg": "^8.23.0"++ },++ "peerDependenciesMeta": {++ "@electric-sql/pglite": {++ "optional": true++ },++ "better-sqlite3": {++ "optional": true++ },++ "iovalkey": {++ "optional": true++ },++ "pg": {++ "optional": true++ }}},"node_modules/openvibe-shared": {-- "version": "1.25.0",-- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0",-- "integrity": "sha512-oqZ+PvljrWi8IEbtg6bDYai17ZV2sALFx/ezMptniELSr10TgFUiGUQt+b1IWwiHv0T2Y1U2iSrkirJxMNgjSw==",++ "version": "1.27.0",++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.27.0",++ "integrity": "sha512-/H7dyFw+cURcdjyfbpyY7mNc6F+Pcb9TwkVwZHR+9d3M62q70KDhZLo/gZi71eaYxUjD8lO8tmbXu70EAa9paQ==","license": "MIT","dependencies": {"jsonwebtoken": "^9.0.0"@@@ -1018,6 +1049,134 @@"integrity": "sha512-A/AGNMFN3c8bOlvV9RreMdrv7jsmF9XIfDeCd87+I8RNg6s78BhJxMu69NEMHBSJFxKidViTEdruRwEk/WIKqA==","license": "MIT"},++ "node_modules/pg": {++ "version": "8.23.0",++ "resolved": "https://registry.npmjs.org/pg/-/pg-8.23.0.tgz",++ "integrity": "sha512-Ip2EQCngowJLGOfCwkFhPXU7/ljlhn6Rxlmy4XYfL2Y+vyRM59+8uR2xqRWKdYmbXmxCFOAmKxBuSUCdF34qLg==",++ "license": "MIT",++ "dependencies": {++ "pg-connection-string": "^2.14.0",++ "pg-pool": "^3.14.0",++ "pg-protocol": "^1.16.0",++ "pg-types": "2.2.0",++ "pgpass": "1.0.5"++ },++ "engines": {++ "node": ">= 16.0.0"++ },++ "optionalDependencies": {++ "pg-cloudflare": "^1.4.0"++ },++ "peerDependencies": {++ "pg-native": ">=3.0.1"++ },++ "peerDependenciesMeta": {++ "pg-native": {++ "optional": true++ }++ }++ },++ "node_modules/pg-cloudflare": {++ "version": "1.4.0",++ "resolved": "https://registry.npmjs.org/pg-cloudflare/-/pg-cloudflare-1.4.0.tgz",++ "integrity": "sha512-Vo7z/6rrQYxpNRylp4Tlob2elzbh+N/MOQbxFVWCxS7oEx6jF53GTJFxK2WWpKuBRkmiin4Mt+xofFDjx09R0A==",++ "license": "MIT",++ "optional": true++ },++ "node_modules/pg-connection-string": {++ "version": "2.14.0",++ "resolved": "https://registry.npmjs.org/pg-connection-string/-/pg-connection-string-2.14.0.tgz",++ "integrity": "sha512-XwWDGcLRGCXAR8F/AM5bG7Q+A3Wm2s6QeEjlOKZLlH3UYcguiqCWKyWXVag5TLTIjR7oOJUY8kcADaZgWPyLeg==",++ "license": "MIT"++ },++ "node_modules/pg-int8": {++ "version": "1.0.1",++ "resolved": "https://registry.npmjs.org/pg-int8/-/pg-int8-1.0.1.tgz",++ "integrity": "sha512-WCtabS6t3c8SkpDBUlb1kjOs7l66xsGdKpIPZsg4wR+B3+u9UAum2odSsF9tnvxg80h4ZxLWMy4pRjOsFIqQpw==",++ "license": "ISC",++ "engines": {++ "node": ">=4.0.0"++ }++ },++ "node_modules/pg-pool": {++ "version": "3.14.0",++ "resolved": "https://registry.npmjs.org/pg-pool/-/pg-pool-3.14.0.tgz",++ "integrity": "sha512-gKtPkFdQPU3DksooVLi9LsjZxrsBUZIpa+7aVx+LV5pNh0KzP4Zleud2po+ConrxbuXGBJ6Hfer6hdgpIBpBaw==",++ "license": "MIT",++ "peerDependencies": {++ "pg": ">=8.0"++ }++ },++ "node_modules/pg-protocol": {++ "version": "1.16.0",++ "resolved": "https://registry.npmjs.org/pg-protocol/-/pg-protocol-1.16.0.tgz",++ "integrity": "sha512-sILXutLVjCLjcDuOmvhX5e2Z4cS5qG/6Bu3VkpFwdf/633ElGLpEh9bgmuI5I4sqKqkifQiGyiCcx1HdtrK7tg==",++ "license": "MIT"++ },++ "node_modules/pg-types": {++ "version": "2.2.0",++ "resolved": "https://registry.npmjs.org/pg-types/-/pg-types-2.2.0.tgz",++ "integrity": "sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA==",++ "license": "MIT",++ "dependencies": {++ "pg-int8": "1.0.1",++ "postgres-array": "~2.0.0",++ "postgres-bytea": "~1.0.0",++ "postgres-date": "~1.0.4",++ "postgres-interval": "^1.1.0"++ },++ "engines": {++ "node": ">=4"++ }++ },++ "node_modules/pgpass": {++ "version": "1.0.5",++ "resolved": "https://registry.npmjs.org/pgpass/-/pgpass-1.0.5.tgz",++ "integrity": "sha512-FdW9r/jQZhSeohs1Z3sI1yxFQNFvMcnmfuj4WBMUTxOrAyLMaTcE1aAMBiTlbMNaXvBCQuVi0R7hd8udDSP7ug==",++ "license": "MIT",++ "dependencies": {++ "split2": "^4.1.0"++ }++ },++ "node_modules/postgres-array": {++ "version": "2.0.0",++ "resolved": "https://registry.npmjs.org/postgres-array/-/postgres-array-2.0.0.tgz",++ "integrity": "sha512-VpZrUqU5A69eQyW2c5CA1jtLecCsN2U/bD6VilrFDWq5+5UIEVO7nazS3TEcHf1zuPYO/sqGvUvW62g86RXZuA==",++ "license": "MIT",++ "engines": {++ "node": ">=4"++ }++ },++ "node_modules/postgres-bytea": {++ "version": "1.0.1",++ "resolved": "https://registry.npmjs.org/postgres-bytea/-/postgres-bytea-1.0.1.tgz",++ "integrity": "sha512-5+5HqXnsZPE65IJZSMkZtURARZelel2oXUEO8rH83VS/hxH5vv1uHquPg5wZs8yMAfdv971IU+kcPUczi7NVBQ==",++ "license": "MIT",++ "engines": {++ "node": ">=0.10.0"++ }++ },++ "node_modules/postgres-date": {++ "version": "1.0.7",++ "resolved": "https://registry.npmjs.org/postgres-date/-/postgres-date-1.0.7.tgz",++ "integrity": "sha512-suDmjLVQg78nMK2UZ454hAG+OAW+HQPZ6n++TNDUX+L0+uUlLywnoxJKDou51Zm+zTCjrCl0Nq6J9C5hP9vK/Q==",++ "license": "MIT",++ "engines": {++ "node": ">=0.10.0"++ }++ },++ "node_modules/postgres-interval": {++ "version": "1.2.0",++ "resolved": "https://registry.npmjs.org/postgres-interval/-/postgres-interval-1.2.0.tgz",++ "integrity": "sha512-9ZhXKM/rw350N1ovuWHbGxnGh/SNJ4cnxHiM0rxE4VN41wsg8P8zWn9hv/buK00RP4WvlOyr/RBDiptyxVbkZQ==",++ "license": "MIT",++ "dependencies": {++ "xtend": "^4.0.0"++ },++ "engines": {++ "node": ">=0.10.0"++ }++ },"node_modules/prebuild-install": {"version": "7.1.3","resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz",@@@ -1356,6 +1515,15 @@"simple-concat": "^1.0.0"}},++ "node_modules/split2": {++ "version": "4.2.0",++ "resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz",++ "integrity": "sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==",++ "license": "ISC",++ "engines": {++ "node": ">= 10.x"++ }++ },"node_modules/statuses": {"version": "2.0.2","resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz",@@@ -1483,6 +1651,15 @@"resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz","integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==","license": "ISC"++ },++ "node_modules/xtend": {++ "version": "4.0.2",++ "resolved": "https://registry.npmjs.org/xtend/-/xtend-4.0.2.tgz",++ "integrity": "sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ==",++ "license": "MIT",++ "engines": {++ "node": ">=0.4"++ }}}}@@@ -8,9 +8,15 @@"start": "node server/index.js","dev": "NODE_ENV=development node server/index.js","test": "node test/run.js",-- "subscribe": "node scripts/subscribe.js"++ "subscribe": "node scripts/subscribe.js",++ "test:pg": "SEARCH_TEST_STORE=pg node test/run.js"},-- "keywords": ["openvibe", "search", "fts5", "acl"],++ "keywords": [++ "openvibe",++ "search",++ "fts5",++ "acl"++ ],"author": "OpenVibers","license": "AGPL-3.0-only","repository": "github:OpenVibers/OpenVibe.Search",@@@ -20,11 +26,15 @@"better-sqlite3": "^11.10.0","dotenv": "^16.4.7","express": "^4.21.2",-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.49.0",-- "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0",-- "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0"++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",++ "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.20.0",++ "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.27.0",++ "pg": "^8.23.0"},"engines": {"node": ">=22.0.0"++ },++ "devDependencies": {++ "@electric-sql/pglite": "^0.5.8"}}nnew file mode 100644@@@ -0,0 +1,26 @@++#!/usr/bin/env node++'use strict';++/**++ * The one-time move of Search's SQLite database (SEARCH_DB_PATH) into its PostgreSQL schema (ADR-035; the procedure is++ * openvibe-sdk docs/migrating-to-postgresql.md, section 6, carried out by openvibe-sdk/db runSqliteMigration).++ *++ * node scripts/migrate-to-postgres.js [--sqlite <file>] [--pglite] [--json]++ *++ * Applies migrations/ as the owner (DATABASE_DIRECT_URL), copies every table into emptied tables, verifies row counts++ * and checksums, and exits 1 unless everything verified. The SQLite file is opened read-only. Every table keeps its++ * name and columns.++ */++require('dotenv').config();++const { runSqliteMigration } = require('openvibe-sdk/db');++const configLib = require('../server/config');++const { MIGRATIONS } = require('../server/db');++++const TABLES = {};++++if (require.main === module) {++ const config = configLib.load();++ runSqliteMigration({ service: 'search', sqlite: config.dbPath, directUrl: config.db.directUrl, migrations: MIGRATIONS, tables: TABLES })++ .then((code) => process.exit(code), (err) => { console.error(`migrate-to-postgres failed: ${err.message}`); process.exit(1); });++}++++module.exports = { TABLES };@@@ -64,7 +64,7 @@ function documentsRouter({ store, auth, db, relay, purges, limits }) {return true;}-- router.put('/api/v1/documents/:owner/:type/:id', guard, limits('search.document.index', index), (req, res) => {++ router.put('/api/v1/documents/:owner/:type/:id', guard, limits('search.document.index', index), async (req, res) => {const ctx = req.ov;if (!ownerCheck(req, res)) return;const body = req.body && typeof req.body === 'object' && !Array.isArray(req.body) ? req.body : null;@@@ -77,12 +77,12 @@ function documentsRouter({ store, auth, db, relay, purges, limits }) {const doc = { ...body, owner: req.params.owner, type: req.params.type, id: req.params.id };const v = validate(doc);if (!v.valid) return http.sendProblem(res, 422, 'search.bad_document', { detail: 'document does not match search.index-document@1', ctx, errors: v.errors });-- const r = store.apply(doc, { via: 'api', traceId: ctx.traceId });++ const r = await store.apply(doc, { via: 'api', traceId: ctx.traceId });if (r.outcome === 'applied' && relay) relay.flush().catch(() => {});return outcomeResponse(res, ctx, r);});-- router.delete('/api/v1/documents/:owner/:type/:id', guard, limits('search.document.remove', index), (req, res) => {++ router.delete('/api/v1/documents/:owner/:type/:id', guard, limits('search.document.remove', index), async (req, res) => {const ctx = req.ov;if (!ownerCheck(req, res)) return;let revision = null;@@@ -90,25 +90,25 @@ function documentsRouter({ store, auth, db, relay, purges, limits }) {if (!/^\d{1,15}$/.test(String(req.query.revision))) return http.sendProblem(res, 400, 'search.bad_revision', { detail: 'revision must be a non-negative integer', ctx });revision = Number(req.query.revision);}-- const r = store.remove(req.params.owner, req.params.type, req.params.id, { revision, via: 'api', traceId: ctx.traceId });++ const r = await store.remove(req.params.owner, req.params.type, req.params.id, { revision, via: 'api', traceId: ctx.traceId });if (r.outcome === 'applied' && relay) relay.flush().catch(() => {});return outcomeResponse(res, ctx, r);});-- router.get('/api/v1/owners/:owner/documents', guard, limits('search.owner.documents'), (req, res) => {++ router.get('/api/v1/owners/:owner/documents', guard, limits('search.owner.documents'), async (req, res) => {const ctx = req.ov;if (!ownerCheck(req, res)) return;const type = req.query.type ? String(req.query.type) : null;if (type && !TYPE_RE.test(type)) return http.sendProblem(res, 400, 'search.bad_identity', { detail: 'type is malformed', ctx });const after = /^\d{1,15}$/.test(String(req.query.after || '')) ? Number(req.query.after) : 0;const limit = Math.min(Math.max(parseInt(req.query.limit, 10) || 500, 1), 1000);-- res.json({ owner: req.params.owner, ...store.ownerPage(req.params.owner, { type, after, limit }) });++ res.json({ owner: req.params.owner, ...await store.ownerPage(req.params.owner, { type, after, limit }) });});-- router.get('/api/v1/owners/:owner/documents/:type/:id', guard, limits('search.owner.document'), (req, res) => {++ router.get('/api/v1/owners/:owner/documents/:type/:id', guard, limits('search.owner.document'), async (req, res) => {const ctx = req.ov;if (!ownerCheck(req, res)) return;-- const found = store.get(req.params.owner, req.params.type, req.params.id);++ const found = await store.get(req.params.owner, req.params.type, req.params.id);if (!found) return http.sendProblem(res, 404, 'search.not_found', { detail: 'no such document', ctx });res.json({document: found.doc,@@@ -120,10 +120,10 @@ function documentsRouter({ store, auth, db, relay, purges, limits }) {});});-- router.get('/api/v1/owners/:owner/rejections', guard, limits('search.owner.rejections'), (req, res) => {++ router.get('/api/v1/owners/:owner/rejections', guard, limits('search.owner.rejections'), async (req, res) => {if (!ownerCheck(req, res)) return;const after = /^\d{1,15}$/.test(String(req.query.after || '')) ? Number(req.query.after) : 0;-- const rows = db.prepare('SELECT * FROM ingest_rejections WHERE owner = ? AND seq > ? ORDER BY seq LIMIT 200').all(req.params.owner, after);++ const rows = await db.prepare('SELECT * FROM ingest_rejections WHERE owner = ? AND seq > ? ORDER BY seq LIMIT 200').all(req.params.owner, after);res.json({rejections: rows.map(r => ({seq: r.seq, event_id: r.event_id, event_type: r.event_type, type: r.type, id: r.id, revision: r.revision,@@@ -133,11 +133,11 @@ function documentsRouter({ store, auth, db, relay, purges, limits }) {});});-- router.get('/api/v1/owners/:owner/removals', guard, limits('search.owner.removals'), (req, res) => {++ router.get('/api/v1/owners/:owner/removals', guard, limits('search.owner.removals'), async (req, res) => {if (!ownerCheck(req, res)) return;const after = /^\d{1,15}$/.test(String(req.query.after || '')) ? Number(req.query.after) : 0;const limit = Math.min(Math.max(parseInt(req.query.limit, 10) || 200, 1), 1000);-- res.json({ owner: req.params.owner, ...purges.ownerFeed(req.params.owner, { after, limit }) });++ res.json({ owner: req.params.owner, ...await purges.ownerFeed(req.params.owner, { after, limit }) });});return router;@@@ -19,7 +19,7 @@const crypto = require('crypto');const express = require('express');const { http } = require('openvibe-contracts');--const { toMatch, snippetHtml } = require('../engine/fts5');++const { toMatch, snippetHtml } = require('../engine/pg');const { EXPOSURE } = require('../document');const { AuthError } = require('../auth');const { OWNER_RE, TYPE_RE, ID_RE } = require('./documents');@@@ -119,7 +119,7 @@ function canView(viewer, row, doc) {* QueryError for a malformed request.*/function createSearcher({ config, store, engine, now: clock = () => Date.now() }) {-- function run({ text = '', query = {}, filters = null, viewer, limit, cursor, facetKeys = [], now = clock() }) {++ async function run({ text = '', query = {}, filters = null, viewer, limit, cursor, facetKeys = [], now = clock() }) {if (text.length > 500) throw new QueryError('search.bad_query', 'q is longer than 500 characters');filters = filters || parseFilters(query, config);limit = Math.min(Math.max(parseInt(limit, 10) || config.query.defaultLimit, 1), config.query.maxLimit);@@@ -136,14 +136,14 @@ function createSearcher({ config, store, engine, now: clock = () => Date.now() }if (match) {if (after && after.rank === undefined) throw new QueryError('search.bad_cursor', 'cursor does not belong to this query');if (after && after.now != null) refNow = after.now;-- page = engine.search({ match, filters, viewer, limit, after, now: refNow });++ page = await engine.search({ match, filters, viewer, limit, after, now: refNow });} else {if (after && after.sort_at === undefined) throw new QueryError('search.bad_cursor', 'cursor does not belong to this query');-- page = engine.browse({ filters, viewer, limit, after });++ page = await engine.browse({ filters, viewer, limit, after });}const results = [];for (const h of page.hits) {-- const found = store.byRid(h.rid);++ const found = await store.byRid(h.rid);// Defence in depth: the engine already filtered; re-check against the stored row.if (!found || !canView(viewer, found.row, found.doc)) continue;results.push(projection(found.doc, found.row, match ? h.snip : undefined));@@@ -152,7 +152,7 @@ function createSearcher({ config, store, engine, now: clock = () => Date.now() }? encodeCursor(match ? { k: 'r', r: page.next.rank, i: page.next.rid, n: refNow, h: hash } : { k: 't', t: page.next.sort_at, i: page.next.rid, h: hash }): null;const body = { results, next_cursor: next };-- if (facetKeys.length) body.facets = engine.facetCounts({ match, filters, viewer, keys: facetKeys });++ if (facetKeys.length) body.facets = await engine.facetCounts({ match, filters, viewer, keys: facetKeys });return body;}@@@ -168,7 +168,7 @@ function createSearcher({ config, store, engine, now: clock = () => Date.now() }* (a per-actor limit, server/actor-limits.js) the viewer is counted as req.viewer before the handler runs.*/function withViewer(auth, handler, limit = null) {-- return (req, res, next) => {++ return async (req, res, next) => {const ctx = req.ov;let viewer;try {@@@ -181,16 +181,16 @@ function withViewer(auth, handler, limit = null) {// personalised answer must never reach a shared cache.res.setHeader('Cache-Control', 'no-store');res.setHeader('Vary', 'Authorization, Cookie, X-OV-Subject, X-OV-Groups, X-OV-Entitlements');-- const run = () => {++ const run = async () => {try {-- return handler(req, res, viewer);++ return await handler(req, res, viewer);} catch (err) {if (err instanceof QueryError) return http.sendProblem(res, 400, err.code, { detail: err.message, ctx });return next(err);}};req.viewer = viewer;-- return limit ? limit(req, res, run) : run();++ return limit ? limit(req, res, run) : await run();};}@@@ -200,9 +200,9 @@ function queryRouter({ config, store, engine, auth, limits, searcher = createSea// Per-actor limits (server/actor-limits.js): queries and document reads take the defaults. Suggest// answers a search box as the person types, several requests a second, so it allows 300 a minute.-- router.get('/api/v1/search', guarded(limits('search.query'), (req, res, viewer) => {++ router.get('/api/v1/search', guarded(limits('search.query'), async (req, res, viewer) => {const q = one(req.query.q);-- res.json(searcher.run({++ res.json(await searcher.run({text: q === undefined ? '' : String(q),query: req.query,viewer,@@@ -212,27 +212,27 @@ function queryRouter({ config, store, engine, auth, limits, searcher = createSea}));}));-- router.get('/api/v1/suggest', guarded(limits('search.suggest', { minute: 300, hour: 6000 }), (req, res, viewer) => {++ router.get('/api/v1/suggest', guarded(limits('search.suggest', { minute: 300, hour: 6000 }), async (req, res, viewer) => {const text = String(one(req.query.q) || '');if (text.length > 100) throw new QueryError('search.bad_query', 'q is longer than 100 characters');const filters = parseFilters(req.query, config);const limit = Math.min(Math.max(parseInt(one(req.query.limit), 10) || 8, 1), 20);const match = toMatch(text, { maxTerms: 6, column: 'title', prefixLast: true });if (!match) return res.json({ suggestions: [] });-- const rows = engine.suggest({ match, filters, viewer, limit });++ const rows = await engine.suggest({ match, filters, viewer, limit });const suggestions = [];for (const r of rows) {-- const found = store.byRid(r.rid);++ const found = await store.byRid(r.rid);if (!found || !canView(viewer, found.row, found.doc)) continue;suggestions.push({ owner: found.doc.owner, type: found.doc.type, id: found.doc.id, title: found.doc.title, canonical_url: found.doc.canonical_url });}res.json({ suggestions });}));-- router.get('/api/v1/documents/:owner/:type/:id', guarded(limits('search.document.read'), (req, res, viewer) => {++ router.get('/api/v1/documents/:owner/:type/:id', guarded(limits('search.document.read'), async (req, res, viewer) => {const { owner, type, id } = req.params;if (!OWNER_RE.test(owner) || !TYPE_RE.test(type) || !ID_RE.test(id)) throw new QueryError('search.bad_identity', 'owner, type or id is malformed');-- const found = store.get(owner, type, id);++ const found = await store.get(owner, type, id);if (!found || !canView(viewer, found.row, found.doc)) {return http.sendProblem(res, 404, 'search.not_found', { detail: 'no such document', ctx: req.ov });}@@@ -53,13 +53,13 @@ function savedRouter({ config, saved, searcher, auth, limits }) {const guarded = (limit, h) => withViewer(auth, h, limit);const change = { minute: 30, hour: 300 };-- router.get('/api/v1/saved-searches', guarded(limits('search.saved.list'), (req, res, viewer) => {++ router.get('/api/v1/saved-searches', guarded(limits('search.saved.list'), async (req, res, viewer) => {const subject = person(req, res, viewer);if (!subject) return;-- res.json({ saved_searches: saved.list(subject), max: saved.maxPerSubject });++ res.json({ saved_searches: await saved.list(subject), max: saved.maxPerSubject });}));-- router.post('/api/v1/saved-searches', guarded(limits('search.saved.create', change), (req, res, viewer) => {++ router.post('/api/v1/saved-searches', guarded(limits('search.saved.create', change), async (req, res, viewer) => {const ctx = req.ov;const subject = person(req, res, viewer, { write: true });if (!subject) return;@@@ -89,39 +89,39 @@ function savedRouter({ config, saved, searcher, auth, limits }) {const text = q.trim();if (text && !searcher.hasWords(text)) throw new QueryError('search.bad_query', 'q has no searchable words');if (!text && !hasFilter) throw new QueryError('search.bad_saved_search', 'a saved search needs words (q) or at least one filter');-- const r = saved.save(subject, { name: name.trim(), q: text, filters });++ const r = await saved.save(subject, { name: name.trim(), q: text, filters });if (r.error === 'limit') return http.sendProblem(res, 409, 'search.saved_search_limit', { detail: `at most ${saved.maxPerSubject} saved searches per person`, ctx });res.status(r.created ? 201 : 200).json({ saved_search: r.saved });}));-- router.get('/api/v1/saved-searches/:id', guarded(limits('search.saved.read'), (req, res, viewer) => {++ router.get('/api/v1/saved-searches/:id', guarded(limits('search.saved.read'), async (req, res, viewer) => {const subject = person(req, res, viewer);if (!subject) return;-- const s = saved.get(subject, req.params.id);++ const s = await saved.get(subject, req.params.id);if (!s) return http.sendProblem(res, 404, 'search.not_found', { detail: 'no such saved search', ctx: req.ov });res.json({ saved_search: s });}));-- router.delete('/api/v1/saved-searches/:id', guarded(limits('search.saved.delete', change), (req, res, viewer) => {++ router.delete('/api/v1/saved-searches/:id', guarded(limits('search.saved.delete', change), async (req, res, viewer) => {const subject = person(req, res, viewer, { write: true });if (!subject) return;-- if (!saved.remove(subject, req.params.id)) return http.sendProblem(res, 404, 'search.not_found', { detail: 'no such saved search', ctx: req.ov });++ if (!await saved.remove(subject, req.params.id)) return http.sendProblem(res, 404, 'search.not_found', { detail: 'no such saved search', ctx: req.ov });res.status(204).end();}));-- router.get('/api/v1/saved-searches/:id/results', guarded(limits('search.saved.run'), (req, res, viewer) => {++ router.get('/api/v1/saved-searches/:id/results', guarded(limits('search.saved.run'), async (req, res, viewer) => {const subject = person(req, res, viewer);if (!subject) return;-- const s = saved.get(subject, req.params.id);++ const s = await saved.get(subject, req.params.id);if (!s) return http.sendProblem(res, 404, 'search.not_found', { detail: 'no such saved search', ctx: req.ov });-- const body = searcher.run({++ const body = await searcher.run({text: s.q,filters: { ...s.filters, facets: s.filters.facets || [] },viewer,limit: one(req.query.limit),cursor: one(req.query.cursor),});-- saved.markRun(s.id);++ await saved.markRun(s.id);res.json({ saved_search: { id: s.id, name: s.name }, ...body });}));@@@ -73,23 +73,23 @@ function verifySignature(raw, header, secrets) {}function createInbox(db, { now = () => Date.now() } = {}) {-- const claim = db.prepare('INSERT OR IGNORE INTO idempotency_receipts (consumer, event_id, outcome, processed_at) VALUES (?, ?, ?, ?)');++ const claim = db.prepare('INSERT INTO idempotency_receipts (consumer, event_id, outcome, processed_at) VALUES (?, ?, ?, ?) ON CONFLICT DO NOTHING');const setOutcome = db.prepare('UPDATE idempotency_receipts SET outcome = ? WHERE consumer = ? AND event_id = ?');const reject = db.prepare(`INSERT INTO ingest_rejections (event_id, event_type, owner, type, id, revision, code, detail, at)VALUES (@event_id, @event_type, @owner, @type, @id, @revision, @code, @detail, @at)`);/** fn() runs inside the transaction and returns an outcome string. */-- function once(eventId, fn) {-- return db.transaction(() => {-- if (claim.run(CONSUMER, eventId, 'processing', now()).changes === 0) return { duplicate: true };-- const outcome = fn();-- setOutcome.run(outcome, CONSUMER, eventId);++ async function once(eventId, fn) {++ return await db.tx(async () => {++ if ((await claim.run(CONSUMER, eventId, 'processing', now())).changes === 0) return { duplicate: true };++ const outcome = await fn();++ await setOutcome.run(outcome, CONSUMER, eventId);return { duplicate: false, outcome };-- })();++ });}-- function recordRejection(r) {-- reject.run({ owner: null, type: null, id: null, revision: null, detail: null, event_type: null, ...r, at: now() });++ async function recordRejection(r) {++ await reject.run({ owner: null, type: null, id: null, revision: null, detail: null, event_type: null, ...r, at: now() });}return { once, recordRejection };@@@ -125,7 +125,7 @@ function webhookRouter({ config, db, store, relay, log = console, now }) {router.post('/internal/events',express.raw({ type: () => true, limit: config.maxBodyBytes }),-- (req, res) => {++ async (req, res, next) => { try {const ctx = req.ov;const secrets = config.events.webhookSecrets;if (!secrets.length) return http.sendProblem(res, 503, 'search.webhook_disabled', { detail: 'SEARCH_EVENTS_SECRET is not set', ctx });@@@ -141,12 +141,12 @@ function webhookRouter({ config, db, store, relay, log = console, now }) {return http.sendProblem(res, 400, 'search.bad_delivery', { detail: 'body must be { event: <envelope>, seq }', ctx });}-- const r = inbox.once(event.event_id, () => {++ const r = await inbox.once(event.event_id, async () => {const parsed = documentFromEvent(event, config.events.owners);if (parsed.ignore) return 'ignored';if (parsed.reject) {const p = event.payload || {};-- inbox.recordRejection({++ await inbox.recordRejection({event_id: event.event_id, event_type: event.event_type, owner: event.source,type: typeof p.type === 'string' ? p.type.slice(0, 40) : null,id: typeof p.id === 'string' ? p.id.slice(0, 128) : null,@@@ -157,12 +157,12 @@ function webhookRouter({ config, db, store, relay, log = console, now }) {return `rejected:${parsed.reject.code}`;}const traceId = typeof event.trace_id === 'string' ? event.trace_id : null;-- const out = store.apply(parsed.doc, { via: 'event', eventId: event.event_id, traceId });++ const out = await store.apply(parsed.doc, { via: 'event', eventId: event.event_id, traceId });return out.outcome;});if (!r.duplicate && r.outcome === 'applied' && relay) relay.flush().catch(() => {});res.status(200).json({ event_id: event.event_id, duplicate: Boolean(r.duplicate), outcome: r.outcome || null });-- });++ } catch (err) { return next(err); } }); // Express 4 does not catch an async handler's rejectionreturn router;}@@@ -23,7 +23,7 @@ function createApp({ config, db, store, engine, auth, keys, outbox, relay, purge// HTTP golden signals by route template, process metrics, release_info and the Search gauges;// GET /metrics answers direct loopback callers only (Track O).const metrics = instrument(app, { service: 'search', release: release.release });-- registerSearchGauges(metrics.registry, { db, outbox, purges });++ registerSearchGauges(metrics.registry, { db, engine, outbox, purges });app.use(http.middleware());// One W3C trace across services (openvibe-shared/trace): calls made while serving a request carry its traceparent.require('openvibe-shared/trace').install(app);@@@ -61,7 +61,7 @@ function createKeyStore({ urls = [], pem = null, fetchImpl = globalThis.fetch, lreturn null;}-- function start() {++ async function start() {if (pem) return Promise.resolve(key);const attempt = async () => {const k = await fetchOnce();@@@ -73,7 +73,7 @@ function createKeyStore({ urls = [], pem = null, fetchImpl = globalThis.fetch, l};refreshTimer = setInterval(() => { fetchOnce().catch(() => {}); }, 6 * 60 * 60 * 1000);refreshTimer.unref?.();-- return attempt();++ return await attempt();}function stop() {@@@ -70,6 +70,9 @@ function load(env = process.env) {clientSecret: env.OV_OAUTH_CLIENT_SECRET || '',},++ // PostgreSQL (ADR-035): DATABASE_URL serves (PgBouncer), DATABASE_DIRECT_URL migrates (owner role).++ db: { url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '' },++ // The SQLite file of releases before the switch: read once by scripts/migrate-to-postgres.js.dbPath: env.SEARCH_DB_PATH || './data/search.db',// Events: the outbox relays only when EVENTS_URL is set (rows wait otherwise).@@@ -1,99 +1,32 @@'use strict';/**-- * The Search database (SQLite, WAL). One file owned by this service only; created on boot,-- * idempotently. The FTS5 tables belong to the engine (server/engine/fts5.js).++ * The Search database: PostgreSQL (ADR-035), owned by this service only; the schema is migrations/NNNN_*.sql, applied++ * at boot. The full-text table search_fts belongs to the engine (server/engine/pg.js).*/const fs = require('fs');const path = require('path');--const Database = require('better-sqlite3');--const outbox = require('./events/outbox');--const purge = require('./purge');--const saved = require('./saved');++const { createDb } = require('openvibe-sdk/db');--const SCHEMA = `--CREATE TABLE IF NOT EXISTS documents (-- rid INTEGER PRIMARY KEY,-- owner TEXT NOT NULL,-- type TEXT NOT NULL,-- id TEXT NOT NULL,-- revision INTEGER NOT NULL,-- deleted INTEGER NOT NULL DEFAULT 0,-- visibility TEXT,-- publication_state TEXT,-- exposure INTEGER NOT NULL DEFAULT 0, -- document.js EXPOSURE (0 none .. 3 public listed)-- index_decision TEXT NOT NULL,-- noindex_reasons TEXT NOT NULL DEFAULT '[]',-- canonical_url TEXT,-- title TEXT,-- language TEXT,-- published_at TEXT,-- updated_at TEXT,-- sort_at TEXT NOT NULL DEFAULT '', -- updated_at, else published_at: browse order-- doc TEXT NOT NULL, -- normalized document JSON (tombstone: identity only)-- hash TEXT NOT NULL,-- via TEXT NOT NULL, -- event | api-- event_id TEXT,-- indexed_at INTEGER NOT NULL,-- UNIQUE (owner, type, id)--);--CREATE INDEX IF NOT EXISTS idx_documents_exposure ON documents (exposure, sort_at, rid);--CREATE INDEX IF NOT EXISTS idx_documents_owner ON documents (owner, type, id);++const MIGRATIONS = path.join(__dirname, '..', 'migrations');++const DEV_PGLITE = path.join(__dirname, '..', 'data', 'pglite');--CREATE TABLE IF NOT EXISTS doc_acl (-- rid INTEGER NOT NULL,-- kind TEXT NOT NULL CHECK (kind IN ('s', 'g', 'e')),-- value TEXT NOT NULL,-- PRIMARY KEY (rid, kind, value)--) WITHOUT ROWID;--CREATE INDEX IF NOT EXISTS idx_doc_acl_value ON doc_acl (kind, value, rid);----CREATE TABLE IF NOT EXISTS doc_facets (-- rid INTEGER NOT NULL,-- key TEXT NOT NULL,-- value TEXT NOT NULL,-- PRIMARY KEY (rid, key, value)--) WITHOUT ROWID;--CREATE INDEX IF NOT EXISTS idx_doc_facets_value ON doc_facets (key, value, rid);----CREATE TABLE IF NOT EXISTS idempotency_receipts (-- consumer TEXT NOT NULL,-- event_id TEXT NOT NULL,-- outcome TEXT NOT NULL,-- processed_at INTEGER NOT NULL,-- PRIMARY KEY (consumer, event_id)--);----CREATE TABLE IF NOT EXISTS ingest_rejections (-- seq INTEGER PRIMARY KEY AUTOINCREMENT,-- event_id TEXT NOT NULL,-- event_type TEXT,-- owner TEXT,-- type TEXT,-- id TEXT,-- revision INTEGER,-- code TEXT NOT NULL,-- detail TEXT,-- at INTEGER NOT NULL--);--CREATE INDEX IF NOT EXISTS idx_ingest_rejections_owner ON ingest_rejections (owner, seq);--`;----function openDb(dbPath) {-- if (dbPath !== ':memory:') fs.mkdirSync(path.dirname(path.resolve(dbPath)), { recursive: true });-- const db = new Database(dbPath);-- db.pragma('journal_mode = WAL');-- db.pragma('synchronous = NORMAL');-- db.pragma('foreign_keys = ON');-- db.pragma('busy_timeout = 5000');-- db.exec(SCHEMA);-- outbox.ensureSchema(db);-- purge.ensureSchema(db);-- saved.ensureSchema(db);-- return db;++/**++ * The serving handle (ADR-035): DATABASE_URL through PgBouncer; in development without it, an embedded PGlite database++ * in data/pglite. Migrations run first, as the owner (DATABASE_DIRECT_URL), or on the embedded handle.++ */++async function openDb(config, { log = console, registry } = {}) {++ if (!config.db.url) {++ if (config.isProduction) throw new Error('DATABASE_URL is not set: production serves from PostgreSQL (OpenVibe.Host roles/data add-service.sh search)');++ log.warn(`[search] DATABASE_URL unset: embedded PGlite database in ${DEV_PGLITE} (development only, one process)`);++ fs.mkdirSync(DEV_PGLITE, { recursive: true });++ const db = createDb({ pglite: DEV_PGLITE, service: 'search', registry, log });++ await db.migrate({ dir: MIGRATIONS, log });++ return db;++ }++ if (!config.db.directUrl) throw new Error('DATABASE_DIRECT_URL is not set: migrations run with the owner role on a direct connection');++ const owner = createDb({ url: config.db.directUrl, service: 'search-migrate', max: 1, log });++ try { await owner.migrate({ dir: MIGRATIONS, log }); } finally { await owner.close(); }++ return createDb({ url: config.db.url, service: 'search', registry, log });}--module.exports = { openDb };++module.exports = { openDb, MIGRATIONS };ddeleted file mode 100644@@@ -1,235 +0,0 @@--'use strict';--/**-- * Search engine: SQLite FTS5 in the service's own database (decision: docs/adr-engine.md).-- *-- * Two full-text tables, split by audience so an anonymous query never touches, ranks against or-- * counts anything but public listed documents:-- *-- * fts_public documents with exposure public_listed (public, published, indexable)-- * fts_restricted documents with exposure restricted (unlisted/members/private, published)-- *-- * Drafts, unpublished, deleted and public-noindex documents are in neither, so no MATCH can-- * reach them. Every query is also filtered by `documents.exposure` and, for restricted rows, by-- * the viewer's ACL keys (doc_acl) in the same SQL: visibility is decided before anything is-- * ranked, counted, snippeted or suggested.-- *-- * The interface (put/remove/search/browse/facetCounts/suggest) is what a PostgreSQL FTS engine-- * would implement; nothing outside this file writes SQL against the FTS tables.-- */--const { EXPOSURE } = require('../document');----const TOKENIZE = "tokenize = 'unicode61 remove_diacritics 2'";--const WEIGHTS = '10.0, 4.0, 1.0'; // title, summary, body--const MARK_OPEN = '\u0001';--const MARK_CLOSE = '\u0002';----const DAY_MS = 24 * 3600 * 1000;----/**-- * Freshness multiplier for a document dated `iso` at reference time `nowMs`:-- *-- * 1 + weight × 2^(−age_days / halfLifeDays)-- *-- * With the defaults (weight 1, half-life 30 days) a document published today counts ×2.0, one-- * 30 days old ×1.5, 60 days ×1.25, 90 days ×1.125, a year old ×1.0002: relevance still decides,-- * and between comparably relevant documents the newer one ranks first. A future date counts as-- * today; no date (or an unparsable one) counts ×1. Weight 0 turns the boost off.-- */--function freshnessBoost(iso, nowMs, halfLifeDays, weight) {-- if (!weight || !iso) return 1;-- const t = Date.parse(iso);-- if (!Number.isFinite(t)) return 1;-- const ageDays = Math.max(0, (nowMs - t) / DAY_MS);-- return 1 + weight * Math.pow(2, -ageDays / halfLifeDays);--}----function createEngine(db, { freshness = { weight: 1, halfLifeDays: 30 } } = {}) {-- // bm25() is negative (lower = more relevant), so multiplying by a boost ≥ 1 ranks newer first.-- db.function('ov_freshness', { deterministic: true }, (iso, nowMs, halfLifeDays, weight) => freshnessBoost(iso, nowMs, halfLifeDays, weight));-- db.exec(`-- CREATE VIRTUAL TABLE IF NOT EXISTS fts_public USING fts5(title, summary, body, ${TOKENIZE});-- CREATE VIRTUAL TABLE IF NOT EXISTS fts_restricted USING fts5(title, summary, body, ${TOKENIZE});-- `);-- const st = {-- delPublic: db.prepare('DELETE FROM fts_public WHERE rowid = ?'),-- delRestricted: db.prepare('DELETE FROM fts_restricted WHERE rowid = ?'),-- insPublic: db.prepare('INSERT INTO fts_public (rowid, title, summary, body) VALUES (?, ?, ?, ?)'),-- insRestricted: db.prepare('INSERT INTO fts_restricted (rowid, title, summary, body) VALUES (?, ?, ?, ?)'),-- };---- /** (Re)index one document according to its exposure. Call inside the store's transaction. */-- function put(rid, doc, exposure) {-- remove(rid);-- if (exposure === EXPOSURE.public_listed) st.insPublic.run(rid, doc.title || '', doc.summary || '', doc.body || '');-- else if (exposure === EXPOSURE.restricted) st.insRestricted.run(rid, doc.title || '', doc.summary || '', doc.body || '');-- }---- function remove(rid) {-- st.delPublic.run(rid);-- st.delRestricted.run(rid);-- }---- // ── SQL building ─────────────────────────────────────────---- /**-- * The ACL predicate for restricted rows (alias d). private: the viewer's subject is in-- * acl.subjects. members/unlisted: subject, group or entitlement match. Returns null when the-- * viewer has no subject (then the restricted branch is not queried at all).-- */-- function aclClause(viewer, params) {-- if (!viewer || !viewer.subject) return null;-- params.v_subject = viewer.subject;-- params.v_groups = JSON.stringify(viewer.groups || []);-- params.v_ents = JSON.stringify(viewer.entitlements || []);-- return `(-- (d.visibility = 'private' AND EXISTS (SELECT 1 FROM doc_acl a WHERE a.rid = d.rid AND a.kind = 's' AND a.value = @v_subject))-- OR (d.visibility IN ('members', 'unlisted') AND EXISTS (SELECT 1 FROM doc_acl a WHERE a.rid = d.rid AND (-- (a.kind = 's' AND a.value = @v_subject)-- OR (a.kind = 'g' AND a.value IN (SELECT value FROM json_each(@v_groups)))-- OR (a.kind = 'e' AND a.value IN (SELECT value FROM json_each(@v_ents))))))-- )`;-- }---- function filterClause(filters, params) {-- const parts = [];-- if (filters.owner) { parts.push('d.owner = @f_owner'); params.f_owner = filters.owner; }-- if (filters.type) { parts.push('d.type = @f_type'); params.f_type = filters.type; }-- if (filters.language) { parts.push('d.language = @f_lang'); params.f_lang = filters.language; }-- (filters.facets || []).forEach(([key, values], i) => {-- params[`f_fk${i}`] = key;-- params[`f_fv${i}`] = JSON.stringify(values);-- parts.push(`EXISTS (SELECT 1 FROM doc_facets x WHERE x.rid = d.rid AND x.key = @f_fk${i} AND x.value IN (SELECT value FROM json_each(@f_fv${i})))`);-- });-- return parts.length ? ` AND ${parts.join(' AND ')}` : '';-- }---- /**-- * The visible, matching set as (rid, rank, snip) rows. match = FTS5 expression. rank is bm25-- * times the freshness boost at reference time `now` (ms), so pages of one query (which carry-- * their first page's `now` in the cursor) rank against the same clock.-- */-- function matchedSet({ match, filters, viewer, snippets = true, now = Date.now() }, params) {-- params.match = match;-- params.r_now = now;-- params.r_half = freshness.halfLifeDays;-- params.r_weight = freshness.weight;-- const where = filterClause(filters, params);-- const snip = (t) => (snippets ? `snippet(${t}, -1, char(1), char(2), '…', 16)` : 'NULL');-- const rank = (t) => `bm25(${t}, ${WEIGHTS}) * ov_freshness(COALESCE(d.published_at, d.updated_at), @r_now, @r_half, @r_weight)`;-- const branches = [`SELECT d.rid AS rid, ${rank('fts_public')} AS rank, ${snip('fts_public')} AS snip-- FROM fts_public JOIN documents d ON d.rid = fts_public.rowid-- WHERE fts_public MATCH @match AND d.exposure = ${EXPOSURE.public_listed} AND d.deleted = 0${where}`];-- const acl = aclClause(viewer, params);-- if (acl) {-- branches.push(`SELECT d.rid AS rid, ${rank('fts_restricted')} AS rank, ${snip('fts_restricted')} AS snip-- FROM fts_restricted JOIN documents d ON d.rid = fts_restricted.rowid-- WHERE fts_restricted MATCH @match AND d.exposure = ${EXPOSURE.restricted} AND d.deleted = 0${where} AND ${acl}`);-- }-- return branches.join('\nUNION ALL\n');-- }---- /** The visible set without a text query (browse). */-- function visibleSet({ filters, viewer }, params) {-- const where = filterClause(filters, params);-- const acl = aclClause(viewer, params);-- const vis = acl-- ? `(d.exposure = ${EXPOSURE.public_listed} OR (d.exposure = ${EXPOSURE.restricted} AND ${acl}))`-- : `d.exposure = ${EXPOSURE.public_listed}`;-- return `SELECT d.rid AS rid, d.sort_at AS sort_at FROM documents d WHERE d.deleted = 0 AND ${vis}${where}`;-- }---- // ── Queries ──────────────────────────────────────────────---- /** Ranked full-text search. after = { rank, rid } keyset cursor. */-- function search({ match, filters = {}, viewer, limit, after, now = Date.now() }) {-- const params = { limit: limit + 1 };-- let sql = `SELECT rid, rank, snip FROM (${matchedSet({ match, filters, viewer, now }, params)})`;-- if (after) {-- sql += ' WHERE (rank > @c_rank OR (rank = @c_rank AND rid > @c_rid))';-- params.c_rank = after.rank;-- params.c_rid = after.rid;-- }-- sql += ' ORDER BY rank, rid LIMIT @limit';-- const rows = db.prepare(sql).all(params);-- const more = rows.length > limit;-- const hits = rows.slice(0, limit);-- const last = hits[hits.length - 1];-- return { hits, next: more && last ? { rank: last.rank, rid: last.rid } : null };-- }---- /** Newest first, no text query. after = { sort_at, rid }. */-- function browse({ filters = {}, viewer, limit, after }) {-- const params = { limit: limit + 1 };-- let sql = `SELECT rid, sort_at FROM (${visibleSet({ filters, viewer }, params)})`;-- if (after) {-- sql += ' WHERE (sort_at < @c_sort OR (sort_at = @c_sort AND rid < @c_rid))';-- params.c_sort = after.sort_at;-- params.c_rid = after.rid;-- }-- sql += ' ORDER BY sort_at DESC, rid DESC LIMIT @limit';-- const rows = db.prepare(sql).all(params);-- const more = rows.length > limit;-- const hits = rows.slice(0, limit).map(r => ({ rid: r.rid, rank: null, snip: null, sort_at: r.sort_at }));-- const last = hits[hits.length - 1];-- return { hits, next: more && last ? { sort_at: last.sort_at, rid: last.rid } : null };-- }---- /**-- * Value counts for facet keys over exactly the visible matching set (same SQL as the hits),-- * so a count can never include a document the viewer cannot see.-- */-- function facetCounts({ match, filters = {}, viewer, keys, perKey = 20 }) {-- const out = {};-- for (const key of keys) {-- const params = { fkey: key, per: perKey };-- const set = match ? matchedSet({ match, filters, viewer, snippets: false }, params) : visibleSet({ filters, viewer }, params);-- const rows = db.prepare(`SELECT f.value AS value, COUNT(*) AS count FROM doc_facets f-- WHERE f.key = @fkey AND f.rid IN (SELECT rid FROM (${set}))-- GROUP BY f.value ORDER BY count DESC, value LIMIT @per`).all(params);-- out[key] = rows;-- }-- return out;-- }---- /** Title prefix suggestions over the visible set. */-- function suggest({ match, filters = {}, viewer, limit }) {-- const params = { limit };-- const sql = `SELECT rid, rank FROM (${matchedSet({ match, filters, viewer, snippets: false }, params)}) ORDER BY rank, rid LIMIT @limit`;-- return db.prepare(sql).all(params);-- }---- function counts() {-- return {-- public_listed: db.prepare('SELECT COUNT(*) AS n FROM fts_public').get().n,-- restricted: db.prepare('SELECT COUNT(*) AS n FROM fts_restricted').get().n,-- };-- }---- return { name: 'sqlite-fts5', put, remove, search, browse, facetCounts, suggest, counts, freshness };--}----// ── Query text → FTS5 expression ─────────────────────────────----/**-- * Turns user text into a safe FTS5 expression: word tokens only, each quoted (no operators,-- * column filters or syntax errors reach FTS5), all required. A trailing `*` on the last word-- * makes it a prefix. Returns null when nothing searchable is left.-- */--function toMatch(q, { maxTerms = 16, column = null, prefixLast = false } = {}) {-- const text = String(q || '').normalize('NFKC');-- const words = (text.toLowerCase().match(/[\p{L}\p{N}_]+/gu) || []).slice(0, maxTerms);-- if (!words.length) return null;-- const wantsPrefix = prefixLast || /[\p{L}\p{N}_]\*\s*$/u.test(text);-- const parts = words.map((w, i) => `"${w}"${wantsPrefix && i === words.length - 1 ? '*' : ''}`);-- const expr = parts.join(' ');-- return column ? `{${column}} : (${expr})` : expr;--}----/** Plain text with the engine's snippet marks → HTML with <mark>, everything else escaped. */--function snippetHtml(snip) {-- if (!snip) return null;-- const esc = String(snip).replace(/[&<>"']/g, c => ({ '&': '&', '<': '<', '>': '>', '"': '"', "'": ''' }[c]));-- return esc.split(MARK_OPEN).join('<mark>').split(MARK_CLOSE).join('</mark>');--}----module.exports = { createEngine, toMatch, snippetHtml, freshnessBoost };nnew file mode 100644@@@ -0,0 +1,293 @@++'use strict';++/**++ * Search engine: PostgreSQL full-text search in the service's own database (ADR-035; it replaced SQLite FTS5, the++ * engine docs/adr-engine.md chose, with the same interface and the same guarantees).++ *++ * One table, search_fts, split by audience so an anonymous query never touches, ranks against or counts anything but++ * public listed documents:++ *++ * audience 3 documents with exposure public_listed (public, published, indexable)++ * audience 1 documents with exposure restricted (unlisted/members/private, published)++ *++ * Drafts, unpublished, deleted and public-noindex documents have no row, so no query can reach them. Every query is++ * also filtered by `documents.exposure` and, for restricted rows, by the viewer's ACL keys (doc_acl) in the same SQL:++ * visibility is decided before anything is ranked, counted, snippeted or suggested.++ *++ * Text is folded before it is indexed or queried (NFKD, combining marks removed: FTS5's remove_diacritics), split into++ * runs of letters and digits (FTS5's unicode61) and tokenized by the `simple` configuration (lower case, no stemming, no stop words: FTS5's unicode61). Title, summary and++ * body weigh 10, 4 and 1 (tsvector weights A, B, C). Every function here is async; put and remove run inside the store's++ * transaction (ambient). Nothing outside this file writes SQL against search_fts.++ */++const { EXPOSURE } = require('../document');++++const CONFIG = 'simple';++const WEIGHTS = '{0.1, 0.1, 0.4, 1.0}'; // D, C body, B summary, A title: FTS5's bm25 weights 10 / 4 / 1++const MARK_OPEN = '\u0001';++const MARK_CLOSE = '\u0002';++const HEADLINE = `StartSel=${MARK_OPEN}, StopSel=${MARK_CLOSE}, MaxWords=16, MinWords=8, ShortWord=0, MaxFragments=1, FragmentDelimiter=" … "`;++++const DAY_MS = 24 * 3600 * 1000;++++/**++ * Freshness multiplier for a document dated `iso` at reference time `nowMs`:++ *++ * 1 + weight × 2^(−age_days / halfLifeDays)++ *++ * With the defaults (weight 1, half-life 30 days) a document published today counts ×2.0, one++ * 30 days old ×1.5, 60 days ×1.25, 90 days ×1.125, a year old ×1.0002: relevance still decides,++ * and between comparably relevant documents the newer one ranks first. A future date counts as++ * today; no date (or an unparsable one) counts ×1. Weight 0 turns the boost off. The SQL below++ * computes the same from search_fts.fresh_at.++ */++function freshnessBoost(iso, nowMs, halfLifeDays, weight) {++ if (!weight || !iso) return 1;++ const t = Date.parse(iso);++ if (!Number.isFinite(t)) return 1;++ const ageDays = Math.max(0, (nowMs - t) / DAY_MS);++ return 1 + weight * Math.pow(2, -ageDays / halfLifeDays);++}++++/** Diacritics folded, as FTS5's remove_diacritics did: the index and every query see the same text. */++function fold(s) {++ return String(s || '').normalize('NFKD').replace(/\p{M}+/gu, '').normalize('NFC');++}++++/**++ * The words FTS5's unicode61 tokenizer saw: runs of letters and digits, everything else a separator. PostgreSQL's++ * parser would keep "OpenVibe.Live", "a_b" or "3.5" as single tokens (host, word, number); separating them first makes++ * "openvibe" find "OpenVibe.Live" as it did.++ */++function words(s) {++ return fold(s).replace(/[^\p{L}\p{N}]+/gu, ' ');++}++++function createEngine(db, { freshness = { weight: 1, halfLifeDays: 30 } } = {}) {++ const st = {++ del: db.prepare('DELETE FROM search_fts WHERE rid = ?'),++ ins: db.prepare(`INSERT INTO search_fts (rid, audience, fresh_at, title, summary, body, tsv)++ VALUES (@rid, @audience, @fresh_at, @title, @summary, @body,++ setweight(to_tsvector('${CONFIG}', @f_title), 'A') || setweight(to_tsvector('${CONFIG}', @f_summary), 'B')++ || setweight(to_tsvector('${CONFIG}', @f_body), 'C'))`),++ };++++ /** (Re)index one document according to its exposure. Call inside the store's transaction. */++ async function put(rid, doc, exposure) {++ await remove(rid);++ if (exposure !== EXPOSURE.public_listed && exposure !== EXPOSURE.restricted) return;++ const at = Date.parse(doc.published_at || doc.updated_at || '');++ const title = doc.title || '';++ const summary = doc.summary || '';++ const body = doc.body || '';++ await st.ins.run({++ rid, audience: exposure, fresh_at: Number.isFinite(at) ? at : null, title, summary, body,++ f_title: words(title), f_summary: words(summary), f_body: words(body),++ });++ }++++ async function remove(rid) {++ await st.del.run(rid);++ }++++ // ── SQL building ─────────────────────────────────────────++++ /**++ * The ACL predicate for restricted rows (alias d). private: the viewer's subject is in++ * acl.subjects. members/unlisted: subject, group or entitlement match. Returns null when the++ * viewer has no subject (then the restricted branch is not queried at all).++ */++ function aclClause(viewer, params) {++ if (!viewer || !viewer.subject) return null;++ params.v_subject = viewer.subject;++ params.v_groups = (viewer.groups || []).map(String);++ params.v_ents = (viewer.entitlements || []).map(String);++ return `(++ (d.visibility = 'private' AND EXISTS (SELECT 1 FROM doc_acl a WHERE a.rid = d.rid AND a.kind = 's' AND a.value = @v_subject))++ OR (d.visibility IN ('members', 'unlisted') AND EXISTS (SELECT 1 FROM doc_acl a WHERE a.rid = d.rid AND (++ (a.kind = 's' AND a.value = @v_subject)++ OR (a.kind = 'g' AND a.value = ANY(@v_groups::text[]))++ OR (a.kind = 'e' AND a.value = ANY(@v_ents::text[])))))++ )`;++ }++++ function filterClause(filters, params) {++ const parts = [];++ if (filters.owner) { parts.push('d.owner = @f_owner'); params.f_owner = filters.owner; }++ if (filters.type) { parts.push('d.type = @f_type'); params.f_type = filters.type; }++ if (filters.language) { parts.push('d.language = @f_lang'); params.f_lang = filters.language; }++ (filters.facets || []).forEach(([key, values], i) => {++ params[`f_fk${i}`] = key;++ params[`f_fv${i}`] = values.map(String);++ parts.push(`EXISTS (SELECT 1 FROM doc_facets x WHERE x.rid = d.rid AND x.key = @f_fk${i} AND x.value = ANY(@f_fv${i}::text[]))`);++ });++ return parts.length ? ` AND ${parts.join(' AND ')}` : '';++ }++++ /**++ * The visible, matching set as (rid, rank) rows. match = toMatch()'s query. rank is minus ts_rank times the++ * freshness boost at reference time `now` (ms), so lower ranks first (as FTS5's bm25 did) and pages of one query++ * (which carry their first page's `now` in the cursor) rank against the same clock.++ */++ function matchedSet({ match, filters, viewer, now = Date.now() }, params) {++ params.match = tsquery(match);++ params.r_now = now;++ params.r_half = freshness.halfLifeDays;++ params.r_weight = freshness.weight;++ const where = filterClause(filters, params);++ const rank = `-(ts_rank('${WEIGHTS}'::float4[], f.tsv, to_tsquery('${CONFIG}', @match), 1)++ * CASE WHEN f.fresh_at IS NULL OR @r_weight::float8 = 0 THEN 1++ ELSE 1 + @r_weight::float8 * power(2::float8, -(GREATEST(0, @r_now::bigint - f.fresh_at)::float8 / ${DAY_MS}) / @r_half::float8) END)`;++ const branches = [`SELECT d.rid AS rid, ${rank} AS rank++ FROM search_fts f JOIN documents d ON d.rid = f.rid++ WHERE f.audience = ${EXPOSURE.public_listed} AND f.tsv @@ to_tsquery('${CONFIG}', @match)++ AND d.exposure = ${EXPOSURE.public_listed} AND d.deleted = 0${where}`];++ const acl = aclClause(viewer, params);++ if (acl) {++ branches.push(`SELECT d.rid AS rid, ${rank} AS rank++ FROM search_fts f JOIN documents d ON d.rid = f.rid++ WHERE f.audience = ${EXPOSURE.restricted} AND f.tsv @@ to_tsquery('${CONFIG}', @match)++ AND d.exposure = ${EXPOSURE.restricted} AND d.deleted = 0${where} AND ${acl}`);++ }++ return branches.join('\nUNION ALL\n');++ }++++ /** The visible set without a text query (browse). */++ function visibleSet({ filters, viewer }, params) {++ const where = filterClause(filters, params);++ const acl = aclClause(viewer, params);++ const vis = acl++ ? `(d.exposure = ${EXPOSURE.public_listed} OR (d.exposure = ${EXPOSURE.restricted} AND ${acl}))`++ : `d.exposure = ${EXPOSURE.public_listed}`;++ return `SELECT d.rid AS rid, d.sort_at AS sort_at FROM documents d WHERE d.deleted = 0 AND ${vis}${where}`;++ }++++ // ── Queries ──────────────────────────────────────────────++++ /**++ * Ranked full-text search. after = { rank, rid } keyset cursor. The snippet is computed for the page's rows only:++ * the summary and body when they match, else the title.++ */++ async function search({ match, filters = {}, viewer, limit, after, now = Date.now() }) {++ const params = { limit: limit + 1 };++ let sql = `SELECT rid, rank FROM (${matchedSet({ match, filters, viewer, now }, params)}) m`;++ if (after) {++ sql += ' WHERE (rank > @c_rank::float8 OR (rank = @c_rank::float8 AND rid > @c_rid::bigint))';++ params.c_rank = after.rank;++ params.c_rid = after.rid;++ }++ sql += ' ORDER BY rank, rid LIMIT @limit';++ const page = `SELECT p.rid, p.rank,++ CASE WHEN to_tsvector('${CONFIG}', f.summary || ' ' || f.body) @@ to_tsquery('${CONFIG}', @match)++ THEN ts_headline('${CONFIG}', f.summary || ' ' || f.body, to_tsquery('${CONFIG}', @match), @hl)++ ELSE ts_headline('${CONFIG}', f.title, to_tsquery('${CONFIG}', @match), @hl) END AS snip++ FROM (${sql}) p JOIN search_fts f ON f.rid = p.rid ORDER BY p.rank, p.rid`;++ params.hl = HEADLINE;++ const rows = await db.prepare(page).all(params);++ const more = rows.length > limit;++ const hits = rows.slice(0, limit);++ const last = hits[hits.length - 1];++ return { hits, next: more && last ? { rank: last.rank, rid: last.rid } : null };++ }++++ /** Newest first, no text query. after = { sort_at, rid }. */++ async function browse({ filters = {}, viewer, limit, after }) {++ const params = { limit: limit + 1 };++ let sql = `SELECT rid, sort_at FROM (${visibleSet({ filters, viewer }, params)}) v`;++ if (after) {++ sql += ' WHERE (sort_at < @c_sort OR (sort_at = @c_sort AND rid < @c_rid::bigint))';++ params.c_sort = after.sort_at;++ params.c_rid = after.rid;++ }++ sql += ' ORDER BY sort_at DESC, rid DESC LIMIT @limit';++ const rows = await db.prepare(sql).all(params);++ const more = rows.length > limit;++ const hits = rows.slice(0, limit).map(r => ({ rid: r.rid, rank: null, snip: null, sort_at: r.sort_at }));++ const last = hits[hits.length - 1];++ return { hits, next: more && last ? { sort_at: last.sort_at, rid: last.rid } : null };++ }++++ /**++ * Value counts for facet keys over exactly the visible matching set (same SQL as the hits),++ * so a count can never include a document the viewer cannot see.++ */++ async function facetCounts({ match, filters = {}, viewer, keys, perKey = 20 }) {++ const out = {};++ for (const key of keys) {++ const params = { fkey: key, per: perKey };++ const set = match ? matchedSet({ match, filters, viewer }, params) : visibleSet({ filters, viewer }, params);++ out[key] = await db.prepare(`SELECT f.value AS value, COUNT(*)::int AS count FROM doc_facets f++ WHERE f.key = @fkey AND f.rid IN (SELECT rid FROM (${set}) s)++ GROUP BY f.value ORDER BY count DESC, value LIMIT @per`).all(params);++ }++ return out;++ }++++ /** Title prefix suggestions over the visible set. */++ async function suggest({ match, filters = {}, viewer, limit }) {++ const params = { limit };++ return await db.prepare(`SELECT rid, rank FROM (${matchedSet({ match, filters, viewer }, params)}) m ORDER BY rank, rid LIMIT @limit`).all(params);++ }++++ async function counts() {++ const rows = await db.prepare('SELECT audience, COUNT(*)::int AS n FROM search_fts GROUP BY audience').all();++ const of = (a) => (rows.find((r) => Number(r.audience) === a) || { n: 0 }).n;++ return { public_listed: of(EXPOSURE.public_listed), restricted: of(EXPOSURE.restricted) };++ }++++ /**++ * Rebuild the index from documents.doc when its counts disagree with the documents table (after the one-time import++ * from SQLite, whose FTS5 tables are not copied). Returns what it did.++ */++ async function reconcile() {++ const docs = await db.prepare(`SELECT++ COUNT(*) FILTER (WHERE exposure = ${EXPOSURE.public_listed} AND deleted = 0)::int AS public_listed,++ COUNT(*) FILTER (WHERE exposure = ${EXPOSURE.restricted} AND deleted = 0)::int AS restricted++ FROM documents`).get();++ const idx = await counts();++ if (idx.public_listed === docs.public_listed && idx.restricted === docs.restricted) return { rebuilt: 0 };++ let rebuilt = 0;++ await db.tx(async () => {++ await db.prepare('DELETE FROM search_fts').run();++ const rows = await db.prepare(`SELECT rid, exposure, doc FROM documents WHERE deleted = 0 AND exposure IN (${EXPOSURE.public_listed}, ${EXPOSURE.restricted}) ORDER BY rid`).all();++ for (const r of rows) {++ await put(r.rid, typeof r.doc === 'string' ? JSON.parse(r.doc) : r.doc, Number(r.exposure));++ rebuilt++;++ }++ });++ return { rebuilt };++ }++++ return { name: 'postgresql-fts', put, remove, search, browse, facetCounts, suggest, counts, reconcile, freshness };++}++++// ── Query text → tsquery ─────────────────────────────────────++++/**++ * Turns user text into a safe query: word tokens only (no operators or syntax reach PostgreSQL), all required. A++ * trailing `*` on the last word makes it a prefix; column 'title' restricts every word to the title. Returns++ * { words, prefix, column } or null when nothing searchable is left.++ */++function toMatch(q, { maxTerms = 16, column = null, prefixLast = false } = {}) {++ const text = String(q || '').normalize('NFKC');++ const words = (fold(text).toLowerCase().match(/[\p{L}\p{N}]+/gu) || []).slice(0, maxTerms);++ if (!words.length) return null;++ const prefix = prefixLast || /[\p{L}\p{N}_]\*\s*$/u.test(text);++ return { words, prefix, column };++}++++/** toMatch()'s query → to_tsquery text: 'w1' & 'w2' & 'w3':* (and :A on each word for the title). */++function tsquery(m) {++ const weight = m.column === 'title' ? 'A' : '';++ return m.words.map((w, i) => {++ const star = m.prefix && i === m.words.length - 1 ? '*' : '';++ return `'${w}'${star || weight ? `:${star}${weight}` : ''}`;++ }).join(' & ');++}++++/** Plain text with the engine's snippet marks → HTML with <mark>, everything else escaped. */++function snippetHtml(snip) {++ if (!snip) return null;++ const esc = String(snip).replace(/[&<>"']/g, c => ({ '&': '&', '<': '<', '>': '>', '"': '"', "'": ''' }[c]));++ return esc.split(MARK_OPEN).join('<mark>').split(MARK_CLOSE).join('</mark>');++}++++module.exports = { createEngine, toMatch, tsquery, snippetHtml, freshnessBoost, fold, words };@@@ -36,8 +36,8 @@ function createOutbox(db, { source, now = () => Date.now() }) {const insert = db.prepare('INSERT INTO event_outbox (event_id, event_type, envelope, created_at) VALUES (?, ?, ?, ?)');/** { event_type, subject, payload, visibility?, priority?, actor?, trace_id? } → envelope */-- function enqueue({ event_type, subject, payload, visibility = 'internal', priority = 'important', actor, trace_id }) {-- if (!db.inTransaction) throw new Error('outbox.enqueue() must run inside the transaction that makes the change');++ async function enqueue({ event_type, subject, payload, visibility = 'internal', priority = 'important', actor, trace_id }) {++ if (!db.inTransaction()) throw new Error('outbox.enqueue() must run inside the transaction that makes the change');const ms = now();const env = {event_id: ids.newId('event', ms),@@@ -54,7 +54,7 @@ function createOutbox(db, { source, now = () => Date.now() }) {if (trace_id && /^[0-9a-f]{32}$/.test(trace_id)) env.trace_id = trace_id;const v = validate('events.event-envelope@1', env);if (!v.valid) throw new Error(`outbox: invalid envelope for ${event_type}: ${v.errors.map(e => `${e.path} ${e.message}`).join('; ')}`);-- insert.run(env.event_id, event_type, JSON.stringify(env), ms);++ await insert.run(env.event_id, event_type, JSON.stringify(env), ms);return env;}@@@ -71,11 +71,11 @@ function createOutbox(db, { source, now = () => Date.now() }) {return {enqueue,-- pending: () => q.pending.get().n,-- rejected: () => q.rejectedCount.get().n,++ pending: async () => (await q.pending.get()).n,++ rejected: async () => (await q.rejectedCount.get()).n,/** Every envelope in order (tests and operators). */-- all: () => q.list.all().map(r => JSON.parse(r.envelope)),-- prune: (olderThanMs = 7 * 24 * 3600 * 1000) => q.prune.run(now() - olderThanMs).changes,++ all: async () => (await q.list.all()).map(r => JSON.parse(r.envelope)),++ prune: async (olderThanMs = 7 * 24 * 3600 * 1000) => (await q.prune.run(now() - olderThanMs)).changes,_q: q,};}@@@ -109,18 +109,18 @@ function createRelay({ outbox, eventsUrl, intervalMs = 2000, tokenClient, tokenOreturn { ok: res.ok, status: res.status, text };}-- function mark(rows, fn) { db.transaction(() => { for (const r of rows) fn(r); })(); }++ async function mark(rows, fn) { await db.tx(() => { for (const r of rows) fn(r); }); }async function publishRows(rows) {let r;try {r = await post(rows);} catch (err) {-- mark(rows, row => q.failed.run(now() + BACKOFF_MS[Math.min(row.attempts, BACKOFF_MS.length - 1)], String(err.message).slice(0, 500), row.seq));++ await mark(rows, async row => await q.failed.run(now() + BACKOFF_MS[Math.min(row.attempts, BACKOFF_MS.length - 1)], String(err.message).slice(0, 500), row.seq));return { sent: 0, failed: rows.length, rejected: 0 };}if (r.ok) {-- mark(rows, row => q.sent.run(now(), row.seq));++ await mark(rows, async row => await q.sent.run(now(), row.seq));return { sent: rows.length, failed: 0, rejected: 0 };}if (permanent(r.status) && rows.length > 1) {@@@ -134,11 +134,11 @@ function createRelay({ outbox, eventsUrl, intervalMs = 2000, tokenClient, tokenO}const msg = `${r.status} ${r.text.slice(0, 300)}`;if (permanent(r.status)) {-- mark(rows, row => q.rejected.run(now(), msg, row.seq));++ await mark(rows, async row => await q.rejected.run(now(), msg, row.seq));log.warn(`[outbox] event refused by Events: ${msg}`);return { sent: 0, failed: 0, rejected: rows.length };}-- mark(rows, row => q.failed.run(now() + BACKOFF_MS[Math.min(row.attempts, BACKOFF_MS.length - 1)], msg, row.seq));++ await mark(rows, async row => await q.failed.run(now() + BACKOFF_MS[Math.min(row.attempts, BACKOFF_MS.length - 1)], msg, row.seq));return { sent: 0, failed: rows.length, rejected: 0 };}@@@ -146,7 +146,7 @@ function createRelay({ outbox, eventsUrl, intervalMs = 2000, tokenClient, tokenOconst total = { sent: 0, failed: 0, rejected: 0 };if (!eventsUrl || !tokens) return total;for (;;) {-- const rows = q.due.all(now(), 50);++ const rows = await q.due.all(now(), 50);if (!rows.length) break;const s = await publishRows(rows);total.sent += s.sent; total.failed += s.failed; total.rejected += s.rejected;@@@ -9,7 +9,7 @@*/const { load } = require('./config');const { openDb } = require('./db');--const { createEngine } = require('./engine/fts5');++const { createEngine } = require('./engine/pg');const { createStore } = require('./store');const { createOutbox, createRelay } = require('./events/outbox');const { createKeyStore, createAuth } = require('./auth');@@@ -17,10 +17,14 @@ const { createPurgeQueue, createPurger } = require('./purge');const { createSavedSearches } = require('./saved');const { createApp } = require('./app');--async function start({ config, now = () => Date.now(), fetchImpl = globalThis.fetch, tokenClient, log = console, listen = true } = {}) {++async function start({ config, db: givenDb = null, now = () => Date.now(), fetchImpl = globalThis.fetch, tokenClient, log = console, listen = true } = {}) {config = config || load();-- const db = openDb(config.dbPath);++ // PostgreSQL (ADR-035): opened and migrated here unless the caller (a test) hands in a migrated handle.++ const db = givenDb || await openDb(config, { log });const engine = createEngine(db, { freshness: config.freshness });++ // The index is derived from documents: rebuilt when it disagrees (after the one-time import from SQLite).++ const rebuilt = await engine.reconcile();++ if (rebuilt.rebuilt) log.log(`[search] full-text index rebuilt: ${rebuilt.rebuilt} document(s)`);const outbox = createOutbox(db, { source: config.serviceId, now });const purges = createPurgeQueue(db, { config: config.purge, now });const purger = createPurger({ db, config: config.purge, fetchImpl, log, now });@@@ -38,7 +42,7 @@ async function start({ config, now = () => Date.now(), fetchImpl = globalThis.feconst keyLoaded = keys.start().catch(() => null);relay.start();purger.start();-- const pruneTimer = setInterval(() => { try { outbox.prune(); } catch (err) { log.error(`[outbox] prune: ${err.message}`); } }, 6 * 3600 * 1000);++ const pruneTimer = setInterval(async () => { try { await outbox.prune(); } catch (err) { log.error(`[outbox] prune: ${err.message}`); } }, 6 * 3600 * 1000);pruneTimer.unref?.();let server = null;@@@ -59,7 +63,7 @@ async function start({ config, now = () => Date.now(), fetchImpl = globalThis.feserver.closeAllConnections?.();await new Promise(resolve => server.close(() => resolve()));}-- db.close();++ if (!givenDb) await db.close();}return { config, db, engine, store, outbox, relay, purges, purger, saved, keys, keyLoaded, auth, app, server, close };@@@ -19,63 +19,63 @@ const { EXPOSURE } = require('./document');const EXPOSURE_NAMES = Object.fromEntries(Object.entries(EXPOSURE).map(([name, n]) => [n, name]));--function readers(db) {-- const byExposure = db.prepare('SELECT exposure, COUNT(*) AS n FROM documents GROUP BY exposure');-- const ftsPublic = db.prepare('SELECT COUNT(*) AS n FROM fts_public');-- const ftsRestricted = db.prepare('SELECT COUNT(*) AS n FROM fts_restricted');++function readers(db, engine) {++ const byExposure = db.prepare('SELECT exposure, COUNT(*)::int AS n FROM documents GROUP BY exposure');return {/** { none, restricted, public_unlisted, public_listed }, every exposure present (0 when empty). */-- documents() {++ async documents() {const out = Object.fromEntries(Object.keys(EXPOSURE).map((k) => [k, 0]));-- for (const r of byExposure.all()) out[EXPOSURE_NAMES[r.exposure] || 'none'] += r.n;++ for (const r of await byExposure.all()) out[EXPOSURE_NAMES[r.exposure] || 'none'] += r.n;return out;},-- indexed: () => ({ public: ftsPublic.get().n, restricted: ftsRestricted.get().n }),++ indexed: async () => { const c = await engine.counts(); return { public: c.public_listed, restricted: c.restricted }; },};}function createSearchReadiness({ db, keys, config, engine, store, outbox, relay, purges = null, purger = null, saved = null, release = null }) {-- const read = readers(db);++ const read = readers(db, engine);return createReadiness({service: 'search',release,checks: [{name: 'db', required: true,-- check: () => {-- db.prepare('SELECT rid FROM documents LIMIT 1').all();-- db.prepare('SELECT rowid FROM fts_public LIMIT 1').all();-- db.prepare('SELECT rowid FROM fts_restricted LIMIT 1').all();-- return true;++ // A real round trip that names the store (postgresql / pglite), and the tables present.++ check: async () => {++ const r = await db.ready();++ if (!r.ok) return r.error;++ await db.prepare('SELECT rid FROM documents LIMIT 1').all();++ await db.prepare('SELECT rid FROM search_fts LIMIT 1').all();++ return { ok: true, detail: r.detail };},},{ name: 'network_jwks', required: false, check: () => keys.loaded() || 'Network signing key not loaded yet: owner writes and signed-in queries cannot be verified' },{name: 'index', required: false, cacheMs: 10_000,-- check: () => {-- const docs = read.documents();-- const fts = read.indexed();++ check: async () => {++ const docs = await read.documents();++ const fts = await read.indexed();const detail = { indexed: fts, expected: { public: docs.public_listed, restricted: docs.restricted } };const off = [];-- if (fts.public !== docs.public_listed) off.push(`fts_public has ${fts.public} entries for ${docs.public_listed} public_listed documents`);-- if (fts.restricted !== docs.restricted) off.push(`fts_restricted has ${fts.restricted} entries for ${docs.restricted} restricted documents`);++ if (fts.public !== docs.public_listed) off.push(`the public index has ${fts.public} entries for ${docs.public_listed} public_listed documents`);++ if (fts.restricted !== docs.restricted) off.push(`the restricted index has ${fts.restricted} entries for ${docs.restricted} restricted documents`);return off.length ? { ok: false, error: `index out of step: ${off.join('; ')}`, detail } : { ok: true, detail };},},],-- details: (body) => {++ details: async (body) => {const dbOk = body.checks.db.status === 'ok';return {engine: engine.name,-- documents: dbOk ? store.counts() : null,-- outbox: dbOk ? { pending: outbox.pending(), rejected: outbox.rejected(), relay: config.events.url ? (relay.running() ? 'running' : 'stopped') : 'off (EVENTS_URL unset)' } : null,++ documents: dbOk ? await store.counts() : null,++ outbox: dbOk ? { pending: await outbox.pending(), rejected: await outbox.rejected(), relay: config.events.url ? (relay.running() ? 'running' : 'stopped') : 'off (EVENTS_URL unset)' } : null,webhook: config.events.webhookSecrets.length ? 'on' : 'off (SEARCH_EVENTS_SECRET unset)',purge: dbOk && purges ? {cdn: purges.cdnOn() ? (purger && purger.running() ? 'cloudflare' : 'cloudflare (stopped)') : 'off (CLOUDFLARE_PURGE_TOKEN unset)',zones: config.purge.zones.length,-- ...purges.cdnCounts(),++ ...await purges.cdnCounts(),} : null,-- saved_searches: dbOk && saved ? saved.total() : null,++ saved_searches: dbOk && saved ? await saved.total() : null,freshness: config.freshness,};},@@@ -83,22 +83,22 @@ function createSearchReadiness({ db, keys, config, engine, store, outbox, relay,}/** Search gauges on the openvibe-shared/metrics registry. */--function registerSearchGauges(registry, { db, outbox, purges = null }) {-- const read = readers(db);++function registerSearchGauges(registry, { db, engine, outbox, purges = null }) {++ const read = readers(db, engine);registry.gauge({name: 'search_documents', help: 'Documents held, by exposure (public_listed and restricted are searchable; tombstones and drafts are none)', labelNames: ['exposure'],-- collect: () => Object.entries(read.documents()).map(([exposure, value]) => ({ labels: { exposure }, value })),++ collect: async () => Object.entries(await read.documents()).map(([exposure, value]) => ({ labels: { exposure }, value })),});registry.gauge({name: 'search_documents_indexed', help: 'Documents in the full-text index, by table', labelNames: ['index'],-- collect: () => Object.entries(read.indexed()).map(([index, value]) => ({ labels: { index }, value })),++ collect: async () => Object.entries(await read.indexed()).map(([index, value]) => ({ labels: { index }, value })),});-- registry.gauge({ name: 'search_outbox_pending', help: 'Events waiting in the outbox', collect: () => outbox.pending() });-- registry.gauge({ name: 'search_outbox_rejected', help: 'Events OpenVibe.Events refused for good', collect: () => outbox.rejected() });++ registry.gauge({ name: 'search_outbox_pending', help: 'Events waiting in the outbox', collect: async () => await outbox.pending() });++ registry.gauge({ name: 'search_outbox_rejected', help: 'Events OpenVibe.Events refused for good', collect: async () => await outbox.rejected() });if (purges) {registry.gauge({name: 'search_cdn_purges', help: 'Cloudflare cache purges of removed documents, by state', labelNames: ['state'],-- collect: () => Object.entries(purges.cdnCounts()).map(([state, value]) => ({ labels: { state }, value })),++ collect: async () => Object.entries(await purges.cdnCounts()).map(([state, value]) => ({ labels: { state }, value })),});}}@@@ -88,7 +88,7 @@ function createPurgeQueue(db, { config, now = () => Date.now() }) {const cdnOn = Boolean(config.cloudflareToken);const st = {insRemoval: db.prepare(`INSERT INTO removals (event_id, owner, type, id, revision, reason, previous_exposure, exposure, canonical_url, at)-- VALUES (@event_id, @owner, @type, @id, @revision, @reason, @previous_exposure, @exposure, @canonical_url, @at)`),++ VALUES (@event_id, @owner, @type, @id, @revision, @reason, @previous_exposure, @exposure, @canonical_url, @at) RETURNING seq`),insPurge: db.prepare(`INSERT INTO cdn_purges (removal_seq, zone_id, url, state, detail, created_at, done_at)VALUES (@removal_seq, @zone_id, @url, @state, @detail, @created_at, @done_at)ON CONFLICT (zone_id, url) WHERE state = 'pending' DO NOTHING`),@@@ -101,17 +101,17 @@ function createPurgeQueue(db, { config, now = () => Date.now() }) {* Record one removal (the payload of the search.document.removed event). Must run inside the* transaction that made the change, like the outbox.*/-- function record(eventId, p) {-- if (!db.inTransaction) throw new Error('purge.record() must run inside the transaction that makes the change');++ async function record(eventId, p) {++ if (!db.inTransaction()) throw new Error('purge.record() must run inside the transaction that makes the change');const at = now();-- const removalSeq = Number(st.insRemoval.run({++ const removalSeq = Number((await st.insRemoval.run({event_id: eventId, owner: p.owner, type: p.type, id: p.id, revision: p.revision, reason: p.reason,previous_exposure: p.previous_exposure, exposure: p.exposure, canonical_url: p.canonical_url || null, at,-- }).lastInsertRowid);++ })).lastInsertRowid);if (!cdnOn || !p.canonical_url) return removalSeq;for (const url of purgeUrls(p.canonical_url, config.relatedPaths)) {const zoneId = zoneFor(url, config.zones);-- st.insPurge.run({++ await st.insPurge.run({removal_seq: removalSeq, zone_id: zoneId, url,state: zoneId ? 'pending' : 'skipped',detail: zoneId ? null : 'no Cloudflare zone configured for this host (CLOUDFLARE_ZONE_IDS)',@@@ -122,8 +122,8 @@ function createPurgeQueue(db, { config, now = () => Date.now() }) {}/** The owner's removal feed, oldest first after a cursor. */-- function ownerFeed(owner, { after = 0, limit = 200 } = {}) {-- const rows = st.ownerFeed.all(owner, after, limit);++ async function ownerFeed(owner, { after = 0, limit = 200 } = {}) {++ const rows = await st.ownerFeed.all(owner, after, limit);return {removals: rows.map(r => ({seq: r.seq, event_id: r.event_id, type: r.type, id: r.id, revision: r.revision, reason: r.reason,@@@ -134,13 +134,13 @@ function createPurgeQueue(db, { config, now = () => Date.now() }) {};}-- function cdnCounts() {++ async function cdnCounts() {const out = { pending: 0, purged: 0, failed: 0, skipped: 0 };-- for (const r of st.cdnCounts.all()) out[r.state] = r.n;++ for (const r of await st.cdnCounts.all()) out[r.state] = r.n;return out;}-- return { record, ownerFeed, cdnCounts, purgesFor: (seq) => st.purgesFor.all(seq), cdnOn: () => cdnOn };++ return { record, ownerFeed, cdnCounts, purgesFor: async (seq) => await st.purgesFor.all(seq), cdnOn: () => cdnOn };}/**@@@ -161,12 +161,12 @@ function createPurger({ db, config, fetchImpl = globalThis.fetch, log = console,const clip = (s) => (token ? String(s || '').split(token).join('[token]') : String(s || '')).slice(0, DETAIL_MAX);-- function mark(rows, fn) { db.transaction(() => { for (const r of rows) fn(r); })(); }++ async function mark(rows, fn) { await db.tx(() => { for (const r of rows) fn(r); }); }-- function retryOrFail(rows, detail) {-- mark(rows, (r) => {-- if (r.attempts + 1 >= MAX_ATTEMPTS) q.failed.run(clip(`gave up after ${MAX_ATTEMPTS} attempts: ${detail}`), now(), r.seq);-- else q.retry.run(now() + BACKOFF_MS[Math.min(r.attempts, BACKOFF_MS.length - 1)], clip(detail), r.seq);++ async function retryOrFail(rows, detail) {++ await mark(rows, async (r) => {++ if (r.attempts + 1 >= MAX_ATTEMPTS) await q.failed.run(clip(`gave up after ${MAX_ATTEMPTS} attempts: ${detail}`), now(), r.seq);++ else await q.retry.run(now() + BACKOFF_MS[Math.min(r.attempts, BACKOFF_MS.length - 1)], clip(detail), r.seq);});}@@@ -188,17 +188,17 @@ function createPurger({ db, config, fetchImpl = globalThis.fetch, log = console,try {r = await call(zoneId, rows.map(x => x.url));} catch (err) {-- retryOrFail(rows, `network: ${err.message}`);++ await retryOrFail(rows, `network: ${err.message}`);return { purged: 0, retrying: rows.length, failed: 0 };}const errors = (r.body && Array.isArray(r.body.errors) ? r.body.errors : []).map(e => `${e.code}: ${e.message}`).join('; ');if (r.status >= 200 && r.status < 300 && r.body && r.body.success === true) {const id = r.body.result && r.body.result.id ? `cloudflare purge ${r.body.result.id}` : 'purged';-- mark(rows, x => q.purged.run(clip(id), now(), x.seq));++ await mark(rows, async x => await q.purged.run(clip(id), now(), x.seq));return { purged: rows.length, retrying: 0, failed: 0 };}if (r.status === 429 || r.status >= 500 || r.status === 408) {-- retryOrFail(rows, `HTTP ${r.status} ${errors}`);++ await retryOrFail(rows, `HTTP ${r.status} ${errors}`);return { purged: 0, retrying: rows.length, failed: 0 };}// Refused (bad URL, token without purge permission, unknown zone): isolate, then fail.@@@ -210,7 +210,7 @@ function createPurger({ db, config, fetchImpl = globalThis.fetch, log = console,}return total;}-- mark(rows, x => q.failed.run(clip(`HTTP ${r.status} ${errors || 'refused'}`), now(), x.seq));++ await mark(rows, async x => await q.failed.run(clip(`HTTP ${r.status} ${errors || 'refused'}`), now(), x.seq));log.warn(`[purge] Cloudflare refused a purge: HTTP ${r.status} ${clip(errors)}`);return { purged: 0, retrying: 0, failed: rows.length };}@@@ -218,7 +218,7 @@ function createPurger({ db, config, fetchImpl = globalThis.fetch, log = console,async function doFlush() {const total = { purged: 0, retrying: 0, failed: 0 };if (!token) return total;-- const rows = q.due.all(now(), 200);++ const rows = await q.due.all(now(), 200);const byZone = new Map();for (const r of rows) {if (!byZone.has(r.zone_id)) byZone.set(r.zone_id, []);@@@ -78,35 +78,35 @@ function createSavedSearches(db, { maxPerSubject = 50, now = () => Date.now() }* Save (subject, q, filters) with a name. The same query again returns the existing row* (renamed when a new name is given): { created, saved } | { error: 'limit' }.*/-- function save(subject, { name, q, filters }) {-- return db.transaction(() => {++ async function save(subject, { name, q, filters }) {++ return await db.tx(async () => {const f = canonicalFilters(filters);const hash = hashOf(q, f);const at = now();-- const existing = st.byHash.get(subject, hash);++ const existing = await st.byHash.get(subject, hash);if (existing) {-- if (name && name !== existing.name) st.rename.run(name, at, existing.id);-- return { created: false, saved: view(st.get.get(existing.id, subject)) };++ if (name && name !== existing.name) await st.rename.run(name, at, existing.id);++ return { created: false, saved: view(await st.get.get(existing.id, subject)) };}-- if (st.count.get(subject).n >= maxPerSubject) return { error: 'limit' };++ if ((await st.count.get(subject)).n >= maxPerSubject) return { error: 'limit' };const id = `svs_${ids.ulid(at).toLowerCase()}`;-- st.insert.run({ id, subject, name: name || q || 'All documents', q, filters: JSON.stringify(f), query_hash: hash, at });-- return { created: true, saved: view(st.get.get(id, subject)) };-- })();++ await st.insert.run({ id, subject, name: name || q || 'All documents', q, filters: JSON.stringify(f), query_hash: hash, at });++ return { created: true, saved: view(await st.get.get(id, subject)) };++ });}return {-- list: (subject) => st.list.all(subject).map(view),++ list: async (subject) => (await st.list.all(subject)).map(view),/** One saved search of this subject; another subject's id is the same as a missing one. */-- get: (subject, id) => {++ get: async (subject, id) => {if (!ID_RE.test(String(id))) return null;-- const r = st.get.get(id, subject);++ const r = await st.get.get(id, subject);return r ? view(r) : null;},save,-- remove: (subject, id) => ID_RE.test(String(id)) && st.del.run(id, subject).changes > 0,-- markRun: (id) => st.ran.run(now(), id),-- total: () => st.total.get().n,++ remove: async (subject, id) => ID_RE.test(String(id)) && (await st.del.run(id, subject)).changes > 0,++ markRun: async (id) => await st.ran.run(now(), id),++ total: async () => (await st.total.get()).n,maxPerSubject,};}@@@ -22,18 +22,18 @@ function createStore({ db, engine, outbox, purges = null, now = () => Date.now()insert: db.prepare(`INSERT INTO documents (owner, type, id, revision, deleted, visibility, publication_state, exposure,index_decision, noindex_reasons, canonical_url, title, language, published_at, updated_at, sort_at, doc, hash, via, event_id, indexed_at)VALUES (@owner, @type, @id, @revision, @deleted, @visibility, @publication_state, @exposure,-- @index_decision, @noindex_reasons, @canonical_url, @title, @language, @published_at, @updated_at, @sort_at, @doc, @hash, @via, @event_id, @indexed_at)`),++ @index_decision, @noindex_reasons, @canonical_url, @title, @language, @published_at, @updated_at, @sort_at, @doc, @hash, @via, @event_id, @indexed_at) RETURNING rid`),update: db.prepare(`UPDATE documents SET revision = @revision, deleted = @deleted, visibility = @visibility,publication_state = @publication_state, exposure = @exposure, index_decision = @index_decision,noindex_reasons = @noindex_reasons, canonical_url = @canonical_url, title = @title, language = @language,published_at = @published_at, updated_at = @updated_at, sort_at = @sort_at, doc = @doc, hash = @hash,via = @via, event_id = @event_id, indexed_at = @indexed_at WHERE rid = @rid`),delAcl: db.prepare('DELETE FROM doc_acl WHERE rid = ?'),-- insAcl: db.prepare('INSERT OR IGNORE INTO doc_acl (rid, kind, value) VALUES (?, ?, ?)'),++ insAcl: db.prepare('INSERT INTO doc_acl (rid, kind, value) VALUES (?, ?, ?) ON CONFLICT DO NOTHING'),delFacets: db.prepare('DELETE FROM doc_facets WHERE rid = ?'),-- insFacet: db.prepare('INSERT OR IGNORE INTO doc_facets (rid, key, value) VALUES (?, ?, ?)'),++ insFacet: db.prepare('INSERT INTO doc_facets (rid, key, value) VALUES (?, ?, ?) ON CONFLICT DO NOTHING'),ownerPage: db.prepare(`SELECT rid, type, id, revision, deleted, hash, exposure, indexed_at FROM documents-- WHERE owner = @owner AND (@type IS NULL OR type = @type) AND rid > @after ORDER BY rid LIMIT @limit`),++ WHERE owner = @owner AND (@type::text IS NULL OR type = @type) AND rid > @after ORDER BY rid LIMIT @limit`),counts: db.prepare(`SELECT COUNT(*) AS total, SUM(deleted) AS tombstones,SUM(CASE WHEN exposure = 3 THEN 1 ELSE 0 END) AS public_listed,SUM(CASE WHEN exposure = 2 THEN 1 ELSE 0 END) AS public_unlisted,@@@ -64,16 +64,16 @@ function createStore({ db, engine, outbox, purges = null, now = () => Date.now()};}-- function writeSideRows(rid, doc, exposure) {-- st.delAcl.run(rid);-- st.delFacets.run(rid);++ async function writeSideRows(rid, doc, exposure) {++ await st.delAcl.run(rid);++ await st.delFacets.run(rid);if (exposure === EXPOSURE.none) return;if (exposure === EXPOSURE.restricted) {-- for (const s of doc.acl.subjects) st.insAcl.run(rid, 's', s);-- for (const g of doc.acl.groups) st.insAcl.run(rid, 'g', g);-- for (const e of doc.acl.entitlements) st.insAcl.run(rid, 'e', e);++ for (const s of doc.acl.subjects) await st.insAcl.run(rid, 's', s);++ for (const g of doc.acl.groups) await st.insAcl.run(rid, 'g', g);++ for (const e of doc.acl.entitlements) await st.insAcl.run(rid, 'e', e);}-- for (const [k, v] of facetPairs(doc.facets)) st.insFacet.run(rid, k, v);++ for (const [k, v] of facetPairs(doc.facets)) await st.insFacet.run(rid, k, v);}function removalReason(prev, doc) {@@@ -84,7 +84,7 @@ function createStore({ db, engine, outbox, purges = null, now = () => Date.now()return 'noindex';}-- function announce(prevRow, prevDoc, doc, exposure, traceId) {++ async function announce(prevRow, prevDoc, doc, exposure, traceId) {const prevExposure = prevRow ? prevRow.exposure : EXPOSURE.none;const subject = { type: 'document', id: `${doc.owner}/${doc.type}/${doc.id}`, revision: doc.revision };const wasPublic = prevExposure >= EXPOSURE.public_unlisted;@@@ -97,12 +97,12 @@ function createStore({ db, engine, outbox, purges = null, now = () => Date.now()// Only a URL that was public already: caches and sitemaps purge it.canonical_url: wasPublic ? prevRow.canonical_url : null,};-- const env = outbox.enqueue({ event_type: 'search.document.removed', subject, trace_id: traceId, payload });++ const env = await outbox.enqueue({ event_type: 'search.document.removed', subject, trace_id: traceId, payload });// Search's own consumer: the owners' removal feed and the CDN purge queue (purge.js).-- if (purges) purges.record(env.event_id, payload);++ if (purges) await purges.record(env.event_id, payload);}if (!doc.deleted && exposure > EXPOSURE.none) {-- outbox.enqueue({++ await outbox.enqueue({event_type: 'search.document.indexed',subject,trace_id: traceId,@@@ -122,10 +122,10 @@ function createStore({ db, engine, outbox, purges = null, now = () => Date.now()* Must be called inside a transaction when combined with other writes (inbox); it opens its* own otherwise.*/-- function apply(input, { via = 'api', eventId = null, traceId = null } = {}) {-- const run = () => {++ async function apply(input, { via = 'api', eventId = null, traceId = null } = {}) {++ const run = async () => {const { doc, hash } = normalize(input);-- const cur = st.get.get(doc.owner, doc.type, doc.id);++ const cur = await st.get.get(doc.owner, doc.type, doc.id);if (cur) {const base = { revision: doc.revision, stored_revision: cur.revision, deleted: Boolean(cur.deleted) };if (doc.revision < cur.revision) return { outcome: 'stale', ...base };@@@ -139,45 +139,45 @@ function createStore({ db, engine, outbox, purges = null, now = () => Date.now()let rid;if (cur) {rid = cur.rid;-- st.update.run({ ...values, rid });++ await st.update.run({ ...values, rid });} else {-- rid = Number(st.insert.run(values).lastInsertRowid);++ rid = Number((await st.insert.run(values)).lastInsertRowid);}-- writeSideRows(rid, doc, values.exposure);-- engine.put(rid, doc, values.exposure);++ await writeSideRows(rid, doc, values.exposure);++ await engine.put(rid, doc, values.exposure);const prevDoc = cur ? JSON.parse(cur.doc) : null;-- announce(cur, prevDoc, doc, values.exposure, traceId);++ await announce(cur, prevDoc, doc, values.exposure, traceId);return { outcome: 'applied', revision: doc.revision, stored_revision: cur ? cur.revision : null, deleted: doc.deleted };};-- return db.inTransaction ? run() : db.transaction(run)();++ return db.inTransaction() ? await run() : await db.tx(run);}/*** Tombstone a document. revision omitted = the stored revision (a deletion wins the tie), or* 0 for a document Search has never seen (so a late older upsert is still refused).*/-- function remove(owner, type, id, { revision = null, via = 'api', eventId = null, traceId = null } = {}) {-- const run = () => {-- const cur = st.get.get(owner, type, id);++ async function remove(owner, type, id, { revision = null, via = 'api', eventId = null, traceId = null } = {}) {++ const run = async () => {++ const cur = await st.get.get(owner, type, id);const rev = revision != null ? revision : (cur ? cur.revision : 0);-- return apply({ owner, type, id, revision: rev, deleted: true }, { via, eventId, traceId });++ return await apply({ owner, type, id, revision: rev, deleted: true }, { via, eventId, traceId });};-- return db.inTransaction ? run() : db.transaction(run)();++ return db.inTransaction() ? await run() : await db.tx(run);}-- function get(owner, type, id) {-- const row = st.get.get(owner, type, id);++ async function get(owner, type, id) {++ const row = await st.get.get(owner, type, id);return row ? { row, doc: JSON.parse(row.doc) } : null;}-- function byRid(rid) {-- const row = st.byRid.get(rid);++ async function byRid(rid) {++ const row = await st.byRid.get(rid);return row ? { row, doc: JSON.parse(row.doc) } : null;}/** Reconciliation page for an owner: (type, id, revision, deleted, hash) by insertion order. */-- function ownerPage(owner, { type = null, after = 0, limit = 500 } = {}) {-- const rows = st.ownerPage.all({ owner, type, after, limit: limit + 1 });++ async function ownerPage(owner, { type = null, after = 0, limit = 500 } = {}) {++ const rows = await st.ownerPage.all({ owner, type, after, limit: limit + 1 });const more = rows.length > limit;const page = rows.slice(0, limit);return {@@@ -189,8 +189,8 @@ function createStore({ db, engine, outbox, purges = null, now = () => Date.now()};}-- function counts() {-- const c = st.counts.get();++ async function counts() {++ const c = await st.counts.get();return {total: c.total || 0, tombstones: c.tombstones || 0, public_listed: c.public_listed || 0,public_unlisted: c.public_unlisted || 0, restricted: c.restricted || 0,@@@ -175,7 +175,7 @@ function pageRouter({ searcher, auth, baseUrl = 'https://search.openvibe.network.send('User-agent: *\nAllow: /$\nAllow: /updates\nDisallow: /?\nDisallow: /api/\nDisallow: /internal/\n');});-- router.get('/', (req, res, next) => {++ router.get('/', async (req, res, next) => {const wantsHtml = /\btext\/html\b/.test(String(req.get('accept') || ''));res.setHeader('Cache-Control', 'no-store');res.setHeader('Vary', 'Accept, Cookie');@@@ -202,7 +202,7 @@ function pageRouter({ searcher, auth, baseUrl = 'https://search.openvibe.networklet status = 200;if (searching) {try {-- const out = searcher.run({ text: q, query: { ...(owner ? { owner } : {}), ...(type ? { type } : {}) }, viewer: viewer || ANONYMOUS, cursor });++ const out = await searcher.run({ text: q, query: { ...(owner ? { owner } : {}), ...(type ? { type } : {}) }, viewer: viewer || ANONYMOUS, cursor });body = out.results.length? resultsHtml(out.results): `<p class="empty">${cursor ? 'No more results.' : 'Nothing found. The index is young: services add documents as they publish them.'}</p>`;@@@ -215,7 +215,7 @@ t('a visibility change or deletion leaves results at once and announces search.dawait request(svc.base, 'DELETE', '/api/v1/documents/wiki/page/flip?revision=3', { token: WIKI });assert.deepStrictEqual(idsOf(await q('q=iotaword', asUser(alice))), []);-- const removed = svc.outbox.all().filter(e => e.event_type === 'search.document.removed' && e.payload.id === 'flip');++ const removed = (await svc.outbox.all()).filter(e => e.event_type === 'search.document.removed' && e.payload.id === 'flip');assert.strictEqual(removed.length, 2);assert.deepStrictEqual(removed.map(e => [e.payload.reason, e.payload.previous_exposure, e.payload.exposure]), [['visibility_changed', 'public_listed', 'restricted'],@@@ -233,7 +233,7 @@ t('publishing → draft and public → noindex also announce removal', async ()const d2 = doc({ id: 'noidx', title: 'Lambdaword', revision: 1 });await put(d2);await put({ ...d2, revision: 2, indexability: { decision: 'noindex', reasons: ['duplicate_without_canonical'] } });-- const reasons = svc.outbox.all().filter(e => e.event_type === 'search.document.removed' && ['unpub', 'noidx'].includes(e.payload.id)).map(e => `${e.payload.id}:${e.payload.reason}`);++ const reasons = (await svc.outbox.all()).filter(e => e.event_type === 'search.document.removed' && ['unpub', 'noidx'].includes(e.payload.id)).map(e => `${e.payload.id}:${e.payload.reason}`);assert.deepStrictEqual(reasons, ['unpub:not_published', 'noidx:noindex']);assert.deepStrictEqual(idsOf(await q('q=kappaword')), []);assert.deepStrictEqual(idsOf(await q('q=lambdaword')), []);@@@ -60,7 +60,7 @@ t('indexing: 120 a minute per service principal, refused before the store is touconst d = doc({ id: 'pg_over' });const r = await call('PUT', `/api/v1/documents/wiki/page/${d.id}`, { token: WIKI, body: d });assert.deepStrictEqual([r.status, r.body.code, r.headers.get('retry-after')], [429, 'rate_limited', '60']);-- assert.strictEqual(svc.store.get('wiki', 'page', 'pg_over'), null, 'nothing stored');++ assert.strictEqual(await svc.store.get('wiki', 'page', 'pg_over'), null, 'nothing stored');const b = doc({ owner: 'blog', type: 'post', id: 'post_1', canonical_url: 'https://openvibe.blog/p/1' });assert.strictEqual((await call('PUT', '/api/v1/documents/blog/post/post_1', { token: BLOG, body: b })).status, 200, 'another service still indexes');});nnew file mode 100644@@@ -0,0 +1,11 @@++'use strict';++/**++ * A migrated database for one test boot (ADR-035), from openvibe-sdk/testing: PGlite by default; with++ * SEARCH_TEST_STORE=pg (npm run test:pg) the PostgreSQL + PgBouncer containers, with roles and a schema of its own.++ */++const { createTestDb, pgAvailable } = require('openvibe-sdk/testing');++const { MIGRATIONS } = require('../server/db');++++const testDb = ({ store = process.env.SEARCH_TEST_STORE || 'pglite', max = 4 } = {}) => createTestDb({ migrations: MIGRATIONS, store, service: 'search', max });++++module.exports = { testDb, pgAvailable };@@@ -6,7 +6,7 @@*/const assert = require('assert');const { boot, request, doc, suite } = require('./helpers');--const { freshnessBoost } = require('../server/engine/fts5');++const { freshnessBoost } = require('../server/engine/pg');const t = suite('freshness');const NOW = Date.parse('2026-09-23T12:00:00Z');@@@ -31,9 +31,9 @@ async function corpus(env = {}) {const svc = await boot({ env, now: () => clock });// Same text, same length: bm25 ties, so only freshness separates them. The old one is indexed// first (lower rid), so without freshness it would come first.-- svc.store.apply(doc({ id: 'old', title: 'Aurora report', body: 'aurora seen tonight', published_at: daysAgo(400), updated_at: daysAgo(400) }));-- svc.store.apply(doc({ id: 'mid', title: 'Aurora report', body: 'aurora seen tonight', published_at: daysAgo(20), updated_at: daysAgo(20) }));-- svc.store.apply(doc({ id: 'new', title: 'Aurora report', body: 'aurora seen tonight', published_at: daysAgo(1), updated_at: daysAgo(1) }));++ await svc.store.apply(doc({ id: 'old', title: 'Aurora report', body: 'aurora seen tonight', published_at: daysAgo(400), updated_at: daysAgo(400) }));++ await svc.store.apply(doc({ id: 'mid', title: 'Aurora report', body: 'aurora seen tonight', published_at: daysAgo(20), updated_at: daysAgo(20) }));++ await svc.store.apply(doc({ id: 'new', title: 'Aurora report', body: 'aurora seen tonight', published_at: daysAgo(1), updated_at: daysAgo(1) }));return { svc, setClock: (c) => { clock = c; } };}@@@ -58,8 +58,8 @@ t('relevance still wins: an old title match outranks a fresh passing mention', alet clock = NOW;const svc = await boot({ now: () => clock });try {-- svc.store.apply(doc({ id: 'fresh-mention', title: 'Weekly notes', summary: 'Many topics this week.', body: `${'filler words about other things '.repeat(40)} glacier`, published_at: daysAgo(0) }));-- svc.store.apply(doc({ id: 'old-title', title: 'Glacier glacier retreat', summary: 'The glacier retreat, measured.', body: 'glacier data', published_at: daysAgo(700) }));++ await svc.store.apply(doc({ id: 'fresh-mention', title: 'Weekly notes', summary: 'Many topics this week.', body: `${'filler words about other things '.repeat(40)} glacier`, published_at: daysAgo(0) }));++ await svc.store.apply(doc({ id: 'old-title', title: 'Glacier glacier retreat', summary: 'The glacier retreat, measured.', body: 'glacier data', published_at: daysAgo(700) }));const r = await request(svc.base, 'GET', '/api/v1/search?q=glacier');assert.deepStrictEqual(r.body.results.map(x => x.id), ['old-title', 'fresh-mention']);} finally { await svc.stop(); }@@@ -56,9 +56,11 @@ async function boot({ env = {}, fetchImpl, tokenClient, now } = {}) {SEARCH_EVENTS_SECRET: WEBHOOK_SECRET,...env,});-- const h = await start({ config, log: silent, fetchImpl, tokenClient, now });++ // One database per boot (PGlite, or SEARCH_TEST_STORE=pg: the containers), dropped when the boot stops.++ const testdb = await require('./db').testDb();++ const h = await start({ config, db: testdb.db, log: silent, fetchImpl, tokenClient, now });const base = `http://127.0.0.1:${h.server.address().port}`;-- return { ...h, base, dir, async stop() { await h.close(); } };++ return { ...h, base, dir, async stop() { await h.close(); await testdb.close(); } };}async function request(base, method, p, { token, body, headers = {}, raw } = {}) {@@@ -122,7 +124,7 @@ async function deliver(base, event, { secret = WEBHOOK_SECRET, seq = 1, badSignaconst key = badSignature ? 'wrong-secret-' + 'y'.repeat(32) : secret;const ts = Math.floor(now / 1000);const v2 = v1Only ? {} : { 'X-OpenVibe-Timestamp': String(ts), 'X-OpenVibe-Signature-V2': signV2(raw, key, ts) };-- return request(base, 'POST', '/internal/events', {++ return await request(base, 'POST', '/internal/events', {raw,headers: { 'Content-Type': 'application/json', 'X-OpenVibe-Signature': sign(raw, key), ...v2, 'X-OpenVibe-Event-Id': event.event_id },});@@@ -39,7 +39,7 @@ t('/api/ready: every check reports; db is the only required one; the index agreeassert.ok(Date.parse(c.checked_at));}assert.deepStrictEqual(r.body.checks.index.detail, { indexed: { public: 2, restricted: 1 }, expected: { public: 2, restricted: 1 } });-- assert.strictEqual(r.body.engine, 'sqlite-fts5');++ assert.strictEqual(r.body.engine, 'postgresql-fts');assert.strictEqual(r.body.documents.total, 4);});@@@ -84,12 +84,12 @@ t('a full-text index out of step with the documents degrades /api/ready (still 2svc = await boot(); // fresh: the index check has no cached resultawait put(doc());await put(doc());-- svc.db.prepare('DELETE FROM fts_public WHERE rowid = (SELECT MIN(rowid) FROM fts_public)').run();++ await svc.db.prepare('DELETE FROM search_fts WHERE rid = (SELECT MIN(rid) FROM search_fts WHERE audience = 3)').run();const r = await request(svc.base, 'GET', '/api/ready');assert.strictEqual(r.status, 200, r.text);assert.strictEqual(r.body.status, 'degraded');assert.deepStrictEqual(r.body.degraded, ['index']);-- assert.match(r.body.checks.index.error, /fts_public has 1 entries for 2 public_listed documents/);++ assert.match(r.body.checks.index.error, /the public index has 1 entries for 2 public_listed documents/);});t('a broken database makes the service unready (503); /metrics still answers', async () => {@@@ -36,12 +36,12 @@ t('outbox envelopes are valid events.event-envelope@1 and are enqueued only withawait request(svc.base, 'PUT', `/api/v1/documents/wiki/page/${d.id}`, { token: WIKI, body: d });await request(svc.base, 'PUT', `/api/v1/documents/wiki/page/${d.id}`, { token: WIKI, body: { ...d, revision: 0 } }); // stale: no eventawait request(svc.base, 'DELETE', `/api/v1/documents/wiki/page/${d.id}?revision=2`, { token: WIKI });-- const all = svc.outbox.all();++ const all = await svc.outbox.all();assert.deepStrictEqual(all.map(e => e.event_type), ['search.document.indexed', 'search.document.removed']);for (const e of all) assert.ok(validate('events.event-envelope@1', e).valid, JSON.stringify(validate('events.event-envelope@1', e).errors));assert.strictEqual(all[1].subject.id, `wiki/page/${d.id}`);assert.strictEqual(all[1].subject.revision, 2);-- assert.throws(() => svc.outbox.enqueue({ event_type: 'search.document.indexed', subject: { type: 'document', id: 'x' } }), /inside the transaction/);++ await assert.rejects(async () => await svc.outbox.enqueue({ event_type: 'search.document.indexed', subject: { type: 'document', id: 'x' } }), /inside the transaction/);} finally {await svc.stop();}@@@ -55,8 +55,8 @@ t('the relay publishes pending events with the service token and marks them sentawait request(svc.base, 'PUT', `/api/v1/documents/wiki/page/${d.id}`, { token: WIKI, body: d });await request(svc.base, 'DELETE', `/api/v1/documents/wiki/page/${d.id}?revision=2`, { token: WIKI });// a write kicks the relay; a flush already in flight may predate the second row-- for (let i = 0; i < 5 && svc.outbox.pending(); i++) await svc.relay.flush();-- assert.strictEqual(svc.outbox.pending(), 0);++ for (let i = 0; i < 5 && await svc.outbox.pending(); i++) await svc.relay.flush();++ assert.strictEqual(await svc.outbox.pending(), 0);const published = events.calls.flatMap(c => (c.body.events ? c.body.events : [c.body]));assert.deepStrictEqual(published.map(e => e.event_type).sort(), ['search.document.indexed', 'search.document.removed']);assert.ok(events.calls.every(c => c.headers.authorization === 'Bearer relay-token'));@@@ -79,13 +79,13 @@ t('an unreachable or failing Events keeps rows pending for retry; a refused rowawait request(svc.base, 'PUT', '/api/v1/documents/wiki/page/fine', { token: WIKI, body: doc({ id: 'fine' }) });await svc.relay.flush();await svc.relay.flush();-- assert.strictEqual(svc.outbox.pending(), 2, 'kept for retry');++ assert.strictEqual(await svc.outbox.pending(), 2, 'kept for retry');mode = 'up';-- svc.db.prepare('UPDATE event_outbox SET next_attempt_at = 0').run();++ await svc.db.prepare('UPDATE event_outbox SET next_attempt_at = 0').run();await svc.relay.flush();await svc.relay.flush();-- assert.strictEqual(svc.outbox.pending(), 0);-- assert.strictEqual(svc.outbox.rejected(), 1, 'the poison row is set aside');++ assert.strictEqual(await svc.outbox.pending(), 0);++ assert.strictEqual(await svc.outbox.rejected(), 1, 'the poison row is set aside');} finally {await svc.stop();await events.close();@@@ -97,7 +97,7 @@ t('readiness reports the outbox and the webhook state', async () => {try {const r = await request(svc.base, 'GET', '/api/ready');assert.strictEqual(r.status, 200);-- assert.strictEqual(r.body.engine, 'sqlite-fts5');++ assert.strictEqual(r.body.engine, 'postgresql-fts');assert.match(r.body.outbox.relay, /^off/);assert.strictEqual(r.body.webhook, 'on');} finally {@@@ -15,10 +15,10 @@ const alice = ids.newId('user');t('boot', async () => {svc = await boot();-- svc.store.apply(doc({ id: 'pub', title: 'Kestrel <script>alert(1)</script> nesting', summary: 'Kestrel "boxes" & more', body: 'kestrel kestrel', canonical_url: 'https://openvibe.wiki/p/kestrel?a=1&b=2' }));-- svc.store.apply(doc({ id: 'prv', visibility: 'private', acl: { subjects: [alice] }, title: 'Kestrel private diary', body: 'kestrel' }));-- svc.store.apply(doc({ id: 'drf', visibility: 'draft', title: 'Kestrel draft', body: 'kestrel' }));-- for (let i = 0; i < 25; i++) svc.store.apply(doc({ id: `many${i}`, title: `Heron sighting ${i}`, body: 'heron' }));++ await svc.store.apply(doc({ id: 'pub', title: 'Kestrel <script>alert(1)</script> nesting', summary: 'Kestrel "boxes" & more', body: 'kestrel kestrel', canonical_url: 'https://openvibe.wiki/p/kestrel?a=1&b=2' }));++ await svc.store.apply(doc({ id: 'prv', visibility: 'private', acl: { subjects: [alice] }, title: 'Kestrel private diary', body: 'kestrel' }));++ await svc.store.apply(doc({ id: 'drf', visibility: 'draft', title: 'Kestrel draft', body: 'kestrel' }));++ for (let i = 0; i < 25; i++) await svc.store.apply(doc({ id: `many${i}`, title: `Heron sighting ${i}`, body: 'heron' }));});t('the front page is a search form, indexable, no-store, with a strict CSP', async () => {@@@ -55,9 +55,9 @@ t('inert without CLOUDFLARE_PURGE_TOKEN: removals are recorded for owners, nothiconst svc = await boot({ env: { CLOUDFLARE_ZONE_IDS: ZONES }, fetchImpl: cf.fetchImpl });try {const d = doc({ id: 'inert1', canonical_url: 'https://openvibe.wiki/p/inert1' });-- svc.store.apply(d);-- svc.store.apply(hide(d));-- assert.deepStrictEqual(svc.purges.cdnCounts(), { pending: 0, purged: 0, failed: 0, skipped: 0 });++ await svc.store.apply(d);++ await svc.store.apply(hide(d));++ assert.deepStrictEqual(await svc.purges.cdnCounts(), { pending: 0, purged: 0, failed: 0, skipped: 0 });assert.deepStrictEqual(await svc.purger.flush(), { purged: 0, retrying: 0, failed: 0 });assert.strictEqual(svc.purger.running(), false);assert.strictEqual(cf.calls.length, 0);@@@ -71,7 +71,7 @@ t('inert without CLOUDFLARE_PURGE_TOKEN: removals are recorded for owners, nothiassert.strictEqual(r.previous_exposure, 'public_listed');assert.strictEqual(r.exposure, 'restricted');// The removal row and the outbox event are the same removal.-- const ev = svc.outbox.all().find(e => e.event_type === 'search.document.removed' && e.payload.id === 'inert1');++ const ev = (await svc.outbox.all()).find(e => e.event_type === 'search.document.removed' && e.payload.id === 'inert1');assert.strictEqual(r.event_id, ev.event_id);const ready = await request(svc.base, 'GET', '/api/ready');assert.match(JSON.stringify(ready.body), /CLOUDFLARE_PURGE_TOKEN unset/);@@@ -82,11 +82,11 @@ t('owner removal feed: owner token only, own owner only, cursor paging, restrictconst svc = await boot();try {const members = doc({ id: 'mem1', visibility: 'members', acl: { groups: ['g1'] } });-- svc.store.apply(members);-- svc.store.remove('wiki', 'page', 'mem1');++ await svc.store.apply(members);++ await svc.store.remove('wiki', 'page', 'mem1');const pub = doc({ id: 'pub2', canonical_url: 'https://openvibe.wiki/p/pub2' });-- svc.store.apply(pub);-- svc.store.remove('wiki', 'page', 'pub2');++ await svc.store.apply(pub);++ await svc.store.remove('wiki', 'page', 'pub2');assert.strictEqual((await request(svc.base, 'GET', '/api/v1/owners/wiki/removals')).status, 401);assert.strictEqual((await request(svc.base, 'GET', '/api/v1/owners/wiki/removals', { token: BLOG })).status, 403);const p1 = await request(svc.base, 'GET', '/api/v1/owners/wiki/removals?limit=1', { token: WIKI });@@@ -105,12 +105,12 @@ t('with a token: the formerly public URL and its sitemap are purged in one callconst a = doc({ id: 'cf1', canonical_url: 'https://openvibe.wiki/p/cf1' });const b = doc({ id: 'cf2', canonical_url: 'https://openvibe.wiki/p/cf2' });const c = doc({ owner: 'wiki', id: 'cf3', canonical_url: 'https://docs.openvibe.network/cf3' });-- for (const d of [a, b, c]) svc.store.apply(d);-- svc.store.apply(hide(a));-- svc.store.remove('wiki', 'page', 'cf2');-- svc.store.apply({ ...c, revision: 2, publication_state: 'retracted' });++ for (const d of [a, b, c]) await svc.store.apply(d);++ await svc.store.apply(hide(a));++ await svc.store.remove('wiki', 'page', 'cf2');++ await svc.store.apply({ ...c, revision: 2, publication_state: 'retracted' });// Two removals on openvibe.wiki share one pending sitemap purge.-- assert.deepStrictEqual(svc.purges.cdnCounts(), { pending: 5, purged: 0, failed: 0, skipped: 0 });++ assert.deepStrictEqual(await svc.purges.cdnCounts(), { pending: 5, purged: 0, failed: 0, skipped: 0 });const s = await svc.purger.flush();assert.deepStrictEqual(s, { purged: 5, retrying: 0, failed: 0 });@@@ -126,7 +126,7 @@ t('with a token: the formerly public URL and its sitemap are purged in one call// Done rows are never sent again; the token is never stored.assert.deepStrictEqual(await svc.purger.flush(), { purged: 0, retrying: 0, failed: 0 });assert.strictEqual(cf.calls.length, 2);-- const dump = JSON.stringify(svc.db.prepare('SELECT * FROM cdn_purges').all()) + JSON.stringify(svc.db.prepare('SELECT * FROM removals').all());++ const dump = JSON.stringify(await svc.db.prepare('SELECT * FROM cdn_purges').all()) + JSON.stringify(await svc.db.prepare('SELECT * FROM removals').all());assert.ok(!dump.includes(TOKEN));assert.strictEqual(svc.purger.running(), true);} finally { await svc.stop(); }@@@ -137,12 +137,12 @@ t('an index change that does not lower exposure, or a URL that was never public,const svc = await boot({ env: { CLOUDFLARE_PURGE_TOKEN: TOKEN, CLOUDFLARE_ZONE_IDS: ZONES }, fetchImpl: cf.fetchImpl });try {const d = doc({ id: 'keep', canonical_url: 'https://openvibe.wiki/p/keep' });-- svc.store.apply(d);-- svc.store.apply({ ...d, revision: 2, title: 'New title' });++ await svc.store.apply(d);++ await svc.store.apply({ ...d, revision: 2, title: 'New title' });const m = doc({ id: 'mem', visibility: 'members', acl: { groups: ['g'] }, canonical_url: 'https://openvibe.wiki/p/mem' });-- svc.store.apply(m);-- svc.store.remove('wiki', 'page', 'mem');-- assert.deepStrictEqual(svc.purges.cdnCounts(), { pending: 0, purged: 0, failed: 0, skipped: 0 });++ await svc.store.apply(m);++ await svc.store.remove('wiki', 'page', 'mem');++ assert.deepStrictEqual(await svc.purges.cdnCounts(), { pending: 0, purged: 0, failed: 0, skipped: 0 });await svc.purger.flush();assert.strictEqual(cf.calls.length, 0);} finally { await svc.stop(); }@@@ -153,12 +153,12 @@ t('a host with no configured zone is recorded as skipped, not sent', async () =>const svc = await boot({ env: { CLOUDFLARE_PURGE_TOKEN: TOKEN, CLOUDFLARE_ZONE_IDS: ZONES }, fetchImpl: cf.fetchImpl });try {const d = doc({ id: 'elsewhere', canonical_url: 'https://example.org/x' });-- svc.store.apply(d);-- svc.store.remove('wiki', 'page', 'elsewhere');-- assert.deepStrictEqual(svc.purges.cdnCounts(), { pending: 0, purged: 0, failed: 0, skipped: 2 });++ await svc.store.apply(d);++ await svc.store.remove('wiki', 'page', 'elsewhere');++ assert.deepStrictEqual(await svc.purges.cdnCounts(), { pending: 0, purged: 0, failed: 0, skipped: 2 });await svc.purger.flush();assert.strictEqual(cf.calls.length, 0);-- const row = svc.db.prepare("SELECT detail FROM cdn_purges WHERE url = 'https://example.org/x'").get();++ const row = await svc.db.prepare("SELECT detail FROM cdn_purges WHERE url = 'https://example.org/x'").get();assert.match(row.detail, /no Cloudflare zone/);} finally { await svc.stop(); }});@@@ -174,9 +174,9 @@ t('429, 5xx and network errors back off and retry; after MAX_ATTEMPTS the purgeconst svc = await boot({ env: { CLOUDFLARE_PURGE_TOKEN: TOKEN, CLOUDFLARE_ZONE_IDS: ZONES, CLOUDFLARE_PURGE_RELATED_PATHS: 'none' }, fetchImpl: cf.fetchImpl, now: () => clock });try {const d = doc({ id: 'retry1', canonical_url: 'https://openvibe.wiki/p/retry1' });-- svc.store.apply(d);-- svc.store.remove('wiki', 'page', 'retry1');-- const n = svc.purges.cdnCounts().pending;++ await svc.store.apply(d);++ await svc.store.remove('wiki', 'page', 'retry1');++ const n = (await svc.purges.cdnCounts()).pending;assert.strictEqual(n, 1, 'related paths off (none)');assert.deepStrictEqual(await svc.purger.flush(), { purged: 0, retrying: n, failed: 0 });// Not due yet: nothing is sent before the backoff elapses.@@@ -189,10 +189,10 @@ t('429, 5xx and network errors back off and retry; after MAX_ATTEMPTS the purge// A URL that never gets through fails after MAX_ATTEMPTS.mode = 'down';const e = doc({ id: 'retry2', canonical_url: 'https://openvibe.wiki/p/retry2' });-- svc.store.apply(e);-- svc.store.remove('wiki', 'page', 'retry2');++ await svc.store.apply(e);++ await svc.store.remove('wiki', 'page', 'retry2');for (let i = 0; i < MAX_ATTEMPTS; i++) { clock += 2 * 3600 * 1000; await svc.purger.flush(); }-- const rows = svc.db.prepare("SELECT state, attempts, detail FROM cdn_purges WHERE url LIKE '%retry2'").all();++ const rows = await svc.db.prepare("SELECT state, attempts, detail FROM cdn_purges WHERE url ILIKE '%retry2'").all();assert.deepStrictEqual(rows.map(r => [r.state, r.attempts]), [['failed', MAX_ATTEMPTS]]);assert.match(rows[0].detail, /gave up after/);} finally { await svc.stop(); }@@@ -205,12 +205,12 @@ t('a refused batch is retried one URL at a time: only the refused URL fails', asconst svc = await boot({ env: { CLOUDFLARE_PURGE_TOKEN: TOKEN, CLOUDFLARE_ZONE_IDS: ZONES, CLOUDFLARE_PURGE_RELATED_PATHS: 'none' }, fetchImpl: cf.fetchImpl });try {for (const id of ['good1', 'refused', 'good2']) {-- svc.store.apply(doc({ id, canonical_url: `https://openvibe.wiki/p/${id}` }));-- svc.store.remove('wiki', 'page', id);++ await svc.store.apply(doc({ id, canonical_url: `https://openvibe.wiki/p/${id}` }));++ await svc.store.remove('wiki', 'page', id);}const s = await svc.purger.flush();assert.deepStrictEqual(s, { purged: 2, retrying: 0, failed: 1 });-- const failed = svc.db.prepare("SELECT url, detail FROM cdn_purges WHERE state = 'failed'").all();++ const failed = await svc.db.prepare("SELECT url, detail FROM cdn_purges WHERE state = 'failed'").all();assert.deepStrictEqual(failed.map(r => r.url), ['https://openvibe.wiki/p/refused']);assert.match(failed[0].detail, /HTTP 400 1012/);} finally { await svc.stop(); }@@@ -224,10 +224,10 @@ t('a token Cloudflare rejects (403) fails the purge without retrying forever andconst { createPurger } = require('../server/purge');const purger = createPurger({ db: svc.db, config: svc.config.purge, fetchImpl: cf.fetchImpl, log: { warn: (m) => logged.push(m), log() {}, error() {} } });try {-- svc.store.apply(doc({ id: 'auth1', canonical_url: 'https://openvibe.wiki/p/auth1' }));-- svc.store.remove('wiki', 'page', 'auth1');++ await svc.store.apply(doc({ id: 'auth1', canonical_url: 'https://openvibe.wiki/p/auth1' }));++ await svc.store.remove('wiki', 'page', 'auth1');assert.deepStrictEqual(await purger.flush(), { purged: 0, retrying: 0, failed: 1 });-- const row = svc.db.prepare("SELECT state, detail FROM cdn_purges WHERE url LIKE '%auth1'").get();++ const row = await svc.db.prepare("SELECT state, detail FROM cdn_purges WHERE url ILIKE '%auth1'").get();assert.strictEqual(row.state, 'failed');assert.ok(!row.detail.includes(TOKEN) && row.detail.includes('[token]'));assert.ok(logged.length && logged.every(m => !m.includes(TOKEN)));@@@ -23,9 +23,9 @@ const call = (method, p, opts = {}) => request(svc.base, method, p, opts);t('boot with a small per-person limit', async () => {svc = await boot({ env: { BASE_URL: ORIGIN, SEARCH_SAVED_MAX_PER_SUBJECT: '3' } });-- svc.store.apply(doc({ id: 'pub', title: 'Zebra migration public', canonical_url: 'https://openvibe.wiki/p/pub', facets: { category: 'animals' } }));-- svc.store.apply(doc({ id: 'mem', visibility: 'members', acl: { subjects: [alice] }, title: 'Zebra migration for alice', facets: { category: 'animals' } }));-- svc.store.apply(doc({ owner: 'blog', type: 'post', id: 'other', title: 'Zebra blog post', canonical_url: 'https://openvibe.blog/p/other' }));++ await svc.store.apply(doc({ id: 'pub', title: 'Zebra migration public', canonical_url: 'https://openvibe.wiki/p/pub', facets: { category: 'animals' } }));++ await svc.store.apply(doc({ id: 'mem', visibility: 'members', acl: { subjects: [alice] }, title: 'Zebra migration for alice', facets: { category: 'animals' } }));++ await svc.store.apply(doc({ owner: 'blog', type: 'post', id: 'other', title: 'Zebra blog post', canonical_url: 'https://openvibe.blog/p/other' }));});t('anonymous callers, guests and services without delegation cannot use saved searches', async () => {@@@ -71,11 +71,11 @@ t('running a saved search queries as the person now, with its filters', async ()assert.ok(after.body.saved_search.last_run_at);// Access is decided at run time: alice loses the members document, it leaves her results.-- svc.store.apply(doc({ id: 'mem', revision: 2, visibility: 'members', acl: { subjects: [bob] }, title: 'Zebra migration for alice', facets: { category: 'animals' } }));++ await svc.store.apply(doc({ id: 'mem', revision: 2, visibility: 'members', acl: { subjects: [bob] }, title: 'Zebra migration for alice', facets: { category: 'animals' } }));const r2 = await call('GET', `/api/v1/saved-searches/${saved.id}/results`, as(alice));assert.deepStrictEqual(r2.body.results.map(x => x.id), ['pub']);// And a deleted public document leaves it too.-- svc.store.remove('wiki', 'page', 'pub');++ await svc.store.remove('wiki', 'page', 'pub');const r3 = await call('GET', `/api/v1/saved-searches/${saved.id}/results`, as(alice));assert.deepStrictEqual(r3.body.results, []);});@@@ -7,7 +7,7 @@ const { boot, request, serviceToken, doc, indexEvent, deliver, suite } = requireconst t = suite('webhook-inbox');let svc;const search = async (q) => (await request(svc.base, 'GET', `/api/v1/search?q=${encodeURIComponent(q)}`)).body.results;--const receipts = () => svc.db.prepare('SELECT COUNT(*) AS n FROM idempotency_receipts').get().n;++const receipts = async () => (await svc.db.prepare('SELECT COUNT(*) AS n FROM idempotency_receipts').get()).n;t('boot', async () => { svc = await boot(); });@@@ -18,7 +18,7 @@ t('an unsigned or wrongly signed delivery is refused and changes nothing', asyncconst r2 = await request(svc.base, 'POST', '/internal/events', { raw: JSON.stringify({ event: e }), headers: { 'Content-Type': 'application/json' } });assert.strictEqual(r2.status, 401);assert.strictEqual((await search('muword')).length, 0);-- assert.strictEqual(receipts(), 0);++ assert.strictEqual(await receipts(), 0);});t('signature v2 is required: a v1-only delivery and a stale v2 timestamp are refused', async () => {@@@ -30,7 +30,7 @@ t('signature v2 is required: a v1-only delivery and a stale v2 timestamp are refconst future = await deliver(svc.base, e, { now: Date.now() + 301000 });assert.strictEqual(future.status, 401, 'v2 from too far in the future: refused');assert.strictEqual((await search('omiword')).length, 0);-- assert.strictEqual(receipts(), 0);++ assert.strictEqual(await receipts(), 0);const ok = await deliver(svc.base, e);assert.strictEqual(ok.status, 200, 'a fresh v2 delivery of the same event is accepted');assert.strictEqual((await search('omiword')).length, 1);@@@ -45,7 +45,7 @@ t('a delivered upsert is indexed; the same event again is a no-op', async () =>const again = await deliver(svc.base, e, { seq: 2 });assert.deepStrictEqual([again.status, again.body.duplicate], [200, true]);assert.strictEqual((await search('nuword')).length, 1);-- const events = svc.outbox.all().filter(x => x.payload.id === d.id && x.event_type === 'search.document.indexed');++ const events = (await svc.outbox.all()).filter(x => x.payload.id === d.id && x.event_type === 'search.document.indexed');assert.strictEqual(events.length, 1, 'one effect, one event');});@@@ -73,7 +73,7 @@ t('a deletion event tombstones and wins over upserts delivered after it', asyncassert.strictEqual((await deliver(svc.base, indexEvent(d))).body.outcome, 'stale');assert.strictEqual((await deliver(svc.base, indexEvent({ ...d, revision: 5 }))).body.outcome, 'stale');assert.strictEqual((await search('omicronword')).length, 0);-- const removed = svc.outbox.all().filter(x => x.event_type === 'search.document.removed' && x.payload.id === d.id);++ const removed = (await svc.outbox.all()).filter(x => x.event_type === 'search.document.removed' && x.payload.id === d.id);assert.deepStrictEqual(removed.map(x => x.payload.reason), ['deleted']);});@@@ -136,7 +136,7 @@ t('a failure while applying leaves no receipt, so the redelivery applies exactlyconst r1 = await deliver(svc.base, e);assert.strictEqual(r1.status, 500);assert.strictEqual((await search('tauword')).length, 0);-- assert.strictEqual(svc.db.prepare('SELECT COUNT(*) AS n FROM idempotency_receipts WHERE event_id = ?').get(e.event_id).n, 0);++ assert.strictEqual((await svc.db.prepare('SELECT COUNT(*) AS n FROM idempotency_receipts WHERE event_id = ?').get(e.event_id)).n, 0);const r2 = await deliver(svc.base, e, { seq: 2 });assert.deepStrictEqual([r2.status, r2.body.outcome], [200, 'applied']);const r3 = await deliver(svc.base, e, { seq: 3 });@@@ -145,7 +145,7 @@ t('a failure while applying leaves no receipt, so the redelivery applies exactlysvc.store.apply = original;}assert.strictEqual((await search('tauword')).length, 1);-- assert.strictEqual(svc.outbox.all().filter(x => x.payload.id === d.id).length, 1);++ assert.strictEqual((await svc.outbox.all()).filter(x => x.payload.id === d.id).length, 1);});t('OpenVibe.Sources item documents (the first real producer) index for staff only', async () => {
@@ -28,6 +28,14 @@ OV_OAUTH_CLIENT_ID=search
@@ -28,6 +28,14 @@ OV_OAUTH_CLIENT_ID=search
OV_OAUTH_CLIENT_SECRET=
OV_OAUTH_CLIENT_SECRET=
# Store (SQLite, FTS5). Production: /var/lib/openvibe-search/search.db (systemd StateDirectory).
# Store (SQLite, FTS5). Production: /var/lib/openvibe-search/search.db (systemd StateDirectory).
+# PostgreSQL (ADR-035), written by OpenVibe.Host roles/data/add-service.sh search: the runtime role through PgBouncer,
+# the owner on a direct connection (migrations and the one-time import), and Valkey (shared limit counters).
+# Without DATABASE_URL, development uses an embedded PGlite database in data/pglite.
+DATABASE_URL=
+DATABASE_DIRECT_URL=
+VALKEY_URL=
+VALKEY_PREFIX=
+# The SQLite file of releases before PostgreSQL: read once by scripts/migrate-to-postgres.js.
SEARCH_DB_PATH=./data/search.db
SEARCH_DB_PATH=./data/search.db
# OpenVibe.Events. The outbox relay runs only when EVENTS_URL is set.
# OpenVibe.Events. The outbox relay runs only when EVENTS_URL is set.
@@ -9,6 +9,9 @@ jobs:
@@ -9,6 +9,9 @@ jobs:
uses: OpenVibers/OpenVibe.Shared/.github/workflows/test.yml@423fe519b5fe9594cc81b670e5ba5cb9f21f6176
uses: OpenVibers/OpenVibe.Shared/.github/workflows/test.yml@423fe519b5fe9594cc81b670e5ba5cb9f21f6176
with:
with:
contracts-service: search
contracts-service: search
+ # ADR-035: PostgreSQL 18 + PgBouncer (transaction mode) + Valkey 9 containers (openvibe-sdk
+ # scripts/test-services.sh). npm test runs every file on PGlite; test:pg runs them all through PgBouncer.
+ test: 'eval "$(node_modules/openvibe-sdk/scripts/test-services.sh up)" && npm test && npm run test:pg'
# Secret scan (gitleaks) and dependency audit, shared by every OpenVibers repo (Track Q).
# Secret scan (gitleaks) and dependency audit, shared by every OpenVibers repo (Track Q).
security:
security:
uses: OpenVibers/OpenVibe.Shared/.github/workflows/security.yml@423fe519b5fe9594cc81b670e5ba5cb9f21f6176
uses: OpenVibers/OpenVibe.Shared/.github/workflows/security.yml@423fe519b5fe9594cc81b670e5ba5cb9f21f6176
@@ -30,7 +30,7 @@ npm test # every test/*.test.js, temp databases, no network needed
@@ -30,7 +30,7 @@ npm test # every test/*.test.js, temp databases, no network needed
Node 22 in production (`fnm exec --using=22.22.1 npm test`). Production: `/opt/openvibe.search`,
Node 22 in production (`fnm exec --using=22.22.1 npm test`). Production: `/opt/openvibe.search`,
env `/etc/openvibe/search.env`, unit [deploy/systemd/openvibe-search.service](deploy/systemd/openvibe-search.service),
env `/etc/openvibe/search.env`, unit [deploy/systemd/openvibe-search.service](deploy/systemd/openvibe-search.service),
-index `/var/lib/openvibe-search/search.db`, nginx [deploy/nginx/search.openvibe.network.conf](deploy/nginx/search.openvibe.network.conf)
+database `ov_search` on the host's data role (ADR-035; `sudo /opt/openvibe.host/roles/data/add-service.sh search`; the one-time move from `/var/lib/openvibe-search/search.db` is `scripts/migrate-to-postgres.js`, and the full-text index rebuilds itself from the documents at boot), nginx [deploy/nginx/search.openvibe.network.conf](deploy/nginx/search.openvibe.network.conf)
(the public vhost exposes the search page, the GET query API, saved searches and health; the owner
(the public vhost exposes the search page, the GET query API, saved searches and health; the owner
API and the Events webhook are loopback-only, and owners and Events call `127.0.0.1:4710`).
API and the Events webhook are loopback-only, and owners and Events call `127.0.0.1:4710`).
@@ -173,9 +173,9 @@ the `search.document.removed` event it records:
@@ -173,9 +173,9 @@ the `search.document.removed` event it records:
## Owns
## Owns
- the index document contract (`search.index-document@1`, released in openvibe-contracts v0.12.0; proposal in [docs/contracts-proposal/](docs/contracts-proposal/))
- the index document contract (`search.index-document@1`, released in openvibe-contracts v0.12.0; proposal in [docs/contracts-proposal/](docs/contracts-proposal/))
-- `documents`, `doc_acl`, `doc_facets`, `fts_public`, `fts_restricted`, `idempotency_receipts`,
- `ingest_rejections`, `event_outbox`, `removals`, `cdn_purges`, `saved_searches` (SQLite; engine
- decision in [docs/adr-engine.md](docs/adr-engine.md))
+- `documents`, `doc_acl`, `doc_facets`, `search_fts` (the full-text index), `idempotency_receipts`,
+ `ingest_rejections`, `event_outbox`, `removals`, `cdn_purges`, `saved_searches` (PostgreSQL, schema in
+ [migrations/](migrations/); engine decision in [docs/adr-engine.md](docs/adr-engine.md))
- the query API, its ranking and ACL filtering, the search page and saved searches; deletion and
- the query API, its ranking and ACL filtering, the search page and saved searches; deletion and
visibility-change propagation, including the CDN purge of removed public URLs
visibility-change propagation, including the CDN purge of removed public URLs
@@ -189,6 +189,7 @@ the `search.document.removed` event it records:
@@ -189,6 +189,7 @@ the `search.document.removed` event it records:
## Depends on
## Depends on
- OpenVibe.Contracts (service tokens, problem details, ids, the event envelope)
- OpenVibe.Contracts (service tokens, problem details, ids, the event envelope)
+- PostgreSQL 18 (OpenVibe.Host `roles/data/`, ADR-035): documents and the full-text index, async through `openvibe-sdk/db`
- OpenVibe.Network (signing key; service principal `search`)
- OpenVibe.Network (signing key; service principal `search`)
- OpenVibe.Events (index-document deliveries in, `search.document.*` out)
- OpenVibe.Events (index-document deliveries in, `search.document.*` out)
@@ -209,7 +210,7 @@ Called elsewhere, as the service principal `search` (client credentials from Ope
@@ -209,7 +210,7 @@ Called elsewhere, as the service principal `search` (client credentials from Ope
| OpenVibe.Events | `events.event.publish` (audience `openvibe.events`) | the outbox relays `search.document.indexed` / `search.document.removed` |
| OpenVibe.Events | `events.event.publish` (audience `openvibe.events`) | the outbox relays `search.document.indexed` / `search.document.removed` |
| OpenVibe.Events | `events.subscription.manage` (`npm run subscribe`, once) | the HMAC-signed subscription to `*.index_document.*` that delivers index events |
| OpenVibe.Events | `events.subscription.manage` (`npm run subscribe`, once) | the HMAC-signed subscription to `*.index_document.*` that delivers index events |
-Released in `openvibe-contracts` v0.12.0 with the service manifest (this repo pins v0.49.0);
+Released in `openvibe-contracts` v0.76.0 with the service manifest (this repo pins v0.49.0);
[server/auth.js](server/auth.js) decides them with the contracts grant rule, and CI runs
[server/auth.js](server/auth.js) decides them with the contracts grant rule, and CI runs
`openvibe-contracts-check --service search`.
`openvibe-contracts-check --service search`.
@@ -294,7 +295,7 @@ fast-forward `/opt/openvibe.search`, install on a lockfile change, restart, wait
@@ -294,7 +295,7 @@ fast-forward `/opt/openvibe.search`, install on a lockfile change, restart, wait
| Unit | `openvibe-search.service` ([deploy/systemd/](deploy/systemd/openvibe-search.service)), runs as `ubuntu` |
| Unit | `openvibe-search.service` ([deploy/systemd/](deploy/systemd/openvibe-search.service)), runs as `ubuntu` |
| Port | `127.0.0.1:4710` |
| Port | `127.0.0.1:4710` |
| Env file | `/etc/openvibe/search.env` |
| Env file | `/etc/openvibe/search.env` |
-| Data | `/var/lib/openvibe-search/search.db` (`StateDirectory`) |
+| Data | PostgreSQL `ov_search` (`DATABASE_URL` through PgBouncer; migrations on `DATABASE_DIRECT_URL`) |
| Public host | nginx [deploy/nginx/search.openvibe.network.conf](deploy/nginx/search.openvibe.network.conf) (below) |
| Public host | nginx [deploy/nginx/search.openvibe.network.conf](deploy/nginx/search.openvibe.network.conf) (below) |
Rollback: ovhost puts the previous sha back by itself when `/api/ready` does not answer 2xx after the
Rollback: ovhost puts the previous sha back by itself when `/api/ready` does not answer 2xx after the
@@ -7,14 +7,22 @@
@@ -7,14 +7,22 @@
"runtime": "node 22, express, better-sqlite3 (FTS5)",
"runtime": "node 22, express, better-sqlite3 (FTS5)",
"port": 4710,
"port": 4710,
"domain": "search.openvibe.network (vhost in deploy/nginx; the service manifest records the search page and query API as public since 2026-09-23)",
"domain": "search.openvibe.network (vhost in deploy/nginx; the service manifest records the search page and query API as public since 2026-09-23)",
- "contracts": "openvibe-contracts v0.49.0",
- "capabilities": ["search.document.write", "search.query.delegate", "search.query.run"],
+ "contracts": "openvibe-contracts v0.76.0",
+ "capabilities": [
+ "search.document.write",
+ "search.query.delegate",
+ "search.query.run"
+ ],
"capabilitiesRegistered": true,
"capabilitiesRegistered": true,
- "contractsProposed": ["search.index-document@1 (released in openvibe-contracts v0.12.0)"],
+ "contractsProposed": [
+ "search.index-document@1 (released in openvibe-contracts v0.12.0)"
+ ],
"engine": "sqlite-fts5 (docs/adr-engine.md)",
"engine": "sqlite-fts5 (docs/adr-engine.md)",
"deployed": true,
"deployed": true,
"deployment": "Deployed with `sudo ovhost deploy search`: /opt/openvibe.search, unit openvibe-search, 127.0.0.1:4710, principal search, Events subscription *.index_document.*, accepted event owners sources, wiki, blog, news, reviews, trade, deals, coupons. On 2026-09-23 the production index held 10 documents, all tombstones (wiki), 0 public. Restore drill passed 2026-09-23.",
"deployment": "Deployed with `sudo ovhost deploy search`: /opt/openvibe.search, unit openvibe-search, 127.0.0.1:4710, principal search, Events subscription *.index_document.*, accepted event owners sources, wiki, blog, news, reviews, trade, deals, coupons. On 2026-09-23 the production index held 10 documents, all tombstones (wiki), 0 public. Restore drill passed 2026-09-23.",
- "indexedProducers": ["wiki (tombstones only so far)"],
+ "indexedProducers": [
+ "wiki (tombstones only so far)"
+ ],
"features": [
"features": [
"ACL-filtered FTS5 query API",
"ACL-filtered FTS5 query API",
"facets",
"facets",
@@ -2,8 +2,8 @@
@@ -2,8 +2,8 @@
Description=OpenVibe.Search — permission-aware index and query API (SQLite FTS5)
Description=OpenVibe.Search — permission-aware index and query API (SQLite FTS5)
# Network (openvibe-network) signs the tokens this service verifies. Search starts without it
# Network (openvibe-network) signs the tokens this service verifies. Search starts without it
# and keeps retrying the JWKS fetch; /api/ready stays 503 until the key has loaded.
# and keeps retrying the JWKS fetch; /api/ready stays 503 until the key has loaded.
-After=network.target
-Wants=openvibe-network.service
+After=network.target postgresql.service pgbouncer.service valkey-server.service
+Wants=openvibe-network.service postgresql.service pgbouncer.service valkey-server.service
[Service]
[Service]
Type=simple
Type=simple
@@ -51,3 +51,16 @@ is no corpus to measure: no product indexes anything today.
@@ -51,3 +51,16 @@ is no corpus to measure: no product indexes anything today.
(Japanese/Chinese get no word segmentation; a later engine or an ICU tokenizer fixes that).
(Japanese/Chinese get no word segmentation; a later engine or an ICU tokenizer fixes that).
- Moving to PostgreSQL is an engine swap plus a re-index from owners (the reconciliation API
- Moving to PostgreSQL is an engine swap plus a re-index from owners (the reconciliation API
exists for exactly that), not a contract change.
exists for exactly that), not a contract change.
+
+## Update 2026-09-28: PostgreSQL full-text search (ADR-035)
+
+Search moved to PostgreSQL with every other service. The engine interface held: `server/engine/pg.js` replaced
+`server/engine/fts5.js` and nothing else learned the engine changed. One table, `search_fts`, split by audience
+(exposure 3 public listed, 1 restricted), with a `tsvector` weighted title A, summary B, body C (FTS5's 10 / 4 / 1 as
+`ts_rank` weights) and a GIN index. Text is folded (diacritics removed) and split into runs of letters and digits before
+the `simple` configuration sees it, so tokens are exactly FTS5 unicode61's: on a copy of production, the match counts
+for sampled terms were identical. The rank is minus `ts_rank` times the same freshness boost, so ascending order and
+the keyset cursors work unchanged. Snippets are `ts_headline` over the summary and body (the title when only it
+matches), computed for the returned page only. The index is derived: at boot, when its counts disagree with the
+documents table (after the one-time import, whose FTS5 tables are not copied), it is rebuilt from `documents.doc`.
+
new file mode 100644
new file mode 100644
@@ -0,0 +1,153 @@
@@ -0,0 +1,153 @@
+-- phase: expand
+-- OpenVibe.Search on PostgreSQL (ADR-035, roadmap WS-X2): the tables as they were on SQLite (converted by openvibe-sdk
+-- tools/asyncify/sqlite-schema-to-pg: text COLLATE "C" compares like SQLite, integers are bigint, identities keep their ids),
+-- then the openvibe-publishing stores and the openvibe-sdk inbox and outbox. Generated once on 2026-09-28; never edited after it runs.
+
+CREATE TABLE documents (
+ rid bigint GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
+ owner text COLLATE "C" NOT NULL,
+ type text COLLATE "C" NOT NULL,
+ id text COLLATE "C" NOT NULL,
+ revision bigint NOT NULL,
+ deleted bigint NOT NULL DEFAULT 0,
+ visibility text COLLATE "C",
+ publication_state text COLLATE "C",
+ exposure bigint NOT NULL DEFAULT 0, -- document.js EXPOSURE (0 none .. 3 public listed)
+ index_decision text COLLATE "C" NOT NULL,
+ noindex_reasons text COLLATE "C" NOT NULL DEFAULT '[]',
+ canonical_url text COLLATE "C",
+ title text COLLATE "C",
+ language text COLLATE "C",
+ published_at text COLLATE "C",
+ updated_at text COLLATE "C",
+ sort_at text COLLATE "C" NOT NULL DEFAULT '', -- updated_at, else published_at: browse order
+ doc text COLLATE "C" NOT NULL, -- normalized document JSON (tombstone: identity only)
+ hash text COLLATE "C" NOT NULL,
+ via text COLLATE "C" NOT NULL, -- event | api
+ event_id text COLLATE "C",
+ indexed_at bigint NOT NULL,
+ UNIQUE (owner, type, id)
+);
+CREATE INDEX idx_documents_exposure ON documents (exposure, sort_at, rid);
+CREATE INDEX idx_documents_owner ON documents (owner, type, id);
+
+-- ACL keys of servable non-public documents: kind s = subject, g = group, e = entitlement.
+CREATE TABLE doc_acl (
+ rid bigint NOT NULL,
+ kind text COLLATE "C" NOT NULL CHECK (kind IN ('s', 'g', 'e')),
+ value text COLLATE "C" NOT NULL,
+ PRIMARY KEY (rid, kind, value)
+);
+CREATE INDEX idx_doc_acl_value ON doc_acl (kind, value, rid);
+
+CREATE TABLE doc_facets (
+ rid bigint NOT NULL,
+ key text COLLATE "C" NOT NULL,
+ value text COLLATE "C" NOT NULL,
+ PRIMARY KEY (rid, key, value)
+);
+CREATE INDEX idx_doc_facets_value ON doc_facets (key, value, rid);
+
+-- Inbox: one receipt per consumed event (exactly-once effects), with what happened.
+CREATE TABLE idempotency_receipts (
+ consumer text COLLATE "C" NOT NULL,
+ event_id text COLLATE "C" NOT NULL,
+ outcome text COLLATE "C" NOT NULL,
+ processed_at bigint NOT NULL,
+ PRIMARY KEY (consumer, event_id)
+);
+
+-- Index-document events Search refused (bad document, wrong owner, subject mismatch). Owners
+-- read their own through GET /api/v1/owners/:owner/rejections.
+CREATE TABLE ingest_rejections (
+ seq bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+ event_id text COLLATE "C" NOT NULL,
+ event_type text COLLATE "C",
+ owner text COLLATE "C",
+ type text COLLATE "C",
+ id text COLLATE "C",
+ revision bigint,
+ code text COLLATE "C" NOT NULL,
+ detail text COLLATE "C",
+ at bigint NOT NULL
+);
+CREATE INDEX idx_ingest_rejections_owner ON ingest_rejections (owner, seq);
+
+-- server/events/outbox.js ensureSchema()
+CREATE TABLE event_outbox (
+ seq bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+ event_id text COLLATE "C" NOT NULL UNIQUE,
+ event_type text COLLATE "C" NOT NULL,
+ envelope text COLLATE "C" NOT NULL,
+ created_at bigint NOT NULL,
+ attempts bigint NOT NULL DEFAULT 0,
+ next_attempt_at bigint NOT NULL DEFAULT 0,
+ sent_at bigint,
+ rejected_at bigint,
+ last_error text COLLATE "C"
+);
+CREATE INDEX idx_event_outbox_due ON event_outbox(sent_at, rejected_at, next_attempt_at);
+
+-- server/purge.js ensureSchema()
+CREATE TABLE removals (
+ seq bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+ event_id text COLLATE "C" NOT NULL, -- the search.document.removed event
+ owner text COLLATE "C" NOT NULL,
+ type text COLLATE "C" NOT NULL,
+ id text COLLATE "C" NOT NULL,
+ revision bigint NOT NULL,
+ reason text COLLATE "C" NOT NULL,
+ previous_exposure text COLLATE "C" NOT NULL,
+ exposure text COLLATE "C" NOT NULL,
+ canonical_url text COLLATE "C", -- only a URL that had been public
+ at bigint NOT NULL
+);
+CREATE INDEX idx_removals_owner ON removals (owner, seq);
+
+CREATE TABLE cdn_purges (
+ seq bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+ removal_seq bigint NOT NULL,
+ provider text COLLATE "C" NOT NULL DEFAULT 'cloudflare',
+ zone_id text COLLATE "C",
+ url text COLLATE "C" NOT NULL,
+ state text COLLATE "C" NOT NULL CHECK (state IN ('pending', 'purged', 'failed', 'skipped')),
+ detail text COLLATE "C",
+ attempts bigint NOT NULL DEFAULT 0,
+ next_attempt_at bigint NOT NULL DEFAULT 0,
+ created_at bigint NOT NULL,
+ done_at bigint
+);
+-- One pending purge per URL: a sitemap shared by many removals is purged once per batch.
+CREATE UNIQUE INDEX idx_cdn_purges_pending ON cdn_purges (zone_id, url) WHERE state = 'pending';
+CREATE INDEX idx_cdn_purges_due ON cdn_purges (state, next_attempt_at, seq);
+
+-- server/saved.js ensureSchema()
+CREATE TABLE saved_searches (
+ id text COLLATE "C" PRIMARY KEY,
+ subject text COLLATE "C" NOT NULL, -- usr_… (the person who saved it)
+ name text COLLATE "C" NOT NULL,
+ q text COLLATE "C" NOT NULL DEFAULT '',
+ filters text COLLATE "C" NOT NULL, -- canonical JSON {owner,type,language,facets}
+ query_hash text COLLATE "C" NOT NULL, -- same query saved twice = one row
+ created_at bigint NOT NULL,
+ updated_at bigint NOT NULL,
+ last_run_at bigint,
+ UNIQUE (subject, query_hash)
+);
+CREATE INDEX idx_saved_searches_subject ON saved_searches (subject, created_at);
+
+-- The full-text index (server/engine/pg.js): one row per servable document, split by audience (exposure 3 public
+-- listed: all an anonymous query ever reads; 1 restricted: ACL-checked in the same SQL). Drafts, unpublished, deleted
+-- and noindex documents have no row. Title, summary and body carry weights A, B and C. Derived from documents.doc and
+-- rebuilt at boot when its counts disagree with the documents table (after the one-time import, for instance).
+CREATE TABLE search_fts (
+ rid bigint PRIMARY KEY REFERENCES documents(rid) ON DELETE CASCADE,
+ audience bigint NOT NULL,
+ fresh_at bigint, -- epoch ms of published_at, else updated_at (the freshness boost)
+ title text NOT NULL,
+ summary text NOT NULL,
+ body text NOT NULL,
+ tsv tsvector NOT NULL
+);
+CREATE INDEX search_fts_tsv ON search_fts USING gin (tsv);
+CREATE INDEX search_fts_audience ON search_fts (audience, rid);
@@ -14,14 +14,25 @@
@@ -14,14 +14,25 @@
"better-sqlite3": "^11.10.0",
"better-sqlite3": "^11.10.0",
"dotenv": "^16.4.7",
"dotenv": "^16.4.7",
"express": "^4.21.2",
"express": "^4.21.2",
- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.49.0",
- "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0",
- "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0"
+ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",
+ "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.20.0",
+ "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.27.0",
+ "pg": "^8.23.0"
+ },
+ "devDependencies": {
+ "@electric-sql/pglite": "^0.5.8"
},
},
"engines": {
"engines": {
"node": ">=22.0.0"
"node": ">=22.0.0"
}
}
},
},
+ "node_modules/@electric-sql/pglite": {
+ "version": "0.5.8",
+ "resolved": "https://registry.npmjs.org/@electric-sql/pglite/-/pglite-0.5.8.tgz",
+ "integrity": "sha512-n9tsbUOhwx2epK1V0ZG9Ar4SHWUju04dhmzZXiSBXwBoleOvIfals33NAaWgagQVAL4Rbvx/Ptsu3P+pA09f6Q==",
+ "devOptional": true,
+ "license": "Apache-2.0"
+ },
"node_modules/accepts": {
"node_modules/accepts": {
"version": "1.3.8",
"version": "1.3.8",
"resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.8.tgz",
"resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.8.tgz",
@@ -970,9 +981,9 @@
@@ -970,9 +981,9 @@
}
}
},
},
"node_modules/openvibe-contracts": {
"node_modules/openvibe-contracts": {
- "version": "0.49.0",
- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.49.0",
- "integrity": "sha512-lapsLoBm3ZpndgiOMELdCBh2PJtuuyqj6SUhQsfwJkxnLdQfyhXfRFgiuLh+iQrg4kqtG9611191Mp8zAfk7fQ==",
+ "version": "0.76.0",
+ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",
+ "integrity": "sha512-zTdx776KWrcJKm1JTnf+0+BnsvswdvXy1ldPyse1hSqhnW/SSr98sX696eRetFUhTl8qVM1xSRCykqIUbqItPw==",
"license": "AGPL-3.0-only",
"license": "AGPL-3.0-only",
"dependencies": {
"dependencies": {
"ajv": "^8.17.1",
"ajv": "^8.17.1",
@@ -986,18 +997,38 @@
@@ -986,18 +997,38 @@
}
}
},
},
"node_modules/openvibe-sdk": {
"node_modules/openvibe-sdk": {
- "version": "0.12.0",
- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0",
- "integrity": "sha512-iG1h905O1f7y/PnP8xLA6GCVzXEBDfSaBj2tc1ucNreWg2sT/AJWosb8NW0WRMK5WS4sIQ6tMsprmDLj/fjjeQ==",
+ "version": "0.20.0",
+ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.20.0",
+ "integrity": "sha512-CqZFfVuSv3ZPoyG1LuWCAdvo8TG2ZuUCeGaqTsxDKS3gsymSFV2QXPq/5QUotTd82WShTGUKKhTgcyp9ggUMfQ==",
"license": "MIT",
"license": "MIT",
"engines": {
"engines": {
"node": ">=20"
"node": ">=20"
+ },
+ "peerDependencies": {
+ "@electric-sql/pglite": "^0.5.8",
+ "better-sqlite3": ">=11",
+ "iovalkey": "^0.4.0",
+ "pg": "^8.23.0"
+ },
+ "peerDependenciesMeta": {
+ "@electric-sql/pglite": {
+ "optional": true
+ },
+ "better-sqlite3": {
+ "optional": true
+ },
+ "iovalkey": {
+ "optional": true
+ },
+ "pg": {
+ "optional": true
+ }
}
}
},
},
"node_modules/openvibe-shared": {
"node_modules/openvibe-shared": {
- "version": "1.25.0",
- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0",
- "integrity": "sha512-oqZ+PvljrWi8IEbtg6bDYai17ZV2sALFx/ezMptniELSr10TgFUiGUQt+b1IWwiHv0T2Y1U2iSrkirJxMNgjSw==",
+ "version": "1.27.0",
+ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.27.0",
+ "integrity": "sha512-/H7dyFw+cURcdjyfbpyY7mNc6F+Pcb9TwkVwZHR+9d3M62q70KDhZLo/gZi71eaYxUjD8lO8tmbXu70EAa9paQ==",
"license": "MIT",
"license": "MIT",
"dependencies": {
"dependencies": {
"jsonwebtoken": "^9.0.0"
"jsonwebtoken": "^9.0.0"
@@ -1018,6 +1049,134 @@
@@ -1018,6 +1049,134 @@
"integrity": "sha512-A/AGNMFN3c8bOlvV9RreMdrv7jsmF9XIfDeCd87+I8RNg6s78BhJxMu69NEMHBSJFxKidViTEdruRwEk/WIKqA==",
"integrity": "sha512-A/AGNMFN3c8bOlvV9RreMdrv7jsmF9XIfDeCd87+I8RNg6s78BhJxMu69NEMHBSJFxKidViTEdruRwEk/WIKqA==",
"license": "MIT"
"license": "MIT"
},
},
+ "node_modules/pg": {
+ "version": "8.23.0",
+ "resolved": "https://registry.npmjs.org/pg/-/pg-8.23.0.tgz",
+ "integrity": "sha512-Ip2EQCngowJLGOfCwkFhPXU7/ljlhn6Rxlmy4XYfL2Y+vyRM59+8uR2xqRWKdYmbXmxCFOAmKxBuSUCdF34qLg==",
+ "license": "MIT",
+ "dependencies": {
+ "pg-connection-string": "^2.14.0",
+ "pg-pool": "^3.14.0",
+ "pg-protocol": "^1.16.0",
+ "pg-types": "2.2.0",
+ "pgpass": "1.0.5"
+ },
+ "engines": {
+ "node": ">= 16.0.0"
+ },
+ "optionalDependencies": {
+ "pg-cloudflare": "^1.4.0"
+ },
+ "peerDependencies": {
+ "pg-native": ">=3.0.1"
+ },
+ "peerDependenciesMeta": {
+ "pg-native": {
+ "optional": true
+ }
+ }
+ },
+ "node_modules/pg-cloudflare": {
+ "version": "1.4.0",
+ "resolved": "https://registry.npmjs.org/pg-cloudflare/-/pg-cloudflare-1.4.0.tgz",
+ "integrity": "sha512-Vo7z/6rrQYxpNRylp4Tlob2elzbh+N/MOQbxFVWCxS7oEx6jF53GTJFxK2WWpKuBRkmiin4Mt+xofFDjx09R0A==",
+ "license": "MIT",
+ "optional": true
+ },
+ "node_modules/pg-connection-string": {
+ "version": "2.14.0",
+ "resolved": "https://registry.npmjs.org/pg-connection-string/-/pg-connection-string-2.14.0.tgz",
+ "integrity": "sha512-XwWDGcLRGCXAR8F/AM5bG7Q+A3Wm2s6QeEjlOKZLlH3UYcguiqCWKyWXVag5TLTIjR7oOJUY8kcADaZgWPyLeg==",
+ "license": "MIT"
+ },
+ "node_modules/pg-int8": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/pg-int8/-/pg-int8-1.0.1.tgz",
+ "integrity": "sha512-WCtabS6t3c8SkpDBUlb1kjOs7l66xsGdKpIPZsg4wR+B3+u9UAum2odSsF9tnvxg80h4ZxLWMy4pRjOsFIqQpw==",
+ "license": "ISC",
+ "engines": {
+ "node": ">=4.0.0"
+ }
+ },
+ "node_modules/pg-pool": {
+ "version": "3.14.0",
+ "resolved": "https://registry.npmjs.org/pg-pool/-/pg-pool-3.14.0.tgz",
+ "integrity": "sha512-gKtPkFdQPU3DksooVLi9LsjZxrsBUZIpa+7aVx+LV5pNh0KzP4Zleud2po+ConrxbuXGBJ6Hfer6hdgpIBpBaw==",
+ "license": "MIT",
+ "peerDependencies": {
+ "pg": ">=8.0"
+ }
+ },
+ "node_modules/pg-protocol": {
+ "version": "1.16.0",
+ "resolved": "https://registry.npmjs.org/pg-protocol/-/pg-protocol-1.16.0.tgz",
+ "integrity": "sha512-sILXutLVjCLjcDuOmvhX5e2Z4cS5qG/6Bu3VkpFwdf/633ElGLpEh9bgmuI5I4sqKqkifQiGyiCcx1HdtrK7tg==",
+ "license": "MIT"
+ },
+ "node_modules/pg-types": {
+ "version": "2.2.0",
+ "resolved": "https://registry.npmjs.org/pg-types/-/pg-types-2.2.0.tgz",
+ "integrity": "sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA==",
+ "license": "MIT",
+ "dependencies": {
+ "pg-int8": "1.0.1",
+ "postgres-array": "~2.0.0",
+ "postgres-bytea": "~1.0.0",
+ "postgres-date": "~1.0.4",
+ "postgres-interval": "^1.1.0"
+ },
+ "engines": {
+ "node": ">=4"
+ }
+ },
+ "node_modules/pgpass": {
+ "version": "1.0.5",
+ "resolved": "https://registry.npmjs.org/pgpass/-/pgpass-1.0.5.tgz",
+ "integrity": "sha512-FdW9r/jQZhSeohs1Z3sI1yxFQNFvMcnmfuj4WBMUTxOrAyLMaTcE1aAMBiTlbMNaXvBCQuVi0R7hd8udDSP7ug==",
+ "license": "MIT",
+ "dependencies": {
+ "split2": "^4.1.0"
+ }
+ },
+ "node_modules/postgres-array": {
+ "version": "2.0.0",
+ "resolved": "https://registry.npmjs.org/postgres-array/-/postgres-array-2.0.0.tgz",
+ "integrity": "sha512-VpZrUqU5A69eQyW2c5CA1jtLecCsN2U/bD6VilrFDWq5+5UIEVO7nazS3TEcHf1zuPYO/sqGvUvW62g86RXZuA==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=4"
+ }
+ },
+ "node_modules/postgres-bytea": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/postgres-bytea/-/postgres-bytea-1.0.1.tgz",
+ "integrity": "sha512-5+5HqXnsZPE65IJZSMkZtURARZelel2oXUEO8rH83VS/hxH5vv1uHquPg5wZs8yMAfdv971IU+kcPUczi7NVBQ==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/postgres-date": {
+ "version": "1.0.7",
+ "resolved": "https://registry.npmjs.org/postgres-date/-/postgres-date-1.0.7.tgz",
+ "integrity": "sha512-suDmjLVQg78nMK2UZ454hAG+OAW+HQPZ6n++TNDUX+L0+uUlLywnoxJKDou51Zm+zTCjrCl0Nq6J9C5hP9vK/Q==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/postgres-interval": {
+ "version": "1.2.0",
+ "resolved": "https://registry.npmjs.org/postgres-interval/-/postgres-interval-1.2.0.tgz",
+ "integrity": "sha512-9ZhXKM/rw350N1ovuWHbGxnGh/SNJ4cnxHiM0rxE4VN41wsg8P8zWn9hv/buK00RP4WvlOyr/RBDiptyxVbkZQ==",
+ "license": "MIT",
+ "dependencies": {
+ "xtend": "^4.0.0"
+ },
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
"node_modules/prebuild-install": {
"node_modules/prebuild-install": {
"version": "7.1.3",
"version": "7.1.3",
"resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz",
"resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz",
@@ -1356,6 +1515,15 @@
@@ -1356,6 +1515,15 @@
"simple-concat": "^1.0.0"
"simple-concat": "^1.0.0"
}
}
},
},
+ "node_modules/split2": {
+ "version": "4.2.0",
+ "resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz",
+ "integrity": "sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==",
+ "license": "ISC",
+ "engines": {
+ "node": ">= 10.x"
+ }
+ },
"node_modules/statuses": {
"node_modules/statuses": {
"version": "2.0.2",
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz",
"resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz",
@@ -1483,6 +1651,15 @@
@@ -1483,6 +1651,15 @@
"resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",
"resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",
"integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==",
"integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==",
"license": "ISC"
"license": "ISC"
+ },
+ "node_modules/xtend": {
+ "version": "4.0.2",
+ "resolved": "https://registry.npmjs.org/xtend/-/xtend-4.0.2.tgz",
+ "integrity": "sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=0.4"
+ }
}
}
}
}
}
}
@@ -8,9 +8,15 @@
@@ -8,9 +8,15 @@
"start": "node server/index.js",
"start": "node server/index.js",
"dev": "NODE_ENV=development node server/index.js",
"dev": "NODE_ENV=development node server/index.js",
"test": "node test/run.js",
"test": "node test/run.js",
- "subscribe": "node scripts/subscribe.js"
+ "subscribe": "node scripts/subscribe.js",
+ "test:pg": "SEARCH_TEST_STORE=pg node test/run.js"
},
},
- "keywords": ["openvibe", "search", "fts5", "acl"],
+ "keywords": [
+ "openvibe",
+ "search",
+ "fts5",
+ "acl"
+ ],
"author": "OpenVibers",
"author": "OpenVibers",
"license": "AGPL-3.0-only",
"license": "AGPL-3.0-only",
"repository": "github:OpenVibers/OpenVibe.Search",
"repository": "github:OpenVibers/OpenVibe.Search",
@@ -20,11 +26,15 @@
@@ -20,11 +26,15 @@
"better-sqlite3": "^11.10.0",
"better-sqlite3": "^11.10.0",
"dotenv": "^16.4.7",
"dotenv": "^16.4.7",
"express": "^4.21.2",
"express": "^4.21.2",
- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.49.0",
- "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0",
- "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0"
+ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",
+ "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.20.0",
+ "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.27.0",
+ "pg": "^8.23.0"
},
},
"engines": {
"engines": {
"node": ">=22.0.0"
"node": ">=22.0.0"
+ },
+ "devDependencies": {
+ "@electric-sql/pglite": "^0.5.8"
}
}
}
}
new file mode 100644
new file mode 100644
@@ -0,0 +1,26 @@
@@ -0,0 +1,26 @@
+#!/usr/bin/env node
+'use strict';
+/**
+ * The one-time move of Search's SQLite database (SEARCH_DB_PATH) into its PostgreSQL schema (ADR-035; the procedure is
+ * openvibe-sdk docs/migrating-to-postgresql.md, section 6, carried out by openvibe-sdk/db runSqliteMigration).
+ *
+ * node scripts/migrate-to-postgres.js [--sqlite <file>] [--pglite] [--json]
+ *
+ * Applies migrations/ as the owner (DATABASE_DIRECT_URL), copies every table into emptied tables, verifies row counts
+ * and checksums, and exits 1 unless everything verified. The SQLite file is opened read-only. Every table keeps its
+ * name and columns.
+ */
+require('dotenv').config();
+const { runSqliteMigration } = require('openvibe-sdk/db');
+const configLib = require('../server/config');
+const { MIGRATIONS } = require('../server/db');
+
+const TABLES = {};
+
+if (require.main === module) {
+ const config = configLib.load();
+ runSqliteMigration({ service: 'search', sqlite: config.dbPath, directUrl: config.db.directUrl, migrations: MIGRATIONS, tables: TABLES })
+ .then((code) => process.exit(code), (err) => { console.error(`migrate-to-postgres failed: ${err.message}`); process.exit(1); });
+}
+
+module.exports = { TABLES };
@@ -64,7 +64,7 @@ function documentsRouter({ store, auth, db, relay, purges, limits }) {
@@ -64,7 +64,7 @@ function documentsRouter({ store, auth, db, relay, purges, limits }) {
return true;
return true;
}
}
- router.put('/api/v1/documents/:owner/:type/:id', guard, limits('search.document.index', index), (req, res) => {
+ router.put('/api/v1/documents/:owner/:type/:id', guard, limits('search.document.index', index), async (req, res) => {
const ctx = req.ov;
const ctx = req.ov;
if (!ownerCheck(req, res)) return;
if (!ownerCheck(req, res)) return;
const body = req.body && typeof req.body === 'object' && !Array.isArray(req.body) ? req.body : null;
const body = req.body && typeof req.body === 'object' && !Array.isArray(req.body) ? req.body : null;
@@ -77,12 +77,12 @@ function documentsRouter({ store, auth, db, relay, purges, limits }) {
@@ -77,12 +77,12 @@ function documentsRouter({ store, auth, db, relay, purges, limits }) {
const doc = { ...body, owner: req.params.owner, type: req.params.type, id: req.params.id };
const doc = { ...body, owner: req.params.owner, type: req.params.type, id: req.params.id };
const v = validate(doc);
const v = validate(doc);
if (!v.valid) return http.sendProblem(res, 422, 'search.bad_document', { detail: 'document does not match search.index-document@1', ctx, errors: v.errors });
if (!v.valid) return http.sendProblem(res, 422, 'search.bad_document', { detail: 'document does not match search.index-document@1', ctx, errors: v.errors });
- const r = store.apply(doc, { via: 'api', traceId: ctx.traceId });
+ const r = await store.apply(doc, { via: 'api', traceId: ctx.traceId });
if (r.outcome === 'applied' && relay) relay.flush().catch(() => {});
if (r.outcome === 'applied' && relay) relay.flush().catch(() => {});
return outcomeResponse(res, ctx, r);
return outcomeResponse(res, ctx, r);
});
});
- router.delete('/api/v1/documents/:owner/:type/:id', guard, limits('search.document.remove', index), (req, res) => {
+ router.delete('/api/v1/documents/:owner/:type/:id', guard, limits('search.document.remove', index), async (req, res) => {
const ctx = req.ov;
const ctx = req.ov;
if (!ownerCheck(req, res)) return;
if (!ownerCheck(req, res)) return;
let revision = null;
let revision = null;
@@ -90,25 +90,25 @@ function documentsRouter({ store, auth, db, relay, purges, limits }) {
@@ -90,25 +90,25 @@ function documentsRouter({ store, auth, db, relay, purges, limits }) {
if (!/^\d{1,15}$/.test(String(req.query.revision))) return http.sendProblem(res, 400, 'search.bad_revision', { detail: 'revision must be a non-negative integer', ctx });
if (!/^\d{1,15}$/.test(String(req.query.revision))) return http.sendProblem(res, 400, 'search.bad_revision', { detail: 'revision must be a non-negative integer', ctx });
revision = Number(req.query.revision);
revision = Number(req.query.revision);
}
}
- const r = store.remove(req.params.owner, req.params.type, req.params.id, { revision, via: 'api', traceId: ctx.traceId });
+ const r = await store.remove(req.params.owner, req.params.type, req.params.id, { revision, via: 'api', traceId: ctx.traceId });
if (r.outcome === 'applied' && relay) relay.flush().catch(() => {});
if (r.outcome === 'applied' && relay) relay.flush().catch(() => {});
return outcomeResponse(res, ctx, r);
return outcomeResponse(res, ctx, r);
});
});
- router.get('/api/v1/owners/:owner/documents', guard, limits('search.owner.documents'), (req, res) => {
+ router.get('/api/v1/owners/:owner/documents', guard, limits('search.owner.documents'), async (req, res) => {
const ctx = req.ov;
const ctx = req.ov;
if (!ownerCheck(req, res)) return;
if (!ownerCheck(req, res)) return;
const type = req.query.type ? String(req.query.type) : null;
const type = req.query.type ? String(req.query.type) : null;
if (type && !TYPE_RE.test(type)) return http.sendProblem(res, 400, 'search.bad_identity', { detail: 'type is malformed', ctx });
if (type && !TYPE_RE.test(type)) return http.sendProblem(res, 400, 'search.bad_identity', { detail: 'type is malformed', ctx });
const after = /^\d{1,15}$/.test(String(req.query.after || '')) ? Number(req.query.after) : 0;
const after = /^\d{1,15}$/.test(String(req.query.after || '')) ? Number(req.query.after) : 0;
const limit = Math.min(Math.max(parseInt(req.query.limit, 10) || 500, 1), 1000);
const limit = Math.min(Math.max(parseInt(req.query.limit, 10) || 500, 1), 1000);
- res.json({ owner: req.params.owner, ...store.ownerPage(req.params.owner, { type, after, limit }) });
+ res.json({ owner: req.params.owner, ...await store.ownerPage(req.params.owner, { type, after, limit }) });
});
});
- router.get('/api/v1/owners/:owner/documents/:type/:id', guard, limits('search.owner.document'), (req, res) => {
+ router.get('/api/v1/owners/:owner/documents/:type/:id', guard, limits('search.owner.document'), async (req, res) => {
const ctx = req.ov;
const ctx = req.ov;
if (!ownerCheck(req, res)) return;
if (!ownerCheck(req, res)) return;
- const found = store.get(req.params.owner, req.params.type, req.params.id);
+ const found = await store.get(req.params.owner, req.params.type, req.params.id);
if (!found) return http.sendProblem(res, 404, 'search.not_found', { detail: 'no such document', ctx });
if (!found) return http.sendProblem(res, 404, 'search.not_found', { detail: 'no such document', ctx });
res.json({
res.json({
document: found.doc,
document: found.doc,
@@ -120,10 +120,10 @@ function documentsRouter({ store, auth, db, relay, purges, limits }) {
@@ -120,10 +120,10 @@ function documentsRouter({ store, auth, db, relay, purges, limits }) {
});
});
});
});
- router.get('/api/v1/owners/:owner/rejections', guard, limits('search.owner.rejections'), (req, res) => {
+ router.get('/api/v1/owners/:owner/rejections', guard, limits('search.owner.rejections'), async (req, res) => {
if (!ownerCheck(req, res)) return;
if (!ownerCheck(req, res)) return;
const after = /^\d{1,15}$/.test(String(req.query.after || '')) ? Number(req.query.after) : 0;
const after = /^\d{1,15}$/.test(String(req.query.after || '')) ? Number(req.query.after) : 0;
- const rows = db.prepare('SELECT * FROM ingest_rejections WHERE owner = ? AND seq > ? ORDER BY seq LIMIT 200').all(req.params.owner, after);
+ const rows = await db.prepare('SELECT * FROM ingest_rejections WHERE owner = ? AND seq > ? ORDER BY seq LIMIT 200').all(req.params.owner, after);
res.json({
res.json({
rejections: rows.map(r => ({
rejections: rows.map(r => ({
seq: r.seq, event_id: r.event_id, event_type: r.event_type, type: r.type, id: r.id, revision: r.revision,
seq: r.seq, event_id: r.event_id, event_type: r.event_type, type: r.type, id: r.id, revision: r.revision,
@@ -133,11 +133,11 @@ function documentsRouter({ store, auth, db, relay, purges, limits }) {
@@ -133,11 +133,11 @@ function documentsRouter({ store, auth, db, relay, purges, limits }) {
});
});
});
});
- router.get('/api/v1/owners/:owner/removals', guard, limits('search.owner.removals'), (req, res) => {
+ router.get('/api/v1/owners/:owner/removals', guard, limits('search.owner.removals'), async (req, res) => {
if (!ownerCheck(req, res)) return;
if (!ownerCheck(req, res)) return;
const after = /^\d{1,15}$/.test(String(req.query.after || '')) ? Number(req.query.after) : 0;
const after = /^\d{1,15}$/.test(String(req.query.after || '')) ? Number(req.query.after) : 0;
const limit = Math.min(Math.max(parseInt(req.query.limit, 10) || 200, 1), 1000);
const limit = Math.min(Math.max(parseInt(req.query.limit, 10) || 200, 1), 1000);
- res.json({ owner: req.params.owner, ...purges.ownerFeed(req.params.owner, { after, limit }) });
+ res.json({ owner: req.params.owner, ...await purges.ownerFeed(req.params.owner, { after, limit }) });
});
});
return router;
return router;
@@ -19,7 +19,7 @@
@@ -19,7 +19,7 @@
const crypto = require('crypto');
const crypto = require('crypto');
const express = require('express');
const express = require('express');
const { http } = require('openvibe-contracts');
const { http } = require('openvibe-contracts');
-const { toMatch, snippetHtml } = require('../engine/fts5');
+const { toMatch, snippetHtml } = require('../engine/pg');
const { EXPOSURE } = require('../document');
const { EXPOSURE } = require('../document');
const { AuthError } = require('../auth');
const { AuthError } = require('../auth');
const { OWNER_RE, TYPE_RE, ID_RE } = require('./documents');
const { OWNER_RE, TYPE_RE, ID_RE } = require('./documents');
@@ -119,7 +119,7 @@ function canView(viewer, row, doc) {
@@ -119,7 +119,7 @@ function canView(viewer, row, doc) {
* QueryError for a malformed request.
* QueryError for a malformed request.
*/
*/
function createSearcher({ config, store, engine, now: clock = () => Date.now() }) {
function createSearcher({ config, store, engine, now: clock = () => Date.now() }) {
- function run({ text = '', query = {}, filters = null, viewer, limit, cursor, facetKeys = [], now = clock() }) {
+ async function run({ text = '', query = {}, filters = null, viewer, limit, cursor, facetKeys = [], now = clock() }) {
if (text.length > 500) throw new QueryError('search.bad_query', 'q is longer than 500 characters');
if (text.length > 500) throw new QueryError('search.bad_query', 'q is longer than 500 characters');
filters = filters || parseFilters(query, config);
filters = filters || parseFilters(query, config);
limit = Math.min(Math.max(parseInt(limit, 10) || config.query.defaultLimit, 1), config.query.maxLimit);
limit = Math.min(Math.max(parseInt(limit, 10) || config.query.defaultLimit, 1), config.query.maxLimit);
@@ -136,14 +136,14 @@ function createSearcher({ config, store, engine, now: clock = () => Date.now() }
@@ -136,14 +136,14 @@ function createSearcher({ config, store, engine, now: clock = () => Date.now() }
if (match) {
if (match) {
if (after && after.rank === undefined) throw new QueryError('search.bad_cursor', 'cursor does not belong to this query');
if (after && after.rank === undefined) throw new QueryError('search.bad_cursor', 'cursor does not belong to this query');
if (after && after.now != null) refNow = after.now;
if (after && after.now != null) refNow = after.now;
- page = engine.search({ match, filters, viewer, limit, after, now: refNow });
+ page = await engine.search({ match, filters, viewer, limit, after, now: refNow });
} else {
} else {
if (after && after.sort_at === undefined) throw new QueryError('search.bad_cursor', 'cursor does not belong to this query');
if (after && after.sort_at === undefined) throw new QueryError('search.bad_cursor', 'cursor does not belong to this query');
- page = engine.browse({ filters, viewer, limit, after });
+ page = await engine.browse({ filters, viewer, limit, after });
}
}
const results = [];
const results = [];
for (const h of page.hits) {
for (const h of page.hits) {
- const found = store.byRid(h.rid);
+ const found = await store.byRid(h.rid);
// Defence in depth: the engine already filtered; re-check against the stored row.
// Defence in depth: the engine already filtered; re-check against the stored row.
if (!found || !canView(viewer, found.row, found.doc)) continue;
if (!found || !canView(viewer, found.row, found.doc)) continue;
results.push(projection(found.doc, found.row, match ? h.snip : undefined));
results.push(projection(found.doc, found.row, match ? h.snip : undefined));
@@ -152,7 +152,7 @@ function createSearcher({ config, store, engine, now: clock = () => Date.now() }
@@ -152,7 +152,7 @@ function createSearcher({ config, store, engine, now: clock = () => Date.now() }
? encodeCursor(match ? { k: 'r', r: page.next.rank, i: page.next.rid, n: refNow, h: hash } : { k: 't', t: page.next.sort_at, i: page.next.rid, h: hash })
? encodeCursor(match ? { k: 'r', r: page.next.rank, i: page.next.rid, n: refNow, h: hash } : { k: 't', t: page.next.sort_at, i: page.next.rid, h: hash })
: null;
: null;
const body = { results, next_cursor: next };
const body = { results, next_cursor: next };
- if (facetKeys.length) body.facets = engine.facetCounts({ match, filters, viewer, keys: facetKeys });
+ if (facetKeys.length) body.facets = await engine.facetCounts({ match, filters, viewer, keys: facetKeys });
return body;
return body;
}
}
@@ -168,7 +168,7 @@ function createSearcher({ config, store, engine, now: clock = () => Date.now() }
@@ -168,7 +168,7 @@ function createSearcher({ config, store, engine, now: clock = () => Date.now() }
* (a per-actor limit, server/actor-limits.js) the viewer is counted as req.viewer before the handler runs.
* (a per-actor limit, server/actor-limits.js) the viewer is counted as req.viewer before the handler runs.
*/
*/
function withViewer(auth, handler, limit = null) {
function withViewer(auth, handler, limit = null) {
- return (req, res, next) => {
+ return async (req, res, next) => {
const ctx = req.ov;
const ctx = req.ov;
let viewer;
let viewer;
try {
try {
@@ -181,16 +181,16 @@ function withViewer(auth, handler, limit = null) {
@@ -181,16 +181,16 @@ function withViewer(auth, handler, limit = null) {
// personalised answer must never reach a shared cache.
// personalised answer must never reach a shared cache.
res.setHeader('Cache-Control', 'no-store');
res.setHeader('Cache-Control', 'no-store');
res.setHeader('Vary', 'Authorization, Cookie, X-OV-Subject, X-OV-Groups, X-OV-Entitlements');
res.setHeader('Vary', 'Authorization, Cookie, X-OV-Subject, X-OV-Groups, X-OV-Entitlements');
- const run = () => {
+ const run = async () => {
try {
try {
- return handler(req, res, viewer);
+ return await handler(req, res, viewer);
} catch (err) {
} catch (err) {
if (err instanceof QueryError) return http.sendProblem(res, 400, err.code, { detail: err.message, ctx });
if (err instanceof QueryError) return http.sendProblem(res, 400, err.code, { detail: err.message, ctx });
return next(err);
return next(err);
}
}
};
};
req.viewer = viewer;
req.viewer = viewer;
- return limit ? limit(req, res, run) : run();
+ return limit ? limit(req, res, run) : await run();
};
};
}
}
@@ -200,9 +200,9 @@ function queryRouter({ config, store, engine, auth, limits, searcher = createSea
@@ -200,9 +200,9 @@ function queryRouter({ config, store, engine, auth, limits, searcher = createSea
// Per-actor limits (server/actor-limits.js): queries and document reads take the defaults. Suggest
// Per-actor limits (server/actor-limits.js): queries and document reads take the defaults. Suggest
// answers a search box as the person types, several requests a second, so it allows 300 a minute.
// answers a search box as the person types, several requests a second, so it allows 300 a minute.
- router.get('/api/v1/search', guarded(limits('search.query'), (req, res, viewer) => {
+ router.get('/api/v1/search', guarded(limits('search.query'), async (req, res, viewer) => {
const q = one(req.query.q);
const q = one(req.query.q);
- res.json(searcher.run({
+ res.json(await searcher.run({
text: q === undefined ? '' : String(q),
text: q === undefined ? '' : String(q),
query: req.query,
query: req.query,
viewer,
viewer,
@@ -212,27 +212,27 @@ function queryRouter({ config, store, engine, auth, limits, searcher = createSea
@@ -212,27 +212,27 @@ function queryRouter({ config, store, engine, auth, limits, searcher = createSea
}));
}));
}));
}));
- router.get('/api/v1/suggest', guarded(limits('search.suggest', { minute: 300, hour: 6000 }), (req, res, viewer) => {
+ router.get('/api/v1/suggest', guarded(limits('search.suggest', { minute: 300, hour: 6000 }), async (req, res, viewer) => {
const text = String(one(req.query.q) || '');
const text = String(one(req.query.q) || '');
if (text.length > 100) throw new QueryError('search.bad_query', 'q is longer than 100 characters');
if (text.length > 100) throw new QueryError('search.bad_query', 'q is longer than 100 characters');
const filters = parseFilters(req.query, config);
const filters = parseFilters(req.query, config);
const limit = Math.min(Math.max(parseInt(one(req.query.limit), 10) || 8, 1), 20);
const limit = Math.min(Math.max(parseInt(one(req.query.limit), 10) || 8, 1), 20);
const match = toMatch(text, { maxTerms: 6, column: 'title', prefixLast: true });
const match = toMatch(text, { maxTerms: 6, column: 'title', prefixLast: true });
if (!match) return res.json({ suggestions: [] });
if (!match) return res.json({ suggestions: [] });
- const rows = engine.suggest({ match, filters, viewer, limit });
+ const rows = await engine.suggest({ match, filters, viewer, limit });
const suggestions = [];
const suggestions = [];
for (const r of rows) {
for (const r of rows) {
- const found = store.byRid(r.rid);
+ const found = await store.byRid(r.rid);
if (!found || !canView(viewer, found.row, found.doc)) continue;
if (!found || !canView(viewer, found.row, found.doc)) continue;
suggestions.push({ owner: found.doc.owner, type: found.doc.type, id: found.doc.id, title: found.doc.title, canonical_url: found.doc.canonical_url });
suggestions.push({ owner: found.doc.owner, type: found.doc.type, id: found.doc.id, title: found.doc.title, canonical_url: found.doc.canonical_url });
}
}
res.json({ suggestions });
res.json({ suggestions });
}));
}));
- router.get('/api/v1/documents/:owner/:type/:id', guarded(limits('search.document.read'), (req, res, viewer) => {
+ router.get('/api/v1/documents/:owner/:type/:id', guarded(limits('search.document.read'), async (req, res, viewer) => {
const { owner, type, id } = req.params;
const { owner, type, id } = req.params;
if (!OWNER_RE.test(owner) || !TYPE_RE.test(type) || !ID_RE.test(id)) throw new QueryError('search.bad_identity', 'owner, type or id is malformed');
if (!OWNER_RE.test(owner) || !TYPE_RE.test(type) || !ID_RE.test(id)) throw new QueryError('search.bad_identity', 'owner, type or id is malformed');
- const found = store.get(owner, type, id);
+ const found = await store.get(owner, type, id);
if (!found || !canView(viewer, found.row, found.doc)) {
if (!found || !canView(viewer, found.row, found.doc)) {
return http.sendProblem(res, 404, 'search.not_found', { detail: 'no such document', ctx: req.ov });
return http.sendProblem(res, 404, 'search.not_found', { detail: 'no such document', ctx: req.ov });
}
}
@@ -53,13 +53,13 @@ function savedRouter({ config, saved, searcher, auth, limits }) {
@@ -53,13 +53,13 @@ function savedRouter({ config, saved, searcher, auth, limits }) {
const guarded = (limit, h) => withViewer(auth, h, limit);
const guarded = (limit, h) => withViewer(auth, h, limit);
const change = { minute: 30, hour: 300 };
const change = { minute: 30, hour: 300 };
- router.get('/api/v1/saved-searches', guarded(limits('search.saved.list'), (req, res, viewer) => {
+ router.get('/api/v1/saved-searches', guarded(limits('search.saved.list'), async (req, res, viewer) => {
const subject = person(req, res, viewer);
const subject = person(req, res, viewer);
if (!subject) return;
if (!subject) return;
- res.json({ saved_searches: saved.list(subject), max: saved.maxPerSubject });
+ res.json({ saved_searches: await saved.list(subject), max: saved.maxPerSubject });
}));
}));
- router.post('/api/v1/saved-searches', guarded(limits('search.saved.create', change), (req, res, viewer) => {
+ router.post('/api/v1/saved-searches', guarded(limits('search.saved.create', change), async (req, res, viewer) => {
const ctx = req.ov;
const ctx = req.ov;
const subject = person(req, res, viewer, { write: true });
const subject = person(req, res, viewer, { write: true });
if (!subject) return;
if (!subject) return;
@@ -89,39 +89,39 @@ function savedRouter({ config, saved, searcher, auth, limits }) {
@@ -89,39 +89,39 @@ function savedRouter({ config, saved, searcher, auth, limits }) {
const text = q.trim();
const text = q.trim();
if (text && !searcher.hasWords(text)) throw new QueryError('search.bad_query', 'q has no searchable words');
if (text && !searcher.hasWords(text)) throw new QueryError('search.bad_query', 'q has no searchable words');
if (!text && !hasFilter) throw new QueryError('search.bad_saved_search', 'a saved search needs words (q) or at least one filter');
if (!text && !hasFilter) throw new QueryError('search.bad_saved_search', 'a saved search needs words (q) or at least one filter');
- const r = saved.save(subject, { name: name.trim(), q: text, filters });
+ const r = await saved.save(subject, { name: name.trim(), q: text, filters });
if (r.error === 'limit') return http.sendProblem(res, 409, 'search.saved_search_limit', { detail: `at most ${saved.maxPerSubject} saved searches per person`, ctx });
if (r.error === 'limit') return http.sendProblem(res, 409, 'search.saved_search_limit', { detail: `at most ${saved.maxPerSubject} saved searches per person`, ctx });
res.status(r.created ? 201 : 200).json({ saved_search: r.saved });
res.status(r.created ? 201 : 200).json({ saved_search: r.saved });
}));
}));
- router.get('/api/v1/saved-searches/:id', guarded(limits('search.saved.read'), (req, res, viewer) => {
+ router.get('/api/v1/saved-searches/:id', guarded(limits('search.saved.read'), async (req, res, viewer) => {
const subject = person(req, res, viewer);
const subject = person(req, res, viewer);
if (!subject) return;
if (!subject) return;
- const s = saved.get(subject, req.params.id);
+ const s = await saved.get(subject, req.params.id);
if (!s) return http.sendProblem(res, 404, 'search.not_found', { detail: 'no such saved search', ctx: req.ov });
if (!s) return http.sendProblem(res, 404, 'search.not_found', { detail: 'no such saved search', ctx: req.ov });
res.json({ saved_search: s });
res.json({ saved_search: s });
}));
}));
- router.delete('/api/v1/saved-searches/:id', guarded(limits('search.saved.delete', change), (req, res, viewer) => {
+ router.delete('/api/v1/saved-searches/:id', guarded(limits('search.saved.delete', change), async (req, res, viewer) => {
const subject = person(req, res, viewer, { write: true });
const subject = person(req, res, viewer, { write: true });
if (!subject) return;
if (!subject) return;
- if (!saved.remove(subject, req.params.id)) return http.sendProblem(res, 404, 'search.not_found', { detail: 'no such saved search', ctx: req.ov });
+ if (!await saved.remove(subject, req.params.id)) return http.sendProblem(res, 404, 'search.not_found', { detail: 'no such saved search', ctx: req.ov });
res.status(204).end();
res.status(204).end();
}));
}));
- router.get('/api/v1/saved-searches/:id/results', guarded(limits('search.saved.run'), (req, res, viewer) => {
+ router.get('/api/v1/saved-searches/:id/results', guarded(limits('search.saved.run'), async (req, res, viewer) => {
const subject = person(req, res, viewer);
const subject = person(req, res, viewer);
if (!subject) return;
if (!subject) return;
- const s = saved.get(subject, req.params.id);
+ const s = await saved.get(subject, req.params.id);
if (!s) return http.sendProblem(res, 404, 'search.not_found', { detail: 'no such saved search', ctx: req.ov });
if (!s) return http.sendProblem(res, 404, 'search.not_found', { detail: 'no such saved search', ctx: req.ov });
- const body = searcher.run({
+ const body = await searcher.run({
text: s.q,
text: s.q,
filters: { ...s.filters, facets: s.filters.facets || [] },
filters: { ...s.filters, facets: s.filters.facets || [] },
viewer,
viewer,
limit: one(req.query.limit),
limit: one(req.query.limit),
cursor: one(req.query.cursor),
cursor: one(req.query.cursor),
});
});
- saved.markRun(s.id);
+ await saved.markRun(s.id);
res.json({ saved_search: { id: s.id, name: s.name }, ...body });
res.json({ saved_search: { id: s.id, name: s.name }, ...body });
}));
}));
@@ -73,23 +73,23 @@ function verifySignature(raw, header, secrets) {
@@ -73,23 +73,23 @@ function verifySignature(raw, header, secrets) {
}
}
function createInbox(db, { now = () => Date.now() } = {}) {
function createInbox(db, { now = () => Date.now() } = {}) {
- const claim = db.prepare('INSERT OR IGNORE INTO idempotency_receipts (consumer, event_id, outcome, processed_at) VALUES (?, ?, ?, ?)');
+ const claim = db.prepare('INSERT INTO idempotency_receipts (consumer, event_id, outcome, processed_at) VALUES (?, ?, ?, ?) ON CONFLICT DO NOTHING');
const setOutcome = db.prepare('UPDATE idempotency_receipts SET outcome = ? WHERE consumer = ? AND event_id = ?');
const setOutcome = db.prepare('UPDATE idempotency_receipts SET outcome = ? WHERE consumer = ? AND event_id = ?');
const reject = db.prepare(`INSERT INTO ingest_rejections (event_id, event_type, owner, type, id, revision, code, detail, at)
const reject = db.prepare(`INSERT INTO ingest_rejections (event_id, event_type, owner, type, id, revision, code, detail, at)
VALUES (@event_id, @event_type, @owner, @type, @id, @revision, @code, @detail, @at)`);
VALUES (@event_id, @event_type, @owner, @type, @id, @revision, @code, @detail, @at)`);
/** fn() runs inside the transaction and returns an outcome string. */
/** fn() runs inside the transaction and returns an outcome string. */
- function once(eventId, fn) {
- return db.transaction(() => {
- if (claim.run(CONSUMER, eventId, 'processing', now()).changes === 0) return { duplicate: true };
- const outcome = fn();
- setOutcome.run(outcome, CONSUMER, eventId);
+ async function once(eventId, fn) {
+ return await db.tx(async () => {
+ if ((await claim.run(CONSUMER, eventId, 'processing', now())).changes === 0) return { duplicate: true };
+ const outcome = await fn();
+ await setOutcome.run(outcome, CONSUMER, eventId);
return { duplicate: false, outcome };
return { duplicate: false, outcome };
- })();
+ });
}
}
- function recordRejection(r) {
- reject.run({ owner: null, type: null, id: null, revision: null, detail: null, event_type: null, ...r, at: now() });
+ async function recordRejection(r) {
+ await reject.run({ owner: null, type: null, id: null, revision: null, detail: null, event_type: null, ...r, at: now() });
}
}
return { once, recordRejection };
return { once, recordRejection };
@@ -125,7 +125,7 @@ function webhookRouter({ config, db, store, relay, log = console, now }) {
@@ -125,7 +125,7 @@ function webhookRouter({ config, db, store, relay, log = console, now }) {
router.post('/internal/events',
router.post('/internal/events',
express.raw({ type: () => true, limit: config.maxBodyBytes }),
express.raw({ type: () => true, limit: config.maxBodyBytes }),
- (req, res) => {
+ async (req, res, next) => { try {
const ctx = req.ov;
const ctx = req.ov;
const secrets = config.events.webhookSecrets;
const secrets = config.events.webhookSecrets;
if (!secrets.length) return http.sendProblem(res, 503, 'search.webhook_disabled', { detail: 'SEARCH_EVENTS_SECRET is not set', ctx });
if (!secrets.length) return http.sendProblem(res, 503, 'search.webhook_disabled', { detail: 'SEARCH_EVENTS_SECRET is not set', ctx });
@@ -141,12 +141,12 @@ function webhookRouter({ config, db, store, relay, log = console, now }) {
@@ -141,12 +141,12 @@ function webhookRouter({ config, db, store, relay, log = console, now }) {
return http.sendProblem(res, 400, 'search.bad_delivery', { detail: 'body must be { event: <envelope>, seq }', ctx });
return http.sendProblem(res, 400, 'search.bad_delivery', { detail: 'body must be { event: <envelope>, seq }', ctx });
}
}
- const r = inbox.once(event.event_id, () => {
+ const r = await inbox.once(event.event_id, async () => {
const parsed = documentFromEvent(event, config.events.owners);
const parsed = documentFromEvent(event, config.events.owners);
if (parsed.ignore) return 'ignored';
if (parsed.ignore) return 'ignored';
if (parsed.reject) {
if (parsed.reject) {
const p = event.payload || {};
const p = event.payload || {};
- inbox.recordRejection({
+ await inbox.recordRejection({
event_id: event.event_id, event_type: event.event_type, owner: event.source,
event_id: event.event_id, event_type: event.event_type, owner: event.source,
type: typeof p.type === 'string' ? p.type.slice(0, 40) : null,
type: typeof p.type === 'string' ? p.type.slice(0, 40) : null,
id: typeof p.id === 'string' ? p.id.slice(0, 128) : null,
id: typeof p.id === 'string' ? p.id.slice(0, 128) : null,
@@ -157,12 +157,12 @@ function webhookRouter({ config, db, store, relay, log = console, now }) {
@@ -157,12 +157,12 @@ function webhookRouter({ config, db, store, relay, log = console, now }) {
return `rejected:${parsed.reject.code}`;
return `rejected:${parsed.reject.code}`;
}
}
const traceId = typeof event.trace_id === 'string' ? event.trace_id : null;
const traceId = typeof event.trace_id === 'string' ? event.trace_id : null;
- const out = store.apply(parsed.doc, { via: 'event', eventId: event.event_id, traceId });
+ const out = await store.apply(parsed.doc, { via: 'event', eventId: event.event_id, traceId });
return out.outcome;
return out.outcome;
});
});
if (!r.duplicate && r.outcome === 'applied' && relay) relay.flush().catch(() => {});
if (!r.duplicate && r.outcome === 'applied' && relay) relay.flush().catch(() => {});
res.status(200).json({ event_id: event.event_id, duplicate: Boolean(r.duplicate), outcome: r.outcome || null });
res.status(200).json({ event_id: event.event_id, duplicate: Boolean(r.duplicate), outcome: r.outcome || null });
- });
+ } catch (err) { return next(err); } }); // Express 4 does not catch an async handler's rejection
return router;
return router;
}
}
@@ -23,7 +23,7 @@ function createApp({ config, db, store, engine, auth, keys, outbox, relay, purge
@@ -23,7 +23,7 @@ function createApp({ config, db, store, engine, auth, keys, outbox, relay, purge
// HTTP golden signals by route template, process metrics, release_info and the Search gauges;
// HTTP golden signals by route template, process metrics, release_info and the Search gauges;
// GET /metrics answers direct loopback callers only (Track O).
// GET /metrics answers direct loopback callers only (Track O).
const metrics = instrument(app, { service: 'search', release: release.release });
const metrics = instrument(app, { service: 'search', release: release.release });
- registerSearchGauges(metrics.registry, { db, outbox, purges });
+ registerSearchGauges(metrics.registry, { db, engine, outbox, purges });
app.use(http.middleware());
app.use(http.middleware());
// One W3C trace across services (openvibe-shared/trace): calls made while serving a request carry its traceparent.
// One W3C trace across services (openvibe-shared/trace): calls made while serving a request carry its traceparent.
require('openvibe-shared/trace').install(app);
require('openvibe-shared/trace').install(app);
@@ -61,7 +61,7 @@ function createKeyStore({ urls = [], pem = null, fetchImpl = globalThis.fetch, l
@@ -61,7 +61,7 @@ function createKeyStore({ urls = [], pem = null, fetchImpl = globalThis.fetch, l
return null;
return null;
}
}
- function start() {
+ async function start() {
if (pem) return Promise.resolve(key);
if (pem) return Promise.resolve(key);
const attempt = async () => {
const attempt = async () => {
const k = await fetchOnce();
const k = await fetchOnce();
@@ -73,7 +73,7 @@ function createKeyStore({ urls = [], pem = null, fetchImpl = globalThis.fetch, l
@@ -73,7 +73,7 @@ function createKeyStore({ urls = [], pem = null, fetchImpl = globalThis.fetch, l
};
};
refreshTimer = setInterval(() => { fetchOnce().catch(() => {}); }, 6 * 60 * 60 * 1000);
refreshTimer = setInterval(() => { fetchOnce().catch(() => {}); }, 6 * 60 * 60 * 1000);
refreshTimer.unref?.();
refreshTimer.unref?.();
- return attempt();
+ return await attempt();
}
}
function stop() {
function stop() {
@@ -70,6 +70,9 @@ function load(env = process.env) {
@@ -70,6 +70,9 @@ function load(env = process.env) {
clientSecret: env.OV_OAUTH_CLIENT_SECRET || '',
clientSecret: env.OV_OAUTH_CLIENT_SECRET || '',
},
},
+ // PostgreSQL (ADR-035): DATABASE_URL serves (PgBouncer), DATABASE_DIRECT_URL migrates (owner role).
+ db: { url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '' },
+ // The SQLite file of releases before the switch: read once by scripts/migrate-to-postgres.js.
dbPath: env.SEARCH_DB_PATH || './data/search.db',
dbPath: env.SEARCH_DB_PATH || './data/search.db',
// Events: the outbox relays only when EVENTS_URL is set (rows wait otherwise).
// Events: the outbox relays only when EVENTS_URL is set (rows wait otherwise).
@@ -1,99 +1,32 @@
@@ -1,99 +1,32 @@
'use strict';
'use strict';
/**
/**
- * The Search database (SQLite, WAL). One file owned by this service only; created on boot,
- * idempotently. The FTS5 tables belong to the engine (server/engine/fts5.js).
+ * The Search database: PostgreSQL (ADR-035), owned by this service only; the schema is migrations/NNNN_*.sql, applied
+ * at boot. The full-text table search_fts belongs to the engine (server/engine/pg.js).
*/
*/
const fs = require('fs');
const fs = require('fs');
const path = require('path');
const path = require('path');
-const Database = require('better-sqlite3');
-const outbox = require('./events/outbox');
-const purge = require('./purge');
-const saved = require('./saved');
+const { createDb } = require('openvibe-sdk/db');
-const SCHEMA = `
-CREATE TABLE IF NOT EXISTS documents (
- rid INTEGER PRIMARY KEY,
- owner TEXT NOT NULL,
- type TEXT NOT NULL,
- id TEXT NOT NULL,
- revision INTEGER NOT NULL,
- deleted INTEGER NOT NULL DEFAULT 0,
- visibility TEXT,
- publication_state TEXT,
- exposure INTEGER NOT NULL DEFAULT 0, -- document.js EXPOSURE (0 none .. 3 public listed)
- index_decision TEXT NOT NULL,
- noindex_reasons TEXT NOT NULL DEFAULT '[]',
- canonical_url TEXT,
- title TEXT,
- language TEXT,
- published_at TEXT,
- updated_at TEXT,
- sort_at TEXT NOT NULL DEFAULT '', -- updated_at, else published_at: browse order
- doc TEXT NOT NULL, -- normalized document JSON (tombstone: identity only)
- hash TEXT NOT NULL,
- via TEXT NOT NULL, -- event | api
- event_id TEXT,
- indexed_at INTEGER NOT NULL,
- UNIQUE (owner, type, id)
-);
-CREATE INDEX IF NOT EXISTS idx_documents_exposure ON documents (exposure, sort_at, rid);
-CREATE INDEX IF NOT EXISTS idx_documents_owner ON documents (owner, type, id);
+const MIGRATIONS = path.join(__dirname, '..', 'migrations');
+const DEV_PGLITE = path.join(__dirname, '..', 'data', 'pglite');
-CREATE TABLE IF NOT EXISTS doc_acl (
- rid INTEGER NOT NULL,
- kind TEXT NOT NULL CHECK (kind IN ('s', 'g', 'e')),
- value TEXT NOT NULL,
- PRIMARY KEY (rid, kind, value)
-) WITHOUT ROWID;
-CREATE INDEX IF NOT EXISTS idx_doc_acl_value ON doc_acl (kind, value, rid);
-
-CREATE TABLE IF NOT EXISTS doc_facets (
- rid INTEGER NOT NULL,
- key TEXT NOT NULL,
- value TEXT NOT NULL,
- PRIMARY KEY (rid, key, value)
-) WITHOUT ROWID;
-CREATE INDEX IF NOT EXISTS idx_doc_facets_value ON doc_facets (key, value, rid);
-
-CREATE TABLE IF NOT EXISTS idempotency_receipts (
- consumer TEXT NOT NULL,
- event_id TEXT NOT NULL,
- outcome TEXT NOT NULL,
- processed_at INTEGER NOT NULL,
- PRIMARY KEY (consumer, event_id)
-);
-
-CREATE TABLE IF NOT EXISTS ingest_rejections (
- seq INTEGER PRIMARY KEY AUTOINCREMENT,
- event_id TEXT NOT NULL,
- event_type TEXT,
- owner TEXT,
- type TEXT,
- id TEXT,
- revision INTEGER,
- code TEXT NOT NULL,
- detail TEXT,
- at INTEGER NOT NULL
-);
-CREATE INDEX IF NOT EXISTS idx_ingest_rejections_owner ON ingest_rejections (owner, seq);
-`;
-
-function openDb(dbPath) {
- if (dbPath !== ':memory:') fs.mkdirSync(path.dirname(path.resolve(dbPath)), { recursive: true });
- const db = new Database(dbPath);
- db.pragma('journal_mode = WAL');
- db.pragma('synchronous = NORMAL');
- db.pragma('foreign_keys = ON');
- db.pragma('busy_timeout = 5000');
- db.exec(SCHEMA);
- outbox.ensureSchema(db);
- purge.ensureSchema(db);
- saved.ensureSchema(db);
- return db;
+/**
+ * The serving handle (ADR-035): DATABASE_URL through PgBouncer; in development without it, an embedded PGlite database
+ * in data/pglite. Migrations run first, as the owner (DATABASE_DIRECT_URL), or on the embedded handle.
+ */
+async function openDb(config, { log = console, registry } = {}) {
+ if (!config.db.url) {
+ if (config.isProduction) throw new Error('DATABASE_URL is not set: production serves from PostgreSQL (OpenVibe.Host roles/data add-service.sh search)');
+ log.warn(`[search] DATABASE_URL unset: embedded PGlite database in ${DEV_PGLITE} (development only, one process)`);
+ fs.mkdirSync(DEV_PGLITE, { recursive: true });
+ const db = createDb({ pglite: DEV_PGLITE, service: 'search', registry, log });
+ await db.migrate({ dir: MIGRATIONS, log });
+ return db;
+ }
+ if (!config.db.directUrl) throw new Error('DATABASE_DIRECT_URL is not set: migrations run with the owner role on a direct connection');
+ const owner = createDb({ url: config.db.directUrl, service: 'search-migrate', max: 1, log });
+ try { await owner.migrate({ dir: MIGRATIONS, log }); } finally { await owner.close(); }
+ return createDb({ url: config.db.url, service: 'search', registry, log });
}
}
-module.exports = { openDb };
+module.exports = { openDb, MIGRATIONS };
deleted file mode 100644
deleted file mode 100644
@@ -1,235 +0,0 @@
@@ -1,235 +0,0 @@
-'use strict';
-/**
- * Search engine: SQLite FTS5 in the service's own database (decision: docs/adr-engine.md).
- *
- * Two full-text tables, split by audience so an anonymous query never touches, ranks against or
- * counts anything but public listed documents:
- *
- * fts_public documents with exposure public_listed (public, published, indexable)
- * fts_restricted documents with exposure restricted (unlisted/members/private, published)
- *
- * Drafts, unpublished, deleted and public-noindex documents are in neither, so no MATCH can
- * reach them. Every query is also filtered by `documents.exposure` and, for restricted rows, by
- * the viewer's ACL keys (doc_acl) in the same SQL: visibility is decided before anything is
- * ranked, counted, snippeted or suggested.
- *
- * The interface (put/remove/search/browse/facetCounts/suggest) is what a PostgreSQL FTS engine
- * would implement; nothing outside this file writes SQL against the FTS tables.
- */
-const { EXPOSURE } = require('../document');
-
-const TOKENIZE = "tokenize = 'unicode61 remove_diacritics 2'";
-const WEIGHTS = '10.0, 4.0, 1.0'; // title, summary, body
-const MARK_OPEN = '\u0001';
-const MARK_CLOSE = '\u0002';
-
-const DAY_MS = 24 * 3600 * 1000;
-
-/**
- * Freshness multiplier for a document dated `iso` at reference time `nowMs`:
- *
- * 1 + weight × 2^(−age_days / halfLifeDays)
- *
- * With the defaults (weight 1, half-life 30 days) a document published today counts ×2.0, one
- * 30 days old ×1.5, 60 days ×1.25, 90 days ×1.125, a year old ×1.0002: relevance still decides,
- * and between comparably relevant documents the newer one ranks first. A future date counts as
- * today; no date (or an unparsable one) counts ×1. Weight 0 turns the boost off.
- */
-function freshnessBoost(iso, nowMs, halfLifeDays, weight) {
- if (!weight || !iso) return 1;
- const t = Date.parse(iso);
- if (!Number.isFinite(t)) return 1;
- const ageDays = Math.max(0, (nowMs - t) / DAY_MS);
- return 1 + weight * Math.pow(2, -ageDays / halfLifeDays);
-}
-
-function createEngine(db, { freshness = { weight: 1, halfLifeDays: 30 } } = {}) {
- // bm25() is negative (lower = more relevant), so multiplying by a boost ≥ 1 ranks newer first.
- db.function('ov_freshness', { deterministic: true }, (iso, nowMs, halfLifeDays, weight) => freshnessBoost(iso, nowMs, halfLifeDays, weight));
- db.exec(`
- CREATE VIRTUAL TABLE IF NOT EXISTS fts_public USING fts5(title, summary, body, ${TOKENIZE});
- CREATE VIRTUAL TABLE IF NOT EXISTS fts_restricted USING fts5(title, summary, body, ${TOKENIZE});
- `);
- const st = {
- delPublic: db.prepare('DELETE FROM fts_public WHERE rowid = ?'),
- delRestricted: db.prepare('DELETE FROM fts_restricted WHERE rowid = ?'),
- insPublic: db.prepare('INSERT INTO fts_public (rowid, title, summary, body) VALUES (?, ?, ?, ?)'),
- insRestricted: db.prepare('INSERT INTO fts_restricted (rowid, title, summary, body) VALUES (?, ?, ?, ?)'),
- };
-
- /** (Re)index one document according to its exposure. Call inside the store's transaction. */
- function put(rid, doc, exposure) {
- remove(rid);
- if (exposure === EXPOSURE.public_listed) st.insPublic.run(rid, doc.title || '', doc.summary || '', doc.body || '');
- else if (exposure === EXPOSURE.restricted) st.insRestricted.run(rid, doc.title || '', doc.summary || '', doc.body || '');
- }
-
- function remove(rid) {
- st.delPublic.run(rid);
- st.delRestricted.run(rid);
- }
-
- // ── SQL building ─────────────────────────────────────────
-
- /**
- * The ACL predicate for restricted rows (alias d). private: the viewer's subject is in
- * acl.subjects. members/unlisted: subject, group or entitlement match. Returns null when the
- * viewer has no subject (then the restricted branch is not queried at all).
- */
- function aclClause(viewer, params) {
- if (!viewer || !viewer.subject) return null;
- params.v_subject = viewer.subject;
- params.v_groups = JSON.stringify(viewer.groups || []);
- params.v_ents = JSON.stringify(viewer.entitlements || []);
- return `(
- (d.visibility = 'private' AND EXISTS (SELECT 1 FROM doc_acl a WHERE a.rid = d.rid AND a.kind = 's' AND a.value = @v_subject))
- OR (d.visibility IN ('members', 'unlisted') AND EXISTS (SELECT 1 FROM doc_acl a WHERE a.rid = d.rid AND (
- (a.kind = 's' AND a.value = @v_subject)
- OR (a.kind = 'g' AND a.value IN (SELECT value FROM json_each(@v_groups)))
- OR (a.kind = 'e' AND a.value IN (SELECT value FROM json_each(@v_ents))))))
- )`;
- }
-
- function filterClause(filters, params) {
- const parts = [];
- if (filters.owner) { parts.push('d.owner = @f_owner'); params.f_owner = filters.owner; }
- if (filters.type) { parts.push('d.type = @f_type'); params.f_type = filters.type; }
- if (filters.language) { parts.push('d.language = @f_lang'); params.f_lang = filters.language; }
- (filters.facets || []).forEach(([key, values], i) => {
- params[`f_fk${i}`] = key;
- params[`f_fv${i}`] = JSON.stringify(values);
- parts.push(`EXISTS (SELECT 1 FROM doc_facets x WHERE x.rid = d.rid AND x.key = @f_fk${i} AND x.value IN (SELECT value FROM json_each(@f_fv${i})))`);
- });
- return parts.length ? ` AND ${parts.join(' AND ')}` : '';
- }
-
- /**
- * The visible, matching set as (rid, rank, snip) rows. match = FTS5 expression. rank is bm25
- * times the freshness boost at reference time `now` (ms), so pages of one query (which carry
- * their first page's `now` in the cursor) rank against the same clock.
- */
- function matchedSet({ match, filters, viewer, snippets = true, now = Date.now() }, params) {
- params.match = match;
- params.r_now = now;
- params.r_half = freshness.halfLifeDays;
- params.r_weight = freshness.weight;
- const where = filterClause(filters, params);
- const snip = (t) => (snippets ? `snippet(${t}, -1, char(1), char(2), '…', 16)` : 'NULL');
- const rank = (t) => `bm25(${t}, ${WEIGHTS}) * ov_freshness(COALESCE(d.published_at, d.updated_at), @r_now, @r_half, @r_weight)`;
- const branches = [`SELECT d.rid AS rid, ${rank('fts_public')} AS rank, ${snip('fts_public')} AS snip
- FROM fts_public JOIN documents d ON d.rid = fts_public.rowid
- WHERE fts_public MATCH @match AND d.exposure = ${EXPOSURE.public_listed} AND d.deleted = 0${where}`];
- const acl = aclClause(viewer, params);
- if (acl) {
- branches.push(`SELECT d.rid AS rid, ${rank('fts_restricted')} AS rank, ${snip('fts_restricted')} AS snip
- FROM fts_restricted JOIN documents d ON d.rid = fts_restricted.rowid
- WHERE fts_restricted MATCH @match AND d.exposure = ${EXPOSURE.restricted} AND d.deleted = 0${where} AND ${acl}`);
- }
- return branches.join('\nUNION ALL\n');
- }
-
- /** The visible set without a text query (browse). */
- function visibleSet({ filters, viewer }, params) {
- const where = filterClause(filters, params);
- const acl = aclClause(viewer, params);
- const vis = acl
- ? `(d.exposure = ${EXPOSURE.public_listed} OR (d.exposure = ${EXPOSURE.restricted} AND ${acl}))`
- : `d.exposure = ${EXPOSURE.public_listed}`;
- return `SELECT d.rid AS rid, d.sort_at AS sort_at FROM documents d WHERE d.deleted = 0 AND ${vis}${where}`;
- }
-
- // ── Queries ──────────────────────────────────────────────
-
- /** Ranked full-text search. after = { rank, rid } keyset cursor. */
- function search({ match, filters = {}, viewer, limit, after, now = Date.now() }) {
- const params = { limit: limit + 1 };
- let sql = `SELECT rid, rank, snip FROM (${matchedSet({ match, filters, viewer, now }, params)})`;
- if (after) {
- sql += ' WHERE (rank > @c_rank OR (rank = @c_rank AND rid > @c_rid))';
- params.c_rank = after.rank;
- params.c_rid = after.rid;
- }
- sql += ' ORDER BY rank, rid LIMIT @limit';
- const rows = db.prepare(sql).all(params);
- const more = rows.length > limit;
- const hits = rows.slice(0, limit);
- const last = hits[hits.length - 1];
- return { hits, next: more && last ? { rank: last.rank, rid: last.rid } : null };
- }
-
- /** Newest first, no text query. after = { sort_at, rid }. */
- function browse({ filters = {}, viewer, limit, after }) {
- const params = { limit: limit + 1 };
- let sql = `SELECT rid, sort_at FROM (${visibleSet({ filters, viewer }, params)})`;
- if (after) {
- sql += ' WHERE (sort_at < @c_sort OR (sort_at = @c_sort AND rid < @c_rid))';
- params.c_sort = after.sort_at;
- params.c_rid = after.rid;
- }
- sql += ' ORDER BY sort_at DESC, rid DESC LIMIT @limit';
- const rows = db.prepare(sql).all(params);
- const more = rows.length > limit;
- const hits = rows.slice(0, limit).map(r => ({ rid: r.rid, rank: null, snip: null, sort_at: r.sort_at }));
- const last = hits[hits.length - 1];
- return { hits, next: more && last ? { sort_at: last.sort_at, rid: last.rid } : null };
- }
-
- /**
- * Value counts for facet keys over exactly the visible matching set (same SQL as the hits),
- * so a count can never include a document the viewer cannot see.
- */
- function facetCounts({ match, filters = {}, viewer, keys, perKey = 20 }) {
- const out = {};
- for (const key of keys) {
- const params = { fkey: key, per: perKey };
- const set = match ? matchedSet({ match, filters, viewer, snippets: false }, params) : visibleSet({ filters, viewer }, params);
- const rows = db.prepare(`SELECT f.value AS value, COUNT(*) AS count FROM doc_facets f
- WHERE f.key = @fkey AND f.rid IN (SELECT rid FROM (${set}))
- GROUP BY f.value ORDER BY count DESC, value LIMIT @per`).all(params);
- out[key] = rows;
- }
- return out;
- }
-
- /** Title prefix suggestions over the visible set. */
- function suggest({ match, filters = {}, viewer, limit }) {
- const params = { limit };
- const sql = `SELECT rid, rank FROM (${matchedSet({ match, filters, viewer, snippets: false }, params)}) ORDER BY rank, rid LIMIT @limit`;
- return db.prepare(sql).all(params);
- }
-
- function counts() {
- return {
- public_listed: db.prepare('SELECT COUNT(*) AS n FROM fts_public').get().n,
- restricted: db.prepare('SELECT COUNT(*) AS n FROM fts_restricted').get().n,
- };
- }
-
- return { name: 'sqlite-fts5', put, remove, search, browse, facetCounts, suggest, counts, freshness };
-}
-
-// ── Query text → FTS5 expression ─────────────────────────────
-
-/**
- * Turns user text into a safe FTS5 expression: word tokens only, each quoted (no operators,
- * column filters or syntax errors reach FTS5), all required. A trailing `*` on the last word
- * makes it a prefix. Returns null when nothing searchable is left.
- */
-function toMatch(q, { maxTerms = 16, column = null, prefixLast = false } = {}) {
- const text = String(q || '').normalize('NFKC');
- const words = (text.toLowerCase().match(/[\p{L}\p{N}_]+/gu) || []).slice(0, maxTerms);
- if (!words.length) return null;
- const wantsPrefix = prefixLast || /[\p{L}\p{N}_]\*\s*$/u.test(text);
- const parts = words.map((w, i) => `"${w}"${wantsPrefix && i === words.length - 1 ? '*' : ''}`);
- const expr = parts.join(' ');
- return column ? `{${column}} : (${expr})` : expr;
-}
-
-/** Plain text with the engine's snippet marks → HTML with <mark>, everything else escaped. */
-function snippetHtml(snip) {
- if (!snip) return null;
- const esc = String(snip).replace(/[&<>"']/g, c => ({ '&': '&', '<': '<', '>': '>', '"': '"', "'": ''' }[c]));
- return esc.split(MARK_OPEN).join('<mark>').split(MARK_CLOSE).join('</mark>');
-}
-
-module.exports = { createEngine, toMatch, snippetHtml, freshnessBoost };
new file mode 100644
new file mode 100644
@@ -0,0 +1,293 @@
@@ -0,0 +1,293 @@
+'use strict';
+/**
+ * Search engine: PostgreSQL full-text search in the service's own database (ADR-035; it replaced SQLite FTS5, the
+ * engine docs/adr-engine.md chose, with the same interface and the same guarantees).
+ *
+ * One table, search_fts, split by audience so an anonymous query never touches, ranks against or counts anything but
+ * public listed documents:
+ *
+ * audience 3 documents with exposure public_listed (public, published, indexable)
+ * audience 1 documents with exposure restricted (unlisted/members/private, published)
+ *
+ * Drafts, unpublished, deleted and public-noindex documents have no row, so no query can reach them. Every query is
+ * also filtered by `documents.exposure` and, for restricted rows, by the viewer's ACL keys (doc_acl) in the same SQL:
+ * visibility is decided before anything is ranked, counted, snippeted or suggested.
+ *
+ * Text is folded before it is indexed or queried (NFKD, combining marks removed: FTS5's remove_diacritics), split into
+ * runs of letters and digits (FTS5's unicode61) and tokenized by the `simple` configuration (lower case, no stemming, no stop words: FTS5's unicode61). Title, summary and
+ * body weigh 10, 4 and 1 (tsvector weights A, B, C). Every function here is async; put and remove run inside the store's
+ * transaction (ambient). Nothing outside this file writes SQL against search_fts.
+ */
+const { EXPOSURE } = require('../document');
+
+const CONFIG = 'simple';
+const WEIGHTS = '{0.1, 0.1, 0.4, 1.0}'; // D, C body, B summary, A title: FTS5's bm25 weights 10 / 4 / 1
+const MARK_OPEN = '\u0001';
+const MARK_CLOSE = '\u0002';
+const HEADLINE = `StartSel=${MARK_OPEN}, StopSel=${MARK_CLOSE}, MaxWords=16, MinWords=8, ShortWord=0, MaxFragments=1, FragmentDelimiter=" … "`;
+
+const DAY_MS = 24 * 3600 * 1000;
+
+/**
+ * Freshness multiplier for a document dated `iso` at reference time `nowMs`:
+ *
+ * 1 + weight × 2^(−age_days / halfLifeDays)
+ *
+ * With the defaults (weight 1, half-life 30 days) a document published today counts ×2.0, one
+ * 30 days old ×1.5, 60 days ×1.25, 90 days ×1.125, a year old ×1.0002: relevance still decides,
+ * and between comparably relevant documents the newer one ranks first. A future date counts as
+ * today; no date (or an unparsable one) counts ×1. Weight 0 turns the boost off. The SQL below
+ * computes the same from search_fts.fresh_at.
+ */
+function freshnessBoost(iso, nowMs, halfLifeDays, weight) {
+ if (!weight || !iso) return 1;
+ const t = Date.parse(iso);
+ if (!Number.isFinite(t)) return 1;
+ const ageDays = Math.max(0, (nowMs - t) / DAY_MS);
+ return 1 + weight * Math.pow(2, -ageDays / halfLifeDays);
+}
+
+/** Diacritics folded, as FTS5's remove_diacritics did: the index and every query see the same text. */
+function fold(s) {
+ return String(s || '').normalize('NFKD').replace(/\p{M}+/gu, '').normalize('NFC');
+}
+
+/**
+ * The words FTS5's unicode61 tokenizer saw: runs of letters and digits, everything else a separator. PostgreSQL's
+ * parser would keep "OpenVibe.Live", "a_b" or "3.5" as single tokens (host, word, number); separating them first makes
+ * "openvibe" find "OpenVibe.Live" as it did.
+ */
+function words(s) {
+ return fold(s).replace(/[^\p{L}\p{N}]+/gu, ' ');
+}
+
+function createEngine(db, { freshness = { weight: 1, halfLifeDays: 30 } } = {}) {
+ const st = {
+ del: db.prepare('DELETE FROM search_fts WHERE rid = ?'),
+ ins: db.prepare(`INSERT INTO search_fts (rid, audience, fresh_at, title, summary, body, tsv)
+ VALUES (@rid, @audience, @fresh_at, @title, @summary, @body,
+ setweight(to_tsvector('${CONFIG}', @f_title), 'A') || setweight(to_tsvector('${CONFIG}', @f_summary), 'B')
+ || setweight(to_tsvector('${CONFIG}', @f_body), 'C'))`),
+ };
+
+ /** (Re)index one document according to its exposure. Call inside the store's transaction. */
+ async function put(rid, doc, exposure) {
+ await remove(rid);
+ if (exposure !== EXPOSURE.public_listed && exposure !== EXPOSURE.restricted) return;
+ const at = Date.parse(doc.published_at || doc.updated_at || '');
+ const title = doc.title || '';
+ const summary = doc.summary || '';
+ const body = doc.body || '';
+ await st.ins.run({
+ rid, audience: exposure, fresh_at: Number.isFinite(at) ? at : null, title, summary, body,
+ f_title: words(title), f_summary: words(summary), f_body: words(body),
+ });
+ }
+
+ async function remove(rid) {
+ await st.del.run(rid);
+ }
+
+ // ── SQL building ─────────────────────────────────────────
+
+ /**
+ * The ACL predicate for restricted rows (alias d). private: the viewer's subject is in
+ * acl.subjects. members/unlisted: subject, group or entitlement match. Returns null when the
+ * viewer has no subject (then the restricted branch is not queried at all).
+ */
+ function aclClause(viewer, params) {
+ if (!viewer || !viewer.subject) return null;
+ params.v_subject = viewer.subject;
+ params.v_groups = (viewer.groups || []).map(String);
+ params.v_ents = (viewer.entitlements || []).map(String);
+ return `(
+ (d.visibility = 'private' AND EXISTS (SELECT 1 FROM doc_acl a WHERE a.rid = d.rid AND a.kind = 's' AND a.value = @v_subject))
+ OR (d.visibility IN ('members', 'unlisted') AND EXISTS (SELECT 1 FROM doc_acl a WHERE a.rid = d.rid AND (
+ (a.kind = 's' AND a.value = @v_subject)
+ OR (a.kind = 'g' AND a.value = ANY(@v_groups::text[]))
+ OR (a.kind = 'e' AND a.value = ANY(@v_ents::text[])))))
+ )`;
+ }
+
+ function filterClause(filters, params) {
+ const parts = [];
+ if (filters.owner) { parts.push('d.owner = @f_owner'); params.f_owner = filters.owner; }
+ if (filters.type) { parts.push('d.type = @f_type'); params.f_type = filters.type; }
+ if (filters.language) { parts.push('d.language = @f_lang'); params.f_lang = filters.language; }
+ (filters.facets || []).forEach(([key, values], i) => {
+ params[`f_fk${i}`] = key;
+ params[`f_fv${i}`] = values.map(String);
+ parts.push(`EXISTS (SELECT 1 FROM doc_facets x WHERE x.rid = d.rid AND x.key = @f_fk${i} AND x.value = ANY(@f_fv${i}::text[]))`);
+ });
+ return parts.length ? ` AND ${parts.join(' AND ')}` : '';
+ }
+
+ /**
+ * The visible, matching set as (rid, rank) rows. match = toMatch()'s query. rank is minus ts_rank times the
+ * freshness boost at reference time `now` (ms), so lower ranks first (as FTS5's bm25 did) and pages of one query
+ * (which carry their first page's `now` in the cursor) rank against the same clock.
+ */
+ function matchedSet({ match, filters, viewer, now = Date.now() }, params) {
+ params.match = tsquery(match);
+ params.r_now = now;
+ params.r_half = freshness.halfLifeDays;
+ params.r_weight = freshness.weight;
+ const where = filterClause(filters, params);
+ const rank = `-(ts_rank('${WEIGHTS}'::float4[], f.tsv, to_tsquery('${CONFIG}', @match), 1)
+ * CASE WHEN f.fresh_at IS NULL OR @r_weight::float8 = 0 THEN 1
+ ELSE 1 + @r_weight::float8 * power(2::float8, -(GREATEST(0, @r_now::bigint - f.fresh_at)::float8 / ${DAY_MS}) / @r_half::float8) END)`;
+ const branches = [`SELECT d.rid AS rid, ${rank} AS rank
+ FROM search_fts f JOIN documents d ON d.rid = f.rid
+ WHERE f.audience = ${EXPOSURE.public_listed} AND f.tsv @@ to_tsquery('${CONFIG}', @match)
+ AND d.exposure = ${EXPOSURE.public_listed} AND d.deleted = 0${where}`];
+ const acl = aclClause(viewer, params);
+ if (acl) {
+ branches.push(`SELECT d.rid AS rid, ${rank} AS rank
+ FROM search_fts f JOIN documents d ON d.rid = f.rid
+ WHERE f.audience = ${EXPOSURE.restricted} AND f.tsv @@ to_tsquery('${CONFIG}', @match)
+ AND d.exposure = ${EXPOSURE.restricted} AND d.deleted = 0${where} AND ${acl}`);
+ }
+ return branches.join('\nUNION ALL\n');
+ }
+
+ /** The visible set without a text query (browse). */
+ function visibleSet({ filters, viewer }, params) {
+ const where = filterClause(filters, params);
+ const acl = aclClause(viewer, params);
+ const vis = acl
+ ? `(d.exposure = ${EXPOSURE.public_listed} OR (d.exposure = ${EXPOSURE.restricted} AND ${acl}))`
+ : `d.exposure = ${EXPOSURE.public_listed}`;
+ return `SELECT d.rid AS rid, d.sort_at AS sort_at FROM documents d WHERE d.deleted = 0 AND ${vis}${where}`;
+ }
+
+ // ── Queries ──────────────────────────────────────────────
+
+ /**
+ * Ranked full-text search. after = { rank, rid } keyset cursor. The snippet is computed for the page's rows only:
+ * the summary and body when they match, else the title.
+ */
+ async function search({ match, filters = {}, viewer, limit, after, now = Date.now() }) {
+ const params = { limit: limit + 1 };
+ let sql = `SELECT rid, rank FROM (${matchedSet({ match, filters, viewer, now }, params)}) m`;
+ if (after) {
+ sql += ' WHERE (rank > @c_rank::float8 OR (rank = @c_rank::float8 AND rid > @c_rid::bigint))';
+ params.c_rank = after.rank;
+ params.c_rid = after.rid;
+ }
+ sql += ' ORDER BY rank, rid LIMIT @limit';
+ const page = `SELECT p.rid, p.rank,
+ CASE WHEN to_tsvector('${CONFIG}', f.summary || ' ' || f.body) @@ to_tsquery('${CONFIG}', @match)
+ THEN ts_headline('${CONFIG}', f.summary || ' ' || f.body, to_tsquery('${CONFIG}', @match), @hl)
+ ELSE ts_headline('${CONFIG}', f.title, to_tsquery('${CONFIG}', @match), @hl) END AS snip
+ FROM (${sql}) p JOIN search_fts f ON f.rid = p.rid ORDER BY p.rank, p.rid`;
+ params.hl = HEADLINE;
+ const rows = await db.prepare(page).all(params);
+ const more = rows.length > limit;
+ const hits = rows.slice(0, limit);
+ const last = hits[hits.length - 1];
+ return { hits, next: more && last ? { rank: last.rank, rid: last.rid } : null };
+ }
+
+ /** Newest first, no text query. after = { sort_at, rid }. */
+ async function browse({ filters = {}, viewer, limit, after }) {
+ const params = { limit: limit + 1 };
+ let sql = `SELECT rid, sort_at FROM (${visibleSet({ filters, viewer }, params)}) v`;
+ if (after) {
+ sql += ' WHERE (sort_at < @c_sort OR (sort_at = @c_sort AND rid < @c_rid::bigint))';
+ params.c_sort = after.sort_at;
+ params.c_rid = after.rid;
+ }
+ sql += ' ORDER BY sort_at DESC, rid DESC LIMIT @limit';
+ const rows = await db.prepare(sql).all(params);
+ const more = rows.length > limit;
+ const hits = rows.slice(0, limit).map(r => ({ rid: r.rid, rank: null, snip: null, sort_at: r.sort_at }));
+ const last = hits[hits.length - 1];
+ return { hits, next: more && last ? { sort_at: last.sort_at, rid: last.rid } : null };
+ }
+
+ /**
+ * Value counts for facet keys over exactly the visible matching set (same SQL as the hits),
+ * so a count can never include a document the viewer cannot see.
+ */
+ async function facetCounts({ match, filters = {}, viewer, keys, perKey = 20 }) {
+ const out = {};
+ for (const key of keys) {
+ const params = { fkey: key, per: perKey };
+ const set = match ? matchedSet({ match, filters, viewer }, params) : visibleSet({ filters, viewer }, params);
+ out[key] = await db.prepare(`SELECT f.value AS value, COUNT(*)::int AS count FROM doc_facets f
+ WHERE f.key = @fkey AND f.rid IN (SELECT rid FROM (${set}) s)
+ GROUP BY f.value ORDER BY count DESC, value LIMIT @per`).all(params);
+ }
+ return out;
+ }
+
+ /** Title prefix suggestions over the visible set. */
+ async function suggest({ match, filters = {}, viewer, limit }) {
+ const params = { limit };
+ return await db.prepare(`SELECT rid, rank FROM (${matchedSet({ match, filters, viewer }, params)}) m ORDER BY rank, rid LIMIT @limit`).all(params);
+ }
+
+ async function counts() {
+ const rows = await db.prepare('SELECT audience, COUNT(*)::int AS n FROM search_fts GROUP BY audience').all();
+ const of = (a) => (rows.find((r) => Number(r.audience) === a) || { n: 0 }).n;
+ return { public_listed: of(EXPOSURE.public_listed), restricted: of(EXPOSURE.restricted) };
+ }
+
+ /**
+ * Rebuild the index from documents.doc when its counts disagree with the documents table (after the one-time import
+ * from SQLite, whose FTS5 tables are not copied). Returns what it did.
+ */
+ async function reconcile() {
+ const docs = await db.prepare(`SELECT
+ COUNT(*) FILTER (WHERE exposure = ${EXPOSURE.public_listed} AND deleted = 0)::int AS public_listed,
+ COUNT(*) FILTER (WHERE exposure = ${EXPOSURE.restricted} AND deleted = 0)::int AS restricted
+ FROM documents`).get();
+ const idx = await counts();
+ if (idx.public_listed === docs.public_listed && idx.restricted === docs.restricted) return { rebuilt: 0 };
+ let rebuilt = 0;
+ await db.tx(async () => {
+ await db.prepare('DELETE FROM search_fts').run();
+ const rows = await db.prepare(`SELECT rid, exposure, doc FROM documents WHERE deleted = 0 AND exposure IN (${EXPOSURE.public_listed}, ${EXPOSURE.restricted}) ORDER BY rid`).all();
+ for (const r of rows) {
+ await put(r.rid, typeof r.doc === 'string' ? JSON.parse(r.doc) : r.doc, Number(r.exposure));
+ rebuilt++;
+ }
+ });
+ return { rebuilt };
+ }
+
+ return { name: 'postgresql-fts', put, remove, search, browse, facetCounts, suggest, counts, reconcile, freshness };
+}
+
+// ── Query text → tsquery ─────────────────────────────────────
+
+/**
+ * Turns user text into a safe query: word tokens only (no operators or syntax reach PostgreSQL), all required. A
+ * trailing `*` on the last word makes it a prefix; column 'title' restricts every word to the title. Returns
+ * { words, prefix, column } or null when nothing searchable is left.
+ */
+function toMatch(q, { maxTerms = 16, column = null, prefixLast = false } = {}) {
+ const text = String(q || '').normalize('NFKC');
+ const words = (fold(text).toLowerCase().match(/[\p{L}\p{N}]+/gu) || []).slice(0, maxTerms);
+ if (!words.length) return null;
+ const prefix = prefixLast || /[\p{L}\p{N}_]\*\s*$/u.test(text);
+ return { words, prefix, column };
+}
+
+/** toMatch()'s query → to_tsquery text: 'w1' & 'w2' & 'w3':* (and :A on each word for the title). */
+function tsquery(m) {
+ const weight = m.column === 'title' ? 'A' : '';
+ return m.words.map((w, i) => {
+ const star = m.prefix && i === m.words.length - 1 ? '*' : '';
+ return `'${w}'${star || weight ? `:${star}${weight}` : ''}`;
+ }).join(' & ');
+}
+
+/** Plain text with the engine's snippet marks → HTML with <mark>, everything else escaped. */
+function snippetHtml(snip) {
+ if (!snip) return null;
+ const esc = String(snip).replace(/[&<>"']/g, c => ({ '&': '&', '<': '<', '>': '>', '"': '"', "'": ''' }[c]));
+ return esc.split(MARK_OPEN).join('<mark>').split(MARK_CLOSE).join('</mark>');
+}
+
+module.exports = { createEngine, toMatch, tsquery, snippetHtml, freshnessBoost, fold, words };
@@ -36,8 +36,8 @@ function createOutbox(db, { source, now = () => Date.now() }) {
@@ -36,8 +36,8 @@ function createOutbox(db, { source, now = () => Date.now() }) {
const insert = db.prepare('INSERT INTO event_outbox (event_id, event_type, envelope, created_at) VALUES (?, ?, ?, ?)');
const insert = db.prepare('INSERT INTO event_outbox (event_id, event_type, envelope, created_at) VALUES (?, ?, ?, ?)');
/** { event_type, subject, payload, visibility?, priority?, actor?, trace_id? } → envelope */
/** { event_type, subject, payload, visibility?, priority?, actor?, trace_id? } → envelope */
- function enqueue({ event_type, subject, payload, visibility = 'internal', priority = 'important', actor, trace_id }) {
- if (!db.inTransaction) throw new Error('outbox.enqueue() must run inside the transaction that makes the change');
+ async function enqueue({ event_type, subject, payload, visibility = 'internal', priority = 'important', actor, trace_id }) {
+ if (!db.inTransaction()) throw new Error('outbox.enqueue() must run inside the transaction that makes the change');
const ms = now();
const ms = now();
const env = {
const env = {
event_id: ids.newId('event', ms),
event_id: ids.newId('event', ms),
@@ -54,7 +54,7 @@ function createOutbox(db, { source, now = () => Date.now() }) {
@@ -54,7 +54,7 @@ function createOutbox(db, { source, now = () => Date.now() }) {
if (trace_id && /^[0-9a-f]{32}$/.test(trace_id)) env.trace_id = trace_id;
if (trace_id && /^[0-9a-f]{32}$/.test(trace_id)) env.trace_id = trace_id;
const v = validate('events.event-envelope@1', env);
const v = validate('events.event-envelope@1', env);
if (!v.valid) throw new Error(`outbox: invalid envelope for ${event_type}: ${v.errors.map(e => `${e.path} ${e.message}`).join('; ')}`);
if (!v.valid) throw new Error(`outbox: invalid envelope for ${event_type}: ${v.errors.map(e => `${e.path} ${e.message}`).join('; ')}`);
- insert.run(env.event_id, event_type, JSON.stringify(env), ms);
+ await insert.run(env.event_id, event_type, JSON.stringify(env), ms);
return env;
return env;
}
}
@@ -71,11 +71,11 @@ function createOutbox(db, { source, now = () => Date.now() }) {
@@ -71,11 +71,11 @@ function createOutbox(db, { source, now = () => Date.now() }) {
return {
return {
enqueue,
enqueue,
- pending: () => q.pending.get().n,
- rejected: () => q.rejectedCount.get().n,
+ pending: async () => (await q.pending.get()).n,
+ rejected: async () => (await q.rejectedCount.get()).n,
/** Every envelope in order (tests and operators). */
/** Every envelope in order (tests and operators). */
- all: () => q.list.all().map(r => JSON.parse(r.envelope)),
- prune: (olderThanMs = 7 * 24 * 3600 * 1000) => q.prune.run(now() - olderThanMs).changes,
+ all: async () => (await q.list.all()).map(r => JSON.parse(r.envelope)),
+ prune: async (olderThanMs = 7 * 24 * 3600 * 1000) => (await q.prune.run(now() - olderThanMs)).changes,
_q: q,
_q: q,
};
};
}
}
@@ -109,18 +109,18 @@ function createRelay({ outbox, eventsUrl, intervalMs = 2000, tokenClient, tokenO
@@ -109,18 +109,18 @@ function createRelay({ outbox, eventsUrl, intervalMs = 2000, tokenClient, tokenO
return { ok: res.ok, status: res.status, text };
return { ok: res.ok, status: res.status, text };
}
}
- function mark(rows, fn) { db.transaction(() => { for (const r of rows) fn(r); })(); }
+ async function mark(rows, fn) { await db.tx(() => { for (const r of rows) fn(r); }); }
async function publishRows(rows) {
async function publishRows(rows) {
let r;
let r;
try {
try {
r = await post(rows);
r = await post(rows);
} catch (err) {
} catch (err) {
- mark(rows, row => q.failed.run(now() + BACKOFF_MS[Math.min(row.attempts, BACKOFF_MS.length - 1)], String(err.message).slice(0, 500), row.seq));
+ await mark(rows, async row => await q.failed.run(now() + BACKOFF_MS[Math.min(row.attempts, BACKOFF_MS.length - 1)], String(err.message).slice(0, 500), row.seq));
return { sent: 0, failed: rows.length, rejected: 0 };
return { sent: 0, failed: rows.length, rejected: 0 };
}
}
if (r.ok) {
if (r.ok) {
- mark(rows, row => q.sent.run(now(), row.seq));
+ await mark(rows, async row => await q.sent.run(now(), row.seq));
return { sent: rows.length, failed: 0, rejected: 0 };
return { sent: rows.length, failed: 0, rejected: 0 };
}
}
if (permanent(r.status) && rows.length > 1) {
if (permanent(r.status) && rows.length > 1) {
@@ -134,11 +134,11 @@ function createRelay({ outbox, eventsUrl, intervalMs = 2000, tokenClient, tokenO
@@ -134,11 +134,11 @@ function createRelay({ outbox, eventsUrl, intervalMs = 2000, tokenClient, tokenO
}
}
const msg = `${r.status} ${r.text.slice(0, 300)}`;
const msg = `${r.status} ${r.text.slice(0, 300)}`;
if (permanent(r.status)) {
if (permanent(r.status)) {
- mark(rows, row => q.rejected.run(now(), msg, row.seq));
+ await mark(rows, async row => await q.rejected.run(now(), msg, row.seq));
log.warn(`[outbox] event refused by Events: ${msg}`);
log.warn(`[outbox] event refused by Events: ${msg}`);
return { sent: 0, failed: 0, rejected: rows.length };
return { sent: 0, failed: 0, rejected: rows.length };
}
}
- mark(rows, row => q.failed.run(now() + BACKOFF_MS[Math.min(row.attempts, BACKOFF_MS.length - 1)], msg, row.seq));
+ await mark(rows, async row => await q.failed.run(now() + BACKOFF_MS[Math.min(row.attempts, BACKOFF_MS.length - 1)], msg, row.seq));
return { sent: 0, failed: rows.length, rejected: 0 };
return { sent: 0, failed: rows.length, rejected: 0 };
}
}
@@ -146,7 +146,7 @@ function createRelay({ outbox, eventsUrl, intervalMs = 2000, tokenClient, tokenO
@@ -146,7 +146,7 @@ function createRelay({ outbox, eventsUrl, intervalMs = 2000, tokenClient, tokenO
const total = { sent: 0, failed: 0, rejected: 0 };
const total = { sent: 0, failed: 0, rejected: 0 };
if (!eventsUrl || !tokens) return total;
if (!eventsUrl || !tokens) return total;
for (;;) {
for (;;) {
- const rows = q.due.all(now(), 50);
+ const rows = await q.due.all(now(), 50);
if (!rows.length) break;
if (!rows.length) break;
const s = await publishRows(rows);
const s = await publishRows(rows);
total.sent += s.sent; total.failed += s.failed; total.rejected += s.rejected;
total.sent += s.sent; total.failed += s.failed; total.rejected += s.rejected;
@@ -9,7 +9,7 @@
@@ -9,7 +9,7 @@
*/
*/
const { load } = require('./config');
const { load } = require('./config');
const { openDb } = require('./db');
const { openDb } = require('./db');
-const { createEngine } = require('./engine/fts5');
+const { createEngine } = require('./engine/pg');
const { createStore } = require('./store');
const { createStore } = require('./store');
const { createOutbox, createRelay } = require('./events/outbox');
const { createOutbox, createRelay } = require('./events/outbox');
const { createKeyStore, createAuth } = require('./auth');
const { createKeyStore, createAuth } = require('./auth');
@@ -17,10 +17,14 @@ const { createPurgeQueue, createPurger } = require('./purge');
@@ -17,10 +17,14 @@ const { createPurgeQueue, createPurger } = require('./purge');
const { createSavedSearches } = require('./saved');
const { createSavedSearches } = require('./saved');
const { createApp } = require('./app');
const { createApp } = require('./app');
-async function start({ config, now = () => Date.now(), fetchImpl = globalThis.fetch, tokenClient, log = console, listen = true } = {}) {
+async function start({ config, db: givenDb = null, now = () => Date.now(), fetchImpl = globalThis.fetch, tokenClient, log = console, listen = true } = {}) {
config = config || load();
config = config || load();
- const db = openDb(config.dbPath);
+ // PostgreSQL (ADR-035): opened and migrated here unless the caller (a test) hands in a migrated handle.
+ const db = givenDb || await openDb(config, { log });
const engine = createEngine(db, { freshness: config.freshness });
const engine = createEngine(db, { freshness: config.freshness });
+ // The index is derived from documents: rebuilt when it disagrees (after the one-time import from SQLite).
+ const rebuilt = await engine.reconcile();
+ if (rebuilt.rebuilt) log.log(`[search] full-text index rebuilt: ${rebuilt.rebuilt} document(s)`);
const outbox = createOutbox(db, { source: config.serviceId, now });
const outbox = createOutbox(db, { source: config.serviceId, now });
const purges = createPurgeQueue(db, { config: config.purge, now });
const purges = createPurgeQueue(db, { config: config.purge, now });
const purger = createPurger({ db, config: config.purge, fetchImpl, log, now });
const purger = createPurger({ db, config: config.purge, fetchImpl, log, now });
@@ -38,7 +42,7 @@ async function start({ config, now = () => Date.now(), fetchImpl = globalThis.fe
@@ -38,7 +42,7 @@ async function start({ config, now = () => Date.now(), fetchImpl = globalThis.fe
const keyLoaded = keys.start().catch(() => null);
const keyLoaded = keys.start().catch(() => null);
relay.start();
relay.start();
purger.start();
purger.start();
- const pruneTimer = setInterval(() => { try { outbox.prune(); } catch (err) { log.error(`[outbox] prune: ${err.message}`); } }, 6 * 3600 * 1000);
+ const pruneTimer = setInterval(async () => { try { await outbox.prune(); } catch (err) { log.error(`[outbox] prune: ${err.message}`); } }, 6 * 3600 * 1000);
pruneTimer.unref?.();
pruneTimer.unref?.();
let server = null;
let server = null;
@@ -59,7 +63,7 @@ async function start({ config, now = () => Date.now(), fetchImpl = globalThis.fe
@@ -59,7 +63,7 @@ async function start({ config, now = () => Date.now(), fetchImpl = globalThis.fe
server.closeAllConnections?.();
server.closeAllConnections?.();
await new Promise(resolve => server.close(() => resolve()));
await new Promise(resolve => server.close(() => resolve()));
}
}
- db.close();
+ if (!givenDb) await db.close();
}
}
return { config, db, engine, store, outbox, relay, purges, purger, saved, keys, keyLoaded, auth, app, server, close };
return { config, db, engine, store, outbox, relay, purges, purger, saved, keys, keyLoaded, auth, app, server, close };
@@ -19,63 +19,63 @@ const { EXPOSURE } = require('./document');
@@ -19,63 +19,63 @@ const { EXPOSURE } = require('./document');
const EXPOSURE_NAMES = Object.fromEntries(Object.entries(EXPOSURE).map(([name, n]) => [n, name]));
const EXPOSURE_NAMES = Object.fromEntries(Object.entries(EXPOSURE).map(([name, n]) => [n, name]));
-function readers(db) {
- const byExposure = db.prepare('SELECT exposure, COUNT(*) AS n FROM documents GROUP BY exposure');
- const ftsPublic = db.prepare('SELECT COUNT(*) AS n FROM fts_public');
- const ftsRestricted = db.prepare('SELECT COUNT(*) AS n FROM fts_restricted');
+function readers(db, engine) {
+ const byExposure = db.prepare('SELECT exposure, COUNT(*)::int AS n FROM documents GROUP BY exposure');
return {
return {
/** { none, restricted, public_unlisted, public_listed }, every exposure present (0 when empty). */
/** { none, restricted, public_unlisted, public_listed }, every exposure present (0 when empty). */
- documents() {
+ async documents() {
const out = Object.fromEntries(Object.keys(EXPOSURE).map((k) => [k, 0]));
const out = Object.fromEntries(Object.keys(EXPOSURE).map((k) => [k, 0]));
- for (const r of byExposure.all()) out[EXPOSURE_NAMES[r.exposure] || 'none'] += r.n;
+ for (const r of await byExposure.all()) out[EXPOSURE_NAMES[r.exposure] || 'none'] += r.n;
return out;
return out;
},
},
- indexed: () => ({ public: ftsPublic.get().n, restricted: ftsRestricted.get().n }),
+ indexed: async () => { const c = await engine.counts(); return { public: c.public_listed, restricted: c.restricted }; },
};
};
}
}
function createSearchReadiness({ db, keys, config, engine, store, outbox, relay, purges = null, purger = null, saved = null, release = null }) {
function createSearchReadiness({ db, keys, config, engine, store, outbox, relay, purges = null, purger = null, saved = null, release = null }) {
- const read = readers(db);
+ const read = readers(db, engine);
return createReadiness({
return createReadiness({
service: 'search',
service: 'search',
release,
release,
checks: [
checks: [
{
{
name: 'db', required: true,
name: 'db', required: true,
- check: () => {
- db.prepare('SELECT rid FROM documents LIMIT 1').all();
- db.prepare('SELECT rowid FROM fts_public LIMIT 1').all();
- db.prepare('SELECT rowid FROM fts_restricted LIMIT 1').all();
- return true;
+ // A real round trip that names the store (postgresql / pglite), and the tables present.
+ check: async () => {
+ const r = await db.ready();
+ if (!r.ok) return r.error;
+ await db.prepare('SELECT rid FROM documents LIMIT 1').all();
+ await db.prepare('SELECT rid FROM search_fts LIMIT 1').all();
+ return { ok: true, detail: r.detail };
},
},
},
},
{ name: 'network_jwks', required: false, check: () => keys.loaded() || 'Network signing key not loaded yet: owner writes and signed-in queries cannot be verified' },
{ name: 'network_jwks', required: false, check: () => keys.loaded() || 'Network signing key not loaded yet: owner writes and signed-in queries cannot be verified' },
{
{
name: 'index', required: false, cacheMs: 10_000,
name: 'index', required: false, cacheMs: 10_000,
- check: () => {
- const docs = read.documents();
- const fts = read.indexed();
+ check: async () => {
+ const docs = await read.documents();
+ const fts = await read.indexed();
const detail = { indexed: fts, expected: { public: docs.public_listed, restricted: docs.restricted } };
const detail = { indexed: fts, expected: { public: docs.public_listed, restricted: docs.restricted } };
const off = [];
const off = [];
- if (fts.public !== docs.public_listed) off.push(`fts_public has ${fts.public} entries for ${docs.public_listed} public_listed documents`);
- if (fts.restricted !== docs.restricted) off.push(`fts_restricted has ${fts.restricted} entries for ${docs.restricted} restricted documents`);
+ if (fts.public !== docs.public_listed) off.push(`the public index has ${fts.public} entries for ${docs.public_listed} public_listed documents`);
+ if (fts.restricted !== docs.restricted) off.push(`the restricted index has ${fts.restricted} entries for ${docs.restricted} restricted documents`);
return off.length ? { ok: false, error: `index out of step: ${off.join('; ')}`, detail } : { ok: true, detail };
return off.length ? { ok: false, error: `index out of step: ${off.join('; ')}`, detail } : { ok: true, detail };
},
},
},
},
],
],
- details: (body) => {
+ details: async (body) => {
const dbOk = body.checks.db.status === 'ok';
const dbOk = body.checks.db.status === 'ok';
return {
return {
engine: engine.name,
engine: engine.name,
- documents: dbOk ? store.counts() : null,
- outbox: dbOk ? { pending: outbox.pending(), rejected: outbox.rejected(), relay: config.events.url ? (relay.running() ? 'running' : 'stopped') : 'off (EVENTS_URL unset)' } : null,
+ documents: dbOk ? await store.counts() : null,
+ outbox: dbOk ? { pending: await outbox.pending(), rejected: await outbox.rejected(), relay: config.events.url ? (relay.running() ? 'running' : 'stopped') : 'off (EVENTS_URL unset)' } : null,
webhook: config.events.webhookSecrets.length ? 'on' : 'off (SEARCH_EVENTS_SECRET unset)',
webhook: config.events.webhookSecrets.length ? 'on' : 'off (SEARCH_EVENTS_SECRET unset)',
purge: dbOk && purges ? {
purge: dbOk && purges ? {
cdn: purges.cdnOn() ? (purger && purger.running() ? 'cloudflare' : 'cloudflare (stopped)') : 'off (CLOUDFLARE_PURGE_TOKEN unset)',
cdn: purges.cdnOn() ? (purger && purger.running() ? 'cloudflare' : 'cloudflare (stopped)') : 'off (CLOUDFLARE_PURGE_TOKEN unset)',
zones: config.purge.zones.length,
zones: config.purge.zones.length,
- ...purges.cdnCounts(),
+ ...await purges.cdnCounts(),
} : null,
} : null,
- saved_searches: dbOk && saved ? saved.total() : null,
+ saved_searches: dbOk && saved ? await saved.total() : null,
freshness: config.freshness,
freshness: config.freshness,
};
};
},
},
@@ -83,22 +83,22 @@ function createSearchReadiness({ db, keys, config, engine, store, outbox, relay,
@@ -83,22 +83,22 @@ function createSearchReadiness({ db, keys, config, engine, store, outbox, relay,
}
}
/** Search gauges on the openvibe-shared/metrics registry. */
/** Search gauges on the openvibe-shared/metrics registry. */
-function registerSearchGauges(registry, { db, outbox, purges = null }) {
- const read = readers(db);
+function registerSearchGauges(registry, { db, engine, outbox, purges = null }) {
+ const read = readers(db, engine);
registry.gauge({
registry.gauge({
name: 'search_documents', help: 'Documents held, by exposure (public_listed and restricted are searchable; tombstones and drafts are none)', labelNames: ['exposure'],
name: 'search_documents', help: 'Documents held, by exposure (public_listed and restricted are searchable; tombstones and drafts are none)', labelNames: ['exposure'],
- collect: () => Object.entries(read.documents()).map(([exposure, value]) => ({ labels: { exposure }, value })),
+ collect: async () => Object.entries(await read.documents()).map(([exposure, value]) => ({ labels: { exposure }, value })),
});
});
registry.gauge({
registry.gauge({
name: 'search_documents_indexed', help: 'Documents in the full-text index, by table', labelNames: ['index'],
name: 'search_documents_indexed', help: 'Documents in the full-text index, by table', labelNames: ['index'],
- collect: () => Object.entries(read.indexed()).map(([index, value]) => ({ labels: { index }, value })),
+ collect: async () => Object.entries(await read.indexed()).map(([index, value]) => ({ labels: { index }, value })),
});
});
- registry.gauge({ name: 'search_outbox_pending', help: 'Events waiting in the outbox', collect: () => outbox.pending() });
- registry.gauge({ name: 'search_outbox_rejected', help: 'Events OpenVibe.Events refused for good', collect: () => outbox.rejected() });
+ registry.gauge({ name: 'search_outbox_pending', help: 'Events waiting in the outbox', collect: async () => await outbox.pending() });
+ registry.gauge({ name: 'search_outbox_rejected', help: 'Events OpenVibe.Events refused for good', collect: async () => await outbox.rejected() });
if (purges) {
if (purges) {
registry.gauge({
registry.gauge({
name: 'search_cdn_purges', help: 'Cloudflare cache purges of removed documents, by state', labelNames: ['state'],
name: 'search_cdn_purges', help: 'Cloudflare cache purges of removed documents, by state', labelNames: ['state'],
- collect: () => Object.entries(purges.cdnCounts()).map(([state, value]) => ({ labels: { state }, value })),
+ collect: async () => Object.entries(await purges.cdnCounts()).map(([state, value]) => ({ labels: { state }, value })),
});
});
}
}
}
}
@@ -88,7 +88,7 @@ function createPurgeQueue(db, { config, now = () => Date.now() }) {
@@ -88,7 +88,7 @@ function createPurgeQueue(db, { config, now = () => Date.now() }) {
const cdnOn = Boolean(config.cloudflareToken);
const cdnOn = Boolean(config.cloudflareToken);
const st = {
const st = {
insRemoval: db.prepare(`INSERT INTO removals (event_id, owner, type, id, revision, reason, previous_exposure, exposure, canonical_url, at)
insRemoval: db.prepare(`INSERT INTO removals (event_id, owner, type, id, revision, reason, previous_exposure, exposure, canonical_url, at)
- VALUES (@event_id, @owner, @type, @id, @revision, @reason, @previous_exposure, @exposure, @canonical_url, @at)`),
+ VALUES (@event_id, @owner, @type, @id, @revision, @reason, @previous_exposure, @exposure, @canonical_url, @at) RETURNING seq`),
insPurge: db.prepare(`INSERT INTO cdn_purges (removal_seq, zone_id, url, state, detail, created_at, done_at)
insPurge: db.prepare(`INSERT INTO cdn_purges (removal_seq, zone_id, url, state, detail, created_at, done_at)
VALUES (@removal_seq, @zone_id, @url, @state, @detail, @created_at, @done_at)
VALUES (@removal_seq, @zone_id, @url, @state, @detail, @created_at, @done_at)
ON CONFLICT (zone_id, url) WHERE state = 'pending' DO NOTHING`),
ON CONFLICT (zone_id, url) WHERE state = 'pending' DO NOTHING`),
@@ -101,17 +101,17 @@ function createPurgeQueue(db, { config, now = () => Date.now() }) {
@@ -101,17 +101,17 @@ function createPurgeQueue(db, { config, now = () => Date.now() }) {
* Record one removal (the payload of the search.document.removed event). Must run inside the
* Record one removal (the payload of the search.document.removed event). Must run inside the
* transaction that made the change, like the outbox.
* transaction that made the change, like the outbox.
*/
*/
- function record(eventId, p) {
- if (!db.inTransaction) throw new Error('purge.record() must run inside the transaction that makes the change');
+ async function record(eventId, p) {
+ if (!db.inTransaction()) throw new Error('purge.record() must run inside the transaction that makes the change');
const at = now();
const at = now();
- const removalSeq = Number(st.insRemoval.run({
+ const removalSeq = Number((await st.insRemoval.run({
event_id: eventId, owner: p.owner, type: p.type, id: p.id, revision: p.revision, reason: p.reason,
event_id: eventId, owner: p.owner, type: p.type, id: p.id, revision: p.revision, reason: p.reason,
previous_exposure: p.previous_exposure, exposure: p.exposure, canonical_url: p.canonical_url || null, at,
previous_exposure: p.previous_exposure, exposure: p.exposure, canonical_url: p.canonical_url || null, at,
- }).lastInsertRowid);
+ })).lastInsertRowid);
if (!cdnOn || !p.canonical_url) return removalSeq;
if (!cdnOn || !p.canonical_url) return removalSeq;
for (const url of purgeUrls(p.canonical_url, config.relatedPaths)) {
for (const url of purgeUrls(p.canonical_url, config.relatedPaths)) {
const zoneId = zoneFor(url, config.zones);
const zoneId = zoneFor(url, config.zones);
- st.insPurge.run({
+ await st.insPurge.run({
removal_seq: removalSeq, zone_id: zoneId, url,
removal_seq: removalSeq, zone_id: zoneId, url,
state: zoneId ? 'pending' : 'skipped',
state: zoneId ? 'pending' : 'skipped',
detail: zoneId ? null : 'no Cloudflare zone configured for this host (CLOUDFLARE_ZONE_IDS)',
detail: zoneId ? null : 'no Cloudflare zone configured for this host (CLOUDFLARE_ZONE_IDS)',
@@ -122,8 +122,8 @@ function createPurgeQueue(db, { config, now = () => Date.now() }) {
@@ -122,8 +122,8 @@ function createPurgeQueue(db, { config, now = () => Date.now() }) {
}
}
/** The owner's removal feed, oldest first after a cursor. */
/** The owner's removal feed, oldest first after a cursor. */
- function ownerFeed(owner, { after = 0, limit = 200 } = {}) {
- const rows = st.ownerFeed.all(owner, after, limit);
+ async function ownerFeed(owner, { after = 0, limit = 200 } = {}) {
+ const rows = await st.ownerFeed.all(owner, after, limit);
return {
return {
removals: rows.map(r => ({
removals: rows.map(r => ({
seq: r.seq, event_id: r.event_id, type: r.type, id: r.id, revision: r.revision, reason: r.reason,
seq: r.seq, event_id: r.event_id, type: r.type, id: r.id, revision: r.revision, reason: r.reason,
@@ -134,13 +134,13 @@ function createPurgeQueue(db, { config, now = () => Date.now() }) {
@@ -134,13 +134,13 @@ function createPurgeQueue(db, { config, now = () => Date.now() }) {
};
};
}
}
- function cdnCounts() {
+ async function cdnCounts() {
const out = { pending: 0, purged: 0, failed: 0, skipped: 0 };
const out = { pending: 0, purged: 0, failed: 0, skipped: 0 };
- for (const r of st.cdnCounts.all()) out[r.state] = r.n;
+ for (const r of await st.cdnCounts.all()) out[r.state] = r.n;
return out;
return out;
}
}
- return { record, ownerFeed, cdnCounts, purgesFor: (seq) => st.purgesFor.all(seq), cdnOn: () => cdnOn };
+ return { record, ownerFeed, cdnCounts, purgesFor: async (seq) => await st.purgesFor.all(seq), cdnOn: () => cdnOn };
}
}
/**
/**
@@ -161,12 +161,12 @@ function createPurger({ db, config, fetchImpl = globalThis.fetch, log = console,
@@ -161,12 +161,12 @@ function createPurger({ db, config, fetchImpl = globalThis.fetch, log = console,
const clip = (s) => (token ? String(s || '').split(token).join('[token]') : String(s || '')).slice(0, DETAIL_MAX);
const clip = (s) => (token ? String(s || '').split(token).join('[token]') : String(s || '')).slice(0, DETAIL_MAX);
- function mark(rows, fn) { db.transaction(() => { for (const r of rows) fn(r); })(); }
+ async function mark(rows, fn) { await db.tx(() => { for (const r of rows) fn(r); }); }
- function retryOrFail(rows, detail) {
- mark(rows, (r) => {
- if (r.attempts + 1 >= MAX_ATTEMPTS) q.failed.run(clip(`gave up after ${MAX_ATTEMPTS} attempts: ${detail}`), now(), r.seq);
- else q.retry.run(now() + BACKOFF_MS[Math.min(r.attempts, BACKOFF_MS.length - 1)], clip(detail), r.seq);
+ async function retryOrFail(rows, detail) {
+ await mark(rows, async (r) => {
+ if (r.attempts + 1 >= MAX_ATTEMPTS) await q.failed.run(clip(`gave up after ${MAX_ATTEMPTS} attempts: ${detail}`), now(), r.seq);
+ else await q.retry.run(now() + BACKOFF_MS[Math.min(r.attempts, BACKOFF_MS.length - 1)], clip(detail), r.seq);
});
});
}
}
@@ -188,17 +188,17 @@ function createPurger({ db, config, fetchImpl = globalThis.fetch, log = console,
@@ -188,17 +188,17 @@ function createPurger({ db, config, fetchImpl = globalThis.fetch, log = console,
try {
try {
r = await call(zoneId, rows.map(x => x.url));
r = await call(zoneId, rows.map(x => x.url));
} catch (err) {
} catch (err) {
- retryOrFail(rows, `network: ${err.message}`);
+ await retryOrFail(rows, `network: ${err.message}`);
return { purged: 0, retrying: rows.length, failed: 0 };
return { purged: 0, retrying: rows.length, failed: 0 };
}
}
const errors = (r.body && Array.isArray(r.body.errors) ? r.body.errors : []).map(e => `${e.code}: ${e.message}`).join('; ');
const errors = (r.body && Array.isArray(r.body.errors) ? r.body.errors : []).map(e => `${e.code}: ${e.message}`).join('; ');
if (r.status >= 200 && r.status < 300 && r.body && r.body.success === true) {
if (r.status >= 200 && r.status < 300 && r.body && r.body.success === true) {
const id = r.body.result && r.body.result.id ? `cloudflare purge ${r.body.result.id}` : 'purged';
const id = r.body.result && r.body.result.id ? `cloudflare purge ${r.body.result.id}` : 'purged';
- mark(rows, x => q.purged.run(clip(id), now(), x.seq));
+ await mark(rows, async x => await q.purged.run(clip(id), now(), x.seq));
return { purged: rows.length, retrying: 0, failed: 0 };
return { purged: rows.length, retrying: 0, failed: 0 };
}
}
if (r.status === 429 || r.status >= 500 || r.status === 408) {
if (r.status === 429 || r.status >= 500 || r.status === 408) {
- retryOrFail(rows, `HTTP ${r.status} ${errors}`);
+ await retryOrFail(rows, `HTTP ${r.status} ${errors}`);
return { purged: 0, retrying: rows.length, failed: 0 };
return { purged: 0, retrying: rows.length, failed: 0 };
}
}
// Refused (bad URL, token without purge permission, unknown zone): isolate, then fail.
// Refused (bad URL, token without purge permission, unknown zone): isolate, then fail.
@@ -210,7 +210,7 @@ function createPurger({ db, config, fetchImpl = globalThis.fetch, log = console,
@@ -210,7 +210,7 @@ function createPurger({ db, config, fetchImpl = globalThis.fetch, log = console,
}
}
return total;
return total;
}
}
- mark(rows, x => q.failed.run(clip(`HTTP ${r.status} ${errors || 'refused'}`), now(), x.seq));
+ await mark(rows, async x => await q.failed.run(clip(`HTTP ${r.status} ${errors || 'refused'}`), now(), x.seq));
log.warn(`[purge] Cloudflare refused a purge: HTTP ${r.status} ${clip(errors)}`);
log.warn(`[purge] Cloudflare refused a purge: HTTP ${r.status} ${clip(errors)}`);
return { purged: 0, retrying: 0, failed: rows.length };
return { purged: 0, retrying: 0, failed: rows.length };
}
}
@@ -218,7 +218,7 @@ function createPurger({ db, config, fetchImpl = globalThis.fetch, log = console,
@@ -218,7 +218,7 @@ function createPurger({ db, config, fetchImpl = globalThis.fetch, log = console,
async function doFlush() {
async function doFlush() {
const total = { purged: 0, retrying: 0, failed: 0 };
const total = { purged: 0, retrying: 0, failed: 0 };
if (!token) return total;
if (!token) return total;
- const rows = q.due.all(now(), 200);
+ const rows = await q.due.all(now(), 200);
const byZone = new Map();
const byZone = new Map();
for (const r of rows) {
for (const r of rows) {
if (!byZone.has(r.zone_id)) byZone.set(r.zone_id, []);
if (!byZone.has(r.zone_id)) byZone.set(r.zone_id, []);
@@ -78,35 +78,35 @@ function createSavedSearches(db, { maxPerSubject = 50, now = () => Date.now() }
@@ -78,35 +78,35 @@ function createSavedSearches(db, { maxPerSubject = 50, now = () => Date.now() }
* Save (subject, q, filters) with a name. The same query again returns the existing row
* Save (subject, q, filters) with a name. The same query again returns the existing row
* (renamed when a new name is given): { created, saved } | { error: 'limit' }.
* (renamed when a new name is given): { created, saved } | { error: 'limit' }.
*/
*/
- function save(subject, { name, q, filters }) {
- return db.transaction(() => {
+ async function save(subject, { name, q, filters }) {
+ return await db.tx(async () => {
const f = canonicalFilters(filters);
const f = canonicalFilters(filters);
const hash = hashOf(q, f);
const hash = hashOf(q, f);
const at = now();
const at = now();
- const existing = st.byHash.get(subject, hash);
+ const existing = await st.byHash.get(subject, hash);
if (existing) {
if (existing) {
- if (name && name !== existing.name) st.rename.run(name, at, existing.id);
- return { created: false, saved: view(st.get.get(existing.id, subject)) };
+ if (name && name !== existing.name) await st.rename.run(name, at, existing.id);
+ return { created: false, saved: view(await st.get.get(existing.id, subject)) };
}
}
- if (st.count.get(subject).n >= maxPerSubject) return { error: 'limit' };
+ if ((await st.count.get(subject)).n >= maxPerSubject) return { error: 'limit' };
const id = `svs_${ids.ulid(at).toLowerCase()}`;
const id = `svs_${ids.ulid(at).toLowerCase()}`;
- st.insert.run({ id, subject, name: name || q || 'All documents', q, filters: JSON.stringify(f), query_hash: hash, at });
- return { created: true, saved: view(st.get.get(id, subject)) };
- })();
+ await st.insert.run({ id, subject, name: name || q || 'All documents', q, filters: JSON.stringify(f), query_hash: hash, at });
+ return { created: true, saved: view(await st.get.get(id, subject)) };
+ });
}
}
return {
return {
- list: (subject) => st.list.all(subject).map(view),
+ list: async (subject) => (await st.list.all(subject)).map(view),
/** One saved search of this subject; another subject's id is the same as a missing one. */
/** One saved search of this subject; another subject's id is the same as a missing one. */
- get: (subject, id) => {
+ get: async (subject, id) => {
if (!ID_RE.test(String(id))) return null;
if (!ID_RE.test(String(id))) return null;
- const r = st.get.get(id, subject);
+ const r = await st.get.get(id, subject);
return r ? view(r) : null;
return r ? view(r) : null;
},
},
save,
save,
- remove: (subject, id) => ID_RE.test(String(id)) && st.del.run(id, subject).changes > 0,
- markRun: (id) => st.ran.run(now(), id),
- total: () => st.total.get().n,
+ remove: async (subject, id) => ID_RE.test(String(id)) && (await st.del.run(id, subject)).changes > 0,
+ markRun: async (id) => await st.ran.run(now(), id),
+ total: async () => (await st.total.get()).n,
maxPerSubject,
maxPerSubject,
};
};
}
}
@@ -22,18 +22,18 @@ function createStore({ db, engine, outbox, purges = null, now = () => Date.now()
@@ -22,18 +22,18 @@ function createStore({ db, engine, outbox, purges = null, now = () => Date.now()
insert: db.prepare(`INSERT INTO documents (owner, type, id, revision, deleted, visibility, publication_state, exposure,
insert: db.prepare(`INSERT INTO documents (owner, type, id, revision, deleted, visibility, publication_state, exposure,
index_decision, noindex_reasons, canonical_url, title, language, published_at, updated_at, sort_at, doc, hash, via, event_id, indexed_at)
index_decision, noindex_reasons, canonical_url, title, language, published_at, updated_at, sort_at, doc, hash, via, event_id, indexed_at)
VALUES (@owner, @type, @id, @revision, @deleted, @visibility, @publication_state, @exposure,
VALUES (@owner, @type, @id, @revision, @deleted, @visibility, @publication_state, @exposure,
- @index_decision, @noindex_reasons, @canonical_url, @title, @language, @published_at, @updated_at, @sort_at, @doc, @hash, @via, @event_id, @indexed_at)`),
+ @index_decision, @noindex_reasons, @canonical_url, @title, @language, @published_at, @updated_at, @sort_at, @doc, @hash, @via, @event_id, @indexed_at) RETURNING rid`),
update: db.prepare(`UPDATE documents SET revision = @revision, deleted = @deleted, visibility = @visibility,
update: db.prepare(`UPDATE documents SET revision = @revision, deleted = @deleted, visibility = @visibility,
publication_state = @publication_state, exposure = @exposure, index_decision = @index_decision,
publication_state = @publication_state, exposure = @exposure, index_decision = @index_decision,
noindex_reasons = @noindex_reasons, canonical_url = @canonical_url, title = @title, language = @language,
noindex_reasons = @noindex_reasons, canonical_url = @canonical_url, title = @title, language = @language,
published_at = @published_at, updated_at = @updated_at, sort_at = @sort_at, doc = @doc, hash = @hash,
published_at = @published_at, updated_at = @updated_at, sort_at = @sort_at, doc = @doc, hash = @hash,
via = @via, event_id = @event_id, indexed_at = @indexed_at WHERE rid = @rid`),
via = @via, event_id = @event_id, indexed_at = @indexed_at WHERE rid = @rid`),
delAcl: db.prepare('DELETE FROM doc_acl WHERE rid = ?'),
delAcl: db.prepare('DELETE FROM doc_acl WHERE rid = ?'),
- insAcl: db.prepare('INSERT OR IGNORE INTO doc_acl (rid, kind, value) VALUES (?, ?, ?)'),
+ insAcl: db.prepare('INSERT INTO doc_acl (rid, kind, value) VALUES (?, ?, ?) ON CONFLICT DO NOTHING'),
delFacets: db.prepare('DELETE FROM doc_facets WHERE rid = ?'),
delFacets: db.prepare('DELETE FROM doc_facets WHERE rid = ?'),
- insFacet: db.prepare('INSERT OR IGNORE INTO doc_facets (rid, key, value) VALUES (?, ?, ?)'),
+ insFacet: db.prepare('INSERT INTO doc_facets (rid, key, value) VALUES (?, ?, ?) ON CONFLICT DO NOTHING'),
ownerPage: db.prepare(`SELECT rid, type, id, revision, deleted, hash, exposure, indexed_at FROM documents
ownerPage: db.prepare(`SELECT rid, type, id, revision, deleted, hash, exposure, indexed_at FROM documents
- WHERE owner = @owner AND (@type IS NULL OR type = @type) AND rid > @after ORDER BY rid LIMIT @limit`),
+ WHERE owner = @owner AND (@type::text IS NULL OR type = @type) AND rid > @after ORDER BY rid LIMIT @limit`),
counts: db.prepare(`SELECT COUNT(*) AS total, SUM(deleted) AS tombstones,
counts: db.prepare(`SELECT COUNT(*) AS total, SUM(deleted) AS tombstones,
SUM(CASE WHEN exposure = 3 THEN 1 ELSE 0 END) AS public_listed,
SUM(CASE WHEN exposure = 3 THEN 1 ELSE 0 END) AS public_listed,
SUM(CASE WHEN exposure = 2 THEN 1 ELSE 0 END) AS public_unlisted,
SUM(CASE WHEN exposure = 2 THEN 1 ELSE 0 END) AS public_unlisted,
@@ -64,16 +64,16 @@ function createStore({ db, engine, outbox, purges = null, now = () => Date.now()
@@ -64,16 +64,16 @@ function createStore({ db, engine, outbox, purges = null, now = () => Date.now()
};
};
}
}
- function writeSideRows(rid, doc, exposure) {
- st.delAcl.run(rid);
- st.delFacets.run(rid);
+ async function writeSideRows(rid, doc, exposure) {
+ await st.delAcl.run(rid);
+ await st.delFacets.run(rid);
if (exposure === EXPOSURE.none) return;
if (exposure === EXPOSURE.none) return;
if (exposure === EXPOSURE.restricted) {
if (exposure === EXPOSURE.restricted) {
- for (const s of doc.acl.subjects) st.insAcl.run(rid, 's', s);
- for (const g of doc.acl.groups) st.insAcl.run(rid, 'g', g);
- for (const e of doc.acl.entitlements) st.insAcl.run(rid, 'e', e);
+ for (const s of doc.acl.subjects) await st.insAcl.run(rid, 's', s);
+ for (const g of doc.acl.groups) await st.insAcl.run(rid, 'g', g);
+ for (const e of doc.acl.entitlements) await st.insAcl.run(rid, 'e', e);
}
}
- for (const [k, v] of facetPairs(doc.facets)) st.insFacet.run(rid, k, v);
+ for (const [k, v] of facetPairs(doc.facets)) await st.insFacet.run(rid, k, v);
}
}
function removalReason(prev, doc) {
function removalReason(prev, doc) {
@@ -84,7 +84,7 @@ function createStore({ db, engine, outbox, purges = null, now = () => Date.now()
@@ -84,7 +84,7 @@ function createStore({ db, engine, outbox, purges = null, now = () => Date.now()
return 'noindex';
return 'noindex';
}
}
- function announce(prevRow, prevDoc, doc, exposure, traceId) {
+ async function announce(prevRow, prevDoc, doc, exposure, traceId) {
const prevExposure = prevRow ? prevRow.exposure : EXPOSURE.none;
const prevExposure = prevRow ? prevRow.exposure : EXPOSURE.none;
const subject = { type: 'document', id: `${doc.owner}/${doc.type}/${doc.id}`, revision: doc.revision };
const subject = { type: 'document', id: `${doc.owner}/${doc.type}/${doc.id}`, revision: doc.revision };
const wasPublic = prevExposure >= EXPOSURE.public_unlisted;
const wasPublic = prevExposure >= EXPOSURE.public_unlisted;
@@ -97,12 +97,12 @@ function createStore({ db, engine, outbox, purges = null, now = () => Date.now()
@@ -97,12 +97,12 @@ function createStore({ db, engine, outbox, purges = null, now = () => Date.now()
// Only a URL that was public already: caches and sitemaps purge it.
// Only a URL that was public already: caches and sitemaps purge it.
canonical_url: wasPublic ? prevRow.canonical_url : null,
canonical_url: wasPublic ? prevRow.canonical_url : null,
};
};
- const env = outbox.enqueue({ event_type: 'search.document.removed', subject, trace_id: traceId, payload });
+ const env = await outbox.enqueue({ event_type: 'search.document.removed', subject, trace_id: traceId, payload });
// Search's own consumer: the owners' removal feed and the CDN purge queue (purge.js).
// Search's own consumer: the owners' removal feed and the CDN purge queue (purge.js).
- if (purges) purges.record(env.event_id, payload);
+ if (purges) await purges.record(env.event_id, payload);
}
}
if (!doc.deleted && exposure > EXPOSURE.none) {
if (!doc.deleted && exposure > EXPOSURE.none) {
- outbox.enqueue({
+ await outbox.enqueue({
event_type: 'search.document.indexed',
event_type: 'search.document.indexed',
subject,
subject,
trace_id: traceId,
trace_id: traceId,
@@ -122,10 +122,10 @@ function createStore({ db, engine, outbox, purges = null, now = () => Date.now()
@@ -122,10 +122,10 @@ function createStore({ db, engine, outbox, purges = null, now = () => Date.now()
* Must be called inside a transaction when combined with other writes (inbox); it opens its
* Must be called inside a transaction when combined with other writes (inbox); it opens its
* own otherwise.
* own otherwise.
*/
*/
- function apply(input, { via = 'api', eventId = null, traceId = null } = {}) {
- const run = () => {
+ async function apply(input, { via = 'api', eventId = null, traceId = null } = {}) {
+ const run = async () => {
const { doc, hash } = normalize(input);
const { doc, hash } = normalize(input);
- const cur = st.get.get(doc.owner, doc.type, doc.id);
+ const cur = await st.get.get(doc.owner, doc.type, doc.id);
if (cur) {
if (cur) {
const base = { revision: doc.revision, stored_revision: cur.revision, deleted: Boolean(cur.deleted) };
const base = { revision: doc.revision, stored_revision: cur.revision, deleted: Boolean(cur.deleted) };
if (doc.revision < cur.revision) return { outcome: 'stale', ...base };
if (doc.revision < cur.revision) return { outcome: 'stale', ...base };
@@ -139,45 +139,45 @@ function createStore({ db, engine, outbox, purges = null, now = () => Date.now()
@@ -139,45 +139,45 @@ function createStore({ db, engine, outbox, purges = null, now = () => Date.now()
let rid;
let rid;
if (cur) {
if (cur) {
rid = cur.rid;
rid = cur.rid;
- st.update.run({ ...values, rid });
+ await st.update.run({ ...values, rid });
} else {
} else {
- rid = Number(st.insert.run(values).lastInsertRowid);
+ rid = Number((await st.insert.run(values)).lastInsertRowid);
}
}
- writeSideRows(rid, doc, values.exposure);
- engine.put(rid, doc, values.exposure);
+ await writeSideRows(rid, doc, values.exposure);
+ await engine.put(rid, doc, values.exposure);
const prevDoc = cur ? JSON.parse(cur.doc) : null;
const prevDoc = cur ? JSON.parse(cur.doc) : null;
- announce(cur, prevDoc, doc, values.exposure, traceId);
+ await announce(cur, prevDoc, doc, values.exposure, traceId);
return { outcome: 'applied', revision: doc.revision, stored_revision: cur ? cur.revision : null, deleted: doc.deleted };
return { outcome: 'applied', revision: doc.revision, stored_revision: cur ? cur.revision : null, deleted: doc.deleted };
};
};
- return db.inTransaction ? run() : db.transaction(run)();
+ return db.inTransaction() ? await run() : await db.tx(run);
}
}
/**
/**
* Tombstone a document. revision omitted = the stored revision (a deletion wins the tie), or
* Tombstone a document. revision omitted = the stored revision (a deletion wins the tie), or
* 0 for a document Search has never seen (so a late older upsert is still refused).
* 0 for a document Search has never seen (so a late older upsert is still refused).
*/
*/
- function remove(owner, type, id, { revision = null, via = 'api', eventId = null, traceId = null } = {}) {
- const run = () => {
- const cur = st.get.get(owner, type, id);
+ async function remove(owner, type, id, { revision = null, via = 'api', eventId = null, traceId = null } = {}) {
+ const run = async () => {
+ const cur = await st.get.get(owner, type, id);
const rev = revision != null ? revision : (cur ? cur.revision : 0);
const rev = revision != null ? revision : (cur ? cur.revision : 0);
- return apply({ owner, type, id, revision: rev, deleted: true }, { via, eventId, traceId });
+ return await apply({ owner, type, id, revision: rev, deleted: true }, { via, eventId, traceId });
};
};
- return db.inTransaction ? run() : db.transaction(run)();
+ return db.inTransaction() ? await run() : await db.tx(run);
}
}
- function get(owner, type, id) {
- const row = st.get.get(owner, type, id);
+ async function get(owner, type, id) {
+ const row = await st.get.get(owner, type, id);
return row ? { row, doc: JSON.parse(row.doc) } : null;
return row ? { row, doc: JSON.parse(row.doc) } : null;
}
}
- function byRid(rid) {
- const row = st.byRid.get(rid);
+ async function byRid(rid) {
+ const row = await st.byRid.get(rid);
return row ? { row, doc: JSON.parse(row.doc) } : null;
return row ? { row, doc: JSON.parse(row.doc) } : null;
}
}
/** Reconciliation page for an owner: (type, id, revision, deleted, hash) by insertion order. */
/** Reconciliation page for an owner: (type, id, revision, deleted, hash) by insertion order. */
- function ownerPage(owner, { type = null, after = 0, limit = 500 } = {}) {
- const rows = st.ownerPage.all({ owner, type, after, limit: limit + 1 });
+ async function ownerPage(owner, { type = null, after = 0, limit = 500 } = {}) {
+ const rows = await st.ownerPage.all({ owner, type, after, limit: limit + 1 });
const more = rows.length > limit;
const more = rows.length > limit;
const page = rows.slice(0, limit);
const page = rows.slice(0, limit);
return {
return {
@@ -189,8 +189,8 @@ function createStore({ db, engine, outbox, purges = null, now = () => Date.now()
@@ -189,8 +189,8 @@ function createStore({ db, engine, outbox, purges = null, now = () => Date.now()
};
};
}
}
- function counts() {
- const c = st.counts.get();
+ async function counts() {
+ const c = await st.counts.get();
return {
return {
total: c.total || 0, tombstones: c.tombstones || 0, public_listed: c.public_listed || 0,
total: c.total || 0, tombstones: c.tombstones || 0, public_listed: c.public_listed || 0,
public_unlisted: c.public_unlisted || 0, restricted: c.restricted || 0,
public_unlisted: c.public_unlisted || 0, restricted: c.restricted || 0,
@@ -175,7 +175,7 @@ function pageRouter({ searcher, auth, baseUrl = 'https://search.openvibe.network
@@ -175,7 +175,7 @@ function pageRouter({ searcher, auth, baseUrl = 'https://search.openvibe.network
.send('User-agent: *\nAllow: /$\nAllow: /updates\nDisallow: /?\nDisallow: /api/\nDisallow: /internal/\n');
.send('User-agent: *\nAllow: /$\nAllow: /updates\nDisallow: /?\nDisallow: /api/\nDisallow: /internal/\n');
});
});
- router.get('/', (req, res, next) => {
+ router.get('/', async (req, res, next) => {
const wantsHtml = /\btext\/html\b/.test(String(req.get('accept') || ''));
const wantsHtml = /\btext\/html\b/.test(String(req.get('accept') || ''));
res.setHeader('Cache-Control', 'no-store');
res.setHeader('Cache-Control', 'no-store');
res.setHeader('Vary', 'Accept, Cookie');
res.setHeader('Vary', 'Accept, Cookie');
@@ -202,7 +202,7 @@ function pageRouter({ searcher, auth, baseUrl = 'https://search.openvibe.network
@@ -202,7 +202,7 @@ function pageRouter({ searcher, auth, baseUrl = 'https://search.openvibe.network
let status = 200;
let status = 200;
if (searching) {
if (searching) {
try {
try {
- const out = searcher.run({ text: q, query: { ...(owner ? { owner } : {}), ...(type ? { type } : {}) }, viewer: viewer || ANONYMOUS, cursor });
+ const out = await searcher.run({ text: q, query: { ...(owner ? { owner } : {}), ...(type ? { type } : {}) }, viewer: viewer || ANONYMOUS, cursor });
body = out.results.length
body = out.results.length
? resultsHtml(out.results)
? resultsHtml(out.results)
: `<p class="empty">${cursor ? 'No more results.' : 'Nothing found. The index is young: services add documents as they publish them.'}</p>`;
: `<p class="empty">${cursor ? 'No more results.' : 'Nothing found. The index is young: services add documents as they publish them.'}</p>`;
@@ -215,7 +215,7 @@ t('a visibility change or deletion leaves results at once and announces search.d
@@ -215,7 +215,7 @@ t('a visibility change or deletion leaves results at once and announces search.d
await request(svc.base, 'DELETE', '/api/v1/documents/wiki/page/flip?revision=3', { token: WIKI });
await request(svc.base, 'DELETE', '/api/v1/documents/wiki/page/flip?revision=3', { token: WIKI });
assert.deepStrictEqual(idsOf(await q('q=iotaword', asUser(alice))), []);
assert.deepStrictEqual(idsOf(await q('q=iotaword', asUser(alice))), []);
- const removed = svc.outbox.all().filter(e => e.event_type === 'search.document.removed' && e.payload.id === 'flip');
+ const removed = (await svc.outbox.all()).filter(e => e.event_type === 'search.document.removed' && e.payload.id === 'flip');
assert.strictEqual(removed.length, 2);
assert.strictEqual(removed.length, 2);
assert.deepStrictEqual(removed.map(e => [e.payload.reason, e.payload.previous_exposure, e.payload.exposure]), [
assert.deepStrictEqual(removed.map(e => [e.payload.reason, e.payload.previous_exposure, e.payload.exposure]), [
['visibility_changed', 'public_listed', 'restricted'],
['visibility_changed', 'public_listed', 'restricted'],
@@ -233,7 +233,7 @@ t('publishing → draft and public → noindex also announce removal', async ()
@@ -233,7 +233,7 @@ t('publishing → draft and public → noindex also announce removal', async ()
const d2 = doc({ id: 'noidx', title: 'Lambdaword', revision: 1 });
const d2 = doc({ id: 'noidx', title: 'Lambdaword', revision: 1 });
await put(d2);
await put(d2);
await put({ ...d2, revision: 2, indexability: { decision: 'noindex', reasons: ['duplicate_without_canonical'] } });
await put({ ...d2, revision: 2, indexability: { decision: 'noindex', reasons: ['duplicate_without_canonical'] } });
- const reasons = svc.outbox.all().filter(e => e.event_type === 'search.document.removed' && ['unpub', 'noidx'].includes(e.payload.id)).map(e => `${e.payload.id}:${e.payload.reason}`);
+ const reasons = (await svc.outbox.all()).filter(e => e.event_type === 'search.document.removed' && ['unpub', 'noidx'].includes(e.payload.id)).map(e => `${e.payload.id}:${e.payload.reason}`);
assert.deepStrictEqual(reasons, ['unpub:not_published', 'noidx:noindex']);
assert.deepStrictEqual(reasons, ['unpub:not_published', 'noidx:noindex']);
assert.deepStrictEqual(idsOf(await q('q=kappaword')), []);
assert.deepStrictEqual(idsOf(await q('q=kappaword')), []);
assert.deepStrictEqual(idsOf(await q('q=lambdaword')), []);
assert.deepStrictEqual(idsOf(await q('q=lambdaword')), []);
@@ -60,7 +60,7 @@ t('indexing: 120 a minute per service principal, refused before the store is tou
@@ -60,7 +60,7 @@ t('indexing: 120 a minute per service principal, refused before the store is tou
const d = doc({ id: 'pg_over' });
const d = doc({ id: 'pg_over' });
const r = await call('PUT', `/api/v1/documents/wiki/page/${d.id}`, { token: WIKI, body: d });
const r = await call('PUT', `/api/v1/documents/wiki/page/${d.id}`, { token: WIKI, body: d });
assert.deepStrictEqual([r.status, r.body.code, r.headers.get('retry-after')], [429, 'rate_limited', '60']);
assert.deepStrictEqual([r.status, r.body.code, r.headers.get('retry-after')], [429, 'rate_limited', '60']);
- assert.strictEqual(svc.store.get('wiki', 'page', 'pg_over'), null, 'nothing stored');
+ assert.strictEqual(await svc.store.get('wiki', 'page', 'pg_over'), null, 'nothing stored');
const b = doc({ owner: 'blog', type: 'post', id: 'post_1', canonical_url: 'https://openvibe.blog/p/1' });
const b = doc({ owner: 'blog', type: 'post', id: 'post_1', canonical_url: 'https://openvibe.blog/p/1' });
assert.strictEqual((await call('PUT', '/api/v1/documents/blog/post/post_1', { token: BLOG, body: b })).status, 200, 'another service still indexes');
assert.strictEqual((await call('PUT', '/api/v1/documents/blog/post/post_1', { token: BLOG, body: b })).status, 200, 'another service still indexes');
});
});
new file mode 100644
new file mode 100644
@@ -0,0 +1,11 @@
@@ -0,0 +1,11 @@
+'use strict';
+/**
+ * A migrated database for one test boot (ADR-035), from openvibe-sdk/testing: PGlite by default; with
+ * SEARCH_TEST_STORE=pg (npm run test:pg) the PostgreSQL + PgBouncer containers, with roles and a schema of its own.
+ */
+const { createTestDb, pgAvailable } = require('openvibe-sdk/testing');
+const { MIGRATIONS } = require('../server/db');
+
+const testDb = ({ store = process.env.SEARCH_TEST_STORE || 'pglite', max = 4 } = {}) => createTestDb({ migrations: MIGRATIONS, store, service: 'search', max });
+
+module.exports = { testDb, pgAvailable };
@@ -6,7 +6,7 @@
@@ -6,7 +6,7 @@
*/
*/
const assert = require('assert');
const assert = require('assert');
const { boot, request, doc, suite } = require('./helpers');
const { boot, request, doc, suite } = require('./helpers');
-const { freshnessBoost } = require('../server/engine/fts5');
+const { freshnessBoost } = require('../server/engine/pg');
const t = suite('freshness');
const t = suite('freshness');
const NOW = Date.parse('2026-09-23T12:00:00Z');
const NOW = Date.parse('2026-09-23T12:00:00Z');
@@ -31,9 +31,9 @@ async function corpus(env = {}) {
@@ -31,9 +31,9 @@ async function corpus(env = {}) {
const svc = await boot({ env, now: () => clock });
const svc = await boot({ env, now: () => clock });
// Same text, same length: bm25 ties, so only freshness separates them. The old one is indexed
// Same text, same length: bm25 ties, so only freshness separates them. The old one is indexed
// first (lower rid), so without freshness it would come first.
// first (lower rid), so without freshness it would come first.
- svc.store.apply(doc({ id: 'old', title: 'Aurora report', body: 'aurora seen tonight', published_at: daysAgo(400), updated_at: daysAgo(400) }));
- svc.store.apply(doc({ id: 'mid', title: 'Aurora report', body: 'aurora seen tonight', published_at: daysAgo(20), updated_at: daysAgo(20) }));
- svc.store.apply(doc({ id: 'new', title: 'Aurora report', body: 'aurora seen tonight', published_at: daysAgo(1), updated_at: daysAgo(1) }));
+ await svc.store.apply(doc({ id: 'old', title: 'Aurora report', body: 'aurora seen tonight', published_at: daysAgo(400), updated_at: daysAgo(400) }));
+ await svc.store.apply(doc({ id: 'mid', title: 'Aurora report', body: 'aurora seen tonight', published_at: daysAgo(20), updated_at: daysAgo(20) }));
+ await svc.store.apply(doc({ id: 'new', title: 'Aurora report', body: 'aurora seen tonight', published_at: daysAgo(1), updated_at: daysAgo(1) }));
return { svc, setClock: (c) => { clock = c; } };
return { svc, setClock: (c) => { clock = c; } };
}
}
@@ -58,8 +58,8 @@ t('relevance still wins: an old title match outranks a fresh passing mention', a
@@ -58,8 +58,8 @@ t('relevance still wins: an old title match outranks a fresh passing mention', a
let clock = NOW;
let clock = NOW;
const svc = await boot({ now: () => clock });
const svc = await boot({ now: () => clock });
try {
try {
- svc.store.apply(doc({ id: 'fresh-mention', title: 'Weekly notes', summary: 'Many topics this week.', body: `${'filler words about other things '.repeat(40)} glacier`, published_at: daysAgo(0) }));
- svc.store.apply(doc({ id: 'old-title', title: 'Glacier glacier retreat', summary: 'The glacier retreat, measured.', body: 'glacier data', published_at: daysAgo(700) }));
+ await svc.store.apply(doc({ id: 'fresh-mention', title: 'Weekly notes', summary: 'Many topics this week.', body: `${'filler words about other things '.repeat(40)} glacier`, published_at: daysAgo(0) }));
+ await svc.store.apply(doc({ id: 'old-title', title: 'Glacier glacier retreat', summary: 'The glacier retreat, measured.', body: 'glacier data', published_at: daysAgo(700) }));
const r = await request(svc.base, 'GET', '/api/v1/search?q=glacier');
const r = await request(svc.base, 'GET', '/api/v1/search?q=glacier');
assert.deepStrictEqual(r.body.results.map(x => x.id), ['old-title', 'fresh-mention']);
assert.deepStrictEqual(r.body.results.map(x => x.id), ['old-title', 'fresh-mention']);
} finally { await svc.stop(); }
} finally { await svc.stop(); }
@@ -56,9 +56,11 @@ async function boot({ env = {}, fetchImpl, tokenClient, now } = {}) {
@@ -56,9 +56,11 @@ async function boot({ env = {}, fetchImpl, tokenClient, now } = {}) {
SEARCH_EVENTS_SECRET: WEBHOOK_SECRET,
SEARCH_EVENTS_SECRET: WEBHOOK_SECRET,
...env,
...env,
});
});
- const h = await start({ config, log: silent, fetchImpl, tokenClient, now });
+ // One database per boot (PGlite, or SEARCH_TEST_STORE=pg: the containers), dropped when the boot stops.
+ const testdb = await require('./db').testDb();
+ const h = await start({ config, db: testdb.db, log: silent, fetchImpl, tokenClient, now });
const base = `http://127.0.0.1:${h.server.address().port}`;
const base = `http://127.0.0.1:${h.server.address().port}`;
- return { ...h, base, dir, async stop() { await h.close(); } };
+ return { ...h, base, dir, async stop() { await h.close(); await testdb.close(); } };
}
}
async function request(base, method, p, { token, body, headers = {}, raw } = {}) {
async function request(base, method, p, { token, body, headers = {}, raw } = {}) {
@@ -122,7 +124,7 @@ async function deliver(base, event, { secret = WEBHOOK_SECRET, seq = 1, badSigna
@@ -122,7 +124,7 @@ async function deliver(base, event, { secret = WEBHOOK_SECRET, seq = 1, badSigna
const key = badSignature ? 'wrong-secret-' + 'y'.repeat(32) : secret;
const key = badSignature ? 'wrong-secret-' + 'y'.repeat(32) : secret;
const ts = Math.floor(now / 1000);
const ts = Math.floor(now / 1000);
const v2 = v1Only ? {} : { 'X-OpenVibe-Timestamp': String(ts), 'X-OpenVibe-Signature-V2': signV2(raw, key, ts) };
const v2 = v1Only ? {} : { 'X-OpenVibe-Timestamp': String(ts), 'X-OpenVibe-Signature-V2': signV2(raw, key, ts) };
- return request(base, 'POST', '/internal/events', {
+ return await request(base, 'POST', '/internal/events', {
raw,
raw,
headers: { 'Content-Type': 'application/json', 'X-OpenVibe-Signature': sign(raw, key), ...v2, 'X-OpenVibe-Event-Id': event.event_id },
headers: { 'Content-Type': 'application/json', 'X-OpenVibe-Signature': sign(raw, key), ...v2, 'X-OpenVibe-Event-Id': event.event_id },
});
});
@@ -39,7 +39,7 @@ t('/api/ready: every check reports; db is the only required one; the index agree
@@ -39,7 +39,7 @@ t('/api/ready: every check reports; db is the only required one; the index agree
assert.ok(Date.parse(c.checked_at));
assert.ok(Date.parse(c.checked_at));
}
}
assert.deepStrictEqual(r.body.checks.index.detail, { indexed: { public: 2, restricted: 1 }, expected: { public: 2, restricted: 1 } });
assert.deepStrictEqual(r.body.checks.index.detail, { indexed: { public: 2, restricted: 1 }, expected: { public: 2, restricted: 1 } });
- assert.strictEqual(r.body.engine, 'sqlite-fts5');
+ assert.strictEqual(r.body.engine, 'postgresql-fts');
assert.strictEqual(r.body.documents.total, 4);
assert.strictEqual(r.body.documents.total, 4);
});
});
@@ -84,12 +84,12 @@ t('a full-text index out of step with the documents degrades /api/ready (still 2
@@ -84,12 +84,12 @@ t('a full-text index out of step with the documents degrades /api/ready (still 2
svc = await boot(); // fresh: the index check has no cached result
svc = await boot(); // fresh: the index check has no cached result
await put(doc());
await put(doc());
await put(doc());
await put(doc());
- svc.db.prepare('DELETE FROM fts_public WHERE rowid = (SELECT MIN(rowid) FROM fts_public)').run();
+ await svc.db.prepare('DELETE FROM search_fts WHERE rid = (SELECT MIN(rid) FROM search_fts WHERE audience = 3)').run();
const r = await request(svc.base, 'GET', '/api/ready');
const r = await request(svc.base, 'GET', '/api/ready');
assert.strictEqual(r.status, 200, r.text);
assert.strictEqual(r.status, 200, r.text);
assert.strictEqual(r.body.status, 'degraded');
assert.strictEqual(r.body.status, 'degraded');
assert.deepStrictEqual(r.body.degraded, ['index']);
assert.deepStrictEqual(r.body.degraded, ['index']);
- assert.match(r.body.checks.index.error, /fts_public has 1 entries for 2 public_listed documents/);
+ assert.match(r.body.checks.index.error, /the public index has 1 entries for 2 public_listed documents/);
});
});
t('a broken database makes the service unready (503); /metrics still answers', async () => {
t('a broken database makes the service unready (503); /metrics still answers', async () => {
@@ -36,12 +36,12 @@ t('outbox envelopes are valid events.event-envelope@1 and are enqueued only with
@@ -36,12 +36,12 @@ t('outbox envelopes are valid events.event-envelope@1 and are enqueued only with
await request(svc.base, 'PUT', `/api/v1/documents/wiki/page/${d.id}`, { token: WIKI, body: d });
await request(svc.base, 'PUT', `/api/v1/documents/wiki/page/${d.id}`, { token: WIKI, body: d });
await request(svc.base, 'PUT', `/api/v1/documents/wiki/page/${d.id}`, { token: WIKI, body: { ...d, revision: 0 } }); // stale: no event
await request(svc.base, 'PUT', `/api/v1/documents/wiki/page/${d.id}`, { token: WIKI, body: { ...d, revision: 0 } }); // stale: no event
await request(svc.base, 'DELETE', `/api/v1/documents/wiki/page/${d.id}?revision=2`, { token: WIKI });
await request(svc.base, 'DELETE', `/api/v1/documents/wiki/page/${d.id}?revision=2`, { token: WIKI });
- const all = svc.outbox.all();
+ const all = await svc.outbox.all();
assert.deepStrictEqual(all.map(e => e.event_type), ['search.document.indexed', 'search.document.removed']);
assert.deepStrictEqual(all.map(e => e.event_type), ['search.document.indexed', 'search.document.removed']);
for (const e of all) assert.ok(validate('events.event-envelope@1', e).valid, JSON.stringify(validate('events.event-envelope@1', e).errors));
for (const e of all) assert.ok(validate('events.event-envelope@1', e).valid, JSON.stringify(validate('events.event-envelope@1', e).errors));
assert.strictEqual(all[1].subject.id, `wiki/page/${d.id}`);
assert.strictEqual(all[1].subject.id, `wiki/page/${d.id}`);
assert.strictEqual(all[1].subject.revision, 2);
assert.strictEqual(all[1].subject.revision, 2);
- assert.throws(() => svc.outbox.enqueue({ event_type: 'search.document.indexed', subject: { type: 'document', id: 'x' } }), /inside the transaction/);
+ await assert.rejects(async () => await svc.outbox.enqueue({ event_type: 'search.document.indexed', subject: { type: 'document', id: 'x' } }), /inside the transaction/);
} finally {
} finally {
await svc.stop();
await svc.stop();
}
}
@@ -55,8 +55,8 @@ t('the relay publishes pending events with the service token and marks them sent
@@ -55,8 +55,8 @@ t('the relay publishes pending events with the service token and marks them sent
await request(svc.base, 'PUT', `/api/v1/documents/wiki/page/${d.id}`, { token: WIKI, body: d });
await request(svc.base, 'PUT', `/api/v1/documents/wiki/page/${d.id}`, { token: WIKI, body: d });
await request(svc.base, 'DELETE', `/api/v1/documents/wiki/page/${d.id}?revision=2`, { token: WIKI });
await request(svc.base, 'DELETE', `/api/v1/documents/wiki/page/${d.id}?revision=2`, { token: WIKI });
// a write kicks the relay; a flush already in flight may predate the second row
// a write kicks the relay; a flush already in flight may predate the second row
- for (let i = 0; i < 5 && svc.outbox.pending(); i++) await svc.relay.flush();
- assert.strictEqual(svc.outbox.pending(), 0);
+ for (let i = 0; i < 5 && await svc.outbox.pending(); i++) await svc.relay.flush();
+ assert.strictEqual(await svc.outbox.pending(), 0);
const published = events.calls.flatMap(c => (c.body.events ? c.body.events : [c.body]));
const published = events.calls.flatMap(c => (c.body.events ? c.body.events : [c.body]));
assert.deepStrictEqual(published.map(e => e.event_type).sort(), ['search.document.indexed', 'search.document.removed']);
assert.deepStrictEqual(published.map(e => e.event_type).sort(), ['search.document.indexed', 'search.document.removed']);
assert.ok(events.calls.every(c => c.headers.authorization === 'Bearer relay-token'));
assert.ok(events.calls.every(c => c.headers.authorization === 'Bearer relay-token'));
@@ -79,13 +79,13 @@ t('an unreachable or failing Events keeps rows pending for retry; a refused row
@@ -79,13 +79,13 @@ t('an unreachable or failing Events keeps rows pending for retry; a refused row
await request(svc.base, 'PUT', '/api/v1/documents/wiki/page/fine', { token: WIKI, body: doc({ id: 'fine' }) });
await request(svc.base, 'PUT', '/api/v1/documents/wiki/page/fine', { token: WIKI, body: doc({ id: 'fine' }) });
await svc.relay.flush();
await svc.relay.flush();
await svc.relay.flush();
await svc.relay.flush();
- assert.strictEqual(svc.outbox.pending(), 2, 'kept for retry');
+ assert.strictEqual(await svc.outbox.pending(), 2, 'kept for retry');
mode = 'up';
mode = 'up';
- svc.db.prepare('UPDATE event_outbox SET next_attempt_at = 0').run();
+ await svc.db.prepare('UPDATE event_outbox SET next_attempt_at = 0').run();
await svc.relay.flush();
await svc.relay.flush();
await svc.relay.flush();
await svc.relay.flush();
- assert.strictEqual(svc.outbox.pending(), 0);
- assert.strictEqual(svc.outbox.rejected(), 1, 'the poison row is set aside');
+ assert.strictEqual(await svc.outbox.pending(), 0);
+ assert.strictEqual(await svc.outbox.rejected(), 1, 'the poison row is set aside');
} finally {
} finally {
await svc.stop();
await svc.stop();
await events.close();
await events.close();
@@ -97,7 +97,7 @@ t('readiness reports the outbox and the webhook state', async () => {
@@ -97,7 +97,7 @@ t('readiness reports the outbox and the webhook state', async () => {
try {
try {
const r = await request(svc.base, 'GET', '/api/ready');
const r = await request(svc.base, 'GET', '/api/ready');
assert.strictEqual(r.status, 200);
assert.strictEqual(r.status, 200);
- assert.strictEqual(r.body.engine, 'sqlite-fts5');
+ assert.strictEqual(r.body.engine, 'postgresql-fts');
assert.match(r.body.outbox.relay, /^off/);
assert.match(r.body.outbox.relay, /^off/);
assert.strictEqual(r.body.webhook, 'on');
assert.strictEqual(r.body.webhook, 'on');
} finally {
} finally {
@@ -15,10 +15,10 @@ const alice = ids.newId('user');
@@ -15,10 +15,10 @@ const alice = ids.newId('user');
t('boot', async () => {
t('boot', async () => {
svc = await boot();
svc = await boot();
- svc.store.apply(doc({ id: 'pub', title: 'Kestrel <script>alert(1)</script> nesting', summary: 'Kestrel "boxes" & more', body: 'kestrel kestrel', canonical_url: 'https://openvibe.wiki/p/kestrel?a=1&b=2' }));
- svc.store.apply(doc({ id: 'prv', visibility: 'private', acl: { subjects: [alice] }, title: 'Kestrel private diary', body: 'kestrel' }));
- svc.store.apply(doc({ id: 'drf', visibility: 'draft', title: 'Kestrel draft', body: 'kestrel' }));
- for (let i = 0; i < 25; i++) svc.store.apply(doc({ id: `many${i}`, title: `Heron sighting ${i}`, body: 'heron' }));
+ await svc.store.apply(doc({ id: 'pub', title: 'Kestrel <script>alert(1)</script> nesting', summary: 'Kestrel "boxes" & more', body: 'kestrel kestrel', canonical_url: 'https://openvibe.wiki/p/kestrel?a=1&b=2' }));
+ await svc.store.apply(doc({ id: 'prv', visibility: 'private', acl: { subjects: [alice] }, title: 'Kestrel private diary', body: 'kestrel' }));
+ await svc.store.apply(doc({ id: 'drf', visibility: 'draft', title: 'Kestrel draft', body: 'kestrel' }));
+ for (let i = 0; i < 25; i++) await svc.store.apply(doc({ id: `many${i}`, title: `Heron sighting ${i}`, body: 'heron' }));
});
});
t('the front page is a search form, indexable, no-store, with a strict CSP', async () => {
t('the front page is a search form, indexable, no-store, with a strict CSP', async () => {
@@ -55,9 +55,9 @@ t('inert without CLOUDFLARE_PURGE_TOKEN: removals are recorded for owners, nothi
@@ -55,9 +55,9 @@ t('inert without CLOUDFLARE_PURGE_TOKEN: removals are recorded for owners, nothi
const svc = await boot({ env: { CLOUDFLARE_ZONE_IDS: ZONES }, fetchImpl: cf.fetchImpl });
const svc = await boot({ env: { CLOUDFLARE_ZONE_IDS: ZONES }, fetchImpl: cf.fetchImpl });
try {
try {
const d = doc({ id: 'inert1', canonical_url: 'https://openvibe.wiki/p/inert1' });
const d = doc({ id: 'inert1', canonical_url: 'https://openvibe.wiki/p/inert1' });
- svc.store.apply(d);
- svc.store.apply(hide(d));
- assert.deepStrictEqual(svc.purges.cdnCounts(), { pending: 0, purged: 0, failed: 0, skipped: 0 });
+ await svc.store.apply(d);
+ await svc.store.apply(hide(d));
+ assert.deepStrictEqual(await svc.purges.cdnCounts(), { pending: 0, purged: 0, failed: 0, skipped: 0 });
assert.deepStrictEqual(await svc.purger.flush(), { purged: 0, retrying: 0, failed: 0 });
assert.deepStrictEqual(await svc.purger.flush(), { purged: 0, retrying: 0, failed: 0 });
assert.strictEqual(svc.purger.running(), false);
assert.strictEqual(svc.purger.running(), false);
assert.strictEqual(cf.calls.length, 0);
assert.strictEqual(cf.calls.length, 0);
@@ -71,7 +71,7 @@ t('inert without CLOUDFLARE_PURGE_TOKEN: removals are recorded for owners, nothi
@@ -71,7 +71,7 @@ t('inert without CLOUDFLARE_PURGE_TOKEN: removals are recorded for owners, nothi
assert.strictEqual(r.previous_exposure, 'public_listed');
assert.strictEqual(r.previous_exposure, 'public_listed');
assert.strictEqual(r.exposure, 'restricted');
assert.strictEqual(r.exposure, 'restricted');
// The removal row and the outbox event are the same removal.
// The removal row and the outbox event are the same removal.
- const ev = svc.outbox.all().find(e => e.event_type === 'search.document.removed' && e.payload.id === 'inert1');
+ const ev = (await svc.outbox.all()).find(e => e.event_type === 'search.document.removed' && e.payload.id === 'inert1');
assert.strictEqual(r.event_id, ev.event_id);
assert.strictEqual(r.event_id, ev.event_id);
const ready = await request(svc.base, 'GET', '/api/ready');
const ready = await request(svc.base, 'GET', '/api/ready');
assert.match(JSON.stringify(ready.body), /CLOUDFLARE_PURGE_TOKEN unset/);
assert.match(JSON.stringify(ready.body), /CLOUDFLARE_PURGE_TOKEN unset/);
@@ -82,11 +82,11 @@ t('owner removal feed: owner token only, own owner only, cursor paging, restrict
@@ -82,11 +82,11 @@ t('owner removal feed: owner token only, own owner only, cursor paging, restrict
const svc = await boot();
const svc = await boot();
try {
try {
const members = doc({ id: 'mem1', visibility: 'members', acl: { groups: ['g1'] } });
const members = doc({ id: 'mem1', visibility: 'members', acl: { groups: ['g1'] } });
- svc.store.apply(members);
- svc.store.remove('wiki', 'page', 'mem1');
+ await svc.store.apply(members);
+ await svc.store.remove('wiki', 'page', 'mem1');
const pub = doc({ id: 'pub2', canonical_url: 'https://openvibe.wiki/p/pub2' });
const pub = doc({ id: 'pub2', canonical_url: 'https://openvibe.wiki/p/pub2' });
- svc.store.apply(pub);
- svc.store.remove('wiki', 'page', 'pub2');
+ await svc.store.apply(pub);
+ await svc.store.remove('wiki', 'page', 'pub2');
assert.strictEqual((await request(svc.base, 'GET', '/api/v1/owners/wiki/removals')).status, 401);
assert.strictEqual((await request(svc.base, 'GET', '/api/v1/owners/wiki/removals')).status, 401);
assert.strictEqual((await request(svc.base, 'GET', '/api/v1/owners/wiki/removals', { token: BLOG })).status, 403);
assert.strictEqual((await request(svc.base, 'GET', '/api/v1/owners/wiki/removals', { token: BLOG })).status, 403);
const p1 = await request(svc.base, 'GET', '/api/v1/owners/wiki/removals?limit=1', { token: WIKI });
const p1 = await request(svc.base, 'GET', '/api/v1/owners/wiki/removals?limit=1', { token: WIKI });
@@ -105,12 +105,12 @@ t('with a token: the formerly public URL and its sitemap are purged in one call
@@ -105,12 +105,12 @@ t('with a token: the formerly public URL and its sitemap are purged in one call
const a = doc({ id: 'cf1', canonical_url: 'https://openvibe.wiki/p/cf1' });
const a = doc({ id: 'cf1', canonical_url: 'https://openvibe.wiki/p/cf1' });
const b = doc({ id: 'cf2', canonical_url: 'https://openvibe.wiki/p/cf2' });
const b = doc({ id: 'cf2', canonical_url: 'https://openvibe.wiki/p/cf2' });
const c = doc({ owner: 'wiki', id: 'cf3', canonical_url: 'https://docs.openvibe.network/cf3' });
const c = doc({ owner: 'wiki', id: 'cf3', canonical_url: 'https://docs.openvibe.network/cf3' });
- for (const d of [a, b, c]) svc.store.apply(d);
- svc.store.apply(hide(a));
- svc.store.remove('wiki', 'page', 'cf2');
- svc.store.apply({ ...c, revision: 2, publication_state: 'retracted' });
+ for (const d of [a, b, c]) await svc.store.apply(d);
+ await svc.store.apply(hide(a));
+ await svc.store.remove('wiki', 'page', 'cf2');
+ await svc.store.apply({ ...c, revision: 2, publication_state: 'retracted' });
// Two removals on openvibe.wiki share one pending sitemap purge.
// Two removals on openvibe.wiki share one pending sitemap purge.
- assert.deepStrictEqual(svc.purges.cdnCounts(), { pending: 5, purged: 0, failed: 0, skipped: 0 });
+ assert.deepStrictEqual(await svc.purges.cdnCounts(), { pending: 5, purged: 0, failed: 0, skipped: 0 });
const s = await svc.purger.flush();
const s = await svc.purger.flush();
assert.deepStrictEqual(s, { purged: 5, retrying: 0, failed: 0 });
assert.deepStrictEqual(s, { purged: 5, retrying: 0, failed: 0 });
@@ -126,7 +126,7 @@ t('with a token: the formerly public URL and its sitemap are purged in one call
@@ -126,7 +126,7 @@ t('with a token: the formerly public URL and its sitemap are purged in one call
// Done rows are never sent again; the token is never stored.
// Done rows are never sent again; the token is never stored.
assert.deepStrictEqual(await svc.purger.flush(), { purged: 0, retrying: 0, failed: 0 });
assert.deepStrictEqual(await svc.purger.flush(), { purged: 0, retrying: 0, failed: 0 });
assert.strictEqual(cf.calls.length, 2);
assert.strictEqual(cf.calls.length, 2);
- const dump = JSON.stringify(svc.db.prepare('SELECT * FROM cdn_purges').all()) + JSON.stringify(svc.db.prepare('SELECT * FROM removals').all());
+ const dump = JSON.stringify(await svc.db.prepare('SELECT * FROM cdn_purges').all()) + JSON.stringify(await svc.db.prepare('SELECT * FROM removals').all());
assert.ok(!dump.includes(TOKEN));
assert.ok(!dump.includes(TOKEN));
assert.strictEqual(svc.purger.running(), true);
assert.strictEqual(svc.purger.running(), true);
} finally { await svc.stop(); }
} finally { await svc.stop(); }
@@ -137,12 +137,12 @@ t('an index change that does not lower exposure, or a URL that was never public,
@@ -137,12 +137,12 @@ t('an index change that does not lower exposure, or a URL that was never public,
const svc = await boot({ env: { CLOUDFLARE_PURGE_TOKEN: TOKEN, CLOUDFLARE_ZONE_IDS: ZONES }, fetchImpl: cf.fetchImpl });
const svc = await boot({ env: { CLOUDFLARE_PURGE_TOKEN: TOKEN, CLOUDFLARE_ZONE_IDS: ZONES }, fetchImpl: cf.fetchImpl });
try {
try {
const d = doc({ id: 'keep', canonical_url: 'https://openvibe.wiki/p/keep' });
const d = doc({ id: 'keep', canonical_url: 'https://openvibe.wiki/p/keep' });
- svc.store.apply(d);
- svc.store.apply({ ...d, revision: 2, title: 'New title' });
+ await svc.store.apply(d);
+ await svc.store.apply({ ...d, revision: 2, title: 'New title' });
const m = doc({ id: 'mem', visibility: 'members', acl: { groups: ['g'] }, canonical_url: 'https://openvibe.wiki/p/mem' });
const m = doc({ id: 'mem', visibility: 'members', acl: { groups: ['g'] }, canonical_url: 'https://openvibe.wiki/p/mem' });
- svc.store.apply(m);
- svc.store.remove('wiki', 'page', 'mem');
- assert.deepStrictEqual(svc.purges.cdnCounts(), { pending: 0, purged: 0, failed: 0, skipped: 0 });
+ await svc.store.apply(m);
+ await svc.store.remove('wiki', 'page', 'mem');
+ assert.deepStrictEqual(await svc.purges.cdnCounts(), { pending: 0, purged: 0, failed: 0, skipped: 0 });
await svc.purger.flush();
await svc.purger.flush();
assert.strictEqual(cf.calls.length, 0);
assert.strictEqual(cf.calls.length, 0);
} finally { await svc.stop(); }
} finally { await svc.stop(); }
@@ -153,12 +153,12 @@ t('a host with no configured zone is recorded as skipped, not sent', async () =>
@@ -153,12 +153,12 @@ t('a host with no configured zone is recorded as skipped, not sent', async () =>
const svc = await boot({ env: { CLOUDFLARE_PURGE_TOKEN: TOKEN, CLOUDFLARE_ZONE_IDS: ZONES }, fetchImpl: cf.fetchImpl });
const svc = await boot({ env: { CLOUDFLARE_PURGE_TOKEN: TOKEN, CLOUDFLARE_ZONE_IDS: ZONES }, fetchImpl: cf.fetchImpl });
try {
try {
const d = doc({ id: 'elsewhere', canonical_url: 'https://example.org/x' });
const d = doc({ id: 'elsewhere', canonical_url: 'https://example.org/x' });
- svc.store.apply(d);
- svc.store.remove('wiki', 'page', 'elsewhere');
- assert.deepStrictEqual(svc.purges.cdnCounts(), { pending: 0, purged: 0, failed: 0, skipped: 2 });
+ await svc.store.apply(d);
+ await svc.store.remove('wiki', 'page', 'elsewhere');
+ assert.deepStrictEqual(await svc.purges.cdnCounts(), { pending: 0, purged: 0, failed: 0, skipped: 2 });
await svc.purger.flush();
await svc.purger.flush();
assert.strictEqual(cf.calls.length, 0);
assert.strictEqual(cf.calls.length, 0);
- const row = svc.db.prepare("SELECT detail FROM cdn_purges WHERE url = 'https://example.org/x'").get();
+ const row = await svc.db.prepare("SELECT detail FROM cdn_purges WHERE url = 'https://example.org/x'").get();
assert.match(row.detail, /no Cloudflare zone/);
assert.match(row.detail, /no Cloudflare zone/);
} finally { await svc.stop(); }
} finally { await svc.stop(); }
});
});
@@ -174,9 +174,9 @@ t('429, 5xx and network errors back off and retry; after MAX_ATTEMPTS the purge
@@ -174,9 +174,9 @@ t('429, 5xx and network errors back off and retry; after MAX_ATTEMPTS the purge
const svc = await boot({ env: { CLOUDFLARE_PURGE_TOKEN: TOKEN, CLOUDFLARE_ZONE_IDS: ZONES, CLOUDFLARE_PURGE_RELATED_PATHS: 'none' }, fetchImpl: cf.fetchImpl, now: () => clock });
const svc = await boot({ env: { CLOUDFLARE_PURGE_TOKEN: TOKEN, CLOUDFLARE_ZONE_IDS: ZONES, CLOUDFLARE_PURGE_RELATED_PATHS: 'none' }, fetchImpl: cf.fetchImpl, now: () => clock });
try {
try {
const d = doc({ id: 'retry1', canonical_url: 'https://openvibe.wiki/p/retry1' });
const d = doc({ id: 'retry1', canonical_url: 'https://openvibe.wiki/p/retry1' });
- svc.store.apply(d);
- svc.store.remove('wiki', 'page', 'retry1');
- const n = svc.purges.cdnCounts().pending;
+ await svc.store.apply(d);
+ await svc.store.remove('wiki', 'page', 'retry1');
+ const n = (await svc.purges.cdnCounts()).pending;
assert.strictEqual(n, 1, 'related paths off (none)');
assert.strictEqual(n, 1, 'related paths off (none)');
assert.deepStrictEqual(await svc.purger.flush(), { purged: 0, retrying: n, failed: 0 });
assert.deepStrictEqual(await svc.purger.flush(), { purged: 0, retrying: n, failed: 0 });
// Not due yet: nothing is sent before the backoff elapses.
// Not due yet: nothing is sent before the backoff elapses.
@@ -189,10 +189,10 @@ t('429, 5xx and network errors back off and retry; after MAX_ATTEMPTS the purge
@@ -189,10 +189,10 @@ t('429, 5xx and network errors back off and retry; after MAX_ATTEMPTS the purge
// A URL that never gets through fails after MAX_ATTEMPTS.
// A URL that never gets through fails after MAX_ATTEMPTS.
mode = 'down';
mode = 'down';
const e = doc({ id: 'retry2', canonical_url: 'https://openvibe.wiki/p/retry2' });
const e = doc({ id: 'retry2', canonical_url: 'https://openvibe.wiki/p/retry2' });
- svc.store.apply(e);
- svc.store.remove('wiki', 'page', 'retry2');
+ await svc.store.apply(e);
+ await svc.store.remove('wiki', 'page', 'retry2');
for (let i = 0; i < MAX_ATTEMPTS; i++) { clock += 2 * 3600 * 1000; await svc.purger.flush(); }
for (let i = 0; i < MAX_ATTEMPTS; i++) { clock += 2 * 3600 * 1000; await svc.purger.flush(); }
- const rows = svc.db.prepare("SELECT state, attempts, detail FROM cdn_purges WHERE url LIKE '%retry2'").all();
+ const rows = await svc.db.prepare("SELECT state, attempts, detail FROM cdn_purges WHERE url ILIKE '%retry2'").all();
assert.deepStrictEqual(rows.map(r => [r.state, r.attempts]), [['failed', MAX_ATTEMPTS]]);
assert.deepStrictEqual(rows.map(r => [r.state, r.attempts]), [['failed', MAX_ATTEMPTS]]);
assert.match(rows[0].detail, /gave up after/);
assert.match(rows[0].detail, /gave up after/);
} finally { await svc.stop(); }
} finally { await svc.stop(); }
@@ -205,12 +205,12 @@ t('a refused batch is retried one URL at a time: only the refused URL fails', as
@@ -205,12 +205,12 @@ t('a refused batch is retried one URL at a time: only the refused URL fails', as
const svc = await boot({ env: { CLOUDFLARE_PURGE_TOKEN: TOKEN, CLOUDFLARE_ZONE_IDS: ZONES, CLOUDFLARE_PURGE_RELATED_PATHS: 'none' }, fetchImpl: cf.fetchImpl });
const svc = await boot({ env: { CLOUDFLARE_PURGE_TOKEN: TOKEN, CLOUDFLARE_ZONE_IDS: ZONES, CLOUDFLARE_PURGE_RELATED_PATHS: 'none' }, fetchImpl: cf.fetchImpl });
try {
try {
for (const id of ['good1', 'refused', 'good2']) {
for (const id of ['good1', 'refused', 'good2']) {
- svc.store.apply(doc({ id, canonical_url: `https://openvibe.wiki/p/${id}` }));
- svc.store.remove('wiki', 'page', id);
+ await svc.store.apply(doc({ id, canonical_url: `https://openvibe.wiki/p/${id}` }));
+ await svc.store.remove('wiki', 'page', id);
}
}
const s = await svc.purger.flush();
const s = await svc.purger.flush();
assert.deepStrictEqual(s, { purged: 2, retrying: 0, failed: 1 });
assert.deepStrictEqual(s, { purged: 2, retrying: 0, failed: 1 });
- const failed = svc.db.prepare("SELECT url, detail FROM cdn_purges WHERE state = 'failed'").all();
+ const failed = await svc.db.prepare("SELECT url, detail FROM cdn_purges WHERE state = 'failed'").all();
assert.deepStrictEqual(failed.map(r => r.url), ['https://openvibe.wiki/p/refused']);
assert.deepStrictEqual(failed.map(r => r.url), ['https://openvibe.wiki/p/refused']);
assert.match(failed[0].detail, /HTTP 400 1012/);
assert.match(failed[0].detail, /HTTP 400 1012/);
} finally { await svc.stop(); }
} finally { await svc.stop(); }
@@ -224,10 +224,10 @@ t('a token Cloudflare rejects (403) fails the purge without retrying forever and
@@ -224,10 +224,10 @@ t('a token Cloudflare rejects (403) fails the purge without retrying forever and
const { createPurger } = require('../server/purge');
const { createPurger } = require('../server/purge');
const purger = createPurger({ db: svc.db, config: svc.config.purge, fetchImpl: cf.fetchImpl, log: { warn: (m) => logged.push(m), log() {}, error() {} } });
const purger = createPurger({ db: svc.db, config: svc.config.purge, fetchImpl: cf.fetchImpl, log: { warn: (m) => logged.push(m), log() {}, error() {} } });
try {
try {
- svc.store.apply(doc({ id: 'auth1', canonical_url: 'https://openvibe.wiki/p/auth1' }));
- svc.store.remove('wiki', 'page', 'auth1');
+ await svc.store.apply(doc({ id: 'auth1', canonical_url: 'https://openvibe.wiki/p/auth1' }));
+ await svc.store.remove('wiki', 'page', 'auth1');
assert.deepStrictEqual(await purger.flush(), { purged: 0, retrying: 0, failed: 1 });
assert.deepStrictEqual(await purger.flush(), { purged: 0, retrying: 0, failed: 1 });
- const row = svc.db.prepare("SELECT state, detail FROM cdn_purges WHERE url LIKE '%auth1'").get();
+ const row = await svc.db.prepare("SELECT state, detail FROM cdn_purges WHERE url ILIKE '%auth1'").get();
assert.strictEqual(row.state, 'failed');
assert.strictEqual(row.state, 'failed');
assert.ok(!row.detail.includes(TOKEN) && row.detail.includes('[token]'));
assert.ok(!row.detail.includes(TOKEN) && row.detail.includes('[token]'));
assert.ok(logged.length && logged.every(m => !m.includes(TOKEN)));
assert.ok(logged.length && logged.every(m => !m.includes(TOKEN)));
@@ -23,9 +23,9 @@ const call = (method, p, opts = {}) => request(svc.base, method, p, opts);
@@ -23,9 +23,9 @@ const call = (method, p, opts = {}) => request(svc.base, method, p, opts);
t('boot with a small per-person limit', async () => {
t('boot with a small per-person limit', async () => {
svc = await boot({ env: { BASE_URL: ORIGIN, SEARCH_SAVED_MAX_PER_SUBJECT: '3' } });
svc = await boot({ env: { BASE_URL: ORIGIN, SEARCH_SAVED_MAX_PER_SUBJECT: '3' } });
- svc.store.apply(doc({ id: 'pub', title: 'Zebra migration public', canonical_url: 'https://openvibe.wiki/p/pub', facets: { category: 'animals' } }));
- svc.store.apply(doc({ id: 'mem', visibility: 'members', acl: { subjects: [alice] }, title: 'Zebra migration for alice', facets: { category: 'animals' } }));
- svc.store.apply(doc({ owner: 'blog', type: 'post', id: 'other', title: 'Zebra blog post', canonical_url: 'https://openvibe.blog/p/other' }));
+ await svc.store.apply(doc({ id: 'pub', title: 'Zebra migration public', canonical_url: 'https://openvibe.wiki/p/pub', facets: { category: 'animals' } }));
+ await svc.store.apply(doc({ id: 'mem', visibility: 'members', acl: { subjects: [alice] }, title: 'Zebra migration for alice', facets: { category: 'animals' } }));
+ await svc.store.apply(doc({ owner: 'blog', type: 'post', id: 'other', title: 'Zebra blog post', canonical_url: 'https://openvibe.blog/p/other' }));
});
});
t('anonymous callers, guests and services without delegation cannot use saved searches', async () => {
t('anonymous callers, guests and services without delegation cannot use saved searches', async () => {
@@ -71,11 +71,11 @@ t('running a saved search queries as the person now, with its filters', async ()
@@ -71,11 +71,11 @@ t('running a saved search queries as the person now, with its filters', async ()
assert.ok(after.body.saved_search.last_run_at);
assert.ok(after.body.saved_search.last_run_at);
// Access is decided at run time: alice loses the members document, it leaves her results.
// Access is decided at run time: alice loses the members document, it leaves her results.
- svc.store.apply(doc({ id: 'mem', revision: 2, visibility: 'members', acl: { subjects: [bob] }, title: 'Zebra migration for alice', facets: { category: 'animals' } }));
+ await svc.store.apply(doc({ id: 'mem', revision: 2, visibility: 'members', acl: { subjects: [bob] }, title: 'Zebra migration for alice', facets: { category: 'animals' } }));
const r2 = await call('GET', `/api/v1/saved-searches/${saved.id}/results`, as(alice));
const r2 = await call('GET', `/api/v1/saved-searches/${saved.id}/results`, as(alice));
assert.deepStrictEqual(r2.body.results.map(x => x.id), ['pub']);
assert.deepStrictEqual(r2.body.results.map(x => x.id), ['pub']);
// And a deleted public document leaves it too.
// And a deleted public document leaves it too.
- svc.store.remove('wiki', 'page', 'pub');
+ await svc.store.remove('wiki', 'page', 'pub');
const r3 = await call('GET', `/api/v1/saved-searches/${saved.id}/results`, as(alice));
const r3 = await call('GET', `/api/v1/saved-searches/${saved.id}/results`, as(alice));
assert.deepStrictEqual(r3.body.results, []);
assert.deepStrictEqual(r3.body.results, []);
});
});
@@ -7,7 +7,7 @@ const { boot, request, serviceToken, doc, indexEvent, deliver, suite } = require
@@ -7,7 +7,7 @@ const { boot, request, serviceToken, doc, indexEvent, deliver, suite } = require
const t = suite('webhook-inbox');
const t = suite('webhook-inbox');
let svc;
let svc;
const search = async (q) => (await request(svc.base, 'GET', `/api/v1/search?q=${encodeURIComponent(q)}`)).body.results;
const search = async (q) => (await request(svc.base, 'GET', `/api/v1/search?q=${encodeURIComponent(q)}`)).body.results;
-const receipts = () => svc.db.prepare('SELECT COUNT(*) AS n FROM idempotency_receipts').get().n;
+const receipts = async () => (await svc.db.prepare('SELECT COUNT(*) AS n FROM idempotency_receipts').get()).n;
t('boot', async () => { svc = await boot(); });
t('boot', async () => { svc = await boot(); });
@@ -18,7 +18,7 @@ t('an unsigned or wrongly signed delivery is refused and changes nothing', async
@@ -18,7 +18,7 @@ t('an unsigned or wrongly signed delivery is refused and changes nothing', async
const r2 = await request(svc.base, 'POST', '/internal/events', { raw: JSON.stringify({ event: e }), headers: { 'Content-Type': 'application/json' } });
const r2 = await request(svc.base, 'POST', '/internal/events', { raw: JSON.stringify({ event: e }), headers: { 'Content-Type': 'application/json' } });
assert.strictEqual(r2.status, 401);
assert.strictEqual(r2.status, 401);
assert.strictEqual((await search('muword')).length, 0);
assert.strictEqual((await search('muword')).length, 0);
- assert.strictEqual(receipts(), 0);
+ assert.strictEqual(await receipts(), 0);
});
});
t('signature v2 is required: a v1-only delivery and a stale v2 timestamp are refused', async () => {
t('signature v2 is required: a v1-only delivery and a stale v2 timestamp are refused', async () => {
@@ -30,7 +30,7 @@ t('signature v2 is required: a v1-only delivery and a stale v2 timestamp are ref
@@ -30,7 +30,7 @@ t('signature v2 is required: a v1-only delivery and a stale v2 timestamp are ref
const future = await deliver(svc.base, e, { now: Date.now() + 301000 });
const future = await deliver(svc.base, e, { now: Date.now() + 301000 });
assert.strictEqual(future.status, 401, 'v2 from too far in the future: refused');
assert.strictEqual(future.status, 401, 'v2 from too far in the future: refused');
assert.strictEqual((await search('omiword')).length, 0);
assert.strictEqual((await search('omiword')).length, 0);
- assert.strictEqual(receipts(), 0);
+ assert.strictEqual(await receipts(), 0);
const ok = await deliver(svc.base, e);
const ok = await deliver(svc.base, e);
assert.strictEqual(ok.status, 200, 'a fresh v2 delivery of the same event is accepted');
assert.strictEqual(ok.status, 200, 'a fresh v2 delivery of the same event is accepted');
assert.strictEqual((await search('omiword')).length, 1);
assert.strictEqual((await search('omiword')).length, 1);
@@ -45,7 +45,7 @@ t('a delivered upsert is indexed; the same event again is a no-op', async () =>
@@ -45,7 +45,7 @@ t('a delivered upsert is indexed; the same event again is a no-op', async () =>
const again = await deliver(svc.base, e, { seq: 2 });
const again = await deliver(svc.base, e, { seq: 2 });
assert.deepStrictEqual([again.status, again.body.duplicate], [200, true]);
assert.deepStrictEqual([again.status, again.body.duplicate], [200, true]);
assert.strictEqual((await search('nuword')).length, 1);
assert.strictEqual((await search('nuword')).length, 1);
- const events = svc.outbox.all().filter(x => x.payload.id === d.id && x.event_type === 'search.document.indexed');
+ const events = (await svc.outbox.all()).filter(x => x.payload.id === d.id && x.event_type === 'search.document.indexed');
assert.strictEqual(events.length, 1, 'one effect, one event');
assert.strictEqual(events.length, 1, 'one effect, one event');
});
});
@@ -73,7 +73,7 @@ t('a deletion event tombstones and wins over upserts delivered after it', async
@@ -73,7 +73,7 @@ t('a deletion event tombstones and wins over upserts delivered after it', async
assert.strictEqual((await deliver(svc.base, indexEvent(d))).body.outcome, 'stale');
assert.strictEqual((await deliver(svc.base, indexEvent(d))).body.outcome, 'stale');
assert.strictEqual((await deliver(svc.base, indexEvent({ ...d, revision: 5 }))).body.outcome, 'stale');
assert.strictEqual((await deliver(svc.base, indexEvent({ ...d, revision: 5 }))).body.outcome, 'stale');
assert.strictEqual((await search('omicronword')).length, 0);
assert.strictEqual((await search('omicronword')).length, 0);
- const removed = svc.outbox.all().filter(x => x.event_type === 'search.document.removed' && x.payload.id === d.id);
+ const removed = (await svc.outbox.all()).filter(x => x.event_type === 'search.document.removed' && x.payload.id === d.id);
assert.deepStrictEqual(removed.map(x => x.payload.reason), ['deleted']);
assert.deepStrictEqual(removed.map(x => x.payload.reason), ['deleted']);
});
});
@@ -136,7 +136,7 @@ t('a failure while applying leaves no receipt, so the redelivery applies exactly
@@ -136,7 +136,7 @@ t('a failure while applying leaves no receipt, so the redelivery applies exactly
const r1 = await deliver(svc.base, e);
const r1 = await deliver(svc.base, e);
assert.strictEqual(r1.status, 500);
assert.strictEqual(r1.status, 500);
assert.strictEqual((await search('tauword')).length, 0);
assert.strictEqual((await search('tauword')).length, 0);
- assert.strictEqual(svc.db.prepare('SELECT COUNT(*) AS n FROM idempotency_receipts WHERE event_id = ?').get(e.event_id).n, 0);
+ assert.strictEqual((await svc.db.prepare('SELECT COUNT(*) AS n FROM idempotency_receipts WHERE event_id = ?').get(e.event_id)).n, 0);
const r2 = await deliver(svc.base, e, { seq: 2 });
const r2 = await deliver(svc.base, e, { seq: 2 });
assert.deepStrictEqual([r2.status, r2.body.outcome], [200, 'applied']);
assert.deepStrictEqual([r2.status, r2.body.outcome], [200, 'applied']);
const r3 = await deliver(svc.base, e, { seq: 3 });
const r3 = await deliver(svc.base, e, { seq: 3 });
@@ -145,7 +145,7 @@ t('a failure while applying leaves no receipt, so the redelivery applies exactly
@@ -145,7 +145,7 @@ t('a failure while applying leaves no receipt, so the redelivery applies exactly
svc.store.apply = original;
svc.store.apply = original;
}
}
assert.strictEqual((await search('tauword')).length, 1);
assert.strictEqual((await search('tauword')).length, 1);
- assert.strictEqual(svc.outbox.all().filter(x => x.payload.id === d.id).length, 1);
+ assert.strictEqual((await svc.outbox.all()).filter(x => x.payload.id === d.id).length, 1);
});
});
t('OpenVibe.Sources item documents (the first real producer) index for staff only', async () => {
t('OpenVibe.Sources item documents (the first real producer) index for staff only', async () => {