openvibers/openvibe.search Commits
The SQLite era is gone (on PostgreSQL since 2026-09-28; no compatibility kept): the one-time migrate-to-postgres script, the *_DB_PATH settings (config, .env.example, systemd unit, test helpers), the unused better-sqlite3 dependency and its lockfile entries, and docs, docstrings and STATUS lines about SQLite or its rollback window. Reviewed independently before shipping. npm test 12/12 test files passed.
Changes
@@@ -2,7 +2,7 @@# OpenVibe.Search — Environment Configuration# Copy to .env and fill in values.# In production this lives at /etc/openvibe/search.env (0600); the systemd unit sets--# NODE_ENV, PORT and SEARCH_DB_PATH itself.++# NODE_ENV and PORT itself.# ═══════════════════════════════════════════════════════════════# Server@@@ -27,16 +27,13 @@ OV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000OV_OAUTH_CLIENT_ID=searchOV_OAUTH_CLIENT_SECRET=--# Store (SQLite, FTS5). Production: /var/lib/openvibe-search/search.db (systemd StateDirectory).--# PostgreSQL (ADR-035), written by OpenVibe.Host roles/data/add-service.sh search: the runtime role through PgBouncer,--# the owner on a direct connection (migrations and the one-time import), and Valkey (shared limit counters).++# Store (PostgreSQL, ADR-035), written by OpenVibe.Host roles/data/add-service.sh search: the runtime role through++# PgBouncer, the owner on a direct connection (migrations), and Valkey (shared limit counters).# Without DATABASE_URL, development uses an embedded PGlite database in data/pglite.DATABASE_URL=DATABASE_DIRECT_URL=VALKEY_URL=VALKEY_PREFIX=--# The SQLite file of releases before PostgreSQL: read once by scripts/migrate-to-postgres.js.--SEARCH_DB_PATH=./data/search.db# OpenVibe.Events. The outbox relay runs only when EVENTS_URL is set.#EVENTS_URL=http://127.0.0.1:4300@@@ -52,7 +49,7 @@ SEARCH_DB_PATH=./data/search.db#SEARCH_MAX_BODY_BYTES=262144# Ranking: freshness weighting of full-text relevance (README "Ranking").--# score = bm25 × (1 + WEIGHT × 2^(−age_days / HALF_LIFE_DAYS)); age from published_at, else++# score = relevance × (1 + WEIGHT × 2^(−age_days / HALF_LIFE_DAYS)); age from published_at, else# updated_at. WEIGHT 0 turns it off.#SEARCH_FRESHNESS_WEIGHT=1#SEARCH_FRESHNESS_HALF_LIFE_DAYS=30@@@ -25,17 +25,17 @@ revision-ordered copy and can be reconciled against it.npm installcp .env.example .envnpm run dev # http://127.0.0.1:4710--npm test # every test/*.test.js, temp databases, no network needed++npm test # every test/*.test.js, a PGlite database per file, no network needed```Node 22 in production (`fnm exec --using=22.22.1 npm test`). Production: `/opt/openvibe.search`,env `/etc/openvibe/search.env`, unit [deploy/systemd/openvibe-search.service](deploy/systemd/openvibe-search.service),--database `ov_search` on the host's data role (ADR-035; `sudo /opt/openvibe.host/roles/data/add-service.sh search`; the one-time move from `/var/lib/openvibe-search/search.db` is `scripts/migrate-to-postgres.js`, and the full-text index rebuilds itself from the documents at boot), nginx [deploy/nginx/search.openvibe.network.conf](deploy/nginx/search.openvibe.network.conf)++database `ov_search` on the host's data role (ADR-035; `sudo /opt/openvibe.host/roles/data/add-service.sh search`; the full-text index rebuilds itself from the documents at boot), nginx [deploy/nginx/search.openvibe.network.conf](deploy/nginx/search.openvibe.network.conf)(the public vhost exposes the search page, the GET query API, saved searches and health; the ownerAPI and the Events webhook are loopback-only, and owners and Events call `127.0.0.1:4710`).`GET /api/health` is liveness. `GET /api/ready` (openvibe-shared/ready) is 503 only when the--database (the documents and both full-text tables) fails; a Network signing key that has not loaded++database (the documents and the full-text table) fails; a Network signing key that has not loaded(anonymous queries still answer) and a full-text index out of step with the documents table degradeit. It also reports document counts by exposure, the outbox backlog and whether the Events webhookis on. `GET /metrics` (openvibe-shared/metrics) answers direct loopback callers only: golden signals@@@ -51,7 +51,7 @@ GET /api/v1/documents/:owner/:type/:id```- `q` is plain text: word tokens only, every word required, a trailing `*` makes the last word a-- prefix. FTS operators and column filters in `q` are neutralised. No `q` = newest first.++ prefix. Query operators and column filters in `q` are neutralised. No `q` = newest first.- Results carry `owner, type, id, revision, visibility, title, summary, canonical_url, facets,language, authorship, provenance, published_at, updated_at, indexable` and a `snippet_html`(escaped text with `<mark>`). Never the ACL. `next_cursor` is bound to the query that made it.@@@ -69,7 +69,7 @@ Who sees what:Drafts, unpublished, retracted and deleted documents are returned to nobody. A hidden document anda missing one give the same 404. A Bearer that does not verify is a 401 (never downgraded toanonymous); delegation headers from a browser are ignored. Visibility is enforced in the engine--SQL (two FTS tables split by audience plus an ACL predicate) and re-checked per hit — see++SQL (the full-text table split by audience plus an ACL predicate) and re-checked per hit — see[docs/adr-engine.md](docs/adr-engine.md).The search page at `/` runs the same query as HTML for browsers (no JavaScript; no-store; result@@@ -78,11 +78,11 @@ route index at `/`.### Ranking--Full-text results are ordered by bm25 relevance (title ×10, summary ×4, body ×1) multiplied by a++Full-text results are ordered by full-text relevance (title ×10, summary ×4, body ×1) multiplied by afreshness boost:```--score = bm25 × (1 + W × 2^(−age_days / H)) W = SEARCH_FRESHNESS_WEIGHT (default 1)++score = relevance × (1 + W × 2^(−age_days / H)) W = SEARCH_FRESHNESS_WEIGHT (default 1)H = SEARCH_FRESHNESS_HALF_LIFE_DAYS (default 30)```@@@ -236,7 +236,7 @@ Never limited: the signed Events deliveries (`POST /internal/events`), `/api/hea## Acceptance (tests)- a private or draft document is never returned to an unauthorized query — hits, snippets, facet-- counts, suggestions, direct gets, cursors and FTS-syntax injection (`test/acl-leak.test.js`)++ counts, suggestions, direct gets, cursors and full-text-syntax injection (`test/acl-leak.test.js`)- a visibility change or deletion removes the document from results immediately and emits`search.document.removed` (`test/acl-leak.test.js`)- revision ordering, tombstones, restore, reconciliation (`test/documents.test.js`)@@@ -4,7 +4,7 @@"code": true,"wave": "W14","plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026)",-- "runtime": "node 22, express, better-sqlite3 (FTS5)",++ "runtime": "node 22, express, PostgreSQL full-text search (openvibe-sdk/db)","port": 4710,"domain": "search.openvibe.network (vhost in deploy/nginx; the service manifest records the search page and query API as public since 2026-09-23)","contracts": "openvibe-contracts v0.76.0",@@@ -17,14 +17,14 @@"contractsProposed": ["search.index-document@1 (released in openvibe-contracts v0.12.0)"],-- "engine": "sqlite-fts5 (docs/adr-engine.md)",++ "engine": "postgresql-fts (docs/adr-engine.md)","deployed": true,"deployment": "Deployed with `sudo ovhost deploy search`: /opt/openvibe.search, unit openvibe-search, 127.0.0.1:4710, principal search, Events subscription *.index_document.*, accepted event owners sources, wiki, blog, news, reviews, trade, deals, coupons. On 2026-09-23 the production index held 10 documents, all tombstones (wiki), 0 public. Restore drill passed 2026-09-23.","indexedProducers": ["wiki (tombstones only so far)"],"features": [-- "ACL-filtered FTS5 query API",++ "ACL-filtered full-text query API","facets","suggestions","freshness-weighted ranking",@@@ -1,5 +1,5 @@[Unit]--Description=OpenVibe.Search — permission-aware index and query API (SQLite FTS5)++Description=OpenVibe.Search — permission-aware index and query API (PostgreSQL full-text search)# Network (openvibe-network) signs the tokens this service verifies. Search starts without it# and keeps retrying the JWKS fetch; /api/ready stays 503 until the key has loaded.After=network.target postgresql.service pgbouncer.service valkey-server.service@@@ -23,9 +23,6 @@ NoNewPrivileges=truePrivateTmp=trueProtectSystem=strictProtectHome=true--# The code tree stays read-only; the index lives in /var/lib/openvibe-search.--StateDirectory=openvibe-search--Environment=SEARCH_DB_PATH=/var/lib/openvibe-search/search.db[Install]WantedBy=multi-user.target@@@ -1,66 +1,54 @@--# ADR: search engine for Wave 14 — SQLite FTS5 now, PostgreSQL FTS when Postgres exists++# ADR: search engine for Wave 14 — PostgreSQL full-text search--**Status:** accepted for alpha (2026-09-22) · **Owner:** OpenVibe.Search · **Roadmap:** §4.2 A, §15.12, Wave 14++**Status:** accepted (2026-09-22; engine updated to PostgreSQL on 2026-09-28, ADR-035) · **Owner:** OpenVibe.Search · **Roadmap:** §4.2 A, §15.12, Wave 14## ContextThe roadmap asks for an event-fed, permission-aware index with the engine chosen only aftermeasuring need: *"PostgreSQL FTS first; Meilisearch or OpenSearch only if justified"*, and warns--against engine shopping. The network runs every service on one host with one SQLite database per--service; PostgreSQL is not provisioned yet (it arrives with the legacy-retirement track). There--is no corpus to measure: no product indexes anything today.++against engine shopping. No external search engine is wanted: the corpus is SQL rows, the ACL++model is SQL, and a visibility change must leave results in the same transaction that makes it.++There is no corpus to measure: no product indexes anything today.## Decision--1. **SQLite FTS5 inside Search's own database** (`/var/lib/openvibe-search/search.db`), no-- external engine, no new daemon. FTS5 ships in better-sqlite3's bundled SQLite (3.49), gives-- BM25 ranking, snippets and prefix queries, and commits in the same transaction as the-- document row, ACL rows, facet rows, the inbox receipt and the outbox event. That transaction-- is what makes "a visibility change or deletion removes the document from results immediately"-- true without a second system to converge.--2. **Behind a small engine interface** — [server/engine/fts5.js](../server/engine/fts5.js):-- `put(rid, doc, exposure)`, `remove(rid)`, `search`, `browse`, `facetCounts`, `suggest`. The-- store, the revision rules, the ACL model and the APIs do not know FTS5 exists.--3. **PostgreSQL FTS is the next engine**, as the roadmap says, once a PostgreSQL instance exists-- for services: a `tsvector` column with `websearch_to_tsquery`/`ts_rank_cd`, GIN index, the same-- ACL predicate as SQL. Meilisearch/OpenSearch stay out unless a measurement justifies them.++1. **PostgreSQL full-text search in the service's own database** (`search_fts`, ADR-035), no++ external engine, no new daemon. A `tsvector` weighted title A, summary B, body C with a GIN++ index gives ranked matches, `ts_headline` snippets and prefix queries, and commits in the same++ transaction as the document row, ACL rows, facet rows, the inbox receipt and the outbox event.++ That transaction is what makes "a visibility change or deletion removes the document from++ results immediately" true without a second system to converge.++2. **Behind a small engine interface** — [server/engine/pg.js](../server/engine/pg.js):++ `put(rid, doc, exposure)`, `remove(rid)`, `search`, `browse`, `facetCounts`, `suggest`, `counts`,++ `reconcile`. The store, the revision rules, the ACL model and the APIs do not know the engine.++3. **Meilisearch/OpenSearch stay out** unless a measurement justifies them, as the roadmap says.## How visibility is enforced inside the engine--- Documents are split into two FTS tables by audience: `fts_public` holds only public,-- published, indexable documents; `fts_restricted` only published unlisted/members/private-- documents. Drafts, unpublished, deleted and public-noindex documents are in neither, so no text-- query can reach them.--- An anonymous query (or a viewer without a subject) only ever reads `fts_public`. BM25 statistics-- are per table, so private documents cannot even shift the ranking an anonymous caller sees.--- A signed-in query adds a second branch over `fts_restricted`, filtered in the same SQL by the-- viewer's ACL keys (`doc_acl`: subject, group, entitlement). Facet counts and suggestions reuse-- the exact same visible-set SQL, so they cannot count or complete what the viewer cannot see.--- Known limitation: within `fts_restricted`, BM25 term statistics include restricted documents the++- One table, `search_fts`, split by the `audience` column: exposure 3 holds only public, published,++ indexable documents; exposure 1 holds published unlisted/members/private documents. Drafts,++ unpublished, deleted and public-noindex documents have no row, so no text query can reach them.++- An anonymous query (or a viewer without a subject) only ever reads audience 3. Ranking statistics++ are computed over the same visible rows, so private documents cannot shift what an anonymous++ caller sees.++- A signed-in query adds a second branch over audience 1, filtered in the same SQL by the viewer's++ ACL keys (`doc_acl`: subject, group, entitlement). Facet counts and suggestions reuse the exact++ same visible-set SQL, so they cannot count or complete what the viewer cannot see.++- Known limitation: within audience 1, `ts_rank` term statistics include restricted documents theviewer cannot see. The effect on ordering is statistical and returns no content, but it is a-- theoretical side channel between signed-in users; a per-audience or per-tenant index (or-- PostgreSQL with ranking that ignores global statistics) removes it if it ever matters.++ theoretical side channel between signed-in users; a per-audience or per-tenant index removes it++ if it ever matters.## Consequences--- One process and one file: backups, restores and moves are the same as every other service.--- Write throughput is SQLite's (single writer, WAL). That is ample for the expected corpus (tens to-- hundreds of thousands of documents); the trigger to revisit is measured: p95 query latency over-- 200 ms at the public edge, index size over a few GB, or write contention in `/api/ready`.--- Tokenizer is `unicode61 remove_diacritics 2` without stemming: predictable across languages-- (Japanese/Chinese get no word segmentation; a later engine or an ICU tokenizer fixes that).--- Moving to PostgreSQL is an engine swap plus a re-index from owners (the reconciliation API-- exists for exactly that), not a contract change.----## Update 2026-09-28: PostgreSQL full-text search (ADR-035)----Search moved to PostgreSQL with every other service. The engine interface held: `server/engine/pg.js` replaced--`server/engine/fts5.js` and nothing else learned the engine changed. One table, `search_fts`, split by audience--(exposure 3 public listed, 1 restricted), with a `tsvector` weighted title A, summary B, body C (FTS5's 10 / 4 / 1 as--`ts_rank` weights) and a GIN index. Text is folded (diacritics removed) and split into runs of letters and digits before--the `simple` configuration sees it, so tokens are exactly FTS5 unicode61's: on a copy of production, the match counts--for sampled terms were identical. The rank is minus `ts_rank` times the same freshness boost, so ascending order and--the keyset cursors work unchanged. Snippets are `ts_headline` over the summary and body (the title when only it--matches), computed for the returned page only. The index is derived: at boot, when its counts disagree with the--documents table (after the one-time import, whose FTS5 tables are not copied), it is rebuilt from `documents.doc`.--++- One database: backups, restores and moves are the same as every other service (PostgreSQL,++ ADR-035).++- Ranking is `ts_rank` with title/summary/body weights 10/4/1 times a freshness boost++ ([README "Ranking"](../README.md#ranking)), returned as minus the score so ascending keyset++ cursors work.++- Text is folded (diacritics removed) and split into runs of letters and digits before the++ `simple` configuration sees it, so tokens are predictable across languages (Japanese/Chinese get++ no word segmentation; a later engine or ICU tokenizer fixes that) and a query and the index++ always agree on what a word is.++- The index is derived from `documents.doc`: at boot, when its counts disagree with the documents++ table, it is rebuilt (the reconciliation API lets owners re-push everything if needed).@@@ -26,5 +26,5 @@"*.index_document.deleted"],"namespacesOwned": [],-- "notes": "Canonical index document (search.index-document@1), event-fed and direct indexing with revision ordering and tombstones, permission-aware query over SQLite FTS5 (docs/adr-engine.md). Anonymous queries need no token."++ "notes": "Canonical index document (search.index-document@1), event-fed and direct indexing with revision ordering and tombstones, permission-aware query over PostgreSQL full-text search (docs/adr-engine.md). Anonymous queries need no token."}@@@ -11,7 +11,6 @@"dependencies": {"ajv": "^8.17.1","ajv-formats": "^3.0.1",-- "better-sqlite3": "^11.10.0","dotenv": "^16.4.7","express": "^4.21.2","openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",@@@ -103,7 +102,9 @@"url": "https://feross.org/support"}],-- "license": "MIT"++ "license": "MIT",++ "optional": true,++ "peer": true},"node_modules/better-sqlite3": {"version": "11.10.0",@@@ -111,6 +112,8 @@"integrity": "sha512-EwhOpyXiOEL/lKzHz9AW1msWFNzGc/z+LzeB3/jnFJpxu+th2yqvzsSWas1v9jgs9+xiXJcD5A8CJxAG2TaghQ==","hasInstallScript": true,"license": "MIT",++ "optional": true,++ "peer": true,"dependencies": {"bindings": "^1.5.0","prebuild-install": "^7.1.1"@@@ -121,6 +124,8 @@"resolved": "https://registry.npmjs.org/bindings/-/bindings-1.5.0.tgz","integrity": "sha512-p2q/t/mhvuOj/UeLlV6566GD/guowlr0hHxClI0W9m7MWYkL1F0hLo+0Aexs9HSPCtR1SXQ0TD3MMKrXZajbiQ==","license": "MIT",++ "optional": true,++ "peer": true,"dependencies": {"file-uri-to-path": "1.0.0"}@@@ -130,6 +135,8 @@"resolved": "https://registry.npmjs.org/bl/-/bl-4.1.0.tgz","integrity": "sha512-1W07cM9gS6DcLperZfFSj+bWLtaPGSOHWhPiGzXmvVJbRLdG82sH/Kn8EtW1VqWVA54AKf2h5k5BbnIbwF3h6w==","license": "MIT",++ "optional": true,++ "peer": true,"dependencies": {"buffer": "^5.5.0","inherits": "^2.0.4",@@@ -179,6 +186,8 @@}],"license": "MIT",++ "optional": true,++ "peer": true,"dependencies": {"base64-js": "^1.3.1","ieee754": "^1.1.13"@@@ -232,7 +241,9 @@"version": "1.1.4","resolved": "https://registry.npmjs.org/chownr/-/chownr-1.1.4.tgz","integrity": "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==",-- "license": "ISC"++ "license": "ISC",++ "optional": true,++ "peer": true},"node_modules/content-disposition": {"version": "0.5.4",@@@ -284,6 +295,8 @@"resolved": "https://registry.npmjs.org/decompress-response/-/decompress-response-6.0.0.tgz","integrity": "sha512-aW35yZM6Bb/4oJlZncMH2LCoZtJXTRxES17vE3hoRiowU2kWHaJKFkSBDnDR+cm9J+9QhXmREyIfv0pji9ejCQ==","license": "MIT",++ "optional": true,++ "peer": true,"dependencies": {"mimic-response": "^3.1.0"},@@@ -299,6 +312,8 @@"resolved": "https://registry.npmjs.org/deep-extend/-/deep-extend-0.6.0.tgz","integrity": "sha512-LOHxIOaPYdHlJRtCQfDIVZtfw/ufM8+rVj649RIHzcm/vGwQRXFt6OPqIFWsm2XEMrNIEtWR64sY1LEKD2vAOA==","license": "MIT",++ "optional": true,++ "peer": true,"engines": {"node": ">=4.0.0"}@@@ -327,6 +342,8 @@"resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz","integrity": "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==","license": "Apache-2.0",++ "optional": true,++ "peer": true,"engines": {"node": ">=8"}@@@ -386,6 +403,8 @@"resolved": "https://registry.npmjs.org/end-of-stream/-/end-of-stream-1.4.5.tgz","integrity": "sha512-ooEGc6HP26xXq/N+GCGOT0JKCLDGrq2bQUZrQ7gyrJiZANJ/8YDTxTpQBXGMn+WbIQXNVpyWymm7KYVICQnyOg==","license": "MIT",++ "optional": true,++ "peer": true,"dependencies": {"once": "^1.4.0"}@@@ -440,6 +459,8 @@"resolved": "https://registry.npmjs.org/expand-template/-/expand-template-2.0.3.tgz","integrity": "sha512-XYfuKMvj4O35f/pOXLObndIRvyQ+/+6AhODh+OKWj9S9498pHHn/IMszH+gt0fBCRWMNfk1ZSp5x3AifmnI2vg==","license": "(MIT OR WTFPL)",++ "optional": true,++ "peer": true,"engines": {"node": ">=6"}@@@ -516,7 +537,9 @@"version": "1.0.0","resolved": "https://registry.npmjs.org/file-uri-to-path/-/file-uri-to-path-1.0.0.tgz","integrity": "sha512-0Zt+s3L7Vf1biwWZ29aARiVYLx7iMGnEUl9x33fbB/j3jR81u/O2LbqK+Bm1CDSNDKVtJ/YjwY7TUd5SkeLQLw==",-- "license": "MIT"++ "license": "MIT",++ "optional": true,++ "peer": true},"node_modules/finalhandler": {"version": "1.3.2",@@@ -558,7 +581,9 @@"version": "1.0.0","resolved": "https://registry.npmjs.org/fs-constants/-/fs-constants-1.0.0.tgz","integrity": "sha512-y6OAwoSIf7FyjMIv94u+b5rdheZEjzR63GTyZJm5qh4Bi+2YgwLCcI/fPFZkL5PSixOt6ZNKm+w+Hfp/Bciwow==",-- "license": "MIT"++ "license": "MIT",++ "optional": true,++ "peer": true},"node_modules/function-bind": {"version": "1.1.2",@@@ -610,7 +635,9 @@"version": "0.0.0","resolved": "https://registry.npmjs.org/github-from-package/-/github-from-package-0.0.0.tgz","integrity": "sha512-SyHy3T1v2NUXn29OsWdxmK6RwHD+vkj3v8en8AOBZ1wBQ/hCAQ5bAQTD02kW4W9tUp/3Qh6J8r9EvntiyCmOOw==",-- "license": "MIT"++ "license": "MIT",++ "optional": true,++ "peer": true},"node_modules/gopd": {"version": "1.2.0",@@@ -698,7 +725,9 @@"url": "https://feross.org/support"}],-- "license": "BSD-3-Clause"++ "license": "BSD-3-Clause",++ "optional": true,++ "peer": true},"node_modules/inherits": {"version": "2.0.4",@@@ -710,7 +739,9 @@"version": "1.3.8","resolved": "https://registry.npmjs.org/ini/-/ini-1.3.8.tgz","integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==",-- "license": "ISC"++ "license": "ISC",++ "optional": true,++ "peer": true},"node_modules/ipaddr.js": {"version": "1.9.1",@@@ -892,6 +923,8 @@"resolved": "https://registry.npmjs.org/mimic-response/-/mimic-response-3.1.0.tgz","integrity": "sha512-z0yWI+4FDrrweS8Zmt4Ej5HdJmky15+L2e6Wgn3+iK5fWzb6T3fhNFq2+MeTRb064c6Wr4N/wv0DzQTjNzHNGQ==","license": "MIT",++ "optional": true,++ "peer": true,"engines": {"node": ">=10"},@@@ -904,6 +937,8 @@"resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.8.tgz","integrity": "sha512-2yyAR8qBkN3YuheJanUpWC5U3bb5osDywNB8RzDVlDwDHbocAJveqqj1u8+SVD7jkWT4yvsHCpWqqWqAxb0zCA==","license": "MIT",++ "optional": true,++ "peer": true,"funding": {"url": "https://github.com/sponsors/ljharb"}@@@ -912,7 +947,9 @@"version": "0.5.3","resolved": "https://registry.npmjs.org/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz","integrity": "sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A==",-- "license": "MIT"++ "license": "MIT",++ "optional": true,++ "peer": true},"node_modules/ms": {"version": "2.0.0",@@@ -924,7 +961,9 @@"version": "2.0.0","resolved": "https://registry.npmjs.org/napi-build-utils/-/napi-build-utils-2.0.0.tgz","integrity": "sha512-GEbrYkbfF7MoNaoh2iGG84Mnf/WZfB0GdGEsM8wz7Expx/LlWf5U8t9nvJKXSp3qr5IsEbK04cBGhol/KwOsWA==",-- "license": "MIT"++ "license": "MIT",++ "optional": true,++ "peer": true},"node_modules/negotiator": {"version": "0.6.3",@@@ -940,6 +979,8 @@"resolved": "https://registry.npmjs.org/node-abi/-/node-abi-3.96.0.tgz","integrity": "sha512-rebQ/lz7i0EkoLzUVSrKRzA69zMkwLp95kKMWoMDkkM00Suxz0D7zEQPwRml5fQum24mj7bPvmlgLAmu2JCiYg==","license": "MIT",++ "optional": true,++ "peer": true,"dependencies": {"semver": "^7.3.5"},@@@ -976,6 +1017,8 @@"resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz","integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==","license": "ISC",++ "optional": true,++ "peer": true,"dependencies": {"wrappy": "1"}@@@ -1183,6 +1226,8 @@"integrity": "sha512-8Mf2cbV7x1cXPUILADGI3wuhfqWvtiLA1iclTDbFRZkgRQS0NqsPZphna9V+HyTEadheuPmjaJMsbzKQFOzLug==","deprecated": "No longer maintained. Please contact the author of the relevant native addon; alternatives are available.","license": "MIT",++ "optional": true,++ "peer": true,"dependencies": {"detect-libc": "^2.0.0","expand-template": "^2.0.3",@@@ -1226,6 +1271,8 @@"resolved": "https://registry.npmjs.org/pump/-/pump-3.0.4.tgz","integrity": "sha512-VS7sjc6KR7e1ukRFhQSY5LM2uBWAUPiOPa/A3mkKmiMwSmRFUITt0xuj+/lesgnCv+dPIEYlkzrcyXgquIHMcA==","license": "MIT",++ "optional": true,++ "peer": true,"dependencies": {"end-of-stream": "^1.1.0","once": "^1.3.1"@@@ -1276,6 +1323,8 @@"resolved": "https://registry.npmjs.org/rc/-/rc-1.2.8.tgz","integrity": "sha512-y3bGgqKj3QBdxLbLkomlohkvsA8gdAiUQlSBJnBhfn+BPxg4bc62d8TcBW15wavDfgexCgccckhcZvywyQYPOw==","license": "(BSD-2-Clause OR MIT OR Apache-2.0)",++ "optional": true,++ "peer": true,"dependencies": {"deep-extend": "^0.6.0","ini": "~1.3.0",@@@ -1291,6 +1340,8 @@"resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz","integrity": "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==","license": "MIT",++ "optional": true,++ "peer": true,"dependencies": {"inherits": "^2.0.3","string_decoder": "^1.1.1",@@@ -1488,7 +1539,9 @@"url": "https://feross.org/support"}],-- "license": "MIT"++ "license": "MIT",++ "optional": true,++ "peer": true},"node_modules/simple-get": {"version": "4.0.1",@@@ -1509,6 +1562,8 @@}],"license": "MIT",++ "optional": true,++ "peer": true,"dependencies": {"decompress-response": "^6.0.0","once": "^1.3.1",@@@ -1538,6 +1593,8 @@"resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.3.0.tgz","integrity": "sha512-hkRX8U1WjJFd8LsDJ2yQ/wWWxaopEsABU1XfkM8A+j0+85JAGppt16cr1Whg6KIbb4okU6Mql6BOj+uup/wKeA==","license": "MIT",++ "optional": true,++ "peer": true,"dependencies": {"safe-buffer": "~5.2.0"}@@@ -1547,6 +1604,8 @@"resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-2.0.1.tgz","integrity": "sha512-4gB8na07fecVVkOI6Rs4e7T6NOTki5EmL7TUduTs6bu3EdnSycntVJ4re8kgZA+wx9IueI2Y11bfbgwtzuE0KQ==","license": "MIT",++ "optional": true,++ "peer": true,"engines": {"node": ">=0.10.0"}@@@ -1556,6 +1615,8 @@"resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-2.1.5.tgz","integrity": "sha512-OboTd8mmMhZDNPV+UjQcK9yKAatXu2aJ+r1w4im1Otd4M4fl2hwvdoXUxIYHFTHWK/3y3FarBP70v3vwmGlOxw==","license": "MIT",++ "optional": true,++ "peer": true,"dependencies": {"chownr": "^1.1.1","mkdirp-classic": "^0.5.2",@@@ -1568,6 +1629,8 @@"resolved": "https://registry.npmjs.org/tar-stream/-/tar-stream-2.2.0.tgz","integrity": "sha512-ujeqbceABgwMZxEJnk2HDY2DlnUZ+9oEcb1KzTVfYHio0UE6dG71n60d8D2I4qNvleWrrXpmjpt7vZeF1LnMZQ==","license": "MIT",++ "optional": true,++ "peer": true,"dependencies": {"bl": "^4.0.3","end-of-stream": "^1.4.1",@@@ -1593,6 +1656,8 @@"resolved": "https://registry.npmjs.org/tunnel-agent/-/tunnel-agent-0.6.0.tgz","integrity": "sha512-McnNiV1l8RYeY8tBgEpuodCC1mLUdbSN+CYBL7kJsJNInOP8UjDDEwdk6Mw60vdLLrr5NHKZhMAOSrR2NZuQ+w==","license": "Apache-2.0",++ "optional": true,++ "peer": true,"dependencies": {"safe-buffer": "^5.0.1"},@@@ -1626,7 +1691,9 @@"version": "1.0.2","resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz","integrity": "sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==",-- "license": "MIT"++ "license": "MIT",++ "optional": true,++ "peer": true},"node_modules/utils-merge": {"version": "1.0.1",@@@ -1650,7 +1717,9 @@"version": "1.0.2","resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz","integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==",-- "license": "ISC"++ "license": "ISC",++ "optional": true,++ "peer": true},"node_modules/xtend": {"version": "4.0.2",@@@ -2,7 +2,7 @@"name": "openvibe-search","version": "0.1.0","private": true,-- "description": "OpenVibe.Search: the canonical index document, event-fed and direct indexing with revision ordering and tombstones, and a permission-aware query API over SQLite FTS5.",++ "description": "OpenVibe.Search: the canonical index document, event-fed and direct indexing with revision ordering and tombstones, and a permission-aware query API over PostgreSQL full-text search.","main": "server/index.js","scripts": {"start": "node server/index.js",@@@ -14,7 +14,7 @@"keywords": ["openvibe","search",-- "fts5",++ "full-text","acl"],"author": "OpenVibers",@@@ -23,7 +23,6 @@"dependencies": {"ajv": "^8.17.1","ajv-formats": "^3.0.1",-- "better-sqlite3": "^11.10.0","dotenv": "^16.4.7","express": "^4.21.2","openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",ddeleted file mode 100644@@@ -1,26 +0,0 @@--#!/usr/bin/env node--'use strict';--/**-- * The one-time move of Search's SQLite database (SEARCH_DB_PATH) into its PostgreSQL schema (ADR-035; the procedure is-- * openvibe-sdk docs/migrating-to-postgresql.md, section 6, carried out by openvibe-sdk/db runSqliteMigration).-- *-- * node scripts/migrate-to-postgres.js [--sqlite <file>] [--pglite] [--json]-- *-- * Applies migrations/ as the owner (DATABASE_DIRECT_URL), copies every table into emptied tables, verifies row counts-- * and checksums, and exits 1 unless everything verified. The SQLite file is opened read-only. Every table keeps its-- * name and columns.-- */--require('dotenv').config();--const { runSqliteMigration } = require('openvibe-sdk/db');--const configLib = require('../server/config');--const { MIGRATIONS } = require('../server/db');----const TABLES = {};----if (require.main === module) {-- const config = configLib.load();-- runSqliteMigration({ service: 'search', sqlite: config.dbPath, directUrl: config.db.directUrl, migrations: MIGRATIONS, tables: TABLES })-- .then((code) => process.exit(code), (err) => { console.error(`migrate-to-postgres failed: ${err.message}`); process.exit(1); });--}----module.exports = { TABLES };@@@ -9,7 +9,7 @@* with envelope subject { type, id, revision } and visibility "internal".** Exactly once: the inbox receipt (consumer, event_id) and the document change commit in one-- * SQLite transaction. A redelivery of a processed event is answered 204 and changes nothing.++ * database transaction. A redelivery of a processed event is answered 204 and changes nothing.* Revision order is the store's rule, so out-of-order delivery is harmless.** Refusals (unknown owner, a source writing another owner's document, a bad document, a subject@@@ -72,8 +72,6 @@ function load(env = process.env) {// PostgreSQL (ADR-035): DATABASE_URL serves (PgBouncer), DATABASE_DIRECT_URL migrates (owner role).db: { url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '' },-- // The SQLite file of releases before the switch: read once by scripts/migrate-to-postgres.js.-- dbPath: env.SEARCH_DB_PATH || './data/search.db',// Events: the outbox relays only when EVENTS_URL is set (rows wait otherwise).events: {@@@ -1,7 +1,7 @@'use strict';/**-- * Search engine: PostgreSQL full-text search in the service's own database (ADR-035; it replaced SQLite FTS5, the-- * engine docs/adr-engine.md chose, with the same interface and the same guarantees).++ * Search engine: PostgreSQL full-text search in the service's own database (ADR-035; the engine docs/adr-engine.md++ * chose, with the same interface and the same guarantees).** One table, search_fts, split by audience so an anonymous query never touches, ranks against or counts anything but* public listed documents:@@@ -13,15 +13,16 @@* also filtered by `documents.exposure` and, for restricted rows, by the viewer's ACL keys (doc_acl) in the same SQL:* visibility is decided before anything is ranked, counted, snippeted or suggested.*-- * Text is folded before it is indexed or queried (NFKD, combining marks removed: FTS5's remove_diacritics), split into-- * runs of letters and digits (FTS5's unicode61) and tokenized by the `simple` configuration (lower case, no stemming, no stop words: FTS5's unicode61). Title, summary and-- * body weigh 10, 4 and 1 (tsvector weights A, B, C). Every function here is async; put and remove run inside the store's++ * Text is folded before it is indexed or queried (NFKD, combining marks removed), split into runs of letters and++ * digits and tokenized by the `simple` configuration (lower case, no stemming, no stop words): the index and every++ * query see the same text. Title, summary and body weigh 10, 4 and 1 (tsvector weights A, B, C), with the freshness++ * boost described below. Every function here is async; put and remove run inside the store's* transaction (ambient). Nothing outside this file writes SQL against search_fts.*/const { EXPOSURE } = require('../document');const CONFIG = 'simple';--const WEIGHTS = '{0.1, 0.1, 0.4, 1.0}'; // D, C body, B summary, A title: FTS5's bm25 weights 10 / 4 / 1++const WEIGHTS = '{0.1, 0.1, 0.4, 1.0}'; // D, C body, B summary, A title: ts_rank weights 1 / 4 / 10const MARK_OPEN = '\u0001';const MARK_CLOSE = '\u0002';const HEADLINE = `StartSel=${MARK_OPEN}, StopSel=${MARK_CLOSE}, MaxWords=16, MinWords=8, ShortWord=0, MaxFragments=1, FragmentDelimiter=" … "`;@@@ -47,15 +48,15 @@ function freshnessBoost(iso, nowMs, halfLifeDays, weight) {return 1 + weight * Math.pow(2, -ageDays / halfLifeDays);}--/** Diacritics folded, as FTS5's remove_diacritics did: the index and every query see the same text. */++/** Diacritics folded, so the index and every query see the same text. */function fold(s) {return String(s || '').normalize('NFKD').replace(/\p{M}+/gu, '').normalize('NFC');}/**-- * The words FTS5's unicode61 tokenizer saw: runs of letters and digits, everything else a separator. PostgreSQL's-- * parser would keep "OpenVibe.Live", "a_b" or "3.5" as single tokens (host, word, number); separating them first makes-- * "openvibe" find "OpenVibe.Live" as it did.++ * Word tokens: runs of letters and digits, everything else a separator. PostgreSQL's parser would++ * keep "OpenVibe.Live", "a_b" or "3.5" as single tokens (host, word, number); separating them first++ * makes "openvibe" find "OpenVibe.Live".*/function words(s) {return fold(s).replace(/[^\p{L}\p{N}]+/gu, ' ');@@@ -124,7 +125,7 @@ function createEngine(db, { freshness = { weight: 1, halfLifeDays: 30 } } = {})/*** The visible, matching set as (rid, rank) rows. match = toMatch()'s query. rank is minus ts_rank times the-- * freshness boost at reference time `now` (ms), so lower ranks first (as FTS5's bm25 did) and pages of one query++ * freshness boost at reference time `now` (ms), so lower ranks first and pages of one query* (which carry their first page's `now` in the cursor) rank against the same clock.*/function matchedSet({ match, filters, viewer, now = Date.now() }, params) {@@@ -234,8 +235,8 @@ function createEngine(db, { freshness = { weight: 1, halfLifeDays: 30 } } = {})}/**-- * Rebuild the index from documents.doc when its counts disagree with the documents table (after the one-time import-- * from SQLite, whose FTS5 tables are not copied). Returns what it did.++ * Rebuild the index from documents.doc when its counts disagree with the documents table (for instance after a++ * restore). Returns what it did.*/async function reconcile() {const docs = await db.prepare(`SELECT@@@ -16,22 +16,6 @@ const { ids, validate, serviceAuth } = require('openvibe-contracts');const BACKOFF_MS = [1000, 5000, 30000, 120000, 600000];--function ensureSchema(db) {-- db.exec(`CREATE TABLE IF NOT EXISTS event_outbox (-- seq INTEGER PRIMARY KEY AUTOINCREMENT,-- event_id TEXT NOT NULL UNIQUE,-- event_type TEXT NOT NULL,-- envelope TEXT NOT NULL,-- created_at INTEGER NOT NULL,-- attempts INTEGER NOT NULL DEFAULT 0,-- next_attempt_at INTEGER NOT NULL DEFAULT 0,-- sent_at INTEGER,-- rejected_at INTEGER,-- last_error TEXT-- );-- CREATE INDEX IF NOT EXISTS idx_event_outbox_due ON event_outbox(sent_at, rejected_at, next_attempt_at);`);--}--function createOutbox(db, { source, now = () => Date.now() }) {const insert = db.prepare('INSERT INTO event_outbox (event_id, event_type, envelope, created_at) VALUES (?, ?, ?, ?)');@@@ -176,4 +160,4 @@ function createRelay({ outbox, eventsUrl, intervalMs = 2000, tokenClient, tokenOreturn { start, stop, flush, running: () => Boolean(timer) };}--module.exports = { ensureSchema, createOutbox, createRelay };++module.exports = { createOutbox, createRelay };@@@ -22,7 +22,7 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetch// PostgreSQL (ADR-035): opened and migrated here unless the caller (a test) hands in a migrated handle.const db = givenDb || await openDb(config, { log });const engine = createEngine(db, { freshness: config.freshness });-- // The index is derived from documents: rebuilt when it disagrees (after the one-time import from SQLite).++ // The index is derived from documents: rebuilt at boot when it disagrees with the documents table.const rebuilt = await engine.reconcile();if (rebuilt.rebuilt) log.log(`[search] full-text index rebuilt: ${rebuilt.rebuilt} document(s)`);const outbox = createOutbox(db, { source: config.serviceId, now });@@@ -3,16 +3,16 @@* Track O: truthful readiness for GET /api/ready and the Search gauges on GET /metrics* (openvibe-shared/ready and openvibe-shared/metrics).*-- * db required a real read of the documents table and of both full-text tables (they-- * live in the same SQLite file): without them nothing can be indexed or found++ * db required a real read of the documents table and of the full-text table (they++ * live in the same database): without them nothing can be indexed or found* network_jwks optional the Network signing key has loaded. Without it anonymous queries still* answer (public documents only), but no token can be verified: owner* writes and signed-in queries fail, so it degrades rather than fails-- * index optional the full-text index agrees with the documents table: fts_public holds-- * exactly the public_listed documents and fts_restricted the restricted-- * ones. A mismatch means queries miss (or would surface) documents++ * index optional the full-text index agrees with the documents table: search_fts holds++ * exactly the public_listed and restricted documents. A mismatch means++ * queries miss (or would surface) documents*-- * Gauges: documents by exposure, full-text entries by table, and the events outbox backlog.++ * Gauges: documents by exposure, full-text entries by audience, and the events outbox backlog.*/const { createReadiness } = require('openvibe-shared/ready');const { EXPOSURE } = require('./document');@@@ -90,7 +90,7 @@ function registerSearchGauges(registry, { db, engine, outbox, purges = null }) {collect: async () => Object.entries(await read.documents()).map(([exposure, value]) => ({ labels: { exposure }, value })),});registry.gauge({-- name: 'search_documents_indexed', help: 'Documents in the full-text index, by table', labelNames: ['index'],++ name: 'search_documents_indexed', help: 'Documents in the full-text index, by audience', labelNames: ['index'],collect: async () => Object.entries(await read.indexed()).map(([index, value]) => ({ labels: { index }, value })),});registry.gauge({ name: 'search_outbox_pending', help: 'Events waiting in the outbox', collect: async () => await outbox.pending() });@@@ -25,42 +25,6 @@ const MAX_ATTEMPTS = 8;const FILES_PER_CALL = 30;const DETAIL_MAX = 300;--function ensureSchema(db) {-- db.exec(`-- CREATE TABLE IF NOT EXISTS removals (-- seq INTEGER PRIMARY KEY AUTOINCREMENT,-- event_id TEXT NOT NULL, -- the search.document.removed event-- owner TEXT NOT NULL,-- type TEXT NOT NULL,-- id TEXT NOT NULL,-- revision INTEGER NOT NULL,-- reason TEXT NOT NULL,-- previous_exposure TEXT NOT NULL,-- exposure TEXT NOT NULL,-- canonical_url TEXT, -- only a URL that had been public-- at INTEGER NOT NULL-- );-- CREATE INDEX IF NOT EXISTS idx_removals_owner ON removals (owner, seq);---- CREATE TABLE IF NOT EXISTS cdn_purges (-- seq INTEGER PRIMARY KEY AUTOINCREMENT,-- removal_seq INTEGER NOT NULL,-- provider TEXT NOT NULL DEFAULT 'cloudflare',-- zone_id TEXT,-- url TEXT NOT NULL,-- state TEXT NOT NULL CHECK (state IN ('pending', 'purged', 'failed', 'skipped')),-- detail TEXT,-- attempts INTEGER NOT NULL DEFAULT 0,-- next_attempt_at INTEGER NOT NULL DEFAULT 0,-- created_at INTEGER NOT NULL,-- done_at INTEGER-- );-- -- One pending purge per URL: a sitemap shared by many removals is purged once per batch.-- CREATE UNIQUE INDEX IF NOT EXISTS idx_cdn_purges_pending ON cdn_purges (zone_id, url) WHERE state = 'pending';-- CREATE INDEX IF NOT EXISTS idx_cdn_purges_due ON cdn_purges (state, next_attempt_at, seq);-- `);--}--/** The zone a URL belongs to: exact host or a parent domain in the map, longest first. */function zoneFor(url, zones) {let u;@@@ -254,4 +218,4 @@ function createPurger({ db, config, fetchImpl = globalThis.fetch, log = console,return { start, stop, flush, running: () => Boolean(timer) };}--module.exports = { ensureSchema, createPurgeQueue, createPurger, zoneFor, purgeUrls, MAX_ATTEMPTS, FILES_PER_CALL };++module.exports = { createPurgeQueue, createPurger, zoneFor, purgeUrls, MAX_ATTEMPTS, FILES_PER_CALL };@@@ -13,24 +13,6 @@ const { ids } = require('openvibe-contracts');const ID_RE = /^svs_[0-9a-z]{26}$/;--function ensureSchema(db) {-- db.exec(`-- CREATE TABLE IF NOT EXISTS saved_searches (-- id TEXT PRIMARY KEY,-- subject TEXT NOT NULL, -- usr_… (the person who saved it)-- name TEXT NOT NULL,-- q TEXT NOT NULL DEFAULT '',-- filters TEXT NOT NULL, -- canonical JSON {owner,type,language,facets}-- query_hash TEXT NOT NULL, -- same query saved twice = one row-- created_at INTEGER NOT NULL,-- updated_at INTEGER NOT NULL,-- last_run_at INTEGER,-- UNIQUE (subject, query_hash)-- );-- CREATE INDEX IF NOT EXISTS idx_saved_searches_subject ON saved_searches (subject, created_at);-- `);--}--/** Filters in one canonical shape, so equal queries hash equally whatever order they came in. */function canonicalFilters(f) {const out = {};@@@ -111,4 +93,4 @@ function createSavedSearches(db, { maxPerSubject = 50, now = () => Date.now() }};}--module.exports = { ensureSchema, createSavedSearches, canonicalFilters, ID_RE };++module.exports = { createSavedSearches, canonicalFilters, ID_RE };@@@ -5,9 +5,6 @@* Events delivery helper. Nothing here needs the network or a running OpenVibe.Network.*/const crypto = require('crypto');--const fs = require('fs');--const os = require('os');--const path = require('path');const { serviceAuth, ids } = require('openvibe-contracts');const { load } = require('../server/config');const { start } = require('../server/index');@@@ -37,21 +34,10 @@ function userToken({ subjectId = ids.newId('user'), role = 'user', aud = ['openv}, key);}--const made = [];--process.on('exit', () => { for (const d of made) fs.rmSync(d, { recursive: true, force: true }); });----function tmpDir() {-- const d = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-search-test-'));-- made.push(d);-- return d;--}--async function boot({ env = {}, fetchImpl, tokenClient, now } = {}) {-- const dir = tmpDir();const config = load({NODE_ENV: 'test',PORT: '0',-- SEARCH_DB_PATH: path.join(dir, 'search.db'),OV_NETWORK_PUBLIC_KEY: publicKey,SEARCH_EVENTS_SECRET: WEBHOOK_SECRET,...env,@@@ -60,7 +46,7 @@ async function boot({ env = {}, fetchImpl, tokenClient, now } = {}) {const testdb = await require('./db').testDb();const h = await start({ config, db: testdb.db, log: silent, fetchImpl, tokenClient, now });const base = `http://127.0.0.1:${h.server.address().port}`;-- return { ...h, base, dir, async stop() { await h.close(); await testdb.close(); } };++ return { ...h, base, async stop() { await h.close(); await testdb.close(); } };}async function request(base, method, p, { token, body, headers = {}, raw } = {}) {@@@ -145,6 +131,6 @@ function suite(name) {}module.exports = {-- ISSUER, WEBHOOK_SECRET, privateKey, publicKey, silent, serviceToken, userToken, tmpDir, boot, request,++ ISSUER, WEBHOOK_SECRET, privateKey, publicKey, silent, serviceToken, userToken, boot, request,doc, indexEvent, deliver, suite,};@@@ -1,8 +1,9 @@#!/usr/bin/env node/*** Runs every test in test/ — the files named *.test.js — each in its own process, and fails-- * if any of them fails. They use temp SQLite databases, generated RSA keys and stub subscribers-- * on random ports; none of them needs the network or a running OpenVibe.Network.++ * if any of them fails. They boot a PGlite database per test file (PostgreSQL with++ * SEARCH_TEST_STORE=pg), with generated RSA keys and stub subscribers on random ports; none of++ * them needs the network or a running OpenVibe.Network.** npm test # everything* npm test -- publish sse # only files whose name contains one of the words
@@ -2,7 +2,7 @@
@@ -2,7 +2,7 @@
# OpenVibe.Search — Environment Configuration
# OpenVibe.Search — Environment Configuration
# Copy to .env and fill in values.
# Copy to .env and fill in values.
# In production this lives at /etc/openvibe/search.env (0600); the systemd unit sets
# In production this lives at /etc/openvibe/search.env (0600); the systemd unit sets
-# NODE_ENV, PORT and SEARCH_DB_PATH itself.
+# NODE_ENV and PORT itself.
# ═══════════════════════════════════════════════════════════════
# ═══════════════════════════════════════════════════════════════
# Server
# Server
@@ -27,16 +27,13 @@ OV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000
@@ -27,16 +27,13 @@ OV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000
OV_OAUTH_CLIENT_ID=search
OV_OAUTH_CLIENT_ID=search
OV_OAUTH_CLIENT_SECRET=
OV_OAUTH_CLIENT_SECRET=
-# Store (SQLite, FTS5). Production: /var/lib/openvibe-search/search.db (systemd StateDirectory).
-# PostgreSQL (ADR-035), written by OpenVibe.Host roles/data/add-service.sh search: the runtime role through PgBouncer,
-# the owner on a direct connection (migrations and the one-time import), and Valkey (shared limit counters).
+# Store (PostgreSQL, ADR-035), written by OpenVibe.Host roles/data/add-service.sh search: the runtime role through
+# PgBouncer, the owner on a direct connection (migrations), and Valkey (shared limit counters).
# Without DATABASE_URL, development uses an embedded PGlite database in data/pglite.
# Without DATABASE_URL, development uses an embedded PGlite database in data/pglite.
DATABASE_URL=
DATABASE_URL=
DATABASE_DIRECT_URL=
DATABASE_DIRECT_URL=
VALKEY_URL=
VALKEY_URL=
VALKEY_PREFIX=
VALKEY_PREFIX=
-# The SQLite file of releases before PostgreSQL: read once by scripts/migrate-to-postgres.js.
-SEARCH_DB_PATH=./data/search.db
# OpenVibe.Events. The outbox relay runs only when EVENTS_URL is set.
# OpenVibe.Events. The outbox relay runs only when EVENTS_URL is set.
#EVENTS_URL=http://127.0.0.1:4300
#EVENTS_URL=http://127.0.0.1:4300
@@ -52,7 +49,7 @@ SEARCH_DB_PATH=./data/search.db
@@ -52,7 +49,7 @@ SEARCH_DB_PATH=./data/search.db
#SEARCH_MAX_BODY_BYTES=262144
#SEARCH_MAX_BODY_BYTES=262144
# Ranking: freshness weighting of full-text relevance (README "Ranking").
# Ranking: freshness weighting of full-text relevance (README "Ranking").
-# score = bm25 × (1 + WEIGHT × 2^(−age_days / HALF_LIFE_DAYS)); age from published_at, else
+# score = relevance × (1 + WEIGHT × 2^(−age_days / HALF_LIFE_DAYS)); age from published_at, else
# updated_at. WEIGHT 0 turns it off.
# updated_at. WEIGHT 0 turns it off.
#SEARCH_FRESHNESS_WEIGHT=1
#SEARCH_FRESHNESS_WEIGHT=1
#SEARCH_FRESHNESS_HALF_LIFE_DAYS=30
#SEARCH_FRESHNESS_HALF_LIFE_DAYS=30
@@ -25,17 +25,17 @@ revision-ordered copy and can be reconciled against it.
@@ -25,17 +25,17 @@ revision-ordered copy and can be reconciled against it.
npm install
npm install
cp .env.example .env
cp .env.example .env
npm run dev # http://127.0.0.1:4710
npm run dev # http://127.0.0.1:4710
-npm test # every test/*.test.js, temp databases, no network needed
+npm test # every test/*.test.js, a PGlite database per file, no network needed
```
```
Node 22 in production (`fnm exec --using=22.22.1 npm test`). Production: `/opt/openvibe.search`,
Node 22 in production (`fnm exec --using=22.22.1 npm test`). Production: `/opt/openvibe.search`,
env `/etc/openvibe/search.env`, unit [deploy/systemd/openvibe-search.service](deploy/systemd/openvibe-search.service),
env `/etc/openvibe/search.env`, unit [deploy/systemd/openvibe-search.service](deploy/systemd/openvibe-search.service),
-database `ov_search` on the host's data role (ADR-035; `sudo /opt/openvibe.host/roles/data/add-service.sh search`; the one-time move from `/var/lib/openvibe-search/search.db` is `scripts/migrate-to-postgres.js`, and the full-text index rebuilds itself from the documents at boot), nginx [deploy/nginx/search.openvibe.network.conf](deploy/nginx/search.openvibe.network.conf)
+database `ov_search` on the host's data role (ADR-035; `sudo /opt/openvibe.host/roles/data/add-service.sh search`; the full-text index rebuilds itself from the documents at boot), nginx [deploy/nginx/search.openvibe.network.conf](deploy/nginx/search.openvibe.network.conf)
(the public vhost exposes the search page, the GET query API, saved searches and health; the owner
(the public vhost exposes the search page, the GET query API, saved searches and health; the owner
API and the Events webhook are loopback-only, and owners and Events call `127.0.0.1:4710`).
API and the Events webhook are loopback-only, and owners and Events call `127.0.0.1:4710`).
`GET /api/health` is liveness. `GET /api/ready` (openvibe-shared/ready) is 503 only when the
`GET /api/health` is liveness. `GET /api/ready` (openvibe-shared/ready) is 503 only when the
-database (the documents and both full-text tables) fails; a Network signing key that has not loaded
+database (the documents and the full-text table) fails; a Network signing key that has not loaded
(anonymous queries still answer) and a full-text index out of step with the documents table degrade
(anonymous queries still answer) and a full-text index out of step with the documents table degrade
it. It also reports document counts by exposure, the outbox backlog and whether the Events webhook
it. It also reports document counts by exposure, the outbox backlog and whether the Events webhook
is on. `GET /metrics` (openvibe-shared/metrics) answers direct loopback callers only: golden signals
is on. `GET /metrics` (openvibe-shared/metrics) answers direct loopback callers only: golden signals
@@ -51,7 +51,7 @@ GET /api/v1/documents/:owner/:type/:id
@@ -51,7 +51,7 @@ GET /api/v1/documents/:owner/:type/:id
```
```
- `q` is plain text: word tokens only, every word required, a trailing `*` makes the last word a
- `q` is plain text: word tokens only, every word required, a trailing `*` makes the last word a
- prefix. FTS operators and column filters in `q` are neutralised. No `q` = newest first.
+ prefix. Query operators and column filters in `q` are neutralised. No `q` = newest first.
- Results carry `owner, type, id, revision, visibility, title, summary, canonical_url, facets,
- Results carry `owner, type, id, revision, visibility, title, summary, canonical_url, facets,
language, authorship, provenance, published_at, updated_at, indexable` and a `snippet_html`
language, authorship, provenance, published_at, updated_at, indexable` and a `snippet_html`
(escaped text with `<mark>`). Never the ACL. `next_cursor` is bound to the query that made it.
(escaped text with `<mark>`). Never the ACL. `next_cursor` is bound to the query that made it.
@@ -69,7 +69,7 @@ Who sees what:
@@ -69,7 +69,7 @@ Who sees what:
Drafts, unpublished, retracted and deleted documents are returned to nobody. A hidden document and
Drafts, unpublished, retracted and deleted documents are returned to nobody. A hidden document and
a missing one give the same 404. A Bearer that does not verify is a 401 (never downgraded to
a missing one give the same 404. A Bearer that does not verify is a 401 (never downgraded to
anonymous); delegation headers from a browser are ignored. Visibility is enforced in the engine
anonymous); delegation headers from a browser are ignored. Visibility is enforced in the engine
-SQL (two FTS tables split by audience plus an ACL predicate) and re-checked per hit — see
+SQL (the full-text table split by audience plus an ACL predicate) and re-checked per hit — see
[docs/adr-engine.md](docs/adr-engine.md).
[docs/adr-engine.md](docs/adr-engine.md).
The search page at `/` runs the same query as HTML for browsers (no JavaScript; no-store; result
The search page at `/` runs the same query as HTML for browsers (no JavaScript; no-store; result
@@ -78,11 +78,11 @@ route index at `/`.
@@ -78,11 +78,11 @@ route index at `/`.
### Ranking
### Ranking
-Full-text results are ordered by bm25 relevance (title ×10, summary ×4, body ×1) multiplied by a
+Full-text results are ordered by full-text relevance (title ×10, summary ×4, body ×1) multiplied by a
freshness boost:
freshness boost:
```
```
-score = bm25 × (1 + W × 2^(−age_days / H)) W = SEARCH_FRESHNESS_WEIGHT (default 1)
+score = relevance × (1 + W × 2^(−age_days / H)) W = SEARCH_FRESHNESS_WEIGHT (default 1)
H = SEARCH_FRESHNESS_HALF_LIFE_DAYS (default 30)
H = SEARCH_FRESHNESS_HALF_LIFE_DAYS (default 30)
```
```
@@ -236,7 +236,7 @@ Never limited: the signed Events deliveries (`POST /internal/events`), `/api/hea
@@ -236,7 +236,7 @@ Never limited: the signed Events deliveries (`POST /internal/events`), `/api/hea
## Acceptance (tests)
## Acceptance (tests)
- a private or draft document is never returned to an unauthorized query — hits, snippets, facet
- a private or draft document is never returned to an unauthorized query — hits, snippets, facet
- counts, suggestions, direct gets, cursors and FTS-syntax injection (`test/acl-leak.test.js`)
+ counts, suggestions, direct gets, cursors and full-text-syntax injection (`test/acl-leak.test.js`)
- a visibility change or deletion removes the document from results immediately and emits
- a visibility change or deletion removes the document from results immediately and emits
`search.document.removed` (`test/acl-leak.test.js`)
`search.document.removed` (`test/acl-leak.test.js`)
- revision ordering, tombstones, restore, reconciliation (`test/documents.test.js`)
- revision ordering, tombstones, restore, reconciliation (`test/documents.test.js`)
@@ -4,7 +4,7 @@
@@ -4,7 +4,7 @@
"code": true,
"code": true,
"wave": "W14",
"wave": "W14",
"plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026)",
"plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026)",
- "runtime": "node 22, express, better-sqlite3 (FTS5)",
+ "runtime": "node 22, express, PostgreSQL full-text search (openvibe-sdk/db)",
"port": 4710,
"port": 4710,
"domain": "search.openvibe.network (vhost in deploy/nginx; the service manifest records the search page and query API as public since 2026-09-23)",
"domain": "search.openvibe.network (vhost in deploy/nginx; the service manifest records the search page and query API as public since 2026-09-23)",
"contracts": "openvibe-contracts v0.76.0",
"contracts": "openvibe-contracts v0.76.0",
@@ -17,14 +17,14 @@
@@ -17,14 +17,14 @@
"contractsProposed": [
"contractsProposed": [
"search.index-document@1 (released in openvibe-contracts v0.12.0)"
"search.index-document@1 (released in openvibe-contracts v0.12.0)"
],
],
- "engine": "sqlite-fts5 (docs/adr-engine.md)",
+ "engine": "postgresql-fts (docs/adr-engine.md)",
"deployed": true,
"deployed": true,
"deployment": "Deployed with `sudo ovhost deploy search`: /opt/openvibe.search, unit openvibe-search, 127.0.0.1:4710, principal search, Events subscription *.index_document.*, accepted event owners sources, wiki, blog, news, reviews, trade, deals, coupons. On 2026-09-23 the production index held 10 documents, all tombstones (wiki), 0 public. Restore drill passed 2026-09-23.",
"deployment": "Deployed with `sudo ovhost deploy search`: /opt/openvibe.search, unit openvibe-search, 127.0.0.1:4710, principal search, Events subscription *.index_document.*, accepted event owners sources, wiki, blog, news, reviews, trade, deals, coupons. On 2026-09-23 the production index held 10 documents, all tombstones (wiki), 0 public. Restore drill passed 2026-09-23.",
"indexedProducers": [
"indexedProducers": [
"wiki (tombstones only so far)"
"wiki (tombstones only so far)"
],
],
"features": [
"features": [
- "ACL-filtered FTS5 query API",
+ "ACL-filtered full-text query API",
"facets",
"facets",
"suggestions",
"suggestions",
"freshness-weighted ranking",
"freshness-weighted ranking",
@@ -1,5 +1,5 @@
@@ -1,5 +1,5 @@
[Unit]
[Unit]
-Description=OpenVibe.Search — permission-aware index and query API (SQLite FTS5)
+Description=OpenVibe.Search — permission-aware index and query API (PostgreSQL full-text search)
# Network (openvibe-network) signs the tokens this service verifies. Search starts without it
# Network (openvibe-network) signs the tokens this service verifies. Search starts without it
# and keeps retrying the JWKS fetch; /api/ready stays 503 until the key has loaded.
# and keeps retrying the JWKS fetch; /api/ready stays 503 until the key has loaded.
After=network.target postgresql.service pgbouncer.service valkey-server.service
After=network.target postgresql.service pgbouncer.service valkey-server.service
@@ -23,9 +23,6 @@ NoNewPrivileges=true
@@ -23,9 +23,6 @@ NoNewPrivileges=true
PrivateTmp=true
PrivateTmp=true
ProtectSystem=strict
ProtectSystem=strict
ProtectHome=true
ProtectHome=true
-# The code tree stays read-only; the index lives in /var/lib/openvibe-search.
-StateDirectory=openvibe-search
-Environment=SEARCH_DB_PATH=/var/lib/openvibe-search/search.db
[Install]
[Install]
WantedBy=multi-user.target
WantedBy=multi-user.target
@@ -1,66 +1,54 @@
@@ -1,66 +1,54 @@
-# ADR: search engine for Wave 14 — SQLite FTS5 now, PostgreSQL FTS when Postgres exists
+# ADR: search engine for Wave 14 — PostgreSQL full-text search
-**Status:** accepted for alpha (2026-09-22) · **Owner:** OpenVibe.Search · **Roadmap:** §4.2 A, §15.12, Wave 14
+**Status:** accepted (2026-09-22; engine updated to PostgreSQL on 2026-09-28, ADR-035) · **Owner:** OpenVibe.Search · **Roadmap:** §4.2 A, §15.12, Wave 14
## Context
## Context
The roadmap asks for an event-fed, permission-aware index with the engine chosen only after
The roadmap asks for an event-fed, permission-aware index with the engine chosen only after
measuring need: *"PostgreSQL FTS first; Meilisearch or OpenSearch only if justified"*, and warns
measuring need: *"PostgreSQL FTS first; Meilisearch or OpenSearch only if justified"*, and warns
-against engine shopping. The network runs every service on one host with one SQLite database per
-service; PostgreSQL is not provisioned yet (it arrives with the legacy-retirement track). There
-is no corpus to measure: no product indexes anything today.
+against engine shopping. No external search engine is wanted: the corpus is SQL rows, the ACL
+model is SQL, and a visibility change must leave results in the same transaction that makes it.
+There is no corpus to measure: no product indexes anything today.
## Decision
## Decision
-1. **SQLite FTS5 inside Search's own database** (`/var/lib/openvibe-search/search.db`), no
- external engine, no new daemon. FTS5 ships in better-sqlite3's bundled SQLite (3.49), gives
- BM25 ranking, snippets and prefix queries, and commits in the same transaction as the
- document row, ACL rows, facet rows, the inbox receipt and the outbox event. That transaction
- is what makes "a visibility change or deletion removes the document from results immediately"
- true without a second system to converge.
-2. **Behind a small engine interface** — [server/engine/fts5.js](../server/engine/fts5.js):
- `put(rid, doc, exposure)`, `remove(rid)`, `search`, `browse`, `facetCounts`, `suggest`. The
- store, the revision rules, the ACL model and the APIs do not know FTS5 exists.
-3. **PostgreSQL FTS is the next engine**, as the roadmap says, once a PostgreSQL instance exists
- for services: a `tsvector` column with `websearch_to_tsquery`/`ts_rank_cd`, GIN index, the same
- ACL predicate as SQL. Meilisearch/OpenSearch stay out unless a measurement justifies them.
+1. **PostgreSQL full-text search in the service's own database** (`search_fts`, ADR-035), no
+ external engine, no new daemon. A `tsvector` weighted title A, summary B, body C with a GIN
+ index gives ranked matches, `ts_headline` snippets and prefix queries, and commits in the same
+ transaction as the document row, ACL rows, facet rows, the inbox receipt and the outbox event.
+ That transaction is what makes "a visibility change or deletion removes the document from
+ results immediately" true without a second system to converge.
+2. **Behind a small engine interface** — [server/engine/pg.js](../server/engine/pg.js):
+ `put(rid, doc, exposure)`, `remove(rid)`, `search`, `browse`, `facetCounts`, `suggest`, `counts`,
+ `reconcile`. The store, the revision rules, the ACL model and the APIs do not know the engine.
+3. **Meilisearch/OpenSearch stay out** unless a measurement justifies them, as the roadmap says.
## How visibility is enforced inside the engine
## How visibility is enforced inside the engine
-- Documents are split into two FTS tables by audience: `fts_public` holds only public,
- published, indexable documents; `fts_restricted` only published unlisted/members/private
- documents. Drafts, unpublished, deleted and public-noindex documents are in neither, so no text
- query can reach them.
-- An anonymous query (or a viewer without a subject) only ever reads `fts_public`. BM25 statistics
- are per table, so private documents cannot even shift the ranking an anonymous caller sees.
-- A signed-in query adds a second branch over `fts_restricted`, filtered in the same SQL by the
- viewer's ACL keys (`doc_acl`: subject, group, entitlement). Facet counts and suggestions reuse
- the exact same visible-set SQL, so they cannot count or complete what the viewer cannot see.
-- Known limitation: within `fts_restricted`, BM25 term statistics include restricted documents the
+- One table, `search_fts`, split by the `audience` column: exposure 3 holds only public, published,
+ indexable documents; exposure 1 holds published unlisted/members/private documents. Drafts,
+ unpublished, deleted and public-noindex documents have no row, so no text query can reach them.
+- An anonymous query (or a viewer without a subject) only ever reads audience 3. Ranking statistics
+ are computed over the same visible rows, so private documents cannot shift what an anonymous
+ caller sees.
+- A signed-in query adds a second branch over audience 1, filtered in the same SQL by the viewer's
+ ACL keys (`doc_acl`: subject, group, entitlement). Facet counts and suggestions reuse the exact
+ same visible-set SQL, so they cannot count or complete what the viewer cannot see.
+- Known limitation: within audience 1, `ts_rank` term statistics include restricted documents the
viewer cannot see. The effect on ordering is statistical and returns no content, but it is a
viewer cannot see. The effect on ordering is statistical and returns no content, but it is a
- theoretical side channel between signed-in users; a per-audience or per-tenant index (or
- PostgreSQL with ranking that ignores global statistics) removes it if it ever matters.
+ theoretical side channel between signed-in users; a per-audience or per-tenant index removes it
+ if it ever matters.
## Consequences
## Consequences
-- One process and one file: backups, restores and moves are the same as every other service.
-- Write throughput is SQLite's (single writer, WAL). That is ample for the expected corpus (tens to
- hundreds of thousands of documents); the trigger to revisit is measured: p95 query latency over
- 200 ms at the public edge, index size over a few GB, or write contention in `/api/ready`.
-- Tokenizer is `unicode61 remove_diacritics 2` without stemming: predictable across languages
- (Japanese/Chinese get no word segmentation; a later engine or an ICU tokenizer fixes that).
-- Moving to PostgreSQL is an engine swap plus a re-index from owners (the reconciliation API
- exists for exactly that), not a contract change.
-
-## Update 2026-09-28: PostgreSQL full-text search (ADR-035)
-
-Search moved to PostgreSQL with every other service. The engine interface held: `server/engine/pg.js` replaced
-`server/engine/fts5.js` and nothing else learned the engine changed. One table, `search_fts`, split by audience
-(exposure 3 public listed, 1 restricted), with a `tsvector` weighted title A, summary B, body C (FTS5's 10 / 4 / 1 as
-`ts_rank` weights) and a GIN index. Text is folded (diacritics removed) and split into runs of letters and digits before
-the `simple` configuration sees it, so tokens are exactly FTS5 unicode61's: on a copy of production, the match counts
-for sampled terms were identical. The rank is minus `ts_rank` times the same freshness boost, so ascending order and
-the keyset cursors work unchanged. Snippets are `ts_headline` over the summary and body (the title when only it
-matches), computed for the returned page only. The index is derived: at boot, when its counts disagree with the
-documents table (after the one-time import, whose FTS5 tables are not copied), it is rebuilt from `documents.doc`.
-
+- One database: backups, restores and moves are the same as every other service (PostgreSQL,
+ ADR-035).
+- Ranking is `ts_rank` with title/summary/body weights 10/4/1 times a freshness boost
+ ([README "Ranking"](../README.md#ranking)), returned as minus the score so ascending keyset
+ cursors work.
+- Text is folded (diacritics removed) and split into runs of letters and digits before the
+ `simple` configuration sees it, so tokens are predictable across languages (Japanese/Chinese get
+ no word segmentation; a later engine or ICU tokenizer fixes that) and a query and the index
+ always agree on what a word is.
+- The index is derived from `documents.doc`: at boot, when its counts disagree with the documents
+ table, it is rebuilt (the reconciliation API lets owners re-push everything if needed).
@@ -26,5 +26,5 @@
@@ -26,5 +26,5 @@
"*.index_document.deleted"
"*.index_document.deleted"
],
],
"namespacesOwned": [],
"namespacesOwned": [],
- "notes": "Canonical index document (search.index-document@1), event-fed and direct indexing with revision ordering and tombstones, permission-aware query over SQLite FTS5 (docs/adr-engine.md). Anonymous queries need no token."
+ "notes": "Canonical index document (search.index-document@1), event-fed and direct indexing with revision ordering and tombstones, permission-aware query over PostgreSQL full-text search (docs/adr-engine.md). Anonymous queries need no token."
}
}
@@ -11,7 +11,6 @@
@@ -11,7 +11,6 @@
"dependencies": {
"dependencies": {
"ajv": "^8.17.1",
"ajv": "^8.17.1",
"ajv-formats": "^3.0.1",
"ajv-formats": "^3.0.1",
- "better-sqlite3": "^11.10.0",
"dotenv": "^16.4.7",
"dotenv": "^16.4.7",
"express": "^4.21.2",
"express": "^4.21.2",
"openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",
"openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",
@@ -103,7 +102,9 @@
@@ -103,7 +102,9 @@
"url": "https://feross.org/support"
"url": "https://feross.org/support"
}
}
],
],
- "license": "MIT"
+ "license": "MIT",
+ "optional": true,
+ "peer": true
},
},
"node_modules/better-sqlite3": {
"node_modules/better-sqlite3": {
"version": "11.10.0",
"version": "11.10.0",
@@ -111,6 +112,8 @@
@@ -111,6 +112,8 @@
"integrity": "sha512-EwhOpyXiOEL/lKzHz9AW1msWFNzGc/z+LzeB3/jnFJpxu+th2yqvzsSWas1v9jgs9+xiXJcD5A8CJxAG2TaghQ==",
"integrity": "sha512-EwhOpyXiOEL/lKzHz9AW1msWFNzGc/z+LzeB3/jnFJpxu+th2yqvzsSWas1v9jgs9+xiXJcD5A8CJxAG2TaghQ==",
"hasInstallScript": true,
"hasInstallScript": true,
"license": "MIT",
"license": "MIT",
+ "optional": true,
+ "peer": true,
"dependencies": {
"dependencies": {
"bindings": "^1.5.0",
"bindings": "^1.5.0",
"prebuild-install": "^7.1.1"
"prebuild-install": "^7.1.1"
@@ -121,6 +124,8 @@
@@ -121,6 +124,8 @@
"resolved": "https://registry.npmjs.org/bindings/-/bindings-1.5.0.tgz",
"resolved": "https://registry.npmjs.org/bindings/-/bindings-1.5.0.tgz",
"integrity": "sha512-p2q/t/mhvuOj/UeLlV6566GD/guowlr0hHxClI0W9m7MWYkL1F0hLo+0Aexs9HSPCtR1SXQ0TD3MMKrXZajbiQ==",
"integrity": "sha512-p2q/t/mhvuOj/UeLlV6566GD/guowlr0hHxClI0W9m7MWYkL1F0hLo+0Aexs9HSPCtR1SXQ0TD3MMKrXZajbiQ==",
"license": "MIT",
"license": "MIT",
+ "optional": true,
+ "peer": true,
"dependencies": {
"dependencies": {
"file-uri-to-path": "1.0.0"
"file-uri-to-path": "1.0.0"
}
}
@@ -130,6 +135,8 @@
@@ -130,6 +135,8 @@
"resolved": "https://registry.npmjs.org/bl/-/bl-4.1.0.tgz",
"resolved": "https://registry.npmjs.org/bl/-/bl-4.1.0.tgz",
"integrity": "sha512-1W07cM9gS6DcLperZfFSj+bWLtaPGSOHWhPiGzXmvVJbRLdG82sH/Kn8EtW1VqWVA54AKf2h5k5BbnIbwF3h6w==",
"integrity": "sha512-1W07cM9gS6DcLperZfFSj+bWLtaPGSOHWhPiGzXmvVJbRLdG82sH/Kn8EtW1VqWVA54AKf2h5k5BbnIbwF3h6w==",
"license": "MIT",
"license": "MIT",
+ "optional": true,
+ "peer": true,
"dependencies": {
"dependencies": {
"buffer": "^5.5.0",
"buffer": "^5.5.0",
"inherits": "^2.0.4",
"inherits": "^2.0.4",
@@ -179,6 +186,8 @@
@@ -179,6 +186,8 @@
}
}
],
],
"license": "MIT",
"license": "MIT",
+ "optional": true,
+ "peer": true,
"dependencies": {
"dependencies": {
"base64-js": "^1.3.1",
"base64-js": "^1.3.1",
"ieee754": "^1.1.13"
"ieee754": "^1.1.13"
@@ -232,7 +241,9 @@
@@ -232,7 +241,9 @@
"version": "1.1.4",
"version": "1.1.4",
"resolved": "https://registry.npmjs.org/chownr/-/chownr-1.1.4.tgz",
"resolved": "https://registry.npmjs.org/chownr/-/chownr-1.1.4.tgz",
"integrity": "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==",
"integrity": "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==",
- "license": "ISC"
+ "license": "ISC",
+ "optional": true,
+ "peer": true
},
},
"node_modules/content-disposition": {
"node_modules/content-disposition": {
"version": "0.5.4",
"version": "0.5.4",
@@ -284,6 +295,8 @@
@@ -284,6 +295,8 @@
"resolved": "https://registry.npmjs.org/decompress-response/-/decompress-response-6.0.0.tgz",
"resolved": "https://registry.npmjs.org/decompress-response/-/decompress-response-6.0.0.tgz",
"integrity": "sha512-aW35yZM6Bb/4oJlZncMH2LCoZtJXTRxES17vE3hoRiowU2kWHaJKFkSBDnDR+cm9J+9QhXmREyIfv0pji9ejCQ==",
"integrity": "sha512-aW35yZM6Bb/4oJlZncMH2LCoZtJXTRxES17vE3hoRiowU2kWHaJKFkSBDnDR+cm9J+9QhXmREyIfv0pji9ejCQ==",
"license": "MIT",
"license": "MIT",
+ "optional": true,
+ "peer": true,
"dependencies": {
"dependencies": {
"mimic-response": "^3.1.0"
"mimic-response": "^3.1.0"
},
},
@@ -299,6 +312,8 @@
@@ -299,6 +312,8 @@
"resolved": "https://registry.npmjs.org/deep-extend/-/deep-extend-0.6.0.tgz",
"resolved": "https://registry.npmjs.org/deep-extend/-/deep-extend-0.6.0.tgz",
"integrity": "sha512-LOHxIOaPYdHlJRtCQfDIVZtfw/ufM8+rVj649RIHzcm/vGwQRXFt6OPqIFWsm2XEMrNIEtWR64sY1LEKD2vAOA==",
"integrity": "sha512-LOHxIOaPYdHlJRtCQfDIVZtfw/ufM8+rVj649RIHzcm/vGwQRXFt6OPqIFWsm2XEMrNIEtWR64sY1LEKD2vAOA==",
"license": "MIT",
"license": "MIT",
+ "optional": true,
+ "peer": true,
"engines": {
"engines": {
"node": ">=4.0.0"
"node": ">=4.0.0"
}
}
@@ -327,6 +342,8 @@
@@ -327,6 +342,8 @@
"resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz",
"resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz",
"integrity": "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==",
"integrity": "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==",
"license": "Apache-2.0",
"license": "Apache-2.0",
+ "optional": true,
+ "peer": true,
"engines": {
"engines": {
"node": ">=8"
"node": ">=8"
}
}
@@ -386,6 +403,8 @@
@@ -386,6 +403,8 @@
"resolved": "https://registry.npmjs.org/end-of-stream/-/end-of-stream-1.4.5.tgz",
"resolved": "https://registry.npmjs.org/end-of-stream/-/end-of-stream-1.4.5.tgz",
"integrity": "sha512-ooEGc6HP26xXq/N+GCGOT0JKCLDGrq2bQUZrQ7gyrJiZANJ/8YDTxTpQBXGMn+WbIQXNVpyWymm7KYVICQnyOg==",
"integrity": "sha512-ooEGc6HP26xXq/N+GCGOT0JKCLDGrq2bQUZrQ7gyrJiZANJ/8YDTxTpQBXGMn+WbIQXNVpyWymm7KYVICQnyOg==",
"license": "MIT",
"license": "MIT",
+ "optional": true,
+ "peer": true,
"dependencies": {
"dependencies": {
"once": "^1.4.0"
"once": "^1.4.0"
}
}
@@ -440,6 +459,8 @@
@@ -440,6 +459,8 @@
"resolved": "https://registry.npmjs.org/expand-template/-/expand-template-2.0.3.tgz",
"resolved": "https://registry.npmjs.org/expand-template/-/expand-template-2.0.3.tgz",
"integrity": "sha512-XYfuKMvj4O35f/pOXLObndIRvyQ+/+6AhODh+OKWj9S9498pHHn/IMszH+gt0fBCRWMNfk1ZSp5x3AifmnI2vg==",
"integrity": "sha512-XYfuKMvj4O35f/pOXLObndIRvyQ+/+6AhODh+OKWj9S9498pHHn/IMszH+gt0fBCRWMNfk1ZSp5x3AifmnI2vg==",
"license": "(MIT OR WTFPL)",
"license": "(MIT OR WTFPL)",
+ "optional": true,
+ "peer": true,
"engines": {
"engines": {
"node": ">=6"
"node": ">=6"
}
}
@@ -516,7 +537,9 @@
@@ -516,7 +537,9 @@
"version": "1.0.0",
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/file-uri-to-path/-/file-uri-to-path-1.0.0.tgz",
"resolved": "https://registry.npmjs.org/file-uri-to-path/-/file-uri-to-path-1.0.0.tgz",
"integrity": "sha512-0Zt+s3L7Vf1biwWZ29aARiVYLx7iMGnEUl9x33fbB/j3jR81u/O2LbqK+Bm1CDSNDKVtJ/YjwY7TUd5SkeLQLw==",
"integrity": "sha512-0Zt+s3L7Vf1biwWZ29aARiVYLx7iMGnEUl9x33fbB/j3jR81u/O2LbqK+Bm1CDSNDKVtJ/YjwY7TUd5SkeLQLw==",
- "license": "MIT"
+ "license": "MIT",
+ "optional": true,
+ "peer": true
},
},
"node_modules/finalhandler": {
"node_modules/finalhandler": {
"version": "1.3.2",
"version": "1.3.2",
@@ -558,7 +581,9 @@
@@ -558,7 +581,9 @@
"version": "1.0.0",
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/fs-constants/-/fs-constants-1.0.0.tgz",
"resolved": "https://registry.npmjs.org/fs-constants/-/fs-constants-1.0.0.tgz",
"integrity": "sha512-y6OAwoSIf7FyjMIv94u+b5rdheZEjzR63GTyZJm5qh4Bi+2YgwLCcI/fPFZkL5PSixOt6ZNKm+w+Hfp/Bciwow==",
"integrity": "sha512-y6OAwoSIf7FyjMIv94u+b5rdheZEjzR63GTyZJm5qh4Bi+2YgwLCcI/fPFZkL5PSixOt6ZNKm+w+Hfp/Bciwow==",
- "license": "MIT"
+ "license": "MIT",
+ "optional": true,
+ "peer": true
},
},
"node_modules/function-bind": {
"node_modules/function-bind": {
"version": "1.1.2",
"version": "1.1.2",
@@ -610,7 +635,9 @@
@@ -610,7 +635,9 @@
"version": "0.0.0",
"version": "0.0.0",
"resolved": "https://registry.npmjs.org/github-from-package/-/github-from-package-0.0.0.tgz",
"resolved": "https://registry.npmjs.org/github-from-package/-/github-from-package-0.0.0.tgz",
"integrity": "sha512-SyHy3T1v2NUXn29OsWdxmK6RwHD+vkj3v8en8AOBZ1wBQ/hCAQ5bAQTD02kW4W9tUp/3Qh6J8r9EvntiyCmOOw==",
"integrity": "sha512-SyHy3T1v2NUXn29OsWdxmK6RwHD+vkj3v8en8AOBZ1wBQ/hCAQ5bAQTD02kW4W9tUp/3Qh6J8r9EvntiyCmOOw==",
- "license": "MIT"
+ "license": "MIT",
+ "optional": true,
+ "peer": true
},
},
"node_modules/gopd": {
"node_modules/gopd": {
"version": "1.2.0",
"version": "1.2.0",
@@ -698,7 +725,9 @@
@@ -698,7 +725,9 @@
"url": "https://feross.org/support"
"url": "https://feross.org/support"
}
}
],
],
- "license": "BSD-3-Clause"
+ "license": "BSD-3-Clause",
+ "optional": true,
+ "peer": true
},
},
"node_modules/inherits": {
"node_modules/inherits": {
"version": "2.0.4",
"version": "2.0.4",
@@ -710,7 +739,9 @@
@@ -710,7 +739,9 @@
"version": "1.3.8",
"version": "1.3.8",
"resolved": "https://registry.npmjs.org/ini/-/ini-1.3.8.tgz",
"resolved": "https://registry.npmjs.org/ini/-/ini-1.3.8.tgz",
"integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==",
"integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==",
- "license": "ISC"
+ "license": "ISC",
+ "optional": true,
+ "peer": true
},
},
"node_modules/ipaddr.js": {
"node_modules/ipaddr.js": {
"version": "1.9.1",
"version": "1.9.1",
@@ -892,6 +923,8 @@
@@ -892,6 +923,8 @@
"resolved": "https://registry.npmjs.org/mimic-response/-/mimic-response-3.1.0.tgz",
"resolved": "https://registry.npmjs.org/mimic-response/-/mimic-response-3.1.0.tgz",
"integrity": "sha512-z0yWI+4FDrrweS8Zmt4Ej5HdJmky15+L2e6Wgn3+iK5fWzb6T3fhNFq2+MeTRb064c6Wr4N/wv0DzQTjNzHNGQ==",
"integrity": "sha512-z0yWI+4FDrrweS8Zmt4Ej5HdJmky15+L2e6Wgn3+iK5fWzb6T3fhNFq2+MeTRb064c6Wr4N/wv0DzQTjNzHNGQ==",
"license": "MIT",
"license": "MIT",
+ "optional": true,
+ "peer": true,
"engines": {
"engines": {
"node": ">=10"
"node": ">=10"
},
},
@@ -904,6 +937,8 @@
@@ -904,6 +937,8 @@
"resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.8.tgz",
"resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.8.tgz",
"integrity": "sha512-2yyAR8qBkN3YuheJanUpWC5U3bb5osDywNB8RzDVlDwDHbocAJveqqj1u8+SVD7jkWT4yvsHCpWqqWqAxb0zCA==",
"integrity": "sha512-2yyAR8qBkN3YuheJanUpWC5U3bb5osDywNB8RzDVlDwDHbocAJveqqj1u8+SVD7jkWT4yvsHCpWqqWqAxb0zCA==",
"license": "MIT",
"license": "MIT",
+ "optional": true,
+ "peer": true,
"funding": {
"funding": {
"url": "https://github.com/sponsors/ljharb"
"url": "https://github.com/sponsors/ljharb"
}
}
@@ -912,7 +947,9 @@
@@ -912,7 +947,9 @@
"version": "0.5.3",
"version": "0.5.3",
"resolved": "https://registry.npmjs.org/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz",
"resolved": "https://registry.npmjs.org/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz",
"integrity": "sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A==",
"integrity": "sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A==",
- "license": "MIT"
+ "license": "MIT",
+ "optional": true,
+ "peer": true
},
},
"node_modules/ms": {
"node_modules/ms": {
"version": "2.0.0",
"version": "2.0.0",
@@ -924,7 +961,9 @@
@@ -924,7 +961,9 @@
"version": "2.0.0",
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/napi-build-utils/-/napi-build-utils-2.0.0.tgz",
"resolved": "https://registry.npmjs.org/napi-build-utils/-/napi-build-utils-2.0.0.tgz",
"integrity": "sha512-GEbrYkbfF7MoNaoh2iGG84Mnf/WZfB0GdGEsM8wz7Expx/LlWf5U8t9nvJKXSp3qr5IsEbK04cBGhol/KwOsWA==",
"integrity": "sha512-GEbrYkbfF7MoNaoh2iGG84Mnf/WZfB0GdGEsM8wz7Expx/LlWf5U8t9nvJKXSp3qr5IsEbK04cBGhol/KwOsWA==",
- "license": "MIT"
+ "license": "MIT",
+ "optional": true,
+ "peer": true
},
},
"node_modules/negotiator": {
"node_modules/negotiator": {
"version": "0.6.3",
"version": "0.6.3",
@@ -940,6 +979,8 @@
@@ -940,6 +979,8 @@
"resolved": "https://registry.npmjs.org/node-abi/-/node-abi-3.96.0.tgz",
"resolved": "https://registry.npmjs.org/node-abi/-/node-abi-3.96.0.tgz",
"integrity": "sha512-rebQ/lz7i0EkoLzUVSrKRzA69zMkwLp95kKMWoMDkkM00Suxz0D7zEQPwRml5fQum24mj7bPvmlgLAmu2JCiYg==",
"integrity": "sha512-rebQ/lz7i0EkoLzUVSrKRzA69zMkwLp95kKMWoMDkkM00Suxz0D7zEQPwRml5fQum24mj7bPvmlgLAmu2JCiYg==",
"license": "MIT",
"license": "MIT",
+ "optional": true,
+ "peer": true,
"dependencies": {
"dependencies": {
"semver": "^7.3.5"
"semver": "^7.3.5"
},
},
@@ -976,6 +1017,8 @@
@@ -976,6 +1017,8 @@
"resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz",
"resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz",
"integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==",
"integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==",
"license": "ISC",
"license": "ISC",
+ "optional": true,
+ "peer": true,
"dependencies": {
"dependencies": {
"wrappy": "1"
"wrappy": "1"
}
}
@@ -1183,6 +1226,8 @@
@@ -1183,6 +1226,8 @@
"integrity": "sha512-8Mf2cbV7x1cXPUILADGI3wuhfqWvtiLA1iclTDbFRZkgRQS0NqsPZphna9V+HyTEadheuPmjaJMsbzKQFOzLug==",
"integrity": "sha512-8Mf2cbV7x1cXPUILADGI3wuhfqWvtiLA1iclTDbFRZkgRQS0NqsPZphna9V+HyTEadheuPmjaJMsbzKQFOzLug==",
"deprecated": "No longer maintained. Please contact the author of the relevant native addon; alternatives are available.",
"deprecated": "No longer maintained. Please contact the author of the relevant native addon; alternatives are available.",
"license": "MIT",
"license": "MIT",
+ "optional": true,
+ "peer": true,
"dependencies": {
"dependencies": {
"detect-libc": "^2.0.0",
"detect-libc": "^2.0.0",
"expand-template": "^2.0.3",
"expand-template": "^2.0.3",
@@ -1226,6 +1271,8 @@
@@ -1226,6 +1271,8 @@
"resolved": "https://registry.npmjs.org/pump/-/pump-3.0.4.tgz",
"resolved": "https://registry.npmjs.org/pump/-/pump-3.0.4.tgz",
"integrity": "sha512-VS7sjc6KR7e1ukRFhQSY5LM2uBWAUPiOPa/A3mkKmiMwSmRFUITt0xuj+/lesgnCv+dPIEYlkzrcyXgquIHMcA==",
"integrity": "sha512-VS7sjc6KR7e1ukRFhQSY5LM2uBWAUPiOPa/A3mkKmiMwSmRFUITt0xuj+/lesgnCv+dPIEYlkzrcyXgquIHMcA==",
"license": "MIT",
"license": "MIT",
+ "optional": true,
+ "peer": true,
"dependencies": {
"dependencies": {
"end-of-stream": "^1.1.0",
"end-of-stream": "^1.1.0",
"once": "^1.3.1"
"once": "^1.3.1"
@@ -1276,6 +1323,8 @@
@@ -1276,6 +1323,8 @@
"resolved": "https://registry.npmjs.org/rc/-/rc-1.2.8.tgz",
"resolved": "https://registry.npmjs.org/rc/-/rc-1.2.8.tgz",
"integrity": "sha512-y3bGgqKj3QBdxLbLkomlohkvsA8gdAiUQlSBJnBhfn+BPxg4bc62d8TcBW15wavDfgexCgccckhcZvywyQYPOw==",
"integrity": "sha512-y3bGgqKj3QBdxLbLkomlohkvsA8gdAiUQlSBJnBhfn+BPxg4bc62d8TcBW15wavDfgexCgccckhcZvywyQYPOw==",
"license": "(BSD-2-Clause OR MIT OR Apache-2.0)",
"license": "(BSD-2-Clause OR MIT OR Apache-2.0)",
+ "optional": true,
+ "peer": true,
"dependencies": {
"dependencies": {
"deep-extend": "^0.6.0",
"deep-extend": "^0.6.0",
"ini": "~1.3.0",
"ini": "~1.3.0",
@@ -1291,6 +1340,8 @@
@@ -1291,6 +1340,8 @@
"resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz",
"resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz",
"integrity": "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==",
"integrity": "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==",
"license": "MIT",
"license": "MIT",
+ "optional": true,
+ "peer": true,
"dependencies": {
"dependencies": {
"inherits": "^2.0.3",
"inherits": "^2.0.3",
"string_decoder": "^1.1.1",
"string_decoder": "^1.1.1",
@@ -1488,7 +1539,9 @@
@@ -1488,7 +1539,9 @@
"url": "https://feross.org/support"
"url": "https://feross.org/support"
}
}
],
],
- "license": "MIT"
+ "license": "MIT",
+ "optional": true,
+ "peer": true
},
},
"node_modules/simple-get": {
"node_modules/simple-get": {
"version": "4.0.1",
"version": "4.0.1",
@@ -1509,6 +1562,8 @@
@@ -1509,6 +1562,8 @@
}
}
],
],
"license": "MIT",
"license": "MIT",
+ "optional": true,
+ "peer": true,
"dependencies": {
"dependencies": {
"decompress-response": "^6.0.0",
"decompress-response": "^6.0.0",
"once": "^1.3.1",
"once": "^1.3.1",
@@ -1538,6 +1593,8 @@
@@ -1538,6 +1593,8 @@
"resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.3.0.tgz",
"resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.3.0.tgz",
"integrity": "sha512-hkRX8U1WjJFd8LsDJ2yQ/wWWxaopEsABU1XfkM8A+j0+85JAGppt16cr1Whg6KIbb4okU6Mql6BOj+uup/wKeA==",
"integrity": "sha512-hkRX8U1WjJFd8LsDJ2yQ/wWWxaopEsABU1XfkM8A+j0+85JAGppt16cr1Whg6KIbb4okU6Mql6BOj+uup/wKeA==",
"license": "MIT",
"license": "MIT",
+ "optional": true,
+ "peer": true,
"dependencies": {
"dependencies": {
"safe-buffer": "~5.2.0"
"safe-buffer": "~5.2.0"
}
}
@@ -1547,6 +1604,8 @@
@@ -1547,6 +1604,8 @@
"resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-2.0.1.tgz",
"resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-2.0.1.tgz",
"integrity": "sha512-4gB8na07fecVVkOI6Rs4e7T6NOTki5EmL7TUduTs6bu3EdnSycntVJ4re8kgZA+wx9IueI2Y11bfbgwtzuE0KQ==",
"integrity": "sha512-4gB8na07fecVVkOI6Rs4e7T6NOTki5EmL7TUduTs6bu3EdnSycntVJ4re8kgZA+wx9IueI2Y11bfbgwtzuE0KQ==",
"license": "MIT",
"license": "MIT",
+ "optional": true,
+ "peer": true,
"engines": {
"engines": {
"node": ">=0.10.0"
"node": ">=0.10.0"
}
}
@@ -1556,6 +1615,8 @@
@@ -1556,6 +1615,8 @@
"resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-2.1.5.tgz",
"resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-2.1.5.tgz",
"integrity": "sha512-OboTd8mmMhZDNPV+UjQcK9yKAatXu2aJ+r1w4im1Otd4M4fl2hwvdoXUxIYHFTHWK/3y3FarBP70v3vwmGlOxw==",
"integrity": "sha512-OboTd8mmMhZDNPV+UjQcK9yKAatXu2aJ+r1w4im1Otd4M4fl2hwvdoXUxIYHFTHWK/3y3FarBP70v3vwmGlOxw==",
"license": "MIT",
"license": "MIT",
+ "optional": true,
+ "peer": true,
"dependencies": {
"dependencies": {
"chownr": "^1.1.1",
"chownr": "^1.1.1",
"mkdirp-classic": "^0.5.2",
"mkdirp-classic": "^0.5.2",
@@ -1568,6 +1629,8 @@
@@ -1568,6 +1629,8 @@
"resolved": "https://registry.npmjs.org/tar-stream/-/tar-stream-2.2.0.tgz",
"resolved": "https://registry.npmjs.org/tar-stream/-/tar-stream-2.2.0.tgz",
"integrity": "sha512-ujeqbceABgwMZxEJnk2HDY2DlnUZ+9oEcb1KzTVfYHio0UE6dG71n60d8D2I4qNvleWrrXpmjpt7vZeF1LnMZQ==",
"integrity": "sha512-ujeqbceABgwMZxEJnk2HDY2DlnUZ+9oEcb1KzTVfYHio0UE6dG71n60d8D2I4qNvleWrrXpmjpt7vZeF1LnMZQ==",
"license": "MIT",
"license": "MIT",
+ "optional": true,
+ "peer": true,
"dependencies": {
"dependencies": {
"bl": "^4.0.3",
"bl": "^4.0.3",
"end-of-stream": "^1.4.1",
"end-of-stream": "^1.4.1",
@@ -1593,6 +1656,8 @@
@@ -1593,6 +1656,8 @@
"resolved": "https://registry.npmjs.org/tunnel-agent/-/tunnel-agent-0.6.0.tgz",
"resolved": "https://registry.npmjs.org/tunnel-agent/-/tunnel-agent-0.6.0.tgz",
"integrity": "sha512-McnNiV1l8RYeY8tBgEpuodCC1mLUdbSN+CYBL7kJsJNInOP8UjDDEwdk6Mw60vdLLrr5NHKZhMAOSrR2NZuQ+w==",
"integrity": "sha512-McnNiV1l8RYeY8tBgEpuodCC1mLUdbSN+CYBL7kJsJNInOP8UjDDEwdk6Mw60vdLLrr5NHKZhMAOSrR2NZuQ+w==",
"license": "Apache-2.0",
"license": "Apache-2.0",
+ "optional": true,
+ "peer": true,
"dependencies": {
"dependencies": {
"safe-buffer": "^5.0.1"
"safe-buffer": "^5.0.1"
},
},
@@ -1626,7 +1691,9 @@
@@ -1626,7 +1691,9 @@
"version": "1.0.2",
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz",
"resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz",
"integrity": "sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==",
"integrity": "sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==",
- "license": "MIT"
+ "license": "MIT",
+ "optional": true,
+ "peer": true
},
},
"node_modules/utils-merge": {
"node_modules/utils-merge": {
"version": "1.0.1",
"version": "1.0.1",
@@ -1650,7 +1717,9 @@
@@ -1650,7 +1717,9 @@
"version": "1.0.2",
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",
"resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",
"integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==",
"integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==",
- "license": "ISC"
+ "license": "ISC",
+ "optional": true,
+ "peer": true
},
},
"node_modules/xtend": {
"node_modules/xtend": {
"version": "4.0.2",
"version": "4.0.2",
@@ -2,7 +2,7 @@
@@ -2,7 +2,7 @@
"name": "openvibe-search",
"name": "openvibe-search",
"version": "0.1.0",
"version": "0.1.0",
"private": true,
"private": true,
- "description": "OpenVibe.Search: the canonical index document, event-fed and direct indexing with revision ordering and tombstones, and a permission-aware query API over SQLite FTS5.",
+ "description": "OpenVibe.Search: the canonical index document, event-fed and direct indexing with revision ordering and tombstones, and a permission-aware query API over PostgreSQL full-text search.",
"main": "server/index.js",
"main": "server/index.js",
"scripts": {
"scripts": {
"start": "node server/index.js",
"start": "node server/index.js",
@@ -14,7 +14,7 @@
@@ -14,7 +14,7 @@
"keywords": [
"keywords": [
"openvibe",
"openvibe",
"search",
"search",
- "fts5",
+ "full-text",
"acl"
"acl"
],
],
"author": "OpenVibers",
"author": "OpenVibers",
@@ -23,7 +23,6 @@
@@ -23,7 +23,6 @@
"dependencies": {
"dependencies": {
"ajv": "^8.17.1",
"ajv": "^8.17.1",
"ajv-formats": "^3.0.1",
"ajv-formats": "^3.0.1",
- "better-sqlite3": "^11.10.0",
"dotenv": "^16.4.7",
"dotenv": "^16.4.7",
"express": "^4.21.2",
"express": "^4.21.2",
"openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",
"openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",
deleted file mode 100644
deleted file mode 100644
@@ -1,26 +0,0 @@
@@ -1,26 +0,0 @@
-#!/usr/bin/env node
-'use strict';
-/**
- * The one-time move of Search's SQLite database (SEARCH_DB_PATH) into its PostgreSQL schema (ADR-035; the procedure is
- * openvibe-sdk docs/migrating-to-postgresql.md, section 6, carried out by openvibe-sdk/db runSqliteMigration).
- *
- * node scripts/migrate-to-postgres.js [--sqlite <file>] [--pglite] [--json]
- *
- * Applies migrations/ as the owner (DATABASE_DIRECT_URL), copies every table into emptied tables, verifies row counts
- * and checksums, and exits 1 unless everything verified. The SQLite file is opened read-only. Every table keeps its
- * name and columns.
- */
-require('dotenv').config();
-const { runSqliteMigration } = require('openvibe-sdk/db');
-const configLib = require('../server/config');
-const { MIGRATIONS } = require('../server/db');
-
-const TABLES = {};
-
-if (require.main === module) {
- const config = configLib.load();
- runSqliteMigration({ service: 'search', sqlite: config.dbPath, directUrl: config.db.directUrl, migrations: MIGRATIONS, tables: TABLES })
- .then((code) => process.exit(code), (err) => { console.error(`migrate-to-postgres failed: ${err.message}`); process.exit(1); });
-}
-
-module.exports = { TABLES };
@@ -9,7 +9,7 @@
@@ -9,7 +9,7 @@
* with envelope subject { type, id, revision } and visibility "internal".
* with envelope subject { type, id, revision } and visibility "internal".
*
*
* Exactly once: the inbox receipt (consumer, event_id) and the document change commit in one
* Exactly once: the inbox receipt (consumer, event_id) and the document change commit in one
- * SQLite transaction. A redelivery of a processed event is answered 204 and changes nothing.
+ * database transaction. A redelivery of a processed event is answered 204 and changes nothing.
* Revision order is the store's rule, so out-of-order delivery is harmless.
* Revision order is the store's rule, so out-of-order delivery is harmless.
*
*
* Refusals (unknown owner, a source writing another owner's document, a bad document, a subject
* Refusals (unknown owner, a source writing another owner's document, a bad document, a subject
@@ -72,8 +72,6 @@ function load(env = process.env) {
@@ -72,8 +72,6 @@ function load(env = process.env) {
// PostgreSQL (ADR-035): DATABASE_URL serves (PgBouncer), DATABASE_DIRECT_URL migrates (owner role).
// PostgreSQL (ADR-035): DATABASE_URL serves (PgBouncer), DATABASE_DIRECT_URL migrates (owner role).
db: { url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '' },
db: { url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '' },
- // The SQLite file of releases before the switch: read once by scripts/migrate-to-postgres.js.
- dbPath: env.SEARCH_DB_PATH || './data/search.db',
// Events: the outbox relays only when EVENTS_URL is set (rows wait otherwise).
// Events: the outbox relays only when EVENTS_URL is set (rows wait otherwise).
events: {
events: {
@@ -1,7 +1,7 @@
@@ -1,7 +1,7 @@
'use strict';
'use strict';
/**
/**
- * Search engine: PostgreSQL full-text search in the service's own database (ADR-035; it replaced SQLite FTS5, the
- * engine docs/adr-engine.md chose, with the same interface and the same guarantees).
+ * Search engine: PostgreSQL full-text search in the service's own database (ADR-035; the engine docs/adr-engine.md
+ * chose, with the same interface and the same guarantees).
*
*
* One table, search_fts, split by audience so an anonymous query never touches, ranks against or counts anything but
* One table, search_fts, split by audience so an anonymous query never touches, ranks against or counts anything but
* public listed documents:
* public listed documents:
@@ -13,15 +13,16 @@
@@ -13,15 +13,16 @@
* also filtered by `documents.exposure` and, for restricted rows, by the viewer's ACL keys (doc_acl) in the same SQL:
* also filtered by `documents.exposure` and, for restricted rows, by the viewer's ACL keys (doc_acl) in the same SQL:
* visibility is decided before anything is ranked, counted, snippeted or suggested.
* visibility is decided before anything is ranked, counted, snippeted or suggested.
*
*
- * Text is folded before it is indexed or queried (NFKD, combining marks removed: FTS5's remove_diacritics), split into
- * runs of letters and digits (FTS5's unicode61) and tokenized by the `simple` configuration (lower case, no stemming, no stop words: FTS5's unicode61). Title, summary and
- * body weigh 10, 4 and 1 (tsvector weights A, B, C). Every function here is async; put and remove run inside the store's
+ * Text is folded before it is indexed or queried (NFKD, combining marks removed), split into runs of letters and
+ * digits and tokenized by the `simple` configuration (lower case, no stemming, no stop words): the index and every
+ * query see the same text. Title, summary and body weigh 10, 4 and 1 (tsvector weights A, B, C), with the freshness
+ * boost described below. Every function here is async; put and remove run inside the store's
* transaction (ambient). Nothing outside this file writes SQL against search_fts.
* transaction (ambient). Nothing outside this file writes SQL against search_fts.
*/
*/
const { EXPOSURE } = require('../document');
const { EXPOSURE } = require('../document');
const CONFIG = 'simple';
const CONFIG = 'simple';
-const WEIGHTS = '{0.1, 0.1, 0.4, 1.0}'; // D, C body, B summary, A title: FTS5's bm25 weights 10 / 4 / 1
+const WEIGHTS = '{0.1, 0.1, 0.4, 1.0}'; // D, C body, B summary, A title: ts_rank weights 1 / 4 / 10
const MARK_OPEN = '\u0001';
const MARK_OPEN = '\u0001';
const MARK_CLOSE = '\u0002';
const MARK_CLOSE = '\u0002';
const HEADLINE = `StartSel=${MARK_OPEN}, StopSel=${MARK_CLOSE}, MaxWords=16, MinWords=8, ShortWord=0, MaxFragments=1, FragmentDelimiter=" … "`;
const HEADLINE = `StartSel=${MARK_OPEN}, StopSel=${MARK_CLOSE}, MaxWords=16, MinWords=8, ShortWord=0, MaxFragments=1, FragmentDelimiter=" … "`;
@@ -47,15 +48,15 @@ function freshnessBoost(iso, nowMs, halfLifeDays, weight) {
@@ -47,15 +48,15 @@ function freshnessBoost(iso, nowMs, halfLifeDays, weight) {
return 1 + weight * Math.pow(2, -ageDays / halfLifeDays);
return 1 + weight * Math.pow(2, -ageDays / halfLifeDays);
}
}
-/** Diacritics folded, as FTS5's remove_diacritics did: the index and every query see the same text. */
+/** Diacritics folded, so the index and every query see the same text. */
function fold(s) {
function fold(s) {
return String(s || '').normalize('NFKD').replace(/\p{M}+/gu, '').normalize('NFC');
return String(s || '').normalize('NFKD').replace(/\p{M}+/gu, '').normalize('NFC');
}
}
/**
/**
- * The words FTS5's unicode61 tokenizer saw: runs of letters and digits, everything else a separator. PostgreSQL's
- * parser would keep "OpenVibe.Live", "a_b" or "3.5" as single tokens (host, word, number); separating them first makes
- * "openvibe" find "OpenVibe.Live" as it did.
+ * Word tokens: runs of letters and digits, everything else a separator. PostgreSQL's parser would
+ * keep "OpenVibe.Live", "a_b" or "3.5" as single tokens (host, word, number); separating them first
+ * makes "openvibe" find "OpenVibe.Live".
*/
*/
function words(s) {
function words(s) {
return fold(s).replace(/[^\p{L}\p{N}]+/gu, ' ');
return fold(s).replace(/[^\p{L}\p{N}]+/gu, ' ');
@@ -124,7 +125,7 @@ function createEngine(db, { freshness = { weight: 1, halfLifeDays: 30 } } = {})
@@ -124,7 +125,7 @@ function createEngine(db, { freshness = { weight: 1, halfLifeDays: 30 } } = {})
/**
/**
* The visible, matching set as (rid, rank) rows. match = toMatch()'s query. rank is minus ts_rank times the
* The visible, matching set as (rid, rank) rows. match = toMatch()'s query. rank is minus ts_rank times the
- * freshness boost at reference time `now` (ms), so lower ranks first (as FTS5's bm25 did) and pages of one query
+ * freshness boost at reference time `now` (ms), so lower ranks first and pages of one query
* (which carry their first page's `now` in the cursor) rank against the same clock.
* (which carry their first page's `now` in the cursor) rank against the same clock.
*/
*/
function matchedSet({ match, filters, viewer, now = Date.now() }, params) {
function matchedSet({ match, filters, viewer, now = Date.now() }, params) {
@@ -234,8 +235,8 @@ function createEngine(db, { freshness = { weight: 1, halfLifeDays: 30 } } = {})
@@ -234,8 +235,8 @@ function createEngine(db, { freshness = { weight: 1, halfLifeDays: 30 } } = {})
}
}
/**
/**
- * Rebuild the index from documents.doc when its counts disagree with the documents table (after the one-time import
- * from SQLite, whose FTS5 tables are not copied). Returns what it did.
+ * Rebuild the index from documents.doc when its counts disagree with the documents table (for instance after a
+ * restore). Returns what it did.
*/
*/
async function reconcile() {
async function reconcile() {
const docs = await db.prepare(`SELECT
const docs = await db.prepare(`SELECT
@@ -16,22 +16,6 @@ const { ids, validate, serviceAuth } = require('openvibe-contracts');
@@ -16,22 +16,6 @@ const { ids, validate, serviceAuth } = require('openvibe-contracts');
const BACKOFF_MS = [1000, 5000, 30000, 120000, 600000];
const BACKOFF_MS = [1000, 5000, 30000, 120000, 600000];
-function ensureSchema(db) {
- db.exec(`CREATE TABLE IF NOT EXISTS event_outbox (
- seq INTEGER PRIMARY KEY AUTOINCREMENT,
- event_id TEXT NOT NULL UNIQUE,
- event_type TEXT NOT NULL,
- envelope TEXT NOT NULL,
- created_at INTEGER NOT NULL,
- attempts INTEGER NOT NULL DEFAULT 0,
- next_attempt_at INTEGER NOT NULL DEFAULT 0,
- sent_at INTEGER,
- rejected_at INTEGER,
- last_error TEXT
- );
- CREATE INDEX IF NOT EXISTS idx_event_outbox_due ON event_outbox(sent_at, rejected_at, next_attempt_at);`);
-}
-
function createOutbox(db, { source, now = () => Date.now() }) {
function createOutbox(db, { source, now = () => Date.now() }) {
const insert = db.prepare('INSERT INTO event_outbox (event_id, event_type, envelope, created_at) VALUES (?, ?, ?, ?)');
const insert = db.prepare('INSERT INTO event_outbox (event_id, event_type, envelope, created_at) VALUES (?, ?, ?, ?)');
@@ -176,4 +160,4 @@ function createRelay({ outbox, eventsUrl, intervalMs = 2000, tokenClient, tokenO
@@ -176,4 +160,4 @@ function createRelay({ outbox, eventsUrl, intervalMs = 2000, tokenClient, tokenO
return { start, stop, flush, running: () => Boolean(timer) };
return { start, stop, flush, running: () => Boolean(timer) };
}
}
-module.exports = { ensureSchema, createOutbox, createRelay };
+module.exports = { createOutbox, createRelay };
@@ -22,7 +22,7 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetch
@@ -22,7 +22,7 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetch
// PostgreSQL (ADR-035): opened and migrated here unless the caller (a test) hands in a migrated handle.
// PostgreSQL (ADR-035): opened and migrated here unless the caller (a test) hands in a migrated handle.
const db = givenDb || await openDb(config, { log });
const db = givenDb || await openDb(config, { log });
const engine = createEngine(db, { freshness: config.freshness });
const engine = createEngine(db, { freshness: config.freshness });
- // The index is derived from documents: rebuilt when it disagrees (after the one-time import from SQLite).
+ // The index is derived from documents: rebuilt at boot when it disagrees with the documents table.
const rebuilt = await engine.reconcile();
const rebuilt = await engine.reconcile();
if (rebuilt.rebuilt) log.log(`[search] full-text index rebuilt: ${rebuilt.rebuilt} document(s)`);
if (rebuilt.rebuilt) log.log(`[search] full-text index rebuilt: ${rebuilt.rebuilt} document(s)`);
const outbox = createOutbox(db, { source: config.serviceId, now });
const outbox = createOutbox(db, { source: config.serviceId, now });
@@ -3,16 +3,16 @@
@@ -3,16 +3,16 @@
* Track O: truthful readiness for GET /api/ready and the Search gauges on GET /metrics
* Track O: truthful readiness for GET /api/ready and the Search gauges on GET /metrics
* (openvibe-shared/ready and openvibe-shared/metrics).
* (openvibe-shared/ready and openvibe-shared/metrics).
*
*
- * db required a real read of the documents table and of both full-text tables (they
- * live in the same SQLite file): without them nothing can be indexed or found
+ * db required a real read of the documents table and of the full-text table (they
+ * live in the same database): without them nothing can be indexed or found
* network_jwks optional the Network signing key has loaded. Without it anonymous queries still
* network_jwks optional the Network signing key has loaded. Without it anonymous queries still
* answer (public documents only), but no token can be verified: owner
* answer (public documents only), but no token can be verified: owner
* writes and signed-in queries fail, so it degrades rather than fails
* writes and signed-in queries fail, so it degrades rather than fails
- * index optional the full-text index agrees with the documents table: fts_public holds
- * exactly the public_listed documents and fts_restricted the restricted
- * ones. A mismatch means queries miss (or would surface) documents
+ * index optional the full-text index agrees with the documents table: search_fts holds
+ * exactly the public_listed and restricted documents. A mismatch means
+ * queries miss (or would surface) documents
*
*
- * Gauges: documents by exposure, full-text entries by table, and the events outbox backlog.
+ * Gauges: documents by exposure, full-text entries by audience, and the events outbox backlog.
*/
*/
const { createReadiness } = require('openvibe-shared/ready');
const { createReadiness } = require('openvibe-shared/ready');
const { EXPOSURE } = require('./document');
const { EXPOSURE } = require('./document');
@@ -90,7 +90,7 @@ function registerSearchGauges(registry, { db, engine, outbox, purges = null }) {
@@ -90,7 +90,7 @@ function registerSearchGauges(registry, { db, engine, outbox, purges = null }) {
collect: async () => Object.entries(await read.documents()).map(([exposure, value]) => ({ labels: { exposure }, value })),
collect: async () => Object.entries(await read.documents()).map(([exposure, value]) => ({ labels: { exposure }, value })),
});
});
registry.gauge({
registry.gauge({
- name: 'search_documents_indexed', help: 'Documents in the full-text index, by table', labelNames: ['index'],
+ name: 'search_documents_indexed', help: 'Documents in the full-text index, by audience', labelNames: ['index'],
collect: async () => Object.entries(await read.indexed()).map(([index, value]) => ({ labels: { index }, value })),
collect: async () => Object.entries(await read.indexed()).map(([index, value]) => ({ labels: { index }, value })),
});
});
registry.gauge({ name: 'search_outbox_pending', help: 'Events waiting in the outbox', collect: async () => await outbox.pending() });
registry.gauge({ name: 'search_outbox_pending', help: 'Events waiting in the outbox', collect: async () => await outbox.pending() });
@@ -25,42 +25,6 @@ const MAX_ATTEMPTS = 8;
@@ -25,42 +25,6 @@ const MAX_ATTEMPTS = 8;
const FILES_PER_CALL = 30;
const FILES_PER_CALL = 30;
const DETAIL_MAX = 300;
const DETAIL_MAX = 300;
-function ensureSchema(db) {
- db.exec(`
- CREATE TABLE IF NOT EXISTS removals (
- seq INTEGER PRIMARY KEY AUTOINCREMENT,
- event_id TEXT NOT NULL, -- the search.document.removed event
- owner TEXT NOT NULL,
- type TEXT NOT NULL,
- id TEXT NOT NULL,
- revision INTEGER NOT NULL,
- reason TEXT NOT NULL,
- previous_exposure TEXT NOT NULL,
- exposure TEXT NOT NULL,
- canonical_url TEXT, -- only a URL that had been public
- at INTEGER NOT NULL
- );
- CREATE INDEX IF NOT EXISTS idx_removals_owner ON removals (owner, seq);
-
- CREATE TABLE IF NOT EXISTS cdn_purges (
- seq INTEGER PRIMARY KEY AUTOINCREMENT,
- removal_seq INTEGER NOT NULL,
- provider TEXT NOT NULL DEFAULT 'cloudflare',
- zone_id TEXT,
- url TEXT NOT NULL,
- state TEXT NOT NULL CHECK (state IN ('pending', 'purged', 'failed', 'skipped')),
- detail TEXT,
- attempts INTEGER NOT NULL DEFAULT 0,
- next_attempt_at INTEGER NOT NULL DEFAULT 0,
- created_at INTEGER NOT NULL,
- done_at INTEGER
- );
- -- One pending purge per URL: a sitemap shared by many removals is purged once per batch.
- CREATE UNIQUE INDEX IF NOT EXISTS idx_cdn_purges_pending ON cdn_purges (zone_id, url) WHERE state = 'pending';
- CREATE INDEX IF NOT EXISTS idx_cdn_purges_due ON cdn_purges (state, next_attempt_at, seq);
- `);
-}
-
/** The zone a URL belongs to: exact host or a parent domain in the map, longest first. */
/** The zone a URL belongs to: exact host or a parent domain in the map, longest first. */
function zoneFor(url, zones) {
function zoneFor(url, zones) {
let u;
let u;
@@ -254,4 +218,4 @@ function createPurger({ db, config, fetchImpl = globalThis.fetch, log = console,
@@ -254,4 +218,4 @@ function createPurger({ db, config, fetchImpl = globalThis.fetch, log = console,
return { start, stop, flush, running: () => Boolean(timer) };
return { start, stop, flush, running: () => Boolean(timer) };
}
}
-module.exports = { ensureSchema, createPurgeQueue, createPurger, zoneFor, purgeUrls, MAX_ATTEMPTS, FILES_PER_CALL };
+module.exports = { createPurgeQueue, createPurger, zoneFor, purgeUrls, MAX_ATTEMPTS, FILES_PER_CALL };
@@ -13,24 +13,6 @@ const { ids } = require('openvibe-contracts');
@@ -13,24 +13,6 @@ const { ids } = require('openvibe-contracts');
const ID_RE = /^svs_[0-9a-z]{26}$/;
const ID_RE = /^svs_[0-9a-z]{26}$/;
-function ensureSchema(db) {
- db.exec(`
- CREATE TABLE IF NOT EXISTS saved_searches (
- id TEXT PRIMARY KEY,
- subject TEXT NOT NULL, -- usr_… (the person who saved it)
- name TEXT NOT NULL,
- q TEXT NOT NULL DEFAULT '',
- filters TEXT NOT NULL, -- canonical JSON {owner,type,language,facets}
- query_hash TEXT NOT NULL, -- same query saved twice = one row
- created_at INTEGER NOT NULL,
- updated_at INTEGER NOT NULL,
- last_run_at INTEGER,
- UNIQUE (subject, query_hash)
- );
- CREATE INDEX IF NOT EXISTS idx_saved_searches_subject ON saved_searches (subject, created_at);
- `);
-}
-
/** Filters in one canonical shape, so equal queries hash equally whatever order they came in. */
/** Filters in one canonical shape, so equal queries hash equally whatever order they came in. */
function canonicalFilters(f) {
function canonicalFilters(f) {
const out = {};
const out = {};
@@ -111,4 +93,4 @@ function createSavedSearches(db, { maxPerSubject = 50, now = () => Date.now() }
@@ -111,4 +93,4 @@ function createSavedSearches(db, { maxPerSubject = 50, now = () => Date.now() }
};
};
}
}
-module.exports = { ensureSchema, createSavedSearches, canonicalFilters, ID_RE };
+module.exports = { createSavedSearches, canonicalFilters, ID_RE };
@@ -5,9 +5,6 @@
@@ -5,9 +5,6 @@
* Events delivery helper. Nothing here needs the network or a running OpenVibe.Network.
* Events delivery helper. Nothing here needs the network or a running OpenVibe.Network.
*/
*/
const crypto = require('crypto');
const crypto = require('crypto');
-const fs = require('fs');
-const os = require('os');
-const path = require('path');
const { serviceAuth, ids } = require('openvibe-contracts');
const { serviceAuth, ids } = require('openvibe-contracts');
const { load } = require('../server/config');
const { load } = require('../server/config');
const { start } = require('../server/index');
const { start } = require('../server/index');
@@ -37,21 +34,10 @@ function userToken({ subjectId = ids.newId('user'), role = 'user', aud = ['openv
@@ -37,21 +34,10 @@ function userToken({ subjectId = ids.newId('user'), role = 'user', aud = ['openv
}, key);
}, key);
}
}
-const made = [];
-process.on('exit', () => { for (const d of made) fs.rmSync(d, { recursive: true, force: true }); });
-
-function tmpDir() {
- const d = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-search-test-'));
- made.push(d);
- return d;
-}
-
async function boot({ env = {}, fetchImpl, tokenClient, now } = {}) {
async function boot({ env = {}, fetchImpl, tokenClient, now } = {}) {
- const dir = tmpDir();
const config = load({
const config = load({
NODE_ENV: 'test',
NODE_ENV: 'test',
PORT: '0',
PORT: '0',
- SEARCH_DB_PATH: path.join(dir, 'search.db'),
OV_NETWORK_PUBLIC_KEY: publicKey,
OV_NETWORK_PUBLIC_KEY: publicKey,
SEARCH_EVENTS_SECRET: WEBHOOK_SECRET,
SEARCH_EVENTS_SECRET: WEBHOOK_SECRET,
...env,
...env,
@@ -60,7 +46,7 @@ async function boot({ env = {}, fetchImpl, tokenClient, now } = {}) {
@@ -60,7 +46,7 @@ async function boot({ env = {}, fetchImpl, tokenClient, now } = {}) {
const testdb = await require('./db').testDb();
const testdb = await require('./db').testDb();
const h = await start({ config, db: testdb.db, log: silent, fetchImpl, tokenClient, now });
const h = await start({ config, db: testdb.db, log: silent, fetchImpl, tokenClient, now });
const base = `http://127.0.0.1:${h.server.address().port}`;
const base = `http://127.0.0.1:${h.server.address().port}`;
- return { ...h, base, dir, async stop() { await h.close(); await testdb.close(); } };
+ return { ...h, base, async stop() { await h.close(); await testdb.close(); } };
}
}
async function request(base, method, p, { token, body, headers = {}, raw } = {}) {
async function request(base, method, p, { token, body, headers = {}, raw } = {}) {
@@ -145,6 +131,6 @@ function suite(name) {
@@ -145,6 +131,6 @@ function suite(name) {
}
}
module.exports = {
module.exports = {
- ISSUER, WEBHOOK_SECRET, privateKey, publicKey, silent, serviceToken, userToken, tmpDir, boot, request,
+ ISSUER, WEBHOOK_SECRET, privateKey, publicKey, silent, serviceToken, userToken, boot, request,
doc, indexEvent, deliver, suite,
doc, indexEvent, deliver, suite,
};
};
@@ -1,8 +1,9 @@
@@ -1,8 +1,9 @@
#!/usr/bin/env node
#!/usr/bin/env node
/**
/**
* Runs every test in test/ — the files named *.test.js — each in its own process, and fails
* Runs every test in test/ — the files named *.test.js — each in its own process, and fails
- * if any of them fails. They use temp SQLite databases, generated RSA keys and stub subscribers
- * on random ports; none of them needs the network or a running OpenVibe.Network.
+ * if any of them fails. They boot a PGlite database per test file (PostgreSQL with
+ * SEARCH_TEST_STORE=pg), with generated RSA keys and stub subscribers on random ports; none of
+ * them needs the network or a running OpenVibe.Network.
*
*
* npm test # everything
* npm test # everything
* npm test -- publish sse # only files whose name contains one of the words
* npm test -- publish sse # only files whose name contains one of the words