openvibers/openvibe.search Commits
W14: removal purge queue, freshness ranking, saved searches, a search page and the public vhost
Changes
@@@ -42,3 +42,23 @@ SEARCH_DB_PATH=./data/search.db# Largest accepted request body (direct PUT and deliveries), bytes.#SEARCH_MAX_BODY_BYTES=262144++++# Ranking: freshness weighting of full-text relevance (README "Ranking").++# score = bm25 × (1 + WEIGHT × 2^(−age_days / HALF_LIFE_DAYS)); age from published_at, else++# updated_at. WEIGHT 0 turns it off.++#SEARCH_FRESHNESS_WEIGHT=1++#SEARCH_FRESHNESS_HALF_LIFE_DAYS=30++++# Saved searches per signed-in person.++#SEARCH_SAVED_MAX_PER_SUBJECT=50++++# Removal purge queue (README "Removals, caches and the CDN"). Every removal is recorded for++# its owner (GET /api/v1/owners/:owner/removals). The Cloudflare purge of a formerly public URL++# runs only when CLOUDFLARE_PURGE_TOKEN is set (inert otherwise). The token needs only++# Zone → Cache Purge → Purge on the zones below. Never commit it: /etc/openvibe/search.env (0600).++#CLOUDFLARE_PURGE_TOKEN=++# host=zone id pairs; a URL uses the zone of its host or nearest parent domain.++#CLOUDFLARE_ZONE_IDS=openvibe.wiki=<zone id>,openvibe.blog=<zone id>,openvibe.network=<zone id>++# Paths on the removed URL's host purged with it (default /sitemap.xml; `none` for none).++#CLOUDFLARE_PURGE_RELATED_PATHS=/sitemap.xml,/feed.atom,/feed.json++#CLOUDFLARE_PURGE_INTERVAL_MS=5000@@@ -6,8 +6,9 @@**Status:** alpha (roadmap Wave 14). Deployed internally, not launched: it runs on the production host(127.0.0.1:4710 only, since 2026-09-23) and is fed through Events, but the index holds only 10 wikitombstones and 0 public documents.--**Domain:** `search.openvibe.network` (no vhost installed yet: the name currently falls through to the--admin.openvibe.network placeholder; it goes public only when the launch rule below holds)++**Domain:** `search.openvibe.network` (vhost in [deploy/nginx/](deploy/nginx/search.openvibe.network.conf),++not installed yet: until it is, the name falls through to the admin.openvibe.network placeholder.++Install steps: [Public host](#public-host))**Plan:** OpenVibe End-to-End Realignment & Implementation Plan, revision 3 — roadmap §4.2 A, §15.12, §29, §32.3.**License:** AGPL-3.0.@@@ -31,7 +32,8 @@ npm test # every test/*.test.js, temp databases, no network neededNode 22 in production (`fnm exec --using=22.22.1 npm test`). Production: `/opt/openvibe.search`,env `/etc/openvibe/search.env`, unit [deploy/systemd/openvibe-search.service](deploy/systemd/openvibe-search.service),index `/var/lib/openvibe-search/search.db`, nginx [deploy/nginx/search.openvibe.network.conf](deploy/nginx/search.openvibe.network.conf)--(the public vhost exposes only the GET query API and health; owners and Events call `127.0.0.1:4710`).++(the public vhost exposes the search page, the GET query API, saved searches and health; the owner++API and the Events webhook are loopback-only, and owners and Events call `127.0.0.1:4710`).`GET /api/health` is liveness. `GET /api/ready` (openvibe-shared/ready) is 503 only when thedatabase (the documents and both full-text tables) fails; a Network signing key that has not loaded@@@ -71,6 +73,50 @@ anonymous); delegation headers from a browser are ignored. Visibility is enforceSQL (two FTS tables split by audience plus an ACL predicate) and re-checked per hit — see[docs/adr-engine.md](docs/adr-engine.md).++The search page at `/` runs the same query as HTML for browsers (no JavaScript; no-store; result++pages `noindex`; anonymous unless this host has an `ov_token` cookie). Other clients get a text++route index at `/`.++++### Ranking++++Full-text results are ordered by bm25 relevance (title ×10, summary ×4, body ×1) multiplied by a++freshness boost:++++```++score = bm25 × (1 + W × 2^(−age_days / H)) W = SEARCH_FRESHNESS_WEIGHT (default 1)++ H = SEARCH_FRESHNESS_HALF_LIFE_DAYS (default 30)++```++++`age_days` counts from `published_at`, else `updated_at` (an edit does not make an old document++new); a future date counts as today and a missing one gets no boost. With the defaults a document++published today counts ×2.0, 30 days old ×1.5, 60 days ×1.25, 90 days ×1.125 and a year old ×1.0002:++relevance still decides, and between comparably relevant documents the newer one ranks first.++`W = 0` turns it off. Suggestions use the same score; browsing without `q` stays newest first. A++cursor carries its first page's clock, so later pages rank on the same scores++(`test/freshness.test.js`).++++### Saved searches++++```http++GET /api/v1/saved-searches++POST /api/v1/saved-searches { "name": "Eclipses", "q": "solar eclipse", "owner": "news", "facets": { "topic": ["space"] } }++GET /api/v1/saved-searches/:id++DELETE /api/v1/saved-searches/:id++GET /api/v1/saved-searches/:id/results?limit=&cursor=++```++++A signed-in person (a Network user token with a `usr_` subject, as Bearer or the `ov_token`++cookie) or a first-party service with `search.query.delegate` acting for `X-OV-Subject: usr_…`++keeps up to `SEARCH_SAVED_MAX_PER_SUBJECT` (50) queries: words, filters or both. Saving the same++query again returns the existing one (renamed if a new name is given). Only the query is stored:++every run is a fresh query as that person at that moment, so what they lost access to (or what was++deleted) is gone and what they gained appears. Someone else's saved search is the same 404 as a++missing one; guests cannot save; a cookie-authenticated POST/DELETE must carry this origin's++`Origin`. Notifications of new matches wait for OpenVibe.Network's notifications (`last_run_at` is++kept for that job). Signing in on `search.openvibe.network` itself needs a Network OAuth client++(`search`), which does not exist yet: today saved searches are used through Bearer tokens or a++product acting for its visitor.++## IndexingOwners call with an OpenVibe.Network client-credentials token (`audience=openvibe.search`) holding@@@ -105,17 +151,40 @@ private/members/unlisted, or public → noindex) leaves results in the same tranpurge); every accepted upsert emits `search.document.indexed`. Both go through a transactionaloutbox relayed to Events when `EVENTS_URL` is set.++### Removals, caches and the CDN++++Search consumes its own removals ([server/purge.js](server/purge.js)); in the same transaction as++the `search.document.removed` event it records:++++- **the owner's removal feed** — `GET /api/v1/owners/:owner/removals?after=&limit=` (owner token,++ own owner only): `seq, event_id, type, id, revision, reason, previous_exposure, exposure,++ canonical_url, at`. Products own their pages, caches and sitemaps, so they poll this feed (or++ subscribe to `search.document.removed`) and drop the document from anything derived they keep.++- **a Cloudflare purge queue** — only when `CLOUDFLARE_PURGE_TOKEN` is set (inert otherwise:++ nothing is queued or sent). For a removal whose URL had been public, the canonical URL and the++ `CLOUDFLARE_PURGE_RELATED_PATHS` on its host (default `/sitemap.xml`) are queued once each and++ purged by URL (`POST /zones/:zone/purge_cache {"files": […]}`, at most 30 per call) in the zone++ `CLOUDFLARE_ZONE_IDS` maps the host (or its nearest parent domain) to. A host with no zone is++ recorded as `skipped`. 429, 5xx and network errors back off and retry (8 attempts); a refusal is++ retried one URL at a time and then marked `failed`. The token needs only *Zone → Cache Purge →++ Purge* on those zones; it is read from the environment, sent only in the Authorization header,++ and never stored or logged. `/api/ready` reports the purge state and counts, `/metrics` the++ `search_cdn_purges{state}` gauge (`test/purge.test.js`, against a mocked Cloudflare API).++## Owns- the index document contract (`search.index-document@1`, released in openvibe-contracts v0.12.0; proposal in [docs/contracts-proposal/](docs/contracts-proposal/))- `documents`, `doc_acl`, `doc_facets`, `fts_public`, `fts_restricted`, `idempotency_receipts`,-- `ingest_rejections`, `event_outbox` (SQLite; engine decision in [docs/adr-engine.md](docs/adr-engine.md))--- the query API and its ACL filtering; deletion and visibility-change propagation++ `ingest_rejections`, `event_outbox`, `removals`, `cdn_purges`, `saved_searches` (SQLite; engine++ decision in [docs/adr-engine.md](docs/adr-engine.md))++- the query API, its ranking and ACL filtering, the search page and saved searches; deletion and++ visibility-change propagation, including the CDN purge of removed public URLs## Does not own- any product's content, publication state or indexability decision (owners and the publishing packages do)--- sitemaps, feeds or page caches (each product; they listen to `search.document.removed`)++- sitemaps, feeds or page caches (each product; they read the removal feed or listen to++ `search.document.removed`). Search only purges the CDN copy of a removed public URL.- identity, groups or entitlements (Network, products, VIP/Billing — Search only matches the keys it is given)## Depends on@@@ -128,8 +197,8 @@ outbox relayed to Events when `EVENTS_URL` is set.| Capability | Routes ||---|---|--| `search.document.write` | `PUT/DELETE /api/v1/documents/:owner/:type/:id`, `/api/v1/owners/:owner/...` (own owner only) |--| `search.query.delegate` | the query routes, acting for `X-OV-Subject` |++| `search.document.write` | `PUT/DELETE /api/v1/documents/:owner/:type/:id`, `/api/v1/owners/:owner/...` including the removal feed (own owner only) |++| `search.query.delegate` | the query and saved-search routes, acting for `X-OV-Subject` |Released in `openvibe-contracts` v0.12.0 with the service manifest (this repo pins v0.13.0);[server/auth.js](server/auth.js) decides them with the contracts grant rule, and CI runs@@@ -146,12 +215,20 @@ Released in `openvibe-contracts` v0.12.0 with the service manifest (this repo piowner spoofing refused (`test/webhook-inbox.test.js`)- outbox relay to Events, retry and poison isolation (`test/outbox.test.js`)- proposals valid against the contracts schemas (`test/proposals.test.js`)++- removals reach the owner feed; the CDN purge is inert without a token and, with one, purges the++ formerly public URL and its sitemap per zone, retries, isolates refusals and never stores the++ token (`test/purge.test.js`)++- freshness decay, newer-first between equals, relevance over age, stable cursors (`test/freshness.test.js`)++- saved searches: signed-in only, own only, run-time ACL, Origin check for cookies, limits++ (`test/saved-searches.test.js`)++- the search page: public only, escaped, no-store, noindex results, paging (`test/page.test.js`)Not yet demonstrated: a document with content indexed end to end in production. The Eventssubscription delivered 20 wiki events; the 10 upserts were rejected (`owner_not_accepted`, sincefixed) and only the 10 tombstones landed. Wiki pages enter the index once a person reviews them;--Sources has every seed disabled. Nobody consumes `search.document.removed` yet (no cache, sitemap--or CDN purge).++Sources has every seed disabled. The removal feed and the CDN purge queue exist, but no product++polls the feed yet and the purge needs a token with Cache Purge permission (the current Cloudflare++token has DNS and rulesets scope only).Restore drill: `ovhost drill search` passed on the production host on 2026-09-23 (integrity check,readiness, identical query answers, row counts; see OpenVibe.Host `docs/restore-drills.md`).@@@ -161,10 +238,37 @@ readiness, identical query answers, row counts; see OpenVibe.Host `docs/restore-This repository does not make the product real, and the domain keeps its placeholder page on[OpenVibers/OpenVibe.Sites](https://github.com/OpenVibers/OpenVibe.Sites) until: an owning runtimewith health/readiness; Network identity and service principals; a public route useful without--JavaScript (there is no HTML search page yet — only the JSON API and a text index at `/`); real++JavaScript (the search page at `/`); realpersistence and end-to-end workflows; capabilities and events registered in OpenVibe.Contracts; a--security review; and acceptance tests. The launch release removes the domain from--`OpenVibe.Sites/sites.json` atomically.++security review; and acceptance tests. The domain is not in `OpenVibe.Sites/sites.json`; this++repository's own vhost serves it.++++## Public host++++`search.openvibe.network` resolves through Cloudflare, but until this vhost is installed nginx++answers it with its default site (the admin placeholder). The vhost++([deploy/nginx/search.openvibe.network.conf](deploy/nginx/search.openvibe.network.conf)) uses the++Network wildcard certificate, sets the client-address headers from `$remote_addr` only, answers++`/metrics` with 404, keeps `/internal/`, `/api/v1/owners/` and document writes loopback-only, and++answers every unknown path with a JSON 404. Install on the host, after the service runs a release++that has the search page and saved searches (`ovhost deploy search`):++++```bash++sudo install -m 0644 /opt/openvibe.search/deploy/nginx/search.openvibe.network.conf /etc/nginx/sites-available/search.openvibe.network.conf++sudo ln -sf /etc/nginx/sites-available/search.openvibe.network.conf /etc/nginx/sites-enabled/search.openvibe.network.conf++sudo nginx -t && sudo systemctl reload nginx++# check++curl -sS https://search.openvibe.network/api/health++curl -sS -H 'Accept: text/html' https://search.openvibe.network/ | grep -o '<title>[^<]*'++curl -sS -o /dev/null -w '%{http_code}\n' https://search.openvibe.network/metrics # 404++curl -sS -o /dev/null -w '%{http_code}\n' -X POST https://search.openvibe.network/internal/events # 403++curl -sS -o /dev/null -w '%{http_code}\n' https://search.openvibe.network/api/v1/owners/wiki/removals # 403++```++++For the CDN purge, create a Cloudflare API token with only *Zone → Cache Purge → Purge* on the++product zones, then add `CLOUDFLARE_PURGE_TOKEN` and `CLOUDFLARE_ZONE_IDS` to++`/etc/openvibe/search.env` (0600) and restart `openvibe-search`; `/api/ready` then shows++`purge.cdn: "cloudflare"`.---@@@ -6,7 +6,7 @@"plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026)","runtime": "node 22, express, better-sqlite3 (FTS5)","port": 4710,-- "domain": "search.openvibe.network (no vhost installed: the name falls through to the admin.openvibe.network placeholder; the service is reachable on the host only)",++ "domain": "search.openvibe.network (vhost in deploy/nginx, not installed: until it is, the name falls through to the admin.openvibe.network placeholder; README \"Public host\")","contracts": "openvibe-contracts v0.13.0","capabilities": ["search.document.write", "search.query.delegate"],"capabilitiesRegistered": true,@@@ -15,5 +15,6 @@"deployed": true,"deployment": "Deployed internally, not launched: /opt/openvibe.search at 5a7e59b, unit openvibe-search, 127.0.0.1:4710 only, principal search, Events subscription *.index_document.*, accepted event owners sources, wiki, blog, news, reviews, trade, deals, coupons. Production index: 10 documents, all tombstones (wiki), 0 public. Restore drill passed 2026-09-23.","indexedProducers": ["wiki (tombstones only so far)"],++ "features": ["ACL-filtered FTS5 query API", "facets", "suggestions", "freshness-weighted ranking", "saved searches (notifications wait for Network notifications)", "server-rendered search page", "owner removal feed", "Cloudflare purge queue (inert until CLOUDFLARE_PURGE_TOKEN is set)"],"updated": "2026-09-23"}@@@ -2,22 +2,26 @@## search.openvibe.network — Nginx Configuration (REFERENCE)#### Deployed to /etc/nginx/sites-available/search.openvibe.network.conf — keep them in sync,--## and reload nginx after changing it.++## and reload nginx after changing it. Install: README "Public host".#### The Node service (port 4710, systemd unit openvibe-search) serves:++## / the search page (HTML for browsers, a text route index++## otherwise) and /robots.txt. Public, no-store, result pages noindex.## /api/v1/search, /api/v1/suggest, GET /api/v1/documents/... public query API (anonymous =## public documents only; the ov_token cookie or a Bearer adds## the caller's ACL matches). Responses are Cache-Control: no-store.--## /api/v1/documents (PUT/DELETE), /api/v1/owners/... owner API (service tokens). Services on--## this host call 127.0.0.1:4710, so the public vhost refuses--## writes; open them only for an off-host owner.--## /internal/events OpenVibe.Events deliveries — host-local only.--## /api/health, /api/ready++## /api/v1/saved-searches[/...] a signed-in person's saved searches (Network user token)++## /api/v1/documents (PUT/DELETE), /api/v1/owners/... owner API (service tokens): loopback only.++## Services on this host call 127.0.0.1:4710 directly.++## /internal/events OpenVibe.Events deliveries: loopback only.++## /api/health, /api/ready, /release.json## /metrics Prometheus text, answered only to direct loopback callers## (never proxied; Node refuses a request carrying X-Forwarded-For)++## Everything else is a JSON 404 from nginx, so the name never falls through to another site.## ═══════════════════════════════════════════════════════════════limit_req_zone $binary_remote_addr zone=ovsearch_query:10m rate=10r/s;++limit_req_zone $binary_remote_addr zone=ovsearch_saved:10m rate=2r/s;# ── HTTP → HTTPS redirect ──────────────────────────────────server {@@@ -50,42 +54,82 @@ server {add_header Referrer-Policy strict-origin-when-cross-origin always;add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;-- location = /api/health { proxy_pass http://127.0.0.1:4710; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $remote_addr; proxy_set_header CF-Connecting-IP $remote_addr; }-- location = /api/ready { proxy_pass http://127.0.0.1:4710; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $remote_addr; proxy_set_header CF-Connecting-IP $remote_addr; }++ # Client address only from $remote_addr (realip has already replaced it with the visitor's++ # address for Cloudflare's ranges); never append to or pass through a client-sent header.++ # Inherited by every location below that proxies (none sets its own).++ proxy_http_version 1.1;++ proxy_set_header Connection "";++ proxy_set_header Host $host;++ proxy_set_header X-Real-IP $remote_addr;++ proxy_set_header X-Forwarded-For $remote_addr;++ proxy_set_header CF-Connecting-IP $remote_addr;++ proxy_set_header X-Forwarded-Proto $scheme;++++ location = /api/health { proxy_pass http://127.0.0.1:4710; }++ location = /api/ready { proxy_pass http://127.0.0.1:4710; }++ location = /release.json { proxy_pass http://127.0.0.1:4710; }# Metrics are for host-local scrapers only (http://127.0.0.1:4710/metrics).location = /metrics { return 404; }++ # ── Search page ─────────────────────────────────────────++ location = / {++ limit_except GET { deny all; }++ limit_req zone=ovsearch_query burst=40 nodelay;++ proxy_pass http://127.0.0.1:4710;++ }++ location = /robots.txt { proxy_pass http://127.0.0.1:4710; }++# ── Public query API (GET only) ─────────────────────────-- location ~ ^/api/v1/(search|suggest|documents/) {++ location ~ ^/api/v1/(search|suggest)$ {limit_except GET { deny all; }limit_req zone=ovsearch_query burst=40 nodelay;proxy_pass http://127.0.0.1:4710;-- proxy_http_version 1.1;-- proxy_set_header Connection "";-- proxy_set_header Host $host;-- proxy_set_header X-Real-IP $remote_addr;-- proxy_set_header X-Forwarded-For $remote_addr;-- proxy_set_header CF-Connecting-IP $remote_addr;-- proxy_set_header X-Forwarded-Proto $scheme;# Never let a shared cache hold a personalised answer.proxy_no_cache 1;proxy_cache_bypass 1;}-- # ── Owner API and the Events webhook: host-local only ───-- location /api/ { return 403; }-- location /internal/ { return 403; }++ # One document by id: GET is public (ACL-checked by Search); PUT/DELETE are owner writes,++ # loopback only.++ location ^~ /api/v1/documents/ {++ limit_except GET {++ allow 127.0.0.1;++ allow ::1;++ deny all;++ }++ limit_req zone=ovsearch_query burst=40 nodelay;++ proxy_pass http://127.0.0.1:4710;++ proxy_no_cache 1;++ proxy_cache_bypass 1;++ }++++ # ── Saved searches (a Network user token; Search checks Origin for cookie changes) ──++ location ~ ^/api/v1/saved-searches(/|$) {++ limit_except GET POST DELETE { deny all; }++ limit_req zone=ovsearch_saved burst=20 nodelay;++ proxy_pass http://127.0.0.1:4710;++ proxy_no_cache 1;++ proxy_cache_bypass 1;++ }-- location = / {++ # ── Loopback only: owner API and the Events webhook ─────++ location ^~ /api/v1/owners/ {++ allow 127.0.0.1;++ allow ::1;++ deny all;++ proxy_pass http://127.0.0.1:4710;++ }++ location ^~ /internal/ {++ allow 127.0.0.1;++ allow ::1;++ deny all;proxy_pass http://127.0.0.1:4710;-- proxy_set_header Host $host;-- proxy_set_header X-Real-IP $remote_addr;-- proxy_set_header X-Forwarded-For $remote_addr;-- proxy_set_header CF-Connecting-IP $remote_addr;}++ # ── Anything else: an honest JSON 404, never another site ──location / {-- return 404;++ default_type application/problem+json;++ return 404 '{"type":"about:blank","title":"Not Found","status":404,"code":"search.not_found","detail":"No such route on search.openvibe.network. The search page is at /, the public query API at /api/v1/search; see https://github.com/OpenVibers/OpenVibe.Search"}';}}@@@ -9,6 +9,9 @@* reconciliation: (type, id, revision, deleted, hash)* GET /api/v1/owners/:owner/documents/:type/:id the stored document + effective indexability* GET /api/v1/owners/:owner/rejections?after= index-document events Search refused++ * GET /api/v1/owners/:owner/removals?after=&limit= documents that left results or public++ * delivery (server/purge.js): the owner drops++ * them from its caches and sitemaps*/const express = require('express');const { http } = require('openvibe-contracts');@@@ -39,7 +42,7 @@ function outcomeResponse(res, ctx, r) {return res.json(r);}--function documentsRouter({ store, auth, db, relay }) {++function documentsRouter({ store, auth, db, relay, purges }) {const router = express.Router();const guard = auth.requireCap(CAPS.write);@@@ -125,6 +128,13 @@ function documentsRouter({ store, auth, db, relay }) {});});++ router.get('/api/v1/owners/:owner/removals', guard, (req, res) => {++ if (!ownerCheck(req, res)) return;++ const after = /^\d{1,15}$/.test(String(req.query.after || '')) ? Number(req.query.after) : 0;++ const limit = Math.min(Math.max(parseInt(req.query.limit, 10) || 200, 1), 1000);++ res.json({ owner: req.params.owner, ...purges.ownerFeed(req.params.owner, { after, limit }) });++ });++return router;}@@@ -69,7 +69,8 @@ function decodeCursor(s, hash) {let c;try { c = JSON.parse(Buffer.from(String(s), 'base64url').toString('utf8')); } catch { c = null; }if (!c || c.h !== hash || !Number.isInteger(c.i)) throw new QueryError('search.bad_cursor', 'cursor does not belong to this query');-- if (c.k === 'r' && typeof c.r === 'number') return { rank: c.r, rid: c.i };++ // n: the first page's reference time for freshness, so later pages rank on the same clock.++ if (c.k === 'r' && typeof c.r === 'number') return { rank: c.r, rid: c.i, now: Number.isFinite(c.n) ? c.n : null };if (c.k === 't' && typeof c.t === 'string') return { sort_at: c.t, rid: c.i };throw new QueryError('search.bad_cursor', 'cursor is malformed');}@@@ -111,51 +112,31 @@ function canView(viewer, row, doc) {return false;}--function queryRouter({ config, store, engine, auth }) {-- const router = express.Router();---- function withViewer(handler) {-- return (req, res) => {-- const ctx = req.ov;-- let viewer;-- try {-- viewer = auth.viewer(req);-- } catch (err) {-- if (err instanceof AuthError) return http.sendProblem(res, err.status, err.code, { detail: err.message, ctx });-- throw err;-- }-- // Never cached: a visibility change or deletion must leave results immediately, and a-- // personalised answer must never reach a shared cache.-- res.setHeader('Cache-Control', 'no-store');-- res.setHeader('Vary', 'Authorization, Cookie, X-OV-Subject, X-OV-Groups, X-OV-Entitlements');-- try {-- return handler(req, res, viewer);-- } catch (err) {-- if (err instanceof QueryError) return http.sendProblem(res, 400, err.code, { detail: err.message, ctx });-- throw err;-- }-- };-- }---- router.get('/api/v1/search', withViewer((req, res, viewer) => {-- const q = one(req.query.q);-- const text = q === undefined ? '' : String(q);++/**++ * One search as a viewer: the query API, saved-search runs and the HTML page all go through here.++ * run({ text, query, viewer, limit, cursor, facetKeys, now }) → { results, next_cursor, facets? }++ * `query` is the query-string-shaped filter source (owner, type, lang, facet.<key>). Throws++ * QueryError for a malformed request.++ */++function createSearcher({ config, store, engine, now: clock = () => Date.now() }) {++ function run({ text = '', query = {}, filters = null, viewer, limit, cursor, facetKeys = [], now = clock() }) {if (text.length > 500) throw new QueryError('search.bad_query', 'q is longer than 500 characters');-- const filters = parseFilters(req.query, config);-- const limit = Math.min(Math.max(parseInt(one(req.query.limit), 10) || config.query.defaultLimit, 1), config.query.maxLimit);-- const facetKeys = String(one(req.query.facets) || '').split(',').map(s => s.trim()).filter(Boolean);++ filters = filters || parseFilters(query, config);++ limit = Math.min(Math.max(parseInt(limit, 10) || config.query.defaultLimit, 1), config.query.maxLimit);if (facetKeys.length > 5 || facetKeys.some(k => !FACET_KEY_RE.test(k))) throw new QueryError('search.bad_query', 'facets: at most 5 well-formed keys');const match = text.trim() ? toMatch(text, { maxTerms: config.query.maxTerms }) : null;const hash = queryHash([match, filters]);-- const after = decodeCursor(one(req.query.cursor), hash);++ const after = decodeCursor(cursor, hash);if (text.trim() && !match) {-- return res.json({ results: [], next_cursor: null, ...(facetKeys.length ? { facets: Object.fromEntries(facetKeys.map(k => [k, []])) } : {}) });++ return { results: [], next_cursor: null, ...(facetKeys.length ? { facets: Object.fromEntries(facetKeys.map(k => [k, []])) } : {}) };}let page;++ let refNow = now;if (match) {if (after && after.rank === undefined) throw new QueryError('search.bad_cursor', 'cursor does not belong to this query');-- page = engine.search({ match, filters, viewer, limit, after });++ if (after && after.now != null) refNow = after.now;++ page = engine.search({ match, filters, viewer, limit, after, now: refNow });} else {if (after && after.sort_at === undefined) throw new QueryError('search.bad_cursor', 'cursor does not belong to this query');page = engine.browse({ filters, viewer, limit, after });@@@ -168,14 +149,61 @@ function queryRouter({ config, store, engine, auth }) {results.push(projection(found.doc, found.row, match ? h.snip : undefined));}const next = page.next-- ? encodeCursor(match ? { k: 'r', r: page.next.rank, i: page.next.rid, h: hash } : { k: 't', t: page.next.sort_at, i: page.next.rid, h: hash })++ ? encodeCursor(match ? { k: 'r', r: page.next.rank, i: page.next.rid, n: refNow, h: hash } : { k: 't', t: page.next.sort_at, i: page.next.rid, h: hash }): null;const body = { results, next_cursor: next };if (facetKeys.length) body.facets = engine.facetCounts({ match, filters, viewer, keys: facetKeys });-- res.json(body);++ return body;++ }++++ return {++ run,++ parseFilters: (query) => parseFilters(query, config),++ hasWords: (text) => toMatch(text, { maxTerms: config.query.maxTerms }) !== null,++ };++}++++/** Wraps a handler: resolves the viewer (AuthError → problem), no-store, QueryError → 400. */++function withViewer(auth, handler) {++ return (req, res, next) => {++ const ctx = req.ov;++ let viewer;++ try {++ viewer = auth.viewer(req);++ } catch (err) {++ if (err instanceof AuthError) return http.sendProblem(res, err.status, err.code, { detail: err.message, ctx });++ return next(err);++ }++ // Never cached: a visibility change or deletion must leave results immediately, and a++ // personalised answer must never reach a shared cache.++ res.setHeader('Cache-Control', 'no-store');++ res.setHeader('Vary', 'Authorization, Cookie, X-OV-Subject, X-OV-Groups, X-OV-Entitlements');++ try {++ return handler(req, res, viewer);++ } catch (err) {++ if (err instanceof QueryError) return http.sendProblem(res, 400, err.code, { detail: err.message, ctx });++ return next(err);++ }++ };++}++++function queryRouter({ config, store, engine, auth, searcher = createSearcher({ config, store, engine }) }) {++ const router = express.Router();++ const guarded = (h) => withViewer(auth, h);++++ router.get('/api/v1/search', guarded((req, res, viewer) => {++ const q = one(req.query.q);++ res.json(searcher.run({++ text: q === undefined ? '' : String(q),++ query: req.query,++ viewer,++ limit: one(req.query.limit),++ cursor: one(req.query.cursor),++ facetKeys: String(one(req.query.facets) || '').split(',').map(s => s.trim()).filter(Boolean),++ }));}));-- router.get('/api/v1/suggest', withViewer((req, res, viewer) => {++ router.get('/api/v1/suggest', guarded((req, res, viewer) => {const text = String(one(req.query.q) || '');if (text.length > 100) throw new QueryError('search.bad_query', 'q is longer than 100 characters');const filters = parseFilters(req.query, config);@@@ -192,7 +220,7 @@ function queryRouter({ config, store, engine, auth }) {res.json({ suggestions });}));-- router.get('/api/v1/documents/:owner/:type/:id', withViewer((req, res, viewer) => {++ router.get('/api/v1/documents/:owner/:type/:id', guarded((req, res, viewer) => {const { owner, type, id } = req.params;if (!OWNER_RE.test(owner) || !TYPE_RE.test(type) || !ID_RE.test(id)) throw new QueryError('search.bad_identity', 'owner, type or id is malformed');const found = store.get(owner, type, id);@@@ -205,4 +233,4 @@ function queryRouter({ config, store, engine, auth }) {return router;}--module.exports = { queryRouter, canView, projection, toMatch };++module.exports = { queryRouter, createSearcher, withViewer, parseFilters, QueryError, canView, projection, toMatch, one };nnew file mode 100644@@@ -0,0 +1,128 @@++'use strict';++/**++ * Saved searches of the signed-in person (server/saved.js).++ *++ * GET /api/v1/saved-searches the caller's saved searches, newest first++ * POST /api/v1/saved-searches { name?, q?, owner?, type?, lang?, facets?: { key: [values] } }++ * 201 new, 200 when the same query was already saved++ * GET /api/v1/saved-searches/:id one++ * DELETE /api/v1/saved-searches/:id 204++ * GET /api/v1/saved-searches/:id/results?limit=&cursor=++ * runs it now, as the caller (the ACL applies at++ * run time, never at save time)++ *++ * Who: a Network user JWT carrying a usr_ subject_id (Bearer, or the ov_token cookie), or a++ * first-party service with search.query.delegate acting for X-OV-Subject: usr_… (a product saving++ * for its signed-in visitor). Guests and anonymous callers cannot save. A saved search of another++ * person and a missing one give the same 404. A cookie-authenticated POST/DELETE must come from++ * this origin (Origin header), so another site cannot save or delete in someone's name.++ */++const express = require('express');++const { http, ids } = require('openvibe-contracts');++const { withViewer, QueryError, one } = require('./query');++++const NAME_MAX = 100;++++function savedRouter({ config, saved, searcher, auth }) {++ const router = express.Router();++ const selfOrigin = (() => { try { return new URL(config.baseUrl).origin; } catch { return null; } })();++++ /** The viewer's usr_ subject, or a problem sent (null). */++ function person(req, res, viewer, { write = false } = {}) {++ const ctx = req.ov;++ if (!viewer.subject) {++ http.sendProblem(res, 401, 'token.missing', { detail: 'sign in to use saved searches', ctx });++ return null;++ }++ if (!ids.isSubjectId('user', viewer.subject)) {++ http.sendProblem(res, 403, 'search.sign_in_required', { detail: 'saved searches belong to signed-in people (usr_ subjects), not guests', ctx });++ return null;++ }++ if (write && viewer.via === 'cookie') {++ const origin = req.get('origin');++ if (!origin || origin !== selfOrigin) {++ http.sendProblem(res, 403, 'search.cross_origin', { detail: 'a cookie-authenticated change must come from this origin', ctx });++ return null;++ }++ }++ return viewer.subject;++ }++++ const guarded = (h) => withViewer(auth, h);++++ router.get('/api/v1/saved-searches', guarded((req, res, viewer) => {++ const subject = person(req, res, viewer);++ if (!subject) return;++ res.json({ saved_searches: saved.list(subject), max: saved.maxPerSubject });++ }));++++ router.post('/api/v1/saved-searches', guarded((req, res, viewer) => {++ const ctx = req.ov;++ const subject = person(req, res, viewer, { write: true });++ if (!subject) return;++ const b = req.body && typeof req.body === 'object' && !Array.isArray(req.body) ? req.body : null;++ if (!b) return http.sendProblem(res, 400, 'search.bad_saved_search', { detail: 'body must be a JSON object', ctx });++ const q = b.q == null ? '' : b.q;++ const name = b.name == null ? '' : b.name;++ if (typeof q !== 'string' || q.length > 500) throw new QueryError('search.bad_query', 'q must be a string of at most 500 characters');++ if (typeof name !== 'string' || name.trim().length > NAME_MAX) throw new QueryError('search.bad_saved_search', `name must be a string of at most ${NAME_MAX} characters`);++ // The filters take the same shape and checks as the query string of /api/v1/search.++ const query = {};++ for (const k of ['owner', 'type', 'lang']) {++ if (b[k] == null) continue;++ if (typeof b[k] !== 'string') throw new QueryError('search.bad_query', `${k} must be a string`);++ query[k] = b[k];++ }++ if (b.facets != null) {++ if (typeof b.facets !== 'object' || Array.isArray(b.facets)) throw new QueryError('search.bad_query', 'facets must be an object of key → values');++ for (const [k, v] of Object.entries(b.facets)) {++ const values = Array.isArray(v) ? v : [v];++ if (values.some(x => typeof x !== 'string')) throw new QueryError('search.bad_query', `facets.${k} values must be strings`);++ query[`facet.${k}`] = values;++ }++ }++ const filters = searcher.parseFilters(query);++ const hasFilter = filters.owner || filters.type || filters.language || filters.facets.length;++ const text = q.trim();++ if (text && !searcher.hasWords(text)) throw new QueryError('search.bad_query', 'q has no searchable words');++ if (!text && !hasFilter) throw new QueryError('search.bad_saved_search', 'a saved search needs words (q) or at least one filter');++ const r = saved.save(subject, { name: name.trim(), q: text, filters });++ if (r.error === 'limit') return http.sendProblem(res, 409, 'search.saved_search_limit', { detail: `at most ${saved.maxPerSubject} saved searches per person`, ctx });++ res.status(r.created ? 201 : 200).json({ saved_search: r.saved });++ }));++++ router.get('/api/v1/saved-searches/:id', guarded((req, res, viewer) => {++ const subject = person(req, res, viewer);++ if (!subject) return;++ const s = saved.get(subject, req.params.id);++ if (!s) return http.sendProblem(res, 404, 'search.not_found', { detail: 'no such saved search', ctx: req.ov });++ res.json({ saved_search: s });++ }));++++ router.delete('/api/v1/saved-searches/:id', guarded((req, res, viewer) => {++ const subject = person(req, res, viewer, { write: true });++ if (!subject) return;++ if (!saved.remove(subject, req.params.id)) return http.sendProblem(res, 404, 'search.not_found', { detail: 'no such saved search', ctx: req.ov });++ res.status(204).end();++ }));++++ router.get('/api/v1/saved-searches/:id/results', guarded((req, res, viewer) => {++ const subject = person(req, res, viewer);++ if (!subject) return;++ const s = saved.get(subject, req.params.id);++ if (!s) return http.sendProblem(res, 404, 'search.not_found', { detail: 'no such saved search', ctx: req.ov });++ const body = searcher.run({++ text: s.q,++ filters: { ...s.filters, facets: s.filters.facets || [] },++ viewer,++ limit: one(req.query.limit),++ cursor: one(req.query.cursor),++ });++ saved.markRun(s.id);++ res.json({ saved_search: { id: s.id, name: s.name }, ...body });++ }));++++ return router;++}++++module.exports = { savedRouter };@@@ -7,11 +7,13 @@ const { instrument } = require('openvibe-shared/metrics');const { createRelease } = require('openvibe-shared/release');const { createSearchReadiness, registerSearchGauges } = require('./observability');const { documentsRouter } = require('./api/documents');--const { queryRouter } = require('./api/query');++const { queryRouter, createSearcher } = require('./api/query');++const { savedRouter } = require('./api/saved');++const { pageRouter } = require('./web/page');const { webhookRouter } = require('./api/webhook');const pkg = require('../package.json');--function createApp({ config, db, store, engine, auth, keys, outbox, relay, log = console, now }) {++function createApp({ config, db, store, engine, auth, keys, outbox, relay, purges, purger, saved, log = console, now }) {const app = express();app.disable('x-powered-by');app.set('trust proxy', 'loopback');@@@ -20,7 +22,7 @@ function createApp({ config, db, store, engine, auth, keys, outbox, relay, log =// HTTP golden signals by route template, process metrics, release_info and the Search gauges;// GET /metrics answers direct loopback callers only (Track O).const metrics = instrument(app, { service: 'search', release: release.release });-- registerSearchGauges(metrics.registry, { db, outbox });++ registerSearchGauges(metrics.registry, { db, outbox, purges });app.use(http.middleware());app.use((req, res, next) => {res.setHeader('X-Content-Type-Options', 'nosniff');@@@ -38,30 +40,16 @@ function createApp({ config, db, store, engine, auth, keys, outbox, relay, log =// Readiness (openvibe-shared/ready): 503 only when the database (documents and full-text tables)// fails; the Network key and index consistency are optional and degrade it (see observability.js).-- const readiness = createSearchReadiness({ db, keys, config, engine, store, outbox, relay, release: release.release });++ const readiness = createSearchReadiness({ db, keys, config, engine, store, outbox, relay, purges, purger, saved, release: release.release });app.get('/api/ready', readiness.handler);app.get('/release.json', release.handler);-- app.use(documentsRouter({ store, auth, db, relay }));-- app.use(queryRouter({ config, store, engine, auth }));---- app.get('/', (_req, res) => {-- res.type('text/plain').send([-- 'OpenVibe.Search: permission-aware search over documents the network\'s services index.',-- '',-- 'GET /api/v1/search?q=&owner=&type=&facet.<key>=&cursor= query (anonymous: public only)',-- 'GET /api/v1/suggest?q= title suggestions',-- 'GET /api/v1/documents/:owner/:type/:id one document the caller may see',-- 'PUT /api/v1/documents/:owner/:type/:id index (owner, search.document.write)',-- 'DELETE /api/v1/documents/:owner/:type/:id?revision= tombstone (owner)',-- 'GET /api/v1/owners/:owner/documents reconciliation (owner)',-- 'POST /internal/events OpenVibe.Events delivery (signed)',-- 'GET /api/health, /api/ready, /release.json',-- '',-- 'Source: https://github.com/OpenVibers/OpenVibe.Search',-- '',-- ].join('\n'));-- });++ const searcher = createSearcher({ config, store, engine, now });++ app.use(documentsRouter({ store, auth, db, relay, purges }));++ app.use(queryRouter({ config, store, engine, auth, searcher }));++ app.use(savedRouter({ config, saved, searcher, auth }));++ // GET / (HTML search page for browsers, the text route index otherwise) and /robots.txt.++ app.use(pageRouter({ searcher, auth }));app.use((req, res) => http.sendProblem(res, 404, 'search.not_found', { detail: `no route ${req.method} ${req.path}`, ctx: req.ov }));@@@ -249,7 +249,10 @@ function createAuth({ config, keys }) {}const fromCookie = cookie(req, 'ov_token');const user = fromCookie ? verifyUser(fromCookie) : null;-- return user ? userViewer(user) : ANONYMOUS;++ if (!user) return ANONYMOUS;++ const v = userViewer(user);++ // `via` lets state-changing routes demand a same-origin request for ambient credentials.++ return v.kind === 'user' ? { ...v, via: 'cookie' } : v;}return { verifyService, verifyUser, requireCap, viewer };@@@ -12,6 +12,30 @@ const int = (v, d) => {};const list = (v, d) => (v == null || v === '' ? d : String(v).split(',').map(s => s.trim()).filter(Boolean));const strip = (v) => String(v || '').replace(/\/$/, '');++const num = (v, d) => {++ const n = parseFloat(v);++ return Number.isFinite(n) ? n : d;++};++++const ZONE_ID_RE = /^[0-9a-f]{32}$/;++const HOST_RE = /^(?=.{1,253}$)([a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\.)+[a-z]{2,63}$/;++++/**++ * CLOUDFLARE_ZONE_IDS: `host=zoneid` pairs, comma-separated (openvibe.wiki=<32 hex>,...). A URL++ * belongs to the zone whose host equals its hostname or is a parent of it (longest match wins).++ * Malformed pairs throw at boot rather than silently purging nothing.++ */++function parseZones(v) {++ const out = [];++ for (const pair of list(v, [])) {++ const i = pair.indexOf('=');++ const host = (i > 0 ? pair.slice(0, i) : '').trim().toLowerCase().replace(/\.$/, '');++ const zoneId = (i > 0 ? pair.slice(i + 1) : '').trim().toLowerCase();++ if (!HOST_RE.test(host) || !ZONE_ID_RE.test(zoneId)) throw new Error(`CLOUDFLARE_ZONE_IDS: "${pair}" is not host=<32-hex zone id>`);++ out.push({ host, zoneId });++ }++ return out.sort((a, b) => b.host.length - a.host.length);++}/*** Services whose index-document events are accepted from OpenVibe.Events. Events already@@@ -64,7 +88,30 @@ function load(env = process.env) {maxFacetFilters: 8,},maxBodyBytes: int(env.SEARCH_MAX_BODY_BYTES, 256 * 1024),++++ // Freshness weighting of full-text relevance (README "Ranking"):++ // score = bm25 × (1 + weight × 2^(−age_days / halfLifeDays)); weight 0 turns it off.++ freshness: {++ weight: Math.max(0, num(env.SEARCH_FRESHNESS_WEIGHT, 1)),++ halfLifeDays: Math.max(0.01, num(env.SEARCH_FRESHNESS_HALF_LIFE_DAYS, 30)),++ },++++ savedSearches: {++ maxPerSubject: Math.max(1, int(env.SEARCH_SAVED_MAX_PER_SUBJECT, 50)),++ },++++ // Removal purge queue (server/purge.js). Every removal is recorded for owners; the++ // Cloudflare purge of a formerly public URL runs only when the token is set.++ purge: {++ cloudflareToken: env.CLOUDFLARE_PURGE_TOKEN || '',++ zones: parseZones(env.CLOUDFLARE_ZONE_IDS),++ // Paths on the removed URL's host purged with it (the product's sitemap, feeds).++ relatedPaths: list(env.CLOUDFLARE_PURGE_RELATED_PATHS, ['/sitemap.xml'])++ .filter(p => /^\/[A-Za-z0-9._~\/-]*$/.test(p)),++ apiBase: strip(env.CLOUDFLARE_API_BASE || 'https://api.cloudflare.com/client/v4'),++ intervalMs: int(env.CLOUDFLARE_PURGE_INTERVAL_MS, 5000),++ },};}--module.exports = { load, DEFAULT_EVENT_OWNERS };++module.exports = { load, parseZones, DEFAULT_EVENT_OWNERS };@@@ -7,6 +7,8 @@ const fs = require('fs');const path = require('path');const Database = require('better-sqlite3');const outbox = require('./events/outbox');++const purge = require('./purge');++const saved = require('./saved');const SCHEMA = `CREATE TABLE IF NOT EXISTS documents (@@@ -89,6 +91,8 @@ function openDb(dbPath) {db.pragma('busy_timeout = 5000');db.exec(SCHEMA);outbox.ensureSchema(db);++ purge.ensureSchema(db);++ saved.ensureSchema(db);return db;}@@@ -23,7 +23,29 @@ const WEIGHTS = '10.0, 4.0, 1.0'; // title, summary, bodyconst MARK_OPEN = '\u0001';const MARK_CLOSE = '\u0002';--function createEngine(db) {++const DAY_MS = 24 * 3600 * 1000;++++/**++ * Freshness multiplier for a document dated `iso` at reference time `nowMs`:++ *++ * 1 + weight × 2^(−age_days / halfLifeDays)++ *++ * With the defaults (weight 1, half-life 30 days) a document published today counts ×2.0, one++ * 30 days old ×1.5, 60 days ×1.25, 90 days ×1.125, a year old ×1.0002: relevance still decides,++ * and between comparably relevant documents the newer one ranks first. A future date counts as++ * today; no date (or an unparsable one) counts ×1. Weight 0 turns the boost off.++ */++function freshnessBoost(iso, nowMs, halfLifeDays, weight) {++ if (!weight || !iso) return 1;++ const t = Date.parse(iso);++ if (!Number.isFinite(t)) return 1;++ const ageDays = Math.max(0, (nowMs - t) / DAY_MS);++ return 1 + weight * Math.pow(2, -ageDays / halfLifeDays);++}++++function createEngine(db, { freshness = { weight: 1, halfLifeDays: 30 } } = {}) {++ // bm25() is negative (lower = more relevant), so multiplying by a boost ≥ 1 ranks newer first.++ db.function('ov_freshness', { deterministic: true }, (iso, nowMs, halfLifeDays, weight) => freshnessBoost(iso, nowMs, halfLifeDays, weight));db.exec(`CREATE VIRTUAL TABLE IF NOT EXISTS fts_public USING fts5(title, summary, body, ${TOKENIZE});CREATE VIRTUAL TABLE IF NOT EXISTS fts_restricted USING fts5(title, summary, body, ${TOKENIZE});@@@ -81,17 +103,25 @@ function createEngine(db) {return parts.length ? ` AND ${parts.join(' AND ')}` : '';}-- /** The visible, matching set as (rid, rank, snip) rows. match = FTS5 expression. */-- function matchedSet({ match, filters, viewer, snippets = true }, params) {++ /**++ * The visible, matching set as (rid, rank, snip) rows. match = FTS5 expression. rank is bm25++ * times the freshness boost at reference time `now` (ms), so pages of one query (which carry++ * their first page's `now` in the cursor) rank against the same clock.++ */++ function matchedSet({ match, filters, viewer, snippets = true, now = Date.now() }, params) {params.match = match;++ params.r_now = now;++ params.r_half = freshness.halfLifeDays;++ params.r_weight = freshness.weight;const where = filterClause(filters, params);const snip = (t) => (snippets ? `snippet(${t}, -1, char(1), char(2), '…', 16)` : 'NULL');-- const branches = [`SELECT d.rid AS rid, bm25(fts_public, ${WEIGHTS}) AS rank, ${snip('fts_public')} AS snip++ const rank = (t) => `bm25(${t}, ${WEIGHTS}) * ov_freshness(COALESCE(d.published_at, d.updated_at), @r_now, @r_half, @r_weight)`;++ const branches = [`SELECT d.rid AS rid, ${rank('fts_public')} AS rank, ${snip('fts_public')} AS snipFROM fts_public JOIN documents d ON d.rid = fts_public.rowidWHERE fts_public MATCH @match AND d.exposure = ${EXPOSURE.public_listed} AND d.deleted = 0${where}`];const acl = aclClause(viewer, params);if (acl) {-- branches.push(`SELECT d.rid AS rid, bm25(fts_restricted, ${WEIGHTS}) AS rank, ${snip('fts_restricted')} AS snip++ branches.push(`SELECT d.rid AS rid, ${rank('fts_restricted')} AS rank, ${snip('fts_restricted')} AS snipFROM fts_restricted JOIN documents d ON d.rid = fts_restricted.rowidWHERE fts_restricted MATCH @match AND d.exposure = ${EXPOSURE.restricted} AND d.deleted = 0${where} AND ${acl}`);}@@@ -111,9 +141,9 @@ function createEngine(db) {// ── Queries ──────────────────────────────────────────────/** Ranked full-text search. after = { rank, rid } keyset cursor. */-- function search({ match, filters = {}, viewer, limit, after }) {++ function search({ match, filters = {}, viewer, limit, after, now = Date.now() }) {const params = { limit: limit + 1 };-- let sql = `SELECT rid, rank, snip FROM (${matchedSet({ match, filters, viewer }, params)})`;++ let sql = `SELECT rid, rank, snip FROM (${matchedSet({ match, filters, viewer, now }, params)})`;if (after) {sql += ' WHERE (rank > @c_rank OR (rank = @c_rank AND rid > @c_rid))';params.c_rank = after.rank;@@@ -175,7 +205,7 @@ function createEngine(db) {};}-- return { name: 'sqlite-fts5', put, remove, search, browse, facetCounts, suggest, counts };++ return { name: 'sqlite-fts5', put, remove, search, browse, facetCounts, suggest, counts, freshness };}// ── Query text → FTS5 expression ─────────────────────────────@@@ -202,4 +232,4 @@ function snippetHtml(snip) {return esc.split(MARK_OPEN).join('<mark>').split(MARK_CLOSE).join('</mark>');}--module.exports = { createEngine, toMatch, snippetHtml };++module.exports = { createEngine, toMatch, snippetHtml, freshnessBoost };@@@ -13,14 +13,19 @@ const { createEngine } = require('./engine/fts5');const { createStore } = require('./store');const { createOutbox, createRelay } = require('./events/outbox');const { createKeyStore, createAuth } = require('./auth');++const { createPurgeQueue, createPurger } = require('./purge');++const { createSavedSearches } = require('./saved');const { createApp } = require('./app');async function start({ config, now = () => Date.now(), fetchImpl = globalThis.fetch, tokenClient, log = console, listen = true } = {}) {config = config || load();const db = openDb(config.dbPath);-- const engine = createEngine(db);++ const engine = createEngine(db, { freshness: config.freshness });const outbox = createOutbox(db, { source: config.serviceId, now });-- const store = createStore({ db, engine, outbox, now });++ const purges = createPurgeQueue(db, { config: config.purge, now });++ const purger = createPurger({ db, config: config.purge, fetchImpl, log, now });++ const saved = createSavedSearches(db, { maxPerSubject: config.savedSearches.maxPerSubject, now });++ const store = createStore({ db, engine, outbox, purges, now });const relay = createRelay({db, outbox, eventsUrl: config.events.url, intervalMs: config.events.relayIntervalMs, fetchImpl, log, now,tokenClient,@@@ -28,10 +33,11 @@ async function start({ config, now = () => Date.now(), fetchImpl = globalThis.fe});const keys = createKeyStore({ urls: [config.networkInternalUrl, config.networkUrl], pem: config.networkPublicKey, fetchImpl, log });const auth = createAuth({ config, keys });-- const app = createApp({ config, db, store, engine, auth, keys, outbox, relay, log, now });++ const app = createApp({ config, db, store, engine, auth, keys, outbox, relay, purges, purger, saved, log, now });const keyLoaded = keys.start().catch(() => null);relay.start();++ purger.start();const pruneTimer = setInterval(() => { try { outbox.prune(); } catch (err) { log.error(`[outbox] prune: ${err.message}`); } }, 6 * 3600 * 1000);pruneTimer.unref?.();@@@ -48,6 +54,7 @@ async function start({ config, now = () => Date.now(), fetchImpl = globalThis.feclearInterval(pruneTimer);keys.stop();await relay.stop();++ await purger.stop();if (server) {server.closeAllConnections?.();await new Promise(resolve => server.close(() => resolve()));@@@ -55,7 +62,7 @@ async function start({ config, now = () => Date.now(), fetchImpl = globalThis.fedb.close();}-- return { config, db, engine, store, outbox, relay, keys, keyLoaded, auth, app, server, close };++ return { config, db, engine, store, outbox, relay, purges, purger, saved, keys, keyLoaded, auth, app, server, close };}if (require.main === module) {@@@ -34,7 +34,7 @@ function readers(db) {};}--function createSearchReadiness({ db, keys, config, engine, store, outbox, relay, release = null }) {++function createSearchReadiness({ db, keys, config, engine, store, outbox, relay, purges = null, purger = null, saved = null, release = null }) {const read = readers(db);return createReadiness({service: 'search',@@@ -70,13 +70,20 @@ function createSearchReadiness({ db, keys, config, engine, store, outbox, relay,documents: dbOk ? store.counts() : null,outbox: dbOk ? { pending: outbox.pending(), rejected: outbox.rejected(), relay: config.events.url ? (relay.running() ? 'running' : 'stopped') : 'off (EVENTS_URL unset)' } : null,webhook: config.events.webhookSecrets.length ? 'on' : 'off (SEARCH_EVENTS_SECRET unset)',++ purge: dbOk && purges ? {++ cdn: purges.cdnOn() ? (purger && purger.running() ? 'cloudflare' : 'cloudflare (stopped)') : 'off (CLOUDFLARE_PURGE_TOKEN unset)',++ zones: config.purge.zones.length,++ ...purges.cdnCounts(),++ } : null,++ saved_searches: dbOk && saved ? saved.total() : null,++ freshness: config.freshness,};},});}/** Search gauges on the openvibe-shared/metrics registry. */--function registerSearchGauges(registry, { db, outbox }) {++function registerSearchGauges(registry, { db, outbox, purges = null }) {const read = readers(db);registry.gauge({name: 'search_documents', help: 'Documents held, by exposure (public_listed and restricted are searchable; tombstones and drafts are none)', labelNames: ['exposure'],@@@ -88,6 +95,12 @@ function registerSearchGauges(registry, { db, outbox }) {});registry.gauge({ name: 'search_outbox_pending', help: 'Events waiting in the outbox', collect: () => outbox.pending() });registry.gauge({ name: 'search_outbox_rejected', help: 'Events OpenVibe.Events refused for good', collect: () => outbox.rejected() });++ if (purges) {++ registry.gauge({++ name: 'search_cdn_purges', help: 'Cloudflare cache purges of removed documents, by state', labelNames: ['state'],++ collect: () => Object.entries(purges.cdnCounts()).map(([state, value]) => ({ labels: { state }, value })),++ });++ }}module.exports = { createSearchReadiness, registerSearchGauges };nnew file mode 100644@@@ -0,0 +1,257 @@++'use strict';++/**++ * The removal purge queue: Search's own consumer of its search.document.removed decisions.++ *++ * Every change that lowers a document's exposure (store.js announce()) records, in the same++ * transaction as the index change and the outbox event:++ *++ * removals one row per removal: the feed products read to drop the document from their own++ * caches and sitemaps (GET /api/v1/owners/:owner/removals, owner token). Products++ * own their pages, so Search tells them rather than reaching into them.++ * cdn_purges one row per URL to purge from Cloudflare's cache: the canonical URL the document++ * had while it was public, plus the configured related paths on the same host++ * (the product's sitemap by default). Written only when CLOUDFLARE_PURGE_TOKEN is++ * set; without it the CDN side is inert and nothing is queued.++ *++ * The purger (createPurger) posts pending URLs to the Cloudflare API, grouped by zone, at most++ * 30 per call (POST /zones/:zone/purge_cache { files }), with backoff on 429/5xx/network errors,++ * permanent failure after MAX_ATTEMPTS or a 4xx refusal (a refused batch is retried one URL at a++ * time so one bad URL cannot sink the others). The token is sent only in the Authorization++ * header and never stored or logged.++ */++++const BACKOFF_MS = [2000, 10000, 60000, 300000, 900000, 1800000, 3600000];++const MAX_ATTEMPTS = 8;++const FILES_PER_CALL = 30;++const DETAIL_MAX = 300;++++function ensureSchema(db) {++ db.exec(`++ CREATE TABLE IF NOT EXISTS removals (++ seq INTEGER PRIMARY KEY AUTOINCREMENT,++ event_id TEXT NOT NULL, -- the search.document.removed event++ owner TEXT NOT NULL,++ type TEXT NOT NULL,++ id TEXT NOT NULL,++ revision INTEGER NOT NULL,++ reason TEXT NOT NULL,++ previous_exposure TEXT NOT NULL,++ exposure TEXT NOT NULL,++ canonical_url TEXT, -- only a URL that had been public++ at INTEGER NOT NULL++ );++ CREATE INDEX IF NOT EXISTS idx_removals_owner ON removals (owner, seq);++++ CREATE TABLE IF NOT EXISTS cdn_purges (++ seq INTEGER PRIMARY KEY AUTOINCREMENT,++ removal_seq INTEGER NOT NULL,++ provider TEXT NOT NULL DEFAULT 'cloudflare',++ zone_id TEXT,++ url TEXT NOT NULL,++ state TEXT NOT NULL CHECK (state IN ('pending', 'purged', 'failed', 'skipped')),++ detail TEXT,++ attempts INTEGER NOT NULL DEFAULT 0,++ next_attempt_at INTEGER NOT NULL DEFAULT 0,++ created_at INTEGER NOT NULL,++ done_at INTEGER++ );++ -- One pending purge per URL: a sitemap shared by many removals is purged once per batch.++ CREATE UNIQUE INDEX IF NOT EXISTS idx_cdn_purges_pending ON cdn_purges (zone_id, url) WHERE state = 'pending';++ CREATE INDEX IF NOT EXISTS idx_cdn_purges_due ON cdn_purges (state, next_attempt_at, seq);++ `);++}++++/** The zone a URL belongs to: exact host or a parent domain in the map, longest first. */++function zoneFor(url, zones) {++ let u;++ try { u = new URL(url); } catch { return null; }++ if (u.protocol !== 'https:' && u.protocol !== 'http:') return null;++ const host = u.hostname.toLowerCase();++ for (const z of zones) {++ if (host === z.host || host.endsWith(`.${z.host}`)) return z.zoneId;++ }++ return null;++}++++/** The canonical URL plus the related paths on its host, without fragments, de-duplicated. */++function purgeUrls(canonicalUrl, relatedPaths) {++ let u;++ try { u = new URL(canonicalUrl); } catch { return []; }++ if (u.protocol !== 'https:' && u.protocol !== 'http:') return [];++ u.hash = '';++ const out = [u.toString()];++ for (const p of relatedPaths) out.push(`${u.origin}${p}`);++ return [...new Set(out)];++}++++function createPurgeQueue(db, { config, now = () => Date.now() }) {++ const cdnOn = Boolean(config.cloudflareToken);++ const st = {++ insRemoval: db.prepare(`INSERT INTO removals (event_id, owner, type, id, revision, reason, previous_exposure, exposure, canonical_url, at)++ VALUES (@event_id, @owner, @type, @id, @revision, @reason, @previous_exposure, @exposure, @canonical_url, @at)`),++ insPurge: db.prepare(`INSERT INTO cdn_purges (removal_seq, zone_id, url, state, detail, created_at, done_at)++ VALUES (@removal_seq, @zone_id, @url, @state, @detail, @created_at, @done_at)++ ON CONFLICT (zone_id, url) WHERE state = 'pending' DO NOTHING`),++ ownerFeed: db.prepare('SELECT * FROM removals WHERE owner = ? AND seq > ? ORDER BY seq LIMIT ?'),++ cdnCounts: db.prepare('SELECT state, COUNT(*) AS n FROM cdn_purges GROUP BY state'),++ purgesFor: db.prepare('SELECT * FROM cdn_purges WHERE removal_seq = ? ORDER BY seq'),++ };++++ /**++ * Record one removal (the payload of the search.document.removed event). Must run inside the++ * transaction that made the change, like the outbox.++ */++ function record(eventId, p) {++ if (!db.inTransaction) throw new Error('purge.record() must run inside the transaction that makes the change');++ const at = now();++ const removalSeq = Number(st.insRemoval.run({++ event_id: eventId, owner: p.owner, type: p.type, id: p.id, revision: p.revision, reason: p.reason,++ previous_exposure: p.previous_exposure, exposure: p.exposure, canonical_url: p.canonical_url || null, at,++ }).lastInsertRowid);++ if (!cdnOn || !p.canonical_url) return removalSeq;++ for (const url of purgeUrls(p.canonical_url, config.relatedPaths)) {++ const zoneId = zoneFor(url, config.zones);++ st.insPurge.run({++ removal_seq: removalSeq, zone_id: zoneId, url,++ state: zoneId ? 'pending' : 'skipped',++ detail: zoneId ? null : 'no Cloudflare zone configured for this host (CLOUDFLARE_ZONE_IDS)',++ created_at: at, done_at: zoneId ? null : at,++ });++ }++ return removalSeq;++ }++++ /** The owner's removal feed, oldest first after a cursor. */++ function ownerFeed(owner, { after = 0, limit = 200 } = {}) {++ const rows = st.ownerFeed.all(owner, after, limit);++ return {++ removals: rows.map(r => ({++ seq: r.seq, event_id: r.event_id, type: r.type, id: r.id, revision: r.revision, reason: r.reason,++ previous_exposure: r.previous_exposure, exposure: r.exposure, canonical_url: r.canonical_url,++ at: new Date(r.at).toISOString(),++ })),++ next_after: rows.length ? rows[rows.length - 1].seq : null,++ };++ }++++ function cdnCounts() {++ const out = { pending: 0, purged: 0, failed: 0, skipped: 0 };++ for (const r of st.cdnCounts.all()) out[r.state] = r.n;++ return out;++ }++++ return { record, ownerFeed, cdnCounts, purgesFor: (seq) => st.purgesFor.all(seq), cdnOn: () => cdnOn };++}++++/**++ * createPurger({ db, config, fetchImpl?, log?, now? }) → { start, stop, flush, running }++ * Does nothing without config.cloudflareToken.++ */++function createPurger({ db, config, fetchImpl = globalThis.fetch, log = console, now = () => Date.now() }) {++ const token = config.cloudflareToken;++ const q = {++ due: db.prepare(`SELECT seq, zone_id, url, attempts FROM cdn_purges++ WHERE state = 'pending' AND next_attempt_at <= ? ORDER BY seq LIMIT ?`),++ purged: db.prepare("UPDATE cdn_purges SET state = 'purged', attempts = attempts + 1, detail = ?, done_at = ? WHERE seq = ?"),++ retry: db.prepare('UPDATE cdn_purges SET attempts = attempts + 1, next_attempt_at = ?, detail = ? WHERE seq = ?'),++ failed: db.prepare("UPDATE cdn_purges SET state = 'failed', attempts = attempts + 1, detail = ?, done_at = ? WHERE seq = ?"),++ };++ let timer = null;++ let flushing = null;++++ const clip = (s) => (token ? String(s || '').split(token).join('[token]') : String(s || '')).slice(0, DETAIL_MAX);++++ function mark(rows, fn) { db.transaction(() => { for (const r of rows) fn(r); })(); }++++ function retryOrFail(rows, detail) {++ mark(rows, (r) => {++ if (r.attempts + 1 >= MAX_ATTEMPTS) q.failed.run(clip(`gave up after ${MAX_ATTEMPTS} attempts: ${detail}`), now(), r.seq);++ else q.retry.run(now() + BACKOFF_MS[Math.min(r.attempts, BACKOFF_MS.length - 1)], clip(detail), r.seq);++ });++ }++++ async function call(zoneId, urls) {++ const res = await fetchImpl(`${config.apiBase}/zones/${encodeURIComponent(zoneId)}/purge_cache`, {++ method: 'POST',++ headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json', Accept: 'application/json' },++ body: JSON.stringify({ files: urls }),++ signal: AbortSignal.timeout(15000),++ });++ let body = null;++ try { body = await res.json(); } catch { body = null; }++ return { status: res.status, body };++ }++++ /** → { purged, retrying, failed } for one zone's batch. */++ async function purgeBatch(zoneId, rows) {++ let r;++ try {++ r = await call(zoneId, rows.map(x => x.url));++ } catch (err) {++ retryOrFail(rows, `network: ${err.message}`);++ return { purged: 0, retrying: rows.length, failed: 0 };++ }++ const errors = (r.body && Array.isArray(r.body.errors) ? r.body.errors : []).map(e => `${e.code}: ${e.message}`).join('; ');++ if (r.status >= 200 && r.status < 300 && r.body && r.body.success === true) {++ const id = r.body.result && r.body.result.id ? `cloudflare purge ${r.body.result.id}` : 'purged';++ mark(rows, x => q.purged.run(clip(id), now(), x.seq));++ return { purged: rows.length, retrying: 0, failed: 0 };++ }++ if (r.status === 429 || r.status >= 500 || r.status === 408) {++ retryOrFail(rows, `HTTP ${r.status} ${errors}`);++ return { purged: 0, retrying: rows.length, failed: 0 };++ }++ // Refused (bad URL, token without purge permission, unknown zone): isolate, then fail.++ if (rows.length > 1) {++ const total = { purged: 0, retrying: 0, failed: 0 };++ for (const row of rows) {++ const s = await purgeBatch(zoneId, [row]);++ total.purged += s.purged; total.retrying += s.retrying; total.failed += s.failed;++ }++ return total;++ }++ mark(rows, x => q.failed.run(clip(`HTTP ${r.status} ${errors || 'refused'}`), now(), x.seq));++ log.warn(`[purge] Cloudflare refused a purge: HTTP ${r.status} ${clip(errors)}`);++ return { purged: 0, retrying: 0, failed: rows.length };++ }++++ async function doFlush() {++ const total = { purged: 0, retrying: 0, failed: 0 };++ if (!token) return total;++ const rows = q.due.all(now(), 200);++ const byZone = new Map();++ for (const r of rows) {++ if (!byZone.has(r.zone_id)) byZone.set(r.zone_id, []);++ byZone.get(r.zone_id).push(r);++ }++ for (const [zoneId, zoneRows] of byZone) {++ for (let i = 0; i < zoneRows.length; i += FILES_PER_CALL) {++ const s = await purgeBatch(zoneId, zoneRows.slice(i, i + FILES_PER_CALL));++ total.purged += s.purged; total.retrying += s.retrying; total.failed += s.failed;++ }++ }++ if (total.retrying) log.warn(`[purge] ${total.retrying} URL(s) not purged yet; will retry`);++ return total;++ }++++ function flush() {++ if (!flushing) flushing = doFlush().finally(() => { flushing = null; });++ return flushing;++ }++++ function start() {++ if (timer || !token) return;++ timer = setInterval(() => { flush().catch(err => log.warn(`[purge] ${err.message}`)); }, config.intervalMs);++ timer.unref?.();++ }++++ function stop() {++ if (timer) clearInterval(timer);++ timer = null;++ return flushing || Promise.resolve();++ }++++ return { start, stop, flush, running: () => Boolean(timer) };++}++++module.exports = { ensureSchema, createPurgeQueue, createPurger, zoneFor, purgeUrls, MAX_ATTEMPTS, FILES_PER_CALL };nnew file mode 100644@@@ -0,0 +1,114 @@++'use strict';++/**++ * Saved searches: a signed-in person (a usr_ subject) keeps a query (text + filters) and runs it++ * again later. The saved row holds the query only, never results: every run is a fresh query as++ * that person at that moment, so a document that became private or was deleted since the save++ * is simply not there, and one they gained access to is.++ *++ * Notifications for new matches wait for OpenVibe.Network's notifications (README "Saved++ * searches"); last_run_at is kept so that job can tell what is new.++ */++const crypto = require('crypto');++const { ids } = require('openvibe-contracts');++++const ID_RE = /^svs_[0-9a-z]{26}$/;++++function ensureSchema(db) {++ db.exec(`++ CREATE TABLE IF NOT EXISTS saved_searches (++ id TEXT PRIMARY KEY,++ subject TEXT NOT NULL, -- usr_… (the person who saved it)++ name TEXT NOT NULL,++ q TEXT NOT NULL DEFAULT '',++ filters TEXT NOT NULL, -- canonical JSON {owner,type,language,facets}++ query_hash TEXT NOT NULL, -- same query saved twice = one row++ created_at INTEGER NOT NULL,++ updated_at INTEGER NOT NULL,++ last_run_at INTEGER,++ UNIQUE (subject, query_hash)++ );++ CREATE INDEX IF NOT EXISTS idx_saved_searches_subject ON saved_searches (subject, created_at);++ `);++}++++/** Filters in one canonical shape, so equal queries hash equally whatever order they came in. */++function canonicalFilters(f) {++ const out = {};++ if (f.owner) out.owner = f.owner;++ if (f.type) out.type = f.type;++ if (f.language) out.language = f.language;++ const facets = (f.facets || [])++ .map(([k, vs]) => [k, [...new Set(vs)].sort()])++ .sort((a, b) => (a[0] < b[0] ? -1 : a[0] > b[0] ? 1 : 0));++ if (facets.length) out.facets = facets;++ return out;++}++++function hashOf(q, filters) {++ return crypto.createHash('sha256').update(JSON.stringify([q, filters])).digest('base64url').slice(0, 22);++}++++function createSavedSearches(db, { maxPerSubject = 50, now = () => Date.now() } = {}) {++ const st = {++ list: db.prepare('SELECT * FROM saved_searches WHERE subject = ? ORDER BY created_at DESC, id DESC'),++ get: db.prepare('SELECT * FROM saved_searches WHERE id = ? AND subject = ?'),++ byHash: db.prepare('SELECT * FROM saved_searches WHERE subject = ? AND query_hash = ?'),++ count: db.prepare('SELECT COUNT(*) AS n FROM saved_searches WHERE subject = ?'),++ insert: db.prepare(`INSERT INTO saved_searches (id, subject, name, q, filters, query_hash, created_at, updated_at)++ VALUES (@id, @subject, @name, @q, @filters, @query_hash, @at, @at)`),++ rename: db.prepare('UPDATE saved_searches SET name = ?, updated_at = ? WHERE id = ?'),++ del: db.prepare('DELETE FROM saved_searches WHERE id = ? AND subject = ?'),++ ran: db.prepare('UPDATE saved_searches SET last_run_at = ? WHERE id = ?'),++ total: db.prepare('SELECT COUNT(*) AS n FROM saved_searches'),++ };++++ function view(r) {++ return {++ id: r.id,++ name: r.name,++ q: r.q,++ filters: JSON.parse(r.filters),++ created_at: new Date(r.created_at).toISOString(),++ updated_at: new Date(r.updated_at).toISOString(),++ last_run_at: r.last_run_at ? new Date(r.last_run_at).toISOString() : null,++ };++ }++++ /**++ * Save (subject, q, filters) with a name. The same query again returns the existing row++ * (renamed when a new name is given): { created, saved } | { error: 'limit' }.++ */++ function save(subject, { name, q, filters }) {++ return db.transaction(() => {++ const f = canonicalFilters(filters);++ const hash = hashOf(q, f);++ const at = now();++ const existing = st.byHash.get(subject, hash);++ if (existing) {++ if (name && name !== existing.name) st.rename.run(name, at, existing.id);++ return { created: false, saved: view(st.get.get(existing.id, subject)) };++ }++ if (st.count.get(subject).n >= maxPerSubject) return { error: 'limit' };++ const id = `svs_${ids.ulid(at).toLowerCase()}`;++ st.insert.run({ id, subject, name: name || q || 'All documents', q, filters: JSON.stringify(f), query_hash: hash, at });++ return { created: true, saved: view(st.get.get(id, subject)) };++ })();++ }++++ return {++ list: (subject) => st.list.all(subject).map(view),++ /** One saved search of this subject; another subject's id is the same as a missing one. */++ get: (subject, id) => {++ if (!ID_RE.test(String(id))) return null;++ const r = st.get.get(id, subject);++ return r ? view(r) : null;++ },++ save,++ remove: (subject, id) => ID_RE.test(String(id)) && st.del.run(id, subject).changes > 0,++ markRun: (id) => st.ran.run(now(), id),++ total: () => st.total.get().n,++ maxPerSubject,++ };++}++++module.exports = { ensureSchema, createSavedSearches, canonicalFilters, ID_RE };@@@ -15,7 +15,7 @@ const { normalize, effectiveIndexability, exposureOf, facetPairs, EXPOSURE } = rconst EXPOSURE_NAME = Object.fromEntries(Object.entries(EXPOSURE).map(([k, v]) => [v, k]));--function createStore({ db, engine, outbox, now = () => Date.now() }) {++function createStore({ db, engine, outbox, purges = null, now = () => Date.now() }) {const st = {get: db.prepare('SELECT * FROM documents WHERE owner = ? AND type = ? AND id = ?'),byRid: db.prepare('SELECT * FROM documents WHERE rid = ?'),@@@ -89,19 +89,17 @@ function createStore({ db, engine, outbox, now = () => Date.now() }) {const subject = { type: 'document', id: `${doc.owner}/${doc.type}/${doc.id}`, revision: doc.revision };const wasPublic = prevExposure >= EXPOSURE.public_unlisted;if (exposure < prevExposure) {-- outbox.enqueue({-- event_type: 'search.document.removed',-- subject,-- trace_id: traceId,-- payload: {-- owner: doc.owner, type: doc.type, id: doc.id, revision: doc.revision,-- reason: removalReason(prevDoc, doc),-- previous_exposure: EXPOSURE_NAME[prevExposure],-- exposure: EXPOSURE_NAME[exposure],-- // Only a URL that was public already: caches and sitemaps purge it.-- canonical_url: wasPublic ? prevRow.canonical_url : null,-- },-- });++ const payload = {++ owner: doc.owner, type: doc.type, id: doc.id, revision: doc.revision,++ reason: removalReason(prevDoc, doc),++ previous_exposure: EXPOSURE_NAME[prevExposure],++ exposure: EXPOSURE_NAME[exposure],++ // Only a URL that was public already: caches and sitemaps purge it.++ canonical_url: wasPublic ? prevRow.canonical_url : null,++ };++ const env = outbox.enqueue({ event_type: 'search.document.removed', subject, trace_id: traceId, payload });++ // Search's own consumer: the owners' removal feed and the CDN purge queue (purge.js).++ if (purges) purges.record(env.event_id, payload);}if (!doc.deleted && exposure > EXPOSURE.none) {outbox.enqueue({nnew file mode 100644@@@ -0,0 +1,176 @@++'use strict';++/**++ * The public search page at search.openvibe.network: server-rendered, works without JavaScript,++ * same searcher and visibility rules as GET /api/v1/search (anonymous = public, published,++ * indexable documents only; an ov_token cookie for this host adds its person's ACL matches).++ *++ * GET / search form; ?q= results, ?owner= / ?type= filters, ?cursor= next page++ * GET /robots.txt the front page may be crawled, result pages and the API not++ *++ * Every answer is no-store (a document that leaves the index leaves this page at once) and result++ * pages are noindex. Clients that do not ask for HTML get the plain-text route index at /.++ */++const express = require('express');++const { AuthError, ANONYMOUS } = require('../auth');++const { QueryError, one } = require('../api/query');++++const CSP = "default-src 'none'; style-src 'unsafe-inline'; img-src 'self' data:; form-action 'self'; base-uri 'none'; frame-ancestors 'none'";++++const esc = (s) => String(s == null ? '' : s).replace(/[&<>"']/g, c => ({ '&': '&', '<': '<', '>': '>', '"': '"', "'": ''' }[c]));++++/** Only http(s) links are rendered as links; anything else is shown as text. */++function safeHref(url) {++ try {++ const u = new URL(url);++ return u.protocol === 'https:' || u.protocol === 'http:' ? u.toString() : null;++ } catch { return null; }++}++++function dateOf(iso) {++ const t = Date.parse(iso || '');++ return Number.isFinite(t) ? new Date(t).toISOString().slice(0, 10) : '';++}++++const TEXT_INDEX = [++ 'OpenVibe.Search: permission-aware search over documents the network\'s services index.',++ '',++ 'GET /?q= search page (HTML)',++ 'GET /api/v1/search?q=&owner=&type=&facet.<key>=&cursor= query (anonymous: public only)',++ 'GET /api/v1/suggest?q= title suggestions',++ 'GET /api/v1/documents/:owner/:type/:id one document the caller may see',++ 'GET /api/v1/saved-searches, POST, GET/DELETE /:id, GET /:id/results saved searches (signed in)',++ 'PUT /api/v1/documents/:owner/:type/:id index (owner, search.document.write)',++ 'DELETE /api/v1/documents/:owner/:type/:id?revision= tombstone (owner)',++ 'GET /api/v1/owners/:owner/documents reconciliation (owner)',++ 'GET /api/v1/owners/:owner/removals?after= removal feed for caches and sitemaps (owner)',++ 'POST /internal/events OpenVibe.Events delivery (signed, host-local)',++ 'GET /api/health, /api/ready, /release.json',++ '',++ 'Source: https://github.com/OpenVibers/OpenVibe.Search',++ '',++].join('\n');++++function layout({ title, q, owner, type, body, noindex }) {++ return `<!doctype html>++<html lang="en">++<head>++<meta charset="utf-8">++<meta name="viewport" content="width=device-width, initial-scale=1">++<title>${esc(title)}</title>++${noindex ? '<meta name="robots" content="noindex, nofollow">\n' : ''}<meta name="description" content="Search what the OpenVibe network's services have published.">++<meta name="color-scheme" content="light dark">++<style>++:root { --bg: #fff; --fg: #1a1a1a; --muted: #5c5c66; --line: #dcdce3; --accent: #2456d6; --mark: #fff2a8; }++@media (prefers-color-scheme: dark) { :root { --bg: #111317; --fg: #e8e8ec; --muted: #a0a0ab; --line: #2c2f36; --accent: #7aa2ff; --mark: #5a4b00; } }++* { box-sizing: border-box; }++body { margin: 0; background: var(--bg); color: var(--fg); font: 16px/1.5 system-ui, -apple-system, "Segoe UI", sans-serif; }++main { max-width: 760px; margin: 0 auto; padding: 24px 16px 48px; }++h1 { font-size: 1.4rem; margin: 0 0 4px; }++.lede { color: var(--muted); margin: 0 0 20px; }++form { display: flex; flex-wrap: wrap; gap: 8px; margin-bottom: 20px; }++input[type=search] { flex: 1 1 260px; min-width: 0; padding: 10px 12px; font: inherit; color: inherit; background: transparent; border: 1px solid var(--line); border-radius: 8px; }++button { padding: 10px 16px; font: inherit; border: 0; border-radius: 8px; background: var(--accent); color: #fff; cursor: pointer; }++ol { list-style: none; padding: 0; margin: 0; }++li { padding: 14px 0; border-top: 1px solid var(--line); overflow-wrap: anywhere; }++li a.t { color: var(--accent); font-size: 1.05rem; text-decoration: none; }++li a.t:hover { text-decoration: underline; }++.meta { color: var(--muted); font-size: .85rem; }++.snip { margin: 4px 0 0; }++mark { background: var(--mark); color: inherit; }++.empty, .note { color: var(--muted); }++.more { display: inline-block; margin-top: 16px; color: var(--accent); }++footer { margin-top: 40px; color: var(--muted); font-size: .85rem; }++footer a { color: inherit; }++</style>++</head>++<body>++<main>++<h1><a href="/" style="color:inherit;text-decoration:none">OpenVibe.Search</a></h1>++<p class="lede">Search what the OpenVibe network's services have published. Alpha: the index holds only what they have sent so far.</p>++<form method="get" action="/" role="search">++<input type="search" name="q" value="${esc(q)}" maxlength="500" placeholder="Search the network" aria-label="Search terms" autofocus>++${owner ? `<input type="hidden" name="owner" value="${esc(owner)}">` : ''}${type ? `<input type="hidden" name="type" value="${esc(type)}">` : ''}<button type="submit">Search</button>++</form>++${body}++<footer>++<p>Private, draft and deleted documents are never shown. A document that is removed or made private leaves these results at once.++JSON API: <a href="/api/v1/search?q=${encodeURIComponent(q || '')}">/api/v1/search</a>.++Source: <a href="https://github.com/OpenVibers/OpenVibe.Search">OpenVibers/OpenVibe.Search</a>.</p>++</footer>++</main>++</body>++</html>++`;++}++++function resultsHtml(results) {++ return `<ol>${results.map((r) => {++ const href = safeHref(r.canonical_url);++ const title = esc(r.title || `${r.owner}/${r.type}/${r.id}`);++ const date = dateOf(r.published_at || r.updated_at);++ return `<li>${href ? `<a class="t" href="${esc(href)}">${title}</a>` : `<span class="t">${title}</span>`}++<div class="meta">${esc(r.owner)} · ${esc(r.type)}${date ? ` · ${date}` : ''}${r.visibility !== 'public' ? ` · ${esc(r.visibility)}` : ''}</div>++${r.snippet_html ? `<p class="snip">${r.snippet_html}</p>` : r.summary ? `<p class="snip">${esc(r.summary)}</p>` : ''}</li>`;++ }).join('\n')}</ol>`;++}++++function pageRouter({ searcher, auth }) {++ const router = express.Router();++++ router.get('/robots.txt', (_req, res) => {++ res.type('text/plain').set('Cache-Control', 'public, max-age=3600')++ .send('User-agent: *\nAllow: /$\nDisallow: /?\nDisallow: /api/\nDisallow: /internal/\n');++ });++++ router.get('/', (req, res, next) => {++ const wantsHtml = /\btext\/html\b/.test(String(req.get('accept') || ''));++ res.setHeader('Cache-Control', 'no-store');++ res.setHeader('Vary', 'Accept, Cookie');++ if (!wantsHtml) return res.type('text/plain').send(TEXT_INDEX);++++ const q = String(one(req.query.q) || '').slice(0, 500);++ const owner = one(req.query.owner) ? String(one(req.query.owner)) : '';++ const type = one(req.query.type) ? String(one(req.query.type)) : '';++ const cursor = one(req.query.cursor);++ const searching = Boolean(q.trim() || owner || type);++ res.setHeader('Content-Security-Policy', CSP);++ res.setHeader('X-Frame-Options', 'DENY');++ res.setHeader('Referrer-Policy', 'strict-origin-when-cross-origin');++ if (searching) res.setHeader('X-Robots-Tag', 'noindex, nofollow');++++ let viewer;++ try {++ viewer = auth.viewer(req);++ } catch (err) {++ if (!(err instanceof AuthError)) return next(err);++ viewer = null; // a browser page never 401s: an unverifiable credential searches as anonymous++ }++ let body = '<p class="note">Type some words to search. Filters: <code>?owner=wiki</code>, <code>?type=page</code>.</p>';++ let status = 200;++ if (searching) {++ try {++ const out = searcher.run({ text: q, query: { ...(owner ? { owner } : {}), ...(type ? { type } : {}) }, viewer: viewer || ANONYMOUS, cursor });++ body = out.results.length++ ? resultsHtml(out.results)++ : `<p class="empty">${cursor ? 'No more results.' : 'Nothing found. The index is young: services add documents as they publish them.'}</p>`;++ if (out.next_cursor) {++ const u = new URLSearchParams();++ if (q) u.set('q', q);++ if (owner) u.set('owner', owner);++ if (type) u.set('type', type);++ u.set('cursor', out.next_cursor);++ body += `<a class="more" href="/?${esc(u.toString())}">More results</a>`;++ }++ } catch (err) {++ if (!(err instanceof QueryError)) return next(err);++ status = 400;++ body = `<p class="empty">${esc(err.message)}.</p>`;++ }++ }++ res.status(status).type('html').send(layout({ title: q ? `${q} · OpenVibe.Search` : 'OpenVibe.Search', q, owner, type, body, noindex: searching }));++ });++++ return router;++}++++module.exports = { pageRouter, TEXT_INDEX };nnew file mode 100644@@@ -0,0 +1,91 @@++'use strict';++/**++ * Freshness weighting: score = bm25 × (1 + weight × 2^(−age_days / half_life)). Newer documents++ * rank higher between comparably relevant ones; relevance still wins over age; pages of one query++ * rank on the first page's clock.++ */++const assert = require('assert');++const { boot, request, doc, suite } = require('./helpers');++const { freshnessBoost } = require('../server/engine/fts5');++++const t = suite('freshness');++const NOW = Date.parse('2026-09-23T12:00:00Z');++const daysAgo = (n) => new Date(NOW - n * 86400000).toISOString();++++t('the decay: ×(1+w) today, ×(1+w/2) one half-life ago, → ×1 when old; future = today; none = ×1', () => {++ const close = (a, b) => assert.ok(Math.abs(a - b) < 1e-9, `${a} ≠ ${b}`);++ close(freshnessBoost(daysAgo(0), NOW, 30, 1), 2);++ close(freshnessBoost(daysAgo(30), NOW, 30, 1), 1.5);++ close(freshnessBoost(daysAgo(60), NOW, 30, 1), 1.25);++ close(freshnessBoost(daysAgo(90), NOW, 30, 1), 1.125);++ assert.ok(freshnessBoost(daysAgo(365), NOW, 30, 1) < 1.001);++ close(freshnessBoost(daysAgo(-10), NOW, 30, 1), 2);++ close(freshnessBoost(null, NOW, 30, 1), 1);++ close(freshnessBoost('not a date', NOW, 30, 1), 1);++ close(freshnessBoost(daysAgo(0), NOW, 30, 0), 1);++ close(freshnessBoost(daysAgo(7), NOW, 7, 0.5), 1.25);++});++++async function corpus(env = {}) {++ let clock = NOW;++ const svc = await boot({ env, now: () => clock });++ // Same text, same length: bm25 ties, so only freshness separates them. The old one is indexed++ // first (lower rid), so without freshness it would come first.++ svc.store.apply(doc({ id: 'old', title: 'Aurora report', body: 'aurora seen tonight', published_at: daysAgo(400), updated_at: daysAgo(400) }));++ svc.store.apply(doc({ id: 'mid', title: 'Aurora report', body: 'aurora seen tonight', published_at: daysAgo(20), updated_at: daysAgo(20) }));++ svc.store.apply(doc({ id: 'new', title: 'Aurora report', body: 'aurora seen tonight', published_at: daysAgo(1), updated_at: daysAgo(1) }));++ return { svc, setClock: (c) => { clock = c; } };++}++++t('between equally relevant documents the newest ranks first', async () => {++ const { svc } = await corpus();++ try {++ const r = await request(svc.base, 'GET', '/api/v1/search?q=aurora');++ assert.deepStrictEqual(r.body.results.map(x => x.id), ['new', 'mid', 'old']);++ } finally { await svc.stop(); }++});++++t('weight 0 turns freshness off (pure bm25, ties by index order)', async () => {++ const { svc } = await corpus({ SEARCH_FRESHNESS_WEIGHT: '0' });++ try {++ const r = await request(svc.base, 'GET', '/api/v1/search?q=aurora');++ assert.deepStrictEqual(r.body.results.map(x => x.id), ['old', 'mid', 'new']);++ assert.deepStrictEqual((await request(svc.base, 'GET', '/api/ready')).body.freshness, { weight: 0, halfLifeDays: 30 });++ } finally { await svc.stop(); }++});++++t('relevance still wins: an old title match outranks a fresh passing mention', async () => {++ let clock = NOW;++ const svc = await boot({ now: () => clock });++ try {++ svc.store.apply(doc({ id: 'fresh-mention', title: 'Weekly notes', summary: 'Many topics this week.', body: `${'filler words about other things '.repeat(40)} glacier`, published_at: daysAgo(0) }));++ svc.store.apply(doc({ id: 'old-title', title: 'Glacier glacier retreat', summary: 'The glacier retreat, measured.', body: 'glacier data', published_at: daysAgo(700) }));++ const r = await request(svc.base, 'GET', '/api/v1/search?q=glacier');++ assert.deepStrictEqual(r.body.results.map(x => x.id), ['old-title', 'fresh-mention']);++ } finally { await svc.stop(); }++});++++t('pages of one query rank on the first page\'s clock: no duplicates or gaps when time moves on', async () => {++ const { svc, setClock } = await corpus();++ try {++ const p1 = await request(svc.base, 'GET', '/api/v1/search?q=aurora&limit=1');++ assert.deepStrictEqual(p1.body.results.map(x => x.id), ['new']);++ setClock(NOW + 200 * 86400000);++ const p2 = await request(svc.base, 'GET', `/api/v1/search?q=aurora&limit=1&cursor=${p1.body.next_cursor}`);++ const p3 = await request(svc.base, 'GET', `/api/v1/search?q=aurora&limit=1&cursor=${p2.body.next_cursor}`);++ assert.deepStrictEqual([...p2.body.results, ...p3.body.results].map(x => x.id), ['mid', 'old']);++ assert.strictEqual(p3.body.next_cursor, null);++ } finally { await svc.stop(); }++});++++t('browse (no q) stays newest first; suggestions prefer fresher titles', async () => {++ const { svc } = await corpus();++ try {++ const b = await request(svc.base, 'GET', '/api/v1/search');++ assert.deepStrictEqual(b.body.results.map(x => x.id), ['new', 'mid', 'old']);++ const s = await request(svc.base, 'GET', '/api/v1/suggest?q=auro');++ assert.deepStrictEqual(s.body.suggestions.map(x => x.id), ['new', 'mid', 'old']);++ } finally { await svc.stop(); }++});++++t.run();nnew file mode 100644@@@ -0,0 +1,88 @@++'use strict';++/**++ * The public HTML search page (server/web/page.js): works without JavaScript, shows only what an++ * anonymous query may see, escapes everything, never caches, and keeps result pages out of++ * search engines. Non-HTML clients still get the text route index.++ */++const assert = require('assert');++const { ids } = require('openvibe-contracts');++const { boot, request, userToken, doc, suite } = require('./helpers');++++const t = suite('page');++let svc;++const html = (p, headers = {}) => request(svc.base, 'GET', p, { headers: { Accept: 'text/html,application/xhtml+xml', ...headers } });++const alice = ids.newId('user');++++t('boot', async () => {++ svc = await boot();++ svc.store.apply(doc({ id: 'pub', title: 'Kestrel <script>alert(1)</script> nesting', summary: 'Kestrel "boxes" & more', body: 'kestrel kestrel', canonical_url: 'https://openvibe.wiki/p/kestrel?a=1&b=2' }));++ svc.store.apply(doc({ id: 'prv', visibility: 'private', acl: { subjects: [alice] }, title: 'Kestrel private diary', body: 'kestrel' }));++ svc.store.apply(doc({ id: 'drf', visibility: 'draft', title: 'Kestrel draft', body: 'kestrel' }));++ for (let i = 0; i < 25; i++) svc.store.apply(doc({ id: `many${i}`, title: `Heron sighting ${i}`, body: 'heron' }));++});++++t('the front page is a search form, indexable, no-store, with a strict CSP', async () => {++ const r = await html('/');++ assert.strictEqual(r.status, 200);++ assert.match(r.headers.get('content-type'), /text\/html/);++ assert.match(r.text, /<form method="get" action="\/" role="search">/);++ assert.ok(!/noindex/.test(r.text));++ assert.strictEqual(r.headers.get('x-robots-tag'), null);++ assert.strictEqual(r.headers.get('cache-control'), 'no-store');++ assert.match(r.headers.get('content-security-policy'), /default-src 'none'/);++ assert.ok(!/<script/i.test(r.text), 'no JavaScript on the page');++});++++t('results show public documents only, escaped, linked to their canonical URL, noindex', async () => {++ const r = await html('/?q=kestrel');++ assert.strictEqual(r.status, 200);++ assert.strictEqual(r.headers.get('x-robots-tag'), 'noindex, nofollow');++ assert.match(r.text, /<meta name="robots" content="noindex, nofollow">/);++ assert.ok(r.text.includes('Kestrel <script>alert(1)</script> nesting'));++ assert.ok(!r.text.includes('<script>alert(1)'));++ assert.ok(r.text.includes('href="https://openvibe.wiki/p/kestrel?a=1&b=2"'));++ assert.match(r.text, /<mark>Kestrel<\/mark>|<mark>kestrel<\/mark>/);++ assert.ok(!/private diary|Kestrel draft/.test(r.text));++});++++t('a signed-in cookie adds that person\'s documents; a bad cookie searches as anonymous', async () => {++ const mine = await html('/?q=kestrel', { Cookie: `ov_token=${userToken({ subjectId: alice, aud: ['openvibe.search'] })}` });++ assert.match(mine.text, /Kestrel private diary/);++ const bad = await html('/?q=kestrel', { Cookie: 'ov_token=not.a.jwt' });++ assert.strictEqual(bad.status, 200);++ assert.ok(!/private diary/.test(bad.text));++ const badBearer = await html('/?q=kestrel', { Authorization: 'Bearer nope.nope.nope' });++ assert.strictEqual(badBearer.status, 200, 'the page never answers 401');++});++++t('paging with More results, filters carried along, and an honest empty state', async () => {++ const p1 = await html('/?q=heron&owner=wiki');++ const more = /<a class="more" href="\/\?([^"]+)">/.exec(p1.text);++ assert.ok(more, 'a next-page link');++ const qs = more[1].replace(/&/g, '&');++ assert.match(qs, /owner=wiki/);++ const p2 = await html(`/?${qs}`);++ const titles = (s) => [...s.matchAll(/Heron sighting (\d+)/g)].map(m => m[1]);++ assert.strictEqual(titles(p1.text).length, 20);++ assert.strictEqual(titles(p2.text).length, 5);++ assert.ok(!titles(p2.text).some(x => titles(p1.text).includes(x)));++ const none = await html('/?q=nothingmatcheszzz');++ assert.match(none.text, /Nothing found/);++ const bad = await html('/?q=x&owner=Not%20Valid');++ assert.strictEqual(bad.status, 400);++ assert.match(bad.text, /owner is malformed/);++});++++t('non-HTML clients get the text route index; robots.txt keeps result pages out', async () => {++ const r = await request(svc.base, 'GET', '/');++ assert.match(r.headers.get('content-type'), /text\/plain/);++ assert.match(r.text, /GET {4}\/api\/v1\/search/);++ assert.match(r.text, /saved-searches/);++ const robots = await request(svc.base, 'GET', '/robots.txt');++ assert.match(robots.text, /Disallow: \/\?/);++ assert.match(robots.text, /Disallow: \/api\//);++});++++t('shutdown', async () => { await svc.stop(); });++++t.run();nnew file mode 100644@@@ -0,0 +1,237 @@++'use strict';++/**++ * The removal purge queue (server/purge.js): the owners' removal feed, and the Cloudflare purge++ * of formerly public URLs against a mocked Cloudflare API. No test talks to Cloudflare.++ */++const assert = require('assert');++const { boot, request, serviceToken, doc, suite } = require('./helpers');++const { load } = require('../server/config');++const { zoneFor, purgeUrls, MAX_ATTEMPTS } = require('../server/purge');++++const t = suite('purge');++const WIKI = serviceToken('wiki', ['search.document.write']);++const BLOG = serviceToken('blog', ['search.document.write']);++const TOKEN = 'cf-test-token-' + 'z'.repeat(24);++const WIKI_ZONE = 'a'.repeat(32);++const NET_ZONE = 'b'.repeat(32);++const ZONES = `openvibe.wiki=${WIKI_ZONE},openvibe.network=${NET_ZONE}`;++++/** A mocked Cloudflare API: records calls, answers with `respond(call)` (default: success). */++function mockCloudflare(respond = () => ({ status: 200, body: { success: true, errors: [], messages: [], result: { id: 'purge-1' } } })) {++ const calls = [];++ const fetchImpl = async (url, opts) => {++ const call = { url, method: opts.method, headers: opts.headers, body: JSON.parse(opts.body) };++ calls.push(call);++ const r = await respond(call);++ if (r instanceof Error) throw r;++ return { status: r.status, ok: r.status >= 200 && r.status < 300, json: async () => r.body, text: async () => JSON.stringify(r.body) };++ };++ return { calls, fetchImpl };++}++++const hide = (d, revision = d.revision + 1) => ({ ...d, revision, visibility: 'private', acl: { subjects: ['usr_01K5ZZZZZZZZZZZZZZZZZZZZZZ'] } });++++t('config: zone map parses host=zoneid pairs and refuses malformed ones', () => {++ const c = load({ CLOUDFLARE_ZONE_IDS: ZONES });++ assert.deepStrictEqual(c.purge.zones.map(z => z.host), ['openvibe.network', 'openvibe.wiki']);++ assert.throws(() => load({ CLOUDFLARE_ZONE_IDS: 'openvibe.wiki=nothex' }), /CLOUDFLARE_ZONE_IDS/);++ assert.throws(() => load({ CLOUDFLARE_ZONE_IDS: WIKI_ZONE }), /CLOUDFLARE_ZONE_IDS/);++ assert.strictEqual(load({}).purge.cloudflareToken, '');++ assert.deepStrictEqual(load({ CLOUDFLARE_PURGE_RELATED_PATHS: '/sitemap.xml,/feed.atom,javascript:x' }).purge.relatedPaths, ['/sitemap.xml', '/feed.atom']);++});++++t('zoneFor matches the host or a parent domain, longest first; purgeUrls adds related paths', () => {++ const zones = load({ CLOUDFLARE_ZONE_IDS: `${ZONES},docs.openvibe.network=${'c'.repeat(32)}` }).purge.zones;++ assert.strictEqual(zoneFor('https://openvibe.wiki/p/x', zones), WIKI_ZONE);++ assert.strictEqual(zoneFor('https://search.openvibe.network/x', zones), NET_ZONE);++ assert.strictEqual(zoneFor('https://docs.openvibe.network/x', zones), 'c'.repeat(32));++ assert.strictEqual(zoneFor('https://notopenvibe.wiki/x', zones), null, 'a suffix that is not a parent domain');++ assert.strictEqual(zoneFor('ftp://openvibe.wiki/x', zones), null);++ assert.deepStrictEqual(purgeUrls('https://openvibe.wiki/p/x#top', ['/sitemap.xml']), ['https://openvibe.wiki/p/x', 'https://openvibe.wiki/sitemap.xml']);++});++++t('inert without CLOUDFLARE_PURGE_TOKEN: removals are recorded for owners, nothing is queued or fetched', async () => {++ const cf = mockCloudflare();++ const svc = await boot({ env: { CLOUDFLARE_ZONE_IDS: ZONES }, fetchImpl: cf.fetchImpl });++ try {++ const d = doc({ id: 'inert1', canonical_url: 'https://openvibe.wiki/p/inert1' });++ svc.store.apply(d);++ svc.store.apply(hide(d));++ assert.deepStrictEqual(svc.purges.cdnCounts(), { pending: 0, purged: 0, failed: 0, skipped: 0 });++ assert.deepStrictEqual(await svc.purger.flush(), { purged: 0, retrying: 0, failed: 0 });++ assert.strictEqual(svc.purger.running(), false);++ assert.strictEqual(cf.calls.length, 0);++ const feed = await request(svc.base, 'GET', '/api/v1/owners/wiki/removals', { token: WIKI });++ assert.strictEqual(feed.status, 200);++ assert.strictEqual(feed.body.removals.length, 1);++ const r = feed.body.removals[0];++ assert.strictEqual(r.id, 'inert1');++ assert.strictEqual(r.reason, 'visibility_changed');++ assert.strictEqual(r.canonical_url, 'https://openvibe.wiki/p/inert1');++ assert.strictEqual(r.previous_exposure, 'public_listed');++ assert.strictEqual(r.exposure, 'restricted');++ // The removal row and the outbox event are the same removal.++ const ev = svc.outbox.all().find(e => e.event_type === 'search.document.removed' && e.payload.id === 'inert1');++ assert.strictEqual(r.event_id, ev.event_id);++ const ready = await request(svc.base, 'GET', '/api/ready');++ assert.match(JSON.stringify(ready.body), /CLOUDFLARE_PURGE_TOKEN unset/);++ } finally { await svc.stop(); }++});++++t('owner removal feed: owner token only, own owner only, cursor paging, restricted removals carry no URL', async () => {++ const svc = await boot();++ try {++ const members = doc({ id: 'mem1', visibility: 'members', acl: { groups: ['g1'] } });++ svc.store.apply(members);++ svc.store.remove('wiki', 'page', 'mem1');++ const pub = doc({ id: 'pub2', canonical_url: 'https://openvibe.wiki/p/pub2' });++ svc.store.apply(pub);++ svc.store.remove('wiki', 'page', 'pub2');++ assert.strictEqual((await request(svc.base, 'GET', '/api/v1/owners/wiki/removals')).status, 401);++ assert.strictEqual((await request(svc.base, 'GET', '/api/v1/owners/wiki/removals', { token: BLOG })).status, 403);++ const p1 = await request(svc.base, 'GET', '/api/v1/owners/wiki/removals?limit=1', { token: WIKI });++ assert.deepStrictEqual(p1.body.removals.map(r => [r.id, r.reason, r.canonical_url]), [['mem1', 'deleted', null]]);++ const p2 = await request(svc.base, 'GET', `/api/v1/owners/wiki/removals?after=${p1.body.next_after}`, { token: WIKI });++ assert.deepStrictEqual(p2.body.removals.map(r => [r.id, r.reason, r.canonical_url]), [['pub2', 'deleted', 'https://openvibe.wiki/p/pub2']]);++ const blog = await request(svc.base, 'GET', '/api/v1/owners/blog/removals', { token: BLOG });++ assert.deepStrictEqual(blog.body.removals, []);++ } finally { await svc.stop(); }++});++++t('with a token: the formerly public URL and its sitemap are purged in one call per zone', async () => {++ const cf = mockCloudflare();++ const svc = await boot({ env: { CLOUDFLARE_PURGE_TOKEN: TOKEN, CLOUDFLARE_ZONE_IDS: ZONES }, fetchImpl: cf.fetchImpl });++ try {++ const a = doc({ id: 'cf1', canonical_url: 'https://openvibe.wiki/p/cf1' });++ const b = doc({ id: 'cf2', canonical_url: 'https://openvibe.wiki/p/cf2' });++ const c = doc({ owner: 'wiki', id: 'cf3', canonical_url: 'https://docs.openvibe.network/cf3' });++ for (const d of [a, b, c]) svc.store.apply(d);++ svc.store.apply(hide(a));++ svc.store.remove('wiki', 'page', 'cf2');++ svc.store.apply({ ...c, revision: 2, publication_state: 'retracted' });++ // Two removals on openvibe.wiki share one pending sitemap purge.++ assert.deepStrictEqual(svc.purges.cdnCounts(), { pending: 5, purged: 0, failed: 0, skipped: 0 });++++ const s = await svc.purger.flush();++ assert.deepStrictEqual(s, { purged: 5, retrying: 0, failed: 0 });++ assert.strictEqual(cf.calls.length, 2);++ const wiki = cf.calls.find(x => x.url.includes(WIKI_ZONE));++ assert.strictEqual(wiki.url, `https://api.cloudflare.com/client/v4/zones/${WIKI_ZONE}/purge_cache`);++ assert.strictEqual(wiki.method, 'POST');++ assert.strictEqual(wiki.headers.Authorization, `Bearer ${TOKEN}`);++ assert.deepStrictEqual(wiki.body.files.sort(), ['https://openvibe.wiki/p/cf1', 'https://openvibe.wiki/p/cf2', 'https://openvibe.wiki/sitemap.xml']);++ const net = cf.calls.find(x => x.url.includes(NET_ZONE));++ assert.deepStrictEqual(net.body.files.sort(), ['https://docs.openvibe.network/cf3', 'https://docs.openvibe.network/sitemap.xml']);++++ // Done rows are never sent again; the token is never stored.++ assert.deepStrictEqual(await svc.purger.flush(), { purged: 0, retrying: 0, failed: 0 });++ assert.strictEqual(cf.calls.length, 2);++ const dump = JSON.stringify(svc.db.prepare('SELECT * FROM cdn_purges').all()) + JSON.stringify(svc.db.prepare('SELECT * FROM removals').all());++ assert.ok(!dump.includes(TOKEN));++ assert.strictEqual(svc.purger.running(), true);++ } finally { await svc.stop(); }++});++++t('an index change that does not lower exposure, or a URL that was never public, purges nothing', async () => {++ const cf = mockCloudflare();++ const svc = await boot({ env: { CLOUDFLARE_PURGE_TOKEN: TOKEN, CLOUDFLARE_ZONE_IDS: ZONES }, fetchImpl: cf.fetchImpl });++ try {++ const d = doc({ id: 'keep', canonical_url: 'https://openvibe.wiki/p/keep' });++ svc.store.apply(d);++ svc.store.apply({ ...d, revision: 2, title: 'New title' });++ const m = doc({ id: 'mem', visibility: 'members', acl: { groups: ['g'] }, canonical_url: 'https://openvibe.wiki/p/mem' });++ svc.store.apply(m);++ svc.store.remove('wiki', 'page', 'mem');++ assert.deepStrictEqual(svc.purges.cdnCounts(), { pending: 0, purged: 0, failed: 0, skipped: 0 });++ await svc.purger.flush();++ assert.strictEqual(cf.calls.length, 0);++ } finally { await svc.stop(); }++});++++t('a host with no configured zone is recorded as skipped, not sent', async () => {++ const cf = mockCloudflare();++ const svc = await boot({ env: { CLOUDFLARE_PURGE_TOKEN: TOKEN, CLOUDFLARE_ZONE_IDS: ZONES }, fetchImpl: cf.fetchImpl });++ try {++ const d = doc({ id: 'elsewhere', canonical_url: 'https://example.org/x' });++ svc.store.apply(d);++ svc.store.remove('wiki', 'page', 'elsewhere');++ assert.deepStrictEqual(svc.purges.cdnCounts(), { pending: 0, purged: 0, failed: 0, skipped: 2 });++ await svc.purger.flush();++ assert.strictEqual(cf.calls.length, 0);++ const row = svc.db.prepare("SELECT detail FROM cdn_purges WHERE url = 'https://example.org/x'").get();++ assert.match(row.detail, /no Cloudflare zone/);++ } finally { await svc.stop(); }++});++++t('429, 5xx and network errors back off and retry; after MAX_ATTEMPTS the purge fails', async () => {++ let clock = Date.parse('2026-09-23T12:00:00Z');++ let mode = 'rate';++ const cf = mockCloudflare(() => {++ if (mode === 'rate') return { status: 429, body: { success: false, errors: [{ code: 971, message: 'Please wait and consider throttling your request speed' }] } };++ if (mode === 'down') return new Error('ECONNRESET');++ return { status: 200, body: { success: true, errors: [], result: { id: 'ok' } } };++ });++ const svc = await boot({ env: { CLOUDFLARE_PURGE_TOKEN: TOKEN, CLOUDFLARE_ZONE_IDS: ZONES, CLOUDFLARE_PURGE_RELATED_PATHS: 'none' }, fetchImpl: cf.fetchImpl, now: () => clock });++ try {++ const d = doc({ id: 'retry1', canonical_url: 'https://openvibe.wiki/p/retry1' });++ svc.store.apply(d);++ svc.store.remove('wiki', 'page', 'retry1');++ const n = svc.purges.cdnCounts().pending;++ assert.strictEqual(n, 1, 'related paths off (none)');++ assert.deepStrictEqual(await svc.purger.flush(), { purged: 0, retrying: n, failed: 0 });++ // Not due yet: nothing is sent before the backoff elapses.++ assert.deepStrictEqual(await svc.purger.flush(), { purged: 0, retrying: 0, failed: 0 });++ assert.strictEqual(cf.calls.length, 1);++ clock += 3000;++ mode = 'ok';++ assert.deepStrictEqual(await svc.purger.flush(), { purged: n, retrying: 0, failed: 0 });++++ // A URL that never gets through fails after MAX_ATTEMPTS.++ mode = 'down';++ const e = doc({ id: 'retry2', canonical_url: 'https://openvibe.wiki/p/retry2' });++ svc.store.apply(e);++ svc.store.remove('wiki', 'page', 'retry2');++ for (let i = 0; i < MAX_ATTEMPTS; i++) { clock += 2 * 3600 * 1000; await svc.purger.flush(); }++ const rows = svc.db.prepare("SELECT state, attempts, detail FROM cdn_purges WHERE url LIKE '%retry2'").all();++ assert.deepStrictEqual(rows.map(r => [r.state, r.attempts]), [['failed', MAX_ATTEMPTS]]);++ assert.match(rows[0].detail, /gave up after/);++ } finally { await svc.stop(); }++});++++t('a refused batch is retried one URL at a time: only the refused URL fails', async () => {++ const cf = mockCloudflare((call) => (call.body.files.some(f => f.includes('refused'))++ ? { status: 400, body: { success: false, errors: [{ code: 1012, message: 'Request must contain one of "purge_everything", "files", "tags", "hosts" or "prefixes"' }] } }++ : { status: 200, body: { success: true, errors: [], result: { id: 'ok' } } }));++ const svc = await boot({ env: { CLOUDFLARE_PURGE_TOKEN: TOKEN, CLOUDFLARE_ZONE_IDS: ZONES, CLOUDFLARE_PURGE_RELATED_PATHS: 'none' }, fetchImpl: cf.fetchImpl });++ try {++ for (const id of ['good1', 'refused', 'good2']) {++ svc.store.apply(doc({ id, canonical_url: `https://openvibe.wiki/p/${id}` }));++ svc.store.remove('wiki', 'page', id);++ }++ const s = await svc.purger.flush();++ assert.deepStrictEqual(s, { purged: 2, retrying: 0, failed: 1 });++ const failed = svc.db.prepare("SELECT url, detail FROM cdn_purges WHERE state = 'failed'").all();++ assert.deepStrictEqual(failed.map(r => r.url), ['https://openvibe.wiki/p/refused']);++ assert.match(failed[0].detail, /HTTP 400 1012/);++ } finally { await svc.stop(); }++});++++t('a token Cloudflare rejects (403) fails the purge without retrying forever and never logs the token', async () => {++ const logged = [];++ const cf = mockCloudflare(() => ({ status: 403, body: { success: false, errors: [{ code: 10000, message: `Authentication error for ${TOKEN}` }] } }));++ const svc = await boot({ env: { CLOUDFLARE_PURGE_TOKEN: TOKEN, CLOUDFLARE_ZONE_IDS: ZONES, CLOUDFLARE_PURGE_RELATED_PATHS: 'none' }, fetchImpl: cf.fetchImpl });++ svc.purger.stop();++ const { createPurger } = require('../server/purge');++ const purger = createPurger({ db: svc.db, config: svc.config.purge, fetchImpl: cf.fetchImpl, log: { warn: (m) => logged.push(m), log() {}, error() {} } });++ try {++ svc.store.apply(doc({ id: 'auth1', canonical_url: 'https://openvibe.wiki/p/auth1' }));++ svc.store.remove('wiki', 'page', 'auth1');++ assert.deepStrictEqual(await purger.flush(), { purged: 0, retrying: 0, failed: 1 });++ const row = svc.db.prepare("SELECT state, detail FROM cdn_purges WHERE url LIKE '%auth1'").get();++ assert.strictEqual(row.state, 'failed');++ assert.ok(!row.detail.includes(TOKEN) && row.detail.includes('[token]'));++ assert.ok(logged.length && logged.every(m => !m.includes(TOKEN)));++ } finally { await svc.stop(); }++});++++t.run();nnew file mode 100644@@@ -0,0 +1,153 @@++'use strict';++/**++ * Saved searches: only a signed-in person (usr_ subject) saves; each person sees only their own;++ * a run is a fresh query as that person (the ACL applies at run time); cookie changes must come++ * from this origin; the per-person limit holds.++ */++const assert = require('assert');++const { ids } = require('openvibe-contracts');++const { boot, request, serviceToken, userToken, doc, suite } = require('./helpers');++++const t = suite('saved-searches');++const ORIGIN = 'https://search.openvibe.network';++const DELEGATE = serviceToken('blog', ['search.query.delegate']);++const NO_DELEGATE = serviceToken('blog', ['search.document.write']);++++const alice = ids.newId('user');++const bob = ids.newId('user');++const guest = ids.newId('guest');++const as = (subjectId, role = 'user') => ({ token: userToken({ subjectId, role, aud: ['openvibe.search'] }) });++++let svc;++const call = (method, p, opts = {}) => request(svc.base, method, p, opts);++++t('boot with a small per-person limit', async () => {++ svc = await boot({ env: { BASE_URL: ORIGIN, SEARCH_SAVED_MAX_PER_SUBJECT: '3' } });++ svc.store.apply(doc({ id: 'pub', title: 'Zebra migration public', canonical_url: 'https://openvibe.wiki/p/pub', facets: { category: 'animals' } }));++ svc.store.apply(doc({ id: 'mem', visibility: 'members', acl: { subjects: [alice] }, title: 'Zebra migration for alice', facets: { category: 'animals' } }));++ svc.store.apply(doc({ owner: 'blog', type: 'post', id: 'other', title: 'Zebra blog post', canonical_url: 'https://openvibe.blog/p/other' }));++});++++t('anonymous callers, guests and services without delegation cannot use saved searches', async () => {++ assert.strictEqual((await call('GET', '/api/v1/saved-searches')).status, 401);++ assert.strictEqual((await call('POST', '/api/v1/saved-searches', { body: { q: 'zebra' } })).status, 401);++ const g = await call('POST', '/api/v1/saved-searches', { token: DELEGATE, headers: { 'X-OV-Subject': guest }, body: { q: 'zebra' } });++ assert.strictEqual(g.status, 403);++ assert.strictEqual(g.body.code || g.body.type.split('/').pop(), 'search.sign_in_required');++ assert.strictEqual((await call('GET', '/api/v1/saved-searches', { token: DELEGATE })).status, 401, 'a service with no subject is nobody');++ assert.strictEqual((await call('GET', '/api/v1/saved-searches', { token: NO_DELEGATE, headers: { 'X-OV-Subject': alice } })).status, 403);++});++++let saved;++t('a person saves a query with filters; the same query again is the same saved search', async () => {++ const r = await call('POST', '/api/v1/saved-searches', { ...as(alice), body: { name: 'Zebras', q: ' zebra migration ', owner: 'wiki', facets: { category: ['animals'] } } });++ assert.strictEqual(r.status, 201, r.text);++ saved = r.body.saved_search;++ assert.match(saved.id, /^svs_[0-9a-z]{26}$/);++ assert.strictEqual(saved.name, 'Zebras');++ assert.strictEqual(saved.q, 'zebra migration');++ assert.deepStrictEqual(saved.filters, { owner: 'wiki', facets: [['category', ['animals']]] });++ assert.strictEqual(saved.last_run_at, null);++ assert.ok(!('subject' in saved));++++ const again = await call('POST', '/api/v1/saved-searches', { ...as(alice), body: { name: 'Zebras (renamed)', q: 'zebra migration', facets: { category: 'animals' }, owner: 'wiki' } });++ assert.strictEqual(again.status, 200);++ assert.strictEqual(again.body.saved_search.id, saved.id);++ assert.strictEqual(again.body.saved_search.name, 'Zebras (renamed)');++++ const list = await call('GET', '/api/v1/saved-searches', as(alice));++ assert.strictEqual(list.status, 200);++ assert.deepStrictEqual(list.body.saved_searches.map(s => s.id), [saved.id]);++ assert.strictEqual(list.body.max, 3);++ assert.strictEqual(list.headers.get('cache-control'), 'no-store');++});++++t('running a saved search queries as the person now, with its filters', async () => {++ const r = await call('GET', `/api/v1/saved-searches/${saved.id}/results`, as(alice));++ assert.strictEqual(r.status, 200, r.text);++ assert.deepStrictEqual(r.body.results.map(x => x.id).sort(), ['mem', 'pub'], 'owner=wiki filter applies; alice sees her members document');++ assert.strictEqual(r.body.saved_search.id, saved.id);++ const after = await call('GET', `/api/v1/saved-searches/${saved.id}`, as(alice));++ assert.ok(after.body.saved_search.last_run_at);++++ // Access is decided at run time: alice loses the members document, it leaves her results.++ svc.store.apply(doc({ id: 'mem', revision: 2, visibility: 'members', acl: { subjects: [bob] }, title: 'Zebra migration for alice', facets: { category: 'animals' } }));++ const r2 = await call('GET', `/api/v1/saved-searches/${saved.id}/results`, as(alice));++ assert.deepStrictEqual(r2.body.results.map(x => x.id), ['pub']);++ // And a deleted public document leaves it too.++ svc.store.remove('wiki', 'page', 'pub');++ const r3 = await call('GET', `/api/v1/saved-searches/${saved.id}/results`, as(alice));++ assert.deepStrictEqual(r3.body.results, []);++});++++t("another person's saved search is the same 404 as a missing one, for read, run and delete", async () => {++ for (const [m, p] of [['GET', `/api/v1/saved-searches/${saved.id}`], ['GET', `/api/v1/saved-searches/${saved.id}/results`], ['DELETE', `/api/v1/saved-searches/${saved.id}`]]) {++ const theirs = await call(m, p, as(bob));++ const missing = await call(m, p.replace(saved.id, 'svs_00000000000000000000000000'), as(bob));++ assert.strictEqual(theirs.status, 404, `${m} ${p}`);++ assert.strictEqual(missing.status, 404);++ assert.strictEqual(theirs.body.detail, missing.body.detail);++ }++ assert.deepStrictEqual((await call('GET', '/api/v1/saved-searches', as(bob))).body.saved_searches, []);++ assert.strictEqual((await call('GET', '/api/v1/saved-searches/not-an-id', as(alice))).status, 404);++});++++t('bad saved searches are refused: nothing to search, malformed filters, oversize name', async () => {++ const bad = async (body) => (await call('POST', '/api/v1/saved-searches', { ...as(alice), body })).status;++ assert.strictEqual(await bad({}), 400);++ assert.strictEqual(await bad({ q: ' ' }), 400);++ assert.strictEqual(await bad({ q: '!!! ???' }), 400);++ assert.strictEqual(await bad({ q: 'x', owner: 'Not Valid' }), 400);++ assert.strictEqual(await bad({ q: 'x', facets: { 'Bad Key': ['v'] } }), 400);++ assert.strictEqual(await bad({ q: 'x', facets: ['v'] }), 400);++ assert.strictEqual(await bad({ q: 'x', facets: { k: [1] } }), 400);++ assert.strictEqual(await bad({ q: 'x', name: 'n'.repeat(101) }), 400);++ assert.strictEqual(await bad({ q: 5 }), 400);++ assert.strictEqual(await bad({ q: 'x'.repeat(501) }), 400);++ // Filters alone are a valid saved search (newest in a product).++ const r = await call('POST', '/api/v1/saved-searches', { ...as(alice), body: { owner: 'blog' } });++ assert.strictEqual(r.status, 201);++ assert.strictEqual(r.body.saved_search.name, 'All documents');++ const run = await call('GET', `/api/v1/saved-searches/${r.body.saved_search.id}/results`, as(alice));++ assert.deepStrictEqual(run.body.results.map(x => x.id), ['other']);++});++++t('a cookie-authenticated change must come from this origin; Bearer and delegated calls need no Origin', async () => {++ const cookie = { headers: { Cookie: `ov_token=${as(alice).token}` } };++ const noOrigin = await call('POST', '/api/v1/saved-searches', { ...cookie, body: { q: 'cookie one' } });++ assert.strictEqual(noOrigin.status, 403);++ const foreign = await call('POST', '/api/v1/saved-searches', { headers: { ...cookie.headers, Origin: 'https://evil.example' }, body: { q: 'cookie one' } });++ assert.strictEqual(foreign.status, 403);++ const sibling = await call('POST', '/api/v1/saved-searches', { headers: { ...cookie.headers, Origin: 'https://openvibe.wiki' }, body: { q: 'cookie one' } });++ assert.strictEqual(sibling.status, 403);++ const same = await call('POST', '/api/v1/saved-searches', { headers: { ...cookie.headers, Origin: ORIGIN }, body: { q: 'cookie one' } });++ assert.strictEqual(same.status, 201);++ // Reading with the cookie needs no Origin.++ assert.strictEqual((await call('GET', '/api/v1/saved-searches', cookie)).status, 200);++ const del = await call('DELETE', `/api/v1/saved-searches/${same.body.saved_search.id}`, cookie);++ assert.strictEqual(del.status, 403);++ assert.strictEqual((await call('DELETE', `/api/v1/saved-searches/${same.body.saved_search.id}`, { headers: { ...cookie.headers, Origin: ORIGIN } })).status, 204);++++ // A product saving for its signed-in visitor.++ const d = await call('POST', '/api/v1/saved-searches', { token: DELEGATE, headers: { 'X-OV-Subject': bob }, body: { q: 'zebra' } });++ assert.strictEqual(d.status, 201);++ assert.deepStrictEqual((await call('GET', '/api/v1/saved-searches', as(bob))).body.saved_searches.map(s => s.id), [d.body.saved_search.id]);++});++++t('the per-person limit holds, and deleting frees a slot', async () => {++ // alice has 2 (zebra, owner=blog); the limit is 3.++ assert.strictEqual((await call('POST', '/api/v1/saved-searches', { ...as(alice), body: { q: 'third' } })).status, 201);++ const over = await call('POST', '/api/v1/saved-searches', { ...as(alice), body: { q: 'fourth' } });++ assert.strictEqual(over.status, 409);++ // Saving a query already saved is not a new one, even at the limit.++ assert.strictEqual((await call('POST', '/api/v1/saved-searches', { ...as(alice), body: { q: 'third' } })).status, 200);++ assert.strictEqual((await call('DELETE', `/api/v1/saved-searches/${saved.id}`, as(alice))).status, 204);++ assert.strictEqual((await call('GET', `/api/v1/saved-searches/${saved.id}`, as(alice))).status, 404);++ assert.strictEqual((await call('POST', '/api/v1/saved-searches', { ...as(alice), body: { q: 'fourth' } })).status, 201);++ const ready = await call('GET', '/api/ready');++ assert.strictEqual(ready.body.saved_searches, 4);++});++++t('shutdown', async () => { await svc.stop(); });++++t.run();
@@ -42,3 +42,23 @@ SEARCH_DB_PATH=./data/search.db
@@ -42,3 +42,23 @@ SEARCH_DB_PATH=./data/search.db
# Largest accepted request body (direct PUT and deliveries), bytes.
# Largest accepted request body (direct PUT and deliveries), bytes.
#SEARCH_MAX_BODY_BYTES=262144
#SEARCH_MAX_BODY_BYTES=262144
+
+# Ranking: freshness weighting of full-text relevance (README "Ranking").
+# score = bm25 × (1 + WEIGHT × 2^(−age_days / HALF_LIFE_DAYS)); age from published_at, else
+# updated_at. WEIGHT 0 turns it off.
+#SEARCH_FRESHNESS_WEIGHT=1
+#SEARCH_FRESHNESS_HALF_LIFE_DAYS=30
+
+# Saved searches per signed-in person.
+#SEARCH_SAVED_MAX_PER_SUBJECT=50
+
+# Removal purge queue (README "Removals, caches and the CDN"). Every removal is recorded for
+# its owner (GET /api/v1/owners/:owner/removals). The Cloudflare purge of a formerly public URL
+# runs only when CLOUDFLARE_PURGE_TOKEN is set (inert otherwise). The token needs only
+# Zone → Cache Purge → Purge on the zones below. Never commit it: /etc/openvibe/search.env (0600).
+#CLOUDFLARE_PURGE_TOKEN=
+# host=zone id pairs; a URL uses the zone of its host or nearest parent domain.
+#CLOUDFLARE_ZONE_IDS=openvibe.wiki=<zone id>,openvibe.blog=<zone id>,openvibe.network=<zone id>
+# Paths on the removed URL's host purged with it (default /sitemap.xml; `none` for none).
+#CLOUDFLARE_PURGE_RELATED_PATHS=/sitemap.xml,/feed.atom,/feed.json
+#CLOUDFLARE_PURGE_INTERVAL_MS=5000
@@ -6,8 +6,9 @@
@@ -6,8 +6,9 @@
**Status:** alpha (roadmap Wave 14). Deployed internally, not launched: it runs on the production host
**Status:** alpha (roadmap Wave 14). Deployed internally, not launched: it runs on the production host
(127.0.0.1:4710 only, since 2026-09-23) and is fed through Events, but the index holds only 10 wiki
(127.0.0.1:4710 only, since 2026-09-23) and is fed through Events, but the index holds only 10 wiki
tombstones and 0 public documents.
tombstones and 0 public documents.
-**Domain:** `search.openvibe.network` (no vhost installed yet: the name currently falls through to the
-admin.openvibe.network placeholder; it goes public only when the launch rule below holds)
+**Domain:** `search.openvibe.network` (vhost in [deploy/nginx/](deploy/nginx/search.openvibe.network.conf),
+not installed yet: until it is, the name falls through to the admin.openvibe.network placeholder.
+Install steps: [Public host](#public-host))
**Plan:** OpenVibe End-to-End Realignment & Implementation Plan, revision 3 — roadmap §4.2 A, §15.12, §29, §32.3.
**Plan:** OpenVibe End-to-End Realignment & Implementation Plan, revision 3 — roadmap §4.2 A, §15.12, §29, §32.3.
**License:** AGPL-3.0.
**License:** AGPL-3.0.
@@ -31,7 +32,8 @@ npm test # every test/*.test.js, temp databases, no network needed
@@ -31,7 +32,8 @@ npm test # every test/*.test.js, temp databases, no network needed
Node 22 in production (`fnm exec --using=22.22.1 npm test`). Production: `/opt/openvibe.search`,
Node 22 in production (`fnm exec --using=22.22.1 npm test`). Production: `/opt/openvibe.search`,
env `/etc/openvibe/search.env`, unit [deploy/systemd/openvibe-search.service](deploy/systemd/openvibe-search.service),
env `/etc/openvibe/search.env`, unit [deploy/systemd/openvibe-search.service](deploy/systemd/openvibe-search.service),
index `/var/lib/openvibe-search/search.db`, nginx [deploy/nginx/search.openvibe.network.conf](deploy/nginx/search.openvibe.network.conf)
index `/var/lib/openvibe-search/search.db`, nginx [deploy/nginx/search.openvibe.network.conf](deploy/nginx/search.openvibe.network.conf)
-(the public vhost exposes only the GET query API and health; owners and Events call `127.0.0.1:4710`).
+(the public vhost exposes the search page, the GET query API, saved searches and health; the owner
+API and the Events webhook are loopback-only, and owners and Events call `127.0.0.1:4710`).
`GET /api/health` is liveness. `GET /api/ready` (openvibe-shared/ready) is 503 only when the
`GET /api/health` is liveness. `GET /api/ready` (openvibe-shared/ready) is 503 only when the
database (the documents and both full-text tables) fails; a Network signing key that has not loaded
database (the documents and both full-text tables) fails; a Network signing key that has not loaded
@@ -71,6 +73,50 @@ anonymous); delegation headers from a browser are ignored. Visibility is enforce
@@ -71,6 +73,50 @@ anonymous); delegation headers from a browser are ignored. Visibility is enforce
SQL (two FTS tables split by audience plus an ACL predicate) and re-checked per hit — see
SQL (two FTS tables split by audience plus an ACL predicate) and re-checked per hit — see
[docs/adr-engine.md](docs/adr-engine.md).
[docs/adr-engine.md](docs/adr-engine.md).
+The search page at `/` runs the same query as HTML for browsers (no JavaScript; no-store; result
+pages `noindex`; anonymous unless this host has an `ov_token` cookie). Other clients get a text
+route index at `/`.
+
+### Ranking
+
+Full-text results are ordered by bm25 relevance (title ×10, summary ×4, body ×1) multiplied by a
+freshness boost:
+
+```
+score = bm25 × (1 + W × 2^(−age_days / H)) W = SEARCH_FRESHNESS_WEIGHT (default 1)
+ H = SEARCH_FRESHNESS_HALF_LIFE_DAYS (default 30)
+```
+
+`age_days` counts from `published_at`, else `updated_at` (an edit does not make an old document
+new); a future date counts as today and a missing one gets no boost. With the defaults a document
+published today counts ×2.0, 30 days old ×1.5, 60 days ×1.25, 90 days ×1.125 and a year old ×1.0002:
+relevance still decides, and between comparably relevant documents the newer one ranks first.
+`W = 0` turns it off. Suggestions use the same score; browsing without `q` stays newest first. A
+cursor carries its first page's clock, so later pages rank on the same scores
+(`test/freshness.test.js`).
+
+### Saved searches
+
+```http
+GET /api/v1/saved-searches
+POST /api/v1/saved-searches { "name": "Eclipses", "q": "solar eclipse", "owner": "news", "facets": { "topic": ["space"] } }
+GET /api/v1/saved-searches/:id
+DELETE /api/v1/saved-searches/:id
+GET /api/v1/saved-searches/:id/results?limit=&cursor=
+```
+
+A signed-in person (a Network user token with a `usr_` subject, as Bearer or the `ov_token`
+cookie) or a first-party service with `search.query.delegate` acting for `X-OV-Subject: usr_…`
+keeps up to `SEARCH_SAVED_MAX_PER_SUBJECT` (50) queries: words, filters or both. Saving the same
+query again returns the existing one (renamed if a new name is given). Only the query is stored:
+every run is a fresh query as that person at that moment, so what they lost access to (or what was
+deleted) is gone and what they gained appears. Someone else's saved search is the same 404 as a
+missing one; guests cannot save; a cookie-authenticated POST/DELETE must carry this origin's
+`Origin`. Notifications of new matches wait for OpenVibe.Network's notifications (`last_run_at` is
+kept for that job). Signing in on `search.openvibe.network` itself needs a Network OAuth client
+(`search`), which does not exist yet: today saved searches are used through Bearer tokens or a
+product acting for its visitor.
+
## Indexing
## Indexing
Owners call with an OpenVibe.Network client-credentials token (`audience=openvibe.search`) holding
Owners call with an OpenVibe.Network client-credentials token (`audience=openvibe.search`) holding
@@ -105,17 +151,40 @@ private/members/unlisted, or public → noindex) leaves results in the same tran
@@ -105,17 +151,40 @@ private/members/unlisted, or public → noindex) leaves results in the same tran
purge); every accepted upsert emits `search.document.indexed`. Both go through a transactional
purge); every accepted upsert emits `search.document.indexed`. Both go through a transactional
outbox relayed to Events when `EVENTS_URL` is set.
outbox relayed to Events when `EVENTS_URL` is set.
+### Removals, caches and the CDN
+
+Search consumes its own removals ([server/purge.js](server/purge.js)); in the same transaction as
+the `search.document.removed` event it records:
+
+- **the owner's removal feed** — `GET /api/v1/owners/:owner/removals?after=&limit=` (owner token,
+ own owner only): `seq, event_id, type, id, revision, reason, previous_exposure, exposure,
+ canonical_url, at`. Products own their pages, caches and sitemaps, so they poll this feed (or
+ subscribe to `search.document.removed`) and drop the document from anything derived they keep.
+- **a Cloudflare purge queue** — only when `CLOUDFLARE_PURGE_TOKEN` is set (inert otherwise:
+ nothing is queued or sent). For a removal whose URL had been public, the canonical URL and the
+ `CLOUDFLARE_PURGE_RELATED_PATHS` on its host (default `/sitemap.xml`) are queued once each and
+ purged by URL (`POST /zones/:zone/purge_cache {"files": […]}`, at most 30 per call) in the zone
+ `CLOUDFLARE_ZONE_IDS` maps the host (or its nearest parent domain) to. A host with no zone is
+ recorded as `skipped`. 429, 5xx and network errors back off and retry (8 attempts); a refusal is
+ retried one URL at a time and then marked `failed`. The token needs only *Zone → Cache Purge →
+ Purge* on those zones; it is read from the environment, sent only in the Authorization header,
+ and never stored or logged. `/api/ready` reports the purge state and counts, `/metrics` the
+ `search_cdn_purges{state}` gauge (`test/purge.test.js`, against a mocked Cloudflare API).
+
## Owns
## Owns
- the index document contract (`search.index-document@1`, released in openvibe-contracts v0.12.0; proposal in [docs/contracts-proposal/](docs/contracts-proposal/))
- the index document contract (`search.index-document@1`, released in openvibe-contracts v0.12.0; proposal in [docs/contracts-proposal/](docs/contracts-proposal/))
- `documents`, `doc_acl`, `doc_facets`, `fts_public`, `fts_restricted`, `idempotency_receipts`,
- `documents`, `doc_acl`, `doc_facets`, `fts_public`, `fts_restricted`, `idempotency_receipts`,
- `ingest_rejections`, `event_outbox` (SQLite; engine decision in [docs/adr-engine.md](docs/adr-engine.md))
-- the query API and its ACL filtering; deletion and visibility-change propagation
+ `ingest_rejections`, `event_outbox`, `removals`, `cdn_purges`, `saved_searches` (SQLite; engine
+ decision in [docs/adr-engine.md](docs/adr-engine.md))
+- the query API, its ranking and ACL filtering, the search page and saved searches; deletion and
+ visibility-change propagation, including the CDN purge of removed public URLs
## Does not own
## Does not own
- any product's content, publication state or indexability decision (owners and the publishing packages do)
- any product's content, publication state or indexability decision (owners and the publishing packages do)
-- sitemaps, feeds or page caches (each product; they listen to `search.document.removed`)
+- sitemaps, feeds or page caches (each product; they read the removal feed or listen to
+ `search.document.removed`). Search only purges the CDN copy of a removed public URL.
- identity, groups or entitlements (Network, products, VIP/Billing — Search only matches the keys it is given)
- identity, groups or entitlements (Network, products, VIP/Billing — Search only matches the keys it is given)
## Depends on
## Depends on
@@ -128,8 +197,8 @@ outbox relayed to Events when `EVENTS_URL` is set.
@@ -128,8 +197,8 @@ outbox relayed to Events when `EVENTS_URL` is set.
| Capability | Routes |
| Capability | Routes |
|---|---|
|---|---|
-| `search.document.write` | `PUT/DELETE /api/v1/documents/:owner/:type/:id`, `/api/v1/owners/:owner/...` (own owner only) |
-| `search.query.delegate` | the query routes, acting for `X-OV-Subject` |
+| `search.document.write` | `PUT/DELETE /api/v1/documents/:owner/:type/:id`, `/api/v1/owners/:owner/...` including the removal feed (own owner only) |
+| `search.query.delegate` | the query and saved-search routes, acting for `X-OV-Subject` |
Released in `openvibe-contracts` v0.12.0 with the service manifest (this repo pins v0.13.0);
Released in `openvibe-contracts` v0.12.0 with the service manifest (this repo pins v0.13.0);
[server/auth.js](server/auth.js) decides them with the contracts grant rule, and CI runs
[server/auth.js](server/auth.js) decides them with the contracts grant rule, and CI runs
@@ -146,12 +215,20 @@ Released in `openvibe-contracts` v0.12.0 with the service manifest (this repo pi
@@ -146,12 +215,20 @@ Released in `openvibe-contracts` v0.12.0 with the service manifest (this repo pi
owner spoofing refused (`test/webhook-inbox.test.js`)
owner spoofing refused (`test/webhook-inbox.test.js`)
- outbox relay to Events, retry and poison isolation (`test/outbox.test.js`)
- outbox relay to Events, retry and poison isolation (`test/outbox.test.js`)
- proposals valid against the contracts schemas (`test/proposals.test.js`)
- proposals valid against the contracts schemas (`test/proposals.test.js`)
+- removals reach the owner feed; the CDN purge is inert without a token and, with one, purges the
+ formerly public URL and its sitemap per zone, retries, isolates refusals and never stores the
+ token (`test/purge.test.js`)
+- freshness decay, newer-first between equals, relevance over age, stable cursors (`test/freshness.test.js`)
+- saved searches: signed-in only, own only, run-time ACL, Origin check for cookies, limits
+ (`test/saved-searches.test.js`)
+- the search page: public only, escaped, no-store, noindex results, paging (`test/page.test.js`)
Not yet demonstrated: a document with content indexed end to end in production. The Events
Not yet demonstrated: a document with content indexed end to end in production. The Events
subscription delivered 20 wiki events; the 10 upserts were rejected (`owner_not_accepted`, since
subscription delivered 20 wiki events; the 10 upserts were rejected (`owner_not_accepted`, since
fixed) and only the 10 tombstones landed. Wiki pages enter the index once a person reviews them;
fixed) and only the 10 tombstones landed. Wiki pages enter the index once a person reviews them;
-Sources has every seed disabled. Nobody consumes `search.document.removed` yet (no cache, sitemap
-or CDN purge).
+Sources has every seed disabled. The removal feed and the CDN purge queue exist, but no product
+polls the feed yet and the purge needs a token with Cache Purge permission (the current Cloudflare
+token has DNS and rulesets scope only).
Restore drill: `ovhost drill search` passed on the production host on 2026-09-23 (integrity check,
Restore drill: `ovhost drill search` passed on the production host on 2026-09-23 (integrity check,
readiness, identical query answers, row counts; see OpenVibe.Host `docs/restore-drills.md`).
readiness, identical query answers, row counts; see OpenVibe.Host `docs/restore-drills.md`).
@@ -161,10 +238,37 @@ readiness, identical query answers, row counts; see OpenVibe.Host `docs/restore-
@@ -161,10 +238,37 @@ readiness, identical query answers, row counts; see OpenVibe.Host `docs/restore-
This repository does not make the product real, and the domain keeps its placeholder page on
This repository does not make the product real, and the domain keeps its placeholder page on
[OpenVibers/OpenVibe.Sites](https://github.com/OpenVibers/OpenVibe.Sites) until: an owning runtime
[OpenVibers/OpenVibe.Sites](https://github.com/OpenVibers/OpenVibe.Sites) until: an owning runtime
with health/readiness; Network identity and service principals; a public route useful without
with health/readiness; Network identity and service principals; a public route useful without
-JavaScript (there is no HTML search page yet — only the JSON API and a text index at `/`); real
+JavaScript (the search page at `/`); real
persistence and end-to-end workflows; capabilities and events registered in OpenVibe.Contracts; a
persistence and end-to-end workflows; capabilities and events registered in OpenVibe.Contracts; a
-security review; and acceptance tests. The launch release removes the domain from
-`OpenVibe.Sites/sites.json` atomically.
+security review; and acceptance tests. The domain is not in `OpenVibe.Sites/sites.json`; this
+repository's own vhost serves it.
+
+## Public host
+
+`search.openvibe.network` resolves through Cloudflare, but until this vhost is installed nginx
+answers it with its default site (the admin placeholder). The vhost
+([deploy/nginx/search.openvibe.network.conf](deploy/nginx/search.openvibe.network.conf)) uses the
+Network wildcard certificate, sets the client-address headers from `$remote_addr` only, answers
+`/metrics` with 404, keeps `/internal/`, `/api/v1/owners/` and document writes loopback-only, and
+answers every unknown path with a JSON 404. Install on the host, after the service runs a release
+that has the search page and saved searches (`ovhost deploy search`):
+
+```bash
+sudo install -m 0644 /opt/openvibe.search/deploy/nginx/search.openvibe.network.conf /etc/nginx/sites-available/search.openvibe.network.conf
+sudo ln -sf /etc/nginx/sites-available/search.openvibe.network.conf /etc/nginx/sites-enabled/search.openvibe.network.conf
+sudo nginx -t && sudo systemctl reload nginx
+# check
+curl -sS https://search.openvibe.network/api/health
+curl -sS -H 'Accept: text/html' https://search.openvibe.network/ | grep -o '<title>[^<]*'
+curl -sS -o /dev/null -w '%{http_code}\n' https://search.openvibe.network/metrics # 404
+curl -sS -o /dev/null -w '%{http_code}\n' -X POST https://search.openvibe.network/internal/events # 403
+curl -sS -o /dev/null -w '%{http_code}\n' https://search.openvibe.network/api/v1/owners/wiki/removals # 403
+```
+
+For the CDN purge, create a Cloudflare API token with only *Zone → Cache Purge → Purge* on the
+product zones, then add `CLOUDFLARE_PURGE_TOKEN` and `CLOUDFLARE_ZONE_IDS` to
+`/etc/openvibe/search.env` (0600) and restart `openvibe-search`; `/api/ready` then shows
+`purge.cdn: "cloudflare"`.
---
---
@@ -6,7 +6,7 @@
@@ -6,7 +6,7 @@
"plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026)",
"plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026)",
"runtime": "node 22, express, better-sqlite3 (FTS5)",
"runtime": "node 22, express, better-sqlite3 (FTS5)",
"port": 4710,
"port": 4710,
- "domain": "search.openvibe.network (no vhost installed: the name falls through to the admin.openvibe.network placeholder; the service is reachable on the host only)",
+ "domain": "search.openvibe.network (vhost in deploy/nginx, not installed: until it is, the name falls through to the admin.openvibe.network placeholder; README \"Public host\")",
"contracts": "openvibe-contracts v0.13.0",
"contracts": "openvibe-contracts v0.13.0",
"capabilities": ["search.document.write", "search.query.delegate"],
"capabilities": ["search.document.write", "search.query.delegate"],
"capabilitiesRegistered": true,
"capabilitiesRegistered": true,
@@ -15,5 +15,6 @@
@@ -15,5 +15,6 @@
"deployed": true,
"deployed": true,
"deployment": "Deployed internally, not launched: /opt/openvibe.search at 5a7e59b, unit openvibe-search, 127.0.0.1:4710 only, principal search, Events subscription *.index_document.*, accepted event owners sources, wiki, blog, news, reviews, trade, deals, coupons. Production index: 10 documents, all tombstones (wiki), 0 public. Restore drill passed 2026-09-23.",
"deployment": "Deployed internally, not launched: /opt/openvibe.search at 5a7e59b, unit openvibe-search, 127.0.0.1:4710 only, principal search, Events subscription *.index_document.*, accepted event owners sources, wiki, blog, news, reviews, trade, deals, coupons. Production index: 10 documents, all tombstones (wiki), 0 public. Restore drill passed 2026-09-23.",
"indexedProducers": ["wiki (tombstones only so far)"],
"indexedProducers": ["wiki (tombstones only so far)"],
+ "features": ["ACL-filtered FTS5 query API", "facets", "suggestions", "freshness-weighted ranking", "saved searches (notifications wait for Network notifications)", "server-rendered search page", "owner removal feed", "Cloudflare purge queue (inert until CLOUDFLARE_PURGE_TOKEN is set)"],
"updated": "2026-09-23"
"updated": "2026-09-23"
}
}
@@ -2,22 +2,26 @@
@@ -2,22 +2,26 @@
## search.openvibe.network — Nginx Configuration (REFERENCE)
## search.openvibe.network — Nginx Configuration (REFERENCE)
##
##
## Deployed to /etc/nginx/sites-available/search.openvibe.network.conf — keep them in sync,
## Deployed to /etc/nginx/sites-available/search.openvibe.network.conf — keep them in sync,
-## and reload nginx after changing it.
+## and reload nginx after changing it. Install: README "Public host".
##
##
## The Node service (port 4710, systemd unit openvibe-search) serves:
## The Node service (port 4710, systemd unit openvibe-search) serves:
+## / the search page (HTML for browsers, a text route index
+## otherwise) and /robots.txt. Public, no-store, result pages noindex.
## /api/v1/search, /api/v1/suggest, GET /api/v1/documents/... public query API (anonymous =
## /api/v1/search, /api/v1/suggest, GET /api/v1/documents/... public query API (anonymous =
## public documents only; the ov_token cookie or a Bearer adds
## public documents only; the ov_token cookie or a Bearer adds
## the caller's ACL matches). Responses are Cache-Control: no-store.
## the caller's ACL matches). Responses are Cache-Control: no-store.
-## /api/v1/documents (PUT/DELETE), /api/v1/owners/... owner API (service tokens). Services on
-## this host call 127.0.0.1:4710, so the public vhost refuses
-## writes; open them only for an off-host owner.
-## /internal/events OpenVibe.Events deliveries — host-local only.
-## /api/health, /api/ready
+## /api/v1/saved-searches[/...] a signed-in person's saved searches (Network user token)
+## /api/v1/documents (PUT/DELETE), /api/v1/owners/... owner API (service tokens): loopback only.
+## Services on this host call 127.0.0.1:4710 directly.
+## /internal/events OpenVibe.Events deliveries: loopback only.
+## /api/health, /api/ready, /release.json
## /metrics Prometheus text, answered only to direct loopback callers
## /metrics Prometheus text, answered only to direct loopback callers
## (never proxied; Node refuses a request carrying X-Forwarded-For)
## (never proxied; Node refuses a request carrying X-Forwarded-For)
+## Everything else is a JSON 404 from nginx, so the name never falls through to another site.
## ═══════════════════════════════════════════════════════════════
## ═══════════════════════════════════════════════════════════════
limit_req_zone $binary_remote_addr zone=ovsearch_query:10m rate=10r/s;
limit_req_zone $binary_remote_addr zone=ovsearch_query:10m rate=10r/s;
+limit_req_zone $binary_remote_addr zone=ovsearch_saved:10m rate=2r/s;
# ── HTTP → HTTPS redirect ──────────────────────────────────
# ── HTTP → HTTPS redirect ──────────────────────────────────
server {
server {
@@ -50,42 +54,82 @@ server {
@@ -50,42 +54,82 @@ server {
add_header Referrer-Policy strict-origin-when-cross-origin always;
add_header Referrer-Policy strict-origin-when-cross-origin always;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
- location = /api/health { proxy_pass http://127.0.0.1:4710; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $remote_addr; proxy_set_header CF-Connecting-IP $remote_addr; }
- location = /api/ready { proxy_pass http://127.0.0.1:4710; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $remote_addr; proxy_set_header CF-Connecting-IP $remote_addr; }
+ # Client address only from $remote_addr (realip has already replaced it with the visitor's
+ # address for Cloudflare's ranges); never append to or pass through a client-sent header.
+ # Inherited by every location below that proxies (none sets its own).
+ proxy_http_version 1.1;
+ proxy_set_header Connection "";
+ proxy_set_header Host $host;
+ proxy_set_header X-Real-IP $remote_addr;
+ proxy_set_header X-Forwarded-For $remote_addr;
+ proxy_set_header CF-Connecting-IP $remote_addr;
+ proxy_set_header X-Forwarded-Proto $scheme;
+
+ location = /api/health { proxy_pass http://127.0.0.1:4710; }
+ location = /api/ready { proxy_pass http://127.0.0.1:4710; }
+ location = /release.json { proxy_pass http://127.0.0.1:4710; }
# Metrics are for host-local scrapers only (http://127.0.0.1:4710/metrics).
# Metrics are for host-local scrapers only (http://127.0.0.1:4710/metrics).
location = /metrics { return 404; }
location = /metrics { return 404; }
+ # ── Search page ─────────────────────────────────────────
+ location = / {
+ limit_except GET { deny all; }
+ limit_req zone=ovsearch_query burst=40 nodelay;
+ proxy_pass http://127.0.0.1:4710;
+ }
+ location = /robots.txt { proxy_pass http://127.0.0.1:4710; }
+
# ── Public query API (GET only) ─────────────────────────
# ── Public query API (GET only) ─────────────────────────
- location ~ ^/api/v1/(search|suggest|documents/) {
+ location ~ ^/api/v1/(search|suggest)$ {
limit_except GET { deny all; }
limit_except GET { deny all; }
limit_req zone=ovsearch_query burst=40 nodelay;
limit_req zone=ovsearch_query burst=40 nodelay;
proxy_pass http://127.0.0.1:4710;
proxy_pass http://127.0.0.1:4710;
- proxy_http_version 1.1;
- proxy_set_header Connection "";
- proxy_set_header Host $host;
- proxy_set_header X-Real-IP $remote_addr;
- proxy_set_header X-Forwarded-For $remote_addr;
- proxy_set_header CF-Connecting-IP $remote_addr;
- proxy_set_header X-Forwarded-Proto $scheme;
# Never let a shared cache hold a personalised answer.
# Never let a shared cache hold a personalised answer.
proxy_no_cache 1;
proxy_no_cache 1;
proxy_cache_bypass 1;
proxy_cache_bypass 1;
}
}
- # ── Owner API and the Events webhook: host-local only ───
- location /api/ { return 403; }
- location /internal/ { return 403; }
+ # One document by id: GET is public (ACL-checked by Search); PUT/DELETE are owner writes,
+ # loopback only.
+ location ^~ /api/v1/documents/ {
+ limit_except GET {
+ allow 127.0.0.1;
+ allow ::1;
+ deny all;
+ }
+ limit_req zone=ovsearch_query burst=40 nodelay;
+ proxy_pass http://127.0.0.1:4710;
+ proxy_no_cache 1;
+ proxy_cache_bypass 1;
+ }
+
+ # ── Saved searches (a Network user token; Search checks Origin for cookie changes) ──
+ location ~ ^/api/v1/saved-searches(/|$) {
+ limit_except GET POST DELETE { deny all; }
+ limit_req zone=ovsearch_saved burst=20 nodelay;
+ proxy_pass http://127.0.0.1:4710;
+ proxy_no_cache 1;
+ proxy_cache_bypass 1;
+ }
- location = / {
+ # ── Loopback only: owner API and the Events webhook ─────
+ location ^~ /api/v1/owners/ {
+ allow 127.0.0.1;
+ allow ::1;
+ deny all;
+ proxy_pass http://127.0.0.1:4710;
+ }
+ location ^~ /internal/ {
+ allow 127.0.0.1;
+ allow ::1;
+ deny all;
proxy_pass http://127.0.0.1:4710;
proxy_pass http://127.0.0.1:4710;
- proxy_set_header Host $host;
- proxy_set_header X-Real-IP $remote_addr;
- proxy_set_header X-Forwarded-For $remote_addr;
- proxy_set_header CF-Connecting-IP $remote_addr;
}
}
+ # ── Anything else: an honest JSON 404, never another site ──
location / {
location / {
- return 404;
+ default_type application/problem+json;
+ return 404 '{"type":"about:blank","title":"Not Found","status":404,"code":"search.not_found","detail":"No such route on search.openvibe.network. The search page is at /, the public query API at /api/v1/search; see https://github.com/OpenVibers/OpenVibe.Search"}';
}
}
}
}
@@ -9,6 +9,9 @@
@@ -9,6 +9,9 @@
* reconciliation: (type, id, revision, deleted, hash)
* reconciliation: (type, id, revision, deleted, hash)
* GET /api/v1/owners/:owner/documents/:type/:id the stored document + effective indexability
* GET /api/v1/owners/:owner/documents/:type/:id the stored document + effective indexability
* GET /api/v1/owners/:owner/rejections?after= index-document events Search refused
* GET /api/v1/owners/:owner/rejections?after= index-document events Search refused
+ * GET /api/v1/owners/:owner/removals?after=&limit= documents that left results or public
+ * delivery (server/purge.js): the owner drops
+ * them from its caches and sitemaps
*/
*/
const express = require('express');
const express = require('express');
const { http } = require('openvibe-contracts');
const { http } = require('openvibe-contracts');
@@ -39,7 +42,7 @@ function outcomeResponse(res, ctx, r) {
@@ -39,7 +42,7 @@ function outcomeResponse(res, ctx, r) {
return res.json(r);
return res.json(r);
}
}
-function documentsRouter({ store, auth, db, relay }) {
+function documentsRouter({ store, auth, db, relay, purges }) {
const router = express.Router();
const router = express.Router();
const guard = auth.requireCap(CAPS.write);
const guard = auth.requireCap(CAPS.write);
@@ -125,6 +128,13 @@ function documentsRouter({ store, auth, db, relay }) {
@@ -125,6 +128,13 @@ function documentsRouter({ store, auth, db, relay }) {
});
});
});
});
+ router.get('/api/v1/owners/:owner/removals', guard, (req, res) => {
+ if (!ownerCheck(req, res)) return;
+ const after = /^\d{1,15}$/.test(String(req.query.after || '')) ? Number(req.query.after) : 0;
+ const limit = Math.min(Math.max(parseInt(req.query.limit, 10) || 200, 1), 1000);
+ res.json({ owner: req.params.owner, ...purges.ownerFeed(req.params.owner, { after, limit }) });
+ });
+
return router;
return router;
}
}
@@ -69,7 +69,8 @@ function decodeCursor(s, hash) {
@@ -69,7 +69,8 @@ function decodeCursor(s, hash) {
let c;
let c;
try { c = JSON.parse(Buffer.from(String(s), 'base64url').toString('utf8')); } catch { c = null; }
try { c = JSON.parse(Buffer.from(String(s), 'base64url').toString('utf8')); } catch { c = null; }
if (!c || c.h !== hash || !Number.isInteger(c.i)) throw new QueryError('search.bad_cursor', 'cursor does not belong to this query');
if (!c || c.h !== hash || !Number.isInteger(c.i)) throw new QueryError('search.bad_cursor', 'cursor does not belong to this query');
- if (c.k === 'r' && typeof c.r === 'number') return { rank: c.r, rid: c.i };
+ // n: the first page's reference time for freshness, so later pages rank on the same clock.
+ if (c.k === 'r' && typeof c.r === 'number') return { rank: c.r, rid: c.i, now: Number.isFinite(c.n) ? c.n : null };
if (c.k === 't' && typeof c.t === 'string') return { sort_at: c.t, rid: c.i };
if (c.k === 't' && typeof c.t === 'string') return { sort_at: c.t, rid: c.i };
throw new QueryError('search.bad_cursor', 'cursor is malformed');
throw new QueryError('search.bad_cursor', 'cursor is malformed');
}
}
@@ -111,51 +112,31 @@ function canView(viewer, row, doc) {
@@ -111,51 +112,31 @@ function canView(viewer, row, doc) {
return false;
return false;
}
}
-function queryRouter({ config, store, engine, auth }) {
- const router = express.Router();
-
- function withViewer(handler) {
- return (req, res) => {
- const ctx = req.ov;
- let viewer;
- try {
- viewer = auth.viewer(req);
- } catch (err) {
- if (err instanceof AuthError) return http.sendProblem(res, err.status, err.code, { detail: err.message, ctx });
- throw err;
- }
- // Never cached: a visibility change or deletion must leave results immediately, and a
- // personalised answer must never reach a shared cache.
- res.setHeader('Cache-Control', 'no-store');
- res.setHeader('Vary', 'Authorization, Cookie, X-OV-Subject, X-OV-Groups, X-OV-Entitlements');
- try {
- return handler(req, res, viewer);
- } catch (err) {
- if (err instanceof QueryError) return http.sendProblem(res, 400, err.code, { detail: err.message, ctx });
- throw err;
- }
- };
- }
-
- router.get('/api/v1/search', withViewer((req, res, viewer) => {
- const q = one(req.query.q);
- const text = q === undefined ? '' : String(q);
+/**
+ * One search as a viewer: the query API, saved-search runs and the HTML page all go through here.
+ * run({ text, query, viewer, limit, cursor, facetKeys, now }) → { results, next_cursor, facets? }
+ * `query` is the query-string-shaped filter source (owner, type, lang, facet.<key>). Throws
+ * QueryError for a malformed request.
+ */
+function createSearcher({ config, store, engine, now: clock = () => Date.now() }) {
+ function run({ text = '', query = {}, filters = null, viewer, limit, cursor, facetKeys = [], now = clock() }) {
if (text.length > 500) throw new QueryError('search.bad_query', 'q is longer than 500 characters');
if (text.length > 500) throw new QueryError('search.bad_query', 'q is longer than 500 characters');
- const filters = parseFilters(req.query, config);
- const limit = Math.min(Math.max(parseInt(one(req.query.limit), 10) || config.query.defaultLimit, 1), config.query.maxLimit);
- const facetKeys = String(one(req.query.facets) || '').split(',').map(s => s.trim()).filter(Boolean);
+ filters = filters || parseFilters(query, config);
+ limit = Math.min(Math.max(parseInt(limit, 10) || config.query.defaultLimit, 1), config.query.maxLimit);
if (facetKeys.length > 5 || facetKeys.some(k => !FACET_KEY_RE.test(k))) throw new QueryError('search.bad_query', 'facets: at most 5 well-formed keys');
if (facetKeys.length > 5 || facetKeys.some(k => !FACET_KEY_RE.test(k))) throw new QueryError('search.bad_query', 'facets: at most 5 well-formed keys');
const match = text.trim() ? toMatch(text, { maxTerms: config.query.maxTerms }) : null;
const match = text.trim() ? toMatch(text, { maxTerms: config.query.maxTerms }) : null;
const hash = queryHash([match, filters]);
const hash = queryHash([match, filters]);
- const after = decodeCursor(one(req.query.cursor), hash);
+ const after = decodeCursor(cursor, hash);
if (text.trim() && !match) {
if (text.trim() && !match) {
- return res.json({ results: [], next_cursor: null, ...(facetKeys.length ? { facets: Object.fromEntries(facetKeys.map(k => [k, []])) } : {}) });
+ return { results: [], next_cursor: null, ...(facetKeys.length ? { facets: Object.fromEntries(facetKeys.map(k => [k, []])) } : {}) };
}
}
let page;
let page;
+ let refNow = now;
if (match) {
if (match) {
if (after && after.rank === undefined) throw new QueryError('search.bad_cursor', 'cursor does not belong to this query');
if (after && after.rank === undefined) throw new QueryError('search.bad_cursor', 'cursor does not belong to this query');
- page = engine.search({ match, filters, viewer, limit, after });
+ if (after && after.now != null) refNow = after.now;
+ page = engine.search({ match, filters, viewer, limit, after, now: refNow });
} else {
} else {
if (after && after.sort_at === undefined) throw new QueryError('search.bad_cursor', 'cursor does not belong to this query');
if (after && after.sort_at === undefined) throw new QueryError('search.bad_cursor', 'cursor does not belong to this query');
page = engine.browse({ filters, viewer, limit, after });
page = engine.browse({ filters, viewer, limit, after });
@@ -168,14 +149,61 @@ function queryRouter({ config, store, engine, auth }) {
@@ -168,14 +149,61 @@ function queryRouter({ config, store, engine, auth }) {
results.push(projection(found.doc, found.row, match ? h.snip : undefined));
results.push(projection(found.doc, found.row, match ? h.snip : undefined));
}
}
const next = page.next
const next = page.next
- ? encodeCursor(match ? { k: 'r', r: page.next.rank, i: page.next.rid, h: hash } : { k: 't', t: page.next.sort_at, i: page.next.rid, h: hash })
+ ? encodeCursor(match ? { k: 'r', r: page.next.rank, i: page.next.rid, n: refNow, h: hash } : { k: 't', t: page.next.sort_at, i: page.next.rid, h: hash })
: null;
: null;
const body = { results, next_cursor: next };
const body = { results, next_cursor: next };
if (facetKeys.length) body.facets = engine.facetCounts({ match, filters, viewer, keys: facetKeys });
if (facetKeys.length) body.facets = engine.facetCounts({ match, filters, viewer, keys: facetKeys });
- res.json(body);
+ return body;
+ }
+
+ return {
+ run,
+ parseFilters: (query) => parseFilters(query, config),
+ hasWords: (text) => toMatch(text, { maxTerms: config.query.maxTerms }) !== null,
+ };
+}
+
+/** Wraps a handler: resolves the viewer (AuthError → problem), no-store, QueryError → 400. */
+function withViewer(auth, handler) {
+ return (req, res, next) => {
+ const ctx = req.ov;
+ let viewer;
+ try {
+ viewer = auth.viewer(req);
+ } catch (err) {
+ if (err instanceof AuthError) return http.sendProblem(res, err.status, err.code, { detail: err.message, ctx });
+ return next(err);
+ }
+ // Never cached: a visibility change or deletion must leave results immediately, and a
+ // personalised answer must never reach a shared cache.
+ res.setHeader('Cache-Control', 'no-store');
+ res.setHeader('Vary', 'Authorization, Cookie, X-OV-Subject, X-OV-Groups, X-OV-Entitlements');
+ try {
+ return handler(req, res, viewer);
+ } catch (err) {
+ if (err instanceof QueryError) return http.sendProblem(res, 400, err.code, { detail: err.message, ctx });
+ return next(err);
+ }
+ };
+}
+
+function queryRouter({ config, store, engine, auth, searcher = createSearcher({ config, store, engine }) }) {
+ const router = express.Router();
+ const guarded = (h) => withViewer(auth, h);
+
+ router.get('/api/v1/search', guarded((req, res, viewer) => {
+ const q = one(req.query.q);
+ res.json(searcher.run({
+ text: q === undefined ? '' : String(q),
+ query: req.query,
+ viewer,
+ limit: one(req.query.limit),
+ cursor: one(req.query.cursor),
+ facetKeys: String(one(req.query.facets) || '').split(',').map(s => s.trim()).filter(Boolean),
+ }));
}));
}));
- router.get('/api/v1/suggest', withViewer((req, res, viewer) => {
+ router.get('/api/v1/suggest', guarded((req, res, viewer) => {
const text = String(one(req.query.q) || '');
const text = String(one(req.query.q) || '');
if (text.length > 100) throw new QueryError('search.bad_query', 'q is longer than 100 characters');
if (text.length > 100) throw new QueryError('search.bad_query', 'q is longer than 100 characters');
const filters = parseFilters(req.query, config);
const filters = parseFilters(req.query, config);
@@ -192,7 +220,7 @@ function queryRouter({ config, store, engine, auth }) {
@@ -192,7 +220,7 @@ function queryRouter({ config, store, engine, auth }) {
res.json({ suggestions });
res.json({ suggestions });
}));
}));
- router.get('/api/v1/documents/:owner/:type/:id', withViewer((req, res, viewer) => {
+ router.get('/api/v1/documents/:owner/:type/:id', guarded((req, res, viewer) => {
const { owner, type, id } = req.params;
const { owner, type, id } = req.params;
if (!OWNER_RE.test(owner) || !TYPE_RE.test(type) || !ID_RE.test(id)) throw new QueryError('search.bad_identity', 'owner, type or id is malformed');
if (!OWNER_RE.test(owner) || !TYPE_RE.test(type) || !ID_RE.test(id)) throw new QueryError('search.bad_identity', 'owner, type or id is malformed');
const found = store.get(owner, type, id);
const found = store.get(owner, type, id);
@@ -205,4 +233,4 @@ function queryRouter({ config, store, engine, auth }) {
@@ -205,4 +233,4 @@ function queryRouter({ config, store, engine, auth }) {
return router;
return router;
}
}
-module.exports = { queryRouter, canView, projection, toMatch };
+module.exports = { queryRouter, createSearcher, withViewer, parseFilters, QueryError, canView, projection, toMatch, one };
new file mode 100644
new file mode 100644
@@ -0,0 +1,128 @@
@@ -0,0 +1,128 @@
+'use strict';
+/**
+ * Saved searches of the signed-in person (server/saved.js).
+ *
+ * GET /api/v1/saved-searches the caller's saved searches, newest first
+ * POST /api/v1/saved-searches { name?, q?, owner?, type?, lang?, facets?: { key: [values] } }
+ * 201 new, 200 when the same query was already saved
+ * GET /api/v1/saved-searches/:id one
+ * DELETE /api/v1/saved-searches/:id 204
+ * GET /api/v1/saved-searches/:id/results?limit=&cursor=
+ * runs it now, as the caller (the ACL applies at
+ * run time, never at save time)
+ *
+ * Who: a Network user JWT carrying a usr_ subject_id (Bearer, or the ov_token cookie), or a
+ * first-party service with search.query.delegate acting for X-OV-Subject: usr_… (a product saving
+ * for its signed-in visitor). Guests and anonymous callers cannot save. A saved search of another
+ * person and a missing one give the same 404. A cookie-authenticated POST/DELETE must come from
+ * this origin (Origin header), so another site cannot save or delete in someone's name.
+ */
+const express = require('express');
+const { http, ids } = require('openvibe-contracts');
+const { withViewer, QueryError, one } = require('./query');
+
+const NAME_MAX = 100;
+
+function savedRouter({ config, saved, searcher, auth }) {
+ const router = express.Router();
+ const selfOrigin = (() => { try { return new URL(config.baseUrl).origin; } catch { return null; } })();
+
+ /** The viewer's usr_ subject, or a problem sent (null). */
+ function person(req, res, viewer, { write = false } = {}) {
+ const ctx = req.ov;
+ if (!viewer.subject) {
+ http.sendProblem(res, 401, 'token.missing', { detail: 'sign in to use saved searches', ctx });
+ return null;
+ }
+ if (!ids.isSubjectId('user', viewer.subject)) {
+ http.sendProblem(res, 403, 'search.sign_in_required', { detail: 'saved searches belong to signed-in people (usr_ subjects), not guests', ctx });
+ return null;
+ }
+ if (write && viewer.via === 'cookie') {
+ const origin = req.get('origin');
+ if (!origin || origin !== selfOrigin) {
+ http.sendProblem(res, 403, 'search.cross_origin', { detail: 'a cookie-authenticated change must come from this origin', ctx });
+ return null;
+ }
+ }
+ return viewer.subject;
+ }
+
+ const guarded = (h) => withViewer(auth, h);
+
+ router.get('/api/v1/saved-searches', guarded((req, res, viewer) => {
+ const subject = person(req, res, viewer);
+ if (!subject) return;
+ res.json({ saved_searches: saved.list(subject), max: saved.maxPerSubject });
+ }));
+
+ router.post('/api/v1/saved-searches', guarded((req, res, viewer) => {
+ const ctx = req.ov;
+ const subject = person(req, res, viewer, { write: true });
+ if (!subject) return;
+ const b = req.body && typeof req.body === 'object' && !Array.isArray(req.body) ? req.body : null;
+ if (!b) return http.sendProblem(res, 400, 'search.bad_saved_search', { detail: 'body must be a JSON object', ctx });
+ const q = b.q == null ? '' : b.q;
+ const name = b.name == null ? '' : b.name;
+ if (typeof q !== 'string' || q.length > 500) throw new QueryError('search.bad_query', 'q must be a string of at most 500 characters');
+ if (typeof name !== 'string' || name.trim().length > NAME_MAX) throw new QueryError('search.bad_saved_search', `name must be a string of at most ${NAME_MAX} characters`);
+ // The filters take the same shape and checks as the query string of /api/v1/search.
+ const query = {};
+ for (const k of ['owner', 'type', 'lang']) {
+ if (b[k] == null) continue;
+ if (typeof b[k] !== 'string') throw new QueryError('search.bad_query', `${k} must be a string`);
+ query[k] = b[k];
+ }
+ if (b.facets != null) {
+ if (typeof b.facets !== 'object' || Array.isArray(b.facets)) throw new QueryError('search.bad_query', 'facets must be an object of key → values');
+ for (const [k, v] of Object.entries(b.facets)) {
+ const values = Array.isArray(v) ? v : [v];
+ if (values.some(x => typeof x !== 'string')) throw new QueryError('search.bad_query', `facets.${k} values must be strings`);
+ query[`facet.${k}`] = values;
+ }
+ }
+ const filters = searcher.parseFilters(query);
+ const hasFilter = filters.owner || filters.type || filters.language || filters.facets.length;
+ const text = q.trim();
+ if (text && !searcher.hasWords(text)) throw new QueryError('search.bad_query', 'q has no searchable words');
+ if (!text && !hasFilter) throw new QueryError('search.bad_saved_search', 'a saved search needs words (q) or at least one filter');
+ const r = saved.save(subject, { name: name.trim(), q: text, filters });
+ if (r.error === 'limit') return http.sendProblem(res, 409, 'search.saved_search_limit', { detail: `at most ${saved.maxPerSubject} saved searches per person`, ctx });
+ res.status(r.created ? 201 : 200).json({ saved_search: r.saved });
+ }));
+
+ router.get('/api/v1/saved-searches/:id', guarded((req, res, viewer) => {
+ const subject = person(req, res, viewer);
+ if (!subject) return;
+ const s = saved.get(subject, req.params.id);
+ if (!s) return http.sendProblem(res, 404, 'search.not_found', { detail: 'no such saved search', ctx: req.ov });
+ res.json({ saved_search: s });
+ }));
+
+ router.delete('/api/v1/saved-searches/:id', guarded((req, res, viewer) => {
+ const subject = person(req, res, viewer, { write: true });
+ if (!subject) return;
+ if (!saved.remove(subject, req.params.id)) return http.sendProblem(res, 404, 'search.not_found', { detail: 'no such saved search', ctx: req.ov });
+ res.status(204).end();
+ }));
+
+ router.get('/api/v1/saved-searches/:id/results', guarded((req, res, viewer) => {
+ const subject = person(req, res, viewer);
+ if (!subject) return;
+ const s = saved.get(subject, req.params.id);
+ if (!s) return http.sendProblem(res, 404, 'search.not_found', { detail: 'no such saved search', ctx: req.ov });
+ const body = searcher.run({
+ text: s.q,
+ filters: { ...s.filters, facets: s.filters.facets || [] },
+ viewer,
+ limit: one(req.query.limit),
+ cursor: one(req.query.cursor),
+ });
+ saved.markRun(s.id);
+ res.json({ saved_search: { id: s.id, name: s.name }, ...body });
+ }));
+
+ return router;
+}
+
+module.exports = { savedRouter };
@@ -7,11 +7,13 @@ const { instrument } = require('openvibe-shared/metrics');
@@ -7,11 +7,13 @@ const { instrument } = require('openvibe-shared/metrics');
const { createRelease } = require('openvibe-shared/release');
const { createRelease } = require('openvibe-shared/release');
const { createSearchReadiness, registerSearchGauges } = require('./observability');
const { createSearchReadiness, registerSearchGauges } = require('./observability');
const { documentsRouter } = require('./api/documents');
const { documentsRouter } = require('./api/documents');
-const { queryRouter } = require('./api/query');
+const { queryRouter, createSearcher } = require('./api/query');
+const { savedRouter } = require('./api/saved');
+const { pageRouter } = require('./web/page');
const { webhookRouter } = require('./api/webhook');
const { webhookRouter } = require('./api/webhook');
const pkg = require('../package.json');
const pkg = require('../package.json');
-function createApp({ config, db, store, engine, auth, keys, outbox, relay, log = console, now }) {
+function createApp({ config, db, store, engine, auth, keys, outbox, relay, purges, purger, saved, log = console, now }) {
const app = express();
const app = express();
app.disable('x-powered-by');
app.disable('x-powered-by');
app.set('trust proxy', 'loopback');
app.set('trust proxy', 'loopback');
@@ -20,7 +22,7 @@ function createApp({ config, db, store, engine, auth, keys, outbox, relay, log =
@@ -20,7 +22,7 @@ function createApp({ config, db, store, engine, auth, keys, outbox, relay, log =
// HTTP golden signals by route template, process metrics, release_info and the Search gauges;
// HTTP golden signals by route template, process metrics, release_info and the Search gauges;
// GET /metrics answers direct loopback callers only (Track O).
// GET /metrics answers direct loopback callers only (Track O).
const metrics = instrument(app, { service: 'search', release: release.release });
const metrics = instrument(app, { service: 'search', release: release.release });
- registerSearchGauges(metrics.registry, { db, outbox });
+ registerSearchGauges(metrics.registry, { db, outbox, purges });
app.use(http.middleware());
app.use(http.middleware());
app.use((req, res, next) => {
app.use((req, res, next) => {
res.setHeader('X-Content-Type-Options', 'nosniff');
res.setHeader('X-Content-Type-Options', 'nosniff');
@@ -38,30 +40,16 @@ function createApp({ config, db, store, engine, auth, keys, outbox, relay, log =
@@ -38,30 +40,16 @@ function createApp({ config, db, store, engine, auth, keys, outbox, relay, log =
// Readiness (openvibe-shared/ready): 503 only when the database (documents and full-text tables)
// Readiness (openvibe-shared/ready): 503 only when the database (documents and full-text tables)
// fails; the Network key and index consistency are optional and degrade it (see observability.js).
// fails; the Network key and index consistency are optional and degrade it (see observability.js).
- const readiness = createSearchReadiness({ db, keys, config, engine, store, outbox, relay, release: release.release });
+ const readiness = createSearchReadiness({ db, keys, config, engine, store, outbox, relay, purges, purger, saved, release: release.release });
app.get('/api/ready', readiness.handler);
app.get('/api/ready', readiness.handler);
app.get('/release.json', release.handler);
app.get('/release.json', release.handler);
- app.use(documentsRouter({ store, auth, db, relay }));
- app.use(queryRouter({ config, store, engine, auth }));
-
- app.get('/', (_req, res) => {
- res.type('text/plain').send([
- 'OpenVibe.Search: permission-aware search over documents the network\'s services index.',
- '',
- 'GET /api/v1/search?q=&owner=&type=&facet.<key>=&cursor= query (anonymous: public only)',
- 'GET /api/v1/suggest?q= title suggestions',
- 'GET /api/v1/documents/:owner/:type/:id one document the caller may see',
- 'PUT /api/v1/documents/:owner/:type/:id index (owner, search.document.write)',
- 'DELETE /api/v1/documents/:owner/:type/:id?revision= tombstone (owner)',
- 'GET /api/v1/owners/:owner/documents reconciliation (owner)',
- 'POST /internal/events OpenVibe.Events delivery (signed)',
- 'GET /api/health, /api/ready, /release.json',
- '',
- 'Source: https://github.com/OpenVibers/OpenVibe.Search',
- '',
- ].join('\n'));
- });
+ const searcher = createSearcher({ config, store, engine, now });
+ app.use(documentsRouter({ store, auth, db, relay, purges }));
+ app.use(queryRouter({ config, store, engine, auth, searcher }));
+ app.use(savedRouter({ config, saved, searcher, auth }));
+ // GET / (HTML search page for browsers, the text route index otherwise) and /robots.txt.
+ app.use(pageRouter({ searcher, auth }));
app.use((req, res) => http.sendProblem(res, 404, 'search.not_found', { detail: `no route ${req.method} ${req.path}`, ctx: req.ov }));
app.use((req, res) => http.sendProblem(res, 404, 'search.not_found', { detail: `no route ${req.method} ${req.path}`, ctx: req.ov }));
@@ -249,7 +249,10 @@ function createAuth({ config, keys }) {
@@ -249,7 +249,10 @@ function createAuth({ config, keys }) {
}
}
const fromCookie = cookie(req, 'ov_token');
const fromCookie = cookie(req, 'ov_token');
const user = fromCookie ? verifyUser(fromCookie) : null;
const user = fromCookie ? verifyUser(fromCookie) : null;
- return user ? userViewer(user) : ANONYMOUS;
+ if (!user) return ANONYMOUS;
+ const v = userViewer(user);
+ // `via` lets state-changing routes demand a same-origin request for ambient credentials.
+ return v.kind === 'user' ? { ...v, via: 'cookie' } : v;
}
}
return { verifyService, verifyUser, requireCap, viewer };
return { verifyService, verifyUser, requireCap, viewer };
@@ -12,6 +12,30 @@ const int = (v, d) => {
@@ -12,6 +12,30 @@ const int = (v, d) => {
};
};
const list = (v, d) => (v == null || v === '' ? d : String(v).split(',').map(s => s.trim()).filter(Boolean));
const list = (v, d) => (v == null || v === '' ? d : String(v).split(',').map(s => s.trim()).filter(Boolean));
const strip = (v) => String(v || '').replace(/\/$/, '');
const strip = (v) => String(v || '').replace(/\/$/, '');
+const num = (v, d) => {
+ const n = parseFloat(v);
+ return Number.isFinite(n) ? n : d;
+};
+
+const ZONE_ID_RE = /^[0-9a-f]{32}$/;
+const HOST_RE = /^(?=.{1,253}$)([a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\.)+[a-z]{2,63}$/;
+
+/**
+ * CLOUDFLARE_ZONE_IDS: `host=zoneid` pairs, comma-separated (openvibe.wiki=<32 hex>,...). A URL
+ * belongs to the zone whose host equals its hostname or is a parent of it (longest match wins).
+ * Malformed pairs throw at boot rather than silently purging nothing.
+ */
+function parseZones(v) {
+ const out = [];
+ for (const pair of list(v, [])) {
+ const i = pair.indexOf('=');
+ const host = (i > 0 ? pair.slice(0, i) : '').trim().toLowerCase().replace(/\.$/, '');
+ const zoneId = (i > 0 ? pair.slice(i + 1) : '').trim().toLowerCase();
+ if (!HOST_RE.test(host) || !ZONE_ID_RE.test(zoneId)) throw new Error(`CLOUDFLARE_ZONE_IDS: "${pair}" is not host=<32-hex zone id>`);
+ out.push({ host, zoneId });
+ }
+ return out.sort((a, b) => b.host.length - a.host.length);
+}
/**
/**
* Services whose index-document events are accepted from OpenVibe.Events. Events already
* Services whose index-document events are accepted from OpenVibe.Events. Events already
@@ -64,7 +88,30 @@ function load(env = process.env) {
@@ -64,7 +88,30 @@ function load(env = process.env) {
maxFacetFilters: 8,
maxFacetFilters: 8,
},
},
maxBodyBytes: int(env.SEARCH_MAX_BODY_BYTES, 256 * 1024),
maxBodyBytes: int(env.SEARCH_MAX_BODY_BYTES, 256 * 1024),
+
+ // Freshness weighting of full-text relevance (README "Ranking"):
+ // score = bm25 × (1 + weight × 2^(−age_days / halfLifeDays)); weight 0 turns it off.
+ freshness: {
+ weight: Math.max(0, num(env.SEARCH_FRESHNESS_WEIGHT, 1)),
+ halfLifeDays: Math.max(0.01, num(env.SEARCH_FRESHNESS_HALF_LIFE_DAYS, 30)),
+ },
+
+ savedSearches: {
+ maxPerSubject: Math.max(1, int(env.SEARCH_SAVED_MAX_PER_SUBJECT, 50)),
+ },
+
+ // Removal purge queue (server/purge.js). Every removal is recorded for owners; the
+ // Cloudflare purge of a formerly public URL runs only when the token is set.
+ purge: {
+ cloudflareToken: env.CLOUDFLARE_PURGE_TOKEN || '',
+ zones: parseZones(env.CLOUDFLARE_ZONE_IDS),
+ // Paths on the removed URL's host purged with it (the product's sitemap, feeds).
+ relatedPaths: list(env.CLOUDFLARE_PURGE_RELATED_PATHS, ['/sitemap.xml'])
+ .filter(p => /^\/[A-Za-z0-9._~\/-]*$/.test(p)),
+ apiBase: strip(env.CLOUDFLARE_API_BASE || 'https://api.cloudflare.com/client/v4'),
+ intervalMs: int(env.CLOUDFLARE_PURGE_INTERVAL_MS, 5000),
+ },
};
};
}
}
-module.exports = { load, DEFAULT_EVENT_OWNERS };
+module.exports = { load, parseZones, DEFAULT_EVENT_OWNERS };
@@ -7,6 +7,8 @@ const fs = require('fs');
@@ -7,6 +7,8 @@ const fs = require('fs');
const path = require('path');
const path = require('path');
const Database = require('better-sqlite3');
const Database = require('better-sqlite3');
const outbox = require('./events/outbox');
const outbox = require('./events/outbox');
+const purge = require('./purge');
+const saved = require('./saved');
const SCHEMA = `
const SCHEMA = `
CREATE TABLE IF NOT EXISTS documents (
CREATE TABLE IF NOT EXISTS documents (
@@ -89,6 +91,8 @@ function openDb(dbPath) {
@@ -89,6 +91,8 @@ function openDb(dbPath) {
db.pragma('busy_timeout = 5000');
db.pragma('busy_timeout = 5000');
db.exec(SCHEMA);
db.exec(SCHEMA);
outbox.ensureSchema(db);
outbox.ensureSchema(db);
+ purge.ensureSchema(db);
+ saved.ensureSchema(db);
return db;
return db;
}
}
@@ -23,7 +23,29 @@ const WEIGHTS = '10.0, 4.0, 1.0'; // title, summary, body
@@ -23,7 +23,29 @@ const WEIGHTS = '10.0, 4.0, 1.0'; // title, summary, body
const MARK_OPEN = '\u0001';
const MARK_OPEN = '\u0001';
const MARK_CLOSE = '\u0002';
const MARK_CLOSE = '\u0002';
-function createEngine(db) {
+const DAY_MS = 24 * 3600 * 1000;
+
+/**
+ * Freshness multiplier for a document dated `iso` at reference time `nowMs`:
+ *
+ * 1 + weight × 2^(−age_days / halfLifeDays)
+ *
+ * With the defaults (weight 1, half-life 30 days) a document published today counts ×2.0, one
+ * 30 days old ×1.5, 60 days ×1.25, 90 days ×1.125, a year old ×1.0002: relevance still decides,
+ * and between comparably relevant documents the newer one ranks first. A future date counts as
+ * today; no date (or an unparsable one) counts ×1. Weight 0 turns the boost off.
+ */
+function freshnessBoost(iso, nowMs, halfLifeDays, weight) {
+ if (!weight || !iso) return 1;
+ const t = Date.parse(iso);
+ if (!Number.isFinite(t)) return 1;
+ const ageDays = Math.max(0, (nowMs - t) / DAY_MS);
+ return 1 + weight * Math.pow(2, -ageDays / halfLifeDays);
+}
+
+function createEngine(db, { freshness = { weight: 1, halfLifeDays: 30 } } = {}) {
+ // bm25() is negative (lower = more relevant), so multiplying by a boost ≥ 1 ranks newer first.
+ db.function('ov_freshness', { deterministic: true }, (iso, nowMs, halfLifeDays, weight) => freshnessBoost(iso, nowMs, halfLifeDays, weight));
db.exec(`
db.exec(`
CREATE VIRTUAL TABLE IF NOT EXISTS fts_public USING fts5(title, summary, body, ${TOKENIZE});
CREATE VIRTUAL TABLE IF NOT EXISTS fts_public USING fts5(title, summary, body, ${TOKENIZE});
CREATE VIRTUAL TABLE IF NOT EXISTS fts_restricted USING fts5(title, summary, body, ${TOKENIZE});
CREATE VIRTUAL TABLE IF NOT EXISTS fts_restricted USING fts5(title, summary, body, ${TOKENIZE});
@@ -81,17 +103,25 @@ function createEngine(db) {
@@ -81,17 +103,25 @@ function createEngine(db) {
return parts.length ? ` AND ${parts.join(' AND ')}` : '';
return parts.length ? ` AND ${parts.join(' AND ')}` : '';
}
}
- /** The visible, matching set as (rid, rank, snip) rows. match = FTS5 expression. */
- function matchedSet({ match, filters, viewer, snippets = true }, params) {
+ /**
+ * The visible, matching set as (rid, rank, snip) rows. match = FTS5 expression. rank is bm25
+ * times the freshness boost at reference time `now` (ms), so pages of one query (which carry
+ * their first page's `now` in the cursor) rank against the same clock.
+ */
+ function matchedSet({ match, filters, viewer, snippets = true, now = Date.now() }, params) {
params.match = match;
params.match = match;
+ params.r_now = now;
+ params.r_half = freshness.halfLifeDays;
+ params.r_weight = freshness.weight;
const where = filterClause(filters, params);
const where = filterClause(filters, params);
const snip = (t) => (snippets ? `snippet(${t}, -1, char(1), char(2), '…', 16)` : 'NULL');
const snip = (t) => (snippets ? `snippet(${t}, -1, char(1), char(2), '…', 16)` : 'NULL');
- const branches = [`SELECT d.rid AS rid, bm25(fts_public, ${WEIGHTS}) AS rank, ${snip('fts_public')} AS snip
+ const rank = (t) => `bm25(${t}, ${WEIGHTS}) * ov_freshness(COALESCE(d.published_at, d.updated_at), @r_now, @r_half, @r_weight)`;
+ const branches = [`SELECT d.rid AS rid, ${rank('fts_public')} AS rank, ${snip('fts_public')} AS snip
FROM fts_public JOIN documents d ON d.rid = fts_public.rowid
FROM fts_public JOIN documents d ON d.rid = fts_public.rowid
WHERE fts_public MATCH @match AND d.exposure = ${EXPOSURE.public_listed} AND d.deleted = 0${where}`];
WHERE fts_public MATCH @match AND d.exposure = ${EXPOSURE.public_listed} AND d.deleted = 0${where}`];
const acl = aclClause(viewer, params);
const acl = aclClause(viewer, params);
if (acl) {
if (acl) {
- branches.push(`SELECT d.rid AS rid, bm25(fts_restricted, ${WEIGHTS}) AS rank, ${snip('fts_restricted')} AS snip
+ branches.push(`SELECT d.rid AS rid, ${rank('fts_restricted')} AS rank, ${snip('fts_restricted')} AS snip
FROM fts_restricted JOIN documents d ON d.rid = fts_restricted.rowid
FROM fts_restricted JOIN documents d ON d.rid = fts_restricted.rowid
WHERE fts_restricted MATCH @match AND d.exposure = ${EXPOSURE.restricted} AND d.deleted = 0${where} AND ${acl}`);
WHERE fts_restricted MATCH @match AND d.exposure = ${EXPOSURE.restricted} AND d.deleted = 0${where} AND ${acl}`);
}
}
@@ -111,9 +141,9 @@ function createEngine(db) {
@@ -111,9 +141,9 @@ function createEngine(db) {
// ── Queries ──────────────────────────────────────────────
// ── Queries ──────────────────────────────────────────────
/** Ranked full-text search. after = { rank, rid } keyset cursor. */
/** Ranked full-text search. after = { rank, rid } keyset cursor. */
- function search({ match, filters = {}, viewer, limit, after }) {
+ function search({ match, filters = {}, viewer, limit, after, now = Date.now() }) {
const params = { limit: limit + 1 };
const params = { limit: limit + 1 };
- let sql = `SELECT rid, rank, snip FROM (${matchedSet({ match, filters, viewer }, params)})`;
+ let sql = `SELECT rid, rank, snip FROM (${matchedSet({ match, filters, viewer, now }, params)})`;
if (after) {
if (after) {
sql += ' WHERE (rank > @c_rank OR (rank = @c_rank AND rid > @c_rid))';
sql += ' WHERE (rank > @c_rank OR (rank = @c_rank AND rid > @c_rid))';
params.c_rank = after.rank;
params.c_rank = after.rank;
@@ -175,7 +205,7 @@ function createEngine(db) {
@@ -175,7 +205,7 @@ function createEngine(db) {
};
};
}
}
- return { name: 'sqlite-fts5', put, remove, search, browse, facetCounts, suggest, counts };
+ return { name: 'sqlite-fts5', put, remove, search, browse, facetCounts, suggest, counts, freshness };
}
}
// ── Query text → FTS5 expression ─────────────────────────────
// ── Query text → FTS5 expression ─────────────────────────────
@@ -202,4 +232,4 @@ function snippetHtml(snip) {
@@ -202,4 +232,4 @@ function snippetHtml(snip) {
return esc.split(MARK_OPEN).join('<mark>').split(MARK_CLOSE).join('</mark>');
return esc.split(MARK_OPEN).join('<mark>').split(MARK_CLOSE).join('</mark>');
}
}
-module.exports = { createEngine, toMatch, snippetHtml };
+module.exports = { createEngine, toMatch, snippetHtml, freshnessBoost };
@@ -13,14 +13,19 @@ const { createEngine } = require('./engine/fts5');
@@ -13,14 +13,19 @@ const { createEngine } = require('./engine/fts5');
const { createStore } = require('./store');
const { createStore } = require('./store');
const { createOutbox, createRelay } = require('./events/outbox');
const { createOutbox, createRelay } = require('./events/outbox');
const { createKeyStore, createAuth } = require('./auth');
const { createKeyStore, createAuth } = require('./auth');
+const { createPurgeQueue, createPurger } = require('./purge');
+const { createSavedSearches } = require('./saved');
const { createApp } = require('./app');
const { createApp } = require('./app');
async function start({ config, now = () => Date.now(), fetchImpl = globalThis.fetch, tokenClient, log = console, listen = true } = {}) {
async function start({ config, now = () => Date.now(), fetchImpl = globalThis.fetch, tokenClient, log = console, listen = true } = {}) {
config = config || load();
config = config || load();
const db = openDb(config.dbPath);
const db = openDb(config.dbPath);
- const engine = createEngine(db);
+ const engine = createEngine(db, { freshness: config.freshness });
const outbox = createOutbox(db, { source: config.serviceId, now });
const outbox = createOutbox(db, { source: config.serviceId, now });
- const store = createStore({ db, engine, outbox, now });
+ const purges = createPurgeQueue(db, { config: config.purge, now });
+ const purger = createPurger({ db, config: config.purge, fetchImpl, log, now });
+ const saved = createSavedSearches(db, { maxPerSubject: config.savedSearches.maxPerSubject, now });
+ const store = createStore({ db, engine, outbox, purges, now });
const relay = createRelay({
const relay = createRelay({
db, outbox, eventsUrl: config.events.url, intervalMs: config.events.relayIntervalMs, fetchImpl, log, now,
db, outbox, eventsUrl: config.events.url, intervalMs: config.events.relayIntervalMs, fetchImpl, log, now,
tokenClient,
tokenClient,
@@ -28,10 +33,11 @@ async function start({ config, now = () => Date.now(), fetchImpl = globalThis.fe
@@ -28,10 +33,11 @@ async function start({ config, now = () => Date.now(), fetchImpl = globalThis.fe
});
});
const keys = createKeyStore({ urls: [config.networkInternalUrl, config.networkUrl], pem: config.networkPublicKey, fetchImpl, log });
const keys = createKeyStore({ urls: [config.networkInternalUrl, config.networkUrl], pem: config.networkPublicKey, fetchImpl, log });
const auth = createAuth({ config, keys });
const auth = createAuth({ config, keys });
- const app = createApp({ config, db, store, engine, auth, keys, outbox, relay, log, now });
+ const app = createApp({ config, db, store, engine, auth, keys, outbox, relay, purges, purger, saved, log, now });
const keyLoaded = keys.start().catch(() => null);
const keyLoaded = keys.start().catch(() => null);
relay.start();
relay.start();
+ purger.start();
const pruneTimer = setInterval(() => { try { outbox.prune(); } catch (err) { log.error(`[outbox] prune: ${err.message}`); } }, 6 * 3600 * 1000);
const pruneTimer = setInterval(() => { try { outbox.prune(); } catch (err) { log.error(`[outbox] prune: ${err.message}`); } }, 6 * 3600 * 1000);
pruneTimer.unref?.();
pruneTimer.unref?.();
@@ -48,6 +54,7 @@ async function start({ config, now = () => Date.now(), fetchImpl = globalThis.fe
@@ -48,6 +54,7 @@ async function start({ config, now = () => Date.now(), fetchImpl = globalThis.fe
clearInterval(pruneTimer);
clearInterval(pruneTimer);
keys.stop();
keys.stop();
await relay.stop();
await relay.stop();
+ await purger.stop();
if (server) {
if (server) {
server.closeAllConnections?.();
server.closeAllConnections?.();
await new Promise(resolve => server.close(() => resolve()));
await new Promise(resolve => server.close(() => resolve()));
@@ -55,7 +62,7 @@ async function start({ config, now = () => Date.now(), fetchImpl = globalThis.fe
@@ -55,7 +62,7 @@ async function start({ config, now = () => Date.now(), fetchImpl = globalThis.fe
db.close();
db.close();
}
}
- return { config, db, engine, store, outbox, relay, keys, keyLoaded, auth, app, server, close };
+ return { config, db, engine, store, outbox, relay, purges, purger, saved, keys, keyLoaded, auth, app, server, close };
}
}
if (require.main === module) {
if (require.main === module) {
@@ -34,7 +34,7 @@ function readers(db) {
@@ -34,7 +34,7 @@ function readers(db) {
};
};
}
}
-function createSearchReadiness({ db, keys, config, engine, store, outbox, relay, release = null }) {
+function createSearchReadiness({ db, keys, config, engine, store, outbox, relay, purges = null, purger = null, saved = null, release = null }) {
const read = readers(db);
const read = readers(db);
return createReadiness({
return createReadiness({
service: 'search',
service: 'search',
@@ -70,13 +70,20 @@ function createSearchReadiness({ db, keys, config, engine, store, outbox, relay,
@@ -70,13 +70,20 @@ function createSearchReadiness({ db, keys, config, engine, store, outbox, relay,
documents: dbOk ? store.counts() : null,
documents: dbOk ? store.counts() : null,
outbox: dbOk ? { pending: outbox.pending(), rejected: outbox.rejected(), relay: config.events.url ? (relay.running() ? 'running' : 'stopped') : 'off (EVENTS_URL unset)' } : null,
outbox: dbOk ? { pending: outbox.pending(), rejected: outbox.rejected(), relay: config.events.url ? (relay.running() ? 'running' : 'stopped') : 'off (EVENTS_URL unset)' } : null,
webhook: config.events.webhookSecrets.length ? 'on' : 'off (SEARCH_EVENTS_SECRET unset)',
webhook: config.events.webhookSecrets.length ? 'on' : 'off (SEARCH_EVENTS_SECRET unset)',
+ purge: dbOk && purges ? {
+ cdn: purges.cdnOn() ? (purger && purger.running() ? 'cloudflare' : 'cloudflare (stopped)') : 'off (CLOUDFLARE_PURGE_TOKEN unset)',
+ zones: config.purge.zones.length,
+ ...purges.cdnCounts(),
+ } : null,
+ saved_searches: dbOk && saved ? saved.total() : null,
+ freshness: config.freshness,
};
};
},
},
});
});
}
}
/** Search gauges on the openvibe-shared/metrics registry. */
/** Search gauges on the openvibe-shared/metrics registry. */
-function registerSearchGauges(registry, { db, outbox }) {
+function registerSearchGauges(registry, { db, outbox, purges = null }) {
const read = readers(db);
const read = readers(db);
registry.gauge({
registry.gauge({
name: 'search_documents', help: 'Documents held, by exposure (public_listed and restricted are searchable; tombstones and drafts are none)', labelNames: ['exposure'],
name: 'search_documents', help: 'Documents held, by exposure (public_listed and restricted are searchable; tombstones and drafts are none)', labelNames: ['exposure'],
@@ -88,6 +95,12 @@ function registerSearchGauges(registry, { db, outbox }) {
@@ -88,6 +95,12 @@ function registerSearchGauges(registry, { db, outbox }) {
});
});
registry.gauge({ name: 'search_outbox_pending', help: 'Events waiting in the outbox', collect: () => outbox.pending() });
registry.gauge({ name: 'search_outbox_pending', help: 'Events waiting in the outbox', collect: () => outbox.pending() });
registry.gauge({ name: 'search_outbox_rejected', help: 'Events OpenVibe.Events refused for good', collect: () => outbox.rejected() });
registry.gauge({ name: 'search_outbox_rejected', help: 'Events OpenVibe.Events refused for good', collect: () => outbox.rejected() });
+ if (purges) {
+ registry.gauge({
+ name: 'search_cdn_purges', help: 'Cloudflare cache purges of removed documents, by state', labelNames: ['state'],
+ collect: () => Object.entries(purges.cdnCounts()).map(([state, value]) => ({ labels: { state }, value })),
+ });
+ }
}
}
module.exports = { createSearchReadiness, registerSearchGauges };
module.exports = { createSearchReadiness, registerSearchGauges };
new file mode 100644
new file mode 100644
@@ -0,0 +1,257 @@
@@ -0,0 +1,257 @@
+'use strict';
+/**
+ * The removal purge queue: Search's own consumer of its search.document.removed decisions.
+ *
+ * Every change that lowers a document's exposure (store.js announce()) records, in the same
+ * transaction as the index change and the outbox event:
+ *
+ * removals one row per removal: the feed products read to drop the document from their own
+ * caches and sitemaps (GET /api/v1/owners/:owner/removals, owner token). Products
+ * own their pages, so Search tells them rather than reaching into them.
+ * cdn_purges one row per URL to purge from Cloudflare's cache: the canonical URL the document
+ * had while it was public, plus the configured related paths on the same host
+ * (the product's sitemap by default). Written only when CLOUDFLARE_PURGE_TOKEN is
+ * set; without it the CDN side is inert and nothing is queued.
+ *
+ * The purger (createPurger) posts pending URLs to the Cloudflare API, grouped by zone, at most
+ * 30 per call (POST /zones/:zone/purge_cache { files }), with backoff on 429/5xx/network errors,
+ * permanent failure after MAX_ATTEMPTS or a 4xx refusal (a refused batch is retried one URL at a
+ * time so one bad URL cannot sink the others). The token is sent only in the Authorization
+ * header and never stored or logged.
+ */
+
+const BACKOFF_MS = [2000, 10000, 60000, 300000, 900000, 1800000, 3600000];
+const MAX_ATTEMPTS = 8;
+const FILES_PER_CALL = 30;
+const DETAIL_MAX = 300;
+
+function ensureSchema(db) {
+ db.exec(`
+ CREATE TABLE IF NOT EXISTS removals (
+ seq INTEGER PRIMARY KEY AUTOINCREMENT,
+ event_id TEXT NOT NULL, -- the search.document.removed event
+ owner TEXT NOT NULL,
+ type TEXT NOT NULL,
+ id TEXT NOT NULL,
+ revision INTEGER NOT NULL,
+ reason TEXT NOT NULL,
+ previous_exposure TEXT NOT NULL,
+ exposure TEXT NOT NULL,
+ canonical_url TEXT, -- only a URL that had been public
+ at INTEGER NOT NULL
+ );
+ CREATE INDEX IF NOT EXISTS idx_removals_owner ON removals (owner, seq);
+
+ CREATE TABLE IF NOT EXISTS cdn_purges (
+ seq INTEGER PRIMARY KEY AUTOINCREMENT,
+ removal_seq INTEGER NOT NULL,
+ provider TEXT NOT NULL DEFAULT 'cloudflare',
+ zone_id TEXT,
+ url TEXT NOT NULL,
+ state TEXT NOT NULL CHECK (state IN ('pending', 'purged', 'failed', 'skipped')),
+ detail TEXT,
+ attempts INTEGER NOT NULL DEFAULT 0,
+ next_attempt_at INTEGER NOT NULL DEFAULT 0,
+ created_at INTEGER NOT NULL,
+ done_at INTEGER
+ );
+ -- One pending purge per URL: a sitemap shared by many removals is purged once per batch.
+ CREATE UNIQUE INDEX IF NOT EXISTS idx_cdn_purges_pending ON cdn_purges (zone_id, url) WHERE state = 'pending';
+ CREATE INDEX IF NOT EXISTS idx_cdn_purges_due ON cdn_purges (state, next_attempt_at, seq);
+ `);
+}
+
+/** The zone a URL belongs to: exact host or a parent domain in the map, longest first. */
+function zoneFor(url, zones) {
+ let u;
+ try { u = new URL(url); } catch { return null; }
+ if (u.protocol !== 'https:' && u.protocol !== 'http:') return null;
+ const host = u.hostname.toLowerCase();
+ for (const z of zones) {
+ if (host === z.host || host.endsWith(`.${z.host}`)) return z.zoneId;
+ }
+ return null;
+}
+
+/** The canonical URL plus the related paths on its host, without fragments, de-duplicated. */
+function purgeUrls(canonicalUrl, relatedPaths) {
+ let u;
+ try { u = new URL(canonicalUrl); } catch { return []; }
+ if (u.protocol !== 'https:' && u.protocol !== 'http:') return [];
+ u.hash = '';
+ const out = [u.toString()];
+ for (const p of relatedPaths) out.push(`${u.origin}${p}`);
+ return [...new Set(out)];
+}
+
+function createPurgeQueue(db, { config, now = () => Date.now() }) {
+ const cdnOn = Boolean(config.cloudflareToken);
+ const st = {
+ insRemoval: db.prepare(`INSERT INTO removals (event_id, owner, type, id, revision, reason, previous_exposure, exposure, canonical_url, at)
+ VALUES (@event_id, @owner, @type, @id, @revision, @reason, @previous_exposure, @exposure, @canonical_url, @at)`),
+ insPurge: db.prepare(`INSERT INTO cdn_purges (removal_seq, zone_id, url, state, detail, created_at, done_at)
+ VALUES (@removal_seq, @zone_id, @url, @state, @detail, @created_at, @done_at)
+ ON CONFLICT (zone_id, url) WHERE state = 'pending' DO NOTHING`),
+ ownerFeed: db.prepare('SELECT * FROM removals WHERE owner = ? AND seq > ? ORDER BY seq LIMIT ?'),
+ cdnCounts: db.prepare('SELECT state, COUNT(*) AS n FROM cdn_purges GROUP BY state'),
+ purgesFor: db.prepare('SELECT * FROM cdn_purges WHERE removal_seq = ? ORDER BY seq'),
+ };
+
+ /**
+ * Record one removal (the payload of the search.document.removed event). Must run inside the
+ * transaction that made the change, like the outbox.
+ */
+ function record(eventId, p) {
+ if (!db.inTransaction) throw new Error('purge.record() must run inside the transaction that makes the change');
+ const at = now();
+ const removalSeq = Number(st.insRemoval.run({
+ event_id: eventId, owner: p.owner, type: p.type, id: p.id, revision: p.revision, reason: p.reason,
+ previous_exposure: p.previous_exposure, exposure: p.exposure, canonical_url: p.canonical_url || null, at,
+ }).lastInsertRowid);
+ if (!cdnOn || !p.canonical_url) return removalSeq;
+ for (const url of purgeUrls(p.canonical_url, config.relatedPaths)) {
+ const zoneId = zoneFor(url, config.zones);
+ st.insPurge.run({
+ removal_seq: removalSeq, zone_id: zoneId, url,
+ state: zoneId ? 'pending' : 'skipped',
+ detail: zoneId ? null : 'no Cloudflare zone configured for this host (CLOUDFLARE_ZONE_IDS)',
+ created_at: at, done_at: zoneId ? null : at,
+ });
+ }
+ return removalSeq;
+ }
+
+ /** The owner's removal feed, oldest first after a cursor. */
+ function ownerFeed(owner, { after = 0, limit = 200 } = {}) {
+ const rows = st.ownerFeed.all(owner, after, limit);
+ return {
+ removals: rows.map(r => ({
+ seq: r.seq, event_id: r.event_id, type: r.type, id: r.id, revision: r.revision, reason: r.reason,
+ previous_exposure: r.previous_exposure, exposure: r.exposure, canonical_url: r.canonical_url,
+ at: new Date(r.at).toISOString(),
+ })),
+ next_after: rows.length ? rows[rows.length - 1].seq : null,
+ };
+ }
+
+ function cdnCounts() {
+ const out = { pending: 0, purged: 0, failed: 0, skipped: 0 };
+ for (const r of st.cdnCounts.all()) out[r.state] = r.n;
+ return out;
+ }
+
+ return { record, ownerFeed, cdnCounts, purgesFor: (seq) => st.purgesFor.all(seq), cdnOn: () => cdnOn };
+}
+
+/**
+ * createPurger({ db, config, fetchImpl?, log?, now? }) → { start, stop, flush, running }
+ * Does nothing without config.cloudflareToken.
+ */
+function createPurger({ db, config, fetchImpl = globalThis.fetch, log = console, now = () => Date.now() }) {
+ const token = config.cloudflareToken;
+ const q = {
+ due: db.prepare(`SELECT seq, zone_id, url, attempts FROM cdn_purges
+ WHERE state = 'pending' AND next_attempt_at <= ? ORDER BY seq LIMIT ?`),
+ purged: db.prepare("UPDATE cdn_purges SET state = 'purged', attempts = attempts + 1, detail = ?, done_at = ? WHERE seq = ?"),
+ retry: db.prepare('UPDATE cdn_purges SET attempts = attempts + 1, next_attempt_at = ?, detail = ? WHERE seq = ?'),
+ failed: db.prepare("UPDATE cdn_purges SET state = 'failed', attempts = attempts + 1, detail = ?, done_at = ? WHERE seq = ?"),
+ };
+ let timer = null;
+ let flushing = null;
+
+ const clip = (s) => (token ? String(s || '').split(token).join('[token]') : String(s || '')).slice(0, DETAIL_MAX);
+
+ function mark(rows, fn) { db.transaction(() => { for (const r of rows) fn(r); })(); }
+
+ function retryOrFail(rows, detail) {
+ mark(rows, (r) => {
+ if (r.attempts + 1 >= MAX_ATTEMPTS) q.failed.run(clip(`gave up after ${MAX_ATTEMPTS} attempts: ${detail}`), now(), r.seq);
+ else q.retry.run(now() + BACKOFF_MS[Math.min(r.attempts, BACKOFF_MS.length - 1)], clip(detail), r.seq);
+ });
+ }
+
+ async function call(zoneId, urls) {
+ const res = await fetchImpl(`${config.apiBase}/zones/${encodeURIComponent(zoneId)}/purge_cache`, {
+ method: 'POST',
+ headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json', Accept: 'application/json' },
+ body: JSON.stringify({ files: urls }),
+ signal: AbortSignal.timeout(15000),
+ });
+ let body = null;
+ try { body = await res.json(); } catch { body = null; }
+ return { status: res.status, body };
+ }
+
+ /** → { purged, retrying, failed } for one zone's batch. */
+ async function purgeBatch(zoneId, rows) {
+ let r;
+ try {
+ r = await call(zoneId, rows.map(x => x.url));
+ } catch (err) {
+ retryOrFail(rows, `network: ${err.message}`);
+ return { purged: 0, retrying: rows.length, failed: 0 };
+ }
+ const errors = (r.body && Array.isArray(r.body.errors) ? r.body.errors : []).map(e => `${e.code}: ${e.message}`).join('; ');
+ if (r.status >= 200 && r.status < 300 && r.body && r.body.success === true) {
+ const id = r.body.result && r.body.result.id ? `cloudflare purge ${r.body.result.id}` : 'purged';
+ mark(rows, x => q.purged.run(clip(id), now(), x.seq));
+ return { purged: rows.length, retrying: 0, failed: 0 };
+ }
+ if (r.status === 429 || r.status >= 500 || r.status === 408) {
+ retryOrFail(rows, `HTTP ${r.status} ${errors}`);
+ return { purged: 0, retrying: rows.length, failed: 0 };
+ }
+ // Refused (bad URL, token without purge permission, unknown zone): isolate, then fail.
+ if (rows.length > 1) {
+ const total = { purged: 0, retrying: 0, failed: 0 };
+ for (const row of rows) {
+ const s = await purgeBatch(zoneId, [row]);
+ total.purged += s.purged; total.retrying += s.retrying; total.failed += s.failed;
+ }
+ return total;
+ }
+ mark(rows, x => q.failed.run(clip(`HTTP ${r.status} ${errors || 'refused'}`), now(), x.seq));
+ log.warn(`[purge] Cloudflare refused a purge: HTTP ${r.status} ${clip(errors)}`);
+ return { purged: 0, retrying: 0, failed: rows.length };
+ }
+
+ async function doFlush() {
+ const total = { purged: 0, retrying: 0, failed: 0 };
+ if (!token) return total;
+ const rows = q.due.all(now(), 200);
+ const byZone = new Map();
+ for (const r of rows) {
+ if (!byZone.has(r.zone_id)) byZone.set(r.zone_id, []);
+ byZone.get(r.zone_id).push(r);
+ }
+ for (const [zoneId, zoneRows] of byZone) {
+ for (let i = 0; i < zoneRows.length; i += FILES_PER_CALL) {
+ const s = await purgeBatch(zoneId, zoneRows.slice(i, i + FILES_PER_CALL));
+ total.purged += s.purged; total.retrying += s.retrying; total.failed += s.failed;
+ }
+ }
+ if (total.retrying) log.warn(`[purge] ${total.retrying} URL(s) not purged yet; will retry`);
+ return total;
+ }
+
+ function flush() {
+ if (!flushing) flushing = doFlush().finally(() => { flushing = null; });
+ return flushing;
+ }
+
+ function start() {
+ if (timer || !token) return;
+ timer = setInterval(() => { flush().catch(err => log.warn(`[purge] ${err.message}`)); }, config.intervalMs);
+ timer.unref?.();
+ }
+
+ function stop() {
+ if (timer) clearInterval(timer);
+ timer = null;
+ return flushing || Promise.resolve();
+ }
+
+ return { start, stop, flush, running: () => Boolean(timer) };
+}
+
+module.exports = { ensureSchema, createPurgeQueue, createPurger, zoneFor, purgeUrls, MAX_ATTEMPTS, FILES_PER_CALL };
new file mode 100644
new file mode 100644
@@ -0,0 +1,114 @@
@@ -0,0 +1,114 @@
+'use strict';
+/**
+ * Saved searches: a signed-in person (a usr_ subject) keeps a query (text + filters) and runs it
+ * again later. The saved row holds the query only, never results: every run is a fresh query as
+ * that person at that moment, so a document that became private or was deleted since the save
+ * is simply not there, and one they gained access to is.
+ *
+ * Notifications for new matches wait for OpenVibe.Network's notifications (README "Saved
+ * searches"); last_run_at is kept so that job can tell what is new.
+ */
+const crypto = require('crypto');
+const { ids } = require('openvibe-contracts');
+
+const ID_RE = /^svs_[0-9a-z]{26}$/;
+
+function ensureSchema(db) {
+ db.exec(`
+ CREATE TABLE IF NOT EXISTS saved_searches (
+ id TEXT PRIMARY KEY,
+ subject TEXT NOT NULL, -- usr_… (the person who saved it)
+ name TEXT NOT NULL,
+ q TEXT NOT NULL DEFAULT '',
+ filters TEXT NOT NULL, -- canonical JSON {owner,type,language,facets}
+ query_hash TEXT NOT NULL, -- same query saved twice = one row
+ created_at INTEGER NOT NULL,
+ updated_at INTEGER NOT NULL,
+ last_run_at INTEGER,
+ UNIQUE (subject, query_hash)
+ );
+ CREATE INDEX IF NOT EXISTS idx_saved_searches_subject ON saved_searches (subject, created_at);
+ `);
+}
+
+/** Filters in one canonical shape, so equal queries hash equally whatever order they came in. */
+function canonicalFilters(f) {
+ const out = {};
+ if (f.owner) out.owner = f.owner;
+ if (f.type) out.type = f.type;
+ if (f.language) out.language = f.language;
+ const facets = (f.facets || [])
+ .map(([k, vs]) => [k, [...new Set(vs)].sort()])
+ .sort((a, b) => (a[0] < b[0] ? -1 : a[0] > b[0] ? 1 : 0));
+ if (facets.length) out.facets = facets;
+ return out;
+}
+
+function hashOf(q, filters) {
+ return crypto.createHash('sha256').update(JSON.stringify([q, filters])).digest('base64url').slice(0, 22);
+}
+
+function createSavedSearches(db, { maxPerSubject = 50, now = () => Date.now() } = {}) {
+ const st = {
+ list: db.prepare('SELECT * FROM saved_searches WHERE subject = ? ORDER BY created_at DESC, id DESC'),
+ get: db.prepare('SELECT * FROM saved_searches WHERE id = ? AND subject = ?'),
+ byHash: db.prepare('SELECT * FROM saved_searches WHERE subject = ? AND query_hash = ?'),
+ count: db.prepare('SELECT COUNT(*) AS n FROM saved_searches WHERE subject = ?'),
+ insert: db.prepare(`INSERT INTO saved_searches (id, subject, name, q, filters, query_hash, created_at, updated_at)
+ VALUES (@id, @subject, @name, @q, @filters, @query_hash, @at, @at)`),
+ rename: db.prepare('UPDATE saved_searches SET name = ?, updated_at = ? WHERE id = ?'),
+ del: db.prepare('DELETE FROM saved_searches WHERE id = ? AND subject = ?'),
+ ran: db.prepare('UPDATE saved_searches SET last_run_at = ? WHERE id = ?'),
+ total: db.prepare('SELECT COUNT(*) AS n FROM saved_searches'),
+ };
+
+ function view(r) {
+ return {
+ id: r.id,
+ name: r.name,
+ q: r.q,
+ filters: JSON.parse(r.filters),
+ created_at: new Date(r.created_at).toISOString(),
+ updated_at: new Date(r.updated_at).toISOString(),
+ last_run_at: r.last_run_at ? new Date(r.last_run_at).toISOString() : null,
+ };
+ }
+
+ /**
+ * Save (subject, q, filters) with a name. The same query again returns the existing row
+ * (renamed when a new name is given): { created, saved } | { error: 'limit' }.
+ */
+ function save(subject, { name, q, filters }) {
+ return db.transaction(() => {
+ const f = canonicalFilters(filters);
+ const hash = hashOf(q, f);
+ const at = now();
+ const existing = st.byHash.get(subject, hash);
+ if (existing) {
+ if (name && name !== existing.name) st.rename.run(name, at, existing.id);
+ return { created: false, saved: view(st.get.get(existing.id, subject)) };
+ }
+ if (st.count.get(subject).n >= maxPerSubject) return { error: 'limit' };
+ const id = `svs_${ids.ulid(at).toLowerCase()}`;
+ st.insert.run({ id, subject, name: name || q || 'All documents', q, filters: JSON.stringify(f), query_hash: hash, at });
+ return { created: true, saved: view(st.get.get(id, subject)) };
+ })();
+ }
+
+ return {
+ list: (subject) => st.list.all(subject).map(view),
+ /** One saved search of this subject; another subject's id is the same as a missing one. */
+ get: (subject, id) => {
+ if (!ID_RE.test(String(id))) return null;
+ const r = st.get.get(id, subject);
+ return r ? view(r) : null;
+ },
+ save,
+ remove: (subject, id) => ID_RE.test(String(id)) && st.del.run(id, subject).changes > 0,
+ markRun: (id) => st.ran.run(now(), id),
+ total: () => st.total.get().n,
+ maxPerSubject,
+ };
+}
+
+module.exports = { ensureSchema, createSavedSearches, canonicalFilters, ID_RE };
@@ -15,7 +15,7 @@ const { normalize, effectiveIndexability, exposureOf, facetPairs, EXPOSURE } = r
@@ -15,7 +15,7 @@ const { normalize, effectiveIndexability, exposureOf, facetPairs, EXPOSURE } = r
const EXPOSURE_NAME = Object.fromEntries(Object.entries(EXPOSURE).map(([k, v]) => [v, k]));
const EXPOSURE_NAME = Object.fromEntries(Object.entries(EXPOSURE).map(([k, v]) => [v, k]));
-function createStore({ db, engine, outbox, now = () => Date.now() }) {
+function createStore({ db, engine, outbox, purges = null, now = () => Date.now() }) {
const st = {
const st = {
get: db.prepare('SELECT * FROM documents WHERE owner = ? AND type = ? AND id = ?'),
get: db.prepare('SELECT * FROM documents WHERE owner = ? AND type = ? AND id = ?'),
byRid: db.prepare('SELECT * FROM documents WHERE rid = ?'),
byRid: db.prepare('SELECT * FROM documents WHERE rid = ?'),
@@ -89,19 +89,17 @@ function createStore({ db, engine, outbox, now = () => Date.now() }) {
@@ -89,19 +89,17 @@ function createStore({ db, engine, outbox, now = () => Date.now() }) {
const subject = { type: 'document', id: `${doc.owner}/${doc.type}/${doc.id}`, revision: doc.revision };
const subject = { type: 'document', id: `${doc.owner}/${doc.type}/${doc.id}`, revision: doc.revision };
const wasPublic = prevExposure >= EXPOSURE.public_unlisted;
const wasPublic = prevExposure >= EXPOSURE.public_unlisted;
if (exposure < prevExposure) {
if (exposure < prevExposure) {
- outbox.enqueue({
- event_type: 'search.document.removed',
- subject,
- trace_id: traceId,
- payload: {
- owner: doc.owner, type: doc.type, id: doc.id, revision: doc.revision,
- reason: removalReason(prevDoc, doc),
- previous_exposure: EXPOSURE_NAME[prevExposure],
- exposure: EXPOSURE_NAME[exposure],
- // Only a URL that was public already: caches and sitemaps purge it.
- canonical_url: wasPublic ? prevRow.canonical_url : null,
- },
- });
+ const payload = {
+ owner: doc.owner, type: doc.type, id: doc.id, revision: doc.revision,
+ reason: removalReason(prevDoc, doc),
+ previous_exposure: EXPOSURE_NAME[prevExposure],
+ exposure: EXPOSURE_NAME[exposure],
+ // Only a URL that was public already: caches and sitemaps purge it.
+ canonical_url: wasPublic ? prevRow.canonical_url : null,
+ };
+ const env = outbox.enqueue({ event_type: 'search.document.removed', subject, trace_id: traceId, payload });
+ // Search's own consumer: the owners' removal feed and the CDN purge queue (purge.js).
+ if (purges) purges.record(env.event_id, payload);
}
}
if (!doc.deleted && exposure > EXPOSURE.none) {
if (!doc.deleted && exposure > EXPOSURE.none) {
outbox.enqueue({
outbox.enqueue({
new file mode 100644
new file mode 100644
@@ -0,0 +1,176 @@
@@ -0,0 +1,176 @@
+'use strict';
+/**
+ * The public search page at search.openvibe.network: server-rendered, works without JavaScript,
+ * same searcher and visibility rules as GET /api/v1/search (anonymous = public, published,
+ * indexable documents only; an ov_token cookie for this host adds its person's ACL matches).
+ *
+ * GET / search form; ?q= results, ?owner= / ?type= filters, ?cursor= next page
+ * GET /robots.txt the front page may be crawled, result pages and the API not
+ *
+ * Every answer is no-store (a document that leaves the index leaves this page at once) and result
+ * pages are noindex. Clients that do not ask for HTML get the plain-text route index at /.
+ */
+const express = require('express');
+const { AuthError, ANONYMOUS } = require('../auth');
+const { QueryError, one } = require('../api/query');
+
+const CSP = "default-src 'none'; style-src 'unsafe-inline'; img-src 'self' data:; form-action 'self'; base-uri 'none'; frame-ancestors 'none'";
+
+const esc = (s) => String(s == null ? '' : s).replace(/[&<>"']/g, c => ({ '&': '&', '<': '<', '>': '>', '"': '"', "'": ''' }[c]));
+
+/** Only http(s) links are rendered as links; anything else is shown as text. */
+function safeHref(url) {
+ try {
+ const u = new URL(url);
+ return u.protocol === 'https:' || u.protocol === 'http:' ? u.toString() : null;
+ } catch { return null; }
+}
+
+function dateOf(iso) {
+ const t = Date.parse(iso || '');
+ return Number.isFinite(t) ? new Date(t).toISOString().slice(0, 10) : '';
+}
+
+const TEXT_INDEX = [
+ 'OpenVibe.Search: permission-aware search over documents the network\'s services index.',
+ '',
+ 'GET /?q= search page (HTML)',
+ 'GET /api/v1/search?q=&owner=&type=&facet.<key>=&cursor= query (anonymous: public only)',
+ 'GET /api/v1/suggest?q= title suggestions',
+ 'GET /api/v1/documents/:owner/:type/:id one document the caller may see',
+ 'GET /api/v1/saved-searches, POST, GET/DELETE /:id, GET /:id/results saved searches (signed in)',
+ 'PUT /api/v1/documents/:owner/:type/:id index (owner, search.document.write)',
+ 'DELETE /api/v1/documents/:owner/:type/:id?revision= tombstone (owner)',
+ 'GET /api/v1/owners/:owner/documents reconciliation (owner)',
+ 'GET /api/v1/owners/:owner/removals?after= removal feed for caches and sitemaps (owner)',
+ 'POST /internal/events OpenVibe.Events delivery (signed, host-local)',
+ 'GET /api/health, /api/ready, /release.json',
+ '',
+ 'Source: https://github.com/OpenVibers/OpenVibe.Search',
+ '',
+].join('\n');
+
+function layout({ title, q, owner, type, body, noindex }) {
+ return `<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>${esc(title)}</title>
+${noindex ? '<meta name="robots" content="noindex, nofollow">\n' : ''}<meta name="description" content="Search what the OpenVibe network's services have published.">
+<meta name="color-scheme" content="light dark">
+<style>
+:root { --bg: #fff; --fg: #1a1a1a; --muted: #5c5c66; --line: #dcdce3; --accent: #2456d6; --mark: #fff2a8; }
+@media (prefers-color-scheme: dark) { :root { --bg: #111317; --fg: #e8e8ec; --muted: #a0a0ab; --line: #2c2f36; --accent: #7aa2ff; --mark: #5a4b00; } }
+* { box-sizing: border-box; }
+body { margin: 0; background: var(--bg); color: var(--fg); font: 16px/1.5 system-ui, -apple-system, "Segoe UI", sans-serif; }
+main { max-width: 760px; margin: 0 auto; padding: 24px 16px 48px; }
+h1 { font-size: 1.4rem; margin: 0 0 4px; }
+.lede { color: var(--muted); margin: 0 0 20px; }
+form { display: flex; flex-wrap: wrap; gap: 8px; margin-bottom: 20px; }
+input[type=search] { flex: 1 1 260px; min-width: 0; padding: 10px 12px; font: inherit; color: inherit; background: transparent; border: 1px solid var(--line); border-radius: 8px; }
+button { padding: 10px 16px; font: inherit; border: 0; border-radius: 8px; background: var(--accent); color: #fff; cursor: pointer; }
+ol { list-style: none; padding: 0; margin: 0; }
+li { padding: 14px 0; border-top: 1px solid var(--line); overflow-wrap: anywhere; }
+li a.t { color: var(--accent); font-size: 1.05rem; text-decoration: none; }
+li a.t:hover { text-decoration: underline; }
+.meta { color: var(--muted); font-size: .85rem; }
+.snip { margin: 4px 0 0; }
+mark { background: var(--mark); color: inherit; }
+.empty, .note { color: var(--muted); }
+.more { display: inline-block; margin-top: 16px; color: var(--accent); }
+footer { margin-top: 40px; color: var(--muted); font-size: .85rem; }
+footer a { color: inherit; }
+</style>
+</head>
+<body>
+<main>
+<h1><a href="/" style="color:inherit;text-decoration:none">OpenVibe.Search</a></h1>
+<p class="lede">Search what the OpenVibe network's services have published. Alpha: the index holds only what they have sent so far.</p>
+<form method="get" action="/" role="search">
+<input type="search" name="q" value="${esc(q)}" maxlength="500" placeholder="Search the network" aria-label="Search terms" autofocus>
+${owner ? `<input type="hidden" name="owner" value="${esc(owner)}">` : ''}${type ? `<input type="hidden" name="type" value="${esc(type)}">` : ''}<button type="submit">Search</button>
+</form>
+${body}
+<footer>
+<p>Private, draft and deleted documents are never shown. A document that is removed or made private leaves these results at once.
+JSON API: <a href="/api/v1/search?q=${encodeURIComponent(q || '')}">/api/v1/search</a>.
+Source: <a href="https://github.com/OpenVibers/OpenVibe.Search">OpenVibers/OpenVibe.Search</a>.</p>
+</footer>
+</main>
+</body>
+</html>
+`;
+}
+
+function resultsHtml(results) {
+ return `<ol>${results.map((r) => {
+ const href = safeHref(r.canonical_url);
+ const title = esc(r.title || `${r.owner}/${r.type}/${r.id}`);
+ const date = dateOf(r.published_at || r.updated_at);
+ return `<li>${href ? `<a class="t" href="${esc(href)}">${title}</a>` : `<span class="t">${title}</span>`}
+<div class="meta">${esc(r.owner)} · ${esc(r.type)}${date ? ` · ${date}` : ''}${r.visibility !== 'public' ? ` · ${esc(r.visibility)}` : ''}</div>
+${r.snippet_html ? `<p class="snip">${r.snippet_html}</p>` : r.summary ? `<p class="snip">${esc(r.summary)}</p>` : ''}</li>`;
+ }).join('\n')}</ol>`;
+}
+
+function pageRouter({ searcher, auth }) {
+ const router = express.Router();
+
+ router.get('/robots.txt', (_req, res) => {
+ res.type('text/plain').set('Cache-Control', 'public, max-age=3600')
+ .send('User-agent: *\nAllow: /$\nDisallow: /?\nDisallow: /api/\nDisallow: /internal/\n');
+ });
+
+ router.get('/', (req, res, next) => {
+ const wantsHtml = /\btext\/html\b/.test(String(req.get('accept') || ''));
+ res.setHeader('Cache-Control', 'no-store');
+ res.setHeader('Vary', 'Accept, Cookie');
+ if (!wantsHtml) return res.type('text/plain').send(TEXT_INDEX);
+
+ const q = String(one(req.query.q) || '').slice(0, 500);
+ const owner = one(req.query.owner) ? String(one(req.query.owner)) : '';
+ const type = one(req.query.type) ? String(one(req.query.type)) : '';
+ const cursor = one(req.query.cursor);
+ const searching = Boolean(q.trim() || owner || type);
+ res.setHeader('Content-Security-Policy', CSP);
+ res.setHeader('X-Frame-Options', 'DENY');
+ res.setHeader('Referrer-Policy', 'strict-origin-when-cross-origin');
+ if (searching) res.setHeader('X-Robots-Tag', 'noindex, nofollow');
+
+ let viewer;
+ try {
+ viewer = auth.viewer(req);
+ } catch (err) {
+ if (!(err instanceof AuthError)) return next(err);
+ viewer = null; // a browser page never 401s: an unverifiable credential searches as anonymous
+ }
+ let body = '<p class="note">Type some words to search. Filters: <code>?owner=wiki</code>, <code>?type=page</code>.</p>';
+ let status = 200;
+ if (searching) {
+ try {
+ const out = searcher.run({ text: q, query: { ...(owner ? { owner } : {}), ...(type ? { type } : {}) }, viewer: viewer || ANONYMOUS, cursor });
+ body = out.results.length
+ ? resultsHtml(out.results)
+ : `<p class="empty">${cursor ? 'No more results.' : 'Nothing found. The index is young: services add documents as they publish them.'}</p>`;
+ if (out.next_cursor) {
+ const u = new URLSearchParams();
+ if (q) u.set('q', q);
+ if (owner) u.set('owner', owner);
+ if (type) u.set('type', type);
+ u.set('cursor', out.next_cursor);
+ body += `<a class="more" href="/?${esc(u.toString())}">More results</a>`;
+ }
+ } catch (err) {
+ if (!(err instanceof QueryError)) return next(err);
+ status = 400;
+ body = `<p class="empty">${esc(err.message)}.</p>`;
+ }
+ }
+ res.status(status).type('html').send(layout({ title: q ? `${q} · OpenVibe.Search` : 'OpenVibe.Search', q, owner, type, body, noindex: searching }));
+ });
+
+ return router;
+}
+
+module.exports = { pageRouter, TEXT_INDEX };
new file mode 100644
new file mode 100644
@@ -0,0 +1,91 @@
@@ -0,0 +1,91 @@
+'use strict';
+/**
+ * Freshness weighting: score = bm25 × (1 + weight × 2^(−age_days / half_life)). Newer documents
+ * rank higher between comparably relevant ones; relevance still wins over age; pages of one query
+ * rank on the first page's clock.
+ */
+const assert = require('assert');
+const { boot, request, doc, suite } = require('./helpers');
+const { freshnessBoost } = require('../server/engine/fts5');
+
+const t = suite('freshness');
+const NOW = Date.parse('2026-09-23T12:00:00Z');
+const daysAgo = (n) => new Date(NOW - n * 86400000).toISOString();
+
+t('the decay: ×(1+w) today, ×(1+w/2) one half-life ago, → ×1 when old; future = today; none = ×1', () => {
+ const close = (a, b) => assert.ok(Math.abs(a - b) < 1e-9, `${a} ≠ ${b}`);
+ close(freshnessBoost(daysAgo(0), NOW, 30, 1), 2);
+ close(freshnessBoost(daysAgo(30), NOW, 30, 1), 1.5);
+ close(freshnessBoost(daysAgo(60), NOW, 30, 1), 1.25);
+ close(freshnessBoost(daysAgo(90), NOW, 30, 1), 1.125);
+ assert.ok(freshnessBoost(daysAgo(365), NOW, 30, 1) < 1.001);
+ close(freshnessBoost(daysAgo(-10), NOW, 30, 1), 2);
+ close(freshnessBoost(null, NOW, 30, 1), 1);
+ close(freshnessBoost('not a date', NOW, 30, 1), 1);
+ close(freshnessBoost(daysAgo(0), NOW, 30, 0), 1);
+ close(freshnessBoost(daysAgo(7), NOW, 7, 0.5), 1.25);
+});
+
+async function corpus(env = {}) {
+ let clock = NOW;
+ const svc = await boot({ env, now: () => clock });
+ // Same text, same length: bm25 ties, so only freshness separates them. The old one is indexed
+ // first (lower rid), so without freshness it would come first.
+ svc.store.apply(doc({ id: 'old', title: 'Aurora report', body: 'aurora seen tonight', published_at: daysAgo(400), updated_at: daysAgo(400) }));
+ svc.store.apply(doc({ id: 'mid', title: 'Aurora report', body: 'aurora seen tonight', published_at: daysAgo(20), updated_at: daysAgo(20) }));
+ svc.store.apply(doc({ id: 'new', title: 'Aurora report', body: 'aurora seen tonight', published_at: daysAgo(1), updated_at: daysAgo(1) }));
+ return { svc, setClock: (c) => { clock = c; } };
+}
+
+t('between equally relevant documents the newest ranks first', async () => {
+ const { svc } = await corpus();
+ try {
+ const r = await request(svc.base, 'GET', '/api/v1/search?q=aurora');
+ assert.deepStrictEqual(r.body.results.map(x => x.id), ['new', 'mid', 'old']);
+ } finally { await svc.stop(); }
+});
+
+t('weight 0 turns freshness off (pure bm25, ties by index order)', async () => {
+ const { svc } = await corpus({ SEARCH_FRESHNESS_WEIGHT: '0' });
+ try {
+ const r = await request(svc.base, 'GET', '/api/v1/search?q=aurora');
+ assert.deepStrictEqual(r.body.results.map(x => x.id), ['old', 'mid', 'new']);
+ assert.deepStrictEqual((await request(svc.base, 'GET', '/api/ready')).body.freshness, { weight: 0, halfLifeDays: 30 });
+ } finally { await svc.stop(); }
+});
+
+t('relevance still wins: an old title match outranks a fresh passing mention', async () => {
+ let clock = NOW;
+ const svc = await boot({ now: () => clock });
+ try {
+ svc.store.apply(doc({ id: 'fresh-mention', title: 'Weekly notes', summary: 'Many topics this week.', body: `${'filler words about other things '.repeat(40)} glacier`, published_at: daysAgo(0) }));
+ svc.store.apply(doc({ id: 'old-title', title: 'Glacier glacier retreat', summary: 'The glacier retreat, measured.', body: 'glacier data', published_at: daysAgo(700) }));
+ const r = await request(svc.base, 'GET', '/api/v1/search?q=glacier');
+ assert.deepStrictEqual(r.body.results.map(x => x.id), ['old-title', 'fresh-mention']);
+ } finally { await svc.stop(); }
+});
+
+t('pages of one query rank on the first page\'s clock: no duplicates or gaps when time moves on', async () => {
+ const { svc, setClock } = await corpus();
+ try {
+ const p1 = await request(svc.base, 'GET', '/api/v1/search?q=aurora&limit=1');
+ assert.deepStrictEqual(p1.body.results.map(x => x.id), ['new']);
+ setClock(NOW + 200 * 86400000);
+ const p2 = await request(svc.base, 'GET', `/api/v1/search?q=aurora&limit=1&cursor=${p1.body.next_cursor}`);
+ const p3 = await request(svc.base, 'GET', `/api/v1/search?q=aurora&limit=1&cursor=${p2.body.next_cursor}`);
+ assert.deepStrictEqual([...p2.body.results, ...p3.body.results].map(x => x.id), ['mid', 'old']);
+ assert.strictEqual(p3.body.next_cursor, null);
+ } finally { await svc.stop(); }
+});
+
+t('browse (no q) stays newest first; suggestions prefer fresher titles', async () => {
+ const { svc } = await corpus();
+ try {
+ const b = await request(svc.base, 'GET', '/api/v1/search');
+ assert.deepStrictEqual(b.body.results.map(x => x.id), ['new', 'mid', 'old']);
+ const s = await request(svc.base, 'GET', '/api/v1/suggest?q=auro');
+ assert.deepStrictEqual(s.body.suggestions.map(x => x.id), ['new', 'mid', 'old']);
+ } finally { await svc.stop(); }
+});
+
+t.run();
new file mode 100644
new file mode 100644
@@ -0,0 +1,88 @@
@@ -0,0 +1,88 @@
+'use strict';
+/**
+ * The public HTML search page (server/web/page.js): works without JavaScript, shows only what an
+ * anonymous query may see, escapes everything, never caches, and keeps result pages out of
+ * search engines. Non-HTML clients still get the text route index.
+ */
+const assert = require('assert');
+const { ids } = require('openvibe-contracts');
+const { boot, request, userToken, doc, suite } = require('./helpers');
+
+const t = suite('page');
+let svc;
+const html = (p, headers = {}) => request(svc.base, 'GET', p, { headers: { Accept: 'text/html,application/xhtml+xml', ...headers } });
+const alice = ids.newId('user');
+
+t('boot', async () => {
+ svc = await boot();
+ svc.store.apply(doc({ id: 'pub', title: 'Kestrel <script>alert(1)</script> nesting', summary: 'Kestrel "boxes" & more', body: 'kestrel kestrel', canonical_url: 'https://openvibe.wiki/p/kestrel?a=1&b=2' }));
+ svc.store.apply(doc({ id: 'prv', visibility: 'private', acl: { subjects: [alice] }, title: 'Kestrel private diary', body: 'kestrel' }));
+ svc.store.apply(doc({ id: 'drf', visibility: 'draft', title: 'Kestrel draft', body: 'kestrel' }));
+ for (let i = 0; i < 25; i++) svc.store.apply(doc({ id: `many${i}`, title: `Heron sighting ${i}`, body: 'heron' }));
+});
+
+t('the front page is a search form, indexable, no-store, with a strict CSP', async () => {
+ const r = await html('/');
+ assert.strictEqual(r.status, 200);
+ assert.match(r.headers.get('content-type'), /text\/html/);
+ assert.match(r.text, /<form method="get" action="\/" role="search">/);
+ assert.ok(!/noindex/.test(r.text));
+ assert.strictEqual(r.headers.get('x-robots-tag'), null);
+ assert.strictEqual(r.headers.get('cache-control'), 'no-store');
+ assert.match(r.headers.get('content-security-policy'), /default-src 'none'/);
+ assert.ok(!/<script/i.test(r.text), 'no JavaScript on the page');
+});
+
+t('results show public documents only, escaped, linked to their canonical URL, noindex', async () => {
+ const r = await html('/?q=kestrel');
+ assert.strictEqual(r.status, 200);
+ assert.strictEqual(r.headers.get('x-robots-tag'), 'noindex, nofollow');
+ assert.match(r.text, /<meta name="robots" content="noindex, nofollow">/);
+ assert.ok(r.text.includes('Kestrel <script>alert(1)</script> nesting'));
+ assert.ok(!r.text.includes('<script>alert(1)'));
+ assert.ok(r.text.includes('href="https://openvibe.wiki/p/kestrel?a=1&b=2"'));
+ assert.match(r.text, /<mark>Kestrel<\/mark>|<mark>kestrel<\/mark>/);
+ assert.ok(!/private diary|Kestrel draft/.test(r.text));
+});
+
+t('a signed-in cookie adds that person\'s documents; a bad cookie searches as anonymous', async () => {
+ const mine = await html('/?q=kestrel', { Cookie: `ov_token=${userToken({ subjectId: alice, aud: ['openvibe.search'] })}` });
+ assert.match(mine.text, /Kestrel private diary/);
+ const bad = await html('/?q=kestrel', { Cookie: 'ov_token=not.a.jwt' });
+ assert.strictEqual(bad.status, 200);
+ assert.ok(!/private diary/.test(bad.text));
+ const badBearer = await html('/?q=kestrel', { Authorization: 'Bearer nope.nope.nope' });
+ assert.strictEqual(badBearer.status, 200, 'the page never answers 401');
+});
+
+t('paging with More results, filters carried along, and an honest empty state', async () => {
+ const p1 = await html('/?q=heron&owner=wiki');
+ const more = /<a class="more" href="\/\?([^"]+)">/.exec(p1.text);
+ assert.ok(more, 'a next-page link');
+ const qs = more[1].replace(/&/g, '&');
+ assert.match(qs, /owner=wiki/);
+ const p2 = await html(`/?${qs}`);
+ const titles = (s) => [...s.matchAll(/Heron sighting (\d+)/g)].map(m => m[1]);
+ assert.strictEqual(titles(p1.text).length, 20);
+ assert.strictEqual(titles(p2.text).length, 5);
+ assert.ok(!titles(p2.text).some(x => titles(p1.text).includes(x)));
+ const none = await html('/?q=nothingmatcheszzz');
+ assert.match(none.text, /Nothing found/);
+ const bad = await html('/?q=x&owner=Not%20Valid');
+ assert.strictEqual(bad.status, 400);
+ assert.match(bad.text, /owner is malformed/);
+});
+
+t('non-HTML clients get the text route index; robots.txt keeps result pages out', async () => {
+ const r = await request(svc.base, 'GET', '/');
+ assert.match(r.headers.get('content-type'), /text\/plain/);
+ assert.match(r.text, /GET {4}\/api\/v1\/search/);
+ assert.match(r.text, /saved-searches/);
+ const robots = await request(svc.base, 'GET', '/robots.txt');
+ assert.match(robots.text, /Disallow: \/\?/);
+ assert.match(robots.text, /Disallow: \/api\//);
+});
+
+t('shutdown', async () => { await svc.stop(); });
+
+t.run();
new file mode 100644
new file mode 100644
@@ -0,0 +1,237 @@
@@ -0,0 +1,237 @@
+'use strict';
+/**
+ * The removal purge queue (server/purge.js): the owners' removal feed, and the Cloudflare purge
+ * of formerly public URLs against a mocked Cloudflare API. No test talks to Cloudflare.
+ */
+const assert = require('assert');
+const { boot, request, serviceToken, doc, suite } = require('./helpers');
+const { load } = require('../server/config');
+const { zoneFor, purgeUrls, MAX_ATTEMPTS } = require('../server/purge');
+
+const t = suite('purge');
+const WIKI = serviceToken('wiki', ['search.document.write']);
+const BLOG = serviceToken('blog', ['search.document.write']);
+const TOKEN = 'cf-test-token-' + 'z'.repeat(24);
+const WIKI_ZONE = 'a'.repeat(32);
+const NET_ZONE = 'b'.repeat(32);
+const ZONES = `openvibe.wiki=${WIKI_ZONE},openvibe.network=${NET_ZONE}`;
+
+/** A mocked Cloudflare API: records calls, answers with `respond(call)` (default: success). */
+function mockCloudflare(respond = () => ({ status: 200, body: { success: true, errors: [], messages: [], result: { id: 'purge-1' } } })) {
+ const calls = [];
+ const fetchImpl = async (url, opts) => {
+ const call = { url, method: opts.method, headers: opts.headers, body: JSON.parse(opts.body) };
+ calls.push(call);
+ const r = await respond(call);
+ if (r instanceof Error) throw r;
+ return { status: r.status, ok: r.status >= 200 && r.status < 300, json: async () => r.body, text: async () => JSON.stringify(r.body) };
+ };
+ return { calls, fetchImpl };
+}
+
+const hide = (d, revision = d.revision + 1) => ({ ...d, revision, visibility: 'private', acl: { subjects: ['usr_01K5ZZZZZZZZZZZZZZZZZZZZZZ'] } });
+
+t('config: zone map parses host=zoneid pairs and refuses malformed ones', () => {
+ const c = load({ CLOUDFLARE_ZONE_IDS: ZONES });
+ assert.deepStrictEqual(c.purge.zones.map(z => z.host), ['openvibe.network', 'openvibe.wiki']);
+ assert.throws(() => load({ CLOUDFLARE_ZONE_IDS: 'openvibe.wiki=nothex' }), /CLOUDFLARE_ZONE_IDS/);
+ assert.throws(() => load({ CLOUDFLARE_ZONE_IDS: WIKI_ZONE }), /CLOUDFLARE_ZONE_IDS/);
+ assert.strictEqual(load({}).purge.cloudflareToken, '');
+ assert.deepStrictEqual(load({ CLOUDFLARE_PURGE_RELATED_PATHS: '/sitemap.xml,/feed.atom,javascript:x' }).purge.relatedPaths, ['/sitemap.xml', '/feed.atom']);
+});
+
+t('zoneFor matches the host or a parent domain, longest first; purgeUrls adds related paths', () => {
+ const zones = load({ CLOUDFLARE_ZONE_IDS: `${ZONES},docs.openvibe.network=${'c'.repeat(32)}` }).purge.zones;
+ assert.strictEqual(zoneFor('https://openvibe.wiki/p/x', zones), WIKI_ZONE);
+ assert.strictEqual(zoneFor('https://search.openvibe.network/x', zones), NET_ZONE);
+ assert.strictEqual(zoneFor('https://docs.openvibe.network/x', zones), 'c'.repeat(32));
+ assert.strictEqual(zoneFor('https://notopenvibe.wiki/x', zones), null, 'a suffix that is not a parent domain');
+ assert.strictEqual(zoneFor('ftp://openvibe.wiki/x', zones), null);
+ assert.deepStrictEqual(purgeUrls('https://openvibe.wiki/p/x#top', ['/sitemap.xml']), ['https://openvibe.wiki/p/x', 'https://openvibe.wiki/sitemap.xml']);
+});
+
+t('inert without CLOUDFLARE_PURGE_TOKEN: removals are recorded for owners, nothing is queued or fetched', async () => {
+ const cf = mockCloudflare();
+ const svc = await boot({ env: { CLOUDFLARE_ZONE_IDS: ZONES }, fetchImpl: cf.fetchImpl });
+ try {
+ const d = doc({ id: 'inert1', canonical_url: 'https://openvibe.wiki/p/inert1' });
+ svc.store.apply(d);
+ svc.store.apply(hide(d));
+ assert.deepStrictEqual(svc.purges.cdnCounts(), { pending: 0, purged: 0, failed: 0, skipped: 0 });
+ assert.deepStrictEqual(await svc.purger.flush(), { purged: 0, retrying: 0, failed: 0 });
+ assert.strictEqual(svc.purger.running(), false);
+ assert.strictEqual(cf.calls.length, 0);
+ const feed = await request(svc.base, 'GET', '/api/v1/owners/wiki/removals', { token: WIKI });
+ assert.strictEqual(feed.status, 200);
+ assert.strictEqual(feed.body.removals.length, 1);
+ const r = feed.body.removals[0];
+ assert.strictEqual(r.id, 'inert1');
+ assert.strictEqual(r.reason, 'visibility_changed');
+ assert.strictEqual(r.canonical_url, 'https://openvibe.wiki/p/inert1');
+ assert.strictEqual(r.previous_exposure, 'public_listed');
+ assert.strictEqual(r.exposure, 'restricted');
+ // The removal row and the outbox event are the same removal.
+ const ev = svc.outbox.all().find(e => e.event_type === 'search.document.removed' && e.payload.id === 'inert1');
+ assert.strictEqual(r.event_id, ev.event_id);
+ const ready = await request(svc.base, 'GET', '/api/ready');
+ assert.match(JSON.stringify(ready.body), /CLOUDFLARE_PURGE_TOKEN unset/);
+ } finally { await svc.stop(); }
+});
+
+t('owner removal feed: owner token only, own owner only, cursor paging, restricted removals carry no URL', async () => {
+ const svc = await boot();
+ try {
+ const members = doc({ id: 'mem1', visibility: 'members', acl: { groups: ['g1'] } });
+ svc.store.apply(members);
+ svc.store.remove('wiki', 'page', 'mem1');
+ const pub = doc({ id: 'pub2', canonical_url: 'https://openvibe.wiki/p/pub2' });
+ svc.store.apply(pub);
+ svc.store.remove('wiki', 'page', 'pub2');
+ assert.strictEqual((await request(svc.base, 'GET', '/api/v1/owners/wiki/removals')).status, 401);
+ assert.strictEqual((await request(svc.base, 'GET', '/api/v1/owners/wiki/removals', { token: BLOG })).status, 403);
+ const p1 = await request(svc.base, 'GET', '/api/v1/owners/wiki/removals?limit=1', { token: WIKI });
+ assert.deepStrictEqual(p1.body.removals.map(r => [r.id, r.reason, r.canonical_url]), [['mem1', 'deleted', null]]);
+ const p2 = await request(svc.base, 'GET', `/api/v1/owners/wiki/removals?after=${p1.body.next_after}`, { token: WIKI });
+ assert.deepStrictEqual(p2.body.removals.map(r => [r.id, r.reason, r.canonical_url]), [['pub2', 'deleted', 'https://openvibe.wiki/p/pub2']]);
+ const blog = await request(svc.base, 'GET', '/api/v1/owners/blog/removals', { token: BLOG });
+ assert.deepStrictEqual(blog.body.removals, []);
+ } finally { await svc.stop(); }
+});
+
+t('with a token: the formerly public URL and its sitemap are purged in one call per zone', async () => {
+ const cf = mockCloudflare();
+ const svc = await boot({ env: { CLOUDFLARE_PURGE_TOKEN: TOKEN, CLOUDFLARE_ZONE_IDS: ZONES }, fetchImpl: cf.fetchImpl });
+ try {
+ const a = doc({ id: 'cf1', canonical_url: 'https://openvibe.wiki/p/cf1' });
+ const b = doc({ id: 'cf2', canonical_url: 'https://openvibe.wiki/p/cf2' });
+ const c = doc({ owner: 'wiki', id: 'cf3', canonical_url: 'https://docs.openvibe.network/cf3' });
+ for (const d of [a, b, c]) svc.store.apply(d);
+ svc.store.apply(hide(a));
+ svc.store.remove('wiki', 'page', 'cf2');
+ svc.store.apply({ ...c, revision: 2, publication_state: 'retracted' });
+ // Two removals on openvibe.wiki share one pending sitemap purge.
+ assert.deepStrictEqual(svc.purges.cdnCounts(), { pending: 5, purged: 0, failed: 0, skipped: 0 });
+
+ const s = await svc.purger.flush();
+ assert.deepStrictEqual(s, { purged: 5, retrying: 0, failed: 0 });
+ assert.strictEqual(cf.calls.length, 2);
+ const wiki = cf.calls.find(x => x.url.includes(WIKI_ZONE));
+ assert.strictEqual(wiki.url, `https://api.cloudflare.com/client/v4/zones/${WIKI_ZONE}/purge_cache`);
+ assert.strictEqual(wiki.method, 'POST');
+ assert.strictEqual(wiki.headers.Authorization, `Bearer ${TOKEN}`);
+ assert.deepStrictEqual(wiki.body.files.sort(), ['https://openvibe.wiki/p/cf1', 'https://openvibe.wiki/p/cf2', 'https://openvibe.wiki/sitemap.xml']);
+ const net = cf.calls.find(x => x.url.includes(NET_ZONE));
+ assert.deepStrictEqual(net.body.files.sort(), ['https://docs.openvibe.network/cf3', 'https://docs.openvibe.network/sitemap.xml']);
+
+ // Done rows are never sent again; the token is never stored.
+ assert.deepStrictEqual(await svc.purger.flush(), { purged: 0, retrying: 0, failed: 0 });
+ assert.strictEqual(cf.calls.length, 2);
+ const dump = JSON.stringify(svc.db.prepare('SELECT * FROM cdn_purges').all()) + JSON.stringify(svc.db.prepare('SELECT * FROM removals').all());
+ assert.ok(!dump.includes(TOKEN));
+ assert.strictEqual(svc.purger.running(), true);
+ } finally { await svc.stop(); }
+});
+
+t('an index change that does not lower exposure, or a URL that was never public, purges nothing', async () => {
+ const cf = mockCloudflare();
+ const svc = await boot({ env: { CLOUDFLARE_PURGE_TOKEN: TOKEN, CLOUDFLARE_ZONE_IDS: ZONES }, fetchImpl: cf.fetchImpl });
+ try {
+ const d = doc({ id: 'keep', canonical_url: 'https://openvibe.wiki/p/keep' });
+ svc.store.apply(d);
+ svc.store.apply({ ...d, revision: 2, title: 'New title' });
+ const m = doc({ id: 'mem', visibility: 'members', acl: { groups: ['g'] }, canonical_url: 'https://openvibe.wiki/p/mem' });
+ svc.store.apply(m);
+ svc.store.remove('wiki', 'page', 'mem');
+ assert.deepStrictEqual(svc.purges.cdnCounts(), { pending: 0, purged: 0, failed: 0, skipped: 0 });
+ await svc.purger.flush();
+ assert.strictEqual(cf.calls.length, 0);
+ } finally { await svc.stop(); }
+});
+
+t('a host with no configured zone is recorded as skipped, not sent', async () => {
+ const cf = mockCloudflare();
+ const svc = await boot({ env: { CLOUDFLARE_PURGE_TOKEN: TOKEN, CLOUDFLARE_ZONE_IDS: ZONES }, fetchImpl: cf.fetchImpl });
+ try {
+ const d = doc({ id: 'elsewhere', canonical_url: 'https://example.org/x' });
+ svc.store.apply(d);
+ svc.store.remove('wiki', 'page', 'elsewhere');
+ assert.deepStrictEqual(svc.purges.cdnCounts(), { pending: 0, purged: 0, failed: 0, skipped: 2 });
+ await svc.purger.flush();
+ assert.strictEqual(cf.calls.length, 0);
+ const row = svc.db.prepare("SELECT detail FROM cdn_purges WHERE url = 'https://example.org/x'").get();
+ assert.match(row.detail, /no Cloudflare zone/);
+ } finally { await svc.stop(); }
+});
+
+t('429, 5xx and network errors back off and retry; after MAX_ATTEMPTS the purge fails', async () => {
+ let clock = Date.parse('2026-09-23T12:00:00Z');
+ let mode = 'rate';
+ const cf = mockCloudflare(() => {
+ if (mode === 'rate') return { status: 429, body: { success: false, errors: [{ code: 971, message: 'Please wait and consider throttling your request speed' }] } };
+ if (mode === 'down') return new Error('ECONNRESET');
+ return { status: 200, body: { success: true, errors: [], result: { id: 'ok' } } };
+ });
+ const svc = await boot({ env: { CLOUDFLARE_PURGE_TOKEN: TOKEN, CLOUDFLARE_ZONE_IDS: ZONES, CLOUDFLARE_PURGE_RELATED_PATHS: 'none' }, fetchImpl: cf.fetchImpl, now: () => clock });
+ try {
+ const d = doc({ id: 'retry1', canonical_url: 'https://openvibe.wiki/p/retry1' });
+ svc.store.apply(d);
+ svc.store.remove('wiki', 'page', 'retry1');
+ const n = svc.purges.cdnCounts().pending;
+ assert.strictEqual(n, 1, 'related paths off (none)');
+ assert.deepStrictEqual(await svc.purger.flush(), { purged: 0, retrying: n, failed: 0 });
+ // Not due yet: nothing is sent before the backoff elapses.
+ assert.deepStrictEqual(await svc.purger.flush(), { purged: 0, retrying: 0, failed: 0 });
+ assert.strictEqual(cf.calls.length, 1);
+ clock += 3000;
+ mode = 'ok';
+ assert.deepStrictEqual(await svc.purger.flush(), { purged: n, retrying: 0, failed: 0 });
+
+ // A URL that never gets through fails after MAX_ATTEMPTS.
+ mode = 'down';
+ const e = doc({ id: 'retry2', canonical_url: 'https://openvibe.wiki/p/retry2' });
+ svc.store.apply(e);
+ svc.store.remove('wiki', 'page', 'retry2');
+ for (let i = 0; i < MAX_ATTEMPTS; i++) { clock += 2 * 3600 * 1000; await svc.purger.flush(); }
+ const rows = svc.db.prepare("SELECT state, attempts, detail FROM cdn_purges WHERE url LIKE '%retry2'").all();
+ assert.deepStrictEqual(rows.map(r => [r.state, r.attempts]), [['failed', MAX_ATTEMPTS]]);
+ assert.match(rows[0].detail, /gave up after/);
+ } finally { await svc.stop(); }
+});
+
+t('a refused batch is retried one URL at a time: only the refused URL fails', async () => {
+ const cf = mockCloudflare((call) => (call.body.files.some(f => f.includes('refused'))
+ ? { status: 400, body: { success: false, errors: [{ code: 1012, message: 'Request must contain one of "purge_everything", "files", "tags", "hosts" or "prefixes"' }] } }
+ : { status: 200, body: { success: true, errors: [], result: { id: 'ok' } } }));
+ const svc = await boot({ env: { CLOUDFLARE_PURGE_TOKEN: TOKEN, CLOUDFLARE_ZONE_IDS: ZONES, CLOUDFLARE_PURGE_RELATED_PATHS: 'none' }, fetchImpl: cf.fetchImpl });
+ try {
+ for (const id of ['good1', 'refused', 'good2']) {
+ svc.store.apply(doc({ id, canonical_url: `https://openvibe.wiki/p/${id}` }));
+ svc.store.remove('wiki', 'page', id);
+ }
+ const s = await svc.purger.flush();
+ assert.deepStrictEqual(s, { purged: 2, retrying: 0, failed: 1 });
+ const failed = svc.db.prepare("SELECT url, detail FROM cdn_purges WHERE state = 'failed'").all();
+ assert.deepStrictEqual(failed.map(r => r.url), ['https://openvibe.wiki/p/refused']);
+ assert.match(failed[0].detail, /HTTP 400 1012/);
+ } finally { await svc.stop(); }
+});
+
+t('a token Cloudflare rejects (403) fails the purge without retrying forever and never logs the token', async () => {
+ const logged = [];
+ const cf = mockCloudflare(() => ({ status: 403, body: { success: false, errors: [{ code: 10000, message: `Authentication error for ${TOKEN}` }] } }));
+ const svc = await boot({ env: { CLOUDFLARE_PURGE_TOKEN: TOKEN, CLOUDFLARE_ZONE_IDS: ZONES, CLOUDFLARE_PURGE_RELATED_PATHS: 'none' }, fetchImpl: cf.fetchImpl });
+ svc.purger.stop();
+ const { createPurger } = require('../server/purge');
+ const purger = createPurger({ db: svc.db, config: svc.config.purge, fetchImpl: cf.fetchImpl, log: { warn: (m) => logged.push(m), log() {}, error() {} } });
+ try {
+ svc.store.apply(doc({ id: 'auth1', canonical_url: 'https://openvibe.wiki/p/auth1' }));
+ svc.store.remove('wiki', 'page', 'auth1');
+ assert.deepStrictEqual(await purger.flush(), { purged: 0, retrying: 0, failed: 1 });
+ const row = svc.db.prepare("SELECT state, detail FROM cdn_purges WHERE url LIKE '%auth1'").get();
+ assert.strictEqual(row.state, 'failed');
+ assert.ok(!row.detail.includes(TOKEN) && row.detail.includes('[token]'));
+ assert.ok(logged.length && logged.every(m => !m.includes(TOKEN)));
+ } finally { await svc.stop(); }
+});
+
+t.run();
new file mode 100644
new file mode 100644
@@ -0,0 +1,153 @@
@@ -0,0 +1,153 @@
+'use strict';
+/**
+ * Saved searches: only a signed-in person (usr_ subject) saves; each person sees only their own;
+ * a run is a fresh query as that person (the ACL applies at run time); cookie changes must come
+ * from this origin; the per-person limit holds.
+ */
+const assert = require('assert');
+const { ids } = require('openvibe-contracts');
+const { boot, request, serviceToken, userToken, doc, suite } = require('./helpers');
+
+const t = suite('saved-searches');
+const ORIGIN = 'https://search.openvibe.network';
+const DELEGATE = serviceToken('blog', ['search.query.delegate']);
+const NO_DELEGATE = serviceToken('blog', ['search.document.write']);
+
+const alice = ids.newId('user');
+const bob = ids.newId('user');
+const guest = ids.newId('guest');
+const as = (subjectId, role = 'user') => ({ token: userToken({ subjectId, role, aud: ['openvibe.search'] }) });
+
+let svc;
+const call = (method, p, opts = {}) => request(svc.base, method, p, opts);
+
+t('boot with a small per-person limit', async () => {
+ svc = await boot({ env: { BASE_URL: ORIGIN, SEARCH_SAVED_MAX_PER_SUBJECT: '3' } });
+ svc.store.apply(doc({ id: 'pub', title: 'Zebra migration public', canonical_url: 'https://openvibe.wiki/p/pub', facets: { category: 'animals' } }));
+ svc.store.apply(doc({ id: 'mem', visibility: 'members', acl: { subjects: [alice] }, title: 'Zebra migration for alice', facets: { category: 'animals' } }));
+ svc.store.apply(doc({ owner: 'blog', type: 'post', id: 'other', title: 'Zebra blog post', canonical_url: 'https://openvibe.blog/p/other' }));
+});
+
+t('anonymous callers, guests and services without delegation cannot use saved searches', async () => {
+ assert.strictEqual((await call('GET', '/api/v1/saved-searches')).status, 401);
+ assert.strictEqual((await call('POST', '/api/v1/saved-searches', { body: { q: 'zebra' } })).status, 401);
+ const g = await call('POST', '/api/v1/saved-searches', { token: DELEGATE, headers: { 'X-OV-Subject': guest }, body: { q: 'zebra' } });
+ assert.strictEqual(g.status, 403);
+ assert.strictEqual(g.body.code || g.body.type.split('/').pop(), 'search.sign_in_required');
+ assert.strictEqual((await call('GET', '/api/v1/saved-searches', { token: DELEGATE })).status, 401, 'a service with no subject is nobody');
+ assert.strictEqual((await call('GET', '/api/v1/saved-searches', { token: NO_DELEGATE, headers: { 'X-OV-Subject': alice } })).status, 403);
+});
+
+let saved;
+t('a person saves a query with filters; the same query again is the same saved search', async () => {
+ const r = await call('POST', '/api/v1/saved-searches', { ...as(alice), body: { name: 'Zebras', q: ' zebra migration ', owner: 'wiki', facets: { category: ['animals'] } } });
+ assert.strictEqual(r.status, 201, r.text);
+ saved = r.body.saved_search;
+ assert.match(saved.id, /^svs_[0-9a-z]{26}$/);
+ assert.strictEqual(saved.name, 'Zebras');
+ assert.strictEqual(saved.q, 'zebra migration');
+ assert.deepStrictEqual(saved.filters, { owner: 'wiki', facets: [['category', ['animals']]] });
+ assert.strictEqual(saved.last_run_at, null);
+ assert.ok(!('subject' in saved));
+
+ const again = await call('POST', '/api/v1/saved-searches', { ...as(alice), body: { name: 'Zebras (renamed)', q: 'zebra migration', facets: { category: 'animals' }, owner: 'wiki' } });
+ assert.strictEqual(again.status, 200);
+ assert.strictEqual(again.body.saved_search.id, saved.id);
+ assert.strictEqual(again.body.saved_search.name, 'Zebras (renamed)');
+
+ const list = await call('GET', '/api/v1/saved-searches', as(alice));
+ assert.strictEqual(list.status, 200);
+ assert.deepStrictEqual(list.body.saved_searches.map(s => s.id), [saved.id]);
+ assert.strictEqual(list.body.max, 3);
+ assert.strictEqual(list.headers.get('cache-control'), 'no-store');
+});
+
+t('running a saved search queries as the person now, with its filters', async () => {
+ const r = await call('GET', `/api/v1/saved-searches/${saved.id}/results`, as(alice));
+ assert.strictEqual(r.status, 200, r.text);
+ assert.deepStrictEqual(r.body.results.map(x => x.id).sort(), ['mem', 'pub'], 'owner=wiki filter applies; alice sees her members document');
+ assert.strictEqual(r.body.saved_search.id, saved.id);
+ const after = await call('GET', `/api/v1/saved-searches/${saved.id}`, as(alice));
+ assert.ok(after.body.saved_search.last_run_at);
+
+ // Access is decided at run time: alice loses the members document, it leaves her results.
+ svc.store.apply(doc({ id: 'mem', revision: 2, visibility: 'members', acl: { subjects: [bob] }, title: 'Zebra migration for alice', facets: { category: 'animals' } }));
+ const r2 = await call('GET', `/api/v1/saved-searches/${saved.id}/results`, as(alice));
+ assert.deepStrictEqual(r2.body.results.map(x => x.id), ['pub']);
+ // And a deleted public document leaves it too.
+ svc.store.remove('wiki', 'page', 'pub');
+ const r3 = await call('GET', `/api/v1/saved-searches/${saved.id}/results`, as(alice));
+ assert.deepStrictEqual(r3.body.results, []);
+});
+
+t("another person's saved search is the same 404 as a missing one, for read, run and delete", async () => {
+ for (const [m, p] of [['GET', `/api/v1/saved-searches/${saved.id}`], ['GET', `/api/v1/saved-searches/${saved.id}/results`], ['DELETE', `/api/v1/saved-searches/${saved.id}`]]) {
+ const theirs = await call(m, p, as(bob));
+ const missing = await call(m, p.replace(saved.id, 'svs_00000000000000000000000000'), as(bob));
+ assert.strictEqual(theirs.status, 404, `${m} ${p}`);
+ assert.strictEqual(missing.status, 404);
+ assert.strictEqual(theirs.body.detail, missing.body.detail);
+ }
+ assert.deepStrictEqual((await call('GET', '/api/v1/saved-searches', as(bob))).body.saved_searches, []);
+ assert.strictEqual((await call('GET', '/api/v1/saved-searches/not-an-id', as(alice))).status, 404);
+});
+
+t('bad saved searches are refused: nothing to search, malformed filters, oversize name', async () => {
+ const bad = async (body) => (await call('POST', '/api/v1/saved-searches', { ...as(alice), body })).status;
+ assert.strictEqual(await bad({}), 400);
+ assert.strictEqual(await bad({ q: ' ' }), 400);
+ assert.strictEqual(await bad({ q: '!!! ???' }), 400);
+ assert.strictEqual(await bad({ q: 'x', owner: 'Not Valid' }), 400);
+ assert.strictEqual(await bad({ q: 'x', facets: { 'Bad Key': ['v'] } }), 400);
+ assert.strictEqual(await bad({ q: 'x', facets: ['v'] }), 400);
+ assert.strictEqual(await bad({ q: 'x', facets: { k: [1] } }), 400);
+ assert.strictEqual(await bad({ q: 'x', name: 'n'.repeat(101) }), 400);
+ assert.strictEqual(await bad({ q: 5 }), 400);
+ assert.strictEqual(await bad({ q: 'x'.repeat(501) }), 400);
+ // Filters alone are a valid saved search (newest in a product).
+ const r = await call('POST', '/api/v1/saved-searches', { ...as(alice), body: { owner: 'blog' } });
+ assert.strictEqual(r.status, 201);
+ assert.strictEqual(r.body.saved_search.name, 'All documents');
+ const run = await call('GET', `/api/v1/saved-searches/${r.body.saved_search.id}/results`, as(alice));
+ assert.deepStrictEqual(run.body.results.map(x => x.id), ['other']);
+});
+
+t('a cookie-authenticated change must come from this origin; Bearer and delegated calls need no Origin', async () => {
+ const cookie = { headers: { Cookie: `ov_token=${as(alice).token}` } };
+ const noOrigin = await call('POST', '/api/v1/saved-searches', { ...cookie, body: { q: 'cookie one' } });
+ assert.strictEqual(noOrigin.status, 403);
+ const foreign = await call('POST', '/api/v1/saved-searches', { headers: { ...cookie.headers, Origin: 'https://evil.example' }, body: { q: 'cookie one' } });
+ assert.strictEqual(foreign.status, 403);
+ const sibling = await call('POST', '/api/v1/saved-searches', { headers: { ...cookie.headers, Origin: 'https://openvibe.wiki' }, body: { q: 'cookie one' } });
+ assert.strictEqual(sibling.status, 403);
+ const same = await call('POST', '/api/v1/saved-searches', { headers: { ...cookie.headers, Origin: ORIGIN }, body: { q: 'cookie one' } });
+ assert.strictEqual(same.status, 201);
+ // Reading with the cookie needs no Origin.
+ assert.strictEqual((await call('GET', '/api/v1/saved-searches', cookie)).status, 200);
+ const del = await call('DELETE', `/api/v1/saved-searches/${same.body.saved_search.id}`, cookie);
+ assert.strictEqual(del.status, 403);
+ assert.strictEqual((await call('DELETE', `/api/v1/saved-searches/${same.body.saved_search.id}`, { headers: { ...cookie.headers, Origin: ORIGIN } })).status, 204);
+
+ // A product saving for its signed-in visitor.
+ const d = await call('POST', '/api/v1/saved-searches', { token: DELEGATE, headers: { 'X-OV-Subject': bob }, body: { q: 'zebra' } });
+ assert.strictEqual(d.status, 201);
+ assert.deepStrictEqual((await call('GET', '/api/v1/saved-searches', as(bob))).body.saved_searches.map(s => s.id), [d.body.saved_search.id]);
+});
+
+t('the per-person limit holds, and deleting frees a slot', async () => {
+ // alice has 2 (zebra, owner=blog); the limit is 3.
+ assert.strictEqual((await call('POST', '/api/v1/saved-searches', { ...as(alice), body: { q: 'third' } })).status, 201);
+ const over = await call('POST', '/api/v1/saved-searches', { ...as(alice), body: { q: 'fourth' } });
+ assert.strictEqual(over.status, 409);
+ // Saving a query already saved is not a new one, even at the limit.
+ assert.strictEqual((await call('POST', '/api/v1/saved-searches', { ...as(alice), body: { q: 'third' } })).status, 200);
+ assert.strictEqual((await call('DELETE', `/api/v1/saved-searches/${saved.id}`, as(alice))).status, 204);
+ assert.strictEqual((await call('GET', `/api/v1/saved-searches/${saved.id}`, as(alice))).status, 404);
+ assert.strictEqual((await call('POST', '/api/v1/saved-searches', { ...as(alice), body: { q: 'fourth' } })).status, 201);
+ const ready = await call('GET', '/api/ready');
+ assert.strictEqual(ready.body.saved_searches, 4);
+});
+
+t('shutdown', async () => { await svc.stop(); });
+
+t.run();