openvibers/openvibe.search README.md
README.md
| 1 | # OpenVibe.Search |
| 2 | |
| 3 | > Permission-aware discovery for the network: the canonical index document, event-fed indexing, |
| 4 | > an ACL-filtered query API, and deletion/visibility propagation. |
| 5 | |
| 6 | **Docs:** https://openvibe.services · **For AI:** https://openvibe.network/llms.txt |
| 7 | **Run it:** `npm ci && npm run dev` (Node 22 or newer) · **Tests:** `npm test` |
| 8 | |
| 9 | **Status:** alpha (roadmap Wave 14). Runs on the production host (127.0.0.1:4710, since 2026-09-23) |
| 10 | and is fed through Events; on 2026-10-10 the index held, per the plan record, 1976 work jobs, 51 help articles and 70 inventory items (2097 documents) plus wiki tombstones; how many are public was not measured. |
| 11 | **Domain:** `search.openvibe.network`, served by this repository's vhost |
| 12 | ([deploy/nginx/](deploy/nginx/search.openvibe.network.conf); the service manifest records the search |
| 13 | page and query API as public since 2026-09-23. Install steps: [Public host](#public-host)) |
| 14 | **Plan:** OpenVibe End-to-End Realignment & Implementation Plan, revision 3 — roadmap §4.2 A, §15.12, §29, §32.3. |
| 15 | **License:** AGPL-3.0. |
| 16 | |
| 17 | ## Purpose |
| 18 | |
| 19 | Every publication product (Wiki, Blog, News, Reviews, Deals, Coupons, Trade) and every other |
| 20 | service with discoverable content tells Search about its resources with one contract — |
| 21 | [`search.index-document@1`](docs/contracts-proposal/) — and Search answers queries without ever |
| 22 | returning a document the caller may not see. The owner stays the source of truth; Search keeps a |
| 23 | revision-ordered copy and can be reconciled against it. |
| 24 | |
| 25 | ## Running it |
| 26 | |
| 27 | ```bash |
| 28 | npm install |
| 29 | cp .env.example .env |
| 30 | npm run dev # http://127.0.0.1:4710 |
| 31 | npm test # every test/*.test.js, a PGlite database per file, no network needed |
| 32 | ``` |
| 33 | |
| 34 | Node 22 in production (`fnm exec --using=22.22.1 npm test`). Production: `/opt/openvibe.search`, |
| 35 | env `/etc/openvibe/search.env`, unit [deploy/systemd/openvibe-search.service](deploy/systemd/openvibe-search.service), |
| 36 | database `ov_search` on the host's data role (ADR-035; `sudo /opt/openvibe.host/roles/data/add-service.sh search`; the full-text index rebuilds itself from the documents at boot), nginx [deploy/nginx/search.openvibe.network.conf](deploy/nginx/search.openvibe.network.conf) |
| 37 | (the public vhost exposes the search page, the GET query API, saved searches and health; the owner |
| 38 | API and the Events webhook are loopback-only, and owners and Events call `127.0.0.1:4710`). |
| 39 | |
| 40 | `GET /api/health` is liveness. `GET /api/ready` (openvibe-shared/ready) is 503 only when the |
| 41 | database (the documents and the full-text table) fails; a Network signing key that has not loaded |
| 42 | (anonymous queries still answer) and a full-text index out of step with the documents table degrade |
| 43 | it. It also reports document counts by exposure, the SDK outbox status (`enabled`, `pending`, |
| 44 | `rejected`, `last_error`) and whether the Events webhook |
| 45 | is on. `GET /metrics` (openvibe-shared/metrics) answers direct loopback callers only: golden signals |
| 46 | by route template, `search_documents{exposure}`, `search_documents_indexed{index}` and the outbox |
| 47 | backlog. |
| 48 | |
| 49 | Request telemetry is aggregated by route template, method and status class, never by raw URL, and |
| 50 | leaves out `/shared` files, static assets, health, readiness and `/metrics`. Once a minute, one |
| 51 | `http.request` sample per key plus the active-requests, p95 latency and event-loop-lag gauges is |
| 52 | written to this service's own `telemetry_samples` table (openvibe-sdk 0.46 telemetry, the |
| 53 | `platform.telemetry-sample@1` shape, created by `migrations/0003_telemetry.sql`), and rows older |
| 54 | than 14 days are deleted daily. `server/app.js` mounts the SDK's `telemetryMiddleware` (a no-op |
| 55 | until init); `server/index.js` calls `telemetry.init()` at production start, runs the 14-day |
| 56 | retention and flushes on stop. |
| 57 | |
| 58 | ## Query API |
| 59 | |
| 60 | ```http |
| 61 | GET /api/v1/search?q=solar+eclipse&owner=news&type=story&lang=en&facet.topic=space&facets=topic&limit=20&cursor=… |
| 62 | GET /api/v1/suggest?q=sol |
| 63 | GET /api/v1/documents/:owner/:type/:id |
| 64 | ``` |
| 65 | |
| 66 | - `q` is plain text: word tokens only, every word required, a trailing `*` makes the last word a |
| 67 | prefix. Query operators and column filters in `q` are neutralised. No `q` = newest first. |
| 68 | - Results carry `owner, type, id, revision, visibility, title, summary, canonical_url, facets, |
| 69 | language, authorship, provenance, published_at, updated_at, indexable` and a `snippet_html` |
| 70 | (escaped text with `<mark>`). Never the ACL. `next_cursor` is bound to the query that made it. |
| 71 | - `facets=k1,k2` returns value counts over exactly the visible matching set. There is no total count. |
| 72 | - Every response is `Cache-Control: no-store`. |
| 73 | |
| 74 | Who sees what: |
| 75 | |
| 76 | | Caller | Sees | |
| 77 | |---|---| |
| 78 | | anonymous (or a user token without `subject_id`) | public, published, indexable documents | |
| 79 | | Network user JWT (`ov_token` cookie or Bearer) | + restricted documents whose ACL names its subject or `role:<role>` | |
| 80 | | service token with `search.query.delegate` + `X-OV-Subject` | + the ACL matches of that `usr_`/`gst_` subject and the `X-OV-Groups` / `X-OV-Entitlements` the service vouches for | |
| 81 | |
| 82 | Drafts, unpublished, retracted and deleted documents are returned to nobody. A hidden document and |
| 83 | a missing one give the same 404. A Bearer that does not verify is a 401 (never downgraded to |
| 84 | anonymous); delegation headers from a browser are ignored. Visibility is enforced in the engine |
| 85 | SQL (the full-text table split by audience plus an ACL predicate) and re-checked per hit — see |
| 86 | [docs/adr-engine.md](docs/adr-engine.md). |
| 87 | |
| 88 | The search page at `/` runs the same query as HTML for browsers (no JavaScript; no-store; result |
| 89 | pages `noindex`; anonymous unless this host has an `ov_token` cookie). Other clients get a text |
| 90 | route index at `/`. |
| 91 | |
| 92 | The plain front page is the network's search front page (`server/web/page.js`, `server/web/front-data.js`): |
| 93 | example searches, the index in numbers (public listed documents, sites sending, kinds, added or updated |
| 94 | this week), shelves of the newest public documents of each kind (streams, blog posts, jobs, |
| 95 | hosting plans, coupons, forum threads, help articles, places, rental cities, quests, tools; read with |
| 96 | the same searcher and anonymous viewer as `GET /api/v1/search?owner=&type=`, so nothing private or draft |
| 97 | can appear; community pastes, whose titles an AI writes from a picture, and placeholder titles are not |
| 98 | shelved), the sources with what each holds, the API samples and the questions (JSON-LD WebSite with its |
| 99 | SearchAction, WebApplication, ItemList of the sources and FAQPage). It is read once per five minutes; |
| 100 | an empty shelf is left out, and without the numbers the sources fall back to the static list. |
| 101 | |
| 102 | ### Ranking |
| 103 | |
| 104 | Full-text results are ordered by a tier first, then full-text relevance (title ×10, summary ×4, body ×1) |
| 105 | multiplied by a freshness boost: |
| 106 | |
| 107 | ``` |
| 108 | tier = site × 10 + field field: title 3 > summary 2 > body 1 |
| 109 | score = tier × 100 + relevance × (1 + W × 2^(−age_days / H)) |
| 110 | W = SEARCH_FRESHNESS_WEIGHT (default 1) |
| 111 | H = SEARCH_FRESHNESS_HALF_LIFE_DAYS (default 30) |
| 112 | ``` |
| 113 | |
| 114 | A query naming a site ("openvibe live", "openvibe tools") puts that site's own records first: the site is |
| 115 | the host `openvibe.<word>` (subdomains too) or a help article under `/a/<host>/`, with Live's old |
| 116 | `openre.stream` name included. A word that matches a document's owner or type is not a text match (the |
| 117 | `owner=` filter is how an owner narrows a search), a title match always outranks a summary-only match, and |
| 118 | a summary-only match outranks a body-only one. Within a tier, a thin document (a username or a very short |
| 119 | title with no real summary: the recorded-stream VODs) ranks below one with a real title and summary, and |
| 120 | one page keeps one result per title and owner. |
| 121 | |
| 122 | A tie in `score` is broken by Live's activity, then by recency (`documents.sort_at`, i.e. `updated_at`, |
| 123 | else `published_at`): a channel that is live now comes before one that is not, a channel with more |
| 124 | followers before one with fewer, and a VOD with a longer duration before a shorter one (Search is not |
| 125 | sent views). A channel is never "thin": its handle title and service summary are normal, so the thin |
| 126 | sink stays for the VODs it was written for. A Live channel whose only match is the service sentence |
| 127 | "<name> streams on OpenVibe.Live." is tied with every other such channel inside its field tier: the |
| 128 | sentence matches all of them, and the length-normalized `ts_rank` would otherwise rank a busier channel |
| 129 | lower because its body holds more past stream titles. A channel whose *title* matches still comes first, |
| 130 | like any other title match. Only Live publishes an activity signal; for other owners the tie-break is |
| 131 | just recency. `test/ranking.test.js` seeds tied channels for both rules. |
| 132 | |
| 133 | `age_days` counts from `published_at`, else `updated_at` (an edit does not make an old document |
| 134 | new); a future date counts as today and a missing one gets no boost. With the defaults a document |
| 135 | published today counts ×2.0, 30 days old ×1.5, 60 days ×1.25, 90 days ×1.125 and a year old ×1.0002: |
| 136 | relevance still decides, and between comparably relevant documents the newer one ranks first. |
| 137 | `W = 0` turns it off. Suggestions use the same score; browsing without `q` stays newest first. A |
| 138 | cursor carries its first page's clock, so later pages rank on the same scores. The hybrid path fuses the |
| 139 | keyword and vector top 100 by RRF and then applies the same tiers to the fused list |
| 140 | (`test/freshness.test.js`, `test/ranking.test.js`). |
| 141 | |
| 142 | ### Saved searches |
| 143 | |
| 144 | ```http |
| 145 | GET /api/v1/saved-searches |
| 146 | POST /api/v1/saved-searches { "name": "Eclipses", "q": "solar eclipse", "owner": "news", "facets": { "topic": ["space"] } } |
| 147 | GET /api/v1/saved-searches/:id |
| 148 | DELETE /api/v1/saved-searches/:id |
| 149 | GET /api/v1/saved-searches/:id/results?limit=&cursor= |
| 150 | ``` |
| 151 | |
| 152 | A signed-in person (a Network user token with a `usr_` subject, as Bearer or the `ov_token` |
| 153 | cookie) or a first-party service with `search.query.delegate` acting for `X-OV-Subject: usr_…` |
| 154 | keeps up to `SEARCH_SAVED_MAX_PER_SUBJECT` (50) queries: words, filters or both. Saving the same |
| 155 | query again returns the existing one (renamed if a new name is given). Only the query is stored: |
| 156 | every run is a fresh query as that person at that moment, so what they lost access to (or what was |
| 157 | deleted) is gone and what they gained appears. Someone else's saved search is the same 404 as a |
| 158 | missing one; guests cannot save; a cookie-authenticated POST/DELETE must carry this origin's |
| 159 | `Origin`. |
| 160 | |
| 161 | **Notifications of new matches** ([server/saved-notifier.js](server/saved-notifier.js)) are off |
| 162 | unless `SEARCH_SAVED_NOTIFY=1`; they also need `OV_OAUTH_CLIENT_SECRET` (the Network OAuth client |
| 163 | `search`, client credentials) and stay off without it, which `/api/ready` reports as |
| 164 | `saved_notify: "off (…)"`, never as a failure. Every `SEARCH_SAVED_NOTIFY_INTERVAL_MS` (15 min) the |
| 165 | notifier takes up to `SEARCH_SAVED_NOTIFY_BATCH` (100) saved searches, longest waiting first, and runs |
| 166 | each as its owner for documents indexed (a new revision) after its watermark, `last_run_at` (else |
| 167 | `created_at`). A search with new hits gets one notification through OpenVibe.Network via |
| 168 | [server/network-push.js](server/network-push.js), an adapter over `openvibe-sdk/notifications` (type |
| 169 | `SEARCH_SAVED_MATCH`, service `search`, category `service`: how many, the top title, a link to the |
| 170 | search page with that query); the person is resolved to Network's user id first |
| 171 | (`GET /internal/identity/resolve`, then `POST /internal/notifications/push`). The watermark moves to |
| 172 | the tick's start only once the push went out (or there was nothing new, or Network no longer knows |
| 173 | the person): a failed push is retried on the next tick and a delivered batch is never notified |
| 174 | twice. Opening the results moves the watermark too, so what the person has seen is not announced. |
| 175 | Network must grant the principal `search` `network.notifications.push` (for its own service) and |
| 176 | `identity.subject.resolve`. The Network OAuth client (`search`) is provisioned on Network with |
| 177 | `node server/setup/service-principal.js create search --env-file /etc/openvibe/search.env` (run in the |
| 178 | OpenVibe.Network checkout with its `DATABASE_URL`, under sudo so the env file is written 0600; then restart |
| 179 | Search so it reads `OV_OAUTH_CLIENT_ID` and `OV_OAUTH_CLIENT_SECRET`), and |
| 180 | saved-search notifications are switched on here with `SEARCH_SAVED_NOTIFY=1`; until someone signs |
| 181 | in that way, saved searches are used through Bearer tokens or a product acting for its visitor. |
| 182 | |
| 183 | ## Search Fabric |
| 184 | |
| 185 | One search over several scopes, routed the cheapest way that answers (`search.fabric.query`): |
| 186 | |
| 187 | | Scope | What it reads | |
| 188 | |---|---| |
| 189 | | `network` | Search's own index, with the same visibility rules as `GET /api/v1/search` | |
| 190 | | `mine` | the same search kept to the caller's non-public documents (needs a signed-in person) | |
| 191 | | `web` | OpenVibe's open index, then the paid/free web providers in a ladder | |
| 192 | | `news`, `video`, `products`, `places`, `code` | facets of the same index, narrowed to the owner and type Search holds | |
| 193 | | `images`, `people` | nothing yet: `skipped: "not indexed yet"` in `route.explain` | |
| 194 | |
| 195 | The web ladder is `open_index` first (free, the configured `SEARCH_OPEN_INDEX_OWNERS`), then |
| 196 | `deepseek` (paid), then `wikipedia` (free). Every step says what happened in `route.explain`: `ok`, |
| 197 | `count`, `ms`, `cached`, `cost_usd`, or `skipped` with the reason. Depth `quick` reads the index and |
| 198 | the cache only; `standard` stops climbing once it has `min(limit, 3)` results; `deep` runs every |
| 199 | step and merges them, de-duplicating by URL. |
| 200 | |
| 201 | Only a signed-in person, or a first-party service, may trigger a paid provider. |
| 202 | Anonymous callers get the open index and cached answers; the cache key is the normalized query, |
| 203 | scope, step, freshness and site, so the same question typed two ways is one entry. DeepSeek calls are |
| 204 | capped per day (`SEARCH_WEB_DAILY`, default 500); past the cap the step is skipped with |
| 205 | `daily cap reached`. Cache TTLs: `day` freshness 1 h, `week` 6 h, otherwise 24 h. |
| 206 | |
| 207 | ```http |
| 208 | POST /api/v1/query { "q": "solar eclipse", "scopes": ["network", "web"], "depth": "standard" } |
| 209 | GET /api/v1/query?q=solar+eclipse&scopes=network,web |
| 210 | GET /api/v1/query/providers |
| 211 | ``` |
| 212 | |
| 213 | The GET query string takes `q`, `scopes` (comma-separated), `freshness`, `depth`, `limit`, `cursor`, |
| 214 | `owner`, `type`, `lang` and `site`. GET and `/providers` answer `Access-Control-Allow-Origin: *` |
| 215 | (anonymous rules apply); POST does not. Results are grouped in the requested scope order, at most |
| 216 | `limit` per scope and 100 in all; `citations` number the web results and `next_cursor` pages the |
| 217 | network or mine scope when it was the only one asked. The answer never contains the query text. |
| 218 | |
| 219 | ```json |
| 220 | { |
| 221 | "results": [ |
| 222 | { "scope": "network", "kind": "document", "title": "Eclipse timings", "url": "https://openvibe.news/s/1", "site": "openvibe.news", "owner": "news", "type": "story", "id": "story_1" }, |
| 223 | { "scope": "web", "kind": "page", "title": "Solar eclipse", "url": "https://en.wikipedia.org/wiki/Solar_eclipse", "site": "en.wikipedia.org", "provider": "wikipedia", "retrieved_at": "2026-10-28T12:00:00.000Z" } |
| 224 | ], |
| 225 | "groups": { "network": 1, "web": 1 }, |
| 226 | "answer": null, |
| 227 | "citations": [{ "n": 1, "url": "https://en.wikipedia.org/wiki/Solar_eclipse", "title": "Solar eclipse", "provider": "wikipedia", "retrieved_at": "2026-10-28T12:00:00.000Z" }], |
| 228 | "route": { |
| 229 | "explain": [ |
| 230 | { "scope": "network", "step": "index", "ok": true, "count": 1, "ms": 4 }, |
| 231 | { "scope": "web", "step": "open_index", "ok": true, "count": 0, "ms": 3 }, |
| 232 | { "scope": "web", "step": "deepseek", "ok": false, "count": 0, "skipped": "paid providers need a signed-in person" }, |
| 233 | { "scope": "web", "step": "wikipedia", "ok": true, "count": 1, "ms": 190, "cost_usd": 0 } |
| 234 | ], |
| 235 | "cost_usd": 0 |
| 236 | }, |
| 237 | "next_cursor": null |
| 238 | } |
| 239 | ``` |
| 240 | |
| 241 | Asking `"answer": true` answers `answer: null` and adds an `answer` explain row: answers are not on |
| 242 | yet. |
| 243 | |
| 244 | ## Code search |
| 245 | |
| 246 | `GET /api/v1/search/code?q=&lang=&repo=&licence=&limit=&cursor=` merges public code from GitHub |
| 247 | (`GITHUB_CODE_TOKEN`; 10 API requests/minute), Sourcegraph's public stream and our own forge |
| 248 | (public repositories only; one `git grep` per repository, because the forge has no global code |
| 249 | route). The same file at the same commit is one result; exact identifier matches rank first, then |
| 250 | stars and recency. Every result is text with a `permalink` to the source's own https host; a private |
| 251 | repository is dropped by the source, never just hidden at display. At most 50 results per page; |
| 252 | `next_cursor` is bound to the filters. The route is public and limited per address/person (30 a |
| 253 | minute, 600 an hour). |
| 254 | |
| 255 | `POST /api/v1/search/code/deep` `{q, lang?, repo?, licence?}` is for a signed-in person. A cheap |
| 256 | model rewrites the question into at most five code queries, the top files are fetched, and a second |
| 257 | cheap model answers with `file:line` citations. The budget is two AI runs per question, at most |
| 258 | `SEARCH_CODE_DEEP_FILES` files and `SEARCH_CODE_DEEP_TOKENS` answer tokens; the response carries the |
| 259 | runs' real tokens and `cost_usd`. It is `503 search.ai_unavailable` when the AI client secret or |
| 260 | `SEARCH_CODE_AI_URL` is unset, and `SEARCH_CODE_DEEP=off` turns it off. AI runs are the metered |
| 261 | `ai.complete` path (the default route is chosen by price and health, so the cheap model answers |
| 262 | first). |
| 263 | |
| 264 | The sources are a small registry: one module per source with a manifest, a per-minute quota, health |
| 265 | and an on/off switch. The owner console at `/console/code-sources` (Network `role:admin`/`role:owner`) |
| 266 | shows each source's switch, key **name** (never its value), quota and health; the JSON is |
| 267 | `GET /api/v1/code/sources` for a service token with `search.code.sources`, and |
| 268 | `PUT /api/v1/code/sources/:id` `{enabled}` flips it. Switches live in `code_source_settings` |
| 269 | (migration 0009) and survive a restart. How to add a source, with the test it must pass: |
| 270 | [docs/code-sources.md](docs/code-sources.md). |
| 271 | |
| 272 | ## MCP |
| 273 | |
| 274 | The same fabric is a [Model Context Protocol](https://modelcontextprotocol.io) server over |
| 275 | Streamable HTTP at **`https://search.openvibe.network/mcp`** (JSON-RPC 2.0, protocol revision |
| 276 | 2025-06-18). It is stateless: one `POST`, one answer, no session id. `GET` and `DELETE` answer 405. |
| 277 | |
| 278 | | Tool | What it does | |
| 279 | |---|---| |
| 280 | | `openvibe_search` | one search over `network`, `mine`, `web`, `news`, `video`, `products`, `places` and `code`; results grouped by scope with `route.explain` and cost | |
| 281 | | `openvibe_search_providers` | the web ladder's providers, with price per call and whether each is available | |
| 282 | |
| 283 | Add it to an MCP client: |
| 284 | |
| 285 | ```json |
| 286 | { |
| 287 | "mcpServers": { |
| 288 | "openvibe-search": { |
| 289 | "type": "http", |
| 290 | "url": "https://search.openvibe.network/mcp" |
| 291 | } |
| 292 | } |
| 293 | } |
| 294 | ``` |
| 295 | |
| 296 | Anonymous callers search the public network and the open web index. A Bearer token (a Network user |
| 297 | token) adds the caller's own `mine` scope and lets the `web` scope reach DeepSeek; it is also the |
| 298 | person the fabric's visibility rules apply to. Each `tools/call` shares the fabric's per-actor |
| 299 | `search.fabric` limit (30 a minute, 600 an hour); `initialize` and `tools/list` are not limited. |
| 300 | |
| 301 | ## Indexing |
| 302 | |
| 303 | Owners call with an OpenVibe.Network client-credentials token (`audience=openvibe.search`) holding |
| 304 | `search.document.write`, and may only touch documents whose `owner` is their own service slug. |
| 305 | |
| 306 | ```http |
| 307 | PUT /api/v1/documents/wiki/page/pg_123 { …search.index-document@1… } |
| 308 | DELETE /api/v1/documents/wiki/page/pg_123?revision=8 |
| 309 | GET /api/v1/owners/wiki/documents?type=page&after=0&limit=500 # reconciliation |
| 310 | GET /api/v1/owners/wiki/documents/page/pg_123 # stored doc + effective indexability |
| 311 | GET /api/v1/owners/wiki/rejections # refused index events |
| 312 | ``` |
| 313 | |
| 314 | Or, preferred, through Events: publish `wiki.index_document.upserted` (payload = the document) or |
| 315 | `wiki.index_document.deleted` (payload = `{type, id, revision}`) from the owner's outbox with |
| 316 | `visibility: "internal"`. Search subscribes to `*.index_document.*` (`npm run subscribe`) and |
| 317 | consumes deliveries at `POST /internal/events`: HMAC-verified (`SEARCH_EVENTS_SECRET`), one inbox |
| 318 | receipt per event in the same transaction as the change, so a redelivery never applies twice. A |
| 319 | source may only write its own owner's documents (`SEARCH_EVENT_OWNERS` limits which sources feed |
| 320 | the index at all). |
| 321 | |
| 322 | Revision order, on both paths: an older revision is `stale` and changes nothing; the same revision |
| 323 | again is `unchanged`; different content at the same revision is a `conflict` (first write wins); at |
| 324 | equal revision a deletion wins; a tombstone keeps the revision, so a late older upsert cannot bring |
| 325 | the document back; a higher revision restores it. Search only makes indexability stricter (drafts, |
| 326 | private/members/unlisted, unpublished, stub-provider provenance and public documents without a |
| 327 | canonical URL are always `noindex`, with the reason recorded). |
| 328 | |
| 329 | Every change that lowers a document's exposure (deleted, drafted, unpublished, made |
| 330 | private/members/unlisted, or public → noindex) leaves results in the same transaction and emits |
| 331 | `search.document.removed` (with the canonical URL if it had been public, so caches and sitemaps |
| 332 | purge); every accepted upsert emits `search.document.indexed`. Both go through a transactional |
| 333 | outbox (`openvibe-sdk/events`, table `service_outbox`) relayed to Events when `EVENTS_URL` and |
| 334 | `OV_OAUTH_CLIENT_SECRET` are set. Pending rows wait when the relay is off. |
| 335 | |
| 336 | ### Removals, caches and the CDN |
| 337 | |
| 338 | Search consumes its own removals ([server/purge.js](server/purge.js)); in the same transaction as |
| 339 | the `search.document.removed` event it records: |
| 340 | |
| 341 | - **the owner's removal feed** — `GET /api/v1/owners/:owner/removals?after=&limit=` (owner token, |
| 342 | own owner only): `seq, event_id, type, id, revision, reason, previous_exposure, exposure, |
| 343 | canonical_url, at`. Products own their pages, caches and sitemaps, so they poll this feed (or |
| 344 | subscribe to `search.document.removed`) and drop the document from anything derived they keep. |
| 345 | - **a Cloudflare purge queue** — only when `CLOUDFLARE_PURGE_TOKEN` is set (inert otherwise: |
| 346 | nothing is queued or sent). For a removal whose URL had been public, the canonical URL and the |
| 347 | `CLOUDFLARE_PURGE_RELATED_PATHS` on its host (default `/sitemap.xml`) are queued once each and |
| 348 | purged by URL (`POST /zones/:zone/purge_cache {"files": […]}`, at most 30 per call) in the zone |
| 349 | `CLOUDFLARE_ZONE_IDS` maps the host (or its nearest parent domain) to. A host with no zone is |
| 350 | recorded as `skipped`. 429, 5xx and network errors back off and retry (8 attempts); a refusal is |
| 351 | retried one URL at a time and then marked `failed`. The token needs only *Zone → Cache Purge → |
| 352 | Purge* on those zones; it is read from the environment, sent only in the Authorization header, |
| 353 | and never stored or logged. `/api/ready` reports the purge state and counts, `/metrics` the |
| 354 | `search_cdn_purges{state}` gauge (`test/purge.test.js`, against a mocked Cloudflare API). |
| 355 | |
| 356 | ## Owns |
| 357 | |
| 358 | - the index document contract (`search.index-document@1`, released in openvibe-contracts v0.12.0; proposal in [docs/contracts-proposal/](docs/contracts-proposal/)) |
| 359 | - `documents`, `doc_acl`, `doc_facets`, `search_fts` (the full-text index), `idempotency_receipts`, |
| 360 | `ingest_rejections`, `service_outbox` (the SDK outbox), `removals`, `cdn_purges`, `saved_searches` (PostgreSQL, schema in |
| 361 | [migrations/](migrations/); engine decision in [docs/adr-engine.md](docs/adr-engine.md)) |
| 362 | - `event_outbox` remains for the N-1 rollback window; the expand migration copies its pending rows to `service_outbox`. |
| 363 | - the query API, its ranking and ACL filtering, the search page and saved searches; deletion and |
| 364 | visibility-change propagation, including the CDN purge of removed public URLs |
| 365 | |
| 366 | ## Does not own |
| 367 | |
| 368 | - any product's content, publication state or indexability decision (owners and the publishing packages do) |
| 369 | - sitemaps, feeds or page caches (each product; they read the removal feed or listen to |
| 370 | `search.document.removed`). Search only purges the CDN copy of a removed public URL. |
| 371 | - identity, groups or entitlements (Network, products, VIP/Billing — Search only matches the keys it is given) |
| 372 | |
| 373 | ## Depends on |
| 374 | |
| 375 | - OpenVibe.Contracts (service tokens, problem details, ids, the event envelope) |
| 376 | - PostgreSQL 18 (OpenVibe.Host `roles/data/`, ADR-035): documents and the full-text index, async through `openvibe-sdk/db` |
| 377 | - OpenVibe.Network (signing key; service principal `search`) |
| 378 | - OpenVibe.Events (index-document deliveries in, `search.document.*` out) |
| 379 | |
| 380 | ## Capabilities (released in openvibe-contracts v0.12.0; proposal in [docs/capabilities-proposal/](docs/capabilities-proposal/)) |
| 381 | |
| 382 | Implemented here (the service manifest's `capabilities`): |
| 383 | |
| 384 | | Capability | Routes | |
| 385 | |---|---| |
| 386 | | `search.document.write` | `PUT/DELETE /api/v1/documents/:owner/:type/:id`, `/api/v1/owners/:owner/...` including the removal feed (own owner only) | |
| 387 | | `search.query.delegate` | the query and saved-search routes, acting for `X-OV-Subject` | |
| 388 | | `search.query.run` | `GET /api/v1/search`, `/api/v1/suggest`, `/api/v1/documents/:owner/:type/:id` as anyone: anonymous or with a person's Network token (no grant needed; a service token must hold `search.query.delegate`) | |
| 389 | | `search.code.sources` | `GET/PUT /api/v1/code/sources`, `GET/POST /console/code-sources` (owner console) | |
| 390 | |
| 391 | Called elsewhere, as the service principal `search` (client credentials from OpenVibe.Network): |
| 392 | |
| 393 | | Service | Grant | Why | |
| 394 | |---|---|---| |
| 395 | | OpenVibe.Events | `events.event.publish` (audience `openvibe.events`) | the outbox relays `search.document.indexed` / `search.document.removed` | |
| 396 | | OpenVibe.Events | `events.subscription.manage` (`npm run subscribe`, once) | the HMAC-signed subscription to `*.index_document.*` that delivers index events | |
| 397 | | OpenVibe.Network | `identity.subject.resolve` (audience `openvibe.network`) | saved-search notifications: a `usr_` subject to Network's user id (only with `SEARCH_SAVED_NOTIFY=1`) | |
| 398 | | OpenVibe.Network | `network.notifications.push` (audience `openvibe.network`, own service `search`) | saved-search notifications: one notification per saved search with new hits | |
| 399 | | OpenVibe.AI | `ai.run.create` (audience `openvibe.ai`, namespace `ai`) | deep code search: two `ai.complete` runs per signed-in question | |
| 400 | |
| 401 | Registered in `openvibe-contracts` with the service manifest (this repo pins v0.96.0); |
| 402 | [server/auth.js](server/auth.js) decides them with the contracts grant rule, and CI runs |
| 403 | `openvibe-contracts-check --service search`. |
| 404 | |
| 405 | ### Per-actor limits |
| 406 | |
| 407 | Every API route also limits who calls it, after its credential and before any work |
| 408 | ([server/actor-limits.js](server/actor-limits.js), openvibe-sdk/limits, roadmap WS-R task 4): the |
| 409 | service principal on owner routes; on query and saved-search routes the person (their own token, or |
| 410 | the `X-OV-Subject` a delegating service vouches for), else the address. A service querying for its |
| 411 | signed-out visitors is not counted (it speaks for all of them). Past a limit: `429` problem+json |
| 412 | `rate_limited` with `Retry-After`, one `[limits]` log line and `search_rate_limited_total{limit,window}`. |
| 413 | |
| 414 | | Route | Per caller | |
| 415 | |---|---| |
| 416 | | Queries, direct gets, owner reconciliation reads, saved-search reads and runs | `SEARCH_LIMITS_MINUTE` / `SEARCH_LIMITS_HOUR` (120 a minute, 3000 an hour) | |
| 417 | | `POST/GET /api/v1/query` (`search.fabric`, a paid-provider ladder) | 30 / 600 | |
| 418 | | `GET /api/v1/query/providers` | `SEARCH_LIMITS_MINUTE` / `SEARCH_LIMITS_HOUR` | |
| 419 | | `GET /api/v1/suggest` | 300 / 6000 (a search box as the person types) | |
| 420 | | `PUT/DELETE /api/v1/documents/…` | 120 / 1200 (a catalog sync is a burst; bulk content comes through Events) | |
| 421 | | Saved-search create, delete | 30 / 300 | |
| 422 | |
| 423 | Never limited: the signed Events deliveries (`POST /internal/events`), `/api/health`, `/api/ready`, |
| 424 | `/release.json`, `/metrics`; the HTML page keeps nginx's address limits. `test/actor-limits.test.js`. |
| 425 | |
| 426 | ## Acceptance (tests) |
| 427 | |
| 428 | - a private or draft document is never returned to an unauthorized query — hits, snippets, facet |
| 429 | counts, suggestions, direct gets, cursors and full-text-syntax injection (`test/acl-leak.test.js`) |
| 430 | - a visibility change or deletion removes the document from results immediately and emits |
| 431 | `search.document.removed` (`test/acl-leak.test.js`) |
| 432 | - revision ordering, tombstones, restore, reconciliation (`test/documents.test.js`) |
| 433 | - signed deliveries, inbox dedupe, out-of-order delivery, crash before commit redelivers once, |
| 434 | owner spoofing refused (`test/webhook-inbox.test.js`) |
| 435 | - outbox relay to Events, retry and poison isolation (`test/outbox.test.js`) |
| 436 | - proposals valid against the contracts schemas (`test/proposals.test.js`) |
| 437 | - removals reach the owner feed; the CDN purge is inert without a token and, with one, purges the |
| 438 | formerly public URL and its sitemap per zone, retries, isolates refusals and never stores the |
| 439 | token (`test/purge.test.js`) |
| 440 | - freshness decay, newer-first between equals, relevance over age, stable cursors (`test/freshness.test.js`) |
| 441 | - saved searches: signed-in only, own only, run-time ACL, Origin check for cookies, limits |
| 442 | (`test/saved-searches.test.js`) |
| 443 | - the Search Fabric: network visibility, mine for a person, the web ladder, cache hits, the daily |
| 444 | cap, provider offers and GET/POST parity (`test/fabric.test.js`) |
| 445 | - saved-search notifications: one push per search with new hits, as its owner, the watermark only |
| 446 | after a push, a failed push retried, never twice, off without the flag and the secret |
| 447 | (`test/saved-notify.test.js`, `test/network-push.test.js`) |
| 448 | - the search page: public only, escaped, no-store, noindex results, paging (`test/page.test.js`) |
| 449 | - code search: three sources merged and de-duplicated, exact identifier first, filters and cursor, |
| 450 | private repositories dropped, https permalinks, no key in an answer (`test/code-search.test.js`, |
| 451 | `test/code-sources.test.js`); deep mode signed-in, two AI runs at most, cited answer, real cost, |
| 452 | AI failure never partial (`test/code-deep.test.js`); the owner switch and console (`test/code-console.test.js`) |
| 453 | - per-actor limits: 429 `rate_limited` with Retry-After per principal, person or address, another |
| 454 | caller unaffected, deliveries and health never limited (`test/actor-limits.test.js`) |
| 455 | |
| 456 | As of 2026-10-10, work, help and inventory documents are indexed in production. The first wiki |
| 457 | upserts were rejected (`owner_not_accepted`, since fixed), and only the wiki tombstones landed. |
| 458 | Wiki pages enter the index once a person reviews them; Sources has every seed disabled. |
| 459 | The removal feed and the CDN purge queue exist, but no product polls the feed yet and the purge needs a token with Cache Purge permission (the current Cloudflare |
| 460 | token has DNS and rulesets scope only). |
| 461 | |
| 462 | Restore drill: `ovhost drill search` passed on the production host on 2026-09-23 (integrity check, |
| 463 | readiness, identical query answers, row counts; see OpenVibe.Host `docs/restore-drills.md`). |
| 464 | |
| 465 | ## Security |
| 466 | |
| 467 | Reporting: [SECURITY.md](SECURITY.md). The rules the code keeps: |
| 468 | |
| 469 | - **Auth.** Tokens are RS256 JWTs signed by OpenVibe.Network (verified through the SDK's JWKS client at `/api/.well-known/jwks`): |
| 470 | service tokens need audience `openvibe.search` and the capability for the route, and an owner may |
| 471 | only write documents under its own service slug. A Bearer that does not verify is a 401, never |
| 472 | anonymous. Delegation headers (`X-OV-Subject`, `X-OV-Groups`, `X-OV-Entitlements`) count only with a |
| 473 | `search.query.delegate` service token. A cookie-authenticated write must carry this origin's `Origin`. |
| 474 | - **Private data.** Visibility is enforced in the engine SQL and re-checked per hit; drafts, unpublished, |
| 475 | retracted and deleted documents reach nobody, a hidden document is the same 404 as a missing one, |
| 476 | results never carry the ACL, and every response is `no-store`. Saved searches store the query only. |
| 477 | - **Network exposure.** The owner API, `/internal/events` and document writes are loopback-only in the |
| 478 | vhost; `/metrics` answers direct loopback callers only; client addresses come from `$remote_addr`. |
| 479 | - **Egress.** Search calls only OpenVibe.Network (keys, tokens, and with `SEARCH_SAVED_NOTIFY=1` identity resolve and notification push), OpenVibe.Events (`EVENTS_URL`), |
| 480 | when `CLOUDFLARE_PURGE_TOKEN` is set the Cloudflare purge API, the three code-source hosts |
| 481 | (`api.github.com`/`raw.githubusercontent.com`, `SOURCEGRAPH_URL`, `FORGE_URL`), and OpenVibe.AI |
| 482 | (`SEARCH_CODE_AI_URL`) for deep code search. It never fetches a URL a user chose. |
| 483 | - **Secrets.** `OV_OAUTH_CLIENT_SECRET`, `SEARCH_EVENTS_SECRET`, `CLOUDFLARE_PURGE_TOKEN`, |
| 484 | `GITHUB_CODE_TOKEN`, `SOURCEGRAPH_TOKEN` and `FORGE_TOKEN` come from the environment |
| 485 | (`/etc/openvibe/search.env`, 0600) and are never stored, returned or logged; the owner console shows |
| 486 | the variable name. A GitHub token for the code source should be a fine-grained public-only token. |
| 487 | Event deliveries are HMAC-verified before anything is read. |
| 488 | |
| 489 | ## Deploy |
| 490 | |
| 491 | Production deploys with `sudo ovhost deploy search` on the host (strategy `git-checkout`: fetch, |
| 492 | fast-forward `/opt/openvibe.search`, install on a lockfile change, restart, wait for `/api/ready`). |
| 493 | |
| 494 | | | | |
| 495 | |---|---| |
| 496 | | Checkout | `/opt/openvibe.search` | |
| 497 | | Unit | `openvibe-search.service` ([deploy/systemd/](deploy/systemd/openvibe-search.service)), runs as `ubuntu` | |
| 498 | | Port | `127.0.0.1:4710` | |
| 499 | | Env file | `/etc/openvibe/search.env` | |
| 500 | | Data | PostgreSQL `ov_search` (`DATABASE_URL` through PgBouncer; migrations on `DATABASE_DIRECT_URL`) | |
| 501 | | Public host | nginx [deploy/nginx/search.openvibe.network.conf](deploy/nginx/search.openvibe.network.conf) (below) | |
| 502 | |
| 503 | Rollback: ovhost puts the previous sha back by itself when `/api/ready` does not answer 2xx after the |
| 504 | restart; afterwards `sudo ovhost rollback search --to <sha>`. Nothing blocks a rollback: the schema |
| 505 | code only adds tables and columns, which an older release ignores. |
| 506 | |
| 507 | ### Public host |
| 508 | |
| 509 | `search.openvibe.network` resolves through Cloudflare; without this vhost nginx answers it with its |
| 510 | default site (the admin placeholder). The vhost |
| 511 | ([deploy/nginx/search.openvibe.network.conf](deploy/nginx/search.openvibe.network.conf)) uses the |
| 512 | Network wildcard certificate, sets the client-address headers from `$remote_addr` only, answers |
| 513 | `/metrics` with 404, keeps `/internal/`, `/api/v1/owners/` and document writes loopback-only, and |
| 514 | answers every unknown path with a JSON 404. Install on the host, after the service runs a release |
| 515 | that has the search page and saved searches (`ovhost deploy search`): |
| 516 | |
| 517 | ```bash |
| 518 | sudo install -m 0644 /opt/openvibe.search/deploy/nginx/search.openvibe.network.conf /etc/nginx/sites-available/search.openvibe.network.conf |
| 519 | sudo ln -sf /etc/nginx/sites-available/search.openvibe.network.conf /etc/nginx/sites-enabled/search.openvibe.network.conf |
| 520 | sudo nginx -t && sudo systemctl reload nginx |
| 521 | # check |
| 522 | curl -sS https://search.openvibe.network/api/health |
| 523 | curl -sS -H 'Accept: text/html' https://search.openvibe.network/ | grep -o '<title>[^<]*' |
| 524 | curl -sS -o /dev/null -w '%{http_code}\n' https://search.openvibe.network/metrics # 404 |
| 525 | curl -sS -o /dev/null -w '%{http_code}\n' -X POST https://search.openvibe.network/internal/events # 403 |
| 526 | curl -sS -o /dev/null -w '%{http_code}\n' https://search.openvibe.network/api/v1/owners/wiki/removals # 403 |
| 527 | ``` |
| 528 | |
| 529 | For the CDN purge, create a Cloudflare API token with only *Zone → Cache Purge → Purge* on the |
| 530 | product zones, then add `CLOUDFLARE_PURGE_TOKEN` and `CLOUDFLARE_ZONE_IDS` to |
| 531 | `/etc/openvibe/search.env` (0600) and restart `openvibe-search`; `/api/ready` then shows |
| 532 | `purge.cdn: "cloudflare"`. |
| 533 | |
| 534 | ## Launch rule |
| 535 | |
| 536 | This repository does not make the product real, and the domain kept its placeholder page on |
| 537 | [OpenVibers/OpenVibe.Sites](https://github.com/OpenVibers/OpenVibe.Sites) until: an owning runtime |
| 538 | with health/readiness; Network identity and service principals; a public route useful without |
| 539 | JavaScript (the search page at `/`); real |
| 540 | persistence and end-to-end workflows; capabilities and events registered in OpenVibe.Contracts; a |
| 541 | security review; and acceptance tests. The domain is not in `OpenVibe.Sites/sites.json`; this |
| 542 | repository's own vhost serves it. |
| 543 | |
| 544 | --- |
| 545 | |
| 546 | Part of the [OpenVibe network](https://openvibe.network). Built in the open by [OpenVibers](https://github.com/OpenVibers). |
| 547 | |
| 548 | <!-- versions:start --> |
| 549 | - openvibe-contracts: v0.419.0 |
| 550 | - openvibe-sdk: v0.161.0 |
| 551 | - openvibe-shared: v4.241.0 |
| 552 | <!-- versions:end --> |
| 553 |