openvibers/openvibe.search Commits
Contracts proposal (search.index-document@1), capability and service manifest proposals, engine ADR, README, STATUS alpha W14, env example, systemd unit, nginx vhost, CI on Node 22.22.1
Changes
nnew file mode 100644@@@ -0,0 +1,44 @@++# ═══════════════════════════════════════════════════════════════++# OpenVibe.Search — Environment Configuration++# Copy to .env and fill in values.++# In production this lives at /etc/openvibe/search.env (0600); the systemd unit sets++# NODE_ENV, PORT and SEARCH_DB_PATH itself.++# ═══════════════════════════════════════════════════════════════++++# Server++PORT=4710++HOST=127.0.0.1++NODE_ENV=development++#BASE_URL=https://search.openvibe.network++++# Identity — OpenVibe.Network signs service tokens (client_credentials, audience++# openvibe.search) and user JWTs. The public key is fetched from++# <url>/api/.well-known/jwks (internal URL first), retried every 30 s until it loads.++OV_NETWORK_URL=https://openvibe.network++OV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000++#OV_NETWORK_ISSUER=https://openvibe.network++# Pin the key instead of fetching it (PEM; \n escapes allowed).++#OV_NETWORK_PUBLIC_KEY=++# A browser's user JWT is accepted when its aud contains one of these.++#SEARCH_USER_AUDIENCES=openvibe.search,openvibe.network++++# This service's own principal (svc:search) — publishes search.document.* to Events++# (events.event.publish) and creates its subscription (events.subscription.manage).++OV_OAUTH_CLIENT_ID=search++OV_OAUTH_CLIENT_SECRET=++++# Store (SQLite, FTS5). Production: /var/lib/openvibe-search/search.db (systemd StateDirectory).++SEARCH_DB_PATH=./data/search.db++++# OpenVibe.Events. The outbox relay runs only when EVENTS_URL is set.++#EVENTS_URL=http://127.0.0.1:4300++#EVENTS_RELAY_INTERVAL_MS=2000++# Signing secret(s) of the index-document subscription (comma-separated during rotation).++# Generate before `npm run subscribe`, e.g. `openssl rand -hex 32`. Unset = webhook off (503).++#SEARCH_EVENTS_SECRET=++# Sources whose <source>.index_document.* events are accepted (default: wiki blog news reviews++# deals coupons trade community live media codes games tools sources).++#SEARCH_EVENT_OWNERS=wiki,blog,news++++# Largest accepted request body (direct PUT and deliveries), bytes.++#SEARCH_MAX_BODY_BYTES=262144nnew file mode 100644@@@ -0,0 +1,19 @@++name: ci++on:++ push:++ branches: [main]++ pull_request:++jobs:++ test:++ runs-on: ubuntu-latest++ steps:++ - uses: actions/checkout@v4++ - uses: actions/setup-node@v4++ with:++ node-version: 22.22.1 # production runs Node 22++ cache: npm++ - run: npm ci --no-audit --no-fund++ - run: npm test++ # Add once the search service manifest and the search.* capabilities ship in++ # openvibe-contracts (proposals in docs/capabilities-proposal/ and docs/service-manifest-proposal.json):++ # - run: npx openvibe-contracts-check --service search --src servernnew file mode 100644@@@ -0,0 +1,158 @@++# OpenVibe.Search++++> Permission-aware discovery for the network: the canonical index document, event-fed indexing,++> an ACL-filtered query API, and deletion/visibility propagation.++++**Status:** alpha (roadmap Wave 14). Runs and is tested; not deployed, and no product indexes into it yet.++**Domain:** `search.openvibe.network` (still a placeholder on OpenVibe.Sites until the launch rule below holds)++**Plan:** OpenVibe End-to-End Realignment & Implementation Plan, revision 3 — roadmap §4.2 A, §15.12, §29, §32.3.++**License:** AGPL-3.0.++++## Purpose++++Every publication product (Wiki, Blog, News, Reviews, Deals, Coupons, Trade) and every other++service with discoverable content tells Search about its resources with one contract —++[`search.index-document@1`](docs/contracts-proposal/) — and Search answers queries without ever++returning a document the caller may not see. The owner stays the source of truth; Search keeps a++revision-ordered copy and can be reconciled against it.++++## Running it++++```bash++npm install++cp .env.example .env++npm run dev # http://127.0.0.1:4710++npm test # every test/*.test.js, temp databases, no network needed++```++++Node 22 in production (`fnm exec --using=22.22.1 npm test`). Production: `/opt/openvibe.search`,++env `/etc/openvibe/search.env`, unit [deploy/systemd/openvibe-search.service](deploy/systemd/openvibe-search.service),++index `/var/lib/openvibe-search/search.db`, nginx [deploy/nginx/search.openvibe.network.conf](deploy/nginx/search.openvibe.network.conf)++(the public vhost exposes only the GET query API and health; owners and Events call `127.0.0.1:4710`).++++`GET /api/health` is liveness. `GET /api/ready` is 200 when the database answers and the Network++signing key has loaded; it also reports document counts by exposure, the outbox backlog and++whether the Events webhook is on.++++## Query API++++```http++GET /api/v1/search?q=solar+eclipse&owner=news&type=story&lang=en&facet.topic=space&facets=topic&limit=20&cursor=…++GET /api/v1/suggest?q=sol++GET /api/v1/documents/:owner/:type/:id++```++++- `q` is plain text: word tokens only, every word required, a trailing `*` makes the last word a++ prefix. FTS operators and column filters in `q` are neutralised. No `q` = newest first.++- Results carry `owner, type, id, revision, visibility, title, summary, canonical_url, facets,++ language, authorship, provenance, published_at, updated_at, indexable` and a `snippet_html`++ (escaped text with `<mark>`). Never the ACL. `next_cursor` is bound to the query that made it.++- `facets=k1,k2` returns value counts over exactly the visible matching set. There is no total count.++- Every response is `Cache-Control: no-store`.++++Who sees what:++++| Caller | Sees |++|---|---|++| anonymous (or a user token without `subject_id`) | public, published, indexable documents |++| Network user JWT (`ov_token` cookie or Bearer) | + restricted documents whose ACL names its subject or `role:<role>` |++| service token with `search.query.delegate` + `X-OV-Subject` | + the ACL matches of that `usr_`/`gst_` subject and the `X-OV-Groups` / `X-OV-Entitlements` the service vouches for |++++Drafts, unpublished, retracted and deleted documents are returned to nobody. A hidden document and++a missing one give the same 404. A Bearer that does not verify is a 401 (never downgraded to++anonymous); delegation headers from a browser are ignored. Visibility is enforced in the engine++SQL (two FTS tables split by audience plus an ACL predicate) and re-checked per hit — see++[docs/adr-engine.md](docs/adr-engine.md).++++## Indexing++++Owners call with an OpenVibe.Network client-credentials token (`audience=openvibe.search`) holding++`search.document.write`, and may only touch documents whose `owner` is their own service slug.++++```http++PUT /api/v1/documents/wiki/page/pg_123 { …search.index-document@1… }++DELETE /api/v1/documents/wiki/page/pg_123?revision=8++GET /api/v1/owners/wiki/documents?type=page&after=0&limit=500 # reconciliation++GET /api/v1/owners/wiki/documents/page/pg_123 # stored doc + effective indexability++GET /api/v1/owners/wiki/rejections # refused index events++```++++Or, preferred, through Events: publish `wiki.index_document.upserted` (payload = the document) or++`wiki.index_document.deleted` (payload = `{type, id, revision}`) from the owner's outbox with++`visibility: "internal"`. Search subscribes to `*.index_document.*` (`npm run subscribe`) and++consumes deliveries at `POST /internal/events`: HMAC-verified (`SEARCH_EVENTS_SECRET`), one inbox++receipt per event in the same transaction as the change, so a redelivery never applies twice. A++source may only write its own owner's documents (`SEARCH_EVENT_OWNERS` limits which sources feed++the index at all).++++Revision order, on both paths: an older revision is `stale` and changes nothing; the same revision++again is `unchanged`; different content at the same revision is a `conflict` (first write wins); at++equal revision a deletion wins; a tombstone keeps the revision, so a late older upsert cannot bring++the document back; a higher revision restores it. Search only makes indexability stricter (drafts,++private/members/unlisted, unpublished, stub-provider provenance and public documents without a++canonical URL are always `noindex`, with the reason recorded).++++Every change that lowers a document's exposure (deleted, drafted, unpublished, made++private/members/unlisted, or public → noindex) leaves results in the same transaction and emits++`search.document.removed` (with the canonical URL if it had been public, so caches and sitemaps++purge); every accepted upsert emits `search.document.indexed`. Both go through a transactional++outbox relayed to Events when `EVENTS_URL` is set.++++## Owns++++- the index document contract (proposed as `search.index-document@1`, [docs/contracts-proposal/](docs/contracts-proposal/))++- `documents`, `doc_acl`, `doc_facets`, `fts_public`, `fts_restricted`, `idempotency_receipts`,++ `ingest_rejections`, `event_outbox` (SQLite; engine decision in [docs/adr-engine.md](docs/adr-engine.md))++- the query API and its ACL filtering; deletion and visibility-change propagation++++## Does not own++++- any product's content, publication state or indexability decision (owners and the publishing packages do)++- sitemaps, feeds or page caches (each product; they listen to `search.document.removed`)++- identity, groups or entitlements (Network, products, VIP/Billing — Search only matches the keys it is given)++++## Depends on++++- OpenVibe.Contracts (service tokens, problem details, ids, the event envelope)++- OpenVibe.Network (signing key; service principal `search`)++- OpenVibe.Events (index-document deliveries in, `search.document.*` out)++++## Capabilities (proposed, [docs/capabilities-proposal/](docs/capabilities-proposal/))++++| Capability | Routes |++|---|---|++| `search.document.write` | `PUT/DELETE /api/v1/documents/:owner/:type/:id`, `/api/v1/owners/:owner/...` (own owner only) |++| `search.query.delegate` | the query routes, acting for `X-OV-Subject` |++++Not in `openvibe-contracts` v0.7.0 yet; [server/auth.js](server/auth.js) decides them with the++contracts grant rule until a release defines them. CI's `openvibe-contracts-check` step is++commented out until the service manifest ([docs/service-manifest-proposal.json](docs/service-manifest-proposal.json)) ships.++++## Acceptance (tests)++++- a private or draft document is never returned to an unauthorized query — hits, snippets, facet++ counts, suggestions, direct gets, cursors and FTS-syntax injection (`test/acl-leak.test.js`)++- a visibility change or deletion removes the document from results immediately and emits++ `search.document.removed` (`test/acl-leak.test.js`)++- revision ordering, tombstones, restore, reconciliation (`test/documents.test.js`)++- signed deliveries, inbox dedupe, out-of-order delivery, crash before commit redelivers once,++ owner spoofing refused (`test/webhook-inbox.test.js`)++- outbox relay to Events, retry and poison isolation (`test/outbox.test.js`)++- proposals valid against the contracts schemas (`test/proposals.test.js`)++++Not yet demonstrated: an end-to-end run with a real product publishing through a deployed Events++(no product exists yet; OpenVibe.Sources' `sources/item` documents are the first producer).++++## Launch rule++++This repository does not make the product real, and the domain keeps its placeholder page on++[OpenVibers/OpenVibe.Sites](https://github.com/OpenVibers/OpenVibe.Sites) until: an owning runtime++with health/readiness; Network identity and service principals; a public route useful without++JavaScript (there is no HTML search page yet — only the JSON API and a text index at `/`); real++persistence and end-to-end workflows; capabilities and events registered in OpenVibe.Contracts; a++security review; and acceptance tests. The launch release removes the domain from++`OpenVibe.Sites/sites.json` atomically.++++---++++Part of the [OpenVibe network](https://openvibe.network). Built in the open by [OpenVibers](https://github.com/OpenVibers).nnew file mode 100644@@@ -0,0 +1,18 @@++{++ "repository": "OpenVibers/OpenVibe.Search",++ "stage": "alpha",++ "code": true,++ "wave": "W14",++ "plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026)",++ "runtime": "node 22, express, better-sqlite3 (FTS5)",++ "port": 4710,++ "domain": "search.openvibe.network",++ "contracts": "openvibe-contracts v0.7.0",++ "capabilities": ["search.document.write", "search.query.delegate"],++ "capabilitiesRegistered": false,++ "contractsProposed": ["search.index-document@1"],++ "engine": "sqlite-fts5 (docs/adr-engine.md)",++ "deployed": false,++ "indexedProducers": [],++ "updated": "2026-09-22"++}nnew file mode 100644@@@ -0,0 +1,82 @@++## ═══════════════════════════════════════════════════════════════++## search.openvibe.network — Nginx Configuration (REFERENCE)++##++## Deployed to /etc/nginx/sites-available/search.openvibe.network.conf — keep them in sync,++## and reload nginx after changing it.++##++## The Node service (port 4710, systemd unit openvibe-search) serves:++## /api/v1/search, /api/v1/suggest, GET /api/v1/documents/... public query API (anonymous =++## public documents only; the ov_token cookie or a Bearer adds++## the caller's ACL matches). Responses are Cache-Control: no-store.++## /api/v1/documents (PUT/DELETE), /api/v1/owners/... owner API (service tokens). Services on++## this host call 127.0.0.1:4710, so the public vhost refuses++## writes; open them only for an off-host owner.++## /internal/events OpenVibe.Events deliveries — host-local only.++## /api/health, /api/ready++## ═══════════════════════════════════════════════════════════════++++limit_req_zone $binary_remote_addr zone=ovsearch_query:10m rate=10r/s;++++# ── HTTP → HTTPS redirect ──────────────────────────────────++server {++ listen 80;++ listen [::]:80;++ server_name search.openvibe.network;++++ location /.well-known/acme-challenge/ { root /var/www/certbot; }++ location / { return 301 https://search.openvibe.network$request_uri; }++}++++# ── HTTPS server ───────────────────────────────────────────++server {++ listen 443 ssl;++ listen [::]:443 ssl;++ http2 on;++ server_name search.openvibe.network;++++ # The host has the openvibe.network wildcard certificate, not a per-name one.++ ssl_certificate /etc/letsencrypt/live/openvibe.network/fullchain.pem;++ ssl_certificate_key /etc/letsencrypt/live/openvibe.network/privkey.pem;++ ssl_protocols TLSv1.2 TLSv1.3;++++ access_log /var/log/nginx/search.openvibe.network.access.log;++ error_log /var/log/nginx/search.openvibe.network.error.log;++++ client_max_body_size 256k;++++ add_header X-Content-Type-Options nosniff always;++ add_header Referrer-Policy strict-origin-when-cross-origin always;++ add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;++++ location = /api/health { proxy_pass http://127.0.0.1:4710; proxy_set_header Host $host; }++ location = /api/ready { proxy_pass http://127.0.0.1:4710; proxy_set_header Host $host; }++++ # ── Public query API (GET only) ─────────────────────────++ location ~ ^/api/v1/(search|suggest|documents/) {++ limit_except GET { deny all; }++ limit_req zone=ovsearch_query burst=40 nodelay;++ proxy_pass http://127.0.0.1:4710;++ proxy_http_version 1.1;++ proxy_set_header Connection "";++ proxy_set_header Host $host;++ proxy_set_header X-Real-IP $remote_addr;++ proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;++ proxy_set_header X-Forwarded-Proto $scheme;++ # Never let a shared cache hold a personalised answer.++ proxy_no_cache 1;++ proxy_cache_bypass 1;++ }++++ # ── Owner API and the Events webhook: host-local only ───++ location /api/ { return 403; }++ location /internal/ { return 403; }++++ location = / {++ proxy_pass http://127.0.0.1:4710;++ proxy_set_header Host $host;++ }++++ location / {++ return 404;++ }++}nnew file mode 100644@@@ -0,0 +1,31 @@++[Unit]++Description=OpenVibe.Search — permission-aware index and query API (SQLite FTS5)++# Network (openvibe-network) signs the tokens this service verifies. Search starts without it++# and keeps retrying the JWKS fetch; /api/ready stays 503 until the key has loaded.++After=network.target++Wants=openvibe-network.service++++[Service]++Type=simple++User=ubuntu++Group=ubuntu++WorkingDirectory=/opt/openvibe.search++EnvironmentFile=/etc/openvibe/search.env++Environment=NODE_ENV=production++Environment=PORT=4710++Environment=PATH=/usr/local/bin:/usr/bin:/bin++ExecStart=/usr/bin/env node /opt/openvibe.search/server/index.js++Restart=always++RestartSec=5++KillSignal=SIGTERM++TimeoutStopSec=15++NoNewPrivileges=true++PrivateTmp=true++ProtectSystem=strict++ProtectHome=true++# The code tree stays read-only; the index lives in /var/lib/openvibe-search.++StateDirectory=openvibe-search++Environment=SEARCH_DB_PATH=/var/lib/openvibe-search/search.db++++[Install]++WantedBy=multi-user.targetnnew file mode 100644@@@ -0,0 +1,53 @@++# ADR: search engine for Wave 14 — SQLite FTS5 now, PostgreSQL FTS when Postgres exists++++**Status:** accepted for alpha (2026-09-22) · **Owner:** OpenVibe.Search · **Roadmap:** §4.2 A, §15.12, Wave 14++++## Context++++The roadmap asks for an event-fed, permission-aware index with the engine chosen only after++measuring need: *"PostgreSQL FTS first; Meilisearch or OpenSearch only if justified"*, and warns++against engine shopping. The network runs every service on one host with one SQLite database per++service; PostgreSQL is not provisioned yet (it arrives with the legacy-retirement track). There++is no corpus to measure: no product indexes anything today.++++## Decision++++1. **SQLite FTS5 inside Search's own database** (`/var/lib/openvibe-search/search.db`), no++ external engine, no new daemon. FTS5 ships in better-sqlite3's bundled SQLite (3.49), gives++ BM25 ranking, snippets and prefix queries, and commits in the same transaction as the++ document row, ACL rows, facet rows, the inbox receipt and the outbox event. That transaction++ is what makes "a visibility change or deletion removes the document from results immediately"++ true without a second system to converge.++2. **Behind a small engine interface** — [server/engine/fts5.js](../server/engine/fts5.js):++ `put(rid, doc, exposure)`, `remove(rid)`, `search`, `browse`, `facetCounts`, `suggest`. The++ store, the revision rules, the ACL model and the APIs do not know FTS5 exists.++3. **PostgreSQL FTS is the next engine**, as the roadmap says, once a PostgreSQL instance exists++ for services: a `tsvector` column with `websearch_to_tsquery`/`ts_rank_cd`, GIN index, the same++ ACL predicate as SQL. Meilisearch/OpenSearch stay out unless a measurement justifies them.++++## How visibility is enforced inside the engine++++- Documents are split into two FTS tables by audience: `fts_public` holds only public,++ published, indexable documents; `fts_restricted` only published unlisted/members/private++ documents. Drafts, unpublished, deleted and public-noindex documents are in neither, so no text++ query can reach them.++- An anonymous query (or a viewer without a subject) only ever reads `fts_public`. BM25 statistics++ are per table, so private documents cannot even shift the ranking an anonymous caller sees.++- A signed-in query adds a second branch over `fts_restricted`, filtered in the same SQL by the++ viewer's ACL keys (`doc_acl`: subject, group, entitlement). Facet counts and suggestions reuse++ the exact same visible-set SQL, so they cannot count or complete what the viewer cannot see.++- Known limitation: within `fts_restricted`, BM25 term statistics include restricted documents the++ viewer cannot see. The effect on ordering is statistical and returns no content, but it is a++ theoretical side channel between signed-in users; a per-audience or per-tenant index (or++ PostgreSQL with ranking that ignores global statistics) removes it if it ever matters.++++## Consequences++++- One process and one file: backups, restores and moves are the same as every other service.++- Write throughput is SQLite's (single writer, WAL). That is ample for the expected corpus (tens to++ hundreds of thousands of documents); the trigger to revisit is measured: p95 query latency over++ 200 ms at the public edge, index size over a few GB, or write contention in `/api/ready`.++- Tokenizer is `unicode61 remove_diacritics 2` without stemming: predictable across languages++ (Japanese/Chinese get no word segmentation; a later engine or an ICU tokenizer fixes that).++- Moving to PostgreSQL is an engine swap plus a re-index from owners (the reconciliation API++ exists for exactly that), not a contract change.nnew file mode 100644@@@ -0,0 +1,26 @@++{++ "id": "search.document.write",++ "version": "1.0",++ "owner": "search",++ "status": "active",++ "visibility": "first-party",++ "description": "Index, replace or tombstone the calling service's own documents (search.index-document@1) and read its own inventory for reconciliation. The owner in the path must be the principal's service slug (svc:wiki writes owner wiki only). Revision-ordered: an older revision never overwrites a newer one, and at equal revision a deletion wins.",++ "inputSchema": "search.index-document@1",++ "permissions": [++ "search:write"++ ],++ "resourceConstraints": [++ "owner"++ ],++ "quotaClass": "search-write",++ "events": [++ "search.document.indexed",++ "search.document.removed"++ ],++ "implementedBy": [++ "PUT|DELETE /api/v1/documents/:owner/:type/:id",++ "GET /api/v1/owners/:owner/documents",++ "GET /api/v1/owners/:owner/documents/:type/:id",++ "GET /api/v1/owners/:owner/rejections"++ ]++}nnew file mode 100644@@@ -0,0 +1,21 @@++{++ "id": "search.query.delegate",++ "version": "1.0",++ "owner": "search",++ "status": "active",++ "visibility": "first-party",++ "description": "Query the index on behalf of a person: X-OV-Subject (usr_/gst_) plus the X-OV-Groups and X-OV-Entitlements the calling first-party service has just resolved for that person. Without X-OV-Subject the service sees public documents only. Drafts, unpublished and deleted documents are never returned to anyone.",++ "permissions": [++ "search:read"++ ],++ "resourceConstraints": [++ "none"++ ],++ "quotaClass": "search-query",++ "events": [],++ "implementedBy": [++ "GET /api/v1/search",++ "GET /api/v1/suggest",++ "GET /api/v1/documents/:owner/:type/:id"++ ]++}nnew file mode 100644@@@ -0,0 +1,80 @@++# Proposal for OpenVibe.Contracts: `search.index-document@1`++++Files to add to OpenVibe.Contracts in its next release:++++| File here | Goes to |++|---|---|++| [contracts/search/index-document.v1.json](contracts/search/index-document.v1.json) | `contracts/search/index-document.v1.json` |++| [catalog-entry.json](catalog-entry.json) | a new entry in `contracts/catalog.json` |++| [../capabilities-proposal/*.json](../capabilities-proposal/) | `manifests/capabilities/` |++| [../service-manifest-proposal.json](../service-manifest-proposal.json) | `manifests/services/search.json` |++++Until then Search validates with this file directly (server/document.js); once the release ships,++it switches to `contracts.validate('search.index-document@1')` and CI turns on++`openvibe-contracts-check --service search`.++++## The document++++One document per owned resource, keyed by `(owner, type, id)` and ordered by the owner's++`revision`:++++| Field | Meaning |++|---|---|++| `owner`, `type`, `id` | Typed reference to the owner's resource (`EntityRef` without the label). Only `svc:<owner>` may write it. |++| `revision` | Owner's monotonic revision. Older never overwrites newer; at equal revision a deletion wins. |++| `deleted` | Deletion marker. A tombstone needs only owner/type/id/revision. |++| `visibility` | `public` · `unlisted` · `members` · `private` · `draft` |++| `acl` | `subjects` (usr_/gst_), `groups`, `entitlements` — who may see a non-public document. Never returned to viewers. |++| `canonical_url`, `title`, `summary`, `body` | What is shown and searched. Plain text; no HTML. |++| `facets` | Filterable key → value(s) (category, tags, space, currency, instrument…). |++| `language`, `authorship` | BCP 47 tag; `human` · `ai_assisted` · `ai_generated` · `imported`. |++| `provenance` | Source references (`{service, type, id, revision?, url?, retrieved_at?, stub?}`) — enough to reconcile a claim against its source record. |++| `publication_state` | `draft` · `scheduled` · `published` · `unpublished` · `retracted` · `archived`. Only `published` is served. |++| `published_at`, `updated_at` | The owner's timestamps; never invented. |++| `indexability` | The publishing gate's decision `index`/`noindex` with explicit reasons (roadmap §32.3). |++++### Visibility rules Search enforces++++| Visibility | In results | By exact id |++|---|---|---|++| `public` (published, indexable) | everyone | everyone |++| `public` (noindex) | nobody | everyone |++| `unlisted` | ACL matches only | any signed-in subject |++| `members` | subject, group or entitlement in `acl` | same |++| `private` | subject in `acl.subjects` | same |++| `draft`, not `published`, deleted | never | never |++++Search only makes indexability stricter: `draft`, `private`, `members_only`, `unlisted`,++`not_published`, `stub_provider` and (for public) `missing_canonical_url` are added as reasons++whatever the owner sent.++++## How owners send documents++++- **Events (preferred):** through the owner's transactional outbox, `event_type`++ `<owner>.index_document.upserted` (payload = the document) or `<owner>.index_document.deleted`++ (payload = `{type, id, revision}`), envelope `subject = {type, id, revision}` of the resource,++ `visibility: "internal"` (never `public`: the payload may describe a private document). The++ body must fit Events' payload limit (64 KB by default).++- **Direct:** `PUT /api/v1/documents/:owner/:type/:id` and `DELETE …?revision=` with a++ `search.document.write` token for audience `openvibe.search`.++++Both paths share one revision order. Reconciliation: `GET /api/v1/owners/:owner/documents`++lists `(type, id, revision, deleted, hash)`; diff it against the source of truth and re-send what++differs. Refused events are listed at `GET /api/v1/owners/:owner/rejections`.++++## Shapes the planned products will send++++| Owner / type | Visibility | Facets | Provenance |++|---|---|---|---|++| `wiki/page` | public, or members with `acl.groups: ["wiki.space:<id>:member"]` | `space`, `tags` | citations → `sources/item` |++| `blog/post` | public, or members with `acl.entitlements: ["vip.plan:<id>"]` | `blog`, `series`, `tags` | — |++| `news/story` | public | `topic`, `cluster` | every claim → `sources/item` |++| `reviews/entity` | public | `category` | signals → `sources/item`; no invented rating |++| `deals/offer` | public | `store`, `currency`, `condition` | price observation → `sources/item` with `retrieved_at` |++| `coupons/coupon` | public | `merchant`, `status` | evidence → `sources/item` |++| `trade/instrument` | public | `exchange`, `kind` | observations/filings → `sources/item` |++| `sources/item` | members, `acl.groups: ["role:admin", "role:global_mod"]` | `category`, `source` | the source item itself |++++Search emits `search.document.indexed` and `search.document.removed` (payload: owner, type, id,++revision, `reason`, previous and new exposure, and the canonical URL only when it had been++public) so caches, sitemaps and feeds can purge.nnew file mode 100644@@@ -0,0 +1,9 @@++{++ "id": "search.index-document",++ "version": "1.0.0",++ "owner": "search",++ "schema": "search/index-document.v1.json",++ "visibility": "first-party",++ "compatibility": "backward",++ "status": "active"++}nnew file mode 100644@@@ -0,0 +1,112 @@++{++ "$schema": "https://json-schema.org/draft/2020-12/schema",++ "$id": "https://openvibe.network/contracts/search/index-document.v1.json",++ "title": "IndexDocument",++ "description": "What an owning service tells OpenVibe.Search about one of its resources (roadmap §15.12, §4.2 A). The owner stays the source of truth; Search keeps a revision-ordered, permission-aware copy for discovery. A document with deleted=true is a tombstone: it removes the resource from every result and wins over any upsert with the same or an older revision.",++ "type": "object",++ "required": ["owner", "type", "id", "revision"],++ "additionalProperties": false,++ "properties": {++ "owner": {++ "type": "string", "pattern": "^[a-z][a-z0-9-]{1,39}$",++ "description": "Owning service id (wiki, blog, news, community, sources…). Only that service may write the document."++ },++ "type": {++ "type": "string", "pattern": "^[a-z][a-z0-9_]{1,39}$",++ "description": "Resource type inside the owner (page, post, story, entity, offer, coupon, instrument, item…)."++ },++ "id": {++ "type": "string", "pattern": "^[A-Za-z0-9][A-Za-z0-9._:~-]{0,127}$",++ "description": "Resource id inside owner+type. Stable across revisions."++ },++ "revision": {++ "type": "integer", "minimum": 0, "maximum": 9007199254740991,++ "description": "Owner's monotonic revision of the resource. An older revision never overwrites a newer one; at equal revision a deletion wins."++ },++ "deleted": {++ "type": "boolean", "default": false,++ "description": "Deletion marker. true = tombstone; every other content field is ignored."++ },++ "visibility": {++ "enum": ["public", "unlisted", "members", "private", "draft"],++ "description": "public: anyone. unlisted: never listed except to ACL matches; fetchable by exact id by a signed-in subject. members: subjects, groups or entitlements in acl. private: subjects in acl.subjects only. draft: never served."++ },++ "acl": {++ "type": "object", "additionalProperties": false,++ "description": "Who may see a non-public document. Ignored for public documents. Empty means nobody (for members/private) or direct-id only (for unlisted).",++ "properties": {++ "subjects": {++ "type": "array", "maxItems": 200, "uniqueItems": true,++ "items": { "type": "string", "pattern": "^(usr|gst)_[0-9A-HJKMNP-TV-Z]{26}$" }++ },++ "groups": {++ "type": "array", "maxItems": 100, "uniqueItems": true,++ "items": { "type": "string", "pattern": "^[a-z][a-z0-9_.:-]{0,127}$" },++ "description": "Opaque group keys owned by some service, e.g. role:admin (Network role), wiki.space:spc_…:member."++ },++ "entitlements": {++ "type": "array", "maxItems": 100, "uniqueItems": true,++ "items": { "type": "string", "pattern": "^[a-z][a-z0-9_.:-]{0,127}$" },++ "description": "Entitlement keys (VIP/Billing), e.g. vip.plan:pln_…. Only a first-party service that just resolved them may present them for a viewer."++ }++ }++ },++ "canonical_url": { "type": "string", "format": "uri", "pattern": "^https?://", "maxLength": 2048 },++ "title": { "type": "string", "maxLength": 500 },++ "summary": { "type": "string", "maxLength": 4000 },++ "body": { "type": "string", "maxLength": 48000, "description": "Plain text for full-text search (no HTML). Keep it under the Events payload limit when publishing through Events." },++ "facets": {++ "type": "object", "maxProperties": 20,++ "propertyNames": { "pattern": "^[a-z][a-z0-9_]{0,39}$" },++ "additionalProperties": {++ "oneOf": [++ { "type": "string", "maxLength": 200 },++ { "type": "number" },++ { "type": "boolean" },++ { "type": "array", "maxItems": 50, "items": { "type": "string", "maxLength": 200 } }++ ]++ },++ "description": "Filterable values (category, tags, space, author subject, currency…). Filter with facet.<key>=<value>."++ },++ "language": { "type": "string", "pattern": "^[a-zA-Z]{2,3}(-[A-Za-z0-9]{2,8})*$", "description": "BCP 47 tag." },++ "authorship": { "enum": ["human", "ai_assisted", "ai_generated", "imported"] },++ "provenance": {++ "type": "array", "maxItems": 50,++ "description": "Where the content came from: typed references to source records (Sources items, citations, Media objects).",++ "items": {++ "type": "object", "required": ["service", "type", "id"], "additionalProperties": false,++ "properties": {++ "service": { "type": "string", "pattern": "^[a-z][a-z0-9-]{1,39}$" },++ "type": { "type": "string", "pattern": "^[a-z][a-z0-9_]{1,39}$" },++ "id": { "type": "string", "minLength": 1, "maxLength": 128 },++ "revision": { "type": "integer", "minimum": 0 },++ "label": { "type": "string", "maxLength": 200 },++ "url": { "type": "string", "format": "uri", "pattern": "^https?://", "maxLength": 2048 },++ "retrieved_at": { "type": "string", "format": "date-time" },++ "stub": { "type": "boolean", "description": "true when the reference came from a stub provider; the document is then never indexable." }++ }++ }++ },++ "publication_state": {++ "enum": ["draft", "scheduled", "published", "unpublished", "retracted", "archived"],++ "description": "Only published documents are ever served."++ },++ "published_at": { "type": ["string", "null"], "format": "date-time" },++ "updated_at": { "type": ["string", "null"], "format": "date-time" },++ "indexability": {++ "type": "object", "required": ["decision"], "additionalProperties": false,++ "description": "The owner's deterministic indexability decision (publishing packages, roadmap §32.3). Search can only make it stricter.",++ "properties": {++ "decision": { "enum": ["index", "noindex"] },++ "reasons": {++ "type": "array", "maxItems": 20, "uniqueItems": true,++ "items": { "type": "string", "pattern": "^[a-z][a-z0-9_]{1,63}$" },++ "description": "Known reasons: draft, private, members_only, unlisted, not_published, deleted, thin_content, duplicate_without_canonical, unsupported_claims, unsourced, stub_provider, sensitive_unreviewed, ai_unreviewed, missing_canonical_url, third_party_content, owner_decision."++ }++ }++ }++ },++ "if": { "properties": { "deleted": { "const": true } }, "required": ["deleted"] },++ "then": {},++ "else": { "required": ["visibility", "publication_state", "title", "indexability"] }++}nnew file mode 100644@@@ -0,0 +1,30 @@++{++ "version": "0.1.0",++ "contractRanges": {++ "openvibe-contracts": ">=0.7.0 <1.0.0"++ },++ "id": "search",++ "name": "OpenVibe.Search",++ "status": "alpha",++ "repository": "OpenVibers/OpenVibe.Search",++ "domains": [++ "search.openvibe.network"++ ],++ "publicOrigin": "https://search.openvibe.network",++ "health": "/api/health",++ "ready": "/api/ready",++ "capabilities": [++ "search.document.write",++ "search.query.delegate"++ ],++ "eventsProduced": [++ "search.document.indexed",++ "search.document.removed"++ ],++ "eventsConsumed": [++ "*.index_document.upserted",++ "*.index_document.deleted"++ ],++ "namespacesOwned": [],++ "notes": "Canonical index document (search.index-document@1), event-fed and direct indexing with revision ordering and tombstones, permission-aware query over SQLite FTS5 (docs/adr-engine.md). Anonymous queries need no token."++}
new file mode 100644
new file mode 100644
@@ -0,0 +1,44 @@
@@ -0,0 +1,44 @@
+# ═══════════════════════════════════════════════════════════════
+# OpenVibe.Search — Environment Configuration
+# Copy to .env and fill in values.
+# In production this lives at /etc/openvibe/search.env (0600); the systemd unit sets
+# NODE_ENV, PORT and SEARCH_DB_PATH itself.
+# ═══════════════════════════════════════════════════════════════
+
+# Server
+PORT=4710
+HOST=127.0.0.1
+NODE_ENV=development
+#BASE_URL=https://search.openvibe.network
+
+# Identity — OpenVibe.Network signs service tokens (client_credentials, audience
+# openvibe.search) and user JWTs. The public key is fetched from
+# <url>/api/.well-known/jwks (internal URL first), retried every 30 s until it loads.
+OV_NETWORK_URL=https://openvibe.network
+OV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000
+#OV_NETWORK_ISSUER=https://openvibe.network
+# Pin the key instead of fetching it (PEM; \n escapes allowed).
+#OV_NETWORK_PUBLIC_KEY=
+# A browser's user JWT is accepted when its aud contains one of these.
+#SEARCH_USER_AUDIENCES=openvibe.search,openvibe.network
+
+# This service's own principal (svc:search) — publishes search.document.* to Events
+# (events.event.publish) and creates its subscription (events.subscription.manage).
+OV_OAUTH_CLIENT_ID=search
+OV_OAUTH_CLIENT_SECRET=
+
+# Store (SQLite, FTS5). Production: /var/lib/openvibe-search/search.db (systemd StateDirectory).
+SEARCH_DB_PATH=./data/search.db
+
+# OpenVibe.Events. The outbox relay runs only when EVENTS_URL is set.
+#EVENTS_URL=http://127.0.0.1:4300
+#EVENTS_RELAY_INTERVAL_MS=2000
+# Signing secret(s) of the index-document subscription (comma-separated during rotation).
+# Generate before `npm run subscribe`, e.g. `openssl rand -hex 32`. Unset = webhook off (503).
+#SEARCH_EVENTS_SECRET=
+# Sources whose <source>.index_document.* events are accepted (default: wiki blog news reviews
+# deals coupons trade community live media codes games tools sources).
+#SEARCH_EVENT_OWNERS=wiki,blog,news
+
+# Largest accepted request body (direct PUT and deliveries), bytes.
+#SEARCH_MAX_BODY_BYTES=262144
new file mode 100644
new file mode 100644
@@ -0,0 +1,19 @@
@@ -0,0 +1,19 @@
+name: ci
+on:
+ push:
+ branches: [main]
+ pull_request:
+jobs:
+ test:
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v4
+ - uses: actions/setup-node@v4
+ with:
+ node-version: 22.22.1 # production runs Node 22
+ cache: npm
+ - run: npm ci --no-audit --no-fund
+ - run: npm test
+ # Add once the search service manifest and the search.* capabilities ship in
+ # openvibe-contracts (proposals in docs/capabilities-proposal/ and docs/service-manifest-proposal.json):
+ # - run: npx openvibe-contracts-check --service search --src server
new file mode 100644
new file mode 100644
@@ -0,0 +1,158 @@
@@ -0,0 +1,158 @@
+# OpenVibe.Search
+
+> Permission-aware discovery for the network: the canonical index document, event-fed indexing,
+> an ACL-filtered query API, and deletion/visibility propagation.
+
+**Status:** alpha (roadmap Wave 14). Runs and is tested; not deployed, and no product indexes into it yet.
+**Domain:** `search.openvibe.network` (still a placeholder on OpenVibe.Sites until the launch rule below holds)
+**Plan:** OpenVibe End-to-End Realignment & Implementation Plan, revision 3 — roadmap §4.2 A, §15.12, §29, §32.3.
+**License:** AGPL-3.0.
+
+## Purpose
+
+Every publication product (Wiki, Blog, News, Reviews, Deals, Coupons, Trade) and every other
+service with discoverable content tells Search about its resources with one contract —
+[`search.index-document@1`](docs/contracts-proposal/) — and Search answers queries without ever
+returning a document the caller may not see. The owner stays the source of truth; Search keeps a
+revision-ordered copy and can be reconciled against it.
+
+## Running it
+
+```bash
+npm install
+cp .env.example .env
+npm run dev # http://127.0.0.1:4710
+npm test # every test/*.test.js, temp databases, no network needed
+```
+
+Node 22 in production (`fnm exec --using=22.22.1 npm test`). Production: `/opt/openvibe.search`,
+env `/etc/openvibe/search.env`, unit [deploy/systemd/openvibe-search.service](deploy/systemd/openvibe-search.service),
+index `/var/lib/openvibe-search/search.db`, nginx [deploy/nginx/search.openvibe.network.conf](deploy/nginx/search.openvibe.network.conf)
+(the public vhost exposes only the GET query API and health; owners and Events call `127.0.0.1:4710`).
+
+`GET /api/health` is liveness. `GET /api/ready` is 200 when the database answers and the Network
+signing key has loaded; it also reports document counts by exposure, the outbox backlog and
+whether the Events webhook is on.
+
+## Query API
+
+```http
+GET /api/v1/search?q=solar+eclipse&owner=news&type=story&lang=en&facet.topic=space&facets=topic&limit=20&cursor=…
+GET /api/v1/suggest?q=sol
+GET /api/v1/documents/:owner/:type/:id
+```
+
+- `q` is plain text: word tokens only, every word required, a trailing `*` makes the last word a
+ prefix. FTS operators and column filters in `q` are neutralised. No `q` = newest first.
+- Results carry `owner, type, id, revision, visibility, title, summary, canonical_url, facets,
+ language, authorship, provenance, published_at, updated_at, indexable` and a `snippet_html`
+ (escaped text with `<mark>`). Never the ACL. `next_cursor` is bound to the query that made it.
+- `facets=k1,k2` returns value counts over exactly the visible matching set. There is no total count.
+- Every response is `Cache-Control: no-store`.
+
+Who sees what:
+
+| Caller | Sees |
+|---|---|
+| anonymous (or a user token without `subject_id`) | public, published, indexable documents |
+| Network user JWT (`ov_token` cookie or Bearer) | + restricted documents whose ACL names its subject or `role:<role>` |
+| service token with `search.query.delegate` + `X-OV-Subject` | + the ACL matches of that `usr_`/`gst_` subject and the `X-OV-Groups` / `X-OV-Entitlements` the service vouches for |
+
+Drafts, unpublished, retracted and deleted documents are returned to nobody. A hidden document and
+a missing one give the same 404. A Bearer that does not verify is a 401 (never downgraded to
+anonymous); delegation headers from a browser are ignored. Visibility is enforced in the engine
+SQL (two FTS tables split by audience plus an ACL predicate) and re-checked per hit — see
+[docs/adr-engine.md](docs/adr-engine.md).
+
+## Indexing
+
+Owners call with an OpenVibe.Network client-credentials token (`audience=openvibe.search`) holding
+`search.document.write`, and may only touch documents whose `owner` is their own service slug.
+
+```http
+PUT /api/v1/documents/wiki/page/pg_123 { …search.index-document@1… }
+DELETE /api/v1/documents/wiki/page/pg_123?revision=8
+GET /api/v1/owners/wiki/documents?type=page&after=0&limit=500 # reconciliation
+GET /api/v1/owners/wiki/documents/page/pg_123 # stored doc + effective indexability
+GET /api/v1/owners/wiki/rejections # refused index events
+```
+
+Or, preferred, through Events: publish `wiki.index_document.upserted` (payload = the document) or
+`wiki.index_document.deleted` (payload = `{type, id, revision}`) from the owner's outbox with
+`visibility: "internal"`. Search subscribes to `*.index_document.*` (`npm run subscribe`) and
+consumes deliveries at `POST /internal/events`: HMAC-verified (`SEARCH_EVENTS_SECRET`), one inbox
+receipt per event in the same transaction as the change, so a redelivery never applies twice. A
+source may only write its own owner's documents (`SEARCH_EVENT_OWNERS` limits which sources feed
+the index at all).
+
+Revision order, on both paths: an older revision is `stale` and changes nothing; the same revision
+again is `unchanged`; different content at the same revision is a `conflict` (first write wins); at
+equal revision a deletion wins; a tombstone keeps the revision, so a late older upsert cannot bring
+the document back; a higher revision restores it. Search only makes indexability stricter (drafts,
+private/members/unlisted, unpublished, stub-provider provenance and public documents without a
+canonical URL are always `noindex`, with the reason recorded).
+
+Every change that lowers a document's exposure (deleted, drafted, unpublished, made
+private/members/unlisted, or public → noindex) leaves results in the same transaction and emits
+`search.document.removed` (with the canonical URL if it had been public, so caches and sitemaps
+purge); every accepted upsert emits `search.document.indexed`. Both go through a transactional
+outbox relayed to Events when `EVENTS_URL` is set.
+
+## Owns
+
+- the index document contract (proposed as `search.index-document@1`, [docs/contracts-proposal/](docs/contracts-proposal/))
+- `documents`, `doc_acl`, `doc_facets`, `fts_public`, `fts_restricted`, `idempotency_receipts`,
+ `ingest_rejections`, `event_outbox` (SQLite; engine decision in [docs/adr-engine.md](docs/adr-engine.md))
+- the query API and its ACL filtering; deletion and visibility-change propagation
+
+## Does not own
+
+- any product's content, publication state or indexability decision (owners and the publishing packages do)
+- sitemaps, feeds or page caches (each product; they listen to `search.document.removed`)
+- identity, groups or entitlements (Network, products, VIP/Billing — Search only matches the keys it is given)
+
+## Depends on
+
+- OpenVibe.Contracts (service tokens, problem details, ids, the event envelope)
+- OpenVibe.Network (signing key; service principal `search`)
+- OpenVibe.Events (index-document deliveries in, `search.document.*` out)
+
+## Capabilities (proposed, [docs/capabilities-proposal/](docs/capabilities-proposal/))
+
+| Capability | Routes |
+|---|---|
+| `search.document.write` | `PUT/DELETE /api/v1/documents/:owner/:type/:id`, `/api/v1/owners/:owner/...` (own owner only) |
+| `search.query.delegate` | the query routes, acting for `X-OV-Subject` |
+
+Not in `openvibe-contracts` v0.7.0 yet; [server/auth.js](server/auth.js) decides them with the
+contracts grant rule until a release defines them. CI's `openvibe-contracts-check` step is
+commented out until the service manifest ([docs/service-manifest-proposal.json](docs/service-manifest-proposal.json)) ships.
+
+## Acceptance (tests)
+
+- a private or draft document is never returned to an unauthorized query — hits, snippets, facet
+ counts, suggestions, direct gets, cursors and FTS-syntax injection (`test/acl-leak.test.js`)
+- a visibility change or deletion removes the document from results immediately and emits
+ `search.document.removed` (`test/acl-leak.test.js`)
+- revision ordering, tombstones, restore, reconciliation (`test/documents.test.js`)
+- signed deliveries, inbox dedupe, out-of-order delivery, crash before commit redelivers once,
+ owner spoofing refused (`test/webhook-inbox.test.js`)
+- outbox relay to Events, retry and poison isolation (`test/outbox.test.js`)
+- proposals valid against the contracts schemas (`test/proposals.test.js`)
+
+Not yet demonstrated: an end-to-end run with a real product publishing through a deployed Events
+(no product exists yet; OpenVibe.Sources' `sources/item` documents are the first producer).
+
+## Launch rule
+
+This repository does not make the product real, and the domain keeps its placeholder page on
+[OpenVibers/OpenVibe.Sites](https://github.com/OpenVibers/OpenVibe.Sites) until: an owning runtime
+with health/readiness; Network identity and service principals; a public route useful without
+JavaScript (there is no HTML search page yet — only the JSON API and a text index at `/`); real
+persistence and end-to-end workflows; capabilities and events registered in OpenVibe.Contracts; a
+security review; and acceptance tests. The launch release removes the domain from
+`OpenVibe.Sites/sites.json` atomically.
+
+---
+
+Part of the [OpenVibe network](https://openvibe.network). Built in the open by [OpenVibers](https://github.com/OpenVibers).
new file mode 100644
new file mode 100644
@@ -0,0 +1,18 @@
@@ -0,0 +1,18 @@
+{
+ "repository": "OpenVibers/OpenVibe.Search",
+ "stage": "alpha",
+ "code": true,
+ "wave": "W14",
+ "plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026)",
+ "runtime": "node 22, express, better-sqlite3 (FTS5)",
+ "port": 4710,
+ "domain": "search.openvibe.network",
+ "contracts": "openvibe-contracts v0.7.0",
+ "capabilities": ["search.document.write", "search.query.delegate"],
+ "capabilitiesRegistered": false,
+ "contractsProposed": ["search.index-document@1"],
+ "engine": "sqlite-fts5 (docs/adr-engine.md)",
+ "deployed": false,
+ "indexedProducers": [],
+ "updated": "2026-09-22"
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,82 @@
@@ -0,0 +1,82 @@
+## ═══════════════════════════════════════════════════════════════
+## search.openvibe.network — Nginx Configuration (REFERENCE)
+##
+## Deployed to /etc/nginx/sites-available/search.openvibe.network.conf — keep them in sync,
+## and reload nginx after changing it.
+##
+## The Node service (port 4710, systemd unit openvibe-search) serves:
+## /api/v1/search, /api/v1/suggest, GET /api/v1/documents/... public query API (anonymous =
+## public documents only; the ov_token cookie or a Bearer adds
+## the caller's ACL matches). Responses are Cache-Control: no-store.
+## /api/v1/documents (PUT/DELETE), /api/v1/owners/... owner API (service tokens). Services on
+## this host call 127.0.0.1:4710, so the public vhost refuses
+## writes; open them only for an off-host owner.
+## /internal/events OpenVibe.Events deliveries — host-local only.
+## /api/health, /api/ready
+## ═══════════════════════════════════════════════════════════════
+
+limit_req_zone $binary_remote_addr zone=ovsearch_query:10m rate=10r/s;
+
+# ── HTTP → HTTPS redirect ──────────────────────────────────
+server {
+ listen 80;
+ listen [::]:80;
+ server_name search.openvibe.network;
+
+ location /.well-known/acme-challenge/ { root /var/www/certbot; }
+ location / { return 301 https://search.openvibe.network$request_uri; }
+}
+
+# ── HTTPS server ───────────────────────────────────────────
+server {
+ listen 443 ssl;
+ listen [::]:443 ssl;
+ http2 on;
+ server_name search.openvibe.network;
+
+ # The host has the openvibe.network wildcard certificate, not a per-name one.
+ ssl_certificate /etc/letsencrypt/live/openvibe.network/fullchain.pem;
+ ssl_certificate_key /etc/letsencrypt/live/openvibe.network/privkey.pem;
+ ssl_protocols TLSv1.2 TLSv1.3;
+
+ access_log /var/log/nginx/search.openvibe.network.access.log;
+ error_log /var/log/nginx/search.openvibe.network.error.log;
+
+ client_max_body_size 256k;
+
+ add_header X-Content-Type-Options nosniff always;
+ add_header Referrer-Policy strict-origin-when-cross-origin always;
+ add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
+
+ location = /api/health { proxy_pass http://127.0.0.1:4710; proxy_set_header Host $host; }
+ location = /api/ready { proxy_pass http://127.0.0.1:4710; proxy_set_header Host $host; }
+
+ # ── Public query API (GET only) ─────────────────────────
+ location ~ ^/api/v1/(search|suggest|documents/) {
+ limit_except GET { deny all; }
+ limit_req zone=ovsearch_query burst=40 nodelay;
+ proxy_pass http://127.0.0.1:4710;
+ proxy_http_version 1.1;
+ proxy_set_header Connection "";
+ proxy_set_header Host $host;
+ proxy_set_header X-Real-IP $remote_addr;
+ proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+ proxy_set_header X-Forwarded-Proto $scheme;
+ # Never let a shared cache hold a personalised answer.
+ proxy_no_cache 1;
+ proxy_cache_bypass 1;
+ }
+
+ # ── Owner API and the Events webhook: host-local only ───
+ location /api/ { return 403; }
+ location /internal/ { return 403; }
+
+ location = / {
+ proxy_pass http://127.0.0.1:4710;
+ proxy_set_header Host $host;
+ }
+
+ location / {
+ return 404;
+ }
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,31 @@
@@ -0,0 +1,31 @@
+[Unit]
+Description=OpenVibe.Search — permission-aware index and query API (SQLite FTS5)
+# Network (openvibe-network) signs the tokens this service verifies. Search starts without it
+# and keeps retrying the JWKS fetch; /api/ready stays 503 until the key has loaded.
+After=network.target
+Wants=openvibe-network.service
+
+[Service]
+Type=simple
+User=ubuntu
+Group=ubuntu
+WorkingDirectory=/opt/openvibe.search
+EnvironmentFile=/etc/openvibe/search.env
+Environment=NODE_ENV=production
+Environment=PORT=4710
+Environment=PATH=/usr/local/bin:/usr/bin:/bin
+ExecStart=/usr/bin/env node /opt/openvibe.search/server/index.js
+Restart=always
+RestartSec=5
+KillSignal=SIGTERM
+TimeoutStopSec=15
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectSystem=strict
+ProtectHome=true
+# The code tree stays read-only; the index lives in /var/lib/openvibe-search.
+StateDirectory=openvibe-search
+Environment=SEARCH_DB_PATH=/var/lib/openvibe-search/search.db
+
+[Install]
+WantedBy=multi-user.target
new file mode 100644
new file mode 100644
@@ -0,0 +1,53 @@
@@ -0,0 +1,53 @@
+# ADR: search engine for Wave 14 — SQLite FTS5 now, PostgreSQL FTS when Postgres exists
+
+**Status:** accepted for alpha (2026-09-22) · **Owner:** OpenVibe.Search · **Roadmap:** §4.2 A, §15.12, Wave 14
+
+## Context
+
+The roadmap asks for an event-fed, permission-aware index with the engine chosen only after
+measuring need: *"PostgreSQL FTS first; Meilisearch or OpenSearch only if justified"*, and warns
+against engine shopping. The network runs every service on one host with one SQLite database per
+service; PostgreSQL is not provisioned yet (it arrives with the legacy-retirement track). There
+is no corpus to measure: no product indexes anything today.
+
+## Decision
+
+1. **SQLite FTS5 inside Search's own database** (`/var/lib/openvibe-search/search.db`), no
+ external engine, no new daemon. FTS5 ships in better-sqlite3's bundled SQLite (3.49), gives
+ BM25 ranking, snippets and prefix queries, and commits in the same transaction as the
+ document row, ACL rows, facet rows, the inbox receipt and the outbox event. That transaction
+ is what makes "a visibility change or deletion removes the document from results immediately"
+ true without a second system to converge.
+2. **Behind a small engine interface** — [server/engine/fts5.js](../server/engine/fts5.js):
+ `put(rid, doc, exposure)`, `remove(rid)`, `search`, `browse`, `facetCounts`, `suggest`. The
+ store, the revision rules, the ACL model and the APIs do not know FTS5 exists.
+3. **PostgreSQL FTS is the next engine**, as the roadmap says, once a PostgreSQL instance exists
+ for services: a `tsvector` column with `websearch_to_tsquery`/`ts_rank_cd`, GIN index, the same
+ ACL predicate as SQL. Meilisearch/OpenSearch stay out unless a measurement justifies them.
+
+## How visibility is enforced inside the engine
+
+- Documents are split into two FTS tables by audience: `fts_public` holds only public,
+ published, indexable documents; `fts_restricted` only published unlisted/members/private
+ documents. Drafts, unpublished, deleted and public-noindex documents are in neither, so no text
+ query can reach them.
+- An anonymous query (or a viewer without a subject) only ever reads `fts_public`. BM25 statistics
+ are per table, so private documents cannot even shift the ranking an anonymous caller sees.
+- A signed-in query adds a second branch over `fts_restricted`, filtered in the same SQL by the
+ viewer's ACL keys (`doc_acl`: subject, group, entitlement). Facet counts and suggestions reuse
+ the exact same visible-set SQL, so they cannot count or complete what the viewer cannot see.
+- Known limitation: within `fts_restricted`, BM25 term statistics include restricted documents the
+ viewer cannot see. The effect on ordering is statistical and returns no content, but it is a
+ theoretical side channel between signed-in users; a per-audience or per-tenant index (or
+ PostgreSQL with ranking that ignores global statistics) removes it if it ever matters.
+
+## Consequences
+
+- One process and one file: backups, restores and moves are the same as every other service.
+- Write throughput is SQLite's (single writer, WAL). That is ample for the expected corpus (tens to
+ hundreds of thousands of documents); the trigger to revisit is measured: p95 query latency over
+ 200 ms at the public edge, index size over a few GB, or write contention in `/api/ready`.
+- Tokenizer is `unicode61 remove_diacritics 2` without stemming: predictable across languages
+ (Japanese/Chinese get no word segmentation; a later engine or an ICU tokenizer fixes that).
+- Moving to PostgreSQL is an engine swap plus a re-index from owners (the reconciliation API
+ exists for exactly that), not a contract change.
new file mode 100644
new file mode 100644
@@ -0,0 +1,26 @@
@@ -0,0 +1,26 @@
+{
+ "id": "search.document.write",
+ "version": "1.0",
+ "owner": "search",
+ "status": "active",
+ "visibility": "first-party",
+ "description": "Index, replace or tombstone the calling service's own documents (search.index-document@1) and read its own inventory for reconciliation. The owner in the path must be the principal's service slug (svc:wiki writes owner wiki only). Revision-ordered: an older revision never overwrites a newer one, and at equal revision a deletion wins.",
+ "inputSchema": "search.index-document@1",
+ "permissions": [
+ "search:write"
+ ],
+ "resourceConstraints": [
+ "owner"
+ ],
+ "quotaClass": "search-write",
+ "events": [
+ "search.document.indexed",
+ "search.document.removed"
+ ],
+ "implementedBy": [
+ "PUT|DELETE /api/v1/documents/:owner/:type/:id",
+ "GET /api/v1/owners/:owner/documents",
+ "GET /api/v1/owners/:owner/documents/:type/:id",
+ "GET /api/v1/owners/:owner/rejections"
+ ]
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,21 @@
@@ -0,0 +1,21 @@
+{
+ "id": "search.query.delegate",
+ "version": "1.0",
+ "owner": "search",
+ "status": "active",
+ "visibility": "first-party",
+ "description": "Query the index on behalf of a person: X-OV-Subject (usr_/gst_) plus the X-OV-Groups and X-OV-Entitlements the calling first-party service has just resolved for that person. Without X-OV-Subject the service sees public documents only. Drafts, unpublished and deleted documents are never returned to anyone.",
+ "permissions": [
+ "search:read"
+ ],
+ "resourceConstraints": [
+ "none"
+ ],
+ "quotaClass": "search-query",
+ "events": [],
+ "implementedBy": [
+ "GET /api/v1/search",
+ "GET /api/v1/suggest",
+ "GET /api/v1/documents/:owner/:type/:id"
+ ]
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,80 @@
@@ -0,0 +1,80 @@
+# Proposal for OpenVibe.Contracts: `search.index-document@1`
+
+Files to add to OpenVibe.Contracts in its next release:
+
+| File here | Goes to |
+|---|---|
+| [contracts/search/index-document.v1.json](contracts/search/index-document.v1.json) | `contracts/search/index-document.v1.json` |
+| [catalog-entry.json](catalog-entry.json) | a new entry in `contracts/catalog.json` |
+| [../capabilities-proposal/*.json](../capabilities-proposal/) | `manifests/capabilities/` |
+| [../service-manifest-proposal.json](../service-manifest-proposal.json) | `manifests/services/search.json` |
+
+Until then Search validates with this file directly (server/document.js); once the release ships,
+it switches to `contracts.validate('search.index-document@1')` and CI turns on
+`openvibe-contracts-check --service search`.
+
+## The document
+
+One document per owned resource, keyed by `(owner, type, id)` and ordered by the owner's
+`revision`:
+
+| Field | Meaning |
+|---|---|
+| `owner`, `type`, `id` | Typed reference to the owner's resource (`EntityRef` without the label). Only `svc:<owner>` may write it. |
+| `revision` | Owner's monotonic revision. Older never overwrites newer; at equal revision a deletion wins. |
+| `deleted` | Deletion marker. A tombstone needs only owner/type/id/revision. |
+| `visibility` | `public` · `unlisted` · `members` · `private` · `draft` |
+| `acl` | `subjects` (usr_/gst_), `groups`, `entitlements` — who may see a non-public document. Never returned to viewers. |
+| `canonical_url`, `title`, `summary`, `body` | What is shown and searched. Plain text; no HTML. |
+| `facets` | Filterable key → value(s) (category, tags, space, currency, instrument…). |
+| `language`, `authorship` | BCP 47 tag; `human` · `ai_assisted` · `ai_generated` · `imported`. |
+| `provenance` | Source references (`{service, type, id, revision?, url?, retrieved_at?, stub?}`) — enough to reconcile a claim against its source record. |
+| `publication_state` | `draft` · `scheduled` · `published` · `unpublished` · `retracted` · `archived`. Only `published` is served. |
+| `published_at`, `updated_at` | The owner's timestamps; never invented. |
+| `indexability` | The publishing gate's decision `index`/`noindex` with explicit reasons (roadmap §32.3). |
+
+### Visibility rules Search enforces
+
+| Visibility | In results | By exact id |
+|---|---|---|
+| `public` (published, indexable) | everyone | everyone |
+| `public` (noindex) | nobody | everyone |
+| `unlisted` | ACL matches only | any signed-in subject |
+| `members` | subject, group or entitlement in `acl` | same |
+| `private` | subject in `acl.subjects` | same |
+| `draft`, not `published`, deleted | never | never |
+
+Search only makes indexability stricter: `draft`, `private`, `members_only`, `unlisted`,
+`not_published`, `stub_provider` and (for public) `missing_canonical_url` are added as reasons
+whatever the owner sent.
+
+## How owners send documents
+
+- **Events (preferred):** through the owner's transactional outbox, `event_type`
+ `<owner>.index_document.upserted` (payload = the document) or `<owner>.index_document.deleted`
+ (payload = `{type, id, revision}`), envelope `subject = {type, id, revision}` of the resource,
+ `visibility: "internal"` (never `public`: the payload may describe a private document). The
+ body must fit Events' payload limit (64 KB by default).
+- **Direct:** `PUT /api/v1/documents/:owner/:type/:id` and `DELETE …?revision=` with a
+ `search.document.write` token for audience `openvibe.search`.
+
+Both paths share one revision order. Reconciliation: `GET /api/v1/owners/:owner/documents`
+lists `(type, id, revision, deleted, hash)`; diff it against the source of truth and re-send what
+differs. Refused events are listed at `GET /api/v1/owners/:owner/rejections`.
+
+## Shapes the planned products will send
+
+| Owner / type | Visibility | Facets | Provenance |
+|---|---|---|---|
+| `wiki/page` | public, or members with `acl.groups: ["wiki.space:<id>:member"]` | `space`, `tags` | citations → `sources/item` |
+| `blog/post` | public, or members with `acl.entitlements: ["vip.plan:<id>"]` | `blog`, `series`, `tags` | — |
+| `news/story` | public | `topic`, `cluster` | every claim → `sources/item` |
+| `reviews/entity` | public | `category` | signals → `sources/item`; no invented rating |
+| `deals/offer` | public | `store`, `currency`, `condition` | price observation → `sources/item` with `retrieved_at` |
+| `coupons/coupon` | public | `merchant`, `status` | evidence → `sources/item` |
+| `trade/instrument` | public | `exchange`, `kind` | observations/filings → `sources/item` |
+| `sources/item` | members, `acl.groups: ["role:admin", "role:global_mod"]` | `category`, `source` | the source item itself |
+
+Search emits `search.document.indexed` and `search.document.removed` (payload: owner, type, id,
+revision, `reason`, previous and new exposure, and the canonical URL only when it had been
+public) so caches, sitemaps and feeds can purge.
new file mode 100644
new file mode 100644
@@ -0,0 +1,9 @@
@@ -0,0 +1,9 @@
+{
+ "id": "search.index-document",
+ "version": "1.0.0",
+ "owner": "search",
+ "schema": "search/index-document.v1.json",
+ "visibility": "first-party",
+ "compatibility": "backward",
+ "status": "active"
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,112 @@
@@ -0,0 +1,112 @@
+{
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
+ "$id": "https://openvibe.network/contracts/search/index-document.v1.json",
+ "title": "IndexDocument",
+ "description": "What an owning service tells OpenVibe.Search about one of its resources (roadmap §15.12, §4.2 A). The owner stays the source of truth; Search keeps a revision-ordered, permission-aware copy for discovery. A document with deleted=true is a tombstone: it removes the resource from every result and wins over any upsert with the same or an older revision.",
+ "type": "object",
+ "required": ["owner", "type", "id", "revision"],
+ "additionalProperties": false,
+ "properties": {
+ "owner": {
+ "type": "string", "pattern": "^[a-z][a-z0-9-]{1,39}$",
+ "description": "Owning service id (wiki, blog, news, community, sources…). Only that service may write the document."
+ },
+ "type": {
+ "type": "string", "pattern": "^[a-z][a-z0-9_]{1,39}$",
+ "description": "Resource type inside the owner (page, post, story, entity, offer, coupon, instrument, item…)."
+ },
+ "id": {
+ "type": "string", "pattern": "^[A-Za-z0-9][A-Za-z0-9._:~-]{0,127}$",
+ "description": "Resource id inside owner+type. Stable across revisions."
+ },
+ "revision": {
+ "type": "integer", "minimum": 0, "maximum": 9007199254740991,
+ "description": "Owner's monotonic revision of the resource. An older revision never overwrites a newer one; at equal revision a deletion wins."
+ },
+ "deleted": {
+ "type": "boolean", "default": false,
+ "description": "Deletion marker. true = tombstone; every other content field is ignored."
+ },
+ "visibility": {
+ "enum": ["public", "unlisted", "members", "private", "draft"],
+ "description": "public: anyone. unlisted: never listed except to ACL matches; fetchable by exact id by a signed-in subject. members: subjects, groups or entitlements in acl. private: subjects in acl.subjects only. draft: never served."
+ },
+ "acl": {
+ "type": "object", "additionalProperties": false,
+ "description": "Who may see a non-public document. Ignored for public documents. Empty means nobody (for members/private) or direct-id only (for unlisted).",
+ "properties": {
+ "subjects": {
+ "type": "array", "maxItems": 200, "uniqueItems": true,
+ "items": { "type": "string", "pattern": "^(usr|gst)_[0-9A-HJKMNP-TV-Z]{26}$" }
+ },
+ "groups": {
+ "type": "array", "maxItems": 100, "uniqueItems": true,
+ "items": { "type": "string", "pattern": "^[a-z][a-z0-9_.:-]{0,127}$" },
+ "description": "Opaque group keys owned by some service, e.g. role:admin (Network role), wiki.space:spc_…:member."
+ },
+ "entitlements": {
+ "type": "array", "maxItems": 100, "uniqueItems": true,
+ "items": { "type": "string", "pattern": "^[a-z][a-z0-9_.:-]{0,127}$" },
+ "description": "Entitlement keys (VIP/Billing), e.g. vip.plan:pln_…. Only a first-party service that just resolved them may present them for a viewer."
+ }
+ }
+ },
+ "canonical_url": { "type": "string", "format": "uri", "pattern": "^https?://", "maxLength": 2048 },
+ "title": { "type": "string", "maxLength": 500 },
+ "summary": { "type": "string", "maxLength": 4000 },
+ "body": { "type": "string", "maxLength": 48000, "description": "Plain text for full-text search (no HTML). Keep it under the Events payload limit when publishing through Events." },
+ "facets": {
+ "type": "object", "maxProperties": 20,
+ "propertyNames": { "pattern": "^[a-z][a-z0-9_]{0,39}$" },
+ "additionalProperties": {
+ "oneOf": [
+ { "type": "string", "maxLength": 200 },
+ { "type": "number" },
+ { "type": "boolean" },
+ { "type": "array", "maxItems": 50, "items": { "type": "string", "maxLength": 200 } }
+ ]
+ },
+ "description": "Filterable values (category, tags, space, author subject, currency…). Filter with facet.<key>=<value>."
+ },
+ "language": { "type": "string", "pattern": "^[a-zA-Z]{2,3}(-[A-Za-z0-9]{2,8})*$", "description": "BCP 47 tag." },
+ "authorship": { "enum": ["human", "ai_assisted", "ai_generated", "imported"] },
+ "provenance": {
+ "type": "array", "maxItems": 50,
+ "description": "Where the content came from: typed references to source records (Sources items, citations, Media objects).",
+ "items": {
+ "type": "object", "required": ["service", "type", "id"], "additionalProperties": false,
+ "properties": {
+ "service": { "type": "string", "pattern": "^[a-z][a-z0-9-]{1,39}$" },
+ "type": { "type": "string", "pattern": "^[a-z][a-z0-9_]{1,39}$" },
+ "id": { "type": "string", "minLength": 1, "maxLength": 128 },
+ "revision": { "type": "integer", "minimum": 0 },
+ "label": { "type": "string", "maxLength": 200 },
+ "url": { "type": "string", "format": "uri", "pattern": "^https?://", "maxLength": 2048 },
+ "retrieved_at": { "type": "string", "format": "date-time" },
+ "stub": { "type": "boolean", "description": "true when the reference came from a stub provider; the document is then never indexable." }
+ }
+ }
+ },
+ "publication_state": {
+ "enum": ["draft", "scheduled", "published", "unpublished", "retracted", "archived"],
+ "description": "Only published documents are ever served."
+ },
+ "published_at": { "type": ["string", "null"], "format": "date-time" },
+ "updated_at": { "type": ["string", "null"], "format": "date-time" },
+ "indexability": {
+ "type": "object", "required": ["decision"], "additionalProperties": false,
+ "description": "The owner's deterministic indexability decision (publishing packages, roadmap §32.3). Search can only make it stricter.",
+ "properties": {
+ "decision": { "enum": ["index", "noindex"] },
+ "reasons": {
+ "type": "array", "maxItems": 20, "uniqueItems": true,
+ "items": { "type": "string", "pattern": "^[a-z][a-z0-9_]{1,63}$" },
+ "description": "Known reasons: draft, private, members_only, unlisted, not_published, deleted, thin_content, duplicate_without_canonical, unsupported_claims, unsourced, stub_provider, sensitive_unreviewed, ai_unreviewed, missing_canonical_url, third_party_content, owner_decision."
+ }
+ }
+ }
+ },
+ "if": { "properties": { "deleted": { "const": true } }, "required": ["deleted"] },
+ "then": {},
+ "else": { "required": ["visibility", "publication_state", "title", "indexability"] }
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,30 @@
@@ -0,0 +1,30 @@
+{
+ "version": "0.1.0",
+ "contractRanges": {
+ "openvibe-contracts": ">=0.7.0 <1.0.0"
+ },
+ "id": "search",
+ "name": "OpenVibe.Search",
+ "status": "alpha",
+ "repository": "OpenVibers/OpenVibe.Search",
+ "domains": [
+ "search.openvibe.network"
+ ],
+ "publicOrigin": "https://search.openvibe.network",
+ "health": "/api/health",
+ "ready": "/api/ready",
+ "capabilities": [
+ "search.document.write",
+ "search.query.delegate"
+ ],
+ "eventsProduced": [
+ "search.document.indexed",
+ "search.document.removed"
+ ],
+ "eventsConsumed": [
+ "*.index_document.upserted",
+ "*.index_document.deleted"
+ ],
+ "namespacesOwned": [],
+ "notes": "Canonical index document (search.index-document@1), event-fed and direct indexing with revision ordering and tombstones, permission-aware query over SQLite FTS5 (docs/adr-engine.md). Anonymous queries need no token."
+}