openvibers/openvibe.search AGENTS.md

AGENTS.md

main 7.4 KB · 124 lines · markdown

Raw HistoryBlame Repository
1# AGENTS.md — OpenVibe.Search
2
3## Project overview
4
5OpenVibe.Search is the network's permission-aware discovery service: it owns the canonical index
6document (`search.index-document@1`), receives documents from product services through OpenVibe.Events
7or the owner API, keeps a revision-ordered copy with tombstones and removals, and answers queries
8through an ACL-filtered PostgreSQL full-text engine (plus optional semantic/fabric ranking and code
9search). It is the only service that stores the network's search index; it does not own the content.
10
11- **Entry:** `server/index.js` (`start()` is also the test entry); Express app in `server/app.js`.
12- **Port / host:** `127.0.0.1:4710`, public `https://search.openvibe.network`; service id `search`.
13- **Index and query:** `server/store.js`, `server/document.js`, `server/engine/pg.js`,
14 `server/ranking.js`, `server/api/`; saved searches in `server/saved.js`.
15- **Ingest:** `POST /internal/events` (`server/app.js`) verifies and applies index-document
16 deliveries; the owner API writes directly. Removals and CDN purges are in `server/purge.js`.
17- **Fabric / semantic search:** `server/fabric/` (local `bge-small-en-v1.5` 384-dim embeddings,
18 vector store, RRF; `docs/adr-engine.md`).
19- **Code search:** `server/code/` (`registry.js`, `search.js`, `deep.js`, hosts from
20 `server/code/sources/`; `docs/code-sources.md`).
21- **Pages:** `server/web/page.js` renders the search page and saved searches with
22 `openvibe-shared/shell`; `server/web-sso.js` mounts Network sign-in. No SPA/build step.
23- **MCP:** `server/api/mcp.js` exposes the read-only search tools at `/mcp`.
24
25## Owns, does not own
26
27Owns: the `documents`, `doc_acl`, `doc_facets`, `search_fts`, `idempotency_receipts`,
28`ingest_rejections`, `service_outbox`, `removals`, `cdn_purges` and `saved_searches` tables; the
29query API and its ranking/ACL filtering; the search page; deletion and visibility propagation,
30including the CDN purge of a removed public URL.
31
32Does not own: any product's content, publication state or indexability decision (the publishing
33service is the source of truth); sitemaps, feeds or page caches (each product reads the removal feed
34or listens to `search.document.removed`); identity, groups or entitlements (Network, products,
35VIP/Billing — Search matches only the keys it is given).
36
37## Who calls it, what it calls
38
39Every publication product (Wiki, Blog, News, Reviews, Deals, Coupons, Trade, Work, Inventory and the
40rest) tells Search about its resources with the one index-document contract, normally by publishing
41an Events topic. SDK `openvibe-sdk/search` is the public query client (`query`, `iterate`,
42`suggest`, `document`, `code`, `codeIterate`); saved searches are the person's own.
43
44Search calls OpenVibe.Network (JWKS keys, service token, with `SEARCH_SAVED_NOTIFY=1` identity
45resolve and notification push), OpenVibe.Events (index-document deliveries and the
46`search.document.*` outbox), the Cloudflare purge API when configured, three code-source hosts, and
47OpenVibe.AI for deep code search (`ai.complete`). It never fetches a URL a user chose.
48
49## Database and migrations
50
51PostgreSQL `ov_search` (ADR-035) through `openvibe-sdk/db`: `DATABASE_URL` through PgBouncer for
52queries, `DATABASE_DIRECT_URL` for migrations. `server/db.js` opens and migrates at boot under an
53advisory lock; `migrations/NNNN_*.sql` are ordered and never edited after they run. The full-text
54index is derived and is reconciled/rebuilt at boot when it disagrees with `documents`. The
55`event_outbox` table remains only for the N-1 rollback window (the expand migration copies pending
56rows to `service_outbox`). A new table or column is an expand migration; removal waits for a contract
57migration.
58
59## Auth and exposure
60
61Tokens are RS256 JWTs from OpenVibe.Network, verified with the SDK JWKS client at
62`/api/.well-known/jwks`. Service tokens need audience `openvibe.search` and the route's capability:
63`search.document.write` (owner writes, own service slug only), `search.query.delegate` (query and
64saved-search routes acting for `X-OV-Subject`), `search.query.run` (public search/suggest/document
65reads) and `search.code.sources`. Delegation headers (`X-OV-Subject`, `X-OV-Groups`,
66`X-OV-Entitlements`) count only with a `search.query.delegate` service token. A bearer that does not
67verify is a 401, never anonymous.
68
69The public vhost exposes the search page, the GET query API, saved searches and `/api/health`. The
70owner API, document writes, `/internal/events` and `/metrics` are loopback-only; nginx takes client
71addresses from `$remote_addr`.
72
73## Deploy
74
75`sudo ovhost deploy search` on the production host (git-checkout: fetch, fast-forward
76`/opt/openvibe.search`, install on a lockfile change, restart, wait for `/api/ready`; rollback
77`sudo ovhost rollback search --to <sha>`). Unit `openvibe-search.service`
78(`deploy/systemd/openvibe-search.service`), env `/etc/openvibe/search.env`, nginx
79`deploy/nginx/search.openvibe.network.conf`. `ovhost deploy` removes the previous sha when readiness
80fails. The schema only adds tables/columns, so an older release still runs after a rollback.
81
82## Commands and tests
83
84```bash
85npm ci
86npm run dev # NODE_ENV=development, http://127.0.0.1:4710
87npm test # every test/*.test.js, PGlite per file, no network
88npm test -- acl outbox # only matching files
89npm run test:pg # the same on PostgreSQL (SEARCH_TEST_STORE=pg)
90npm run subscribe # create the Events subscription once (HMAC)
91node --check <file.js> # syntax check; no linter
92```
93
94`test/run.js` delegates to `openvibe-shared/test-runner` (60 s, serial): each file gets its own
95PGlite database, generated RSA keys and stub subscribers on random ports; a skipped file is not
96counted as passed. `test/acl-leak.test.js` is the privacy guard; `test/documents.test.js` covers
97revision ordering and tombstones; `test/outbox*.test.js` the N-1 outbox; `test/front.test.js` and
98`test/page.test.js` the pages.
99
100## Conventions
101
102- CommonJS, 4-space indent, single quotes, semicolons; no build step and no frontend framework.
103- Errors and responses are problem+json (`openvibe-contracts/http`); never leak an ACL or a raw
104 database row. Every API response is `no-store`.
105- Owner writes only under the caller's own service slug; visibility is enforced in the engine SQL and
106 re-checked per hit (hidden = the same 404 as missing).
107- Runtime state (timers, subscribers, sockets) starts in `server/index.js` after config load, never at
108 module load; the signal handler stops them in order.
109- Limits go through `server/actor-limits.js` (`openvibe-sdk/limits`), after the credential and before
110 any work.
111
112## Key pitfalls
113
114- **An ACL leak is the worst bug here.** `test/acl-leak.test.js` exists because a query hit must never
115 expose a document the caller cannot see; a hidden document is a 404, not a 403.
116- **Revision ordering and tombstones:** a stale or out-of-order event must not resurrect a removed or
117 older document; `documents.revision` and `removals` decide, not arrival time.
118- **The index is derived:** do not hand-write `search_fts`; reconcile/rebuild is the boot path.
119- **Events outbox:** `emit` inside the change's transaction, or the event and the row disagree after a
120 crash. `event_outbox` is only for the rollback window.
121- **Public/loopback split:** adding a route under the public vhost by accident is a security change;
122 update `deploy/nginx/search.openvibe.network.conf` and the security tests together.
123- **Saved searches store the query only** — never the person's results or identity in clear text.
124