openvibers/openvibe.search Commits
AGENTS.md: the context an agent starts from (#291)
Changes
nnew file mode 100644@@@ -0,0 +1,123 @@++# AGENTS.md — OpenVibe.Search++++## Project overview++++OpenVibe.Search is the network's permission-aware discovery service: it owns the canonical index++document (`search.index-document@1`), receives documents from product services through OpenVibe.Events++or the owner API, keeps a revision-ordered copy with tombstones and removals, and answers queries++through an ACL-filtered PostgreSQL full-text engine (plus optional semantic/fabric ranking and code++search). It is the only service that stores the network's search index; it does not own the content.++++- **Entry:** `server/index.js` (`start()` is also the test entry); Express app in `server/app.js`.++- **Port / host:** `127.0.0.1:4710`, public `https://search.openvibe.network`; service id `search`.++- **Index and query:** `server/store.js`, `server/document.js`, `server/engine/pg.js`,++ `server/ranking.js`, `server/api/`; saved searches in `server/saved.js`.++- **Ingest:** `POST /internal/events` (`server/app.js`) verifies and applies index-document++ deliveries; the owner API writes directly. Removals and CDN purges are in `server/purge.js`.++- **Fabric / semantic search:** `server/fabric/` (local `bge-small-en-v1.5` 384-dim embeddings,++ vector store, RRF; `docs/adr-engine.md`).++- **Code search:** `server/code/` (`registry.js`, `search.js`, `deep.js`, hosts from++ `server/code/sources/`; `docs/code-sources.md`).++- **Pages:** `server/web/page.js` renders the search page and saved searches with++ `openvibe-shared/shell`; `server/web-sso.js` mounts Network sign-in. No SPA/build step.++- **MCP:** `server/api/mcp.js` exposes the read-only search tools at `/mcp`.++++## Owns, does not own++++Owns: the `documents`, `doc_acl`, `doc_facets`, `search_fts`, `idempotency_receipts`,++`ingest_rejections`, `service_outbox`, `removals`, `cdn_purges` and `saved_searches` tables; the++query API and its ranking/ACL filtering; the search page; deletion and visibility propagation,++including the CDN purge of a removed public URL.++++Does not own: any product's content, publication state or indexability decision (the publishing++service is the source of truth); sitemaps, feeds or page caches (each product reads the removal feed++or listens to `search.document.removed`); identity, groups or entitlements (Network, products,++VIP/Billing — Search matches only the keys it is given).++++## Who calls it, what it calls++++Every publication product (Wiki, Blog, News, Reviews, Deals, Coupons, Trade, Work, Inventory and the++rest) tells Search about its resources with the one index-document contract, normally by publishing++an Events topic. SDK `openvibe-sdk/search` is the public query client (`query`, `iterate`,++`suggest`, `document`, `code`, `codeIterate`); saved searches are the person's own.++++Search calls OpenVibe.Network (JWKS keys, service token, with `SEARCH_SAVED_NOTIFY=1` identity++resolve and notification push), OpenVibe.Events (index-document deliveries and the++`search.document.*` outbox), the Cloudflare purge API when configured, three code-source hosts, and++OpenVibe.AI for deep code search (`ai.complete`). It never fetches a URL a user chose.++++## Database and migrations++++PostgreSQL `ov_search` (ADR-035) through `openvibe-sdk/db`: `DATABASE_URL` through PgBouncer for++queries, `DATABASE_DIRECT_URL` for migrations. `server/db.js` opens and migrates at boot under an++advisory lock; `migrations/NNNN_*.sql` are ordered and never edited after they run. The full-text++index is derived and is reconciled/rebuilt at boot when it disagrees with `documents`. The++`event_outbox` table remains only for the N-1 rollback window (the expand migration copies pending++rows to `service_outbox`). A new table or column is an expand migration; removal waits for a contract++migration.++++## Auth and exposure++++Tokens are RS256 JWTs from OpenVibe.Network, verified with the SDK JWKS client at++`/api/.well-known/jwks`. Service tokens need audience `openvibe.search` and the route's capability:++`search.document.write` (owner writes, own service slug only), `search.query.delegate` (query and++saved-search routes acting for `X-OV-Subject`), `search.query.run` (public search/suggest/document++reads) and `search.code.sources`. Delegation headers (`X-OV-Subject`, `X-OV-Groups`,++`X-OV-Entitlements`) count only with a `search.query.delegate` service token. A bearer that does not++verify is a 401, never anonymous.++++The public vhost exposes the search page, the GET query API, saved searches and `/api/health`. The++owner API, document writes, `/internal/events` and `/metrics` are loopback-only; nginx takes client++addresses from `$remote_addr`.++++## Deploy++++`sudo ovhost deploy search` on the production host (git-checkout: fetch, fast-forward++`/opt/openvibe.search`, install on a lockfile change, restart, wait for `/api/ready`; rollback++`sudo ovhost rollback search --to <sha>`). Unit `openvibe-search.service`++(`deploy/systemd/openvibe-search.service`), env `/etc/openvibe/search.env`, nginx++`deploy/nginx/search.openvibe.network.conf`. `ovhost deploy` removes the previous sha when readiness++fails. The schema only adds tables/columns, so an older release still runs after a rollback.++++## Commands and tests++++```bash++npm ci++npm run dev # NODE_ENV=development, http://127.0.0.1:4710++npm test # every test/*.test.js, PGlite per file, no network++npm test -- acl outbox # only matching files++npm run test:pg # the same on PostgreSQL (SEARCH_TEST_STORE=pg)++npm run subscribe # create the Events subscription once (HMAC)++node --check <file.js> # syntax check; no linter++```++++`test/run.js` delegates to `openvibe-shared/test-runner` (60 s, serial): each file gets its own++PGlite database, generated RSA keys and stub subscribers on random ports; a skipped file is not++counted as passed. `test/acl-leak.test.js` is the privacy guard; `test/documents.test.js` covers++revision ordering and tombstones; `test/outbox*.test.js` the N-1 outbox; `test/front.test.js` and++`test/page.test.js` the pages.++++## Conventions++++- CommonJS, 4-space indent, single quotes, semicolons; no build step and no frontend framework.++- Errors and responses are problem+json (`openvibe-contracts/http`); never leak an ACL or a raw++ database row. Every API response is `no-store`.++- Owner writes only under the caller's own service slug; visibility is enforced in the engine SQL and++ re-checked per hit (hidden = the same 404 as missing).++- Runtime state (timers, subscribers, sockets) starts in `server/index.js` after config load, never at++ module load; the signal handler stops them in order.++- Limits go through `server/actor-limits.js` (`openvibe-sdk/limits`), after the credential and before++ any work.++++## Key pitfalls++++- **An ACL leak is the worst bug here.** `test/acl-leak.test.js` exists because a query hit must never++ expose a document the caller cannot see; a hidden document is a 404, not a 403.++- **Revision ordering and tombstones:** a stale or out-of-order event must not resurrect a removed or++ older document; `documents.revision` and `removals` decide, not arrival time.++- **The index is derived:** do not hand-write `search_fts`; reconcile/rebuild is the boot path.++- **Events outbox:** `emit` inside the change's transaction, or the event and the row disagree after a++ crash. `event_outbox` is only for the rollback window.++- **Public/loopback split:** adding a route under the public vhost by accident is a security change;++ update `deploy/nginx/search.openvibe.network.conf` and the security tests together.++- **Saved searches store the query only** — never the person's results or identity in clear text.nnew file mode 100644@@@ -0,0 +1 @@
new file mode 100644
new file mode 100644
@@ -0,0 +1,123 @@
@@ -0,0 +1,123 @@
+# AGENTS.md — OpenVibe.Search
+
+## Project overview
+
+OpenVibe.Search is the network's permission-aware discovery service: it owns the canonical index
+document (`search.index-document@1`), receives documents from product services through OpenVibe.Events
+or the owner API, keeps a revision-ordered copy with tombstones and removals, and answers queries
+through an ACL-filtered PostgreSQL full-text engine (plus optional semantic/fabric ranking and code
+search). It is the only service that stores the network's search index; it does not own the content.
+
+- **Entry:** `server/index.js` (`start()` is also the test entry); Express app in `server/app.js`.
+- **Port / host:** `127.0.0.1:4710`, public `https://search.openvibe.network`; service id `search`.
+- **Index and query:** `server/store.js`, `server/document.js`, `server/engine/pg.js`,
+ `server/ranking.js`, `server/api/`; saved searches in `server/saved.js`.
+- **Ingest:** `POST /internal/events` (`server/app.js`) verifies and applies index-document
+ deliveries; the owner API writes directly. Removals and CDN purges are in `server/purge.js`.
+- **Fabric / semantic search:** `server/fabric/` (local `bge-small-en-v1.5` 384-dim embeddings,
+ vector store, RRF; `docs/adr-engine.md`).
+- **Code search:** `server/code/` (`registry.js`, `search.js`, `deep.js`, hosts from
+ `server/code/sources/`; `docs/code-sources.md`).
+- **Pages:** `server/web/page.js` renders the search page and saved searches with
+ `openvibe-shared/shell`; `server/web-sso.js` mounts Network sign-in. No SPA/build step.
+- **MCP:** `server/api/mcp.js` exposes the read-only search tools at `/mcp`.
+
+## Owns, does not own
+
+Owns: the `documents`, `doc_acl`, `doc_facets`, `search_fts`, `idempotency_receipts`,
+`ingest_rejections`, `service_outbox`, `removals`, `cdn_purges` and `saved_searches` tables; the
+query API and its ranking/ACL filtering; the search page; deletion and visibility propagation,
+including the CDN purge of a removed public URL.
+
+Does not own: any product's content, publication state or indexability decision (the publishing
+service is the source of truth); sitemaps, feeds or page caches (each product reads the removal feed
+or listens to `search.document.removed`); identity, groups or entitlements (Network, products,
+VIP/Billing — Search matches only the keys it is given).
+
+## Who calls it, what it calls
+
+Every publication product (Wiki, Blog, News, Reviews, Deals, Coupons, Trade, Work, Inventory and the
+rest) tells Search about its resources with the one index-document contract, normally by publishing
+an Events topic. SDK `openvibe-sdk/search` is the public query client (`query`, `iterate`,
+`suggest`, `document`, `code`, `codeIterate`); saved searches are the person's own.
+
+Search calls OpenVibe.Network (JWKS keys, service token, with `SEARCH_SAVED_NOTIFY=1` identity
+resolve and notification push), OpenVibe.Events (index-document deliveries and the
+`search.document.*` outbox), the Cloudflare purge API when configured, three code-source hosts, and
+OpenVibe.AI for deep code search (`ai.complete`). It never fetches a URL a user chose.
+
+## Database and migrations
+
+PostgreSQL `ov_search` (ADR-035) through `openvibe-sdk/db`: `DATABASE_URL` through PgBouncer for
+queries, `DATABASE_DIRECT_URL` for migrations. `server/db.js` opens and migrates at boot under an
+advisory lock; `migrations/NNNN_*.sql` are ordered and never edited after they run. The full-text
+index is derived and is reconciled/rebuilt at boot when it disagrees with `documents`. The
+`event_outbox` table remains only for the N-1 rollback window (the expand migration copies pending
+rows to `service_outbox`). A new table or column is an expand migration; removal waits for a contract
+migration.
+
+## Auth and exposure
+
+Tokens are RS256 JWTs from OpenVibe.Network, verified with the SDK JWKS client at
+`/api/.well-known/jwks`. Service tokens need audience `openvibe.search` and the route's capability:
+`search.document.write` (owner writes, own service slug only), `search.query.delegate` (query and
+saved-search routes acting for `X-OV-Subject`), `search.query.run` (public search/suggest/document
+reads) and `search.code.sources`. Delegation headers (`X-OV-Subject`, `X-OV-Groups`,
+`X-OV-Entitlements`) count only with a `search.query.delegate` service token. A bearer that does not
+verify is a 401, never anonymous.
+
+The public vhost exposes the search page, the GET query API, saved searches and `/api/health`. The
+owner API, document writes, `/internal/events` and `/metrics` are loopback-only; nginx takes client
+addresses from `$remote_addr`.
+
+## Deploy
+
+`sudo ovhost deploy search` on the production host (git-checkout: fetch, fast-forward
+`/opt/openvibe.search`, install on a lockfile change, restart, wait for `/api/ready`; rollback
+`sudo ovhost rollback search --to <sha>`). Unit `openvibe-search.service`
+(`deploy/systemd/openvibe-search.service`), env `/etc/openvibe/search.env`, nginx
+`deploy/nginx/search.openvibe.network.conf`. `ovhost deploy` removes the previous sha when readiness
+fails. The schema only adds tables/columns, so an older release still runs after a rollback.
+
+## Commands and tests
+
+```bash
+npm ci
+npm run dev # NODE_ENV=development, http://127.0.0.1:4710
+npm test # every test/*.test.js, PGlite per file, no network
+npm test -- acl outbox # only matching files
+npm run test:pg # the same on PostgreSQL (SEARCH_TEST_STORE=pg)
+npm run subscribe # create the Events subscription once (HMAC)
+node --check <file.js> # syntax check; no linter
+```
+
+`test/run.js` delegates to `openvibe-shared/test-runner` (60 s, serial): each file gets its own
+PGlite database, generated RSA keys and stub subscribers on random ports; a skipped file is not
+counted as passed. `test/acl-leak.test.js` is the privacy guard; `test/documents.test.js` covers
+revision ordering and tombstones; `test/outbox*.test.js` the N-1 outbox; `test/front.test.js` and
+`test/page.test.js` the pages.
+
+## Conventions
+
+- CommonJS, 4-space indent, single quotes, semicolons; no build step and no frontend framework.
+- Errors and responses are problem+json (`openvibe-contracts/http`); never leak an ACL or a raw
+ database row. Every API response is `no-store`.
+- Owner writes only under the caller's own service slug; visibility is enforced in the engine SQL and
+ re-checked per hit (hidden = the same 404 as missing).
+- Runtime state (timers, subscribers, sockets) starts in `server/index.js` after config load, never at
+ module load; the signal handler stops them in order.
+- Limits go through `server/actor-limits.js` (`openvibe-sdk/limits`), after the credential and before
+ any work.
+
+## Key pitfalls
+
+- **An ACL leak is the worst bug here.** `test/acl-leak.test.js` exists because a query hit must never
+ expose a document the caller cannot see; a hidden document is a 404, not a 403.
+- **Revision ordering and tombstones:** a stale or out-of-order event must not resurrect a removed or
+ older document; `documents.revision` and `removals` decide, not arrival time.
+- **The index is derived:** do not hand-write `search_fts`; reconcile/rebuild is the boot path.
+- **Events outbox:** `emit` inside the change's transaction, or the event and the row disagree after a
+ crash. `event_outbox` is only for the rollback window.
+- **Public/loopback split:** adding a route under the public vhost by accident is a security change;
+ update `deploy/nginx/search.openvibe.network.conf` and the security tests together.
+- **Saved searches store the query only** — never the person's results or identity in clear text.
new file mode 100644
new file mode 100644
@@ -0,0 +1 @@
@@ -0,0 +1 @@