openvibers/openvibe.search Commits

AGENTS.md: the context an agent starts from (#291)

OP OpenVibers committed 1 days ago · 666dc313f3 · 2 files, +124 −0

Adds `AGENTS.md` (125 lines) as the context an agent starts from, plus `CLAUDE.md` (`@AGENTS.md`).

It covers: what Search owns (the index document, index/ACL tables, query API, pages) and what it
must never own (product content, identity, entitlements); the event-fed and direct ingest paths;
the services it calls and the capabilities it requires; the PostgreSQL/migrations rule and the
derived full-text index; the public-vhost vs loopback split; deploy across `ovhost`, systemd and
nginx; the test runner and the ACL/tombstone/outbox guards; conventions; and the pitfalls visible
in its tests and docs.

Checked: every cited path exists on main; no code, dependency or version changed.

Changes

ddiff --git a/AGENTS.md b/AGENTS.md
nnew file mode 100644
iindex 0000000..42f5977
---- /dev/null
++++ b/AGENTS.md
@@@ -0,0 +1,123 @@
++# AGENTS.md — OpenVibe.Search
++
++## Project overview
++
++OpenVibe.Search is the network's permission-aware discovery service: it owns the canonical index
++document (`search.index-document@1`), receives documents from product services through OpenVibe.Events
++or the owner API, keeps a revision-ordered copy with tombstones and removals, and answers queries
++through an ACL-filtered PostgreSQL full-text engine (plus optional semantic/fabric ranking and code
++search). It is the only service that stores the network's search index; it does not own the content.
++
++- **Entry:** `server/index.js` (`start()` is also the test entry); Express app in `server/app.js`.
++- **Port / host:** `127.0.0.1:4710`, public `https://search.openvibe.network`; service id `search`.
++- **Index and query:** `server/store.js`, `server/document.js`, `server/engine/pg.js`,
++ `server/ranking.js`, `server/api/`; saved searches in `server/saved.js`.
++- **Ingest:** `POST /internal/events` (`server/app.js`) verifies and applies index-document
++ deliveries; the owner API writes directly. Removals and CDN purges are in `server/purge.js`.
++- **Fabric / semantic search:** `server/fabric/` (local `bge-small-en-v1.5` 384-dim embeddings,
++ vector store, RRF; `docs/adr-engine.md`).
++- **Code search:** `server/code/` (`registry.js`, `search.js`, `deep.js`, hosts from
++ `server/code/sources/`; `docs/code-sources.md`).
++- **Pages:** `server/web/page.js` renders the search page and saved searches with
++ `openvibe-shared/shell`; `server/web-sso.js` mounts Network sign-in. No SPA/build step.
++- **MCP:** `server/api/mcp.js` exposes the read-only search tools at `/mcp`.
++
++## Owns, does not own
++
++Owns: the `documents`, `doc_acl`, `doc_facets`, `search_fts`, `idempotency_receipts`,
++`ingest_rejections`, `service_outbox`, `removals`, `cdn_purges` and `saved_searches` tables; the
++query API and its ranking/ACL filtering; the search page; deletion and visibility propagation,
++including the CDN purge of a removed public URL.
++
++Does not own: any product's content, publication state or indexability decision (the publishing
++service is the source of truth); sitemaps, feeds or page caches (each product reads the removal feed
++or listens to `search.document.removed`); identity, groups or entitlements (Network, products,
++VIP/Billing — Search matches only the keys it is given).
++
++## Who calls it, what it calls
++
++Every publication product (Wiki, Blog, News, Reviews, Deals, Coupons, Trade, Work, Inventory and the
++rest) tells Search about its resources with the one index-document contract, normally by publishing
++an Events topic. SDK `openvibe-sdk/search` is the public query client (`query`, `iterate`,
++`suggest`, `document`, `code`, `codeIterate`); saved searches are the person's own.
++
++Search calls OpenVibe.Network (JWKS keys, service token, with `SEARCH_SAVED_NOTIFY=1` identity
++resolve and notification push), OpenVibe.Events (index-document deliveries and the
++`search.document.*` outbox), the Cloudflare purge API when configured, three code-source hosts, and
++OpenVibe.AI for deep code search (`ai.complete`). It never fetches a URL a user chose.
++
++## Database and migrations
++
++PostgreSQL `ov_search` (ADR-035) through `openvibe-sdk/db`: `DATABASE_URL` through PgBouncer for
++queries, `DATABASE_DIRECT_URL` for migrations. `server/db.js` opens and migrates at boot under an
++advisory lock; `migrations/NNNN_*.sql` are ordered and never edited after they run. The full-text
++index is derived and is reconciled/rebuilt at boot when it disagrees with `documents`. The
++`event_outbox` table remains only for the N-1 rollback window (the expand migration copies pending
++rows to `service_outbox`). A new table or column is an expand migration; removal waits for a contract
++migration.
++
++## Auth and exposure
++
++Tokens are RS256 JWTs from OpenVibe.Network, verified with the SDK JWKS client at
++`/api/.well-known/jwks`. Service tokens need audience `openvibe.search` and the route's capability:
++`search.document.write` (owner writes, own service slug only), `search.query.delegate` (query and
++saved-search routes acting for `X-OV-Subject`), `search.query.run` (public search/suggest/document
++reads) and `search.code.sources`. Delegation headers (`X-OV-Subject`, `X-OV-Groups`,
++`X-OV-Entitlements`) count only with a `search.query.delegate` service token. A bearer that does not
++verify is a 401, never anonymous.
++
++The public vhost exposes the search page, the GET query API, saved searches and `/api/health`. The
++owner API, document writes, `/internal/events` and `/metrics` are loopback-only; nginx takes client
++addresses from `$remote_addr`.
++
++## Deploy
++
++`sudo ovhost deploy search` on the production host (git-checkout: fetch, fast-forward
++`/opt/openvibe.search`, install on a lockfile change, restart, wait for `/api/ready`; rollback
++`sudo ovhost rollback search --to <sha>`). Unit `openvibe-search.service`
++(`deploy/systemd/openvibe-search.service`), env `/etc/openvibe/search.env`, nginx
++`deploy/nginx/search.openvibe.network.conf`. `ovhost deploy` removes the previous sha when readiness
++fails. The schema only adds tables/columns, so an older release still runs after a rollback.
++
++## Commands and tests
++
++```bash
++npm ci
++npm run dev # NODE_ENV=development, http://127.0.0.1:4710
++npm test # every test/*.test.js, PGlite per file, no network
++npm test -- acl outbox # only matching files
++npm run test:pg # the same on PostgreSQL (SEARCH_TEST_STORE=pg)
++npm run subscribe # create the Events subscription once (HMAC)
++node --check <file.js> # syntax check; no linter
++```
++
++`test/run.js` delegates to `openvibe-shared/test-runner` (60 s, serial): each file gets its own
++PGlite database, generated RSA keys and stub subscribers on random ports; a skipped file is not
++counted as passed. `test/acl-leak.test.js` is the privacy guard; `test/documents.test.js` covers
++revision ordering and tombstones; `test/outbox*.test.js` the N-1 outbox; `test/front.test.js` and
++`test/page.test.js` the pages.
++
++## Conventions
++
++- CommonJS, 4-space indent, single quotes, semicolons; no build step and no frontend framework.
++- Errors and responses are problem+json (`openvibe-contracts/http`); never leak an ACL or a raw
++ database row. Every API response is `no-store`.
++- Owner writes only under the caller's own service slug; visibility is enforced in the engine SQL and
++ re-checked per hit (hidden = the same 404 as missing).
++- Runtime state (timers, subscribers, sockets) starts in `server/index.js` after config load, never at
++ module load; the signal handler stops them in order.
++- Limits go through `server/actor-limits.js` (`openvibe-sdk/limits`), after the credential and before
++ any work.
++
++## Key pitfalls
++
++- **An ACL leak is the worst bug here.** `test/acl-leak.test.js` exists because a query hit must never
++ expose a document the caller cannot see; a hidden document is a 404, not a 403.
++- **Revision ordering and tombstones:** a stale or out-of-order event must not resurrect a removed or
++ older document; `documents.revision` and `removals` decide, not arrival time.
++- **The index is derived:** do not hand-write `search_fts`; reconcile/rebuild is the boot path.
++- **Events outbox:** `emit` inside the change's transaction, or the event and the row disagree after a
++ crash. `event_outbox` is only for the rollback window.
++- **Public/loopback split:** adding a route under the public vhost by accident is a security change;
++ update `deploy/nginx/search.openvibe.network.conf` and the security tests together.
++- **Saved searches store the query only** — never the person's results or identity in clear text.
ddiff --git a/CLAUDE.md b/CLAUDE.md
nnew file mode 100644
iindex 0000000..43c994c
---- /dev/null
++++ b/CLAUDE.md
@@@ -0,0 +1 @@