openvibers/openvibe.sources AGENTS.md
AGENTS.md
| 1 | # AGENTS.md — OpenVibe.Sources |
| 2 | |
| 3 | ## Project overview |
| 4 | |
| 5 | OpenVibe.Sources is the one place outside material enters the network: a registry of sources with |
| 6 | their terms and a polite fetcher/ingester that records provenance, retrieval time, content hashes and |
| 7 | explicit failure states. News, Reviews, Deals, Coupons, Trade and AI summarisation read its items; |
| 8 | the products decide what, if anything, to publish. |
| 9 | |
| 10 | - **Entry:** `server/index.js` (Express; `start()` is also the test entry); port `4720`, |
| 11 | service id `sources`, internal host `sources.openvibe.network`. |
| 12 | - **Registry:** `server/registry.js`, `server/settings.js`, contracts `sources.source@1`; seeds in |
| 13 | `seeds/sources.json`. |
| 14 | - **Fetch pipeline:** `server/net/guard.js` (public-address check), `server/net/fetcher.js`, |
| 15 | `server/robots.js`, `server/spacer.js`, `server/ingest.js`, `server/scheduler.js`. |
| 16 | - **Adapters:** `server/adapters/` (feed, sitemap, jsonld, api, bulk, crawl, manual); evidence-derived |
| 17 | adapters in `adapters/<domain>.json`; `server/crawl.js` for the crawl type. |
| 18 | - **Items and runs:** `server/items.js` (identity, revisions, content hash); jobs in |
| 19 | `server/jobs/`; events in `server/events/`. |
| 20 | - **Merchant opt-in:** `server/merchant.js`, `server/merchant-auth.js`, `server/merchant-pages.js`, |
| 21 | `server/views/merchant.js`; public pages in `server/views/home.js`. |
| 22 | - **Pages:** server-rendered with `openvibe-shared/shell`; `public/merchant-arrival.js` is the only |
| 23 | own browser script. No SPA and no build step. |
| 24 | |
| 25 | ## Owns, does not own |
| 26 | |
| 27 | Owns the source registry (`sources`, `endpoint_state`, `robots_cache`), merchant opt-in claims |
| 28 | (`merchant_sources`) and their proof checks, ingestion (`fetch_runs`, `items`, `item_revisions`, the |
| 29 | scheduler, robots and rate-limit state), and the `sources.*` events. |
| 30 | |
| 31 | Does not own: publication — what a product shows, summarises, rates or indexes from an item (News, |
| 32 | Reviews, Deals, Coupons, Trade and the publishing packages' indexability gate); AI summarisation |
| 33 | (OpenVibe.AI invokes the adapters' output, it does not fetch here); legal clearance (a terms note |
| 34 | records what a person verified, it does not replace verifying). |
| 35 | |
| 36 | **The AI ↔ Sources client does not exist:** Sources never calls AI, and AI never fetches from |
| 37 | Sources. The product is the only client of both, passing Sources items to AI with provenance. |
| 38 | |
| 39 | ## Who calls it, what it calls |
| 40 | |
| 41 | Publication products call the API with a Network client-credentials token for audience |
| 42 | `openvibe.sources`; every route has one capability (`sources.source.read`, item reads, and the |
| 43 | merchant routes). A first-party service may act for a person with `X-OV-Subject`; the merchant API |
| 44 | and pages accept the person's own session (Bearer on the API, the first-party cookie on the pages). |
| 45 | SDK `openvibe-sdk/sources` is the client surface. There is **no public JSON API**: the vhost keeps |
| 46 | `/api/v1/*` loopback-only; only `/merchant*` and `/shared/*` are public. |
| 47 | |
| 48 | Sources calls the outside hosts named by its sources (subject to robots, spacing and the address |
| 49 | guard), OpenVibe.Network (JWKS, service token, account export/deletion, merchant identity) and |
| 50 | OpenVibe.Events (outbox). It never fetches a URL a user chose. |
| 51 | |
| 52 | ## Database and migrations |
| 53 | |
| 54 | PostgreSQL `ov_sources` (ADR-035) through `openvibe-sdk/db`: `DATABASE_URL` through PgBouncer, |
| 55 | `DATABASE_DIRECT_URL` for migrations; `server/db.js` opens and migrates at boot under an advisory |
| 56 | lock. `migrations/0001..0012` include crawl, merchant sources, account data, source state, item |
| 57 | retention, config, telemetry and the outbox rejection index; never edit one that has run. A failed |
| 58 | fetch writes a `fetch_runs` row and **never creates or modifies an item**; only the `ok` path touches |
| 59 | items, and validators (ETag/Last-Modified) are stored only after a good parse. |
| 60 | |
| 61 | ## Auth and exposure |
| 62 | |
| 63 | Network RS256 JWTs verified through the SDK JWKS client. Every `/api/v1` route needs a service token |
| 64 | for audience `openvibe.sources` with that route's capability, then passes per-actor limits. The |
| 65 | merchant API/pages are the person-facing exception: a person's session owns their claims, and a |
| 66 | first-party service may act for a person only with `X-OV-Subject`. A source credential is read from |
| 67 | the environment variable it names (`SOURCES_CRED_*` only), sent only to the configured origin |
| 68 | (dropped on a cross-origin redirect), and never stored or logged. |
| 69 | |
| 70 | ## Deploy |
| 71 | |
| 72 | `sudo ovhost deploy sources` (git-checkout: fetch, fast-forward `/opt/openvibe.sources`, install on a |
| 73 | lockfile change, restart, wait for `/api/ready`; rollback `sudo ovhost rollback sources --to <sha>`). |
| 74 | Unit `deploy/systemd/openvibe-sources.service`, env `/etc/openvibe/sources.env`, database `ov_sources`, |
| 75 | nginx `deploy/nginx/sources.openvibe.network.conf` (API loopback-only, merchant + shared public). |
| 76 | The scheduler runs in-process; it starts after migrations in `server/index.js`, never at module load. |
| 77 | |
| 78 | ## Commands and tests |
| 79 | |
| 80 | ```bash |
| 81 | npm install |
| 82 | npm run seed # create the sources in seeds/sources.json, disabled |
| 83 | npm run dev # http://127.0.0.1:4720 |
| 84 | npm test # every test/*.test.js against local stub sites, no internet |
| 85 | npm test -- crawl robots # only matching files |
| 86 | npm run test:pg # the same through PgBouncer |
| 87 | npm run probe -- <domain> # write evidence/<domain>.json (run from the host's egress) |
| 88 | npm run adapters # evidence → adapters or compliance/review-queue.yaml |
| 89 | npm run register-adapters # bring the registry to the adapters |
| 90 | ``` |
| 91 | |
| 92 | `test/run.js` uses `openvibe-shared/test-runner`; tests boot local stub sites and PGlite and never |
| 93 | touch the internet. Coverage includes adapters, extract, crawl, robots/rate limiting, the API |
| 94 | (CSV/TSV/join/map), scheduler, merchant proof, account data/deletion and the service manifest. |
| 95 | |
| 96 | ## Conventions and pitfalls |
| 97 | |
| 98 | - CommonJS, 4-space indent, single quotes, semicolons; problem+json errors, no raw rows. |
| 99 | - **Provenance is the product:** every item keeps source, fetch time, content hash and parser |
| 100 | version; a failure stays a recorded failure and is never filled with invented content. |
| 101 | - **ADR 0007 is Accepted and the fixed technique list is not up for discussion.** Read |
| 102 | `docs/adr/0007-source-access.md`; a probe verdict from `evidence/<domain>.json` decides access — |
| 103 | probe first, read existing evidence, do not re-derive or re-litigate it. |
| 104 | - **Never enable a source without a terms note** (and a robots note when fetched); an adapter |
| 105 | existing is not permission. A source with items cannot be deleted, only disabled. |
| 106 | - **Politeness is enforced:** robots.txt (RFC 9309, 4xx = none, 429/5xx = disallow), max request |
| 107 | spacing (`min_interval_ms`, `Crawl-delay`), conditional GET, one response deadline, byte cap, |
| 108 | public-address check at connect time, manual re-checked redirects, exponential backoff. |
| 109 | - **Dedupe and classification split:** Sources dedupes within a source (identity + content hash) and |
| 110 | assigns a coarse category; cross-source dedupe and fine classification are the product's. |
| 111 | - **A failed run does not touch items and does not store validators** — the next pass re-fetches in |
| 112 | full. Keep the `fetch_runs` state names exact. |
| 113 |