openvibers/openvibe.sources Commits
AGENTS.md: the context an agent starts from (#282)
Changes
nnew file mode 100644@@@ -0,0 +1,112 @@++# AGENTS.md — OpenVibe.Sources++++## Project overview++++OpenVibe.Sources is the one place outside material enters the network: a registry of sources with++their terms and a polite fetcher/ingester that records provenance, retrieval time, content hashes and++explicit failure states. News, Reviews, Deals, Coupons, Trade and AI summarisation read its items;++the products decide what, if anything, to publish.++++- **Entry:** `server/index.js` (Express; `start()` is also the test entry); port `4720`,++ service id `sources`, internal host `sources.openvibe.network`.++- **Registry:** `server/registry.js`, `server/settings.js`, contracts `sources.source@1`; seeds in++ `seeds/sources.json`.++- **Fetch pipeline:** `server/net/guard.js` (public-address check), `server/net/fetcher.js`,++ `server/robots.js`, `server/spacer.js`, `server/ingest.js`, `server/scheduler.js`.++- **Adapters:** `server/adapters/` (feed, sitemap, jsonld, api, bulk, crawl, manual); evidence-derived++ adapters in `adapters/<domain>.json`; `server/crawl.js` for the crawl type.++- **Items and runs:** `server/items.js` (identity, revisions, content hash); jobs in++ `server/jobs/`; events in `server/events/`.++- **Merchant opt-in:** `server/merchant.js`, `server/merchant-auth.js`, `server/merchant-pages.js`,++ `server/views/merchant.js`; public pages in `server/views/home.js`.++- **Pages:** server-rendered with `openvibe-shared/shell`; `public/merchant-arrival.js` is the only++ own browser script. No SPA and no build step.++++## Owns, does not own++++Owns the source registry (`sources`, `endpoint_state`, `robots_cache`), merchant opt-in claims++(`merchant_sources`) and their proof checks, ingestion (`fetch_runs`, `items`, `item_revisions`, the++scheduler, robots and rate-limit state), and the `sources.*` events.++++Does not own: publication — what a product shows, summarises, rates or indexes from an item (News,++Reviews, Deals, Coupons, Trade and the publishing packages' indexability gate); AI summarisation++(OpenVibe.AI invokes the adapters' output, it does not fetch here); legal clearance (a terms note++records what a person verified, it does not replace verifying).++++**The AI ↔ Sources client does not exist:** Sources never calls AI, and AI never fetches from++Sources. The product is the only client of both, passing Sources items to AI with provenance.++++## Who calls it, what it calls++++Publication products call the API with a Network client-credentials token for audience++`openvibe.sources`; every route has one capability (`sources.source.read`, item reads, and the++merchant routes). A first-party service may act for a person with `X-OV-Subject`; the merchant API++and pages accept the person's own session (Bearer on the API, the first-party cookie on the pages).++SDK `openvibe-sdk/sources` is the client surface. There is **no public JSON API**: the vhost keeps++`/api/v1/*` loopback-only; only `/merchant*` and `/shared/*` are public.++++Sources calls the outside hosts named by its sources (subject to robots, spacing and the address++guard), OpenVibe.Network (JWKS, service token, account export/deletion, merchant identity) and++OpenVibe.Events (outbox). It never fetches a URL a user chose.++++## Database and migrations++++PostgreSQL `ov_sources` (ADR-035) through `openvibe-sdk/db`: `DATABASE_URL` through PgBouncer,++`DATABASE_DIRECT_URL` for migrations; `server/db.js` opens and migrates at boot under an advisory++lock. `migrations/0001..0012` include crawl, merchant sources, account data, source state, item++retention, config, telemetry and the outbox rejection index; never edit one that has run. A failed++fetch writes a `fetch_runs` row and **never creates or modifies an item**; only the `ok` path touches++items, and validators (ETag/Last-Modified) are stored only after a good parse.++++## Auth and exposure++++Network RS256 JWTs verified through the SDK JWKS client. Every `/api/v1` route needs a service token++for audience `openvibe.sources` with that route's capability, then passes per-actor limits. The++merchant API/pages are the person-facing exception: a person's session owns their claims, and a++first-party service may act for a person only with `X-OV-Subject`. A source credential is read from++the environment variable it names (`SOURCES_CRED_*` only), sent only to the configured origin++(dropped on a cross-origin redirect), and never stored or logged.++++## Deploy++++`sudo ovhost deploy sources` (git-checkout: fetch, fast-forward `/opt/openvibe.sources`, install on a++lockfile change, restart, wait for `/api/ready`; rollback `sudo ovhost rollback sources --to <sha>`).++Unit `deploy/systemd/openvibe-sources.service`, env `/etc/openvibe/sources.env`, database `ov_sources`,++nginx `deploy/nginx/sources.openvibe.network.conf` (API loopback-only, merchant + shared public).++The scheduler runs in-process; it starts after migrations in `server/index.js`, never at module load.++++## Commands and tests++++```bash++npm install++npm run seed # create the sources in seeds/sources.json, disabled++npm run dev # http://127.0.0.1:4720++npm test # every test/*.test.js against local stub sites, no internet++npm test -- crawl robots # only matching files++npm run test:pg # the same through PgBouncer++npm run probe -- <domain> # write evidence/<domain>.json (run from the host's egress)++npm run adapters # evidence → adapters or compliance/review-queue.yaml++npm run register-adapters # bring the registry to the adapters++```++++`test/run.js` uses `openvibe-shared/test-runner`; tests boot local stub sites and PGlite and never++touch the internet. Coverage includes adapters, extract, crawl, robots/rate limiting, the API++(CSV/TSV/join/map), scheduler, merchant proof, account data/deletion and the service manifest.++++## Conventions and pitfalls++++- CommonJS, 4-space indent, single quotes, semicolons; problem+json errors, no raw rows.++- **Provenance is the product:** every item keeps source, fetch time, content hash and parser++ version; a failure stays a recorded failure and is never filled with invented content.++- **ADR 0007 is Accepted and the fixed technique list is not up for discussion.** Read++ `docs/adr/0007-source-access.md`; a probe verdict from `evidence/<domain>.json` decides access —++ probe first, read existing evidence, do not re-derive or re-litigate it.++- **Never enable a source without a terms note** (and a robots note when fetched); an adapter++ existing is not permission. A source with items cannot be deleted, only disabled.++- **Politeness is enforced:** robots.txt (RFC 9309, 4xx = none, 429/5xx = disallow), max request++ spacing (`min_interval_ms`, `Crawl-delay`), conditional GET, one response deadline, byte cap,++ public-address check at connect time, manual re-checked redirects, exponential backoff.++- **Dedupe and classification split:** Sources dedupes within a source (identity + content hash) and++ assigns a coarse category; cross-source dedupe and fine classification are the product's.++- **A failed run does not touch items and does not store validators** — the next pass re-fetches in++ full. Keep the `fetch_runs` state names exact.@@@ -1,8 +1 @@--# OpenVibe.Sources: notes for agents----Read README.md first. Source access:----Access verdicts come from evidence/<domain>.json, written by scripts/probe.mjs.--Probe first; if evidence exists, read it and do not re-derive it.--docs/adr/0007-source-access.md is Accepted. Re-litigating it is a code comment.--The fixed technique list in ADR 0007 is not up for discussion.
new file mode 100644
new file mode 100644
@@ -0,0 +1,112 @@
@@ -0,0 +1,112 @@
+# AGENTS.md — OpenVibe.Sources
+
+## Project overview
+
+OpenVibe.Sources is the one place outside material enters the network: a registry of sources with
+their terms and a polite fetcher/ingester that records provenance, retrieval time, content hashes and
+explicit failure states. News, Reviews, Deals, Coupons, Trade and AI summarisation read its items;
+the products decide what, if anything, to publish.
+
+- **Entry:** `server/index.js` (Express; `start()` is also the test entry); port `4720`,
+ service id `sources`, internal host `sources.openvibe.network`.
+- **Registry:** `server/registry.js`, `server/settings.js`, contracts `sources.source@1`; seeds in
+ `seeds/sources.json`.
+- **Fetch pipeline:** `server/net/guard.js` (public-address check), `server/net/fetcher.js`,
+ `server/robots.js`, `server/spacer.js`, `server/ingest.js`, `server/scheduler.js`.
+- **Adapters:** `server/adapters/` (feed, sitemap, jsonld, api, bulk, crawl, manual); evidence-derived
+ adapters in `adapters/<domain>.json`; `server/crawl.js` for the crawl type.
+- **Items and runs:** `server/items.js` (identity, revisions, content hash); jobs in
+ `server/jobs/`; events in `server/events/`.
+- **Merchant opt-in:** `server/merchant.js`, `server/merchant-auth.js`, `server/merchant-pages.js`,
+ `server/views/merchant.js`; public pages in `server/views/home.js`.
+- **Pages:** server-rendered with `openvibe-shared/shell`; `public/merchant-arrival.js` is the only
+ own browser script. No SPA and no build step.
+
+## Owns, does not own
+
+Owns the source registry (`sources`, `endpoint_state`, `robots_cache`), merchant opt-in claims
+(`merchant_sources`) and their proof checks, ingestion (`fetch_runs`, `items`, `item_revisions`, the
+scheduler, robots and rate-limit state), and the `sources.*` events.
+
+Does not own: publication — what a product shows, summarises, rates or indexes from an item (News,
+Reviews, Deals, Coupons, Trade and the publishing packages' indexability gate); AI summarisation
+(OpenVibe.AI invokes the adapters' output, it does not fetch here); legal clearance (a terms note
+records what a person verified, it does not replace verifying).
+
+**The AI ↔ Sources client does not exist:** Sources never calls AI, and AI never fetches from
+Sources. The product is the only client of both, passing Sources items to AI with provenance.
+
+## Who calls it, what it calls
+
+Publication products call the API with a Network client-credentials token for audience
+`openvibe.sources`; every route has one capability (`sources.source.read`, item reads, and the
+merchant routes). A first-party service may act for a person with `X-OV-Subject`; the merchant API
+and pages accept the person's own session (Bearer on the API, the first-party cookie on the pages).
+SDK `openvibe-sdk/sources` is the client surface. There is **no public JSON API**: the vhost keeps
+`/api/v1/*` loopback-only; only `/merchant*` and `/shared/*` are public.
+
+Sources calls the outside hosts named by its sources (subject to robots, spacing and the address
+guard), OpenVibe.Network (JWKS, service token, account export/deletion, merchant identity) and
+OpenVibe.Events (outbox). It never fetches a URL a user chose.
+
+## Database and migrations
+
+PostgreSQL `ov_sources` (ADR-035) through `openvibe-sdk/db`: `DATABASE_URL` through PgBouncer,
+`DATABASE_DIRECT_URL` for migrations; `server/db.js` opens and migrates at boot under an advisory
+lock. `migrations/0001..0012` include crawl, merchant sources, account data, source state, item
+retention, config, telemetry and the outbox rejection index; never edit one that has run. A failed
+fetch writes a `fetch_runs` row and **never creates or modifies an item**; only the `ok` path touches
+items, and validators (ETag/Last-Modified) are stored only after a good parse.
+
+## Auth and exposure
+
+Network RS256 JWTs verified through the SDK JWKS client. Every `/api/v1` route needs a service token
+for audience `openvibe.sources` with that route's capability, then passes per-actor limits. The
+merchant API/pages are the person-facing exception: a person's session owns their claims, and a
+first-party service may act for a person only with `X-OV-Subject`. A source credential is read from
+the environment variable it names (`SOURCES_CRED_*` only), sent only to the configured origin
+(dropped on a cross-origin redirect), and never stored or logged.
+
+## Deploy
+
+`sudo ovhost deploy sources` (git-checkout: fetch, fast-forward `/opt/openvibe.sources`, install on a
+lockfile change, restart, wait for `/api/ready`; rollback `sudo ovhost rollback sources --to <sha>`).
+Unit `deploy/systemd/openvibe-sources.service`, env `/etc/openvibe/sources.env`, database `ov_sources`,
+nginx `deploy/nginx/sources.openvibe.network.conf` (API loopback-only, merchant + shared public).
+The scheduler runs in-process; it starts after migrations in `server/index.js`, never at module load.
+
+## Commands and tests
+
+```bash
+npm install
+npm run seed # create the sources in seeds/sources.json, disabled
+npm run dev # http://127.0.0.1:4720
+npm test # every test/*.test.js against local stub sites, no internet
+npm test -- crawl robots # only matching files
+npm run test:pg # the same through PgBouncer
+npm run probe -- <domain> # write evidence/<domain>.json (run from the host's egress)
+npm run adapters # evidence → adapters or compliance/review-queue.yaml
+npm run register-adapters # bring the registry to the adapters
+```
+
+`test/run.js` uses `openvibe-shared/test-runner`; tests boot local stub sites and PGlite and never
+touch the internet. Coverage includes adapters, extract, crawl, robots/rate limiting, the API
+(CSV/TSV/join/map), scheduler, merchant proof, account data/deletion and the service manifest.
+
+## Conventions and pitfalls
+
+- CommonJS, 4-space indent, single quotes, semicolons; problem+json errors, no raw rows.
+- **Provenance is the product:** every item keeps source, fetch time, content hash and parser
+ version; a failure stays a recorded failure and is never filled with invented content.
+- **ADR 0007 is Accepted and the fixed technique list is not up for discussion.** Read
+ `docs/adr/0007-source-access.md`; a probe verdict from `evidence/<domain>.json` decides access —
+ probe first, read existing evidence, do not re-derive or re-litigate it.
+- **Never enable a source without a terms note** (and a robots note when fetched); an adapter
+ existing is not permission. A source with items cannot be deleted, only disabled.
+- **Politeness is enforced:** robots.txt (RFC 9309, 4xx = none, 429/5xx = disallow), max request
+ spacing (`min_interval_ms`, `Crawl-delay`), conditional GET, one response deadline, byte cap,
+ public-address check at connect time, manual re-checked redirects, exponential backoff.
+- **Dedupe and classification split:** Sources dedupes within a source (identity + content hash) and
+ assigns a coarse category; cross-source dedupe and fine classification are the product's.
+- **A failed run does not touch items and does not store validators** — the next pass re-fetches in
+ full. Keep the `fetch_runs` state names exact.
@@ -1,8 +1 @@
@@ -1,8 +1 @@
-# OpenVibe.Sources: notes for agents
-
-Read README.md first. Source access:
-
-Access verdicts come from evidence/<domain>.json, written by scripts/probe.mjs.
-Probe first; if evidence exists, read it and do not re-derive it.
-docs/adr/0007-source-access.md is Accepted. Re-litigating it is a code comment.
-The fixed technique list in ADR 0007 is not up for discussion.