README.md

main 49.4 KB · 614 lines · markdown

1	# OpenVibe.Sources
2
3	> The source registry and the ingestion workers the publication products depend on: every item
4	> carries where it came from, when it was retrieved and on what terms, and a failed fetch is a
5	> recorded failure — never replaced by invented content.
6
7	Docs: https://openvibe.services · For AI: https://openvibe.services/llms.txt
8	Run it: `npm ci && npm run dev` (Node 22 or newer) · Tests: `npm test`
9
10	Status: alpha (roadmap Wave 14). Deployed internally: the API runs on the production host (127.0.0.1:4720 only,
11	since 2026-09-23). On 2026-10-22, 43 of its 47 sources were enabled and it held 55,683 items from 4,762 fetch runs
12	(the first on 2026-10-07; 431 in the last 24 hours), and none of the enabled sources was failing. The counts by category
13	are under [Acceptance](#acceptance-tests).
14	Domain: `sources.openvibe.network` (a short "internal service" page and health on this
15	repository's vhost; the API is loopback-only). Install steps: [Public host](#public-host).
16	Plan: OpenVibe End-to-End Realignment & Implementation Plan, revision 3 — roadmap §4.2 B, §15.12, §29, anti-goals 13, 26, 27.
17	License: AGPL-3.0.
18
19	## Purpose
20
21	News, Reviews, Deals, Coupons, Trade (and AI summarisation) all need outside material. Sources
22	owns the one place it enters the network: a registry of sources with their terms, and workers that
23	fetch them politely and record exactly what happened. Products read items with provenance and
24	decide — under their own review and indexability rules — what, if anything, to publish.
25
26	## Running it
27
28	```bash
29	npm install
30	cp .env.example .env
31	npm run seed # the 67 sources in seeds/sources.json, created disabled
32	npm run dev # http://127.0.0.1:4720
33	npm test # every test/*.test.js against local stub sites; no internet
34	```
35
36	Node 22 in production (`fnm exec --using=22.22.1 npm test`). Production: `/opt/openvibe.sources`,
37	env `/etc/openvibe/sources.env`, unit [deploy/systemd/openvibe-sources.service](deploy/systemd/openvibe-sources.service),
38	database `ov_sources` on the host's data role (ADR-035; `sudo /opt/openvibe.host/roles/data/add-service.sh sources`), nginx [deploy/nginx/sources.openvibe.network.conf](deploy/nginx/sources.openvibe.network.conf).
39
40	`GET /api/health` is liveness. `GET /api/ready` (openvibe-shared/ready) is 503 only when the
41	database fails; a Network key that has not loaded and a fetcher that is off, stopped or behind (a
42	source due for more than 15 minutes) degrade it. It reports source counts by health status, runs in
43	flight and the outbox backlog. `GET /metrics` (openvibe-shared/metrics) answers direct loopback
44	callers only: golden signals by route template, `sources_sources{status}`, `sources_items{state}`,
45	the fetch queue (`sources_fetch_due`, `sources_fetch_oldest_wait_seconds`,
46	`sources_fetch_in_flight`), `sources_last_fetch_timestamp_seconds` and
47	`sources_last_success_timestamp_seconds`.
48
49	Requests are aggregated by route template, method and status class, never by raw URL; health,
50	readiness, `/metrics`, `/shared` and static files are not counted. Once a minute one `http.request`
51	sample per key, plus the active-requests, p95 and event-loop-lag gauges, is written to this service's
52	own `telemetry_samples` table (openvibe-sdk 0.46 telemetry, `platform.telemetry-sample@1`, migration
53	`0003_telemetry.sql`); rows older than 14 days are deleted daily. `server/app.js` mounts the SDK's
54	no-op-until-init `telemetryMiddleware`; `server/index.js` calls `telemetry.init()` at production
55	start and flushes on stop.
56
57	## The registry
58
59	A source records: `key`, `name`, `type` (`rss\|atom\|sitemap\|jsonld\|api\|bulk\|crawl\|manual`), `category`
60	(`news\|blog\|reviews\|deals\|coupons\|trade\|hosting\|crowdfunding\|games\|housing\|food\|jobs\|code`, read from `sources.source@1`; housing, food and jobs since contracts 0.210.0, code since 0.411.1), `endpoints` (for `api`: a field mapping, no code per
61	provider; for `bulk`: one dataset file with a format, a byte cap and a row limit), `auth` (`none\|header\|bearer\|query` with the name of an environment variable —
62	names must start with `SOURCES_CRED_`, so an entry can never point at this service's own
63	secrets), `robots_note`, `terms_note`, `license_note`, the rate limit (`min_interval_ms`) — the GitHub read uses
64	`GITHUB_TOKEN` (or `SOURCES_GITHUB_TOKEN`, or the owner's D11 setting) by name when set, and is anonymous within
65	GitHub's 10-a-minute search limit otherwise —
66	`poll_interval_sec`, `stale_after_sec`, `max_items`, `enabled`, `review_required`,
67	`sensitivity`, `default_indexability`, `search_visibility` and `retire_after_sec` (since contracts 0.211.0).
68
69	A source cannot be enabled without a terms note (and a robots note if it is fetched): an adapter
70	existing is not permission to ingest (anti-goal 26). A source with items cannot be deleted —
71	disable it — so provenance stays resolvable. Contract: [`sources.source@1`](docs/contracts-proposal/), released in openvibe-contracts v0.12.0.
72
73	## Ingestion
74
75	An in-process scheduler starts runs for enabled sources whose poll is due (at most
76	`SOURCES_MAX_CONCURRENT` at once, never two of one source). A run fetches each endpoint in turn:
77
78	1. Credential read from the named variable at that moment (missing → `disabled`), sent only
79	to the configured origin (dropped on a cross-origin redirect), never stored or logged.
80	2. robots.txt (RFC 9309) for the origin: our `OpenVibeSources` group or `*`, longest match,
81	`*`/`$`; 4xx = no restrictions, 429/5xx/unreachable = complete disallow; cached 24 h;
82	`Crawl-delay` honoured. Every redirect hop is checked too.
83	3. Spacing: max(`SOURCES_HOST_MIN_INTERVAL_MS`, the source's `min_interval_ms`,
84	Crawl-delay) between requests to a host. A trigger before the source's interval has passed,
85	or before a `Retry-After`, records `rate_limited` and requests nothing.
86	4. Fetch: address guard (http(s) only, ports 80/443, every resolved address must be public —
87	checked at connect time, so DNS rebinding cannot slip through; loopback, RFC 1918, link-local,
88	CGNAT, metadata and v4-mapped/NAT64/6to4 forms are refused), manual redirects (≤ 5, each
89	re-checked), conditional GET (`If-None-Match`/`If-Modified-Since`), one deadline for the whole
90	response (`SOURCES_FETCH_TIMEOUT_MS`), a byte cap on the decoded body (`SOURCES_MAX_BYTES`).
91	5. Parse with the adapter (a `bulk` dataset is parsed chunk by chunk and the stream is stopped at its row limit; a resolver walks its JSON documents); on success, in one transaction: the run row, the items (created /
92	new revision if the content hash changed / retrieval time refreshed if unchanged), the
93	validators, and the events.
94
95	Every attempt writes a `fetch_runs` row with an explicit state: `ok`, `not_modified`,
96	`http_error`, `timeout`, `robots_denied`, `parse_error`, `rate_limited` or `disabled`, plus the
97	HTTP status, an error code, the raw-body hash and item counts. **A failed run never creates or
98	modifies an item** (only the `ok` path touches items), and ETag/Last-Modified are stored only after
99	a good parse, so an unreadable body is re-fetched in full next time. Failing sources back off
100	exponentially (capped at `SOURCES_MAX_BACKOFF_MS`).
101
102	Staleness is computed: a fetched source is stale when its last success (`ok` or
103	`not_modified`) is older than `stale_after_sec` (default 3 × the poll interval) or it never
104	succeeded. Health status: `healthy \| stale \| failing \| never_fetched \| disabled \| manual`.
105
106	### Adapters (parser versions recorded on every item)
107
108	\| Type \| Reads \| Notes \|
109	\|---\|---\|---\|
110	\| `rss`, `atom` (`feed@3`) \| RSS 2.0, RSS 1.0/RDF, Atom 1.0 \| guid/id identity, else link; text-only summary capped at 1000 chars; a deal feed's own offer elements (DealNews' `dealnews:price` with its currency, `dealnews:retailer`, `dealnews:expires`) become `fields.price`, `currency`, `seller` and `valid_until` exactly as stated (the item stays an article; nothing is read out of the headline); its `dealnews:category` fills `fields.categories` when the item has no `<category>`, and `dealnews:staffPick` is `fields.staff_pick` (true/false, absent when not stated) \|
111	\| `sitemap` (`sitemap@1`) \| `<urlset>` (+ Google News extension), `<sitemapindex>` \| index children are listed, not followed \|
112	\| `jsonld` (`jsonld@2`) \| schema.org Product, Offer/AggregateOffer, Review, NewsArticle, BlogPosting (+ Article) \| prices, ratings and availability exactly as the page states; missing = null; embedded review text not kept; a page without JSON-LD is a `parse_error`; `list_price` from a ListPrice/StrikethroughPrice/MSRP/SRP `priceSpecification`, as stated \|
113	\| `api` (`api@1`) \| JSON, XML, TSV or CSV official files via a registry mapping \| `items_path`, `items_shape` (`array` default, `map` for records keyed by id), `fields`, `extra`, `where`/`filter`; epoch fields only where mapped as such; `resolver: "hn-whoishiring"` walks the Hacker News thread's JSON documents instead; `resolver: "github"` walks GitHub's repository search plan (server/github.js; trending is the last 24 hours because GitHub's index has no same-day bucket) \|
114	\| `bulk` (`bulk@1`) \| one dataset file, streamed (CSV, JSONL or TSV) \| `max_bytes` is the size cap, `max_rows` the per-run row limit; the stream stops at the limit, so a multi-GB file is never held in memory. `resolver: "off-delta"` reads a text index and only the gzipped JSONL delta parts it lists, keeping the last finished part and the current part's row offset in `source_state` (Open Food Facts) \|
115	\| `crawl` (`jsonld@2`, `page@1`) \| a whole site from its seeds: its sitemaps and, if asked, its links (see below) \| only pages that publish JSON-LD are kept, unless `require_structured: false` keeps a `page` item from the page's own search metadata \|
116	\| `manual` (`manual@1`) \| items entered through the API with an evidence URL \| no fetcher \|
117
118	XML with DTD entity declarations is refused; the parser expands no entities (values are decoded
119	per field). Fixtures and expectations: `test/adapters.test.js`.
120
121	Some official price lists key their records by id instead of listing them — Porkbun's TLD pricing
122	is `{ "pricing": { "com": {…}, "net": {…} } }`. `items_shape: "map"` reads that object: each own
123	property becomes one record in sorted key order, and the property's key is available as the
124	virtual path `$key` (so `fields.identity: "$key"` makes the id the identity). A property whose
125	value is not an object is skipped, and an `items_path` of the wrong shape is a parse error, never
126	an empty success. With the default `items_shape: "array"`, `$index` is a record's 0-based position in
127	the array (as a whole path, e.g. `"extra": { "position": "$index" }`), so a ranked list keeps its
128	order (Steam's top sellers).
129
130	A CSV or TSV body likewise names its columns on the first line: each further line is a record of
131	trimmed text, and `items_path` must be "". CSV follows RFC 4180 quoting (a quoted field may
132	hold commas, doubled quotes and line breaks); a row whose value count differs from the header is
133	skipped, and a body that ends inside a quoted field is a parse error. When no single column is unique,
134	`identity_fields` names two to eight paths whose values, joined with " \| ", are the record's identity
135	(the USDA ERS table prices Apples in four forms); `fields.identity` cannot be given with it.
136
137	```json
138	{ "url": "https://api.porkbun.com/api/json/v3/pricing/get", "format": "json",
139	"items_path": "pricing", "items_shape": "map", "item_kind": "tld",
140	"fields": { "identity": "$key" },
141	"extra": { "tld": "$key", "registration": "registration", "renewal": "renewal" } }
142	```
143
144	A path segment `*` steps into the single child of an object (a path that meets an array, or an
145	object with zero or more than one own property, resolves to null). A path starting with `$root.`
146	resolves from the parsed response body rather than the record, and inside it `{$key}` stands for
147	the record's map key — so a keyed price list can be joined to prices held elsewhere in the same
148	response: `"price_hourly": "$root.terms.OnDemand.{$key}..priceDimensions..pricePerUnit.USD"`.
149	An endpoint may also carry `where`, up to 10 `path -> expected` conditions (expected is a string,
150	number or boolean, or an array of up to 20 of them, matching any): a record that fails any
151	condition is left out before `maxItems` is counted and reported as `meta.filtered`, e.g.
152	`"where": { "productFamily": "Lightsail Instance", "attributes.operatingSystem": "Linux" }`.
153	An expectation may instead be `{ "pattern": "<regular expression, at most 200 characters>" }`, which the value must
154	match as text.
155
156	Catalogs that keep plans, hardware and prices apart (OVHcloud's public order catalog, `ovhcloud-vps`). A key may
157	carry a selector, `key[field=value,…]` (one to four conditions): `key` must be an array, and the path steps into its
158	first element whose fields equal every value, compared as text after `scalar()` (so `interval=1` matches the number 1).
159	A value is literal text, `$rec.<path>` (read from the record) or `$root.<path>` (read from the response), and either
160	path may carry its own selectors. That reaches a plan's default addon and then that addon's own product without code
161	per provider; no match is null.
162
163	```json
164	"where": { "planCode": { "pattern": "^vps-[0-9]{4}-model[0-9]+$" } },
165	"extra": { "vcpu": "$root.products[name=$rec.product].blobs.technical.cpu.cores",
166	"memory_gb": "$root.products[name=$root.addons[planCode=$rec.addonFamilies[name=memory].default].product].blobs.technical.memory.size",
167	"price_monthly": "pricings[mode=default,commitment=0,intervalUnit=month,interval=1].price" }
168	```
169
170	`max_bytes` (an integer from 1 KiB to 64 MiB) sets one endpoint's body cap instead of `SOURCES_MAX_BYTES`. OVHcloud's
171	VPS catalog is about 12 MB, its bare-metal catalog about 32 MB and its Eco catalog about 10 MB.
172
173	`timeout_ms` (an integer from 1 s to 120 s) sets one endpoint's deadline instead of `SOURCES_FETCH_TIMEOUT_MS`. NYC Open
174	Data computes the HPD per-building violation tally (`nyc-hpd-open-violations`) in 12 to 60 seconds.
175
176	`request_content_type` (`application/json` only) is sent as the Content-Type of the endpoint's GET, which has no body.
177	Open Collective's public GraphQL API answers a GET only with it (its CSRF guard): `opencollective-accounts` (category
178	`crowdfunding`) reads the 100 most active and the 100 newest collectives, projects and funds every six hours, each a
179	`campaign` item with the amounts and counts as Open Collective states them, for OpenVibe.Deals' crowdfunding discovery.
180
181	Facts stated in prose (`extract`, rss/atom only). A feed endpoint may carry
182	`extract: { kind, require, fields }` ([server/adapters/extract.js](server/adapters/extract.js)). Each field is read from
183	the item's own `title` or `summary` by a pattern with one capturing group. With `as: "date"`, "October 20" becomes
184	2026-10-20: without a year, the first such day on or after the item's publication. An item that yields every `require`d
185	field is stored as `kind` with those fields; any other item is skipped and counted in the run's `items_skipped` (the
186	plain feed source keeps it). The registry refuses a pattern that does not compile or has no group, an unknown option,
187	and `extract` on an api source. `dealnews-coupon-codes` (category `coupons`) is the first: the code, the shop and the
188	stated end date of the DealNews deals that state a code, exactly as written.
189
190	### Crawl sources (`server/crawl.js`, contracts 0.224.0)
191
192	The owner, 2026-10-28: "aggregate all websites and select deals on all sites… a full comprehensive crawler system… the
193	open source Google". A crawl source walks one site from its endpoints (the seeds). Its `crawl` settings:
194
195	\| Field \| Default \| What it does \|
196	\|---\|---\|---\|
197	\| `sitemaps` \| `true` \| read the site's sitemaps: robots.txt `Sitemap` lines, else `/sitemap.xml`; indexes and `.xml.gz` followed \|
198	\| `follow_links` \| `false` \| also follow same-host links (not `rel=nofollow`, not from a `nofollow` page) \|
199	\| `max_depth` \| `2` \| with `follow_links`, how many links from a seed or sitemap page \|
200	\| `include`, `exclude` \| none \| regular expressions over the path and query (seeds are fetched whatever they say) \|
201	\| `max_pages` \| `5000` \| the frontier's size for the site \|
202	\| `pages_per_run` \| `100` \| requests in one run (sitemaps count) \|
203	\| `recrawl_sec` \| `86400` \| when a page that changed is due again; an unchanged page waits up to 4× longer \|
204	\| `require_structured` \| `true` \| keep only pages with schema.org JSON-LD \|
205
206	The frontier (`crawl_frontier`, migration 0006) holds every page and sitemap the crawl knows, with its state:
207
208	- Page states: `new`, `kept`, `hub`, `unstructured`, `noindex`, `duplicate`, `not_html`, `too_large`, `excluded`,
209	`denied`, `missing`, `gone` and `failing`.
210	- What it records per page: when the page is due, its validators, its body hash, and the fetch run its items were
211	last read in.
212	- Run order: each run reads the due sitemaps first, then the due pages, oldest first.
213
214	Every request is a source request like any other:
215	- robots.txt on the URL and on every redirect hop;
216	- per-host spacing with Crawl-delay and the source's `min_interval_ms`;
217	- conditional GETs;
218	- the address guard;
219	- one `fetch_runs` row per page.
220
221	Scope: a crawl never leaves its seeds' hosts (or their `www` twins).
222
223	Pages that ask not to be kept:
224	- Canonical: a page whose canonical is another page of the crawl is read there instead.
225	- noindex: a page that says noindex (`<meta name="robots">`, `<meta name="openvibesources">`, `X-Robots-Tag`) is
226	not kept, and its items are retired.
227	- Gone: a page that answers 404 twice, or 410 once, is gone and its items are retired as `gone_from_source`.
228
229	A page's failure is the page's: the source fails (backoff, one `sources.fetch.failed`) only when the site does:
230	- robots.txt unreachable;
231	- a 429 or a 503 with Retry-After;
232	- five requests in a row with no usable answer.
233
234	Scheduling:
235	- at most `SOURCES_MAX_CONCURRENT_CRAWLS` (2) crawls run at once;
236	- a run lasts at most `SOURCES_CRAWL_RUN_MS` (15 min);
237	- while pages are still due, the next run comes after `SOURCES_CRAWL_SOON_MS` (60 s), not `poll_interval_sec`.
238
239	Settings that don't apply: a crawl source has no `retire_after_sec`, because a page leaves with its items as
240	described above.
241
242	API:
243	- `GET /api/v1/sources/:key/crawl` shows the frontier: pages by state, sitemaps, how many are due, and the latest
244	fetches.
245	- `POST /api/v1/sources/:key/fetch` starts a crawl and answers 202.
246
247	Tests: `test/crawl.test.js`.
248
249	### Items
250
251	Each item: `source_key`, `category`, `kind`, `identity`, `canonical_url` (tracking parameters and
252	fragments removed), `title`, `summary`, `authors`, `published_at`, `source_updated_at`, `fields`,
253	`revision`, and `provenance` (`retrieved_at`, `first_seen_at`, `content_hash`, `raw_body_hash`,
254	`parser_version`, `fetch_run_id`, `license_note`, `terms_note`, `entered_by`). Earlier revisions
255	are kept in `item_revisions`. Removal (takedown, licence) is explicit, needs a reason, and is
256	sticky: a later fetch that still lists the item does not bring it back.
257
258	A source of jobs or listings sets `retire_after_sec` (an hour to a year; null, the default, keeps every item, as news
259	and datasets should). After a run in which every endpoint answered (`ok` or `not_modified`) and none failed, an item
260	the feed has not shown for that long is removed with reason `gone_from_source`. That is a change like any removal, so a
261	consumer following `GET /api/v1/items?include_removed=1` learns that the job is gone. A 304 counts as seeing again what
262	the endpoint's last good parse saw. At most 500 items are retired a run; the rest go on the next. Contract (released in openvibe-contracts v0.12.0):
263	[`sources.item@1`](docs/contracts-proposal/).
264
265	Retention. Items do not grow forever: `server/retention.js` declares, in one place, a window per category (news 30
266	days, blog 90, reviews 180, deals 14, coupons 90, trade 365, hosting 30, crowdfunding 90, games 14, food 365, jobs 14, code 30)
267	and one `keepLatest` rule for housing, where HUD's yearly Fair Market Rents must keep the latest value per source key.
268	An item's age is `retrieved_at` while it is live and `removed_at` once removed, so a consumer following the change feed
269	still sees a removal for the full window; `first_seen_at` is never the age, since a source that keeps listing a job or
270	a price keeps it current. The daily job (`server/jobs/retention.js`, started with the worker) deletes at most 2,000 rows
271	a statement, 50 batches a category a run, and its log line carries counts only. The rules are in code because Sources
272	has no `openvibe-shared/config` namespace, and each one is the `sources.category@1` document for its category:
273	`test/retention.test.js` validates every `{ key, retention }` against the contract (openvibe-contracts v0.295.0) and
274	holds the strict one-to-one check between `sources.source@1`'s category list, `sources.category@1`'s keys and the rule
275	table, so a pin bump fails CI and names the missing rule. At runtime a category the contract has but the rules do not is
276	kept (never pruned) and warned about once a run, and a rule for a category the contract lacks is ignored with a warning,
277	so a Contracts pin bump cannot stop Sources at boot.
278
279	## API (service tokens, one capability per route)
280
281	Callers use an OpenVibe.Network client-credentials token for audience `openvibe.sources`. The merchant routes also
282	accept a signed-in person's session (see [Merchant opt-in](#merchant-opt-in-list-your-store-on-openvibe)); a first-party
283	service calls them with its service token and `X-OV-Subject: usr_…`.
284
285	\| Capability \| Routes \|
286	\|---\|---\|
287	\| `sources.source.read` \| `GET /api/v1/sources`, `/api/v1/sources/:key`, `/api/v1/sources/:key/runs?before=`, `/api/v1/runs?state=failed&after=`, `/api/v1/health`; `GET /api/v1/merchant/sources[/:id]` \|
288	\| `sources.item.read` \| `GET /api/v1/items?source=&category=&after=<change_seq>&include_removed=1`, `GET /api/v1/items/:id?revisions=1`; `GET /api/v1/code/repos?list=trending\\|week\\|month\\|hot\\|new\\|top&lang=&topic=&limit=`, `GET /api/v1/code/search?q=&lang=&sort=stars\\|updated\\|best&page=` \|
289	\| `sources.source.manage` \| `POST /api/v1/sources`, `PATCH`/`DELETE /api/v1/sources/:key`, `POST /api/v1/sources/:key/fetch`, `POST /api/v1/sources/:key/items` (manual), `DELETE /api/v1/items/:id` `{reason}`; `POST /api/v1/merchant/sources`, `/:id/verify`, `/:id/withdraw` \|
290
291	`GET /api/v1/items` pages in change order (creations, revisions and — with `include_removed=1` —
292	removals); resume from `next_after`. Every page carries the status and staleness of the sources
293	it contains. The capability ids (first proposed in [docs/capabilities-proposal/](docs/capabilities-proposal/))
294	are released in `openvibe-contracts` v0.12.0 (this repo pins v0.96.0); [server/auth.js](server/auth.js)
295	decides them with the contracts grant rule.
296
297	### Per-actor limits
298
299	Every API route also limits the principal that passed its capability guard, before any work:
300	`server/api/actor-limits.js`, openvibe-sdk/limits, roadmap WS-R task 4. Past a limit: `429`
301	problem+json `rate_limited` with `Retry-After`, one `[limits]` log line and
302	`sources_rate_limited_total{limit,window}`.
303
304	\| Routes \| Per principal, a minute / an hour \|
305	\|---\|---\|
306	\| Reads (`sources.source.read`, `sources.item.read`) by an app or module (`app:…`, `mod:…`) \| `SOURCES_LIMITS_MINUTE` / `SOURCES_LIMITS_HOUR` (120 / 3000) \|
307	\| `POST`, `PATCH`, `DELETE /api/v1/sources[/:key]` \| 30 / 300 \|
308	\| `POST /api/v1/sources/:key/fetch` \| 10 / 100 \|
309	\| `POST /api/v1/sources/:key/items` (manual) \| 60 / 1200 \|
310	\| `GET /api/v1/merchant/sources[/:id]` (a merchant or a service acting for one) \| 120 / 3000 \|
311	\| `POST /api/v1/merchant/sources`, `/:id/verify`, `/:id/withdraw` \| 30 / 300 \|
312	\| `DELETE /api/v1/items/:id` (a takedown may cover many items) \| 120 / 3000 \|
313
314	A first-party service (`svc:…`) is not counted on reads: News, Reviews and Trade pull cursor pages
315	for all their readers, and Wiki asks about every item its editors cite. Sources' own workers (the
316	scheduler, fetches, the outbox relay) run in-process and never pass through HTTP. Never limited:
317	`/api/health`, `/api/ready`, `/release.json`, `/metrics` and the home page. `test/actor-limits.test.js`.
318
319	## Merchant opt-in ("List your store on OpenVibe")
320
321	The first permitted channel in plan §4.4. A signed-in merchant claims a domain, proves control of it, and names the
322	sitemap or product feed Sources may read. Nothing is created in `sources` before the proof is found: a pending claim
323	is only a `merchant_sources` row (migration `0007_merchant_sources.sql`). Verification creates one enabled source in the
324	same transaction, and withdrawal disables it while keeping the note.
325
326	- Proof. One of two challenges, the OpenVibe.Space pattern:
327	- DNS: `_openvibe-sources.<domain> TXT "openvibe-sources-verification=<token>"`.
328	- Meta: `<meta name="openvibe-sources-verification" content="<token>">` on `https://<domain>/` (or the `meta_url` the
329	merchant names, on the same site). Redirects off the domain are refused.
330	- Fetched politely. A sitemap becomes a `crawl` source seeded with that sitemap (no guessed sitemaps); a product feed
331	becomes an `rss`/`atom` source. Either way the ordinary fetcher runs: robots.txt on every request and redirect hop,
332	per-host spacing, SSRF guard, conditional GETs. Explicit permission is not a robots override.
333	- The terms note records the merchant subject (and display name when the token carries one), the UTC verification
334	time, the domain, the named sitemap/feed, and how control was verified ("DNS TXT record at _openvibe-sources.…" or
335	"meta tag at …"); it also says the merchant may withdraw and that robots.txt is still obeyed.
336	- Withdrawal sets the claim to `withdrawn`, disables the source, and leaves `terms_note` and the claim row intact.
337	The domain is free to claim again (a new claim, a new source key). Unlike a served custom domain (Space's daily
338	recheck), the permission persists until the merchant withdraws: removing the TXT/meta record later does not silently
339	disable the source.
340	- Pending proofs are rechecked by a slow in-process pass (`SOURCES_MERCHANT_RECHECK_MS`, default 10 min; `0` turns
341	it off) and fail after `SOURCES_MERCHANT_PENDING_DAYS` (14). A merchant can also press "Check now".
342	- Pages. `GET /merchant` (guest explanation, or the signed-in merchant's list and add form),
343	`GET /merchant/sources/:id` (challenge, status, terms note, last crawl), the POST actions
344	`/merchant/sources`, `/:id/verify`, `/:id/withdraw`, and `/merchant/login\|callback\|logout`. Forms carry an HMAC CSRF
345	token; the session is the Sources `ov_token` cookie, verified offline with `verifyUserToken`.
346	- JSON read for Report. `GET /api/v1/merchant/sources` (alias `/claims`) returns each claim with its source, health,
347	last run, recent runs and — for a crawl — `last_crawl` (`pages`, `due`, `next_due_at`, state counts, recent pages).
348	Report uses its service token and `X-OV-Subject: usr_…`; a merchant's own session needs no extra header. The read is
349	the contract for the Report dashboard (which is not built here).
350	- Identity. The OAuth client is `OV_OAUTH_CLIENT_ID` (default `sources`) with
351	`OV_OAUTH_REDIRECT_URI` defaulting to `<BASE_URL>/merchant/callback`; Network must register that exact redirect. A
352	first-party service token may act for a merchant only with `X-OV-Subject` (an app/module token cannot). The merchant
353	routes reuse the released `sources.source.read` / `sources.source.manage` capabilities; no new contract id is needed.
354	- Config. `SOURCES_SESSION_SECRET` (defaults to `OV_OAUTH_CLIENT_SECRET`), `SOURCES_COOKIE_SECURE`,
355	`SOURCES_MERCHANT_PENDING_DAYS`, `SOURCES_MERCHANT_RECHECK_MS`, `SOURCES_MERCHANT_META_MAX_BYTES`,
356	`SOURCES_MERCHANT_DNS_SERVERS`, and the source-shape knobs `SOURCES_MERCHANT_*` documented in
357	[.env.example](.env.example). Tests: `test/merchant-optin.test.js`, `test/merchant-pages.test.js`.
358
359	## Events (transactional outbox → OpenVibe.Events when `EVENTS_URL` is set)
360
361	- `sources.item.created`, `sources.item.updated` (with the previous content hash), `sources.item.removed` (with the reason)
362	- `sources.fetch.failed` for `http_error`, `timeout`, `robots_denied`, `parse_error` and upstream 429 (with the state, error code, HTTP status and consecutive failures). Our own rate limiter and disabled sources are not failures.
363	- `sources.index_document.upserted\|deleted` for sources with `search_visibility: "members"`: raw items indexed in OpenVibe.Search for staff only (`acl.groups: role:admin, role:global_mod`), always `noindex` (`third_party_content`).
364
365	All with `visibility: "internal"`. Envelopes validate as `events.event-envelope@1`.
366	The openvibe-sdk service outbox stores pending events in `service_outbox` and relays them with the
367	Sources service token. The expand migration copies pending legacy rows; `event_outbox` remains
368	for the N-1 rollback window.
369
370	## Seeds
371
372	[seeds/sources.json](seeds/sources.json) holds the 69 registered sources, each with its robots, terms and licence notes (the production rows are the record; the terms notes match them as of the 2026-10-23 review).
373	`npm run seed` creates the ones that do not exist yet, all disabled, and never changes an existing one; a person
374	enables a source after reading its terms (`PATCH /api/v1/sources/:key {"enabled": true}`), and the fetcher re-checks
375	robots.txt on every run. Production on 2026-10-22 (the food row checked 2026-11-06):
376
377	\| Category \| Sources (type) \| Enabled in production \|
378	\|---\|---\|---\|
379	\| news \| `nasa-news-releases`, `eu-press-corner`, `fda-food-recalls`, `noaa-news` (rss); `govuk-news` (atom) \| 4 of 5: `noaa-news` is off (NOAA's robots.txt answers 403 to our user agent, so its rules cannot be read) \|
380	\| blog \| `nodejs-blog` (rss) \| 1 of 1 \|
381	\| reviews \| `steam-reviews-<game>` for 19 games (api: Steam `appreviews` signals only, review text not mapped); `cfpb-complaints-by-company`, `nyc-restaurant-inspections`, `nyc-hpd-open-violations` (api: official records) \| 22 of 22 \|
382	\| deals \| `dealnews-daily` (rss, `feed@3`: the offer's price, retailer, end date, category and editors' pick) \| 1 of 1 \|
383	\| coupons \| `dealnews-coupon-codes` (rss with `extract`: only items that state a code and the shop, as kind `coupon`); `staff-coupon-codes` (manual, with the evidence URL) \| 2 of 2 \|
384	\| trade \| `sec-xbrl-filings`, `ecb-euro-reference-rates` (rss); `sec-company-tickers`, `frankfurter-eur-rates` (api) \| 4 of 4 \|
385	\| hosting \| `porkbun-tld-pricing`, `ovhcloud-vps`, `ovhcloud-baremetal`, `linode-types`, `aws-lightsail-bundles`, `vultr-plans` (api); `lowendspirit-offers`, `lowendtalk-offers`, `lowendbox-posts` (rss) \| 5 of 9: `vultr-plans` is off (its API host's robots.txt disallows every path), LowEndTalk's and LowEndBox's feeds answer 403 to the production host, and `ovhcloud-baremetal` is seeded disabled for review \|
386	\| crowdfunding \| `opencollective-accounts` (api) \| 1 of 1 \|
387	\| games \| `steam-top-sellers`, `steam-new-releases`, `steam-specials` (api: the store's featured lists, in their order) \| 3 of 3 \|
388	\| food \| `open-food-facts`, `openfda-food-enforcement`, `bls-average-prices`, `usda-ers-fruit-vegetable-prices` (api); `themealdb` \| 4 of 5: `themealdb` is off (it needs a key, which is the owner's call); `usda-ers-fruit-vegetable-prices` was enabled on 2026-11-06 after its terms review, and its 155 prices are recorded in `test/fixtures/` \|
389	\| code \| `github` (api, `resolver: "github"`: REST search only, star history per day) \| seeded disabled: enable after the terms review \|
390
391	Candidates rejected on 2026-09-22: CheapShark's API (`/api/1.0/` is disallowed in its
392	robots.txt) and EDGAR's `getcurrent` Atom feed (`/cgi-bin` disallowed). The seeded feeds were
393	downloaded once by hand and parsed offline with these adapters (NASA 10 items, Node.js 50,
394	Steam 20, DealNews 50, SEC 200); that check is not part of the test suite and proves nothing about
395	the terms.
396
397	## Source access (ADR 0007)
398
399	Whether a site may be read, and how, is computed from evidence, never judged:
400	[docs/adr/0007-source-access.md](docs/adr/0007-source-access.md). `node scripts/probe.mjs <domain>` (run from the
401	production host, whose egress is the crawler's) writes `evidence/<domain>.json`: robots.txt, sitemap.xml,
402	/products.json, llms.txt, /api/ and the home page, with statuses and challenge markers, and a verdict.
403	`node scripts/adapters-from-evidence.mjs` turns evidence into adapters (`ok` → an enabled crawl source,
404	`structured-only` → a disabled one, Lane 1) or a row in [compliance/review-queue.yaml](compliance/review-queue.yaml)
405	(`queued`); `node scripts/register-adapters.js` brings the registry to the adapters. The crawler identifies itself as
406	`Mozilla/5.0 (compatible; OpenVibeBot/1.0; +https://openvibe.network/bot)` and obeys robots.txt groups, robots meta
407	and X-Robots-Tag for either token, `OpenVibeBot` or `OpenVibeSources`. The crawl type itself:
408	[docs/adr/0006-crawl-source-type.md](docs/adr/0006-crawl-source-type.md).
409
410	Probed 2026-10-28 (`c` = challenge page):
411
412	\| Domain \| Verdict \| Statuses \| Lane \| Adapter \|
413	\|---\|---\|---\|---\|---\|
414	\| adorama.com \| structured-only \| `/robots.txt:200 /sitemap.xml:404c /products.json:403c /llms.txt:403c /api/:403c /:403c` \| Lane 1 (feeds, affiliate, API, opt-in); adapter disabled \| [adapters/adorama.com.json](adapters/adorama.com.json) \|
415	\| allbirds.com \| ok \| `/robots.txt:200 /sitemap.xml:200 /products.json:200 /llms.txt:200 /api/:404 /:200` \| crawl adapter, enabled \| [adapters/allbirds.com.json](adapters/allbirds.com.json) \|
416	\| anker.com \| ok \| `/robots.txt:200 /sitemap.xml:200 /products.json:404 /llms.txt:200 /api/:404 /:200` \| crawl adapter, enabled \| [adapters/anker.com.json](adapters/anker.com.json) \|
417	\| backcountry.com \| queued \| `/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403` \| review queue; probed again after 24 h \| [compliance/review-queue.yaml](compliance/review-queue.yaml) \|
418	\| bestbuy.com \| ok \| `/robots.txt:200 /sitemap.xml:200 /products.json:404 /llms.txt:404 /api/:404 /:200` \| crawl adapter, enabled \| [adapters/bestbuy.com.json](adapters/bestbuy.com.json) \|
419	\| bhphotovideo.com \| structured-only \| `/robots.txt:200 /sitemap.xml:403c /products.json:403c /llms.txt:403c /api/:403c /:403c` \| Lane 1 (feeds, affiliate, API, opt-in); adapter disabled \| [adapters/bhphotovideo.com.json](adapters/bhphotovideo.com.json) \|
420	\| brooksrunning.com \| queued \| `/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403` \| review queue; probed again after 24 h \| [compliance/review-queue.yaml](compliance/review-queue.yaml) \|
421	\| converse.com \| queued \| `/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403` \| review queue; probed again after 24 h \| [compliance/review-queue.yaml](compliance/review-queue.yaml) \|
422	\| dsw.com \| ok \| `/robots.txt:200 /sitemap.xml:404 /products.json:404 /llms.txt:404 /api/:200 /:200` \| crawl adapter, enabled \| [adapters/dsw.com.json](adapters/dsw.com.json) \|
423	\| footlocker.com \| ok \| `/robots.txt:200 /sitemap.xml:200 /products.json:200 /llms.txt:200 /api/:403 /:200` \| crawl adapter, enabled \| [adapters/footlocker.com.json](adapters/footlocker.com.json) \|
424	\| frame.work \| queued \| `/robots.txt:403c /sitemap.xml:403c /products.json:403c /llms.txt:403c /api/:403c /:403c` \| review queue; probed again after 24 h \| [compliance/review-queue.yaml](compliance/review-queue.yaml) \|
425	\| gog.com \| ok \| `/robots.txt:200 /sitemap.xml:404 /products.json:404 /llms.txt:404 /api/:404 /:200` \| crawl adapter, enabled \| [adapters/gog.com.json](adapters/gog.com.json) \|
426	\| hoka.com \| structured-only \| `/robots.txt:200 /sitemap.xml:404 /products.json:410 /llms.txt:410 /api/:500 /:0` \| Lane 1 (feeds, affiliate, API, opt-in); adapter disabled \| [adapters/hoka.com.json](adapters/hoka.com.json) \|
427	\| homedepot.com \| ok \| `/robots.txt:200 /sitemap.xml:404 /products.json:404 /llms.txt:404 /api/:404 /:200` \| crawl adapter, enabled \| [adapters/homedepot.com.json](adapters/homedepot.com.json) \|
428	\| humblebundle.com \| ok \| `/robots.txt:200 /sitemap.xml:200 /products.json:404 /llms.txt:404 /api/:404 /:200` \| crawl adapter, enabled \| [adapters/humblebundle.com.json](adapters/humblebundle.com.json) \|
429	\| ikea.com \| queued \| `/robots.txt:403c /sitemap.xml:403c /products.json:403c /llms.txt:403c /api/:403c /:403c` \| review queue; probed again after 24 h \| [compliance/review-queue.yaml](compliance/review-queue.yaml) \|
430	\| microcenter.com \| queued \| `/robots.txt:403c /sitemap.xml:403c /products.json:403c /llms.txt:403c /api/:403c /:403c` \| review queue; probed again after 24 h \| [compliance/review-queue.yaml](compliance/review-queue.yaml) \|
431	\| monoprice.com \| structured-only \| `/robots.txt:200 /sitemap.xml:403c /products.json:403c /llms.txt:404 /api/:403c /:403c` \| Lane 1 (feeds, affiliate, API, opt-in); adapter disabled \| [adapters/monoprice.com.json](adapters/monoprice.com.json) \|
432	\| newbalance.com \| queued \| `/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403` \| review queue; probed again after 24 h \| [compliance/review-queue.yaml](compliance/review-queue.yaml) \|
433	\| newegg.com \| queued \| `/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403` \| review queue; probed again after 24 h \| [compliance/review-queue.yaml](compliance/review-queue.yaml) \|
434	\| on.com \| ok \| `/robots.txt:200 /sitemap.xml:200 /products.json:200 /llms.txt:200 /api/:403 /:200` \| crawl adapter, enabled \| [adapters/on.com.json](adapters/on.com.json) \|
435	\| patagonia.com \| ok \| `/robots.txt:200 /sitemap.xml:200 /products.json:200 /llms.txt:200 /api/:200 /:200` \| crawl adapter, enabled \| [adapters/patagonia.com.json](adapters/patagonia.com.json) \|
436	\| rei.com \| queued \| `/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403` \| review queue; probed again after 24 h \| [compliance/review-queue.yaml](compliance/review-queue.yaml) \|
437	\| runningwarehouse.com \| queued \| `/robots.txt:0 /sitemap.xml:0 /products.json:0 /llms.txt:0 /api/:0 /:0` \| review queue; probed again after 24 h \| [compliance/review-queue.yaml](compliance/review-queue.yaml) \|
438	\| saucony.com \| structured-only \| `/robots.txt:200 /sitemap.xml:404 /products.json:404c /llms.txt:404c /api/:404c /:200c` \| Lane 1 (feeds, affiliate, API, opt-in); adapter disabled \| [adapters/saucony.com.json](adapters/saucony.com.json) \|
439	\| vans.com \| queued \| `/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403` \| review queue; probed again after 24 h \| [compliance/review-queue.yaml](compliance/review-queue.yaml) \|
440	\| wayfair.com \| structured-only \| `/robots.txt:200 /sitemap.xml:404 /products.json:404 /llms.txt:429 /api/:429 /:429` \| Lane 1 (feeds, affiliate, API, opt-in); adapter disabled \| [adapters/wayfair.com.json](adapters/wayfair.com.json) \|
441	\| woot.com \| ok \| `/robots.txt:200 /sitemap.xml:200 /products.json:404 /llms.txt:404 /api/:404 /:200` \| crawl adapter, enabled \| [adapters/woot.com.json](adapters/woot.com.json) \|
442	\| zappos.com \| ok \| `/robots.txt:200 /sitemap.xml:404 /products.json:404 /llms.txt:404 /api/:200 /:200` \| crawl adapter, enabled \| [adapters/zappos.com.json](adapters/zappos.com.json) \|
443
444	## Processing jobs: who does what (roadmap WS-O task 7, decided 2026-09-26)
445
446	\| Job \| Where \| How \|
447	\|---\|---\|---\|
448	\| Dedupe within a source \| Sources \| an item's `identity` (feed guid/id, else canonical URL) and `content_hash`: a refetch updates the item and keeps a revision, never a second item \|
449	\| Dedupe across sources (one story from several outlets) \| the product \| News groups items into a story before `news.summarize_story`; Deals and Coupons match on merchant and code. Sources does not decide two outlets' items are "the same". \|
450	\| Classify \| Sources, coarsely; the product, finely \| every source has one `category` (news, reviews, deals, coupons, trade…) and its items inherit it. Topic, entity or instrument classification is the product's, with its own review. \|
451	\| Summarise \| the product, through OpenVibe.AI \| products call AI's seeded workflows (`news.summarize_story`, `reviews.summarize_entity`, `deals.enrich_deal`, `coupons.extract_coupon`, `trade.summarize_market_context`), passing Sources items as numbered `sources` with their provenance. The result is a draft/evidence package with citations and gaps, never published by AI or Sources. \|
452	\| Quality check \| the product \| the publishing packages' indexability gate and each product's review queue decide what is shown; Sources records fetch failures as failures and never fills a gap with invented content \|
453
454	The AI ↔ Sources client: there is none. Sources never calls AI, and AI never fetches from Sources.
455	The product is the only client of both: it reads items from Sources (`sources.item.read`) and sends
456	them to AI as run input (`ai.run.create`). Provenance then travels with the draft, and no service
457	publishes another's output.
458
459	## Owns
460
461	- the source registry (`sources`, `endpoint_state`, `robots_cache`) and the source/item contracts
462	- the merchant opt-in claims (`merchant_sources`), their proof checks and the pages/JSON read
463	- ingestion: `fetch_runs`, `items`, `item_revisions`, the scheduler, robots and rate-limit state
464	- `sources.*` events
465
466	## Does not own
467
468	- publication: what a product shows, summarises, rates or indexes from an item (News, Reviews,
469	Deals, Coupons, Trade, the publishing packages' indexability gate)
470	- AI summarisation (OpenVibe.AI invokes adapters' output; it does not fetch here)
471	- legal clearance: a terms note records what a person verified; it does not replace verifying
472
473	## Depends on
474
475	- PostgreSQL 18 (OpenVibe.Host `roles/data/`, ADR-035): every read and write is async through `openvibe-sdk/db`
476	- OpenVibe.Contracts (service tokens, problem details, ids, the event envelope)
477	- OpenVibe.Network (signing key; service principal `sources`; OAuth client `sources` for merchant sign-in)
478	- OpenVibe.Events (outbound events)
479
480	## Capabilities
481
482	Implemented here (the service manifest's `capabilities`; routes in [API](#api-service-tokens-one-capability-per-route)):
483	`sources.source.read`, `sources.item.read` and `sources.source.manage`, each checked by
484	[server/auth.js](server/auth.js) with the contracts grant rule for audience `openvibe.sources`.
485
486	Called elsewhere, as the service principal `sources` (client credentials from OpenVibe.Network):
487
488	\| Service \| Grant \| Why \|
489	\|---\|---\|---\|
490	\| OpenVibe.Events \| `events.event.publish` (audience `openvibe.events`) \| the outbox relays `sources.item.`, `sources.fetch.failed` and `sources.index_document.` \|
491
492	Sources calls no other OpenVibe service: it never calls AI, and Search receives its index documents
493	through Events.
494
495	## Acceptance (tests)
496
497	- failure never fabricates: HTTP errors, timeouts, parse errors, oversize bodies and 429s leave
498	every item byte-for-byte unchanged, each recorded with its state (`test/ingest.test.js`)
499	- robots.txt respected, including redirects, unreachable = deny, Crawl-delay (`test/robots-ssrf.test.js`)
500	- rate limit spacing and early triggers recorded as `rate_limited` without a request (`test/ingest.test.js`)
501	- conditional GET with ETag and Last-Modified (`test/ingest.test.js`)
502	- loopback/private/metadata addresses, DNS answers and redirects into them refused (`test/robots-ssrf.test.js`)
503	- credentials by variable name, never stored, not forwarded cross-origin (`test/ingest.test.js`)
504	- parsers against fixtures (`test/adapters.test.js`, and ERS's two published CSVs in `test/usda-ers.test.js`); API, registry rules, paging, manual items,
505	removal (`test/api.test.js`); scheduler, backoff, relay to Events (`test/scheduler-events.test.js`);
506	proposals and seeds (`test/proposals.test.js`)
507	- the public host shows only an honest "internal service" page and the merchant opt-in pages, and the vhost keeps the
508	API loopback-only (`test/public-host.test.js`)
509	- merchant opt-in: no source before the DNS/meta proof, the terms note records who/when/how, robots.txt is still obeyed,
510	withdrawal disables and keeps the note, and the JSON read carries last crawl stats (`test/merchant-optin.test.js`,
511	`test/merchant-pages.test.js`)
512
513	Production on 2026-10-22 (counts read from `ov_sources`):
514	- 43 of 47 sources enabled, each after its terms and robots were read (by category in [Seeds](#seeds)); none of them
515	failing. 4,762 fetch runs since the first on 2026-10-07, 431 in the last 24 hours.
516	- 55,683 items, none removed: reviews 33,635, trade 15,736, deals 3,465, hosting 1,097, news 529, crowdfunding 504,
517	games 362, coupons 297 and blog 58; 58,355 item revisions. The database is 253 MB.
518	- Their items feed News, Reviews (Steam signals and the CFPB, NYC Health and NYC HPD records), Deals (DealNews and
519	Open Collective for crowdfunding), Coupons, Trade (SEC, ECB and Frankfurter rates), Host (Porkbun, OVHcloud, Linode,
520	Lightsail and LowEndSpirit), Games (Steam's featured lists) and Blog's desks.
521
522	The first production fetch run was on 2026-10-07; until then every seed stayed disabled.
523
524	Restore drill: `ovhost drill sources` passed on the production host on 2026-09-23 (integrity check,
525	readiness, row counts; see OpenVibe.Host `docs/restore-drills.md`).
526
527	## Security
528
529	Reporting: [SECURITY.md](SECURITY.md). The rules the code keeps:
530
531	- Auth. Every `/api/v1` route needs an OpenVibe.Network service token for audience
532	`openvibe.sources` with that route's capability, then passes the per-actor limits. The merchant API and pages are the
533	one person-facing exception: a Network person session (Bearer on the API, the first-party `ov_token` cookie on the
534	pages) owns its claims, and a first-party service may act for a person only with `X-OV-Subject`. There is still no
535	public JSON API: the vhost keeps `/api/v1/` loopback-only, and only `/merchant` and `/shared/*` are public.
536	- Egress. Sources fetches outside URLs by design, only through its guard: http(s) on ports 80/443,
537	every resolved address public at connect time (no DNS rebinding), each redirect re-checked, robots.txt
538	obeyed, per-host spacing, one deadline and a byte cap per response (`test/robots-ssrf.test.js`,
539	`test/security-ssrf.test.js`). XML entity declarations are refused.
540	- Secrets. A source's credential is the name of a `SOURCES_CRED_*` variable, read at fetch time,
541	sent only to the configured origin, never stored or logged (`test/security-secrets.test.js`).
542	`OV_OAUTH_CLIENT_SECRET` and the source credentials live in `/etc/openvibe/sources.env` (0600).
543	- Data. A failed fetch never creates or changes an item; items keep their provenance and terms;
544	raw items reach Search only for staff (`acl.groups`), always `noindex`.
545
546	## Deploy
547
548	Production deploys with `sudo ovhost deploy sources` on the host (strategy `git-checkout`: fetch,
549	fast-forward `/opt/openvibe.sources`, install on a lockfile change, restart, wait for `/api/ready`).
550
551	\| \| \|
552	\|---\|---\|
553	\| Checkout \| `/opt/openvibe.sources` \|
554	\| Unit \| `openvibe-sources.service` ([deploy/systemd/](deploy/systemd/openvibe-sources.service)) \|
555	\| Port \| `127.0.0.1:4720` \|
556	\| Env file \| `/etc/openvibe/sources.env` \|
557	\| Data \| PostgreSQL `ov_sources` (`DATABASE_URL` through PgBouncer; migrations on `DATABASE_DIRECT_URL`; schema in [migrations/](migrations/)) \|
558	\| Public host \| nginx [deploy/nginx/sources.openvibe.network.conf](deploy/nginx/sources.openvibe.network.conf) (below) \|
559
560	Rollback: ovhost puts the previous sha back by itself when `/api/ready` does not answer 2xx after the
561	restart; afterwards `sudo ovhost rollback sources --to <sha>`. Migrations only add tables and columns.
562
563	### Public host
564
565	`sources.openvibe.network` resolves through Cloudflare; without this vhost nginx answers it with
566	its default site (the admin placeholder). The vhost
567	([deploy/nginx/sources.openvibe.network.conf](deploy/nginx/sources.openvibe.network.conf)) uses
568	the Network wildcard certificate and answers only:
569
570	\| Path \| Answer \|
571	\|---\|---\|
572	\| `/` \| a short page: an internal ingestion service, nothing to browse (HTML for browsers, the text route index otherwise; noindex, no-store) \|
573	\| `/merchant`, `/merchant/*` \| the merchant opt-in pages (guest explanation, signed-in claims, proof/withdraw forms; noindex, no-store) \|
574	\| `/shared/*` \| this release's pinned `theme-loader.js`, `kit.css`, `kit-web.js` (served by `openvibe-shared/serve`) \|
575	\| `/api/health`, `/api/ready`, `/release.json` \| proxied \|
576	\| `/robots.txt` \| `Disallow: /` \|
577	\| `/metrics` \| 404 (Prometheus scrapes `127.0.0.1:4720/metrics` directly) \|
578	\| `/api/v1/*` \| loopback only (`allow 127.0.0.1; allow ::1; deny all`); services call `127.0.0.1:4720` \|
579	\| anything else \| a JSON 404 that says what the host is \|
580
581	Client-address headers come from `$remote_addr` only. Install on the host, after the service runs
582	a release with the new `/` page (`ovhost deploy sources`):
583
584	```bash
585	sudo install -m 0644 /opt/openvibe.sources/deploy/nginx/sources.openvibe.network.conf /etc/nginx/sites-available/sources.openvibe.network.conf
586	sudo ln -sf /etc/nginx/sites-available/sources.openvibe.network.conf /etc/nginx/sites-enabled/sources.openvibe.network.conf
587	sudo nginx -t && sudo systemctl reload nginx
588	# check
589	curl -sS -H 'Accept: text/html' https://sources.openvibe.network/ \| grep -o '<title>[^<]*' # OpenVibe.Sources
590	curl -sS -o /dev/null -w '%{http_code}\n' https://sources.openvibe.network/merchant # 200
591	curl -sS -o /dev/null -w '%{http_code}\n' https://sources.openvibe.network/shared/kit.css # 200
592	curl -sS https://sources.openvibe.network/api/health
593	curl -sS -o /dev/null -w '%{http_code}\n' https://sources.openvibe.network/api/v1/sources # 403
594	curl -sS -o /dev/null -w '%{http_code}\n' https://sources.openvibe.network/metrics # 404
595	curl -sS https://sources.openvibe.network/admin # JSON 404
596	```
597
598
599	## Launch rule
600
601	Sources has no public product pages by design: the only person-facing pages are the merchant opt-in flow (`/merchant`),
602	which exist so a merchant can prove a domain and name a sitemap/feed. It never "launches" as a site. The domain is not
603	in [OpenVibers/OpenVibe.Sites](https://github.com/OpenVibers/OpenVibe.Sites); this repository's own vhost serves it.
604
605	---
606
607	Part of the [OpenVibe network](https://openvibe.network). Built in the open by [OpenVibers](https://github.com/OpenVibers).
608
609	<!-- versions:start -->
610	- openvibe-contracts: v0.419.0
611	- openvibe-sdk: v0.161.0
612	- openvibe-shared: v4.241.0
613	<!-- versions:end -->
614