openvibers/openvibe.sources Commits
Contracts proposals (sources.source@1, sources.item@1), capability and service manifest proposals, README, STATUS alpha W14, env example, systemd unit, nginx vhost, CI on Node 22.22.1
Changes
nnew file mode 100644@@@ -0,0 +1,53 @@++# ═══════════════════════════════════════════════════════════════++# OpenVibe.Sources — Environment Configuration++# Copy to .env and fill in values.++# In production this lives at /etc/openvibe/sources.env (0600); the systemd unit sets++# NODE_ENV, PORT and SOURCES_DB_PATH itself.++# ═══════════════════════════════════════════════════════════════++++# Server++PORT=4720++HOST=127.0.0.1++NODE_ENV=development++#BASE_URL=https://sources.openvibe.network++++# Identity — OpenVibe.Network signs the service tokens (audience openvibe.sources) callers present.++OV_NETWORK_URL=https://openvibe.network++OV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000++#OV_NETWORK_ISSUER=https://openvibe.network++#OV_NETWORK_PUBLIC_KEY=++++# This service's own principal (svc:sources) — publishes sources.* events (events.event.publish).++OV_OAUTH_CLIENT_ID=sources++OV_OAUTH_CLIENT_SECRET=++++# Store (SQLite). Production: /var/lib/openvibe-sources/sources.db (systemd StateDirectory).++SOURCES_DB_PATH=./data/sources.db++++# OpenVibe.Events. The outbox relay runs only when EVENTS_URL is set.++#EVENTS_URL=http://127.0.0.1:4300++#EVENTS_RELAY_INTERVAL_MS=2000++++# Ingestion worker (in-process scheduler)++#SOURCES_WORKER=on++#SOURCES_TICK_MS=5000++#SOURCES_MAX_CONCURRENT=4++#SOURCES_MAX_BACKOFF_MS=21600000++++# Fetching++#SOURCES_USER_AGENT=OpenVibeSources/0.1.0 (+https://sources.openvibe.network; [email protected])++#SOURCES_ROBOTS_AGENT=OpenVibeSources++#SOURCES_FETCH_TIMEOUT_MS=20000++#SOURCES_MAX_BYTES=5242880++#SOURCES_MAX_REDIRECTS=5++#SOURCES_ROBOTS_TTL_MS=86400000++# Least time between two requests to one host, whatever a source says.++#SOURCES_HOST_MIN_INTERVAL_MS=1000++#SOURCES_ALLOWED_PORTS=80,443++# Exact hosts/IPs allowed to resolve to private/loopback addresses. Leave EMPTY in production.++#SOURCES_ALLOW_PRIVATE_HOSTS=++#SOURCES_MAX_ITEMS_PER_FETCH=500++++# Source credentials: a registry entry names the variable (auth.env), never the value. Names++# must start with SOURCES_CRED_. Example (none of the seeds needs one):++#SOURCES_CRED_EXAMPLE_API_KEY=nnew file mode 100644@@@ -0,0 +1,19 @@++name: ci++on:++ push:++ branches: [main]++ pull_request:++jobs:++ test:++ runs-on: ubuntu-latest++ steps:++ - uses: actions/checkout@v4++ - uses: actions/setup-node@v4++ with:++ node-version: 22.22.1 # production runs Node 22++ cache: npm++ - run: npm ci --no-audit --no-fund++ - run: npm test++ # Add once the sources service manifest and the sources.* capabilities ship in++ # openvibe-contracts (proposals in docs/capabilities-proposal/ and docs/service-manifest-proposal.json):++ # - run: npx openvibe-contracts-check --service sources --src servernnew file mode 100644@@@ -0,0 +1,198 @@++# OpenVibe.Sources++++> The source registry and the ingestion workers the publication products depend on: every item++> carries where it came from, when it was retrieved and on what terms, and a failed fetch is a++> recorded failure — never replaced by invented content.++++**Status:** alpha (roadmap Wave 14). Runs and is tested against stub sites; not deployed; the six seeded sources are **disabled**.++**Domain:** `sources.openvibe.network` (health only on the public vhost; the API is host-local)++**Plan:** OpenVibe End-to-End Realignment & Implementation Plan, revision 3 — roadmap §4.2 B, §15.12, §29, anti-goals 13, 26, 27.++**License:** AGPL-3.0.++++## Purpose++++News, Reviews, Deals, Coupons, Trade (and AI summarisation) all need outside material. Sources++owns the one place it enters the network: a registry of sources with their terms, and workers that++fetch them politely and record exactly what happened. Products read items with provenance and++decide — under their own review and indexability rules — what, if anything, to publish.++++## Running it++++```bash++npm install++cp .env.example .env++npm run seed # six real sources, one per category, all disabled++npm run dev # http://127.0.0.1:4720++npm test # every test/*.test.js against local stub sites; no internet++```++++Node 22 in production (`fnm exec --using=22.22.1 npm test`). Production: `/opt/openvibe.sources`,++env `/etc/openvibe/sources.env`, unit [deploy/systemd/openvibe-sources.service](deploy/systemd/openvibe-sources.service),++store `/var/lib/openvibe-sources/sources.db`, nginx [deploy/nginx/sources.openvibe.network.conf](deploy/nginx/sources.openvibe.network.conf).++++`GET /api/health` is liveness. `GET /api/ready` is 200 when the database answers, the Network key++has loaded and the scheduler runs; it reports source counts by health status, runs in flight and++the outbox backlog.++++## The registry++++A source records: `key`, `name`, `type` (`rss|atom|sitemap|jsonld|api|manual`), `category`++(`news|blog|reviews|deals|coupons|trade`), `endpoints` (for `api`: a field mapping, no code per++provider), `auth` (`none|header|bearer|query` with the **name** of an environment variable —++names must start with `SOURCES_CRED_`, so an entry can never point at this service's own++secrets), `robots_note`, `terms_note`, `license_note`, the rate limit (`min_interval_ms`),++`poll_interval_sec`, `stale_after_sec`, `max_items`, `enabled`, `review_required`,++`sensitivity`, `default_indexability` and `search_visibility`.++++A source cannot be enabled without a terms note (and a robots note if it is fetched): an adapter++existing is not permission to ingest (anti-goal 26). A source with items cannot be deleted —++disable it — so provenance stays resolvable. Proposed contract: [`sources.source@1`](docs/contracts-proposal/).++++## Ingestion++++An in-process scheduler starts runs for enabled sources whose poll is due (at most++`SOURCES_MAX_CONCURRENT` at once, never two of one source). A run fetches each endpoint in turn:++++1. **Credential** read from the named variable at that moment (missing → `disabled`), sent only++ to the configured origin (dropped on a cross-origin redirect), never stored or logged.++2. **robots.txt** (RFC 9309) for the origin: our `OpenVibeSources` group or `*`, longest match,++ `*`/`$`; 4xx = no restrictions, 429/5xx/unreachable = complete disallow; cached 24 h;++ `Crawl-delay` honoured. Every redirect hop is checked too.++3. **Spacing**: max(`SOURCES_HOST_MIN_INTERVAL_MS`, the source's `min_interval_ms`,++ Crawl-delay) between requests to a host. A trigger before the source's interval has passed,++ or before a `Retry-After`, records `rate_limited` and requests nothing.++4. **Fetch**: address guard (http(s) only, ports 80/443, every resolved address must be public —++ checked at connect time, so DNS rebinding cannot slip through; loopback, RFC 1918, link-local,++ CGNAT, metadata and v4-mapped/NAT64/6to4 forms are refused), manual redirects (≤ 5, each++ re-checked), conditional GET (`If-None-Match`/`If-Modified-Since`), one deadline for the whole++ response (`SOURCES_FETCH_TIMEOUT_MS`), a byte cap on the decoded body (`SOURCES_MAX_BYTES`).++5. **Parse** with the adapter; on success, in one transaction: the run row, the items (created /++ new revision if the content hash changed / retrieval time refreshed if unchanged), the++ validators, and the events.++++Every attempt writes a `fetch_runs` row with an explicit state: `ok`, `not_modified`,++`http_error`, `timeout`, `robots_denied`, `parse_error`, `rate_limited` or `disabled`, plus the++HTTP status, an error code, the raw-body hash and item counts. **A failed run never creates or++modifies an item** (only the `ok` path touches items), and ETag/Last-Modified are stored only after++a good parse, so an unreadable body is re-fetched in full next time. Failing sources back off++exponentially (capped at `SOURCES_MAX_BACKOFF_MS`).++++**Staleness** is computed: a fetched source is stale when its last success (`ok` or++`not_modified`) is older than `stale_after_sec` (default 3 × the poll interval) or it never++succeeded. Health status: `healthy | stale | failing | never_fetched | disabled | manual`.++++### Adapters (parser versions recorded on every item)++++| Type | Reads | Notes |++|---|---|---|++| `rss`, `atom` (`feed@1`) | RSS 2.0, RSS 1.0/RDF, Atom 1.0 | guid/id identity, else link; text-only summary capped at 1000 chars |++| `sitemap` (`sitemap@1`) | `<urlset>` (+ Google News extension), `<sitemapindex>` | index children are listed, not followed |++| `jsonld` (`jsonld@1`) | schema.org Product, Offer/AggregateOffer, Review, NewsArticle, BlogPosting (+ Article) | prices, ratings and availability exactly as the page states; missing = null; embedded review text not kept; a page without JSON-LD is a `parse_error` |++| `api` (`api@1`) | JSON or XML official APIs via a registry mapping | `items_path`, `fields`, `extra`; epoch fields only where mapped as such |++| `manual` (`manual@1`) | items entered through the API with an evidence URL | no fetcher |++++XML with DTD entity declarations is refused; the parser expands no entities (values are decoded++per field). Fixtures and expectations: `test/adapters.test.js`.++++### Items++++Each item: `source_key`, `category`, `kind`, `identity`, `canonical_url` (tracking parameters and++fragments removed), `title`, `summary`, `authors`, `published_at`, `source_updated_at`, `fields`,++`revision`, and `provenance` (`retrieved_at`, `first_seen_at`, `content_hash`, `raw_body_hash`,++`parser_version`, `fetch_run_id`, `license_note`, `terms_note`, `entered_by`). Earlier revisions++are kept in `item_revisions`. Removal (takedown, licence) is explicit, needs a reason, and is++sticky: a later fetch that still lists the item does not bring it back. Proposed contract:++[`sources.item@1`](docs/contracts-proposal/).++++## API (service tokens, one capability per route)++++Callers use an OpenVibe.Network client-credentials token for audience `openvibe.sources`.++++| Capability | Routes |++|---|---|++| `sources.source.read` | `GET /api/v1/sources`, `/api/v1/sources/:key`, `/api/v1/sources/:key/runs?before=`, `/api/v1/runs?state=failed&after=`, `/api/v1/health` |++| `sources.item.read` | `GET /api/v1/items?source=&category=&after=<change_seq>&include_removed=1`, `GET /api/v1/items/:id?revisions=1` |++| `sources.source.manage` | `POST /api/v1/sources`, `PATCH`/`DELETE /api/v1/sources/:key`, `POST /api/v1/sources/:key/fetch`, `POST /api/v1/sources/:key/items` (manual), `DELETE /api/v1/items/:id` `{reason}` |++++`GET /api/v1/items` pages in change order (creations, revisions and — with `include_removed=1` —++removals); resume from `next_after`. Every page carries the status and staleness of the sources++it contains. The capability ids are proposed in [docs/capabilities-proposal/](docs/capabilities-proposal/)++and not in `openvibe-contracts` v0.7.0; [server/auth.js](server/auth.js) decides them with the++contracts grant rule until a release defines them.++++## Events (transactional outbox → OpenVibe.Events when `EVENTS_URL` is set)++++- `sources.item.created`, `sources.item.updated` (with the previous content hash), `sources.item.removed` (with the reason)++- `sources.fetch.failed` for `http_error`, `timeout`, `robots_denied`, `parse_error` and upstream 429 (with the state, error code, HTTP status and consecutive failures). Our own rate limiter and disabled sources are not failures.++- `sources.index_document.upserted|deleted` for sources with `search_visibility: "members"`: raw items indexed in OpenVibe.Search for staff only (`acl.groups: role:admin, role:global_mod`), always `noindex` (`third_party_content`).++++All with `visibility: "internal"`. Envelopes validate as `events.event-envelope@1`.++++## Seeds++++[seeds/sources.json](seeds/sources.json) — one real source per category, **all disabled**, robots++checked by hand on 2026-09-22 (the fetcher re-checks on every run), terms to be re-verified by a++person before enabling:++++| Key | Category | Type | Endpoint |++|---|---|---|---|++| `nasa-news-releases` | news | rss | nasa.gov news-release feed |++| `nodejs-blog` | blog | rss | nodejs.org blog feed |++| `steam-reviews-portal-2` | reviews | api | Steam `appreviews` for app 620 — signals only, review text not mapped |++| `dealnews-daily` | deals | rss | dealnews.com RSS (Crawl-delay 2) |++| `staff-coupon-codes` | coupons | manual | codes a merchant published itself, entered with the evidence URL |++| `sec-xbrl-filings` | trade | rss | SEC EDGAR XBRL filings RSS (`/cgi-bin` is robots-disallowed, so the getcurrent feed is not used) |++++Candidates rejected on 2026-09-22: CheapShark's API (`/api/1.0/` is disallowed in its++robots.txt) and EDGAR's `getcurrent` Atom feed (`/cgi-bin` disallowed). The seeded feeds were++downloaded once by hand and parsed offline with these adapters (NASA 10 items, Node.js 50,++Steam 20, DealNews 50, SEC 200); that check is not part of the test suite and proves nothing about++the terms.++++## Owns++++- the source registry (`sources`, `endpoint_state`, `robots_cache`) and the source/item contracts++- ingestion: `fetch_runs`, `items`, `item_revisions`, the scheduler, robots and rate-limit state++- `sources.*` events++++## Does not own++++- publication: what a product shows, summarises, rates or indexes from an item (News, Reviews,++ Deals, Coupons, Trade, the publishing packages' indexability gate)++- AI summarisation (OpenVibe.AI invokes adapters' output; it does not fetch here)++- legal clearance: a terms note records what a person verified; it does not replace verifying++++## Depends on++++- OpenVibe.Contracts (service tokens, problem details, ids, the event envelope)++- OpenVibe.Network (signing key; service principal `sources`)++- OpenVibe.Events (outbound events)++++## Acceptance (tests)++++- failure never fabricates: HTTP errors, timeouts, parse errors, oversize bodies and 429s leave++ every item byte-for-byte unchanged, each recorded with its state (`test/ingest.test.js`)++- robots.txt respected, including redirects, unreachable = deny, Crawl-delay (`test/robots-ssrf.test.js`)++- rate limit spacing and early triggers recorded as `rate_limited` without a request (`test/ingest.test.js`)++- conditional GET with ETag and Last-Modified (`test/ingest.test.js`)++- loopback/private/metadata addresses, DNS answers and redirects into them refused (`test/robots-ssrf.test.js`)++- credentials by variable name, never stored, not forwarded cross-origin (`test/ingest.test.js`)++- parsers against fixtures (`test/adapters.test.js`); API, registry rules, paging, manual items,++ removal (`test/api.test.js`); scheduler, backoff, relay to Events (`test/scheduler-events.test.js`);++ proposals and seeds (`test/proposals.test.js`)++++Not yet demonstrated: a run against a real source from the deployed service (every seed is disabled++until a person verifies its terms), and a consuming product (News/Reviews/Deals/Coupons/Trade do++not exist yet).++++## Launch rule++++The domain keeps its placeholder page on [OpenVibers/OpenVibe.Sites](https://github.com/OpenVibers/OpenVibe.Sites)++until the plan's launch rule holds (owning runtime with health/readiness, identity and service++principals, real persistence and workflows, capabilities and events registered in++OpenVibe.Contracts, a security review, acceptance tests). Sources has no public pages by design;++its public vhost answers health and a `Disallow: /` robots.txt.++++---++++Part of the [OpenVibe network](https://openvibe.network). Built in the open by [OpenVibers](https://github.com/OpenVibers).nnew file mode 100644@@@ -0,0 +1,19 @@++{++ "repository": "OpenVibers/OpenVibe.Sources",++ "stage": "alpha",++ "code": true,++ "wave": "W14",++ "plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026)",++ "runtime": "node 22, express, better-sqlite3, fast-xml-parser",++ "port": 4720,++ "domain": "sources.openvibe.network",++ "contracts": "openvibe-contracts v0.7.0",++ "capabilities": ["sources.source.read", "sources.item.read", "sources.source.manage"],++ "capabilitiesRegistered": false,++ "contractsProposed": ["sources.source@1", "sources.item@1"],++ "adapters": ["rss", "atom", "sitemap", "jsonld", "api", "manual"],++ "seededSources": 6,++ "enabledSources": 0,++ "deployed": false,++ "updated": "2026-09-22"++}nnew file mode 100644@@@ -0,0 +1,75 @@++## ═══════════════════════════════════════════════════════════════++## sources.openvibe.network — Nginx Configuration (REFERENCE)++##++## Deployed to /etc/nginx/sites-available/sources.openvibe.network.conf — keep them in sync,++## and reload nginx after changing it.++##++## The Node service (port 4720, systemd unit openvibe-sources) serves:++## /api/health, /api/ready++## /api/v1/* service-to-service API (every route needs a service token). Services on this++## host call 127.0.0.1:4720 directly, so the public vhost refuses it; open the++## location below only when a service on another host needs it.++## Nothing here is a public page: Sources holds raw third-party material, which products++## publish (or not) under their own rules.++## ═══════════════════════════════════════════════════════════════++++# ── HTTP → HTTPS redirect ──────────────────────────────────++server {++ listen 80;++ listen [::]:80;++ server_name sources.openvibe.network;++++ location /.well-known/acme-challenge/ { root /var/www/certbot; }++ location / { return 301 https://sources.openvibe.network$request_uri; }++}++++# ── HTTPS server ───────────────────────────────────────────++server {++ listen 443 ssl;++ listen [::]:443 ssl;++ http2 on;++ server_name sources.openvibe.network;++++ # The host has the openvibe.network wildcard certificate, not a per-name one.++ ssl_certificate /etc/letsencrypt/live/openvibe.network/fullchain.pem;++ ssl_certificate_key /etc/letsencrypt/live/openvibe.network/privkey.pem;++ ssl_protocols TLSv1.2 TLSv1.3;++++ access_log /var/log/nginx/sources.openvibe.network.access.log;++ error_log /var/log/nginx/sources.openvibe.network.error.log;++++ client_max_body_size 256k;++++ add_header X-Content-Type-Options nosniff always;++ add_header Referrer-Policy strict-origin-when-cross-origin always;++ add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;++ add_header X-Robots-Tag "noindex, nofollow" always;++++ location = /api/health { proxy_pass http://127.0.0.1:4720; proxy_set_header Host $host; }++ location = /api/ready { proxy_pass http://127.0.0.1:4720; proxy_set_header Host $host; }++++ # ── Service API: host-local only (see header) ───────────++ location /api/ {++ return 403;++ # To expose it (token-guarded) to off-host services instead:++ # proxy_pass http://127.0.0.1:4720;++ # proxy_http_version 1.1;++ # proxy_set_header Host $host;++ # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;++ # proxy_set_header X-Forwarded-Proto $scheme;++ }++++ location = / {++ proxy_pass http://127.0.0.1:4720;++ proxy_set_header Host $host;++ }++++ location = /robots.txt {++ default_type text/plain;++ return 200 "User-agent: *\nDisallow: /\n";++ }++++ location / {++ return 404;++ }++}nnew file mode 100644@@@ -0,0 +1,32 @@++[Unit]++Description=OpenVibe.Sources — source registry and ingestion workers (RSS/Atom, sitemaps, JSON-LD, APIs)++# Network (openvibe-network) signs the tokens this service verifies. Sources starts without it++# and keeps retrying the JWKS fetch; /api/ready stays 503 until the key has loaded.++After=network.target++Wants=openvibe-network.service++++[Service]++Type=simple++User=ubuntu++Group=ubuntu++WorkingDirectory=/opt/openvibe.sources++EnvironmentFile=/etc/openvibe/sources.env++Environment=NODE_ENV=production++Environment=PORT=4720++Environment=PATH=/usr/local/bin:/usr/bin:/bin++ExecStart=/usr/bin/env node /opt/openvibe.sources/server/index.js++Restart=always++RestartSec=5++# SIGTERM: the scheduler stops starting runs; runs in flight finish (each is bounded by the fetch timeout).++KillSignal=SIGTERM++TimeoutStopSec=30++NoNewPrivileges=true++PrivateTmp=true++ProtectSystem=strict++ProtectHome=true++# The code tree stays read-only; the registry and items live in /var/lib/openvibe-sources.++StateDirectory=openvibe-sources++Environment=SOURCES_DB_PATH=/var/lib/openvibe-sources/sources.db++++[Install]++WantedBy=multi-user.targetnnew file mode 100644@@@ -0,0 +1,21 @@++{++ "id": "sources.item.read",++ "version": "1.0",++ "owner": "sources",++ "status": "active",++ "visibility": "first-party",++ "description": "Read source items with provenance (source, canonical URL, retrieval time, content and raw-body hashes, parser version, terms) in change order, and one item with its revisions. Each page states the staleness of the sources it contains.",++ "outputSchema": "sources.item@1",++ "permissions": [++ "sources:read"++ ],++ "resourceConstraints": [++ "none"++ ],++ "quotaClass": "sources-read",++ "events": [],++ "implementedBy": [++ "GET /api/v1/items",++ "GET /api/v1/items/:id"++ ]++}nnew file mode 100644@@@ -0,0 +1,28 @@++{++ "id": "sources.source.manage",++ "version": "1.0",++ "owner": "sources",++ "status": "active",++ "visibility": "first-party",++ "description": "Create, change, enable or disable and delete (only while it has no items) sources; trigger a fetch; add items to manual sources with an evidence URL; remove an item with a reason (sticky). Enabling needs a terms note, and a robots note for fetched sources.",++ "permissions": [++ "sources:admin"++ ],++ "resourceConstraints": [++ "none"++ ],++ "quotaClass": "sources-admin",++ "events": [++ "sources.item.created",++ "sources.item.updated",++ "sources.item.removed",++ "sources.fetch.failed"++ ],++ "implementedBy": [++ "POST /api/v1/sources",++ "PATCH|DELETE /api/v1/sources/:key",++ "POST /api/v1/sources/:key/fetch",++ "POST /api/v1/sources/:key/items",++ "DELETE /api/v1/items/:id"++ ]++}nnew file mode 100644@@@ -0,0 +1,24 @@++{++ "id": "sources.source.read",++ "version": "1.0",++ "owner": "sources",++ "status": "active",++ "visibility": "first-party",++ "description": "List registered sources with health and staleness, their fetch runs (every attempt with an explicit state) and the health summary. Credential values are never part of any response; only the variable name is.",++ "outputSchema": "sources.source@1",++ "permissions": [++ "sources:read"++ ],++ "resourceConstraints": [++ "none"++ ],++ "quotaClass": "sources-read",++ "events": [],++ "implementedBy": [++ "GET /api/v1/sources",++ "GET /api/v1/sources/:key",++ "GET /api/v1/sources/:key/runs",++ "GET /api/v1/runs",++ "GET /api/v1/health"++ ]++}nnew file mode 100644@@@ -0,0 +1,38 @@++# Proposal for OpenVibe.Contracts: `sources.source@1`, `sources.item@1`++++Wave 14 asks OpenVibe.Contracts for the source-adapter contract. Files to add in its next release:++++| File here | Goes to |++|---|---|++| [contracts/sources/source.v1.json](contracts/sources/source.v1.json) | `contracts/sources/source.v1.json` |++| [contracts/sources/item.v1.json](contracts/sources/item.v1.json) | `contracts/sources/item.v1.json` |++| [catalog-entries.json](catalog-entries.json) | two new entries in `contracts/catalog.json` |++| [../capabilities-proposal/*.json](../capabilities-proposal/) | `manifests/capabilities/` |++| [../service-manifest-proposal.json](../service-manifest-proposal.json) | `manifests/services/sources.json` |++++`test/proposals.test.js` checks that the API's source and item views validate against these++schemas, that the capability and service manifests validate against the contracts' own++schemas, and that the seeds are valid.++++## Events++++| Type | Subject | Payload |++|---|---|---|++| `sources.item.created` | `{type: item, id, revision}` | item_id, source_key, category, kind, canonical_url, title, revision, content_hash, retrieved_at |++| `sources.item.updated` | same | + previous_content_hash |++| `sources.item.removed` | same | + reason |++| `sources.fetch.failed` | `{type: source, id: <key>}` | source_key, category, run_id, endpoint_url, state, error_code, http_status, consecutive_failures |++| `sources.index_document.upserted` | `{type: item, id, revision}` | a `search.index-document@1` (staff-only, noindex) |++| `sources.index_document.deleted` | same | `{type, id, revision}` |++++OpenVibe.Events' default source prefixes do not include `sources` yet (`EVENTS_SOURCE_PREFIXES`++or a default-list change is needed before these publish).++++## How products are expected to use items++++A product never shows an item as its own fact: it keeps a typed reference++(`{service: 'sources', type: 'item', id, revision}`) as provenance, shows `retrieved_at` as the++observation time (a price is "as of", never "current"), treats `null` as unknown, honours++`terms_note`/`license_note`, starts its indexability decision from the source's++`default_indexability`/`review_required`/`sensitivity`, and reacts to `sources.item.updated` /++`removed` with a revision or a correction of its own.nnew file mode 100644@@@ -0,0 +1,4 @@++[++ { "id": "sources.source", "version": "1.0.0", "owner": "sources", "schema": "sources/source.v1.json", "visibility": "first-party", "compatibility": "backward", "status": "active" },++ { "id": "sources.item", "version": "1.0.0", "owner": "sources", "schema": "sources/item.v1.json", "visibility": "first-party", "compatibility": "backward", "status": "active" }++]nnew file mode 100644@@@ -0,0 +1,46 @@++{++ "$schema": "https://json-schema.org/draft/2020-12/schema",++ "$id": "https://openvibe.network/contracts/sources/item.v1.json",++ "title": "SourceItem",++ "description": "One thing a source said, with its provenance: source, canonical URL, retrieval time, content hash, raw-body hash, parser version and the terms it was ingested under. A field the source did not state is null — never guessed. Items change only through successful fetches (a new revision per content change) or an explicit, audited removal.",++ "type": "object",++ "required": ["id", "source_key", "category", "kind", "identity", "canonical_url", "title", "summary", "authors", "published_at", "source_updated_at", "fields", "revision", "provenance", "removed"],++ "additionalProperties": false,++ "properties": {++ "id": { "type": "string", "pattern": "^itm_[0-9A-HJKMNP-TV-Z]{26}$" },++ "source_key": { "type": "string", "pattern": "^[a-z0-9][a-z0-9-]{1,63}$" },++ "category": { "enum": ["news", "blog", "reviews", "deals", "coupons", "trade"] },++ "kind": { "type": "string", "pattern": "^[a-z][a-z0-9_]{1,39}$", "description": "article | url | sitemap | product | offer | review | review_signal | record | coupon …" },++ "identity": { "type": "string", "minLength": 1, "maxLength": 2048, "description": "The source's own id (guid, @id, API id) or the canonical URL." },++ "canonical_url": { "type": ["string", "null"], "format": "uri" },++ "title": { "type": ["string", "null"] },++ "summary": { "type": ["string", "null"], "maxLength": 1000, "description": "Plain text, capped. Sources keeps metadata and short descriptions, not republishable full text." },++ "authors": { "type": "array", "items": { "type": "string" } },++ "published_at": { "type": ["string", "null"], "format": "date-time" },++ "source_updated_at": { "type": ["string", "null"], "format": "date-time" },++ "fields": { "type": "object", "description": "Adapter-specific parsed fields (offers, ratings, prices as stated, API extras)." },++ "revision": { "type": "integer", "minimum": 1 },++ "provenance": {++ "type": "object",++ "required": ["retrieved_at", "first_seen_at", "content_hash", "raw_body_hash", "parser_version", "fetch_run_id", "license_note", "terms_note", "entered_by"],++ "additionalProperties": false,++ "properties": {++ "retrieved_at": { "type": "string", "format": "date-time", "description": "Last successful fetch that contained this item (observation time)." },++ "first_seen_at": { "type": "string", "format": "date-time" },++ "content_hash": { "type": "string", "pattern": "^[0-9a-f]{64}$" },++ "raw_body_hash": { "type": ["string", "null"], "pattern": "^[0-9a-f]{64}$" },++ "parser_version": { "type": "string", "pattern": "^[a-z]+@\\d+$" },++ "fetch_run_id": { "type": ["string", "null"], "pattern": "^frn_[0-9A-HJKMNP-TV-Z]{26}$" },++ "license_note": { "type": ["string", "null"] },++ "terms_note": { "type": ["string", "null"] },++ "entered_by": { "type": ["string", "null"], "description": "Principal that entered a manual item." }++ }++ },++ "removed": {++ "type": ["object", "null"], "additionalProperties": false, "required": ["at", "reason"],++ "properties": { "at": { "type": "string", "format": "date-time" }, "reason": { "type": "string" } }++ },++ "change_seq": { "type": "integer", "minimum": 1 },++ "revisions": { "type": "array" }++ }++}nnew file mode 100644@@@ -0,0 +1,66 @@++{++ "$schema": "https://json-schema.org/draft/2020-12/schema",++ "$id": "https://openvibe.network/contracts/sources/source.v1.json",++ "title": "Source",++ "description": "A registered source (roadmap §4.2 B, Wave 14): where it is fetched from and on what terms. Credentials are referenced by environment-variable NAME only. A source's existence is never permission to fetch: robots and terms are verified before it is enabled, and the fetcher re-checks robots.txt on every run.",++ "type": "object",++ "required": ["key", "name", "type", "category", "endpoints", "auth", "min_interval_ms", "poll_interval_sec", "stale_after_sec", "max_items", "enabled", "review_required", "sensitivity", "default_indexability"],++ "additionalProperties": false,++ "properties": {++ "key": { "type": "string", "pattern": "^[a-z0-9][a-z0-9-]{1,63}$" },++ "name": { "type": "string", "minLength": 1, "maxLength": 200 },++ "type": { "enum": ["rss", "atom", "sitemap", "jsonld", "api", "manual"] },++ "category": { "enum": ["news", "blog", "reviews", "deals", "coupons", "trade"] },++ "homepage_url": { "type": ["string", "null"], "format": "uri" },++ "endpoints": {++ "type": "array", "maxItems": 20,++ "items": {++ "type": "object", "required": ["url"], "additionalProperties": false,++ "properties": {++ "url": { "type": "string", "format": "uri", "pattern": "^https?://" },++ "format": { "enum": ["json", "xml"] },++ "items_path": { "type": "string" },++ "item_kind": { "type": "string", "pattern": "^[a-z][a-z0-9_]{1,39}$" },++ "fields": { "type": "object", "additionalProperties": { "type": "string" } },++ "extra": { "type": "object", "additionalProperties": { "type": "string" } }++ }++ }++ },++ "auth": {++ "type": "object", "required": ["mode"], "additionalProperties": false,++ "properties": {++ "mode": { "enum": ["none", "header", "bearer", "query"] },++ "env": { "type": "string", "pattern": "^SOURCES_CRED_[A-Z0-9_]{2,60}$", "description": "Name of the environment variable holding the credential. Never a value." },++ "header": { "type": "string", "pattern": "^[A-Za-z][A-Za-z0-9-]{0,63}$" },++ "param": { "type": "string", "pattern": "^[A-Za-z0-9_.-]{1,64}$" }++ }++ },++ "robots_note": { "type": ["string", "null"], "maxLength": 2000 },++ "terms_note": { "type": ["string", "null"], "maxLength": 2000 },++ "license_note": { "type": ["string", "null"], "maxLength": 2000 },++ "min_interval_ms": { "type": "integer", "minimum": 0, "description": "Rate limit: the least time between two requests to this source (the per-host floor and robots Crawl-delay may raise it)." },++ "poll_interval_sec": { "type": "integer", "minimum": 1 },++ "stale_after_sec": { "type": "integer", "minimum": 1 },++ "max_items": { "type": "integer", "minimum": 1 },++ "enabled": { "type": "boolean" },++ "review_required": { "type": "boolean", "description": "Products must review what they derive from this source before it can be indexable." },++ "sensitivity": { "enum": ["none", "financial", "health", "political", "legal", "adult"] },++ "default_indexability": { "enum": ["index", "noindex"], "description": "What a product's indexability gate starts from for content derived from this source." },++ "search_visibility": { "type": ["string", "null"], "enum": ["members", null], "description": "members = raw items are indexed in OpenVibe.Search for staff only (never indexable)." },++ "health": {++ "type": "object", "additionalProperties": false,++ "properties": {++ "status": { "enum": ["healthy", "stale", "failing", "never_fetched", "disabled", "manual"] },++ "stale": { "type": "boolean" },++ "last_success_at": { "type": ["string", "null"], "format": "date-time" },++ "last_run_at": { "type": ["string", "null"], "format": "date-time" },++ "last_state": { "type": ["string", "null"] },++ "consecutive_failures": { "type": "integer", "minimum": 0 },++ "next_due_at": { "type": ["string", "null"], "format": "date-time" },++ "stale_after_sec": { "type": "integer" }++ }++ },++ "created_at": { "type": "string", "format": "date-time" },++ "updated_at": { "type": "string", "format": "date-time" }++ }++}nnew file mode 100644@@@ -0,0 +1,32 @@++{++ "version": "0.1.0",++ "contractRanges": {++ "openvibe-contracts": ">=0.7.0 <1.0.0"++ },++ "id": "sources",++ "name": "OpenVibe.Sources",++ "status": "alpha",++ "repository": "OpenVibers/OpenVibe.Sources",++ "domains": [++ "sources.openvibe.network"++ ],++ "publicOrigin": "https://sources.openvibe.network",++ "health": "/api/health",++ "ready": "/api/ready",++ "capabilities": [++ "sources.source.read",++ "sources.item.read",++ "sources.source.manage"++ ],++ "eventsProduced": [++ "sources.item.created",++ "sources.item.updated",++ "sources.item.removed",++ "sources.fetch.failed",++ "sources.index_document.upserted",++ "sources.index_document.deleted"++ ],++ "eventsConsumed": [],++ "namespacesOwned": [],++ "notes": "Source registry and ingestion workers (RSS/Atom, sitemaps, JSON-LD, mapped official APIs, manual). Every item carries provenance; every fetch is a recorded run with an explicit state; failures never create or modify items."++}
new file mode 100644
new file mode 100644
@@ -0,0 +1,53 @@
@@ -0,0 +1,53 @@
+# ═══════════════════════════════════════════════════════════════
+# OpenVibe.Sources — Environment Configuration
+# Copy to .env and fill in values.
+# In production this lives at /etc/openvibe/sources.env (0600); the systemd unit sets
+# NODE_ENV, PORT and SOURCES_DB_PATH itself.
+# ═══════════════════════════════════════════════════════════════
+
+# Server
+PORT=4720
+HOST=127.0.0.1
+NODE_ENV=development
+#BASE_URL=https://sources.openvibe.network
+
+# Identity — OpenVibe.Network signs the service tokens (audience openvibe.sources) callers present.
+OV_NETWORK_URL=https://openvibe.network
+OV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000
+#OV_NETWORK_ISSUER=https://openvibe.network
+#OV_NETWORK_PUBLIC_KEY=
+
+# This service's own principal (svc:sources) — publishes sources.* events (events.event.publish).
+OV_OAUTH_CLIENT_ID=sources
+OV_OAUTH_CLIENT_SECRET=
+
+# Store (SQLite). Production: /var/lib/openvibe-sources/sources.db (systemd StateDirectory).
+SOURCES_DB_PATH=./data/sources.db
+
+# OpenVibe.Events. The outbox relay runs only when EVENTS_URL is set.
+#EVENTS_URL=http://127.0.0.1:4300
+#EVENTS_RELAY_INTERVAL_MS=2000
+
+# Ingestion worker (in-process scheduler)
+#SOURCES_WORKER=on
+#SOURCES_TICK_MS=5000
+#SOURCES_MAX_CONCURRENT=4
+#SOURCES_MAX_BACKOFF_MS=21600000
+
+# Fetching
+#SOURCES_USER_AGENT=OpenVibeSources/0.1.0 (+https://sources.openvibe.network; [email protected])
+#SOURCES_ROBOTS_AGENT=OpenVibeSources
+#SOURCES_FETCH_TIMEOUT_MS=20000
+#SOURCES_MAX_BYTES=5242880
+#SOURCES_MAX_REDIRECTS=5
+#SOURCES_ROBOTS_TTL_MS=86400000
+# Least time between two requests to one host, whatever a source says.
+#SOURCES_HOST_MIN_INTERVAL_MS=1000
+#SOURCES_ALLOWED_PORTS=80,443
+# Exact hosts/IPs allowed to resolve to private/loopback addresses. Leave EMPTY in production.
+#SOURCES_ALLOW_PRIVATE_HOSTS=
+#SOURCES_MAX_ITEMS_PER_FETCH=500
+
+# Source credentials: a registry entry names the variable (auth.env), never the value. Names
+# must start with SOURCES_CRED_. Example (none of the seeds needs one):
+#SOURCES_CRED_EXAMPLE_API_KEY=
new file mode 100644
new file mode 100644
@@ -0,0 +1,19 @@
@@ -0,0 +1,19 @@
+name: ci
+on:
+ push:
+ branches: [main]
+ pull_request:
+jobs:
+ test:
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v4
+ - uses: actions/setup-node@v4
+ with:
+ node-version: 22.22.1 # production runs Node 22
+ cache: npm
+ - run: npm ci --no-audit --no-fund
+ - run: npm test
+ # Add once the sources service manifest and the sources.* capabilities ship in
+ # openvibe-contracts (proposals in docs/capabilities-proposal/ and docs/service-manifest-proposal.json):
+ # - run: npx openvibe-contracts-check --service sources --src server
new file mode 100644
new file mode 100644
@@ -0,0 +1,198 @@
@@ -0,0 +1,198 @@
+# OpenVibe.Sources
+
+> The source registry and the ingestion workers the publication products depend on: every item
+> carries where it came from, when it was retrieved and on what terms, and a failed fetch is a
+> recorded failure — never replaced by invented content.
+
+**Status:** alpha (roadmap Wave 14). Runs and is tested against stub sites; not deployed; the six seeded sources are **disabled**.
+**Domain:** `sources.openvibe.network` (health only on the public vhost; the API is host-local)
+**Plan:** OpenVibe End-to-End Realignment & Implementation Plan, revision 3 — roadmap §4.2 B, §15.12, §29, anti-goals 13, 26, 27.
+**License:** AGPL-3.0.
+
+## Purpose
+
+News, Reviews, Deals, Coupons, Trade (and AI summarisation) all need outside material. Sources
+owns the one place it enters the network: a registry of sources with their terms, and workers that
+fetch them politely and record exactly what happened. Products read items with provenance and
+decide — under their own review and indexability rules — what, if anything, to publish.
+
+## Running it
+
+```bash
+npm install
+cp .env.example .env
+npm run seed # six real sources, one per category, all disabled
+npm run dev # http://127.0.0.1:4720
+npm test # every test/*.test.js against local stub sites; no internet
+```
+
+Node 22 in production (`fnm exec --using=22.22.1 npm test`). Production: `/opt/openvibe.sources`,
+env `/etc/openvibe/sources.env`, unit [deploy/systemd/openvibe-sources.service](deploy/systemd/openvibe-sources.service),
+store `/var/lib/openvibe-sources/sources.db`, nginx [deploy/nginx/sources.openvibe.network.conf](deploy/nginx/sources.openvibe.network.conf).
+
+`GET /api/health` is liveness. `GET /api/ready` is 200 when the database answers, the Network key
+has loaded and the scheduler runs; it reports source counts by health status, runs in flight and
+the outbox backlog.
+
+## The registry
+
+A source records: `key`, `name`, `type` (`rss|atom|sitemap|jsonld|api|manual`), `category`
+(`news|blog|reviews|deals|coupons|trade`), `endpoints` (for `api`: a field mapping, no code per
+provider), `auth` (`none|header|bearer|query` with the **name** of an environment variable —
+names must start with `SOURCES_CRED_`, so an entry can never point at this service's own
+secrets), `robots_note`, `terms_note`, `license_note`, the rate limit (`min_interval_ms`),
+`poll_interval_sec`, `stale_after_sec`, `max_items`, `enabled`, `review_required`,
+`sensitivity`, `default_indexability` and `search_visibility`.
+
+A source cannot be enabled without a terms note (and a robots note if it is fetched): an adapter
+existing is not permission to ingest (anti-goal 26). A source with items cannot be deleted —
+disable it — so provenance stays resolvable. Proposed contract: [`sources.source@1`](docs/contracts-proposal/).
+
+## Ingestion
+
+An in-process scheduler starts runs for enabled sources whose poll is due (at most
+`SOURCES_MAX_CONCURRENT` at once, never two of one source). A run fetches each endpoint in turn:
+
+1. **Credential** read from the named variable at that moment (missing → `disabled`), sent only
+ to the configured origin (dropped on a cross-origin redirect), never stored or logged.
+2. **robots.txt** (RFC 9309) for the origin: our `OpenVibeSources` group or `*`, longest match,
+ `*`/`$`; 4xx = no restrictions, 429/5xx/unreachable = complete disallow; cached 24 h;
+ `Crawl-delay` honoured. Every redirect hop is checked too.
+3. **Spacing**: max(`SOURCES_HOST_MIN_INTERVAL_MS`, the source's `min_interval_ms`,
+ Crawl-delay) between requests to a host. A trigger before the source's interval has passed,
+ or before a `Retry-After`, records `rate_limited` and requests nothing.
+4. **Fetch**: address guard (http(s) only, ports 80/443, every resolved address must be public —
+ checked at connect time, so DNS rebinding cannot slip through; loopback, RFC 1918, link-local,
+ CGNAT, metadata and v4-mapped/NAT64/6to4 forms are refused), manual redirects (≤ 5, each
+ re-checked), conditional GET (`If-None-Match`/`If-Modified-Since`), one deadline for the whole
+ response (`SOURCES_FETCH_TIMEOUT_MS`), a byte cap on the decoded body (`SOURCES_MAX_BYTES`).
+5. **Parse** with the adapter; on success, in one transaction: the run row, the items (created /
+ new revision if the content hash changed / retrieval time refreshed if unchanged), the
+ validators, and the events.
+
+Every attempt writes a `fetch_runs` row with an explicit state: `ok`, `not_modified`,
+`http_error`, `timeout`, `robots_denied`, `parse_error`, `rate_limited` or `disabled`, plus the
+HTTP status, an error code, the raw-body hash and item counts. **A failed run never creates or
+modifies an item** (only the `ok` path touches items), and ETag/Last-Modified are stored only after
+a good parse, so an unreadable body is re-fetched in full next time. Failing sources back off
+exponentially (capped at `SOURCES_MAX_BACKOFF_MS`).
+
+**Staleness** is computed: a fetched source is stale when its last success (`ok` or
+`not_modified`) is older than `stale_after_sec` (default 3 × the poll interval) or it never
+succeeded. Health status: `healthy | stale | failing | never_fetched | disabled | manual`.
+
+### Adapters (parser versions recorded on every item)
+
+| Type | Reads | Notes |
+|---|---|---|
+| `rss`, `atom` (`feed@1`) | RSS 2.0, RSS 1.0/RDF, Atom 1.0 | guid/id identity, else link; text-only summary capped at 1000 chars |
+| `sitemap` (`sitemap@1`) | `<urlset>` (+ Google News extension), `<sitemapindex>` | index children are listed, not followed |
+| `jsonld` (`jsonld@1`) | schema.org Product, Offer/AggregateOffer, Review, NewsArticle, BlogPosting (+ Article) | prices, ratings and availability exactly as the page states; missing = null; embedded review text not kept; a page without JSON-LD is a `parse_error` |
+| `api` (`api@1`) | JSON or XML official APIs via a registry mapping | `items_path`, `fields`, `extra`; epoch fields only where mapped as such |
+| `manual` (`manual@1`) | items entered through the API with an evidence URL | no fetcher |
+
+XML with DTD entity declarations is refused; the parser expands no entities (values are decoded
+per field). Fixtures and expectations: `test/adapters.test.js`.
+
+### Items
+
+Each item: `source_key`, `category`, `kind`, `identity`, `canonical_url` (tracking parameters and
+fragments removed), `title`, `summary`, `authors`, `published_at`, `source_updated_at`, `fields`,
+`revision`, and `provenance` (`retrieved_at`, `first_seen_at`, `content_hash`, `raw_body_hash`,
+`parser_version`, `fetch_run_id`, `license_note`, `terms_note`, `entered_by`). Earlier revisions
+are kept in `item_revisions`. Removal (takedown, licence) is explicit, needs a reason, and is
+sticky: a later fetch that still lists the item does not bring it back. Proposed contract:
+[`sources.item@1`](docs/contracts-proposal/).
+
+## API (service tokens, one capability per route)
+
+Callers use an OpenVibe.Network client-credentials token for audience `openvibe.sources`.
+
+| Capability | Routes |
+|---|---|
+| `sources.source.read` | `GET /api/v1/sources`, `/api/v1/sources/:key`, `/api/v1/sources/:key/runs?before=`, `/api/v1/runs?state=failed&after=`, `/api/v1/health` |
+| `sources.item.read` | `GET /api/v1/items?source=&category=&after=<change_seq>&include_removed=1`, `GET /api/v1/items/:id?revisions=1` |
+| `sources.source.manage` | `POST /api/v1/sources`, `PATCH`/`DELETE /api/v1/sources/:key`, `POST /api/v1/sources/:key/fetch`, `POST /api/v1/sources/:key/items` (manual), `DELETE /api/v1/items/:id` `{reason}` |
+
+`GET /api/v1/items` pages in change order (creations, revisions and — with `include_removed=1` —
+removals); resume from `next_after`. Every page carries the status and staleness of the sources
+it contains. The capability ids are proposed in [docs/capabilities-proposal/](docs/capabilities-proposal/)
+and not in `openvibe-contracts` v0.7.0; [server/auth.js](server/auth.js) decides them with the
+contracts grant rule until a release defines them.
+
+## Events (transactional outbox → OpenVibe.Events when `EVENTS_URL` is set)
+
+- `sources.item.created`, `sources.item.updated` (with the previous content hash), `sources.item.removed` (with the reason)
+- `sources.fetch.failed` for `http_error`, `timeout`, `robots_denied`, `parse_error` and upstream 429 (with the state, error code, HTTP status and consecutive failures). Our own rate limiter and disabled sources are not failures.
+- `sources.index_document.upserted|deleted` for sources with `search_visibility: "members"`: raw items indexed in OpenVibe.Search for staff only (`acl.groups: role:admin, role:global_mod`), always `noindex` (`third_party_content`).
+
+All with `visibility: "internal"`. Envelopes validate as `events.event-envelope@1`.
+
+## Seeds
+
+[seeds/sources.json](seeds/sources.json) — one real source per category, **all disabled**, robots
+checked by hand on 2026-09-22 (the fetcher re-checks on every run), terms to be re-verified by a
+person before enabling:
+
+| Key | Category | Type | Endpoint |
+|---|---|---|---|
+| `nasa-news-releases` | news | rss | nasa.gov news-release feed |
+| `nodejs-blog` | blog | rss | nodejs.org blog feed |
+| `steam-reviews-portal-2` | reviews | api | Steam `appreviews` for app 620 — signals only, review text not mapped |
+| `dealnews-daily` | deals | rss | dealnews.com RSS (Crawl-delay 2) |
+| `staff-coupon-codes` | coupons | manual | codes a merchant published itself, entered with the evidence URL |
+| `sec-xbrl-filings` | trade | rss | SEC EDGAR XBRL filings RSS (`/cgi-bin` is robots-disallowed, so the getcurrent feed is not used) |
+
+Candidates rejected on 2026-09-22: CheapShark's API (`/api/1.0/` is disallowed in its
+robots.txt) and EDGAR's `getcurrent` Atom feed (`/cgi-bin` disallowed). The seeded feeds were
+downloaded once by hand and parsed offline with these adapters (NASA 10 items, Node.js 50,
+Steam 20, DealNews 50, SEC 200); that check is not part of the test suite and proves nothing about
+the terms.
+
+## Owns
+
+- the source registry (`sources`, `endpoint_state`, `robots_cache`) and the source/item contracts
+- ingestion: `fetch_runs`, `items`, `item_revisions`, the scheduler, robots and rate-limit state
+- `sources.*` events
+
+## Does not own
+
+- publication: what a product shows, summarises, rates or indexes from an item (News, Reviews,
+ Deals, Coupons, Trade, the publishing packages' indexability gate)
+- AI summarisation (OpenVibe.AI invokes adapters' output; it does not fetch here)
+- legal clearance: a terms note records what a person verified; it does not replace verifying
+
+## Depends on
+
+- OpenVibe.Contracts (service tokens, problem details, ids, the event envelope)
+- OpenVibe.Network (signing key; service principal `sources`)
+- OpenVibe.Events (outbound events)
+
+## Acceptance (tests)
+
+- failure never fabricates: HTTP errors, timeouts, parse errors, oversize bodies and 429s leave
+ every item byte-for-byte unchanged, each recorded with its state (`test/ingest.test.js`)
+- robots.txt respected, including redirects, unreachable = deny, Crawl-delay (`test/robots-ssrf.test.js`)
+- rate limit spacing and early triggers recorded as `rate_limited` without a request (`test/ingest.test.js`)
+- conditional GET with ETag and Last-Modified (`test/ingest.test.js`)
+- loopback/private/metadata addresses, DNS answers and redirects into them refused (`test/robots-ssrf.test.js`)
+- credentials by variable name, never stored, not forwarded cross-origin (`test/ingest.test.js`)
+- parsers against fixtures (`test/adapters.test.js`); API, registry rules, paging, manual items,
+ removal (`test/api.test.js`); scheduler, backoff, relay to Events (`test/scheduler-events.test.js`);
+ proposals and seeds (`test/proposals.test.js`)
+
+Not yet demonstrated: a run against a real source from the deployed service (every seed is disabled
+until a person verifies its terms), and a consuming product (News/Reviews/Deals/Coupons/Trade do
+not exist yet).
+
+## Launch rule
+
+The domain keeps its placeholder page on [OpenVibers/OpenVibe.Sites](https://github.com/OpenVibers/OpenVibe.Sites)
+until the plan's launch rule holds (owning runtime with health/readiness, identity and service
+principals, real persistence and workflows, capabilities and events registered in
+OpenVibe.Contracts, a security review, acceptance tests). Sources has no public pages by design;
+its public vhost answers health and a `Disallow: /` robots.txt.
+
+---
+
+Part of the [OpenVibe network](https://openvibe.network). Built in the open by [OpenVibers](https://github.com/OpenVibers).
new file mode 100644
new file mode 100644
@@ -0,0 +1,19 @@
@@ -0,0 +1,19 @@
+{
+ "repository": "OpenVibers/OpenVibe.Sources",
+ "stage": "alpha",
+ "code": true,
+ "wave": "W14",
+ "plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026)",
+ "runtime": "node 22, express, better-sqlite3, fast-xml-parser",
+ "port": 4720,
+ "domain": "sources.openvibe.network",
+ "contracts": "openvibe-contracts v0.7.0",
+ "capabilities": ["sources.source.read", "sources.item.read", "sources.source.manage"],
+ "capabilitiesRegistered": false,
+ "contractsProposed": ["sources.source@1", "sources.item@1"],
+ "adapters": ["rss", "atom", "sitemap", "jsonld", "api", "manual"],
+ "seededSources": 6,
+ "enabledSources": 0,
+ "deployed": false,
+ "updated": "2026-09-22"
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,75 @@
@@ -0,0 +1,75 @@
+## ═══════════════════════════════════════════════════════════════
+## sources.openvibe.network — Nginx Configuration (REFERENCE)
+##
+## Deployed to /etc/nginx/sites-available/sources.openvibe.network.conf — keep them in sync,
+## and reload nginx after changing it.
+##
+## The Node service (port 4720, systemd unit openvibe-sources) serves:
+## /api/health, /api/ready
+## /api/v1/* service-to-service API (every route needs a service token). Services on this
+## host call 127.0.0.1:4720 directly, so the public vhost refuses it; open the
+## location below only when a service on another host needs it.
+## Nothing here is a public page: Sources holds raw third-party material, which products
+## publish (or not) under their own rules.
+## ═══════════════════════════════════════════════════════════════
+
+# ── HTTP → HTTPS redirect ──────────────────────────────────
+server {
+ listen 80;
+ listen [::]:80;
+ server_name sources.openvibe.network;
+
+ location /.well-known/acme-challenge/ { root /var/www/certbot; }
+ location / { return 301 https://sources.openvibe.network$request_uri; }
+}
+
+# ── HTTPS server ───────────────────────────────────────────
+server {
+ listen 443 ssl;
+ listen [::]:443 ssl;
+ http2 on;
+ server_name sources.openvibe.network;
+
+ # The host has the openvibe.network wildcard certificate, not a per-name one.
+ ssl_certificate /etc/letsencrypt/live/openvibe.network/fullchain.pem;
+ ssl_certificate_key /etc/letsencrypt/live/openvibe.network/privkey.pem;
+ ssl_protocols TLSv1.2 TLSv1.3;
+
+ access_log /var/log/nginx/sources.openvibe.network.access.log;
+ error_log /var/log/nginx/sources.openvibe.network.error.log;
+
+ client_max_body_size 256k;
+
+ add_header X-Content-Type-Options nosniff always;
+ add_header Referrer-Policy strict-origin-when-cross-origin always;
+ add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
+ add_header X-Robots-Tag "noindex, nofollow" always;
+
+ location = /api/health { proxy_pass http://127.0.0.1:4720; proxy_set_header Host $host; }
+ location = /api/ready { proxy_pass http://127.0.0.1:4720; proxy_set_header Host $host; }
+
+ # ── Service API: host-local only (see header) ───────────
+ location /api/ {
+ return 403;
+ # To expose it (token-guarded) to off-host services instead:
+ # proxy_pass http://127.0.0.1:4720;
+ # proxy_http_version 1.1;
+ # proxy_set_header Host $host;
+ # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+ # proxy_set_header X-Forwarded-Proto $scheme;
+ }
+
+ location = / {
+ proxy_pass http://127.0.0.1:4720;
+ proxy_set_header Host $host;
+ }
+
+ location = /robots.txt {
+ default_type text/plain;
+ return 200 "User-agent: *\nDisallow: /\n";
+ }
+
+ location / {
+ return 404;
+ }
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,32 @@
@@ -0,0 +1,32 @@
+[Unit]
+Description=OpenVibe.Sources — source registry and ingestion workers (RSS/Atom, sitemaps, JSON-LD, APIs)
+# Network (openvibe-network) signs the tokens this service verifies. Sources starts without it
+# and keeps retrying the JWKS fetch; /api/ready stays 503 until the key has loaded.
+After=network.target
+Wants=openvibe-network.service
+
+[Service]
+Type=simple
+User=ubuntu
+Group=ubuntu
+WorkingDirectory=/opt/openvibe.sources
+EnvironmentFile=/etc/openvibe/sources.env
+Environment=NODE_ENV=production
+Environment=PORT=4720
+Environment=PATH=/usr/local/bin:/usr/bin:/bin
+ExecStart=/usr/bin/env node /opt/openvibe.sources/server/index.js
+Restart=always
+RestartSec=5
+# SIGTERM: the scheduler stops starting runs; runs in flight finish (each is bounded by the fetch timeout).
+KillSignal=SIGTERM
+TimeoutStopSec=30
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectSystem=strict
+ProtectHome=true
+# The code tree stays read-only; the registry and items live in /var/lib/openvibe-sources.
+StateDirectory=openvibe-sources
+Environment=SOURCES_DB_PATH=/var/lib/openvibe-sources/sources.db
+
+[Install]
+WantedBy=multi-user.target
new file mode 100644
new file mode 100644
@@ -0,0 +1,21 @@
@@ -0,0 +1,21 @@
+{
+ "id": "sources.item.read",
+ "version": "1.0",
+ "owner": "sources",
+ "status": "active",
+ "visibility": "first-party",
+ "description": "Read source items with provenance (source, canonical URL, retrieval time, content and raw-body hashes, parser version, terms) in change order, and one item with its revisions. Each page states the staleness of the sources it contains.",
+ "outputSchema": "sources.item@1",
+ "permissions": [
+ "sources:read"
+ ],
+ "resourceConstraints": [
+ "none"
+ ],
+ "quotaClass": "sources-read",
+ "events": [],
+ "implementedBy": [
+ "GET /api/v1/items",
+ "GET /api/v1/items/:id"
+ ]
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,28 @@
@@ -0,0 +1,28 @@
+{
+ "id": "sources.source.manage",
+ "version": "1.0",
+ "owner": "sources",
+ "status": "active",
+ "visibility": "first-party",
+ "description": "Create, change, enable or disable and delete (only while it has no items) sources; trigger a fetch; add items to manual sources with an evidence URL; remove an item with a reason (sticky). Enabling needs a terms note, and a robots note for fetched sources.",
+ "permissions": [
+ "sources:admin"
+ ],
+ "resourceConstraints": [
+ "none"
+ ],
+ "quotaClass": "sources-admin",
+ "events": [
+ "sources.item.created",
+ "sources.item.updated",
+ "sources.item.removed",
+ "sources.fetch.failed"
+ ],
+ "implementedBy": [
+ "POST /api/v1/sources",
+ "PATCH|DELETE /api/v1/sources/:key",
+ "POST /api/v1/sources/:key/fetch",
+ "POST /api/v1/sources/:key/items",
+ "DELETE /api/v1/items/:id"
+ ]
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,24 @@
@@ -0,0 +1,24 @@
+{
+ "id": "sources.source.read",
+ "version": "1.0",
+ "owner": "sources",
+ "status": "active",
+ "visibility": "first-party",
+ "description": "List registered sources with health and staleness, their fetch runs (every attempt with an explicit state) and the health summary. Credential values are never part of any response; only the variable name is.",
+ "outputSchema": "sources.source@1",
+ "permissions": [
+ "sources:read"
+ ],
+ "resourceConstraints": [
+ "none"
+ ],
+ "quotaClass": "sources-read",
+ "events": [],
+ "implementedBy": [
+ "GET /api/v1/sources",
+ "GET /api/v1/sources/:key",
+ "GET /api/v1/sources/:key/runs",
+ "GET /api/v1/runs",
+ "GET /api/v1/health"
+ ]
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,38 @@
@@ -0,0 +1,38 @@
+# Proposal for OpenVibe.Contracts: `sources.source@1`, `sources.item@1`
+
+Wave 14 asks OpenVibe.Contracts for the source-adapter contract. Files to add in its next release:
+
+| File here | Goes to |
+|---|---|
+| [contracts/sources/source.v1.json](contracts/sources/source.v1.json) | `contracts/sources/source.v1.json` |
+| [contracts/sources/item.v1.json](contracts/sources/item.v1.json) | `contracts/sources/item.v1.json` |
+| [catalog-entries.json](catalog-entries.json) | two new entries in `contracts/catalog.json` |
+| [../capabilities-proposal/*.json](../capabilities-proposal/) | `manifests/capabilities/` |
+| [../service-manifest-proposal.json](../service-manifest-proposal.json) | `manifests/services/sources.json` |
+
+`test/proposals.test.js` checks that the API's source and item views validate against these
+schemas, that the capability and service manifests validate against the contracts' own
+schemas, and that the seeds are valid.
+
+## Events
+
+| Type | Subject | Payload |
+|---|---|---|
+| `sources.item.created` | `{type: item, id, revision}` | item_id, source_key, category, kind, canonical_url, title, revision, content_hash, retrieved_at |
+| `sources.item.updated` | same | + previous_content_hash |
+| `sources.item.removed` | same | + reason |
+| `sources.fetch.failed` | `{type: source, id: <key>}` | source_key, category, run_id, endpoint_url, state, error_code, http_status, consecutive_failures |
+| `sources.index_document.upserted` | `{type: item, id, revision}` | a `search.index-document@1` (staff-only, noindex) |
+| `sources.index_document.deleted` | same | `{type, id, revision}` |
+
+OpenVibe.Events' default source prefixes do not include `sources` yet (`EVENTS_SOURCE_PREFIXES`
+or a default-list change is needed before these publish).
+
+## How products are expected to use items
+
+A product never shows an item as its own fact: it keeps a typed reference
+(`{service: 'sources', type: 'item', id, revision}`) as provenance, shows `retrieved_at` as the
+observation time (a price is "as of", never "current"), treats `null` as unknown, honours
+`terms_note`/`license_note`, starts its indexability decision from the source's
+`default_indexability`/`review_required`/`sensitivity`, and reacts to `sources.item.updated` /
+`removed` with a revision or a correction of its own.
new file mode 100644
new file mode 100644
@@ -0,0 +1,4 @@
@@ -0,0 +1,4 @@
+[
+ { "id": "sources.source", "version": "1.0.0", "owner": "sources", "schema": "sources/source.v1.json", "visibility": "first-party", "compatibility": "backward", "status": "active" },
+ { "id": "sources.item", "version": "1.0.0", "owner": "sources", "schema": "sources/item.v1.json", "visibility": "first-party", "compatibility": "backward", "status": "active" }
+]
new file mode 100644
new file mode 100644
@@ -0,0 +1,46 @@
@@ -0,0 +1,46 @@
+{
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
+ "$id": "https://openvibe.network/contracts/sources/item.v1.json",
+ "title": "SourceItem",
+ "description": "One thing a source said, with its provenance: source, canonical URL, retrieval time, content hash, raw-body hash, parser version and the terms it was ingested under. A field the source did not state is null — never guessed. Items change only through successful fetches (a new revision per content change) or an explicit, audited removal.",
+ "type": "object",
+ "required": ["id", "source_key", "category", "kind", "identity", "canonical_url", "title", "summary", "authors", "published_at", "source_updated_at", "fields", "revision", "provenance", "removed"],
+ "additionalProperties": false,
+ "properties": {
+ "id": { "type": "string", "pattern": "^itm_[0-9A-HJKMNP-TV-Z]{26}$" },
+ "source_key": { "type": "string", "pattern": "^[a-z0-9][a-z0-9-]{1,63}$" },
+ "category": { "enum": ["news", "blog", "reviews", "deals", "coupons", "trade"] },
+ "kind": { "type": "string", "pattern": "^[a-z][a-z0-9_]{1,39}$", "description": "article | url | sitemap | product | offer | review | review_signal | record | coupon …" },
+ "identity": { "type": "string", "minLength": 1, "maxLength": 2048, "description": "The source's own id (guid, @id, API id) or the canonical URL." },
+ "canonical_url": { "type": ["string", "null"], "format": "uri" },
+ "title": { "type": ["string", "null"] },
+ "summary": { "type": ["string", "null"], "maxLength": 1000, "description": "Plain text, capped. Sources keeps metadata and short descriptions, not republishable full text." },
+ "authors": { "type": "array", "items": { "type": "string" } },
+ "published_at": { "type": ["string", "null"], "format": "date-time" },
+ "source_updated_at": { "type": ["string", "null"], "format": "date-time" },
+ "fields": { "type": "object", "description": "Adapter-specific parsed fields (offers, ratings, prices as stated, API extras)." },
+ "revision": { "type": "integer", "minimum": 1 },
+ "provenance": {
+ "type": "object",
+ "required": ["retrieved_at", "first_seen_at", "content_hash", "raw_body_hash", "parser_version", "fetch_run_id", "license_note", "terms_note", "entered_by"],
+ "additionalProperties": false,
+ "properties": {
+ "retrieved_at": { "type": "string", "format": "date-time", "description": "Last successful fetch that contained this item (observation time)." },
+ "first_seen_at": { "type": "string", "format": "date-time" },
+ "content_hash": { "type": "string", "pattern": "^[0-9a-f]{64}$" },
+ "raw_body_hash": { "type": ["string", "null"], "pattern": "^[0-9a-f]{64}$" },
+ "parser_version": { "type": "string", "pattern": "^[a-z]+@\\d+$" },
+ "fetch_run_id": { "type": ["string", "null"], "pattern": "^frn_[0-9A-HJKMNP-TV-Z]{26}$" },
+ "license_note": { "type": ["string", "null"] },
+ "terms_note": { "type": ["string", "null"] },
+ "entered_by": { "type": ["string", "null"], "description": "Principal that entered a manual item." }
+ }
+ },
+ "removed": {
+ "type": ["object", "null"], "additionalProperties": false, "required": ["at", "reason"],
+ "properties": { "at": { "type": "string", "format": "date-time" }, "reason": { "type": "string" } }
+ },
+ "change_seq": { "type": "integer", "minimum": 1 },
+ "revisions": { "type": "array" }
+ }
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,66 @@
@@ -0,0 +1,66 @@
+{
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
+ "$id": "https://openvibe.network/contracts/sources/source.v1.json",
+ "title": "Source",
+ "description": "A registered source (roadmap §4.2 B, Wave 14): where it is fetched from and on what terms. Credentials are referenced by environment-variable NAME only. A source's existence is never permission to fetch: robots and terms are verified before it is enabled, and the fetcher re-checks robots.txt on every run.",
+ "type": "object",
+ "required": ["key", "name", "type", "category", "endpoints", "auth", "min_interval_ms", "poll_interval_sec", "stale_after_sec", "max_items", "enabled", "review_required", "sensitivity", "default_indexability"],
+ "additionalProperties": false,
+ "properties": {
+ "key": { "type": "string", "pattern": "^[a-z0-9][a-z0-9-]{1,63}$" },
+ "name": { "type": "string", "minLength": 1, "maxLength": 200 },
+ "type": { "enum": ["rss", "atom", "sitemap", "jsonld", "api", "manual"] },
+ "category": { "enum": ["news", "blog", "reviews", "deals", "coupons", "trade"] },
+ "homepage_url": { "type": ["string", "null"], "format": "uri" },
+ "endpoints": {
+ "type": "array", "maxItems": 20,
+ "items": {
+ "type": "object", "required": ["url"], "additionalProperties": false,
+ "properties": {
+ "url": { "type": "string", "format": "uri", "pattern": "^https?://" },
+ "format": { "enum": ["json", "xml"] },
+ "items_path": { "type": "string" },
+ "item_kind": { "type": "string", "pattern": "^[a-z][a-z0-9_]{1,39}$" },
+ "fields": { "type": "object", "additionalProperties": { "type": "string" } },
+ "extra": { "type": "object", "additionalProperties": { "type": "string" } }
+ }
+ }
+ },
+ "auth": {
+ "type": "object", "required": ["mode"], "additionalProperties": false,
+ "properties": {
+ "mode": { "enum": ["none", "header", "bearer", "query"] },
+ "env": { "type": "string", "pattern": "^SOURCES_CRED_[A-Z0-9_]{2,60}$", "description": "Name of the environment variable holding the credential. Never a value." },
+ "header": { "type": "string", "pattern": "^[A-Za-z][A-Za-z0-9-]{0,63}$" },
+ "param": { "type": "string", "pattern": "^[A-Za-z0-9_.-]{1,64}$" }
+ }
+ },
+ "robots_note": { "type": ["string", "null"], "maxLength": 2000 },
+ "terms_note": { "type": ["string", "null"], "maxLength": 2000 },
+ "license_note": { "type": ["string", "null"], "maxLength": 2000 },
+ "min_interval_ms": { "type": "integer", "minimum": 0, "description": "Rate limit: the least time between two requests to this source (the per-host floor and robots Crawl-delay may raise it)." },
+ "poll_interval_sec": { "type": "integer", "minimum": 1 },
+ "stale_after_sec": { "type": "integer", "minimum": 1 },
+ "max_items": { "type": "integer", "minimum": 1 },
+ "enabled": { "type": "boolean" },
+ "review_required": { "type": "boolean", "description": "Products must review what they derive from this source before it can be indexable." },
+ "sensitivity": { "enum": ["none", "financial", "health", "political", "legal", "adult"] },
+ "default_indexability": { "enum": ["index", "noindex"], "description": "What a product's indexability gate starts from for content derived from this source." },
+ "search_visibility": { "type": ["string", "null"], "enum": ["members", null], "description": "members = raw items are indexed in OpenVibe.Search for staff only (never indexable)." },
+ "health": {
+ "type": "object", "additionalProperties": false,
+ "properties": {
+ "status": { "enum": ["healthy", "stale", "failing", "never_fetched", "disabled", "manual"] },
+ "stale": { "type": "boolean" },
+ "last_success_at": { "type": ["string", "null"], "format": "date-time" },
+ "last_run_at": { "type": ["string", "null"], "format": "date-time" },
+ "last_state": { "type": ["string", "null"] },
+ "consecutive_failures": { "type": "integer", "minimum": 0 },
+ "next_due_at": { "type": ["string", "null"], "format": "date-time" },
+ "stale_after_sec": { "type": "integer" }
+ }
+ },
+ "created_at": { "type": "string", "format": "date-time" },
+ "updated_at": { "type": "string", "format": "date-time" }
+ }
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,32 @@
@@ -0,0 +1,32 @@
+{
+ "version": "0.1.0",
+ "contractRanges": {
+ "openvibe-contracts": ">=0.7.0 <1.0.0"
+ },
+ "id": "sources",
+ "name": "OpenVibe.Sources",
+ "status": "alpha",
+ "repository": "OpenVibers/OpenVibe.Sources",
+ "domains": [
+ "sources.openvibe.network"
+ ],
+ "publicOrigin": "https://sources.openvibe.network",
+ "health": "/api/health",
+ "ready": "/api/ready",
+ "capabilities": [
+ "sources.source.read",
+ "sources.item.read",
+ "sources.source.manage"
+ ],
+ "eventsProduced": [
+ "sources.item.created",
+ "sources.item.updated",
+ "sources.item.removed",
+ "sources.fetch.failed",
+ "sources.index_document.upserted",
+ "sources.index_document.deleted"
+ ],
+ "eventsConsumed": [],
+ "namespacesOwned": [],
+ "notes": "Source registry and ingestion workers (RSS/Atom, sitemaps, JSON-LD, mapped official APIs, manual). Every item carries provenance; every fetch is a recorded run with an explicit state; failures never create or modify items."
+}