openvibers/openvibe.sources Commits

Contracts proposals (sources.source@1, sources.item@1), capability and service manifest proposals, README, STATUS alpha W14, env example, systemd unit, nginx vhost, CI on Node 22.22.1

OP OpenVibers committed 23 Sept 2026 · 65b893b621 · 14 files, +655 −0


Changes

ddiff --git a/.env.example b/.env.example
nnew file mode 100644
iindex 0000000..19459af
---- /dev/null
++++ b/.env.example
@@@ -0,0 +1,53 @@
++# ═══════════════════════════════════════════════════════════════
++# OpenVibe.Sources — Environment Configuration
++# Copy to .env and fill in values.
++# In production this lives at /etc/openvibe/sources.env (0600); the systemd unit sets
++# NODE_ENV, PORT and SOURCES_DB_PATH itself.
++# ═══════════════════════════════════════════════════════════════
++
++# Server
++PORT=4720
++HOST=127.0.0.1
++NODE_ENV=development
++#BASE_URL=https://sources.openvibe.network
++
++# Identity — OpenVibe.Network signs the service tokens (audience openvibe.sources) callers present.
++OV_NETWORK_URL=https://openvibe.network
++OV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000
++#OV_NETWORK_ISSUER=https://openvibe.network
++#OV_NETWORK_PUBLIC_KEY=
++
++# This service's own principal (svc:sources) — publishes sources.* events (events.event.publish).
++OV_OAUTH_CLIENT_ID=sources
++OV_OAUTH_CLIENT_SECRET=
++
++# Store (SQLite). Production: /var/lib/openvibe-sources/sources.db (systemd StateDirectory).
++SOURCES_DB_PATH=./data/sources.db
++
++# OpenVibe.Events. The outbox relay runs only when EVENTS_URL is set.
++#EVENTS_URL=http://127.0.0.1:4300
++#EVENTS_RELAY_INTERVAL_MS=2000
++
++# Ingestion worker (in-process scheduler)
++#SOURCES_WORKER=on
++#SOURCES_TICK_MS=5000
++#SOURCES_MAX_CONCURRENT=4
++#SOURCES_MAX_BACKOFF_MS=21600000
++
++# Fetching
++#SOURCES_USER_AGENT=OpenVibeSources/0.1.0 (+https://sources.openvibe.network; [email protected])
++#SOURCES_ROBOTS_AGENT=OpenVibeSources
++#SOURCES_FETCH_TIMEOUT_MS=20000
++#SOURCES_MAX_BYTES=5242880
++#SOURCES_MAX_REDIRECTS=5
++#SOURCES_ROBOTS_TTL_MS=86400000
++# Least time between two requests to one host, whatever a source says.
++#SOURCES_HOST_MIN_INTERVAL_MS=1000
++#SOURCES_ALLOWED_PORTS=80,443
++# Exact hosts/IPs allowed to resolve to private/loopback addresses. Leave EMPTY in production.
++#SOURCES_ALLOW_PRIVATE_HOSTS=
++#SOURCES_MAX_ITEMS_PER_FETCH=500
++
++# Source credentials: a registry entry names the variable (auth.env), never the value. Names
++# must start with SOURCES_CRED_. Example (none of the seeds needs one):
++#SOURCES_CRED_EXAMPLE_API_KEY=
ddiff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
nnew file mode 100644
iindex 0000000..4728fc0
---- /dev/null
++++ b/.github/workflows/ci.yml
@@@ -0,0 +1,19 @@
++name: ci
++on:
++ push:
++ branches: [main]
++ pull_request:
++jobs:
++ test:
++ runs-on: ubuntu-latest
++ steps:
++ - uses: actions/checkout@v4
++ - uses: actions/setup-node@v4
++ with:
++ node-version: 22.22.1 # production runs Node 22
++ cache: npm
++ - run: npm ci --no-audit --no-fund
++ - run: npm test
++ # Add once the sources service manifest and the sources.* capabilities ship in
++ # openvibe-contracts (proposals in docs/capabilities-proposal/ and docs/service-manifest-proposal.json):
++ # - run: npx openvibe-contracts-check --service sources --src server
ddiff --git a/README.md b/README.md
nnew file mode 100644
iindex 0000000..582e03a
---- /dev/null
++++ b/README.md
@@@ -0,0 +1,198 @@
++# OpenVibe.Sources
++
++> The source registry and the ingestion workers the publication products depend on: every item
++> carries where it came from, when it was retrieved and on what terms, and a failed fetch is a
++> recorded failure — never replaced by invented content.
++
++**Status:** alpha (roadmap Wave 14). Runs and is tested against stub sites; not deployed; the six seeded sources are **disabled**.
++**Domain:** `sources.openvibe.network` (health only on the public vhost; the API is host-local)
++**Plan:** OpenVibe End-to-End Realignment & Implementation Plan, revision 3 — roadmap §4.2 B, §15.12, §29, anti-goals 13, 26, 27.
++**License:** AGPL-3.0.
++
++## Purpose
++
++News, Reviews, Deals, Coupons, Trade (and AI summarisation) all need outside material. Sources
++owns the one place it enters the network: a registry of sources with their terms, and workers that
++fetch them politely and record exactly what happened. Products read items with provenance and
++decide — under their own review and indexability rules — what, if anything, to publish.
++
++## Running it
++
++```bash
++npm install
++cp .env.example .env
++npm run seed # six real sources, one per category, all disabled
++npm run dev # http://127.0.0.1:4720
++npm test # every test/*.test.js against local stub sites; no internet
++```
++
++Node 22 in production (`fnm exec --using=22.22.1 npm test`). Production: `/opt/openvibe.sources`,
++env `/etc/openvibe/sources.env`, unit [deploy/systemd/openvibe-sources.service](deploy/systemd/openvibe-sources.service),
++store `/var/lib/openvibe-sources/sources.db`, nginx [deploy/nginx/sources.openvibe.network.conf](deploy/nginx/sources.openvibe.network.conf).
++
++`GET /api/health` is liveness. `GET /api/ready` is 200 when the database answers, the Network key
++has loaded and the scheduler runs; it reports source counts by health status, runs in flight and
++the outbox backlog.
++
++## The registry
++
++A source records: `key`, `name`, `type` (`rss|atom|sitemap|jsonld|api|manual`), `category`
++(`news|blog|reviews|deals|coupons|trade`), `endpoints` (for `api`: a field mapping, no code per
++provider), `auth` (`none|header|bearer|query` with the **name** of an environment variable —
++names must start with `SOURCES_CRED_`, so an entry can never point at this service's own
++secrets), `robots_note`, `terms_note`, `license_note`, the rate limit (`min_interval_ms`),
++`poll_interval_sec`, `stale_after_sec`, `max_items`, `enabled`, `review_required`,
++`sensitivity`, `default_indexability` and `search_visibility`.
++
++A source cannot be enabled without a terms note (and a robots note if it is fetched): an adapter
++existing is not permission to ingest (anti-goal 26). A source with items cannot be deleted —
++disable it — so provenance stays resolvable. Proposed contract: [`sources.source@1`](docs/contracts-proposal/).
++
++## Ingestion
++
++An in-process scheduler starts runs for enabled sources whose poll is due (at most
++`SOURCES_MAX_CONCURRENT` at once, never two of one source). A run fetches each endpoint in turn:
++
++1. **Credential** read from the named variable at that moment (missing → `disabled`), sent only
++ to the configured origin (dropped on a cross-origin redirect), never stored or logged.
++2. **robots.txt** (RFC 9309) for the origin: our `OpenVibeSources` group or `*`, longest match,
++ `*`/`$`; 4xx = no restrictions, 429/5xx/unreachable = complete disallow; cached 24 h;
++ `Crawl-delay` honoured. Every redirect hop is checked too.
++3. **Spacing**: max(`SOURCES_HOST_MIN_INTERVAL_MS`, the source's `min_interval_ms`,
++ Crawl-delay) between requests to a host. A trigger before the source's interval has passed,
++ or before a `Retry-After`, records `rate_limited` and requests nothing.
++4. **Fetch**: address guard (http(s) only, ports 80/443, every resolved address must be public —
++ checked at connect time, so DNS rebinding cannot slip through; loopback, RFC 1918, link-local,
++ CGNAT, metadata and v4-mapped/NAT64/6to4 forms are refused), manual redirects (≤ 5, each
++ re-checked), conditional GET (`If-None-Match`/`If-Modified-Since`), one deadline for the whole
++ response (`SOURCES_FETCH_TIMEOUT_MS`), a byte cap on the decoded body (`SOURCES_MAX_BYTES`).
++5. **Parse** with the adapter; on success, in one transaction: the run row, the items (created /
++ new revision if the content hash changed / retrieval time refreshed if unchanged), the
++ validators, and the events.
++
++Every attempt writes a `fetch_runs` row with an explicit state: `ok`, `not_modified`,
++`http_error`, `timeout`, `robots_denied`, `parse_error`, `rate_limited` or `disabled`, plus the
++HTTP status, an error code, the raw-body hash and item counts. **A failed run never creates or
++modifies an item** (only the `ok` path touches items), and ETag/Last-Modified are stored only after
++a good parse, so an unreadable body is re-fetched in full next time. Failing sources back off
++exponentially (capped at `SOURCES_MAX_BACKOFF_MS`).
++
++**Staleness** is computed: a fetched source is stale when its last success (`ok` or
++`not_modified`) is older than `stale_after_sec` (default 3 × the poll interval) or it never
++succeeded. Health status: `healthy | stale | failing | never_fetched | disabled | manual`.
++
++### Adapters (parser versions recorded on every item)
++
++| Type | Reads | Notes |
++|---|---|---|
++| `rss`, `atom` (`feed@1`) | RSS 2.0, RSS 1.0/RDF, Atom 1.0 | guid/id identity, else link; text-only summary capped at 1000 chars |
++| `sitemap` (`sitemap@1`) | `<urlset>` (+ Google News extension), `<sitemapindex>` | index children are listed, not followed |
++| `jsonld` (`jsonld@1`) | schema.org Product, Offer/AggregateOffer, Review, NewsArticle, BlogPosting (+ Article) | prices, ratings and availability exactly as the page states; missing = null; embedded review text not kept; a page without JSON-LD is a `parse_error` |
++| `api` (`api@1`) | JSON or XML official APIs via a registry mapping | `items_path`, `fields`, `extra`; epoch fields only where mapped as such |
++| `manual` (`manual@1`) | items entered through the API with an evidence URL | no fetcher |
++
++XML with DTD entity declarations is refused; the parser expands no entities (values are decoded
++per field). Fixtures and expectations: `test/adapters.test.js`.
++
++### Items
++
++Each item: `source_key`, `category`, `kind`, `identity`, `canonical_url` (tracking parameters and
++fragments removed), `title`, `summary`, `authors`, `published_at`, `source_updated_at`, `fields`,
++`revision`, and `provenance` (`retrieved_at`, `first_seen_at`, `content_hash`, `raw_body_hash`,
++`parser_version`, `fetch_run_id`, `license_note`, `terms_note`, `entered_by`). Earlier revisions
++are kept in `item_revisions`. Removal (takedown, licence) is explicit, needs a reason, and is
++sticky: a later fetch that still lists the item does not bring it back. Proposed contract:
++[`sources.item@1`](docs/contracts-proposal/).
++
++## API (service tokens, one capability per route)
++
++Callers use an OpenVibe.Network client-credentials token for audience `openvibe.sources`.
++
++| Capability | Routes |
++|---|---|
++| `sources.source.read` | `GET /api/v1/sources`, `/api/v1/sources/:key`, `/api/v1/sources/:key/runs?before=`, `/api/v1/runs?state=failed&after=`, `/api/v1/health` |
++| `sources.item.read` | `GET /api/v1/items?source=&category=&after=<change_seq>&include_removed=1`, `GET /api/v1/items/:id?revisions=1` |
++| `sources.source.manage` | `POST /api/v1/sources`, `PATCH`/`DELETE /api/v1/sources/:key`, `POST /api/v1/sources/:key/fetch`, `POST /api/v1/sources/:key/items` (manual), `DELETE /api/v1/items/:id` `{reason}` |
++
++`GET /api/v1/items` pages in change order (creations, revisions and — with `include_removed=1` —
++removals); resume from `next_after`. Every page carries the status and staleness of the sources
++it contains. The capability ids are proposed in [docs/capabilities-proposal/](docs/capabilities-proposal/)
++and not in `openvibe-contracts` v0.7.0; [server/auth.js](server/auth.js) decides them with the
++contracts grant rule until a release defines them.
++
++## Events (transactional outbox → OpenVibe.Events when `EVENTS_URL` is set)
++
++- `sources.item.created`, `sources.item.updated` (with the previous content hash), `sources.item.removed` (with the reason)
++- `sources.fetch.failed` for `http_error`, `timeout`, `robots_denied`, `parse_error` and upstream 429 (with the state, error code, HTTP status and consecutive failures). Our own rate limiter and disabled sources are not failures.
++- `sources.index_document.upserted|deleted` for sources with `search_visibility: "members"`: raw items indexed in OpenVibe.Search for staff only (`acl.groups: role:admin, role:global_mod`), always `noindex` (`third_party_content`).
++
++All with `visibility: "internal"`. Envelopes validate as `events.event-envelope@1`.
++
++## Seeds
++
++[seeds/sources.json](seeds/sources.json) — one real source per category, **all disabled**, robots
++checked by hand on 2026-09-22 (the fetcher re-checks on every run), terms to be re-verified by a
++person before enabling:
++
++| Key | Category | Type | Endpoint |
++|---|---|---|---|
++| `nasa-news-releases` | news | rss | nasa.gov news-release feed |
++| `nodejs-blog` | blog | rss | nodejs.org blog feed |
++| `steam-reviews-portal-2` | reviews | api | Steam `appreviews` for app 620 — signals only, review text not mapped |
++| `dealnews-daily` | deals | rss | dealnews.com RSS (Crawl-delay 2) |
++| `staff-coupon-codes` | coupons | manual | codes a merchant published itself, entered with the evidence URL |
++| `sec-xbrl-filings` | trade | rss | SEC EDGAR XBRL filings RSS (`/cgi-bin` is robots-disallowed, so the getcurrent feed is not used) |
++
++Candidates rejected on 2026-09-22: CheapShark's API (`/api/1.0/` is disallowed in its
++robots.txt) and EDGAR's `getcurrent` Atom feed (`/cgi-bin` disallowed). The seeded feeds were
++downloaded once by hand and parsed offline with these adapters (NASA 10 items, Node.js 50,
++Steam 20, DealNews 50, SEC 200); that check is not part of the test suite and proves nothing about
++the terms.
++
++## Owns
++
++- the source registry (`sources`, `endpoint_state`, `robots_cache`) and the source/item contracts
++- ingestion: `fetch_runs`, `items`, `item_revisions`, the scheduler, robots and rate-limit state
++- `sources.*` events
++
++## Does not own
++
++- publication: what a product shows, summarises, rates or indexes from an item (News, Reviews,
++ Deals, Coupons, Trade, the publishing packages' indexability gate)
++- AI summarisation (OpenVibe.AI invokes adapters' output; it does not fetch here)
++- legal clearance: a terms note records what a person verified; it does not replace verifying
++
++## Depends on
++
++- OpenVibe.Contracts (service tokens, problem details, ids, the event envelope)
++- OpenVibe.Network (signing key; service principal `sources`)
++- OpenVibe.Events (outbound events)
++
++## Acceptance (tests)
++
++- failure never fabricates: HTTP errors, timeouts, parse errors, oversize bodies and 429s leave
++ every item byte-for-byte unchanged, each recorded with its state (`test/ingest.test.js`)
++- robots.txt respected, including redirects, unreachable = deny, Crawl-delay (`test/robots-ssrf.test.js`)
++- rate limit spacing and early triggers recorded as `rate_limited` without a request (`test/ingest.test.js`)
++- conditional GET with ETag and Last-Modified (`test/ingest.test.js`)
++- loopback/private/metadata addresses, DNS answers and redirects into them refused (`test/robots-ssrf.test.js`)
++- credentials by variable name, never stored, not forwarded cross-origin (`test/ingest.test.js`)
++- parsers against fixtures (`test/adapters.test.js`); API, registry rules, paging, manual items,
++ removal (`test/api.test.js`); scheduler, backoff, relay to Events (`test/scheduler-events.test.js`);
++ proposals and seeds (`test/proposals.test.js`)
++
++Not yet demonstrated: a run against a real source from the deployed service (every seed is disabled
++until a person verifies its terms), and a consuming product (News/Reviews/Deals/Coupons/Trade do
++not exist yet).
++
++## Launch rule
++
++The domain keeps its placeholder page on [OpenVibers/OpenVibe.Sites](https://github.com/OpenVibers/OpenVibe.Sites)
++until the plan's launch rule holds (owning runtime with health/readiness, identity and service
++principals, real persistence and workflows, capabilities and events registered in
++OpenVibe.Contracts, a security review, acceptance tests). Sources has no public pages by design;
++its public vhost answers health and a `Disallow: /` robots.txt.
++
++---
++
++Part of the [OpenVibe network](https://openvibe.network). Built in the open by [OpenVibers](https://github.com/OpenVibers).
ddiff --git a/STATUS.json b/STATUS.json
nnew file mode 100644
iindex 0000000..6e9b5c3
---- /dev/null
++++ b/STATUS.json
@@@ -0,0 +1,19 @@
++{
++ "repository": "OpenVibers/OpenVibe.Sources",
++ "stage": "alpha",
++ "code": true,
++ "wave": "W14",
++ "plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026)",
++ "runtime": "node 22, express, better-sqlite3, fast-xml-parser",
++ "port": 4720,
++ "domain": "sources.openvibe.network",
++ "contracts": "openvibe-contracts v0.7.0",
++ "capabilities": ["sources.source.read", "sources.item.read", "sources.source.manage"],
++ "capabilitiesRegistered": false,
++ "contractsProposed": ["sources.source@1", "sources.item@1"],
++ "adapters": ["rss", "atom", "sitemap", "jsonld", "api", "manual"],
++ "seededSources": 6,
++ "enabledSources": 0,
++ "deployed": false,
++ "updated": "2026-09-22"
++}
ddiff --git a/deploy/nginx/sources.openvibe.network.conf b/deploy/nginx/sources.openvibe.network.conf
nnew file mode 100644
iindex 0000000..4a0240e
---- /dev/null
++++ b/deploy/nginx/sources.openvibe.network.conf
@@@ -0,0 +1,75 @@
++## ═══════════════════════════════════════════════════════════════
++## sources.openvibe.network — Nginx Configuration (REFERENCE)
++##
++## Deployed to /etc/nginx/sites-available/sources.openvibe.network.conf — keep them in sync,
++## and reload nginx after changing it.
++##
++## The Node service (port 4720, systemd unit openvibe-sources) serves:
++## /api/health, /api/ready
++## /api/v1/* service-to-service API (every route needs a service token). Services on this
++## host call 127.0.0.1:4720 directly, so the public vhost refuses it; open the
++## location below only when a service on another host needs it.
++## Nothing here is a public page: Sources holds raw third-party material, which products
++## publish (or not) under their own rules.
++## ═══════════════════════════════════════════════════════════════
++
++# ── HTTP → HTTPS redirect ──────────────────────────────────
++server {
++ listen 80;
++ listen [::]:80;
++ server_name sources.openvibe.network;
++
++ location /.well-known/acme-challenge/ { root /var/www/certbot; }
++ location / { return 301 https://sources.openvibe.network$request_uri; }
++}
++
++# ── HTTPS server ───────────────────────────────────────────
++server {
++ listen 443 ssl;
++ listen [::]:443 ssl;
++ http2 on;
++ server_name sources.openvibe.network;
++
++ # The host has the openvibe.network wildcard certificate, not a per-name one.
++ ssl_certificate /etc/letsencrypt/live/openvibe.network/fullchain.pem;
++ ssl_certificate_key /etc/letsencrypt/live/openvibe.network/privkey.pem;
++ ssl_protocols TLSv1.2 TLSv1.3;
++
++ access_log /var/log/nginx/sources.openvibe.network.access.log;
++ error_log /var/log/nginx/sources.openvibe.network.error.log;
++
++ client_max_body_size 256k;
++
++ add_header X-Content-Type-Options nosniff always;
++ add_header Referrer-Policy strict-origin-when-cross-origin always;
++ add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
++ add_header X-Robots-Tag "noindex, nofollow" always;
++
++ location = /api/health { proxy_pass http://127.0.0.1:4720; proxy_set_header Host $host; }
++ location = /api/ready { proxy_pass http://127.0.0.1:4720; proxy_set_header Host $host; }
++
++ # ── Service API: host-local only (see header) ───────────
++ location /api/ {
++ return 403;
++ # To expose it (token-guarded) to off-host services instead:
++ # proxy_pass http://127.0.0.1:4720;
++ # proxy_http_version 1.1;
++ # proxy_set_header Host $host;
++ # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
++ # proxy_set_header X-Forwarded-Proto $scheme;
++ }
++
++ location = / {
++ proxy_pass http://127.0.0.1:4720;
++ proxy_set_header Host $host;
++ }
++
++ location = /robots.txt {
++ default_type text/plain;
++ return 200 "User-agent: *\nDisallow: /\n";
++ }
++
++ location / {
++ return 404;
++ }
++}
ddiff --git a/deploy/systemd/openvibe-sources.service b/deploy/systemd/openvibe-sources.service
nnew file mode 100644
iindex 0000000..22356bf
---- /dev/null
++++ b/deploy/systemd/openvibe-sources.service
@@@ -0,0 +1,32 @@
++[Unit]
++Description=OpenVibe.Sources — source registry and ingestion workers (RSS/Atom, sitemaps, JSON-LD, APIs)
++# Network (openvibe-network) signs the tokens this service verifies. Sources starts without it
++# and keeps retrying the JWKS fetch; /api/ready stays 503 until the key has loaded.
++After=network.target
++Wants=openvibe-network.service
++
++[Service]
++Type=simple
++User=ubuntu
++Group=ubuntu
++WorkingDirectory=/opt/openvibe.sources
++EnvironmentFile=/etc/openvibe/sources.env
++Environment=NODE_ENV=production
++Environment=PORT=4720
++Environment=PATH=/usr/local/bin:/usr/bin:/bin
++ExecStart=/usr/bin/env node /opt/openvibe.sources/server/index.js
++Restart=always
++RestartSec=5
++# SIGTERM: the scheduler stops starting runs; runs in flight finish (each is bounded by the fetch timeout).
++KillSignal=SIGTERM
++TimeoutStopSec=30
++NoNewPrivileges=true
++PrivateTmp=true
++ProtectSystem=strict
++ProtectHome=true
++# The code tree stays read-only; the registry and items live in /var/lib/openvibe-sources.
++StateDirectory=openvibe-sources
++Environment=SOURCES_DB_PATH=/var/lib/openvibe-sources/sources.db
++
++[Install]
++WantedBy=multi-user.target
ddiff --git a/docs/capabilities-proposal/sources.item.read.json b/docs/capabilities-proposal/sources.item.read.json
nnew file mode 100644
iindex 0000000..ef1c497
---- /dev/null
++++ b/docs/capabilities-proposal/sources.item.read.json
@@@ -0,0 +1,21 @@
++{
++ "id": "sources.item.read",
++ "version": "1.0",
++ "owner": "sources",
++ "status": "active",
++ "visibility": "first-party",
++ "description": "Read source items with provenance (source, canonical URL, retrieval time, content and raw-body hashes, parser version, terms) in change order, and one item with its revisions. Each page states the staleness of the sources it contains.",
++ "outputSchema": "sources.item@1",
++ "permissions": [
++ "sources:read"
++ ],
++ "resourceConstraints": [
++ "none"
++ ],
++ "quotaClass": "sources-read",
++ "events": [],
++ "implementedBy": [
++ "GET /api/v1/items",
++ "GET /api/v1/items/:id"
++ ]
++}
ddiff --git a/docs/capabilities-proposal/sources.source.manage.json b/docs/capabilities-proposal/sources.source.manage.json
nnew file mode 100644
iindex 0000000..0f2c092
---- /dev/null
++++ b/docs/capabilities-proposal/sources.source.manage.json
@@@ -0,0 +1,28 @@
++{
++ "id": "sources.source.manage",
++ "version": "1.0",
++ "owner": "sources",
++ "status": "active",
++ "visibility": "first-party",
++ "description": "Create, change, enable or disable and delete (only while it has no items) sources; trigger a fetch; add items to manual sources with an evidence URL; remove an item with a reason (sticky). Enabling needs a terms note, and a robots note for fetched sources.",
++ "permissions": [
++ "sources:admin"
++ ],
++ "resourceConstraints": [
++ "none"
++ ],
++ "quotaClass": "sources-admin",
++ "events": [
++ "sources.item.created",
++ "sources.item.updated",
++ "sources.item.removed",
++ "sources.fetch.failed"
++ ],
++ "implementedBy": [
++ "POST /api/v1/sources",
++ "PATCH|DELETE /api/v1/sources/:key",
++ "POST /api/v1/sources/:key/fetch",
++ "POST /api/v1/sources/:key/items",
++ "DELETE /api/v1/items/:id"
++ ]
++}
ddiff --git a/docs/capabilities-proposal/sources.source.read.json b/docs/capabilities-proposal/sources.source.read.json
nnew file mode 100644
iindex 0000000..f6c7b54
---- /dev/null
++++ b/docs/capabilities-proposal/sources.source.read.json
@@@ -0,0 +1,24 @@
++{
++ "id": "sources.source.read",
++ "version": "1.0",
++ "owner": "sources",
++ "status": "active",
++ "visibility": "first-party",
++ "description": "List registered sources with health and staleness, their fetch runs (every attempt with an explicit state) and the health summary. Credential values are never part of any response; only the variable name is.",
++ "outputSchema": "sources.source@1",
++ "permissions": [
++ "sources:read"
++ ],
++ "resourceConstraints": [
++ "none"
++ ],
++ "quotaClass": "sources-read",
++ "events": [],
++ "implementedBy": [
++ "GET /api/v1/sources",
++ "GET /api/v1/sources/:key",
++ "GET /api/v1/sources/:key/runs",
++ "GET /api/v1/runs",
++ "GET /api/v1/health"
++ ]
++}
ddiff --git a/docs/contracts-proposal/README.md b/docs/contracts-proposal/README.md
nnew file mode 100644
iindex 0000000..15187a5
---- /dev/null
++++ b/docs/contracts-proposal/README.md
@@@ -0,0 +1,38 @@
++# Proposal for OpenVibe.Contracts: `sources.source@1`, `sources.item@1`
++
++Wave 14 asks OpenVibe.Contracts for the source-adapter contract. Files to add in its next release:
++
++| File here | Goes to |
++|---|---|
++| [contracts/sources/source.v1.json](contracts/sources/source.v1.json) | `contracts/sources/source.v1.json` |
++| [contracts/sources/item.v1.json](contracts/sources/item.v1.json) | `contracts/sources/item.v1.json` |
++| [catalog-entries.json](catalog-entries.json) | two new entries in `contracts/catalog.json` |
++| [../capabilities-proposal/*.json](../capabilities-proposal/) | `manifests/capabilities/` |
++| [../service-manifest-proposal.json](../service-manifest-proposal.json) | `manifests/services/sources.json` |
++
++`test/proposals.test.js` checks that the API's source and item views validate against these
++schemas, that the capability and service manifests validate against the contracts' own
++schemas, and that the seeds are valid.
++
++## Events
++
++| Type | Subject | Payload |
++|---|---|---|
++| `sources.item.created` | `{type: item, id, revision}` | item_id, source_key, category, kind, canonical_url, title, revision, content_hash, retrieved_at |
++| `sources.item.updated` | same | + previous_content_hash |
++| `sources.item.removed` | same | + reason |
++| `sources.fetch.failed` | `{type: source, id: <key>}` | source_key, category, run_id, endpoint_url, state, error_code, http_status, consecutive_failures |
++| `sources.index_document.upserted` | `{type: item, id, revision}` | a `search.index-document@1` (staff-only, noindex) |
++| `sources.index_document.deleted` | same | `{type, id, revision}` |
++
++OpenVibe.Events' default source prefixes do not include `sources` yet (`EVENTS_SOURCE_PREFIXES`
++or a default-list change is needed before these publish).
++
++## How products are expected to use items
++
++A product never shows an item as its own fact: it keeps a typed reference
++(`{service: 'sources', type: 'item', id, revision}`) as provenance, shows `retrieved_at` as the
++observation time (a price is "as of", never "current"), treats `null` as unknown, honours
++`terms_note`/`license_note`, starts its indexability decision from the source's
++`default_indexability`/`review_required`/`sensitivity`, and reacts to `sources.item.updated` /
++`removed` with a revision or a correction of its own.
ddiff --git a/docs/contracts-proposal/catalog-entries.json b/docs/contracts-proposal/catalog-entries.json
nnew file mode 100644
iindex 0000000..cd47c26
---- /dev/null
++++ b/docs/contracts-proposal/catalog-entries.json
@@@ -0,0 +1,4 @@
++[
++ { "id": "sources.source", "version": "1.0.0", "owner": "sources", "schema": "sources/source.v1.json", "visibility": "first-party", "compatibility": "backward", "status": "active" },
++ { "id": "sources.item", "version": "1.0.0", "owner": "sources", "schema": "sources/item.v1.json", "visibility": "first-party", "compatibility": "backward", "status": "active" }
++]
ddiff --git a/docs/contracts-proposal/contracts/sources/item.v1.json b/docs/contracts-proposal/contracts/sources/item.v1.json
nnew file mode 100644
iindex 0000000..1d76322
---- /dev/null
++++ b/docs/contracts-proposal/contracts/sources/item.v1.json
@@@ -0,0 +1,46 @@
++{
++ "$schema": "https://json-schema.org/draft/2020-12/schema",
++ "$id": "https://openvibe.network/contracts/sources/item.v1.json",
++ "title": "SourceItem",
++ "description": "One thing a source said, with its provenance: source, canonical URL, retrieval time, content hash, raw-body hash, parser version and the terms it was ingested under. A field the source did not state is null — never guessed. Items change only through successful fetches (a new revision per content change) or an explicit, audited removal.",
++ "type": "object",
++ "required": ["id", "source_key", "category", "kind", "identity", "canonical_url", "title", "summary", "authors", "published_at", "source_updated_at", "fields", "revision", "provenance", "removed"],
++ "additionalProperties": false,
++ "properties": {
++ "id": { "type": "string", "pattern": "^itm_[0-9A-HJKMNP-TV-Z]{26}$" },
++ "source_key": { "type": "string", "pattern": "^[a-z0-9][a-z0-9-]{1,63}$" },
++ "category": { "enum": ["news", "blog", "reviews", "deals", "coupons", "trade"] },
++ "kind": { "type": "string", "pattern": "^[a-z][a-z0-9_]{1,39}$", "description": "article | url | sitemap | product | offer | review | review_signal | record | coupon …" },
++ "identity": { "type": "string", "minLength": 1, "maxLength": 2048, "description": "The source's own id (guid, @id, API id) or the canonical URL." },
++ "canonical_url": { "type": ["string", "null"], "format": "uri" },
++ "title": { "type": ["string", "null"] },
++ "summary": { "type": ["string", "null"], "maxLength": 1000, "description": "Plain text, capped. Sources keeps metadata and short descriptions, not republishable full text." },
++ "authors": { "type": "array", "items": { "type": "string" } },
++ "published_at": { "type": ["string", "null"], "format": "date-time" },
++ "source_updated_at": { "type": ["string", "null"], "format": "date-time" },
++ "fields": { "type": "object", "description": "Adapter-specific parsed fields (offers, ratings, prices as stated, API extras)." },
++ "revision": { "type": "integer", "minimum": 1 },
++ "provenance": {
++ "type": "object",
++ "required": ["retrieved_at", "first_seen_at", "content_hash", "raw_body_hash", "parser_version", "fetch_run_id", "license_note", "terms_note", "entered_by"],
++ "additionalProperties": false,
++ "properties": {
++ "retrieved_at": { "type": "string", "format": "date-time", "description": "Last successful fetch that contained this item (observation time)." },
++ "first_seen_at": { "type": "string", "format": "date-time" },
++ "content_hash": { "type": "string", "pattern": "^[0-9a-f]{64}$" },
++ "raw_body_hash": { "type": ["string", "null"], "pattern": "^[0-9a-f]{64}$" },
++ "parser_version": { "type": "string", "pattern": "^[a-z]+@\\d+$" },
++ "fetch_run_id": { "type": ["string", "null"], "pattern": "^frn_[0-9A-HJKMNP-TV-Z]{26}$" },
++ "license_note": { "type": ["string", "null"] },
++ "terms_note": { "type": ["string", "null"] },
++ "entered_by": { "type": ["string", "null"], "description": "Principal that entered a manual item." }
++ }
++ },
++ "removed": {
++ "type": ["object", "null"], "additionalProperties": false, "required": ["at", "reason"],
++ "properties": { "at": { "type": "string", "format": "date-time" }, "reason": { "type": "string" } }
++ },
++ "change_seq": { "type": "integer", "minimum": 1 },
++ "revisions": { "type": "array" }
++ }
++}
ddiff --git a/docs/contracts-proposal/contracts/sources/source.v1.json b/docs/contracts-proposal/contracts/sources/source.v1.json
nnew file mode 100644
iindex 0000000..9c62f53
---- /dev/null
++++ b/docs/contracts-proposal/contracts/sources/source.v1.json
@@@ -0,0 +1,66 @@
++{
++ "$schema": "https://json-schema.org/draft/2020-12/schema",
++ "$id": "https://openvibe.network/contracts/sources/source.v1.json",
++ "title": "Source",
++ "description": "A registered source (roadmap §4.2 B, Wave 14): where it is fetched from and on what terms. Credentials are referenced by environment-variable NAME only. A source's existence is never permission to fetch: robots and terms are verified before it is enabled, and the fetcher re-checks robots.txt on every run.",
++ "type": "object",
++ "required": ["key", "name", "type", "category", "endpoints", "auth", "min_interval_ms", "poll_interval_sec", "stale_after_sec", "max_items", "enabled", "review_required", "sensitivity", "default_indexability"],
++ "additionalProperties": false,
++ "properties": {
++ "key": { "type": "string", "pattern": "^[a-z0-9][a-z0-9-]{1,63}$" },
++ "name": { "type": "string", "minLength": 1, "maxLength": 200 },
++ "type": { "enum": ["rss", "atom", "sitemap", "jsonld", "api", "manual"] },
++ "category": { "enum": ["news", "blog", "reviews", "deals", "coupons", "trade"] },
++ "homepage_url": { "type": ["string", "null"], "format": "uri" },
++ "endpoints": {
++ "type": "array", "maxItems": 20,
++ "items": {
++ "type": "object", "required": ["url"], "additionalProperties": false,
++ "properties": {
++ "url": { "type": "string", "format": "uri", "pattern": "^https?://" },
++ "format": { "enum": ["json", "xml"] },
++ "items_path": { "type": "string" },
++ "item_kind": { "type": "string", "pattern": "^[a-z][a-z0-9_]{1,39}$" },
++ "fields": { "type": "object", "additionalProperties": { "type": "string" } },
++ "extra": { "type": "object", "additionalProperties": { "type": "string" } }
++ }
++ }
++ },
++ "auth": {
++ "type": "object", "required": ["mode"], "additionalProperties": false,
++ "properties": {
++ "mode": { "enum": ["none", "header", "bearer", "query"] },
++ "env": { "type": "string", "pattern": "^SOURCES_CRED_[A-Z0-9_]{2,60}$", "description": "Name of the environment variable holding the credential. Never a value." },
++ "header": { "type": "string", "pattern": "^[A-Za-z][A-Za-z0-9-]{0,63}$" },
++ "param": { "type": "string", "pattern": "^[A-Za-z0-9_.-]{1,64}$" }
++ }
++ },
++ "robots_note": { "type": ["string", "null"], "maxLength": 2000 },
++ "terms_note": { "type": ["string", "null"], "maxLength": 2000 },
++ "license_note": { "type": ["string", "null"], "maxLength": 2000 },
++ "min_interval_ms": { "type": "integer", "minimum": 0, "description": "Rate limit: the least time between two requests to this source (the per-host floor and robots Crawl-delay may raise it)." },
++ "poll_interval_sec": { "type": "integer", "minimum": 1 },
++ "stale_after_sec": { "type": "integer", "minimum": 1 },
++ "max_items": { "type": "integer", "minimum": 1 },
++ "enabled": { "type": "boolean" },
++ "review_required": { "type": "boolean", "description": "Products must review what they derive from this source before it can be indexable." },
++ "sensitivity": { "enum": ["none", "financial", "health", "political", "legal", "adult"] },
++ "default_indexability": { "enum": ["index", "noindex"], "description": "What a product's indexability gate starts from for content derived from this source." },
++ "search_visibility": { "type": ["string", "null"], "enum": ["members", null], "description": "members = raw items are indexed in OpenVibe.Search for staff only (never indexable)." },
++ "health": {
++ "type": "object", "additionalProperties": false,
++ "properties": {
++ "status": { "enum": ["healthy", "stale", "failing", "never_fetched", "disabled", "manual"] },
++ "stale": { "type": "boolean" },
++ "last_success_at": { "type": ["string", "null"], "format": "date-time" },
++ "last_run_at": { "type": ["string", "null"], "format": "date-time" },
++ "last_state": { "type": ["string", "null"] },
++ "consecutive_failures": { "type": "integer", "minimum": 0 },
++ "next_due_at": { "type": ["string", "null"], "format": "date-time" },
++ "stale_after_sec": { "type": "integer" }
++ }
++ },
++ "created_at": { "type": "string", "format": "date-time" },
++ "updated_at": { "type": "string", "format": "date-time" }
++ }
++}
ddiff --git a/docs/service-manifest-proposal.json b/docs/service-manifest-proposal.json
nnew file mode 100644
iindex 0000000..a488c7f
---- /dev/null
++++ b/docs/service-manifest-proposal.json
@@@ -0,0 +1,32 @@
++{
++ "version": "0.1.0",
++ "contractRanges": {
++ "openvibe-contracts": ">=0.7.0 <1.0.0"
++ },
++ "id": "sources",
++ "name": "OpenVibe.Sources",
++ "status": "alpha",
++ "repository": "OpenVibers/OpenVibe.Sources",
++ "domains": [
++ "sources.openvibe.network"
++ ],
++ "publicOrigin": "https://sources.openvibe.network",
++ "health": "/api/health",
++ "ready": "/api/ready",
++ "capabilities": [
++ "sources.source.read",
++ "sources.item.read",
++ "sources.source.manage"
++ ],
++ "eventsProduced": [
++ "sources.item.created",
++ "sources.item.updated",
++ "sources.item.removed",
++ "sources.fetch.failed",
++ "sources.index_document.upserted",
++ "sources.index_document.deleted"
++ ],
++ "eventsConsumed": [],
++ "namespacesOwned": [],
++ "notes": "Source registry and ingestion workers (RSS/Atom, sitemaps, JSON-LD, mapped official APIs, manual). Every item carries provenance; every fetch is a recorded run with an explicit state; failures never create or modify items."
++}