openvibers/openvibe.sources Commits
List your store on OpenVibe: merchant opt-in with DNS or meta proof, terms note, withdrawal (#136)
Changes
@@@ -29,6 +29,29 @@ OV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000OV_OAUTH_CLIENT_ID=sourcesOV_OAUTH_CLIENT_SECRET=++# Merchant opt-in pages: the Network OAuth client must be registered with the redirect URI below. BASE_URL decides the++# default; set OV_OAUTH_REDIRECT_URI explicitly if the public host differs. The session cookie is Secure in production.++#OV_OAUTH_REDIRECT_URI=https://sources.openvibe.network/merchant/callback++#OV_OAUTH_SCOPE=profile++#SOURCES_SESSION_SECRET=++#SOURCES_COOKIE_SECURE=1++# Pending proofs: recheck cadence (0 disables the worker pass), expiry, meta page cap, optional DNS servers.++#SOURCES_MERCHANT_RECHECK_MS=600000++#SOURCES_MERCHANT_PENDING_DAYS=14++#SOURCES_MERCHANT_META_MAX_BYTES=1048576++#SOURCES_MERCHANT_META_TIMEOUT_MS=10000++#SOURCES_MERCHANT_DNS_SERVERS=++# The source created after verification: crawl/feed spacing, polling, staleness and crawl limits.++#SOURCES_MERCHANT_CRAWL_MIN_INTERVAL_MS=1000++#SOURCES_MERCHANT_FEED_MIN_INTERVAL_MS=60000++#SOURCES_MERCHANT_POLL_INTERVAL_SEC=3600++#SOURCES_MERCHANT_CRAWL_STALE_AFTER_SEC=86400++#SOURCES_MERCHANT_FEED_STALE_AFTER_SEC=10800++#SOURCES_MERCHANT_MAX_ITEMS=1000++#SOURCES_MERCHANT_MAX_PAGES=5000++#SOURCES_MERCHANT_PAGES_PER_RUN=100++#SOURCES_MERCHANT_RECRAWL_SEC=86400++# Store (PostgreSQL, ADR-035), written by OpenVibe.Host roles/data/add-service.sh sources: the runtime role through# PgBouncer, the owner on a direct connection (migrations), and Valkey (shared limit counters).# Without DATABASE_URL, development uses an embedded PGlite database in data/pglite.@@@ -248,13 +248,15 @@ the endpoint's last good parse saw. At most 500 items are retired a run; the res## API (service tokens, one capability per route)--Callers use an OpenVibe.Network client-credentials token for audience `openvibe.sources`.++Callers use an OpenVibe.Network client-credentials token for audience `openvibe.sources`. The merchant routes also++accept a signed-in person's session (see [Merchant opt-in](#merchant-opt-in-list-your-store-on-openvibe)); a first-party++service calls them with its service token and `X-OV-Subject: usr_…`.| Capability | Routes ||---|---|--| `sources.source.read` | `GET /api/v1/sources`, `/api/v1/sources/:key`, `/api/v1/sources/:key/runs?before=`, `/api/v1/runs?state=failed&after=`, `/api/v1/health` |++| `sources.source.read` | `GET /api/v1/sources`, `/api/v1/sources/:key`, `/api/v1/sources/:key/runs?before=`, `/api/v1/runs?state=failed&after=`, `/api/v1/health`; `GET /api/v1/merchant/sources[/:id]` || `sources.item.read` | `GET /api/v1/items?source=&category=&after=<change_seq>&include_removed=1`, `GET /api/v1/items/:id?revisions=1` |--| `sources.source.manage` | `POST /api/v1/sources`, `PATCH`/`DELETE /api/v1/sources/:key`, `POST /api/v1/sources/:key/fetch`, `POST /api/v1/sources/:key/items` (manual), `DELETE /api/v1/items/:id` `{reason}` |++| `sources.source.manage` | `POST /api/v1/sources`, `PATCH`/`DELETE /api/v1/sources/:key`, `POST /api/v1/sources/:key/fetch`, `POST /api/v1/sources/:key/items` (manual), `DELETE /api/v1/items/:id` `{reason}`; `POST /api/v1/merchant/sources`, `/:id/verify`, `/:id/withdraw` |`GET /api/v1/items` pages in change order (creations, revisions and — with `include_removed=1` —removals); resume from `next_after`. Every page carries the status and staleness of the sources@@@ -275,6 +277,8 @@ problem+json `rate_limited` with `Retry-After`, one `[limits]` log line and| `POST`, `PATCH`, `DELETE /api/v1/sources[/:key]` | 30 / 300 || `POST /api/v1/sources/:key/fetch` | 10 / 100 || `POST /api/v1/sources/:key/items` (manual) | 60 / 1200 |++| `GET /api/v1/merchant/sources[/:id]` (a merchant or a service acting for one) | 120 / 3000 |++| `POST /api/v1/merchant/sources`, `/:id/verify`, `/:id/withdraw` | 30 / 300 || `DELETE /api/v1/items/:id` (a takedown may cover many items) | 120 / 3000 |A first-party service (`svc:…`) is not counted on reads: News, Reviews and Trade pull cursor pages@@@ -282,6 +286,46 @@ for all their readers, and Wiki asks about every item its editors cite. Sources'scheduler, fetches, the outbox relay) run in-process and never pass through HTTP. Never limited:`/api/health`, `/api/ready`, `/release.json`, `/metrics` and the home page. `test/actor-limits.test.js`.++## Merchant opt-in ("List your store on OpenVibe")++++The first permitted channel in plan §4.4. A signed-in merchant claims a domain, proves control of it, and names the++sitemap or product feed Sources may read. **Nothing is created in `sources` before the proof is found**: a pending claim++is only a `merchant_sources` row (migration `0007_merchant_sources.sql`). Verification creates one enabled source in the++same transaction, and withdrawal disables it while keeping the note.++++- **Proof.** One of two challenges, the OpenVibe.Space pattern:++ - DNS: `_openvibe-sources.<domain> TXT "openvibe-sources-verification=<token>"`.++ - Meta: `<meta name="openvibe-sources-verification" content="<token>">` on `https://<domain>/` (or the `meta_url` the++ merchant names, on the same site). Redirects off the domain are refused.++- **Fetched politely.** A sitemap becomes a `crawl` source seeded with that sitemap (no guessed sitemaps); a product feed++ becomes an `rss`/`atom` source. Either way the ordinary fetcher runs: robots.txt on every request and redirect hop,++ per-host spacing, SSRF guard, conditional GETs. Explicit permission is not a robots override.++- **The terms note** records the merchant subject (and display name when the token carries one), the UTC verification++ time, the domain, the named sitemap/feed, and how control was verified ("DNS TXT record at _openvibe-sources.…" or++ "meta tag at …"); it also says the merchant may withdraw and that robots.txt is still obeyed.++- **Withdrawal** sets the claim to `withdrawn`, disables the source, and leaves `terms_note` and the claim row intact.++ The domain is free to claim again (a new claim, a new source key). Unlike a *served* custom domain (Space's daily++ recheck), the permission persists until the merchant withdraws: removing the TXT/meta record later does not silently++ disable the source.++- **Pending proofs** are rechecked by a slow in-process pass (`SOURCES_MERCHANT_RECHECK_MS`, default 10 min; `0` turns++ it off) and fail after `SOURCES_MERCHANT_PENDING_DAYS` (14). A merchant can also press "Check now".++- **Pages.** `GET /merchant` (guest explanation, or the signed-in merchant's list and add form),++ `GET /merchant/sources/:id` (challenge, status, terms note, last crawl), the POST actions++ `/merchant/sources`, `/:id/verify`, `/:id/withdraw`, and `/merchant/login|callback|logout`. Forms carry an HMAC CSRF++ token; the session is the Sources `ov_token` cookie, verified offline with `verifyUserToken`.++- **JSON read for Report.** `GET /api/v1/merchant/sources` (alias `/claims`) returns each claim with its source, health,++ last run, recent runs and — for a crawl — `last_crawl` (`pages`, `due`, `next_due_at`, state counts, recent pages).++ Report uses its service token and `X-OV-Subject: usr_…`; a merchant's own session needs no extra header. The read is++ the contract for the Report dashboard (which is not built here).++- **Identity.** The OAuth client is `OV_OAUTH_CLIENT_ID` (default `sources`) with++ `OV_OAUTH_REDIRECT_URI` defaulting to `<BASE_URL>/merchant/callback`; Network must register that exact redirect. A++ first-party service token may act for a merchant only with `X-OV-Subject` (an app/module token cannot). The merchant++ routes reuse the released `sources.source.read` / `sources.source.manage` capabilities; no new contract id is needed.++- **Config.** `SOURCES_SESSION_SECRET` (defaults to `OV_OAUTH_CLIENT_SECRET`), `SOURCES_COOKIE_SECURE`,++ `SOURCES_MERCHANT_PENDING_DAYS`, `SOURCES_MERCHANT_RECHECK_MS`, `SOURCES_MERCHANT_META_MAX_BYTES`,++ `SOURCES_MERCHANT_DNS_SERVERS`, and the source-shape knobs `SOURCES_MERCHANT_*` documented in++ [.env.example](.env.example). Tests: `test/merchant-optin.test.js`, `test/merchant-pages.test.js`.++## Events (transactional outbox → OpenVibe.Events when `EVENTS_URL` is set)- `sources.item.created`, `sources.item.updated` (with the previous content hash), `sources.item.removed` (with the reason)@@@ -383,6 +427,7 @@ publishes another's output.## Owns- the source registry (`sources`, `endpoint_state`, `robots_cache`) and the source/item contracts++- the merchant opt-in claims (`merchant_sources`), their proof checks and the pages/JSON read- ingestion: `fetch_runs`, `items`, `item_revisions`, the scheduler, robots and rate-limit state- `sources.*` events@@@ -397,7 +442,7 @@ publishes another's output.- PostgreSQL 18 (OpenVibe.Host `roles/data/`, ADR-035): every read and write is async through `openvibe-sdk/db`- OpenVibe.Contracts (service tokens, problem details, ids, the event envelope)--- OpenVibe.Network (signing key; service principal `sources`)++- OpenVibe.Network (signing key; service principal `sources`; OAuth client `sources` for merchant sign-in)- OpenVibe.Events (outbound events)## Capabilities@@@ -427,8 +472,11 @@ through Events.- parsers against fixtures (`test/adapters.test.js`); API, registry rules, paging, manual items,removal (`test/api.test.js`); scheduler, backoff, relay to Events (`test/scheduler-events.test.js`);proposals and seeds (`test/proposals.test.js`)--- the public host shows only an honest "internal service" page, and the vhost keeps the API-- loopback-only (`test/public-host.test.js`)++- the public host shows only an honest "internal service" page and the merchant opt-in pages, and the vhost keeps the++ API loopback-only (`test/public-host.test.js`)++- merchant opt-in: no source before the DNS/meta proof, the terms note records who/when/how, robots.txt is still obeyed,++ withdrawal disables and keeps the note, and the JSON read carries last crawl stats (`test/merchant-optin.test.js`,++ `test/merchant-pages.test.js`)Production on 2026-10-22 (counts read from `ov_sources`):- 43 of 47 sources enabled, each after its terms and robots were read (by category in [Seeds](#seeds)); none of them@@@ -449,8 +497,10 @@ readiness, row counts; see OpenVibe.Host `docs/restore-drills.md`).Reporting: [SECURITY.md](SECURITY.md). The rules the code keeps:- **Auth.** Every `/api/v1` route needs an OpenVibe.Network service token for audience-- `openvibe.sources` with that route's capability, then passes the per-actor limits. There are no user-- sessions and no public API: the vhost keeps `/api/v1/*` loopback-only.++ `openvibe.sources` with that route's capability, then passes the per-actor limits. The merchant API and pages are the++ one person-facing exception: a Network person session (Bearer on the API, the first-party `ov_token` cookie on the++ pages) owns its claims, and a first-party service may act for a person only with `X-OV-Subject`. There is still no++ public JSON API: the vhost keeps `/api/v1/*` loopback-only, and only `/merchant*` and `/shared/*` are public.- **Egress.** Sources fetches outside URLs by design, only through its guard: http(s) on ports 80/443,every resolved address public at connect time (no DNS rebinding), each redirect re-checked, robots.txtobeyed, per-host spacing, one deadline and a byte cap per response (`test/robots-ssrf.test.js`,@@@ -488,6 +538,8 @@ the Network wildcard certificate and answers only:| Path | Answer ||---|---|| `/` | a short page: an internal ingestion service, nothing to browse (HTML for browsers, the text route index otherwise; noindex, no-store) |++| `/merchant`, `/merchant/*` | the merchant opt-in pages (guest explanation, signed-in claims, proof/withdraw forms; noindex, no-store) |++| `/shared/*` | this release's pinned `theme-loader.js`, `kit.css`, `kit-web.js` (served by `openvibe-shared/serve`) || `/api/health`, `/api/ready`, `/release.json` | proxied || `/robots.txt` | `Disallow: /` || `/metrics` | 404 (Prometheus scrapes `127.0.0.1:4720/metrics` directly) |@@@ -503,6 +555,8 @@ sudo ln -sf /etc/nginx/sites-available/sources.openvibe.network.conf /etc/nginx/sudo nginx -t && sudo systemctl reload nginx# checkcurl -sS -H 'Accept: text/html' https://sources.openvibe.network/ | grep -o '<title>[^<]*' # OpenVibe.Sources++curl -sS -o /dev/null -w '%{http_code}\n' https://sources.openvibe.network/merchant # 200++curl -sS -o /dev/null -w '%{http_code}\n' https://sources.openvibe.network/shared/kit.css # 200curl -sS https://sources.openvibe.network/api/healthcurl -sS -o /dev/null -w '%{http_code}\n' https://sources.openvibe.network/api/v1/sources # 403curl -sS -o /dev/null -w '%{http_code}\n' https://sources.openvibe.network/metrics # 404@@@ -512,9 +566,9 @@ curl -sS https://sources.openvibe.network/admin## Launch rule--Sources has no public pages by design, so it never "launches" as a site. The domain is not in--[OpenVibers/OpenVibe.Sites](https://github.com/OpenVibers/OpenVibe.Sites); this repository's own--vhost serves it.++Sources has no public product pages by design: the only person-facing pages are the merchant opt-in flow (`/merchant`),++which exist so a merchant can prove a domain and name a sitemap/feed. It never "launches" as a site. The domain is not++in [OpenVibers/OpenVibe.Sites](https://github.com/OpenVibers/OpenVibe.Sites); this repository's own vhost serves it.---@@@ -6,7 +6,7 @@"plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026)","runtime": "node 22, express, PostgreSQL 18 (openvibe-sdk/db, async) + Valkey 9, fast-xml-parser","port": 4720,-- "domain": "sources.openvibe.network (vhost in deploy/nginx: an internal-service page, health and JSON 404s; the API is loopback-only; README \"Public host\")",++ "domain": "sources.openvibe.network (vhost in deploy/nginx: an internal-service page, the merchant opt-in pages, health and JSON 404s; the JSON API is loopback-only; README \"Public host\")","contracts": "openvibe-contracts v0.224.0","sdk": "openvibe-sdk v0.67.0","shared": "openvibe-shared v4.33.0",@@@ -26,7 +26,8 @@"sitemap","jsonld","api",-- "manual"++ "manual",++ "crawl"],"seededSources": 47,"enabledSources": 43,@@@ -34,7 +35,8 @@"deployment": "Deployed with `sudo ovhost deploy sources`: /opt/openvibe.sources, unit openvibe-sources, 127.0.0.1:4720, principal sources. On 2026-10-22, 43 of the 47 sources were enabled in production (reviews 22, hosting 5, news 4, trade 4, games 3, coupons 2, and blog, crowdfunding and deals 1 each; none failing), with 55,683 items from 4,762 fetch runs since the first on 2026-10-07; they feed OpenVibe.News, Reviews, Deals (with its crowdfunding search), Coupons, Trade, Host, Games and Blog's desks. Restore drill passed 2026-09-23.","features": ["source registry with terms, robots and licence notes",-- "rss, atom, sitemap, jsonld, api and manual adapters",++ "rss, atom, sitemap, jsonld, api, manual and crawl adapters",++ "merchant opt-in: a signed-in merchant proves a domain by DNS TXT or meta tag, names a sitemap or product feed, and a source is created only on verification with the explicit permission in its terms note; withdrawal disables it and keeps the note; /api/v1/merchant/sources is the JSON read for Report","categories: news, blog, reviews, deals, coupons, trade, hosting, crowdfunding, games, and since contracts 0.210.0 housing, food and jobs (read from sources.source@1)","a deal feed's own offer elements kept as stated (feed@3: DealNews' price, currency, retailer, end date, category and editors' pick)","api mappings: keyed maps, where conditions and patterns, $index for ranked lists; per-endpoint timeout_ms and max_bytes",@@@ -4,14 +4,16 @@## Deployed to /etc/nginx/sites-available/sources.openvibe.network.conf — keep them in sync,## and reload nginx after changing it. Install: README "Public host".##--## Sources is an internal ingestion service with no public pages: Sources holds raw third-party--## material, which products publish (or not) under their own rules. The public host answers only:++## Sources is an internal ingestion service, with one person-facing surface: the merchant opt-in flow.++## Sources holds raw third-party material, which products publish (or not) under their own rules. The public host answers:## / a short page saying what the service is (HTML for browsers, text otherwise)++## /merchant* the merchant opt-in pages (signed-in merchants add, prove and withdraw a store)++## /shared/* this release's pinned kit.css, kit-web.js and theme-loader.js## /api/health, /api/ready, /release.json## /robots.txt disallow everything## and keeps the rest off the public edge:## /api/v1/* service-to-service API (every route needs a service token): loopback only.--## Services on this host call 127.0.0.1:4720 directly.++## Services on this host call 127.0.0.1:4720 directly; /api/v1/merchant/* is reachable there too.## /metrics 404 here; Prometheus text for direct loopback callers of 127.0.0.1:4720## anything else a JSON 404 that says why, so the name never falls through to another site## ═══════════════════════════════════════════════════════════════@@@ -77,6 +79,23 @@ server {return 200 "User-agent: *\nDisallow: /\n";}++ # ── Merchant opt-in pages (the one person-facing surface) ──++ # A signed-in merchant adds a domain, proves it with DNS or a meta tag, names a sitemap/feed, and may withdraw.++ # The Network OAuth client `sources` must be registered with https://sources.openvibe.network/merchant/callback.++ location = /merchant {++ limit_except GET POST { deny all; }++ proxy_pass http://127.0.0.1:4720;++ }++ location ^~ /merchant/ {++ limit_except GET POST { deny all; }++ proxy_pass http://127.0.0.1:4720;++ }++ # This release's pinned OpenVibe Kit and theme loader.++ location ^~ /shared/ {++ limit_except GET { deny all; }++ proxy_pass http://127.0.0.1:4720;++ }++# ── Service API: loopback only (see header) ─────────────location ^~ /api/v1/ {allow 127.0.0.1;@@@ -88,6 +107,6 @@ server {# ── Anything else: an honest JSON 404, never another site ──location / {default_type application/problem+json;-- return 404 '{"type":"about:blank","title":"Not Found","status":404,"code":"sources.not_found","detail":"sources.openvibe.network is an internal ingestion service of the OpenVibe network with no public pages or API. Its API is for OpenVibe services on the host. About: https://sources.openvibe.network/ and https://github.com/OpenVibers/OpenVibe.Sources"}';++ return 404 '{"type":"about:blank","title":"Not Found","status":404,"code":"sources.not_found","detail":"sources.openvibe.network is an internal ingestion service of the OpenVibe network. Its JSON API is for OpenVibe services on the host; the merchant opt-in pages are at /merchant. About: https://sources.openvibe.network/ and https://github.com/OpenVibers/OpenVibe.Sources"}';}}nnew file mode 100644@@@ -0,0 +1,44 @@++-- phase: expand++-- Merchant opt-in ("List your store on OpenVibe", plan §4.4 channel 1).++--++-- A signed-in merchant claims a domain, proves control of it with a DNS TXT record or a meta tag, and names the++-- sitemap or product feed Sources may read. Nothing is created in `sources` before the proof is present: a pending++-- claim is only this row. On verification the source is created enabled, and its terms_note records who gave the++-- explicit permission, when, and how control was verified. Withdrawal disables that source and keeps the note (and++-- this row) for the audit trail. The crawler and the fetcher still obey robots.txt on every request.++CREATE TABLE IF NOT EXISTS merchant_sources (++ id text COLLATE "C" PRIMARY KEY, -- msrc_<ULID>++ owner_sub text COLLATE "C" NOT NULL, -- usr_<ULID>, the signed-in merchant++ owner_name text COLLATE "C", -- display name the token carried, for people reading the note++ store_name text COLLATE "C",++ domain text COLLATE "C" NOT NULL, -- lowercase hostname, no port++ method text COLLATE "C" NOT NULL CHECK (method IN ('dns', 'meta')),++ token text COLLATE "C" NOT NULL, -- verification challenge, shown only to the owner++ listing_url text COLLATE "C" NOT NULL, -- the sitemap or product feed the merchant named++ listing_kind text COLLATE "C" NOT NULL CHECK (listing_kind IN ('sitemap', 'feed')),++ meta_url text COLLATE "C", -- where the meta tag lives (meta method only)++ status text COLLATE "C" NOT NULL DEFAULT 'pending'++ CHECK (status IN ('pending', 'verified', 'withdrawn', 'failed')),++ source_key text COLLATE "C", -- set only when verified++ terms_note text COLLATE "C", -- final explicit-permission note, kept after withdrawal++ verified_at bigint,++ verified_by text COLLATE "C", -- the actor that completed the check (the merchant or svc:sources)++ verification_detail text COLLATE "C", -- how the proof was found, without the challenge value++ last_checked_at bigint,++ check_count bigint NOT NULL DEFAULT 0,++ last_error text COLLATE "C",++ created_at bigint NOT NULL,++ updated_at bigint NOT NULL,++ withdrawn_at bigint,++ withdrawn_by text COLLATE "C"++);++++-- One live claim per domain: a withdrawn or failed claim frees the name for a new opt-in.++CREATE UNIQUE INDEX IF NOT EXISTS merchant_sources_live_domain++ ON merchant_sources (domain) WHERE status IN ('pending', 'verified');++CREATE UNIQUE INDEX IF NOT EXISTS merchant_sources_source_key++ ON merchant_sources (source_key) WHERE source_key IS NOT NULL;++CREATE INDEX IF NOT EXISTS merchant_sources_owner++ ON merchant_sources (owner_sub, created_at DESC);++CREATE INDEX IF NOT EXISTS merchant_sources_pending++ ON merchant_sources (last_checked_at) WHERE status = 'pending';@@@ -43,6 +43,10 @@ const BUDGETS = {// Creating, changing or deleting a source is a staff decision recorded with its terms and robots// notes: a form every few seconds at most.'sources.source.manage': { minute: 30, hour: 300 },++ // Merchant opt-in reads (the page and the JSON read Report calls): a person refreshing their small list.++ 'sources.merchant.read': { minute: 120, hour: 3000 },++ // Claiming a domain, checking the proof, withdrawing: a handful of decisions, not a polling loop.++ 'sources.merchant.manage': { minute: 30, hour: 300 },// A manual fetch requests every endpoint of the source now (the per-host spacing still applies):// the scheduler already polls each source, so a person's "fetch now" is a rare nudge.'sources.source.fetch': { minute: 10, hour: 100 },nnew file mode 100644@@@ -0,0 +1,141 @@++'use strict';++/**++ * Merchant opt-in JSON API.++ *++ * sources.source.read (or the merchant's own person token)++ * GET /api/v1/merchant/sources the merchant's claims and sources, with last run/crawl stats++ * GET /api/v1/merchant/sources/:id++ * sources.source.manage (or the merchant's own person token)++ * POST /api/v1/merchant/sources add a domain and name its sitemap/product feed (pending)++ * POST /api/v1/merchant/sources/:id/verify++ * POST /api/v1/merchant/sources/:id/withdraw++ *++ * `/api/v1/merchant/claims` is an alias for the same collection (a claim is a source before verification). A first-party++ * service token must name the merchant with X-OV-Subject; a person token needs nothing else. Report calls the read with++ * its service token and the signed-in merchant's subject. The merchant dashboard itself lives in Report, not here.++ */++const express = require('express');++const { http } = require('openvibe-contracts');++const { MerchantError, CHALLENGE_PREFIX, TXT_PREFIX, META_NAME } = require('../merchant');++const { CAPS } = require('../auth');++const { runView } = require('./sources');++++const COLLECTION = ['/api/v1/merchant/sources', '/api/v1/merchant/claims'];++const ITEM = ['/api/v1/merchant/sources/:id', '/api/v1/merchant/claims/:id'];++const VERIFY = ['/api/v1/merchant/sources/:id/verify', '/api/v1/merchant/claims/:id/verify'];++const WITHDRAW = ['/api/v1/merchant/sources/:id/withdraw', '/api/v1/merchant/claims/:id/withdraw'];++++function crawlSummary(crawl) {++ if (!crawl || crawl.not_crawl) return null;++ return {++ pages: crawl.pages,++ max_pages: crawl.max_pages,++ due: crawl.due,++ next_due_at: crawl.next_due_at,++ pages_by_state: crawl.pages_by_state,++ sitemaps_by_state: crawl.sitemaps_by_state,++ recent: (crawl.recent || []).slice(0, 5),++ };++}++++function createMerchantView({ db, merchant, registry, ingest }) {++ return async function decorate(row) {++ const out = { ...merchant.view(row), source: null, last_run: null, last_crawl: null, recent_runs: [] };++ if (!row.source_key) return out;++ const source = await registry.get(row.source_key);++ if (!source) return out;++ out.source = registry.view(source);++ const runs = await db.prepare('SELECT * FROM fetch_runs WHERE source_key = ? ORDER BY rid DESC LIMIT 5').all(source.key);++ out.recent_runs = runs.map(runView);++ out.last_run = out.recent_runs[0] || null;++ if (source.type === 'crawl') {++ try { out.last_crawl = crawlSummary(await ingest.crawlStatus(source.key)); } catch { out.last_crawl = null; }++ }++ out.stats = {++ health: out.source.health,++ last_run: out.last_run,++ last_crawl: out.last_crawl,++ recent_runs: out.recent_runs,++ };++ return out;++ };++}++++function merchantRouter({ db, merchant, registry, ingest, auth, merchantAuth, limits }) {++ const router = express.Router();++ const read = [merchantAuth.api(CAPS.read), limits.budget('sources.merchant.read')];++ const manage = [merchantAuth.api(CAPS.manage), limits.budget('sources.merchant.manage')];++ const decorate = createMerchantView({ db, merchant, registry, ingest });++++ const fail = (res, req, err) => {++ if (err instanceof MerchantError) return http.sendProblem(res, err.status, err.code, { detail: err.message, ctx: req.ov });++ if (err && typeof err.code === 'string' && err.code.startsWith('sources.')) {++ return http.sendProblem(res, 422, err.code, { detail: err.message, ctx: req.ov });++ }++ throw err;++ };++++ router.get(COLLECTION, read, async (req, res, next) => {++ try {++ const rows = await merchant.list(req.merchant);++ res.json({++ merchant: { subject: req.merchant.subject, name: req.merchant.name || null },++ sources: await Promise.all(rows.map(decorate)),++ });++ } catch (err) { next(err); }++ });++++ router.get(ITEM, read, async (req, res, next) => {++ try {++ const row = await merchant.get(req.merchant, req.params.id);++ res.json({ merchant_source: await decorate(row) });++ } catch (err) {++ if (err instanceof MerchantError) return fail(res, req, err);++ next(err);++ }++ });++++ router.post(COLLECTION, manage, async (req, res, next) => {++ try {++ const row = await merchant.add(req.merchant, req.body || {});++ res.status(201).json({++ merchant_source: await decorate(row),++ challenge: {++ dns: { name: `${CHALLENGE_PREFIX}.${row.domain}`, value: `${TXT_PREFIX}${row.token}` },++ meta: { name: META_NAME, value: row.token, url: row.meta_url || `https://${row.domain}/` },++ },++ });++ } catch (err) {++ if (err instanceof MerchantError) return fail(res, req, err);++ next(err);++ }++ });++++ router.post(VERIFY, manage, async (req, res, next) => {++ try {++ const out = await merchant.verify(req.merchant, req.params.id);++ res.json({++ merchant_source: await decorate(out.row),++ verified: Boolean(out.verified),++ source_created: Boolean(out.source),++ });++ } catch (err) {++ if (err instanceof MerchantError) return fail(res, req, err);++ next(err);++ }++ });++++ router.post(WITHDRAW, manage, async (req, res, next) => {++ try {++ const row = await merchant.withdraw(req.merchant, req.params.id);++ res.json({ merchant_source: await decorate(row) });++ } catch (err) {++ if (err instanceof MerchantError) return fail(res, req, err);++ next(err);++ }++ });++++ return router;++}++++module.exports = { merchantRouter, createMerchantView, crawlSummary };@@@ -8,6 +8,8 @@ const { createRelease } = require('openvibe-shared/release');const { createSourcesReadiness, registerSourcesGauges } = require('./observability');const { sourcesRouter } = require('./api/sources');const { itemsRouter } = require('./api/items');++const { merchantRouter } = require('./api/merchant');++const { merchantPages } = require('./merchant-pages');const { createActorLimits } = require('./api/actor-limits');const pkg = require('../package.json');@@@ -16,12 +18,14 @@ const TEXT_INDEX = ['Every item carries its source, canonical URL, retrieval time, content hash and terms; a failed fetch is','recorded as a failure and never replaced with invented content.','',-- 'This is an internal service. Its API is for OpenVibe services on the production host (service tokens);',-- 'sources.openvibe.network answers only this page, /api/health, /api/ready and /release.json.',++ 'This is an internal service. Its JSON API is for OpenVibe services on the production host (service tokens),',++ 'and its merchant opt-in pages are for signed-in merchants: sources.openvibe.network answers /, /merchant,',++ '/merchant/*, /shared/*, /api/health, /api/ready and /release.json.','','GET /api/v1/sources, /api/v1/sources/:key, /api/v1/sources/:key/runs, /api/v1/runs, /api/v1/health (sources.source.read)','GET /api/v1/items?source=&category=&after=, /api/v1/items/:id (sources.item.read)','POST/PATCH/DELETE /api/v1/sources[/:key], POST /api/v1/sources/:key/fetch, /items, DELETE /api/v1/items/:id (sources.source.manage)',++ 'GET/POST /api/v1/merchant/sources[/:id[/verify|/withdraw]] (sources.source.read/manage, or a merchant session)','GET /api/health, /api/ready, /release.json','','Source: https://github.com/OpenVibers/OpenVibe.Sources',@@@ -55,15 +59,17 @@ a { color: var(--accent); }where every item came from, when it was fetched and under which terms. A failed fetch is recorded as afailure, never replaced with invented content.</p><p>There is nothing to browse here. What reaches readers is published by the products themselves, under--their own rules.</p>++their own rules. Merchants who want their store listed can <a href="/merchant">list it on OpenVibe</a>:++claim the domain, prove control of it, and name the sitemap or product feed we may read.</p><p class="muted">Health: <a href="/api/health">/api/health</a> · Readiness: <a href="/api/ready">/api/ready</a> ·++Merchant opt-in: <a href="/merchant">/merchant</a> ·Source code: <a href="https://github.com/OpenVibers/OpenVibe.Sources">OpenVibers/OpenVibe.Sources</a></p></main></body></html>`;--function createApp({ config, db, registry, items, ingest, scheduler, auth, outbox, now, log = console, limitsNow = null }) {++function createApp({ config, db, registry, items, ingest, scheduler, auth, outbox, now, log = console, limitsNow = null, merchant = null, merchantAuth = null }) {const app = express();app.disable('x-powered-by');app.set('trust proxy', 'loopback');@@@ -100,9 +106,24 @@ function createApp({ config, db, registry, items, ingest, scheduler, auth, outboapp.use(sourcesRouter({ db, registry, ingest, auth, limits }));app.use(itemsRouter({ db, registry, items, auth, outbox, now, limits }));-- // The public host (sources.openvibe.network) shows only this page, health, readiness and-- // /release.json: Sources is internal. Browsers get a short honest HTML page, other clients the-- // text route index. Never cached, never indexed.++ // Merchant opt-in (plan §4.4 channel 1): the JSON read/write API, and the signed-in pages that are its front door.++ // /shared carries this release's own kit.css/kit-web.js/theme-loader.js, so the pages do not depend on another host.++ if (merchant && merchantAuth) {++ app.use('/shared', require('openvibe-shared/serve').handler());++ app.use(merchantRouter({ db, merchant, registry, ingest, auth, merchantAuth, limits }));++ const networkOrigin = (() => { try { return new URL(config.networkUrl).origin; } catch { return 'https://openvibe.network'; } })();++ app.use('/merchant', (req, res, next) => {++ res.setHeader('X-Robots-Tag', 'noindex, nofollow');++ res.setHeader('Content-Security-Policy', `default-src 'none'; style-src 'self' 'unsafe-inline'; script-src 'self'; img-src 'self' data: https:; connect-src 'self' ${networkOrigin}; form-action 'self'; base-uri 'none'; frame-ancestors 'none'`);++ next();++ }, express.urlencoded({ extended: false, limit: '32kb' }), merchantAuth.page, merchantPages({++ db, merchant, registry, ingest, merchantAuth, limits, log,++ }));++ }++++ // The public host (sources.openvibe.network) shows the internal-service page, the merchant opt-in pages and++ // health/readiness/release.json: the JSON API stays loopback-only. Browsers get a short honest HTML page, other++ // clients the text route index. Never cached, never indexed.app.get('/', (req, res) => {res.setHeader('X-Robots-Tag', 'noindex, nofollow');res.setHeader('Vary', 'Accept');@@@ -1,17 +1,22 @@'use strict';/**-- * Service-token authentication and capability checks. Every API route is for services (other-- * OpenVibe services, or the Network admin surface acting through its own principal): an-- * RS256 client-credentials token from OpenVibe.Network (audience openvibe.sources), verified-- * offline against Network's JWKS (openvibe-sdk/auth, one client per URL, kept fresh and served-- * through an outage), and ONE capability per route.++ * Service-token authentication and capability checks. API routes for services (other OpenVibe++ * services, or the Network admin surface acting through its own principal) use an RS256++ * client-credentials token from OpenVibe.Network (audience openvibe.sources), verified offline++ * against Network's JWKS (openvibe-sdk/auth, one client per URL, kept fresh and served through an++ * outage), and ONE capability per route.++ *++ * The merchant opt-in routes also accept a person's Network session token (the same RS256 key and++ * issuer, no `typ`, a usr_… `subject_id`): the merchant is the owner of the claim. Pages use the++ * same verification on their own first-party cookie. A first-party service may act for a person++ * only through the X-OV-Subject header (server/merchant-auth.js), as OpenVibe.Space does.** The sources.* capabilities are released in openvibe-contracts (docs/capabilities-proposal/ is* kept in step with those manifests), so the library's own grant rule decides every one: an exact* id or a `family.*` grant, and capability.unknown for an id it does not know.*/--const { serviceAuth, capabilities, http } = require('openvibe-contracts');--const { jwksClient } = require('openvibe-sdk/auth');++const { serviceAuth, capabilities, http, ids } = require('openvibe-contracts');++const { jwksClient, verifyUserToken } = require('openvibe-sdk/auth');const CAPS = Object.freeze({read: 'sources.source.read',@@@ -23,8 +28,9 @@ function checkCapability(claims, capabilityId) {return capabilities.check(claims, capabilityId);}--function createAuth({ config, log = console }) {++function createAuth({ config, log = console, jwks = null }) {const headerKid = (token) => { try { return JSON.parse(Buffer.from(String(token).split('.')[0], 'base64url').toString('utf8')).kid || null; } catch { return null; } };++ const userKeys = jwks || jwksClient(config.jwksUrl, { log });/** { ok, claims } or { ok: false, code, reason }. The SDK's error names the internal JWKS URL: logged, never answered. */async function verifyService(token) {@@@ -43,6 +49,23 @@ function createAuth({ config, log = console }) {return last;}++ /**++ * A person's Network session token → { claims, subject: usr_…, name } or null. Typed tokens (FedCM assertions,++ * realtime tickets) and service/app principals are refused by verifyUserToken; the subject must be a usr_ id.++ */++ async function verifyUser(token) {++ if (!token) return null;++ try {++ const claims = await verifyUserToken(String(token), { jwks: userKeys, issuer: config.issuer });++ const subject = ids.isSubjectId('user', claims.subject_id) ? claims.subject_id++ : (ids.isSubjectId('user', claims.sub) ? claims.sub : null);++ if (!subject) return null;++ return { claims, subject, name: claims.username || claims.display_name || claims.name || null };++ } catch {++ return null;++ }++ }++/** Express guard: one capability. Sets req.principal = { sub, cap, jti }. */function requireCap(id) {return async function capGuard(req, res, next) {@@@ -62,7 +85,7 @@ function createAuth({ config, log = console }) {return next();};}-- return { requireCap };++ return { requireCap, verifyService, verifyUser };}module.exports = { CAPS, createAuth, checkCapability };@@@ -25,13 +25,14 @@ function load(env = process.env) {// The JWKS the SDK's client (openvibe-sdk/auth jwksClient) fetches, caches and keeps fresh; one URL,// since the SDK keeps one client per URL. Defaults to Network's internal JWKS.const jwksUrl = strip(env.OV_NETWORK_JWKS_URL || `${networkInternalUrl}/api/.well-known/jwks`);++ const baseUrl = strip(env.BASE_URL || (isProduction ? 'https://sources.openvibe.network' : `http://localhost:${port}`));return {port,host: env.HOST || '127.0.0.1',nodeEnv,isProduction,serviceId: 'sources',-- baseUrl: strip(env.BASE_URL || (isProduction ? 'https://sources.openvibe.network' : `http://localhost:${port}`)),++ baseUrl,networkUrl,networkInternalUrl,@@@ -47,6 +48,35 @@ function load(env = process.env) {oauth: {clientId: env.OV_OAUTH_CLIENT_ID || 'sources',clientSecret: env.OV_OAUTH_CLIENT_SECRET || '',++ // The Network OAuth client `sources` must be registered with this exact redirect URI.++ redirectUri: strip(env.OV_OAUTH_REDIRECT_URI || `${baseUrl}/merchant/callback`),++ scope: env.OV_OAUTH_SCOPE || 'profile',++ },++ cookies: {++ // A first-party session cookie; Secure everywhere except local tests/development over http.++ secure: env.SOURCES_COOKIE_SECURE != null ? env.SOURCES_COOKIE_SECURE === '1' : isProduction,++ },++++ // Merchant opt-in (server/merchant.js): proof checks, the session form secret, and the shape of the source++ // created only after verification. A sitemap becomes a crawl; a product feed uses the feed adapter.++ merchant: {++ pendingDays: Math.max(1, int(env.SOURCES_MERCHANT_PENDING_DAYS, 14)),++ // Pending claims are checked again on this cadence; 0 turns the worker pass off (tests).++ recheckIntervalMs: Math.max(0, int(env.SOURCES_MERCHANT_RECHECK_MS, 10 * 60 * 1000)),++ metaMaxBytes: Math.min(5 * 1024 * 1024, Math.max(1024, int(env.SOURCES_MERCHANT_META_MAX_BYTES, 1024 * 1024))),++ metaTimeoutMs: Math.min(120000, Math.max(1000, int(env.SOURCES_MERCHANT_META_TIMEOUT_MS, 10000))),++ dnsServers: list(env.SOURCES_MERCHANT_DNS_SERVERS, []),++ // CSRF/state HMAC key. Production should set SOURCES_SESSION_SECRET (or shares OV_OAUTH_CLIENT_SECRET).++ sessionSecret: env.SOURCES_SESSION_SECRET || env.OV_OAUTH_CLIENT_SECRET || 'sources-merchant-development',++ crawlMinIntervalMs: Math.min(86400000, Math.max(0, int(env.SOURCES_MERCHANT_CRAWL_MIN_INTERVAL_MS, 1000))),++ feedMinIntervalMs: Math.min(86400000, Math.max(0, int(env.SOURCES_MERCHANT_FEED_MIN_INTERVAL_MS, 60000))),++ pollIntervalSec: Math.min(7 * 86400, Math.max(1, int(env.SOURCES_MERCHANT_POLL_INTERVAL_SEC, 3600))),++ crawlStaleAfterSec: Math.min(90 * 86400, Math.max(1, int(env.SOURCES_MERCHANT_CRAWL_STALE_AFTER_SEC, 86400))),++ feedStaleAfterSec: Math.min(90 * 86400, Math.max(1, int(env.SOURCES_MERCHANT_FEED_STALE_AFTER_SEC, 10800))),++ maxItems: Math.min(12000, Math.max(1, int(env.SOURCES_MERCHANT_MAX_ITEMS, 1000))),++ crawlMaxPages: Math.min(200000, Math.max(1, int(env.SOURCES_MERCHANT_MAX_PAGES, 5000))),++ crawlPagesPerRun: Math.min(1000, Math.max(1, int(env.SOURCES_MERCHANT_PAGES_PER_RUN, 100))),++ crawlRecrawlSec: Math.min(30 * 86400, Math.max(3600, int(env.SOURCES_MERCHANT_RECRAWL_SEC, 86400))),},// PostgreSQL (ADR-035): DATABASE_URL serves (PgBouncer), DATABASE_DIRECT_URL migrates (owner role).@@@ -25,9 +25,11 @@ const { jwksClient } = require('openvibe-sdk/auth');const { gracefulStop, telemetry } = require('openvibe-sdk/service');const { createPgTelemetrySink, startTelemetryRetention } = require('openvibe-sdk/telemetry');const { createAuth } = require('./auth');++const { createMerchant } = require('./merchant');++const { createMerchantAuth } = require('./merchant-auth');const { createApp } = require('./app');--async function start({ config, db: givenDb = null, now = () => Date.now(), fetchImpl = globalThis.fetch, lookupImpl, log = console, listen = true, limitsNow = null } = {}) {++async function start({ config, db: givenDb = null, now = () => Date.now(), fetchImpl = globalThis.fetch, lookupImpl, merchantResolver = null, log = console, listen = true, limitsNow = null } = {}) {config = config || load();// PostgreSQL (ADR-035): opened and migrated here unless the caller (a test) hands in a migrated handle.const db = givenDb || await openDb(config, { log });@@@ -53,14 +55,26 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetchconst ingest = createIngest({ db, config, registry, items, robots, fetcher, spacer, emitEvent, now, log, outbox });const scheduler = createScheduler({ db, ingest, config, now, log });const keys = jwksClient(config.jwksUrl, { fetch: fetchImpl, log });-- const auth = createAuth({ config, log });-- const app = createApp({ config, db, registry, items, ingest, scheduler, auth, outbox, now, log, limitsNow });++ const auth = createAuth({ config, log, jwks: keys });++ const merchant = createMerchant({ db, config, registry, fetcher, resolver: merchantResolver, now, log });++ const merchantAuth = createMerchantAuth({ config, auth, log, fetchImpl });++ const app = createApp({ config, db, registry, items, ingest, scheduler, auth, outbox, now, log, limitsNow, merchant, merchantAuth });// One JWKS client for the process (the SDK shares it with verifyUserToken): refresh in the background// on an unref'd timer, keeping the last good keys through a Network outage.keys.start();const keyLoaded = keys.keys().catch(() => null);outbox.start();if (config.worker.enabled) scheduler.start();++ // Pending merchant proofs are checked again on a slow cadence (the merchant can always press "Check now"). A claim++ // that never verifies fails after merchant.pendingDays. 0 turns the worker pass off (tests).++ let merchantTimer = null;++ let merchantKick = null;++ if (config.merchant.recheckIntervalMs > 0) {++ merchantTimer = setInterval(() => merchant.recheck().catch((err) => log.error(`[merchant] recheck: ${err.stack || err}`)), config.merchant.recheckIntervalMs);++ merchantTimer.unref?.();++ merchantKick = setTimeout(() => merchant.recheck().catch((err) => log.error(`[merchant] recheck: ${err.stack || err}`)), 5000);++ merchantKick.unref?.();++ }const pruneTimer = setInterval(async () => { try { await outbox.outbox.prune(); } catch (err) { log.error(`[outbox] prune: ${err.message}`); } }, 6 * 3600 * 1000);pruneTimer.unref?.();@@@ -75,6 +89,8 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetchasync function close() {clearInterval(pruneTimer);++ if (merchantTimer) clearInterval(merchantTimer);++ if (merchantKick) clearTimeout(merchantKick);keys.stop();await scheduler.stop();await outbox.stop();@@@ -85,7 +101,7 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetchif (!givenDb) await db.close();}-- return { config, db, registry, items, guard, fetcher, robots, ingest, scheduler, outbox, keys, keyLoaded, auth, app, server, close };++ return { config, db, registry, items, guard, fetcher, robots, ingest, scheduler, outbox, keys, keyLoaded, auth, merchant, merchantAuth, app, server, close };}if (require.main === module) {nnew file mode 100644@@@ -0,0 +1,225 @@++'use strict';++/**++ * Merchant authentication for the opt-in flow.++ *++ * API `/api/v1/merchant/*`++ * - a person's Network session token in `Authorization: Bearer` (the merchant owns the claims); or++ * - a first-party service token with `sources.source.read` / `sources.source.manage` and X-OV-Subject,++ * the way OpenVibe.Space lets a first-party service act for a person. An app/module token cannot delegate.++ * Pages `/merchant/*`++ * - the Sources first-party `ov_token` cookie, set by the OAuth callback below and verified offline with++ * openvibe-sdk/auth verifyUserToken. There is no client-side token storage.++ *++ * The OAuth client (`OV_OAUTH_CLIENT_ID`, default `sources`) must be registered at OpenVibe.Network with++ * `<BASE_URL>/merchant/callback` as a redirect URI. Forms carry an HMAC CSRF token bound to the merchant subject;++ * cookies are httpOnly, SameSite=Lax, Secure in production.++ */++const crypto = require('crypto');++const { http, ids } = require('openvibe-contracts');++const { checkCapability } = require('./auth');++++const ACCESS_COOKIE = 'ov_token';++const STATE_COOKIE = 'ov_merchant_state';++const NEXT_COOKIE = 'ov_merchant_next';++const FLOW_MAX_AGE = 10 * 60 * 1000;++const SESSION_MAX_AGE = 24 * 3600 * 1000;++const PRINCIPAL_SUB = /^(svc|app|mod|node|agent):/;++++function decodePayload(token) {++ const parts = String(token || '').split('.');++ if (parts.length !== 3) return null;++ try { return JSON.parse(Buffer.from(parts[1], 'base64url').toString('utf8')); } catch { return null; }++}++++function cookie(req, name) {++ const m = String((req && req.headers && req.headers.cookie) || '').match(new RegExp(`(?:^|;\\s*)${name}=([^;]*)`));++ if (!m) return null;++ try { return decodeURIComponent(m[1]); } catch { return null; }++}++++function bearer(req) {++ const h = String((req && req.headers && req.headers.authorization) || '');++ return h.startsWith('Bearer ') ? h.slice(7).trim() || null : null;++}++++/** Only a same-site merchant path is a post-sign-in destination. */++function safeNext(value) {++ const n = String(value == null ? '' : value);++ if (!n || /[\u0000-\u001f\u007f\\]/.test(n)) return '/merchant';++ if (!/^\/merchant(?:\/|$|\?|#)/.test(n)) return '/merchant';++ return n;++}++++/** `/oauth/authorize` on the public Network, with the state the callback checks. */++function authorizeUrl(config, state) {++ const u = new URL('/oauth/authorize', config.networkUrl);++ u.searchParams.set('client_id', config.oauth.clientId);++ u.searchParams.set('redirect_uri', config.oauth.redirectUri);++ u.searchParams.set('response_type', 'code');++ u.searchParams.set('scope', config.oauth.scope || 'profile');++ u.searchParams.set('state', state);++ return u.toString();++}++++function createMerchantAuth({ config, auth, log = console, fetchImpl = globalThis.fetch }) {++ const secret = config.merchant.sessionSecret;++ const secure = Boolean(config.cookies && config.cookies.secure);++ const baseCookie = { sameSite: 'lax', secure, httpOnly: true, path: '/' };++ const flowCookie = { sameSite: 'lax', secure, httpOnly: true, path: '/merchant', maxAge: FLOW_MAX_AGE };++++ const csrfFor = (subject) => crypto.createHmac('sha256', secret).update(`sources-merchant:${subject}`).digest('base64url').slice(0, 32);++++ function checkCsrf(principal, token) {++ if (!principal || !principal.subject || typeof token !== 'string') return false;++ const expected = csrfFor(principal.subject);++ const a = Buffer.from(expected);++ const b = Buffer.from(token);++ return a.length === b.length && crypto.timingSafeEqual(a, b);++ }++++ function setSession(res, token) {++ res.cookie(ACCESS_COOKIE, token, { ...baseCookie, maxAge: SESSION_MAX_AGE });++ }++++ function clearSession(res) {++ res.clearCookie(ACCESS_COOKIE, { ...baseCookie, maxAge: undefined });++ }++++ async function resolvePage(req) {++ const token = bearer(req) || cookie(req, ACCESS_COOKIE);++ if (!token) return null;++ const payload = decodePayload(token);++ if (payload && (payload.actor_type || PRINCIPAL_SUB.test(String(payload.sub || '')))) return null;++ const u = await auth.verifyUser(token);++ if (!u) return null;++ return { subject: u.subject, name: u.name, kind: 'user', claims: u.claims };++ }++++ /** Attach req.merchant when the page cookie is a live person session; anonymous stays anonymous. */++ async function page(req, _res, next) {++ try {++ const merchant = await resolvePage(req);++ if (merchant) {++ req.merchant = merchant;++ req.principal = { sub: `user:${merchant.subject}` };++ }++ return next();++ } catch (err) { return next(err); }++ }++++ function requirePage(req, res, next) {++ if (req.merchant) return next();++ const nextPath = req.method === 'GET' || req.method === 'HEAD' ? safeNext(req.originalUrl) : '/merchant';++ return res.redirect(303, `/merchant/login?next=${encodeURIComponent(nextPath)}`);++ }++++ function problem(res, req, status, code, detail) {++ return http.sendProblem(res, status, code, { detail, ctx: req.ov });++ }++++ /**++ * API guard: a person's token (owner) or a first-party service token with `cap` plus X-OV-Subject. Sets++ * req.merchant = { subject, name, kind } and req.principal for the per-actor limiter.++ */++ function api(cap) {++ return async function merchantGuard(req, res, next) {++ const token = bearer(req);++ if (!token) return problem(res, req, 401, 'token.missing', 'a merchant session or a service token is required');++ const payload = decodePayload(token);++ const principalToken = payload && (payload.actor_type || PRINCIPAL_SUB.test(String(payload.sub || '')));++ try {++ if (principalToken) {++ const r = await auth.verifyService(token);++ if (!r.ok) return problem(res, req, r.code === 'token.unavailable' ? 503 : 401, r.code, r.reason);++ const c = checkCapability(r.claims, cap);++ if (!c.allowed) return problem(res, req, 403, c.code, c.reason);++ const delegated = req.get('x-ov-subject') || req.query.subject || null;++ if (!delegated) return problem(res, req, 400, 'merchant.subject_required', 'a first-party service must name the merchant with X-OV-Subject');++ if (!String(r.claims.sub).startsWith('svc:')) return problem(res, req, 403, 'merchant.subject_forbidden', 'only a first-party service may act for a person');++ if (!ids.isSubjectId('user', String(delegated))) return problem(res, req, 400, 'merchant.subject_invalid', 'X-OV-Subject must be a usr_… subject id');++ req.merchant = { subject: String(delegated), name: null, kind: 'service', service: r.claims.sub };++ req.principal = { sub: r.claims.sub, cap: r.claims.cap, jti: r.claims.jti };++ return next();++ }++ const u = await auth.verifyUser(token);++ if (!u) return problem(res, req, 401, 'merchant.auth_required', 'the bearer token is not a valid OpenVibe person session');++ req.merchant = { subject: u.subject, name: u.name, kind: 'user', claims: u.claims };++ req.principal = { sub: `user:${u.subject}` };++ return next();++ } catch (err) { return next(err); }++ };++ }++++ async function login(req, res) {++ const next = safeNext(req.query.next);++ const existing = await resolvePage(req);++ if (existing) return res.redirect(303, next);++ const state = crypto.randomBytes(16).toString('hex');++ res.cookie(STATE_COOKIE, state, flowCookie);++ res.cookie(NEXT_COOKIE, next, flowCookie);++ return res.redirect(authorizeUrl(config, state));++ }++++ async function tokenGrant(body) {++ let lastErr = null;++ for (const base of [config.networkInternalUrl, config.networkUrl]) {++ if (!base) continue;++ try {++ const res = await fetchImpl(`${base}/oauth/token`, {++ method: 'POST',++ headers: { 'Content-Type': 'application/json' },++ body: JSON.stringify({ client_id: config.oauth.clientId, client_secret: config.oauth.clientSecret, ...body }),++ signal: AbortSignal.timeout(10_000),++ });++ const data = await res.json().catch(() => ({}));++ if (!res.ok) {++ const err = new Error(data.error_description || data.error || `token grant failed (${res.status})`);++ err.status = res.status;++ err.error = data.error || 'invalid_grant';++ throw err;++ }++ return data;++ } catch (err) {++ lastErr = err;++ if (err.status && err.status < 500) throw err;++ }++ }++ throw lastErr || new Error('Network unreachable');++ }++++ async function callback(req, res) {++ const { code, state, error } = req.query;++ const next = safeNext(cookie(req, NEXT_COOKIE));++ const expected = cookie(req, STATE_COOKIE);++ res.clearCookie(STATE_COOKIE, { ...flowCookie, maxAge: undefined });++ res.clearCookie(NEXT_COOKIE, { ...flowCookie, maxAge: undefined });++ if (error) return res.redirect(303, `/merchant?auth_error=${encodeURIComponent(String(error).slice(0, 80))}`);++ if (!code || !state || !expected || String(state).length !== String(expected).length++ || !crypto.timingSafeEqual(Buffer.from(String(state)), Buffer.from(String(expected)))) {++ return res.status(400).type('html').send('<!doctype html><meta charset="utf-8"><title>Sign-in failed</title><p>The sign-in response did not match this browser. <a href="/merchant/login">Try again</a>.</p>');++ }++ try {++ const data = await tokenGrant({ grant_type: 'authorization_code', redirect_uri: config.oauth.redirectUri, code });++ const u = await auth.verifyUser(data.access_token);++ if (!u) throw Object.assign(new Error('the Network returned a token without a person subject'), { status: 401 });++ setSession(res, data.access_token);++ return res.redirect(303, next);++ } catch (err) {++ log.error(`[merchant] sign-in failed: ${err.message}`);++ return res.status(502).type('html').send('<!doctype html><meta charset="utf-8"><title>Sign-in failed</title><p>OpenVibe.Network could not complete the sign-in. <a href="/merchant/login">Try again</a>.</p>');++ }++ }++++ function logout(req, res) {++ clearSession(res);++ return res.redirect(303, '/merchant');++ }++++ return {++ api, page, requirePage, login, callback, logout,++ resolvePage, checkCsrf, csrfFor, setSession, clearSession,++ ACCESS_COOKIE, STATE_COOKIE, NEXT_COOKIE, safeNext, authorizeUrl,++ };++}++++module.exports = { createMerchantAuth, authorizeUrl, safeNext, decodePayload, cookie, bearer };nnew file mode 100644@@@ -0,0 +1,139 @@++'use strict';++/**++ * The merchant opt-in pages (plain forms, no client-side state). The JSON API and these pages share server/merchant.js:++ * the page is the signed-in merchant's front door, and the API is what Report calls later.++ *++ * GET /merchant the merchant's claims, or the guest explanation++ * GET /merchant/sources/:id proof instructions, status, terms note and last crawl++ * POST /merchant/sources add a domain (CSRF)++ * POST /merchant/sources/:id/verify check the proof now (CSRF)++ * POST /merchant/sources/:id/withdraw disable the source and keep the note (CSRF)++ * GET /merchant/login|callback|logout++ */++const express = require('express');++const { MerchantError } = require('./merchant');++const views = require('./views/merchant');++const { createMerchantView } = require('./api/merchant');++++const NOTICES = {++ created: 'Domain added. Publish the proof below, then check now. Nothing is fetched or enabled before the proof is found.',++ verified: 'Proof found. Your source was created and enabled.',++ pending: 'The proof was not found yet. Publish it and check again.',++ withdrawn: 'Permission withdrawn. The source is disabled and its terms note is kept.',++};++++function merchantPages({ db, merchant, registry, ingest, merchantAuth, limits, log = console }) {++ const router = express.Router();++ const decorate = createMerchantView({ db, merchant, registry, ingest });++ const readLimit = limits.budget('sources.merchant.read');++ const manageLimit = limits.budget('sources.merchant.manage');++++ const noticeFor = (query) => Object.keys(NOTICES).filter(k => query[k] != null && query[k] !== '0').map(k => NOTICES[k]).join(' ');++ const send = (res, status, html) => res.status(status).type('html').send(html);++++ async function listPage(req, res, status = 200, extra = {}) {++ const claims = req.merchant ? await Promise.all((await merchant.list(req.merchant)).map(decorate)) : [];++ return send(res, status, views.merchantPage({++ merchant: req.merchant,++ claims,++ csrf: req.merchant ? merchantAuth.csrfFor(req.merchant.subject) : '',++ notice: noticeFor(req.query),++ authError: req.query.auth_error ? String(req.query.auth_error).slice(0, 80) : '',++ ...extra,++ }));++ }++++ async function detailPage(req, res, status = 200, extra = {}) {++ const row = await merchant.get(req.merchant, req.params.id);++ return send(res, status, views.detailPage({++ merchant: req.merchant,++ claim: await decorate(row),++ csrf: merchantAuth.csrfFor(req.merchant.subject),++ notice: noticeFor(req.query),++ noticeKind: req.query.pending != null && req.query.pending !== '0' ? 'warn' : 'ok',++ ...extra,++ }));++ }++++ function csrfOk(req) {++ return merchantAuth.checkCsrf(req.merchant, req.body && req.body.csrf);++ }++++ router.get('/', readLimit, async (req, res, next) => {++ try {++ if (!req.merchant) {++ return send(res, 200, views.guestPage({++ authError: req.query.auth_error ? String(req.query.auth_error).slice(0, 80) : '',++ }));++ }++ return await listPage(req, res);++ } catch (err) { next(err); }++ });++++ router.get('/login', async (req, res, next) => {++ try { await merchantAuth.login(req, res); } catch (err) { next(err); }++ });++ router.get('/callback', async (req, res, next) => {++ try { await merchantAuth.callback(req, res); } catch (err) { next(err); }++ });++ router.get('/logout', (req, res) => merchantAuth.logout(req, res));++++ router.get('/sources', (req, res) => res.redirect(303, '/merchant'));++ router.get('/sources/:id', merchantAuth.requirePage, readLimit, async (req, res, next) => {++ try { return await detailPage(req, res); } catch (err) {++ if (err instanceof MerchantError) return send(res, err.status, views.errorPage({ status: err.status, message: err.message }).html);++ next(err);++ }++ });++++ router.post('/sources', merchantAuth.requirePage, manageLimit, async (req, res, next) => {++ try {++ if (!csrfOk(req)) return send(res, 403, views.errorPage({ status: 403, message: 'This form expired or did not come from this page. Reload and try again.' }).html);++ const row = await merchant.add(req.merchant, req.body || {});++ return res.redirect(303, `/merchant/sources/${row.id}?created=1`);++ } catch (err) {++ if (err instanceof MerchantError) {++ try { return await listPage(req, res, err.status, { error: err.message }); } catch (inner) { return next(inner); }++ }++ next(err);++ }++ });++++ router.post('/sources/:id/verify', merchantAuth.requirePage, manageLimit, async (req, res, next) => {++ try {++ if (!csrfOk(req)) return send(res, 403, views.errorPage({ status: 403, message: 'This form expired or did not come from this page. Reload and try again.' }).html);++ const out = await merchant.verify(req.merchant, req.params.id);++ const verified = out.row && out.row.status === 'verified';++ return res.redirect(303, `/merchant/sources/${out.row.id}?${verified ? 'verified=1' : 'pending=1'}`);++ } catch (err) {++ if (err instanceof MerchantError) {++ try { return await detailPage(req, res, err.status, { error: err.message }); } catch (inner) { return next(inner); }++ }++ next(err);++ }++ });++++ router.post('/sources/:id/withdraw', merchantAuth.requirePage, manageLimit, async (req, res, next) => {++ try {++ if (!csrfOk(req)) return send(res, 403, views.errorPage({ status: 403, message: 'This form expired or did not come from this page. Reload and try again.' }).html);++ const row = await merchant.withdraw(req.merchant, req.params.id);++ return res.redirect(303, `/merchant/sources/${row.id}?withdrawn=1`);++ } catch (err) {++ if (err instanceof MerchantError) {++ try { return await detailPage(req, res, err.status, { error: err.message }); } catch (inner) { return next(inner); }++ }++ next(err);++ }++ });++++ // A page error is HTML, not a JSON problem: the merchant can read it and go back.++ router.use((err, req, res, _next) => {++ if (res.headersSent) return res.end();++ log.error(`[merchant] ${req.method} ${req.originalUrl}: ${err.stack || err}`);++ return send(res, 500, views.errorPage({ status: 500, message: 'Something went wrong on our side. Please try again.' }).html);++ });++++ return router;++}++++module.exports = { merchantPages, NOTICES };nnew file mode 100644@@@ -0,0 +1,398 @@++'use strict';++/**++ * Merchant opt-in ("List your store on OpenVibe", plan §4.4 channel 1).++ *++ * A signed-in merchant adds a domain, proves control of it (the OpenVibe.Space pattern: a DNS TXT record at++ * _openvibe-sources.<domain>, or a meta tag on a page of the domain), and names the sitemap or product feed Sources may++ * read. Nothing enters `sources` before the proof is present: a pending claim lives only in `merchant_sources`. On++ * verification one enabled source is created in the same transaction, and its terms_note records who gave the explicit++ * permission, when, and how control was verified. Withdrawal disables that source and keeps the note (and the claim) as++ * the audit trail. The fetcher/crawler still obeys robots.txt on every request; explicit permission is not a robots++ * override.++ *++ * The challenge is a random token in a public DNS/meta record. It is not a credential and is shown only to the owner (or++ * a first-party service acting for them, with X-OV-Subject).++ *++ * ADR 0007 still governs technique: this is a source whose access verdict is the merchant's explicit proof, not a probe++ * of a site we have no permission from; every request it causes goes through the same fetcher, UA, robots and spacing.++ */++const crypto = require('crypto');++const dns = require('dns');++const { ids } = require('openvibe-contracts');++const { decodeBody, decodeEntities } = require('./adapters/util');++++const CHALLENGE_PREFIX = '_openvibe-sources';++const TXT_PREFIX = 'openvibe-sources-verification=';++const META_NAME = 'openvibe-sources-verification';++const METHODS = ['dns', 'meta'];++const LISTING_KINDS = ['sitemap', 'feed'];++const DAY = 24 * 3600 * 1000;++++const LABEL = '[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?';++const HOSTNAME_RE = new RegExp(`^(?:${LABEL}\\.)+[a-z][a-z0-9-]{0,61}[a-z0-9]$`);++const RESERVED_SUFFIXES = ['.local', '.internal', '.localhost', '.test', '.invalid', '.example', '.onion', '.home.arpa'];++const ID_RE = /^msrc_[0-9A-HJKMNP-TV-Z]{26}$/;++++class MerchantError extends Error {++ constructor(status, code, detail) { super(detail); this.status = status; this.code = code; }++}++++const clip = (v, n) => String(v == null ? '' : v).replace(/[\u0000-\u001f\u007f]/g, ' ').trim().slice(0, n);++const iso = (v) => (v == null ? null : new Date(Number(v)).toISOString());++++function isHostname(h) {++ return typeof h === 'string' && h.length <= 253 && HOSTNAME_RE.test(h);++}++++/** A claimed domain: lowercase, no port, a real DNS name, not an OpenVibe property or a reserved name. */++function checkDomain(raw) {++ const domain = String(raw || '').trim().toLowerCase().replace(/\.$/, '');++ if (!isHostname(domain)) throw new MerchantError(422, 'merchant.domain_invalid', 'the domain must be a fully qualified host name such as shop.example.org');++ if (RESERVED_SUFFIXES.some(s => domain.endsWith(s))) throw new MerchantError(422, 'merchant.domain_reserved', `${domain} is a reserved name and cannot be claimed`);++ if (/(^|\.)openvibe\./.test(domain)) throw new MerchantError(422, 'merchant.domain_reserved', `${domain} is an OpenVibe domain and cannot be claimed`);++ return domain;++}++++/** A URL the merchant names: http(s), no credentials, at most 2048 characters, no secret-looking query parameter. */++function checkUrl(raw, what) {++ const s = String(raw == null ? '' : raw).trim();++ if (!s || s.length > 2048) throw new MerchantError(422, 'merchant.url_invalid', `${what} must be an http(s) URL of at most 2048 characters`);++ let u;++ try { u = new URL(s); } catch { throw new MerchantError(422, 'merchant.url_invalid', `${what} is not a URL`); }++ if (u.protocol !== 'http:' && u.protocol !== 'https:') throw new MerchantError(422, 'merchant.url_invalid', `${what} must be http(s)`);++ if (u.username || u.password) throw new MerchantError(422, 'merchant.url_invalid', `${what} must not carry credentials`);++ for (const k of u.searchParams.keys()) {++ if (/^(api[_-]?key|apikey|key|token|access[_-]?token|auth|secret|client[_-]?secret|password|passwd|sig|signature)$/i.test(k)) {++ throw new MerchantError(422, 'merchant.url_secret', `${what} must not carry a credential in its query string`);++ }++ }++ u.hash = '';++ return u.toString();++}++++/** The claimed domain and a URL's host are the same site: the domain, its www twin, or a subdomain of it. */++function sameSite(domain, host) {++ const d = String(domain || '').toLowerCase().replace(/\.$/, '');++ const h = String(host || '').toLowerCase().replace(/\.$/, '');++ if (!d || !h) return false;++ return h === d || h === `www.${d}` || d === `www.${h}` || h.endsWith(`.${d}`);++}++++function sourceKeyFor(row) {++ const slug = String(row.domain).replace(/[^a-z0-9]+/g, '-').replace(/^-+|-+$/g, '').slice(0, 40).replace(/-+$/g, '') || 'site';++ const suffix = String(row.id).replace(/^msrc_/, '').slice(-8).toLowerCase();++ return `merchant-${slug}-${suffix}`;++}++++/** The explicit-permission note products read: who, when, how, and that withdrawal is possible. */++function termsNoteFor(row, verifiedAt) {++ const who = row.owner_name ? `${row.owner_sub} (${clip(row.owner_name, 80)})` : row.owner_sub;++ const how = row.method === 'dns'++ ? `a DNS TXT record at ${CHALLENGE_PREFIX}.${row.domain}`++ : `a meta tag at ${clip(row.meta_url || `https://${row.domain}/`, 400)}`;++ const what = row.listing_kind === 'sitemap' ? 'sitemap' : 'product feed';++ return clip(`Merchant opt-in: ${who} gave OpenVibe explicit permission on ${iso(verifiedAt)} to list ${row.domain} from the ${what} at ${clip(row.listing_url, 500)}, verified by ${how}. The merchant may withdraw this permission, which disables this source; robots.txt is still checked on every request.`, 2000);++}++++/** The source definition created only after the proof. A sitemap becomes a crawl; a feed uses the feed adapter. */++function sourceFor(row, termsNote, config) {++ const crawl = row.listing_kind === 'sitemap';++ const atom = !crawl && /\.atom(?:\.xml)?$/i.test(new URL(row.listing_url).pathname);++ const m = config.merchant;++ return {++ key: sourceKeyFor(row),++ name: clip(row.store_name || `${row.domain} (merchant opt-in)`, 200),++ type: crawl ? 'crawl' : (atom ? 'atom' : 'rss'),++ category: 'deals',++ homepage_url: `https://${row.domain}/`,++ endpoints: [row.listing_url],++ auth: { mode: 'none' },++ robots_note: 'robots.txt is checked on every request and every redirect hop; a disallowed page is not fetched.',++ terms_note: termsNote,++ license_note: null,++ min_interval_ms: crawl ? m.crawlMinIntervalMs : m.feedMinIntervalMs,++ poll_interval_sec: m.pollIntervalSec,++ stale_after_sec: crawl ? m.crawlStaleAfterSec : m.feedStaleAfterSec,++ max_items: Math.min(m.maxItems, config.items.maxPerFetch),++ enabled: true,++ review_required: false,++ sensitivity: 'none',++ default_indexability: 'index',++ search_visibility: null,++ retire_after_sec: null,++ crawl: crawl ? {++ // The merchant named this sitemap: follow it (and any sitemap index below it), not other sitemaps found by guess.++ sitemaps: false,++ follow_links: false,++ include: [],++ exclude: [],++ max_pages: m.crawlMaxPages,++ pages_per_run: m.crawlPagesPerRun,++ recrawl_sec: m.crawlRecrawlSec,++ max_depth: 0,++ require_structured: true,++ } : null,++ };++}++++function defaultResolver(servers) {++ const r = new dns.promises.Resolver({ timeout: 5000, tries: 2 });++ if (servers && servers.length) r.setServers(servers);++ return { resolveTxt: (name) => r.resolveTxt(name) };++}++++/** True when the page carries <meta name="openvibe-sources-verification" content="<token>">, in any attribute order. */++function hasVerificationMeta(html, token) {++ const ATTR = /([A-Za-z_:][-A-Za-z0-9_:.]*)\s*=\s*(?:"([^"]*)"|'([^']*)'|([^\s"'=<>`]+))/g;++ for (const tag of String(html || '').match(/<meta\b[^>]*>/gi) || []) {++ const attrs = {};++ let m;++ ATTR.lastIndex = 0;++ while ((m = ATTR.exec(tag))) attrs[m[1].toLowerCase()] = m[2] ?? m[3] ?? m[4] ?? '';++ const name = attrs.name || attrs.property || '';++ if (name.trim().toLowerCase() === META_NAME && decodeEntities(attrs.content || '').trim() === token) return true;++ }++ return false;++}++++function createMerchant({ db, config, registry, fetcher, resolver = null, now = () => Date.now(), log = console }) {++ const dnsResolver = resolver || defaultResolver(config.merchant.dnsServers);++ const q = {++ byId: db.prepare('SELECT * FROM merchant_sources WHERE id = ?'),++ liveForDomain: db.prepare("SELECT * FROM merchant_sources WHERE domain = ? AND status IN ('pending','verified')"),++ forOwner: db.prepare('SELECT * FROM merchant_sources WHERE owner_sub = ? ORDER BY created_at DESC, id DESC'),++ insert: db.prepare(`INSERT INTO merchant_sources++ (id, owner_sub, owner_name, store_name, domain, method, token, listing_url, listing_kind, meta_url, status,++ created_at, updated_at)++ VALUES (@id, @owner_sub, @owner_name, @store_name, @domain, @method, @token, @listing_url, @listing_kind, @meta_url,++ 'pending', @now, @now)`),++ checked: db.prepare(`UPDATE merchant_sources SET last_checked_at = ?, check_count = check_count + 1,++ last_error = ?, updated_at = ? WHERE id = ?`),++ verified: db.prepare(`UPDATE merchant_sources SET status = 'verified', source_key = ?, terms_note = ?, verified_at = ?,++ verified_by = ?, verification_detail = ?, last_checked_at = ?, check_count = check_count + 1, last_error = NULL,++ updated_at = ? WHERE id = ? AND status = 'pending'`),++ failed: db.prepare("UPDATE merchant_sources SET status = 'failed', last_error = ?, updated_at = ? WHERE id = ? AND status = 'pending'"),++ withdraw: db.prepare(`UPDATE merchant_sources SET status = 'withdrawn', withdrawn_at = ?, withdrawn_by = ?, updated_at = ?++ WHERE id = ? AND status != 'withdrawn'`),++ duePending: db.prepare(`SELECT * FROM merchant_sources WHERE status = 'pending'++ AND COALESCE(last_checked_at, 0) < ? ORDER BY COALESCE(last_checked_at, 0), created_at LIMIT ?`),++ };++++ function view(row) {++ if (!row) return null;++ const challenge = row.method === 'dns'++ ? { type: 'TXT', name: `${CHALLENGE_PREFIX}.${row.domain}`, value: `${TXT_PREFIX}${row.token}`, note: 'Publish this record, then check now. You can remove it after verification.' }++ : { type: 'meta', name: META_NAME, value: row.token, url: row.meta_url || `https://${row.domain}/`, html: `<meta name="${META_NAME}" content="${row.token}">` };++ return {++ id: row.id,++ domain: row.domain,++ status: row.status,++ owner: { subject: row.owner_sub, name: row.owner_name || null },++ store_name: row.store_name || null,++ method: row.method,++ listing: { kind: row.listing_kind, url: row.listing_url },++ meta_url: row.meta_url || null,++ source_key: row.source_key || null,++ terms_note: row.terms_note || null,++ verification: {++ method: row.method,++ challenge,++ verified_at: iso(row.verified_at),++ verified_by: row.verified_by || null,++ detail: row.verification_detail || null,++ last_checked_at: iso(row.last_checked_at),++ check_count: Number(row.check_count || 0),++ last_error: row.last_error || null,++ },++ withdrawal: row.withdrawn_at ? { at: iso(row.withdrawn_at), by: row.withdrawn_by || null } : null,++ created_at: iso(row.created_at),++ updated_at: iso(row.updated_at),++ };++ }++++ async function load(principal, id) {++ if (!principal || !ids.isSubjectId('user', principal.subject)) throw new MerchantError(401, 'merchant.auth_required', 'a signed-in merchant is required');++ if (!ID_RE.test(String(id || ''))) throw new MerchantError(404, 'merchant.not_found', 'no such merchant source');++ const row = await q.byId.get(id);++ if (!row || row.owner_sub !== principal.subject) throw new MerchantError(404, 'merchant.not_found', 'no such merchant source');++ return row;++ }++++ async function add(principal, input = {}) {++ if (!principal || !ids.isSubjectId('user', principal.subject)) throw new MerchantError(401, 'merchant.auth_required', 'a signed-in merchant is required');++ const body = input && typeof input === 'object' && !Array.isArray(input) ? input : {};++ const domain = checkDomain(body.domain ?? body.hostname ?? body.host);++ const method = String(body.method ?? body.verification_method ?? '').trim().toLowerCase();++ if (!METHODS.includes(method)) throw new MerchantError(422, 'merchant.method_invalid', 'method must be dns or meta');++ const listingKind = String(body.listing_kind || body.listing || (body.sitemap_url ? 'sitemap' : '') || (body.feed_url ? 'feed' : '')).trim().toLowerCase();++ if (!LISTING_KINDS.includes(listingKind)) throw new MerchantError(422, 'merchant.listing_invalid', 'listing_kind must be sitemap or feed');++ const listingUrl = checkUrl(body.listing_url ?? body.url ?? body.sitemap_url ?? body.feed_url, 'listing_url');++ if (!sameSite(domain, new URL(listingUrl).hostname)) throw new MerchantError(422, 'merchant.listing_host', `the ${listingKind} must be on ${domain} or one of its subdomains`);++ let metaUrl = null;++ if (method === 'meta') {++ const chosenMeta = body.meta_url ?? body.verification_url;++ metaUrl = chosenMeta ? checkUrl(chosenMeta, 'meta_url') : `https://${domain}/`;++ if (!sameSite(domain, new URL(metaUrl).hostname)) throw new MerchantError(422, 'merchant.meta_host', `the meta tag page must be on ${domain} or one of its subdomains`);++ } else if (body.meta_url || body.verification_url) {++ throw new MerchantError(422, 'merchant.meta_unused', 'meta_url is only for method meta');++ }++ if ((body.store_name != null && typeof body.store_name !== 'string') || (body.name != null && typeof body.name !== 'string')) {++ throw new MerchantError(422, 'merchant.store_name_invalid', 'store_name must be text');++ }++ const storeName = body.store_name == null && body.name == null ? null : clip(body.store_name ?? body.name, 120) || null;++++ const live = await q.liveForDomain.get(domain);++ if (live) {++ if (live.owner_sub === principal.subject) throw new MerchantError(409, 'merchant.exists', `${domain} already has a ${live.status} claim on this account`);++ throw new MerchantError(409, 'merchant.domain_taken', `${domain} is already claimed`);++ }++ const t = now();++ const row = {++ id: `msrc_${ids.ulid(t)}`,++ owner_sub: principal.subject,++ owner_name: principal.name ? clip(principal.name, 80) : null,++ store_name: storeName,++ domain,++ method,++ token: crypto.randomBytes(20).toString('hex'),++ listing_url: listingUrl,++ listing_kind: listingKind,++ meta_url: metaUrl,++ now: t,++ };++ try { await q.insert.run(row); } catch (err) {++ if (err && err.code === '23505') throw new MerchantError(409, 'merchant.domain_taken', `${domain} is already claimed`);++ throw err;++ }++ return await q.byId.get(row.id);++ }++++ async function lookupDns(row) {++ let records;++ try { records = await dnsResolver.resolveTxt(`${CHALLENGE_PREFIX}.${row.domain}`); } catch (err) {++ const code = err && err.code;++ if (code === 'ENOTFOUND' || code === 'ENODATA' || code === 'NXDOMAIN') {++ return { ok: false, error: `no TXT record at ${CHALLENGE_PREFIX}.${row.domain} yet` };++ }++ return { ok: false, error: `DNS lookup failed (${code || 'error'})` };++ }++ const values = (records || []).map(chunks => (Array.isArray(chunks) ? chunks.join('') : String(chunks)));++ const expected = `${TXT_PREFIX}${row.token}`;++ if (values.includes(expected)) return { ok: true, detail: `DNS TXT record ${CHALLENGE_PREFIX}.${row.domain} matched` };++ return { ok: false, error: values.length ? `${CHALLENGE_PREFIX}.${row.domain} has TXT records, but not the expected value` : `no TXT record at ${CHALLENGE_PREFIX}.${row.domain} yet` };++ }++++ async function lookupMeta(row) {++ const urls = row.meta_url ? [row.meta_url] : [`https://${row.domain}/`, `http://${row.domain}/`];++ let lastError = `no meta tag at ${urls[0]}`;++ for (const url of urls) {++ try {++ const res = await fetcher.fetchUrl(url, {++ headers: { Accept: 'text/html, application/xhtml+xml;q=0.9, */*;q=0.1' },++ capBytes: config.merchant.metaMaxBytes,++ timeoutMs: config.merchant.metaTimeoutMs,++ });++ if (res.status !== 200) { lastError = `HTTP ${res.status} at ${url}`; continue; }++ const finalHost = new URL(res.finalUrl || url).hostname;++ if (!sameSite(row.domain, finalHost)) { lastError = `redirected outside ${row.domain} (${finalHost})`; continue; }++ const text = decodeBody(res.body, res.headers['content-type']);++ if (hasVerificationMeta(text, row.token)) return { ok: true, detail: `meta tag at ${url} matched` };++ lastError = `no ${META_NAME} meta tag at ${url}`;++ } catch (err) {++ lastError = `${url}: ${err.code || 'fetch failed'}`;++ }++ }++ return { ok: false, error: lastError };++ }++++ /** Check the proof and, when present, create the source in the same transaction that verifies the claim. */++ async function check(row, actor) {++ const result = row.method === 'dns' ? await lookupDns(row) : await lookupMeta(row);++ const checkedAt = now();++ if (!result.ok) {++ await q.checked.run(checkedAt, clip(result.error, 500) || 'not verified', checkedAt, row.id);++ return { row: await q.byId.get(row.id), verified: false, source: null };++ }++ const terms = termsNoteFor(row, checkedAt);++ const sourceInput = sourceFor(row, terms, config);++ let source = null;++ try {++ await db.tx(async () => {++ const fresh = await db.prepare('SELECT * FROM merchant_sources WHERE id = ? FOR UPDATE').get(row.id);++ if (!fresh || fresh.status !== 'pending') return;++ source = await registry.create(sourceInput, fresh.owner_sub);++ const changed = await q.verified.run(source.key, terms, checkedAt, actor && actor.subject ? actor.subject : fresh.owner_sub,++ clip(result.detail, 500), checkedAt, checkedAt, row.id);++ if (!changed || changed.changes === 0) throw new Error('the claim changed while it was being verified');++ });++ } catch (err) {++ await q.checked.run(now(), clip(`verification found the proof, but the source was not created: ${err.message}`, 500), now(), row.id);++ if (err && typeof err.code === 'string' && err.code.startsWith('sources.')) {++ throw new MerchantError(422, err.code, `the source was not created: ${err.message}`);++ }++ throw err;++ }++ const final = await q.byId.get(row.id);++ return { row: final, verified: Boolean(final && final.status === 'verified'), source };++ }++++ async function verify(principal, id) {++ const row = await load(principal, id);++ if (row.status === 'verified') return { row, verified: true, source: null, already: true };++ if (row.status === 'withdrawn') throw new MerchantError(409, 'merchant.withdrawn', 'this claim was withdrawn; add the domain again to opt in');++ if (row.status === 'failed') throw new MerchantError(409, 'merchant.failed', 'this claim failed before verification; add the domain again to opt in');++ return await check(row, principal);++ }++++ async function withdraw(principal, id) {++ const row = await load(principal, id);++ if (row.status === 'withdrawn') return row;++ const t = now();++ await db.tx(async () => {++ if (row.source_key) {++ const source = await registry.get(row.source_key);++ // The terms note stays as it was: the history of the permission is the point.++ if (source) await registry.patch(row.source_key, { enabled: false }, principal.subject);++ }++ await q.withdraw.run(t, principal.subject, t, row.id);++ });++ return await q.byId.get(row.id);++ }++++ async function list(principal) {++ if (!principal || !ids.isSubjectId('user', principal.subject)) throw new MerchantError(401, 'merchant.auth_required', 'a signed-in merchant is required');++ return await q.forOwner.all(principal.subject);++ }++++ /** Worker pass: pending claims are checked again; a claim that never verifies fails after merchant.pendingDays. */++ async function recheck({ limit = 50 } = {}) {++ const t = now();++ const summary = { checked: 0, verified: 0, failed: 0 };++ for (const row of await q.duePending.all(t - config.merchant.recheckIntervalMs + 1000, limit)) {++ if (t - row.created_at > config.merchant.pendingDays * DAY) {++ await q.failed.run(`not verified within ${config.merchant.pendingDays} days`, t, row.id);++ summary.failed++;++ continue;++ }++ try {++ const out = await check(row, { subject: 'svc:sources' });++ summary.checked++;++ if (out.row && out.row.status === 'verified') summary.verified++;++ } catch (err) {++ log.error(`[merchant] recheck ${row.id}: ${err.stack || err}`);++ }++ }++ return summary;++ }++++ return {++ add, list, load, verify, withdraw, recheck, view,++ get: async (principal, id) => await load(principal, id),++ getById: async (id) => (ID_RE.test(String(id || '')) ? await q.byId.get(id) || null : null),++ };++}++++module.exports = {++ createMerchant, MerchantError, checkDomain, checkUrl, sameSite, sourceKeyFor, termsNoteFor, hasVerificationMeta,++ isHostname, CHALLENGE_PREFIX, TXT_PREFIX, META_NAME, METHODS, LISTING_KINDS,++};nnew file mode 100644@@@ -0,0 +1,220 @@++'use strict';++/**++ * The merchant opt-in pages: plain semantic markup built with the OpenVibe Kit (openvibe-shared/kit), so the page keeps++ * the network's classes and the main session styles the pieces it adds. No client-side state: every action is a form.++ */++const kit = require('openvibe-shared/kit');++++const esc = kit.esc;++++const STATUS = {++ pending: ['pending', 'warn'],++ verified: ['verified', 'ok'],++ withdrawn: ['withdrawn', ''],++ failed: ['failed', 'bad'],++};++++const statusBadge = (status) => {++ const [label, tone] = STATUS[status] || [status, ''];++ return kit.badge(label, tone);++};++++const when = (iso) => {++ if (!iso) return '';++ const t = kit.when(iso);++ return t.text ? `<time datetime="${esc(t.iso)}">${esc(t.text)}</time>` : '';++};++++const csrfInput = (csrf) => `<input type="hidden" name="csrf" value="${esc(csrf)}">`;++++function layout({ title = 'List your store on OpenVibe', body = '', description = '' }) {++ return `<!doctype html>++<html lang="en">++<head>++<meta charset="utf-8">++<meta name="viewport" content="width=device-width, initial-scale=1">++<meta name="robots" content="noindex, nofollow">++<meta name="color-scheme" content="dark light">++<title>${esc(title)} · OpenVibe.Sources</title>++${description ? `<meta name="description" content="${esc(description)}">` : ''}++<script src="/shared/theme-loader.js"></script>++<link rel="stylesheet" href="/shared/kit.css">++<style>html{background:var(--ovk-bg);color:var(--ovk-text)}body{margin:0}</style>++</head>++<body>++<main class="ovk-page">++${body}++</main>++<script src="/shared/kit-web.js" defer></script>++</body>++</html>`;++}++++function notice(kind, text) {++ return text ? kit.notice({ kind, text }) : '';++}++++function guestPage({ error = '', authError = '' } = {}) {++ const body = `${kit.head({ title: 'List your store on OpenVibe', sub: 'Free, with honest ranking and no pay-to-win.' })}++${notice('error', error)}++${authError ? kit.notice({ kind: 'warn', text: `Sign-in did not complete (${authError}). Please try again.` }) : ''}++${kit.section({++ title: 'How it works',++ body: `<ol class="ovk-list">++<li class="ovk-row ovk-row--static"><span class="ovk-row-main"><span class="ovk-row-top"><strong>1. Claim your domain</strong></span><span class="ovk-row-sub"><span>Sign in, add the domain your shop uses, and name the sitemap or product feed we may read.</span></span></span></li>++<li class="ovk-row ovk-row--static"><span class="ovk-row-main"><span class="ovk-row-top"><strong>2. Prove you control it</strong></span><span class="ovk-row-sub"><span>Publish a DNS TXT record or a meta tag. Nothing is crawled or enabled until the proof is found.</span></span></span></li>++<li class="ovk-row ovk-row--static"><span class="ovk-row-main"><span class="ovk-row-top"><strong>3. We crawl politely</strong></span><span class="ovk-row-sub"><span>robots.txt is checked on every request. We read what your pages publish for machines, with provenance.</span></span></span></li>++<li class="ovk-row ovk-row--static"><span class="ovk-row-main"><span class="ovk-row-top"><strong>4. You stay in control</strong></span><span class="ovk-row-sub"><span>Withdraw at any time: the source is disabled and the permission note is kept for the audit trail.</span></span></span></li>++</ol>`,++ })}++${kit.section({++ title: 'Sign in to start',++ body: `<p class="ovk-note">A merchant signs in with an OpenVibe.Network account. The claim belongs to that account.</p>++<p>${kit.button({ href: '/merchant/login?next=%2Fmerchant', label: 'Sign in with OpenVibe', icon: 'user' })}</p>`,++ })}`;++ return layout({ body });++}++++function claimRow(claim) {++ const source = claim.source;++ const sub = `${claim.listing.kind === 'sitemap' ? 'Sitemap' : 'Product feed'} · ${claim.listing.url}`;++ const trail = source ? statusBadge(source.health.status) : '';++ return kit.row({++ href: `/merchant/sources/${claim.id}`,++ title: claim.domain,++ badge: statusBadge(claim.status),++ sub,++ meta: claim.created_at ? kit.when(claim.created_at).text : '',++ metaIso: claim.created_at || '',++ trail,++ });++}++++function merchantPage({ merchant, claims = [], csrf, notice: noticeText = '', error = '', authError = '' }) {++ const list = claims.length++ ? kit.list(claims.map(claimRow), { title: 'Your merchant sources' })++ : kit.empty({ title: 'No store listed yet', text: 'Add your first domain below. It becomes a source only after you prove you control it and the proof is found.', icon: 'globe' });++++ const form = `<form class="ovk-form" method="post" action="/merchant/sources">++${csrfInput(csrf)}++<label>Store domain <input name="domain" required maxlength="253" autocomplete="off" placeholder="shop.example.com" spellcheck="false"></label>++<label>Store name (optional) <input name="store_name" maxlength="120" autocomplete="organization" placeholder="Example Shop"></label>++${kit.seg({ name: 'method', label: 'How will you prove control?', value: 'dns', options: [['dns', 'DNS TXT record'], ['meta', 'Meta tag']] })}++${kit.selectRow({ name: 'listing_kind', label: 'What should we read?', hint: 'A sitemap of product pages, or an RSS/Atom product feed.', options: [['sitemap', 'Sitemap'], ['feed', 'Product feed (RSS/Atom)']], value: 'sitemap' })}++<label>Sitemap or product feed URL <input name="listing_url" type="url" required maxlength="2048" autocomplete="url" placeholder="https://shop.example.com/sitemap.xml" spellcheck="false"></label>++<label>Meta tag page (only for method meta; defaults to your home page) <input name="meta_url" type="url" maxlength="2048" autocomplete="url" placeholder="https://shop.example.com/" spellcheck="false"></label>++<p class="ovk-note">Nothing is created or enabled before the proof is found. robots.txt is still obeyed after that.</p>++${kit.button({ label: 'Add domain', icon: 'plus' })}++</form>`;++++ const body = `${kit.head({++ title: 'List your store on OpenVibe',++ sub: `Signed in as ${merchant.name || merchant.subject}.`,++ actions: [kit.button({ href: '/merchant/logout', label: 'Sign out', kind: 'quiet', icon: 'leave', small: true })],++ })}++${notice('ok', noticeText)}++${notice('error', error)}++${authError ? kit.notice({ kind: 'warn', text: `Sign-in did not complete (${authError}). Please try again.` }) : ''}++${kit.section({ title: 'Add a store', body: form })}++${list}++<p class="ovk-note">A full merchant dashboard with charts and history lives in OpenVibe.Report later. This page is the opt-in flow; the same sources are readable as JSON at <code>/api/v1/merchant/sources</code> with your session or a first-party service token.</p>`;++ return layout({ body });++}++++function sourcePanel(claim) {++ const source = claim.source;++ const terms = claim.terms_note++ ? `<h3>Terms note</h3><p class="ovk-note">${esc(claim.terms_note)}</p>`++ : '';++ if (!source) return `${terms}<p class="ovk-note">No source exists yet. Verification creates it; nothing is fetched before that.</p>`;++ const health = source.health;++ const last = claim.last_run;++ const crawl = claim.last_crawl;++ const runs = claim.recent_runs && claim.recent_runs.length++ ? `<h3>Last fetches</h3><ul class="ovk-list">${claim.recent_runs.map(r => `<li class="ovk-row ovk-row--static"><span class="ovk-row-main"><span class="ovk-row-top"><strong>${esc(r.state)}</strong><span class="ovk-row-meta">${esc(r.started_at ? kit.when(r.started_at).text : '')}</span></span><span class="ovk-row-sub"><span>${esc(r.endpoint_url || '')} · ${Number(r.items.seen)} seen, ${Number(r.items.created)} created</span></span></span></li>`).join('')}</ul>`++ : '<p class="ovk-note">No fetch has run yet.</p>';++ const crawler = crawl++ ? `<p>${Number(crawl.pages)} page(s) known, ${Number(crawl.due)} due${crawl.next_due_at ? ` · next ${when(crawl.next_due_at)}` : ''}.</p>++${Object.keys(crawl.pages_by_state || {}).length ? `<p class="ovk-note">Pages: ${Object.entries(crawl.pages_by_state).map(([k, v]) => `${esc(k)} ${Number(v)}`).join(' · ')}</p>` : ''}++${crawl.recent && crawl.recent.length ? `<ul class="ovk-list">${crawl.recent.map(p => `<li class="ovk-row ovk-row--static"><span class="ovk-row-main"><span class="ovk-row-top"><strong>${esc(p.url)}</strong><span class="ovk-row-meta">${esc(p.state)}</span></span><span class="ovk-row-sub"><span>${Number(p.items)} item(s) · ${p.http_status == null ? 'not fetched' : `HTTP ${Number(p.http_status)}`}</span></span></span></li>`).join('')}</ul>` : ''}`++ : '';++ return `${claim.store_name ? `<p class="ovk-note">${esc(claim.store_name)}</p>` : ''}${terms}++<dl class="ovk-stats">++<div><dt>source</dt><dd><code>${esc(source.key)}</code></dd></div>++<div><dt>type</dt><dd>${esc(source.type)}</dd></div>++<div><dt>enabled</dt><dd>${source.enabled ? 'yes' : 'no'}</dd></div>++<div><dt>health</dt><dd>${esc(health.status)}</dd></div>++</dl>++${health.last_success_at ? `<p class="ovk-note">Last success ${when(health.last_success_at)}${health.last_state ? ` · last state ${esc(health.last_state)}` : ''}.</p>` : ''}++${last ? `<p class="ovk-note">Last run ${esc(last.state)} at ${when(last.started_at)}${last.detail ? `: ${esc(last.detail)}` : ''}.</p>` : ''}++${crawler}++${runs}`;++}++++function detailPage({ merchant, claim, csrf, notice: noticeText = '', noticeKind = 'ok', error = '' }) {++ const challenge = claim.verification.challenge;++ const pending = claim.status === 'pending';++ const verified = claim.status === 'verified';++ const withdrawn = claim.status === 'withdrawn';++ const failed = claim.status === 'failed';++++ let proof = '';++ if (pending) {++ const how = claim.method === 'dns'++ ? `<p>Publish this DNS record at your DNS provider:</p>++${kit.snippet({ code: `${challenge.name} IN TXT "${challenge.value}"` })}++<dl class="ovk-stats">++<div><dt>type</dt><dd>TXT</dd></div>++<div><dt>name</dt><dd><code>${esc(challenge.name)}</code></dd></div>++<div><dt>value</dt><dd><code>${esc(challenge.value)}</code></dd></div>++</dl>`++ : `<p>Put this tag in the <code><head></code> of <a href="${esc(challenge.url)}" rel="noopener">${esc(challenge.url)}</a>:</p>++${kit.snippet({ code: challenge.html })}++<p class="ovk-note">The page must stay on ${esc(claim.domain)} (a redirect to another host is refused).</p>`;++ proof = `${how}++<div>${kit.action({ action: `/merchant/sources/${claim.id}/verify`, hidden: { csrf }, label: 'Check now', icon: 'retry' })}</div>++${claim.verification.last_error ? kit.notice({ kind: 'warn', text: claim.verification.last_error }) : ''}`;++ if (claim.verification.last_checked_at) proof += `<p class="ovk-note">Checked ${Number(claim.verification.check_count || 0)} time(s); last ${when(claim.verification.last_checked_at)}.</p>`;++ } else if (verified) {++ proof = kit.notice({ kind: 'ok', text: `Verified ${claim.verification.detail || (claim.method === 'dns' ? 'by DNS TXT record' : 'by meta tag')} at ${claim.verification.verified_at}. The source is enabled.` });++ proof += `<p class="ovk-note">You can keep the ${claim.method === 'dns' ? 'TXT record' : 'meta tag'} in place, but it is no longer needed to keep the source enabled. Withdrawing is the explicit way to stop it.</p>`;++ } else if (withdrawn) {++ proof = kit.notice({ kind: 'info', text: `Permission withdrawn at ${claim.withdrawal && claim.withdrawal.at ? claim.withdrawal.at : claim.updated_at}. The source is disabled and its terms note is kept.` });++ } else if (failed) {++ proof = kit.notice({ kind: 'error', text: `This claim failed before verification: ${claim.verification.last_error || 'the proof was not found in time'}. Add the domain again to opt in.` });++ }++++ const actions = [];++ if (!withdrawn) {++ actions.push(kit.action({++ action: `/merchant/sources/${claim.id}/withdraw`,++ hidden: { csrf },++ label: 'Withdraw permission',++ icon: 'ban',++ kind: 'danger',++ confirm: 'Withdraw this permission? The source is disabled and the permission note is kept.',++ }));++ }++++ const body = `${kit.head({++ title: claim.domain,++ sub: `${claim.listing.kind === 'sitemap' ? 'Sitemap' : 'Product feed'} · ${claim.listing.url}`,++ back: { href: '/merchant', label: 'Merchant opt-in' },++ actions: [statusBadge(claim.status)],++ })}++${notice(noticeKind, noticeText)}++${notice('error', error)}++${kit.section({ title: pending ? 'Prove you control this domain' : 'Verification', body: proof })}++${kit.section({ title: 'Source and last crawl', body: sourcePanel(claim) })}++${actions.length ? kit.section({ title: 'Withdraw', body: actions.join('') }) : ''}`;++ return layout({ title: `${claim.domain} · Merchant opt-in`, body });++}++++function errorPage({ status = 400, message = 'Something went wrong.' } = {}) {++ const body = `${kit.head({ title: 'Merchant opt-in' })}++${kit.notice({ kind: 'error', text: message })}++<p><a href="/merchant">Back to merchant opt-in</a></p>`;++ return { status, html: layout({ title: 'Merchant opt-in', body }) };++}++++module.exports = { layout, guestPage, merchantPage, detailPage, errorPage, statusBadge, claimRow };@@@ -30,11 +30,23 @@ function serviceToken(slug, cap, { aud = 'openvibe.sources', exp = Math.floor(Da}, key);}++/** A person's Network session token: user id in `sub`, the canonical usr_… id in `subject_id`, no principal/typed claims. */++function userToken(subject, { username = 'merchant', exp = Math.floor(Date.now() / 1000) + 300, key = privateKey, extra = {} } = {}) {++ const now = Math.floor(Date.now() / 1000);++ const header = { alg: 'RS256', typ: 'JWT' };++ const claims = {++ iss: ISSUER, sub: String(now % 1000000), subject_id: subject, username, display_name: username,++ aud: ['openvibe.network'], iat: now, exp, jti: `usr_tok_${crypto.randomBytes(8).toString('hex')}`, ...extra,++ };++ const input = `${Buffer.from(JSON.stringify(header)).toString('base64url')}.${Buffer.from(JSON.stringify(claims)).toString('base64url')}`;++ return `${input}.${crypto.sign('RSA-SHA256', Buffer.from(input), key).toString('base64url')}`;++}++/*** Boot Sources; the scheduler is off unless worker: 'on'. Loopback is allowlisted for the stubs.* log: Sources' logger (default silent); limitsNow: the per-actor limiter's clock (default the wall clock).*/--async function boot({ env = {}, worker = 'off', lookupImpl, now, log = silent, limitsNow = null } = {}) {++async function boot({ env = {}, worker = 'off', lookupImpl, resolver = null, now, log = silent, limitsNow = null } = {}) {// A stub Network JWKS serving the generated signing key, so the SDK's JWKS client has something real// to fetch and verify against (no test touches the internet). Its URL is overridable via env.const jwksSite = await site({ '/api/.well-known/jwks': () => ({ body: JSON.stringify({ keys: [publicJwk] }) }) });@@@ -47,20 +59,28 @@ async function boot({ env = {}, worker = 'off', lookupImpl, now, log = silent, lSOURCES_HOST_MIN_INTERVAL_MS: '0',SOURCES_FETCH_TIMEOUT_MS: '1500',SOURCES_TICK_MS: '50',++ SOURCES_MERCHANT_RECHECK_MS: '0',...env,});// One database per boot (PGlite, or SOURCES_TEST_STORE=pg: the containers), dropped when the boot stops.const testdb = await require('./db').testDb();-- const h = await start({ config, db: testdb.db, log, lookupImpl, limitsNow, ...(now ? { now } : {}) });++ const h = await start({ config, db: testdb.db, log, lookupImpl, merchantResolver: resolver, limitsNow, ...(now ? { now } : {}) });const base = `http://127.0.0.1:${h.server.address().port}`;return { ...h, base, jwksSite, async stop() { await h.close(); await testdb.close(); await jwksSite.close(); } };}--async function request(base, method, p, { token, body, headers = {} } = {}) {++async function request(base, method, p, { token, body, form, headers = {}, redirect = 'follow' } = {}) {const h = { ...headers };if (token) h.Authorization = `Bearer ${token}`;-- if (body !== undefined) h['Content-Type'] = 'application/json';-- const res = await fetch(base + p, { method, headers: h, body: body !== undefined ? JSON.stringify(body) : undefined });++ let payload;++ if (form !== undefined) {++ h['Content-Type'] = 'application/x-www-form-urlencoded';++ payload = new URLSearchParams(form).toString();++ } else if (body !== undefined) {++ h['Content-Type'] = 'application/json';++ payload = JSON.stringify(body);++ }++ const res = await fetch(base + p, { method, headers: h, body: payload, redirect });const text = await res.text();let json = null;try { json = text ? JSON.parse(text) : null; } catch { json = null; }@@@ -135,4 +155,4 @@ function suite(name) {return t;}--module.exports = { ISSUER, privateKey, publicKey, silent, serviceToken, boot, request, site, sourceDef, rss, fixture, sleep, suite };++module.exports = { ISSUER, privateKey, publicKey, silent, serviceToken, userToken, boot, request, site, sourceDef, rss, fixture, sleep, suite };nnew file mode 100644@@@ -0,0 +1,246 @@++'use strict';++/**++ * Merchant opt-in (plan §4.4 channel 1): a signed-in merchant proves a domain with DNS or a meta tag, a source is++ * created only then with an explicit-permission terms note, the crawler still obeys robots.txt, and withdrawal++ * disables the source while keeping the note. Also covers the JSON read Report calls, including last crawl stats.++ */++const assert = require('assert');++const { ids } = require('openvibe-contracts');++const { boot, request, serviceToken, userToken, site, fixture, suite } = require('./helpers');++++const t = suite('merchant-optin');++let svc;++let shopWeb;++let metaWeb;++let blockedWeb;++let dns;++let shopClaim;++let shopSourceKey;++++const ALICE = ids.newId('user');++const BOB = ids.newId('user');++const ALICE_TOKEN = userToken(ALICE, { username: 'alice' });++const BOB_TOKEN = userToken(BOB, { username: 'bob' });++const READER = serviceToken('report', ['sources.source.read']);++const ADMIN = serviceToken('network', ['sources.source.manage']);++++const api = (method, p, opts = {}) => request(svc.base, method, p, opts);++++t('boot: a stub shop, meta site and a robots-closed site, with an injectable TXT resolver', async () => {++ dns = {++ records: new Map(),++ async resolveTxt(name) {++ if (!this.records.has(name)) { const err = new Error('no record'); err.code = 'ENOTFOUND'; throw err; }++ return this.records.get(name);++ },++ set(domain, value) { this.records.set(`_openvibe-sources.${domain}`, [[value]]); },++ };++ // Every stub name resolves to the loopback server; SOURCES_ALLOW_PRIVATE_HOSTS exempts the names below.++ const lookupImpl = (_host, _opts, cb) => cb(null, [{ address: '127.0.0.1', family: 4 }]);++ shopWeb = await site({});++ metaWeb = await site({});++ blockedWeb = await site({});++ const shopBase = `http://shop.example.com:${new URL(shopWeb.origin).port}`;++ const metaBase = `http://meta.example.com:${new URL(metaWeb.origin).port}`;++ const blockedBase = `http://blocked.example.com:${new URL(blockedWeb.origin).port}`;++++ shopWeb.routes['/robots.txt'] = () => ({ body: 'User-agent: *\nAllow: /\n', headers: { 'content-type': 'text/plain' } });++ shopWeb.routes['/sitemap.xml'] = () => ({++ body: `<?xml version="1.0"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"><url><loc>${shopBase}/products/1</loc></url></urlset>`,++ headers: { 'content-type': 'application/xml' },++ });++ shopWeb.routes['/products/1'] = () => ({ body: fixture('product.html'), headers: { 'content-type': 'text/html' } });++++ let metaToken = '';++ metaWeb.routes['/robots.txt'] = () => ({ body: 'User-agent: *\nAllow: /\n', headers: { 'content-type': 'text/plain' } });++ metaWeb.routes['/'] = () => ({++ body: `<!doctype html><html><head><meta name="openvibe-sources-verification" content="${metaToken}"><title>Meta Shop</title></head><body>meta</body></html>`,++ headers: { 'content-type': 'text/html' },++ });++ metaWeb.routes['/feed.xml'] = () => ({++ body: '<?xml version="1.0"?><rss version="2.0"><channel><title>Meta Shop</title><item><guid>p1</guid><title>Product one</title><link>https://meta.example.com/p/1</link><description>One.</description></item></channel></rss>',++ headers: { 'content-type': 'application/rss+xml' },++ });++++ blockedWeb.routes['/robots.txt'] = () => ({ body: 'User-agent: *\nDisallow: /\n', headers: { 'content-type': 'text/plain' } });++ blockedWeb.routes['/sitemap.xml'] = () => ({ body: '<?xml version="1.0"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"></urlset>', headers: { 'content-type': 'application/xml' } });++++ svc = await boot({++ resolver: dns,++ lookupImpl,++ env: {++ BASE_URL: 'https://sources.test',++ SOURCES_ALLOW_PRIVATE_HOSTS: '127.0.0.1,shop.example.com,meta.example.com,blocked.example.com',++ },++ });++ svc.metaBase = metaBase;++ svc.shopBase = shopBase;++ svc.blockedBase = blockedBase;++ svc.setMetaToken = (v) => { metaToken = v; };++});++++t('a pending claim creates no source; verification creates one with who/when/how in its terms note', async () => {++ const c = await api('POST', '/api/v1/merchant/sources', { token: ALICE_TOKEN, body: {++ domain: 'shop.example.com', store_name: 'Example Shop', method: 'dns', listing_kind: 'sitemap',++ listing_url: `${svc.shopBase}/sitemap.xml`,++ } });++ assert.strictEqual(c.status, 201, c.text);++ shopClaim = c.body.merchant_source;++ assert.deepStrictEqual([shopClaim.status, shopClaim.source_key, shopClaim.method], ['pending', null, 'dns']);++ assert.strictEqual(shopClaim.verification.challenge.name, '_openvibe-sources.shop.example.com');++ assert.match(shopClaim.verification.challenge.value, /^openvibe-sources-verification=[a-f0-9]{40}$/);++ assert.strictEqual((await svc.db.prepare("SELECT COUNT(*) AS n FROM sources WHERE key LIKE 'merchant-%'").get()).n, 0, 'no source before verification');++++ const before = await api('POST', `/api/v1/merchant/sources/${shopClaim.id}/verify`, { token: ALICE_TOKEN });++ assert.strictEqual(before.status, 200);++ assert.deepStrictEqual([before.body.verified, before.body.merchant_source.status, before.body.merchant_source.source_key], [false, 'pending', null]);++ assert.match(before.body.merchant_source.verification.last_error, /no TXT record/);++++ dns.set('shop.example.com', shopClaim.verification.challenge.value);++ const after = await api('POST', `/api/v1/merchant/sources/${shopClaim.id}/verify`, { token: ALICE_TOKEN });++ assert.strictEqual(after.status, 200, after.text);++ assert.deepStrictEqual([after.body.verified, after.body.merchant_source.status, after.body.source_created], [true, 'verified', true]);++ shopSourceKey = after.body.merchant_source.source_key;++ assert.match(shopSourceKey, /^merchant-shop-example-com-[a-z0-9]{8}$/);++ const terms = after.body.merchant_source.terms_note;++ assert.ok(terms.length <= 2000);++ for (const bit of [ALICE, 'alice', 'explicit permission', 'shop.example.com', 'sitemap', 'DNS TXT record', 'robots.txt', 'withdraw']) {++ assert.ok(terms.includes(bit), `terms note should mention ${bit}`);++ }++++ const source = await api('GET', `/api/v1/sources/${shopSourceKey}`, { token: READER });++ assert.strictEqual(source.status, 200);++ assert.deepStrictEqual([source.body.source.enabled, source.body.source.type, source.body.source.category], [true, 'crawl', 'deals']);++ assert.strictEqual(source.body.source.endpoints[0].url, `${svc.shopBase}/sitemap.xml`);++ assert.strictEqual(source.body.source.terms_note, terms);++ assert.match(source.body.source.robots_note, /robots\.txt/);++ assert.strictEqual(source.body.source.crawl.sitemaps, false, 'the merchant named this sitemap; no guessed ones');++});++++t('the crawl reads the named sitemap and the merchant read reports its last run and frontier', async () => {++ const out = await svc.ingest.run(shopSourceKey, { trigger: 'manual' });++ assert.deepStrictEqual(out.runs.map(r => r.state), ['ok', 'ok']);++ assert.strictEqual(out.runs[1].items_created, 1);++++ const read = await api('GET', '/api/v1/merchant/sources', { token: ALICE_TOKEN });++ assert.strictEqual(read.status, 200);++ const listed = read.body.sources.find(s => s.id === shopClaim.id);++ assert.ok(listed);++ assert.deepStrictEqual([listed.source.enabled, listed.source.health.status], [true, 'healthy']);++ assert.strictEqual(listed.last_run.state, 'ok');++ assert.strictEqual(listed.last_run.items.created, 1);++ assert.strictEqual(listed.last_crawl.pages, 1);++ assert.strictEqual(listed.last_crawl.pages_by_state.kept, 1);++ assert.ok(listed.last_crawl.recent.some(p => p.url.endsWith('/products/1') && p.state === 'kept'));++++ // Report reads the same by naming the merchant it acts for.++ const report = await api('GET', '/api/v1/merchant/sources', { token: READER, headers: { 'X-OV-Subject': ALICE } });++ assert.strictEqual(report.status, 200);++ assert.strictEqual(report.body.sources[0].last_crawl.pages, 1);++ assert.strictEqual((await api('GET', '/api/v1/merchant/sources', { token: READER })).status, 400);++ // Another merchant sees nothing of this claim, even by id.++ assert.strictEqual((await api('GET', `/api/v1/merchant/sources/${shopClaim.id}`, { token: BOB_TOKEN })).status, 404);++ assert.strictEqual((await api('GET', '/api/v1/merchant/sources', { token: BOB_TOKEN })).body.sources.length, 0);++});++++t('withdrawal disables the source and keeps the permission note', async () => {++ const before = await api('GET', `/api/v1/merchant/sources/${shopClaim.id}`, { token: ALICE_TOKEN });++ const w = await api('POST', `/api/v1/merchant/sources/${shopClaim.id}/withdraw`, { token: ALICE_TOKEN });++ assert.strictEqual(w.status, 200, w.text);++ assert.strictEqual(w.body.merchant_source.status, 'withdrawn');++ assert.strictEqual(w.body.merchant_source.terms_note, before.body.merchant_source.terms_note, 'the note is kept verbatim');++ assert.strictEqual(w.body.merchant_source.source.enabled, false);++ assert.strictEqual(w.body.merchant_source.source.health.status, 'disabled');++ assert.ok(w.body.merchant_source.withdrawal.at);++++ const source = await api('GET', `/api/v1/sources/${shopSourceKey}`, { token: READER });++ assert.deepStrictEqual([source.body.source.enabled, source.body.source.terms_note], [false, before.body.merchant_source.terms_note]);++ assert.strictEqual((await api('POST', `/api/v1/merchant/sources/${shopClaim.id}/verify`, { token: ALICE_TOKEN })).status, 409);++ // A withdrawn claim frees the domain: the merchant may opt in again (a new claim, a new source).++ assert.strictEqual((await api('POST', '/api/v1/merchant/sources', { token: ALICE_TOKEN, body: {++ domain: 'shop.example.com', method: 'dns', listing_kind: 'feed', listing_url: `${svc.shopBase}/feed.xml`,++ } })).status, 201);++});++++t('a meta tag proves control and a product feed becomes an enabled rss source', async () => {++ const c = await api('POST', '/api/v1/merchant/sources', { token: ALICE_TOKEN, body: {++ domain: 'meta.example.com', method: 'meta', meta_url: `${svc.metaBase}/`, listing_kind: 'feed',++ listing_url: `${svc.metaBase}/feed.xml`,++ } });++ assert.strictEqual(c.status, 201, c.text);++ const claim = c.body.merchant_source;++ assert.strictEqual(claim.verification.challenge.name, 'openvibe-sources-verification');++ const before = await api('POST', `/api/v1/merchant/sources/${claim.id}/verify`, { token: ALICE_TOKEN });++ assert.strictEqual(before.body.verified, false);++ assert.match(before.body.merchant_source.verification.last_error, /no openvibe-sources-verification meta tag/);++++ svc.setMetaToken(claim.verification.challenge.value);++ const after = await api('POST', `/api/v1/merchant/sources/${claim.id}/verify`, { token: ALICE_TOKEN });++ assert.strictEqual(after.status, 200, after.text);++ assert.deepStrictEqual([after.body.verified, after.body.merchant_source.status], [true, 'verified']);++ assert.match(after.body.merchant_source.terms_note, /meta tag at/);++ const source = await api('GET', `/api/v1/sources/${after.body.merchant_source.source_key}`, { token: READER });++ assert.deepStrictEqual([source.body.source.enabled, source.body.source.type], [true, 'rss']);++ const run = await svc.ingest.run(after.body.merchant_source.source_key, { trigger: 'manual' });++ assert.deepStrictEqual(run.runs.map(r => [r.state, r.items_created]), [['ok', 1]]);++});++++t('a merchant opt-in still obeys robots.txt: a disallowed site is never read', async () => {++ const c = await api('POST', '/api/v1/merchant/sources', { token: ALICE_TOKEN, body: {++ domain: 'blocked.example.com', method: 'dns', listing_kind: 'sitemap', listing_url: `${svc.blockedBase}/sitemap.xml`,++ } });++ const claim = c.body.merchant_source;++ dns.set('blocked.example.com', claim.verification.challenge.value);++ const v = await api('POST', `/api/v1/merchant/sources/${claim.id}/verify`, { token: ALICE_TOKEN });++ assert.strictEqual(v.body.merchant_source.status, 'verified');++ const key = v.body.merchant_source.source_key;++ const run = await svc.ingest.run(key, { trigger: 'manual' });++ assert.deepStrictEqual(run.runs.map(r => [r.state, r.items_seen, r.items_created]), [['robots_denied', 0, 0]]);++ assert.strictEqual((await svc.db.prepare('SELECT COUNT(*) AS n FROM items WHERE source_key = ?').get(key)).n, 0);++ const read = await api('GET', `/api/v1/merchant/sources/${claim.id}`, { token: ALICE_TOKEN });++ assert.strictEqual(read.body.merchant_source.last_run.state, 'robots_denied');++ assert.strictEqual(read.body.merchant_source.last_crawl.pages, 0);++});++++t('domains, listing URLs and duplicate claims are validated', async () => {++ const bad = async (body, re) => {++ const r = await api('POST', '/api/v1/merchant/sources', { token: ALICE_TOKEN, body });++ assert.strictEqual(r.status, 422, JSON.stringify(body) + ' -> ' + r.status + ' ' + r.text);++ assert.match(r.body.detail, re);++ };++ await bad({ domain: 'localhost', method: 'dns', listing_kind: 'sitemap', listing_url: 'https://localhost/sitemap.xml' }, /fully qualified/);++ await bad({ domain: 'shop.example.com', method: 'dns', listing_kind: 'sitemap', listing_url: 'https://evil.example.net/sitemap.xml' }, /on shop\.example\.com/);++ await bad({ domain: 'shop.example.com', method: 'dns', listing_kind: 'feed', listing_url: 'https://shop.example.com/feed.xml?api_key=secret' }, /credential/);++ await bad({ domain: 'shop.example.com', method: 'fax', listing_kind: 'sitemap', listing_url: 'https://shop.example.com/sitemap.xml' }, /method/);++ await bad({ domain: 'shop.example.com', method: 'dns', listing_kind: 'csv', listing_url: 'https://shop.example.com/sitemap.xml' }, /listing_kind/);++ // The second claim from the previous test is pending on shop.example.com.++ const dup = await api('POST', '/api/v1/merchant/sources', { token: BOB_TOKEN, body: {++ domain: 'shop.example.com', method: 'dns', listing_kind: 'sitemap', listing_url: 'https://shop.example.com/sitemap.xml',++ } });++ assert.strictEqual(dup.status, 409);++ assert.strictEqual(dup.body.code, 'merchant.domain_taken');++++ // `/claims` is the same resource; a first-party service may act for the merchant with X-OV-Subject, and needs manage.++ assert.strictEqual((await api('GET', '/api/v1/merchant/claims', { token: ALICE_TOKEN })).status, 200);++ assert.strictEqual((await api('POST', '/api/v1/merchant/claims', {++ token: READER, headers: { 'X-OV-Subject': ALICE }, body: {++ domain: 'report.example.com', method: 'dns', listing_kind: 'feed', listing_url: 'https://report.example.com/feed.xml',++ },++ })).status, 403);++ const delegated = await api('POST', '/api/v1/merchant/claims', {++ token: ADMIN, headers: { 'X-OV-Subject': ALICE }, body: {++ domain: 'report.example.com', method: 'dns', listing_kind: 'feed', listing_url: 'https://report.example.com/feed.xml',++ },++ });++ assert.strictEqual(delegated.status, 201, delegated.text);++ assert.strictEqual(delegated.body.merchant_source.owner.subject, ALICE);++});++++t('done', async () => {++ await shopWeb.close();++ await metaWeb.close();++ await blockedWeb.close();++ await svc.stop();++});++++t.run();nnew file mode 100644@@@ -0,0 +1,140 @@++'use strict';++/** The merchant opt-in pages: a guest sees how it works, a signed-in merchant adds/proves/withdraws with plain forms. */++const assert = require('assert');++const { ids } = require('openvibe-contracts');++const { boot, request, userToken, suite } = require('./helpers');++++const t = suite('merchant-pages');++let svc;++const ALICE = ids.newId('user');++const BOB = ids.newId('user');++const ALICE_TOKEN = userToken(ALICE, { username: 'alice' });++const BOB_TOKEN = userToken(BOB, { username: 'bob' });++let claimId;++let sourceKey;++++const cookies = (token) => ({ cookie: `ov_token=${token}` });++const page = (method, p, token, opts = {}) => request(svc.base, method, p, { ...opts, headers: { ...(token ? cookies(token) : {}), ...(opts.headers || {}) } });++const csrfOf = (html) => {++ const m = String(html).match(/name="csrf" value="([^"]+)"/);++ assert.ok(m, 'the page carries a CSRF token');++ return m[1];++};++++t('boot with an injectable DNS resolver', async () => {++ const records = new Map();++ svc = await boot({++ resolver: {++ async resolveTxt(name) {++ if (!records.has(name)) { const err = new Error('none'); err.code = 'ENOTFOUND'; throw err; }++ return records.get(name);++ },++ },++ env: { BASE_URL: 'https://sources.test' },++ });++ svc.dns = records;++});++++t('a guest sees the pitch and a sign-in link, not the form', async () => {++ const r = await page('GET', '/merchant');++ assert.strictEqual(r.status, 200);++ assert.match(r.text, /List your store on OpenVibe/);++ assert.match(r.text, /Sign in with OpenVibe/);++ assert.match(r.text, /DNS TXT record or a meta tag/);++ assert.doesNotMatch(r.text, /name="listing_url"/);++ assert.strictEqual((await page('POST', '/merchant/sources', null, { redirect: 'manual', form: {} })).status, 303);++});++++t('a signed-in merchant adds a domain and sees the DNS instructions', async () => {++ const list = await page('GET', '/merchant', ALICE_TOKEN);++ assert.strictEqual(list.status, 200);++ assert.match(list.text, /Signed in as alice/);++ assert.match(list.text, /Add a store/);++ const csrf = csrfOf(list.text);++ const created = await page('POST', '/merchant/sources', ALICE_TOKEN, {++ redirect: 'manual',++ form: {++ csrf, domain: 'shop.example.com', store_name: 'Example Shop', method: 'dns', listing_kind: 'sitemap',++ listing_url: 'https://shop.example.com/sitemap.xml', meta_url: '',++ },++ });++ assert.strictEqual(created.status, 303, created.text);++ const location = created.headers.get('location');++ assert.match(location, /^\/merchant\/sources\/msrc_[0-9A-HJKMNP-TV-Z]{26}\?created=1$/);++ claimId = location.match(/\/merchant\/sources\/(msrc_[0-9A-HJKMNP-TV-Z]{26})/)[1];++++ const detail = await page('GET', `/merchant/sources/${claimId}`, ALICE_TOKEN);++ assert.strictEqual(detail.status, 200);++ assert.match(detail.text, /_openvibe-sources\.shop\.example\.com/);++ assert.match(detail.text, /openvibe-sources-verification=[a-f0-9]{40}/);++ assert.match(detail.text, /Check now/);++ assert.match(detail.text, /No source exists yet/);++});++++t('check now says pending; after the TXT record is published it creates and shows the source', async () => {++ const detail = await page('GET', `/merchant/sources/${claimId}`, ALICE_TOKEN);++ const csrf = csrfOf(detail.text);++ const pending = await page('POST', `/merchant/sources/${claimId}/verify`, ALICE_TOKEN, {++ redirect: 'manual', form: { csrf },++ });++ assert.strictEqual(pending.status, 303);++ assert.match(pending.headers.get('location'), /\?pending=1$/);++ const still = await page('GET', `/merchant/sources/${claimId}?pending=1`, ALICE_TOKEN);++ assert.match(still.text, /proof was not found yet|no TXT record/i);++++ const tokenValue = still.text.match(/openvibe-sources-verification=([a-f0-9]{40})/)[1];++ svc.dns.set('_openvibe-sources.shop.example.com', [[`openvibe-sources-verification=${tokenValue}`]]);++ const ok = await page('POST', `/merchant/sources/${claimId}/verify`, ALICE_TOKEN, {++ redirect: 'manual', form: { csrf: csrfOf(still.text) },++ });++ assert.strictEqual(ok.status, 303);++ assert.match(ok.headers.get('location'), /\?verified=1$/);++++ const done = await page('GET', `/merchant/sources/${claimId}?verified=1`, ALICE_TOKEN);++ assert.match(done.text, /The source is enabled/);++ assert.match(done.text, /explicit permission/);++ assert.match(done.text, /robots\.txt/);++ assert.match(done.text, /Example Shop/);++ sourceKey = done.text.match(/<code>(merchant-shop-example-com-[a-z0-9]{8})<\/code>/)[1];++ assert.ok(sourceKey);++});++++t('another merchant cannot open the claim; a missing CSRF token is refused', async () => {++ assert.strictEqual((await page('GET', `/merchant/sources/${claimId}`, BOB_TOKEN)).status, 404);++ const detail = await page('GET', `/merchant/sources/${claimId}`, ALICE_TOKEN);++ const forged = await page('POST', `/merchant/sources/${claimId}/verify`, ALICE_TOKEN, { form: { csrf: 'not-the-token' } });++ assert.strictEqual(forged.status, 403);++ assert.match(forged.text, /form expired/);++ assert.ok(detail.text.includes('Withdraw permission'));++});++++t('withdrawal disables the source and the page says the note is kept', async () => {++ const detail = await page('GET', `/merchant/sources/${claimId}`, ALICE_TOKEN);++ const withdrawn = await page('POST', `/merchant/sources/${claimId}/withdraw`, ALICE_TOKEN, {++ redirect: 'manual', form: { csrf: csrfOf(detail.text) },++ });++ assert.strictEqual(withdrawn.status, 303);++ assert.match(withdrawn.headers.get('location'), /\?withdrawn=1$/);++ const after = await page('GET', `/merchant/sources/${claimId}?withdrawn=1`, ALICE_TOKEN);++ assert.match(after.text, /Permission withdrawn/);++ assert.match(after.text, /terms note is kept/);++ assert.match(after.text, /explicit permission/, 'the permission note is still visible');++ assert.match(after.text, /<dd>no<\/dd>/, 'the source is disabled');++ assert.doesNotMatch(after.text, /Withdraw permission/);++});++++t('sign-in redirects to the Network OAuth client with the merchant callback', async () => {++ const r = await request(svc.base, 'GET', '/merchant/login?next=%2Fmerchant', { redirect: 'manual' });++ assert.strictEqual(r.status, 302);++ const location = new URL(r.headers.get('location'));++ assert.strictEqual(location.origin, 'https://openvibe.network');++ assert.strictEqual(location.pathname, '/oauth/authorize');++ assert.strictEqual(location.searchParams.get('client_id'), 'sources');++ assert.strictEqual(location.searchParams.get('redirect_uri'), 'https://sources.test/merchant/callback');++ assert.ok(location.searchParams.get('state'));++ assert.match(r.headers.get('set-cookie') || '', /ov_merchant_state=/);++});++++t('done', async () => { await svc.stop(); });++++t.run();@@@ -63,7 +63,8 @@ t('every route, as every kind of caller, with real and nonsense ids: the clientfor (const id of ['keyed', '1', 'nope', "'\"<x>", 'x'.repeat(300)]) {const p = fill(route.path, id);for (const [who, token] of Object.entries(callers)) {-- const r = await request(svc.base, method.toUpperCase(), `${p}${method === 'get' ? '?q=x&limit=-1' : ''}`, { token, ...(method === 'get' ? {} : { headers: { 'content-type': 'application/json' } }) });++ // manual redirects: /merchant/login points at OpenVibe.Network, and this test must not call the internet.++ const r = await request(svc.base, method.toUpperCase(), `${p}${method === 'get' ? '?q=x&limit=-1' : ''}`, { token, redirect: 'manual', ...(method === 'get' ? {} : { headers: { 'content-type': 'application/json' } }) });const text = r.text + JSON.stringify([...r.headers.entries()]);if (text.includes(SECRET)) found.push(`${who}: ${method} ${p} → ${r.status} carries the client secret`);if (text.includes(FEED_KEY) && who !== 'source reader') found.push(`${who}: ${method} ${p} → ${r.status} carries the feed key`);@@@ -151,6 +151,7 @@ t('a source registered over the API with internal endpoints is stored, and its ft('ratchet: every file that makes an outbound request itself is reviewed', () => {const REVIEWED = {'server/net/fetcher.js': 'every source fetch; the guard (checkUrl up front, lookup at connect, on every redirect hop)',++ 'server/merchant-auth.js': 'OAuth code exchange only, to the operator-configured Network URLs (never a merchant-supplied URL)',};const root = path.join(__dirname, '..');const found = [];
@@ -29,6 +29,29 @@ OV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000
@@ -29,6 +29,29 @@ OV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000
OV_OAUTH_CLIENT_ID=sources
OV_OAUTH_CLIENT_ID=sources
OV_OAUTH_CLIENT_SECRET=
OV_OAUTH_CLIENT_SECRET=
+# Merchant opt-in pages: the Network OAuth client must be registered with the redirect URI below. BASE_URL decides the
+# default; set OV_OAUTH_REDIRECT_URI explicitly if the public host differs. The session cookie is Secure in production.
+#OV_OAUTH_REDIRECT_URI=https://sources.openvibe.network/merchant/callback
+#OV_OAUTH_SCOPE=profile
+#SOURCES_SESSION_SECRET=
+#SOURCES_COOKIE_SECURE=1
+# Pending proofs: recheck cadence (0 disables the worker pass), expiry, meta page cap, optional DNS servers.
+#SOURCES_MERCHANT_RECHECK_MS=600000
+#SOURCES_MERCHANT_PENDING_DAYS=14
+#SOURCES_MERCHANT_META_MAX_BYTES=1048576
+#SOURCES_MERCHANT_META_TIMEOUT_MS=10000
+#SOURCES_MERCHANT_DNS_SERVERS=
+# The source created after verification: crawl/feed spacing, polling, staleness and crawl limits.
+#SOURCES_MERCHANT_CRAWL_MIN_INTERVAL_MS=1000
+#SOURCES_MERCHANT_FEED_MIN_INTERVAL_MS=60000
+#SOURCES_MERCHANT_POLL_INTERVAL_SEC=3600
+#SOURCES_MERCHANT_CRAWL_STALE_AFTER_SEC=86400
+#SOURCES_MERCHANT_FEED_STALE_AFTER_SEC=10800
+#SOURCES_MERCHANT_MAX_ITEMS=1000
+#SOURCES_MERCHANT_MAX_PAGES=5000
+#SOURCES_MERCHANT_PAGES_PER_RUN=100
+#SOURCES_MERCHANT_RECRAWL_SEC=86400
+
# Store (PostgreSQL, ADR-035), written by OpenVibe.Host roles/data/add-service.sh sources: the runtime role through
# Store (PostgreSQL, ADR-035), written by OpenVibe.Host roles/data/add-service.sh sources: the runtime role through
# PgBouncer, the owner on a direct connection (migrations), and Valkey (shared limit counters).
# PgBouncer, the owner on a direct connection (migrations), and Valkey (shared limit counters).
# Without DATABASE_URL, development uses an embedded PGlite database in data/pglite.
# Without DATABASE_URL, development uses an embedded PGlite database in data/pglite.
@@ -248,13 +248,15 @@ the endpoint's last good parse saw. At most 500 items are retired a run; the res
@@ -248,13 +248,15 @@ the endpoint's last good parse saw. At most 500 items are retired a run; the res
## API (service tokens, one capability per route)
## API (service tokens, one capability per route)
-Callers use an OpenVibe.Network client-credentials token for audience `openvibe.sources`.
+Callers use an OpenVibe.Network client-credentials token for audience `openvibe.sources`. The merchant routes also
+accept a signed-in person's session (see [Merchant opt-in](#merchant-opt-in-list-your-store-on-openvibe)); a first-party
+service calls them with its service token and `X-OV-Subject: usr_…`.
| Capability | Routes |
| Capability | Routes |
|---|---|
|---|---|
-| `sources.source.read` | `GET /api/v1/sources`, `/api/v1/sources/:key`, `/api/v1/sources/:key/runs?before=`, `/api/v1/runs?state=failed&after=`, `/api/v1/health` |
+| `sources.source.read` | `GET /api/v1/sources`, `/api/v1/sources/:key`, `/api/v1/sources/:key/runs?before=`, `/api/v1/runs?state=failed&after=`, `/api/v1/health`; `GET /api/v1/merchant/sources[/:id]` |
| `sources.item.read` | `GET /api/v1/items?source=&category=&after=<change_seq>&include_removed=1`, `GET /api/v1/items/:id?revisions=1` |
| `sources.item.read` | `GET /api/v1/items?source=&category=&after=<change_seq>&include_removed=1`, `GET /api/v1/items/:id?revisions=1` |
-| `sources.source.manage` | `POST /api/v1/sources`, `PATCH`/`DELETE /api/v1/sources/:key`, `POST /api/v1/sources/:key/fetch`, `POST /api/v1/sources/:key/items` (manual), `DELETE /api/v1/items/:id` `{reason}` |
+| `sources.source.manage` | `POST /api/v1/sources`, `PATCH`/`DELETE /api/v1/sources/:key`, `POST /api/v1/sources/:key/fetch`, `POST /api/v1/sources/:key/items` (manual), `DELETE /api/v1/items/:id` `{reason}`; `POST /api/v1/merchant/sources`, `/:id/verify`, `/:id/withdraw` |
`GET /api/v1/items` pages in change order (creations, revisions and — with `include_removed=1` —
`GET /api/v1/items` pages in change order (creations, revisions and — with `include_removed=1` —
removals); resume from `next_after`. Every page carries the status and staleness of the sources
removals); resume from `next_after`. Every page carries the status and staleness of the sources
@@ -275,6 +277,8 @@ problem+json `rate_limited` with `Retry-After`, one `[limits]` log line and
@@ -275,6 +277,8 @@ problem+json `rate_limited` with `Retry-After`, one `[limits]` log line and
| `POST`, `PATCH`, `DELETE /api/v1/sources[/:key]` | 30 / 300 |
| `POST`, `PATCH`, `DELETE /api/v1/sources[/:key]` | 30 / 300 |
| `POST /api/v1/sources/:key/fetch` | 10 / 100 |
| `POST /api/v1/sources/:key/fetch` | 10 / 100 |
| `POST /api/v1/sources/:key/items` (manual) | 60 / 1200 |
| `POST /api/v1/sources/:key/items` (manual) | 60 / 1200 |
+| `GET /api/v1/merchant/sources[/:id]` (a merchant or a service acting for one) | 120 / 3000 |
+| `POST /api/v1/merchant/sources`, `/:id/verify`, `/:id/withdraw` | 30 / 300 |
| `DELETE /api/v1/items/:id` (a takedown may cover many items) | 120 / 3000 |
| `DELETE /api/v1/items/:id` (a takedown may cover many items) | 120 / 3000 |
A first-party service (`svc:…`) is not counted on reads: News, Reviews and Trade pull cursor pages
A first-party service (`svc:…`) is not counted on reads: News, Reviews and Trade pull cursor pages
@@ -282,6 +286,46 @@ for all their readers, and Wiki asks about every item its editors cite. Sources'
@@ -282,6 +286,46 @@ for all their readers, and Wiki asks about every item its editors cite. Sources'
scheduler, fetches, the outbox relay) run in-process and never pass through HTTP. Never limited:
scheduler, fetches, the outbox relay) run in-process and never pass through HTTP. Never limited:
`/api/health`, `/api/ready`, `/release.json`, `/metrics` and the home page. `test/actor-limits.test.js`.
`/api/health`, `/api/ready`, `/release.json`, `/metrics` and the home page. `test/actor-limits.test.js`.
+## Merchant opt-in ("List your store on OpenVibe")
+
+The first permitted channel in plan §4.4. A signed-in merchant claims a domain, proves control of it, and names the
+sitemap or product feed Sources may read. **Nothing is created in `sources` before the proof is found**: a pending claim
+is only a `merchant_sources` row (migration `0007_merchant_sources.sql`). Verification creates one enabled source in the
+same transaction, and withdrawal disables it while keeping the note.
+
+- **Proof.** One of two challenges, the OpenVibe.Space pattern:
+ - DNS: `_openvibe-sources.<domain> TXT "openvibe-sources-verification=<token>"`.
+ - Meta: `<meta name="openvibe-sources-verification" content="<token>">` on `https://<domain>/` (or the `meta_url` the
+ merchant names, on the same site). Redirects off the domain are refused.
+- **Fetched politely.** A sitemap becomes a `crawl` source seeded with that sitemap (no guessed sitemaps); a product feed
+ becomes an `rss`/`atom` source. Either way the ordinary fetcher runs: robots.txt on every request and redirect hop,
+ per-host spacing, SSRF guard, conditional GETs. Explicit permission is not a robots override.
+- **The terms note** records the merchant subject (and display name when the token carries one), the UTC verification
+ time, the domain, the named sitemap/feed, and how control was verified ("DNS TXT record at _openvibe-sources.…" or
+ "meta tag at …"); it also says the merchant may withdraw and that robots.txt is still obeyed.
+- **Withdrawal** sets the claim to `withdrawn`, disables the source, and leaves `terms_note` and the claim row intact.
+ The domain is free to claim again (a new claim, a new source key). Unlike a *served* custom domain (Space's daily
+ recheck), the permission persists until the merchant withdraws: removing the TXT/meta record later does not silently
+ disable the source.
+- **Pending proofs** are rechecked by a slow in-process pass (`SOURCES_MERCHANT_RECHECK_MS`, default 10 min; `0` turns
+ it off) and fail after `SOURCES_MERCHANT_PENDING_DAYS` (14). A merchant can also press "Check now".
+- **Pages.** `GET /merchant` (guest explanation, or the signed-in merchant's list and add form),
+ `GET /merchant/sources/:id` (challenge, status, terms note, last crawl), the POST actions
+ `/merchant/sources`, `/:id/verify`, `/:id/withdraw`, and `/merchant/login|callback|logout`. Forms carry an HMAC CSRF
+ token; the session is the Sources `ov_token` cookie, verified offline with `verifyUserToken`.
+- **JSON read for Report.** `GET /api/v1/merchant/sources` (alias `/claims`) returns each claim with its source, health,
+ last run, recent runs and — for a crawl — `last_crawl` (`pages`, `due`, `next_due_at`, state counts, recent pages).
+ Report uses its service token and `X-OV-Subject: usr_…`; a merchant's own session needs no extra header. The read is
+ the contract for the Report dashboard (which is not built here).
+- **Identity.** The OAuth client is `OV_OAUTH_CLIENT_ID` (default `sources`) with
+ `OV_OAUTH_REDIRECT_URI` defaulting to `<BASE_URL>/merchant/callback`; Network must register that exact redirect. A
+ first-party service token may act for a merchant only with `X-OV-Subject` (an app/module token cannot). The merchant
+ routes reuse the released `sources.source.read` / `sources.source.manage` capabilities; no new contract id is needed.
+- **Config.** `SOURCES_SESSION_SECRET` (defaults to `OV_OAUTH_CLIENT_SECRET`), `SOURCES_COOKIE_SECURE`,
+ `SOURCES_MERCHANT_PENDING_DAYS`, `SOURCES_MERCHANT_RECHECK_MS`, `SOURCES_MERCHANT_META_MAX_BYTES`,
+ `SOURCES_MERCHANT_DNS_SERVERS`, and the source-shape knobs `SOURCES_MERCHANT_*` documented in
+ [.env.example](.env.example). Tests: `test/merchant-optin.test.js`, `test/merchant-pages.test.js`.
+
## Events (transactional outbox → OpenVibe.Events when `EVENTS_URL` is set)
## Events (transactional outbox → OpenVibe.Events when `EVENTS_URL` is set)
- `sources.item.created`, `sources.item.updated` (with the previous content hash), `sources.item.removed` (with the reason)
- `sources.item.created`, `sources.item.updated` (with the previous content hash), `sources.item.removed` (with the reason)
@@ -383,6 +427,7 @@ publishes another's output.
@@ -383,6 +427,7 @@ publishes another's output.
## Owns
## Owns
- the source registry (`sources`, `endpoint_state`, `robots_cache`) and the source/item contracts
- the source registry (`sources`, `endpoint_state`, `robots_cache`) and the source/item contracts
+- the merchant opt-in claims (`merchant_sources`), their proof checks and the pages/JSON read
- ingestion: `fetch_runs`, `items`, `item_revisions`, the scheduler, robots and rate-limit state
- ingestion: `fetch_runs`, `items`, `item_revisions`, the scheduler, robots and rate-limit state
- `sources.*` events
- `sources.*` events
@@ -397,7 +442,7 @@ publishes another's output.
@@ -397,7 +442,7 @@ publishes another's output.
- PostgreSQL 18 (OpenVibe.Host `roles/data/`, ADR-035): every read and write is async through `openvibe-sdk/db`
- PostgreSQL 18 (OpenVibe.Host `roles/data/`, ADR-035): every read and write is async through `openvibe-sdk/db`
- OpenVibe.Contracts (service tokens, problem details, ids, the event envelope)
- OpenVibe.Contracts (service tokens, problem details, ids, the event envelope)
-- OpenVibe.Network (signing key; service principal `sources`)
+- OpenVibe.Network (signing key; service principal `sources`; OAuth client `sources` for merchant sign-in)
- OpenVibe.Events (outbound events)
- OpenVibe.Events (outbound events)
## Capabilities
## Capabilities
@@ -427,8 +472,11 @@ through Events.
@@ -427,8 +472,11 @@ through Events.
- parsers against fixtures (`test/adapters.test.js`); API, registry rules, paging, manual items,
- parsers against fixtures (`test/adapters.test.js`); API, registry rules, paging, manual items,
removal (`test/api.test.js`); scheduler, backoff, relay to Events (`test/scheduler-events.test.js`);
removal (`test/api.test.js`); scheduler, backoff, relay to Events (`test/scheduler-events.test.js`);
proposals and seeds (`test/proposals.test.js`)
proposals and seeds (`test/proposals.test.js`)
-- the public host shows only an honest "internal service" page, and the vhost keeps the API
- loopback-only (`test/public-host.test.js`)
+- the public host shows only an honest "internal service" page and the merchant opt-in pages, and the vhost keeps the
+ API loopback-only (`test/public-host.test.js`)
+- merchant opt-in: no source before the DNS/meta proof, the terms note records who/when/how, robots.txt is still obeyed,
+ withdrawal disables and keeps the note, and the JSON read carries last crawl stats (`test/merchant-optin.test.js`,
+ `test/merchant-pages.test.js`)
Production on 2026-10-22 (counts read from `ov_sources`):
Production on 2026-10-22 (counts read from `ov_sources`):
- 43 of 47 sources enabled, each after its terms and robots were read (by category in [Seeds](#seeds)); none of them
- 43 of 47 sources enabled, each after its terms and robots were read (by category in [Seeds](#seeds)); none of them
@@ -449,8 +497,10 @@ readiness, row counts; see OpenVibe.Host `docs/restore-drills.md`).
@@ -449,8 +497,10 @@ readiness, row counts; see OpenVibe.Host `docs/restore-drills.md`).
Reporting: [SECURITY.md](SECURITY.md). The rules the code keeps:
Reporting: [SECURITY.md](SECURITY.md). The rules the code keeps:
- **Auth.** Every `/api/v1` route needs an OpenVibe.Network service token for audience
- **Auth.** Every `/api/v1` route needs an OpenVibe.Network service token for audience
- `openvibe.sources` with that route's capability, then passes the per-actor limits. There are no user
- sessions and no public API: the vhost keeps `/api/v1/*` loopback-only.
+ `openvibe.sources` with that route's capability, then passes the per-actor limits. The merchant API and pages are the
+ one person-facing exception: a Network person session (Bearer on the API, the first-party `ov_token` cookie on the
+ pages) owns its claims, and a first-party service may act for a person only with `X-OV-Subject`. There is still no
+ public JSON API: the vhost keeps `/api/v1/*` loopback-only, and only `/merchant*` and `/shared/*` are public.
- **Egress.** Sources fetches outside URLs by design, only through its guard: http(s) on ports 80/443,
- **Egress.** Sources fetches outside URLs by design, only through its guard: http(s) on ports 80/443,
every resolved address public at connect time (no DNS rebinding), each redirect re-checked, robots.txt
every resolved address public at connect time (no DNS rebinding), each redirect re-checked, robots.txt
obeyed, per-host spacing, one deadline and a byte cap per response (`test/robots-ssrf.test.js`,
obeyed, per-host spacing, one deadline and a byte cap per response (`test/robots-ssrf.test.js`,
@@ -488,6 +538,8 @@ the Network wildcard certificate and answers only:
@@ -488,6 +538,8 @@ the Network wildcard certificate and answers only:
| Path | Answer |
| Path | Answer |
|---|---|
|---|---|
| `/` | a short page: an internal ingestion service, nothing to browse (HTML for browsers, the text route index otherwise; noindex, no-store) |
| `/` | a short page: an internal ingestion service, nothing to browse (HTML for browsers, the text route index otherwise; noindex, no-store) |
+| `/merchant`, `/merchant/*` | the merchant opt-in pages (guest explanation, signed-in claims, proof/withdraw forms; noindex, no-store) |
+| `/shared/*` | this release's pinned `theme-loader.js`, `kit.css`, `kit-web.js` (served by `openvibe-shared/serve`) |
| `/api/health`, `/api/ready`, `/release.json` | proxied |
| `/api/health`, `/api/ready`, `/release.json` | proxied |
| `/robots.txt` | `Disallow: /` |
| `/robots.txt` | `Disallow: /` |
| `/metrics` | 404 (Prometheus scrapes `127.0.0.1:4720/metrics` directly) |
| `/metrics` | 404 (Prometheus scrapes `127.0.0.1:4720/metrics` directly) |
@@ -503,6 +555,8 @@ sudo ln -sf /etc/nginx/sites-available/sources.openvibe.network.conf /etc/nginx/
@@ -503,6 +555,8 @@ sudo ln -sf /etc/nginx/sites-available/sources.openvibe.network.conf /etc/nginx/
sudo nginx -t && sudo systemctl reload nginx
sudo nginx -t && sudo systemctl reload nginx
# check
# check
curl -sS -H 'Accept: text/html' https://sources.openvibe.network/ | grep -o '<title>[^<]*' # OpenVibe.Sources
curl -sS -H 'Accept: text/html' https://sources.openvibe.network/ | grep -o '<title>[^<]*' # OpenVibe.Sources
+curl -sS -o /dev/null -w '%{http_code}\n' https://sources.openvibe.network/merchant # 200
+curl -sS -o /dev/null -w '%{http_code}\n' https://sources.openvibe.network/shared/kit.css # 200
curl -sS https://sources.openvibe.network/api/health
curl -sS https://sources.openvibe.network/api/health
curl -sS -o /dev/null -w '%{http_code}\n' https://sources.openvibe.network/api/v1/sources # 403
curl -sS -o /dev/null -w '%{http_code}\n' https://sources.openvibe.network/api/v1/sources # 403
curl -sS -o /dev/null -w '%{http_code}\n' https://sources.openvibe.network/metrics # 404
curl -sS -o /dev/null -w '%{http_code}\n' https://sources.openvibe.network/metrics # 404
@@ -512,9 +566,9 @@ curl -sS https://sources.openvibe.network/admin
@@ -512,9 +566,9 @@ curl -sS https://sources.openvibe.network/admin
## Launch rule
## Launch rule
-Sources has no public pages by design, so it never "launches" as a site. The domain is not in
-[OpenVibers/OpenVibe.Sites](https://github.com/OpenVibers/OpenVibe.Sites); this repository's own
-vhost serves it.
+Sources has no public product pages by design: the only person-facing pages are the merchant opt-in flow (`/merchant`),
+which exist so a merchant can prove a domain and name a sitemap/feed. It never "launches" as a site. The domain is not
+in [OpenVibers/OpenVibe.Sites](https://github.com/OpenVibers/OpenVibe.Sites); this repository's own vhost serves it.
---
---
@@ -6,7 +6,7 @@
@@ -6,7 +6,7 @@
"plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026)",
"plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026)",
"runtime": "node 22, express, PostgreSQL 18 (openvibe-sdk/db, async) + Valkey 9, fast-xml-parser",
"runtime": "node 22, express, PostgreSQL 18 (openvibe-sdk/db, async) + Valkey 9, fast-xml-parser",
"port": 4720,
"port": 4720,
- "domain": "sources.openvibe.network (vhost in deploy/nginx: an internal-service page, health and JSON 404s; the API is loopback-only; README \"Public host\")",
+ "domain": "sources.openvibe.network (vhost in deploy/nginx: an internal-service page, the merchant opt-in pages, health and JSON 404s; the JSON API is loopback-only; README \"Public host\")",
"contracts": "openvibe-contracts v0.224.0",
"contracts": "openvibe-contracts v0.224.0",
"sdk": "openvibe-sdk v0.67.0",
"sdk": "openvibe-sdk v0.67.0",
"shared": "openvibe-shared v4.33.0",
"shared": "openvibe-shared v4.33.0",
@@ -26,7 +26,8 @@
@@ -26,7 +26,8 @@
"sitemap",
"sitemap",
"jsonld",
"jsonld",
"api",
"api",
- "manual"
+ "manual",
+ "crawl"
],
],
"seededSources": 47,
"seededSources": 47,
"enabledSources": 43,
"enabledSources": 43,
@@ -34,7 +35,8 @@
@@ -34,7 +35,8 @@
"deployment": "Deployed with `sudo ovhost deploy sources`: /opt/openvibe.sources, unit openvibe-sources, 127.0.0.1:4720, principal sources. On 2026-10-22, 43 of the 47 sources were enabled in production (reviews 22, hosting 5, news 4, trade 4, games 3, coupons 2, and blog, crowdfunding and deals 1 each; none failing), with 55,683 items from 4,762 fetch runs since the first on 2026-10-07; they feed OpenVibe.News, Reviews, Deals (with its crowdfunding search), Coupons, Trade, Host, Games and Blog's desks. Restore drill passed 2026-09-23.",
"deployment": "Deployed with `sudo ovhost deploy sources`: /opt/openvibe.sources, unit openvibe-sources, 127.0.0.1:4720, principal sources. On 2026-10-22, 43 of the 47 sources were enabled in production (reviews 22, hosting 5, news 4, trade 4, games 3, coupons 2, and blog, crowdfunding and deals 1 each; none failing), with 55,683 items from 4,762 fetch runs since the first on 2026-10-07; they feed OpenVibe.News, Reviews, Deals (with its crowdfunding search), Coupons, Trade, Host, Games and Blog's desks. Restore drill passed 2026-09-23.",
"features": [
"features": [
"source registry with terms, robots and licence notes",
"source registry with terms, robots and licence notes",
- "rss, atom, sitemap, jsonld, api and manual adapters",
+ "rss, atom, sitemap, jsonld, api, manual and crawl adapters",
+ "merchant opt-in: a signed-in merchant proves a domain by DNS TXT or meta tag, names a sitemap or product feed, and a source is created only on verification with the explicit permission in its terms note; withdrawal disables it and keeps the note; /api/v1/merchant/sources is the JSON read for Report",
"categories: news, blog, reviews, deals, coupons, trade, hosting, crowdfunding, games, and since contracts 0.210.0 housing, food and jobs (read from sources.source@1)",
"categories: news, blog, reviews, deals, coupons, trade, hosting, crowdfunding, games, and since contracts 0.210.0 housing, food and jobs (read from sources.source@1)",
"a deal feed's own offer elements kept as stated (feed@3: DealNews' price, currency, retailer, end date, category and editors' pick)",
"a deal feed's own offer elements kept as stated (feed@3: DealNews' price, currency, retailer, end date, category and editors' pick)",
"api mappings: keyed maps, where conditions and patterns, $index for ranked lists; per-endpoint timeout_ms and max_bytes",
"api mappings: keyed maps, where conditions and patterns, $index for ranked lists; per-endpoint timeout_ms and max_bytes",
@@ -4,14 +4,16 @@
@@ -4,14 +4,16 @@
## Deployed to /etc/nginx/sites-available/sources.openvibe.network.conf — keep them in sync,
## Deployed to /etc/nginx/sites-available/sources.openvibe.network.conf — keep them in sync,
## and reload nginx after changing it. Install: README "Public host".
## and reload nginx after changing it. Install: README "Public host".
##
##
-## Sources is an internal ingestion service with no public pages: Sources holds raw third-party
-## material, which products publish (or not) under their own rules. The public host answers only:
+## Sources is an internal ingestion service, with one person-facing surface: the merchant opt-in flow.
+## Sources holds raw third-party material, which products publish (or not) under their own rules. The public host answers:
## / a short page saying what the service is (HTML for browsers, text otherwise)
## / a short page saying what the service is (HTML for browsers, text otherwise)
+## /merchant* the merchant opt-in pages (signed-in merchants add, prove and withdraw a store)
+## /shared/* this release's pinned kit.css, kit-web.js and theme-loader.js
## /api/health, /api/ready, /release.json
## /api/health, /api/ready, /release.json
## /robots.txt disallow everything
## /robots.txt disallow everything
## and keeps the rest off the public edge:
## and keeps the rest off the public edge:
## /api/v1/* service-to-service API (every route needs a service token): loopback only.
## /api/v1/* service-to-service API (every route needs a service token): loopback only.
-## Services on this host call 127.0.0.1:4720 directly.
+## Services on this host call 127.0.0.1:4720 directly; /api/v1/merchant/* is reachable there too.
## /metrics 404 here; Prometheus text for direct loopback callers of 127.0.0.1:4720
## /metrics 404 here; Prometheus text for direct loopback callers of 127.0.0.1:4720
## anything else a JSON 404 that says why, so the name never falls through to another site
## anything else a JSON 404 that says why, so the name never falls through to another site
## ═══════════════════════════════════════════════════════════════
## ═══════════════════════════════════════════════════════════════
@@ -77,6 +79,23 @@ server {
@@ -77,6 +79,23 @@ server {
return 200 "User-agent: *\nDisallow: /\n";
return 200 "User-agent: *\nDisallow: /\n";
}
}
+ # ── Merchant opt-in pages (the one person-facing surface) ──
+ # A signed-in merchant adds a domain, proves it with DNS or a meta tag, names a sitemap/feed, and may withdraw.
+ # The Network OAuth client `sources` must be registered with https://sources.openvibe.network/merchant/callback.
+ location = /merchant {
+ limit_except GET POST { deny all; }
+ proxy_pass http://127.0.0.1:4720;
+ }
+ location ^~ /merchant/ {
+ limit_except GET POST { deny all; }
+ proxy_pass http://127.0.0.1:4720;
+ }
+ # This release's pinned OpenVibe Kit and theme loader.
+ location ^~ /shared/ {
+ limit_except GET { deny all; }
+ proxy_pass http://127.0.0.1:4720;
+ }
+
# ── Service API: loopback only (see header) ─────────────
# ── Service API: loopback only (see header) ─────────────
location ^~ /api/v1/ {
location ^~ /api/v1/ {
allow 127.0.0.1;
allow 127.0.0.1;
@@ -88,6 +107,6 @@ server {
@@ -88,6 +107,6 @@ server {
# ── Anything else: an honest JSON 404, never another site ──
# ── Anything else: an honest JSON 404, never another site ──
location / {
location / {
default_type application/problem+json;
default_type application/problem+json;
- return 404 '{"type":"about:blank","title":"Not Found","status":404,"code":"sources.not_found","detail":"sources.openvibe.network is an internal ingestion service of the OpenVibe network with no public pages or API. Its API is for OpenVibe services on the host. About: https://sources.openvibe.network/ and https://github.com/OpenVibers/OpenVibe.Sources"}';
+ return 404 '{"type":"about:blank","title":"Not Found","status":404,"code":"sources.not_found","detail":"sources.openvibe.network is an internal ingestion service of the OpenVibe network. Its JSON API is for OpenVibe services on the host; the merchant opt-in pages are at /merchant. About: https://sources.openvibe.network/ and https://github.com/OpenVibers/OpenVibe.Sources"}';
}
}
}
}
new file mode 100644
new file mode 100644
@@ -0,0 +1,44 @@
@@ -0,0 +1,44 @@
+-- phase: expand
+-- Merchant opt-in ("List your store on OpenVibe", plan §4.4 channel 1).
+--
+-- A signed-in merchant claims a domain, proves control of it with a DNS TXT record or a meta tag, and names the
+-- sitemap or product feed Sources may read. Nothing is created in `sources` before the proof is present: a pending
+-- claim is only this row. On verification the source is created enabled, and its terms_note records who gave the
+-- explicit permission, when, and how control was verified. Withdrawal disables that source and keeps the note (and
+-- this row) for the audit trail. The crawler and the fetcher still obey robots.txt on every request.
+CREATE TABLE IF NOT EXISTS merchant_sources (
+ id text COLLATE "C" PRIMARY KEY, -- msrc_<ULID>
+ owner_sub text COLLATE "C" NOT NULL, -- usr_<ULID>, the signed-in merchant
+ owner_name text COLLATE "C", -- display name the token carried, for people reading the note
+ store_name text COLLATE "C",
+ domain text COLLATE "C" NOT NULL, -- lowercase hostname, no port
+ method text COLLATE "C" NOT NULL CHECK (method IN ('dns', 'meta')),
+ token text COLLATE "C" NOT NULL, -- verification challenge, shown only to the owner
+ listing_url text COLLATE "C" NOT NULL, -- the sitemap or product feed the merchant named
+ listing_kind text COLLATE "C" NOT NULL CHECK (listing_kind IN ('sitemap', 'feed')),
+ meta_url text COLLATE "C", -- where the meta tag lives (meta method only)
+ status text COLLATE "C" NOT NULL DEFAULT 'pending'
+ CHECK (status IN ('pending', 'verified', 'withdrawn', 'failed')),
+ source_key text COLLATE "C", -- set only when verified
+ terms_note text COLLATE "C", -- final explicit-permission note, kept after withdrawal
+ verified_at bigint,
+ verified_by text COLLATE "C", -- the actor that completed the check (the merchant or svc:sources)
+ verification_detail text COLLATE "C", -- how the proof was found, without the challenge value
+ last_checked_at bigint,
+ check_count bigint NOT NULL DEFAULT 0,
+ last_error text COLLATE "C",
+ created_at bigint NOT NULL,
+ updated_at bigint NOT NULL,
+ withdrawn_at bigint,
+ withdrawn_by text COLLATE "C"
+);
+
+-- One live claim per domain: a withdrawn or failed claim frees the name for a new opt-in.
+CREATE UNIQUE INDEX IF NOT EXISTS merchant_sources_live_domain
+ ON merchant_sources (domain) WHERE status IN ('pending', 'verified');
+CREATE UNIQUE INDEX IF NOT EXISTS merchant_sources_source_key
+ ON merchant_sources (source_key) WHERE source_key IS NOT NULL;
+CREATE INDEX IF NOT EXISTS merchant_sources_owner
+ ON merchant_sources (owner_sub, created_at DESC);
+CREATE INDEX IF NOT EXISTS merchant_sources_pending
+ ON merchant_sources (last_checked_at) WHERE status = 'pending';
@@ -43,6 +43,10 @@ const BUDGETS = {
@@ -43,6 +43,10 @@ const BUDGETS = {
// Creating, changing or deleting a source is a staff decision recorded with its terms and robots
// Creating, changing or deleting a source is a staff decision recorded with its terms and robots
// notes: a form every few seconds at most.
// notes: a form every few seconds at most.
'sources.source.manage': { minute: 30, hour: 300 },
'sources.source.manage': { minute: 30, hour: 300 },
+ // Merchant opt-in reads (the page and the JSON read Report calls): a person refreshing their small list.
+ 'sources.merchant.read': { minute: 120, hour: 3000 },
+ // Claiming a domain, checking the proof, withdrawing: a handful of decisions, not a polling loop.
+ 'sources.merchant.manage': { minute: 30, hour: 300 },
// A manual fetch requests every endpoint of the source now (the per-host spacing still applies):
// A manual fetch requests every endpoint of the source now (the per-host spacing still applies):
// the scheduler already polls each source, so a person's "fetch now" is a rare nudge.
// the scheduler already polls each source, so a person's "fetch now" is a rare nudge.
'sources.source.fetch': { minute: 10, hour: 100 },
'sources.source.fetch': { minute: 10, hour: 100 },
new file mode 100644
new file mode 100644
@@ -0,0 +1,141 @@
@@ -0,0 +1,141 @@
+'use strict';
+/**
+ * Merchant opt-in JSON API.
+ *
+ * sources.source.read (or the merchant's own person token)
+ * GET /api/v1/merchant/sources the merchant's claims and sources, with last run/crawl stats
+ * GET /api/v1/merchant/sources/:id
+ * sources.source.manage (or the merchant's own person token)
+ * POST /api/v1/merchant/sources add a domain and name its sitemap/product feed (pending)
+ * POST /api/v1/merchant/sources/:id/verify
+ * POST /api/v1/merchant/sources/:id/withdraw
+ *
+ * `/api/v1/merchant/claims` is an alias for the same collection (a claim is a source before verification). A first-party
+ * service token must name the merchant with X-OV-Subject; a person token needs nothing else. Report calls the read with
+ * its service token and the signed-in merchant's subject. The merchant dashboard itself lives in Report, not here.
+ */
+const express = require('express');
+const { http } = require('openvibe-contracts');
+const { MerchantError, CHALLENGE_PREFIX, TXT_PREFIX, META_NAME } = require('../merchant');
+const { CAPS } = require('../auth');
+const { runView } = require('./sources');
+
+const COLLECTION = ['/api/v1/merchant/sources', '/api/v1/merchant/claims'];
+const ITEM = ['/api/v1/merchant/sources/:id', '/api/v1/merchant/claims/:id'];
+const VERIFY = ['/api/v1/merchant/sources/:id/verify', '/api/v1/merchant/claims/:id/verify'];
+const WITHDRAW = ['/api/v1/merchant/sources/:id/withdraw', '/api/v1/merchant/claims/:id/withdraw'];
+
+function crawlSummary(crawl) {
+ if (!crawl || crawl.not_crawl) return null;
+ return {
+ pages: crawl.pages,
+ max_pages: crawl.max_pages,
+ due: crawl.due,
+ next_due_at: crawl.next_due_at,
+ pages_by_state: crawl.pages_by_state,
+ sitemaps_by_state: crawl.sitemaps_by_state,
+ recent: (crawl.recent || []).slice(0, 5),
+ };
+}
+
+function createMerchantView({ db, merchant, registry, ingest }) {
+ return async function decorate(row) {
+ const out = { ...merchant.view(row), source: null, last_run: null, last_crawl: null, recent_runs: [] };
+ if (!row.source_key) return out;
+ const source = await registry.get(row.source_key);
+ if (!source) return out;
+ out.source = registry.view(source);
+ const runs = await db.prepare('SELECT * FROM fetch_runs WHERE source_key = ? ORDER BY rid DESC LIMIT 5').all(source.key);
+ out.recent_runs = runs.map(runView);
+ out.last_run = out.recent_runs[0] || null;
+ if (source.type === 'crawl') {
+ try { out.last_crawl = crawlSummary(await ingest.crawlStatus(source.key)); } catch { out.last_crawl = null; }
+ }
+ out.stats = {
+ health: out.source.health,
+ last_run: out.last_run,
+ last_crawl: out.last_crawl,
+ recent_runs: out.recent_runs,
+ };
+ return out;
+ };
+}
+
+function merchantRouter({ db, merchant, registry, ingest, auth, merchantAuth, limits }) {
+ const router = express.Router();
+ const read = [merchantAuth.api(CAPS.read), limits.budget('sources.merchant.read')];
+ const manage = [merchantAuth.api(CAPS.manage), limits.budget('sources.merchant.manage')];
+ const decorate = createMerchantView({ db, merchant, registry, ingest });
+
+ const fail = (res, req, err) => {
+ if (err instanceof MerchantError) return http.sendProblem(res, err.status, err.code, { detail: err.message, ctx: req.ov });
+ if (err && typeof err.code === 'string' && err.code.startsWith('sources.')) {
+ return http.sendProblem(res, 422, err.code, { detail: err.message, ctx: req.ov });
+ }
+ throw err;
+ };
+
+ router.get(COLLECTION, read, async (req, res, next) => {
+ try {
+ const rows = await merchant.list(req.merchant);
+ res.json({
+ merchant: { subject: req.merchant.subject, name: req.merchant.name || null },
+ sources: await Promise.all(rows.map(decorate)),
+ });
+ } catch (err) { next(err); }
+ });
+
+ router.get(ITEM, read, async (req, res, next) => {
+ try {
+ const row = await merchant.get(req.merchant, req.params.id);
+ res.json({ merchant_source: await decorate(row) });
+ } catch (err) {
+ if (err instanceof MerchantError) return fail(res, req, err);
+ next(err);
+ }
+ });
+
+ router.post(COLLECTION, manage, async (req, res, next) => {
+ try {
+ const row = await merchant.add(req.merchant, req.body || {});
+ res.status(201).json({
+ merchant_source: await decorate(row),
+ challenge: {
+ dns: { name: `${CHALLENGE_PREFIX}.${row.domain}`, value: `${TXT_PREFIX}${row.token}` },
+ meta: { name: META_NAME, value: row.token, url: row.meta_url || `https://${row.domain}/` },
+ },
+ });
+ } catch (err) {
+ if (err instanceof MerchantError) return fail(res, req, err);
+ next(err);
+ }
+ });
+
+ router.post(VERIFY, manage, async (req, res, next) => {
+ try {
+ const out = await merchant.verify(req.merchant, req.params.id);
+ res.json({
+ merchant_source: await decorate(out.row),
+ verified: Boolean(out.verified),
+ source_created: Boolean(out.source),
+ });
+ } catch (err) {
+ if (err instanceof MerchantError) return fail(res, req, err);
+ next(err);
+ }
+ });
+
+ router.post(WITHDRAW, manage, async (req, res, next) => {
+ try {
+ const row = await merchant.withdraw(req.merchant, req.params.id);
+ res.json({ merchant_source: await decorate(row) });
+ } catch (err) {
+ if (err instanceof MerchantError) return fail(res, req, err);
+ next(err);
+ }
+ });
+
+ return router;
+}
+
+module.exports = { merchantRouter, createMerchantView, crawlSummary };
@@ -8,6 +8,8 @@ const { createRelease } = require('openvibe-shared/release');
@@ -8,6 +8,8 @@ const { createRelease } = require('openvibe-shared/release');
const { createSourcesReadiness, registerSourcesGauges } = require('./observability');
const { createSourcesReadiness, registerSourcesGauges } = require('./observability');
const { sourcesRouter } = require('./api/sources');
const { sourcesRouter } = require('./api/sources');
const { itemsRouter } = require('./api/items');
const { itemsRouter } = require('./api/items');
+const { merchantRouter } = require('./api/merchant');
+const { merchantPages } = require('./merchant-pages');
const { createActorLimits } = require('./api/actor-limits');
const { createActorLimits } = require('./api/actor-limits');
const pkg = require('../package.json');
const pkg = require('../package.json');
@@ -16,12 +18,14 @@ const TEXT_INDEX = [
@@ -16,12 +18,14 @@ const TEXT_INDEX = [
'Every item carries its source, canonical URL, retrieval time, content hash and terms; a failed fetch is',
'Every item carries its source, canonical URL, retrieval time, content hash and terms; a failed fetch is',
'recorded as a failure and never replaced with invented content.',
'recorded as a failure and never replaced with invented content.',
'',
'',
- 'This is an internal service. Its API is for OpenVibe services on the production host (service tokens);',
- 'sources.openvibe.network answers only this page, /api/health, /api/ready and /release.json.',
+ 'This is an internal service. Its JSON API is for OpenVibe services on the production host (service tokens),',
+ 'and its merchant opt-in pages are for signed-in merchants: sources.openvibe.network answers /, /merchant,',
+ '/merchant/*, /shared/*, /api/health, /api/ready and /release.json.',
'',
'',
'GET /api/v1/sources, /api/v1/sources/:key, /api/v1/sources/:key/runs, /api/v1/runs, /api/v1/health (sources.source.read)',
'GET /api/v1/sources, /api/v1/sources/:key, /api/v1/sources/:key/runs, /api/v1/runs, /api/v1/health (sources.source.read)',
'GET /api/v1/items?source=&category=&after=, /api/v1/items/:id (sources.item.read)',
'GET /api/v1/items?source=&category=&after=, /api/v1/items/:id (sources.item.read)',
'POST/PATCH/DELETE /api/v1/sources[/:key], POST /api/v1/sources/:key/fetch, /items, DELETE /api/v1/items/:id (sources.source.manage)',
'POST/PATCH/DELETE /api/v1/sources[/:key], POST /api/v1/sources/:key/fetch, /items, DELETE /api/v1/items/:id (sources.source.manage)',
+ 'GET/POST /api/v1/merchant/sources[/:id[/verify|/withdraw]] (sources.source.read/manage, or a merchant session)',
'GET /api/health, /api/ready, /release.json',
'GET /api/health, /api/ready, /release.json',
'',
'',
'Source: https://github.com/OpenVibers/OpenVibe.Sources',
'Source: https://github.com/OpenVibers/OpenVibe.Sources',
@@ -55,15 +59,17 @@ a { color: var(--accent); }
@@ -55,15 +59,17 @@ a { color: var(--accent); }
where every item came from, when it was fetched and under which terms. A failed fetch is recorded as a
where every item came from, when it was fetched and under which terms. A failed fetch is recorded as a
failure, never replaced with invented content.</p>
failure, never replaced with invented content.</p>
<p>There is nothing to browse here. What reaches readers is published by the products themselves, under
<p>There is nothing to browse here. What reaches readers is published by the products themselves, under
-their own rules.</p>
+their own rules. Merchants who want their store listed can <a href="/merchant">list it on OpenVibe</a>:
+claim the domain, prove control of it, and name the sitemap or product feed we may read.</p>
<p class="muted">Health: <a href="/api/health">/api/health</a> · Readiness: <a href="/api/ready">/api/ready</a> ·
<p class="muted">Health: <a href="/api/health">/api/health</a> · Readiness: <a href="/api/ready">/api/ready</a> ·
+Merchant opt-in: <a href="/merchant">/merchant</a> ·
Source code: <a href="https://github.com/OpenVibers/OpenVibe.Sources">OpenVibers/OpenVibe.Sources</a></p>
Source code: <a href="https://github.com/OpenVibers/OpenVibe.Sources">OpenVibers/OpenVibe.Sources</a></p>
</main>
</main>
</body>
</body>
</html>
</html>
`;
`;
-function createApp({ config, db, registry, items, ingest, scheduler, auth, outbox, now, log = console, limitsNow = null }) {
+function createApp({ config, db, registry, items, ingest, scheduler, auth, outbox, now, log = console, limitsNow = null, merchant = null, merchantAuth = null }) {
const app = express();
const app = express();
app.disable('x-powered-by');
app.disable('x-powered-by');
app.set('trust proxy', 'loopback');
app.set('trust proxy', 'loopback');
@@ -100,9 +106,24 @@ function createApp({ config, db, registry, items, ingest, scheduler, auth, outbo
@@ -100,9 +106,24 @@ function createApp({ config, db, registry, items, ingest, scheduler, auth, outbo
app.use(sourcesRouter({ db, registry, ingest, auth, limits }));
app.use(sourcesRouter({ db, registry, ingest, auth, limits }));
app.use(itemsRouter({ db, registry, items, auth, outbox, now, limits }));
app.use(itemsRouter({ db, registry, items, auth, outbox, now, limits }));
- // The public host (sources.openvibe.network) shows only this page, health, readiness and
- // /release.json: Sources is internal. Browsers get a short honest HTML page, other clients the
- // text route index. Never cached, never indexed.
+ // Merchant opt-in (plan §4.4 channel 1): the JSON read/write API, and the signed-in pages that are its front door.
+ // /shared carries this release's own kit.css/kit-web.js/theme-loader.js, so the pages do not depend on another host.
+ if (merchant && merchantAuth) {
+ app.use('/shared', require('openvibe-shared/serve').handler());
+ app.use(merchantRouter({ db, merchant, registry, ingest, auth, merchantAuth, limits }));
+ const networkOrigin = (() => { try { return new URL(config.networkUrl).origin; } catch { return 'https://openvibe.network'; } })();
+ app.use('/merchant', (req, res, next) => {
+ res.setHeader('X-Robots-Tag', 'noindex, nofollow');
+ res.setHeader('Content-Security-Policy', `default-src 'none'; style-src 'self' 'unsafe-inline'; script-src 'self'; img-src 'self' data: https:; connect-src 'self' ${networkOrigin}; form-action 'self'; base-uri 'none'; frame-ancestors 'none'`);
+ next();
+ }, express.urlencoded({ extended: false, limit: '32kb' }), merchantAuth.page, merchantPages({
+ db, merchant, registry, ingest, merchantAuth, limits, log,
+ }));
+ }
+
+ // The public host (sources.openvibe.network) shows the internal-service page, the merchant opt-in pages and
+ // health/readiness/release.json: the JSON API stays loopback-only. Browsers get a short honest HTML page, other
+ // clients the text route index. Never cached, never indexed.
app.get('/', (req, res) => {
app.get('/', (req, res) => {
res.setHeader('X-Robots-Tag', 'noindex, nofollow');
res.setHeader('X-Robots-Tag', 'noindex, nofollow');
res.setHeader('Vary', 'Accept');
res.setHeader('Vary', 'Accept');
@@ -1,17 +1,22 @@
@@ -1,17 +1,22 @@
'use strict';
'use strict';
/**
/**
- * Service-token authentication and capability checks. Every API route is for services (other
- * OpenVibe services, or the Network admin surface acting through its own principal): an
- * RS256 client-credentials token from OpenVibe.Network (audience openvibe.sources), verified
- * offline against Network's JWKS (openvibe-sdk/auth, one client per URL, kept fresh and served
- * through an outage), and ONE capability per route.
+ * Service-token authentication and capability checks. API routes for services (other OpenVibe
+ * services, or the Network admin surface acting through its own principal) use an RS256
+ * client-credentials token from OpenVibe.Network (audience openvibe.sources), verified offline
+ * against Network's JWKS (openvibe-sdk/auth, one client per URL, kept fresh and served through an
+ * outage), and ONE capability per route.
+ *
+ * The merchant opt-in routes also accept a person's Network session token (the same RS256 key and
+ * issuer, no `typ`, a usr_… `subject_id`): the merchant is the owner of the claim. Pages use the
+ * same verification on their own first-party cookie. A first-party service may act for a person
+ * only through the X-OV-Subject header (server/merchant-auth.js), as OpenVibe.Space does.
*
*
* The sources.* capabilities are released in openvibe-contracts (docs/capabilities-proposal/ is
* The sources.* capabilities are released in openvibe-contracts (docs/capabilities-proposal/ is
* kept in step with those manifests), so the library's own grant rule decides every one: an exact
* kept in step with those manifests), so the library's own grant rule decides every one: an exact
* id or a `family.*` grant, and capability.unknown for an id it does not know.
* id or a `family.*` grant, and capability.unknown for an id it does not know.
*/
*/
-const { serviceAuth, capabilities, http } = require('openvibe-contracts');
-const { jwksClient } = require('openvibe-sdk/auth');
+const { serviceAuth, capabilities, http, ids } = require('openvibe-contracts');
+const { jwksClient, verifyUserToken } = require('openvibe-sdk/auth');
const CAPS = Object.freeze({
const CAPS = Object.freeze({
read: 'sources.source.read',
read: 'sources.source.read',
@@ -23,8 +28,9 @@ function checkCapability(claims, capabilityId) {
@@ -23,8 +28,9 @@ function checkCapability(claims, capabilityId) {
return capabilities.check(claims, capabilityId);
return capabilities.check(claims, capabilityId);
}
}
-function createAuth({ config, log = console }) {
+function createAuth({ config, log = console, jwks = null }) {
const headerKid = (token) => { try { return JSON.parse(Buffer.from(String(token).split('.')[0], 'base64url').toString('utf8')).kid || null; } catch { return null; } };
const headerKid = (token) => { try { return JSON.parse(Buffer.from(String(token).split('.')[0], 'base64url').toString('utf8')).kid || null; } catch { return null; } };
+ const userKeys = jwks || jwksClient(config.jwksUrl, { log });
/** { ok, claims } or { ok: false, code, reason }. The SDK's error names the internal JWKS URL: logged, never answered. */
/** { ok, claims } or { ok: false, code, reason }. The SDK's error names the internal JWKS URL: logged, never answered. */
async function verifyService(token) {
async function verifyService(token) {
@@ -43,6 +49,23 @@ function createAuth({ config, log = console }) {
@@ -43,6 +49,23 @@ function createAuth({ config, log = console }) {
return last;
return last;
}
}
+ /**
+ * A person's Network session token → { claims, subject: usr_…, name } or null. Typed tokens (FedCM assertions,
+ * realtime tickets) and service/app principals are refused by verifyUserToken; the subject must be a usr_ id.
+ */
+ async function verifyUser(token) {
+ if (!token) return null;
+ try {
+ const claims = await verifyUserToken(String(token), { jwks: userKeys, issuer: config.issuer });
+ const subject = ids.isSubjectId('user', claims.subject_id) ? claims.subject_id
+ : (ids.isSubjectId('user', claims.sub) ? claims.sub : null);
+ if (!subject) return null;
+ return { claims, subject, name: claims.username || claims.display_name || claims.name || null };
+ } catch {
+ return null;
+ }
+ }
+
/** Express guard: one capability. Sets req.principal = { sub, cap, jti }. */
/** Express guard: one capability. Sets req.principal = { sub, cap, jti }. */
function requireCap(id) {
function requireCap(id) {
return async function capGuard(req, res, next) {
return async function capGuard(req, res, next) {
@@ -62,7 +85,7 @@ function createAuth({ config, log = console }) {
@@ -62,7 +85,7 @@ function createAuth({ config, log = console }) {
return next();
return next();
};
};
}
}
- return { requireCap };
+ return { requireCap, verifyService, verifyUser };
}
}
module.exports = { CAPS, createAuth, checkCapability };
module.exports = { CAPS, createAuth, checkCapability };
@@ -25,13 +25,14 @@ function load(env = process.env) {
@@ -25,13 +25,14 @@ function load(env = process.env) {
// The JWKS the SDK's client (openvibe-sdk/auth jwksClient) fetches, caches and keeps fresh; one URL,
// The JWKS the SDK's client (openvibe-sdk/auth jwksClient) fetches, caches and keeps fresh; one URL,
// since the SDK keeps one client per URL. Defaults to Network's internal JWKS.
// since the SDK keeps one client per URL. Defaults to Network's internal JWKS.
const jwksUrl = strip(env.OV_NETWORK_JWKS_URL || `${networkInternalUrl}/api/.well-known/jwks`);
const jwksUrl = strip(env.OV_NETWORK_JWKS_URL || `${networkInternalUrl}/api/.well-known/jwks`);
+ const baseUrl = strip(env.BASE_URL || (isProduction ? 'https://sources.openvibe.network' : `http://localhost:${port}`));
return {
return {
port,
port,
host: env.HOST || '127.0.0.1',
host: env.HOST || '127.0.0.1',
nodeEnv,
nodeEnv,
isProduction,
isProduction,
serviceId: 'sources',
serviceId: 'sources',
- baseUrl: strip(env.BASE_URL || (isProduction ? 'https://sources.openvibe.network' : `http://localhost:${port}`)),
+ baseUrl,
networkUrl,
networkUrl,
networkInternalUrl,
networkInternalUrl,
@@ -47,6 +48,35 @@ function load(env = process.env) {
@@ -47,6 +48,35 @@ function load(env = process.env) {
oauth: {
oauth: {
clientId: env.OV_OAUTH_CLIENT_ID || 'sources',
clientId: env.OV_OAUTH_CLIENT_ID || 'sources',
clientSecret: env.OV_OAUTH_CLIENT_SECRET || '',
clientSecret: env.OV_OAUTH_CLIENT_SECRET || '',
+ // The Network OAuth client `sources` must be registered with this exact redirect URI.
+ redirectUri: strip(env.OV_OAUTH_REDIRECT_URI || `${baseUrl}/merchant/callback`),
+ scope: env.OV_OAUTH_SCOPE || 'profile',
+ },
+ cookies: {
+ // A first-party session cookie; Secure everywhere except local tests/development over http.
+ secure: env.SOURCES_COOKIE_SECURE != null ? env.SOURCES_COOKIE_SECURE === '1' : isProduction,
+ },
+
+ // Merchant opt-in (server/merchant.js): proof checks, the session form secret, and the shape of the source
+ // created only after verification. A sitemap becomes a crawl; a product feed uses the feed adapter.
+ merchant: {
+ pendingDays: Math.max(1, int(env.SOURCES_MERCHANT_PENDING_DAYS, 14)),
+ // Pending claims are checked again on this cadence; 0 turns the worker pass off (tests).
+ recheckIntervalMs: Math.max(0, int(env.SOURCES_MERCHANT_RECHECK_MS, 10 * 60 * 1000)),
+ metaMaxBytes: Math.min(5 * 1024 * 1024, Math.max(1024, int(env.SOURCES_MERCHANT_META_MAX_BYTES, 1024 * 1024))),
+ metaTimeoutMs: Math.min(120000, Math.max(1000, int(env.SOURCES_MERCHANT_META_TIMEOUT_MS, 10000))),
+ dnsServers: list(env.SOURCES_MERCHANT_DNS_SERVERS, []),
+ // CSRF/state HMAC key. Production should set SOURCES_SESSION_SECRET (or shares OV_OAUTH_CLIENT_SECRET).
+ sessionSecret: env.SOURCES_SESSION_SECRET || env.OV_OAUTH_CLIENT_SECRET || 'sources-merchant-development',
+ crawlMinIntervalMs: Math.min(86400000, Math.max(0, int(env.SOURCES_MERCHANT_CRAWL_MIN_INTERVAL_MS, 1000))),
+ feedMinIntervalMs: Math.min(86400000, Math.max(0, int(env.SOURCES_MERCHANT_FEED_MIN_INTERVAL_MS, 60000))),
+ pollIntervalSec: Math.min(7 * 86400, Math.max(1, int(env.SOURCES_MERCHANT_POLL_INTERVAL_SEC, 3600))),
+ crawlStaleAfterSec: Math.min(90 * 86400, Math.max(1, int(env.SOURCES_MERCHANT_CRAWL_STALE_AFTER_SEC, 86400))),
+ feedStaleAfterSec: Math.min(90 * 86400, Math.max(1, int(env.SOURCES_MERCHANT_FEED_STALE_AFTER_SEC, 10800))),
+ maxItems: Math.min(12000, Math.max(1, int(env.SOURCES_MERCHANT_MAX_ITEMS, 1000))),
+ crawlMaxPages: Math.min(200000, Math.max(1, int(env.SOURCES_MERCHANT_MAX_PAGES, 5000))),
+ crawlPagesPerRun: Math.min(1000, Math.max(1, int(env.SOURCES_MERCHANT_PAGES_PER_RUN, 100))),
+ crawlRecrawlSec: Math.min(30 * 86400, Math.max(3600, int(env.SOURCES_MERCHANT_RECRAWL_SEC, 86400))),
},
},
// PostgreSQL (ADR-035): DATABASE_URL serves (PgBouncer), DATABASE_DIRECT_URL migrates (owner role).
// PostgreSQL (ADR-035): DATABASE_URL serves (PgBouncer), DATABASE_DIRECT_URL migrates (owner role).
@@ -25,9 +25,11 @@ const { jwksClient } = require('openvibe-sdk/auth');
@@ -25,9 +25,11 @@ const { jwksClient } = require('openvibe-sdk/auth');
const { gracefulStop, telemetry } = require('openvibe-sdk/service');
const { gracefulStop, telemetry } = require('openvibe-sdk/service');
const { createPgTelemetrySink, startTelemetryRetention } = require('openvibe-sdk/telemetry');
const { createPgTelemetrySink, startTelemetryRetention } = require('openvibe-sdk/telemetry');
const { createAuth } = require('./auth');
const { createAuth } = require('./auth');
+const { createMerchant } = require('./merchant');
+const { createMerchantAuth } = require('./merchant-auth');
const { createApp } = require('./app');
const { createApp } = require('./app');
-async function start({ config, db: givenDb = null, now = () => Date.now(), fetchImpl = globalThis.fetch, lookupImpl, log = console, listen = true, limitsNow = null } = {}) {
+async function start({ config, db: givenDb = null, now = () => Date.now(), fetchImpl = globalThis.fetch, lookupImpl, merchantResolver = null, log = console, listen = true, limitsNow = null } = {}) {
config = config || load();
config = config || load();
// PostgreSQL (ADR-035): opened and migrated here unless the caller (a test) hands in a migrated handle.
// PostgreSQL (ADR-035): opened and migrated here unless the caller (a test) hands in a migrated handle.
const db = givenDb || await openDb(config, { log });
const db = givenDb || await openDb(config, { log });
@@ -53,14 +55,26 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetch
@@ -53,14 +55,26 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetch
const ingest = createIngest({ db, config, registry, items, robots, fetcher, spacer, emitEvent, now, log, outbox });
const ingest = createIngest({ db, config, registry, items, robots, fetcher, spacer, emitEvent, now, log, outbox });
const scheduler = createScheduler({ db, ingest, config, now, log });
const scheduler = createScheduler({ db, ingest, config, now, log });
const keys = jwksClient(config.jwksUrl, { fetch: fetchImpl, log });
const keys = jwksClient(config.jwksUrl, { fetch: fetchImpl, log });
- const auth = createAuth({ config, log });
- const app = createApp({ config, db, registry, items, ingest, scheduler, auth, outbox, now, log, limitsNow });
+ const auth = createAuth({ config, log, jwks: keys });
+ const merchant = createMerchant({ db, config, registry, fetcher, resolver: merchantResolver, now, log });
+ const merchantAuth = createMerchantAuth({ config, auth, log, fetchImpl });
+ const app = createApp({ config, db, registry, items, ingest, scheduler, auth, outbox, now, log, limitsNow, merchant, merchantAuth });
// One JWKS client for the process (the SDK shares it with verifyUserToken): refresh in the background
// One JWKS client for the process (the SDK shares it with verifyUserToken): refresh in the background
// on an unref'd timer, keeping the last good keys through a Network outage.
// on an unref'd timer, keeping the last good keys through a Network outage.
keys.start();
keys.start();
const keyLoaded = keys.keys().catch(() => null);
const keyLoaded = keys.keys().catch(() => null);
outbox.start();
outbox.start();
if (config.worker.enabled) scheduler.start();
if (config.worker.enabled) scheduler.start();
+ // Pending merchant proofs are checked again on a slow cadence (the merchant can always press "Check now"). A claim
+ // that never verifies fails after merchant.pendingDays. 0 turns the worker pass off (tests).
+ let merchantTimer = null;
+ let merchantKick = null;
+ if (config.merchant.recheckIntervalMs > 0) {
+ merchantTimer = setInterval(() => merchant.recheck().catch((err) => log.error(`[merchant] recheck: ${err.stack || err}`)), config.merchant.recheckIntervalMs);
+ merchantTimer.unref?.();
+ merchantKick = setTimeout(() => merchant.recheck().catch((err) => log.error(`[merchant] recheck: ${err.stack || err}`)), 5000);
+ merchantKick.unref?.();
+ }
const pruneTimer = setInterval(async () => { try { await outbox.outbox.prune(); } catch (err) { log.error(`[outbox] prune: ${err.message}`); } }, 6 * 3600 * 1000);
const pruneTimer = setInterval(async () => { try { await outbox.outbox.prune(); } catch (err) { log.error(`[outbox] prune: ${err.message}`); } }, 6 * 3600 * 1000);
pruneTimer.unref?.();
pruneTimer.unref?.();
@@ -75,6 +89,8 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetch
@@ -75,6 +89,8 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetch
async function close() {
async function close() {
clearInterval(pruneTimer);
clearInterval(pruneTimer);
+ if (merchantTimer) clearInterval(merchantTimer);
+ if (merchantKick) clearTimeout(merchantKick);
keys.stop();
keys.stop();
await scheduler.stop();
await scheduler.stop();
await outbox.stop();
await outbox.stop();
@@ -85,7 +101,7 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetch
@@ -85,7 +101,7 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetch
if (!givenDb) await db.close();
if (!givenDb) await db.close();
}
}
- return { config, db, registry, items, guard, fetcher, robots, ingest, scheduler, outbox, keys, keyLoaded, auth, app, server, close };
+ return { config, db, registry, items, guard, fetcher, robots, ingest, scheduler, outbox, keys, keyLoaded, auth, merchant, merchantAuth, app, server, close };
}
}
if (require.main === module) {
if (require.main === module) {
new file mode 100644
new file mode 100644
@@ -0,0 +1,225 @@
@@ -0,0 +1,225 @@
+'use strict';
+/**
+ * Merchant authentication for the opt-in flow.
+ *
+ * API `/api/v1/merchant/*`
+ * - a person's Network session token in `Authorization: Bearer` (the merchant owns the claims); or
+ * - a first-party service token with `sources.source.read` / `sources.source.manage` and X-OV-Subject,
+ * the way OpenVibe.Space lets a first-party service act for a person. An app/module token cannot delegate.
+ * Pages `/merchant/*`
+ * - the Sources first-party `ov_token` cookie, set by the OAuth callback below and verified offline with
+ * openvibe-sdk/auth verifyUserToken. There is no client-side token storage.
+ *
+ * The OAuth client (`OV_OAUTH_CLIENT_ID`, default `sources`) must be registered at OpenVibe.Network with
+ * `<BASE_URL>/merchant/callback` as a redirect URI. Forms carry an HMAC CSRF token bound to the merchant subject;
+ * cookies are httpOnly, SameSite=Lax, Secure in production.
+ */
+const crypto = require('crypto');
+const { http, ids } = require('openvibe-contracts');
+const { checkCapability } = require('./auth');
+
+const ACCESS_COOKIE = 'ov_token';
+const STATE_COOKIE = 'ov_merchant_state';
+const NEXT_COOKIE = 'ov_merchant_next';
+const FLOW_MAX_AGE = 10 * 60 * 1000;
+const SESSION_MAX_AGE = 24 * 3600 * 1000;
+const PRINCIPAL_SUB = /^(svc|app|mod|node|agent):/;
+
+function decodePayload(token) {
+ const parts = String(token || '').split('.');
+ if (parts.length !== 3) return null;
+ try { return JSON.parse(Buffer.from(parts[1], 'base64url').toString('utf8')); } catch { return null; }
+}
+
+function cookie(req, name) {
+ const m = String((req && req.headers && req.headers.cookie) || '').match(new RegExp(`(?:^|;\\s*)${name}=([^;]*)`));
+ if (!m) return null;
+ try { return decodeURIComponent(m[1]); } catch { return null; }
+}
+
+function bearer(req) {
+ const h = String((req && req.headers && req.headers.authorization) || '');
+ return h.startsWith('Bearer ') ? h.slice(7).trim() || null : null;
+}
+
+/** Only a same-site merchant path is a post-sign-in destination. */
+function safeNext(value) {
+ const n = String(value == null ? '' : value);
+ if (!n || /[\u0000-\u001f\u007f\\]/.test(n)) return '/merchant';
+ if (!/^\/merchant(?:\/|$|\?|#)/.test(n)) return '/merchant';
+ return n;
+}
+
+/** `/oauth/authorize` on the public Network, with the state the callback checks. */
+function authorizeUrl(config, state) {
+ const u = new URL('/oauth/authorize', config.networkUrl);
+ u.searchParams.set('client_id', config.oauth.clientId);
+ u.searchParams.set('redirect_uri', config.oauth.redirectUri);
+ u.searchParams.set('response_type', 'code');
+ u.searchParams.set('scope', config.oauth.scope || 'profile');
+ u.searchParams.set('state', state);
+ return u.toString();
+}
+
+function createMerchantAuth({ config, auth, log = console, fetchImpl = globalThis.fetch }) {
+ const secret = config.merchant.sessionSecret;
+ const secure = Boolean(config.cookies && config.cookies.secure);
+ const baseCookie = { sameSite: 'lax', secure, httpOnly: true, path: '/' };
+ const flowCookie = { sameSite: 'lax', secure, httpOnly: true, path: '/merchant', maxAge: FLOW_MAX_AGE };
+
+ const csrfFor = (subject) => crypto.createHmac('sha256', secret).update(`sources-merchant:${subject}`).digest('base64url').slice(0, 32);
+
+ function checkCsrf(principal, token) {
+ if (!principal || !principal.subject || typeof token !== 'string') return false;
+ const expected = csrfFor(principal.subject);
+ const a = Buffer.from(expected);
+ const b = Buffer.from(token);
+ return a.length === b.length && crypto.timingSafeEqual(a, b);
+ }
+
+ function setSession(res, token) {
+ res.cookie(ACCESS_COOKIE, token, { ...baseCookie, maxAge: SESSION_MAX_AGE });
+ }
+
+ function clearSession(res) {
+ res.clearCookie(ACCESS_COOKIE, { ...baseCookie, maxAge: undefined });
+ }
+
+ async function resolvePage(req) {
+ const token = bearer(req) || cookie(req, ACCESS_COOKIE);
+ if (!token) return null;
+ const payload = decodePayload(token);
+ if (payload && (payload.actor_type || PRINCIPAL_SUB.test(String(payload.sub || '')))) return null;
+ const u = await auth.verifyUser(token);
+ if (!u) return null;
+ return { subject: u.subject, name: u.name, kind: 'user', claims: u.claims };
+ }
+
+ /** Attach req.merchant when the page cookie is a live person session; anonymous stays anonymous. */
+ async function page(req, _res, next) {
+ try {
+ const merchant = await resolvePage(req);
+ if (merchant) {
+ req.merchant = merchant;
+ req.principal = { sub: `user:${merchant.subject}` };
+ }
+ return next();
+ } catch (err) { return next(err); }
+ }
+
+ function requirePage(req, res, next) {
+ if (req.merchant) return next();
+ const nextPath = req.method === 'GET' || req.method === 'HEAD' ? safeNext(req.originalUrl) : '/merchant';
+ return res.redirect(303, `/merchant/login?next=${encodeURIComponent(nextPath)}`);
+ }
+
+ function problem(res, req, status, code, detail) {
+ return http.sendProblem(res, status, code, { detail, ctx: req.ov });
+ }
+
+ /**
+ * API guard: a person's token (owner) or a first-party service token with `cap` plus X-OV-Subject. Sets
+ * req.merchant = { subject, name, kind } and req.principal for the per-actor limiter.
+ */
+ function api(cap) {
+ return async function merchantGuard(req, res, next) {
+ const token = bearer(req);
+ if (!token) return problem(res, req, 401, 'token.missing', 'a merchant session or a service token is required');
+ const payload = decodePayload(token);
+ const principalToken = payload && (payload.actor_type || PRINCIPAL_SUB.test(String(payload.sub || '')));
+ try {
+ if (principalToken) {
+ const r = await auth.verifyService(token);
+ if (!r.ok) return problem(res, req, r.code === 'token.unavailable' ? 503 : 401, r.code, r.reason);
+ const c = checkCapability(r.claims, cap);
+ if (!c.allowed) return problem(res, req, 403, c.code, c.reason);
+ const delegated = req.get('x-ov-subject') || req.query.subject || null;
+ if (!delegated) return problem(res, req, 400, 'merchant.subject_required', 'a first-party service must name the merchant with X-OV-Subject');
+ if (!String(r.claims.sub).startsWith('svc:')) return problem(res, req, 403, 'merchant.subject_forbidden', 'only a first-party service may act for a person');
+ if (!ids.isSubjectId('user', String(delegated))) return problem(res, req, 400, 'merchant.subject_invalid', 'X-OV-Subject must be a usr_… subject id');
+ req.merchant = { subject: String(delegated), name: null, kind: 'service', service: r.claims.sub };
+ req.principal = { sub: r.claims.sub, cap: r.claims.cap, jti: r.claims.jti };
+ return next();
+ }
+ const u = await auth.verifyUser(token);
+ if (!u) return problem(res, req, 401, 'merchant.auth_required', 'the bearer token is not a valid OpenVibe person session');
+ req.merchant = { subject: u.subject, name: u.name, kind: 'user', claims: u.claims };
+ req.principal = { sub: `user:${u.subject}` };
+ return next();
+ } catch (err) { return next(err); }
+ };
+ }
+
+ async function login(req, res) {
+ const next = safeNext(req.query.next);
+ const existing = await resolvePage(req);
+ if (existing) return res.redirect(303, next);
+ const state = crypto.randomBytes(16).toString('hex');
+ res.cookie(STATE_COOKIE, state, flowCookie);
+ res.cookie(NEXT_COOKIE, next, flowCookie);
+ return res.redirect(authorizeUrl(config, state));
+ }
+
+ async function tokenGrant(body) {
+ let lastErr = null;
+ for (const base of [config.networkInternalUrl, config.networkUrl]) {
+ if (!base) continue;
+ try {
+ const res = await fetchImpl(`${base}/oauth/token`, {
+ method: 'POST',
+ headers: { 'Content-Type': 'application/json' },
+ body: JSON.stringify({ client_id: config.oauth.clientId, client_secret: config.oauth.clientSecret, ...body }),
+ signal: AbortSignal.timeout(10_000),
+ });
+ const data = await res.json().catch(() => ({}));
+ if (!res.ok) {
+ const err = new Error(data.error_description || data.error || `token grant failed (${res.status})`);
+ err.status = res.status;
+ err.error = data.error || 'invalid_grant';
+ throw err;
+ }
+ return data;
+ } catch (err) {
+ lastErr = err;
+ if (err.status && err.status < 500) throw err;
+ }
+ }
+ throw lastErr || new Error('Network unreachable');
+ }
+
+ async function callback(req, res) {
+ const { code, state, error } = req.query;
+ const next = safeNext(cookie(req, NEXT_COOKIE));
+ const expected = cookie(req, STATE_COOKIE);
+ res.clearCookie(STATE_COOKIE, { ...flowCookie, maxAge: undefined });
+ res.clearCookie(NEXT_COOKIE, { ...flowCookie, maxAge: undefined });
+ if (error) return res.redirect(303, `/merchant?auth_error=${encodeURIComponent(String(error).slice(0, 80))}`);
+ if (!code || !state || !expected || String(state).length !== String(expected).length
+ || !crypto.timingSafeEqual(Buffer.from(String(state)), Buffer.from(String(expected)))) {
+ return res.status(400).type('html').send('<!doctype html><meta charset="utf-8"><title>Sign-in failed</title><p>The sign-in response did not match this browser. <a href="/merchant/login">Try again</a>.</p>');
+ }
+ try {
+ const data = await tokenGrant({ grant_type: 'authorization_code', redirect_uri: config.oauth.redirectUri, code });
+ const u = await auth.verifyUser(data.access_token);
+ if (!u) throw Object.assign(new Error('the Network returned a token without a person subject'), { status: 401 });
+ setSession(res, data.access_token);
+ return res.redirect(303, next);
+ } catch (err) {
+ log.error(`[merchant] sign-in failed: ${err.message}`);
+ return res.status(502).type('html').send('<!doctype html><meta charset="utf-8"><title>Sign-in failed</title><p>OpenVibe.Network could not complete the sign-in. <a href="/merchant/login">Try again</a>.</p>');
+ }
+ }
+
+ function logout(req, res) {
+ clearSession(res);
+ return res.redirect(303, '/merchant');
+ }
+
+ return {
+ api, page, requirePage, login, callback, logout,
+ resolvePage, checkCsrf, csrfFor, setSession, clearSession,
+ ACCESS_COOKIE, STATE_COOKIE, NEXT_COOKIE, safeNext, authorizeUrl,
+ };
+}
+
+module.exports = { createMerchantAuth, authorizeUrl, safeNext, decodePayload, cookie, bearer };
new file mode 100644
new file mode 100644
@@ -0,0 +1,139 @@
@@ -0,0 +1,139 @@
+'use strict';
+/**
+ * The merchant opt-in pages (plain forms, no client-side state). The JSON API and these pages share server/merchant.js:
+ * the page is the signed-in merchant's front door, and the API is what Report calls later.
+ *
+ * GET /merchant the merchant's claims, or the guest explanation
+ * GET /merchant/sources/:id proof instructions, status, terms note and last crawl
+ * POST /merchant/sources add a domain (CSRF)
+ * POST /merchant/sources/:id/verify check the proof now (CSRF)
+ * POST /merchant/sources/:id/withdraw disable the source and keep the note (CSRF)
+ * GET /merchant/login|callback|logout
+ */
+const express = require('express');
+const { MerchantError } = require('./merchant');
+const views = require('./views/merchant');
+const { createMerchantView } = require('./api/merchant');
+
+const NOTICES = {
+ created: 'Domain added. Publish the proof below, then check now. Nothing is fetched or enabled before the proof is found.',
+ verified: 'Proof found. Your source was created and enabled.',
+ pending: 'The proof was not found yet. Publish it and check again.',
+ withdrawn: 'Permission withdrawn. The source is disabled and its terms note is kept.',
+};
+
+function merchantPages({ db, merchant, registry, ingest, merchantAuth, limits, log = console }) {
+ const router = express.Router();
+ const decorate = createMerchantView({ db, merchant, registry, ingest });
+ const readLimit = limits.budget('sources.merchant.read');
+ const manageLimit = limits.budget('sources.merchant.manage');
+
+ const noticeFor = (query) => Object.keys(NOTICES).filter(k => query[k] != null && query[k] !== '0').map(k => NOTICES[k]).join(' ');
+ const send = (res, status, html) => res.status(status).type('html').send(html);
+
+ async function listPage(req, res, status = 200, extra = {}) {
+ const claims = req.merchant ? await Promise.all((await merchant.list(req.merchant)).map(decorate)) : [];
+ return send(res, status, views.merchantPage({
+ merchant: req.merchant,
+ claims,
+ csrf: req.merchant ? merchantAuth.csrfFor(req.merchant.subject) : '',
+ notice: noticeFor(req.query),
+ authError: req.query.auth_error ? String(req.query.auth_error).slice(0, 80) : '',
+ ...extra,
+ }));
+ }
+
+ async function detailPage(req, res, status = 200, extra = {}) {
+ const row = await merchant.get(req.merchant, req.params.id);
+ return send(res, status, views.detailPage({
+ merchant: req.merchant,
+ claim: await decorate(row),
+ csrf: merchantAuth.csrfFor(req.merchant.subject),
+ notice: noticeFor(req.query),
+ noticeKind: req.query.pending != null && req.query.pending !== '0' ? 'warn' : 'ok',
+ ...extra,
+ }));
+ }
+
+ function csrfOk(req) {
+ return merchantAuth.checkCsrf(req.merchant, req.body && req.body.csrf);
+ }
+
+ router.get('/', readLimit, async (req, res, next) => {
+ try {
+ if (!req.merchant) {
+ return send(res, 200, views.guestPage({
+ authError: req.query.auth_error ? String(req.query.auth_error).slice(0, 80) : '',
+ }));
+ }
+ return await listPage(req, res);
+ } catch (err) { next(err); }
+ });
+
+ router.get('/login', async (req, res, next) => {
+ try { await merchantAuth.login(req, res); } catch (err) { next(err); }
+ });
+ router.get('/callback', async (req, res, next) => {
+ try { await merchantAuth.callback(req, res); } catch (err) { next(err); }
+ });
+ router.get('/logout', (req, res) => merchantAuth.logout(req, res));
+
+ router.get('/sources', (req, res) => res.redirect(303, '/merchant'));
+ router.get('/sources/:id', merchantAuth.requirePage, readLimit, async (req, res, next) => {
+ try { return await detailPage(req, res); } catch (err) {
+ if (err instanceof MerchantError) return send(res, err.status, views.errorPage({ status: err.status, message: err.message }).html);
+ next(err);
+ }
+ });
+
+ router.post('/sources', merchantAuth.requirePage, manageLimit, async (req, res, next) => {
+ try {
+ if (!csrfOk(req)) return send(res, 403, views.errorPage({ status: 403, message: 'This form expired or did not come from this page. Reload and try again.' }).html);
+ const row = await merchant.add(req.merchant, req.body || {});
+ return res.redirect(303, `/merchant/sources/${row.id}?created=1`);
+ } catch (err) {
+ if (err instanceof MerchantError) {
+ try { return await listPage(req, res, err.status, { error: err.message }); } catch (inner) { return next(inner); }
+ }
+ next(err);
+ }
+ });
+
+ router.post('/sources/:id/verify', merchantAuth.requirePage, manageLimit, async (req, res, next) => {
+ try {
+ if (!csrfOk(req)) return send(res, 403, views.errorPage({ status: 403, message: 'This form expired or did not come from this page. Reload and try again.' }).html);
+ const out = await merchant.verify(req.merchant, req.params.id);
+ const verified = out.row && out.row.status === 'verified';
+ return res.redirect(303, `/merchant/sources/${out.row.id}?${verified ? 'verified=1' : 'pending=1'}`);
+ } catch (err) {
+ if (err instanceof MerchantError) {
+ try { return await detailPage(req, res, err.status, { error: err.message }); } catch (inner) { return next(inner); }
+ }
+ next(err);
+ }
+ });
+
+ router.post('/sources/:id/withdraw', merchantAuth.requirePage, manageLimit, async (req, res, next) => {
+ try {
+ if (!csrfOk(req)) return send(res, 403, views.errorPage({ status: 403, message: 'This form expired or did not come from this page. Reload and try again.' }).html);
+ const row = await merchant.withdraw(req.merchant, req.params.id);
+ return res.redirect(303, `/merchant/sources/${row.id}?withdrawn=1`);
+ } catch (err) {
+ if (err instanceof MerchantError) {
+ try { return await detailPage(req, res, err.status, { error: err.message }); } catch (inner) { return next(inner); }
+ }
+ next(err);
+ }
+ });
+
+ // A page error is HTML, not a JSON problem: the merchant can read it and go back.
+ router.use((err, req, res, _next) => {
+ if (res.headersSent) return res.end();
+ log.error(`[merchant] ${req.method} ${req.originalUrl}: ${err.stack || err}`);
+ return send(res, 500, views.errorPage({ status: 500, message: 'Something went wrong on our side. Please try again.' }).html);
+ });
+
+ return router;
+}
+
+module.exports = { merchantPages, NOTICES };
new file mode 100644
new file mode 100644
@@ -0,0 +1,398 @@
@@ -0,0 +1,398 @@
+'use strict';
+/**
+ * Merchant opt-in ("List your store on OpenVibe", plan §4.4 channel 1).
+ *
+ * A signed-in merchant adds a domain, proves control of it (the OpenVibe.Space pattern: a DNS TXT record at
+ * _openvibe-sources.<domain>, or a meta tag on a page of the domain), and names the sitemap or product feed Sources may
+ * read. Nothing enters `sources` before the proof is present: a pending claim lives only in `merchant_sources`. On
+ * verification one enabled source is created in the same transaction, and its terms_note records who gave the explicit
+ * permission, when, and how control was verified. Withdrawal disables that source and keeps the note (and the claim) as
+ * the audit trail. The fetcher/crawler still obeys robots.txt on every request; explicit permission is not a robots
+ * override.
+ *
+ * The challenge is a random token in a public DNS/meta record. It is not a credential and is shown only to the owner (or
+ * a first-party service acting for them, with X-OV-Subject).
+ *
+ * ADR 0007 still governs technique: this is a source whose access verdict is the merchant's explicit proof, not a probe
+ * of a site we have no permission from; every request it causes goes through the same fetcher, UA, robots and spacing.
+ */
+const crypto = require('crypto');
+const dns = require('dns');
+const { ids } = require('openvibe-contracts');
+const { decodeBody, decodeEntities } = require('./adapters/util');
+
+const CHALLENGE_PREFIX = '_openvibe-sources';
+const TXT_PREFIX = 'openvibe-sources-verification=';
+const META_NAME = 'openvibe-sources-verification';
+const METHODS = ['dns', 'meta'];
+const LISTING_KINDS = ['sitemap', 'feed'];
+const DAY = 24 * 3600 * 1000;
+
+const LABEL = '[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?';
+const HOSTNAME_RE = new RegExp(`^(?:${LABEL}\\.)+[a-z][a-z0-9-]{0,61}[a-z0-9]$`);
+const RESERVED_SUFFIXES = ['.local', '.internal', '.localhost', '.test', '.invalid', '.example', '.onion', '.home.arpa'];
+const ID_RE = /^msrc_[0-9A-HJKMNP-TV-Z]{26}$/;
+
+class MerchantError extends Error {
+ constructor(status, code, detail) { super(detail); this.status = status; this.code = code; }
+}
+
+const clip = (v, n) => String(v == null ? '' : v).replace(/[\u0000-\u001f\u007f]/g, ' ').trim().slice(0, n);
+const iso = (v) => (v == null ? null : new Date(Number(v)).toISOString());
+
+function isHostname(h) {
+ return typeof h === 'string' && h.length <= 253 && HOSTNAME_RE.test(h);
+}
+
+/** A claimed domain: lowercase, no port, a real DNS name, not an OpenVibe property or a reserved name. */
+function checkDomain(raw) {
+ const domain = String(raw || '').trim().toLowerCase().replace(/\.$/, '');
+ if (!isHostname(domain)) throw new MerchantError(422, 'merchant.domain_invalid', 'the domain must be a fully qualified host name such as shop.example.org');
+ if (RESERVED_SUFFIXES.some(s => domain.endsWith(s))) throw new MerchantError(422, 'merchant.domain_reserved', `${domain} is a reserved name and cannot be claimed`);
+ if (/(^|\.)openvibe\./.test(domain)) throw new MerchantError(422, 'merchant.domain_reserved', `${domain} is an OpenVibe domain and cannot be claimed`);
+ return domain;
+}
+
+/** A URL the merchant names: http(s), no credentials, at most 2048 characters, no secret-looking query parameter. */
+function checkUrl(raw, what) {
+ const s = String(raw == null ? '' : raw).trim();
+ if (!s || s.length > 2048) throw new MerchantError(422, 'merchant.url_invalid', `${what} must be an http(s) URL of at most 2048 characters`);
+ let u;
+ try { u = new URL(s); } catch { throw new MerchantError(422, 'merchant.url_invalid', `${what} is not a URL`); }
+ if (u.protocol !== 'http:' && u.protocol !== 'https:') throw new MerchantError(422, 'merchant.url_invalid', `${what} must be http(s)`);
+ if (u.username || u.password) throw new MerchantError(422, 'merchant.url_invalid', `${what} must not carry credentials`);
+ for (const k of u.searchParams.keys()) {
+ if (/^(api[_-]?key|apikey|key|token|access[_-]?token|auth|secret|client[_-]?secret|password|passwd|sig|signature)$/i.test(k)) {
+ throw new MerchantError(422, 'merchant.url_secret', `${what} must not carry a credential in its query string`);
+ }
+ }
+ u.hash = '';
+ return u.toString();
+}
+
+/** The claimed domain and a URL's host are the same site: the domain, its www twin, or a subdomain of it. */
+function sameSite(domain, host) {
+ const d = String(domain || '').toLowerCase().replace(/\.$/, '');
+ const h = String(host || '').toLowerCase().replace(/\.$/, '');
+ if (!d || !h) return false;
+ return h === d || h === `www.${d}` || d === `www.${h}` || h.endsWith(`.${d}`);
+}
+
+function sourceKeyFor(row) {
+ const slug = String(row.domain).replace(/[^a-z0-9]+/g, '-').replace(/^-+|-+$/g, '').slice(0, 40).replace(/-+$/g, '') || 'site';
+ const suffix = String(row.id).replace(/^msrc_/, '').slice(-8).toLowerCase();
+ return `merchant-${slug}-${suffix}`;
+}
+
+/** The explicit-permission note products read: who, when, how, and that withdrawal is possible. */
+function termsNoteFor(row, verifiedAt) {
+ const who = row.owner_name ? `${row.owner_sub} (${clip(row.owner_name, 80)})` : row.owner_sub;
+ const how = row.method === 'dns'
+ ? `a DNS TXT record at ${CHALLENGE_PREFIX}.${row.domain}`
+ : `a meta tag at ${clip(row.meta_url || `https://${row.domain}/`, 400)}`;
+ const what = row.listing_kind === 'sitemap' ? 'sitemap' : 'product feed';
+ return clip(`Merchant opt-in: ${who} gave OpenVibe explicit permission on ${iso(verifiedAt)} to list ${row.domain} from the ${what} at ${clip(row.listing_url, 500)}, verified by ${how}. The merchant may withdraw this permission, which disables this source; robots.txt is still checked on every request.`, 2000);
+}
+
+/** The source definition created only after the proof. A sitemap becomes a crawl; a feed uses the feed adapter. */
+function sourceFor(row, termsNote, config) {
+ const crawl = row.listing_kind === 'sitemap';
+ const atom = !crawl && /\.atom(?:\.xml)?$/i.test(new URL(row.listing_url).pathname);
+ const m = config.merchant;
+ return {
+ key: sourceKeyFor(row),
+ name: clip(row.store_name || `${row.domain} (merchant opt-in)`, 200),
+ type: crawl ? 'crawl' : (atom ? 'atom' : 'rss'),
+ category: 'deals',
+ homepage_url: `https://${row.domain}/`,
+ endpoints: [row.listing_url],
+ auth: { mode: 'none' },
+ robots_note: 'robots.txt is checked on every request and every redirect hop; a disallowed page is not fetched.',
+ terms_note: termsNote,
+ license_note: null,
+ min_interval_ms: crawl ? m.crawlMinIntervalMs : m.feedMinIntervalMs,
+ poll_interval_sec: m.pollIntervalSec,
+ stale_after_sec: crawl ? m.crawlStaleAfterSec : m.feedStaleAfterSec,
+ max_items: Math.min(m.maxItems, config.items.maxPerFetch),
+ enabled: true,
+ review_required: false,
+ sensitivity: 'none',
+ default_indexability: 'index',
+ search_visibility: null,
+ retire_after_sec: null,
+ crawl: crawl ? {
+ // The merchant named this sitemap: follow it (and any sitemap index below it), not other sitemaps found by guess.
+ sitemaps: false,
+ follow_links: false,
+ include: [],
+ exclude: [],
+ max_pages: m.crawlMaxPages,
+ pages_per_run: m.crawlPagesPerRun,
+ recrawl_sec: m.crawlRecrawlSec,
+ max_depth: 0,
+ require_structured: true,
+ } : null,
+ };
+}
+
+function defaultResolver(servers) {
+ const r = new dns.promises.Resolver({ timeout: 5000, tries: 2 });
+ if (servers && servers.length) r.setServers(servers);
+ return { resolveTxt: (name) => r.resolveTxt(name) };
+}
+
+/** True when the page carries <meta name="openvibe-sources-verification" content="<token>">, in any attribute order. */
+function hasVerificationMeta(html, token) {
+ const ATTR = /([A-Za-z_:][-A-Za-z0-9_:.]*)\s*=\s*(?:"([^"]*)"|'([^']*)'|([^\s"'=<>`]+))/g;
+ for (const tag of String(html || '').match(/<meta\b[^>]*>/gi) || []) {
+ const attrs = {};
+ let m;
+ ATTR.lastIndex = 0;
+ while ((m = ATTR.exec(tag))) attrs[m[1].toLowerCase()] = m[2] ?? m[3] ?? m[4] ?? '';
+ const name = attrs.name || attrs.property || '';
+ if (name.trim().toLowerCase() === META_NAME && decodeEntities(attrs.content || '').trim() === token) return true;
+ }
+ return false;
+}
+
+function createMerchant({ db, config, registry, fetcher, resolver = null, now = () => Date.now(), log = console }) {
+ const dnsResolver = resolver || defaultResolver(config.merchant.dnsServers);
+ const q = {
+ byId: db.prepare('SELECT * FROM merchant_sources WHERE id = ?'),
+ liveForDomain: db.prepare("SELECT * FROM merchant_sources WHERE domain = ? AND status IN ('pending','verified')"),
+ forOwner: db.prepare('SELECT * FROM merchant_sources WHERE owner_sub = ? ORDER BY created_at DESC, id DESC'),
+ insert: db.prepare(`INSERT INTO merchant_sources
+ (id, owner_sub, owner_name, store_name, domain, method, token, listing_url, listing_kind, meta_url, status,
+ created_at, updated_at)
+ VALUES (@id, @owner_sub, @owner_name, @store_name, @domain, @method, @token, @listing_url, @listing_kind, @meta_url,
+ 'pending', @now, @now)`),
+ checked: db.prepare(`UPDATE merchant_sources SET last_checked_at = ?, check_count = check_count + 1,
+ last_error = ?, updated_at = ? WHERE id = ?`),
+ verified: db.prepare(`UPDATE merchant_sources SET status = 'verified', source_key = ?, terms_note = ?, verified_at = ?,
+ verified_by = ?, verification_detail = ?, last_checked_at = ?, check_count = check_count + 1, last_error = NULL,
+ updated_at = ? WHERE id = ? AND status = 'pending'`),
+ failed: db.prepare("UPDATE merchant_sources SET status = 'failed', last_error = ?, updated_at = ? WHERE id = ? AND status = 'pending'"),
+ withdraw: db.prepare(`UPDATE merchant_sources SET status = 'withdrawn', withdrawn_at = ?, withdrawn_by = ?, updated_at = ?
+ WHERE id = ? AND status != 'withdrawn'`),
+ duePending: db.prepare(`SELECT * FROM merchant_sources WHERE status = 'pending'
+ AND COALESCE(last_checked_at, 0) < ? ORDER BY COALESCE(last_checked_at, 0), created_at LIMIT ?`),
+ };
+
+ function view(row) {
+ if (!row) return null;
+ const challenge = row.method === 'dns'
+ ? { type: 'TXT', name: `${CHALLENGE_PREFIX}.${row.domain}`, value: `${TXT_PREFIX}${row.token}`, note: 'Publish this record, then check now. You can remove it after verification.' }
+ : { type: 'meta', name: META_NAME, value: row.token, url: row.meta_url || `https://${row.domain}/`, html: `<meta name="${META_NAME}" content="${row.token}">` };
+ return {
+ id: row.id,
+ domain: row.domain,
+ status: row.status,
+ owner: { subject: row.owner_sub, name: row.owner_name || null },
+ store_name: row.store_name || null,
+ method: row.method,
+ listing: { kind: row.listing_kind, url: row.listing_url },
+ meta_url: row.meta_url || null,
+ source_key: row.source_key || null,
+ terms_note: row.terms_note || null,
+ verification: {
+ method: row.method,
+ challenge,
+ verified_at: iso(row.verified_at),
+ verified_by: row.verified_by || null,
+ detail: row.verification_detail || null,
+ last_checked_at: iso(row.last_checked_at),
+ check_count: Number(row.check_count || 0),
+ last_error: row.last_error || null,
+ },
+ withdrawal: row.withdrawn_at ? { at: iso(row.withdrawn_at), by: row.withdrawn_by || null } : null,
+ created_at: iso(row.created_at),
+ updated_at: iso(row.updated_at),
+ };
+ }
+
+ async function load(principal, id) {
+ if (!principal || !ids.isSubjectId('user', principal.subject)) throw new MerchantError(401, 'merchant.auth_required', 'a signed-in merchant is required');
+ if (!ID_RE.test(String(id || ''))) throw new MerchantError(404, 'merchant.not_found', 'no such merchant source');
+ const row = await q.byId.get(id);
+ if (!row || row.owner_sub !== principal.subject) throw new MerchantError(404, 'merchant.not_found', 'no such merchant source');
+ return row;
+ }
+
+ async function add(principal, input = {}) {
+ if (!principal || !ids.isSubjectId('user', principal.subject)) throw new MerchantError(401, 'merchant.auth_required', 'a signed-in merchant is required');
+ const body = input && typeof input === 'object' && !Array.isArray(input) ? input : {};
+ const domain = checkDomain(body.domain ?? body.hostname ?? body.host);
+ const method = String(body.method ?? body.verification_method ?? '').trim().toLowerCase();
+ if (!METHODS.includes(method)) throw new MerchantError(422, 'merchant.method_invalid', 'method must be dns or meta');
+ const listingKind = String(body.listing_kind || body.listing || (body.sitemap_url ? 'sitemap' : '') || (body.feed_url ? 'feed' : '')).trim().toLowerCase();
+ if (!LISTING_KINDS.includes(listingKind)) throw new MerchantError(422, 'merchant.listing_invalid', 'listing_kind must be sitemap or feed');
+ const listingUrl = checkUrl(body.listing_url ?? body.url ?? body.sitemap_url ?? body.feed_url, 'listing_url');
+ if (!sameSite(domain, new URL(listingUrl).hostname)) throw new MerchantError(422, 'merchant.listing_host', `the ${listingKind} must be on ${domain} or one of its subdomains`);
+ let metaUrl = null;
+ if (method === 'meta') {
+ const chosenMeta = body.meta_url ?? body.verification_url;
+ metaUrl = chosenMeta ? checkUrl(chosenMeta, 'meta_url') : `https://${domain}/`;
+ if (!sameSite(domain, new URL(metaUrl).hostname)) throw new MerchantError(422, 'merchant.meta_host', `the meta tag page must be on ${domain} or one of its subdomains`);
+ } else if (body.meta_url || body.verification_url) {
+ throw new MerchantError(422, 'merchant.meta_unused', 'meta_url is only for method meta');
+ }
+ if ((body.store_name != null && typeof body.store_name !== 'string') || (body.name != null && typeof body.name !== 'string')) {
+ throw new MerchantError(422, 'merchant.store_name_invalid', 'store_name must be text');
+ }
+ const storeName = body.store_name == null && body.name == null ? null : clip(body.store_name ?? body.name, 120) || null;
+
+ const live = await q.liveForDomain.get(domain);
+ if (live) {
+ if (live.owner_sub === principal.subject) throw new MerchantError(409, 'merchant.exists', `${domain} already has a ${live.status} claim on this account`);
+ throw new MerchantError(409, 'merchant.domain_taken', `${domain} is already claimed`);
+ }
+ const t = now();
+ const row = {
+ id: `msrc_${ids.ulid(t)}`,
+ owner_sub: principal.subject,
+ owner_name: principal.name ? clip(principal.name, 80) : null,
+ store_name: storeName,
+ domain,
+ method,
+ token: crypto.randomBytes(20).toString('hex'),
+ listing_url: listingUrl,
+ listing_kind: listingKind,
+ meta_url: metaUrl,
+ now: t,
+ };
+ try { await q.insert.run(row); } catch (err) {
+ if (err && err.code === '23505') throw new MerchantError(409, 'merchant.domain_taken', `${domain} is already claimed`);
+ throw err;
+ }
+ return await q.byId.get(row.id);
+ }
+
+ async function lookupDns(row) {
+ let records;
+ try { records = await dnsResolver.resolveTxt(`${CHALLENGE_PREFIX}.${row.domain}`); } catch (err) {
+ const code = err && err.code;
+ if (code === 'ENOTFOUND' || code === 'ENODATA' || code === 'NXDOMAIN') {
+ return { ok: false, error: `no TXT record at ${CHALLENGE_PREFIX}.${row.domain} yet` };
+ }
+ return { ok: false, error: `DNS lookup failed (${code || 'error'})` };
+ }
+ const values = (records || []).map(chunks => (Array.isArray(chunks) ? chunks.join('') : String(chunks)));
+ const expected = `${TXT_PREFIX}${row.token}`;
+ if (values.includes(expected)) return { ok: true, detail: `DNS TXT record ${CHALLENGE_PREFIX}.${row.domain} matched` };
+ return { ok: false, error: values.length ? `${CHALLENGE_PREFIX}.${row.domain} has TXT records, but not the expected value` : `no TXT record at ${CHALLENGE_PREFIX}.${row.domain} yet` };
+ }
+
+ async function lookupMeta(row) {
+ const urls = row.meta_url ? [row.meta_url] : [`https://${row.domain}/`, `http://${row.domain}/`];
+ let lastError = `no meta tag at ${urls[0]}`;
+ for (const url of urls) {
+ try {
+ const res = await fetcher.fetchUrl(url, {
+ headers: { Accept: 'text/html, application/xhtml+xml;q=0.9, */*;q=0.1' },
+ capBytes: config.merchant.metaMaxBytes,
+ timeoutMs: config.merchant.metaTimeoutMs,
+ });
+ if (res.status !== 200) { lastError = `HTTP ${res.status} at ${url}`; continue; }
+ const finalHost = new URL(res.finalUrl || url).hostname;
+ if (!sameSite(row.domain, finalHost)) { lastError = `redirected outside ${row.domain} (${finalHost})`; continue; }
+ const text = decodeBody(res.body, res.headers['content-type']);
+ if (hasVerificationMeta(text, row.token)) return { ok: true, detail: `meta tag at ${url} matched` };
+ lastError = `no ${META_NAME} meta tag at ${url}`;
+ } catch (err) {
+ lastError = `${url}: ${err.code || 'fetch failed'}`;
+ }
+ }
+ return { ok: false, error: lastError };
+ }
+
+ /** Check the proof and, when present, create the source in the same transaction that verifies the claim. */
+ async function check(row, actor) {
+ const result = row.method === 'dns' ? await lookupDns(row) : await lookupMeta(row);
+ const checkedAt = now();
+ if (!result.ok) {
+ await q.checked.run(checkedAt, clip(result.error, 500) || 'not verified', checkedAt, row.id);
+ return { row: await q.byId.get(row.id), verified: false, source: null };
+ }
+ const terms = termsNoteFor(row, checkedAt);
+ const sourceInput = sourceFor(row, terms, config);
+ let source = null;
+ try {
+ await db.tx(async () => {
+ const fresh = await db.prepare('SELECT * FROM merchant_sources WHERE id = ? FOR UPDATE').get(row.id);
+ if (!fresh || fresh.status !== 'pending') return;
+ source = await registry.create(sourceInput, fresh.owner_sub);
+ const changed = await q.verified.run(source.key, terms, checkedAt, actor && actor.subject ? actor.subject : fresh.owner_sub,
+ clip(result.detail, 500), checkedAt, checkedAt, row.id);
+ if (!changed || changed.changes === 0) throw new Error('the claim changed while it was being verified');
+ });
+ } catch (err) {
+ await q.checked.run(now(), clip(`verification found the proof, but the source was not created: ${err.message}`, 500), now(), row.id);
+ if (err && typeof err.code === 'string' && err.code.startsWith('sources.')) {
+ throw new MerchantError(422, err.code, `the source was not created: ${err.message}`);
+ }
+ throw err;
+ }
+ const final = await q.byId.get(row.id);
+ return { row: final, verified: Boolean(final && final.status === 'verified'), source };
+ }
+
+ async function verify(principal, id) {
+ const row = await load(principal, id);
+ if (row.status === 'verified') return { row, verified: true, source: null, already: true };
+ if (row.status === 'withdrawn') throw new MerchantError(409, 'merchant.withdrawn', 'this claim was withdrawn; add the domain again to opt in');
+ if (row.status === 'failed') throw new MerchantError(409, 'merchant.failed', 'this claim failed before verification; add the domain again to opt in');
+ return await check(row, principal);
+ }
+
+ async function withdraw(principal, id) {
+ const row = await load(principal, id);
+ if (row.status === 'withdrawn') return row;
+ const t = now();
+ await db.tx(async () => {
+ if (row.source_key) {
+ const source = await registry.get(row.source_key);
+ // The terms note stays as it was: the history of the permission is the point.
+ if (source) await registry.patch(row.source_key, { enabled: false }, principal.subject);
+ }
+ await q.withdraw.run(t, principal.subject, t, row.id);
+ });
+ return await q.byId.get(row.id);
+ }
+
+ async function list(principal) {
+ if (!principal || !ids.isSubjectId('user', principal.subject)) throw new MerchantError(401, 'merchant.auth_required', 'a signed-in merchant is required');
+ return await q.forOwner.all(principal.subject);
+ }
+
+ /** Worker pass: pending claims are checked again; a claim that never verifies fails after merchant.pendingDays. */
+ async function recheck({ limit = 50 } = {}) {
+ const t = now();
+ const summary = { checked: 0, verified: 0, failed: 0 };
+ for (const row of await q.duePending.all(t - config.merchant.recheckIntervalMs + 1000, limit)) {
+ if (t - row.created_at > config.merchant.pendingDays * DAY) {
+ await q.failed.run(`not verified within ${config.merchant.pendingDays} days`, t, row.id);
+ summary.failed++;
+ continue;
+ }
+ try {
+ const out = await check(row, { subject: 'svc:sources' });
+ summary.checked++;
+ if (out.row && out.row.status === 'verified') summary.verified++;
+ } catch (err) {
+ log.error(`[merchant] recheck ${row.id}: ${err.stack || err}`);
+ }
+ }
+ return summary;
+ }
+
+ return {
+ add, list, load, verify, withdraw, recheck, view,
+ get: async (principal, id) => await load(principal, id),
+ getById: async (id) => (ID_RE.test(String(id || '')) ? await q.byId.get(id) || null : null),
+ };
+}
+
+module.exports = {
+ createMerchant, MerchantError, checkDomain, checkUrl, sameSite, sourceKeyFor, termsNoteFor, hasVerificationMeta,
+ isHostname, CHALLENGE_PREFIX, TXT_PREFIX, META_NAME, METHODS, LISTING_KINDS,
+};
new file mode 100644
new file mode 100644
@@ -0,0 +1,220 @@
@@ -0,0 +1,220 @@
+'use strict';
+/**
+ * The merchant opt-in pages: plain semantic markup built with the OpenVibe Kit (openvibe-shared/kit), so the page keeps
+ * the network's classes and the main session styles the pieces it adds. No client-side state: every action is a form.
+ */
+const kit = require('openvibe-shared/kit');
+
+const esc = kit.esc;
+
+const STATUS = {
+ pending: ['pending', 'warn'],
+ verified: ['verified', 'ok'],
+ withdrawn: ['withdrawn', ''],
+ failed: ['failed', 'bad'],
+};
+
+const statusBadge = (status) => {
+ const [label, tone] = STATUS[status] || [status, ''];
+ return kit.badge(label, tone);
+};
+
+const when = (iso) => {
+ if (!iso) return '';
+ const t = kit.when(iso);
+ return t.text ? `<time datetime="${esc(t.iso)}">${esc(t.text)}</time>` : '';
+};
+
+const csrfInput = (csrf) => `<input type="hidden" name="csrf" value="${esc(csrf)}">`;
+
+function layout({ title = 'List your store on OpenVibe', body = '', description = '' }) {
+ return `<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<meta name="robots" content="noindex, nofollow">
+<meta name="color-scheme" content="dark light">
+<title>${esc(title)} · OpenVibe.Sources</title>
+${description ? `<meta name="description" content="${esc(description)}">` : ''}
+<script src="/shared/theme-loader.js"></script>
+<link rel="stylesheet" href="/shared/kit.css">
+<style>html{background:var(--ovk-bg);color:var(--ovk-text)}body{margin:0}</style>
+</head>
+<body>
+<main class="ovk-page">
+${body}
+</main>
+<script src="/shared/kit-web.js" defer></script>
+</body>
+</html>`;
+}
+
+function notice(kind, text) {
+ return text ? kit.notice({ kind, text }) : '';
+}
+
+function guestPage({ error = '', authError = '' } = {}) {
+ const body = `${kit.head({ title: 'List your store on OpenVibe', sub: 'Free, with honest ranking and no pay-to-win.' })}
+${notice('error', error)}
+${authError ? kit.notice({ kind: 'warn', text: `Sign-in did not complete (${authError}). Please try again.` }) : ''}
+${kit.section({
+ title: 'How it works',
+ body: `<ol class="ovk-list">
+<li class="ovk-row ovk-row--static"><span class="ovk-row-main"><span class="ovk-row-top"><strong>1. Claim your domain</strong></span><span class="ovk-row-sub"><span>Sign in, add the domain your shop uses, and name the sitemap or product feed we may read.</span></span></span></li>
+<li class="ovk-row ovk-row--static"><span class="ovk-row-main"><span class="ovk-row-top"><strong>2. Prove you control it</strong></span><span class="ovk-row-sub"><span>Publish a DNS TXT record or a meta tag. Nothing is crawled or enabled until the proof is found.</span></span></span></li>
+<li class="ovk-row ovk-row--static"><span class="ovk-row-main"><span class="ovk-row-top"><strong>3. We crawl politely</strong></span><span class="ovk-row-sub"><span>robots.txt is checked on every request. We read what your pages publish for machines, with provenance.</span></span></span></li>
+<li class="ovk-row ovk-row--static"><span class="ovk-row-main"><span class="ovk-row-top"><strong>4. You stay in control</strong></span><span class="ovk-row-sub"><span>Withdraw at any time: the source is disabled and the permission note is kept for the audit trail.</span></span></span></li>
+</ol>`,
+ })}
+${kit.section({
+ title: 'Sign in to start',
+ body: `<p class="ovk-note">A merchant signs in with an OpenVibe.Network account. The claim belongs to that account.</p>
+<p>${kit.button({ href: '/merchant/login?next=%2Fmerchant', label: 'Sign in with OpenVibe', icon: 'user' })}</p>`,
+ })}`;
+ return layout({ body });
+}
+
+function claimRow(claim) {
+ const source = claim.source;
+ const sub = `${claim.listing.kind === 'sitemap' ? 'Sitemap' : 'Product feed'} · ${claim.listing.url}`;
+ const trail = source ? statusBadge(source.health.status) : '';
+ return kit.row({
+ href: `/merchant/sources/${claim.id}`,
+ title: claim.domain,
+ badge: statusBadge(claim.status),
+ sub,
+ meta: claim.created_at ? kit.when(claim.created_at).text : '',
+ metaIso: claim.created_at || '',
+ trail,
+ });
+}
+
+function merchantPage({ merchant, claims = [], csrf, notice: noticeText = '', error = '', authError = '' }) {
+ const list = claims.length
+ ? kit.list(claims.map(claimRow), { title: 'Your merchant sources' })
+ : kit.empty({ title: 'No store listed yet', text: 'Add your first domain below. It becomes a source only after you prove you control it and the proof is found.', icon: 'globe' });
+
+ const form = `<form class="ovk-form" method="post" action="/merchant/sources">
+${csrfInput(csrf)}
+<label>Store domain <input name="domain" required maxlength="253" autocomplete="off" placeholder="shop.example.com" spellcheck="false"></label>
+<label>Store name (optional) <input name="store_name" maxlength="120" autocomplete="organization" placeholder="Example Shop"></label>
+${kit.seg({ name: 'method', label: 'How will you prove control?', value: 'dns', options: [['dns', 'DNS TXT record'], ['meta', 'Meta tag']] })}
+${kit.selectRow({ name: 'listing_kind', label: 'What should we read?', hint: 'A sitemap of product pages, or an RSS/Atom product feed.', options: [['sitemap', 'Sitemap'], ['feed', 'Product feed (RSS/Atom)']], value: 'sitemap' })}
+<label>Sitemap or product feed URL <input name="listing_url" type="url" required maxlength="2048" autocomplete="url" placeholder="https://shop.example.com/sitemap.xml" spellcheck="false"></label>
+<label>Meta tag page (only for method meta; defaults to your home page) <input name="meta_url" type="url" maxlength="2048" autocomplete="url" placeholder="https://shop.example.com/" spellcheck="false"></label>
+<p class="ovk-note">Nothing is created or enabled before the proof is found. robots.txt is still obeyed after that.</p>
+${kit.button({ label: 'Add domain', icon: 'plus' })}
+</form>`;
+
+ const body = `${kit.head({
+ title: 'List your store on OpenVibe',
+ sub: `Signed in as ${merchant.name || merchant.subject}.`,
+ actions: [kit.button({ href: '/merchant/logout', label: 'Sign out', kind: 'quiet', icon: 'leave', small: true })],
+ })}
+${notice('ok', noticeText)}
+${notice('error', error)}
+${authError ? kit.notice({ kind: 'warn', text: `Sign-in did not complete (${authError}). Please try again.` }) : ''}
+${kit.section({ title: 'Add a store', body: form })}
+${list}
+<p class="ovk-note">A full merchant dashboard with charts and history lives in OpenVibe.Report later. This page is the opt-in flow; the same sources are readable as JSON at <code>/api/v1/merchant/sources</code> with your session or a first-party service token.</p>`;
+ return layout({ body });
+}
+
+function sourcePanel(claim) {
+ const source = claim.source;
+ const terms = claim.terms_note
+ ? `<h3>Terms note</h3><p class="ovk-note">${esc(claim.terms_note)}</p>`
+ : '';
+ if (!source) return `${terms}<p class="ovk-note">No source exists yet. Verification creates it; nothing is fetched before that.</p>`;
+ const health = source.health;
+ const last = claim.last_run;
+ const crawl = claim.last_crawl;
+ const runs = claim.recent_runs && claim.recent_runs.length
+ ? `<h3>Last fetches</h3><ul class="ovk-list">${claim.recent_runs.map(r => `<li class="ovk-row ovk-row--static"><span class="ovk-row-main"><span class="ovk-row-top"><strong>${esc(r.state)}</strong><span class="ovk-row-meta">${esc(r.started_at ? kit.when(r.started_at).text : '')}</span></span><span class="ovk-row-sub"><span>${esc(r.endpoint_url || '')} · ${Number(r.items.seen)} seen, ${Number(r.items.created)} created</span></span></span></li>`).join('')}</ul>`
+ : '<p class="ovk-note">No fetch has run yet.</p>';
+ const crawler = crawl
+ ? `<p>${Number(crawl.pages)} page(s) known, ${Number(crawl.due)} due${crawl.next_due_at ? ` · next ${when(crawl.next_due_at)}` : ''}.</p>
+${Object.keys(crawl.pages_by_state || {}).length ? `<p class="ovk-note">Pages: ${Object.entries(crawl.pages_by_state).map(([k, v]) => `${esc(k)} ${Number(v)}`).join(' · ')}</p>` : ''}
+${crawl.recent && crawl.recent.length ? `<ul class="ovk-list">${crawl.recent.map(p => `<li class="ovk-row ovk-row--static"><span class="ovk-row-main"><span class="ovk-row-top"><strong>${esc(p.url)}</strong><span class="ovk-row-meta">${esc(p.state)}</span></span><span class="ovk-row-sub"><span>${Number(p.items)} item(s) · ${p.http_status == null ? 'not fetched' : `HTTP ${Number(p.http_status)}`}</span></span></span></li>`).join('')}</ul>` : ''}`
+ : '';
+ return `${claim.store_name ? `<p class="ovk-note">${esc(claim.store_name)}</p>` : ''}${terms}
+<dl class="ovk-stats">
+<div><dt>source</dt><dd><code>${esc(source.key)}</code></dd></div>
+<div><dt>type</dt><dd>${esc(source.type)}</dd></div>
+<div><dt>enabled</dt><dd>${source.enabled ? 'yes' : 'no'}</dd></div>
+<div><dt>health</dt><dd>${esc(health.status)}</dd></div>
+</dl>
+${health.last_success_at ? `<p class="ovk-note">Last success ${when(health.last_success_at)}${health.last_state ? ` · last state ${esc(health.last_state)}` : ''}.</p>` : ''}
+${last ? `<p class="ovk-note">Last run ${esc(last.state)} at ${when(last.started_at)}${last.detail ? `: ${esc(last.detail)}` : ''}.</p>` : ''}
+${crawler}
+${runs}`;
+}
+
+function detailPage({ merchant, claim, csrf, notice: noticeText = '', noticeKind = 'ok', error = '' }) {
+ const challenge = claim.verification.challenge;
+ const pending = claim.status === 'pending';
+ const verified = claim.status === 'verified';
+ const withdrawn = claim.status === 'withdrawn';
+ const failed = claim.status === 'failed';
+
+ let proof = '';
+ if (pending) {
+ const how = claim.method === 'dns'
+ ? `<p>Publish this DNS record at your DNS provider:</p>
+${kit.snippet({ code: `${challenge.name} IN TXT "${challenge.value}"` })}
+<dl class="ovk-stats">
+<div><dt>type</dt><dd>TXT</dd></div>
+<div><dt>name</dt><dd><code>${esc(challenge.name)}</code></dd></div>
+<div><dt>value</dt><dd><code>${esc(challenge.value)}</code></dd></div>
+</dl>`
+ : `<p>Put this tag in the <code><head></code> of <a href="${esc(challenge.url)}" rel="noopener">${esc(challenge.url)}</a>:</p>
+${kit.snippet({ code: challenge.html })}
+<p class="ovk-note">The page must stay on ${esc(claim.domain)} (a redirect to another host is refused).</p>`;
+ proof = `${how}
+<div>${kit.action({ action: `/merchant/sources/${claim.id}/verify`, hidden: { csrf }, label: 'Check now', icon: 'retry' })}</div>
+${claim.verification.last_error ? kit.notice({ kind: 'warn', text: claim.verification.last_error }) : ''}`;
+ if (claim.verification.last_checked_at) proof += `<p class="ovk-note">Checked ${Number(claim.verification.check_count || 0)} time(s); last ${when(claim.verification.last_checked_at)}.</p>`;
+ } else if (verified) {
+ proof = kit.notice({ kind: 'ok', text: `Verified ${claim.verification.detail || (claim.method === 'dns' ? 'by DNS TXT record' : 'by meta tag')} at ${claim.verification.verified_at}. The source is enabled.` });
+ proof += `<p class="ovk-note">You can keep the ${claim.method === 'dns' ? 'TXT record' : 'meta tag'} in place, but it is no longer needed to keep the source enabled. Withdrawing is the explicit way to stop it.</p>`;
+ } else if (withdrawn) {
+ proof = kit.notice({ kind: 'info', text: `Permission withdrawn at ${claim.withdrawal && claim.withdrawal.at ? claim.withdrawal.at : claim.updated_at}. The source is disabled and its terms note is kept.` });
+ } else if (failed) {
+ proof = kit.notice({ kind: 'error', text: `This claim failed before verification: ${claim.verification.last_error || 'the proof was not found in time'}. Add the domain again to opt in.` });
+ }
+
+ const actions = [];
+ if (!withdrawn) {
+ actions.push(kit.action({
+ action: `/merchant/sources/${claim.id}/withdraw`,
+ hidden: { csrf },
+ label: 'Withdraw permission',
+ icon: 'ban',
+ kind: 'danger',
+ confirm: 'Withdraw this permission? The source is disabled and the permission note is kept.',
+ }));
+ }
+
+ const body = `${kit.head({
+ title: claim.domain,
+ sub: `${claim.listing.kind === 'sitemap' ? 'Sitemap' : 'Product feed'} · ${claim.listing.url}`,
+ back: { href: '/merchant', label: 'Merchant opt-in' },
+ actions: [statusBadge(claim.status)],
+ })}
+${notice(noticeKind, noticeText)}
+${notice('error', error)}
+${kit.section({ title: pending ? 'Prove you control this domain' : 'Verification', body: proof })}
+${kit.section({ title: 'Source and last crawl', body: sourcePanel(claim) })}
+${actions.length ? kit.section({ title: 'Withdraw', body: actions.join('') }) : ''}`;
+ return layout({ title: `${claim.domain} · Merchant opt-in`, body });
+}
+
+function errorPage({ status = 400, message = 'Something went wrong.' } = {}) {
+ const body = `${kit.head({ title: 'Merchant opt-in' })}
+${kit.notice({ kind: 'error', text: message })}
+<p><a href="/merchant">Back to merchant opt-in</a></p>`;
+ return { status, html: layout({ title: 'Merchant opt-in', body }) };
+}
+
+module.exports = { layout, guestPage, merchantPage, detailPage, errorPage, statusBadge, claimRow };
@@ -30,11 +30,23 @@ function serviceToken(slug, cap, { aud = 'openvibe.sources', exp = Math.floor(Da
@@ -30,11 +30,23 @@ function serviceToken(slug, cap, { aud = 'openvibe.sources', exp = Math.floor(Da
}, key);
}, key);
}
}
+/** A person's Network session token: user id in `sub`, the canonical usr_… id in `subject_id`, no principal/typed claims. */
+function userToken(subject, { username = 'merchant', exp = Math.floor(Date.now() / 1000) + 300, key = privateKey, extra = {} } = {}) {
+ const now = Math.floor(Date.now() / 1000);
+ const header = { alg: 'RS256', typ: 'JWT' };
+ const claims = {
+ iss: ISSUER, sub: String(now % 1000000), subject_id: subject, username, display_name: username,
+ aud: ['openvibe.network'], iat: now, exp, jti: `usr_tok_${crypto.randomBytes(8).toString('hex')}`, ...extra,
+ };
+ const input = `${Buffer.from(JSON.stringify(header)).toString('base64url')}.${Buffer.from(JSON.stringify(claims)).toString('base64url')}`;
+ return `${input}.${crypto.sign('RSA-SHA256', Buffer.from(input), key).toString('base64url')}`;
+}
+
/**
/**
* Boot Sources; the scheduler is off unless worker: 'on'. Loopback is allowlisted for the stubs.
* Boot Sources; the scheduler is off unless worker: 'on'. Loopback is allowlisted for the stubs.
* log: Sources' logger (default silent); limitsNow: the per-actor limiter's clock (default the wall clock).
* log: Sources' logger (default silent); limitsNow: the per-actor limiter's clock (default the wall clock).
*/
*/
-async function boot({ env = {}, worker = 'off', lookupImpl, now, log = silent, limitsNow = null } = {}) {
+async function boot({ env = {}, worker = 'off', lookupImpl, resolver = null, now, log = silent, limitsNow = null } = {}) {
// A stub Network JWKS serving the generated signing key, so the SDK's JWKS client has something real
// A stub Network JWKS serving the generated signing key, so the SDK's JWKS client has something real
// to fetch and verify against (no test touches the internet). Its URL is overridable via env.
// to fetch and verify against (no test touches the internet). Its URL is overridable via env.
const jwksSite = await site({ '/api/.well-known/jwks': () => ({ body: JSON.stringify({ keys: [publicJwk] }) }) });
const jwksSite = await site({ '/api/.well-known/jwks': () => ({ body: JSON.stringify({ keys: [publicJwk] }) }) });
@@ -47,20 +59,28 @@ async function boot({ env = {}, worker = 'off', lookupImpl, now, log = silent, l
@@ -47,20 +59,28 @@ async function boot({ env = {}, worker = 'off', lookupImpl, now, log = silent, l
SOURCES_HOST_MIN_INTERVAL_MS: '0',
SOURCES_HOST_MIN_INTERVAL_MS: '0',
SOURCES_FETCH_TIMEOUT_MS: '1500',
SOURCES_FETCH_TIMEOUT_MS: '1500',
SOURCES_TICK_MS: '50',
SOURCES_TICK_MS: '50',
+ SOURCES_MERCHANT_RECHECK_MS: '0',
...env,
...env,
});
});
// One database per boot (PGlite, or SOURCES_TEST_STORE=pg: the containers), dropped when the boot stops.
// One database per boot (PGlite, or SOURCES_TEST_STORE=pg: the containers), dropped when the boot stops.
const testdb = await require('./db').testDb();
const testdb = await require('./db').testDb();
- const h = await start({ config, db: testdb.db, log, lookupImpl, limitsNow, ...(now ? { now } : {}) });
+ const h = await start({ config, db: testdb.db, log, lookupImpl, merchantResolver: resolver, limitsNow, ...(now ? { now } : {}) });
const base = `http://127.0.0.1:${h.server.address().port}`;
const base = `http://127.0.0.1:${h.server.address().port}`;
return { ...h, base, jwksSite, async stop() { await h.close(); await testdb.close(); await jwksSite.close(); } };
return { ...h, base, jwksSite, async stop() { await h.close(); await testdb.close(); await jwksSite.close(); } };
}
}
-async function request(base, method, p, { token, body, headers = {} } = {}) {
+async function request(base, method, p, { token, body, form, headers = {}, redirect = 'follow' } = {}) {
const h = { ...headers };
const h = { ...headers };
if (token) h.Authorization = `Bearer ${token}`;
if (token) h.Authorization = `Bearer ${token}`;
- if (body !== undefined) h['Content-Type'] = 'application/json';
- const res = await fetch(base + p, { method, headers: h, body: body !== undefined ? JSON.stringify(body) : undefined });
+ let payload;
+ if (form !== undefined) {
+ h['Content-Type'] = 'application/x-www-form-urlencoded';
+ payload = new URLSearchParams(form).toString();
+ } else if (body !== undefined) {
+ h['Content-Type'] = 'application/json';
+ payload = JSON.stringify(body);
+ }
+ const res = await fetch(base + p, { method, headers: h, body: payload, redirect });
const text = await res.text();
const text = await res.text();
let json = null;
let json = null;
try { json = text ? JSON.parse(text) : null; } catch { json = null; }
try { json = text ? JSON.parse(text) : null; } catch { json = null; }
@@ -135,4 +155,4 @@ function suite(name) {
@@ -135,4 +155,4 @@ function suite(name) {
return t;
return t;
}
}
-module.exports = { ISSUER, privateKey, publicKey, silent, serviceToken, boot, request, site, sourceDef, rss, fixture, sleep, suite };
+module.exports = { ISSUER, privateKey, publicKey, silent, serviceToken, userToken, boot, request, site, sourceDef, rss, fixture, sleep, suite };
new file mode 100644
new file mode 100644
@@ -0,0 +1,246 @@
@@ -0,0 +1,246 @@
+'use strict';
+/**
+ * Merchant opt-in (plan §4.4 channel 1): a signed-in merchant proves a domain with DNS or a meta tag, a source is
+ * created only then with an explicit-permission terms note, the crawler still obeys robots.txt, and withdrawal
+ * disables the source while keeping the note. Also covers the JSON read Report calls, including last crawl stats.
+ */
+const assert = require('assert');
+const { ids } = require('openvibe-contracts');
+const { boot, request, serviceToken, userToken, site, fixture, suite } = require('./helpers');
+
+const t = suite('merchant-optin');
+let svc;
+let shopWeb;
+let metaWeb;
+let blockedWeb;
+let dns;
+let shopClaim;
+let shopSourceKey;
+
+const ALICE = ids.newId('user');
+const BOB = ids.newId('user');
+const ALICE_TOKEN = userToken(ALICE, { username: 'alice' });
+const BOB_TOKEN = userToken(BOB, { username: 'bob' });
+const READER = serviceToken('report', ['sources.source.read']);
+const ADMIN = serviceToken('network', ['sources.source.manage']);
+
+const api = (method, p, opts = {}) => request(svc.base, method, p, opts);
+
+t('boot: a stub shop, meta site and a robots-closed site, with an injectable TXT resolver', async () => {
+ dns = {
+ records: new Map(),
+ async resolveTxt(name) {
+ if (!this.records.has(name)) { const err = new Error('no record'); err.code = 'ENOTFOUND'; throw err; }
+ return this.records.get(name);
+ },
+ set(domain, value) { this.records.set(`_openvibe-sources.${domain}`, [[value]]); },
+ };
+ // Every stub name resolves to the loopback server; SOURCES_ALLOW_PRIVATE_HOSTS exempts the names below.
+ const lookupImpl = (_host, _opts, cb) => cb(null, [{ address: '127.0.0.1', family: 4 }]);
+ shopWeb = await site({});
+ metaWeb = await site({});
+ blockedWeb = await site({});
+ const shopBase = `http://shop.example.com:${new URL(shopWeb.origin).port}`;
+ const metaBase = `http://meta.example.com:${new URL(metaWeb.origin).port}`;
+ const blockedBase = `http://blocked.example.com:${new URL(blockedWeb.origin).port}`;
+
+ shopWeb.routes['/robots.txt'] = () => ({ body: 'User-agent: *\nAllow: /\n', headers: { 'content-type': 'text/plain' } });
+ shopWeb.routes['/sitemap.xml'] = () => ({
+ body: `<?xml version="1.0"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"><url><loc>${shopBase}/products/1</loc></url></urlset>`,
+ headers: { 'content-type': 'application/xml' },
+ });
+ shopWeb.routes['/products/1'] = () => ({ body: fixture('product.html'), headers: { 'content-type': 'text/html' } });
+
+ let metaToken = '';
+ metaWeb.routes['/robots.txt'] = () => ({ body: 'User-agent: *\nAllow: /\n', headers: { 'content-type': 'text/plain' } });
+ metaWeb.routes['/'] = () => ({
+ body: `<!doctype html><html><head><meta name="openvibe-sources-verification" content="${metaToken}"><title>Meta Shop</title></head><body>meta</body></html>`,
+ headers: { 'content-type': 'text/html' },
+ });
+ metaWeb.routes['/feed.xml'] = () => ({
+ body: '<?xml version="1.0"?><rss version="2.0"><channel><title>Meta Shop</title><item><guid>p1</guid><title>Product one</title><link>https://meta.example.com/p/1</link><description>One.</description></item></channel></rss>',
+ headers: { 'content-type': 'application/rss+xml' },
+ });
+
+ blockedWeb.routes['/robots.txt'] = () => ({ body: 'User-agent: *\nDisallow: /\n', headers: { 'content-type': 'text/plain' } });
+ blockedWeb.routes['/sitemap.xml'] = () => ({ body: '<?xml version="1.0"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"></urlset>', headers: { 'content-type': 'application/xml' } });
+
+ svc = await boot({
+ resolver: dns,
+ lookupImpl,
+ env: {
+ BASE_URL: 'https://sources.test',
+ SOURCES_ALLOW_PRIVATE_HOSTS: '127.0.0.1,shop.example.com,meta.example.com,blocked.example.com',
+ },
+ });
+ svc.metaBase = metaBase;
+ svc.shopBase = shopBase;
+ svc.blockedBase = blockedBase;
+ svc.setMetaToken = (v) => { metaToken = v; };
+});
+
+t('a pending claim creates no source; verification creates one with who/when/how in its terms note', async () => {
+ const c = await api('POST', '/api/v1/merchant/sources', { token: ALICE_TOKEN, body: {
+ domain: 'shop.example.com', store_name: 'Example Shop', method: 'dns', listing_kind: 'sitemap',
+ listing_url: `${svc.shopBase}/sitemap.xml`,
+ } });
+ assert.strictEqual(c.status, 201, c.text);
+ shopClaim = c.body.merchant_source;
+ assert.deepStrictEqual([shopClaim.status, shopClaim.source_key, shopClaim.method], ['pending', null, 'dns']);
+ assert.strictEqual(shopClaim.verification.challenge.name, '_openvibe-sources.shop.example.com');
+ assert.match(shopClaim.verification.challenge.value, /^openvibe-sources-verification=[a-f0-9]{40}$/);
+ assert.strictEqual((await svc.db.prepare("SELECT COUNT(*) AS n FROM sources WHERE key LIKE 'merchant-%'").get()).n, 0, 'no source before verification');
+
+ const before = await api('POST', `/api/v1/merchant/sources/${shopClaim.id}/verify`, { token: ALICE_TOKEN });
+ assert.strictEqual(before.status, 200);
+ assert.deepStrictEqual([before.body.verified, before.body.merchant_source.status, before.body.merchant_source.source_key], [false, 'pending', null]);
+ assert.match(before.body.merchant_source.verification.last_error, /no TXT record/);
+
+ dns.set('shop.example.com', shopClaim.verification.challenge.value);
+ const after = await api('POST', `/api/v1/merchant/sources/${shopClaim.id}/verify`, { token: ALICE_TOKEN });
+ assert.strictEqual(after.status, 200, after.text);
+ assert.deepStrictEqual([after.body.verified, after.body.merchant_source.status, after.body.source_created], [true, 'verified', true]);
+ shopSourceKey = after.body.merchant_source.source_key;
+ assert.match(shopSourceKey, /^merchant-shop-example-com-[a-z0-9]{8}$/);
+ const terms = after.body.merchant_source.terms_note;
+ assert.ok(terms.length <= 2000);
+ for (const bit of [ALICE, 'alice', 'explicit permission', 'shop.example.com', 'sitemap', 'DNS TXT record', 'robots.txt', 'withdraw']) {
+ assert.ok(terms.includes(bit), `terms note should mention ${bit}`);
+ }
+
+ const source = await api('GET', `/api/v1/sources/${shopSourceKey}`, { token: READER });
+ assert.strictEqual(source.status, 200);
+ assert.deepStrictEqual([source.body.source.enabled, source.body.source.type, source.body.source.category], [true, 'crawl', 'deals']);
+ assert.strictEqual(source.body.source.endpoints[0].url, `${svc.shopBase}/sitemap.xml`);
+ assert.strictEqual(source.body.source.terms_note, terms);
+ assert.match(source.body.source.robots_note, /robots\.txt/);
+ assert.strictEqual(source.body.source.crawl.sitemaps, false, 'the merchant named this sitemap; no guessed ones');
+});
+
+t('the crawl reads the named sitemap and the merchant read reports its last run and frontier', async () => {
+ const out = await svc.ingest.run(shopSourceKey, { trigger: 'manual' });
+ assert.deepStrictEqual(out.runs.map(r => r.state), ['ok', 'ok']);
+ assert.strictEqual(out.runs[1].items_created, 1);
+
+ const read = await api('GET', '/api/v1/merchant/sources', { token: ALICE_TOKEN });
+ assert.strictEqual(read.status, 200);
+ const listed = read.body.sources.find(s => s.id === shopClaim.id);
+ assert.ok(listed);
+ assert.deepStrictEqual([listed.source.enabled, listed.source.health.status], [true, 'healthy']);
+ assert.strictEqual(listed.last_run.state, 'ok');
+ assert.strictEqual(listed.last_run.items.created, 1);
+ assert.strictEqual(listed.last_crawl.pages, 1);
+ assert.strictEqual(listed.last_crawl.pages_by_state.kept, 1);
+ assert.ok(listed.last_crawl.recent.some(p => p.url.endsWith('/products/1') && p.state === 'kept'));
+
+ // Report reads the same by naming the merchant it acts for.
+ const report = await api('GET', '/api/v1/merchant/sources', { token: READER, headers: { 'X-OV-Subject': ALICE } });
+ assert.strictEqual(report.status, 200);
+ assert.strictEqual(report.body.sources[0].last_crawl.pages, 1);
+ assert.strictEqual((await api('GET', '/api/v1/merchant/sources', { token: READER })).status, 400);
+ // Another merchant sees nothing of this claim, even by id.
+ assert.strictEqual((await api('GET', `/api/v1/merchant/sources/${shopClaim.id}`, { token: BOB_TOKEN })).status, 404);
+ assert.strictEqual((await api('GET', '/api/v1/merchant/sources', { token: BOB_TOKEN })).body.sources.length, 0);
+});
+
+t('withdrawal disables the source and keeps the permission note', async () => {
+ const before = await api('GET', `/api/v1/merchant/sources/${shopClaim.id}`, { token: ALICE_TOKEN });
+ const w = await api('POST', `/api/v1/merchant/sources/${shopClaim.id}/withdraw`, { token: ALICE_TOKEN });
+ assert.strictEqual(w.status, 200, w.text);
+ assert.strictEqual(w.body.merchant_source.status, 'withdrawn');
+ assert.strictEqual(w.body.merchant_source.terms_note, before.body.merchant_source.terms_note, 'the note is kept verbatim');
+ assert.strictEqual(w.body.merchant_source.source.enabled, false);
+ assert.strictEqual(w.body.merchant_source.source.health.status, 'disabled');
+ assert.ok(w.body.merchant_source.withdrawal.at);
+
+ const source = await api('GET', `/api/v1/sources/${shopSourceKey}`, { token: READER });
+ assert.deepStrictEqual([source.body.source.enabled, source.body.source.terms_note], [false, before.body.merchant_source.terms_note]);
+ assert.strictEqual((await api('POST', `/api/v1/merchant/sources/${shopClaim.id}/verify`, { token: ALICE_TOKEN })).status, 409);
+ // A withdrawn claim frees the domain: the merchant may opt in again (a new claim, a new source).
+ assert.strictEqual((await api('POST', '/api/v1/merchant/sources', { token: ALICE_TOKEN, body: {
+ domain: 'shop.example.com', method: 'dns', listing_kind: 'feed', listing_url: `${svc.shopBase}/feed.xml`,
+ } })).status, 201);
+});
+
+t('a meta tag proves control and a product feed becomes an enabled rss source', async () => {
+ const c = await api('POST', '/api/v1/merchant/sources', { token: ALICE_TOKEN, body: {
+ domain: 'meta.example.com', method: 'meta', meta_url: `${svc.metaBase}/`, listing_kind: 'feed',
+ listing_url: `${svc.metaBase}/feed.xml`,
+ } });
+ assert.strictEqual(c.status, 201, c.text);
+ const claim = c.body.merchant_source;
+ assert.strictEqual(claim.verification.challenge.name, 'openvibe-sources-verification');
+ const before = await api('POST', `/api/v1/merchant/sources/${claim.id}/verify`, { token: ALICE_TOKEN });
+ assert.strictEqual(before.body.verified, false);
+ assert.match(before.body.merchant_source.verification.last_error, /no openvibe-sources-verification meta tag/);
+
+ svc.setMetaToken(claim.verification.challenge.value);
+ const after = await api('POST', `/api/v1/merchant/sources/${claim.id}/verify`, { token: ALICE_TOKEN });
+ assert.strictEqual(after.status, 200, after.text);
+ assert.deepStrictEqual([after.body.verified, after.body.merchant_source.status], [true, 'verified']);
+ assert.match(after.body.merchant_source.terms_note, /meta tag at/);
+ const source = await api('GET', `/api/v1/sources/${after.body.merchant_source.source_key}`, { token: READER });
+ assert.deepStrictEqual([source.body.source.enabled, source.body.source.type], [true, 'rss']);
+ const run = await svc.ingest.run(after.body.merchant_source.source_key, { trigger: 'manual' });
+ assert.deepStrictEqual(run.runs.map(r => [r.state, r.items_created]), [['ok', 1]]);
+});
+
+t('a merchant opt-in still obeys robots.txt: a disallowed site is never read', async () => {
+ const c = await api('POST', '/api/v1/merchant/sources', { token: ALICE_TOKEN, body: {
+ domain: 'blocked.example.com', method: 'dns', listing_kind: 'sitemap', listing_url: `${svc.blockedBase}/sitemap.xml`,
+ } });
+ const claim = c.body.merchant_source;
+ dns.set('blocked.example.com', claim.verification.challenge.value);
+ const v = await api('POST', `/api/v1/merchant/sources/${claim.id}/verify`, { token: ALICE_TOKEN });
+ assert.strictEqual(v.body.merchant_source.status, 'verified');
+ const key = v.body.merchant_source.source_key;
+ const run = await svc.ingest.run(key, { trigger: 'manual' });
+ assert.deepStrictEqual(run.runs.map(r => [r.state, r.items_seen, r.items_created]), [['robots_denied', 0, 0]]);
+ assert.strictEqual((await svc.db.prepare('SELECT COUNT(*) AS n FROM items WHERE source_key = ?').get(key)).n, 0);
+ const read = await api('GET', `/api/v1/merchant/sources/${claim.id}`, { token: ALICE_TOKEN });
+ assert.strictEqual(read.body.merchant_source.last_run.state, 'robots_denied');
+ assert.strictEqual(read.body.merchant_source.last_crawl.pages, 0);
+});
+
+t('domains, listing URLs and duplicate claims are validated', async () => {
+ const bad = async (body, re) => {
+ const r = await api('POST', '/api/v1/merchant/sources', { token: ALICE_TOKEN, body });
+ assert.strictEqual(r.status, 422, JSON.stringify(body) + ' -> ' + r.status + ' ' + r.text);
+ assert.match(r.body.detail, re);
+ };
+ await bad({ domain: 'localhost', method: 'dns', listing_kind: 'sitemap', listing_url: 'https://localhost/sitemap.xml' }, /fully qualified/);
+ await bad({ domain: 'shop.example.com', method: 'dns', listing_kind: 'sitemap', listing_url: 'https://evil.example.net/sitemap.xml' }, /on shop\.example\.com/);
+ await bad({ domain: 'shop.example.com', method: 'dns', listing_kind: 'feed', listing_url: 'https://shop.example.com/feed.xml?api_key=secret' }, /credential/);
+ await bad({ domain: 'shop.example.com', method: 'fax', listing_kind: 'sitemap', listing_url: 'https://shop.example.com/sitemap.xml' }, /method/);
+ await bad({ domain: 'shop.example.com', method: 'dns', listing_kind: 'csv', listing_url: 'https://shop.example.com/sitemap.xml' }, /listing_kind/);
+ // The second claim from the previous test is pending on shop.example.com.
+ const dup = await api('POST', '/api/v1/merchant/sources', { token: BOB_TOKEN, body: {
+ domain: 'shop.example.com', method: 'dns', listing_kind: 'sitemap', listing_url: 'https://shop.example.com/sitemap.xml',
+ } });
+ assert.strictEqual(dup.status, 409);
+ assert.strictEqual(dup.body.code, 'merchant.domain_taken');
+
+ // `/claims` is the same resource; a first-party service may act for the merchant with X-OV-Subject, and needs manage.
+ assert.strictEqual((await api('GET', '/api/v1/merchant/claims', { token: ALICE_TOKEN })).status, 200);
+ assert.strictEqual((await api('POST', '/api/v1/merchant/claims', {
+ token: READER, headers: { 'X-OV-Subject': ALICE }, body: {
+ domain: 'report.example.com', method: 'dns', listing_kind: 'feed', listing_url: 'https://report.example.com/feed.xml',
+ },
+ })).status, 403);
+ const delegated = await api('POST', '/api/v1/merchant/claims', {
+ token: ADMIN, headers: { 'X-OV-Subject': ALICE }, body: {
+ domain: 'report.example.com', method: 'dns', listing_kind: 'feed', listing_url: 'https://report.example.com/feed.xml',
+ },
+ });
+ assert.strictEqual(delegated.status, 201, delegated.text);
+ assert.strictEqual(delegated.body.merchant_source.owner.subject, ALICE);
+});
+
+t('done', async () => {
+ await shopWeb.close();
+ await metaWeb.close();
+ await blockedWeb.close();
+ await svc.stop();
+});
+
+t.run();
new file mode 100644
new file mode 100644
@@ -0,0 +1,140 @@
@@ -0,0 +1,140 @@
+'use strict';
+/** The merchant opt-in pages: a guest sees how it works, a signed-in merchant adds/proves/withdraws with plain forms. */
+const assert = require('assert');
+const { ids } = require('openvibe-contracts');
+const { boot, request, userToken, suite } = require('./helpers');
+
+const t = suite('merchant-pages');
+let svc;
+const ALICE = ids.newId('user');
+const BOB = ids.newId('user');
+const ALICE_TOKEN = userToken(ALICE, { username: 'alice' });
+const BOB_TOKEN = userToken(BOB, { username: 'bob' });
+let claimId;
+let sourceKey;
+
+const cookies = (token) => ({ cookie: `ov_token=${token}` });
+const page = (method, p, token, opts = {}) => request(svc.base, method, p, { ...opts, headers: { ...(token ? cookies(token) : {}), ...(opts.headers || {}) } });
+const csrfOf = (html) => {
+ const m = String(html).match(/name="csrf" value="([^"]+)"/);
+ assert.ok(m, 'the page carries a CSRF token');
+ return m[1];
+};
+
+t('boot with an injectable DNS resolver', async () => {
+ const records = new Map();
+ svc = await boot({
+ resolver: {
+ async resolveTxt(name) {
+ if (!records.has(name)) { const err = new Error('none'); err.code = 'ENOTFOUND'; throw err; }
+ return records.get(name);
+ },
+ },
+ env: { BASE_URL: 'https://sources.test' },
+ });
+ svc.dns = records;
+});
+
+t('a guest sees the pitch and a sign-in link, not the form', async () => {
+ const r = await page('GET', '/merchant');
+ assert.strictEqual(r.status, 200);
+ assert.match(r.text, /List your store on OpenVibe/);
+ assert.match(r.text, /Sign in with OpenVibe/);
+ assert.match(r.text, /DNS TXT record or a meta tag/);
+ assert.doesNotMatch(r.text, /name="listing_url"/);
+ assert.strictEqual((await page('POST', '/merchant/sources', null, { redirect: 'manual', form: {} })).status, 303);
+});
+
+t('a signed-in merchant adds a domain and sees the DNS instructions', async () => {
+ const list = await page('GET', '/merchant', ALICE_TOKEN);
+ assert.strictEqual(list.status, 200);
+ assert.match(list.text, /Signed in as alice/);
+ assert.match(list.text, /Add a store/);
+ const csrf = csrfOf(list.text);
+ const created = await page('POST', '/merchant/sources', ALICE_TOKEN, {
+ redirect: 'manual',
+ form: {
+ csrf, domain: 'shop.example.com', store_name: 'Example Shop', method: 'dns', listing_kind: 'sitemap',
+ listing_url: 'https://shop.example.com/sitemap.xml', meta_url: '',
+ },
+ });
+ assert.strictEqual(created.status, 303, created.text);
+ const location = created.headers.get('location');
+ assert.match(location, /^\/merchant\/sources\/msrc_[0-9A-HJKMNP-TV-Z]{26}\?created=1$/);
+ claimId = location.match(/\/merchant\/sources\/(msrc_[0-9A-HJKMNP-TV-Z]{26})/)[1];
+
+ const detail = await page('GET', `/merchant/sources/${claimId}`, ALICE_TOKEN);
+ assert.strictEqual(detail.status, 200);
+ assert.match(detail.text, /_openvibe-sources\.shop\.example\.com/);
+ assert.match(detail.text, /openvibe-sources-verification=[a-f0-9]{40}/);
+ assert.match(detail.text, /Check now/);
+ assert.match(detail.text, /No source exists yet/);
+});
+
+t('check now says pending; after the TXT record is published it creates and shows the source', async () => {
+ const detail = await page('GET', `/merchant/sources/${claimId}`, ALICE_TOKEN);
+ const csrf = csrfOf(detail.text);
+ const pending = await page('POST', `/merchant/sources/${claimId}/verify`, ALICE_TOKEN, {
+ redirect: 'manual', form: { csrf },
+ });
+ assert.strictEqual(pending.status, 303);
+ assert.match(pending.headers.get('location'), /\?pending=1$/);
+ const still = await page('GET', `/merchant/sources/${claimId}?pending=1`, ALICE_TOKEN);
+ assert.match(still.text, /proof was not found yet|no TXT record/i);
+
+ const tokenValue = still.text.match(/openvibe-sources-verification=([a-f0-9]{40})/)[1];
+ svc.dns.set('_openvibe-sources.shop.example.com', [[`openvibe-sources-verification=${tokenValue}`]]);
+ const ok = await page('POST', `/merchant/sources/${claimId}/verify`, ALICE_TOKEN, {
+ redirect: 'manual', form: { csrf: csrfOf(still.text) },
+ });
+ assert.strictEqual(ok.status, 303);
+ assert.match(ok.headers.get('location'), /\?verified=1$/);
+
+ const done = await page('GET', `/merchant/sources/${claimId}?verified=1`, ALICE_TOKEN);
+ assert.match(done.text, /The source is enabled/);
+ assert.match(done.text, /explicit permission/);
+ assert.match(done.text, /robots\.txt/);
+ assert.match(done.text, /Example Shop/);
+ sourceKey = done.text.match(/<code>(merchant-shop-example-com-[a-z0-9]{8})<\/code>/)[1];
+ assert.ok(sourceKey);
+});
+
+t('another merchant cannot open the claim; a missing CSRF token is refused', async () => {
+ assert.strictEqual((await page('GET', `/merchant/sources/${claimId}`, BOB_TOKEN)).status, 404);
+ const detail = await page('GET', `/merchant/sources/${claimId}`, ALICE_TOKEN);
+ const forged = await page('POST', `/merchant/sources/${claimId}/verify`, ALICE_TOKEN, { form: { csrf: 'not-the-token' } });
+ assert.strictEqual(forged.status, 403);
+ assert.match(forged.text, /form expired/);
+ assert.ok(detail.text.includes('Withdraw permission'));
+});
+
+t('withdrawal disables the source and the page says the note is kept', async () => {
+ const detail = await page('GET', `/merchant/sources/${claimId}`, ALICE_TOKEN);
+ const withdrawn = await page('POST', `/merchant/sources/${claimId}/withdraw`, ALICE_TOKEN, {
+ redirect: 'manual', form: { csrf: csrfOf(detail.text) },
+ });
+ assert.strictEqual(withdrawn.status, 303);
+ assert.match(withdrawn.headers.get('location'), /\?withdrawn=1$/);
+ const after = await page('GET', `/merchant/sources/${claimId}?withdrawn=1`, ALICE_TOKEN);
+ assert.match(after.text, /Permission withdrawn/);
+ assert.match(after.text, /terms note is kept/);
+ assert.match(after.text, /explicit permission/, 'the permission note is still visible');
+ assert.match(after.text, /<dd>no<\/dd>/, 'the source is disabled');
+ assert.doesNotMatch(after.text, /Withdraw permission/);
+});
+
+t('sign-in redirects to the Network OAuth client with the merchant callback', async () => {
+ const r = await request(svc.base, 'GET', '/merchant/login?next=%2Fmerchant', { redirect: 'manual' });
+ assert.strictEqual(r.status, 302);
+ const location = new URL(r.headers.get('location'));
+ assert.strictEqual(location.origin, 'https://openvibe.network');
+ assert.strictEqual(location.pathname, '/oauth/authorize');
+ assert.strictEqual(location.searchParams.get('client_id'), 'sources');
+ assert.strictEqual(location.searchParams.get('redirect_uri'), 'https://sources.test/merchant/callback');
+ assert.ok(location.searchParams.get('state'));
+ assert.match(r.headers.get('set-cookie') || '', /ov_merchant_state=/);
+});
+
+t('done', async () => { await svc.stop(); });
+
+t.run();
@@ -63,7 +63,8 @@ t('every route, as every kind of caller, with real and nonsense ids: the client
@@ -63,7 +63,8 @@ t('every route, as every kind of caller, with real and nonsense ids: the client
for (const id of ['keyed', '1', 'nope', "'\"<x>", 'x'.repeat(300)]) {
for (const id of ['keyed', '1', 'nope', "'\"<x>", 'x'.repeat(300)]) {
const p = fill(route.path, id);
const p = fill(route.path, id);
for (const [who, token] of Object.entries(callers)) {
for (const [who, token] of Object.entries(callers)) {
- const r = await request(svc.base, method.toUpperCase(), `${p}${method === 'get' ? '?q=x&limit=-1' : ''}`, { token, ...(method === 'get' ? {} : { headers: { 'content-type': 'application/json' } }) });
+ // manual redirects: /merchant/login points at OpenVibe.Network, and this test must not call the internet.
+ const r = await request(svc.base, method.toUpperCase(), `${p}${method === 'get' ? '?q=x&limit=-1' : ''}`, { token, redirect: 'manual', ...(method === 'get' ? {} : { headers: { 'content-type': 'application/json' } }) });
const text = r.text + JSON.stringify([...r.headers.entries()]);
const text = r.text + JSON.stringify([...r.headers.entries()]);
if (text.includes(SECRET)) found.push(`${who}: ${method} ${p} → ${r.status} carries the client secret`);
if (text.includes(SECRET)) found.push(`${who}: ${method} ${p} → ${r.status} carries the client secret`);
if (text.includes(FEED_KEY) && who !== 'source reader') found.push(`${who}: ${method} ${p} → ${r.status} carries the feed key`);
if (text.includes(FEED_KEY) && who !== 'source reader') found.push(`${who}: ${method} ${p} → ${r.status} carries the feed key`);
@@ -151,6 +151,7 @@ t('a source registered over the API with internal endpoints is stored, and its f
@@ -151,6 +151,7 @@ t('a source registered over the API with internal endpoints is stored, and its f
t('ratchet: every file that makes an outbound request itself is reviewed', () => {
t('ratchet: every file that makes an outbound request itself is reviewed', () => {
const REVIEWED = {
const REVIEWED = {
'server/net/fetcher.js': 'every source fetch; the guard (checkUrl up front, lookup at connect, on every redirect hop)',
'server/net/fetcher.js': 'every source fetch; the guard (checkUrl up front, lookup at connect, on every redirect hop)',
+ 'server/merchant-auth.js': 'OAuth code exchange only, to the operator-configured Network URLs (never a merchant-supplied URL)',
};
};
const root = path.join(__dirname, '..');
const root = path.join(__dirname, '..');
const found = [];
const found = [];