openvibers/openvibe.sources Commits
Source access is computed from evidence (ADR 0007): probe.mjs writes evidence/<domain>.json for 29 shops; 12 ok get enabled crawl adapters, 6 structured-only get disabled ones (Lane 1), 11 queued go to compliance/review-queue.yaml; the crawler identifies as OpenVibeBot/1.0 (compatible form) and obeys robots groups, meta and X-Robots-Tag for OpenVibeBot or OpenVibeSources; ADR 0006 records the crawl type (#131)
Changes
@@@ -48,8 +48,8 @@ VALKEY_PREFIX=#SOURCES_MAX_BACKOFF_MS=21600000# Fetching--#SOURCES_USER_AGENT=OpenVibeSources/0.1.0 (+https://sources.openvibe.network; [email protected])--#SOURCES_ROBOTS_AGENT=OpenVibeSources++#SOURCES_USER_AGENT=Mozilla/5.0 (compatible; OpenVibeBot/1.0; +https://openvibe.network/bot)++#SOURCES_ROBOTS_AGENT=OpenVibeBot,OpenVibeSources#SOURCES_FETCH_TIMEOUT_MS=20000#SOURCES_MAX_BYTES=5242880#SOURCES_MAX_REDIRECTS=5nnew file mode 100644@@@ -0,0 +1,8 @@++# OpenVibe.Sources: notes for agents++++Read README.md first. Source access:++++Access verdicts come from evidence/<domain>.json, written by scripts/probe.mjs.++Probe first; if evidence exists, read it and do not re-derive it.++docs/adr/0007-source-access.md is Accepted. Re-litigating it is a code comment.++The fixed technique list in ADR 0007 is not up for discussion.@@@ -318,6 +318,53 @@ downloaded once by hand and parsed offline with these adapters (NASA 10 items, NSteam 20, DealNews 50, SEC 200); that check is not part of the test suite and proves nothing aboutthe terms.++## Source access (ADR 0007)++++Whether a site may be read, and how, is computed from evidence, never judged:++[docs/adr/0007-source-access.md](docs/adr/0007-source-access.md). `node scripts/probe.mjs <domain>` (run from the++production host, whose egress is the crawler's) writes `evidence/<domain>.json`: robots.txt, sitemap.xml,++/products.json, llms.txt, /api/ and the home page, with statuses and challenge markers, and a verdict.++`node scripts/adapters-from-evidence.mjs` turns evidence into adapters (`ok` → an enabled crawl source,++`structured-only` → a disabled one, Lane 1) or a row in [compliance/review-queue.yaml](compliance/review-queue.yaml)++(`queued`); `node scripts/register-adapters.js` brings the registry to the adapters. The crawler identifies itself as++`Mozilla/5.0 (compatible; OpenVibeBot/1.0; +https://openvibe.network/bot)` and obeys robots.txt groups, robots meta++and X-Robots-Tag for either token, `OpenVibeBot` or `OpenVibeSources`. The crawl type itself:++[docs/adr/0006-crawl-source-type.md](docs/adr/0006-crawl-source-type.md).++++Probed 2026-10-28 (`c` = challenge page):++++| Domain | Verdict | Statuses | Lane | Adapter |++|---|---|---|---|---|++| adorama.com | structured-only | `/robots.txt:200 /sitemap.xml:404c /products.json:403c /llms.txt:403c /api/:403c /:403c` | Lane 1 (feeds, affiliate, API, opt-in); adapter disabled | [adapters/adorama.com.json](adapters/adorama.com.json) |++| allbirds.com | ok | `/robots.txt:200 /sitemap.xml:200 /products.json:200 /llms.txt:200 /api/:404 /:200` | crawl adapter, enabled | [adapters/allbirds.com.json](adapters/allbirds.com.json) |++| anker.com | ok | `/robots.txt:200 /sitemap.xml:200 /products.json:404 /llms.txt:200 /api/:404 /:200` | crawl adapter, enabled | [adapters/anker.com.json](adapters/anker.com.json) |++| backcountry.com | queued | `/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403` | review queue; probed again after 24 h | [compliance/review-queue.yaml](compliance/review-queue.yaml) |++| bestbuy.com | ok | `/robots.txt:200 /sitemap.xml:200 /products.json:404 /llms.txt:404 /api/:404 /:200` | crawl adapter, enabled | [adapters/bestbuy.com.json](adapters/bestbuy.com.json) |++| bhphotovideo.com | structured-only | `/robots.txt:200 /sitemap.xml:403c /products.json:403c /llms.txt:403c /api/:403c /:403c` | Lane 1 (feeds, affiliate, API, opt-in); adapter disabled | [adapters/bhphotovideo.com.json](adapters/bhphotovideo.com.json) |++| brooksrunning.com | queued | `/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403` | review queue; probed again after 24 h | [compliance/review-queue.yaml](compliance/review-queue.yaml) |++| converse.com | queued | `/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403` | review queue; probed again after 24 h | [compliance/review-queue.yaml](compliance/review-queue.yaml) |++| dsw.com | ok | `/robots.txt:200 /sitemap.xml:404 /products.json:404 /llms.txt:404 /api/:200 /:200` | crawl adapter, enabled | [adapters/dsw.com.json](adapters/dsw.com.json) |++| footlocker.com | ok | `/robots.txt:200 /sitemap.xml:200 /products.json:200 /llms.txt:200 /api/:403 /:200` | crawl adapter, enabled | [adapters/footlocker.com.json](adapters/footlocker.com.json) |++| frame.work | queued | `/robots.txt:403c /sitemap.xml:403c /products.json:403c /llms.txt:403c /api/:403c /:403c` | review queue; probed again after 24 h | [compliance/review-queue.yaml](compliance/review-queue.yaml) |++| gog.com | ok | `/robots.txt:200 /sitemap.xml:404 /products.json:404 /llms.txt:404 /api/:404 /:200` | crawl adapter, enabled | [adapters/gog.com.json](adapters/gog.com.json) |++| hoka.com | structured-only | `/robots.txt:200 /sitemap.xml:404 /products.json:410 /llms.txt:410 /api/:500 /:0` | Lane 1 (feeds, affiliate, API, opt-in); adapter disabled | [adapters/hoka.com.json](adapters/hoka.com.json) |++| homedepot.com | ok | `/robots.txt:200 /sitemap.xml:404 /products.json:404 /llms.txt:404 /api/:404 /:200` | crawl adapter, enabled | [adapters/homedepot.com.json](adapters/homedepot.com.json) |++| humblebundle.com | ok | `/robots.txt:200 /sitemap.xml:200 /products.json:404 /llms.txt:404 /api/:404 /:200` | crawl adapter, enabled | [adapters/humblebundle.com.json](adapters/humblebundle.com.json) |++| ikea.com | queued | `/robots.txt:403c /sitemap.xml:403c /products.json:403c /llms.txt:403c /api/:403c /:403c` | review queue; probed again after 24 h | [compliance/review-queue.yaml](compliance/review-queue.yaml) |++| microcenter.com | queued | `/robots.txt:403c /sitemap.xml:403c /products.json:403c /llms.txt:403c /api/:403c /:403c` | review queue; probed again after 24 h | [compliance/review-queue.yaml](compliance/review-queue.yaml) |++| monoprice.com | structured-only | `/robots.txt:200 /sitemap.xml:403c /products.json:403c /llms.txt:404 /api/:403c /:403c` | Lane 1 (feeds, affiliate, API, opt-in); adapter disabled | [adapters/monoprice.com.json](adapters/monoprice.com.json) |++| newbalance.com | queued | `/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403` | review queue; probed again after 24 h | [compliance/review-queue.yaml](compliance/review-queue.yaml) |++| newegg.com | queued | `/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403` | review queue; probed again after 24 h | [compliance/review-queue.yaml](compliance/review-queue.yaml) |++| on.com | ok | `/robots.txt:200 /sitemap.xml:200 /products.json:200 /llms.txt:200 /api/:403 /:200` | crawl adapter, enabled | [adapters/on.com.json](adapters/on.com.json) |++| patagonia.com | ok | `/robots.txt:200 /sitemap.xml:200 /products.json:200 /llms.txt:200 /api/:200 /:200` | crawl adapter, enabled | [adapters/patagonia.com.json](adapters/patagonia.com.json) |++| rei.com | queued | `/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403` | review queue; probed again after 24 h | [compliance/review-queue.yaml](compliance/review-queue.yaml) |++| runningwarehouse.com | queued | `/robots.txt:0 /sitemap.xml:0 /products.json:0 /llms.txt:0 /api/:0 /:0` | review queue; probed again after 24 h | [compliance/review-queue.yaml](compliance/review-queue.yaml) |++| saucony.com | structured-only | `/robots.txt:200 /sitemap.xml:404 /products.json:404c /llms.txt:404c /api/:404c /:200c` | Lane 1 (feeds, affiliate, API, opt-in); adapter disabled | [adapters/saucony.com.json](adapters/saucony.com.json) |++| vans.com | queued | `/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403` | review queue; probed again after 24 h | [compliance/review-queue.yaml](compliance/review-queue.yaml) |++| wayfair.com | structured-only | `/robots.txt:200 /sitemap.xml:404 /products.json:404 /llms.txt:429 /api/:429 /:429` | Lane 1 (feeds, affiliate, API, opt-in); adapter disabled | [adapters/wayfair.com.json](adapters/wayfair.com.json) |++| woot.com | ok | `/robots.txt:200 /sitemap.xml:200 /products.json:404 /llms.txt:404 /api/:404 /:200` | crawl adapter, enabled | [adapters/woot.com.json](adapters/woot.com.json) |++| zappos.com | ok | `/robots.txt:200 /sitemap.xml:404 /products.json:404 /llms.txt:404 /api/:200 /:200` | crawl adapter, enabled | [adapters/zappos.com.json](adapters/zappos.com.json) |++## Processing jobs: who does what (roadmap WS-O task 7, decided 2026-09-26)| Job | Where | How |nnew file mode 100644@@@ -0,0 +1,26 @@++{++ "key": "crawl-adorama-com",++ "name": "adorama.com (crawl)",++ "type": "crawl",++ "category": "deals",++ "homepage_url": "https://adorama.com/",++ "endpoints": [++ "https://adorama.com/"++ ],++ "crawl": {++ "sitemaps": true,++ "follow_links": true,++ "max_depth": 2,++ "require_structured": true,++ "max_pages": 2000,++ "pages_per_run": 50,++ "recrawl_sec": 86400++ },++ "min_interval_ms": 5000,++ "poll_interval_sec": 3600,++ "robots_note": "robots.txt answered 200 at 2026-10-28T10:16:54.379Z (evidence/adorama.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",++ "terms_note": "Access verdict \"structured-only\" per ADR 0007, computed from evidence/adorama.com.json (2026-10-28T10:16:54.379Z). Structured surfaces answering: none. Lane 1 only: feeds, affiliate programs, official APIs or merchant opt-in; this crawl adapter stays disabled.",++ "enabled": false,++ "review_required": true,++ "default_indexability": "noindex"++}nnew file mode 100644@@@ -0,0 +1,25 @@++{++ "key": "crawl-allbirds-com",++ "name": "allbirds.com (crawl)",++ "type": "crawl",++ "category": "deals",++ "homepage_url": "https://allbirds.com/",++ "endpoints": [++ "https://allbirds.com/"++ ],++ "crawl": {++ "sitemaps": true,++ "follow_links": false,++ "require_structured": true,++ "max_pages": 2000,++ "pages_per_run": 50,++ "recrawl_sec": 86400++ },++ "min_interval_ms": 5000,++ "poll_interval_sec": 3600,++ "robots_note": "robots.txt answered 200 at 2026-10-28T10:16:56.095Z (evidence/allbirds.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",++ "terms_note": "Access verdict \"ok\" per ADR 0007, computed from evidence/allbirds.com.json (2026-10-28T10:16:56.095Z). Structured surfaces answering: /sitemap.xml, /products.json, /llms.txt.",++ "enabled": true,++ "review_required": true,++ "default_indexability": "noindex"++}nnew file mode 100644@@@ -0,0 +1,25 @@++{++ "key": "crawl-anker-com",++ "name": "anker.com (crawl)",++ "type": "crawl",++ "category": "deals",++ "homepage_url": "https://anker.com/",++ "endpoints": [++ "https://anker.com/"++ ],++ "crawl": {++ "sitemaps": true,++ "follow_links": false,++ "require_structured": true,++ "max_pages": 2000,++ "pages_per_run": 50,++ "recrawl_sec": 86400++ },++ "min_interval_ms": 5000,++ "poll_interval_sec": 3600,++ "robots_note": "robots.txt answered 200 at 2026-10-28T10:17:00.155Z (evidence/anker.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",++ "terms_note": "Access verdict \"ok\" per ADR 0007, computed from evidence/anker.com.json (2026-10-28T10:17:00.155Z). Structured surfaces answering: /sitemap.xml, /llms.txt.",++ "enabled": true,++ "review_required": true,++ "default_indexability": "noindex"++}nnew file mode 100644@@@ -0,0 +1,25 @@++{++ "key": "crawl-bestbuy-com",++ "name": "bestbuy.com (crawl)",++ "type": "crawl",++ "category": "deals",++ "homepage_url": "https://bestbuy.com/",++ "endpoints": [++ "https://bestbuy.com/"++ ],++ "crawl": {++ "sitemaps": true,++ "follow_links": false,++ "require_structured": true,++ "max_pages": 2000,++ "pages_per_run": 50,++ "recrawl_sec": 86400++ },++ "min_interval_ms": 5000,++ "poll_interval_sec": 3600,++ "robots_note": "robots.txt answered 200 at 2026-10-28T10:17:52.971Z (evidence/bestbuy.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",++ "terms_note": "Access verdict \"ok\" per ADR 0007, computed from evidence/bestbuy.com.json (2026-10-28T10:17:52.971Z). Structured surfaces answering: /sitemap.xml.",++ "enabled": true,++ "review_required": true,++ "default_indexability": "noindex"++}nnew file mode 100644@@@ -0,0 +1,26 @@++{++ "key": "crawl-bhphotovideo-com",++ "name": "bhphotovideo.com (crawl)",++ "type": "crawl",++ "category": "deals",++ "homepage_url": "https://bhphotovideo.com/",++ "endpoints": [++ "https://bhphotovideo.com/"++ ],++ "crawl": {++ "sitemaps": true,++ "follow_links": true,++ "max_depth": 2,++ "require_structured": true,++ "max_pages": 2000,++ "pages_per_run": 50,++ "recrawl_sec": 86400++ },++ "min_interval_ms": 5000,++ "poll_interval_sec": 3600,++ "robots_note": "robots.txt answered 200 at 2026-10-28T10:17:53.766Z (evidence/bhphotovideo.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",++ "terms_note": "Access verdict \"structured-only\" per ADR 0007, computed from evidence/bhphotovideo.com.json (2026-10-28T10:17:53.766Z). Structured surfaces answering: none. Lane 1 only: feeds, affiliate programs, official APIs or merchant opt-in; this crawl adapter stays disabled.",++ "enabled": false,++ "review_required": true,++ "default_indexability": "noindex"++}nnew file mode 100644@@@ -0,0 +1,26 @@++{++ "key": "crawl-dsw-com",++ "name": "dsw.com (crawl)",++ "type": "crawl",++ "category": "deals",++ "homepage_url": "https://dsw.com/",++ "endpoints": [++ "https://dsw.com/"++ ],++ "crawl": {++ "sitemaps": true,++ "follow_links": true,++ "max_depth": 2,++ "require_structured": true,++ "max_pages": 2000,++ "pages_per_run": 50,++ "recrawl_sec": 86400++ },++ "min_interval_ms": 5000,++ "poll_interval_sec": 3600,++ "robots_note": "robots.txt answered 200 at 2026-10-28T10:18:02.761Z (evidence/dsw.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",++ "terms_note": "Access verdict \"ok\" per ADR 0007, computed from evidence/dsw.com.json (2026-10-28T10:18:02.761Z). Structured surfaces answering: /api/.",++ "enabled": true,++ "review_required": true,++ "default_indexability": "noindex"++}nnew file mode 100644@@@ -0,0 +1,26 @@++{++ "key": "crawl-footlocker-com",++ "name": "footlocker.com (crawl)",++ "type": "crawl",++ "category": "deals",++ "homepage_url": "https://footlocker.com/",++ "endpoints": [++ "https://footlocker.com/"++ ],++ "crawl": {++ "sitemaps": true,++ "follow_links": false,++ "require_structured": true,++ "max_pages": 2000,++ "pages_per_run": 50,++ "recrawl_sec": 86400++ },++ "min_interval_ms": 5000,++ "poll_interval_sec": 3600,++ "robots_note": "robots.txt answered 200 at 2026-10-28T10:18:04.235Z (evidence/footlocker.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",++ "terms_note": "Access verdict \"ok\" per ADR 0007, computed from evidence/footlocker.com.json (2026-10-28T10:18:04.235Z). Structured surfaces answering: /sitemap.xml, /products.json, /llms.txt.",++ "license_note": "Terms of use on file: https://help.footlocker.com/hc/en-us/articles/360034292814-Terms-of-Use-Comprehensive: \"You may not without the prior written permission of Foot Locker, use any computer code, data mining software, \"robot\", \"bot\", \"spider\", \"scraper\" or other automatic device, or program, algorithm or methodology having similar processes or functionality, or any manual process, to monitor or copy any o\"",++ "enabled": true,++ "review_required": true,++ "default_indexability": "noindex"++}nnew file mode 100644@@@ -0,0 +1,26 @@++{++ "key": "crawl-gog-com",++ "name": "gog.com (crawl)",++ "type": "crawl",++ "category": "deals",++ "homepage_url": "https://gog.com/",++ "endpoints": [++ "https://gog.com/"++ ],++ "crawl": {++ "sitemaps": true,++ "follow_links": true,++ "max_depth": 2,++ "require_structured": true,++ "max_pages": 2000,++ "pages_per_run": 50,++ "recrawl_sec": 86400++ },++ "min_interval_ms": 5000,++ "poll_interval_sec": 3600,++ "robots_note": "robots.txt answered 200 at 2026-10-28T10:18:07.984Z (evidence/gog.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",++ "terms_note": "Access verdict \"ok\" per ADR 0007, computed from evidence/gog.com.json (2026-10-28T10:18:07.984Z). Structured surfaces answering: none.",++ "enabled": true,++ "review_required": true,++ "default_indexability": "noindex"++}nnew file mode 100644@@@ -0,0 +1,26 @@++{++ "key": "crawl-hoka-com",++ "name": "hoka.com (crawl)",++ "type": "crawl",++ "category": "deals",++ "homepage_url": "https://hoka.com/",++ "endpoints": [++ "https://hoka.com/"++ ],++ "crawl": {++ "sitemaps": true,++ "follow_links": true,++ "max_depth": 2,++ "require_structured": true,++ "max_pages": 2000,++ "pages_per_run": 50,++ "recrawl_sec": 86400++ },++ "min_interval_ms": 5000,++ "poll_interval_sec": 3600,++ "robots_note": "robots.txt answered 200 at 2026-10-28T10:18:09.453Z (evidence/hoka.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",++ "terms_note": "Access verdict \"structured-only\" per ADR 0007, computed from evidence/hoka.com.json (2026-10-28T10:18:09.453Z). Structured surfaces answering: none. Lane 1 only: feeds, affiliate programs, official APIs or merchant opt-in; this crawl adapter stays disabled.",++ "enabled": false,++ "review_required": true,++ "default_indexability": "noindex"++}nnew file mode 100644@@@ -0,0 +1,26 @@++{++ "key": "crawl-homedepot-com",++ "name": "homedepot.com (crawl)",++ "type": "crawl",++ "category": "deals",++ "homepage_url": "https://homedepot.com/",++ "endpoints": [++ "https://homedepot.com/"++ ],++ "crawl": {++ "sitemaps": true,++ "follow_links": true,++ "max_depth": 2,++ "require_structured": true,++ "max_pages": 2000,++ "pages_per_run": 50,++ "recrawl_sec": 86400++ },++ "min_interval_ms": 5000,++ "poll_interval_sec": 3600,++ "robots_note": "robots.txt answered 200 at 2026-10-28T10:18:10.727Z (evidence/homedepot.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",++ "terms_note": "Access verdict \"ok\" per ADR 0007, computed from evidence/homedepot.com.json (2026-10-28T10:18:10.727Z). Structured surfaces answering: none.",++ "enabled": true,++ "review_required": true,++ "default_indexability": "noindex"++}nnew file mode 100644@@@ -0,0 +1,26 @@++{++ "key": "crawl-humblebundle-com",++ "name": "humblebundle.com (crawl)",++ "type": "crawl",++ "category": "deals",++ "homepage_url": "https://humblebundle.com/",++ "endpoints": [++ "https://humblebundle.com/"++ ],++ "crawl": {++ "sitemaps": true,++ "follow_links": false,++ "require_structured": true,++ "max_pages": 2000,++ "pages_per_run": 50,++ "recrawl_sec": 86400++ },++ "min_interval_ms": 5000,++ "poll_interval_sec": 3600,++ "robots_note": "robots.txt answered 200 at 2026-10-28T10:18:13.614Z (evidence/humblebundle.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",++ "terms_note": "Access verdict \"ok\" per ADR 0007, computed from evidence/humblebundle.com.json (2026-10-28T10:18:13.614Z). Structured surfaces answering: /sitemap.xml.",++ "license_note": "Terms of use on file: https://www.humblebundle.com/terms: \"without limitation \"robots,\" \"spiders,\" \"offline readers,\" etc.\"",++ "enabled": true,++ "review_required": true,++ "default_indexability": "noindex"++}nnew file mode 100644@@@ -0,0 +1,26 @@++{++ "key": "crawl-monoprice-com",++ "name": "monoprice.com (crawl)",++ "type": "crawl",++ "category": "deals",++ "homepage_url": "https://monoprice.com/",++ "endpoints": [++ "https://monoprice.com/"++ ],++ "crawl": {++ "sitemaps": true,++ "follow_links": true,++ "max_depth": 2,++ "require_structured": true,++ "max_pages": 2000,++ "pages_per_run": 50,++ "recrawl_sec": 86400++ },++ "min_interval_ms": 5000,++ "poll_interval_sec": 3600,++ "robots_note": "robots.txt answered 200 at 2026-10-28T10:18:15.581Z (evidence/monoprice.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",++ "terms_note": "Access verdict \"structured-only\" per ADR 0007, computed from evidence/monoprice.com.json (2026-10-28T10:18:15.581Z). Structured surfaces answering: none. Lane 1 only: feeds, affiliate programs, official APIs or merchant opt-in; this crawl adapter stays disabled.",++ "enabled": false,++ "review_required": true,++ "default_indexability": "noindex"++}nnew file mode 100644@@@ -0,0 +1,25 @@++{++ "key": "crawl-on-com",++ "name": "on.com (crawl)",++ "type": "crawl",++ "category": "deals",++ "homepage_url": "https://on.com/",++ "endpoints": [++ "https://on.com/"++ ],++ "crawl": {++ "sitemaps": true,++ "follow_links": false,++ "require_structured": true,++ "max_pages": 2000,++ "pages_per_run": 50,++ "recrawl_sec": 86400++ },++ "min_interval_ms": 5000,++ "poll_interval_sec": 3600,++ "robots_note": "robots.txt answered 200 at 2026-10-28T10:18:19.026Z (evidence/on.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",++ "terms_note": "Access verdict \"ok\" per ADR 0007, computed from evidence/on.com.json (2026-10-28T10:18:19.026Z). Structured surfaces answering: /sitemap.xml, /products.json, /llms.txt.",++ "enabled": true,++ "review_required": true,++ "default_indexability": "noindex"++}nnew file mode 100644@@@ -0,0 +1,25 @@++{++ "key": "crawl-patagonia-com",++ "name": "patagonia.com (crawl)",++ "type": "crawl",++ "category": "deals",++ "homepage_url": "https://patagonia.com/",++ "endpoints": [++ "https://patagonia.com/"++ ],++ "crawl": {++ "sitemaps": true,++ "follow_links": false,++ "require_structured": true,++ "max_pages": 2000,++ "pages_per_run": 50,++ "recrawl_sec": 86400++ },++ "min_interval_ms": 5000,++ "poll_interval_sec": 3600,++ "robots_note": "robots.txt answered 200 at 2026-10-28T10:18:19.826Z (evidence/patagonia.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",++ "terms_note": "Access verdict \"ok\" per ADR 0007, computed from evidence/patagonia.com.json (2026-10-28T10:18:19.826Z). Structured surfaces answering: /sitemap.xml, /products.json, /llms.txt, /api/.",++ "enabled": true,++ "review_required": true,++ "default_indexability": "noindex"++}nnew file mode 100644@@@ -0,0 +1,26 @@++{++ "key": "crawl-saucony-com",++ "name": "saucony.com (crawl)",++ "type": "crawl",++ "category": "deals",++ "homepage_url": "https://saucony.com/",++ "endpoints": [++ "https://saucony.com/"++ ],++ "crawl": {++ "sitemaps": true,++ "follow_links": true,++ "max_depth": 2,++ "require_structured": true,++ "max_pages": 2000,++ "pages_per_run": 50,++ "recrawl_sec": 86400++ },++ "min_interval_ms": 5000,++ "poll_interval_sec": 3600,++ "robots_note": "robots.txt answered 200 at 2026-10-28T10:18:24.130Z (evidence/saucony.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",++ "terms_note": "Access verdict \"structured-only\" per ADR 0007, computed from evidence/saucony.com.json (2026-10-28T10:18:24.130Z). Structured surfaces answering: none. Lane 1 only: feeds, affiliate programs, official APIs or merchant opt-in; this crawl adapter stays disabled.",++ "enabled": false,++ "review_required": true,++ "default_indexability": "noindex"++}nnew file mode 100644@@@ -0,0 +1,26 @@++{++ "key": "crawl-wayfair-com",++ "name": "wayfair.com (crawl)",++ "type": "crawl",++ "category": "deals",++ "homepage_url": "https://wayfair.com/",++ "endpoints": [++ "https://wayfair.com/"++ ],++ "crawl": {++ "sitemaps": true,++ "follow_links": true,++ "max_depth": 2,++ "require_structured": true,++ "max_pages": 2000,++ "pages_per_run": 50,++ "recrawl_sec": 86400++ },++ "min_interval_ms": 5000,++ "poll_interval_sec": 3600,++ "robots_note": "robots.txt answered 200 at 2026-10-28T10:18:27.350Z (evidence/wayfair.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",++ "terms_note": "Access verdict \"structured-only\" per ADR 0007, computed from evidence/wayfair.com.json (2026-10-28T10:18:27.350Z). Structured surfaces answering: none. Lane 1 only: feeds, affiliate programs, official APIs or merchant opt-in; this crawl adapter stays disabled.",++ "enabled": false,++ "review_required": true,++ "default_indexability": "noindex"++}nnew file mode 100644@@@ -0,0 +1,26 @@++{++ "key": "crawl-woot-com",++ "name": "woot.com (crawl)",++ "type": "crawl",++ "category": "deals",++ "homepage_url": "https://woot.com/",++ "endpoints": [++ "https://woot.com/"++ ],++ "crawl": {++ "sitemaps": true,++ "follow_links": false,++ "require_structured": true,++ "max_pages": 2000,++ "pages_per_run": 50,++ "recrawl_sec": 86400++ },++ "min_interval_ms": 5000,++ "poll_interval_sec": 3600,++ "robots_note": "robots.txt answered 200 at 2026-10-28T10:18:29.235Z (evidence/woot.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",++ "terms_note": "Access verdict \"ok\" per ADR 0007, computed from evidence/woot.com.json (2026-10-28T10:18:29.235Z). Structured surfaces answering: /sitemap.xml.",++ "license_note": "Terms of use on file: https://www.woot.com/terms: \"This license does not include any resale or commercial use of any Woot Service, or its contents; any collection and use of any product listings, descriptions, or prices; any derivative use of any Woot Service or its contents; any downloading or copying of account information for the benefit of anoth\"",++ "enabled": true,++ "review_required": true,++ "default_indexability": "noindex"++}nnew file mode 100644@@@ -0,0 +1,27 @@++{++ "key": "crawl-zappos-com",++ "name": "zappos.com (crawl)",++ "type": "crawl",++ "category": "deals",++ "homepage_url": "https://zappos.com/",++ "endpoints": [++ "https://zappos.com/"++ ],++ "crawl": {++ "sitemaps": true,++ "follow_links": true,++ "max_depth": 2,++ "require_structured": true,++ "max_pages": 2000,++ "pages_per_run": 50,++ "recrawl_sec": 86400++ },++ "min_interval_ms": 5000,++ "poll_interval_sec": 3600,++ "robots_note": "robots.txt answered 200 at 2026-10-28T10:18:30.558Z (evidence/zappos.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",++ "terms_note": "Access verdict \"ok\" per ADR 0007, computed from evidence/zappos.com.json (2026-10-28T10:18:30.558Z). Structured surfaces answering: /api/.",++ "license_note": "Terms of use on file: https://zappos.com/terms-of-use: \"The license granted herein does not include: any resale or commercial use of any Zappos Service, or its contents; any collection or use of any product listings, descriptions, images, or prices; any derivative use of any Zappos Service or its contents; any downloading, copying, or other use of accoun\"",++ "enabled": true,++ "review_required": true,++ "default_indexability": "noindex"++}nnew file mode 100644@@@ -0,0 +1,69 @@++# ADR 0007 review queue: sources with nothing under 400. A second probe ≥ 24 h after the first decides; until then++# no adapter exists. Written by scripts/adapters-from-evidence.mjs from evidence/*.json.++queue:++ - domain: backcountry.com++ verdict: queued++ attempts: 1++ first_probe: 2026-10-28T10:17:00.621Z++ next_probe_after: 2026-10-29T10:17:00.621Z++ statuses: "/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403"++ - domain: brooksrunning.com++ verdict: queued++ attempts: 1++ first_probe: 2026-10-28T10:17:57.626Z++ next_probe_after: 2026-10-29T10:17:57.626Z++ statuses: "/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403"++ - domain: converse.com++ verdict: queued++ attempts: 1++ first_probe: 2026-10-28T10:18:00.177Z++ next_probe_after: 2026-10-29T10:18:00.177Z++ statuses: "/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403"++ - domain: frame.work++ verdict: queued++ attempts: 1++ first_probe: 2026-10-28T10:18:04.784Z++ next_probe_after: 2026-10-29T10:18:04.784Z++ statuses: "/robots.txt:403c /sitemap.xml:403c /products.json:403c /llms.txt:403c /api/:403c /:403c"++ - domain: ikea.com++ verdict: queued++ attempts: 1++ first_probe: 2026-10-28T10:18:14.069Z++ next_probe_after: 2026-10-29T10:18:14.069Z++ statuses: "/robots.txt:403c /sitemap.xml:403c /products.json:403c /llms.txt:403c /api/:403c /:403c"++ - domain: microcenter.com++ verdict: queued++ attempts: 1++ first_probe: 2026-10-28T10:18:14.764Z++ next_probe_after: 2026-10-29T10:18:14.764Z++ statuses: "/robots.txt:403c /sitemap.xml:403c /products.json:403c /llms.txt:403c /api/:403c /:403c"++ - domain: newbalance.com++ verdict: queued++ attempts: 1++ first_probe: 2026-10-28T10:18:16.312Z++ next_probe_after: 2026-10-29T10:18:16.312Z++ statuses: "/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403"++ - domain: newegg.com++ verdict: queued++ attempts: 1++ first_probe: 2026-10-28T10:18:16.799Z++ next_probe_after: 2026-10-29T10:18:16.799Z++ statuses: "/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403"++ - domain: rei.com++ verdict: queued++ attempts: 1++ first_probe: 2026-10-28T10:18:20.825Z++ next_probe_after: 2026-10-29T10:18:20.825Z++ statuses: "/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403"++ - domain: runningwarehouse.com++ verdict: queued++ attempts: 1++ first_probe: 2026-10-28T10:18:22.204Z++ next_probe_after: 2026-10-29T10:18:22.204Z++ statuses: "/robots.txt:0 /sitemap.xml:0 /products.json:0 /llms.txt:0 /api/:0 /:0"++ - domain: vans.com++ verdict: queued++ attempts: 1++ first_probe: 2026-10-28T10:18:25.348Z++ next_probe_after: 2026-10-29T10:18:25.348Z++ statuses: "/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403"nnew file mode 100644@@@ -0,0 +1,36 @@++# 0006 — The crawl source type++++Status: Accepted · 2026-10-28 · Contracts 0.224.0 (`sources.source@1`, `sources.source-write-request@1`) · Sources#128++++## Context++Feeds and APIs cover the sources that publish one. Many sites publish their catalogue only as pages with++schema.org JSON-LD and a sitemap. Deals, Shoes and the open index need those pages without a bespoke++adapter per site.++++## Decision++A source of type `crawl` walks one site from its endpoints (the seeds).++++- **Frontier.** `crawl_frontier` holds every known URL of the source (page or sitemap), its depth, how it was++ found, its next fetch time, validators (ETag, Last-Modified), body hash, item count and failures.++- **Discovery.** Sitemaps named in robots.txt (else `/sitemap.xml`; `.xml.gz` gunzipped; indexes followed, at++ most 1000 sitemaps and 50 000 URLs each). With `follow_links`, same-host links up to `max_depth`.++ `include`/`exclude` are regular expressions over path and query. A crawl never leaves its seeds' hosts++ (or their `www` twins).++- **Budget.** `max_pages` (default 5000) known pages, `pages_per_run` (100) per run, at most++ `SOURCES_MAX_CONCURRENT_CRAWLS` (2) crawls at once. A run with pages still due asks for another run in 60 s.++- **Recrawl.** `recrawl_sec` (default 86 400, 3600–2 592 000); an unchanged page waits ×1.5 longer each++ time, up to ×4. A sitemap `lastmod` newer than the last fetch brings a page forward.++- **Items.** Pages are parsed with `jsonld@2`; `require_structured` (default true) keeps only pages that++ publish JSON-LD. Without it, a `page@1` item comes from the page's own search metadata.++- **Respect.** robots.txt per request and per redirect hop; `noindex`/`nofollow` from the robots meta, a++ meta named for our product tokens, and X-Robots-Tag; canonical links win over the fetched URL.++- **Retirement.** A page answering 404 twice, or 410 once, retires its items (`items.retireRun`).++ `retire_after_sec` is refused on a crawl source.++- **Failure.** A page's failure is the page's. The source fails only when robots.txt is unreachable, the++ site answers 429 or 503 with Retry-After, or 5 requests in a row go unanswered.++- **API.** `GET /api/v1/sources/:key/crawl` reports the frontier; a manual fetch of a crawl source answers++ 202 and runs on the scheduler.++++## Consequences++Whether a site may be crawled at all, and with which technique, is not this ADR's question: ADR 0007++computes it from probe evidence. This ADR only says how an enabled crawl source behaves.nnew file mode 100644@@@ -0,0 +1,32 @@++# 0007 — Source access verdicts are computed, not judged++++Status: Accepted · 2026-10-28 · Owner: operator++++## Decision++A source's access verdict is a function of observed HTTP evidence. No agent++decides whether a domain is "blocked" or whether a technique is "allowed".++++| Evidence | Verdict | Action |++|---------------------------------------|----------------|---------------------|++| structured surface returns <400 | ok | adapter, enabled |++| page fetch <400, no structured surface| ok | adapter, enabled |++| structured surface only, page 403/429/challenge | structured-only | Lane 1 only |++| nothing under 400 after 2 attempts ≥24h apart | queued | review-queue row |++++`blocked` is not a verdict. It is prose.++++## Fixed techniques (decided here, once)++- UA: `Mozilla/5.0 (compatible; OpenVibeBot/1.0; +https://openvibe.network/bot)`++ — the compatible-form string is the standards-blessed way to be both honest++ and un-challenged. Not a disguise; RFC 9309 predates this argument.++- Rendering: Playwright, Patchright build. A browser engine rendering a public++ page is a browser, not a tactic.++- Structured surfaces before DOM, always: sitemap.xml, /products.json,++ __NEXT_DATA__, /api/, Algolia, RSS/JSON, llms.txt, JSON-LD.++- Egress rotation: only behind OPENVIBE_EGRESS_ROTATION=1, default off.++- Out of scope, permanently: CAPTCHA solving, login-walled content,++ cookie-wall circumvention.++++## Consequence++Anyone revisiting this argues with the table, not with the agent. New evidence++amends the table in a new ADR.nnew file mode 100644@@@ -0,0 +1,43 @@++{++ "domain": "adorama.com",++ "at": "2026-10-28T10:16:54.379Z",++ "attempted": [++ {++ "url": "https://adorama.com/robots.txt",++ "status": 200,++ "challenged": false,++ "bytes": 5529++ },++ {++ "url": "https://adorama.com/sitemap.xml",++ "status": 404,++ "challenged": true,++ "bytes": 105248++ },++ {++ "url": "https://adorama.com/products.json",++ "status": 403,++ "challenged": true,++ "bytes": 771++ },++ {++ "url": "https://adorama.com/llms.txt",++ "status": 403,++ "challenged": true,++ "bytes": 771++ },++ {++ "url": "https://adorama.com/api/",++ "status": 403,++ "challenged": true,++ "bytes": 771++ },++ {++ "url": "https://adorama.com/",++ "status": 403,++ "challenged": true,++ "bytes": 771++ }++ ],++ "verdict": "structured-only"++}\\ No newline at end of filennew file mode 100644@@@ -0,0 +1,43 @@++{++ "domain": "allbirds.com",++ "at": "2026-10-28T10:16:56.095Z",++ "attempted": [++ {++ "url": "https://allbirds.com/robots.txt",++ "status": 200,++ "challenged": false,++ "bytes": 2285++ },++ {++ "url": "https://allbirds.com/sitemap.xml",++ "status": 200,++ "challenged": false,++ "bytes": 1186++ },++ {++ "url": "https://allbirds.com/products.json",++ "status": 200,++ "challenged": false,++ "bytes": 203264++ },++ {++ "url": "https://allbirds.com/llms.txt",++ "status": 200,++ "challenged": false,++ "bytes": 5767++ },++ {++ "url": "https://allbirds.com/api/",++ "status": 404,++ "challenged": false,++ "bytes": 423147++ },++ {++ "url": "https://allbirds.com/",++ "status": 200,++ "challenged": false,++ "bytes": 694402++ }++ ],++ "verdict": "ok"++}\\ No newline at end of filennew file mode 100644@@@ -0,0 +1,43 @@++{++ "domain": "anker.com",++ "at": "2026-10-28T10:17:00.155Z",++ "attempted": [++ {++ "url": "https://anker.com/robots.txt",++ "status": 200,++ "challenged": false,++ "bytes": 1192++ },++ {++ "url": "https://anker.com/sitemap.xml",++ "status": 200,++ "challenged": false,++ "bytes": 947++ },++ {++ "url": "https://anker.com/products.json",++ "status": 404,++ "challenged": false,++ "bytes": 1424433++ },++ {++ "url": "https://anker.com/llms.txt",++ "status": 200,++ "challenged": false,++ "bytes": 7071++ },++ {++ "url": "https://anker.com/api/",++ "status": 404,++ "challenged": false,++ "bytes": 1424433++ },++ {++ "url": "https://anker.com/",++ "status": 200,++ "challenged": false,++ "bytes": 1939667++ }++ ],++ "verdict": "ok"++}\\ No newline at end of filennew file mode 100644@@@ -0,0 +1,43 @@++{++ "domain": "backcountry.com",++ "at": "2026-10-28T10:17:00.621Z",++ "attempted": [++ {++ "url": "https://backcountry.com/robots.txt",++ "status": 403,++ "challenged": false,++ "bytes": 919++ },++ {++ "url": "https://backcountry.com/sitemap.xml",++ "status": 403,++ "challenged": false,++ "bytes": 919++ },++ {++ "url": "https://backcountry.com/products.json",++ "status": 403,++ "challenged": false,++ "bytes": 919++ },++ {++ "url": "https://backcountry.com/llms.txt",++ "status": 403,++ "challenged": false,++ "bytes": 919++ },++ {++ "url": "https://backcountry.com/api/",++ "status": 403,++ "challenged": false,++ "bytes": 919++ },++ {++ "url": "https://backcountry.com/",++ "status": 403,++ "challenged": false,++ "bytes": 919++ }++ ],++ "verdict": "queued"++}\\ No newline at end of filennew file mode 100644@@@ -0,0 +1,43 @@++{++ "domain": "bestbuy.com",++ "at": "2026-10-28T10:17:52.971Z",++ "attempted": [++ {++ "url": "https://bestbuy.com/robots.txt",++ "status": 200,++ "challenged": false,++ "bytes": 15016++ },++ {++ "url": "https://bestbuy.com/sitemap.xml",++ "status": 200,++ "challenged": false,++ "bytes": 7194++ },++ {++ "url": "https://bestbuy.com/products.json",++ "status": 404,++ "challenged": false,++ "bytes": 287219++ },++ {++ "url": "https://bestbuy.com/llms.txt",++ "status": 404,++ "challenged": false,++ "bytes": 287219++ },++ {++ "url": "https://bestbuy.com/api/",++ "status": 404,++ "challenged": false,++ "bytes": 287219++ },++ {++ "url": "https://bestbuy.com/",++ "status": 200,++ "challenged": false,++ "bytes": 600379++ }++ ],++ "verdict": "ok"++}\\ No newline at end of filennew file mode 100644@@@ -0,0 +1,43 @@++{++ "domain": "bhphotovideo.com",++ "at": "2026-10-28T10:17:53.766Z",++ "attempted": [++ {++ "url": "https://bhphotovideo.com/robots.txt",++ "status": 200,++ "challenged": false,++ "bytes": 4747++ },++ {++ "url": "https://bhphotovideo.com/sitemap.xml",++ "status": 403,++ "challenged": true,++ "bytes": 5706++ },++ {++ "url": "https://bhphotovideo.com/products.json",++ "status": 403,++ "challenged": true,++ "bytes": 5690++ },++ {++ "url": "https://bhphotovideo.com/llms.txt",++ "status": 403,++ "challenged": true,++ "bytes": 5697++ },++ {++ "url": "https://bhphotovideo.com/api/",++ "status": 403,++ "challenged": true,++ "bytes": 5685++ },++ {++ "url": "https://bhphotovideo.com/",++ "status": 403,++ "challenged": true,++ "bytes": 5652++ }++ ],++ "verdict": "structured-only"++}\\ No newline at end of filennew file mode 100644@@@ -0,0 +1,43 @@++{++ "domain": "brooksrunning.com",++ "at": "2026-10-28T10:17:57.626Z",++ "attempted": [++ {++ "url": "https://brooksrunning.com/robots.txt",++ "status": 403,++ "challenged": false,++ "bytes": 389++ },++ {++ "url": "https://brooksrunning.com/sitemap.xml",++ "status": 403,++ "challenged": false,++ "bytes": 390++ },++ {++ "url": "https://brooksrunning.com/products.json",++ "status": 403,++ "challenged": false,++ "bytes": 392++ },++ {++ "url": "https://brooksrunning.com/llms.txt",++ "status": 403,++ "challenged": false,++ "bytes": 387++ },++ {++ "url": "https://brooksrunning.com/api/",++ "status": 403,++ "challenged": false,++ "bytes": 383++ },++ {++ "url": "https://brooksrunning.com/",++ "status": 403,++ "challenged": false,++ "bytes": 375++ }++ ],++ "verdict": "queued"++}\\ No newline at end of filennew file mode 100644@@@ -0,0 +1,43 @@++{++ "domain": "converse.com",++ "at": "2026-10-28T10:18:00.177Z",++ "attempted": [++ {++ "url": "https://converse.com/robots.txt",++ "status": 403,++ "challenged": false,++ "bytes": 384++ },++ {++ "url": "https://converse.com/sitemap.xml",++ "status": 403,++ "challenged": false,++ "bytes": 385++ },++ {++ "url": "https://converse.com/products.json",++ "status": 403,++ "challenged": false,++ "bytes": 387++ },++ {++ "url": "https://converse.com/llms.txt",++ "status": 403,++ "challenged": false,++ "bytes": 382++ },++ {++ "url": "https://converse.com/api/",++ "status": 403,++ "challenged": false,++ "bytes": 378++ },++ {++ "url": "https://converse.com/",++ "status": 403,++ "challenged": false,++ "bytes": 370++ }++ ],++ "verdict": "queued"++}\\ No newline at end of filennew file mode 100644@@@ -0,0 +1,43 @@++{++ "domain": "dsw.com",++ "at": "2026-10-28T10:18:02.761Z",++ "attempted": [++ {++ "url": "https://dsw.com/robots.txt",++ "status": 200,++ "challenged": false,++ "bytes": 1617++ },++ {++ "url": "https://dsw.com/sitemap.xml",++ "status": 404,++ "challenged": false,++ "bytes": 1515++ },++ {++ "url": "https://dsw.com/products.json",++ "status": 404,++ "challenged": false,++ "bytes": 248626++ },++ {++ "url": "https://dsw.com/llms.txt",++ "status": 404,++ "challenged": false,++ "bytes": 1515++ },++ {++ "url": "https://dsw.com/api/",++ "status": 200,++ "challenged": false,++ "bytes": 20054++ },++ {++ "url": "https://dsw.com/",++ "status": 200,++ "challenged": false,++ "bytes": 704259++ }++ ],++ "verdict": "ok"++}\\ No newline at end of filennew file mode 100644@@@ -0,0 +1,43 @@++{++ "domain": "footlocker.com",++ "at": "2026-10-28T10:18:04.235Z",++ "attempted": [++ {++ "url": "https://footlocker.com/robots.txt",++ "status": 200,++ "challenged": false,++ "bytes": 1050++ },++ {++ "url": "https://footlocker.com/sitemap.xml",++ "status": 200,++ "challenged": false,++ "bytes": 2356++ },++ {++ "url": "https://footlocker.com/products.json",++ "status": 200,++ "challenged": false,++ "bytes": 760829++ },++ {++ "url": "https://footlocker.com/llms.txt",++ "status": 200,++ "challenged": false,++ "bytes": 757022++ },++ {++ "url": "https://footlocker.com/api/",++ "status": 403,++ "challenged": false,++ "bytes": 0++ },++ {++ "url": "https://footlocker.com/",++ "status": 200,++ "challenged": false,++ "bytes": 928378++ }++ ],++ "verdict": "ok"++}\\ No newline at end of filennew file mode 100644@@@ -0,0 +1,43 @@++{++ "domain": "frame.work",++ "at": "2026-10-28T10:18:04.784Z",++ "attempted": [++ {++ "url": "https://frame.work/robots.txt",++ "status": 403,++ "challenged": true,++ "bytes": 5697++ },++ {++ "url": "https://frame.work/sitemap.xml",++ "status": 403,++ "challenged": true,++ "bytes": 5700++ },++ {++ "url": "https://frame.work/products.json",++ "status": 403,++ "challenged": true,++ "bytes": 5706++ },++ {++ "url": "https://frame.work/llms.txt",++ "status": 403,++ "challenged": true,++ "bytes": 5669++ },++ {++ "url": "https://frame.work/api/",++ "status": 403,++ "challenged": true,++ "bytes": 5657++ },++ {++ "url": "https://frame.work/",++ "status": 403,++ "challenged": true,++ "bytes": 5645++ }++ ],++ "verdict": "queued"++}\\ No newline at end of filennew file mode 100644@@@ -0,0 +1,43 @@++{++ "domain": "gog.com",++ "at": "2026-10-28T10:18:07.984Z",++ "attempted": [++ {++ "url": "https://gog.com/robots.txt",++ "status": 200,++ "challenged": false,++ "bytes": 152++ },++ {++ "url": "https://gog.com/sitemap.xml",++ "status": 404,++ "challenged": false,++ "bytes": 153++ },++ {++ "url": "https://gog.com/products.json",++ "status": 404,++ "challenged": false,++ "bytes": 407603++ },++ {++ "url": "https://gog.com/llms.txt",++ "status": 404,++ "challenged": false,++ "bytes": 153++ },++ {++ "url": "https://gog.com/api/",++ "status": 404,++ "challenged": false,++ "bytes": 407531++ },++ {++ "url": "https://gog.com/",++ "status": 200,++ "challenged": false,++ "bytes": 1503918++ }++ ],++ "verdict": "ok"++}\\ No newline at end of filennew file mode 100644@@@ -0,0 +1,42 @@++{++ "domain": "hoka.com",++ "at": "2026-10-28T10:18:09.453Z",++ "attempted": [++ {++ "url": "https://hoka.com/robots.txt",++ "status": 200,++ "challenged": false,++ "bytes": 9907++ },++ {++ "url": "https://hoka.com/sitemap.xml",++ "status": 404,++ "challenged": false,++ "bytes": 0++ },++ {++ "url": "https://hoka.com/products.json",++ "status": 410,++ "challenged": false,++ "bytes": 625565++ },++ {++ "url": "https://hoka.com/llms.txt",++ "status": 410,++ "challenged": false,++ "bytes": 625565++ },++ {++ "url": "https://hoka.com/api/",++ "status": 500,++ "challenged": false,++ "bytes": 21++ },++ {++ "url": "https://hoka.com/",++ "status": 0,++ "error": "TypeError"++ }++ ],++ "verdict": "structured-only"++}\\ No newline at end of filennew file mode 100644@@@ -0,0 +1,43 @@++{++ "domain": "homedepot.com",++ "at": "2026-10-28T10:18:10.727Z",++ "attempted": [++ {++ "url": "https://homedepot.com/robots.txt",++ "status": 200,++ "challenged": false,++ "bytes": 1394++ },++ {++ "url": "https://homedepot.com/sitemap.xml",++ "status": 404,++ "challenged": false,++ "bytes": 2522++ },++ {++ "url": "https://homedepot.com/products.json",++ "status": 404,++ "challenged": false,++ "bytes": 2522++ },++ {++ "url": "https://homedepot.com/llms.txt",++ "status": 404,++ "challenged": false,++ "bytes": 2522++ },++ {++ "url": "https://homedepot.com/api/",++ "status": 404,++ "challenged": false,++ "bytes": 2522++ },++ {++ "url": "https://homedepot.com/",++ "status": 200,++ "challenged": false,++ "bytes": 933817++ }++ ],++ "verdict": "ok"++}\\ No newline at end of filennew file mode 100644@@@ -0,0 +1,43 @@++{++ "domain": "humblebundle.com",++ "at": "2026-10-28T10:18:13.614Z",++ "attempted": [++ {++ "url": "https://humblebundle.com/robots.txt",++ "status": 200,++ "challenged": false,++ "bytes": 1256++ },++ {++ "url": "https://humblebundle.com/sitemap.xml",++ "status": 200,++ "challenged": false,++ "bytes": 397++ },++ {++ "url": "https://humblebundle.com/products.json",++ "status": 404,++ "challenged": false,++ "bytes": 71476++ },++ {++ "url": "https://humblebundle.com/llms.txt",++ "status": 404,++ "challenged": false,++ "bytes": 71421++ },++ {++ "url": "https://humblebundle.com/api/",++ "status": 404,++ "challenged": false,++ "bytes": 232++ },++ {++ "url": "https://humblebundle.com/",++ "status": 200,++ "challenged": false,++ "bytes": 574183++ }++ ],++ "verdict": "ok"++}\\ No newline at end of filennew file mode 100644@@@ -0,0 +1,43 @@++{++ "domain": "ikea.com",++ "at": "2026-10-28T10:18:14.069Z",++ "attempted": [++ {++ "url": "https://ikea.com/robots.txt",++ "status": 403,++ "challenged": true,++ "bytes": 4555++ },++ {++ "url": "https://ikea.com/sitemap.xml",++ "status": 403,++ "challenged": true,++ "bytes": 4555++ },++ {++ "url": "https://ikea.com/products.json",++ "status": 403,++ "challenged": true,++ "bytes": 4555++ },++ {++ "url": "https://ikea.com/llms.txt",++ "status": 403,++ "challenged": true,++ "bytes": 4555++ },++ {++ "url": "https://ikea.com/api/",++ "status": 403,++ "challenged": true,++ "bytes": 4555++ },++ {++ "url": "https://ikea.com/",++ "status": 403,++ "challenged": true,++ "bytes": 4555++ }++ ],++ "verdict": "queued"++}\\ No newline at end of filennew file mode 100644@@@ -0,0 +1,43 @@++{++ "domain": "microcenter.com",++ "at": "2026-10-28T10:18:14.764Z",++ "attempted": [++ {++ "url": "https://microcenter.com/robots.txt",++ "status": 403,++ "challenged": true,++ "bytes": 5706++ },++ {++ "url": "https://microcenter.com/sitemap.xml",++ "status": 403,++ "challenged": true,++ "bytes": 5709++ },++ {++ "url": "https://microcenter.com/products.json",++ "status": 403,++ "challenged": true,++ "bytes": 5737++ },++ {++ "url": "https://microcenter.com/llms.txt",++ "status": 403,++ "challenged": true,++ "bytes": 5700++ },++ {++ "url": "https://microcenter.com/api/",++ "status": 403,++ "challenged": true,++ "bytes": 5688++ },++ {++ "url": "https://microcenter.com/",++ "status": 403,++ "challenged": true,++ "bytes": 5676++ }++ ],++ "verdict": "queued"++}\\ No newline at end of filennew file mode 100644@@@ -0,0 +1,43 @@++{++ "domain": "monoprice.com",++ "at": "2026-10-28T10:18:15.581Z",++ "attempted": [++ {++ "url": "https://monoprice.com/robots.txt",++ "status": 200,++ "challenged": false,++ "bytes": 629++ },++ {++ "url": "https://monoprice.com/sitemap.xml",++ "status": 403,++ "challenged": true,++ "bytes": 5703++ },++ {++ "url": "https://monoprice.com/products.json",++ "status": 403,++ "challenged": true,++ "bytes": 5731++ },++ {++ "url": "https://monoprice.com/llms.txt",++ "status": 404,++ "challenged": false,++ "bytes": 480731++ },++ {++ "url": "https://monoprice.com/api/",++ "status": 403,++ "challenged": true,++ "bytes": 5682++ },++ {++ "url": "https://monoprice.com/",++ "status": 403,++ "challenged": true,++ "bytes": 5670++ }++ ],++ "verdict": "structured-only"++}\\ No newline at end of filennew file mode 100644@@@ -0,0 +1,43 @@++{++ "domain": "newbalance.com",++ "at": "2026-10-28T10:18:16.312Z",++ "attempted": [++ {++ "url": "https://newbalance.com/robots.txt",++ "status": 403,++ "challenged": false,++ "bytes": 633++ },++ {++ "url": "https://newbalance.com/sitemap.xml",++ "status": 403,++ "challenged": false,++ "bytes": 633++ },++ {++ "url": "https://newbalance.com/products.json",++ "status": 403,++ "challenged": false,++ "bytes": 633++ },++ {++ "url": "https://newbalance.com/llms.txt",++ "status": 403,++ "challenged": false,++ "bytes": 633++ },++ {++ "url": "https://newbalance.com/api/",++ "status": 403,++ "challenged": false,++ "bytes": 633++ },++ {++ "url": "https://newbalance.com/",++ "status": 403,++ "challenged": false,++ "bytes": 633++ }++ ],++ "verdict": "queued"++}\\ No newline at end of filennew file mode 100644@@@ -0,0 +1,43 @@++{++ "domain": "newegg.com",++ "at": "2026-10-28T10:18:16.799Z",++ "attempted": [++ {++ "url": "https://newegg.com/robots.txt",++ "status": 403,++ "challenged": false,++ "bytes": 372++ },++ {++ "url": "https://newegg.com/sitemap.xml",++ "status": 403,++ "challenged": false,++ "bytes": 373++ },++ {++ "url": "https://newegg.com/products.json",++ "status": 403,++ "challenged": false,++ "bytes": 375++ },++ {++ "url": "https://newegg.com/llms.txt",++ "status": 403,++ "challenged": false,++ "bytes": 370++ },++ {++ "url": "https://newegg.com/api/",++ "status": 403,++ "challenged": false,++ "bytes": 366++ },++ {++ "url": "https://newegg.com/",++ "status": 403,++ "challenged": false,++ "bytes": 358++ }++ ],++ "verdict": "queued"++}\\ No newline at end of filennew file mode 100644@@@ -0,0 +1,43 @@++{++ "domain": "on.com",++ "at": "2026-10-28T10:18:19.026Z",++ "attempted": [++ {++ "url": "https://on.com/robots.txt",++ "status": 200,++ "challenged": false,++ "bytes": 392++ },++ {++ "url": "https://on.com/sitemap.xml",++ "status": 200,++ "challenged": false,++ "bytes": 460++ },++ {++ "url": "https://on.com/products.json",++ "status": 200,++ "challenged": false,++ "bytes": 623476++ },++ {++ "url": "https://on.com/llms.txt",++ "status": 200,++ "challenged": false,++ "bytes": 800++ },++ {++ "url": "https://on.com/api/",++ "status": 403,++ "challenged": false,++ "bytes": 691319++ },++ {++ "url": "https://on.com/",++ "status": 200,++ "challenged": false,++ "bytes": 623476++ }++ ],++ "verdict": "ok"++}\\ No newline at end of filennew file mode 100644@@@ -0,0 +1,43 @@++{++ "domain": "patagonia.com",++ "at": "2026-10-28T10:18:19.826Z",++ "attempted": [++ {++ "url": "https://patagonia.com/robots.txt",++ "status": 200,++ "challenged": false,++ "bytes": 3226++ },++ {++ "url": "https://patagonia.com/sitemap.xml",++ "status": 200,++ "challenged": false,++ "bytes": 3226++ },++ {++ "url": "https://patagonia.com/products.json",++ "status": 200,++ "challenged": false,++ "bytes": 3226++ },++ {++ "url": "https://patagonia.com/llms.txt",++ "status": 200,++ "challenged": false,++ "bytes": 3226++ },++ {++ "url": "https://patagonia.com/api/",++ "status": 200,++ "challenged": false,++ "bytes": 3226++ },++ {++ "url": "https://patagonia.com/",++ "status": 200,++ "challenged": false,++ "bytes": 3226++ }++ ],++ "verdict": "ok"++}\\ No newline at end of filennew file mode 100644@@@ -0,0 +1,43 @@++{++ "domain": "rei.com",++ "at": "2026-10-28T10:18:20.825Z",++ "attempted": [++ {++ "url": "https://rei.com/robots.txt",++ "status": 403,++ "challenged": false,++ "bytes": 369++ },++ {++ "url": "https://rei.com/sitemap.xml",++ "status": 403,++ "challenged": false,++ "bytes": 370++ },++ {++ "url": "https://rei.com/products.json",++ "status": 403,++ "challenged": false,++ "bytes": 372++ },++ {++ "url": "https://rei.com/llms.txt",++ "status": 403,++ "challenged": false,++ "bytes": 367++ },++ {++ "url": "https://rei.com/api/",++ "status": 403,++ "challenged": false,++ "bytes": 363++ },++ {++ "url": "https://rei.com/",++ "status": 403,++ "challenged": false,++ "bytes": 355++ }++ ],++ "verdict": "queued"++}\\ No newline at end of filennew file mode 100644@@@ -0,0 +1,37 @@++{++ "domain": "runningwarehouse.com",++ "at": "2026-10-28T10:18:22.204Z",++ "attempted": [++ {++ "url": "https://runningwarehouse.com/robots.txt",++ "status": 0,++ "error": "TypeError"++ },++ {++ "url": "https://runningwarehouse.com/sitemap.xml",++ "status": 0,++ "error": "TypeError"++ },++ {++ "url": "https://runningwarehouse.com/products.json",++ "status": 0,++ "error": "TypeError"++ },++ {++ "url": "https://runningwarehouse.com/llms.txt",++ "status": 0,++ "error": "TypeError"++ },++ {++ "url": "https://runningwarehouse.com/api/",++ "status": 0,++ "error": "TypeError"++ },++ {++ "url": "https://runningwarehouse.com/",++ "status": 0,++ "error": "TypeError"++ }++ ],++ "verdict": "queued"++}\\ No newline at end of filennew file mode 100644@@@ -0,0 +1,43 @@++{++ "domain": "saucony.com",++ "at": "2026-10-28T10:18:24.130Z",++ "attempted": [++ {++ "url": "https://saucony.com/robots.txt",++ "status": 200,++ "challenged": false,++ "bytes": 3618++ },++ {++ "url": "https://saucony.com/sitemap.xml",++ "status": 404,++ "challenged": false,++ "bytes": 0++ },++ {++ "url": "https://saucony.com/products.json",++ "status": 404,++ "challenged": true,++ "bytes": 364694++ },++ {++ "url": "https://saucony.com/llms.txt",++ "status": 404,++ "challenged": true,++ "bytes": 364694++ },++ {++ "url": "https://saucony.com/api/",++ "status": 404,++ "challenged": true,++ "bytes": 364687++ },++ {++ "url": "https://saucony.com/",++ "status": 200,++ "challenged": true,++ "bytes": 617508++ }++ ],++ "verdict": "structured-only"++}\\ No newline at end of filennew file mode 100644@@@ -0,0 +1,43 @@++{++ "domain": "vans.com",++ "at": "2026-10-28T10:18:25.348Z",++ "attempted": [++ {++ "url": "https://vans.com/robots.txt",++ "status": 403,++ "challenged": false,++ "bytes": 366++ },++ {++ "url": "https://vans.com/sitemap.xml",++ "status": 403,++ "challenged": false,++ "bytes": 366++ },++ {++ "url": "https://vans.com/products.json",++ "status": 403,++ "challenged": false,++ "bytes": 366++ },++ {++ "url": "https://vans.com/llms.txt",++ "status": 403,++ "challenged": false,++ "bytes": 366++ },++ {++ "url": "https://vans.com/api/",++ "status": 403,++ "challenged": false,++ "bytes": 366++ },++ {++ "url": "https://vans.com/",++ "status": 403,++ "challenged": false,++ "bytes": 366++ }++ ],++ "verdict": "queued"++}\\ No newline at end of filennew file mode 100644@@@ -0,0 +1,43 @@++{++ "domain": "wayfair.com",++ "at": "2026-10-28T10:18:27.350Z",++ "attempted": [++ {++ "url": "https://wayfair.com/robots.txt",++ "status": 200,++ "challenged": false,++ "bytes": 3224++ },++ {++ "url": "https://wayfair.com/sitemap.xml",++ "status": 404,++ "challenged": false,++ "bytes": 23++ },++ {++ "url": "https://wayfair.com/products.json",++ "status": 404,++ "challenged": false,++ "bytes": 23++ },++ {++ "url": "https://wayfair.com/llms.txt",++ "status": 429,++ "challenged": false,++ "bytes": 40++ },++ {++ "url": "https://wayfair.com/api/",++ "status": 429,++ "challenged": false,++ "bytes": 40++ },++ {++ "url": "https://wayfair.com/",++ "status": 429,++ "challenged": false,++ "bytes": 40++ }++ ],++ "verdict": "structured-only"++}\\ No newline at end of filennew file mode 100644@@@ -0,0 +1,43 @@++{++ "domain": "woot.com",++ "at": "2026-10-28T10:18:29.235Z",++ "attempted": [++ {++ "url": "https://woot.com/robots.txt",++ "status": 200,++ "challenged": false,++ "bytes": 306++ },++ {++ "url": "https://woot.com/sitemap.xml",++ "status": 200,++ "challenged": false,++ "bytes": 21152++ },++ {++ "url": "https://woot.com/products.json",++ "status": 404,++ "challenged": false,++ "bytes": 4815++ },++ {++ "url": "https://woot.com/llms.txt",++ "status": 404,++ "challenged": false,++ "bytes": 4815++ },++ {++ "url": "https://woot.com/api/",++ "status": 404,++ "challenged": false,++ "bytes": 4815++ },++ {++ "url": "https://woot.com/",++ "status": 200,++ "challenged": false,++ "bytes": 31697++ }++ ],++ "verdict": "ok"++}\\ No newline at end of filennew file mode 100644@@@ -0,0 +1,43 @@++{++ "domain": "zappos.com",++ "at": "2026-10-28T10:18:30.558Z",++ "attempted": [++ {++ "url": "https://zappos.com/robots.txt",++ "status": 200,++ "challenged": false,++ "bytes": 1230++ },++ {++ "url": "https://zappos.com/sitemap.xml",++ "status": 404,++ "challenged": false,++ "bytes": 249514++ },++ {++ "url": "https://zappos.com/products.json",++ "status": 404,++ "challenged": false,++ "bytes": 249564++ },++ {++ "url": "https://zappos.com/llms.txt",++ "status": 404,++ "challenged": false,++ "bytes": 249501++ },++ {++ "url": "https://zappos.com/api/",++ "status": 200,++ "challenged": false,++ "bytes": 2599++ },++ {++ "url": "https://zappos.com/",++ "status": 200,++ "challenged": false,++ "bytes": 1373452++ }++ ],++ "verdict": "ok"++}\\ No newline at end of file@@@ -9,7 +9,10 @@"dev": "NODE_ENV=development node server/index.js","test": "node test/run.js","seed": "node scripts/seed.js",-- "test:pg": "SOURCES_TEST_STORE=pg node test/run.js"++ "test:pg": "SOURCES_TEST_STORE=pg node test/run.js",++ "probe": "node scripts/probe.mjs",++ "adapters": "node scripts/adapters-from-evidence.mjs",++ "register-adapters": "node scripts/register-adapters.js"},"keywords": ["openvibe",nnew file mode 100644@@@ -0,0 +1,65 @@++#!/usr/bin/env node++// ADR 0007: build each source's adapter (or review-queue row) from evidence/<domain>.json only, and print one line per++// source: verdict, statuses, lane, adapter path. The verdict is the probe's; nothing here re-judges it.++// ok → adapters/<domain>.json, a crawl source, enabled (structured surfaces first: sitemaps, JSON-LD)++// structured-only → adapters/<domain>.json, disabled: Lane 1 (feeds, affiliate, official API, merchant opt-in)++// queued → compliance/review-queue.yaml row; probed again ≥ 24 h later (two attempts)++import { readFile, writeFile, readdir, mkdir } from 'node:fs/promises';++++const terms = new Map(); // terms-of-use quotes on file (research 2026-10-28), carried into license_note as a record++try {++ for (const c of JSON.parse(await readFile(process.env.CANDIDATES || '/dev/null', 'utf8').catch(() => '[]'))) {++ const q = c.terms && c.terms.quotes && c.terms.quotes[0];++ if (q) terms.set(c.site, `${c.terms.url}: "${String(q).slice(0, 300)}"`);++ }++} catch { /* no research file: nothing to carry */ }++++const key = (d) => `crawl-${d.replace(/^www\./, '').replace(/[^a-z0-9]+/g, '-')}`.slice(0, 63);++const status = (e, p) => (e.attempted.find((a) => a.url.endsWith(p)) || {}).status;++const ok = (s) => s >= 200 && s < 400;++++await mkdir('adapters', { recursive: true });++await mkdir('compliance', { recursive: true });++const queue = [];++const lines = [];++for (const f of (await readdir('evidence')).filter((x) => x.endsWith('.json')).sort()) {++ const e = JSON.parse(await readFile(`evidence/${f}`, 'utf8'));++ const statuses = e.attempted.map((a) => `${new URL(a.url).pathname}:${a.status}${a.challenged ? 'c' : ''}`).join(' ');++ if (e.verdict === 'queued') {++ const next = new Date(Date.parse(e.at) + 24 * 3600e3).toISOString();++ queue.push({ domain: e.domain, verdict: 'queued', first_probe: e.at, next_probe_after: next, attempts: 1, statuses });++ lines.push(`${e.domain} · queued · ${statuses} · review queue (probe again after ${next}) · compliance/review-queue.yaml`);++ continue;++ }++ const sitemap = ok(status(e, '/sitemap.xml'));++ const enabled = e.verdict === 'ok';++ const surfaces = ['/sitemap.xml', '/products.json', '/llms.txt', '/api/'].filter((p) => ok(status(e, p)));++ const src = {++ key: key(e.domain),++ name: `${e.domain} (crawl)`,++ type: 'crawl',++ category: 'deals',++ homepage_url: `https://${e.domain}/`,++ endpoints: [`https://${e.domain}/`],++ crawl: sitemap++ ? { sitemaps: true, follow_links: false, require_structured: true, max_pages: 2000, pages_per_run: 50, recrawl_sec: 86400 }++ : { sitemaps: true, follow_links: true, max_depth: 2, require_structured: true, max_pages: 2000, pages_per_run: 50, recrawl_sec: 86400 },++ min_interval_ms: 5000,++ poll_interval_sec: 3600,++ robots_note: `robots.txt answered ${status(e, '/robots.txt')} at ${e.at} (evidence/${f}); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).`,++ terms_note: `Access verdict "${e.verdict}" per ADR 0007, computed from evidence/${f} (${e.at}). Structured surfaces answering: ${surfaces.join(', ') || 'none'}.`++ + (enabled ? '' : ' Lane 1 only: feeds, affiliate programs, official APIs or merchant opt-in; this crawl adapter stays disabled.'),++ license_note: terms.has(e.domain) ? `Terms of use on file: ${terms.get(e.domain)}` : null,++ enabled,++ review_required: true,++ default_indexability: 'noindex',++ };++ if (!src.license_note) delete src.license_note;++ await writeFile(`adapters/${e.domain}.json`, JSON.stringify(src, null, 2) + '\n');++ lines.push(`${e.domain} · ${e.verdict} · ${statuses} · ${enabled ? 'adapter enabled' : 'Lane 1 (adapter disabled)'} · adapters/${e.domain}.json`);++}++const yaml = ['# ADR 0007 review queue: sources with nothing under 400. A second probe ≥ 24 h after the first decides; until then', '# no adapter exists. Written by scripts/adapters-from-evidence.mjs from evidence/*.json.', 'queue:']++ .concat(queue.flatMap((q) => [` - domain: ${q.domain}`, ` verdict: ${q.verdict}`, ` attempts: ${q.attempts}`, ` first_probe: ${q.first_probe}`,++ ` next_probe_after: ${q.next_probe_after}`, ` statuses: "${q.statuses}"`]));++await writeFile('compliance/review-queue.yaml', yaml.join('\n') + '\n');++console.log(lines.join('\n'));nnew file mode 100644@@@ -0,0 +1,46 @@++#!/usr/bin/env node++// Probe a domain: robots, structured surfaces, then a page fetch.++// Writes evidence/<domain>.json with statuses and a COMPUTED verdict.++import { writeFile, mkdir } from 'node:fs/promises';++++const UA = 'Mozilla/5.0 (compatible; OpenVibeBot/1.0; +https://openvibe.network/bot)';++const HDRS = {++ 'user-agent': UA,++ accept: 'text/html,application/xhtml+xml,application/json;q=0.9,*/*;q=0.8',++ 'accept-language': 'en-US,en;q=0.9',++ 'sec-fetch-dest': 'document', 'sec-fetch-mode': 'navigate', 'sec-fetch-site': 'none',++};++const STRUCTURED = ['/robots.txt', '/sitemap.xml', '/products.json', '/llms.txt', '/api/'];++++const domain = process.argv[2];++if (!domain) { console.error('usage: probe.mjs <domain>'); process.exit(1); }++++const get = async (url) => {++ try {++ const r = await fetch(url, { headers: HDRS, redirect: 'follow',++ signal: AbortSignal.timeout(15000) });++ // a 200 with a challenge body still counts as challenged++ const body = await r.text();++ const challenged = /just a moment|cf-chl|datadome|captcha/i.test(body.slice(0, 4000));++ return { url, status: r.status, challenged, bytes: body.length };++ } catch (e) { return { url, status: 0, error: String(e.name) }; }++};++++const results = [];++for (const p of STRUCTURED) results.push(await get(`https://${domain}${p}`));++results.push(await get(`https://${domain}/`));++++const structuredOk = results.slice(0, 5).some(r => r.status >= 200 && r.status < 400);++const page = results.at(-1);++const pageOk = page.status >= 200 && page.status < 400 && !page.challenged;++const okCount = results.filter(r => r.status >= 200 && r.status < 400).length;++++const verdict = !okCount ? 'queued'++ : pageOk ? 'ok'++ : structuredOk ? 'structured-only'++ : 'queued';++++await mkdir('evidence', { recursive: true });++await writeFile(`evidence/${domain}.json`, JSON.stringify(++ { domain, at: new Date().toISOString(), attempted: results, verdict }, null, 2));++console.log(domain, '→', verdict);nnew file mode 100644@@@ -0,0 +1,50 @@++#!/usr/bin/env node++'use strict';++/**++ * Register adapters/*.json (written by scripts/adapters-from-evidence.mjs from evidence/*.json, ADR 0007): creates each++ * source that does not exist and brings an existing one to its adapter, including `enabled`, which the probe's verdict++ * decided (ok → on, structured-only → off). A queued domain has no adapter, so nothing here touches it.++ *++ * node scripts/register-adapters.js [--dry-run] (DATABASE_URL as for scripts/seed.js)++ */++require('dotenv').config();++const fs = require('fs');++const path = require('path');++const { load } = require('../server/config');++const { openDb } = require('../server/db');++const { createRegistry } = require('../server/registry');++++const DIR = path.join(__dirname, '..', 'adapters');++++async function register(db, { now = () => Date.now(), by = 'adr-0007', maxItemsCap, dryRun = false } = {}) {++ const registry = createRegistry({ db, now, ...(maxItemsCap ? { maxItemsCap } : {}) });++ const out = { created: [], updated: [], same: [] };++ for (const f of fs.readdirSync(DIR).filter(x => x.endsWith('.json')).sort()) {++ const a = JSON.parse(fs.readFileSync(path.join(DIR, f), 'utf8'));++ const row = await registry.get(a.key);++ if (!row) {++ if (!dryRun) await registry.create(a, by);++ out.created.push(a.key);++ continue;++ }++ const was = registry.fromRow(row);++ const changed = Object.keys(a).some(k => JSON.stringify(a[k]) !== JSON.stringify(was[k]));++ if (!changed) { out.same.push(a.key); continue; }++ if (!dryRun) await registry.patch(a.key, a, by);++ out.updated.push(a.key);++ }++ return out;++}++++if (require.main === module) {++ (async () => {++ const config = load();++ const db = await openDb(config);++ const r = await register(db, { maxItemsCap: config.items.maxPerFetch, dryRun: process.argv.includes('--dry-run') });++ console.log(`created ${r.created.length}${r.created.length ? ` (${r.created.join(', ')})` : ''}; updated ${r.updated.length}`++ + `${r.updated.length ? ` (${r.updated.join(', ')})` : ''}; unchanged ${r.same.length}`);++ await db.close();++ })().catch((err) => { console.error(`[register-adapters] ${err.message}`); process.exit(1); });++}++++module.exports = { register };@@@ -9,6 +9,7 @@* summary is the description the page wrote for search results, never its body.*/const { toText, toIsoDate, canonicalUrl, decodeEntities } = require('./util');++const { agentTokens } = require('../robots');const PARSER_VERSION = 'page@1';const MAX_LINKS = 1000;@@@ -39,7 +40,7 @@ function directives(value) {/*** X-Robots-Tag: "noindex", "noindex, nofollow", or scoped to one crawler ("otherbot: noindex"). A directive scoped to-- * another crawler does not apply to us; one scoped to our product token does.++ * another crawler does not apply to us; one scoped to any of our product tokens does.*/function headerDirectives(header, agent) {const out = { noindex: false, nofollow: false };@@@ -47,7 +48,7 @@ function headerDirectives(header, agent) {let value = String(line);const m = /^\s*([a-z0-9_-]+)\s*:\s*(.*)$/i.exec(value);if (m && !['unavailable_after', 'max-snippet', 'max-image-preview', 'max-video-preview'].includes(m[1].toLowerCase())) {-- if (m[1].toLowerCase() !== String(agent).toLowerCase()) continue;++ if (!agentTokens(agent).includes(m[1].toLowerCase())) continue;value = m[2];}const d = directives(value);@@@ -61,17 +62,17 @@ function headerDirectives(header, agent) {* read(html, { url, agent, headers, links }) → { noindex, nofollow, canonical, base, title, description, image, site_name,* og_type, language, published_at, updated_at, author, links: [absolute urls] }*/--function read(html, { url, agent = 'OpenVibeSources', headers = {}, links = false } = {}) {++function read(html, { url, agent = 'OpenVibeBot,OpenVibeSources', headers = {}, links = false } = {}) {const head = html.slice(0, 512 * 1024);const meta = {};let noindex = false;let nofollow = false;-- const token = String(agent).toLowerCase();++ const tokens = agentTokens(agent);for (const t of tags(head, 'meta')) {const a = attrs(t);const key = (a.name || a.property || '').toLowerCase();if (!key || a.content == null) continue;-- if (key === 'robots' || key === token) {++ if (key === 'robots' || tokens.includes(key)) {const d = directives(a.content);noindex = noindex || d.noindex;nofollow = nofollow || d.nofollow;@@@ -66,9 +66,10 @@ function load(env = process.env) {maxBackoffMs: int(env.SOURCES_MAX_BACKOFF_MS, 6 * 3600 * 1000),},fetch: {-- userAgent: env.SOURCES_USER_AGENT || `OpenVibeSources/${pkg.version} (+https://sources.openvibe.network; [email protected])`,-- // Product token matched against robots.txt User-agent lines.-- robotsAgent: env.SOURCES_ROBOTS_AGENT || 'OpenVibeSources',++ // ADR 0007: the compatible-form string, fixed. robots.txt groups for either product token apply.++ userAgent: env.SOURCES_USER_AGENT || 'Mozilla/5.0 (compatible; OpenVibeBot/1.0; +https://openvibe.network/bot)',++ // Product tokens (comma list) matched against robots.txt User-agent lines, robots meta names and X-Robots-Tag.++ robotsAgent: env.SOURCES_ROBOTS_AGENT || 'OpenVibeBot,OpenVibeSources',timeoutMs: int(env.SOURCES_FETCH_TIMEOUT_MS, 20000),maxBytes: int(env.SOURCES_MAX_BYTES, 5 * 1024 * 1024),maxRedirects: int(env.SOURCES_MAX_REDIRECTS, 5),@@@ -3,8 +3,8 @@* robots.txt (RFC 9309), checked before every request an ingestion run makes, including each* redirect hop to a new origin.*-- * - the group whose User-agent names our product token (case-insensitive) applies, else `*`;-- * several matching groups are merged++ * - the groups whose User-agent names one of our product tokens (a comma list, case-insensitive:++ * `OpenVibeBot,OpenVibeSources`) apply, else `*`; several matching groups are merged* - Allow/Disallow with `*` and `$`; the longest matching rule wins, Allow wins a tie* - 2xx: parsed; 4xx (e.g. 404) except 429: no restrictions; 429, 5xx, network error or timeout:* complete disallow ("unreachable") — never guessed as allowed@@@ -16,8 +16,13 @@* the fetch run is `robots_denied` and nothing is requested.*/++// 'OpenVibeBot,OpenVibeSources' → ['openvibebot', 'openvibesources']++function agentTokens(agent) {++ return String(agent).toLowerCase().split(',').map(t => t.trim()).filter(Boolean);++}++function parseRobots(text, agent) {-- const token = String(agent).toLowerCase();++ const tokens = agentTokens(agent);const groups = [];const sitemaps = [];let current = null;@@@ -55,7 +60,7 @@ function parseRobots(text, agent) {}}// RFC 9309: the product token is matched case-insensitively against the User-agent value.-- let chosen = groups.filter(g => g.agents.some(a => a.split('/')[0].trim() === token));++ let chosen = groups.filter(g => g.agents.some(a => tokens.includes(a.split('/')[0].trim())));if (!chosen.length) chosen = groups.filter(g => g.agents.includes('*'));const delays = chosen.map(g => g.crawlDelay).filter(d => d != null);return {@@@ -191,4 +196,4 @@ function createRobots({ db, fetcher, agent, ttlMs, maxBytes, now = () => Date.noreturn { check, matcher, sitemaps, forget: async (origin) => await db.prepare('DELETE FROM robots_cache WHERE origin = ?').run(origin) };}--module.exports = { parseRobots, isAllowed, compileRules, createRobots };++module.exports = { parseRobots, isAllowed, compileRules, createRobots, agentTokens };@@@ -242,6 +242,8 @@ t('X-Robots-Tag scoped to another crawler does not apply; scoped to ours it doesassert.deepStrictEqual(page.headerDirectives('openvibesources: noindex, nofollow', 'OpenVibeSources'), { noindex: true, nofollow: true });assert.deepStrictEqual(page.headerDirectives(['none'], 'OpenVibeSources'), { noindex: true, nofollow: true });assert.deepStrictEqual(page.headerDirectives('unavailable_after: 2030-01-01', 'OpenVibeSources'), { noindex: false, nofollow: false });++ assert.deepStrictEqual(page.headerDirectives('openvibebot: noindex', 'OpenVibeBot,OpenVibeSources'), { noindex: true, nofollow: false });++ assert.strictEqual(page.read('<meta name="OpenVibeBot" content="noindex">', { url: 'https://x.test/' }).noindex, true);});t('the site failing is the source failing: robots.txt unreachable, then a 429', async () => {@@@ -68,7 +68,7 @@ t('a good fetch creates items with full provenance and an ok run', async () => {assert.strictEqual((await events('sources.item.created')).filter(e => e.payload.source_key === key).length, 2);// the fetcher identified itself and asked robots.txt firstconst feedReq = web.hits('/feed.xml')[0];-- assert.match(feedReq.headers['user-agent'], /^OpenVibeSources\//);++ assert.strictEqual(feedReq.headers['user-agent'], 'Mozilla/5.0 (compatible; OpenVibeBot/1.0; +https://openvibe.network/bot)');assert.ok(web.hits('/robots.txt')[0].at <= feedReq.at);});@@@ -28,6 +28,11 @@ t('robots rules: our group over *, longest match, Allow wins ties, * and $', ()assert.strictEqual(isAllowed(empty, '/anything'), true);const prefix = parseRobots('User-agent: OpenVibe\nDisallow: /\n', 'OpenVibeSources');assert.strictEqual(isAllowed(prefix, '/x'), true, 'a different token (even a prefix of ours) does not apply');++ // ADR 0007: a comma list of product tokens; a group naming either applies++ const both = 'User-agent: *\nDisallow: /all\n\nUser-agent: OpenVibeBot\nDisallow: /bot-only\n';++ assert.strictEqual(isAllowed(parseRobots(both, 'OpenVibeBot,OpenVibeSources'), '/bot-only'), false);++ assert.strictEqual(isAllowed(parseRobots(both, 'OpenVibeBot,OpenVibeSources'), '/all'), true, 'our group replaces *');++ assert.strictEqual(isAllowed(parseRobots('User-agent: OpenVibeSources\nDisallow: /\n', 'OpenVibeBot,OpenVibeSources'), '/x'), false);});t('address policy', () => {nnew file mode 100644@@@ -0,0 +1,53 @@++'use strict';++/** ADR 0007: adapters follow evidence/*.json exactly — ok → enabled crawl, structured-only → disabled, queued → review queue. */++const assert = require('assert');++const fs = require('fs');++const path = require('path');++const { validateSource } = require('../server/registry');++const { load } = require('../server/config');++const { suite } = require('./helpers');++++const t = suite('source-access');++const ROOT = path.join(__dirname, '..');++const evidence = fs.readdirSync(path.join(ROOT, 'evidence')).filter(f => f.endsWith('.json'))++ .map(f => JSON.parse(fs.readFileSync(path.join(ROOT, 'evidence', f), 'utf8')));++const adapter = (d) => {++ const p = path.join(ROOT, 'adapters', `${d}.json`);++ return fs.existsSync(p) ? JSON.parse(fs.readFileSync(p, 'utf8')) : null;++};++const queue = fs.readFileSync(path.join(ROOT, 'compliance', 'review-queue.yaml'), 'utf8');++++t('every evidence file carries a computed verdict', () => {++ assert.ok(evidence.length >= 29);++ for (const e of evidence) assert.ok(['ok', 'structured-only', 'queued'].includes(e.verdict), `${e.domain}: ${e.verdict}`);++});++++t('ok → an enabled crawl adapter; structured-only → a disabled one; queued → a review-queue row and no adapter', () => {++ for (const e of evidence) {++ const a = adapter(e.domain);++ if (e.verdict === 'queued') {++ assert.strictEqual(a, null, `${e.domain} is queued and has an adapter`);++ assert.ok(queue.includes(`- domain: ${e.domain}\n`), `${e.domain} missing from the review queue`);++ continue;++ }++ assert.ok(a, `${e.domain} (${e.verdict}) has no adapter`);++ assert.strictEqual(a.type, 'crawl');++ assert.strictEqual(a.enabled, e.verdict === 'ok', `${e.domain}: enabled must follow the verdict`);++ assert.ok(a.terms_note.includes(`evidence/${e.domain}.json`), `${e.domain}: terms_note cites its evidence`);++ assert.ok(a.crawl.require_structured, `${e.domain}: structured surfaces first`);++ validateSource(a);++ }++});++++t('no adapter exists without evidence', () => {++ const known = new Set(evidence.map(e => e.domain));++ for (const f of fs.readdirSync(path.join(ROOT, 'adapters'))) assert.ok(known.has(f.replace(/\.json$/, '')), `adapters/${f} has no evidence`);++});++++t('the crawler sends the ADR 0007 user agent and answers to both product tokens', () => {++ const { fetch } = load({});++ assert.strictEqual(fetch.userAgent, 'Mozilla/5.0 (compatible; OpenVibeBot/1.0; +https://openvibe.network/bot)');++ assert.deepStrictEqual(fetch.robotsAgent.split(','), ['OpenVibeBot', 'OpenVibeSources']);++});++++t.run();
@@ -48,8 +48,8 @@ VALKEY_PREFIX=
@@ -48,8 +48,8 @@ VALKEY_PREFIX=
#SOURCES_MAX_BACKOFF_MS=21600000
#SOURCES_MAX_BACKOFF_MS=21600000
# Fetching
# Fetching
-#SOURCES_USER_AGENT=OpenVibeSources/0.1.0 (+https://sources.openvibe.network; [email protected])
-#SOURCES_ROBOTS_AGENT=OpenVibeSources
+#SOURCES_USER_AGENT=Mozilla/5.0 (compatible; OpenVibeBot/1.0; +https://openvibe.network/bot)
+#SOURCES_ROBOTS_AGENT=OpenVibeBot,OpenVibeSources
#SOURCES_FETCH_TIMEOUT_MS=20000
#SOURCES_FETCH_TIMEOUT_MS=20000
#SOURCES_MAX_BYTES=5242880
#SOURCES_MAX_BYTES=5242880
#SOURCES_MAX_REDIRECTS=5
#SOURCES_MAX_REDIRECTS=5
new file mode 100644
new file mode 100644
@@ -0,0 +1,8 @@
@@ -0,0 +1,8 @@
+# OpenVibe.Sources: notes for agents
+
+Read README.md first. Source access:
+
+Access verdicts come from evidence/<domain>.json, written by scripts/probe.mjs.
+Probe first; if evidence exists, read it and do not re-derive it.
+docs/adr/0007-source-access.md is Accepted. Re-litigating it is a code comment.
+The fixed technique list in ADR 0007 is not up for discussion.
@@ -318,6 +318,53 @@ downloaded once by hand and parsed offline with these adapters (NASA 10 items, N
@@ -318,6 +318,53 @@ downloaded once by hand and parsed offline with these adapters (NASA 10 items, N
Steam 20, DealNews 50, SEC 200); that check is not part of the test suite and proves nothing about
Steam 20, DealNews 50, SEC 200); that check is not part of the test suite and proves nothing about
the terms.
the terms.
+## Source access (ADR 0007)
+
+Whether a site may be read, and how, is computed from evidence, never judged:
+[docs/adr/0007-source-access.md](docs/adr/0007-source-access.md). `node scripts/probe.mjs <domain>` (run from the
+production host, whose egress is the crawler's) writes `evidence/<domain>.json`: robots.txt, sitemap.xml,
+/products.json, llms.txt, /api/ and the home page, with statuses and challenge markers, and a verdict.
+`node scripts/adapters-from-evidence.mjs` turns evidence into adapters (`ok` → an enabled crawl source,
+`structured-only` → a disabled one, Lane 1) or a row in [compliance/review-queue.yaml](compliance/review-queue.yaml)
+(`queued`); `node scripts/register-adapters.js` brings the registry to the adapters. The crawler identifies itself as
+`Mozilla/5.0 (compatible; OpenVibeBot/1.0; +https://openvibe.network/bot)` and obeys robots.txt groups, robots meta
+and X-Robots-Tag for either token, `OpenVibeBot` or `OpenVibeSources`. The crawl type itself:
+[docs/adr/0006-crawl-source-type.md](docs/adr/0006-crawl-source-type.md).
+
+Probed 2026-10-28 (`c` = challenge page):
+
+| Domain | Verdict | Statuses | Lane | Adapter |
+|---|---|---|---|---|
+| adorama.com | structured-only | `/robots.txt:200 /sitemap.xml:404c /products.json:403c /llms.txt:403c /api/:403c /:403c` | Lane 1 (feeds, affiliate, API, opt-in); adapter disabled | [adapters/adorama.com.json](adapters/adorama.com.json) |
+| allbirds.com | ok | `/robots.txt:200 /sitemap.xml:200 /products.json:200 /llms.txt:200 /api/:404 /:200` | crawl adapter, enabled | [adapters/allbirds.com.json](adapters/allbirds.com.json) |
+| anker.com | ok | `/robots.txt:200 /sitemap.xml:200 /products.json:404 /llms.txt:200 /api/:404 /:200` | crawl adapter, enabled | [adapters/anker.com.json](adapters/anker.com.json) |
+| backcountry.com | queued | `/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403` | review queue; probed again after 24 h | [compliance/review-queue.yaml](compliance/review-queue.yaml) |
+| bestbuy.com | ok | `/robots.txt:200 /sitemap.xml:200 /products.json:404 /llms.txt:404 /api/:404 /:200` | crawl adapter, enabled | [adapters/bestbuy.com.json](adapters/bestbuy.com.json) |
+| bhphotovideo.com | structured-only | `/robots.txt:200 /sitemap.xml:403c /products.json:403c /llms.txt:403c /api/:403c /:403c` | Lane 1 (feeds, affiliate, API, opt-in); adapter disabled | [adapters/bhphotovideo.com.json](adapters/bhphotovideo.com.json) |
+| brooksrunning.com | queued | `/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403` | review queue; probed again after 24 h | [compliance/review-queue.yaml](compliance/review-queue.yaml) |
+| converse.com | queued | `/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403` | review queue; probed again after 24 h | [compliance/review-queue.yaml](compliance/review-queue.yaml) |
+| dsw.com | ok | `/robots.txt:200 /sitemap.xml:404 /products.json:404 /llms.txt:404 /api/:200 /:200` | crawl adapter, enabled | [adapters/dsw.com.json](adapters/dsw.com.json) |
+| footlocker.com | ok | `/robots.txt:200 /sitemap.xml:200 /products.json:200 /llms.txt:200 /api/:403 /:200` | crawl adapter, enabled | [adapters/footlocker.com.json](adapters/footlocker.com.json) |
+| frame.work | queued | `/robots.txt:403c /sitemap.xml:403c /products.json:403c /llms.txt:403c /api/:403c /:403c` | review queue; probed again after 24 h | [compliance/review-queue.yaml](compliance/review-queue.yaml) |
+| gog.com | ok | `/robots.txt:200 /sitemap.xml:404 /products.json:404 /llms.txt:404 /api/:404 /:200` | crawl adapter, enabled | [adapters/gog.com.json](adapters/gog.com.json) |
+| hoka.com | structured-only | `/robots.txt:200 /sitemap.xml:404 /products.json:410 /llms.txt:410 /api/:500 /:0` | Lane 1 (feeds, affiliate, API, opt-in); adapter disabled | [adapters/hoka.com.json](adapters/hoka.com.json) |
+| homedepot.com | ok | `/robots.txt:200 /sitemap.xml:404 /products.json:404 /llms.txt:404 /api/:404 /:200` | crawl adapter, enabled | [adapters/homedepot.com.json](adapters/homedepot.com.json) |
+| humblebundle.com | ok | `/robots.txt:200 /sitemap.xml:200 /products.json:404 /llms.txt:404 /api/:404 /:200` | crawl adapter, enabled | [adapters/humblebundle.com.json](adapters/humblebundle.com.json) |
+| ikea.com | queued | `/robots.txt:403c /sitemap.xml:403c /products.json:403c /llms.txt:403c /api/:403c /:403c` | review queue; probed again after 24 h | [compliance/review-queue.yaml](compliance/review-queue.yaml) |
+| microcenter.com | queued | `/robots.txt:403c /sitemap.xml:403c /products.json:403c /llms.txt:403c /api/:403c /:403c` | review queue; probed again after 24 h | [compliance/review-queue.yaml](compliance/review-queue.yaml) |
+| monoprice.com | structured-only | `/robots.txt:200 /sitemap.xml:403c /products.json:403c /llms.txt:404 /api/:403c /:403c` | Lane 1 (feeds, affiliate, API, opt-in); adapter disabled | [adapters/monoprice.com.json](adapters/monoprice.com.json) |
+| newbalance.com | queued | `/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403` | review queue; probed again after 24 h | [compliance/review-queue.yaml](compliance/review-queue.yaml) |
+| newegg.com | queued | `/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403` | review queue; probed again after 24 h | [compliance/review-queue.yaml](compliance/review-queue.yaml) |
+| on.com | ok | `/robots.txt:200 /sitemap.xml:200 /products.json:200 /llms.txt:200 /api/:403 /:200` | crawl adapter, enabled | [adapters/on.com.json](adapters/on.com.json) |
+| patagonia.com | ok | `/robots.txt:200 /sitemap.xml:200 /products.json:200 /llms.txt:200 /api/:200 /:200` | crawl adapter, enabled | [adapters/patagonia.com.json](adapters/patagonia.com.json) |
+| rei.com | queued | `/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403` | review queue; probed again after 24 h | [compliance/review-queue.yaml](compliance/review-queue.yaml) |
+| runningwarehouse.com | queued | `/robots.txt:0 /sitemap.xml:0 /products.json:0 /llms.txt:0 /api/:0 /:0` | review queue; probed again after 24 h | [compliance/review-queue.yaml](compliance/review-queue.yaml) |
+| saucony.com | structured-only | `/robots.txt:200 /sitemap.xml:404 /products.json:404c /llms.txt:404c /api/:404c /:200c` | Lane 1 (feeds, affiliate, API, opt-in); adapter disabled | [adapters/saucony.com.json](adapters/saucony.com.json) |
+| vans.com | queued | `/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403` | review queue; probed again after 24 h | [compliance/review-queue.yaml](compliance/review-queue.yaml) |
+| wayfair.com | structured-only | `/robots.txt:200 /sitemap.xml:404 /products.json:404 /llms.txt:429 /api/:429 /:429` | Lane 1 (feeds, affiliate, API, opt-in); adapter disabled | [adapters/wayfair.com.json](adapters/wayfair.com.json) |
+| woot.com | ok | `/robots.txt:200 /sitemap.xml:200 /products.json:404 /llms.txt:404 /api/:404 /:200` | crawl adapter, enabled | [adapters/woot.com.json](adapters/woot.com.json) |
+| zappos.com | ok | `/robots.txt:200 /sitemap.xml:404 /products.json:404 /llms.txt:404 /api/:200 /:200` | crawl adapter, enabled | [adapters/zappos.com.json](adapters/zappos.com.json) |
+
## Processing jobs: who does what (roadmap WS-O task 7, decided 2026-09-26)
## Processing jobs: who does what (roadmap WS-O task 7, decided 2026-09-26)
| Job | Where | How |
| Job | Where | How |
new file mode 100644
new file mode 100644
@@ -0,0 +1,26 @@
@@ -0,0 +1,26 @@
+{
+ "key": "crawl-adorama-com",
+ "name": "adorama.com (crawl)",
+ "type": "crawl",
+ "category": "deals",
+ "homepage_url": "https://adorama.com/",
+ "endpoints": [
+ "https://adorama.com/"
+ ],
+ "crawl": {
+ "sitemaps": true,
+ "follow_links": true,
+ "max_depth": 2,
+ "require_structured": true,
+ "max_pages": 2000,
+ "pages_per_run": 50,
+ "recrawl_sec": 86400
+ },
+ "min_interval_ms": 5000,
+ "poll_interval_sec": 3600,
+ "robots_note": "robots.txt answered 200 at 2026-10-28T10:16:54.379Z (evidence/adorama.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",
+ "terms_note": "Access verdict \"structured-only\" per ADR 0007, computed from evidence/adorama.com.json (2026-10-28T10:16:54.379Z). Structured surfaces answering: none. Lane 1 only: feeds, affiliate programs, official APIs or merchant opt-in; this crawl adapter stays disabled.",
+ "enabled": false,
+ "review_required": true,
+ "default_indexability": "noindex"
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,25 @@
@@ -0,0 +1,25 @@
+{
+ "key": "crawl-allbirds-com",
+ "name": "allbirds.com (crawl)",
+ "type": "crawl",
+ "category": "deals",
+ "homepage_url": "https://allbirds.com/",
+ "endpoints": [
+ "https://allbirds.com/"
+ ],
+ "crawl": {
+ "sitemaps": true,
+ "follow_links": false,
+ "require_structured": true,
+ "max_pages": 2000,
+ "pages_per_run": 50,
+ "recrawl_sec": 86400
+ },
+ "min_interval_ms": 5000,
+ "poll_interval_sec": 3600,
+ "robots_note": "robots.txt answered 200 at 2026-10-28T10:16:56.095Z (evidence/allbirds.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",
+ "terms_note": "Access verdict \"ok\" per ADR 0007, computed from evidence/allbirds.com.json (2026-10-28T10:16:56.095Z). Structured surfaces answering: /sitemap.xml, /products.json, /llms.txt.",
+ "enabled": true,
+ "review_required": true,
+ "default_indexability": "noindex"
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,25 @@
@@ -0,0 +1,25 @@
+{
+ "key": "crawl-anker-com",
+ "name": "anker.com (crawl)",
+ "type": "crawl",
+ "category": "deals",
+ "homepage_url": "https://anker.com/",
+ "endpoints": [
+ "https://anker.com/"
+ ],
+ "crawl": {
+ "sitemaps": true,
+ "follow_links": false,
+ "require_structured": true,
+ "max_pages": 2000,
+ "pages_per_run": 50,
+ "recrawl_sec": 86400
+ },
+ "min_interval_ms": 5000,
+ "poll_interval_sec": 3600,
+ "robots_note": "robots.txt answered 200 at 2026-10-28T10:17:00.155Z (evidence/anker.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",
+ "terms_note": "Access verdict \"ok\" per ADR 0007, computed from evidence/anker.com.json (2026-10-28T10:17:00.155Z). Structured surfaces answering: /sitemap.xml, /llms.txt.",
+ "enabled": true,
+ "review_required": true,
+ "default_indexability": "noindex"
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,25 @@
@@ -0,0 +1,25 @@
+{
+ "key": "crawl-bestbuy-com",
+ "name": "bestbuy.com (crawl)",
+ "type": "crawl",
+ "category": "deals",
+ "homepage_url": "https://bestbuy.com/",
+ "endpoints": [
+ "https://bestbuy.com/"
+ ],
+ "crawl": {
+ "sitemaps": true,
+ "follow_links": false,
+ "require_structured": true,
+ "max_pages": 2000,
+ "pages_per_run": 50,
+ "recrawl_sec": 86400
+ },
+ "min_interval_ms": 5000,
+ "poll_interval_sec": 3600,
+ "robots_note": "robots.txt answered 200 at 2026-10-28T10:17:52.971Z (evidence/bestbuy.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",
+ "terms_note": "Access verdict \"ok\" per ADR 0007, computed from evidence/bestbuy.com.json (2026-10-28T10:17:52.971Z). Structured surfaces answering: /sitemap.xml.",
+ "enabled": true,
+ "review_required": true,
+ "default_indexability": "noindex"
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,26 @@
@@ -0,0 +1,26 @@
+{
+ "key": "crawl-bhphotovideo-com",
+ "name": "bhphotovideo.com (crawl)",
+ "type": "crawl",
+ "category": "deals",
+ "homepage_url": "https://bhphotovideo.com/",
+ "endpoints": [
+ "https://bhphotovideo.com/"
+ ],
+ "crawl": {
+ "sitemaps": true,
+ "follow_links": true,
+ "max_depth": 2,
+ "require_structured": true,
+ "max_pages": 2000,
+ "pages_per_run": 50,
+ "recrawl_sec": 86400
+ },
+ "min_interval_ms": 5000,
+ "poll_interval_sec": 3600,
+ "robots_note": "robots.txt answered 200 at 2026-10-28T10:17:53.766Z (evidence/bhphotovideo.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",
+ "terms_note": "Access verdict \"structured-only\" per ADR 0007, computed from evidence/bhphotovideo.com.json (2026-10-28T10:17:53.766Z). Structured surfaces answering: none. Lane 1 only: feeds, affiliate programs, official APIs or merchant opt-in; this crawl adapter stays disabled.",
+ "enabled": false,
+ "review_required": true,
+ "default_indexability": "noindex"
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,26 @@
@@ -0,0 +1,26 @@
+{
+ "key": "crawl-dsw-com",
+ "name": "dsw.com (crawl)",
+ "type": "crawl",
+ "category": "deals",
+ "homepage_url": "https://dsw.com/",
+ "endpoints": [
+ "https://dsw.com/"
+ ],
+ "crawl": {
+ "sitemaps": true,
+ "follow_links": true,
+ "max_depth": 2,
+ "require_structured": true,
+ "max_pages": 2000,
+ "pages_per_run": 50,
+ "recrawl_sec": 86400
+ },
+ "min_interval_ms": 5000,
+ "poll_interval_sec": 3600,
+ "robots_note": "robots.txt answered 200 at 2026-10-28T10:18:02.761Z (evidence/dsw.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",
+ "terms_note": "Access verdict \"ok\" per ADR 0007, computed from evidence/dsw.com.json (2026-10-28T10:18:02.761Z). Structured surfaces answering: /api/.",
+ "enabled": true,
+ "review_required": true,
+ "default_indexability": "noindex"
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,26 @@
@@ -0,0 +1,26 @@
+{
+ "key": "crawl-footlocker-com",
+ "name": "footlocker.com (crawl)",
+ "type": "crawl",
+ "category": "deals",
+ "homepage_url": "https://footlocker.com/",
+ "endpoints": [
+ "https://footlocker.com/"
+ ],
+ "crawl": {
+ "sitemaps": true,
+ "follow_links": false,
+ "require_structured": true,
+ "max_pages": 2000,
+ "pages_per_run": 50,
+ "recrawl_sec": 86400
+ },
+ "min_interval_ms": 5000,
+ "poll_interval_sec": 3600,
+ "robots_note": "robots.txt answered 200 at 2026-10-28T10:18:04.235Z (evidence/footlocker.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",
+ "terms_note": "Access verdict \"ok\" per ADR 0007, computed from evidence/footlocker.com.json (2026-10-28T10:18:04.235Z). Structured surfaces answering: /sitemap.xml, /products.json, /llms.txt.",
+ "license_note": "Terms of use on file: https://help.footlocker.com/hc/en-us/articles/360034292814-Terms-of-Use-Comprehensive: \"You may not without the prior written permission of Foot Locker, use any computer code, data mining software, \"robot\", \"bot\", \"spider\", \"scraper\" or other automatic device, or program, algorithm or methodology having similar processes or functionality, or any manual process, to monitor or copy any o\"",
+ "enabled": true,
+ "review_required": true,
+ "default_indexability": "noindex"
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,26 @@
@@ -0,0 +1,26 @@
+{
+ "key": "crawl-gog-com",
+ "name": "gog.com (crawl)",
+ "type": "crawl",
+ "category": "deals",
+ "homepage_url": "https://gog.com/",
+ "endpoints": [
+ "https://gog.com/"
+ ],
+ "crawl": {
+ "sitemaps": true,
+ "follow_links": true,
+ "max_depth": 2,
+ "require_structured": true,
+ "max_pages": 2000,
+ "pages_per_run": 50,
+ "recrawl_sec": 86400
+ },
+ "min_interval_ms": 5000,
+ "poll_interval_sec": 3600,
+ "robots_note": "robots.txt answered 200 at 2026-10-28T10:18:07.984Z (evidence/gog.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",
+ "terms_note": "Access verdict \"ok\" per ADR 0007, computed from evidence/gog.com.json (2026-10-28T10:18:07.984Z). Structured surfaces answering: none.",
+ "enabled": true,
+ "review_required": true,
+ "default_indexability": "noindex"
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,26 @@
@@ -0,0 +1,26 @@
+{
+ "key": "crawl-hoka-com",
+ "name": "hoka.com (crawl)",
+ "type": "crawl",
+ "category": "deals",
+ "homepage_url": "https://hoka.com/",
+ "endpoints": [
+ "https://hoka.com/"
+ ],
+ "crawl": {
+ "sitemaps": true,
+ "follow_links": true,
+ "max_depth": 2,
+ "require_structured": true,
+ "max_pages": 2000,
+ "pages_per_run": 50,
+ "recrawl_sec": 86400
+ },
+ "min_interval_ms": 5000,
+ "poll_interval_sec": 3600,
+ "robots_note": "robots.txt answered 200 at 2026-10-28T10:18:09.453Z (evidence/hoka.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",
+ "terms_note": "Access verdict \"structured-only\" per ADR 0007, computed from evidence/hoka.com.json (2026-10-28T10:18:09.453Z). Structured surfaces answering: none. Lane 1 only: feeds, affiliate programs, official APIs or merchant opt-in; this crawl adapter stays disabled.",
+ "enabled": false,
+ "review_required": true,
+ "default_indexability": "noindex"
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,26 @@
@@ -0,0 +1,26 @@
+{
+ "key": "crawl-homedepot-com",
+ "name": "homedepot.com (crawl)",
+ "type": "crawl",
+ "category": "deals",
+ "homepage_url": "https://homedepot.com/",
+ "endpoints": [
+ "https://homedepot.com/"
+ ],
+ "crawl": {
+ "sitemaps": true,
+ "follow_links": true,
+ "max_depth": 2,
+ "require_structured": true,
+ "max_pages": 2000,
+ "pages_per_run": 50,
+ "recrawl_sec": 86400
+ },
+ "min_interval_ms": 5000,
+ "poll_interval_sec": 3600,
+ "robots_note": "robots.txt answered 200 at 2026-10-28T10:18:10.727Z (evidence/homedepot.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",
+ "terms_note": "Access verdict \"ok\" per ADR 0007, computed from evidence/homedepot.com.json (2026-10-28T10:18:10.727Z). Structured surfaces answering: none.",
+ "enabled": true,
+ "review_required": true,
+ "default_indexability": "noindex"
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,26 @@
@@ -0,0 +1,26 @@
+{
+ "key": "crawl-humblebundle-com",
+ "name": "humblebundle.com (crawl)",
+ "type": "crawl",
+ "category": "deals",
+ "homepage_url": "https://humblebundle.com/",
+ "endpoints": [
+ "https://humblebundle.com/"
+ ],
+ "crawl": {
+ "sitemaps": true,
+ "follow_links": false,
+ "require_structured": true,
+ "max_pages": 2000,
+ "pages_per_run": 50,
+ "recrawl_sec": 86400
+ },
+ "min_interval_ms": 5000,
+ "poll_interval_sec": 3600,
+ "robots_note": "robots.txt answered 200 at 2026-10-28T10:18:13.614Z (evidence/humblebundle.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",
+ "terms_note": "Access verdict \"ok\" per ADR 0007, computed from evidence/humblebundle.com.json (2026-10-28T10:18:13.614Z). Structured surfaces answering: /sitemap.xml.",
+ "license_note": "Terms of use on file: https://www.humblebundle.com/terms: \"without limitation \"robots,\" \"spiders,\" \"offline readers,\" etc.\"",
+ "enabled": true,
+ "review_required": true,
+ "default_indexability": "noindex"
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,26 @@
@@ -0,0 +1,26 @@
+{
+ "key": "crawl-monoprice-com",
+ "name": "monoprice.com (crawl)",
+ "type": "crawl",
+ "category": "deals",
+ "homepage_url": "https://monoprice.com/",
+ "endpoints": [
+ "https://monoprice.com/"
+ ],
+ "crawl": {
+ "sitemaps": true,
+ "follow_links": true,
+ "max_depth": 2,
+ "require_structured": true,
+ "max_pages": 2000,
+ "pages_per_run": 50,
+ "recrawl_sec": 86400
+ },
+ "min_interval_ms": 5000,
+ "poll_interval_sec": 3600,
+ "robots_note": "robots.txt answered 200 at 2026-10-28T10:18:15.581Z (evidence/monoprice.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",
+ "terms_note": "Access verdict \"structured-only\" per ADR 0007, computed from evidence/monoprice.com.json (2026-10-28T10:18:15.581Z). Structured surfaces answering: none. Lane 1 only: feeds, affiliate programs, official APIs or merchant opt-in; this crawl adapter stays disabled.",
+ "enabled": false,
+ "review_required": true,
+ "default_indexability": "noindex"
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,25 @@
@@ -0,0 +1,25 @@
+{
+ "key": "crawl-on-com",
+ "name": "on.com (crawl)",
+ "type": "crawl",
+ "category": "deals",
+ "homepage_url": "https://on.com/",
+ "endpoints": [
+ "https://on.com/"
+ ],
+ "crawl": {
+ "sitemaps": true,
+ "follow_links": false,
+ "require_structured": true,
+ "max_pages": 2000,
+ "pages_per_run": 50,
+ "recrawl_sec": 86400
+ },
+ "min_interval_ms": 5000,
+ "poll_interval_sec": 3600,
+ "robots_note": "robots.txt answered 200 at 2026-10-28T10:18:19.026Z (evidence/on.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",
+ "terms_note": "Access verdict \"ok\" per ADR 0007, computed from evidence/on.com.json (2026-10-28T10:18:19.026Z). Structured surfaces answering: /sitemap.xml, /products.json, /llms.txt.",
+ "enabled": true,
+ "review_required": true,
+ "default_indexability": "noindex"
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,25 @@
@@ -0,0 +1,25 @@
+{
+ "key": "crawl-patagonia-com",
+ "name": "patagonia.com (crawl)",
+ "type": "crawl",
+ "category": "deals",
+ "homepage_url": "https://patagonia.com/",
+ "endpoints": [
+ "https://patagonia.com/"
+ ],
+ "crawl": {
+ "sitemaps": true,
+ "follow_links": false,
+ "require_structured": true,
+ "max_pages": 2000,
+ "pages_per_run": 50,
+ "recrawl_sec": 86400
+ },
+ "min_interval_ms": 5000,
+ "poll_interval_sec": 3600,
+ "robots_note": "robots.txt answered 200 at 2026-10-28T10:18:19.826Z (evidence/patagonia.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",
+ "terms_note": "Access verdict \"ok\" per ADR 0007, computed from evidence/patagonia.com.json (2026-10-28T10:18:19.826Z). Structured surfaces answering: /sitemap.xml, /products.json, /llms.txt, /api/.",
+ "enabled": true,
+ "review_required": true,
+ "default_indexability": "noindex"
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,26 @@
@@ -0,0 +1,26 @@
+{
+ "key": "crawl-saucony-com",
+ "name": "saucony.com (crawl)",
+ "type": "crawl",
+ "category": "deals",
+ "homepage_url": "https://saucony.com/",
+ "endpoints": [
+ "https://saucony.com/"
+ ],
+ "crawl": {
+ "sitemaps": true,
+ "follow_links": true,
+ "max_depth": 2,
+ "require_structured": true,
+ "max_pages": 2000,
+ "pages_per_run": 50,
+ "recrawl_sec": 86400
+ },
+ "min_interval_ms": 5000,
+ "poll_interval_sec": 3600,
+ "robots_note": "robots.txt answered 200 at 2026-10-28T10:18:24.130Z (evidence/saucony.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",
+ "terms_note": "Access verdict \"structured-only\" per ADR 0007, computed from evidence/saucony.com.json (2026-10-28T10:18:24.130Z). Structured surfaces answering: none. Lane 1 only: feeds, affiliate programs, official APIs or merchant opt-in; this crawl adapter stays disabled.",
+ "enabled": false,
+ "review_required": true,
+ "default_indexability": "noindex"
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,26 @@
@@ -0,0 +1,26 @@
+{
+ "key": "crawl-wayfair-com",
+ "name": "wayfair.com (crawl)",
+ "type": "crawl",
+ "category": "deals",
+ "homepage_url": "https://wayfair.com/",
+ "endpoints": [
+ "https://wayfair.com/"
+ ],
+ "crawl": {
+ "sitemaps": true,
+ "follow_links": true,
+ "max_depth": 2,
+ "require_structured": true,
+ "max_pages": 2000,
+ "pages_per_run": 50,
+ "recrawl_sec": 86400
+ },
+ "min_interval_ms": 5000,
+ "poll_interval_sec": 3600,
+ "robots_note": "robots.txt answered 200 at 2026-10-28T10:18:27.350Z (evidence/wayfair.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",
+ "terms_note": "Access verdict \"structured-only\" per ADR 0007, computed from evidence/wayfair.com.json (2026-10-28T10:18:27.350Z). Structured surfaces answering: none. Lane 1 only: feeds, affiliate programs, official APIs or merchant opt-in; this crawl adapter stays disabled.",
+ "enabled": false,
+ "review_required": true,
+ "default_indexability": "noindex"
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,26 @@
@@ -0,0 +1,26 @@
+{
+ "key": "crawl-woot-com",
+ "name": "woot.com (crawl)",
+ "type": "crawl",
+ "category": "deals",
+ "homepage_url": "https://woot.com/",
+ "endpoints": [
+ "https://woot.com/"
+ ],
+ "crawl": {
+ "sitemaps": true,
+ "follow_links": false,
+ "require_structured": true,
+ "max_pages": 2000,
+ "pages_per_run": 50,
+ "recrawl_sec": 86400
+ },
+ "min_interval_ms": 5000,
+ "poll_interval_sec": 3600,
+ "robots_note": "robots.txt answered 200 at 2026-10-28T10:18:29.235Z (evidence/woot.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",
+ "terms_note": "Access verdict \"ok\" per ADR 0007, computed from evidence/woot.com.json (2026-10-28T10:18:29.235Z). Structured surfaces answering: /sitemap.xml.",
+ "license_note": "Terms of use on file: https://www.woot.com/terms: \"This license does not include any resale or commercial use of any Woot Service, or its contents; any collection and use of any product listings, descriptions, or prices; any derivative use of any Woot Service or its contents; any downloading or copying of account information for the benefit of anoth\"",
+ "enabled": true,
+ "review_required": true,
+ "default_indexability": "noindex"
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,27 @@
@@ -0,0 +1,27 @@
+{
+ "key": "crawl-zappos-com",
+ "name": "zappos.com (crawl)",
+ "type": "crawl",
+ "category": "deals",
+ "homepage_url": "https://zappos.com/",
+ "endpoints": [
+ "https://zappos.com/"
+ ],
+ "crawl": {
+ "sitemaps": true,
+ "follow_links": true,
+ "max_depth": 2,
+ "require_structured": true,
+ "max_pages": 2000,
+ "pages_per_run": 50,
+ "recrawl_sec": 86400
+ },
+ "min_interval_ms": 5000,
+ "poll_interval_sec": 3600,
+ "robots_note": "robots.txt answered 200 at 2026-10-28T10:18:30.558Z (evidence/zappos.com.json); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).",
+ "terms_note": "Access verdict \"ok\" per ADR 0007, computed from evidence/zappos.com.json (2026-10-28T10:18:30.558Z). Structured surfaces answering: /api/.",
+ "license_note": "Terms of use on file: https://zappos.com/terms-of-use: \"The license granted herein does not include: any resale or commercial use of any Zappos Service, or its contents; any collection or use of any product listings, descriptions, images, or prices; any derivative use of any Zappos Service or its contents; any downloading, copying, or other use of accoun\"",
+ "enabled": true,
+ "review_required": true,
+ "default_indexability": "noindex"
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,69 @@
@@ -0,0 +1,69 @@
+# ADR 0007 review queue: sources with nothing under 400. A second probe ≥ 24 h after the first decides; until then
+# no adapter exists. Written by scripts/adapters-from-evidence.mjs from evidence/*.json.
+queue:
+ - domain: backcountry.com
+ verdict: queued
+ attempts: 1
+ first_probe: 2026-10-28T10:17:00.621Z
+ next_probe_after: 2026-10-29T10:17:00.621Z
+ statuses: "/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403"
+ - domain: brooksrunning.com
+ verdict: queued
+ attempts: 1
+ first_probe: 2026-10-28T10:17:57.626Z
+ next_probe_after: 2026-10-29T10:17:57.626Z
+ statuses: "/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403"
+ - domain: converse.com
+ verdict: queued
+ attempts: 1
+ first_probe: 2026-10-28T10:18:00.177Z
+ next_probe_after: 2026-10-29T10:18:00.177Z
+ statuses: "/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403"
+ - domain: frame.work
+ verdict: queued
+ attempts: 1
+ first_probe: 2026-10-28T10:18:04.784Z
+ next_probe_after: 2026-10-29T10:18:04.784Z
+ statuses: "/robots.txt:403c /sitemap.xml:403c /products.json:403c /llms.txt:403c /api/:403c /:403c"
+ - domain: ikea.com
+ verdict: queued
+ attempts: 1
+ first_probe: 2026-10-28T10:18:14.069Z
+ next_probe_after: 2026-10-29T10:18:14.069Z
+ statuses: "/robots.txt:403c /sitemap.xml:403c /products.json:403c /llms.txt:403c /api/:403c /:403c"
+ - domain: microcenter.com
+ verdict: queued
+ attempts: 1
+ first_probe: 2026-10-28T10:18:14.764Z
+ next_probe_after: 2026-10-29T10:18:14.764Z
+ statuses: "/robots.txt:403c /sitemap.xml:403c /products.json:403c /llms.txt:403c /api/:403c /:403c"
+ - domain: newbalance.com
+ verdict: queued
+ attempts: 1
+ first_probe: 2026-10-28T10:18:16.312Z
+ next_probe_after: 2026-10-29T10:18:16.312Z
+ statuses: "/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403"
+ - domain: newegg.com
+ verdict: queued
+ attempts: 1
+ first_probe: 2026-10-28T10:18:16.799Z
+ next_probe_after: 2026-10-29T10:18:16.799Z
+ statuses: "/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403"
+ - domain: rei.com
+ verdict: queued
+ attempts: 1
+ first_probe: 2026-10-28T10:18:20.825Z
+ next_probe_after: 2026-10-29T10:18:20.825Z
+ statuses: "/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403"
+ - domain: runningwarehouse.com
+ verdict: queued
+ attempts: 1
+ first_probe: 2026-10-28T10:18:22.204Z
+ next_probe_after: 2026-10-29T10:18:22.204Z
+ statuses: "/robots.txt:0 /sitemap.xml:0 /products.json:0 /llms.txt:0 /api/:0 /:0"
+ - domain: vans.com
+ verdict: queued
+ attempts: 1
+ first_probe: 2026-10-28T10:18:25.348Z
+ next_probe_after: 2026-10-29T10:18:25.348Z
+ statuses: "/robots.txt:403 /sitemap.xml:403 /products.json:403 /llms.txt:403 /api/:403 /:403"
new file mode 100644
new file mode 100644
@@ -0,0 +1,36 @@
@@ -0,0 +1,36 @@
+# 0006 — The crawl source type
+
+Status: Accepted · 2026-10-28 · Contracts 0.224.0 (`sources.source@1`, `sources.source-write-request@1`) · Sources#128
+
+## Context
+Feeds and APIs cover the sources that publish one. Many sites publish their catalogue only as pages with
+schema.org JSON-LD and a sitemap. Deals, Shoes and the open index need those pages without a bespoke
+adapter per site.
+
+## Decision
+A source of type `crawl` walks one site from its endpoints (the seeds).
+
+- **Frontier.** `crawl_frontier` holds every known URL of the source (page or sitemap), its depth, how it was
+ found, its next fetch time, validators (ETag, Last-Modified), body hash, item count and failures.
+- **Discovery.** Sitemaps named in robots.txt (else `/sitemap.xml`; `.xml.gz` gunzipped; indexes followed, at
+ most 1000 sitemaps and 50 000 URLs each). With `follow_links`, same-host links up to `max_depth`.
+ `include`/`exclude` are regular expressions over path and query. A crawl never leaves its seeds' hosts
+ (or their `www` twins).
+- **Budget.** `max_pages` (default 5000) known pages, `pages_per_run` (100) per run, at most
+ `SOURCES_MAX_CONCURRENT_CRAWLS` (2) crawls at once. A run with pages still due asks for another run in 60 s.
+- **Recrawl.** `recrawl_sec` (default 86 400, 3600–2 592 000); an unchanged page waits ×1.5 longer each
+ time, up to ×4. A sitemap `lastmod` newer than the last fetch brings a page forward.
+- **Items.** Pages are parsed with `jsonld@2`; `require_structured` (default true) keeps only pages that
+ publish JSON-LD. Without it, a `page@1` item comes from the page's own search metadata.
+- **Respect.** robots.txt per request and per redirect hop; `noindex`/`nofollow` from the robots meta, a
+ meta named for our product tokens, and X-Robots-Tag; canonical links win over the fetched URL.
+- **Retirement.** A page answering 404 twice, or 410 once, retires its items (`items.retireRun`).
+ `retire_after_sec` is refused on a crawl source.
+- **Failure.** A page's failure is the page's. The source fails only when robots.txt is unreachable, the
+ site answers 429 or 503 with Retry-After, or 5 requests in a row go unanswered.
+- **API.** `GET /api/v1/sources/:key/crawl` reports the frontier; a manual fetch of a crawl source answers
+ 202 and runs on the scheduler.
+
+## Consequences
+Whether a site may be crawled at all, and with which technique, is not this ADR's question: ADR 0007
+computes it from probe evidence. This ADR only says how an enabled crawl source behaves.
new file mode 100644
new file mode 100644
@@ -0,0 +1,32 @@
@@ -0,0 +1,32 @@
+# 0007 — Source access verdicts are computed, not judged
+
+Status: Accepted · 2026-10-28 · Owner: operator
+
+## Decision
+A source's access verdict is a function of observed HTTP evidence. No agent
+decides whether a domain is "blocked" or whether a technique is "allowed".
+
+| Evidence | Verdict | Action |
+|---------------------------------------|----------------|---------------------|
+| structured surface returns <400 | ok | adapter, enabled |
+| page fetch <400, no structured surface| ok | adapter, enabled |
+| structured surface only, page 403/429/challenge | structured-only | Lane 1 only |
+| nothing under 400 after 2 attempts ≥24h apart | queued | review-queue row |
+
+`blocked` is not a verdict. It is prose.
+
+## Fixed techniques (decided here, once)
+- UA: `Mozilla/5.0 (compatible; OpenVibeBot/1.0; +https://openvibe.network/bot)`
+ — the compatible-form string is the standards-blessed way to be both honest
+ and un-challenged. Not a disguise; RFC 9309 predates this argument.
+- Rendering: Playwright, Patchright build. A browser engine rendering a public
+ page is a browser, not a tactic.
+- Structured surfaces before DOM, always: sitemap.xml, /products.json,
+ __NEXT_DATA__, /api/, Algolia, RSS/JSON, llms.txt, JSON-LD.
+- Egress rotation: only behind OPENVIBE_EGRESS_ROTATION=1, default off.
+- Out of scope, permanently: CAPTCHA solving, login-walled content,
+ cookie-wall circumvention.
+
+## Consequence
+Anyone revisiting this argues with the table, not with the agent. New evidence
+amends the table in a new ADR.
new file mode 100644
new file mode 100644
@@ -0,0 +1,43 @@
@@ -0,0 +1,43 @@
+{
+ "domain": "adorama.com",
+ "at": "2026-10-28T10:16:54.379Z",
+ "attempted": [
+ {
+ "url": "https://adorama.com/robots.txt",
+ "status": 200,
+ "challenged": false,
+ "bytes": 5529
+ },
+ {
+ "url": "https://adorama.com/sitemap.xml",
+ "status": 404,
+ "challenged": true,
+ "bytes": 105248
+ },
+ {
+ "url": "https://adorama.com/products.json",
+ "status": 403,
+ "challenged": true,
+ "bytes": 771
+ },
+ {
+ "url": "https://adorama.com/llms.txt",
+ "status": 403,
+ "challenged": true,
+ "bytes": 771
+ },
+ {
+ "url": "https://adorama.com/api/",
+ "status": 403,
+ "challenged": true,
+ "bytes": 771
+ },
+ {
+ "url": "https://adorama.com/",
+ "status": 403,
+ "challenged": true,
+ "bytes": 771
+ }
+ ],
+ "verdict": "structured-only"
+}
\ No newline at end of file
\ No newline at end of file
new file mode 100644
new file mode 100644
@@ -0,0 +1,43 @@
@@ -0,0 +1,43 @@
+{
+ "domain": "allbirds.com",
+ "at": "2026-10-28T10:16:56.095Z",
+ "attempted": [
+ {
+ "url": "https://allbirds.com/robots.txt",
+ "status": 200,
+ "challenged": false,
+ "bytes": 2285
+ },
+ {
+ "url": "https://allbirds.com/sitemap.xml",
+ "status": 200,
+ "challenged": false,
+ "bytes": 1186
+ },
+ {
+ "url": "https://allbirds.com/products.json",
+ "status": 200,
+ "challenged": false,
+ "bytes": 203264
+ },
+ {
+ "url": "https://allbirds.com/llms.txt",
+ "status": 200,
+ "challenged": false,
+ "bytes": 5767
+ },
+ {
+ "url": "https://allbirds.com/api/",
+ "status": 404,
+ "challenged": false,
+ "bytes": 423147
+ },
+ {
+ "url": "https://allbirds.com/",
+ "status": 200,
+ "challenged": false,
+ "bytes": 694402
+ }
+ ],
+ "verdict": "ok"
+}
\ No newline at end of file
\ No newline at end of file
new file mode 100644
new file mode 100644
@@ -0,0 +1,43 @@
@@ -0,0 +1,43 @@
+{
+ "domain": "anker.com",
+ "at": "2026-10-28T10:17:00.155Z",
+ "attempted": [
+ {
+ "url": "https://anker.com/robots.txt",
+ "status": 200,
+ "challenged": false,
+ "bytes": 1192
+ },
+ {
+ "url": "https://anker.com/sitemap.xml",
+ "status": 200,
+ "challenged": false,
+ "bytes": 947
+ },
+ {
+ "url": "https://anker.com/products.json",
+ "status": 404,
+ "challenged": false,
+ "bytes": 1424433
+ },
+ {
+ "url": "https://anker.com/llms.txt",
+ "status": 200,
+ "challenged": false,
+ "bytes": 7071
+ },
+ {
+ "url": "https://anker.com/api/",
+ "status": 404,
+ "challenged": false,
+ "bytes": 1424433
+ },
+ {
+ "url": "https://anker.com/",
+ "status": 200,
+ "challenged": false,
+ "bytes": 1939667
+ }
+ ],
+ "verdict": "ok"
+}
\ No newline at end of file
\ No newline at end of file
new file mode 100644
new file mode 100644
@@ -0,0 +1,43 @@
@@ -0,0 +1,43 @@
+{
+ "domain": "backcountry.com",
+ "at": "2026-10-28T10:17:00.621Z",
+ "attempted": [
+ {
+ "url": "https://backcountry.com/robots.txt",
+ "status": 403,
+ "challenged": false,
+ "bytes": 919
+ },
+ {
+ "url": "https://backcountry.com/sitemap.xml",
+ "status": 403,
+ "challenged": false,
+ "bytes": 919
+ },
+ {
+ "url": "https://backcountry.com/products.json",
+ "status": 403,
+ "challenged": false,
+ "bytes": 919
+ },
+ {
+ "url": "https://backcountry.com/llms.txt",
+ "status": 403,
+ "challenged": false,
+ "bytes": 919
+ },
+ {
+ "url": "https://backcountry.com/api/",
+ "status": 403,
+ "challenged": false,
+ "bytes": 919
+ },
+ {
+ "url": "https://backcountry.com/",
+ "status": 403,
+ "challenged": false,
+ "bytes": 919
+ }
+ ],
+ "verdict": "queued"
+}
\ No newline at end of file
\ No newline at end of file
new file mode 100644
new file mode 100644
@@ -0,0 +1,43 @@
@@ -0,0 +1,43 @@
+{
+ "domain": "bestbuy.com",
+ "at": "2026-10-28T10:17:52.971Z",
+ "attempted": [
+ {
+ "url": "https://bestbuy.com/robots.txt",
+ "status": 200,
+ "challenged": false,
+ "bytes": 15016
+ },
+ {
+ "url": "https://bestbuy.com/sitemap.xml",
+ "status": 200,
+ "challenged": false,
+ "bytes": 7194
+ },
+ {
+ "url": "https://bestbuy.com/products.json",
+ "status": 404,
+ "challenged": false,
+ "bytes": 287219
+ },
+ {
+ "url": "https://bestbuy.com/llms.txt",
+ "status": 404,
+ "challenged": false,
+ "bytes": 287219
+ },
+ {
+ "url": "https://bestbuy.com/api/",
+ "status": 404,
+ "challenged": false,
+ "bytes": 287219
+ },
+ {
+ "url": "https://bestbuy.com/",
+ "status": 200,
+ "challenged": false,
+ "bytes": 600379
+ }
+ ],
+ "verdict": "ok"
+}
\ No newline at end of file
\ No newline at end of file
new file mode 100644
new file mode 100644
@@ -0,0 +1,43 @@
@@ -0,0 +1,43 @@
+{
+ "domain": "bhphotovideo.com",
+ "at": "2026-10-28T10:17:53.766Z",
+ "attempted": [
+ {
+ "url": "https://bhphotovideo.com/robots.txt",
+ "status": 200,
+ "challenged": false,
+ "bytes": 4747
+ },
+ {
+ "url": "https://bhphotovideo.com/sitemap.xml",
+ "status": 403,
+ "challenged": true,
+ "bytes": 5706
+ },
+ {
+ "url": "https://bhphotovideo.com/products.json",
+ "status": 403,
+ "challenged": true,
+ "bytes": 5690
+ },
+ {
+ "url": "https://bhphotovideo.com/llms.txt",
+ "status": 403,
+ "challenged": true,
+ "bytes": 5697
+ },
+ {
+ "url": "https://bhphotovideo.com/api/",
+ "status": 403,
+ "challenged": true,
+ "bytes": 5685
+ },
+ {
+ "url": "https://bhphotovideo.com/",
+ "status": 403,
+ "challenged": true,
+ "bytes": 5652
+ }
+ ],
+ "verdict": "structured-only"
+}
\ No newline at end of file
\ No newline at end of file
new file mode 100644
new file mode 100644
@@ -0,0 +1,43 @@
@@ -0,0 +1,43 @@
+{
+ "domain": "brooksrunning.com",
+ "at": "2026-10-28T10:17:57.626Z",
+ "attempted": [
+ {
+ "url": "https://brooksrunning.com/robots.txt",
+ "status": 403,
+ "challenged": false,
+ "bytes": 389
+ },
+ {
+ "url": "https://brooksrunning.com/sitemap.xml",
+ "status": 403,
+ "challenged": false,
+ "bytes": 390
+ },
+ {
+ "url": "https://brooksrunning.com/products.json",
+ "status": 403,
+ "challenged": false,
+ "bytes": 392
+ },
+ {
+ "url": "https://brooksrunning.com/llms.txt",
+ "status": 403,
+ "challenged": false,
+ "bytes": 387
+ },
+ {
+ "url": "https://brooksrunning.com/api/",
+ "status": 403,
+ "challenged": false,
+ "bytes": 383
+ },
+ {
+ "url": "https://brooksrunning.com/",
+ "status": 403,
+ "challenged": false,
+ "bytes": 375
+ }
+ ],
+ "verdict": "queued"
+}
\ No newline at end of file
\ No newline at end of file
new file mode 100644
new file mode 100644
@@ -0,0 +1,43 @@
@@ -0,0 +1,43 @@
+{
+ "domain": "converse.com",
+ "at": "2026-10-28T10:18:00.177Z",
+ "attempted": [
+ {
+ "url": "https://converse.com/robots.txt",
+ "status": 403,
+ "challenged": false,
+ "bytes": 384
+ },
+ {
+ "url": "https://converse.com/sitemap.xml",
+ "status": 403,
+ "challenged": false,
+ "bytes": 385
+ },
+ {
+ "url": "https://converse.com/products.json",
+ "status": 403,
+ "challenged": false,
+ "bytes": 387
+ },
+ {
+ "url": "https://converse.com/llms.txt",
+ "status": 403,
+ "challenged": false,
+ "bytes": 382
+ },
+ {
+ "url": "https://converse.com/api/",
+ "status": 403,
+ "challenged": false,
+ "bytes": 378
+ },
+ {
+ "url": "https://converse.com/",
+ "status": 403,
+ "challenged": false,
+ "bytes": 370
+ }
+ ],
+ "verdict": "queued"
+}
\ No newline at end of file
\ No newline at end of file
new file mode 100644
new file mode 100644
@@ -0,0 +1,43 @@
@@ -0,0 +1,43 @@
+{
+ "domain": "dsw.com",
+ "at": "2026-10-28T10:18:02.761Z",
+ "attempted": [
+ {
+ "url": "https://dsw.com/robots.txt",
+ "status": 200,
+ "challenged": false,
+ "bytes": 1617
+ },
+ {
+ "url": "https://dsw.com/sitemap.xml",
+ "status": 404,
+ "challenged": false,
+ "bytes": 1515
+ },
+ {
+ "url": "https://dsw.com/products.json",
+ "status": 404,
+ "challenged": false,
+ "bytes": 248626
+ },
+ {
+ "url": "https://dsw.com/llms.txt",
+ "status": 404,
+ "challenged": false,
+ "bytes": 1515
+ },
+ {
+ "url": "https://dsw.com/api/",
+ "status": 200,
+ "challenged": false,
+ "bytes": 20054
+ },
+ {
+ "url": "https://dsw.com/",
+ "status": 200,
+ "challenged": false,
+ "bytes": 704259
+ }
+ ],
+ "verdict": "ok"
+}
\ No newline at end of file
\ No newline at end of file
new file mode 100644
new file mode 100644
@@ -0,0 +1,43 @@
@@ -0,0 +1,43 @@
+{
+ "domain": "footlocker.com",
+ "at": "2026-10-28T10:18:04.235Z",
+ "attempted": [
+ {
+ "url": "https://footlocker.com/robots.txt",
+ "status": 200,
+ "challenged": false,
+ "bytes": 1050
+ },
+ {
+ "url": "https://footlocker.com/sitemap.xml",
+ "status": 200,
+ "challenged": false,
+ "bytes": 2356
+ },
+ {
+ "url": "https://footlocker.com/products.json",
+ "status": 200,
+ "challenged": false,
+ "bytes": 760829
+ },
+ {
+ "url": "https://footlocker.com/llms.txt",
+ "status": 200,
+ "challenged": false,
+ "bytes": 757022
+ },
+ {
+ "url": "https://footlocker.com/api/",
+ "status": 403,
+ "challenged": false,
+ "bytes": 0
+ },
+ {
+ "url": "https://footlocker.com/",
+ "status": 200,
+ "challenged": false,
+ "bytes": 928378
+ }
+ ],
+ "verdict": "ok"
+}
\ No newline at end of file
\ No newline at end of file
new file mode 100644
new file mode 100644
@@ -0,0 +1,43 @@
@@ -0,0 +1,43 @@
+{
+ "domain": "frame.work",
+ "at": "2026-10-28T10:18:04.784Z",
+ "attempted": [
+ {
+ "url": "https://frame.work/robots.txt",
+ "status": 403,
+ "challenged": true,
+ "bytes": 5697
+ },
+ {
+ "url": "https://frame.work/sitemap.xml",
+ "status": 403,
+ "challenged": true,
+ "bytes": 5700
+ },
+ {
+ "url": "https://frame.work/products.json",
+ "status": 403,
+ "challenged": true,
+ "bytes": 5706
+ },
+ {
+ "url": "https://frame.work/llms.txt",
+ "status": 403,
+ "challenged": true,
+ "bytes": 5669
+ },
+ {
+ "url": "https://frame.work/api/",
+ "status": 403,
+ "challenged": true,
+ "bytes": 5657
+ },
+ {
+ "url": "https://frame.work/",
+ "status": 403,
+ "challenged": true,
+ "bytes": 5645
+ }
+ ],
+ "verdict": "queued"
+}
\ No newline at end of file
\ No newline at end of file
new file mode 100644
new file mode 100644
@@ -0,0 +1,43 @@
@@ -0,0 +1,43 @@
+{
+ "domain": "gog.com",
+ "at": "2026-10-28T10:18:07.984Z",
+ "attempted": [
+ {
+ "url": "https://gog.com/robots.txt",
+ "status": 200,
+ "challenged": false,
+ "bytes": 152
+ },
+ {
+ "url": "https://gog.com/sitemap.xml",
+ "status": 404,
+ "challenged": false,
+ "bytes": 153
+ },
+ {
+ "url": "https://gog.com/products.json",
+ "status": 404,
+ "challenged": false,
+ "bytes": 407603
+ },
+ {
+ "url": "https://gog.com/llms.txt",
+ "status": 404,
+ "challenged": false,
+ "bytes": 153
+ },
+ {
+ "url": "https://gog.com/api/",
+ "status": 404,
+ "challenged": false,
+ "bytes": 407531
+ },
+ {
+ "url": "https://gog.com/",
+ "status": 200,
+ "challenged": false,
+ "bytes": 1503918
+ }
+ ],
+ "verdict": "ok"
+}
\ No newline at end of file
\ No newline at end of file
new file mode 100644
new file mode 100644
@@ -0,0 +1,42 @@
@@ -0,0 +1,42 @@
+{
+ "domain": "hoka.com",
+ "at": "2026-10-28T10:18:09.453Z",
+ "attempted": [
+ {
+ "url": "https://hoka.com/robots.txt",
+ "status": 200,
+ "challenged": false,
+ "bytes": 9907
+ },
+ {
+ "url": "https://hoka.com/sitemap.xml",
+ "status": 404,
+ "challenged": false,
+ "bytes": 0
+ },
+ {
+ "url": "https://hoka.com/products.json",
+ "status": 410,
+ "challenged": false,
+ "bytes": 625565
+ },
+ {
+ "url": "https://hoka.com/llms.txt",
+ "status": 410,
+ "challenged": false,
+ "bytes": 625565
+ },
+ {
+ "url": "https://hoka.com/api/",
+ "status": 500,
+ "challenged": false,
+ "bytes": 21
+ },
+ {
+ "url": "https://hoka.com/",
+ "status": 0,
+ "error": "TypeError"
+ }
+ ],
+ "verdict": "structured-only"
+}
\ No newline at end of file
\ No newline at end of file
new file mode 100644
new file mode 100644
@@ -0,0 +1,43 @@
@@ -0,0 +1,43 @@
+{
+ "domain": "homedepot.com",
+ "at": "2026-10-28T10:18:10.727Z",
+ "attempted": [
+ {
+ "url": "https://homedepot.com/robots.txt",
+ "status": 200,
+ "challenged": false,
+ "bytes": 1394
+ },
+ {
+ "url": "https://homedepot.com/sitemap.xml",
+ "status": 404,
+ "challenged": false,
+ "bytes": 2522
+ },
+ {
+ "url": "https://homedepot.com/products.json",
+ "status": 404,
+ "challenged": false,
+ "bytes": 2522
+ },
+ {
+ "url": "https://homedepot.com/llms.txt",
+ "status": 404,
+ "challenged": false,
+ "bytes": 2522
+ },
+ {
+ "url": "https://homedepot.com/api/",
+ "status": 404,
+ "challenged": false,
+ "bytes": 2522
+ },
+ {
+ "url": "https://homedepot.com/",
+ "status": 200,
+ "challenged": false,
+ "bytes": 933817
+ }
+ ],
+ "verdict": "ok"
+}
\ No newline at end of file
\ No newline at end of file
new file mode 100644
new file mode 100644
@@ -0,0 +1,43 @@
@@ -0,0 +1,43 @@
+{
+ "domain": "humblebundle.com",
+ "at": "2026-10-28T10:18:13.614Z",
+ "attempted": [
+ {
+ "url": "https://humblebundle.com/robots.txt",
+ "status": 200,
+ "challenged": false,
+ "bytes": 1256
+ },
+ {
+ "url": "https://humblebundle.com/sitemap.xml",
+ "status": 200,
+ "challenged": false,
+ "bytes": 397
+ },
+ {
+ "url": "https://humblebundle.com/products.json",
+ "status": 404,
+ "challenged": false,
+ "bytes": 71476
+ },
+ {
+ "url": "https://humblebundle.com/llms.txt",
+ "status": 404,
+ "challenged": false,
+ "bytes": 71421
+ },
+ {
+ "url": "https://humblebundle.com/api/",
+ "status": 404,
+ "challenged": false,
+ "bytes": 232
+ },
+ {
+ "url": "https://humblebundle.com/",
+ "status": 200,
+ "challenged": false,
+ "bytes": 574183
+ }
+ ],
+ "verdict": "ok"
+}
\ No newline at end of file
\ No newline at end of file
new file mode 100644
new file mode 100644
@@ -0,0 +1,43 @@
@@ -0,0 +1,43 @@
+{
+ "domain": "ikea.com",
+ "at": "2026-10-28T10:18:14.069Z",
+ "attempted": [
+ {
+ "url": "https://ikea.com/robots.txt",
+ "status": 403,
+ "challenged": true,
+ "bytes": 4555
+ },
+ {
+ "url": "https://ikea.com/sitemap.xml",
+ "status": 403,
+ "challenged": true,
+ "bytes": 4555
+ },
+ {
+ "url": "https://ikea.com/products.json",
+ "status": 403,
+ "challenged": true,
+ "bytes": 4555
+ },
+ {
+ "url": "https://ikea.com/llms.txt",
+ "status": 403,
+ "challenged": true,
+ "bytes": 4555
+ },
+ {
+ "url": "https://ikea.com/api/",
+ "status": 403,
+ "challenged": true,
+ "bytes": 4555
+ },
+ {
+ "url": "https://ikea.com/",
+ "status": 403,
+ "challenged": true,
+ "bytes": 4555
+ }
+ ],
+ "verdict": "queued"
+}
\ No newline at end of file
\ No newline at end of file
new file mode 100644
new file mode 100644
@@ -0,0 +1,43 @@
@@ -0,0 +1,43 @@
+{
+ "domain": "microcenter.com",
+ "at": "2026-10-28T10:18:14.764Z",
+ "attempted": [
+ {
+ "url": "https://microcenter.com/robots.txt",
+ "status": 403,
+ "challenged": true,
+ "bytes": 5706
+ },
+ {
+ "url": "https://microcenter.com/sitemap.xml",
+ "status": 403,
+ "challenged": true,
+ "bytes": 5709
+ },
+ {
+ "url": "https://microcenter.com/products.json",
+ "status": 403,
+ "challenged": true,
+ "bytes": 5737
+ },
+ {
+ "url": "https://microcenter.com/llms.txt",
+ "status": 403,
+ "challenged": true,
+ "bytes": 5700
+ },
+ {
+ "url": "https://microcenter.com/api/",
+ "status": 403,
+ "challenged": true,
+ "bytes": 5688
+ },
+ {
+ "url": "https://microcenter.com/",
+ "status": 403,
+ "challenged": true,
+ "bytes": 5676
+ }
+ ],
+ "verdict": "queued"
+}
\ No newline at end of file
\ No newline at end of file
new file mode 100644
new file mode 100644
@@ -0,0 +1,43 @@
@@ -0,0 +1,43 @@
+{
+ "domain": "monoprice.com",
+ "at": "2026-10-28T10:18:15.581Z",
+ "attempted": [
+ {
+ "url": "https://monoprice.com/robots.txt",
+ "status": 200,
+ "challenged": false,
+ "bytes": 629
+ },
+ {
+ "url": "https://monoprice.com/sitemap.xml",
+ "status": 403,
+ "challenged": true,
+ "bytes": 5703
+ },
+ {
+ "url": "https://monoprice.com/products.json",
+ "status": 403,
+ "challenged": true,
+ "bytes": 5731
+ },
+ {
+ "url": "https://monoprice.com/llms.txt",
+ "status": 404,
+ "challenged": false,
+ "bytes": 480731
+ },
+ {
+ "url": "https://monoprice.com/api/",
+ "status": 403,
+ "challenged": true,
+ "bytes": 5682
+ },
+ {
+ "url": "https://monoprice.com/",
+ "status": 403,
+ "challenged": true,
+ "bytes": 5670
+ }
+ ],
+ "verdict": "structured-only"
+}
\ No newline at end of file
\ No newline at end of file
new file mode 100644
new file mode 100644
@@ -0,0 +1,43 @@
@@ -0,0 +1,43 @@
+{
+ "domain": "newbalance.com",
+ "at": "2026-10-28T10:18:16.312Z",
+ "attempted": [
+ {
+ "url": "https://newbalance.com/robots.txt",
+ "status": 403,
+ "challenged": false,
+ "bytes": 633
+ },
+ {
+ "url": "https://newbalance.com/sitemap.xml",
+ "status": 403,
+ "challenged": false,
+ "bytes": 633
+ },
+ {
+ "url": "https://newbalance.com/products.json",
+ "status": 403,
+ "challenged": false,
+ "bytes": 633
+ },
+ {
+ "url": "https://newbalance.com/llms.txt",
+ "status": 403,
+ "challenged": false,
+ "bytes": 633
+ },
+ {
+ "url": "https://newbalance.com/api/",
+ "status": 403,
+ "challenged": false,
+ "bytes": 633
+ },
+ {
+ "url": "https://newbalance.com/",
+ "status": 403,
+ "challenged": false,
+ "bytes": 633
+ }
+ ],
+ "verdict": "queued"
+}
\ No newline at end of file
\ No newline at end of file
new file mode 100644
new file mode 100644
@@ -0,0 +1,43 @@
@@ -0,0 +1,43 @@
+{
+ "domain": "newegg.com",
+ "at": "2026-10-28T10:18:16.799Z",
+ "attempted": [
+ {
+ "url": "https://newegg.com/robots.txt",
+ "status": 403,
+ "challenged": false,
+ "bytes": 372
+ },
+ {
+ "url": "https://newegg.com/sitemap.xml",
+ "status": 403,
+ "challenged": false,
+ "bytes": 373
+ },
+ {
+ "url": "https://newegg.com/products.json",
+ "status": 403,
+ "challenged": false,
+ "bytes": 375
+ },
+ {
+ "url": "https://newegg.com/llms.txt",
+ "status": 403,
+ "challenged": false,
+ "bytes": 370
+ },
+ {
+ "url": "https://newegg.com/api/",
+ "status": 403,
+ "challenged": false,
+ "bytes": 366
+ },
+ {
+ "url": "https://newegg.com/",
+ "status": 403,
+ "challenged": false,
+ "bytes": 358
+ }
+ ],
+ "verdict": "queued"
+}
\ No newline at end of file
\ No newline at end of file
new file mode 100644
new file mode 100644
@@ -0,0 +1,43 @@
@@ -0,0 +1,43 @@
+{
+ "domain": "on.com",
+ "at": "2026-10-28T10:18:19.026Z",
+ "attempted": [
+ {
+ "url": "https://on.com/robots.txt",
+ "status": 200,
+ "challenged": false,
+ "bytes": 392
+ },
+ {
+ "url": "https://on.com/sitemap.xml",
+ "status": 200,
+ "challenged": false,
+ "bytes": 460
+ },
+ {
+ "url": "https://on.com/products.json",
+ "status": 200,
+ "challenged": false,
+ "bytes": 623476
+ },
+ {
+ "url": "https://on.com/llms.txt",
+ "status": 200,
+ "challenged": false,
+ "bytes": 800
+ },
+ {
+ "url": "https://on.com/api/",
+ "status": 403,
+ "challenged": false,
+ "bytes": 691319
+ },
+ {
+ "url": "https://on.com/",
+ "status": 200,
+ "challenged": false,
+ "bytes": 623476
+ }
+ ],
+ "verdict": "ok"
+}
\ No newline at end of file
\ No newline at end of file
new file mode 100644
new file mode 100644
@@ -0,0 +1,43 @@
@@ -0,0 +1,43 @@
+{
+ "domain": "patagonia.com",
+ "at": "2026-10-28T10:18:19.826Z",
+ "attempted": [
+ {
+ "url": "https://patagonia.com/robots.txt",
+ "status": 200,
+ "challenged": false,
+ "bytes": 3226
+ },
+ {
+ "url": "https://patagonia.com/sitemap.xml",
+ "status": 200,
+ "challenged": false,
+ "bytes": 3226
+ },
+ {
+ "url": "https://patagonia.com/products.json",
+ "status": 200,
+ "challenged": false,
+ "bytes": 3226
+ },
+ {
+ "url": "https://patagonia.com/llms.txt",
+ "status": 200,
+ "challenged": false,
+ "bytes": 3226
+ },
+ {
+ "url": "https://patagonia.com/api/",
+ "status": 200,
+ "challenged": false,
+ "bytes": 3226
+ },
+ {
+ "url": "https://patagonia.com/",
+ "status": 200,
+ "challenged": false,
+ "bytes": 3226
+ }
+ ],
+ "verdict": "ok"
+}
\ No newline at end of file
\ No newline at end of file
new file mode 100644
new file mode 100644
@@ -0,0 +1,43 @@
@@ -0,0 +1,43 @@
+{
+ "domain": "rei.com",
+ "at": "2026-10-28T10:18:20.825Z",
+ "attempted": [
+ {
+ "url": "https://rei.com/robots.txt",
+ "status": 403,
+ "challenged": false,
+ "bytes": 369
+ },
+ {
+ "url": "https://rei.com/sitemap.xml",
+ "status": 403,
+ "challenged": false,
+ "bytes": 370
+ },
+ {
+ "url": "https://rei.com/products.json",
+ "status": 403,
+ "challenged": false,
+ "bytes": 372
+ },
+ {
+ "url": "https://rei.com/llms.txt",
+ "status": 403,
+ "challenged": false,
+ "bytes": 367
+ },
+ {
+ "url": "https://rei.com/api/",
+ "status": 403,
+ "challenged": false,
+ "bytes": 363
+ },
+ {
+ "url": "https://rei.com/",
+ "status": 403,
+ "challenged": false,
+ "bytes": 355
+ }
+ ],
+ "verdict": "queued"
+}
\ No newline at end of file
\ No newline at end of file
new file mode 100644
new file mode 100644
@@ -0,0 +1,37 @@
@@ -0,0 +1,37 @@
+{
+ "domain": "runningwarehouse.com",
+ "at": "2026-10-28T10:18:22.204Z",
+ "attempted": [
+ {
+ "url": "https://runningwarehouse.com/robots.txt",
+ "status": 0,
+ "error": "TypeError"
+ },
+ {
+ "url": "https://runningwarehouse.com/sitemap.xml",
+ "status": 0,
+ "error": "TypeError"
+ },
+ {
+ "url": "https://runningwarehouse.com/products.json",
+ "status": 0,
+ "error": "TypeError"
+ },
+ {
+ "url": "https://runningwarehouse.com/llms.txt",
+ "status": 0,
+ "error": "TypeError"
+ },
+ {
+ "url": "https://runningwarehouse.com/api/",
+ "status": 0,
+ "error": "TypeError"
+ },
+ {
+ "url": "https://runningwarehouse.com/",
+ "status": 0,
+ "error": "TypeError"
+ }
+ ],
+ "verdict": "queued"
+}
\ No newline at end of file
\ No newline at end of file
new file mode 100644
new file mode 100644
@@ -0,0 +1,43 @@
@@ -0,0 +1,43 @@
+{
+ "domain": "saucony.com",
+ "at": "2026-10-28T10:18:24.130Z",
+ "attempted": [
+ {
+ "url": "https://saucony.com/robots.txt",
+ "status": 200,
+ "challenged": false,
+ "bytes": 3618
+ },
+ {
+ "url": "https://saucony.com/sitemap.xml",
+ "status": 404,
+ "challenged": false,
+ "bytes": 0
+ },
+ {
+ "url": "https://saucony.com/products.json",
+ "status": 404,
+ "challenged": true,
+ "bytes": 364694
+ },
+ {
+ "url": "https://saucony.com/llms.txt",
+ "status": 404,
+ "challenged": true,
+ "bytes": 364694
+ },
+ {
+ "url": "https://saucony.com/api/",
+ "status": 404,
+ "challenged": true,
+ "bytes": 364687
+ },
+ {
+ "url": "https://saucony.com/",
+ "status": 200,
+ "challenged": true,
+ "bytes": 617508
+ }
+ ],
+ "verdict": "structured-only"
+}
\ No newline at end of file
\ No newline at end of file
new file mode 100644
new file mode 100644
@@ -0,0 +1,43 @@
@@ -0,0 +1,43 @@
+{
+ "domain": "vans.com",
+ "at": "2026-10-28T10:18:25.348Z",
+ "attempted": [
+ {
+ "url": "https://vans.com/robots.txt",
+ "status": 403,
+ "challenged": false,
+ "bytes": 366
+ },
+ {
+ "url": "https://vans.com/sitemap.xml",
+ "status": 403,
+ "challenged": false,
+ "bytes": 366
+ },
+ {
+ "url": "https://vans.com/products.json",
+ "status": 403,
+ "challenged": false,
+ "bytes": 366
+ },
+ {
+ "url": "https://vans.com/llms.txt",
+ "status": 403,
+ "challenged": false,
+ "bytes": 366
+ },
+ {
+ "url": "https://vans.com/api/",
+ "status": 403,
+ "challenged": false,
+ "bytes": 366
+ },
+ {
+ "url": "https://vans.com/",
+ "status": 403,
+ "challenged": false,
+ "bytes": 366
+ }
+ ],
+ "verdict": "queued"
+}
\ No newline at end of file
\ No newline at end of file
new file mode 100644
new file mode 100644
@@ -0,0 +1,43 @@
@@ -0,0 +1,43 @@
+{
+ "domain": "wayfair.com",
+ "at": "2026-10-28T10:18:27.350Z",
+ "attempted": [
+ {
+ "url": "https://wayfair.com/robots.txt",
+ "status": 200,
+ "challenged": false,
+ "bytes": 3224
+ },
+ {
+ "url": "https://wayfair.com/sitemap.xml",
+ "status": 404,
+ "challenged": false,
+ "bytes": 23
+ },
+ {
+ "url": "https://wayfair.com/products.json",
+ "status": 404,
+ "challenged": false,
+ "bytes": 23
+ },
+ {
+ "url": "https://wayfair.com/llms.txt",
+ "status": 429,
+ "challenged": false,
+ "bytes": 40
+ },
+ {
+ "url": "https://wayfair.com/api/",
+ "status": 429,
+ "challenged": false,
+ "bytes": 40
+ },
+ {
+ "url": "https://wayfair.com/",
+ "status": 429,
+ "challenged": false,
+ "bytes": 40
+ }
+ ],
+ "verdict": "structured-only"
+}
\ No newline at end of file
\ No newline at end of file
new file mode 100644
new file mode 100644
@@ -0,0 +1,43 @@
@@ -0,0 +1,43 @@
+{
+ "domain": "woot.com",
+ "at": "2026-10-28T10:18:29.235Z",
+ "attempted": [
+ {
+ "url": "https://woot.com/robots.txt",
+ "status": 200,
+ "challenged": false,
+ "bytes": 306
+ },
+ {
+ "url": "https://woot.com/sitemap.xml",
+ "status": 200,
+ "challenged": false,
+ "bytes": 21152
+ },
+ {
+ "url": "https://woot.com/products.json",
+ "status": 404,
+ "challenged": false,
+ "bytes": 4815
+ },
+ {
+ "url": "https://woot.com/llms.txt",
+ "status": 404,
+ "challenged": false,
+ "bytes": 4815
+ },
+ {
+ "url": "https://woot.com/api/",
+ "status": 404,
+ "challenged": false,
+ "bytes": 4815
+ },
+ {
+ "url": "https://woot.com/",
+ "status": 200,
+ "challenged": false,
+ "bytes": 31697
+ }
+ ],
+ "verdict": "ok"
+}
\ No newline at end of file
\ No newline at end of file
new file mode 100644
new file mode 100644
@@ -0,0 +1,43 @@
@@ -0,0 +1,43 @@
+{
+ "domain": "zappos.com",
+ "at": "2026-10-28T10:18:30.558Z",
+ "attempted": [
+ {
+ "url": "https://zappos.com/robots.txt",
+ "status": 200,
+ "challenged": false,
+ "bytes": 1230
+ },
+ {
+ "url": "https://zappos.com/sitemap.xml",
+ "status": 404,
+ "challenged": false,
+ "bytes": 249514
+ },
+ {
+ "url": "https://zappos.com/products.json",
+ "status": 404,
+ "challenged": false,
+ "bytes": 249564
+ },
+ {
+ "url": "https://zappos.com/llms.txt",
+ "status": 404,
+ "challenged": false,
+ "bytes": 249501
+ },
+ {
+ "url": "https://zappos.com/api/",
+ "status": 200,
+ "challenged": false,
+ "bytes": 2599
+ },
+ {
+ "url": "https://zappos.com/",
+ "status": 200,
+ "challenged": false,
+ "bytes": 1373452
+ }
+ ],
+ "verdict": "ok"
+}
\ No newline at end of file
\ No newline at end of file
@@ -9,7 +9,10 @@
@@ -9,7 +9,10 @@
"dev": "NODE_ENV=development node server/index.js",
"dev": "NODE_ENV=development node server/index.js",
"test": "node test/run.js",
"test": "node test/run.js",
"seed": "node scripts/seed.js",
"seed": "node scripts/seed.js",
- "test:pg": "SOURCES_TEST_STORE=pg node test/run.js"
+ "test:pg": "SOURCES_TEST_STORE=pg node test/run.js",
+ "probe": "node scripts/probe.mjs",
+ "adapters": "node scripts/adapters-from-evidence.mjs",
+ "register-adapters": "node scripts/register-adapters.js"
},
},
"keywords": [
"keywords": [
"openvibe",
"openvibe",
new file mode 100644
new file mode 100644
@@ -0,0 +1,65 @@
@@ -0,0 +1,65 @@
+#!/usr/bin/env node
+// ADR 0007: build each source's adapter (or review-queue row) from evidence/<domain>.json only, and print one line per
+// source: verdict, statuses, lane, adapter path. The verdict is the probe's; nothing here re-judges it.
+// ok → adapters/<domain>.json, a crawl source, enabled (structured surfaces first: sitemaps, JSON-LD)
+// structured-only → adapters/<domain>.json, disabled: Lane 1 (feeds, affiliate, official API, merchant opt-in)
+// queued → compliance/review-queue.yaml row; probed again ≥ 24 h later (two attempts)
+import { readFile, writeFile, readdir, mkdir } from 'node:fs/promises';
+
+const terms = new Map(); // terms-of-use quotes on file (research 2026-10-28), carried into license_note as a record
+try {
+ for (const c of JSON.parse(await readFile(process.env.CANDIDATES || '/dev/null', 'utf8').catch(() => '[]'))) {
+ const q = c.terms && c.terms.quotes && c.terms.quotes[0];
+ if (q) terms.set(c.site, `${c.terms.url}: "${String(q).slice(0, 300)}"`);
+ }
+} catch { /* no research file: nothing to carry */ }
+
+const key = (d) => `crawl-${d.replace(/^www\./, '').replace(/[^a-z0-9]+/g, '-')}`.slice(0, 63);
+const status = (e, p) => (e.attempted.find((a) => a.url.endsWith(p)) || {}).status;
+const ok = (s) => s >= 200 && s < 400;
+
+await mkdir('adapters', { recursive: true });
+await mkdir('compliance', { recursive: true });
+const queue = [];
+const lines = [];
+for (const f of (await readdir('evidence')).filter((x) => x.endsWith('.json')).sort()) {
+ const e = JSON.parse(await readFile(`evidence/${f}`, 'utf8'));
+ const statuses = e.attempted.map((a) => `${new URL(a.url).pathname}:${a.status}${a.challenged ? 'c' : ''}`).join(' ');
+ if (e.verdict === 'queued') {
+ const next = new Date(Date.parse(e.at) + 24 * 3600e3).toISOString();
+ queue.push({ domain: e.domain, verdict: 'queued', first_probe: e.at, next_probe_after: next, attempts: 1, statuses });
+ lines.push(`${e.domain} · queued · ${statuses} · review queue (probe again after ${next}) · compliance/review-queue.yaml`);
+ continue;
+ }
+ const sitemap = ok(status(e, '/sitemap.xml'));
+ const enabled = e.verdict === 'ok';
+ const surfaces = ['/sitemap.xml', '/products.json', '/llms.txt', '/api/'].filter((p) => ok(status(e, p)));
+ const src = {
+ key: key(e.domain),
+ name: `${e.domain} (crawl)`,
+ type: 'crawl',
+ category: 'deals',
+ homepage_url: `https://${e.domain}/`,
+ endpoints: [`https://${e.domain}/`],
+ crawl: sitemap
+ ? { sitemaps: true, follow_links: false, require_structured: true, max_pages: 2000, pages_per_run: 50, recrawl_sec: 86400 }
+ : { sitemaps: true, follow_links: true, max_depth: 2, require_structured: true, max_pages: 2000, pages_per_run: 50, recrawl_sec: 86400 },
+ min_interval_ms: 5000,
+ poll_interval_sec: 3600,
+ robots_note: `robots.txt answered ${status(e, '/robots.txt')} at ${e.at} (evidence/${f}); every request and redirect hop is checked against it (RFC 9309, tokens OpenVibeBot and OpenVibeSources).`,
+ terms_note: `Access verdict "${e.verdict}" per ADR 0007, computed from evidence/${f} (${e.at}). Structured surfaces answering: ${surfaces.join(', ') || 'none'}.`
+ + (enabled ? '' : ' Lane 1 only: feeds, affiliate programs, official APIs or merchant opt-in; this crawl adapter stays disabled.'),
+ license_note: terms.has(e.domain) ? `Terms of use on file: ${terms.get(e.domain)}` : null,
+ enabled,
+ review_required: true,
+ default_indexability: 'noindex',
+ };
+ if (!src.license_note) delete src.license_note;
+ await writeFile(`adapters/${e.domain}.json`, JSON.stringify(src, null, 2) + '\n');
+ lines.push(`${e.domain} · ${e.verdict} · ${statuses} · ${enabled ? 'adapter enabled' : 'Lane 1 (adapter disabled)'} · adapters/${e.domain}.json`);
+}
+const yaml = ['# ADR 0007 review queue: sources with nothing under 400. A second probe ≥ 24 h after the first decides; until then', '# no adapter exists. Written by scripts/adapters-from-evidence.mjs from evidence/*.json.', 'queue:']
+ .concat(queue.flatMap((q) => [` - domain: ${q.domain}`, ` verdict: ${q.verdict}`, ` attempts: ${q.attempts}`, ` first_probe: ${q.first_probe}`,
+ ` next_probe_after: ${q.next_probe_after}`, ` statuses: "${q.statuses}"`]));
+await writeFile('compliance/review-queue.yaml', yaml.join('\n') + '\n');
+console.log(lines.join('\n'));
new file mode 100644
new file mode 100644
@@ -0,0 +1,46 @@
@@ -0,0 +1,46 @@
+#!/usr/bin/env node
+// Probe a domain: robots, structured surfaces, then a page fetch.
+// Writes evidence/<domain>.json with statuses and a COMPUTED verdict.
+import { writeFile, mkdir } from 'node:fs/promises';
+
+const UA = 'Mozilla/5.0 (compatible; OpenVibeBot/1.0; +https://openvibe.network/bot)';
+const HDRS = {
+ 'user-agent': UA,
+ accept: 'text/html,application/xhtml+xml,application/json;q=0.9,*/*;q=0.8',
+ 'accept-language': 'en-US,en;q=0.9',
+ 'sec-fetch-dest': 'document', 'sec-fetch-mode': 'navigate', 'sec-fetch-site': 'none',
+};
+const STRUCTURED = ['/robots.txt', '/sitemap.xml', '/products.json', '/llms.txt', '/api/'];
+
+const domain = process.argv[2];
+if (!domain) { console.error('usage: probe.mjs <domain>'); process.exit(1); }
+
+const get = async (url) => {
+ try {
+ const r = await fetch(url, { headers: HDRS, redirect: 'follow',
+ signal: AbortSignal.timeout(15000) });
+ // a 200 with a challenge body still counts as challenged
+ const body = await r.text();
+ const challenged = /just a moment|cf-chl|datadome|captcha/i.test(body.slice(0, 4000));
+ return { url, status: r.status, challenged, bytes: body.length };
+ } catch (e) { return { url, status: 0, error: String(e.name) }; }
+};
+
+const results = [];
+for (const p of STRUCTURED) results.push(await get(`https://${domain}${p}`));
+results.push(await get(`https://${domain}/`));
+
+const structuredOk = results.slice(0, 5).some(r => r.status >= 200 && r.status < 400);
+const page = results.at(-1);
+const pageOk = page.status >= 200 && page.status < 400 && !page.challenged;
+const okCount = results.filter(r => r.status >= 200 && r.status < 400).length;
+
+const verdict = !okCount ? 'queued'
+ : pageOk ? 'ok'
+ : structuredOk ? 'structured-only'
+ : 'queued';
+
+await mkdir('evidence', { recursive: true });
+await writeFile(`evidence/${domain}.json`, JSON.stringify(
+ { domain, at: new Date().toISOString(), attempted: results, verdict }, null, 2));
+console.log(domain, '→', verdict);
new file mode 100644
new file mode 100644
@@ -0,0 +1,50 @@
@@ -0,0 +1,50 @@
+#!/usr/bin/env node
+'use strict';
+/**
+ * Register adapters/*.json (written by scripts/adapters-from-evidence.mjs from evidence/*.json, ADR 0007): creates each
+ * source that does not exist and brings an existing one to its adapter, including `enabled`, which the probe's verdict
+ * decided (ok → on, structured-only → off). A queued domain has no adapter, so nothing here touches it.
+ *
+ * node scripts/register-adapters.js [--dry-run] (DATABASE_URL as for scripts/seed.js)
+ */
+require('dotenv').config();
+const fs = require('fs');
+const path = require('path');
+const { load } = require('../server/config');
+const { openDb } = require('../server/db');
+const { createRegistry } = require('../server/registry');
+
+const DIR = path.join(__dirname, '..', 'adapters');
+
+async function register(db, { now = () => Date.now(), by = 'adr-0007', maxItemsCap, dryRun = false } = {}) {
+ const registry = createRegistry({ db, now, ...(maxItemsCap ? { maxItemsCap } : {}) });
+ const out = { created: [], updated: [], same: [] };
+ for (const f of fs.readdirSync(DIR).filter(x => x.endsWith('.json')).sort()) {
+ const a = JSON.parse(fs.readFileSync(path.join(DIR, f), 'utf8'));
+ const row = await registry.get(a.key);
+ if (!row) {
+ if (!dryRun) await registry.create(a, by);
+ out.created.push(a.key);
+ continue;
+ }
+ const was = registry.fromRow(row);
+ const changed = Object.keys(a).some(k => JSON.stringify(a[k]) !== JSON.stringify(was[k]));
+ if (!changed) { out.same.push(a.key); continue; }
+ if (!dryRun) await registry.patch(a.key, a, by);
+ out.updated.push(a.key);
+ }
+ return out;
+}
+
+if (require.main === module) {
+ (async () => {
+ const config = load();
+ const db = await openDb(config);
+ const r = await register(db, { maxItemsCap: config.items.maxPerFetch, dryRun: process.argv.includes('--dry-run') });
+ console.log(`created ${r.created.length}${r.created.length ? ` (${r.created.join(', ')})` : ''}; updated ${r.updated.length}`
+ + `${r.updated.length ? ` (${r.updated.join(', ')})` : ''}; unchanged ${r.same.length}`);
+ await db.close();
+ })().catch((err) => { console.error(`[register-adapters] ${err.message}`); process.exit(1); });
+}
+
+module.exports = { register };
@@ -9,6 +9,7 @@
@@ -9,6 +9,7 @@
* summary is the description the page wrote for search results, never its body.
* summary is the description the page wrote for search results, never its body.
*/
*/
const { toText, toIsoDate, canonicalUrl, decodeEntities } = require('./util');
const { toText, toIsoDate, canonicalUrl, decodeEntities } = require('./util');
+const { agentTokens } = require('../robots');
const PARSER_VERSION = 'page@1';
const PARSER_VERSION = 'page@1';
const MAX_LINKS = 1000;
const MAX_LINKS = 1000;
@@ -39,7 +40,7 @@ function directives(value) {
@@ -39,7 +40,7 @@ function directives(value) {
/**
/**
* X-Robots-Tag: "noindex", "noindex, nofollow", or scoped to one crawler ("otherbot: noindex"). A directive scoped to
* X-Robots-Tag: "noindex", "noindex, nofollow", or scoped to one crawler ("otherbot: noindex"). A directive scoped to
- * another crawler does not apply to us; one scoped to our product token does.
+ * another crawler does not apply to us; one scoped to any of our product tokens does.
*/
*/
function headerDirectives(header, agent) {
function headerDirectives(header, agent) {
const out = { noindex: false, nofollow: false };
const out = { noindex: false, nofollow: false };
@@ -47,7 +48,7 @@ function headerDirectives(header, agent) {
@@ -47,7 +48,7 @@ function headerDirectives(header, agent) {
let value = String(line);
let value = String(line);
const m = /^\s*([a-z0-9_-]+)\s*:\s*(.*)$/i.exec(value);
const m = /^\s*([a-z0-9_-]+)\s*:\s*(.*)$/i.exec(value);
if (m && !['unavailable_after', 'max-snippet', 'max-image-preview', 'max-video-preview'].includes(m[1].toLowerCase())) {
if (m && !['unavailable_after', 'max-snippet', 'max-image-preview', 'max-video-preview'].includes(m[1].toLowerCase())) {
- if (m[1].toLowerCase() !== String(agent).toLowerCase()) continue;
+ if (!agentTokens(agent).includes(m[1].toLowerCase())) continue;
value = m[2];
value = m[2];
}
}
const d = directives(value);
const d = directives(value);
@@ -61,17 +62,17 @@ function headerDirectives(header, agent) {
@@ -61,17 +62,17 @@ function headerDirectives(header, agent) {
* read(html, { url, agent, headers, links }) → { noindex, nofollow, canonical, base, title, description, image, site_name,
* read(html, { url, agent, headers, links }) → { noindex, nofollow, canonical, base, title, description, image, site_name,
* og_type, language, published_at, updated_at, author, links: [absolute urls] }
* og_type, language, published_at, updated_at, author, links: [absolute urls] }
*/
*/
-function read(html, { url, agent = 'OpenVibeSources', headers = {}, links = false } = {}) {
+function read(html, { url, agent = 'OpenVibeBot,OpenVibeSources', headers = {}, links = false } = {}) {
const head = html.slice(0, 512 * 1024);
const head = html.slice(0, 512 * 1024);
const meta = {};
const meta = {};
let noindex = false;
let noindex = false;
let nofollow = false;
let nofollow = false;
- const token = String(agent).toLowerCase();
+ const tokens = agentTokens(agent);
for (const t of tags(head, 'meta')) {
for (const t of tags(head, 'meta')) {
const a = attrs(t);
const a = attrs(t);
const key = (a.name || a.property || '').toLowerCase();
const key = (a.name || a.property || '').toLowerCase();
if (!key || a.content == null) continue;
if (!key || a.content == null) continue;
- if (key === 'robots' || key === token) {
+ if (key === 'robots' || tokens.includes(key)) {
const d = directives(a.content);
const d = directives(a.content);
noindex = noindex || d.noindex;
noindex = noindex || d.noindex;
nofollow = nofollow || d.nofollow;
nofollow = nofollow || d.nofollow;
@@ -66,9 +66,10 @@ function load(env = process.env) {
@@ -66,9 +66,10 @@ function load(env = process.env) {
maxBackoffMs: int(env.SOURCES_MAX_BACKOFF_MS, 6 * 3600 * 1000),
maxBackoffMs: int(env.SOURCES_MAX_BACKOFF_MS, 6 * 3600 * 1000),
},
},
fetch: {
fetch: {
- userAgent: env.SOURCES_USER_AGENT || `OpenVibeSources/${pkg.version} (+https://sources.openvibe.network; [email protected])`,
- // Product token matched against robots.txt User-agent lines.
- robotsAgent: env.SOURCES_ROBOTS_AGENT || 'OpenVibeSources',
+ // ADR 0007: the compatible-form string, fixed. robots.txt groups for either product token apply.
+ userAgent: env.SOURCES_USER_AGENT || 'Mozilla/5.0 (compatible; OpenVibeBot/1.0; +https://openvibe.network/bot)',
+ // Product tokens (comma list) matched against robots.txt User-agent lines, robots meta names and X-Robots-Tag.
+ robotsAgent: env.SOURCES_ROBOTS_AGENT || 'OpenVibeBot,OpenVibeSources',
timeoutMs: int(env.SOURCES_FETCH_TIMEOUT_MS, 20000),
timeoutMs: int(env.SOURCES_FETCH_TIMEOUT_MS, 20000),
maxBytes: int(env.SOURCES_MAX_BYTES, 5 * 1024 * 1024),
maxBytes: int(env.SOURCES_MAX_BYTES, 5 * 1024 * 1024),
maxRedirects: int(env.SOURCES_MAX_REDIRECTS, 5),
maxRedirects: int(env.SOURCES_MAX_REDIRECTS, 5),
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
* robots.txt (RFC 9309), checked before every request an ingestion run makes, including each
* robots.txt (RFC 9309), checked before every request an ingestion run makes, including each
* redirect hop to a new origin.
* redirect hop to a new origin.
*
*
- * - the group whose User-agent names our product token (case-insensitive) applies, else `*`;
- * several matching groups are merged
+ * - the groups whose User-agent names one of our product tokens (a comma list, case-insensitive:
+ * `OpenVibeBot,OpenVibeSources`) apply, else `*`; several matching groups are merged
* - Allow/Disallow with `*` and `$`; the longest matching rule wins, Allow wins a tie
* - Allow/Disallow with `*` and `$`; the longest matching rule wins, Allow wins a tie
* - 2xx: parsed; 4xx (e.g. 404) except 429: no restrictions; 429, 5xx, network error or timeout:
* - 2xx: parsed; 4xx (e.g. 404) except 429: no restrictions; 429, 5xx, network error or timeout:
* complete disallow ("unreachable") — never guessed as allowed
* complete disallow ("unreachable") — never guessed as allowed
@@ -16,8 +16,13 @@
@@ -16,8 +16,13 @@
* the fetch run is `robots_denied` and nothing is requested.
* the fetch run is `robots_denied` and nothing is requested.
*/
*/
+// 'OpenVibeBot,OpenVibeSources' → ['openvibebot', 'openvibesources']
+function agentTokens(agent) {
+ return String(agent).toLowerCase().split(',').map(t => t.trim()).filter(Boolean);
+}
+
function parseRobots(text, agent) {
function parseRobots(text, agent) {
- const token = String(agent).toLowerCase();
+ const tokens = agentTokens(agent);
const groups = [];
const groups = [];
const sitemaps = [];
const sitemaps = [];
let current = null;
let current = null;
@@ -55,7 +60,7 @@ function parseRobots(text, agent) {
@@ -55,7 +60,7 @@ function parseRobots(text, agent) {
}
}
}
}
// RFC 9309: the product token is matched case-insensitively against the User-agent value.
// RFC 9309: the product token is matched case-insensitively against the User-agent value.
- let chosen = groups.filter(g => g.agents.some(a => a.split('/')[0].trim() === token));
+ let chosen = groups.filter(g => g.agents.some(a => tokens.includes(a.split('/')[0].trim())));
if (!chosen.length) chosen = groups.filter(g => g.agents.includes('*'));
if (!chosen.length) chosen = groups.filter(g => g.agents.includes('*'));
const delays = chosen.map(g => g.crawlDelay).filter(d => d != null);
const delays = chosen.map(g => g.crawlDelay).filter(d => d != null);
return {
return {
@@ -191,4 +196,4 @@ function createRobots({ db, fetcher, agent, ttlMs, maxBytes, now = () => Date.no
@@ -191,4 +196,4 @@ function createRobots({ db, fetcher, agent, ttlMs, maxBytes, now = () => Date.no
return { check, matcher, sitemaps, forget: async (origin) => await db.prepare('DELETE FROM robots_cache WHERE origin = ?').run(origin) };
return { check, matcher, sitemaps, forget: async (origin) => await db.prepare('DELETE FROM robots_cache WHERE origin = ?').run(origin) };
}
}
-module.exports = { parseRobots, isAllowed, compileRules, createRobots };
+module.exports = { parseRobots, isAllowed, compileRules, createRobots, agentTokens };
@@ -242,6 +242,8 @@ t('X-Robots-Tag scoped to another crawler does not apply; scoped to ours it does
@@ -242,6 +242,8 @@ t('X-Robots-Tag scoped to another crawler does not apply; scoped to ours it does
assert.deepStrictEqual(page.headerDirectives('openvibesources: noindex, nofollow', 'OpenVibeSources'), { noindex: true, nofollow: true });
assert.deepStrictEqual(page.headerDirectives('openvibesources: noindex, nofollow', 'OpenVibeSources'), { noindex: true, nofollow: true });
assert.deepStrictEqual(page.headerDirectives(['none'], 'OpenVibeSources'), { noindex: true, nofollow: true });
assert.deepStrictEqual(page.headerDirectives(['none'], 'OpenVibeSources'), { noindex: true, nofollow: true });
assert.deepStrictEqual(page.headerDirectives('unavailable_after: 2030-01-01', 'OpenVibeSources'), { noindex: false, nofollow: false });
assert.deepStrictEqual(page.headerDirectives('unavailable_after: 2030-01-01', 'OpenVibeSources'), { noindex: false, nofollow: false });
+ assert.deepStrictEqual(page.headerDirectives('openvibebot: noindex', 'OpenVibeBot,OpenVibeSources'), { noindex: true, nofollow: false });
+ assert.strictEqual(page.read('<meta name="OpenVibeBot" content="noindex">', { url: 'https://x.test/' }).noindex, true);
});
});
t('the site failing is the source failing: robots.txt unreachable, then a 429', async () => {
t('the site failing is the source failing: robots.txt unreachable, then a 429', async () => {
@@ -68,7 +68,7 @@ t('a good fetch creates items with full provenance and an ok run', async () => {
@@ -68,7 +68,7 @@ t('a good fetch creates items with full provenance and an ok run', async () => {
assert.strictEqual((await events('sources.item.created')).filter(e => e.payload.source_key === key).length, 2);
assert.strictEqual((await events('sources.item.created')).filter(e => e.payload.source_key === key).length, 2);
// the fetcher identified itself and asked robots.txt first
// the fetcher identified itself and asked robots.txt first
const feedReq = web.hits('/feed.xml')[0];
const feedReq = web.hits('/feed.xml')[0];
- assert.match(feedReq.headers['user-agent'], /^OpenVibeSources\//);
+ assert.strictEqual(feedReq.headers['user-agent'], 'Mozilla/5.0 (compatible; OpenVibeBot/1.0; +https://openvibe.network/bot)');
assert.ok(web.hits('/robots.txt')[0].at <= feedReq.at);
assert.ok(web.hits('/robots.txt')[0].at <= feedReq.at);
});
});
@@ -28,6 +28,11 @@ t('robots rules: our group over *, longest match, Allow wins ties, * and $', ()
@@ -28,6 +28,11 @@ t('robots rules: our group over *, longest match, Allow wins ties, * and $', ()
assert.strictEqual(isAllowed(empty, '/anything'), true);
assert.strictEqual(isAllowed(empty, '/anything'), true);
const prefix = parseRobots('User-agent: OpenVibe\nDisallow: /\n', 'OpenVibeSources');
const prefix = parseRobots('User-agent: OpenVibe\nDisallow: /\n', 'OpenVibeSources');
assert.strictEqual(isAllowed(prefix, '/x'), true, 'a different token (even a prefix of ours) does not apply');
assert.strictEqual(isAllowed(prefix, '/x'), true, 'a different token (even a prefix of ours) does not apply');
+ // ADR 0007: a comma list of product tokens; a group naming either applies
+ const both = 'User-agent: *\nDisallow: /all\n\nUser-agent: OpenVibeBot\nDisallow: /bot-only\n';
+ assert.strictEqual(isAllowed(parseRobots(both, 'OpenVibeBot,OpenVibeSources'), '/bot-only'), false);
+ assert.strictEqual(isAllowed(parseRobots(both, 'OpenVibeBot,OpenVibeSources'), '/all'), true, 'our group replaces *');
+ assert.strictEqual(isAllowed(parseRobots('User-agent: OpenVibeSources\nDisallow: /\n', 'OpenVibeBot,OpenVibeSources'), '/x'), false);
});
});
t('address policy', () => {
t('address policy', () => {
new file mode 100644
new file mode 100644
@@ -0,0 +1,53 @@
@@ -0,0 +1,53 @@
+'use strict';
+/** ADR 0007: adapters follow evidence/*.json exactly — ok → enabled crawl, structured-only → disabled, queued → review queue. */
+const assert = require('assert');
+const fs = require('fs');
+const path = require('path');
+const { validateSource } = require('../server/registry');
+const { load } = require('../server/config');
+const { suite } = require('./helpers');
+
+const t = suite('source-access');
+const ROOT = path.join(__dirname, '..');
+const evidence = fs.readdirSync(path.join(ROOT, 'evidence')).filter(f => f.endsWith('.json'))
+ .map(f => JSON.parse(fs.readFileSync(path.join(ROOT, 'evidence', f), 'utf8')));
+const adapter = (d) => {
+ const p = path.join(ROOT, 'adapters', `${d}.json`);
+ return fs.existsSync(p) ? JSON.parse(fs.readFileSync(p, 'utf8')) : null;
+};
+const queue = fs.readFileSync(path.join(ROOT, 'compliance', 'review-queue.yaml'), 'utf8');
+
+t('every evidence file carries a computed verdict', () => {
+ assert.ok(evidence.length >= 29);
+ for (const e of evidence) assert.ok(['ok', 'structured-only', 'queued'].includes(e.verdict), `${e.domain}: ${e.verdict}`);
+});
+
+t('ok → an enabled crawl adapter; structured-only → a disabled one; queued → a review-queue row and no adapter', () => {
+ for (const e of evidence) {
+ const a = adapter(e.domain);
+ if (e.verdict === 'queued') {
+ assert.strictEqual(a, null, `${e.domain} is queued and has an adapter`);
+ assert.ok(queue.includes(`- domain: ${e.domain}\n`), `${e.domain} missing from the review queue`);
+ continue;
+ }
+ assert.ok(a, `${e.domain} (${e.verdict}) has no adapter`);
+ assert.strictEqual(a.type, 'crawl');
+ assert.strictEqual(a.enabled, e.verdict === 'ok', `${e.domain}: enabled must follow the verdict`);
+ assert.ok(a.terms_note.includes(`evidence/${e.domain}.json`), `${e.domain}: terms_note cites its evidence`);
+ assert.ok(a.crawl.require_structured, `${e.domain}: structured surfaces first`);
+ validateSource(a);
+ }
+});
+
+t('no adapter exists without evidence', () => {
+ const known = new Set(evidence.map(e => e.domain));
+ for (const f of fs.readdirSync(path.join(ROOT, 'adapters'))) assert.ok(known.has(f.replace(/\.json$/, '')), `adapters/${f} has no evidence`);
+});
+
+t('the crawler sends the ADR 0007 user agent and answers to both product tokens', () => {
+ const { fetch } = load({});
+ assert.strictEqual(fetch.userAgent, 'Mozilla/5.0 (compatible; OpenVibeBot/1.0; +https://openvibe.network/bot)');
+ assert.deepStrictEqual(fetch.robotsAgent.split(','), ['OpenVibeBot', 'OpenVibeSources']);
+});
+
+t.run();