openvibers/openvibe.sources Commits

W14: sources.openvibe.network answers an honest internal-service page, never the admin placeholder

OP OpenVibers committed 23 Sept 2026 · 2e986af27a · 5 files, +193 −49

Sources is an internal ingestion service with no public pages. The vhost now answers only: / (a
short page saying what the service is and that there is nothing to browse; HTML for browsers, the
text route index otherwise; noindex, no-store, no script), /api/health, /api/ready, /release.json
and a Disallow-all robots.txt. /metrics is 404, /api/v1/ is loopback-only (allow 127.0.0.1 and ::1,
deny all) instead of a bare 403, and every other path is a JSON 404 explaining the host, so the name
no longer falls through to nginx's default site. Client-address headers come from $remote_addr only,
set once for every proxied location. README "Public host" has the install commands (not installed).
test/public-host.test.js.

Changes

ddiff --git a/README.md b/README.md
iindex 38272e8..285f369 100644
---- a/README.md
++++ b/README.md
@@@ -7,8 +7,9 @@
  **Status:** alpha (roadmap Wave 14). Deployed internally, not launched: it runs on the production host
  (127.0.0.1:4720 only, since 2026-09-23) with the six seeded sources **disabled**, so it has made 0 fetch
  runs and holds 0 items.  
--**Domain:** `sources.openvibe.network` (health only on the public vhost; the API is host-local). The
--vhost is not installed yet: the name currently falls through to the admin.openvibe.network placeholder.  
++**Domain:** `sources.openvibe.network` (a short "internal service" page and health on the public
++vhost; the API is loopback-only). The vhost is not installed yet: until it is, the name falls through
++to the admin.openvibe.network placeholder. Install steps: [Public host](#public-host).  
  **Plan:** OpenVibe End-to-End Realignment & Implementation Plan, revision 3 — roadmap §4.2 B, §15.12, §29, anti-goals 13, 26, 27.  
  **License:** AGPL-3.0.
  
@@@ -188,6 +189,8 @@ the terms.
  - parsers against fixtures (`test/adapters.test.js`); API, registry rules, paging, manual items,
    removal (`test/api.test.js`); scheduler, backoff, relay to Events (`test/scheduler-events.test.js`);
    proposals and seeds (`test/proposals.test.js`)
++- the public host shows only an honest "internal service" page, and the vhost keeps the API
++  loopback-only (`test/public-host.test.js`)
  
  Not yet demonstrated: a run against a real source from the deployed service (every seed is disabled
  until a person verifies its terms), and a consuming product using real items. News and Reviews run
@@@ -200,11 +203,40 @@ readiness, row counts; see OpenVibe.Host `docs/restore-drills.md`).
  
  ## Launch rule
  
--The domain keeps its placeholder page on [OpenVibers/OpenVibe.Sites](https://github.com/OpenVibers/OpenVibe.Sites)
--until the plan's launch rule holds (owning runtime with health/readiness, identity and service
--principals, real persistence and workflows, capabilities and events registered in
--OpenVibe.Contracts, a security review, acceptance tests). Sources has no public pages by design;
--its public vhost answers health and a `Disallow: /` robots.txt.
++Sources has no public pages by design, so it never "launches" as a site. The domain is not in
++[OpenVibers/OpenVibe.Sites](https://github.com/OpenVibers/OpenVibe.Sites); this repository's own
++vhost serves it once installed.
++
++## Public host
++
++`sources.openvibe.network` resolves through Cloudflare, but until this vhost is installed nginx
++answers it with its default site (the admin placeholder). The vhost
++([deploy/nginx/sources.openvibe.network.conf](deploy/nginx/sources.openvibe.network.conf)) uses
++the Network wildcard certificate and answers only:
++
++| Path | Answer |
++|---|---|
++| `/` | a short page: an internal ingestion service, nothing to browse (HTML for browsers, the text route index otherwise; noindex, no-store) |
++| `/api/health`, `/api/ready`, `/release.json` | proxied |
++| `/robots.txt` | `Disallow: /` |
++| `/metrics` | 404 (Prometheus scrapes `127.0.0.1:4720/metrics` directly) |
++| `/api/v1/*` | loopback only (`allow 127.0.0.1; allow ::1; deny all`); services call `127.0.0.1:4720` |
++| anything else | a JSON 404 that says what the host is |
++
++Client-address headers come from `$remote_addr` only. Install on the host, after the service runs
++a release with the new `/` page (`ovhost deploy sources`):
++
++```bash
++sudo install -m 0644 /opt/openvibe.sources/deploy/nginx/sources.openvibe.network.conf /etc/nginx/sites-available/sources.openvibe.network.conf
++sudo ln -sf /etc/nginx/sites-available/sources.openvibe.network.conf /etc/nginx/sites-enabled/sources.openvibe.network.conf
++sudo nginx -t && sudo systemctl reload nginx
++# check
++curl -sS -H 'Accept: text/html' https://sources.openvibe.network/ | grep -o '<title>[^<]*'   # OpenVibe.Sources
++curl -sS https://sources.openvibe.network/api/health
++curl -sS -o /dev/null -w '%{http_code}\n' https://sources.openvibe.network/api/v1/sources    # 403
++curl -sS -o /dev/null -w '%{http_code}\n' https://sources.openvibe.network/metrics           # 404
++curl -sS https://sources.openvibe.network/admin                                               # JSON 404
++```
  
  ---
  
ddiff --git a/STATUS.json b/STATUS.json
iindex ae37f48..daaea10 100644
---- a/STATUS.json
++++ b/STATUS.json
@@@ -6,7 +6,7 @@
    "plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026)",
    "runtime": "node 22, express, better-sqlite3, fast-xml-parser",
    "port": 4720,
--  "domain": "sources.openvibe.network (no vhost installed: the name falls through to the admin.openvibe.network placeholder; the service is reachable on the host only)",
++  "domain": "sources.openvibe.network (vhost in deploy/nginx, not installed: until it is, the name falls through to the admin.openvibe.network placeholder; once installed it answers an internal-service page, health and JSON 404s, API loopback-only; README \"Public host\")",
    "contracts": "openvibe-contracts v0.13.0",
    "capabilities": ["sources.source.read", "sources.item.read", "sources.source.manage"],
    "capabilitiesRegistered": true,
ddiff --git a/deploy/nginx/sources.openvibe.network.conf b/deploy/nginx/sources.openvibe.network.conf
iindex f62c241..77f37c6 100644
---- a/deploy/nginx/sources.openvibe.network.conf
++++ b/deploy/nginx/sources.openvibe.network.conf
@@@ -2,16 +2,18 @@
  ## sources.openvibe.network — Nginx Configuration (REFERENCE)
  ##
  ## Deployed to /etc/nginx/sites-available/sources.openvibe.network.conf — keep them in sync,
--## and reload nginx after changing it.
++## and reload nginx after changing it. Install: README "Public host".
  ##
--## The Node service (port 4720, systemd unit openvibe-sources) serves:
--##   /api/health, /api/ready
--##   /metrics    Prometheus text, answered only to direct loopback callers (never proxied)
--##   /api/v1/*   service-to-service API (every route needs a service token). Services on this
--##               host call 127.0.0.1:4720 directly, so the public vhost refuses it; open the
--##               location below only when a service on another host needs it.
--## Nothing here is a public page: Sources holds raw third-party material, which products
--## publish (or not) under their own rules.
++## Sources is an internal ingestion service with no public pages: Sources holds raw third-party
++## material, which products publish (or not) under their own rules. The public host answers only:
++##   /                  a short page saying what the service is (HTML for browsers, text otherwise)
++##   /api/health, /api/ready, /release.json
++##   /robots.txt        disallow everything
++## and keeps the rest off the public edge:
++##   /api/v1/*          service-to-service API (every route needs a service token): loopback only.
++##                      Services on this host call 127.0.0.1:4720 directly.
++##   /metrics           404 here; Prometheus text for direct loopback callers of 127.0.0.1:4720
++##   anything else      a JSON 404 that says why, so the name never falls through to another site
  ## ═══════════════════════════════════════════════════════════════
  
  # ── HTTP → HTTPS redirect ──────────────────────────────────
@@@ -46,29 +48,27 @@ server {
      add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
      add_header X-Robots-Tag "noindex, nofollow" always;
  
--    location = /api/health { proxy_pass http://127.0.0.1:4720; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $remote_addr; proxy_set_header CF-Connecting-IP $remote_addr; }
--    location = /api/ready  { proxy_pass http://127.0.0.1:4720; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $remote_addr; proxy_set_header CF-Connecting-IP $remote_addr; }
++    # Client address only from $remote_addr (realip has already replaced it with the visitor's
++    # address for Cloudflare's ranges); never append to or pass through a client-sent header.
++    # Inherited by every location below that proxies (none sets its own).
++    proxy_http_version 1.1;
++    proxy_set_header Connection "";
++    proxy_set_header Host $host;
++    proxy_set_header X-Real-IP $remote_addr;
++    proxy_set_header X-Forwarded-For $remote_addr;
++    proxy_set_header CF-Connecting-IP $remote_addr;
++    proxy_set_header X-Forwarded-Proto $scheme;
++
++    location = /api/health   { proxy_pass http://127.0.0.1:4720; }
++    location = /api/ready    { proxy_pass http://127.0.0.1:4720; }
++    location = /release.json { proxy_pass http://127.0.0.1:4720; }
  
      # Metrics are for host-local scrapers only (http://127.0.0.1:4720/metrics).
      location = /metrics { return 404; }
  
--    # ── Service API: host-local only (see header) ───────────
--    location /api/ {
--        return 403;
--        # To expose it (token-guarded) to off-host services instead:
--        # proxy_pass http://127.0.0.1:4720;
--        # proxy_http_version 1.1;
--        # proxy_set_header Host $host;
--        # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
--        # proxy_set_header X-Forwarded-Proto $scheme;
--    }
--
      location = / {
++        limit_except GET { deny all; }
          proxy_pass http://127.0.0.1:4720;
--        proxy_set_header Host $host;
--        proxy_set_header X-Real-IP $remote_addr;
--        proxy_set_header X-Forwarded-For $remote_addr;
--        proxy_set_header CF-Connecting-IP $remote_addr;
      }
  
      location = /robots.txt {
@@@ -76,7 +76,17 @@ server {
          return 200 "User-agent: *\nDisallow: /\n";
      }
  
++    # ── Service API: loopback only (see header) ─────────────
++    location ^~ /api/v1/ {
++        allow 127.0.0.1;
++        allow ::1;
++        deny all;
++        proxy_pass http://127.0.0.1:4720;
++    }
++
++    # ── Anything else: an honest JSON 404, never another site ──
      location / {
--        return 404;
++        default_type application/problem+json;
++        return 404 '{"type":"about:blank","title":"Not Found","status":404,"code":"sources.not_found","detail":"sources.openvibe.network is an internal ingestion service of the OpenVibe network with no public pages or API. Its API is for OpenVibe services on the host. About: https://sources.openvibe.network/ and https://github.com/OpenVibers/OpenVibe.Sources"}';
      }
  }
ddiff --git a/server/app.js b/server/app.js
iindex f07025a..55d790a 100644
---- a/server/app.js
++++ b/server/app.js
@@@ -10,6 +10,58 @@ const { sourcesRouter } = require('./api/sources');
  const { itemsRouter } = require('./api/items');
  const pkg = require('../package.json');
  
++const TEXT_INDEX = [
++    'OpenVibe.Sources: the source registry and ingestion workers for the network\'s publication products.',
++    'Every item carries its source, canonical URL, retrieval time, content hash and terms; a failed fetch is',
++    'recorded as a failure and never replaced with invented content.',
++    '',
++    'This is an internal service. Its API is for OpenVibe services on the production host (service tokens);',
++    'sources.openvibe.network answers only this page, /api/health, /api/ready and /release.json.',
++    '',
++    'GET  /api/v1/sources, /api/v1/sources/:key, /api/v1/sources/:key/runs, /api/v1/runs, /api/v1/health   (sources.source.read)',
++    'GET  /api/v1/items?source=&category=&after=, /api/v1/items/:id                                          (sources.item.read)',
++    'POST/PATCH/DELETE /api/v1/sources[/:key], POST /api/v1/sources/:key/fetch, /items, DELETE /api/v1/items/:id (sources.source.manage)',
++    'GET  /api/health, /api/ready, /release.json',
++    '',
++    'Source: https://github.com/OpenVibers/OpenVibe.Sources',
++    '',
++].join('\n');
++
++const HOME_HTML = `<!doctype html>
++<html lang="en">
++<head>
++<meta charset="utf-8">
++<meta name="viewport" content="width=device-width, initial-scale=1">
++<meta name="robots" content="noindex, nofollow">
++<meta name="color-scheme" content="light dark">
++<title>OpenVibe.Sources</title>
++<style>
++:root { --bg: #fff; --fg: #1a1a1a; --muted: #5c5c66; --accent: #2456d6; }
++@media (prefers-color-scheme: dark) { :root { --bg: #111317; --fg: #e8e8ec; --muted: #a0a0ab; --accent: #7aa2ff; } }
++body { margin: 0; background: var(--bg); color: var(--fg); font: 16px/1.55 system-ui, -apple-system, "Segoe UI", sans-serif; }
++main { max-width: 640px; margin: 0 auto; padding: 32px 16px; }
++h1 { font-size: 1.4rem; margin: 0 0 12px; }
++p { margin: 0 0 12px; }
++.muted { color: var(--muted); font-size: .9rem; }
++a { color: var(--accent); }
++</style>
++</head>
++<body>
++<main>
++<h1>OpenVibe.Sources</h1>
++<p>This is an internal service of the OpenVibe network, not a website. It keeps the list of outside sources
++(feeds, sitemaps, APIs) the network's publication products may read, fetches them politely, and records
++where every item came from, when it was fetched and under which terms. A failed fetch is recorded as a
++failure, never replaced with invented content.</p>
++<p>There is nothing to browse here. What reaches readers is published by the products themselves, under
++their own rules.</p>
++<p class="muted">Health: <a href="/api/health">/api/health</a> · Readiness: <a href="/api/ready">/api/ready</a> ·
++Source code: <a href="https://github.com/OpenVibers/OpenVibe.Sources">OpenVibers/OpenVibe.Sources</a></p>
++</main>
++</body>
++</html>
++`;
++
  function createApp({ config, db, registry, items, ingest, scheduler, auth, keys, outbox, relay, now, log = console }) {
      const app = express();
      app.disable('x-powered-by');
@@@ -40,20 +92,17 @@ function createApp({ config, db, registry, items, ingest, scheduler, auth, keys,
      app.use(sourcesRouter({ db, registry, ingest, auth }));
      app.use(itemsRouter({ db, registry, items, auth, relay, now }));
  
--    app.get('/', (_req, res) => {
--        res.type('text/plain').send([
--            'OpenVibe.Sources: the source registry and ingestion workers for the network\'s publication products.',
--            'Every item carries its source, canonical URL, retrieval time, content hash and terms; a failed fetch is',
--            'recorded as a failure and never replaced with invented content.',
--            '',
--            'GET  /api/v1/sources, /api/v1/sources/:key, /api/v1/sources/:key/runs, /api/v1/runs, /api/v1/health   (sources.source.read)',
--            'GET  /api/v1/items?source=&category=&after=, /api/v1/items/:id                                          (sources.item.read)',
--            'POST/PATCH/DELETE /api/v1/sources[/:key], POST /api/v1/sources/:key/fetch, /items, DELETE /api/v1/items/:id (sources.source.manage)',
--            'GET  /api/health, /api/ready, /release.json',
--            '',
--            'Source: https://github.com/OpenVibers/OpenVibe.Sources',
--            '',
--        ].join('\n'));
++    // The public host (sources.openvibe.network) shows only this page, health, readiness and
++    // /release.json: Sources is internal. Browsers get a short honest HTML page, other clients the
++    // text route index. Never cached, never indexed.
++    app.get('/', (req, res) => {
++        res.setHeader('X-Robots-Tag', 'noindex, nofollow');
++        res.setHeader('Vary', 'Accept');
++        if (/\btext\/html\b/.test(String(req.get('accept') || ''))) {
++            res.setHeader('Content-Security-Policy', "default-src 'none'; style-src 'unsafe-inline'; base-uri 'none'; frame-ancestors 'none'");
++            return res.type('html').send(HOME_HTML);
++        }
++        return res.type('text/plain').send(TEXT_INDEX);
      });
  
      app.use((req, res) => http.sendProblem(res, 404, 'sources.not_found', { detail: `no route ${req.method} ${req.path}`, ctx: req.ov }));
ddiff --git a/test/public-host.test.js b/test/public-host.test.js
nnew file mode 100644
iindex 0000000..feb3b55
---- /dev/null
++++ b/test/public-host.test.js
@@@ -0,0 +1,53 @@
++'use strict';
++/**
++ * What sources.openvibe.network shows the public: a short honest page at / (HTML for browsers,
++ * the text route index otherwise), never cached or indexed, with no JavaScript. The nginx vhost
++ * (deploy/nginx/sources.openvibe.network.conf) keeps /api/v1/ loopback-only and answers every
++ * other path with a JSON 404.
++ */
++const assert = require('assert');
++const fs = require('fs');
++const path = require('path');
++const { boot, request, suite } = require('./helpers');
++
++const t = suite('public-host');
++let svc;
++
++t('boot', async () => { svc = await boot(); });
++
++t('browsers get a short honest HTML page: internal service, nothing to browse, noindex, no-store, no script', async () => {
++    const r = await request(svc.base, 'GET', '/', { headers: { Accept: 'text/html,application/xhtml+xml' } });
++    assert.strictEqual(r.status, 200);
++    assert.match(r.headers.get('content-type'), /text\/html/);
++    assert.match(r.text, /internal service/);
++    assert.match(r.text, /nothing to browse/);
++    assert.match(r.text, /<meta name="robots" content="noindex, nofollow">/);
++    assert.strictEqual(r.headers.get('x-robots-tag'), 'noindex, nofollow');
++    assert.strictEqual(r.headers.get('cache-control'), 'no-store');
++    assert.match(r.headers.get('content-security-policy'), /default-src 'none'/);
++    assert.ok(!/<script/i.test(r.text));
++    assert.ok(!/admin/i.test(r.text));
++});
++
++t('other clients get the text route index', async () => {
++    const r = await request(svc.base, 'GET', '/');
++    assert.match(r.headers.get('content-type'), /text\/plain/);
++    assert.match(r.text, /internal service/);
++    assert.match(r.text, /GET {2}\/api\/v1\/sources/);
++});
++
++t('the vhost: wildcard certificate, client address from $remote_addr, /metrics 404, loopback-only API, JSON 404', () => {
++    const conf = fs.readFileSync(path.join(__dirname, '..', 'deploy', 'nginx', 'sources.openvibe.network.conf'), 'utf8');
++    assert.match(conf, /ssl_certificate\s+\/etc\/letsencrypt\/live\/openvibe\.network\/fullchain\.pem;/);
++    for (const h of ['X-Real-IP', 'X-Forwarded-For', 'CF-Connecting-IP']) assert.match(conf, new RegExp(`proxy_set_header ${h} \\$remote_addr;`));
++    assert.ok(!/proxy_add_x_forwarded_for|\$http_cf_connecting_ip|\$http_x_forwarded_for/.test(conf), 'never a client-sent address');
++    assert.match(conf, /location = \/metrics \{ return 404; \}/);
++    assert.match(conf, /location \^~ \/api\/v1\/ \{\s*allow 127\.0\.0\.1;\s*allow ::1;\s*deny all;/);
++    assert.match(conf, /location \/ \{\s*default_type application\/problem\+json;\s*return 404 '\{/);
++    const body = /return 404 '(\{.*\})';/.exec(conf)[1];
++    assert.strictEqual(JSON.parse(body).status, 404);
++});
++
++t('shutdown', async () => { await svc.stop(); });
++
++t.run();