openvibers/openvibe.host Commits
Host tells search engines when a hosted site changes (plan T11, IndexNow): with INDEXNOW_KEY set, a production site's deploy, rollback, domain change or removal pings the engines for that site (its root and sitemap); sandbox sites never; unset, nothing is sent. A dashboard performance budget runs in CI. Tests green (verified).
Changes
@@@ -330,6 +330,10 @@ Server-rendered pages with the shared chrome (`openvibe-shared` v2.2.0: navbar a`GET /api/ready` (openvibe-shared/ready): `db` and `storage` are required; `network_jwks`, `events_relay` and `domain_checks` are optional and report degradation. `GET /release.json`. `GET /metrics` for direct loopback callers only (nginx also returns 404): HTTP golden signals by route template (tenant requests are one `tenant_site` series), process metrics, release info and `host_sites`.++### IndexNow (openvibe-shared/indexnow)++++`INDEXNOW_KEY` (`openvibe-shared` v2.2.0) turns on IndexNow. The key file is served at `/<key>.txt` on every host (the dashboard and each tenant site, so an engine can verify a ping that names a tenant host). When a tenant site's active deploy appears, changes (a new deploy or a rollback) or goes away (the site or its project is deleted), the site's page and its `/sitemap.xml` are queued to `api.indexnow.org`; a verified custom domain is queued when it starts serving. The module batches and debounces (one POST per 30 s window), filters to https URLs on the pinged host and never throws, so a failed ping cannot fail a deploy. A ready deploy that is not active (a draft) and a sandbox site (already `noindex`) never ping. Unset → off: no key route is mounted and nothing is sent. Tests and drills never set it (the drill's override in [`host.example.json`](host.example.json) sets `INDEXNOW_KEY` empty).++### Grants and registration (for the lead)- OpenVibe.Network OAuth client **`host`**, redirect `https://openvibe.host/auth/callback`, scope `profile theme`. The same client is the service principal `svc:host`.@@@ -345,7 +349,7 @@ The service runs on the host (loopback, since 2026-09-23) and the pending tenant2. **Wildcard certificate.** A DNS-01 challenge is required for `*.openvibe.host`, for example `certbot certonly --dns-cloudflare --dns-cloudflare-credentials /root/.secrets/certbot-cloudflare.ini -d openvibe.host -d '*.openvibe.host'`. The credentials file is root-only (0600), outside every repository, and never passed to Host. The certificate lands in `/etc/letsencrypt/live/openvibe.host/`; another location is passed with `--wildcard-cert <name|dir>`.3. **Network.** Register the OAuth client `host` and the grant above.4. **Code and state.** `sudo -u ubuntu git clone https://github.com/OpenVibers/OpenVibe.Host /opt/openvibe.host && cd /opt/openvibe.host && sudo -u ubuntu npm ci --omit=dev --no-audit --no-fund`. This checkout is the service; the root-owned `/usr/local/lib/openvibe-host` stays the `ovhost` CLI. The unit's state directory is `/var/lib/openvibe-host-api` (database and objects), deliberately **not** `/var/lib/openvibe-host`, which is `ovhost`'s own root-only state.--5. **Environment.** `/etc/openvibe/host.env` (0600), from [`.env.example`](.env.example): at least `BASE_URL=https://openvibe.host`, `HOST_SITES_DOMAIN=openvibe.host`, `OV_NETWORK_URL`, `OV_NETWORK_INTERNAL_URL`, `OV_OAUTH_CLIENT_ID=host`, `OV_OAUTH_CLIENT_SECRET`, `HOST_FORM_SECRET`, `EVENTS_URL`.++5. **Environment.** `/etc/openvibe/host.env` (0600), from [`.env.example`](.env.example): at least `BASE_URL=https://openvibe.host`, `HOST_SITES_DOMAIN=openvibe.host`, `OV_NETWORK_URL`, `OV_NETWORK_INTERNAL_URL`, `OV_OAUTH_CLIENT_ID=host`, `OV_OAUTH_CLIENT_SECRET`, `HOST_FORM_SECRET`, `EVENTS_URL`, `INDEXNOW_KEY` (optional).6. **Unit.** `sudo cp deploy/systemd/openvibe-host.service /etc/systemd/system/ && sudo systemctl daemon-reload && sudo systemctl enable --now openvibe-host`, then `curl -s http://127.0.0.1:4910/api/ready`.7. **nginx.** Add the `host` entry from [`host.example.json`](host.example.json) to `/etc/openvibe/host.json`, update `/usr/local/lib/openvibe-host`, then `sudo ovhost nginx tenants host` (review) and `sudo ovhost nginx tenants host --install` (writes `openvibe.host.conf` and `openvibe.host-custom-domains.conf`, `nginx -t`, reload; restored on failure). [`deploy/nginx/openvibe.host.conf`](deploy/nginx/openvibe.host.conf) is a reference copy.8. **Custom domains** (whenever tenants verify one): `sudo ovhost nginx tenants host` lists the verified domains still waiting for a certificate. For each, `sudo certbot certonly --webroot -w /var/www/certbot -d <hostname>` (the generated port-80 block already answers the ACME challenge), then `sudo ovhost nginx tenants host --install` again. Until then, the domain is verified but only answers over HTTP with the challenge. Certificates never pass through the Host API.@@@ -374,6 +378,8 @@ The service runs on the host (loopback, since 2026-09-23) and the pending tenant- **Lifecycle, headers, events** (`test/host-lifecycle.test.js`, `test/host-contracts-events.test.js`): content types, ETag/304, ranges, immutable caching, CSP, nosniff, custom 404, 405; valid event envelopes delivered by the relay with a Network service token; the proposals match the guarded routes.- **Tenant vhosts** (`test/nginx-tenants.test.js`): the wildcard vhost with the certificate path parameter, HTTPS only for verified domains that have a certificate, hostile database values refused before they reach nginx, transactional install that removes the pending vhost in the same change, `www.` redirect, client address headers from `$remote_addr` only, no key file read.- **Abuse controls** (`test/host-abuse.test.js`): staff takedowns (451 with Clear-Site-Data on every host, content kept, members cannot publish or delete around them, lift), certificate-validation paths refused, uploads validated two at a time (503 + Retry-After), the disk floor (507), the per-person project cap, fingerprinted assets capped at an hour at a CDN.++- **IndexNow** (`test/indexnow.test.js`): off without `INDEXNOW_KEY` (no key route, nothing sent); with a key the key file is served at `/<key>.txt` as `text/plain` on the dashboard and on tenant hosts; activating a deploy pings the site page and its sitemap, a rollback or a site deletion pings again, a ready-but-not-active deploy and a sandbox site never ping.++- **Home-page size budget** (`test/perf-budget.test.js`): the server on a fresh database, the home page measured with `openvibe-shared/perf-budget` — html 15.9 KB (4.3 br), js 4 files 212.2 KB (49.9 br), css 1 file 3.1 KB (0.9 br), 0 external — within the committed budgets.- The isolation test also covers cross-origin reads from a tenant page (no CORS grant anywhere), ETag/Range existence oracles, delegated service tokens, app principals and the dashboard.Not demonstrated yet: any of this on the production host.@@@ -1002,9 +1002,16 @@"HOST_WORKER": "off","EVENTS_URL": "","OV_OAUTH_CLIENT_SECRET": "",++ "INDEXNOW_KEY": "","HOST_STORAGE_DIR": "{tmp}/objects","VALKEY_URL": "redis://127.0.0.1:9/0"},++ "requires": [++ {++ "file": "server/config.js",++ "contains": "INDEXNOW_KEY"++ }++ ],"dirs": ["{tmp}/objects"],@@@ -19,6 +19,7 @@ const cookieParser = require('cookie-parser');const rateLimit = require('express-rate-limit');const contracts = require('openvibe-contracts');const sharedMetrics = require('openvibe-shared/metrics');++const { createIndexNow } = require('openvibe-shared/indexnow');const configLib = require('./config');const { openStore } = require('./db');@@@ -73,13 +74,40 @@ async function createApp(opts = {}) {const blobs = createBlobStore(opts.storageDir || config.storageDir);require('./http/errors').setLogger(log);++ // IndexNow (openvibe-shared/indexnow): created once at boot from INDEXNOW_KEY. Unset → off, nothing++ // mounted, nothing sent; tests and drills never set it. The key file is served on every host (it is++ // mounted below, before the tenant dispatch), and one module per public host keeps a tenant site's++ // URL on its own host, which the protocol (and the module) requires. Tests inject a single object++ // through opts.indexnow (a spy) and every ping goes through it.++ const indexnow = opts.indexnow !== undefined ? opts.indexnow : createIndexNow({++ host: config.baseUrl, key: config.indexnow.key, ...(opts.fetchImpl ? { fetch: opts.fetchImpl } : {}), log,++ });++ const indexnowHosts = new Map();++ function indexnowFor(hostname) {++ if (!indexnowHosts.has(hostname)) indexnowHosts.set(hostname, createIndexNow({++ host: hostname, key: config.indexnow.key, ...(opts.fetchImpl ? { fetch: opts.fetchImpl } : {}), log,++ }));++ return indexnowHosts.get(hostname);++ }++ /**++ * Queue a page that appeared, changed or left the index: one absolute URL per path on `hostname`,++ * sent as a debounced batch. With an injected module (tests) it is used for every host. Never throws.++ */++ function announce(hostname, paths) {++ if (!indexnow.enabled || !hostname) return;++ const u = new URL(config.baseUrl);++ const origin = `${u.protocol}//${hostname}${u.port ? `:${u.port}` : ''}`;++ const target = opts.indexnow !== undefined ? indexnow : indexnowFor(hostname);++ target.pingSoon(paths.map((p) => new URL(p, origin).href));++ }++const outbox = createHostOutbox({ db: store.db, config, fetchImpl: opts.fetchImpl, now: store.now, log });const access = createAccess({ store });const takedowns = createTakedowns({ store });-- const projects = createProjects({ store, config, access, blobs, takedowns, log });-- const sites = createSites({ store, config, access, projects, takedowns });-- const deploys = createDeploys({ store, config, access, projects, sites, blobs, outbox, takedowns, log });-- const domains = createDomains({ store, config, access, projects, sites, outbox, resolver: opts.resolver, log });++ const projects = createProjects({ store, config, access, blobs, takedowns, log, indexnow: announce });++ const sites = createSites({ store, config, access, projects, takedowns, indexnow: announce });++ const deploys = createDeploys({ store, config, access, projects, sites, blobs, outbox, takedowns, log, indexnow: announce });++ const domains = createDomains({ store, config, access, projects, sites, outbox, resolver: opts.resolver, log, indexnow: announce });const auth = opts.auth || createSsoClient({site: 'host',baseUrl: config.baseUrl,@@@ -99,7 +127,7 @@ async function createApp(opts = {}) {const uploadGate = createUploadGate(config.uploads.maxConcurrent);const worker = createWorker({ config, store, domains, blobs, outbox, log });-- const ctx = { config, store, blobs, outbox, access, takedowns, projects, sites, deploys, domains, auth, viewers, tenant, worker, uploadGate, log };++ const ctx = { config, store, blobs, outbox, access, takedowns, projects, sites, deploys, domains, auth, viewers, tenant, worker, uploadGate, indexnow, log };const app = express();app.disable('x-powered-by');@@@ -127,6 +155,11 @@ async function createApp(opts = {}) {app.use(httpMetrics.middleware);++ // GET /<key>.txt — the IndexNow key file (openvibe-shared/indexnow). Served before the host dispatch++ // so it answers on the dashboard host and on every tenant host (an engine fetches it from the host a++ // ping names). Mounted only with a key; it answers that one path and falls through for everything else.++ if (indexnow.enabled) app.use(indexnow.keyFile);++// ── Host dispatch: tenant sites never reach anything below ──// resolve/handle read the database (ADR-035: async), so a rejection here must reach the error// handler below: Express 4 does not catch a rejected async handler, which would hang the request.@@@ -45,7 +45,9 @@ function load(env = process.env) {trustProxy: env.TRUST_PROXY != null ? Number(env.TRUST_PROXY) : 1, // nginx (realip) → Node// PostgreSQL (ADR-035): DATABASE_URL serves (PgBouncer), DATABASE_DIRECT_URL migrates (owner role).-- db: { url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '' },++ // HOST_PGLITE_DIR: where the development-only embedded database lives (default data/pglite); tests++ // point it at a temp directory so a spawned server gets a database of its own.++ db: { url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '', pgliteDir: env.HOST_PGLITE_DIR || '' },valkey: { url: env.VALKEY_URL || '', prefix: env.VALKEY_PREFIX || 'ov:host:' },// The SQLite file of releases before the switch: read once by scripts/migrate-to-postgres.js.dbPath: env.HOST_DB_PATH || './data/host.db',@@@ -65,6 +67,12 @@ function load(env = process.env) {// Signs the dashboard form tokens (CSRF). Unset: a random per-process key.formSecret: env.HOST_FORM_SECRET || '',++ // IndexNow (openvibe-shared/indexnow): a key makes search engines recrawl a tenant site the++ // moment its active deploy appears, changes or goes away (the key file is served at++ // /<key>.txt on every host). Unset → off: no key file, nothing sent. Tests and drills never++ // set it (the drill sets INDEXNOW_KEY empty in host.example.json).++ indexnow: { key: String(env.INDEXNOW_KEY || '').trim() },++// OpenVibe.Events: the outbox relay runs only when EVENTS_URL and the client secret are set.events: {url: trim(env.EVENTS_URL || ''),@@@ -36,10 +36,13 @@ const CHARTER_TABLES = ['host_projects', 'host_project_members', 'host_quotas',*/async function openDb(config, { log = console, registry } = {}) {if (!config.db.url) {++ // HOST_PGLITE_DIR lets a test (or a second local process) point the embedded database somewhere++ // of its own; production never takes this path.++ const pgliteDir = config.db.pgliteDir || DEV_PGLITE;if (config.isProduction) throw new Error('DATABASE_URL is not set: production serves from PostgreSQL (OpenVibe.Host roles/data add-service.sh host)');-- log.warn(`[Host] DATABASE_URL unset: embedded PGlite database in ${DEV_PGLITE} (development only, one process)`);-- fs.mkdirSync(DEV_PGLITE, { recursive: true });-- const db = createDb({ pglite: DEV_PGLITE, service: 'host', registry, log });++ log.warn(`[Host] DATABASE_URL unset: embedded PGlite database in ${pgliteDir} (development only, one process)`);++ fs.mkdirSync(pgliteDir, { recursive: true });++ const db = createDb({ pglite: pgliteDir, service: 'host', registry, log });await db.migrate({ dir: MIGRATIONS, log });return db;}@@@ -23,8 +23,14 @@ const { principalOf, actorRef } = require('./access');const DAY = 24 * 3600 * 1000;const fmtBytes = (n) => (n >= 1048576 ? `${(n / 1048576).toFixed(1)} MiB` : n >= 1024 ? `${(n / 1024).toFixed(1)} KiB` : `${n} B`);--function createDeploys({ store, config = null, access, projects, sites, blobs, outbox, takedowns, log = console }) {++function createDeploys({ store, config = null, access, projects, sites, blobs, outbox, takedowns, log = console, indexnow = null }) {const { db } = store;++ // IndexNow: a site's public page appeared, changed or left the index (never a sandbox site, whose++ // responses carry X-Robots-Tag noindex). `indexnow(hostname, paths)` queues one debounced batch.++ const announce = typeof indexnow === 'function' ? indexnow : () => {};++ const announceSite = (site, project, paths = ['/', '/sitemap.xml']) => {++ if (!project || project.environment !== 'sandbox') announce(sites.defaultHostname(site.name), paths);++ };const q = {byId: db.prepare('SELECT * FROM host_deploys WHERE id = ?'),forSite: db.prepare("SELECT * FROM host_deploys WHERE site_id = ? AND state <> 'deleted' ORDER BY created_at DESC, seq DESC LIMIT ?"),@@@ -174,6 +180,7 @@ function createDeploys({ store, config = null, access, projects, sites, blobs, oawait writeLog(id, lines, now);});outbox.kick();++ if (activated && activated.changed) announceSite(site, project);return { deploy: await get(id), log: await q.logFor.all(id), activated };}@@@ -199,16 +206,17 @@ function createDeploys({ store, config = null, access, projects, sites, blobs, o}async function activate(viewer, id, { expectedActive, traceparent } = {}) {-- const { deploy, site } = await load(viewer, id, 'deploy');++ const { deploy, site, project } = await load(viewer, id, 'deploy');await takedowns.assertOpen(site);const out = await store.tx(async () => await switchPointer(viewer, site, deploy, 'activate', { expectedActive, traceparent }));outbox.kick();++ if (out.changed) announceSite(site, project);return out;}/** To `deployId`, or else to the most recent previously active deploy that still exists. */async function rollback(viewer, siteId, { deployId, expectedActive, traceparent } = {}) {-- const { site } = await sites.load(viewer, siteId, 'deploy');++ const { site, project } = await sites.load(viewer, siteId, 'deploy');await takedowns.assertOpen(site);const out = await store.tx(async () => {const current = (await db.prepare('SELECT active_deploy_id FROM host_sites WHERE id = ?').get(site.id)).active_deploy_id;@@@ -230,6 +238,7 @@ function createDeploys({ store, config = null, access, projects, sites, blobs, oreturn await switchPointer(viewer, site, target, 'rollback', { expectedActive, traceparent });});outbox.kick();++ if (out.changed) announceSite(site, project);return out;}@@@ -51,8 +51,10 @@ function defaultResolver(servers) {return { resolveTxt: (name) => r.resolveTxt(name) };}--function createDomains({ store, config, access, projects, sites, outbox, resolver = null, log = console }) {++function createDomains({ store, config, access, projects, sites, outbox, resolver = null, log = console, indexnow = null }) {const { db } = store;++ // IndexNow: a verified custom domain starts serving the site; tell the engines its page is up.++ const announce = typeof indexnow === 'function' ? indexnow : () => {};const dnsResolver = resolver || defaultResolver(config.domains.dnsServers);const reserved = firstPartyDomains(config);const q = {@@@ -165,6 +167,7 @@ function createDomains({ store, config, access, projects, sites, outbox, resolvethrow err;}outbox.kick();++ announce(domain.hostname, ['/', '/sitemap.xml']);return await q.byId.get(domain.id);}@@@ -18,8 +18,9 @@ const PRINCIPAL_RE = /^(usr_[0-9A-HJKMNP-TV-Z]{26}|app:app_[0-9A-HJKMNP-TV-Z]{26const QUOTA_FIELDS = { storage_bytes: 'storageBytes', deploys_per_day: 'deploysPerDay', max_files: 'maxFiles', max_file_bytes: 'maxFileBytes', sites: 'sites', custom_domains: 'customDomains' };const DAY = 24 * 3600 * 1000;--function createProjects({ store, config, access, blobs, takedowns, log = console }) {++function createProjects({ store, config, access, blobs, takedowns, log = console, indexnow = null }) {const { db } = store;++ const announce = typeof indexnow === 'function' ? indexnow : () => {};const q = {byId: db.prepare('SELECT * FROM host_projects WHERE id = ?'),forPrincipal: db.prepare(`SELECT p.*, m.role FROM host_projects p JOIN host_project_members m ON m.project_id = p.id@@@ -149,8 +150,9 @@ function createProjects({ store, config, access, blobs, takedowns, log = consoleawait access.authorize(project, viewer, 'own');await takedowns.assertDeletable(viewer, { projectId: project.id });const now = store.now();++ const doomed = await sites.listForProject(project.id);await store.tx(async () => {-- for (const s of await sites.listForProject(project.id)) await sites.markDeleted(s, now);++ for (const s of doomed) await sites.markDeleted(s, now);await db.prepare("UPDATE host_deploys SET state = 'deleted', deleted_at = ? WHERE project_id = ? AND state <> 'deleted'").run(now, project.id);await db.prepare("DELETE FROM host_deploy_files WHERE deploy_id IN (SELECT id FROM host_deploys WHERE project_id = ?)").run(project.id);await db.prepare('DELETE FROM host_blobs WHERE project_id = ?').run(project.id);@@@ -158,6 +160,8 @@ function createProjects({ store, config, access, blobs, takedowns, log = consoleawait q.markDeleted.run(now, now, project.id);});try { blobs.removeProject(project.id); } catch (err) { log.warn('[Host] could not remove project objects:', err.message); }++ // IndexNow: every site of the project left the index (never a sandbox project, which is noindex).++ if (project.environment !== 'sandbox') for (const s of doomed) announce(`${s.name}.${config.sitesDomain}`, ['/', '/sitemap.xml']);return { deleted: true };}@@@ -21,7 +21,7 @@ const RESERVED = new Set([]);const HOLD_MS = 30 * 24 * 3600 * 1000;--function createSites({ store, config, access, projects, takedowns }) {++function createSites({ store, config, access, projects, takedowns, indexnow = null }) {const { db } = store;const reserved = new Set([...RESERVED, ...config.sites.extraReservedNames]);const q = {@@@ -37,6 +37,11 @@ function createSites({ store, config, access, projects, takedowns }) {};const defaultHostname = (name) => `${name}.${config.sitesDomain}`;++ // IndexNow: the site's public pages left the index (never a sandbox site, which is noindex).++ const announce = typeof indexnow === 'function' ? indexnow : () => {};++ const announceSite = (site, project) => {++ if (!project || project.environment !== 'sandbox') announce(defaultHostname(site.name), ['/', '/sitemap.xml']);++ };async function get(id) {return isId('site', id) ? await q.byId.get(id) || null : null;@@@ -105,6 +110,7 @@ function createSites({ store, config, access, projects, takedowns }) {freed = await deploys.collectGarbage(project.id);});deploys.unlinkBlobs(project.id, freed);++ announceSite(site, project);return { deleted: true };}nnew file mode 100644@@@ -0,0 +1,104 @@++'use strict';++/**++ * IndexNow (openvibe-shared/indexnow): INDEXNOW_KEY unset → the feature is off (no key route, nothing++ * sent). With a key the key file is served at /<key>.txt as text/plain on every host, and a tenant++ * site whose active deploy appears, changes or goes away pings the engines with the site's page and++ * its sitemap. A ready deploy that is not active (a draft) never pings; a sandbox site (noindex) never++ * pings.++ */++const assert = require('assert');++const { boot, check, done, DASHBOARD } = require('./stageb/boot');++++const KEY = 'k'.repeat(32);++++/** A spy in place of the module's HTTP send: records every pingSoon batch (the repo's test pattern). */++function makeSpy() {++ const pings = [];++ return {++ pings,++ module: {++ enabled: true,++ keyFile: (_req, _res, next) => next(),++ pingSoon: (urls) => { const a = Array.isArray(urls) ? urls : [urls]; pings.push(...a); return a.length; },++ ping: async () => ({ sent: 0, status: 0 }),++ flush: async () => ({ sent: 0, status: 0 }),++ },++ };++}++++(async () => {++ const off = await boot();++ await check('without a key IndexNow is off: no key route and nothing sent', async () => {++ assert.strictEqual(off.ctx.indexnow.enabled, false);++ const res = await off.get(DASHBOARD, `/${KEY}.txt`);++ assert.strictEqual(res.status, 404, res.text);++ });++ await off.close();++++ const on = await boot({ env: { INDEXNOW_KEY: KEY } });++ await check('with a key the key file answers text/plain with the key (dashboard and tenant hosts)', async () => {++ assert.strictEqual(on.ctx.indexnow.enabled, true);++ const dash = await on.get(DASHBOARD, `/${KEY}.txt`);++ assert.strictEqual(dash.status, 200, dash.text);++ assert.match(dash.headers['content-type'], /text\/plain/);++ assert.strictEqual(dash.text, KEY);++ const tenant = await on.get('anything.openvibe.host', `/${KEY}.txt`);++ assert.strictEqual(tenant.status, 200, tenant.text);++ assert.strictEqual(tenant.text, KEY);++ });++ await on.close();++++ const s = makeSpy();++ const t = await boot({ indexnow: s.module });++ const alice = t.user('alice', { display_name: 'Alice' });++ const project = await t.project(alice, 'Widgets');++ const site = await t.site(alice, project.id, 'widgets');++ const origin = 'https://widgets.openvibe.host';++ let deploy;++++ await check('a ready deploy that is not active (a draft) never pings', async () => {++ const r = await t.upload(alice, site.id, { 'index.html': '<h1>Widgets</h1>' }, { activate: false });++ assert.strictEqual(r.status, 201, r.text);++ deploy = r.json().deploy;++ assert.deepStrictEqual(s.pings, []);++ });++++ await check('activating the deploy pings the site page and the sitemap', async () => {++ const r = await t.api('POST', `/api/v1/deploys/${deploy.id}/activate`, { as: alice, json: {} });++ assert.strictEqual(r.status, 200, r.text);++ assert.ok(r.json().changed, r.text);++ assert.ok(s.pings.includes(`${origin}/`), JSON.stringify(s.pings));++ assert.ok(s.pings.includes(`${origin}/sitemap.xml`), JSON.stringify(s.pings));++ });++++ await check('a rollback pings the page and the sitemap again', async () => {++ const r2 = await t.upload(alice, site.id, { 'index.html': '<h1>Widgets v2</h1>' }, { activate: true });++ assert.strictEqual(r2.status, 201, r2.text);++ s.pings.length = 0;++ const rb = await t.api('POST', `/api/v1/sites/${site.id}/rollback`, { as: alice, json: {} });++ assert.strictEqual(rb.status, 200, rb.text);++ assert.strictEqual(rb.json().changed, true);++ assert.ok(s.pings.includes(`${origin}/`), JSON.stringify(s.pings));++ assert.ok(s.pings.includes(`${origin}/sitemap.xml`), JSON.stringify(s.pings));++ });++++ await check('deleting the site pings the page and the sitemap again', async () => {++ s.pings.length = 0;++ const r = await t.api('DELETE', `/api/v1/sites/${site.id}`, { as: alice, json: {} });++ assert.strictEqual(r.status, 200, r.text);++ assert.ok(s.pings.includes(`${origin}/`), JSON.stringify(s.pings));++ assert.ok(s.pings.includes(`${origin}/sitemap.xml`), JSON.stringify(s.pings));++ });++++ await check('a sandbox site (noindex) never pings, even when it activates', async () => {++ const sandboxProject = await t.project(alice, 'Sandbox', { environment: 'sandbox' });++ const sandboxSite = await t.site(alice, sandboxProject.id, 'playground');++ s.pings.length = 0;++ const r = await t.upload(alice, sandboxSite.id, { 'index.html': '<h1>Play</h1>' }, { activate: true });++ assert.strictEqual(r.status, 201, r.text);++ assert.deepStrictEqual(s.pings, []);++ });++ await t.close();++++ done();++})().catch((err) => { console.error(err); process.exit(1); });nnew file mode 100644@@@ -0,0 +1,64 @@++'use strict';++// Size budgets for openvibe.host's home page (roadmap WS-T task 1, openvibe-shared/perf-budget): the++// server as it runs (a fresh database), measured without a browser. Budgets sit a little above the++// 2026-09-29 measurement; raising one is a decision to state in the commit.++// node test/perf-budget.test.js++const assert = require('assert');++const fs = require('fs');++const net = require('net');++const os = require('os');++const path = require('path');++const { spawn } = require('child_process');++const { measure, check, format } = require('openvibe-shared/perf-budget');++++const BUDGETS = {++ htmlRawKB: 18, // measured 15.9 (fresh database)++ htmlBrotliKB: 5, // 4.3++ jsFiles: 5, // 4++ jsRawKB: 230, // 212.2++ jsBrotliKB: 53, // 49.9++ cssFiles: 2, // 1++ cssRawKB: 5, // 3.1++ cssBrotliKB: 2, // 0.9++ externalFiles: 1, // 0++};++++const freePort = () => new Promise((resolve) => { const s = net.createServer(); s.listen(0, '127.0.0.1', () => { const { port } = s.address(); s.close(() => resolve(port)); }); });++++(async () => {++ // A database of its own (not the shared dev PGlite in data/pglite, and not whatever the caller's++ // DATABASE_URL names), so the measurement is the server on a fresh database and the run leaves nothing.++ const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-host-budget-'));++ const pgliteDir = path.join(dir, 'pglite');++ const port = await freePort();++ const child = spawn(process.execPath, [path.join(__dirname, '..', 'server', 'index.js')], {++ cwd: path.join(__dirname, '..'),++ env: {++ ...process.env, PORT: String(port), HOST: '127.0.0.1', NODE_ENV: 'test',++ BASE_URL: 'https://openvibe.host', HOST_SITES_DOMAIN: 'openvibe.host', COOKIE_SECURE: 'false',++ DATABASE_URL: '', DATABASE_DIRECT_URL: '', VALKEY_URL: '', HOST_PGLITE_DIR: pgliteDir,++ HOST_STORAGE_DIR: path.join(dir, 'objects'), HOST_WORKER: 'off', HOST_MIN_FREE_BYTES: '0',++ },++ stdio: ['ignore', 'ignore', 'pipe'],++ });++ let stderr = '';++ child.stderr.on('data', (d) => { stderr = (stderr + d).slice(-2000); });++ const base = `http://127.0.0.1:${port}`;++ try {++ let up = false;++ for (let i = 0; i < 150 && !up; i++) {++ up = await fetch(`${base}/api/health`).then((r) => r.ok).catch(() => false);++ if (!up) await new Promise((r) => setTimeout(r, 100));++ }++ assert.ok(up, `the server did not start:\n${stderr}`);++ assert.ok(fs.existsSync(pgliteDir), 'the server did not use the isolated database (HOST_PGLITE_DIR)');++ const m = await measure({ base });++ const over = check(m, BUDGETS);++ assert.deepStrictEqual(over, [], format(m, over));++ console.log(format(m));++ console.log('perf budget: all checks passed');++ } finally {++ child.kill();++ fs.rmSync(dir, { recursive: true, force: true });++ }++})().catch((err) => { console.error(err); process.exitCode = 1; });@@@ -64,7 +64,7 @@ async function boot(opts = {}) {async function start() {const config = configLib.load(env);-- built = await createApp({ config, store: createStore(testdb.db, { now: clock.now }), now: clock.now, log: quiet, resolver, fetchImpl: opts.fetchImpl });++ built = await createApp({ config, store: createStore(testdb.db, { now: clock.now }), now: clock.now, log: quiet, resolver, fetchImpl: opts.fetchImpl, indexnow: opts.indexnow });server = await new Promise((resolve) => { const s = http.createServer(built.app); s.listen(0, '127.0.0.1', () => resolve(s)); });t.port = server.address().port;t.app = built.app;
@@ -330,6 +330,10 @@ Server-rendered pages with the shared chrome (`openvibe-shared` v2.2.0: navbar a
@@ -330,6 +330,10 @@ Server-rendered pages with the shared chrome (`openvibe-shared` v2.2.0: navbar a
`GET /api/ready` (openvibe-shared/ready): `db` and `storage` are required; `network_jwks`, `events_relay` and `domain_checks` are optional and report degradation. `GET /release.json`. `GET /metrics` for direct loopback callers only (nginx also returns 404): HTTP golden signals by route template (tenant requests are one `tenant_site` series), process metrics, release info and `host_sites`.
`GET /api/ready` (openvibe-shared/ready): `db` and `storage` are required; `network_jwks`, `events_relay` and `domain_checks` are optional and report degradation. `GET /release.json`. `GET /metrics` for direct loopback callers only (nginx also returns 404): HTTP golden signals by route template (tenant requests are one `tenant_site` series), process metrics, release info and `host_sites`.
+### IndexNow (openvibe-shared/indexnow)
+
+`INDEXNOW_KEY` (`openvibe-shared` v2.2.0) turns on IndexNow. The key file is served at `/<key>.txt` on every host (the dashboard and each tenant site, so an engine can verify a ping that names a tenant host). When a tenant site's active deploy appears, changes (a new deploy or a rollback) or goes away (the site or its project is deleted), the site's page and its `/sitemap.xml` are queued to `api.indexnow.org`; a verified custom domain is queued when it starts serving. The module batches and debounces (one POST per 30 s window), filters to https URLs on the pinged host and never throws, so a failed ping cannot fail a deploy. A ready deploy that is not active (a draft) and a sandbox site (already `noindex`) never ping. Unset → off: no key route is mounted and nothing is sent. Tests and drills never set it (the drill's override in [`host.example.json`](host.example.json) sets `INDEXNOW_KEY` empty).
+
### Grants and registration (for the lead)
### Grants and registration (for the lead)
- OpenVibe.Network OAuth client **`host`**, redirect `https://openvibe.host/auth/callback`, scope `profile theme`. The same client is the service principal `svc:host`.
- OpenVibe.Network OAuth client **`host`**, redirect `https://openvibe.host/auth/callback`, scope `profile theme`. The same client is the service principal `svc:host`.
@@ -345,7 +349,7 @@ The service runs on the host (loopback, since 2026-09-23) and the pending tenant
@@ -345,7 +349,7 @@ The service runs on the host (loopback, since 2026-09-23) and the pending tenant
2. **Wildcard certificate.** A DNS-01 challenge is required for `*.openvibe.host`, for example `certbot certonly --dns-cloudflare --dns-cloudflare-credentials /root/.secrets/certbot-cloudflare.ini -d openvibe.host -d '*.openvibe.host'`. The credentials file is root-only (0600), outside every repository, and never passed to Host. The certificate lands in `/etc/letsencrypt/live/openvibe.host/`; another location is passed with `--wildcard-cert <name|dir>`.
2. **Wildcard certificate.** A DNS-01 challenge is required for `*.openvibe.host`, for example `certbot certonly --dns-cloudflare --dns-cloudflare-credentials /root/.secrets/certbot-cloudflare.ini -d openvibe.host -d '*.openvibe.host'`. The credentials file is root-only (0600), outside every repository, and never passed to Host. The certificate lands in `/etc/letsencrypt/live/openvibe.host/`; another location is passed with `--wildcard-cert <name|dir>`.
3. **Network.** Register the OAuth client `host` and the grant above.
3. **Network.** Register the OAuth client `host` and the grant above.
4. **Code and state.** `sudo -u ubuntu git clone https://github.com/OpenVibers/OpenVibe.Host /opt/openvibe.host && cd /opt/openvibe.host && sudo -u ubuntu npm ci --omit=dev --no-audit --no-fund`. This checkout is the service; the root-owned `/usr/local/lib/openvibe-host` stays the `ovhost` CLI. The unit's state directory is `/var/lib/openvibe-host-api` (database and objects), deliberately **not** `/var/lib/openvibe-host`, which is `ovhost`'s own root-only state.
4. **Code and state.** `sudo -u ubuntu git clone https://github.com/OpenVibers/OpenVibe.Host /opt/openvibe.host && cd /opt/openvibe.host && sudo -u ubuntu npm ci --omit=dev --no-audit --no-fund`. This checkout is the service; the root-owned `/usr/local/lib/openvibe-host` stays the `ovhost` CLI. The unit's state directory is `/var/lib/openvibe-host-api` (database and objects), deliberately **not** `/var/lib/openvibe-host`, which is `ovhost`'s own root-only state.
-5. **Environment.** `/etc/openvibe/host.env` (0600), from [`.env.example`](.env.example): at least `BASE_URL=https://openvibe.host`, `HOST_SITES_DOMAIN=openvibe.host`, `OV_NETWORK_URL`, `OV_NETWORK_INTERNAL_URL`, `OV_OAUTH_CLIENT_ID=host`, `OV_OAUTH_CLIENT_SECRET`, `HOST_FORM_SECRET`, `EVENTS_URL`.
+5. **Environment.** `/etc/openvibe/host.env` (0600), from [`.env.example`](.env.example): at least `BASE_URL=https://openvibe.host`, `HOST_SITES_DOMAIN=openvibe.host`, `OV_NETWORK_URL`, `OV_NETWORK_INTERNAL_URL`, `OV_OAUTH_CLIENT_ID=host`, `OV_OAUTH_CLIENT_SECRET`, `HOST_FORM_SECRET`, `EVENTS_URL`, `INDEXNOW_KEY` (optional).
6. **Unit.** `sudo cp deploy/systemd/openvibe-host.service /etc/systemd/system/ && sudo systemctl daemon-reload && sudo systemctl enable --now openvibe-host`, then `curl -s http://127.0.0.1:4910/api/ready`.
6. **Unit.** `sudo cp deploy/systemd/openvibe-host.service /etc/systemd/system/ && sudo systemctl daemon-reload && sudo systemctl enable --now openvibe-host`, then `curl -s http://127.0.0.1:4910/api/ready`.
7. **nginx.** Add the `host` entry from [`host.example.json`](host.example.json) to `/etc/openvibe/host.json`, update `/usr/local/lib/openvibe-host`, then `sudo ovhost nginx tenants host` (review) and `sudo ovhost nginx tenants host --install` (writes `openvibe.host.conf` and `openvibe.host-custom-domains.conf`, `nginx -t`, reload; restored on failure). [`deploy/nginx/openvibe.host.conf`](deploy/nginx/openvibe.host.conf) is a reference copy.
7. **nginx.** Add the `host` entry from [`host.example.json`](host.example.json) to `/etc/openvibe/host.json`, update `/usr/local/lib/openvibe-host`, then `sudo ovhost nginx tenants host` (review) and `sudo ovhost nginx tenants host --install` (writes `openvibe.host.conf` and `openvibe.host-custom-domains.conf`, `nginx -t`, reload; restored on failure). [`deploy/nginx/openvibe.host.conf`](deploy/nginx/openvibe.host.conf) is a reference copy.
8. **Custom domains** (whenever tenants verify one): `sudo ovhost nginx tenants host` lists the verified domains still waiting for a certificate. For each, `sudo certbot certonly --webroot -w /var/www/certbot -d <hostname>` (the generated port-80 block already answers the ACME challenge), then `sudo ovhost nginx tenants host --install` again. Until then, the domain is verified but only answers over HTTP with the challenge. Certificates never pass through the Host API.
8. **Custom domains** (whenever tenants verify one): `sudo ovhost nginx tenants host` lists the verified domains still waiting for a certificate. For each, `sudo certbot certonly --webroot -w /var/www/certbot -d <hostname>` (the generated port-80 block already answers the ACME challenge), then `sudo ovhost nginx tenants host --install` again. Until then, the domain is verified but only answers over HTTP with the challenge. Certificates never pass through the Host API.
@@ -374,6 +378,8 @@ The service runs on the host (loopback, since 2026-09-23) and the pending tenant
@@ -374,6 +378,8 @@ The service runs on the host (loopback, since 2026-09-23) and the pending tenant
- **Lifecycle, headers, events** (`test/host-lifecycle.test.js`, `test/host-contracts-events.test.js`): content types, ETag/304, ranges, immutable caching, CSP, nosniff, custom 404, 405; valid event envelopes delivered by the relay with a Network service token; the proposals match the guarded routes.
- **Lifecycle, headers, events** (`test/host-lifecycle.test.js`, `test/host-contracts-events.test.js`): content types, ETag/304, ranges, immutable caching, CSP, nosniff, custom 404, 405; valid event envelopes delivered by the relay with a Network service token; the proposals match the guarded routes.
- **Tenant vhosts** (`test/nginx-tenants.test.js`): the wildcard vhost with the certificate path parameter, HTTPS only for verified domains that have a certificate, hostile database values refused before they reach nginx, transactional install that removes the pending vhost in the same change, `www.` redirect, client address headers from `$remote_addr` only, no key file read.
- **Tenant vhosts** (`test/nginx-tenants.test.js`): the wildcard vhost with the certificate path parameter, HTTPS only for verified domains that have a certificate, hostile database values refused before they reach nginx, transactional install that removes the pending vhost in the same change, `www.` redirect, client address headers from `$remote_addr` only, no key file read.
- **Abuse controls** (`test/host-abuse.test.js`): staff takedowns (451 with Clear-Site-Data on every host, content kept, members cannot publish or delete around them, lift), certificate-validation paths refused, uploads validated two at a time (503 + Retry-After), the disk floor (507), the per-person project cap, fingerprinted assets capped at an hour at a CDN.
- **Abuse controls** (`test/host-abuse.test.js`): staff takedowns (451 with Clear-Site-Data on every host, content kept, members cannot publish or delete around them, lift), certificate-validation paths refused, uploads validated two at a time (503 + Retry-After), the disk floor (507), the per-person project cap, fingerprinted assets capped at an hour at a CDN.
+- **IndexNow** (`test/indexnow.test.js`): off without `INDEXNOW_KEY` (no key route, nothing sent); with a key the key file is served at `/<key>.txt` as `text/plain` on the dashboard and on tenant hosts; activating a deploy pings the site page and its sitemap, a rollback or a site deletion pings again, a ready-but-not-active deploy and a sandbox site never ping.
+- **Home-page size budget** (`test/perf-budget.test.js`): the server on a fresh database, the home page measured with `openvibe-shared/perf-budget` — html 15.9 KB (4.3 br), js 4 files 212.2 KB (49.9 br), css 1 file 3.1 KB (0.9 br), 0 external — within the committed budgets.
- The isolation test also covers cross-origin reads from a tenant page (no CORS grant anywhere), ETag/Range existence oracles, delegated service tokens, app principals and the dashboard.
- The isolation test also covers cross-origin reads from a tenant page (no CORS grant anywhere), ETag/Range existence oracles, delegated service tokens, app principals and the dashboard.
Not demonstrated yet: any of this on the production host.
Not demonstrated yet: any of this on the production host.
@@ -1002,9 +1002,16 @@
@@ -1002,9 +1002,16 @@
"HOST_WORKER": "off",
"HOST_WORKER": "off",
"EVENTS_URL": "",
"EVENTS_URL": "",
"OV_OAUTH_CLIENT_SECRET": "",
"OV_OAUTH_CLIENT_SECRET": "",
+ "INDEXNOW_KEY": "",
"HOST_STORAGE_DIR": "{tmp}/objects",
"HOST_STORAGE_DIR": "{tmp}/objects",
"VALKEY_URL": "redis://127.0.0.1:9/0"
"VALKEY_URL": "redis://127.0.0.1:9/0"
},
},
+ "requires": [
+ {
+ "file": "server/config.js",
+ "contains": "INDEXNOW_KEY"
+ }
+ ],
"dirs": [
"dirs": [
"{tmp}/objects"
"{tmp}/objects"
],
],
@@ -19,6 +19,7 @@ const cookieParser = require('cookie-parser');
@@ -19,6 +19,7 @@ const cookieParser = require('cookie-parser');
const rateLimit = require('express-rate-limit');
const rateLimit = require('express-rate-limit');
const contracts = require('openvibe-contracts');
const contracts = require('openvibe-contracts');
const sharedMetrics = require('openvibe-shared/metrics');
const sharedMetrics = require('openvibe-shared/metrics');
+const { createIndexNow } = require('openvibe-shared/indexnow');
const configLib = require('./config');
const configLib = require('./config');
const { openStore } = require('./db');
const { openStore } = require('./db');
@@ -73,13 +74,40 @@ async function createApp(opts = {}) {
@@ -73,13 +74,40 @@ async function createApp(opts = {}) {
const blobs = createBlobStore(opts.storageDir || config.storageDir);
const blobs = createBlobStore(opts.storageDir || config.storageDir);
require('./http/errors').setLogger(log);
require('./http/errors').setLogger(log);
+ // IndexNow (openvibe-shared/indexnow): created once at boot from INDEXNOW_KEY. Unset → off, nothing
+ // mounted, nothing sent; tests and drills never set it. The key file is served on every host (it is
+ // mounted below, before the tenant dispatch), and one module per public host keeps a tenant site's
+ // URL on its own host, which the protocol (and the module) requires. Tests inject a single object
+ // through opts.indexnow (a spy) and every ping goes through it.
+ const indexnow = opts.indexnow !== undefined ? opts.indexnow : createIndexNow({
+ host: config.baseUrl, key: config.indexnow.key, ...(opts.fetchImpl ? { fetch: opts.fetchImpl } : {}), log,
+ });
+ const indexnowHosts = new Map();
+ function indexnowFor(hostname) {
+ if (!indexnowHosts.has(hostname)) indexnowHosts.set(hostname, createIndexNow({
+ host: hostname, key: config.indexnow.key, ...(opts.fetchImpl ? { fetch: opts.fetchImpl } : {}), log,
+ }));
+ return indexnowHosts.get(hostname);
+ }
+ /**
+ * Queue a page that appeared, changed or left the index: one absolute URL per path on `hostname`,
+ * sent as a debounced batch. With an injected module (tests) it is used for every host. Never throws.
+ */
+ function announce(hostname, paths) {
+ if (!indexnow.enabled || !hostname) return;
+ const u = new URL(config.baseUrl);
+ const origin = `${u.protocol}//${hostname}${u.port ? `:${u.port}` : ''}`;
+ const target = opts.indexnow !== undefined ? indexnow : indexnowFor(hostname);
+ target.pingSoon(paths.map((p) => new URL(p, origin).href));
+ }
+
const outbox = createHostOutbox({ db: store.db, config, fetchImpl: opts.fetchImpl, now: store.now, log });
const outbox = createHostOutbox({ db: store.db, config, fetchImpl: opts.fetchImpl, now: store.now, log });
const access = createAccess({ store });
const access = createAccess({ store });
const takedowns = createTakedowns({ store });
const takedowns = createTakedowns({ store });
- const projects = createProjects({ store, config, access, blobs, takedowns, log });
- const sites = createSites({ store, config, access, projects, takedowns });
- const deploys = createDeploys({ store, config, access, projects, sites, blobs, outbox, takedowns, log });
- const domains = createDomains({ store, config, access, projects, sites, outbox, resolver: opts.resolver, log });
+ const projects = createProjects({ store, config, access, blobs, takedowns, log, indexnow: announce });
+ const sites = createSites({ store, config, access, projects, takedowns, indexnow: announce });
+ const deploys = createDeploys({ store, config, access, projects, sites, blobs, outbox, takedowns, log, indexnow: announce });
+ const domains = createDomains({ store, config, access, projects, sites, outbox, resolver: opts.resolver, log, indexnow: announce });
const auth = opts.auth || createSsoClient({
const auth = opts.auth || createSsoClient({
site: 'host',
site: 'host',
baseUrl: config.baseUrl,
baseUrl: config.baseUrl,
@@ -99,7 +127,7 @@ async function createApp(opts = {}) {
@@ -99,7 +127,7 @@ async function createApp(opts = {}) {
const uploadGate = createUploadGate(config.uploads.maxConcurrent);
const uploadGate = createUploadGate(config.uploads.maxConcurrent);
const worker = createWorker({ config, store, domains, blobs, outbox, log });
const worker = createWorker({ config, store, domains, blobs, outbox, log });
- const ctx = { config, store, blobs, outbox, access, takedowns, projects, sites, deploys, domains, auth, viewers, tenant, worker, uploadGate, log };
+ const ctx = { config, store, blobs, outbox, access, takedowns, projects, sites, deploys, domains, auth, viewers, tenant, worker, uploadGate, indexnow, log };
const app = express();
const app = express();
app.disable('x-powered-by');
app.disable('x-powered-by');
@@ -127,6 +155,11 @@ async function createApp(opts = {}) {
@@ -127,6 +155,11 @@ async function createApp(opts = {}) {
app.use(httpMetrics.middleware);
app.use(httpMetrics.middleware);
+ // GET /<key>.txt — the IndexNow key file (openvibe-shared/indexnow). Served before the host dispatch
+ // so it answers on the dashboard host and on every tenant host (an engine fetches it from the host a
+ // ping names). Mounted only with a key; it answers that one path and falls through for everything else.
+ if (indexnow.enabled) app.use(indexnow.keyFile);
+
// ── Host dispatch: tenant sites never reach anything below ──
// ── Host dispatch: tenant sites never reach anything below ──
// resolve/handle read the database (ADR-035: async), so a rejection here must reach the error
// resolve/handle read the database (ADR-035: async), so a rejection here must reach the error
// handler below: Express 4 does not catch a rejected async handler, which would hang the request.
// handler below: Express 4 does not catch a rejected async handler, which would hang the request.
@@ -45,7 +45,9 @@ function load(env = process.env) {
@@ -45,7 +45,9 @@ function load(env = process.env) {
trustProxy: env.TRUST_PROXY != null ? Number(env.TRUST_PROXY) : 1, // nginx (realip) → Node
trustProxy: env.TRUST_PROXY != null ? Number(env.TRUST_PROXY) : 1, // nginx (realip) → Node
// PostgreSQL (ADR-035): DATABASE_URL serves (PgBouncer), DATABASE_DIRECT_URL migrates (owner role).
// PostgreSQL (ADR-035): DATABASE_URL serves (PgBouncer), DATABASE_DIRECT_URL migrates (owner role).
- db: { url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '' },
+ // HOST_PGLITE_DIR: where the development-only embedded database lives (default data/pglite); tests
+ // point it at a temp directory so a spawned server gets a database of its own.
+ db: { url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '', pgliteDir: env.HOST_PGLITE_DIR || '' },
valkey: { url: env.VALKEY_URL || '', prefix: env.VALKEY_PREFIX || 'ov:host:' },
valkey: { url: env.VALKEY_URL || '', prefix: env.VALKEY_PREFIX || 'ov:host:' },
// The SQLite file of releases before the switch: read once by scripts/migrate-to-postgres.js.
// The SQLite file of releases before the switch: read once by scripts/migrate-to-postgres.js.
dbPath: env.HOST_DB_PATH || './data/host.db',
dbPath: env.HOST_DB_PATH || './data/host.db',
@@ -65,6 +67,12 @@ function load(env = process.env) {
@@ -65,6 +67,12 @@ function load(env = process.env) {
// Signs the dashboard form tokens (CSRF). Unset: a random per-process key.
// Signs the dashboard form tokens (CSRF). Unset: a random per-process key.
formSecret: env.HOST_FORM_SECRET || '',
formSecret: env.HOST_FORM_SECRET || '',
+ // IndexNow (openvibe-shared/indexnow): a key makes search engines recrawl a tenant site the
+ // moment its active deploy appears, changes or goes away (the key file is served at
+ // /<key>.txt on every host). Unset → off: no key file, nothing sent. Tests and drills never
+ // set it (the drill sets INDEXNOW_KEY empty in host.example.json).
+ indexnow: { key: String(env.INDEXNOW_KEY || '').trim() },
+
// OpenVibe.Events: the outbox relay runs only when EVENTS_URL and the client secret are set.
// OpenVibe.Events: the outbox relay runs only when EVENTS_URL and the client secret are set.
events: {
events: {
url: trim(env.EVENTS_URL || ''),
url: trim(env.EVENTS_URL || ''),
@@ -36,10 +36,13 @@ const CHARTER_TABLES = ['host_projects', 'host_project_members', 'host_quotas',
@@ -36,10 +36,13 @@ const CHARTER_TABLES = ['host_projects', 'host_project_members', 'host_quotas',
*/
*/
async function openDb(config, { log = console, registry } = {}) {
async function openDb(config, { log = console, registry } = {}) {
if (!config.db.url) {
if (!config.db.url) {
+ // HOST_PGLITE_DIR lets a test (or a second local process) point the embedded database somewhere
+ // of its own; production never takes this path.
+ const pgliteDir = config.db.pgliteDir || DEV_PGLITE;
if (config.isProduction) throw new Error('DATABASE_URL is not set: production serves from PostgreSQL (OpenVibe.Host roles/data add-service.sh host)');
if (config.isProduction) throw new Error('DATABASE_URL is not set: production serves from PostgreSQL (OpenVibe.Host roles/data add-service.sh host)');
- log.warn(`[Host] DATABASE_URL unset: embedded PGlite database in ${DEV_PGLITE} (development only, one process)`);
- fs.mkdirSync(DEV_PGLITE, { recursive: true });
- const db = createDb({ pglite: DEV_PGLITE, service: 'host', registry, log });
+ log.warn(`[Host] DATABASE_URL unset: embedded PGlite database in ${pgliteDir} (development only, one process)`);
+ fs.mkdirSync(pgliteDir, { recursive: true });
+ const db = createDb({ pglite: pgliteDir, service: 'host', registry, log });
await db.migrate({ dir: MIGRATIONS, log });
await db.migrate({ dir: MIGRATIONS, log });
return db;
return db;
}
}
@@ -23,8 +23,14 @@ const { principalOf, actorRef } = require('./access');
@@ -23,8 +23,14 @@ const { principalOf, actorRef } = require('./access');
const DAY = 24 * 3600 * 1000;
const DAY = 24 * 3600 * 1000;
const fmtBytes = (n) => (n >= 1048576 ? `${(n / 1048576).toFixed(1)} MiB` : n >= 1024 ? `${(n / 1024).toFixed(1)} KiB` : `${n} B`);
const fmtBytes = (n) => (n >= 1048576 ? `${(n / 1048576).toFixed(1)} MiB` : n >= 1024 ? `${(n / 1024).toFixed(1)} KiB` : `${n} B`);
-function createDeploys({ store, config = null, access, projects, sites, blobs, outbox, takedowns, log = console }) {
+function createDeploys({ store, config = null, access, projects, sites, blobs, outbox, takedowns, log = console, indexnow = null }) {
const { db } = store;
const { db } = store;
+ // IndexNow: a site's public page appeared, changed or left the index (never a sandbox site, whose
+ // responses carry X-Robots-Tag noindex). `indexnow(hostname, paths)` queues one debounced batch.
+ const announce = typeof indexnow === 'function' ? indexnow : () => {};
+ const announceSite = (site, project, paths = ['/', '/sitemap.xml']) => {
+ if (!project || project.environment !== 'sandbox') announce(sites.defaultHostname(site.name), paths);
+ };
const q = {
const q = {
byId: db.prepare('SELECT * FROM host_deploys WHERE id = ?'),
byId: db.prepare('SELECT * FROM host_deploys WHERE id = ?'),
forSite: db.prepare("SELECT * FROM host_deploys WHERE site_id = ? AND state <> 'deleted' ORDER BY created_at DESC, seq DESC LIMIT ?"),
forSite: db.prepare("SELECT * FROM host_deploys WHERE site_id = ? AND state <> 'deleted' ORDER BY created_at DESC, seq DESC LIMIT ?"),
@@ -174,6 +180,7 @@ function createDeploys({ store, config = null, access, projects, sites, blobs, o
@@ -174,6 +180,7 @@ function createDeploys({ store, config = null, access, projects, sites, blobs, o
await writeLog(id, lines, now);
await writeLog(id, lines, now);
});
});
outbox.kick();
outbox.kick();
+ if (activated && activated.changed) announceSite(site, project);
return { deploy: await get(id), log: await q.logFor.all(id), activated };
return { deploy: await get(id), log: await q.logFor.all(id), activated };
}
}
@@ -199,16 +206,17 @@ function createDeploys({ store, config = null, access, projects, sites, blobs, o
@@ -199,16 +206,17 @@ function createDeploys({ store, config = null, access, projects, sites, blobs, o
}
}
async function activate(viewer, id, { expectedActive, traceparent } = {}) {
async function activate(viewer, id, { expectedActive, traceparent } = {}) {
- const { deploy, site } = await load(viewer, id, 'deploy');
+ const { deploy, site, project } = await load(viewer, id, 'deploy');
await takedowns.assertOpen(site);
await takedowns.assertOpen(site);
const out = await store.tx(async () => await switchPointer(viewer, site, deploy, 'activate', { expectedActive, traceparent }));
const out = await store.tx(async () => await switchPointer(viewer, site, deploy, 'activate', { expectedActive, traceparent }));
outbox.kick();
outbox.kick();
+ if (out.changed) announceSite(site, project);
return out;
return out;
}
}
/** To `deployId`, or else to the most recent previously active deploy that still exists. */
/** To `deployId`, or else to the most recent previously active deploy that still exists. */
async function rollback(viewer, siteId, { deployId, expectedActive, traceparent } = {}) {
async function rollback(viewer, siteId, { deployId, expectedActive, traceparent } = {}) {
- const { site } = await sites.load(viewer, siteId, 'deploy');
+ const { site, project } = await sites.load(viewer, siteId, 'deploy');
await takedowns.assertOpen(site);
await takedowns.assertOpen(site);
const out = await store.tx(async () => {
const out = await store.tx(async () => {
const current = (await db.prepare('SELECT active_deploy_id FROM host_sites WHERE id = ?').get(site.id)).active_deploy_id;
const current = (await db.prepare('SELECT active_deploy_id FROM host_sites WHERE id = ?').get(site.id)).active_deploy_id;
@@ -230,6 +238,7 @@ function createDeploys({ store, config = null, access, projects, sites, blobs, o
@@ -230,6 +238,7 @@ function createDeploys({ store, config = null, access, projects, sites, blobs, o
return await switchPointer(viewer, site, target, 'rollback', { expectedActive, traceparent });
return await switchPointer(viewer, site, target, 'rollback', { expectedActive, traceparent });
});
});
outbox.kick();
outbox.kick();
+ if (out.changed) announceSite(site, project);
return out;
return out;
}
}
@@ -51,8 +51,10 @@ function defaultResolver(servers) {
@@ -51,8 +51,10 @@ function defaultResolver(servers) {
return { resolveTxt: (name) => r.resolveTxt(name) };
return { resolveTxt: (name) => r.resolveTxt(name) };
}
}
-function createDomains({ store, config, access, projects, sites, outbox, resolver = null, log = console }) {
+function createDomains({ store, config, access, projects, sites, outbox, resolver = null, log = console, indexnow = null }) {
const { db } = store;
const { db } = store;
+ // IndexNow: a verified custom domain starts serving the site; tell the engines its page is up.
+ const announce = typeof indexnow === 'function' ? indexnow : () => {};
const dnsResolver = resolver || defaultResolver(config.domains.dnsServers);
const dnsResolver = resolver || defaultResolver(config.domains.dnsServers);
const reserved = firstPartyDomains(config);
const reserved = firstPartyDomains(config);
const q = {
const q = {
@@ -165,6 +167,7 @@ function createDomains({ store, config, access, projects, sites, outbox, resolve
@@ -165,6 +167,7 @@ function createDomains({ store, config, access, projects, sites, outbox, resolve
throw err;
throw err;
}
}
outbox.kick();
outbox.kick();
+ announce(domain.hostname, ['/', '/sitemap.xml']);
return await q.byId.get(domain.id);
return await q.byId.get(domain.id);
}
}
@@ -18,8 +18,9 @@ const PRINCIPAL_RE = /^(usr_[0-9A-HJKMNP-TV-Z]{26}|app:app_[0-9A-HJKMNP-TV-Z]{26
@@ -18,8 +18,9 @@ const PRINCIPAL_RE = /^(usr_[0-9A-HJKMNP-TV-Z]{26}|app:app_[0-9A-HJKMNP-TV-Z]{26
const QUOTA_FIELDS = { storage_bytes: 'storageBytes', deploys_per_day: 'deploysPerDay', max_files: 'maxFiles', max_file_bytes: 'maxFileBytes', sites: 'sites', custom_domains: 'customDomains' };
const QUOTA_FIELDS = { storage_bytes: 'storageBytes', deploys_per_day: 'deploysPerDay', max_files: 'maxFiles', max_file_bytes: 'maxFileBytes', sites: 'sites', custom_domains: 'customDomains' };
const DAY = 24 * 3600 * 1000;
const DAY = 24 * 3600 * 1000;
-function createProjects({ store, config, access, blobs, takedowns, log = console }) {
+function createProjects({ store, config, access, blobs, takedowns, log = console, indexnow = null }) {
const { db } = store;
const { db } = store;
+ const announce = typeof indexnow === 'function' ? indexnow : () => {};
const q = {
const q = {
byId: db.prepare('SELECT * FROM host_projects WHERE id = ?'),
byId: db.prepare('SELECT * FROM host_projects WHERE id = ?'),
forPrincipal: db.prepare(`SELECT p.*, m.role FROM host_projects p JOIN host_project_members m ON m.project_id = p.id
forPrincipal: db.prepare(`SELECT p.*, m.role FROM host_projects p JOIN host_project_members m ON m.project_id = p.id
@@ -149,8 +150,9 @@ function createProjects({ store, config, access, blobs, takedowns, log = console
@@ -149,8 +150,9 @@ function createProjects({ store, config, access, blobs, takedowns, log = console
await access.authorize(project, viewer, 'own');
await access.authorize(project, viewer, 'own');
await takedowns.assertDeletable(viewer, { projectId: project.id });
await takedowns.assertDeletable(viewer, { projectId: project.id });
const now = store.now();
const now = store.now();
+ const doomed = await sites.listForProject(project.id);
await store.tx(async () => {
await store.tx(async () => {
- for (const s of await sites.listForProject(project.id)) await sites.markDeleted(s, now);
+ for (const s of doomed) await sites.markDeleted(s, now);
await db.prepare("UPDATE host_deploys SET state = 'deleted', deleted_at = ? WHERE project_id = ? AND state <> 'deleted'").run(now, project.id);
await db.prepare("UPDATE host_deploys SET state = 'deleted', deleted_at = ? WHERE project_id = ? AND state <> 'deleted'").run(now, project.id);
await db.prepare("DELETE FROM host_deploy_files WHERE deploy_id IN (SELECT id FROM host_deploys WHERE project_id = ?)").run(project.id);
await db.prepare("DELETE FROM host_deploy_files WHERE deploy_id IN (SELECT id FROM host_deploys WHERE project_id = ?)").run(project.id);
await db.prepare('DELETE FROM host_blobs WHERE project_id = ?').run(project.id);
await db.prepare('DELETE FROM host_blobs WHERE project_id = ?').run(project.id);
@@ -158,6 +160,8 @@ function createProjects({ store, config, access, blobs, takedowns, log = console
@@ -158,6 +160,8 @@ function createProjects({ store, config, access, blobs, takedowns, log = console
await q.markDeleted.run(now, now, project.id);
await q.markDeleted.run(now, now, project.id);
});
});
try { blobs.removeProject(project.id); } catch (err) { log.warn('[Host] could not remove project objects:', err.message); }
try { blobs.removeProject(project.id); } catch (err) { log.warn('[Host] could not remove project objects:', err.message); }
+ // IndexNow: every site of the project left the index (never a sandbox project, which is noindex).
+ if (project.environment !== 'sandbox') for (const s of doomed) announce(`${s.name}.${config.sitesDomain}`, ['/', '/sitemap.xml']);
return { deleted: true };
return { deleted: true };
}
}
@@ -21,7 +21,7 @@ const RESERVED = new Set([
@@ -21,7 +21,7 @@ const RESERVED = new Set([
]);
]);
const HOLD_MS = 30 * 24 * 3600 * 1000;
const HOLD_MS = 30 * 24 * 3600 * 1000;
-function createSites({ store, config, access, projects, takedowns }) {
+function createSites({ store, config, access, projects, takedowns, indexnow = null }) {
const { db } = store;
const { db } = store;
const reserved = new Set([...RESERVED, ...config.sites.extraReservedNames]);
const reserved = new Set([...RESERVED, ...config.sites.extraReservedNames]);
const q = {
const q = {
@@ -37,6 +37,11 @@ function createSites({ store, config, access, projects, takedowns }) {
@@ -37,6 +37,11 @@ function createSites({ store, config, access, projects, takedowns }) {
};
};
const defaultHostname = (name) => `${name}.${config.sitesDomain}`;
const defaultHostname = (name) => `${name}.${config.sitesDomain}`;
+ // IndexNow: the site's public pages left the index (never a sandbox site, which is noindex).
+ const announce = typeof indexnow === 'function' ? indexnow : () => {};
+ const announceSite = (site, project) => {
+ if (!project || project.environment !== 'sandbox') announce(defaultHostname(site.name), ['/', '/sitemap.xml']);
+ };
async function get(id) {
async function get(id) {
return isId('site', id) ? await q.byId.get(id) || null : null;
return isId('site', id) ? await q.byId.get(id) || null : null;
@@ -105,6 +110,7 @@ function createSites({ store, config, access, projects, takedowns }) {
@@ -105,6 +110,7 @@ function createSites({ store, config, access, projects, takedowns }) {
freed = await deploys.collectGarbage(project.id);
freed = await deploys.collectGarbage(project.id);
});
});
deploys.unlinkBlobs(project.id, freed);
deploys.unlinkBlobs(project.id, freed);
+ announceSite(site, project);
return { deleted: true };
return { deleted: true };
}
}
new file mode 100644
new file mode 100644
@@ -0,0 +1,104 @@
@@ -0,0 +1,104 @@
+'use strict';
+/**
+ * IndexNow (openvibe-shared/indexnow): INDEXNOW_KEY unset → the feature is off (no key route, nothing
+ * sent). With a key the key file is served at /<key>.txt as text/plain on every host, and a tenant
+ * site whose active deploy appears, changes or goes away pings the engines with the site's page and
+ * its sitemap. A ready deploy that is not active (a draft) never pings; a sandbox site (noindex) never
+ * pings.
+ */
+const assert = require('assert');
+const { boot, check, done, DASHBOARD } = require('./stageb/boot');
+
+const KEY = 'k'.repeat(32);
+
+/** A spy in place of the module's HTTP send: records every pingSoon batch (the repo's test pattern). */
+function makeSpy() {
+ const pings = [];
+ return {
+ pings,
+ module: {
+ enabled: true,
+ keyFile: (_req, _res, next) => next(),
+ pingSoon: (urls) => { const a = Array.isArray(urls) ? urls : [urls]; pings.push(...a); return a.length; },
+ ping: async () => ({ sent: 0, status: 0 }),
+ flush: async () => ({ sent: 0, status: 0 }),
+ },
+ };
+}
+
+(async () => {
+ const off = await boot();
+ await check('without a key IndexNow is off: no key route and nothing sent', async () => {
+ assert.strictEqual(off.ctx.indexnow.enabled, false);
+ const res = await off.get(DASHBOARD, `/${KEY}.txt`);
+ assert.strictEqual(res.status, 404, res.text);
+ });
+ await off.close();
+
+ const on = await boot({ env: { INDEXNOW_KEY: KEY } });
+ await check('with a key the key file answers text/plain with the key (dashboard and tenant hosts)', async () => {
+ assert.strictEqual(on.ctx.indexnow.enabled, true);
+ const dash = await on.get(DASHBOARD, `/${KEY}.txt`);
+ assert.strictEqual(dash.status, 200, dash.text);
+ assert.match(dash.headers['content-type'], /text\/plain/);
+ assert.strictEqual(dash.text, KEY);
+ const tenant = await on.get('anything.openvibe.host', `/${KEY}.txt`);
+ assert.strictEqual(tenant.status, 200, tenant.text);
+ assert.strictEqual(tenant.text, KEY);
+ });
+ await on.close();
+
+ const s = makeSpy();
+ const t = await boot({ indexnow: s.module });
+ const alice = t.user('alice', { display_name: 'Alice' });
+ const project = await t.project(alice, 'Widgets');
+ const site = await t.site(alice, project.id, 'widgets');
+ const origin = 'https://widgets.openvibe.host';
+ let deploy;
+
+ await check('a ready deploy that is not active (a draft) never pings', async () => {
+ const r = await t.upload(alice, site.id, { 'index.html': '<h1>Widgets</h1>' }, { activate: false });
+ assert.strictEqual(r.status, 201, r.text);
+ deploy = r.json().deploy;
+ assert.deepStrictEqual(s.pings, []);
+ });
+
+ await check('activating the deploy pings the site page and the sitemap', async () => {
+ const r = await t.api('POST', `/api/v1/deploys/${deploy.id}/activate`, { as: alice, json: {} });
+ assert.strictEqual(r.status, 200, r.text);
+ assert.ok(r.json().changed, r.text);
+ assert.ok(s.pings.includes(`${origin}/`), JSON.stringify(s.pings));
+ assert.ok(s.pings.includes(`${origin}/sitemap.xml`), JSON.stringify(s.pings));
+ });
+
+ await check('a rollback pings the page and the sitemap again', async () => {
+ const r2 = await t.upload(alice, site.id, { 'index.html': '<h1>Widgets v2</h1>' }, { activate: true });
+ assert.strictEqual(r2.status, 201, r2.text);
+ s.pings.length = 0;
+ const rb = await t.api('POST', `/api/v1/sites/${site.id}/rollback`, { as: alice, json: {} });
+ assert.strictEqual(rb.status, 200, rb.text);
+ assert.strictEqual(rb.json().changed, true);
+ assert.ok(s.pings.includes(`${origin}/`), JSON.stringify(s.pings));
+ assert.ok(s.pings.includes(`${origin}/sitemap.xml`), JSON.stringify(s.pings));
+ });
+
+ await check('deleting the site pings the page and the sitemap again', async () => {
+ s.pings.length = 0;
+ const r = await t.api('DELETE', `/api/v1/sites/${site.id}`, { as: alice, json: {} });
+ assert.strictEqual(r.status, 200, r.text);
+ assert.ok(s.pings.includes(`${origin}/`), JSON.stringify(s.pings));
+ assert.ok(s.pings.includes(`${origin}/sitemap.xml`), JSON.stringify(s.pings));
+ });
+
+ await check('a sandbox site (noindex) never pings, even when it activates', async () => {
+ const sandboxProject = await t.project(alice, 'Sandbox', { environment: 'sandbox' });
+ const sandboxSite = await t.site(alice, sandboxProject.id, 'playground');
+ s.pings.length = 0;
+ const r = await t.upload(alice, sandboxSite.id, { 'index.html': '<h1>Play</h1>' }, { activate: true });
+ assert.strictEqual(r.status, 201, r.text);
+ assert.deepStrictEqual(s.pings, []);
+ });
+ await t.close();
+
+ done();
+})().catch((err) => { console.error(err); process.exit(1); });
new file mode 100644
new file mode 100644
@@ -0,0 +1,64 @@
@@ -0,0 +1,64 @@
+'use strict';
+// Size budgets for openvibe.host's home page (roadmap WS-T task 1, openvibe-shared/perf-budget): the
+// server as it runs (a fresh database), measured without a browser. Budgets sit a little above the
+// 2026-09-29 measurement; raising one is a decision to state in the commit.
+// node test/perf-budget.test.js
+const assert = require('assert');
+const fs = require('fs');
+const net = require('net');
+const os = require('os');
+const path = require('path');
+const { spawn } = require('child_process');
+const { measure, check, format } = require('openvibe-shared/perf-budget');
+
+const BUDGETS = {
+ htmlRawKB: 18, // measured 15.9 (fresh database)
+ htmlBrotliKB: 5, // 4.3
+ jsFiles: 5, // 4
+ jsRawKB: 230, // 212.2
+ jsBrotliKB: 53, // 49.9
+ cssFiles: 2, // 1
+ cssRawKB: 5, // 3.1
+ cssBrotliKB: 2, // 0.9
+ externalFiles: 1, // 0
+};
+
+const freePort = () => new Promise((resolve) => { const s = net.createServer(); s.listen(0, '127.0.0.1', () => { const { port } = s.address(); s.close(() => resolve(port)); }); });
+
+(async () => {
+ // A database of its own (not the shared dev PGlite in data/pglite, and not whatever the caller's
+ // DATABASE_URL names), so the measurement is the server on a fresh database and the run leaves nothing.
+ const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-host-budget-'));
+ const pgliteDir = path.join(dir, 'pglite');
+ const port = await freePort();
+ const child = spawn(process.execPath, [path.join(__dirname, '..', 'server', 'index.js')], {
+ cwd: path.join(__dirname, '..'),
+ env: {
+ ...process.env, PORT: String(port), HOST: '127.0.0.1', NODE_ENV: 'test',
+ BASE_URL: 'https://openvibe.host', HOST_SITES_DOMAIN: 'openvibe.host', COOKIE_SECURE: 'false',
+ DATABASE_URL: '', DATABASE_DIRECT_URL: '', VALKEY_URL: '', HOST_PGLITE_DIR: pgliteDir,
+ HOST_STORAGE_DIR: path.join(dir, 'objects'), HOST_WORKER: 'off', HOST_MIN_FREE_BYTES: '0',
+ },
+ stdio: ['ignore', 'ignore', 'pipe'],
+ });
+ let stderr = '';
+ child.stderr.on('data', (d) => { stderr = (stderr + d).slice(-2000); });
+ const base = `http://127.0.0.1:${port}`;
+ try {
+ let up = false;
+ for (let i = 0; i < 150 && !up; i++) {
+ up = await fetch(`${base}/api/health`).then((r) => r.ok).catch(() => false);
+ if (!up) await new Promise((r) => setTimeout(r, 100));
+ }
+ assert.ok(up, `the server did not start:\n${stderr}`);
+ assert.ok(fs.existsSync(pgliteDir), 'the server did not use the isolated database (HOST_PGLITE_DIR)');
+ const m = await measure({ base });
+ const over = check(m, BUDGETS);
+ assert.deepStrictEqual(over, [], format(m, over));
+ console.log(format(m));
+ console.log('perf budget: all checks passed');
+ } finally {
+ child.kill();
+ fs.rmSync(dir, { recursive: true, force: true });
+ }
+})().catch((err) => { console.error(err); process.exitCode = 1; });
@@ -64,7 +64,7 @@ async function boot(opts = {}) {
@@ -64,7 +64,7 @@ async function boot(opts = {}) {
async function start() {
async function start() {
const config = configLib.load(env);
const config = configLib.load(env);
- built = await createApp({ config, store: createStore(testdb.db, { now: clock.now }), now: clock.now, log: quiet, resolver, fetchImpl: opts.fetchImpl });
+ built = await createApp({ config, store: createStore(testdb.db, { now: clock.now }), now: clock.now, log: quiet, resolver, fetchImpl: opts.fetchImpl, indexnow: opts.indexnow });
server = await new Promise((resolve) => { const s = http.createServer(built.app); s.listen(0, '127.0.0.1', () => resolve(s)); });
server = await new Promise((resolve) => { const s = http.createServer(built.app); s.listen(0, '127.0.0.1', () => resolve(s)); });
t.port = server.address().port;
t.port = server.address().port;
t.app = built.app;
t.app = built.app;