openvibers/openvibe.media Commits
Crawler basics at the openvibe.media origin: /robots.txt (openvibe-shared seo.robotsTxt: AI and search crawlers named; /api/, /auth/, /internal/, /o/, /metrics, /release-metrics and the /live/ dev-data endpoints disallowed; the sitemap), /sitemap.xml and /llms.txt, each cached an hour. The sitemap lists the media index and the watch pages Media is the canonical home of: public, ready VODs and clips of apps other than Live, with their thumbnails as image entries. Live's items are left out on purpose: their watch pages here are noindex with the canonical on openvibe.live/vod and /clip, which Live's own sitemap lists, and a sitemap URL marked noindex is an error for search engines. AI clips, private, unlisted, developer-sandbox, recording, quarantined and failed items are never listed. The sitemap and the page share one predicate (pages.watchIndexable), so every listed page renders index, follow. Watch pages: AI clips (auto_generated) are now noindex, follow for every app, labelled AI clip, attributed to no person (creator Organization OpenVibe AI instead of a Person author) and isBasedOn their source VOD, as on Live; the schema.org VideoObject is emitted on public pages only (none on unlisted or private ones). test/crawl.test.js.
Changes
@@@ -338,8 +338,16 @@ never the headers (`test/trust-proxy.test.js`).| `GET /live/:sel/transcript.json` | **transcript + AI timeline API** — full audio-transcription log and AI overview timeline. `:sel` = slot id / slot slug → slot-scoped sessions; **`@username`** → user-scoped (all their slots, works while offline). Returns `{ live, current, sessions[] (each: ai_overview, transcript, duration…), streamer (overview + stream memories), user }`. `?limit=1..50` sessions (default 10), `?app=`. Cached **30s** (that's the rate limit), CORS-open. || `GET /live/:sel/chat-insight.json` | **chat insight API** — a user's chat-related AI insight/timeline (`:sel` = `@username` or numeric user id): today-vs-alltime chat overviews, condensed memory, event timeline, plus their streamer overview + stream memories when they stream. Proxied from the app's public chat-AI API over loopback; cached **30s**, CORS-open. || `GET /v/:id/transcript.json` | **VOD transcript API** — transcript + AI overview for one existing VOD id (`{ vod_id, title, duration_seconds, ai_overview, transcript, ai_analyzed_at }`). Private VODs → 404. Cached 30s, CORS-open. |++| `GET /robots.txt` | crawler policy (`openvibe-shared/seo.robotsTxt`: AI and search crawlers named, `/api/`, `/auth/`, `/internal/`, `/o/`, `/metrics`, `/live/` disallowed) and the sitemap |++| `GET /sitemap.xml` | the media index and the watch pages Media is the canonical home of: public, ready VODs and clips of apps other than Live (Live lists its own `/vod` and `/clip` pages), never AI clips, private, unlisted, sandbox, recording or failed items; thumbnails as image entries. Every listed page renders `index, follow` |++| `GET /llms.txt` | a map of the site for language-model crawlers || `GET /live/:sel/frame.jpg` | **live frame API** — near-realtime JPEG frame of an actively-live stream slot, extracted from its in-progress recording. `:sel` = slot id (`1`), slot **slug** (`whip`), or **`@username`** (that streamer's top-viewed live slot; slug/username resolve via the app's `/api/streams` listing, cached 5s). Optional `?w=64..1920` scales the width, `?app=` selects the tenant (default `live`; internal base URLs from `APP_INTERNAL_URLS` JSON env or `LIVE_APP_INTERNAL_URL`). Cached **5s per slot** (that cache is the rate limit), CORS-open for external APIs/bots/dashboards. Not live → **404 with a styled OFFLINE card** (real JPEG bytes — dev pipelines decode the body as image/jpeg) so `<img>` embeds degrade nicely (`?format=json` for JSON errors); `503` + card when live but a frame can't be cut. |++Watch pages (`/v/:id`, `/c/:id` on a browser navigation) carry a schema.org `VideoObject` when the++item is public (none on unlisted or private pages). AI clips (`auto_generated`) are `noindex, follow`++wherever they belong, labelled "AI clip", attributed to no person (`creator` OpenVibe AI) and++`isBasedOn` their source VOD, as on Live.++Private items (and legacy rows with no visibility and `is_public = 0`) answerexactly like a missing id — the same 404 and body — unless the request bears theowning app's API key, optionally acting for the owner via `X-OV-User-Id`. That holdsnnew file mode 100644@@@ -0,0 +1,113 @@++/**++ * OpenVibe.Media — crawler files at the origin: /robots.txt, /sitemap.xml, /llms.txt.++ *++ * The sitemap lists the pages search engines may index here: the media index (/) and the watch++ * pages of public, ready VODs and clips that Media is the canonical home of (pages.watchIndexable:++ * not Live's, which Live lists under its own /vod and /clip pages; never AI clips; never private or++ * unlisted; never a developer project's sandbox). A URL here always renders `index, follow`.++ */++'use strict';++++const express = require('express');++const config = require('../config');++const db = require('../db/database');++const seo = require('openvibe-shared/seo');++const pages = require('./pages');++++const MAX_URLS = 45000; // under the 50,000-URL sitemap limit++const router = express.Router();++++const iso = (v) => {++ if (!v) return null;++ const s = String(v);++ const d = new Date(/^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}/.test(s) ? `${s.replace(' ', 'T')}Z` : s);++ return Number.isNaN(d.getTime()) ? null : d.toISOString();++};++const abs = (u) => (!u ? null : (/^https?:\/\//i.test(u) ? u : `${config.publicUrl}${u.startsWith('/') ? '' : '/'}${u}`));++++function sandboxApps() {++ return new Set(db.all("SELECT app_id FROM apps WHERE env = 'sandbox'").map(r => r.app_id));++}++++/** The indexable watch pages, newest first: [{ loc, lastmod, images }]. */++function sitemapEntries() {++ const sandbox = sandboxApps();++ const out = [];++ const vods = db.all(`SELECT id, app_id, visibility, is_public, thumbnail_url, created_at FROM vods++ WHERE COALESCE(is_recording, 0) = 0 AND COALESCE(clips_only, 0) = 0 AND is_public = 1 AND COALESCE(visibility, 'public') = 'public'++ AND quarantined_at IS NULL AND COALESCE(health_status, 'ok') NOT IN ('corrupt', 'zero_byte', 'missing_file', 'needs_review')++ AND file_path IS NOT NULL AND app_id != 'live'++ ORDER BY created_at DESC LIMIT ?`, [MAX_URLS]);++ for (const v of vods) {++ if (sandbox.has(v.app_id) || !pages.watchIndexable('vod', v)) continue;++ out.push({ loc: `${config.publicUrl}/v/${v.id}`, lastmod: iso(v.created_at), images: v.thumbnail_url ? [abs(v.thumbnail_url)] : [] });++ }++ const clips = db.all(`SELECT id, app_id, visibility, is_public, auto_generated, thumbnail_url, created_at FROM clips++ WHERE COALESCE(status, 'ready') = 'ready' AND COALESCE(is_public, 1) = 1 AND COALESCE(visibility, 'public') = 'public'++ AND COALESCE(auto_generated, 0) = 0 AND COALESCE(file_path, '') != '' AND app_id != 'live'++ ORDER BY created_at DESC LIMIT ?`, [MAX_URLS]);++ for (const c of clips) {++ if (sandbox.has(c.app_id) || !pages.watchIndexable('clip', c)) continue;++ out.push({ loc: `${config.publicUrl}/c/${c.id}`, lastmod: iso(c.created_at), images: c.thumbnail_url ? [abs(c.thumbnail_url)] : [] });++ }++ out.sort((a, b) => String(b.lastmod || '').localeCompare(String(a.lastmod || '')));++ return out.slice(0, MAX_URLS - 1);++}++++function robotsTxt() {++ return seo.robotsTxt({++ sitemaps: [`${config.publicUrl}/sitemap.xml`],++ // APIs, sign-in, signed object bytes, operator and dev-data endpoints are not pages.++ disallow: ['/api/', '/auth/', '/internal/', '/o/', '/metrics', '/release-metrics', '/live/'],++ });++}++++function llmsTxt() {++ const base = config.publicUrl;++ return seo.llmsTxt({++ name: 'OpenVibe.Media',++ summary: 'The media service of the OpenVibe network: it stores and serves recorded live streams (VODs), clips, thumbnails, screenshots and files for the network\'s apps.',++ details: [++ 'Most media here belongs to OpenVibe.Live, whose /vod/:id and /clip/:id pages are the canonical pages for it (chat, transcript, the streamer\'s channel); the watch pages here point there.',++ 'Clips marked "AI clip" were picked and cut automatically from a stream, not made by a person.',++ 'Private and unlisted media is never listed here or in the sitemap.',++ ].join(' '),++ sections: [++ { title: 'Browse', links: [++ { title: 'Media index', url: `${base}/`, note: 'every public video, clip, image and file, newest first, each linking to its source page' },++ { title: 'Sitemap', url: `${base}/sitemap.xml`, note: 'the watch pages Media is the canonical home of' },++ ] },++ { title: 'URLs', links: [++ { title: 'VOD watch page', url: `${base}/v/{id}`, note: 'a page for browsers; ?raw=1 returns the video bytes' },++ { title: 'Clip watch page', url: `${base}/c/{id}`, note: 'as above, for clips' },++ { title: 'VOD transcript', url: `${base}/v/{id}/transcript.json`, note: 'transcript and AI overview of a public VOD (JSON)' },++ ] },++ { title: 'The network', links: [++ { title: 'OpenVibe.Live', url: 'https://openvibe.live/', note: 'live streams, channels, VOD and clip pages' },++ { title: 'OpenVibe.Community', url: 'https://openvibe.community/', note: 'pastes and discussion' },++ { title: 'OpenVibe.Network', url: 'https://openvibe.network/', note: 'accounts and the network directory' },++ ] },++ ],++ });++}++++function send(res, type, body) {++ res.set('Cache-Control', 'public, max-age=3600');++ res.type(type).send(body);++}++++router.get('/robots.txt', (req, res) => send(res, 'text/plain', robotsTxt()));++router.get('/llms.txt', (req, res) => send(res, 'text/plain', llmsTxt()));++router.get('/sitemap.xml', (req, res) => {++ try {++ send(res, 'application/xml', seo.sitemapXml([{ loc: `${config.publicUrl}/`, changefreq: 'hourly' }, ...sitemapEntries()]));++ } catch (err) {++ console.error('[Public] sitemap error:', err.message);++ res.status(500).type('text/plain').send('sitemap unavailable');++ }++});++++module.exports = router;++module.exports.sitemapEntries = sitemapEntries;++module.exports.robotsTxt = robotsTxt;++module.exports.llmsTxt = llmsTxt;@@@ -51,6 +51,22 @@ function sourcePage(kind, record) {return `${base}/${kind === 'vod' ? 'vod' : 'clip'}/${record.id}`;}++/** A clip the owning app's automation cut (AI moments), not a person. */++function isAiClip(kind, record) {++ return kind === 'clip' && !!Number(record.auto_generated || 0);++}++++/**++ * May search engines index this watch page? Only when it is public, Media is its canonical home (the++ * owning app is not Live, which serves its own /vod and /clip pages), and a person made it: AI clips++ * are noindex, as on Live. The sitemap lists exactly these pages (server/public/crawl.js).++ */++function watchIndexable(kind, record) {++ const visibility = record.visibility || (record.is_public ? 'public' : 'private');++ const ownedByLive = (record.app_id || 'live') === 'live';++ return visibility === 'public' && !ownedByLive && !isAiClip(kind, record);++}++/*** /v/:id and /c/:id as a page. Canonical stays with the owning app when that* app serves the item (Live's /vod/:id and /clip/:id); anything else is@@@ -62,14 +78,16 @@ function renderWatchPage(kind, record) {const self = `${config.publicUrl}/${isVod ? 'v' : 'c'}/${id}`;const rawUrl = `${self}?raw=1`;const title = snip(record.title || (isVod ? `VOD #${id}` : `Clip #${id}`), 120);-- const by = creatorLine(record);-- const description = snip(record.description || record.ai_overview || `${isVod ? 'A recorded live stream' : 'A clip'}${by ? ` by ${by}` : ''} on the OpenVibe network.`, 200);++ const ai = isAiClip(kind, record);++ // An AI clip is attributed to no person: the automation cut it from someone's stream.++ const by = ai ? null : creatorLine(record);++ const description = snip(record.description || record.ai_overview || `${isVod ? 'A recorded live stream' : ai ? 'An AI-picked clip from a live stream' : 'A clip'}${by ? ` by ${by}` : ''} on the OpenVibe network.`, 200);const thumb = abs(record.thumbnail_url);const ownedByLive = (record.app_id || 'live') === 'live';const canonical = ownedByLive ? sourcePage(kind, record) : self;-- const visibility = record.visibility || (record.is_public ? 'public' : 'private');-- // The owning app has the canonical page; unlisted items never index.-- const robots = ownedByLive || visibility !== 'public' ? 'noindex, follow' : 'index, follow';++ // The owning app has the canonical page; unlisted items and AI clips never index.++ const robots = watchIndexable(kind, record) ? 'index, follow' : 'noindex, follow';++ const sourceVod = ai && record.vod_id ? `${config.publicUrl}/v/${record.vod_id}` : null;const mime = VIDEO_MIME[path.extname(record.file_path || '').toLowerCase()] || 'video/mp4';const uploadDate = isoDate(record.created_at);const duration = isoDuration(record.duration_seconds);@@@ -86,13 +104,15 @@ function renderWatchPage(kind, record) {...(uploadDate ? { uploadDate } : {}),...(duration ? { duration } : {}),...(by ? { author: { '@type': 'Person', name: by } } : {}),++ ...(ai ? { creator: { '@type': 'Organization', name: 'OpenVibe AI', url: NETWORK_URL } } : {}),++ ...(sourceVod ? { isBasedOn: sourceVod } : {}),...(record.view_count ? { interactionStatistic: { '@type': 'InteractionCounter', interactionType: 'https://schema.org/WatchAction', userInteractionCount: record.view_count } } : {}),publisher: { '@type': 'Organization', name: 'OpenVibe', url: NETWORK_URL },isFamilyFriendly: true,};const metaBits = [-- isVod ? 'VOD' : 'Clip',++ isVod ? 'VOD' : ai ? 'AI clip' : 'Clip',record.duration_seconds ? fmtDuration(record.duration_seconds) : null,record.view_count != null ? `${Number(record.view_count).toLocaleString('en-US')} views` : null,fmtDate(record.created_at) || null,@@@ -125,7 +145,8 @@ function renderWatchPage(kind, record) {title: `${title} — ${SITE_NAME}`, description, canonical, robots,image: thumb || undefined, ogType: 'video.other',video: { url: rawUrl, type: mime },-- jsonLd: [videoObject],++ // Structured data describes public items only; unlisted and private pages carry none.++ jsonLd: (record.visibility || (record.is_public ? 'public' : 'private')) === 'public' ? [videoObject] : [],},css, body,history: { type: isVod ? 'vod' : 'clip', title },@@@ -206,4 +227,4 @@ function renderPastePage(paste) {});}--module.exports = { wantsHtmlPage, renderWatchPage, renderPastePage, sourcePage };++module.exports = { wantsHtmlPage, renderWatchPage, renderPastePage, sourcePage, watchIndexable, isAiClip };@@@ -13,6 +13,8 @@* GET /t/:id thumbnails (id = filename)* GET /f/:key files with correct Content-Type + Range* GET /og-image.png the site's share card++ * GET /robots.txt, /sitemap.xml, /llms.txt crawler files (crawl.js): the sitemap lists only++ * the watch pages Media is canonical for (public, not Live's, not AI clips)** /v /c /t /f bytes carry X-Robots-Tag: noindex — the owning app has the* canonical page (the watch page says so with rel=canonical). The paste@@@ -32,6 +34,8 @@ const pages = require('./pages');const drill = require('../drill');const router = express.Router();++// /robots.txt, /sitemap.xml, /llms.txt (server/public/crawl.js).++router.use(require('./crawl'));const MIME_TYPES = {'.webm': 'video/webm', '.mp4': 'video/mp4', '.mkv': 'video/x-matroska', '.avi': 'video/x-msvideo',nnew file mode 100644@@@ -0,0 +1,109 @@++'use strict';++// Crawler basics at the origin (server/public/crawl.js, pages.js): /robots.txt names the sitemap and++// keeps APIs out; /sitemap.xml lists the home and the watch pages Media is canonical for (public,++// ready, not Live's, not AI clips, never private/unlisted/sandbox), and every listed page renders++// `index, follow`; /llms.txt maps the site; public watch pages carry a VideoObject; AI clips are++// noindex with no Person author, like on Live.++const assert = require('assert');++const fs = require('fs');++const os = require('os');++const path = require('path');++const http = require('http');++const express = require('express');++++const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-crawl-'));++Object.assign(process.env, { DB_PATH: path.join(tmp, 'media.db'), MEDIA_PUBLIC_URL: 'https://media.test', VOD_PATH: path.join(tmp, 'vods'), CLIPS_PATH: path.join(tmp, 'clips') });++++const db = require('../server/db/database');++require('../server/views/service').ensureSchema();++const pages = require('../server/public/pages');++++db.upsertApp({ app_id: 'live', api_key: 'k1-live-crawl' });++db.upsertApp({ app_id: 'games', api_key: 'k2-games-crawl' });++db.ensureProjectTenant('prj_01TESTCRAWL00000000000000A', 'sandbox', 1024);++const raw = db.getDb();++const vod = raw.prepare(`INSERT INTO vods (id, app_id, title, file_path, is_public, visibility, is_recording, health_status, thumbnail_url, created_at, duration_seconds)++ VALUES (?, ?, ?, '/x.mp4', ?, ?, ?, ?, ?, ?, 60)`);++const clip = raw.prepare(`INSERT INTO clips (id, app_id, vod_id, title, file_path, is_public, visibility, auto_generated, status, created_at)++ VALUES (?, ?, ?, ?, '/c.webm', ?, ?, ?, ?, ?)`);++vod.run(1, 'games', 'Games public', 1, 'public', 0, 'ok', '/t/g1.jpg', '2026-09-20 10:00:00');++vod.run(2, 'games', 'Games unlisted', 0, 'unlisted', 0, 'ok', null, '2026-09-20 11:00:00');++vod.run(3, 'games', 'Games private', 0, 'private', 0, 'ok', null, '2026-09-20 12:00:00');++vod.run(4, 'live', 'Live public', 1, 'public', 0, 'ok', null, '2026-09-21 10:00:00');++vod.run(5, 'games', 'Games recording', 1, 'public', 1, 'unknown', null, '2026-09-21 11:00:00');++vod.run(6, 'games', 'Games failed', 1, 'public', 0, 'corrupt', null, '2026-09-21 12:00:00');++vod.run(7, 'prj_01TESTCRAWL00000000000000A-sandbox', 'Sandbox', 1, 'public', 0, 'ok', null, '2026-09-21 13:00:00');++clip.run(11, 'games', 1, 'Games clip', 1, 'public', 0, 'ready', '2026-09-22 10:00:00');++clip.run(12, 'games', 1, 'AI moment', 1, 'public', 1, 'ready', '2026-09-22 11:00:00');++clip.run(13, 'games', 1, 'Cutting', 1, 'public', 0, 'processing', '2026-09-22 12:00:00');++clip.run(14, 'games', 1, 'Unlisted clip', 0, 'unlisted', 0, 'ready', '2026-09-22 13:00:00');++clip.run(15, 'live', 4, 'Live clip', 1, 'public', 0, 'ready', '2026-09-22 14:00:00');++++const app = express();++app.use('/', require('../server/public/routes'));++const server = app.listen(0, '127.0.0.1');++const get = (p, headers = {}) => new Promise((resolve, reject) => {++ http.get({ host: '127.0.0.1', port: server.address().port, path: p, headers }, (res) => {++ let body = ''; res.setEncoding('utf8'); res.on('data', c => { body += c; }); res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body }));++ }).on('error', reject);++});++++(async () => {++ await new Promise((res) => (server.listening ? res() : server.once('listening', res)));++ // ── robots.txt ──++ let r = await get('/robots.txt');++ assert.strictEqual(r.status, 200);++ assert.ok(r.headers['content-type'].startsWith('text/plain'));++ assert.ok(r.body.includes('Sitemap: https://media.test/sitemap.xml'));++ for (const d of ['/api/', '/auth/', '/o/', '/internal/']) assert.ok(r.body.includes(`Disallow: ${d}`), `robots keeps ${d} out`);++ assert.ok(/User-agent: GPTBot/.test(r.body) && /User-agent: ClaudeBot/.test(r.body), 'the AI-crawler policy is named');++ console.log('✅ /robots.txt: sitemap, APIs disallowed, AI crawlers named');++++ // ── sitemap.xml ──++ r = await get('/sitemap.xml');++ assert.strictEqual(r.status, 200);++ assert.ok(r.headers['content-type'].startsWith('application/xml'));++ const locs = [...r.body.matchAll(/<loc>([^<]+)<\/loc>/g)].map(m => m[1]);++ assert.deepStrictEqual(locs.filter(l => !l.includes('/t/')).sort(), ['https://media.test/', 'https://media.test/c/11', 'https://media.test/v/1'].sort());++ assert.ok(r.body.includes('<image:loc>https://media.test/t/g1.jpg</image:loc>'), 'thumbnails as image entries');++ for (const never of ['/v/2', '/v/3', '/v/4', '/v/5', '/v/6', '/v/7', '/c/12', '/c/13', '/c/14', '/c/15']) {++ assert.ok(!locs.includes(`https://media.test${never}`), `${never} is not listed`);++ }++ // Every listed watch page renders index, follow (no "submitted URL marked noindex").++ const byId = (t, id) => db.get(`SELECT * FROM ${t} WHERE id = ?`, [id]);++ assert.ok(pages.renderWatchPage('vod', byId('vods', 1)).includes('content="index, follow"'));++ assert.ok(pages.renderWatchPage('clip', byId('clips', 11)).includes('content="index, follow"'));++ console.log('✅ /sitemap.xml: public ready items Media is canonical for; private, unlisted, sandbox, Live-owned, AI, recording and failed never');++++ // ── llms.txt ──++ r = await get('/llms.txt');++ assert.strictEqual(r.status, 200);++ assert.ok(r.body.startsWith('# OpenVibe.Media') && r.body.includes('https://media.test/sitemap.xml') && r.body.includes('AI clip'));++ assert.ok(!/\bfree\b/i.test(r.body), 'no cost claims');++ console.log('✅ /llms.txt maps the site');++++ // ── Watch pages: VideoObject on public pages; AI clips noindex, no Person, based on the VOD ──++ const ldOf = (html) => { const m = /<script type="application\/ld\+json">([\s\S]*?)<\/script>/.exec(html); return m ? JSON.parse(m[1]) : null; };++ const pub = pages.renderWatchPage('vod', { ...byId('vods', 1), username: 'gamer' });++ assert.strictEqual(ldOf(pub)['@type'], 'VideoObject');++ assert.deepStrictEqual(ldOf(pub).author, { '@type': 'Person', name: 'gamer' });++ const aiHtml = pages.renderWatchPage('clip', { ...byId('clips', 12), username: 'gamer' });++ assert.ok(aiHtml.includes('content="noindex, follow"'), 'AI clips never index');++ const aiLd = ldOf(aiHtml);++ assert.strictEqual(aiLd.author, undefined, 'no Person attribution for an AI clip');++ assert.deepStrictEqual(aiLd.creator, { '@type': 'Organization', name: 'OpenVibe AI', url: 'https://openvibe.network' });++ assert.strictEqual(aiLd.isBasedOn, 'https://media.test/v/1');++ assert.ok(aiHtml.includes('<span>AI clip</span>') && !aiHtml.includes('by gamer'));++ assert.strictEqual(ldOf(pages.renderWatchPage('vod', byId('vods', 2))), null, 'unlisted pages carry no structured data');++ assert.ok(pages.renderWatchPage('clip', byId('clips', 15)).includes('content="noindex, follow"'), 'Live-owned: canonical on Live');++ r = await get('/c/12', { accept: 'text/html', 'sec-fetch-dest': 'document' });++ assert.strictEqual(r.status, 200);++ assert.ok(r.body.includes('content="noindex, follow"'));++ console.log('✅ watch pages: VideoObject when public; AI clips noindex, attributed to OpenVibe AI and their source VOD');++++ server.close();++ db.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('\n✅ All crawler tests passed');++ process.exit(0);++})().catch((err) => { console.error(err); server.close(); process.exit(1); });
@@ -338,8 +338,16 @@ never the headers (`test/trust-proxy.test.js`).
@@ -338,8 +338,16 @@ never the headers (`test/trust-proxy.test.js`).
| `GET /live/:sel/transcript.json` | **transcript + AI timeline API** — full audio-transcription log and AI overview timeline. `:sel` = slot id / slot slug → slot-scoped sessions; **`@username`** → user-scoped (all their slots, works while offline). Returns `{ live, current, sessions[] (each: ai_overview, transcript, duration…), streamer (overview + stream memories), user }`. `?limit=1..50` sessions (default 10), `?app=`. Cached **30s** (that's the rate limit), CORS-open. |
| `GET /live/:sel/transcript.json` | **transcript + AI timeline API** — full audio-transcription log and AI overview timeline. `:sel` = slot id / slot slug → slot-scoped sessions; **`@username`** → user-scoped (all their slots, works while offline). Returns `{ live, current, sessions[] (each: ai_overview, transcript, duration…), streamer (overview + stream memories), user }`. `?limit=1..50` sessions (default 10), `?app=`. Cached **30s** (that's the rate limit), CORS-open. |
| `GET /live/:sel/chat-insight.json` | **chat insight API** — a user's chat-related AI insight/timeline (`:sel` = `@username` or numeric user id): today-vs-alltime chat overviews, condensed memory, event timeline, plus their streamer overview + stream memories when they stream. Proxied from the app's public chat-AI API over loopback; cached **30s**, CORS-open. |
| `GET /live/:sel/chat-insight.json` | **chat insight API** — a user's chat-related AI insight/timeline (`:sel` = `@username` or numeric user id): today-vs-alltime chat overviews, condensed memory, event timeline, plus their streamer overview + stream memories when they stream. Proxied from the app's public chat-AI API over loopback; cached **30s**, CORS-open. |
| `GET /v/:id/transcript.json` | **VOD transcript API** — transcript + AI overview for one existing VOD id (`{ vod_id, title, duration_seconds, ai_overview, transcript, ai_analyzed_at }`). Private VODs → 404. Cached 30s, CORS-open. |
| `GET /v/:id/transcript.json` | **VOD transcript API** — transcript + AI overview for one existing VOD id (`{ vod_id, title, duration_seconds, ai_overview, transcript, ai_analyzed_at }`). Private VODs → 404. Cached 30s, CORS-open. |
+| `GET /robots.txt` | crawler policy (`openvibe-shared/seo.robotsTxt`: AI and search crawlers named, `/api/`, `/auth/`, `/internal/`, `/o/`, `/metrics`, `/live/` disallowed) and the sitemap |
+| `GET /sitemap.xml` | the media index and the watch pages Media is the canonical home of: public, ready VODs and clips of apps other than Live (Live lists its own `/vod` and `/clip` pages), never AI clips, private, unlisted, sandbox, recording or failed items; thumbnails as image entries. Every listed page renders `index, follow` |
+| `GET /llms.txt` | a map of the site for language-model crawlers |
| `GET /live/:sel/frame.jpg` | **live frame API** — near-realtime JPEG frame of an actively-live stream slot, extracted from its in-progress recording. `:sel` = slot id (`1`), slot **slug** (`whip`), or **`@username`** (that streamer's top-viewed live slot; slug/username resolve via the app's `/api/streams` listing, cached 5s). Optional `?w=64..1920` scales the width, `?app=` selects the tenant (default `live`; internal base URLs from `APP_INTERNAL_URLS` JSON env or `LIVE_APP_INTERNAL_URL`). Cached **5s per slot** (that cache is the rate limit), CORS-open for external APIs/bots/dashboards. Not live → **404 with a styled OFFLINE card** (real JPEG bytes — dev pipelines decode the body as image/jpeg) so `<img>` embeds degrade nicely (`?format=json` for JSON errors); `503` + card when live but a frame can't be cut. |
| `GET /live/:sel/frame.jpg` | **live frame API** — near-realtime JPEG frame of an actively-live stream slot, extracted from its in-progress recording. `:sel` = slot id (`1`), slot **slug** (`whip`), or **`@username`** (that streamer's top-viewed live slot; slug/username resolve via the app's `/api/streams` listing, cached 5s). Optional `?w=64..1920` scales the width, `?app=` selects the tenant (default `live`; internal base URLs from `APP_INTERNAL_URLS` JSON env or `LIVE_APP_INTERNAL_URL`). Cached **5s per slot** (that cache is the rate limit), CORS-open for external APIs/bots/dashboards. Not live → **404 with a styled OFFLINE card** (real JPEG bytes — dev pipelines decode the body as image/jpeg) so `<img>` embeds degrade nicely (`?format=json` for JSON errors); `503` + card when live but a frame can't be cut. |
+Watch pages (`/v/:id`, `/c/:id` on a browser navigation) carry a schema.org `VideoObject` when the
+item is public (none on unlisted or private pages). AI clips (`auto_generated`) are `noindex, follow`
+wherever they belong, labelled "AI clip", attributed to no person (`creator` OpenVibe AI) and
+`isBasedOn` their source VOD, as on Live.
+
Private items (and legacy rows with no visibility and `is_public = 0`) answer
Private items (and legacy rows with no visibility and `is_public = 0`) answer
exactly like a missing id — the same 404 and body — unless the request bears the
exactly like a missing id — the same 404 and body — unless the request bears the
owning app's API key, optionally acting for the owner via `X-OV-User-Id`. That holds
owning app's API key, optionally acting for the owner via `X-OV-User-Id`. That holds
new file mode 100644
new file mode 100644
@@ -0,0 +1,113 @@
@@ -0,0 +1,113 @@
+/**
+ * OpenVibe.Media — crawler files at the origin: /robots.txt, /sitemap.xml, /llms.txt.
+ *
+ * The sitemap lists the pages search engines may index here: the media index (/) and the watch
+ * pages of public, ready VODs and clips that Media is the canonical home of (pages.watchIndexable:
+ * not Live's, which Live lists under its own /vod and /clip pages; never AI clips; never private or
+ * unlisted; never a developer project's sandbox). A URL here always renders `index, follow`.
+ */
+'use strict';
+
+const express = require('express');
+const config = require('../config');
+const db = require('../db/database');
+const seo = require('openvibe-shared/seo');
+const pages = require('./pages');
+
+const MAX_URLS = 45000; // under the 50,000-URL sitemap limit
+const router = express.Router();
+
+const iso = (v) => {
+ if (!v) return null;
+ const s = String(v);
+ const d = new Date(/^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}/.test(s) ? `${s.replace(' ', 'T')}Z` : s);
+ return Number.isNaN(d.getTime()) ? null : d.toISOString();
+};
+const abs = (u) => (!u ? null : (/^https?:\/\//i.test(u) ? u : `${config.publicUrl}${u.startsWith('/') ? '' : '/'}${u}`));
+
+function sandboxApps() {
+ return new Set(db.all("SELECT app_id FROM apps WHERE env = 'sandbox'").map(r => r.app_id));
+}
+
+/** The indexable watch pages, newest first: [{ loc, lastmod, images }]. */
+function sitemapEntries() {
+ const sandbox = sandboxApps();
+ const out = [];
+ const vods = db.all(`SELECT id, app_id, visibility, is_public, thumbnail_url, created_at FROM vods
+ WHERE COALESCE(is_recording, 0) = 0 AND COALESCE(clips_only, 0) = 0 AND is_public = 1 AND COALESCE(visibility, 'public') = 'public'
+ AND quarantined_at IS NULL AND COALESCE(health_status, 'ok') NOT IN ('corrupt', 'zero_byte', 'missing_file', 'needs_review')
+ AND file_path IS NOT NULL AND app_id != 'live'
+ ORDER BY created_at DESC LIMIT ?`, [MAX_URLS]);
+ for (const v of vods) {
+ if (sandbox.has(v.app_id) || !pages.watchIndexable('vod', v)) continue;
+ out.push({ loc: `${config.publicUrl}/v/${v.id}`, lastmod: iso(v.created_at), images: v.thumbnail_url ? [abs(v.thumbnail_url)] : [] });
+ }
+ const clips = db.all(`SELECT id, app_id, visibility, is_public, auto_generated, thumbnail_url, created_at FROM clips
+ WHERE COALESCE(status, 'ready') = 'ready' AND COALESCE(is_public, 1) = 1 AND COALESCE(visibility, 'public') = 'public'
+ AND COALESCE(auto_generated, 0) = 0 AND COALESCE(file_path, '') != '' AND app_id != 'live'
+ ORDER BY created_at DESC LIMIT ?`, [MAX_URLS]);
+ for (const c of clips) {
+ if (sandbox.has(c.app_id) || !pages.watchIndexable('clip', c)) continue;
+ out.push({ loc: `${config.publicUrl}/c/${c.id}`, lastmod: iso(c.created_at), images: c.thumbnail_url ? [abs(c.thumbnail_url)] : [] });
+ }
+ out.sort((a, b) => String(b.lastmod || '').localeCompare(String(a.lastmod || '')));
+ return out.slice(0, MAX_URLS - 1);
+}
+
+function robotsTxt() {
+ return seo.robotsTxt({
+ sitemaps: [`${config.publicUrl}/sitemap.xml`],
+ // APIs, sign-in, signed object bytes, operator and dev-data endpoints are not pages.
+ disallow: ['/api/', '/auth/', '/internal/', '/o/', '/metrics', '/release-metrics', '/live/'],
+ });
+}
+
+function llmsTxt() {
+ const base = config.publicUrl;
+ return seo.llmsTxt({
+ name: 'OpenVibe.Media',
+ summary: 'The media service of the OpenVibe network: it stores and serves recorded live streams (VODs), clips, thumbnails, screenshots and files for the network\'s apps.',
+ details: [
+ 'Most media here belongs to OpenVibe.Live, whose /vod/:id and /clip/:id pages are the canonical pages for it (chat, transcript, the streamer\'s channel); the watch pages here point there.',
+ 'Clips marked "AI clip" were picked and cut automatically from a stream, not made by a person.',
+ 'Private and unlisted media is never listed here or in the sitemap.',
+ ].join(' '),
+ sections: [
+ { title: 'Browse', links: [
+ { title: 'Media index', url: `${base}/`, note: 'every public video, clip, image and file, newest first, each linking to its source page' },
+ { title: 'Sitemap', url: `${base}/sitemap.xml`, note: 'the watch pages Media is the canonical home of' },
+ ] },
+ { title: 'URLs', links: [
+ { title: 'VOD watch page', url: `${base}/v/{id}`, note: 'a page for browsers; ?raw=1 returns the video bytes' },
+ { title: 'Clip watch page', url: `${base}/c/{id}`, note: 'as above, for clips' },
+ { title: 'VOD transcript', url: `${base}/v/{id}/transcript.json`, note: 'transcript and AI overview of a public VOD (JSON)' },
+ ] },
+ { title: 'The network', links: [
+ { title: 'OpenVibe.Live', url: 'https://openvibe.live/', note: 'live streams, channels, VOD and clip pages' },
+ { title: 'OpenVibe.Community', url: 'https://openvibe.community/', note: 'pastes and discussion' },
+ { title: 'OpenVibe.Network', url: 'https://openvibe.network/', note: 'accounts and the network directory' },
+ ] },
+ ],
+ });
+}
+
+function send(res, type, body) {
+ res.set('Cache-Control', 'public, max-age=3600');
+ res.type(type).send(body);
+}
+
+router.get('/robots.txt', (req, res) => send(res, 'text/plain', robotsTxt()));
+router.get('/llms.txt', (req, res) => send(res, 'text/plain', llmsTxt()));
+router.get('/sitemap.xml', (req, res) => {
+ try {
+ send(res, 'application/xml', seo.sitemapXml([{ loc: `${config.publicUrl}/`, changefreq: 'hourly' }, ...sitemapEntries()]));
+ } catch (err) {
+ console.error('[Public] sitemap error:', err.message);
+ res.status(500).type('text/plain').send('sitemap unavailable');
+ }
+});
+
+module.exports = router;
+module.exports.sitemapEntries = sitemapEntries;
+module.exports.robotsTxt = robotsTxt;
+module.exports.llmsTxt = llmsTxt;
@@ -51,6 +51,22 @@ function sourcePage(kind, record) {
@@ -51,6 +51,22 @@ function sourcePage(kind, record) {
return `${base}/${kind === 'vod' ? 'vod' : 'clip'}/${record.id}`;
return `${base}/${kind === 'vod' ? 'vod' : 'clip'}/${record.id}`;
}
}
+/** A clip the owning app's automation cut (AI moments), not a person. */
+function isAiClip(kind, record) {
+ return kind === 'clip' && !!Number(record.auto_generated || 0);
+}
+
+/**
+ * May search engines index this watch page? Only when it is public, Media is its canonical home (the
+ * owning app is not Live, which serves its own /vod and /clip pages), and a person made it: AI clips
+ * are noindex, as on Live. The sitemap lists exactly these pages (server/public/crawl.js).
+ */
+function watchIndexable(kind, record) {
+ const visibility = record.visibility || (record.is_public ? 'public' : 'private');
+ const ownedByLive = (record.app_id || 'live') === 'live';
+ return visibility === 'public' && !ownedByLive && !isAiClip(kind, record);
+}
+
/**
/**
* /v/:id and /c/:id as a page. Canonical stays with the owning app when that
* /v/:id and /c/:id as a page. Canonical stays with the owning app when that
* app serves the item (Live's /vod/:id and /clip/:id); anything else is
* app serves the item (Live's /vod/:id and /clip/:id); anything else is
@@ -62,14 +78,16 @@ function renderWatchPage(kind, record) {
@@ -62,14 +78,16 @@ function renderWatchPage(kind, record) {
const self = `${config.publicUrl}/${isVod ? 'v' : 'c'}/${id}`;
const self = `${config.publicUrl}/${isVod ? 'v' : 'c'}/${id}`;
const rawUrl = `${self}?raw=1`;
const rawUrl = `${self}?raw=1`;
const title = snip(record.title || (isVod ? `VOD #${id}` : `Clip #${id}`), 120);
const title = snip(record.title || (isVod ? `VOD #${id}` : `Clip #${id}`), 120);
- const by = creatorLine(record);
- const description = snip(record.description || record.ai_overview || `${isVod ? 'A recorded live stream' : 'A clip'}${by ? ` by ${by}` : ''} on the OpenVibe network.`, 200);
+ const ai = isAiClip(kind, record);
+ // An AI clip is attributed to no person: the automation cut it from someone's stream.
+ const by = ai ? null : creatorLine(record);
+ const description = snip(record.description || record.ai_overview || `${isVod ? 'A recorded live stream' : ai ? 'An AI-picked clip from a live stream' : 'A clip'}${by ? ` by ${by}` : ''} on the OpenVibe network.`, 200);
const thumb = abs(record.thumbnail_url);
const thumb = abs(record.thumbnail_url);
const ownedByLive = (record.app_id || 'live') === 'live';
const ownedByLive = (record.app_id || 'live') === 'live';
const canonical = ownedByLive ? sourcePage(kind, record) : self;
const canonical = ownedByLive ? sourcePage(kind, record) : self;
- const visibility = record.visibility || (record.is_public ? 'public' : 'private');
- // The owning app has the canonical page; unlisted items never index.
- const robots = ownedByLive || visibility !== 'public' ? 'noindex, follow' : 'index, follow';
+ // The owning app has the canonical page; unlisted items and AI clips never index.
+ const robots = watchIndexable(kind, record) ? 'index, follow' : 'noindex, follow';
+ const sourceVod = ai && record.vod_id ? `${config.publicUrl}/v/${record.vod_id}` : null;
const mime = VIDEO_MIME[path.extname(record.file_path || '').toLowerCase()] || 'video/mp4';
const mime = VIDEO_MIME[path.extname(record.file_path || '').toLowerCase()] || 'video/mp4';
const uploadDate = isoDate(record.created_at);
const uploadDate = isoDate(record.created_at);
const duration = isoDuration(record.duration_seconds);
const duration = isoDuration(record.duration_seconds);
@@ -86,13 +104,15 @@ function renderWatchPage(kind, record) {
@@ -86,13 +104,15 @@ function renderWatchPage(kind, record) {
...(uploadDate ? { uploadDate } : {}),
...(uploadDate ? { uploadDate } : {}),
...(duration ? { duration } : {}),
...(duration ? { duration } : {}),
...(by ? { author: { '@type': 'Person', name: by } } : {}),
...(by ? { author: { '@type': 'Person', name: by } } : {}),
+ ...(ai ? { creator: { '@type': 'Organization', name: 'OpenVibe AI', url: NETWORK_URL } } : {}),
+ ...(sourceVod ? { isBasedOn: sourceVod } : {}),
...(record.view_count ? { interactionStatistic: { '@type': 'InteractionCounter', interactionType: 'https://schema.org/WatchAction', userInteractionCount: record.view_count } } : {}),
...(record.view_count ? { interactionStatistic: { '@type': 'InteractionCounter', interactionType: 'https://schema.org/WatchAction', userInteractionCount: record.view_count } } : {}),
publisher: { '@type': 'Organization', name: 'OpenVibe', url: NETWORK_URL },
publisher: { '@type': 'Organization', name: 'OpenVibe', url: NETWORK_URL },
isFamilyFriendly: true,
isFamilyFriendly: true,
};
};
const metaBits = [
const metaBits = [
- isVod ? 'VOD' : 'Clip',
+ isVod ? 'VOD' : ai ? 'AI clip' : 'Clip',
record.duration_seconds ? fmtDuration(record.duration_seconds) : null,
record.duration_seconds ? fmtDuration(record.duration_seconds) : null,
record.view_count != null ? `${Number(record.view_count).toLocaleString('en-US')} views` : null,
record.view_count != null ? `${Number(record.view_count).toLocaleString('en-US')} views` : null,
fmtDate(record.created_at) || null,
fmtDate(record.created_at) || null,
@@ -125,7 +145,8 @@ function renderWatchPage(kind, record) {
@@ -125,7 +145,8 @@ function renderWatchPage(kind, record) {
title: `${title} — ${SITE_NAME}`, description, canonical, robots,
title: `${title} — ${SITE_NAME}`, description, canonical, robots,
image: thumb || undefined, ogType: 'video.other',
image: thumb || undefined, ogType: 'video.other',
video: { url: rawUrl, type: mime },
video: { url: rawUrl, type: mime },
- jsonLd: [videoObject],
+ // Structured data describes public items only; unlisted and private pages carry none.
+ jsonLd: (record.visibility || (record.is_public ? 'public' : 'private')) === 'public' ? [videoObject] : [],
},
},
css, body,
css, body,
history: { type: isVod ? 'vod' : 'clip', title },
history: { type: isVod ? 'vod' : 'clip', title },
@@ -206,4 +227,4 @@ function renderPastePage(paste) {
@@ -206,4 +227,4 @@ function renderPastePage(paste) {
});
});
}
}
-module.exports = { wantsHtmlPage, renderWatchPage, renderPastePage, sourcePage };
+module.exports = { wantsHtmlPage, renderWatchPage, renderPastePage, sourcePage, watchIndexable, isAiClip };
@@ -13,6 +13,8 @@
@@ -13,6 +13,8 @@
* GET /t/:id thumbnails (id = filename)
* GET /t/:id thumbnails (id = filename)
* GET /f/:key files with correct Content-Type + Range
* GET /f/:key files with correct Content-Type + Range
* GET /og-image.png the site's share card
* GET /og-image.png the site's share card
+ * GET /robots.txt, /sitemap.xml, /llms.txt crawler files (crawl.js): the sitemap lists only
+ * the watch pages Media is canonical for (public, not Live's, not AI clips)
*
*
* /v /c /t /f bytes carry X-Robots-Tag: noindex — the owning app has the
* /v /c /t /f bytes carry X-Robots-Tag: noindex — the owning app has the
* canonical page (the watch page says so with rel=canonical). The paste
* canonical page (the watch page says so with rel=canonical). The paste
@@ -32,6 +34,8 @@ const pages = require('./pages');
@@ -32,6 +34,8 @@ const pages = require('./pages');
const drill = require('../drill');
const drill = require('../drill');
const router = express.Router();
const router = express.Router();
+// /robots.txt, /sitemap.xml, /llms.txt (server/public/crawl.js).
+router.use(require('./crawl'));
const MIME_TYPES = {
const MIME_TYPES = {
'.webm': 'video/webm', '.mp4': 'video/mp4', '.mkv': 'video/x-matroska', '.avi': 'video/x-msvideo',
'.webm': 'video/webm', '.mp4': 'video/mp4', '.mkv': 'video/x-matroska', '.avi': 'video/x-msvideo',
new file mode 100644
new file mode 100644
@@ -0,0 +1,109 @@
@@ -0,0 +1,109 @@
+'use strict';
+// Crawler basics at the origin (server/public/crawl.js, pages.js): /robots.txt names the sitemap and
+// keeps APIs out; /sitemap.xml lists the home and the watch pages Media is canonical for (public,
+// ready, not Live's, not AI clips, never private/unlisted/sandbox), and every listed page renders
+// `index, follow`; /llms.txt maps the site; public watch pages carry a VideoObject; AI clips are
+// noindex with no Person author, like on Live.
+const assert = require('assert');
+const fs = require('fs');
+const os = require('os');
+const path = require('path');
+const http = require('http');
+const express = require('express');
+
+const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-crawl-'));
+Object.assign(process.env, { DB_PATH: path.join(tmp, 'media.db'), MEDIA_PUBLIC_URL: 'https://media.test', VOD_PATH: path.join(tmp, 'vods'), CLIPS_PATH: path.join(tmp, 'clips') });
+
+const db = require('../server/db/database');
+require('../server/views/service').ensureSchema();
+const pages = require('../server/public/pages');
+
+db.upsertApp({ app_id: 'live', api_key: 'k1-live-crawl' });
+db.upsertApp({ app_id: 'games', api_key: 'k2-games-crawl' });
+db.ensureProjectTenant('prj_01TESTCRAWL00000000000000A', 'sandbox', 1024);
+const raw = db.getDb();
+const vod = raw.prepare(`INSERT INTO vods (id, app_id, title, file_path, is_public, visibility, is_recording, health_status, thumbnail_url, created_at, duration_seconds)
+ VALUES (?, ?, ?, '/x.mp4', ?, ?, ?, ?, ?, ?, 60)`);
+const clip = raw.prepare(`INSERT INTO clips (id, app_id, vod_id, title, file_path, is_public, visibility, auto_generated, status, created_at)
+ VALUES (?, ?, ?, ?, '/c.webm', ?, ?, ?, ?, ?)`);
+vod.run(1, 'games', 'Games public', 1, 'public', 0, 'ok', '/t/g1.jpg', '2026-09-20 10:00:00');
+vod.run(2, 'games', 'Games unlisted', 0, 'unlisted', 0, 'ok', null, '2026-09-20 11:00:00');
+vod.run(3, 'games', 'Games private', 0, 'private', 0, 'ok', null, '2026-09-20 12:00:00');
+vod.run(4, 'live', 'Live public', 1, 'public', 0, 'ok', null, '2026-09-21 10:00:00');
+vod.run(5, 'games', 'Games recording', 1, 'public', 1, 'unknown', null, '2026-09-21 11:00:00');
+vod.run(6, 'games', 'Games failed', 1, 'public', 0, 'corrupt', null, '2026-09-21 12:00:00');
+vod.run(7, 'prj_01TESTCRAWL00000000000000A-sandbox', 'Sandbox', 1, 'public', 0, 'ok', null, '2026-09-21 13:00:00');
+clip.run(11, 'games', 1, 'Games clip', 1, 'public', 0, 'ready', '2026-09-22 10:00:00');
+clip.run(12, 'games', 1, 'AI moment', 1, 'public', 1, 'ready', '2026-09-22 11:00:00');
+clip.run(13, 'games', 1, 'Cutting', 1, 'public', 0, 'processing', '2026-09-22 12:00:00');
+clip.run(14, 'games', 1, 'Unlisted clip', 0, 'unlisted', 0, 'ready', '2026-09-22 13:00:00');
+clip.run(15, 'live', 4, 'Live clip', 1, 'public', 0, 'ready', '2026-09-22 14:00:00');
+
+const app = express();
+app.use('/', require('../server/public/routes'));
+const server = app.listen(0, '127.0.0.1');
+const get = (p, headers = {}) => new Promise((resolve, reject) => {
+ http.get({ host: '127.0.0.1', port: server.address().port, path: p, headers }, (res) => {
+ let body = ''; res.setEncoding('utf8'); res.on('data', c => { body += c; }); res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body }));
+ }).on('error', reject);
+});
+
+(async () => {
+ await new Promise((res) => (server.listening ? res() : server.once('listening', res)));
+ // ── robots.txt ──
+ let r = await get('/robots.txt');
+ assert.strictEqual(r.status, 200);
+ assert.ok(r.headers['content-type'].startsWith('text/plain'));
+ assert.ok(r.body.includes('Sitemap: https://media.test/sitemap.xml'));
+ for (const d of ['/api/', '/auth/', '/o/', '/internal/']) assert.ok(r.body.includes(`Disallow: ${d}`), `robots keeps ${d} out`);
+ assert.ok(/User-agent: GPTBot/.test(r.body) && /User-agent: ClaudeBot/.test(r.body), 'the AI-crawler policy is named');
+ console.log('✅ /robots.txt: sitemap, APIs disallowed, AI crawlers named');
+
+ // ── sitemap.xml ──
+ r = await get('/sitemap.xml');
+ assert.strictEqual(r.status, 200);
+ assert.ok(r.headers['content-type'].startsWith('application/xml'));
+ const locs = [...r.body.matchAll(/<loc>([^<]+)<\/loc>/g)].map(m => m[1]);
+ assert.deepStrictEqual(locs.filter(l => !l.includes('/t/')).sort(), ['https://media.test/', 'https://media.test/c/11', 'https://media.test/v/1'].sort());
+ assert.ok(r.body.includes('<image:loc>https://media.test/t/g1.jpg</image:loc>'), 'thumbnails as image entries');
+ for (const never of ['/v/2', '/v/3', '/v/4', '/v/5', '/v/6', '/v/7', '/c/12', '/c/13', '/c/14', '/c/15']) {
+ assert.ok(!locs.includes(`https://media.test${never}`), `${never} is not listed`);
+ }
+ // Every listed watch page renders index, follow (no "submitted URL marked noindex").
+ const byId = (t, id) => db.get(`SELECT * FROM ${t} WHERE id = ?`, [id]);
+ assert.ok(pages.renderWatchPage('vod', byId('vods', 1)).includes('content="index, follow"'));
+ assert.ok(pages.renderWatchPage('clip', byId('clips', 11)).includes('content="index, follow"'));
+ console.log('✅ /sitemap.xml: public ready items Media is canonical for; private, unlisted, sandbox, Live-owned, AI, recording and failed never');
+
+ // ── llms.txt ──
+ r = await get('/llms.txt');
+ assert.strictEqual(r.status, 200);
+ assert.ok(r.body.startsWith('# OpenVibe.Media') && r.body.includes('https://media.test/sitemap.xml') && r.body.includes('AI clip'));
+ assert.ok(!/\bfree\b/i.test(r.body), 'no cost claims');
+ console.log('✅ /llms.txt maps the site');
+
+ // ── Watch pages: VideoObject on public pages; AI clips noindex, no Person, based on the VOD ──
+ const ldOf = (html) => { const m = /<script type="application\/ld\+json">([\s\S]*?)<\/script>/.exec(html); return m ? JSON.parse(m[1]) : null; };
+ const pub = pages.renderWatchPage('vod', { ...byId('vods', 1), username: 'gamer' });
+ assert.strictEqual(ldOf(pub)['@type'], 'VideoObject');
+ assert.deepStrictEqual(ldOf(pub).author, { '@type': 'Person', name: 'gamer' });
+ const aiHtml = pages.renderWatchPage('clip', { ...byId('clips', 12), username: 'gamer' });
+ assert.ok(aiHtml.includes('content="noindex, follow"'), 'AI clips never index');
+ const aiLd = ldOf(aiHtml);
+ assert.strictEqual(aiLd.author, undefined, 'no Person attribution for an AI clip');
+ assert.deepStrictEqual(aiLd.creator, { '@type': 'Organization', name: 'OpenVibe AI', url: 'https://openvibe.network' });
+ assert.strictEqual(aiLd.isBasedOn, 'https://media.test/v/1');
+ assert.ok(aiHtml.includes('<span>AI clip</span>') && !aiHtml.includes('by gamer'));
+ assert.strictEqual(ldOf(pages.renderWatchPage('vod', byId('vods', 2))), null, 'unlisted pages carry no structured data');
+ assert.ok(pages.renderWatchPage('clip', byId('clips', 15)).includes('content="noindex, follow"'), 'Live-owned: canonical on Live');
+ r = await get('/c/12', { accept: 'text/html', 'sec-fetch-dest': 'document' });
+ assert.strictEqual(r.status, 200);
+ assert.ok(r.body.includes('content="noindex, follow"'));
+ console.log('✅ watch pages: VideoObject when public; AI clips noindex, attributed to OpenVibe AI and their source VOD');
+
+ server.close();
+ db.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('\n✅ All crawler tests passed');
+ process.exit(0);
+})().catch((err) => { console.error(err); server.close(); process.exit(1); });