openvibers/openvibe.shared Commits
Render page lists in llmsFull and an ai-summary block in pageSummary (#7)
Changes
@@@ -6,6 +6,13 @@ A release is the git tag `vX.Y.Z`; consumers pin the tag's tarball (see README).## Unreleased++- `seo.llmsFull` takes `base` and sections of `pages: [{ title, url, text | html }]`: each page++ renders as `### title`, `URL: …` and its full text (HTML stripped), in input order, and++ `maxBytes` ends the file with a `(truncated: N more pages at <base>/llms.txt)` line. The++ 2.4.0 `{ url, body }` sections render unchanged. `seo.pageSummary` now converts to an HTML++ string — `<meta name="ai-summary">`, WebPage JSON-LD and, with string `facts`, a++ `<noscript><section data-ai-summary>` block — and has `.head`/`.body`; its `meta`, `jsonLd` and++ `html` are unchanged. `templates/site` serves `/llms-full.txt` and puts the summary in its head.- `perf-budget` measures Core Web Vitals (plan T11). `measure({ browser })` opens the page inChrome through `browser-harness` after the byte pass, clicks once, and adds`cwv: { lcpMs, inpMs, cls }`. `check()` accepts `lcpMs`, `inpMs` and `cls` budgets and throws@@@ -39,7 +39,7 @@ npm run create-site -- my-site```The command creates `./my-site` with a pinned Shared dependency, a small Express server,--discovery routes, and a smoke test. Set `SITE_URL` to the site's public origin before serving it.++discovery routes (`/robots.txt`, `/sitemap.xml`, `/llms.txt`, `/llms-full.txt`), and a smoke test. Set `SITE_URL` to the site's public origin before serving it.### Server side: pin a release tarball@@@ -62,18 +62,42 @@ const { AnalyticsTracker } = require('openvibe-shared/analytics');### SEO kit`openvibe-shared/seo` exports the page head, sitemap, robots and `llms.txt` helpers, plus--the discovery formats below. The package root also exports them as `seo`.++the discovery formats below. The package root also exports them as `seo`. A site serves++`/robots.txt`, `/sitemap.xml`, `/llms.txt` and `/llms-full.txt` (`templates/site` has all four).++++`llmsFull({ site, summary, base, sections, maxBytes })` renders `/llms-full.txt`: the `llms.txt`++header, then `## <section>` and, for each page, `### <title>`, `URL: <absolute url>` and the page's++full plain text — `text` as given (newlines kept) or `html` stripped to text; a page with neither is++listed with its URL only. Input order is kept and bodies are never clipped. With `maxBytes` it stops++before the first page that would pass the limit and ends with++`(truncated: N more pages at <base>/llms.txt)`. Sections with no pages are left out. (Sections in the++2.4.0 shape, `{ title, url, body }` with `maxChars`/`maxTotal`, still render as before.)```jsconst seo = require('openvibe-shared/seo');++const full = seo.llmsFull({ site: 'Example', summary: 'What Example publishes', base: 'https://example.com/',++ sections: [{ title: 'Guides', pages: [{ title: 'Start', url: '/start', text: 'Step one.\nStep two.' },++ { title: 'API', url: '/api', html: '<h1>API</h1><p>Call get().</p>' }]}],++ maxBytes: 500000 }); // serve as text/plain at /llms-full.txt++```++++`pageSummary({ title, summary, url, updated, facts })` is the AI-readable summary of a page. As a++string it is `<meta name="ai-summary">` (the summary clipped to 160 like the description, which++`headTags` still owns), a WebPage JSON-LD tag (`name`, `description`, `url`, `dateModified` as ISO,++`abstract`; written with `jsonLdTag`) and, when `facts` (strings) are given, a++`<noscript><section data-ai-summary>` block with the title, summary and facts. Everything is++escaped. `.head` is the meta and JSON-LD on their own and `.body` the noscript block, for pages that++put facts in `<body>`; `.meta`, `.jsonLd` and `.html` are the 2.4.0 values.++++```js++const summary = seo.pageSummary({ title: 'Start', summary: 'How to set up Example in five minutes',++ url: 'https://example.com/start', updated: '2026-10-01', facts: ['Takes five minutes', 'No account needed'] });++`<head>${seo.headTags({ title: 'Start', description: 'How to set up Example', canonical: 'https://example.com/start' })}++${summary.head}</head><body>${summary.body}…`; // or `${summary}` in <head> when there are no facts++```++++```jsconst site = { name: 'Example', url: 'https://example.com/' };--const full = seo.llmsFull({ site, summary: 'What Example publishes',-- sections: [{ title: 'Guide', url: '/guide', body: 'The full plain-text guide.' }],-- maxChars: 20000, maxTotal: 100000 }); // serve as text/plain at /llms-full.txt--const page = seo.pageSummary({ title: 'Guide', summary: 'A short guide',-- facts: [['Topic', 'Getting started']], url: 'https://example.com/guide', updated: '2026-10-01' });--// page.meta is an escaped description attribute value; page.jsonLd is a WebPage object;--// page.html is an escaped, hidden summary section. Use jsonLdTag(page.jsonLd) in <head>.const feed = { title: 'Example', link: site.url, description: 'Recent guides',updated: '2026-10-01T12:00:00Z', items: [{ title: 'Guide', url: '/guide', id: 'guide-1',published: '2026-10-01T12:00:00Z', summary: 'A short guide', author: 'Example' }] };@@@ -84,7 +108,7 @@ seo.feedLinkTags({ rss: 'https://example.com/feed.xml', atom: 'https://example.c`jsonLd` also builds `product`, `review`, `aggregateRating`, `imageGallery` (an ImageObjectarray) and `videoObject` (the same VideoObject shape as `video`). Feed URLs should be absolute,--or relative to the absolute `link`; `llmsFull` resolves section URLs against `site.url`.++or relative to the absolute `link`; `llmsFull` resolves page URLs against `base` (or `site.url`).### Images and responsive pictures (`assets.js`)@@@ -205,29 +205,89 @@ function limitText(value, max) {return size ? text.slice(0, size - 1).trimEnd() + '…' : '';}--/** /llms-full.txt: the llms.txt header followed by full, plain-text sections. */--function llmsFull({ site, summary, sections = [], maxChars, maxTotal } = {}) {++const ENTITIES = { amp: '&', lt: '<', gt: '>', quot: '"', apos: "'", nbsp: ' ' };++++/** Plain text from HTML: scripts, styles and comments dropped, block ends kept as line breaks. */++function htmlText(html) {++ return String(html == null ? '' : html)++ .replace(/<!--[\s\S]*?-->/g, '')++ .replace(/<(script|style|template)\b[\s\S]*?<\/\1\s*>/gi, '')++ .replace(/<br\s*\/?>|<\/(li|dt|dd|tr)\s*>/gi, '\n')++ .replace(/<\/(p|div|section|article|header|footer|main|aside|nav|ul|ol|dl|table|h[1-6]|blockquote|pre|figure)\s*>/gi, '\n\n')++ .replace(/<[^>]*>/g, '')++ .replace(/&(#x[0-9a-f]+|#\d+|[a-z]+);/gi, (m, e) => {++ if (e[0] !== '#') return ENTITIES[e.toLowerCase()] || m;++ const code = e[1] === 'x' || e[1] === 'X' ? parseInt(e.slice(2), 16) : parseInt(e.slice(1), 10);++ return code > 0 && code <= 0x10ffff ? String.fromCodePoint(code) : m;++ })++ .split('\n').map((line) => line.replace(/\s+/g, ' ').trim()).join('\n')++ .replace(/\n{3,}/g, '\n\n')++ .trim();++}++++/** One /llms-full.txt page: `### title`, `URL: …` and its full text (`text`, or `html` stripped to text). */++function llmsPage(page, base) {++ const url = absolute(page.url, base);++ const text = page.text != null ? String(page.text).replace(/\r\n?/g, '\n').trim() : htmlText(page.html);++ return `\n### ${page.title || ''}\n${url ? `\nURL: ${url}\n` : ''}${text ? `\n${text}\n` : ''}`;++}++++/**++ * /llms-full.txt: the llms.txt header followed by full, plain-text sections. A section with++ * `pages: [{ title, url, text | html }]` renders `## title` then every page in input order, never++ * clipped; `maxBytes` stops before the first page that would pass it and ends with a++ * `(truncated: N more pages at <base>/llms.txt)` line. A section with `url`/`body` renders as in 2.4.0.++ */++function llmsFull({ site, summary, base, sections = [], maxChars, maxTotal, maxBytes } = {}) {const name = site && typeof site === 'object' ? site.name || site.title : site;-- const base = site && typeof site === 'object' ? site.url : site;++ base = base || (site && typeof site === 'object' ? site.url : site);let out = llmsTxt({ name, summary });++ let left = 0;for (const section of sections) {if (!section) continue;++ if (Array.isArray(section.pages)) {++ let heading = `\n## ${section.title || ''}\n`;++ for (const page of section.pages) {++ if (!page) continue;++ const chunk = heading + llmsPage(page, base);++ if (left || (maxBytes != null && Buffer.byteLength(out + chunk) > maxBytes)) { left++; continue; }++ out += chunk;++ heading = '';++ }++ continue;++ }++ if (left) continue;const url = absolute(section.url, base);const body = limitText(String(section.body == null ? '' : section.body).trim(), section.maxChars == null ? maxChars : section.maxChars);out += `\n## ${section.title || ''}\n\n${url}\n\n${body}\n`;}++ if (left) out += `\n(truncated: ${left} more page${left === 1 ? '' : 's'} at ${absolute('/llms.txt', base)})\n`;return limitText(out, maxTotal);}--/** A description value, WebPage data, and a hidden text snapshot for a page. */++/**++ * AI-readable page summary. As a string (`${seo.pageSummary(…)}`, `String(…)`) it is the HTML:++ * `<meta name="ai-summary">` (clipped to 160 like the description), a WebPage JSON-LD tag and, when++ * `facts` (strings) are given, a `<noscript><section data-ai-summary>` block. `.head` and `.body` hold++ * the head part and the noscript block on their own; `.meta`, `.jsonLd` and `.html` are the 2.4.0 values.++ */function pageSummary({ title, summary, facts = [], url, updated } = {}) {const pageUrl = absolute(url);const text = String(summary == null ? '' : summary);const jsonLd = { '@context': 'https://schema.org', '@type': 'WebPage', name: title, url: pageUrl,abstract: text, ...(updated ? { dateModified: updated } : {}) };-- const rows = facts.map(([key, value]) => `<dt>${esc(key)}</dt><dd>${esc(value)}</dd>`).join('');++ const rows = facts.filter(Array.isArray).map(([key, value]) => `<dt>${esc(key)}</dt><dd>${esc(value)}</dd>`).join('');const html = `<section data-ov-summary hidden><h2>${esc(title)}</h2><p>${esc(text)}</p>${rows ? `<dl>${rows}</dl>` : ''}${pageUrl ? `<a href="${esc(pageUrl)}">${esc(pageUrl)}</a>` : ''}${updated ? `<time datetime="${esc(updated)}">${esc(updated)}</time>` : ''}</section>`;-- return { meta: esc(clip(text, 160)), jsonLd, html };++ const modified = feedDate(updated, 'iso');++ const ld = { '@context': 'https://schema.org', '@type': 'WebPage', ...(title != null ? { name: String(title) } : {}),++ description: text, ...(pageUrl ? { url: pageUrl } : {}), ...(modified ? { dateModified: modified } : {}), abstract: text };++ const head = [text ? `<meta name="ai-summary" content="${esc(clip(text, 160))}">` : '', jsonLdTag(ld)].filter(Boolean).join('\n');++ const items = facts.map((f) => (Array.isArray(f) ? f.filter((x) => x != null && x !== '').join(': ') : String(f == null ? '' : f)).trim()).filter(Boolean);++ const body = items.length ? `<noscript><section data-ai-summary><h2>${esc(title)}</h2><p>${esc(text)}</p><ul>${items.map((f) => `<li>${esc(f)}</li>`).join('')}</ul></section></noscript>` : '';++ const result = { meta: esc(clip(text, 160)), jsonLd, html };++ const tags = body ? `${head}\n${body}` : head;++ Object.defineProperties(result, { head: { value: head }, body: { value: body }, toString: { value: () => tags } });++ return result;}function feedDate(value, format) {@@@ -29,14 +29,21 @@ app.get('/llms.txt', (req, res) => {res.type('text/plain').send(seo.llmsTxt({ name, summary: name, sections: [{ title: 'Pages', links: [{ title: 'Home', url: home }] }] }));});++app.get('/llms-full.txt', (req, res) => {++ res.set('Cache-Control', cache.assetHeaders(req.path));++ res.type('text/plain').send(seo.llmsFull({ site: name, summary: name, base: home, sections: [{ title: 'Pages', pages: [{ title: 'Home', url: '/', text: name }] }] }));++});++app.get('/', cache.applyHtml(), (req, res) => {const head = seo.headTags({ title: name, siteName: name, description: name, canonical: home });++ const summary = seo.pageSummary({ title: name, summary: name, url: home });res.type('html').send(`<!doctype html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1">${head}++${summary}<script src="${serve.url('web-runtime.js')}" defer></script><script src="${serve.url('navbar.js')}" defer></script><script src="${serve.url('footer.js')}" defer></script>@@@ -26,6 +26,8 @@ function request(server, path) {assert.equal(home.status, 200);assert.ok(home.body.includes(`<link rel="canonical" href="${homeUrl}">`));assert.ok(home.body.includes(`<meta property="og:url" content="${homeUrl}">`));++ assert.match(home.body, /<meta name="ai-summary" content="[^"]+">/);++ assert.match(home.body, /"@type":"WebPage"/);assert.match(home.body, /OpenVibeNavbar\.init\(/);assert.match(home.body, /OVWebRuntime\.create\(\{ features: \{\}, routes: \[\] \}\)\.boot\(\)/);assert.equal(home.headers['cache-control'], 'public, max-age=120, stale-while-revalidate=3600');@@@ -44,6 +46,11 @@ function request(server, path) {assert.equal(llms.status, 200);assert.match(llms.headers['content-type'], /^text\/plain/);++ const llmsFull = await request(server, '/llms-full.txt');++ assert.equal(llmsFull.status, 200);++ assert.match(llmsFull.headers['content-type'], /^text\/plain/);++ assert.ok(llmsFull.body.includes(`URL: ${homeUrl}\n`));++const script = await request(server, '/shared/web-runtime.js');assert.equal(script.status, 200);assert.equal(script.headers['cache-control'], 'public, max-age=300, stale-while-revalidate=60');nnew file mode 100644@@@ -0,0 +1,69 @@++'use strict';++const assert = require('assert');++const seo = require('../seo');++++for (const name of ['llmsFull', 'pageSummary']) assert.strictEqual(typeof seo[name], 'function', `${name} resolves`);++++const input = { site: 'Example', summary: ' Docs and blog ', base: 'https://example.test/', sections: [++ { title: 'Docs', pages: [++ { title: 'Intro', url: '/intro', text: 'Line one.\r\nLine two.\n\nLast paragraph.' },++ { title: 'API', url: 'api/v1', html: '<h1>API</h1><!-- note --><p>Use <b>get()</b> & <i>set()</i>.</p><style>p{}</style><script>alert("x")</script><ul><li>One</li><li>Two!</li></ul>' },++ ] },++ { title: 'Empty', pages: [] },++ { title: 'Blog', pages: [{ title: 'Launch', url: 'https://other.test/launch' }] },++] };++const full = seo.llmsFull(input);++assert.strictEqual(full, seo.llmsFull(input), 'deterministic');++assert.ok(full.startsWith(seo.llmsTxt({ name: 'Example', summary: ' Docs and blog ' })), 'llms.txt header');++assert.ok(full.endsWith('\n') && !full.endsWith('\n\n'), 'one trailing newline');++assert.ok(full.indexOf('## Docs') < full.indexOf('### Intro') && full.indexOf('### Intro') < full.indexOf('### API')++ && full.indexOf('### API') < full.indexOf('## Blog') && full.indexOf('## Blog') < full.indexOf('### Launch'), 'input order kept');++assert.ok(full.includes('### Intro\n\nURL: https://example.test/intro\n\nLine one.\nLine two.\n\nLast paragraph.\n'), 'text kept with its newlines');++assert.ok(full.includes('URL: https://example.test/api/v1\n\nAPI\n\nUse get() & set().\n\nOne\nTwo!\n'), 'html stripped to text');++assert.ok(!/<|alert|p\{\}|note/.test(full), 'no tags, scripts, styles or comments');++assert.ok(full.endsWith('### Launch\n\nURL: https://other.test/launch\n'), 'a page without text lists its URL only');++assert.ok(!full.includes('## Empty'), 'an empty section is left out');++assert.strictEqual(seo.llmsFull({ site: 'X', summary: 's', sections: [{ title: 'None', pages: [] }] }), seo.llmsTxt({ name: 'X', summary: 's' }));++const long = 'word '.repeat(5000).trim();++assert.ok(seo.llmsFull({ site: 'X', base: 'https://x.test', sections: [{ title: 'A', pages: [{ title: 'Long', url: '/', text: long }] }] }).includes(long), 'body never clipped');++assert.match(seo.llmsFull({ site: { name: 'S', url: 'https://s.test/' }, sections: [{ title: 'A', pages: [{ title: 'P', url: '/p' }] }] }), /URL: https:\/\/s\.test\/p\n/, 'site.url is the base when base is absent');++++const pages = (n) => Array.from({ length: n }, (_, i) => ({ title: `P${i}`, url: `/p${i}`, text: `Body ${i} ünïcode` }));++const all = seo.llmsFull({ site: 'X', base: 'https://x.test/', sections: [{ title: 'A', pages: pages(2) }, { title: 'B', pages: pages(2) }] });++const upTo = (marker) => Buffer.byteLength(all.slice(0, all.indexOf(marker)));++const cut = seo.llmsFull({ site: 'X', base: 'https://x.test/', maxBytes: upTo('### P1'), sections: [{ title: 'A', pages: pages(2) }, { title: 'B', pages: pages(2) }] });++assert.ok(cut.includes('### P0') && !cut.includes('### P1') && !cut.includes('## B'), 'stops before the page that would pass maxBytes');++assert.ok(cut.endsWith('\n\n(truncated: 3 more pages at https://x.test/llms.txt)\n'), 'truncation line counts every page left');++assert.ok(Buffer.byteLength(cut.slice(0, cut.indexOf('\n(truncated'))) <= upTo('### P1'), 'content stays within maxBytes');++assert.strictEqual(seo.llmsFull({ site: 'X', base: 'https://x.test/', maxBytes: Buffer.byteLength(all), sections: [{ title: 'A', pages: pages(2) }, { title: 'B', pages: pages(2) }] }), all, 'no line when everything fits');++assert.match(seo.llmsFull({ site: 'X', base: 'https://x.test/', maxBytes: upTo('## B'), sections: [{ title: 'A', pages: pages(2) }, { title: 'B', pages: pages(1) }] }), /\n\(truncated: 1 more page at https:\/\/x\.test\/llms\.txt\)\n$/);++++const ps = seo.pageSummary({ title: 'Guide "A" <b> & co', summary: 'Say "hi" <b>bold</b> & </script><script>alert(1)</script>', url: 'https://example.test/g?a=1&b=2', updated: '2026-10-01T12:00:00Z', facts: ['Fast & "small"', '<i>Two</i>'] });++const html = String(ps);++assert.strictEqual(`${ps}`, html);++assert.strictEqual(html, `${ps.head}\n${ps.body}`);++assert.ok(html.includes('<meta name="ai-summary" content="Say "hi" <b>bold</b> & </script><script>alert(1)</script>">'), 'meta content escaped');++assert.ok(!html.includes('name="description"'), 'headTags owns the description');++assert.strictEqual(html.match(/<\/script>/g).length, 1, '</script> in the summary cannot close the JSON-LD tag');++const ld = JSON.parse(html.match(/<script type="application\/ld\+json">(.*?)<\/script>/)[1]);++assert.deepStrictEqual(ld, { '@context': 'https://schema.org', '@type': 'WebPage', name: 'Guide "A" <b> & co',++ description: 'Say "hi" <b>bold</b> & </script><script>alert(1)</script>', url: 'https://example.test/g?a=1&b=2',++ dateModified: '2026-10-01T12:00:00.000Z', abstract: 'Say "hi" <b>bold</b> & </script><script>alert(1)</script>' });++assert.ok(html.includes(seo.jsonLdTag(ld)), 'built with jsonLdTag');++assert.ok(ps.body.startsWith('<noscript><section data-ai-summary><h2>Guide "A" <b> & co</h2><p>Say "hi"'));++assert.ok(ps.body.endsWith('<ul><li>Fast & "small"</li><li><i>Two</i></li></ul></section></noscript>'));++assert.deepStrictEqual(Object.keys(ps), ['meta', 'jsonLd', 'html'], 'the 2.4.0 result shape is unchanged');++++const bare = seo.pageSummary({ title: 'T', summary: 'S', url: '/t' });++assert.ok(!String(bare).includes('<noscript'), 'no noscript block without facts');++assert.strictEqual(bare.body, '');++assert.ok(!String(seo.pageSummary({ title: 'T', summary: 'S', facts: [] })).includes('<noscript'));++assert.ok(!/dateModified/.test(String(bare)) && !/dateModified/.test(String(seo.pageSummary({ summary: 'S', updated: 'not a date' }))), 'dateModified only for a valid date');++++const words = 'alpha beta gamma delta '.repeat(20);++const content = String(seo.pageSummary({ title: 'T', summary: words })).match(/name="ai-summary" content="([^"]*)"/)[1];++assert.strictEqual(content, seo.clip(words, 160), 'clipped like the description');++assert.ok(content.length <= 160 && content.endsWith('…'));++assert.ok(String(seo.pageSummary({ title: 'T', summary: words })).includes(`"abstract":"${words}"`), 'JSON-LD keeps the full summary');++++console.log('seo-llms-full: all checks passed');
@@ -6,6 +6,13 @@ A release is the git tag `vX.Y.Z`; consumers pin the tag's tarball (see README).
@@ -6,6 +6,13 @@ A release is the git tag `vX.Y.Z`; consumers pin the tag's tarball (see README).
## Unreleased
## Unreleased
+- `seo.llmsFull` takes `base` and sections of `pages: [{ title, url, text | html }]`: each page
+ renders as `### title`, `URL: …` and its full text (HTML stripped), in input order, and
+ `maxBytes` ends the file with a `(truncated: N more pages at <base>/llms.txt)` line. The
+ 2.4.0 `{ url, body }` sections render unchanged. `seo.pageSummary` now converts to an HTML
+ string — `<meta name="ai-summary">`, WebPage JSON-LD and, with string `facts`, a
+ `<noscript><section data-ai-summary>` block — and has `.head`/`.body`; its `meta`, `jsonLd` and
+ `html` are unchanged. `templates/site` serves `/llms-full.txt` and puts the summary in its head.
- `perf-budget` measures Core Web Vitals (plan T11). `measure({ browser })` opens the page in
- `perf-budget` measures Core Web Vitals (plan T11). `measure({ browser })` opens the page in
Chrome through `browser-harness` after the byte pass, clicks once, and adds
Chrome through `browser-harness` after the byte pass, clicks once, and adds
`cwv: { lcpMs, inpMs, cls }`. `check()` accepts `lcpMs`, `inpMs` and `cls` budgets and throws
`cwv: { lcpMs, inpMs, cls }`. `check()` accepts `lcpMs`, `inpMs` and `cls` budgets and throws
@@ -39,7 +39,7 @@ npm run create-site -- my-site
@@ -39,7 +39,7 @@ npm run create-site -- my-site
```
```
The command creates `./my-site` with a pinned Shared dependency, a small Express server,
The command creates `./my-site` with a pinned Shared dependency, a small Express server,
-discovery routes, and a smoke test. Set `SITE_URL` to the site's public origin before serving it.
+discovery routes (`/robots.txt`, `/sitemap.xml`, `/llms.txt`, `/llms-full.txt`), and a smoke test. Set `SITE_URL` to the site's public origin before serving it.
### Server side: pin a release tarball
### Server side: pin a release tarball
@@ -62,18 +62,42 @@ const { AnalyticsTracker } = require('openvibe-shared/analytics');
@@ -62,18 +62,42 @@ const { AnalyticsTracker } = require('openvibe-shared/analytics');
### SEO kit
### SEO kit
`openvibe-shared/seo` exports the page head, sitemap, robots and `llms.txt` helpers, plus
`openvibe-shared/seo` exports the page head, sitemap, robots and `llms.txt` helpers, plus
-the discovery formats below. The package root also exports them as `seo`.
+the discovery formats below. The package root also exports them as `seo`. A site serves
+`/robots.txt`, `/sitemap.xml`, `/llms.txt` and `/llms-full.txt` (`templates/site` has all four).
+
+`llmsFull({ site, summary, base, sections, maxBytes })` renders `/llms-full.txt`: the `llms.txt`
+header, then `## <section>` and, for each page, `### <title>`, `URL: <absolute url>` and the page's
+full plain text — `text` as given (newlines kept) or `html` stripped to text; a page with neither is
+listed with its URL only. Input order is kept and bodies are never clipped. With `maxBytes` it stops
+before the first page that would pass the limit and ends with
+`(truncated: N more pages at <base>/llms.txt)`. Sections with no pages are left out. (Sections in the
+2.4.0 shape, `{ title, url, body }` with `maxChars`/`maxTotal`, still render as before.)
```js
```js
const seo = require('openvibe-shared/seo');
const seo = require('openvibe-shared/seo');
+const full = seo.llmsFull({ site: 'Example', summary: 'What Example publishes', base: 'https://example.com/',
+ sections: [{ title: 'Guides', pages: [{ title: 'Start', url: '/start', text: 'Step one.\nStep two.' },
+ { title: 'API', url: '/api', html: '<h1>API</h1><p>Call get().</p>' }]}],
+ maxBytes: 500000 }); // serve as text/plain at /llms-full.txt
+```
+
+`pageSummary({ title, summary, url, updated, facts })` is the AI-readable summary of a page. As a
+string it is `<meta name="ai-summary">` (the summary clipped to 160 like the description, which
+`headTags` still owns), a WebPage JSON-LD tag (`name`, `description`, `url`, `dateModified` as ISO,
+`abstract`; written with `jsonLdTag`) and, when `facts` (strings) are given, a
+`<noscript><section data-ai-summary>` block with the title, summary and facts. Everything is
+escaped. `.head` is the meta and JSON-LD on their own and `.body` the noscript block, for pages that
+put facts in `<body>`; `.meta`, `.jsonLd` and `.html` are the 2.4.0 values.
+
+```js
+const summary = seo.pageSummary({ title: 'Start', summary: 'How to set up Example in five minutes',
+ url: 'https://example.com/start', updated: '2026-10-01', facts: ['Takes five minutes', 'No account needed'] });
+`<head>${seo.headTags({ title: 'Start', description: 'How to set up Example', canonical: 'https://example.com/start' })}
+${summary.head}</head><body>${summary.body}…`; // or `${summary}` in <head> when there are no facts
+```
+
+```js
const site = { name: 'Example', url: 'https://example.com/' };
const site = { name: 'Example', url: 'https://example.com/' };
-const full = seo.llmsFull({ site, summary: 'What Example publishes',
- sections: [{ title: 'Guide', url: '/guide', body: 'The full plain-text guide.' }],
- maxChars: 20000, maxTotal: 100000 }); // serve as text/plain at /llms-full.txt
-const page = seo.pageSummary({ title: 'Guide', summary: 'A short guide',
- facts: [['Topic', 'Getting started']], url: 'https://example.com/guide', updated: '2026-10-01' });
-// page.meta is an escaped description attribute value; page.jsonLd is a WebPage object;
-// page.html is an escaped, hidden summary section. Use jsonLdTag(page.jsonLd) in <head>.
const feed = { title: 'Example', link: site.url, description: 'Recent guides',
const feed = { title: 'Example', link: site.url, description: 'Recent guides',
updated: '2026-10-01T12:00:00Z', items: [{ title: 'Guide', url: '/guide', id: 'guide-1',
updated: '2026-10-01T12:00:00Z', items: [{ title: 'Guide', url: '/guide', id: 'guide-1',
published: '2026-10-01T12:00:00Z', summary: 'A short guide', author: 'Example' }] };
published: '2026-10-01T12:00:00Z', summary: 'A short guide', author: 'Example' }] };
@@ -84,7 +108,7 @@ seo.feedLinkTags({ rss: 'https://example.com/feed.xml', atom: 'https://example.c
@@ -84,7 +108,7 @@ seo.feedLinkTags({ rss: 'https://example.com/feed.xml', atom: 'https://example.c
`jsonLd` also builds `product`, `review`, `aggregateRating`, `imageGallery` (an ImageObject
`jsonLd` also builds `product`, `review`, `aggregateRating`, `imageGallery` (an ImageObject
array) and `videoObject` (the same VideoObject shape as `video`). Feed URLs should be absolute,
array) and `videoObject` (the same VideoObject shape as `video`). Feed URLs should be absolute,
-or relative to the absolute `link`; `llmsFull` resolves section URLs against `site.url`.
+or relative to the absolute `link`; `llmsFull` resolves page URLs against `base` (or `site.url`).
### Images and responsive pictures (`assets.js`)
### Images and responsive pictures (`assets.js`)
@@ -205,29 +205,89 @@ function limitText(value, max) {
@@ -205,29 +205,89 @@ function limitText(value, max) {
return size ? text.slice(0, size - 1).trimEnd() + '…' : '';
return size ? text.slice(0, size - 1).trimEnd() + '…' : '';
}
}
-/** /llms-full.txt: the llms.txt header followed by full, plain-text sections. */
-function llmsFull({ site, summary, sections = [], maxChars, maxTotal } = {}) {
+const ENTITIES = { amp: '&', lt: '<', gt: '>', quot: '"', apos: "'", nbsp: ' ' };
+
+/** Plain text from HTML: scripts, styles and comments dropped, block ends kept as line breaks. */
+function htmlText(html) {
+ return String(html == null ? '' : html)
+ .replace(/<!--[\s\S]*?-->/g, '')
+ .replace(/<(script|style|template)\b[\s\S]*?<\/\1\s*>/gi, '')
+ .replace(/<br\s*\/?>|<\/(li|dt|dd|tr)\s*>/gi, '\n')
+ .replace(/<\/(p|div|section|article|header|footer|main|aside|nav|ul|ol|dl|table|h[1-6]|blockquote|pre|figure)\s*>/gi, '\n\n')
+ .replace(/<[^>]*>/g, '')
+ .replace(/&(#x[0-9a-f]+|#\d+|[a-z]+);/gi, (m, e) => {
+ if (e[0] !== '#') return ENTITIES[e.toLowerCase()] || m;
+ const code = e[1] === 'x' || e[1] === 'X' ? parseInt(e.slice(2), 16) : parseInt(e.slice(1), 10);
+ return code > 0 && code <= 0x10ffff ? String.fromCodePoint(code) : m;
+ })
+ .split('\n').map((line) => line.replace(/\s+/g, ' ').trim()).join('\n')
+ .replace(/\n{3,}/g, '\n\n')
+ .trim();
+}
+
+/** One /llms-full.txt page: `### title`, `URL: …` and its full text (`text`, or `html` stripped to text). */
+function llmsPage(page, base) {
+ const url = absolute(page.url, base);
+ const text = page.text != null ? String(page.text).replace(/\r\n?/g, '\n').trim() : htmlText(page.html);
+ return `\n### ${page.title || ''}\n${url ? `\nURL: ${url}\n` : ''}${text ? `\n${text}\n` : ''}`;
+}
+
+/**
+ * /llms-full.txt: the llms.txt header followed by full, plain-text sections. A section with
+ * `pages: [{ title, url, text | html }]` renders `## title` then every page in input order, never
+ * clipped; `maxBytes` stops before the first page that would pass it and ends with a
+ * `(truncated: N more pages at <base>/llms.txt)` line. A section with `url`/`body` renders as in 2.4.0.
+ */
+function llmsFull({ site, summary, base, sections = [], maxChars, maxTotal, maxBytes } = {}) {
const name = site && typeof site === 'object' ? site.name || site.title : site;
const name = site && typeof site === 'object' ? site.name || site.title : site;
- const base = site && typeof site === 'object' ? site.url : site;
+ base = base || (site && typeof site === 'object' ? site.url : site);
let out = llmsTxt({ name, summary });
let out = llmsTxt({ name, summary });
+ let left = 0;
for (const section of sections) {
for (const section of sections) {
if (!section) continue;
if (!section) continue;
+ if (Array.isArray(section.pages)) {
+ let heading = `\n## ${section.title || ''}\n`;
+ for (const page of section.pages) {
+ if (!page) continue;
+ const chunk = heading + llmsPage(page, base);
+ if (left || (maxBytes != null && Buffer.byteLength(out + chunk) > maxBytes)) { left++; continue; }
+ out += chunk;
+ heading = '';
+ }
+ continue;
+ }
+ if (left) continue;
const url = absolute(section.url, base);
const url = absolute(section.url, base);
const body = limitText(String(section.body == null ? '' : section.body).trim(), section.maxChars == null ? maxChars : section.maxChars);
const body = limitText(String(section.body == null ? '' : section.body).trim(), section.maxChars == null ? maxChars : section.maxChars);
out += `\n## ${section.title || ''}\n\n${url}\n\n${body}\n`;
out += `\n## ${section.title || ''}\n\n${url}\n\n${body}\n`;
}
}
+ if (left) out += `\n(truncated: ${left} more page${left === 1 ? '' : 's'} at ${absolute('/llms.txt', base)})\n`;
return limitText(out, maxTotal);
return limitText(out, maxTotal);
}
}
-/** A description value, WebPage data, and a hidden text snapshot for a page. */
+/**
+ * AI-readable page summary. As a string (`${seo.pageSummary(…)}`, `String(…)`) it is the HTML:
+ * `<meta name="ai-summary">` (clipped to 160 like the description), a WebPage JSON-LD tag and, when
+ * `facts` (strings) are given, a `<noscript><section data-ai-summary>` block. `.head` and `.body` hold
+ * the head part and the noscript block on their own; `.meta`, `.jsonLd` and `.html` are the 2.4.0 values.
+ */
function pageSummary({ title, summary, facts = [], url, updated } = {}) {
function pageSummary({ title, summary, facts = [], url, updated } = {}) {
const pageUrl = absolute(url);
const pageUrl = absolute(url);
const text = String(summary == null ? '' : summary);
const text = String(summary == null ? '' : summary);
const jsonLd = { '@context': 'https://schema.org', '@type': 'WebPage', name: title, url: pageUrl,
const jsonLd = { '@context': 'https://schema.org', '@type': 'WebPage', name: title, url: pageUrl,
abstract: text, ...(updated ? { dateModified: updated } : {}) };
abstract: text, ...(updated ? { dateModified: updated } : {}) };
- const rows = facts.map(([key, value]) => `<dt>${esc(key)}</dt><dd>${esc(value)}</dd>`).join('');
+ const rows = facts.filter(Array.isArray).map(([key, value]) => `<dt>${esc(key)}</dt><dd>${esc(value)}</dd>`).join('');
const html = `<section data-ov-summary hidden><h2>${esc(title)}</h2><p>${esc(text)}</p>${rows ? `<dl>${rows}</dl>` : ''}${pageUrl ? `<a href="${esc(pageUrl)}">${esc(pageUrl)}</a>` : ''}${updated ? `<time datetime="${esc(updated)}">${esc(updated)}</time>` : ''}</section>`;
const html = `<section data-ov-summary hidden><h2>${esc(title)}</h2><p>${esc(text)}</p>${rows ? `<dl>${rows}</dl>` : ''}${pageUrl ? `<a href="${esc(pageUrl)}">${esc(pageUrl)}</a>` : ''}${updated ? `<time datetime="${esc(updated)}">${esc(updated)}</time>` : ''}</section>`;
- return { meta: esc(clip(text, 160)), jsonLd, html };
+ const modified = feedDate(updated, 'iso');
+ const ld = { '@context': 'https://schema.org', '@type': 'WebPage', ...(title != null ? { name: String(title) } : {}),
+ description: text, ...(pageUrl ? { url: pageUrl } : {}), ...(modified ? { dateModified: modified } : {}), abstract: text };
+ const head = [text ? `<meta name="ai-summary" content="${esc(clip(text, 160))}">` : '', jsonLdTag(ld)].filter(Boolean).join('\n');
+ const items = facts.map((f) => (Array.isArray(f) ? f.filter((x) => x != null && x !== '').join(': ') : String(f == null ? '' : f)).trim()).filter(Boolean);
+ const body = items.length ? `<noscript><section data-ai-summary><h2>${esc(title)}</h2><p>${esc(text)}</p><ul>${items.map((f) => `<li>${esc(f)}</li>`).join('')}</ul></section></noscript>` : '';
+ const result = { meta: esc(clip(text, 160)), jsonLd, html };
+ const tags = body ? `${head}\n${body}` : head;
+ Object.defineProperties(result, { head: { value: head }, body: { value: body }, toString: { value: () => tags } });
+ return result;
}
}
function feedDate(value, format) {
function feedDate(value, format) {
@@ -29,14 +29,21 @@ app.get('/llms.txt', (req, res) => {
@@ -29,14 +29,21 @@ app.get('/llms.txt', (req, res) => {
res.type('text/plain').send(seo.llmsTxt({ name, summary: name, sections: [{ title: 'Pages', links: [{ title: 'Home', url: home }] }] }));
res.type('text/plain').send(seo.llmsTxt({ name, summary: name, sections: [{ title: 'Pages', links: [{ title: 'Home', url: home }] }] }));
});
});
+app.get('/llms-full.txt', (req, res) => {
+ res.set('Cache-Control', cache.assetHeaders(req.path));
+ res.type('text/plain').send(seo.llmsFull({ site: name, summary: name, base: home, sections: [{ title: 'Pages', pages: [{ title: 'Home', url: '/', text: name }] }] }));
+});
+
app.get('/', cache.applyHtml(), (req, res) => {
app.get('/', cache.applyHtml(), (req, res) => {
const head = seo.headTags({ title: name, siteName: name, description: name, canonical: home });
const head = seo.headTags({ title: name, siteName: name, description: name, canonical: home });
+ const summary = seo.pageSummary({ title: name, summary: name, url: home });
res.type('html').send(`<!doctype html>
res.type('html').send(`<!doctype html>
<html lang="en">
<html lang="en">
<head>
<head>
<meta charset="utf-8">
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="viewport" content="width=device-width, initial-scale=1">
${head}
${head}
+${summary}
<script src="${serve.url('web-runtime.js')}" defer></script>
<script src="${serve.url('web-runtime.js')}" defer></script>
<script src="${serve.url('navbar.js')}" defer></script>
<script src="${serve.url('navbar.js')}" defer></script>
<script src="${serve.url('footer.js')}" defer></script>
<script src="${serve.url('footer.js')}" defer></script>
@@ -26,6 +26,8 @@ function request(server, path) {
@@ -26,6 +26,8 @@ function request(server, path) {
assert.equal(home.status, 200);
assert.equal(home.status, 200);
assert.ok(home.body.includes(`<link rel="canonical" href="${homeUrl}">`));
assert.ok(home.body.includes(`<link rel="canonical" href="${homeUrl}">`));
assert.ok(home.body.includes(`<meta property="og:url" content="${homeUrl}">`));
assert.ok(home.body.includes(`<meta property="og:url" content="${homeUrl}">`));
+ assert.match(home.body, /<meta name="ai-summary" content="[^"]+">/);
+ assert.match(home.body, /"@type":"WebPage"/);
assert.match(home.body, /OpenVibeNavbar\.init\(/);
assert.match(home.body, /OpenVibeNavbar\.init\(/);
assert.match(home.body, /OVWebRuntime\.create\(\{ features: \{\}, routes: \[\] \}\)\.boot\(\)/);
assert.match(home.body, /OVWebRuntime\.create\(\{ features: \{\}, routes: \[\] \}\)\.boot\(\)/);
assert.equal(home.headers['cache-control'], 'public, max-age=120, stale-while-revalidate=3600');
assert.equal(home.headers['cache-control'], 'public, max-age=120, stale-while-revalidate=3600');
@@ -44,6 +46,11 @@ function request(server, path) {
@@ -44,6 +46,11 @@ function request(server, path) {
assert.equal(llms.status, 200);
assert.equal(llms.status, 200);
assert.match(llms.headers['content-type'], /^text\/plain/);
assert.match(llms.headers['content-type'], /^text\/plain/);
+ const llmsFull = await request(server, '/llms-full.txt');
+ assert.equal(llmsFull.status, 200);
+ assert.match(llmsFull.headers['content-type'], /^text\/plain/);
+ assert.ok(llmsFull.body.includes(`URL: ${homeUrl}\n`));
+
const script = await request(server, '/shared/web-runtime.js');
const script = await request(server, '/shared/web-runtime.js');
assert.equal(script.status, 200);
assert.equal(script.status, 200);
assert.equal(script.headers['cache-control'], 'public, max-age=300, stale-while-revalidate=60');
assert.equal(script.headers['cache-control'], 'public, max-age=300, stale-while-revalidate=60');
new file mode 100644
new file mode 100644
@@ -0,0 +1,69 @@
@@ -0,0 +1,69 @@
+'use strict';
+const assert = require('assert');
+const seo = require('../seo');
+
+for (const name of ['llmsFull', 'pageSummary']) assert.strictEqual(typeof seo[name], 'function', `${name} resolves`);
+
+const input = { site: 'Example', summary: ' Docs and blog ', base: 'https://example.test/', sections: [
+ { title: 'Docs', pages: [
+ { title: 'Intro', url: '/intro', text: 'Line one.\r\nLine two.\n\nLast paragraph.' },
+ { title: 'API', url: 'api/v1', html: '<h1>API</h1><!-- note --><p>Use <b>get()</b> & <i>set()</i>.</p><style>p{}</style><script>alert("x")</script><ul><li>One</li><li>Two!</li></ul>' },
+ ] },
+ { title: 'Empty', pages: [] },
+ { title: 'Blog', pages: [{ title: 'Launch', url: 'https://other.test/launch' }] },
+] };
+const full = seo.llmsFull(input);
+assert.strictEqual(full, seo.llmsFull(input), 'deterministic');
+assert.ok(full.startsWith(seo.llmsTxt({ name: 'Example', summary: ' Docs and blog ' })), 'llms.txt header');
+assert.ok(full.endsWith('\n') && !full.endsWith('\n\n'), 'one trailing newline');
+assert.ok(full.indexOf('## Docs') < full.indexOf('### Intro') && full.indexOf('### Intro') < full.indexOf('### API')
+ && full.indexOf('### API') < full.indexOf('## Blog') && full.indexOf('## Blog') < full.indexOf('### Launch'), 'input order kept');
+assert.ok(full.includes('### Intro\n\nURL: https://example.test/intro\n\nLine one.\nLine two.\n\nLast paragraph.\n'), 'text kept with its newlines');
+assert.ok(full.includes('URL: https://example.test/api/v1\n\nAPI\n\nUse get() & set().\n\nOne\nTwo!\n'), 'html stripped to text');
+assert.ok(!/<|alert|p\{\}|note/.test(full), 'no tags, scripts, styles or comments');
+assert.ok(full.endsWith('### Launch\n\nURL: https://other.test/launch\n'), 'a page without text lists its URL only');
+assert.ok(!full.includes('## Empty'), 'an empty section is left out');
+assert.strictEqual(seo.llmsFull({ site: 'X', summary: 's', sections: [{ title: 'None', pages: [] }] }), seo.llmsTxt({ name: 'X', summary: 's' }));
+const long = 'word '.repeat(5000).trim();
+assert.ok(seo.llmsFull({ site: 'X', base: 'https://x.test', sections: [{ title: 'A', pages: [{ title: 'Long', url: '/', text: long }] }] }).includes(long), 'body never clipped');
+assert.match(seo.llmsFull({ site: { name: 'S', url: 'https://s.test/' }, sections: [{ title: 'A', pages: [{ title: 'P', url: '/p' }] }] }), /URL: https:\/\/s\.test\/p\n/, 'site.url is the base when base is absent');
+
+const pages = (n) => Array.from({ length: n }, (_, i) => ({ title: `P${i}`, url: `/p${i}`, text: `Body ${i} ünïcode` }));
+const all = seo.llmsFull({ site: 'X', base: 'https://x.test/', sections: [{ title: 'A', pages: pages(2) }, { title: 'B', pages: pages(2) }] });
+const upTo = (marker) => Buffer.byteLength(all.slice(0, all.indexOf(marker)));
+const cut = seo.llmsFull({ site: 'X', base: 'https://x.test/', maxBytes: upTo('### P1'), sections: [{ title: 'A', pages: pages(2) }, { title: 'B', pages: pages(2) }] });
+assert.ok(cut.includes('### P0') && !cut.includes('### P1') && !cut.includes('## B'), 'stops before the page that would pass maxBytes');
+assert.ok(cut.endsWith('\n\n(truncated: 3 more pages at https://x.test/llms.txt)\n'), 'truncation line counts every page left');
+assert.ok(Buffer.byteLength(cut.slice(0, cut.indexOf('\n(truncated'))) <= upTo('### P1'), 'content stays within maxBytes');
+assert.strictEqual(seo.llmsFull({ site: 'X', base: 'https://x.test/', maxBytes: Buffer.byteLength(all), sections: [{ title: 'A', pages: pages(2) }, { title: 'B', pages: pages(2) }] }), all, 'no line when everything fits');
+assert.match(seo.llmsFull({ site: 'X', base: 'https://x.test/', maxBytes: upTo('## B'), sections: [{ title: 'A', pages: pages(2) }, { title: 'B', pages: pages(1) }] }), /\n\(truncated: 1 more page at https:\/\/x\.test\/llms\.txt\)\n$/);
+
+const ps = seo.pageSummary({ title: 'Guide "A" <b> & co', summary: 'Say "hi" <b>bold</b> & </script><script>alert(1)</script>', url: 'https://example.test/g?a=1&b=2', updated: '2026-10-01T12:00:00Z', facts: ['Fast & "small"', '<i>Two</i>'] });
+const html = String(ps);
+assert.strictEqual(`${ps}`, html);
+assert.strictEqual(html, `${ps.head}\n${ps.body}`);
+assert.ok(html.includes('<meta name="ai-summary" content="Say "hi" <b>bold</b> & </script><script>alert(1)</script>">'), 'meta content escaped');
+assert.ok(!html.includes('name="description"'), 'headTags owns the description');
+assert.strictEqual(html.match(/<\/script>/g).length, 1, '</script> in the summary cannot close the JSON-LD tag');
+const ld = JSON.parse(html.match(/<script type="application\/ld\+json">(.*?)<\/script>/)[1]);
+assert.deepStrictEqual(ld, { '@context': 'https://schema.org', '@type': 'WebPage', name: 'Guide "A" <b> & co',
+ description: 'Say "hi" <b>bold</b> & </script><script>alert(1)</script>', url: 'https://example.test/g?a=1&b=2',
+ dateModified: '2026-10-01T12:00:00.000Z', abstract: 'Say "hi" <b>bold</b> & </script><script>alert(1)</script>' });
+assert.ok(html.includes(seo.jsonLdTag(ld)), 'built with jsonLdTag');
+assert.ok(ps.body.startsWith('<noscript><section data-ai-summary><h2>Guide "A" <b> & co</h2><p>Say "hi"'));
+assert.ok(ps.body.endsWith('<ul><li>Fast & "small"</li><li><i>Two</i></li></ul></section></noscript>'));
+assert.deepStrictEqual(Object.keys(ps), ['meta', 'jsonLd', 'html'], 'the 2.4.0 result shape is unchanged');
+
+const bare = seo.pageSummary({ title: 'T', summary: 'S', url: '/t' });
+assert.ok(!String(bare).includes('<noscript'), 'no noscript block without facts');
+assert.strictEqual(bare.body, '');
+assert.ok(!String(seo.pageSummary({ title: 'T', summary: 'S', facts: [] })).includes('<noscript'));
+assert.ok(!/dateModified/.test(String(bare)) && !/dateModified/.test(String(seo.pageSummary({ summary: 'S', updated: 'not a date' }))), 'dateModified only for a valid date');
+
+const words = 'alpha beta gamma delta '.repeat(20);
+const content = String(seo.pageSummary({ title: 'T', summary: words })).match(/name="ai-summary" content="([^"]*)"/)[1];
+assert.strictEqual(content, seo.clip(words, 160), 'clipped like the description');
+assert.ok(content.length <= 160 && content.endsWith('…'));
+assert.ok(String(seo.pageSummary({ title: 'T', summary: words })).includes(`"abstract":"${words}"`), 'JSON-LD keeps the full summary');
+
+console.log('seo-llms-full: all checks passed');