openvibers/openvibe.shared Commits

Render page lists in llmsFull and an ai-summary block in pageSummary (#7)

OP OpenVibers committed 2 Oct 2026 · a72d4caf7b · 6 files, +190 −16

local checks ✓ 3/3 · CI ✓ 3/3 · review SHIP by space-bunny-alpha

https://github.com/OpenVibers/OpenVibe.Shared/pull/7

Changes

ddiff --git a/CHANGELOG.md b/CHANGELOG.md
iindex 263e33f..8fa18d4 100644
---- a/CHANGELOG.md
++++ b/CHANGELOG.md
@@@ -6,6 +6,13 @@ A release is the git tag `vX.Y.Z`; consumers pin the tag's tarball (see README).
## Unreleased
++- `seo.llmsFull` takes `base` and sections of `pages: [{ title, url, text | html }]`: each page
++ renders as `### title`, `URL: …` and its full text (HTML stripped), in input order, and
++ `maxBytes` ends the file with a `(truncated: N more pages at <base>/llms.txt)` line. The
++ 2.4.0 `{ url, body }` sections render unchanged. `seo.pageSummary` now converts to an HTML
++ string — `<meta name="ai-summary">`, WebPage JSON-LD and, with string `facts`, a
++ `<noscript><section data-ai-summary>` block — and has `.head`/`.body`; its `meta`, `jsonLd` and
++ `html` are unchanged. `templates/site` serves `/llms-full.txt` and puts the summary in its head.
- `perf-budget` measures Core Web Vitals (plan T11). `measure({ browser })` opens the page in
Chrome through `browser-harness` after the byte pass, clicks once, and adds
`cwv: { lcpMs, inpMs, cls }`. `check()` accepts `lcpMs`, `inpMs` and `cls` budgets and throws
ddiff --git a/README.md b/README.md
iindex af51b07..8c99a14 100644
---- a/README.md
++++ b/README.md
@@@ -39,7 +39,7 @@ npm run create-site -- my-site
```
The command creates `./my-site` with a pinned Shared dependency, a small Express server,
--discovery routes, and a smoke test. Set `SITE_URL` to the site's public origin before serving it.
++discovery routes (`/robots.txt`, `/sitemap.xml`, `/llms.txt`, `/llms-full.txt`), and a smoke test. Set `SITE_URL` to the site's public origin before serving it.
### Server side: pin a release tarball
@@@ -62,18 +62,42 @@ const { AnalyticsTracker } = require('openvibe-shared/analytics');
### SEO kit
`openvibe-shared/seo` exports the page head, sitemap, robots and `llms.txt` helpers, plus
--the discovery formats below. The package root also exports them as `seo`.
++the discovery formats below. The package root also exports them as `seo`. A site serves
++`/robots.txt`, `/sitemap.xml`, `/llms.txt` and `/llms-full.txt` (`templates/site` has all four).
++
++`llmsFull({ site, summary, base, sections, maxBytes })` renders `/llms-full.txt`: the `llms.txt`
++header, then `## <section>` and, for each page, `### <title>`, `URL: <absolute url>` and the page's
++full plain text — `text` as given (newlines kept) or `html` stripped to text; a page with neither is
++listed with its URL only. Input order is kept and bodies are never clipped. With `maxBytes` it stops
++before the first page that would pass the limit and ends with
++`(truncated: N more pages at <base>/llms.txt)`. Sections with no pages are left out. (Sections in the
++2.4.0 shape, `{ title, url, body }` with `maxChars`/`maxTotal`, still render as before.)
```js
const seo = require('openvibe-shared/seo');
++const full = seo.llmsFull({ site: 'Example', summary: 'What Example publishes', base: 'https://example.com/',
++ sections: [{ title: 'Guides', pages: [{ title: 'Start', url: '/start', text: 'Step one.\nStep two.' },
++ { title: 'API', url: '/api', html: '<h1>API</h1><p>Call get().</p>' }]}],
++ maxBytes: 500000 }); // serve as text/plain at /llms-full.txt
++```
++
++`pageSummary({ title, summary, url, updated, facts })` is the AI-readable summary of a page. As a
++string it is `<meta name="ai-summary">` (the summary clipped to 160 like the description, which
++`headTags` still owns), a WebPage JSON-LD tag (`name`, `description`, `url`, `dateModified` as ISO,
++`abstract`; written with `jsonLdTag`) and, when `facts` (strings) are given, a
++`<noscript><section data-ai-summary>` block with the title, summary and facts. Everything is
++escaped. `.head` is the meta and JSON-LD on their own and `.body` the noscript block, for pages that
++put facts in `<body>`; `.meta`, `.jsonLd` and `.html` are the 2.4.0 values.
++
++```js
++const summary = seo.pageSummary({ title: 'Start', summary: 'How to set up Example in five minutes',
++ url: 'https://example.com/start', updated: '2026-10-01', facts: ['Takes five minutes', 'No account needed'] });
++`<head>${seo.headTags({ title: 'Start', description: 'How to set up Example', canonical: 'https://example.com/start' })}
++${summary.head}</head><body>${summary.body}…`; // or `${summary}` in <head> when there are no facts
++```
++
++```js
const site = { name: 'Example', url: 'https://example.com/' };
--const full = seo.llmsFull({ site, summary: 'What Example publishes',
-- sections: [{ title: 'Guide', url: '/guide', body: 'The full plain-text guide.' }],
-- maxChars: 20000, maxTotal: 100000 }); // serve as text/plain at /llms-full.txt
--const page = seo.pageSummary({ title: 'Guide', summary: 'A short guide',
-- facts: [['Topic', 'Getting started']], url: 'https://example.com/guide', updated: '2026-10-01' });
--// page.meta is an escaped description attribute value; page.jsonLd is a WebPage object;
--// page.html is an escaped, hidden summary section. Use jsonLdTag(page.jsonLd) in <head>.
const feed = { title: 'Example', link: site.url, description: 'Recent guides',
updated: '2026-10-01T12:00:00Z', items: [{ title: 'Guide', url: '/guide', id: 'guide-1',
published: '2026-10-01T12:00:00Z', summary: 'A short guide', author: 'Example' }] };
@@@ -84,7 +108,7 @@ seo.feedLinkTags({ rss: 'https://example.com/feed.xml', atom: 'https://example.c
`jsonLd` also builds `product`, `review`, `aggregateRating`, `imageGallery` (an ImageObject
array) and `videoObject` (the same VideoObject shape as `video`). Feed URLs should be absolute,
--or relative to the absolute `link`; `llmsFull` resolves section URLs against `site.url`.
++or relative to the absolute `link`; `llmsFull` resolves page URLs against `base` (or `site.url`).
### Images and responsive pictures (`assets.js`)
ddiff --git a/seo.js b/seo.js
iindex 188083e..1ee39a8 100644
---- a/seo.js
++++ b/seo.js
@@@ -205,29 +205,89 @@ function limitText(value, max) {
return size ? text.slice(0, size - 1).trimEnd() + '…' : '';
}
--/** /llms-full.txt: the llms.txt header followed by full, plain-text sections. */
--function llmsFull({ site, summary, sections = [], maxChars, maxTotal } = {}) {
++const ENTITIES = { amp: '&', lt: '<', gt: '>', quot: '"', apos: "'", nbsp: ' ' };
++
++/** Plain text from HTML: scripts, styles and comments dropped, block ends kept as line breaks. */
++function htmlText(html) {
++ return String(html == null ? '' : html)
++ .replace(/<!--[\s\S]*?-->/g, '')
++ .replace(/<(script|style|template)\b[\s\S]*?<\/\1\s*>/gi, '')
++ .replace(/<br\s*\/?>|<\/(li|dt|dd|tr)\s*>/gi, '\n')
++ .replace(/<\/(p|div|section|article|header|footer|main|aside|nav|ul|ol|dl|table|h[1-6]|blockquote|pre|figure)\s*>/gi, '\n\n')
++ .replace(/<[^>]*>/g, '')
++ .replace(/&(#x[0-9a-f]+|#\d+|[a-z]+);/gi, (m, e) => {
++ if (e[0] !== '#') return ENTITIES[e.toLowerCase()] || m;
++ const code = e[1] === 'x' || e[1] === 'X' ? parseInt(e.slice(2), 16) : parseInt(e.slice(1), 10);
++ return code > 0 && code <= 0x10ffff ? String.fromCodePoint(code) : m;
++ })
++ .split('\n').map((line) => line.replace(/\s+/g, ' ').trim()).join('\n')
++ .replace(/\n{3,}/g, '\n\n')
++ .trim();
++}
++
++/** One /llms-full.txt page: `### title`, `URL: …` and its full text (`text`, or `html` stripped to text). */
++function llmsPage(page, base) {
++ const url = absolute(page.url, base);
++ const text = page.text != null ? String(page.text).replace(/\r\n?/g, '\n').trim() : htmlText(page.html);
++ return `\n### ${page.title || ''}\n${url ? `\nURL: ${url}\n` : ''}${text ? `\n${text}\n` : ''}`;
++}
++
++/**
++ * /llms-full.txt: the llms.txt header followed by full, plain-text sections. A section with
++ * `pages: [{ title, url, text | html }]` renders `## title` then every page in input order, never
++ * clipped; `maxBytes` stops before the first page that would pass it and ends with a
++ * `(truncated: N more pages at <base>/llms.txt)` line. A section with `url`/`body` renders as in 2.4.0.
++ */
++function llmsFull({ site, summary, base, sections = [], maxChars, maxTotal, maxBytes } = {}) {
const name = site && typeof site === 'object' ? site.name || site.title : site;
-- const base = site && typeof site === 'object' ? site.url : site;
++ base = base || (site && typeof site === 'object' ? site.url : site);
let out = llmsTxt({ name, summary });
++ let left = 0;
for (const section of sections) {
if (!section) continue;
++ if (Array.isArray(section.pages)) {
++ let heading = `\n## ${section.title || ''}\n`;
++ for (const page of section.pages) {
++ if (!page) continue;
++ const chunk = heading + llmsPage(page, base);
++ if (left || (maxBytes != null && Buffer.byteLength(out + chunk) > maxBytes)) { left++; continue; }
++ out += chunk;
++ heading = '';
++ }
++ continue;
++ }
++ if (left) continue;
const url = absolute(section.url, base);
const body = limitText(String(section.body == null ? '' : section.body).trim(), section.maxChars == null ? maxChars : section.maxChars);
out += `\n## ${section.title || ''}\n\n${url}\n\n${body}\n`;
}
++ if (left) out += `\n(truncated: ${left} more page${left === 1 ? '' : 's'} at ${absolute('/llms.txt', base)})\n`;
return limitText(out, maxTotal);
}
--/** A description value, WebPage data, and a hidden text snapshot for a page. */
++/**
++ * AI-readable page summary. As a string (`${seo.pageSummary(…)}`, `String(…)`) it is the HTML:
++ * `<meta name="ai-summary">` (clipped to 160 like the description), a WebPage JSON-LD tag and, when
++ * `facts` (strings) are given, a `<noscript><section data-ai-summary>` block. `.head` and `.body` hold
++ * the head part and the noscript block on their own; `.meta`, `.jsonLd` and `.html` are the 2.4.0 values.
++ */
function pageSummary({ title, summary, facts = [], url, updated } = {}) {
const pageUrl = absolute(url);
const text = String(summary == null ? '' : summary);
const jsonLd = { '@context': 'https://schema.org', '@type': 'WebPage', name: title, url: pageUrl,
abstract: text, ...(updated ? { dateModified: updated } : {}) };
-- const rows = facts.map(([key, value]) => `<dt>${esc(key)}</dt><dd>${esc(value)}</dd>`).join('');
++ const rows = facts.filter(Array.isArray).map(([key, value]) => `<dt>${esc(key)}</dt><dd>${esc(value)}</dd>`).join('');
const html = `<section data-ov-summary hidden><h2>${esc(title)}</h2><p>${esc(text)}</p>${rows ? `<dl>${rows}</dl>` : ''}${pageUrl ? `<a href="${esc(pageUrl)}">${esc(pageUrl)}</a>` : ''}${updated ? `<time datetime="${esc(updated)}">${esc(updated)}</time>` : ''}</section>`;
-- return { meta: esc(clip(text, 160)), jsonLd, html };
++ const modified = feedDate(updated, 'iso');
++ const ld = { '@context': 'https://schema.org', '@type': 'WebPage', ...(title != null ? { name: String(title) } : {}),
++ description: text, ...(pageUrl ? { url: pageUrl } : {}), ...(modified ? { dateModified: modified } : {}), abstract: text };
++ const head = [text ? `<meta name="ai-summary" content="${esc(clip(text, 160))}">` : '', jsonLdTag(ld)].filter(Boolean).join('\n');
++ const items = facts.map((f) => (Array.isArray(f) ? f.filter((x) => x != null && x !== '').join(': ') : String(f == null ? '' : f)).trim()).filter(Boolean);
++ const body = items.length ? `<noscript><section data-ai-summary><h2>${esc(title)}</h2><p>${esc(text)}</p><ul>${items.map((f) => `<li>${esc(f)}</li>`).join('')}</ul></section></noscript>` : '';
++ const result = { meta: esc(clip(text, 160)), jsonLd, html };
++ const tags = body ? `${head}\n${body}` : head;
++ Object.defineProperties(result, { head: { value: head }, body: { value: body }, toString: { value: () => tags } });
++ return result;
}
function feedDate(value, format) {
ddiff --git a/templates/site/server/app.js b/templates/site/server/app.js
iindex ea68863..0c0f54b 100644
---- a/templates/site/server/app.js
++++ b/templates/site/server/app.js
@@@ -29,14 +29,21 @@ app.get('/llms.txt', (req, res) => {
res.type('text/plain').send(seo.llmsTxt({ name, summary: name, sections: [{ title: 'Pages', links: [{ title: 'Home', url: home }] }] }));
});
++app.get('/llms-full.txt', (req, res) => {
++ res.set('Cache-Control', cache.assetHeaders(req.path));
++ res.type('text/plain').send(seo.llmsFull({ site: name, summary: name, base: home, sections: [{ title: 'Pages', pages: [{ title: 'Home', url: '/', text: name }] }] }));
++});
++
app.get('/', cache.applyHtml(), (req, res) => {
const head = seo.headTags({ title: name, siteName: name, description: name, canonical: home });
++ const summary = seo.pageSummary({ title: name, summary: name, url: home });
res.type('html').send(`<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
${head}
++${summary}
<script src="${serve.url('web-runtime.js')}" defer></script>
<script src="${serve.url('navbar.js')}" defer></script>
<script src="${serve.url('footer.js')}" defer></script>
ddiff --git a/templates/site/test/smoke.test.js b/templates/site/test/smoke.test.js
iindex b0e07dd..0add8b9 100644
---- a/templates/site/test/smoke.test.js
++++ b/templates/site/test/smoke.test.js
@@@ -26,6 +26,8 @@ function request(server, path) {
assert.equal(home.status, 200);
assert.ok(home.body.includes(`<link rel="canonical" href="${homeUrl}">`));
assert.ok(home.body.includes(`<meta property="og:url" content="${homeUrl}">`));
++ assert.match(home.body, /<meta name="ai-summary" content="[^"]+">/);
++ assert.match(home.body, /"@type":"WebPage"/);
assert.match(home.body, /OpenVibeNavbar\.init\(/);
assert.match(home.body, /OVWebRuntime\.create\(\{ features: \{\}, routes: \[\] \}\)\.boot\(\)/);
assert.equal(home.headers['cache-control'], 'public, max-age=120, stale-while-revalidate=3600');
@@@ -44,6 +46,11 @@ function request(server, path) {
assert.equal(llms.status, 200);
assert.match(llms.headers['content-type'], /^text\/plain/);
++ const llmsFull = await request(server, '/llms-full.txt');
++ assert.equal(llmsFull.status, 200);
++ assert.match(llmsFull.headers['content-type'], /^text\/plain/);
++ assert.ok(llmsFull.body.includes(`URL: ${homeUrl}\n`));
++
const script = await request(server, '/shared/web-runtime.js');
assert.equal(script.status, 200);
assert.equal(script.headers['cache-control'], 'public, max-age=300, stale-while-revalidate=60');
ddiff --git a/test/seo-llms-full.test.js b/test/seo-llms-full.test.js
nnew file mode 100644
iindex 0000000..2942fb6
---- /dev/null
++++ b/test/seo-llms-full.test.js
@@@ -0,0 +1,69 @@
++'use strict';
++const assert = require('assert');
++const seo = require('../seo');
++
++for (const name of ['llmsFull', 'pageSummary']) assert.strictEqual(typeof seo[name], 'function', `${name} resolves`);
++
++const input = { site: 'Example', summary: ' Docs and blog ', base: 'https://example.test/', sections: [
++ { title: 'Docs', pages: [
++ { title: 'Intro', url: '/intro', text: 'Line one.\r\nLine two.\n\nLast paragraph.' },
++ { title: 'API', url: 'api/v1', html: '<h1>API</h1><!-- note --><p>Use <b>get()</b> &amp; <i>set()</i>.</p><style>p{}</style><script>alert("x")</script><ul><li>One</li><li>Two&#33;</li></ul>' },
++ ] },
++ { title: 'Empty', pages: [] },
++ { title: 'Blog', pages: [{ title: 'Launch', url: 'https://other.test/launch' }] },
++] };
++const full = seo.llmsFull(input);
++assert.strictEqual(full, seo.llmsFull(input), 'deterministic');
++assert.ok(full.startsWith(seo.llmsTxt({ name: 'Example', summary: ' Docs and blog ' })), 'llms.txt header');
++assert.ok(full.endsWith('\n') && !full.endsWith('\n\n'), 'one trailing newline');
++assert.ok(full.indexOf('## Docs') < full.indexOf('### Intro') && full.indexOf('### Intro') < full.indexOf('### API')
++ && full.indexOf('### API') < full.indexOf('## Blog') && full.indexOf('## Blog') < full.indexOf('### Launch'), 'input order kept');
++assert.ok(full.includes('### Intro\n\nURL: https://example.test/intro\n\nLine one.\nLine two.\n\nLast paragraph.\n'), 'text kept with its newlines');
++assert.ok(full.includes('URL: https://example.test/api/v1\n\nAPI\n\nUse get() & set().\n\nOne\nTwo!\n'), 'html stripped to text');
++assert.ok(!/<|alert|p\{\}|note/.test(full), 'no tags, scripts, styles or comments');
++assert.ok(full.endsWith('### Launch\n\nURL: https://other.test/launch\n'), 'a page without text lists its URL only');
++assert.ok(!full.includes('## Empty'), 'an empty section is left out');
++assert.strictEqual(seo.llmsFull({ site: 'X', summary: 's', sections: [{ title: 'None', pages: [] }] }), seo.llmsTxt({ name: 'X', summary: 's' }));
++const long = 'word '.repeat(5000).trim();
++assert.ok(seo.llmsFull({ site: 'X', base: 'https://x.test', sections: [{ title: 'A', pages: [{ title: 'Long', url: '/', text: long }] }] }).includes(long), 'body never clipped');
++assert.match(seo.llmsFull({ site: { name: 'S', url: 'https://s.test/' }, sections: [{ title: 'A', pages: [{ title: 'P', url: '/p' }] }] }), /URL: https:\/\/s\.test\/p\n/, 'site.url is the base when base is absent');
++
++const pages = (n) => Array.from({ length: n }, (_, i) => ({ title: `P${i}`, url: `/p${i}`, text: `Body ${i} ünïcode` }));
++const all = seo.llmsFull({ site: 'X', base: 'https://x.test/', sections: [{ title: 'A', pages: pages(2) }, { title: 'B', pages: pages(2) }] });
++const upTo = (marker) => Buffer.byteLength(all.slice(0, all.indexOf(marker)));
++const cut = seo.llmsFull({ site: 'X', base: 'https://x.test/', maxBytes: upTo('### P1'), sections: [{ title: 'A', pages: pages(2) }, { title: 'B', pages: pages(2) }] });
++assert.ok(cut.includes('### P0') && !cut.includes('### P1') && !cut.includes('## B'), 'stops before the page that would pass maxBytes');
++assert.ok(cut.endsWith('\n\n(truncated: 3 more pages at https://x.test/llms.txt)\n'), 'truncation line counts every page left');
++assert.ok(Buffer.byteLength(cut.slice(0, cut.indexOf('\n(truncated'))) <= upTo('### P1'), 'content stays within maxBytes');
++assert.strictEqual(seo.llmsFull({ site: 'X', base: 'https://x.test/', maxBytes: Buffer.byteLength(all), sections: [{ title: 'A', pages: pages(2) }, { title: 'B', pages: pages(2) }] }), all, 'no line when everything fits');
++assert.match(seo.llmsFull({ site: 'X', base: 'https://x.test/', maxBytes: upTo('## B'), sections: [{ title: 'A', pages: pages(2) }, { title: 'B', pages: pages(1) }] }), /\n\(truncated: 1 more page at https:\/\/x\.test\/llms\.txt\)\n$/);
++
++const ps = seo.pageSummary({ title: 'Guide "A" <b> & co', summary: 'Say "hi" <b>bold</b> & </script><script>alert(1)</script>', url: 'https://example.test/g?a=1&b=2', updated: '2026-10-01T12:00:00Z', facts: ['Fast & "small"', '<i>Two</i>'] });
++const html = String(ps);
++assert.strictEqual(`${ps}`, html);
++assert.strictEqual(html, `${ps.head}\n${ps.body}`);
++assert.ok(html.includes('<meta name="ai-summary" content="Say &quot;hi&quot; &lt;b&gt;bold&lt;/b&gt; &amp; &lt;/script&gt;&lt;script&gt;alert(1)&lt;/script&gt;">'), 'meta content escaped');
++assert.ok(!html.includes('name="description"'), 'headTags owns the description');
++assert.strictEqual(html.match(/<\/script>/g).length, 1, '</script> in the summary cannot close the JSON-LD tag');
++const ld = JSON.parse(html.match(/<script type="application\/ld\+json">(.*?)<\/script>/)[1]);
++assert.deepStrictEqual(ld, { '@context': 'https://schema.org', '@type': 'WebPage', name: 'Guide "A" <b> & co',
++ description: 'Say "hi" <b>bold</b> & </script><script>alert(1)</script>', url: 'https://example.test/g?a=1&b=2',
++ dateModified: '2026-10-01T12:00:00.000Z', abstract: 'Say "hi" <b>bold</b> & </script><script>alert(1)</script>' });
++assert.ok(html.includes(seo.jsonLdTag(ld)), 'built with jsonLdTag');
++assert.ok(ps.body.startsWith('<noscript><section data-ai-summary><h2>Guide &quot;A&quot; &lt;b&gt; &amp; co</h2><p>Say &quot;hi&quot;'));
++assert.ok(ps.body.endsWith('<ul><li>Fast &amp; &quot;small&quot;</li><li>&lt;i&gt;Two&lt;/i&gt;</li></ul></section></noscript>'));
++assert.deepStrictEqual(Object.keys(ps), ['meta', 'jsonLd', 'html'], 'the 2.4.0 result shape is unchanged');
++
++const bare = seo.pageSummary({ title: 'T', summary: 'S', url: '/t' });
++assert.ok(!String(bare).includes('<noscript'), 'no noscript block without facts');
++assert.strictEqual(bare.body, '');
++assert.ok(!String(seo.pageSummary({ title: 'T', summary: 'S', facts: [] })).includes('<noscript'));
++assert.ok(!/dateModified/.test(String(bare)) && !/dateModified/.test(String(seo.pageSummary({ summary: 'S', updated: 'not a date' }))), 'dateModified only for a valid date');
++
++const words = 'alpha beta gamma delta '.repeat(20);
++const content = String(seo.pageSummary({ title: 'T', summary: words })).match(/name="ai-summary" content="([^"]*)"/)[1];
++assert.strictEqual(content, seo.clip(words, 160), 'clipped like the description');
++assert.ok(content.length <= 160 && content.endsWith('…'));
++assert.ok(String(seo.pageSummary({ title: 'T', summary: words })).includes(`"abstract":"${words}"`), 'JSON-LD keeps the full summary');
++
++console.log('seo-llms-full: all checks passed');