openvibers/openvibe.deals Commits

SEO-16/SEO-34: the sitemap is an index of bounded children, and llms.txt lists only URLs that answer (#133)

OP OpenVibers committed 15 days ago · 10987099f9 · 7 files, +136 −41

* SEO-16: the sitemap is an index of bounded children

/sitemap.xml named three sections whose catalog child was one unbounded <urlset>
(24,908 URLs today, read with files[0], so anything past the first 50,000 was
silently dropped). It is now a sitemap index (shared seo.sitemapIndexXml) over
/sitemap-pages.xml and one /sitemap-<n>.xml per 10,000 catalog URLs: every
indexable offer, then every product, in the order the site has always listed
them. Each child carries its own newest lastmod; a number past the last chunk
answers 404; robots.txt still names /sitemap.xml and the content type and
cache-policy headers are unchanged.

test/sitemap-build.test.js seeds 20,005 offers and a product: the index names
every child, no child holds more than 10,000, the union of the children is
exactly the old catalog sitemap in the same order, and /sitemap-3.xml,
/sitemap-99.xml, /sitemap-00.xml and /sitemap-01.xml are 404.

Co-authored-by: CommandCodeBot <[email protected]>

* llms.txt lists only URLs that answer (SEO-34): the Product JSON link pointed at /sitemaps/products.xml, which no longer exists once the sitemap is an index, and /sitemap.xml itself was a single 24,908-URL file a crawler waited out. The link now names the index, and the page test sweeps every URL in the file for an answer below 400.

---------

Co-authored-by: CommandCodeBot <[email protected]>

Changes

ddiff --git a/CHANGELOG.md b/CHANGELOG.md
iindex ef19fc6..be5d3ec 100644
---- a/CHANGELOG.md
++++ b/CHANGELOG.md
@@@ -2,6 +2,8 @@
  
  ## Unreleased
  
++- `llms.txt` lists only URLs that answer (SEO-34): the *Product JSON* link pointed at `/sitemaps/products.xml`, which no longer exists once the sitemap is an index, and `/sitemap.xml` itself was a single 24,908-URL file a crawler waited out. The link now names the index, and `test/pages.test.js` sweeps every URL in the file for an answer below 400.
++- The sitemaps are a bounded index (SEO-16): `GET /sitemap.xml` (openvibe-shared `seo.sitemapIndexXml`) names `/sitemap-pages.xml` and one `/sitemap-<n>.xml` child per 10,000 catalog URLs — every indexable offer, then every product, in the order the site has always listed them — so no child can pass a crawler's per-file limit and the union of the children is exactly the old catalog sitemap. A chunk number past the last answers 404; `robots.txt` still names `/sitemap.xml` and the content type and `cache-policy` headers are unchanged. `test/sitemap-build.test.js` seeds 20,005 offers and a product and pins all of it.
  - The front page is curated (`server/render/home.js`): a search and the site's numbers beside the hero, then editors' picks, hot, just in, ending soon, under $20, categories, shops and crowdfunding, how it works and six questions (FAQPage and ItemList JSON-LD). The hot list moved to `/hot` (`/?page=N` → 301); new `/ending`, `/picks`, `/c` and `/c/:category`; `/sitemaps/pages.xml` lists them all (the sitemap had no page a person browses). Imported deals keep their source's category and editors' pick (Sources feed@3; migration `0007_offer_category_pick.sql` adds `staff_pick`). Budgets: html 29.1 KB, css 24.2 KB.
  - Account export and deletion (ADR-033): `network.account.export_requested` and `network.account.deleted` are answered at `POST /internal/events` by `server/domain/account-data.js` over `openvibe-sdk/account-data` (SDK v0.37.0, was v0.35.0). The person's watches, votes, name and rate windows are deleted; what they contributed to the shared catalog stays without their id; staff moderation records stay attributed. The two subscriptions are created at boot. Migration `0003_account_data.sql` adds `account_data_events`. `test/account-data.test.js`.
  - openvibe-contracts moves from v0.79.0 to v0.97.0 (pin, lockfile and `node_modules`); nothing in the range breaks Deals, and the contracts' own service check is green. All twelve `deals.*` ids are now defined by the release, so `server/auth/capabilities.js` drops its local fallback for proposed ids and sends every check through the library's grant rule; `test/capabilities.test.js` pins that every guarded id is defined and that exact, prefix, denied and unknown ids answer as `capabilities.check()` does. README and STATUS.json name v0.97.0.
ddiff --git a/README.md b/README.md
iindex ffb5112..7f7f621 100644
---- a/README.md
++++ b/README.md
@@@ -74,7 +74,7 @@ gauges to `telemetry_samples` every minute and on stop, and deletes rows older t
  | `GET/POST /watches` | watches and saved searches; `POST /watches/:id/delete` |
  | `GET /mod` | moderators: open flags (incl. vote rings), text waiting for review, possible duplicates; `POST /mod/offers/:slug/(merge\|unmerge\|disable\|enable\|review\|expire)`, `POST /mod/flags/:id/(resolve\|dismiss)` |
  | `GET /feed.xml`, `/atom.xml`, `/feed.json` | newest active deals, with "as of" times and stable ids `deals:offer:<id>` |
--| `GET /robots.txt`, `/llms.txt`, `/sitemap.xml`, `/sitemaps/pages.xml`, `/sitemaps/offers.xml`, `/sitemaps/products.xml` | discovery (below); `pages.xml` lists the front page and every list (hot, new, ending, picks, each category and shop, crowdfunding) while it has deals |
++| `GET /robots.txt`, `/llms.txt`, `/sitemap.xml`, `/sitemap-pages.xml`, `/sitemap-<n>.xml` | discovery (below): the index names `sitemap-pages.xml` (the front page and every list — hot, new, ending, picks, each category and shop, crowdfunding — while it has deals) and one `/sitemap-<n>.xml` child per 10,000 catalog URLs (offers, then products, always in that order); a child number past the last is 404 |
  | `GET /api/health`, `/api/ready`, `/release.json`, `/metrics` | operations (`/metrics` loopback only; nginx never proxies it) |
  | `/auth/*` | Network SSO (the same session layer as Community/Blog) |
  
@@@ -137,7 +137,7 @@ two subscriptions are created at boot when missing (`startSubscriptions`, needs
  - The latest observation is **fresh** while younger than `DEALS_FRESHNESS_HOURS` (48 h), **stale**
    after; no observation is **unobserved**. Pages print "as of <time>", a stale price says it may no
    longer be available, and nothing on the site calls a price "current".
--- Stale → `noindex` (gate reason `stale_price`), out of `sitemaps/offers.xml`, the Offer JSON-LD
++- Stale → `noindex` (gate reason `stale_price`), out of the catalog sitemaps (`/sitemap-<n>.xml`), the Offer JSON-LD
    loses its price/currency/availability, and the worker re-sends the Search document (freshness
    facet, `noindex`) — only when it actually changed.
  - A price needs its currency: `10` without `USD` is refused (422), never completed. `$10` is refused
ddiff --git a/server/http/discovery.js b/server/http/discovery.js
iindex 68e2164..a8743d5 100644
---- a/server/http/discovery.js
++++ b/server/http/discovery.js
@@@ -7,13 +7,13 @@
   *   GET /llms.txt               orientation for language models
   *   GET /llms-full.txt          the same header plus a short summary of every indexable deal (never
   *                               the full source text)
-- *   GET /sitemap.xml            sitemap index over the sections below
-- *   GET /sitemaps/offers.xml    offers the gate calls indexable RIGHT NOW (fresh observation, not
-- *                               expired, reviewed where review is needed); lastmod = the latest
-- *                               observation or edit, never "now"
-- *   GET /sitemaps/products.xml  product pages with at least one fresh offer
-- *   GET /sitemaps/pages.xml     the front page and the lists: hot, new, ending soon, picks, every category and shop,
++ *   GET /sitemap.xml            sitemap index over the children below, each at most SITEMAP_CHILD_MAX URLs
++ *   GET /sitemap-pages.xml      the front page and the lists: hot, new, ending soon, picks, every category and shop,
   *                               crowdfunding (an empty list is noindex and left out)
++ *   GET /sitemap-<n>.xml        the catalog, split in SITEMAP_CHILD_MAX-URL chunks in one stable order: offers the gate
++ *                               calls indexable RIGHT NOW (fresh observation, not expired, reviewed where review is
++ *                               needed; lastmod = the latest observation or edit, never "now"), then product pages with
++ *                               at least one fresh offer. A chunk number past the last chunk is 404.
   *
   * Built from the database, never from the viewer (the offer and product lists at most once per
   * config.sitemapTtlMs, shared by concurrent readers). Stale, expired, disabled, merged and unreviewed
@@@ -28,6 +28,18 @@ const { SITE_SUMMARY } = require('../render/layout');
  // Offers or products whose page view a list build reads at once (the pool has a handful of connections for everyone).
  const VIEWS_PARALLEL = 8;
  
++// URLs in one sitemap file. The catalog keeps growing, so the index names one child per chunk instead of serving one
++// unbounded <urlset> that would eventually pass a crawler's limit.
++const SITEMAP_CHILD_MAX = 10000;
++
++/** entries in their given order, SITEMAP_CHILD_MAX at a time; an empty catalog still has one (empty) chunk. */
++function chunks(entries) {
++    if (!entries.length) return [[]];
++    const out = [];
++    for (let i = 0; i < entries.length; i += SITEMAP_CHILD_MAX) out.push(entries.slice(i, i + SITEMAP_CHILD_MAX));
++    return out;
++}
++
  /** fn over items, `limit` at a time; results in the items' order. */
  async function mapLimit(items, limit, fn) {
      const out = new Array(items.length);
@@@ -88,6 +100,22 @@ function createDiscoveryRoutes({ config, store, publication, listings }) {
          return lists.map((l) => ({ loc: abs(l.path), lastmod: l.lastmod || null, decision: decide(l.path, l.empty) }));
      }
  
++    /** The catalog the chunks split: every indexable offer, then every indexable product — the order the site has used. */
++    async function catalogEntries() {
++        const indexable = (list) => list.filter((e) => e.decision.indexable);
++        return [...indexable(await offerEntries()), ...indexable(await productEntries())];
++    }
++
++    /** The newest lastmod among entries, as the index's own date for that child; none is invented when there is none. */
++    const newestLastmod = (entries) => {
++        let newest = 0;
++        for (const e of entries) {
++            const t = e.lastmod ? new Date(e.lastmod).getTime() : 0;
++            if (t > newest) newest = t;
++        }
++        return newest ? { lastmod: new Date(newest).toISOString() } : {};
++    };
++
      router.get('/robots.txt', (_req, res) => {
          const body = [
              '# openvibe.deals automated-consumer policy: search engines and AI crawlers are welcome to read',
@@@ -115,7 +143,7 @@ function createDiscoveryRoutes({ config, store, publication, listings }) {
                  { title: 'Feeds', links: [{ title: 'RSS', url: abs('/feed.xml') }, { title: 'Atom', url: abs('/atom.xml') }, { title: 'JSON Feed', url: abs('/feed.json') }] },
                  { title: 'Data', links: [
                      { title: 'Deal JSON', url: abs('/new'), note: 'append .json to any deal URL (/d/<slug>.json)' },
--                    { title: 'Product JSON', url: abs('/sitemaps/products.xml'), note: 'append .json to any product URL (/p/<slug>.json)' },
++                    { title: 'Product JSON', url: abs('/sitemap.xml'), note: 'product pages are in the sitemap; append .json to any product URL (/p/<slug>.json)' },
                  ] },
              ],
          }));
@@@ -143,17 +171,23 @@ function createDiscoveryRoutes({ config, store, publication, listings }) {
      });
  
      router.get('/sitemap.xml', async (_req, res) => {
--        const o = (await offerEntries()).filter((e) => e.decision.indexable).map((e) => e.lastmod).filter(Boolean);
--        const p = (await productEntries()).filter((e) => e.decision.indexable).map((e) => e.lastmod).filter(Boolean);
--        const g = (await pageEntries()).filter((e) => e.decision.indexable).map((e) => e.lastmod).filter(Boolean);
--        const iso = (list) => (list.length ? { lastmod: new Date(Math.max(...list)).toISOString() } : {});
--        xml(res, seo.sitemapIndex([{ loc: abs('/sitemaps/pages.xml'), ...iso(g) }, { loc: abs('/sitemaps/offers.xml'), ...iso(o) }, { loc: abs('/sitemaps/products.xml'), ...iso(p) }]));
++        const pages = (await pageEntries()).filter((e) => e.decision.indexable);
++        const catalog = chunks(await catalogEntries());
++        const children = [
++            { loc: abs('/sitemap-pages.xml'), ...newestLastmod(pages) },
++            ...catalog.map((part, i) => ({ loc: abs(`/sitemap-${i}.xml`), ...newestLastmod(part) })),
++        ];
++        xml(res, sharedSeo.sitemapIndexXml(children));
      });
  
--    router.get('/sitemaps/pages.xml', async (_req, res) => xml(res, seo.sitemap(await pageEntries()).files[0]));
++    router.get('/sitemap-pages.xml', async (_req, res) => xml(res, seo.sitemap(await pageEntries()).files[0]));
  
--    router.get('/sitemaps/offers.xml', async (_req, res) => xml(res, seo.sitemap(await offerEntries()).files[0]));
--    router.get('/sitemaps/products.xml', async (_req, res) => xml(res, seo.sitemap(await productEntries()).files[0]));
++    // /sitemap-<n>.xml: the nth chunk of the catalog. Past the last chunk the number does not exist, so 404.
++    router.get(/^\/sitemap-(0|[1-9]\d{0,5})\.xml$/, async (req, res, next) => {
++        const part = chunks(await catalogEntries())[Number(req.params[0])];
++        if (!part) return next();
++        xml(res, seo.sitemap(part).files[0]);
++    });
  
      return router;
  }
ddiff --git a/test/freshness.test.js b/test/freshness.test.js
iindex 13f7ac2..34145b3 100644
---- a/test/freshness.test.js
++++ b/test/freshness.test.js
@@@ -46,7 +46,7 @@ const { sourceItem } = require('./helpers/mocks');
          assert.strictEqual(json.freshness.state, 'stale');
          assert.deepStrictEqual(json.indexability.reasons.map((r) => r.code), ['stale_price']);
          assert.strictEqual(json.latest_observation.price, '49.99', 'history is kept, with its time');
--        const sm = await t.get('/sitemaps/offers.xml');
++        const sm = await t.get('/sitemap-0.xml');
          assert.ok(!sm.text.includes(offer.slug));
          const list = await t.get('/');
          assert.match(list.text, /data-freshness="stale"/);
ddiff --git a/test/import.test.js b/test/import.test.js
iindex be68a5d..8272184 100644
---- a/test/import.test.js
++++ b/test/import.test.js
@@@ -256,7 +256,7 @@ const { signDelivery, signDeliveryHeaders } = require('openvibe-sdk/events');
          await t.ctx.importer.pull();
          assert.deepStrictEqual([(await byUrl(link)).category, (await byUrl(link)).staff_pick], ['Electronics', false]);
          assert.strictEqual((await t.get('/c/tvs')).status, 404, 'an empty category is gone');
--        const sm = await t.get('/sitemaps/pages.xml');
++        const sm = await t.get('/sitemap-pages.xml');
          assert.ok(sm.text.includes('<loc>https://openvibe.deals/c/electronics</loc>') && sm.text.includes('<loc>https://openvibe.deals/at/amazon</loc>'));
      });
  
ddiff --git a/test/pages.test.js b/test/pages.test.js
iindex df82acc..4e6c6ca 100644
---- a/test/pages.test.js
++++ b/test/pages.test.js
@@@ -138,7 +138,7 @@ const { boot, check, done, robotsOf, jsonLd, HOUR } = require('./helpers/boot');
          assert.strictEqual(e.length, 1);
          assert.strictEqual(e[0].payload.reason, 'submitter');
          assert.ok(!(await t.get('/')).text.includes(deal.slug));
--        assert.ok(!(await t.get('/sitemaps/offers.xml')).text.includes(deal.slug));
++        assert.ok(!(await t.get('/sitemap-0.xml')).text.includes(deal.slug));
      });
  
      await check('a stated end time expires the deal on the worker tick; an unknown end never expires by itself', async () => {
@@@ -203,10 +203,17 @@ const { boot, check, done, robotsOf, jsonLd, HOUR } = require('./helpers/boot');
          const llms = await t.get('/llms.txt');
          assert.match(llms.text, /OpenVibe.Deals/);
          assert.match(llms.text, /never 0/);
++        // Every URL the file lists must answer when it is fetched as a link: a language model that
++        // follows one and gets a 404 or a timeout has learned nothing from it.
++        for (const url of [...llms.text.matchAll(/\((https?:\/\/[^)]+)\)/g)].map((m) => m[1])) {
++            if (!url.startsWith('https://openvibe.deals/')) continue;
++            const u = new URL(url);
++            assert.ok((await t.get(u.pathname + u.search)).status < 400, url + ' does not answer');
++        }
          const idx = await t.get('/sitemap.xml');
--        assert.match(idx.text, /sitemaps\/offers.xml/);
--        assert.match(idx.text, /sitemaps\/products.xml/);
--        const offers = await t.get('/sitemaps/offers.xml');
++        assert.match(idx.text, /<loc>https:\/\/openvibe\.deals\/sitemap-pages\.xml<\/loc>/);
++        assert.match(idx.text, /<loc>https:\/\/openvibe\.deals\/sitemap-0\.xml<\/loc>/);
++        const offers = await t.get('/sitemap-0.xml');
          const fresh = (await t.ctx.store.db.prepare("SELECT slug FROM deal_offers WHERE title = 'Hardcover novel'").get()).slug;
          assert.ok(offers.text.includes(fresh));
          assertSitemapLastmods(offers.text);
@@@ -214,13 +221,13 @@ const { boot, check, done, robotsOf, jsonLd, HOUR } = require('./helpers/boot');
          const observation = await t.ctx.store.db.prepare('SELECT MAX(observed_at) AS time FROM deal_price_observations WHERE offer_id = ?').get(offerRow.id);
          const expected = new Date(Math.max(offerRow.updated_at, observation.time || 0)).toISOString().slice(0, 10);
          assert.ok(offers.text.includes(`<lastmod>${expected}</lastmod>`));
--        const products = (await t.get('/sitemaps/products.xml')).text;
--        assertSitemapLastmods(products);
--        // An index entry carries its map's newest lastmod; an empty map has no honest date and none is invented.
--        const offersEntry = /<sitemap><loc>[^<]*\/sitemaps\/offers\.xml<\/loc>(<lastmod>[^<]+<\/lastmod>)?<\/sitemap>/.exec(idx.text);
--        assert.ok(offersEntry && offersEntry[1], idx.text);
--        if (!products.includes('<url>')) assert.ok(/<loc>[^<]*\/sitemaps\/products\.xml<\/loc><\/sitemap>/.test(idx.text), 'no lastmod for an empty map');
--        else assertSitemapLastmods(idx.text, 'sitemap');
++        // An index entry carries its child's newest lastmod; an empty child has no honest date and none is invented.
++        const catalogEntry = /<sitemap><loc>[^<]*\/sitemap-0\.xml<\/loc>(<lastmod>[^<]+<\/lastmod>)?<\/sitemap>/.exec(idx.text);
++        assert.ok(catalogEntry, idx.text);
++        if (!offers.text.includes('<url>')) assert.ok(!catalogEntry[1], 'no lastmod for an empty map');
++        else assert.ok(catalogEntry[1], 'the child\'s newest lastmod');
++        assertSitemapLastmods(idx.text, 'sitemap');
++        assert.strictEqual((await t.get('/sitemap-1.xml')).status, 404, 'a chunk number that does not exist');
          assert.ok(!offers.text.includes(timed.slug), 'expired');
          assert.ok(!offers.text.includes(deal.slug), 'expired');
      });
@@@ -235,7 +242,7 @@ const { boot, check, done, robotsOf, jsonLd, HOUR } = require('./helpers/boot');
          const urls = [...full.text.matchAll(/^URL: (\S+)$/gm)].map((m) => m[1]);
          assert.ok(urls.length >= 1, 'lists an indexable deal');
          // The same gate as the sitemap: every listed URL is one the sitemap also lists as indexable.
--        const locs = new Set([...(await t.get('/sitemaps/offers.xml')).text.matchAll(/<loc>([^<]+)<\/loc>/g)].map((m) => m[1]));
++        const locs = new Set([...(await t.get('/sitemap-0.xml')).text.matchAll(/<loc>([^<]+)<\/loc>/g)].map((m) => m[1]));
          for (const u of urls) assert.ok(locs.has(u), `only indexable deals: ${u}`);
          assert.ok(!urls.some((u) => u.endsWith(`/d/${timed.slug}`)), 'an expired deal stays out');
          assert.ok(!urls.some((u) => u.endsWith(`/d/${deal.slug}`)), 'an expired deal stays out');
@@@ -260,8 +267,8 @@ const { boot, check, done, robotsOf, jsonLd, HOUR } = require('./helpers/boot');
          const old = await t.get('/?page=2');
          assert.deepStrictEqual([old.status, old.headers.get('location')], [301, '/hot?page=2']);
          for (const p of ['/hot', '/ending', '/picks', '/c']) assert.strictEqual((await t.get(p)).status, 200, p);
--        assert.ok((await t.get('/sitemap.xml')).text.includes('<loc>https://openvibe.deals/sitemaps/pages.xml</loc>'));
--        assert.ok((await t.get('/sitemaps/pages.xml')).text.includes('<loc>https://openvibe.deals/</loc>'));
++        assert.ok((await t.get('/sitemap.xml')).text.includes('<loc>https://openvibe.deals/sitemap-pages.xml</loc>'));
++        assert.ok((await t.get('/sitemap-pages.xml')).text.includes('<loc>https://openvibe.deals/</loc>'));
      });
  
      await check('operations: health, readiness, release.json, metrics (loopback), legal pages', async () => {
ddiff --git a/test/sitemap-build.test.js b/test/sitemap-build.test.js
iindex ed10b1e..f648ec0 100644
---- a/test/sitemap-build.test.js
++++ b/test/sitemap-build.test.js
@@@ -4,18 +4,26 @@
   * it runs, at most 8 page views at once, reused for DEALS_SITEMAP_TTL_MS (production: 10 minutes; 0 rebuilds). On
   * 2026-10-23 each crawler fetch of /sitemap.xml read every offer's and product's view at once: 5,000-24,000 queries
   * queued on the pool for a minute while every page waited behind them.
++ *
++ * The catalog is split in SITEMAP_CHILD_MAX-URL chunks: /sitemap.xml names the pages child and one child per chunk,
++ * and the chunks hold exactly the URLs (in the same order) the old single catalog sitemap did.
   */
  const assert = require('assert');
  const http = require('http');
  const express = require('express');
++const seo = require('openvibe-publishing/seo');
  const { createDiscoveryRoutes } = require('../server/http/discovery');
  
  const seen = { now: 0, max: 0, views: 0 };
  const tick = () => new Promise((r) => setImmediate(r));
--const offers = Array.from({ length: 20 }, (_x, i) => ({ id: `dof_${i}`, slug: `deal-${i}`, updated_at: 1792700000000 + i }));
  const indexable = { indexable: true, listable: true, robots: 'index, follow', reasons: [], codes: [] };
  
--function routes(ttlMs, clock) {
++function makeOffers(count) {
++    return Array.from({ length: count }, (_x, i) => ({ id: `dof_${i}`, slug: `deal-${i}`, updated_at: 1792700000000 + i }));
++}
++
++function routes(ttlMs, clock, { offerCount = 20, products = [] } = {}) {
++    const offers = makeOffers(offerCount);
      const publication = {
          async offerView(o) {
              seen.now++; seen.views++; seen.max = Math.max(seen.max, seen.now);
@@@ -28,30 +36,33 @@ function routes(ttlMs, clock) {
          productPath: (p) => `/p/${p.slug}`,
      };
      const listings = {
--        indexable: async () => offers, products: async () => [], newest: async () => [], countActive: async () => offers.length,
++        indexable: async () => offers, products: async () => products, newest: async () => [], countActive: async () => offers.length,
          countEnding: async () => 0, countPicks: async () => 0, categories: async () => [], sellers: async () => [],
      };
      return createDiscoveryRoutes({ config: { baseUrl: 'https://deals.test', sitemapTtlMs: ttlMs }, store: { now: () => clock.t }, publication, listings });
  }
  
  async function serve(router) {
--    const app = express().use(router);
++    const app = express().use(router).use((_req, res) => res.status(404).type('text/plain').send('Not found'));
      const server = await new Promise((r) => { const s = http.createServer(app).listen(0, '127.0.0.1', () => r(s)); });
      const base = `http://127.0.0.1:${server.address().port}`;
      return { get: async (p) => { const res = await fetch(base + p); return { status: res.status, text: await res.text() }; }, close: () => server.close() };
  }
  
++const locs = (text) => [...text.matchAll(/<loc>([^<]+)<\/loc>/g)].map((m) => m[1]);
++const maps = (text) => [...text.matchAll(/<sitemap><loc>([^<]+)<\/loc>/g)].map((m) => m[1]);
++
  (async () => {
      const clock = { t: Date.parse('2026-10-23T21:26:00Z') };
  
      // No reuse (tests, development): concurrent requests share one build; the next request rebuilds.
      let s = await serve(routes(0, clock));
--    const [a, b, c] = await Promise.all([s.get('/sitemap.xml'), s.get('/sitemaps/offers.xml'), s.get('/llms-full.txt')]);
++    const [a, b, c] = await Promise.all([s.get('/sitemap.xml'), s.get('/sitemap-0.xml'), s.get('/llms-full.txt')]);
      assert.deepStrictEqual([a.status, b.status, c.status], [200, 200, 200]);
      assert.ok(b.text.includes('https://deals.test/d/deal-19'), 'every offer listed');
      assert.strictEqual(seen.views, 20, `three readers, one build (${seen.views} views)`);
      assert.ok(seen.max <= 8 && seen.max >= 2, `at most 8 views at once (saw ${seen.max})`);
--    await s.get('/sitemaps/offers.xml');
++    await s.get('/sitemap-0.xml');
      assert.strictEqual(seen.views, 40, 'no TTL: rebuilt');
      s.close();
  
@@@ -60,11 +71,52 @@ async function serve(router) {
      s = await serve(routes(600000, clock));
      await s.get('/sitemap.xml');
      clock.t += 9 * 60 * 1000;
--    await s.get('/sitemaps/offers.xml');
++    await s.get('/sitemap-0.xml');
      assert.strictEqual(seen.views, 20, 'reused inside ten minutes');
      clock.t += 2 * 60 * 1000;
--    await s.get('/sitemaps/offers.xml');
++    await s.get('/sitemap-0.xml');
      assert.strictEqual(seen.views, 40, 'rebuilt after');
      s.close();
++
++    // A catalog past 10,000 URLs: the index names every child and the children are the old single sitemap, split.
++    const offerCount = 20005;
++    const offers = makeOffers(offerCount);
++    const products = [{ id: 'dpr_1', slug: 'novel', updated_at: 1792800000000 }];
++    s = await serve(routes(0, clock, { offerCount, products }));
++    const index = await s.get('/sitemap.xml');
++    assert.strictEqual(index.status, 200);
++    assert.match(index.text, /<sitemapindex xmlns="http:\/\/www\.sitemaps\.org\/schemas\/sitemap\/0\.9">/);
++    assert.deepStrictEqual(maps(index.text), [
++        'https://deals.test/sitemap-pages.xml',
++        'https://deals.test/sitemap-0.xml',
++        'https://deals.test/sitemap-1.xml',
++        'https://deals.test/sitemap-2.xml',
++    ], 'the pages child, then one child per catalog chunk');
++
++    const sizes = [];
++    const catalog = [];
++    for (const url of maps(index.text)) {
++        const child = await s.get(new URL(url).pathname);
++        assert.strictEqual(child.status, 200, url);
++        assert.match(child.text, /<urlset xmlns="http:\/\/www\.sitemaps\.org\/schemas\/sitemap\/0\.9"/, url);
++        const found = locs(child.text);
++        assert.ok(found.length <= 10000, `${url} holds ${found.length} URLs`);
++        if (/\/sitemap-\d+\.xml$/.test(url)) { sizes.push(found.length); catalog.push(...found); }
++        else assert.ok(found.includes('https://deals.test/') && found.includes('https://deals.test/hot'), 'the pages child');
++    }
++    assert.deepStrictEqual(sizes, [10000, 10000, 6], 'offers, then products: at most 10,000 a child');
++
++    const oldSingle = seo.sitemap([
++        ...offers.map((o) => ({ loc: `https://deals.test/d/${o.slug}`, lastmod: o.updated_at, decision: indexable })),
++        ...products.map((p) => ({ loc: `https://deals.test/p/${p.slug}`, lastmod: p.updated_at, decision: indexable })),
++    ]).files[0];
++    assert.deepStrictEqual(catalog, locs(oldSingle), 'the union of the children is the old catalog sitemap, same order');
++    assert.strictEqual(new Set(catalog).size, catalog.length, 'no URL twice');
++
++    // A chunk number past the last one does not exist.
++    for (const path of ['/sitemap-3.xml', '/sitemap-99.xml', '/sitemap-00.xml', '/sitemap-01.xml']) {
++        assert.strictEqual((await s.get(path)).status, 404, path);
++    }
++    s.close();
      console.log('sitemap-build: all checks passed');
  })().catch((e) => { console.error(e); process.exit(1); });