openvibers/openvibe.publishing Commits

Publishing 1.1.0: the shared ingest and publication chassis (plan T9 J1, additive). openvibe-publishing/ingest: the Sources client, a named change cursor (<prefix>_ingest_cursor), pullChanges (one transaction per page, one savepoint per item, apply → applied|hold|removed, a bad item isolated), the signed event consumer with an exactly-once inbox, and the normalize/hosts/freshness helpers the five product copies share (119 fixture cases citing their source). openvibe-publishing/publication: createPublication (gate → document → sequencer.stamp → events → outbox on the caller's transaction, tombstones, the IndexNow ping). No existing export, table or document changed. Both suites green (verified).

OP OpenVibers committed 29 Sept 2026 · bd7e6c2fa8 · 12 files, +2928 −34


Changes

ddiff --git a/CHANGELOG.md b/CHANGELOG.md
iindex afb8c7a..e5af20e 100644
---- a/CHANGELOG.md
++++ b/CHANGELOG.md
@@@ -4,6 +4,79 @@ All notable changes to `openvibe-publishing`. Versions follow [semver](https://s
breaking change to any exported function, table layout, reason code or document shape is a new
major (a minor while 0.x). A release is the git tag `vX.Y.Z`; consumers pin the tag's tarball.
++## 1.1.0 — 2026-09-29
++
++**Additive: two new entry points, `openvibe-publishing/ingest` and `openvibe-publishing/publication`.**
++No existing export, table layout, reason code or document shape changes; a product on v1.0.0 upgrades by
++pinning v1.1.0, and the five content products (News, Reviews, Deals, Coupons, Trade) each delete their
++own copy of this code when they convert. `require('openvibe-publishing')` exposes `.ingest` and
++`.publication` lazily, and root `schema({ … })` gains the `ingest` key.
++
++### Added — `openvibe-publishing/ingest`
++
++The chassis the five content products each re-implemented (~4,150 lines): primitives and thin factories,
++not one opinionated pipeline, because their domains diverge.
++
++- `createSourcesClient({ config, fetchImpl, now, timeoutMs })` — the OpenVibe.Sources client
++ (`GET /api/v1/items` in change order, `item`, `sources`, `source`) with a client-credentials token for
++ audience `openvibe.sources` and scopes `sources.item.read`/`sources.source.read`. Failures throw a
++ `SourcesError` with a stable code (`sources.not_configured`, `sources.token_unavailable`,
++ `sources.unavailable`, `sources.http_<status>`, `sources.bad_response`); nothing invents an item.
++- `createChangeCursor(db, { prefix, now })` — a named change cursor in `<prefix>_ingest_cursor`
++ (`name`, `cursor`, `updated_at`), `schema()` gives its DDL for the product's migration,
++ `ensureSchema()` runs it for tests; `get(name)` reads and `set(t, name, value)` advances the cursor on
++ the caller's handle, monotonically (`GREATEST`).
++- `pullChanges({ db, cursor, source, apply, name, maxPages, pageSize, onItem })` — one transaction per
++ page (the page's writes and the cursor advance commit together), one savepoint per item (an item whose
++ `apply()` throws is rolled back alone and counted `failed`, so a bad item never stalls its page or the
++ cursor), and `apply(item, t)` returns `'applied' | 'hold' | 'removed'` (a `{ outcome }` object is
++ accepted; anything else is isolated as failed). `hold` is not an error: it is counted and the cursor
++ moves past it (the product records the hold itself). Returns
++ `{ pages, applied, hold, removed, failed, after }`.
++- `createEventConsumer({ db, secrets, consumer, now, table })` — the shared half of `POST /internal/events`:
++ the `X-OpenVibe-Signature` v2 HMAC (±300 s, v1-only or stale refused), and exactly-once via the SDK's
++ pg inbox claimed in the same transaction as the handler, so a redelivery changes nothing and a failed
++ handler rolls back its receipt. `apply(raw, headers, handler) → { status, code?, event_id?, duplicate?, outcome? }`.
++- `normalize` — the shared helpers, byte-for-byte from the five copies: News's `fold/titleKey/urlKey/
++ hostOf/publisherDomain/shingles/jaccard/stem/terms/entities/licensedSummary/independentSources`, Deals's
++ `normalizeUrl/hostPathKey/domainOf/slugify/parseAmount/parseCurrency/parseInstant/parseEnum/parseDecimal/
++ parseTime/text/longText/slug/tokens/str/httpUrl/json` and its alias normalizers
++ (`alias/tryAlias/name/url/code/gtin/sourceKey/external`), Reviews's key normalizers and Trade's
++ time/decimal/string parsers. `test/ingest.test.js` proves all 119 captured cases from the five current
++ product files (`test/fixtures/normalize.json`, each case citing `file:line`).
++- `hosts` — the generic PSL and registrable-host helpers moved out of Coupons
++ (`normalizeHost/publicSuffix/registrableDomain/registrable/hostOfUrl/normalizePathPrefix/checkRule/
++ bestRule`); and `freshness` (`verdict/view`), the generic staleness rule (fresh inside the window,
++ stale after, never invented), with the threshold a parameter — no policy constant and no date ever
++ defaults to now.
++
++### Added — `openvibe-publishing/publication`
++
++`createPublication({ owner, sequencer, outbox, baseUrl, indexnow, now, decide, document, page })` — the
++glue the five products each re-implemented: gate → document → `sequencer.stamp(t, doc)` → events →
++`outbox.enqueue(t, …)`, always on the **caller's** transaction handle, so an index document, its event
++and the product's own change commit or roll back together (ADR-004).
++
++- `index(t, { document, page, traceparent })` — stamps and enqueues one document; an unchanged document
++ (same revision) is a no-op; a resource that was never indexed sends no tombstone.
++- `tombstone(t, { type, id, page, traceparent })` — the tombstone for unpublish/merge.
++- `publication(t, { type, action, id, revision, document, decision, actor, extra, traceparent })` — the
++ product's own `<owner>.<type>.<action>` event.
++- `sync(t, entity, { traceparent, forSearch })` — the whole chain for one entity for a product that
++ supplies `decide`/`document`/`page`.
++- It emits exactly `search.index-document@1` carried by `<owner>.index_document.upserted|deleted`, and
++ the publication events (`index-hooks.publicationEvent`), both validated by openvibe-contracts; and it
++ owns the IndexNow ping (config stays per product): an indexable page that appeared/changed or a page
++ Search already had that went away is announced, a draft/private/noindex page never is, and
++ `pingSoon` is a no-op without a key and never throws.
++
++### Tests
++
++`test/ingest.test.js` and `test/publication.test.js` (new) plus the extended `test/schema.test.js`
++(the `<prefix>_ingest_cursor` DDL, idempotent, served by its primary key).
++`npm test` (Node 22): 18 files, 135 cases; 17 files and 1 skipped (postgres) when `OV_TEST_PG_URL` is
++not set.
++
## 1.0.0 — 2026-09-28
**Breaking: the stores run on PostgreSQL through `openvibe-sdk/db`, and every store API is async**
ddiff --git a/README.md b/README.md
iindex 0816384..972cad7 100644
---- a/README.md
++++ b/README.md
@@@ -3,14 +3,16 @@
> Shared publishing packages for the OpenVibe publication products: Wiki, Blog, News, Reviews,
> Deals, Coupons and Trade.
--**Status:** alpha, **1.0.0**: the stores run on PostgreSQL through the `openvibe-sdk/db` async data
++**Status:** alpha, **1.1.0**: the stores run on PostgreSQL through the `openvibe-sdk/db` async data
layer (ADR-035: PostgreSQL 18 behind PgBouncer in production, PGlite in tests). Every store method is
async, takes the product's `openvibe-sdk/db` handle, and accepts the caller's transaction handle; each
--store gives its DDL as `schema(prefix)` for the product's migrations. The exit proof
--(`examples/two-products`) runs in `npm test`. Releases v0.1.0 to v1.0.0 are tagged (see
--[CHANGELOG.md](CHANGELOG.md)). The seven products (Wiki and Blog, public at openvibe.wiki and
--openvibe.blog; News, Reviews, Deals, Coupons and Trade, deployed loopback-only, not launched) still pin
--v0.4.x on SQLite; each moves to 1.x in its own PostgreSQL migration.
++store gives its DDL as `schema(prefix)` for the product's migrations. v1.1.0 adds the shared ingest
++chassis (`openvibe-publishing/ingest`) and publication glue (`openvibe-publishing/publication`) —
++additive, no existing export changed. The exit proof (`examples/two-products`) runs in `npm test`.
++Releases v0.1.0 to v1.0.0 are tagged (see [CHANGELOG.md](CHANGELOG.md)). All seven products (Wiki and
++Blog, public at openvibe.wiki and openvibe.blog; News, Reviews, Deals, Coupons and Trade, deployed
++loopback-only, not launched) run v1.0.0 on PostgreSQL today; the five content products move to v1.1.0 as
++each adopts the chassis, and no product is released until its conversion lands.
**Package:** `openvibe-publishing` (CommonJS, Node ≥ 20, production runs Node 22).
**License:** MIT, like OpenVibe.Shared.
@@@ -61,16 +63,18 @@ Every module is its own entry point and can be used alone.
| `openvibe-publishing/index-hooks` | `search.index-document@1` documents and tombstones, the `<owner>.index_document.upserted\|deleted` events OpenVibe.Search consumes, a monotonic index-revision sequencer, and the product's own `<product>.<type>.published\|updated\|unpublished\|deleted` events | `<prefix>_index_revisions` |
| `openvibe-publishing/ai` | Ask OpenVibe.AI for a draft: run a registered workflow with the product's service token (polls a slow run), read its citations, get the `{ id, version, runId, model }` an AI authorship record needs; coded `AiRunError`s, never a partial draft | — |
| `openvibe-publishing/ssr` | auto-escaping `html` tagged templates with `raw()`, a safe Markdown subset, plain-text extraction, word count, server pagination, breadcrumbs, diff markup, honest `<time>` | — |
++| `openvibe-publishing/ingest` | The shared ingest chassis: the OpenVibe.Sources client, a named change cursor and the `pullChanges` loop (per-item savepoint isolation, `{applied\|hold\|removed}`), the signed event consumer with an exactly-once inbox, the shared normalisers, and the generic PSL/registrable-host and freshness helpers | `<prefix>_ingest_cursor` |
++| `openvibe-publishing/publication` | The shared publication glue: gate → document → `sequencer.stamp` → events → `outbox.enqueue` on the caller's transaction handle, tombstones on unpublish/merge, and the IndexNow ping; it emits exactly `search.index-document@1` and the product's events | — |
--`require('openvibe-publishing')` exposes all of them lazily (`.revisions`, `.seo`, `.indexHooks`, …),
--plus `schema({ … })`, the DDL of several stores at once (below).
++`require('openvibe-publishing')` exposes all of them lazily (`.revisions`, `.seo`, `.indexHooks`,
++`.ingest`, `.publication`, …), plus `schema({ … })`, the DDL of several stores at once (below).
## Install
Pin the release tarball, like every OpenVibe package (never a `file:` link or a vendored copy):
```json
--"openvibe-publishing": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.0.0",
++"openvibe-publishing": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.1.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.15.0",
"pg": "^8.23.0"
```
@@@ -121,6 +125,7 @@ const ddl = publishing.schema({ revisions: 'wiki_page', citations: 'wiki', seo:
| taxonomy | `schema(prefix)` | `<prefix>_terms`, `<prefix>_term_links` |
| seo | `redirectsSchema(prefix)` | `<prefix>_redirects` |
| index-hooks | `sequencerSchema(prefix)` | `<prefix>_index_revisions` |
++| ingest | `schema(prefix)` | `<prefix>_ingest_cursor` |
Every store also has `store.schema()` (the same text) and `await store.ensureSchema()`, which runs it
on the handle: for tests and PGlite, where the handle may create tables. The text is idempotent
@@@ -257,7 +262,72 @@ await db.tx(async (t) => {
URL, publication state, indexability; public only for public, listable content). Neither sets
`event_id`: the OpenVibe.Events outbox assigns it. Enqueue both in the same transaction as the
product's state change. Tests validate every document and envelope with openvibe-contracts
--v0.49.0 and mirror Search's webhook checks.
++v0.76.0 and mirror Search's webhook checks.
++
++## Ingesting from Sources (the ingest chassis)
++
++The five content products (News, Reviews, Deals, Coupons, Trade) each pull OpenVibe.Sources items, track a
++change cursor, consume signed OpenVibe.Events deliveries and normalise item fields. They share that code
++here; each keeps its own domain tables, gate facts, facets and item mapping.
++
++```js
++const { createSourcesClient, createChangeCursor, pullChanges, createEventConsumer }
++ = require('openvibe-publishing/ingest');
++
++const sources = createSourcesClient({ config }); // { sources, oauth }, from the product
++const cursor = createChangeCursor(db, { prefix: 'deals' }); // <prefix>_ingest_cursor in a migration
++const summary = await pullChanges({
++ db, cursor, source: sources, name: 'sources.deals.after',
++ apply: async (item, t) => { // one savepoint per item
++ const row = await offers.upsertFromItem(t, item);
++ return row.held ? 'hold' : 'applied'; // 'removed' for a deleted item
++ },
++});
++// → { pages, applied, hold, removed, failed, after }
++```
++
++`pullChanges` commits one transaction per page — the page's writes **and** the cursor advance — and runs
++`apply(item, t)` in a savepoint, so one unreadable item is rolled back alone (`failed`) and never stalls
++the page or the cursor. `apply` returns `'applied' | 'hold' | 'removed'`; `hold` is counted and the cursor
++moves past it (the product records the hold itself).
++
++`createEventConsumer({ db, secrets, consumer }).apply(raw, headers, handler)` verifies the
++`X-OpenVibe-Signature` v2 HMAC (±300 s; a v1-only or stale delivery is refused) and runs the handler
++exactly once per `event_id` through the SDK's pg inbox, so a redelivery changes nothing and a failed
++handler rolls back its receipt.
++
++`ingest.normalize` holds the normalisers the products share (folding, title/URL keys, hosts, amounts,
++currencies, instants, enum/text parsing, the Reviews key/alias normalisers), `ingest.hosts` the PSL and
++registrable-host helpers, and `ingest.freshness` the generic staleness rule. Nothing defaults a date to
++"now", and no policy threshold is a constant: a product passes its own.
++
++## The publication glue (createPublication)
++
++`createPublication` is the half the five content products each re-implement: it takes the product's gate
++(`decide`), document builder (`document`), canonical path (`page`), the index sequencer and the outbox, and
++runs gate → document → `sequencer.stamp(t, doc)` → event → `outbox.enqueue(t, …)` on the **caller's**
++transaction handle, so the index document, its event and the product's own change commit or roll back
++together.
++
++```js
++const { createPublication } = require('openvibe-publishing/publication');
++const publication = createPublication({
++ owner: 'deals', sequencer, outbox, baseUrl: config.baseUrl, indexnow,
++ decide: (offer, now) => publication.decideOffer(offer), // the product's gate facts
++ document: (offer, decision) => offerDocument(offer), // hooks.buildIndexDocument/tombstone
++ page: (offer) => `/d/${offer.slug}`,
++});
++await db.tx(async (t) => {
++ await t.exec(sql`UPDATE deal_offers SET … WHERE id = ${id}`);
++ await publication.sync(t, offer); // stamps + enqueues in the SAME tx
++});
++```
++
++It emits exactly `search.index-document@1` (carried by `<owner>.index_document.upserted|deleted`) and the
++product's own `<owner>.<type>.<action>` events; an unchanged document is a no-op, a resource that was
++never indexed gets no tombstone, and unpublish/merge call `tombstone(t, …)`. It owns the IndexNow ping
++(config stays per product): an indexable page that appeared or changed, or a page Search already had that
++went away, is announced; a draft, private or noindex page never is.
## Exit proof: two products
@@@ -276,10 +346,12 @@ survives a worker restart.
## Capabilities
--The package implements no capability and holds no grant. Three modules call a service with the
++The package implements no capability and holds no grant. Four modules call a service with the
**product's** token client, so the product needs the grant: `discussion` calls OpenVibe.Community
(`community.comment.write` to resolve a thread), `ai` calls OpenVibe.AI (`ai.run.create`,
--`ai.run.read`) and `media`'s `verify()` reads OpenVibe.Media objects (`media.object.read`).
++`ai.run.read`), `media`'s `verify()` reads OpenVibe.Media objects (`media.object.read`), and `ingest`'s
++Sources client reads OpenVibe.Sources items and registry records (`sources.item.read`,
++`sources.source.read`).
## Acceptance
@@@ -291,7 +363,12 @@ states (`media.test.js`); discussion references without content (`discussion.tes
redirects, structured data and feeds built only from given fields (`seo.test.js`); AI content held
until a person's review (`authorship.test.js`, `ai.test.js`); Search documents and envelopes valid
against openvibe-contracts, and `stamp(t, …)` committing with its event (`index-hooks.test.js`);
--escaping and the Markdown subset, including the ReDoS fix (`ssr.test.js`); every subpath export loading
++escaping and the Markdown subset, including the ReDoS fix (`ssr.test.js`); the ingest chassis — the
++normalisers and host helpers byte-for-byte against 119 cases captured from the five current product files,
++the Sources client against a stub server, cursor advancement with per-item savepoint isolation and hold,
++and the signed event consumer's window and exactly-once inbox (`ingest.test.js`); the publication glue —
++contracts-valid documents and events, `stamp` + `enqueue` in one transaction, tombstones and the IndexNow
++ping (`publication.test.js`); every subpath export loading
alone without a driver (`package.test.js`); the DDL: idempotent, 63-byte names, PostgreSQL types,
CHECKs, PL/pgSQL guards, applied by `db.migrate()`, and an index under every query the stores send
(`schema.test.js`); callers' transactions spanning store writes, rollbacks leaving nothing, savepoints
@@@ -342,5 +419,5 @@ eval "$(node_modules/openvibe-sdk/scripts/test-services.sh up)" && fnm exec --us
- PostgreSQL 18 (through PgBouncer in transaction mode), the product's own database (ADR-035).
- `openvibe-shared` ≥ 1.5.0 (`seo`), a peer dependency supplied by the consumer (tests use v1.25.0).
- Contracts it produces for: `common.entity-ref@1`, `media.media-ref@1`, `events.event-envelope@1`
-- and `search.index-document@1` (validated in tests against `openvibe-contracts` v0.49.0, a
++ and `search.index-document@1` (validated in tests against `openvibe-contracts` v0.76.0, a
devDependency: nothing in `lib/` needs it at runtime).
ddiff --git a/STATUS.json b/STATUS.json
iindex ab70052..ddb6cff 100644
---- a/STATUS.json
++++ b/STATUS.json
@@@ -5,7 +5,7 @@
"wave": "W15",
"kind": "packages",
"package": "openvibe-publishing",
-- "version": "1.0.0",
++ "version": "1.1.0",
"plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026)",
"runtime": null,
"port": null,
@@@ -22,26 +22,28 @@
"authorship",
"index-hooks",
"ai",
-- "ssr"
++ "ssr",
++ "ingest",
++ "publication"
],
"consumers": [
-- "OpenVibe.Wiki (v0.4.0 on SQLite until it moves to 1.x; public at https://openvibe.wiki)",
-- "OpenVibe.Blog (v0.4.0 on SQLite until it moves to 1.x; public at https://openvibe.blog)",
-- "OpenVibe.News (v0.4.0 on SQLite until it moves to 1.x; deployed loopback-only, not launched)",
-- "OpenVibe.Reviews (v0.4.0 on SQLite until it moves to 1.x; deployed loopback-only, not launched)",
-- "OpenVibe.Deals (v0.4.0 on SQLite until it moves to 1.x; deployed loopback-only, not launched)",
-- "OpenVibe.Coupons (v0.4.0 on SQLite until it moves to 1.x; deployed loopback-only, not launched)",
-- "OpenVibe.Trade (v0.4.0 on SQLite until it moves to 1.x; deployed loopback-only, not launched)"
++ "OpenVibe.Wiki (v1.0.0 on PostgreSQL; public at https://openvibe.wiki)",
++ "OpenVibe.Blog (v1.0.0 on PostgreSQL; public at https://openvibe.blog)",
++ "OpenVibe.News (v1.0.0 on PostgreSQL; deployed loopback-only, not launched)",
++ "OpenVibe.Reviews (v1.0.0 on PostgreSQL; deployed loopback-only, not launched)",
++ "OpenVibe.Deals (v1.0.0 on PostgreSQL; deployed loopback-only, not launched)",
++ "OpenVibe.Coupons (v1.0.0 on PostgreSQL; deployed loopback-only, not launched)",
++ "OpenVibe.Trade (v1.0.0 on PostgreSQL; deployed loopback-only, not launched)"
],
"plannedConsumers": [],
"exitProof": "examples/two-products + test/two-products.test.js",
-- "contracts": "openvibe-contracts v0.49.0 (devDependency): emits search.index-document@1 and events.event-envelope@1 (validated in tests)",
++ "contracts": "openvibe-contracts v0.76.0 (devDependency): emits search.index-document@1 and events.event-envelope@1 (validated in tests)",
"dependsOn": [
"openvibe-sdk >=0.15.0 (peer; tests pin v0.15.0)",
"openvibe-shared >=1.5.0 (optional peer; tests pin v1.25.0)",
"PostgreSQL 18 through PgBouncer (the product's database)"
],
-- "tests": "npm test (Node 22): 16 files, 121 cases; test/postgres.test.js runs on PostgreSQL 18 + PgBouncer when OV_TEST_PG_URL is set (CI starts the containers), else it is listed as skipped",
++ "tests": "npm test (Node 22): 18 files, 135 cases; test/postgres.test.js runs on PostgreSQL 18 + PgBouncer when OV_TEST_PG_URL is set (CI starts the containers), else it is listed as skipped",
"released": true,
"deployed": false,
"deployment": "none: a package released as git tags; products pin the tag and deploy it with their own releases",
@@@ -57,7 +59,9 @@
"authorship and AI disclosure, held until a person's review",
"OpenVibe.AI draft client",
"Search index documents, tombstones and publication events",
-- "auto-escaping SSR and a safe Markdown subset"
++ "auto-escaping SSR and a safe Markdown subset",
++ "the ingest chassis: the OpenVibe.Sources client, a change cursor + pull loop with per-item savepoint isolation and an {applied|hold|removed} apply contract, a signed event consumer with an exactly-once inbox, the normalisers and PSL/registrable-host and freshness helpers the five content products share",
++ "the publication glue: gate → document → sequencer.stamp → events → outbox.enqueue on the caller's transaction handle, tombstones on unpublish/merge, and the IndexNow ping"
],
-- "updated": "2026-09-28"
++ "updated": "2026-09-29"
}
ddiff --git a/index.js b/index.js
iindex 41a9445..032503c 100644
---- a/index.js
++++ b/index.js
@@@ -2,7 +2,7 @@
/**
* openvibe-publishing — every module is also its own entry point and can be used alone:
* require('openvibe-publishing/revisions'), …/schedule, …/taxonomy, …/citations, …/media,
-- * …/discussion, …/seo, …/authorship, …/index-hooks, …/ssr, …/ai
++ * …/discussion, …/seo, …/authorship, …/index-hooks, …/ssr, …/ai, …/ingest, …/publication
* This root export loads each one lazily, on first access.
*
* schema({ revisions: 'wiki_page', citations: 'wiki', … }) is the DDL of every store a product uses,
@@@ -20,6 +20,8 @@ const MODULES = {
indexHooks: './lib/index-hooks',
ssr: './lib/ssr',
ai: './lib/ai',
++ ingest: './lib/ingest',
++ publication: './lib/publication',
};
/** Which function gives each storing module's DDL. */
@@@ -33,6 +35,7 @@ const SCHEMAS = {
taxonomy: (p) => require('./lib/taxonomy').schema(p),
seo: (p) => require('./lib/seo').redirectsSchema(p),
indexHooks: (p) => require('./lib/index-hooks').sequencerSchema(p),
++ ingest: (p) => require('./lib/ingest').schema(p),
};
function schema(stores = {}) {
ddiff --git a/lib/ingest.js b/lib/ingest.js
nnew file mode 100644
iindex 0000000..0186eba
---- /dev/null
++++ b/lib/ingest.js
@@@ -0,0 +1,938 @@
++'use strict';
++/**
++ * openvibe-publishing/ingest — the shared ingest chassis: the primitives the five content products
++ * (News, Reviews, Deals, Coupons, Trade) each keep a private copy of today.
++ *
++ * const ingest = require('openvibe-publishing/ingest');
++ *
++ * const sources = ingest.createSourcesClient({ config }); // Sources HTTP client
++ * const cursor = await ingest.createChangeCursor(db, { prefix: 'deals' }).ensureSchema();
++ * await ingest.pullChanges({ db, cursor, source: sources, apply: async (item, t) => 'applied' });
++ * const consumer = ingest.createEventConsumer({ db, secrets, consumer: 'deals-sources' });
++ * const r = await consumer.apply(raw, req.headers, (event) => importer.kick());
++ *
++ * It is primitives + thin factories, not one opinionated pipeline: the products' domains diverge
++ * (News clustering, Reviews signals, Trade observations, …), so only what they share lives here —
++ * the Sources client, the change-cursor pull with per-item isolation, the signed event consumer with
++ * an exactly-once inbox, the shared pure normalisers, the PSL/registrable-host helpers and the
++ * freshness verdict.
++ *
++ * Honesty: nothing here defaults a missing date to "now", invents a value or reads fields of a
++ * Sources item beyond what the caller asks for. The cursor stores what Sources' change feed said and
++ * moves only past pages that committed.
++ *
++ * Every method that touches data takes the caller's openvibe-sdk/db transaction handle as its first
++ * argument (the store convention of this package), so the chassis writes commit or roll back with the
++ * caller's own writes.
++ */
++const { isHandle, assertPrefix, clockOf, PublishingError } = require('./internal');
++
++// ─────────────────────────────────────────────────────────────────────────────
++// normalize — the pure helpers the five copies share, ported byte-for-byte.
++// Each entry names the copy it came from (product path:line in the source tree).
++// ─────────────────────────────────────────────────────────────────────────────
++
++// News `server/domain/text.js`
++const STOPWORDS = new Set(('a about above after again against all am an and any are as at be because been before being below between both but by '
++ + 'can could did do does doing down during each few for from further had has have having he her here hers him his how i if in into is it its '
++ + 'itself just me more most my no nor not now of off on once only or other our ours out over own same she should so some such than that the '
++ + 'their theirs them then there these they this those through to too under until up very was we were what when where which while who whom '
++ + 'why will with would you your yours says said say new news report reports update updates live latest via amid over after more first year '
++ + 'years today yesterday week one two three four five six seven eight nine ten'
++).split(/\s+/));
++const NOT_ENTITIES = new Set(['the', 'a', 'an', 'in', 'on', 'at', 'as', 'how', 'why', 'what', 'when', 'where', 'who', 'new', 'breaking', 'update', 'live', 'watch', 'exclusive', 'opinion', 'analysis', 'report']);
++
++/** Lowercase, strip accents and punctuation, collapse spaces. */
++function fold(s) {
++ return String(s == null ? '' : s).normalize('NFKD').replace(/[\u0300-\u036f]/g, '')
++ .toLowerCase().replace(/[’']/g, '').replace(/[^a-z0-9]+/g, ' ').trim();
++}
++
++/** A headline normalised for comparison. */
++function titleKey(title) { return fold(title); }
++
++const TRACKING = /^(utm_[a-z]+|fbclid|gclid|mc_cid|mc_eid|ref|ref_src|cmpid|ocid|igshid|smid)$/i;
++
++/**
++ * The URL key dedupe compares: https, lowercase host without "www.", no fragment, no tracking
++ * parameters, remaining parameters sorted, no trailing slash. null for anything that is not http(s).
++ */
++function urlKey(url) {
++ if (!url) return null;
++ let u;
++ try { u = new URL(String(url)); } catch { return null; }
++ if (u.protocol !== 'http:' && u.protocol !== 'https:') return null;
++ const host = u.hostname.toLowerCase().replace(/^www\./, '');
++ const params = [...u.searchParams.entries()].filter(([k]) => !TRACKING.test(k)).sort(([a, x], [b, y]) => (a < b ? -1 : a > b ? 1 : x < y ? -1 : x > y ? 1 : 0));
++ const q = params.length ? `?${new URLSearchParams(params).toString()}` : '';
++ let p = u.pathname.replace(/\/{2,}/g, '/');
++ if (p.length > 1) p = p.replace(/\/+$/, '');
++ return `https://${host}${u.port && u.port !== '443' && u.port !== '80' ? `:${u.port}` : ''}${p}${q}`;
++}
++
++/** The outlet shown when the registry has no name: the URL's host without "www.". */
++function hostOf(url) {
++ try { return new URL(String(url)).hostname.toLowerCase().replace(/^www\./, ''); } catch { return null; }
++}
++
++// Second-level labels under a two-letter country code that are not a publisher's own name
++// (bbc.co.uk, abc.net.au): the publisher is the label before them.
++const COUNTRY_SECOND_LEVEL = new Set(['co', 'com', 'org', 'net', 'gov', 'edu', 'ac', 'ne', 'or', 'go', 'gob', 'nic', 'mil']);
++
++/**
++ * The publisher's domain of a URL: its host without subdomains (news.example.com and
++ * www.example.com → example.com; news.bbc.co.uk → bbc.co.uk). Deliberately coarse.
++ */
++function publisherDomain(url) {
++ const host = hostOf(url);
++ if (!host) return null;
++ if (/^[\d.]+$/.test(host) || host.startsWith('[')) return host;
++ const labels = host.split('.').filter(Boolean);
++ if (labels.length <= 2) return labels.join('.');
++ const n = labels.length;
++ const take = labels[n - 1].length === 2 && COUNTRY_SECOND_LEVEL.has(labels[n - 2]) ? 3 : 2;
++ return labels.slice(-take).join('.');
++}
++
++/**
++ * Independent sources among source items: items are one source when they share the Sources source,
++ * the publisher's domain, the outlet name, or the original they duplicate; transitive. → groups of ids.
++ */
++function independentSources(items) {
++ const parent = new Map();
++ const find = (k) => { while (parent.get(k) !== k) { parent.set(k, parent.get(parent.get(k))); k = parent.get(k); } return k; };
++ const union = (a, b) => { const x = find(a); const y = find(b); if (x !== y) parent.set(y, x); };
++ for (const it of items) {
++ const node = `item:${it.id}`;
++ if (!parent.has(node)) parent.set(node, node);
++ const domain = publisherDomain(it.canonical_url);
++ const outlet = fold(it.outlet);
++ const keys = [`source:${it.source_key}`, `original:${it.duplicate_of || it.id}`, domain ? `domain:${domain}` : null, outlet ? `outlet:${outlet}` : null].filter(Boolean);
++ for (const k of keys) {
++ if (!parent.has(k)) parent.set(k, k);
++ union(node, k);
++ }
++ }
++ const groups = new Map();
++ for (const it of items) {
++ const root = find(`item:${it.id}`);
++ if (!groups.has(root)) groups.set(root, []);
++ groups.get(root).push(it.id);
++ }
++ return [...groups.values()];
++}
++
++/** Very light stemming: plural and possessive endings, so "rockets" and "rocket" meet. */
++function stem(w) {
++ if (w.length > 4 && w.endsWith('ies')) return `${w.slice(0, -3)}y`;
++ if (w.length > 3 && w.endsWith('s') && !w.endsWith('ss') && !w.endsWith('us') && !w.endsWith('is')) return w.slice(0, -1);
++ return w;
++}
++
++/** Content terms of a text: folded words of 3+ letters (or any number), minus stopwords, stemmed, unique, sorted. */
++function terms(text) {
++ const out = new Set();
++ for (const w of fold(text).split(' ')) {
++ if (!w || STOPWORDS.has(w)) continue;
++ if (w.length < 3 && !/^\d+$/.test(w)) continue;
++ out.add(stem(w));
++ }
++ return [...out].sort();
++}
++
++/**
++ * Named entities, deterministically: runs of Capitalised words (and ALL-CAPS acronyms of 2+ letters),
++ * folded. Sorted, unique.
++ */
++function entities(text) {
++ const src = String(text || '');
++ const tokens = src.split(/\s+/).filter(Boolean);
++ const out = new Set();
++ let run = [];
++ const laterCapitalised = (w, idx) => tokens.slice(idx + 1).some((t) => t.replace(/[^\p{L}\p{N}]/gu, '') === w);
++ const flush = () => {
++ if (run.length) {
++ const words = run.map((r) => r.word);
++ const startsText = run[0].index === 0;
++ const acronym = words.some((w) => /^[A-Z0-9]{2,}$/.test(w));
++ if (!(startsText && words.length === 1 && !acronym && !laterCapitalised(words[0], run[0].index))) {
++ const f = fold(words.join(' '));
++ if (f && !NOT_ENTITIES.has(f) && !STOPWORDS.has(f)) out.add(f);
++ }
++ }
++ run = [];
++ };
++ tokens.forEach((raw, index) => {
++ const word = raw.replace(/^[^\p{L}\p{N}]+|[^\p{L}\p{N}]+$/gu, '');
++ const cap = /^\p{Lu}/u.test(word) && !NOT_ENTITIES.has(word.toLowerCase());
++ if (word && cap) run.push({ word, index }); else flush();
++ if (/[.!?:;,]$/.test(raw)) flush();
++ });
++ flush();
++ return [...out].sort();
++}
++
++/** Word 2-shingles of a folded headline (single-word headlines give the word itself). */
++function shingles(title) {
++ const w = titleKey(title).split(' ').filter(Boolean);
++ if (w.length < 2) return new Set(w);
++ const s = new Set();
++ for (let i = 0; i < w.length - 1; i++) s.add(`${w[i]} ${w[i + 1]}`);
++ return s;
++}
++
++function jaccard(a, b) {
++ if (!a.size && !b.size) return 0;
++ let inter = 0;
++ for (const x of a) if (b.has(x)) inter++;
++ return inter / (a.size + b.size - inter);
++}
++
++const ALLOWS_SUMMARY = /\b(short|brief)\s+summar(y|ies)\b|\bsummar(y|ies)\s+(are|is)\s+(allowed|permitted)\b/i;
++const FORBIDS_SUMMARY = /\b(no|not|never|without)\s+(short\s+|brief\s+)?summar(y|ies)\b|\bsummar(y|ies)\s+(are|is)\s+not\b|\bheadlines?\s+(and\s+links\s+)?only\b|\btitles?\s+and\s+links\s+only\b/i;
++
++/**
++ * What of a source's summary may be kept: a short summary, only when the item's terms or licence
++ * note explicitly allows it (and nothing forbids it), cut to maxChars at a word boundary.
++ * → { summary, basis }
++ */
++function licensedSummary(summary, { termsNote, licenseNote, maxChars = 280 } = {}) {
++ const notes = `${termsNote || ''}\n${licenseNote || ''}`;
++ const text = String(summary || '').replace(/\s+/g, ' ').trim();
++ if (!text) return { summary: null, basis: 'none_provided' };
++ if (!maxChars) return { summary: null, basis: 'summaries_disabled' };
++ if (FORBIDS_SUMMARY.test(notes)) return { summary: null, basis: 'terms_forbid_summaries' };
++ if (!ALLOWS_SUMMARY.test(notes)) return { summary: null, basis: 'terms_do_not_allow_summaries' };
++ if (text.length <= maxChars) return { summary: text, basis: 'terms_allow_short_summaries' };
++ const cut = text.slice(0, maxChars - 1);
++ const at = cut.lastIndexOf(' ');
++ return { summary: `${(at > maxChars * 0.6 ? cut.slice(0, at) : cut).replace(/[\s,;:.-]+$/, '')}…`, basis: 'terms_allow_short_summaries' };
++}
++
++// Deals `server/domain/util.js`
++/** http(s) URL with the fragment and tracking parameters removed, host lower-cased, params sorted. */
++function normalizeUrl(input) {
++ if (input == null || input === '') return null;
++ let u;
++ try { u = new URL(String(input).trim()); } catch { return null; }
++ if (u.protocol !== 'http:' && u.protocol !== 'https:') return null;
++ if (u.username || u.password) return null;
++ u.hash = '';
++ u.hostname = u.hostname.toLowerCase();
++ const keep = [...u.searchParams.entries()].filter(([k]) => !TRACKING.test(k)).sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0));
++ u.search = '';
++ for (const [k, v] of keep) u.searchParams.append(k, v);
++ let s = u.toString();
++ if (u.pathname.length > 1 && s.endsWith('/') && !u.search) s = s.slice(0, -1);
++ return s.length <= 2048 ? s : null;
++}
++
++/** The identity used to find duplicates: normalised URL without scheme and without "www.". */
++function hostPathKey(normalized) {
++ if (!normalized) return null;
++ return normalized.replace(/^https?:\/\//, '').replace(/^www\./, '');
++}
++
++function domainOf(normalized) {
++ try { return new URL(normalized).hostname.replace(/^www\./, ''); } catch { return null; }
++}
++
++function slugify(text, suffixFrom) {
++ const base = String(text || '').normalize('NFKD').replace(/[\u0300-\u036f]/g, '').toLowerCase()
++ .replace(/[^a-z0-9]+/g, '-').replace(/^-+|-+$/g, '').slice(0, 60).replace(/-+$/g, '') || 'deal';
++ const suffix = String(suffixFrom || '').replace(/^[a-z]+_/, '').slice(-6).toLowerCase();
++ return suffix ? `${base}-${suffix}` : base;
++}
++
++/**
++ * A decimal amount as stated: '12', '12.5', '1,299.99'. The digits stay as given; only thousands
++ * commas and surrounding space go. → { text, num } | null ; throws 422 on anything else.
++ */
++function parseAmount(v, field) {
++ if (v == null || v === '') return null;
++ if (typeof v === 'number') {
++ if (!Number.isFinite(v) || v < 0) throw new PublishingError(422, 'request.invalid', `${field} must be a non-negative number`);
++ return { text: String(v), num: v };
++ }
++ const s = String(v).trim().replace(/,(?=\d{3}(\D|$))/g, '');
++ if (s === '') return null;
++ if (!/^\d{1,9}(\.\d{1,4})?$/.test(s)) throw new PublishingError(422, 'request.invalid', `${field} must be a plain amount like 19.99 (no currency symbols)`);
++ return { text: s.replace(/^0+(?=\d)/, ''), num: Number(s) };
++}
++
++function parseCurrency(v, field = 'currency') {
++ if (v == null || v === '') return null;
++ const s = String(v).trim().toUpperCase();
++ if (!/^[A-Z]{3}$/.test(s)) throw new PublishingError(422, 'request.invalid', `${field} must be an ISO 4217 code such as USD or EUR`);
++ return s;
++}
++
++function parseEnum(v, allowed, field) {
++ if (v == null || v === '') return null;
++ const s = String(v).trim().toLowerCase();
++ if (!allowed.includes(s)) throw new PublishingError(422, 'request.invalid', `${field} must be one of ${allowed.join(', ')}`);
++ return s;
++}
++
++/** An instant stated by a person or source (ISO string or epoch ms) → ms; blank → null. */
++function parseInstant(v, field) {
++ if (v == null || v === '') return null;
++ const t = typeof v === 'number' ? v : Date.parse(String(v));
++ if (!Number.isFinite(t)) throw new PublishingError(422, 'request.invalid', `${field} must be a date`);
++ return t;
++}
++
++function text(v, { field, min = 0, max = 1000, required = false } = {}) {
++ const s = v == null ? '' : String(v).replace(/\s+/g, ' ').trim();
++ if (!s) {
++ if (required) throw new PublishingError(422, 'request.invalid', `${field} is required`);
++ return null;
++ }
++ if (s.length < min) throw new PublishingError(422, 'request.invalid', `${field} must be at least ${min} characters`);
++ return s.slice(0, max);
++}
++
++/** Multi-line text (descriptions): keeps line breaks, trims, caps. */
++function longText(v, max = 4000) {
++ if (v == null) return null;
++ const s = String(v).replace(/\r\n?/g, '\n').replace(/[ \t]+\n/g, '\n').trim();
++ return s ? s.slice(0, max) : null;
++}
++
++/** Lower-case word tokens (letters and digits of any script). */
++function tokens(s) {
++ return String(s || '').toLowerCase().normalize('NFKC').split(/[^\p{L}\p{N}]+/u).filter((t) => t.length > 0);
++}
++
++// Trade `server/domain/util.js`
++/**
++ * A date the caller STATES: an ISO 8601 date-time with a zone (…Z or ±hh:mm), or epoch ms as a
++ * number. Anything else (including a missing value) is null — never "now".
++ */
++function parseTime(v) {
++ if (v == null || v === '') return null;
++ if (typeof v === 'number') return Number.isFinite(v) && v > 0 ? Math.floor(v) : null;
++ const s = String(v).trim();
++ if (!/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(:\d{2}(\.\d{1,9})?)?(Z|[+-]\d{2}:?\d{2})$/i.test(s)) return null;
++ const t = Date.parse(s);
++ return Number.isFinite(t) ? t : null;
++}
++
++const DECIMAL_RE = /^-?(\d{1,20})(\.\d{1,12})?([eE][-+]?\d{1,3})?$/;
++
++/** A decimal as stated: → { text, num } or null. Numbers are kept as their shortest string form. */
++function parseDecimal(v) {
++ if (typeof v === 'number') {
++ if (!Number.isFinite(v)) return null;
++ return { text: String(v), num: v };
++ }
++ if (typeof v !== 'string') return null;
++ const s = v.trim();
++ if (!DECIMAL_RE.test(s)) return null;
++ const num = Number(s);
++ return Number.isFinite(num) ? { text: s, num } : null;
++}
++
++const invalid = (detail, code = 'request.invalid') => new PublishingError(422, code, detail);
++
++function str(v, max, name, { required = false } = {}) {
++ if (v == null || v === '') {
++ if (required) throw invalid(`${name} is required`);
++ return null;
++ }
++ if (typeof v !== 'string') throw invalid(`${name} must be a string`);
++ const s = v.trim();
++ if (!s && required) throw invalid(`${name} is required`);
++ if (s.length > max) throw invalid(`${name} is longer than ${max} characters`);
++ return s || null;
++}
++
++function httpUrl(v, name) {
++ if (v == null || v === '') return null;
++ let u;
++ try { u = new URL(String(v)); } catch { throw invalid(`${name} must be an absolute URL`); }
++ if (u.protocol !== 'https:' && u.protocol !== 'http:') throw invalid(`${name} must be http(s)`);
++ return u.toString();
++}
++
++/** Parse JSON stored by a service; a broken value is an empty default, never a crash. */
++function json(text, def) {
++ if (text == null) return def;
++ try { return JSON.parse(text); } catch { return def; }
++}
++
++/** epoch ms → ISO string, or null. */
++const iso = (ms) => (ms == null ? null : new Date(ms).toISOString());
++
++// Reviews `server/reviews/normalize.js`
++class NormalizeError extends Error {
++ constructor(message) { super(message); this.name = 'NormalizeError'; this.status = 422; this.code = 'alias.invalid'; }
++}
++
++function name(value) {
++ const s = String(value == null ? '' : value).normalize('NFKD').replace(/[\u0300-\u036f]/g, '')
++ .toLowerCase().replace(/&/g, ' and ').replace(/[^a-z0-9]+/g, ' ').trim();
++ if (!s) throw new NormalizeError('A name needs at least one letter or digit');
++ return s.slice(0, 200);
++}
++
++/** host (without www.) + path (+ query minus tracking parameters); http and https are the same thing. */
++function url(value) {
++ let u;
++ try { u = new URL(String(value || '').trim()); } catch { throw new NormalizeError('A URL alias must be an absolute http(s) URL'); }
++ if (u.protocol !== 'http:' && u.protocol !== 'https:') throw new NormalizeError('A URL alias must be http(s)');
++ const host = u.hostname.toLowerCase().replace(/^www\./, '');
++ const keep = [...u.searchParams.entries()].filter(([k]) => !/^(utm_|fbclid$|gclid$|ref$|mc_)/i.test(k)).sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0));
++ let pathname = u.pathname.replace(/\/{2,}/g, '/');
++ if (pathname.length > 1) pathname = pathname.replace(/\/+$/, '');
++ const query = keep.length ? `?${new URLSearchParams(keep).toString()}` : '';
++ return `${host}${pathname === '/' ? '' : pathname}${query}`.slice(0, 1000);
++}
++
++function code(value, what) {
++ const s = String(value == null ? '' : value).trim().toUpperCase().replace(/\s+/g, '');
++ if (!s || s.length > 100) throw new NormalizeError(`A ${what} is 1–100 characters`);
++ return s;
++}
++
++/** GTIN-8/12/13/14 → 14 digits, so the same product under two lengths is one key. */
++function gtin(value) {
++ const d = String(value == null ? '' : value).replace(/[\s-]/g, '');
++ if (!/^\d+$/.test(d) || ![8, 12, 13, 14].includes(d.length)) throw new NormalizeError('A GTIN has 8, 12, 13 or 14 digits');
++ return d.padStart(14, '0');
++}
++
++function sourceKey(value) {
++ const s = String(value == null ? '' : value).trim().toLowerCase();
++ if (!/^[a-z0-9][a-z0-9-]{1,63}$/.test(s)) throw new NormalizeError('A source binding is an OpenVibe.Sources source key');
++ return s;
++}
++
++/** namespace:id, e.g. steam_app:620 */
++function external(value) {
++ const m = String(value == null ? '' : value).trim().match(/^([A-Za-z][A-Za-z0-9_.-]{0,39}):(.{1,200})$/);
++ if (!m) throw new NormalizeError('An external id is namespace:id (e.g. steam_app:620)');
++ return `${m[1].toLowerCase()}:${m[2].trim()}`;
++}
++
++const BY_TYPE = { name, url, sku: (v) => code(v, 'SKU'), mpn: (v) => code(v, 'MPN'), gtin, source: sourceKey, external };
++
++function alias(type, value) {
++ const fn = BY_TYPE[type];
++ if (!fn) throw new NormalizeError(`alias type must be one of ${Object.keys(BY_TYPE).join(', ')}`);
++ return fn(value);
++}
++
++/** Try to normalise; null when the value is not a valid identifier of that type. */
++function tryAlias(type, value) {
++ if (value == null || value === '') return null;
++ try { return alias(type, value); } catch { return null; }
++}
++
++function slug(value) {
++ const s = String(value == null ? '' : value).normalize('NFKD').replace(/[\u0300-\u036f]/g, '')
++ .toLowerCase().replace(/&/g, '-and-').replace(/[^a-z0-9]+/g, '-').replace(/^-+|-+$/g, '').slice(0, 80).replace(/-+$/, '');
++ return s || null;
++}
++
++const normalize = Object.freeze({
++ // News
++ fold, titleKey, urlKey, hostOf, publisherDomain, independentSources, terms, entities, shingles, jaccard, stem, licensedSummary,
++ // Deals
++ normalizeUrl, hostPathKey, domainOf, slugify, parseAmount, parseCurrency, parseEnum, parseInstant, text, longText, tokens,
++ // Trade
++ parseTime, parseDecimal, str, httpUrl, json,
++ // Reviews
++ name, url, code, gtin, sourceKey, external, alias, tryAlias, slug, NormalizeError, ALIAS_TYPES: Object.keys(BY_TYPE),
++ // shared
++ iso,
++});
++
++// ─────────────────────────────────────────────────────────────────────────────
++// hosts — the PSL and registrable-host helpers (Coupons `server/domain/psl.js` and `hosts.js`).
++// ─────────────────────────────────────────────────────────────────────────────
++
++/**
++ * A small, bundled subset of the Public Suffix List (https://publicsuffix.org/list/, MPL-2.0),
++ * enough to find the registrable domain (eTLD+1) of a host. Longest matching rule wins, `*.x`
++ * matches one label, `!a.x` is an exception, else `*`. Private-section entries are included so
++ * foo.myshopify.com and bar.myshopify.com never resolve to the same site. A subset on purpose:
++ * no network fetch, no runtime dependency.
++ */
++const PSL_RULES = [
++ 'co.uk', 'org.uk', 'me.uk', 'ltd.uk', 'plc.uk', 'net.uk', 'ac.uk', 'gov.uk', 'sch.uk',
++ 'com.au', 'net.au', 'org.au', 'edu.au', 'gov.au', 'asn.au', 'id.au',
++ 'co.nz', 'net.nz', 'org.nz', 'ac.nz', 'govt.nz',
++ 'co.jp', 'ne.jp', 'or.jp', 'ac.jp', 'go.jp', 'ad.jp', 'ed.jp', 'gr.jp', 'lg.jp',
++ 'co.kr', 'or.kr', 'ne.kr', 'go.kr', 'ac.kr',
++ 'com.br', 'net.br', 'org.br', 'gov.br', 'edu.br',
++ 'com.mx', 'org.mx', 'net.mx', 'gob.mx', 'edu.mx',
++ 'com.ar', 'net.ar', 'org.ar', 'gob.ar',
++ 'com.co', 'net.co', 'org.co', 'gov.co',
++ 'com.pe', 'org.pe', 'net.pe',
++ 'com.cn', 'net.cn', 'org.cn', 'gov.cn', 'edu.cn',
++ 'com.hk', 'net.hk', 'org.hk', 'edu.hk', 'gov.hk',
++ 'com.tw', 'net.tw', 'org.tw', 'edu.tw', 'gov.tw',
++ 'com.sg', 'net.sg', 'org.sg', 'edu.sg', 'gov.sg',
++ 'com.my', 'net.my', 'org.my', 'edu.my', 'gov.my',
++ 'co.id', 'or.id', 'web.id', 'ac.id', 'go.id',
++ 'com.ph', 'net.ph', 'org.ph', 'gov.ph',
++ 'com.vn', 'net.vn', 'org.vn', 'gov.vn',
++ 'co.th', 'in.th', 'or.th', 'ac.th', 'go.th',
++ 'co.in', 'net.in', 'org.in', 'firm.in', 'gen.in', 'ind.in', 'ac.in', 'gov.in', 'edu.in',
++ 'com.tr', 'net.tr', 'org.tr', 'gov.tr', 'edu.tr',
++ 'com.ua', 'net.ua', 'org.ua', 'gov.ua',
++ 'com.pl', 'net.pl', 'org.pl',
++ 'co.za', 'org.za', 'net.za', 'gov.za', 'ac.za',
++ 'co.il', 'org.il', 'net.il', 'ac.il', 'gov.il',
++ 'com.sa', 'net.sa', 'org.sa', 'gov.sa',
++ 'com.eg', 'net.eg', 'org.eg', 'gov.eg',
++ 'co.ke', 'or.ke', 'ne.ke', 'go.ke',
++ 'com.ng', 'org.ng', 'net.ng', 'gov.ng',
++ '*.ck', '!www.ck',
++ '*.bd', '*.er', '*.fk', '*.jm', '*.kh', '*.mm', '*.np', '*.pg',
++ 'myshopify.com', 'bigcartel.com', 'wixsite.com',
++ 'github.io', 'gitlab.io', 'blogspot.com', 'herokuapp.com', 'netlify.app', 'vercel.app',
++ 'pages.dev', 'workers.dev', 'web.app', 'firebaseapp.com', 'appspot.com', 'azurewebsites.net',
++ 'cloudfront.net',
++];
++
++const PSL_EXACT = new Set();
++const PSL_WILDCARD = new Set();
++const PSL_EXCEPTION = new Set();
++for (const r of PSL_RULES) {
++ if (r.startsWith('!')) PSL_EXCEPTION.add(r.slice(1));
++ else if (r.startsWith('*.')) PSL_WILDCARD.add(r.slice(2));
++ else PSL_EXACT.add(r);
++}
++
++/** Number of labels of the public suffix of `host` (already normalized). */
++function suffixLabels(host) {
++ const labels = host.split('.');
++ let best = 1;
++ for (let i = 0; i < labels.length; i++) {
++ const candidate = labels.slice(i).join('.');
++ const n = labels.length - i;
++ if (PSL_EXCEPTION.has(candidate)) return n - 1;
++ if (PSL_EXACT.has(candidate) && n > best) best = n;
++ if (i > 0 && PSL_WILDCARD.has(candidate) && n + 1 > best) best = n + 1;
++ }
++ return best;
++}
++
++/** The public suffix of a normalized host (e.g. 'co.uk'). */
++function publicSuffix(host) {
++ const labels = host.split('.');
++ return labels.slice(labels.length - Math.min(suffixLabels(host), labels.length)).join('.');
++}
++
++/** eTLD+1, or null when the host IS a public suffix (e.g. 'co.uk', 'myshopify.com'). */
++function registrableDomain(host) {
++ const labels = host.split('.');
++ const n = suffixLabels(host);
++ if (labels.length <= n) return null;
++ return labels.slice(labels.length - n - 1).join('.');
++}
++
++const { domainToASCII } = require('url');
++
++const LABEL_RE = /^(?!-)[a-z0-9-]{1,63}(?<!-)$/;
++const SPECIAL = ['localhost', 'local', 'internal', 'invalid', 'arpa', 'test', 'example', 'onion', 'lan', 'home', 'corp'];
++
++class HostError extends Error {
++ constructor(detail) { super(detail); this.name = 'HostError'; this.status = 400; this.code = 'host.invalid'; }
++}
++
++/** → normalized host, or throws HostError. */
++function normalizeHost(input) {
++ if (typeof input !== 'string') throw new HostError('host must be a string');
++ let h = input.trim().toLowerCase();
++ if (!h || h.length > 300) throw new HostError('host is empty or too long');
++ if (/[\s/\\?#@]/.test(h)) throw new HostError('host must be a hostname only (no scheme, path, query or credentials)');
++ if (h.startsWith('[')) throw new HostError('IP addresses are not merchants');
++ const colon = h.indexOf(':');
++ if (colon !== -1) {
++ if (!/^\d{1,5}$/.test(h.slice(colon + 1))) throw new HostError('malformed port');
++ h = h.slice(0, colon);
++ }
++ h = h.replace(/\.$/, '');
++ const ascii = domainToASCII(h);
++ if (!ascii) throw new HostError('not a valid domain name');
++ h = ascii.toLowerCase();
++ if (h.length > 253) throw new HostError('host is too long');
++ const labels = h.split('.');
++ if (labels.length < 2) throw new HostError('host needs at least two labels');
++ if (!labels.every((l) => LABEL_RE.test(l))) throw new HostError('host has a malformed label');
++ if (labels.every((l) => /^\d+$/.test(l)) || /^\d+$/.test(labels[labels.length - 1])) throw new HostError('IP addresses are not merchants');
++ if (SPECIAL.includes(labels[labels.length - 1])) throw new HostError('special-use names are not merchants');
++ return h;
++}
++
++/** Normalize, or null. */
++function tryHost(input) {
++ try { return normalizeHost(input); } catch { return null; }
++}
++
++/** The registrable domain (eTLD+1) of a normalized host, or null when it is a public suffix. */
++function registrable(host) { return registrableDomain(host); }
++
++/** Host and path of an http(s) URL (for evidence and Sources items), or null. */
++function hostOfUrl(url) {
++ let u;
++ try { u = new URL(String(url)); } catch { return null; }
++ if (u.protocol !== 'https:' && u.protocol !== 'http:') return null;
++ const host = tryHost(u.hostname);
++ return host ? { host, path: u.pathname || '/' } : null;
++}
++
++/** '/Shop/x/' → '/Shop/x' ; '' | '/' → '' ; throws on anything that is not a plain path. */
++function normalizePathPrefix(p) {
++ if (p == null || p === '' || p === '/') return '';
++ const s = String(p).trim();
++ if (!s.startsWith('/') || s.length > 200 || /[\s?#\\]/.test(s) || s.includes('//') || /(^|\/)\.\.?(\/|$)/.test(s)) {
++ throw new HostError('path_prefix must be a plain path like /shop/name');
++ }
++ return s.replace(/\/+$/, '');
++}
++
++/** Does `path` start with `prefix` at a segment boundary? */
++function pathMatches(prefix, path) {
++ if (!prefix) return true;
++ if (path == null) return false;
++ return path === prefix || path.startsWith(`${prefix}/`);
++}
++
++/** Validate a new domain rule. → { host, registrable, path_prefix, include_subdomains }. */
++function checkRule({ host, include_subdomains = true, path_prefix = '' } = {}) {
++ const h = normalizeHost(host);
++ const reg = registrable(h);
++ if (!reg) throw new HostError(`${h} is a public suffix (${publicSuffix(h)}), not a site`);
++ return { host: h, registrable: reg, path_prefix: normalizePathPrefix(path_prefix), include_subdomains: include_subdomains === false || include_subdomains === 0 || include_subdomains === '0' ? 0 : 1 };
++}
++
++/** The best rule for (host, path) among candidate rule rows, or null. */
++function bestRule(rules, host, path = null) {
++ const reg = registrable(host);
++ if (!reg) return null;
++ let best = null;
++ for (const r of rules) {
++ const exact = r.host === host;
++ const parent = !exact && r.include_subdomains && host.endsWith(`.${r.host}`);
++ if (!exact && !parent) continue;
++ if (r.host.length < reg.length || !(r.host === reg || r.host.endsWith(`.${reg}`))) continue;
++ if (!pathMatches(r.path_prefix, path)) continue;
++ if (!best || r.host.length > best.host.length || (r.host.length === best.host.length && r.path_prefix.length > best.path_prefix.length)) best = r;
++ }
++ return best;
++}
++
++/** Every host a lookup could match a rule on: the host and its parents down to the registrable domain. */
++function candidateHosts(host) {
++ const reg = registrable(host);
++ if (!reg) return [];
++ const out = [];
++ let h = host;
++ for (;;) {
++ out.push(h);
++ if (h === reg) break;
++ h = h.slice(h.indexOf('.') + 1);
++ }
++ return out;
++}
++
++const hosts = Object.freeze({
++ PSL_RULES, suffixLabels, publicSuffix, registrableDomain,
++ HostError, normalizeHost, tryHost, registrable, hostOfUrl, normalizePathPrefix, pathMatches, checkRule, bestRule, candidateHosts,
++});
++
++// ─────────────────────────────────────────────────────────────────────────────
++// freshness — the generic staleness verdict (Trade `server/domain/freshness.js`, pure half).
++// ─────────────────────────────────────────────────────────────────────────────
++
++/**
++ * The verdict for one source-status row at `now` (epoch ms). A row:
++ * { stale_after_sec, last_success_at, stale_since, name, upstream_status, reported_at, terms_note, license_note }.
++ * A source is fresh only when its last successful fetch is known and younger than its window
++ * (row.stale_after_sec || defaultStaleAfterSec). Everything else is stale; nothing is invented.
++ */
++function freshnessVerdict(row, now, defaultStaleAfterSec) {
++ if (!row) return { known: false, stale: true, staleSince: null, window: defaultStaleAfterSec, lastSuccessAt: null };
++ const window = row.stale_after_sec || defaultStaleAfterSec;
++ if (row.last_success_at == null) return { known: true, stale: true, staleSince: row.stale_since, window, lastSuccessAt: null };
++ const until = row.last_success_at + window * 1000;
++ return { known: true, stale: now > until, staleSince: now > until ? until : null, window, lastSuccessAt: row.last_success_at };
++}
++
++/** The serialisable view of one status row (Trade's freshness.view shape). */
++function freshnessView(row, key, now, defaultStaleAfterSec) {
++ const v = freshnessVerdict(row, now, defaultStaleAfterSec);
++ return {
++ key,
++ name: row ? row.name : null,
++ known: v.known,
++ status: !v.known ? 'unknown' : v.stale ? 'stale' : 'fresh',
++ stale: v.stale,
++ stale_since: iso(v.staleSince),
++ last_success_at: iso(v.lastSuccessAt),
++ stale_after_sec: v.window,
++ upstream_status: row ? row.upstream_status : null,
++ reported_at: row ? iso(row.reported_at) : null,
++ terms_note: row ? row.terms_note : null,
++ license_note: row ? row.license_note : null,
++ };
++}
++
++const freshness = Object.freeze({ verdict: freshnessVerdict, view: freshnessView });
++
++// ─────────────────────────────────────────────────────────────────────────────
++// Sources client
++// ─────────────────────────────────────────────────────────────────────────────
++
++class SourcesError extends Error {
++ constructor(code, message, status = null) { super(message); this.name = 'SourcesError'; this.code = code; this.status = status; }
++}
++
++/**
++ * OpenVibe.Sources client (host-local API, client-credentials token for audience openvibe.sources).
++ *
++ * sources.item.read GET /api/v1/items?category=<cat>&after=&limit=&include_removed=1 (change order)
++ * GET /api/v1/items/:id
++ * sources.source.read GET /api/v1/sources (health and staleness of every source)
++ * GET /api/v1/sources/:key (a registry record)
++ *
++ * config: { networkInternalUrl, oauth: { clientId, clientSecret }, sources: { internalUrl, category, timeoutMs } }.
++ * Failures throw a SourcesError with a stable code (sources.not_configured, sources.token_unavailable,
++ * sources.unavailable, sources.http_<status>, sources.bad_response); the caller records them. Nothing
++ * here invents an item.
++ */
++function createSourcesClient({ config, fetchImpl = globalThis.fetch, now = () => Date.now(), timeoutMs = null } = {}) {
++ const src = (config && config.sources) || {};
++ const base = src.internalUrl;
++ const enabled = Boolean(config && config.oauth && config.oauth.clientSecret && base);
++ const { createServiceTokenClient } = require('openvibe-sdk/auth');
++ const tokens = enabled ? createServiceTokenClient({
++ network: config.networkInternalUrl, tokenUrl: config.networkInternalUrl ? `${String(config.networkInternalUrl).replace(/\/+$/, '')}/oauth/token` : undefined,
++ clientId: config.oauth.clientId, clientSecret: config.oauth.clientSecret, fetch: fetchImpl, now,
++ }) : null;
++ const wallMs = timeoutMs || src.timeoutMs || 10000;
++ const ITEM = { audience: 'openvibe.sources', scope: 'sources.item.read' };
++ const SOURCE = { audience: 'openvibe.sources', scope: 'sources.source.read' };
++
++ async function call(ctx, path) {
++ if (!tokens) throw new SourcesError('sources.not_configured', 'OV_OAUTH_CLIENT_SECRET or OV_SOURCES_INTERNAL_URL is not set');
++ let auth;
++ try { auth = await tokens.authHeaders(ctx); } catch (err) {
++ throw new SourcesError('sources.token_unavailable', `No service token for OpenVibe.Sources: ${err.message}`);
++ }
++ let res;
++ try {
++ res = await fetchImpl(`${base}${path}`, { headers: { Accept: 'application/json', ...auth }, signal: AbortSignal.timeout(wallMs) });
++ } catch (err) {
++ throw new SourcesError('sources.unavailable', `OpenVibe.Sources is unreachable: ${err.message}`);
++ }
++ if (res.status === 401) tokens.invalidate(ctx);
++ const body = await res.json().catch(() => null);
++ if (!res.ok) {
++ throw new SourcesError(`sources.http_${res.status}`, `OpenVibe.Sources answered ${res.status}${body && (body.code || body.detail) ? `: ${body.code || ''} ${body.detail || ''}`.trimEnd() : ''}`, res.status);
++ }
++ if (!body || typeof body !== 'object') throw new SourcesError('sources.bad_response', 'OpenVibe.Sources sent no JSON');
++ return body;
++ }
++
++ function category() {
++ if (!src.category) throw new SourcesError('sources.bad_response', 'config.sources.category is required');
++ return src.category;
++ }
++
++ return {
++ enabled,
++ SourcesError,
++ /** One page of items in change order. → { items, next_after, more, sources } */
++ async listItems({ after = 0, limit = 100 } = {}) {
++ const qs = new URLSearchParams({ category: category(), after: String(after), limit: String(limit), include_removed: '1' });
++ const body = await call(ITEM, `/api/v1/items?${qs}`);
++ if (!Array.isArray(body.items) || !Number.isInteger(body.next_after)) throw new SourcesError('sources.bad_response', 'items page without items/next_after');
++ return body;
++ },
++ /** One item (with its source's health). → { item, source } */
++ async getItem(id) {
++ const body = await call(ITEM, `/api/v1/items/${encodeURIComponent(id)}`);
++ if (!body.item || typeof body.item.id !== 'string') throw new SourcesError('sources.bad_response', 'item response without item');
++ return body;
++ },
++ /** Every source known to Sources. → { sources: [...] } */
++ async listSources() {
++ const body = await call(SOURCE, '/api/v1/sources');
++ if (!Array.isArray(body.sources)) throw new SourcesError('sources.bad_response', 'sources response without sources');
++ return body;
++ },
++ /** A registry record (name, homepage, health). Failure is the caller's "unknown". */
++ async getSource(key) {
++ const body = await call(SOURCE, `/api/v1/sources/${encodeURIComponent(key)}`);
++ return body.source || body;
++ },
++ };
++}
++
++// ─────────────────────────────────────────────────────────────────────────────
++// Change cursor + pull
++// ─────────────────────────────────────────────────────────────────────────────
++
++/** The ingest cursor's DDL for one prefix (idempotent), for the service's migration file. */
++function ingestSchema(prefix) {
++ assertPrefix(prefix);
++ const T = `${prefix}_ingest_cursor`;
++ return `CREATE TABLE IF NOT EXISTS ${T} (
++ name text COLLATE "C" NOT NULL,
++ cursor bigint NOT NULL,
++ updated_at bigint NOT NULL,
++ PRIMARY KEY (name)
++);
++`;
++}
++
++/**
++ * A named change cursor in <prefix>_ingest_cursor of the product's database. One row per feed
++ * (e.g. 'sources'), advanced by pullChanges only past pages that committed. get() reads on the
++ * store's handle; set(t, name, value) takes the caller's transaction handle first.
++ */
++function createChangeCursor(db, { prefix, now } = {}) {
++ const { assertDb, sqlOf } = require('./internal');
++ assertDb(db);
++ assertPrefix(prefix);
++ const sql = sqlOf(db);
++ const clock = clockOf(now);
++ const T = `${prefix}_ingest_cursor`;
++ const t$ = sql.ident(T);
++ const api = {
++ table: T,
++ schema: () => ingestSchema(prefix),
++ /** Create the table where the handle may (tests, PGlite); services put schema() in a migration. */
++ async ensureSchema() { await db.query(ingestSchema(prefix)); return api; },
++ /** The stored cursor for `name` (0 when the row is absent). */
++ async get(name = 'default') {
++ return (await db.value(sql`SELECT cursor FROM ${t$} WHERE name = ${String(name)}`)) || 0;
++ },
++ /** Advance the cursor for `name`. set(t, name, value) writes on the caller's handle. */
++ async set(...args) {
++ const t = isHandle(args[0]) ? args[0] : db;
++ const [name = 'default', value] = args.slice(isHandle(args[0]) ? 1 : 0);
++ const n = Math.floor(Number(value));
++ if (!Number.isFinite(n) || n < 0) throw new TypeError('cursor value must be a non-negative integer');
++ await t.exec(sql`INSERT INTO ${t$} (name, cursor, updated_at) VALUES (${String(name)}, ${n}, ${clock()})
++ ON CONFLICT (name) DO UPDATE SET cursor = GREATEST(${t$}.cursor, excluded.cursor), updated_at = excluded.updated_at`);
++ return n;
++ },
++ };
++ return api;
++}
++
++/** The apply() result, one of these three strings (or { outcome }). Anything else is a bug. */
++const APPLY_OUTCOMES = new Set(['applied', 'hold', 'removed']);
++function applyOutcome(value) {
++ const o = value && typeof value === 'object' ? value.outcome : value;
++ if (!APPLY_OUTCOMES.has(o)) throw new TypeError(`apply(item, t) must return one of ${[...APPLY_OUTCOMES].join(', ')}`);
++ return o;
++}
++
++/**
++ * Pull the change feed from the cursor, applying each item:
++ *
++ * - one transaction per page: the page's writes AND the cursor advance commit together, so a crash
++ * replays the page (product writes must be idempotent on (item id, revision));
++ * - one savepoint per item: an item whose apply() throws is rolled back alone and counted as
++ * `failed`, so a bad item never stalls its page or the cursor;
++ * - `hold` is not an error and does not block: the item is counted and the cursor moves past it
++ * (the product records the hold itself, e.g. "no merchant yet").
++ *
++ * apply(item, t) → 'applied' | 'hold' | 'removed'. A fetch/list failure throws (the caller records it
++ * and tries again). → { pages, applied, hold, removed, failed, after }.
++ */
++async function pullChanges({ db, cursor, source, apply, name = 'default', maxPages = 25, pageSize = 200, onItem = null } = {}) {
++ if (typeof apply !== 'function') throw new TypeError('pullChanges needs apply(item, t)');
++ if (!source || typeof source.listItems !== 'function') throw new TypeError('pullChanges needs a source with listItems({ after, limit })');
++ const counts = { pages: 0, applied: 0, hold: 0, removed: 0, failed: 0 };
++ let after = await cursor.get(name);
++ const limit = Number.isInteger(pageSize) && pageSize > 0 ? pageSize : 200;
++ for (let p = 0; p < maxPages; p++) {
++ const body = await source.listItems({ after, limit });
++ const items = Array.isArray(body.items) ? body.items : [];
++ await db.tx(async (t) => {
++ for (const item of items) {
++ let outcome;
++ try {
++ outcome = await t.tx(async (st) => applyOutcome(await apply(item, st)));
++ } catch (err) {
++ outcome = 'failed';
++ if (onItem) onItem(item, err, 'failed');
++ }
++ if (outcome !== 'failed' && onItem) onItem(item, null, outcome);
++ counts[outcome] = (counts[outcome] || 0) + 1;
++ }
++ const next = Number.isInteger(body.next_after) ? Math.max(body.next_after, after) : after;
++ await cursor.set(t, name, next);
++ after = next;
++ });
++ counts.pages++;
++ if (!body.more) break;
++ }
++ return { ...counts, after };
++}
++
++// ─────────────────────────────────────────────────────────────────────────────
++// Signed event consumer
++// ─────────────────────────────────────────────────────────────────────────────
++
++/**
++ * A signed OpenVibe.Events delivery consumer: the shared half of POST /internal/events.
++ *
++ * const consumer = createEventConsumer({ db, secrets: config.events.webhookSecrets, consumer: 'deals-sources' });
++ * const r = await consumer.apply(req.body, req.headers, async (event) => { ... });
++ * // r: { status, code? , event_id?, duplicate?, outcome? } — map status onto your HTTP response.
++ *
++ * The signature is X-OpenVibe-Signature v2 (HMAC-SHA256 of "<t>.<raw body>"), accepted only within
++ * ±300 s; a v1-only or stale delivery is refused (openvibe-sdk/events parseDelivery requireV2).
++ * Exactly once: the inbox receipt (consumer, event_id) claims in the same transaction as the handler,
++ * so a redelivery changes nothing and a failed handler rolls back its receipt and is retried.
++ */
++function createEventConsumer({ db, secrets, consumer, now = () => Date.now(), table } = {}) {
++ const { assertDb } = require('./internal');
++ assertDb(db);
++ if (!consumer || typeof consumer !== 'string') throw new TypeError('consumer is required (e.g. "deals-sources")');
++ const list = [].concat(secrets || []).filter(Boolean);
++ const { createPgInbox, parseDelivery } = require('openvibe-sdk/events');
++ const inbox = createPgInbox(db, table ? { table, now } : { now });
++
++ function verify(raw, headers) {
++ if (!list.length) return null;
++ for (const secret of list) {
++ const delivery = parseDelivery(raw, headers, secret, { requireV2: true });
++ if (delivery) return delivery;
++ }
++ return null;
++ }
++
++ return {
++ inbox,
++ verify,
++ /** One delivery: verify the signature, then run handler(event) once per event_id. */
++ async apply(raw, headers, handler) {
++ if (!list.length) return { status: 503, code: 'ingest.webhook_disabled' };
++ const delivery = verify(raw, headers);
++ if (!delivery) return { status: 401, code: 'ingest.bad_signature' };
++ const event = delivery.event;
++ if (!event || typeof event.event_id !== 'string' || !/^evt_[0-9A-HJKMNP-TV-Z]{26}$/.test(event.event_id)) {
++ return { status: 400, code: 'ingest.bad_delivery' };
++ }
++ const r = await inbox.once(consumer, event.event_id, () => handler(event));
++ return { status: 200, event_id: event.event_id, duplicate: r.duplicate, outcome: r.duplicate ? null : (r.result === undefined ? null : r.result) };
++ },
++ };
++}
++
++module.exports = {
++ createSourcesClient, SourcesError,
++ createChangeCursor, pullChanges, schema: ingestSchema,
++ createEventConsumer,
++ normalize, hosts, freshness,
++};
ddiff --git a/lib/publication.js b/lib/publication.js
nnew file mode 100644
iindex 0000000..c09308a
---- /dev/null
++++ b/lib/publication.js
@@@ -0,0 +1,116 @@
++'use strict';
++/**
++ * openvibe-publishing/publication — the shared publication glue: gate decision → index document →
++ * sequencer.stamp → events → outbox.enqueue, plus the IndexNow ping. It is the half the five content
++ * products each re-implement (News domain/publication.js, Reviews service.js, Deals domain/indexing.js,
++ * Coupons domain/publication.js, Trade domain/indexing.js).
++ *
++ * const publication = createPublication({
++ * owner: 'deals', baseUrl: config.baseUrl, sequencer: store.sequencer, outbox,
++ * indexnow, // optional { enabled, pingSoon([urls]) }
++ * decide: (offer, now) => publication.decideOffer(offer), // the product's gate facts
++ * document: (offer, decision) => offerDocument(offer), // hooks.buildIndexDocument/tombstone
++ * page: (offer) => `/d/${offer.slug}`,
++ * });
++ * await db.tx(async (t) => {
++ * await t.exec(sql`UPDATE deal_offers SET … WHERE id = ${id}`);
++ * await publication.sync(t, offer); // stamps + enqueues in the SAME transaction
++ * });
++ *
++ * What stays per product: the gate facts, the facets, the document body and the domain tables. This
++ * module only owns the mechanics — the sequencer revision, the two events, the outbox write and the
++ * IndexNow ping — and it always writes them on the CALLER's transaction handle, so an index document,
++ * its event and the product's own change commit or roll back together (ADR-004).
++ *
++ * Events are exactly what OpenVibe.Search consumes: `search.index-document@1` carried by
++ * `<owner>.index_document.upserted|deleted` (openvibe-publishing/index-hooks), and the product's own
++ * `<owner>.<type>.<action>` publication events. Nothing non-listable is ever sent as a document: it is
++ * a tombstone, so a state or visibility change always removes the old copy from Search. A resource
++ * that was never indexed gets no tombstone.
++ */
++const seo = require('./seo');
++const hooks = require('./index-hooks');
++const { isHandle } = require('./internal');
++
++const OWNER_RE = /^[a-z][a-z0-9_]{1,39}$/;
++
++/** outbox.enqueue(t, envelope, { traceparent }) is the SDK's pg outbox; emitIn is the service kit's. */
++function writerOf(outbox) {
++ if (outbox && typeof outbox.enqueue === 'function') return (t, env, opts) => outbox.enqueue(t, env, opts);
++ if (outbox && typeof outbox.emitIn === 'function') return (t, env, opts) => outbox.emitIn(t, env, opts);
++ throw new TypeError('createPublication needs an outbox with enqueue(t, envelope) (or emitIn)');
++}
++
++/**
++ * The product's publication factory. `sequencer` is openvibe-publishing/index-hooks'
++ * createIndexSequencer (its stamp(t, doc) takes the caller's handle); `outbox` is the SDK's pg outbox
++ * (or the service kit's). `decide`/`document`/`page` are the product's gate, document builder and
++ * canonical path.
++ */
++function createPublication({
++ owner, sequencer, outbox, baseUrl = null, indexnow = null, db = null,
++ now = () => Date.now(), decide = null, document = null, page = null,
++} = {}) {
++ if (!OWNER_RE.test(String(owner || ''))) throw new TypeError('owner must be a service slug of [a-z0-9_], e.g. "deals"');
++ if (!sequencer || typeof sequencer.stamp !== 'function' || typeof sequencer.current !== 'function') {
++ throw new TypeError('sequencer must be openvibe-publishing/index-hooks createIndexSequencer');
++ }
++ const enqueue = writerOf(outbox);
++ const clock = () => {
++ const t = Number(now());
++ if (!Number.isFinite(t)) throw new TypeError('now() must return epoch milliseconds');
++ return t;
++ };
++ const abs = (p) => (baseUrl ? seo.canonicalUrl(baseUrl, p) : p);
++
++ /**
++ * Stamp and enqueue one document on the caller's transaction handle. The revision is the sequencer's
++ * (same document → same revision → no event). Never-indexed tombstones send nothing.
++ */
++ async function index(t, { document: doc, page: pagePath = null, traceparent } = {}) {
++ if (!isHandle(t)) throw new TypeError('index(t, …): pass the transaction handle db.tx gives you (or db)');
++ if (!doc || typeof doc !== 'object') throw new TypeError('index needs a document (buildIndexDocument/tombstone)');
++ const current = await sequencer.current(t, owner, doc.type, doc.id);
++ if (doc.deleted && current == null) return null; // never indexed: nothing to remove
++ const stamped = await sequencer.stamp(t, doc);
++ if (current != null && stamped.revision === current) return null; // unchanged: no event
++ await enqueue(t, hooks.indexEvent({ document: stamped, now: clock() }), { traceparent });
++ // IndexNow: an indexable page appeared or changed, or a page Search already had went away.
++ // A draft, private or noindex page never pings. pingSoon never throws and is a no-op without a key.
++ if (indexnow && indexnow.enabled && pagePath) {
++ const indexable = !stamped.deleted && stamped.indexability && stamped.indexability.decision === 'index';
++ const removed = stamped.deleted && current != null;
++ if (indexable || removed) indexnow.pingSoon([abs(pagePath), abs('/sitemap.xml')]);
++ }
++ return stamped;
++ }
++
++ /** A tombstone for (type, id): sends nothing when the resource was never indexed. */
++ async function tombstone(t, { type, id, page: pagePath = null, traceparent } = {}) {
++ return await index(t, { document: hooks.tombstone({ owner, type, id, revision: 0 }), page: pagePath, traceparent });
++ }
++
++ /** The product event `<owner>.<type>.<action>` for a transition, on the caller's transaction handle. */
++ async function publication(t, { type, action, id, revision, document: doc, decision = null, actor = null, extra = {}, traceparent } = {}) {
++ if (!isHandle(t)) throw new TypeError('publication(t, …): pass the transaction handle db.tx gives you (or db)');
++ const env = hooks.publicationEvent({ product: owner, type, action, id, revision, actor, document: doc, decision, now: clock(), extra });
++ await enqueue(t, env, { traceparent });
++ return env;
++ }
++
++ /**
++ * gate → document → stamp → event → outbox, on the caller's handle: the whole chain for one entity,
++ * when the product supplied decide/document. `forSearch` is passed to document() so a product can
++ * render a full document for its own events and a tombstone for Search.
++ */
++ async function sync(t, entity, { traceparent, forSearch = true } = {}) {
++ if (typeof decide !== 'function' || typeof document !== 'function') throw new TypeError('sync needs decide(entity, now) and document(entity, decision, now)');
++ const decision = await decide(entity, clock());
++ const doc = await document(entity, decision, clock(), { forSearch });
++ return await index(t, { document: doc, page: page ? page(entity) : null, traceparent });
++ }
++
++ return { owner, db, sequencer, outbox, indexnow, abs, index, tombstone, publication, sync };
++}
++
++module.exports = { createPublication };
ddiff --git a/package.json b/package.json
iindex d272fbc..4fd0d71 100644
---- a/package.json
++++ b/package.json
@@@ -1,7 +1,7 @@
{
"name": "openvibe-publishing",
-- "version": "1.0.0",
-- "description": "Shared publishing packages for the OpenVibe publication products, on PostgreSQL through openvibe-sdk/db: revisions, scheduling, taxonomy, citations, media attachments, discussion references, the SEO indexability gate, authorship, index hooks and SSR helpers. Packages, not an authority.",
++ "version": "1.1.0",
++ "description": "Shared publishing packages for the OpenVibe publication products, on PostgreSQL through openvibe-sdk/db: revisions, scheduling, taxonomy, citations, media attachments, discussion references, the SEO indexability gate, authorship, index hooks, SSR helpers, the ingest chassis (Sources client, change cursor, signed event consumer, normalisers, PSL) and the publication glue. Packages, not an authority.",
"main": "index.js",
"exports": {
".": "./index.js",
@@@ -16,7 +16,9 @@
"./authorship": "./lib/authorship.js",
"./index-hooks": "./lib/index-hooks.js",
"./ssr": "./lib/ssr.js",
-- "./ai": "./lib/ai.js"
++ "./ai": "./lib/ai.js",
++ "./ingest": "./lib/ingest.js",
++ "./publication": "./lib/publication.js"
},
"files": [
"index.js",
ddiff --git a/test/fixtures/normalize.json b/test/fixtures/normalize.json
nnew file mode 100644
iindex 0000000..f414945
---- /dev/null
++++ b/test/fixtures/normalize.json
@@@ -0,0 +1,1288 @@
++{
++ "_about": "Captured from the five current product copies (read-only) so the ported helpers can be proven byte-for-byte. Each case cites its source file:line; regenerate from the J1 capture notes.",
++ "cases": [
++ {
++ "ns": "normalize",
++ "fn": "fold",
++ "args": [
++ "Héllo, World!"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:22",
++ "value": "hello world"
++ },
++ {
++ "ns": "normalize",
++ "fn": "titleKey",
++ "args": [
++ "Héllo, World!"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:28",
++ "value": "hello world"
++ },
++ {
++ "ns": "normalize",
++ "fn": "fold",
++ "args": [
++ "It's the Rye — a grass"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:22",
++ "value": "its the rye a grass"
++ },
++ {
++ "ns": "normalize",
++ "fn": "titleKey",
++ "args": [
++ "It's the Rye — a grass"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:28",
++ "value": "its the rye a grass"
++ },
++ {
++ "ns": "normalize",
++ "fn": "fold",
++ "args": [
++ "ABC 123"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:22",
++ "value": "abc 123"
++ },
++ {
++ "ns": "normalize",
++ "fn": "titleKey",
++ "args": [
++ "ABC 123"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:28",
++ "value": "abc 123"
++ },
++ {
++ "ns": "normalize",
++ "fn": "fold",
++ "args": [
++ ""
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:22",
++ "value": ""
++ },
++ {
++ "ns": "normalize",
++ "fn": "titleKey",
++ "args": [
++ ""
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:28",
++ "value": ""
++ },
++ {
++ "ns": "normalize",
++ "fn": "fold",
++ "args": [
++ null
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:22",
++ "value": ""
++ },
++ {
++ "ns": "normalize",
++ "fn": "titleKey",
++ "args": [
++ null
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:28",
++ "value": ""
++ },
++ {
++ "ns": "normalize",
++ "fn": "fold",
++ "args": [
++ "naïve café"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:22",
++ "value": "naive cafe"
++ },
++ {
++ "ns": "normalize",
++ "fn": "titleKey",
++ "args": [
++ "naïve café"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:28",
++ "value": "naive cafe"
++ },
++ {
++ "ns": "normalize",
++ "fn": "urlKey",
++ "args": [
++ "https://WWW.Example.com/a/b/?utm_source=x&b=2&a=1#frag"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:36",
++ "value": "https://example.com/a/b?a=1&b=2"
++ },
++ {
++ "ns": "normalize",
++ "fn": "hostOf",
++ "args": [
++ "https://WWW.Example.com/a/b/?utm_source=x&b=2&a=1#frag"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:50",
++ "value": "example.com"
++ },
++ {
++ "ns": "normalize",
++ "fn": "publisherDomain",
++ "args": [
++ "https://WWW.Example.com/a/b/?utm_source=x&b=2&a=1#frag"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:63",
++ "value": "example.com"
++ },
++ {
++ "ns": "normalize",
++ "fn": "urlKey",
++ "args": [
++ "http://example.com/path"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:36",
++ "value": "https://example.com/path"
++ },
++ {
++ "ns": "normalize",
++ "fn": "hostOf",
++ "args": [
++ "http://example.com/path"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:50",
++ "value": "example.com"
++ },
++ {
++ "ns": "normalize",
++ "fn": "publisherDomain",
++ "args": [
++ "http://example.com/path"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:63",
++ "value": "example.com"
++ },
++ {
++ "ns": "normalize",
++ "fn": "urlKey",
++ "args": [
++ "https://example.com/"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:36",
++ "value": "https://example.com/"
++ },
++ {
++ "ns": "normalize",
++ "fn": "hostOf",
++ "args": [
++ "https://example.com/"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:50",
++ "value": "example.com"
++ },
++ {
++ "ns": "normalize",
++ "fn": "publisherDomain",
++ "args": [
++ "https://example.com/"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:63",
++ "value": "example.com"
++ },
++ {
++ "ns": "normalize",
++ "fn": "urlKey",
++ "args": [
++ "https://news.bbc.co.uk/x"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:36",
++ "value": "https://news.bbc.co.uk/x"
++ },
++ {
++ "ns": "normalize",
++ "fn": "hostOf",
++ "args": [
++ "https://news.bbc.co.uk/x"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:50",
++ "value": "news.bbc.co.uk"
++ },
++ {
++ "ns": "normalize",
++ "fn": "publisherDomain",
++ "args": [
++ "https://news.bbc.co.uk/x"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:63",
++ "value": "bbc.co.uk"
++ },
++ {
++ "ns": "normalize",
++ "fn": "urlKey",
++ "args": [
++ "ftp://example.com/x"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:36",
++ "value": null
++ },
++ {
++ "ns": "normalize",
++ "fn": "hostOf",
++ "args": [
++ "ftp://example.com/x"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:50",
++ "value": "example.com"
++ },
++ {
++ "ns": "normalize",
++ "fn": "publisherDomain",
++ "args": [
++ "ftp://example.com/x"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:63",
++ "value": "example.com"
++ },
++ {
++ "ns": "normalize",
++ "fn": "urlKey",
++ "args": [
++ "not a url"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:36",
++ "value": null
++ },
++ {
++ "ns": "normalize",
++ "fn": "hostOf",
++ "args": [
++ "not a url"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:50",
++ "value": null
++ },
++ {
++ "ns": "normalize",
++ "fn": "publisherDomain",
++ "args": [
++ "not a url"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:63",
++ "value": null
++ },
++ {
++ "ns": "normalize",
++ "fn": "urlKey",
++ "args": [
++ ""
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:36",
++ "value": null
++ },
++ {
++ "ns": "normalize",
++ "fn": "hostOf",
++ "args": [
++ ""
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:50",
++ "value": null
++ },
++ {
++ "ns": "normalize",
++ "fn": "publisherDomain",
++ "args": [
++ ""
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:63",
++ "value": null
++ },
++ {
++ "ns": "normalize",
++ "fn": "terms",
++ "args": [
++ "Rockets launch into the sky, rockets everywhere"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:115",
++ "resultMode": "set",
++ "value": [
++ "everywhere",
++ "launch",
++ "rocket",
++ "sky"
++ ]
++ },
++ {
++ "ns": "normalize",
++ "fn": "entities",
++ "args": [
++ "President Smith met NASA in New York. The market rose."
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:130",
++ "resultMode": "set",
++ "value": [
++ "nasa",
++ "president smith",
++ "york"
++ ]
++ },
++ {
++ "ns": "normalize",
++ "fn": "shingles",
++ "args": [
++ "a quick brown fox jumps"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:159",
++ "resultMode": "set",
++ "value": [
++ "a quick",
++ "brown fox",
++ "fox jumps",
++ "quick brown"
++ ]
++ },
++ {
++ "ns": "normalize",
++ "fn": "jaccard",
++ "args": [
++ "a quick brown fox",
++ "a quick brown fox jumps"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:167",
++ "resultMode": "jaccardPair",
++ "value": 0.75
++ },
++ {
++ "ns": "normalize",
++ "fn": "stem",
++ "args": [
++ "rockets"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:108",
++ "value": "rocket"
++ },
++ {
++ "ns": "normalize",
++ "fn": "stem",
++ "args": [
++ "parties"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:108",
++ "value": "party"
++ },
++ {
++ "ns": "normalize",
++ "fn": "stem",
++ "args": [
++ "class"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:108",
++ "value": "class"
++ },
++ {
++ "ns": "normalize",
++ "fn": "stem",
++ "args": [
++ "bus"
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:108",
++ "value": "bus"
++ },
++ {
++ "ns": "normalize",
++ "fn": "licensedSummary",
++ "args": [
++ "A short summary that goes on and on and on beyond the limit.",
++ {
++ "termsNote": "Short summaries allowed with attribution."
++ }
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:184",
++ "value": {
++ "summary": "A short summary that goes on and on and on beyond the limit.",
++ "basis": "terms_allow_short_summaries"
++ }
++ },
++ {
++ "ns": "normalize",
++ "fn": "licensedSummary",
++ "args": [
++ "A short summary that goes on and on and on beyond the limit.",
++ {
++ "termsNote": "Headlines and links only."
++ }
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:184",
++ "value": {
++ "summary": null,
++ "basis": "terms_forbid_summaries"
++ }
++ },
++ {
++ "ns": "normalize",
++ "fn": "licensedSummary",
++ "args": [
++ "A short summary that goes on and on and on beyond the limit.",
++ {
++ "termsNote": "No summaries."
++ }
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:184",
++ "value": {
++ "summary": null,
++ "basis": "terms_forbid_summaries"
++ }
++ },
++ {
++ "ns": "normalize",
++ "fn": "licensedSummary",
++ "args": [
++ "A short summary that goes on and on and on beyond the limit.",
++ {
++ "licenseNote": "CC BY; summaries are allowed."
++ }
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:184",
++ "value": {
++ "summary": "A short summary that goes on and on and on beyond the limit.",
++ "basis": "terms_allow_short_summaries"
++ }
++ },
++ {
++ "ns": "normalize",
++ "fn": "licensedSummary",
++ "args": [
++ "A short summary that goes on and on and on beyond the limit.",
++ {}
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:184",
++ "value": {
++ "summary": null,
++ "basis": "terms_do_not_allow_summaries"
++ }
++ },
++ {
++ "ns": "normalize",
++ "fn": "licensedSummary",
++ "args": [
++ "A short summary that goes on and on and on beyond the limit.",
++ {
++ "termsNote": "Short summaries allowed.",
++ "maxChars": 0
++ }
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:184",
++ "value": {
++ "summary": null,
++ "basis": "summaries_disabled"
++ }
++ },
++ {
++ "ns": "normalize",
++ "fn": "licensedSummary",
++ "args": [
++ "A short summary that goes on and on and on beyond the limit.",
++ {
++ "termsNote": "Short summaries allowed.",
++ "maxChars": 20
++ }
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:184",
++ "value": {
++ "summary": "A short summary…",
++ "basis": "terms_allow_short_summaries"
++ }
++ },
++ {
++ "ns": "normalize",
++ "fn": "independentSources",
++ "args": [
++ [
++ {
++ "id": "a",
++ "source_key": "s1",
++ "canonical_url": "https://news.example.com/1",
++ "outlet": "Example",
++ "duplicate_of": null
++ },
++ {
++ "id": "b",
++ "source_key": "s2",
++ "canonical_url": "https://other.example.com/2",
++ "outlet": "Other",
++ "duplicate_of": null
++ },
++ {
++ "id": "c",
++ "source_key": "s3",
++ "canonical_url": "https://news.bbc.co.uk/3",
++ "outlet": "BBC",
++ "duplicate_of": "a"
++ }
++ ]
++ ],
++ "source": "OpenVibe.News/server/domain/text.js:83",
++ "resultMode": "groups",
++ "value": [
++ [
++ "a",
++ "b",
++ "c"
++ ]
++ ]
++ },
++ {
++ "ns": "normalize",
++ "fn": "normalizeUrl",
++ "args": [
++ "https://WWW.Example.com/a/b/?utm_source=x&b=2&a=1#frag"
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:39",
++ "value": "https://www.example.com/a/b/?a=1&b=2"
++ },
++ {
++ "ns": "normalize",
++ "fn": "normalizeUrl",
++ "args": [
++ "http://example.com/path"
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:39",
++ "value": "http://example.com/path"
++ },
++ {
++ "ns": "normalize",
++ "fn": "normalizeUrl",
++ "args": [
++ "https://example.com/"
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:39",
++ "value": "https://example.com/"
++ },
++ {
++ "ns": "normalize",
++ "fn": "normalizeUrl",
++ "args": [
++ "https://news.bbc.co.uk/x"
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:39",
++ "value": "https://news.bbc.co.uk/x"
++ },
++ {
++ "ns": "normalize",
++ "fn": "normalizeUrl",
++ "args": [
++ "ftp://example.com/x"
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:39",
++ "value": null
++ },
++ {
++ "ns": "normalize",
++ "fn": "normalizeUrl",
++ "args": [
++ "not a url"
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:39",
++ "value": null
++ },
++ {
++ "ns": "normalize",
++ "fn": "normalizeUrl",
++ "args": [
++ ""
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:39",
++ "value": null
++ },
++ {
++ "ns": "normalize",
++ "fn": "hostPathKey",
++ "args": [
++ "https://www.example.com/a/b/?a=1&b=2"
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:56",
++ "value": "example.com/a/b/?a=1&b=2"
++ },
++ {
++ "ns": "normalize",
++ "fn": "hostPathKey",
++ "args": [
++ "http://example.com/path"
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:56",
++ "value": "example.com/path"
++ },
++ {
++ "ns": "normalize",
++ "fn": "hostPathKey",
++ "args": [
++ "https://example.com/"
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:56",
++ "value": "example.com/"
++ },
++ {
++ "ns": "normalize",
++ "fn": "hostPathKey",
++ "args": [
++ "https://news.bbc.co.uk/x"
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:56",
++ "value": "news.bbc.co.uk/x"
++ },
++ {
++ "ns": "normalize",
++ "fn": "hostPathKey",
++ "args": [
++ null
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:56",
++ "value": null
++ },
++ {
++ "ns": "normalize",
++ "fn": "hostPathKey",
++ "args": [
++ null
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:56",
++ "value": null
++ },
++ {
++ "ns": "normalize",
++ "fn": "hostPathKey",
++ "args": [
++ null
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:56",
++ "value": null
++ },
++ {
++ "ns": "normalize",
++ "fn": "domainOf",
++ "args": [
++ "https://www.Example.com/x"
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:61",
++ "value": "example.com"
++ },
++ {
++ "ns": "normalize",
++ "fn": "domainOf",
++ "args": [
++ null
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:61",
++ "value": null
++ },
++ {
++ "ns": "normalize",
++ "fn": "domainOf",
++ "args": [
++ "x"
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:61",
++ "value": null
++ },
++ {
++ "ns": "normalize",
++ "fn": "slugify",
++ "args": [
++ "A Great Deal",
++ "off_01ABC"
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:65",
++ "value": "a-great-deal-01abc"
++ },
++ {
++ "ns": "normalize",
++ "fn": "parseAmount",
++ "args": [
++ "1,299.99",
++ "price"
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:77",
++ "value": {
++ "text": "1299.99",
++ "num": 1299.99
++ }
++ },
++ {
++ "ns": "normalize",
++ "fn": "parseAmount",
++ "args": [
++ "-3",
++ "price"
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:77",
++ "throws": {
++ "status": 422,
++ "code": "request.invalid",
++ "message": "price must be a plain amount like 19.99 (no currency symbols)"
++ }
++ },
++ {
++ "ns": "normalize",
++ "fn": "parseAmount",
++ "args": [
++ 12.5,
++ "price"
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:77",
++ "value": {
++ "text": "12.5",
++ "num": 12.5
++ }
++ },
++ {
++ "ns": "normalize",
++ "fn": "parseCurrency",
++ "args": [
++ "usd"
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:89",
++ "value": "USD"
++ },
++ {
++ "ns": "normalize",
++ "fn": "parseCurrency",
++ "args": [
++ "US"
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:89",
++ "throws": {
++ "status": 422,
++ "code": "request.invalid",
++ "message": "currency must be an ISO 4217 code such as USD or EUR"
++ }
++ },
++ {
++ "ns": "normalize",
++ "fn": "parseEnum",
++ "args": [
++ "NEW",
++ [
++ "new",
++ "used"
++ ],
++ "condition"
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:93",
++ "value": "new"
++ },
++ {
++ "ns": "normalize",
++ "fn": "parseEnum",
++ "args": [
++ "broken",
++ [
++ "new",
++ "used"
++ ],
++ "condition"
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:93",
++ "throws": {
++ "status": 422,
++ "code": "request.invalid",
++ "message": "condition must be one of new, used"
++ }
++ },
++ {
++ "ns": "normalize",
++ "fn": "parseInstant",
++ "args": [
++ "2026-09-20T10:00:00Z",
++ "at"
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:99",
++ "value": 1789898400000
++ },
++ {
++ "ns": "normalize",
++ "fn": "parseInstant",
++ "args": [
++ "nope",
++ "at"
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:99",
++ "throws": {
++ "status": 422,
++ "code": "request.invalid",
++ "message": "at must be a date"
++ }
++ },
++ {
++ "ns": "normalize",
++ "fn": "text",
++ "args": [
++ " a b ",
++ {
++ "field": "title",
++ "max": 5
++ }
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:111",
++ "value": "a b"
++ },
++ {
++ "ns": "normalize",
++ "fn": "longText",
++ "args": [
++ "a\r\nb \n",
++ 10
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:122",
++ "value": "a\nb"
++ },
++ {
++ "ns": "normalize",
++ "fn": "tokens",
++ "args": [
++ "Héllo, World 123"
++ ],
++ "source": "OpenVibe.Deals/server/domain/util.js:129",
++ "value": [
++ "héllo",
++ "world",
++ "123"
++ ]
++ },
++ {
++ "ns": "normalize",
++ "fn": "parseTime",
++ "args": [
++ "2026-09-20T10:00:00Z"
++ ],
++ "source": "OpenVibe.Trade/server/domain/util.js:16",
++ "value": 1789898400000
++ },
++ {
++ "ns": "normalize",
++ "fn": "parseTime",
++ "args": [
++ "2026-09-20"
++ ],
++ "source": "OpenVibe.Trade/server/domain/util.js:16",
++ "value": null
++ },
++ {
++ "ns": "normalize",
++ "fn": "parseTime",
++ "args": [
++ 1758362400000
++ ],
++ "source": "OpenVibe.Trade/server/domain/util.js:16",
++ "value": 1758362400000
++ },
++ {
++ "ns": "normalize",
++ "fn": "parseDecimal",
++ "args": [
++ "-12.5"
++ ],
++ "source": "OpenVibe.Trade/server/domain/util.js:28",
++ "value": {
++ "text": "-12.5",
++ "num": -12.5
++ }
++ },
++ {
++ "ns": "normalize",
++ "fn": "parseDecimal",
++ "args": [
++ "1e3"
++ ],
++ "source": "OpenVibe.Trade/server/domain/util.js:28",
++ "value": {
++ "text": "1e3",
++ "num": 1000
++ }
++ },
++ {
++ "ns": "normalize",
++ "fn": "parseDecimal",
++ "args": [
++ "abc"
++ ],
++ "source": "OpenVibe.Trade/server/domain/util.js:28",
++ "value": null
++ },
++ {
++ "ns": "normalize",
++ "fn": "str",
++ "args": [
++ " x ",
++ 5,
++ "name"
++ ],
++ "source": "OpenVibe.Trade/server/domain/util.js:42",
++ "value": "x"
++ },
++ {
++ "ns": "normalize",
++ "fn": "str",
++ "args": [
++ "xxxxxx",
++ 5,
++ "name"
++ ],
++ "source": "OpenVibe.Trade/server/domain/util.js:42",
++ "throws": {
++ "status": 422,
++ "code": "request.invalid",
++ "message": "name is longer than 5 characters"
++ }
++ },
++ {
++ "ns": "normalize",
++ "fn": "httpUrl",
++ "args": [
++ "https://example.com/x"
++ ],
++ "source": "OpenVibe.Trade/server/domain/util.js:54",
++ "value": "https://example.com/x"
++ },
++ {
++ "ns": "normalize",
++ "fn": "httpUrl",
++ "args": [
++ "ftp://example.com"
++ ],
++ "source": "OpenVibe.Trade/server/domain/util.js:54",
++ "throws": {
++ "status": 422,
++ "code": "request.invalid",
++ "message": "undefined must be http(s)"
++ }
++ },
++ {
++ "ns": "normalize",
++ "fn": "json",
++ "args": [
++ "{\"a\":1}",
++ {
++ "z": 0
++ }
++ ],
++ "source": "OpenVibe.Trade/server/domain/util.js:63",
++ "value": {
++ "a": 1
++ }
++ },
++ {
++ "ns": "normalize",
++ "fn": "json",
++ "args": [
++ "{oops",
++ {
++ "z": 0
++ }
++ ],
++ "source": "OpenVibe.Trade/server/domain/util.js:63",
++ "value": {
++ "z": 0
++ }
++ },
++ {
++ "ns": "normalize",
++ "fn": "alias",
++ "args": [
++ "name",
++ "ACME & Co."
++ ],
++ "source": "OpenVibe.Reviews/server/reviews/normalize.js:11 (via :59)",
++ "value": "acme and co"
++ },
++ {
++ "ns": "normalize",
++ "fn": "alias",
++ "args": [
++ "name",
++ " "
++ ],
++ "source": "OpenVibe.Reviews/server/reviews/normalize.js:11 (via :59)",
++ "throws": {
++ "status": 422,
++ "code": "alias.invalid",
++ "message": "A name needs at least one letter or digit"
++ }
++ },
++ {
++ "ns": "normalize",
++ "fn": "alias",
++ "args": [
++ "url",
++ "https://WWW.Example.com/a?utm_source=x&b=2"
++ ],
++ "source": "OpenVibe.Reviews/server/reviews/normalize.js:19 (via :59)",
++ "value": "example.com/a?b=2"
++ },
++ {
++ "ns": "normalize",
++ "fn": "alias",
++ "args": [
++ "url",
++ "not a url"
++ ],
++ "source": "OpenVibe.Reviews/server/reviews/normalize.js:19 (via :59)",
++ "throws": {
++ "status": 422,
++ "code": "alias.invalid",
++ "message": "A URL alias must be an absolute http(s) URL"
++ }
++ },
++ {
++ "ns": "normalize",
++ "fn": "alias",
++ "args": [
++ "sku",
++ " ab c "
++ ],
++ "source": "OpenVibe.Reviews/server/reviews/normalize.js:31 (via :59)",
++ "value": "ABC"
++ },
++ {
++ "ns": "normalize",
++ "fn": "alias",
++ "args": [
++ "gtin",
++ "12345678"
++ ],
++ "source": "OpenVibe.Reviews/server/reviews/normalize.js:38 (via :59)",
++ "value": "00000012345678"
++ },
++ {
++ "ns": "normalize",
++ "fn": "alias",
++ "args": [
++ "source",
++ "Staff-Codes"
++ ],
++ "source": "OpenVibe.Reviews/server/reviews/normalize.js:44 (via :59)",
++ "value": "staff-codes"
++ },
++ {
++ "ns": "normalize",
++ "fn": "alias",
++ "args": [
++ "external",
++ "Steam_App:620"
++ ],
++ "source": "OpenVibe.Reviews/server/reviews/normalize.js:51 (via :59)",
++ "value": "steam_app:620"
++ },
++ {
++ "ns": "normalize",
++ "fn": "alias",
++ "args": [
++ "nope",
++ "x"
++ ],
++ "source": "OpenVibe.Reviews/server/reviews/normalize.js:59",
++ "throws": {
++ "status": 422,
++ "code": "alias.invalid",
++ "message": "alias type must be one of name, url, sku, mpn, gtin, source, external"
++ }
++ },
++ {
++ "ns": "normalize",
++ "fn": "tryAlias",
++ "args": [
++ "gtin",
++ "nope"
++ ],
++ "source": "OpenVibe.Reviews/server/reviews/normalize.js:66",
++ "value": null
++ },
++ {
++ "ns": "normalize",
++ "fn": "slug",
++ "args": [
++ "Ünïcode & Things"
++ ],
++ "source": "OpenVibe.Reviews/server/reviews/normalize.js:71",
++ "value": "unicode-and-things"
++ },
++ {
++ "ns": "hosts",
++ "fn": "normalizeHost",
++ "args": [
++ "WWW.Example-Shop.co.uk."
++ ],
++ "source": "OpenVibe.Coupons/server/domain/hosts.js:36",
++ "value": "www.example-shop.co.uk"
++ },
++ {
++ "ns": "hosts",
++ "fn": "normalizeHost",
++ "args": [
++ "bücher.example"
++ ],
++ "source": "OpenVibe.Coupons/server/domain/hosts.js:36",
++ "throws": {
++ "status": 400,
++ "code": "host.invalid",
++ "message": "special-use names are not merchants"
++ }
++ },
++ {
++ "ns": "hosts",
++ "fn": "normalizeHost",
++ "args": [
++ "localhost"
++ ],
++ "source": "OpenVibe.Coupons/server/domain/hosts.js:36",
++ "throws": {
++ "status": 400,
++ "code": "host.invalid",
++ "message": "host needs at least two labels"
++ }
++ },
++ {
++ "ns": "hosts",
++ "fn": "normalizeHost",
++ "args": [
++ "news.bbc.co.uk"
++ ],
++ "source": "OpenVibe.Coupons/server/domain/hosts.js:36",
++ "value": "news.bbc.co.uk"
++ },
++ {
++ "ns": "hosts",
++ "fn": "normalizeHost",
++ "args": [
++ "a.b.myshopify.com"
++ ],
++ "source": "OpenVibe.Coupons/server/domain/hosts.js:36",
++ "value": "a.b.myshopify.com"
++ },
++ {
++ "ns": "hosts",
++ "fn": "hostOfUrl",
++ "args": [
++ "https://www.Example.com/x/y"
++ ],
++ "source": "OpenVibe.Coupons/server/domain/hosts.js:69",
++ "value": {
++ "host": "www.example.com",
++ "path": "/x/y"
++ }
++ },
++ {
++ "ns": "hosts",
++ "fn": "hostOfUrl",
++ "args": [
++ "ftp://example.com"
++ ],
++ "source": "OpenVibe.Coupons/server/domain/hosts.js:69",
++ "value": null
++ },
++ {
++ "ns": "hosts",
++ "fn": "normalizePathPrefix",
++ "args": [
++ "/Shop/x/"
++ ],
++ "source": "OpenVibe.Coupons/server/domain/hosts.js:78",
++ "value": "/Shop/x"
++ },
++ {
++ "ns": "hosts",
++ "fn": "checkRule",
++ "args": [
++ {
++ "host": "shop.example.co.uk",
++ "path_prefix": "/deals"
++ }
++ ],
++ "source": "OpenVibe.Coupons/server/domain/hosts.js:96",
++ "value": {
++ "host": "shop.example.co.uk",
++ "registrable": "example.co.uk",
++ "path_prefix": "/deals",
++ "include_subdomains": 1
++ }
++ },
++ {
++ "ns": "hosts",
++ "fn": "checkRule",
++ "args": [
++ {
++ "host": "co.uk"
++ }
++ ],
++ "source": "OpenVibe.Coupons/server/domain/hosts.js:96",
++ "throws": {
++ "status": 400,
++ "code": "host.invalid",
++ "message": "co.uk is a public suffix (co.uk), not a site"
++ }
++ },
++ {
++ "ns": "hosts",
++ "fn": "registrableDomain",
++ "args": [
++ "www.example-shop.co.uk"
++ ],
++ "source": "OpenVibe.Coupons/server/domain/psl.js:93",
++ "value": "example-shop.co.uk"
++ },
++ {
++ "ns": "hosts",
++ "fn": "registrableDomain",
++ "args": [
++ "news.bbc.co.uk"
++ ],
++ "source": "OpenVibe.Coupons/server/domain/psl.js:93",
++ "value": "bbc.co.uk"
++ },
++ {
++ "ns": "hosts",
++ "fn": "registrableDomain",
++ "args": [
++ "co.uk"
++ ],
++ "source": "OpenVibe.Coupons/server/domain/psl.js:93",
++ "value": null
++ },
++ {
++ "ns": "hosts",
++ "fn": "registrableDomain",
++ "args": [
++ "a.b.myshopify.com"
++ ],
++ "source": "OpenVibe.Coupons/server/domain/psl.js:93",
++ "value": "b.myshopify.com"
++ },
++ {
++ "ns": "hosts",
++ "fn": "registrableDomain",
++ "args": [
++ "foo.github.io"
++ ],
++ "source": "OpenVibe.Coupons/server/domain/psl.js:93",
++ "value": "foo.github.io"
++ },
++ {
++ "ns": "hosts",
++ "fn": "publicSuffix",
++ "args": [
++ "www.example-shop.co.uk"
++ ],
++ "source": "OpenVibe.Coupons/server/domain/psl.js:87",
++ "value": "co.uk"
++ },
++ {
++ "ns": "hosts",
++ "fn": "publicSuffix",
++ "args": [
++ "news.bbc.co.uk"
++ ],
++ "source": "OpenVibe.Coupons/server/domain/psl.js:87",
++ "value": "co.uk"
++ }
++ ]
++}
\\ No newline at end of file
ddiff --git a/test/ingest.test.js b/test/ingest.test.js
nnew file mode 100644
iindex 0000000..5e5bad4
---- /dev/null
++++ b/test/ingest.test.js
@@@ -0,0 +1,197 @@
++'use strict';
++/**
++ * openvibe-publishing/ingest (1.1.0): the shared ingest chassis.
++ *
++ * - normalize/hosts byte-for-byte against fixtures captured from the five current product copies
++ * (test/fixtures/normalize.json; each case cites its source file:line);
++ * - createSourcesClient against an in-process stub of OpenVibe.Sources (fake keys made at runtime);
++ * - createChangeCursor/pullChanges: cursor advancement, per-item savepoint isolation, hold;
++ * - createEventConsumer: signature window and inbox exactly-once.
++ */
++const assert = require('assert');
++const crypto = require('crypto');
++const http = require('http');
++const { openDb, fakeClock, suite, sql } = require('./helpers/db');
++const ingest = require('../lib/ingest');
++const { signDeliveryHeaders } = require('openvibe-sdk/events');
++
++const { test, run } = suite();
++const FIXTURES = require('./fixtures/normalize.json');
++
++test('normalize/hosts match the five current copies byte-for-byte (fixtures, source file:line cited)', () => {
++ let checked = 0;
++ for (const c of FIXTURES.cases) {
++ const fn = ingest[c.ns][c.fn];
++ assert.strictEqual(typeof fn, 'function', `${c.ns}.${c.fn} (source ${c.source})`);
++ if (c.resultMode === 'jaccardPair') {
++ const got = fn(ingest.normalize.shingles(c.args[0]), ingest.normalize.shingles(c.args[1]));
++ assert.strictEqual(got, c.value, `${c.fn}(${JSON.stringify(c.args)}) [${c.source}]`);
++ } else if (c.throws) {
++ assert.throws(
++ () => fn(...c.args),
++ (e) => e.status === c.throws.status && e.code === c.throws.code && e.message === c.throws.message,
++ `${c.ns}.${c.fn}(${JSON.stringify(c.args)}) should throw ${JSON.stringify(c.throws)} [${c.source}]`,
++ );
++ } else {
++ let got = fn(...c.args);
++ if (c.resultMode === 'set') got = [...got].sort();
++ if (c.resultMode === 'groups') got = got.map((g) => [...g].sort()).sort((a, b) => a.join().localeCompare(b.join()));
++ assert.deepStrictEqual(got, c.value, `${c.ns}.${c.fn}(${JSON.stringify(c.args)}) [${c.source}]`);
++ }
++ checked++;
++ }
++ assert.strictEqual(checked, FIXTURES.cases.length);
++ assert.ok(checked > 100, `only ${checked} fixture cases`);
++});
++
++test('freshness.verdict is the generic staleness rule: fresh inside the window, stale after, never invented', () => {
++ const def = 3600;
++ const now = Date.parse('2026-09-22T12:00:00Z');
++ assert.deepStrictEqual(ingest.freshness.verdict(null, now, def), { known: false, stale: true, staleSince: null, window: def, lastSuccessAt: null });
++ const fresh = ingest.freshness.verdict({ last_success_at: now - 60 * 1000, stale_after_sec: null }, now, def);
++ assert.deepStrictEqual([fresh.stale, fresh.window, fresh.staleSince], [false, def, null]);
++ const stale = ingest.freshness.verdict({ last_success_at: now - 2 * 3600 * 1000, stale_after_sec: null, stale_since: null }, now, def);
++ assert.strictEqual(stale.stale, true);
++ assert.strictEqual(stale.staleSince, now - 2 * 3600 * 1000 + def * 1000);
++ const never = ingest.freshness.verdict({ last_success_at: null, stale_since: 5 }, now, def);
++ assert.deepStrictEqual([never.known, never.stale, never.lastSuccessAt], [true, true, null]);
++ assert.strictEqual(ingest.freshness.view(null, 'k', now, def).status, 'unknown');
++ assert.strictEqual(ingest.freshness.view(null, 'k', now, def).last_success_at, null);
++});
++
++// ── Sources client against a stub server ──────────────────────────────────────
++
++function listen(handler) {
++ return new Promise((resolve) => {
++ const server = http.createServer(handler).listen(0, '127.0.0.1', () => {
++ const base = `http://127.0.0.1:${server.address().port}`;
++ resolve({ base, close: () => new Promise((r) => server.close(r)) });
++ });
++ });
++}
++
++/** A stub OpenVibe.Sources: token endpoint + items / items/:id / sources / sources/:key. */
++async function stubSources() {
++ const seen = { tokens: 0, items: 0, item: 0, sources: 0, source: 0 };
++ const server = await listen((req, res) => {
++ const url = new URL(req.url, 'http://x');
++ const send = (status, body, type = 'application/json') => { res.writeHead(status, { 'Content-Type': type }); res.end(typeof body === 'string' ? body : JSON.stringify(body)); };
++ if (url.pathname === '/oauth/token') { seen.tokens++; return send(200, { access_token: 'tok', token_type: 'Bearer', expires_in: 300 }); }
++ if (url.pathname === '/api/v1/items') {
++ seen.items++;
++ if (!/^Bearer /.test(req.headers.authorization || '')) return send(401, { code: 'unauthorized' });
++ const cat = url.searchParams.get('category');
++ if (cat === 'boom') return send(500, { code: 'sources.error' });
++ if (cat === 'bad') return send(200, 'not json', 'text/plain');
++ return send(200, { items: [{ id: 'itm_01J8Z6Q3KX0000000000000000', revision: 1, source_key: 's1', canonical_url: 'https://example.com/a' }], next_after: 5, more: false, sources: { s1: { status: 'ok', stale: false, last_success_at: '2026-09-20T10:00:00Z' } } });
++ }
++ if (/^\/api\/v1\/items\/.+/.test(url.pathname)) { seen.item++; return send(200, { item: { id: decodeURIComponent(url.pathname.split('/').pop()) }, source: { key: 's1' } }); }
++ if (url.pathname === '/api/v1/sources') { seen.sources++; return send(200, { sources: [{ key: 's1', category: 'reviews', health: { status: 'ok' } }] }); }
++ if (/^\/api\/v1\/sources\/.+/.test(url.pathname)) { seen.source++; return send(200, { source: { key: url.pathname.split('/').pop(), name: 'S1' } }); }
++ send(404, { code: 'route.not_found' });
++ });
++ return { seen, server };
++}
++
++const creds = () => ({ clientId: 'svc:reviews', clientSecret: crypto.randomBytes(24).toString('hex') });
++
++test('createSourcesClient: items/item/sources, one token per scope, stable error codes', async () => {
++ const { seen, server } = await stubSources();
++ try {
++ const config = { networkInternalUrl: server.base, oauth: creds(), sources: { internalUrl: server.base, category: 'reviews', timeoutMs: 2000 } };
++ const src = ingest.createSourcesClient({ config });
++ assert.strictEqual(src.enabled, true);
++ const page = await src.listItems({ after: 0, limit: 50 });
++ assert.strictEqual(page.next_after, 5);
++ assert.strictEqual(page.items[0].id, 'itm_01J8Z6Q3KX0000000000000000');
++ assert.strictEqual((await src.getItem('itm_01J8Z6Q3KX0000000000000000')).item.id, 'itm_01J8Z6Q3KX0000000000000000');
++ assert.strictEqual((await src.listSources()).sources[0].key, 's1');
++ assert.strictEqual((await src.getSource('s1')).name, 'S1');
++ assert.strictEqual(seen.items, 1);
++ assert.strictEqual(seen.tokens, 2, 'one cached token per scope (items, sources), reused at the second call');
++
++ const boom = ingest.createSourcesClient({ config: { ...config, oauth: creds(), sources: { ...config.sources, category: 'boom' } } });
++ await assert.rejects(boom.listItems({}), (e) => e instanceof ingest.SourcesError && e.code === 'sources.http_500' && e.status === 500);
++ const bad = ingest.createSourcesClient({ config: { ...config, oauth: creds(), sources: { ...config.sources, category: 'bad' } } });
++ await assert.rejects(bad.listItems({}), (e) => e.code === 'sources.bad_response');
++ } finally { await server.close(); }
++});
++
++test('createSourcesClient: not configured is a stable code, never an invented page', async () => {
++ const off = ingest.createSourcesClient({ config: { sources: { internalUrl: 'http://127.0.0.1:1', category: 'reviews' } } });
++ assert.strictEqual(off.enabled, false);
++ await assert.rejects(off.listItems({}), (e) => e instanceof ingest.SourcesError && e.code === 'sources.not_configured');
++});
++
++// ── Change cursor + pull ──────────────────────────────────────────────────────
++
++test('pullChanges: one page transaction, per-item savepoint isolation, hold does not stall the cursor', async () => {
++ const db = await openDb();
++ const cursor = await ingest.createChangeCursor(db, { prefix: 't', now: fakeClock() }).ensureSchema();
++ assert.strictEqual(cursor.table, 't_ingest_cursor');
++ await db.query('CREATE TABLE t_seen (item text PRIMARY KEY)');
++ const pages = [
++ { items: [{ id: 'a' }, { id: 'b' }, { id: 'c' }, { id: 'd' }], next_after: 10, more: true },
++ { items: [{ id: 'e' }], next_after: 12, more: false },
++ ];
++ let p = 0;
++ const source = { async listItems() { return pages[p++] || { items: [], next_after: 12, more: false }; } };
++ const apply = async (item, t) => {
++ await t.exec(sql`INSERT INTO t_seen (item) VALUES (${item.id})`);
++ if (item.id === 'b') throw new Error('unreadable item'); // its savepoint rolls back alone
++ if (item.id === 'c') return 'hold'; // no merchant yet: counted, cursor moves on
++ if (item.id === 'd') return { outcome: 'removed' }; // the object form is accepted too
++ return 'applied';
++ };
++ const summary = await ingest.pullChanges({ db, cursor, source, apply, pageSize: 10 });
++ assert.deepStrictEqual(summary, { pages: 2, applied: 2, hold: 1, removed: 1, failed: 1, after: 12 });
++ assert.strictEqual(await cursor.get('default'), 12, 'the cursor advanced past both pages');
++ assert.deepStrictEqual((await db.many(sql`SELECT item FROM t_seen ORDER BY item`)).map((r) => r.item), ['a', 'c', 'd', 'e'], 'the bad item b rolled back; every other item committed');
++ assert.strictEqual((await ingest.pullChanges({ db, cursor, source, apply })).pages, 1, 'a replay from the advanced cursor finds nothing new');
++ assert.strictEqual(await cursor.get('default'), 12);
++ const bad = await ingest.pullChanges({ db, cursor, source: { async listItems() { return { items: [{ id: 'f' }], next_after: 13, more: false }; } }, apply: async () => 'created' });
++ assert.deepStrictEqual(bad, { pages: 1, applied: 0, hold: 0, removed: 0, failed: 1, after: 13 }, 'a value outside the contract is isolated as failed');
++});
++
++// ── Event consumer ────────────────────────────────────────────────────────────
++
++const EVENT_ID = 'evt_01J8Z6Q3KX0000000000000000';
++const ITEM_ID = 'itm_01J8Z6Q3KX0000000000000000';
++function delivery(event, secret, now = Date.now()) {
++ const raw = Buffer.from(JSON.stringify({ event, seq: 1 }));
++ return { raw, headers: signDeliveryHeaders(raw, secret, { now }) };
++}
++const envelope = (id = EVENT_ID) => ({
++ event_id: id, event_type: 'sources.item.created', version: 1, source: 'sources',
++ actor: { type: 'service', id: 'sources' }, timestamp: '2026-09-22T12:00:00.000Z', visibility: 'internal',
++ subject: { type: 'item', id: ITEM_ID }, payload: { category: 'reviews', item_id: ITEM_ID },
++});
++
++test('createEventConsumer: v2 signature within ±300 s, exactly once through the inbox', async () => {
++ const db = await openDb();
++ const secret = crypto.randomBytes(32).toString('hex');
++ const other = crypto.randomBytes(32).toString('hex');
++ const consumer = ingest.createEventConsumer({ db, secrets: [secret], consumer: 'reviews-sources' });
++ await consumer.inbox.ensureSchema();
++ let calls = 0;
++ const a = delivery(envelope(), secret);
++ const first = await consumer.apply(a.raw, a.headers, async (event) => { calls++; return `saw:${event.payload.category}`; });
++ assert.deepStrictEqual([first.status, first.event_id, first.duplicate, first.outcome], [200, EVENT_ID, false, 'saw:reviews']);
++ const again = await consumer.apply(a.raw, a.headers, async () => { calls++; return 'no'; });
++ assert.deepStrictEqual([again.duplicate, again.outcome, calls], [true, null, 1], 'the handler runs once per event_id');
++
++ const bad = delivery(envelope('evt_01J8Z6Q3KX0000000000000001'), other);
++ assert.strictEqual((await consumer.apply(bad.raw, bad.headers, async () => 'no')).status, 401);
++ const stale = delivery(envelope('evt_01J8Z6Q3KX0000000000000002'), secret, Date.now() - 10 * 60 * 1000);
++ assert.strictEqual((await consumer.apply(stale.raw, stale.headers, async () => 'no')).status, 401, 'outside the ±300 s window');
++ const malformed = delivery({ ...envelope('nope'), event_id: 'nope' }, secret);
++ assert.strictEqual((await consumer.apply(malformed.raw, malformed.headers, async () => 'no')).status, 400);
++});
++
++test('createEventConsumer: no secrets configured is 503, never an accepted delivery', async () => {
++ const db = await openDb();
++ const consumer = ingest.createEventConsumer({ db, secrets: [], consumer: 'x-sources' });
++ assert.deepStrictEqual(await consumer.apply(Buffer.from('{}'), {}, async () => 'no'), { status: 503, code: 'ingest.webhook_disabled' });
++});
++
++run();
ddiff --git a/test/package.test.js b/test/package.test.js
iindex a630ac4..f6f9151 100644
---- a/test/package.test.js
++++ b/test/package.test.js
@@@ -12,7 +12,7 @@ const SUBPATHS = Object.keys(pkg.exports).filter((k) => k !== '.' && k !== './pa
test('every subpath export resolves to an existing module', () => {
assert.deepStrictEqual(SUBPATHS.map((s) => s.slice(2)).sort(),
-- ['ai', 'authorship', 'citations', 'discussion', 'index-hooks', 'media', 'revisions', 'schedule', 'seo', 'ssr', 'taxonomy']);
++ ['ai', 'authorship', 'citations', 'discussion', 'index-hooks', 'ingest', 'media', 'publication', 'revisions', 'schedule', 'seo', 'ssr', 'taxonomy']);
for (const s of SUBPATHS) assert.ok(require(`openvibe-publishing/${s.slice(2)}`), s);
});
@@@ -37,7 +37,7 @@ test('the root export is lazy and exposes every module', () => {
assert.strictEqual(out.before, 0);
assert.ok(out.after > 0);
assert.strictEqual(out.v, pkg.version);
-- assert.deepStrictEqual(out.names.sort(), ['PublishingError', 'ai', 'authorship', 'citations', 'discussion', 'indexHooks', 'media', 'revisions', 'schedule', 'schema', 'seo', 'ssr', 'taxonomy', 'version']);
++ assert.deepStrictEqual(out.names.sort(), ['PublishingError', 'ai', 'authorship', 'citations', 'discussion', 'indexHooks', 'ingest', 'media', 'publication', 'revisions', 'schedule', 'schema', 'seo', 'ssr', 'taxonomy', 'version']);
});
test('the package owns no runtime: no listen(), no database of its own, no env config in lib/', () => {
ddiff --git a/test/publication.test.js b/test/publication.test.js
nnew file mode 100644
iindex 0000000..3a15265
---- /dev/null
++++ b/test/publication.test.js
@@@ -0,0 +1,173 @@
++'use strict';
++/**
++ * openvibe-publishing/publication (1.1.0): the shared publication glue.
++ *
++ * - createPublication stamps + enqueues on the CALLER's transaction handle: the sequencer revision and
++ * the outbox row commit together or roll back together (ADR-004);
++ * - it emits exactly search.index-document@1 (carried by <owner>.index_document.upserted|deleted,
++ * checked the way OpenVibe.Search's webhook reads it) and valid <owner>.<type>.<action> events;
++ * - tombstones on unpublish/merge, and nothing for a resource that was never indexed;
++ * - the IndexNow ping for an indexable page that appeared/changed or a page that went away.
++ */
++const assert = require('assert');
++const contracts = require('openvibe-contracts');
++const { createPgOutbox } = require('openvibe-sdk/events');
++const { openDb, fakeClock, suite, sql } = require('./helpers/db');
++const hooks = require('../lib/index-hooks');
++const seo = require('../lib/seo');
++const { createIndexSequencer } = require('../lib/index-hooks');
++const { createPublication } = require('../lib/publication');
++
++const { test, run } = suite();
++const USER = 'usr_01J8Z6Q3KX0000000000000000';
++
++const valid = (doc) => {
++ const v = contracts.validate('search.index-document@1', doc);
++ assert.ok(v.valid, `search.index-document@1: ${JSON.stringify(v.errors)} in ${JSON.stringify(doc)}`);
++ return doc;
++};
++const validEnvelope = (env) => {
++ const v = contracts.validate('events.event-envelope@1', { ...env, event_id: env.event_id || contracts.ids.newId('event') });
++ assert.ok(v.valid, `events.event-envelope@1: ${JSON.stringify(v.errors)}`);
++ return env;
++};
++/** What OpenVibe.Search's webhook (server/api/webhook.js documentFromEvent) does with an envelope. */
++function searchReads(env) {
++ const m = /^([a-z][a-z0-9-]{1,39})\.index_document\.(upserted|deleted)$/.exec(env.event_type);
++ assert.ok(m, `not an index_document event: ${env.event_type}`);
++ assert.strictEqual(m[1], env.source, 'event_type owner must be the source');
++ const p = env.payload;
++ if (p.owner !== undefined) assert.strictEqual(p.owner, env.source);
++ const doc = m[2] === 'deleted' ? { owner: env.source, type: p.type, id: p.id, revision: p.revision, deleted: true } : { ...p, owner: env.source };
++ assert.strictEqual(env.subject.type, doc.type);
++ assert.strictEqual(env.subject.id, doc.id);
++ assert.strictEqual(env.subject.revision, doc.revision);
++ assert.strictEqual(env.visibility, 'internal');
++ return valid(doc);
++}
++
++const facts = { state: 'published', visibility: 'public', canonicalUrl: 'https://openvibe.wiki/p/rye', wordCount: 400, citationCount: 2 };
++const decisionFor = (extra = {}) => seo.evaluate({ ...facts, ...extra }, { policy: { requireSources: true } });
++const input = (extra = {}) => ({
++ owner: 'wiki', type: 'page', id: 'pg_1', revision: 0, state: 'published', visibility: 'public',
++ canonicalUrl: 'https://openvibe.wiki/p/rye', title: 'Rye', summary: 'A cereal grass.', body: 'Rye is a grass grown as a grain.',
++ facets: { space: 'food' }, authorship: { mode: 'human' },
++ decision: decisionFor(), publishedAt: '2026-09-20T12:00:00Z', updatedAt: '2026-09-21T12:00:00Z', language: 'en', ...extra,
++});
++
++/** A publication wired to a real pg outbox with a stub Events client (only enqueue is exercised). */
++async function setup() {
++ const db = await openDb();
++ const clock = fakeClock();
++ const sequencer = await createIndexSequencer(db, { prefix: 'x', now: clock }).ensureSchema();
++ const events = { prepare: (env) => ({ ...env, event_id: contracts.ids.newId('event') }), publish: async () => ({ results: [] }) };
++ const outbox = createPgOutbox(db, { events, table: 'x_outbox', now: clock });
++ await outbox.ensureSchema();
++ const indexnow = { enabled: true, pings: [], pingSoon(urls) { this.pings.push(urls); } };
++ const publication = createPublication({ owner: 'wiki', sequencer, outbox, baseUrl: 'https://openvibe.wiki', indexnow, now: clock });
++ const envelopes = () => db.many(sql`SELECT envelope FROM x_outbox ORDER BY id`);
++ return { db, clock, sequencer, outbox, indexnow, publication, envelopes };
++}
++
++test('index: the sequencer revision and the Search event commit on the caller\'s handle, exactly search.index-document@1', async () => {
++ const { db, sequencer, publication, envelopes, indexnow } = await setup();
++ const doc = hooks.buildIndexDocument(input());
++ let stamped;
++ await db.tx(async (t) => { stamped = await publication.index(t, { document: doc, page: '/p/rye' }); });
++ assert.strictEqual(stamped.revision, 1);
++ assert.strictEqual(await sequencer.current('wiki', 'page', 'pg_1'), 1);
++ const rows = await envelopes();
++ assert.strictEqual(rows.length, 1);
++ const env = validEnvelope(rows[0].envelope);
++ assert.strictEqual(env.event_type, 'wiki.index_document.upserted');
++ assert.deepStrictEqual(searchReads(env), stamped);
++ assert.deepStrictEqual(indexnow.pings, [['https://openvibe.wiki/p/rye', 'https://openvibe.wiki/sitemap.xml']]);
++ // An unchanged document is a no-op: same revision, no event, no ping.
++ indexnow.pings.length = 0;
++ const again = await db.tx(async (t) => await publication.index(t, { document: doc }));
++ assert.strictEqual(again, null);
++ assert.strictEqual((await envelopes()).length, 1);
++ assert.deepStrictEqual(indexnow.pings, []);
++});
++
++test('index: a failed transaction rolls back the sequencer revision and its event together', async () => {
++ const { db, sequencer, publication, envelopes } = await setup();
++ const doc = hooks.buildIndexDocument(input());
++ await db.tx(async (t) => { await publication.index(t, { document: doc }); });
++ assert.strictEqual(await sequencer.current('wiki', 'page', 'pg_1'), 1);
++ const changed = hooks.buildIndexDocument(input({ title: 'Rye (revised)' }));
++ await assert.rejects(db.tx(async (t) => { await publication.index(t, { document: changed }); throw new Error('the product write failed'); }), /the product write failed/);
++ assert.strictEqual(await sequencer.current('wiki', 'page', 'pg_1'), 1, 'the stamp rolled back with the product write');
++ assert.strictEqual((await envelopes()).length, 1, 'no event survived the rollback');
++ await assert.rejects(publication.index(doc), /transaction handle/);
++});
++
++test('tombstones on unpublish and merge; never-indexed resources send nothing', async () => {
++ const { db, publication, envelopes, indexnow } = await setup();
++ const doc = hooks.buildIndexDocument(input());
++ await db.tx(async (t) => { await publication.index(t, { document: doc, page: '/p/rye' }); });
++ indexnow.pings.length = 0;
++ const tomb = await db.tx(async (t) => await publication.tombstone(t, { type: 'page', id: 'pg_1', page: '/p/rye' }));
++ assert.strictEqual(tomb.revision, 2);
++ const rows = await envelopes();
++ const del = validEnvelope(rows[rows.length - 1].envelope);
++ assert.strictEqual(del.event_type, 'wiki.index_document.deleted');
++ assert.deepStrictEqual(del.payload, { type: 'page', id: 'pg_1', revision: 2 });
++ assert.deepStrictEqual(searchReads(del), { owner: 'wiki', type: 'page', id: 'pg_1', revision: 2, deleted: true });
++ assert.deepStrictEqual(indexnow.pings, [['https://openvibe.wiki/p/rye', 'https://openvibe.wiki/sitemap.xml']], 'a page Search had that went away is announced');
++ // A second tombstone is a no-op.
++ assert.strictEqual(await db.tx(async (t) => await publication.tombstone(t, { type: 'page', id: 'pg_1', page: '/p/rye' })), null);
++ // A merge: the duplicate had been indexed, so its page is removed and announced.
++ await db.tx(async (t) => { await publication.index(t, { document: hooks.buildIndexDocument(input({ id: 'pg_2', canonicalUrl: 'https://openvibe.wiki/p/rye-2' })), page: '/p/rye-2' }); });
++ const before = (await envelopes()).length;
++ const merged = await db.tx(async (t) => await publication.tombstone(t, { type: 'page', id: 'pg_2', page: '/p/rye-2' }));
++ assert.strictEqual(merged.revision, 2);
++ assert.strictEqual((await envelopes()).length, before + 1);
++ // A resource that was never indexed gets no tombstone.
++ assert.strictEqual(await db.tx(async (t) => await publication.tombstone(t, { type: 'page', id: 'pg_nope' })), null);
++});
++
++test('noindex documents are sent (Search keeps the noindex copy) but never pinged', async () => {
++ const { db, publication, envelopes, indexnow } = await setup();
++ const decision = decisionFor({ wordCount: 3, citationCount: 0 });
++ assert.strictEqual(decision.indexable, false);
++ const doc = hooks.buildIndexDocument(input({ id: 'pg_3', decision }));
++ await db.tx(async (t) => { await publication.index(t, { document: doc, page: '/p/thin' }); });
++ assert.strictEqual((await envelopes()).length, 1, 'the noindex document still reaches Search');
++ assert.deepStrictEqual(indexnow.pings, [], 'a noindex page is never announced');
++});
++
++test('publication events <owner>.<type>.<action> are valid and public only when listable', async () => {
++ const { db, publication, envelopes } = await setup();
++ const doc = hooks.buildIndexDocument(input());
++ await db.tx(async (t) => {
++ await publication.publication(t, { type: 'page', action: 'published', id: 'pg_1', revision: 1, document: doc, decision: decisionFor(), actor: USER });
++ });
++ const env = validEnvelope((await envelopes())[0].envelope);
++ assert.strictEqual(env.event_type, 'wiki.page.published');
++ assert.strictEqual(env.visibility, 'public');
++ assert.deepStrictEqual(env.payload, { canonical_url: 'https://openvibe.wiki/p/rye', publication_state: 'published', indexability: { decision: 'index', reasons: [] } });
++ const tomb = hooks.tombstone({ owner: 'wiki', type: 'page', id: 'pg_1', revision: 2 });
++ await db.tx(async (t) => {
++ await publication.publication(t, { type: 'page', action: 'unpublished', id: 'pg_1', revision: 2, document: tomb, actor: 'svc:wiki' });
++ });
++ const del = validEnvelope((await envelopes())[1].envelope);
++ assert.strictEqual(del.event_type, 'wiki.page.unpublished');
++ assert.strictEqual(del.visibility, 'internal');
++ assert.strictEqual(del.payload.publication_state, 'unpublished');
++});
++
++test('sync runs gate -> document -> stamp -> event in the caller\'s transaction', async () => {
++ const { db, sequencer, publication, envelopes } = await setup();
++ const entity = { id: 'pg_9', slug: 'nine' };
++ const decide = () => seo.evaluate({ state: 'published', visibility: 'public', canonicalUrl: 'https://openvibe.wiki/p/nine', wordCount: 400, citationCount: 2 }, { policy: { requireSources: true } });
++ const document = (e, decision) => hooks.buildIndexDocument({ owner: 'wiki', type: 'page', id: e.id, revision: 0, state: 'published', visibility: 'public', canonicalUrl: 'https://openvibe.wiki/p/nine', title: 'Nine', decision });
++ const pub = createPublication({ owner: 'wiki', sequencer, outbox: publication.outbox, baseUrl: 'https://openvibe.wiki', now: () => Date.now(), decide, document, page: (e) => `/p/${e.slug}` });
++ let stamped;
++ await db.tx(async (t) => { stamped = await pub.sync(t, entity); });
++ assert.strictEqual(stamped.revision, 1);
++ assert.strictEqual(await sequencer.current('wiki', 'page', 'pg_9'), 1);
++ assert.strictEqual((await envelopes()).length, 1);
++});
++
++run();
ddiff --git a/test/schema.test.js b/test/schema.test.js
iindex 4a2e19e..4ce26da 100644
---- a/test/schema.test.js
++++ b/test/schema.test.js
@@@ -20,9 +20,10 @@ const { createReviewLog } = require('../lib/authorship');
const { createRedirectStore } = require('../lib/seo');
const { createDiscussionRefs } = require('../lib/discussion');
const { createIndexSequencer, tombstone } = require('../lib/index-hooks');
++const { createChangeCursor } = require('../lib/ingest');
const { test, run } = suite();
--const MODULES = ['revisions', 'citations', 'media', 'discussion', 'schedule', 'authorship', 'taxonomy', 'seo', 'indexHooks'];
++const MODULES = ['revisions', 'citations', 'media', 'discussion', 'schedule', 'authorship', 'taxonomy', 'seo', 'indexHooks', 'ingest'];
const LONGEST = `p${'x'.repeat(39)}`; // PREFIX_RE allows 40 characters
const all = (prefix) => publishing.schema(Object.fromEntries(MODULES.map((m) => [m, prefix])));
@@@ -31,11 +32,14 @@ test('every storing module exports its DDL; the root schema() joins them', () =>
for (const m of ['citations', 'media', 'discussion', 'schedule', 'authorship', 'taxonomy']) assert.strictEqual(typeof require(`../lib/${m}`).schema, 'function', m);
assert.strictEqual(typeof require('../lib/seo').redirectsSchema, 'function');
assert.strictEqual(typeof require('../lib/index-hooks').sequencerSchema, 'function');
++ assert.strictEqual(typeof require('../lib/ingest').schema, 'function');
const text = publishing.schema({ revisions: ['wiki_page', 'wiki_talk'], seo: 'wiki_page' });
assert.match(text, /-- openvibe-publishing\/revisions \(prefix wiki_page\)/);
assert.match(text, /CREATE TABLE IF NOT EXISTS wiki_talk_revisions/);
assert.match(text, /CREATE TABLE IF NOT EXISTS wiki_page_redirects/);
++ assert.match(publishing.schema({ ingest: 'news' }), /-- openvibe-publishing\/ingest \(prefix news\)\nCREATE TABLE IF NOT EXISTS news_ingest_cursor/);
assert.throws(() => publishing.schema({ ssr: 'x' }), /no store in module "ssr"/);
++ assert.throws(() => publishing.schema({ publication: 'x' }), /no store in module "publication"/);
assert.throws(() => publishing.schema({ revisions: 'Bad Prefix' }), /prefix/);
assert.strictEqual(require('../lib/media').schema('blog_post'), createAttachmentStore({ query() {}, tx() {}, many() {}, maybe() {}, exec() {}, sql }, { prefix: 'blog_post' }).schema());
});
@@@ -65,6 +69,25 @@ test('the DDL is idempotent and applies as a service migration (db.migrate), onc
assert.ok(await db.value(sql`SELECT to_regclass('blog_revisions') IS NOT NULL AS ok`));
});
++test('the ingest cursor DDL: one <prefix>_ingest_cursor keyed by feed name, idempotent, served by its PK', async () => {
++ const db = await openDb();
++ const cursor = await createChangeCursor(db, { prefix: 'news', now: fakeClock() }).ensureSchema();
++ assert.strictEqual(cursor.table, 'news_ingest_cursor');
++ await db.query(cursor.schema()); // idempotent
++ const cols = await db.many(sql`SELECT column_name, data_type, collation_name FROM information_schema.columns WHERE table_name = 'news_ingest_cursor' ORDER BY ordinal_position`);
++ assert.deepStrictEqual(cols.map((c) => c.column_name), ['name', 'cursor', 'updated_at']);
++ assert.deepStrictEqual(cols.map((c) => c.data_type), ['text', 'bigint', 'bigint']);
++ assert.strictEqual(cols[0].collation_name, 'C');
++ assert.strictEqual(await cursor.get('default'), 0);
++ await db.tx((t) => cursor.set(t, 'sources', 7));
++ assert.strictEqual(await cursor.get('sources'), 7);
++ await db.tx((t) => cursor.set(t, 'sources', 3));
++ assert.strictEqual(await cursor.get('sources'), 7, 'the cursor never goes back');
++ await db.exec(sql`SET enable_seqscan = off`);
++ const plan = await db.one(sql`EXPLAIN SELECT cursor FROM news_ingest_cursor WHERE name = 'sources'`);
++ assert.match(String(Object.values(plan)[0]), /Index/, 'the primary key serves the cursor lookup');
++});
++
test('PostgreSQL types: bigint identities, jsonb, bigint epoch ms, timestamptz for datetime text', async () => {
const db = await openDb();
await db.query(all('t'));